2024-02-19T14:40:07.764 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T14:40:07.764 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468 --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} --owner scheduled_leonidus@teuthology --verbose --name leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/orig.config.yaml 2024-02-19T14:40:07.808 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T14:40:07.894 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468 branch: wip-lusov-qdb description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} email: null first_in_suite: false job_id: '7566468' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_DAMAGE\) - \(FS_DEGRADED\) sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: disabled: true install: ceph: flavor: default sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee owner: scheduled_leonidus@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 2280 sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee sleep_before_teardown: 0 subset: 111/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEFJ716bM4kXdY3SWaq4E/9AKO2aR9oARi5WQKiL9KctLMdfkOuM0QpNZUyOUbjV2DvVftK/40LaKvuq62Mw7tA= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2zy5WOe4j8G1rIhiUDwy2DrvGN7z8LqJEVTU1UzoJPp1BXF97WPZD0W3wQVGOZ1sfaE4HPcOT5L0Db8pL1jFQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_14:33:01 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-19T14:40:07.894 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa; will attempt to use it 2024-02-19T14:40:07.894 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks 2024-02-19T14:40:07.895 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T14:40:07.896 INFO:teuthology.task.internal:Checking packages... 2024-02-19T14:40:07.920 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee' 2024-02-19T14:40:07.920 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T14:40:07.921 INFO:teuthology.packaging:ref: None 2024-02-19T14:40:07.921 INFO:teuthology.packaging:tag: None 2024-02-19T14:40:07.921 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-02-19T14:40:07.921 INFO:teuthology.packaging:sha1: 9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T14:40:07.921 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-02-19T14:40:08.070 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1451.g9f8908ee 2024-02-19T14:40:08.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T14:40:08.078 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T14:40:08.078 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T14:40:08.087 INFO:teuthology.task.internal:Saving configuration 2024-02-19T14:40:08.104 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T14:40:08.112 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T14:40:08.135 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 14:34:19.578089', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEFJ716bM4kXdY3SWaq4E/9AKO2aR9oARi5WQKiL9KctLMdfkOuM0QpNZUyOUbjV2DvVftK/40LaKvuq62Mw7tA='} 2024-02-19T14:40:08.162 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 14:34:19.577033', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2zy5WOe4j8G1rIhiUDwy2DrvGN7z8LqJEVTU1UzoJPp1BXF97WPZD0W3wQVGOZ1sfaE4HPcOT5L0Db8pL1jFQ='} 2024-02-19T14:40:08.162 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T14:40:08.168 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-19T14:40:08.168 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-02-19T14:40:08.168 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T14:40:08.234 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f95afd7b0d0>, signals=[15]) 2024-02-19T14:40:08.235 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T14:40:08.243 INFO:teuthology.task.internal:Opening connections... 2024-02-19T14:40:08.243 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:40:08.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:40:08.337 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:40:08.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:40:08.425 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T14:40:08.431 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-02-19T14:40:08.448 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-02-19T14:40:08.449 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-02-19T14:40:08.504 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-02-19T14:40:08.504 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-02-19T14:40:08.504 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-02-19T14:40:08.504 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-02-19T14:40:08.504 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-02-19T14:40:08.504 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T14:40:08.505 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T14:40:08.505 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-02-19T14:40:08.526 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-02-19T14:40:08.543 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-02-19T14:40:08.543 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-02-19T14:40:08.598 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-02-19T14:40:08.598 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-02-19T14:40:08.598 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T14:40:08.599 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T14:40:08.599 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-02-19T14:40:08.625 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T14:40:08.635 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T14:40:08.643 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T14:40:08.644 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-02-19T14:40:08.646 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-02-19T14:40:08.661 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T14:40:08.669 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T14:40:08.669 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-02-19T14:40:08.702 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-02-19T14:40:08.729 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T14:40:08.780 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T14:40:08.796 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-02-19T14:40:08.796 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2024-02-19T14:40:08.796 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-02-19T14:40:08.796 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T14:40:08.797 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T14:40:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T14:40:08.812 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-02-19T14:40:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T14:40:08.813 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-19T14:40:08.867 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-361.el9.x86_64 2024-02-19T14:40:08.867 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-361.el9.x86_64 2024-02-19T14:40:08.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-02-19T14:40:08.869 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-02-19T14:40:08.869 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-02-19T14:40:08.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-02-19T14:40:10.929 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-19T14:40:11.076 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-19T14:40:12.337 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-02-19T14:40:12.707 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-02-19T14:40:20.911 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 992 kB/s | 8.0 MB 00:08 2024-02-19T14:40:22.540 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 861 kB/s | 8.0 MB 00:09 2024-02-19T14:40:22.929 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-02-19T14:40:26.133 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-02-19T14:40:29.029 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-02-19T14:40:30.355 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 15 kB 00:00 2024-02-19T14:40:32.215 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-02-19T14:40:37.173 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-02-19T14:40:38.525 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 704 kB/s | 6.1 MB 00:08 2024-02-19T14:40:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T14:40:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T14:40:39.453 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-02-19T14:40:39.469 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-02-19T14:40:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:40:39.471 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 100 M 2024-02-19T14:40:39.471 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 123 M 2024-02-19T14:40:39.471 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-02-19T14:40:39.816 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 18 MB/s | 6.1 MB 00:00 2024-02-19T14:40:39.900 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-02-19T14:40:40.350 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-19T14:40:41.016 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 28 MB/s | 33 MB 00:01 2024-02-19T14:40:41.188 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-02-19T14:40:41.492 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 19 MB/s | 39 MB 00:02 2024-02-19T14:40:41.495 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-02-19T14:40:41.495 INFO:teuthology.orchestra.run.smithi078.stdout:Total 49 MB/s | 100 MB 00:02 2024-02-19T14:40:41.495 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-02-19T14:40:41.877 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-02-19T14:40:41.877 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-02-19T14:40:42.900 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-02-19T14:40:42.900 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-02-19T14:40:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-02-19T14:40:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T14:40:46.143 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T14:40:46.244 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-02-19T14:40:48.417 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T14:40:48.417 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T14:40:48.513 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-19T14:40:48.520 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-02-19T14:40:48.521 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:40:48.522 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 100 M 2024-02-19T14:40:48.522 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 123 M 2024-02-19T14:40:48.522 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-19T14:40:48.886 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 17 MB/s | 6.1 MB 00:00 2024-02-19T14:40:48.911 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T14:40:49.096 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T14:40:49.428 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-19T14:40:50.729 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-19T14:40:51.029 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-core-5.14.0-421.el9.x86_6 16 MB/s | 33 MB 00:02 2024-02-19T14:40:51.032 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-19T14:40:51.032 INFO:teuthology.orchestra.run.smithi150.stdout:Total 40 MB/s | 100 MB 00:02 2024-02-19T14:40:51.032 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-19T14:40:51.401 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-19T14:40:51.402 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-19T14:40:52.415 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-19T14:40:52.415 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-19T14:40:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T14:40:54.035 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:40:54.840 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-19T14:40:55.543 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T14:40:55.688 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T14:40:58.467 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T14:40:58.639 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T14:40:58.798 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:03.350 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T14:41:03.602 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:08.321 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:28.113 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:28.113 INFO:teuthology.orchestra.run.smithi078.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T14:41:28.113 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:41:28.182 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:28.646 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:28.647 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T14:41:28.647 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T14:41:28.647 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T14:41:29.039 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:29.039 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:41:29.039 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-02-19T14:41:29.039 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T14:41:29.039 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T14:41:29.040 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T14:41:29.040 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T14:41:29.040 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:41:29.040 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-02-19T14:41:29.589 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T14:41:29.725 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T14:41:29.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T14:41:29.725 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T14:41:29.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T14:41:29.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T14:41:29.725 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-02-19T14:41:29.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T14:41:29.725 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T14:41:29.834 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T14:41:29.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T14:41:29.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-02-19T14:41:29.868 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-02-19T14:41:29.868 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 68.el9 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8264254 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-02-19T14:41:29.869 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:41:29.870 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T14:41:29.871 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T14:41:29.871 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-421.el9.x86_64 2024-02-19T14:41:29.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T14:41:30.510 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-02-19T14:41:31.852 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T14:41:31.889 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-02-19T14:41:31.891 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-02-19T14:41:31.906 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.I1VWzk77Hh 2024-02-19T14:41:31.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I1VWzk77Hh 2024-02-19T14:41:31.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.I1VWzk77Hh 2024-02-19T14:41:32.119 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.I1VWzk77Hh is 6KB 2024-02-19T14:41:32.168 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.I1VWzk77Hh 2024-02-19T14:41:32.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T14:41:32.247 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T14:41:32.247 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T14:41:32.247 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T14:41:32.247 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T14:41:32.249 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T14:41:32.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-02-19T14:41:37.578 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:37.578 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T14:41:37.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:41:37.649 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:38.048 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:38.048 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T14:41:38.048 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T14:41:38.048 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:41:38.483 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-19T14:41:39.006 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T14:41:39.136 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T14:41:39.137 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T14:41:39.137 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T14:41:39.137 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T14:41:39.137 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T14:41:39.138 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-02-19T14:41:39.138 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T14:41:39.138 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T14:41:39.246 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T14:41:39.246 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T14:41:39.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T14:41:39.280 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T14:41:39.281 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T14:41:39.282 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-02-19T14:41:39.282 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:41:39.282 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T14:41:39.282 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T14:41:39.283 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-421.el9.x86_64 2024-02-19T14:41:39.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T14:41:39.977 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-02-19T14:41:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T14:41:41.480 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-02-19T14:41:41.481 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-19T14:41:41.496 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.XwtDfyAs45 2024-02-19T14:41:41.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XwtDfyAs45 2024-02-19T14:41:41.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.XwtDfyAs45 2024-02-19T14:41:41.699 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.XwtDfyAs45 is 6KB 2024-02-19T14:41:41.719 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.XwtDfyAs45 2024-02-19T14:41:41.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T14:41:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T14:41:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T14:41:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T14:41:41.807 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T14:41:41.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T14:41:41.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-02-19T14:42:02.412 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-02-19T14:42:02.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:42:02.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:42:11.972 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-02-19T14:42:11.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-19T14:42:11.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:42:36.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-02-19T14:42:45.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:42:45.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:42:46.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-02-19T14:42:49.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-02-19T14:42:55.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-19T14:42:55.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:43:01.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:43:01.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:43:13.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-02-19T14:43:25.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-19T14:43:25.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:43:27.312 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-19T14:43:27.773 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-19T14:43:27.773 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-421.el9.x86_64"... 2024-02-19T14:43:27.773 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-19T14:43:27.787 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-421.el9.x86_64 2024-02-19T14:43:27.787 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T14:43:27.787 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T14:43:27.787 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T14:43:28.788 DEBUG:teuthology.parallel:result is None 2024-02-19T14:44:01.059 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T14:44:16.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:44:16.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:44:16.504 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-02-19T14:44:16.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:44:16.904 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-421.el9.x86_64"... 2024-02-19T14:44:16.904 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-02-19T14:44:16.919 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-421.el9.x86_64 2024-02-19T14:44:16.919 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T14:44:16.919 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T14:44:16.919 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T14:44:17.920 DEBUG:teuthology.parallel:result is None 2024-02-19T14:44:17.920 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T14:44:17.929 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T14:44:17.929 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T14:44:17.931 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T14:44:17.952 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T14:44:17.961 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T14:44:17.967 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T14:44:17.968 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T14:44:17.989 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T14:44:18.028 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T14:44:18.034 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T14:44:18.034 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T14:44:18.064 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T14:44:18.152 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T14:44:18.169 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T14:44:18.188 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T14:44:18.204 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T14:44:18.205 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T14:44:18.213 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T14:44:18.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T14:44:18.215 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T14:44:18.246 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T14:44:18.259 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T14:44:18.259 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T14:44:18.286 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T14:44:18.303 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T14:44:18.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T14:44:18.430 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T14:44:18.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T14:44:18.556 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:44:18.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T14:44:18.624 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T14:44:18.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T14:44:18.726 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T14:44:18.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T14:44:18.858 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:44:18.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T14:44:18.929 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-02-19T14:44:18.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-02-19T14:44:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T14:44:19.034 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T14:44:19.435 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T14:44:19.443 INFO:teuthology.task.internal:Starting timer... 2024-02-19T14:44:19.443 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T14:44:19.454 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T14:44:19.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-02-19T14:44:19.515 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-02-19T14:44:19.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-02-19T14:44:19.564 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-02-19T14:44:19.564 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T14:44:19.564 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-02-19T14:44:19.591 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-02-19T14:44:19.592 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-02-19T14:44:19.626 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-02-19T14:44:19.627 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2024-02-19T14:44:19.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T14:44:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T14:44:19.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T14:44:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T14:44:19.696 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T14:44:19.696 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-02-19T14:44:19.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-02-19T14:44:19.771 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T14:44:19.782 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T14:44:19.799 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T14:44:19.801 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T14:48:33.628 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-02-19T14:48:33.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:48:33.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:48:33.723 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-02-19T14:48:33.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-02-19T14:48:33.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-19T14:48:33.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T14:48:33.894 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-19T14:48:33.974 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-19T14:48:33.974 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T14:48:33.984 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T14:48:33.984 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T14:48:33.984 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T14:48:33.987 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T14:48:33.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T14:48:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T14:48:34.031 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T14:48:34.045 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T14:48:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-02-19T14:48:34.061 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T14:48:34.074 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-02-19T14:48:34.090 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T14:48:34.090 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-02-19T14:48:34.104 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-02-19T14:48:34.106 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T14:48:34.120 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T14:48:34.136 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T14:48:34.159 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-02-19T14:48:34.162 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T14:48:34.162 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-02-19T14:48:34.162 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.162 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.162 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.162 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.192 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-19T14:48:34.194 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T14:48:34.194 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-19T14:48:34.195 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.195 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.195 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.195 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T14:48:34.196 INFO:teuthology.run_tasks:Running task install... 2024-02-19T14:48:34.204 DEBUG:teuthology.task.install:project ceph 2024-02-19T14:48:34.204 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}} 2024-02-19T14:48:34.204 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'} 2024-02-19T14:48:34.205 INFO:teuthology.task.install:Using flavor: default 2024-02-19T14:48:34.210 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T14:48:34.210 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-02-19T14:48:34.211 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'tag': None, 'wait_for_package': False} 2024-02-19T14:48:34.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T14:48:34.213 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'tag': None, 'wait_for_package': False} 2024-02-19T14:48:34.213 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T14:48:34.358 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/ 2024-02-19T14:48:34.358 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1451.g9f8908ee 2024-02-19T14:48:34.363 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/ 2024-02-19T14:48:34.363 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1451.g9f8908ee 2024-02-19T14:48:34.572 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T14:48:34.572 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:48:34.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T14:48:34.581 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-qdb/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T14:48:34.581 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:48:34.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T14:48:34.599 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-19T14:48:34.600 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T14:48:34.609 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-19T14:48:34.609 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T14:48:34.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T14:48:34.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T14:48:34.748 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T14:48:34.756 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T14:48:34.814 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-02-19T14:48:34.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-02-19T14:48:34.822 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-02-19T14:48:34.823 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-02-19T14:48:35.053 INFO:teuthology.orchestra.run.smithi150.stdout:74 files removed 2024-02-19T14:48:35.059 INFO:teuthology.orchestra.run.smithi078.stdout:74 files removed 2024-02-19T14:48:35.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-19T14:48:35.086 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-19T14:48:35.892 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 168 kB/s | 83 kB 00:00 2024-02-19T14:48:35.905 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 166 kB/s | 83 kB 00:00 2024-02-19T14:48:36.273 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-02-19T14:48:36.351 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-02-19T14:48:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-02-19T14:48:36.727 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-02-19T14:48:37.229 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-19T14:48:37.667 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.7 MB 00:00 2024-02-19T14:48:38.871 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-02-19T14:48:39.117 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-02-19T14:48:40.558 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-02-19T14:48:44.086 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-02-19T14:48:49.377 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 803 kB/s | 8.0 MB 00:10 2024-02-19T14:48:52.811 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-02-19T14:48:56.930 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-02-19T14:48:58.330 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 15 kB 00:00 2024-02-19T14:49:03.633 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 387 kB/s | 6.1 MB 00:16 2024-02-19T14:49:13.656 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 20 MB 00:15 2024-02-19T14:49:18.539 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-19T14:49:21.024 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 947 B/s | 15 kB 00:16 2024-02-19T14:49:21.100 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-19T14:49:21.100 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-19T14:49:21.112 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-19T14:49:21.113 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-19T14:49:21.114 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-19T14:49:21.115 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-19T14:49:21.117 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-19T14:49:21.119 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-19T14:49:21.120 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-19T14:49:21.200 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-19T14:49:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-02-19T14:49:21.224 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-02-19T14:49:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-02-19T14:49:21.224 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-19T14:49:21.224 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-02-19T14:49:21.224 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 6.6 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 5.3 M 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 887 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 145 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 1.5 M 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 138 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 1.8 M 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 7.4 M 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 52 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 9.7 M 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 47 M 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 264 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 748 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 217 k 2024-02-19T14:49:21.225 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 27 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 18 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 31 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 744 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 127 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-02-19T14:49:21.226 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 162 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 320 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 298 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 99 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 85 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 3.1 M 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 171 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager x86_64 1:1.45.91-1.el9 CentOS-BaseOS 2.3 M 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm x86_64 1:1.45.91-1.el9 CentOS-BaseOS 1.8 M 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team x86_64 1:1.45.91-1.el9 CentOS-BaseOS 41 k 2024-02-19T14:49:21.227 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui x86_64 1:1.45.91-1.el9 CentOS-BaseOS 246 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 3.4 M 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 3.1 M 2024-02-19T14:49:21.228 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 20 M 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 31 k 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 2.3 M 2024-02-19T14:49:21.229 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 248 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 4.7 M 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 17 M 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 15 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 25 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 163 k 2024-02-19T14:49:21.230 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 498 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 5.1 M 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T14:49:21.231 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 45 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 131 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T14:49:21.232 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T14:49:21.233 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T14:49:21.234 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T14:49:21.235 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-02-19T14:49:21.236 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout:Install 150 Packages 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 26 Packages 2024-02-19T14:49:21.237 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:49:21.238 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 223 M 2024-02-19T14:49:21.238 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-19T14:49:21.807 INFO:teuthology.orchestra.run.smithi150.stdout:(1/176): ceph-19.0.0-1451.g9f8908ee.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-02-19T14:49:23.169 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:02 2024-02-19T14:49:23.282 INFO:teuthology.orchestra.run.smithi150.stdout:(2/176): ceph-fuse-19.0.0-1451.g9f8908ee.el9.x8 601 kB/s | 887 kB 00:01 2024-02-19T14:49:23.490 INFO:teuthology.orchestra.run.smithi150.stdout:(3/176): ceph-immutable-object-cache-19.0.0-145 698 kB/s | 145 kB 00:00 2024-02-19T14:49:25.091 INFO:teuthology.orchestra.run.smithi150.stdout:(4/176): ceph-mds-19.0.0-1451.g9f8908ee.el9.x86 1.4 MB/s | 2.3 MB 00:01 2024-02-19T14:49:25.691 INFO:teuthology.orchestra.run.smithi150.stdout:(5/176): ceph-base-19.0.0-1451.g9f8908ee.el9.x8 1.2 MB/s | 5.3 MB 00:04 2024-02-19T14:49:25.727 INFO:teuthology.orchestra.run.smithi150.stdout:(6/176): ceph-mgr-19.0.0-1451.g9f8908ee.el9.x86 2.3 MB/s | 1.5 MB 00:00 2024-02-19T14:49:28.110 INFO:teuthology.orchestra.run.smithi150.stdout:(7/176): ceph-mon-19.0.0-1451.g9f8908ee.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-02-19T14:49:28.192 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-02-19T14:49:28.877 INFO:teuthology.orchestra.run.smithi150.stdout:(8/176): ceph-osd-19.0.0-1451.g9f8908ee.el9.x86 5.3 MB/s | 17 MB 00:03 2024-02-19T14:49:28.944 INFO:teuthology.orchestra.run.smithi150.stdout:(9/176): ceph-selinux-19.0.0-1451.g9f8908ee.el9 378 kB/s | 25 kB 00:00 2024-02-19T14:49:30.721 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-19T14:49:30.722 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-19T14:49:30.736 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-19T14:49:30.737 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-19T14:49:30.738 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-19T14:49:30.739 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-19T14:49:30.741 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-19T14:49:30.744 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-19T14:49:30.746 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-19T14:49:30.824 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-02-19T14:49:30.844 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-02-19T14:49:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-02-19T14:49:30.844 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 6.6 k 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 5.3 M 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 887 k 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 145 k 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 1.5 M 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 138 k 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 1.8 M 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 7.4 M 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 52 k 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 9.7 M 2024-02-19T14:49:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 47 M 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 264 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 748 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 217 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 27 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 18 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 31 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 744 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 127 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-02-19T14:49:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 162 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 320 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 298 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 99 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 85 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 3.1 M 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 171 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-02-19T14:49:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager x86_64 1:1.45.91-1.el9 CentOS-BaseOS 2.3 M 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm x86_64 1:1.45.91-1.el9 CentOS-BaseOS 1.8 M 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team x86_64 1:1.45.91-1.el9 CentOS-BaseOS 41 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui x86_64 1:1.45.91-1.el9 CentOS-BaseOS 246 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-02-19T14:49:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 3.4 M 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 3.1 M 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-02-19T14:49:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 20 M 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 31 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 2.3 M 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 248 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 4.7 M 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 17 M 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1451.g9f8908ee.el9 ceph-noarch 15 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 25 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T14:49:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 163 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T14:49:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 498 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 5.1 M 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-19T14:49:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 45 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:19.0.0-1451.g9f8908ee.el9 ceph 131 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-19T14:49:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T14:49:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-19T14:49:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T14:49:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T14:49:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout:Install 150 Packages 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 26 Packages 2024-02-19T14:49:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:49:30.859 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 223 M 2024-02-19T14:49:30.859 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-02-19T14:49:30.961 INFO:teuthology.orchestra.run.smithi150.stdout:(10/176): ceph-common-19.0.0-1451.g9f8908ee.el9 2.1 MB/s | 20 MB 00:09 2024-02-19T14:49:31.170 INFO:teuthology.orchestra.run.smithi150.stdout:(11/176): cephfs-mirror-19.0.0-1451.g9f8908ee.e 1.0 MB/s | 217 kB 00:00 2024-02-19T14:49:31.278 INFO:teuthology.orchestra.run.smithi150.stdout:(12/176): cephfs-shell-19.0.0-1451.g9f8908ee.el 250 kB/s | 27 kB 00:00 2024-02-19T14:49:31.403 INFO:teuthology.orchestra.run.smithi150.stdout:(13/176): libcephfs-devel-19.0.0-1451.g9f8908ee 249 kB/s | 31 kB 00:00 2024-02-19T14:49:31.795 INFO:teuthology.orchestra.run.smithi150.stdout:(14/176): libcephfs2-19.0.0-1451.g9f8908ee.el9. 1.9 MB/s | 744 kB 00:00 2024-02-19T14:49:31.887 INFO:teuthology.orchestra.run.smithi078.stdout:(1/176): ceph-19.0.0-1451.g9f8908ee.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-02-19T14:49:31.929 INFO:teuthology.orchestra.run.smithi150.stdout:(15/176): libcephsqlite-19.0.0-1451.g9f8908ee.e 1.2 MB/s | 163 kB 00:00 2024-02-19T14:49:32.046 INFO:teuthology.orchestra.run.smithi150.stdout:(16/176): librados-devel-19.0.0-1451.g9f8908ee. 1.1 MB/s | 127 kB 00:00 2024-02-19T14:49:32.329 INFO:teuthology.orchestra.run.smithi150.stdout:(17/176): libradosstriper1-19.0.0-1451.g9f8908e 1.7 MB/s | 498 kB 00:00 2024-02-19T14:49:33.005 INFO:teuthology.orchestra.run.smithi150.stdout:(18/176): ceph-radosgw-19.0.0-1451.g9f8908ee.el 2.0 MB/s | 9.7 MB 00:04 2024-02-19T14:49:33.122 INFO:teuthology.orchestra.run.smithi150.stdout:(19/176): python3-ceph-argparse-19.0.0-1451.g9f 388 kB/s | 45 kB 00:00 2024-02-19T14:49:33.243 INFO:teuthology.orchestra.run.smithi078.stdout:(2/176): ceph-fuse-19.0.0-1451.g9f8908ee.el9.x8 653 kB/s | 887 kB 00:01 2024-02-19T14:49:33.263 INFO:teuthology.orchestra.run.smithi150.stdout:(20/176): python3-ceph-common-19.0.0-1451.g9f89 927 kB/s | 131 kB 00:00 2024-02-19T14:49:33.485 INFO:teuthology.orchestra.run.smithi078.stdout:(3/176): ceph-immutable-object-cache-19.0.0-145 600 kB/s | 145 kB 00:00 2024-02-19T14:49:33.497 INFO:teuthology.orchestra.run.smithi150.stdout:(21/176): python3-cephfs-19.0.0-1451.g9f8908ee. 693 kB/s | 162 kB 00:00 2024-02-19T14:49:33.697 INFO:teuthology.orchestra.run.smithi150.stdout:(22/176): python3-rados-19.0.0-1451.g9f8908ee.e 1.6 MB/s | 320 kB 00:00 2024-02-19T14:49:33.923 INFO:teuthology.orchestra.run.smithi150.stdout:(23/176): python3-rbd-19.0.0-1451.g9f8908ee.el9 1.3 MB/s | 298 kB 00:00 2024-02-19T14:49:34.031 INFO:teuthology.orchestra.run.smithi150.stdout:(24/176): python3-rgw-19.0.0-1451.g9f8908ee.el9 917 kB/s | 99 kB 00:00 2024-02-19T14:49:34.148 INFO:teuthology.orchestra.run.smithi150.stdout:(25/176): rbd-fuse-19.0.0-1451.g9f8908ee.el9.x8 730 kB/s | 85 kB 00:00 2024-02-19T14:49:34.852 INFO:teuthology.orchestra.run.smithi078.stdout:(4/176): ceph-base-19.0.0-1451.g9f8908ee.el9.x8 1.6 MB/s | 5.3 MB 00:03 2024-02-19T14:49:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:(5/176): ceph-mds-19.0.0-1451.g9f8908ee.el9.x86 1.5 MB/s | 2.3 MB 00:01 2024-02-19T14:49:35.007 INFO:teuthology.orchestra.run.smithi150.stdout:(26/176): librgw2-19.0.0-1451.g9f8908ee.el9.x86 1.9 MB/s | 5.1 MB 00:02 2024-02-19T14:49:35.199 INFO:teuthology.orchestra.run.smithi150.stdout:(27/176): rbd-nbd-19.0.0-1451.g9f8908ee.el9.x86 893 kB/s | 171 kB 00:00 2024-02-19T14:49:35.330 INFO:teuthology.orchestra.run.smithi150.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1451.g 240 kB/s | 31 kB 00:00 2024-02-19T14:49:35.427 INFO:teuthology.orchestra.run.smithi078.stdout:(6/176): ceph-mgr-19.0.0-1451.g9f8908ee.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-02-19T14:49:35.432 INFO:teuthology.orchestra.run.smithi150.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1451.g9f8908e 1.3 MB/s | 138 kB 00:00 2024-02-19T14:49:35.716 INFO:teuthology.orchestra.run.smithi150.stdout:(30/176): rbd-mirror-19.0.0-1451.g9f8908ee.el9. 2.0 MB/s | 3.1 MB 00:01 2024-02-19T14:49:36.403 INFO:teuthology.orchestra.run.smithi078.stdout:(7/176): ceph-mon-19.0.0-1451.g9f8908ee.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-02-19T14:49:36.741 INFO:teuthology.orchestra.run.smithi150.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1451.g9f890 1.4 MB/s | 1.8 MB 00:01 2024-02-19T14:49:37.017 INFO:teuthology.orchestra.run.smithi150.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1451.g9f 903 kB/s | 248 kB 00:00 2024-02-19T14:49:37.125 INFO:teuthology.orchestra.run.smithi150.stdout:(33/176): ceph-mgr-rook-19.0.0-1451.g9f8908ee.e 483 kB/s | 52 kB 00:00 2024-02-19T14:49:37.226 INFO:teuthology.orchestra.run.smithi150.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1451.g9 153 kB/s | 15 kB 00:00 2024-02-19T14:49:37.718 INFO:teuthology.orchestra.run.smithi150.stdout:(35/176): ceph-volume-19.0.0-1451.g9f8908ee.el9 537 kB/s | 264 kB 00:00 2024-02-19T14:49:38.193 INFO:teuthology.orchestra.run.smithi150.stdout:(36/176): cephadm-19.0.0-1451.g9f8908ee.el9.noa 1.5 MB/s | 748 kB 00:00 2024-02-19T14:49:38.203 INFO:teuthology.orchestra.run.smithi078.stdout:(8/176): ceph-radosgw-19.0.0-1451.g9f8908ee.el9 5.4 MB/s | 9.7 MB 00:01 2024-02-19T14:49:38.251 INFO:teuthology.orchestra.run.smithi150.stdout:(37/176): cephfs-top-19.0.0-1451.g9f8908ee.el9. 314 kB/s | 18 kB 00:00 2024-02-19T14:49:38.270 INFO:teuthology.orchestra.run.smithi078.stdout:(9/176): ceph-selinux-19.0.0-1451.g9f8908ee.el9 378 kB/s | 25 kB 00:00 2024-02-19T14:49:38.396 INFO:teuthology.orchestra.run.smithi150.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.5 MB/s | 662 kB 00:00 2024-02-19T14:49:38.452 INFO:teuthology.orchestra.run.smithi150.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2024-02-19T14:49:38.486 INFO:teuthology.orchestra.run.smithi150.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-02-19T14:49:38.519 INFO:teuthology.orchestra.run.smithi150.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 609 kB/s | 20 kB 00:00 2024-02-19T14:49:38.561 INFO:teuthology.orchestra.run.smithi150.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-02-19T14:49:38.595 INFO:teuthology.orchestra.run.smithi150.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 950 kB/s | 32 kB 00:00 2024-02-19T14:49:38.703 INFO:teuthology.orchestra.run.smithi150.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-02-19T14:49:38.745 INFO:teuthology.orchestra.run.smithi150.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 411 kB/s | 17 kB 00:00 2024-02-19T14:49:38.779 INFO:teuthology.orchestra.run.smithi150.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-02-19T14:49:38.813 INFO:teuthology.orchestra.run.smithi150.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-02-19T14:49:38.846 INFO:teuthology.orchestra.run.smithi150.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-02-19T14:49:38.897 INFO:teuthology.orchestra.run.smithi150.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-02-19T14:49:38.939 INFO:teuthology.orchestra.run.smithi150.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-02-19T14:49:38.972 INFO:teuthology.orchestra.run.smithi150.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 735 kB/s | 24 kB 00:00 2024-02-19T14:49:39.006 INFO:teuthology.orchestra.run.smithi150.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-02-19T14:49:39.048 INFO:teuthology.orchestra.run.smithi150.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-02-19T14:49:39.081 INFO:teuthology.orchestra.run.smithi150.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-02-19T14:49:39.115 INFO:teuthology.orchestra.run.smithi150.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-02-19T14:49:39.240 INFO:teuthology.orchestra.run.smithi150.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-02-19T14:49:39.333 INFO:teuthology.orchestra.run.smithi150.stdout:(57/176): openssl-devel-3.0.7-25.el9.x86_64.rpm 33 MB/s | 3.0 MB 00:00 2024-02-19T14:49:39.416 INFO:teuthology.orchestra.run.smithi150.stdout:(58/176): ceph-mgr-diskprediction-local-19.0.0- 2.0 MB/s | 7.4 MB 00:03 2024-02-19T14:49:39.458 INFO:teuthology.orchestra.run.smithi150.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-02-19T14:49:39.491 INFO:teuthology.orchestra.run.smithi150.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-02-19T14:49:39.516 INFO:teuthology.orchestra.run.smithi150.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-02-19T14:49:39.642 INFO:teuthology.orchestra.run.smithi150.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-02-19T14:49:39.676 INFO:teuthology.orchestra.run.smithi150.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-02-19T14:49:39.701 INFO:teuthology.orchestra.run.smithi150.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-02-19T14:49:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-02-19T14:49:39.843 INFO:teuthology.orchestra.run.smithi150.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 227 kB 00:00 2024-02-19T14:49:39.860 INFO:teuthology.orchestra.run.smithi150.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 588 kB/s | 24 kB 00:00 2024-02-19T14:49:39.877 INFO:teuthology.orchestra.run.smithi150.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 538 kB/s | 18 kB 00:00 2024-02-19T14:49:39.903 INFO:teuthology.orchestra.run.smithi150.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 144 kB 00:00 2024-02-19T14:49:39.953 INFO:teuthology.orchestra.run.smithi150.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-02-19T14:49:40.037 INFO:teuthology.orchestra.run.smithi150.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-02-19T14:49:40.062 INFO:teuthology.orchestra.run.smithi150.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-02-19T14:49:40.087 INFO:teuthology.orchestra.run.smithi150.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-02-19T14:49:40.112 INFO:teuthology.orchestra.run.smithi150.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-02-19T14:49:40.138 INFO:teuthology.orchestra.run.smithi150.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-02-19T14:49:40.163 INFO:teuthology.orchestra.run.smithi150.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-02-19T14:49:40.188 INFO:teuthology.orchestra.run.smithi150.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-02-19T14:49:40.214 INFO:teuthology.orchestra.run.smithi150.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-02-19T14:49:40.239 INFO:teuthology.orchestra.run.smithi150.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 952 kB/s | 47 kB 00:00 2024-02-19T14:49:40.264 INFO:teuthology.orchestra.run.smithi150.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2024-02-19T14:49:40.406 INFO:teuthology.orchestra.run.smithi150.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 277 kB/s | 39 kB 00:00 2024-02-19T14:49:40.573 INFO:teuthology.orchestra.run.smithi150.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-19T14:49:40.598 INFO:teuthology.orchestra.run.smithi150.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 216 kB/s | 41 kB 00:00 2024-02-19T14:49:40.624 INFO:teuthology.orchestra.run.smithi150.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-02-19T14:49:40.649 INFO:teuthology.orchestra.run.smithi150.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 675 kB/s | 34 kB 00:00 2024-02-19T14:49:40.674 INFO:teuthology.orchestra.run.smithi150.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 866 kB/s | 43 kB 00:00 2024-02-19T14:49:40.699 INFO:teuthology.orchestra.run.smithi150.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-02-19T14:49:40.726 INFO:teuthology.orchestra.run.smithi150.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-02-19T14:49:40.750 INFO:teuthology.orchestra.run.smithi150.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-02-19T14:49:40.775 INFO:teuthology.orchestra.run.smithi150.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-02-19T14:49:40.801 INFO:teuthology.orchestra.run.smithi150.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 874 kB/s | 44 kB 00:00 2024-02-19T14:49:40.811 INFO:teuthology.orchestra.run.smithi078.stdout:(10/176): ceph-osd-19.0.0-1451.g9f8908ee.el9.x8 3.1 MB/s | 17 MB 00:05 2024-02-19T14:49:40.826 INFO:teuthology.orchestra.run.smithi150.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-02-19T14:49:40.851 INFO:teuthology.orchestra.run.smithi150.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-19T14:49:40.885 INFO:teuthology.orchestra.run.smithi150.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-02-19T14:49:40.910 INFO:teuthology.orchestra.run.smithi150.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-02-19T14:49:40.936 INFO:teuthology.orchestra.run.smithi150.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-02-19T14:49:40.961 INFO:teuthology.orchestra.run.smithi150.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-19T14:49:40.995 INFO:teuthology.orchestra.run.smithi150.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-02-19T14:49:41.011 INFO:teuthology.orchestra.run.smithi150.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-02-19T14:49:41.037 INFO:teuthology.orchestra.run.smithi150.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.9 MB/s | 124 kB 00:00 2024-02-19T14:49:41.062 INFO:teuthology.orchestra.run.smithi150.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-02-19T14:49:41.087 INFO:teuthology.orchestra.run.smithi150.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-02-19T14:49:41.112 INFO:teuthology.orchestra.run.smithi078.stdout:(11/176): cephfs-mirror-19.0.0-1451.g9f8908ee.e 725 kB/s | 217 kB 00:00 2024-02-19T14:49:41.187 INFO:teuthology.orchestra.run.smithi078.stdout:(12/176): cephfs-shell-19.0.0-1451.g9f8908ee.el 361 kB/s | 27 kB 00:00 2024-02-19T14:49:41.270 INFO:teuthology.orchestra.run.smithi078.stdout:(13/176): libcephfs-devel-19.0.0-1451.g9f8908ee 375 kB/s | 31 kB 00:00 2024-02-19T14:49:41.563 INFO:teuthology.orchestra.run.smithi150.stdout:(103/176): ceph-test-19.0.0-1451.g9f8908ee.el9. 3.7 MB/s | 47 MB 00:12 2024-02-19T14:49:41.672 INFO:teuthology.orchestra.run.smithi150.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 403 kB/s | 245 kB 00:00 2024-02-19T14:49:42.037 INFO:teuthology.orchestra.run.smithi078.stdout:(14/176): libcephfs2-19.0.0-1451.g9f8908ee.el9. 971 kB/s | 744 kB 00:00 2024-02-19T14:49:42.080 INFO:teuthology.orchestra.run.smithi150.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 22 kB/s | 22 kB 00:00 2024-02-19T14:49:42.213 INFO:teuthology.orchestra.run.smithi078.stdout:(15/176): libcephsqlite-19.0.0-1451.g9f8908ee.e 933 kB/s | 163 kB 00:00 2024-02-19T14:49:42.297 INFO:teuthology.orchestra.run.smithi150.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 877 kB/s | 548 kB 00:00 2024-02-19T14:49:42.388 INFO:teuthology.orchestra.run.smithi078.stdout:(16/176): librados-devel-19.0.0-1451.g9f8908ee. 725 kB/s | 127 kB 00:00 2024-02-19T14:49:42.405 INFO:teuthology.orchestra.run.smithi150.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-02-19T14:49:42.656 INFO:teuthology.orchestra.run.smithi150.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 537 kB/s | 309 kB 00:00 2024-02-19T14:49:42.755 INFO:teuthology.orchestra.run.smithi078.stdout:(17/176): libradosstriper1-19.0.0-1451.g9f8908e 1.3 MB/s | 498 kB 00:00 2024-02-19T14:49:42.756 INFO:teuthology.orchestra.run.smithi150.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-02-19T14:49:42.823 INFO:teuthology.orchestra.run.smithi150.stdout:(110/176): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-02-19T14:49:42.865 INFO:teuthology.orchestra.run.smithi150.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-02-19T14:49:42.923 INFO:teuthology.orchestra.run.smithi150.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-02-19T14:49:42.999 INFO:teuthology.orchestra.run.smithi150.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-02-19T14:49:43.032 INFO:teuthology.orchestra.run.smithi150.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-02-19T14:49:43.133 INFO:teuthology.orchestra.run.smithi150.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 434 kB/s | 43 kB 00:00 2024-02-19T14:49:43.199 INFO:teuthology.orchestra.run.smithi150.stdout:(116/176): python3-asyncssh-2.13.2-4.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-02-19T14:49:43.233 INFO:teuthology.orchestra.run.smithi150.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-02-19T14:49:43.300 INFO:teuthology.orchestra.run.smithi150.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-02-19T14:49:43.342 INFO:teuthology.orchestra.run.smithi150.stdout:(119/176): protobuf-compiler-3.14.0-13.el9.x86_ 485 kB/s | 863 kB 00:01 2024-02-19T14:49:43.376 INFO:teuthology.orchestra.run.smithi150.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-02-19T14:49:43.418 INFO:teuthology.orchestra.run.smithi150.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-02-19T14:49:43.468 INFO:teuthology.orchestra.run.smithi150.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 385 kB/s | 35 kB 00:00 2024-02-19T14:49:43.527 INFO:teuthology.orchestra.run.smithi150.stdout:(123/176): python3-google-auth-2.27.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-02-19T14:49:43.619 INFO:teuthology.orchestra.run.smithi150.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-02-19T14:49:43.644 INFO:teuthology.orchestra.run.smithi150.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-02-19T14:49:43.711 INFO:teuthology.orchestra.run.smithi150.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-02-19T14:49:43.736 INFO:teuthology.orchestra.run.smithi150.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-02-19T14:49:43.803 INFO:teuthology.orchestra.run.smithi150.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-02-19T14:49:43.837 INFO:teuthology.orchestra.run.smithi150.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-02-19T14:49:43.887 INFO:teuthology.orchestra.run.smithi150.stdout:(130/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 557 kB/s | 303 kB 00:00 2024-02-19T14:49:43.912 INFO:teuthology.orchestra.run.smithi150.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-02-19T14:49:43.996 INFO:teuthology.orchestra.run.smithi150.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-02-19T14:49:44.021 INFO:teuthology.orchestra.run.smithi150.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 726 kB/s | 79 kB 00:00 2024-02-19T14:49:44.046 INFO:teuthology.orchestra.run.smithi150.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-02-19T14:49:44.105 INFO:teuthology.orchestra.run.smithi150.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-02-19T14:49:44.130 INFO:teuthology.orchestra.run.smithi150.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-02-19T14:49:44.155 INFO:teuthology.orchestra.run.smithi150.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-02-19T14:49:44.222 INFO:teuthology.orchestra.run.smithi150.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 767 kB/s | 90 kB 00:00 2024-02-19T14:49:44.247 INFO:teuthology.orchestra.run.smithi150.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-02-19T14:49:44.273 INFO:teuthology.orchestra.run.smithi150.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 262 kB/s | 31 kB 00:00 2024-02-19T14:49:44.348 INFO:teuthology.orchestra.run.smithi150.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-02-19T14:49:44.373 INFO:teuthology.orchestra.run.smithi150.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-02-19T14:49:44.398 INFO:teuthology.orchestra.run.smithi150.stdout:(143/176): python3-routes-2.5.1-5.el9.noarch.rp 1.0 MB/s | 188 kB 00:00 2024-02-19T14:49:44.449 INFO:teuthology.orchestra.run.smithi150.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-02-19T14:49:44.474 INFO:teuthology.orchestra.run.smithi150.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-02-19T14:49:44.516 INFO:teuthology.orchestra.run.smithi150.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 764 kB/s | 90 kB 00:00 2024-02-19T14:49:44.566 INFO:teuthology.orchestra.run.smithi150.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-02-19T14:49:44.592 INFO:teuthology.orchestra.run.smithi150.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-02-19T14:49:44.650 INFO:teuthology.orchestra.run.smithi150.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-02-19T14:49:44.701 INFO:teuthology.orchestra.run.smithi150.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-02-19T14:49:44.734 INFO:teuthology.orchestra.run.smithi150.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-02-19T14:49:44.835 INFO:teuthology.orchestra.run.smithi150.stdout:(152/176): NetworkManager-1.45.91-1.el9.x86_64. 23 MB/s | 2.3 MB 00:00 2024-02-19T14:49:44.918 INFO:teuthology.orchestra.run.smithi150.stdout:(153/176): NetworkManager-libnm-1.45.91-1.el9.x 22 MB/s | 1.8 MB 00:00 2024-02-19T14:49:44.952 INFO:teuthology.orchestra.run.smithi150.stdout:(154/176): NetworkManager-team-1.45.91-1.el9.x8 1.2 MB/s | 41 kB 00:00 2024-02-19T14:49:44.995 INFO:teuthology.orchestra.run.smithi150.stdout:(155/176): NetworkManager-tui-1.45.91-1.el9.x86 5.8 MB/s | 246 kB 00:00 2024-02-19T14:49:45.027 INFO:teuthology.orchestra.run.smithi150.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-02-19T14:49:45.061 INFO:teuthology.orchestra.run.smithi150.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-02-19T14:49:45.103 INFO:teuthology.orchestra.run.smithi150.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-02-19T14:49:45.145 INFO:teuthology.orchestra.run.smithi150.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-02-19T14:49:45.195 INFO:teuthology.orchestra.run.smithi150.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-02-19T14:49:45.237 INFO:teuthology.orchestra.run.smithi150.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-02-19T14:49:45.271 INFO:teuthology.orchestra.run.smithi150.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 659 kB/s | 22 kB 00:00 2024-02-19T14:49:45.304 INFO:teuthology.orchestra.run.smithi150.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 107 kB 00:00 2024-02-19T14:49:45.346 INFO:teuthology.orchestra.run.smithi150.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-02-19T14:49:45.388 INFO:teuthology.orchestra.run.smithi150.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-02-19T14:49:45.413 INFO:teuthology.orchestra.run.smithi150.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 2.5 MB/s | 63 kB 00:00 2024-02-19T14:49:45.439 INFO:teuthology.orchestra.run.smithi150.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-02-19T14:49:45.505 INFO:teuthology.orchestra.run.smithi078.stdout:(18/176): ceph-test-19.0.0-1451.g9f8908ee.el9.x 6.5 MB/s | 47 MB 00:07 2024-02-19T14:49:45.514 INFO:teuthology.orchestra.run.smithi150.stdout:(168/176): openssl-3.0.7-25.el9.x86_64.rpm 15 MB/s | 1.2 MB 00:00 2024-02-19T14:49:45.598 INFO:teuthology.orchestra.run.smithi150.stdout:(169/176): openssl-libs-3.0.7-25.el9.x86_64.rpm 26 MB/s | 2.1 MB 00:00 2024-02-19T14:49:45.681 INFO:teuthology.orchestra.run.smithi150.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-02-19T14:49:45.722 INFO:teuthology.orchestra.run.smithi078.stdout:(19/176): ceph-common-19.0.0-1451.g9f8908ee.el9 1.4 MB/s | 20 MB 00:14 2024-02-19T14:49:45.723 INFO:teuthology.orchestra.run.smithi150.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 10 MB/s | 438 kB 00:00 2024-02-19T14:49:45.739 INFO:teuthology.orchestra.run.smithi078.stdout:(20/176): python3-ceph-argparse-19.0.0-1451.g9f 193 kB/s | 45 kB 00:00 2024-02-19T14:49:45.757 INFO:teuthology.orchestra.run.smithi150.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 906 kB/s | 30 kB 00:00 2024-02-19T14:49:45.798 INFO:teuthology.orchestra.run.smithi150.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-02-19T14:49:45.823 INFO:teuthology.orchestra.run.smithi078.stdout:(21/176): librgw2-19.0.0-1451.g9f8908ee.el9.x86 1.7 MB/s | 5.1 MB 00:03 2024-02-19T14:49:45.848 INFO:teuthology.orchestra.run.smithi078.stdout:(22/176): python3-ceph-common-19.0.0-1451.g9f89 1.0 MB/s | 131 kB 00:00 2024-02-19T14:49:45.857 INFO:teuthology.orchestra.run.smithi150.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-02-19T14:49:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:(23/176): python3-cephfs-19.0.0-1451.g9f8908ee. 1.2 MB/s | 162 kB 00:00 2024-02-19T14:49:45.982 INFO:teuthology.orchestra.run.smithi078.stdout:(24/176): python3-rgw-19.0.0-1451.g9f8908ee.el9 916 kB/s | 99 kB 00:00 2024-02-19T14:49:46.032 INFO:teuthology.orchestra.run.smithi078.stdout:(25/176): python3-rbd-19.0.0-1451.g9f8908ee.el9 1.6 MB/s | 298 kB 00:00 2024-02-19T14:49:46.057 INFO:teuthology.orchestra.run.smithi078.stdout:(26/176): python3-rados-19.0.0-1451.g9f8908ee.e 1.3 MB/s | 320 kB 00:00 2024-02-19T14:49:46.082 INFO:teuthology.orchestra.run.smithi078.stdout:(27/176): rbd-fuse-19.0.0-1451.g9f8908ee.el9.x8 849 kB/s | 85 kB 00:00 2024-02-19T14:49:46.183 INFO:teuthology.orchestra.run.smithi078.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1451.g 314 kB/s | 31 kB 00:00 2024-02-19T14:49:46.208 INFO:teuthology.orchestra.run.smithi078.stdout:(29/176): rbd-nbd-19.0.0-1451.g9f8908ee.el9.x86 1.1 MB/s | 171 kB 00:00 2024-02-19T14:49:46.283 INFO:teuthology.orchestra.run.smithi078.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1451.g9f8908e 1.3 MB/s | 138 kB 00:00 2024-02-19T14:49:46.474 INFO:teuthology.orchestra.run.smithi150.stdout:(175/176): librados2-19.0.0-1451.g9f8908ee.el9. 1.8 MB/s | 3.4 MB 00:01 2024-02-19T14:49:46.532 INFO:teuthology.orchestra.run.smithi150.stdout:(176/176): librbd1-19.0.0-1451.g9f8908ee.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-02-19T14:49:46.536 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-19T14:49:46.537 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.8 MB/s | 223 MB 00:25 2024-02-19T14:49:46.585 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-19T14:49:46.750 INFO:teuthology.orchestra.run.smithi078.stdout:(31/176): rbd-mirror-19.0.0-1451.g9f8908ee.el9. 4.3 MB/s | 3.1 MB 00:00 2024-02-19T14:49:46.850 INFO:teuthology.orchestra.run.smithi078.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1451.g9f 2.4 MB/s | 248 kB 00:00 2024-02-19T14:49:46.886 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-19T14:49:46.886 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-19T14:49:46.959 INFO:teuthology.orchestra.run.smithi078.stdout:(33/176): ceph-mgr-rook-19.0.0-1451.g9f8908ee.e 483 kB/s | 52 kB 00:00 2024-02-19T14:49:47.059 INFO:teuthology.orchestra.run.smithi078.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1451.g9 153 kB/s | 15 kB 00:00 2024-02-19T14:49:47.118 INFO:teuthology.orchestra.run.smithi078.stdout:(35/176): ceph-mgr-dashboard-19.0.0-1451.g9f890 2.0 MB/s | 1.8 MB 00:00 2024-02-19T14:49:47.168 INFO:teuthology.orchestra.run.smithi078.stdout:(36/176): ceph-volume-19.0.0-1451.g9f8908ee.el9 2.4 MB/s | 264 kB 00:00 2024-02-19T14:49:47.268 INFO:teuthology.orchestra.run.smithi078.stdout:(37/176): cephfs-top-19.0.0-1451.g9f8908ee.el9. 182 kB/s | 18 kB 00:00 2024-02-19T14:49:47.402 INFO:teuthology.orchestra.run.smithi078.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-02-19T14:49:47.464 INFO:teuthology.orchestra.run.smithi078.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-02-19T14:49:47.494 INFO:teuthology.orchestra.run.smithi078.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-02-19T14:49:47.519 INFO:teuthology.orchestra.run.smithi078.stdout:(41/176): cephadm-19.0.0-1451.g9f8908ee.el9.noa 1.8 MB/s | 748 kB 00:00 2024-02-19T14:49:47.544 INFO:teuthology.orchestra.run.smithi078.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 406 kB/s | 20 kB 00:00 2024-02-19T14:49:47.570 INFO:teuthology.orchestra.run.smithi078.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-02-19T14:49:47.586 INFO:teuthology.orchestra.run.smithi078.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 768 kB/s | 32 kB 00:00 2024-02-19T14:49:47.679 INFO:teuthology.orchestra.run.smithi078.stdout:(45/176): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-02-19T14:49:47.729 INFO:teuthology.orchestra.run.smithi078.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-02-19T14:49:47.745 INFO:teuthology.orchestra.run.smithi078.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 108 kB/s | 17 kB 00:00 2024-02-19T14:49:47.762 INFO:teuthology.orchestra.run.smithi078.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 123 kB/s | 10 kB 00:00 2024-02-19T14:49:47.780 INFO:teuthology.orchestra.run.smithi078.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-02-19T14:49:47.805 INFO:teuthology.orchestra.run.smithi078.stdout:(50/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-02-19T14:49:47.830 INFO:teuthology.orchestra.run.smithi078.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-02-19T14:49:47.855 INFO:teuthology.orchestra.run.smithi078.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-02-19T14:49:47.872 INFO:teuthology.orchestra.run.smithi078.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 596 kB/s | 24 kB 00:00 2024-02-19T14:49:47.889 INFO:teuthology.orchestra.run.smithi078.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 805 kB/s | 46 kB 00:00 2024-02-19T14:49:47.914 INFO:teuthology.orchestra.run.smithi078.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-02-19T14:49:47.931 INFO:teuthology.orchestra.run.smithi078.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 827 kB/s | 34 kB 00:00 2024-02-19T14:49:47.957 INFO:teuthology.orchestra.run.smithi078.stdout:(57/176): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-02-19T14:49:48.049 INFO:teuthology.orchestra.run.smithi078.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-02-19T14:49:48.107 INFO:teuthology.orchestra.run.smithi078.stdout:(59/176): openssl-devel-3.0.7-25.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-02-19T14:49:48.149 INFO:teuthology.orchestra.run.smithi078.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 8.0 MB/s | 1.5 MB 00:00 2024-02-19T14:49:48.174 INFO:teuthology.orchestra.run.smithi078.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 6.4 MB/s | 811 kB 00:00 2024-02-19T14:49:48.199 INFO:teuthology.orchestra.run.smithi078.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-02-19T14:49:48.241 INFO:teuthology.orchestra.run.smithi078.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-02-19T14:49:48.342 INFO:teuthology.orchestra.run.smithi078.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-02-19T14:49:48.384 INFO:teuthology.orchestra.run.smithi078.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 425 kB/s | 78 kB 00:00 2024-02-19T14:49:48.467 INFO:teuthology.orchestra.run.smithi078.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-02-19T14:49:48.492 INFO:teuthology.orchestra.run.smithi078.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 227 kB 00:00 2024-02-19T14:49:48.509 INFO:teuthology.orchestra.run.smithi078.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 198 kB/s | 24 kB 00:00 2024-02-19T14:49:48.526 INFO:teuthology.orchestra.run.smithi078.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 311 kB/s | 18 kB 00:00 2024-02-19T14:49:48.543 INFO:teuthology.orchestra.run.smithi078.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 944 kB/s | 32 kB 00:00 2024-02-19T14:49:48.568 INFO:teuthology.orchestra.run.smithi078.stdout:(71/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-02-19T14:49:48.660 INFO:teuthology.orchestra.run.smithi078.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-02-19T14:49:48.685 INFO:teuthology.orchestra.run.smithi078.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-02-19T14:49:48.710 INFO:teuthology.orchestra.run.smithi078.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-02-19T14:49:48.736 INFO:teuthology.orchestra.run.smithi078.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-02-19T14:49:48.761 INFO:teuthology.orchestra.run.smithi078.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-02-19T14:49:48.786 INFO:teuthology.orchestra.run.smithi078.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-02-19T14:49:48.811 INFO:teuthology.orchestra.run.smithi078.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-19T14:49:48.837 INFO:teuthology.orchestra.run.smithi078.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-02-19T14:49:48.838 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-19T14:49:48.838 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-19T14:49:48.862 INFO:teuthology.orchestra.run.smithi078.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-02-19T14:49:48.887 INFO:teuthology.orchestra.run.smithi078.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 573 kB/s | 43 kB 00:00 2024-02-19T14:49:48.912 INFO:teuthology.orchestra.run.smithi078.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-02-19T14:49:49.079 INFO:teuthology.orchestra.run.smithi078.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 214 kB/s | 41 kB 00:00 2024-02-19T14:49:49.113 INFO:teuthology.orchestra.run.smithi078.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 215 kB 00:00 2024-02-19T14:49:49.272 INFO:teuthology.orchestra.run.smithi078.stdout:(85/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-02-19T14:49:49.297 INFO:teuthology.orchestra.run.smithi078.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 157 kB/s | 34 kB 00:00 2024-02-19T14:49:49.322 INFO:teuthology.orchestra.run.smithi078.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 209 kB/s | 43 kB 00:00 2024-02-19T14:49:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-02-19T14:49:49.373 INFO:teuthology.orchestra.run.smithi078.stdout:(89/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-02-19T14:49:49.389 INFO:teuthology.orchestra.run.smithi078.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.3 MB/s | 87 kB 00:00 2024-02-19T14:49:49.415 INFO:teuthology.orchestra.run.smithi078.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 942 kB/s | 63 kB 00:00 2024-02-19T14:49:49.440 INFO:teuthology.orchestra.run.smithi078.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-02-19T14:49:49.465 INFO:teuthology.orchestra.run.smithi078.stdout:(93/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-02-19T14:49:49.490 INFO:teuthology.orchestra.run.smithi078.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-02-19T14:49:49.524 INFO:teuthology.orchestra.run.smithi078.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-02-19T14:49:49.541 INFO:teuthology.orchestra.run.smithi078.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-02-19T14:49:49.566 INFO:teuthology.orchestra.run.smithi078.stdout:(97/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 194 kB 00:00 2024-02-19T14:49:49.591 INFO:teuthology.orchestra.run.smithi078.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-02-19T14:49:49.625 INFO:teuthology.orchestra.run.smithi078.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-02-19T14:49:49.650 INFO:teuthology.orchestra.run.smithi078.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.1 MB/s | 124 kB 00:00 2024-02-19T14:49:49.667 INFO:teuthology.orchestra.run.smithi078.stdout:(101/176): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-02-19T14:49:49.692 INFO:teuthology.orchestra.run.smithi078.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.7 MB/s | 113 kB 00:00 2024-02-19T14:49:49.720 INFO:teuthology.orchestra.run.smithi078.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.7 MB/s | 187 kB 00:00 2024-02-19T14:49:49.793 INFO:teuthology.orchestra.run.smithi078.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2024-02-19T14:49:49.901 INFO:teuthology.orchestra.run.smithi078.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 4.6 MB/s | 863 kB 00:00 2024-02-19T14:49:49.960 INFO:teuthology.orchestra.run.smithi078.stdout:(106/176): python3-devel-3.9.17-2.el9.x86_64.rp 920 kB/s | 245 kB 00:00 2024-02-19T14:49:50.210 INFO:teuthology.orchestra.run.smithi078.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-02-19T14:49:50.410 INFO:teuthology.orchestra.run.smithi078.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 888 kB/s | 548 kB 00:00 2024-02-19T14:49:50.461 INFO:teuthology.orchestra.run.smithi078.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 552 kB/s | 309 kB 00:00 2024-02-19T14:49:50.502 INFO:teuthology.orchestra.run.smithi078.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-02-19T14:49:50.578 INFO:teuthology.orchestra.run.smithi078.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-02-19T14:49:50.603 INFO:teuthology.orchestra.run.smithi078.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-02-19T14:49:50.728 INFO:teuthology.orchestra.run.smithi078.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-02-19T14:49:50.829 INFO:teuthology.orchestra.run.smithi078.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-02-19T14:49:50.921 INFO:teuthology.orchestra.run.smithi078.stdout:(115/176): libarrow-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-02-19T14:49:50.954 INFO:teuthology.orchestra.run.smithi078.stdout:(116/176): python3-asyncssh-2.13.2-4.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-02-19T14:49:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:(117/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 289 kB/s | 43 kB 00:00 2024-02-19T14:49:51.030 INFO:teuthology.orchestra.run.smithi078.stdout:(118/176): python3-cachetools-4.2.4-1.el9.noarc 295 kB/s | 32 kB 00:00 2024-02-19T14:49:51.064 INFO:teuthology.orchestra.run.smithi078.stdout:(119/176): python3-certifi-2021.10.8-2.el9.noar 137 kB/s | 15 kB 00:00 2024-02-19T14:49:51.105 INFO:teuthology.orchestra.run.smithi078.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-02-19T14:49:51.172 INFO:teuthology.orchestra.run.smithi078.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-02-19T14:49:51.264 INFO:teuthology.orchestra.run.smithi078.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 223 kB/s | 35 kB 00:00 2024-02-19T14:49:51.289 INFO:teuthology.orchestra.run.smithi078.stdout:(123/176): python3-google-auth-2.27.0-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-02-19T14:49:51.323 INFO:teuthology.orchestra.run.smithi078.stdout:(124/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.1 MB/s | 303 kB 00:00 2024-02-19T14:49:51.407 INFO:teuthology.orchestra.run.smithi078.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-02-19T14:49:51.440 INFO:teuthology.orchestra.run.smithi078.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-02-19T14:49:51.515 INFO:teuthology.orchestra.run.smithi078.stdout:(127/176): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-02-19T14:49:51.540 INFO:teuthology.orchestra.run.smithi078.stdout:(128/176): python3-jaraco-classes-3.2.1-5.el9.n 132 kB/s | 18 kB 00:00 2024-02-19T14:49:51.566 INFO:teuthology.orchestra.run.smithi078.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 184 kB/s | 23 kB 00:00 2024-02-19T14:49:51.608 INFO:teuthology.orchestra.run.smithi078.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-02-19T14:49:51.633 INFO:teuthology.orchestra.run.smithi078.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-02-19T14:49:51.733 INFO:teuthology.orchestra.run.smithi078.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-02-19T14:49:51.758 INFO:teuthology.orchestra.run.smithi078.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 630 kB/s | 79 kB 00:00 2024-02-19T14:49:51.792 INFO:teuthology.orchestra.run.smithi078.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-02-19T14:49:51.859 INFO:teuthology.orchestra.run.smithi078.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 460 kB/s | 58 kB 00:00 2024-02-19T14:49:51.884 INFO:teuthology.orchestra.run.smithi078.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-02-19T14:49:51.910 INFO:teuthology.orchestra.run.smithi078.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-02-19T14:49:51.985 INFO:teuthology.orchestra.run.smithi078.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 717 kB/s | 90 kB 00:00 2024-02-19T14:49:52.018 INFO:teuthology.orchestra.run.smithi078.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 211 kB/s | 28 kB 00:00 2024-02-19T14:49:52.044 INFO:teuthology.orchestra.run.smithi078.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 229 kB/s | 31 kB 00:00 2024-02-19T14:49:52.102 INFO:teuthology.orchestra.run.smithi078.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-02-19T14:49:52.128 INFO:teuthology.orchestra.run.smithi078.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 542 kB/s | 59 kB 00:00 2024-02-19T14:49:52.211 INFO:teuthology.orchestra.run.smithi078.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 214 kB/s | 36 kB 00:00 2024-02-19T14:49:52.261 INFO:teuthology.orchestra.run.smithi078.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 321 kB/s | 51 kB 00:00 2024-02-19T14:49:52.286 INFO:teuthology.orchestra.run.smithi078.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-02-19T14:49:52.362 INFO:teuthology.orchestra.run.smithi078.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 596 kB/s | 90 kB 00:00 2024-02-19T14:49:52.412 INFO:teuthology.orchestra.run.smithi078.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-02-19T14:49:52.437 INFO:teuthology.orchestra.run.smithi078.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-02-19T14:49:52.463 INFO:teuthology.orchestra.run.smithi078.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-02-19T14:49:52.571 INFO:teuthology.orchestra.run.smithi078.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-02-19T14:49:52.614 INFO:teuthology.orchestra.run.smithi078.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-02-19T14:49:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:(152/176): NetworkManager-1.45.91-1.el9.x86_64. 21 MB/s | 2.3 MB 00:00 2024-02-19T14:49:52.806 INFO:teuthology.orchestra.run.smithi078.stdout:(153/176): NetworkManager-libnm-1.45.91-1.el9.x 22 MB/s | 1.8 MB 00:00 2024-02-19T14:49:52.831 INFO:teuthology.orchestra.run.smithi078.stdout:(154/176): NetworkManager-team-1.45.91-1.el9.x8 1.6 MB/s | 41 kB 00:00 2024-02-19T14:49:52.873 INFO:teuthology.orchestra.run.smithi078.stdout:(155/176): NetworkManager-tui-1.45.91-1.el9.x86 5.8 MB/s | 246 kB 00:00 2024-02-19T14:49:52.906 INFO:teuthology.orchestra.run.smithi078.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-02-19T14:49:52.932 INFO:teuthology.orchestra.run.smithi078.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-02-19T14:49:52.974 INFO:teuthology.orchestra.run.smithi078.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-02-19T14:49:53.015 INFO:teuthology.orchestra.run.smithi078.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-02-19T14:49:53.074 INFO:teuthology.orchestra.run.smithi078.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-02-19T14:49:53.116 INFO:teuthology.orchestra.run.smithi078.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-02-19T14:49:53.149 INFO:teuthology.orchestra.run.smithi078.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 662 kB/s | 22 kB 00:00 2024-02-19T14:49:53.191 INFO:teuthology.orchestra.run.smithi078.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-02-19T14:49:53.225 INFO:teuthology.orchestra.run.smithi078.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.5 MB/s | 154 kB 00:00 2024-02-19T14:49:53.259 INFO:teuthology.orchestra.run.smithi078.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.9 MB/s | 135 kB 00:00 2024-02-19T14:49:53.317 INFO:teuthology.orchestra.run.smithi078.stdout:(166/176): librados2-19.0.0-1451.g9f8908ee.el9. 3.8 MB/s | 3.4 MB 00:00 2024-02-19T14:49:53.342 INFO:teuthology.orchestra.run.smithi078.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 753 kB/s | 63 kB 00:00 2024-02-19T14:49:53.368 INFO:teuthology.orchestra.run.smithi078.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 553 kB/s | 28 kB 00:00 2024-02-19T14:49:53.410 INFO:teuthology.orchestra.run.smithi078.stdout:(169/176): openssl-3.0.7-25.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-02-19T14:49:53.460 INFO:teuthology.orchestra.run.smithi078.stdout:(170/176): openssl-libs-3.0.7-25.el9.x86_64.rpm 24 MB/s | 2.1 MB 00:00 2024-02-19T14:49:53.519 INFO:teuthology.orchestra.run.smithi078.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.2 MB 00:00 2024-02-19T14:49:53.544 INFO:teuthology.orchestra.run.smithi078.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.3 MB/s | 438 kB 00:00 2024-02-19T14:49:53.569 INFO:teuthology.orchestra.run.smithi078.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 609 kB/s | 30 kB 00:00 2024-02-19T14:49:53.594 INFO:teuthology.orchestra.run.smithi078.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-02-19T14:49:53.636 INFO:teuthology.orchestra.run.smithi078.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-02-19T14:49:53.720 INFO:teuthology.orchestra.run.smithi078.stdout:(176/176): librbd1-19.0.0-1451.g9f8908ee.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-02-19T14:49:53.724 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-02-19T14:49:53.724 INFO:teuthology.orchestra.run.smithi078.stdout:Total 9.8 MB/s | 223 MB 00:22 2024-02-19T14:49:53.771 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-02-19T14:49:54.072 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-02-19T14:49:54.072 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-02-19T14:49:55.755 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-19T14:49:55.850 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/202 2024-02-19T14:49:55.888 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-02-19T14:49:55.927 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-19T14:49:55.951 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-19T14:49:56.014 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-02-19T14:49:56.036 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-02-19T14:49:56.036 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-02-19T14:49:56.051 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-02-19T14:49:56.107 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-02-19T14:49:56.139 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-02-19T14:49:56.165 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-02-19T14:49:56.203 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-02-19T14:49:56.551 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-02-19T14:49:56.607 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 11/202 2024-02-19T14:49:56.720 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 11/202 2024-02-19T14:49:56.757 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 12/202 2024-02-19T14:49:56.849 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 12/202 2024-02-19T14:49:56.878 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-1451.g9f8908ee.el9.x86_64 13/202 2024-02-19T14:49:57.028 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-02-19T14:49:57.084 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-02-19T14:49:57.112 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 16/202 2024-02-19T14:49:57.215 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 16/202 2024-02-19T14:49:57.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86 17/202 2024-02-19T14:49:57.277 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86 17/202 2024-02-19T14:49:57.308 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-02-19T14:49:57.331 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-19T14:49:57.346 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-19T14:49:57.495 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-19T14:49:57.521 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-19T14:49:57.568 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-02-19T14:49:57.748 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-02-19T14:49:57.879 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 23/202 2024-02-19T14:49:57.940 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 23/202 2024-02-19T14:49:58.040 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 24/202 2024-02-19T14:49:58.101 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-1:1.45.91-1.el9.x86_64 24/202 2024-02-19T14:49:58.334 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 24/202 2024-02-19T14:49:58.466 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-02-19T14:49:58.560 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-02-19T14:49:58.589 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-02-19T14:49:58.658 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-02-19T14:49:58.711 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-02-19T14:49:58.751 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-02-19T14:49:58.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-02-19T14:49:58.818 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-02-19T14:49:58.881 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-02-19T14:49:58.917 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-02-19T14:49:58.961 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-02-19T14:49:58.994 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-02-19T14:49:59.021 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-02-19T14:49:59.092 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-02-19T14:49:59.126 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-1451.g9f8908ee.el9. 39/202 2024-02-19T14:49:59.170 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-1451.g9f8908ee.el 40/202 2024-02-19T14:49:59.198 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-1451.g9f8908ee.el9.x86_6 41/202 2024-02-19T14:49:59.268 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-02-19T14:49:59.313 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-02-19T14:49:59.386 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-02-19T14:49:59.521 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-02-19T14:49:59.573 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-02-19T14:50:00.094 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-02-19T14:50:00.177 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-02-19T14:50:00.258 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-02-19T14:50:00.297 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-19T14:50:00.306 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-19T14:50:00.345 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-02-19T14:50:00.382 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-1451.g9f8908ee.el9.x86_6 52/202 2024-02-19T14:50:00.415 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-02-19T14:50:00.514 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-02-19T14:50:00.583 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-02-19T14:50:00.858 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-02-19T14:50:01.189 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-02-19T14:50:01.386 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-02-19T14:50:01.430 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-02-19T14:50:01.568 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-02-19T14:50:01.625 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 61/202 2024-02-19T14:50:01.680 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-02-19T14:50:01.752 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-02-19T14:50:01.784 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-02-19T14:50:01.820 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-02-19T14:50:01.858 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-02-19T14:50:01.894 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-02-19T14:50:01.938 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-02-19T14:50:01.979 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-02-19T14:50:02.112 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-02-19T14:50:02.152 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-02-19T14:50:02.182 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-02-19T14:50:02.224 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-02-19T14:50:02.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-02-19T14:50:02.357 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 75/202 2024-02-19T14:50:02.397 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-02-19T14:50:02.423 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-02-19T14:50:02.447 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-02-19T14:50:02.545 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-02-19T14:50:02.674 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-02-19T14:50:02.780 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-02-19T14:50:02.830 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-02-19T14:50:02.879 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-02-19T14:50:02.934 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-02-19T14:50:02.940 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-02-19T14:50:03.028 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/202 2024-02-19T14:50:03.059 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-02-19T14:50:03.089 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-19T14:50:03.110 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-19T14:50:03.150 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-02-19T14:50:03.159 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-02-19T14:50:03.189 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-02-19T14:50:03.201 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-02-19T14:50:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-02-19T14:50:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-02-19T14:50:03.302 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-02-19T14:50:03.321 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-02-19T14:50:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-02-19T14:50:03.366 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-02-19T14:50:03.399 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-02-19T14:50:03.431 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-02-19T14:50:03.466 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-02-19T14:50:03.497 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-02-19T14:50:03.525 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-02-19T14:50:03.559 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-02-19T14:50:03.598 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-02-19T14:50:03.621 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-02-19T14:50:03.652 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-02-19T14:50:03.727 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-02-19T14:50:03.741 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-02-19T14:50:03.762 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-02-19T14:50:03.769 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 11/202 2024-02-19T14:50:03.795 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-02-19T14:50:03.837 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-02-19T14:50:03.871 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-02-19T14:50:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 11/202 2024-02-19T14:50:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 12/202 2024-02-19T14:50:03.959 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-02-19T14:50:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 12/202 2024-02-19T14:50:04.000 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:19.0.0-1451.g9f8908ee.el9.x86_64 13/202 2024-02-19T14:50:04.042 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-02-19T14:50:04.087 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-02-19T14:50:04.125 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-02-19T14:50:04.164 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-02-19T14:50:04.173 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-02-19T14:50:04.214 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-02-19T14:50:04.241 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 16/202 2024-02-19T14:50:04.269 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-02-19T14:50:04.319 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-02-19T14:50:04.344 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 16/202 2024-02-19T14:50:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86 17/202 2024-02-19T14:50:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86 17/202 2024-02-19T14:50:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-02-19T14:50:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-19T14:50:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-19T14:50:04.698 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-19T14:50:04.734 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-19T14:50:04.773 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-02-19T14:50:04.943 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-02-19T14:50:05.043 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-02-19T14:50:05.091 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-02-19T14:50:05.100 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 23/202 2024-02-19T14:50:05.125 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-02-19T14:50:05.168 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 23/202 2024-02-19T14:50:05.181 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-02-19T14:50:05.265 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 24/202 2024-02-19T14:50:05.268 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-02-19T14:50:05.317 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-02-19T14:50:05.319 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-1:1.45.91-1.el9.x86_64 24/202 2024-02-19T14:50:05.494 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-02-19T14:50:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-02-19T14:50:05.534 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 24/202 2024-02-19T14:50:05.592 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-02-19T14:50:05.645 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-02-19T14:50:05.770 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-02-19T14:50:05.816 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-02-19T14:50:05.885 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-02-19T14:50:05.938 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-02-19T14:50:05.969 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-02-19T14:50:06.005 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-02-19T14:50:06.045 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-02-19T14:50:06.107 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-02-19T14:50:06.143 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-02-19T14:50:06.178 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-02-19T14:50:06.202 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-02-19T14:50:06.231 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-02-19T14:50:06.243 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-02-19T14:50:06.299 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-02-19T14:50:06.335 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:19.0.0-1451.g9f8908ee.el9. 39/202 2024-02-19T14:50:06.369 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-02-19T14:50:06.379 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:19.0.0-1451.g9f8908ee.el 40/202 2024-02-19T14:50:06.407 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:19.0.0-1451.g9f8908ee.el9.x86_6 41/202 2024-02-19T14:50:06.475 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-02-19T14:50:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-02-19T14:50:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-02-19T14:50:06.727 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-02-19T14:50:06.790 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-02-19T14:50:07.303 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-02-19T14:50:07.380 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-02-19T14:50:07.459 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-02-19T14:50:07.508 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-19T14:50:07.517 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-19T14:50:07.555 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-02-19T14:50:07.581 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-02-19T14:50:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:19.0.0-1451.g9f8908ee.el9.x86_6 52/202 2024-02-19T14:50:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-02-19T14:50:07.660 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-02-19T14:50:07.678 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-02-19T14:50:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-02-19T14:50:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-02-19T14:50:07.978 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-02-19T14:50:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 125/202 2024-02-19T14:50:08.032 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-02-19T14:50:08.082 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 125/202 2024-02-19T14:50:08.113 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 126/202 2024-02-19T14:50:08.149 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-02-19T14:50:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-02-19T14:50:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-02-19T14:50:08.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-02-19T14:50:08.491 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-02-19T14:50:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 61/202 2024-02-19T14:50:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-02-19T14:50:08.607 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-02-19T14:50:08.636 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 129/202 2024-02-19T14:50:08.659 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-02-19T14:50:08.691 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 129/202 2024-02-19T14:50:08.691 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-02-19T14:50:08.721 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 130/202 2024-02-19T14:50:08.728 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-02-19T14:50:08.774 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-02-19T14:50:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-02-19T14:50:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-02-19T14:50:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-02-19T14:50:09.027 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-02-19T14:50:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-02-19T14:50:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-02-19T14:50:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-02-19T14:50:09.241 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-02-19T14:50:09.281 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 75/202 2024-02-19T14:50:09.321 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-02-19T14:50:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-02-19T14:50:09.372 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-02-19T14:50:09.448 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-02-19T14:50:09.572 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-02-19T14:50:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-02-19T14:50:09.713 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-02-19T14:50:09.761 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-02-19T14:50:09.818 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-02-19T14:50:10.000 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-02-19T14:50:10.042 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-02-19T14:50:10.171 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-02-19T14:50:10.208 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-02-19T14:50:10.240 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-02-19T14:50:10.272 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-02-19T14:50:10.307 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-02-19T14:50:10.338 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-02-19T14:50:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-02-19T14:50:10.400 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-02-19T14:50:10.439 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-02-19T14:50:10.462 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-02-19T14:50:10.492 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-02-19T14:50:10.568 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-02-19T14:50:10.603 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-02-19T14:50:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-02-19T14:50:10.678 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-02-19T14:50:10.712 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-02-19T14:50:10.801 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-02-19T14:50:10.882 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-02-19T14:50:10.926 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 131/202 2024-02-19T14:50:10.934 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-02-19T14:50:10.959 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 131/202 2024-02-19T14:50:10.966 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-02-19T14:50:11.012 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-02-19T14:50:11.130 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-02-19T14:50:11.184 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-02-19T14:50:11.505 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 131/202 2024-02-19T14:50:11.538 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 132/202 2024-02-19T14:50:11.614 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 132/202 2024-02-19T14:50:11.614 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T14:50:11.614 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T14:50:11.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:11.656 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 133/202 2024-02-19T14:50:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-02-19T14:50:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-02-19T14:50:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-02-19T14:50:12.038 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-02-19T14:50:12.125 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-02-19T14:50:12.175 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-02-19T14:50:12.352 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-02-19T14:50:12.381 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-02-19T14:50:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-02-19T14:50:13.094 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-02-19T14:50:13.211 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-02-19T14:50:14.420 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-02-19T14:50:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-02-19T14:50:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-02-19T14:50:14.845 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-02-19T14:50:14.875 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 125/202 2024-02-19T14:50:14.956 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 125/202 2024-02-19T14:50:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 126/202 2024-02-19T14:50:15.024 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-02-19T14:50:15.478 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-02-19T14:50:15.502 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 129/202 2024-02-19T14:50:15.557 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 129/202 2024-02-19T14:50:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 130/202 2024-02-19T14:50:17.800 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 131/202 2024-02-19T14:50:17.834 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 131/202 2024-02-19T14:50:18.383 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 131/202 2024-02-19T14:50:18.420 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 132/202 2024-02-19T14:50:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 132/202 2024-02-19T14:50:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T14:50:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T14:50:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:18.530 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 133/202 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 133/202 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-02-19T14:50:23.525 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:23.686 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 134/202 2024-02-19T14:50:23.700 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 134/202 2024-02-19T14:50:23.700 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:23.701 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-19T14:50:23.701 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:25.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 135/202 2024-02-19T14:50:25.239 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 135/202 2024-02-19T14:50:25.239 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:25.239 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T14:50:25.239 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T14:50:25.239 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T14:50:25.239 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:25.473 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 136/202 2024-02-19T14:50:25.495 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 136/202 2024-02-19T14:50:25.495 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:25.495 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T14:50:25.495 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T14:50:25.495 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T14:50:25.495 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:25.940 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 137/202 2024-02-19T14:50:25.962 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 137/202 2024-02-19T14:50:25.963 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:25.963 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T14:50:25.963 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T14:50:25.963 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T14:50:25.963 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:26.050 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-02-19T14:50:26.384 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 139/202 2024-02-19T14:50:26.415 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 139/202 2024-02-19T14:50:26.446 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1451.g9f8908ee.e 140/202 2024-02-19T14:50:26.507 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1451.g9f8908ee. 141/202 2024-02-19T14:50:26.530 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noa 142/202 2024-02-19T14:50:26.755 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noa 142/202 2024-02-19T14:50:26.781 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.n 143/202 2024-02-19T14:50:27.734 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.n 143/202 2024-02-19T14:50:27.756 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8 144/202 2024-02-19T14:50:27.855 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8 144/202 2024-02-19T14:50:28.018 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1451.g9f8908ee.el 145/202 2024-02-19T14:50:28.050 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 146/202 2024-02-19T14:50:28.073 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 146/202 2024-02-19T14:50:28.073 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:28.073 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T14:50:28.073 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T14:50:28.073 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T14:50:28.073 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:28.117 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 147/202 2024-02-19T14:50:28.128 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 147/202 2024-02-19T14:50:28.175 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-1451.g9f8908ee.el9.x86_64 148/202 2024-02-19T14:50:29.025 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-02-19T14:50:29.061 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 150/202 2024-02-19T14:50:29.083 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 150/202 2024-02-19T14:50:29.083 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:29.083 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T14:50:29.084 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T14:50:29.084 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T14:50:29.084 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:29.136 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1451.g9f890 151/202 2024-02-19T14:50:29.158 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1451.g9f890 151/202 2024-02-19T14:50:29.158 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:29.158 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T14:50:29.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:29.204 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 152/202 2024-02-19T14:50:29.226 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 152/202 2024-02-19T14:50:29.226 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:29.226 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-19T14:50:29.226 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-19T14:50:29.226 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-19T14:50:29.226 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:29.529 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 153/202 2024-02-19T14:50:29.552 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 153/202 2024-02-19T14:50:29.552 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:29.552 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T14:50:29.552 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T14:50:29.552 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T14:50:29.552 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:30.353 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 133/202 2024-02-19T14:50:30.353 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-02-19T14:50:30.353 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-02-19T14:50:30.353 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-02-19T14:50:30.354 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-02-19T14:50:30.354 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-02-19T14:50:30.354 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-02-19T14:50:30.354 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-02-19T14:50:30.354 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:30.525 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 134/202 2024-02-19T14:50:30.541 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 134/202 2024-02-19T14:50:30.541 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:30.541 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-19T14:50:30.541 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:32.005 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 135/202 2024-02-19T14:50:32.030 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 135/202 2024-02-19T14:50:32.030 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:32.031 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T14:50:32.031 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T14:50:32.031 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T14:50:32.031 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:32.256 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 136/202 2024-02-19T14:50:32.278 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 136/202 2024-02-19T14:50:32.278 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:32.278 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T14:50:32.278 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T14:50:32.278 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T14:50:32.278 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:32.723 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 137/202 2024-02-19T14:50:32.745 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 137/202 2024-02-19T14:50:32.746 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:32.746 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T14:50:32.746 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T14:50:32.746 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T14:50:32.746 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:32.833 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-02-19T14:50:33.251 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 139/202 2024-02-19T14:50:33.281 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 139/202 2024-02-19T14:50:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1451.g9f8908ee.e 140/202 2024-02-19T14:50:33.372 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1451.g9f8908ee. 141/202 2024-02-19T14:50:33.396 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noa 142/202 2024-02-19T14:50:33.621 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noa 142/202 2024-02-19T14:50:33.655 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.n 143/202 2024-02-19T14:50:33.992 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-1451.g9f8908ee.el9.x86_64 154/202 2024-02-19T14:50:34.042 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 155/202 2024-02-19T14:50:34.072 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 156/202 2024-02-19T14:50:34.099 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-02-19T14:50:34.281 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-shell-2:19.0.0-1451.g9f8908ee.el9.x86_64 158/202 2024-02-19T14:50:34.414 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-02-19T14:50:34.456 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 160/202 2024-02-19T14:50:34.516 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-02-19T14:50:34.573 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-02-19T14:50:34.608 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.n 143/202 2024-02-19T14:50:34.639 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8 144/202 2024-02-19T14:50:34.653 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-02-19T14:50:34.691 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-02-19T14:50:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-02-19T14:50:34.738 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8 144/202 2024-02-19T14:50:34.765 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-1451.g9f8908ee.el9.x86_ 166/202 2024-02-19T14:50:34.794 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-team-1:1.45.91-1.el9.x86_64 167/202 2024-02-19T14:50:34.865 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-tui-1:1.45.91-1.el9.x86_64 168/202 2024-02-19T14:50:34.909 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1451.g9f8908ee.el 145/202 2024-02-19T14:50:34.932 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-02-19T14:50:34.941 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 146/202 2024-02-19T14:50:34.963 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 146/202 2024-02-19T14:50:34.964 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:34.964 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T14:50:34.964 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T14:50:34.964 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T14:50:34.964 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:34.968 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-02-19T14:50:35.008 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 147/202 2024-02-19T14:50:35.019 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 147/202 2024-02-19T14:50:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:19.0.0-1451.g9f8908ee.el9.x86_64 148/202 2024-02-19T14:50:35.601 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-19T14:50:35.641 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-19T14:50:35.661 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-19T14:50:35.922 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-02-19T14:50:35.952 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 150/202 2024-02-19T14:50:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 150/202 2024-02-19T14:50:35.975 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:35.975 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T14:50:35.975 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T14:50:35.975 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T14:50:35.975 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1451.g9f890 151/202 2024-02-19T14:50:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1451.g9f890 151/202 2024-02-19T14:50:36.050 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:36.050 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T14:50:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 152/202 2024-02-19T14:50:36.116 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 152/202 2024-02-19T14:50:36.117 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:36.117 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-19T14:50:36.117 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-19T14:50:36.117 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-19T14:50:36.117 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:36.421 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 153/202 2024-02-19T14:50:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 153/202 2024-02-19T14:50:36.443 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T14:50:36.443 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T14:50:36.443 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T14:50:36.443 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T14:50:36.443 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:36.476 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:19.0.0-1451.g9f8908ee.el9.noarch 172/202 2024-02-19T14:50:36.544 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 173/202 2024-02-19T14:50:36.585 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-02-19T14:50:36.676 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-02-19T14:50:36.706 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-02-19T14:50:36.731 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-02-19T14:50:36.765 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-02-19T14:50:36.791 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-02-19T14:50:36.791 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-19T14:50:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-19T14:50:36.904 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-02-19T14:50:36.934 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-02-19T14:50:36.934 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-19T14:50:37.019 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-19T14:50:37.052 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-02-19T14:50:37.079 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-02-19T14:50:37.112 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-02-19T14:50:37.140 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-02-19T14:50:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-02-19T14:50:37.215 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-02-19T14:50:37.261 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-02-19T14:50:37.301 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-02-19T14:50:37.343 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-02-19T14:50:37.369 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-02-19T14:50:37.394 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-02-19T14:50:37.420 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-02-19T14:50:37.426 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-19T14:50:37.426 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-19T14:50:37.616 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-19T14:50:37.616 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-19T14:50:37.682 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-19T14:50:37.707 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-02-19T14:50:37.733 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-02-19T14:50:37.757 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-02-19T14:50:37.799 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-02-19T14:50:37.866 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-19T14:50:40.858 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:19.0.0-1451.g9f8908ee.el9.x86_64 154/202 2024-02-19T14:50:40.908 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 155/202 2024-02-19T14:50:40.938 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 156/202 2024-02-19T14:50:40.965 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-02-19T14:50:41.146 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-shell-2:19.0.0-1451.g9f8908ee.el9.x86_64 158/202 2024-02-19T14:50:41.264 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-02-19T14:50:41.305 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 160/202 2024-02-19T14:50:41.359 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-02-19T14:50:41.422 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-02-19T14:50:41.491 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-02-19T14:50:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-02-19T14:50:41.576 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-02-19T14:50:41.607 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:19.0.0-1451.g9f8908ee.el9.x86_ 166/202 2024-02-19T14:50:41.637 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-team-1:1.45.91-1.el9.x86_64 167/202 2024-02-19T14:50:41.735 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-tui-1:1.45.91-1.el9.x86_64 168/202 2024-02-19T14:50:41.800 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-02-19T14:50:41.851 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-02-19T14:50:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-19T14:50:42.441 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-19T14:50:42.461 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-19T14:50:42.682 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-19T14:50:42.682 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-1451.g9f8908ee.el9.x86_64 1/202 2024-02-19T14:50:42.682 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 2/202 2024-02-19T14:50:42.682 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 3/202 2024-02-19T14:50:42.682 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 4/202 2024-02-19T14:50:42.682 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1451.g9f890 5/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 6/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 7/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 8/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 9/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 10/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 11/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-1451.g9f8908ee.el9.x86_64 12/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 13/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-shell-2:19.0.0-1451.g9f8908ee.el9.x86_64 14/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-1451.g9f8908ee.el9.x86_ 15/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 16/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 17/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-1451.g9f8908ee.el9.x86_6 18/202 2024-02-19T14:50:42.683 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86 19/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 20/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1451.g9f8908ee.el 21/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-1451.g9f8908ee.el9. 22/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-1451.g9f8908ee.el9.x86_6 23/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-1451.g9f8908ee.el9.x86_64 24/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 25/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 26/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 27/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 28/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 29/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1451.g9f8908ee. 30/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noa 31/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.n 32/202 2024-02-19T14:50:42.684 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8 33/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1451.g9f8908ee.el 34/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 35/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1451.g9f8908ee.e 36/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 37/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 38/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:19.0.0-1451.g9f8908ee.el9.noarch 39/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-02-19T14:50:42.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 59/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-02-19T14:50:42.686 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-02-19T14:50:42.687 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-02-19T14:50:42.688 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-02-19T14:50:42.689 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-02-19T14:50:42.690 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-02-19T14:50:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-02-19T14:50:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 151/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 153/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.45.91-1.el9.x86_64 157/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 159/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.45.91-1.el9.x86_64 161/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.45.91-1.el9.x86_64 163/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-02-19T14:50:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-02-19T14:50:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 189/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 191/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-02-19T14:50:42.695 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-02-19T14:50:42.696 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-02-19T14:50:42.696 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-02-19T14:50:42.696 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-02-19T14:50:42.696 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-02-19T14:50:43.265 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:19.0.0-1451.g9f8908ee.el9.noarch 172/202 2024-02-19T14:50:43.336 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 173/202 2024-02-19T14:50:43.386 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-02-19T14:50:43.474 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-02-19T14:50:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-02-19T14:50:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-02-19T14:50:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-02-19T14:50:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-02-19T14:50:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-19T14:50:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-19T14:50:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-02-19T14:50:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-02-19T14:50:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-19T14:50:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-19T14:50:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-02-19T14:50:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-02-19T14:50:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-02-19T14:50:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-02-19T14:50:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-02-19T14:50:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-02-19T14:50:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-02-19T14:50:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-02-19T14:50:44.154 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-02-19T14:50:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-02-19T14:50:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-02-19T14:50:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-02-19T14:50:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-19T14:50:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-19T14:50:44.425 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-19T14:50:44.425 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-19T14:50:44.507 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-19T14:50:44.532 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-02-19T14:50:44.558 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-02-19T14:50:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-02-19T14:50:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-02-19T14:50:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: acl-2.3.1-4.el9.x86_64 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-19T14:50:45.742 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: iproute-6.7.0-2.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libmount-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-19T14:50:45.743 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.2-8.el9.noarch 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.7.4-5.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.744 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.4-9.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-19T14:50:45.746 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T14:50:45.747 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T14:50:45.748 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-13.11-1.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T14:50:45.749 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-19T14:50:45.750 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T14:50:45.751 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T14:50:45.752 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T14:50:45.753 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-19T14:50:45.754 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T14:50:45.755 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-19T14:50:45.755 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-19T14:50:45.755 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T14:50:45.755 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-19T14:50:45.755 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:45.755 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-19T14:50:46.333 DEBUG:teuthology.parallel:result is None 2024-02-19T14:50:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-19T14:50:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:19.0.0-1451.g9f8908ee.el9.x86_64 1/202 2024-02-19T14:50:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 2/202 2024-02-19T14:50:49.730 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 3/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 4/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1451.g9f890 5/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 6/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 7/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 8/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 9/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 10/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 11/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:19.0.0-1451.g9f8908ee.el9.x86_64 12/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 13/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-shell-2:19.0.0-1451.g9f8908ee.el9.x86_64 14/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:19.0.0-1451.g9f8908ee.el9.x86_ 15/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 16/202 2024-02-19T14:50:49.731 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 17/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:19.0.0-1451.g9f8908ee.el9.x86_6 18/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86 19/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 20/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1451.g9f8908ee.el 21/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:19.0.0-1451.g9f8908ee.el9. 22/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:19.0.0-1451.g9f8908ee.el9.x86_6 23/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:19.0.0-1451.g9f8908ee.el9.x86_64 24/202 2024-02-19T14:50:49.732 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 25/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 26/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 27/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 28/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 29/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1451.g9f8908ee. 30/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noa 31/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.n 32/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8 33/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1451.g9f8908ee.el 34/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 35/202 2024-02-19T14:50:49.749 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1451.g9f8908ee.e 36/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 37/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 38/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:19.0.0-1451.g9f8908ee.el9.noarch 39/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-02-19T14:50:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 59/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-02-19T14:50:49.751 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-02-19T14:50:49.752 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-02-19T14:50:49.753 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-02-19T14:50:49.754 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-02-19T14:50:49.755 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-02-19T14:50:49.756 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-02-19T14:50:49.757 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 151/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 153/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.45.91-1.el9.x86_64 157/202 2024-02-19T14:50:49.758 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 159/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.45.91-1.el9.x86_64 161/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.45.91-1.el9.x86_64 163/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-02-19T14:50:49.759 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-02-19T14:50:49.760 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 189/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 191/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-02-19T14:50:49.761 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-02-19T14:50:49.762 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-02-19T14:50:49.762 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-02-19T14:50:52.626 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-02-19T14:50:52.626 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:52.626 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-02-19T14:50:52.626 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:52.626 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:52.626 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui-1:1.45.91-1.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: acl-2.3.1-4.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-6.7.0-2.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: libmount-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.627 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.2-8.el9.noarch 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.7.4-5.el9.x86_64 2024-02-19T14:50:52.628 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:19.0.0-1451.g9f8908ee.el9.noarch 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.4-9.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T14:50:52.630 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-16.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T14:50:52.631 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: lua-5.4.4-4.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T14:50:52.632 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-13.11-1.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T14:50:52.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-19T14:50:52.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T14:50:52.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-19T14:50:52.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T14:50:52.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:19.0.0-1451.g9f8908ee.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: re2-1:20211101-3.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-19T14:50:52.638 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T14:50:52.639 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-19T14:50:52.639 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:52.639 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-02-19T14:50:53.203 DEBUG:teuthology.parallel:result is None 2024-02-19T14:50:53.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T14:50:53.456 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T14:50:53.481 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-1451.g9f8908ee.el9 2024-02-19T14:50:53.481 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1451.g9f8908ee.el9 2024-02-19T14:50:53.481 INFO:teuthology.task.install:The correct ceph version 19.0.0-1451.g9f8908ee is installed. 2024-02-19T14:50:53.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9f8908eed2b2d7e8c316bd5be7ed6e49024931ee 2024-02-19T14:50:53.623 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T14:50:53.649 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-1451.g9f8908ee.el9 2024-02-19T14:50:53.649 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1451.g9f8908ee.el9 2024-02-19T14:50:53.650 INFO:teuthology.task.install:The correct ceph version 19.0.0-1451.g9f8908ee is installed. 2024-02-19T14:50:53.651 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T14:50:53.652 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:53.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T14:50:53.685 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:53.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T14:50:53.724 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T14:50:53.725 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:53.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T14:50:53.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T14:50:53.822 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:53.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T14:50:53.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T14:50:53.921 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T14:50:53.922 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:53.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T14:50:53.950 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T14:50:54.017 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:54.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T14:50:54.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T14:50:54.115 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T14:50:54.115 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:54.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T14:50:54.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T14:50:54.209 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:54.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T14:50:54.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T14:50:54.305 INFO:teuthology.run_tasks:Running task ceph... 2024-02-19T14:50:54.387 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-19T14:50:54.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-02-19T14:50:54.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-02-19T14:50:54.417 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-19T14:50:54.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-19T14:50:54.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-19T14:50:54.484 INFO:tasks.ceph:Creating extra log directories... 2024-02-19T14:50:54.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-19T14:50:54.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-19T14:50:54.557 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-02-19T14:50:54.557 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-02-19T14:50:54.557 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-02-19T14:50:54.557 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:54.557 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-02-19T14:50:54.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-02-19T14:50:54.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-02-19T14:50:54.747 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:54.747 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-02-19T14:50:54.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-02-19T14:50:54.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-02-19T14:50:54.909 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-02-19T14:50:54.910 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-19T14:50:54.910 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468', 'branch': 'wip-lusov-qdb', 'description': 'fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}', 'email': None, 'first_in_suite': False, 'job_id': '7566468', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 2280, 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'sleep_before_teardown': 0, 'subset': '111/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEFJ716bM4kXdY3SWaq4E/9AKO2aR9oARi5WQKiL9KctLMdfkOuM0QpNZUyOUbjV2DvVftK/40LaKvuq62Mw7tA=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2zy5WOe4j8G1rIhiUDwy2DrvGN7z8LqJEVTU1UzoJPp1BXF97WPZD0W3wQVGOZ1sfaE4HPcOT5L0Db8pL1jFQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '9f8908eed2b2d7e8c316bd5be7ed6e49024931ee', 'cluster': 'ceph'}}, {'kclient': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_failover']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-19_14:33:01', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-19T14:50:54.910 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-19T14:50:54.912 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-19T14:50:54.960 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-02-19T14:50:54.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-02-19T14:50:55.027 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:55.027 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T14:50:55.047 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T14:50:55.047 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-02-19 14:50:45.072706622 +0000 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-02-19 14:48:10.194533198 +0000 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-02-19 14:48:10.194533198 +0000 2024-02-19T14:50:55.103 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-02-19 14:48:10.194533198 +0000 2024-02-19T14:50:55.104 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T14:50:55.169 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-02-19T14:50:55.169 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-02-19T14:50:55.169 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000183452 s, 2.8 MB/s 2024-02-19T14:50:55.170 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T14:50:55.227 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-02-19 14:50:45.073706603 +0000 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-02-19 14:48:10.549526470 +0000 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-02-19 14:48:10.549526470 +0000 2024-02-19T14:50:55.282 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-02-19 14:48:10.549526470 +0000 2024-02-19T14:50:55.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T14:50:55.347 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-02-19T14:50:55.347 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-02-19T14:50:55.347 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000213218 s, 2.4 MB/s 2024-02-19T14:50:55.349 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T14:50:55.404 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-02-19 14:50:45.073706603 +0000 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-02-19 14:48:10.888520045 +0000 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-02-19 14:48:10.888520045 +0000 2024-02-19T14:50:55.459 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-02-19 14:48:10.888520045 +0000 2024-02-19T14:50:55.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T14:50:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-02-19T14:50:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-02-19T14:50:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000169526 s, 3.0 MB/s 2024-02-19T14:50:55.526 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T14:50:55.581 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-02-19 14:50:45.074706585 +0000 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-02-19 14:48:11.236513450 +0000 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-02-19 14:48:11.236513450 +0000 2024-02-19T14:50:55.637 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-02-19 14:48:11.236513450 +0000 2024-02-19T14:50:55.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T14:50:55.702 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-02-19T14:50:55.702 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-02-19T14:50:55.702 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000211145 s, 2.4 MB/s 2024-02-19T14:50:55.704 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T14:50:55.759 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T14:50:55.759 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:55.759 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T14:50:55.776 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T14:50:55.776 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-19 14:50:38.217073784 +0000 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-19 14:48:10.185960404 +0000 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-19 14:48:10.185960404 +0000 2024-02-19T14:50:55.831 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-19 14:48:10.185960404 +0000 2024-02-19T14:50:55.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T14:50:55.898 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-19T14:50:55.898 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-19T14:50:55.898 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000186855 s, 2.7 MB/s 2024-02-19T14:50:55.900 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T14:50:55.956 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-02-19T14:50:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T14:50:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:56.010 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-19T14:50:56.010 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:56.011 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:56.011 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-19 14:50:38.218073764 +0000 2024-02-19T14:50:56.011 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-19 14:48:10.540953213 +0000 2024-02-19T14:50:56.011 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-19 14:48:10.540953213 +0000 2024-02-19T14:50:56.011 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-19 14:48:10.540953213 +0000 2024-02-19T14:50:56.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T14:50:56.076 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-19T14:50:56.077 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-19T14:50:56.077 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000242121 s, 2.1 MB/s 2024-02-19T14:50:56.078 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T14:50:56.134 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-19 14:50:38.218073764 +0000 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-19 14:48:10.875946428 +0000 2024-02-19T14:50:56.189 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-19 14:48:10.875946428 +0000 2024-02-19T14:50:56.190 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-19 14:48:10.875946428 +0000 2024-02-19T14:50:56.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T14:50:56.255 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-19T14:50:56.255 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-19T14:50:56.255 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000221076 s, 2.3 MB/s 2024-02-19T14:50:56.256 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T14:50:56.312 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-02-19T14:50:56.366 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T14:50:56.366 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T14:50:56.366 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-19T14:50:56.366 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T14:50:56.367 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T14:50:56.367 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-19 14:50:38.219073744 +0000 2024-02-19T14:50:56.367 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-19 14:48:11.227939299 +0000 2024-02-19T14:50:56.367 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-19 14:48:11.227939299 +0000 2024-02-19T14:50:56.367 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-19 14:48:11.227939299 +0000 2024-02-19T14:50:56.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T14:50:56.432 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-19T14:50:56.432 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-19T14:50:56.432 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00021492 s, 2.4 MB/s 2024-02-19T14:50:56.433 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T14:50:56.489 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T14:50:56.489 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-19T14:50:56.489 INFO:tasks.ceph:Generating config... 2024-02-19T14:50:56.490 INFO:tasks.ceph:[client] client mount timeout = 600 2024-02-19T14:50:56.490 INFO:tasks.ceph:[client] debug client = 20 2024-02-19T14:50:56.490 INFO:tasks.ceph:[client] debug ms = 1 2024-02-19T14:50:56.491 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-02-19T14:50:56.491 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-02-19T14:50:56.491 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] debug mds = 20 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] debug ms = 1 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] mds debug frag = True 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-02-19T14:50:56.491 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-19T14:50:56.492 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-02-19T14:50:56.492 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-19T14:50:56.492 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-19T14:50:56.492 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-02-19T14:50:56.492 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-02-19T14:50:56.492 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-19T14:50:56.492 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-19T14:50:56.493 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-02-19T14:50:56.493 INFO:tasks.ceph:Setting up mon.a... 2024-02-19T14:50:56.493 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-02-19T14:50:56.561 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.keyring 2024-02-19T14:50:56.564 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-02-19T14:50:56.610 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-19T14:50:56.636 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.78'), ('mon.b', '172.21.15.150'), ('mon.c', '[v2:172.21.15.150:3301,v1:172.21.15.150:6790]')] 2024-02-19T14:50:56.637 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.78,172.21.15.150,[v2:172.21.15.150:3301,v1:172.21.15.150:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-02-19T14:50:56.637 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:56.637 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-19T14:50:56.691 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --add b 172.21.15.150 --addv c '[v2:172.21.15.150:3301,v1:172.21.15.150:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:50:56.775 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid 7cde3b0a-2943-4d9b-b265-05e609d5a6b3 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:fsid 7cde3b0a-2943-4d9b-b265-05e609d5a6b3 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2024-02-19T14:50:56.774288+0000 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-02-19T14:50:56.774288+0000 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 17 (quincy) 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-02-19T14:50:56.776 INFO:teuthology.orchestra.run.smithi078.stdout:2: [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] mon.c 2024-02-19T14:50:56.777 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-02-19T14:50:56.778 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-19T14:50:56.793 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7cde3b0a-2943-4d9b-b265-05e609d5a6b3... 2024-02-19T14:50:56.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T14:50:56.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-19T14:50:56.883 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-02-19T14:50:56.883 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-02-19T14:50:56.883 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T14:50:56.883 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-02-19T14:50:56.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-02-19T14:50:56.885 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-02-19T14:50:56.886 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "172.21.15.78,172.21.15.150,[v2:172.21.15.150:3301,v1:172.21.15.150:6790]" 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on skipped message = False 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 7cde3b0a-2943-4d9b-b265-05e609d5a6b3 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-02-19T14:50:56.887 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-02-19T14:50:56.888 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 20 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 20 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 10 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-02-19T14:50:56.889 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: osd op complaint time = 180 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-02-19T14:50:56.890 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T14:50:56.891 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-02-19T14:50:56.892 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: mon op complaint time = 120 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: client mount timeout = 600 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: debug client = 20 2024-02-19T14:50:56.893 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: rados mon op timeout = 900 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: rados osd op timeout = 900 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout:[mds] 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: debug mds = 20 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: debug mds balancer = 20 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: mds debug frag = True 2024-02-19T14:50:56.894 INFO:teuthology.orchestra.run.smithi078.stdout: mds debug scatterstat = True 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: mds op complaint time = 180 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: mds verify scatter = True 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: osd op complaint time = 180 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: rados mon op timeout = 900 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi078.stdout: rados osd op timeout = 900 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-02-19T14:50:56.895 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-02-19T14:50:56.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-02-19T14:50:56.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.78,172.21.15.150,[v2:172.21.15.150:3301,v1:172.21.15.150:6790]" 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 7cde3b0a-2943-4d9b-b265-05e609d5a6b3 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-02-19T14:50:56.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.900 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-02-19T14:50:56.901 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-02-19T14:50:56.902 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-02-19T14:50:56.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-02-19T14:50:56.904 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-02-19T14:50:56.905 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-02-19T14:50:56.906 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-02-19T14:50:56.907 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-02-19T14:50:56.909 INFO:tasks.ceph:Creating admin key on mon.a... 2024-02-19T14:50:56.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-02-19T14:50:57.003 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-19T14:50:57.004 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:57.004 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-02-19T14:50:57.019 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:57.019 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-02-19T14:50:57.074 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:50:57.074 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:57.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-19T14:50:57.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-19T14:50:57.153 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:50:57.153 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:50:57.208 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:50:57.208 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:57.208 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-19T14:50:57.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-19T14:50:57.249 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:50:57.249 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:50:57.304 INFO:tasks.ceph:Setting up mon nodes... 2024-02-19T14:50:57.305 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-19T14:50:57.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-02-19T14:50:57.359 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-02-19T14:50:57.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-02-19T14:50:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-02-19T14:50:57.427 INFO:tasks.ceph:Setting up mds nodes... 2024-02-19T14:50:57.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-02-19T14:50:57.481 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-02-19T14:50:57.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-02-19T14:50:57.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-02-19T14:50:57.604 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-02-19T14:50:57.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-02-19T14:50:57.633 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-02-19T14:50:57.688 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-02-19T14:50:57.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-02-19T14:50:57.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-02-19T14:50:57.812 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-02-19T14:50:57.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-02-19T14:50:57.841 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-19T14:50:57.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-19T14:50:57.883 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-02-19T14:50:57.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-19T14:50:57.940 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-02-19T14:50:57.955 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-19T14:50:57.955 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-19T14:50:57.955 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-02-19T14:50:57.981 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T14:50:57.981 INFO:tasks.ceph:role: osd.0 2024-02-19T14:50:57.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:50:57.982 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:50:58.056 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:50:58.057 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:50:58.549 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-02-19T14:50:58.551 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-02-19T14:50:58.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-02-19T14:50:58.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-02-19T14:50:58.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-02-19T14:50:58.787 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T14:50:58.787 INFO:tasks.ceph:role: osd.1 2024-02-19T14:50:58.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:50:58.787 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:50:58.862 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:50:59.331 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-02-19T14:50:59.333 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-02-19T14:50:59.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-02-19T14:50:59.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-02-19T14:50:59.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-02-19T14:50:59.525 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T14:50:59.526 INFO:tasks.ceph:role: osd.2 2024-02-19T14:50:59.526 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:50:59.526 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:50:59.600 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:50:59.601 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:50:59.601 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:50:59.601 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:51:00.084 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-02-19T14:51:00.086 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-02-19T14:51:00.086 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-02-19T14:51:00.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-02-19T14:51:00.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-02-19T14:51:00.322 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-19T14:51:00.322 INFO:tasks.ceph:role: osd.3 2024-02-19T14:51:00.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:51:00.323 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:51:00.398 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:51:00.893 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-02-19T14:51:00.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-02-19T14:51:00.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-02-19T14:51:01.025 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-02-19T14:51:01.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:01.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:01.150+0000 7f412fa0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-02-19T14:51:01.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:01.150+0000 7f412fa0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-02-19T14:51:01.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:01.150+0000 7f412fa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-19T14:51:01.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:01.150+0000 7f412fa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-19T14:51:01.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:01.150+0000 7f412fa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-19T14:51:01.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:01.150+0000 7f412fa0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-02-19T14:51:04.007 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T14:51:04.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:04.124+0000 7f3fb9e99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-02-19T14:51:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:04.124+0000 7f3fb9e99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-02-19T14:51:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:04.124+0000 7f3fb9e99640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-19T14:51:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:04.124+0000 7f3fb9e99640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-19T14:51:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:04.124+0000 7f3fb9e99640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-19T14:51:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:04.125+0000 7f3fb9e99640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-02-19T14:51:06.969 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T14:51:07.001 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:07.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:07.093+0000 7f35fe824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-02-19T14:51:07.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:07.093+0000 7f35fe824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-02-19T14:51:07.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:07.094+0000 7f35fe824640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-19T14:51:07.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:07.094+0000 7f35fe824640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-19T14:51:07.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:07.094+0000 7f35fe824640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-19T14:51:07.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:07.094+0000 7f35fe824640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-02-19T14:51:09.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T14:51:09.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:10.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:10.058+0000 7f7c47413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-02-19T14:51:10.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:10.058+0000 7f7c47413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-02-19T14:51:10.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:10.058+0000 7f7c47413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-19T14:51:10.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:10.058+0000 7f7c47413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-19T14:51:10.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:10.058+0000 7f7c47413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-19T14:51:10.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:10.058+0000 7f7c47413640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-02-19T14:51:12.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T14:51:12.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-02-19T14:51:12.976 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T14:51:12.977 INFO:tasks.ceph:role: osd.4 2024-02-19T14:51:12.977 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:51:12.977 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-19T14:51:13.055 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:51:13.055 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:51:13.056 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:51:13.569 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-19T14:51:13.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-19T14:51:13.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-02-19T14:51:13.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-02-19T14:51:13.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-02-19T14:51:13.791 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T14:51:13.791 INFO:tasks.ceph:role: osd.5 2024-02-19T14:51:13.791 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:51:13.791 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-19T14:51:13.868 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:51:13.869 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:51:14.325 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-19T14:51:14.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-19T14:51:14.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-02-19T14:51:14.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-02-19T14:51:14.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-02-19T14:51:14.518 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T14:51:14.518 INFO:tasks.ceph:role: osd.6 2024-02-19T14:51:14.518 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:51:14.518 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-19T14:51:14.592 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:51:14.592 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:51:14.592 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:51:14.592 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:51:14.592 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:51:14.593 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:51:14.593 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:51:14.593 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:51:14.593 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:51:14.593 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:51:15.077 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-19T14:51:15.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-19T14:51:15.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-02-19T14:51:15.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-02-19T14:51:15.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-02-19T14:51:15.329 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-19T14:51:15.330 INFO:tasks.ceph:role: osd.7 2024-02-19T14:51:15.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:51:15.330 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-19T14:51:15.404 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-19T14:51:15.404 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-19T14:51:15.404 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-19T14:51:15.404 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-19T14:51:15.405 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-19T14:51:15.405 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-02-19T14:51:15.405 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-19T14:51:15.405 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-19T14:51:15.405 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-19T14:51:15.405 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-19T14:51:15.891 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-02-19T14:51:15.893 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-02-19T14:51:15.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-02-19T14:51:16.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-02-19T14:51:16.060 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:16.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:16.180+0000 7fadf4c68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-02-19T14:51:16.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:16.181+0000 7fadf4c68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-02-19T14:51:16.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:16.181+0000 7fadf4c68640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-19T14:51:16.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:16.181+0000 7fadf4c68640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-19T14:51:16.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:16.181+0000 7fadf4c68640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-19T14:51:16.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:16.182+0000 7fadf4c68640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-02-19T14:51:19.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T14:51:19.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:19.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:19.155+0000 7f578320d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-02-19T14:51:19.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:19.155+0000 7f578320d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-02-19T14:51:19.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:19.155+0000 7f578320d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-19T14:51:19.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:19.155+0000 7f578320d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-19T14:51:19.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:19.155+0000 7f578320d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-19T14:51:19.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:19.156+0000 7f578320d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-02-19T14:51:22.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T14:51:22.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:22.131+0000 7f8763a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-02-19T14:51:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:22.132+0000 7f8763a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-02-19T14:51:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:22.132+0000 7f8763a13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-19T14:51:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:22.132+0000 7f8763a13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-19T14:51:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:22.132+0000 7f8763a13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-19T14:51:22.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:22.132+0000 7f8763a13640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-02-19T14:51:24.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:51:24.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:51:24.991 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-19T14:51:25.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:25.113+0000 7f99a0622640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-02-19T14:51:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:25.114+0000 7f99a0622640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-02-19T14:51:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:25.114+0000 7f99a0622640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-19T14:51:25.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:25.114+0000 7f99a0622640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-19T14:51:25.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:25.114+0000 7f99a0622640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-19T14:51:25.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:25.114+0000 7f99a0622640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-02-19T14:51:27.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-19T14:51:27.996 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-19T14:51:27.996 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:27.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-02-19T14:51:28.029 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.029 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-02-19T14:51:28.099 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.099 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-02-19T14:51:28.168 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.168 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-02-19T14:51:28.236 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.236 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-02-19T14:51:28.303 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.304 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-02-19T14:51:28.371 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-02-19T14:51:28.438 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-02-19T14:51:28.470 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-02-19T14:51:28.541 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-02-19T14:51:28.610 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-02-19T14:51:28.678 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-02-19T14:51:28.746 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-02-19T14:51:28.814 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-02-19T14:51:28.881 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:28.882 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-02-19T14:51:28.897 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:28.897 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-02-19T14:51:28.936 INFO:tasks.ceph:Adding keys to all mons... 2024-02-19T14:51:28.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-19T14:51:28.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-19T14:51:28.979 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.y] 2024-02-19T14:51:28.979 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ== 2024-02-19T14:51:28.979 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A== 2024-02-19T14:51:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.6] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.7] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2024-02-19T14:51:28.981 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw== 2024-02-19T14:51:29.001 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2024-02-19T14:51:29.001 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ== 2024-02-19T14:51:29.001 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-02-19T14:51:29.001 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A== 2024-02-19T14:51:29.002 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:51:29.003 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2024-02-19T14:51:29.004 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw== 2024-02-19T14:51:29.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T14:51:29.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T14:51:29.096 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.319 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T14:51:29.427 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-19T14:51:29.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-19T14:51:29.592 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.646 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-19T14:51:29.809 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T14:51:29.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T14:51:29.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T14:51:29.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-19T14:51:29.923 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-19T14:51:29.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-02-19T14:51:29.950 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-19T14:51:30.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-02-19T14:51:30.073 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-02-19T14:51:30.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-19T14:51:30.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-02-19T14:51:30.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-02-19T14:51:30.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-19T14:51:30.385 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-02-19T14:51:30.411 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:30.413 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-19T14:51:30.466 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-02-19T14:51:30.467 INFO:tasks.ceph.mon.a:Restarting daemon 2024-02-19T14:51:30.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-02-19T14:51:30.469 INFO:tasks.ceph.mon.a:Started 2024-02-19T14:51:30.469 INFO:tasks.ceph.mon.b:Restarting daemon 2024-02-19T14:51:30.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-02-19T14:51:30.509 INFO:tasks.ceph.mon.b:Started 2024-02-19T14:51:30.509 INFO:tasks.ceph.mon.c:Restarting daemon 2024-02-19T14:51:30.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-02-19T14:51:30.511 INFO:tasks.ceph.mon.c:Started 2024-02-19T14:51:30.511 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-02-19T14:51:30.511 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-02-19T14:51:30.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-02-19T14:51:30.514 INFO:tasks.ceph.mgr.y:Started 2024-02-19T14:51:30.514 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-02-19T14:51:30.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-02-19T14:51:30.517 INFO:tasks.ceph.mgr.x:Started 2024-02-19T14:51:30.517 INFO:tasks.ceph:Setting crush tunables to default 2024-02-19T14:51:30.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd crush tunables default 2024-02-19T14:51:30.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.615+0000 7f69728d4640 1 Processor -- start 2024-02-19T14:51:30.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.615+0000 7f69728d4640 1 -- start start 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f69728d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f69728d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f69728d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f69728d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c070bf0 con 0x7f696c150720 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f69728d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c06fc00 con 0x7f696c0c4c50 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f69728d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c4700 con 0x7f696c070270 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43744/0 (socket says 172.21.15.78:43744) 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 learned_addr learned my addr 172.21.15.78:0/2151135161 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.616+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T14:51:30.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:30.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-19T14:51:30.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:30.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:30.817+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-19T14:51:31.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:31.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:31.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:31.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T14:51:31.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:31.218+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T14:51:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696b7fe640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-19T14:51:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-19T14:51:32.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T14:51:32.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:32.019+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 msgr2=0x7f696c0c38e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f696c070270 0x7f696c0c38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 msgr2=0x7f696c0c6a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.150:3300/0 conn(0x7f696c0c4c50 0x7f696c0c6a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 -- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 msgr2=0x7f696c0c64d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 --2- 172.21.15.78:0/2151135161 >> v2:172.21.15.78:3300/0 conn(0x7f696c150720 0x7f696c0c64d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 -- 172.21.15.78:0/2151135161 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c153a60 con 0x7f6954003670 2024-02-19T14:51:33.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:33.616+0000 7f6969ffb640 1 -- 172.21.15.78:0/2151135161 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695400c080 con 0x7f6954007300 2024-02-19T14:51:35.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991085208 0 0) 0x7f696c153a60 con 0x7f6954003670 2024-02-19T14:51:35.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6958003520 con 0x7f6954003670 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477190921 0 0) 0x7f695400c080 con 0x7f6954007300 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696c153a60 con 0x7f6954007300 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306542744 0 0) 0x7f696c153a60 con 0x7f6954007300 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 >> v1:172.21.15.78:6789/0 conn(0x7f6954003670 legacy=0x7f6954005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695400d2e0 con 0x7f6954007300 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.835+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f696000d580 con 0x7f6954007300 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 >> v1:172.21.15.150:6789/0 conn(0x7f6954007300 legacy=0x7f69540097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f6970e71640 1 -- 172.21.15.78:0/2151135161 reap_dead start 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f6958003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:35.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f6958004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f695800bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695800c350 con 0x7f6958003ba0 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695800c490 con 0x7f69580047c0 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695800c650 con 0x7f6958009970 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f6958003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f6958004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f6970e71640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f695800bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 msgr2=0x7f695800bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f695800bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 msgr2=0x7f6958004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f6958004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696bfff640 1 -- 172.21.15.78:0/2151135161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c153a60 con 0x7f6958003ba0 2024-02-19T14:51:35.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.836+0000 7f696b7fe640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f6958004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.837+0000 7f696bfff640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f6958003f80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f695c012a70 tx=0x7f69580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a3063c8f517a3a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.838+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c036630 con 0x7f6958003ba0 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.838+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c00c040 con 0x7f6958003ba0 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f696affd640 1 -- 172.21.15.78:0/2151135161 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c036a10 con 0x7f6958003ba0 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 -- 172.21.15.78:0/2151135161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 msgr2=0x7f6958003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f6958003f80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f695c012a70 tx=0x7f69580057f0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 -- 172.21.15.78:0/2151135161 shutdown_connections 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f695800bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f6958004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 --2- 172.21.15.78:0/2151135161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f6958003f80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.839+0000 7f69728d4640 1 -- 172.21.15.78:0/2151135161 >> 172.21.15.78:0/2151135161 conn(0x7f696c0c5050 msgr2=0x7f696c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:35.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.840+0000 7f69728d4640 1 -- 172.21.15.78:0/2151135161 shutdown_connections 2024-02-19T14:51:35.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.840+0000 7f69728d4640 1 -- 172.21.15.78:0/2151135161 wait complete. 2024-02-19T14:51:35.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.841+0000 7f69728d4640 1 Processor -- start 2024-02-19T14:51:35.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 -- start start 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f696c14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f696c14c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f696c14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1489c0 con 0x7f6958003ba0 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c2110 con 0x7f69580047c0 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.842+0000 7f69728d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c148c10 con 0x7f6958009970 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f696c14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f696c14d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43842/0 (socket says 172.21.15.78:43842) 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f6970e71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f696c14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696bfff640 1 -- 172.21.15.78:0/1839140341 learned_addr learned my addr 172.21.15.78:0/1839140341 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f696c14c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 -- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 msgr2=0x7f696c14b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f696c14b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 -- 172.21.15.78:0/1839140341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 msgr2=0x7f696c14d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f696c14d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 -- 172.21.15.78:0/1839140341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c148ef0 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f6970e71640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f696c14b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696bfff640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f696c14d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.843+0000 7f696b7fe640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f696c14c690 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6960006e30 tx=0x7f6960015be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960015e30 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960013a40 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1491d0 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960013db0 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c14dcb0 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6960004020 con 0x7f69580047c0 2024-02-19T14:51:35.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.844+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6960003e20 con 0x7f69580047c0 2024-02-19T14:51:35.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.845+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c0c22e0 con 0x7f69580047c0 2024-02-19T14:51:35.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.848+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f696c0c22e0 con 0x7f69580047c0 2024-02-19T14:51:35.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.914+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f696c074990 con 0x7f69580047c0 2024-02-19T14:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.920+0000 7f6968ff9640 1 -- 172.21.15.78:0/1839140341 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6960026060 con 0x7f69580047c0 2024-02-19T14:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-02-19T14:51:35.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 msgr2=0x7f696c14c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f696c14c690 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6960006e30 tx=0x7f6960015be0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 shutdown_connections 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6958009970 0x7f696c14b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580047c0 0x7f696c14c690 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 --2- 172.21.15.78:0/1839140341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6958003ba0 0x7f696c14d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 >> 172.21.15.78:0/1839140341 conn(0x7f696c0c5050 msgr2=0x7f696c149c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 shutdown_connections 2024-02-19T14:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:35.921+0000 7f69728d4640 1 -- 172.21.15.78:0/1839140341 wait complete. 2024-02-19T14:51:35.936 INFO:tasks.ceph:check_enable_crimson: False 2024-02-19T14:51:35.936 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-02-19T14:51:35.936 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:35.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-02-19T14:51:35.964 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:35.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-02-19T14:51:36.031 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:36.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-02-19T14:51:36.098 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:51:36.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-02-19T14:51:36.164 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:36.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-02-19T14:51:36.193 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:36.193 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-02-19T14:51:36.260 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:36.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-02-19T14:51:36.316 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:36.314+0000 7f544b44b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-19T14:51:36.328 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:51:36.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-02-19T14:51:36.341 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:36.338+0000 7fe58942e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-19T14:51:36.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new cfeee3e6-366d-43a1-ae94-10a089e7e8c1 0 2024-02-19T14:51:36.369 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:36.366+0000 7f544b44b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T14:51:36.398 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:36.395+0000 7fe58942e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T14:51:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.495+0000 7f5b16e76640 1 Processor -- start 2024-02-19T14:51:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b16e76640 1 -- start start 2024-02-19T14:51:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b16e76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5b100c6d60 0x7f5b10154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b16e76640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b1006ff80 0x7f5b100c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b16e76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1014f730 con 0x7f5b1006e830 2024-02-19T14:51:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b16e76640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100c1940 con 0x7f5b100c6d60 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b16e76640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100c16d0 con 0x7f5b1006ff80 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b15413640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b1006ff80 0x7f5b100c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b15413640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b1006ff80 0x7f5b100c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54602/0 (socket says 172.21.15.150:54602) 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.496+0000 7f5b15413640 1 -- 172.21.15.150:0/2666169318 learned_addr learned my addr 172.21.15.150:0/2666169318 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b14c12640 1 --2- 172.21.15.150:0/2666169318 >> v2:172.21.15.78:3300/0 conn(0x7f5b100c6d60 0x7f5b10154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b15413640 1 -- 172.21.15.150:0/2666169318 >> v2:172.21.15.78:3300/0 conn(0x7f5b100c6d60 msgr2=0x7f5b10154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b15413640 1 --2- 172.21.15.150:0/2666169318 >> v2:172.21.15.78:3300/0 conn(0x7f5b100c6d60 0x7f5b10154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b15413640 1 -- 172.21.15.150:0/2666169318 >> v1:172.21.15.78:6789/0 conn(0x7f5b1006e830 legacy=0x7f5b100c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b15413640 1 -- 172.21.15.150:0/2666169318 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10072f70 con 0x7f5b1006ff80 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b14c12640 1 --2- 172.21.15.150:0/2666169318 >> v2:172.21.15.78:3300/0 conn(0x7f5b100c6d60 0x7f5b10154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.497+0000 7f5b15413640 1 --2- 172.21.15.150:0/2666169318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b100c7850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5b1006de00 tx=0x7f5b0802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3456381d3fa40a03 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b0f7fe640 1 -- 172.21.15.150:0/2666169318 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0802cb20 con 0x7f5b1006ff80 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b0f7fe640 1 -- 172.21.15.150:0/2666169318 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0802cc80 con 0x7f5b1006ff80 2024-02-19T14:51:36.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b0f7fe640 1 -- 172.21.15.150:0/2666169318 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080343f0 con 0x7f5b1006ff80 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 -- 172.21.15.150:0/2666169318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 msgr2=0x7f5b100c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 --2- 172.21.15.150:0/2666169318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b100c7850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5b1006de00 tx=0x7f5b0802bf80 comp rx=0 tx=0).stop 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 -- 172.21.15.150:0/2666169318 shutdown_connections 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 --2- 172.21.15.150:0/2666169318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b100c7850 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 --2- 172.21.15.150:0/2666169318 >> v2:172.21.15.78:3300/0 conn(0x7f5b100c6d60 0x7f5b10154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 -- 172.21.15.150:0/2666169318 >> 172.21.15.150:0/2666169318 conn(0x7f5b1006c170 msgr2=0x7f5b1006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 -- 172.21.15.150:0/2666169318 shutdown_connections 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.498+0000 7f5b16e76640 1 -- 172.21.15.150:0/2666169318 wait complete. 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.500+0000 7f5b16e76640 1 Processor -- start 2024-02-19T14:51:36.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.500+0000 7f5b16e76640 1 -- start start 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.501+0000 7f5b16e76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 0x7f5b100a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.501+0000 7f5b16e76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b10158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.501+0000 7f5b16e76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b100c6d60 0x7f5b10158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b16e76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100c22d0 con 0x7f5b100c6d60 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b16e76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100a7c80 con 0x7f5b1006ff80 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b16e76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100c26f0 con 0x7f5b1006e830 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 0x7f5b100a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b0ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b10158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 0x7f5b100a7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39712/0 (socket says 172.21.15.150:39712) 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 -- 172.21.15.150:0/4242198237 learned_addr learned my addr 172.21.15.150:0/4242198237 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b15413640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b100c6d60 0x7f5b10158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 -- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 msgr2=0x7f5b10158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b10158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 -- 172.21.15.150:0/4242198237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b100c6d60 msgr2=0x7f5b10158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b100c6d60 0x7f5b10158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 -- 172.21.15.150:0/4242198237 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08009b30 con 0x7f5b1006e830 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b0ffff640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b10158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b14c12640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 0x7f5b100a7ec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400e7e0 tx=0x7f5b0400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04004030 con 0x7f5b1006e830 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04004190 con 0x7f5b1006e830 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04004520 con 0x7f5b1006e830 2024-02-19T14:51:36.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b100c4c00 con 0x7f5b1006e830 2024-02-19T14:51:36.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b100c1ef0 con 0x7f5b1006e830 2024-02-19T14:51:36.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.502+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5b0400ca60 con 0x7f5b1006e830 2024-02-19T14:51:36.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.503+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5b04013970 con 0x7f5b1006e830 2024-02-19T14:51:36.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.503+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b100c2540 con 0x7f5b1006e830 2024-02-19T14:51:36.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.505+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5b04020060 con 0x7f5b1006e830 2024-02-19T14:51:36.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.576+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cfeee3e6-366d-43a1-ae94-10a089e7e8c1", "id": 0} v 0) v1 -- 0x7f5b10074990 con 0x7f5b1006e830 2024-02-19T14:51:36.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.579+0000 7f5b0d7fa640 1 -- 172.21.15.150:0/4242198237 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cfeee3e6-366d-43a1-ae94-10a089e7e8c1", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b0400f040 con 0x7f5b1006e830 2024-02-19T14:51:36.582 INFO:teuthology.orchestra.run.smithi150.stdout:0 2024-02-19T14:51:36.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 msgr2=0x7f5b100a7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 0x7f5b100a7ec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400e7e0 tx=0x7f5b0400eca0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 shutdown_connections 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b100c6d60 0x7f5b10158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1006ff80 0x7f5b10158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 --2- 172.21.15.150:0/4242198237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b1006e830 0x7f5b100a7ec0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 >> 172.21.15.150:0/4242198237 conn(0x7f5b1006c170 msgr2=0x7f5b100721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 shutdown_connections 2024-02-19T14:51:36.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.580+0000 7f5b16e76640 1 -- 172.21.15.150:0/4242198237 wait complete. 2024-02-19T14:51:36.593 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new eb08ce5d-c579-413d-8058-91c426ab9dc0 1 2024-02-19T14:51:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.692+0000 7f3443cf4640 1 Processor -- start 2024-02-19T14:51:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.692+0000 7f3443cf4640 1 -- start start 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3443cf4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f343c06e830 0x7f343c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3443cf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06ff80 0x7f343c0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3443cf4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c14f730 con 0x7f343c0c6d60 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3443cf4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c0c1940 con 0x7f343c06e830 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3443cf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c0c16d0 con 0x7f343c06ff80 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3441a90640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f343c0c6d60 0x7f343c0c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40760/0 (socket says 172.21.15.150:40760) 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3441a90640 1 -- 172.21.15.150:0/86974975 learned_addr learned my addr 172.21.15.150:0/86974975 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3442291640 1 --2- 172.21.15.150:0/86974975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06ff80 0x7f343c0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3440a8e640 1 -- 172.21.15.150:0/86974975 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234389954 0 0) 0x7f343c14f730 con 0x7f343c0c6d60 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f344128f640 1 --2- 172.21.15.150:0/86974975 >> v2:172.21.15.78:3300/0 conn(0x7f343c06e830 0x7f343c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3440a8e640 1 -- 172.21.15.150:0/86974975 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3428003520 con 0x7f343c0c6d60 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3442291640 1 -- 172.21.15.150:0/86974975 >> v2:172.21.15.78:3300/0 conn(0x7f343c06e830 msgr2=0x7f343c154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3442291640 1 --2- 172.21.15.150:0/86974975 >> v2:172.21.15.78:3300/0 conn(0x7f343c06e830 0x7f343c154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3442291640 1 -- 172.21.15.150:0/86974975 >> v1:172.21.15.150:6789/0 conn(0x7f343c0c6d60 legacy=0x7f343c0c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.693+0000 7f3442291640 1 -- 172.21.15.150:0/86974975 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c072f70 con 0x7f343c06ff80 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.694+0000 7f3442291640 1 --2- 172.21.15.150:0/86974975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06ff80 0x7f343c0c7850 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3434009d90 tx=0x7f343402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1036c715b7b5deff server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.694+0000 7f3440a8e640 1 -- 172.21.15.150:0/86974975 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343402cbc0 con 0x7f343c06ff80 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.694+0000 7f3440a8e640 1 -- 172.21.15.150:0/86974975 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343402cd20 con 0x7f343c06ff80 2024-02-19T14:51:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.694+0000 7f3440a8e640 1 -- 172.21.15.150:0/86974975 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434034660 con 0x7f343c06ff80 2024-02-19T14:51:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 -- 172.21.15.150:0/86974975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06ff80 msgr2=0x7f343c0c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 --2- 172.21.15.150:0/86974975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06ff80 0x7f343c0c7850 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3434009d90 tx=0x7f343402be10 comp rx=0 tx=0).stop 2024-02-19T14:51:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 -- 172.21.15.150:0/86974975 shutdown_connections 2024-02-19T14:51:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 --2- 172.21.15.150:0/86974975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06ff80 0x7f343c0c7850 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 --2- 172.21.15.150:0/86974975 >> v2:172.21.15.78:3300/0 conn(0x7f343c06e830 0x7f343c154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 -- 172.21.15.150:0/86974975 >> 172.21.15.150:0/86974975 conn(0x7f343c06c170 msgr2=0x7f343c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 -- 172.21.15.150:0/86974975 shutdown_connections 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.695+0000 7f3443cf4640 1 -- 172.21.15.150:0/86974975 wait complete. 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.696+0000 7f3443cf4640 1 Processor -- start 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.696+0000 7f3443cf4640 1 -- start start 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.696+0000 7f3443cf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 0x7f343c1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.696+0000 7f3443cf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f343c06ff80 0x7f343c0a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3443cf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f343c0c6d60 0x7f343c0a8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3443cf4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c14a580 con 0x7f343c0c6d60 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3443cf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c149ce0 con 0x7f343c06ff80 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3443cf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c149fc0 con 0x7f343c06e830 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 0x7f343c1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f344128f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f343c06ff80 0x7f343c0a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 0x7f343c1525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39728/0 (socket says 172.21.15.150:39728) 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 -- 172.21.15.150:0/2689024574 learned_addr learned my addr 172.21.15.150:0/2689024574 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3442291640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f343c0c6d60 0x7f343c0a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 -- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f343c06ff80 msgr2=0x7f343c0a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f343c06ff80 0x7f343c0a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 -- 172.21.15.150:0/2689024574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f343c0c6d60 msgr2=0x7f343c0a8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f343c0c6d60 0x7f343c0a8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 -- 172.21.15.150:0/2689024574 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434009a70 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3441a90640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 0x7f343c1525a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f343800c430 tx=0x7f343800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438005ce0 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438005e40 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c14a2a0 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34380072f0 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0a8650 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3438007450 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.697+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f34380128d0 con 0x7f343c06e830 2024-02-19T14:51:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.698+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c0a6f40 con 0x7f343c06e830 2024-02-19T14:51:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.700+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3438017070 con 0x7f343c06e830 2024-02-19T14:51:36.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.773+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "eb08ce5d-c579-413d-8058-91c426ab9dc0", "id": 1} v 0) v1 -- 0x7f343c1ee2f0 con 0x7f343c06e830 2024-02-19T14:51:36.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.779+0000 7f34327fc640 1 -- 172.21.15.150:0/2689024574 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb08ce5d-c579-413d-8058-91c426ab9dc0", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f343c1ee2f0 con 0x7f343c06e830 2024-02-19T14:51:36.781 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-02-19T14:51:36.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.779+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 msgr2=0x7f343c1525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.779+0000 7f3443cf4640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 0x7f343c1525a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f343800c430 tx=0x7f343800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.779+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 shutdown_connections 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.779+0000 7f3443cf4640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f343c0c6d60 0x7f343c0a8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.779+0000 7f3443cf4640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f343c06ff80 0x7f343c0a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.780+0000 7f3443cf4640 1 --2- 172.21.15.150:0/2689024574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f343c06e830 0x7f343c1525a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.780+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 >> 172.21.15.150:0/2689024574 conn(0x7f343c06c170 msgr2=0x7f343c155590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.780+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 shutdown_connections 2024-02-19T14:51:36.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.780+0000 7f3443cf4640 1 -- 172.21.15.150:0/2689024574 wait complete. 2024-02-19T14:51:36.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 068ab687-b58f-475b-9b39-e16546079e64 2 2024-02-19T14:51:36.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.894+0000 7feb355bc640 1 Processor -- start 2024-02-19T14:51:36.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.894+0000 7feb355bc640 1 -- start start 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb355bc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb300c6d60 0x7feb30154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb355bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb355bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3014f730 con 0x7feb3006ff80 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb355bc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300c1940 con 0x7feb300c6d60 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb355bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb30072f70 con 0x7feb3006e830 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39732/0 (socket says 172.21.15.150:39732) 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 -- 172.21.15.150:0/2288751154 learned_addr learned my addr 172.21.15.150:0/2288751154 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2effd640 1 --2- 172.21.15.150:0/2288751154 >> v2:172.21.15.78:3300/0 conn(0x7feb300c6d60 0x7feb30154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 -- 172.21.15.150:0/2288751154 >> v2:172.21.15.78:3300/0 conn(0x7feb300c6d60 msgr2=0x7feb30154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:36.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 --2- 172.21.15.150:0/2288751154 >> v2:172.21.15.78:3300/0 conn(0x7feb300c6d60 0x7feb30154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 -- 172.21.15.150:0/2288751154 >> v1:172.21.15.78:6789/0 conn(0x7feb3006ff80 legacy=0x7feb300c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2e7fc640 1 -- 172.21.15.150:0/2288751154 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb300c1b50 con 0x7feb3006e830 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.895+0000 7feb2effd640 1 --2- 172.21.15.150:0/2288751154 >> v2:172.21.15.78:3300/0 conn(0x7feb300c6d60 0x7feb30154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.896+0000 7feb2e7fc640 1 --2- 172.21.15.150:0/2288751154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c0fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feb20009870 tx=0x7feb2002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40195045fb660cef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.896+0000 7feb2dffb640 1 -- 172.21.15.150:0/2288751154 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20031640 con 0x7feb3006e830 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.896+0000 7feb2dffb640 1 -- 172.21.15.150:0/2288751154 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb200317a0 con 0x7feb3006e830 2024-02-19T14:51:36.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.896+0000 7feb2dffb640 1 -- 172.21.15.150:0/2288751154 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20031b30 con 0x7feb3006e830 2024-02-19T14:51:36.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 -- 172.21.15.150:0/2288751154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 msgr2=0x7feb300c0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 --2- 172.21.15.150:0/2288751154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c0fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feb20009870 tx=0x7feb2002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 -- 172.21.15.150:0/2288751154 shutdown_connections 2024-02-19T14:51:36.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 --2- 172.21.15.150:0/2288751154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c0fc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 --2- 172.21.15.150:0/2288751154 >> v2:172.21.15.78:3300/0 conn(0x7feb300c6d60 0x7feb30154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 -- 172.21.15.150:0/2288751154 >> 172.21.15.150:0/2288751154 conn(0x7feb3006c170 msgr2=0x7feb3006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 -- 172.21.15.150:0/2288751154 shutdown_connections 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.897+0000 7feb355bc640 1 -- 172.21.15.150:0/2288751154 wait complete. 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.898+0000 7feb355bc640 1 Processor -- start 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.898+0000 7feb355bc640 1 -- start start 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 0x7feb30149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb300c6d60 0x7feb300c1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300a7fb0 con 0x7feb300c6d60 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300a85f0 con 0x7feb3006ff80 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c43f0 con 0x7feb3006e830 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 0x7feb30149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39740/0 (socket says 172.21.15.150:39740) 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 -- 172.21.15.150:0/1832204833 learned_addr learned my addr 172.21.15.150:0/1832204833 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 0x7feb30149ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54628/0 (socket says 172.21.15.150:54628) 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2f7fe640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb300c6d60 0x7feb300c1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 -- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 msgr2=0x7feb30149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 0x7feb30149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 -- 172.21.15.150:0/1832204833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb300c6d60 msgr2=0x7feb300c1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb300c6d60 0x7feb300c1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 -- 172.21.15.150:0/1832204833 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200094d0 con 0x7feb3006e830 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2e7fc640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 0x7feb30149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb2effd640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c4880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feb2400d800 tx=0x7feb2400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24004030 con 0x7feb3006e830 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24004190 con 0x7feb3006e830 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb300a7cd0 con 0x7feb3006e830 2024-02-19T14:51:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24004520 con 0x7feb3006e830 2024-02-19T14:51:36.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb300c4080 con 0x7feb3006e830 2024-02-19T14:51:36.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.899+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7feb240096a0 con 0x7feb3006e830 2024-02-19T14:51:36.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.900+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7feb24003cf0 con 0x7feb3006e830 2024-02-19T14:51:36.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.900+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb300c3c00 con 0x7feb3006e830 2024-02-19T14:51:36.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.902+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7feb300c3c00 con 0x7feb3006e830 2024-02-19T14:51:36.935 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:36.933+0000 7f544b44b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T14:51:36.964 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:36.961+0000 7fe58942e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T14:51:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.975+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "068ab687-b58f-475b-9b39-e16546079e64", "id": 2} v 0) v1 -- 0x7feb30074990 con 0x7feb3006e830 2024-02-19T14:51:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb177fe640 1 -- 172.21.15.150:0/1832204833 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "068ab687-b58f-475b-9b39-e16546079e64", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb240037e0 con 0x7feb3006e830 2024-02-19T14:51:36.983 INFO:teuthology.orchestra.run.smithi150.stdout:2 2024-02-19T14:51:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 msgr2=0x7feb300c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c4880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feb2400d800 tx=0x7feb2400dcc0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 shutdown_connections 2024-02-19T14:51:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb300c6d60 0x7feb300c1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3006ff80 0x7feb30149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 --2- 172.21.15.150:0/1832204833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006e830 0x7feb300c4880 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.981+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 >> 172.21.15.150:0/1832204833 conn(0x7feb3006c170 msgr2=0x7feb300721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.982+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 shutdown_connections 2024-02-19T14:51:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:36.982+0000 7feb355bc640 1 -- 172.21.15.150:0/1832204833 wait complete. 2024-02-19T14:51:36.990 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:36.987+0000 7f544b44b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T14:51:36.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 043d280c-df76-4b38-b704-cb2fc622c5b6 3 2024-02-19T14:51:37.022 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.020+0000 7fe58942e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T14:51:37.034 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.032+0000 7f544b44b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T14:51:37.070 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.067+0000 7fe58942e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T14:51:37.088 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.085+0000 7f544b44b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T14:51:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.093+0000 7fc87f727640 1 Processor -- start 2024-02-19T14:51:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.093+0000 7fc87f727640 1 -- start start 2024-02-19T14:51:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87f727640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 0x7fc878159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87f727640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc8780c4d80 0x7fc8780c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87f727640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 0x7fc8781523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87f727640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87806b720 con 0x7fc8780c4d80 2024-02-19T14:51:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87f727640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87806ffa0 con 0x7fc8780c48f0 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87f727640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87806fc70 con 0x7fc8780c78a0 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87dcc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 0x7fc8781523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87d4c3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 0x7fc878159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87dcc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 0x7fc8781523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54636/0 (socket says 172.21.15.150:54636) 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87d4c3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 0x7fc878159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39752/0 (socket says 172.21.15.150:39752) 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87dcc4640 1 -- 172.21.15.150:0/3906470597 learned_addr learned my addr 172.21.15.150:0/3906470597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.094+0000 7fc87ccc2640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.78:3300/0 conn(0x7fc8780c4d80 0x7fc8780c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87d4c3640 1 -- 172.21.15.150:0/3906470597 >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 msgr2=0x7fc8781523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87d4c3640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 0x7fc8781523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87d4c3640 1 -- 172.21.15.150:0/3906470597 >> v2:172.21.15.78:3300/0 conn(0x7fc8780c4d80 msgr2=0x7fc8780c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87d4c3640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.78:3300/0 conn(0x7fc8780c4d80 0x7fc8780c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87d4c3640 1 -- 172.21.15.150:0/3906470597 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87806b930 con 0x7fc8780c78a0 2024-02-19T14:51:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87ccc2640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.78:3300/0 conn(0x7fc8780c4d80 0x7fc8780c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87dcc4640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 0x7fc8781523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.095+0000 7fc87d4c3640 1 --2- 172.21.15.150:0/3906470597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 0x7fc878159010 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8780c8680 tx=0x7fc86802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80fecd0c2c96b954 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc86ffff640 1 -- 172.21.15.150:0/3906470597 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86802f050 con 0x7fc8780c78a0 2024-02-19T14:51:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc86ffff640 1 -- 172.21.15.150:0/3906470597 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc868031700 con 0x7fc8780c78a0 2024-02-19T14:51:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc86ffff640 1 -- 172.21.15.150:0/3906470597 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868031a90 con 0x7fc8780c78a0 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 -- 172.21.15.150:0/3906470597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 msgr2=0x7fc878159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 --2- 172.21.15.150:0/3906470597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 0x7fc878159010 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8780c8680 tx=0x7fc86802b7a0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 -- 172.21.15.150:0/3906470597 shutdown_connections 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.150:3300/0 conn(0x7fc8780c48f0 0x7fc8781523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 --2- 172.21.15.150:0/3906470597 >> v2:172.21.15.78:3300/0 conn(0x7fc8780c4d80 0x7fc8780c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 --2- 172.21.15.150:0/3906470597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c78a0 0x7fc878159010 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 -- 172.21.15.150:0/3906470597 >> 172.21.15.150:0/3906470597 conn(0x7fc878066e70 msgr2=0x7fc8780a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 -- 172.21.15.150:0/3906470597 shutdown_connections 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.096+0000 7fc87f727640 1 -- 172.21.15.150:0/3906470597 wait complete. 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.097+0000 7fc87f727640 1 Processor -- start 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.097+0000 7fc87f727640 1 -- start start 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87f727640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8780c48f0 0x7fc8781ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87f727640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 0x7fc87814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87f727640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8780c78a0 0x7fc8781481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87f727640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8781ee850 con 0x7fc8780c48f0 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87f727640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8781edc40 con 0x7fc8780c78a0 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87f727640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8781edf20 con 0x7fc8780c4d80 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 0x7fc87814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 0x7fc87814e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39768/0 (socket says 172.21.15.150:39768) 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 -- 172.21.15.150:0/1674930438 learned_addr learned my addr 172.21.15.150:0/1674930438 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87dcc4640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8780c78a0 0x7fc8781481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87d4c3640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8780c48f0 0x7fc8781ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 -- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8780c78a0 msgr2=0x7fc8781481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8780c78a0 0x7fc8781481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 -- 172.21.15.150:0/1674930438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8780c48f0 msgr2=0x7fc8781ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.098+0000 7fc87ccc2640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8780c48f0 0x7fc8781ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc87ccc2640 1 -- 172.21.15.150:0/1674930438 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870009b30 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc87dcc4640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8780c78a0 0x7fc8781481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc87ccc2640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 0x7fc87814e190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc87400e7e0 tx=0x7fc87400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874004030 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc874004190 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc868009400 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc874004520 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8781ee4e0 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc87400ca60 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.099+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc87400cdd0 con 0x7fc8780c4d80 2024-02-19T14:51:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.100+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc878074990 con 0x7fc8780c4d80 2024-02-19T14:51:37.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.102+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc874020060 con 0x7fc8780c4d80 2024-02-19T14:51:37.128 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.125+0000 7fe58942e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T14:51:37.131 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.128+0000 7f544b44b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T14:51:37.172 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.169+0000 7fe58942e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T14:51:37.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.175+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "043d280c-df76-4b38-b704-cb2fc622c5b6", "id": 3} v 0) v1 -- 0x7fc8780c54a0 con 0x7fc8780c4d80 2024-02-19T14:51:37.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.180+0000 7fc86dffb640 1 -- 172.21.15.150:0/1674930438 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "043d280c-df76-4b38-b704-cb2fc622c5b6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc87400f040 con 0x7fc8780c4d80 2024-02-19T14:51:37.183 INFO:teuthology.orchestra.run.smithi150.stdout:3 2024-02-19T14:51:37.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 msgr2=0x7fc87814e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 0x7fc87814e190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc87400e7e0 tx=0x7fc87400eca0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 shutdown_connections 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8780c78a0 0x7fc8781481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8780c4d80 0x7fc87814e190 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 --2- 172.21.15.150:0/1674930438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8780c48f0 0x7fc8781ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 >> 172.21.15.150:0/1674930438 conn(0x7fc878066e70 msgr2=0x7fc878153460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.181+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 shutdown_connections 2024-02-19T14:51:37.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.182+0000 7fc87f727640 1 -- 172.21.15.150:0/1674930438 wait complete. 2024-02-19T14:51:37.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new ca904970-a7f2-4b96-bab2-fc2dc65022ac 4 2024-02-19T14:51:37.245 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.243+0000 7f544b44b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-19T14:51:37.294 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.292+0000 7fe58942e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-19T14:51:37.314 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.312+0000 7f544b44b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T14:51:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.334+0000 7f809e8f4640 1 Processor -- start 2024-02-19T14:51:37.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.334+0000 7f809e8f4640 1 -- start start 2024-02-19T14:51:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f809e8f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f809806ea50 0x7f8098072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f809e8f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8098152680 0x7f80980c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f809e8f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80980c5970 con 0x7f809814f1a0 2024-02-19T14:51:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f809e8f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80980c5750 con 0x7f809806ea50 2024-02-19T14:51:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f809e8f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80980702d0 con 0x7f8098152680 2024-02-19T14:51:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f809806ea50 0x7f8098072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f809806ea50 0x7f8098072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54662/0 (socket says 172.21.15.150:54662) 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f809ce91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8098152680 0x7f80980c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 -- 172.21.15.150:0/4174971824 learned_addr learned my addr 172.21.15.150:0/4174971824 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 -- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8098152680 msgr2=0x7f80980c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 --2- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8098152680 0x7f80980c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 -- 172.21.15.150:0/4174971824 >> v1:172.21.15.150:6789/0 conn(0x7f809814f1a0 legacy=0x7f809806fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f8096ffd640 1 -- 172.21.15.150:0/4174971824 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58900839 0 0) 0x7f80980c5970 con 0x7f809814f1a0 2024-02-19T14:51:37.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.335+0000 7f80977fe640 1 -- 172.21.15.150:0/4174971824 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80980c5bc0 con 0x7f809806ea50 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.336+0000 7f80977fe640 1 --2- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f809806ea50 0x7f8098072d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8088009870 tx=0x7f808802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f069afeb9703af server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.336+0000 7f8096ffd640 1 -- 172.21.15.150:0/4174971824 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80880317c0 con 0x7f809806ea50 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.336+0000 7f8096ffd640 1 -- 172.21.15.150:0/4174971824 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8088031920 con 0x7f809806ea50 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.336+0000 7f8096ffd640 1 -- 172.21.15.150:0/4174971824 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088031cb0 con 0x7f809806ea50 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 -- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f809806ea50 msgr2=0x7f8098072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f809806ea50 0x7f8098072d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8088009870 tx=0x7f808802b860 comp rx=0 tx=0).stop 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 -- 172.21.15.150:0/4174971824 shutdown_connections 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8098152680 0x7f80980c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4174971824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f809806ea50 0x7f8098072d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 -- 172.21.15.150:0/4174971824 >> 172.21.15.150:0/4174971824 conn(0x7f809806c170 msgr2=0x7f809806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 -- 172.21.15.150:0/4174971824 shutdown_connections 2024-02-19T14:51:37.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.337+0000 7f809e8f4640 1 -- 172.21.15.150:0/4174971824 wait complete. 2024-02-19T14:51:37.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.338+0000 7f809e8f4640 1 Processor -- start 2024-02-19T14:51:37.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.338+0000 7f809e8f4640 1 -- start start 2024-02-19T14:51:37.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809806ea50 0x7f8098153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f809814f1a0 0x7f80980c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 0x7f80981e5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8098155e00 con 0x7f809806ea50 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80980c3890 con 0x7f8098152680 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80980c3b70 con 0x7f809814f1a0 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 0x7f80981e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 0x7f80981e5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54666/0 (socket says 172.21.15.150:54666) 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f80977fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f809814f1a0 0x7f80980c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 -- 172.21.15.150:0/4116827116 learned_addr learned my addr 172.21.15.150:0/4116827116 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f8097fff640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809806ea50 0x7f8098153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 -- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f809814f1a0 msgr2=0x7f80980c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f809814f1a0 0x7f80980c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 -- 172.21.15.150:0/4116827116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809806ea50 msgr2=0x7f8098153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809806ea50 0x7f8098153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 -- 172.21.15.150:0/4116827116 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80880094d0 con 0x7f8098152680 2024-02-19T14:51:37.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f8097fff640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809806ea50 0x7f8098153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809ce91640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 0x7f80981e5750 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f809000efc0 tx=0x7f809000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090014480 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809000f040 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8098155740 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090014820 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.339+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8098154070 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.340+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8090014d10 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.340+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f80900139e0 con 0x7f8098152680 2024-02-19T14:51:37.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.341+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8098074990 con 0x7f8098152680 2024-02-19T14:51:37.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.343+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8090018020 con 0x7f8098152680 2024-02-19T14:51:37.360 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.358+0000 7f544b44b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T14:51:37.369 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.366+0000 7fe58942e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T14:51:37.411 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.409+0000 7f544b44b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T14:51:37.417 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.414+0000 7fe58942e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T14:51:37.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.416+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ca904970-a7f2-4b96-bab2-fc2dc65022ac", "id": 4} v 0) v1 -- 0x7f809806c940 con 0x7f8098152680 2024-02-19T14:51:37.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.421+0000 7f8094ff9640 1 -- 172.21.15.150:0/4116827116 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca904970-a7f2-4b96-bab2-fc2dc65022ac", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8090013e10 con 0x7f8098152680 2024-02-19T14:51:37.424 INFO:teuthology.orchestra.run.smithi150.stdout:4 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 msgr2=0x7f80981e5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 0x7f80981e5750 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f809000efc0 tx=0x7f809000c490 comp rx=0 tx=0).stop 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 shutdown_connections 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8098152680 0x7f80981e5750 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f809814f1a0 0x7f80980c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 --2- 172.21.15.150:0/4116827116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f809806ea50 0x7f8098153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 >> 172.21.15.150:0/4116827116 conn(0x7f809806c170 msgr2=0x7f809814c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 shutdown_connections 2024-02-19T14:51:37.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.422+0000 7f809e8f4640 1 -- 172.21.15.150:0/4116827116 wait complete. 2024-02-19T14:51:37.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 294f6374-334d-4b72-8374-40cf962e05f5 5 2024-02-19T14:51:37.458 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.456+0000 7f544b44b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-19T14:51:37.474 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.471+0000 7fe58942e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T14:51:37.523 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.521+0000 7fe58942e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-19T14:51:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.532+0000 7f0b0f6d4640 1 Processor -- start 2024-02-19T14:51:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0f6d4640 1 -- start start 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0f6d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b0806e830 0x7f0b08154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0f6d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0814f730 con 0x7f0b0806ff80 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0f6d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b080c16d0 con 0x7f0b080c6d60 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0f6d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b080c1940 con 0x7f0b0806e830 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0cc6f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b0806e830 0x7f0b08154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0d470640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0b080c6d60 0x7f0b080c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40768/0 (socket says 172.21.15.150:40768) 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.533+0000 7f0b0d470640 1 -- 172.21.15.150:0/1034230930 learned_addr learned my addr 172.21.15.150:0/1034230930 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.534+0000 7f0b0cc6f640 1 -- 172.21.15.150:0/1034230930 >> v1:172.21.15.150:6789/0 conn(0x7f0b080c6d60 legacy=0x7f0b080c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.534+0000 7f0b0cc6f640 1 -- 172.21.15.150:0/1034230930 >> v1:172.21.15.78:6789/0 conn(0x7f0b0806ff80 legacy=0x7f0b080c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.534+0000 7f0afffff640 1 -- 172.21.15.150:0/1034230930 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514957172 0 0) 0x7f0b080c16d0 con 0x7f0b080c6d60 2024-02-19T14:51:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.534+0000 7f0b0cc6f640 1 -- 172.21.15.150:0/1034230930 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b08072f70 con 0x7f0b0806e830 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.535+0000 7f0b0cc6f640 1 --2- 172.21.15.150:0/1034230930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b08154f70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0af80097a0 tx=0x7f0af802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cd394989a339e73 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.535+0000 7f0afffff640 1 -- 172.21.15.150:0/1034230930 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8030610 con 0x7f0b0806e830 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.535+0000 7f0afffff640 1 -- 172.21.15.150:0/1034230930 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af8030770 con 0x7f0b0806e830 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.535+0000 7f0afffff640 1 -- 172.21.15.150:0/1034230930 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8030b00 con 0x7f0b0806e830 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/1034230930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 msgr2=0x7f0b08154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 --2- 172.21.15.150:0/1034230930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b08154f70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0af80097a0 tx=0x7f0af802b960 comp rx=0 tx=0).stop 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/1034230930 shutdown_connections 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 --2- 172.21.15.150:0/1034230930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b08154f70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/1034230930 >> 172.21.15.150:0/1034230930 conn(0x7f0b0806c170 msgr2=0x7f0b0806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/1034230930 shutdown_connections 2024-02-19T14:51:37.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.536+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/1034230930 wait complete. 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.537+0000 7f0b0f6d4640 1 Processor -- start 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.537+0000 7f0b0f6d4640 1 -- start start 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.537+0000 7f0b0f6d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b081574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.537+0000 7f0b0f6d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0806ff80 0x7f0b08143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0f6d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b080c6d60 0x7f0b08143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0f6d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b081507e0 con 0x7f0b080c6d60 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0f6d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b081eaff0 con 0x7f0b0806e830 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0f6d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080a7bf0 con 0x7f0b0806ff80 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b081574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0cc6f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0806ff80 0x7f0b08143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b081574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54676/0 (socket says 172.21.15.150:54676) 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 -- 172.21.15.150:0/3763763649 learned_addr learned my addr 172.21.15.150:0/3763763649 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0dc71640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b080c6d60 0x7f0b08143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 -- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0806ff80 msgr2=0x7f0b08143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0806ff80 0x7f0b08143800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 -- 172.21.15.150:0/3763763649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b080c6d60 msgr2=0x7f0b08143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b080c6d60 0x7f0b08143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 -- 172.21.15.150:0/3763763649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af8009400 con 0x7f0b0806e830 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0d470640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b081574b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c4a0 tx=0x7f0b0400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b040074e0 con 0x7f0b0806e830 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b04007640 con 0x7f0b0806e830 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b081eb200 con 0x7f0b0806e830 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.538+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04004a00 con 0x7f0b0806e830 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.539+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b081eb7c0 con 0x7f0b0806e830 2024-02-19T14:51:37.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.539+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0b04004b60 con 0x7f0b0806e830 2024-02-19T14:51:37.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.539+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f0b04012980 con 0x7f0b0806e830 2024-02-19T14:51:37.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.539+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0806ef70 con 0x7f0b0806e830 2024-02-19T14:51:37.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.542+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0b04018070 con 0x7f0b0806e830 2024-02-19T14:51:37.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.614+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "294f6374-334d-4b72-8374-40cf962e05f5", "id": 5} v 0) v1 -- 0x7f0b080c62c0 con 0x7f0b0806e830 2024-02-19T14:51:37.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.617+0000 7f0afdffb640 1 -- 172.21.15.150:0/3763763649 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "294f6374-334d-4b72-8374-40cf962e05f5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0b040124d0 con 0x7f0b0806e830 2024-02-19T14:51:37.620 INFO:teuthology.orchestra.run.smithi150.stdout:5 2024-02-19T14:51:37.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 msgr2=0x7f0b081574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b081574b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400c4a0 tx=0x7f0b0400c960 comp rx=0 tx=0).stop 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 shutdown_connections 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b080c6d60 0x7f0b08143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0806ff80 0x7f0b08143800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 --2- 172.21.15.150:0/3763763649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b0806e830 0x7f0b081574b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 >> 172.21.15.150:0/3763763649 conn(0x7f0b0806c170 msgr2=0x7f0b080a9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 shutdown_connections 2024-02-19T14:51:37.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.618+0000 7f0b0f6d4640 1 -- 172.21.15.150:0/3763763649 wait complete. 2024-02-19T14:51:37.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 32d93613-4035-426d-8554-d3b5dddd289d 6 2024-02-19T14:51:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.770+0000 7f57271d0640 1 Processor -- start 2024-02-19T14:51:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.770+0000 7f57271d0640 1 -- start start 2024-02-19T14:51:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f57271d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f5720154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f57271d0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572014f730 con 0x7f572006ff80 2024-02-19T14:51:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f57271d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200c16d0 con 0x7f57200c6d60 2024-02-19T14:51:37.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f57271d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200c1940 con 0x7f572006e830 2024-02-19T14:51:37.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f571ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f5720154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f5724f6c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f57200c6d60 0x7f57200c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40782/0 (socket says 172.21.15.150:40782) 2024-02-19T14:51:37.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.771+0000 7f5724f6c640 1 -- 172.21.15.150:0/69428453 learned_addr learned my addr 172.21.15.150:0/69428453 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-02-19T14:51:37.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.772+0000 7f571f7fe640 1 -- 172.21.15.150:0/69428453 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469510989 0 0) 0x7f57200c16d0 con 0x7f57200c6d60 2024-02-19T14:51:37.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.772+0000 7f571ffff640 1 -- 172.21.15.150:0/69428453 >> v1:172.21.15.150:6789/0 conn(0x7f57200c6d60 legacy=0x7f57200c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.772+0000 7f571ffff640 1 -- 172.21.15.150:0/69428453 >> v1:172.21.15.78:6789/0 conn(0x7f572006ff80 legacy=0x7f57200c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.772+0000 7f571ffff640 1 -- 172.21.15.150:0/69428453 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5720072f70 con 0x7f572006e830 2024-02-19T14:51:37.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.772+0000 7f571ffff640 1 --2- 172.21.15.150:0/69428453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f5720154f70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097d0 tx=0x7f570c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1364f8704651b791 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f571f7fe640 1 -- 172.21.15.150:0/69428453 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c031800 con 0x7f572006e830 2024-02-19T14:51:37.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f571f7fe640 1 -- 172.21.15.150:0/69428453 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c031960 con 0x7f572006e830 2024-02-19T14:51:37.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f571f7fe640 1 -- 172.21.15.150:0/69428453 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c031cf0 con 0x7f572006e830 2024-02-19T14:51:37.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 -- 172.21.15.150:0/69428453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 msgr2=0x7f5720154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 --2- 172.21.15.150:0/69428453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f5720154f70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f570c0097d0 tx=0x7f570c02b8a0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 -- 172.21.15.150:0/69428453 shutdown_connections 2024-02-19T14:51:37.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 --2- 172.21.15.150:0/69428453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f5720154f70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 -- 172.21.15.150:0/69428453 >> 172.21.15.150:0/69428453 conn(0x7f572006c170 msgr2=0x7f572006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 -- 172.21.15.150:0/69428453 shutdown_connections 2024-02-19T14:51:37.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.773+0000 7f57271d0640 1 -- 172.21.15.150:0/69428453 wait complete. 2024-02-19T14:51:37.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.774+0000 7f57271d0640 1 Processor -- start 2024-02-19T14:51:37.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.774+0000 7f57271d0640 1 -- start start 2024-02-19T14:51:37.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f57271d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f57200c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f57271d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 0x7f5720149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f57271d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57200c6d60 0x7f57201eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f57271d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57201eb080 con 0x7f57200c6d60 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f57271d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57201eeb00 con 0x7f572006ff80 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f57271d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200aa090 con 0x7f572006e830 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f5724f6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f57200c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f571ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 0x7f5720149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f5724f6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f57200c34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39810/0 (socket says 172.21.15.150:39810) 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f5724f6c640 1 -- 172.21.15.150:0/43149850 learned_addr learned my addr 172.21.15.150:0/43149850 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.775+0000 7f571ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 0x7f5720149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54686/0 (socket says 172.21.15.150:54686) 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f572576d640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57200c6d60 0x7f57201eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f5724f6c640 1 -- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 msgr2=0x7f5720149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f5724f6c640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 0x7f5720149db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f5724f6c640 1 -- 172.21.15.150:0/43149850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57200c6d60 msgr2=0x7f57201eb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f5724f6c640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57200c6d60 0x7f57201eb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f5724f6c640 1 -- 172.21.15.150:0/43149850 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c009400 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f571ffff640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 0x7f5720149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f5724f6c640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f57200c34c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f571400c4d0 tx=0x7f571400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57140074e0 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714007640 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714004a20 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57201eace0 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572014a9c0 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5714004b80 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.776+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f5714004e30 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.777+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5720074990 con 0x7f572006e830 2024-02-19T14:51:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.779+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5714018070 con 0x7f572006e830 2024-02-19T14:51:37.834 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.832+0000 7f544b44b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T14:51:37.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.853+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "32d93613-4035-426d-8554-d3b5dddd289d", "id": 6} v 0) v1 -- 0x7f57200c09c0 con 0x7f572006e830 2024-02-19T14:51:37.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.858+0000 7f571d7fa640 1 -- 172.21.15.150:0/43149850 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "32d93613-4035-426d-8554-d3b5dddd289d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f571401d570 con 0x7f572006e830 2024-02-19T14:51:37.861 INFO:teuthology.orchestra.run.smithi150.stdout:6 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 msgr2=0x7f57200c34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f57200c34c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f571400c4d0 tx=0x7f571400c990 comp rx=0 tx=0).stop 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 shutdown_connections 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57200c6d60 0x7f57201eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f572006ff80 0x7f5720149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 --2- 172.21.15.150:0/43149850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f572006e830 0x7f57200c34c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 >> 172.21.15.150:0/43149850 conn(0x7f572006c170 msgr2=0x7f57200c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 shutdown_connections 2024-02-19T14:51:37.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.859+0000 7f57271d0640 1 -- 172.21.15.150:0/43149850 wait complete. 2024-02-19T14:51:37.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 7115141e-af2d-4ee8-84bf-8876f91b641e 7 2024-02-19T14:51:37.929 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:37.926+0000 7fe58942e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T14:51:37.967 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:37.965+0000 7f544b44b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T14:51:37.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 Processor -- start 2024-02-19T14:51:37.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 -- start start 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 0x7f30a4072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30a4152680 0x7f30a40c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40c5970 con 0x7f30a414f1a0 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40c5750 con 0x7f30a4152680 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.971+0000 7f30ac777640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40702d0 con 0x7f30a406ea50 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30a4152680 0x7f30a40c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30a9d12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 0x7f30a4072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30a4152680 0x7f30a40c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54694/0 (socket says 172.21.15.150:54694) 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 -- 172.21.15.150:0/831983729 learned_addr learned my addr 172.21.15.150:0/831983729 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30a9d12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 0x7f30a4072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39822/0 (socket says 172.21.15.150:39822) 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 -- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 msgr2=0x7f30a4072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 --2- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 0x7f30a4072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 -- 172.21.15.150:0/831983729 >> v1:172.21.15.78:6789/0 conn(0x7f30a414f1a0 legacy=0x7f30a406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30aad14640 1 -- 172.21.15.150:0/831983729 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a40c5bc0 con 0x7f30a4152680 2024-02-19T14:51:37.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.972+0000 7f30a9d12640 1 --2- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 0x7f30a4072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:37.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.973+0000 7f30aad14640 1 --2- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a4152680 0x7f30a40c78a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f309c00d150 tx=0x7f309c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24333a0a2d828d52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.973+0000 7f30a9511640 1 -- 172.21.15.150:0/831983729 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c02e9e0 con 0x7f30a4152680 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.973+0000 7f30a9511640 1 -- 172.21.15.150:0/831983729 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c02eb40 con 0x7f30a4152680 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.973+0000 7f30a9511640 1 -- 172.21.15.150:0/831983729 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c009600 con 0x7f30a4152680 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.973+0000 7f30ac777640 1 -- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a4152680 msgr2=0x7f30a40c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.973+0000 7f30ac777640 1 --2- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a4152680 0x7f30a40c78a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f309c00d150 tx=0x7f309c02e040 comp rx=0 tx=0).stop 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.974+0000 7f30ac777640 1 -- 172.21.15.150:0/831983729 shutdown_connections 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.974+0000 7f30ac777640 1 --2- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a4152680 0x7f30a40c78a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.974+0000 7f30ac777640 1 --2- 172.21.15.150:0/831983729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a406ea50 0x7f30a4072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.974+0000 7f30ac777640 1 -- 172.21.15.150:0/831983729 >> 172.21.15.150:0/831983729 conn(0x7f30a406c170 msgr2=0x7f30a406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.974+0000 7f30ac777640 1 -- 172.21.15.150:0/831983729 shutdown_connections 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.974+0000 7f30ac777640 1 -- 172.21.15.150:0/831983729 wait complete. 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 Processor -- start 2024-02-19T14:51:37.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 -- start start 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a406ea50 0x7f30a40bcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 0x7f30a414ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30a4152680 0x7f30a40c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4156190 con 0x7f30a4152680 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a41e6140 con 0x7f30a406ea50 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30ac777640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a41e7110 con 0x7f30a414f1a0 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30aa513640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a406ea50 0x7f30a40bcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30a9d12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 0x7f30a414ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30a9d12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 0x7f30a414ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:39824/0 (socket says 172.21.15.150:39824) 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30a9d12640 1 -- 172.21.15.150:0/3877647628 learned_addr learned my addr 172.21.15.150:0/3877647628 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-02-19T14:51:37.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30aa513640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a406ea50 0x7f30a40bcfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54698/0 (socket says 172.21.15.150:54698) 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.975+0000 7f30aad14640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30a4152680 0x7f30a40c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30a9d12640 1 -- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a406ea50 msgr2=0x7f30a40bcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30a9d12640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a406ea50 0x7f30a40bcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30a9d12640 1 -- 172.21.15.150:0/3877647628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30a4152680 msgr2=0x7f30a40c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30a9d12640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30a4152680 0x7f30a40c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30a9d12640 1 -- 172.21.15.150:0/3877647628 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094009400 con 0x7f30a414f1a0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30a9d12640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 0x7f30a414ac80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f309400e960 tx=0x7f309400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309400ca00 con 0x7f30a414f1a0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309400cb60 con 0x7f30a414f1a0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c00bd40 con 0x7f30a414f1a0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094013470 con 0x7f30a414f1a0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41e6350 con 0x7f30a414f1a0 2024-02-19T14:51:37.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3094005990 con 0x7f30a414f1a0 2024-02-19T14:51:37.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.976+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f3094013c30 con 0x7f30a414f1a0 2024-02-19T14:51:37.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.977+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4074990 con 0x7f30a414f1a0 2024-02-19T14:51:37.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:37.979+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3094021060 con 0x7f30a414f1a0 2024-02-19T14:51:38.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.051+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7115141e-af2d-4ee8-84bf-8876f91b641e", "id": 7} v 0) v1 -- 0x7f30a40bcda0 con 0x7f30a414f1a0 2024-02-19T14:51:38.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.059+0000 7f309affd640 1 -- 172.21.15.150:0/3877647628 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7115141e-af2d-4ee8-84bf-8876f91b641e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3094017070 con 0x7f30a414f1a0 2024-02-19T14:51:38.062 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-02-19T14:51:38.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 msgr2=0x7f30a414ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 0x7f30a414ac80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f309400e960 tx=0x7f309400ee20 comp rx=0 tx=0).stop 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 shutdown_connections 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30a4152680 0x7f30a40c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a414f1a0 0x7f30a414ac80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 --2- 172.21.15.150:0/3877647628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a406ea50 0x7f30a40bcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.060+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 >> 172.21.15.150:0/3877647628 conn(0x7f30a406c170 msgr2=0x7f30a414dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.061+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 shutdown_connections 2024-02-19T14:51:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-02-19T14:51:38.061+0000 7f30ac777640 1 -- 172.21.15.150:0/3877647628 wait complete. 2024-02-19T14:51:38.071 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.068+0000 7fe58942e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T14:51:38.074 INFO:tasks.ceph.osd.0:Restarting daemon 2024-02-19T14:51:38.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-02-19T14:51:38.076 INFO:tasks.ceph.osd.0:Started 2024-02-19T14:51:38.076 INFO:tasks.ceph.osd.1:Restarting daemon 2024-02-19T14:51:38.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-02-19T14:51:38.078 INFO:tasks.ceph.osd.1:Started 2024-02-19T14:51:38.078 INFO:tasks.ceph.osd.2:Restarting daemon 2024-02-19T14:51:38.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-02-19T14:51:38.080 INFO:tasks.ceph.osd.2:Started 2024-02-19T14:51:38.081 INFO:tasks.ceph.osd.3:Restarting daemon 2024-02-19T14:51:38.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-02-19T14:51:38.083 INFO:tasks.ceph.osd.3:Started 2024-02-19T14:51:38.083 INFO:tasks.ceph.osd.4:Restarting daemon 2024-02-19T14:51:38.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-02-19T14:51:38.085 INFO:tasks.ceph.osd.4:Started 2024-02-19T14:51:38.085 INFO:tasks.ceph.osd.5:Restarting daemon 2024-02-19T14:51:38.085 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-02-19T14:51:38.088 INFO:tasks.ceph.osd.5:Started 2024-02-19T14:51:38.088 INFO:tasks.ceph.osd.6:Restarting daemon 2024-02-19T14:51:38.088 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-02-19T14:51:38.090 INFO:tasks.ceph.osd.6:Started 2024-02-19T14:51:38.090 INFO:tasks.ceph.osd.7:Restarting daemon 2024-02-19T14:51:38.090 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-02-19T14:51:38.094 INFO:tasks.ceph.osd.7:Started 2024-02-19T14:51:38.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:51:38.098 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.096+0000 7f544b44b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T14:51:38.154 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.152+0000 7f544b44b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T14:51:38.210 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.207+0000 7fe58942e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T14:51:38.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.213+0000 7f6030e5c640 1 Processor -- start 2024-02-19T14:51:38.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.213+0000 7f6030e5c640 1 -- start start 2024-02-19T14:51:38.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f6030e5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f602c152700 0x7f602c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:38.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f6030e5c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c1551f0 con 0x7f602c0c8e00 2024-02-19T14:51:38.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f602a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f602c152700 0x7f602c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:38.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f602a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f602c152700 0x7f602c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56046/0 (socket says 172.21.15.78:56046) 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f602ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f602c0c8e00 0x7f602c15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41450/0 (socket says 172.21.15.78:41450) 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f602a59c640 1 -- 172.21.15.78:0/3574211140 learned_addr learned my addr 172.21.15.78:0/3574211140 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c155490 con 0x7f602c0c5730 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.214+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c155770 con 0x7f602c152700 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.215+0000 7f602a59c640 1 -- 172.21.15.78:0/3574211140 >> v1:172.21.15.150:6789/0 conn(0x7f602c0c5730 legacy=0x7f602c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.215+0000 7f602a59c640 1 -- 172.21.15.78:0/3574211140 >> v1:172.21.15.78:6789/0 conn(0x7f602c0c8e00 legacy=0x7f602c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.215+0000 7f602a59c640 1 -- 172.21.15.78:0/3574211140 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c155a50 con 0x7f602c152700 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.215+0000 7f602a59c640 1 --2- 172.21.15.78:0/3574211140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c152700 0x7f602c0c51f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f601c009870 tx=0x7f601c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b15ed85ccff71a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6029d9b640 1 -- 172.21.15.78:0/3574211140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c030860 con 0x7f602c152700 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6029d9b640 1 -- 172.21.15.78:0/3574211140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c0309c0 con 0x7f602c152700 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6029d9b640 1 -- 172.21.15.78:0/3574211140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c030d50 con 0x7f602c152700 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c152700 msgr2=0x7f602c0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 --2- 172.21.15.78:0/3574211140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c152700 0x7f602c0c51f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f601c009870 tx=0x7f601c02b9a0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 shutdown_connections 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 --2- 172.21.15.78:0/3574211140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c152700 0x7f602c0c51f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 >> 172.21.15.78:0/3574211140 conn(0x7f602c0cac90 msgr2=0x7f602c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:38.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 shutdown_connections 2024-02-19T14:51:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.216+0000 7f6030e5c640 1 -- 172.21.15.78:0/3574211140 wait complete. 2024-02-19T14:51:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.217+0000 7f6030e5c640 1 Processor -- start 2024-02-19T14:51:38.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.217+0000 7f6030e5c640 1 -- start start 2024-02-19T14:51:38.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6030e5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f602c0c5730 0x7f602c1ee400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:38.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6030e5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 0x7f602c0aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:38.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6030e5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f602c152700 0x7f602c0ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:38.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6030e5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c0c1940 con 0x7f602c0c8e00 2024-02-19T14:51:38.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6030e5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c0c1e70 con 0x7f602c0c5730 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6030e5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602c1eef10 con 0x7f602c152700 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6023fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 0x7f602c0aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6023fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 0x7f602c0aabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56060/0 (socket says 172.21.15.78:56060) 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6023fff640 1 -- 172.21.15.78:0/4220907349 learned_addr learned my addr 172.21.15.78:0/4220907349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f602a59c640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f602c0c5730 0x7f602c1ee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f602ad9d640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f602c152700 0x7f602c0ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.218+0000 7f6023fff640 1 -- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f602c152700 msgr2=0x7f602c0ab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6023fff640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f602c152700 0x7f602c0ab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6023fff640 1 -- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f602c0c5730 msgr2=0x7f602c1ee400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6023fff640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f602c0c5730 0x7f602c1ee400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6023fff640 1 -- 172.21.15.78:0/4220907349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c0094d0 con 0x7f602c0c8e00 2024-02-19T14:51:38.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f602a59c640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f602c0c5730 0x7f602c1ee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6023fff640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 0x7f602c0aabe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f601400c450 tx=0x7f601400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60140052d0 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6014005430 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c0c1b90 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014012870 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c0ab660 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f60140129d0 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.219+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f601401bad0 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.220+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c074990 con 0x7f602c0c8e00 2024-02-19T14:51:38.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.222+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f601401b400 con 0x7f602c0c8e00 2024-02-19T14:51:38.257 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.254+0000 7fe58942e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T14:51:38.281 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.278+0000 7f544b44b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T14:51:38.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.281+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f602c0c1600 con 0x7f602c0c8e00 2024-02-19T14:51:38.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.282+0000 7f6022ffd640 1 -- 172.21.15.78:0/4220907349 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f601402a090 con 0x7f602c0c8e00 2024-02-19T14:51:38.285 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:51:38.285 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:51:38.054002+0000","last_up_change":"0.000000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:51:38.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 msgr2=0x7f602c0aabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 0x7f602c0aabe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f601400c450 tx=0x7f601400c910 comp rx=0 tx=0).stop 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 shutdown_connections 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f602c0c8e00 0x7f602c0aabe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f602c152700 0x7f602c0ab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 --2- 172.21.15.78:0/4220907349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f602c0c5730 0x7f602c1ee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 >> 172.21.15.78:0/4220907349 conn(0x7f602c0cac90 msgr2=0x7f602c150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 shutdown_connections 2024-02-19T14:51:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:38.283+0000 7f6030e5c640 1 -- 172.21.15.78:0/4220907349 wait complete. 2024-02-19T14:51:38.296 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-02-19T14:51:38.296 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-19T14:51:38.369 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.367+0000 7fe58942e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T14:51:38.378 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.376+0000 7f544b44b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T14:51:38.466 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.463+0000 7fe58942e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T14:51:38.533 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.530+0000 7f544b44b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T14:51:38.575 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.572+0000 7f544b44b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T14:51:38.621 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.618+0000 7fe58942e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T14:51:38.663 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.661+0000 7fe58942e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T14:51:38.693 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-02-19T14:51:38.690+0000 7f4a6f49d640 -1 Falling back to public interface 2024-02-19T14:51:38.698 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-02-19T14:51:38.695+0000 7fb01181a640 -1 Falling back to public interface 2024-02-19T14:51:38.698 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-02-19T14:51:38.695+0000 7f99e3749640 -1 Falling back to public interface 2024-02-19T14:51:38.702 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-19T14:51:38.700+0000 7feff7c0f640 -1 Falling back to public interface 2024-02-19T14:51:38.703 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-02-19T14:51:38.700+0000 7fa5cd266640 -1 Falling back to public interface 2024-02-19T14:51:38.707 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-02-19T14:51:38.705+0000 7f9cb080e640 -1 Falling back to public interface 2024-02-19T14:51:38.710 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-02-19T14:51:38.707+0000 7f9889af9640 -1 Falling back to public interface 2024-02-19T14:51:38.711 INFO:tasks.ceph.osd.7.smithi150.stderr:2024-02-19T14:51:38.708+0000 7f9972d1d640 -1 Falling back to public interface 2024-02-19T14:51:38.731 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:38.728+0000 7f544b44b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T14:51:38.824 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:38.821+0000 7fe58942e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T14:51:39.103 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-19T14:51:39.194 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.191+0000 7fadd513e640 1 Processor -- start 2024-02-19T14:51:39.194 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.191+0000 7fadd513e640 1 -- start start 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadd513e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fadd00bcc60 0x7fadd0158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadd513e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fadd0159400 0x7fadd006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadd513e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd0158750 con 0x7fadd01580e0 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadd513e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd0158a30 con 0x7fadd0159400 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadd513e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd00c11b0 con 0x7fadd00bcc60 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadced9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fadd01580e0 0x7fadd0155b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41456/0 (socket says 172.21.15.78:41456) 2024-02-19T14:51:39.196 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadced9d640 1 -- 172.21.15.78:0/1500232078 learned_addr learned my addr 172.21.15.78:0/1500232078 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadcf59e640 1 --2- 172.21.15.78:0/1500232078 >> v2:172.21.15.78:3300/0 conn(0x7fadd0159400 0x7fadd006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadc7fff640 1 --2- 172.21.15.78:0/1500232078 >> v2:172.21.15.150:3300/0 conn(0x7fadd00bcc60 0x7fadd0158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadce59c640 1 -- 172.21.15.78:0/1500232078 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912208897 0 0) 0x7fadd0158750 con 0x7fadd01580e0 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadce59c640 1 -- 172.21.15.78:0/1500232078 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadbc003520 con 0x7fadd01580e0 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadcf59e640 1 -- 172.21.15.78:0/1500232078 >> v2:172.21.15.150:3300/0 conn(0x7fadd00bcc60 msgr2=0x7fadd0158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadcf59e640 1 --2- 172.21.15.78:0/1500232078 >> v2:172.21.15.150:3300/0 conn(0x7fadd00bcc60 0x7fadd0158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadcf59e640 1 -- 172.21.15.78:0/1500232078 >> v1:172.21.15.78:6789/0 conn(0x7fadd01580e0 legacy=0x7fadd0155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.192+0000 7fadcf59e640 1 -- 172.21.15.78:0/1500232078 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd01584c0 con 0x7fadd0159400 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadcf59e640 1 --2- 172.21.15.78:0/1500232078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd0159400 0x7fadd006e8c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fadd006f970 tx=0x7fadc802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b377c7c22bfca30 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadce59c640 1 -- 172.21.15.78:0/1500232078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc802cbf0 con 0x7fadd0159400 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadce59c640 1 -- 172.21.15.78:0/1500232078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc802cd50 con 0x7fadd0159400 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadce59c640 1 -- 172.21.15.78:0/1500232078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8034660 con 0x7fadd0159400 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadd513e640 1 -- 172.21.15.78:0/1500232078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd0159400 msgr2=0x7fadd006e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadd513e640 1 --2- 172.21.15.78:0/1500232078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd0159400 0x7fadd006e8c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fadd006f970 tx=0x7fadc802be40 comp rx=0 tx=0).stop 2024-02-19T14:51:39.197 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadd513e640 1 -- 172.21.15.78:0/1500232078 shutdown_connections 2024-02-19T14:51:39.198 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.193+0000 7fadd513e640 1 --2- 172.21.15.78:0/1500232078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd0159400 0x7fadd006e8c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.198 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.194+0000 7fadd513e640 1 --2- 172.21.15.78:0/1500232078 >> v2:172.21.15.150:3300/0 conn(0x7fadd00bcc60 0x7fadd0158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.198 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.194+0000 7fadd513e640 1 -- 172.21.15.78:0/1500232078 >> 172.21.15.78:0/1500232078 conn(0x7fadd00c6660 msgr2=0x7fadd00c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:39.198 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.194+0000 7fadd513e640 1 -- 172.21.15.78:0/1500232078 shutdown_connections 2024-02-19T14:51:39.198 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.194+0000 7fadd513e640 1 -- 172.21.15.78:0/1500232078 wait complete. 2024-02-19T14:51:39.200 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.195+0000 7fadd513e640 1 Processor -- start 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.195+0000 7fadd513e640 1 -- start start 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.195+0000 7fadd513e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd00bcc60 0x7fadd01ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.195+0000 7fadd513e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 0x7fadd0072e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.195+0000 7fadd513e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadd00c3580 0x7fadd00c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadd513e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd014e650 con 0x7fadd01580e0 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadd513e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd014e160 con 0x7fadd00bcc60 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadd513e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd014f9c0 con 0x7fadd00c3580 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 0x7fadd0072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 0x7fadd0072e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56078/0 (socket says 172.21.15.78:56078) 2024-02-19T14:51:39.201 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 -- 172.21.15.78:0/3001534154 learned_addr learned my addr 172.21.15.78:0/3001534154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadcf59e640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadd00c3580 0x7fadd00c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadced9d640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd00bcc60 0x7fadd01ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 -- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadd00c3580 msgr2=0x7fadd00c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadd00c3580 0x7fadd00c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 -- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd00bcc60 msgr2=0x7fadd01ee720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd00bcc60 0x7fadd01ee720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.202 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 -- 172.21.15.78:0/3001534154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc8009a70 con 0x7fadd01580e0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc7fff640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 0x7fadd0072e90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fadb800d800 tx=0x7fadb800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800b6c0 con 0x7fadd01580e0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb800b820 con 0x7fadd01580e0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800bbb0 con 0x7fadd01580e0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd014e850 con 0x7fadd01580e0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd014fc60 con 0x7fadd01580e0 2024-02-19T14:51:39.203 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fadb800bd10 con 0x7fadd01580e0 2024-02-19T14:51:39.204 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.196+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fadb8014c90 con 0x7fadd01580e0 2024-02-19T14:51:39.204 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.197+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd00749c0 con 0x7fadd01580e0 2024-02-19T14:51:39.204 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.199+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fadb80145c0 con 0x7fadd01580e0 2024-02-19T14:51:39.263 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.261+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadd006e6a0 con 0x7fadd01580e0 2024-02-19T14:51:39.264 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.261+0000 7fadc77fe640 1 -- 172.21.15.78:0/3001534154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fadb80147c0 con 0x7fadd01580e0 2024-02-19T14:51:39.264 INFO:teuthology.misc.health.smithi078.stdout: 2024-02-19T14:51:39.264 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":10,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:51:38.054002+0000","last_up_change":"0.000000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 msgr2=0x7fadd0072e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 0x7fadd0072e90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fadb800d800 tx=0x7fadb800dcc0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 shutdown_connections 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadd01580e0 0x7fadd0072e90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadd00c3580 0x7fadd00c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 --2- 172.21.15.78:0/3001534154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd00bcc60 0x7fadd01ee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:39.265 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 >> 172.21.15.78:0/3001534154 conn(0x7fadd00c6660 msgr2=0x7fadd0153f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:39.266 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 shutdown_connections 2024-02-19T14:51:39.266 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:39.262+0000 7fadd513e640 1 -- 172.21.15.78:0/3001534154 wait complete. 2024-02-19T14:51:39.273 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-19T14:51:39.602 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:39.599+0000 7f544b44b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T14:51:39.877 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:39.875+0000 7f544b44b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T14:51:39.922 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:39.920+0000 7fe58942e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T14:51:39.927 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:39.925+0000 7f544b44b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T14:51:40.041 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:40.039+0000 7f544b44b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T14:51:40.206 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:40.203+0000 7fe58942e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T14:51:40.256 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:40.253+0000 7fe58942e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T14:51:40.374 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-02-19T14:51:40.371+0000 7fe58942e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T14:51:41.067 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:41.064+0000 7f53ec142640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-19T14:51:41.067 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-02-19T14:51:41.065+0000 7f53ec142640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-19T14:51:42.180 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-02-19T14:51:42.178+0000 7f4a6f49d640 -1 osd.5 0 log_to_monitors true 2024-02-19T14:51:42.216 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-02-19T14:51:42.213+0000 7f9889af9640 -1 osd.4 0 log_to_monitors true 2024-02-19T14:51:42.221 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-02-19T14:51:42.219+0000 7fa5cd266640 -1 osd.0 0 log_to_monitors true 2024-02-19T14:51:42.224 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-02-19T14:51:42.221+0000 7f99e3749640 -1 osd.1 0 log_to_monitors true 2024-02-19T14:51:42.224 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-19T14:51:42.222+0000 7feff7c0f640 -1 osd.6 0 log_to_monitors true 2024-02-19T14:51:42.227 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-02-19T14:51:42.225+0000 7f9cb080e640 -1 osd.3 0 log_to_monitors true 2024-02-19T14:51:42.228 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-02-19T14:51:42.226+0000 7fb01181a640 -1 osd.2 0 log_to_monitors true 2024-02-19T14:51:42.242 INFO:tasks.ceph.osd.7.smithi150.stderr:2024-02-19T14:51:42.240+0000 7f9972d1d640 -1 osd.7 0 log_to_monitors true 2024-02-19T14:51:46.080 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-19T14:51:46.109 INFO:tasks.ceph.osd.7.smithi150.stderr:2024-02-19T14:51:46.106+0000 7f996ecb3640 -1 osd.7 0 waiting for initial osdmap 2024-02-19T14:51:46.121 INFO:tasks.ceph.osd.7.smithi150.stderr:2024-02-19T14:51:46.119+0000 7f9966ad3640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.191 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-02-19T14:51:46.188+0000 7f4a6bc47640 -1 osd.5 0 waiting for initial osdmap 2024-02-19T14:51:46.196 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-19T14:51:46.194+0000 7feff357d640 -1 osd.6 0 waiting for initial osdmap 2024-02-19T14:51:46.203 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-02-19T14:51:46.201+0000 7f4a63253640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.210 INFO:tasks.ceph.osd.6.smithi150.stderr:2024-02-19T14:51:46.208+0000 7fefeb39d640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.213 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-02-19T14:51:46.211+0000 7f9885a8f640 -1 osd.4 0 waiting for initial osdmap 2024-02-19T14:51:46.227 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.225+0000 7fa030577640 1 Processor -- start 2024-02-19T14:51:46.228 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.225+0000 7fa030577640 1 -- start start 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.226+0000 7fa030577640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0280c1060 0x7fa02815d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.226+0000 7fa030577640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0280c2980 0x7fa0280c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.226+0000 7fa030577640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa028159560 con 0x7fa028156af0 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.226+0000 7fa030577640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02815a550 con 0x7fa0280c2980 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.226+0000 7fa030577640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028159790 con 0x7fa0280c1060 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0280c1060 0x7fa02815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0280c1060 0x7fa02815d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:34950/0 (socket says 172.21.15.78:34950) 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 -- 172.21.15.78:0/2798752492 learned_addr learned my addr 172.21.15.78:0/2798752492 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02eb14640 1 --2- 172.21.15.78:0/2798752492 >> v2:172.21.15.78:3300/0 conn(0x7fa0280c2980 0x7fa0280c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 -- 172.21.15.78:0/2798752492 >> v2:172.21.15.78:3300/0 conn(0x7fa0280c2980 msgr2=0x7fa0280c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:46.230 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 --2- 172.21.15.78:0/2798752492 >> v2:172.21.15.78:3300/0 conn(0x7fa0280c2980 0x7fa0280c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.231 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 -- 172.21.15.78:0/2798752492 >> v1:172.21.15.78:6789/0 conn(0x7fa028156af0 legacy=0x7fa02806e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:46.231 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.227+0000 7fa02db12640 1 -- 172.21.15.78:0/2798752492 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028159a70 con 0x7fa0280c1060 2024-02-19T14:51:46.231 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-02-19T14:51:46.227+0000 7f987d8af640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.231 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-02-19T14:51:46.229+0000 7fa5c91fc640 -1 osd.0 0 waiting for initial osdmap 2024-02-19T14:51:46.232 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.229+0000 7fa02db12640 1 --2- 172.21.15.78:0/2798752492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c1060 0x7fa02815d560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa020009e50 tx=0x7fa02002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63cacba01ed36adb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:46.232 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.230+0000 7fa02d311640 1 -- 172.21.15.78:0/2798752492 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02002cb10 con 0x7fa0280c1060 2024-02-19T14:51:46.234 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.230+0000 7fa02d311640 1 -- 172.21.15.78:0/2798752492 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02002cc70 con 0x7fa0280c1060 2024-02-19T14:51:46.234 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.230+0000 7fa02d311640 1 -- 172.21.15.78:0/2798752492 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0200343f0 con 0x7fa0280c1060 2024-02-19T14:51:46.235 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.232+0000 7fa030577640 1 -- 172.21.15.78:0/2798752492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c1060 msgr2=0x7fa02815d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:46.235 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.232+0000 7fa030577640 1 --2- 172.21.15.78:0/2798752492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c1060 0x7fa02815d560 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa020009e50 tx=0x7fa02002bf70 comp rx=0 tx=0).stop 2024-02-19T14:51:46.235 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.233+0000 7fa030577640 1 -- 172.21.15.78:0/2798752492 shutdown_connections 2024-02-19T14:51:46.235 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.233+0000 7fa030577640 1 --2- 172.21.15.78:0/2798752492 >> v2:172.21.15.78:3300/0 conn(0x7fa0280c2980 0x7fa0280c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.235 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.233+0000 7fa030577640 1 --2- 172.21.15.78:0/2798752492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c1060 0x7fa02815d560 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa020009e50 tx=0x7fa02002bf70 comp rx=0 tx=0).stop 2024-02-19T14:51:46.235 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.233+0000 7fa030577640 1 -- 172.21.15.78:0/2798752492 >> 172.21.15.78:0/2798752492 conn(0x7fa0280c6660 msgr2=0x7fa0280c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:46.238 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.234+0000 7fa030577640 1 -- 172.21.15.78:0/2798752492 shutdown_connections 2024-02-19T14:51:46.238 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.234+0000 7fa030577640 1 -- 172.21.15.78:0/2798752492 wait complete. 2024-02-19T14:51:46.238 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.236+0000 7fa030577640 1 Processor -- start 2024-02-19T14:51:46.239 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.236+0000 7fa030577640 1 -- start start 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.237+0000 7fa030577640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c2980 0x7fa02815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.237+0000 7fa030577640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 0x7fa02815bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.237+0000 7fa030577640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280aaff0 0x7fa0280711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.237+0000 7fa030577640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02815ace0 con 0x7fa028156af0 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.237+0000 7fa030577640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0280a9f30 con 0x7fa0280c2980 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.237+0000 7fa030577640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02815b820 con 0x7fa0280aaff0 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.238+0000 7fa02db12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 0x7fa02815bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.238+0000 7fa02db12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 0x7fa02815bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56212/0 (socket says 172.21.15.78:56212) 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.238+0000 7fa02db12640 1 -- 172.21.15.78:0/4140735506 learned_addr learned my addr 172.21.15.78:0/4140735506 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:46.241 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.238+0000 7fa02e313640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c2980 0x7fa02815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.238+0000 7fa02eb14640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280aaff0 0x7fa0280711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.239+0000 7fa02db12640 1 -- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280aaff0 msgr2=0x7fa0280711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.239+0000 7fa02db12640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280aaff0 0x7fa0280711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.239+0000 7fa02db12640 1 -- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c2980 msgr2=0x7fa02815c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.239+0000 7fa02db12640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c2980 0x7fa02815c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.239+0000 7fa02db12640 1 -- 172.21.15.78:0/4140735506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa020009b30 con 0x7fa028156af0 2024-02-19T14:51:46.242 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.240+0000 7fa02db12640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 0x7fa02815bcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa020009f70 tx=0x7fa020004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:46.243 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.240+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02003e450 con 0x7fa028156af0 2024-02-19T14:51:46.243 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.240+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02815cde0 con 0x7fa028156af0 2024-02-19T14:51:46.244 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.241+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02003e5d0 con 0x7fa028156af0 2024-02-19T14:51:46.244 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.242+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02003e780 con 0x7fa028156af0 2024-02-19T14:51:46.246 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.242+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02815aef0 con 0x7fa028156af0 2024-02-19T14:51:46.246 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.242+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 89732+0+0 (secure 0 0 0) 0x7fa02003e9a0 con 0x7fa028156af0 2024-02-19T14:51:46.246 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.243+0000 7fa01effd640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b880 0x7fa00c06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:46.246 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.243+0000 7fa02e313640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b880 0x7fa00c06dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:46.246 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.243+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0280749c0 con 0x7fa028156af0 2024-02-19T14:51:46.248 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.243+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fa0200b0a90 con 0x7fa028156af0 2024-02-19T14:51:46.248 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.246+0000 7fa02e313640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b880 0x7fa00c06dd40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa024000f60 tx=0x7fa024009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:46.249 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-02-19T14:51:46.247+0000 7fa5c101c640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.255 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.253+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa0200b1060 con 0x7fa028156af0 2024-02-19T14:51:46.264 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-02-19T14:51:46.261+0000 7f9cac0c6640 -1 osd.3 0 waiting for initial osdmap 2024-02-19T14:51:46.266 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-02-19T14:51:46.264+0000 7fb00d20c640 -1 osd.2 0 waiting for initial osdmap 2024-02-19T14:51:46.269 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-02-19T14:51:46.267+0000 7f99df6df640 -1 osd.1 0 waiting for initial osdmap 2024-02-19T14:51:46.275 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-02-19T14:51:46.273+0000 7f9ca3ee6640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.278 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-02-19T14:51:46.276+0000 7fb00502c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.282 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-02-19T14:51:46.280+0000 7f99d74ff640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T14:51:46.432 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.430+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa02815b960 con 0x7fa028156af0 2024-02-19T14:51:46.434 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.432+0000 7fa01effd640 1 -- 172.21.15.78:0/4140735506 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa020043030 con 0x7fa028156af0 2024-02-19T14:51:46.435 INFO:teuthology.misc.health.smithi078.stdout: 2024-02-19T14:51:46.435 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":12,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:51:44.065319+0000","last_up_change":"0.000000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b880 msgr2=0x7fa00c06dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b880 0x7fa00c06dd40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa024000f60 tx=0x7fa024009040 comp rx=0 tx=0).stop 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 msgr2=0x7fa02815bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 0x7fa02815bcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa020009f70 tx=0x7fa020004a40 comp rx=0 tx=0).stop 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 shutdown_connections 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b880 0x7fa00c06dd40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa028156af0 0x7fa02815bcb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.437 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280aaff0 0x7fa0280711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.438 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 --2- 172.21.15.78:0/4140735506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280c2980 0x7fa02815c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:46.438 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 >> 172.21.15.78:0/4140735506 conn(0x7fa0280c6660 msgr2=0x7fa0280c6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:46.438 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 shutdown_connections 2024-02-19T14:51:46.438 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:46.434+0000 7fa030577640 1 -- 172.21.15.78:0/4140735506 wait complete. 2024-02-19T14:51:46.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-19T14:51:53.254 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-19T14:51:53.358 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.355+0000 7f9aec10d640 1 Processor -- start 2024-02-19T14:51:53.358 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 -- start start 2024-02-19T14:51:53.361 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9ae4158250 0x7f9ae406c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.361 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae40c3400 0x7f9ae40c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.361 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9ae4158f60 0x7f9ae40c3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.361 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae406ccb0 con 0x7f9ae4158250 2024-02-19T14:51:53.361 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae41587d0 con 0x7f9ae4158f60 2024-02-19T14:51:53.361 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.356+0000 7f9aec10d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c72f0 con 0x7f9ae40c3400 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9ae4158250 0x7f9ae406c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9ae4158250 0x7f9ae406c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58584/0 (socket says 172.21.15.78:58584) 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/359926325 learned_addr learned my addr 172.21.15.78:0/359926325 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9aea6aa640 1 --2- 172.21.15.78:0/359926325 >> v2:172.21.15.150:3300/0 conn(0x7f9ae4158f60 0x7f9ae40c3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae96a8640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae40c3400 0x7f9ae40c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/359926325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae40c3400 msgr2=0x7f9ae40c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae40c3400 0x7f9ae40c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/359926325 >> v2:172.21.15.150:3300/0 conn(0x7f9ae4158f60 msgr2=0x7f9ae40c3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 --2- 172.21.15.78:0/359926325 >> v2:172.21.15.150:3300/0 conn(0x7f9ae4158f60 0x7f9ae40c3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/359926325 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40c7590 con 0x7f9ae4158250 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.357+0000 7f9ae96a8640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae40c3400 0x7f9ae40c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9ae9ea9640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4158250 0x7f9ae406c770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40c8650 tx=0x7f9ad402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b6b3631a9e39c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9ae8ea7640 1 -- 172.21.15.78:0/359926325 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40317e0 con 0x7f9ae4158250 2024-02-19T14:51:53.362 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9ae8ea7640 1 -- 172.21.15.78:0/359926325 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4031940 con 0x7f9ae4158250 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9ae8ea7640 1 -- 172.21.15.78:0/359926325 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4031cd0 con 0x7f9ae4158250 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9aec10d640 1 -- 172.21.15.78:0/359926325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4158250 msgr2=0x7f9ae406c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9aec10d640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4158250 0x7f9ae406c770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40c8650 tx=0x7f9ad402b880 comp rx=0 tx=0).stop 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9aec10d640 1 -- 172.21.15.78:0/359926325 shutdown_connections 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9aec10d640 1 --2- 172.21.15.78:0/359926325 >> v2:172.21.15.150:3300/0 conn(0x7f9ae4158f60 0x7f9ae40c3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9aec10d640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae40c3400 0x7f9ae40c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.358+0000 7f9aec10d640 1 --2- 172.21.15.78:0/359926325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae4158250 0x7f9ae406c770 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.359+0000 7f9aec10d640 1 -- 172.21.15.78:0/359926325 >> 172.21.15.78:0/359926325 conn(0x7f9ae40c87f0 msgr2=0x7f9ae40c8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.359+0000 7f9aec10d640 1 -- 172.21.15.78:0/359926325 shutdown_connections 2024-02-19T14:51:53.363 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.359+0000 7f9aec10d640 1 -- 172.21.15.78:0/359926325 wait complete. 2024-02-19T14:51:53.366 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 Processor -- start 2024-02-19T14:51:53.366 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 -- start start 2024-02-19T14:51:53.366 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 0x7f9ae4156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4158250 0x7f9ae41e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae4158f60 0x7f9ae40ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41ee6a0 con 0x7f9ae40c3400 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae41ee980 con 0x7f9ae4158250 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.360+0000 7f9aec10d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40a8510 con 0x7f9ae4158f60 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 0x7f9ae4156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 0x7f9ae4156e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58600/0 (socket says 172.21.15.78:58600) 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/407216014 learned_addr learned my addr 172.21.15.78:0/407216014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae96a8640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4158250 0x7f9ae41e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9aea6aa640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae4158f60 0x7f9ae40ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae4158f60 msgr2=0x7f9ae40ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.367 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae4158f60 0x7f9ae40ac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4158250 msgr2=0x7f9ae41e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4158250 0x7f9ae41e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 -- 172.21.15.78:0/407216014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40094d0 con 0x7f9ae40c3400 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9aea6aa640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae4158f60 0x7f9ae40ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae96a8640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4158250 0x7f9ae41e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ae9ea9640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 0x7f9ae4156e60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4000c00 tx=0x7f9ad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.368 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4032880 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4032a20 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae40a87f0 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4032dd0 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.361+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae41573a0 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.363+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae40749c0 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.363+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9ad404e450 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.363+0000 7f9ada7fc640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ac806b810 0x7f9ac806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.363+0000 7f9ae96a8640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ac806b810 0x7f9ac806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.363+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9ad40abf20 con 0x7f9ae40c3400 2024-02-19T14:51:53.369 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.364+0000 7f9ae96a8640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ac806b810 0x7f9ac806dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40adf90 tx=0x7f9ae0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.370 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.368+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9ad4076f00 con 0x7f9ae40c3400 2024-02-19T14:51:53.542 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.539+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ae40c39e0 con 0x7f9ae40c3400 2024-02-19T14:51:53.542 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.540+0000 7f9ada7fc640 1 -- 172.21.15.78:0/407216014 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7f9ad407a5c0 con 0x7f9ae40c3400 2024-02-19T14:51:53.543 INFO:teuthology.misc.health.smithi078.stdout: 2024-02-19T14:51:53.543 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":16,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:51:50.115948+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ac806b810 msgr2=0x7f9ac806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ac806b810 0x7f9ac806dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40adf90 tx=0x7f9ae0008040 comp rx=0 tx=0).stop 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 msgr2=0x7f9ae4156e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 0x7f9ae4156e60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4000c00 tx=0x7f9ad40046c0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 shutdown_connections 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ac806b810 0x7f9ac806dcd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae4158f60 0x7f9ae40ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4158250 0x7f9ae41e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 --2- 172.21.15.78:0/407216014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ae40c3400 0x7f9ae4156e60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.545 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 >> 172.21.15.78:0/407216014 conn(0x7f9ae40c87f0 msgr2=0x7f9ae40c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:53.546 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 shutdown_connections 2024-02-19T14:51:53.546 INFO:teuthology.misc.health.smithi078.stderr:2024-02-19T14:51:53.542+0000 7f9aec10d640 1 -- 172.21.15.78:0/407216014 wait complete. 2024-02-19T14:51:53.554 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-02-19T14:51:53.554 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-02-19T14:51:53.555 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:51:53.555 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:51:53.558 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:51:53.558 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:51:53.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:51:53.560 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:51:53.560 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:51:53.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:51:53.563 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:51:53.563 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:51:53.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:51:53.565 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:51:53.565 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-02-19T14:51:53.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:51:53.633 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:51:53.636 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:51:53.647 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:51:53.648 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:51:53.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.669+0000 7f1794b93640 1 Processor -- start 2024-02-19T14:51:53.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.669+0000 7f1794b93640 1 -- start start 2024-02-19T14:51:53.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f1794b93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1790150460 0x7f17900c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f1794b93640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17900721a0 0x7f17900c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f1794b93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17900c7d90 0x7f17900c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f1794b93640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1790153e80 con 0x7f1790150460 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f1794b93640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1790072c60 con 0x7f17900721a0 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f1794b93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17901538c0 con 0x7f17900c7d90 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1790150460 0x7f17900c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1790150460 0x7f17900c1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58642/0 (socket says 172.21.15.78:58642) 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 -- 172.21.15.78:0/348183203 learned_addr learned my addr 172.21.15.78:0/348183203 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 -- 172.21.15.78:0/348183203 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17900c7d90 msgr2=0x7f17900c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 --2- 172.21.15.78:0/348183203 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17900c7d90 0x7f17900c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 -- 172.21.15.78:0/348183203 >> v2:172.21.15.150:3300/0 conn(0x7f17900721a0 msgr2=0x7f17900c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 --2- 172.21.15.78:0/348183203 >> v2:172.21.15.150:3300/0 conn(0x7f17900721a0 0x7f17900c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.670+0000 7f178e59c640 1 -- 172.21.15.78:0/348183203 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790153ba0 con 0x7f1790150460 2024-02-19T14:51:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.671+0000 7f178e59c640 1 --2- 172.21.15.78:0/348183203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1790150460 0x7f17900c1810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1790072130 tx=0x7f178002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae8e74e35b1ce83 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.671+0000 7f178dd9b640 1 -- 172.21.15.78:0/348183203 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780031870 con 0x7f1790150460 2024-02-19T14:51:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.671+0000 7f178dd9b640 1 -- 172.21.15.78:0/348183203 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17800319d0 con 0x7f1790150460 2024-02-19T14:51:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.671+0000 7f178dd9b640 1 -- 172.21.15.78:0/348183203 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780031d60 con 0x7f1790150460 2024-02-19T14:51:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 -- 172.21.15.78:0/348183203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1790150460 msgr2=0x7f17900c1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 --2- 172.21.15.78:0/348183203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1790150460 0x7f17900c1810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1790072130 tx=0x7f178002b910 comp rx=0 tx=0).stop 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 -- 172.21.15.78:0/348183203 shutdown_connections 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 --2- 172.21.15.78:0/348183203 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17900c7d90 0x7f17900c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 --2- 172.21.15.78:0/348183203 >> v2:172.21.15.150:3300/0 conn(0x7f17900721a0 0x7f17900c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 --2- 172.21.15.78:0/348183203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1790150460 0x7f17900c1810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 -- 172.21.15.78:0/348183203 >> 172.21.15.78:0/348183203 conn(0x7f179006c170 msgr2=0x7f179006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 -- 172.21.15.78:0/348183203 shutdown_connections 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.672+0000 7f1794b93640 1 -- 172.21.15.78:0/348183203 wait complete. 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.673+0000 7f1794b93640 1 Processor -- start 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.673+0000 7f1794b93640 1 -- start start 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17900721a0 0x7f17901e79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 0x7f17900c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1790150460 0x7f17900aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900abe20 con 0x7f17900c7d90 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1790156b10 con 0x7f17900721a0 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17901eafb0 con 0x7f1790150460 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 0x7f17900c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 0x7f17900c09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58648/0 (socket says 172.21.15.78:58648) 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 -- 172.21.15.78:0/3285882644 learned_addr learned my addr 172.21.15.78:0/3285882644 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f178e59c640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17900721a0 0x7f17901e79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 -- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1790150460 msgr2=0x7f17900aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f178ed9d640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1790150460 0x7f17900aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1790150460 0x7f17900aa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 -- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17900721a0 msgr2=0x7f17901e79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17900721a0 0x7f17901e79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 -- 172.21.15.78:0/3285882644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17800094d0 con 0x7f17900c7d90 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f178e59c640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17900721a0 0x7f17901e79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1787fff640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 0x7f17900c09c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f177800b4d0 tx=0x7f177800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778010870 con 0x7f17900c7d90 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17780109d0 con 0x7f17900c7d90 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17900c6880 con 0x7f17900c7d90 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.674+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778010d80 con 0x7f17900c7d90 2024-02-19T14:51:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.675+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17900c6b40 con 0x7f17900c7d90 2024-02-19T14:51:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.676+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1790070150 con 0x7f17900c7d90 2024-02-19T14:51:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.676+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f177800f3c0 con 0x7f17900c7d90 2024-02-19T14:51:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.676+0000 7f1786ffd640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f177006b810 0x7f177006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.677+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1778089eb0 con 0x7f17900c7d90 2024-02-19T14:51:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.677+0000 7f178e59c640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f177006b810 0x7f177006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.677+0000 7f178e59c640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f177006b810 0x7f177006dcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1780036000 tx=0x7f178003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.681+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1778054c20 con 0x7f17900c7d90 2024-02-19T14:51:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.853+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17900c6d60 con 0x7f17900c7d90 2024-02-19T14:51:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.853+0000 7f1786ffd640 1 -- 172.21.15.78:0/3285882644 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7f1778014090 con 0x7f17900c7d90 2024-02-19T14:51:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:51:53.856 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:51:50.115948+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:51:53.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f177006b810 msgr2=0x7f177006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f177006b810 0x7f177006dcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1780036000 tx=0x7f178003c040 comp rx=0 tx=0).stop 2024-02-19T14:51:53.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 msgr2=0x7f17900c09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 0x7f17900c09c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f177800b4d0 tx=0x7f177800b990 comp rx=0 tx=0).stop 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 shutdown_connections 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f177006b810 0x7f177006dcd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17900c7d90 0x7f17900c09c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1790150460 0x7f17900aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 --2- 172.21.15.78:0/3285882644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17900721a0 0x7f17901e79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 >> 172.21.15.78:0/3285882644 conn(0x7f179006c170 msgr2=0x7f179006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 shutdown_connections 2024-02-19T14:51:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.855+0000 7f1794b93640 1 -- 172.21.15.78:0/3285882644 wait complete. 2024-02-19T14:51:53.869 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:51:53.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:51:53.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.969+0000 7f4987055640 1 Processor -- start 2024-02-19T14:51:53.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.969+0000 7f4987055640 1 -- start start 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4987055640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49800c1880 0x7f4980152980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4987055640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f49800c1d40 0x7f498006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4987055640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498006b710 con 0x7f4980152180 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4987055640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4980072430 con 0x7f49800c1880 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4987055640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49800715e0 con 0x7f49800c1d40 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f497ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49800c1880 0x7f4980152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4984df1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4980152180 0x7f4980072830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46698/0 (socket says 172.21.15.78:46698) 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f4984df1640 1 -- 172.21.15.78:0/763272815 learned_addr learned my addr 172.21.15.78:0/763272815 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f49855f2640 1 --2- 172.21.15.78:0/763272815 >> v2:172.21.15.150:3300/0 conn(0x7f49800c1d40 0x7f498006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f497ffff640 1 -- 172.21.15.78:0/763272815 >> v2:172.21.15.150:3300/0 conn(0x7f49800c1d40 msgr2=0x7f498006e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f497ffff640 1 --2- 172.21.15.78:0/763272815 >> v2:172.21.15.150:3300/0 conn(0x7f49800c1d40 0x7f498006e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f497ffff640 1 -- 172.21.15.78:0/763272815 >> v1:172.21.15.78:6789/0 conn(0x7f4980152180 legacy=0x7f4980072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f497ffff640 1 -- 172.21.15.78:0/763272815 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4980152560 con 0x7f49800c1880 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.970+0000 7f497f7fe640 1 -- 172.21.15.78:0/763272815 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368064369 0 0) 0x7f498006b710 con 0x7f4980152180 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.971+0000 7f497ffff640 1 --2- 172.21.15.78:0/763272815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1880 0x7f4980152980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f498006d3f0 tx=0x7f497402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673dccdef7b5079b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.971+0000 7f497f7fe640 1 -- 172.21.15.78:0/763272815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497402f050 con 0x7f49800c1880 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.971+0000 7f497f7fe640 1 -- 172.21.15.78:0/763272815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4974031700 con 0x7f49800c1880 2024-02-19T14:51:53.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.971+0000 7f497f7fe640 1 -- 172.21.15.78:0/763272815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974031a90 con 0x7f49800c1880 2024-02-19T14:51:53.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 -- 172.21.15.78:0/763272815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1880 msgr2=0x7f4980152980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 --2- 172.21.15.78:0/763272815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1880 0x7f4980152980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f498006d3f0 tx=0x7f497402b7a0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 -- 172.21.15.78:0/763272815 shutdown_connections 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 --2- 172.21.15.78:0/763272815 >> v2:172.21.15.150:3300/0 conn(0x7f49800c1d40 0x7f498006e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 --2- 172.21.15.78:0/763272815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1880 0x7f4980152980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 -- 172.21.15.78:0/763272815 >> 172.21.15.78:0/763272815 conn(0x7f49800a8700 msgr2=0x7f49800a8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 -- 172.21.15.78:0/763272815 shutdown_connections 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.972+0000 7f4987055640 1 -- 172.21.15.78:0/763272815 wait complete. 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.973+0000 7f4987055640 1 Processor -- start 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.973+0000 7f4987055640 1 -- start start 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4987055640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49800c1880 0x7f49801ec320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4987055640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 0x7f4980149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4987055640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4980152180 0x7f49801e8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4987055640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49800c5fa0 con 0x7f49800c1d40 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4987055640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49801e8380 con 0x7f4980152180 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4987055640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49801ec860 con 0x7f49800c1880 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 0x7f4980149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 0x7f4980149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58662/0 (socket says 172.21.15.78:58662) 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 -- 172.21.15.78:0/3394352157 learned_addr learned my addr 172.21.15.78:0/3394352157 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f49855f2640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4980152180 0x7f49801e8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f4984df1640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49800c1880 0x7f49801ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 -- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49800c1880 msgr2=0x7f49801ec320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49800c1880 0x7f49801ec320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 -- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4980152180 msgr2=0x7f49801e8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4980152180 0x7f49801e8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 -- 172.21.15.78:0/3394352157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974009400 con 0x7f49800c1d40 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.974+0000 7f497ffff640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 0x7f4980149c20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4974036040 tx=0x7f49740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.975+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974033c30 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.975+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4974033db0 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.975+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49800c5700 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.975+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497403c640 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.975+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49800c5a90 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.976+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49800c2a30 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.976+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f497403a040 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.976+0000 7f497d7fa640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f496c06b810 0x7f496c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.976+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f49740ab3c0 con 0x7f49800c1d40 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.976+0000 7f4984df1640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f496c06b810 0x7f496c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:53.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.977+0000 7f4984df1640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f496c06b810 0x7f496c06dcd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4970000f60 tx=0x7f4970007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:53.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:53.981+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4974076130 con 0x7f49800c1d40 2024-02-19T14:51:54.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.153+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f498006bc00 con 0x7f49800c1d40 2024-02-19T14:51:54.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.153+0000 7f497d7fa640 1 -- 172.21.15.78:0/3394352157 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49740380a0 con 0x7f49800c1d40 2024-02-19T14:51:54.156 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f496c06b810 msgr2=0x7f496c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f496c06b810 0x7f496c06dcd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4970000f60 tx=0x7f4970007b50 comp rx=0 tx=0).stop 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 msgr2=0x7f4980149c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 0x7f4980149c20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4974036040 tx=0x7f49740046c0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 shutdown_connections 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f496c06b810 0x7f496c06dcd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49800c1d40 0x7f4980149c20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4980152180 0x7f49801e8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 --2- 172.21.15.78:0/3394352157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49800c1880 0x7f49801ec320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 >> 172.21.15.78:0/3394352157 conn(0x7f49800a8700 msgr2=0x7f49801534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:54.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 shutdown_connections 2024-02-19T14:51:54.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.155+0000 7f4987055640 1 -- 172.21.15.78:0/3394352157 wait complete. 2024-02-19T14:51:54.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.269+0000 7f038787b640 1 Processor -- start 2024-02-19T14:51:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f038787b640 1 -- start start 2024-02-19T14:51:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f038787b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f03800c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f038787b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03800c3d90 0x7f03800c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f038787b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038006ff90 con 0x7f038014d210 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f038787b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0380070270 con 0x7f03800c3d90 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f038787b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03800c3970 con 0x7f03800719e0 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f0385617640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f038014d210 0x7f03800c4290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46706/0 (socket says 172.21.15.78:46706) 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.270+0000 7f0385617640 1 -- 172.21.15.78:0/2997589926 learned_addr learned my addr 172.21.15.78:0/2997589926 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303293626 0 0) 0x7f038006ff90 con 0x7f038014d210 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f036c003520 con 0x7f038014d210 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0385e18640 1 --2- 172.21.15.78:0/2997589926 >> v2:172.21.15.150:3300/0 conn(0x7f03800c3d90 0x7f03800c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f03800c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457636893 0 0) 0x7f036c003520 con 0x7f038014d210 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 msgr2=0x7f03800c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f03800c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 >> v2:172.21.15.150:3300/0 conn(0x7f03800c3d90 msgr2=0x7f03800c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 --2- 172.21.15.78:0/2997589926 >> v2:172.21.15.150:3300/0 conn(0x7f03800c3d90 0x7f03800c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380156fc0 con 0x7f038014d210 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f037c0019f0 con 0x7f038014d210 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 >> v1:172.21.15.78:6789/0 conn(0x7f038014d210 legacy=0x7f03800c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f036c003970 0x7f036c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f036c0047c0 0x7f036c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f036c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0380070270 con 0x7f036c0047c0 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03800c3970 con 0x7f036c003970 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c00bae0 con 0x7f03800719e0 2024-02-19T14:51:54.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f036c0047c0 0x7f036c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0385617640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f036c003970 0x7f036c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0385e18640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f036c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 -- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 msgr2=0x7f036c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f036c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 -- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f036c003970 msgr2=0x7f036c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f036c003970 0x7f036c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0384e16640 1 -- 172.21.15.78:0/2997589926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c003520 con 0x7f036c0047c0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0385617640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f036c003970 0x7f036c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.271+0000 7f0385e18640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03800719e0 0x7f036c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.272+0000 7f0384e16640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f036c0047c0 0x7f036c004c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0380071dc0 tx=0x7f036c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6bc8052a85c091 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.272+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0370033670 con 0x7f036c0047c0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.272+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03700337d0 con 0x7f036c0047c0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.272+0000 7f0377fff640 1 -- 172.21.15.78:0/2997589926 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0370033b60 con 0x7f036c0047c0 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f038787b640 1 -- 172.21.15.78:0/2997589926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f036c0047c0 msgr2=0x7f036c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f038787b640 1 --2- 172.21.15.78:0/2997589926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f036c0047c0 0x7f036c004c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0380071dc0 tx=0x7f036c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f0385e18640 1 -- 172.21.15.78:0/2997589926 reap_dead start 2024-02-19T14:51:54.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f038787b640 1 -- 172.21.15.78:0/2997589926 shutdown_connections 2024-02-19T14:51:54.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f038787b640 1 -- 172.21.15.78:0/2997589926 >> 172.21.15.78:0/2997589926 conn(0x7f03800c5050 msgr2=0x7f03800c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:54.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f038787b640 1 -- 172.21.15.78:0/2997589926 shutdown_connections 2024-02-19T14:51:54.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.273+0000 7f038787b640 1 -- 172.21.15.78:0/2997589926 wait complete. 2024-02-19T14:51:54.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 Processor -- start 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 -- start start 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036c0047c0 0x7f03800c0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 0x7f03800be6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03800bebf0 0x7f03801ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03800bdf80 con 0x7f038014d630 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03800bfa90 con 0x7f03800bebf0 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.274+0000 7f038787b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03800bfd30 con 0x7f036c0047c0 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 0x7f03800be6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 0x7f03800be6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58680/0 (socket says 172.21.15.78:58680) 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 -- 172.21.15.78:0/3338432026 learned_addr learned my addr 172.21.15.78:0/3338432026 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 -- 172.21.15.78:0/3338432026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036c0047c0 msgr2=0x7f03800c0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036c0047c0 0x7f03800c0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 -- 172.21.15.78:0/3338432026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03800bebf0 msgr2=0x7f03801ebaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03800bebf0 0x7f03801ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 -- 172.21.15.78:0/3338432026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03800c0010 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0384e16640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 0x7f03800be6b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f037000baf0 tx=0x7f0370004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0370046600 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0370040070 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03800be190 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03700469a0 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.275+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038014e8b0 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.276+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03800bfe70 con 0x7f038014d630 2024-02-19T14:51:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.277+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0370046b00 con 0x7f038014d630 2024-02-19T14:51:54.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.277+0000 7f0375ffb640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f036406b810 0x7f036406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:54.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.277+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f037002d040 con 0x7f038014d630 2024-02-19T14:51:54.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.277+0000 7f0385617640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f036406b810 0x7f036406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:54.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.277+0000 7f0385617640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f036406b810 0x7f036406dcd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f037c002230 tx=0x7f037c001e90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.281+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f03800bfe70 con 0x7f038014d630 2024-02-19T14:51:54.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.489+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03800bfe70 con 0x7f038014d630 2024-02-19T14:51:54.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.489+0000 7f0375ffb640 1 -- 172.21.15.78:0/3338432026 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f03800bfe70 con 0x7f038014d630 2024-02-19T14:51:54.492 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:51:54.492 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-02-19T14:51:54.492 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f036406b810 msgr2=0x7f036406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f036406b810 0x7f036406dcd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f037c002230 tx=0x7f037c001e90 comp rx=0 tx=0).stop 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 msgr2=0x7f03800be6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 0x7f03800be6b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f037000baf0 tx=0x7f0370004730 comp rx=0 tx=0).stop 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 shutdown_connections 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f036406b810 0x7f036406dcd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f038014d630 0x7f03800be6b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03800bebf0 0x7f03801ebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 --2- 172.21.15.78:0/3338432026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036c0047c0 0x7f03800c0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:54.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 >> 172.21.15.78:0/3338432026 conn(0x7f03800c5050 msgr2=0x7f03801552c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:54.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 shutdown_connections 2024-02-19T14:51:54.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:54.491+0000 7f038787b640 1 -- 172.21.15.78:0/3338432026 wait complete. 2024-02-19T14:51:55.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:51:55.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:51:56.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.619+0000 7fb81627b640 1 Processor -- start 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.619+0000 7fb81627b640 1 -- start start 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb81627b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb810152480 0x7fb8101559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb81627b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8100c87a0 0x7fb8100c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb81627b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb810155ef0 con 0x7fb8100c7070 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb81627b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81006dec0 con 0x7fb810152480 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb81627b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81006d520 con 0x7fb8100c87a0 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb80ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb810152480 0x7fb8101559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb80ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb810152480 0x7fb8101559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58684/0 (socket says 172.21.15.78:58684) 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.620+0000 7fb80ffff640 1 -- 172.21.15.78:0/1032193519 learned_addr learned my addr 172.21.15.78:0/1032193519 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:56.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.621+0000 7fb814818640 1 --2- 172.21.15.78:0/1032193519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8100c87a0 0x7fb8100c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.621+0000 7fb80ffff640 1 -- 172.21.15.78:0/1032193519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8100c87a0 msgr2=0x7fb8100c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.621+0000 7fb80ffff640 1 --2- 172.21.15.78:0/1032193519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8100c87a0 0x7fb8100c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.621+0000 7fb80ffff640 1 -- 172.21.15.78:0/1032193519 >> v1:172.21.15.150:6789/0 conn(0x7fb8100c7070 legacy=0x7fb81006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.621+0000 7fb80ffff640 1 -- 172.21.15.78:0/1032193519 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100c75c0 con 0x7fb810152480 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.621+0000 7fb80ffff640 1 --2- 172.21.15.78:0/1032193519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb810152480 0x7fb8101559b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8100aa1b0 tx=0x7fb80002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7582c3b9e58e5c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb80effd640 1 -- 172.21.15.78:0/1032193519 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800031640 con 0x7fb810152480 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb80effd640 1 -- 172.21.15.78:0/1032193519 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8000317a0 con 0x7fb810152480 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb80effd640 1 -- 172.21.15.78:0/1032193519 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800031b30 con 0x7fb810152480 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 -- 172.21.15.78:0/1032193519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb810152480 msgr2=0x7fb8101559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 --2- 172.21.15.78:0/1032193519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb810152480 0x7fb8101559b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8100aa1b0 tx=0x7fb80002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 -- 172.21.15.78:0/1032193519 shutdown_connections 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 --2- 172.21.15.78:0/1032193519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8100c87a0 0x7fb8100c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 --2- 172.21.15.78:0/1032193519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb810152480 0x7fb8101559b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 -- 172.21.15.78:0/1032193519 >> 172.21.15.78:0/1032193519 conn(0x7fb8100ca8a0 msgr2=0x7fb8100cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:56.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 -- 172.21.15.78:0/1032193519 shutdown_connections 2024-02-19T14:51:56.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.622+0000 7fb81627b640 1 -- 172.21.15.78:0/1032193519 wait complete. 2024-02-19T14:51:56.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.623+0000 7fb81627b640 1 Processor -- start 2024-02-19T14:51:56.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 -- start start 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8100c7070 0x7fb8100b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 0x7fb8100acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb810152480 0x7fb8100abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100afc40 con 0x7fb8100c87a0 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb810147a10 con 0x7fb8100c7070 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb81627b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100ad540 con 0x7fb810152480 2024-02-19T14:51:56.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 0x7fb8100acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 0x7fb8100acd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58698/0 (socket says 172.21.15.78:58698) 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb80f7fe640 1 -- 172.21.15.78:0/3704864166 learned_addr learned my addr 172.21.15.78:0/3704864166 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb80ffff640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8100c7070 0x7fb8100b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.624+0000 7fb814818640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb810152480 0x7fb8100abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80f7fe640 1 -- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb810152480 msgr2=0x7fb8100abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80f7fe640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb810152480 0x7fb8100abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80f7fe640 1 -- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8100c7070 msgr2=0x7fb8100b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80f7fe640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8100c7070 0x7fb8100b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80f7fe640 1 -- 172.21.15.78:0/3704864166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8000094d0 con 0x7fb8100c87a0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80f7fe640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 0x7fb8100acd80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c400 tx=0x7fb7f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8011790 con 0x7fb8100c87a0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8011930 con 0x7fb8100c87a0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8100ad820 con 0x7fb8100c87a0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8011ce0 con 0x7fb8100c87a0 2024-02-19T14:51:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.625+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8101e13b0 con 0x7fb8100c87a0 2024-02-19T14:51:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.626+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8100add40 con 0x7fb8100c87a0 2024-02-19T14:51:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.627+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb7f801c460 con 0x7fb8100c87a0 2024-02-19T14:51:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.627+0000 7fb80cff9640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7f006b810 0x7fb7f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.627+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb7f8089c10 con 0x7fb8100c87a0 2024-02-19T14:51:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.627+0000 7fb80ffff640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7f006b810 0x7fb7f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.628+0000 7fb80ffff640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7f006b810 0x7fb7f006dcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb800036000 tx=0x7fb80003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:56.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.632+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb7f8054ac0 con 0x7fb8100c87a0 2024-02-19T14:51:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.842+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb810074990 con 0x7fb8100c87a0 2024-02-19T14:51:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.843+0000 7fb80cff9640 1 -- 172.21.15.78:0/3704864166 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7fb7f8058180 con 0x7fb8100c87a0 2024-02-19T14:51:56.845 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:51:56.845 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-19T14:51:56.846 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2024-02-19T14:51:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7f006b810 msgr2=0x7fb7f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7f006b810 0x7fb7f006dcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb800036000 tx=0x7fb80003c040 comp rx=0 tx=0).stop 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 msgr2=0x7fb8100acd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 0x7fb8100acd80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c400 tx=0x7fb7f800c8c0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 shutdown_connections 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7f006b810 0x7fb7f006dcd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8100c87a0 0x7fb8100acd80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb810152480 0x7fb8100abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 --2- 172.21.15.78:0/3704864166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8100c7070 0x7fb8100b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 >> 172.21.15.78:0/3704864166 conn(0x7fb8100ca8a0 msgr2=0x7fb81006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 shutdown_connections 2024-02-19T14:51:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.845+0000 7fb81627b640 1 -- 172.21.15.78:0/3704864166 wait complete. 2024-02-19T14:51:56.858 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:51:56.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.956+0000 7f9fd946b640 1 Processor -- start 2024-02-19T14:51:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.956+0000 7f9fd946b640 1 -- start start 2024-02-19T14:51:56.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd946b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 0x7f9fd40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd946b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd41551f0 con 0x7f9fd40c8e00 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd946b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd4155490 con 0x7f9fd40c5730 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd946b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd4155770 con 0x7f9fd4152700 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9fd40c8e00 0x7f9fd415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46716/0 (socket says 172.21.15.78:46716) 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd37fe640 1 -- 172.21.15.78:0/2076949337 learned_addr learned my addr 172.21.15.78:0/2076949337 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938168215 0 0) 0x7f9fd41551f0 con 0x7f9fd40c8e00 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc0003520 con 0x7f9fd40c8e00 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd2ffd640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 0x7f9fd40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2157781045 0 0) 0x7f9fc0003520 con 0x7f9fd40c8e00 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 msgr2=0x7f9fd40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 0x7f9fd40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 >> v1:172.21.15.150:6789/0 conn(0x7f9fd40c5730 legacy=0x7f9fd406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4155a50 con 0x7f9fd40c8e00 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9fcc0019f0 con 0x7f9fd40c8e00 2024-02-19T14:51:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 >> v1:172.21.15.78:6789/0 conn(0x7f9fd40c8e00 legacy=0x7f9fd415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fc00039a0 0x7f9fc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 0x7f9fc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd4155770 con 0x7f9fc0004d00 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc000ba30 con 0x7f9fc00039a0 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc000bbf0 con 0x7f9fd4152700 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.957+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fca7fc640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 0x7f9fc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd2ffd640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fc00039a0 0x7f9fc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd37fe640 1 -- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 msgr2=0x7f9fc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd4152700 0x7f9fc00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd37fe640 1 -- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fc00039a0 msgr2=0x7f9fc0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fc00039a0 0x7f9fc0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd37fe640 1 -- 172.21.15.78:0/2076949337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc0003520 con 0x7f9fc0004d00 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0013d0 tx=0x7f9fc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c9de147b006d6a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc034630 con 0x7f9fc0004d00 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc02f070 con 0x7f9fc0004d00 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.958+0000 7f9fd27fc640 1 -- 172.21.15.78:0/2076949337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc0349f0 con 0x7f9fc0004d00 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd946b640 1 -- 172.21.15.78:0/2076949337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc0004d00 msgr2=0x7f9fc0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd946b640 1 --2- 172.21.15.78:0/2076949337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc0004d00 0x7f9fc0005180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0013d0 tx=0x7f9fc00057f0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd37fe640 1 -- 172.21.15.78:0/2076949337 reap_dead start 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd946b640 1 -- 172.21.15.78:0/2076949337 shutdown_connections 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd946b640 1 -- 172.21.15.78:0/2076949337 >> 172.21.15.78:0/2076949337 conn(0x7f9fd40cac90 msgr2=0x7f9fd40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd946b640 1 -- 172.21.15.78:0/2076949337 shutdown_connections 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.959+0000 7f9fd946b640 1 -- 172.21.15.78:0/2076949337 wait complete. 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.960+0000 7f9fd946b640 1 Processor -- start 2024-02-19T14:51:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.960+0000 7f9fd946b640 1 -- start start 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd946b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd40c3f80 0x7f9fd40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd946b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd40ace50 0x7f9fd40c1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd946b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 0x7f9fd40aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd946b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40beba0 con 0x7f9fd40c4360 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd946b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40c2840 con 0x7f9fd40ace50 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd946b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd4145c10 con 0x7f9fd40c3f80 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 0x7f9fd40aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 0x7f9fd40aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58712/0 (socket says 172.21.15.78:58712) 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 -- 172.21.15.78:0/978039220 learned_addr learned my addr 172.21.15.78:0/978039220 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fca7fc640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd40ace50 0x7f9fd40c1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd2ffd640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd40c3f80 0x7f9fd40c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 -- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd40c3f80 msgr2=0x7f9fd40c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd40c3f80 0x7f9fd40c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 -- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd40ace50 msgr2=0x7f9fd40c1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd40ace50 0x7f9fd40c1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 -- 172.21.15.78:0/978039220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4145ff0 con 0x7f9fd40c4360 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.961+0000 7f9fd37fe640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 0x7f9fd40aa1d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc001cb0 tx=0x7f9fd406d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.962+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc034840 con 0x7f9fd40c4360 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.962+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc0349a0 con 0x7f9fd40c4360 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.962+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd40c2560 con 0x7f9fd40c4360 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.962+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc034d10 con 0x7f9fd40c4360 2024-02-19T14:51:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.962+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd41498c0 con 0x7f9fd40c4360 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.962+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd40c23c0 con 0x7f9fd40c4360 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.963+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9fcc031030 con 0x7f9fd40c4360 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.963+0000 7f9fcbfff640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fb406b840 0x7f9fb406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.963+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9fcc02ce40 con 0x7f9fd40c4360 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.963+0000 7f9fd2ffd640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fb406b840 0x7f9fb406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.964+0000 7f9fd2ffd640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fb406b840 0x7f9fb406dd00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40acbc0 tx=0x7f9fc4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:56.968+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9fd40c23c0 con 0x7f9fd40c4360 2024-02-19T14:51:57.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.141+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fd41ebcf0 con 0x7f9fd40c4360 2024-02-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.142+0000 7f9fcbfff640 1 -- 172.21.15.78:0/978039220 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7f9fd41ebcf0 con 0x7f9fd40c4360 2024-02-19T14:51:57.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:51:57.145 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:51:50.115948+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fb406b840 msgr2=0x7f9fb406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fb406b840 0x7f9fb406dd00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40acbc0 tx=0x7f9fc4008040 comp rx=0 tx=0).stop 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 msgr2=0x7f9fd40aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 0x7f9fd40aa1d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc001cb0 tx=0x7f9fd406d6b0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 shutdown_connections 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fb406b840 0x7f9fb406dd00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fd40c4360 0x7f9fd40aa1d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd40ace50 0x7f9fd40c1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 --2- 172.21.15.78:0/978039220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fd40c3f80 0x7f9fd40c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 >> 172.21.15.78:0/978039220 conn(0x7f9fd40cac90 msgr2=0x7f9fd40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 shutdown_connections 2024-02-19T14:51:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.144+0000 7f9fd946b640 1 -- 172.21.15.78:0/978039220 wait complete. 2024-02-19T14:51:57.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:51:57.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:51:57.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.258+0000 7fcdc02da640 1 Processor -- start 2024-02-19T14:51:57.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.258+0000 7fcdc02da640 1 -- start start 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdc02da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 0x7fcdb8071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdc02da640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdb80c8b60 0x7fcdb81536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdc02da640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb806fe40 con 0x7fcdb80c29e0 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdc02da640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb80c32c0 con 0x7fcdb80c8b60 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdc02da640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8070500 con 0x7fcdb80713a0 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd875640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcdb80c29e0 0x7fcdb80c64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46718/0 (socket says 172.21.15.78:46718) 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd875640 1 -- 172.21.15.78:0/2550638758 learned_addr learned my addr 172.21.15.78:0/2550638758 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:51:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbe877640 1 --2- 172.21.15.78:0/2550638758 >> v2:172.21.15.150:3300/0 conn(0x7fcdb80c8b60 0x7fcdb81536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383402563 0 0) 0x7fcdb806fe40 con 0x7fcdb80c29e0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda4003520 con 0x7fcdb80c29e0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbe076640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 0x7fcdb8071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 672176780 0 0) 0x7fcda4003520 con 0x7fcdb80c29e0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 msgr2=0x7fcdb8071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 0x7fcdb8071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 >> v2:172.21.15.150:3300/0 conn(0x7fcdb80c8b60 msgr2=0x7fcdb81536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 --2- 172.21.15.78:0/2550638758 >> v2:172.21.15.150:3300/0 conn(0x7fcdb80c8b60 0x7fcdb81536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8153bf0 con 0x7fcdb80c29e0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fcda8001670 con 0x7fcdb80c29e0 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 >> v1:172.21.15.78:6789/0 conn(0x7fcdb80c29e0 legacy=0x7fcdb80c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda4003970 0x7fcda4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 0x7fcda40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.259+0000 7fcdbd074640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda4004d00 0x7fcda4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda400b980 con 0x7fcda4003970 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda400bac0 con 0x7fcda4004d00 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda400bc00 con 0x7fcdb80713a0 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe076640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda4003970 0x7fcda4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe877640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda4004d00 0x7fcda4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbd875640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 0x7fcda40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe076640 1 -- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 msgr2=0x7fcda40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe076640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdb80713a0 0x7fcda40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe076640 1 -- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda4004d00 msgr2=0x7fcda4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe076640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda4004d00 0x7fcda4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.260+0000 7fcdbe076640 1 -- 172.21.15.78:0/2550638758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4003520 con 0x7fcda4003970 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdbe076640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda4003970 0x7fcda4003d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009ac0 tx=0x7fcda40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc938639a0579b74 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb402c6b0 con 0x7fcda4003970 2024-02-19T14:51:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb402c810 con 0x7fcda4003970 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdbd074640 1 -- 172.21.15.78:0/2550638758 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb402cba0 con 0x7fcda4003970 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdc02da640 1 -- 172.21.15.78:0/2550638758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda4003970 msgr2=0x7fcda4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdc02da640 1 --2- 172.21.15.78:0/2550638758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcda4003970 0x7fcda4003d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009ac0 tx=0x7fcda40057f0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdbe877640 1 -- 172.21.15.78:0/2550638758 reap_dead start 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdc02da640 1 -- 172.21.15.78:0/2550638758 shutdown_connections 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdc02da640 1 -- 172.21.15.78:0/2550638758 >> 172.21.15.78:0/2550638758 conn(0x7fcdb80aa300 msgr2=0x7fcdb80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdc02da640 1 -- 172.21.15.78:0/2550638758 shutdown_connections 2024-02-19T14:51:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.261+0000 7fcdc02da640 1 -- 172.21.15.78:0/2550638758 wait complete. 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.262+0000 7fcdc02da640 1 Processor -- start 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.262+0000 7fcdc02da640 1 -- start start 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdc02da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcda4003970 0x7fcdb80c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdc02da640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 0x7fcdb814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdc02da640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdb80a7740 0x7fcdb814f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdc02da640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb81ee3d0 con 0x7fcdb8155150 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdc02da640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb806dcb0 con 0x7fcdb80a7740 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdc02da640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb806d980 con 0x7fcda4003970 2024-02-19T14:51:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 0x7fcdb814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 0x7fcdb814ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58736/0 (socket says 172.21.15.78:58736) 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 -- 172.21.15.78:0/3587440078 learned_addr learned my addr 172.21.15.78:0/3587440078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbe877640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdb80a7740 0x7fcdb814f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 -- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcda4003970 msgr2=0x7fcdb80c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbe076640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcda4003970 0x7fcdb80c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcda4003970 0x7fcdb80c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 -- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdb80a7740 msgr2=0x7fcdb814f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdb80a7740 0x7fcdb814f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 -- 172.21.15.78:0/3587440078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb81ede10 con 0x7fcdb8155150 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbe877640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdb80a7740 0x7fcdb814f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.263+0000 7fcdbd875640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 0x7fcdb814ed00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcda800e820 tx=0x7fcda800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.264+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda800eed0 con 0x7fcdb8155150 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.264+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda800c870 con 0x7fcdb8155150 2024-02-19T14:51:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.264+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb81ee0f0 con 0x7fcdb8155150 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.264+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda800cc00 con 0x7fcdb8155150 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.264+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb8099a60 con 0x7fcdb8155150 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.265+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb806d7e0 con 0x7fcdb8155150 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.265+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcda801d460 con 0x7fcdb8155150 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.265+0000 7fcdaeffd640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd9c06b810 0x7fcd9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.266+0000 7fcdbe076640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd9c06b810 0x7fcd9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.266+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcda8089df0 con 0x7fcdb8155150 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.266+0000 7fcdbe076640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd9c06b810 0x7fcd9c06dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4036000 tx=0x7fcdb4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.270+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcdb806d7e0 con 0x7fcdb8155150 2024-02-19T14:51:57.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.441+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcdb806d7e0 con 0x7fcdb8155150 2024-02-19T14:51:57.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.441+0000 7fcdaeffd640 1 -- 172.21.15.78:0/3587440078 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcdb806d7e0 con 0x7fcdb8155150 2024-02-19T14:51:57.444 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:51:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd9c06b810 msgr2=0x7fcd9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd9c06b810 0x7fcd9c06dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4036000 tx=0x7fcdb4030000 comp rx=0 tx=0).stop 2024-02-19T14:51:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 msgr2=0x7fcdb814ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 0x7fcdb814ed00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcda800e820 tx=0x7fcda800ece0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 shutdown_connections 2024-02-19T14:51:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd9c06b810 0x7fcd9c06dcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdb8155150 0x7fcdb814ed00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdb80a7740 0x7fcdb814f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 --2- 172.21.15.78:0/3587440078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcda4003970 0x7fcdb80c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 >> 172.21.15.78:0/3587440078 conn(0x7fcdb80aa300 msgr2=0x7fcdb80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 shutdown_connections 2024-02-19T14:51:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.443+0000 7fcdc02da640 1 -- 172.21.15.78:0/3587440078 wait complete. 2024-02-19T14:51:57.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:51:57.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.554+0000 7f9b97f0d640 1 Processor -- start 2024-02-19T14:51:57.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.554+0000 7f9b97f0d640 1 -- start start 2024-02-19T14:51:57.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b97f0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9b900c8500 0x7f9b900c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b97f0d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b900c34a0 con 0x7f9b900c2f50 2024-02-19T14:51:57.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b97f0d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9006c8e0 con 0x7f9b900c42e0 2024-02-19T14:51:57.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b97f0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9006c090 con 0x7f9b900c8500 2024-02-19T14:51:57.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b95ca9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9b900c8500 0x7f9b900c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b964aa640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9b900c2f50 0x7f9b9015f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49758/0 (socket says 172.21.15.78:49758) 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b964aa640 1 -- 172.21.15.78:0/3342329240 learned_addr learned my addr 172.21.15.78:0/3342329240 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b95ca9640 1 -- 172.21.15.78:0/3342329240 >> v1:172.21.15.150:6789/0 conn(0x7f9b900c42e0 legacy=0x7f9b90158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b95ca9640 1 -- 172.21.15.78:0/3342329240 >> v1:172.21.15.78:6789/0 conn(0x7f9b900c2f50 legacy=0x7f9b9015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.555+0000 7f9b95ca9640 1 -- 172.21.15.78:0/3342329240 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9006cbc0 con 0x7f9b900c8500 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.556+0000 7f9b95ca9640 1 --2- 172.21.15.78:0/3342329240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b900c88e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097a0 tx=0x7f9b8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de794c346128258 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.556+0000 7f9b94ca7640 1 -- 172.21.15.78:0/3342329240 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c02e070 con 0x7f9b900c8500 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.556+0000 7f9b94ca7640 1 -- 172.21.15.78:0/3342329240 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c0315a0 con 0x7f9b900c8500 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.556+0000 7f9b94ca7640 1 -- 172.21.15.78:0/3342329240 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c031930 con 0x7f9b900c8500 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 -- 172.21.15.78:0/3342329240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 msgr2=0x7f9b900c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/3342329240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b900c88e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0097a0 tx=0x7f9b8c02b840 comp rx=0 tx=0).stop 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 -- 172.21.15.78:0/3342329240 shutdown_connections 2024-02-19T14:51:57.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/3342329240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b900c88e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 -- 172.21.15.78:0/3342329240 >> 172.21.15.78:0/3342329240 conn(0x7f9b900aa300 msgr2=0x7f9b900aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 -- 172.21.15.78:0/3342329240 shutdown_connections 2024-02-19T14:51:57.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.557+0000 7f9b97f0d640 1 -- 172.21.15.78:0/3342329240 wait complete. 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.558+0000 7f9b97f0d640 1 Processor -- start 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.558+0000 7f9b97f0d640 1 -- start start 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.558+0000 7f9b97f0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b900c2f50 0x7f9b901e20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.558+0000 7f9b97f0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b90159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.558+0000 7f9b97f0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9006f3f0 0x7f9b901568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b97f0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90070c60 con 0x7f9b900c8500 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b97f0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90157200 con 0x7f9b9006f3f0 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b97f0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90071430 con 0x7f9b900c2f50 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b90159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b90159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49650/0 (socket says 172.21.15.78:49650) 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 -- 172.21.15.78:0/1401417441 learned_addr learned my addr 172.21.15.78:0/1401417441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:57.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b964aa640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9006f3f0 0x7f9b901568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 -- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b900c2f50 msgr2=0x7f9b901e20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b900c2f50 0x7f9b901e20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 -- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9006f3f0 msgr2=0x7f9b901568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9006f3f0 0x7f9b901568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 -- 172.21.15.78:0/1401417441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c009400 con 0x7f9b900c8500 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b954a8640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b90159120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000c430 tx=0x7f9b8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011790 con 0x7f9b900c8500 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b800118f0 con 0x7f9b900c8500 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b90070e70 con 0x7f9b900c8500 2024-02-19T14:51:57.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011ce0 con 0x7f9b900c8500 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.559+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b901eae00 con 0x7f9b900c8500 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.560+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b90071c20 con 0x7f9b900c8500 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.561+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9b8001c460 con 0x7f9b900c8500 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.561+0000 7f9b867fc640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b7406b810 0x7f9b7406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.561+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9b80089e50 con 0x7f9b900c8500 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.561+0000 7f9b95ca9640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b7406b810 0x7f9b7406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.561+0000 7f9b95ca9640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b7406b810 0x7f9b7406dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c036000 tx=0x7f9b8c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.565+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9b80054bd0 con 0x7f9b900c8500 2024-02-19T14:51:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.739+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9b90074990 con 0x7f9b900c8500 2024-02-19T14:51:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.739+0000 7f9b867fc640 1 -- 172.21.15.78:0/1401417441 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9b80058290 con 0x7f9b900c8500 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:51:57.742 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b7406b810 msgr2=0x7f9b7406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b7406b810 0x7f9b7406dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c036000 tx=0x7f9b8c030000 comp rx=0 tx=0).stop 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 msgr2=0x7f9b90159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b90159120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000c430 tx=0x7f9b8000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 shutdown_connections 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b7406b810 0x7f9b7406dcd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b900c8500 0x7f9b90159120 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9006f3f0 0x7f9b901568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 --2- 172.21.15.78:0/1401417441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b900c2f50 0x7f9b901e20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 >> 172.21.15.78:0/1401417441 conn(0x7f9b900aa300 msgr2=0x7f9b900aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 shutdown_connections 2024-02-19T14:51:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.741+0000 7f9b97f0d640 1 -- 172.21.15.78:0/1401417441 wait complete. 2024-02-19T14:51:57.754 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:51:57.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:51:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 Processor -- start 2024-02-19T14:51:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 -- start start 2024-02-19T14:51:57.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b141503a0 0x7f8b140c5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b14156350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8b140719d0 0x7f8b14156890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b14156dd0 con 0x7f8b141503a0 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140c65a0 con 0x7f8b140719d0 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b19ca4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140a8100 con 0x7f8b140c2ba0 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.855+0000 7f8b137fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b141503a0 0x7f8b140c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8b141503a0 0x7f8b140c5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49660/0 (socket says 172.21.15.78:49660) 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 -- 172.21.15.78:0/3401592985 learned_addr learned my addr 172.21.15.78:0/3401592985 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b0bfff640 1 --2- 172.21.15.78:0/3401592985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b14156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b13fff640 1 --2- 172.21.15.78:0/3401592985 >> v2:172.21.15.150:3300/0 conn(0x7f8b140719d0 0x7f8b14156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 -- 172.21.15.78:0/3401592985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 msgr2=0x7f8b14156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 --2- 172.21.15.78:0/3401592985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b14156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 -- 172.21.15.78:0/3401592985 >> v2:172.21.15.150:3300/0 conn(0x7f8b140719d0 msgr2=0x7f8b14156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:57.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 --2- 172.21.15.78:0/3401592985 >> v2:172.21.15.150:3300/0 conn(0x7f8b140719d0 0x7f8b14156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 -- 172.21.15.78:0/3401592985 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b14150780 con 0x7f8b141503a0 2024-02-19T14:51:57.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b13fff640 1 --2- 172.21.15.78:0/3401592985 >> v2:172.21.15.150:3300/0 conn(0x7f8b140719d0 0x7f8b14156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:57.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.856+0000 7f8b137fe640 1 --2- 172.21.15.78:0/3401592985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b141503a0 0x7f8b140c5130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b14150970 tx=0x7f8b0402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c441d7816c52a816 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b12ffd640 1 -- 172.21.15.78:0/3401592985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04031740 con 0x7f8b141503a0 2024-02-19T14:51:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b12ffd640 1 -- 172.21.15.78:0/3401592985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b040318a0 con 0x7f8b141503a0 2024-02-19T14:51:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b12ffd640 1 -- 172.21.15.78:0/3401592985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04031c30 con 0x7f8b141503a0 2024-02-19T14:51:57.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 -- 172.21.15.78:0/3401592985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b141503a0 msgr2=0x7f8b140c5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/3401592985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b141503a0 0x7f8b140c5130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b14150970 tx=0x7f8b0402b7e0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 -- 172.21.15.78:0/3401592985 shutdown_connections 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/3401592985 >> v2:172.21.15.150:3300/0 conn(0x7f8b140719d0 0x7f8b14156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/3401592985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b14156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/3401592985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b141503a0 0x7f8b140c5130 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 -- 172.21.15.78:0/3401592985 >> 172.21.15.78:0/3401592985 conn(0x7f8b140c86a0 msgr2=0x7f8b140c8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 -- 172.21.15.78:0/3401592985 shutdown_connections 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.857+0000 7f8b19ca4640 1 -- 172.21.15.78:0/3401592985 wait complete. 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.858+0000 7f8b19ca4640 1 Processor -- start 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.858+0000 7f8b19ca4640 1 -- start start 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b19ca4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 0x7f8b141dcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b19ca4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b140c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b19ca4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b141503a0 0x7f8b141eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b19ca4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b14151c20 con 0x7f8b140719d0 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b19ca4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b141dcb90 con 0x7f8b141503a0 2024-02-19T14:51:57.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b19ca4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b141eaa20 con 0x7f8b140c2ba0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 0x7f8b141dcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 0x7f8b141dcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49670/0 (socket says 172.21.15.78:49670) 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 -- 172.21.15.78:0/2973982542 learned_addr learned my addr 172.21.15.78:0/2973982542 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b0bfff640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b140c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b13fff640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b141503a0 0x7f8b141eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 -- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 msgr2=0x7f8b140c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b140c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 -- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b141503a0 msgr2=0x7f8b141eaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b141503a0 0x7f8b141eaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.859+0000 7f8b137fe640 1 -- 172.21.15.78:0/2973982542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b040094d0 con 0x7f8b140719d0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.860+0000 7f8b137fe640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 0x7f8b141dcdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b0402b820 tx=0x7f8b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.860+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b040339c0 con 0x7f8b140719d0 2024-02-19T14:51:57.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.860+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04033b20 con 0x7f8b140719d0 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.860+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b141dc5d0 con 0x7f8b140719d0 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.860+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0403c400 con 0x7f8b140719d0 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.860+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1406da90 con 0x7f8b140719d0 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.861+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b14150970 con 0x7f8b140719d0 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.861+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8b0403a040 con 0x7f8b140719d0 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.861+0000 7f8b10ff9640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8af406b810 0x7f8af406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:57.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.862+0000 7f8b0bfff640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8af406b810 0x7f8af406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.862+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b040ab730 con 0x7f8b140719d0 2024-02-19T14:51:57.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.862+0000 7f8b0bfff640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8af406b810 0x7f8af406dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b140c8100 tx=0x7f8afc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:57.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:57.866+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b04076530 con 0x7f8b140719d0 2024-02-19T14:51:58.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.036+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b14074990 con 0x7f8b140719d0 2024-02-19T14:51:58.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.228+0000 7f8b10ff9640 1 -- 172.21.15.78:0/2973982542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8b0402f0b0 con 0x7f8b140719d0 2024-02-19T14:51:58.231 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8af406b810 msgr2=0x7f8af406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8af406b810 0x7f8af406dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b140c8100 tx=0x7f8afc008040 comp rx=0 tx=0).stop 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 msgr2=0x7f8b141dcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 0x7f8b141dcdb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b0402b820 tx=0x7f8b040046c0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 shutdown_connections 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8af406b810 0x7f8af406dcd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b141503a0 0x7f8b141eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8b140c2ba0 0x7f8b140c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 --2- 172.21.15.78:0/2973982542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b140719d0 0x7f8b141dcdb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 >> 172.21.15.78:0/2973982542 conn(0x7f8b140c86a0 msgr2=0x7f8b1414e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 shutdown_connections 2024-02-19T14:51:58.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.230+0000 7f8b19ca4640 1 -- 172.21.15.78:0/2973982542 wait complete. 2024-02-19T14:51:58.247 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-02-19T14:51:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.350+0000 7f1c2e1a2640 1 Processor -- start 2024-02-19T14:51:58.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.351+0000 7f1c2e1a2640 1 -- start start 2024-02-19T14:51:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.351+0000 7f1c2e1a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c280c8a30 0x7f1c280c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.351+0000 7f1c2e1a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c280c5560 0x7f1c28151f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.351+0000 7f1c2e1a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280c5a00 con 0x7f1c28152490 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.351+0000 7f1c2e1a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c28155ee0 con 0x7f1c280c8a30 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.351+0000 7f1c2e1a2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c28155120 con 0x7f1c280c5560 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c280c8a30 0x7f1c280c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c280c8a30 0x7f1c280c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49686/0 (socket says 172.21.15.78:49686) 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 -- 172.21.15.78:0/1629798479 learned_addr learned my addr 172.21.15.78:0/1629798479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2c99f640 1 --2- 172.21.15.78:0/1629798479 >> v2:172.21.15.150:3300/0 conn(0x7f1c280c5560 0x7f1c28151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 -- 172.21.15.78:0/1629798479 >> v2:172.21.15.150:3300/0 conn(0x7f1c280c5560 msgr2=0x7f1c28151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 --2- 172.21.15.78:0/1629798479 >> v2:172.21.15.150:3300/0 conn(0x7f1c280c5560 0x7f1c28151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 -- 172.21.15.78:0/1629798479 >> v1:172.21.15.150:6789/0 conn(0x7f1c28152490 legacy=0x7f1c2806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.352+0000 7f1c2d1a0640 1 -- 172.21.15.78:0/1629798479 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28155400 con 0x7f1c280c8a30 2024-02-19T14:51:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2d1a0640 1 --2- 172.21.15.78:0/1629798479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280c5020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c280c6aa0 tx=0x7f1c2402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3aa5c557cac7ce server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c1ffff640 1 -- 172.21.15.78:0/1629798479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c240317e0 con 0x7f1c280c8a30 2024-02-19T14:51:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c1ffff640 1 -- 172.21.15.78:0/1629798479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c24031940 con 0x7f1c280c8a30 2024-02-19T14:51:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c1ffff640 1 -- 172.21.15.78:0/1629798479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24031cd0 con 0x7f1c280c8a30 2024-02-19T14:51:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/1629798479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 msgr2=0x7f1c280c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/1629798479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280c5020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c280c6aa0 tx=0x7f1c2402b880 comp rx=0 tx=0).stop 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/1629798479 shutdown_connections 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/1629798479 >> v2:172.21.15.150:3300/0 conn(0x7f1c280c5560 0x7f1c28151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/1629798479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280c5020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/1629798479 >> 172.21.15.78:0/1629798479 conn(0x7f1c280ac270 msgr2=0x7f1c280ac680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/1629798479 shutdown_connections 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.353+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/1629798479 wait complete. 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.354+0000 7f1c2e1a2640 1 Processor -- start 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 -- start start 2024-02-19T14:51:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c280c5560 0x7f1c28158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c28152490 0x7f1c28147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28154e70 con 0x7f1c280c8a30 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28159230 con 0x7f1c280c5560 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2e1a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281548b0 con 0x7f1c28152490 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2c99f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2c99f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280ab400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49702/0 (socket says 172.21.15.78:49702) 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.355+0000 7f1c2c99f640 1 -- 172.21.15.78:0/604982393 learned_addr learned my addr 172.21.15.78:0/604982393 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2c99f640 1 -- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c28152490 msgr2=0x7f1c28147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2d1a0640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c280c5560 0x7f1c28158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2c99f640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c28152490 0x7f1c28147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2c99f640 1 -- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c280c5560 msgr2=0x7f1c28158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2c99f640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c280c5560 0x7f1c28158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2c99f640 1 -- 172.21.15.78:0/604982393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240094d0 con 0x7f1c280c8a30 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2c99f640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280ab400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c7f0 tx=0x7f1c1800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18004220 con 0x7f1c280c8a30 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c180043a0 con 0x7f1c280c8a30 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c28154b90 con 0x7f1c280c8a30 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18007860 con 0x7f1c280c8a30 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.356+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c280c4180 con 0x7f1c280c8a30 2024-02-19T14:51:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.357+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c280c3ce0 con 0x7f1c280c8a30 2024-02-19T14:51:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.357+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1c18007bd0 con 0x7f1c280c8a30 2024-02-19T14:51:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.358+0000 7f1c1dffb640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0c06b810 0x7f1c0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.358+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f1c18089fc0 con 0x7f1c280c8a30 2024-02-19T14:51:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.358+0000 7f1c2d1a0640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0c06b810 0x7f1c0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:58.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.358+0000 7f1c2d1a0640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0c06b810 0x7f1c0c06dcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1c24036000 tx=0x7f1c2403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.362+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1c180bddc0 con 0x7f1c280c8a30 2024-02-19T14:51:58.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:58.533+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f1c280c4370 con 0x7f1c280c8a30 2024-02-19T14:51:59.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.238+0000 7f1c1dffb640 1 -- 172.21.15.78:0/604982393 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f1c1801e020 con 0x7f1c280c8a30 2024-02-19T14:51:59.241 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0c06b810 msgr2=0x7f1c0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0c06b810 0x7f1c0c06dcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1c24036000 tx=0x7f1c2403d040 comp rx=0 tx=0).stop 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 msgr2=0x7f1c280ab400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280ab400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c7f0 tx=0x7f1c1800ccb0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 shutdown_connections 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0c06b810 0x7f1c0c06dcd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c280c8a30 0x7f1c280ab400 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c28152490 0x7f1c28147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 --2- 172.21.15.78:0/604982393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c280c5560 0x7f1c28158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 >> 172.21.15.78:0/604982393 conn(0x7f1c280ac270 msgr2=0x7f1c2814cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 shutdown_connections 2024-02-19T14:51:59.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.241+0000 7f1c2e1a2640 1 -- 172.21.15.78:0/604982393 wait complete. 2024-02-19T14:51:59.265 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:51:59.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.370+0000 7fd0e8d5c640 1 Processor -- start 2024-02-19T14:51:59.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.370+0000 7fd0e8d5c640 1 -- start start 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e8d5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd0e4152700 0x7fd0e40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e8d5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e8d5c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd0e40c8e00 0x7fd0e406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e8d5c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e406cd20 con 0x7fd0e4152700 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e8d5c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e41551f0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e8d5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4155490 con 0x7fd0e40c5730 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd0e4152700 0x7fd0e40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd0e4152700 0x7fd0e40c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49708/0 (socket says 172.21.15.78:49708) 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 -- 172.21.15.78:0/3035975430 learned_addr learned my addr 172.21.15.78:0/3035975430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e2d9d640 1 --2- 172.21.15.78:0/3035975430 >> v2:172.21.15.150:3300/0 conn(0x7fd0e40c8e00 0x7fd0e406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e1d9b640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 -- 172.21.15.78:0/3035975430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 msgr2=0x7fd0e406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 -- 172.21.15.78:0/3035975430 >> v2:172.21.15.150:3300/0 conn(0x7fd0e40c8e00 msgr2=0x7fd0e406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 --2- 172.21.15.78:0/3035975430 >> v2:172.21.15.150:3300/0 conn(0x7fd0e40c8e00 0x7fd0e406c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e259c640 1 -- 172.21.15.78:0/3035975430 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4155770 con 0x7fd0e4152700 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e1d9b640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.371+0000 7fd0e2d9d640 1 --2- 172.21.15.78:0/3035975430 >> v2:172.21.15.150:3300/0 conn(0x7fd0e40c8e00 0x7fd0e406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.372+0000 7fd0e259c640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e4152700 0x7fd0e40c51f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40c8bc0 tx=0x7fd0d402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a281d2a5685ecbd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.372+0000 7fd0e159a640 1 -- 172.21.15.78:0/3035975430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4031860 con 0x7fd0e4152700 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.372+0000 7fd0e159a640 1 -- 172.21.15.78:0/3035975430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d40319c0 con 0x7fd0e4152700 2024-02-19T14:51:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.372+0000 7fd0e159a640 1 -- 172.21.15.78:0/3035975430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4031d50 con 0x7fd0e4152700 2024-02-19T14:51:59.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/3035975430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e4152700 msgr2=0x7fd0e40c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:59.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e4152700 0x7fd0e40c51f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40c8bc0 tx=0x7fd0d402b900 comp rx=0 tx=0).stop 2024-02-19T14:51:59.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/3035975430 shutdown_connections 2024-02-19T14:51:59.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/3035975430 >> v2:172.21.15.150:3300/0 conn(0x7fd0e40c8e00 0x7fd0e406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/3035975430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e4152700 0x7fd0e40c51f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/3035975430 >> 172.21.15.78:0/3035975430 conn(0x7fd0e40cac90 msgr2=0x7fd0e40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/3035975430 shutdown_connections 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.373+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/3035975430 wait complete. 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.374+0000 7fd0e8d5c640 1 Processor -- start 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.374+0000 7fd0e8d5c640 1 -- start start 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e8d5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e40ab450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e8d5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 0x7fd0e414d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e8d5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0e4152700 0x7fd0e40aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e8d5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40aa710 con 0x7fd0e40c8e00 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e8d5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40ad250 con 0x7fd0e4152700 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e8d5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40accd0 con 0x7fd0e40c5730 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 0x7fd0e414d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 0x7fd0e414d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49716/0 (socket says 172.21.15.78:49716) 2024-02-19T14:51:59.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 -- 172.21.15.78:0/1443360904 learned_addr learned my addr 172.21.15.78:0/1443360904 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e2d9d640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0e4152700 0x7fd0e40aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e259c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e40ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 -- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 msgr2=0x7fd0e40ab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e40ab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 -- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0e4152700 msgr2=0x7fd0e40aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0e4152700 0x7fd0e40aaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.375+0000 7fd0e1d9b640 1 -- 172.21.15.78:0/1443360904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40094d0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.376+0000 7fd0e1d9b640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 0x7fd0e414d0d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c7f0 tx=0x7fd0d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.376+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8007a30 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.376+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8007b90 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.376+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e40aa3e0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.376+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d800ce40 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.376+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e414d610 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.377+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e40aa8c0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.377+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd0d80113f0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.377+0000 7fd0d2ffd640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd0c406b840 0x7fd0c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:51:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.378+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd0d808a1e0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.378+0000 7fd0e259c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd0c406b840 0x7fd0c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:51:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.378+0000 7fd0e259c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd0c406b840 0x7fd0c406dd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4036000 tx=0x7fd0d403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:51:59.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.382+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd0e40aa8c0 con 0x7fd0e40c8e00 2024-02-19T14:51:59.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:51:59.589+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd0e4074990 con 0x7fd0e40c8e00 2024-02-19T14:52:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.251+0000 7fd0d2ffd640 1 -- 172.21.15.78:0/1443360904 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd0d8054d10 con 0x7fd0e40c8e00 2024-02-19T14:52:00.254 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2024-02-19T14:52:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd0c406b840 msgr2=0x7fd0c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd0c406b840 0x7fd0c406dd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4036000 tx=0x7fd0d403c040 comp rx=0 tx=0).stop 2024-02-19T14:52:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 msgr2=0x7fd0e414d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 0x7fd0e414d0d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c7f0 tx=0x7fd0d800cbf0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 shutdown_connections 2024-02-19T14:52:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd0c406b840 0x7fd0c406dd00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0e40c8e00 0x7fd0e414d0d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0e4152700 0x7fd0e40aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 --2- 172.21.15.78:0/1443360904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0e40c5730 0x7fd0e40ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.256+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 >> 172.21.15.78:0/1443360904 conn(0x7fd0e40cac90 msgr2=0x7fd0e40cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.257+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 shutdown_connections 2024-02-19T14:52:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.258+0000 7fd0e8d5c640 1 -- 172.21.15.78:0/1443360904 wait complete. 2024-02-19T14:52:00.280 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-02-19T14:52:00.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-02-19T14:52:00.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.383+0000 7f431f872640 1 Processor -- start 2024-02-19T14:52:00.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.383+0000 7f431f872640 1 -- start start 2024-02-19T14:52:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431f872640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f431806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431f872640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431806dc70 con 0x7f4318154ea0 2024-02-19T14:52:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431f872640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318158ef0 con 0x7f43180c8730 2024-02-19T14:52:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431f872640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4318155a40 con 0x7f4318152670 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431de0f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4318154ea0 0x7f4318155330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49772/0 (socket says 172.21.15.78:49772) 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431de0f640 1 -- 172.21.15.78:0/427274086 learned_addr learned my addr 172.21.15.78:0/427274086 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f431d60e640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f431806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288849079 0 0) 0x7f431806dc70 con 0x7f4318154ea0 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.384+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4304003520 con 0x7f4318154ea0 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2510164268 0 0) 0x7f4304003520 con 0x7f4318154ea0 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 msgr2=0x7f431806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f431806d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 >> v1:172.21.15.150:6789/0 conn(0x7f43180c8730 legacy=0x7f43180c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180be190 con 0x7f4318154ea0 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f43100019f0 con 0x7f4318154ea0 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 >> v1:172.21.15.78:6789/0 conn(0x7f4318154ea0 legacy=0x7f4318155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f4304003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43040047c0 0x7f4304004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f4304005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430400bb10 con 0x7f43040039a0 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430400bc50 con 0x7f43040047c0 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430400be10 con 0x7f4318152670 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431d60e640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f4304003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431ce0d640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43040047c0 0x7f4304004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431de0f640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f4304005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431d60e640 1 -- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 msgr2=0x7f4304005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431d60e640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f4304005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431d60e640 1 -- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43040047c0 msgr2=0x7f4304004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431d60e640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43040047c0 0x7f4304004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431d60e640 1 -- 172.21.15.78:0/427274086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304003520 con 0x7f43040039a0 2024-02-19T14:52:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.385+0000 7f431de0f640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4318152670 0x7f4304005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f431d60e640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f4304003d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4314009ac0 tx=0x7f43040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3938c42116c65288 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431402c6b0 con 0x7f43040039a0 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431402c810 con 0x7f43040039a0 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f430ffff640 1 -- 172.21.15.78:0/427274086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431402cba0 con 0x7f43040039a0 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f431f872640 1 -- 172.21.15.78:0/427274086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 msgr2=0x7f4304003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f431f872640 1 --2- 172.21.15.78:0/427274086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f4304003d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4314009ac0 tx=0x7f43040057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.386+0000 7f431de0f640 1 -- 172.21.15.78:0/427274086 reap_dead start 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.387+0000 7f431f872640 1 -- 172.21.15.78:0/427274086 shutdown_connections 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.387+0000 7f431f872640 1 -- 172.21.15.78:0/427274086 >> 172.21.15.78:0/427274086 conn(0x7f4318072790 msgr2=0x7f4318072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.387+0000 7f431f872640 1 -- 172.21.15.78:0/427274086 shutdown_connections 2024-02-19T14:52:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.387+0000 7f431f872640 1 -- 172.21.15.78:0/427274086 wait complete. 2024-02-19T14:52:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 Processor -- start 2024-02-19T14:52:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 -- start start 2024-02-19T14:52:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f43180c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43180ad720 0x7f43181edc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43180c09a0 0x7f43180be320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180bf570 con 0x7f43040039a0 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180bfe70 con 0x7f43180c09a0 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431f872640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43180bedb0 con 0x7f43180ad720 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431d60e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f43180c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431d60e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f43180c4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49740/0 (socket says 172.21.15.78:49740) 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.388+0000 7f431d60e640 1 -- 172.21.15.78:0/4093979350 learned_addr learned my addr 172.21.15.78:0/4093979350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431de0f640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43180c09a0 0x7f43180be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431ce0d640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43180ad720 0x7f43181edc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431d60e640 1 -- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43180ad720 msgr2=0x7f43181edc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431d60e640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43180ad720 0x7f43181edc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431d60e640 1 -- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43180c09a0 msgr2=0x7f43180be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431d60e640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43180c09a0 0x7f43180be320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431d60e640 1 -- 172.21.15.78:0/4093979350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43181ee580 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431de0f640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43180c09a0 0x7f43180be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431d60e640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f43180c4400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4314000c00 tx=0x7f4314004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314003810 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314003970 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314003d20 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43180bf050 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.389+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43180c4020 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.390+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4318074990 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.391+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4314034440 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.391+0000 7f430dffb640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f42fc06b810 0x7f42fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.391+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43140abfe0 con 0x7f43040039a0 2024-02-19T14:52:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.391+0000 7f431ce0d640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f42fc06b810 0x7f42fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.391+0000 7f431ce0d640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f42fc06b810 0x7f42fc06dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f43180adb20 tx=0x7f4308009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.395+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4314076c60 con 0x7f43040039a0 2024-02-19T14:52:00.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:00.577+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f43181522d0 con 0x7f43040039a0 2024-02-19T14:52:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.241+0000 7f430dffb640 1 -- 172.21.15.78:0/4093979350 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f431407a320 con 0x7f43040039a0 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f42fc06b810 msgr2=0x7f42fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f42fc06b810 0x7f42fc06dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f43180adb20 tx=0x7f4308009040 comp rx=0 tx=0).stop 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 msgr2=0x7f43180c4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f43180c4400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4314000c00 tx=0x7f4314004730 comp rx=0 tx=0).stop 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 shutdown_connections 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f42fc06b810 0x7f42fc06dcd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43180c09a0 0x7f43180be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43180ad720 0x7f43181edc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 --2- 172.21.15.78:0/4093979350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43040039a0 0x7f43180c4400 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 >> 172.21.15.78:0/4093979350 conn(0x7f4318072790 msgr2=0x7f43180c50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 shutdown_connections 2024-02-19T14:52:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.244+0000 7f431f872640 1 -- 172.21.15.78:0/4093979350 wait complete. 2024-02-19T14:52:01.258 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:52:01.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.367+0000 7fccfb3e9640 1 Processor -- start 2024-02-19T14:52:01.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.367+0000 7fccfb3e9640 1 -- start start 2024-02-19T14:52:01.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccfb3e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf41502f0 0x7fccf4072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:01.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccfb3e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fccf406ae80 0x7fccf406b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccfb3e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf41506d0 con 0x7fccf40c5cf0 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccfb3e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf40724c0 con 0x7fccf406ae80 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccfb3e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf4072800 con 0x7fccf41502f0 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccf9986640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fccf406ae80 0x7fccf406b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.368+0000 7fccf9986640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fccf406ae80 0x7fccf406b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49754/0 (socket says 172.21.15.78:49754) 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.369+0000 7fccf9986640 1 -- 172.21.15.78:0/1984419222 learned_addr learned my addr 172.21.15.78:0/1984419222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.369+0000 7fccf9185640 1 --2- 172.21.15.78:0/1984419222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf41502f0 0x7fccf4072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.369+0000 7fccf9986640 1 -- 172.21.15.78:0/1984419222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf41502f0 msgr2=0x7fccf4072fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.369+0000 7fccf9986640 1 --2- 172.21.15.78:0/1984419222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf41502f0 0x7fccf4072fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.369+0000 7fccf9986640 1 -- 172.21.15.78:0/1984419222 >> v1:172.21.15.150:6789/0 conn(0x7fccf40c5cf0 legacy=0x7fccf4152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:01.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.369+0000 7fccf9986640 1 -- 172.21.15.78:0/1984419222 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf4073500 con 0x7fccf406ae80 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccf9986640 1 --2- 172.21.15.78:0/1984419222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf406ae80 0x7fccf406b290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fccf40736f0 tx=0x7fccec02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bcc6efe261bd1b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccebfff640 1 -- 172.21.15.78:0/1984419222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec02ca50 con 0x7fccf406ae80 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccebfff640 1 -- 172.21.15.78:0/1984419222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec02cbb0 con 0x7fccf406ae80 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccebfff640 1 -- 172.21.15.78:0/1984419222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec0343f0 con 0x7fccf406ae80 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 -- 172.21.15.78:0/1984419222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf406ae80 msgr2=0x7fccf406b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/1984419222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf406ae80 0x7fccf406b290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fccf40736f0 tx=0x7fccec02beb0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 -- 172.21.15.78:0/1984419222 shutdown_connections 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/1984419222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf406ae80 0x7fccf406b290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/1984419222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf41502f0 0x7fccf4072fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 -- 172.21.15.78:0/1984419222 >> 172.21.15.78:0/1984419222 conn(0x7fccf406c700 msgr2=0x7fccf40a9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:01.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 -- 172.21.15.78:0/1984419222 shutdown_connections 2024-02-19T14:52:01.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.370+0000 7fccfb3e9640 1 -- 172.21.15.78:0/1984419222 wait complete. 2024-02-19T14:52:01.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.371+0000 7fccfb3e9640 1 Processor -- start 2024-02-19T14:52:01.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.371+0000 7fccfb3e9640 1 -- start start 2024-02-19T14:52:01.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccfb3e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf406ae80 0x7fccf40a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:01.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccfb3e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 0x7fccf40c0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:01.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccfb3e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf41502f0 0x7fccf40c0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:01.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccfb3e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf4155b20 con 0x7fccf40c5cf0 2024-02-19T14:52:01.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccfb3e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf4156b00 con 0x7fccf41502f0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccfb3e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf41527f0 con 0x7fccf406ae80 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccf8984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 0x7fccf40c0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccf8984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 0x7fccf40c0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49760/0 (socket says 172.21.15.78:49760) 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccf8984640 1 -- 172.21.15.78:0/114404939 learned_addr learned my addr 172.21.15.78:0/114404939 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccf9185640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf406ae80 0x7fccf40a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.372+0000 7fccf9986640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf41502f0 0x7fccf40c0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccf8984640 1 -- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf406ae80 msgr2=0x7fccf40a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccf8984640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf406ae80 0x7fccf40a89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccf8984640 1 -- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf41502f0 msgr2=0x7fccf40c0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccf8984640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf41502f0 0x7fccf40c0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccf8984640 1 -- 172.21.15.78:0/114404939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec009b30 con 0x7fccf40c5cf0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccf8984640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 0x7fccf40c0940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcce400c400 tx=0x7fcce400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce40052d0 con 0x7fccf40c5cf0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce4005450 con 0x7fccf40c5cf0 2024-02-19T14:52:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf41ec3a0 con 0x7fccf40c5cf0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4007260 con 0x7fccf40c5cf0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.373+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf4152a40 con 0x7fccf40c5cf0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.374+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf4075190 con 0x7fccf40c5cf0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.374+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcce40073c0 con 0x7fccf40c5cf0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.375+0000 7fcce9ffb640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd806b810 0x7fccd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.375+0000 7fccf9185640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd806b810 0x7fccd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.375+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fcce40895f0 con 0x7fccf40c5cf0 2024-02-19T14:52:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.375+0000 7fccf9185640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd806b810 0x7fccd806dcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fccf000aac0 tx=0x7fccf0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:01.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.379+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcce4054020 con 0x7fccf40c5cf0 2024-02-19T14:52:01.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:01.627+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fccf4072230 con 0x7fccf40c5cf0 2024-02-19T14:52:03.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.256+0000 7fcce9ffb640 1 -- 172.21.15.78:0/114404939 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7fcce40576e0 con 0x7fccf40c5cf0 2024-02-19T14:52:03.259 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' created 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd806b810 msgr2=0x7fccd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd806b810 0x7fccd806dcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fccf000aac0 tx=0x7fccf0008040 comp rx=0 tx=0).stop 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 msgr2=0x7fccf40c0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 0x7fccf40c0940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcce400c400 tx=0x7fcce400c8c0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 shutdown_connections 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd806b810 0x7fccd806dcd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf40c5cf0 0x7fccf40c0940 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf41502f0 0x7fccf40c0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 --2- 172.21.15.78:0/114404939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf406ae80 0x7fccf40a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 >> 172.21.15.78:0/114404939 conn(0x7fccf406c700 msgr2=0x7fccf414f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:03.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 shutdown_connections 2024-02-19T14:52:03.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.259+0000 7fccfb3e9640 1 -- 172.21.15.78:0/114404939 wait complete. 2024-02-19T14:52:03.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-02-19T14:52:03.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.380+0000 7fdc9a15c640 1 Processor -- start 2024-02-19T14:52:03.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.381+0000 7fdc9a15c640 1 -- start start 2024-02-19T14:52:03.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc9a15c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc94151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:03.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc9a15c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 0x7fdc9415c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:03.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc9a15c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9414f610 con 0x7fdc9414f230 2024-02-19T14:52:03.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc9a15c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc940c5810 con 0x7fdc9415a4f0 2024-02-19T14:52:03.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc9a15c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc940c5530 con 0x7fdc94151b80 2024-02-19T14:52:03.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc94151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc93fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 0x7fdc9415c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc94151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:55228/0 (socket says 172.21.15.78:55228) 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 -- 172.21.15.78:0/961024343 learned_addr learned my addr 172.21.15.78:0/961024343 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc93fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 0x7fdc9415c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57680/0 (socket says 172.21.15.78:57680) 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc937fe640 1 --1- 172.21.15.78:0/961024343 >> v1:172.21.15.150:6789/0 conn(0x7fdc9414f230 0x7fdc940c2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58322/0 (socket says 172.21.15.78:58322) 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 -- 172.21.15.78:0/961024343 >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 msgr2=0x7fdc9415c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 --2- 172.21.15.78:0/961024343 >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 0x7fdc9415c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 -- 172.21.15.78:0/961024343 >> v1:172.21.15.150:6789/0 conn(0x7fdc9414f230 legacy=0x7fdc940c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc92ffd640 1 -- 172.21.15.78:0/961024343 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc94155cb0 con 0x7fdc94151b80 2024-02-19T14:52:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.382+0000 7fdc93fff640 1 --2- 172.21.15.78:0/961024343 >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 0x7fdc9415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:03.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.383+0000 7fdc92ffd640 1 --2- 172.21.15.78:0/961024343 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc94151fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009870 tx=0x7fdc8402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e2680908aecd480 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:03.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.383+0000 7fdc927fc640 1 -- 172.21.15.78:0/961024343 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc840317c0 con 0x7fdc94151b80 2024-02-19T14:52:03.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.383+0000 7fdc927fc640 1 -- 172.21.15.78:0/961024343 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc84031920 con 0x7fdc94151b80 2024-02-19T14:52:03.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.383+0000 7fdc927fc640 1 -- 172.21.15.78:0/961024343 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84031cb0 con 0x7fdc94151b80 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 -- 172.21.15.78:0/961024343 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 msgr2=0x7fdc94151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/961024343 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc94151fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009870 tx=0x7fdc8402b860 comp rx=0 tx=0).stop 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 -- 172.21.15.78:0/961024343 shutdown_connections 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/961024343 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc94151fe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/961024343 >> v2:172.21.15.150:3300/0 conn(0x7fdc9415a4f0 0x7fdc9415c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 -- 172.21.15.78:0/961024343 >> 172.21.15.78:0/961024343 conn(0x7fdc9406c3b0 msgr2=0x7fdc9406c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 -- 172.21.15.78:0/961024343 shutdown_connections 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.384+0000 7fdc9a15c640 1 -- 172.21.15.78:0/961024343 wait complete. 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.386+0000 7fdc9a15c640 1 Processor -- start 2024-02-19T14:52:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.386+0000 7fdc9a15c640 1 -- start start 2024-02-19T14:52:03.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc9a15c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 0x7fdc941564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc9a15c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc9406b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc9a15c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc9415a4f0 0x7fdc94158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc9a15c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94158210 con 0x7fdc9414f230 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc9a15c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94146880 con 0x7fdc9415a4f0 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc9a15c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94146b20 con 0x7fdc94151b80 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 0x7fdc941564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 0x7fdc941564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49764/0 (socket says 172.21.15.78:49764) 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc937fe640 1 -- 172.21.15.78:0/4285744483 learned_addr learned my addr 172.21.15.78:0/4285744483 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc93fff640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc9415a4f0 0x7fdc94158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.387+0000 7fdc92ffd640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc9406b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc937fe640 1 -- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 msgr2=0x7fdc9406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc937fe640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc9406b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc937fe640 1 -- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc9415a4f0 msgr2=0x7fdc94158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc937fe640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc9415a4f0 0x7fdc94158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc937fe640 1 -- 172.21.15.78:0/4285744483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c009a70 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc937fe640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 0x7fdc941564d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800c3c0 tx=0x7fdc8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88005c70 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88005dd0 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc840094d0 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc880072d0 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.388+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc94158740 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.389+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc94146d40 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.389+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdc88007430 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.390+0000 7fdc7bfff640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdc8006b840 0x7fdc8006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.390+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdc880898b0 con 0x7fdc9414f230 2024-02-19T14:52:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.390+0000 7fdc92ffd640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdc8006b840 0x7fdc8006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:03.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.390+0000 7fdc92ffd640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdc8006b840 0x7fdc8006dd00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc84036000 tx=0x7fdc8403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.394+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdc88054060 con 0x7fdc9414f230 2024-02-19T14:52:03.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:03.570+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fdc94074990 con 0x7fdc9414f230 2024-02-19T14:52:04.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.258+0000 7fdc7bfff640 1 -- 172.21.15.78:0/4285744483 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdc880170c0 con 0x7fdc9414f230 2024-02-19T14:52:04.261 INFO:teuthology.orchestra.run.smithi078.stderr:set pool 4 allow_ec_overwrites to true 2024-02-19T14:52:04.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.261+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdc8006b840 msgr2=0x7fdc8006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:04.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.261+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdc8006b840 0x7fdc8006dd00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc84036000 tx=0x7fdc8403d040 comp rx=0 tx=0).stop 2024-02-19T14:52:04.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.261+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 msgr2=0x7fdc941564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:04.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.261+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 0x7fdc941564d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800c3c0 tx=0x7fdc8800c880 comp rx=0 tx=0).stop 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 shutdown_connections 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdc8006b840 0x7fdc8006dd00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc9415a4f0 0x7fdc94158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc94151b80 0x7fdc9406b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 --2- 172.21.15.78:0/4285744483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc9414f230 0x7fdc941564d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 >> 172.21.15.78:0/4285744483 conn(0x7fdc9406c3b0 msgr2=0x7fdc9414b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 shutdown_connections 2024-02-19T14:52:04.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.262+0000 7fdc9a15c640 1 -- 172.21.15.78:0/4285744483 wait complete. 2024-02-19T14:52:04.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-02-19T14:52:04.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.392+0000 7ff217725640 1 Processor -- start 2024-02-19T14:52:04.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.392+0000 7ff217725640 1 -- start start 2024-02-19T14:52:04.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff217725640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff2100ace90 0x7ff2100c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff217725640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff2100c8ee0 0x7ff2100c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff217725640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff210152a60 con 0x7ff210152680 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff217725640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2100aa050 con 0x7ff2100c8ee0 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff217725640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2101557e0 con 0x7ff2100ace90 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff2154c1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff210152680 0x7ff210155000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49788/0 (socket says 172.21.15.78:49788) 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff215cc2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff2100c8ee0 0x7ff2100c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff2154c1640 1 -- 172.21.15.78:0/572076309 learned_addr learned my addr 172.21.15.78:0/572076309 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff215cc2640 1 -- 172.21.15.78:0/572076309 >> v2:172.21.15.150:3300/0 conn(0x7ff2100ace90 msgr2=0x7ff2100c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff214cc0640 1 --2- 172.21.15.78:0/572076309 >> v2:172.21.15.150:3300/0 conn(0x7ff2100ace90 0x7ff2100c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff215cc2640 1 --2- 172.21.15.78:0/572076309 >> v2:172.21.15.150:3300/0 conn(0x7ff2100ace90 0x7ff2100c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.393+0000 7ff207fff640 1 -- 172.21.15.78:0/572076309 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978606691 0 0) 0x7ff210152a60 con 0x7ff210152680 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff215cc2640 1 -- 172.21.15.78:0/572076309 >> v1:172.21.15.78:6789/0 conn(0x7ff210152680 legacy=0x7ff210155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:04.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff215cc2640 1 -- 172.21.15.78:0/572076309 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff210155ac0 con 0x7ff2100c8ee0 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff214cc0640 1 --2- 172.21.15.78:0/572076309 >> v2:172.21.15.150:3300/0 conn(0x7ff2100ace90 0x7ff2100c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff215cc2640 1 --2- 172.21.15.78:0/572076309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c9b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff21008bf60 tx=0x7ff20802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34a07bc755f1b35 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff207fff640 1 -- 172.21.15.78:0/572076309 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20802cbf0 con 0x7ff2100c8ee0 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff207fff640 1 -- 172.21.15.78:0/572076309 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20802cd50 con 0x7ff2100c8ee0 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.394+0000 7ff207fff640 1 -- 172.21.15.78:0/572076309 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208034660 con 0x7ff2100c8ee0 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 -- 172.21.15.78:0/572076309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100c8ee0 msgr2=0x7ff2100c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:04.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 --2- 172.21.15.78:0/572076309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c9b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff21008bf60 tx=0x7ff20802be40 comp rx=0 tx=0).stop 2024-02-19T14:52:04.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 -- 172.21.15.78:0/572076309 shutdown_connections 2024-02-19T14:52:04.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 --2- 172.21.15.78:0/572076309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c9b80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 --2- 172.21.15.78:0/572076309 >> v2:172.21.15.150:3300/0 conn(0x7ff2100ace90 0x7ff2100c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 -- 172.21.15.78:0/572076309 >> 172.21.15.78:0/572076309 conn(0x7ff2100ac370 msgr2=0x7ff2100ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:04.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 -- 172.21.15.78:0/572076309 shutdown_connections 2024-02-19T14:52:04.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.395+0000 7ff217725640 1 -- 172.21.15.78:0/572076309 wait complete. 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.396+0000 7ff217725640 1 Processor -- start 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.396+0000 7ff217725640 1 -- start start 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 0x7ff2101edba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff210152680 0x7ff2100c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101561d0 con 0x7ff2100ace90 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101ee470 con 0x7ff2100c8ee0 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100c0bf0 con 0x7ff210152680 2024-02-19T14:52:04.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 0x7ff2101edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 0x7ff2101edba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49780/0 (socket says 172.21.15.78:49780) 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 -- 172.21.15.78:0/581709818 learned_addr learned my addr 172.21.15.78:0/581709818 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff214cc0640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff215cc2640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff210152680 0x7ff2100c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 -- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff210152680 msgr2=0x7ff2100c2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff210152680 0x7ff2100c2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 -- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2100c8ee0 msgr2=0x7ff2100c2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 -- 172.21.15.78:0/581709818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208009a70 con 0x7ff2100ace90 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff2154c1640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 0x7ff2101edba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c430 tx=0x7ff20000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2000074e0 con 0x7ff2100ace90 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200007660 con 0x7ff2100ace90 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100c0ed0 con 0x7ff2100ace90 2024-02-19T14:52:04.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.397+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200004a20 con 0x7ff2100ace90 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.398+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2100c30f0 con 0x7ff2100ace90 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.399+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff210158fb0 con 0x7ff2100ace90 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.399+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff200004b80 con 0x7ff2100ace90 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.400+0000 7ff205ffb640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1f406b810 0x7ff1f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.400+0000 7ff214cc0640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1f406b810 0x7ff1f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.400+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff20008a260 con 0x7ff2100ace90 2024-02-19T14:52:04.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.400+0000 7ff214cc0640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1f406b810 0x7ff1f406dcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2100c4ac0 tx=0x7ff20c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:04.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.408+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff210158fb0 con 0x7ff2100ace90 2024-02-19T14:52:04.658 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:52:04.656+0000 7f781f2f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:52:04.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:04.684+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7ff210158fb0 con 0x7ff2100ace90 2024-02-19T14:52:05.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.283+0000 7ff205ffb640 1 -- 172.21.15.78:0/581709818 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7ff210158fb0 con 0x7ff2100ace90 2024-02-19T14:52:05.286 INFO:teuthology.orchestra.run.smithi078.stderr:added data pool 4 to fsmap 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1f406b810 msgr2=0x7ff1f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1f406b810 0x7ff1f406dcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2100c4ac0 tx=0x7ff20c008040 comp rx=0 tx=0).stop 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 msgr2=0x7ff2101edba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 0x7ff2101edba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff20000c430 tx=0x7ff20000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 shutdown_connections 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1f406b810 0x7ff1f406dcd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff210152680 0x7ff2100c2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2100c8ee0 0x7ff2100c2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 --2- 172.21.15.78:0/581709818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2100ace90 0x7ff2101edba0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 >> 172.21.15.78:0/581709818 conn(0x7ff2100ac370 msgr2=0x7ff21006ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 shutdown_connections 2024-02-19T14:52:05.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.286+0000 7ff217725640 1 -- 172.21.15.78:0/581709818 wait complete. 2024-02-19T14:52:05.300 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.398+0000 7f03dc581640 1 Processor -- start 2024-02-19T14:52:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.398+0000 7f03dc581640 1 -- start start 2024-02-19T14:52:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03dc581640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03d414d210 0x7f03d40c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03dc581640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f03d40719e0 0x7f03d40c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03dc581640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d40c3d90 0x7f03d40c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03dc581640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d4070270 con 0x7f03d40719e0 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03dc581640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40c3970 con 0x7f03d414d210 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03dc581640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d406ff90 con 0x7f03d40c3d90 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f03d40719e0 0x7f03d40c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f03d40719e0 0x7f03d40c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49794/0 (socket says 172.21.15.78:49794) 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 -- 172.21.15.78:0/530898683 learned_addr learned my addr 172.21.15.78:0/530898683 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03da31d640 1 --2- 172.21.15.78:0/530898683 >> v2:172.21.15.150:3300/0 conn(0x7f03d414d210 0x7f03d40c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 -- 172.21.15.78:0/530898683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d40c3d90 msgr2=0x7f03d40c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 --2- 172.21.15.78:0/530898683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d40c3d90 0x7f03d40c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 -- 172.21.15.78:0/530898683 >> v2:172.21.15.150:3300/0 conn(0x7f03d414d210 msgr2=0x7f03d40c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 --2- 172.21.15.78:0/530898683 >> v2:172.21.15.150:3300/0 conn(0x7f03d414d210 0x7f03d40c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03d9b1c640 1 -- 172.21.15.78:0/530898683 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4156fc0 con 0x7f03d40719e0 2024-02-19T14:52:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.399+0000 7f03da31d640 1 --2- 172.21.15.78:0/530898683 >> v2:172.21.15.150:3300/0 conn(0x7f03d414d210 0x7f03d40c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.400+0000 7f03d9b1c640 1 --2- 172.21.15.78:0/530898683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40c64d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03d40c5de0 tx=0x7f03c402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce4f8fec59c3f11 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.400+0000 7f03d931b640 1 -- 172.21.15.78:0/530898683 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4031800 con 0x7f03d40719e0 2024-02-19T14:52:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.400+0000 7f03d931b640 1 -- 172.21.15.78:0/530898683 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4031960 con 0x7f03d40719e0 2024-02-19T14:52:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.400+0000 7f03d931b640 1 -- 172.21.15.78:0/530898683 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4031cf0 con 0x7f03d40719e0 2024-02-19T14:52:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.400+0000 7f03dc581640 1 -- 172.21.15.78:0/530898683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 msgr2=0x7f03d40c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.400+0000 7f03dc581640 1 --2- 172.21.15.78:0/530898683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40c64d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03d40c5de0 tx=0x7f03c402b8a0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 -- 172.21.15.78:0/530898683 shutdown_connections 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 --2- 172.21.15.78:0/530898683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d40c3d90 0x7f03d40c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 --2- 172.21.15.78:0/530898683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40c64d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 --2- 172.21.15.78:0/530898683 >> v2:172.21.15.150:3300/0 conn(0x7f03d414d210 0x7f03d40c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 -- 172.21.15.78:0/530898683 >> 172.21.15.78:0/530898683 conn(0x7f03d40c5050 msgr2=0x7f03d40c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 -- 172.21.15.78:0/530898683 shutdown_connections 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.401+0000 7f03dc581640 1 -- 172.21.15.78:0/530898683 wait complete. 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 Processor -- start 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 -- start start 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d40c3d90 0x7f03d4155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d414d210 0x7f03d4152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41eb5b0 con 0x7f03d40719e0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d406c130 con 0x7f03d40c3d90 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03dc581640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40add00 con 0x7f03d414d210 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03da31d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03da31d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49808/0 (socket says 172.21.15.78:49808) 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.402+0000 7f03da31d640 1 -- 172.21.15.78:0/3480396491 learned_addr learned my addr 172.21.15.78:0/3480396491 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03d9b1c640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d40c3d90 0x7f03d4155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03dab1e640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d414d210 0x7f03d4152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03da31d640 1 -- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d414d210 msgr2=0x7f03d4152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03da31d640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d414d210 0x7f03d4152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03da31d640 1 -- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d40c3d90 msgr2=0x7f03d4155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03da31d640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d40c3d90 0x7f03d4155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03da31d640 1 -- 172.21.15.78:0/3480396491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40094d0 con 0x7f03d40719e0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03da31d640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40ad7c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03d000c7f0 tx=0x7f03d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0007820 con 0x7f03d40719e0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d0007980 con 0x7f03d40719e0 2024-02-19T14:52:05.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d40c3660 con 0x7f03d40719e0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0007cf0 con 0x7f03d40719e0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.403+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d41eb2d0 con 0x7f03d40719e0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.404+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d406bf90 con 0x7f03d40719e0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.404+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f03d00113f0 con 0x7f03d40719e0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.405+0000 7f03caffd640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03b806b810 0x7f03b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.405+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f03d008a600 con 0x7f03d40719e0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.405+0000 7f03d9b1c640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03b806b810 0x7f03b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.405+0000 7f03d9b1c640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03b806b810 0x7f03b806dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03c4036000 tx=0x7f03c403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:05.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.409+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f03d406bf90 con 0x7f03d40719e0 2024-02-19T14:52:05.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.621+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03d4074990 con 0x7f03d40719e0 2024-02-19T14:52:05.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.621+0000 7f03caffd640 1 -- 172.21.15.78:0/3480396491 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4940 (secure 0 0 0) 0x7f03d0054d70 con 0x7f03d40719e0 2024-02-19T14:52:05.624 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:05.624 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:04.685406+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-19T14:52:05.625 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2024-02-19T14:52:05.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03b806b810 msgr2=0x7f03b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03b806b810 0x7f03b806dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03c4036000 tx=0x7f03c403c040 comp rx=0 tx=0).stop 2024-02-19T14:52:05.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 msgr2=0x7f03d40ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40ad7c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03d000c7f0 tx=0x7f03d000ccb0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 shutdown_connections 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03b806b810 0x7f03b806dcd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03d414d210 0x7f03d4152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d40c3d90 0x7f03d4155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 --2- 172.21.15.78:0/3480396491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03d40719e0 0x7f03d40ad7c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 >> 172.21.15.78:0/3480396491 conn(0x7f03d40c5050 msgr2=0x7f03d41498c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.623+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 shutdown_connections 2024-02-19T14:52:05.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.624+0000 7f03dc581640 1 -- 172.21.15.78:0/3480396491 wait complete. 2024-02-19T14:52:05.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.733+0000 7f1e1a1c7640 1 Processor -- start 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.733+0000 7f1e1a1c7640 1 -- start start 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e1a1c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e14152480 0x7f1e141559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e1a1c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e140c87a0 0x7f1e140c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e1a1c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e14155ef0 con 0x7f1e140c7070 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e1a1c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1406dec0 con 0x7f1e140c87a0 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e1a1c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1406d520 con 0x7f1e14152480 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e140c87a0 0x7f1e140c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e140c87a0 0x7f1e140c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49816/0 (socket says 172.21.15.78:49816) 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 -- 172.21.15.78:0/3053637648 learned_addr learned my addr 172.21.15.78:0/3053637648 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e137fe640 1 --2- 172.21.15.78:0/3053637648 >> v2:172.21.15.150:3300/0 conn(0x7f1e14152480 0x7f1e141559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 -- 172.21.15.78:0/3053637648 >> v2:172.21.15.150:3300/0 conn(0x7f1e14152480 msgr2=0x7f1e141559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 --2- 172.21.15.78:0/3053637648 >> v2:172.21.15.150:3300/0 conn(0x7f1e14152480 0x7f1e141559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 -- 172.21.15.78:0/3053637648 >> v1:172.21.15.150:6789/0 conn(0x7f1e140c7070 legacy=0x7f1e1406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.734+0000 7f1e13fff640 1 -- 172.21.15.78:0/3053637648 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e140c75c0 con 0x7f1e140c87a0 2024-02-19T14:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.735+0000 7f1e13fff640 1 --2- 172.21.15.78:0/3053637648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e140c5220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1e140aa1b0 tx=0x7f1e0c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43a0b67ac9ff810 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.735+0000 7f1e12ffd640 1 -- 172.21.15.78:0/3053637648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c02cb20 con 0x7f1e140c87a0 2024-02-19T14:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.735+0000 7f1e12ffd640 1 -- 172.21.15.78:0/3053637648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c02cc80 con 0x7f1e140c87a0 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.735+0000 7f1e12ffd640 1 -- 172.21.15.78:0/3053637648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0343f0 con 0x7f1e140c87a0 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/3053637648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 msgr2=0x7f1e140c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/3053637648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e140c5220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1e140aa1b0 tx=0x7f1e0c02bf80 comp rx=0 tx=0).stop 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/3053637648 shutdown_connections 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/3053637648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e140c5220 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/3053637648 >> v2:172.21.15.150:3300/0 conn(0x7f1e14152480 0x7f1e141559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/3053637648 >> 172.21.15.78:0/3053637648 conn(0x7f1e140ca8a0 msgr2=0x7f1e140cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/3053637648 shutdown_connections 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.736+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/3053637648 wait complete. 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.737+0000 7f1e1a1c7640 1 Processor -- start 2024-02-19T14:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.737+0000 7f1e1a1c7640 1 -- start start 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e140c7070 0x7f1e14150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e141eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e14152480 0x7f1e141eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140c2290 con 0x7f1e140c87a0 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140c1cd0 con 0x7f1e140c7070 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140c1fb0 con 0x7f1e14152480 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e141eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e141eaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49820/0 (socket says 172.21.15.78:49820) 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 -- 172.21.15.78:0/2803430275 learned_addr learned my addr 172.21.15.78:0/2803430275 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e137fe640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e140c7070 0x7f1e14150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e13fff640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e14152480 0x7f1e141eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 -- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e14152480 msgr2=0x7f1e141eb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e14152480 0x7f1e141eb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 -- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e140c7070 msgr2=0x7f1e14150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e140c7070 0x7f1e14150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 -- 172.21.15.78:0/2803430275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c009b30 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e137fe640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e140c7070 0x7f1e14150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e0affd640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e141eaf80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c470 tx=0x7f1dfc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0052d0 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc005430 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc007260 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e141eed80 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.738+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e14151470 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.739+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e14074990 con 0x7f1e140c87a0 2024-02-19T14:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.740+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1dfc0073c0 con 0x7f1e140c87a0 2024-02-19T14:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.740+0000 7f1e10ff9640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1df406b840 0x7f1df406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.740+0000 7f1e137fe640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1df406b840 0x7f1df406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.740+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1dfc0896f0 con 0x7f1e140c87a0 2024-02-19T14:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.741+0000 7f1e137fe640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1df406b840 0x7f1df406dd00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e140af6d0 tx=0x7f1e04008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:05.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.745+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1dfc053e60 con 0x7f1e140c87a0 2024-02-19T14:52:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.921+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e14155ef0 con 0x7f1e140c87a0 2024-02-19T14:52:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.922+0000 7f1e10ff9640 1 -- 172.21.15.78:0/2803430275 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f1e14155ef0 con 0x7f1e140c87a0 2024-02-19T14:52:05.925 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:05.925 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1df406b840 msgr2=0x7f1df406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1df406b840 0x7f1df406dd00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e140af6d0 tx=0x7f1e04008040 comp rx=0 tx=0).stop 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 msgr2=0x7f1e141eaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e141eaf80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c470 tx=0x7f1dfc001e70 comp rx=0 tx=0).stop 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 shutdown_connections 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1df406b840 0x7f1df406dd00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e140c87a0 0x7f1e141eaf80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e14152480 0x7f1e141eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 --2- 172.21.15.78:0/2803430275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e140c7070 0x7f1e14150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 >> 172.21.15.78:0/2803430275 conn(0x7f1e140ca8a0 msgr2=0x7f1e140c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:05.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 shutdown_connections 2024-02-19T14:52:05.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:05.924+0000 7f1e1a1c7640 1 -- 172.21.15.78:0/2803430275 wait complete. 2024-02-19T14:52:05.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:06.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.040+0000 7f962f1e9640 1 Processor -- start 2024-02-19T14:52:06.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.040+0000 7f962f1e9640 1 -- start start 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962f1e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96281537a0 0x7f96280c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962f1e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9628156860 0x7f9628156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962f1e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96281563b0 con 0x7f962814cd50 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962f1e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9628153cd0 con 0x7f9628156860 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962f1e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962806b6e0 con 0x7f96281537a0 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962d786640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9628156860 0x7f9628156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962d786640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9628156860 0x7f9628156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49832/0 (socket says 172.21.15.78:49832) 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.041+0000 7f962d786640 1 -- 172.21.15.78:0/1300950691 learned_addr learned my addr 172.21.15.78:0/1300950691 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:06.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f9627fff640 1 --2- 172.21.15.78:0/1300950691 >> v2:172.21.15.150:3300/0 conn(0x7f96281537a0 0x7f96280c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f962d786640 1 -- 172.21.15.78:0/1300950691 >> v2:172.21.15.150:3300/0 conn(0x7f96281537a0 msgr2=0x7f96280c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f962d786640 1 --2- 172.21.15.78:0/1300950691 >> v2:172.21.15.150:3300/0 conn(0x7f96281537a0 0x7f96280c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f962d786640 1 -- 172.21.15.78:0/1300950691 >> v1:172.21.15.150:6789/0 conn(0x7f962814cd50 legacy=0x7f96280c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f962d786640 1 -- 172.21.15.78:0/1300950691 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962806b980 con 0x7f9628156860 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f9627fff640 1 --2- 172.21.15.78:0/1300950691 >> v2:172.21.15.150:3300/0 conn(0x7f96281537a0 0x7f96280c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f962d786640 1 --2- 172.21.15.78:0/1300950691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f9628156cd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96280acad0 tx=0x7f961802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcbc782c400c99e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.042+0000 7f96277fe640 1 -- 172.21.15.78:0/1300950691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618031750 con 0x7f9628156860 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f96277fe640 1 -- 172.21.15.78:0/1300950691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96180318b0 con 0x7f9628156860 2024-02-19T14:52:06.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f96277fe640 1 -- 172.21.15.78:0/1300950691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618031c40 con 0x7f9628156860 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 -- 172.21.15.78:0/1300950691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 msgr2=0x7f9628156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1300950691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f9628156cd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f96280acad0 tx=0x7f961802b7f0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 -- 172.21.15.78:0/1300950691 shutdown_connections 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1300950691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f9628156cd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1300950691 >> v2:172.21.15.150:3300/0 conn(0x7f96281537a0 0x7f96280c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 -- 172.21.15.78:0/1300950691 >> 172.21.15.78:0/1300950691 conn(0x7f9628066e70 msgr2=0x7f96280a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 -- 172.21.15.78:0/1300950691 shutdown_connections 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.043+0000 7f962f1e9640 1 -- 172.21.15.78:0/1300950691 wait complete. 2024-02-19T14:52:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.044+0000 7f962f1e9640 1 Processor -- start 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.044+0000 7f962f1e9640 1 -- start start 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962f1e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f962814cd50 0x7f96281dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962f1e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96281537a0 0x7f96281ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962f1e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f96281dd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962f1e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281508d0 con 0x7f9628156860 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962f1e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280c1d40 con 0x7f96281537a0 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962f1e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9628154a10 con 0x7f962814cd50 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962d786640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f96281dd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962d786640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f96281dd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49844/0 (socket says 172.21.15.78:49844) 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.045+0000 7f962d786640 1 -- 172.21.15.78:0/1548867375 learned_addr learned my addr 172.21.15.78:0/1548867375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f9627fff640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96281537a0 0x7f96281ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962d786640 1 -- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f962814cd50 msgr2=0x7f96281dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962cf85640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f962814cd50 0x7f96281dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962d786640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f962814cd50 0x7f96281dc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962d786640 1 -- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96281537a0 msgr2=0x7f96281ebd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962d786640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96281537a0 0x7f96281ebd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962d786640 1 -- 172.21.15.78:0/1548867375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96180094d0 con 0x7f9628156860 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f9627fff640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96281537a0 0x7f96281ebd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962d786640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f96281dd170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f961802b830 tx=0x7f96180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618033c80 con 0x7f9628156860 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618033de0 con 0x7f9628156860 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96281ec620 con 0x7f9628156860 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961803c640 con 0x7f9628156860 2024-02-19T14:52:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.046+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96281504f0 con 0x7f9628156860 2024-02-19T14:52:06.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.047+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962806d000 con 0x7f9628156860 2024-02-19T14:52:06.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.048+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f961803a040 con 0x7f9628156860 2024-02-19T14:52:06.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.048+0000 7f96257fa640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f961406b810 0x7f961406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.048+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f96180ac1d0 con 0x7f9628156860 2024-02-19T14:52:06.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.048+0000 7f962cf85640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f961406b810 0x7f961406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.048+0000 7f962cf85640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f961406b810 0x7f961406dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9620007bf0 tx=0x7f9620007240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.052+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f962806d000 con 0x7f9628156860 2024-02-19T14:52:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.228+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f962806d000 con 0x7f9628156860 2024-02-19T14:52:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.230+0000 7f96257fa640 1 -- 172.21.15.78:0/1548867375 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f962806d000 con 0x7f9628156860 2024-02-19T14:52:06.233 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:06.233 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f961406b810 msgr2=0x7f961406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f961406b810 0x7f961406dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9620007bf0 tx=0x7f9620007240 comp rx=0 tx=0).stop 2024-02-19T14:52:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 msgr2=0x7f96281dd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f96281dd170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f961802b830 tx=0x7f96180046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 shutdown_connections 2024-02-19T14:52:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f961406b810 0x7f961406dcd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9628156860 0x7f96281dd170 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96281537a0 0x7f96281ebd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 --2- 172.21.15.78:0/1548867375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f962814cd50 0x7f96281dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 >> 172.21.15.78:0/1548867375 conn(0x7f9628066e70 msgr2=0x7f9628151550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 shutdown_connections 2024-02-19T14:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.232+0000 7f962f1e9640 1 -- 172.21.15.78:0/1548867375 wait complete. 2024-02-19T14:52:06.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:06.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.366+0000 7f29802c7640 1 Processor -- start 2024-02-19T14:52:06.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 -- start start 2024-02-19T14:52:06.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f297814d210 0x7f29780c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f29780719e0 0x7f29780c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2978070270 con 0x7f29780719e0 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29780c3970 con 0x7f297814d210 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.367+0000 7f29802c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297806ff90 con 0x7f29780c3d90 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f29780719e0 0x7f29780c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f29780719e0 0x7f29780c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49848/0 (socket says 172.21.15.78:49848) 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 -- 172.21.15.78:0/3686995272 learned_addr learned my addr 172.21.15.78:0/3686995272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297e864640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297e063640 1 --2- 172.21.15.78:0/3686995272 >> v2:172.21.15.150:3300/0 conn(0x7f297814d210 0x7f29780c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 -- 172.21.15.78:0/3686995272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 msgr2=0x7f29780c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 -- 172.21.15.78:0/3686995272 >> v2:172.21.15.150:3300/0 conn(0x7f297814d210 msgr2=0x7f29780c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:06.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 --2- 172.21.15.78:0/3686995272 >> v2:172.21.15.150:3300/0 conn(0x7f297814d210 0x7f29780c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297d862640 1 -- 172.21.15.78:0/3686995272 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978156fc0 con 0x7f29780719e0 2024-02-19T14:52:06.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297e864640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:06.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.368+0000 7f297e063640 1 --2- 172.21.15.78:0/3686995272 >> v2:172.21.15.150:3300/0 conn(0x7f297814d210 0x7f29780c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.369+0000 7f297d862640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29780719e0 0x7f29780c64d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29780c5de0 tx=0x7f296802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f07f84b4c6f616 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.369+0000 7f297d061640 1 -- 172.21.15.78:0/3686995272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29680317e0 con 0x7f29780719e0 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.369+0000 7f297d061640 1 -- 172.21.15.78:0/3686995272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2968031940 con 0x7f29780719e0 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.369+0000 7f297d061640 1 -- 172.21.15.78:0/3686995272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968031cd0 con 0x7f29780719e0 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 -- 172.21.15.78:0/3686995272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29780719e0 msgr2=0x7f29780c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29780719e0 0x7f29780c64d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29780c5de0 tx=0x7f296802b880 comp rx=0 tx=0).stop 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 -- 172.21.15.78:0/3686995272 shutdown_connections 2024-02-19T14:52:06.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 --2- 172.21.15.78:0/3686995272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29780719e0 0x7f29780c64d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 --2- 172.21.15.78:0/3686995272 >> v2:172.21.15.150:3300/0 conn(0x7f297814d210 0x7f29780c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 -- 172.21.15.78:0/3686995272 >> 172.21.15.78:0/3686995272 conn(0x7f29780c5050 msgr2=0x7f29780c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 -- 172.21.15.78:0/3686995272 shutdown_connections 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.370+0000 7f29802c7640 1 -- 172.21.15.78:0/3686995272 wait complete. 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 Processor -- start 2024-02-19T14:52:06.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 -- start start 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29780719e0 0x7f29781ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 0x7f29780ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781df1e0 con 0x7f297814d210 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781ec480 con 0x7f29780719e0 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.371+0000 7f29802c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29781ec760 con 0x7f29780c3d90 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 0x7f29780ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 0x7f29780ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49864/0 (socket says 172.21.15.78:49864) 2024-02-19T14:52:06.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 -- 172.21.15.78:0/304699831 learned_addr learned my addr 172.21.15.78:0/304699831 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e063640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29780719e0 0x7f29781ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297d862640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 -- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 msgr2=0x7f29780abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 -- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29780719e0 msgr2=0x7f29781ebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29780719e0 0x7f29781ebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 -- 172.21.15.78:0/304699831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29680094d0 con 0x7f297814d210 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e063640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29780719e0 0x7f29781ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f297e864640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 0x7f29780ac300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f297000cdf0 tx=0x7f297000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970003b90 con 0x7f297814d210 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970003d10 con 0x7f297814d210 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970004260 con 0x7f297814d210 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29780c3660 con 0x7f297814d210 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.372+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29781ec180 con 0x7f297814d210 2024-02-19T14:52:06.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.373+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29781deb40 con 0x7f297814d210 2024-02-19T14:52:06.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.374+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f29700043c0 con 0x7f297814d210 2024-02-19T14:52:06.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.374+0000 7f296effd640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295c06b840 0x7f295c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.374+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f297008ad50 con 0x7f297814d210 2024-02-19T14:52:06.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.374+0000 7f297e063640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295c06b840 0x7f295c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.375+0000 7f297e063640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295c06b840 0x7f295c06dd00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f297400a990 tx=0x7f2974008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.383+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29781deb40 con 0x7f297814d210 2024-02-19T14:52:06.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.589+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29781ec920 con 0x7f297814d210 2024-02-19T14:52:06.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.590+0000 7f296effd640 1 -- 172.21.15.78:0/304699831 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4940 (secure 0 0 0) 0x7f29781ec920 con 0x7f297814d210 2024-02-19T14:52:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:06.593 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:04.685406+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-19T14:52:06.593 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2024-02-19T14:52:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295c06b840 msgr2=0x7f295c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295c06b840 0x7f295c06dd00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f297400a990 tx=0x7f2974008040 comp rx=0 tx=0).stop 2024-02-19T14:52:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 msgr2=0x7f29780ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 0x7f29780ac300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f297000cdf0 tx=0x7f297000a710 comp rx=0 tx=0).stop 2024-02-19T14:52:06.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 shutdown_connections 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295c06b840 0x7f295c06dd00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f297814d210 0x7f29780ac300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29780c3d90 0x7f29780abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.591+0000 7f29802c7640 1 --2- 172.21.15.78:0/304699831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29780719e0 0x7f29781ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.592+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 >> 172.21.15.78:0/304699831 conn(0x7f29780c5050 msgr2=0x7f29780c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.592+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 shutdown_connections 2024-02-19T14:52:06.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.592+0000 7f29802c7640 1 -- 172.21.15.78:0/304699831 wait complete. 2024-02-19T14:52:06.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:06.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.703+0000 7f57fe447640 1 Processor -- start 2024-02-19T14:52:06.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.703+0000 7f57fe447640 1 -- start start 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fe447640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f80c8e00 0x7f57f806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fe447640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f57f806d910 0x7f57f806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fe447640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f806cd20 con 0x7f57f8152700 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fe447640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f81551f0 con 0x7f57f806d910 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fe447640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f8155490 con 0x7f57f80c8e00 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f80c8e00 0x7f57f806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fc9e4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f57f806d910 0x7f57f806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fc9e4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f57f806d910 0x7f57f806c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57788/0 (socket says 172.21.15.78:57788) 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57fc9e4640 1 -- 172.21.15.78:0/1758568507 learned_addr learned my addr 172.21.15.78:0/1758568507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.704+0000 7f57f6ffd640 1 -- 172.21.15.78:0/1758568507 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298582650 0 0) 0x7f57f806cd20 con 0x7f57f8152700 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.705+0000 7f57f6ffd640 1 -- 172.21.15.78:0/1758568507 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e0003520 con 0x7f57f8152700 2024-02-19T14:52:06.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.705+0000 7f57fc9e4640 1 -- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f80c8e00 msgr2=0x7f57f806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.705+0000 7f57fc9e4640 1 --2- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f80c8e00 0x7f57f806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.705+0000 7f57fc9e4640 1 -- 172.21.15.78:0/1758568507 >> v1:172.21.15.150:6789/0 conn(0x7f57f8152700 legacy=0x7f57f80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.705+0000 7f57fc9e4640 1 -- 172.21.15.78:0/1758568507 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f8155770 con 0x7f57f806d910 2024-02-19T14:52:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.705+0000 7f57f77fe640 1 --2- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f80c8e00 0x7f57f806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.706+0000 7f57fc9e4640 1 --2- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f806d910 0x7f57f806c7e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e50 tx=0x7f57f002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aefee3e8045059c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.706+0000 7f57f6ffd640 1 -- 172.21.15.78:0/1758568507 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f002ca70 con 0x7f57f806d910 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.706+0000 7f57f6ffd640 1 -- 172.21.15.78:0/1758568507 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f002cbd0 con 0x7f57f806d910 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.706+0000 7f57f6ffd640 1 -- 172.21.15.78:0/1758568507 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f00343f0 con 0x7f57f806d910 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 -- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f806d910 msgr2=0x7f57f806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 --2- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f806d910 0x7f57f806c7e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e50 tx=0x7f57f002bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 -- 172.21.15.78:0/1758568507 shutdown_connections 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 --2- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f80c8e00 0x7f57f806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 --2- 172.21.15.78:0/1758568507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f806d910 0x7f57f806c7e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 -- 172.21.15.78:0/1758568507 >> 172.21.15.78:0/1758568507 conn(0x7f57f80cac90 msgr2=0x7f57f80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 -- 172.21.15.78:0/1758568507 shutdown_connections 2024-02-19T14:52:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.707+0000 7f57fe447640 1 -- 172.21.15.78:0/1758568507 wait complete. 2024-02-19T14:52:06.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.708+0000 7f57fe447640 1 Processor -- start 2024-02-19T14:52:06.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.709+0000 7f57fe447640 1 -- start start 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f806d910 0x7f57f81ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 0x7f57f8148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f8152700 0x7f57f81eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f80aa670 con 0x7f57f80c8e00 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f81ee730 con 0x7f57f8152700 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f80acc50 con 0x7f57f806d910 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 0x7f57f8148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 0x7f57f8148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49870/0 (socket says 172.21.15.78:49870) 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 -- 172.21.15.78:0/3340941103 learned_addr learned my addr 172.21.15.78:0/3340941103 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fc9e4640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f8152700 0x7f57f81eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f7fff640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f806d910 0x7f57f81ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 -- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f806d910 msgr2=0x7f57f81ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f806d910 0x7f57f81ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 -- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f8152700 msgr2=0x7f57f81eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f8152700 0x7f57f81eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 -- 172.21.15.78:0/3340941103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec009400 con 0x7f57f80c8e00 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f77fe640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 0x7f57f8148e40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00ec10 tx=0x7f57ec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec009760 con 0x7f57f80c8e00 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec00cc30 con 0x7f57f80c8e00 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f0009b30 con 0x7f57f80c8e00 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.710+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec013470 con 0x7f57f80c8e00 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.711+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f81eade0 con 0x7f57f80c8e00 2024-02-19T14:52:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.711+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f8074990 con 0x7f57f80c8e00 2024-02-19T14:52:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.712+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f57ec0135d0 con 0x7f57f80c8e00 2024-02-19T14:52:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.712+0000 7f57f4ff9640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57e406b840 0x7f57e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.712+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57ec08a6e0 con 0x7f57f80c8e00 2024-02-19T14:52:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.712+0000 7f57f7fff640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57e406b840 0x7f57e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.713+0000 7f57f7fff640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57e406b840 0x7f57e406dd00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f57e80011e0 tx=0x7f57e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.716+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57ec054f00 con 0x7f57f80c8e00 2024-02-19T14:52:06.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.892+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57f80bf540 con 0x7f57f80c8e00 2024-02-19T14:52:06.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.893+0000 7f57f4ff9640 1 -- 172.21.15.78:0/3340941103 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f57ec0585c0 con 0x7f57f80c8e00 2024-02-19T14:52:06.896 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:06.896 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:06.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57e406b840 msgr2=0x7f57e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57e406b840 0x7f57e406dd00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f57e80011e0 tx=0x7f57e8009040 comp rx=0 tx=0).stop 2024-02-19T14:52:06.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 msgr2=0x7f57f8148e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:06.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 0x7f57f8148e40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00ec10 tx=0x7f57ec00c520 comp rx=0 tx=0).stop 2024-02-19T14:52:06.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 shutdown_connections 2024-02-19T14:52:06.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57e406b840 0x7f57e406dd00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80c8e00 0x7f57f8148e40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57f8152700 0x7f57f81eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 --2- 172.21.15.78:0/3340941103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57f806d910 0x7f57f81ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:06.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.895+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 >> 172.21.15.78:0/3340941103 conn(0x7f57f80cac90 msgr2=0x7f57f80cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:06.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.896+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 shutdown_connections 2024-02-19T14:52:06.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:06.896+0000 7f57fe447640 1 -- 172.21.15.78:0/3340941103 wait complete. 2024-02-19T14:52:07.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:51:59.591061+0000', 'modified': '2024-02-19T14:52:04.685406+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/1940908201', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 1940908201}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1940908201}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-02-19T14:52:07.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:52:07.312 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:52:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.412+0000 7f12f9ec0640 1 Processor -- start 2024-02-19T14:52:07.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.412+0000 7f12f9ec0640 1 -- start start 2024-02-19T14:52:07.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f9ec0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12f414f1a0 0x7f12f4072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f9ec0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f4152680 0x7f12f40c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f9ec0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f40c5970 con 0x7f12f406ea50 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f9ec0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f4152af0 con 0x7f12f414f1a0 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f9ec0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40c5750 con 0x7f12f4152680 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12f414f1a0 0x7f12f4072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12f414f1a0 0x7f12f4072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49878/0 (socket says 172.21.15.78:49878) 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 -- 172.21.15.78:0/1036660673 learned_addr learned my addr 172.21.15.78:0/1036660673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f3fff640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f4152680 0x7f12f40c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 -- 172.21.15.78:0/1036660673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f4152680 msgr2=0x7f12f40c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f4152680 0x7f12f40c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 -- 172.21.15.78:0/1036660673 >> v1:172.21.15.150:6789/0 conn(0x7f12f406ea50 legacy=0x7f12f406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f37fe640 1 -- 172.21.15.78:0/1036660673 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40702d0 con 0x7f12f414f1a0 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.413+0000 7f12f3fff640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f4152680 0x7f12f40c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.414+0000 7f12f37fe640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f414f1a0 0x7f12f4072d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009e50 tx=0x7f12e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c7b3cb68bc77dc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.414+0000 7f12f27fc640 1 -- 172.21.15.78:0/1036660673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e402c9e0 con 0x7f12f414f1a0 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.414+0000 7f12f27fc640 1 -- 172.21.15.78:0/1036660673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e402cb40 con 0x7f12f414f1a0 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.414+0000 7f12f27fc640 1 -- 172.21.15.78:0/1036660673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40343f0 con 0x7f12f414f1a0 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 -- 172.21.15.78:0/1036660673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f414f1a0 msgr2=0x7f12f4072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f414f1a0 0x7f12f4072d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f12e4009e50 tx=0x7f12e402be40 comp rx=0 tx=0).stop 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 -- 172.21.15.78:0/1036660673 shutdown_connections 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f4152680 0x7f12f40c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/1036660673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f414f1a0 0x7f12f4072d70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 -- 172.21.15.78:0/1036660673 >> 172.21.15.78:0/1036660673 conn(0x7f12f406c170 msgr2=0x7f12f406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 -- 172.21.15.78:0/1036660673 shutdown_connections 2024-02-19T14:52:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.415+0000 7f12f9ec0640 1 -- 172.21.15.78:0/1036660673 wait complete. 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.416+0000 7f12f9ec0640 1 Processor -- start 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.416+0000 7f12f9ec0640 1 -- start start 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f9ec0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f406ea50 0x7f12f41e7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f9ec0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f414f1a0 0x7f12f4145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f9ec0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 0x7f12f41eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f9ec0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f4149d00 con 0x7f12f4152680 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f9ec0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41eae40 con 0x7f12f406ea50 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f9ec0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40c46a0 con 0x7f12f414f1a0 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 0x7f12f41eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 0x7f12f41eb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49886/0 (socket says 172.21.15.78:49886) 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 -- 172.21.15.78:0/13348322 learned_addr learned my addr 172.21.15.78:0/13348322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f37fe640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f406ea50 0x7f12f41e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f414f1a0 0x7f12f4145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 -- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f414f1a0 msgr2=0x7f12f4145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f414f1a0 0x7f12f4145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 -- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f406ea50 msgr2=0x7f12f41e7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f406ea50 0x7f12f41e7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.417+0000 7f12f3fff640 1 -- 172.21.15.78:0/13348322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4009b30 con 0x7f12f4152680 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.418+0000 7f12f3fff640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 0x7f12f41eb260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c7f0 tx=0x7f12ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.418+0000 7f12ebfff640 1 -- 172.21.15.78:0/13348322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec007820 con 0x7f12f4152680 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.418+0000 7f12ebfff640 1 -- 172.21.15.78:0/13348322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec007980 con 0x7f12f4152680 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.418+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f40c4980 con 0x7f12f4152680 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.418+0000 7f12ebfff640 1 -- 172.21.15.78:0/13348322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec007d70 con 0x7f12f4152680 2024-02-19T14:52:07.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.418+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f4146240 con 0x7f12f4152680 2024-02-19T14:52:07.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.420+0000 7f12ebfff640 1 -- 172.21.15.78:0/13348322 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f12ec0113f0 con 0x7f12f4152680 2024-02-19T14:52:07.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.420+0000 7f12ebfff640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12d406b810 0x7f12d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.420+0000 7f12ebfff640 1 -- 172.21.15.78:0/13348322 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12ec08a770 con 0x7f12f4152680 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.420+0000 7f12f37fe640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12d406b810 0x7f12d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.420+0000 7f12fcb72740 1 Processor -- start 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 -- start start 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12f37fe640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12d406b810 0x7f12d406dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12e4035000 tx=0x7f12e4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa1d0880 0x55f2aa20d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa20a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa217680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa18bcd0 con 0x55f2aa1d0c60 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa13ef20 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa13c450 con 0x55f2aa1d0880 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa20a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa20a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49896/0 (socket says 172.21.15.78:49896) 2024-02-19T14:52:07.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.421+0000 7f12f2ffd640 1 -- 172.21.15.78:0/1943678871 learned_addr learned my addr 172.21.15.78:0/1943678871 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f37fe640 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa1d0880 0x55f2aa20d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f3fff640 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa217680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f2ffd640 1 -- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa1d0880 msgr2=0x55f2aa20d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa1d0880 0x55f2aa20d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f2ffd640 1 -- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 msgr2=0x55f2aa217680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa217680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f2ffd640 1 -- 172.21.15.78:0/1943678871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2aa0ecf20 con 0x55f2aa1d0c60 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa20a8c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0015f0 tx=0x7f12dc007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2c1b84e2d6c902 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12e97fa640 1 -- 172.21.15.78:0/1943678871 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0050d0 con 0x55f2aa1d0c60 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12e97fa640 1 -- 172.21.15.78:0/1943678871 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc005230 con 0x55f2aa1d0c60 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.422+0000 7f12e97fa640 1 -- 172.21.15.78:0/1943678871 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00f7b0 con 0x55f2aa1d0c60 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 -- 172.21.15.78:0/1943678871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 msgr2=0x55f2aa20a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa20a8c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0015f0 tx=0x7f12dc007e50 comp rx=0 tx=0).stop 2024-02-19T14:52:07.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 -- 172.21.15.78:0/1943678871 shutdown_connections 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa20a8c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa217680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 --2- 172.21.15.78:0/1943678871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa1d0880 0x55f2aa20d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 -- 172.21.15.78:0/1943678871 >> 172.21.15.78:0/1943678871 conn(0x55f2aa1cd120 msgr2=0x55f2aa1f1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 -- 172.21.15.78:0/1943678871 shutdown_connections 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 -- 172.21.15.78:0/1943678871 wait complete. 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 -- start start 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.423+0000 7f12fcb72740 1 Processor -- start 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa206eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2043b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa1ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa137c30 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa0dcd30 con 0x55f2aa1d0c60 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2a9e0b310 con 0x55f2aa2091b0 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12fcb72740 10 client.0 _ll_register_callbacks cb 0x55f2aa0e4d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:52:07.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2043b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49910/0 (socket says 172.21.15.78:49910) 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12f2ffd640 1 -- 172.21.15.78:0/128751372 learned_addr learned my addr 172.21.15.78:0/128751372 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12f37fe640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa206eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.424+0000 7f12f3fff640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa1ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12f2ffd640 1 -- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 msgr2=0x55f2aa1ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa1ee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12f2ffd640 1 -- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 msgr2=0x55f2aa206eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa206eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12f2ffd640 1 -- 172.21.15.78:0/128751372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f40c5970 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2043b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f12dc014000 tx=0x7f12dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d415c242722e4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12d27fc640 1 -- 172.21.15.78:0/128751372 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0050d0 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12d27fc640 1 -- 172.21.15.78:0/128751372 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12dc000f70 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2a9e376c0 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12d27fc640 1 -- 172.21.15.78:0/128751372 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12dc007020 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 10 client.4431 fetch_fsmap learned FSMap version 6 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55f2aa1362b0 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12d27fc640 1 -- 172.21.15.78:0/128751372 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f2a9e376c0 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12d27fc640 1 -- 172.21.15.78:0/128751372 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 6) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f12dc012070 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 10 client.4431 fetch_fsmap finished waiting for FSMap version 6 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 10 client.4431 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 20 client.4431 populate_metadata read hostname 'smithi078' 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa222000 0x55f2aa2223e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 4 client.4431 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.425+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 --> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2a9e88270 con 0x55f2aa222000 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.426+0000 7f12f37fe640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa222000 0x55f2aa2223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.426+0000 7f12d27fc640 10 client.4431 ms_handle_connect on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.426+0000 7f12f37fe640 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa222000 0x55f2aa2223e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20b505f3965ff7bc server_cookie=edf35cab50c8524a in_seq=0 out_seq=0 2024-02-19T14:52:07.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.426+0000 7f12d27fc640 1 -- 172.21.15.78:0/128751372 <== mds.0 v2:172.21.15.78:6836/1940908201 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f2a9e88270 con 0x55f2aa222000 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.426+0000 7f12d27fc640 10 client.4431 handle_command_reply: tid=0 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.426+0000 7f12fcb72740 1 client.4431 shutdown 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 msgr2=0x55f2aa2043b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2043b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f12dc014000 tx=0x7f12dc000bc0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 shutdown_connections 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa222000 0x55f2aa2223e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2043b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa1ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 --2- 172.21.15.78:0/128751372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa206eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 >> 172.21.15.78:0/128751372 conn(0x55f2aa1d0880 msgr2=0x55f2aa20b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12d27fc640 0 client.4431 ms_handle_reset on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 shutdown_connections 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 1 -- 172.21.15.78:0/128751372 wait complete. 2024-02-19T14:52:07.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.427+0000 7f12fcb72740 20 client.4431 trim_cache size 0 max 16384 2024-02-19T14:52:07.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.458+0000 7f12fcb72740 1 Processor -- start 2024-02-19T14:52:07.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.458+0000 7f12fcb72740 1 -- start start 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa207e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2775c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa277b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12fcb72740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa0bd490 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa228f10 con 0x55f2aa1d0c60 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa1ee380 con 0x55f2aa2091b0 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2775c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2775c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59112/0 (socket says 172.21.15.78:59112) 2024-02-19T14:52:07.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.459+0000 7f12f2ffd640 1 -- 172.21.15.78:0/788253975 learned_addr learned my addr 172.21.15.78:0/788253975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f3fff640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa277b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f37fe640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa207e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f2ffd640 1 -- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 msgr2=0x55f2aa277b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa277b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f2ffd640 1 -- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 msgr2=0x55f2aa207e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa207e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f2ffd640 1 -- 172.21.15.78:0/788253975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2aa0d2a20 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f3fff640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa277b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f37fe640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa207e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2775c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f12dc007a50 tx=0x7f12dc01f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e0800ab46e100e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12e8ff9640 1 -- 172.21.15.78:0/788253975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc0050d0 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12e8ff9640 1 -- 172.21.15.78:0/788253975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc021070 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.460+0000 7f12e8ff9640 1 -- 172.21.15.78:0/788253975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc000f70 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 -- 172.21.15.78:0/788253975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 msgr2=0x55f2aa2775c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2775c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f12dc007a50 tx=0x7f12dc01f070 comp rx=0 tx=0).stop 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 -- 172.21.15.78:0/788253975 shutdown_connections 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa2775c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa277b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 --2- 172.21.15.78:0/788253975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d0c60 0x55f2aa207e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 -- 172.21.15.78:0/788253975 >> 172.21.15.78:0/788253975 conn(0x55f2aa1d0880 msgr2=0x55f2aa1ee770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 -- 172.21.15.78:0/788253975 shutdown_connections 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 -- 172.21.15.78:0/788253975 wait complete. 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 -- start start 2024-02-19T14:52:07.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.461+0000 7f12fcb72740 1 Processor -- start 2024-02-19T14:52:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa223420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa231f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa21b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa277d50 con 0x55f2aa1d0c60 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa223d70 con 0x55f2aa1d1ce0 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2aa232480 con 0x55f2aa2091b0 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12fcb72740 10 client.0 _ll_register_callbacks cb 0x7f12dc007cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa223420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa223420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59120/0 (socket says 172.21.15.78:59120) 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 -- 172.21.15.78:0/3139950556 learned_addr learned my addr 172.21.15.78:0/3139950556 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa231f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f3fff640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa21b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 -- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 msgr2=0x55f2aa21b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa21b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 -- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 msgr2=0x55f2aa231f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa231f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f37fe640 1 -- 172.21.15.78:0/3139950556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2aa1362b0 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.462+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa231f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12f37fe640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa223420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f12e402c450 tx=0x7f12e402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6335ae43294385 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12d27fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12d27fc640 1 -- 172.21.15.78:0/3139950556 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4053070 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12d27fc640 1 -- 172.21.15.78:0/3139950556 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12e403f440 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12dc005230 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12d27fc640 1 -- 172.21.15.78:0/3139950556 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12e4034e90 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 10 client.4437 fetch_fsmap learned FSMap version 6 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55f2aa0f8780 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12d27fc640 1 -- 172.21.15.78:0/3139950556 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc005230 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12d27fc640 1 -- 172.21.15.78:0/3139950556 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 6) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f12e404b040 con 0x55f2aa1d0c60 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 10 client.4437 fetch_fsmap finished waiting for FSMap version 6 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 10 client.4437 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 20 client.4437 populate_metadata read hostname 'smithi078' 2024-02-19T14:52:07.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa274170 0x55f2aa283710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 4 client.4437 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 --> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2a9e74610 con 0x55f2aa274170 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.463+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa274170 0x55f2aa283710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12d27fc640 10 client.4437 ms_handle_connect on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12f2ffd640 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa274170 0x55f2aa283710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6295601af1ab58 server_cookie=4be376cd558cd458 in_seq=0 out_seq=0 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12d27fc640 1 -- 172.21.15.78:0/3139950556 <== mds.0 v2:172.21.15.78:6836/1940908201 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55f2a9e74610 con 0x55f2aa274170 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12d27fc640 10 client.4437 handle_command_reply: tid=0 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 client.4437 shutdown 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 msgr2=0x55f2aa223420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa223420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f12e402c450 tx=0x7f12e402e070 comp rx=0 tx=0).stop 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 shutdown_connections 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x55f2aa274170 0x55f2aa283710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55f2aa2091b0 0x55f2aa21b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12d27fc640 0 client.4437 ms_handle_reset on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f2aa1d1ce0 0x55f2aa231f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 --2- 172.21.15.78:0/3139950556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2aa1d0c60 0x55f2aa223420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 >> 172.21.15.78:0/3139950556 conn(0x55f2aa1d0880 msgr2=0x55f2aa1f17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 shutdown_connections 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 1 -- 172.21.15.78:0/3139950556 wait complete. 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.464+0000 7f12fcb72740 20 client.4437 trim_cache size 0 max 16384 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4290, 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:52:07.474 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 7.2115864820000004, 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 6, 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 24, 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 24, 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 13.828443804000001 2024-02-19T14:52:07.475 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12d406b810 msgr2=0x7f12d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12d406b810 0x7f12d406dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12e4035000 tx=0x7f12e4030000 comp rx=0 tx=0).stop 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 msgr2=0x7f12f41eb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 0x7f12f41eb260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c7f0 tx=0x7f12ec00ccb0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 shutdown_connections 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12d406b810 0x7f12d406dcd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12f4152680 0x7f12f41eb260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12f414f1a0 0x7f12f4145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 --2- 172.21.15.78:0/13348322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f406ea50 0x7f12f41e7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 >> 172.21.15.78:0/13348322 conn(0x7f12f406c170 msgr2=0x7f12f406fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 shutdown_connections 2024-02-19T14:52:07.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.465+0000 7f12f9ec0640 1 -- 172.21.15.78:0/13348322 wait complete. 2024-02-19T14:52:07.481 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T14:52:07.481 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:07.481 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T14:52:07.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.579+0000 7fba52381640 1 Processor -- start 2024-02-19T14:52:07.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.579+0000 7fba52381640 1 -- start start 2024-02-19T14:52:07.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.580+0000 7fba52381640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fba4c06bb00 0x7fba4c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba52381640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba4c0721a0 0x7fba4c0c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba52381640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c156e80 con 0x7fba4c150560 2024-02-19T14:52:07.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba52381640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4c06ffc0 con 0x7fba4c06bb00 2024-02-19T14:52:07.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba52381640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4c072d70 con 0x7fba4c0721a0 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fba4c06bb00 0x7fba4c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fba4c150560 0x7fba4c0c77d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39270/0 (socket says 172.21.15.78:39270) 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4bfff640 1 -- 172.21.15.78:0/2439386667 learned_addr learned my addr 172.21.15.78:0/2439386667 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fba4c06bb00 0x7fba4c0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59134/0 (socket says 172.21.15.78:59134) 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba5091e640 1 --2- 172.21.15.78:0/2439386667 >> v2:172.21.15.150:3300/0 conn(0x7fba4c0721a0 0x7fba4c0c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4b7fe640 1 -- 172.21.15.78:0/2439386667 >> v2:172.21.15.150:3300/0 conn(0x7fba4c0721a0 msgr2=0x7fba4c0c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4b7fe640 1 --2- 172.21.15.78:0/2439386667 >> v2:172.21.15.150:3300/0 conn(0x7fba4c0721a0 0x7fba4c0c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4affd640 1 -- 172.21.15.78:0/2439386667 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266171593 0 0) 0x7fba4c156e80 con 0x7fba4c150560 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4b7fe640 1 -- 172.21.15.78:0/2439386667 >> v1:172.21.15.78:6789/0 conn(0x7fba4c150560 legacy=0x7fba4c0c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba4b7fe640 1 -- 172.21.15.78:0/2439386667 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c073050 con 0x7fba4c06bb00 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.581+0000 7fba5091e640 1 --2- 172.21.15.78:0/2439386667 >> v2:172.21.15.150:3300/0 conn(0x7fba4c0721a0 0x7fba4c0c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba4b7fe640 1 --2- 172.21.15.78:0/2439386667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c06bb00 0x7fba4c0c6080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fba4c06b810 tx=0x7fba4002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d754e7afb11d763e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba4affd640 1 -- 172.21.15.78:0/2439386667 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4002e070 con 0x7fba4c06bb00 2024-02-19T14:52:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba4affd640 1 -- 172.21.15.78:0/2439386667 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba400315a0 con 0x7fba4c06bb00 2024-02-19T14:52:07.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba4affd640 1 -- 172.21.15.78:0/2439386667 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40031930 con 0x7fba4c06bb00 2024-02-19T14:52:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 -- 172.21.15.78:0/2439386667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c06bb00 msgr2=0x7fba4c0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 --2- 172.21.15.78:0/2439386667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c06bb00 0x7fba4c0c6080 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fba4c06b810 tx=0x7fba4002b840 comp rx=0 tx=0).stop 2024-02-19T14:52:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 -- 172.21.15.78:0/2439386667 shutdown_connections 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 --2- 172.21.15.78:0/2439386667 >> v2:172.21.15.150:3300/0 conn(0x7fba4c0721a0 0x7fba4c0c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 --2- 172.21.15.78:0/2439386667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c06bb00 0x7fba4c0c6080 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 -- 172.21.15.78:0/2439386667 >> 172.21.15.78:0/2439386667 conn(0x7fba4c06c170 msgr2=0x7fba4c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 -- 172.21.15.78:0/2439386667 shutdown_connections 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.582+0000 7fba52381640 1 -- 172.21.15.78:0/2439386667 wait complete. 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.583+0000 7fba52381640 1 Processor -- start 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 -- start start 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba4c06bb00 0x7fba4c1eb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba4c0721a0 0x7fba4c1422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 0x7fba4c0c20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c1561c0 con 0x7fba4c150560 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c1430d0 con 0x7fba4c06bb00 2024-02-19T14:52:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba52381640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c1eb950 con 0x7fba4c0721a0 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba5091e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 0x7fba4c0c20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba5091e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 0x7fba4c0c20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59142/0 (socket says 172.21.15.78:59142) 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.584+0000 7fba5091e640 1 -- 172.21.15.78:0/402794869 learned_addr learned my addr 172.21.15.78:0/402794869 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba4bfff640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba4c06bb00 0x7fba4c1eb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba4b7fe640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba4c0721a0 0x7fba4c1422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba5091e640 1 -- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba4c0721a0 msgr2=0x7fba4c1422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba5091e640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba4c0721a0 0x7fba4c1422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba5091e640 1 -- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba4c06bb00 msgr2=0x7fba4c1eb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba5091e640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba4c06bb00 0x7fba4c1eb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba5091e640 1 -- 172.21.15.78:0/402794869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40009400 con 0x7fba4c150560 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba4bfff640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba4c06bb00 0x7fba4c1eb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba5091e640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 0x7fba4c0c20d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba4400cf20 tx=0x7fba4400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4400d840 con 0x7fba4c150560 2024-02-19T14:52:07.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4400d9a0 con 0x7fba4c150560 2024-02-19T14:52:07.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c0c47d0 con 0x7fba4c150560 2024-02-19T14:52:07.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4400dd50 con 0x7fba4c150560 2024-02-19T14:52:07.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.585+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c142a80 con 0x7fba4c150560 2024-02-19T14:52:07.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.586+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c0c1d90 con 0x7fba4c150560 2024-02-19T14:52:07.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.587+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fba4401d360 con 0x7fba4c150560 2024-02-19T14:52:07.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.587+0000 7fba48ff9640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba3806b810 0x7fba3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:07.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.587+0000 7fba4bfff640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba3806b810 0x7fba3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:07.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.587+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fba4408aef0 con 0x7fba4c150560 2024-02-19T14:52:07.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.588+0000 7fba4bfff640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba3806b810 0x7fba3806dcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba3c000f60 tx=0x7fba3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:07.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.591+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba44055720 con 0x7fba4c150560 2024-02-19T14:52:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.804+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fba4c074990 con 0x7fba4c150560 2024-02-19T14:52:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.804+0000 7fba48ff9640 1 -- 172.21.15.78:0/402794869 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7fba4401a020 con 0x7fba4c150560 2024-02-19T14:52:07.807 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-19T14:52:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba3806b810 msgr2=0x7fba3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba3806b810 0x7fba3806dcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba3c000f60 tx=0x7fba3c009040 comp rx=0 tx=0).stop 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 msgr2=0x7fba4c0c20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 0x7fba4c0c20d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba4400cf20 tx=0x7fba4400a7d0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 shutdown_connections 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba3806b810 0x7fba3806dcd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4c150560 0x7fba4c0c20d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba4c0721a0 0x7fba4c1422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 --2- 172.21.15.78:0/402794869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba4c06bb00 0x7fba4c1eb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 >> 172.21.15.78:0/402794869 conn(0x7fba4c06c170 msgr2=0x7fba4c14ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 shutdown_connections 2024-02-19T14:52:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:07.806+0000 7fba52381640 1 -- 172.21.15.78:0/402794869 wait complete. 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T14:52:07.819 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T14:52:07.820 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-02-19T14:52:07.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:07.820 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42687sec preferred_lft 42687sec 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-02-19T14:52:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:07.837 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T14:52:07.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T14:52:07.837 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:07.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:07.992 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T14:52:08.019 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-02-19T14:52:08.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.021 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-02-19T14:52:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-02-19T14:52:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T14:52:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T14:52:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T14:52:08.089 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T14:52:08.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.090 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:08.090 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:08.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T14:52:08.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T14:52:08.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T14:52:08.090 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:08.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.356 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-02-19T14:52:08.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.371 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-02-19T14:52:08.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.427 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:08.427 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:08.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-19T14:52:08.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-02-19T14:52:08.427 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:08.516 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-02-19T14:52:08.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-02-19T14:52:08.517 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:08.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.622 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:08.622 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:08.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-02-19T14:52:08.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T14:52:08.622 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:08.665 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-02-19T14:52:08.685 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T14:52:08.685 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:08.720 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-02-19T14:52:08.720 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T14:52:08.720 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:08.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-02-19T14:52:08.847 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-02-19T14:52:08.903 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-02-19T14:52:08.903 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-02-19T14:52:08.903 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-02-19T14:52:08.903 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-02-19T14:52:08.903 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-02-19T14:52:08.904 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-0 2024-02-19T14:52:08.905 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-1 2024-02-19T14:52:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-2 2024-02-19T14:52:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-3 2024-02-19T14:52:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-02-19T14:52:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-19T14:52:08.906 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-19T14:52:08.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:08.907 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-02-19T14:52:08.957 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-02-19T14:52:08.958 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-02-19T14:52:08.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-02-19T14:52:09.026 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-02-19T14:52:09.082 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-02-19T14:52:09.083 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-0 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-1 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-2 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-3 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-19T14:52:09.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:09.084 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-02-19T14:52:09.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:2024-02-19T14:52:09.086+0000 7f305e437480 -1 init, newargv = 0x55fe37869560 newargc=15 2024-02-19T14:52:09.092 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[44948]: starting ceph client 2024-02-19T14:52:09.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[44948]: starting fuse 2024-02-19T14:52:09.106 INFO:teuthology.orchestra.run.smithi078.stdout:43 2024-02-19T14:52:09.106 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-02-19T14:52:09.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-02-19T14:52:09.107 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.admin")) 2024-02-19T14:52:09.108 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-02-19T14:52:09.227 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.admin.44948.asok 2024-02-19T14:52:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-19T14:52:09.237 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44948.asok 2024-02-19T14:52:09.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:09.237 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44948.asok status 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "9f8908eed2b2d7e8c316bd5be7ed6e49024931ee", 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-1451-g9f8908ee (9f8908eed2b2d7e8c316bd5be7ed6e49024931ee) squid (dev)", 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "admin", 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "44948", 2024-02-19T14:52:09.381 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4452, 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4452 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-02-19T14:52:09.382 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3576227116 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3576227116 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.4452 192.168.0.1:0/3576227116", 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/3576227116", 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 6, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 24, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-02-19T14:52:09.383 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-02-19T14:52:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:52:09.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:09.394 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-02-19T14:52:09.394 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:09.409 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-02-19T14:52:09.409 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:09.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:09.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:09.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:09.481 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-02-19T14:52:09.549 INFO:teuthology.orchestra.run.smithi078.stdout:# file: . 2024-02-19T14:52:09.549 INFO:teuthology.orchestra.run.smithi078.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-02-19T14:52:09.549 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:09.549 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:52:09.550 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:09.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:09.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.744+0000 7f95d948f640 1 Processor -- start 2024-02-19T14:52:09.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.744+0000 7f95d948f640 1 -- start start 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d948f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d948f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95d40c5730 0x7f95d406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d948f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95d40c8e00 0x7f95d406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d948f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d406cd20 con 0x7f95d40c8e00 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d948f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d41551f0 con 0x7f95d40c5730 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d948f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4155490 con 0x7f95d4152700 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95d40c8e00 0x7f95d406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95d40c8e00 0x7f95d406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59150/0 (socket says 172.21.15.78:59150) 2024-02-19T14:52:09.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 -- 172.21.15.78:0/861605670 learned_addr learned my addr 172.21.15.78:0/861605670 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95cbfff640 1 --2- 172.21.15.78:0/861605670 >> v2:172.21.15.150:3300/0 conn(0x7f95d40c5730 0x7f95d406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d2ffd640 1 --2- 172.21.15.78:0/861605670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 -- 172.21.15.78:0/861605670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 msgr2=0x7f95d40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 --2- 172.21.15.78:0/861605670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 -- 172.21.15.78:0/861605670 >> v2:172.21.15.150:3300/0 conn(0x7f95d40c5730 msgr2=0x7f95d406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 --2- 172.21.15.78:0/861605670 >> v2:172.21.15.150:3300/0 conn(0x7f95d40c5730 0x7f95d406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.745+0000 7f95d37fe640 1 -- 172.21.15.78:0/861605670 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d4155770 con 0x7f95d40c8e00 2024-02-19T14:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.746+0000 7f95d37fe640 1 --2- 172.21.15.78:0/861605670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d406c7e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f95d408bfd0 tx=0x7f95cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f9044fa73514a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.746+0000 7f95d27fc640 1 -- 172.21.15.78:0/861605670 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc02ca70 con 0x7f95d40c8e00 2024-02-19T14:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.746+0000 7f95d27fc640 1 -- 172.21.15.78:0/861605670 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc02cbd0 con 0x7f95d40c8e00 2024-02-19T14:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.746+0000 7f95d27fc640 1 -- 172.21.15.78:0/861605670 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0343f0 con 0x7f95d40c8e00 2024-02-19T14:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 -- 172.21.15.78:0/861605670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 msgr2=0x7f95d406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 --2- 172.21.15.78:0/861605670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d406c7e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f95d408bfd0 tx=0x7f95cc02bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 -- 172.21.15.78:0/861605670 shutdown_connections 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 --2- 172.21.15.78:0/861605670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d406c7e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 --2- 172.21.15.78:0/861605670 >> v2:172.21.15.150:3300/0 conn(0x7f95d40c5730 0x7f95d406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 --2- 172.21.15.78:0/861605670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 -- 172.21.15.78:0/861605670 >> 172.21.15.78:0/861605670 conn(0x7f95d40cac90 msgr2=0x7f95d40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 -- 172.21.15.78:0/861605670 shutdown_connections 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.747+0000 7f95d948f640 1 -- 172.21.15.78:0/861605670 wait complete. 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 Processor -- start 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 -- start start 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d40c5730 0x7f95d41548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d40abff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4156a20 con 0x7f95d40c8e00 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40aa750 con 0x7f95d40c5730 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95d948f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d41eb3d0 con 0x7f95d4152700 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.748+0000 7f95cbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d40abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d40abff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59162/0 (socket says 172.21.15.78:59162) 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 -- 172.21.15.78:0/1232137930 learned_addr learned my addr 172.21.15.78:0/1232137930 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95d2ffd640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d40c5730 0x7f95d41548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95d37fe640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 -- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 msgr2=0x7f95d40ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 -- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d40c5730 msgr2=0x7f95d41548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d40c5730 0x7f95d41548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 -- 172.21.15.78:0/1232137930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc009b30 con 0x7f95d40c8e00 2024-02-19T14:52:09.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95d37fe640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:09.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cbfff640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d40abff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00e9b0 tx=0x7f95bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:09.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc009670 con 0x7f95d40c8e00 2024-02-19T14:52:09.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc00c9a0 con 0x7f95d40c8e00 2024-02-19T14:52:09.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d41eb0f0 con 0x7f95d40c8e00 2024-02-19T14:52:09.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00cd50 con 0x7f95d40c8e00 2024-02-19T14:52:09.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.749+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d4154e20 con 0x7f95d40c8e00 2024-02-19T14:52:09.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.750+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d40aa900 con 0x7f95d40c8e00 2024-02-19T14:52:09.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.751+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f95bc013470 con 0x7f95d40c8e00 2024-02-19T14:52:09.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.751+0000 7f95cb7fe640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95b406b810 0x7f95b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:09.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.751+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f95bc08a6b0 con 0x7f95d40c8e00 2024-02-19T14:52:09.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.751+0000 7f95d2ffd640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95b406b810 0x7f95b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:09.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.752+0000 7f95d2ffd640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95b406b810 0x7f95b406dcd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f95d41512b0 tx=0x7f95c4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:09.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.756+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f95d40aa900 con 0x7f95d40c8e00 2024-02-19T14:52:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.935+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95d40aa900 con 0x7f95d40c8e00 2024-02-19T14:52:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.936+0000 7f95cb7fe640 1 -- 172.21.15.78:0/1232137930 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f95d40aa900 con 0x7f95d40c8e00 2024-02-19T14:52:09.939 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:09.939 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:09.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95b406b810 msgr2=0x7f95b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:09.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95b406b810 0x7f95b406dcd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f95d41512b0 tx=0x7f95c4009190 comp rx=0 tx=0).stop 2024-02-19T14:52:09.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 msgr2=0x7f95d40abff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d40abff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00e9b0 tx=0x7f95bc00ee70 comp rx=0 tx=0).stop 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 shutdown_connections 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95b406b810 0x7f95b406dcd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95d40c8e00 0x7f95d40abff0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95d4152700 0x7f95d40ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 --2- 172.21.15.78:0/1232137930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d40c5730 0x7f95d41548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 >> 172.21.15.78:0/1232137930 conn(0x7f95d40cac90 msgr2=0x7f95d40cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 shutdown_connections 2024-02-19T14:52:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:09.938+0000 7f95d948f640 1 -- 172.21.15.78:0/1232137930 wait complete. 2024-02-19T14:52:09.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:09.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:10.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.055+0000 7f00466b1640 1 Processor -- start 2024-02-19T14:52:10.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.055+0000 7f00466b1640 1 -- start start 2024-02-19T14:52:10.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f00466b1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 0x7f00400c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f00466b1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0040156e80 con 0x7f00400721a0 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f00466b1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0040072d70 con 0x7f004006bb00 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f00466b1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004006ffc0 con 0x7f0040150560 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f0044c4e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f00400721a0 0x7f004015d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39284/0 (socket says 172.21.15.78:39284) 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f0044c4e640 1 -- 172.21.15.78:0/104667663 learned_addr learned my addr 172.21.15.78:0/104667663 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294994616 0 0) 0x7f0040156e80 con 0x7f00400721a0 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003ffff640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 0x7f00400c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0028003520 con 0x7f00400721a0 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916935974 0 0) 0x7f0028003520 con 0x7f00400721a0 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 msgr2=0x7f00400c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 0x7f00400c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 >> v1:172.21.15.150:6789/0 conn(0x7f004006bb00 legacy=0x7f00400c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040073050 con 0x7f00400721a0 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f00380019f0 con 0x7f00400721a0 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 >> v1:172.21.15.78:6789/0 conn(0x7f00400721a0 legacy=0x7f004015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 0x7f00280039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0028003ee0 0x7f00280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0028004d00 0x7f0028005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002800bb10 con 0x7f0028003ee0 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002800bc80 con 0x7f0028004d00 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.056+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002800be40 con 0x7f0040150560 2024-02-19T14:52:10.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003f7fe640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0028003ee0 0x7f00280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003ffff640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 0x7f00280039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f0044c4e640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0028004d00 0x7f0028005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003f7fe640 1 -- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 msgr2=0x7f00280039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003f7fe640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0040150560 0x7f00280039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003f7fe640 1 -- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0028004d00 msgr2=0x7f0028005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003f7fe640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0028004d00 0x7f0028005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.057+0000 7f003f7fe640 1 -- 172.21.15.78:0/104667663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0028003520 con 0x7f0028003ee0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.058+0000 7f003f7fe640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0028003ee0 0x7f00280047c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f00400c5540 tx=0x7f00280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca067801b556e22 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.058+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030033630 con 0x7f0028003ee0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.058+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030033790 con 0x7f0028003ee0 2024-02-19T14:52:10.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.058+0000 7f003effd640 1 -- 172.21.15.78:0/104667663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030033b60 con 0x7f0028003ee0 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f00466b1640 1 -- 172.21.15.78:0/104667663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0028003ee0 msgr2=0x7f00280047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f00466b1640 1 --2- 172.21.15.78:0/104667663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0028003ee0 0x7f00280047c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f00400c5540 tx=0x7f00280057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f0044c4e640 1 -- 172.21.15.78:0/104667663 reap_dead start 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f00466b1640 1 -- 172.21.15.78:0/104667663 shutdown_connections 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f00466b1640 1 -- 172.21.15.78:0/104667663 >> 172.21.15.78:0/104667663 conn(0x7f004006c170 msgr2=0x7f004006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f00466b1640 1 -- 172.21.15.78:0/104667663 shutdown_connections 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.059+0000 7f00466b1640 1 -- 172.21.15.78:0/104667663 wait complete. 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.060+0000 7f00466b1640 1 Processor -- start 2024-02-19T14:52:10.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.060+0000 7f00466b1640 1 -- start start 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f00466b1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0028003ee0 0x7f00401534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f00466b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0040150d90 0x7f00401ebe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f00466b1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 0x7f00401e8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f00466b1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00401eb640 con 0x7f00401e8010 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f00466b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0040151650 con 0x7f0040150d90 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f00466b1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0040151910 con 0x7f0028003ee0 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 0x7f00401e8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 0x7f00401e8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59180/0 (socket says 172.21.15.78:59180) 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 -- 172.21.15.78:0/2585906613 learned_addr learned my addr 172.21.15.78:0/2585906613 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f003f7fe640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0040150d90 0x7f00401ebe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 -- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0028003ee0 msgr2=0x7f00401534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f003ffff640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0028003ee0 0x7f00401534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0028003ee0 0x7f00401534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 -- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0040150d90 msgr2=0x7f00401ebe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0040150d90 0x7f00401ebe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 -- 172.21.15.78:0/2585906613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040151bf0 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f003f7fe640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0040150d90 0x7f00401ebe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f003ffff640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0028003ee0 0x7f00401534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.061+0000 7f0044c4e640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 0x7f00401e8490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0038001460 tx=0x7f0038007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.062+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003800ab30 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.062+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003800ac90 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.062+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00380133f0 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.062+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00401eb8e0 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.062+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0040153a20 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.063+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0040074990 con 0x7f00401e8010 2024-02-19T14:52:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.063+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0038013570 con 0x7f00401e8010 2024-02-19T14:52:10.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.063+0000 7f003cff9640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002c06b840 0x7f002c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.064+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f003808adc0 con 0x7f00401e8010 2024-02-19T14:52:10.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.064+0000 7f003ffff640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002c06b840 0x7f002c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.064+0000 7f003ffff640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002c06b840 0x7f002c06dd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f00400c45c0 tx=0x7f0034008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.072+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0038055530 con 0x7f00401e8010 2024-02-19T14:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.250+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f004006b8e0 con 0x7f00401e8010 2024-02-19T14:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.250+0000 7f003cff9640 1 -- 172.21.15.78:0/2585906613 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0038017070 con 0x7f00401e8010 2024-02-19T14:52:10.254 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002c06b840 msgr2=0x7f002c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002c06b840 0x7f002c06dd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f00400c45c0 tx=0x7f0034008040 comp rx=0 tx=0).stop 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 msgr2=0x7f00401e8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 0x7f00401e8490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0038001460 tx=0x7f0038007bd0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 shutdown_connections 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002c06b840 0x7f002c06dd00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00401e8010 0x7f00401e8490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0040150d90 0x7f00401ebe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 --2- 172.21.15.78:0/2585906613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0028003ee0 0x7f00401534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.252+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 >> 172.21.15.78:0/2585906613 conn(0x7f004006c170 msgr2=0x7f00400c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:10.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.253+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 shutdown_connections 2024-02-19T14:52:10.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.253+0000 7f00466b1640 1 -- 172.21.15.78:0/2585906613 wait complete. 2024-02-19T14:52:10.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.396+0000 7f71d2423640 1 Processor -- start 2024-02-19T14:52:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.396+0000 7f71d2423640 1 -- start start 2024-02-19T14:52:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d2423640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc152700 0x7f71cc0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d2423640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f71cc0c5730 0x7f71cc06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d2423640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71cc0c8e00 0x7f71cc06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d2423640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71cc06cd20 con 0x7f71cc0c8e00 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d2423640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71cc1551f0 con 0x7f71cc0c5730 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d2423640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc155490 con 0x7f71cc152700 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d09c0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71cc0c8e00 0x7f71cc06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d09c0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71cc0c8e00 0x7f71cc06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59186/0 (socket says 172.21.15.78:59186) 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.397+0000 7f71d09c0640 1 -- 172.21.15.78:0/1103808193 learned_addr learned my addr 172.21.15.78:0/1103808193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71d09c0640 1 -- 172.21.15.78:0/1103808193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc152700 msgr2=0x7f71cc0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71d09c0640 1 --2- 172.21.15.78:0/1103808193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc152700 0x7f71cc0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71d09c0640 1 -- 172.21.15.78:0/1103808193 >> v2:172.21.15.150:3300/0 conn(0x7f71cc0c5730 msgr2=0x7f71cc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71d09c0640 1 --2- 172.21.15.78:0/1103808193 >> v2:172.21.15.150:3300/0 conn(0x7f71cc0c5730 0x7f71cc06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71d09c0640 1 -- 172.21.15.78:0/1103808193 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc155770 con 0x7f71cc0c8e00 2024-02-19T14:52:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71d09c0640 1 --2- 172.21.15.78:0/1103808193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc0c8e00 0x7f71cc06c7e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f71cc08bfd0 tx=0x7f71c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1133122d1b109b4d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.398+0000 7f71caffd640 1 -- 172.21.15.78:0/1103808193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c402ca70 con 0x7f71cc0c8e00 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71caffd640 1 -- 172.21.15.78:0/1103808193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c402cbd0 con 0x7f71cc0c8e00 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71caffd640 1 -- 172.21.15.78:0/1103808193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c40343f0 con 0x7f71cc0c8e00 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 -- 172.21.15.78:0/1103808193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc0c8e00 msgr2=0x7f71cc06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 --2- 172.21.15.78:0/1103808193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc0c8e00 0x7f71cc06c7e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f71cc08bfd0 tx=0x7f71c402bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 -- 172.21.15.78:0/1103808193 shutdown_connections 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 --2- 172.21.15.78:0/1103808193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc0c8e00 0x7f71cc06c7e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 --2- 172.21.15.78:0/1103808193 >> v2:172.21.15.150:3300/0 conn(0x7f71cc0c5730 0x7f71cc06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 --2- 172.21.15.78:0/1103808193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc152700 0x7f71cc0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 -- 172.21.15.78:0/1103808193 >> 172.21.15.78:0/1103808193 conn(0x7f71cc0cac90 msgr2=0x7f71cc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 -- 172.21.15.78:0/1103808193 shutdown_connections 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.399+0000 7f71d2423640 1 -- 172.21.15.78:0/1103808193 wait complete. 2024-02-19T14:52:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.400+0000 7f71d2423640 1 Processor -- start 2024-02-19T14:52:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.400+0000 7f71d2423640 1 -- start start 2024-02-19T14:52:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d2423640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71cc0c5730 0x7f71cc1edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d2423640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc0c8e00 0x7f71cc148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d2423640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 0x7f71cc148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d2423640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0aa2b0 con 0x7f71cc152700 2024-02-19T14:52:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d2423640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc1494e0 con 0x7f71cc0c5730 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d2423640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0ac370 con 0x7f71cc0c8e00 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d09c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 0x7f71cc148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d09c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 0x7f71cc148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59194/0 (socket says 172.21.15.78:59194) 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d09c0640 1 -- 172.21.15.78:0/622461300 learned_addr learned my addr 172.21.15.78:0/622461300 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71cbfff640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71cc0c5730 0x7f71cc1edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71cb7fe640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc0c8e00 0x7f71cc148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.401+0000 7f71d09c0640 1 -- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc0c8e00 msgr2=0x7f71cc148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d09c0640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc0c8e00 0x7f71cc148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d09c0640 1 -- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71cc0c5730 msgr2=0x7f71cc1edcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d09c0640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71cc0c5730 0x7f71cc1edcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d09c0640 1 -- 172.21.15.78:0/622461300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c4009b30 con 0x7f71cc152700 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d09c0640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 0x7f71cc148fa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009f40 tx=0x7f71c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c403e450 con 0x7f71cc152700 2024-02-19T14:52:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71cc0acf20 con 0x7f71cc152700 2024-02-19T14:52:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c403e5d0 con 0x7f71cc152700 2024-02-19T14:52:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c403e980 con 0x7f71cc152700 2024-02-19T14:52:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.402+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71cc1ee530 con 0x7f71cc152700 2024-02-19T14:52:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.403+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71cc074990 con 0x7f71cc152700 2024-02-19T14:52:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.404+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f71c403eae0 con 0x7f71cc152700 2024-02-19T14:52:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.404+0000 7f71c8ff9640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71ac06b840 0x7f71ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.404+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f71c40ad0f0 con 0x7f71cc152700 2024-02-19T14:52:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.404+0000 7f71cbfff640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71ac06b840 0x7f71ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.405+0000 7f71cbfff640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71ac06b840 0x7f71ac06dd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00b810 tx=0x7f71bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.408+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f71c40777e0 con 0x7f71cc152700 2024-02-19T14:52:10.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.585+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71cc0c8770 con 0x7f71cc152700 2024-02-19T14:52:10.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.585+0000 7f71c8ff9640 1 -- 172.21.15.78:0/622461300 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f71c407aea0 con 0x7f71cc152700 2024-02-19T14:52:10.588 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71ac06b840 msgr2=0x7f71ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71ac06b840 0x7f71ac06dd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00b810 tx=0x7f71bc008040 comp rx=0 tx=0).stop 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 msgr2=0x7f71cc148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 0x7f71cc148fa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009f40 tx=0x7f71c4004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 shutdown_connections 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71ac06b840 0x7f71ac06dd00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71cc152700 0x7f71cc148fa0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71cc0c8e00 0x7f71cc148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 --2- 172.21.15.78:0/622461300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71cc0c5730 0x7f71cc1edcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 >> 172.21.15.78:0/622461300 conn(0x7f71cc0cac90 msgr2=0x7f71cc06c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:10.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 shutdown_connections 2024-02-19T14:52:10.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.587+0000 7f71d2423640 1 -- 172.21.15.78:0/622461300 wait complete. 2024-02-19T14:52:10.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.702+0000 7f5ee6b3e640 1 Processor -- start 2024-02-19T14:52:10.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 -- start start 2024-02-19T14:52:10.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5ee0150560 0x7f5ee00c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5ee006bb00 0x7f5ee00c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee00721a0 0x7f5ee00c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee006ffc0 con 0x7f5ee006bb00 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee0156e80 con 0x7f5ee0150560 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5ee6b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0072d70 con 0x7f5ee00721a0 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5edffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5ee006bb00 0x7f5ee00c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5edffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5ee006bb00 0x7f5ee00c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59210/0 (socket says 172.21.15.78:59210) 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.703+0000 7f5edffff640 1 -- 172.21.15.78:0/581096998 learned_addr learned my addr 172.21.15.78:0/581096998 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5ee50db640 1 --2- 172.21.15.78:0/581096998 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee00721a0 0x7f5ee00c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5ee48da640 1 --2- 172.21.15.78:0/581096998 >> v2:172.21.15.150:3300/0 conn(0x7f5ee0150560 0x7f5ee00c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5edffff640 1 -- 172.21.15.78:0/581096998 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee00721a0 msgr2=0x7f5ee00c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5edffff640 1 --2- 172.21.15.78:0/581096998 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee00721a0 0x7f5ee00c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5edffff640 1 -- 172.21.15.78:0/581096998 >> v2:172.21.15.150:3300/0 conn(0x7f5ee0150560 msgr2=0x7f5ee00c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5edffff640 1 --2- 172.21.15.78:0/581096998 >> v2:172.21.15.150:3300/0 conn(0x7f5ee0150560 0x7f5ee00c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.704+0000 7f5edffff640 1 -- 172.21.15.78:0/581096998 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0073050 con 0x7f5ee006bb00 2024-02-19T14:52:10.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.705+0000 7f5edffff640 1 --2- 172.21.15.78:0/581096998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee006bb00 0x7f5ee00c6080 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00ac950 tx=0x7f5ecc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5baea11c5ced8c61 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.705+0000 7f5edf7fe640 1 -- 172.21.15.78:0/581096998 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc0317e0 con 0x7f5ee006bb00 2024-02-19T14:52:10.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.705+0000 7f5edf7fe640 1 -- 172.21.15.78:0/581096998 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc031940 con 0x7f5ee006bb00 2024-02-19T14:52:10.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.705+0000 7f5edf7fe640 1 -- 172.21.15.78:0/581096998 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc031cd0 con 0x7f5ee006bb00 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/581096998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee006bb00 msgr2=0x7f5ee00c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/581096998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee006bb00 0x7f5ee00c6080 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00ac950 tx=0x7f5ecc02b880 comp rx=0 tx=0).stop 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/581096998 shutdown_connections 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/581096998 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee00721a0 0x7f5ee00c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/581096998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee006bb00 0x7f5ee00c6080 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/581096998 >> v2:172.21.15.150:3300/0 conn(0x7f5ee0150560 0x7f5ee00c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/581096998 >> 172.21.15.78:0/581096998 conn(0x7f5ee006c170 msgr2=0x7f5ee006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/581096998 shutdown_connections 2024-02-19T14:52:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.706+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/581096998 wait complete. 2024-02-19T14:52:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.707+0000 7f5ee6b3e640 1 Processor -- start 2024-02-19T14:52:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.707+0000 7f5ee6b3e640 1 -- start start 2024-02-19T14:52:10.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ee006bb00 0x7f5ee01e9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 0x7f5ee0155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee0150560 0x7f5ee01e87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0150b80 con 0x7f5ee00721a0 2024-02-19T14:52:10.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0150e60 con 0x7f5ee006bb00 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0151140 con 0x7f5ee0150560 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 0x7f5ee0155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 0x7f5ee0155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59222/0 (socket says 172.21.15.78:59222) 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 -- 172.21.15.78:0/4056910017 learned_addr learned my addr 172.21.15.78:0/4056910017 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee48da640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ee006bb00 0x7f5ee01e9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee50db640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee0150560 0x7f5ee01e87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 -- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee0150560 msgr2=0x7f5ee01e87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee0150560 0x7f5ee01e87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 -- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ee006bb00 msgr2=0x7f5ee01e9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ee006bb00 0x7f5ee01e9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 -- 172.21.15.78:0/4056910017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0094d0 con 0x7f5ee00721a0 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee48da640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ee006bb00 0x7f5ee01e9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee50db640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee0150560 0x7f5ee01e87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edffff640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 0x7f5ee0155430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc000c00 tx=0x7f5ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc032880 con 0x7f5ee00721a0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc0329e0 con 0x7f5ee00721a0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee01e82b0 con 0x7f5ee00721a0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.709+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc032d90 con 0x7f5ee00721a0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.710+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee01e7fa0 con 0x7f5ee00721a0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.711+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0074990 con 0x7f5ee00721a0 2024-02-19T14:52:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.711+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5ecc04e450 con 0x7f5ee00721a0 2024-02-19T14:52:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.711+0000 7f5edd7fa640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ec406b840 0x7f5ec406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.711+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5ecc0ac400 con 0x7f5ee00721a0 2024-02-19T14:52:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.712+0000 7f5ee48da640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ec406b840 0x7f5ec406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.712+0000 7f5ee48da640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ec406b840 0x7f5ec406dd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009b90 tx=0x7f5ed4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:10.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.716+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5ecc076e00 con 0x7f5ee00721a0 2024-02-19T14:52:10.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.893+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ee006b8e0 con 0x7f5ee00721a0 2024-02-19T14:52:10.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.893+0000 7f5edd7fa640 1 -- 172.21.15.78:0/4056910017 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ecc07a4c0 con 0x7f5ee00721a0 2024-02-19T14:52:10.896 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:10.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ec406b840 msgr2=0x7f5ec406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ec406b840 0x7f5ec406dd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009b90 tx=0x7f5ed4008040 comp rx=0 tx=0).stop 2024-02-19T14:52:10.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 msgr2=0x7f5ee0155430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 0x7f5ee0155430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc000c00 tx=0x7f5ecc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 shutdown_connections 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ec406b840 0x7f5ec406dd00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ee00721a0 0x7f5ee0155430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ee0150560 0x7f5ee01e87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 --2- 172.21.15.78:0/4056910017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ee006bb00 0x7f5ee01e9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 >> 172.21.15.78:0/4056910017 conn(0x7f5ee006c170 msgr2=0x7f5ee00c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 shutdown_connections 2024-02-19T14:52:10.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:10.895+0000 7f5ee6b3e640 1 -- 172.21.15.78:0/4056910017 wait complete. 2024-02-19T14:52:10.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.010+0000 7ff22bc42640 1 Processor -- start 2024-02-19T14:52:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.010+0000 7ff22bc42640 1 -- start start 2024-02-19T14:52:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff22bc42640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff2240c7f50 0x7ff224160850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff22bc42640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240c8380 con 0x7ff2240c7b70 2024-02-19T14:52:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff22bc42640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22406db90 con 0x7ff224155bb0 2024-02-19T14:52:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff22bc42640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff224154ec0 con 0x7ff2240c7f50 2024-02-19T14:52:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff2299de640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff2240c7b70 0x7ff2241584b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39286/0 (socket says 172.21.15.78:39286) 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff2299de640 1 -- 172.21.15.78:0/551731805 learned_addr learned my addr 172.21.15.78:0/551731805 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.011+0000 7ff22a1df640 1 --2- 172.21.15.78:0/551731805 >> v2:172.21.15.150:3300/0 conn(0x7ff2240c7f50 0x7ff224160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.012+0000 7ff2289dc640 1 -- 172.21.15.78:0/551731805 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082509606 0 0) 0x7ff2240c8380 con 0x7ff2240c7b70 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.012+0000 7ff2289dc640 1 -- 172.21.15.78:0/551731805 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff210003520 con 0x7ff2240c7b70 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.012+0000 7ff22a1df640 1 -- 172.21.15.78:0/551731805 >> v1:172.21.15.150:6789/0 conn(0x7ff224155bb0 legacy=0x7ff22406d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.012+0000 7ff22a1df640 1 -- 172.21.15.78:0/551731805 >> v1:172.21.15.78:6789/0 conn(0x7ff2240c7b70 legacy=0x7ff2241584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.012+0000 7ff22a1df640 1 -- 172.21.15.78:0/551731805 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2241550d0 con 0x7ff2240c7f50 2024-02-19T14:52:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22a1df640 1 --2- 172.21.15.78:0/551731805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240c7f50 0x7ff224160850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009d90 tx=0x7ff21c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f335f9806c4aac41 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff2289dc640 1 -- 172.21.15.78:0/551731805 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c02cc60 con 0x7ff2240c7f50 2024-02-19T14:52:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff2289dc640 1 -- 172.21.15.78:0/551731805 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c02cdc0 con 0x7ff2240c7f50 2024-02-19T14:52:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff2289dc640 1 -- 172.21.15.78:0/551731805 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c034640 con 0x7ff2240c7f50 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22bc42640 1 -- 172.21.15.78:0/551731805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240c7f50 msgr2=0x7ff224160850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22bc42640 1 --2- 172.21.15.78:0/551731805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240c7f50 0x7ff224160850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009d90 tx=0x7ff21c02beb0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22bc42640 1 -- 172.21.15.78:0/551731805 shutdown_connections 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22bc42640 1 --2- 172.21.15.78:0/551731805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240c7f50 0x7ff224160850 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22bc42640 1 -- 172.21.15.78:0/551731805 >> 172.21.15.78:0/551731805 conn(0x7ff2240c5050 msgr2=0x7ff2240c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.013+0000 7ff22bc42640 1 -- 172.21.15.78:0/551731805 shutdown_connections 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.014+0000 7ff22bc42640 1 -- 172.21.15.78:0/551731805 wait complete. 2024-02-19T14:52:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.015+0000 7ff22bc42640 1 Processor -- start 2024-02-19T14:52:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.015+0000 7ff22bc42640 1 -- start start 2024-02-19T14:52:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff22bc42640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2240c7b70 0x7ff2241de7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff22bc42640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 0x7ff224157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff22bc42640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224155bb0 0x7ff22414f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff22bc42640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2241def80 con 0x7ff2240c7f50 2024-02-19T14:52:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff22bc42640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2241eb8d0 con 0x7ff224155bb0 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff22bc42640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22414b1c0 con 0x7ff2240c7b70 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff2291dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 0x7ff224157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff2291dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 0x7ff224157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59230/0 (socket says 172.21.15.78:59230) 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.016+0000 7ff2291dd640 1 -- 172.21.15.78:0/33192109 learned_addr learned my addr 172.21.15.78:0/33192109 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2291dd640 1 -- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2240c7b70 msgr2=0x7ff2241de7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff22a1df640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224155bb0 0x7ff22414f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2299de640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2240c7b70 0x7ff2241de7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2291dd640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2240c7b70 0x7ff2241de7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2291dd640 1 -- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224155bb0 msgr2=0x7ff22414f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2291dd640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224155bb0 0x7ff22414f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2291dd640 1 -- 172.21.15.78:0/33192109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c009a70 con 0x7ff2240c7f50 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff22a1df640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224155bb0 0x7ff22414f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff2291dd640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 0x7ff224157500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff22000c3c0 tx=0x7ff22000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2200052d0 con 0x7ff2240c7f50 2024-02-19T14:52:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff220005450 con 0x7ff2240c7f50 2024-02-19T14:52:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22414af30 con 0x7ff2240c7f50 2024-02-19T14:52:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220007260 con 0x7ff2240c7f50 2024-02-19T14:52:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.017+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2241eb560 con 0x7ff2240c7f50 2024-02-19T14:52:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.018+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22406fc80 con 0x7ff2240c7f50 2024-02-19T14:52:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.018+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff2200073c0 con 0x7ff2240c7f50 2024-02-19T14:52:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.019+0000 7ff21a7fc640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff20806b810 0x7ff20806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.019+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff220089800 con 0x7ff2240c7f50 2024-02-19T14:52:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.019+0000 7ff2299de640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff20806b810 0x7ff20806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.019+0000 7ff2299de640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff20806b810 0x7ff20806dcd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff2140011e0 tx=0x7ff214009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.027+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff220053f70 con 0x7ff2240c7f50 2024-02-19T14:52:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.204+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff2240c61a0 con 0x7ff2240c7f50 2024-02-19T14:52:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.204+0000 7ff21a7fc640 1 -- 172.21.15.78:0/33192109 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff220057630 con 0x7ff2240c7f50 2024-02-19T14:52:11.207 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.206+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff20806b810 msgr2=0x7ff20806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.206+0000 7ff22bc42640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff20806b810 0x7ff20806dcd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff2140011e0 tx=0x7ff214009040 comp rx=0 tx=0).stop 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.206+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 msgr2=0x7ff224157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.206+0000 7ff22bc42640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 0x7ff224157500 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff22000c3c0 tx=0x7ff22000c880 comp rx=0 tx=0).stop 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 shutdown_connections 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff20806b810 0x7ff20806dcd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2240c7f50 0x7ff224157500 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224155bb0 0x7ff22414f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 --2- 172.21.15.78:0/33192109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2240c7b70 0x7ff2241de7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 >> 172.21.15.78:0/33192109 conn(0x7ff2240c5050 msgr2=0x7ff22406f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 shutdown_connections 2024-02-19T14:52:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.207+0000 7ff22bc42640 1 -- 172.21.15.78:0/33192109 wait complete. 2024-02-19T14:52:11.220 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T14:52:11.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.323+0000 7fcf095bc640 1 Processor -- start 2024-02-19T14:52:11.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.323+0000 7fcf095bc640 1 -- start start 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.324+0000 7fcf095bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf040c7280 0x7fcf040c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.324+0000 7fcf095bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf041586a0 con 0x7fcf04152830 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.324+0000 7fcf095bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf04158980 con 0x7fcf0406c570 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.324+0000 7fcf095bc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf040c6550 con 0x7fcf040c7280 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.324+0000 7fcf02ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcf04152830 0x7fcf040c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39296/0 (socket says 172.21.15.78:39296) 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.324+0000 7fcf02ffd640 1 -- 172.21.15.78:0/2676196156 learned_addr learned my addr 172.21.15.78:0/2676196156 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866378410 0 0) 0x7fcf041586a0 con 0x7fcf04152830 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf027fc640 1 --2- 172.21.15.78:0/2676196156 >> v2:172.21.15.150:3300/0 conn(0x7fcf040c7280 0x7fcf040c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec003520 con 0x7fcf04152830 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3543872513 0 0) 0x7fceec003520 con 0x7fcf04152830 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 >> v2:172.21.15.150:3300/0 conn(0x7fcf040c7280 msgr2=0x7fcf040c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 --2- 172.21.15.78:0/2676196156 >> v2:172.21.15.150:3300/0 conn(0x7fcf040c7280 0x7fcf040c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 >> v1:172.21.15.150:6789/0 conn(0x7fcf0406c570 legacy=0x7fcf0406c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04158c60 con 0x7fcf04152830 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fcef4001670 con 0x7fcf04152830 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 >> v1:172.21.15.78:6789/0 conn(0x7fcf04152830 legacy=0x7fcf040c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec003970 0x7fceec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0047c0 0x7fceec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec005180 0x7fceec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c6550 con 0x7fceec005180 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec00c120 con 0x7fceec0047c0 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec00c2e0 con 0x7fceec003970 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf037fe640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec005180 0x7fceec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf02ffd640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec003970 0x7fceec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf027fc640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0047c0 0x7fceec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf037fe640 1 -- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec003970 msgr2=0x7fceec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:11.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf037fe640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec003970 0x7fceec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf037fe640 1 -- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0047c0 msgr2=0x7fceec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf037fe640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0047c0 0x7fceec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.325+0000 7fcf037fe640 1 -- 172.21.15.78:0/2676196156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec003520 con 0x7fceec005180 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.326+0000 7fcf037fe640 1 -- 172.21.15.78:0/2676196156 reap_dead start 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.326+0000 7fcf037fe640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec005180 0x7fceec00b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcefc001e50 tx=0x7fceec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1cce1f1dcf7b29 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.326+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc003b70 con 0x7fceec005180 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.326+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc003d10 con 0x7fceec005180 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.326+0000 7fcf01ffb640 1 -- 172.21.15.78:0/2676196156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc005310 con 0x7fceec005180 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 -- 172.21.15.78:0/2676196156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec005180 msgr2=0x7fceec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec005180 0x7fceec00b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcefc001e50 tx=0x7fceec0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 -- 172.21.15.78:0/2676196156 shutdown_connections 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 --2- 172.21.15.78:0/2676196156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec005180 0x7fceec00b980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 -- 172.21.15.78:0/2676196156 >> 172.21.15.78:0/2676196156 conn(0x7fcf040ca8a0 msgr2=0x7fcf040cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:11.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 -- 172.21.15.78:0/2676196156 shutdown_connections 2024-02-19T14:52:11.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.327+0000 7fcf095bc640 1 -- 172.21.15.78:0/2676196156 wait complete. 2024-02-19T14:52:11.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.328+0000 7fcf095bc640 1 Processor -- start 2024-02-19T14:52:11.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.328+0000 7fcf095bc640 1 -- start start 2024-02-19T14:52:11.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec005180 0x7fcf041eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 0x7fcf040ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf040c0c50 0x7fcf040c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c7c40 con 0x7fcf040ac520 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04159320 con 0x7fceec005180 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040acf40 con 0x7fcf040c0c50 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 0x7fcf040ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 0x7fcf040ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59244/0 (socket says 172.21.15.78:59244) 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 -- 172.21.15.78:0/1588390973 learned_addr learned my addr 172.21.15.78:0/1588390973 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf037fe640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf040c0c50 0x7fcf040c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf02ffd640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec005180 0x7fcf041eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 -- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf040c0c50 msgr2=0x7fcf040c10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf040c0c50 0x7fcf040c10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 -- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec005180 msgr2=0x7fcf041eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec005180 0x7fcf041eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 -- 172.21.15.78:0/1588390973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf041ebdf0 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf02ffd640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec005180 0x7fcf041eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf027fc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 0x7fcf040ab6b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcef80118e0 tx=0x7fcef8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800f9d0 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef800fb30 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf040ac980 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800fec0 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.329+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf040aaeb0 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.330+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf041497c0 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.331+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcef8014440 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.331+0000 7fceeb7fe640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b840 0x7fcef006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.331+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcef808ac10 con 0x7fcf040ac520 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.332+0000 7fcf02ffd640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b840 0x7fcef006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.332+0000 7fcf02ffd640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b840 0x7fcef006dd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcef4007c60 tx=0x7fcef4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.336+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcef8055380 con 0x7fcf040ac520 2024-02-19T14:52:11.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.526+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcf040aad10 con 0x7fcf040ac520 2024-02-19T14:52:11.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.527+0000 7fceeb7fe640 1 -- 172.21.15.78:0/1588390973 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcf040aad10 con 0x7fcf040ac520 2024-02-19T14:52:11.529 INFO:teuthology.orchestra.run.smithi078.stderr:listed 0 entries 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b840 msgr2=0x7fcef006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b840 0x7fcef006dd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcef4007c60 tx=0x7fcef4009040 comp rx=0 tx=0).stop 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 msgr2=0x7fcf040ab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 0x7fcf040ab6b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcef80118e0 tx=0x7fcef8011da0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 shutdown_connections 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b840 0x7fcef006dd00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf040ac520 0x7fcf040ab6b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf040c0c50 0x7fcf040c10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 --2- 172.21.15.78:0/1588390973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec005180 0x7fcf041eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 >> 172.21.15.78:0/1588390973 conn(0x7fcf040ca8a0 msgr2=0x7fcf0414d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:11.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 shutdown_connections 2024-02-19T14:52:11.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.529+0000 7fcf095bc640 1 -- 172.21.15.78:0/1588390973 wait complete. 2024-02-19T14:52:11.543 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:52:11.543 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:11.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.697+0000 7f12b92ae640 1 Processor -- start 2024-02-19T14:52:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.697+0000 7f12b92ae640 1 -- start start 2024-02-19T14:52:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b92ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12b4152700 0x7f12b40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b92ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b41551f0 con 0x7f12b40c8e00 2024-02-19T14:52:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b92ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b4155490 con 0x7f12b40c5730 2024-02-19T14:52:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b92ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b4155770 con 0x7f12b4152700 2024-02-19T14:52:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f12b40c8e00 0x7f12b415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39308/0 (socket says 172.21.15.78:39308) 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b37fe640 1 -- 172.21.15.78:0/3434668991 learned_addr learned my addr 172.21.15.78:0/3434668991 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b2ffd640 1 --2- 172.21.15.78:0/3434668991 >> v2:172.21.15.150:3300/0 conn(0x7f12b4152700 0x7f12b40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067726930 0 0) 0x7f12b41551f0 con 0x7f12b40c8e00 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12b40c8e00 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222878406 0 0) 0x7f12a0003520 con 0x7f12b40c8e00 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 >> v2:172.21.15.150:3300/0 conn(0x7f12b4152700 msgr2=0x7f12b40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 --2- 172.21.15.78:0/3434668991 >> v2:172.21.15.150:3300/0 conn(0x7f12b4152700 0x7f12b40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 >> v1:172.21.15.150:6789/0 conn(0x7f12b40c5730 legacy=0x7f12b406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4155a50 con 0x7f12b40c8e00 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.698+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f12a4001670 con 0x7f12b40c8e00 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 >> v1:172.21.15.78:6789/0 conn(0x7f12b40c8e00 legacy=0x7f12b415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a0003970 0x7f12a0003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12a0004370 0x7f12a0004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b4155770 con 0x7f12a00050e0 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000c020 con 0x7f12a0003970 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000c1e0 con 0x7f12a0004370 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b37fe640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b2ffd640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a0003970 0x7f12a0003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b27fc640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12a0004370 0x7f12a0004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b37fe640 1 -- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12a0004370 msgr2=0x7f12a0004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b37fe640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12a0004370 0x7f12a0004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b37fe640 1 -- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a0003970 msgr2=0x7f12a0003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b37fe640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a0003970 0x7f12a0003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.699+0000 7f12b37fe640 1 -- 172.21.15.78:0/3434668991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0003520 con 0x7f12a00050e0 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b37fe640 1 -- 172.21.15.78:0/3434668991 reap_dead start 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b37fe640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12a000b980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c160 tx=0x7f12a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5b272001621262 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4005ad0 con 0x7f12a00050e0 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4005c50 con 0x7f12a00050e0 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b1ffb640 1 -- 172.21.15.78:0/3434668991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a402c830 con 0x7f12a00050e0 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 -- 172.21.15.78:0/3434668991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 msgr2=0x7f12a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12a000b980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c160 tx=0x7f12a00057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 -- 172.21.15.78:0/3434668991 shutdown_connections 2024-02-19T14:52:11.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 --2- 172.21.15.78:0/3434668991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12a000b980 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 -- 172.21.15.78:0/3434668991 >> 172.21.15.78:0/3434668991 conn(0x7f12b40cac90 msgr2=0x7f12b40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:11.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 -- 172.21.15.78:0/3434668991 shutdown_connections 2024-02-19T14:52:11.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.700+0000 7f12b92ae640 1 -- 172.21.15.78:0/3434668991 wait complete. 2024-02-19T14:52:11.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 Processor -- start 2024-02-19T14:52:11.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 -- start start 2024-02-19T14:52:11.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12b40be370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b40c27d0 0x7f12b40c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40c2c80 0x7f12b40c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40beb00 con 0x7f12a00050e0 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b41477b0 con 0x7f12b40c2c80 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b92ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40c1d80 con 0x7f12b40c27d0 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12b40be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12b40be370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59256/0 (socket says 172.21.15.78:59256) 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.702+0000 7f12b2ffd640 1 -- 172.21.15.78:0/1048314607 learned_addr learned my addr 172.21.15.78:0/1048314607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b27fc640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b40c27d0 0x7f12b40c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b37fe640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40c2c80 0x7f12b40c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b2ffd640 1 -- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b40c27d0 msgr2=0x7f12b40c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b2ffd640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b40c27d0 0x7f12b40c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b2ffd640 1 -- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40c2c80 msgr2=0x7f12b40c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b2ffd640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40c2c80 0x7f12b40c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b2ffd640 1 -- 172.21.15.78:0/1048314607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40c2340 con 0x7f12a00050e0 2024-02-19T14:52:11.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b37fe640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40c2c80 0x7f12b40c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b2ffd640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12b40be370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009fc0 tx=0x7f12ac00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac00e040 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac00f040 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b41eb810 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac015870 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.703+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b41ebab0 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.704+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4074990 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.704+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f12ac0159d0 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.705+0000 7f12ab7fe640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129406b810 0x7f129406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.705+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12ac08b370 con 0x7f12a00050e0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.705+0000 7f12b27fc640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129406b810 0x7f129406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:11.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.705+0000 7f12b27fc640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129406b810 0x7f129406dcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12b41e8b30 tx=0x7f129c007bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:11.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.709+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f12ac0559c0 con 0x7f12a00050e0 2024-02-19T14:52:11.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.964+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12b40b04c0 con 0x7f12a00050e0 2024-02-19T14:52:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.965+0000 7f12ab7fe640 1 -- 172.21.15.78:0/1048314607 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f12ac00fea0 con 0x7f12a00050e0 2024-02-19T14:52:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:11.968 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.967+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129406b810 msgr2=0x7f129406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129406b810 0x7f129406dcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12b41e8b30 tx=0x7f129c007bb0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 msgr2=0x7f12b40be370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12b40be370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009fc0 tx=0x7f12ac00f580 comp rx=0 tx=0).stop 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 shutdown_connections 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129406b810 0x7f129406dcd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40c2c80 0x7f12b40c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b40c27d0 0x7f12b40c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 --2- 172.21.15.78:0/1048314607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12a00050e0 0x7f12b40be370 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 >> 172.21.15.78:0/1048314607 conn(0x7f12b40cac90 msgr2=0x7f12b414ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 shutdown_connections 2024-02-19T14:52:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:11.968+0000 7f12b92ae640 1 -- 172.21.15.78:0/1048314607 wait complete. 2024-02-19T14:52:11.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:11.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.082+0000 7f367fe12640 1 Processor -- start 2024-02-19T14:52:12.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.082+0000 7f367fe12640 1 -- start start 2024-02-19T14:52:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367fe12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f367806ea50 0x7f3678072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367fe12640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3678152680 0x7f36780c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367fe12640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36780c5970 con 0x7f367814f1a0 2024-02-19T14:52:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367fe12640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3678152af0 con 0x7f3678152680 2024-02-19T14:52:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367fe12640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36780c5750 con 0x7f367806ea50 2024-02-19T14:52:12.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367e3af640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3678152680 0x7f36780c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367dbae640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f367814f1a0 0x7f367806fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39322/0 (socket says 172.21.15.78:39322) 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367dbae640 1 -- 172.21.15.78:0/1856986748 learned_addr learned my addr 172.21.15.78:0/1856986748 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.083+0000 7f367d3ad640 1 --2- 172.21.15.78:0/1856986748 >> v2:172.21.15.150:3300/0 conn(0x7f367806ea50 0x7f3678072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367e3af640 1 -- 172.21.15.78:0/1856986748 >> v2:172.21.15.150:3300/0 conn(0x7f367806ea50 msgr2=0x7f3678072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367e3af640 1 --2- 172.21.15.78:0/1856986748 >> v2:172.21.15.150:3300/0 conn(0x7f367806ea50 0x7f3678072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367e3af640 1 -- 172.21.15.78:0/1856986748 >> v1:172.21.15.78:6789/0 conn(0x7f367814f1a0 legacy=0x7f367806fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367e3af640 1 -- 172.21.15.78:0/1856986748 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780702d0 con 0x7f3678152680 2024-02-19T14:52:12.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367cbac640 1 -- 172.21.15.78:0/1856986748 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908610382 0 0) 0x7f36780c5970 con 0x7f367814f1a0 2024-02-19T14:52:12.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367e3af640 1 --2- 172.21.15.78:0/1856986748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3678152680 0x7f36780c78a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f367806f090 tx=0x7f367402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69f185dd3783fc0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367cbac640 1 -- 172.21.15.78:0/1856986748 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674031710 con 0x7f3678152680 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367cbac640 1 -- 172.21.15.78:0/1856986748 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367402f050 con 0x7f3678152680 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.084+0000 7f367cbac640 1 -- 172.21.15.78:0/1856986748 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674031a90 con 0x7f3678152680 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 -- 172.21.15.78:0/1856986748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3678152680 msgr2=0x7f36780c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 --2- 172.21.15.78:0/1856986748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3678152680 0x7f36780c78a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f367806f090 tx=0x7f367402b7b0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 -- 172.21.15.78:0/1856986748 shutdown_connections 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 --2- 172.21.15.78:0/1856986748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3678152680 0x7f36780c78a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 --2- 172.21.15.78:0/1856986748 >> v2:172.21.15.150:3300/0 conn(0x7f367806ea50 0x7f3678072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 -- 172.21.15.78:0/1856986748 >> 172.21.15.78:0/1856986748 conn(0x7f367806c170 msgr2=0x7f367806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 -- 172.21.15.78:0/1856986748 shutdown_connections 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.085+0000 7f367fe12640 1 -- 172.21.15.78:0/1856986748 wait complete. 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.086+0000 7f367fe12640 1 Processor -- start 2024-02-19T14:52:12.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.086+0000 7f367fe12640 1 -- start start 2024-02-19T14:52:12.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367fe12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f367806ea50 0x7f367814a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367fe12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 0x7f36781489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367fe12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3678152680 0x7f3678148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367fe12640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3678149470 con 0x7f367814f1a0 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367fe12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780c3fc0 con 0x7f3678152680 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367fe12640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36781ee030 con 0x7f367806ea50 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 0x7f36781489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 0x7f36781489f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59274/0 (socket says 172.21.15.78:59274) 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 -- 172.21.15.78:0/3222975765 learned_addr learned my addr 172.21.15.78:0/3222975765 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367e3af640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3678152680 0x7f3678148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367dbae640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f367806ea50 0x7f367814a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 -- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f367806ea50 msgr2=0x7f367814a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f367806ea50 0x7f367814a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 -- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3678152680 msgr2=0x7f3678148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3678152680 0x7f3678148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 -- 172.21.15.78:0/3222975765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674009400 con 0x7f367814f1a0 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367e3af640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3678152680 0x7f3678148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.087+0000 7f367d3ad640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 0x7f36781489f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f366800d5f0 tx=0x7f366800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.088+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680125f0 con 0x7f367814f1a0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.088+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668012750 con 0x7f367814f1a0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.088+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36781ee310 con 0x7f367814f1a0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.088+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668012b00 con 0x7f367814f1a0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.088+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36781ee8d0 con 0x7f367814f1a0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.089+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3678074990 con 0x7f367814f1a0 2024-02-19T14:52:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.089+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3668012c60 con 0x7f367814f1a0 2024-02-19T14:52:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.089+0000 7f366e7fc640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f365c06b810 0x7f365c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.090+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f366808a960 con 0x7f367814f1a0 2024-02-19T14:52:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.090+0000 7f367dbae640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f365c06b810 0x7f365c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.090+0000 7f367dbae640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f365c06b810 0x7f365c06dcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36700012e0 tx=0x7f3670009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.094+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f36680550d0 con 0x7f367814f1a0 2024-02-19T14:52:12.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.270+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f367806e830 con 0x7f367814f1a0 2024-02-19T14:52:12.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.271+0000 7f366e7fc640 1 -- 172.21.15.78:0/3222975765 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3668058790 con 0x7f367814f1a0 2024-02-19T14:52:12.274 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:12.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f365c06b810 msgr2=0x7f365c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f365c06b810 0x7f365c06dcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36700012e0 tx=0x7f3670009040 comp rx=0 tx=0).stop 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 msgr2=0x7f36781489f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 0x7f36781489f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f366800d5f0 tx=0x7f366800dab0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 shutdown_connections 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f365c06b810 0x7f365c06dcd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f367814f1a0 0x7f36781489f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3678152680 0x7f3678148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 --2- 172.21.15.78:0/3222975765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f367806ea50 0x7f367814a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 >> 172.21.15.78:0/3222975765 conn(0x7f367806c170 msgr2=0x7f367806fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 shutdown_connections 2024-02-19T14:52:12.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.273+0000 7f367fe12640 1 -- 172.21.15.78:0/3222975765 wait complete. 2024-02-19T14:52:12.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.388+0000 7f1e89f3e640 1 Processor -- start 2024-02-19T14:52:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e89f3e640 1 -- start start 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e89f3e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e8414d210 0x7f1e840c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e89f3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840719e0 0x7f1e840c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e89f3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8406ff90 con 0x7f1e840c3d90 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e89f3e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e84070270 con 0x7f1e8414d210 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e89f3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e840c3970 con 0x7f1e840719e0 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e837fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e8414d210 0x7f1e840c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e837fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e8414d210 0x7f1e840c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59288/0 (socket says 172.21.15.78:59288) 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.389+0000 7f1e837fe640 1 -- 172.21.15.78:0/2440609580 learned_addr learned my addr 172.21.15.78:0/2440609580 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.390+0000 7f1e82ffd640 1 --2- 172.21.15.78:0/2440609580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840719e0 0x7f1e840c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.390+0000 7f1e837fe640 1 -- 172.21.15.78:0/2440609580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840719e0 msgr2=0x7f1e840c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.390+0000 7f1e837fe640 1 --2- 172.21.15.78:0/2440609580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840719e0 0x7f1e840c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.390+0000 7f1e837fe640 1 -- 172.21.15.78:0/2440609580 >> v1:172.21.15.150:6789/0 conn(0x7f1e840c3d90 legacy=0x7f1e84152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.390+0000 7f1e837fe640 1 -- 172.21.15.78:0/2440609580 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84156fc0 con 0x7f1e8414d210 2024-02-19T14:52:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.390+0000 7f1e837fe640 1 --2- 172.21.15.78:0/2440609580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8414d210 0x7f1e840c4290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009870 tx=0x7f1e7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9317bb7baa06ae0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e827fc640 1 -- 172.21.15.78:0/2440609580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740317c0 con 0x7f1e8414d210 2024-02-19T14:52:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e827fc640 1 -- 172.21.15.78:0/2440609580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e74031920 con 0x7f1e8414d210 2024-02-19T14:52:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e827fc640 1 -- 172.21.15.78:0/2440609580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74031cb0 con 0x7f1e8414d210 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 -- 172.21.15.78:0/2440609580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8414d210 msgr2=0x7f1e840c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/2440609580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8414d210 0x7f1e840c4290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009870 tx=0x7f1e7402b860 comp rx=0 tx=0).stop 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 -- 172.21.15.78:0/2440609580 shutdown_connections 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/2440609580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840719e0 0x7f1e840c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/2440609580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8414d210 0x7f1e840c4290 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 -- 172.21.15.78:0/2440609580 >> 172.21.15.78:0/2440609580 conn(0x7f1e840c5050 msgr2=0x7f1e840c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 -- 172.21.15.78:0/2440609580 shutdown_connections 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.391+0000 7f1e89f3e640 1 -- 172.21.15.78:0/2440609580 wait complete. 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.392+0000 7f1e89f3e640 1 Processor -- start 2024-02-19T14:52:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 -- start start 2024-02-19T14:52:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 0x7f1e840bdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840c3d90 0x7f1e84149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8414d210 0x7f1e840be230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841e9260 con 0x7f1e840719e0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841dea90 con 0x7f1e8414d210 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e89f3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e840bfcb0 con 0x7f1e840c3d90 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 0x7f1e840bdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 0x7f1e840bdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59290/0 (socket says 172.21.15.78:59290) 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e837fe640 1 -- 172.21.15.78:0/3825460287 learned_addr learned my addr 172.21.15.78:0/3825460287 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e82ffd640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840c3d90 0x7f1e84149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.393+0000 7f1e83fff640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8414d210 0x7f1e840be230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e837fe640 1 -- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840c3d90 msgr2=0x7f1e84149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e837fe640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840c3d90 0x7f1e84149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e837fe640 1 -- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8414d210 msgr2=0x7f1e840be230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e837fe640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8414d210 0x7f1e840be230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e837fe640 1 -- 172.21.15.78:0/3825460287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740094d0 con 0x7f1e840719e0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e83fff640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8414d210 0x7f1e840be230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e837fe640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 0x7f1e840bdb10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1e74000c00 tx=0x7f1e740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740329a0 con 0x7f1e840719e0 2024-02-19T14:52:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e74032b20 con 0x7f1e840719e0 2024-02-19T14:52:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e840c3660 con 0x7f1e840719e0 2024-02-19T14:52:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740453f0 con 0x7f1e840719e0 2024-02-19T14:52:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.394+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e840bd830 con 0x7f1e840719e0 2024-02-19T14:52:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.395+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84074990 con 0x7f1e840719e0 2024-02-19T14:52:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.396+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1e7404e450 con 0x7f1e840719e0 2024-02-19T14:52:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.396+0000 7f1e6bfff640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1e7006b840 0x7f1e7006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.396+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1e740ac760 con 0x7f1e840719e0 2024-02-19T14:52:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.396+0000 7f1e82ffd640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1e7006b840 0x7f1e7006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.396+0000 7f1e82ffd640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1e7006b840 0x7f1e7006dd00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1e841de160 tx=0x7f1e78008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.400+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1e74076ed0 con 0x7f1e840719e0 2024-02-19T14:52:12.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.579+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e840c3100 con 0x7f1e840719e0 2024-02-19T14:52:12.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.579+0000 7f1e6bfff640 1 -- 172.21.15.78:0/3825460287 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e7407a590 con 0x7f1e840719e0 2024-02-19T14:52:12.582 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1e7006b840 msgr2=0x7f1e7006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1e7006b840 0x7f1e7006dd00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1e841de160 tx=0x7f1e78008040 comp rx=0 tx=0).stop 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 msgr2=0x7f1e840bdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 0x7f1e840bdb10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1e74000c00 tx=0x7f1e740046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 shutdown_connections 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1e7006b840 0x7f1e7006dd00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8414d210 0x7f1e840be230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e840c3d90 0x7f1e84149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 --2- 172.21.15.78:0/3825460287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e840719e0 0x7f1e840bdb10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 >> 172.21.15.78:0/3825460287 conn(0x7f1e840c5050 msgr2=0x7f1e8406f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 shutdown_connections 2024-02-19T14:52:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.581+0000 7f1e89f3e640 1 -- 172.21.15.78:0/3825460287 wait complete. 2024-02-19T14:52:12.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:12.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.719+0000 7faaa16d4640 1 Processor -- start 2024-02-19T14:52:12.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.719+0000 7faaa16d4640 1 -- start start 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faaa16d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faa9c152700 0x7faa9c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faaa16d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 0x7faa9c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faaa16d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c1551f0 con 0x7faa9c0c8e00 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faaa16d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9c155490 con 0x7faa9c152700 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faaa16d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c155770 con 0x7faa9c0c5730 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faa9b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7faa9c0c8e00 0x7faa9c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39324/0 (socket says 172.21.15.78:39324) 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.720+0000 7faa9b7fe640 1 -- 172.21.15.78:0/2885643797 learned_addr learned my addr 172.21.15.78:0/2885643797 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9affd640 1 --2- 172.21.15.78:0/2885643797 >> v2:172.21.15.150:3300/0 conn(0x7faa9c152700 0x7faa9c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9a7fc640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 0x7faa9c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658561929 0 0) 0x7faa9c1551f0 con 0x7faa9c0c8e00 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa88003520 con 0x7faa9c0c8e00 2024-02-19T14:52:12.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113388030 0 0) 0x7faa88003520 con 0x7faa9c0c8e00 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 msgr2=0x7faa9c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 0x7faa9c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 >> v2:172.21.15.150:3300/0 conn(0x7faa9c152700 msgr2=0x7faa9c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 --2- 172.21.15.78:0/2885643797 >> v2:172.21.15.150:3300/0 conn(0x7faa9c152700 0x7faa9c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c155a50 con 0x7faa9c0c8e00 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7faa94001670 con 0x7faa9c0c8e00 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 >> v1:172.21.15.78:6789/0 conn(0x7faa9c0c8e00 legacy=0x7faa9c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa88003970 0x7faa88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 0x7faa880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c155490 con 0x7faa88004d00 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c155770 con 0x7faa88003970 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8800b980 con 0x7faa9c0c5730 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9b7fe640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9affd640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa88003970 0x7faa88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9a7fc640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 0x7faa880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9b7fe640 1 -- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 msgr2=0x7faa880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9b7fe640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c0c5730 0x7faa880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9b7fe640 1 -- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa88003970 msgr2=0x7faa88003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9b7fe640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa88003970 0x7faa88003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.721+0000 7faa9b7fe640 1 -- 172.21.15.78:0/2885643797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88003520 con 0x7faa88004d00 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.722+0000 7faa9b7fe640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa88005180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa9400c210 tx=0x7faa880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e287921cde96cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.722+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94005ad0 con 0x7faa88004d00 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.722+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa94005c30 con 0x7faa88004d00 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.722+0000 7faa99ffb640 1 -- 172.21.15.78:0/2885643797 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9402c8d0 con 0x7faa88004d00 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faaa16d4640 1 -- 172.21.15.78:0/2885643797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 msgr2=0x7faa88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faaa16d4640 1 --2- 172.21.15.78:0/2885643797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa88005180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa9400c210 tx=0x7faa880057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faa9b7fe640 1 -- 172.21.15.78:0/2885643797 reap_dead start 2024-02-19T14:52:12.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faaa16d4640 1 -- 172.21.15.78:0/2885643797 shutdown_connections 2024-02-19T14:52:12.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faaa16d4640 1 -- 172.21.15.78:0/2885643797 >> 172.21.15.78:0/2885643797 conn(0x7faa9c0cac90 msgr2=0x7faa9c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:12.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faaa16d4640 1 -- 172.21.15.78:0/2885643797 shutdown_connections 2024-02-19T14:52:12.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.723+0000 7faaa16d4640 1 -- 172.21.15.78:0/2885643797 wait complete. 2024-02-19T14:52:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.724+0000 7faaa16d4640 1 Processor -- start 2024-02-19T14:52:12.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.724+0000 7faaa16d4640 1 -- start start 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faaa16d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa9c1ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faaa16d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c1ee3a0 0x7faa9c1e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faaa16d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c1e7c30 0x7faa9c1ec670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faaa16d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c1ec250 con 0x7faa88004d00 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faaa16d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c1e80c0 con 0x7faa9c1e7c30 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faaa16d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c156970 con 0x7faa9c1ee3a0 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa9c1ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa9c1ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59304/0 (socket says 172.21.15.78:59304) 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 -- 172.21.15.78:0/3566166134 learned_addr learned my addr 172.21.15.78:0/3566166134 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9a7fc640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c1ee3a0 0x7faa9c1e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9b7fe640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c1e7c30 0x7faa9c1ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 -- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c1ee3a0 msgr2=0x7faa9c1e76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c1ee3a0 0x7faa9c1e76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 -- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c1e7c30 msgr2=0x7faa9c1ec670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c1e7c30 0x7faa9c1ec670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 -- 172.21.15.78:0/3566166134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c1ee850 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.725+0000 7faa9affd640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa9c1ede60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009fc0 tx=0x7faa8c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.726+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c00e040 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.726+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c00f040 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.726+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c1ebb90 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.726+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c015670 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.726+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c0c3430 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.727+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c074990 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.727+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7faa8c0157d0 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.727+0000 7faa937fe640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.728+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7faa8c08b1c0 con 0x7faa88004d00 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.728+0000 7faa9a7fc640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.728+0000 7faa9a7fc640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa9c1ee780 tx=0x7faa84009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.732+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa8c0558a0 con 0x7faa88004d00 2024-02-19T14:52:12.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.908+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faa9c0c8770 con 0x7faa88004d00 2024-02-19T14:52:12.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.908+0000 7faa937fe640 1 -- 172.21.15.78:0/3566166134 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7faa8c058f60 con 0x7faa88004d00 2024-02-19T14:52:12.911 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 msgr2=0x7faa7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa9c1ee780 tx=0x7faa84009190 comp rx=0 tx=0).stop 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 msgr2=0x7faa9c1ede60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa9c1ede60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009fc0 tx=0x7faa8c00f580 comp rx=0 tx=0).stop 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 shutdown_connections 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c1e7c30 0x7faa9c1ec670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa9c1ee3a0 0x7faa9c1e76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 --2- 172.21.15.78:0/3566166134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa88004d00 0x7faa9c1ede60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 >> 172.21.15.78:0/3566166134 conn(0x7faa9c0cac90 msgr2=0x7faa9c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 shutdown_connections 2024-02-19T14:52:12.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:12.910+0000 7faaa16d4640 1 -- 172.21.15.78:0/3566166134 wait complete. 2024-02-19T14:52:12.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:13.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.037+0000 7f628658a640 1 Processor -- start 2024-02-19T14:52:13.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.037+0000 7f628658a640 1 -- start start 2024-02-19T14:52:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f628658a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62800c78a0 0x7f62800c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f628658a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6280072ee0 0x7f628014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f628658a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628014f8f0 con 0x7f628014ef50 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f628658a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62800c35c0 con 0x7f6280072ee0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f628658a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62800c38a0 con 0x7f62800c78a0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6280072ee0 0x7f628014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6280072ee0 0x7f628014f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59308/0 (socket says 172.21.15.78:59308) 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 -- 172.21.15.78:0/4044214100 learned_addr learned my addr 172.21.15.78:0/4044214100 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f627ffff640 1 --2- 172.21.15.78:0/4044214100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62800c78a0 0x7f62800c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 -- 172.21.15.78:0/4044214100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62800c78a0 msgr2=0x7f62800c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 --2- 172.21.15.78:0/4044214100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62800c78a0 0x7f62800c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 -- 172.21.15.78:0/4044214100 >> v1:172.21.15.150:6789/0 conn(0x7f628014ef50 legacy=0x7f62800727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.038+0000 7f6284b27640 1 -- 172.21.15.78:0/4044214100 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800c3b80 con 0x7f6280072ee0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.039+0000 7f6284b27640 1 --2- 172.21.15.78:0/4044214100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f628014f3b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6278009e50 tx=0x7f627802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9e223dbd3d13f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.039+0000 7f627effd640 1 -- 172.21.15.78:0/4044214100 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627802cb00 con 0x7f6280072ee0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.039+0000 7f627effd640 1 -- 172.21.15.78:0/4044214100 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627802cc60 con 0x7f6280072ee0 2024-02-19T14:52:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.039+0000 7f627effd640 1 -- 172.21.15.78:0/4044214100 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780343f0 con 0x7f6280072ee0 2024-02-19T14:52:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 -- 172.21.15.78:0/4044214100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 msgr2=0x7f628014f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 --2- 172.21.15.78:0/4044214100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f628014f3b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6278009e50 tx=0x7f627802bf60 comp rx=0 tx=0).stop 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 -- 172.21.15.78:0/4044214100 shutdown_connections 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 --2- 172.21.15.78:0/4044214100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f628014f3b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 --2- 172.21.15.78:0/4044214100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62800c78a0 0x7f62800c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 -- 172.21.15.78:0/4044214100 >> 172.21.15.78:0/4044214100 conn(0x7f6280066e70 msgr2=0x7f62800a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 -- 172.21.15.78:0/4044214100 shutdown_connections 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.040+0000 7f628658a640 1 -- 172.21.15.78:0/4044214100 wait complete. 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.041+0000 7f628658a640 1 Processor -- start 2024-02-19T14:52:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.041+0000 7f628658a640 1 -- start start 2024-02-19T14:52:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f62801e89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62800c78a0 0x7f62800bea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628014ef50 0x7f62800c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628006ce10 con 0x7f6280072ee0 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628006ae80 con 0x7f62800c78a0 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62800c0560 con 0x7f628014ef50 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f62801e89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f62801e89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59314/0 (socket says 172.21.15.78:59314) 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 -- 172.21.15.78:0/1503029531 learned_addr learned my addr 172.21.15.78:0/1503029531 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f6284b27640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628014ef50 0x7f62800c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627f7fe640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62800c78a0 0x7f62800bea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 -- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628014ef50 msgr2=0x7f62800c0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628014ef50 0x7f62800c0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 -- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62800c78a0 msgr2=0x7f62800bea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62800c78a0 0x7f62800bea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 -- 172.21.15.78:0/1503029531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278009b30 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627ffff640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f62801e89d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f627000d830 tx=0x7f627000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627000b6c0 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627000b820 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62800bf430 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627000bbb0 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.042+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62801e8f10 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.043+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62800be8d0 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.044+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f627001e460 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.044+0000 7f627cff9640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f626c06b840 0x7f626c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.044+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f627008abd0 con 0x7f6280072ee0 2024-02-19T14:52:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.044+0000 7f627f7fe640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f626c06b840 0x7f626c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.045+0000 7f627f7fe640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f626c06b840 0x7f626c06dd00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6274007fd0 tx=0x7f6274009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.053+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6270055340 con 0x7f6280072ee0 2024-02-19T14:52:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.230+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6280074990 con 0x7f6280072ee0 2024-02-19T14:52:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.230+0000 7f627cff9640 1 -- 172.21.15.78:0/1503029531 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6270058a00 con 0x7f6280072ee0 2024-02-19T14:52:13.233 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f626c06b840 msgr2=0x7f626c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f626c06b840 0x7f626c06dd00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6274007fd0 tx=0x7f6274009040 comp rx=0 tx=0).stop 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 msgr2=0x7f62801e89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f62801e89d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f627000d830 tx=0x7f627000dcf0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 shutdown_connections 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f626c06b840 0x7f626c06dd00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628014ef50 0x7f62800c0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62800c78a0 0x7f62800bea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 --2- 172.21.15.78:0/1503029531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6280072ee0 0x7f62801e89d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 >> 172.21.15.78:0/1503029531 conn(0x7f6280066e70 msgr2=0x7f6280150740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:13.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 shutdown_connections 2024-02-19T14:52:13.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.232+0000 7f628658a640 1 -- 172.21.15.78:0/1503029531 wait complete. 2024-02-19T14:52:13.246 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T14:52:13.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.345+0000 7f522016c640 1 Processor -- start 2024-02-19T14:52:13.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.345+0000 7f522016c640 1 -- start start 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f522016c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f521814ef50 0x7f521814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f522016c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52180727d0 0x7f5218072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f522016c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180c35c0 con 0x7f52180c78a0 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f522016c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52180c4bc0 con 0x7f52180727d0 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f522016c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521814f920 con 0x7f521814ef50 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f521e709640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52180727d0 0x7f5218072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f521e709640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52180727d0 0x7f5218072bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59318/0 (socket says 172.21.15.78:59318) 2024-02-19T14:52:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f521e709640 1 -- 172.21.15.78:0/2544290220 learned_addr learned my addr 172.21.15.78:0/2544290220 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.346+0000 7f521d707640 1 --2- 172.21.15.78:0/2544290220 >> v2:172.21.15.150:3300/0 conn(0x7f521814ef50 0x7f521814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.347+0000 7f521e709640 1 -- 172.21.15.78:0/2544290220 >> v2:172.21.15.150:3300/0 conn(0x7f521814ef50 msgr2=0x7f521814f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.347+0000 7f521e709640 1 --2- 172.21.15.78:0/2544290220 >> v2:172.21.15.150:3300/0 conn(0x7f521814ef50 0x7f521814f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.347+0000 7f521e709640 1 -- 172.21.15.78:0/2544290220 >> v1:172.21.15.150:6789/0 conn(0x7f52180c78a0 legacy=0x7f52180c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.347+0000 7f521e709640 1 -- 172.21.15.78:0/2544290220 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180c38a0 con 0x7f52180727d0 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.347+0000 7f521e709640 1 --2- 172.21.15.78:0/2544290220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f5218072bf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52180c1eb0 tx=0x7f521002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694034cdc796201b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.347+0000 7f521cf06640 1 -- 172.21.15.78:0/2544290220 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210030860 con 0x7f52180727d0 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f521cf06640 1 -- 172.21.15.78:0/2544290220 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52100309c0 con 0x7f52180727d0 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f521cf06640 1 -- 172.21.15.78:0/2544290220 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210030d50 con 0x7f52180727d0 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 -- 172.21.15.78:0/2544290220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 msgr2=0x7f5218072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 --2- 172.21.15.78:0/2544290220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f5218072bf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f52180c1eb0 tx=0x7f521002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 -- 172.21.15.78:0/2544290220 shutdown_connections 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 --2- 172.21.15.78:0/2544290220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f5218072bf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 --2- 172.21.15.78:0/2544290220 >> v2:172.21.15.150:3300/0 conn(0x7f521814ef50 0x7f521814f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 -- 172.21.15.78:0/2544290220 >> 172.21.15.78:0/2544290220 conn(0x7f5218066e70 msgr2=0x7f52180a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 -- 172.21.15.78:0/2544290220 shutdown_connections 2024-02-19T14:52:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.348+0000 7f522016c640 1 -- 172.21.15.78:0/2544290220 wait complete. 2024-02-19T14:52:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.349+0000 7f522016c640 1 Processor -- start 2024-02-19T14:52:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.349+0000 7f522016c640 1 -- start start 2024-02-19T14:52:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f522016c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f52181ec210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f522016c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52180c78a0 0x7f521806b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f522016c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521814ef50 0x7f52181e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f522016c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52180c0b50 con 0x7f52180727d0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f522016c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52181d9400 con 0x7f52180c78a0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f522016c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180c0340 con 0x7f521814ef50 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f52181ec210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f52181ec210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59330/0 (socket says 172.21.15.78:59330) 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 -- 172.21.15.78:0/229119626 learned_addr learned my addr 172.21.15.78:0/229119626 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521d707640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52180c78a0 0x7f521806b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521e709640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521814ef50 0x7f52181e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 -- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521814ef50 msgr2=0x7f52181e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521814ef50 0x7f52181e8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 -- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52180c78a0 msgr2=0x7f521806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52180c78a0 0x7f521806b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 -- 172.21.15.78:0/229119626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100094d0 con 0x7f52180727d0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.350+0000 7f521df08640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f52181ec210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f521400c790 tx=0x7f521400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.351+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400cea0 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.351+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214006d50 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.351+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52181da3b0 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.351+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214003bb0 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.351+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52181ec7b0 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.352+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5218074990 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.352+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5214003d10 con 0x7f52180727d0 2024-02-19T14:52:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.352+0000 7f520e7fc640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51fc06b840 0x7f51fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.353+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f521408a040 con 0x7f52180727d0 2024-02-19T14:52:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.353+0000 7f521d707640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51fc06b840 0x7f51fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.353+0000 7f521d707640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51fc06b840 0x7f51fc06dd00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f52180c74a0 tx=0x7f5208008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.357+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f52140582e0 con 0x7f52180727d0 2024-02-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.545+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f52180c0930 con 0x7f52180727d0 2024-02-19T14:52:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.546+0000 7f520e7fc640 1 -- 172.21.15.78:0/229119626 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5214057e70 con 0x7f52180727d0 2024-02-19T14:52:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:listed 0 entries 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51fc06b840 msgr2=0x7f51fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51fc06b840 0x7f51fc06dd00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f52180c74a0 tx=0x7f5208008040 comp rx=0 tx=0).stop 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 msgr2=0x7f52181ec210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f52181ec210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f521400c790 tx=0x7f521400cc50 comp rx=0 tx=0).stop 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 shutdown_connections 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51fc06b840 0x7f51fc06dd00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521814ef50 0x7f52181e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52180c78a0 0x7f521806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 --2- 172.21.15.78:0/229119626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52180727d0 0x7f52181ec210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 >> 172.21.15.78:0/229119626 conn(0x7f5218066e70 msgr2=0x7f52180c5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 shutdown_connections 2024-02-19T14:52:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.548+0000 7f522016c640 1 -- 172.21.15.78:0/229119626 wait complete. 2024-02-19T14:52:13.562 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-02-19T14:52:13.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:13.562 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-02-19T14:52:13.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[44948]: fuse finished with error 0 and tester_r 0 2024-02-19T14:52:13.589 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:52:13.678 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:52:13.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:13.679 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-02-19T14:52:13.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:13.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.848+0000 7f8087bbc640 1 Processor -- start 2024-02-19T14:52:13.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.848+0000 7f8087bbc640 1 -- start start 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8087bbc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8080158670 0x7f8080158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8087bbc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80800c1d80 con 0x7f80800acb50 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8087bbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808006c5a0 con 0x7f808006da10 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8087bbc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808006caf0 con 0x7f8080158670 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8086159640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8080158670 0x7f8080158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8086159640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8080158670 0x7f8080158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59342/0 (socket says 172.21.15.78:59342) 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8086159640 1 -- 172.21.15.78:0/4205753885 learned_addr learned my addr 172.21.15.78:0/4205753885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8086159640 1 -- 172.21.15.78:0/4205753885 >> v1:172.21.15.150:6789/0 conn(0x7f808006da10 legacy=0x7f80800723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8086159640 1 -- 172.21.15.78:0/4205753885 >> v1:172.21.15.78:6789/0 conn(0x7f80800acb50 legacy=0x7f8080155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.849+0000 7f8086159640 1 -- 172.21.15.78:0/4205753885 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808006c810 con 0x7f8080158670 2024-02-19T14:52:13.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.850+0000 7f8086159640 1 --2- 172.21.15.78:0/4205753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8080158670 0x7f8080158aa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8078009e50 tx=0x7f807802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffde6f1ec94637c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.850+0000 7f8084956640 1 -- 172.21.15.78:0/4205753885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807802cb20 con 0x7f8080158670 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.850+0000 7f8084956640 1 -- 172.21.15.78:0/4205753885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807802cc80 con 0x7f8080158670 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.850+0000 7f8084956640 1 -- 172.21.15.78:0/4205753885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780343f0 con 0x7f8080158670 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 -- 172.21.15.78:0/4205753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8080158670 msgr2=0x7f8080158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 --2- 172.21.15.78:0/4205753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8080158670 0x7f8080158aa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8078009e50 tx=0x7f807802bf80 comp rx=0 tx=0).stop 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 -- 172.21.15.78:0/4205753885 shutdown_connections 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 --2- 172.21.15.78:0/4205753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8080158670 0x7f8080158aa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 -- 172.21.15.78:0/4205753885 >> 172.21.15.78:0/4205753885 conn(0x7f80800718e0 msgr2=0x7f8080071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 -- 172.21.15.78:0/4205753885 shutdown_connections 2024-02-19T14:52:13.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.851+0000 7f8087bbc640 1 -- 172.21.15.78:0/4205753885 wait complete. 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 Processor -- start 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 -- start start 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 0x7f8080139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80800acb50 0x7f80800c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8080158670 0x7f8080137f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801356a0 con 0x7f808006da10 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8080135e10 con 0x7f80800acb50 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8087bbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8080137b90 con 0x7f8080158670 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.852+0000 7f8085958640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 0x7f8080139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 0x7f8080139c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59354/0 (socket says 172.21.15.78:59354) 2024-02-19T14:52:13.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 -- 172.21.15.78:0/127132041 learned_addr learned my addr 172.21.15.78:0/127132041 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8086159640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8080158670 0x7f8080137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085157640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80800acb50 0x7f80800c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 -- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8080158670 msgr2=0x7f8080137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8080158670 0x7f8080137f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 -- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80800acb50 msgr2=0x7f80800c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80800acb50 0x7f80800c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 -- 172.21.15.78:0/127132041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078009b30 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8085958640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 0x7f8080139c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f807c00c400 tx=0x7f807c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0052d0 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8080136740 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c005430 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c003830 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.853+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80801358f0 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.854+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8080136c60 con 0x7f808006da10 2024-02-19T14:52:13.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.854+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f807c003990 con 0x7f808006da10 2024-02-19T14:52:13.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.855+0000 7f80767fc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f806406b840 0x7f806406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:13.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.855+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f807c089b00 con 0x7f808006da10 2024-02-19T14:52:13.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.855+0000 7f8085157640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f806406b840 0x7f806406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:13.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.855+0000 7f8085157640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f806406b840 0x7f806406dd00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8070007fd0 tx=0x7f8070009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:13.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:13.859+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f807c054160 con 0x7f808006da10 2024-02-19T14:52:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.183+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f808006f780 con 0x7f808006da10 2024-02-19T14:52:14.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.184+0000 7f80767fc640 1 -- 172.21.15.78:0/127132041 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f807c057820 con 0x7f808006da10 2024-02-19T14:52:14.187 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:14.187 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f806406b840 msgr2=0x7f806406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f806406b840 0x7f806406dd00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8070007fd0 tx=0x7f8070009040 comp rx=0 tx=0).stop 2024-02-19T14:52:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 msgr2=0x7f8080139c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 0x7f8080139c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f807c00c400 tx=0x7f807c00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 shutdown_connections 2024-02-19T14:52:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f806406b840 0x7f806406dd00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8080158670 0x7f8080137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80800acb50 0x7f80800c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 --2- 172.21.15.78:0/127132041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f808006da10 0x7f8080139c90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 >> 172.21.15.78:0/127132041 conn(0x7f80800718e0 msgr2=0x7f808013cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 shutdown_connections 2024-02-19T14:52:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.186+0000 7f8087bbc640 1 -- 172.21.15.78:0/127132041 wait complete. 2024-02-19T14:52:14.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.298+0000 7f91851c2640 1 Processor -- start 2024-02-19T14:52:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.298+0000 7f91851c2640 1 -- start start 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f91851c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f9180159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f91851c2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918006ffa0 con 0x7f91800c78a0 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f91851c2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918006fc70 con 0x7f9180155010 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f91851c2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918006b720 con 0x7f91800c4d80 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f917ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f91800c78a0 0x7f91800c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39350/0 (socket says 172.21.15.78:39350) 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f917ed9d640 1 -- 172.21.15.78:0/1252781618 learned_addr learned my addr 172.21.15.78:0/1252781618 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f917e59c640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f9180159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.299+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165091758 0 0) 0x7f918006ffa0 con 0x7f91800c78a0 2024-02-19T14:52:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c003520 con 0x7f91800c78a0 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484655320 0 0) 0x7f916c003520 con 0x7f91800c78a0 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 msgr2=0x7f9180159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f9180159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 >> v1:172.21.15.150:6789/0 conn(0x7f9180155010 legacy=0x7f91801523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918006b930 con 0x7f91800c78a0 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9178001670 con 0x7f91800c78a0 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 >> v1:172.21.15.78:6789/0 conn(0x7f91800c78a0 legacy=0x7f91800c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f916c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f916c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c004d00 0x7f916c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918006b720 con 0x7f916c003f20 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c00bb10 con 0x7f916c004d00 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c00bcd0 con 0x7f91800c4d80 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917e59c640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f916c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917ed9d640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f916c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917f59e640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c004d00 0x7f916c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917e59c640 1 -- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 msgr2=0x7f916c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917e59c640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f916c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917e59c640 1 -- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c004d00 msgr2=0x7f916c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917e59c640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c004d00 0x7f916c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917e59c640 1 -- 172.21.15.78:0/1252781618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c003520 con 0x7f916c003f20 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.300+0000 7f917ed9d640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91800c4d80 0x7f916c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.301+0000 7f917e59c640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f916c0047c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ac0 tx=0x7f916c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63aaba4f2ce78ee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.301+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916802c6b0 con 0x7f916c003f20 2024-02-19T14:52:14.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.301+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916802c810 con 0x7f916c003f20 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.301+0000 7f917dd9b640 1 -- 172.21.15.78:0/1252781618 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916802cba0 con 0x7f916c003f20 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f91851c2640 1 -- 172.21.15.78:0/1252781618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 msgr2=0x7f916c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f91851c2640 1 --2- 172.21.15.78:0/1252781618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f916c0047c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ac0 tx=0x7f916c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f917f59e640 1 -- 172.21.15.78:0/1252781618 reap_dead start 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f91851c2640 1 -- 172.21.15.78:0/1252781618 shutdown_connections 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f91851c2640 1 -- 172.21.15.78:0/1252781618 >> 172.21.15.78:0/1252781618 conn(0x7f9180066e70 msgr2=0x7f91800a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f91851c2640 1 -- 172.21.15.78:0/1252781618 shutdown_connections 2024-02-19T14:52:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.302+0000 7f91851c2640 1 -- 172.21.15.78:0/1252781618 wait complete. 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.303+0000 7f91851c2640 1 Processor -- start 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.304+0000 7f91851c2640 1 -- start start 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.304+0000 7f91851c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f91801ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f91851c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9180151070 0x7f918014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f91851c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9180158d20 0x7f91801ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f91851c2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91801519c0 con 0x7f916c003f20 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f91851c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91801eadf0 con 0x7f9180158d20 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f91851c2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91801ee230 con 0x7f9180151070 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f91801ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f91801ea8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59366/0 (socket says 172.21.15.78:59366) 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 -- 172.21.15.78:0/2138832339 learned_addr learned my addr 172.21.15.78:0/2138832339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917f59e640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9180158d20 0x7f91801ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917e59c640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9180151070 0x7f918014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 -- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9180151070 msgr2=0x7f918014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9180151070 0x7f918014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 -- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9180158d20 msgr2=0x7f91801ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9180158d20 0x7f91801ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 -- 172.21.15.78:0/2138832339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918014fe60 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.305+0000 7f917ed9d640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f91801ea8b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f91780072f0 tx=0x7f9178004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.306+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917800fb40 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.306+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917800fca0 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.306+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178012350 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.306+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9180150140 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.306+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91801edec0 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.307+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180074990 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.307+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f91780124b0 con 0x7f916c003f20 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.308+0000 7f91777fe640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f916006b840 0x7f916006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.308+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9178089e60 con 0x7f916c003f20 2024-02-19T14:52:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.308+0000 7f917e59c640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f916006b840 0x7f916006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.308+0000 7f917e59c640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f916006b840 0x7f916006dd00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ac0 tx=0x7f9168030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:14.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.314+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9178054680 con 0x7f916c003f20 2024-02-19T14:52:14.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.590+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f918014e010 con 0x7f916c003f20 2024-02-19T14:52:14.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.591+0000 7f91777fe640 1 -- 172.21.15.78:0/2138832339 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f917801a020 con 0x7f916c003f20 2024-02-19T14:52:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:14.594 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:14.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f916006b840 msgr2=0x7f916006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f916006b840 0x7f916006dd00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ac0 tx=0x7f9168030000 comp rx=0 tx=0).stop 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 msgr2=0x7f91801ea8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f91801ea8b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f91780072f0 tx=0x7f9178004f90 comp rx=0 tx=0).stop 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 shutdown_connections 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f916006b840 0x7f916006dd00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9180158d20 0x7f91801ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9180151070 0x7f918014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 --2- 172.21.15.78:0/2138832339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f916c003f20 0x7f91801ea8b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 >> 172.21.15.78:0/2138832339 conn(0x7f9180066e70 msgr2=0x7f9180155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 shutdown_connections 2024-02-19T14:52:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.594+0000 7f91851c2640 1 -- 172.21.15.78:0/2138832339 wait complete. 2024-02-19T14:52:14.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:52:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.712+0000 7fdd556a4640 1 Processor -- start 2024-02-19T14:52:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.712+0000 7fdd556a4640 1 -- start start 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd556a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdd50149790 0x7fdd5006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd556a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdd500723d0 0x7fdd50072800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd556a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd501530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd556a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd50149b70 con 0x7fdd50149790 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd556a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5006f420 con 0x7fdd500723d0 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd556a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd500ab760 con 0x7fdd50152c30 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdd50149790 0x7fdd5006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdd50149790 0x7fdd5006ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59378/0 (socket says 172.21.15.78:59378) 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 -- 172.21.15.78:0/654582056 learned_addr learned my addr 172.21.15.78:0/654582056 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 -- 172.21.15.78:0/654582056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 msgr2=0x7fdd501530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 --2- 172.21.15.78:0/654582056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd501530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4e7fc640 1 --2- 172.21.15.78:0/654582056 >> v2:172.21.15.150:3300/0 conn(0x7fdd500723d0 0x7fdd50072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 -- 172.21.15.78:0/654582056 >> v2:172.21.15.150:3300/0 conn(0x7fdd500723d0 msgr2=0x7fdd50072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 --2- 172.21.15.78:0/654582056 >> v2:172.21.15.150:3300/0 conn(0x7fdd500723d0 0x7fdd50072800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.713+0000 7fdd4effd640 1 -- 172.21.15.78:0/654582056 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500aba40 con 0x7fdd50149790 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.714+0000 7fdd4effd640 1 --2- 172.21.15.78:0/654582056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd50149790 0x7fdd5006ed40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd50073440 tx=0x7fdd4002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8568dd3eb98943f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.714+0000 7fdd4dffb640 1 -- 172.21.15.78:0/654582056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40031750 con 0x7fdd50149790 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.714+0000 7fdd4dffb640 1 -- 172.21.15.78:0/654582056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd400318b0 con 0x7fdd50149790 2024-02-19T14:52:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.714+0000 7fdd4dffb640 1 -- 172.21.15.78:0/654582056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40031c40 con 0x7fdd50149790 2024-02-19T14:52:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 -- 172.21.15.78:0/654582056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd50149790 msgr2=0x7fdd5006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 --2- 172.21.15.78:0/654582056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd50149790 0x7fdd5006ed40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdd50073440 tx=0x7fdd4002b7f0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 -- 172.21.15.78:0/654582056 shutdown_connections 2024-02-19T14:52:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 --2- 172.21.15.78:0/654582056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd501530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 --2- 172.21.15.78:0/654582056 >> v2:172.21.15.150:3300/0 conn(0x7fdd500723d0 0x7fdd50072800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 --2- 172.21.15.78:0/654582056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd50149790 0x7fdd5006ed40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 -- 172.21.15.78:0/654582056 >> 172.21.15.78:0/654582056 conn(0x7fdd500718e0 msgr2=0x7fdd50071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 -- 172.21.15.78:0/654582056 shutdown_connections 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.715+0000 7fdd556a4640 1 -- 172.21.15.78:0/654582056 wait complete. 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.716+0000 7fdd556a4640 1 Processor -- start 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.716+0000 7fdd556a4640 1 -- start start 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd556a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 0x7fdd501ddd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd556a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd50149790 0x7fdd5014beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd556a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd5014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd556a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5014cdb0 con 0x7fdd500723d0 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd556a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd501debb0 con 0x7fdd50149790 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd556a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd501dee90 con 0x7fdd50152c30 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 0x7fdd501ddd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 0x7fdd501ddd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59388/0 (socket says 172.21.15.78:59388) 2024-02-19T14:52:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 -- 172.21.15.78:0/3374474118 learned_addr learned my addr 172.21.15.78:0/3374474118 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4e7fc640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd50149790 0x7fdd5014beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 -- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 msgr2=0x7fdd5014abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4f7fe640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd5014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd5014abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 -- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd50149790 msgr2=0x7fdd5014beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd50149790 0x7fdd5014beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 -- 172.21.15.78:0/3374474118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd400094d0 con 0x7fdd500723d0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4f7fe640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd5014abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4e7fc640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd50149790 0x7fdd5014beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd4effd640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 0x7fdd501ddd50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdd40000c00 tx=0x7fdd400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.717+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40033c80 con 0x7fdd500723d0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.718+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd40033de0 con 0x7fdd500723d0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.718+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5014c3f0 con 0x7fdd500723d0 2024-02-19T14:52:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.718+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4003c640 con 0x7fdd500723d0 2024-02-19T14:52:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.718+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd500c6810 con 0x7fdd500723d0 2024-02-19T14:52:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.719+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd501dea10 con 0x7fdd500723d0 2024-02-19T14:52:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.719+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdd4003a040 con 0x7fdd500723d0 2024-02-19T14:52:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.719+0000 7fdd377fe640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdd3c06b810 0x7fdd3c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.719+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdd400ac1c0 con 0x7fdd500723d0 2024-02-19T14:52:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.720+0000 7fdd4e7fc640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdd3c06b810 0x7fdd3c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.720+0000 7fdd4e7fc640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdd3c06b810 0x7fdd3c06dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400b820 tx=0x7fdd44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:14.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.724+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdd501dea10 con 0x7fdd500723d0 2024-02-19T14:52:14.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:14.940+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdd501dea10 con 0x7fdd500723d0 2024-02-19T14:52:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.313+0000 7fdd377fe640 1 -- 172.21.15.78:0/3374474118 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdd501dea10 con 0x7fdd500723d0 2024-02-19T14:52:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdd3c06b810 msgr2=0x7fdd3c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdd3c06b810 0x7fdd3c06dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400b820 tx=0x7fdd44008040 comp rx=0 tx=0).stop 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 msgr2=0x7fdd501ddd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 0x7fdd501ddd50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdd40000c00 tx=0x7fdd400046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 shutdown_connections 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdd3c06b810 0x7fdd3c06dcd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdd50152c30 0x7fdd5014abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd50149790 0x7fdd5014beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 --2- 172.21.15.78:0/3374474118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd500723d0 0x7fdd501ddd50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 >> 172.21.15.78:0/3374474118 conn(0x7fdd500718e0 msgr2=0x7fdd50071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 shutdown_connections 2024-02-19T14:52:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.316+0000 7fdd556a4640 1 -- 172.21.15.78:0/3374474118 wait complete. 2024-02-19T14:52:15.329 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-02-19T14:52:15.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:52:15.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.430+0000 7f2d57d3e640 1 Processor -- start 2024-02-19T14:52:15.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.430+0000 7f2d57d3e640 1 -- start start 2024-02-19T14:52:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d57d3e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2d500c5560 0x7f2d500c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d57d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d50070720 0x7f2d50070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:15.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d57d3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d500c65a0 con 0x7f2d500c7910 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d57d3e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d500c7530 con 0x7f2d500c5560 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d57d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501537b0 con 0x7f2d50070720 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2d500c5560 0x7f2d500c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2d500c5560 0x7f2d500c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59398/0 (socket says 172.21.15.78:59398) 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 -- 172.21.15.78:0/3733148590 learned_addr learned my addr 172.21.15.78:0/3733148590 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d562db640 1 --2- 172.21.15.78:0/3733148590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d50070720 0x7f2d50070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 -- 172.21.15.78:0/3733148590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d50070720 msgr2=0x7f2d50070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 --2- 172.21.15.78:0/3733148590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d50070720 0x7f2d50070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 -- 172.21.15.78:0/3733148590 >> v1:172.21.15.150:6789/0 conn(0x7f2d500c7910 legacy=0x7f2d50154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.431+0000 7f2d55ada640 1 -- 172.21.15.78:0/3733148590 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d50153a90 con 0x7f2d500c5560 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.432+0000 7f2d55ada640 1 --2- 172.21.15.78:0/3733148590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c5560 0x7f2d500c6ff0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009870 tx=0x7f2d4c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad6ffdbe7153f6e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.432+0000 7f2d54ad8640 1 -- 172.21.15.78:0/3733148590 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0317c0 con 0x7f2d500c5560 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.432+0000 7f2d54ad8640 1 -- 172.21.15.78:0/3733148590 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c031920 con 0x7f2d500c5560 2024-02-19T14:52:15.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.432+0000 7f2d54ad8640 1 -- 172.21.15.78:0/3733148590 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c031cb0 con 0x7f2d500c5560 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3733148590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c5560 msgr2=0x7f2d500c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3733148590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c5560 0x7f2d500c6ff0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009870 tx=0x7f2d4c02b860 comp rx=0 tx=0).stop 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3733148590 shutdown_connections 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3733148590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d50070720 0x7f2d50070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3733148590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c5560 0x7f2d500c6ff0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3733148590 >> 172.21.15.78:0/3733148590 conn(0x7f2d5006b4b0 msgr2=0x7f2d500c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3733148590 shutdown_connections 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.433+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3733148590 wait complete. 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.434+0000 7f2d57d3e640 1 Processor -- start 2024-02-19T14:52:15.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.434+0000 7f2d57d3e640 1 -- start start 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d57d3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d50070720 0x7f2d501487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d57d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d500c5560 0x7f2d501ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d57d3e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 0x7f2d501e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d57d3e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501dc910 con 0x7f2d500c7910 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d57d3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501dc390 con 0x7f2d50070720 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d57d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d501ec520 con 0x7f2d500c5560 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 0x7f2d501e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 0x7f2d501e9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59414/0 (socket says 172.21.15.78:59414) 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 -- 172.21.15.78:0/3516876759 learned_addr learned my addr 172.21.15.78:0/3516876759 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d552d9640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d500c5560 0x7f2d501ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d55ada640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d50070720 0x7f2d501487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 -- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d500c5560 msgr2=0x7f2d501ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d500c5560 0x7f2d501ebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 -- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d50070720 msgr2=0x7f2d501487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d50070720 0x7f2d501487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 -- 172.21.15.78:0/3516876759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c0094d0 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d552d9640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d500c5560 0x7f2d501ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d562db640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 0x7f2d501e9cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d48007e60 tx=0x7f2d4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48014420 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.435+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d480145a0 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.436+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d500c5ac0 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.436+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48014950 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.436+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d501ec800 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.436+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d50074990 con 0x7f2d500c7910 2024-02-19T14:52:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.437+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2d48014ab0 con 0x7f2d500c7910 2024-02-19T14:52:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.437+0000 7f2d467fc640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d3406b810 0x7f2d3406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.437+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2d4808ade0 con 0x7f2d500c7910 2024-02-19T14:52:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.437+0000 7f2d55ada640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d3406b810 0x7f2d3406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.438+0000 7f2d55ada640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d3406b810 0x7f2d3406dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c036000 tx=0x7f2d4c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:15.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.442+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d480554c0 con 0x7f2d500c7910 2024-02-19T14:52:15.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:15.693+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2d500c76f0 con 0x7f2d500c7910 2024-02-19T14:52:16.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.320+0000 7f2d467fc640 1 -- 172.21.15.78:0/3516876759 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d48058b80 con 0x7f2d500c7910 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d3406b810 msgr2=0x7f2d3406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d3406b810 0x7f2d3406dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c036000 tx=0x7f2d4c03d040 comp rx=0 tx=0).stop 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 msgr2=0x7f2d501e9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 0x7f2d501e9cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d48007e60 tx=0x7f2d4800e4b0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 shutdown_connections 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d3406b810 0x7f2d3406dcd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d500c7910 0x7f2d501e9cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d500c5560 0x7f2d501ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 --2- 172.21.15.78:0/3516876759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d50070720 0x7f2d501487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 >> 172.21.15.78:0/3516876759 conn(0x7f2d5006b4b0 msgr2=0x7f2d500c4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 shutdown_connections 2024-02-19T14:52:16.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.322+0000 7f2d57d3e640 1 -- 172.21.15.78:0/3516876759 wait complete. 2024-02-19T14:52:16.339 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:52:16.339 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:16.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.437+0000 7ffb004a4640 1 Processor -- start 2024-02-19T14:52:16.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.437+0000 7ffb004a4640 1 -- start start 2024-02-19T14:52:16.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.438+0000 7ffb004a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffaf80c79f0 0x7ffaf80c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.438+0000 7ffb004a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80c27e0 con 0x7ffaf8155da0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.438+0000 7ffb004a4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80c3a90 con 0x7ffaf80c6960 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.438+0000 7ffb004a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80c3d70 con 0x7ffaf80c79f0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.438+0000 7ffafe240640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ffaf8155da0 0x7ffaf80c72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39364/0 (socket says 172.21.15.78:39364) 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.438+0000 7ffafe240640 1 -- 172.21.15.78:0/2418166607 learned_addr learned my addr 172.21.15.78:0/2418166607 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafea41640 1 --2- 172.21.15.78:0/2418166607 >> v2:172.21.15.150:3300/0 conn(0x7ffaf80c79f0 0x7ffaf80c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298545306 0 0) 0x7ffaf80c27e0 con 0x7ffaf8155da0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae4003520 con 0x7ffaf8155da0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097322754 0 0) 0x7ffae4003520 con 0x7ffaf8155da0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 >> v2:172.21.15.150:3300/0 conn(0x7ffaf80c79f0 msgr2=0x7ffaf80c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 --2- 172.21.15.78:0/2418166607 >> v2:172.21.15.150:3300/0 conn(0x7ffaf80c79f0 0x7ffaf80c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 >> v1:172.21.15.150:6789/0 conn(0x7ffaf80c6960 legacy=0x7ffaf80c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf806d070 con 0x7ffaf8155da0 2024-02-19T14:52:16.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7ffaf4001670 con 0x7ffaf8155da0 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 >> v1:172.21.15.78:6789/0 conn(0x7ffaf8155da0 legacy=0x7ffaf80c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae4003970 0x7ffae4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40047c0 0x7ffae4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4005180 0x7ffae400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80c3d70 con 0x7ffae4003970 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae400c120 con 0x7ffae40047c0 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae400c2e0 con 0x7ffae4005180 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafe240640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae4003970 0x7ffae4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafda3f640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40047c0 0x7ffae4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafea41640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4005180 0x7ffae400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafe240640 1 -- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4005180 msgr2=0x7ffae400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafe240640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4005180 0x7ffae400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafe240640 1 -- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40047c0 msgr2=0x7ffae4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:16.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafe240640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40047c0 0x7ffae4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.439+0000 7ffafe240640 1 -- 172.21.15.78:0/2418166607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4003520 con 0x7ffae4003970 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.440+0000 7ffafea41640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4005180 0x7ffae400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.440+0000 7ffafea41640 1 -- 172.21.15.78:0/2418166607 reap_dead start 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.441+0000 7ffafe240640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae4003970 0x7ffae4003d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400c240 tx=0x7ffae40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e5dd21cc9bac6f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.441+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4005ad0 con 0x7ffae4003970 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.441+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4005c30 con 0x7ffae4003970 2024-02-19T14:52:16.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.441+0000 7ffafd23e640 1 -- 172.21.15.78:0/2418166607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf402ca90 con 0x7ffae4003970 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.441+0000 7ffb004a4640 1 -- 172.21.15.78:0/2418166607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae4003970 msgr2=0x7ffae4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.441+0000 7ffb004a4640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae4003970 0x7ffae4003d90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400c240 tx=0x7ffae40057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.442+0000 7ffb004a4640 1 -- 172.21.15.78:0/2418166607 shutdown_connections 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.442+0000 7ffb004a4640 1 --2- 172.21.15.78:0/2418166607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae4003970 0x7ffae4003d90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.442+0000 7ffb004a4640 1 -- 172.21.15.78:0/2418166607 >> 172.21.15.78:0/2418166607 conn(0x7ffaf80aa3e0 msgr2=0x7ffaf80aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.442+0000 7ffb004a4640 1 -- 172.21.15.78:0/2418166607 shutdown_connections 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.442+0000 7ffb004a4640 1 -- 172.21.15.78:0/2418166607 wait complete. 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.443+0000 7ffb004a4640 1 Processor -- start 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.443+0000 7ffb004a4640 1 -- start start 2024-02-19T14:52:16.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.444+0000 7ffb004a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4003970 0x7ffaf81eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.444+0000 7ffb004a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaf80c8a00 0x7ffaf8150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.444+0000 7ffb004a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 0x7ffaf81edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.444+0000 7ffb004a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf81eb4a0 con 0x7ffaf8150e40 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.444+0000 7ffb004a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8151280 con 0x7ffaf80c8a00 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.444+0000 7ffb004a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8099d90 con 0x7ffae4003970 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 0x7ffaf81edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 0x7ffaf81edd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59440/0 (socket says 172.21.15.78:59440) 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 -- 172.21.15.78:0/4062709968 learned_addr learned my addr 172.21.15.78:0/4062709968 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafda3f640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaf80c8a00 0x7ffaf8150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafe240640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4003970 0x7ffaf81eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 -- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4003970 msgr2=0x7ffaf81eaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4003970 0x7ffaf81eaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 -- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaf80c8a00 msgr2=0x7ffaf8150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaf80c8a00 0x7ffaf8150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 -- 172.21.15.78:0/4062709968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf806d710 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffafea41640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 0x7ffaf81edd90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80c4920 tx=0x7ffaf0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf001b070 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf000f040 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf000e050 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf8156c80 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.445+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80ad5e0 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.446+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf806e120 con 0x7ffaf8150e40 2024-02-19T14:52:16.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.447+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffaf0003bd0 con 0x7ffaf8150e40 2024-02-19T14:52:16.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.447+0000 7ffaeeffd640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffadc06b810 0x7ffadc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.447+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ffaf008a900 con 0x7ffaf8150e40 2024-02-19T14:52:16.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.447+0000 7ffafe240640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffadc06b810 0x7ffadc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.448+0000 7ffafe240640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffadc06b810 0x7ffadc06dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4034000 tx=0x7ffaf4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:16.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.451+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffaf0055070 con 0x7ffaf8150e40 2024-02-19T14:52:16.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.666+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaf8074990 con 0x7ffaf8150e40 2024-02-19T14:52:16.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.666+0000 7ffaeeffd640 1 -- 172.21.15.78:0/4062709968 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7ffaf0058730 con 0x7ffaf8150e40 2024-02-19T14:52:16.670 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:16.670 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:16.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffadc06b810 msgr2=0x7ffadc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffadc06b810 0x7ffadc06dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4034000 tx=0x7ffaf4009030 comp rx=0 tx=0).stop 2024-02-19T14:52:16.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 msgr2=0x7ffaf81edd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 0x7ffaf81edd90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80c4920 tx=0x7ffaf0010590 comp rx=0 tx=0).stop 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 shutdown_connections 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffadc06b810 0x7ffadc06dcd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffaf8150e40 0x7ffaf81edd90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaf80c8a00 0x7ffaf8150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 --2- 172.21.15.78:0/4062709968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae4003970 0x7ffaf81eaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.668+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 >> 172.21.15.78:0/4062709968 conn(0x7ffaf80aa3e0 msgr2=0x7ffaf80a96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.669+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 shutdown_connections 2024-02-19T14:52:16.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.669+0000 7ffb004a4640 1 -- 172.21.15.78:0/4062709968 wait complete. 2024-02-19T14:52:16.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.779+0000 7fd430d0c640 1 Processor -- start 2024-02-19T14:52:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.779+0000 7fd430d0c640 1 -- start start 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd430d0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd42c0c5560 0x7fd42c0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd430d0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd42c0c7910 0x7fd42c154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd430d0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd430d0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd42c0c65a0 con 0x7fd42c0c5560 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd430d0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd42c0c7530 con 0x7fd42c0c7910 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd430d0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c1537b0 con 0x7fd42c070720 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd42a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd42c0c5560 0x7fd42c0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.780+0000 7fd42a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd42c0c5560 0x7fd42c0c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59448/0 (socket says 172.21.15.78:59448) 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42a59c640 1 -- 172.21.15.78:0/2189706812 learned_addr learned my addr 172.21.15.78:0/2189706812 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd429d9b640 1 --2- 172.21.15.78:0/2189706812 >> v2:172.21.15.150:3300/0 conn(0x7fd42c0c7910 0x7fd42c154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/2189706812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42a59c640 1 -- 172.21.15.78:0/2189706812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 msgr2=0x7fd42c070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42a59c640 1 --2- 172.21.15.78:0/2189706812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42a59c640 1 -- 172.21.15.78:0/2189706812 >> v2:172.21.15.150:3300/0 conn(0x7fd42c0c7910 msgr2=0x7fd42c154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42a59c640 1 --2- 172.21.15.78:0/2189706812 >> v2:172.21.15.150:3300/0 conn(0x7fd42c0c7910 0x7fd42c154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd42a59c640 1 -- 172.21.15.78:0/2189706812 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c153a90 con 0x7fd42c0c5560 2024-02-19T14:52:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.781+0000 7fd429d9b640 1 --2- 172.21.15.78:0/2189706812 >> v2:172.21.15.150:3300/0 conn(0x7fd42c0c7910 0x7fd42c154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:16.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.782+0000 7fd42a59c640 1 --2- 172.21.15.78:0/2189706812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c5560 0x7fd42c0c6ff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd42c0c91b0 tx=0x7fd41c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54e235d64b2df49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:16.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.782+0000 7fd42959a640 1 -- 172.21.15.78:0/2189706812 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c031750 con 0x7fd42c0c5560 2024-02-19T14:52:16.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.782+0000 7fd42959a640 1 -- 172.21.15.78:0/2189706812 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41c0318b0 con 0x7fd42c0c5560 2024-02-19T14:52:16.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.782+0000 7fd42959a640 1 -- 172.21.15.78:0/2189706812 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41c031c40 con 0x7fd42c0c5560 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 -- 172.21.15.78:0/2189706812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c5560 msgr2=0x7fd42c0c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 --2- 172.21.15.78:0/2189706812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c5560 0x7fd42c0c6ff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd42c0c91b0 tx=0x7fd41c02b7f0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 -- 172.21.15.78:0/2189706812 shutdown_connections 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 --2- 172.21.15.78:0/2189706812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 --2- 172.21.15.78:0/2189706812 >> v2:172.21.15.150:3300/0 conn(0x7fd42c0c7910 0x7fd42c154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 --2- 172.21.15.78:0/2189706812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c5560 0x7fd42c0c6ff0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 -- 172.21.15.78:0/2189706812 >> 172.21.15.78:0/2189706812 conn(0x7fd42c06b4b0 msgr2=0x7fd42c0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 -- 172.21.15.78:0/2189706812 shutdown_connections 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.783+0000 7fd430d0c640 1 -- 172.21.15.78:0/2189706812 wait complete. 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.784+0000 7fd430d0c640 1 Processor -- start 2024-02-19T14:52:16.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.784+0000 7fd430d0c640 1 -- start start 2024-02-19T14:52:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0c5560 0x7fd42c1488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 0x7fd42c148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c0ad190 con 0x7fd42c0c7910 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c14af40 con 0x7fd42c0c5560 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c14b570 con 0x7fd42c070720 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 0x7fd42c148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 0x7fd42c148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59456/0 (socket says 172.21.15.78:59456) 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 -- 172.21.15.78:0/320139297 learned_addr learned my addr 172.21.15.78:0/320139297 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 -- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 msgr2=0x7fd42c149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd429d9b640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0c5560 0x7fd42c1488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42a59c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c149990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 -- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0c5560 msgr2=0x7fd42c1488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0c5560 0x7fd42c1488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 -- 172.21.15.78:0/320139297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c0094d0 con 0x7fd42c0c7910 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 0x7fd42c148e20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd424007d20 tx=0x7fd42400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424014420 con 0x7fd42c0c7910 2024-02-19T14:52:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4240145a0 con 0x7fd42c0c7910 2024-02-19T14:52:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42c0ac610 con 0x7fd42c0c7910 2024-02-19T14:52:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424014950 con 0x7fd42c0c7910 2024-02-19T14:52:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.785+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd42c0acbd0 con 0x7fd42c0c7910 2024-02-19T14:52:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.787+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42c074990 con 0x7fd42c0c7910 2024-02-19T14:52:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.788+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd424014ab0 con 0x7fd42c0c7910 2024-02-19T14:52:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.788+0000 7fd41affd640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.788+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd42408bcb0 con 0x7fd42c0c7910 2024-02-19T14:52:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.788+0000 7fd42a59c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.788+0000 7fd42a59c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd41c035000 tx=0x7fd41c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.792+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4240563e0 con 0x7fd42c0c7910 2024-02-19T14:52:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.970+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd42c0c76f0 con 0x7fd42c0c7910 2024-02-19T14:52:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.972+0000 7fd41affd640 1 -- 172.21.15.78:0/320139297 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7fd424059aa0 con 0x7fd42c0c7910 2024-02-19T14:52:16.975 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:16.976 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 msgr2=0x7fd40c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd41c035000 tx=0x7fd41c008030 comp rx=0 tx=0).stop 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 msgr2=0x7fd42c148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 0x7fd42c148e20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd424007d20 tx=0x7fd42400e4b0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 shutdown_connections 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0c7910 0x7fd42c148e20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0c5560 0x7fd42c1488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 --2- 172.21.15.78:0/320139297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c070720 0x7fd42c149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 >> 172.21.15.78:0/320139297 conn(0x7fd42c06b4b0 msgr2=0x7fd42c070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 shutdown_connections 2024-02-19T14:52:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:16.974+0000 7fd430d0c640 1 -- 172.21.15.78:0/320139297 wait complete. 2024-02-19T14:52:16.988 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:52:16.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:52:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.089+0000 7fdb2c9c7640 1 Processor -- start 2024-02-19T14:52:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2c9c7640 1 -- start start 2024-02-19T14:52:17.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2c9c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb280c78a0 0x7fdb280c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb2814ef50 0x7fdb2814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2c9c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2814f8f0 con 0x7fdb280727d0 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2c9c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb280c35c0 con 0x7fdb280c78a0 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2c9c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280c38a0 con 0x7fdb2814ef50 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb280c78a0 0x7fdb280c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb26d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdb280727d0 0x7fdb28072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39374/0 (socket says 172.21.15.78:39374) 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb2659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb280c78a0 0x7fdb280c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59472/0 (socket says 172.21.15.78:59472) 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.090+0000 7fdb26d9d640 1 -- 172.21.15.78:0/2377697853 learned_addr learned my addr 172.21.15.78:0/2377697853 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb1ffff640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb2814ef50 0x7fdb2814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb2659c640 1 -- 172.21.15.78:0/2377697853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb2814ef50 msgr2=0x7fdb2814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb2659c640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb2814ef50 0x7fdb2814f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb2659c640 1 -- 172.21.15.78:0/2377697853 >> v1:172.21.15.78:6789/0 conn(0x7fdb280727d0 legacy=0x7fdb28072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb25d9b640 1 -- 172.21.15.78:0/2377697853 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199344908 0 0) 0x7fdb2814f8f0 con 0x7fdb280727d0 2024-02-19T14:52:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb2659c640 1 -- 172.21.15.78:0/2377697853 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280c3b80 con 0x7fdb280c78a0 2024-02-19T14:52:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb1ffff640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb2814ef50 0x7fdb2814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.091+0000 7fdb2659c640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb280c78a0 0x7fdb280c44b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdb180097a0 tx=0x7fdb1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68782ff9b386b5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb25d9b640 1 -- 172.21.15.78:0/2377697853 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18031830 con 0x7fdb280c78a0 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb25d9b640 1 -- 172.21.15.78:0/2377697853 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb18031990 con 0x7fdb280c78a0 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb25d9b640 1 -- 172.21.15.78:0/2377697853 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18031d20 con 0x7fdb280c78a0 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/2377697853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb280c78a0 msgr2=0x7fdb280c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb280c78a0 0x7fdb280c44b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdb180097a0 tx=0x7fdb1802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/2377697853 shutdown_connections 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb2814ef50 0x7fdb2814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/2377697853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb280c78a0 0x7fdb280c44b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/2377697853 >> 172.21.15.78:0/2377697853 conn(0x7fdb28066e70 msgr2=0x7fdb280a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/2377697853 shutdown_connections 2024-02-19T14:52:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.092+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/2377697853 wait complete. 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.093+0000 7fdb2c9c7640 1 Processor -- start 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.093+0000 7fdb2c9c7640 1 -- start start 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb280727d0 0x7fdb28146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb280c78a0 0x7fdb281487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 0x7fdb281e8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2c9c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281e8930 con 0x7fdb2814ef50 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2c9c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281ecc40 con 0x7fdb280727d0 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2c9c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2814a680 con 0x7fdb280c78a0 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 0x7fdb281e8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 0x7fdb281e8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59478/0 (socket says 172.21.15.78:59478) 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 -- 172.21.15.78:0/19899659 learned_addr learned my addr 172.21.15.78:0/19899659 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:17.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb1ffff640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb280c78a0 0x7fdb281487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2659c640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb280727d0 0x7fdb28146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 -- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb280c78a0 msgr2=0x7fdb281487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb280c78a0 0x7fdb281487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 -- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb280727d0 msgr2=0x7fdb28146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb280727d0 0x7fdb28146460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb26d9d640 1 -- 172.21.15.78:0/19899659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb18009400 con 0x7fdb2814ef50 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.094+0000 7fdb2659c640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb280727d0 0x7fdb28146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.095+0000 7fdb26d9d640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 0x7fdb281e8d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c7a0 tx=0x7fdb2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.095+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000a640 con 0x7fdb2814ef50 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.095+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2000a7a0 con 0x7fdb2814ef50 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.095+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2814a960 con 0x7fdb2814ef50 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.095+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000abd0 con 0x7fdb2814ef50 2024-02-19T14:52:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.095+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb28148cf0 con 0x7fdb2814ef50 2024-02-19T14:52:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.096+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb281e8aa0 con 0x7fdb2814ef50 2024-02-19T14:52:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.096+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdb2000ad30 con 0x7fdb2814ef50 2024-02-19T14:52:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.097+0000 7fdb1effd640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb0806b810 0x7fdb0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.097+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdb20089ca0 con 0x7fdb2814ef50 2024-02-19T14:52:17.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.097+0000 7fdb2659c640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb0806b810 0x7fdb0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.097+0000 7fdb2659c640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb0806b810 0x7fdb0806dcd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdb18036000 tx=0x7fdb18008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.105+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdb281e8aa0 con 0x7fdb2814ef50 2024-02-19T14:52:17.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.275+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb280c3290 con 0x7fdb0806b810 2024-02-19T14:52:17.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.284+0000 7fdb1effd640 1 -- 172.21.15.78:0/19899659 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345318 (secure 0 0 0) 0x7fdb18031c00 con 0x7fdb0806b810 2024-02-19T14:52:17.292 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:17.294 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb0806b810 msgr2=0x7fdb0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb0806b810 0x7fdb0806dcd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdb18036000 tx=0x7fdb18008030 comp rx=0 tx=0).stop 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 msgr2=0x7fdb281e8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 0x7fdb281e8d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c7a0 tx=0x7fdb2000cc60 comp rx=0 tx=0).stop 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 shutdown_connections 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb0806b810 0x7fdb0806dcd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb2814ef50 0x7fdb281e8d80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb280c78a0 0x7fdb281487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 --2- 172.21.15.78:0/19899659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb280727d0 0x7fdb28146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 >> 172.21.15.78:0/19899659 conn(0x7fdb28066e70 msgr2=0x7fdb280c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:17.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 shutdown_connections 2024-02-19T14:52:17.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.287+0000 7fdb2c9c7640 1 -- 172.21.15.78:0/19899659 wait complete. 2024-02-19T14:52:17.302 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-02-19T14:52:16.063123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219384,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755336,"statfs":{"total":773094113280,"available":772869464064,"internally_reserved":0,"allocated":3284992,"data_stored":1991782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.782425"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226366+0000","last_change":"2024-02-19T14:51:59.266929+0000","last_active":"2024-02-19T14:52:06.226366+0000","last_peered":"2024-02-19T14:52:06.226366+0000","last_clean":"2024-02-19T14:52:06.226366+0000","last_became_active":"2024-02-19T14:51:59.266644+0000","last_became_peered":"2024-02-19T14:51:59.266644+0000","last_unstale":"2024-02-19T14:52:06.226366+0000","last_undegraded":"2024-02-19T14:52:06.226366+0000","last_fullsized":"2024-02-19T14:52:06.226366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335294+0000","last_change":"2024-02-19T14:52:00.264687+0000","last_active":"2024-02-19T14:52:07.335294+0000","last_peered":"2024-02-19T14:52:07.335294+0000","last_clean":"2024-02-19T14:52:07.335294+0000","last_became_active":"2024-02-19T14:52:00.264424+0000","last_became_peered":"2024-02-19T14:52:00.264424+0000","last_unstale":"2024-02-19T14:52:07.335294+0000","last_undegraded":"2024-02-19T14:52:07.335294+0000","last_fullsized":"2024-02-19T14:52:07.335294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334297+0000","last_change":"2024-02-19T14:52:04.435336+0000","last_active":"2024-02-19T14:52:07.334297+0000","last_peered":"2024-02-19T14:52:07.334297+0000","last_clean":"2024-02-19T14:52:07.334297+0000","last_became_active":"2024-02-19T14:52:04.434977+0000","last_became_peered":"2024-02-19T14:52:04.434977+0000","last_unstale":"2024-02-19T14:52:07.334297+0000","last_undegraded":"2024-02-19T14:52:07.334297+0000","last_fullsized":"2024-02-19T14:52:07.334297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334116+0000","last_change":"2024-02-19T14:51:59.262430+0000","last_active":"2024-02-19T14:52:07.334116+0000","last_peered":"2024-02-19T14:52:07.334116+0000","last_clean":"2024-02-19T14:52:07.334116+0000","last_became_active":"2024-02-19T14:51:59.262168+0000","last_became_peered":"2024-02-19T14:51:59.262168+0000","last_unstale":"2024-02-19T14:52:07.334116+0000","last_undegraded":"2024-02-19T14:52:07.334116+0000","last_fullsized":"2024-02-19T14:52:07.334116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226267+0000","last_change":"2024-02-19T14:52:00.263014+0000","last_active":"2024-02-19T14:52:06.226267+0000","last_peered":"2024-02-19T14:52:06.226267+0000","last_clean":"2024-02-19T14:52:06.226267+0000","last_became_active":"2024-02-19T14:52:00.262531+0000","last_became_peered":"2024-02-19T14:52:00.262531+0000","last_unstale":"2024-02-19T14:52:06.226267+0000","last_undegraded":"2024-02-19T14:52:06.226267+0000","last_fullsized":"2024-02-19T14:52:06.226267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282898+0000","last_change":"2024-02-19T14:52:04.277572+0000","last_active":"2024-02-19T14:52:05.282898+0000","last_peered":"2024-02-19T14:52:05.282898+0000","last_clean":"2024-02-19T14:52:05.282898+0000","last_became_active":"2024-02-19T14:52:04.276970+0000","last_became_peered":"2024-02-19T14:52:04.276970+0000","last_unstale":"2024-02-19T14:52:05.282898+0000","last_undegraded":"2024-02-19T14:52:05.282898+0000","last_fullsized":"2024-02-19T14:52:05.282898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:14.095185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247290+0000","last_change":"2024-02-19T14:51:59.265003+0000","last_active":"2024-02-19T14:52:06.247290+0000","last_peered":"2024-02-19T14:52:06.247290+0000","last_clean":"2024-02-19T14:52:06.247290+0000","last_became_active":"2024-02-19T14:51:59.264719+0000","last_became_peered":"2024-02-19T14:51:59.264719+0000","last_unstale":"2024-02-19T14:52:06.247290+0000","last_undegraded":"2024-02-19T14:52:06.247290+0000","last_fullsized":"2024-02-19T14:52:06.247290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283484+0000","last_change":"2024-02-19T14:52:00.252606+0000","last_active":"2024-02-19T14:52:05.283484+0000","last_peered":"2024-02-19T14:52:05.283484+0000","last_clean":"2024-02-19T14:52:05.283484+0000","last_became_active":"2024-02-19T14:52:00.249782+0000","last_became_peered":"2024-02-19T14:52:00.249782+0000","last_unstale":"2024-02-19T14:52:05.283484+0000","last_undegraded":"2024-02-19T14:52:05.283484+0000","last_fullsized":"2024-02-19T14:52:05.283484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283979+0000","last_change":"2024-02-19T14:52:04.278014+0000","last_active":"2024-02-19T14:52:05.283979+0000","last_peered":"2024-02-19T14:52:05.283979+0000","last_clean":"2024-02-19T14:52:05.283979+0000","last_became_active":"2024-02-19T14:52:04.277126+0000","last_became_peered":"2024-02-19T14:52:04.277126+0000","last_unstale":"2024-02-19T14:52:05.283979+0000","last_undegraded":"2024-02-19T14:52:05.283979+0000","last_fullsized":"2024-02-19T14:52:05.283979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:27.674760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247080+0000","last_change":"2024-02-19T14:51:59.263462+0000","last_active":"2024-02-19T14:52:06.247080+0000","last_peered":"2024-02-19T14:52:06.247080+0000","last_clean":"2024-02-19T14:52:06.247080+0000","last_became_active":"2024-02-19T14:51:59.262932+0000","last_became_peered":"2024-02-19T14:51:59.262932+0000","last_unstale":"2024-02-19T14:52:06.247080+0000","last_undegraded":"2024-02-19T14:52:06.247080+0000","last_fullsized":"2024-02-19T14:52:06.247080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284101+0000","last_change":"2024-02-19T14:52:00.252460+0000","last_active":"2024-02-19T14:52:05.284101+0000","last_peered":"2024-02-19T14:52:05.284101+0000","last_clean":"2024-02-19T14:52:05.284101+0000","last_became_active":"2024-02-19T14:52:00.251603+0000","last_became_peered":"2024-02-19T14:52:00.251603+0000","last_unstale":"2024-02-19T14:52:05.284101+0000","last_undegraded":"2024-02-19T14:52:05.284101+0000","last_fullsized":"2024-02-19T14:52:05.284101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283155+0000","last_change":"2024-02-19T14:52:04.278038+0000","last_active":"2024-02-19T14:52:05.283155+0000","last_peered":"2024-02-19T14:52:05.283155+0000","last_clean":"2024-02-19T14:52:05.283155+0000","last_became_active":"2024-02-19T14:52:04.277620+0000","last_became_peered":"2024-02-19T14:52:04.277620+0000","last_unstale":"2024-02-19T14:52:05.283155+0000","last_undegraded":"2024-02-19T14:52:05.283155+0000","last_fullsized":"2024-02-19T14:52:05.283155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:12.987808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530027+0000","last_change":"2024-02-19T14:52:00.264133+0000","last_active":"2024-02-19T14:52:05.530027+0000","last_peered":"2024-02-19T14:52:05.530027+0000","last_clean":"2024-02-19T14:52:05.530027+0000","last_became_active":"2024-02-19T14:52:00.263475+0000","last_became_peered":"2024-02-19T14:52:00.263475+0000","last_unstale":"2024-02-19T14:52:05.530027+0000","last_undegraded":"2024-02-19T14:52:05.530027+0000","last_fullsized":"2024-02-19T14:52:05.530027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247110+0000","last_change":"2024-02-19T14:51:59.263095+0000","last_active":"2024-02-19T14:52:06.247110+0000","last_peered":"2024-02-19T14:52:06.247110+0000","last_clean":"2024-02-19T14:52:06.247110+0000","last_became_active":"2024-02-19T14:51:59.262392+0000","last_became_peered":"2024-02-19T14:51:59.262392+0000","last_unstale":"2024-02-19T14:52:06.247110+0000","last_undegraded":"2024-02-19T14:52:06.247110+0000","last_fullsized":"2024-02-19T14:52:06.247110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:04.278534+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:04.277850+0000","last_became_peered":"2024-02-19T14:52:04.277850+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334968+0000","last_change":"2024-02-19T14:52:00.268782+0000","last_active":"2024-02-19T14:52:07.334968+0000","last_peered":"2024-02-19T14:52:07.334968+0000","last_clean":"2024-02-19T14:52:07.334968+0000","last_became_active":"2024-02-19T14:52:00.266478+0000","last_became_peered":"2024-02-19T14:52:00.266478+0000","last_unstale":"2024-02-19T14:52:07.334968+0000","last_undegraded":"2024-02-19T14:52:07.334968+0000","last_fullsized":"2024-02-19T14:52:07.334968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228818+0000","last_change":"2024-02-19T14:51:59.260219+0000","last_active":"2024-02-19T14:52:07.228818+0000","last_peered":"2024-02-19T14:52:07.228818+0000","last_clean":"2024-02-19T14:52:07.228818+0000","last_became_active":"2024-02-19T14:51:59.259917+0000","last_became_peered":"2024-02-19T14:51:59.259917+0000","last_unstale":"2024-02-19T14:52:07.228818+0000","last_undegraded":"2024-02-19T14:52:07.228818+0000","last_fullsized":"2024-02-19T14:52:07.228818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282889+0000","last_change":"2024-02-19T14:52:04.277635+0000","last_active":"2024-02-19T14:52:05.282889+0000","last_peered":"2024-02-19T14:52:05.282889+0000","last_clean":"2024-02-19T14:52:05.282889+0000","last_became_active":"2024-02-19T14:52:04.277231+0000","last_became_peered":"2024-02-19T14:52:04.277231+0000","last_unstale":"2024-02-19T14:52:05.282889+0000","last_undegraded":"2024-02-19T14:52:05.282889+0000","last_fullsized":"2024-02-19T14:52:05.282889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:46.471360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334393+0000","last_change":"2024-02-19T14:52:00.263688+0000","last_active":"2024-02-19T14:52:07.334393+0000","last_peered":"2024-02-19T14:52:07.334393+0000","last_clean":"2024-02-19T14:52:07.334393+0000","last_became_active":"2024-02-19T14:52:00.263381+0000","last_became_peered":"2024-02-19T14:52:00.263381+0000","last_unstale":"2024-02-19T14:52:07.334393+0000","last_undegraded":"2024-02-19T14:52:07.334393+0000","last_fullsized":"2024-02-19T14:52:07.334393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228547+0000","last_change":"2024-02-19T14:51:59.262590+0000","last_active":"2024-02-19T14:52:07.228547+0000","last_peered":"2024-02-19T14:52:07.228547+0000","last_clean":"2024-02-19T14:52:07.228547+0000","last_became_active":"2024-02-19T14:51:59.262310+0000","last_became_peered":"2024-02-19T14:51:59.262310+0000","last_unstale":"2024-02-19T14:52:07.228547+0000","last_undegraded":"2024-02-19T14:52:07.228547+0000","last_fullsized":"2024-02-19T14:52:07.228547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229123+0000","last_change":"2024-02-19T14:52:04.434727+0000","last_active":"2024-02-19T14:52:07.229123+0000","last_peered":"2024-02-19T14:52:07.229123+0000","last_clean":"2024-02-19T14:52:07.229123+0000","last_became_active":"2024-02-19T14:52:04.432708+0000","last_became_peered":"2024-02-19T14:52:04.432708+0000","last_unstale":"2024-02-19T14:52:07.229123+0000","last_undegraded":"2024-02-19T14:52:07.229123+0000","last_fullsized":"2024-02-19T14:52:07.229123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:19.410143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284506+0000","last_change":"2024-02-19T14:52:00.260620+0000","last_active":"2024-02-19T14:52:05.284506+0000","last_peered":"2024-02-19T14:52:05.284506+0000","last_clean":"2024-02-19T14:52:05.284506+0000","last_became_active":"2024-02-19T14:52:00.260357+0000","last_became_peered":"2024-02-19T14:52:00.260357+0000","last_unstale":"2024-02-19T14:52:05.284506+0000","last_undegraded":"2024-02-19T14:52:05.284506+0000","last_fullsized":"2024-02-19T14:52:05.284506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:09.799959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246702+0000","last_change":"2024-02-19T14:51:59.267802+0000","last_active":"2024-02-19T14:52:06.246702+0000","last_peered":"2024-02-19T14:52:06.246702+0000","last_clean":"2024-02-19T14:52:06.246702+0000","last_became_active":"2024-02-19T14:51:59.267518+0000","last_became_peered":"2024-02-19T14:51:59.267518+0000","last_unstale":"2024-02-19T14:52:06.246702+0000","last_undegraded":"2024-02-19T14:52:06.246702+0000","last_fullsized":"2024-02-19T14:52:06.246702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:45.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283176+0000","last_change":"2024-02-19T14:52:04.276717+0000","last_active":"2024-02-19T14:52:05.283176+0000","last_peered":"2024-02-19T14:52:05.283176+0000","last_clean":"2024-02-19T14:52:05.283176+0000","last_became_active":"2024-02-19T14:52:04.276273+0000","last_became_peered":"2024-02-19T14:52:04.276273+0000","last_unstale":"2024-02-19T14:52:05.283176+0000","last_undegraded":"2024-02-19T14:52:05.283176+0000","last_fullsized":"2024-02-19T14:52:05.283176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:30.898405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226156+0000","last_change":"2024-02-19T14:52:00.259379+0000","last_active":"2024-02-19T14:52:06.226156+0000","last_peered":"2024-02-19T14:52:06.226156+0000","last_clean":"2024-02-19T14:52:06.226156+0000","last_became_active":"2024-02-19T14:52:00.259061+0000","last_became_peered":"2024-02-19T14:52:00.259061+0000","last_unstale":"2024-02-19T14:52:06.226156+0000","last_undegraded":"2024-02-19T14:52:06.226156+0000","last_fullsized":"2024-02-19T14:52:06.226156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:51:59.262618+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:51:59.262333+0000","last_became_peered":"2024-02-19T14:51:59.262333+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284070+0000","last_change":"2024-02-19T14:52:04.436152+0000","last_active":"2024-02-19T14:52:05.284070+0000","last_peered":"2024-02-19T14:52:05.284070+0000","last_clean":"2024-02-19T14:52:05.284070+0000","last_became_active":"2024-02-19T14:52:04.435251+0000","last_became_peered":"2024-02-19T14:52:04.435251+0000","last_unstale":"2024-02-19T14:52:05.284070+0000","last_undegraded":"2024-02-19T14:52:05.284070+0000","last_fullsized":"2024-02-19T14:52:05.284070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:33.762974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335436+0000","last_change":"2024-02-19T14:52:00.272464+0000","last_active":"2024-02-19T14:52:07.335436+0000","last_peered":"2024-02-19T14:52:07.335436+0000","last_clean":"2024-02-19T14:52:07.335436+0000","last_became_active":"2024-02-19T14:52:00.272184+0000","last_became_peered":"2024-02-19T14:52:00.272184+0000","last_unstale":"2024-02-19T14:52:07.335436+0000","last_undegraded":"2024-02-19T14:52:07.335436+0000","last_fullsized":"2024-02-19T14:52:07.335436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.256170+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.255855+0000","last_became_peered":"2024-02-19T14:51:59.255855+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334561+0000","last_change":"2024-02-19T14:52:04.433402+0000","last_active":"2024-02-19T14:52:07.334561+0000","last_peered":"2024-02-19T14:52:07.334561+0000","last_clean":"2024-02-19T14:52:07.334561+0000","last_became_active":"2024-02-19T14:52:04.432815+0000","last_became_peered":"2024-02-19T14:52:04.432815+0000","last_unstale":"2024-02-19T14:52:07.334561+0000","last_undegraded":"2024-02-19T14:52:07.334561+0000","last_fullsized":"2024-02-19T14:52:07.334561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:20.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228931+0000","last_change":"2024-02-19T14:52:00.261954+0000","last_active":"2024-02-19T14:52:07.228931+0000","last_peered":"2024-02-19T14:52:07.228931+0000","last_clean":"2024-02-19T14:52:07.228931+0000","last_became_active":"2024-02-19T14:52:00.261678+0000","last_became_peered":"2024-02-19T14:52:00.261678+0000","last_unstale":"2024-02-19T14:52:07.228931+0000","last_undegraded":"2024-02-19T14:52:07.228931+0000","last_fullsized":"2024-02-19T14:52:07.228931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247420+0000","last_change":"2024-02-19T14:51:59.267996+0000","last_active":"2024-02-19T14:52:06.247420+0000","last_peered":"2024-02-19T14:52:06.247420+0000","last_clean":"2024-02-19T14:52:06.247420+0000","last_became_active":"2024-02-19T14:51:59.267426+0000","last_became_peered":"2024-02-19T14:51:59.267426+0000","last_unstale":"2024-02-19T14:52:06.247420+0000","last_undegraded":"2024-02-19T14:52:06.247420+0000","last_fullsized":"2024-02-19T14:52:06.247420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225861+0000","last_change":"2024-02-19T14:52:04.433162+0000","last_active":"2024-02-19T14:52:06.225861+0000","last_peered":"2024-02-19T14:52:06.225861+0000","last_clean":"2024-02-19T14:52:06.225861+0000","last_became_active":"2024-02-19T14:52:04.432722+0000","last_became_peered":"2024-02-19T14:52:04.432722+0000","last_unstale":"2024-02-19T14:52:06.225861+0000","last_undegraded":"2024-02-19T14:52:06.225861+0000","last_fullsized":"2024-02-19T14:52:06.225861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:00.272249+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:00.266375+0000","last_became_peered":"2024-02-19T14:52:00.266375+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229033+0000","last_change":"2024-02-19T14:51:59.260559+0000","last_active":"2024-02-19T14:52:07.229033+0000","last_peered":"2024-02-19T14:52:07.229033+0000","last_clean":"2024-02-19T14:52:07.229033+0000","last_became_active":"2024-02-19T14:51:59.260290+0000","last_became_peered":"2024-02-19T14:51:59.260290+0000","last_unstale":"2024-02-19T14:52:07.229033+0000","last_undegraded":"2024-02-19T14:52:07.229033+0000","last_fullsized":"2024-02-19T14:52:07.229033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228634+0000","last_change":"2024-02-19T14:52:04.283819+0000","last_active":"2024-02-19T14:52:07.228634+0000","last_peered":"2024-02-19T14:52:07.228634+0000","last_clean":"2024-02-19T14:52:07.228634+0000","last_became_active":"2024-02-19T14:52:04.283469+0000","last_became_peered":"2024-02-19T14:52:04.283469+0000","last_unstale":"2024-02-19T14:52:07.228634+0000","last_undegraded":"2024-02-19T14:52:07.228634+0000","last_fullsized":"2024-02-19T14:52:07.228634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284410+0000","last_change":"2024-02-19T14:52:00.266384+0000","last_active":"2024-02-19T14:52:05.284410+0000","last_peered":"2024-02-19T14:52:05.284410+0000","last_clean":"2024-02-19T14:52:05.284410+0000","last_became_active":"2024-02-19T14:52:00.265954+0000","last_became_peered":"2024-02-19T14:52:00.265954+0000","last_unstale":"2024-02-19T14:52:05.284410+0000","last_undegraded":"2024-02-19T14:52:05.284410+0000","last_fullsized":"2024-02-19T14:52:05.284410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283363+0000","last_change":"2024-02-19T14:51:59.259377+0000","last_active":"2024-02-19T14:52:05.283363+0000","last_peered":"2024-02-19T14:52:05.283363+0000","last_clean":"2024-02-19T14:52:05.283363+0000","last_became_active":"2024-02-19T14:51:59.259004+0000","last_became_peered":"2024-02-19T14:51:59.259004+0000","last_unstale":"2024-02-19T14:52:05.283363+0000","last_undegraded":"2024-02-19T14:52:05.283363+0000","last_fullsized":"2024-02-19T14:52:05.283363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229000+0000","last_change":"2024-02-19T14:52:04.282027+0000","last_active":"2024-02-19T14:52:07.229000+0000","last_peered":"2024-02-19T14:52:07.229000+0000","last_clean":"2024-02-19T14:52:07.229000+0000","last_became_active":"2024-02-19T14:52:04.281679+0000","last_became_peered":"2024-02-19T14:52:04.281679+0000","last_unstale":"2024-02-19T14:52:07.229000+0000","last_undegraded":"2024-02-19T14:52:07.229000+0000","last_fullsized":"2024-02-19T14:52:07.229000+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:41.900564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228565+0000","last_change":"2024-02-19T14:52:00.261302+0000","last_active":"2024-02-19T14:52:07.228565+0000","last_peered":"2024-02-19T14:52:07.228565+0000","last_clean":"2024-02-19T14:52:07.228565+0000","last_became_active":"2024-02-19T14:52:00.261027+0000","last_became_peered":"2024-02-19T14:52:00.261027+0000","last_unstale":"2024-02-19T14:52:07.228565+0000","last_undegraded":"2024-02-19T14:52:07.228565+0000","last_fullsized":"2024-02-19T14:52:07.228565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281412+0000","last_change":"2024-02-19T14:51:59.256034+0000","last_active":"2024-02-19T14:52:05.281412+0000","last_peered":"2024-02-19T14:52:05.281412+0000","last_clean":"2024-02-19T14:52:05.281412+0000","last_became_active":"2024-02-19T14:51:59.255701+0000","last_became_peered":"2024-02-19T14:51:59.255701+0000","last_unstale":"2024-02-19T14:52:05.281412+0000","last_undegraded":"2024-02-19T14:52:05.281412+0000","last_fullsized":"2024-02-19T14:52:05.281412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229180+0000","last_change":"2024-02-19T14:52:04.283973+0000","last_active":"2024-02-19T14:52:07.229180+0000","last_peered":"2024-02-19T14:52:07.229180+0000","last_clean":"2024-02-19T14:52:07.229180+0000","last_became_active":"2024-02-19T14:52:04.283562+0000","last_became_peered":"2024-02-19T14:52:04.283562+0000","last_unstale":"2024-02-19T14:52:07.229180+0000","last_undegraded":"2024-02-19T14:52:07.229180+0000","last_fullsized":"2024-02-19T14:52:07.229180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225524+0000","last_change":"2024-02-19T14:52:00.260330+0000","last_active":"2024-02-19T14:52:06.225524+0000","last_peered":"2024-02-19T14:52:06.225524+0000","last_clean":"2024-02-19T14:52:06.225524+0000","last_became_active":"2024-02-19T14:52:00.260039+0000","last_became_peered":"2024-02-19T14:52:00.260039+0000","last_unstale":"2024-02-19T14:52:06.225524+0000","last_undegraded":"2024-02-19T14:52:06.225524+0000","last_fullsized":"2024-02-19T14:52:06.225524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:47.726577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246718+0000","last_change":"2024-02-19T14:51:59.267730+0000","last_active":"2024-02-19T14:52:06.246718+0000","last_peered":"2024-02-19T14:52:06.246718+0000","last_clean":"2024-02-19T14:52:06.246718+0000","last_became_active":"2024-02-19T14:51:59.267415+0000","last_became_peered":"2024-02-19T14:51:59.267415+0000","last_unstale":"2024-02-19T14:52:06.246718+0000","last_undegraded":"2024-02-19T14:52:06.246718+0000","last_fullsized":"2024-02-19T14:52:06.246718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226143+0000","last_change":"2024-02-19T14:52:04.433516+0000","last_active":"2024-02-19T14:52:06.226143+0000","last_peered":"2024-02-19T14:52:06.226143+0000","last_clean":"2024-02-19T14:52:06.226143+0000","last_became_active":"2024-02-19T14:52:04.432635+0000","last_became_peered":"2024-02-19T14:52:04.432635+0000","last_unstale":"2024-02-19T14:52:06.226143+0000","last_undegraded":"2024-02-19T14:52:06.226143+0000","last_fullsized":"2024-02-19T14:52:06.226143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334629+0000","last_change":"2024-02-19T14:52:00.266772+0000","last_active":"2024-02-19T14:52:07.334629+0000","last_peered":"2024-02-19T14:52:07.334629+0000","last_clean":"2024-02-19T14:52:07.334629+0000","last_became_active":"2024-02-19T14:52:00.265956+0000","last_became_peered":"2024-02-19T14:52:00.265956+0000","last_unstale":"2024-02-19T14:52:07.334629+0000","last_undegraded":"2024-02-19T14:52:07.334629+0000","last_fullsized":"2024-02-19T14:52:07.334629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282780+0000","last_change":"2024-02-19T14:51:59.252004+0000","last_active":"2024-02-19T14:52:05.282780+0000","last_peered":"2024-02-19T14:52:05.282780+0000","last_clean":"2024-02-19T14:52:05.282780+0000","last_became_active":"2024-02-19T14:51:59.251707+0000","last_became_peered":"2024-02-19T14:51:59.251707+0000","last_unstale":"2024-02-19T14:52:05.282780+0000","last_undegraded":"2024-02-19T14:52:05.282780+0000","last_fullsized":"2024-02-19T14:52:05.282780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246877+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:52:06.246877+0000","last_peered":"2024-02-19T14:52:06.246877+0000","last_clean":"2024-02-19T14:52:06.246877+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:52:06.246877+0000","last_undegraded":"2024-02-19T14:52:06.246877+0000","last_fullsized":"2024-02-19T14:52:06.246877+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530437+0000","last_change":"2024-02-19T14:52:04.436623+0000","last_active":"2024-02-19T14:52:05.530437+0000","last_peered":"2024-02-19T14:52:05.530437+0000","last_clean":"2024-02-19T14:52:05.530437+0000","last_became_active":"2024-02-19T14:52:04.436086+0000","last_became_peered":"2024-02-19T14:52:04.436086+0000","last_unstale":"2024-02-19T14:52:05.530437+0000","last_undegraded":"2024-02-19T14:52:05.530437+0000","last_fullsized":"2024-02-19T14:52:05.530437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247184+0000","last_change":"2024-02-19T14:52:00.267316+0000","last_active":"2024-02-19T14:52:06.247184+0000","last_peered":"2024-02-19T14:52:06.247184+0000","last_clean":"2024-02-19T14:52:06.247184+0000","last_became_active":"2024-02-19T14:52:00.266997+0000","last_became_peered":"2024-02-19T14:52:00.266997+0000","last_unstale":"2024-02-19T14:52:06.247184+0000","last_undegraded":"2024-02-19T14:52:06.247184+0000","last_fullsized":"2024-02-19T14:52:06.247184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281644+0000","last_change":"2024-02-19T14:51:59.253304+0000","last_active":"2024-02-19T14:52:05.281644+0000","last_peered":"2024-02-19T14:52:05.281644+0000","last_clean":"2024-02-19T14:52:05.281644+0000","last_became_active":"2024-02-19T14:51:59.253037+0000","last_became_peered":"2024-02-19T14:51:59.253037+0000","last_unstale":"2024-02-19T14:52:05.281644+0000","last_undegraded":"2024-02-19T14:52:05.281644+0000","last_fullsized":"2024-02-19T14:52:05.281644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530279+0000","last_change":"2024-02-19T14:52:04.433265+0000","last_active":"2024-02-19T14:52:05.530279+0000","last_peered":"2024-02-19T14:52:05.530279+0000","last_clean":"2024-02-19T14:52:05.530279+0000","last_became_active":"2024-02-19T14:52:04.432784+0000","last_became_peered":"2024-02-19T14:52:04.432784+0000","last_unstale":"2024-02-19T14:52:05.530279+0000","last_undegraded":"2024-02-19T14:52:05.530279+0000","last_fullsized":"2024-02-19T14:52:05.530279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283546+0000","last_change":"2024-02-19T14:52:00.266600+0000","last_active":"2024-02-19T14:52:05.283546+0000","last_peered":"2024-02-19T14:52:05.283546+0000","last_clean":"2024-02-19T14:52:05.283546+0000","last_became_active":"2024-02-19T14:52:00.265662+0000","last_became_peered":"2024-02-19T14:52:00.265662+0000","last_unstale":"2024-02-19T14:52:05.283546+0000","last_undegraded":"2024-02-19T14:52:05.283546+0000","last_fullsized":"2024-02-19T14:52:05.283546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229111+0000","last_change":"2024-02-19T14:51:59.258772+0000","last_active":"2024-02-19T14:52:07.229111+0000","last_peered":"2024-02-19T14:52:07.229111+0000","last_clean":"2024-02-19T14:52:07.229111+0000","last_became_active":"2024-02-19T14:51:59.258453+0000","last_became_peered":"2024-02-19T14:51:59.258453+0000","last_unstale":"2024-02-19T14:52:07.229111+0000","last_undegraded":"2024-02-19T14:52:07.229111+0000","last_fullsized":"2024-02-19T14:52:07.229111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225963+0000","last_change":"2024-02-19T14:52:04.281309+0000","last_active":"2024-02-19T14:52:06.225963+0000","last_peered":"2024-02-19T14:52:06.225963+0000","last_clean":"2024-02-19T14:52:06.225963+0000","last_became_active":"2024-02-19T14:52:04.280950+0000","last_became_peered":"2024-02-19T14:52:04.280950+0000","last_unstale":"2024-02-19T14:52:06.225963+0000","last_undegraded":"2024-02-19T14:52:06.225963+0000","last_fullsized":"2024-02-19T14:52:06.225963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247558+0000","last_change":"2024-02-19T14:52:00.260071+0000","last_active":"2024-02-19T14:52:06.247558+0000","last_peered":"2024-02-19T14:52:06.247558+0000","last_clean":"2024-02-19T14:52:06.247558+0000","last_became_active":"2024-02-19T14:52:00.259802+0000","last_became_peered":"2024-02-19T14:52:00.259802+0000","last_unstale":"2024-02-19T14:52:06.247558+0000","last_undegraded":"2024-02-19T14:52:06.247558+0000","last_fullsized":"2024-02-19T14:52:06.247558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:37.874309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281686+0000","last_change":"2024-02-19T14:51:59.252653+0000","last_active":"2024-02-19T14:52:05.281686+0000","last_peered":"2024-02-19T14:52:05.281686+0000","last_clean":"2024-02-19T14:52:05.281686+0000","last_became_active":"2024-02-19T14:51:59.252339+0000","last_became_peered":"2024-02-19T14:51:59.252339+0000","last_unstale":"2024-02-19T14:52:05.281686+0000","last_undegraded":"2024-02-19T14:52:05.281686+0000","last_fullsized":"2024-02-19T14:52:05.281686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282491+0000","last_change":"2024-02-19T14:52:04.435715+0000","last_active":"2024-02-19T14:52:05.282491+0000","last_peered":"2024-02-19T14:52:05.282491+0000","last_clean":"2024-02-19T14:52:05.282491+0000","last_became_active":"2024-02-19T14:52:04.435125+0000","last_became_peered":"2024-02-19T14:52:04.435125+0000","last_unstale":"2024-02-19T14:52:05.282491+0000","last_undegraded":"2024-02-19T14:52:05.282491+0000","last_fullsized":"2024-02-19T14:52:05.282491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247332+0000","last_change":"2024-02-19T14:52:00.262931+0000","last_active":"2024-02-19T14:52:06.247332+0000","last_peered":"2024-02-19T14:52:06.247332+0000","last_clean":"2024-02-19T14:52:06.247332+0000","last_became_active":"2024-02-19T14:52:00.262693+0000","last_became_peered":"2024-02-19T14:52:00.262693+0000","last_unstale":"2024-02-19T14:52:06.247332+0000","last_undegraded":"2024-02-19T14:52:06.247332+0000","last_fullsized":"2024-02-19T14:52:06.247332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283630+0000","last_change":"2024-02-19T14:51:59.260972+0000","last_active":"2024-02-19T14:52:05.283630+0000","last_peered":"2024-02-19T14:52:05.283630+0000","last_clean":"2024-02-19T14:52:05.283630+0000","last_became_active":"2024-02-19T14:51:59.260713+0000","last_became_peered":"2024-02-19T14:51:59.260713+0000","last_unstale":"2024-02-19T14:52:05.283630+0000","last_undegraded":"2024-02-19T14:52:05.283630+0000","last_fullsized":"2024-02-19T14:52:05.283630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283685+0000","last_change":"2024-02-19T14:52:04.277753+0000","last_active":"2024-02-19T14:52:05.283685+0000","last_peered":"2024-02-19T14:52:05.283685+0000","last_clean":"2024-02-19T14:52:05.283685+0000","last_became_active":"2024-02-19T14:52:04.277055+0000","last_became_peered":"2024-02-19T14:52:04.277055+0000","last_unstale":"2024-02-19T14:52:05.283685+0000","last_undegraded":"2024-02-19T14:52:05.283685+0000","last_fullsized":"2024-02-19T14:52:05.283685+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:26.703718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246886+0000","last_change":"2024-02-19T14:52:00.262083+0000","last_active":"2024-02-19T14:52:06.246886+0000","last_peered":"2024-02-19T14:52:06.246886+0000","last_clean":"2024-02-19T14:52:06.246886+0000","last_became_active":"2024-02-19T14:52:00.261797+0000","last_became_peered":"2024-02-19T14:52:00.261797+0000","last_unstale":"2024-02-19T14:52:06.246886+0000","last_undegraded":"2024-02-19T14:52:06.246886+0000","last_fullsized":"2024-02-19T14:52:06.246886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281279+0000","last_change":"2024-02-19T14:51:59.257068+0000","last_active":"2024-02-19T14:52:05.281279+0000","last_peered":"2024-02-19T14:52:05.281279+0000","last_clean":"2024-02-19T14:52:05.281279+0000","last_became_active":"2024-02-19T14:51:59.256775+0000","last_became_peered":"2024-02-19T14:51:59.256775+0000","last_unstale":"2024-02-19T14:52:05.281279+0000","last_undegraded":"2024-02-19T14:52:05.281279+0000","last_fullsized":"2024-02-19T14:52:05.281279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226044+0000","last_change":"2024-02-19T14:52:04.279890+0000","last_active":"2024-02-19T14:52:06.226044+0000","last_peered":"2024-02-19T14:52:06.226044+0000","last_clean":"2024-02-19T14:52:06.226044+0000","last_became_active":"2024-02-19T14:52:04.279485+0000","last_became_peered":"2024-02-19T14:52:04.279485+0000","last_unstale":"2024-02-19T14:52:06.226044+0000","last_undegraded":"2024-02-19T14:52:06.226044+0000","last_fullsized":"2024-02-19T14:52:06.226044+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:52:00.269756+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:52:00.269054+0000","last_became_peered":"2024-02-19T14:52:00.269054+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:51:59.251817+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:51:59.251496+0000","last_became_peered":"2024-02-19T14:51:59.251496+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335425+0000","last_change":"2024-02-19T14:52:04.436129+0000","last_active":"2024-02-19T14:52:07.335425+0000","last_peered":"2024-02-19T14:52:07.335425+0000","last_clean":"2024-02-19T14:52:07.335425+0000","last_became_active":"2024-02-19T14:52:04.435697+0000","last_became_peered":"2024-02-19T14:52:04.435697+0000","last_unstale":"2024-02-19T14:52:07.335425+0000","last_undegraded":"2024-02-19T14:52:07.335425+0000","last_fullsized":"2024-02-19T14:52:07.335425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:07.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247106+0000","last_change":"2024-02-19T14:52:00.263880+0000","last_active":"2024-02-19T14:52:06.247106+0000","last_peered":"2024-02-19T14:52:06.247106+0000","last_clean":"2024-02-19T14:52:06.247106+0000","last_became_active":"2024-02-19T14:52:00.263627+0000","last_became_peered":"2024-02-19T14:52:00.263627+0000","last_unstale":"2024-02-19T14:52:06.247106+0000","last_undegraded":"2024-02-19T14:52:06.247106+0000","last_fullsized":"2024-02-19T14:52:06.247106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283287+0000","last_change":"2024-02-19T14:51:59.252506+0000","last_active":"2024-02-19T14:52:05.283287+0000","last_peered":"2024-02-19T14:52:05.283287+0000","last_clean":"2024-02-19T14:52:05.283287+0000","last_became_active":"2024-02-19T14:51:59.251983+0000","last_became_peered":"2024-02-19T14:51:59.251983+0000","last_unstale":"2024-02-19T14:52:05.283287+0000","last_undegraded":"2024-02-19T14:52:05.283287+0000","last_fullsized":"2024-02-19T14:52:05.283287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335287+0000","last_change":"2024-02-19T14:52:04.433237+0000","last_active":"2024-02-19T14:52:07.335287+0000","last_peered":"2024-02-19T14:52:07.335287+0000","last_clean":"2024-02-19T14:52:07.335287+0000","last_became_active":"2024-02-19T14:52:04.432799+0000","last_became_peered":"2024-02-19T14:52:04.432799+0000","last_unstale":"2024-02-19T14:52:07.335287+0000","last_undegraded":"2024-02-19T14:52:07.335287+0000","last_fullsized":"2024-02-19T14:52:07.335287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:19.099988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225727+0000","last_change":"2024-02-19T14:52:00.267669+0000","last_active":"2024-02-19T14:52:06.225727+0000","last_peered":"2024-02-19T14:52:06.225727+0000","last_clean":"2024-02-19T14:52:06.225727+0000","last_became_active":"2024-02-19T14:52:00.267407+0000","last_became_peered":"2024-02-19T14:52:00.267407+0000","last_unstale":"2024-02-19T14:52:06.225727+0000","last_undegraded":"2024-02-19T14:52:06.225727+0000","last_fullsized":"2024-02-19T14:52:06.225727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226030+0000","last_change":"2024-02-19T14:51:59.261832+0000","last_active":"2024-02-19T14:52:06.226030+0000","last_peered":"2024-02-19T14:52:06.226030+0000","last_clean":"2024-02-19T14:52:06.226030+0000","last_became_active":"2024-02-19T14:51:59.261032+0000","last_became_peered":"2024-02-19T14:51:59.261032+0000","last_unstale":"2024-02-19T14:52:06.226030+0000","last_undegraded":"2024-02-19T14:52:06.226030+0000","last_fullsized":"2024-02-19T14:52:06.226030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284040+0000","last_change":"2024-02-19T14:52:04.433282+0000","last_active":"2024-02-19T14:52:05.284040+0000","last_peered":"2024-02-19T14:52:05.284040+0000","last_clean":"2024-02-19T14:52:05.284040+0000","last_became_active":"2024-02-19T14:52:04.432811+0000","last_became_peered":"2024-02-19T14:52:04.432811+0000","last_unstale":"2024-02-19T14:52:05.284040+0000","last_undegraded":"2024-02-19T14:52:05.284040+0000","last_fullsized":"2024-02-19T14:52:05.284040+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247064+0000","last_change":"2024-02-19T14:52:00.265115+0000","last_active":"2024-02-19T14:52:06.247064+0000","last_peered":"2024-02-19T14:52:06.247064+0000","last_clean":"2024-02-19T14:52:06.247064+0000","last_became_active":"2024-02-19T14:52:00.264838+0000","last_became_peered":"2024-02-19T14:52:00.264838+0000","last_unstale":"2024-02-19T14:52:06.247064+0000","last_undegraded":"2024-02-19T14:52:06.247064+0000","last_fullsized":"2024-02-19T14:52:06.247064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:09.092501+0000","last_change":"2024-02-19T14:51:59.260164+0000","last_active":"2024-02-19T14:52:09.092501+0000","last_peered":"2024-02-19T14:52:09.092501+0000","last_clean":"2024-02-19T14:52:09.092501+0000","last_became_active":"2024-02-19T14:51:59.259921+0000","last_became_peered":"2024-02-19T14:51:59.259921+0000","last_unstale":"2024-02-19T14:52:09.092501+0000","last_undegraded":"2024-02-19T14:52:09.092501+0000","last_fullsized":"2024-02-19T14:52:09.092501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228768+0000","last_change":"2024-02-19T14:52:04.282972+0000","last_active":"2024-02-19T14:52:07.228768+0000","last_peered":"2024-02-19T14:52:07.228768+0000","last_clean":"2024-02-19T14:52:07.228768+0000","last_became_active":"2024-02-19T14:52:04.282593+0000","last_became_peered":"2024-02-19T14:52:04.282593+0000","last_unstale":"2024-02-19T14:52:07.228768+0000","last_undegraded":"2024-02-19T14:52:07.228768+0000","last_fullsized":"2024-02-19T14:52:07.228768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283293+0000","last_change":"2024-02-19T14:52:00.263099+0000","last_active":"2024-02-19T14:52:05.283293+0000","last_peered":"2024-02-19T14:52:05.283293+0000","last_clean":"2024-02-19T14:52:05.283293+0000","last_became_active":"2024-02-19T14:52:00.260826+0000","last_became_peered":"2024-02-19T14:52:00.260826+0000","last_unstale":"2024-02-19T14:52:05.283293+0000","last_undegraded":"2024-02-19T14:52:05.283293+0000","last_fullsized":"2024-02-19T14:52:05.283293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:49.639204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.258165+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.257847+0000","last_became_peered":"2024-02-19T14:51:59.257847+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284313+0000","last_change":"2024-02-19T14:52:04.275097+0000","last_active":"2024-02-19T14:52:05.284313+0000","last_peered":"2024-02-19T14:52:05.284313+0000","last_clean":"2024-02-19T14:52:05.284313+0000","last_became_active":"2024-02-19T14:52:04.274720+0000","last_became_peered":"2024-02-19T14:52:04.274720+0000","last_unstale":"2024-02-19T14:52:05.284313+0000","last_undegraded":"2024-02-19T14:52:05.284313+0000","last_fullsized":"2024-02-19T14:52:05.284313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226316+0000","last_change":"2024-02-19T14:52:00.262791+0000","last_active":"2024-02-19T14:52:06.226316+0000","last_peered":"2024-02-19T14:52:06.226316+0000","last_clean":"2024-02-19T14:52:06.226316+0000","last_became_active":"2024-02-19T14:52:00.262201+0000","last_became_peered":"2024-02-19T14:52:00.262201+0000","last_unstale":"2024-02-19T14:52:06.226316+0000","last_undegraded":"2024-02-19T14:52:06.226316+0000","last_fullsized":"2024-02-19T14:52:06.226316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:23.961482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283591+0000","last_change":"2024-02-19T14:51:59.255762+0000","last_active":"2024-02-19T14:52:05.283591+0000","last_peered":"2024-02-19T14:52:05.283591+0000","last_clean":"2024-02-19T14:52:05.283591+0000","last_became_active":"2024-02-19T14:51:59.255418+0000","last_became_peered":"2024-02-19T14:51:59.255418+0000","last_unstale":"2024-02-19T14:52:05.283591+0000","last_undegraded":"2024-02-19T14:52:05.283591+0000","last_fullsized":"2024-02-19T14:52:05.283591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334347+0000","last_change":"2024-02-19T14:52:04.275183+0000","last_active":"2024-02-19T14:52:07.334347+0000","last_peered":"2024-02-19T14:52:07.334347+0000","last_clean":"2024-02-19T14:52:07.334347+0000","last_became_active":"2024-02-19T14:52:04.274796+0000","last_became_peered":"2024-02-19T14:52:04.274796+0000","last_unstale":"2024-02-19T14:52:07.334347+0000","last_undegraded":"2024-02-19T14:52:07.334347+0000","last_fullsized":"2024-02-19T14:52:07.334347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:17.099296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283483+0000","last_change":"2024-02-19T14:52:00.260440+0000","last_active":"2024-02-19T14:52:05.283483+0000","last_peered":"2024-02-19T14:52:05.283483+0000","last_clean":"2024-02-19T14:52:05.283483+0000","last_became_active":"2024-02-19T14:52:00.259406+0000","last_became_peered":"2024-02-19T14:52:00.259406+0000","last_unstale":"2024-02-19T14:52:05.283483+0000","last_undegraded":"2024-02-19T14:52:05.283483+0000","last_fullsized":"2024-02-19T14:52:05.283483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530584+0000","last_change":"2024-02-19T14:51:59.254054+0000","last_active":"2024-02-19T14:52:05.530584+0000","last_peered":"2024-02-19T14:52:05.530584+0000","last_clean":"2024-02-19T14:52:05.530584+0000","last_became_active":"2024-02-19T14:51:59.253751+0000","last_became_peered":"2024-02-19T14:51:59.253751+0000","last_unstale":"2024-02-19T14:52:05.530584+0000","last_undegraded":"2024-02-19T14:52:05.530584+0000","last_fullsized":"2024-02-19T14:52:05.530584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334305+0000","last_change":"2024-02-19T14:52:04.433602+0000","last_active":"2024-02-19T14:52:07.334305+0000","last_peered":"2024-02-19T14:52:07.334305+0000","last_clean":"2024-02-19T14:52:07.334305+0000","last_became_active":"2024-02-19T14:52:04.433250+0000","last_became_peered":"2024-02-19T14:52:04.433250+0000","last_unstale":"2024-02-19T14:52:07.334305+0000","last_undegraded":"2024-02-19T14:52:07.334305+0000","last_fullsized":"2024-02-19T14:52:07.334305+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226005+0000","last_change":"2024-02-19T14:52:04.435369+0000","last_active":"2024-02-19T14:52:06.226005+0000","last_peered":"2024-02-19T14:52:06.226005+0000","last_clean":"2024-02-19T14:52:06.226005+0000","last_became_active":"2024-02-19T14:52:04.435024+0000","last_became_peered":"2024-02-19T14:52:04.435024+0000","last_unstale":"2024-02-19T14:52:06.226005+0000","last_undegraded":"2024-02-19T14:52:06.226005+0000","last_fullsized":"2024-02-19T14:52:06.226005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:58.843829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228450+0000","last_change":"2024-02-19T14:52:00.266810+0000","last_active":"2024-02-19T14:52:07.228450+0000","last_peered":"2024-02-19T14:52:07.228450+0000","last_clean":"2024-02-19T14:52:07.228450+0000","last_became_active":"2024-02-19T14:52:00.266548+0000","last_became_peered":"2024-02-19T14:52:00.266548+0000","last_unstale":"2024-02-19T14:52:07.228450+0000","last_undegraded":"2024-02-19T14:52:07.228450+0000","last_fullsized":"2024-02-19T14:52:07.228450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282610+0000","last_change":"2024-02-19T14:51:59.248141+0000","last_active":"2024-02-19T14:52:05.282610+0000","last_peered":"2024-02-19T14:52:05.282610+0000","last_clean":"2024-02-19T14:52:05.282610+0000","last_became_active":"2024-02-19T14:51:59.247849+0000","last_became_peered":"2024-02-19T14:51:59.247849+0000","last_unstale":"2024-02-19T14:52:05.282610+0000","last_undegraded":"2024-02-19T14:52:05.282610+0000","last_fullsized":"2024-02-19T14:52:05.282610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530585+0000","last_change":"2024-02-19T14:52:04.282329+0000","last_active":"2024-02-19T14:52:05.530585+0000","last_peered":"2024-02-19T14:52:05.530585+0000","last_clean":"2024-02-19T14:52:05.530585+0000","last_became_active":"2024-02-19T14:52:04.281869+0000","last_became_peered":"2024-02-19T14:52:04.281869+0000","last_unstale":"2024-02-19T14:52:05.530585+0000","last_undegraded":"2024-02-19T14:52:05.530585+0000","last_fullsized":"2024-02-19T14:52:05.530585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334363+0000","last_change":"2024-02-19T14:52:00.266718+0000","last_active":"2024-02-19T14:52:07.334363+0000","last_peered":"2024-02-19T14:52:07.334363+0000","last_clean":"2024-02-19T14:52:07.334363+0000","last_became_active":"2024-02-19T14:52:00.266351+0000","last_became_peered":"2024-02-19T14:52:00.266351+0000","last_unstale":"2024-02-19T14:52:07.334363+0000","last_undegraded":"2024-02-19T14:52:07.334363+0000","last_fullsized":"2024-02-19T14:52:07.334363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333980+0000","last_change":"2024-02-19T14:51:59.259820+0000","last_active":"2024-02-19T14:52:07.333980+0000","last_peered":"2024-02-19T14:52:07.333980+0000","last_clean":"2024-02-19T14:52:07.333980+0000","last_became_active":"2024-02-19T14:51:59.259418+0000","last_became_peered":"2024-02-19T14:51:59.259418+0000","last_unstale":"2024-02-19T14:52:07.333980+0000","last_undegraded":"2024-02-19T14:52:07.333980+0000","last_fullsized":"2024-02-19T14:52:07.333980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530291+0000","last_change":"2024-02-19T14:51:59.252686+0000","last_active":"2024-02-19T14:52:05.530291+0000","last_peered":"2024-02-19T14:52:05.530291+0000","last_clean":"2024-02-19T14:52:05.530291+0000","last_became_active":"2024-02-19T14:51:59.252420+0000","last_became_peered":"2024-02-19T14:51:59.252420+0000","last_unstale":"2024-02-19T14:52:05.530291+0000","last_undegraded":"2024-02-19T14:52:05.530291+0000","last_fullsized":"2024-02-19T14:52:05.530291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226023+0000","last_change":"2024-02-19T14:52:00.264522+0000","last_active":"2024-02-19T14:52:06.226023+0000","last_peered":"2024-02-19T14:52:06.226023+0000","last_clean":"2024-02-19T14:52:06.226023+0000","last_became_active":"2024-02-19T14:52:00.264273+0000","last_became_peered":"2024-02-19T14:52:00.264273+0000","last_unstale":"2024-02-19T14:52:06.226023+0000","last_undegraded":"2024-02-19T14:52:06.226023+0000","last_fullsized":"2024-02-19T14:52:06.226023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281616+0000","last_change":"2024-02-19T14:52:04.275545+0000","last_active":"2024-02-19T14:52:05.281616+0000","last_peered":"2024-02-19T14:52:05.281616+0000","last_clean":"2024-02-19T14:52:05.281616+0000","last_became_active":"2024-02-19T14:52:04.275140+0000","last_became_peered":"2024-02-19T14:52:04.275140+0000","last_unstale":"2024-02-19T14:52:05.281616+0000","last_undegraded":"2024-02-19T14:52:05.281616+0000","last_fullsized":"2024-02-19T14:52:05.281616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:27.987737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333998+0000","last_change":"2024-02-19T14:51:59.262930+0000","last_active":"2024-02-19T14:52:07.333998+0000","last_peered":"2024-02-19T14:52:07.333998+0000","last_clean":"2024-02-19T14:52:07.333998+0000","last_became_active":"2024-02-19T14:51:59.262635+0000","last_became_peered":"2024-02-19T14:51:59.262635+0000","last_unstale":"2024-02-19T14:52:07.333998+0000","last_undegraded":"2024-02-19T14:52:07.333998+0000","last_fullsized":"2024-02-19T14:52:07.333998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:21.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284375+0000","last_change":"2024-02-19T14:52:00.260213+0000","last_active":"2024-02-19T14:52:05.284375+0000","last_peered":"2024-02-19T14:52:05.284375+0000","last_clean":"2024-02-19T14:52:05.284375+0000","last_became_active":"2024-02-19T14:52:00.257148+0000","last_became_peered":"2024-02-19T14:52:00.257148+0000","last_unstale":"2024-02-19T14:52:05.284375+0000","last_undegraded":"2024-02-19T14:52:05.284375+0000","last_fullsized":"2024-02-19T14:52:05.284375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:56.918797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335231+0000","last_change":"2024-02-19T14:52:04.276033+0000","last_active":"2024-02-19T14:52:07.335231+0000","last_peered":"2024-02-19T14:52:07.335231+0000","last_clean":"2024-02-19T14:52:07.335231+0000","last_became_active":"2024-02-19T14:52:04.275671+0000","last_became_peered":"2024-02-19T14:52:04.275671+0000","last_unstale":"2024-02-19T14:52:07.335231+0000","last_undegraded":"2024-02-19T14:52:07.335231+0000","last_fullsized":"2024-02-19T14:52:07.335231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:42.880381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.529955+0000","last_change":"2024-02-19T14:52:00.270697+0000","last_active":"2024-02-19T14:52:05.529955+0000","last_peered":"2024-02-19T14:52:05.529955+0000","last_clean":"2024-02-19T14:52:05.529955+0000","last_became_active":"2024-02-19T14:52:00.269449+0000","last_became_peered":"2024-02-19T14:52:00.269449+0000","last_unstale":"2024-02-19T14:52:05.529955+0000","last_undegraded":"2024-02-19T14:52:05.529955+0000","last_fullsized":"2024-02-19T14:52:05.529955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281315+0000","last_change":"2024-02-19T14:52:00.263056+0000","last_active":"2024-02-19T14:52:05.281315+0000","last_peered":"2024-02-19T14:52:05.281315+0000","last_clean":"2024-02-19T14:52:05.281315+0000","last_became_active":"2024-02-19T14:52:00.262801+0000","last_became_peered":"2024-02-19T14:52:00.262801+0000","last_unstale":"2024-02-19T14:52:05.281315+0000","last_undegraded":"2024-02-19T14:52:05.281315+0000","last_fullsized":"2024-02-19T14:52:05.281315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283161+0000","last_change":"2024-02-19T14:52:00.270389+0000","last_active":"2024-02-19T14:52:05.283161+0000","last_peered":"2024-02-19T14:52:05.283161+0000","last_clean":"2024-02-19T14:52:05.283161+0000","last_became_active":"2024-02-19T14:52:00.267246+0000","last_became_peered":"2024-02-19T14:52:00.267246+0000","last_unstale":"2024-02-19T14:52:05.283161+0000","last_undegraded":"2024-02-19T14:52:05.283161+0000","last_fullsized":"2024-02-19T14:52:05.283161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225914+0000","last_change":"2024-02-19T14:52:00.265624+0000","last_active":"2024-02-19T14:52:06.225914+0000","last_peered":"2024-02-19T14:52:06.225914+0000","last_clean":"2024-02-19T14:52:06.225914+0000","last_became_active":"2024-02-19T14:52:00.265377+0000","last_became_peered":"2024-02-19T14:52:00.265377+0000","last_unstale":"2024-02-19T14:52:06.225914+0000","last_undegraded":"2024-02-19T14:52:06.225914+0000","last_fullsized":"2024-02-19T14:52:06.225914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335142+0000","last_change":"2024-02-19T14:52:00.263043+0000","last_active":"2024-02-19T14:52:07.335142+0000","last_peered":"2024-02-19T14:52:07.335142+0000","last_clean":"2024-02-19T14:52:07.335142+0000","last_became_active":"2024-02-19T14:52:00.262754+0000","last_became_peered":"2024-02-19T14:52:00.262754+0000","last_unstale":"2024-02-19T14:52:07.335142+0000","last_undegraded":"2024-02-19T14:52:07.335142+0000","last_fullsized":"2024-02-19T14:52:07.335142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228927+0000","last_change":"2024-02-19T14:52:00.266890+0000","last_active":"2024-02-19T14:52:07.228927+0000","last_peered":"2024-02-19T14:52:07.228927+0000","last_clean":"2024-02-19T14:52:07.228927+0000","last_became_active":"2024-02-19T14:52:00.266574+0000","last_became_peered":"2024-02-19T14:52:00.266574+0000","last_unstale":"2024-02-19T14:52:07.228927+0000","last_undegraded":"2024-02-19T14:52:07.228927+0000","last_fullsized":"2024-02-19T14:52:07.228927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530706+0000","last_change":"2024-02-19T14:52:00.261225+0000","last_active":"2024-02-19T14:52:05.530706+0000","last_peered":"2024-02-19T14:52:05.530706+0000","last_clean":"2024-02-19T14:52:05.530706+0000","last_became_active":"2024-02-19T14:52:00.259666+0000","last_became_peered":"2024-02-19T14:52:00.259666+0000","last_unstale":"2024-02-19T14:52:05.530706+0000","last_undegraded":"2024-02-19T14:52:05.530706+0000","last_fullsized":"2024-02-19T14:52:05.530706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335354+0000","last_change":"2024-02-19T14:52:00.264377+0000","last_active":"2024-02-19T14:52:07.335354+0000","last_peered":"2024-02-19T14:52:07.335354+0000","last_clean":"2024-02-19T14:52:07.335354+0000","last_became_active":"2024-02-19T14:52:00.264082+0000","last_became_peered":"2024-02-19T14:52:00.264082+0000","last_unstale":"2024-02-19T14:52:07.335354+0000","last_undegraded":"2024-02-19T14:52:07.335354+0000","last_fullsized":"2024-02-19T14:52:07.335354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282561+0000","last_change":"2024-02-19T14:52:00.265423+0000","last_active":"2024-02-19T14:52:05.282561+0000","last_peered":"2024-02-19T14:52:05.282561+0000","last_clean":"2024-02-19T14:52:05.282561+0000","last_became_active":"2024-02-19T14:52:00.265132+0000","last_became_peered":"2024-02-19T14:52:00.265132+0000","last_unstale":"2024-02-19T14:52:05.282561+0000","last_undegraded":"2024-02-19T14:52:05.282561+0000","last_fullsized":"2024-02-19T14:52:05.282561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334543+0000","last_change":"2024-02-19T14:52:00.259058+0000","last_active":"2024-02-19T14:52:07.334543+0000","last_peered":"2024-02-19T14:52:07.334543+0000","last_clean":"2024-02-19T14:52:07.334543+0000","last_became_active":"2024-02-19T14:52:00.258228+0000","last_became_peered":"2024-02-19T14:52:00.258228+0000","last_unstale":"2024-02-19T14:52:07.334543+0000","last_undegraded":"2024-02-19T14:52:07.334543+0000","last_fullsized":"2024-02-19T14:52:07.334543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283058+0000","last_change":"2024-02-19T14:52:00.258261+0000","last_active":"2024-02-19T14:52:05.283058+0000","last_peered":"2024-02-19T14:52:05.283058+0000","last_clean":"2024-02-19T14:52:05.283058+0000","last_became_active":"2024-02-19T14:52:00.257970+0000","last_became_peered":"2024-02-19T14:52:00.257970+0000","last_unstale":"2024-02-19T14:52:05.283058+0000","last_undegraded":"2024-02-19T14:52:05.283058+0000","last_fullsized":"2024-02-19T14:52:05.283058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225756+0000","last_change":"2024-02-19T14:52:00.268547+0000","last_active":"2024-02-19T14:52:06.225756+0000","last_peered":"2024-02-19T14:52:06.225756+0000","last_clean":"2024-02-19T14:52:06.225756+0000","last_became_active":"2024-02-19T14:52:00.268241+0000","last_became_peered":"2024-02-19T14:52:00.268241+0000","last_unstale":"2024-02-19T14:52:06.225756+0000","last_undegraded":"2024-02-19T14:52:06.225756+0000","last_fullsized":"2024-02-19T14:52:06.225756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530412+0000","last_change":"2024-02-19T14:52:00.262779+0000","last_active":"2024-02-19T14:52:05.530412+0000","last_peered":"2024-02-19T14:52:05.530412+0000","last_clean":"2024-02-19T14:52:05.530412+0000","last_became_active":"2024-02-19T14:52:00.262476+0000","last_became_peered":"2024-02-19T14:52:00.262476+0000","last_unstale":"2024-02-19T14:52:05.530412+0000","last_undegraded":"2024-02-19T14:52:05.530412+0000","last_fullsized":"2024-02-19T14:52:05.530412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247311+0000","last_change":"2024-02-19T14:52:00.272996+0000","last_active":"2024-02-19T14:52:06.247311+0000","last_peered":"2024-02-19T14:52:06.247311+0000","last_clean":"2024-02-19T14:52:06.247311+0000","last_became_active":"2024-02-19T14:52:00.271013+0000","last_became_peered":"2024-02-19T14:52:00.271013+0000","last_unstale":"2024-02-19T14:52:06.247311+0000","last_undegraded":"2024-02-19T14:52:06.247311+0000","last_fullsized":"2024-02-19T14:52:06.247311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281593+0000","last_change":"2024-02-19T14:52:00.263763+0000","last_active":"2024-02-19T14:52:05.281593+0000","last_peered":"2024-02-19T14:52:05.281593+0000","last_clean":"2024-02-19T14:52:05.281593+0000","last_became_active":"2024-02-19T14:52:00.263488+0000","last_became_peered":"2024-02-19T14:52:00.263488+0000","last_unstale":"2024-02-19T14:52:05.281593+0000","last_undegraded":"2024-02-19T14:52:05.281593+0000","last_fullsized":"2024-02-19T14:52:05.281593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:46.599373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284503+0000","last_change":"2024-02-19T14:52:00.258879+0000","last_active":"2024-02-19T14:52:05.284503+0000","last_peered":"2024-02-19T14:52:05.284503+0000","last_clean":"2024-02-19T14:52:05.284503+0000","last_became_active":"2024-02-19T14:52:00.257612+0000","last_became_peered":"2024-02-19T14:52:00.257612+0000","last_unstale":"2024-02-19T14:52:05.284503+0000","last_undegraded":"2024-02-19T14:52:05.284503+0000","last_fullsized":"2024-02-19T14:52:05.284503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282447+0000","last_change":"2024-02-19T14:52:00.268671+0000","last_active":"2024-02-19T14:52:05.282447+0000","last_peered":"2024-02-19T14:52:05.282447+0000","last_clean":"2024-02-19T14:52:05.282447+0000","last_became_active":"2024-02-19T14:52:00.268351+0000","last_became_peered":"2024-02-19T14:52:00.268351+0000","last_unstale":"2024-02-19T14:52:05.282447+0000","last_undegraded":"2024-02-19T14:52:05.282447+0000","last_fullsized":"2024-02-19T14:52:05.282447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228525+0000","last_change":"2024-02-19T14:52:00.249695+0000","last_active":"2024-02-19T14:52:07.228525+0000","last_peered":"2024-02-19T14:52:07.228525+0000","last_clean":"2024-02-19T14:52:07.228525+0000","last_became_active":"2024-02-19T14:52:00.249406+0000","last_became_peered":"2024-02-19T14:52:00.249406+0000","last_unstale":"2024-02-19T14:52:07.228525+0000","last_undegraded":"2024-02-19T14:52:07.228525+0000","last_fullsized":"2024-02-19T14:52:07.228525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225669+0000","last_change":"2024-02-19T14:52:00.259566+0000","last_active":"2024-02-19T14:52:06.225669+0000","last_peered":"2024-02-19T14:52:06.225669+0000","last_clean":"2024-02-19T14:52:06.225669+0000","last_became_active":"2024-02-19T14:52:00.259290+0000","last_became_peered":"2024-02-19T14:52:00.259290+0000","last_unstale":"2024-02-19T14:52:06.225669+0000","last_undegraded":"2024-02-19T14:52:06.225669+0000","last_fullsized":"2024-02-19T14:52:06.225669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334965+0000","last_change":"2024-02-19T14:52:00.267359+0000","last_active":"2024-02-19T14:52:07.334965+0000","last_peered":"2024-02-19T14:52:07.334965+0000","last_clean":"2024-02-19T14:52:07.334965+0000","last_became_active":"2024-02-19T14:52:00.266119+0000","last_became_peered":"2024-02-19T14:52:00.266119+0000","last_unstale":"2024-02-19T14:52:07.334965+0000","last_undegraded":"2024-02-19T14:52:07.334965+0000","last_fullsized":"2024-02-19T14:52:07.334965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283181+0000","last_change":"2024-02-19T14:52:00.262531+0000","last_active":"2024-02-19T14:52:05.283181+0000","last_peered":"2024-02-19T14:52:05.283181+0000","last_clean":"2024-02-19T14:52:05.283181+0000","last_became_active":"2024-02-19T14:52:00.262240+0000","last_became_peered":"2024-02-19T14:52:00.262240+0000","last_unstale":"2024-02-19T14:52:05.283181+0000","last_undegraded":"2024-02-19T14:52:05.283181+0000","last_fullsized":"2024-02-19T14:52:05.283181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:41.341079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530384+0000","last_change":"2024-02-19T14:52:00.266149+0000","last_active":"2024-02-19T14:52:05.530384+0000","last_peered":"2024-02-19T14:52:05.530384+0000","last_clean":"2024-02-19T14:52:05.530384+0000","last_became_active":"2024-02-19T14:52:00.265873+0000","last_became_peered":"2024-02-19T14:52:00.265873+0000","last_unstale":"2024-02-19T14:52:05.530384+0000","last_undegraded":"2024-02-19T14:52:05.530384+0000","last_fullsized":"2024-02-19T14:52:05.530384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:52:00.266144+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:52:00.265852+0000","last_became_peered":"2024-02-19T14:52:00.265852+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247502+0000","last_change":"2024-02-19T14:52:00.265908+0000","last_active":"2024-02-19T14:52:06.247502+0000","last_peered":"2024-02-19T14:52:06.247502+0000","last_clean":"2024-02-19T14:52:06.247502+0000","last_became_active":"2024-02-19T14:52:00.265630+0000","last_became_peered":"2024-02-19T14:52:00.265630+0000","last_unstale":"2024-02-19T14:52:06.247502+0000","last_undegraded":"2024-02-19T14:52:06.247502+0000","last_fullsized":"2024-02-19T14:52:06.247502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:06.056284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334181+0000","last_change":"2024-02-19T14:52:00.267195+0000","last_active":"2024-02-19T14:52:07.334181+0000","last_peered":"2024-02-19T14:52:07.334181+0000","last_clean":"2024-02-19T14:52:07.334181+0000","last_became_active":"2024-02-19T14:52:00.266185+0000","last_became_peered":"2024-02-19T14:52:00.266185+0000","last_unstale":"2024-02-19T14:52:07.334181+0000","last_undegraded":"2024-02-19T14:52:07.334181+0000","last_fullsized":"2024-02-19T14:52:07.334181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246858+0000","last_change":"2024-02-19T14:52:00.266943+0000","last_active":"2024-02-19T14:52:06.246858+0000","last_peered":"2024-02-19T14:52:06.246858+0000","last_clean":"2024-02-19T14:52:06.246858+0000","last_became_active":"2024-02-19T14:52:00.266665+0000","last_became_peered":"2024-02-19T14:52:00.266665+0000","last_unstale":"2024-02-19T14:52:06.246858+0000","last_undegraded":"2024-02-19T14:52:06.246858+0000","last_fullsized":"2024-02-19T14:52:06.246858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282952+0000","last_change":"2024-02-19T14:52:00.261505+0000","last_active":"2024-02-19T14:52:05.282952+0000","last_peered":"2024-02-19T14:52:05.282952+0000","last_clean":"2024-02-19T14:52:05.282952+0000","last_became_active":"2024-02-19T14:52:00.261212+0000","last_became_peered":"2024-02-19T14:52:00.261212+0000","last_unstale":"2024-02-19T14:52:05.282952+0000","last_undegraded":"2024-02-19T14:52:05.282952+0000","last_fullsized":"2024-02-19T14:52:05.282952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246984+0000","last_change":"2024-02-19T14:52:00.261481+0000","last_active":"2024-02-19T14:52:06.246984+0000","last_peered":"2024-02-19T14:52:06.246984+0000","last_clean":"2024-02-19T14:52:06.246984+0000","last_became_active":"2024-02-19T14:52:00.261205+0000","last_became_peered":"2024-02-19T14:52:00.261205+0000","last_unstale":"2024-02-19T14:52:06.246984+0000","last_undegraded":"2024-02-19T14:52:06.246984+0000","last_fullsized":"2024-02-19T14:52:06.246984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283153+0000","last_change":"2024-02-19T14:52:00.266814+0000","last_active":"2024-02-19T14:52:05.283153+0000","last_peered":"2024-02-19T14:52:05.283153+0000","last_clean":"2024-02-19T14:52:05.283153+0000","last_became_active":"2024-02-19T14:52:00.266019+0000","last_became_peered":"2024-02-19T14:52:00.266019+0000","last_unstale":"2024-02-19T14:52:05.283153+0000","last_undegraded":"2024-02-19T14:52:05.283153+0000","last_fullsized":"2024-02-19T14:52:05.283153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225905+0000","last_change":"2024-02-19T14:52:00.260863+0000","last_active":"2024-02-19T14:52:06.225905+0000","last_peered":"2024-02-19T14:52:06.225905+0000","last_clean":"2024-02-19T14:52:06.225905+0000","last_became_active":"2024-02-19T14:52:00.260543+0000","last_became_peered":"2024-02-19T14:52:00.260543+0000","last_unstale":"2024-02-19T14:52:06.225905+0000","last_undegraded":"2024-02-19T14:52:06.225905+0000","last_fullsized":"2024-02-19T14:52:06.225905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228999+0000","last_change":"2024-02-19T14:52:00.260306+0000","last_active":"2024-02-19T14:52:07.228999+0000","last_peered":"2024-02-19T14:52:07.228999+0000","last_clean":"2024-02-19T14:52:07.228999+0000","last_became_active":"2024-02-19T14:52:00.260026+0000","last_became_peered":"2024-02-19T14:52:00.260026+0000","last_unstale":"2024-02-19T14:52:07.228999+0000","last_undegraded":"2024-02-19T14:52:07.228999+0000","last_fullsized":"2024-02-19T14:52:07.228999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247442+0000","last_change":"2024-02-19T14:52:00.263320+0000","last_active":"2024-02-19T14:52:06.247442+0000","last_peered":"2024-02-19T14:52:06.247442+0000","last_clean":"2024-02-19T14:52:06.247442+0000","last_became_active":"2024-02-19T14:52:00.263044+0000","last_became_peered":"2024-02-19T14:52:00.263044+0000","last_unstale":"2024-02-19T14:52:06.247442+0000","last_undegraded":"2024-02-19T14:52:06.247442+0000","last_fullsized":"2024-02-19T14:52:06.247442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:08.747083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:52:17.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:52:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.440+0000 7fcb580eb640 1 Processor -- start 2024-02-19T14:52:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.440+0000 7fcb580eb640 1 -- start start 2024-02-19T14:52:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb580eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcb50155da0 0x7fcb500c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb580eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb500c6960 0x7fcb500c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb580eb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500c27e0 con 0x7fcb500adde0 2024-02-19T14:52:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb580eb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb500c3a90 con 0x7fcb50155da0 2024-02-19T14:52:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb580eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500c3d70 con 0x7fcb500c6960 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb55e87640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcb50155da0 0x7fcb500c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb55e87640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcb50155da0 0x7fcb500c72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59482/0 (socket says 172.21.15.78:59482) 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb55e87640 1 -- 172.21.15.78:0/2008179078 learned_addr learned my addr 172.21.15.78:0/2008179078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.441+0000 7fcb55686640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb500c6960 0x7fcb500c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb55e87640 1 -- 172.21.15.78:0/2008179078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb500c6960 msgr2=0x7fcb500c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb55e87640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb500c6960 0x7fcb500c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb55e87640 1 -- 172.21.15.78:0/2008179078 >> v1:172.21.15.150:6789/0 conn(0x7fcb500adde0 legacy=0x7fcb500c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb55e87640 1 -- 172.21.15.78:0/2008179078 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5006d070 con 0x7fcb50155da0 2024-02-19T14:52:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb55686640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb500c6960 0x7fcb500c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb55e87640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb50155da0 0x7fcb500c72e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009e50 tx=0x7fcb4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ecf979ee2b4e86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb54e85640 1 -- 172.21.15.78:0/2008179078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4802ca70 con 0x7fcb50155da0 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb54e85640 1 -- 172.21.15.78:0/2008179078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4802cbd0 con 0x7fcb50155da0 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.442+0000 7fcb54e85640 1 -- 172.21.15.78:0/2008179078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480343f0 con 0x7fcb50155da0 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 -- 172.21.15.78:0/2008179078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb50155da0 msgr2=0x7fcb500c72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb50155da0 0x7fcb500c72e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009e50 tx=0x7fcb4802bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 -- 172.21.15.78:0/2008179078 shutdown_connections 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb500c6960 0x7fcb500c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2008179078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb50155da0 0x7fcb500c72e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 -- 172.21.15.78:0/2008179078 >> 172.21.15.78:0/2008179078 conn(0x7fcb500aa3e0 msgr2=0x7fcb500aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 -- 172.21.15.78:0/2008179078 shutdown_connections 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.443+0000 7fcb580eb640 1 -- 172.21.15.78:0/2008179078 wait complete. 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.444+0000 7fcb580eb640 1 Processor -- start 2024-02-19T14:52:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.444+0000 7fcb580eb640 1 -- start start 2024-02-19T14:52:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb580eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb500adde0 0x7fcb501ea730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb580eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 0x7fcb501ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb580eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb50155da0 0x7fcb501e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb580eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501ee070 con 0x7fcb500c6960 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb580eb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50157740 con 0x7fcb500adde0 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb580eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501edd20 con 0x7fcb50155da0 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb55686640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 0x7fcb501ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb55686640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 0x7fcb501ee490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59494/0 (socket says 172.21.15.78:59494) 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.445+0000 7fcb55686640 1 -- 172.21.15.78:0/2764465843 learned_addr learned my addr 172.21.15.78:0/2764465843 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb56688640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb50155da0 0x7fcb501e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55e87640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb500adde0 0x7fcb501ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55686640 1 -- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb50155da0 msgr2=0x7fcb501e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55686640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb50155da0 0x7fcb501e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55686640 1 -- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb500adde0 msgr2=0x7fcb501ea730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55686640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb500adde0 0x7fcb501ea730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55686640 1 -- 172.21.15.78:0/2764465843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009b30 con 0x7fcb500c6960 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb56688640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb50155da0 0x7fcb501e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55e87640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb500adde0 0x7fcb501ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb55686640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 0x7fcb501ee490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400c7f0 tx=0x7fcb4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400cea0 con 0x7fcb500c6960 2024-02-19T14:52:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb44004230 con 0x7fcb500c6960 2024-02-19T14:52:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5006d710 con 0x7fcb500c6960 2024-02-19T14:52:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44007830 con 0x7fcb500c6960 2024-02-19T14:52:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.446+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb500ac750 con 0x7fcb500c6960 2024-02-19T14:52:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.447+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5006e120 con 0x7fcb500c6960 2024-02-19T14:52:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.448+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcb44007990 con 0x7fcb500c6960 2024-02-19T14:52:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.448+0000 7fcb427fc640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb3406b810 0x7fcb3406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.448+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcb4408a410 con 0x7fcb500c6960 2024-02-19T14:52:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.448+0000 7fcb55e87640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb3406b810 0x7fcb3406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.448+0000 7fcb55e87640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb3406b810 0x7fcb3406dcd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcb48035000 tx=0x7fcb48030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.452+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcb44054b80 con 0x7fcb500c6960 2024-02-19T14:52:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.620+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb50074990 con 0x7fcb3406b810 2024-02-19T14:52:17.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.636+0000 7fcb427fc640 1 -- 172.21.15.78:0/2764465843 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345318 (secure 0 0 0) 0x7fcb50074990 con 0x7fcb3406b810 2024-02-19T14:52:17.646 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:17.649 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:52:17.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb3406b810 msgr2=0x7fcb3406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb3406b810 0x7fcb3406dcd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcb48035000 tx=0x7fcb48030000 comp rx=0 tx=0).stop 2024-02-19T14:52:17.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 msgr2=0x7fcb501ee490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 0x7fcb501ee490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400c7f0 tx=0x7fcb4400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 shutdown_connections 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb3406b810 0x7fcb3406dcd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb500c6960 0x7fcb501ee490 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb50155da0 0x7fcb501e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 --2- 172.21.15.78:0/2764465843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb500adde0 0x7fcb501ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 >> 172.21.15.78:0/2764465843 conn(0x7fcb500aa3e0 msgr2=0x7fcb500aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 shutdown_connections 2024-02-19T14:52:17.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.640+0000 7fcb580eb640 1 -- 172.21.15.78:0/2764465843 wait complete. 2024-02-19T14:52:17.654 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-02-19T14:52:16.063123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219384,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755336,"statfs":{"total":773094113280,"available":772869464064,"internally_reserved":0,"allocated":3284992,"data_stored":1991782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.782425"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226366+0000","last_change":"2024-02-19T14:51:59.266929+0000","last_active":"2024-02-19T14:52:06.226366+0000","last_peered":"2024-02-19T14:52:06.226366+0000","last_clean":"2024-02-19T14:52:06.226366+0000","last_became_active":"2024-02-19T14:51:59.266644+0000","last_became_peered":"2024-02-19T14:51:59.266644+0000","last_unstale":"2024-02-19T14:52:06.226366+0000","last_undegraded":"2024-02-19T14:52:06.226366+0000","last_fullsized":"2024-02-19T14:52:06.226366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335294+0000","last_change":"2024-02-19T14:52:00.264687+0000","last_active":"2024-02-19T14:52:07.335294+0000","last_peered":"2024-02-19T14:52:07.335294+0000","last_clean":"2024-02-19T14:52:07.335294+0000","last_became_active":"2024-02-19T14:52:00.264424+0000","last_became_peered":"2024-02-19T14:52:00.264424+0000","last_unstale":"2024-02-19T14:52:07.335294+0000","last_undegraded":"2024-02-19T14:52:07.335294+0000","last_fullsized":"2024-02-19T14:52:07.335294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334297+0000","last_change":"2024-02-19T14:52:04.435336+0000","last_active":"2024-02-19T14:52:07.334297+0000","last_peered":"2024-02-19T14:52:07.334297+0000","last_clean":"2024-02-19T14:52:07.334297+0000","last_became_active":"2024-02-19T14:52:04.434977+0000","last_became_peered":"2024-02-19T14:52:04.434977+0000","last_unstale":"2024-02-19T14:52:07.334297+0000","last_undegraded":"2024-02-19T14:52:07.334297+0000","last_fullsized":"2024-02-19T14:52:07.334297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334116+0000","last_change":"2024-02-19T14:51:59.262430+0000","last_active":"2024-02-19T14:52:07.334116+0000","last_peered":"2024-02-19T14:52:07.334116+0000","last_clean":"2024-02-19T14:52:07.334116+0000","last_became_active":"2024-02-19T14:51:59.262168+0000","last_became_peered":"2024-02-19T14:51:59.262168+0000","last_unstale":"2024-02-19T14:52:07.334116+0000","last_undegraded":"2024-02-19T14:52:07.334116+0000","last_fullsized":"2024-02-19T14:52:07.334116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226267+0000","last_change":"2024-02-19T14:52:00.263014+0000","last_active":"2024-02-19T14:52:06.226267+0000","last_peered":"2024-02-19T14:52:06.226267+0000","last_clean":"2024-02-19T14:52:06.226267+0000","last_became_active":"2024-02-19T14:52:00.262531+0000","last_became_peered":"2024-02-19T14:52:00.262531+0000","last_unstale":"2024-02-19T14:52:06.226267+0000","last_undegraded":"2024-02-19T14:52:06.226267+0000","last_fullsized":"2024-02-19T14:52:06.226267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282898+0000","last_change":"2024-02-19T14:52:04.277572+0000","last_active":"2024-02-19T14:52:05.282898+0000","last_peered":"2024-02-19T14:52:05.282898+0000","last_clean":"2024-02-19T14:52:05.282898+0000","last_became_active":"2024-02-19T14:52:04.276970+0000","last_became_peered":"2024-02-19T14:52:04.276970+0000","last_unstale":"2024-02-19T14:52:05.282898+0000","last_undegraded":"2024-02-19T14:52:05.282898+0000","last_fullsized":"2024-02-19T14:52:05.282898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:14.095185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247290+0000","last_change":"2024-02-19T14:51:59.265003+0000","last_active":"2024-02-19T14:52:06.247290+0000","last_peered":"2024-02-19T14:52:06.247290+0000","last_clean":"2024-02-19T14:52:06.247290+0000","last_became_active":"2024-02-19T14:51:59.264719+0000","last_became_peered":"2024-02-19T14:51:59.264719+0000","last_unstale":"2024-02-19T14:52:06.247290+0000","last_undegraded":"2024-02-19T14:52:06.247290+0000","last_fullsized":"2024-02-19T14:52:06.247290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283484+0000","last_change":"2024-02-19T14:52:00.252606+0000","last_active":"2024-02-19T14:52:05.283484+0000","last_peered":"2024-02-19T14:52:05.283484+0000","last_clean":"2024-02-19T14:52:05.283484+0000","last_became_active":"2024-02-19T14:52:00.249782+0000","last_became_peered":"2024-02-19T14:52:00.249782+0000","last_unstale":"2024-02-19T14:52:05.283484+0000","last_undegraded":"2024-02-19T14:52:05.283484+0000","last_fullsized":"2024-02-19T14:52:05.283484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283979+0000","last_change":"2024-02-19T14:52:04.278014+0000","last_active":"2024-02-19T14:52:05.283979+0000","last_peered":"2024-02-19T14:52:05.283979+0000","last_clean":"2024-02-19T14:52:05.283979+0000","last_became_active":"2024-02-19T14:52:04.277126+0000","last_became_peered":"2024-02-19T14:52:04.277126+0000","last_unstale":"2024-02-19T14:52:05.283979+0000","last_undegraded":"2024-02-19T14:52:05.283979+0000","last_fullsized":"2024-02-19T14:52:05.283979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:27.674760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247080+0000","last_change":"2024-02-19T14:51:59.263462+0000","last_active":"2024-02-19T14:52:06.247080+0000","last_peered":"2024-02-19T14:52:06.247080+0000","last_clean":"2024-02-19T14:52:06.247080+0000","last_became_active":"2024-02-19T14:51:59.262932+0000","last_became_peered":"2024-02-19T14:51:59.262932+0000","last_unstale":"2024-02-19T14:52:06.247080+0000","last_undegraded":"2024-02-19T14:52:06.247080+0000","last_fullsized":"2024-02-19T14:52:06.247080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284101+0000","last_change":"2024-02-19T14:52:00.252460+0000","last_active":"2024-02-19T14:52:05.284101+0000","last_peered":"2024-02-19T14:52:05.284101+0000","last_clean":"2024-02-19T14:52:05.284101+0000","last_became_active":"2024-02-19T14:52:00.251603+0000","last_became_peered":"2024-02-19T14:52:00.251603+0000","last_unstale":"2024-02-19T14:52:05.284101+0000","last_undegraded":"2024-02-19T14:52:05.284101+0000","last_fullsized":"2024-02-19T14:52:05.284101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283155+0000","last_change":"2024-02-19T14:52:04.278038+0000","last_active":"2024-02-19T14:52:05.283155+0000","last_peered":"2024-02-19T14:52:05.283155+0000","last_clean":"2024-02-19T14:52:05.283155+0000","last_became_active":"2024-02-19T14:52:04.277620+0000","last_became_peered":"2024-02-19T14:52:04.277620+0000","last_unstale":"2024-02-19T14:52:05.283155+0000","last_undegraded":"2024-02-19T14:52:05.283155+0000","last_fullsized":"2024-02-19T14:52:05.283155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:12.987808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530027+0000","last_change":"2024-02-19T14:52:00.264133+0000","last_active":"2024-02-19T14:52:05.530027+0000","last_peered":"2024-02-19T14:52:05.530027+0000","last_clean":"2024-02-19T14:52:05.530027+0000","last_became_active":"2024-02-19T14:52:00.263475+0000","last_became_peered":"2024-02-19T14:52:00.263475+0000","last_unstale":"2024-02-19T14:52:05.530027+0000","last_undegraded":"2024-02-19T14:52:05.530027+0000","last_fullsized":"2024-02-19T14:52:05.530027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247110+0000","last_change":"2024-02-19T14:51:59.263095+0000","last_active":"2024-02-19T14:52:06.247110+0000","last_peered":"2024-02-19T14:52:06.247110+0000","last_clean":"2024-02-19T14:52:06.247110+0000","last_became_active":"2024-02-19T14:51:59.262392+0000","last_became_peered":"2024-02-19T14:51:59.262392+0000","last_unstale":"2024-02-19T14:52:06.247110+0000","last_undegraded":"2024-02-19T14:52:06.247110+0000","last_fullsized":"2024-02-19T14:52:06.247110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:04.278534+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:04.277850+0000","last_became_peered":"2024-02-19T14:52:04.277850+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334968+0000","last_change":"2024-02-19T14:52:00.268782+0000","last_active":"2024-02-19T14:52:07.334968+0000","last_peered":"2024-02-19T14:52:07.334968+0000","last_clean":"2024-02-19T14:52:07.334968+0000","last_became_active":"2024-02-19T14:52:00.266478+0000","last_became_peered":"2024-02-19T14:52:00.266478+0000","last_unstale":"2024-02-19T14:52:07.334968+0000","last_undegraded":"2024-02-19T14:52:07.334968+0000","last_fullsized":"2024-02-19T14:52:07.334968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228818+0000","last_change":"2024-02-19T14:51:59.260219+0000","last_active":"2024-02-19T14:52:07.228818+0000","last_peered":"2024-02-19T14:52:07.228818+0000","last_clean":"2024-02-19T14:52:07.228818+0000","last_became_active":"2024-02-19T14:51:59.259917+0000","last_became_peered":"2024-02-19T14:51:59.259917+0000","last_unstale":"2024-02-19T14:52:07.228818+0000","last_undegraded":"2024-02-19T14:52:07.228818+0000","last_fullsized":"2024-02-19T14:52:07.228818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282889+0000","last_change":"2024-02-19T14:52:04.277635+0000","last_active":"2024-02-19T14:52:05.282889+0000","last_peered":"2024-02-19T14:52:05.282889+0000","last_clean":"2024-02-19T14:52:05.282889+0000","last_became_active":"2024-02-19T14:52:04.277231+0000","last_became_peered":"2024-02-19T14:52:04.277231+0000","last_unstale":"2024-02-19T14:52:05.282889+0000","last_undegraded":"2024-02-19T14:52:05.282889+0000","last_fullsized":"2024-02-19T14:52:05.282889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:46.471360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334393+0000","last_change":"2024-02-19T14:52:00.263688+0000","last_active":"2024-02-19T14:52:07.334393+0000","last_peered":"2024-02-19T14:52:07.334393+0000","last_clean":"2024-02-19T14:52:07.334393+0000","last_became_active":"2024-02-19T14:52:00.263381+0000","last_became_peered":"2024-02-19T14:52:00.263381+0000","last_unstale":"2024-02-19T14:52:07.334393+0000","last_undegraded":"2024-02-19T14:52:07.334393+0000","last_fullsized":"2024-02-19T14:52:07.334393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228547+0000","last_change":"2024-02-19T14:51:59.262590+0000","last_active":"2024-02-19T14:52:07.228547+0000","last_peered":"2024-02-19T14:52:07.228547+0000","last_clean":"2024-02-19T14:52:07.228547+0000","last_became_active":"2024-02-19T14:51:59.262310+0000","last_became_peered":"2024-02-19T14:51:59.262310+0000","last_unstale":"2024-02-19T14:52:07.228547+0000","last_undegraded":"2024-02-19T14:52:07.228547+0000","last_fullsized":"2024-02-19T14:52:07.228547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229123+0000","last_change":"2024-02-19T14:52:04.434727+0000","last_active":"2024-02-19T14:52:07.229123+0000","last_peered":"2024-02-19T14:52:07.229123+0000","last_clean":"2024-02-19T14:52:07.229123+0000","last_became_active":"2024-02-19T14:52:04.432708+0000","last_became_peered":"2024-02-19T14:52:04.432708+0000","last_unstale":"2024-02-19T14:52:07.229123+0000","last_undegraded":"2024-02-19T14:52:07.229123+0000","last_fullsized":"2024-02-19T14:52:07.229123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:19.410143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284506+0000","last_change":"2024-02-19T14:52:00.260620+0000","last_active":"2024-02-19T14:52:05.284506+0000","last_peered":"2024-02-19T14:52:05.284506+0000","last_clean":"2024-02-19T14:52:05.284506+0000","last_became_active":"2024-02-19T14:52:00.260357+0000","last_became_peered":"2024-02-19T14:52:00.260357+0000","last_unstale":"2024-02-19T14:52:05.284506+0000","last_undegraded":"2024-02-19T14:52:05.284506+0000","last_fullsized":"2024-02-19T14:52:05.284506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:09.799959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246702+0000","last_change":"2024-02-19T14:51:59.267802+0000","last_active":"2024-02-19T14:52:06.246702+0000","last_peered":"2024-02-19T14:52:06.246702+0000","last_clean":"2024-02-19T14:52:06.246702+0000","last_became_active":"2024-02-19T14:51:59.267518+0000","last_became_peered":"2024-02-19T14:51:59.267518+0000","last_unstale":"2024-02-19T14:52:06.246702+0000","last_undegraded":"2024-02-19T14:52:06.246702+0000","last_fullsized":"2024-02-19T14:52:06.246702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:45.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283176+0000","last_change":"2024-02-19T14:52:04.276717+0000","last_active":"2024-02-19T14:52:05.283176+0000","last_peered":"2024-02-19T14:52:05.283176+0000","last_clean":"2024-02-19T14:52:05.283176+0000","last_became_active":"2024-02-19T14:52:04.276273+0000","last_became_peered":"2024-02-19T14:52:04.276273+0000","last_unstale":"2024-02-19T14:52:05.283176+0000","last_undegraded":"2024-02-19T14:52:05.283176+0000","last_fullsized":"2024-02-19T14:52:05.283176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:30.898405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226156+0000","last_change":"2024-02-19T14:52:00.259379+0000","last_active":"2024-02-19T14:52:06.226156+0000","last_peered":"2024-02-19T14:52:06.226156+0000","last_clean":"2024-02-19T14:52:06.226156+0000","last_became_active":"2024-02-19T14:52:00.259061+0000","last_became_peered":"2024-02-19T14:52:00.259061+0000","last_unstale":"2024-02-19T14:52:06.226156+0000","last_undegraded":"2024-02-19T14:52:06.226156+0000","last_fullsized":"2024-02-19T14:52:06.226156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:51:59.262618+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:51:59.262333+0000","last_became_peered":"2024-02-19T14:51:59.262333+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284070+0000","last_change":"2024-02-19T14:52:04.436152+0000","last_active":"2024-02-19T14:52:05.284070+0000","last_peered":"2024-02-19T14:52:05.284070+0000","last_clean":"2024-02-19T14:52:05.284070+0000","last_became_active":"2024-02-19T14:52:04.435251+0000","last_became_peered":"2024-02-19T14:52:04.435251+0000","last_unstale":"2024-02-19T14:52:05.284070+0000","last_undegraded":"2024-02-19T14:52:05.284070+0000","last_fullsized":"2024-02-19T14:52:05.284070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:33.762974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335436+0000","last_change":"2024-02-19T14:52:00.272464+0000","last_active":"2024-02-19T14:52:07.335436+0000","last_peered":"2024-02-19T14:52:07.335436+0000","last_clean":"2024-02-19T14:52:07.335436+0000","last_became_active":"2024-02-19T14:52:00.272184+0000","last_became_peered":"2024-02-19T14:52:00.272184+0000","last_unstale":"2024-02-19T14:52:07.335436+0000","last_undegraded":"2024-02-19T14:52:07.335436+0000","last_fullsized":"2024-02-19T14:52:07.335436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.256170+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.255855+0000","last_became_peered":"2024-02-19T14:51:59.255855+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334561+0000","last_change":"2024-02-19T14:52:04.433402+0000","last_active":"2024-02-19T14:52:07.334561+0000","last_peered":"2024-02-19T14:52:07.334561+0000","last_clean":"2024-02-19T14:52:07.334561+0000","last_became_active":"2024-02-19T14:52:04.432815+0000","last_became_peered":"2024-02-19T14:52:04.432815+0000","last_unstale":"2024-02-19T14:52:07.334561+0000","last_undegraded":"2024-02-19T14:52:07.334561+0000","last_fullsized":"2024-02-19T14:52:07.334561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:20.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228931+0000","last_change":"2024-02-19T14:52:00.261954+0000","last_active":"2024-02-19T14:52:07.228931+0000","last_peered":"2024-02-19T14:52:07.228931+0000","last_clean":"2024-02-19T14:52:07.228931+0000","last_became_active":"2024-02-19T14:52:00.261678+0000","last_became_peered":"2024-02-19T14:52:00.261678+0000","last_unstale":"2024-02-19T14:52:07.228931+0000","last_undegraded":"2024-02-19T14:52:07.228931+0000","last_fullsized":"2024-02-19T14:52:07.228931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247420+0000","last_change":"2024-02-19T14:51:59.267996+0000","last_active":"2024-02-19T14:52:06.247420+0000","last_peered":"2024-02-19T14:52:06.247420+0000","last_clean":"2024-02-19T14:52:06.247420+0000","last_became_active":"2024-02-19T14:51:59.267426+0000","last_became_peered":"2024-02-19T14:51:59.267426+0000","last_unstale":"2024-02-19T14:52:06.247420+0000","last_undegraded":"2024-02-19T14:52:06.247420+0000","last_fullsized":"2024-02-19T14:52:06.247420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225861+0000","last_change":"2024-02-19T14:52:04.433162+0000","last_active":"2024-02-19T14:52:06.225861+0000","last_peered":"2024-02-19T14:52:06.225861+0000","last_clean":"2024-02-19T14:52:06.225861+0000","last_became_active":"2024-02-19T14:52:04.432722+0000","last_became_peered":"2024-02-19T14:52:04.432722+0000","last_unstale":"2024-02-19T14:52:06.225861+0000","last_undegraded":"2024-02-19T14:52:06.225861+0000","last_fullsized":"2024-02-19T14:52:06.225861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:00.272249+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:00.266375+0000","last_became_peered":"2024-02-19T14:52:00.266375+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229033+0000","last_change":"2024-02-19T14:51:59.260559+0000","last_active":"2024-02-19T14:52:07.229033+0000","last_peered":"2024-02-19T14:52:07.229033+0000","last_clean":"2024-02-19T14:52:07.229033+0000","last_became_active":"2024-02-19T14:51:59.260290+0000","last_became_peered":"2024-02-19T14:51:59.260290+0000","last_unstale":"2024-02-19T14:52:07.229033+0000","last_undegraded":"2024-02-19T14:52:07.229033+0000","last_fullsized":"2024-02-19T14:52:07.229033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228634+0000","last_change":"2024-02-19T14:52:04.283819+0000","last_active":"2024-02-19T14:52:07.228634+0000","last_peered":"2024-02-19T14:52:07.228634+0000","last_clean":"2024-02-19T14:52:07.228634+0000","last_became_active":"2024-02-19T14:52:04.283469+0000","last_became_peered":"2024-02-19T14:52:04.283469+0000","last_unstale":"2024-02-19T14:52:07.228634+0000","last_undegraded":"2024-02-19T14:52:07.228634+0000","last_fullsized":"2024-02-19T14:52:07.228634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284410+0000","last_change":"2024-02-19T14:52:00.266384+0000","last_active":"2024-02-19T14:52:05.284410+0000","last_peered":"2024-02-19T14:52:05.284410+0000","last_clean":"2024-02-19T14:52:05.284410+0000","last_became_active":"2024-02-19T14:52:00.265954+0000","last_became_peered":"2024-02-19T14:52:00.265954+0000","last_unstale":"2024-02-19T14:52:05.284410+0000","last_undegraded":"2024-02-19T14:52:05.284410+0000","last_fullsized":"2024-02-19T14:52:05.284410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283363+0000","last_change":"2024-02-19T14:51:59.259377+0000","last_active":"2024-02-19T14:52:05.283363+0000","last_peered":"2024-02-19T14:52:05.283363+0000","last_clean":"2024-02-19T14:52:05.283363+0000","last_became_active":"2024-02-19T14:51:59.259004+0000","last_became_peered":"2024-02-19T14:51:59.259004+0000","last_unstale":"2024-02-19T14:52:05.283363+0000","last_undegraded":"2024-02-19T14:52:05.283363+0000","last_fullsized":"2024-02-19T14:52:05.283363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229000+0000","last_change":"2024-02-19T14:52:04.282027+0000","last_active":"2024-02-19T14:52:07.229000+0000","last_peered":"2024-02-19T14:52:07.229000+0000","last_clean":"2024-02-19T14:52:07.229000+0000","last_became_active":"2024-02-19T14:52:04.281679+0000","last_became_peered":"2024-02-19T14:52:04.281679+0000","last_unstale":"2024-02-19T14:52:07.229000+0000","last_undegraded":"2024-02-19T14:52:07.229000+0000","last_fullsized":"2024-02-19T14:52:07.229000+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:41.900564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228565+0000","last_change":"2024-02-19T14:52:00.261302+0000","last_active":"2024-02-19T14:52:07.228565+0000","last_peered":"2024-02-19T14:52:07.228565+0000","last_clean":"2024-02-19T14:52:07.228565+0000","last_became_active":"2024-02-19T14:52:00.261027+0000","last_became_peered":"2024-02-19T14:52:00.261027+0000","last_unstale":"2024-02-19T14:52:07.228565+0000","last_undegraded":"2024-02-19T14:52:07.228565+0000","last_fullsized":"2024-02-19T14:52:07.228565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281412+0000","last_change":"2024-02-19T14:51:59.256034+0000","last_active":"2024-02-19T14:52:05.281412+0000","last_peered":"2024-02-19T14:52:05.281412+0000","last_clean":"2024-02-19T14:52:05.281412+0000","last_became_active":"2024-02-19T14:51:59.255701+0000","last_became_peered":"2024-02-19T14:51:59.255701+0000","last_unstale":"2024-02-19T14:52:05.281412+0000","last_undegraded":"2024-02-19T14:52:05.281412+0000","last_fullsized":"2024-02-19T14:52:05.281412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229180+0000","last_change":"2024-02-19T14:52:04.283973+0000","last_active":"2024-02-19T14:52:07.229180+0000","last_peered":"2024-02-19T14:52:07.229180+0000","last_clean":"2024-02-19T14:52:07.229180+0000","last_became_active":"2024-02-19T14:52:04.283562+0000","last_became_peered":"2024-02-19T14:52:04.283562+0000","last_unstale":"2024-02-19T14:52:07.229180+0000","last_undegraded":"2024-02-19T14:52:07.229180+0000","last_fullsized":"2024-02-19T14:52:07.229180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225524+0000","last_change":"2024-02-19T14:52:00.260330+0000","last_active":"2024-02-19T14:52:06.225524+0000","last_peered":"2024-02-19T14:52:06.225524+0000","last_clean":"2024-02-19T14:52:06.225524+0000","last_became_active":"2024-02-19T14:52:00.260039+0000","last_became_peered":"2024-02-19T14:52:00.260039+0000","last_unstale":"2024-02-19T14:52:06.225524+0000","last_undegraded":"2024-02-19T14:52:06.225524+0000","last_fullsized":"2024-02-19T14:52:06.225524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:47.726577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246718+0000","last_change":"2024-02-19T14:51:59.267730+0000","last_active":"2024-02-19T14:52:06.246718+0000","last_peered":"2024-02-19T14:52:06.246718+0000","last_clean":"2024-02-19T14:52:06.246718+0000","last_became_active":"2024-02-19T14:51:59.267415+0000","last_became_peered":"2024-02-19T14:51:59.267415+0000","last_unstale":"2024-02-19T14:52:06.246718+0000","last_undegraded":"2024-02-19T14:52:06.246718+0000","last_fullsized":"2024-02-19T14:52:06.246718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226143+0000","last_change":"2024-02-19T14:52:04.433516+0000","last_active":"2024-02-19T14:52:06.226143+0000","last_peered":"2024-02-19T14:52:06.226143+0000","last_clean":"2024-02-19T14:52:06.226143+0000","last_became_active":"2024-02-19T14:52:04.432635+0000","last_became_peered":"2024-02-19T14:52:04.432635+0000","last_unstale":"2024-02-19T14:52:06.226143+0000","last_undegraded":"2024-02-19T14:52:06.226143+0000","last_fullsized":"2024-02-19T14:52:06.226143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334629+0000","last_change":"2024-02-19T14:52:00.266772+0000","last_active":"2024-02-19T14:52:07.334629+0000","last_peered":"2024-02-19T14:52:07.334629+0000","last_clean":"2024-02-19T14:52:07.334629+0000","last_became_active":"2024-02-19T14:52:00.265956+0000","last_became_peered":"2024-02-19T14:52:00.265956+0000","last_unstale":"2024-02-19T14:52:07.334629+0000","last_undegraded":"2024-02-19T14:52:07.334629+0000","last_fullsized":"2024-02-19T14:52:07.334629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282780+0000","last_change":"2024-02-19T14:51:59.252004+0000","last_active":"2024-02-19T14:52:05.282780+0000","last_peered":"2024-02-19T14:52:05.282780+0000","last_clean":"2024-02-19T14:52:05.282780+0000","last_became_active":"2024-02-19T14:51:59.251707+0000","last_became_peered":"2024-02-19T14:51:59.251707+0000","last_unstale":"2024-02-19T14:52:05.282780+0000","last_undegraded":"2024-02-19T14:52:05.282780+0000","last_fullsized":"2024-02-19T14:52:05.282780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246877+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:52:06.246877+0000","last_peered":"2024-02-19T14:52:06.246877+0000","last_clean":"2024-02-19T14:52:06.246877+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:52:06.246877+0000","last_undegraded":"2024-02-19T14:52:06.246877+0000","last_fullsized":"2024-02-19T14:52:06.246877+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530437+0000","last_change":"2024-02-19T14:52:04.436623+0000","last_active":"2024-02-19T14:52:05.530437+0000","last_peered":"2024-02-19T14:52:05.530437+0000","last_clean":"2024-02-19T14:52:05.530437+0000","last_became_active":"2024-02-19T14:52:04.436086+0000","last_became_peered":"2024-02-19T14:52:04.436086+0000","last_unstale":"2024-02-19T14:52:05.530437+0000","last_undegraded":"2024-02-19T14:52:05.530437+0000","last_fullsized":"2024-02-19T14:52:05.530437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247184+0000","last_change":"2024-02-19T14:52:00.267316+0000","last_active":"2024-02-19T14:52:06.247184+0000","last_peered":"2024-02-19T14:52:06.247184+0000","last_clean":"2024-02-19T14:52:06.247184+0000","last_became_active":"2024-02-19T14:52:00.266997+0000","last_became_peered":"2024-02-19T14:52:00.266997+0000","last_unstale":"2024-02-19T14:52:06.247184+0000","last_undegraded":"2024-02-19T14:52:06.247184+0000","last_fullsized":"2024-02-19T14:52:06.247184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281644+0000","last_change":"2024-02-19T14:51:59.253304+0000","last_active":"2024-02-19T14:52:05.281644+0000","last_peered":"2024-02-19T14:52:05.281644+0000","last_clean":"2024-02-19T14:52:05.281644+0000","last_became_active":"2024-02-19T14:51:59.253037+0000","last_became_peered":"2024-02-19T14:51:59.253037+0000","last_unstale":"2024-02-19T14:52:05.281644+0000","last_undegraded":"2024-02-19T14:52:05.281644+0000","last_fullsized":"2024-02-19T14:52:05.281644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530279+0000","last_change":"2024-02-19T14:52:04.433265+0000","last_active":"2024-02-19T14:52:05.530279+0000","last_peered":"2024-02-19T14:52:05.530279+0000","last_clean":"2024-02-19T14:52:05.530279+0000","last_became_active":"2024-02-19T14:52:04.432784+0000","last_became_peered":"2024-02-19T14:52:04.432784+0000","last_unstale":"2024-02-19T14:52:05.530279+0000","last_undegraded":"2024-02-19T14:52:05.530279+0000","last_fullsized":"2024-02-19T14:52:05.530279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283546+0000","last_change":"2024-02-19T14:52:00.266600+0000","last_active":"2024-02-19T14:52:05.283546+0000","last_peered":"2024-02-19T14:52:05.283546+0000","last_clean":"2024-02-19T14:52:05.283546+0000","last_became_active":"2024-02-19T14:52:00.265662+0000","last_became_peered":"2024-02-19T14:52:00.265662+0000","last_unstale":"2024-02-19T14:52:05.283546+0000","last_undegraded":"2024-02-19T14:52:05.283546+0000","last_fullsized":"2024-02-19T14:52:05.283546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229111+0000","last_change":"2024-02-19T14:51:59.258772+0000","last_active":"2024-02-19T14:52:07.229111+0000","last_peered":"2024-02-19T14:52:07.229111+0000","last_clean":"2024-02-19T14:52:07.229111+0000","last_became_active":"2024-02-19T14:51:59.258453+0000","last_became_peered":"2024-02-19T14:51:59.258453+0000","last_unstale":"2024-02-19T14:52:07.229111+0000","last_undegraded":"2024-02-19T14:52:07.229111+0000","last_fullsized":"2024-02-19T14:52:07.229111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225963+0000","last_change":"2024-02-19T14:52:04.281309+0000","last_active":"2024-02-19T14:52:06.225963+0000","last_peered":"2024-02-19T14:52:06.225963+0000","last_clean":"2024-02-19T14:52:06.225963+0000","last_became_active":"2024-02-19T14:52:04.280950+0000","last_became_peered":"2024-02-19T14:52:04.280950+0000","last_unstale":"2024-02-19T14:52:06.225963+0000","last_undegraded":"2024-02-19T14:52:06.225963+0000","last_fullsized":"2024-02-19T14:52:06.225963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247558+0000","last_change":"2024-02-19T14:52:00.260071+0000","last_active":"2024-02-19T14:52:06.247558+0000","last_peered":"2024-02-19T14:52:06.247558+0000","last_clean":"2024-02-19T14:52:06.247558+0000","last_became_active":"2024-02-19T14:52:00.259802+0000","last_became_peered":"2024-02-19T14:52:00.259802+0000","last_unstale":"2024-02-19T14:52:06.247558+0000","last_undegraded":"2024-02-19T14:52:06.247558+0000","last_fullsized":"2024-02-19T14:52:06.247558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:37.874309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281686+0000","last_change":"2024-02-19T14:51:59.252653+0000","last_active":"2024-02-19T14:52:05.281686+0000","last_peered":"2024-02-19T14:52:05.281686+0000","last_clean":"2024-02-19T14:52:05.281686+0000","last_became_active":"2024-02-19T14:51:59.252339+0000","last_became_peered":"2024-02-19T14:51:59.252339+0000","last_unstale":"2024-02-19T14:52:05.281686+0000","last_undegraded":"2024-02-19T14:52:05.281686+0000","last_fullsized":"2024-02-19T14:52:05.281686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282491+0000","last_change":"2024-02-19T14:52:04.435715+0000","last_active":"2024-02-19T14:52:05.282491+0000","last_peered":"2024-02-19T14:52:05.282491+0000","last_clean":"2024-02-19T14:52:05.282491+0000","last_became_active":"2024-02-19T14:52:04.435125+0000","last_became_peered":"2024-02-19T14:52:04.435125+0000","last_unstale":"2024-02-19T14:52:05.282491+0000","last_undegraded":"2024-02-19T14:52:05.282491+0000","last_fullsized":"2024-02-19T14:52:05.282491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247332+0000","last_change":"2024-02-19T14:52:00.262931+0000","last_active":"2024-02-19T14:52:06.247332+0000","last_peered":"2024-02-19T14:52:06.247332+0000","last_clean":"2024-02-19T14:52:06.247332+0000","last_became_active":"2024-02-19T14:52:00.262693+0000","last_became_peered":"2024-02-19T14:52:00.262693+0000","last_unstale":"2024-02-19T14:52:06.247332+0000","last_undegraded":"2024-02-19T14:52:06.247332+0000","last_fullsized":"2024-02-19T14:52:06.247332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283630+0000","last_change":"2024-02-19T14:51:59.260972+0000","last_active":"2024-02-19T14:52:05.283630+0000","last_peered":"2024-02-19T14:52:05.283630+0000","last_clean":"2024-02-19T14:52:05.283630+0000","last_became_active":"2024-02-19T14:51:59.260713+0000","last_became_peered":"2024-02-19T14:51:59.260713+0000","last_unstale":"2024-02-19T14:52:05.283630+0000","last_undegraded":"2024-02-19T14:52:05.283630+0000","last_fullsized":"2024-02-19T14:52:05.283630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283685+0000","last_change":"2024-02-19T14:52:04.277753+0000","last_active":"2024-02-19T14:52:05.283685+0000","last_peered":"2024-02-19T14:52:05.283685+0000","last_clean":"2024-02-19T14:52:05.283685+0000","last_became_active":"2024-02-19T14:52:04.277055+0000","last_became_peered":"2024-02-19T14:52:04.277055+0000","last_unstale":"2024-02-19T14:52:05.283685+0000","last_undegraded":"2024-02-19T14:52:05.283685+0000","last_fullsized":"2024-02-19T14:52:05.283685+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:26.703718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246886+0000","last_change":"2024-02-19T14:52:00.262083+0000","last_active":"2024-02-19T14:52:06.246886+0000","last_peered":"2024-02-19T14:52:06.246886+0000","last_clean":"2024-02-19T14:52:06.246886+0000","last_became_active":"2024-02-19T14:52:00.261797+0000","last_became_peered":"2024-02-19T14:52:00.261797+0000","last_unstale":"2024-02-19T14:52:06.246886+0000","last_undegraded":"2024-02-19T14:52:06.246886+0000","last_fullsized":"2024-02-19T14:52:06.246886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281279+0000","last_change":"2024-02-19T14:51:59.257068+0000","last_active":"2024-02-19T14:52:05.281279+0000","last_peered":"2024-02-19T14:52:05.281279+0000","last_clean":"2024-02-19T14:52:05.281279+0000","last_became_active":"2024-02-19T14:51:59.256775+0000","last_became_peered":"2024-02-19T14:51:59.256775+0000","last_unstale":"2024-02-19T14:52:05.281279+0000","last_undegraded":"2024-02-19T14:52:05.281279+0000","last_fullsized":"2024-02-19T14:52:05.281279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226044+0000","last_change":"2024-02-19T14:52:04.279890+0000","last_active":"2024-02-19T14:52:06.226044+0000","last_peered":"2024-02-19T14:52:06.226044+0000","last_clean":"2024-02-19T14:52:06.226044+0000","last_became_active":"2024-02-19T14:52:04.279485+0000","last_became_peered":"2024-02-19T14:52:04.279485+0000","last_unstale":"2024-02-19T14:52:06.226044+0000","last_undegraded":"2024-02-19T14:52:06.226044+0000","last_fullsized":"2024-02-19T14:52:06.226044+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:52:00.269756+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:52:00.269054+0000","last_became_peered":"2024-02-19T14:52:00.269054+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:51:59.251817+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:51:59.251496+0000","last_became_peered":"2024-02-19T14:51:59.251496+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335425+0000","last_change":"2024-02-19T14:52:04.436129+0000","last_active":"2024-02-19T14:52:07.335425+0000","last_peered":"2024-02-19T14:52:07.335425+0000","last_clean":"2024-02-19T14:52:07.335425+0000","last_became_active":"2024-02-19T14:52:04.435697+0000","last_became_peered":"2024-02-19T14:52:04.435697+0000","last_unstale":"2024-02-19T14:52:07.335425+0000","last_undegraded":"2024-02-19T14:52:07.335425+0000","last_fullsized":"2024-02-19T14:52:07.335425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:07.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247106+0000","last_change":"2024-02-19T14:52:00.263880+0000","last_active":"2024-02-19T14:52:06.247106+0000","last_peered":"2024-02-19T14:52:06.247106+0000","last_clean":"2024-02-19T14:52:06.247106+0000","last_became_active":"2024-02-19T14:52:00.263627+0000","last_became_peered":"2024-02-19T14:52:00.263627+0000","last_unstale":"2024-02-19T14:52:06.247106+0000","last_undegraded":"2024-02-19T14:52:06.247106+0000","last_fullsized":"2024-02-19T14:52:06.247106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283287+0000","last_change":"2024-02-19T14:51:59.252506+0000","last_active":"2024-02-19T14:52:05.283287+0000","last_peered":"2024-02-19T14:52:05.283287+0000","last_clean":"2024-02-19T14:52:05.283287+0000","last_became_active":"2024-02-19T14:51:59.251983+0000","last_became_peered":"2024-02-19T14:51:59.251983+0000","last_unstale":"2024-02-19T14:52:05.283287+0000","last_undegraded":"2024-02-19T14:52:05.283287+0000","last_fullsized":"2024-02-19T14:52:05.283287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335287+0000","last_change":"2024-02-19T14:52:04.433237+0000","last_active":"2024-02-19T14:52:07.335287+0000","last_peered":"2024-02-19T14:52:07.335287+0000","last_clean":"2024-02-19T14:52:07.335287+0000","last_became_active":"2024-02-19T14:52:04.432799+0000","last_became_peered":"2024-02-19T14:52:04.432799+0000","last_unstale":"2024-02-19T14:52:07.335287+0000","last_undegraded":"2024-02-19T14:52:07.335287+0000","last_fullsized":"2024-02-19T14:52:07.335287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:19.099988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225727+0000","last_change":"2024-02-19T14:52:00.267669+0000","last_active":"2024-02-19T14:52:06.225727+0000","last_peered":"2024-02-19T14:52:06.225727+0000","last_clean":"2024-02-19T14:52:06.225727+0000","last_became_active":"2024-02-19T14:52:00.267407+0000","last_became_peered":"2024-02-19T14:52:00.267407+0000","last_unstale":"2024-02-19T14:52:06.225727+0000","last_undegraded":"2024-02-19T14:52:06.225727+0000","last_fullsized":"2024-02-19T14:52:06.225727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226030+0000","last_change":"2024-02-19T14:51:59.261832+0000","last_active":"2024-02-19T14:52:06.226030+0000","last_peered":"2024-02-19T14:52:06.226030+0000","last_clean":"2024-02-19T14:52:06.226030+0000","last_became_active":"2024-02-19T14:51:59.261032+0000","last_became_peered":"2024-02-19T14:51:59.261032+0000","last_unstale":"2024-02-19T14:52:06.226030+0000","last_undegraded":"2024-02-19T14:52:06.226030+0000","last_fullsized":"2024-02-19T14:52:06.226030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284040+0000","last_change":"2024-02-19T14:52:04.433282+0000","last_active":"2024-02-19T14:52:05.284040+0000","last_peered":"2024-02-19T14:52:05.284040+0000","last_clean":"2024-02-19T14:52:05.284040+0000","last_became_active":"2024-02-19T14:52:04.432811+0000","last_became_peered":"2024-02-19T14:52:04.432811+0000","last_unstale":"2024-02-19T14:52:05.284040+0000","last_undegraded":"2024-02-19T14:52:05.284040+0000","last_fullsized":"2024-02-19T14:52:05.284040+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247064+0000","last_change":"2024-02-19T14:52:00.265115+0000","last_active":"2024-02-19T14:52:06.247064+0000","last_peered":"2024-02-19T14:52:06.247064+0000","last_clean":"2024-02-19T14:52:06.247064+0000","last_became_active":"2024-02-19T14:52:00.264838+0000","last_became_peered":"2024-02-19T14:52:00.264838+0000","last_unstale":"2024-02-19T14:52:06.247064+0000","last_undegraded":"2024-02-19T14:52:06.247064+0000","last_fullsized":"2024-02-19T14:52:06.247064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:09.092501+0000","last_change":"2024-02-19T14:51:59.260164+0000","last_active":"2024-02-19T14:52:09.092501+0000","last_peered":"2024-02-19T14:52:09.092501+0000","last_clean":"2024-02-19T14:52:09.092501+0000","last_became_active":"2024-02-19T14:51:59.259921+0000","last_became_peered":"2024-02-19T14:51:59.259921+0000","last_unstale":"2024-02-19T14:52:09.092501+0000","last_undegraded":"2024-02-19T14:52:09.092501+0000","last_fullsized":"2024-02-19T14:52:09.092501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228768+0000","last_change":"2024-02-19T14:52:04.282972+0000","last_active":"2024-02-19T14:52:07.228768+0000","last_peered":"2024-02-19T14:52:07.228768+0000","last_clean":"2024-02-19T14:52:07.228768+0000","last_became_active":"2024-02-19T14:52:04.282593+0000","last_became_peered":"2024-02-19T14:52:04.282593+0000","last_unstale":"2024-02-19T14:52:07.228768+0000","last_undegraded":"2024-02-19T14:52:07.228768+0000","last_fullsized":"2024-02-19T14:52:07.228768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283293+0000","last_change":"2024-02-19T14:52:00.263099+0000","last_active":"2024-02-19T14:52:05.283293+0000","last_peered":"2024-02-19T14:52:05.283293+0000","last_clean":"2024-02-19T14:52:05.283293+0000","last_became_active":"2024-02-19T14:52:00.260826+0000","last_became_peered":"2024-02-19T14:52:00.260826+0000","last_unstale":"2024-02-19T14:52:05.283293+0000","last_undegraded":"2024-02-19T14:52:05.283293+0000","last_fullsized":"2024-02-19T14:52:05.283293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:49.639204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.258165+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.257847+0000","last_became_peered":"2024-02-19T14:51:59.257847+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284313+0000","last_change":"2024-02-19T14:52:04.275097+0000","last_active":"2024-02-19T14:52:05.284313+0000","last_peered":"2024-02-19T14:52:05.284313+0000","last_clean":"2024-02-19T14:52:05.284313+0000","last_became_active":"2024-02-19T14:52:04.274720+0000","last_became_peered":"2024-02-19T14:52:04.274720+0000","last_unstale":"2024-02-19T14:52:05.284313+0000","last_undegraded":"2024-02-19T14:52:05.284313+0000","last_fullsized":"2024-02-19T14:52:05.284313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226316+0000","last_change":"2024-02-19T14:52:00.262791+0000","last_active":"2024-02-19T14:52:06.226316+0000","last_peered":"2024-02-19T14:52:06.226316+0000","last_clean":"2024-02-19T14:52:06.226316+0000","last_became_active":"2024-02-19T14:52:00.262201+0000","last_became_peered":"2024-02-19T14:52:00.262201+0000","last_unstale":"2024-02-19T14:52:06.226316+0000","last_undegraded":"2024-02-19T14:52:06.226316+0000","last_fullsized":"2024-02-19T14:52:06.226316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:23.961482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283591+0000","last_change":"2024-02-19T14:51:59.255762+0000","last_active":"2024-02-19T14:52:05.283591+0000","last_peered":"2024-02-19T14:52:05.283591+0000","last_clean":"2024-02-19T14:52:05.283591+0000","last_became_active":"2024-02-19T14:51:59.255418+0000","last_became_peered":"2024-02-19T14:51:59.255418+0000","last_unstale":"2024-02-19T14:52:05.283591+0000","last_undegraded":"2024-02-19T14:52:05.283591+0000","last_fullsized":"2024-02-19T14:52:05.283591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334347+0000","last_change":"2024-02-19T14:52:04.275183+0000","last_active":"2024-02-19T14:52:07.334347+0000","last_peered":"2024-02-19T14:52:07.334347+0000","last_clean":"2024-02-19T14:52:07.334347+0000","last_became_active":"2024-02-19T14:52:04.274796+0000","last_became_peered":"2024-02-19T14:52:04.274796+0000","last_unstale":"2024-02-19T14:52:07.334347+0000","last_undegraded":"2024-02-19T14:52:07.334347+0000","last_fullsized":"2024-02-19T14:52:07.334347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:17.099296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283483+0000","last_change":"2024-02-19T14:52:00.260440+0000","last_active":"2024-02-19T14:52:05.283483+0000","last_peered":"2024-02-19T14:52:05.283483+0000","last_clean":"2024-02-19T14:52:05.283483+0000","last_became_active":"2024-02-19T14:52:00.259406+0000","last_became_peered":"2024-02-19T14:52:00.259406+0000","last_unstale":"2024-02-19T14:52:05.283483+0000","last_undegraded":"2024-02-19T14:52:05.283483+0000","last_fullsized":"2024-02-19T14:52:05.283483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530584+0000","last_change":"2024-02-19T14:51:59.254054+0000","last_active":"2024-02-19T14:52:05.530584+0000","last_peered":"2024-02-19T14:52:05.530584+0000","last_clean":"2024-02-19T14:52:05.530584+0000","last_became_active":"2024-02-19T14:51:59.253751+0000","last_became_peered":"2024-02-19T14:51:59.253751+0000","last_unstale":"2024-02-19T14:52:05.530584+0000","last_undegraded":"2024-02-19T14:52:05.530584+0000","last_fullsized":"2024-02-19T14:52:05.530584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334305+0000","last_change":"2024-02-19T14:52:04.433602+0000","last_active":"2024-02-19T14:52:07.334305+0000","last_peered":"2024-02-19T14:52:07.334305+0000","last_clean":"2024-02-19T14:52:07.334305+0000","last_became_active":"2024-02-19T14:52:04.433250+0000","last_became_peered":"2024-02-19T14:52:04.433250+0000","last_unstale":"2024-02-19T14:52:07.334305+0000","last_undegraded":"2024-02-19T14:52:07.334305+0000","last_fullsized":"2024-02-19T14:52:07.334305+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226005+0000","last_change":"2024-02-19T14:52:04.435369+0000","last_active":"2024-02-19T14:52:06.226005+0000","last_peered":"2024-02-19T14:52:06.226005+0000","last_clean":"2024-02-19T14:52:06.226005+0000","last_became_active":"2024-02-19T14:52:04.435024+0000","last_became_peered":"2024-02-19T14:52:04.435024+0000","last_unstale":"2024-02-19T14:52:06.226005+0000","last_undegraded":"2024-02-19T14:52:06.226005+0000","last_fullsized":"2024-02-19T14:52:06.226005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:58.843829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228450+0000","last_change":"2024-02-19T14:52:00.266810+0000","last_active":"2024-02-19T14:52:07.228450+0000","last_peered":"2024-02-19T14:52:07.228450+0000","last_clean":"2024-02-19T14:52:07.228450+0000","last_became_active":"2024-02-19T14:52:00.266548+0000","last_became_peered":"2024-02-19T14:52:00.266548+0000","last_unstale":"2024-02-19T14:52:07.228450+0000","last_undegraded":"2024-02-19T14:52:07.228450+0000","last_fullsized":"2024-02-19T14:52:07.228450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282610+0000","last_change":"2024-02-19T14:51:59.248141+0000","last_active":"2024-02-19T14:52:05.282610+0000","last_peered":"2024-02-19T14:52:05.282610+0000","last_clean":"2024-02-19T14:52:05.282610+0000","last_became_active":"2024-02-19T14:51:59.247849+0000","last_became_peered":"2024-02-19T14:51:59.247849+0000","last_unstale":"2024-02-19T14:52:05.282610+0000","last_undegraded":"2024-02-19T14:52:05.282610+0000","last_fullsized":"2024-02-19T14:52:05.282610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530585+0000","last_change":"2024-02-19T14:52:04.282329+0000","last_active":"2024-02-19T14:52:05.530585+0000","last_peered":"2024-02-19T14:52:05.530585+0000","last_clean":"2024-02-19T14:52:05.530585+0000","last_became_active":"2024-02-19T14:52:04.281869+0000","last_became_peered":"2024-02-19T14:52:04.281869+0000","last_unstale":"2024-02-19T14:52:05.530585+0000","last_undegraded":"2024-02-19T14:52:05.530585+0000","last_fullsized":"2024-02-19T14:52:05.530585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334363+0000","last_change":"2024-02-19T14:52:00.266718+0000","last_active":"2024-02-19T14:52:07.334363+0000","last_peered":"2024-02-19T14:52:07.334363+0000","last_clean":"2024-02-19T14:52:07.334363+0000","last_became_active":"2024-02-19T14:52:00.266351+0000","last_became_peered":"2024-02-19T14:52:00.266351+0000","last_unstale":"2024-02-19T14:52:07.334363+0000","last_undegraded":"2024-02-19T14:52:07.334363+0000","last_fullsized":"2024-02-19T14:52:07.334363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333980+0000","last_change":"2024-02-19T14:51:59.259820+0000","last_active":"2024-02-19T14:52:07.333980+0000","last_peered":"2024-02-19T14:52:07.333980+0000","last_clean":"2024-02-19T14:52:07.333980+0000","last_became_active":"2024-02-19T14:51:59.259418+0000","last_became_peered":"2024-02-19T14:51:59.259418+0000","last_unstale":"2024-02-19T14:52:07.333980+0000","last_undegraded":"2024-02-19T14:52:07.333980+0000","last_fullsized":"2024-02-19T14:52:07.333980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530291+0000","last_change":"2024-02-19T14:51:59.252686+0000","last_active":"2024-02-19T14:52:05.530291+0000","last_peered":"2024-02-19T14:52:05.530291+0000","last_clean":"2024-02-19T14:52:05.530291+0000","last_became_active":"2024-02-19T14:51:59.252420+0000","last_became_peered":"2024-02-19T14:51:59.252420+0000","last_unstale":"2024-02-19T14:52:05.530291+0000","last_undegraded":"2024-02-19T14:52:05.530291+0000","last_fullsized":"2024-02-19T14:52:05.530291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226023+0000","last_change":"2024-02-19T14:52:00.264522+0000","last_active":"2024-02-19T14:52:06.226023+0000","last_peered":"2024-02-19T14:52:06.226023+0000","last_clean":"2024-02-19T14:52:06.226023+0000","last_became_active":"2024-02-19T14:52:00.264273+0000","last_became_peered":"2024-02-19T14:52:00.264273+0000","last_unstale":"2024-02-19T14:52:06.226023+0000","last_undegraded":"2024-02-19T14:52:06.226023+0000","last_fullsized":"2024-02-19T14:52:06.226023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281616+0000","last_change":"2024-02-19T14:52:04.275545+0000","last_active":"2024-02-19T14:52:05.281616+0000","last_peered":"2024-02-19T14:52:05.281616+0000","last_clean":"2024-02-19T14:52:05.281616+0000","last_became_active":"2024-02-19T14:52:04.275140+0000","last_became_peered":"2024-02-19T14:52:04.275140+0000","last_unstale":"2024-02-19T14:52:05.281616+0000","last_undegraded":"2024-02-19T14:52:05.281616+0000","last_fullsized":"2024-02-19T14:52:05.281616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:27.987737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333998+0000","last_change":"2024-02-19T14:51:59.262930+0000","last_active":"2024-02-19T14:52:07.333998+0000","last_peered":"2024-02-19T14:52:07.333998+0000","last_clean":"2024-02-19T14:52:07.333998+0000","last_became_active":"2024-02-19T14:51:59.262635+0000","last_became_peered":"2024-02-19T14:51:59.262635+0000","last_unstale":"2024-02-19T14:52:07.333998+0000","last_undegraded":"2024-02-19T14:52:07.333998+0000","last_fullsized":"2024-02-19T14:52:07.333998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:21.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284375+0000","last_change":"2024-02-19T14:52:00.260213+0000","last_active":"2024-02-19T14:52:05.284375+0000","last_peered":"2024-02-19T14:52:05.284375+0000","last_clean":"2024-02-19T14:52:05.284375+0000","last_became_active":"2024-02-19T14:52:00.257148+0000","last_became_peered":"2024-02-19T14:52:00.257148+0000","last_unstale":"2024-02-19T14:52:05.284375+0000","last_undegraded":"2024-02-19T14:52:05.284375+0000","last_fullsized":"2024-02-19T14:52:05.284375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:56.918797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335231+0000","last_change":"2024-02-19T14:52:04.276033+0000","last_active":"2024-02-19T14:52:07.335231+0000","last_peered":"2024-02-19T14:52:07.335231+0000","last_clean":"2024-02-19T14:52:07.335231+0000","last_became_active":"2024-02-19T14:52:04.275671+0000","last_became_peered":"2024-02-19T14:52:04.275671+0000","last_unstale":"2024-02-19T14:52:07.335231+0000","last_undegraded":"2024-02-19T14:52:07.335231+0000","last_fullsized":"2024-02-19T14:52:07.335231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:42.880381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.529955+0000","last_change":"2024-02-19T14:52:00.270697+0000","last_active":"2024-02-19T14:52:05.529955+0000","last_peered":"2024-02-19T14:52:05.529955+0000","last_clean":"2024-02-19T14:52:05.529955+0000","last_became_active":"2024-02-19T14:52:00.269449+0000","last_became_peered":"2024-02-19T14:52:00.269449+0000","last_unstale":"2024-02-19T14:52:05.529955+0000","last_undegraded":"2024-02-19T14:52:05.529955+0000","last_fullsized":"2024-02-19T14:52:05.529955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281315+0000","last_change":"2024-02-19T14:52:00.263056+0000","last_active":"2024-02-19T14:52:05.281315+0000","last_peered":"2024-02-19T14:52:05.281315+0000","last_clean":"2024-02-19T14:52:05.281315+0000","last_became_active":"2024-02-19T14:52:00.262801+0000","last_became_peered":"2024-02-19T14:52:00.262801+0000","last_unstale":"2024-02-19T14:52:05.281315+0000","last_undegraded":"2024-02-19T14:52:05.281315+0000","last_fullsized":"2024-02-19T14:52:05.281315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283161+0000","last_change":"2024-02-19T14:52:00.270389+0000","last_active":"2024-02-19T14:52:05.283161+0000","last_peered":"2024-02-19T14:52:05.283161+0000","last_clean":"2024-02-19T14:52:05.283161+0000","last_became_active":"2024-02-19T14:52:00.267246+0000","last_became_peered":"2024-02-19T14:52:00.267246+0000","last_unstale":"2024-02-19T14:52:05.283161+0000","last_undegraded":"2024-02-19T14:52:05.283161+0000","last_fullsized":"2024-02-19T14:52:05.283161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225914+0000","last_change":"2024-02-19T14:52:00.265624+0000","last_active":"2024-02-19T14:52:06.225914+0000","last_peered":"2024-02-19T14:52:06.225914+0000","last_clean":"2024-02-19T14:52:06.225914+0000","last_became_active":"2024-02-19T14:52:00.265377+0000","last_became_peered":"2024-02-19T14:52:00.265377+0000","last_unstale":"2024-02-19T14:52:06.225914+0000","last_undegraded":"2024-02-19T14:52:06.225914+0000","last_fullsized":"2024-02-19T14:52:06.225914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335142+0000","last_change":"2024-02-19T14:52:00.263043+0000","last_active":"2024-02-19T14:52:07.335142+0000","last_peered":"2024-02-19T14:52:07.335142+0000","last_clean":"2024-02-19T14:52:07.335142+0000","last_became_active":"2024-02-19T14:52:00.262754+0000","last_became_peered":"2024-02-19T14:52:00.262754+0000","last_unstale":"2024-02-19T14:52:07.335142+0000","last_undegraded":"2024-02-19T14:52:07.335142+0000","last_fullsized":"2024-02-19T14:52:07.335142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228927+0000","last_change":"2024-02-19T14:52:00.266890+0000","last_active":"2024-02-19T14:52:07.228927+0000","last_peered":"2024-02-19T14:52:07.228927+0000","last_clean":"2024-02-19T14:52:07.228927+0000","last_became_active":"2024-02-19T14:52:00.266574+0000","last_became_peered":"2024-02-19T14:52:00.266574+0000","last_unstale":"2024-02-19T14:52:07.228927+0000","last_undegraded":"2024-02-19T14:52:07.228927+0000","last_fullsized":"2024-02-19T14:52:07.228927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530706+0000","last_change":"2024-02-19T14:52:00.261225+0000","last_active":"2024-02-19T14:52:05.530706+0000","last_peered":"2024-02-19T14:52:05.530706+0000","last_clean":"2024-02-19T14:52:05.530706+0000","last_became_active":"2024-02-19T14:52:00.259666+0000","last_became_peered":"2024-02-19T14:52:00.259666+0000","last_unstale":"2024-02-19T14:52:05.530706+0000","last_undegraded":"2024-02-19T14:52:05.530706+0000","last_fullsized":"2024-02-19T14:52:05.530706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335354+0000","last_change":"2024-02-19T14:52:00.264377+0000","last_active":"2024-02-19T14:52:07.335354+0000","last_peered":"2024-02-19T14:52:07.335354+0000","last_clean":"2024-02-19T14:52:07.335354+0000","last_became_active":"2024-02-19T14:52:00.264082+0000","last_became_peered":"2024-02-19T14:52:00.264082+0000","last_unstale":"2024-02-19T14:52:07.335354+0000","last_undegraded":"2024-02-19T14:52:07.335354+0000","last_fullsized":"2024-02-19T14:52:07.335354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282561+0000","last_change":"2024-02-19T14:52:00.265423+0000","last_active":"2024-02-19T14:52:05.282561+0000","last_peered":"2024-02-19T14:52:05.282561+0000","last_clean":"2024-02-19T14:52:05.282561+0000","last_became_active":"2024-02-19T14:52:00.265132+0000","last_became_peered":"2024-02-19T14:52:00.265132+0000","last_unstale":"2024-02-19T14:52:05.282561+0000","last_undegraded":"2024-02-19T14:52:05.282561+0000","last_fullsized":"2024-02-19T14:52:05.282561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334543+0000","last_change":"2024-02-19T14:52:00.259058+0000","last_active":"2024-02-19T14:52:07.334543+0000","last_peered":"2024-02-19T14:52:07.334543+0000","last_clean":"2024-02-19T14:52:07.334543+0000","last_became_active":"2024-02-19T14:52:00.258228+0000","last_became_peered":"2024-02-19T14:52:00.258228+0000","last_unstale":"2024-02-19T14:52:07.334543+0000","last_undegraded":"2024-02-19T14:52:07.334543+0000","last_fullsized":"2024-02-19T14:52:07.334543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283058+0000","last_change":"2024-02-19T14:52:00.258261+0000","last_active":"2024-02-19T14:52:05.283058+0000","last_peered":"2024-02-19T14:52:05.283058+0000","last_clean":"2024-02-19T14:52:05.283058+0000","last_became_active":"2024-02-19T14:52:00.257970+0000","last_became_peered":"2024-02-19T14:52:00.257970+0000","last_unstale":"2024-02-19T14:52:05.283058+0000","last_undegraded":"2024-02-19T14:52:05.283058+0000","last_fullsized":"2024-02-19T14:52:05.283058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225756+0000","last_change":"2024-02-19T14:52:00.268547+0000","last_active":"2024-02-19T14:52:06.225756+0000","last_peered":"2024-02-19T14:52:06.225756+0000","last_clean":"2024-02-19T14:52:06.225756+0000","last_became_active":"2024-02-19T14:52:00.268241+0000","last_became_peered":"2024-02-19T14:52:00.268241+0000","last_unstale":"2024-02-19T14:52:06.225756+0000","last_undegraded":"2024-02-19T14:52:06.225756+0000","last_fullsized":"2024-02-19T14:52:06.225756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530412+0000","last_change":"2024-02-19T14:52:00.262779+0000","last_active":"2024-02-19T14:52:05.530412+0000","last_peered":"2024-02-19T14:52:05.530412+0000","last_clean":"2024-02-19T14:52:05.530412+0000","last_became_active":"2024-02-19T14:52:00.262476+0000","last_became_peered":"2024-02-19T14:52:00.262476+0000","last_unstale":"2024-02-19T14:52:05.530412+0000","last_undegraded":"2024-02-19T14:52:05.530412+0000","last_fullsized":"2024-02-19T14:52:05.530412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247311+0000","last_change":"2024-02-19T14:52:00.272996+0000","last_active":"2024-02-19T14:52:06.247311+0000","last_peered":"2024-02-19T14:52:06.247311+0000","last_clean":"2024-02-19T14:52:06.247311+0000","last_became_active":"2024-02-19T14:52:00.271013+0000","last_became_peered":"2024-02-19T14:52:00.271013+0000","last_unstale":"2024-02-19T14:52:06.247311+0000","last_undegraded":"2024-02-19T14:52:06.247311+0000","last_fullsized":"2024-02-19T14:52:06.247311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281593+0000","last_change":"2024-02-19T14:52:00.263763+0000","last_active":"2024-02-19T14:52:05.281593+0000","last_peered":"2024-02-19T14:52:05.281593+0000","last_clean":"2024-02-19T14:52:05.281593+0000","last_became_active":"2024-02-19T14:52:00.263488+0000","last_became_peered":"2024-02-19T14:52:00.263488+0000","last_unstale":"2024-02-19T14:52:05.281593+0000","last_undegraded":"2024-02-19T14:52:05.281593+0000","last_fullsized":"2024-02-19T14:52:05.281593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:46.599373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284503+0000","last_change":"2024-02-19T14:52:00.258879+0000","last_active":"2024-02-19T14:52:05.284503+0000","last_peered":"2024-02-19T14:52:05.284503+0000","last_clean":"2024-02-19T14:52:05.284503+0000","last_became_active":"2024-02-19T14:52:00.257612+0000","last_became_peered":"2024-02-19T14:52:00.257612+0000","last_unstale":"2024-02-19T14:52:05.284503+0000","last_undegraded":"2024-02-19T14:52:05.284503+0000","last_fullsized":"2024-02-19T14:52:05.284503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282447+0000","last_change":"2024-02-19T14:52:00.268671+0000","last_active":"2024-02-19T14:52:05.282447+0000","last_peered":"2024-02-19T14:52:05.282447+0000","last_clean":"2024-02-19T14:52:05.282447+0000","last_became_active":"2024-02-19T14:52:00.268351+0000","last_became_peered":"2024-02-19T14:52:00.268351+0000","last_unstale":"2024-02-19T14:52:05.282447+0000","last_undegraded":"2024-02-19T14:52:05.282447+0000","last_fullsized":"2024-02-19T14:52:05.282447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228525+0000","last_change":"2024-02-19T14:52:00.249695+0000","last_active":"2024-02-19T14:52:07.228525+0000","last_peered":"2024-02-19T14:52:07.228525+0000","last_clean":"2024-02-19T14:52:07.228525+0000","last_became_active":"2024-02-19T14:52:00.249406+0000","last_became_peered":"2024-02-19T14:52:00.249406+0000","last_unstale":"2024-02-19T14:52:07.228525+0000","last_undegraded":"2024-02-19T14:52:07.228525+0000","last_fullsized":"2024-02-19T14:52:07.228525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225669+0000","last_change":"2024-02-19T14:52:00.259566+0000","last_active":"2024-02-19T14:52:06.225669+0000","last_peered":"2024-02-19T14:52:06.225669+0000","last_clean":"2024-02-19T14:52:06.225669+0000","last_became_active":"2024-02-19T14:52:00.259290+0000","last_became_peered":"2024-02-19T14:52:00.259290+0000","last_unstale":"2024-02-19T14:52:06.225669+0000","last_undegraded":"2024-02-19T14:52:06.225669+0000","last_fullsized":"2024-02-19T14:52:06.225669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334965+0000","last_change":"2024-02-19T14:52:00.267359+0000","last_active":"2024-02-19T14:52:07.334965+0000","last_peered":"2024-02-19T14:52:07.334965+0000","last_clean":"2024-02-19T14:52:07.334965+0000","last_became_active":"2024-02-19T14:52:00.266119+0000","last_became_peered":"2024-02-19T14:52:00.266119+0000","last_unstale":"2024-02-19T14:52:07.334965+0000","last_undegraded":"2024-02-19T14:52:07.334965+0000","last_fullsized":"2024-02-19T14:52:07.334965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283181+0000","last_change":"2024-02-19T14:52:00.262531+0000","last_active":"2024-02-19T14:52:05.283181+0000","last_peered":"2024-02-19T14:52:05.283181+0000","last_clean":"2024-02-19T14:52:05.283181+0000","last_became_active":"2024-02-19T14:52:00.262240+0000","last_became_peered":"2024-02-19T14:52:00.262240+0000","last_unstale":"2024-02-19T14:52:05.283181+0000","last_undegraded":"2024-02-19T14:52:05.283181+0000","last_fullsized":"2024-02-19T14:52:05.283181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:41.341079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530384+0000","last_change":"2024-02-19T14:52:00.266149+0000","last_active":"2024-02-19T14:52:05.530384+0000","last_peered":"2024-02-19T14:52:05.530384+0000","last_clean":"2024-02-19T14:52:05.530384+0000","last_became_active":"2024-02-19T14:52:00.265873+0000","last_became_peered":"2024-02-19T14:52:00.265873+0000","last_unstale":"2024-02-19T14:52:05.530384+0000","last_undegraded":"2024-02-19T14:52:05.530384+0000","last_fullsized":"2024-02-19T14:52:05.530384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:52:00.266144+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:52:00.265852+0000","last_became_peered":"2024-02-19T14:52:00.265852+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247502+0000","last_change":"2024-02-19T14:52:00.265908+0000","last_active":"2024-02-19T14:52:06.247502+0000","last_peered":"2024-02-19T14:52:06.247502+0000","last_clean":"2024-02-19T14:52:06.247502+0000","last_became_active":"2024-02-19T14:52:00.265630+0000","last_became_peered":"2024-02-19T14:52:00.265630+0000","last_unstale":"2024-02-19T14:52:06.247502+0000","last_undegraded":"2024-02-19T14:52:06.247502+0000","last_fullsized":"2024-02-19T14:52:06.247502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:06.056284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334181+0000","last_change":"2024-02-19T14:52:00.267195+0000","last_active":"2024-02-19T14:52:07.334181+0000","last_peered":"2024-02-19T14:52:07.334181+0000","last_clean":"2024-02-19T14:52:07.334181+0000","last_became_active":"2024-02-19T14:52:00.266185+0000","last_became_peered":"2024-02-19T14:52:00.266185+0000","last_unstale":"2024-02-19T14:52:07.334181+0000","last_undegraded":"2024-02-19T14:52:07.334181+0000","last_fullsized":"2024-02-19T14:52:07.334181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246858+0000","last_change":"2024-02-19T14:52:00.266943+0000","last_active":"2024-02-19T14:52:06.246858+0000","last_peered":"2024-02-19T14:52:06.246858+0000","last_clean":"2024-02-19T14:52:06.246858+0000","last_became_active":"2024-02-19T14:52:00.266665+0000","last_became_peered":"2024-02-19T14:52:00.266665+0000","last_unstale":"2024-02-19T14:52:06.246858+0000","last_undegraded":"2024-02-19T14:52:06.246858+0000","last_fullsized":"2024-02-19T14:52:06.246858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282952+0000","last_change":"2024-02-19T14:52:00.261505+0000","last_active":"2024-02-19T14:52:05.282952+0000","last_peered":"2024-02-19T14:52:05.282952+0000","last_clean":"2024-02-19T14:52:05.282952+0000","last_became_active":"2024-02-19T14:52:00.261212+0000","last_became_peered":"2024-02-19T14:52:00.261212+0000","last_unstale":"2024-02-19T14:52:05.282952+0000","last_undegraded":"2024-02-19T14:52:05.282952+0000","last_fullsized":"2024-02-19T14:52:05.282952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246984+0000","last_change":"2024-02-19T14:52:00.261481+0000","last_active":"2024-02-19T14:52:06.246984+0000","last_peered":"2024-02-19T14:52:06.246984+0000","last_clean":"2024-02-19T14:52:06.246984+0000","last_became_active":"2024-02-19T14:52:00.261205+0000","last_became_peered":"2024-02-19T14:52:00.261205+0000","last_unstale":"2024-02-19T14:52:06.246984+0000","last_undegraded":"2024-02-19T14:52:06.246984+0000","last_fullsized":"2024-02-19T14:52:06.246984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283153+0000","last_change":"2024-02-19T14:52:00.266814+0000","last_active":"2024-02-19T14:52:05.283153+0000","last_peered":"2024-02-19T14:52:05.283153+0000","last_clean":"2024-02-19T14:52:05.283153+0000","last_became_active":"2024-02-19T14:52:00.266019+0000","last_became_peered":"2024-02-19T14:52:00.266019+0000","last_unstale":"2024-02-19T14:52:05.283153+0000","last_undegraded":"2024-02-19T14:52:05.283153+0000","last_fullsized":"2024-02-19T14:52:05.283153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225905+0000","last_change":"2024-02-19T14:52:00.260863+0000","last_active":"2024-02-19T14:52:06.225905+0000","last_peered":"2024-02-19T14:52:06.225905+0000","last_clean":"2024-02-19T14:52:06.225905+0000","last_became_active":"2024-02-19T14:52:00.260543+0000","last_became_peered":"2024-02-19T14:52:00.260543+0000","last_unstale":"2024-02-19T14:52:06.225905+0000","last_undegraded":"2024-02-19T14:52:06.225905+0000","last_fullsized":"2024-02-19T14:52:06.225905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228999+0000","last_change":"2024-02-19T14:52:00.260306+0000","last_active":"2024-02-19T14:52:07.228999+0000","last_peered":"2024-02-19T14:52:07.228999+0000","last_clean":"2024-02-19T14:52:07.228999+0000","last_became_active":"2024-02-19T14:52:00.260026+0000","last_became_peered":"2024-02-19T14:52:00.260026+0000","last_unstale":"2024-02-19T14:52:07.228999+0000","last_undegraded":"2024-02-19T14:52:07.228999+0000","last_fullsized":"2024-02-19T14:52:07.228999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247442+0000","last_change":"2024-02-19T14:52:00.263320+0000","last_active":"2024-02-19T14:52:06.247442+0000","last_peered":"2024-02-19T14:52:06.247442+0000","last_clean":"2024-02-19T14:52:06.247442+0000","last_became_active":"2024-02-19T14:52:00.263044+0000","last_became_peered":"2024-02-19T14:52:00.263044+0000","last_unstale":"2024-02-19T14:52:06.247442+0000","last_undegraded":"2024-02-19T14:52:06.247442+0000","last_fullsized":"2024-02-19T14:52:06.247442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:08.747083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:52:17.659 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:52:17.660 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-19T14:52:17.660 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-02-19T14:52:17.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-02-19T14:52:17.660 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-19T14:52:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.791+0000 7f6abe33a640 1 Processor -- start 2024-02-19T14:52:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.791+0000 7f6abe33a640 1 -- start start 2024-02-19T14:52:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6abe33a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 0x7f6ab80c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6abe33a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ab80c5270 0x7f6ab80719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6abe33a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab80c1120 con 0x7f6ab8071f10 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6abe33a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab80c39d0 con 0x7f6ab80c5270 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6abe33a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80c6080 con 0x7f6ab814ece0 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 0x7f6ab80c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ab80c5270 0x7f6ab80719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 0x7f6ab80c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50794/0 (socket says 172.21.15.78:50794) 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 -- 172.21.15.78:0/3709258333 learned_addr learned my addr 172.21.15.78:0/3709258333 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 -- 172.21.15.78:0/3709258333 >> v2:172.21.15.150:3300/0 conn(0x7f6ab80c5270 msgr2=0x7f6ab80719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 --2- 172.21.15.78:0/3709258333 >> v2:172.21.15.150:3300/0 conn(0x7f6ab80c5270 0x7f6ab80719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 -- 172.21.15.78:0/3709258333 >> v1:172.21.15.150:6789/0 conn(0x7f6ab8071f10 legacy=0x7f6ab80c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab7fff640 1 -- 172.21.15.78:0/3709258333 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80c57e0 con 0x7f6ab814ece0 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.792+0000 7f6ab77fe640 1 --2- 172.21.15.78:0/3709258333 >> v2:172.21.15.150:3300/0 conn(0x7f6ab80c5270 0x7f6ab80719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.793+0000 7f6ab7fff640 1 --2- 172.21.15.78:0/3709258333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 0x7f6ab80c4d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009870 tx=0x7f6aa802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c18ea654725c674f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.793+0000 7f6ab6ffd640 1 -- 172.21.15.78:0/3709258333 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8030840 con 0x7f6ab814ece0 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.793+0000 7f6ab6ffd640 1 -- 172.21.15.78:0/3709258333 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa80309a0 con 0x7f6ab814ece0 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.793+0000 7f6ab6ffd640 1 -- 172.21.15.78:0/3709258333 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8030d30 con 0x7f6ab814ece0 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 -- 172.21.15.78:0/3709258333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 msgr2=0x7f6ab80c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 --2- 172.21.15.78:0/3709258333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 0x7f6ab80c4d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009870 tx=0x7f6aa802b980 comp rx=0 tx=0).stop 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 -- 172.21.15.78:0/3709258333 shutdown_connections 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 --2- 172.21.15.78:0/3709258333 >> v2:172.21.15.150:3300/0 conn(0x7f6ab80c5270 0x7f6ab80719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 --2- 172.21.15.78:0/3709258333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab814ece0 0x7f6ab80c4d30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 -- 172.21.15.78:0/3709258333 >> 172.21.15.78:0/3709258333 conn(0x7f6ab806b270 msgr2=0x7f6ab806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 -- 172.21.15.78:0/3709258333 shutdown_connections 2024-02-19T14:52:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.794+0000 7f6abe33a640 1 -- 172.21.15.78:0/3709258333 wait complete. 2024-02-19T14:52:17.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.795+0000 7f6abe33a640 1 Processor -- start 2024-02-19T14:52:17.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.795+0000 7f6abe33a640 1 -- start start 2024-02-19T14:52:17.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abe33a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab8071f10 0x7f6ab8154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abe33a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab80c5270 0x7f6ab81e8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abe33a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 0x7f6ab81e8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abe33a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81eb610 con 0x7f6ab814ece0 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abe33a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81e8a80 con 0x7f6ab80c5270 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abe33a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81eb8b0 con 0x7f6ab8071f10 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 0x7f6ab81e8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 0x7f6ab81e8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52284/0 (socket says 172.21.15.78:52284) 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 -- 172.21.15.78:0/1134281781 learned_addr learned my addr 172.21.15.78:0/1134281781 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6ab7fff640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab8071f10 0x7f6ab8154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6ab77fe640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab80c5270 0x7f6ab81e8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 -- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab8071f10 msgr2=0x7f6ab8154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab8071f10 0x7f6ab8154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 -- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab80c5270 msgr2=0x7f6ab81e8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab80c5270 0x7f6ab81e8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 -- 172.21.15.78:0/1134281781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac009400 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.796+0000 7f6abc8d7640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 0x7f6ab81e8540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000c7a0 tx=0x7f6ab000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.797+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0003b70 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.797+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab0003cf0 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.797+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa80094d0 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.797+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0003ea0 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.797+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab81ebe70 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.798+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8074990 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.798+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6ab000aa90 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.799+0000 7f6ab4ff9640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6aa406b810 0x7f6aa406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.799+0000 7f6ab7fff640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6aa406b810 0x7f6aa406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.799+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6ab008a170 con 0x7f6ab814ece0 2024-02-19T14:52:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.799+0000 7f6ab7fff640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6aa406b810 0x7f6aa406dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8036000 tx=0x7f6aa8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:17.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:17.803+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ab0054990 con 0x7f6ab814ece0 2024-02-19T14:52:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.027+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6ab80c49f0 con 0x7f6ab814ece0 2024-02-19T14:52:18.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.031+0000 7f6ab4ff9640 1 -- 172.21.15.78:0/1134281781 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171791 (secure 0 0 0) 0x7f6ab0058050 con 0x7f6ab814ece0 2024-02-19T14:52:18.036 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6aa406b810 msgr2=0x7f6aa406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6aa406b810 0x7f6aa406dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8036000 tx=0x7f6aa8008030 comp rx=0 tx=0).stop 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 msgr2=0x7f6ab81e8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 0x7f6ab81e8540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000c7a0 tx=0x7f6ab000cc60 comp rx=0 tx=0).stop 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 shutdown_connections 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6aa406b810 0x7f6aa406dcd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab814ece0 0x7f6ab81e8540 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab80c5270 0x7f6ab81e8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 --2- 172.21.15.78:0/1134281781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab8071f10 0x7f6ab8154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 >> 172.21.15.78:0/1134281781 conn(0x7f6ab806b270 msgr2=0x7f6ab80c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 shutdown_connections 2024-02-19T14:52:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.033+0000 7f6abe33a640 1 -- 172.21.15.78:0/1134281781 wait complete. 2024-02-19T14:52:18.048 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":359840476},{"type":"v1","addr":"172.21.15.78:6833","nonce":359840476}]},"active_addr":"172.21.15.78:6833/359840476","active_change":"2024-02-19T14:51:40.052666+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1050653338}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2295346394}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":880500493}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1829408796}]}]} 2024-02-19T14:52:18.051 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-02-19T14:52:18.051 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-19T14:52:18.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.170+0000 7f8d1fe9a640 1 Processor -- start 2024-02-19T14:52:18.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.170+0000 7f8d1fe9a640 1 -- start start 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1fe9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d18152480 0x7f8d181559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1fe9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 0x7f8d1806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1fe9a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d18155ef0 con 0x7f8d1806d350 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1fe9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1806dec0 con 0x7f8d18152480 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1fe9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180c75c0 con 0x7f8d180c7070 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1d435640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 0x7f8d1806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1dc36640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d18152480 0x7f8d181559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1d435640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 0x7f8d1806ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50804/0 (socket says 172.21.15.78:50804) 2024-02-19T14:52:18.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.171+0000 7f8d1d435640 1 -- 172.21.15.78:0/1611702326 learned_addr learned my addr 172.21.15.78:0/1611702326 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.172+0000 7f8d1d435640 1 -- 172.21.15.78:0/1611702326 >> v2:172.21.15.150:3300/0 conn(0x7f8d18152480 msgr2=0x7f8d181559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.172+0000 7f8d1d435640 1 --2- 172.21.15.78:0/1611702326 >> v2:172.21.15.150:3300/0 conn(0x7f8d18152480 0x7f8d181559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.172+0000 7f8d1d435640 1 -- 172.21.15.78:0/1611702326 >> v1:172.21.15.150:6789/0 conn(0x7f8d1806d350 legacy=0x7f8d180c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.172+0000 7f8d1d435640 1 -- 172.21.15.78:0/1611702326 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180c78a0 con 0x7f8d180c7070 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.172+0000 7f8d1d435640 1 --2- 172.21.15.78:0/1611702326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 0x7f8d1806ce10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d080098a0 tx=0x7f8d0802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440b8f6d95be0c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1cc34640 1 -- 172.21.15.78:0/1611702326 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08030840 con 0x7f8d180c7070 2024-02-19T14:52:18.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1cc34640 1 -- 172.21.15.78:0/1611702326 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d080309a0 con 0x7f8d180c7070 2024-02-19T14:52:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1cc34640 1 -- 172.21.15.78:0/1611702326 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08030d30 con 0x7f8d180c7070 2024-02-19T14:52:18.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1611702326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 msgr2=0x7f8d1806ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1611702326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 0x7f8d1806ce10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8d080098a0 tx=0x7f8d0802b980 comp rx=0 tx=0).stop 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1611702326 shutdown_connections 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1611702326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d180c7070 0x7f8d1806ce10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1611702326 >> v2:172.21.15.150:3300/0 conn(0x7f8d18152480 0x7f8d181559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1611702326 >> 172.21.15.78:0/1611702326 conn(0x7f8d180ca8a0 msgr2=0x7f8d180cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1611702326 shutdown_connections 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.173+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1611702326 wait complete. 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.174+0000 7f8d1fe9a640 1 Processor -- start 2024-02-19T14:52:18.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.174+0000 7f8d1fe9a640 1 -- start start 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1fe9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 0x7f8d181ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1fe9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d18152480 0x7f8d180ac430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1fe9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d18146e90 0x7f8d180ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1fe9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180adca0 con 0x7f8d180c7070 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1fe9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1814d430 con 0x7f8d18152480 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1fe9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d181eec60 con 0x7f8d18146e90 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1dc36640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 0x7f8d181ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1dc36640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 0x7f8d181ee1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52296/0 (socket says 172.21.15.78:52296) 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1dc36640 1 -- 172.21.15.78:0/1736248772 learned_addr learned my addr 172.21.15.78:0/1736248772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1d435640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d18152480 0x7f8d180ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.175+0000 7f8d1e437640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d18146e90 0x7f8d180ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1dc36640 1 -- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d18146e90 msgr2=0x7f8d180ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1dc36640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d18146e90 0x7f8d180ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1dc36640 1 -- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d18152480 msgr2=0x7f8d180ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1dc36640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d18152480 0x7f8d180ac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1dc36640 1 -- 172.21.15.78:0/1736248772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d080094d0 con 0x7f8d180c7070 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1dc36640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 0x7f8d181ee1e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009d10 tx=0x7f8d1000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10009b80 con 0x7f8d180c7070 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1000f040 con 0x7f8d180c7070 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10014680 con 0x7f8d180c7070 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d180ad930 con 0x7f8d180c7070 2024-02-19T14:52:18.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.176+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d180aceb0 con 0x7f8d180c7070 2024-02-19T14:52:18.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.177+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d180ad100 con 0x7f8d180c7070 2024-02-19T14:52:18.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.177+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8d100147e0 con 0x7f8d180c7070 2024-02-19T14:52:18.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.178+0000 7f8d0e7fc640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cfc06b810 0x7f8cfc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.178+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8d1008ae30 con 0x7f8d180c7070 2024-02-19T14:52:18.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.178+0000 7f8d1d435640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cfc06b810 0x7f8cfc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.178+0000 7f8d1d435640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cfc06b810 0x7f8cfc06dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8d08036000 tx=0x7f8d08008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.182+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8d10055650 con 0x7f8d180c7070 2024-02-19T14:52:18.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.404+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d181eee80 con 0x7f8d180c7070 2024-02-19T14:52:18.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.405+0000 7f8d0e7fc640 1 -- 172.21.15.78:0/1736248772 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f8d10058d10 con 0x7f8d180c7070 2024-02-19T14:52:18.408 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:18.409 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cfc06b810 msgr2=0x7f8cfc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cfc06b810 0x7f8cfc06dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8d08036000 tx=0x7f8d08008030 comp rx=0 tx=0).stop 2024-02-19T14:52:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 msgr2=0x7f8d181ee1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 0x7f8d181ee1e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009d10 tx=0x7f8d1000c490 comp rx=0 tx=0).stop 2024-02-19T14:52:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 shutdown_connections 2024-02-19T14:52:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cfc06b810 0x7f8cfc06dcd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d18146e90 0x7f8d180ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d18152480 0x7f8d180ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 --2- 172.21.15.78:0/1736248772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d180c7070 0x7f8d181ee1e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 >> 172.21.15.78:0/1736248772 conn(0x7f8d180ca8a0 msgr2=0x7f8d18152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 shutdown_connections 2024-02-19T14:52:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.407+0000 7f8d1fe9a640 1 -- 172.21.15.78:0/1736248772 wait complete. 2024-02-19T14:52:18.422 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-02-19T14:52:18.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:18.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.519+0000 7f0d25977640 1 Processor -- start 2024-02-19T14:52:18.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.519+0000 7f0d25977640 1 -- start start 2024-02-19T14:52:18.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d25977640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d200c8e00 0x7f0d2006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d25977640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d2006d910 0x7f0d2006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d25977640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2006cd20 con 0x7f0d20152700 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d25977640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d201551f0 con 0x7f0d2006d910 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d25977640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d20155490 con 0x7f0d200c8e00 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d2006d910 0x7f0d2006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0d20152700 0x7f0d200c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57672/0 (socket says 172.21.15.78:57672) 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1effd640 1 -- 172.21.15.78:0/849861992 learned_addr learned my addr 172.21.15.78:0/849861992 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1e7fc640 1 --2- 172.21.15.78:0/849861992 >> v2:172.21.15.150:3300/0 conn(0x7f0d200c8e00 0x7f0d2006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1f7fe640 1 -- 172.21.15.78:0/849861992 >> v2:172.21.15.150:3300/0 conn(0x7f0d200c8e00 msgr2=0x7f0d2006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1f7fe640 1 --2- 172.21.15.78:0/849861992 >> v2:172.21.15.150:3300/0 conn(0x7f0d200c8e00 0x7f0d2006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1f7fe640 1 -- 172.21.15.78:0/849861992 >> v1:172.21.15.78:6789/0 conn(0x7f0d20152700 legacy=0x7f0d200c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1f7fe640 1 -- 172.21.15.78:0/849861992 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20155770 con 0x7f0d2006d910 2024-02-19T14:52:18.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1dffb640 1 -- 172.21.15.78:0/849861992 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156713560 0 0) 0x7f0d2006cd20 con 0x7f0d20152700 2024-02-19T14:52:18.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.520+0000 7f0d1e7fc640 1 --2- 172.21.15.78:0/849861992 >> v2:172.21.15.150:3300/0 conn(0x7f0d200c8e00 0x7f0d2006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:18.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.521+0000 7f0d1f7fe640 1 --2- 172.21.15.78:0/849861992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d2006d910 0x7f0d2006c7e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0d200c7620 tx=0x7f0d1802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b2f3a9da6d83b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.521+0000 7f0d1dffb640 1 -- 172.21.15.78:0/849861992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1802cc80 con 0x7f0d2006d910 2024-02-19T14:52:18.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.521+0000 7f0d1dffb640 1 -- 172.21.15.78:0/849861992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1802cde0 con 0x7f0d2006d910 2024-02-19T14:52:18.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.521+0000 7f0d1dffb640 1 -- 172.21.15.78:0/849861992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18034660 con 0x7f0d2006d910 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 -- 172.21.15.78:0/849861992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d2006d910 msgr2=0x7f0d2006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 --2- 172.21.15.78:0/849861992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d2006d910 0x7f0d2006c7e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0d200c7620 tx=0x7f0d1802bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 -- 172.21.15.78:0/849861992 shutdown_connections 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 --2- 172.21.15.78:0/849861992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d2006d910 0x7f0d2006c7e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 --2- 172.21.15.78:0/849861992 >> v2:172.21.15.150:3300/0 conn(0x7f0d200c8e00 0x7f0d2006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 -- 172.21.15.78:0/849861992 >> 172.21.15.78:0/849861992 conn(0x7f0d200cac90 msgr2=0x7f0d200cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 -- 172.21.15.78:0/849861992 shutdown_connections 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.522+0000 7f0d25977640 1 -- 172.21.15.78:0/849861992 wait complete. 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.523+0000 7f0d25977640 1 Processor -- start 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.523+0000 7f0d25977640 1 -- start start 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d25977640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d2006d910 0x7f0d201590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d25977640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 0x7f0d201edc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d25977640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d20152700 0x7f0d200c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d25977640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d20156f40 con 0x7f0d200c8e00 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d25977640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200c41a0 con 0x7f0d2006d910 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d25977640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200c4480 con 0x7f0d20152700 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 0x7f0d201edc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 0x7f0d201edc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52320/0 (socket says 172.21.15.78:52320) 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 -- 172.21.15.78:0/3812346354 learned_addr learned my addr 172.21.15.78:0/3812346354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1effd640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d2006d910 0x7f0d201590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1f7fe640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d20152700 0x7f0d200c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 -- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d20152700 msgr2=0x7f0d200c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d20152700 0x7f0d200c0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 -- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d2006d910 msgr2=0x7f0d201590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d2006d910 0x7f0d201590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 -- 172.21.15.78:0/3812346354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d18009a70 con 0x7f0d200c8e00 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1f7fe640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d20152700 0x7f0d200c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.524+0000 7f0d1e7fc640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 0x7f0d201edc10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c7f0 tx=0x7f0d1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.525+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10007a30 con 0x7f0d200c8e00 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.525+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10007b90 con 0x7f0d200c8e00 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.525+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1000ce40 con 0x7f0d200c8e00 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.525+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d201ee550 con 0x7f0d200c8e00 2024-02-19T14:52:18.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.525+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d200ad0b0 con 0x7f0d200c8e00 2024-02-19T14:52:18.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.526+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d200c4640 con 0x7f0d200c8e00 2024-02-19T14:52:18.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.526+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0d100113f0 con 0x7f0d200c8e00 2024-02-19T14:52:18.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.526+0000 7f0d07fff640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d0c06b810 0x7f0d0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.526+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0d1008a670 con 0x7f0d200c8e00 2024-02-19T14:52:18.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.526+0000 7f0d1effd640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d0c06b810 0x7f0d0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.527+0000 7f0d1effd640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d0c06b810 0x7f0d0c06dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007fd0 tx=0x7f0d14007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.531+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0d10054e00 con 0x7f0d200c8e00 2024-02-19T14:52:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.703+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d20152ae0 con 0x7f0d200c8e00 2024-02-19T14:52:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.704+0000 7f0d07fff640 1 -- 172.21.15.78:0/3812346354 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f0d100584c0 con 0x7f0d200c8e00 2024-02-19T14:52:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:18.708 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d0c06b810 msgr2=0x7f0d0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d0c06b810 0x7f0d0c06dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007fd0 tx=0x7f0d14007de0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 msgr2=0x7f0d201edc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 0x7f0d201edc10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c7f0 tx=0x7f0d1000cbf0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 shutdown_connections 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d0c06b810 0x7f0d0c06dcd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d200c8e00 0x7f0d201edc10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d20152700 0x7f0d200c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 --2- 172.21.15.78:0/3812346354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d2006d910 0x7f0d201590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 >> 172.21.15.78:0/3812346354 conn(0x7f0d200cac90 msgr2=0x7f0d20150770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 shutdown_connections 2024-02-19T14:52:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.706+0000 7f0d25977640 1 -- 172.21.15.78:0/3812346354 wait complete. 2024-02-19T14:52:18.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-02-19T14:52:18.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-02-19T14:52:18.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-02-19T14:52:18.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-02-19T14:52:18.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-02-19T14:52:18.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-02-19T14:52:18.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-02-19T14:52:18.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-02-19T14:52:18.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.897+0000 7fa3a7fff640 1 Processor -- start 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.897+0000 7fa3a7fff640 1 -- start start 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a00d3c40 0x7fa3a004ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3a004b1b0 0x7fa3a00d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a00d99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a7fff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a0046580 con 0x7fa3a004b1b0 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a7fff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a0047660 con 0x7fa3a00d3c40 2024-02-19T14:52:18.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a7fff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a00d65d0 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.897+0000 7fdeaf25c640 1 Processor -- start 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.897+0000 7fdeaf25c640 1 -- start start 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fdeaf25c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdea00071a0 0x7fdea0007580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fdeaf25c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdea004db40 0x7fdea00e2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fdeaf25c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea000e520 con 0x7fdea000e9b0 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fdeaf25c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea00e2c60 con 0x7fdea004db40 2024-02-19T14:52:18.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fdeaf25c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea004a550 con 0x7fdea00071a0 2024-02-19T14:52:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7f4d2aa39640 1 Processor -- start 2024-02-19T14:52:18.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7f4d2aa39640 1 -- start start 2024-02-19T14:52:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.901+0000 7f4d2aa39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d240c47e0 0x7f4d2415cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.899+0000 7fdea7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdea004db40 0x7fdea00e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.899+0000 7fdea7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdea004db40 0x7fdea00e2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52336/0 (socket says 172.21.15.78:52336) 2024-02-19T14:52:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.899+0000 7fdea7fff640 1 -- 172.21.15.78:0/3713551967 learned_addr learned my addr 172.21.15.78:0/3713551967 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fdea7fff640 1 -- 172.21.15.78:0/3713551967 >> v2:172.21.15.150:3300/0 conn(0x7fdea00071a0 msgr2=0x7fdea0007580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fdea7fff640 1 --2- 172.21.15.78:0/3713551967 >> v2:172.21.15.150:3300/0 conn(0x7fdea00071a0 0x7fdea0007580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fdea7fff640 1 -- 172.21.15.78:0/3713551967 >> v1:172.21.15.78:6789/0 conn(0x7fdea000e9b0 legacy=0x7fdea00de790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fdea7fff640 1 -- 172.21.15.78:0/3713551967 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea000e1f0 con 0x7fdea004db40 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.901+0000 7f4d2aa39640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2406e170 con 0x7f4d2414e800 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.901+0000 7f4d2aa39640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d240c5490 con 0x7f4d2406f440 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.901+0000 7f4d2aa39640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d240c1d30 con 0x7f4d240c47e0 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.901+0000 7f4d237fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4d2406f440 0x7f4d24159850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50226/0 (socket says 172.21.15.78:50226) 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.901+0000 7f4d237fe640 1 -- 172.21.15.78:0/2746408790 learned_addr learned my addr 172.21.15.78:0/2746408790 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d22ffd640 1 -- 172.21.15.78:0/2746408790 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199688059 0 0) 0x7f4d240c5490 con 0x7f4d2406f440 2024-02-19T14:52:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d28fd6640 1 --2- 172.21.15.78:0/2746408790 >> v2:172.21.15.150:3300/0 conn(0x7f4d240c47e0 0x7f4d2415cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d22ffd640 1 -- 172.21.15.78:0/2746408790 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c003520 con 0x7f4d2406f440 2024-02-19T14:52:18.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d22ffd640 1 -- 172.21.15.78:0/2746408790 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659923223 0 0) 0x7f4d2406e170 con 0x7f4d2414e800 2024-02-19T14:52:18.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d22ffd640 1 -- 172.21.15.78:0/2746408790 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d240c5490 con 0x7f4d2414e800 2024-02-19T14:52:18.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d28fd6640 1 -- 172.21.15.78:0/2746408790 >> v1:172.21.15.150:6789/0 conn(0x7f4d2406f440 legacy=0x7f4d24159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.903+0000 7fdea7fff640 1 --2- 172.21.15.78:0/3713551967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00e2720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdea0050c40 tx=0x7fde9402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b9c13a93941b9b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.903+0000 7fdea77fe640 1 -- 172.21.15.78:0/3713551967 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94031890 con 0x7fdea004db40 2024-02-19T14:52:18.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.902+0000 7f4d28fd6640 1 -- 172.21.15.78:0/2746408790 >> v1:172.21.15.78:6789/0 conn(0x7f4d2414e800 legacy=0x7f4d241515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.903+0000 7f4d28fd6640 1 -- 172.21.15.78:0/2746408790 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d240c4ed0 con 0x7f4d240c47e0 2024-02-19T14:52:18.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.904+0000 7fdea77fe640 1 -- 172.21.15.78:0/3713551967 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde940319f0 con 0x7fdea004db40 2024-02-19T14:52:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.904+0000 7fdea77fe640 1 -- 172.21.15.78:0/3713551967 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde94031d60 con 0x7fdea004db40 2024-02-19T14:52:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.904+0000 7f4d28fd6640 1 --2- 172.21.15.78:0/2746408790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d240c47e0 0x7f4d2415cf80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009720 tx=0x7f4d1802b620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=525a60290d5621ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.904+0000 7f4d22ffd640 1 -- 172.21.15.78:0/2746408790 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1802bea0 con 0x7f4d240c47e0 2024-02-19T14:52:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.904+0000 7f4d22ffd640 1 -- 172.21.15.78:0/2746408790 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1802e050 con 0x7f4d240c47e0 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a00d3c40 0x7fa3a004ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.898+0000 7fa3a6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a00d3c40 0x7fa3a004ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45694/0 (socket says 172.21.15.78:45694) 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.899+0000 7fa3a6ffd640 1 -- 172.21.15.78:0/2959003766 learned_addr learned my addr 172.21.15.78:0/2959003766 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fa3a77fe640 1 --2- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a00d99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fa3a6ffd640 1 -- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 msgr2=0x7fa3a00d99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fa3a6ffd640 1 --2- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a00d99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fa3a6ffd640 1 -- 172.21.15.78:0/2959003766 >> v2:172.21.15.78:3300/0 conn(0x7fa3a004b1b0 msgr2=0x7fa3a00d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fa3a6ffd640 1 --2- 172.21.15.78:0/2959003766 >> v2:172.21.15.78:3300/0 conn(0x7fa3a004b1b0 0x7fa3a00d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.900+0000 7fa3a6ffd640 1 -- 172.21.15.78:0/2959003766 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a00468d0 con 0x7fa3a00d3c40 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.904+0000 7fa3a6ffd640 1 --2- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00d3c40 0x7fa3a004ac70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002b240 tx=0x7fa3980344b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bc29420380ce807 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7fa3a5ffb640 1 -- 172.21.15.78:0/2959003766 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39803a480 con 0x7fa3a00d3c40 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7fa3a5ffb640 1 -- 172.21.15.78:0/2959003766 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398037030 con 0x7fa3a00d3c40 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2746408790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d240c47e0 msgr2=0x7f4d2415cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7f4d2aa39640 1 --2- 172.21.15.78:0/2746408790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d240c47e0 0x7f4d2415cf80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009720 tx=0x7f4d1802b620 comp rx=0 tx=0).stop 2024-02-19T14:52:18.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2746408790 shutdown_connections 2024-02-19T14:52:18.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7f4d2aa39640 1 --2- 172.21.15.78:0/2746408790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d240c47e0 0x7f4d2415cf80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.905+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2746408790 >> 172.21.15.78:0/2746408790 conn(0x7f4d2406b270 msgr2=0x7f4d2406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 -- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00d3c40 msgr2=0x7fa3a004ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 --2- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00d3c40 0x7fa3a004ac70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002b240 tx=0x7fa3980344b0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 -- 172.21.15.78:0/2959003766 shutdown_connections 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 --2- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a00d99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fdeaf25c640 1 -- 172.21.15.78:0/3713551967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 msgr2=0x7fdea00e2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fdeaf25c640 1 --2- 172.21.15.78:0/3713551967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00e2720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdea0050c40 tx=0x7fde9402b930 comp rx=0 tx=0).stop 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fdeaf25c640 1 -- 172.21.15.78:0/3713551967 shutdown_connections 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fdeaf25c640 1 --2- 172.21.15.78:0/3713551967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00e2720 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fdeaf25c640 1 --2- 172.21.15.78:0/3713551967 >> v2:172.21.15.150:3300/0 conn(0x7fdea00071a0 0x7fdea0007580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fdeaf25c640 1 -- 172.21.15.78:0/3713551967 >> 172.21.15.78:0/3713551967 conn(0x7fdea004c650 msgr2=0x7fdea004ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 --2- 172.21.15.78:0/2959003766 >> v2:172.21.15.78:3300/0 conn(0x7fa3a004b1b0 0x7fa3a00d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 --2- 172.21.15.78:0/2959003766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00d3c40 0x7fa3a004ac70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.906+0000 7fa3a7fff640 1 -- 172.21.15.78:0/2959003766 >> 172.21.15.78:0/2959003766 conn(0x7fa3a0005e90 msgr2=0x7fa3a00062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fdeaf25c640 1 -- 172.21.15.78:0/3713551967 shutdown_connections 2024-02-19T14:52:18.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fdeaf25c640 1 -- 172.21.15.78:0/3713551967 wait complete. 2024-02-19T14:52:18.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fa3a7fff640 1 -- 172.21.15.78:0/2959003766 shutdown_connections 2024-02-19T14:52:18.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2746408790 shutdown_connections 2024-02-19T14:52:18.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7fa3a7fff640 1 -- 172.21.15.78:0/2959003766 wait complete. 2024-02-19T14:52:18.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.907+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2746408790 wait complete. 2024-02-19T14:52:18.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.908+0000 7fdeaf25c640 1 Processor -- start 2024-02-19T14:52:18.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.908+0000 7fdeaf25c640 1 -- start start 2024-02-19T14:52:18.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.909+0000 7f4d2aa39640 1 Processor -- start 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.910+0000 7fdeaf25c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdea00071a0 0x7fdea00c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.910+0000 7fdeaf25c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdea000e9b0 0x7fdea0042540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.910+0000 7fdeaf25c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.910+0000 7fdeacff8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdea00071a0 0x7fdea00c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.910+0000 7fdeacff8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdea00071a0 0x7fdea00c9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50828/0 (socket says 172.21.15.78:50828) 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.910+0000 7fdea7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdea000e9b0 0x7fdea0042540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdea7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdea000e9b0 0x7fdea0042540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45714/0 (socket says 172.21.15.78:45714) 2024-02-19T14:52:18.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdeaf25c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00cd5c0 con 0x7fdea004db40 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00cc5b0 con 0x7fdea000e9b0 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00cc800 con 0x7fdea00071a0 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdeacff8640 1 -- 172.21.15.78:0/2676233706 learned_addr learned my addr 172.21.15.78:0/2676233706 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdead7f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdeacff8640 1 -- 172.21.15.78:0/2676233706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdea000e9b0 msgr2=0x7fdea0042540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.911+0000 7fdeacff8640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdea000e9b0 0x7fdea0042540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.912+0000 7fdeacff8640 1 -- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 msgr2=0x7fdea00c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.912+0000 7fdeacff8640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.912+0000 7fdeacff8640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940094d0 con 0x7fdea00071a0 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.912+0000 7fdead7f9640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdea004db40 0x7fdea00c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.912+0000 7fdeacff8640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdea00071a0 0x7fdea00c9f20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdea80507a0 tx=0x7fdea80689b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7fa3a7fff640 1 Processor -- start 2024-02-19T14:52:18.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80748b0 con 0x7fdea00071a0 2024-02-19T14:52:18.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea8074a50 con 0x7fdea00071a0 2024-02-19T14:52:18.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea00ccae0 con 0x7fdea00071a0 2024-02-19T14:52:18.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7fa3a7fff640 1 -- start start 2024-02-19T14:52:18.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8092580 con 0x7fdea00071a0 2024-02-19T14:52:18.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.915+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a004b1b0 0x7fa3a00d8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea00cd0a0 con 0x7fdea00071a0 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.915+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a00d3c40 0x7fa3a00407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.915+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a0040cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.916+0000 7fa3a7fff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a00d94d0 con 0x7fa3a00d3c40 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.916+0000 7fa3a7fff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0041c50 con 0x7fa3a004b1b0 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.916+0000 7fa3a7fff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0041970 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.916+0000 7fa3a77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a0040cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a0040cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50848/0 (socket says 172.21.15.78:50848) 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 -- 172.21.15.78:0/1541744532 learned_addr learned my addr 172.21.15.78:0/1541744532 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a6ffd640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a004b1b0 0x7fa3a00d8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde88001030 con 0x7fdea00071a0 2024-02-19T14:52:18.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a67fc640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a00d3c40 0x7fa3a00407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdea80929b0 con 0x7fdea00071a0 2024-02-19T14:52:18.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 -- 172.21.15.78:0/1541744532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a004b1b0 msgr2=0x7fa3a00d8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a004b1b0 0x7fa3a00d8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 -- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a00d3c40 msgr2=0x7fa3a00407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a00d3c40 0x7fa3a00407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.917+0000 7fa3a77fe640 1 -- 172.21.15.78:0/1541744532 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39800a570 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.918+0000 7fa3a77fe640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a0040cf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa3a806a1b0 tx=0x7fa3a80689b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.909+0000 7f4d2aa39640 1 -- start start 2024-02-19T14:52:18.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d2aa39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d2406f440 0x7f4d24154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d2aa39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d240c47e0 0x7f4d24153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d2aa39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d2414e800 0x7f4d2414d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d2aa39640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d241531b0 con 0x7f4d240c47e0 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d2aa39640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d24153670 con 0x7f4d2406f440 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d2aa39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d241f8d40 con 0x7f4d2414e800 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d237fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d240c47e0 0x7f4d24153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d237fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d240c47e0 0x7f4d24153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52356/0 (socket says 172.21.15.78:52356) 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d237fe640 1 -- 172.21.15.78:0/2439564419 learned_addr learned my addr 172.21.15.78:0/2439564419 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d23fff640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d2406f440 0x7f4d24154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.913+0000 7f4d28fd6640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d2414e800 0x7f4d2414d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d237fe640 1 -- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d2414e800 msgr2=0x7f4d2414d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d237fe640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d2414e800 0x7f4d2414d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d237fe640 1 -- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d2406f440 msgr2=0x7f4d24154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d237fe640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d2406f440 0x7f4d24154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d237fe640 1 -- 172.21.15.78:0/2439564419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18009400 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d237fe640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d240c47e0 0x7f4d24153a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c001f50 tx=0x7f4d1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.914+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00ab40 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.915+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d24153390 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.915+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d241f89d0 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.918+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c00aca0 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.919+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c007310 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.919+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4d1c007470 con 0x7f4d240c47e0 2024-02-19T14:52:18.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.919+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80746b0 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.919+0000 7fa3a7fff640 1 -- 172.21.15.78:0/1541744532 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a016f900 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.920+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d04001030 con 0x7f4d240c47e0 2024-02-19T14:52:18.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.919+0000 7fa3a7fff640 1 -- 172.21.15.78:0/1541744532 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a0032c20 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.920+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a8074810 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.920+0000 7f6215a8f640 1 Processor -- start 2024-02-19T14:52:18.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.920+0000 7f6215a8f640 1 -- start start 2024-02-19T14:52:18.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.920+0000 7fdea57fa640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde8c06b8e0 0x7fde8c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.920+0000 7f4d20ff9640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1006b910 0x7f4d1006ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8094440 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7fdea7fff640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde8c06b8e0 0x7fde8c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7fdea7fff640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde8c06b8e0 0x7fde8c06dda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde94036000 tx=0x7fde94008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdea81050e0 con 0x7fdea00071a0 2024-02-19T14:52:18.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7fa3a7fff640 1 -- 172.21.15.78:0/1541744532 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa388001030 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa3a80945a0 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7fdea57fa640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7fde8c075060 0x7fde8c0774a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde8c077b70 con 0x7fde8c075060 2024-02-19T14:52:18.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdea8107700 con 0x7fdea00071a0 2024-02-19T14:52:18.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7f290f072640 1 Processor -- start 2024-02-19T14:52:18.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7fdead7f9640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7fde8c075060 0x7fde8c0774a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7fdead7f9640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7fde8c075060 0x7fde8c0774a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4d1c08a1a0 con 0x7f4d240c47e0 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f4d20ff9640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7f4d10074f10 0x7f4d10077350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 --> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d10077a20 con 0x7f4d10074f10 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d1c08c550 con 0x7f4d240c47e0 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f4d23fff640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1006b910 0x7f4d1006ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7f4d28fd6640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7f4d10074f10 0x7f4d10077350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7f4d28fd6640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7f4d10074f10 0x7f4d10077350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f6215a8f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f621014e800 0x7f62100c3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f6215a8f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f621006e170 con 0x7f62100c3990 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f6215a8f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62100c5490 con 0x7f62100c47e0 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f6215a8f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62100c1d30 con 0x7f621014e800 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f620ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f62100c3990 0x7f621015b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57688/0 (socket says 172.21.15.78:57688) 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f620ffff640 1 -- 172.21.15.78:0/2428603256 learned_addr learned my addr 172.21.15.78:0/2428603256 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.921+0000 7f620f7fe640 1 --2- 172.21.15.78:0/2428603256 >> v2:172.21.15.78:3300/0 conn(0x7f621014e800 0x7f62100c3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f620f7fe640 1 -- 172.21.15.78:0/2428603256 >> v1:172.21.15.150:6789/0 conn(0x7f62100c47e0 legacy=0x7f62101515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f620f7fe640 1 -- 172.21.15.78:0/2428603256 >> v1:172.21.15.78:6789/0 conn(0x7f62100c3990 legacy=0x7f621015b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.922+0000 7f620f7fe640 1 -- 172.21.15.78:0/2428603256 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62100c4ed0 con 0x7f621014e800 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f620f7fe640 1 --2- 172.21.15.78:0/2428603256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f621014e800 0x7f62100c3450 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62040097a0 tx=0x7f620402b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317a1905c81a03c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f620e7fc640 1 -- 172.21.15.78:0/2428603256 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204030710 con 0x7f621014e800 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f620e7fc640 1 -- 172.21.15.78:0/2428603256 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620402e050 con 0x7f621014e800 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7f290f072640 1 -- start start 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f290f072640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f29000071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f290f072640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f290004e020 0x7f290004e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f290f072640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290000fa00 con 0x7f290004db10 2024-02-19T14:52:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f290f072640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29000da4e0 con 0x7f290004e020 2024-02-19T14:52:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f290f072640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000da7c0 con 0x7f29000d7b10 2024-02-19T14:52:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== osd.0 v2:172.21.15.78:6816/2796609917 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fde8c077b70 con 0x7fde8c075060 2024-02-19T14:52:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f29000071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f29000071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50864/0 (socket says 172.21.15.78:50864) 2024-02-19T14:52:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 -- 172.21.15.78:0/2193882817 learned_addr learned my addr 172.21.15.78:0/2193882817 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f2907fff640 1 --2- 172.21.15.78:0/2193882817 >> v2:172.21.15.150:3300/0 conn(0x7f290004e020 0x7f290004e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 -- 172.21.15.78:0/2193882817 >> v2:172.21.15.150:3300/0 conn(0x7f290004e020 msgr2=0x7f290004e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 --2- 172.21.15.78:0/2193882817 >> v2:172.21.15.150:3300/0 conn(0x7f290004e020 0x7f290004e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 -- 172.21.15.78:0/2193882817 >> v1:172.21.15.150:6789/0 conn(0x7f290004db10 legacy=0x7f290004ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f290ce0e640 1 -- 172.21.15.78:0/2193882817 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000daaa0 con 0x7f29000d7b10 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f2907fff640 1 --2- 172.21.15.78:0/2193882817 >> v2:172.21.15.150:3300/0 conn(0x7f290004e020 0x7f290004e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7fa38f7fe640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa39406b8e0 0x7fa39406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.923+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa3a8104d50 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7fa38f7fe640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7fa394074fd0 0x7fa394077410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 --> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa394077ae0 con 0x7fa394074fd0 2024-02-19T14:52:18.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7fa3a6ffd640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa39406b8e0 0x7fa39406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7fa3a67fc640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7fa394074fd0 0x7fa394077410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7fa3a6ffd640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa39406b8e0 0x7fa39406dda0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa39803f000 tx=0x7fa398038000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.924+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== osd.5 v2:172.21.15.150:6800/911852656 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4d10077a20 con 0x7f4d10074f10 2024-02-19T14:52:18.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f6215a8f640 1 -- 172.21.15.78:0/2428603256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f621014e800 msgr2=0x7f62100c3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f620e7fc640 1 -- 172.21.15.78:0/2428603256 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204030710 con 0x7f621014e800 2024-02-19T14:52:18.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f6215a8f640 1 --2- 172.21.15.78:0/2428603256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f621014e800 0x7f62100c3450 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62040097a0 tx=0x7f620402b570 comp rx=0 tx=0).stop 2024-02-19T14:52:18.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.926+0000 7f4d23fff640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1006b910 0x7f4d1006ddd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4d14001640 tx=0x7f4d14009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f6215a8f640 1 -- 172.21.15.78:0/2428603256 shutdown_connections 2024-02-19T14:52:18.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f6215a8f640 1 --2- 172.21.15.78:0/2428603256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f621014e800 0x7f62100c3450 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7f6215a8f640 1 -- 172.21.15.78:0/2428603256 >> 172.21.15.78:0/2428603256 conn(0x7f621006b270 msgr2=0x7f621006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.926+0000 7f6215a8f640 1 -- 172.21.15.78:0/2428603256 shutdown_connections 2024-02-19T14:52:18.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.926+0000 7f6215a8f640 1 -- 172.21.15.78:0/2428603256 wait complete. 2024-02-19T14:52:18.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.925+0000 7fa3a67fc640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7fa394074fd0 0x7fa394077410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.926+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3a80cf590 con 0x7fa3a00d6fa0 2024-02-19T14:52:18.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.926+0000 7fd164df3640 1 Processor -- start 2024-02-19T14:52:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.926+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== osd.3 v2:172.21.15.78:6824/3716944111 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa394077ae0 con 0x7fa394074fd0 2024-02-19T14:52:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.927+0000 7fd164df3640 1 -- start start 2024-02-19T14:52:18.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.927+0000 7f6215a8f640 1 Processor -- start 2024-02-19T14:52:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7fd164df3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd1600c85c0 0x7fd1600c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7fd164df3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160155dc0 con 0x7fd16006cc00 2024-02-19T14:52:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7fd15ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd16006cc00 0x7fd16006d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57692/0 (socket says 172.21.15.78:57692) 2024-02-19T14:52:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7fd15ffff640 1 -- 172.21.15.78:0/1079608996 learned_addr learned my addr 172.21.15.78:0/1079608996 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:18.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160155000 con 0x7fd1601525c0 2024-02-19T14:52:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1601529a0 con 0x7fd1600c85c0 2024-02-19T14:52:18.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.928+0000 7f6215a8f640 1 -- start start 2024-02-19T14:52:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15effd640 1 --2- 172.21.15.78:0/1079608996 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c85c0 0x7fd1600c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462851811 0 0) 0x7fd160155dc0 con 0x7fd16006cc00 2024-02-19T14:52:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd148003520 con 0x7fd16006cc00 2024-02-19T14:52:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048043027 0 0) 0x7fd160155000 con 0x7fd1601525c0 2024-02-19T14:52:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd160155dc0 con 0x7fd1601525c0 2024-02-19T14:52:18.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f290ce0e640 1 --2- 172.21.15.78:0/2193882817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f29000071e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009870 tx=0x7f28fc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359427b34d250ea5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2031472898 0 0) 0x7fd148003520 con 0x7fd16006cc00 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c85c0 msgr2=0x7fd1600c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 --2- 172.21.15.78:0/1079608996 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c85c0 0x7fd1600c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 >> v1:172.21.15.150:6789/0 conn(0x7fd1601525c0 legacy=0x7fd1600c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1601552e0 con 0x7fd16006cc00 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fd154001670 con 0x7fd16006cc00 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 >> v1:172.21.15.78:6789/0 conn(0x7fd16006cc00 legacy=0x7fd16006d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd148003b50 0x7fd148003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1480047c0 0x7fd148004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1480051d0 0x7fd14800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601529a0 con 0x7fd148003b50 2024-02-19T14:52:18.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c100 con 0x7fd1480047c0 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.929+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c2c0 con 0x7fd1480051d0 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd148003b50 0x7fd148003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15ffff640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1480051d0 0x7fd14800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f29077fe640 1 -- 172.21.15.78:0/2193882817 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc031640 con 0x7f29000d7b10 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f29077fe640 1 -- 172.21.15.78:0/2193882817 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc0317a0 con 0x7f29000d7b10 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f290f072640 1 -- 172.21.15.78:0/2193882817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 msgr2=0x7f29000071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f29077fe640 1 -- 172.21.15.78:0/2193882817 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc031640 con 0x7f29000d7b10 2024-02-19T14:52:18.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f290f072640 1 --2- 172.21.15.78:0/2193882817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f29000071e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009870 tx=0x7f28fc02b8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15effd640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1480047c0 0x7fd148004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f6215a8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62100c3990 0x7f62101f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f6215a8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62100c47e0 0x7f62101e2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f6215a8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6210154040 0x7f62101ef340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f6215a8f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62101eef20 con 0x7f62100c47e0 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f6215a8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62101eec30 con 0x7f6210154040 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f6215a8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6210152b90 con 0x7f62100c3990 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7f620effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62100c47e0 0x7f62101e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62100c47e0 0x7f62101e2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52392/0 (socket says 172.21.15.78:52392) 2024-02-19T14:52:18.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 -- 172.21.15.78:0/381679873 learned_addr learned my addr 172.21.15.78:0/381679873 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620f7fe640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62100c3990 0x7f62101f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15f7fe640 1 -- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1480051d0 msgr2=0x7fd14800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f290f072640 1 -- 172.21.15.78:0/2193882817 shutdown_connections 2024-02-19T14:52:18.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f290f072640 1 --2- 172.21.15.78:0/2193882817 >> v2:172.21.15.150:3300/0 conn(0x7f290004e020 0x7f290004e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f290f072640 1 --2- 172.21.15.78:0/2193882817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f29000071e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f290f072640 1 -- 172.21.15.78:0/2193882817 >> 172.21.15.78:0/2193882817 conn(0x7f290004c690 msgr2=0x7f290004ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1480051d0 0x7fd14800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15f7fe640 1 -- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1480047c0 msgr2=0x7fd148004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f290f072640 1 -- 172.21.15.78:0/2193882817 shutdown_connections 2024-02-19T14:52:18.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1480047c0 0x7fd148004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.930+0000 7fd15f7fe640 1 -- 172.21.15.78:0/1079608996 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148003520 con 0x7fd148003b50 2024-02-19T14:52:18.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd148003b50 0x7fd148003f50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd158001ea0 tx=0x7fd1480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be85c0e9846b807 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7fd15ffff640 1 -- 172.21.15.78:0/1079608996 reap_dead start 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 -- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62100c3990 msgr2=0x7f62101f2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62100c3990 0x7f62101f2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 -- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6210154040 msgr2=0x7f62101ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6210154040 0x7f62101ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 -- 172.21.15.78:0/381679873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6204009400 con 0x7f62100c47e0 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.931+0000 7f620effd640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62100c47e0 0x7f62101e2840 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f620000c780 tx=0x7f620000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6200005270 con 0x7f62100c47e0 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62101dfc30 con 0x7f62100c47e0 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.933+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62101525d0 con 0x7f62100c47e0 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.933+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620000ce90 con 0x7f62100c47e0 2024-02-19T14:52:18.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.933+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000a640 con 0x7f62100c47e0 2024-02-19T14:52:18.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7f290f072640 1 -- 172.21.15.78:0/2193882817 wait complete. 2024-02-19T14:52:18.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.933+0000 7f290f072640 1 Processor -- start 2024-02-19T14:52:18.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7f290f072640 1 -- start start 2024-02-19T14:52:18.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158003b70 con 0x7fd148003b50 2024-02-19T14:52:18.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7fd15e7fc640 1 -- 172.21.15.78:0/1079608996 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158003cd0 con 0x7fd148003b50 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7ff0e987b640 1 Processor -- start 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.932+0000 7ff0e987b640 1 -- start start 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e987b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 0x7ff0dc0ea110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e987b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0dc0311a0 con 0x7ff0dc0dc010 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e987b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0dc04aaa0 con 0x7ff0dc00b100 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e987b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc031460 con 0x7ff0dc0070f0 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 0x7ff0dc0ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 0x7ff0dc0ea110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50884/0 (socket says 172.21.15.78:50884) 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e37fe640 1 -- 172.21.15.78:0/2374977832 learned_addr learned my addr 172.21.15.78:0/2374977832 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e27fc640 1 --1- 172.21.15.78:0/2374977832 >> v1:172.21.15.150:6789/0 conn(0x7ff0dc00b100 0x7ff0dc0069e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50266/0 (socket says 172.21.15.78:50266) 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e37fe640 1 -- 172.21.15.78:0/2374977832 >> v1:172.21.15.150:6789/0 conn(0x7ff0dc00b100 legacy=0x7ff0dc0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.934+0000 7ff0e37fe640 1 -- 172.21.15.78:0/2374977832 >> v1:172.21.15.78:6789/0 conn(0x7ff0dc0dc010 legacy=0x7ff0dc03e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7ff0e37fe640 1 -- 172.21.15.78:0/2374977832 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc04b120 con 0x7ff0dc0070f0 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd148003b50 msgr2=0x7fd148003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7fd164df3640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd148003b50 0x7fd148003f50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd158001ea0 tx=0x7fd1480057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 shutdown_connections 2024-02-19T14:52:18.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7fd164df3640 1 --2- 172.21.15.78:0/1079608996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd148003b50 0x7fd148003f50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 >> 172.21.15.78:0/1079608996 conn(0x7fd1600ac270 msgr2=0x7fd1600ac680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f290f072640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f290004db10 0x7f2900043740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f290f072640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f290004e020 0x7f29000d9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f290f072640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f2900170580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f2907fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f290004e020 0x7f29000d9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f2907fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f290004e020 0x7f29000d9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45764/0 (socket says 172.21.15.78:45764) 2024-02-19T14:52:18.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f2907fff640 1 -- 172.21.15.78:0/2617691543 learned_addr learned my addr 172.21.15.78:0/2617691543 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f0f77fff640 1 Processor -- start 2024-02-19T14:52:18.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 shutdown_connections 2024-02-19T14:52:18.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7fd164df3640 1 -- 172.21.15.78:0/1079608996 wait complete. 2024-02-19T14:52:18.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2900042e60 con 0x7f290004db10 2024-02-19T14:52:18.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000d88f0 con 0x7f290004e020 2024-02-19T14:52:18.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2900043c80 con 0x7f29000d7b10 2024-02-19T14:52:18.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.935+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f620000a840 con 0x7f62100c47e0 2024-02-19T14:52:18.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f0f77fff640 1 -- start start 2024-02-19T14:52:18.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7ff0e37fe640 1 --2- 172.21.15.78:0/2374977832 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 0x7ff0dc0ea110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80098a0 tx=0x7ff0d802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354c4a6d72250b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f61f7fff640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61fc06b8e0 0x7f61fc06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6200089870 con 0x7f62100c47e0 2024-02-19T14:52:18.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f620f7fe640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61fc06b8e0 0x7f61fc06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7ff0e1ffb640 1 -- 172.21.15.78:0/2374977832 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8031640 con 0x7ff0dc0070f0 2024-02-19T14:52:18.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7ff0e1ffb640 1 -- 172.21.15.78:0/2374977832 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d80317a0 con 0x7ff0dc0070f0 2024-02-19T14:52:18.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7ff0e1ffb640 1 -- 172.21.15.78:0/2374977832 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8031b30 con 0x7ff0dc0070f0 2024-02-19T14:52:18.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f6215a8f640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f61f0001650 0x7f61f0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.936+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 --> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61f0006bb0 con 0x7f61f0001650 2024-02-19T14:52:18.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7f620ffff640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f61f0001650 0x7f61f0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7f290ce0e640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f290004db10 0x7f2900043740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7f290d60f640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f2900170580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f77fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155070 0x7f0f781554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f77fff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7806faf0 con 0x7f0f7806c790 2024-02-19T14:52:18.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f77fff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f780c1b60 con 0x7f0f78155a10 2024-02-19T14:52:18.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f77fff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78158420 con 0x7f0f78155070 2024-02-19T14:52:18.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7fd164df3640 1 Processor -- start 2024-02-19T14:52:18.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.937+0000 7f620f7fe640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61fc06b8e0 0x7f61fc06dda0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6204034000 tx=0x7f6204008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 -- 172.21.15.78:0/2374977832 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 msgr2=0x7ff0dc0ea110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 --2- 172.21.15.78:0/2374977832 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 0x7ff0dc0ea110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80098a0 tx=0x7ff0d802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 -- 172.21.15.78:0/2374977832 shutdown_connections 2024-02-19T14:52:18.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 --2- 172.21.15.78:0/2374977832 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc0070f0 0x7ff0dc0ea110 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 -- 172.21.15.78:0/2374977832 >> 172.21.15.78:0/2374977832 conn(0x7ff0dc005ec0 msgr2=0x7ff0dc0062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f76ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0f7806c790 0x7f0f780c1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57710/0 (socket says 172.21.15.78:57710) 2024-02-19T14:52:18.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f76ffd640 1 -- 172.21.15.78:0/1272568225 learned_addr learned my addr 172.21.15.78:0/1272568225 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:18.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7fd164df3640 1 -- start start 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f767fc640 1 --2- 172.21.15.78:0/1272568225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155070 0x7f0f781554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f290ce0e640 1 -- 172.21.15.78:0/2617691543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 msgr2=0x7f2900170580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f290ce0e640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f2900170580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f290ce0e640 1 -- 172.21.15.78:0/2617691543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f290004e020 msgr2=0x7f29000d9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f290ce0e640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f290004e020 0x7f29000d9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f290ce0e640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0094d0 con 0x7f290004db10 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 -- 172.21.15.78:0/2374977832 shutdown_connections 2024-02-19T14:52:18.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7ff0e987b640 1 -- 172.21.15.78:0/2374977832 wait complete. 2024-02-19T14:52:18.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.938+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127613842 0 0) 0x7f0f7806faf0 con 0x7f0f7806c790 2024-02-19T14:52:18.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f290ce0e640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f290004db10 0x7f2900043740 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28fc033000 tx=0x7f28fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc003810 con 0x7f290004db10 2024-02-19T14:52:18.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc03e070 con 0x7f290004db10 2024-02-19T14:52:18.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc003810 con 0x7f290004db10 2024-02-19T14:52:18.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7fd164df3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd148003b50 0x7fd1601ea920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f60003520 con 0x7f0f7806c790 2024-02-19T14:52:18.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971324333 0 0) 0x7f0f780c1b60 con 0x7f0f78155a10 2024-02-19T14:52:18.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f7806faf0 con 0x7f0f78155a10 2024-02-19T14:52:18.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7fd164df3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600bdb20 0x7fd16014d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7fd164df3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c43a0 0x7fd1601ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7fd15f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd148003b50 0x7fd1601ea920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd148003b50 0x7fd1601ea920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45768/0 (socket says 172.21.15.78:45768) 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c43a0 0x7fd1601ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c43a0 0x7fd1601ee550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52418/0 (socket says 172.21.15.78:52418) 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 -- 172.21.15.78:0/3771036465 learned_addr learned my addr 172.21.15.78:0/3771036465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600c1c10 con 0x7fd1600c43a0 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600c2080 con 0x7fd148003b50 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600c3f80 con 0x7fd1600bdb20 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f0f767fc640 1 -- 172.21.15.78:0/1272568225 >> v1:172.21.15.150:6789/0 conn(0x7f0f78155a10 legacy=0x7f0f7806ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f0f767fc640 1 -- 172.21.15.78:0/1272568225 >> v1:172.21.15.78:6789/0 conn(0x7f0f7806c790 legacy=0x7f0f780c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.939+0000 7f0f767fc640 1 -- 172.21.15.78:0/1272568225 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7806eee0 con 0x7f0f78155070 2024-02-19T14:52:18.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7ff0e987b640 1 Processor -- start 2024-02-19T14:52:18.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7f290d60f640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29000d7b10 0x7f2900170580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:18.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7ff0e987b640 1 -- start start 2024-02-19T14:52:18.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 -- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600bdb20 msgr2=0x7fd16014d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600bdb20 0x7fd16014d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 -- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c43a0 msgr2=0x7fd1601ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c43a0 0x7fd1601ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.940+0000 7fd15f7fe640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600c1e10 con 0x7fd148003b50 2024-02-19T14:52:18.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.941+0000 7f620ffff640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f61f0001650 0x7f61f0003b10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.941+0000 7fd15ffff640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c43a0 0x7fd1601ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:18.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.941+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29000d9af0 con 0x7f290004db10 2024-02-19T14:52:18.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.941+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2900049ac0 con 0x7f290004db10 2024-02-19T14:52:18.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.941+0000 7f0f767fc640 1 --2- 172.21.15.78:0/1272568225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155070 0x7f0f781554d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0097d0 tx=0x7f0f6c02b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05828529621a81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e987b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0dc0070f0 0x7ff0dc0d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== osd.7 v2:172.21.15.150:6819/1386990918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f61f0006bb0 con 0x7f61f0001650 2024-02-19T14:52:18.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e987b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc00b100 0x7ff0dc169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29000d7340 con 0x7f290004db10 2024-02-19T14:52:18.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f680019f0 con 0x7f0f78155070 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0dc0070f0 0x7ff0dc0d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0dc0070f0 0x7ff0dc0d6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45770/0 (socket says 172.21.15.78:45770) 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e987b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0dc0dc010 0x7ff0dc0d6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e27fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc00b100 0x7ff0dc169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e27fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc00b100 0x7ff0dc169a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50914/0 (socket says 172.21.15.78:50914) 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0dc0dc010 0x7ff0dc0d6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0dc0dc010 0x7ff0dc0d6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52420/0 (socket says 172.21.15.78:52420) 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e2ffd640 1 -- 172.21.15.78:0/3737506923 learned_addr learned my addr 172.21.15.78:0/3737506923 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc16b580 con 0x7ff0dc0dc010 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc178a40 con 0x7ff0dc0070f0 2024-02-19T14:52:18.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0e2320 con 0x7ff0dc00b100 2024-02-19T14:52:18.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.942+0000 7fd15f7fe640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd148003b50 0x7fd1601ea920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd158001fc0 tx=0x7fd158001580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c02e050 con 0x7f0f78155070 2024-02-19T14:52:18.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7f0f75ffb640 1 -- 172.21.15.78:0/1272568225 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c030730 con 0x7f0f78155070 2024-02-19T14:52:18.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f28fc02f050 con 0x7f290004db10 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e2ffd640 1 -- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc00b100 msgr2=0x7ff0dc169a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e2ffd640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc00b100 0x7ff0dc169a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e2ffd640 1 -- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0dc0dc010 msgr2=0x7ff0dc0d6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e2ffd640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0dc0dc010 0x7ff0dc0d6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e2ffd640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d80094d0 con 0x7ff0dc0070f0 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e27fc640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0dc00b100 0x7ff0dc169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e37fe640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0dc0dc010 0x7ff0dc0d6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:18.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0e2ffd640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0dc0070f0 0x7ff0dc0d6de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400b3e0 tx=0x7ff0d400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15803e070 con 0x7fd148003b50 2024-02-19T14:52:18.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4011830 con 0x7ff0dc0070f0 2024-02-19T14:52:18.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d40119b0 con 0x7ff0dc0070f0 2024-02-19T14:52:18.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d4011d60 con 0x7ff0dc0070f0 2024-02-19T14:52:18.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158003e10 con 0x7fd148003b50 2024-02-19T14:52:18.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc0e2600 con 0x7ff0dc0070f0 2024-02-19T14:52:18.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.943+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1600c1930 con 0x7fd148003b50 2024-02-19T14:52:18.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16014dbc0 con 0x7fd148003b50 2024-02-19T14:52:18.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc16a8e0 con 0x7ff0dc0070f0 2024-02-19T14:52:18.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158039290 con 0x7fd148003b50 2024-02-19T14:52:18.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7f29057fa640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28ec06b870 0x7f28ec06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f28fc040070 con 0x7f290004db10 2024-02-19T14:52:18.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7f29057fa640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f28ec074f60 0x7f28ec0773a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28ec077a70 con 0x7f28ec074f60 2024-02-19T14:52:18.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28fc0ac6c0 con 0x7f290004db10 2024-02-19T14:52:18.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.944+0000 7f290d60f640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f28ec074f60 0x7f28ec0773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.945+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd140001030 con 0x7fd148003b50 2024-02-19T14:52:18.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.945+0000 7f290d60f640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f28ec074f60 0x7f28ec0773a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.945+0000 7f2907fff640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28ec06b870 0x7f28ec06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 -- 172.21.15.78:0/1272568225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155070 msgr2=0x7f0f781554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 --2- 172.21.15.78:0/1272568225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155070 0x7f0f781554d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0097d0 tx=0x7f0f6c02b690 comp rx=0 tx=0).stop 2024-02-19T14:52:18.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 -- 172.21.15.78:0/1272568225 shutdown_connections 2024-02-19T14:52:18.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 --2- 172.21.15.78:0/1272568225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155070 0x7f0f781554d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 -- 172.21.15.78:0/1272568225 >> 172.21.15.78:0/1272568225 conn(0x7f0f780c7490 msgr2=0x7f0f780c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f2907fff640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28ec06b870 0x7f28ec06dd30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2908068580 tx=0x7f290808d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 -- 172.21.15.78:0/1272568225 shutdown_connections 2024-02-19T14:52:18.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== osd.2 v2:172.21.15.78:6800/3664023701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28ec077a70 con 0x7f28ec074f60 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.947+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd158005070 con 0x7fd148003b50 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.947+0000 7fd147fff640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14c06b910 0x7fd14c06ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.948+0000 7fd15effd640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14c06b910 0x7fd14c06ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7fd15effd640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14c06b910 0x7fd14c06ddd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd150006b10 tx=0x7fd150006a20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd1580ac090 con 0x7fd148003b50 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7fd147fff640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd14c075080 0x7fd14c0774c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd14c077b90 con 0x7fd14c075080 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1580ac3c0 con 0x7fd148003b50 2024-02-19T14:52:18.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7fd15ffff640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd14c075080 0x7fd14c0774c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.946+0000 7f0f77fff640 1 -- 172.21.15.78:0/1272568225 wait complete. 2024-02-19T14:52:18.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.948+0000 7f0f77fff640 1 Processor -- start 2024-02-19T14:52:18.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.948+0000 7f0f77fff640 1 -- start start 2024-02-19T14:52:18.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff0c4001030 con 0x7ff0dc0070f0 2024-02-19T14:52:18.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f77fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f7806c790 0x7f0f7813f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f78155070 0x7f0f78130f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f77fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155a10 0x7f0f78131440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f77fff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78149040 con 0x7f0f7806c790 2024-02-19T14:52:18.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f77fff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78152770 con 0x7f0f78155070 2024-02-19T14:52:18.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f77fff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78148290 con 0x7f0f78155a10 2024-02-19T14:52:18.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.949+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155a10 0x7f0f78131440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7fd15ffff640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd14c075080 0x7fd14c0774c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155a10 0x7f0f78131440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50920/0 (socket says 172.21.15.78:50920) 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 -- 172.21.15.78:0/713408469 learned_addr learned my addr 172.21.15.78:0/713408469 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f767fc640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f78155070 0x7f0f78130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f76ffd640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f7806c790 0x7f0f7813f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 -- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f78155070 msgr2=0x7f0f78130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f78155070 0x7f0f78130f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 -- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f7806c790 msgr2=0x7f0f7813f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f7806c790 0x7f0f7813f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f777fe640 1 -- 172.21.15.78:0/713408469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c009400 con 0x7f0f78155a10 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f767fc640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f78155070 0x7f0f78130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.950+0000 7f0f76ffd640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f7806c790 0x7f0f7813f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.951+0000 7f0f777fe640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155a10 0x7f0f78131440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c490 tx=0x7f0f7000c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.951+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70015e30 con 0x7f0f78155a10 2024-02-19T14:52:18.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.951+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f700050d0 con 0x7f0f78155a10 2024-02-19T14:52:18.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.952+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== osd.4 v2:172.21.15.150:6816/1744914063 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd14c077b90 con 0x7fd14c075080 2024-02-19T14:52:18.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.952+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70005460 con 0x7f0f78155a10 2024-02-19T14:52:18.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.952+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f781484c0 con 0x7f0f78155a10 2024-02-19T14:52:18.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.952+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f78148a80 con 0x7f0f78155a10 2024-02-19T14:52:18.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.953+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 --> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde88002c70 con 0x7fde8c075060 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.953+0000 7fdea57fa640 1 -- 172.21.15.78:0/2676233706 <== osd.0 v2:172.21.15.78:6816/2796609917 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde88002c70 con 0x7fde8c075060 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7fde8c075060 msgr2=0x7fde8c0774a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdeaf25c640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7fde8c075060 0x7fde8c0774a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde8c06b8e0 msgr2=0x7fde8c06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdeaf25c640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde8c06b8e0 0x7fde8c06dda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde94036000 tx=0x7fde94008030 comp rx=0 tx=0).stop 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdea00071a0 msgr2=0x7fdea00c9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdeaf25c640 1 --2- 172.21.15.78:0/2676233706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdea00071a0 0x7fdea00c9f20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdea80507a0 tx=0x7fdea80689b0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7fdead7f9640 1 -- 172.21.15.78:0/2676233706 reap_dead start 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 shutdown_connections 2024-02-19T14:52:18.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 >> 172.21.15.78:0/2676233706 conn(0x7fdea004c650 msgr2=0x7fdea00e1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0f700035f0 con 0x7f0f78155a10 2024-02-19T14:52:18.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f7806f780 con 0x7f0f78155a10 2024-02-19T14:52:18.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7f0f5ffff640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0f6406b810 0x7f0f6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.953+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff0d401a400 con 0x7ff0dc0070f0 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0f700923d0 con 0x7f0f78155a10 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7f0f5ffff640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f0f64074e90 0x7f0f640772d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 --> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f640779a0 con 0x7f0f64074e90 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7f0f767fc640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f0f64074e90 0x7f0f640772d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f70007e70 con 0x7f0f78155a10 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.956+0000 7f0f76ffd640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0f6406b810 0x7f0f6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.956+0000 7f0f767fc640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f0f64074e90 0x7f0f640772d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.956+0000 7f0f76ffd640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0f6406b810 0x7f0f6406dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0f7808ac80 tx=0x7f0f68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.953+0000 7ff0cbfff640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0d006b8e0 0x7ff0d006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.953+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff0d408a4d0 con 0x7ff0dc0070f0 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7ff0cbfff640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff0d0075050 0x7ff0d0077490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0d0077b60 con 0x7ff0d0075050 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0d408caf0 con 0x7ff0dc0070f0 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7ff0e27fc640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0d006b8e0 0x7ff0d006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.954+0000 7ff0e37fe640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff0d0075050 0x7ff0d0077490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7ff0e27fc640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0d006b8e0 0x7ff0d006dda0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40504c0 tx=0x7ff0e4068c50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.955+0000 7ff0e37fe640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff0d0075050 0x7ff0d0077490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.957+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== osd.6 v2:172.21.15.150:6808/3532732509 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff0d0077b60 con 0x7ff0d0075050 2024-02-19T14:52:18.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.957+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 --> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4d04002c70 con 0x7f4d10074f10 2024-02-19T14:52:18.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.957+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== osd.1 v2:172.21.15.78:6802/289106328 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0f640779a0 con 0x7f0f64074e90 2024-02-19T14:52:18.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.961+0000 7f4d20ff9640 1 -- 172.21.15.78:0/2439564419 <== osd.5 v2:172.21.15.150:6800/911852656 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4d04002c70 con 0x7f4d10074f10 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.961+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7f4d10074f10 msgr2=0x7f4d10077350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.961+0000 7f4d2aa39640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7f4d10074f10 0x7f4d10077350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1006b910 msgr2=0x7f4d1006ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d2aa39640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1006b910 0x7f4d1006ddd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4d14001640 tx=0x7f4d14009040 comp rx=0 tx=0).stop 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d240c47e0 msgr2=0x7f4d24153a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d2aa39640 1 --2- 172.21.15.78:0/2439564419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d240c47e0 0x7f4d24153a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c001f50 tx=0x7f4d1c00a430 comp rx=0 tx=0).stop 2024-02-19T14:52:18.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 shutdown_connections 2024-02-19T14:52:18.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d28fd6640 1 -- 172.21.15.78:0/2439564419 reap_dead start 2024-02-19T14:52:18.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7fdeaf25c640 1 -- 172.21.15.78:0/2676233706 wait complete. 2024-02-19T14:52:18.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 shutdown_connections 2024-02-19T14:52:18.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.962+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 >> 172.21.15.78:0/2439564419 conn(0x7f4d2406b270 msgr2=0x7f4d240c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:18.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.964+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 shutdown_connections 2024-02-19T14:52:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.964+0000 7f4d2aa39640 1 -- 172.21.15.78:0/2439564419 wait complete. 2024-02-19T14:52:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.964+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 --> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61f0005c90 con 0x7f61f0001650 2024-02-19T14:52:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.965+0000 7f61f7fff640 1 -- 172.21.15.78:0/381679873 <== osd.7 v2:172.21.15.150:6819/1386990918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f61f0005c90 con 0x7f61f0001650 2024-02-19T14:52:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.966+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f61f0001650 msgr2=0x7f61f0003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.966+0000 7f6215a8f640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f61f0001650 0x7f61f0003b10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.968+0000 7fa3a7fff640 1 -- 172.21.15.78:0/1541744532 --> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa388002d30 con 0x7fa394074fd0 2024-02-19T14:52:19.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61fc06b8e0 msgr2=0x7f61fc06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61fc06b8e0 0x7f61fc06dda0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6204034000 tx=0x7f6204008030 comp rx=0 tx=0).stop 2024-02-19T14:52:19.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7fa38f7fe640 1 -- 172.21.15.78:0/1541744532 <== osd.3 v2:172.21.15.78:6824/3716944111 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa388002d30 con 0x7fa394074fd0 2024-02-19T14:52:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62100c47e0 msgr2=0x7f62101e2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 --2- 172.21.15.78:0/381679873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62100c47e0 0x7f62101e2840 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f620000c780 tx=0x7f620000cc40 comp rx=0 tx=0).stop 2024-02-19T14:52:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f620ffff640 1 -- 172.21.15.78:0/381679873 reap_dead start 2024-02-19T14:52:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 shutdown_connections 2024-02-19T14:52:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 >> 172.21.15.78:0/381679873 conn(0x7f621006b270 msgr2=0x7f621014f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.969+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 shutdown_connections 2024-02-19T14:52:19.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.972+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7fa394074fd0 msgr2=0x7fa394077410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.972+0000 7fa38d7fa640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7fa394074fd0 0x7fa394077410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.973+0000 7f6215a8f640 1 -- 172.21.15.78:0/381679873 wait complete. 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.975+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa39406b8e0 msgr2=0x7fa39406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.975+0000 7fa38d7fa640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa39406b8e0 0x7fa39406dda0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa39803f000 tx=0x7fa398038000 comp rx=0 tx=0).stop 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.975+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 msgr2=0x7fa3a0040cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.975+0000 7fa38d7fa640 1 --2- 172.21.15.78:0/1541744532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a00d6fa0 0x7fa3a0040cf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa3a806a1b0 tx=0x7fa3a80689b0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.976+0000 7fa3a77fe640 1 -- 172.21.15.78:0/1541744532 reap_dead start 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.976+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 shutdown_connections 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.976+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 >> 172.21.15.78:0/1541744532 conn(0x7fa3a0005e90 msgr2=0x7fa3a00d2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.976+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 shutdown_connections 2024-02-19T14:52:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.976+0000 7fa38d7fa640 1 -- 172.21.15.78:0/1541744532 wait complete. 2024-02-19T14:52:19.004 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:19.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-19T14:52:19.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.979+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 --> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f290004e4e0 con 0x7f28ec074f60 2024-02-19T14:52:19.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.979+0000 7f29057fa640 1 -- 172.21.15.78:0/2617691543 <== osd.2 v2:172.21.15.78:6800/3664023701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f28ec077a70 con 0x7f28ec074f60 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f28ec074f60 msgr2=0x7f28ec0773a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f28ec074f60 0x7f28ec0773a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28ec06b870 msgr2=0x7f28ec06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28ec06b870 0x7f28ec06dd30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2908068580 tx=0x7f290808d040 comp rx=0 tx=0).stop 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f290004db10 msgr2=0x7f2900043740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 --2- 172.21.15.78:0/2617691543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f290004db10 0x7f2900043740 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28fc033000 tx=0x7f28fc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290d60f640 1 -- 172.21.15.78:0/2617691543 reap_dead start 2024-02-19T14:52:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 shutdown_connections 2024-02-19T14:52:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 >> 172.21.15.78:0/2617691543 conn(0x7f290004c690 msgr2=0x7f290004e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.980+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 shutdown_connections 2024-02-19T14:52:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.981+0000 7f290f072640 1 -- 172.21.15.78:0/2617691543 wait complete. 2024-02-19T14:52:19.007 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:19.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-19T14:52:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.983+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 --> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f7806f4a0 con 0x7f0f64074e90 2024-02-19T14:52:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.984+0000 7f0f5ffff640 1 -- 172.21.15.78:0/713408469 <== osd.1 v2:172.21.15.78:6802/289106328 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f6c02be90 con 0x7f0f64074e90 2024-02-19T14:52:19.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.984+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f0f64074e90 msgr2=0x7f0f640772d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.984+0000 7f0f77fff640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f0f64074e90 0x7f0f640772d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0f6406b810 msgr2=0x7f0f6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0f6406b810 0x7f0f6406dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0f7808ac80 tx=0x7f0f68009040 comp rx=0 tx=0).stop 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155a10 msgr2=0x7f0f78131440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 --2- 172.21.15.78:0/713408469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f78155a10 0x7f0f78131440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c490 tx=0x7f0f7000c950 comp rx=0 tx=0).stop 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f777fe640 1 -- 172.21.15.78:0/713408469 reap_dead start 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 shutdown_connections 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 >> 172.21.15.78:0/713408469 conn(0x7f0f780c7490 msgr2=0x7f0f780c1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 shutdown_connections 2024-02-19T14:52:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.985+0000 7f0f77fff640 1 -- 172.21.15.78:0/713408469 wait complete. 2024-02-19T14:52:19.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.988+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 --> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd140002d30 con 0x7fd14c075080 2024-02-19T14:52:19.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.989+0000 7fd147fff640 1 -- 172.21.15.78:0/3771036465 <== osd.4 v2:172.21.15.150:6816/1744914063 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd140002d30 con 0x7fd14c075080 2024-02-19T14:52:19.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.990+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 --> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff0c4002d30 con 0x7ff0d0075050 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.990+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd14c075080 msgr2=0x7fd14c0774c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.990+0000 7fd164df3640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd14c075080 0x7fd14c0774c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14c06b910 msgr2=0x7fd14c06ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14c06b910 0x7fd14c06ddd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd150006b10 tx=0x7fd150006a20 comp rx=0 tx=0).stop 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd148003b50 msgr2=0x7fd1601ea920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 --2- 172.21.15.78:0/3771036465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd148003b50 0x7fd1601ea920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd158001fc0 tx=0x7fd158001580 comp rx=0 tx=0).stop 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd15ffff640 1 -- 172.21.15.78:0/3771036465 reap_dead start 2024-02-19T14:52:19.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 shutdown_connections 2024-02-19T14:52:19.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 >> 172.21.15.78:0/3771036465 conn(0x7fd1600ac270 msgr2=0x7fd1600c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 shutdown_connections 2024-02-19T14:52:19.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7fd164df3640 1 -- 172.21.15.78:0/3771036465 wait complete. 2024-02-19T14:52:19.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.991+0000 7ff0cbfff640 1 -- 172.21.15.78:0/3737506923 <== osd.6 v2:172.21.15.150:6808/3532732509 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff0c4002d30 con 0x7ff0d0075050 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff0d0075050 msgr2=0x7ff0d0077490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff0d0075050 0x7ff0d0077490 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0d006b8e0 msgr2=0x7ff0d006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0d006b8e0 0x7ff0d006dda0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40504c0 tx=0x7ff0e4068c50 comp rx=0 tx=0).stop 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0dc0070f0 msgr2=0x7ff0dc0d6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 --2- 172.21.15.78:0/3737506923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0dc0070f0 0x7ff0dc0d6de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0d400b3e0 tx=0x7ff0d400b8a0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e37fe640 1 -- 172.21.15.78:0/3737506923 reap_dead start 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 shutdown_connections 2024-02-19T14:52:19.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.992+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 >> 172.21.15.78:0/3737506923 conn(0x7ff0dc005ec0 msgr2=0x7ff0dc02e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.993+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 shutdown_connections 2024-02-19T14:52:19.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:18.993+0000 7ff0e987b640 1 -- 172.21.15.78:0/3737506923 wait complete. 2024-02-19T14:52:19.015 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:19.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-19T14:52:19.016 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:19.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-19T14:52:19.017 INFO:teuthology.orchestra.run.smithi078.stdout:55834574859 2024-02-19T14:52:19.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-19T14:52:19.018 INFO:teuthology.orchestra.run.smithi078.stdout:55834574859 2024-02-19T14:52:19.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-19T14:52:19.019 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:19.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-19T14:52:19.020 INFO:teuthology.orchestra.run.smithi078.stdout:55834574857 2024-02-19T14:52:19.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-19T14:52:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.223+0000 7fb6f5072640 1 Processor -- start 2024-02-19T14:52:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.223+0000 7fb6f5072640 1 -- start start 2024-02-19T14:52:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.224+0000 7fb6f5072640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6e800b100 0x7fb6e804aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.224+0000 7fb6f5072640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e8047690 con 0x7fb6e80d3be0 2024-02-19T14:52:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.224+0000 7fb6f5072640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e80d6600 con 0x7fb6e80d71a0 2024-02-19T14:52:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.224+0000 7fb6f5072640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e80465b0 con 0x7fb6e800b100 2024-02-19T14:52:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb6ee59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb6e80d3be0 0x7fb6e80d6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57724/0 (socket says 172.21.15.78:57724) 2024-02-19T14:52:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb6ee59c640 1 -- 172.21.15.78:0/1568856547 learned_addr learned my addr 172.21.15.78:0/1568856547 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb057dbc640 1 Processor -- start 2024-02-19T14:52:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7f3b3a042640 1 Processor -- start 2024-02-19T14:52:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb6eed9d640 1 --2- 172.21.15.78:0/1568856547 >> v2:172.21.15.150:3300/0 conn(0x7fb6e800b100 0x7fb6e804aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb057dbc640 1 -- start start 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695236644 0 0) 0x7fb6e4001110 con 0x7fb6e80d3be0 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d8003520 con 0x7fb6e80d3be0 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.225+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146418829 0 0) 0x7fb6e80d6600 con 0x7fb6e80d71a0 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.226+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e4001110 con 0x7fb6e80d71a0 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.226+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659206596 0 0) 0x7fb6d8003520 con 0x7fb6e80d3be0 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.226+0000 7fb6eed9d640 1 -- 172.21.15.78:0/1568856547 >> v1:172.21.15.150:6789/0 conn(0x7fb6e80d71a0 legacy=0x7fb6e80dacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.226+0000 7fb6eed9d640 1 -- 172.21.15.78:0/1568856547 >> v1:172.21.15.78:6789/0 conn(0x7fb6e80d3be0 legacy=0x7fb6e80d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.226+0000 7fb6eed9d640 1 -- 172.21.15.78:0/1568856547 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8046900 con 0x7fb6e800b100 2024-02-19T14:52:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.227+0000 7fb6eed9d640 1 --2- 172.21.15.78:0/1568856547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e800b100 0x7fb6e804aca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00649e0 tx=0x7fb6f00a6890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d534f8e1bcd579d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.227+0000 7f3b3a042640 1 -- start start 2024-02-19T14:52:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.226+0000 7fb057dbc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb0500c6f00 0x7fb05014e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00b1070 con 0x7fb6e800b100 2024-02-19T14:52:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.227+0000 7fb055b58640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb050070e90 0x7fb05006e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50302/0 (socket says 172.21.15.78:50302) 2024-02-19T14:52:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f00ac630 con 0x7fb6e800b100 2024-02-19T14:52:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb6edd9b640 1 -- 172.21.15.78:0/1568856547 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4001800 con 0x7fb6e800b100 2024-02-19T14:52:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3a042640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3414e460 0x7f3b3406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3a042640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b3406b3c0 0x7f3b34154c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3a042640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340c2040 con 0x7f3b340c1580 2024-02-19T14:52:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3a042640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b34072d80 con 0x7f3b3406b3c0 2024-02-19T14:52:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3a042640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340719d0 con 0x7f3b3414e460 2024-02-19T14:52:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3883f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3b340c1580 0x7f3b34154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57738/0 (socket says 172.21.15.78:57738) 2024-02-19T14:52:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b3883f640 1 -- 172.21.15.78:0/2494854933 learned_addr learned my addr 172.21.15.78:0/2494854933 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7f3b39841640 1 --2- 172.21.15.78:0/2494854933 >> v2:172.21.15.150:3300/0 conn(0x7f3b3406b3c0 0x7f3b34154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7f3b39040640 1 --2- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3414e460 0x7f3b3406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb6f5072640 1 -- 172.21.15.78:0/1568856547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e800b100 msgr2=0x7fb6e804aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb6f5072640 1 --2- 172.21.15.78:0/1568856547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e800b100 0x7fb6e804aca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00649e0 tx=0x7fb6f00a6890 comp rx=0 tx=0).stop 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb6f5072640 1 -- 172.21.15.78:0/1568856547 shutdown_connections 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb6f5072640 1 --2- 172.21.15.78:0/1568856547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e800b100 0x7fb6e804aca0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb6f5072640 1 -- 172.21.15.78:0/1568856547 >> 172.21.15.78:0/1568856547 conn(0x7fb6e8005ec0 msgr2=0x7fb6e80062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb6f5072640 1 -- 172.21.15.78:0/1568856547 shutdown_connections 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7f3b39841640 1 -- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3414e460 msgr2=0x7f3b3406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7f3b39841640 1 --2- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3414e460 0x7f3b3406ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7f3b39841640 1 -- 172.21.15.78:0/2494854933 >> v1:172.21.15.78:6789/0 conn(0x7f3b340c1580 legacy=0x7f3b34154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7f3b39841640 1 -- 172.21.15.78:0/2494854933 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34071cb0 con 0x7f3b3406b3c0 2024-02-19T14:52:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb057dbc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05006ae80 con 0x7fb0500712b0 2024-02-19T14:52:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb055b58640 1 -- 172.21.15.78:0/4198303233 learned_addr learned my addr 172.21.15.78:0/4198303233 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb050071c10 con 0x7fb050070e90 2024-02-19T14:52:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0500724b0 con 0x7fb0500c6f00 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb055357640 1 --2- 172.21.15.78:0/4198303233 >> v2:172.21.15.78:3300/0 conn(0x7fb0500c6f00 0x7fb05014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397363749 0 0) 0x7fb050071c10 con 0x7fb050070e90 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.228+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb03c003520 con 0x7fb050070e90 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375959655 0 0) 0x7fb03c003520 con 0x7fb050070e90 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 >> v2:172.21.15.78:3300/0 conn(0x7fb0500c6f00 msgr2=0x7fb05014e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 --2- 172.21.15.78:0/4198303233 >> v2:172.21.15.78:3300/0 conn(0x7fb0500c6f00 0x7fb05014e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 >> v1:172.21.15.78:6789/0 conn(0x7fb0500712b0 legacy=0x7fb050151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb050071ef0 con 0x7fb050070e90 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb040001670 con 0x7fb050070e90 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 >> v1:172.21.15.150:6789/0 conn(0x7fb050070e90 legacy=0x7fb05006e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.229+0000 7fb054b56640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb03c003970 0x7fb03c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb055b58640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb03c003970 0x7fb03c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb054b56640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c0047c0 0x7fb03c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb054b56640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb03c0052a0 0x7fb03c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500724b0 con 0x7fb03c0052a0 2024-02-19T14:52:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c00c1f0 con 0x7fb03c0047c0 2024-02-19T14:52:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c00c3b0 con 0x7fb03c003970 2024-02-19T14:52:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.230+0000 7fb6f5072640 1 -- 172.21.15.78:0/1568856547 wait complete. 2024-02-19T14:52:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.231+0000 7f3b39841640 1 --2- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3406b3c0 0x7f3b34154c20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009d90 tx=0x7f3b2c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5645a22d121f4719 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.231+0000 7f3b2bfff640 1 -- 172.21.15.78:0/2494854933 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c02cc50 con 0x7f3b3406b3c0 2024-02-19T14:52:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.231+0000 7f3b2bfff640 1 -- 172.21.15.78:0/2494854933 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c02cdb0 con 0x7f3b3406b3c0 2024-02-19T14:52:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.231+0000 7f3b2bfff640 1 -- 172.21.15.78:0/2494854933 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c034660 con 0x7f3b3406b3c0 2024-02-19T14:52:19.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb056359640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb03c0052a0 0x7fb03c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb6f5072640 1 Processor -- start 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f3b3a042640 1 -- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3406b3c0 msgr2=0x7f3b34154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3406b3c0 0x7f3b34154c20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009d90 tx=0x7f3b2c02bea0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f3b3a042640 1 -- 172.21.15.78:0/2494854933 shutdown_connections 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3406b3c0 0x7f3b34154c20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2494854933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3414e460 0x7f3b3406ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f3b3a042640 1 -- 172.21.15.78:0/2494854933 >> 172.21.15.78:0/2494854933 conn(0x7f3b340c6410 msgr2=0x7f3b340c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7f3b3a042640 1 -- 172.21.15.78:0/2494854933 shutdown_connections 2024-02-19T14:52:19.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7f3b3a042640 1 -- 172.21.15.78:0/2494854933 wait complete. 2024-02-19T14:52:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f6cac90a640 1 Processor -- start 2024-02-19T14:52:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7f6cac90a640 1 -- start start 2024-02-19T14:52:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb6f5072640 1 -- start start 2024-02-19T14:52:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb055b58640 1 -- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c0047c0 msgr2=0x7fb03c004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-19T14:52:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6f5072640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e800b100 0x7fb6e8049b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6f5072640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 0x7fb6e80e8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6f5072640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 0x7fb6e80e8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6f5072640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e8005760 con 0x7fb6e80d3be0 2024-02-19T14:52:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6f5072640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e817d1c0 con 0x7fb6e80d71a0 2024-02-19T14:52:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6f5072640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80d9ad0 con 0x7fb6e800b100 2024-02-19T14:52:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb055b58640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c0047c0 0x7fb03c004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb055b58640 1 -- 172.21.15.78:0/4198303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb03c0052a0 msgr2=0x7fb03c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb055b58640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb03c0052a0 0x7fb03c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb055b58640 1 -- 172.21.15.78:0/4198303233 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c003520 con 0x7fb03c003970 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.232+0000 7fb056359640 1 -- 172.21.15.78:0/4198303233 reap_dead start 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb055b58640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb03c003970 0x7fb03c003d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb040001eb0 tx=0x7fb03c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97a03e2ed4266da9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6ee59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 0x7fb6e80e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6ef59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 0x7fb6e80e8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6ee59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 0x7fb6e80e8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52450/0 (socket says 172.21.15.78:52450) 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6ee59c640 1 -- 172.21.15.78:0/965672847 learned_addr learned my addr 172.21.15.78:0/965672847 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.233+0000 7fb6ef59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 0x7fb6e80e8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45814/0 (socket says 172.21.15.78:45814) 2024-02-19T14:52:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040030dd0 con 0x7fb03c003970 2024-02-19T14:52:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400058d0 con 0x7fb03c003970 2024-02-19T14:52:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb054b56640 1 -- 172.21.15.78:0/4198303233 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040005ce0 con 0x7fb03c003970 2024-02-19T14:52:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7f6cac90a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ca80c2270 0x7f6ca8155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ef59e640 1 -- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e800b100 msgr2=0x7fb6e8049b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:52:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ef59e640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e800b100 0x7fb6e8049b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ef59e640 1 -- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 msgr2=0x7fb6e80e8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ef59e640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 0x7fb6e80e8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ef59e640 1 -- 172.21.15.78:0/965672847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e80d9db0 con 0x7fb6e80d71a0 2024-02-19T14:52:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ee59c640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 0x7fb6e80e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7f6ca77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6ca80c1e90 0x7f6ca806c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50314/0 (socket says 172.21.15.78:50314) 2024-02-19T14:52:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7f6c9ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ca80c2270 0x7f6ca8155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7f6c9ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ca80c2270 0x7f6ca8155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52466/0 (socket says 172.21.15.78:52466) 2024-02-19T14:52:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7fb6ef59e640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 0x7fb6e80e8940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c390 tx=0x7fb6e0001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7f6cac90a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca815c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.234+0000 7f6c9ffff640 1 -- 172.21.15.78:0/20213739 learned_addr learned my addr 172.21.15.78:0/20213739 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00052d0 con 0x7fb6e80d71a0 2024-02-19T14:52:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca806faf0 con 0x7f6ca80c1e90 2024-02-19T14:52:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80c26d0 con 0x7f6ca80c2270 2024-02-19T14:52:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80c1b60 con 0x7f6ca81555b0 2024-02-19T14:52:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f3b3a042640 1 Processor -- start 2024-02-19T14:52:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f3b3a042640 1 -- start start 2024-02-19T14:52:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e817d860 con 0x7fb6e80d71a0 2024-02-19T14:52:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e802c4f0 con 0x7fb6e80d71a0 2024-02-19T14:52:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e0005430 con 0x7fb6e80d71a0 2024-02-19T14:52:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0003830 con 0x7fb6e80d71a0 2024-02-19T14:52:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6c9ffff640 1 -- 172.21.15.78:0/20213739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 msgr2=0x7f6ca815c430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-19T14:52:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb03c003970 msgr2=0x7fb03c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6c9ffff640 1 --2- 172.21.15.78:0/20213739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca815c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6c9ffff640 1 -- 172.21.15.78:0/20213739 >> v1:172.21.15.150:6789/0 conn(0x7f6ca80c1e90 legacy=0x7f6ca806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7f6c9ffff640 1 -- 172.21.15.78:0/20213739 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca806eee0 con 0x7f6ca80c2270 2024-02-19T14:52:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.235+0000 7fb057dbc640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb03c003970 0x7fb03c003d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb040001eb0 tx=0x7fb03c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.236+0000 7f3b3a042640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3406b3c0 0x7f3b34152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.236+0000 7f3b3a042640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 0x7f3b340a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.236+0000 7f6c9ffff640 1 --2- 172.21.15.78:0/20213739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 0x7f6ca8155070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009870 tx=0x7f6c9002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78509e168834f9a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b3a042640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 0x7f3b341e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.236+0000 7f6ca6ffd640 1 -- 172.21.15.78:0/20213739 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900317e0 con 0x7f6ca80c2270 2024-02-19T14:52:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 shutdown_connections 2024-02-19T14:52:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7fb057dbc640 1 --2- 172.21.15.78:0/4198303233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb03c003970 0x7fb03c003d70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 >> 172.21.15.78:0/4198303233 conn(0x7fb0500c6410 msgr2=0x7fb0500c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f6ca6ffd640 1 -- 172.21.15.78:0/20213739 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c90031940 con 0x7f6ca80c2270 2024-02-19T14:52:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f6ca6ffd640 1 -- 172.21.15.78:0/20213739 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90031cd0 con 0x7f6ca80c2270 2024-02-19T14:52:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b39841640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 0x7f3b341e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b39841640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 0x7f3b341e6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52482/0 (socket says 172.21.15.78:52482) 2024-02-19T14:52:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b3883f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 0x7f3b340a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b3883f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 0x7f3b340a9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45822/0 (socket says 172.21.15.78:45822) 2024-02-19T14:52:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b39841640 1 -- 172.21.15.78:0/2030877653 learned_addr learned my addr 172.21.15.78:0/2030877653 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340aa8d0 con 0x7f3b3414e460 2024-02-19T14:52:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3414acb0 con 0x7f3b340c1580 2024-02-19T14:52:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3414b3a0 con 0x7f3b3406b3c0 2024-02-19T14:52:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 shutdown_connections 2024-02-19T14:52:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7f3b39841640 1 -- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3406b3c0 msgr2=0x7f3b34152080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:52:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7f3b39841640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3406b3c0 0x7f3b34152080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7f3b39841640 1 -- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 msgr2=0x7f3b340a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7f3b39841640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 0x7f3b340a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7f3b39841640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009a70 con 0x7f3b3414e460 2024-02-19T14:52:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.237+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb6e0003990 con 0x7fb6e80d71a0 2024-02-19T14:52:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7f3b3883f640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 0x7f3b340a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7fb6d77fe640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6dc06b8e0 0x7fb6dc06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d0005290 con 0x7fb6e80d71a0 2024-02-19T14:52:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb6e00898f0 con 0x7fb6e80d71a0 2024-02-19T14:52:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.239+0000 7fb6eed9d640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6dc06b8e0 0x7fb6dc06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.238+0000 7fb057dbc640 1 -- 172.21.15.78:0/4198303233 wait complete. 2024-02-19T14:52:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.239+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 msgr2=0x7f6ca8155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.239+0000 7f6cac90a640 1 --2- 172.21.15.78:0/20213739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 0x7f6ca8155070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009870 tx=0x7f6c9002b880 comp rx=0 tx=0).stop 2024-02-19T14:52:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.239+0000 7f62ad1d7640 1 Processor -- start 2024-02-19T14:52:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.239+0000 7fb6eed9d640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6dc06b8e0 0x7fb6dc06dda0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8008590 tx=0x7fb6f00ac000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.239+0000 7f62ad1d7640 1 -- start start 2024-02-19T14:52:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7fb057dbc640 1 Processor -- start 2024-02-19T14:52:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7fb057dbc640 1 -- start start 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 shutdown_connections 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f6cac90a640 1 --2- 172.21.15.78:0/20213739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca815c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f6cac90a640 1 --2- 172.21.15.78:0/20213739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 0x7f6ca8155070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 >> 172.21.15.78:0/20213739 conn(0x7f6ca80c7490 msgr2=0x7f6ca80c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 shutdown_connections 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f6cac90a640 1 -- 172.21.15.78:0/20213739 wait complete. 2024-02-19T14:52:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f3b39841640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 0x7f3b341e6e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009d90 tx=0x7f3b2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c03e610 con 0x7f3b3414e460 2024-02-19T14:52:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c035e90 con 0x7f3b3414e460 2024-02-19T14:52:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f62ad1d7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62a80c1ef0 0x7f62a814b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.240+0000 7f62ad1d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62a8151d20 0x7f62a8066c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f62ad1d7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a8152290 con 0x7f62a80a84c0 2024-02-19T14:52:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f62ad1d7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a806da10 con 0x7f62a80c1ef0 2024-02-19T14:52:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f62ad1d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a806dcf0 con 0x7f62a8151d20 2024-02-19T14:52:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62a8151d20 0x7f62a8066c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7fb057dbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c003970 0x7fb0500aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62a8151d20 0x7f62a8066c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45832/0 (socket says 172.21.15.78:45832) 2024-02-19T14:52:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f62a6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62a80c1ef0 0x7f62a814b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 -- 172.21.15.78:0/3758674367 learned_addr learned my addr 172.21.15.78:0/3758674367 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 -- 172.21.15.78:0/3758674367 >> v2:172.21.15.78:3300/0 conn(0x7f62a80c1ef0 msgr2=0x7f62a814b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 --2- 172.21.15.78:0/3758674367 >> v2:172.21.15.78:3300/0 conn(0x7f62a80c1ef0 0x7f62a814b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 -- 172.21.15.78:0/3758674367 >> v1:172.21.15.78:6789/0 conn(0x7f62a80a84c0 legacy=0x7f62a806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f629ffff640 1 -- 172.21.15.78:0/3758674367 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a814b7a0 con 0x7f62a8151d20 2024-02-19T14:52:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7fb055b58640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c003970 0x7fb0500aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7fb055b58640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c003970 0x7fb0500aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45848/0 (socket says 172.21.15.78:45848) 2024-02-19T14:52:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7fb057dbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0500c0690 0x7fb0501f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c03e990 con 0x7f3b3414e460 2024-02-19T14:52:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3414af00 con 0x7f3b3414e460 2024-02-19T14:52:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.241+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b340aa2a0 con 0x7f3b3414e460 2024-02-19T14:52:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.242+0000 7f6cac90a640 1 Processor -- start 2024-02-19T14:52:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.242+0000 7fb055357640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0500c0690 0x7fb0501f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.242+0000 7fb055357640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0500c0690 0x7fb0501f3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52494/0 (socket says 172.21.15.78:52494) 2024-02-19T14:52:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.242+0000 7f6cac90a640 1 -- start start 2024-02-19T14:52:19.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.242+0000 7fb055b58640 1 -- 172.21.15.78:0/3527997989 learned_addr learned my addr 172.21.15.78:0/3527997989 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.242+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0501f43e0 0x7fb0501fa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0501f4860 con 0x7fb0500c0690 2024-02-19T14:52:19.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7f629ffff640 1 --2- 172.21.15.78:0/3758674367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a8151d20 0x7f62a8066c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62a8071870 tx=0x7f629802b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172bbe2994f48dea server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0501f7b20 con 0x7fb03c003970 2024-02-19T14:52:19.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7fb056359640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0501f43e0 0x7fb0501fa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b341e6b70 con 0x7f3b3414e460 2024-02-19T14:52:19.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0501f6ea0 con 0x7fb0501f43e0 2024-02-19T14:52:19.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7f66269c7640 1 Processor -- start 2024-02-19T14:52:19.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7f66269c7640 1 -- start start 2024-02-19T14:52:19.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.243+0000 7f6cac90a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 0x7f6ca81e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7fb056359640 1 -- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c003970 msgr2=0x7fb0500aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7fb056359640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c003970 0x7fb0500aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7fb056359640 1 -- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0500c0690 msgr2=0x7fb0501f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7fb056359640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0500c0690 0x7fb0501f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7fb056359640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0501f7180 con 0x7fb0501f43e0 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6cac90a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 0x7f6ca81487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6ca77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 0x7f6ca81e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6ca77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 0x7f6ca81e9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45854/0 (socket says 172.21.15.78:45854) 2024-02-19T14:52:19.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6ca77fe640 1 -- 172.21.15.78:0/4277432453 learned_addr learned my addr 172.21.15.78:0/4277432453 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f62a659c640 1 -- 172.21.15.78:0/3758674367 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62980317f0 con 0x7f62a8151d20 2024-02-19T14:52:19.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f62a659c640 1 -- 172.21.15.78:0/3758674367 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6298031950 con 0x7f62a8151d20 2024-02-19T14:52:19.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f62a659c640 1 -- 172.21.15.78:0/3758674367 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298031ce0 con 0x7f62a8151d20 2024-02-19T14:52:19.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6cac90a640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca8146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca8143f80 con 0x7f6ca80c2270 2024-02-19T14:52:19.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7fb056359640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0501f43e0 0x7fb0501fa420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0500c7490 tx=0x7fb04800ea00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6ca7fff640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca8146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81ea580 con 0x7f6ca80c1e90 2024-02-19T14:52:19.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.244+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81435e0 con 0x7f6ca81555b0 2024-02-19T14:52:19.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3758674367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a8151d20 msgr2=0x7f62a8066c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3758674367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a8151d20 0x7f62a8066c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62a8071870 tx=0x7f629802b890 comp rx=0 tx=0).stop 2024-02-19T14:52:19.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb6e0054040 con 0x7fb6e80d71a0 2024-02-19T14:52:19.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6ca77fe640 1 -- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 msgr2=0x7f6ca8146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6ca77fe640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca8146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6ca77fe640 1 -- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 msgr2=0x7f6ca81487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6ca77fe640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 0x7f6ca81487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6ca77fe640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900094d0 con 0x7f6ca80c1e90 2024-02-19T14:52:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f66269c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 0x7f66200c3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f66269c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f662015ca30 0x7f662015cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f66269c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66200c19d0 con 0x7f6620158df0 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f66269c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66200c4740 con 0x7f66200c32c0 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f66269c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200c4a20 con 0x7f662015ca30 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f661f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 0x7f66200c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f661f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 0x7f66200c3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45860/0 (socket says 172.21.15.78:45860) 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f661f7fe640 1 -- 172.21.15.78:0/3768635007 learned_addr learned my addr 172.21.15.78:0/3768635007 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6624f64640 1 --2- 172.21.15.78:0/3768635007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f662015ca30 0x7f662015cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3758674367 shutdown_connections 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3758674367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a8151d20 0x7f62a8066c90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3758674367 >> v2:172.21.15.78:3300/0 conn(0x7f62a80c1ef0 0x7f62a814b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3758674367 >> 172.21.15.78:0/3758674367 conn(0x7f62a80718e0 msgr2=0x7f62a8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3758674367 shutdown_connections 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3758674367 wait complete. 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048017070 con 0x7fb0501f43e0 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb048003b80 con 0x7fb0501f43e0 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048003d50 con 0x7fb0501f43e0 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0501f7460 con 0x7fb0501f43e0 2024-02-19T14:52:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0500ab1d0 con 0x7fb0501f43e0 2024-02-19T14:52:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.245+0000 7f6ca77fe640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 0x7f6ca81e9f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b410 tx=0x7f6c9800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6624f64640 1 -- 172.21.15.78:0/3768635007 >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 msgr2=0x7f66200c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6624f64640 1 --2- 172.21.15.78:0/3768635007 >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 0x7f66200c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6624f64640 1 -- 172.21.15.78:0/3768635007 >> v1:172.21.15.150:6789/0 conn(0x7f6620158df0 legacy=0x7f66200c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98011830 con 0x7f6ca80c1e90 2024-02-19T14:52:19.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98011990 con 0x7f6ca80c1e90 2024-02-19T14:52:19.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98011dc0 con 0x7f6ca80c1e90 2024-02-19T14:52:19.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca81438c0 con 0x7f6ca80c1e90 2024-02-19T14:52:19.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3b2c03b050 con 0x7f3b3414e460 2024-02-19T14:52:19.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.246+0000 7f6624f64640 1 -- 172.21.15.78:0/3768635007 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66200c4d00 con 0x7f662015ca30 2024-02-19T14:52:19.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.247+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca8148d10 con 0x7f6ca80c1e90 2024-02-19T14:52:19.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.247+0000 7f3b29ffb640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b1806b8e0 0x7f3b1806dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f62ad1d7640 1 Processor -- start 2024-02-19T14:52:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.247+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3b2c0ac920 con 0x7f3b3414e460 2024-02-19T14:52:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f3b39040640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b1806b8e0 0x7f3b1806dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.247+0000 7f661f7fe640 1 --2- 172.21.15.78:0/3768635007 >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 0x7f66200c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f6624f64640 1 --2- 172.21.15.78:0/3768635007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f662015ca30 0x7f662015cec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66100097a0 tx=0x7f661002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d42060d1e6cc4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb0480050d0 con 0x7fb0501f43e0 2024-02-19T14:52:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0501fa140 con 0x7fb0501f43e0 2024-02-19T14:52:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f3b39040640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b1806b8e0 0x7f3b1806dda0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3b300098c0 tx=0x7f3b30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7fcd9b979640 1 Processor -- start 2024-02-19T14:52:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f661effd640 1 -- 172.21.15.78:0/3768635007 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66100317c0 con 0x7f662015ca30 2024-02-19T14:52:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f661effd640 1 -- 172.21.15.78:0/3768635007 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6610031920 con 0x7f662015ca30 2024-02-19T14:52:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f661effd640 1 -- 172.21.15.78:0/3768635007 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610031cf0 con 0x7f662015ca30 2024-02-19T14:52:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f66269c7640 1 -- 172.21.15.78:0/3768635007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f662015ca30 msgr2=0x7f662015cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f66269c7640 1 --2- 172.21.15.78:0/3768635007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f662015ca30 0x7f662015cec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66100097a0 tx=0x7f661002b860 comp rx=0 tx=0).stop 2024-02-19T14:52:19.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f66269c7640 1 -- 172.21.15.78:0/3768635007 shutdown_connections 2024-02-19T14:52:19.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f66269c7640 1 --2- 172.21.15.78:0/3768635007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f662015ca30 0x7f662015cec0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f66269c7640 1 --2- 172.21.15.78:0/3768635007 >> v2:172.21.15.150:3300/0 conn(0x7f66200c32c0 0x7f66200c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f66269c7640 1 -- 172.21.15.78:0/3768635007 >> 172.21.15.78:0/3768635007 conn(0x7f662006b270 msgr2=0x7f662006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f66269c7640 1 -- 172.21.15.78:0/3768635007 shutdown_connections 2024-02-19T14:52:19.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.248+0000 7f62ad1d7640 1 -- start start 2024-02-19T14:52:19.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62ad1d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a80a84c0 0x7f62a814a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62ad1d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62a80c1ef0 0x7f62a815e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62ad1d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8151d20 0x7f62a8136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62ad1d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80c4b00 con 0x7f62a8151d20 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62ad1d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80ac810 con 0x7f62a80a84c0 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62ad1d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80c4da0 con 0x7f62a80c1ef0 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f629ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62a80c1ef0 0x7f62a815e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62a6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a80a84c0 0x7f62a814a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f62a759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8151d20 0x7f62a8136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f629ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62a80c1ef0 0x7f62a815e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51002/0 (socket says 172.21.15.78:51002) 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7f629ffff640 1 -- 172.21.15.78:0/3666731516 learned_addr learned my addr 172.21.15.78:0/3666731516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f62a759e640 1 -- 172.21.15.78:0/3666731516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62a80c1ef0 msgr2=0x7f62a815e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f62a759e640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62a80c1ef0 0x7f62a815e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f62a759e640 1 -- 172.21.15.78:0/3666731516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a80a84c0 msgr2=0x7f62a814a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f62a759e640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a80a84c0 0x7f62a814a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f62a759e640 1 -- 172.21.15.78:0/3666731516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980094d0 con 0x7f62a8151d20 2024-02-19T14:52:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.249+0000 7fcd9b979640 1 -- start start 2024-02-19T14:52:19.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7fcd9b979640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd9414e800 0x7fcd94154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f62a759e640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8151d20 0x7f62a8136940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f62a000caf0 tx=0x7f62a00020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c9801a400 con 0x7f6ca80c1e90 2024-02-19T14:52:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f6ca4ff9640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8806b8e0 0x7f6c8806dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd99715640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd9414e800 0x7fcd94154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7f6c9ffff640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8806b8e0 0x7f6c8806dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6c9808a5a0 con 0x7f6ca80c1e90 2024-02-19T14:52:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd99715640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd9414e800 0x7fcd94154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52518/0 (socket says 172.21.15.78:52518) 2024-02-19T14:52:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd99715640 1 -- 172.21.15.78:0/2564211501 learned_addr learned my addr 172.21.15.78:0/2564211501 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.250+0000 7f66269c7640 1 -- 172.21.15.78:0/3768635007 wait complete. 2024-02-19T14:52:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7f6c9ffff640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8806b8e0 0x7f6c8806dda0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6c90036000 tx=0x7f6c90030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7f66269c7640 1 Processor -- start 2024-02-19T14:52:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fb0467fc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b910 0x7fb03406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fb055b58640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b910 0x7fb03406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.252+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb048089710 con 0x7fb0501f43e0 2024-02-19T14:52:19.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.252+0000 7f66269c7640 1 -- start start 2024-02-19T14:52:19.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940c5be0 con 0x7fcd940c5dd0 2024-02-19T14:52:19.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940c1a30 con 0x7fcd9406b590 2024-02-19T14:52:19.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd940c1ff0 con 0x7fcd9414e800 2024-02-19T14:52:19.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd99715640 1 -- 172.21.15.78:0/2564211501 >> v1:172.21.15.150:6789/0 conn(0x7fcd9406b590 legacy=0x7fcd94151a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-19T14:52:19.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd99715640 1 -- 172.21.15.78:0/2564211501 >> v1:172.21.15.78:6789/0 conn(0x7fcd940c5dd0 legacy=0x7fcd940c48c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:52:19.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.251+0000 7fcd99715640 1 -- 172.21.15.78:0/2564211501 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940c1d10 con 0x7fcd9414e800 2024-02-19T14:52:19.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f66269c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66200c32c0 0x7f66200725d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f66269c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6620158df0 0x7f6620155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f66269c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f662015ca30 0x7f6620155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f66269c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66200aace0 con 0x7f662015ca30 2024-02-19T14:52:19.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f66269c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66200a8fe0 con 0x7f66200c32c0 2024-02-19T14:52:19.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f66269c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66201561a0 con 0x7f6620158df0 2024-02-19T14:52:19.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7fb055b58640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b910 0x7fb03406ddd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb0400058a0 tx=0x7fb04003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f661f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6620158df0 0x7f6620155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f661ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66200c32c0 0x7f66200725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f6624f64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f662015ca30 0x7f6620155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f661ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66200c32c0 0x7f66200725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45866/0 (socket says 172.21.15.78:45866) 2024-02-19T14:52:19.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f6624f64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f662015ca30 0x7f6620155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52522/0 (socket says 172.21.15.78:52522) 2024-02-19T14:52:19.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f661ffff640 1 -- 172.21.15.78:0/1358181277 learned_addr learned my addr 172.21.15.78:0/1358181277 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7fcd99715640 1 --2- 172.21.15.78:0/2564211501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd94154fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009b30 tx=0x7fcd90034310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc461612e9b24f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0006b50 con 0x7f62a8151d20 2024-02-19T14:52:19.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7fcd8bfff640 1 -- 172.21.15.78:0/2564211501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9003c440 con 0x7fcd9414e800 2024-02-19T14:52:19.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7fcd8bfff640 1 -- 172.21.15.78:0/2564211501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9003c5a0 con 0x7fcd9414e800 2024-02-19T14:52:19.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7fcd8bfff640 1 -- 172.21.15.78:0/2564211501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9003c930 con 0x7fcd9414e800 2024-02-19T14:52:19.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3b2c077190 con 0x7f3b3414e460 2024-02-19T14:52:19.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.253+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a80c5080 con 0x7f62a8151d20 2024-02-19T14:52:19.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a80c31d0 con 0x7f62a8151d20 2024-02-19T14:52:19.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f661f7fe640 1 -- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66200c32c0 msgr2=0x7f66200725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f661f7fe640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66200c32c0 0x7f66200725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f661f7fe640 1 -- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f662015ca30 msgr2=0x7f6620155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f661f7fe640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f662015ca30 0x7f6620155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f661f7fe640 1 -- 172.21.15.78:0/1358181277 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610009400 con 0x7f6620158df0 2024-02-19T14:52:19.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a0006cb0 con 0x7f62a8151d20 2024-02-19T14:52:19.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0003b70 con 0x7f62a8151d20 2024-02-19T14:52:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 msgr2=0x7fcd94154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.254+0000 7fcd9b979640 1 --2- 172.21.15.78:0/2564211501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd94154fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009b30 tx=0x7fcd90034310 comp rx=0 tx=0).stop 2024-02-19T14:52:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7f661f7fe640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6620158df0 0x7f6620155720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f661400e630 tx=0x7f661400eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 shutdown_connections 2024-02-19T14:52:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7fcd9b979640 1 --2- 172.21.15.78:0/2564211501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd94154fb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 >> 172.21.15.78:0/2564211501 conn(0x7fcd940c6fa0 msgr2=0x7fcd9406ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca8074990 con 0x7f6ca80c1e90 2024-02-19T14:52:19.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 shutdown_connections 2024-02-19T14:52:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.255+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6280005290 con 0x7f62a8151d20 2024-02-19T14:52:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.256+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f62a0003cd0 con 0x7f62a8151d20 2024-02-19T14:52:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.256+0000 7f629f7fe640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f628c06b910 0x7f628c06ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.256+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f62a00abd20 con 0x7f62a8151d20 2024-02-19T14:52:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.257+0000 7f62a6d9d640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f628c06b910 0x7f628c06ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.257+0000 7fcd9b979640 1 -- 172.21.15.78:0/2564211501 wait complete. 2024-02-19T14:52:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.257+0000 7f62a6d9d640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f628c06b910 0x7f628c06ddd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6290009b40 tx=0x7f6290009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.258+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb0501fa140 con 0x7fb0501f43e0 2024-02-19T14:52:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.258+0000 7fcd9b979640 1 Processor -- start 2024-02-19T14:52:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.258+0000 7fcd9b979640 1 -- start start 2024-02-19T14:52:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.257+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614004230 con 0x7f6620158df0 2024-02-19T14:52:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.257+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614004390 con 0x7f6620158df0 2024-02-19T14:52:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.258+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614003830 con 0x7f6620158df0 2024-02-19T14:52:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.258+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66200a83d0 con 0x7f6620158df0 2024-02-19T14:52:19.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.258+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66200a8990 con 0x7f6620158df0 2024-02-19T14:52:19.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.259+0000 7fcd9b979640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd9406b590 0x7fcd940aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.259+0000 7fcd9b979640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd940c5dd0 0x7fcd9414a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.259+0000 7fcd9b979640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd941e9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.259+0000 7fcd9b979640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd94151620 con 0x7fcd9414e800 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.260+0000 7fcd9b979640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940bf110 con 0x7fcd940c5dd0 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.260+0000 7fcd9b979640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd940bc180 con 0x7fcd9406b590 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.260+0000 7fcd99f16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd941e9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.260+0000 7fcd99f16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd941e9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52530/0 (socket says 172.21.15.78:52530) 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.260+0000 7fcd99f16640 1 -- 172.21.15.78:0/2887232049 learned_addr learned my addr 172.21.15.78:0/2887232049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.260+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6620074990 con 0x7f6620158df0 2024-02-19T14:52:19.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7fcd99f16640 1 -- 172.21.15.78:0/2887232049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd9406b590 msgr2=0x7fcd940aafc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:52:19.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7fcd99f16640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd9406b590 0x7fcd940aafc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7fcd99f16640 1 -- 172.21.15.78:0/2887232049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd940c5dd0 msgr2=0x7fcd9414a950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:52:19.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7fcd99f16640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd940c5dd0 0x7fcd9414a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7fcd99f16640 1 -- 172.21.15.78:0/2887232049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd900094d0 con 0x7fcd9414e800 2024-02-19T14:52:19.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c98054cf0 con 0x7f6ca80c1e90 2024-02-19T14:52:19.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.262+0000 7fcd99f16640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd941e9aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00b970 tx=0x7fcd8c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.261+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6614003990 con 0x7f6620158df0 2024-02-19T14:52:19.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.262+0000 7f661cff9640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f660c06b8e0 0x7f660c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.262+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6614089f80 con 0x7f6620158df0 2024-02-19T14:52:19.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.263+0000 7f661ffff640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f660c06b8e0 0x7f660c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.264+0000 7f661ffff640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f660c06b8e0 0x7f660c06dda0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6618001fe0 tx=0x7f6618009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c00cae0 con 0x7fcd9414e800 2024-02-19T14:52:19.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62a00764a0 con 0x7f62a8151d20 2024-02-19T14:52:19.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c00cc40 con 0x7fcd9414e800 2024-02-19T14:52:19.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7fcd9b979640 1 -- 172.21.15.78:0/2887232049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd941e6f30 con 0x7fcd9414e800 2024-02-19T14:52:19.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6614054780 con 0x7f6620158df0 2024-02-19T14:52:19.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c014440 con 0x7fcd9414e800 2024-02-19T14:52:19.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.266+0000 7fcd9b979640 1 -- 172.21.15.78:0/2887232049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd940bbe10 con 0x7fcd9414e800 2024-02-19T14:52:19.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.269+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd74005290 con 0x7fcd9414e800 2024-02-19T14:52:19.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.269+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcd8c014660 con 0x7fcd9414e800 2024-02-19T14:52:19.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.270+0000 7fcd89ffb640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd7806b860 0x7fcd7806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.279+0000 7f4ff733e640 1 Processor -- start 2024-02-19T14:52:19.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.279+0000 7f4ff733e640 1 -- start start 2024-02-19T14:52:19.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.279+0000 7fcd99715640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd7806b860 0x7fcd7806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.279+0000 7fcd99715640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd7806b860 0x7fcd7806dd20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd9003e000 tx=0x7fcd90038000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.280+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcd8c08aa10 con 0x7fcd9414e800 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.280+0000 7f4ff48d9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4ff006ea50 0x7f4ff00c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57758/0 (socket says 172.21.15.78:57758) 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.281+0000 7f4ff48d9640 1 -- 172.21.15.78:0/3908636466 learned_addr learned my addr 172.21.15.78:0/3908636466 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.281+0000 7f4ff733e640 1 --2- 172.21.15.78:0/3908636466 >> v2:172.21.15.78:3300/0 conn(0x7f4ff0072d70 0x7f4ff0152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.281+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00c5970 con 0x7f4ff006ea50 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.281+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00c5750 con 0x7f4ff014f1a0 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.281+0000 7f4ff58db640 1 --2- 172.21.15.78:0/3908636466 >> v2:172.21.15.78:3300/0 conn(0x7f4ff0072d70 0x7f4ff0152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.281+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff00702d0 con 0x7f4ff0072d70 2024-02-19T14:52:19.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.282+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd8c054a70 con 0x7fcd9414e800 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.282+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997459668 0 0) 0x7f4ff00c5970 con 0x7f4ff006ea50 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.282+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003520 con 0x7f4ff006ea50 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939073259 0 0) 0x7f4fdc003520 con 0x7f4ff006ea50 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 >> v2:172.21.15.78:3300/0 conn(0x7f4ff0072d70 msgr2=0x7f4ff0152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 --2- 172.21.15.78:0/3908636466 >> v2:172.21.15.78:3300/0 conn(0x7f4ff0072d70 0x7f4ff0152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 >> v1:172.21.15.150:6789/0 conn(0x7f4ff014f1a0 legacy=0x7f4ff006fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00c5bc0 con 0x7f4ff006ea50 2024-02-19T14:52:19.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4fec001670 con 0x7f4ff006ea50 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 >> v1:172.21.15.78:6789/0 conn(0x7f4ff006ea50 legacy=0x7f4ff00c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0039c0 0x7f4fdc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fdc0047c0 0x7f4fdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00702d0 con 0x7f4fdc0047c0 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00c1a0 con 0x7f4fdc0039c0 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00c360 con 0x7f4fdc0052a0 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4ff48d9640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fdc0047c0 0x7f4fdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4ff58db640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.283+0000 7f4ff50da640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0039c0 0x7f4fdc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.284+0000 7f4ff58db640 1 -- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0039c0 msgr2=0x7f4fdc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.284+0000 7f4ff58db640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0039c0 0x7f4fdc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.284+0000 7f4ff58db640 1 -- 172.21.15.78:0/3908636466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fdc0047c0 msgr2=0x7f4fdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.284+0000 7f4ff58db640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fdc0047c0 0x7f4fdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.284+0000 7f4ff58db640 1 -- 172.21.15.78:0/3908636466 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc003520 con 0x7f4fdc0052a0 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.285+0000 7f4ff58db640 1 -- 172.21.15.78:0/3908636466 reap_dead start 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.285+0000 7f4ff58db640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009fc0 tx=0x7f4fdc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1445a5be7ff2bb03 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.287+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe802cc60 con 0x7f4fdc0052a0 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.287+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe800f040 con 0x7f4fdc0052a0 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.287+0000 7f4fe7fff640 1 -- 172.21.15.78:0/3908636466 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8003990 con 0x7f4fdc0052a0 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.288+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 msgr2=0x7f4fdc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.288+0000 7f4ff733e640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009fc0 tx=0x7f4fdc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.291+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 shutdown_connections 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.291+0000 7f4ff733e640 1 --2- 172.21.15.78:0/3908636466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4fdc00b980 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.291+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 >> 172.21.15.78:0/3908636466 conn(0x7f4ff006c170 msgr2=0x7f4ff006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.296+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 shutdown_connections 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.296+0000 7f4ff733e640 1 -- 172.21.15.78:0/3908636466 wait complete. 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.298+0000 7f4ff733e640 1 Processor -- start 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.298+0000 7f4ff733e640 1 -- start start 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.299+0000 7f4ff733e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4ff00c2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff733e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ff01554e0 0x7f4ff01e9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff733e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff0155bc0 0x7f4ff014d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff733e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01e8610 con 0x7f4ff01554e0 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff50da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4ff00c2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff50da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4ff00c2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51034/0 (socket says 172.21.15.78:51034) 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff50da640 1 -- 172.21.15.78:0/1410277264 learned_addr learned my addr 172.21.15.78:0/1410277264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff733e640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01e8bc0 con 0x7f4ff0155bc0 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.300+0000 7f4ff733e640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0153a00 con 0x7f4fdc0052a0 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.301+0000 7f4ff48d9640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ff01554e0 0x7f4ff01e9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4ff48d9640 1 -- 172.21.15.78:0/1410277264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 msgr2=0x7f4ff00c2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4ff48d9640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4ff00c2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4ff48d9640 1 -- 172.21.15.78:0/1410277264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff0155bc0 msgr2=0x7f4ff014d960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4ff48d9640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff0155bc0 0x7f4ff014d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4ff48d9640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00c2560 con 0x7f4ff01554e0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4ff48d9640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ff01554e0 0x7f4ff01e9050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0074040 tx=0x7f4fec00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.302+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00c830 con 0x7f4ff01554e0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.303+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec00c990 con 0x7f4ff01554e0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.303+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00cd40 con 0x7f4ff01554e0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.303+0000 7f4ff733e640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff01e9590 con 0x7f4ff01554e0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.303+0000 7f4ff733e640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff01e8850 con 0x7f4ff01554e0 2024-02-19T14:52:19.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.308+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd0005290 con 0x7f4ff01554e0 2024-02-19T14:52:19.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.308+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4fec019070 con 0x7f4ff01554e0 2024-02-19T14:52:19.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.308+0000 7f4fe5ffb640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fd406b8e0 0x7f4fd406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:19.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.309+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4fec08e0d0 con 0x7f4ff01554e0 2024-02-19T14:52:19.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.309+0000 7f4ff50da640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fd406b8e0 0x7f4fd406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:19.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.310+0000 7f4ff50da640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fd406b8e0 0x7f4fd406dda0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00a91b0 tx=0x7f4fe0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:19.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.320+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4fec058820 con 0x7f4ff01554e0 2024-02-19T14:52:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.658+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3b341e6b70 con 0x7f3b3414e460 2024-02-19T14:52:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.659+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2030877653 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b2c039020 con 0x7f3b3414e460 2024-02-19T14:52:19.670 INFO:teuthology.orchestra.run.smithi078.stdout:55834574856 2024-02-19T14:52:19.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b1806b8e0 msgr2=0x7f3b1806dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b1806b8e0 0x7f3b1806dda0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3b300098c0 tx=0x7f3b30008040 comp rx=0 tx=0).stop 2024-02-19T14:52:19.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 msgr2=0x7f3b341e6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 0x7f3b341e6e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009d90 tx=0x7f3b2c004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:19.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 shutdown_connections 2024-02-19T14:52:19.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b1806b8e0 0x7f3b1806dda0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b3414e460 0x7f3b341e6e50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b340c1580 0x7f3b340a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 --2- 172.21.15.78:0/2030877653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b3406b3c0 0x7f3b34152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.670+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 >> 172.21.15.78:0/2030877653 conn(0x7f3b340c6410 msgr2=0x7f3b3414bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.671+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 shutdown_connections 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.671+0000 7f3b3a042640 1 -- 172.21.15.78:0/2030877653 wait complete. 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.671+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb6d0005540 con 0x7fb6e80d71a0 2024-02-19T14:52:19.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.672+0000 7fb6d77fe640 1 -- 172.21.15.78:0/965672847 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb6e0057700 con 0x7fb6e80d71a0 2024-02-19T14:52:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:55834574856 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6dc06b8e0 msgr2=0x7fb6dc06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6dc06b8e0 0x7fb6dc06dda0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8008590 tx=0x7fb6f00ac000 comp rx=0 tx=0).stop 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 msgr2=0x7fb6e80e8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 0x7fb6e80e8940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c390 tx=0x7fb6e0001e70 comp rx=0 tx=0).stop 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 shutdown_connections 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6dc06b8e0 0x7fb6dc06dda0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e80d3be0 0x7fb6e80e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.675+0000 7fb6f5072640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e80d71a0 0x7fb6e80e8940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.676+0000 7fb6f5072640 1 --2- 172.21.15.78:0/965672847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e800b100 0x7fb6e8049b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.676+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 >> 172.21.15.78:0/965672847 conn(0x7fb6e8005ec0 msgr2=0x7fb6e8030ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.676+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 shutdown_connections 2024-02-19T14:52:19.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.676+0000 7fb6f5072640 1 -- 172.21.15.78:0/965672847 wait complete. 2024-02-19T14:52:19.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.676+0000 7f6cac90a640 1 -- 172.21.15.78:0/4277432453 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6ca806c570 con 0x7f6ca80c1e90 2024-02-19T14:52:19.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.677+0000 7f6ca4ff9640 1 -- 172.21.15.78:0/4277432453 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c98015030 con 0x7f6ca80c1e90 2024-02-19T14:52:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:55834574857 2024-02-19T14:52:19.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 -- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8806b8e0 msgr2=0x7f6c8806dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8806b8e0 0x7f6c8806dda0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6c90036000 tx=0x7f6c90030000 comp rx=0 tx=0).stop 2024-02-19T14:52:19.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 -- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 msgr2=0x7f6ca81e9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 0x7f6ca81e9f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b410 tx=0x7f6c9800b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f66200c4480 con 0x7f6620158df0 2024-02-19T14:52:19.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb0501fa140 con 0x7fb0501f43e0 2024-02-19T14:52:19.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 -- 172.21.15.78:0/4277432453 shutdown_connections 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8806b8e0 0x7f6c8806dda0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca80c2270 0x7f6ca81487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca81555b0 0x7f6ca8146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 --2- 172.21.15.78:0/4277432453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca80c1e90 0x7f6ca81e9f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 -- 172.21.15.78:0/4277432453 >> 172.21.15.78:0/4277432453 conn(0x7f6ca80c7490 msgr2=0x7f6ca806c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 -- 172.21.15.78:0/4277432453 shutdown_connections 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.681+0000 7f6c9dffb640 1 -- 172.21.15.78:0/4277432453 wait complete. 2024-02-19T14:52:19.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.682+0000 7f661cff9640 1 -- 172.21.15.78:0/1358181277 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6614020020 con 0x7f6620158df0 2024-02-19T14:52:19.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.682+0000 7fb0467fc640 1 -- 172.21.15.78:0/3527997989 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0501fa140 con 0x7fb0501f43e0 2024-02-19T14:52:19.687 INFO:teuthology.orchestra.run.smithi078.stdout:55834574857 2024-02-19T14:52:19.687 INFO:teuthology.orchestra.run.smithi078.stdout:55834574856 2024-02-19T14:52:19.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b910 msgr2=0x7fb03406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f660c06b8e0 msgr2=0x7f660c06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7f66269c7640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f660c06b8e0 0x7f660c06dda0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6618001fe0 tx=0x7f6618009040 comp rx=0 tx=0).stop 2024-02-19T14:52:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b910 0x7fb03406ddd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb0400058a0 tx=0x7fb04003b040 comp rx=0 tx=0).stop 2024-02-19T14:52:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0501f43e0 msgr2=0x7fb0501fa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0501f43e0 0x7fb0501fa420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0500c7490 tx=0x7fb04800ea00 comp rx=0 tx=0).stop 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 shutdown_connections 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b910 0x7fb03406ddd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0500c0690 0x7fb0501f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0501f43e0 0x7fb0501fa420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 --2- 172.21.15.78:0/3527997989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb03c003970 0x7fb0500aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 >> 172.21.15.78:0/3527997989 conn(0x7fb0500c6410 msgr2=0x7fb0500c2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 shutdown_connections 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7fb057dbc640 1 -- 172.21.15.78:0/3527997989 wait complete. 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6620158df0 msgr2=0x7f6620155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7f66269c7640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6620158df0 0x7f6620155720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f661400e630 tx=0x7f661400eaf0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 shutdown_connections 2024-02-19T14:52:19.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.685+0000 7f66269c7640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f660c06b8e0 0x7f660c06dda0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7f66269c7640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f662015ca30 0x7f6620155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7f66269c7640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6620158df0 0x7f6620155720 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7f66269c7640 1 --2- 172.21.15.78:0/1358181277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66200c32c0 0x7f66200725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 >> 172.21.15.78:0/1358181277 conn(0x7f662006b270 msgr2=0x7f66200c0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 shutdown_connections 2024-02-19T14:52:19.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.686+0000 7f66269c7640 1 -- 172.21.15.78:0/1358181277 wait complete. 2024-02-19T14:52:19.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.0 2024-02-19T14:52:19.697 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.3 2024-02-19T14:52:19.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.700+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6280005540 con 0x7f62a8151d20 2024-02-19T14:52:19.704 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.2 2024-02-19T14:52:19.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.700+0000 7f629f7fe640 1 -- 172.21.15.78:0/3666731516 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f62a0038020 con 0x7f62a8151d20 2024-02-19T14:52:19.705 INFO:teuthology.orchestra.run.smithi078.stdout:55834574856 2024-02-19T14:52:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f628c06b910 msgr2=0x7f628c06ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f628c06b910 0x7f628c06ddd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6290009b40 tx=0x7f6290009190 comp rx=0 tx=0).stop 2024-02-19T14:52:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8151d20 msgr2=0x7f62a8136940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8151d20 0x7f62a8136940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f62a000caf0 tx=0x7f62a00020a0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 shutdown_connections 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f628c06b910 0x7f628c06ddd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62a8151d20 0x7f62a8136940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62a80c1ef0 0x7f62a815e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 --2- 172.21.15.78:0/3666731516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62a80a84c0 0x7f62a814a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 >> 172.21.15.78:0/3666731516 conn(0x7f62a80718e0 msgr2=0x7f62a8139990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.703+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 shutdown_connections 2024-02-19T14:52:19.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.704+0000 7f62ad1d7640 1 -- 172.21.15.78:0/3666731516 wait complete. 2024-02-19T14:52:19.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.4 2024-02-19T14:52:19.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.1 2024-02-19T14:52:19.718 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.5 2024-02-19T14:52:19.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.784+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcd74005540 con 0x7fcd9414e800 2024-02-19T14:52:19.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.784+0000 7fcd89ffb640 1 -- 172.21.15.78:0/2887232049 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd8c019020 con 0x7fcd9414e800 2024-02-19T14:52:19.788 INFO:teuthology.orchestra.run.smithi078.stdout:55834574856 2024-02-19T14:52:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd7806b860 msgr2=0x7fcd7806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd7806b860 0x7fcd7806dd20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd9003e000 tx=0x7fcd90038000 comp rx=0 tx=0).stop 2024-02-19T14:52:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 msgr2=0x7fcd941e9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd941e9aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00b970 tx=0x7fcd8c00be30 comp rx=0 tx=0).stop 2024-02-19T14:52:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 shutdown_connections 2024-02-19T14:52:19.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd7806b860 0x7fcd7806dd20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9414e800 0x7fcd941e9aa0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd940c5dd0 0x7fcd9414a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 --2- 172.21.15.78:0/2887232049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd9406b590 0x7fcd940aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 >> 172.21.15.78:0/2887232049 conn(0x7fcd940c6fa0 msgr2=0x7fcd94151a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 shutdown_connections 2024-02-19T14:52:19.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.786+0000 7fcd7f7fe640 1 -- 172.21.15.78:0/2887232049 wait complete. 2024-02-19T14:52:19.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.7 2024-02-19T14:52:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.856+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4fd0005540 con 0x7f4ff01554e0 2024-02-19T14:52:19.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.856+0000 7f4fe5ffb640 1 -- 172.21.15.78:0/1410277264 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4fec05bee0 con 0x7f4ff01554e0 2024-02-19T14:52:19.860 INFO:teuthology.orchestra.run.smithi078.stdout:55834574855 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fd406b8e0 msgr2=0x7f4fd406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fd406b8e0 0x7f4fd406dda0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00a91b0 tx=0x7f4fe0009040 comp rx=0 tx=0).stop 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ff01554e0 msgr2=0x7f4ff01e9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ff01554e0 0x7f4ff01e9050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0074040 tx=0x7f4fec00eb50 comp rx=0 tx=0).stop 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 shutdown_connections 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fd406b8e0 0x7f4fd406dda0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ff01554e0 0x7f4ff01e9050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff0155bc0 0x7f4ff014d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 --2- 172.21.15.78:0/1410277264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0052a0 0x7f4ff00c2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 >> 172.21.15.78:0/1410277264 conn(0x7f4ff006c170 msgr2=0x7f4ff00ac3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 shutdown_connections 2024-02-19T14:52:19.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:19.858+0000 7f4fdb7fe640 1 -- 172.21.15.78:0/1410277264 wait complete. 2024-02-19T14:52:19.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.6 2024-02-19T14:52:20.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-19T14:52:20.698 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-19T14:52:20.706 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-19T14:52:20.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-19T14:52:20.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-19T14:52:20.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-19T14:52:20.805 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-19T14:52:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.827+0000 7fcf14847640 1 Processor -- start 2024-02-19T14:52:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.828+0000 7fcf14847640 1 -- start start 2024-02-19T14:52:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.837+0000 7fcf14847640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf1014e800 0x7fcf100c3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.838+0000 7fcf0e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf1014e800 0x7fcf100c3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.838+0000 7fcf0e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf1014e800 0x7fcf100c3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45914/0 (socket says 172.21.15.78:45914) 2024-02-19T14:52:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.838+0000 7fcf14847640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf101515d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.838+0000 7fcf0e59c640 1 -- 172.21.15.78:0/3025346516 learned_addr learned my addr 172.21.15.78:0/3025346516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.839+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1006e170 con 0x7fcf100c3990 2024-02-19T14:52:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.839+0000 7fcf0dd9b640 1 --2- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf101515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.839+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf100c1d30 con 0x7fcf1014e800 2024-02-19T14:52:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.839+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf100c5490 con 0x7fcf100c47e0 2024-02-19T14:52:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.839+0000 7fcf0d59a640 1 -- 172.21.15.78:0/3025346516 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866380047 0 0) 0x7fcf1006e170 con 0x7fcf100c3990 2024-02-19T14:52:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.840+0000 7fcf0d59a640 1 -- 172.21.15.78:0/3025346516 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef8003520 con 0x7fcf100c3990 2024-02-19T14:52:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.840+0000 7fcf0e59c640 1 -- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 msgr2=0x7fcf101515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.840+0000 7fcf0e59c640 1 --2- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf101515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.840+0000 7fcf0e59c640 1 -- 172.21.15.78:0/3025346516 >> v1:172.21.15.150:6789/0 conn(0x7fcf100c3990 legacy=0x7fcf10154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.840+0000 7fcf0e59c640 1 -- 172.21.15.78:0/3025346516 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf100c4ed0 con 0x7fcf1014e800 2024-02-19T14:52:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.842+0000 7fcf0e59c640 1 --2- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100c3450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009e50 tx=0x7fcf0002e130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b915882984f5f45 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.842+0000 7fcf0d59a640 1 -- 172.21.15.78:0/3025346516 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0002ecd0 con 0x7fcf1014e800 2024-02-19T14:52:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.842+0000 7fcf0d59a640 1 -- 172.21.15.78:0/3025346516 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0002ee30 con 0x7fcf1014e800 2024-02-19T14:52:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.842+0000 7fcf0d59a640 1 -- 172.21.15.78:0/3025346516 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000365f0 con 0x7fcf1014e800 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 msgr2=0x7fcf100c3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 --2- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100c3450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009e50 tx=0x7fcf0002e130 comp rx=0 tx=0).stop 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 shutdown_connections 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 --2- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf101515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 --2- 172.21.15.78:0/3025346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100c3450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 >> 172.21.15.78:0/3025346516 conn(0x7fcf1006b270 msgr2=0x7fcf1006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.843+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 shutdown_connections 2024-02-19T14:52:20.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.844+0000 7fcf14847640 1 -- 172.21.15.78:0/3025346516 wait complete. 2024-02-19T14:52:20.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.846+0000 7fcf14847640 1 Processor -- start 2024-02-19T14:52:20.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.846+0000 7fcf14847640 1 -- start start 2024-02-19T14:52:20.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf14847640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf100c3990 0x7fcf101dc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf14847640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf100a9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf14847640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf14847640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf100ab360 con 0x7fcf100c3990 2024-02-19T14:52:20.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf14847640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf100c0310 con 0x7fcf1014e800 2024-02-19T14:52:20.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf14847640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf100c05f0 con 0x7fcf100c47e0 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.847+0000 7fcf0dd9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf100a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.848+0000 7fcf0ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.848+0000 7fcf0e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf100c3990 0x7fcf101dc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.848+0000 7fcf0ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100a9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45918/0 (socket says 172.21.15.78:45918) 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.848+0000 7fcf0ed9d640 1 -- 172.21.15.78:0/945256411 learned_addr learned my addr 172.21.15.78:0/945256411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.848+0000 7fcf0dd9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf100a9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51046/0 (socket says 172.21.15.78:51046) 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.849+0000 7fcf0e59c640 1 -- 172.21.15.78:0/945256411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 msgr2=0x7fcf100a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.849+0000 7fcf0e59c640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf100a9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.849+0000 7fcf0e59c640 1 -- 172.21.15.78:0/945256411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 msgr2=0x7fcf100a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.849+0000 7fcf0e59c640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.849+0000 7fcf0e59c640 1 -- 172.21.15.78:0/945256411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00009b30 con 0x7fcf100c3990 2024-02-19T14:52:20.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.849+0000 7fcf0e59c640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf100c3990 0x7fcf101dc7c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf00034000 tx=0x7fcf00004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.850+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000405f0 con 0x7fcf100c3990 2024-02-19T14:52:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.850+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf100ab9f0 con 0x7fcf100c3990 2024-02-19T14:52:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.850+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf100abfb0 con 0x7fcf100c3990 2024-02-19T14:52:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.850+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf00040770 con 0x7fcf100c3990 2024-02-19T14:52:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.850+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf00040b20 con 0x7fcf100c3990 2024-02-19T14:52:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.853+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcf00040c80 con 0x7fcf100c3990 2024-02-19T14:52:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.854+0000 7fcefeffd640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b8e0 0x7fcef006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.854+0000 7fcf0dd9b640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b8e0 0x7fcef006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.855+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcf000ac570 con 0x7fcf100c3990 2024-02-19T14:52:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.863+0000 7fcf0dd9b640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b8e0 0x7fcef006dda0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf1006d8a0 tx=0x7fcf0400c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.864+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef4005290 con 0x7fcf100c3990 2024-02-19T14:52:20.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-19T14:52:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.872+0000 7efc65de5640 1 Processor -- start 2024-02-19T14:52:20.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.872+0000 7efc65de5640 1 -- start start 2024-02-19T14:52:20.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.873+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf00076d70 con 0x7fcf100c3990 2024-02-19T14:52:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.880+0000 7efc65de5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc58048490 0x7efc580474c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.881+0000 7efc65de5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 0x7efc580e26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.882+0000 7efc5ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 0x7efc580e26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.882+0000 7efc5ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 0x7efc580e26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51054/0 (socket says 172.21.15.78:51054) 2024-02-19T14:52:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.883+0000 7efc5ffff640 1 -- 172.21.15.78:0/1756469239 learned_addr learned my addr 172.21.15.78:0/1756469239 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.883+0000 7efc64de3640 1 --2- 172.21.15.78:0/1756469239 >> v2:172.21.15.78:3300/0 conn(0x7efc58048490 0x7efc580474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.884+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580df290 con 0x7efc5803f020 2024-02-19T14:52:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.884+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc58046f50 con 0x7efc58048490 2024-02-19T14:52:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.884+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc580dc2d0 con 0x7efc58006630 2024-02-19T14:52:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.886+0000 7efc5f7fe640 1 -- 172.21.15.78:0/1756469239 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290024262 0 0) 0x7efc580df290 con 0x7efc5803f020 2024-02-19T14:52:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.886+0000 7efc5f7fe640 1 -- 172.21.15.78:0/1756469239 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50003520 con 0x7efc5803f020 2024-02-19T14:52:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.886+0000 7efc5ffff640 1 -- 172.21.15.78:0/1756469239 >> v2:172.21.15.78:3300/0 conn(0x7efc58048490 msgr2=0x7efc580474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.886+0000 7efc5f7fe640 1 -- 172.21.15.78:0/1756469239 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244363326 0 0) 0x7efc50003520 con 0x7efc5803f020 2024-02-19T14:52:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.887+0000 7efc5ffff640 1 --2- 172.21.15.78:0/1756469239 >> v2:172.21.15.78:3300/0 conn(0x7efc58048490 0x7efc580474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.887+0000 7efc5ffff640 1 -- 172.21.15.78:0/1756469239 >> v1:172.21.15.150:6789/0 conn(0x7efc5803f020 legacy=0x7efc58007ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.887+0000 7efc5ffff640 1 -- 172.21.15.78:0/1756469239 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58047a00 con 0x7efc58006630 2024-02-19T14:52:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.889+0000 7efc64de3640 1 --2- 172.21.15.78:0/1756469239 >> v2:172.21.15.78:3300/0 conn(0x7efc58048490 0x7efc580474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:20.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7efc5ffff640 1 --2- 172.21.15.78:0/1756469239 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 0x7efc580e26f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efc54009870 tx=0x7efc5402b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2feba7bd712c4c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.895+0000 7f58128d3640 1 Processor -- start 2024-02-19T14:52:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.895+0000 7f58128d3640 1 -- start start 2024-02-19T14:52:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.895+0000 7fe66d272640 1 Processor -- start 2024-02-19T14:52:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.895+0000 7fe66d272640 1 -- start start 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58128d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f580c156880 0x7f580c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58128d3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f580c0c9220 0x7f580c0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58128d3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580c0c5040 con 0x7f580c0ca230 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58128d3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c0c7000 con 0x7f580c0c9220 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58128d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c0be080 con 0x7f580c156880 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58120d2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f580c0c9220 0x7f580c0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58120d2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f580c0c9220 0x7f580c0c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52596/0 (socket says 172.21.15.78:52596) 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7f58120d2640 1 -- 172.21.15.78:0/715894679 learned_addr learned my addr 172.21.15.78:0/715894679 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7f58120d2640 1 -- 172.21.15.78:0/715894679 >> v2:172.21.15.150:3300/0 conn(0x7f580c156880 msgr2=0x7f580c0c9770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:52:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7f58120d2640 1 --2- 172.21.15.78:0/715894679 >> v2:172.21.15.150:3300/0 conn(0x7f580c156880 0x7f580c0c9770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7f58120d2640 1 -- 172.21.15.78:0/715894679 >> v1:172.21.15.78:6789/0 conn(0x7f580c0ca230 legacy=0x7f580c0c5a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7f58120d2640 1 -- 172.21.15.78:0/715894679 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c159460 con 0x7f580c0c9220 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.896+0000 7fe66d272640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe66004ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe66d272640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66004c480 con 0x7fe66004b8b0 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe66d272640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66004ca40 con 0x7fe66004d410 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe66d272640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600d81f0 con 0x7fe6600d5940 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe6677fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe66004b8b0 0x7fe660048330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57764/0 (socket says 172.21.15.78:57764) 2024-02-19T14:52:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe6677fe640 1 -- 172.21.15.78:0/1198000412 learned_addr learned my addr 172.21.15.78:0/1198000412 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe667fff640 1 --2- 172.21.15.78:0/1198000412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe66004ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe666ffd640 1 -- 172.21.15.78:0/1198000412 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590746378 0 0) 0x7fe66004c480 con 0x7fe66004b8b0 2024-02-19T14:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe667fff640 1 -- 172.21.15.78:0/1198000412 >> v1:172.21.15.150:6789/0 conn(0x7fe66004d410 legacy=0x7fe6600e2b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-19T14:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe667fff640 1 -- 172.21.15.78:0/1198000412 >> v1:172.21.15.78:6789/0 conn(0x7fe66004b8b0 legacy=0x7fe660048330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.897+0000 7fe667fff640 1 -- 172.21.15.78:0/1198000412 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6600d8400 con 0x7fe6600d5940 2024-02-19T14:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7efc5f7fe640 1 -- 172.21.15.78:0/1756469239 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60062b20 con 0x7efc58006630 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7efc5f7fe640 1 -- 172.21.15.78:0/1756469239 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5402e050 con 0x7efc58006630 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7efc5f7fe640 1 -- 172.21.15.78:0/1756469239 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54030730 con 0x7efc58006630 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.899+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 msgr2=0x7efc580e26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.899+0000 7efc65de5640 1 --2- 172.21.15.78:0/1756469239 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 0x7efc580e26f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efc54009870 tx=0x7efc5402b740 comp rx=0 tx=0).stop 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 shutdown_connections 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7efc65de5640 1 --2- 172.21.15.78:0/1756469239 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58006630 0x7efc580e26f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7efc65de5640 1 --2- 172.21.15.78:0/1756469239 >> v2:172.21.15.78:3300/0 conn(0x7efc58048490 0x7efc580474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 >> 172.21.15.78:0/1756469239 conn(0x7efc5802db80 msgr2=0x7efc5802df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f1b941ef640 1 Processor -- start 2024-02-19T14:52:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f8c52c42640 1 Processor -- start 2024-02-19T14:52:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 shutdown_connections 2024-02-19T14:52:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f8c52c42640 1 -- start start 2024-02-19T14:52:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f8c52c42640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c4c156920 0x7f8c4c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f8c52c42640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f8c52c42640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4c159e20 con 0x7f8c4c06f530 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f8c52c42640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4c0c3b00 con 0x7f8c4c156920 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f8c52c42640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c0c3520 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c4c156920 0x7f8c4c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c4c156920 0x7f8c4c0c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45920/0 (socket says 172.21.15.78:45920) 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 -- 172.21.15.78:0/1650202494 learned_addr learned my addr 172.21.15.78:0/1650202494 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c4bfff640 1 --2- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f1b941ef640 1 -- start start 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f1b941ef640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b8c158df0 0x7f1b8c0c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f1b941ef640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0c19d0 con 0x7f1b8c15ca30 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f1b941ef640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0c4740 con 0x7f1b8c0c14c0 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f1b941ef640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c0c4a20 con 0x7f1b8c158df0 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7f58120d2640 1 --2- 172.21.15.78:0/715894679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c0c9220 0x7f580c0c61a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f580400b1d0 tx=0x7f580402c030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e894cce5615756 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7f58108cf640 1 -- 172.21.15.78:0/715894679 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580402c9d0 con 0x7f580c0c9220 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7f58108cf640 1 -- 172.21.15.78:0/715894679 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580402cb30 con 0x7f580c0c9220 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.899+0000 7f58108cf640 1 -- 172.21.15.78:0/715894679 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58040343f0 con 0x7f580c0c9220 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.899+0000 7f58128d3640 1 -- 172.21.15.78:0/715894679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c0c9220 msgr2=0x7f580c0c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.899+0000 7f58128d3640 1 --2- 172.21.15.78:0/715894679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c0c9220 0x7f580c0c61a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f580400b1d0 tx=0x7f580402c030 comp rx=0 tx=0).stop 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f58128d3640 1 -- 172.21.15.78:0/715894679 shutdown_connections 2024-02-19T14:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f58128d3640 1 --2- 172.21.15.78:0/715894679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c0c9220 0x7f580c0c61a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f58128d3640 1 --2- 172.21.15.78:0/715894679 >> v2:172.21.15.150:3300/0 conn(0x7f580c156880 0x7f580c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f58128d3640 1 -- 172.21.15.78:0/715894679 >> 172.21.15.78:0/715894679 conn(0x7f580c0ca6c0 msgr2=0x7f580c0caad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7f58128d3640 1 -- 172.21.15.78:0/715894679 shutdown_connections 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7f58128d3640 1 -- 172.21.15.78:0/715894679 wait complete. 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.898+0000 7fe667fff640 1 --2- 172.21.15.78:0/1198000412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe66004ced0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe66804f770 tx=0x7fe6680a6910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2edc9e7370db9b41 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7fe666ffd640 1 -- 172.21.15.78:0/1198000412 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6680ac930 con 0x7fe6600d5940 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7fe666ffd640 1 -- 172.21.15.78:0/1198000412 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6680aca90 con 0x7fe6600d5940 2024-02-19T14:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.901+0000 7fe666ffd640 1 -- 172.21.15.78:0/1198000412 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6680ace80 con 0x7fe6600d5940 2024-02-19T14:52:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.900+0000 7efc65de5640 1 -- 172.21.15.78:0/1756469239 wait complete. 2024-02-19T14:52:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7efc65de5640 1 Processor -- start 2024-02-19T14:52:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7efc65de5640 1 -- start start 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 -- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 msgr2=0x7f8c4c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 --2- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c06eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 -- 172.21.15.78:0/1650202494 >> v1:172.21.15.150:6789/0 conn(0x7f8c4c06f530 legacy=0x7f8c4c070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f8c509de640 1 -- 172.21.15.78:0/1650202494 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c15a380 con 0x7f8c4c156920 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f1b9278c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1b8c15ca30 0x7f1b8c15ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57774/0 (socket says 172.21.15.78:57774) 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f1b9278c640 1 -- 172.21.15.78:0/3643182397 learned_addr learned my addr 172.21.15.78:0/3643182397 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7f1b91f8b640 1 --2- 172.21.15.78:0/3643182397 >> v2:172.21.15.150:3300/0 conn(0x7f1b8c158df0 0x7f1b8c0c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7f1b91f8b640 1 -- 172.21.15.78:0/3643182397 >> v1:172.21.15.150:6789/0 conn(0x7f1b8c0c14c0 legacy=0x7f1b8c0c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7f1b91f8b640 1 -- 172.21.15.78:0/3643182397 >> v1:172.21.15.78:6789/0 conn(0x7f1b8c15ca30 legacy=0x7f1b8c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7f1b91f8b640 1 -- 172.21.15.78:0/3643182397 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0c4d00 con 0x7f1b8c158df0 2024-02-19T14:52:20.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f8c509de640 1 --2- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0c4980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c440098a0 tx=0x7f8c4402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8d98f83435797e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7efc65de5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 0x7efc58177bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc64de3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 0x7efc58177bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc64de3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 0x7efc58177bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52600/0 (socket says 172.21.15.78:52600) 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc64de3640 1 -- 172.21.15.78:0/1464122813 learned_addr learned my addr 172.21.15.78:0/1464122813 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5803f020 0x7efc580e0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f1b91f8b640 1 --2- 172.21.15.78:0/3643182397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c0c0f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0097a0 tx=0x7f1b7c02b570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ab65479664385 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f8c4b7fe640 1 -- 172.21.15.78:0/1650202494 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44030710 con 0x7f8c4c156920 2024-02-19T14:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f8c4b7fe640 1 -- 172.21.15.78:0/1650202494 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4402e050 con 0x7f8c4c156920 2024-02-19T14:52:20.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f8c4b7fe640 1 -- 172.21.15.78:0/1650202494 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44030a90 con 0x7f8c4c156920 2024-02-19T14:52:20.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58048490 0x7efc580d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58043830 con 0x7efc58006630 2024-02-19T14:52:20.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc5ffff640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5803f020 0x7efc580e0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc580e1b70 con 0x7efc5803f020 2024-02-19T14:52:20.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5802d9c0 con 0x7efc58048490 2024-02-19T14:52:20.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f1b90f89640 1 -- 172.21.15.78:0/3643182397 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c030710 con 0x7f1b8c158df0 2024-02-19T14:52:20.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7f58128d3640 1 Processor -- start 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7f58128d3640 1 -- start start 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58128d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f580c0c9220 0x7f580c1490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58128d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f580c0ca230 0x7f580c0c1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58128d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 0x7f580c157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58128d3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c157080 con 0x7f580c156880 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58128d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c3130 con 0x7f580c0ca230 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58128d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c149620 con 0x7f580c0c9220 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58120d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 0x7f580c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58120d2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 0x7f580c157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52612/0 (socket says 172.21.15.78:52612) 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58120d2640 1 -- 172.21.15.78:0/2254585012 learned_addr learned my addr 172.21.15.78:0/2254585012 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.904+0000 7f58118d1640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f580c0c9220 0x7f580c1490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58110d0640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f580c0ca230 0x7f580c0c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58120d2640 1 -- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f580c0c9220 msgr2=0x7f580c1490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58120d2640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f580c0c9220 0x7f580c1490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f1b90f89640 1 -- 172.21.15.78:0/3643182397 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c02e050 con 0x7f1b8c158df0 2024-02-19T14:52:20.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58120d2640 1 -- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f580c0ca230 msgr2=0x7f580c0c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58120d2640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f580c0ca230 0x7f580c0c1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58120d2640 1 -- 172.21.15.78:0/2254585012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804009b30 con 0x7f580c156880 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.905+0000 7f58120d2640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 0x7f580c157510 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5804036040 tx=0x7f5804004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580402c9d0 con 0x7f580c156880 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f58128d3640 1 -- 172.21.15.78:0/2254585012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c0c3550 con 0x7f580c156880 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f8c52c42640 1 -- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 msgr2=0x7f8c4c0c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f8c52c42640 1 --2- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0c4980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c440098a0 tx=0x7f8c4402b7c0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f1b90f89640 1 -- 172.21.15.78:0/3643182397 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c030a90 con 0x7f1b8c158df0 2024-02-19T14:52:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f58128d3640 1 -- 172.21.15.78:0/2254585012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c0c3b10 con 0x7f580c156880 2024-02-19T14:52:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f8c52c42640 1 -- 172.21.15.78:0/1650202494 shutdown_connections 2024-02-19T14:52:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f8c52c42640 1 --2- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c06eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f8c52c42640 1 --2- 172.21.15.78:0/1650202494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0c4980 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7f8c52c42640 1 -- 172.21.15.78:0/1650202494 >> 172.21.15.78:0/1650202494 conn(0x7f8c4c0c85e0 msgr2=0x7f8c4c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7efc655e4640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58048490 0x7efc580d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7efc64de3640 1 -- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58048490 msgr2=0x7efc580d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f1b941ef640 1 -- 172.21.15.78:0/3643182397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 msgr2=0x7f1b8c0c0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f1b941ef640 1 --2- 172.21.15.78:0/3643182397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c0c0f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0097a0 tx=0x7f1b7c02b570 comp rx=0 tx=0).stop 2024-02-19T14:52:20.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7efc64de3640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58048490 0x7efc580d6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7efc64de3640 1 -- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5803f020 msgr2=0x7efc580e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7fe66d272640 1 -- 172.21.15.78:0/1198000412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 msgr2=0x7fe66004ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.902+0000 7fe66d272640 1 --2- 172.21.15.78:0/1198000412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe66004ced0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe66804f770 tx=0x7fe6680a6910 comp rx=0 tx=0).stop 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7fe66d272640 1 -- 172.21.15.78:0/1198000412 shutdown_connections 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7fe66d272640 1 --2- 172.21.15.78:0/1198000412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe66004ced0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7fe66d272640 1 -- 172.21.15.78:0/1198000412 >> 172.21.15.78:0/1198000412 conn(0x7fe66002f4a0 msgr2=0x7fe66002f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7fe66d272640 1 -- 172.21.15.78:0/1198000412 shutdown_connections 2024-02-19T14:52:20.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.903+0000 7fe66d272640 1 -- 172.21.15.78:0/1198000412 wait complete. 2024-02-19T14:52:20.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7fe66d272640 1 Processor -- start 2024-02-19T14:52:20.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7fe66d272640 1 -- start start 2024-02-19T14:52:20.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7efc64de3640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5803f020 0x7efc580e0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.906+0000 7efc64de3640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc540094d0 con 0x7efc58006630 2024-02-19T14:52:20.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7efc5ffff640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5803f020 0x7efc580e0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:20.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f8c52c42640 1 -- 172.21.15.78:0/1650202494 shutdown_connections 2024-02-19T14:52:20.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f1b941ef640 1 -- 172.21.15.78:0/3643182397 shutdown_connections 2024-02-19T14:52:20.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f1b941ef640 1 --2- 172.21.15.78:0/3643182397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c0c0f80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f1b941ef640 1 -- 172.21.15.78:0/3643182397 >> 172.21.15.78:0/3643182397 conn(0x7f1b8c06b270 msgr2=0x7f1b8c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:20.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580403e450 con 0x7f580c156880 2024-02-19T14:52:20.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7f1b941ef640 1 -- 172.21.15.78:0/3643182397 shutdown_connections 2024-02-19T14:52:20.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7efc64de3640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 0x7efc58177bf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00bfd0 tx=0x7efc4c00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.907+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580403e7e0 con 0x7f580c156880 2024-02-19T14:52:20.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e8005290 con 0x7f580c156880 2024-02-19T14:52:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c004030 con 0x7efc58006630 2024-02-19T14:52:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c004190 con 0x7efc58006630 2024-02-19T14:52:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c004520 con 0x7efc58006630 2024-02-19T14:52:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7f8c52c42640 1 -- 172.21.15.78:0/1650202494 wait complete. 2024-02-19T14:52:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f8c52c42640 1 Processor -- start 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.908+0000 7f1b941ef640 1 -- 172.21.15.78:0/3643182397 wait complete. 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.909+0000 7f1b941ef640 1 Processor -- start 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.909+0000 7f1b941ef640 1 -- start start 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b941ef640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b8c0c14c0 0x7f1b8c0a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b941ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c1fae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b941ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b8c15ca30 0x7f1b8c15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b941ef640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0a9740 con 0x7f1b8c15ca30 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b941ef640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0a6c20 con 0x7f1b8c158df0 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b941ef640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c155860 con 0x7f1b8c0c14c0 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b9178a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c1fae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b9178a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c1fae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45960/0 (socket says 172.21.15.78:45960) 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b9178a640 1 -- 172.21.15.78:0/1741122415 learned_addr learned my addr 172.21.15.78:0/1741122415 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b9278c640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b8c15ca30 0x7f1b8c15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f1b9278c640 1 -- 172.21.15.78:0/1741122415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b8c0c14c0 msgr2=0x7f1b8c0a66e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f1b9278c640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b8c0c14c0 0x7f1b8c0a66e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f1b9278c640 1 -- 172.21.15.78:0/1741122415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 msgr2=0x7f1b8c1fae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f1b9278c640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c1fae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f1b9278c640 1 -- 172.21.15.78:0/1741122415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c009400 con 0x7f1b8c15ca30 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f1b9278c640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b8c15ca30 0x7f1b8c15b4d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c730 tx=0x7f1b8400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66d272640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe66004b8b0 0x7fe66000d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66d272640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 0x7fe6600081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66d272640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe660191210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66d272640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66000a060 con 0x7fe66004b8b0 2024-02-19T14:52:20.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66d272640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66000a340 con 0x7fe66004d410 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66d272640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66000a620 con 0x7fe6600d5940 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe6677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 0x7fe6600081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe66ca71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe660191210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe6677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 0x7fe6600081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45976/0 (socket says 172.21.15.78:45976) 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7fe6677fe640 1 -- 172.21.15.78:0/1167022575 learned_addr learned my addr 172.21.15.78:0/1167022575 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7fe6677fe640 1 -- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 msgr2=0x7fe660191210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7fe6677fe640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe660191210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7fe6677fe640 1 -- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe66004b8b0 msgr2=0x7fe66000d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7fe6677fe640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe66004b8b0 0x7fe66000d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7fe6677fe640 1 -- 172.21.15.78:0/1167022575 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe668050140 con 0x7fe66004d410 2024-02-19T14:52:20.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5802d120 con 0x7efc58006630 2024-02-19T14:52:20.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5802d6e0 con 0x7efc58006630 2024-02-19T14:52:20.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f580403e940 con 0x7f580c156880 2024-02-19T14:52:20.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f58027fc640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57f006b8e0 0x7f57f006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f58040ad840 con 0x7f580c156880 2024-02-19T14:52:20.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.910+0000 7f8c52c42640 1 -- start start 2024-02-19T14:52:20.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.911+0000 7fe6677fe640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 0x7fe6600081d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe65800c460 tx=0x7fe65800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c52c42640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c06f530 0x7f8c4c0c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c52c42640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c0ad270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c52c42640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c52c42640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c1440b0 con 0x7f8c4c06f530 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c52c42640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c0acde0 con 0x7f8c4c156920 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c52c42640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c15d830 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c4bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c0ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c4bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c0ad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51124/0 (socket says 172.21.15.78:51124) 2024-02-19T14:52:20.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6580074e0 con 0x7fe66004d410 2024-02-19T14:52:20.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c4bfff640 1 -- 172.21.15.78:0/1357822684 learned_addr learned my addr 172.21.15.78:0/1357822684 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:20.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c509de640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c06f530 0x7f8c4c0c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc58036680 con 0x7efc58006630 2024-02-19T14:52:20.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7f8c511df640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c4bfff640 1 -- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 msgr2=0x7f8c4c0ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c4bfff640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0ad7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c4bfff640 1 -- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c06f530 msgr2=0x7f8c4c0c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c4bfff640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c06f530 0x7f8c4c0c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c4bfff640 1 -- 172.21.15.78:0/1357822684 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c440094d0 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f58118d1640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57f006b8e0 0x7f57f006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.912+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe658007640 con 0x7fe66004d410 2024-02-19T14:52:20.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66000a900 con 0x7fe66004d410 2024-02-19T14:52:20.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660007b50 con 0x7fe66004d410 2024-02-19T14:52:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c4bfff640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c0ad270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400e9b0 tx=0x7f8c3400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34009670 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3400c9a0 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400cd50 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f58118d1640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57f006b8e0 0x7f57f006dda0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f580c0c6680 tx=0x7f5808009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c52c42640 1 -- 172.21.15.78:0/1357822684 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c0ac4b0 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.913+0000 7f8c52c42640 1 -- 172.21.15.78:0/1357822684 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c0aca70 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.915+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658004a20 con 0x7fe66004d410 2024-02-19T14:52:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.915+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe658004c00 con 0x7fe66004d410 2024-02-19T14:52:20.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8c34013470 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c24005290 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7fe664ff9640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe65406b8e0 0x7fe65406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648005290 con 0x7fe66004d410 2024-02-19T14:52:20.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe658089a40 con 0x7fe66004d410 2024-02-19T14:52:20.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7fe667fff640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe65406b8e0 0x7fe65406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7f8c497fa640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8c2c06b810 0x7f8c2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7f8c509de640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8c2c06b810 0x7f8c2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.916+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8c3408a6d0 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.917+0000 7f8c509de640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8c2c06b810 0x7f8c2c06dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c44034000 tx=0x7f8c44008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.917+0000 7fe667fff640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe65406b8e0 0x7fe65406dda0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe66004e950 tx=0x7fe6680ab000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.917+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5804078160 con 0x7f580c156880 2024-02-19T14:52:20.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.918+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400ce40 con 0x7f1b8c15ca30 2024-02-19T14:52:20.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.918+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c0a9460 con 0x7f1b8c15ca30 2024-02-19T14:52:20.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.918+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c1fa930 con 0x7f1b8c15ca30 2024-02-19T14:52:20.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.919+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efc4c009760 con 0x7efc58006630 2024-02-19T14:52:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.919+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400a640 con 0x7f1b8c15ca30 2024-02-19T14:52:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.919+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8400a9f0 con 0x7f1b8c15ca30 2024-02-19T14:52:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.920+0000 7efc5d7fa640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4406b910 0x7efc4406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.920+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efc4c08b220 con 0x7efc58006630 2024-02-19T14:52:20.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.920+0000 7efc5ffff640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4406b910 0x7efc4406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.921+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74005290 con 0x7f1b8c15ca30 2024-02-19T14:52:20.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.921+0000 7efc5ffff640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4406b910 0x7efc4406ddd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efc54009870 tx=0x7efc54008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.927+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe658054340 con 0x7fe66004d410 2024-02-19T14:52:20.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.927+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efc4c0558e0 con 0x7efc58006630 2024-02-19T14:52:20.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.927+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8c34054e50 con 0x7f8c4c0c2f60 2024-02-19T14:52:20.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.939+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1b8400ab50 con 0x7f1b8c15ca30 2024-02-19T14:52:20.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.939+0000 7f1b827fc640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b7006b810 0x7f1b7006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.939+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1b84089c70 con 0x7f1b8c15ca30 2024-02-19T14:52:20.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.940+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b8408a0f0 con 0x7f1b8c15ca30 2024-02-19T14:52:20.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.940+0000 7f1b91f8b640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b7006b810 0x7f1b7006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.941+0000 7f1b91f8b640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b7006b810 0x7f1b7006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c034000 tx=0x7f1b7c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.981+0000 7f914bdf4640 1 Processor -- start 2024-02-19T14:52:20.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.985+0000 7f914bdf4640 1 -- start start 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914bdf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440c7a40 0x7f9144158950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914bdf4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f91440721a0 0x7f9144158e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914bdf4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914415d6c0 con 0x7f91440a6890 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914bdf4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914415cab0 con 0x7f91440721a0 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914bdf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914415cd90 con 0x7f91440c7a40 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914938f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f91440a6890 0x7f9144159580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57790/0 (socket says 172.21.15.78:57790) 2024-02-19T14:52:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.986+0000 7f914938f640 1 -- 172.21.15.78:0/1163411024 learned_addr learned my addr 172.21.15.78:0/1163411024 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.987+0000 7f914a391640 1 --2- 172.21.15.78:0/1163411024 >> v2:172.21.15.150:3300/0 conn(0x7f91440721a0 0x7f9144158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.989+0000 7f9148b8e640 1 -- 172.21.15.78:0/1163411024 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914773336 0 0) 0x7f914415d6c0 con 0x7f91440a6890 2024-02-19T14:52:20.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.989+0000 7f9148b8e640 1 -- 172.21.15.78:0/1163411024 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9130003520 con 0x7f91440a6890 2024-02-19T14:52:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.993+0000 7f9148b8e640 1 -- 172.21.15.78:0/1163411024 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928105226 0 0) 0x7f9130003520 con 0x7f91440a6890 2024-02-19T14:52:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.993+0000 7f914a391640 1 -- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440c7a40 msgr2=0x7f9144158950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-19T14:52:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.993+0000 7f914a391640 1 --2- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440c7a40 0x7f9144158950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.993+0000 7f914a391640 1 -- 172.21.15.78:0/1163411024 >> v1:172.21.15.78:6789/0 conn(0x7f91440a6890 legacy=0x7f9144159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:20.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.993+0000 7f914a391640 1 -- 172.21.15.78:0/1163411024 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914415d070 con 0x7f91440721a0 2024-02-19T14:52:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:20.998+0000 7f914a391640 1 --2- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440721a0 0x7f9144158e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f91400097a0 tx=0x7f914002b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2454b47bd8d806c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.001+0000 7f9148b8e640 1 -- 172.21.15.78:0/1163411024 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c0019f0 con 0x7f91440721a0 2024-02-19T14:52:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.001+0000 7f9148b8e640 1 -- 172.21.15.78:0/1163411024 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914002e050 con 0x7f91440721a0 2024-02-19T14:52:21.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.001+0000 7f9148b8e640 1 -- 172.21.15.78:0/1163411024 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140030730 con 0x7f91440721a0 2024-02-19T14:52:21.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.005+0000 7f914bdf4640 1 -- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440721a0 msgr2=0x7f9144158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.005+0000 7f914bdf4640 1 --2- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440721a0 0x7f9144158e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f91400097a0 tx=0x7f914002b680 comp rx=0 tx=0).stop 2024-02-19T14:52:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.012+0000 7f914bdf4640 1 -- 172.21.15.78:0/1163411024 shutdown_connections 2024-02-19T14:52:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.012+0000 7f914bdf4640 1 --2- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440721a0 0x7f9144158e90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.012+0000 7f914bdf4640 1 --2- 172.21.15.78:0/1163411024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440c7a40 0x7f9144158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.012+0000 7f914bdf4640 1 -- 172.21.15.78:0/1163411024 >> 172.21.15.78:0/1163411024 conn(0x7f914406c170 msgr2=0x7f914406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.025+0000 7f914bdf4640 1 -- 172.21.15.78:0/1163411024 shutdown_connections 2024-02-19T14:52:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.026+0000 7f914bdf4640 1 -- 172.21.15.78:0/1163411024 wait complete. 2024-02-19T14:52:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.027+0000 7f914bdf4640 1 Processor -- start 2024-02-19T14:52:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.027+0000 7f914bdf4640 1 -- start start 2024-02-19T14:52:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.028+0000 7f914bdf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440721a0 0x7f91440bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.028+0000 7f914bdf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 0x7f91440c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914bdf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440c7a40 0x7f91441538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914bdf4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91440befb0 con 0x7f91440a6890 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914bdf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914414f280 con 0x7f91440c7a40 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914bdf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914414f560 con 0x7f91440721a0 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914938f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 0x7f91440c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914938f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 0x7f91440c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52644/0 (socket says 172.21.15.78:52644) 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914938f640 1 -- 172.21.15.78:0/2316534234 learned_addr learned my addr 172.21.15.78:0/2316534234 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f9149b90640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440721a0 0x7f91440bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914938f640 1 -- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440721a0 msgr2=0x7f91440bea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.029+0000 7f914a391640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440c7a40 0x7f91441538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.030+0000 7f914938f640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440721a0 0x7f91440bea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.031+0000 7f914938f640 1 -- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440c7a40 msgr2=0x7f91441538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.031+0000 7f914938f640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440c7a40 0x7f91441538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.031+0000 7f914938f640 1 -- 172.21.15.78:0/2316534234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140009400 con 0x7f91440a6890 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.032+0000 7f914938f640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 0x7f91440c42d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f913c001f70 tx=0x7f913c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.032+0000 7f914a391640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440c7a40 0x7f91441538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.032+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c012400 con 0x7f91440a6890 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.032+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914414f840 con 0x7f91440a6890 2024-02-19T14:52:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.032+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914414fe00 con 0x7f91440a6890 2024-02-19T14:52:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.033+0000 7f9149b90640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440721a0 0x7f91440bea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.036+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913c012560 con 0x7f91440a6890 2024-02-19T14:52:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.036+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c012930 con 0x7f91440a6890 2024-02-19T14:52:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.036+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f913c012b50 con 0x7f91440a6890 2024-02-19T14:52:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.037+0000 7f913a7fc640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f912806b940 0x7f912806de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.037+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f913c08a6f0 con 0x7f91440a6890 2024-02-19T14:52:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.038+0000 7f9149b90640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f912806b940 0x7f912806de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.038+0000 7f9149b90640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f912806b940 0x7f912806de00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f91440c4d20 tx=0x7f9134008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.039+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9144074990 con 0x7f91440a6890 2024-02-19T14:52:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.048+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f913c054ef0 con 0x7f91440a6890 2024-02-19T14:52:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.052+0000 7f2cf3fff640 1 Processor -- start 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.052+0000 7f2cf3fff640 1 -- start start 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.053+0000 7f2cf3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2cf40c1e90 0x7f2cf40c2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.053+0000 7f2cf3fff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf406faf0 con 0x7f2cf406c790 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.053+0000 7f2cf3fff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf40c1b60 con 0x7f2cf4155780 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.053+0000 7f2cf3fff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf406eee0 con 0x7f2cf40c1e90 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.054+0000 7f2cf37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2cf4155780 0x7f2cf415c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50372/0 (socket says 172.21.15.78:50372) 2024-02-19T14:52:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.054+0000 7f2cf37fe640 1 -- 172.21.15.78:0/2516020388 learned_addr learned my addr 172.21.15.78:0/2516020388 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.054+0000 7f2cf2ffd640 1 --2- 172.21.15.78:0/2516020388 >> v2:172.21.15.150:3300/0 conn(0x7f2cf40c1e90 0x7f2cf40c2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.056+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102268006 0 0) 0x7f2cf40c1b60 con 0x7f2cf4155780 2024-02-19T14:52:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cdc003520 con 0x7f2cf4155780 2024-02-19T14:52:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133868832 0 0) 0x7f2cf406faf0 con 0x7f2cf406c790 2024-02-19T14:52:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf40c1b60 con 0x7f2cf406c790 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285081793 0 0) 0x7f2cf40c1b60 con 0x7f2cf406c790 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 >> v2:172.21.15.150:3300/0 conn(0x7f2cf40c1e90 msgr2=0x7f2cf40c2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 --2- 172.21.15.78:0/2516020388 >> v2:172.21.15.150:3300/0 conn(0x7f2cf40c1e90 0x7f2cf40c2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 >> v1:172.21.15.150:6789/0 conn(0x7f2cf4155780 legacy=0x7f2cf415c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf406f1c0 con 0x7f2cf406c790 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.057+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2cec0018a0 con 0x7f2cf406c790 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 >> v1:172.21.15.78:6789/0 conn(0x7f2cf406c790 legacy=0x7f2cf4155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc003ba0 0x7f2cdc003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cdc0047c0 0x7f2cdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cdc0051f0 0x7f2cdc00ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf406eee0 con 0x7f2cdc0047c0 2024-02-19T14:52:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc00c1d0 con 0x7f2cdc003ba0 2024-02-19T14:52:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf27fc640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cdc0047c0 0x7f2cdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.058+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc00c390 con 0x7f2cdc0051f0 2024-02-19T14:52:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.059+0000 7f2cf27fc640 1 -- 172.21.15.78:0/2516020388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cdc0051f0 msgr2=0x7f2cdc00ba00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:52:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.059+0000 7f2cf27fc640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cdc0051f0 0x7f2cdc00ba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.059+0000 7f2cf27fc640 1 -- 172.21.15.78:0/2516020388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc003ba0 msgr2=0x7f2cdc003fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-19T14:52:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.059+0000 7f2cf27fc640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc003ba0 0x7f2cdc003fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.059+0000 7f2cf27fc640 1 -- 172.21.15.78:0/2516020388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40c1b60 con 0x7f2cdc0047c0 2024-02-19T14:52:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.061+0000 7f2cf37fe640 1 -- 172.21.15.78:0/2516020388 reap_dead start 2024-02-19T14:52:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.061+0000 7f2cf27fc640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cdc0047c0 0x7f2cdc004c40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00c210 tx=0x7f2cdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4581965967a0433 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.061+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec02cb20 con 0x7f2cdc0047c0 2024-02-19T14:52:21.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.062+0000 7f2cf1ffb640 1 -- 172.21.15.78:0/2516020388 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec02cca0 con 0x7f2cdc0047c0 2024-02-19T14:52:21.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 -- 172.21.15.78:0/2516020388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cdc0047c0 msgr2=0x7f2cdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cdc0047c0 0x7f2cdc004c40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00c210 tx=0x7f2cdc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 -- 172.21.15.78:0/2516020388 shutdown_connections 2024-02-19T14:52:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/2516020388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cdc0047c0 0x7f2cdc004c40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 -- 172.21.15.78:0/2516020388 >> 172.21.15.78:0/2516020388 conn(0x7f2cf40c7490 msgr2=0x7f2cf40c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 -- 172.21.15.78:0/2516020388 shutdown_connections 2024-02-19T14:52:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.063+0000 7f2cf3fff640 1 -- 172.21.15.78:0/2516020388 wait complete. 2024-02-19T14:52:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.066+0000 7f2cf3fff640 1 Processor -- start 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.066+0000 7f2cf3fff640 1 -- start start 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc0047c0 0x7f2cf41e9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf41ea260 0x7f2cf40c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf3fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 0x7f2cf40c1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf3fff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf414f4c0 con 0x7f2cf41ea260 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf3fff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4150a50 con 0x7f2cdc0047c0 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf3fff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf41ea7a0 con 0x7f2cf41452b0 2024-02-19T14:52:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 0x7f2cf40c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 0x7f2cf40c1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51158/0 (socket says 172.21.15.78:51158) 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf37fe640 1 -- 172.21.15.78:0/1315020147 learned_addr learned my addr 172.21.15.78:0/1315020147 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.067+0000 7f2cf2ffd640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc0047c0 0x7f2cf41e9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.068+0000 7f2cf37fe640 1 -- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc0047c0 msgr2=0x7f2cf41e9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.068+0000 7f2cf37fe640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc0047c0 0x7f2cf41e9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.068+0000 7f2cf37fe640 1 -- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf41ea260 msgr2=0x7f2cf40c0ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.068+0000 7f2cf37fe640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf41ea260 0x7f2cf40c0ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.068+0000 7f2cf37fe640 1 -- 172.21.15.78:0/1315020147 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf414f980 con 0x7f2cf41452b0 2024-02-19T14:52:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.068+0000 7f2cf37fe640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 0x7f2cf40c1020 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40c3790 tx=0x7f2ce400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.073+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4005ce0 con 0x7f2cf41452b0 2024-02-19T14:52:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.073+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce4005e40 con 0x7f2cf41452b0 2024-02-19T14:52:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.073+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4007290 con 0x7f2cf41452b0 2024-02-19T14:52:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.073+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf414fc60 con 0x7f2cf41452b0 2024-02-19T14:52:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.073+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf4144d60 con 0x7f2cf41452b0 2024-02-19T14:52:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.075+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd4005290 con 0x7f2cf41452b0 2024-02-19T14:52:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.083+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2ce40073f0 con 0x7f2cf41452b0 2024-02-19T14:52:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.083+0000 7f2cdb7fe640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ce006b910 0x7f2ce006ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.084+0000 7f2cf2ffd640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ce006b910 0x7f2ce006ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.084+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2ce4089eb0 con 0x7f2cf41452b0 2024-02-19T14:52:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.084+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ce408a330 con 0x7f2cf41452b0 2024-02-19T14:52:21.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.098+0000 7f2cf2ffd640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ce006b910 0x7f2ce006ddd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80043f0 tx=0x7f2ce8014040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.266+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcef4005540 con 0x7fcf100c3990 2024-02-19T14:52:21.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.267+0000 7fcefeffd640 1 -- 172.21.15.78:0/945256411 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf00045030 con 0x7fcf100c3990 2024-02-19T14:52:21.271 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:21.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.269+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b8e0 msgr2=0x7fcef006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b8e0 0x7fcef006dda0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf1006d8a0 tx=0x7fcf0400c040 comp rx=0 tx=0).stop 2024-02-19T14:52:21.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf100c3990 msgr2=0x7fcf101dc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf100c3990 0x7fcf101dc7c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf00034000 tx=0x7fcf00004ab0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 shutdown_connections 2024-02-19T14:52:21.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcef006b8e0 0x7fcef006dda0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1014e800 0x7fcf100a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf100c47e0 0x7fcf100a9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 --2- 172.21.15.78:0/945256411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf100c3990 0x7fcf101dc7c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 >> 172.21.15.78:0/945256411 conn(0x7fcf1006b270 msgr2=0x7fcf1014da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 shutdown_connections 2024-02-19T14:52:21.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.270+0000 7fcf14847640 1 -- 172.21.15.78:0/945256411 wait complete. 2024-02-19T14:52:21.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-02-19T14:52:21.290 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.310+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f57e8005810 con 0x7f580c156880 2024-02-19T14:52:21.317 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.311+0000 7f58027fc640 1 -- 172.21.15.78:0/2254585012 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f580407b820 con 0x7f580c156880 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.314+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57f006b8e0 msgr2=0x7f57f006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.314+0000 7f57effff640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57f006b8e0 0x7f57f006dda0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f580c0c6680 tx=0x7f5808009060 comp rx=0 tx=0).stop 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.314+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 msgr2=0x7f580c157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.314+0000 7f57effff640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 0x7f580c157510 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5804036040 tx=0x7f5804004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.315+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 shutdown_connections 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.315+0000 7f57effff640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57f006b8e0 0x7f57f006dda0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.315+0000 7f57effff640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f580c156880 0x7f580c157510 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.315+0000 7f57effff640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f580c0ca230 0x7f580c0c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.315+0000 7f57effff640 1 --2- 172.21.15.78:0/2254585012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f580c0c9220 0x7f580c1490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.315+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 >> 172.21.15.78:0/2254585012 conn(0x7f580c0ca6c0 msgr2=0x7f580c06d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.316+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 shutdown_connections 2024-02-19T14:52:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.316+0000 7f57effff640 1 -- 172.21.15.78:0/2254585012 wait complete. 2024-02-19T14:52:21.322 INFO:teuthology.orchestra.run.smithi078.stdout:55834574859 2024-02-19T14:52:21.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.318+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1b74005810 con 0x7f1b8c15ca30 2024-02-19T14:52:21.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.318+0000 7f1b827fc640 1 -- 172.21.15.78:0/1741122415 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b84018020 con 0x7f1b8c15ca30 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.322+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b7006b810 msgr2=0x7f1b7006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.322+0000 7f1b941ef640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b7006b810 0x7f1b7006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c034000 tx=0x7f1b7c008030 comp rx=0 tx=0).stop 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.322+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b8c15ca30 msgr2=0x7f1b8c15b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.322+0000 7f1b941ef640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b8c15ca30 0x7f1b8c15b4d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400c730 tx=0x7f1b8400cbf0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 shutdown_connections 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b7006b810 0x7f1b7006dcd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b8c15ca30 0x7f1b8c15b4d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b8c158df0 0x7f1b8c1fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 --2- 172.21.15.78:0/1741122415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b8c0c14c0 0x7f1b8c0a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 >> 172.21.15.78:0/1741122415 conn(0x7f1b8c06b270 msgr2=0x7f1b8c157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 shutdown_connections 2024-02-19T14:52:21.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.323+0000 7f1b941ef640 1 -- 172.21.15.78:0/1741122415 wait complete. 2024-02-19T14:52:21.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.4 2024-02-19T14:52:21.339 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-02-19T14:52:21.342 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.346+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7efc580057d0 con 0x7efc58006630 2024-02-19T14:52:21.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.346+0000 7efc5d7fa640 1 -- 172.21.15.78:0/1464122813 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc4c058fa0 con 0x7efc58006630 2024-02-19T14:52:21.350 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:21.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe648005540 con 0x7fe66004d410 2024-02-19T14:52:21.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7fe664ff9640 1 -- 172.21.15.78:0/1167022575 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe658018020 con 0x7fe66004d410 2024-02-19T14:52:21.352 INFO:teuthology.orchestra.run.smithi078.stdout:55834574859 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4406b910 msgr2=0x7efc4406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4406b910 0x7efc4406ddd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efc54009870 tx=0x7efc54008030 comp rx=0 tx=0).stop 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 msgr2=0x7efc58177bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 0x7efc58177bf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00bfd0 tx=0x7efc4c00dda0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 shutdown_connections 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4406b910 0x7efc4406ddd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc58048490 0x7efc580d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5803f020 0x7efc580e0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 --2- 172.21.15.78:0/1464122813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58006630 0x7efc58177bf0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 >> 172.21.15.78:0/1464122813 conn(0x7efc5802db80 msgr2=0x7efc580ddd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 shutdown_connections 2024-02-19T14:52:21.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.349+0000 7efc65de5640 1 -- 172.21.15.78:0/1464122813 wait complete. 2024-02-19T14:52:21.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe65406b8e0 msgr2=0x7fe65406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe65406b8e0 0x7fe65406dda0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe66004e950 tx=0x7fe6680ab000 comp rx=0 tx=0).stop 2024-02-19T14:52:21.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 msgr2=0x7fe6600081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 0x7fe6600081d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe65800c460 tx=0x7fe65800c920 comp rx=0 tx=0).stop 2024-02-19T14:52:21.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 shutdown_connections 2024-02-19T14:52:21.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe65406b8e0 0x7fe65406dda0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6600d5940 0x7fe660191210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe66004d410 0x7fe6600081d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 --2- 172.21.15.78:0/1167022575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe66004b8b0 0x7fe66000d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.351+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 >> 172.21.15.78:0/1167022575 conn(0x7fe66002f4a0 msgr2=0x7fe6600d2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.352+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 shutdown_connections 2024-02-19T14:52:21.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.352+0000 7fe66d272640 1 -- 172.21.15.78:0/1167022575 wait complete. 2024-02-19T14:52:21.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2024-02-19T14:52:21.365 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2024-02-19T14:52:21.366 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.398+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9144072810 con 0x7f91440a6890 2024-02-19T14:52:21.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.399+0000 7f913a7fc640 1 -- 172.21.15.78:0/2316534234 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f913c017090 con 0x7f91440a6890 2024-02-19T14:52:21.402 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f912806b940 msgr2=0x7f912806de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f912806b940 0x7f912806de00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f91440c4d20 tx=0x7f9134008040 comp rx=0 tx=0).stop 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 msgr2=0x7f91440c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 0x7f91440c42d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f913c001f70 tx=0x7f913c00a430 comp rx=0 tx=0).stop 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 shutdown_connections 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f912806b940 0x7f912806de00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91440a6890 0x7f91440c42d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91440c7a40 0x7f91441538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 --2- 172.21.15.78:0/2316534234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91440721a0 0x7f91440bea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 >> 172.21.15.78:0/2316534234 conn(0x7f914406c170 msgr2=0x7f914406af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 shutdown_connections 2024-02-19T14:52:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.401+0000 7f914bdf4640 1 -- 172.21.15.78:0/2316534234 wait complete. 2024-02-19T14:52:21.419 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-02-19T14:52:21.419 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.458+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2cd4005810 con 0x7f2cf41452b0 2024-02-19T14:52:21.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.458+0000 7f2cdb7fe640 1 -- 172.21.15.78:0/1315020147 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ce4054600 con 0x7f2cf41452b0 2024-02-19T14:52:21.462 INFO:teuthology.orchestra.run.smithi078.stdout:55834574857 2024-02-19T14:52:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ce006b910 msgr2=0x7f2ce006ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ce006b910 0x7f2ce006ddd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80043f0 tx=0x7f2ce8014040 comp rx=0 tx=0).stop 2024-02-19T14:52:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 msgr2=0x7f2cf40c1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 0x7f2cf40c1020 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40c3790 tx=0x7f2ce400c820 comp rx=0 tx=0).stop 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 shutdown_connections 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ce006b910 0x7f2ce006ddd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cf41ea260 0x7f2cf40c0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cf41452b0 0x7f2cf40c1020 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 --2- 172.21.15.78:0/1315020147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cdc0047c0 0x7f2cf41e9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 >> 172.21.15.78:0/1315020147 conn(0x7f2cf40c7490 msgr2=0x7f2cf40c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 shutdown_connections 2024-02-19T14:52:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.461+0000 7f2cf3fff640 1 -- 172.21.15.78:0/1315020147 wait complete. 2024-02-19T14:52:21.475 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-02-19T14:52:21.475 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.493+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f8c24005540 con 0x7f8c4c0c2f60 2024-02-19T14:52:21.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.493+0000 7f8c497fa640 1 -- 172.21.15.78:0/1357822684 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c34058510 con 0x7f8c4c0c2f60 2024-02-19T14:52:21.497 INFO:teuthology.orchestra.run.smithi078.stdout:55834574858 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8c2c06b810 msgr2=0x7f8c2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8c2c06b810 0x7f8c2c06dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c44034000 tx=0x7f8c44008030 comp rx=0 tx=0).stop 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 msgr2=0x7f8c4c0ad270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c0ad270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400e9b0 tx=0x7f8c3400ee70 comp rx=0 tx=0).stop 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 shutdown_connections 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8c2c06b810 0x7f8c2c06dcd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c156920 0x7f8c4c0ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c4c0c2f60 0x7f8c4c0ad270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 --2- 172.21.15.78:0/1357822684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c06f530 0x7f8c4c0c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 >> 172.21.15.78:0/1357822684 conn(0x7f8c4c0c85e0 msgr2=0x7f8c4c158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 shutdown_connections 2024-02-19T14:52:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.496+0000 7f8c42ffd640 1 -- 172.21.15.78:0/1357822684 wait complete. 2024-02-19T14:52:21.514 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-02-19T14:52:21.514 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:21.514 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-19T14:52:21.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:52:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.652+0000 7f1b12cc3640 1 Processor -- start 2024-02-19T14:52:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.652+0000 7f1b12cc3640 1 -- start start 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b12cc3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b0c14ef50 0x7f1b0c14f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b12cc3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b0c0727d0 0x7f1b0c072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b12cc3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c0c35c0 con 0x7f1b0c0c78a0 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b12cc3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c0c4bc0 con 0x7f1b0c14ef50 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b12cc3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c14f920 con 0x7f1b0c0727d0 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b0c14ef50 0x7f1b0c14f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b0c14ef50 0x7f1b0c14f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52660/0 (socket says 172.21.15.78:52660) 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b10a5f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1b0c0c78a0 0x7f1b0c0c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57812/0 (socket says 172.21.15.78:57812) 2024-02-19T14:52:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 -- 172.21.15.78:0/1159077890 learned_addr learned my addr 172.21.15.78:0/1159077890 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b11260640 1 --2- 172.21.15.78:0/1159077890 >> v2:172.21.15.150:3300/0 conn(0x7f1b0c0727d0 0x7f1b0c072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 -- 172.21.15.78:0/1159077890 >> v2:172.21.15.150:3300/0 conn(0x7f1b0c0727d0 msgr2=0x7f1b0c072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 --2- 172.21.15.78:0/1159077890 >> v2:172.21.15.150:3300/0 conn(0x7f1b0c0727d0 0x7f1b0c072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 -- 172.21.15.78:0/1159077890 >> v1:172.21.15.78:6789/0 conn(0x7f1b0c0c78a0 legacy=0x7f1b0c0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0bfff640 1 -- 172.21.15.78:0/1159077890 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0c38a0 con 0x7f1b0c14ef50 2024-02-19T14:52:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.653+0000 7f1b0b7fe640 1 -- 172.21.15.78:0/1159077890 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440439836 0 0) 0x7f1b0c0c35c0 con 0x7f1b0c0c78a0 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.654+0000 7f1b0bfff640 1 --2- 172.21.15.78:0/1159077890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c14f3e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0a8ea0 tx=0x7f1b0002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c1f5cfdedb3a9e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.654+0000 7f1b0b7fe640 1 -- 172.21.15.78:0/1159077890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00036440 con 0x7f1b0c14ef50 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.654+0000 7f1b0b7fe640 1 -- 172.21.15.78:0/1159077890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b000365a0 con 0x7f1b0c14ef50 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.654+0000 7f1b0b7fe640 1 -- 172.21.15.78:0/1159077890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00036930 con 0x7f1b0c14ef50 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 -- 172.21.15.78:0/1159077890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 msgr2=0x7f1b0c14f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/1159077890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c14f3e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0a8ea0 tx=0x7f1b0002e170 comp rx=0 tx=0).stop 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 -- 172.21.15.78:0/1159077890 shutdown_connections 2024-02-19T14:52:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/1159077890 >> v2:172.21.15.150:3300/0 conn(0x7f1b0c0727d0 0x7f1b0c072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/1159077890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c14f3e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 -- 172.21.15.78:0/1159077890 >> 172.21.15.78:0/1159077890 conn(0x7f1b0c066e70 msgr2=0x7f1b0c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 -- 172.21.15.78:0/1159077890 shutdown_connections 2024-02-19T14:52:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.655+0000 7f1b12cc3640 1 -- 172.21.15.78:0/1159077890 wait complete. 2024-02-19T14:52:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.656+0000 7f1b12cc3640 1 Processor -- start 2024-02-19T14:52:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.657+0000 7f1b12cc3640 1 -- start start 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b12cc3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c0727d0 0x7f1b0c150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b12cc3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b0c0c78a0 0x7f1b0c142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b12cc3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b12cc3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0c1090 con 0x7f1b0c14ef50 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b12cc3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0be550 con 0x7f1b0c0727d0 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b12cc3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0bd270 con 0x7f1b0c0c78a0 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52670/0 (socket says 172.21.15.78:52670) 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 -- 172.21.15.78:0/2600453959 learned_addr learned my addr 172.21.15.78:0/2600453959 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b10a5f640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c0727d0 0x7f1b0c150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b0bfff640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b0c0c78a0 0x7f1b0c142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 -- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b0c0c78a0 msgr2=0x7f1b0c142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b0c0c78a0 0x7f1b0c142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 -- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c0727d0 msgr2=0x7f1b0c150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c0727d0 0x7f1b0c150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 -- 172.21.15.78:0/2600453959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0000b770 con 0x7f1b0c14ef50 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.658+0000 7f1b11260640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c142f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1b04007e60 tx=0x7f1b0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.659+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04014420 con 0x7f1b0c14ef50 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.659+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b040145a0 con 0x7f1b0c14ef50 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.659+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c0bd550 con 0x7f1b0c14ef50 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.659+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04014950 con 0x7f1b0c14ef50 2024-02-19T14:52:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.659+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c0bdb10 con 0x7f1b0c14ef50 2024-02-19T14:52:21.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.660+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c0bdd00 con 0x7f1b0c14ef50 2024-02-19T14:52:21.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.660+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1b04014ab0 con 0x7f1b0c14ef50 2024-02-19T14:52:21.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.660+0000 7f1b097fa640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1af806b840 0x7f1af806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:21.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.660+0000 7f1b10a5f640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1af806b840 0x7f1af806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:21.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.660+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1b0408ad00 con 0x7f1b0c14ef50 2024-02-19T14:52:21.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.661+0000 7f1b10a5f640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1af806b840 0x7f1af806dd00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000f60 tx=0x7f1afc007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:21.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.665+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b04055490 con 0x7f1b0c14ef50 2024-02-19T14:52:21.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.838+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b0c0c3e60 con 0x7f1af806b840 2024-02-19T14:52:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.847+0000 7f1b097fa640 1 -- 172.21.15.78:0/2600453959 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345262 (secure 0 0 0) 0x7f1afc004c70 con 0x7f1af806b840 2024-02-19T14:52:21.855 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:21.857 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1af806b840 msgr2=0x7f1af806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1af806b840 0x7f1af806dd00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000f60 tx=0x7f1afc007d50 comp rx=0 tx=0).stop 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 msgr2=0x7f1b0c142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c142f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1b04007e60 tx=0x7f1b0400e4b0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 shutdown_connections 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1af806b840 0x7f1af806dd00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b0c14ef50 0x7f1b0c142f80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b0c0c78a0 0x7f1b0c142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 --2- 172.21.15.78:0/2600453959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c0727d0 0x7f1b0c150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 >> 172.21.15.78:0/2600453959 conn(0x7f1b0c066e70 msgr2=0x7f1b0c06af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:21.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 shutdown_connections 2024-02-19T14:52:21.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:21.851+0000 7f1b12cc3640 1 -- 172.21.15.78:0/2600453959 wait complete. 2024-02-19T14:52:21.865 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-02-19T14:52:20.063774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219380,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755340,"statfs":{"total":773094113280,"available":772869468160,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002403"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226366+0000","last_change":"2024-02-19T14:51:59.266929+0000","last_active":"2024-02-19T14:52:06.226366+0000","last_peered":"2024-02-19T14:52:06.226366+0000","last_clean":"2024-02-19T14:52:06.226366+0000","last_became_active":"2024-02-19T14:51:59.266644+0000","last_became_peered":"2024-02-19T14:51:59.266644+0000","last_unstale":"2024-02-19T14:52:06.226366+0000","last_undegraded":"2024-02-19T14:52:06.226366+0000","last_fullsized":"2024-02-19T14:52:06.226366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335294+0000","last_change":"2024-02-19T14:52:00.264687+0000","last_active":"2024-02-19T14:52:07.335294+0000","last_peered":"2024-02-19T14:52:07.335294+0000","last_clean":"2024-02-19T14:52:07.335294+0000","last_became_active":"2024-02-19T14:52:00.264424+0000","last_became_peered":"2024-02-19T14:52:00.264424+0000","last_unstale":"2024-02-19T14:52:07.335294+0000","last_undegraded":"2024-02-19T14:52:07.335294+0000","last_fullsized":"2024-02-19T14:52:07.335294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334297+0000","last_change":"2024-02-19T14:52:04.435336+0000","last_active":"2024-02-19T14:52:07.334297+0000","last_peered":"2024-02-19T14:52:07.334297+0000","last_clean":"2024-02-19T14:52:07.334297+0000","last_became_active":"2024-02-19T14:52:04.434977+0000","last_became_peered":"2024-02-19T14:52:04.434977+0000","last_unstale":"2024-02-19T14:52:07.334297+0000","last_undegraded":"2024-02-19T14:52:07.334297+0000","last_fullsized":"2024-02-19T14:52:07.334297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334116+0000","last_change":"2024-02-19T14:51:59.262430+0000","last_active":"2024-02-19T14:52:07.334116+0000","last_peered":"2024-02-19T14:52:07.334116+0000","last_clean":"2024-02-19T14:52:07.334116+0000","last_became_active":"2024-02-19T14:51:59.262168+0000","last_became_peered":"2024-02-19T14:51:59.262168+0000","last_unstale":"2024-02-19T14:52:07.334116+0000","last_undegraded":"2024-02-19T14:52:07.334116+0000","last_fullsized":"2024-02-19T14:52:07.334116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226267+0000","last_change":"2024-02-19T14:52:00.263014+0000","last_active":"2024-02-19T14:52:06.226267+0000","last_peered":"2024-02-19T14:52:06.226267+0000","last_clean":"2024-02-19T14:52:06.226267+0000","last_became_active":"2024-02-19T14:52:00.262531+0000","last_became_peered":"2024-02-19T14:52:00.262531+0000","last_unstale":"2024-02-19T14:52:06.226267+0000","last_undegraded":"2024-02-19T14:52:06.226267+0000","last_fullsized":"2024-02-19T14:52:06.226267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282898+0000","last_change":"2024-02-19T14:52:04.277572+0000","last_active":"2024-02-19T14:52:05.282898+0000","last_peered":"2024-02-19T14:52:05.282898+0000","last_clean":"2024-02-19T14:52:05.282898+0000","last_became_active":"2024-02-19T14:52:04.276970+0000","last_became_peered":"2024-02-19T14:52:04.276970+0000","last_unstale":"2024-02-19T14:52:05.282898+0000","last_undegraded":"2024-02-19T14:52:05.282898+0000","last_fullsized":"2024-02-19T14:52:05.282898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:14.095185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247290+0000","last_change":"2024-02-19T14:51:59.265003+0000","last_active":"2024-02-19T14:52:06.247290+0000","last_peered":"2024-02-19T14:52:06.247290+0000","last_clean":"2024-02-19T14:52:06.247290+0000","last_became_active":"2024-02-19T14:51:59.264719+0000","last_became_peered":"2024-02-19T14:51:59.264719+0000","last_unstale":"2024-02-19T14:52:06.247290+0000","last_undegraded":"2024-02-19T14:52:06.247290+0000","last_fullsized":"2024-02-19T14:52:06.247290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283484+0000","last_change":"2024-02-19T14:52:00.252606+0000","last_active":"2024-02-19T14:52:05.283484+0000","last_peered":"2024-02-19T14:52:05.283484+0000","last_clean":"2024-02-19T14:52:05.283484+0000","last_became_active":"2024-02-19T14:52:00.249782+0000","last_became_peered":"2024-02-19T14:52:00.249782+0000","last_unstale":"2024-02-19T14:52:05.283484+0000","last_undegraded":"2024-02-19T14:52:05.283484+0000","last_fullsized":"2024-02-19T14:52:05.283484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283979+0000","last_change":"2024-02-19T14:52:04.278014+0000","last_active":"2024-02-19T14:52:05.283979+0000","last_peered":"2024-02-19T14:52:05.283979+0000","last_clean":"2024-02-19T14:52:05.283979+0000","last_became_active":"2024-02-19T14:52:04.277126+0000","last_became_peered":"2024-02-19T14:52:04.277126+0000","last_unstale":"2024-02-19T14:52:05.283979+0000","last_undegraded":"2024-02-19T14:52:05.283979+0000","last_fullsized":"2024-02-19T14:52:05.283979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:27.674760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247080+0000","last_change":"2024-02-19T14:51:59.263462+0000","last_active":"2024-02-19T14:52:06.247080+0000","last_peered":"2024-02-19T14:52:06.247080+0000","last_clean":"2024-02-19T14:52:06.247080+0000","last_became_active":"2024-02-19T14:51:59.262932+0000","last_became_peered":"2024-02-19T14:51:59.262932+0000","last_unstale":"2024-02-19T14:52:06.247080+0000","last_undegraded":"2024-02-19T14:52:06.247080+0000","last_fullsized":"2024-02-19T14:52:06.247080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284101+0000","last_change":"2024-02-19T14:52:00.252460+0000","last_active":"2024-02-19T14:52:05.284101+0000","last_peered":"2024-02-19T14:52:05.284101+0000","last_clean":"2024-02-19T14:52:05.284101+0000","last_became_active":"2024-02-19T14:52:00.251603+0000","last_became_peered":"2024-02-19T14:52:00.251603+0000","last_unstale":"2024-02-19T14:52:05.284101+0000","last_undegraded":"2024-02-19T14:52:05.284101+0000","last_fullsized":"2024-02-19T14:52:05.284101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283155+0000","last_change":"2024-02-19T14:52:04.278038+0000","last_active":"2024-02-19T14:52:05.283155+0000","last_peered":"2024-02-19T14:52:05.283155+0000","last_clean":"2024-02-19T14:52:05.283155+0000","last_became_active":"2024-02-19T14:52:04.277620+0000","last_became_peered":"2024-02-19T14:52:04.277620+0000","last_unstale":"2024-02-19T14:52:05.283155+0000","last_undegraded":"2024-02-19T14:52:05.283155+0000","last_fullsized":"2024-02-19T14:52:05.283155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:12.987808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530027+0000","last_change":"2024-02-19T14:52:00.264133+0000","last_active":"2024-02-19T14:52:05.530027+0000","last_peered":"2024-02-19T14:52:05.530027+0000","last_clean":"2024-02-19T14:52:05.530027+0000","last_became_active":"2024-02-19T14:52:00.263475+0000","last_became_peered":"2024-02-19T14:52:00.263475+0000","last_unstale":"2024-02-19T14:52:05.530027+0000","last_undegraded":"2024-02-19T14:52:05.530027+0000","last_fullsized":"2024-02-19T14:52:05.530027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247110+0000","last_change":"2024-02-19T14:51:59.263095+0000","last_active":"2024-02-19T14:52:06.247110+0000","last_peered":"2024-02-19T14:52:06.247110+0000","last_clean":"2024-02-19T14:52:06.247110+0000","last_became_active":"2024-02-19T14:51:59.262392+0000","last_became_peered":"2024-02-19T14:51:59.262392+0000","last_unstale":"2024-02-19T14:52:06.247110+0000","last_undegraded":"2024-02-19T14:52:06.247110+0000","last_fullsized":"2024-02-19T14:52:06.247110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:04.278534+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:04.277850+0000","last_became_peered":"2024-02-19T14:52:04.277850+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334968+0000","last_change":"2024-02-19T14:52:00.268782+0000","last_active":"2024-02-19T14:52:07.334968+0000","last_peered":"2024-02-19T14:52:07.334968+0000","last_clean":"2024-02-19T14:52:07.334968+0000","last_became_active":"2024-02-19T14:52:00.266478+0000","last_became_peered":"2024-02-19T14:52:00.266478+0000","last_unstale":"2024-02-19T14:52:07.334968+0000","last_undegraded":"2024-02-19T14:52:07.334968+0000","last_fullsized":"2024-02-19T14:52:07.334968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228818+0000","last_change":"2024-02-19T14:51:59.260219+0000","last_active":"2024-02-19T14:52:07.228818+0000","last_peered":"2024-02-19T14:52:07.228818+0000","last_clean":"2024-02-19T14:52:07.228818+0000","last_became_active":"2024-02-19T14:51:59.259917+0000","last_became_peered":"2024-02-19T14:51:59.259917+0000","last_unstale":"2024-02-19T14:52:07.228818+0000","last_undegraded":"2024-02-19T14:52:07.228818+0000","last_fullsized":"2024-02-19T14:52:07.228818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282889+0000","last_change":"2024-02-19T14:52:04.277635+0000","last_active":"2024-02-19T14:52:05.282889+0000","last_peered":"2024-02-19T14:52:05.282889+0000","last_clean":"2024-02-19T14:52:05.282889+0000","last_became_active":"2024-02-19T14:52:04.277231+0000","last_became_peered":"2024-02-19T14:52:04.277231+0000","last_unstale":"2024-02-19T14:52:05.282889+0000","last_undegraded":"2024-02-19T14:52:05.282889+0000","last_fullsized":"2024-02-19T14:52:05.282889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:46.471360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334393+0000","last_change":"2024-02-19T14:52:00.263688+0000","last_active":"2024-02-19T14:52:07.334393+0000","last_peered":"2024-02-19T14:52:07.334393+0000","last_clean":"2024-02-19T14:52:07.334393+0000","last_became_active":"2024-02-19T14:52:00.263381+0000","last_became_peered":"2024-02-19T14:52:00.263381+0000","last_unstale":"2024-02-19T14:52:07.334393+0000","last_undegraded":"2024-02-19T14:52:07.334393+0000","last_fullsized":"2024-02-19T14:52:07.334393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228547+0000","last_change":"2024-02-19T14:51:59.262590+0000","last_active":"2024-02-19T14:52:07.228547+0000","last_peered":"2024-02-19T14:52:07.228547+0000","last_clean":"2024-02-19T14:52:07.228547+0000","last_became_active":"2024-02-19T14:51:59.262310+0000","last_became_peered":"2024-02-19T14:51:59.262310+0000","last_unstale":"2024-02-19T14:52:07.228547+0000","last_undegraded":"2024-02-19T14:52:07.228547+0000","last_fullsized":"2024-02-19T14:52:07.228547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229123+0000","last_change":"2024-02-19T14:52:04.434727+0000","last_active":"2024-02-19T14:52:07.229123+0000","last_peered":"2024-02-19T14:52:07.229123+0000","last_clean":"2024-02-19T14:52:07.229123+0000","last_became_active":"2024-02-19T14:52:04.432708+0000","last_became_peered":"2024-02-19T14:52:04.432708+0000","last_unstale":"2024-02-19T14:52:07.229123+0000","last_undegraded":"2024-02-19T14:52:07.229123+0000","last_fullsized":"2024-02-19T14:52:07.229123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:19.410143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284506+0000","last_change":"2024-02-19T14:52:00.260620+0000","last_active":"2024-02-19T14:52:05.284506+0000","last_peered":"2024-02-19T14:52:05.284506+0000","last_clean":"2024-02-19T14:52:05.284506+0000","last_became_active":"2024-02-19T14:52:00.260357+0000","last_became_peered":"2024-02-19T14:52:00.260357+0000","last_unstale":"2024-02-19T14:52:05.284506+0000","last_undegraded":"2024-02-19T14:52:05.284506+0000","last_fullsized":"2024-02-19T14:52:05.284506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:09.799959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246702+0000","last_change":"2024-02-19T14:51:59.267802+0000","last_active":"2024-02-19T14:52:06.246702+0000","last_peered":"2024-02-19T14:52:06.246702+0000","last_clean":"2024-02-19T14:52:06.246702+0000","last_became_active":"2024-02-19T14:51:59.267518+0000","last_became_peered":"2024-02-19T14:51:59.267518+0000","last_unstale":"2024-02-19T14:52:06.246702+0000","last_undegraded":"2024-02-19T14:52:06.246702+0000","last_fullsized":"2024-02-19T14:52:06.246702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:45.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283176+0000","last_change":"2024-02-19T14:52:04.276717+0000","last_active":"2024-02-19T14:52:05.283176+0000","last_peered":"2024-02-19T14:52:05.283176+0000","last_clean":"2024-02-19T14:52:05.283176+0000","last_became_active":"2024-02-19T14:52:04.276273+0000","last_became_peered":"2024-02-19T14:52:04.276273+0000","last_unstale":"2024-02-19T14:52:05.283176+0000","last_undegraded":"2024-02-19T14:52:05.283176+0000","last_fullsized":"2024-02-19T14:52:05.283176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:30.898405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226156+0000","last_change":"2024-02-19T14:52:00.259379+0000","last_active":"2024-02-19T14:52:06.226156+0000","last_peered":"2024-02-19T14:52:06.226156+0000","last_clean":"2024-02-19T14:52:06.226156+0000","last_became_active":"2024-02-19T14:52:00.259061+0000","last_became_peered":"2024-02-19T14:52:00.259061+0000","last_unstale":"2024-02-19T14:52:06.226156+0000","last_undegraded":"2024-02-19T14:52:06.226156+0000","last_fullsized":"2024-02-19T14:52:06.226156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:51:59.262618+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:51:59.262333+0000","last_became_peered":"2024-02-19T14:51:59.262333+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284070+0000","last_change":"2024-02-19T14:52:04.436152+0000","last_active":"2024-02-19T14:52:05.284070+0000","last_peered":"2024-02-19T14:52:05.284070+0000","last_clean":"2024-02-19T14:52:05.284070+0000","last_became_active":"2024-02-19T14:52:04.435251+0000","last_became_peered":"2024-02-19T14:52:04.435251+0000","last_unstale":"2024-02-19T14:52:05.284070+0000","last_undegraded":"2024-02-19T14:52:05.284070+0000","last_fullsized":"2024-02-19T14:52:05.284070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:33.762974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335436+0000","last_change":"2024-02-19T14:52:00.272464+0000","last_active":"2024-02-19T14:52:07.335436+0000","last_peered":"2024-02-19T14:52:07.335436+0000","last_clean":"2024-02-19T14:52:07.335436+0000","last_became_active":"2024-02-19T14:52:00.272184+0000","last_became_peered":"2024-02-19T14:52:00.272184+0000","last_unstale":"2024-02-19T14:52:07.335436+0000","last_undegraded":"2024-02-19T14:52:07.335436+0000","last_fullsized":"2024-02-19T14:52:07.335436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.256170+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.255855+0000","last_became_peered":"2024-02-19T14:51:59.255855+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334561+0000","last_change":"2024-02-19T14:52:04.433402+0000","last_active":"2024-02-19T14:52:07.334561+0000","last_peered":"2024-02-19T14:52:07.334561+0000","last_clean":"2024-02-19T14:52:07.334561+0000","last_became_active":"2024-02-19T14:52:04.432815+0000","last_became_peered":"2024-02-19T14:52:04.432815+0000","last_unstale":"2024-02-19T14:52:07.334561+0000","last_undegraded":"2024-02-19T14:52:07.334561+0000","last_fullsized":"2024-02-19T14:52:07.334561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:20.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228931+0000","last_change":"2024-02-19T14:52:00.261954+0000","last_active":"2024-02-19T14:52:07.228931+0000","last_peered":"2024-02-19T14:52:07.228931+0000","last_clean":"2024-02-19T14:52:07.228931+0000","last_became_active":"2024-02-19T14:52:00.261678+0000","last_became_peered":"2024-02-19T14:52:00.261678+0000","last_unstale":"2024-02-19T14:52:07.228931+0000","last_undegraded":"2024-02-19T14:52:07.228931+0000","last_fullsized":"2024-02-19T14:52:07.228931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247420+0000","last_change":"2024-02-19T14:51:59.267996+0000","last_active":"2024-02-19T14:52:06.247420+0000","last_peered":"2024-02-19T14:52:06.247420+0000","last_clean":"2024-02-19T14:52:06.247420+0000","last_became_active":"2024-02-19T14:51:59.267426+0000","last_became_peered":"2024-02-19T14:51:59.267426+0000","last_unstale":"2024-02-19T14:52:06.247420+0000","last_undegraded":"2024-02-19T14:52:06.247420+0000","last_fullsized":"2024-02-19T14:52:06.247420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225861+0000","last_change":"2024-02-19T14:52:04.433162+0000","last_active":"2024-02-19T14:52:06.225861+0000","last_peered":"2024-02-19T14:52:06.225861+0000","last_clean":"2024-02-19T14:52:06.225861+0000","last_became_active":"2024-02-19T14:52:04.432722+0000","last_became_peered":"2024-02-19T14:52:04.432722+0000","last_unstale":"2024-02-19T14:52:06.225861+0000","last_undegraded":"2024-02-19T14:52:06.225861+0000","last_fullsized":"2024-02-19T14:52:06.225861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:00.272249+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:00.266375+0000","last_became_peered":"2024-02-19T14:52:00.266375+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229033+0000","last_change":"2024-02-19T14:51:59.260559+0000","last_active":"2024-02-19T14:52:07.229033+0000","last_peered":"2024-02-19T14:52:07.229033+0000","last_clean":"2024-02-19T14:52:07.229033+0000","last_became_active":"2024-02-19T14:51:59.260290+0000","last_became_peered":"2024-02-19T14:51:59.260290+0000","last_unstale":"2024-02-19T14:52:07.229033+0000","last_undegraded":"2024-02-19T14:52:07.229033+0000","last_fullsized":"2024-02-19T14:52:07.229033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228634+0000","last_change":"2024-02-19T14:52:04.283819+0000","last_active":"2024-02-19T14:52:07.228634+0000","last_peered":"2024-02-19T14:52:07.228634+0000","last_clean":"2024-02-19T14:52:07.228634+0000","last_became_active":"2024-02-19T14:52:04.283469+0000","last_became_peered":"2024-02-19T14:52:04.283469+0000","last_unstale":"2024-02-19T14:52:07.228634+0000","last_undegraded":"2024-02-19T14:52:07.228634+0000","last_fullsized":"2024-02-19T14:52:07.228634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284410+0000","last_change":"2024-02-19T14:52:00.266384+0000","last_active":"2024-02-19T14:52:05.284410+0000","last_peered":"2024-02-19T14:52:05.284410+0000","last_clean":"2024-02-19T14:52:05.284410+0000","last_became_active":"2024-02-19T14:52:00.265954+0000","last_became_peered":"2024-02-19T14:52:00.265954+0000","last_unstale":"2024-02-19T14:52:05.284410+0000","last_undegraded":"2024-02-19T14:52:05.284410+0000","last_fullsized":"2024-02-19T14:52:05.284410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283363+0000","last_change":"2024-02-19T14:51:59.259377+0000","last_active":"2024-02-19T14:52:05.283363+0000","last_peered":"2024-02-19T14:52:05.283363+0000","last_clean":"2024-02-19T14:52:05.283363+0000","last_became_active":"2024-02-19T14:51:59.259004+0000","last_became_peered":"2024-02-19T14:51:59.259004+0000","last_unstale":"2024-02-19T14:52:05.283363+0000","last_undegraded":"2024-02-19T14:52:05.283363+0000","last_fullsized":"2024-02-19T14:52:05.283363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229000+0000","last_change":"2024-02-19T14:52:04.282027+0000","last_active":"2024-02-19T14:52:07.229000+0000","last_peered":"2024-02-19T14:52:07.229000+0000","last_clean":"2024-02-19T14:52:07.229000+0000","last_became_active":"2024-02-19T14:52:04.281679+0000","last_became_peered":"2024-02-19T14:52:04.281679+0000","last_unstale":"2024-02-19T14:52:07.229000+0000","last_undegraded":"2024-02-19T14:52:07.229000+0000","last_fullsized":"2024-02-19T14:52:07.229000+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:41.900564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228565+0000","last_change":"2024-02-19T14:52:00.261302+0000","last_active":"2024-02-19T14:52:07.228565+0000","last_peered":"2024-02-19T14:52:07.228565+0000","last_clean":"2024-02-19T14:52:07.228565+0000","last_became_active":"2024-02-19T14:52:00.261027+0000","last_became_peered":"2024-02-19T14:52:00.261027+0000","last_unstale":"2024-02-19T14:52:07.228565+0000","last_undegraded":"2024-02-19T14:52:07.228565+0000","last_fullsized":"2024-02-19T14:52:07.228565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281412+0000","last_change":"2024-02-19T14:51:59.256034+0000","last_active":"2024-02-19T14:52:05.281412+0000","last_peered":"2024-02-19T14:52:05.281412+0000","last_clean":"2024-02-19T14:52:05.281412+0000","last_became_active":"2024-02-19T14:51:59.255701+0000","last_became_peered":"2024-02-19T14:51:59.255701+0000","last_unstale":"2024-02-19T14:52:05.281412+0000","last_undegraded":"2024-02-19T14:52:05.281412+0000","last_fullsized":"2024-02-19T14:52:05.281412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229180+0000","last_change":"2024-02-19T14:52:04.283973+0000","last_active":"2024-02-19T14:52:07.229180+0000","last_peered":"2024-02-19T14:52:07.229180+0000","last_clean":"2024-02-19T14:52:07.229180+0000","last_became_active":"2024-02-19T14:52:04.283562+0000","last_became_peered":"2024-02-19T14:52:04.283562+0000","last_unstale":"2024-02-19T14:52:07.229180+0000","last_undegraded":"2024-02-19T14:52:07.229180+0000","last_fullsized":"2024-02-19T14:52:07.229180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225524+0000","last_change":"2024-02-19T14:52:00.260330+0000","last_active":"2024-02-19T14:52:06.225524+0000","last_peered":"2024-02-19T14:52:06.225524+0000","last_clean":"2024-02-19T14:52:06.225524+0000","last_became_active":"2024-02-19T14:52:00.260039+0000","last_became_peered":"2024-02-19T14:52:00.260039+0000","last_unstale":"2024-02-19T14:52:06.225524+0000","last_undegraded":"2024-02-19T14:52:06.225524+0000","last_fullsized":"2024-02-19T14:52:06.225524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:47.726577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246718+0000","last_change":"2024-02-19T14:51:59.267730+0000","last_active":"2024-02-19T14:52:06.246718+0000","last_peered":"2024-02-19T14:52:06.246718+0000","last_clean":"2024-02-19T14:52:06.246718+0000","last_became_active":"2024-02-19T14:51:59.267415+0000","last_became_peered":"2024-02-19T14:51:59.267415+0000","last_unstale":"2024-02-19T14:52:06.246718+0000","last_undegraded":"2024-02-19T14:52:06.246718+0000","last_fullsized":"2024-02-19T14:52:06.246718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226143+0000","last_change":"2024-02-19T14:52:04.433516+0000","last_active":"2024-02-19T14:52:06.226143+0000","last_peered":"2024-02-19T14:52:06.226143+0000","last_clean":"2024-02-19T14:52:06.226143+0000","last_became_active":"2024-02-19T14:52:04.432635+0000","last_became_peered":"2024-02-19T14:52:04.432635+0000","last_unstale":"2024-02-19T14:52:06.226143+0000","last_undegraded":"2024-02-19T14:52:06.226143+0000","last_fullsized":"2024-02-19T14:52:06.226143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334629+0000","last_change":"2024-02-19T14:52:00.266772+0000","last_active":"2024-02-19T14:52:07.334629+0000","last_peered":"2024-02-19T14:52:07.334629+0000","last_clean":"2024-02-19T14:52:07.334629+0000","last_became_active":"2024-02-19T14:52:00.265956+0000","last_became_peered":"2024-02-19T14:52:00.265956+0000","last_unstale":"2024-02-19T14:52:07.334629+0000","last_undegraded":"2024-02-19T14:52:07.334629+0000","last_fullsized":"2024-02-19T14:52:07.334629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282780+0000","last_change":"2024-02-19T14:51:59.252004+0000","last_active":"2024-02-19T14:52:05.282780+0000","last_peered":"2024-02-19T14:52:05.282780+0000","last_clean":"2024-02-19T14:52:05.282780+0000","last_became_active":"2024-02-19T14:51:59.251707+0000","last_became_peered":"2024-02-19T14:51:59.251707+0000","last_unstale":"2024-02-19T14:52:05.282780+0000","last_undegraded":"2024-02-19T14:52:05.282780+0000","last_fullsized":"2024-02-19T14:52:05.282780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246877+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:52:06.246877+0000","last_peered":"2024-02-19T14:52:06.246877+0000","last_clean":"2024-02-19T14:52:06.246877+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:52:06.246877+0000","last_undegraded":"2024-02-19T14:52:06.246877+0000","last_fullsized":"2024-02-19T14:52:06.246877+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530437+0000","last_change":"2024-02-19T14:52:04.436623+0000","last_active":"2024-02-19T14:52:05.530437+0000","last_peered":"2024-02-19T14:52:05.530437+0000","last_clean":"2024-02-19T14:52:05.530437+0000","last_became_active":"2024-02-19T14:52:04.436086+0000","last_became_peered":"2024-02-19T14:52:04.436086+0000","last_unstale":"2024-02-19T14:52:05.530437+0000","last_undegraded":"2024-02-19T14:52:05.530437+0000","last_fullsized":"2024-02-19T14:52:05.530437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247184+0000","last_change":"2024-02-19T14:52:00.267316+0000","last_active":"2024-02-19T14:52:06.247184+0000","last_peered":"2024-02-19T14:52:06.247184+0000","last_clean":"2024-02-19T14:52:06.247184+0000","last_became_active":"2024-02-19T14:52:00.266997+0000","last_became_peered":"2024-02-19T14:52:00.266997+0000","last_unstale":"2024-02-19T14:52:06.247184+0000","last_undegraded":"2024-02-19T14:52:06.247184+0000","last_fullsized":"2024-02-19T14:52:06.247184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281644+0000","last_change":"2024-02-19T14:51:59.253304+0000","last_active":"2024-02-19T14:52:05.281644+0000","last_peered":"2024-02-19T14:52:05.281644+0000","last_clean":"2024-02-19T14:52:05.281644+0000","last_became_active":"2024-02-19T14:51:59.253037+0000","last_became_peered":"2024-02-19T14:51:59.253037+0000","last_unstale":"2024-02-19T14:52:05.281644+0000","last_undegraded":"2024-02-19T14:52:05.281644+0000","last_fullsized":"2024-02-19T14:52:05.281644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530279+0000","last_change":"2024-02-19T14:52:04.433265+0000","last_active":"2024-02-19T14:52:05.530279+0000","last_peered":"2024-02-19T14:52:05.530279+0000","last_clean":"2024-02-19T14:52:05.530279+0000","last_became_active":"2024-02-19T14:52:04.432784+0000","last_became_peered":"2024-02-19T14:52:04.432784+0000","last_unstale":"2024-02-19T14:52:05.530279+0000","last_undegraded":"2024-02-19T14:52:05.530279+0000","last_fullsized":"2024-02-19T14:52:05.530279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283546+0000","last_change":"2024-02-19T14:52:00.266600+0000","last_active":"2024-02-19T14:52:05.283546+0000","last_peered":"2024-02-19T14:52:05.283546+0000","last_clean":"2024-02-19T14:52:05.283546+0000","last_became_active":"2024-02-19T14:52:00.265662+0000","last_became_peered":"2024-02-19T14:52:00.265662+0000","last_unstale":"2024-02-19T14:52:05.283546+0000","last_undegraded":"2024-02-19T14:52:05.283546+0000","last_fullsized":"2024-02-19T14:52:05.283546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229111+0000","last_change":"2024-02-19T14:51:59.258772+0000","last_active":"2024-02-19T14:52:07.229111+0000","last_peered":"2024-02-19T14:52:07.229111+0000","last_clean":"2024-02-19T14:52:07.229111+0000","last_became_active":"2024-02-19T14:51:59.258453+0000","last_became_peered":"2024-02-19T14:51:59.258453+0000","last_unstale":"2024-02-19T14:52:07.229111+0000","last_undegraded":"2024-02-19T14:52:07.229111+0000","last_fullsized":"2024-02-19T14:52:07.229111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225963+0000","last_change":"2024-02-19T14:52:04.281309+0000","last_active":"2024-02-19T14:52:06.225963+0000","last_peered":"2024-02-19T14:52:06.225963+0000","last_clean":"2024-02-19T14:52:06.225963+0000","last_became_active":"2024-02-19T14:52:04.280950+0000","last_became_peered":"2024-02-19T14:52:04.280950+0000","last_unstale":"2024-02-19T14:52:06.225963+0000","last_undegraded":"2024-02-19T14:52:06.225963+0000","last_fullsized":"2024-02-19T14:52:06.225963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247558+0000","last_change":"2024-02-19T14:52:00.260071+0000","last_active":"2024-02-19T14:52:06.247558+0000","last_peered":"2024-02-19T14:52:06.247558+0000","last_clean":"2024-02-19T14:52:06.247558+0000","last_became_active":"2024-02-19T14:52:00.259802+0000","last_became_peered":"2024-02-19T14:52:00.259802+0000","last_unstale":"2024-02-19T14:52:06.247558+0000","last_undegraded":"2024-02-19T14:52:06.247558+0000","last_fullsized":"2024-02-19T14:52:06.247558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:37.874309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281686+0000","last_change":"2024-02-19T14:51:59.252653+0000","last_active":"2024-02-19T14:52:05.281686+0000","last_peered":"2024-02-19T14:52:05.281686+0000","last_clean":"2024-02-19T14:52:05.281686+0000","last_became_active":"2024-02-19T14:51:59.252339+0000","last_became_peered":"2024-02-19T14:51:59.252339+0000","last_unstale":"2024-02-19T14:52:05.281686+0000","last_undegraded":"2024-02-19T14:52:05.281686+0000","last_fullsized":"2024-02-19T14:52:05.281686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282491+0000","last_change":"2024-02-19T14:52:04.435715+0000","last_active":"2024-02-19T14:52:05.282491+0000","last_peered":"2024-02-19T14:52:05.282491+0000","last_clean":"2024-02-19T14:52:05.282491+0000","last_became_active":"2024-02-19T14:52:04.435125+0000","last_became_peered":"2024-02-19T14:52:04.435125+0000","last_unstale":"2024-02-19T14:52:05.282491+0000","last_undegraded":"2024-02-19T14:52:05.282491+0000","last_fullsized":"2024-02-19T14:52:05.282491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247332+0000","last_change":"2024-02-19T14:52:00.262931+0000","last_active":"2024-02-19T14:52:06.247332+0000","last_peered":"2024-02-19T14:52:06.247332+0000","last_clean":"2024-02-19T14:52:06.247332+0000","last_became_active":"2024-02-19T14:52:00.262693+0000","last_became_peered":"2024-02-19T14:52:00.262693+0000","last_unstale":"2024-02-19T14:52:06.247332+0000","last_undegraded":"2024-02-19T14:52:06.247332+0000","last_fullsized":"2024-02-19T14:52:06.247332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283630+0000","last_change":"2024-02-19T14:51:59.260972+0000","last_active":"2024-02-19T14:52:05.283630+0000","last_peered":"2024-02-19T14:52:05.283630+0000","last_clean":"2024-02-19T14:52:05.283630+0000","last_became_active":"2024-02-19T14:51:59.260713+0000","last_became_peered":"2024-02-19T14:51:59.260713+0000","last_unstale":"2024-02-19T14:52:05.283630+0000","last_undegraded":"2024-02-19T14:52:05.283630+0000","last_fullsized":"2024-02-19T14:52:05.283630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283685+0000","last_change":"2024-02-19T14:52:04.277753+0000","last_active":"2024-02-19T14:52:05.283685+0000","last_peered":"2024-02-19T14:52:05.283685+0000","last_clean":"2024-02-19T14:52:05.283685+0000","last_became_active":"2024-02-19T14:52:04.277055+0000","last_became_peered":"2024-02-19T14:52:04.277055+0000","last_unstale":"2024-02-19T14:52:05.283685+0000","last_undegraded":"2024-02-19T14:52:05.283685+0000","last_fullsized":"2024-02-19T14:52:05.283685+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:26.703718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246886+0000","last_change":"2024-02-19T14:52:00.262083+0000","last_active":"2024-02-19T14:52:06.246886+0000","last_peered":"2024-02-19T14:52:06.246886+0000","last_clean":"2024-02-19T14:52:06.246886+0000","last_became_active":"2024-02-19T14:52:00.261797+0000","last_became_peered":"2024-02-19T14:52:00.261797+0000","last_unstale":"2024-02-19T14:52:06.246886+0000","last_undegraded":"2024-02-19T14:52:06.246886+0000","last_fullsized":"2024-02-19T14:52:06.246886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281279+0000","last_change":"2024-02-19T14:51:59.257068+0000","last_active":"2024-02-19T14:52:05.281279+0000","last_peered":"2024-02-19T14:52:05.281279+0000","last_clean":"2024-02-19T14:52:05.281279+0000","last_became_active":"2024-02-19T14:51:59.256775+0000","last_became_peered":"2024-02-19T14:51:59.256775+0000","last_unstale":"2024-02-19T14:52:05.281279+0000","last_undegraded":"2024-02-19T14:52:05.281279+0000","last_fullsized":"2024-02-19T14:52:05.281279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226044+0000","last_change":"2024-02-19T14:52:04.279890+0000","last_active":"2024-02-19T14:52:06.226044+0000","last_peered":"2024-02-19T14:52:06.226044+0000","last_clean":"2024-02-19T14:52:06.226044+0000","last_became_active":"2024-02-19T14:52:04.279485+0000","last_became_peered":"2024-02-19T14:52:04.279485+0000","last_unstale":"2024-02-19T14:52:06.226044+0000","last_undegraded":"2024-02-19T14:52:06.226044+0000","last_fullsized":"2024-02-19T14:52:06.226044+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:52:00.269756+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:52:00.269054+0000","last_became_peered":"2024-02-19T14:52:00.269054+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:51:59.251817+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:51:59.251496+0000","last_became_peered":"2024-02-19T14:51:59.251496+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335425+0000","last_change":"2024-02-19T14:52:04.436129+0000","last_active":"2024-02-19T14:52:07.335425+0000","last_peered":"2024-02-19T14:52:07.335425+0000","last_clean":"2024-02-19T14:52:07.335425+0000","last_became_active":"2024-02-19T14:52:04.435697+0000","last_became_peered":"2024-02-19T14:52:04.435697+0000","last_unstale":"2024-02-19T14:52:07.335425+0000","last_undegraded":"2024-02-19T14:52:07.335425+0000","last_fullsized":"2024-02-19T14:52:07.335425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:07.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247106+0000","last_change":"2024-02-19T14:52:00.263880+0000","last_active":"2024-02-19T14:52:06.247106+0000","last_peered":"2024-02-19T14:52:06.247106+0000","last_clean":"2024-02-19T14:52:06.247106+0000","last_became_active":"2024-02-19T14:52:00.263627+0000","last_became_peered":"2024-02-19T14:52:00.263627+0000","last_unstale":"2024-02-19T14:52:06.247106+0000","last_undegraded":"2024-02-19T14:52:06.247106+0000","last_fullsized":"2024-02-19T14:52:06.247106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283287+0000","last_change":"2024-02-19T14:51:59.252506+0000","last_active":"2024-02-19T14:52:05.283287+0000","last_peered":"2024-02-19T14:52:05.283287+0000","last_clean":"2024-02-19T14:52:05.283287+0000","last_became_active":"2024-02-19T14:51:59.251983+0000","last_became_peered":"2024-02-19T14:51:59.251983+0000","last_unstale":"2024-02-19T14:52:05.283287+0000","last_undegraded":"2024-02-19T14:52:05.283287+0000","last_fullsized":"2024-02-19T14:52:05.283287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335287+0000","last_change":"2024-02-19T14:52:04.433237+0000","last_active":"2024-02-19T14:52:07.335287+0000","last_peered":"2024-02-19T14:52:07.335287+0000","last_clean":"2024-02-19T14:52:07.335287+0000","last_became_active":"2024-02-19T14:52:04.432799+0000","last_became_peered":"2024-02-19T14:52:04.432799+0000","last_unstale":"2024-02-19T14:52:07.335287+0000","last_undegraded":"2024-02-19T14:52:07.335287+0000","last_fullsized":"2024-02-19T14:52:07.335287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:19.099988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225727+0000","last_change":"2024-02-19T14:52:00.267669+0000","last_active":"2024-02-19T14:52:06.225727+0000","last_peered":"2024-02-19T14:52:06.225727+0000","last_clean":"2024-02-19T14:52:06.225727+0000","last_became_active":"2024-02-19T14:52:00.267407+0000","last_became_peered":"2024-02-19T14:52:00.267407+0000","last_unstale":"2024-02-19T14:52:06.225727+0000","last_undegraded":"2024-02-19T14:52:06.225727+0000","last_fullsized":"2024-02-19T14:52:06.225727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226030+0000","last_change":"2024-02-19T14:51:59.261832+0000","last_active":"2024-02-19T14:52:06.226030+0000","last_peered":"2024-02-19T14:52:06.226030+0000","last_clean":"2024-02-19T14:52:06.226030+0000","last_became_active":"2024-02-19T14:51:59.261032+0000","last_became_peered":"2024-02-19T14:51:59.261032+0000","last_unstale":"2024-02-19T14:52:06.226030+0000","last_undegraded":"2024-02-19T14:52:06.226030+0000","last_fullsized":"2024-02-19T14:52:06.226030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284040+0000","last_change":"2024-02-19T14:52:04.433282+0000","last_active":"2024-02-19T14:52:05.284040+0000","last_peered":"2024-02-19T14:52:05.284040+0000","last_clean":"2024-02-19T14:52:05.284040+0000","last_became_active":"2024-02-19T14:52:04.432811+0000","last_became_peered":"2024-02-19T14:52:04.432811+0000","last_unstale":"2024-02-19T14:52:05.284040+0000","last_undegraded":"2024-02-19T14:52:05.284040+0000","last_fullsized":"2024-02-19T14:52:05.284040+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247064+0000","last_change":"2024-02-19T14:52:00.265115+0000","last_active":"2024-02-19T14:52:06.247064+0000","last_peered":"2024-02-19T14:52:06.247064+0000","last_clean":"2024-02-19T14:52:06.247064+0000","last_became_active":"2024-02-19T14:52:00.264838+0000","last_became_peered":"2024-02-19T14:52:00.264838+0000","last_unstale":"2024-02-19T14:52:06.247064+0000","last_undegraded":"2024-02-19T14:52:06.247064+0000","last_fullsized":"2024-02-19T14:52:06.247064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:09.545834+0000","last_change":"2024-02-19T14:51:59.260164+0000","last_active":"2024-02-19T14:52:09.545834+0000","last_peered":"2024-02-19T14:52:09.545834+0000","last_clean":"2024-02-19T14:52:09.545834+0000","last_became_active":"2024-02-19T14:51:59.259921+0000","last_became_peered":"2024-02-19T14:51:59.259921+0000","last_unstale":"2024-02-19T14:52:09.545834+0000","last_undegraded":"2024-02-19T14:52:09.545834+0000","last_fullsized":"2024-02-19T14:52:09.545834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228768+0000","last_change":"2024-02-19T14:52:04.282972+0000","last_active":"2024-02-19T14:52:07.228768+0000","last_peered":"2024-02-19T14:52:07.228768+0000","last_clean":"2024-02-19T14:52:07.228768+0000","last_became_active":"2024-02-19T14:52:04.282593+0000","last_became_peered":"2024-02-19T14:52:04.282593+0000","last_unstale":"2024-02-19T14:52:07.228768+0000","last_undegraded":"2024-02-19T14:52:07.228768+0000","last_fullsized":"2024-02-19T14:52:07.228768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283293+0000","last_change":"2024-02-19T14:52:00.263099+0000","last_active":"2024-02-19T14:52:05.283293+0000","last_peered":"2024-02-19T14:52:05.283293+0000","last_clean":"2024-02-19T14:52:05.283293+0000","last_became_active":"2024-02-19T14:52:00.260826+0000","last_became_peered":"2024-02-19T14:52:00.260826+0000","last_unstale":"2024-02-19T14:52:05.283293+0000","last_undegraded":"2024-02-19T14:52:05.283293+0000","last_fullsized":"2024-02-19T14:52:05.283293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:49.639204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.258165+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.257847+0000","last_became_peered":"2024-02-19T14:51:59.257847+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284313+0000","last_change":"2024-02-19T14:52:04.275097+0000","last_active":"2024-02-19T14:52:05.284313+0000","last_peered":"2024-02-19T14:52:05.284313+0000","last_clean":"2024-02-19T14:52:05.284313+0000","last_became_active":"2024-02-19T14:52:04.274720+0000","last_became_peered":"2024-02-19T14:52:04.274720+0000","last_unstale":"2024-02-19T14:52:05.284313+0000","last_undegraded":"2024-02-19T14:52:05.284313+0000","last_fullsized":"2024-02-19T14:52:05.284313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226316+0000","last_change":"2024-02-19T14:52:00.262791+0000","last_active":"2024-02-19T14:52:06.226316+0000","last_peered":"2024-02-19T14:52:06.226316+0000","last_clean":"2024-02-19T14:52:06.226316+0000","last_became_active":"2024-02-19T14:52:00.262201+0000","last_became_peered":"2024-02-19T14:52:00.262201+0000","last_unstale":"2024-02-19T14:52:06.226316+0000","last_undegraded":"2024-02-19T14:52:06.226316+0000","last_fullsized":"2024-02-19T14:52:06.226316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:23.961482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283591+0000","last_change":"2024-02-19T14:51:59.255762+0000","last_active":"2024-02-19T14:52:05.283591+0000","last_peered":"2024-02-19T14:52:05.283591+0000","last_clean":"2024-02-19T14:52:05.283591+0000","last_became_active":"2024-02-19T14:51:59.255418+0000","last_became_peered":"2024-02-19T14:51:59.255418+0000","last_unstale":"2024-02-19T14:52:05.283591+0000","last_undegraded":"2024-02-19T14:52:05.283591+0000","last_fullsized":"2024-02-19T14:52:05.283591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334347+0000","last_change":"2024-02-19T14:52:04.275183+0000","last_active":"2024-02-19T14:52:07.334347+0000","last_peered":"2024-02-19T14:52:07.334347+0000","last_clean":"2024-02-19T14:52:07.334347+0000","last_became_active":"2024-02-19T14:52:04.274796+0000","last_became_peered":"2024-02-19T14:52:04.274796+0000","last_unstale":"2024-02-19T14:52:07.334347+0000","last_undegraded":"2024-02-19T14:52:07.334347+0000","last_fullsized":"2024-02-19T14:52:07.334347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:17.099296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283483+0000","last_change":"2024-02-19T14:52:00.260440+0000","last_active":"2024-02-19T14:52:05.283483+0000","last_peered":"2024-02-19T14:52:05.283483+0000","last_clean":"2024-02-19T14:52:05.283483+0000","last_became_active":"2024-02-19T14:52:00.259406+0000","last_became_peered":"2024-02-19T14:52:00.259406+0000","last_unstale":"2024-02-19T14:52:05.283483+0000","last_undegraded":"2024-02-19T14:52:05.283483+0000","last_fullsized":"2024-02-19T14:52:05.283483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530584+0000","last_change":"2024-02-19T14:51:59.254054+0000","last_active":"2024-02-19T14:52:05.530584+0000","last_peered":"2024-02-19T14:52:05.530584+0000","last_clean":"2024-02-19T14:52:05.530584+0000","last_became_active":"2024-02-19T14:51:59.253751+0000","last_became_peered":"2024-02-19T14:51:59.253751+0000","last_unstale":"2024-02-19T14:52:05.530584+0000","last_undegraded":"2024-02-19T14:52:05.530584+0000","last_fullsized":"2024-02-19T14:52:05.530584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334305+0000","last_change":"2024-02-19T14:52:04.433602+0000","last_active":"2024-02-19T14:52:07.334305+0000","last_peered":"2024-02-19T14:52:07.334305+0000","last_clean":"2024-02-19T14:52:07.334305+0000","last_became_active":"2024-02-19T14:52:04.433250+0000","last_became_peered":"2024-02-19T14:52:04.433250+0000","last_unstale":"2024-02-19T14:52:07.334305+0000","last_undegraded":"2024-02-19T14:52:07.334305+0000","last_fullsized":"2024-02-19T14:52:07.334305+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226005+0000","last_change":"2024-02-19T14:52:04.435369+0000","last_active":"2024-02-19T14:52:06.226005+0000","last_peered":"2024-02-19T14:52:06.226005+0000","last_clean":"2024-02-19T14:52:06.226005+0000","last_became_active":"2024-02-19T14:52:04.435024+0000","last_became_peered":"2024-02-19T14:52:04.435024+0000","last_unstale":"2024-02-19T14:52:06.226005+0000","last_undegraded":"2024-02-19T14:52:06.226005+0000","last_fullsized":"2024-02-19T14:52:06.226005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:58.843829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228450+0000","last_change":"2024-02-19T14:52:00.266810+0000","last_active":"2024-02-19T14:52:07.228450+0000","last_peered":"2024-02-19T14:52:07.228450+0000","last_clean":"2024-02-19T14:52:07.228450+0000","last_became_active":"2024-02-19T14:52:00.266548+0000","last_became_peered":"2024-02-19T14:52:00.266548+0000","last_unstale":"2024-02-19T14:52:07.228450+0000","last_undegraded":"2024-02-19T14:52:07.228450+0000","last_fullsized":"2024-02-19T14:52:07.228450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282610+0000","last_change":"2024-02-19T14:51:59.248141+0000","last_active":"2024-02-19T14:52:05.282610+0000","last_peered":"2024-02-19T14:52:05.282610+0000","last_clean":"2024-02-19T14:52:05.282610+0000","last_became_active":"2024-02-19T14:51:59.247849+0000","last_became_peered":"2024-02-19T14:51:59.247849+0000","last_unstale":"2024-02-19T14:52:05.282610+0000","last_undegraded":"2024-02-19T14:52:05.282610+0000","last_fullsized":"2024-02-19T14:52:05.282610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530585+0000","last_change":"2024-02-19T14:52:04.282329+0000","last_active":"2024-02-19T14:52:05.530585+0000","last_peered":"2024-02-19T14:52:05.530585+0000","last_clean":"2024-02-19T14:52:05.530585+0000","last_became_active":"2024-02-19T14:52:04.281869+0000","last_became_peered":"2024-02-19T14:52:04.281869+0000","last_unstale":"2024-02-19T14:52:05.530585+0000","last_undegraded":"2024-02-19T14:52:05.530585+0000","last_fullsized":"2024-02-19T14:52:05.530585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334363+0000","last_change":"2024-02-19T14:52:00.266718+0000","last_active":"2024-02-19T14:52:07.334363+0000","last_peered":"2024-02-19T14:52:07.334363+0000","last_clean":"2024-02-19T14:52:07.334363+0000","last_became_active":"2024-02-19T14:52:00.266351+0000","last_became_peered":"2024-02-19T14:52:00.266351+0000","last_unstale":"2024-02-19T14:52:07.334363+0000","last_undegraded":"2024-02-19T14:52:07.334363+0000","last_fullsized":"2024-02-19T14:52:07.334363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333980+0000","last_change":"2024-02-19T14:51:59.259820+0000","last_active":"2024-02-19T14:52:07.333980+0000","last_peered":"2024-02-19T14:52:07.333980+0000","last_clean":"2024-02-19T14:52:07.333980+0000","last_became_active":"2024-02-19T14:51:59.259418+0000","last_became_peered":"2024-02-19T14:51:59.259418+0000","last_unstale":"2024-02-19T14:52:07.333980+0000","last_undegraded":"2024-02-19T14:52:07.333980+0000","last_fullsized":"2024-02-19T14:52:07.333980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530291+0000","last_change":"2024-02-19T14:51:59.252686+0000","last_active":"2024-02-19T14:52:05.530291+0000","last_peered":"2024-02-19T14:52:05.530291+0000","last_clean":"2024-02-19T14:52:05.530291+0000","last_became_active":"2024-02-19T14:51:59.252420+0000","last_became_peered":"2024-02-19T14:51:59.252420+0000","last_unstale":"2024-02-19T14:52:05.530291+0000","last_undegraded":"2024-02-19T14:52:05.530291+0000","last_fullsized":"2024-02-19T14:52:05.530291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226023+0000","last_change":"2024-02-19T14:52:00.264522+0000","last_active":"2024-02-19T14:52:06.226023+0000","last_peered":"2024-02-19T14:52:06.226023+0000","last_clean":"2024-02-19T14:52:06.226023+0000","last_became_active":"2024-02-19T14:52:00.264273+0000","last_became_peered":"2024-02-19T14:52:00.264273+0000","last_unstale":"2024-02-19T14:52:06.226023+0000","last_undegraded":"2024-02-19T14:52:06.226023+0000","last_fullsized":"2024-02-19T14:52:06.226023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281616+0000","last_change":"2024-02-19T14:52:04.275545+0000","last_active":"2024-02-19T14:52:05.281616+0000","last_peered":"2024-02-19T14:52:05.281616+0000","last_clean":"2024-02-19T14:52:05.281616+0000","last_became_active":"2024-02-19T14:52:04.275140+0000","last_became_peered":"2024-02-19T14:52:04.275140+0000","last_unstale":"2024-02-19T14:52:05.281616+0000","last_undegraded":"2024-02-19T14:52:05.281616+0000","last_fullsized":"2024-02-19T14:52:05.281616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:27.987737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333998+0000","last_change":"2024-02-19T14:51:59.262930+0000","last_active":"2024-02-19T14:52:07.333998+0000","last_peered":"2024-02-19T14:52:07.333998+0000","last_clean":"2024-02-19T14:52:07.333998+0000","last_became_active":"2024-02-19T14:51:59.262635+0000","last_became_peered":"2024-02-19T14:51:59.262635+0000","last_unstale":"2024-02-19T14:52:07.333998+0000","last_undegraded":"2024-02-19T14:52:07.333998+0000","last_fullsized":"2024-02-19T14:52:07.333998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:21.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284375+0000","last_change":"2024-02-19T14:52:00.260213+0000","last_active":"2024-02-19T14:52:05.284375+0000","last_peered":"2024-02-19T14:52:05.284375+0000","last_clean":"2024-02-19T14:52:05.284375+0000","last_became_active":"2024-02-19T14:52:00.257148+0000","last_became_peered":"2024-02-19T14:52:00.257148+0000","last_unstale":"2024-02-19T14:52:05.284375+0000","last_undegraded":"2024-02-19T14:52:05.284375+0000","last_fullsized":"2024-02-19T14:52:05.284375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:56.918797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335231+0000","last_change":"2024-02-19T14:52:04.276033+0000","last_active":"2024-02-19T14:52:07.335231+0000","last_peered":"2024-02-19T14:52:07.335231+0000","last_clean":"2024-02-19T14:52:07.335231+0000","last_became_active":"2024-02-19T14:52:04.275671+0000","last_became_peered":"2024-02-19T14:52:04.275671+0000","last_unstale":"2024-02-19T14:52:07.335231+0000","last_undegraded":"2024-02-19T14:52:07.335231+0000","last_fullsized":"2024-02-19T14:52:07.335231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:42.880381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.529955+0000","last_change":"2024-02-19T14:52:00.270697+0000","last_active":"2024-02-19T14:52:05.529955+0000","last_peered":"2024-02-19T14:52:05.529955+0000","last_clean":"2024-02-19T14:52:05.529955+0000","last_became_active":"2024-02-19T14:52:00.269449+0000","last_became_peered":"2024-02-19T14:52:00.269449+0000","last_unstale":"2024-02-19T14:52:05.529955+0000","last_undegraded":"2024-02-19T14:52:05.529955+0000","last_fullsized":"2024-02-19T14:52:05.529955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281315+0000","last_change":"2024-02-19T14:52:00.263056+0000","last_active":"2024-02-19T14:52:05.281315+0000","last_peered":"2024-02-19T14:52:05.281315+0000","last_clean":"2024-02-19T14:52:05.281315+0000","last_became_active":"2024-02-19T14:52:00.262801+0000","last_became_peered":"2024-02-19T14:52:00.262801+0000","last_unstale":"2024-02-19T14:52:05.281315+0000","last_undegraded":"2024-02-19T14:52:05.281315+0000","last_fullsized":"2024-02-19T14:52:05.281315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283161+0000","last_change":"2024-02-19T14:52:00.270389+0000","last_active":"2024-02-19T14:52:05.283161+0000","last_peered":"2024-02-19T14:52:05.283161+0000","last_clean":"2024-02-19T14:52:05.283161+0000","last_became_active":"2024-02-19T14:52:00.267246+0000","last_became_peered":"2024-02-19T14:52:00.267246+0000","last_unstale":"2024-02-19T14:52:05.283161+0000","last_undegraded":"2024-02-19T14:52:05.283161+0000","last_fullsized":"2024-02-19T14:52:05.283161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225914+0000","last_change":"2024-02-19T14:52:00.265624+0000","last_active":"2024-02-19T14:52:06.225914+0000","last_peered":"2024-02-19T14:52:06.225914+0000","last_clean":"2024-02-19T14:52:06.225914+0000","last_became_active":"2024-02-19T14:52:00.265377+0000","last_became_peered":"2024-02-19T14:52:00.265377+0000","last_unstale":"2024-02-19T14:52:06.225914+0000","last_undegraded":"2024-02-19T14:52:06.225914+0000","last_fullsized":"2024-02-19T14:52:06.225914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335142+0000","last_change":"2024-02-19T14:52:00.263043+0000","last_active":"2024-02-19T14:52:07.335142+0000","last_peered":"2024-02-19T14:52:07.335142+0000","last_clean":"2024-02-19T14:52:07.335142+0000","last_became_active":"2024-02-19T14:52:00.262754+0000","last_became_peered":"2024-02-19T14:52:00.262754+0000","last_unstale":"2024-02-19T14:52:07.335142+0000","last_undegraded":"2024-02-19T14:52:07.335142+0000","last_fullsized":"2024-02-19T14:52:07.335142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228927+0000","last_change":"2024-02-19T14:52:00.266890+0000","last_active":"2024-02-19T14:52:07.228927+0000","last_peered":"2024-02-19T14:52:07.228927+0000","last_clean":"2024-02-19T14:52:07.228927+0000","last_became_active":"2024-02-19T14:52:00.266574+0000","last_became_peered":"2024-02-19T14:52:00.266574+0000","last_unstale":"2024-02-19T14:52:07.228927+0000","last_undegraded":"2024-02-19T14:52:07.228927+0000","last_fullsized":"2024-02-19T14:52:07.228927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530706+0000","last_change":"2024-02-19T14:52:00.261225+0000","last_active":"2024-02-19T14:52:05.530706+0000","last_peered":"2024-02-19T14:52:05.530706+0000","last_clean":"2024-02-19T14:52:05.530706+0000","last_became_active":"2024-02-19T14:52:00.259666+0000","last_became_peered":"2024-02-19T14:52:00.259666+0000","last_unstale":"2024-02-19T14:52:05.530706+0000","last_undegraded":"2024-02-19T14:52:05.530706+0000","last_fullsized":"2024-02-19T14:52:05.530706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335354+0000","last_change":"2024-02-19T14:52:00.264377+0000","last_active":"2024-02-19T14:52:07.335354+0000","last_peered":"2024-02-19T14:52:07.335354+0000","last_clean":"2024-02-19T14:52:07.335354+0000","last_became_active":"2024-02-19T14:52:00.264082+0000","last_became_peered":"2024-02-19T14:52:00.264082+0000","last_unstale":"2024-02-19T14:52:07.335354+0000","last_undegraded":"2024-02-19T14:52:07.335354+0000","last_fullsized":"2024-02-19T14:52:07.335354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282561+0000","last_change":"2024-02-19T14:52:00.265423+0000","last_active":"2024-02-19T14:52:05.282561+0000","last_peered":"2024-02-19T14:52:05.282561+0000","last_clean":"2024-02-19T14:52:05.282561+0000","last_became_active":"2024-02-19T14:52:00.265132+0000","last_became_peered":"2024-02-19T14:52:00.265132+0000","last_unstale":"2024-02-19T14:52:05.282561+0000","last_undegraded":"2024-02-19T14:52:05.282561+0000","last_fullsized":"2024-02-19T14:52:05.282561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334543+0000","last_change":"2024-02-19T14:52:00.259058+0000","last_active":"2024-02-19T14:52:07.334543+0000","last_peered":"2024-02-19T14:52:07.334543+0000","last_clean":"2024-02-19T14:52:07.334543+0000","last_became_active":"2024-02-19T14:52:00.258228+0000","last_became_peered":"2024-02-19T14:52:00.258228+0000","last_unstale":"2024-02-19T14:52:07.334543+0000","last_undegraded":"2024-02-19T14:52:07.334543+0000","last_fullsized":"2024-02-19T14:52:07.334543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283058+0000","last_change":"2024-02-19T14:52:00.258261+0000","last_active":"2024-02-19T14:52:05.283058+0000","last_peered":"2024-02-19T14:52:05.283058+0000","last_clean":"2024-02-19T14:52:05.283058+0000","last_became_active":"2024-02-19T14:52:00.257970+0000","last_became_peered":"2024-02-19T14:52:00.257970+0000","last_unstale":"2024-02-19T14:52:05.283058+0000","last_undegraded":"2024-02-19T14:52:05.283058+0000","last_fullsized":"2024-02-19T14:52:05.283058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225756+0000","last_change":"2024-02-19T14:52:00.268547+0000","last_active":"2024-02-19T14:52:06.225756+0000","last_peered":"2024-02-19T14:52:06.225756+0000","last_clean":"2024-02-19T14:52:06.225756+0000","last_became_active":"2024-02-19T14:52:00.268241+0000","last_became_peered":"2024-02-19T14:52:00.268241+0000","last_unstale":"2024-02-19T14:52:06.225756+0000","last_undegraded":"2024-02-19T14:52:06.225756+0000","last_fullsized":"2024-02-19T14:52:06.225756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530412+0000","last_change":"2024-02-19T14:52:00.262779+0000","last_active":"2024-02-19T14:52:05.530412+0000","last_peered":"2024-02-19T14:52:05.530412+0000","last_clean":"2024-02-19T14:52:05.530412+0000","last_became_active":"2024-02-19T14:52:00.262476+0000","last_became_peered":"2024-02-19T14:52:00.262476+0000","last_unstale":"2024-02-19T14:52:05.530412+0000","last_undegraded":"2024-02-19T14:52:05.530412+0000","last_fullsized":"2024-02-19T14:52:05.530412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247311+0000","last_change":"2024-02-19T14:52:00.272996+0000","last_active":"2024-02-19T14:52:06.247311+0000","last_peered":"2024-02-19T14:52:06.247311+0000","last_clean":"2024-02-19T14:52:06.247311+0000","last_became_active":"2024-02-19T14:52:00.271013+0000","last_became_peered":"2024-02-19T14:52:00.271013+0000","last_unstale":"2024-02-19T14:52:06.247311+0000","last_undegraded":"2024-02-19T14:52:06.247311+0000","last_fullsized":"2024-02-19T14:52:06.247311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281593+0000","last_change":"2024-02-19T14:52:00.263763+0000","last_active":"2024-02-19T14:52:05.281593+0000","last_peered":"2024-02-19T14:52:05.281593+0000","last_clean":"2024-02-19T14:52:05.281593+0000","last_became_active":"2024-02-19T14:52:00.263488+0000","last_became_peered":"2024-02-19T14:52:00.263488+0000","last_unstale":"2024-02-19T14:52:05.281593+0000","last_undegraded":"2024-02-19T14:52:05.281593+0000","last_fullsized":"2024-02-19T14:52:05.281593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:46.599373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284503+0000","last_change":"2024-02-19T14:52:00.258879+0000","last_active":"2024-02-19T14:52:05.284503+0000","last_peered":"2024-02-19T14:52:05.284503+0000","last_clean":"2024-02-19T14:52:05.284503+0000","last_became_active":"2024-02-19T14:52:00.257612+0000","last_became_peered":"2024-02-19T14:52:00.257612+0000","last_unstale":"2024-02-19T14:52:05.284503+0000","last_undegraded":"2024-02-19T14:52:05.284503+0000","last_fullsized":"2024-02-19T14:52:05.284503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282447+0000","last_change":"2024-02-19T14:52:00.268671+0000","last_active":"2024-02-19T14:52:05.282447+0000","last_peered":"2024-02-19T14:52:05.282447+0000","last_clean":"2024-02-19T14:52:05.282447+0000","last_became_active":"2024-02-19T14:52:00.268351+0000","last_became_peered":"2024-02-19T14:52:00.268351+0000","last_unstale":"2024-02-19T14:52:05.282447+0000","last_undegraded":"2024-02-19T14:52:05.282447+0000","last_fullsized":"2024-02-19T14:52:05.282447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228525+0000","last_change":"2024-02-19T14:52:00.249695+0000","last_active":"2024-02-19T14:52:07.228525+0000","last_peered":"2024-02-19T14:52:07.228525+0000","last_clean":"2024-02-19T14:52:07.228525+0000","last_became_active":"2024-02-19T14:52:00.249406+0000","last_became_peered":"2024-02-19T14:52:00.249406+0000","last_unstale":"2024-02-19T14:52:07.228525+0000","last_undegraded":"2024-02-19T14:52:07.228525+0000","last_fullsized":"2024-02-19T14:52:07.228525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225669+0000","last_change":"2024-02-19T14:52:00.259566+0000","last_active":"2024-02-19T14:52:06.225669+0000","last_peered":"2024-02-19T14:52:06.225669+0000","last_clean":"2024-02-19T14:52:06.225669+0000","last_became_active":"2024-02-19T14:52:00.259290+0000","last_became_peered":"2024-02-19T14:52:00.259290+0000","last_unstale":"2024-02-19T14:52:06.225669+0000","last_undegraded":"2024-02-19T14:52:06.225669+0000","last_fullsized":"2024-02-19T14:52:06.225669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334965+0000","last_change":"2024-02-19T14:52:00.267359+0000","last_active":"2024-02-19T14:52:07.334965+0000","last_peered":"2024-02-19T14:52:07.334965+0000","last_clean":"2024-02-19T14:52:07.334965+0000","last_became_active":"2024-02-19T14:52:00.266119+0000","last_became_peered":"2024-02-19T14:52:00.266119+0000","last_unstale":"2024-02-19T14:52:07.334965+0000","last_undegraded":"2024-02-19T14:52:07.334965+0000","last_fullsized":"2024-02-19T14:52:07.334965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283181+0000","last_change":"2024-02-19T14:52:00.262531+0000","last_active":"2024-02-19T14:52:05.283181+0000","last_peered":"2024-02-19T14:52:05.283181+0000","last_clean":"2024-02-19T14:52:05.283181+0000","last_became_active":"2024-02-19T14:52:00.262240+0000","last_became_peered":"2024-02-19T14:52:00.262240+0000","last_unstale":"2024-02-19T14:52:05.283181+0000","last_undegraded":"2024-02-19T14:52:05.283181+0000","last_fullsized":"2024-02-19T14:52:05.283181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:41.341079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530384+0000","last_change":"2024-02-19T14:52:00.266149+0000","last_active":"2024-02-19T14:52:05.530384+0000","last_peered":"2024-02-19T14:52:05.530384+0000","last_clean":"2024-02-19T14:52:05.530384+0000","last_became_active":"2024-02-19T14:52:00.265873+0000","last_became_peered":"2024-02-19T14:52:00.265873+0000","last_unstale":"2024-02-19T14:52:05.530384+0000","last_undegraded":"2024-02-19T14:52:05.530384+0000","last_fullsized":"2024-02-19T14:52:05.530384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:52:00.266144+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:52:00.265852+0000","last_became_peered":"2024-02-19T14:52:00.265852+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247502+0000","last_change":"2024-02-19T14:52:00.265908+0000","last_active":"2024-02-19T14:52:06.247502+0000","last_peered":"2024-02-19T14:52:06.247502+0000","last_clean":"2024-02-19T14:52:06.247502+0000","last_became_active":"2024-02-19T14:52:00.265630+0000","last_became_peered":"2024-02-19T14:52:00.265630+0000","last_unstale":"2024-02-19T14:52:06.247502+0000","last_undegraded":"2024-02-19T14:52:06.247502+0000","last_fullsized":"2024-02-19T14:52:06.247502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:06.056284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334181+0000","last_change":"2024-02-19T14:52:00.267195+0000","last_active":"2024-02-19T14:52:07.334181+0000","last_peered":"2024-02-19T14:52:07.334181+0000","last_clean":"2024-02-19T14:52:07.334181+0000","last_became_active":"2024-02-19T14:52:00.266185+0000","last_became_peered":"2024-02-19T14:52:00.266185+0000","last_unstale":"2024-02-19T14:52:07.334181+0000","last_undegraded":"2024-02-19T14:52:07.334181+0000","last_fullsized":"2024-02-19T14:52:07.334181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246858+0000","last_change":"2024-02-19T14:52:00.266943+0000","last_active":"2024-02-19T14:52:06.246858+0000","last_peered":"2024-02-19T14:52:06.246858+0000","last_clean":"2024-02-19T14:52:06.246858+0000","last_became_active":"2024-02-19T14:52:00.266665+0000","last_became_peered":"2024-02-19T14:52:00.266665+0000","last_unstale":"2024-02-19T14:52:06.246858+0000","last_undegraded":"2024-02-19T14:52:06.246858+0000","last_fullsized":"2024-02-19T14:52:06.246858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282952+0000","last_change":"2024-02-19T14:52:00.261505+0000","last_active":"2024-02-19T14:52:05.282952+0000","last_peered":"2024-02-19T14:52:05.282952+0000","last_clean":"2024-02-19T14:52:05.282952+0000","last_became_active":"2024-02-19T14:52:00.261212+0000","last_became_peered":"2024-02-19T14:52:00.261212+0000","last_unstale":"2024-02-19T14:52:05.282952+0000","last_undegraded":"2024-02-19T14:52:05.282952+0000","last_fullsized":"2024-02-19T14:52:05.282952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246984+0000","last_change":"2024-02-19T14:52:00.261481+0000","last_active":"2024-02-19T14:52:06.246984+0000","last_peered":"2024-02-19T14:52:06.246984+0000","last_clean":"2024-02-19T14:52:06.246984+0000","last_became_active":"2024-02-19T14:52:00.261205+0000","last_became_peered":"2024-02-19T14:52:00.261205+0000","last_unstale":"2024-02-19T14:52:06.246984+0000","last_undegraded":"2024-02-19T14:52:06.246984+0000","last_fullsized":"2024-02-19T14:52:06.246984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283153+0000","last_change":"2024-02-19T14:52:00.266814+0000","last_active":"2024-02-19T14:52:05.283153+0000","last_peered":"2024-02-19T14:52:05.283153+0000","last_clean":"2024-02-19T14:52:05.283153+0000","last_became_active":"2024-02-19T14:52:00.266019+0000","last_became_peered":"2024-02-19T14:52:00.266019+0000","last_unstale":"2024-02-19T14:52:05.283153+0000","last_undegraded":"2024-02-19T14:52:05.283153+0000","last_fullsized":"2024-02-19T14:52:05.283153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225905+0000","last_change":"2024-02-19T14:52:00.260863+0000","last_active":"2024-02-19T14:52:06.225905+0000","last_peered":"2024-02-19T14:52:06.225905+0000","last_clean":"2024-02-19T14:52:06.225905+0000","last_became_active":"2024-02-19T14:52:00.260543+0000","last_became_peered":"2024-02-19T14:52:00.260543+0000","last_unstale":"2024-02-19T14:52:06.225905+0000","last_undegraded":"2024-02-19T14:52:06.225905+0000","last_fullsized":"2024-02-19T14:52:06.225905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228999+0000","last_change":"2024-02-19T14:52:00.260306+0000","last_active":"2024-02-19T14:52:07.228999+0000","last_peered":"2024-02-19T14:52:07.228999+0000","last_clean":"2024-02-19T14:52:07.228999+0000","last_became_active":"2024-02-19T14:52:00.260026+0000","last_became_peered":"2024-02-19T14:52:00.260026+0000","last_unstale":"2024-02-19T14:52:07.228999+0000","last_undegraded":"2024-02-19T14:52:07.228999+0000","last_fullsized":"2024-02-19T14:52:07.228999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247442+0000","last_change":"2024-02-19T14:52:00.263320+0000","last_active":"2024-02-19T14:52:06.247442+0000","last_peered":"2024-02-19T14:52:06.247442+0000","last_clean":"2024-02-19T14:52:06.247442+0000","last_became_active":"2024-02-19T14:52:00.263044+0000","last_became_peered":"2024-02-19T14:52:00.263044+0000","last_unstale":"2024-02-19T14:52:06.247442+0000","last_undegraded":"2024-02-19T14:52:06.247442+0000","last_fullsized":"2024-02-19T14:52:06.247442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:08.747083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:52:21.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:52:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.002+0000 7fbf2350d640 1 Processor -- start 2024-02-19T14:52:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.002+0000 7fbf2350d640 1 -- start start 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf2350d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 0x7fbf1c0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf2350d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf1c06d040 0x7fbf1c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf2350d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1c06db90 con 0x7fbf1c0c7b70 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf2350d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c06d990 con 0x7fbf1c06d040 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf2350d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c154ec0 con 0x7fbf1c155bb0 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf212a9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbf1c0c7b70 0x7fbf1c1584b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57818/0 (socket says 172.21.15.78:57818) 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf212a9640 1 -- 172.21.15.78:0/1247284010 learned_addr learned my addr 172.21.15.78:0/1247284010 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333905289 0 0) 0x7fbf1c06db90 con 0x7fbf1c0c7b70 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf21aaa640 1 --2- 172.21.15.78:0/1247284010 >> v2:172.21.15.150:3300/0 conn(0x7fbf1c06d040 0x7fbf1c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.003+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf08003520 con 0x7fbf1c0c7b70 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf20aa8640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 0x7fbf1c0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1933458694 0 0) 0x7fbf08003520 con 0x7fbf1c0c7b70 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 msgr2=0x7fbf1c0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 0x7fbf1c0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 >> v2:172.21.15.150:3300/0 conn(0x7fbf1c06d040 msgr2=0x7fbf1c06d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 --2- 172.21.15.78:0/1247284010 >> v2:172.21.15.150:3300/0 conn(0x7fbf1c06d040 0x7fbf1c06d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c1550d0 con 0x7fbf1c0c7b70 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fbf18001670 con 0x7fbf1c0c7b70 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 >> v1:172.21.15.78:6789/0 conn(0x7fbf1c0c7b70 legacy=0x7fbf1c1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf08003970 0x7fbf08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 0x7fbf080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08004d00 0x7fbf08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c06d990 con 0x7fbf08003970 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c154ec0 con 0x7fbf08004d00 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0800b980 con 0x7fbf1c155bb0 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf08003970 0x7fbf08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf20aa8640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 0x7fbf080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf21aaa640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08004d00 0x7fbf08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf212a9640 1 -- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 msgr2=0x7fbf080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c155bb0 0x7fbf080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf212a9640 1 -- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08004d00 msgr2=0x7fbf08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08004d00 0x7fbf08005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.004+0000 7fbf212a9640 1 -- 172.21.15.78:0/1247284010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08003520 con 0x7fbf08003970 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.005+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf08003970 0x7fbf08003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800c210 tx=0x7fbf080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65df6490c30ab402 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.005+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1802cae0 con 0x7fbf08003970 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.005+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1802cc40 con 0x7fbf08003970 2024-02-19T14:52:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.005+0000 7fbf13fff640 1 -- 172.21.15.78:0/1247284010 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf180058d0 con 0x7fbf08003970 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf2350d640 1 -- 172.21.15.78:0/1247284010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf08003970 msgr2=0x7fbf08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1247284010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf08003970 0x7fbf08003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800c210 tx=0x7fbf080057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf21aaa640 1 -- 172.21.15.78:0/1247284010 reap_dead start 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf2350d640 1 -- 172.21.15.78:0/1247284010 shutdown_connections 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf2350d640 1 -- 172.21.15.78:0/1247284010 >> 172.21.15.78:0/1247284010 conn(0x7fbf1c0c5050 msgr2=0x7fbf1c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf2350d640 1 -- 172.21.15.78:0/1247284010 shutdown_connections 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.006+0000 7fbf2350d640 1 -- 172.21.15.78:0/1247284010 wait complete. 2024-02-19T14:52:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.007+0000 7fbf2350d640 1 Processor -- start 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.007+0000 7fbf2350d640 1 -- start start 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08003970 0x7fbf1c1eb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 0x7fbf1c150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c14f560 0x7fbf1c147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c148590 con 0x7fbf1c152000 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1eaeb0 con 0x7fbf08003970 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c148770 con 0x7fbf1c14f560 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 0x7fbf1c150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 0x7fbf1c150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52688/0 (socket says 172.21.15.78:52688) 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 -- 172.21.15.78:0/1478993086 learned_addr learned my addr 172.21.15.78:0/1478993086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08003970 0x7fbf1c1eb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf21aaa640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c14f560 0x7fbf1c147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 -- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c14f560 msgr2=0x7fbf1c147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c14f560 0x7fbf1c147860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 -- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08003970 msgr2=0x7fbf1c1eb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08003970 0x7fbf1c1eb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 -- 172.21.15.78:0/1478993086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c1eab40 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf20aa8640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 0x7fbf1c150e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0119a0 tx=0x7fbf0c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c00faf0 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c00fc50 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c014440 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.009+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c151370 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.010+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c152480 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.011+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c0ae4a0 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.011+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbf0c0145a0 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.011+0000 7fbf11ffb640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf0006b810 0x7fbf0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.012+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbf0c08ad60 con 0x7fbf1c152000 2024-02-19T14:52:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.012+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf0006b810 0x7fbf0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.012+0000 7fbf212a9640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf0006b810 0x7fbf0006dcd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbf18034000 tx=0x7fbf18007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.016+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbf0c0554d0 con 0x7fbf1c152000 2024-02-19T14:52:22.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.191+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf1c155690 con 0x7fbf0006b810 2024-02-19T14:52:22.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.199+0000 7fbf11ffb640 1 -- 172.21.15.78:0/1478993086 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345262 (secure 0 0 0) 0x7fbf1802c8e0 con 0x7fbf0006b810 2024-02-19T14:52:22.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf0006b810 msgr2=0x7fbf0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf0006b810 0x7fbf0006dcd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbf18034000 tx=0x7fbf18007000 comp rx=0 tx=0).stop 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 msgr2=0x7fbf1c150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 0x7fbf1c150e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0119a0 tx=0x7fbf0c011e60 comp rx=0 tx=0).stop 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 shutdown_connections 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf0006b810 0x7fbf0006dcd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf1c152000 0x7fbf1c150e30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf1c14f560 0x7fbf1c147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 --2- 172.21.15.78:0/1478993086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf08003970 0x7fbf1c1eb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 >> 172.21.15.78:0/1478993086 conn(0x7fbf1c0c5050 msgr2=0x7fbf1c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:22.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 shutdown_connections 2024-02-19T14:52:22.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.203+0000 7fbf2350d640 1 -- 172.21.15.78:0/1478993086 wait complete. 2024-02-19T14:52:22.216 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-02-19T14:52:22.064360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219380,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755340,"statfs":{"total":773094113280,"available":772869468160,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002649"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226366+0000","last_change":"2024-02-19T14:51:59.266929+0000","last_active":"2024-02-19T14:52:06.226366+0000","last_peered":"2024-02-19T14:52:06.226366+0000","last_clean":"2024-02-19T14:52:06.226366+0000","last_became_active":"2024-02-19T14:51:59.266644+0000","last_became_peered":"2024-02-19T14:51:59.266644+0000","last_unstale":"2024-02-19T14:52:06.226366+0000","last_undegraded":"2024-02-19T14:52:06.226366+0000","last_fullsized":"2024-02-19T14:52:06.226366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335294+0000","last_change":"2024-02-19T14:52:00.264687+0000","last_active":"2024-02-19T14:52:07.335294+0000","last_peered":"2024-02-19T14:52:07.335294+0000","last_clean":"2024-02-19T14:52:07.335294+0000","last_became_active":"2024-02-19T14:52:00.264424+0000","last_became_peered":"2024-02-19T14:52:00.264424+0000","last_unstale":"2024-02-19T14:52:07.335294+0000","last_undegraded":"2024-02-19T14:52:07.335294+0000","last_fullsized":"2024-02-19T14:52:07.335294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334297+0000","last_change":"2024-02-19T14:52:04.435336+0000","last_active":"2024-02-19T14:52:07.334297+0000","last_peered":"2024-02-19T14:52:07.334297+0000","last_clean":"2024-02-19T14:52:07.334297+0000","last_became_active":"2024-02-19T14:52:04.434977+0000","last_became_peered":"2024-02-19T14:52:04.434977+0000","last_unstale":"2024-02-19T14:52:07.334297+0000","last_undegraded":"2024-02-19T14:52:07.334297+0000","last_fullsized":"2024-02-19T14:52:07.334297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334116+0000","last_change":"2024-02-19T14:51:59.262430+0000","last_active":"2024-02-19T14:52:07.334116+0000","last_peered":"2024-02-19T14:52:07.334116+0000","last_clean":"2024-02-19T14:52:07.334116+0000","last_became_active":"2024-02-19T14:51:59.262168+0000","last_became_peered":"2024-02-19T14:51:59.262168+0000","last_unstale":"2024-02-19T14:52:07.334116+0000","last_undegraded":"2024-02-19T14:52:07.334116+0000","last_fullsized":"2024-02-19T14:52:07.334116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226267+0000","last_change":"2024-02-19T14:52:00.263014+0000","last_active":"2024-02-19T14:52:06.226267+0000","last_peered":"2024-02-19T14:52:06.226267+0000","last_clean":"2024-02-19T14:52:06.226267+0000","last_became_active":"2024-02-19T14:52:00.262531+0000","last_became_peered":"2024-02-19T14:52:00.262531+0000","last_unstale":"2024-02-19T14:52:06.226267+0000","last_undegraded":"2024-02-19T14:52:06.226267+0000","last_fullsized":"2024-02-19T14:52:06.226267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282898+0000","last_change":"2024-02-19T14:52:04.277572+0000","last_active":"2024-02-19T14:52:05.282898+0000","last_peered":"2024-02-19T14:52:05.282898+0000","last_clean":"2024-02-19T14:52:05.282898+0000","last_became_active":"2024-02-19T14:52:04.276970+0000","last_became_peered":"2024-02-19T14:52:04.276970+0000","last_unstale":"2024-02-19T14:52:05.282898+0000","last_undegraded":"2024-02-19T14:52:05.282898+0000","last_fullsized":"2024-02-19T14:52:05.282898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:14.095185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247290+0000","last_change":"2024-02-19T14:51:59.265003+0000","last_active":"2024-02-19T14:52:06.247290+0000","last_peered":"2024-02-19T14:52:06.247290+0000","last_clean":"2024-02-19T14:52:06.247290+0000","last_became_active":"2024-02-19T14:51:59.264719+0000","last_became_peered":"2024-02-19T14:51:59.264719+0000","last_unstale":"2024-02-19T14:52:06.247290+0000","last_undegraded":"2024-02-19T14:52:06.247290+0000","last_fullsized":"2024-02-19T14:52:06.247290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283484+0000","last_change":"2024-02-19T14:52:00.252606+0000","last_active":"2024-02-19T14:52:05.283484+0000","last_peered":"2024-02-19T14:52:05.283484+0000","last_clean":"2024-02-19T14:52:05.283484+0000","last_became_active":"2024-02-19T14:52:00.249782+0000","last_became_peered":"2024-02-19T14:52:00.249782+0000","last_unstale":"2024-02-19T14:52:05.283484+0000","last_undegraded":"2024-02-19T14:52:05.283484+0000","last_fullsized":"2024-02-19T14:52:05.283484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283979+0000","last_change":"2024-02-19T14:52:04.278014+0000","last_active":"2024-02-19T14:52:05.283979+0000","last_peered":"2024-02-19T14:52:05.283979+0000","last_clean":"2024-02-19T14:52:05.283979+0000","last_became_active":"2024-02-19T14:52:04.277126+0000","last_became_peered":"2024-02-19T14:52:04.277126+0000","last_unstale":"2024-02-19T14:52:05.283979+0000","last_undegraded":"2024-02-19T14:52:05.283979+0000","last_fullsized":"2024-02-19T14:52:05.283979+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:27.674760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247080+0000","last_change":"2024-02-19T14:51:59.263462+0000","last_active":"2024-02-19T14:52:06.247080+0000","last_peered":"2024-02-19T14:52:06.247080+0000","last_clean":"2024-02-19T14:52:06.247080+0000","last_became_active":"2024-02-19T14:51:59.262932+0000","last_became_peered":"2024-02-19T14:51:59.262932+0000","last_unstale":"2024-02-19T14:52:06.247080+0000","last_undegraded":"2024-02-19T14:52:06.247080+0000","last_fullsized":"2024-02-19T14:52:06.247080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284101+0000","last_change":"2024-02-19T14:52:00.252460+0000","last_active":"2024-02-19T14:52:05.284101+0000","last_peered":"2024-02-19T14:52:05.284101+0000","last_clean":"2024-02-19T14:52:05.284101+0000","last_became_active":"2024-02-19T14:52:00.251603+0000","last_became_peered":"2024-02-19T14:52:00.251603+0000","last_unstale":"2024-02-19T14:52:05.284101+0000","last_undegraded":"2024-02-19T14:52:05.284101+0000","last_fullsized":"2024-02-19T14:52:05.284101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283155+0000","last_change":"2024-02-19T14:52:04.278038+0000","last_active":"2024-02-19T14:52:05.283155+0000","last_peered":"2024-02-19T14:52:05.283155+0000","last_clean":"2024-02-19T14:52:05.283155+0000","last_became_active":"2024-02-19T14:52:04.277620+0000","last_became_peered":"2024-02-19T14:52:04.277620+0000","last_unstale":"2024-02-19T14:52:05.283155+0000","last_undegraded":"2024-02-19T14:52:05.283155+0000","last_fullsized":"2024-02-19T14:52:05.283155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:12.987808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530027+0000","last_change":"2024-02-19T14:52:00.264133+0000","last_active":"2024-02-19T14:52:05.530027+0000","last_peered":"2024-02-19T14:52:05.530027+0000","last_clean":"2024-02-19T14:52:05.530027+0000","last_became_active":"2024-02-19T14:52:00.263475+0000","last_became_peered":"2024-02-19T14:52:00.263475+0000","last_unstale":"2024-02-19T14:52:05.530027+0000","last_undegraded":"2024-02-19T14:52:05.530027+0000","last_fullsized":"2024-02-19T14:52:05.530027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247110+0000","last_change":"2024-02-19T14:51:59.263095+0000","last_active":"2024-02-19T14:52:06.247110+0000","last_peered":"2024-02-19T14:52:06.247110+0000","last_clean":"2024-02-19T14:52:06.247110+0000","last_became_active":"2024-02-19T14:51:59.262392+0000","last_became_peered":"2024-02-19T14:51:59.262392+0000","last_unstale":"2024-02-19T14:52:06.247110+0000","last_undegraded":"2024-02-19T14:52:06.247110+0000","last_fullsized":"2024-02-19T14:52:06.247110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:04.278534+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:04.277850+0000","last_became_peered":"2024-02-19T14:52:04.277850+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334968+0000","last_change":"2024-02-19T14:52:00.268782+0000","last_active":"2024-02-19T14:52:07.334968+0000","last_peered":"2024-02-19T14:52:07.334968+0000","last_clean":"2024-02-19T14:52:07.334968+0000","last_became_active":"2024-02-19T14:52:00.266478+0000","last_became_peered":"2024-02-19T14:52:00.266478+0000","last_unstale":"2024-02-19T14:52:07.334968+0000","last_undegraded":"2024-02-19T14:52:07.334968+0000","last_fullsized":"2024-02-19T14:52:07.334968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228818+0000","last_change":"2024-02-19T14:51:59.260219+0000","last_active":"2024-02-19T14:52:07.228818+0000","last_peered":"2024-02-19T14:52:07.228818+0000","last_clean":"2024-02-19T14:52:07.228818+0000","last_became_active":"2024-02-19T14:51:59.259917+0000","last_became_peered":"2024-02-19T14:51:59.259917+0000","last_unstale":"2024-02-19T14:52:07.228818+0000","last_undegraded":"2024-02-19T14:52:07.228818+0000","last_fullsized":"2024-02-19T14:52:07.228818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282889+0000","last_change":"2024-02-19T14:52:04.277635+0000","last_active":"2024-02-19T14:52:05.282889+0000","last_peered":"2024-02-19T14:52:05.282889+0000","last_clean":"2024-02-19T14:52:05.282889+0000","last_became_active":"2024-02-19T14:52:04.277231+0000","last_became_peered":"2024-02-19T14:52:04.277231+0000","last_unstale":"2024-02-19T14:52:05.282889+0000","last_undegraded":"2024-02-19T14:52:05.282889+0000","last_fullsized":"2024-02-19T14:52:05.282889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:46.471360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334393+0000","last_change":"2024-02-19T14:52:00.263688+0000","last_active":"2024-02-19T14:52:07.334393+0000","last_peered":"2024-02-19T14:52:07.334393+0000","last_clean":"2024-02-19T14:52:07.334393+0000","last_became_active":"2024-02-19T14:52:00.263381+0000","last_became_peered":"2024-02-19T14:52:00.263381+0000","last_unstale":"2024-02-19T14:52:07.334393+0000","last_undegraded":"2024-02-19T14:52:07.334393+0000","last_fullsized":"2024-02-19T14:52:07.334393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228547+0000","last_change":"2024-02-19T14:51:59.262590+0000","last_active":"2024-02-19T14:52:07.228547+0000","last_peered":"2024-02-19T14:52:07.228547+0000","last_clean":"2024-02-19T14:52:07.228547+0000","last_became_active":"2024-02-19T14:51:59.262310+0000","last_became_peered":"2024-02-19T14:51:59.262310+0000","last_unstale":"2024-02-19T14:52:07.228547+0000","last_undegraded":"2024-02-19T14:52:07.228547+0000","last_fullsized":"2024-02-19T14:52:07.228547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229123+0000","last_change":"2024-02-19T14:52:04.434727+0000","last_active":"2024-02-19T14:52:07.229123+0000","last_peered":"2024-02-19T14:52:07.229123+0000","last_clean":"2024-02-19T14:52:07.229123+0000","last_became_active":"2024-02-19T14:52:04.432708+0000","last_became_peered":"2024-02-19T14:52:04.432708+0000","last_unstale":"2024-02-19T14:52:07.229123+0000","last_undegraded":"2024-02-19T14:52:07.229123+0000","last_fullsized":"2024-02-19T14:52:07.229123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:19.410143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284506+0000","last_change":"2024-02-19T14:52:00.260620+0000","last_active":"2024-02-19T14:52:05.284506+0000","last_peered":"2024-02-19T14:52:05.284506+0000","last_clean":"2024-02-19T14:52:05.284506+0000","last_became_active":"2024-02-19T14:52:00.260357+0000","last_became_peered":"2024-02-19T14:52:00.260357+0000","last_unstale":"2024-02-19T14:52:05.284506+0000","last_undegraded":"2024-02-19T14:52:05.284506+0000","last_fullsized":"2024-02-19T14:52:05.284506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:09.799959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246702+0000","last_change":"2024-02-19T14:51:59.267802+0000","last_active":"2024-02-19T14:52:06.246702+0000","last_peered":"2024-02-19T14:52:06.246702+0000","last_clean":"2024-02-19T14:52:06.246702+0000","last_became_active":"2024-02-19T14:51:59.267518+0000","last_became_peered":"2024-02-19T14:51:59.267518+0000","last_unstale":"2024-02-19T14:52:06.246702+0000","last_undegraded":"2024-02-19T14:52:06.246702+0000","last_fullsized":"2024-02-19T14:52:06.246702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:45.594311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283176+0000","last_change":"2024-02-19T14:52:04.276717+0000","last_active":"2024-02-19T14:52:05.283176+0000","last_peered":"2024-02-19T14:52:05.283176+0000","last_clean":"2024-02-19T14:52:05.283176+0000","last_became_active":"2024-02-19T14:52:04.276273+0000","last_became_peered":"2024-02-19T14:52:04.276273+0000","last_unstale":"2024-02-19T14:52:05.283176+0000","last_undegraded":"2024-02-19T14:52:05.283176+0000","last_fullsized":"2024-02-19T14:52:05.283176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:30.898405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226156+0000","last_change":"2024-02-19T14:52:00.259379+0000","last_active":"2024-02-19T14:52:06.226156+0000","last_peered":"2024-02-19T14:52:06.226156+0000","last_clean":"2024-02-19T14:52:06.226156+0000","last_became_active":"2024-02-19T14:52:00.259061+0000","last_became_peered":"2024-02-19T14:52:00.259061+0000","last_unstale":"2024-02-19T14:52:06.226156+0000","last_undegraded":"2024-02-19T14:52:06.226156+0000","last_fullsized":"2024-02-19T14:52:06.226156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:51:59.262618+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:51:59.262333+0000","last_became_peered":"2024-02-19T14:51:59.262333+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284070+0000","last_change":"2024-02-19T14:52:04.436152+0000","last_active":"2024-02-19T14:52:05.284070+0000","last_peered":"2024-02-19T14:52:05.284070+0000","last_clean":"2024-02-19T14:52:05.284070+0000","last_became_active":"2024-02-19T14:52:04.435251+0000","last_became_peered":"2024-02-19T14:52:04.435251+0000","last_unstale":"2024-02-19T14:52:05.284070+0000","last_undegraded":"2024-02-19T14:52:05.284070+0000","last_fullsized":"2024-02-19T14:52:05.284070+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:33.762974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335436+0000","last_change":"2024-02-19T14:52:00.272464+0000","last_active":"2024-02-19T14:52:07.335436+0000","last_peered":"2024-02-19T14:52:07.335436+0000","last_clean":"2024-02-19T14:52:07.335436+0000","last_became_active":"2024-02-19T14:52:00.272184+0000","last_became_peered":"2024-02-19T14:52:00.272184+0000","last_unstale":"2024-02-19T14:52:07.335436+0000","last_undegraded":"2024-02-19T14:52:07.335436+0000","last_fullsized":"2024-02-19T14:52:07.335436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.256170+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.255855+0000","last_became_peered":"2024-02-19T14:51:59.255855+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334561+0000","last_change":"2024-02-19T14:52:04.433402+0000","last_active":"2024-02-19T14:52:07.334561+0000","last_peered":"2024-02-19T14:52:07.334561+0000","last_clean":"2024-02-19T14:52:07.334561+0000","last_became_active":"2024-02-19T14:52:04.432815+0000","last_became_peered":"2024-02-19T14:52:04.432815+0000","last_unstale":"2024-02-19T14:52:07.334561+0000","last_undegraded":"2024-02-19T14:52:07.334561+0000","last_fullsized":"2024-02-19T14:52:07.334561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:20.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228931+0000","last_change":"2024-02-19T14:52:00.261954+0000","last_active":"2024-02-19T14:52:07.228931+0000","last_peered":"2024-02-19T14:52:07.228931+0000","last_clean":"2024-02-19T14:52:07.228931+0000","last_became_active":"2024-02-19T14:52:00.261678+0000","last_became_peered":"2024-02-19T14:52:00.261678+0000","last_unstale":"2024-02-19T14:52:07.228931+0000","last_undegraded":"2024-02-19T14:52:07.228931+0000","last_fullsized":"2024-02-19T14:52:07.228931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247420+0000","last_change":"2024-02-19T14:51:59.267996+0000","last_active":"2024-02-19T14:52:06.247420+0000","last_peered":"2024-02-19T14:52:06.247420+0000","last_clean":"2024-02-19T14:52:06.247420+0000","last_became_active":"2024-02-19T14:51:59.267426+0000","last_became_peered":"2024-02-19T14:51:59.267426+0000","last_unstale":"2024-02-19T14:52:06.247420+0000","last_undegraded":"2024-02-19T14:52:06.247420+0000","last_fullsized":"2024-02-19T14:52:06.247420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225861+0000","last_change":"2024-02-19T14:52:04.433162+0000","last_active":"2024-02-19T14:52:06.225861+0000","last_peered":"2024-02-19T14:52:06.225861+0000","last_clean":"2024-02-19T14:52:06.225861+0000","last_became_active":"2024-02-19T14:52:04.432722+0000","last_became_peered":"2024-02-19T14:52:04.432722+0000","last_unstale":"2024-02-19T14:52:06.225861+0000","last_undegraded":"2024-02-19T14:52:06.225861+0000","last_fullsized":"2024-02-19T14:52:06.225861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284214+0000","last_change":"2024-02-19T14:52:00.272249+0000","last_active":"2024-02-19T14:52:05.284214+0000","last_peered":"2024-02-19T14:52:05.284214+0000","last_clean":"2024-02-19T14:52:05.284214+0000","last_became_active":"2024-02-19T14:52:00.266375+0000","last_became_peered":"2024-02-19T14:52:00.266375+0000","last_unstale":"2024-02-19T14:52:05.284214+0000","last_undegraded":"2024-02-19T14:52:05.284214+0000","last_fullsized":"2024-02-19T14:52:05.284214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229033+0000","last_change":"2024-02-19T14:51:59.260559+0000","last_active":"2024-02-19T14:52:07.229033+0000","last_peered":"2024-02-19T14:52:07.229033+0000","last_clean":"2024-02-19T14:52:07.229033+0000","last_became_active":"2024-02-19T14:51:59.260290+0000","last_became_peered":"2024-02-19T14:51:59.260290+0000","last_unstale":"2024-02-19T14:52:07.229033+0000","last_undegraded":"2024-02-19T14:52:07.229033+0000","last_fullsized":"2024-02-19T14:52:07.229033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:48.634143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228634+0000","last_change":"2024-02-19T14:52:04.283819+0000","last_active":"2024-02-19T14:52:07.228634+0000","last_peered":"2024-02-19T14:52:07.228634+0000","last_clean":"2024-02-19T14:52:07.228634+0000","last_became_active":"2024-02-19T14:52:04.283469+0000","last_became_peered":"2024-02-19T14:52:04.283469+0000","last_unstale":"2024-02-19T14:52:07.228634+0000","last_undegraded":"2024-02-19T14:52:07.228634+0000","last_fullsized":"2024-02-19T14:52:07.228634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284410+0000","last_change":"2024-02-19T14:52:00.266384+0000","last_active":"2024-02-19T14:52:05.284410+0000","last_peered":"2024-02-19T14:52:05.284410+0000","last_clean":"2024-02-19T14:52:05.284410+0000","last_became_active":"2024-02-19T14:52:00.265954+0000","last_became_peered":"2024-02-19T14:52:00.265954+0000","last_unstale":"2024-02-19T14:52:05.284410+0000","last_undegraded":"2024-02-19T14:52:05.284410+0000","last_fullsized":"2024-02-19T14:52:05.284410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283363+0000","last_change":"2024-02-19T14:51:59.259377+0000","last_active":"2024-02-19T14:52:05.283363+0000","last_peered":"2024-02-19T14:52:05.283363+0000","last_clean":"2024-02-19T14:52:05.283363+0000","last_became_active":"2024-02-19T14:51:59.259004+0000","last_became_peered":"2024-02-19T14:51:59.259004+0000","last_unstale":"2024-02-19T14:52:05.283363+0000","last_undegraded":"2024-02-19T14:52:05.283363+0000","last_fullsized":"2024-02-19T14:52:05.283363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229000+0000","last_change":"2024-02-19T14:52:04.282027+0000","last_active":"2024-02-19T14:52:07.229000+0000","last_peered":"2024-02-19T14:52:07.229000+0000","last_clean":"2024-02-19T14:52:07.229000+0000","last_became_active":"2024-02-19T14:52:04.281679+0000","last_became_peered":"2024-02-19T14:52:04.281679+0000","last_unstale":"2024-02-19T14:52:07.229000+0000","last_undegraded":"2024-02-19T14:52:07.229000+0000","last_fullsized":"2024-02-19T14:52:07.229000+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:41.900564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228565+0000","last_change":"2024-02-19T14:52:00.261302+0000","last_active":"2024-02-19T14:52:07.228565+0000","last_peered":"2024-02-19T14:52:07.228565+0000","last_clean":"2024-02-19T14:52:07.228565+0000","last_became_active":"2024-02-19T14:52:00.261027+0000","last_became_peered":"2024-02-19T14:52:00.261027+0000","last_unstale":"2024-02-19T14:52:07.228565+0000","last_undegraded":"2024-02-19T14:52:07.228565+0000","last_fullsized":"2024-02-19T14:52:07.228565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281412+0000","last_change":"2024-02-19T14:51:59.256034+0000","last_active":"2024-02-19T14:52:05.281412+0000","last_peered":"2024-02-19T14:52:05.281412+0000","last_clean":"2024-02-19T14:52:05.281412+0000","last_became_active":"2024-02-19T14:51:59.255701+0000","last_became_peered":"2024-02-19T14:51:59.255701+0000","last_unstale":"2024-02-19T14:52:05.281412+0000","last_undegraded":"2024-02-19T14:52:05.281412+0000","last_fullsized":"2024-02-19T14:52:05.281412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229180+0000","last_change":"2024-02-19T14:52:04.283973+0000","last_active":"2024-02-19T14:52:07.229180+0000","last_peered":"2024-02-19T14:52:07.229180+0000","last_clean":"2024-02-19T14:52:07.229180+0000","last_became_active":"2024-02-19T14:52:04.283562+0000","last_became_peered":"2024-02-19T14:52:04.283562+0000","last_unstale":"2024-02-19T14:52:07.229180+0000","last_undegraded":"2024-02-19T14:52:07.229180+0000","last_fullsized":"2024-02-19T14:52:07.229180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225524+0000","last_change":"2024-02-19T14:52:00.260330+0000","last_active":"2024-02-19T14:52:06.225524+0000","last_peered":"2024-02-19T14:52:06.225524+0000","last_clean":"2024-02-19T14:52:06.225524+0000","last_became_active":"2024-02-19T14:52:00.260039+0000","last_became_peered":"2024-02-19T14:52:00.260039+0000","last_unstale":"2024-02-19T14:52:06.225524+0000","last_undegraded":"2024-02-19T14:52:06.225524+0000","last_fullsized":"2024-02-19T14:52:06.225524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:10:47.726577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246718+0000","last_change":"2024-02-19T14:51:59.267730+0000","last_active":"2024-02-19T14:52:06.246718+0000","last_peered":"2024-02-19T14:52:06.246718+0000","last_clean":"2024-02-19T14:52:06.246718+0000","last_became_active":"2024-02-19T14:51:59.267415+0000","last_became_peered":"2024-02-19T14:51:59.267415+0000","last_unstale":"2024-02-19T14:52:06.246718+0000","last_undegraded":"2024-02-19T14:52:06.246718+0000","last_fullsized":"2024-02-19T14:52:06.246718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226143+0000","last_change":"2024-02-19T14:52:04.433516+0000","last_active":"2024-02-19T14:52:06.226143+0000","last_peered":"2024-02-19T14:52:06.226143+0000","last_clean":"2024-02-19T14:52:06.226143+0000","last_became_active":"2024-02-19T14:52:04.432635+0000","last_became_peered":"2024-02-19T14:52:04.432635+0000","last_unstale":"2024-02-19T14:52:06.226143+0000","last_undegraded":"2024-02-19T14:52:06.226143+0000","last_fullsized":"2024-02-19T14:52:06.226143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334629+0000","last_change":"2024-02-19T14:52:00.266772+0000","last_active":"2024-02-19T14:52:07.334629+0000","last_peered":"2024-02-19T14:52:07.334629+0000","last_clean":"2024-02-19T14:52:07.334629+0000","last_became_active":"2024-02-19T14:52:00.265956+0000","last_became_peered":"2024-02-19T14:52:00.265956+0000","last_unstale":"2024-02-19T14:52:07.334629+0000","last_undegraded":"2024-02-19T14:52:07.334629+0000","last_fullsized":"2024-02-19T14:52:07.334629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282780+0000","last_change":"2024-02-19T14:51:59.252004+0000","last_active":"2024-02-19T14:52:05.282780+0000","last_peered":"2024-02-19T14:52:05.282780+0000","last_clean":"2024-02-19T14:52:05.282780+0000","last_became_active":"2024-02-19T14:51:59.251707+0000","last_became_peered":"2024-02-19T14:51:59.251707+0000","last_unstale":"2024-02-19T14:52:05.282780+0000","last_undegraded":"2024-02-19T14:52:05.282780+0000","last_fullsized":"2024-02-19T14:52:05.282780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246877+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:52:06.246877+0000","last_peered":"2024-02-19T14:52:06.246877+0000","last_clean":"2024-02-19T14:52:06.246877+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:52:06.246877+0000","last_undegraded":"2024-02-19T14:52:06.246877+0000","last_fullsized":"2024-02-19T14:52:06.246877+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530437+0000","last_change":"2024-02-19T14:52:04.436623+0000","last_active":"2024-02-19T14:52:05.530437+0000","last_peered":"2024-02-19T14:52:05.530437+0000","last_clean":"2024-02-19T14:52:05.530437+0000","last_became_active":"2024-02-19T14:52:04.436086+0000","last_became_peered":"2024-02-19T14:52:04.436086+0000","last_unstale":"2024-02-19T14:52:05.530437+0000","last_undegraded":"2024-02-19T14:52:05.530437+0000","last_fullsized":"2024-02-19T14:52:05.530437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247184+0000","last_change":"2024-02-19T14:52:00.267316+0000","last_active":"2024-02-19T14:52:06.247184+0000","last_peered":"2024-02-19T14:52:06.247184+0000","last_clean":"2024-02-19T14:52:06.247184+0000","last_became_active":"2024-02-19T14:52:00.266997+0000","last_became_peered":"2024-02-19T14:52:00.266997+0000","last_unstale":"2024-02-19T14:52:06.247184+0000","last_undegraded":"2024-02-19T14:52:06.247184+0000","last_fullsized":"2024-02-19T14:52:06.247184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:10.863254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281644+0000","last_change":"2024-02-19T14:51:59.253304+0000","last_active":"2024-02-19T14:52:05.281644+0000","last_peered":"2024-02-19T14:52:05.281644+0000","last_clean":"2024-02-19T14:52:05.281644+0000","last_became_active":"2024-02-19T14:51:59.253037+0000","last_became_peered":"2024-02-19T14:51:59.253037+0000","last_unstale":"2024-02-19T14:52:05.281644+0000","last_undegraded":"2024-02-19T14:52:05.281644+0000","last_fullsized":"2024-02-19T14:52:05.281644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530279+0000","last_change":"2024-02-19T14:52:04.433265+0000","last_active":"2024-02-19T14:52:05.530279+0000","last_peered":"2024-02-19T14:52:05.530279+0000","last_clean":"2024-02-19T14:52:05.530279+0000","last_became_active":"2024-02-19T14:52:04.432784+0000","last_became_peered":"2024-02-19T14:52:04.432784+0000","last_unstale":"2024-02-19T14:52:05.530279+0000","last_undegraded":"2024-02-19T14:52:05.530279+0000","last_fullsized":"2024-02-19T14:52:05.530279+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:59.188668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283546+0000","last_change":"2024-02-19T14:52:00.266600+0000","last_active":"2024-02-19T14:52:05.283546+0000","last_peered":"2024-02-19T14:52:05.283546+0000","last_clean":"2024-02-19T14:52:05.283546+0000","last_became_active":"2024-02-19T14:52:00.265662+0000","last_became_peered":"2024-02-19T14:52:00.265662+0000","last_unstale":"2024-02-19T14:52:05.283546+0000","last_undegraded":"2024-02-19T14:52:05.283546+0000","last_fullsized":"2024-02-19T14:52:05.283546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229111+0000","last_change":"2024-02-19T14:51:59.258772+0000","last_active":"2024-02-19T14:52:07.229111+0000","last_peered":"2024-02-19T14:52:07.229111+0000","last_clean":"2024-02-19T14:52:07.229111+0000","last_became_active":"2024-02-19T14:51:59.258453+0000","last_became_peered":"2024-02-19T14:51:59.258453+0000","last_unstale":"2024-02-19T14:52:07.229111+0000","last_undegraded":"2024-02-19T14:52:07.229111+0000","last_fullsized":"2024-02-19T14:52:07.229111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:00.225293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225963+0000","last_change":"2024-02-19T14:52:04.281309+0000","last_active":"2024-02-19T14:52:06.225963+0000","last_peered":"2024-02-19T14:52:06.225963+0000","last_clean":"2024-02-19T14:52:06.225963+0000","last_became_active":"2024-02-19T14:52:04.280950+0000","last_became_peered":"2024-02-19T14:52:04.280950+0000","last_unstale":"2024-02-19T14:52:06.225963+0000","last_undegraded":"2024-02-19T14:52:06.225963+0000","last_fullsized":"2024-02-19T14:52:06.225963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247558+0000","last_change":"2024-02-19T14:52:00.260071+0000","last_active":"2024-02-19T14:52:06.247558+0000","last_peered":"2024-02-19T14:52:06.247558+0000","last_clean":"2024-02-19T14:52:06.247558+0000","last_became_active":"2024-02-19T14:52:00.259802+0000","last_became_peered":"2024-02-19T14:52:00.259802+0000","last_unstale":"2024-02-19T14:52:06.247558+0000","last_undegraded":"2024-02-19T14:52:06.247558+0000","last_fullsized":"2024-02-19T14:52:06.247558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:10:37.874309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281686+0000","last_change":"2024-02-19T14:51:59.252653+0000","last_active":"2024-02-19T14:52:05.281686+0000","last_peered":"2024-02-19T14:52:05.281686+0000","last_clean":"2024-02-19T14:52:05.281686+0000","last_became_active":"2024-02-19T14:51:59.252339+0000","last_became_peered":"2024-02-19T14:51:59.252339+0000","last_unstale":"2024-02-19T14:52:05.281686+0000","last_undegraded":"2024-02-19T14:52:05.281686+0000","last_fullsized":"2024-02-19T14:52:05.281686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282491+0000","last_change":"2024-02-19T14:52:04.435715+0000","last_active":"2024-02-19T14:52:05.282491+0000","last_peered":"2024-02-19T14:52:05.282491+0000","last_clean":"2024-02-19T14:52:05.282491+0000","last_became_active":"2024-02-19T14:52:04.435125+0000","last_became_peered":"2024-02-19T14:52:04.435125+0000","last_unstale":"2024-02-19T14:52:05.282491+0000","last_undegraded":"2024-02-19T14:52:05.282491+0000","last_fullsized":"2024-02-19T14:52:05.282491+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:46:38.342852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247332+0000","last_change":"2024-02-19T14:52:00.262931+0000","last_active":"2024-02-19T14:52:06.247332+0000","last_peered":"2024-02-19T14:52:06.247332+0000","last_clean":"2024-02-19T14:52:06.247332+0000","last_became_active":"2024-02-19T14:52:00.262693+0000","last_became_peered":"2024-02-19T14:52:00.262693+0000","last_unstale":"2024-02-19T14:52:06.247332+0000","last_undegraded":"2024-02-19T14:52:06.247332+0000","last_fullsized":"2024-02-19T14:52:06.247332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283630+0000","last_change":"2024-02-19T14:51:59.260972+0000","last_active":"2024-02-19T14:52:05.283630+0000","last_peered":"2024-02-19T14:52:05.283630+0000","last_clean":"2024-02-19T14:52:05.283630+0000","last_became_active":"2024-02-19T14:51:59.260713+0000","last_became_peered":"2024-02-19T14:51:59.260713+0000","last_unstale":"2024-02-19T14:52:05.283630+0000","last_undegraded":"2024-02-19T14:52:05.283630+0000","last_fullsized":"2024-02-19T14:52:05.283630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:57.879783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283685+0000","last_change":"2024-02-19T14:52:04.277753+0000","last_active":"2024-02-19T14:52:05.283685+0000","last_peered":"2024-02-19T14:52:05.283685+0000","last_clean":"2024-02-19T14:52:05.283685+0000","last_became_active":"2024-02-19T14:52:04.277055+0000","last_became_peered":"2024-02-19T14:52:04.277055+0000","last_unstale":"2024-02-19T14:52:05.283685+0000","last_undegraded":"2024-02-19T14:52:05.283685+0000","last_fullsized":"2024-02-19T14:52:05.283685+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:26.703718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246886+0000","last_change":"2024-02-19T14:52:00.262083+0000","last_active":"2024-02-19T14:52:06.246886+0000","last_peered":"2024-02-19T14:52:06.246886+0000","last_clean":"2024-02-19T14:52:06.246886+0000","last_became_active":"2024-02-19T14:52:00.261797+0000","last_became_peered":"2024-02-19T14:52:00.261797+0000","last_unstale":"2024-02-19T14:52:06.246886+0000","last_undegraded":"2024-02-19T14:52:06.246886+0000","last_fullsized":"2024-02-19T14:52:06.246886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281279+0000","last_change":"2024-02-19T14:51:59.257068+0000","last_active":"2024-02-19T14:52:05.281279+0000","last_peered":"2024-02-19T14:52:05.281279+0000","last_clean":"2024-02-19T14:52:05.281279+0000","last_became_active":"2024-02-19T14:51:59.256775+0000","last_became_peered":"2024-02-19T14:51:59.256775+0000","last_unstale":"2024-02-19T14:52:05.281279+0000","last_undegraded":"2024-02-19T14:52:05.281279+0000","last_fullsized":"2024-02-19T14:52:05.281279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226044+0000","last_change":"2024-02-19T14:52:04.279890+0000","last_active":"2024-02-19T14:52:06.226044+0000","last_peered":"2024-02-19T14:52:06.226044+0000","last_clean":"2024-02-19T14:52:06.226044+0000","last_became_active":"2024-02-19T14:52:04.279485+0000","last_became_peered":"2024-02-19T14:52:04.279485+0000","last_unstale":"2024-02-19T14:52:06.226044+0000","last_undegraded":"2024-02-19T14:52:06.226044+0000","last_fullsized":"2024-02-19T14:52:06.226044+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:10.082539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:52:00.269756+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:52:00.269054+0000","last_became_peered":"2024-02-19T14:52:00.269054+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530102+0000","last_change":"2024-02-19T14:51:59.251817+0000","last_active":"2024-02-19T14:52:05.530102+0000","last_peered":"2024-02-19T14:52:05.530102+0000","last_clean":"2024-02-19T14:52:05.530102+0000","last_became_active":"2024-02-19T14:51:59.251496+0000","last_became_peered":"2024-02-19T14:51:59.251496+0000","last_unstale":"2024-02-19T14:52:05.530102+0000","last_undegraded":"2024-02-19T14:52:05.530102+0000","last_fullsized":"2024-02-19T14:52:05.530102+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335425+0000","last_change":"2024-02-19T14:52:04.436129+0000","last_active":"2024-02-19T14:52:07.335425+0000","last_peered":"2024-02-19T14:52:07.335425+0000","last_clean":"2024-02-19T14:52:07.335425+0000","last_became_active":"2024-02-19T14:52:04.435697+0000","last_became_peered":"2024-02-19T14:52:04.435697+0000","last_unstale":"2024-02-19T14:52:07.335425+0000","last_undegraded":"2024-02-19T14:52:07.335425+0000","last_fullsized":"2024-02-19T14:52:07.335425+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:07.823976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247106+0000","last_change":"2024-02-19T14:52:00.263880+0000","last_active":"2024-02-19T14:52:06.247106+0000","last_peered":"2024-02-19T14:52:06.247106+0000","last_clean":"2024-02-19T14:52:06.247106+0000","last_became_active":"2024-02-19T14:52:00.263627+0000","last_became_peered":"2024-02-19T14:52:00.263627+0000","last_unstale":"2024-02-19T14:52:06.247106+0000","last_undegraded":"2024-02-19T14:52:06.247106+0000","last_fullsized":"2024-02-19T14:52:06.247106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:55.162413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283287+0000","last_change":"2024-02-19T14:51:59.252506+0000","last_active":"2024-02-19T14:52:05.283287+0000","last_peered":"2024-02-19T14:52:05.283287+0000","last_clean":"2024-02-19T14:52:05.283287+0000","last_became_active":"2024-02-19T14:51:59.251983+0000","last_became_peered":"2024-02-19T14:51:59.251983+0000","last_unstale":"2024-02-19T14:52:05.283287+0000","last_undegraded":"2024-02-19T14:52:05.283287+0000","last_fullsized":"2024-02-19T14:52:05.283287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335287+0000","last_change":"2024-02-19T14:52:04.433237+0000","last_active":"2024-02-19T14:52:07.335287+0000","last_peered":"2024-02-19T14:52:07.335287+0000","last_clean":"2024-02-19T14:52:07.335287+0000","last_became_active":"2024-02-19T14:52:04.432799+0000","last_became_peered":"2024-02-19T14:52:04.432799+0000","last_unstale":"2024-02-19T14:52:07.335287+0000","last_undegraded":"2024-02-19T14:52:07.335287+0000","last_fullsized":"2024-02-19T14:52:07.335287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:19.099988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225727+0000","last_change":"2024-02-19T14:52:00.267669+0000","last_active":"2024-02-19T14:52:06.225727+0000","last_peered":"2024-02-19T14:52:06.225727+0000","last_clean":"2024-02-19T14:52:06.225727+0000","last_became_active":"2024-02-19T14:52:00.267407+0000","last_became_peered":"2024-02-19T14:52:00.267407+0000","last_unstale":"2024-02-19T14:52:06.225727+0000","last_undegraded":"2024-02-19T14:52:06.225727+0000","last_fullsized":"2024-02-19T14:52:06.225727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226030+0000","last_change":"2024-02-19T14:51:59.261832+0000","last_active":"2024-02-19T14:52:06.226030+0000","last_peered":"2024-02-19T14:52:06.226030+0000","last_clean":"2024-02-19T14:52:06.226030+0000","last_became_active":"2024-02-19T14:51:59.261032+0000","last_became_peered":"2024-02-19T14:51:59.261032+0000","last_unstale":"2024-02-19T14:52:06.226030+0000","last_undegraded":"2024-02-19T14:52:06.226030+0000","last_fullsized":"2024-02-19T14:52:06.226030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:53.417631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284040+0000","last_change":"2024-02-19T14:52:04.433282+0000","last_active":"2024-02-19T14:52:05.284040+0000","last_peered":"2024-02-19T14:52:05.284040+0000","last_clean":"2024-02-19T14:52:05.284040+0000","last_became_active":"2024-02-19T14:52:04.432811+0000","last_became_peered":"2024-02-19T14:52:04.432811+0000","last_unstale":"2024-02-19T14:52:05.284040+0000","last_undegraded":"2024-02-19T14:52:05.284040+0000","last_fullsized":"2024-02-19T14:52:05.284040+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247064+0000","last_change":"2024-02-19T14:52:00.265115+0000","last_active":"2024-02-19T14:52:06.247064+0000","last_peered":"2024-02-19T14:52:06.247064+0000","last_clean":"2024-02-19T14:52:06.247064+0000","last_became_active":"2024-02-19T14:52:00.264838+0000","last_became_peered":"2024-02-19T14:52:00.264838+0000","last_unstale":"2024-02-19T14:52:06.247064+0000","last_undegraded":"2024-02-19T14:52:06.247064+0000","last_fullsized":"2024-02-19T14:52:06.247064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:09.545834+0000","last_change":"2024-02-19T14:51:59.260164+0000","last_active":"2024-02-19T14:52:09.545834+0000","last_peered":"2024-02-19T14:52:09.545834+0000","last_clean":"2024-02-19T14:52:09.545834+0000","last_became_active":"2024-02-19T14:51:59.259921+0000","last_became_peered":"2024-02-19T14:51:59.259921+0000","last_unstale":"2024-02-19T14:52:09.545834+0000","last_undegraded":"2024-02-19T14:52:09.545834+0000","last_fullsized":"2024-02-19T14:52:09.545834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:50:08.794898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228768+0000","last_change":"2024-02-19T14:52:04.282972+0000","last_active":"2024-02-19T14:52:07.228768+0000","last_peered":"2024-02-19T14:52:07.228768+0000","last_clean":"2024-02-19T14:52:07.228768+0000","last_became_active":"2024-02-19T14:52:04.282593+0000","last_became_peered":"2024-02-19T14:52:04.282593+0000","last_unstale":"2024-02-19T14:52:07.228768+0000","last_undegraded":"2024-02-19T14:52:07.228768+0000","last_fullsized":"2024-02-19T14:52:07.228768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:12.773338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283293+0000","last_change":"2024-02-19T14:52:00.263099+0000","last_active":"2024-02-19T14:52:05.283293+0000","last_peered":"2024-02-19T14:52:05.283293+0000","last_clean":"2024-02-19T14:52:05.283293+0000","last_became_active":"2024-02-19T14:52:00.260826+0000","last_became_peered":"2024-02-19T14:52:00.260826+0000","last_unstale":"2024-02-19T14:52:05.283293+0000","last_undegraded":"2024-02-19T14:52:05.283293+0000","last_fullsized":"2024-02-19T14:52:05.283293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:49.639204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.229103+0000","last_change":"2024-02-19T14:51:59.258165+0000","last_active":"2024-02-19T14:52:07.229103+0000","last_peered":"2024-02-19T14:52:07.229103+0000","last_clean":"2024-02-19T14:52:07.229103+0000","last_became_active":"2024-02-19T14:51:59.257847+0000","last_became_peered":"2024-02-19T14:51:59.257847+0000","last_unstale":"2024-02-19T14:52:07.229103+0000","last_undegraded":"2024-02-19T14:52:07.229103+0000","last_fullsized":"2024-02-19T14:52:07.229103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:40.336017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284313+0000","last_change":"2024-02-19T14:52:04.275097+0000","last_active":"2024-02-19T14:52:05.284313+0000","last_peered":"2024-02-19T14:52:05.284313+0000","last_clean":"2024-02-19T14:52:05.284313+0000","last_became_active":"2024-02-19T14:52:04.274720+0000","last_became_peered":"2024-02-19T14:52:04.274720+0000","last_unstale":"2024-02-19T14:52:05.284313+0000","last_undegraded":"2024-02-19T14:52:05.284313+0000","last_fullsized":"2024-02-19T14:52:05.284313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:14.889510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226316+0000","last_change":"2024-02-19T14:52:00.262791+0000","last_active":"2024-02-19T14:52:06.226316+0000","last_peered":"2024-02-19T14:52:06.226316+0000","last_clean":"2024-02-19T14:52:06.226316+0000","last_became_active":"2024-02-19T14:52:00.262201+0000","last_became_peered":"2024-02-19T14:52:00.262201+0000","last_unstale":"2024-02-19T14:52:06.226316+0000","last_undegraded":"2024-02-19T14:52:06.226316+0000","last_fullsized":"2024-02-19T14:52:06.226316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:23.961482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283591+0000","last_change":"2024-02-19T14:51:59.255762+0000","last_active":"2024-02-19T14:52:05.283591+0000","last_peered":"2024-02-19T14:52:05.283591+0000","last_clean":"2024-02-19T14:52:05.283591+0000","last_became_active":"2024-02-19T14:51:59.255418+0000","last_became_peered":"2024-02-19T14:51:59.255418+0000","last_unstale":"2024-02-19T14:52:05.283591+0000","last_undegraded":"2024-02-19T14:52:05.283591+0000","last_fullsized":"2024-02-19T14:52:05.283591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:23.794135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334347+0000","last_change":"2024-02-19T14:52:04.275183+0000","last_active":"2024-02-19T14:52:07.334347+0000","last_peered":"2024-02-19T14:52:07.334347+0000","last_clean":"2024-02-19T14:52:07.334347+0000","last_became_active":"2024-02-19T14:52:04.274796+0000","last_became_peered":"2024-02-19T14:52:04.274796+0000","last_unstale":"2024-02-19T14:52:07.334347+0000","last_undegraded":"2024-02-19T14:52:07.334347+0000","last_fullsized":"2024-02-19T14:52:07.334347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:31:17.099296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283483+0000","last_change":"2024-02-19T14:52:00.260440+0000","last_active":"2024-02-19T14:52:05.283483+0000","last_peered":"2024-02-19T14:52:05.283483+0000","last_clean":"2024-02-19T14:52:05.283483+0000","last_became_active":"2024-02-19T14:52:00.259406+0000","last_became_peered":"2024-02-19T14:52:00.259406+0000","last_unstale":"2024-02-19T14:52:05.283483+0000","last_undegraded":"2024-02-19T14:52:05.283483+0000","last_fullsized":"2024-02-19T14:52:05.283483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:59:23.648505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530584+0000","last_change":"2024-02-19T14:51:59.254054+0000","last_active":"2024-02-19T14:52:05.530584+0000","last_peered":"2024-02-19T14:52:05.530584+0000","last_clean":"2024-02-19T14:52:05.530584+0000","last_became_active":"2024-02-19T14:51:59.253751+0000","last_became_peered":"2024-02-19T14:51:59.253751+0000","last_unstale":"2024-02-19T14:52:05.530584+0000","last_undegraded":"2024-02-19T14:52:05.530584+0000","last_fullsized":"2024-02-19T14:52:05.530584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:07.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334305+0000","last_change":"2024-02-19T14:52:04.433602+0000","last_active":"2024-02-19T14:52:07.334305+0000","last_peered":"2024-02-19T14:52:07.334305+0000","last_clean":"2024-02-19T14:52:07.334305+0000","last_became_active":"2024-02-19T14:52:04.433250+0000","last_became_peered":"2024-02-19T14:52:04.433250+0000","last_unstale":"2024-02-19T14:52:07.334305+0000","last_undegraded":"2024-02-19T14:52:07.334305+0000","last_fullsized":"2024-02-19T14:52:07.334305+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:08:52.470860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226005+0000","last_change":"2024-02-19T14:52:04.435369+0000","last_active":"2024-02-19T14:52:06.226005+0000","last_peered":"2024-02-19T14:52:06.226005+0000","last_clean":"2024-02-19T14:52:06.226005+0000","last_became_active":"2024-02-19T14:52:04.435024+0000","last_became_peered":"2024-02-19T14:52:04.435024+0000","last_unstale":"2024-02-19T14:52:06.226005+0000","last_undegraded":"2024-02-19T14:52:06.226005+0000","last_fullsized":"2024-02-19T14:52:06.226005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:58.843829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228450+0000","last_change":"2024-02-19T14:52:00.266810+0000","last_active":"2024-02-19T14:52:07.228450+0000","last_peered":"2024-02-19T14:52:07.228450+0000","last_clean":"2024-02-19T14:52:07.228450+0000","last_became_active":"2024-02-19T14:52:00.266548+0000","last_became_peered":"2024-02-19T14:52:00.266548+0000","last_unstale":"2024-02-19T14:52:07.228450+0000","last_undegraded":"2024-02-19T14:52:07.228450+0000","last_fullsized":"2024-02-19T14:52:07.228450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282610+0000","last_change":"2024-02-19T14:51:59.248141+0000","last_active":"2024-02-19T14:52:05.282610+0000","last_peered":"2024-02-19T14:52:05.282610+0000","last_clean":"2024-02-19T14:52:05.282610+0000","last_became_active":"2024-02-19T14:51:59.247849+0000","last_became_peered":"2024-02-19T14:51:59.247849+0000","last_unstale":"2024-02-19T14:52:05.282610+0000","last_undegraded":"2024-02-19T14:52:05.282610+0000","last_fullsized":"2024-02-19T14:52:05.282610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530585+0000","last_change":"2024-02-19T14:52:04.282329+0000","last_active":"2024-02-19T14:52:05.530585+0000","last_peered":"2024-02-19T14:52:05.530585+0000","last_clean":"2024-02-19T14:52:05.530585+0000","last_became_active":"2024-02-19T14:52:04.281869+0000","last_became_peered":"2024-02-19T14:52:04.281869+0000","last_unstale":"2024-02-19T14:52:05.530585+0000","last_undegraded":"2024-02-19T14:52:05.530585+0000","last_fullsized":"2024-02-19T14:52:05.530585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:49.675162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334363+0000","last_change":"2024-02-19T14:52:00.266718+0000","last_active":"2024-02-19T14:52:07.334363+0000","last_peered":"2024-02-19T14:52:07.334363+0000","last_clean":"2024-02-19T14:52:07.334363+0000","last_became_active":"2024-02-19T14:52:00.266351+0000","last_became_peered":"2024-02-19T14:52:00.266351+0000","last_unstale":"2024-02-19T14:52:07.334363+0000","last_undegraded":"2024-02-19T14:52:07.334363+0000","last_fullsized":"2024-02-19T14:52:07.334363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333980+0000","last_change":"2024-02-19T14:51:59.259820+0000","last_active":"2024-02-19T14:52:07.333980+0000","last_peered":"2024-02-19T14:52:07.333980+0000","last_clean":"2024-02-19T14:52:07.333980+0000","last_became_active":"2024-02-19T14:51:59.259418+0000","last_became_peered":"2024-02-19T14:51:59.259418+0000","last_unstale":"2024-02-19T14:52:07.333980+0000","last_undegraded":"2024-02-19T14:52:07.333980+0000","last_fullsized":"2024-02-19T14:52:07.333980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:42:41.281357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530291+0000","last_change":"2024-02-19T14:51:59.252686+0000","last_active":"2024-02-19T14:52:05.530291+0000","last_peered":"2024-02-19T14:52:05.530291+0000","last_clean":"2024-02-19T14:52:05.530291+0000","last_became_active":"2024-02-19T14:51:59.252420+0000","last_became_peered":"2024-02-19T14:51:59.252420+0000","last_unstale":"2024-02-19T14:52:05.530291+0000","last_undegraded":"2024-02-19T14:52:05.530291+0000","last_fullsized":"2024-02-19T14:52:05.530291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:55.913735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.226023+0000","last_change":"2024-02-19T14:52:00.264522+0000","last_active":"2024-02-19T14:52:06.226023+0000","last_peered":"2024-02-19T14:52:06.226023+0000","last_clean":"2024-02-19T14:52:06.226023+0000","last_became_active":"2024-02-19T14:52:00.264273+0000","last_became_peered":"2024-02-19T14:52:00.264273+0000","last_unstale":"2024-02-19T14:52:06.226023+0000","last_undegraded":"2024-02-19T14:52:06.226023+0000","last_fullsized":"2024-02-19T14:52:06.226023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281616+0000","last_change":"2024-02-19T14:52:04.275545+0000","last_active":"2024-02-19T14:52:05.281616+0000","last_peered":"2024-02-19T14:52:05.281616+0000","last_clean":"2024-02-19T14:52:05.281616+0000","last_became_active":"2024-02-19T14:52:04.275140+0000","last_became_peered":"2024-02-19T14:52:04.275140+0000","last_unstale":"2024-02-19T14:52:05.281616+0000","last_undegraded":"2024-02-19T14:52:05.281616+0000","last_fullsized":"2024-02-19T14:52:05.281616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:27.987737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.333998+0000","last_change":"2024-02-19T14:51:59.262930+0000","last_active":"2024-02-19T14:52:07.333998+0000","last_peered":"2024-02-19T14:52:07.333998+0000","last_clean":"2024-02-19T14:52:07.333998+0000","last_became_active":"2024-02-19T14:51:59.262635+0000","last_became_peered":"2024-02-19T14:51:59.262635+0000","last_unstale":"2024-02-19T14:52:07.333998+0000","last_undegraded":"2024-02-19T14:52:07.333998+0000","last_fullsized":"2024-02-19T14:52:07.333998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:58.224602+0000","last_clean_scrub_stamp":"2024-02-19T14:51:58.224602+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:21.322778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284375+0000","last_change":"2024-02-19T14:52:00.260213+0000","last_active":"2024-02-19T14:52:05.284375+0000","last_peered":"2024-02-19T14:52:05.284375+0000","last_clean":"2024-02-19T14:52:05.284375+0000","last_became_active":"2024-02-19T14:52:00.257148+0000","last_became_peered":"2024-02-19T14:52:00.257148+0000","last_unstale":"2024-02-19T14:52:05.284375+0000","last_undegraded":"2024-02-19T14:52:05.284375+0000","last_fullsized":"2024-02-19T14:52:05.284375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:56.918797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335231+0000","last_change":"2024-02-19T14:52:04.276033+0000","last_active":"2024-02-19T14:52:07.335231+0000","last_peered":"2024-02-19T14:52:07.335231+0000","last_clean":"2024-02-19T14:52:07.335231+0000","last_became_active":"2024-02-19T14:52:04.275671+0000","last_became_peered":"2024-02-19T14:52:04.275671+0000","last_unstale":"2024-02-19T14:52:07.335231+0000","last_undegraded":"2024-02-19T14:52:07.335231+0000","last_fullsized":"2024-02-19T14:52:07.335231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:03.255919+0000","last_clean_scrub_stamp":"2024-02-19T14:52:03.255919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:22:42.880381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.529955+0000","last_change":"2024-02-19T14:52:00.270697+0000","last_active":"2024-02-19T14:52:05.529955+0000","last_peered":"2024-02-19T14:52:05.529955+0000","last_clean":"2024-02-19T14:52:05.529955+0000","last_became_active":"2024-02-19T14:52:00.269449+0000","last_became_peered":"2024-02-19T14:52:00.269449+0000","last_unstale":"2024-02-19T14:52:05.529955+0000","last_undegraded":"2024-02-19T14:52:05.529955+0000","last_fullsized":"2024-02-19T14:52:05.529955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281315+0000","last_change":"2024-02-19T14:52:00.263056+0000","last_active":"2024-02-19T14:52:05.281315+0000","last_peered":"2024-02-19T14:52:05.281315+0000","last_clean":"2024-02-19T14:52:05.281315+0000","last_became_active":"2024-02-19T14:52:00.262801+0000","last_became_peered":"2024-02-19T14:52:00.262801+0000","last_unstale":"2024-02-19T14:52:05.281315+0000","last_undegraded":"2024-02-19T14:52:05.281315+0000","last_fullsized":"2024-02-19T14:52:05.281315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:20:01.230354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283161+0000","last_change":"2024-02-19T14:52:00.270389+0000","last_active":"2024-02-19T14:52:05.283161+0000","last_peered":"2024-02-19T14:52:05.283161+0000","last_clean":"2024-02-19T14:52:05.283161+0000","last_became_active":"2024-02-19T14:52:00.267246+0000","last_became_peered":"2024-02-19T14:52:00.267246+0000","last_unstale":"2024-02-19T14:52:05.283161+0000","last_undegraded":"2024-02-19T14:52:05.283161+0000","last_fullsized":"2024-02-19T14:52:05.283161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:29.736719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225914+0000","last_change":"2024-02-19T14:52:00.265624+0000","last_active":"2024-02-19T14:52:06.225914+0000","last_peered":"2024-02-19T14:52:06.225914+0000","last_clean":"2024-02-19T14:52:06.225914+0000","last_became_active":"2024-02-19T14:52:00.265377+0000","last_became_peered":"2024-02-19T14:52:00.265377+0000","last_unstale":"2024-02-19T14:52:06.225914+0000","last_undegraded":"2024-02-19T14:52:06.225914+0000","last_fullsized":"2024-02-19T14:52:06.225914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335142+0000","last_change":"2024-02-19T14:52:00.263043+0000","last_active":"2024-02-19T14:52:07.335142+0000","last_peered":"2024-02-19T14:52:07.335142+0000","last_clean":"2024-02-19T14:52:07.335142+0000","last_became_active":"2024-02-19T14:52:00.262754+0000","last_became_peered":"2024-02-19T14:52:00.262754+0000","last_unstale":"2024-02-19T14:52:07.335142+0000","last_undegraded":"2024-02-19T14:52:07.335142+0000","last_fullsized":"2024-02-19T14:52:07.335142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228927+0000","last_change":"2024-02-19T14:52:00.266890+0000","last_active":"2024-02-19T14:52:07.228927+0000","last_peered":"2024-02-19T14:52:07.228927+0000","last_clean":"2024-02-19T14:52:07.228927+0000","last_became_active":"2024-02-19T14:52:00.266574+0000","last_became_peered":"2024-02-19T14:52:00.266574+0000","last_unstale":"2024-02-19T14:52:07.228927+0000","last_undegraded":"2024-02-19T14:52:07.228927+0000","last_fullsized":"2024-02-19T14:52:07.228927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530706+0000","last_change":"2024-02-19T14:52:00.261225+0000","last_active":"2024-02-19T14:52:05.530706+0000","last_peered":"2024-02-19T14:52:05.530706+0000","last_clean":"2024-02-19T14:52:05.530706+0000","last_became_active":"2024-02-19T14:52:00.259666+0000","last_became_peered":"2024-02-19T14:52:00.259666+0000","last_unstale":"2024-02-19T14:52:05.530706+0000","last_undegraded":"2024-02-19T14:52:05.530706+0000","last_fullsized":"2024-02-19T14:52:05.530706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.335354+0000","last_change":"2024-02-19T14:52:00.264377+0000","last_active":"2024-02-19T14:52:07.335354+0000","last_peered":"2024-02-19T14:52:07.335354+0000","last_clean":"2024-02-19T14:52:07.335354+0000","last_became_active":"2024-02-19T14:52:00.264082+0000","last_became_peered":"2024-02-19T14:52:00.264082+0000","last_unstale":"2024-02-19T14:52:07.335354+0000","last_undegraded":"2024-02-19T14:52:07.335354+0000","last_fullsized":"2024-02-19T14:52:07.335354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282561+0000","last_change":"2024-02-19T14:52:00.265423+0000","last_active":"2024-02-19T14:52:05.282561+0000","last_peered":"2024-02-19T14:52:05.282561+0000","last_clean":"2024-02-19T14:52:05.282561+0000","last_became_active":"2024-02-19T14:52:00.265132+0000","last_became_peered":"2024-02-19T14:52:00.265132+0000","last_unstale":"2024-02-19T14:52:05.282561+0000","last_undegraded":"2024-02-19T14:52:05.282561+0000","last_fullsized":"2024-02-19T14:52:05.282561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334543+0000","last_change":"2024-02-19T14:52:00.259058+0000","last_active":"2024-02-19T14:52:07.334543+0000","last_peered":"2024-02-19T14:52:07.334543+0000","last_clean":"2024-02-19T14:52:07.334543+0000","last_became_active":"2024-02-19T14:52:00.258228+0000","last_became_peered":"2024-02-19T14:52:00.258228+0000","last_unstale":"2024-02-19T14:52:07.334543+0000","last_undegraded":"2024-02-19T14:52:07.334543+0000","last_fullsized":"2024-02-19T14:52:07.334543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283058+0000","last_change":"2024-02-19T14:52:00.258261+0000","last_active":"2024-02-19T14:52:05.283058+0000","last_peered":"2024-02-19T14:52:05.283058+0000","last_clean":"2024-02-19T14:52:05.283058+0000","last_became_active":"2024-02-19T14:52:00.257970+0000","last_became_peered":"2024-02-19T14:52:00.257970+0000","last_unstale":"2024-02-19T14:52:05.283058+0000","last_undegraded":"2024-02-19T14:52:05.283058+0000","last_fullsized":"2024-02-19T14:52:05.283058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225756+0000","last_change":"2024-02-19T14:52:00.268547+0000","last_active":"2024-02-19T14:52:06.225756+0000","last_peered":"2024-02-19T14:52:06.225756+0000","last_clean":"2024-02-19T14:52:06.225756+0000","last_became_active":"2024-02-19T14:52:00.268241+0000","last_became_peered":"2024-02-19T14:52:00.268241+0000","last_unstale":"2024-02-19T14:52:06.225756+0000","last_undegraded":"2024-02-19T14:52:06.225756+0000","last_fullsized":"2024-02-19T14:52:06.225756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530412+0000","last_change":"2024-02-19T14:52:00.262779+0000","last_active":"2024-02-19T14:52:05.530412+0000","last_peered":"2024-02-19T14:52:05.530412+0000","last_clean":"2024-02-19T14:52:05.530412+0000","last_became_active":"2024-02-19T14:52:00.262476+0000","last_became_peered":"2024-02-19T14:52:00.262476+0000","last_unstale":"2024-02-19T14:52:05.530412+0000","last_undegraded":"2024-02-19T14:52:05.530412+0000","last_fullsized":"2024-02-19T14:52:05.530412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247311+0000","last_change":"2024-02-19T14:52:00.272996+0000","last_active":"2024-02-19T14:52:06.247311+0000","last_peered":"2024-02-19T14:52:06.247311+0000","last_clean":"2024-02-19T14:52:06.247311+0000","last_became_active":"2024-02-19T14:52:00.271013+0000","last_became_peered":"2024-02-19T14:52:00.271013+0000","last_unstale":"2024-02-19T14:52:06.247311+0000","last_undegraded":"2024-02-19T14:52:06.247311+0000","last_fullsized":"2024-02-19T14:52:06.247311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:54.817573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.281593+0000","last_change":"2024-02-19T14:52:00.263763+0000","last_active":"2024-02-19T14:52:05.281593+0000","last_peered":"2024-02-19T14:52:05.281593+0000","last_clean":"2024-02-19T14:52:05.281593+0000","last_became_active":"2024-02-19T14:52:00.263488+0000","last_became_peered":"2024-02-19T14:52:00.263488+0000","last_unstale":"2024-02-19T14:52:05.281593+0000","last_undegraded":"2024-02-19T14:52:05.281593+0000","last_fullsized":"2024-02-19T14:52:05.281593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:46.599373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.284503+0000","last_change":"2024-02-19T14:52:00.258879+0000","last_active":"2024-02-19T14:52:05.284503+0000","last_peered":"2024-02-19T14:52:05.284503+0000","last_clean":"2024-02-19T14:52:05.284503+0000","last_became_active":"2024-02-19T14:52:00.257612+0000","last_became_peered":"2024-02-19T14:52:00.257612+0000","last_unstale":"2024-02-19T14:52:05.284503+0000","last_undegraded":"2024-02-19T14:52:05.284503+0000","last_fullsized":"2024-02-19T14:52:05.284503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:45.648907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282447+0000","last_change":"2024-02-19T14:52:00.268671+0000","last_active":"2024-02-19T14:52:05.282447+0000","last_peered":"2024-02-19T14:52:05.282447+0000","last_clean":"2024-02-19T14:52:05.282447+0000","last_became_active":"2024-02-19T14:52:00.268351+0000","last_became_peered":"2024-02-19T14:52:00.268351+0000","last_unstale":"2024-02-19T14:52:05.282447+0000","last_undegraded":"2024-02-19T14:52:05.282447+0000","last_fullsized":"2024-02-19T14:52:05.282447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:43.427530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228525+0000","last_change":"2024-02-19T14:52:00.249695+0000","last_active":"2024-02-19T14:52:07.228525+0000","last_peered":"2024-02-19T14:52:07.228525+0000","last_clean":"2024-02-19T14:52:07.228525+0000","last_became_active":"2024-02-19T14:52:00.249406+0000","last_became_peered":"2024-02-19T14:52:00.249406+0000","last_unstale":"2024-02-19T14:52:07.228525+0000","last_undegraded":"2024-02-19T14:52:07.228525+0000","last_fullsized":"2024-02-19T14:52:07.228525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:30:19.756628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225669+0000","last_change":"2024-02-19T14:52:00.259566+0000","last_active":"2024-02-19T14:52:06.225669+0000","last_peered":"2024-02-19T14:52:06.225669+0000","last_clean":"2024-02-19T14:52:06.225669+0000","last_became_active":"2024-02-19T14:52:00.259290+0000","last_became_peered":"2024-02-19T14:52:00.259290+0000","last_unstale":"2024-02-19T14:52:06.225669+0000","last_undegraded":"2024-02-19T14:52:06.225669+0000","last_fullsized":"2024-02-19T14:52:06.225669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:42:39.149178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334965+0000","last_change":"2024-02-19T14:52:00.267359+0000","last_active":"2024-02-19T14:52:07.334965+0000","last_peered":"2024-02-19T14:52:07.334965+0000","last_clean":"2024-02-19T14:52:07.334965+0000","last_became_active":"2024-02-19T14:52:00.266119+0000","last_became_peered":"2024-02-19T14:52:00.266119+0000","last_unstale":"2024-02-19T14:52:07.334965+0000","last_undegraded":"2024-02-19T14:52:07.334965+0000","last_fullsized":"2024-02-19T14:52:07.334965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283181+0000","last_change":"2024-02-19T14:52:00.262531+0000","last_active":"2024-02-19T14:52:05.283181+0000","last_peered":"2024-02-19T14:52:05.283181+0000","last_clean":"2024-02-19T14:52:05.283181+0000","last_became_active":"2024-02-19T14:52:00.262240+0000","last_became_peered":"2024-02-19T14:52:00.262240+0000","last_unstale":"2024-02-19T14:52:05.283181+0000","last_undegraded":"2024-02-19T14:52:05.283181+0000","last_fullsized":"2024-02-19T14:52:05.283181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:41.341079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530384+0000","last_change":"2024-02-19T14:52:00.266149+0000","last_active":"2024-02-19T14:52:05.530384+0000","last_peered":"2024-02-19T14:52:05.530384+0000","last_clean":"2024-02-19T14:52:05.530384+0000","last_became_active":"2024-02-19T14:52:00.265873+0000","last_became_peered":"2024-02-19T14:52:00.265873+0000","last_unstale":"2024-02-19T14:52:05.530384+0000","last_undegraded":"2024-02-19T14:52:05.530384+0000","last_fullsized":"2024-02-19T14:52:05.530384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:49.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.530665+0000","last_change":"2024-02-19T14:52:00.266144+0000","last_active":"2024-02-19T14:52:05.530665+0000","last_peered":"2024-02-19T14:52:05.530665+0000","last_clean":"2024-02-19T14:52:05.530665+0000","last_became_active":"2024-02-19T14:52:00.265852+0000","last_became_peered":"2024-02-19T14:52:00.265852+0000","last_unstale":"2024-02-19T14:52:05.530665+0000","last_undegraded":"2024-02-19T14:52:05.530665+0000","last_fullsized":"2024-02-19T14:52:05.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:11:42.445104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247502+0000","last_change":"2024-02-19T14:52:00.265908+0000","last_active":"2024-02-19T14:52:06.247502+0000","last_peered":"2024-02-19T14:52:06.247502+0000","last_clean":"2024-02-19T14:52:06.247502+0000","last_became_active":"2024-02-19T14:52:00.265630+0000","last_became_peered":"2024-02-19T14:52:00.265630+0000","last_unstale":"2024-02-19T14:52:06.247502+0000","last_undegraded":"2024-02-19T14:52:06.247502+0000","last_fullsized":"2024-02-19T14:52:06.247502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:11:06.056284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.334181+0000","last_change":"2024-02-19T14:52:00.267195+0000","last_active":"2024-02-19T14:52:07.334181+0000","last_peered":"2024-02-19T14:52:07.334181+0000","last_clean":"2024-02-19T14:52:07.334181+0000","last_became_active":"2024-02-19T14:52:00.266185+0000","last_became_peered":"2024-02-19T14:52:00.266185+0000","last_unstale":"2024-02-19T14:52:07.334181+0000","last_undegraded":"2024-02-19T14:52:07.334181+0000","last_fullsized":"2024-02-19T14:52:07.334181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:08.948966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246858+0000","last_change":"2024-02-19T14:52:00.266943+0000","last_active":"2024-02-19T14:52:06.246858+0000","last_peered":"2024-02-19T14:52:06.246858+0000","last_clean":"2024-02-19T14:52:06.246858+0000","last_became_active":"2024-02-19T14:52:00.266665+0000","last_became_peered":"2024-02-19T14:52:00.266665+0000","last_unstale":"2024-02-19T14:52:06.246858+0000","last_undegraded":"2024-02-19T14:52:06.246858+0000","last_fullsized":"2024-02-19T14:52:06.246858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:26.872150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.282952+0000","last_change":"2024-02-19T14:52:00.261505+0000","last_active":"2024-02-19T14:52:05.282952+0000","last_peered":"2024-02-19T14:52:05.282952+0000","last_clean":"2024-02-19T14:52:05.282952+0000","last_became_active":"2024-02-19T14:52:00.261212+0000","last_became_peered":"2024-02-19T14:52:00.261212+0000","last_unstale":"2024-02-19T14:52:05.282952+0000","last_undegraded":"2024-02-19T14:52:05.282952+0000","last_fullsized":"2024-02-19T14:52:05.282952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.246984+0000","last_change":"2024-02-19T14:52:00.261481+0000","last_active":"2024-02-19T14:52:06.246984+0000","last_peered":"2024-02-19T14:52:06.246984+0000","last_clean":"2024-02-19T14:52:06.246984+0000","last_became_active":"2024-02-19T14:52:00.261205+0000","last_became_peered":"2024-02-19T14:52:00.261205+0000","last_unstale":"2024-02-19T14:52:06.246984+0000","last_undegraded":"2024-02-19T14:52:06.246984+0000","last_fullsized":"2024-02-19T14:52:06.246984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:22.677463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:05.283153+0000","last_change":"2024-02-19T14:52:00.266814+0000","last_active":"2024-02-19T14:52:05.283153+0000","last_peered":"2024-02-19T14:52:05.283153+0000","last_clean":"2024-02-19T14:52:05.283153+0000","last_became_active":"2024-02-19T14:52:00.266019+0000","last_became_peered":"2024-02-19T14:52:00.266019+0000","last_unstale":"2024-02-19T14:52:05.283153+0000","last_undegraded":"2024-02-19T14:52:05.283153+0000","last_fullsized":"2024-02-19T14:52:05.283153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:06:10.068930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.225905+0000","last_change":"2024-02-19T14:52:00.260863+0000","last_active":"2024-02-19T14:52:06.225905+0000","last_peered":"2024-02-19T14:52:06.225905+0000","last_clean":"2024-02-19T14:52:06.225905+0000","last_became_active":"2024-02-19T14:52:00.260543+0000","last_became_peered":"2024-02-19T14:52:00.260543+0000","last_unstale":"2024-02-19T14:52:06.225905+0000","last_undegraded":"2024-02-19T14:52:06.225905+0000","last_fullsized":"2024-02-19T14:52:06.225905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:48:56.561657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:07.228999+0000","last_change":"2024-02-19T14:52:00.260306+0000","last_active":"2024-02-19T14:52:07.228999+0000","last_peered":"2024-02-19T14:52:07.228999+0000","last_clean":"2024-02-19T14:52:07.228999+0000","last_became_active":"2024-02-19T14:52:00.260026+0000","last_became_peered":"2024-02-19T14:52:00.260026+0000","last_unstale":"2024-02-19T14:52:07.228999+0000","last_undegraded":"2024-02-19T14:52:07.228999+0000","last_fullsized":"2024-02-19T14:52:07.228999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:09:08.961552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-19T14:52:06.247442+0000","last_change":"2024-02-19T14:52:00.263320+0000","last_active":"2024-02-19T14:52:06.247442+0000","last_peered":"2024-02-19T14:52:06.247442+0000","last_clean":"2024-02-19T14:52:06.247442+0000","last_became_active":"2024-02-19T14:52:00.263044+0000","last_became_peered":"2024-02-19T14:52:00.263044+0000","last_unstale":"2024-02-19T14:52:06.247442+0000","last_undegraded":"2024-02-19T14:52:06.247442+0000","last_fullsized":"2024-02-19T14:52:06.247442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:59.229663+0000","last_clean_scrub_stamp":"2024-02-19T14:51:59.229663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:08.747083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:52:22.222 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-19T14:52:22.222 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-19T14:52:22.222 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-02-19T14:52:22.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T14:52:22.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.381+0000 7fc6e5bc9640 1 Processor -- start 2024-02-19T14:52:22.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.381+0000 7fc6e5bc9640 1 -- start start 2024-02-19T14:52:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6e5bc9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc6e00ac390 0x7fc6e00ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6e5bc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e0158c40 con 0x7fc6e00c55f0 2024-02-19T14:52:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6e5bc9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e00c8de0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6e5bc9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e00c5a30 con 0x7fc6e00ac390 2024-02-19T14:52:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6df7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc6e00ac390 0x7fc6e00ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6dffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc6e00c55f0 0x7fc6e015f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57832/0 (socket says 172.21.15.78:57832) 2024-02-19T14:52:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.383+0000 7fc6dffff640 1 -- 172.21.15.78:0/1910423759 learned_addr learned my addr 172.21.15.78:0/1910423759 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.382+0000 7fc6df7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc6e00ac390 0x7fc6e00ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52690/0 (socket says 172.21.15.78:52690) 2024-02-19T14:52:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.383+0000 7fc6df7fe640 1 -- 172.21.15.78:0/1910423759 >> v1:172.21.15.150:6789/0 conn(0x7fc6e00bdf20 legacy=0x7fc6e0158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.383+0000 7fc6df7fe640 1 -- 172.21.15.78:0/1910423759 >> v1:172.21.15.78:6789/0 conn(0x7fc6e00c55f0 legacy=0x7fc6e015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.383+0000 7fc6df7fe640 1 -- 172.21.15.78:0/1910423759 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e0155a60 con 0x7fc6e00ac390 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.383+0000 7fc6df7fe640 1 --2- 172.21.15.78:0/1910423759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00ac390 0x7fc6e00ac770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009d90 tx=0x7fc6d002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f572dc92ba1ba8c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6de7fc640 1 -- 172.21.15.78:0/1910423759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d002cbd0 con 0x7fc6e00ac390 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6de7fc640 1 -- 172.21.15.78:0/1910423759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d002cd30 con 0x7fc6e00ac390 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6de7fc640 1 -- 172.21.15.78:0/1910423759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d002cea0 con 0x7fc6e00ac390 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/1910423759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00ac390 msgr2=0x7fc6e00ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/1910423759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00ac390 0x7fc6e00ac770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009d90 tx=0x7fc6d002be20 comp rx=0 tx=0).stop 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/1910423759 shutdown_connections 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/1910423759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00ac390 0x7fc6e00ac770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/1910423759 >> 172.21.15.78:0/1910423759 conn(0x7fc6e00ca890 msgr2=0x7fc6e00caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:22.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/1910423759 shutdown_connections 2024-02-19T14:52:22.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.384+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/1910423759 wait complete. 2024-02-19T14:52:22.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.385+0000 7fc6e5bc9640 1 Processor -- start 2024-02-19T14:52:22.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 -- start start 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6e00ac390 0x7fc6e01518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 0x7fc6e0151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6e00c55f0 0x7fc6e01ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00be7e0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e0152370 con 0x7fc6e00c55f0 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6e5bc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e00c3c10 con 0x7fc6e00ac390 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6deffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 0x7fc6e0151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6deffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 0x7fc6e0151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52698/0 (socket says 172.21.15.78:52698) 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6deffd640 1 -- 172.21.15.78:0/938516637 learned_addr learned my addr 172.21.15.78:0/938516637 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6dffff640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6e00c55f0 0x7fc6e01ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.386+0000 7fc6df7fe640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6e00ac390 0x7fc6e01518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6deffd640 1 -- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6e00ac390 msgr2=0x7fc6e01518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6deffd640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6e00ac390 0x7fc6e01518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6deffd640 1 -- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6e00c55f0 msgr2=0x7fc6e01ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6deffd640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6e00c55f0 0x7fc6e01ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6deffd640 1 -- 172.21.15.78:0/938516637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d0009a70 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6dffff640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6e00c55f0 0x7fc6e01ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6deffd640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 0x7fc6e0151e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c440 tx=0x7fc6c8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80052d0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c8005430 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8007260 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e00c3ef0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.387+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e00bea30 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.388+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e0074990 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.388+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc6c80073c0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.389+0000 7fc6d7fff640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc6c006b810 0x7fc6c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.389+0000 7fc6df7fe640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc6c006b810 0x7fc6c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.389+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc6c8089870 con 0x7fc6e00bdf20 2024-02-19T14:52:22.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.389+0000 7fc6df7fe640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc6c006b810 0x7fc6c006dcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0036000 tx=0x7fc6d0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.393+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc6c8053fe0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.604+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc6e006d300 con 0x7fc6e00bdf20 2024-02-19T14:52:22.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.604+0000 7fc6d7fff640 1 -- 172.21.15.78:0/938516637 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc6c80576a0 con 0x7fc6e00bdf20 2024-02-19T14:52:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:22.607 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc6c006b810 msgr2=0x7fc6c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc6c006b810 0x7fc6c006dcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0036000 tx=0x7fc6d0030000 comp rx=0 tx=0).stop 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 msgr2=0x7fc6e0151e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 0x7fc6e0151e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c440 tx=0x7fc6c8001e70 comp rx=0 tx=0).stop 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 shutdown_connections 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc6c006b810 0x7fc6c006dcd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e00bdf20 0x7fc6e0151e30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6e00c55f0 0x7fc6e01ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 --2- 172.21.15.78:0/938516637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6e00ac390 0x7fc6e01518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.606+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 >> 172.21.15.78:0/938516637 conn(0x7fc6e00ca890 msgr2=0x7fc6e0153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.607+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 shutdown_connections 2024-02-19T14:52:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.607+0000 7fc6e5bc9640 1 -- 172.21.15.78:0/938516637 wait complete. 2024-02-19T14:52:22.620 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-02-19T14:52:22.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.719+0000 7fcd1131e640 1 Processor -- start 2024-02-19T14:52:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.719+0000 7fcd1131e640 1 -- start start 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd1131e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fcd0c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd1131e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c1551f0 con 0x7fcd0c06d910 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd1131e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c155490 con 0x7fcd0c152700 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd1131e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c155770 con 0x7fcd0c0c8e00 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcd0c06d910 0x7fcd0c15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57838/0 (socket says 172.21.15.78:57838) 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/1582751830 learned_addr learned my addr 172.21.15.78:0/1582751830 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:22.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd03fff640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fcd0c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147019941 0 0) 0x7fcd0c1551f0 con 0x7fcd0c06d910 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf8003520 con 0x7fcd0c06d910 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191405434 0 0) 0x7fcd0c155490 con 0x7fcd0c152700 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd0c1551f0 con 0x7fcd0c152700 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169278042 0 0) 0x7fccf8003520 con 0x7fcd0c06d910 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 msgr2=0x7fcd0c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fcd0c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 >> v1:172.21.15.150:6789/0 conn(0x7fcd0c152700 legacy=0x7fcd0c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c155a50 con 0x7fcd0c06d910 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fcd040019f0 con 0x7fcd0c06d910 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd03fff640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fcd0c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 >> v1:172.21.15.78:6789/0 conn(0x7fcd0c06d910 legacy=0x7fcd0c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.720+0000 7fcd0a7fc640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8003b30 0x7fccf8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0a7fc640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fccf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0a7fc640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf8004d00 0x7fccf8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c155770 con 0x7fccf8004d00 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf800ba60 con 0x7fccf8003b30 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf800bc20 con 0x7fcd0c0c8e00 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf8004d00 0x7fccf8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0affd640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8003b30 0x7fccf8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd03fff640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fccf80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 msgr2=0x7fccf80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0c8e00 0x7fccf80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8003b30 msgr2=0x7fccf8003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8003b30 0x7fccf8003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.721+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/1582751830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf4009400 con 0x7fccf8004d00 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf8004d00 0x7fccf8005180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400a7b0 tx=0x7fccf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1993868be2accb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04034630 con 0x7fccf8004d00 2024-02-19T14:52:22.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0402f070 con 0x7fccf8004d00 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd0a7fc640 1 -- 172.21.15.78:0/1582751830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd040349d0 con 0x7fccf8004d00 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd1131e640 1 -- 172.21.15.78:0/1582751830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf8004d00 msgr2=0x7fccf8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd1131e640 1 --2- 172.21.15.78:0/1582751830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf8004d00 0x7fccf8005180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400a7b0 tx=0x7fccf80057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/1582751830 reap_dead start 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd1131e640 1 -- 172.21.15.78:0/1582751830 shutdown_connections 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd1131e640 1 -- 172.21.15.78:0/1582751830 >> 172.21.15.78:0/1582751830 conn(0x7fcd0c0cac90 msgr2=0x7fcd0c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd1131e640 1 -- 172.21.15.78:0/1582751830 shutdown_connections 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.722+0000 7fcd1131e640 1 -- 172.21.15.78:0/1582751830 wait complete. 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.723+0000 7fcd1131e640 1 Processor -- start 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.723+0000 7fcd1131e640 1 -- start start 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd1131e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8004d00 0x7fcd0c1ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd1131e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0ab8d0 0x7fcd0c1ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd1131e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 0x7fcd0c1e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd1131e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0ac0d0 con 0x7fcd0c1e7920 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd1131e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0ad1b0 con 0x7fccf8004d00 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd1131e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0acbf0 con 0x7fcd0c0ab8d0 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd0b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 0x7fcd0c1e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd0b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 0x7fcd0c1e7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52720/0 (socket says 172.21.15.78:52720) 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/2518335193 learned_addr learned my addr 172.21.15.78:0/2518335193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd03fff640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0ab8d0 0x7fcd0c1ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd0affd640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8004d00 0x7fcd0c1ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.724+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0ab8d0 msgr2=0x7fcd0c1ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0ab8d0 0x7fcd0c1ebff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8004d00 msgr2=0x7fcd0c1ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8004d00 0x7fcd0c1ebab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/2518335193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c0aced0 con 0x7fcd0c1e7920 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd03fff640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0ab8d0 0x7fcd0c1ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:22.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 0x7fcd0c1e7dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400a8d0 tx=0x7fcd0c06d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04034630 con 0x7fcd0c1e7920 2024-02-19T14:52:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd04034790 con 0x7fcd0c1e7920 2024-02-19T14:52:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c0aadf0 con 0x7fcd0c1e7920 2024-02-19T14:52:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04034b00 con 0x7fcd0c1e7920 2024-02-19T14:52:22.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.725+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c0ab3b0 con 0x7fcd0c1e7920 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.726+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0c074990 con 0x7fcd0c1e7920 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.726+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcd04031030 con 0x7fcd0c1e7920 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.727+0000 7fcd037fe640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccec06b810 0x7fccec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.727+0000 7fcd0affd640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccec06b810 0x7fccec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.727+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcd040ac8c0 con 0x7fcd0c1e7920 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.727+0000 7fcd0affd640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccec06b810 0x7fccec06dcd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fccfc0018d0 tx=0x7fccfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:22.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.731+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd04077150 con 0x7fcd0c1e7920 2024-02-19T14:52:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.905+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0c074990 con 0x7fcd0c1e7920 2024-02-19T14:52:22.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.906+0000 7fcd037fe640 1 -- 172.21.15.78:0/2518335193 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7fcd0403ee60 con 0x7fcd0c1e7920 2024-02-19T14:52:22.909 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:22.909 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccec06b810 msgr2=0x7fccec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccec06b810 0x7fccec06dcd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fccfc0018d0 tx=0x7fccfc009040 comp rx=0 tx=0).stop 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 msgr2=0x7fcd0c1e7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 0x7fcd0c1e7dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400a8d0 tx=0x7fcd0c06d6b0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 shutdown_connections 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccec06b810 0x7fccec06dcd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd0c1e7920 0x7fcd0c1e7dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd0c0ab8d0 0x7fcd0c1ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 --2- 172.21.15.78:0/2518335193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf8004d00 0x7fcd0c1ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:22.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 >> 172.21.15.78:0/2518335193 conn(0x7fcd0c0cac90 msgr2=0x7fcd0c0c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:22.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 shutdown_connections 2024-02-19T14:52:22.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:22.908+0000 7fcd1131e640 1 -- 172.21.15.78:0/2518335193 wait complete. 2024-02-19T14:52:22.922 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:22.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.023+0000 7fcf86207640 1 Processor -- start 2024-02-19T14:52:23.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.023+0000 7fcf86207640 1 -- start start 2024-02-19T14:52:23.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf86207640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcf8014f1a0 0x7fcf80072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf86207640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006ea50 0x7fcf80152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf86207640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf8006fbc0 0x7fcf800c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf86207640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800c5970 con 0x7fcf8014f1a0 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf86207640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800c5750 con 0x7fcf8006fbc0 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf86207640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c5bc0 con 0x7fcf8006ea50 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcf8014f1a0 0x7fcf80072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcf8014f1a0 0x7fcf80072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52736/0 (socket says 172.21.15.78:52736) 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/1693733765 learned_addr learned my addr 172.21.15.78:0/1693733765 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7ffff640 1 --2- 172.21.15.78:0/1693733765 >> v2:172.21.15.150:3300/0 conn(0x7fcf8006fbc0 0x7fcf800c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7effd640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006ea50 0x7fcf80152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/1693733765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006ea50 msgr2=0x7fcf80152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006ea50 0x7fcf80152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/1693733765 >> v2:172.21.15.150:3300/0 conn(0x7fcf8006fbc0 msgr2=0x7fcf800c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 --2- 172.21.15.78:0/1693733765 >> v2:172.21.15.150:3300/0 conn(0x7fcf8006fbc0 0x7fcf800c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/1693733765 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800c5ea0 con 0x7fcf8014f1a0 2024-02-19T14:52:23.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7effd640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006ea50 0x7fcf80152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:23.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.024+0000 7fcf7ffff640 1 --2- 172.21.15.78:0/1693733765 >> v2:172.21.15.150:3300/0 conn(0x7fcf8006fbc0 0x7fcf800c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:23.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf7f7fe640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8014f1a0 0x7fcf80072d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf80071f60 tx=0x7fcf7002dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a6f02aa3287be1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf7e7fc640 1 -- 172.21.15.78:0/1693733765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7002ed70 con 0x7fcf8014f1a0 2024-02-19T14:52:23.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf7e7fc640 1 -- 172.21.15.78:0/1693733765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70036440 con 0x7fcf8014f1a0 2024-02-19T14:52:23.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf7e7fc640 1 -- 172.21.15.78:0/1693733765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf700367d0 con 0x7fcf8014f1a0 2024-02-19T14:52:23.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 -- 172.21.15.78:0/1693733765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8014f1a0 msgr2=0x7fcf80072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8014f1a0 0x7fcf80072d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf80071f60 tx=0x7fcf7002dfc0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 -- 172.21.15.78:0/1693733765 shutdown_connections 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 --2- 172.21.15.78:0/1693733765 >> v2:172.21.15.150:3300/0 conn(0x7fcf8006fbc0 0x7fcf800c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006ea50 0x7fcf80152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 --2- 172.21.15.78:0/1693733765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8014f1a0 0x7fcf80072d70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.025+0000 7fcf86207640 1 -- 172.21.15.78:0/1693733765 >> 172.21.15.78:0/1693733765 conn(0x7fcf8006c170 msgr2=0x7fcf8006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.026+0000 7fcf86207640 1 -- 172.21.15.78:0/1693733765 shutdown_connections 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.026+0000 7fcf86207640 1 -- 172.21.15.78:0/1693733765 wait complete. 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.027+0000 7fcf86207640 1 Processor -- start 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.027+0000 7fcf86207640 1 -- start start 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf86207640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 0x7fcf801e6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf86207640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006fbc0 0x7fcf800aada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf86207640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf8014f1a0 0x7fcf800ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf86207640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c3900 con 0x7fcf8006ea50 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf86207640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf801e6c60 con 0x7fcf8014f1a0 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf86207640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c4780 con 0x7fcf8006fbc0 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 0x7fcf801e6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 0x7fcf801e6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52742/0 (socket says 172.21.15.78:52742) 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/3638178347 learned_addr learned my addr 172.21.15.78:0/3638178347 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7ffff640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf8014f1a0 0x7fcf800ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7effd640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006fbc0 0x7fcf800aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006fbc0 msgr2=0x7fcf800aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006fbc0 0x7fcf800aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf8014f1a0 msgr2=0x7fcf800ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf8014f1a0 0x7fcf800ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 -- 172.21.15.78:0/3638178347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7000b770 con 0x7fcf8006ea50 2024-02-19T14:52:23.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.028+0000 7fcf7f7fe640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 0x7fcf801e6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002e000 tx=0x7fcf700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.029+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70041850 con 0x7fcf8006ea50 2024-02-19T14:52:23.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.029+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf700419b0 con 0x7fcf8006ea50 2024-02-19T14:52:23.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.029+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70041d60 con 0x7fcf8006ea50 2024-02-19T14:52:23.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.029+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf800c4a60 con 0x7fcf8006ea50 2024-02-19T14:52:23.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.029+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf800c2020 con 0x7fcf8006ea50 2024-02-19T14:52:23.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.030+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf800c48c0 con 0x7fcf8006ea50 2024-02-19T14:52:23.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.030+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcf7003d040 con 0x7fcf8006ea50 2024-02-19T14:52:23.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.030+0000 7fcf67fff640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf6c06b810 0x7fcf6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.031+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcf700aef70 con 0x7fcf8006ea50 2024-02-19T14:52:23.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.031+0000 7fcf7effd640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf6c06b810 0x7fcf6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.031+0000 7fcf7effd640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf6c06b810 0x7fcf6c06dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf800a84b0 tx=0x7fcf74008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.035+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf700e2dc0 con 0x7fcf8006ea50 2024-02-19T14:52:23.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.265+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf80074990 con 0x7fcf8006ea50 2024-02-19T14:52:23.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.265+0000 7fcf67fff640 1 -- 172.21.15.78:0/3638178347 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf800c48c0 con 0x7fcf8006ea50 2024-02-19T14:52:23.268 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf6c06b810 msgr2=0x7fcf6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf6c06b810 0x7fcf6c06dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf800a84b0 tx=0x7fcf74008040 comp rx=0 tx=0).stop 2024-02-19T14:52:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 msgr2=0x7fcf801e6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 0x7fcf801e6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcf7002e000 tx=0x7fcf700046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 shutdown_connections 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf6c06b810 0x7fcf6c06dcd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf8014f1a0 0x7fcf800ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf8006fbc0 0x7fcf800aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 --2- 172.21.15.78:0/3638178347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf8006ea50 0x7fcf801e6190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.267+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 >> 172.21.15.78:0/3638178347 conn(0x7fcf8006c170 msgr2=0x7fcf8014b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.268+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 shutdown_connections 2024-02-19T14:52:23.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.268+0000 7fcf86207640 1 -- 172.21.15.78:0/3638178347 wait complete. 2024-02-19T14:52:23.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:23.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.391+0000 7f0bc44f4640 1 Processor -- start 2024-02-19T14:52:23.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.391+0000 7f0bc44f4640 1 -- start start 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc44f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0bbc152830 0x7f0bbc0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc44f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bbc0c7990 0x7f0bbc06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc44f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc1586a0 con 0x7f0bbc0c5760 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc44f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c6550 con 0x7f0bbc0c7990 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc44f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc159320 con 0x7f0bbc152830 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bbc0c7990 0x7f0bbc06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bbc0c7990 0x7f0bbc06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52750/0 (socket says 172.21.15.78:52750) 2024-02-19T14:52:23.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 -- 172.21.15.78:0/3204945810 learned_addr learned my addr 172.21.15.78:0/3204945810 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:23.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2290640 1 --2- 172.21.15.78:0/3204945810 >> v2:172.21.15.150:3300/0 conn(0x7f0bbc152830 0x7f0bbc0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 -- 172.21.15.78:0/3204945810 >> v2:172.21.15.150:3300/0 conn(0x7f0bbc152830 msgr2=0x7f0bbc0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:23.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 --2- 172.21.15.78:0/3204945810 >> v2:172.21.15.150:3300/0 conn(0x7f0bbc152830 0x7f0bbc0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 -- 172.21.15.78:0/3204945810 >> v1:172.21.15.150:6789/0 conn(0x7f0bbc0c5760 legacy=0x7f0bbc0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:23.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.392+0000 7f0bc2a91640 1 -- 172.21.15.78:0/3204945810 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc158980 con 0x7f0bbc0c7990 2024-02-19T14:52:23.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.393+0000 7f0bc2a91640 1 --2- 172.21.15.78:0/3204945810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c7990 0x7f0bbc06c570 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0c7250 tx=0x7f0bb802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5825361b0572d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc128e640 1 -- 172.21.15.78:0/3204945810 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb802c9f0 con 0x7f0bbc0c7990 2024-02-19T14:52:23.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc128e640 1 -- 172.21.15.78:0/3204945810 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb802cb50 con 0x7f0bbc0c7990 2024-02-19T14:52:23.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc128e640 1 -- 172.21.15.78:0/3204945810 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80343f0 con 0x7f0bbc0c7990 2024-02-19T14:52:23.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 -- 172.21.15.78:0/3204945810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c7990 msgr2=0x7f0bbc06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/3204945810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c7990 0x7f0bbc06c570 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0c7250 tx=0x7f0bb802be50 comp rx=0 tx=0).stop 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 -- 172.21.15.78:0/3204945810 shutdown_connections 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/3204945810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c7990 0x7f0bbc06c570 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/3204945810 >> v2:172.21.15.150:3300/0 conn(0x7f0bbc152830 0x7f0bbc0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 -- 172.21.15.78:0/3204945810 >> 172.21.15.78:0/3204945810 conn(0x7f0bbc0ca8a0 msgr2=0x7f0bbc0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 -- 172.21.15.78:0/3204945810 shutdown_connections 2024-02-19T14:52:23.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.394+0000 7f0bc44f4640 1 -- 172.21.15.78:0/3204945810 wait complete. 2024-02-19T14:52:23.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.395+0000 7f0bc44f4640 1 Processor -- start 2024-02-19T14:52:23.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.395+0000 7f0bc44f4640 1 -- start start 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc44f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 0x7f0bbc14aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc44f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc152830 0x7f0bbc0be3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc44f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0acf70 0x7f0bbc0be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc44f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0bfd60 con 0x7f0bbc0c5760 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc44f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c0c30 con 0x7f0bbc0acf70 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc44f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c0ed0 con 0x7f0bbc152830 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 0x7f0bbc14aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 0x7f0bbc14aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52752/0 (socket says 172.21.15.78:52752) 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 -- 172.21.15.78:0/1823299125 learned_addr learned my addr 172.21.15.78:0/1823299125 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2a91640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0acf70 0x7f0bbc0be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc1a8f640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc152830 0x7f0bbc0be3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 -- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc152830 msgr2=0x7f0bbc0be3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc152830 0x7f0bbc0be3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 -- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0acf70 msgr2=0x7f0bbc0be8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0acf70 0x7f0bbc0be8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc2290640 1 -- 172.21.15.78:0/1823299125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8009b30 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.396+0000 7f0bc1a8f640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc152830 0x7f0bbc0be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.397+0000 7f0bc2290640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 0x7f0bbc14aed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400d5f0 tx=0x7f0bb400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.397+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40125f0 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.397+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4012750 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.397+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc0c11b0 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.397+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4012b00 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.397+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc14b410 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.398+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc068950 con 0x7f0bbc0c5760 2024-02-19T14:52:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.398+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0bb4012c60 con 0x7f0bbc0c5760 2024-02-19T14:52:23.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.399+0000 7f0bb2ffd640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b810 0x7f0ba006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.399+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0bb408a8c0 con 0x7f0bbc0c5760 2024-02-19T14:52:23.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.399+0000 7f0bc1a8f640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b810 0x7f0ba006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.399+0000 7f0bc1a8f640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b810 0x7f0ba006dcd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0cb2b0 tx=0x7f0bac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.403+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0bb4055030 con 0x7f0bbc0c5760 2024-02-19T14:52:23.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.579+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bbc0bffd0 con 0x7f0bbc0c5760 2024-02-19T14:52:23.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.580+0000 7f0bb2ffd640 1 -- 172.21.15.78:0/1823299125 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bb40586f0 con 0x7f0bbc0c5760 2024-02-19T14:52:23.583 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:23.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b810 msgr2=0x7f0ba006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b810 0x7f0ba006dcd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0cb2b0 tx=0x7f0bac009040 comp rx=0 tx=0).stop 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 msgr2=0x7f0bbc14aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 0x7f0bbc14aed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400d5f0 tx=0x7f0bb400dab0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 shutdown_connections 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b810 0x7f0ba006dcd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0acf70 0x7f0bbc0be8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc152830 0x7f0bbc0be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 --2- 172.21.15.78:0/1823299125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c5760 0x7f0bbc14aed0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 >> 172.21.15.78:0/1823299125 conn(0x7f0bbc0ca8a0 msgr2=0x7f0bbc0c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 shutdown_connections 2024-02-19T14:52:23.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.582+0000 7f0bc44f4640 1 -- 172.21.15.78:0/1823299125 wait complete. 2024-02-19T14:52:23.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:23.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.697+0000 7f7d6c16c640 1 Processor -- start 2024-02-19T14:52:23.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.697+0000 7f7d6c16c640 1 -- start start 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d6c16c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7d64156550 0x7f7d640ad700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d6c16c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d64157040 0x7f7d640c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d6c16c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d64071050 con 0x7f7d6406f170 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d6c16c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d640c6ea0 con 0x7f7d64156550 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d6c16c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640c7120 con 0x7f7d64157040 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d69f08640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7d64156550 0x7f7d640ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d69f08640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7d64156550 0x7f7d640ad700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52756/0 (socket says 172.21.15.78:52756) 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d69f08640 1 -- 172.21.15.78:0/496776186 learned_addr learned my addr 172.21.15.78:0/496776186 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d6a709640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d64157040 0x7f7d640c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d69f08640 1 -- 172.21.15.78:0/496776186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d64157040 msgr2=0x7f7d640c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d69f08640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d64157040 0x7f7d640c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.698+0000 7f7d69f08640 1 -- 172.21.15.78:0/496776186 >> v1:172.21.15.150:6789/0 conn(0x7f7d6406f170 legacy=0x7f7d64156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.699+0000 7f7d69f08640 1 -- 172.21.15.78:0/496776186 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640c76c0 con 0x7f7d64156550 2024-02-19T14:52:23.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.699+0000 7f7d6a709640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d64157040 0x7f7d640c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:23.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.699+0000 7f7d69f08640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d64156550 0x7f7d640ad700 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009870 tx=0x7f7d5402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6873ec7f853e48 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.699+0000 7f7d68f06640 1 -- 172.21.15.78:0/496776186 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031640 con 0x7f7d64156550 2024-02-19T14:52:23.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.699+0000 7f7d68f06640 1 -- 172.21.15.78:0/496776186 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d540317a0 con 0x7f7d64156550 2024-02-19T14:52:23.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.699+0000 7f7d68f06640 1 -- 172.21.15.78:0/496776186 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031b30 con 0x7f7d64156550 2024-02-19T14:52:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 -- 172.21.15.78:0/496776186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d64156550 msgr2=0x7f7d640ad700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d64156550 0x7f7d640ad700 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009870 tx=0x7f7d5402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 -- 172.21.15.78:0/496776186 shutdown_connections 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d64157040 0x7f7d640c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/496776186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d64156550 0x7f7d640ad700 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 -- 172.21.15.78:0/496776186 >> 172.21.15.78:0/496776186 conn(0x7f7d640aa3e0 msgr2=0x7f7d640aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 -- 172.21.15.78:0/496776186 shutdown_connections 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.700+0000 7f7d6c16c640 1 -- 172.21.15.78:0/496776186 wait complete. 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.701+0000 7f7d6c16c640 1 Processor -- start 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.701+0000 7f7d6c16c640 1 -- start start 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 0x7f7d6414c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d64156550 0x7f7d640c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d641488a0 0x7f7d64148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6414aa80 con 0x7f7d6406f170 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d64152370 con 0x7f7d64156550 2024-02-19T14:52:23.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640ad7e0 con 0x7f7d641488a0 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 0x7f7d6414c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 0x7f7d6414c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52762/0 (socket says 172.21.15.78:52762) 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 -- 172.21.15.78:0/3371848067 learned_addr learned my addr 172.21.15.78:0/3371848067 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69707640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d64156550 0x7f7d640c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 -- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d641488a0 msgr2=0x7f7d64148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d641488a0 0x7f7d64148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 -- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d64156550 msgr2=0x7f7d640c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d64156550 0x7f7d640c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 -- 172.21.15.78:0/3371848067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d540094d0 con 0x7f7d6406f170 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69707640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d64156550 0x7f7d640c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d69f08640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 0x7f7d6414c5b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7d54000c00 tx=0x7f7d540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5403ac60 con 0x7f7d6406f170 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5403ade0 con 0x7f7d6406f170 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6414a210 con 0x7f7d6406f170 2024-02-19T14:52:23.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.702+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54044630 con 0x7f7d6406f170 2024-02-19T14:52:23.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.703+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d64151d20 con 0x7f7d6406f170 2024-02-19T14:52:23.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.703+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d64074990 con 0x7f7d6406f170 2024-02-19T14:52:23.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.704+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7d5402f050 con 0x7f7d6406f170 2024-02-19T14:52:23.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.704+0000 7f7d5a7fc640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d4806b810 0x7f7d4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:23.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.704+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7d540ac670 con 0x7f7d6406f170 2024-02-19T14:52:23.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.705+0000 7f7d69707640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d4806b810 0x7f7d4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:23.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.705+0000 7f7d69707640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d4806b810 0x7f7d4806dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d6414f510 tx=0x7f7d60009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:23.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.713+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7d54076d50 con 0x7f7d6406f170 2024-02-19T14:52:23.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.919+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d640c94f0 con 0x7f7d6406f170 2024-02-19T14:52:23.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.920+0000 7f7d5a7fc640 1 -- 172.21.15.78:0/3371848067 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d5404d450 con 0x7f7d6406f170 2024-02-19T14:52:23.923 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:23.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d4806b810 msgr2=0x7f7d4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d4806b810 0x7f7d4806dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d6414f510 tx=0x7f7d60009040 comp rx=0 tx=0).stop 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 msgr2=0x7f7d6414c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 0x7f7d6414c5b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7d54000c00 tx=0x7f7d540046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 shutdown_connections 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d4806b810 0x7f7d4806dcd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d641488a0 0x7f7d64148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d64156550 0x7f7d640c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 --2- 172.21.15.78:0/3371848067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d6406f170 0x7f7d6414c5b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 >> 172.21.15.78:0/3371848067 conn(0x7f7d640aa3e0 msgr2=0x7f7d64155070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 shutdown_connections 2024-02-19T14:52:23.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:23.922+0000 7f7d6c16c640 1 -- 172.21.15.78:0/3371848067 wait complete. 2024-02-19T14:52:23.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.035+0000 7fe98512c640 1 Processor -- start 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.035+0000 7fe98512c640 1 -- start start 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe98512c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe98014d210 0x7fe9800c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe98512c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98006ff90 con 0x7fe9800c3d90 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe98512c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe980156fc0 con 0x7fe9800719e0 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe98512c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe980070270 con 0x7fe98014d210 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe98014d210 0x7fe9800c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe98014d210 0x7fe9800c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52764/0 (socket says 172.21.15.78:52764) 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97f59e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe9800c3d90 0x7fe98015d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57850/0 (socket says 172.21.15.78:57850) 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97ed9d640 1 -- 172.21.15.78:0/4279812310 learned_addr learned my addr 172.21.15.78:0/4279812310 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97ed9d640 1 -- 172.21.15.78:0/4279812310 >> v1:172.21.15.150:6789/0 conn(0x7fe9800719e0 legacy=0x7fe9800c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97dd9b640 1 -- 172.21.15.78:0/4279812310 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961412118 0 0) 0x7fe98006ff90 con 0x7fe9800c3d90 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97ed9d640 1 -- 172.21.15.78:0/4279812310 >> v1:172.21.15.78:6789/0 conn(0x7fe9800c3d90 legacy=0x7fe98015d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.036+0000 7fe97ed9d640 1 -- 172.21.15.78:0/4279812310 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9801572a0 con 0x7fe98014d210 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.037+0000 7fe97ed9d640 1 --2- 172.21.15.78:0/4279812310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe98014d210 0x7fe9800c4290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe978009e50 tx=0x7fe97802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3443830e2794c4f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.037+0000 7fe97dd9b640 1 -- 172.21.15.78:0/4279812310 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97802cb20 con 0x7fe98014d210 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.037+0000 7fe97dd9b640 1 -- 172.21.15.78:0/4279812310 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97802cc80 con 0x7fe98014d210 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.037+0000 7fe97dd9b640 1 -- 172.21.15.78:0/4279812310 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9780343f0 con 0x7fe98014d210 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 -- 172.21.15.78:0/4279812310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe98014d210 msgr2=0x7fe9800c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 --2- 172.21.15.78:0/4279812310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe98014d210 0x7fe9800c4290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe978009e50 tx=0x7fe97802bf80 comp rx=0 tx=0).stop 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 -- 172.21.15.78:0/4279812310 shutdown_connections 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 --2- 172.21.15.78:0/4279812310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe98014d210 0x7fe9800c4290 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 -- 172.21.15.78:0/4279812310 >> 172.21.15.78:0/4279812310 conn(0x7fe9800c5050 msgr2=0x7fe9800c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 -- 172.21.15.78:0/4279812310 shutdown_connections 2024-02-19T14:52:24.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.038+0000 7fe98512c640 1 -- 172.21.15.78:0/4279812310 wait complete. 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.039+0000 7fe98512c640 1 Processor -- start 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.039+0000 7fe98512c640 1 -- start start 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe98512c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 0x7fe98006da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe98512c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9800c3d90 0x7fe9800c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe98512c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe98014d210 0x7fe980152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe98512c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800c3970 con 0x7fe9800719e0 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe98512c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9801ebed0 con 0x7fe98014d210 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe98512c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9801eb000 con 0x7fe9800c3d90 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe97ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 0x7fe98006da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe97ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 0x7fe98006da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52774/0 (socket says 172.21.15.78:52774) 2024-02-19T14:52:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe97ed9d640 1 -- 172.21.15.78:0/734727767 learned_addr learned my addr 172.21.15.78:0/734727767 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe97f59e640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe98014d210 0x7fe980152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe97e59c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9800c3d90 0x7fe9800c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.040+0000 7fe97ed9d640 1 -- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9800c3d90 msgr2=0x7fe9800c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe97ed9d640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9800c3d90 0x7fe9800c1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe97ed9d640 1 -- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe98014d210 msgr2=0x7fe980152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe97ed9d640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe98014d210 0x7fe980152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe97ed9d640 1 -- 172.21.15.78:0/734727767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe978009b30 con 0x7fe9800719e0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe97f59e640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe98014d210 0x7fe980152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe97ed9d640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 0x7fe98006da70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe978009e50 tx=0x7fe978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe978009510 con 0x7fe9800719e0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97803e450 con 0x7fe9800719e0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9800c3660 con 0x7fe9800719e0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97803e800 con 0x7fe9800719e0 2024-02-19T14:52:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.041+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9801eb5a0 con 0x7fe9800719e0 2024-02-19T14:52:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.042+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe980074990 con 0x7fe9800719e0 2024-02-19T14:52:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.043+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe97803e960 con 0x7fe9800719e0 2024-02-19T14:52:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.043+0000 7fe9677fe640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe96c06b840 0x7fe96c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.043+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe9780ad130 con 0x7fe9800719e0 2024-02-19T14:52:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.043+0000 7fe97e59c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe96c06b840 0x7fe96c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.044+0000 7fe97e59c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe96c06b840 0x7fe96c06dd00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe9800a7dd0 tx=0x7fe974009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.048+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe978077950 con 0x7fe9800719e0 2024-02-19T14:52:24.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.324+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe9800c3100 con 0x7fe9800719e0 2024-02-19T14:52:24.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.325+0000 7fe9677fe640 1 -- 172.21.15.78:0/734727767 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe97807b010 con 0x7fe9800719e0 2024-02-19T14:52:24.327 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe96c06b840 msgr2=0x7fe96c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe96c06b840 0x7fe96c06dd00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe9800a7dd0 tx=0x7fe974009040 comp rx=0 tx=0).stop 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 msgr2=0x7fe98006da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 0x7fe98006da70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe978009e50 tx=0x7fe978004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 shutdown_connections 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe96c06b840 0x7fe96c06dd00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe98014d210 0x7fe980152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9800c3d90 0x7fe9800c1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 --2- 172.21.15.78:0/734727767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9800719e0 0x7fe98006da70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 >> 172.21.15.78:0/734727767 conn(0x7fe9800c5050 msgr2=0x7fe9800a9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 shutdown_connections 2024-02-19T14:52:24.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.327+0000 7fe98512c640 1 -- 172.21.15.78:0/734727767 wait complete. 2024-02-19T14:52:24.344 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:24.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.443+0000 7f4f8e6a4640 1 Processor -- start 2024-02-19T14:52:24.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.443+0000 7f4f8e6a4640 1 -- start start 2024-02-19T14:52:24.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f8e6a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f8806bb00 0x7f4f880c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f8e6a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f880721a0 0x7f4f880c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f8e6a4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f88156e80 con 0x7f4f88150560 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f8e6a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8806ffc0 con 0x7f4f8806bb00 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f8e6a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f88072d70 con 0x7f4f880721a0 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f8806bb00 0x7f4f880c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f8806bb00 0x7f4f880c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52776/0 (socket says 172.21.15.78:52776) 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 -- 172.21.15.78:0/1768712235 learned_addr learned my addr 172.21.15.78:0/1768712235 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 -- 172.21.15.78:0/1768712235 >> v2:172.21.15.150:3300/0 conn(0x7f4f880721a0 msgr2=0x7f4f880c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 --2- 172.21.15.78:0/1768712235 >> v2:172.21.15.150:3300/0 conn(0x7f4f880721a0 0x7f4f880c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 -- 172.21.15.78:0/1768712235 >> v1:172.21.15.150:6789/0 conn(0x7f4f88150560 legacy=0x7f4f880c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.444+0000 7f4f877fe640 1 -- 172.21.15.78:0/1768712235 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f88073050 con 0x7f4f8806bb00 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.445+0000 7f4f877fe640 1 --2- 172.21.15.78:0/1768712235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8806bb00 0x7f4f880c6080 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4f8806b810 tx=0x7f4f7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f451a0aa64ff9a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.445+0000 7f4f86ffd640 1 -- 172.21.15.78:0/1768712235 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0317e0 con 0x7f4f8806bb00 2024-02-19T14:52:24.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.445+0000 7f4f86ffd640 1 -- 172.21.15.78:0/1768712235 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c031940 con 0x7f4f8806bb00 2024-02-19T14:52:24.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.445+0000 7f4f86ffd640 1 -- 172.21.15.78:0/1768712235 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c031cd0 con 0x7f4f8806bb00 2024-02-19T14:52:24.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/1768712235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8806bb00 msgr2=0x7f4f880c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/1768712235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8806bb00 0x7f4f880c6080 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4f8806b810 tx=0x7f4f7c02b880 comp rx=0 tx=0).stop 2024-02-19T14:52:24.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/1768712235 shutdown_connections 2024-02-19T14:52:24.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/1768712235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f8806bb00 0x7f4f880c6080 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/1768712235 >> v2:172.21.15.150:3300/0 conn(0x7f4f880721a0 0x7f4f880c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/1768712235 >> 172.21.15.78:0/1768712235 conn(0x7f4f8806c170 msgr2=0x7f4f8806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/1768712235 shutdown_connections 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.446+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/1768712235 wait complete. 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.447+0000 7f4f8e6a4640 1 Processor -- start 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.447+0000 7f4f8e6a4640 1 -- start start 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.447+0000 7f4f8e6a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8806bb00 0x7f4f880a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.447+0000 7f4f8e6a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 0x7f4f88155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8e6a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f88150560 0x7f4f88146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8e6a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88147f90 con 0x7f4f880721a0 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8e6a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88148d60 con 0x7f4f8806bb00 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8e6a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88148a80 con 0x7f4f88150560 2024-02-19T14:52:24.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 0x7f4f88155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 0x7f4f88155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52786/0 (socket says 172.21.15.78:52786) 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 -- 172.21.15.78:0/767969846 learned_addr learned my addr 172.21.15.78:0/767969846 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f87fff640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8806bb00 0x7f4f880a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 -- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f88150560 msgr2=0x7f4f88146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8cc41640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f88150560 0x7f4f88146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f88150560 0x7f4f88146730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 -- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8806bb00 msgr2=0x7f4f880a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8806bb00 0x7f4f880a9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 -- 172.21.15.78:0/767969846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0094d0 con 0x7f4f880721a0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f877fe640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 0x7f4f88155080 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c000c00 tx=0x7f4f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c032880 con 0x7f4f880721a0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0329e0 con 0x7f4f880721a0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f881e9600 con 0x7f4f880721a0 2024-02-19T14:52:24.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c032d90 con 0x7f4f880721a0 2024-02-19T14:52:24.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.448+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f881481e0 con 0x7f4f880721a0 2024-02-19T14:52:24.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.449+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f881483d0 con 0x7f4f880721a0 2024-02-19T14:52:24.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.450+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4f7c04d3f0 con 0x7f4f880721a0 2024-02-19T14:52:24.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.450+0000 7f4f84ff9640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f7406b810 0x7f4f7406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.450+0000 7f4f87fff640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f7406b810 0x7f4f7406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.450+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4f7c0ac3d0 con 0x7f4f880721a0 2024-02-19T14:52:24.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.451+0000 7f4f87fff640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f7406b810 0x7f4f7406dcd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f78007fd0 tx=0x7f4f78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.455+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4f7c076b40 con 0x7f4f880721a0 2024-02-19T14:52:24.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.662+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f88074990 con 0x7f4f880721a0 2024-02-19T14:52:24.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.663+0000 7f4f84ff9640 1 -- 172.21.15.78:0/767969846 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7f4f7c07a200 con 0x7f4f880721a0 2024-02-19T14:52:24.665 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:24.666 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:24.666 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:24.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f7406b810 msgr2=0x7f4f7406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f7406b810 0x7f4f7406dcd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f78007fd0 tx=0x7f4f78009040 comp rx=0 tx=0).stop 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 msgr2=0x7f4f88155080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 0x7f4f88155080 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c000c00 tx=0x7f4f7c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 shutdown_connections 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f7406b810 0x7f4f7406dcd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f880721a0 0x7f4f88155080 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f88150560 0x7f4f88146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 --2- 172.21.15.78:0/767969846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8806bb00 0x7f4f880a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 >> 172.21.15.78:0/767969846 conn(0x7f4f8806c170 msgr2=0x7f4f8806c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 shutdown_connections 2024-02-19T14:52:24.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.665+0000 7f4f8e6a4640 1 -- 172.21.15.78:0/767969846 wait complete. 2024-02-19T14:52:24.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:24.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.778+0000 7f80ee44e640 1 Processor -- start 2024-02-19T14:52:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.778+0000 7f80ee44e640 1 -- start start 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ee44e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80e8152700 0x7f80e80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ee44e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c5730 0x7f80e806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ee44e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80e80c8e00 0x7f80e806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ee44e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e806cd20 con 0x7f80e80c8e00 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ee44e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e81551f0 con 0x7f80e8152700 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ee44e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8155490 con 0x7f80e80c5730 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80e80c8e00 0x7f80e806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80e80c8e00 0x7f80e806c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52788/0 (socket says 172.21.15.78:52788) 2024-02-19T14:52:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 -- 172.21.15.78:0/3169144959 learned_addr learned my addr 172.21.15.78:0/3169144959 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80e7fff640 1 --2- 172.21.15.78:0/3169144959 >> v2:172.21.15.150:3300/0 conn(0x7f80e8152700 0x7f80e80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80e77fe640 1 --2- 172.21.15.78:0/3169144959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c5730 0x7f80e806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 -- 172.21.15.78:0/3169144959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c5730 msgr2=0x7f80e806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 --2- 172.21.15.78:0/3169144959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c5730 0x7f80e806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 -- 172.21.15.78:0/3169144959 >> v2:172.21.15.150:3300/0 conn(0x7f80e8152700 msgr2=0x7f80e80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 --2- 172.21.15.78:0/3169144959 >> v2:172.21.15.150:3300/0 conn(0x7f80e8152700 0x7f80e80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.779+0000 7f80ec9eb640 1 -- 172.21.15.78:0/3169144959 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e8155770 con 0x7f80e80c8e00 2024-02-19T14:52:24.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.780+0000 7f80ec9eb640 1 --2- 172.21.15.78:0/3169144959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c8e00 0x7f80e806c7e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f80e808bad0 tx=0x7f80e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bd0fce51bd962b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.780+0000 7f80e6ffd640 1 -- 172.21.15.78:0/3169144959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e002ca70 con 0x7f80e80c8e00 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.780+0000 7f80e6ffd640 1 -- 172.21.15.78:0/3169144959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e002cbd0 con 0x7f80e80c8e00 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.780+0000 7f80e6ffd640 1 -- 172.21.15.78:0/3169144959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00343f0 con 0x7f80e80c8e00 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 -- 172.21.15.78:0/3169144959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c8e00 msgr2=0x7f80e806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 --2- 172.21.15.78:0/3169144959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c8e00 0x7f80e806c7e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f80e808bad0 tx=0x7f80e002bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 -- 172.21.15.78:0/3169144959 shutdown_connections 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 --2- 172.21.15.78:0/3169144959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c8e00 0x7f80e806c7e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 --2- 172.21.15.78:0/3169144959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c5730 0x7f80e806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 --2- 172.21.15.78:0/3169144959 >> v2:172.21.15.150:3300/0 conn(0x7f80e8152700 0x7f80e80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 -- 172.21.15.78:0/3169144959 >> 172.21.15.78:0/3169144959 conn(0x7f80e80cac90 msgr2=0x7f80e80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 -- 172.21.15.78:0/3169144959 shutdown_connections 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.781+0000 7f80ee44e640 1 -- 172.21.15.78:0/3169144959 wait complete. 2024-02-19T14:52:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.782+0000 7f80ee44e640 1 Processor -- start 2024-02-19T14:52:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.782+0000 7f80ee44e640 1 -- start start 2024-02-19T14:52:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 0x7f80e80c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c8e00 0x7f80e81ed850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80e8152700 0x7f80e81edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81ee870 con 0x7f80e80c5730 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81496b0 con 0x7f80e8152700 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80abdd0 con 0x7f80e80c8e00 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 0x7f80e80c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 0x7f80e80c0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52804/0 (socket says 172.21.15.78:52804) 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 -- 172.21.15.78:0/292164949 learned_addr learned my addr 172.21.15.78:0/292164949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ec9eb640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80e8152700 0x7f80e81edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e77fe640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c8e00 0x7f80e81ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 -- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c8e00 msgr2=0x7f80e81ed850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c8e00 0x7f80e81ed850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 -- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80e8152700 msgr2=0x7f80e81edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80e8152700 0x7f80e81edd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 -- 172.21.15.78:0/292164949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0009b30 con 0x7f80e80c5730 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e77fe640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c8e00 0x7f80e81ed850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e7fff640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 0x7f80e80c0f70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00e870 tx=0x7f80dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0094f0 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc00c8e0 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00cc90 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e8157040 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.783+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e81ee2d0 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.784+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e8156e80 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.785+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f80dc013470 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.785+0000 7f80e4ff9640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80d406b810 0x7f80d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.785+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f80dc08a3b0 con 0x7f80e80c5730 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.785+0000 7f80e77fe640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80d406b810 0x7f80d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:24.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.786+0000 7f80e77fe640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80d406b810 0x7f80d406dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f80e806dbc0 tx=0x7f80d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:24.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.789+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f80dc054bd0 con 0x7f80e80c5730 2024-02-19T14:52:24.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.968+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80e8074990 con 0x7f80e80c5730 2024-02-19T14:52:24.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.969+0000 7f80e4ff9640 1 -- 172.21.15.78:0/292164949 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f80dc058290 con 0x7f80e80c5730 2024-02-19T14:52:24.972 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:24.972 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80d406b810 msgr2=0x7f80d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80d406b810 0x7f80d406dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f80e806dbc0 tx=0x7f80d8008040 comp rx=0 tx=0).stop 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 msgr2=0x7f80e80c0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 0x7f80e80c0f70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00e870 tx=0x7f80dc00ed30 comp rx=0 tx=0).stop 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 shutdown_connections 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80d406b810 0x7f80d406dcd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80e8152700 0x7f80e81edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80e80c8e00 0x7f80e81ed850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 --2- 172.21.15.78:0/292164949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80e80c5730 0x7f80e80c0f70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 >> 172.21.15.78:0/292164949 conn(0x7f80e80cac90 msgr2=0x7f80e814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 shutdown_connections 2024-02-19T14:52:24.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:24.971+0000 7f80ee44e640 1 -- 172.21.15.78:0/292164949 wait complete. 2024-02-19T14:52:24.985 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:24.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:25.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.085+0000 7f504f0ae640 1 Processor -- start 2024-02-19T14:52:25.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.085+0000 7f504f0ae640 1 -- start start 2024-02-19T14:52:25.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504f0ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50480c57a0 0x7f504806f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504f0ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150590 0x7f5048156e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504f0ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50481509b0 con 0x7f5048150100 2024-02-19T14:52:25.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504f0ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5048157370 con 0x7f50480c57a0 2024-02-19T14:52:25.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504f0ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5048152d70 con 0x7f5048150590 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504ce4a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50480c57a0 0x7f504806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f504d64b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150590 0x7f5048156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f5047fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5048150100 0x7f5048156720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50456/0 (socket says 172.21.15.78:50456) 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.086+0000 7f5047fff640 1 -- 172.21.15.78:0/2859171428 learned_addr learned my addr 172.21.15.78:0/2859171428 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.087+0000 7f504d64b640 1 -- 172.21.15.78:0/2859171428 >> v2:172.21.15.150:3300/0 conn(0x7f50480c57a0 msgr2=0x7f504806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.087+0000 7f504d64b640 1 --2- 172.21.15.78:0/2859171428 >> v2:172.21.15.150:3300/0 conn(0x7f50480c57a0 0x7f504806f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.087+0000 7f504d64b640 1 -- 172.21.15.78:0/2859171428 >> v1:172.21.15.150:6789/0 conn(0x7f5048150100 legacy=0x7f5048156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.087+0000 7f504d64b640 1 -- 172.21.15.78:0/2859171428 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50480725a0 con 0x7f5048150590 2024-02-19T14:52:25.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.087+0000 7f504ce4a640 1 --2- 172.21.15.78:0/2859171428 >> v2:172.21.15.150:3300/0 conn(0x7f50480c57a0 0x7f504806f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:25.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.087+0000 7f504d64b640 1 --2- 172.21.15.78:0/2859171428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150590 0x7f5048156e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f50380097a0 tx=0x7f503802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a57548a358151f06 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f50477fe640 1 -- 172.21.15.78:0/2859171428 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503802e070 con 0x7f5048150590 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f50477fe640 1 -- 172.21.15.78:0/2859171428 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50380315a0 con 0x7f5048150590 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f50477fe640 1 -- 172.21.15.78:0/2859171428 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038031930 con 0x7f5048150590 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 -- 172.21.15.78:0/2859171428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150590 msgr2=0x7f5048156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 --2- 172.21.15.78:0/2859171428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150590 0x7f5048156e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f50380097a0 tx=0x7f503802b840 comp rx=0 tx=0).stop 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 -- 172.21.15.78:0/2859171428 shutdown_connections 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 --2- 172.21.15.78:0/2859171428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150590 0x7f5048156e30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 --2- 172.21.15.78:0/2859171428 >> v2:172.21.15.150:3300/0 conn(0x7f50480c57a0 0x7f504806f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 -- 172.21.15.78:0/2859171428 >> 172.21.15.78:0/2859171428 conn(0x7f504806c3b0 msgr2=0x7f504806c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 -- 172.21.15.78:0/2859171428 shutdown_connections 2024-02-19T14:52:25.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.088+0000 7f504f0ae640 1 -- 172.21.15.78:0/2859171428 wait complete. 2024-02-19T14:52:25.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.090+0000 7f504f0ae640 1 Processor -- start 2024-02-19T14:52:25.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.090+0000 7f504f0ae640 1 -- start start 2024-02-19T14:52:25.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.091+0000 7f504f0ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50480c57a0 0x7f50480c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.091+0000 7f504f0ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150100 0x7f5048150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.091+0000 7f504f0ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 0x7f50481e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.091+0000 7f504f0ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50481520b0 con 0x7f5048150590 2024-02-19T14:52:25.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.091+0000 7f504f0ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480c2a30 con 0x7f50480c57a0 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.091+0000 7f504f0ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5048152290 con 0x7f5048150100 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 0x7f50481e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504ce4a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50480c57a0 0x7f50480c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 0x7f50481e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52820/0 (socket says 172.21.15.78:52820) 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 -- 172.21.15.78:0/1665450862 learned_addr learned my addr 172.21.15.78:0/1665450862 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f5047fff640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150100 0x7f5048150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 -- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150100 msgr2=0x7f5048150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150100 0x7f5048150e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 -- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50480c57a0 msgr2=0x7f50480c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50480c57a0 0x7f50480c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 -- 172.21.15.78:0/1665450862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040009a70 con 0x7f5048150590 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504ce4a640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50480c57a0 0x7f50480c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504d64b640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 0x7f50481e8960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f50380097a0 tx=0x7f50380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038032840 con 0x7f5048150590 2024-02-19T14:52:25.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50380329a0 con 0x7f5048150590 2024-02-19T14:52:25.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5038009400 con 0x7f5048150590 2024-02-19T14:52:25.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038032d50 con 0x7f5048150590 2024-02-19T14:52:25.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.092+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5048152850 con 0x7f5048150590 2024-02-19T14:52:25.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.094+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5048074990 con 0x7f5048150590 2024-02-19T14:52:25.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.094+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f50380453f0 con 0x7f5048150590 2024-02-19T14:52:25.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.094+0000 7f50457fa640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503406b810 0x7f503406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.095+0000 7f504ce4a640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503406b810 0x7f503406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.095+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f50380ac480 con 0x7f5048150590 2024-02-19T14:52:25.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.095+0000 7f504ce4a640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503406b810 0x7f503406dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f504814ec60 tx=0x7f504000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.102+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5038076dc0 con 0x7f5048150590 2024-02-19T14:52:25.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.279+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50480ace60 con 0x7f5048150590 2024-02-19T14:52:25.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.279+0000 7f50457fa640 1 -- 172.21.15.78:0/1665450862 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f503807a480 con 0x7f5048150590 2024-02-19T14:52:25.282 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:25.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.281+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503406b810 msgr2=0x7f503406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.281+0000 7f504f0ae640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503406b810 0x7f503406dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f504814ec60 tx=0x7f504000b040 comp rx=0 tx=0).stop 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.281+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 msgr2=0x7f50481e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.281+0000 7f504f0ae640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 0x7f50481e8960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f50380097a0 tx=0x7f50380046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 shutdown_connections 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503406b810 0x7f503406dcd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5048150590 0x7f50481e8960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5048150100 0x7f5048150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 --2- 172.21.15.78:0/1665450862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50480c57a0 0x7f50480c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 >> 172.21.15.78:0/1665450862 conn(0x7f504806c3b0 msgr2=0x7f50480c44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 shutdown_connections 2024-02-19T14:52:25.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.282+0000 7f504f0ae640 1 -- 172.21.15.78:0/1665450862 wait complete. 2024-02-19T14:52:25.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:25.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:52:25.341 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:52:25.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.398+0000 7f8257bae640 1 Processor -- start 2024-02-19T14:52:25.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.398+0000 7f8257bae640 1 -- start start 2024-02-19T14:52:25.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f8257bae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f825014f1a0 0x7f8250072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f8257bae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 0x7f8250152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f8257bae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500c5970 con 0x7f825006fbc0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f8257bae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82500c5750 con 0x7f825014f1a0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f8257bae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82500c5bc0 con 0x7f825006ea50 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f825614b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f825006fbc0 0x7f82500c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57858/0 (socket says 172.21.15.78:57858) 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.399+0000 7f825614b640 1 -- 172.21.15.78:0/233209636 learned_addr learned my addr 172.21.15.78:0/233209636 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825594a640 1 --2- 172.21.15.78:0/233209636 >> v2:172.21.15.150:3300/0 conn(0x7f825014f1a0 0x7f8250072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124601297 0 0) 0x7f82500c5970 con 0x7f825006fbc0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8255149640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 0x7f8250152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003520 con 0x7f825006fbc0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179164488 0 0) 0x7f823c003520 con 0x7f825006fbc0 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 msgr2=0x7f8250152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 0x7f8250152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 >> v2:172.21.15.150:3300/0 conn(0x7f825014f1a0 msgr2=0x7f8250072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 --2- 172.21.15.78:0/233209636 >> v2:172.21.15.150:3300/0 conn(0x7f825014f1a0 0x7f8250072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500c5ea0 con 0x7f825006fbc0 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f82480019f0 con 0x7f825006fbc0 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 >> v1:172.21.15.78:6789/0 conn(0x7f825006fbc0 legacy=0x7f82500c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 0x7f823c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c003ef0 0x7f823c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823c004d00 0x7f823c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82500c5750 con 0x7f823c004d00 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82500c5bc0 con 0x7f823c003ef0 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c00ba60 con 0x7f825006ea50 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825614b640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823c004d00 0x7f823c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825594a640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 0x7f823c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f8255149640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c003ef0 0x7f823c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825614b640 1 -- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 msgr2=0x7f823c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825614b640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f825006ea50 0x7f823c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825614b640 1 -- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c003ef0 msgr2=0x7f823c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825614b640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c003ef0 0x7f823c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.400+0000 7f825614b640 1 -- 172.21.15.78:0/233209636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c003520 con 0x7f823c004d00 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.401+0000 7f825614b640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823c004d00 0x7f823c005180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f824800a830 tx=0x7f823c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9a3d107dd771fc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.401+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248034630 con 0x7f823c004d00 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.401+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824802f070 con 0x7f823c004d00 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.401+0000 7f8254948640 1 -- 172.21.15.78:0/233209636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82480349d0 con 0x7f823c004d00 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f8257bae640 1 -- 172.21.15.78:0/233209636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823c004d00 msgr2=0x7f823c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f8257bae640 1 --2- 172.21.15.78:0/233209636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f823c004d00 0x7f823c005180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f824800a830 tx=0x7f823c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f825614b640 1 -- 172.21.15.78:0/233209636 reap_dead start 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f8257bae640 1 -- 172.21.15.78:0/233209636 shutdown_connections 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f8257bae640 1 -- 172.21.15.78:0/233209636 >> 172.21.15.78:0/233209636 conn(0x7f825006c170 msgr2=0x7f825006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f8257bae640 1 -- 172.21.15.78:0/233209636 shutdown_connections 2024-02-19T14:52:25.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.402+0000 7f8257bae640 1 -- 172.21.15.78:0/233209636 wait complete. 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.403+0000 7f8257bae640 1 Processor -- start 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.403+0000 7f8257bae640 1 -- start start 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82500c0c30 0x7f825014a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 0x7f825014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82500c34d0 0x7f82500c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8250155580 con 0x7f82501549e0 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82501560c0 con 0x7f82500c0c30 2024-02-19T14:52:25.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8250155820 con 0x7f82500c34d0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 0x7f825014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 0x7f825014ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52850/0 (socket says 172.21.15.78:52850) 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 -- 172.21.15.78:0/3101144839 learned_addr learned my addr 172.21.15.78:0/3101144839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f825594a640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82500c0c30 0x7f825014a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f825614b640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82500c34d0 0x7f82500c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 -- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82500c34d0 msgr2=0x7f82500c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82500c34d0 0x7f82500c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 -- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82500c0c30 msgr2=0x7f825014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82500c0c30 0x7f825014a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 -- 172.21.15.78:0/3101144839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8250155b00 con 0x7f82501549e0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f825594a640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82500c0c30 0x7f825014a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8255149640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 0x7f825014ae30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82400119a0 tx=0x7f8240011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824000faf0 con 0x7f82501549e0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824000fc50 con 0x7f82501549e0 2024-02-19T14:52:25.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8250155de0 con 0x7f82501549e0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240014440 con 0x7f82501549e0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.404+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82500c1010 con 0x7f82501549e0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.406+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82500c3a70 con 0x7f82501549e0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.406+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f82400145a0 con 0x7f82501549e0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.407+0000 7f82467fc640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f823406b7b0 0x7f823406dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.407+0000 7f825594a640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f823406b7b0 0x7f823406dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.407+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f824008adc0 con 0x7f82501549e0 2024-02-19T14:52:25.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.407+0000 7f825594a640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f823406b7b0 0x7f823406dc70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f824c009c40 tx=0x7f824c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.411+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8240055530 con 0x7f82501549e0 2024-02-19T14:52:25.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.588+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8250074990 con 0x7f82501549e0 2024-02-19T14:52:25.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.588+0000 7f82467fc640 1 -- 172.21.15.78:0/3101144839 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8240058bf0 con 0x7f82501549e0 2024-02-19T14:52:25.591 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f823406b7b0 msgr2=0x7f823406dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f823406b7b0 0x7f823406dc70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f824c009c40 tx=0x7f824c008040 comp rx=0 tx=0).stop 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 msgr2=0x7f825014ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 0x7f825014ae30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82400119a0 tx=0x7f8240011e60 comp rx=0 tx=0).stop 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 shutdown_connections 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f823406b7b0 0x7f823406dc70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82501549e0 0x7f825014ae30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82500c34d0 0x7f82500c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 --2- 172.21.15.78:0/3101144839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82500c0c30 0x7f825014a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 >> 172.21.15.78:0/3101144839 conn(0x7f825006c170 msgr2=0x7f8250072b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:25.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 shutdown_connections 2024-02-19T14:52:25.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.590+0000 7f8257bae640 1 -- 172.21.15.78:0/3101144839 wait complete. 2024-02-19T14:52:25.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.747+0000 7fc47dfbd640 1 Processor -- start 2024-02-19T14:52:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.747+0000 7fc47dfbd640 1 -- start start 2024-02-19T14:52:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc47dfbd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc4780c6d60 0x7fc478154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc47dfbd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc47806e830 0x7fc4780c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc47dfbd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47814f730 con 0x7fc47806ff80 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc47dfbd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4780c1940 con 0x7fc47806e830 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc47dfbd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc478072f70 con 0x7fc4780c6d60 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc476ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc47806e830 0x7fc4780c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc476ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc47806e830 0x7fc4780c0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52856/0 (socket says 172.21.15.78:52856) 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc477fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc47806ff80 0x7fc4780c7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57862/0 (socket says 172.21.15.78:57862) 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc476ffd640 1 -- 172.21.15.78:0/3903263122 learned_addr learned my addr 172.21.15.78:0/3903263122 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc4777fe640 1 --2- 172.21.15.78:0/3903263122 >> v2:172.21.15.150:3300/0 conn(0x7fc4780c6d60 0x7fc478154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc476ffd640 1 -- 172.21.15.78:0/3903263122 >> v2:172.21.15.150:3300/0 conn(0x7fc4780c6d60 msgr2=0x7fc478154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc476ffd640 1 --2- 172.21.15.78:0/3903263122 >> v2:172.21.15.150:3300/0 conn(0x7fc4780c6d60 0x7fc478154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc476ffd640 1 -- 172.21.15.78:0/3903263122 >> v1:172.21.15.78:6789/0 conn(0x7fc47806ff80 legacy=0x7fc4780c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.749+0000 7fc476ffd640 1 -- 172.21.15.78:0/3903263122 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780c1b50 con 0x7fc47806e830 2024-02-19T14:52:25.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.748+0000 7fc4767fc640 1 -- 172.21.15.78:0/3903263122 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543606139 0 0) 0x7fc47814f730 con 0x7fc47806ff80 2024-02-19T14:52:25.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.749+0000 7fc4777fe640 1 --2- 172.21.15.78:0/3903263122 >> v2:172.21.15.150:3300/0 conn(0x7fc4780c6d60 0x7fc478154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc476ffd640 1 --2- 172.21.15.78:0/3903263122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc47806e830 0x7fc4780c0fc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc4780704a0 tx=0x7fc46002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad85915782331af server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc4767fc640 1 -- 172.21.15.78:0/3903263122 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460030860 con 0x7fc47806e830 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc4767fc640 1 -- 172.21.15.78:0/3903263122 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4600309c0 con 0x7fc47806e830 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc4767fc640 1 -- 172.21.15.78:0/3903263122 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460030d50 con 0x7fc47806e830 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 -- 172.21.15.78:0/3903263122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc47806e830 msgr2=0x7fc4780c0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/3903263122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc47806e830 0x7fc4780c0fc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc4780704a0 tx=0x7fc46002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 -- 172.21.15.78:0/3903263122 shutdown_connections 2024-02-19T14:52:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/3903263122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc47806e830 0x7fc4780c0fc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/3903263122 >> v2:172.21.15.150:3300/0 conn(0x7fc4780c6d60 0x7fc478154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 -- 172.21.15.78:0/3903263122 >> 172.21.15.78:0/3903263122 conn(0x7fc47806c170 msgr2=0x7fc47806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 -- 172.21.15.78:0/3903263122 shutdown_connections 2024-02-19T14:52:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.750+0000 7fc47dfbd640 1 -- 172.21.15.78:0/3903263122 wait complete. 2024-02-19T14:52:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.751+0000 7fc47dfbd640 1 Processor -- start 2024-02-19T14:52:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 -- start start 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc47806e830 0x7fc4781499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc47806ff80 0x7fc4781eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 0x7fc4781eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780c4a70 con 0x7fc4780c6d60 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780a9350 con 0x7fc47806ff80 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc47dfbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780c46b0 con 0x7fc47806e830 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc477fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 0x7fc4781eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc477fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 0x7fc4781eafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52860/0 (socket says 172.21.15.78:52860) 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc477fff640 1 -- 172.21.15.78:0/1286649257 learned_addr learned my addr 172.21.15.78:0/1286649257 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.752+0000 7fc476ffd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc47806ff80 0x7fc4781eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc4777fe640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc47806e830 0x7fc4781499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc477fff640 1 -- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc47806e830 msgr2=0x7fc4781499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc477fff640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc47806e830 0x7fc4781499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc477fff640 1 -- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc47806ff80 msgr2=0x7fc4781eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc477fff640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc47806ff80 0x7fc4781eaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc477fff640 1 -- 172.21.15.78:0/1286649257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600094d0 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc477fff640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 0x7fc4781eafc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc47000c7d0 tx=0x7fc47000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000a640 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47000a7a0 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000ab90 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4780a7d90 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.753+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc478149f30 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.754+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47806efd0 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.755+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc47000acf0 con 0x7fc4780c6d60 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.755+0000 7fc46ffff640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc45806b810 0x7fc45806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:25.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.755+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc470089c30 con 0x7fc4780c6d60 2024-02-19T14:52:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.755+0000 7fc4777fe640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc45806b810 0x7fc45806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:25.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.755+0000 7fc4777fe640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc45806b810 0x7fc45806dcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc47814ee50 tx=0x7fc468008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:25.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.759+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc470054450 con 0x7fc4780c6d60 2024-02-19T14:52:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.932+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc478074990 con 0x7fc4780c6d60 2024-02-19T14:52:25.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.933+0000 7fc46ffff640 1 -- 172.21.15.78:0/1286649257 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc470018020 con 0x7fc4780c6d60 2024-02-19T14:52:25.935 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:25.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc45806b810 msgr2=0x7fc45806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc45806b810 0x7fc45806dcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc47814ee50 tx=0x7fc468008040 comp rx=0 tx=0).stop 2024-02-19T14:52:25.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 msgr2=0x7fc4781eafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:25.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 0x7fc4781eafc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc47000c7d0 tx=0x7fc47000cc90 comp rx=0 tx=0).stop 2024-02-19T14:52:25.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 shutdown_connections 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc45806b810 0x7fc45806dcd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4780c6d60 0x7fc4781eafc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc47806ff80 0x7fc4781eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 --2- 172.21.15.78:0/1286649257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc47806e830 0x7fc4781499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.934+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 >> 172.21.15.78:0/1286649257 conn(0x7fc47806c170 msgr2=0x7fc4780723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.935+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 shutdown_connections 2024-02-19T14:52:25.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:25.935+0000 7fc47dfbd640 1 -- 172.21.15.78:0/1286649257 wait complete. 2024-02-19T14:52:25.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:26.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.066+0000 7f1f64f13640 1 Processor -- start 2024-02-19T14:52:26.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.066+0000 7f1f64f13640 1 -- start start 2024-02-19T14:52:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f64f13640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f6006e830 0x7f1f60154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f64f13640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6014f730 con 0x7f1f600c6d60 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f64f13640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f600c16d0 con 0x7f1f6006ff80 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f64f13640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f600c1940 con 0x7f1f6006e830 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f5e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1f600c6d60 0x7f1f600c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57878/0 (socket says 172.21.15.78:57878) 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f5e59c640 1 -- 172.21.15.78:0/244993440 learned_addr learned my addr 172.21.15.78:0/244993440 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383221340 0 0) 0x7f1f6014f730 con 0x7f1f600c6d60 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f4c003520 con 0x7f1f600c6d60 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.067+0000 7f1f55d9b640 1 --2- 172.21.15.78:0/244993440 >> v2:172.21.15.150:3300/0 conn(0x7f1f6006e830 0x7f1f60154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844954199 0 0) 0x7f1f4c003520 con 0x7f1f600c6d60 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 >> v2:172.21.15.150:3300/0 conn(0x7f1f6006e830 msgr2=0x7f1f60154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 --2- 172.21.15.78:0/244993440 >> v2:172.21.15.150:3300/0 conn(0x7f1f6006e830 0x7f1f60154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 >> v1:172.21.15.150:6789/0 conn(0x7f1f6006ff80 legacy=0x7f1f600c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60072f70 con 0x7f1f600c6d60 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1f50001670 con 0x7f1f600c6d60 2024-02-19T14:52:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 >> v1:172.21.15.78:6789/0 conn(0x7f1f600c6d60 legacy=0x7f1f600c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f4c003970 0x7f1f4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f4c0047c0 0x7f1f4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c005180 0x7f1f4c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c1940 con 0x7f1f4c003970 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c00c0d0 con 0x7f1f4c005180 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c00c290 con 0x7f1f4c0047c0 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f55d9b640 1 --2- 172.21.15.78:0/244993440 >> v2:172.21.15.150:3300/0 conn(0x7f1f6006e830 0x7f1f60154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f4c003970 0x7f1f4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5ed9d640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c005180 0x7f1f4c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f55d9b640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f4c0047c0 0x7f1f4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5e59c640 1 -- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f4c0047c0 msgr2=0x7f1f4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f4c0047c0 0x7f1f4c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5e59c640 1 -- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c005180 msgr2=0x7f1f4c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c005180 0x7f1f4c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5ed9d640 1 -- 172.21.15.78:0/244993440 reap_dead start 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.068+0000 7f1f5e59c640 1 -- 172.21.15.78:0/244993440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c003520 con 0x7f1f4c003970 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f4c003970 0x7f1f4c003d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c190 tx=0x7f1f4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154ef7bf919af35d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50005ad0 con 0x7f1f4c003970 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f50005c30 con 0x7f1f4c003970 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f5dd9b640 1 -- 172.21.15.78:0/244993440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5002cae0 con 0x7f1f4c003970 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f64f13640 1 -- 172.21.15.78:0/244993440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f4c003970 msgr2=0x7f1f4c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f64f13640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f4c003970 0x7f1f4c003d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c190 tx=0x7f1f4c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f64f13640 1 -- 172.21.15.78:0/244993440 shutdown_connections 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f64f13640 1 --2- 172.21.15.78:0/244993440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f4c003970 0x7f1f4c003d90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f64f13640 1 -- 172.21.15.78:0/244993440 >> 172.21.15.78:0/244993440 conn(0x7f1f6006c170 msgr2=0x7f1f6006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.069+0000 7f1f64f13640 1 -- 172.21.15.78:0/244993440 shutdown_connections 2024-02-19T14:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.070+0000 7f1f64f13640 1 -- 172.21.15.78:0/244993440 wait complete. 2024-02-19T14:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.071+0000 7f1f64f13640 1 Processor -- start 2024-02-19T14:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.071+0000 7f1f64f13640 1 -- start start 2024-02-19T14:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c003970 0x7f1f600a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f6014e320 0x7f1f600c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 0x7f1f60144b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60152970 con 0x7f1f601446a0 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600a7970 con 0x7f1f4c003970 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600a8a10 con 0x7f1f6014e320 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 0x7f1f60144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 0x7f1f60144b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52882/0 (socket says 172.21.15.78:52882) 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 -- 172.21.15.78:0/883183194 learned_addr learned my addr 172.21.15.78:0/883183194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c003970 0x7f1f600a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f55d9b640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f6014e320 0x7f1f600c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 -- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f6014e320 msgr2=0x7f1f600c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f6014e320 0x7f1f600c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 -- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c003970 msgr2=0x7f1f600a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c003970 0x7f1f600a9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 -- 172.21.15.78:0/883183194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60152440 con 0x7f1f601446a0 2024-02-19T14:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f5ed9d640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 0x7f1f60144b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1f58007fd0 tx=0x7f1f5800e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58006d60 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58003970 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58003d20 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f600c47e0 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.072+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f60152690 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.074+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f60074990 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.074+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1f58005070 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.074+0000 7f1f577fe640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f4006b840 0x7f1f4006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.074+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1f580898f0 con 0x7f1f601446a0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.074+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f4006b840 0x7f1f4006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.075+0000 7f1f5e59c640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f4006b840 0x7f1f4006dd00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1f50034000 tx=0x7f1f50009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.079+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1f58054080 con 0x7f1f601446a0 2024-02-19T14:52:26.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.255+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1f600ac870 con 0x7f1f601446a0 2024-02-19T14:52:26.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.255+0000 7f1f577fe640 1 -- 172.21.15.78:0/883183194 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1f58057740 con 0x7f1f601446a0 2024-02-19T14:52:26.258 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:26.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f4006b840 msgr2=0x7f1f4006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f4006b840 0x7f1f4006dd00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1f50034000 tx=0x7f1f50009030 comp rx=0 tx=0).stop 2024-02-19T14:52:26.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 msgr2=0x7f1f60144b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 0x7f1f60144b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1f58007fd0 tx=0x7f1f5800e980 comp rx=0 tx=0).stop 2024-02-19T14:52:26.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 shutdown_connections 2024-02-19T14:52:26.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f4006b840 0x7f1f4006dd00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f601446a0 0x7f1f60144b20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f6014e320 0x7f1f600c38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 --2- 172.21.15.78:0/883183194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f4c003970 0x7f1f600a9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 >> 172.21.15.78:0/883183194 conn(0x7f1f6006c170 msgr2=0x7f1f600c5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 shutdown_connections 2024-02-19T14:52:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.257+0000 7f1f64f13640 1 -- 172.21.15.78:0/883183194 wait complete. 2024-02-19T14:52:26.272 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:26.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.370+0000 7fc443e46640 1 Processor -- start 2024-02-19T14:52:26.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.371+0000 7fc443e46640 1 -- start start 2024-02-19T14:52:26.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.371+0000 7fc443e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.371+0000 7fc443e46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 0x7fc43c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.371+0000 7fc443e46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c0c65a0 con 0x7fc43c0c7910 2024-02-19T14:52:26.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.371+0000 7fc443e46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c0c7530 con 0x7fc43c070720 2024-02-19T14:52:26.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.371+0000 7fc443e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c1537b0 con 0x7fc43c0c5560 2024-02-19T14:52:26.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc4423e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 0x7fc43c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc441be2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc4413e1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc43c0c7910 0x7fc43c154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50474/0 (socket says 172.21.15.78:50474) 2024-02-19T14:52:26.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc4413e1640 1 -- 172.21.15.78:0/1026005342 learned_addr learned my addr 172.21.15.78:0/1026005342 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:26.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc4423e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 0x7fc43c070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:46300/0 (socket says 172.21.15.78:46300) 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc441be2640 1 -- 172.21.15.78:0/1026005342 >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 msgr2=0x7fc43c070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc441be2640 1 --2- 172.21.15.78:0/1026005342 >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 0x7fc43c070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc441be2640 1 -- 172.21.15.78:0/1026005342 >> v1:172.21.15.150:6789/0 conn(0x7fc43c0c7910 legacy=0x7fc43c154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc441be2640 1 -- 172.21.15.78:0/1026005342 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c153a90 con 0x7fc43c0c5560 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.372+0000 7fc4423e3640 1 --2- 172.21.15.78:0/1026005342 >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 0x7fc43c070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.373+0000 7fc441be2640 1 --2- 172.21.15.78:0/1026005342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c6ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc4380097a0 tx=0x7fc43802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=869339dcbd2ce4d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.373+0000 7fc440be0640 1 -- 172.21.15.78:0/1026005342 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43802f050 con 0x7fc43c0c5560 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.373+0000 7fc440be0640 1 -- 172.21.15.78:0/1026005342 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc438031700 con 0x7fc43c0c5560 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.373+0000 7fc440be0640 1 -- 172.21.15.78:0/1026005342 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438031a90 con 0x7fc43c0c5560 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 -- 172.21.15.78:0/1026005342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 msgr2=0x7fc43c0c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 --2- 172.21.15.78:0/1026005342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c6ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc4380097a0 tx=0x7fc43802b7a0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 -- 172.21.15.78:0/1026005342 shutdown_connections 2024-02-19T14:52:26.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 --2- 172.21.15.78:0/1026005342 >> v2:172.21.15.150:3300/0 conn(0x7fc43c070720 0x7fc43c070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 --2- 172.21.15.78:0/1026005342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c6ff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 -- 172.21.15.78:0/1026005342 >> 172.21.15.78:0/1026005342 conn(0x7fc43c06b4b0 msgr2=0x7fc43c0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:26.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 -- 172.21.15.78:0/1026005342 shutdown_connections 2024-02-19T14:52:26.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.374+0000 7fc443e46640 1 -- 172.21.15.78:0/1026005342 wait complete. 2024-02-19T14:52:26.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.375+0000 7fc443e46640 1 Processor -- start 2024-02-19T14:52:26.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.375+0000 7fc443e46640 1 -- start start 2024-02-19T14:52:26.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c070720 0x7fc43c148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 0x7fc43c06c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c06df20 con 0x7fc43c0c7910 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c1490d0 con 0x7fc43c070720 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c1488f0 con 0x7fc43c0c5560 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 0x7fc43c06c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 0x7fc43c06c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52896/0 (socket says 172.21.15.78:52896) 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 -- 172.21.15.78:0/62633866 learned_addr learned my addr 172.21.15.78:0/62633866 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc441be2640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c070720 0x7fc43c148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4413e1640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 -- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 msgr2=0x7fc43c0c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 -- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c070720 msgr2=0x7fc43c148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c070720 0x7fc43c148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 -- 172.21.15.78:0/62633866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc434009a70 con 0x7fc43c0c7910 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4423e3640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 0x7fc43c06c670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc43400efc0 tx=0x7fc43400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434009cd0 con 0x7fc43c0c7910 2024-02-19T14:52:26.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43400f040 con 0x7fc43c0c7910 2024-02-19T14:52:26.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438009400 con 0x7fc43c0c7910 2024-02-19T14:52:26.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.376+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434003990 con 0x7fc43c0c7910 2024-02-19T14:52:26.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.377+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c0c5ac0 con 0x7fc43c0c7910 2024-02-19T14:52:26.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.378+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c074990 con 0x7fc43c0c7910 2024-02-19T14:52:26.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.378+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc434014480 con 0x7fc43c0c7910 2024-02-19T14:52:26.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.378+0000 7fc4327fc640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42006b810 0x7fc42006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.378+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc43408af40 con 0x7fc43c0c7910 2024-02-19T14:52:26.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.378+0000 7fc441be2640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42006b810 0x7fc42006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.379+0000 7fc441be2640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42006b810 0x7fc42006dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc438035000 tx=0x7fc438008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.382+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc434055760 con 0x7fc43c0c7910 2024-02-19T14:52:26.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.597+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc43c0c76f0 con 0x7fc43c0c7910 2024-02-19T14:52:26.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.597+0000 7fc4327fc640 1 -- 172.21.15.78:0/62633866 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7fc434018090 con 0x7fc43c0c7910 2024-02-19T14:52:26.600 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:26.600 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:26.600 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42006b810 msgr2=0x7fc42006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42006b810 0x7fc42006dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc438035000 tx=0x7fc438008030 comp rx=0 tx=0).stop 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 msgr2=0x7fc43c06c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 0x7fc43c06c670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc43400efc0 tx=0x7fc43400c490 comp rx=0 tx=0).stop 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 shutdown_connections 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42006b810 0x7fc42006dcd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc43c0c7910 0x7fc43c06c670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc43c0c5560 0x7fc43c0c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 --2- 172.21.15.78:0/62633866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c070720 0x7fc43c148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 >> 172.21.15.78:0/62633866 conn(0x7fc43c06b4b0 msgr2=0x7fc43c1548e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 shutdown_connections 2024-02-19T14:52:26.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.600+0000 7fc443e46640 1 -- 172.21.15.78:0/62633866 wait complete. 2024-02-19T14:52:26.613 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:26.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.713+0000 7f18cb9e9640 1 Processor -- start 2024-02-19T14:52:26.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.713+0000 7f18cb9e9640 1 -- start start 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18cb9e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f18c414cd50 0x7f18c40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18cb9e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f18c41537a0 0x7f18c40c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18cb9e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c41563b0 con 0x7f18c40c7b70 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18cb9e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c4153cd0 con 0x7f18c414cd50 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18cb9e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c406b6e0 con 0x7f18c41537a0 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18c9785640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f18c414cd50 0x7f18c40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18c9785640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f18c414cd50 0x7f18c40c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52900/0 (socket says 172.21.15.78:52900) 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18c9785640 1 -- 172.21.15.78:0/4151836108 learned_addr learned my addr 172.21.15.78:0/4151836108 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.714+0000 7f18c8f84640 1 --2- 172.21.15.78:0/4151836108 >> v2:172.21.15.150:3300/0 conn(0x7f18c41537a0 0x7f18c40c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18c9785640 1 -- 172.21.15.78:0/4151836108 >> v2:172.21.15.150:3300/0 conn(0x7f18c41537a0 msgr2=0x7f18c40c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18c9785640 1 --2- 172.21.15.78:0/4151836108 >> v2:172.21.15.150:3300/0 conn(0x7f18c41537a0 0x7f18c40c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18c9785640 1 -- 172.21.15.78:0/4151836108 >> v1:172.21.15.150:6789/0 conn(0x7f18c40c7b70 legacy=0x7f18c4156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18c9785640 1 -- 172.21.15.78:0/4151836108 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c406b980 con 0x7f18c414cd50 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18c8f84640 1 --2- 172.21.15.78:0/4151836108 >> v2:172.21.15.150:3300/0 conn(0x7f18c41537a0 0x7f18c40c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18c9785640 1 --2- 172.21.15.78:0/4151836108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c414cd50 0x7f18c40c4cd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f18c40acad0 tx=0x7f18c002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec6540ca257b673 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18bbfff640 1 -- 172.21.15.78:0/4151836108 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0031750 con 0x7f18c414cd50 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18bbfff640 1 -- 172.21.15.78:0/4151836108 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c00318b0 con 0x7f18c414cd50 2024-02-19T14:52:26.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.715+0000 7f18bbfff640 1 -- 172.21.15.78:0/4151836108 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0031c40 con 0x7f18c414cd50 2024-02-19T14:52:26.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 -- 172.21.15.78:0/4151836108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c414cd50 msgr2=0x7f18c40c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/4151836108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c414cd50 0x7f18c40c4cd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f18c40acad0 tx=0x7f18c002b7f0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 -- 172.21.15.78:0/4151836108 shutdown_connections 2024-02-19T14:52:26.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/4151836108 >> v2:172.21.15.150:3300/0 conn(0x7f18c41537a0 0x7f18c40c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/4151836108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c414cd50 0x7f18c40c4cd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 -- 172.21.15.78:0/4151836108 >> 172.21.15.78:0/4151836108 conn(0x7f18c4066e70 msgr2=0x7f18c40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:26.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 -- 172.21.15.78:0/4151836108 shutdown_connections 2024-02-19T14:52:26.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.716+0000 7f18cb9e9640 1 -- 172.21.15.78:0/4151836108 wait complete. 2024-02-19T14:52:26.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.718+0000 7f18cb9e9640 1 Processor -- start 2024-02-19T14:52:26.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.718+0000 7f18cb9e9640 1 -- start start 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18cb9e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 0x7f18c41ec080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18cb9e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18c414cd50 0x7f18c4146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18cb9e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18c41537a0 0x7f18c41470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18cb9e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40c6cf0 con 0x7f18c40c7b70 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18cb9e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c414c240 con 0x7f18c414cd50 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18cb9e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4146150 con 0x7f18c41537a0 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 0x7f18c41ec080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 0x7f18c41ec080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52910/0 (socket says 172.21.15.78:52910) 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 -- 172.21.15.78:0/1615407936 learned_addr learned my addr 172.21.15.78:0/1615407936 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:26.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c8f84640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18c414cd50 0x7f18c4146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9f86640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18c41537a0 0x7f18c41470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 -- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18c41537a0 msgr2=0x7f18c41470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18c41537a0 0x7f18c41470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 -- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18c414cd50 msgr2=0x7f18c4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18c414cd50 0x7f18c4146b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.719+0000 7f18c9785640 1 -- 172.21.15.78:0/1615407936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c00094d0 con 0x7f18c40c7b70 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.720+0000 7f18c9785640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 0x7f18c41ec080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f18c0000c00 tx=0x7f18c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.720+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0033c80 con 0x7f18c40c7b70 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.720+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0033de0 con 0x7f18c40c7b70 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.720+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c4145e70 con 0x7f18c40c7b70 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.720+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c003c640 con 0x7f18c40c7b70 2024-02-19T14:52:26.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.720+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c41ecba0 con 0x7f18c40c7b70 2024-02-19T14:52:26.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.721+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c40c6e60 con 0x7f18c40c7b70 2024-02-19T14:52:26.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.721+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f18c003a040 con 0x7f18c40c7b70 2024-02-19T14:52:26.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.722+0000 7f18b9ffb640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18a806b810 0x7f18a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:26.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.722+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f18c00ac180 con 0x7f18c40c7b70 2024-02-19T14:52:26.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.722+0000 7f18c8f84640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18a806b810 0x7f18a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:26.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.722+0000 7f18c8f84640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18a806b810 0x7f18a806dcd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f18c41482c0 tx=0x7f18b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:26.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.726+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f18c00769a0 con 0x7f18c40c7b70 2024-02-19T14:52:26.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.904+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18c4074990 con 0x7f18c40c7b70 2024-02-19T14:52:26.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.905+0000 7f18b9ffb640 1 -- 172.21.15.78:0/1615407936 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f18c007a060 con 0x7f18c40c7b70 2024-02-19T14:52:26.908 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:26.908 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:26.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18a806b810 msgr2=0x7f18a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18a806b810 0x7f18a806dcd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f18c41482c0 tx=0x7f18b4008040 comp rx=0 tx=0).stop 2024-02-19T14:52:26.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 msgr2=0x7f18c41ec080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:26.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 0x7f18c41ec080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f18c0000c00 tx=0x7f18c00046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 shutdown_connections 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18a806b810 0x7f18a806dcd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18c41537a0 0x7f18c41470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.907+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18c414cd50 0x7f18c4146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.908+0000 7f18cb9e9640 1 --2- 172.21.15.78:0/1615407936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18c40c7b70 0x7f18c41ec080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.908+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 >> 172.21.15.78:0/1615407936 conn(0x7f18c4066e70 msgr2=0x7f18c406f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.908+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 shutdown_connections 2024-02-19T14:52:26.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:26.908+0000 7f18cb9e9640 1 -- 172.21.15.78:0/1615407936 wait complete. 2024-02-19T14:52:26.921 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.018+0000 7f98c51a1640 1 Processor -- start 2024-02-19T14:52:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.018+0000 7f98c51a1640 1 -- start start 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98c51a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98c51a1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c00c65a0 con 0x7f98c00c7910 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98c51a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c00c7530 con 0x7f98c0070720 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98c51a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c01537b0 con 0x7f98c00c5560 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98be59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f98c00c7910 0x7f98c0154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57890/0 (socket says 172.21.15.78:57890) 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98be59c640 1 -- 172.21.15.78:0/3791008985 learned_addr learned my addr 172.21.15.78:0/3791008985 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bed9d640 1 --2- 172.21.15.78:0/3791008985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612454752 0 0) 0x7f98c00c65a0 con 0x7f98c00c7910 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a8003520 con 0x7f98c00c7910 2024-02-19T14:52:27.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588128283 0 0) 0x7f98c00c7530 con 0x7f98c0070720 2024-02-19T14:52:27.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c00c65a0 con 0x7f98c0070720 2024-02-19T14:52:27.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bed9d640 1 -- 172.21.15.78:0/3791008985 >> v1:172.21.15.150:6789/0 conn(0x7f98c0070720 legacy=0x7f98c0070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bed9d640 1 -- 172.21.15.78:0/3791008985 >> v1:172.21.15.78:6789/0 conn(0x7f98c00c7910 legacy=0x7f98c0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.020+0000 7f98bed9d640 1 -- 172.21.15.78:0/3791008985 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0153a90 con 0x7f98c00c5560 2024-02-19T14:52:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.021+0000 7f98bed9d640 1 --2- 172.21.15.78:0/3791008985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c00c6ff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98b80097d0 tx=0x7f98b802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930b340dbce2986c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.021+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8031770 con 0x7f98c00c5560 2024-02-19T14:52:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.021+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b80318d0 con 0x7f98c00c5560 2024-02-19T14:52:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.021+0000 7f98bdd9b640 1 -- 172.21.15.78:0/3791008985 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8031c40 con 0x7f98c00c5560 2024-02-19T14:52:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 -- 172.21.15.78:0/3791008985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 msgr2=0x7f98c00c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 --2- 172.21.15.78:0/3791008985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c00c6ff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98b80097d0 tx=0x7f98b802b810 comp rx=0 tx=0).stop 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 -- 172.21.15.78:0/3791008985 shutdown_connections 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 --2- 172.21.15.78:0/3791008985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c00c6ff0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 -- 172.21.15.78:0/3791008985 >> 172.21.15.78:0/3791008985 conn(0x7f98c006b4b0 msgr2=0x7f98c00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 -- 172.21.15.78:0/3791008985 shutdown_connections 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.022+0000 7f98c51a1640 1 -- 172.21.15.78:0/3791008985 wait complete. 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.024+0000 7f98c51a1640 1 Processor -- start 2024-02-19T14:52:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.024+0000 7f98c51a1640 1 -- start start 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98c51a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98c0070720 0x7f98c00aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98c51a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c01df1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98c51a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 0x7f98c01df720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98c51a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c01dffc0 con 0x7f98c00c7910 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98c51a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c01dfd20 con 0x7f98c0070720 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98c51a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c014d580 con 0x7f98c00c5560 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98bf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 0x7f98c01df720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98bf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 0x7f98c01df720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52926/0 (socket says 172.21.15.78:52926) 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.025+0000 7f98bf59e640 1 -- 172.21.15.78:0/522981737 learned_addr learned my addr 172.21.15.78:0/522981737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bed9d640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98c0070720 0x7f98c00aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98be59c640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c01df1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bf59e640 1 -- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 msgr2=0x7f98c01df1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bf59e640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c01df1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bf59e640 1 -- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98c0070720 msgr2=0x7f98c00aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bf59e640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98c0070720 0x7f98c00aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bf59e640 1 -- 172.21.15.78:0/522981737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8009400 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98be59c640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c01df1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98bf59e640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 0x7f98c01df720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f98b0001eb0 tx=0x7f98b000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b00052d0 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b0005430 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c00c5ac0 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b00037f0 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.026+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c01e95a0 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.027+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c0074990 con 0x7f98c00c7910 2024-02-19T14:52:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.028+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f98b0003950 con 0x7f98c00c7910 2024-02-19T14:52:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.028+0000 7f98a77fe640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98ac06b840 0x7f98ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.028+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f98b0089920 con 0x7f98c00c7910 2024-02-19T14:52:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.028+0000 7f98bed9d640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98ac06b840 0x7f98ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.028+0000 7f98bed9d640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98ac06b840 0x7f98ac06dd00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f98b8036000 tx=0x7f98b803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.036+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98b00540b0 con 0x7f98c00c7910 2024-02-19T14:52:27.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.250+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98c00c2920 con 0x7f98c00c7910 2024-02-19T14:52:27.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.251+0000 7f98a77fe640 1 -- 172.21.15.78:0/522981737 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7f98b0057770 con 0x7f98c00c7910 2024-02-19T14:52:27.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:27.254 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:27.254 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:27.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98ac06b840 msgr2=0x7f98ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98ac06b840 0x7f98ac06dd00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f98b8036000 tx=0x7f98b803d040 comp rx=0 tx=0).stop 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 msgr2=0x7f98c01df720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 0x7f98c01df720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f98b0001eb0 tx=0x7f98b000c820 comp rx=0 tx=0).stop 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 shutdown_connections 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98ac06b840 0x7f98ac06dd00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98c00c7910 0x7f98c01df720 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98c00c5560 0x7f98c01df1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 --2- 172.21.15.78:0/522981737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98c0070720 0x7f98c00aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 >> 172.21.15.78:0/522981737 conn(0x7f98c006b4b0 msgr2=0x7f98c0070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 shutdown_connections 2024-02-19T14:52:27.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.253+0000 7f98c51a1640 1 -- 172.21.15.78:0/522981737 wait complete. 2024-02-19T14:52:27.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:51:59.591061+0000', 'modified': '2024-02-19T14:52:15.695071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/1940908201', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 1940908201}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1940908201}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:52:27.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:52:27.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:52:27.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.769+0000 7fd000bc9640 1 Processor -- start 2024-02-19T14:52:27.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.769+0000 7fd000bc9640 1 -- start start 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fd000bc9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcffc0c78a0 0x7fcffc159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fd000bc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 0x7fcffc0c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fd000bc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc06ffa0 con 0x7fcffc0c48f0 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fd000bc9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcffc06b720 con 0x7fcffc0c78a0 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fd000bc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc06fc70 con 0x7fcffc0c4d80 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcffad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcffc0c48f0 0x7fcffc1523d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51472/0 (socket says 172.21.15.78:51472) 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcffad9d640 1 -- 172.21.15.78:0/4050309828 learned_addr learned my addr 172.21.15.78:0/4050309828 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4050309828 >> v2:172.21.15.150:3300/0 conn(0x7fcffc0c78a0 0x7fcffc159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 0x7fcffc0c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692079915 0 0) 0x7fcffc06ffa0 con 0x7fcffc0c48f0 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe4003520 con 0x7fcffc0c48f0 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934857443 0 0) 0x7fcfe4003520 con 0x7fcffc0c48f0 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 msgr2=0x7fcffc0c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 0x7fcffc0c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 >> v2:172.21.15.150:3300/0 conn(0x7fcffc0c78a0 msgr2=0x7fcffc159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 --2- 172.21.15.78:0/4050309828 >> v2:172.21.15.150:3300/0 conn(0x7fcffc0c78a0 0x7fcffc159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc06b930 con 0x7fcffc0c48f0 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fcff40019f0 con 0x7fcffc0c48f0 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 >> v1:172.21.15.78:6789/0 conn(0x7fcffc0c48f0 legacy=0x7fcffc1523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe4003970 0x7fcfe4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 0x7fcfe40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4004d00 0x7fcfe4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc06b720 con 0x7fcfe4003970 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc06fc70 con 0x7fcfe4004d00 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.770+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe400b980 con 0x7fcffc0c4d80 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4050309828 >> v2:172.21.15.150:3300/0 conn(0x7fcffc0c78a0 0x7fcffc159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 0x7fcfe40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe4003970 0x7fcfe4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4004d00 0x7fcfe4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 -- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 msgr2=0x7fcfe40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c4d80 0x7fcfe40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 -- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4004d00 msgr2=0x7fcfe4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4004d00 0x7fcfe4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.771+0000 7fcffa59c640 1 -- 172.21.15.78:0/4050309828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe4003520 con 0x7fcfe4003970 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe4003970 0x7fcfe4003d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009c70 tx=0x7fcfe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bf0ee9e1ded74f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec004020 con 0x7fcfe4003970 2024-02-19T14:52:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec004180 con 0x7fcfe4003970 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fcff959a640 1 -- 172.21.15.78:0/4050309828 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02c960 con 0x7fcfe4003970 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fd000bc9640 1 -- 172.21.15.78:0/4050309828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe4003970 msgr2=0x7fcfe4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fd000bc9640 1 --2- 172.21.15.78:0/4050309828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcfe4003970 0x7fcfe4003d90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009c70 tx=0x7fcfe40057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fcffad9d640 1 -- 172.21.15.78:0/4050309828 reap_dead start 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fd000bc9640 1 -- 172.21.15.78:0/4050309828 shutdown_connections 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fd000bc9640 1 -- 172.21.15.78:0/4050309828 >> 172.21.15.78:0/4050309828 conn(0x7fcffc066e70 msgr2=0x7fcffc0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fd000bc9640 1 -- 172.21.15.78:0/4050309828 shutdown_connections 2024-02-19T14:52:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.772+0000 7fd000bc9640 1 -- 172.21.15.78:0/4050309828 wait complete. 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 Processor -- start 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 -- start start 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4003970 0x7fcffc158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c5290 0x7fcffc0aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 0x7fcffc1e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1e8080 con 0x7fcffc1e76c0 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc14c990 con 0x7fcfe4003970 2024-02-19T14:52:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fd000bc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc156550 con 0x7fcffc0c5290 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.774+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 0x7fcffc1e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 0x7fcffc1e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53596/0 (socket says 172.21.15.78:53596) 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 -- 172.21.15.78:0/3239233732 learned_addr learned my addr 172.21.15.78:0/3239233732 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c5290 0x7fcffc0aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffa59c640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4003970 0x7fcffc158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 -- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c5290 msgr2=0x7fcffc0aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c5290 0x7fcffc0aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 -- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4003970 msgr2=0x7fcffc158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4003970 0x7fcffc158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 -- 172.21.15.78:0/3239233732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc1ee510 con 0x7fcffc1e76c0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcffad9d640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 0x7fcffc1e7b40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcff4001330 tx=0x7fcff4007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcfeaffd640 1 -- 172.21.15.78:0/3239233732 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff400ace0 con 0x7fcffc1e76c0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcfeaffd640 1 -- 172.21.15.78:0/3239233732 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff40133f0 con 0x7fcffc1e76c0 2024-02-19T14:52:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc1ee7f0 con 0x7fcffc1e76c0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fcfeaffd640 1 -- 172.21.15.78:0/3239233732 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff400aea0 con 0x7fcffc1e76c0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.775+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc0abf50 con 0x7fcffc1e76c0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.777+0000 7fcfeaffd640 1 -- 172.21.15.78:0/3239233732 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcff40137a0 con 0x7fcffc1e76c0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.777+0000 7fcfeaffd640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcfdc06b840 0x7fcfdc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.777+0000 7fcfeaffd640 1 -- 172.21.15.78:0/3239233732 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcff408aba0 con 0x7fcffc1e76c0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.777+0000 7fcffa59c640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcfdc06b840 0x7fcfdc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.778+0000 7fcffa59c640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcfdc06b840 0x7fcfdc06dd00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcfec035000 tx=0x7fcfec030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.778+0000 7fd0037a3740 1 Processor -- start 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.778+0000 7fd0037a3740 1 -- start start 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec609dae0 0x560ec60d7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec609c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fd0037a3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec6057cd0 con 0x560ec60da8b0 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec600af20 con 0x560ec60d7740 2024-02-19T14:52:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ec6008450 con 0x560ec609dae0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec609c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec609c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53598/0 (socket says 172.21.15.78:53598) 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 -- 172.21.15.78:0/4095773298 learned_addr learned my addr 172.21.15.78:0/4095773298 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec609dae0 0x560ec60d7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 -- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec609dae0 msgr2=0x560ec60d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec609dae0 0x560ec60d7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 -- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 msgr2=0x560ec60d88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 -- 172.21.15.78:0/4095773298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ec5fb8f20 con 0x560ec60da8b0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec609c860 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcff4088d00 tx=0x7fcff4058940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd288afe6cf49b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.779+0000 7fcfdbfff640 1 -- 172.21.15.78:0/4095773298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4070070 con 0x560ec60da8b0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fcfdbfff640 1 -- 172.21.15.78:0/4095773298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff405d130 con 0x560ec60da8b0 2024-02-19T14:52:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fcfdbfff640 1 -- 172.21.15.78:0/4095773298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff40616f0 con 0x560ec60da8b0 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 -- 172.21.15.78:0/4095773298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 msgr2=0x560ec609c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec609c860 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcff4088d00 tx=0x7fcff4058940 comp rx=0 tx=0).stop 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 -- 172.21.15.78:0/4095773298 shutdown_connections 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec609c860 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec609dae0 0x560ec60d7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4095773298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.780+0000 7fd0037a3740 1 -- 172.21.15.78:0/4095773298 >> 172.21.15.78:0/4095773298 conn(0x560ec6099120 msgr2=0x560ec60bdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.781+0000 7fd0037a3740 1 -- 172.21.15.78:0/4095773298 shutdown_connections 2024-02-19T14:52:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.781+0000 7fd0037a3740 1 -- 172.21.15.78:0/4095773298 wait complete. 2024-02-19T14:52:27.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.781+0000 7fd0037a3740 1 -- start start 2024-02-19T14:52:27.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.781+0000 7fd0037a3740 1 Processor -- start 2024-02-19T14:52:27.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 0x560ec60eca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60da8b0 0x560ec60d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60afe10 0x560ec609b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec6003c30 con 0x560ec60d7740 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec5fa8d30 con 0x560ec60afe10 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ec5cd7310 con 0x560ec60da8b0 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fd0037a3740 10 client.0 _ll_register_callbacks cb 0x560ec5fb0d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 0x560ec60eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 0x560ec60eca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53600/0 (socket says 172.21.15.78:53600) 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 -- 172.21.15.78:0/346701136 learned_addr learned my addr 172.21.15.78:0/346701136 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffad9d640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60afe10 0x560ec609b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60da8b0 0x560ec60d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 -- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60da8b0 msgr2=0x560ec60d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60da8b0 0x560ec60d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 -- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60afe10 msgr2=0x560ec609b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60afe10 0x560ec609b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcffa59c640 1 -- 172.21.15.78:0/346701136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc0abf50 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.782+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60da8b0 0x560ec60d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcffa59c640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 0x560ec60eca40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d60 tx=0x7fcfec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a739c0e5e46a2d35 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcfd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcfd97fa640 1 -- 172.21.15.78:0/346701136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02cc30 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcfd97fa640 1 -- 172.21.15.78:0/346701136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcfec02cd90 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ec5d036c0 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcfd97fa640 1 -- 172.21.15.78:0/346701136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfec006040 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 10 client.4836 fetch_fsmap learned FSMap version 8 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560ec60022b0 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcfd97fa640 1 -- 172.21.15.78:0/346701136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560ec5d036c0 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcfd97fa640 1 -- 172.21.15.78:0/346701136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 8) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fcfec0061a0 con 0x560ec60d7740 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 10 client.4836 fetch_fsmap finished waiting for FSMap version 8 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 10 client.4836 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 20 client.4836 populate_metadata read hostname 'smithi078' 2024-02-19T14:52:27.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec60cff40 0x560ec60d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 4 client.4836 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 --> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ec5d54270 con 0x560ec60cff40 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.783+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec60cff40 0x560ec60d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fcfd97fa640 10 client.4836 ms_handle_connect on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec60cff40 0x560ec60d0320 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1428626e7e0b55c5 server_cookie=c85c5772ab743c9b in_seq=0 out_seq=0 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fcfd97fa640 1 -- 172.21.15.78:0/346701136 <== mds.0 v2:172.21.15.78:6836/1940908201 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560ec5d54270 con 0x560ec60cff40 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fcfd97fa640 10 client.4836 handle_command_reply: tid=0 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 client.4836 shutdown 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 msgr2=0x560ec60eca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 0x560ec60eca40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009d60 tx=0x7fcfec0046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 shutdown_connections 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec60cff40 0x560ec60d0320 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60afe10 0x560ec609b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60da8b0 0x560ec60d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fcfd97fa640 0 client.4836 ms_handle_reset on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 --2- 172.21.15.78:0/346701136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60d7740 0x560ec60eca40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.784+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 >> 172.21.15.78:0/346701136 conn(0x560ec609dae0 msgr2=0x560ec6095f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.785+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 shutdown_connections 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.785+0000 7fd0037a3740 1 -- 172.21.15.78:0/346701136 wait complete. 2024-02-19T14:52:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.785+0000 7fd0037a3740 20 client.4836 trim_cache size 0 max 16384 2024-02-19T14:52:27.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.801+0000 7fd0037a3740 1 Processor -- start 2024-02-19T14:52:27.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.801+0000 7fd0037a3740 1 -- start start 2024-02-19T14:52:27.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 0x560ec60d9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60d7740 0x560ec613ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60da8b0 0x560ec613ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fd0037a3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec5f89490 con 0x560ec60afe10 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec60ba990 con 0x560ec60da8b0 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ec60af870 con 0x560ec60d7740 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 0x560ec60d9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 0x560ec60d9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53604/0 (socket says 172.21.15.78:53604) 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 -- 172.21.15.78:0/2804287379 learned_addr learned my addr 172.21.15.78:0/2804287379 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 -- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60d7740 msgr2=0x560ec613ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60d7740 0x560ec613ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 -- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60da8b0 msgr2=0x560ec613ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffad9d640 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60da8b0 0x560ec613ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60da8b0 0x560ec613ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 -- 172.21.15.78:0/2804287379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ec5f9ea20 con 0x560ec60afe10 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcffa59c640 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 0x560ec60d9030 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00f170 tx=0x7fcfec04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1edaa244cc743e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcfdb7fe640 1 -- 172.21.15.78:0/2804287379 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec005d60 con 0x560ec60afe10 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcfdb7fe640 1 -- 172.21.15.78:0/2804287379 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec006610 con 0x560ec60afe10 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.802+0000 7fcfdb7fe640 1 -- 172.21.15.78:0/2804287379 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0061f0 con 0x560ec60afe10 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 -- 172.21.15.78:0/2804287379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 msgr2=0x560ec60d9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 0x560ec60d9030 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00f170 tx=0x7fcfec04b000 comp rx=0 tx=0).stop 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 -- 172.21.15.78:0/2804287379 shutdown_connections 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60da8b0 0x560ec613ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60d7740 0x560ec613ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 --2- 172.21.15.78:0/2804287379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60afe10 0x560ec60d9030 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 -- 172.21.15.78:0/2804287379 >> 172.21.15.78:0/2804287379 conn(0x560ec609dae0 msgr2=0x560ec6095f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 -- 172.21.15.78:0/2804287379 shutdown_connections 2024-02-19T14:52:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.803+0000 7fd0037a3740 1 -- 172.21.15.78:0/2804287379 wait complete. 2024-02-19T14:52:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 -- start start 2024-02-19T14:52:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 Processor -- start 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60afe10 0x560ec60ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec613a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec613b9d0 con 0x560ec60da8b0 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec609b830 con 0x560ec60d7740 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ec60e4fe0 con 0x560ec60afe10 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fd0037a3740 10 client.0 _ll_register_callbacks cb 0x7fcfe40045d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:52:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.804+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec613a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec613a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53618/0 (socket says 172.21.15.78:53618) 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 -- 172.21.15.78:0/4253535971 learned_addr learned my addr 172.21.15.78:0/4253535971 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcff9d9b640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60afe10 0x560ec60ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 -- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60afe10 msgr2=0x560ec60ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60afe10 0x560ec60ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 -- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 msgr2=0x560ec60d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 -- 172.21.15.78:0/4253535971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ec60022b0 con 0x560ec60da8b0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60afe10 0x560ec60ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcffad9d640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec613a8d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcff4088de0 tx=0x7fcff4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e059884eae321d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcfd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcfd97fa640 1 -- 172.21.15.78:0/4253535971 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4070040 con 0x560ec60da8b0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcfd97fa640 1 -- 172.21.15.78:0/4253535971 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcff40616f0 con 0x560ec60da8b0 2024-02-19T14:52:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcff405d130 con 0x560ec60da8b0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcfd97fa640 1 -- 172.21.15.78:0/4253535971 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcff4058630 con 0x560ec60da8b0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fd0037a3740 10 client.4842 fetch_fsmap learned FSMap version 8 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560ec5fc4780 con 0x560ec60da8b0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcfd97fa640 1 -- 172.21.15.78:0/4253535971 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff405d130 con 0x560ec60da8b0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fcfd97fa640 1 -- 172.21.15.78:0/4253535971 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 8) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fcff4012070 con 0x560ec60da8b0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fd0037a3740 10 client.4842 fetch_fsmap finished waiting for FSMap version 8 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fd0037a3740 10 client.4842 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.805+0000 7fd0037a3740 20 client.4842 populate_metadata read hostname 'smithi078' 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec613ae10 0x560ec6096de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 4 client.4842 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 --> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560ec5d40610 con 0x560ec613ae10 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec613ae10 0x560ec6096de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fcfd97fa640 10 client.4842 ms_handle_connect on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fcffa59c640 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec613ae10 0x560ec6096de0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3131c0b0ae387a6 server_cookie=3b5e9b2912bfd6b2 in_seq=0 out_seq=0 2024-02-19T14:52:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fcfd97fa640 1 -- 172.21.15.78:0/4253535971 <== mds.0 v2:172.21.15.78:6836/1940908201 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x560ec5d40610 con 0x560ec613ae10 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fcfd97fa640 10 client.4842 handle_command_reply: tid=0 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 client.4842 shutdown 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 msgr2=0x560ec613a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec613a8d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcff4088de0 tx=0x7fcff4000f40 comp rx=0 tx=0).stop 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 shutdown_connections 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:6836/1940908201,v1:172.21.15.78:6837/1940908201] conn(0x560ec613ae10 0x560ec6096de0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560ec60da8b0 0x560ec613a8d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.806+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec60d7740 0x560ec60d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd0037a3740 1 --2- 172.21.15.78:0/4253535971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560ec60afe10 0x560ec60ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fcfd97fa640 0 client.4842 ms_handle_reset on v2:172.21.15.78:6836/1940908201 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 >> 172.21.15.78:0/4253535971 conn(0x560ec609dae0 msgr2=0x560ec6095f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 shutdown_connections 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd0037a3740 1 -- 172.21.15.78:0/4253535971 wait complete. 2024-02-19T14:52:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd0037a3740 20 client.4842 trim_cache size 0 max 16384 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4290, 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 27.553936745000001, 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 8, 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 24, 2024-02-19T14:52:27.817 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 24, 2024-02-19T14:52:27.818 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 34.170797339000003 2024-02-19T14:52:27.818 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:52:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcfdc06b840 msgr2=0x7fcfdc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcfdc06b840 0x7fcfdc06dd00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcfec035000 tx=0x7fcfec030000 comp rx=0 tx=0).stop 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 msgr2=0x7fcffc1e7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 0x7fcffc1e7b40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcff4001330 tx=0x7fcff4007bd0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 shutdown_connections 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcfdc06b840 0x7fcfdc06dd00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcffc1e76c0 0x7fcffc1e7b40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcffc0c5290 0x7fcffc0aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 --2- 172.21.15.78:0/3239233732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfe4003970 0x7fcffc158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 >> 172.21.15.78:0/3239233732 conn(0x7fcffc066e70 msgr2=0x7fcffc0a8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 shutdown_connections 2024-02-19T14:52:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:27.807+0000 7fd000bc9640 1 -- 172.21.15.78:0/3239233732 wait complete. 2024-02-19T14:52:27.821 INFO:teuthology.run_tasks:Running task kclient... 2024-02-19T14:52:27.837 INFO:tasks.kclient:Mounting kernel clients... 2024-02-19T14:52:27.837 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-19T14:52:27.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:27.837 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-02-19T14:52:27.852 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-02-19T14:52:27.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:27.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-19T14:52:27.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:27.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-02-19T14:52:28.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.011 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-02-19T14:52:28.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-02-19T14:52:28.104 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2024-02-19T14:52:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T14:52:28.106 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-02-19T14:52:28.106 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T14:52:28.106 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T14:52:28.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T14:52:28.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.208+0000 7fa21b3c2640 1 Processor -- start 2024-02-19T14:52:28.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.208+0000 7fa21b3c2640 1 -- start start 2024-02-19T14:52:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21b3c2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2140c29c0 0x7fa2141526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21b3c2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa214072250 con 0x7fa2140c5be0 2024-02-19T14:52:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21b3c2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa214072480 con 0x7fa2140c25e0 2024-02-19T14:52:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21b3c2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa214152c90 con 0x7fa2140c29c0 2024-02-19T14:52:28.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21895d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2140c29c0 0x7fa2141526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21995f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa2140c5be0 0x7fa21406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51478/0 (socket says 172.21.15.78:51478) 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21995f640 1 -- 172.21.15.78:0/326107773 learned_addr learned my addr 172.21.15.78:0/326107773 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.209+0000 7fa21895d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2140c29c0 0x7fa2141526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53632/0 (socket says 172.21.15.78:53632) 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa21895d640 1 -- 172.21.15.78:0/326107773 >> v1:172.21.15.150:6789/0 conn(0x7fa2140c25e0 legacy=0x7fa2140c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa21895d640 1 -- 172.21.15.78:0/326107773 >> v1:172.21.15.78:6789/0 conn(0x7fa2140c5be0 legacy=0x7fa21406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa20bfff640 1 -- 172.21.15.78:0/326107773 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37775935 0 0) 0x7fa214072250 con 0x7fa2140c5be0 2024-02-19T14:52:28.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa21895d640 1 -- 172.21.15.78:0/326107773 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140c4b90 con 0x7fa2140c29c0 2024-02-19T14:52:28.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa21895d640 1 --2- 172.21.15.78:0/326107773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa2141526a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa2040097a0 tx=0x7fa20402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419d7b0d8d7ec44e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa20bfff640 1 -- 172.21.15.78:0/326107773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204031830 con 0x7fa2140c29c0 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa20bfff640 1 -- 172.21.15.78:0/326107773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa204031990 con 0x7fa2140c29c0 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.210+0000 7fa20bfff640 1 -- 172.21.15.78:0/326107773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204031d20 con 0x7fa2140c29c0 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 -- 172.21.15.78:0/326107773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 msgr2=0x7fa2141526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/326107773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa2141526a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa2040097a0 tx=0x7fa20402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 -- 172.21.15.78:0/326107773 shutdown_connections 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/326107773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa2141526a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 -- 172.21.15.78:0/326107773 >> 172.21.15.78:0/326107773 conn(0x7fa21406b270 msgr2=0x7fa21406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 -- 172.21.15.78:0/326107773 shutdown_connections 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.211+0000 7fa21b3c2640 1 -- 172.21.15.78:0/326107773 wait complete. 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.212+0000 7fa21b3c2640 1 Processor -- start 2024-02-19T14:52:28.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.212+0000 7fa21b3c2640 1 -- start start 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21b3c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2140c25e0 0x7fa2140bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21b3c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa21414c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21b3c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2140c5be0 0x7fa21414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21b3c2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2141ee3b0 con 0x7fa2140c29c0 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21b3c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21414cdc0 con 0x7fa2140c25e0 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21b3c2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2141ed730 con 0x7fa2140c5be0 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa21414c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa21414c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53642/0 (socket says 172.21.15.78:53642) 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 -- 172.21.15.78:0/2812119643 learned_addr learned my addr 172.21.15.78:0/2812119643 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21915e640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2140c25e0 0x7fa2140bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:28.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21995f640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2140c5be0 0x7fa21414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 -- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2140c5be0 msgr2=0x7fa21414c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2140c5be0 0x7fa21414c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 -- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2140c25e0 msgr2=0x7fa2140bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2140c25e0 0x7fa2140bf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.213+0000 7fa21895d640 1 -- 172.21.15.78:0/2812119643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204009400 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.214+0000 7fa21895d640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa21414c340 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa2040098c0 tx=0x7fa2040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.214+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20403ba70 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.214+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20403bbf0 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.214+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2141eda10 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.214+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2040453f0 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.214+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2141edfd0 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.215+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2141ee4f0 con 0x7fa2140c29c0 2024-02-19T14:52:28.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.216+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa20404e450 con 0x7fa2140c29c0 2024-02-19T14:52:28.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.216+0000 7fa209ffb640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa1f806b840 0x7fa1f806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:28.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.216+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa2040ac570 con 0x7fa2140c29c0 2024-02-19T14:52:28.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.216+0000 7fa21915e640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa1f806b840 0x7fa1f806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:28.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.217+0000 7fa21915e640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa1f806b840 0x7fa1f806dd00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa210000f60 tx=0x7fa210009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:28.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.220+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa204076d90 con 0x7fa2140c29c0 2024-02-19T14:52:28.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.440+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa214074990 con 0x7fa2140c29c0 2024-02-19T14:52:28.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.441+0000 7fa209ffb640 1 -- 172.21.15.78:0/2812119643 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fa20407a450 con 0x7fa2140c29c0 2024-02-19T14:52:28.444 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa1f806b840 msgr2=0x7fa1f806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa1f806b840 0x7fa1f806dd00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa210000f60 tx=0x7fa210009040 comp rx=0 tx=0).stop 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 msgr2=0x7fa21414c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa21414c340 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa2040098c0 tx=0x7fa2040046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 shutdown_connections 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa1f806b840 0x7fa1f806dd00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2140c29c0 0x7fa21414c340 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2140c5be0 0x7fa21414c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 --2- 172.21.15.78:0/2812119643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2140c25e0 0x7fa2140bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:28.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 >> 172.21.15.78:0/2812119643 conn(0x7fa21406b270 msgr2=0x7fa2140c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:28.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 shutdown_connections 2024-02-19T14:52:28.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:28.443+0000 7fa21b3c2640 1 -- 172.21.15.78:0/2812119643 wait complete. 2024-02-19T14:52:28.456 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T14:52:28.456 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T14:52:28.457 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T14:52:28.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.457 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-02-19T14:52:28.472 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T14:52:28.472 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T14:52:28.472 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T14:52:28.472 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:28.472 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-02-19T14:52:28.472 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42667sec preferred_lft 42667sec 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-02-19T14:52:28.473 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:28.474 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T14:52:28.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T14:52:28.474 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:28.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.581 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T14:52:28.606 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-02-19T14:52:28.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.608 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-02-19T14:52:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-02-19T14:52:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T14:52:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T14:52:28.662 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T14:52:28.663 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T14:52:28.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.663 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:28.663 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:28.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T14:52:28.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T14:52:28.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T14:52:28.663 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:28.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.758 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-02-19T14:52:28.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.772 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-02-19T14:52:28.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.826 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:28.826 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:28.827 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T14:52:28.827 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-19T14:52:28.827 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:28.907 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-19T14:52:28.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-19T14:52:28.908 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:29.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.034 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-02-19T14:52:29.034 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-02-19T14:52:29.034 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-02-19T14:52:29.034 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T14:52:29.034 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-02-19T14:52:29.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T14:52:29.075 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T14:52:29.130 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T14:52:29.130 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T14:52:29.130 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T14:52:29.184 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-02-19T14:52:29.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.185 DEBUG:teuthology.orchestra.run.smithi078:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-19T14:52:29.270 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-19T14:52:29.270 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.150:3300,172.21.15.150:3301" 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with new device syntax: 0@7cde3b0a-2943-4d9b-b265-05e609d5a6b3.cephfs=/ 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.78:3300/172.21.15.150:3300/172.21.15.150:3301" will pass to kernel 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout: You just mounted a file system that supports labels which does not 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T14:52:29.382 INFO:teuthology.orchestra.run.smithi078.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T14:52:29.383 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T14:52:29.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T14:52:29.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.410 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T14:52:29.466 INFO:teuthology.orchestra.run.smithi078.stdout:client4738 2024-02-19T14:52:29.466 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:52:29.466 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:29.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.521 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T14:52:29.577 INFO:teuthology.orchestra.run.smithi078.stdout:client4738 2024-02-19T14:52:29.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.577 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T14:52:29.634 INFO:teuthology.orchestra.run.smithi078.stdout:client4738 2024-02-19T14:52:29.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:29.634 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7cde3b0a-2943-4d9b-b265-05e609d5a6b3.client4738/status') 2024-02-19T14:52:29.700 INFO:teuthology.orchestra.run.smithi078.stdout:instance: client.4738 (3)172.21.15.78:0/509434158 2024-02-19T14:52:29.700 INFO:teuthology.orchestra.run.smithi078.stdout:blocklisted: false 2024-02-19T14:52:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2024-02-19T14:52:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2024-02-19T14:52:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:69 bytes copied, 2.087e-05 s, 3.3 MB/s 2024-02-19T14:52:29.701 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-02-19T14:52:29.701 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T14:52:29.701 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T14:52:29.702 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-19T14:52:29.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.839+0000 7f94b4484640 1 Processor -- start 2024-02-19T14:52:29.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.839+0000 7f94b4484640 1 -- start start 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.839+0000 7f94b4484640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94ac14d210 0x7f94ac0c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.839+0000 7f94b4484640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac06ff90 con 0x7f94ac0719e0 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.839+0000 7f94b4484640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac156fc0 con 0x7f94ac0c3d90 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.839+0000 7f94b4484640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac070270 con 0x7f94ac14d210 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b1a1f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f94ac0719e0 0x7f94ac0c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51488/0 (socket says 172.21.15.78:51488) 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b1a1f640 1 -- 172.21.15.78:0/1717227291 learned_addr learned my addr 172.21.15.78:0/1717227291 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b2220640 1 --2- 172.21.15.78:0/1717227291 >> v2:172.21.15.150:3300/0 conn(0x7f94ac14d210 0x7f94ac0c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608179683 0 0) 0x7f94ac06ff90 con 0x7f94ac0719e0 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9498003520 con 0x7f94ac0719e0 2024-02-19T14:52:29.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644465225 0 0) 0x7f94ac156fc0 con 0x7f94ac0c3d90 2024-02-19T14:52:29.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac06ff90 con 0x7f94ac0c3d90 2024-02-19T14:52:29.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196785738 0 0) 0x7f9498003520 con 0x7f94ac0719e0 2024-02-19T14:52:29.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b2220640 1 -- 172.21.15.78:0/1717227291 >> v1:172.21.15.150:6789/0 conn(0x7f94ac0c3d90 legacy=0x7f94ac15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:29.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b2220640 1 -- 172.21.15.78:0/1717227291 >> v1:172.21.15.78:6789/0 conn(0x7f94ac0719e0 legacy=0x7f94ac0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:29.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.840+0000 7f94b2220640 1 -- 172.21.15.78:0/1717227291 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac1572a0 con 0x7f94ac14d210 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.841+0000 7f94b2220640 1 --2- 172.21.15.78:0/1717227291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac14d210 0x7f94ac0c4290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f94a80097a0 tx=0x7f94a802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78c0232db98c7967 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.841+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c001670 con 0x7f94ac14d210 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.841+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a802e050 con 0x7f94ac14d210 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.841+0000 7f94b121e640 1 -- 172.21.15.78:0/1717227291 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8030710 con 0x7f94ac14d210 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 -- 172.21.15.78:0/1717227291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac14d210 msgr2=0x7f94ac0c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 --2- 172.21.15.78:0/1717227291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac14d210 0x7f94ac0c4290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f94a80097a0 tx=0x7f94a802b790 comp rx=0 tx=0).stop 2024-02-19T14:52:29.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 -- 172.21.15.78:0/1717227291 shutdown_connections 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 --2- 172.21.15.78:0/1717227291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac14d210 0x7f94ac0c4290 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 -- 172.21.15.78:0/1717227291 >> 172.21.15.78:0/1717227291 conn(0x7f94ac0c5050 msgr2=0x7f94ac0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 -- 172.21.15.78:0/1717227291 shutdown_connections 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.842+0000 7f94b4484640 1 -- 172.21.15.78:0/1717227291 wait complete. 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 Processor -- start 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 -- start start 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 0x7f94ac155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c3d90 0x7f94ac154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94ac14d210 0x7f94ac154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0c3970 con 0x7f94ac0719e0 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1de6e0 con 0x7f94ac0c3d90 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.843+0000 7f94b4484640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1ebc00 con 0x7f94ac14d210 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 0x7f94ac155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:29.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 0x7f94ac155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53648/0 (socket says 172.21.15.78:53648) 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 -- 172.21.15.78:0/2547990518 learned_addr learned my addr 172.21.15.78:0/2547990518 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2a21640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94ac14d210 0x7f94ac154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b1a1f640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c3d90 0x7f94ac154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 -- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94ac14d210 msgr2=0x7f94ac154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94ac14d210 0x7f94ac154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 -- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c3d90 msgr2=0x7f94ac154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c3d90 0x7f94ac154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 -- 172.21.15.78:0/2547990518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8009400 con 0x7f94ac0719e0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b2220640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 0x7f94ac155bc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f94a802b7d0 tx=0x7f94a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8003ea0 con 0x7f94ac0719e0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8030710 con 0x7f94ac0719e0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8030aa0 con 0x7f94ac0719e0 2024-02-19T14:52:29.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac0c3660 con 0x7f94ac0719e0 2024-02-19T14:52:29.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.844+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac1eaff0 con 0x7f94ac0719e0 2024-02-19T14:52:29.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.845+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac074990 con 0x7f94ac0719e0 2024-02-19T14:52:29.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.846+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f94a8039050 con 0x7f94ac0719e0 2024-02-19T14:52:29.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.846+0000 7f94a2ffd640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f949006b840 0x7f949006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:29.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.846+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f94a80abf50 con 0x7f94ac0719e0 2024-02-19T14:52:29.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.846+0000 7f94b1a1f640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f949006b840 0x7f949006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:29.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.846+0000 7f94b1a1f640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f949006b840 0x7f949006dd00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94ac1e8ae0 tx=0x7f949c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:29.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:29.855+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f94a8076640 con 0x7f94ac0719e0 2024-02-19T14:52:30.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.070+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f94ac0c3100 con 0x7f94ac0719e0 2024-02-19T14:52:30.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.070+0000 7f94a2ffd640 1 -- 172.21.15.78:0/2547990518 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f94a804c070 con 0x7f94ac0719e0 2024-02-19T14:52:30.073 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f949006b840 msgr2=0x7f949006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f949006b840 0x7f949006dd00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f94ac1e8ae0 tx=0x7f949c009040 comp rx=0 tx=0).stop 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 msgr2=0x7f94ac155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 0x7f94ac155bc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f94a802b7d0 tx=0x7f94a80046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 shutdown_connections 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f949006b840 0x7f949006dd00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94ac14d210 0x7f94ac154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c3d90 0x7f94ac154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:30.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 --2- 172.21.15.78:0/2547990518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94ac0719e0 0x7f94ac155bc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:30.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 >> 172.21.15.78:0/2547990518 conn(0x7f94ac0c5050 msgr2=0x7f94ac0a94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:30.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 shutdown_connections 2024-02-19T14:52:30.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:30.072+0000 7f94b4484640 1 -- 172.21.15.78:0/2547990518 wait complete. 2024-02-19T14:52:30.085 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T14:52:30.086 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-19T14:52:30.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.086 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-02-19T14:52:30.105 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T14:52:30.105 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T14:52:30.105 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T14:52:30.105 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:30.105 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42657sec preferred_lft 42657sec 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-02-19T14:52:30.106 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-02-19T14:52:30.107 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T14:52:30.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T14:52:30.107 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-02-19T14:52:30.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.366 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T14:52:30.394 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-02-19T14:52:30.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.396 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-02-19T14:52:30.653 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-02-19T14:52:30.653 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T14:52:30.653 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T14:52:30.653 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T14:52:30.653 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T14:52:30.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.654 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-02-19T14:52:30.654 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-02-19T14:52:30.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T14:52:30.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T14:52:30.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T14:52:30.654 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-02-19T14:52:30.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.885 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-02-19T14:52:30.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.899 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-02-19T14:52:30.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:30.955 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-02-19T14:52:30.955 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-02-19T14:52:30.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T14:52:30.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-19T14:52:30.955 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-02-19T14:52:31.041 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-19T14:52:31.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.041 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-02-19T14:52:31.041 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-02-19T14:52:31.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-19T14:52:31.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T14:52:31.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-19T14:52:31.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-19T14:52:31.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-19T14:52:31.042 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-02-19T14:52:31.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.178 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-02-19T14:52:31.178 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-02-19T14:52:31.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-02-19T14:52:31.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T14:52:31.178 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-02-19T14:52:31.220 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T14:52:31.221 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-19T14:52:31.273 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-19T14:52:31.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T14:52:31.273 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-19T14:52:31.328 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-02-19T14:52:31.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-19T14:52:31.414 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-19T14:52:31.414 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-19T14:52:31.538 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-19T14:52:31.538 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-19T14:52:31.538 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.150:3300,172.21.15.150:3301" 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with new device syntax: 1@7cde3b0a-2943-4d9b-b265-05e609d5a6b3.cephfs=/ 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.78:3300/172.21.15.150:3300/172.21.15.150:3301" will pass to kernel 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T14:52:31.539 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T14:52:31.541 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T14:52:31.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-19T14:52:31.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.568 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T14:52:31.623 INFO:teuthology.orchestra.run.smithi150.stdout:client4565 2024-02-19T14:52:31.624 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:52:31.624 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:31.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.679 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T14:52:31.734 INFO:teuthology.orchestra.run.smithi150.stdout:client4565 2024-02-19T14:52:31.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.734 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T14:52:31.788 INFO:teuthology.orchestra.run.smithi150.stdout:client4565 2024-02-19T14:52:31.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:31.789 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7cde3b0a-2943-4d9b-b265-05e609d5a6b3.client4565/status') 2024-02-19T14:52:31.855 INFO:teuthology.orchestra.run.smithi150.stdout:instance: client.4565 (3)192.168.0.1:0/3332699686 2024-02-19T14:52:31.855 INFO:teuthology.orchestra.run.smithi150.stdout:blocklisted: false 2024-02-19T14:52:31.855 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records in 2024-02-19T14:52:31.855 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records out 2024-02-19T14:52:31.856 INFO:teuthology.orchestra.run.smithi150.stderr:69 bytes copied, 2.2049e-05 s, 3.1 MB/s 2024-02-19T14:52:31.857 INFO:teuthology.run_tasks:Running task exec... 2024-02-19T14:52:31.869 INFO:teuthology.task.exec:Executing custom commands... 2024-02-19T14:52:31.870 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:52:31.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/crash/warn_recent_interval 0' 2024-02-19T14:52:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.969+0000 7efc88f6c640 1 Processor -- start 2024-02-19T14:52:31.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.969+0000 7efc88f6c640 1 -- start start 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc88f6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc840732e0 0x7efc840c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc88f6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc840c1d20 0x7efc840c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc88f6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc840c2c90 con 0x7efc841526f0 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc88f6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc84156010 con 0x7efc840732e0 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc88f6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8406f8f0 con 0x7efc840c1d20 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc840732e0 0x7efc840c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc840732e0 0x7efc840c77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53650/0 (socket says 172.21.15.78:53650) 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 -- 172.21.15.78:0/3440430673 learned_addr learned my addr 172.21.15.78:0/3440430673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc82d9d640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc840c1d20 0x7efc840c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 -- 172.21.15.78:0/3440430673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc840c1d20 msgr2=0x7efc840c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc840c1d20 0x7efc840c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 -- 172.21.15.78:0/3440430673 >> v1:172.21.15.150:6789/0 conn(0x7efc841526f0 legacy=0x7efc8406fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc79d9b640 1 -- 172.21.15.78:0/3440430673 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc840c2ee0 con 0x7efc840732e0 2024-02-19T14:52:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.970+0000 7efc82d9d640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc840c1d20 0x7efc840c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.971+0000 7efc79d9b640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc840c77c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009870 tx=0x7efc6c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2422a7412d9b4427 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.971+0000 7efc81d9b640 1 -- 172.21.15.78:0/3440430673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c031740 con 0x7efc840732e0 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.971+0000 7efc81d9b640 1 -- 172.21.15.78:0/3440430673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c0318a0 con 0x7efc840732e0 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.971+0000 7efc81d9b640 1 -- 172.21.15.78:0/3440430673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c031c30 con 0x7efc840732e0 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 -- 172.21.15.78:0/3440430673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 msgr2=0x7efc840c77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc840c77c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009870 tx=0x7efc6c02b7e0 comp rx=0 tx=0).stop 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 -- 172.21.15.78:0/3440430673 shutdown_connections 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc840c1d20 0x7efc840c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 --2- 172.21.15.78:0/3440430673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc840c77c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 -- 172.21.15.78:0/3440430673 >> 172.21.15.78:0/3440430673 conn(0x7efc8406aed0 msgr2=0x7efc8406b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 -- 172.21.15.78:0/3440430673 shutdown_connections 2024-02-19T14:52:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.972+0000 7efc88f6c640 1 -- 172.21.15.78:0/3440430673 wait complete. 2024-02-19T14:52:31.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.973+0000 7efc88f6c640 1 Processor -- start 2024-02-19T14:52:31.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.973+0000 7efc88f6c640 1 -- start start 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc88f6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc841eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc88f6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc840c1d20 0x7efc841e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc88f6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc841526f0 0x7efc841e8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc88f6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84158b90 con 0x7efc840732e0 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc88f6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84158e70 con 0x7efc840c1d20 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc88f6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc84159150 con 0x7efc841526f0 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc8259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc841eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc8259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc841eabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53658/0 (socket says 172.21.15.78:53658) 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc8259c640 1 -- 172.21.15.78:0/2302435009 learned_addr learned my addr 172.21.15.78:0/2302435009 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc82d9d640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc841526f0 0x7efc841e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:31.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.974+0000 7efc79d9b640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc840c1d20 0x7efc841e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc8259c640 1 -- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc841526f0 msgr2=0x7efc841e8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc8259c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc841526f0 0x7efc841e8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc8259c640 1 -- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc840c1d20 msgr2=0x7efc841e7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc8259c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc840c1d20 0x7efc841e7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc8259c640 1 -- 172.21.15.78:0/2302435009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0094d0 con 0x7efc840732e0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc8259c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc841eabb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efc7400c430 tx=0x7efc7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74011790 con 0x7efc840732e0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc740118f0 con 0x7efc840732e0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc840a85b0 con 0x7efc840732e0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74011ce0 con 0x7efc840732e0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.975+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc841eb5a0 con 0x7efc840732e0 2024-02-19T14:52:31.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.976+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc84156fa0 con 0x7efc840732e0 2024-02-19T14:52:31.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.977+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efc7401c460 con 0x7efc840732e0 2024-02-19T14:52:31.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.977+0000 7efc7b7fe640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc6406b810 0x7efc6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:31.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.977+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efc7408a310 con 0x7efc840732e0 2024-02-19T14:52:31.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.977+0000 7efc79d9b640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc6406b810 0x7efc6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:31.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.978+0000 7efc79d9b640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc6406b810 0x7efc6406dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efc6c035000 tx=0x7efc6c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:31.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:31.982+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efc74054b30 con 0x7efc840732e0 2024-02-19T14:52:32.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.156+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/crash/warn_recent_interval}] v 0) v1 -- 0x7efc8404f250 con 0x7efc840732e0 2024-02-19T14:52:32.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.159+0000 7efc7b7fe640 1 -- 172.21.15.78:0/2302435009 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/crash/warn_recent_interval}]=0 v10)=0 v10) v1 ==== 128+0+0 (secure 0 0 0) 0x7efc740581f0 con 0x7efc840732e0 2024-02-19T14:52:32.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc6406b810 msgr2=0x7efc6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc6406b810 0x7efc6406dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efc6c035000 tx=0x7efc6c008030 comp rx=0 tx=0).stop 2024-02-19T14:52:32.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 msgr2=0x7efc841eabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc841eabb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efc7400c430 tx=0x7efc7400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 shutdown_connections 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc6406b810 0x7efc6406dcd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc841526f0 0x7efc841e8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc840c1d20 0x7efc841e7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 --2- 172.21.15.78:0/2302435009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc840732e0 0x7efc841eabb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.161+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 >> 172.21.15.78:0/2302435009 conn(0x7efc8406aed0 msgr2=0x7efc8406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.162+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 shutdown_connections 2024-02-19T14:52:32.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.162+0000 7efc88f6c640 1 -- 172.21.15.78:0/2302435009 wait complete. 2024-02-19T14:52:32.175 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-02-19T14:52:32.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:32.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.294+0000 7f41ecd0d640 1 Processor -- start 2024-02-19T14:52:32.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.294+0000 7f41ecd0d640 1 -- start start 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41ecd0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f41e80bdf20 0x7f41e80ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41ecd0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41e80c55f0 0x7f41e815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41ecd0d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e8158c40 con 0x7f41e80ac390 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41ecd0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e80c8de0 con 0x7f41e80bdf20 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41ecd0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e80c5a30 con 0x7f41e80c55f0 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e5d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f41e80bdf20 0x7f41e80ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f41e80ac390 0x7f41e8158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51494/0 (socket says 172.21.15.78:51494) 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e659c640 1 -- 172.21.15.78:0/3532297907 learned_addr learned my addr 172.21.15.78:0/3532297907 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e6d9d640 1 --2- 172.21.15.78:0/3532297907 >> v2:172.21.15.150:3300/0 conn(0x7f41e80c55f0 0x7f41e815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e5d9b640 1 -- 172.21.15.78:0/3532297907 >> v2:172.21.15.150:3300/0 conn(0x7f41e80c55f0 msgr2=0x7f41e815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e5d9b640 1 --2- 172.21.15.78:0/3532297907 >> v2:172.21.15.150:3300/0 conn(0x7f41e80c55f0 0x7f41e815f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e5d9b640 1 -- 172.21.15.78:0/3532297907 >> v1:172.21.15.78:6789/0 conn(0x7f41e80ac390 legacy=0x7f41e8158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.295+0000 7f41e559a640 1 -- 172.21.15.78:0/3532297907 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266639614 0 0) 0x7f41e8158c40 con 0x7f41e80ac390 2024-02-19T14:52:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.296+0000 7f41e5d9b640 1 -- 172.21.15.78:0/3532297907 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8155a60 con 0x7f41e80bdf20 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.296+0000 7f41e5d9b640 1 --2- 172.21.15.78:0/3532297907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80bdf20 0x7f41e80ac770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f41e806d9a0 tx=0x7f41dc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916ecbfb3c6b09ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.296+0000 7f41e559a640 1 -- 172.21.15.78:0/3532297907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc031710 con 0x7f41e80bdf20 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.296+0000 7f41e559a640 1 -- 172.21.15.78:0/3532297907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc02f050 con 0x7f41e80bdf20 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.296+0000 7f41e559a640 1 -- 172.21.15.78:0/3532297907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc031a90 con 0x7f41e80bdf20 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 -- 172.21.15.78:0/3532297907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80bdf20 msgr2=0x7f41e80ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/3532297907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80bdf20 0x7f41e80ac770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f41e806d9a0 tx=0x7f41dc02b7b0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 -- 172.21.15.78:0/3532297907 shutdown_connections 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/3532297907 >> v2:172.21.15.150:3300/0 conn(0x7f41e80c55f0 0x7f41e815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/3532297907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80bdf20 0x7f41e80ac770 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 -- 172.21.15.78:0/3532297907 >> 172.21.15.78:0/3532297907 conn(0x7f41e80ca890 msgr2=0x7f41e80caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 -- 172.21.15.78:0/3532297907 shutdown_connections 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.297+0000 7f41ecd0d640 1 -- 172.21.15.78:0/3532297907 wait complete. 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.298+0000 7f41ecd0d640 1 Processor -- start 2024-02-19T14:52:32.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.298+0000 7f41ecd0d640 1 -- start start 2024-02-19T14:52:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41ecd0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 0x7f41e80c0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41ecd0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e80bdf20 0x7f41e8157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41ecd0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e80c55f0 0x7f41e80c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41ecd0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80bf0e0 con 0x7f41e80ac390 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41ecd0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80bf570 con 0x7f41e80bdf20 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41ecd0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80bedb0 con 0x7f41e80c55f0 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41e659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 0x7f41e80c0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41e659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 0x7f41e80c0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53674/0 (socket says 172.21.15.78:53674) 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.299+0000 7f41e659c640 1 -- 172.21.15.78:0/1805519534 learned_addr learned my addr 172.21.15.78:0/1805519534 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e5d9b640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e80bdf20 0x7f41e8157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e6d9d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e80c55f0 0x7f41e80c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e659c640 1 -- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e80c55f0 msgr2=0x7f41e80c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e659c640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e80c55f0 0x7f41e80c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e659c640 1 -- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e80bdf20 msgr2=0x7f41e8157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e659c640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e80bdf20 0x7f41e8157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e659c640 1 -- 172.21.15.78:0/1805519534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc009400 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e5d9b640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e80bdf20 0x7f41e8157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41e659c640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 0x7f41e80c0b50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c890 tx=0x7f41e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e000a640 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e000a7c0 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e000ab70 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e80bf960 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.300+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e81581a0 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.301+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e8074990 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.301+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f41e000acd0 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.302+0000 7f41d6ffd640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41c806b840 0x7f41c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.302+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f41e0089d30 con 0x7f41e80ac390 2024-02-19T14:52:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.302+0000 7f41e5d9b640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41c806b840 0x7f41c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.302+0000 7f41e5d9b640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41c806b840 0x7f41c806dd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f41dc035000 tx=0x7f41dc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:32.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.306+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f41e00544a0 con 0x7f41e80ac390 2024-02-19T14:52:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.603+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41e806da00 con 0x7f41e80ac390 2024-02-19T14:52:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.605+0000 7f41d6ffd640 1 -- 172.21.15.78:0/1805519534 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f41e0018020 con 0x7f41e80ac390 2024-02-19T14:52:32.607 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:32.607 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41c806b840 msgr2=0x7f41c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41c806b840 0x7f41c806dd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f41dc035000 tx=0x7f41dc008030 comp rx=0 tx=0).stop 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 msgr2=0x7f41e80c0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 0x7f41e80c0b50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c890 tx=0x7f41e000cd50 comp rx=0 tx=0).stop 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 shutdown_connections 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41c806b840 0x7f41c806dd00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e80c55f0 0x7f41e80c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e80bdf20 0x7f41e8157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 --2- 172.21.15.78:0/1805519534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e80ac390 0x7f41e80c0b50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 >> 172.21.15.78:0/1805519534 conn(0x7f41e80ca890 msgr2=0x7f41e80c9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 shutdown_connections 2024-02-19T14:52:32.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.607+0000 7f41ecd0d640 1 -- 172.21.15.78:0/1805519534 wait complete. 2024-02-19T14:52:32.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:32.621 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:32.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 Processor -- start 2024-02-19T14:52:32.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 -- start start 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b8c14d210 0x7f4b8c0c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b8c0719e0 0x7f4b8c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c06ff90 con 0x7f4b8c0c3d90 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c070270 con 0x7f4b8c14d210 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.726+0000 7f4b9321c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c0c3970 con 0x7f4b8c0719e0 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b90fb8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b8c14d210 0x7f4b8c0c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b917b9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4b8c0c3d90 0x7f4b8c152ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51500/0 (socket says 172.21.15.78:51500) 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b90fb8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b8c14d210 0x7f4b8c0c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53690/0 (socket says 172.21.15.78:53690) 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b917b9640 1 -- 172.21.15.78:0/684028257 learned_addr learned my addr 172.21.15.78:0/684028257 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b8bfff640 1 --2- 172.21.15.78:0/684028257 >> v2:172.21.15.150:3300/0 conn(0x7f4b8c0719e0 0x7f4b8c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b90fb8640 1 -- 172.21.15.78:0/684028257 >> v2:172.21.15.150:3300/0 conn(0x7f4b8c0719e0 msgr2=0x7f4b8c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b90fb8640 1 --2- 172.21.15.78:0/684028257 >> v2:172.21.15.150:3300/0 conn(0x7f4b8c0719e0 0x7f4b8c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b90fb8640 1 -- 172.21.15.78:0/684028257 >> v1:172.21.15.78:6789/0 conn(0x7f4b8c0c3d90 legacy=0x7f4b8c152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b8b7fe640 1 -- 172.21.15.78:0/684028257 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692228208 0 0) 0x7f4b8c06ff90 con 0x7f4b8c0c3d90 2024-02-19T14:52:32.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.727+0000 7f4b90fb8640 1 -- 172.21.15.78:0/684028257 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c156fc0 con 0x7f4b8c14d210 2024-02-19T14:52:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b90fb8640 1 --2- 172.21.15.78:0/684028257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c14d210 0x7f4b8c0c4290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ab3a0 tx=0x7f4b8002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7362c89aeaeb1b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b8b7fe640 1 -- 172.21.15.78:0/684028257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8002e070 con 0x7f4b8c14d210 2024-02-19T14:52:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b8b7fe640 1 -- 172.21.15.78:0/684028257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b800315a0 con 0x7f4b8c14d210 2024-02-19T14:52:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b8b7fe640 1 -- 172.21.15.78:0/684028257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80031930 con 0x7f4b8c14d210 2024-02-19T14:52:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 -- 172.21.15.78:0/684028257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c14d210 msgr2=0x7f4b8c0c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 --2- 172.21.15.78:0/684028257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c14d210 0x7f4b8c0c4290 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ab3a0 tx=0x7f4b8002b840 comp rx=0 tx=0).stop 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 -- 172.21.15.78:0/684028257 shutdown_connections 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 --2- 172.21.15.78:0/684028257 >> v2:172.21.15.150:3300/0 conn(0x7f4b8c0719e0 0x7f4b8c0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 --2- 172.21.15.78:0/684028257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c14d210 0x7f4b8c0c4290 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 -- 172.21.15.78:0/684028257 >> 172.21.15.78:0/684028257 conn(0x7f4b8c0c5050 msgr2=0x7f4b8c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 -- 172.21.15.78:0/684028257 shutdown_connections 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.728+0000 7f4b9321c640 1 -- 172.21.15.78:0/684028257 wait complete. 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.729+0000 7f4b9321c640 1 Processor -- start 2024-02-19T14:52:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 -- start start 2024-02-19T14:52:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 0x7f4b8c0a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b8c0c3d90 0x7f4b8c0ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b8c14d210 0x7f4b8c0ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0ac460 con 0x7f4b8c0719e0 2024-02-19T14:52:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0ac180 con 0x7f4b8c0c3d90 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b9321c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1453b0 con 0x7f4b8c14d210 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b90fb8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 0x7f4b8c0a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b90fb8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 0x7f4b8c0a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53706/0 (socket says 172.21.15.78:53706) 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.730+0000 7f4b90fb8640 1 -- 172.21.15.78:0/1722803178 learned_addr learned my addr 172.21.15.78:0/1722803178 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b8bfff640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b8c0c3d90 0x7f4b8c0ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b917b9640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b8c14d210 0x7f4b8c0ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b90fb8640 1 -- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b8c14d210 msgr2=0x7f4b8c0ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b90fb8640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b8c14d210 0x7f4b8c0ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b90fb8640 1 -- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b8c0c3d90 msgr2=0x7f4b8c0ac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b90fb8640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b8c0c3d90 0x7f4b8c0ac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b90fb8640 1 -- 172.21.15.78:0/1722803178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80009400 con 0x7f4b8c0719e0 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b90fb8640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 0x7f4b8c0a9660 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4b800098c0 tx=0x7f4b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80004230 con 0x7f4b8c0719e0 2024-02-19T14:52:32.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80031f00 con 0x7f4b8c0719e0 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c0c3660 con 0x7f4b8c0719e0 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80032640 con 0x7f4b8c0719e0 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.731+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c143da0 con 0x7f4b8c0719e0 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.732+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c074990 con 0x7f4b8c0719e0 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.733+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b800327a0 con 0x7f4b8c0719e0 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.733+0000 7f4b897fa640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7006b810 0x7f4b7006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:32.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.733+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4b800ac470 con 0x7f4b8c0719e0 2024-02-19T14:52:32.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.733+0000 7f4b8bfff640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7006b810 0x7f4b7006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:32.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.734+0000 7f4b8bfff640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7006b810 0x7f4b7006dcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ab390 tx=0x7f4b78008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:32.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.738+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b80076db0 con 0x7f4b8c0719e0 2024-02-19T14:52:32.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.915+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b8c0c3100 con 0x7f4b8c0719e0 2024-02-19T14:52:32.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.915+0000 7f4b897fa640 1 -- 172.21.15.78:0/1722803178 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b8007a470 con 0x7f4b8c0719e0 2024-02-19T14:52:32.918 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7006b810 msgr2=0x7f4b7006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7006b810 0x7f4b7006dcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ab390 tx=0x7f4b78008040 comp rx=0 tx=0).stop 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 msgr2=0x7f4b8c0a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 0x7f4b8c0a9660 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4b800098c0 tx=0x7f4b800046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 shutdown_connections 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7006b810 0x7f4b7006dcd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b8c14d210 0x7f4b8c0ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b8c0c3d90 0x7f4b8c0ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 --2- 172.21.15.78:0/1722803178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8c0719e0 0x7f4b8c0a9660 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 >> 172.21.15.78:0/1722803178 conn(0x7f4b8c0c5050 msgr2=0x7f4b8c14ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 shutdown_connections 2024-02-19T14:52:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:32.918+0000 7f4b9321c640 1 -- 172.21.15.78:0/1722803178 wait complete. 2024-02-19T14:52:32.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.034+0000 7f7bf725b640 1 Processor -- start 2024-02-19T14:52:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.034+0000 7f7bf725b640 1 -- start start 2024-02-19T14:52:33.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7bf725b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 0x7f7bf00c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7bf725b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf00c6460 con 0x7f7bf0155070 2024-02-19T14:52:33.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7bf725b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf0158720 con 0x7f7bf01525c0 2024-02-19T14:52:33.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7bf725b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00c1f50 con 0x7f7bf00c6870 2024-02-19T14:52:33.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7bf57f8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7bf0155070 0x7f7bf0155500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51508/0 (socket says 172.21.15.78:51508) 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7bf57f8640 1 -- 172.21.15.78:0/297743713 learned_addr learned my addr 172.21.15.78:0/297743713 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7be7fff640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 0x7f7bf00c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131601807 0 0) 0x7f7bf00c6460 con 0x7f7bf0155070 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.035+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd8003520 con 0x7f7bf0155070 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3035856952 0 0) 0x7f7bd8003520 con 0x7f7bf0155070 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 msgr2=0x7f7bf00c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 0x7f7bf00c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 >> v1:172.21.15.150:6789/0 conn(0x7f7bf01525c0 legacy=0x7f7bf006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf0158a00 con 0x7f7bf0155070 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f7be8001670 con 0x7f7bf0155070 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 >> v1:172.21.15.78:6789/0 conn(0x7f7bf0155070 legacy=0x7f7bf0155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 0x7f7bd80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bd8003f20 0x7f7bd80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd8004d00 0x7f7bd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00c1f50 con 0x7f7bd8004d00 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd800bb10 con 0x7f7bd8003f20 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd800bcd0 con 0x7f7bf00c6870 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd8004d00 0x7f7bd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf4ff7640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 0x7f7bd80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7be7fff640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bd8003f20 0x7f7bd80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf57f8640 1 -- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 msgr2=0x7f7bd80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf00c6870 0x7f7bd80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf57f8640 1 -- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bd8003f20 msgr2=0x7f7bd80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bd8003f20 0x7f7bd80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.036+0000 7f7bf57f8640 1 -- 172.21.15.78:0/297743713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8003520 con 0x7f7bd8004d00 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.037+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd8004d00 0x7f7bd8005180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be8001050 tx=0x7f7bd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb59f3a52fb4413 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.037+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8005ad0 con 0x7f7bd8004d00 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.037+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8005c30 con 0x7f7bd8004d00 2024-02-19T14:52:33.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.037+0000 7f7be77fe640 1 -- 172.21.15.78:0/297743713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be802c890 con 0x7f7bd8004d00 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.037+0000 7f7bf725b640 1 -- 172.21.15.78:0/297743713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd8004d00 msgr2=0x7f7bd8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.037+0000 7f7bf725b640 1 --2- 172.21.15.78:0/297743713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bd8004d00 0x7f7bd8005180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be8001050 tx=0x7f7bd80057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.038+0000 7f7bf57f8640 1 -- 172.21.15.78:0/297743713 reap_dead start 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.038+0000 7f7bf725b640 1 -- 172.21.15.78:0/297743713 shutdown_connections 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.038+0000 7f7bf725b640 1 -- 172.21.15.78:0/297743713 >> 172.21.15.78:0/297743713 conn(0x7f7bf00cab20 msgr2=0x7f7bf00caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.038+0000 7f7bf725b640 1 -- 172.21.15.78:0/297743713 shutdown_connections 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.038+0000 7f7bf725b640 1 -- 172.21.15.78:0/297743713 wait complete. 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 Processor -- start 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 -- start start 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf01502d0 0x7f7bf01eadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bf014b670 0x7f7bf01576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 0x7f7bf0149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01593a0 con 0x7f7bf0157c20 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0156460 con 0x7f7bf014b670 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.039+0000 7f7bf725b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0156640 con 0x7f7bf01502d0 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 0x7f7bf0149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 0x7f7bf0149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53720/0 (socket says 172.21.15.78:53720) 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 -- 172.21.15.78:0/3086589388 learned_addr learned my addr 172.21.15.78:0/3086589388 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7be7fff640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bf014b670 0x7f7bf01576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf4ff7640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf01502d0 0x7f7bf01eadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 -- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf01502d0 msgr2=0x7f7bf01eadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf01502d0 0x7f7bf01eadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 -- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bf014b670 msgr2=0x7f7bf01576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bf014b670 0x7f7bf01576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 -- 172.21.15.78:0/3086589388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf0156920 con 0x7f7bf0157c20 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf57f8640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 0x7f7bf0149fb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7be800c300 tx=0x7f7be8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be803b070 con 0x7f7bf0157c20 2024-02-19T14:52:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8001c40 con 0x7f7bf0157c20 2024-02-19T14:52:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf0156c00 con 0x7f7bf0157c20 2024-02-19T14:52:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be80048a0 con 0x7f7bf0157c20 2024-02-19T14:52:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.040+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf01571c0 con 0x7f7bf0157c20 2024-02-19T14:52:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.041+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf01573e0 con 0x7f7bf0157c20 2024-02-19T14:52:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.042+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7be8032070 con 0x7f7bf0157c20 2024-02-19T14:52:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.042+0000 7f7be57fa640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bdc06b840 0x7f7bdc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.042+0000 7f7bf4ff7640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bdc06b840 0x7f7bdc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.042+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7be80aafe0 con 0x7f7bf0157c20 2024-02-19T14:52:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.043+0000 7f7bf4ff7640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bdc06b840 0x7f7bdc06dd00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7bec003fb0 tx=0x7f7bec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.047+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7be80791f0 con 0x7f7bf0157c20 2024-02-19T14:52:33.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.225+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7bf0074990 con 0x7f7bf0157c20 2024-02-19T14:52:33.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.225+0000 7f7be57fa640 1 -- 172.21.15.78:0/3086589388 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7be8078d80 con 0x7f7bf0157c20 2024-02-19T14:52:33.228 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bdc06b840 msgr2=0x7f7bdc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bdc06b840 0x7f7bdc06dd00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7bec003fb0 tx=0x7f7bec009040 comp rx=0 tx=0).stop 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 msgr2=0x7f7bf0149fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 0x7f7bf0149fb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7be800c300 tx=0x7f7be8000fe0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 shutdown_connections 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bdc06b840 0x7f7bdc06dd00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bf0157c20 0x7f7bf0149fb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bf014b670 0x7f7bf01576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 --2- 172.21.15.78:0/3086589388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bf01502d0 0x7f7bf01eadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 >> 172.21.15.78:0/3086589388 conn(0x7f7bf00cab20 msgr2=0x7f7bf00c0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 shutdown_connections 2024-02-19T14:52:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.227+0000 7f7bf725b640 1 -- 172.21.15.78:0/3086589388 wait complete. 2024-02-19T14:52:33.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:33.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.341+0000 7f440daa4640 1 Processor -- start 2024-02-19T14:52:33.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.341+0000 7f440daa4640 1 -- start start 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f440daa4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f440814cd50 0x7f44080c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f440daa4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 0x7f44080c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f440daa4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44081563b0 con 0x7f44080c7b70 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f440daa4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4408153cd0 con 0x7f440814cd50 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f440daa4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440806b6e0 con 0x7f44081537a0 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f4407fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f44080c7b70 0x7f4408156860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51522/0 (socket says 172.21.15.78:51522) 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.342+0000 7f4407fff640 1 -- 172.21.15.78:0/2809136485 learned_addr learned my addr 172.21.15.78:0/2809136485 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078467804 0 0) 0x7f44081563b0 con 0x7f44080c7b70 2024-02-19T14:52:33.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f4406ffd640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 0x7f44080c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44077fe640 1 --2- 172.21.15.78:0/2809136485 >> v2:172.21.15.150:3300/0 conn(0x7f440814cd50 0x7f44080c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f0003520 con 0x7f44080c7b70 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358585046 0 0) 0x7f43f0003520 con 0x7f44080c7b70 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 msgr2=0x7f44080c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 0x7f44080c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 >> v2:172.21.15.150:3300/0 conn(0x7f440814cd50 msgr2=0x7f44080c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 --2- 172.21.15.78:0/2809136485 >> v2:172.21.15.150:3300/0 conn(0x7f440814cd50 0x7f44080c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440806b980 con 0x7f44080c7b70 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f44000019f0 con 0x7f44080c7b70 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 >> v1:172.21.15.78:6789/0 conn(0x7f44080c7b70 legacy=0x7f4408156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 0x7f43f0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43f0003ef0 0x7f43f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43f0004d00 0x7f43f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4408153cd0 con 0x7f43f0003ef0 2024-02-19T14:52:33.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440806b6e0 con 0x7f43f0004d00 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f000ba60 con 0x7f44081537a0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f4406ffd640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43f0003ef0 0x7f43f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f4407fff640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43f0004d00 0x7f43f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f44077fe640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 0x7f43f0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f4406ffd640 1 -- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 msgr2=0x7f43f0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f4406ffd640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44081537a0 0x7f43f0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.343+0000 7f4406ffd640 1 -- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43f0004d00 msgr2=0x7f43f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f4406ffd640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43f0004d00 0x7f43f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f4406ffd640 1 -- 172.21.15.78:0/2809136485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0003520 con 0x7f43f0003ef0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f4407fff640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43f0004d00 0x7f43f0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f4406ffd640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43f0003ef0 0x7f43f00047c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009a90 tx=0x7f43f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2793ff862cc0fb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f802c680 con 0x7f43f0003ef0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f802c7e0 con 0x7f43f0003ef0 2024-02-19T14:52:33.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.344+0000 7f44067fc640 1 -- 172.21.15.78:0/2809136485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f802cb70 con 0x7f43f0003ef0 2024-02-19T14:52:33.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f440daa4640 1 -- 172.21.15.78:0/2809136485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43f0003ef0 msgr2=0x7f43f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f440daa4640 1 --2- 172.21.15.78:0/2809136485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43f0003ef0 0x7f43f00047c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009a90 tx=0x7f43f00057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f4407fff640 1 -- 172.21.15.78:0/2809136485 reap_dead start 2024-02-19T14:52:33.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f440daa4640 1 -- 172.21.15.78:0/2809136485 shutdown_connections 2024-02-19T14:52:33.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f440daa4640 1 -- 172.21.15.78:0/2809136485 >> 172.21.15.78:0/2809136485 conn(0x7f4408066e70 msgr2=0x7f44080a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f440daa4640 1 -- 172.21.15.78:0/2809136485 shutdown_connections 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.345+0000 7f440daa4640 1 -- 172.21.15.78:0/2809136485 wait complete. 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.347+0000 7f440daa4640 1 Processor -- start 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.347+0000 7f440daa4640 1 -- start start 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f440daa4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43f0003ef0 0x7f4408154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f440daa4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44080c1fa0 0x7f44080c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f440daa4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 0x7f44080c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f440daa4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44081e9340 con 0x7f4408154110 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f440daa4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4408155e80 con 0x7f44080c1fa0 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f440daa4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4408155b50 con 0x7f43f0003ef0 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 0x7f44080c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 0x7f44080c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53738/0 (socket says 172.21.15.78:53738) 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 -- 172.21.15.78:0/3278326170 learned_addr learned my addr 172.21.15.78:0/3278326170 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f44077fe640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43f0003ef0 0x7f4408154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4406ffd640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44080c1fa0 0x7f44080c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 -- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43f0003ef0 msgr2=0x7f4408154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43f0003ef0 0x7f4408154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 -- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44080c1fa0 msgr2=0x7f44080c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44080c1fa0 0x7f44080c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.348+0000 7f4407fff640 1 -- 172.21.15.78:0/3278326170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44081e9530 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.349+0000 7f4407fff640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 0x7f44080c2e90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4400001330 tx=0x7f440000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.349+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440000eb90 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.349+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440000ed10 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.349+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400013660 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.349+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44081e9810 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.349+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44080c33d0 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.350+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44080c75b0 con 0x7f4408154110 2024-02-19T14:52:33.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.350+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f44000137c0 con 0x7f4408154110 2024-02-19T14:52:33.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.350+0000 7f43effff640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43f406b840 0x7f43f406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.351+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f440008a690 con 0x7f4408154110 2024-02-19T14:52:33.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.351+0000 7f44077fe640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43f406b840 0x7f43f406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.351+0000 7f44077fe640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43f406b840 0x7f43f406dd00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44080a8f00 tx=0x7f43fc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.356+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4400054e00 con 0x7f4408154110 2024-02-19T14:52:33.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.534+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44080c35c0 con 0x7f4408154110 2024-02-19T14:52:33.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.534+0000 7f43effff640 1 -- 172.21.15.78:0/3278326170 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44000584c0 con 0x7f4408154110 2024-02-19T14:52:33.537 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43f406b840 msgr2=0x7f43f406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43f406b840 0x7f43f406dd00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44080a8f00 tx=0x7f43fc009190 comp rx=0 tx=0).stop 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 msgr2=0x7f44080c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 0x7f44080c2e90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4400001330 tx=0x7f440000e530 comp rx=0 tx=0).stop 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 shutdown_connections 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43f406b840 0x7f43f406dd00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4408154110 0x7f44080c2e90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44080c1fa0 0x7f44080c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 --2- 172.21.15.78:0/3278326170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43f0003ef0 0x7f4408154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 >> 172.21.15.78:0/3278326170 conn(0x7f4408066e70 msgr2=0x7f440814b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:33.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 shutdown_connections 2024-02-19T14:52:33.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.536+0000 7f440daa4640 1 -- 172.21.15.78:0/3278326170 wait complete. 2024-02-19T14:52:33.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.654+0000 7fc08657a640 1 Processor -- start 2024-02-19T14:52:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.654+0000 7fc08657a640 1 -- start start 2024-02-19T14:52:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc08657a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc080152700 0x7fc0800c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc08657a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc0800c8e00 0x7fc08015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc08657a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0801551f0 con 0x7fc0800c5730 2024-02-19T14:52:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc08657a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc080155490 con 0x7fc080152700 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc08657a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc080155770 con 0x7fc0800c8e00 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc07ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc080152700 0x7fc0800c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc07f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc0800c5730 0x7fc08006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51532/0 (socket says 172.21.15.78:51532) 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc07f7fe640 1 -- 172.21.15.78:0/3817607890 learned_addr learned my addr 172.21.15.78:0/3817607890 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.655+0000 7fc084b17640 1 --2- 172.21.15.78:0/3817607890 >> v2:172.21.15.150:3300/0 conn(0x7fc0800c8e00 0x7fc08015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc07ffff640 1 -- 172.21.15.78:0/3817607890 >> v2:172.21.15.150:3300/0 conn(0x7fc0800c8e00 msgr2=0x7fc08015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc07ffff640 1 --2- 172.21.15.78:0/3817607890 >> v2:172.21.15.150:3300/0 conn(0x7fc0800c8e00 0x7fc08015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc07ffff640 1 -- 172.21.15.78:0/3817607890 >> v1:172.21.15.78:6789/0 conn(0x7fc0800c5730 legacy=0x7fc08006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc07ffff640 1 -- 172.21.15.78:0/3817607890 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080155a50 con 0x7fc080152700 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc07effd640 1 -- 172.21.15.78:0/3817607890 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583839148 0 0) 0x7fc0801551f0 con 0x7fc0800c5730 2024-02-19T14:52:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc084b17640 1 --2- 172.21.15.78:0/3817607890 >> v2:172.21.15.150:3300/0 conn(0x7fc0800c8e00 0x7fc08015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.656+0000 7fc07ffff640 1 --2- 172.21.15.78:0/3817607890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800c51f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc0801596f0 tx=0x7fc07002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d58fd7c6571516d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc07effd640 1 -- 172.21.15.78:0/3817607890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07002e070 con 0x7fc080152700 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc07effd640 1 -- 172.21.15.78:0/3817607890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0700315a0 con 0x7fc080152700 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc07effd640 1 -- 172.21.15.78:0/3817607890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070031930 con 0x7fc080152700 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 -- 172.21.15.78:0/3817607890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 msgr2=0x7fc0800c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 --2- 172.21.15.78:0/3817607890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800c51f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc0801596f0 tx=0x7fc07002b840 comp rx=0 tx=0).stop 2024-02-19T14:52:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 -- 172.21.15.78:0/3817607890 shutdown_connections 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 --2- 172.21.15.78:0/3817607890 >> v2:172.21.15.150:3300/0 conn(0x7fc0800c8e00 0x7fc08015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 --2- 172.21.15.78:0/3817607890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800c51f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 -- 172.21.15.78:0/3817607890 >> 172.21.15.78:0/3817607890 conn(0x7fc0800cac90 msgr2=0x7fc0800cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 -- 172.21.15.78:0/3817607890 shutdown_connections 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.657+0000 7fc08657a640 1 -- 172.21.15.78:0/3817607890 wait complete. 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.658+0000 7fc08657a640 1 Processor -- start 2024-02-19T14:52:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.658+0000 7fc08657a640 1 -- start start 2024-02-19T14:52:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc08657a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0800c5730 0x7fc0801edc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc08657a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc0800c8e00 0x7fc0801ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc08657a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc08657a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800afae0 con 0x7fc080152700 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc08657a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc080151bf0 con 0x7fc0800c5730 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc08657a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800af720 con 0x7fc0800c8e00 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc084b17640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc084b17640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53768/0 (socket says 172.21.15.78:53768) 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc084b17640 1 -- 172.21.15.78:0/3014360813 learned_addr learned my addr 172.21.15.78:0/3014360813 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc07f7fe640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc0800c8e00 0x7fc0801ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.659+0000 7fc07ffff640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0800c5730 0x7fc0801edc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc084b17640 1 -- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc0800c8e00 msgr2=0x7fc0801ee150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc084b17640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc0800c8e00 0x7fc0801ee150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc084b17640 1 -- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0800c5730 msgr2=0x7fc0801edc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc084b17640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0800c5730 0x7fc0801edc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc084b17640 1 -- 172.21.15.78:0/3014360813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc070009400 con 0x7fc080152700 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc084b17640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800ade70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc078009fa0 tx=0x7fc07800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078003b90 con 0x7fc080152700 2024-02-19T14:52:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07800f040 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078004030 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0801ee690 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.660+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0800c4700 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.661+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0800c4920 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.661+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc078004190 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.662+0000 7fc07cff9640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc06006b810 0x7fc06006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.662+0000 7fc07ffff640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc06006b810 0x7fc06006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.662+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc07808a7a0 con 0x7fc080152700 2024-02-19T14:52:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.662+0000 7fc07ffff640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc06006b810 0x7fc06006dcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc070036000 tx=0x7fc07003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.666+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc078054fc0 con 0x7fc080152700 2024-02-19T14:52:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.872+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc080074990 con 0x7fc080152700 2024-02-19T14:52:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.873+0000 7fc07cff9640 1 -- 172.21.15.78:0/3014360813 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc078058680 con 0x7fc080152700 2024-02-19T14:52:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc06006b810 msgr2=0x7fc06006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc06006b810 0x7fc06006dcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc070036000 tx=0x7fc07003d040 comp rx=0 tx=0).stop 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 msgr2=0x7fc0800ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800ade70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc078009fa0 tx=0x7fc07800d560 comp rx=0 tx=0).stop 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 shutdown_connections 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc06006b810 0x7fc06006dcd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc080152700 0x7fc0800ade70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc0800c8e00 0x7fc0801ee150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 --2- 172.21.15.78:0/3014360813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0800c5730 0x7fc0801edc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 >> 172.21.15.78:0/3014360813 conn(0x7fc0800cac90 msgr2=0x7fc0801509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 shutdown_connections 2024-02-19T14:52:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.875+0000 7fc08657a640 1 -- 172.21.15.78:0/3014360813 wait complete. 2024-02-19T14:52:33.890 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:34.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.997+0000 7f6b2764e640 1 Processor -- start 2024-02-19T14:52:34.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.997+0000 7f6b2764e640 1 -- start start 2024-02-19T14:52:34.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.998+0000 7f6b2764e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6b200c5560 0x7f6b200c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.998+0000 7f6b2764e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200c65a0 con 0x7f6b20070720 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.998+0000 7f6b2764e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200c7530 con 0x7f6b200c7910 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.998+0000 7f6b2764e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b201537b0 con 0x7f6b200c5560 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.998+0000 7f6b25beb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6b20070720 0x7f6b20070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51542/0 (socket says 172.21.15.78:51542) 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.998+0000 7f6b25beb640 1 -- 172.21.15.78:0/1488956454 learned_addr learned my addr 172.21.15.78:0/1488956454 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b253ea640 1 --2- 172.21.15.78:0/1488956454 >> v2:172.21.15.150:3300/0 conn(0x7f6b200c5560 0x7f6b200c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567273354 0 0) 0x7f6b200c65a0 con 0x7f6b20070720 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b0c003520 con 0x7f6b20070720 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942598681 0 0) 0x7f6b200c7530 con 0x7f6b200c7910 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b200c65a0 con 0x7f6b200c7910 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865921645 0 0) 0x7f6b0c003520 con 0x7f6b20070720 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b253ea640 1 -- 172.21.15.78:0/1488956454 >> v1:172.21.15.150:6789/0 conn(0x7f6b200c7910 legacy=0x7f6b20154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b253ea640 1 -- 172.21.15.78:0/1488956454 >> v1:172.21.15.78:6789/0 conn(0x7f6b20070720 legacy=0x7f6b20070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:33.999+0000 7f6b253ea640 1 -- 172.21.15.78:0/1488956454 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20153a90 con 0x7f6b200c5560 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.000+0000 7f6b253ea640 1 --2- 172.21.15.78:0/1488956454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200c6ff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6b180097a0 tx=0x7f6b1802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dbff97f562ccab6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.000+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0019f0 con 0x7f6b200c5560 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.000+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1802e050 con 0x7f6b200c5560 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.000+0000 7f6b17fff640 1 -- 172.21.15.78:0/1488956454 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18030730 con 0x7f6b200c5560 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.000+0000 7f6b2764e640 1 -- 172.21.15.78:0/1488956454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 msgr2=0x7f6b200c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.000+0000 7f6b2764e640 1 --2- 172.21.15.78:0/1488956454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200c6ff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6b180097a0 tx=0x7f6b1802b710 comp rx=0 tx=0).stop 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.001+0000 7f6b2764e640 1 -- 172.21.15.78:0/1488956454 shutdown_connections 2024-02-19T14:52:34.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.001+0000 7f6b2764e640 1 --2- 172.21.15.78:0/1488956454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200c6ff0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.001+0000 7f6b2764e640 1 -- 172.21.15.78:0/1488956454 >> 172.21.15.78:0/1488956454 conn(0x7f6b2006b4b0 msgr2=0x7f6b200c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.001+0000 7f6b2764e640 1 -- 172.21.15.78:0/1488956454 shutdown_connections 2024-02-19T14:52:34.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.001+0000 7f6b2764e640 1 -- 172.21.15.78:0/1488956454 wait complete. 2024-02-19T14:52:34.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 Processor -- start 2024-02-19T14:52:34.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 -- start start 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b20070720 0x7f6b200aa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200ad460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 0x7f6b201ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b201dfd80 con 0x7f6b200c7910 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b201e01a0 con 0x7f6b200c5560 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b2764e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b20152190 con 0x7f6b20070720 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b25beb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 0x7f6b201ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b25beb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 0x7f6b201ebe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53784/0 (socket says 172.21.15.78:53784) 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.002+0000 7f6b25beb640 1 -- 172.21.15.78:0/3394247501 learned_addr learned my addr 172.21.15.78:0/3394247501 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b253ea640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b20070720 0x7f6b200aa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b25beb640 1 -- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b20070720 msgr2=0x7f6b200aa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b24be9640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b25beb640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b20070720 0x7f6b200aa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b25beb640 1 -- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 msgr2=0x7f6b200ad460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b25beb640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200ad460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b25beb640 1 -- 172.21.15.78:0/3394247501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b18009400 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b253ea640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b20070720 0x7f6b200aa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b24be9640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200ad460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b25beb640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 0x7f6b201ebe40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00ccb0 tx=0x7f6b1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c012400 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c012560 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b200c5ac0 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c012910 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.003+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b200aabb0 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.004+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6b1c012a70 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.005+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b201dfec0 con 0x7f6b200c7910 2024-02-19T14:52:34.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.005+0000 7f6b15ffb640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b0406b840 0x7f6b0406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.005+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6b1c08ab90 con 0x7f6b200c7910 2024-02-19T14:52:34.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.005+0000 7f6b253ea640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b0406b840 0x7f6b0406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.005+0000 7f6b253ea640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b0406b840 0x7f6b0406dd00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6b18034000 tx=0x7f6b18008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.009+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6b1c055300 con 0x7f6b200c7910 2024-02-19T14:52:34.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.222+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b201532a0 con 0x7f6b200c7910 2024-02-19T14:52:34.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.223+0000 7f6b15ffb640 1 -- 172.21.15.78:0/3394247501 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f6b1c0589c0 con 0x7f6b200c7910 2024-02-19T14:52:34.226 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:34.226 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:34.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b0406b840 msgr2=0x7f6b0406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b0406b840 0x7f6b0406dd00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6b18034000 tx=0x7f6b18008030 comp rx=0 tx=0).stop 2024-02-19T14:52:34.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 msgr2=0x7f6b201ebe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 0x7f6b201ebe40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00ccb0 tx=0x7f6b1c00a430 comp rx=0 tx=0).stop 2024-02-19T14:52:34.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 shutdown_connections 2024-02-19T14:52:34.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b0406b840 0x7f6b0406dd00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b200c7910 0x7f6b201ebe40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b200c5560 0x7f6b200ad460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 --2- 172.21.15.78:0/3394247501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b20070720 0x7f6b200aa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 >> 172.21.15.78:0/3394247501 conn(0x7f6b2006b4b0 msgr2=0x7f6b200c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 shutdown_connections 2024-02-19T14:52:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.225+0000 7f6b2764e640 1 -- 172.21.15.78:0/3394247501 wait complete. 2024-02-19T14:52:34.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:34.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:34.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.351+0000 7fa77098c640 1 Processor -- start 2024-02-19T14:52:34.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.351+0000 7fa77098c640 1 -- start start 2024-02-19T14:52:34.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa77098c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa76c155010 0x7fa76c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa77098c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c06ffa0 con 0x7fa76c0c78a0 2024-02-19T14:52:34.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa77098c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c06fc70 con 0x7fa76c0c4d80 2024-02-19T14:52:34.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa77098c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76c06b720 con 0x7fa76c155010 2024-02-19T14:52:34.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa76ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa76c155010 0x7fa76c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa76ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa76c155010 0x7fa76c159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53788/0 (socket says 172.21.15.78:53788) 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa76a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa76c0c78a0 0x7fa76c0c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51558/0 (socket says 172.21.15.78:51558) 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa76ad9d640 1 -- 172.21.15.78:0/875790509 learned_addr learned my addr 172.21.15.78:0/875790509 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.352+0000 7fa76ad9d640 1 -- 172.21.15.78:0/875790509 >> v1:172.21.15.150:6789/0 conn(0x7fa76c0c4d80 legacy=0x7fa76c1523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76ad9d640 1 -- 172.21.15.78:0/875790509 >> v1:172.21.15.78:6789/0 conn(0x7fa76c0c78a0 legacy=0x7fa76c0c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76959a640 1 -- 172.21.15.78:0/875790509 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679186666 0 0) 0x7fa76c06ffa0 con 0x7fa76c0c78a0 2024-02-19T14:52:34.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76ad9d640 1 -- 172.21.15.78:0/875790509 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c06b930 con 0x7fa76c155010 2024-02-19T14:52:34.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76ad9d640 1 --2- 172.21.15.78:0/875790509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c155010 0x7fa76c159010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa764009e50 tx=0x7fa76402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7b0709462d04cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76959a640 1 -- 172.21.15.78:0/875790509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76402ca80 con 0x7fa76c155010 2024-02-19T14:52:34.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76959a640 1 -- 172.21.15.78:0/875790509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76402cbe0 con 0x7fa76c155010 2024-02-19T14:52:34.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.353+0000 7fa76959a640 1 -- 172.21.15.78:0/875790509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7640343f0 con 0x7fa76c155010 2024-02-19T14:52:34.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 -- 172.21.15.78:0/875790509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c155010 msgr2=0x7fa76c159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 --2- 172.21.15.78:0/875790509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c155010 0x7fa76c159010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa764009e50 tx=0x7fa76402bee0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 -- 172.21.15.78:0/875790509 shutdown_connections 2024-02-19T14:52:34.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 --2- 172.21.15.78:0/875790509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c155010 0x7fa76c159010 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 -- 172.21.15.78:0/875790509 >> 172.21.15.78:0/875790509 conn(0x7fa76c066e70 msgr2=0x7fa76c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 -- 172.21.15.78:0/875790509 shutdown_connections 2024-02-19T14:52:34.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.354+0000 7fa77098c640 1 -- 172.21.15.78:0/875790509 wait complete. 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.355+0000 7fa77098c640 1 Processor -- start 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.355+0000 7fa77098c640 1 -- start start 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa77098c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 0x7fa76c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa77098c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0c78a0 0x7fa76c1ea420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa77098c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c155010 0x7fa76c1ea960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa77098c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c0ac4d0 con 0x7fa76c0c4d80 2024-02-19T14:52:34.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa77098c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c154740 con 0x7fa76c155010 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa77098c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c1ee2a0 con 0x7fa76c0c78a0 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 0x7fa76c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 0x7fa76c14e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53798/0 (socket says 172.21.15.78:53798) 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 -- 172.21.15.78:0/3940899010 learned_addr learned my addr 172.21.15.78:0/3940899010 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa769d9b640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0c78a0 0x7fa76c1ea420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76ad9d640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c155010 0x7fa76c1ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 -- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0c78a0 msgr2=0x7fa76c1ea420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0c78a0 0x7fa76c1ea420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 -- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c155010 msgr2=0x7fa76c1ea960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c155010 0x7fa76c1ea960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.356+0000 7fa76a59c640 1 -- 172.21.15.78:0/3940899010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa764009b30 con 0x7fa76c0c4d80 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.357+0000 7fa76a59c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 0x7fa76c14e100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00c460 tx=0x7fa75c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.357+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0074e0 con 0x7fa76c0c4d80 2024-02-19T14:52:34.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.357+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c007660 con 0x7fa76c0c4d80 2024-02-19T14:52:34.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.357+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c004a20 con 0x7fa76c0c4d80 2024-02-19T14:52:34.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.357+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c1ee580 con 0x7fa76c0c4d80 2024-02-19T14:52:34.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.357+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c1549b0 con 0x7fa76c0c4d80 2024-02-19T14:52:34.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.358+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76c1ee7c0 con 0x7fa76c0c4d80 2024-02-19T14:52:34.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.358+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa75c004b80 con 0x7fa76c0c4d80 2024-02-19T14:52:34.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.359+0000 7fa762ffd640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa74c06b810 0x7fa74c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.359+0000 7fa769d9b640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa74c06b810 0x7fa74c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.359+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa75c089970 con 0x7fa76c0c4d80 2024-02-19T14:52:34.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.359+0000 7fa769d9b640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa74c06b810 0x7fa74c06dcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa754006fd0 tx=0x7fa754008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.363+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa75c0540e0 con 0x7fa76c0c4d80 2024-02-19T14:52:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.541+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa76c074990 con 0x7fa76c0c4d80 2024-02-19T14:52:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.542+0000 7fa762ffd640 1 -- 172.21.15.78:0/3940899010 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa75c018020 con 0x7fa76c0c4d80 2024-02-19T14:52:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa74c06b810 msgr2=0x7fa74c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa74c06b810 0x7fa74c06dcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa754006fd0 tx=0x7fa754008040 comp rx=0 tx=0).stop 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 msgr2=0x7fa76c14e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 0x7fa76c14e100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00c460 tx=0x7fa75c00c920 comp rx=0 tx=0).stop 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 shutdown_connections 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa74c06b810 0x7fa74c06dcd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c155010 0x7fa76c1ea960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0c78a0 0x7fa76c1ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 --2- 172.21.15.78:0/3940899010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa76c0c4d80 0x7fa76c14e100 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 >> 172.21.15.78:0/3940899010 conn(0x7fa76c066e70 msgr2=0x7fa76c0c3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 shutdown_connections 2024-02-19T14:52:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.544+0000 7fa77098c640 1 -- 172.21.15.78:0/3940899010 wait complete. 2024-02-19T14:52:34.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.659+0000 7fb7a7365640 1 Processor -- start 2024-02-19T14:52:34.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.659+0000 7fb7a7365640 1 -- start start 2024-02-19T14:52:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a7365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a7365640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb7a00727d0 0x7fb7a0072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a7365640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a014f8f0 con 0x7fb7a00c78a0 2024-02-19T14:52:34.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a7365640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a00c4bc0 con 0x7fb7a00727d0 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a7365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a00c35c0 con 0x7fb7a014ef50 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a5902640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb7a00727d0 0x7fb7a0072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a5101640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb7a00c78a0 0x7fb7a00c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51570/0 (socket says 172.21.15.78:51570) 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a5101640 1 -- 172.21.15.78:0/2518164105 learned_addr learned my addr 172.21.15.78:0/2518164105 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a5902640 1 --2- 172.21.15.78:0/2518164105 >> v2:172.21.15.78:3300/0 conn(0x7fb7a00727d0 0x7fb7a0072bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53810/0 (socket says 172.21.15.78:53810) 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.660+0000 7fb7a4900640 1 --2- 172.21.15.78:0/2518164105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.661+0000 7fb7a5902640 1 -- 172.21.15.78:0/2518164105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 msgr2=0x7fb7a014f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.661+0000 7fb7a5902640 1 --2- 172.21.15.78:0/2518164105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a014f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.661+0000 7fb7a5902640 1 -- 172.21.15.78:0/2518164105 >> v1:172.21.15.78:6789/0 conn(0x7fb7a00c78a0 legacy=0x7fb7a00c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.661+0000 7fb797fff640 1 -- 172.21.15.78:0/2518164105 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923329748 0 0) 0x7fb7a014f8f0 con 0x7fb7a00c78a0 2024-02-19T14:52:34.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.661+0000 7fb7a5902640 1 -- 172.21.15.78:0/2518164105 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a00c38a0 con 0x7fb7a00727d0 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.661+0000 7fb7a5902640 1 --2- 172.21.15.78:0/2518164105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a0072bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0097a0 tx=0x7fb79c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dccf6635f5579aa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb797fff640 1 -- 172.21.15.78:0/2518164105 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c031830 con 0x7fb7a00727d0 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb797fff640 1 -- 172.21.15.78:0/2518164105 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c031990 con 0x7fb7a00727d0 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb797fff640 1 -- 172.21.15.78:0/2518164105 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c031d20 con 0x7fb7a00727d0 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 -- 172.21.15.78:0/2518164105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 msgr2=0x7fb7a0072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 --2- 172.21.15.78:0/2518164105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a0072bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0097a0 tx=0x7fb79c02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 -- 172.21.15.78:0/2518164105 shutdown_connections 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 --2- 172.21.15.78:0/2518164105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a0072bf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 --2- 172.21.15.78:0/2518164105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a014f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 -- 172.21.15.78:0/2518164105 >> 172.21.15.78:0/2518164105 conn(0x7fb7a0066e70 msgr2=0x7fb7a00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 -- 172.21.15.78:0/2518164105 shutdown_connections 2024-02-19T14:52:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.662+0000 7fb7a7365640 1 -- 172.21.15.78:0/2518164105 wait complete. 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.664+0000 7fb7a7365640 1 Processor -- start 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.664+0000 7fb7a7365640 1 -- start start 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a7365640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a7365640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00c78a0 0x7fb7a00bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a7365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a00c0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a7365640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a014da60 con 0x7fb7a00727d0 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a7365640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a01e8240 con 0x7fb7a00c78a0 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a7365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a01e8500 con 0x7fb7a014ef50 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a006b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53826/0 (socket says 172.21.15.78:53826) 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 -- 172.21.15.78:0/1260216299 learned_addr learned my addr 172.21.15.78:0/1260216299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5902640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a00c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a4900640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00c78a0 0x7fb7a00bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 -- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 msgr2=0x7fb7a00c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a00c0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 -- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00c78a0 msgr2=0x7fb7a00bfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00c78a0 0x7fb7a00bfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a5101640 1 -- 172.21.15.78:0/1260216299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009400 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.665+0000 7fb7a4900640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00c78a0 0x7fb7a00bfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.666+0000 7fb7a5101640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a006b0f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c890 tx=0x7fb79800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.666+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79800a640 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.666+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79800a7c0 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.666+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a01e87e0 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.666+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79800ab70 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.666+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a01e8da0 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.667+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a0074990 con 0x7fb7a00727d0 2024-02-19T14:52:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.667+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb79800acd0 con 0x7fb7a00727d0 2024-02-19T14:52:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.668+0000 7fb795ffb640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb78406b810 0x7fb78406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.668+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb798089d40 con 0x7fb7a00727d0 2024-02-19T14:52:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.668+0000 7fb7a4900640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb78406b810 0x7fb78406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.668+0000 7fb7a4900640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb78406b810 0x7fb78406dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00c4400 tx=0x7fb790008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.672+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb7980544b0 con 0x7fb7a00727d0 2024-02-19T14:52:34.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.848+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb7a00c1a60 con 0x7fb7a00727d0 2024-02-19T14:52:34.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.849+0000 7fb795ffb640 1 -- 172.21.15.78:0/1260216299 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7980180a0 con 0x7fb7a00727d0 2024-02-19T14:52:34.851 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb78406b810 msgr2=0x7fb78406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb78406b810 0x7fb78406dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00c4400 tx=0x7fb790008040 comp rx=0 tx=0).stop 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 msgr2=0x7fb7a006b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a006b0f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb79800c890 tx=0x7fb79800cd50 comp rx=0 tx=0).stop 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 shutdown_connections 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb78406b810 0x7fb78406dcd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7a014ef50 0x7fb7a00c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7a00c78a0 0x7fb7a00bfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 --2- 172.21.15.78:0/1260216299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7a00727d0 0x7fb7a006b0f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 >> 172.21.15.78:0/1260216299 conn(0x7fb7a0066e70 msgr2=0x7fb7a0151ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 shutdown_connections 2024-02-19T14:52:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.851+0000 7fb7a7365640 1 -- 172.21.15.78:0/1260216299 wait complete. 2024-02-19T14:52:34.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:34.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.965+0000 7f2504483640 1 Processor -- start 2024-02-19T14:52:34.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.965+0000 7f2504483640 1 -- start start 2024-02-19T14:52:34.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2504483640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f24fc0719d0 0x7f24fc071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2504483640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc0c15b0 0x7f24fc0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2504483640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24fc0c1120 con 0x7f24fc14ece0 2024-02-19T14:52:34.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2504483640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0c6080 con 0x7f24fc0719d0 2024-02-19T14:52:34.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2504483640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0c57e0 con 0x7f24fc0c15b0 2024-02-19T14:52:34.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2501a1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f24fc0719d0 0x7f24fc071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2501a1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f24fc0719d0 0x7f24fc071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53832/0 (socket says 172.21.15.78:53832) 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f2501a1e640 1 -- 172.21.15.78:0/4104097982 learned_addr learned my addr 172.21.15.78:0/4104097982 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.966+0000 7f250221f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f24fc14ece0 0x7f24fc0c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51584/0 (socket says 172.21.15.78:51584) 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f2502a20640 1 --2- 172.21.15.78:0/4104097982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc0c15b0 0x7f24fc0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f2501a1e640 1 -- 172.21.15.78:0/4104097982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc0c15b0 msgr2=0x7f24fc0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f2501a1e640 1 --2- 172.21.15.78:0/4104097982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc0c15b0 0x7f24fc0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f2501a1e640 1 -- 172.21.15.78:0/4104097982 >> v1:172.21.15.78:6789/0 conn(0x7f24fc14ece0 legacy=0x7f24fc0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f250121d640 1 -- 172.21.15.78:0/4104097982 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056477365 0 0) 0x7f24fc0c1120 con 0x7f24fc14ece0 2024-02-19T14:52:34.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f2501a1e640 1 -- 172.21.15.78:0/4104097982 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc0c5ac0 con 0x7f24fc0719d0 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.967+0000 7f2501a1e640 1 --2- 172.21.15.78:0/4104097982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc071e50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00bb10 tx=0x7f24ec02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23126c7b10d6f3d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f250121d640 1 -- 172.21.15.78:0/4104097982 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec036440 con 0x7f24fc0719d0 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f250121d640 1 -- 172.21.15.78:0/4104097982 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec0365a0 con 0x7f24fc0719d0 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f250121d640 1 -- 172.21.15.78:0/4104097982 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec036930 con 0x7f24fc0719d0 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 -- 172.21.15.78:0/4104097982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 msgr2=0x7f24fc071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 --2- 172.21.15.78:0/4104097982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc071e50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00bb10 tx=0x7f24ec02e150 comp rx=0 tx=0).stop 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 -- 172.21.15.78:0/4104097982 shutdown_connections 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 --2- 172.21.15.78:0/4104097982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc071e50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 --2- 172.21.15.78:0/4104097982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc0c15b0 0x7f24fc0c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 -- 172.21.15.78:0/4104097982 >> 172.21.15.78:0/4104097982 conn(0x7f24fc06b270 msgr2=0x7f24fc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:34.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 -- 172.21.15.78:0/4104097982 shutdown_connections 2024-02-19T14:52:34.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.968+0000 7f2504483640 1 -- 172.21.15.78:0/4104097982 wait complete. 2024-02-19T14:52:34.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.969+0000 7f2504483640 1 Processor -- start 2024-02-19T14:52:34.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.969+0000 7f2504483640 1 -- start start 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2504483640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc0a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2504483640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0c15b0 0x7f24fc1ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2504483640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc14ece0 0x7f24fc155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2504483640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0bfe50 con 0x7f24fc0719d0 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2504483640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc1eec00 con 0x7f24fc0c15b0 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2504483640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc1ea880 con 0x7f24fc14ece0 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc0a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc0a9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53840/0 (socket says 172.21.15.78:53840) 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 -- 172.21.15.78:0/1497734845 learned_addr learned my addr 172.21.15.78:0/1497734845 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:34.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2502a20640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc14ece0 0x7f24fc155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f2501a1e640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0c15b0 0x7f24fc1ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 -- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc14ece0 msgr2=0x7f24fc155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc14ece0 0x7f24fc155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 -- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0c15b0 msgr2=0x7f24fc1ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0c15b0 0x7f24fc1ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.970+0000 7f250221f640 1 -- 172.21.15.78:0/1497734845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec00b770 con 0x7f24fc0719d0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.971+0000 7f250221f640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc0a9a80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f24f800c4d0 tx=0x7f24f800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.971+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80074e0 con 0x7f24fc0719d0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.971+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8007660 con 0x7f24fc0719d0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.971+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8004a20 con 0x7f24fc0719d0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.971+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc14a580 con 0x7f24fc0719d0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.971+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc1eab20 con 0x7f24fc0719d0 2024-02-19T14:52:34.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.972+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc14d6c0 con 0x7f24fc0719d0 2024-02-19T14:52:34.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.972+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f24f8004b80 con 0x7f24fc0719d0 2024-02-19T14:52:34.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.972+0000 7f24f2ffd640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f24e006b810 0x7f24e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:34.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.972+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f24f8089990 con 0x7f24fc0719d0 2024-02-19T14:52:34.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.973+0000 7f2501a1e640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f24e006b810 0x7f24e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:34.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.973+0000 7f2501a1e640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f24e006b810 0x7f24e006dcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24ec038000 tx=0x7f24ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:34.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:34.977+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f24f8054100 con 0x7f24fc0719d0 2024-02-19T14:52:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.154+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24fc074990 con 0x7f24fc0719d0 2024-02-19T14:52:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.155+0000 7f24f2ffd640 1 -- 172.21.15.78:0/1497734845 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24f80577c0 con 0x7f24fc0719d0 2024-02-19T14:52:35.157 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.156+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f24e006b810 msgr2=0x7f24e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.156+0000 7f2504483640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f24e006b810 0x7f24e006dcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24ec038000 tx=0x7f24ec008040 comp rx=0 tx=0).stop 2024-02-19T14:52:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.156+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 msgr2=0x7f24fc0a9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.156+0000 7f2504483640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc0a9a80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f24f800c4d0 tx=0x7f24f800c990 comp rx=0 tx=0).stop 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.156+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 shutdown_connections 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.156+0000 7f2504483640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f24e006b810 0x7f24e006dcd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.157+0000 7f2504483640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24fc14ece0 0x7f24fc155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.157+0000 7f2504483640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24fc0c15b0 0x7f24fc1ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.157+0000 7f2504483640 1 --2- 172.21.15.78:0/1497734845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24fc0719d0 0x7f24fc0a9a80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.157+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 >> 172.21.15.78:0/1497734845 conn(0x7f24fc06b270 msgr2=0x7f24fc1536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.157+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 shutdown_connections 2024-02-19T14:52:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.157+0000 7f2504483640 1 -- 172.21.15.78:0/1497734845 wait complete. 2024-02-19T14:52:35.173 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.275+0000 7f782add7640 1 Processor -- start 2024-02-19T14:52:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.275+0000 7f782add7640 1 -- start start 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f782add7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7824152700 0x7f78240c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f782add7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78240c8e00 0x7f782415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f782add7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78241551f0 con 0x7f78240c5730 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f782add7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7824155490 con 0x7f7824152700 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f782add7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7824155770 con 0x7f78240c8e00 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f7828b73640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7824152700 0x7f78240c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f7828b73640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7824152700 0x7f78240c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53848/0 (socket says 172.21.15.78:53848) 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f7828b73640 1 -- 172.21.15.78:0/1845774805 learned_addr learned my addr 172.21.15.78:0/1845774805 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f7828b73640 1 -- 172.21.15.78:0/1845774805 >> v2:172.21.15.150:3300/0 conn(0x7f78240c8e00 msgr2=0x7f782415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:35.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f7828b73640 1 --2- 172.21.15.78:0/1845774805 >> v2:172.21.15.150:3300/0 conn(0x7f78240c8e00 0x7f782415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.276+0000 7f7828b73640 1 -- 172.21.15.78:0/1845774805 >> v1:172.21.15.150:6789/0 conn(0x7f78240c5730 legacy=0x7f782406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:35.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.277+0000 7f7828b73640 1 -- 172.21.15.78:0/1845774805 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824155a50 con 0x7f7824152700 2024-02-19T14:52:35.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.277+0000 7f7828b73640 1 --2- 172.21.15.78:0/1845774805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7824152700 0x7f78240c51f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f78241596f0 tx=0x7f782002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8b4f00fdb5bda4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.277+0000 7f781b7fe640 1 -- 172.21.15.78:0/1845774805 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78200317e0 con 0x7f7824152700 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.277+0000 7f781b7fe640 1 -- 172.21.15.78:0/1845774805 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7820031940 con 0x7f7824152700 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.277+0000 7f781b7fe640 1 -- 172.21.15.78:0/1845774805 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820031cd0 con 0x7f7824152700 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 -- 172.21.15.78:0/1845774805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7824152700 msgr2=0x7f78240c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 --2- 172.21.15.78:0/1845774805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7824152700 0x7f78240c51f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f78241596f0 tx=0x7f782002b880 comp rx=0 tx=0).stop 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 -- 172.21.15.78:0/1845774805 shutdown_connections 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 --2- 172.21.15.78:0/1845774805 >> v2:172.21.15.150:3300/0 conn(0x7f78240c8e00 0x7f782415f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 --2- 172.21.15.78:0/1845774805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7824152700 0x7f78240c51f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 -- 172.21.15.78:0/1845774805 >> 172.21.15.78:0/1845774805 conn(0x7f78240cac90 msgr2=0x7f78240cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 -- 172.21.15.78:0/1845774805 shutdown_connections 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.278+0000 7f782add7640 1 -- 172.21.15.78:0/1845774805 wait complete. 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.279+0000 7f782add7640 1 Processor -- start 2024-02-19T14:52:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.279+0000 7f782add7640 1 -- start start 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f782add7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 0x7f78241ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f782add7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8e00 0x7f78240c3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f782add7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7824152700 0x7f7824148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f782add7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240c4010 con 0x7f78240c5730 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f782add7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240c1e50 con 0x7f7824152700 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f782add7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78241eb2a0 con 0x7f78240c8e00 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 0x7f78241ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 0x7f78241ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53862/0 (socket says 172.21.15.78:53862) 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 -- 172.21.15.78:0/3646143110 learned_addr learned my addr 172.21.15.78:0/3646143110 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7829374640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7824152700 0x7f7824148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f781bfff640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8e00 0x7f78240c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 -- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8e00 msgr2=0x7f78240c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8e00 0x7f78240c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 -- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7824152700 msgr2=0x7f7824148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7824152700 0x7f7824148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 -- 172.21.15.78:0/3646143110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78200094d0 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f781bfff640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8e00 0x7f78240c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.280+0000 7f7828b73640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 0x7f78241ead60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f782002b8c0 tx=0x7f78200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.281+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820032880 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.281+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78200329e0 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.281+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7824154eb0 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.281+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820032d90 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.281+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78240acd30 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.282+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78240acf20 con 0x7f78240c5730 2024-02-19T14:52:35.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.282+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f78200453f0 con 0x7f78240c5730 2024-02-19T14:52:35.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.282+0000 7f78197fa640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f781006b840 0x7f781006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.283+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f78200ac990 con 0x7f78240c5730 2024-02-19T14:52:35.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.283+0000 7f781bfff640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f781006b840 0x7f781006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.283+0000 7f781bfff640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f781006b840 0x7f781006dd00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f78240c1590 tx=0x7f7814009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.287+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7820077220 con 0x7f78240c5730 2024-02-19T14:52:35.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.462+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7824074990 con 0x7f78240c5730 2024-02-19T14:52:35.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.462+0000 7f78197fa640 1 -- 172.21.15.78:0/3646143110 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f782007a8e0 con 0x7f78240c5730 2024-02-19T14:52:35.465 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f781006b840 msgr2=0x7f781006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f781006b840 0x7f781006dd00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f78240c1590 tx=0x7f7814009040 comp rx=0 tx=0).stop 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 msgr2=0x7f78241ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 0x7f78241ead60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f782002b8c0 tx=0x7f78200046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 shutdown_connections 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f781006b840 0x7f781006dd00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7824152700 0x7f7824148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8e00 0x7f78240c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 --2- 172.21.15.78:0/3646143110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240c5730 0x7f78241ead60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 >> 172.21.15.78:0/3646143110 conn(0x7f78240cac90 msgr2=0x7f78240cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 shutdown_connections 2024-02-19T14:52:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.465+0000 7f782add7640 1 -- 172.21.15.78:0/3646143110 wait complete. 2024-02-19T14:52:35.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.581+0000 7f6adcd0d640 1 Processor -- start 2024-02-19T14:52:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.581+0000 7f6adcd0d640 1 -- start start 2024-02-19T14:52:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.582+0000 7f6adcd0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ad8149790 0x7f6ad806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.582+0000 7f6adcd0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ad80723d0 0x7f6ad8072800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.582+0000 7f6adcd0d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad806f420 con 0x7f6ad8152c30 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.582+0000 7f6adcd0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad8149b70 con 0x7f6ad80723d0 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.582+0000 7f6adcd0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad80ab760 con 0x7f6ad8149790 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ad80723d0 0x7f6ad8072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ad80723d0 0x7f6ad8072800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53874/0 (socket says 172.21.15.78:53874) 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 -- 172.21.15.78:0/2225218949 learned_addr learned my addr 172.21.15.78:0/2225218949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad659c640 1 --2- 172.21.15.78:0/2225218949 >> v2:172.21.15.150:3300/0 conn(0x7f6ad8149790 0x7f6ad806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 -- 172.21.15.78:0/2225218949 >> v2:172.21.15.150:3300/0 conn(0x7f6ad8149790 msgr2=0x7f6ad806ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 --2- 172.21.15.78:0/2225218949 >> v2:172.21.15.150:3300/0 conn(0x7f6ad8149790 0x7f6ad806ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 -- 172.21.15.78:0/2225218949 >> v1:172.21.15.150:6789/0 conn(0x7f6ad8152c30 legacy=0x7f6ad81530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.583+0000 7f6ad5d9b640 1 -- 172.21.15.78:0/2225218949 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad80aba40 con 0x7f6ad80723d0 2024-02-19T14:52:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6ad5d9b640 1 --2- 172.21.15.78:0/2225218949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad80723d0 0x7f6ad8072800 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6ad806ccd0 tx=0x7f6acc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196b47d33ec4595d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6ad559a640 1 -- 172.21.15.78:0/2225218949 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc031750 con 0x7f6ad80723d0 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6ad559a640 1 -- 172.21.15.78:0/2225218949 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc0318b0 con 0x7f6ad80723d0 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6ad559a640 1 -- 172.21.15.78:0/2225218949 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc031c40 con 0x7f6ad80723d0 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 -- 172.21.15.78:0/2225218949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad80723d0 msgr2=0x7f6ad8072800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/2225218949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad80723d0 0x7f6ad8072800 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6ad806ccd0 tx=0x7f6acc02b7f0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 -- 172.21.15.78:0/2225218949 shutdown_connections 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/2225218949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad80723d0 0x7f6ad8072800 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/2225218949 >> v2:172.21.15.150:3300/0 conn(0x7f6ad8149790 0x7f6ad806ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 -- 172.21.15.78:0/2225218949 >> 172.21.15.78:0/2225218949 conn(0x7f6ad80718e0 msgr2=0x7f6ad8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 -- 172.21.15.78:0/2225218949 shutdown_connections 2024-02-19T14:52:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.584+0000 7f6adcd0d640 1 -- 172.21.15.78:0/2225218949 wait complete. 2024-02-19T14:52:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.586+0000 7f6adcd0d640 1 Processor -- start 2024-02-19T14:52:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.586+0000 7f6adcd0d640 1 -- start start 2024-02-19T14:52:35.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6adcd0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ad80723d0 0x7f6ad81de420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6adcd0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ad8149790 0x7f6ad81de960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6adcd0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 0x7f6ad8142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6adcd0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad8140c50 con 0x7f6ad8152c30 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6adcd0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad81deea0 con 0x7f6ad80723d0 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6adcd0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad81e9a90 con 0x7f6ad8149790 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.587+0000 7f6ad6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 0x7f6ad8142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 0x7f6ad8142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53884/0 (socket says 172.21.15.78:53884) 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 -- 172.21.15.78:0/719758575 learned_addr learned my addr 172.21.15.78:0/719758575 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad659c640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ad80723d0 0x7f6ad81de420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad5d9b640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ad8149790 0x7f6ad81de960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 -- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ad8149790 msgr2=0x7f6ad81de960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ad8149790 0x7f6ad81de960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 -- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ad80723d0 msgr2=0x7f6ad81de420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ad80723d0 0x7f6ad81de420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 -- 172.21.15.78:0/719758575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0094d0 con 0x7f6ad8152c30 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ad6d9d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 0x7f6ad8142b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000c7b0 tx=0x7f6ad000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0003d80 con 0x7f6ad8152c30 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad0005070 con 0x7f6ad8152c30 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0005420 con 0x7f6ad8152c30 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad81e9d70 con 0x7f6ad8152c30 2024-02-19T14:52:35.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.588+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad8141180 con 0x7f6ad8152c30 2024-02-19T14:52:35.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.589+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad8141370 con 0x7f6ad8152c30 2024-02-19T14:52:35.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.590+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6ad0005580 con 0x7f6ad8152c30 2024-02-19T14:52:35.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.590+0000 7f6ac6ffd640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ab806b810 0x7f6ab806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.590+0000 7f6ad659c640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ab806b810 0x7f6ab806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.590+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6ad0089d90 con 0x7f6ad8152c30 2024-02-19T14:52:35.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.591+0000 7f6ad659c640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ab806b810 0x7f6ab806dcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b7c0 tx=0x7f6ac8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.595+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ad0054470 con 0x7f6ad8152c30 2024-02-19T14:52:35.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.772+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ad8074990 con 0x7f6ad8152c30 2024-02-19T14:52:35.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.773+0000 7f6ac6ffd640 1 -- 172.21.15.78:0/719758575 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7f6ad0017070 con 0x7f6ad8152c30 2024-02-19T14:52:35.776 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:35.776 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ab806b810 msgr2=0x7f6ab806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ab806b810 0x7f6ab806dcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b7c0 tx=0x7f6ac8008040 comp rx=0 tx=0).stop 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 msgr2=0x7f6ad8142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 0x7f6ad8142b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000c7b0 tx=0x7f6ad000cc70 comp rx=0 tx=0).stop 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 shutdown_connections 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ab806b810 0x7f6ab806dcd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ad8152c30 0x7f6ad8142b70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ad8149790 0x7f6ad81de960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 --2- 172.21.15.78:0/719758575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ad80723d0 0x7f6ad81de420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 >> 172.21.15.78:0/719758575 conn(0x7f6ad80718e0 msgr2=0x7f6ad8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 shutdown_connections 2024-02-19T14:52:35.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.775+0000 7f6adcd0d640 1 -- 172.21.15.78:0/719758575 wait complete. 2024-02-19T14:52:35.788 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:35.789 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:35.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.890+0000 7fa36cfdd640 1 Processor -- start 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.890+0000 7fa36cfdd640 1 -- start start 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa36cfdd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3680c78a0 0x7fa3680c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa36cfdd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa368072ee0 0x7fa36814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa36cfdd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680c35c0 con 0x7fa36814ef50 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa36cfdd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36814f920 con 0x7fa368072ee0 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa36cfdd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3680c38a0 con 0x7fa3680c78a0 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa366d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa368072ee0 0x7fa36814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa366d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa368072ee0 0x7fa36814f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53888/0 (socket says 172.21.15.78:53888) 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa366d9d640 1 -- 172.21.15.78:0/643535252 learned_addr learned my addr 172.21.15.78:0/643535252 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:35.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa36659c640 1 --2- 172.21.15.78:0/643535252 >> v2:172.21.15.150:3300/0 conn(0x7fa3680c78a0 0x7fa3680c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.891+0000 7fa366d9d640 1 -- 172.21.15.78:0/643535252 >> v2:172.21.15.150:3300/0 conn(0x7fa3680c78a0 msgr2=0x7fa3680c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa366d9d640 1 --2- 172.21.15.78:0/643535252 >> v2:172.21.15.150:3300/0 conn(0x7fa3680c78a0 0x7fa3680c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa366d9d640 1 -- 172.21.15.78:0/643535252 >> v1:172.21.15.150:6789/0 conn(0x7fa36814ef50 legacy=0x7fa3680727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa366d9d640 1 -- 172.21.15.78:0/643535252 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680c3b80 con 0x7fa368072ee0 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa366d9d640 1 --2- 172.21.15.78:0/643535252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa36814f3e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa3680c80d0 tx=0x7fa36002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88302f130606abbb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa36559a640 1 -- 172.21.15.78:0/643535252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36002ca80 con 0x7fa368072ee0 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa36559a640 1 -- 172.21.15.78:0/643535252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36002cbe0 con 0x7fa368072ee0 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.892+0000 7fa36559a640 1 -- 172.21.15.78:0/643535252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3600343f0 con 0x7fa368072ee0 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 -- 172.21.15.78:0/643535252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 msgr2=0x7fa36814f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/643535252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa36814f3e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa3680c80d0 tx=0x7fa36002bee0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 -- 172.21.15.78:0/643535252 shutdown_connections 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/643535252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa36814f3e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/643535252 >> v2:172.21.15.150:3300/0 conn(0x7fa3680c78a0 0x7fa3680c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 -- 172.21.15.78:0/643535252 >> 172.21.15.78:0/643535252 conn(0x7fa368066e70 msgr2=0x7fa3680a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 -- 172.21.15.78:0/643535252 shutdown_connections 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.893+0000 7fa36cfdd640 1 -- 172.21.15.78:0/643535252 wait complete. 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.894+0000 7fa36cfdd640 1 Processor -- start 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.894+0000 7fa36cfdd640 1 -- start start 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36cfdd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa3680bcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36cfdd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3680c78a0 0x7fa3680c0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36cfdd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36814ef50 0x7fa3681eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36cfdd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680c0470 con 0x7fa368072ee0 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36cfdd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3681480f0 con 0x7fa36814ef50 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36cfdd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa368147b30 con 0x7fa3680c78a0 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa3680bcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa3680bcfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53894/0 (socket says 172.21.15.78:53894) 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 -- 172.21.15.78:0/3502275254 learned_addr learned my addr 172.21.15.78:0/3502275254 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa365d9b640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3680c78a0 0x7fa3680c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa366d9d640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36814ef50 0x7fa3681eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 -- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3680c78a0 msgr2=0x7fa3680c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3680c78a0 0x7fa3680c0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 -- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36814ef50 msgr2=0x7fa3681eaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36814ef50 0x7fa3681eaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.895+0000 7fa36659c640 1 -- 172.21.15.78:0/3502275254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360009b30 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.896+0000 7fa36659c640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa3680bcfa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa35800d800 tx=0x7fa35800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.896+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35800b6c0 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.896+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35800b820 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.896+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa368147e10 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.896+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35800bbb0 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.896+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3680c0090 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.897+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa368074990 con 0x7fa368072ee0 2024-02-19T14:52:35.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.897+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa35800bd10 con 0x7fa368072ee0 2024-02-19T14:52:35.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.898+0000 7fa356ffd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa34806b810 0x7fa34806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:35.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.898+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa35808aa20 con 0x7fa368072ee0 2024-02-19T14:52:35.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.898+0000 7fa365d9b640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa34806b810 0x7fa34806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:35.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.898+0000 7fa365d9b640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa34806b810 0x7fa34806dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006fd0 tx=0x7fa35c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:35.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:35.902+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa358055190 con 0x7fa368072ee0 2024-02-19T14:52:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.079+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3680c1c90 con 0x7fa368072ee0 2024-02-19T14:52:36.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.079+0000 7fa356ffd640 1 -- 172.21.15.78:0/3502275254 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa358058850 con 0x7fa368072ee0 2024-02-19T14:52:36.082 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa34806b810 msgr2=0x7fa34806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa34806b810 0x7fa34806dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006fd0 tx=0x7fa35c008040 comp rx=0 tx=0).stop 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 msgr2=0x7fa3680bcfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa3680bcfa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa35800d800 tx=0x7fa35800dcc0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 shutdown_connections 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa34806b810 0x7fa34806dcd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36814ef50 0x7fa3681eaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3680c78a0 0x7fa3680c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 --2- 172.21.15.78:0/3502275254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa368072ee0 0x7fa3680bcfa0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 >> 172.21.15.78:0/3502275254 conn(0x7fa368066e70 msgr2=0x7fa368150030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 shutdown_connections 2024-02-19T14:52:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.081+0000 7fa36cfdd640 1 -- 172.21.15.78:0/3502275254 wait complete. 2024-02-19T14:52:36.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 Processor -- start 2024-02-19T14:52:36.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 -- start start 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d80acb50 0x7fc3d8155900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3d806da10 0x7fc3d80723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d80c1d80 con 0x7fc3d8072910 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d806caf0 con 0x7fc3d806da10 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.197+0000 7fc3dff83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d80c37e0 con 0x7fc3d80acb50 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3de520640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc3d8072910 0x7fc3d8158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51586/0 (socket says 172.21.15.78:51586) 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3de520640 1 -- 172.21.15.78:0/1314979146 learned_addr learned my addr 172.21.15.78:0/1314979146 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3ddd1f640 1 --2- 172.21.15.78:0/1314979146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d80acb50 0x7fc3d8155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3dd51e640 1 --2- 172.21.15.78:0/1314979146 >> v2:172.21.15.150:3300/0 conn(0x7fc3d806da10 0x7fc3d80723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3dcd1d640 1 -- 172.21.15.78:0/1314979146 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220991297 0 0) 0x7fc3d80c1d80 con 0x7fc3d8072910 2024-02-19T14:52:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3dcd1d640 1 -- 172.21.15.78:0/1314979146 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c4003520 con 0x7fc3d8072910 2024-02-19T14:52:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3ddd1f640 1 -- 172.21.15.78:0/1314979146 >> v2:172.21.15.150:3300/0 conn(0x7fc3d806da10 msgr2=0x7fc3d80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3ddd1f640 1 --2- 172.21.15.78:0/1314979146 >> v2:172.21.15.150:3300/0 conn(0x7fc3d806da10 0x7fc3d80723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3ddd1f640 1 -- 172.21.15.78:0/1314979146 >> v1:172.21.15.78:6789/0 conn(0x7fc3d8072910 legacy=0x7fc3d8158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.198+0000 7fc3ddd1f640 1 -- 172.21.15.78:0/1314979146 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d806c5a0 con 0x7fc3d80acb50 2024-02-19T14:52:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.199+0000 7fc3ddd1f640 1 --2- 172.21.15.78:0/1314979146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d80acb50 0x7fc3d8155900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40097a0 tx=0x7fc3d402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b94ec1d8e126d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.199+0000 7fc3dcd1d640 1 -- 172.21.15.78:0/1314979146 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d402e070 con 0x7fc3d80acb50 2024-02-19T14:52:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.199+0000 7fc3dcd1d640 1 -- 172.21.15.78:0/1314979146 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d40315a0 con 0x7fc3d80acb50 2024-02-19T14:52:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.199+0000 7fc3dcd1d640 1 -- 172.21.15.78:0/1314979146 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4031910 con 0x7fc3d80acb50 2024-02-19T14:52:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 -- 172.21.15.78:0/1314979146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d80acb50 msgr2=0x7fc3d8155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1314979146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d80acb50 0x7fc3d8155900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40097a0 tx=0x7fc3d402b840 comp rx=0 tx=0).stop 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 -- 172.21.15.78:0/1314979146 shutdown_connections 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1314979146 >> v2:172.21.15.150:3300/0 conn(0x7fc3d806da10 0x7fc3d80723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1314979146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d80acb50 0x7fc3d8155900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 -- 172.21.15.78:0/1314979146 >> 172.21.15.78:0/1314979146 conn(0x7fc3d80718e0 msgr2=0x7fc3d8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 -- 172.21.15.78:0/1314979146 shutdown_connections 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.200+0000 7fc3dff83640 1 -- 172.21.15.78:0/1314979146 wait complete. 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.201+0000 7fc3dff83640 1 Processor -- start 2024-02-19T14:52:36.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.201+0000 7fc3dff83640 1 -- start start 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.201+0000 7fc3dff83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3d806da10 0x7fc3d81eac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.201+0000 7fc3dff83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d8072910 0x7fc3d80c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dff83640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 0x7fc3d8146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dff83640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8149d30 con 0x7fc3d80acb50 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dff83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8152770 con 0x7fc3d806da10 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dff83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8153de0 con 0x7fc3d8072910 2024-02-19T14:52:36.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 0x7fc3d8146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 0x7fc3d8146730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53902/0 (socket says 172.21.15.78:53902) 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 -- 172.21.15.78:0/1534602841 learned_addr learned my addr 172.21.15.78:0/1534602841 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3ddd1f640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3d806da10 0x7fc3d81eac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dd51e640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d8072910 0x7fc3d80c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 -- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d8072910 msgr2=0x7fc3d80c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d8072910 0x7fc3d80c0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 -- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3d806da10 msgr2=0x7fc3d81eac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3d806da10 0x7fc3d81eac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 -- 172.21.15.78:0/1534602841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4009400 con 0x7fc3d80acb50 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3de520640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 0x7fc3d8146730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000caa0 tx=0x7fc3d000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000ac00 con 0x7fc3d80acb50 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d000ad60 con 0x7fc3d80acb50 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0012620 con 0x7fc3d80acb50 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d81472f0 con 0x7fc3d80acb50 2024-02-19T14:52:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.202+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d8152400 con 0x7fc3d80acb50 2024-02-19T14:52:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.203+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d8147810 con 0x7fc3d80acb50 2024-02-19T14:52:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.204+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc3d0007070 con 0x7fc3d80acb50 2024-02-19T14:52:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.204+0000 7fc3ce7fc640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3bc06b810 0x7fc3bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.204+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc3d008a3c0 con 0x7fc3d80acb50 2024-02-19T14:52:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.204+0000 7fc3ddd1f640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3bc06b810 0x7fc3bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.205+0000 7fc3ddd1f640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3bc06b810 0x7fc3bc06dcd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4036000 tx=0x7fc3d403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.208+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc3d8147810 con 0x7fc3d80acb50 2024-02-19T14:52:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.389+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3d8152260 con 0x7fc3d80acb50 2024-02-19T14:52:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.389+0000 7fc3ce7fc640 1 -- 172.21.15.78:0/1534602841 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc3d8152260 con 0x7fc3d80acb50 2024-02-19T14:52:36.392 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3bc06b810 msgr2=0x7fc3bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3bc06b810 0x7fc3bc06dcd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4036000 tx=0x7fc3d403d040 comp rx=0 tx=0).stop 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 msgr2=0x7fc3d8146730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 0x7fc3d8146730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000caa0 tx=0x7fc3d000cf60 comp rx=0 tx=0).stop 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 shutdown_connections 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3bc06b810 0x7fc3bc06dcd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3d80acb50 0x7fc3d8146730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3d8072910 0x7fc3d80c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 --2- 172.21.15.78:0/1534602841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3d806da10 0x7fc3d81eac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 >> 172.21.15.78:0/1534602841 conn(0x7fc3d80718e0 msgr2=0x7fc3d814ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 shutdown_connections 2024-02-19T14:52:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.391+0000 7fc3dff83640 1 -- 172.21.15.78:0/1534602841 wait complete. 2024-02-19T14:52:36.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:36.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.507+0000 7f7f1c21c640 1 Processor -- start 2024-02-19T14:52:36.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.508+0000 7f7f1c21c640 1 -- start start 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1c21c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f14149790 0x7f7f1406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1c21c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f140727f0 0x7f7f14153340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1c21c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f14072c00 con 0x7f7f140723d0 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1c21c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f14149b70 con 0x7f7f140727f0 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1c21c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1406f420 con 0x7f7f14149790 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f140727f0 0x7f7f14153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f140727f0 0x7f7f14153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53906/0 (socket says 172.21.15.78:53906) 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/77346268 learned_addr learned my addr 172.21.15.78:0/77346268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f19fb8640 1 --2- 172.21.15.78:0/77346268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f14149790 0x7f7f1406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/77346268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f14149790 msgr2=0x7f7f1406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:36.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 --2- 172.21.15.78:0/77346268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f14149790 0x7f7f1406ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/77346268 >> v1:172.21.15.150:6789/0 conn(0x7f7f140723d0 legacy=0x7f7f14152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:36.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.509+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/77346268 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140ab760 con 0x7f7f140727f0 2024-02-19T14:52:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.510+0000 7f7f1a7b9640 1 --2- 172.21.15.78:0/77346268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f140727f0 0x7f7f14153340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009e50 tx=0x7f7f0c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e53bc1bbd782bdc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.510+0000 7f7f18fb6640 1 -- 172.21.15.78:0/77346268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c02c950 con 0x7f7f140727f0 2024-02-19T14:52:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.510+0000 7f7f18fb6640 1 -- 172.21.15.78:0/77346268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c02cab0 con 0x7f7f140727f0 2024-02-19T14:52:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.510+0000 7f7f18fb6640 1 -- 172.21.15.78:0/77346268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c02ce40 con 0x7f7f140727f0 2024-02-19T14:52:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 -- 172.21.15.78:0/77346268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f140727f0 msgr2=0x7f7f14153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/77346268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f140727f0 0x7f7f14153340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009e50 tx=0x7f7f0c02bdb0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 -- 172.21.15.78:0/77346268 shutdown_connections 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/77346268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f140727f0 0x7f7f14153340 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/77346268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f14149790 0x7f7f1406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 -- 172.21.15.78:0/77346268 >> 172.21.15.78:0/77346268 conn(0x7f7f140718e0 msgr2=0x7f7f14071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 -- 172.21.15.78:0/77346268 shutdown_connections 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.511+0000 7f7f1c21c640 1 -- 172.21.15.78:0/77346268 wait complete. 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 Processor -- start 2024-02-19T14:52:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 -- start start 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f140723d0 0x7f7f141de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f140727f0 0x7f7f141463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 0x7f7f141468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1414ad30 con 0x7f7f14149790 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14146e20 con 0x7f7f140727f0 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1c21c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1414c230 con 0x7f7f140723d0 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.512+0000 7f7f1a7b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 0x7f7f141468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 0x7f7f141468e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53910/0 (socket says 172.21.15.78:53910) 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/4288778396 learned_addr learned my addr 172.21.15.78:0/4288778396 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f19fb8640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f140723d0 0x7f7f141de2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f197b7640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f140727f0 0x7f7f141463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f140723d0 msgr2=0x7f7f141de2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f140723d0 0x7f7f141de2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f140727f0 msgr2=0x7f7f141463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f140727f0 0x7f7f141463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 -- 172.21.15.78:0/4288778396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c009b30 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1a7b9640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 0x7f7f141468e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009e50 tx=0x7f7f0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c009510 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c03e450 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1414aa50 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c03e7e0 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.513+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1414c740 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.514+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f14074990 con 0x7f7f14149790 2024-02-19T14:52:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.515+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7f0c03e940 con 0x7f7f14149790 2024-02-19T14:52:36.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.515+0000 7f7f0a7fc640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef806b810 0x7f7ef806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.515+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7f0c0ad120 con 0x7f7f14149790 2024-02-19T14:52:36.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.515+0000 7f7f19fb8640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef806b810 0x7f7ef806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.515+0000 7f7f19fb8640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef806b810 0x7f7ef806dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f10006fd0 tx=0x7f7f10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.519+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7f0c077890 con 0x7f7f14149790 2024-02-19T14:52:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.691+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f140acff0 con 0x7f7f14149790 2024-02-19T14:52:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.691+0000 7f7f0a7fc640 1 -- 172.21.15.78:0/4288778396 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f0c043020 con 0x7f7f14149790 2024-02-19T14:52:36.694 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef806b810 msgr2=0x7f7ef806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef806b810 0x7f7ef806dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f10006fd0 tx=0x7f7f10008040 comp rx=0 tx=0).stop 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 msgr2=0x7f7f141468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 0x7f7f141468e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009e50 tx=0x7f7f0c004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 shutdown_connections 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef806b810 0x7f7ef806dcd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f14149790 0x7f7f141468e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f140727f0 0x7f7f141463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 --2- 172.21.15.78:0/4288778396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f140723d0 0x7f7f141de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 >> 172.21.15.78:0/4288778396 conn(0x7f7f140718e0 msgr2=0x7f7f14071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 shutdown_connections 2024-02-19T14:52:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.693+0000 7f7f1c21c640 1 -- 172.21.15.78:0/4288778396 wait complete. 2024-02-19T14:52:36.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.809+0000 7f38941b5640 1 Processor -- start 2024-02-19T14:52:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.809+0000 7f38941b5640 1 -- start start 2024-02-19T14:52:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f38941b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 0x7f388c0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f38941b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c0c1d80 con 0x7f388c072910 2024-02-19T14:52:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f38941b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c06c5a0 con 0x7f388c0acb50 2024-02-19T14:52:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f38941b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c06caf0 con 0x7f388c06da10 2024-02-19T14:52:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f3892752640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f388c072910 0x7f388c158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51588/0 (socket says 172.21.15.78:51588) 2024-02-19T14:52:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f3892752640 1 -- 172.21.15.78:0/2415191507 learned_addr learned my addr 172.21.15.78:0/2415191507 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f3891750640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 0x7f388c0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515008534 0 0) 0x7f388c0c1d80 con 0x7f388c072910 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.810+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3878003520 con 0x7f388c072910 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701873382 0 0) 0x7f3878003520 con 0x7f388c072910 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 msgr2=0x7f388c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 0x7f388c0723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 >> v1:172.21.15.150:6789/0 conn(0x7f388c0acb50 legacy=0x7f388c155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c06c810 con 0x7f388c072910 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f38840019f0 con 0x7f388c072910 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 >> v1:172.21.15.78:6789/0 conn(0x7f388c072910 legacy=0x7f388c158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38780039a0 0x7f3878003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 0x7f38780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3878004d00 0x7f3878005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c06caf0 con 0x7f3878004d00 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387800bab0 con 0x7f38780039a0 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387800bc70 con 0x7f388c06da10 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3892752640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3878004d00 0x7f3878005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3891f51640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38780039a0 0x7f3878003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3891750640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 0x7f38780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3892752640 1 -- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 msgr2=0x7f38780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3892752640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c06da10 0x7f38780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3892752640 1 -- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38780039a0 msgr2=0x7f3878003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3892752640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38780039a0 0x7f3878003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.811+0000 7f3892752640 1 -- 172.21.15.78:0/2415191507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3878003520 con 0x7f3878004d00 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.812+0000 7f3892752640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3878004d00 0x7f3878005180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f38840013d0 tx=0x7f38780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f25306888861af4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.812+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884034630 con 0x7f3878004d00 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.812+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388402f070 con 0x7f3878004d00 2024-02-19T14:52:36.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.812+0000 7f3890f4f640 1 -- 172.21.15.78:0/2415191507 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884034a10 con 0x7f3878004d00 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f38941b5640 1 -- 172.21.15.78:0/2415191507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3878004d00 msgr2=0x7f3878005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f38941b5640 1 --2- 172.21.15.78:0/2415191507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3878004d00 0x7f3878005180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f38840013d0 tx=0x7f38780057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f3892752640 1 -- 172.21.15.78:0/2415191507 reap_dead start 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f38941b5640 1 -- 172.21.15.78:0/2415191507 shutdown_connections 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f38941b5640 1 -- 172.21.15.78:0/2415191507 >> 172.21.15.78:0/2415191507 conn(0x7f388c0718e0 msgr2=0x7f388c071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f38941b5640 1 -- 172.21.15.78:0/2415191507 shutdown_connections 2024-02-19T14:52:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.813+0000 7f38941b5640 1 -- 172.21.15.78:0/2415191507 wait complete. 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 Processor -- start 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 -- start start 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f388c14c210 0x7f388c1ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c1ebda0 0x7f388c1eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 0x7f388c1438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1edea0 con 0x7f388c143450 2024-02-19T14:52:36.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1ee460 con 0x7f388c14c210 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.814+0000 7f38941b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1eee30 con 0x7f388c1ebda0 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 0x7f388c1438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 0x7f388c1438d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53932/0 (socket says 172.21.15.78:53932) 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 -- 172.21.15.78:0/2691633740 learned_addr learned my addr 172.21.15.78:0/2691633740 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3891750640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c1ebda0 0x7f388c1eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3891f51640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f388c14c210 0x7f388c1ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 -- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c1ebda0 msgr2=0x7f388c1eb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c1ebda0 0x7f388c1eb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 -- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f388c14c210 msgr2=0x7f388c1ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f388c14c210 0x7f388c1ee8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 -- 172.21.15.78:0/2691633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c143e10 con 0x7f388c143450 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3891750640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c1ebda0 0x7f388c1eb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f3892752640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 0x7f388c1438d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f38840013d0 tx=0x7f3884001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38840019c0 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3884034430 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c14f760 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38840347c0 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.815+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c14fd20 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.816+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c14fd20 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.817+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3884031030 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.817+0000 7f38827fc640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f387006b810 0x7f387006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.817+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f38840ac120 con 0x7f388c143450 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.817+0000 7f3891f51640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f387006b810 0x7f387006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:36.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.818+0000 7f3891f51640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f387006b810 0x7f387006dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f388c14f6b0 tx=0x7f387c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:36.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.822+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f38840769b0 con 0x7f388c143450 2024-02-19T14:52:37.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.999+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f388c14fd20 con 0x7f388c143450 2024-02-19T14:52:37.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:36.999+0000 7f38827fc640 1 -- 172.21.15.78:0/2691633740 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f388407a070 con 0x7f388c143450 2024-02-19T14:52:37.002 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f387006b810 msgr2=0x7f387006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f387006b810 0x7f387006dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f388c14f6b0 tx=0x7f387c009040 comp rx=0 tx=0).stop 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 msgr2=0x7f388c1438d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 0x7f388c1438d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f38840013d0 tx=0x7f3884001360 comp rx=0 tx=0).stop 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 shutdown_connections 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f387006b810 0x7f387006dcd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c143450 0x7f388c1438d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f388c1ebda0 0x7f388c1eb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 --2- 172.21.15.78:0/2691633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f388c14c210 0x7f388c1ee8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 >> 172.21.15.78:0/2691633740 conn(0x7f388c0718e0 msgr2=0x7f388c153450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:37.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 shutdown_connections 2024-02-19T14:52:37.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.001+0000 7f38941b5640 1 -- 172.21.15.78:0/2691633740 wait complete. 2024-02-19T14:52:37.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:37.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.116+0000 7fa572d16640 1 Processor -- start 2024-02-19T14:52:37.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.116+0000 7fa572d16640 1 -- start start 2024-02-19T14:52:37.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa572d16640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa56c152700 0x7fa56c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa572d16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 0x7fa56c15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa572d16640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c1551f0 con 0x7fa56c0c5730 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa572d16640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c155490 con 0x7fa56c152700 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa572d16640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c155770 con 0x7fa56c0c8e00 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa5712b3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 0x7fa56c15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa570ab2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa56c152700 0x7fa56c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa5712b3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 0x7fa56c15f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:45528/0 (socket says 172.21.15.78:45528) 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa5712b3640 1 -- 172.21.15.78:0/3676924915 learned_addr learned my addr 172.21.15.78:0/3676924915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa570ab2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa56c152700 0x7fa56c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:48668/0 (socket says 172.21.15.78:48668) 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa56bfff640 1 --1- 172.21.15.78:0/3676924915 >> v1:172.21.15.150:6789/0 conn(0x7fa56c0c5730 0x7fa56c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58884/0 (socket says 172.21.15.78:58884) 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa56b7fe640 1 -- 172.21.15.78:0/3676924915 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129223257 0 0) 0x7fa56c1551f0 con 0x7fa56c0c5730 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.117+0000 7fa56b7fe640 1 -- 172.21.15.78:0/3676924915 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa558003520 con 0x7fa56c0c5730 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.118+0000 7fa570ab2640 1 -- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 msgr2=0x7fa56c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.118+0000 7fa570ab2640 1 --2- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 0x7fa56c15f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.118+0000 7fa570ab2640 1 -- 172.21.15.78:0/3676924915 >> v1:172.21.15.150:6789/0 conn(0x7fa56c0c5730 legacy=0x7fa56c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.118+0000 7fa570ab2640 1 -- 172.21.15.78:0/3676924915 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c155a50 con 0x7fa56c152700 2024-02-19T14:52:37.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.118+0000 7fa5712b3640 1 --2- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 0x7fa56c15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.119+0000 7fa570ab2640 1 --2- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c152700 0x7fa56c0c51f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0097a0 tx=0x7fa55c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c362253fb389f86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.119+0000 7fa56b7fe640 1 -- 172.21.15.78:0/3676924915 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c031830 con 0x7fa56c152700 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.119+0000 7fa56b7fe640 1 -- 172.21.15.78:0/3676924915 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55c031990 con 0x7fa56c152700 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.119+0000 7fa56b7fe640 1 -- 172.21.15.78:0/3676924915 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c031d00 con 0x7fa56c152700 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.119+0000 7fa572d16640 1 -- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c152700 msgr2=0x7fa56c0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.119+0000 7fa572d16640 1 --2- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c152700 0x7fa56c0c51f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0097a0 tx=0x7fa55c02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.120+0000 7fa572d16640 1 -- 172.21.15.78:0/3676924915 shutdown_connections 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.120+0000 7fa572d16640 1 --2- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c0c8e00 0x7fa56c15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.120+0000 7fa572d16640 1 --2- 172.21.15.78:0/3676924915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c152700 0x7fa56c0c51f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.120+0000 7fa572d16640 1 -- 172.21.15.78:0/3676924915 >> 172.21.15.78:0/3676924915 conn(0x7fa56c0cac90 msgr2=0x7fa56c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:37.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.120+0000 7fa572d16640 1 -- 172.21.15.78:0/3676924915 shutdown_connections 2024-02-19T14:52:37.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.120+0000 7fa572d16640 1 -- 172.21.15.78:0/3676924915 wait complete. 2024-02-19T14:52:37.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 Processor -- start 2024-02-19T14:52:37.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 -- start start 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c0c5730 0x7fa56c0c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 0x7fa56c1e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c152700 0x7fa56c1e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0aa280 con 0x7fa56c0c8e00 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0ace70 con 0x7fa56c0c5730 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.121+0000 7fa572d16640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1ee920 con 0x7fa56c152700 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 0x7fa56c1e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 0x7fa56c1e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53940/0 (socket says 172.21.15.78:53940) 2024-02-19T14:52:37.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 -- 172.21.15.78:0/2186688747 learned_addr learned my addr 172.21.15.78:0/2186688747 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa5712b3640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c152700 0x7fa56c1e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa570ab2640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c0c5730 0x7fa56c0c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 -- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c152700 msgr2=0x7fa56c1e7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c152700 0x7fa56c1e7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 -- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c0c5730 msgr2=0x7fa56c0c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c0c5730 0x7fa56c0c0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 -- 172.21.15.78:0/2186688747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564009b30 con 0x7fa56c0c8e00 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa56bfff640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 0x7fa56c1e7780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa55400c430 tx=0x7fa55400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5540074e0 con 0x7fa56c0c8e00 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa554007660 con 0x7fa56c0c8e00 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c009400 con 0x7fa56c0c8e00 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554004a20 con 0x7fa56c0c8e00 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.122+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c0aba40 con 0x7fa56c0c8e00 2024-02-19T14:52:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.124+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c074990 con 0x7fa56c0c8e00 2024-02-19T14:52:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.124+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa554004b80 con 0x7fa56c0c8e00 2024-02-19T14:52:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.124+0000 7fa5697fa640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa54c06b810 0x7fa54c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.124+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa554089da0 con 0x7fa56c0c8e00 2024-02-19T14:52:37.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.125+0000 7fa570ab2640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa54c06b810 0x7fa54c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.125+0000 7fa570ab2640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa54c06b810 0x7fa54c06dcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa55c036000 tx=0x7fa55c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.129+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa5540581b0 con 0x7fa56c0c8e00 2024-02-19T14:52:37.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.305+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa56c0c8770 con 0x7fa56c0c8e00 2024-02-19T14:52:37.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.306+0000 7fa5697fa640 1 -- 172.21.15.78:0/2186688747 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7fa554057d40 con 0x7fa56c0c8e00 2024-02-19T14:52:37.309 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:37.309 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa54c06b810 msgr2=0x7fa54c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa54c06b810 0x7fa54c06dcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa55c036000 tx=0x7fa55c03c040 comp rx=0 tx=0).stop 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 msgr2=0x7fa56c1e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 0x7fa56c1e7780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa55400c430 tx=0x7fa55400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 shutdown_connections 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa54c06b810 0x7fa54c06dcd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56c0c8e00 0x7fa56c1e7780 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa56c152700 0x7fa56c1e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 --2- 172.21.15.78:0/2186688747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa56c0c5730 0x7fa56c0c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 >> 172.21.15.78:0/2186688747 conn(0x7fa56c0cac90 msgr2=0x7fa56c14fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 shutdown_connections 2024-02-19T14:52:37.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.309+0000 7fa572d16640 1 -- 172.21.15.78:0/2186688747 wait complete. 2024-02-19T14:52:37.326 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:37.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.426+0000 7f952f31e640 1 Processor -- start 2024-02-19T14:52:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.427+0000 7f952f31e640 1 -- start start 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.427+0000 7f952f31e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95280c3d90 0x7f952815e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.427+0000 7f952f31e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952806ff90 con 0x7f95280719e0 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.427+0000 7f952f31e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9528156fc0 con 0x7f952814d210 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.427+0000 7f952f31e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9528070270 con 0x7f95280c3d90 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.427+0000 7f952c8b9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f95280719e0 0x7f95280c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51604/0 (socket says 172.21.15.78:51604) 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 -- 172.21.15.78:0/2833194614 learned_addr learned my addr 172.21.15.78:0/2833194614 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490441139 0 0) 0x7f952806ff90 con 0x7f95280719e0 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9514003520 con 0x7f95280719e0 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952d8bb640 1 --2- 172.21.15.78:0/2833194614 >> v2:172.21.15.150:3300/0 conn(0x7f95280c3d90 0x7f952815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1939646925 0 0) 0x7f9514003520 con 0x7f95280719e0 2024-02-19T14:52:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 >> v2:172.21.15.150:3300/0 conn(0x7f95280c3d90 msgr2=0x7f952815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 --2- 172.21.15.78:0/2833194614 >> v2:172.21.15.150:3300/0 conn(0x7f95280c3d90 0x7f952815e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 >> v1:172.21.15.150:6789/0 conn(0x7f952814d210 legacy=0x7f95280c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95281572a0 con 0x7f95280719e0 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9518001670 con 0x7f95280719e0 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 >> v1:172.21.15.78:6789/0 conn(0x7f95280719e0 legacy=0x7f95280c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9514003970 0x7f9514003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f9514004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9514005180 0x7f951400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9528070270 con 0x7f95140047c0 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951400c120 con 0x7f9514003970 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951400c2e0 con 0x7f9514005180 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f9514004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952d0ba640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9514003970 0x7f9514003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952d8bb640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9514005180 0x7f951400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 -- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9514005180 msgr2=0x7f951400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9514005180 0x7f951400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 -- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9514003970 msgr2=0x7f9514003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9514003970 0x7f9514003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952c8b9640 1 -- 172.21.15.78:0/2833194614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9514003520 con 0x7f95140047c0 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952d0ba640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9514003970 0x7f9514003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.428+0000 7f952d8bb640 1 -- 172.21.15.78:0/2833194614 reap_dead start 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.429+0000 7f952c8b9640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f9514004c40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9518007fd0 tx=0x7f95140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b92a60d835d4656 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.429+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9518005ad0 con 0x7f95140047c0 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.429+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9518005c50 con 0x7f95140047c0 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.429+0000 7f951ffff640 1 -- 172.21.15.78:0/2833194614 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951802c810 con 0x7f95140047c0 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 -- 172.21.15.78:0/2833194614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 msgr2=0x7f9514004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f9514004c40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9518007fd0 tx=0x7f95140057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 -- 172.21.15.78:0/2833194614 shutdown_connections 2024-02-19T14:52:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 --2- 172.21.15.78:0/2833194614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f9514004c40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 -- 172.21.15.78:0/2833194614 >> 172.21.15.78:0/2833194614 conn(0x7f95280c5050 msgr2=0x7f95280c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 -- 172.21.15.78:0/2833194614 shutdown_connections 2024-02-19T14:52:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.430+0000 7f952f31e640 1 -- 172.21.15.78:0/2833194614 wait complete. 2024-02-19T14:52:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.431+0000 7f952f31e640 1 Processor -- start 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.431+0000 7f952f31e640 1 -- start start 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952f31e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f95281ebfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952f31e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9528153790 0x7f95281ec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952f31e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9528149610 0x7f9528149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952f31e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95280c3970 con 0x7f95140047c0 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952f31e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95281eca20 con 0x7f9528149610 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952f31e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952814de70 con 0x7f9528153790 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f95281ebfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f95281ebfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53948/0 (socket says 172.21.15.78:53948) 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 -- 172.21.15.78:0/3350737151 learned_addr learned my addr 172.21.15.78:0/3350737151 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952c8b9640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9528153790 0x7f95281ec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d8bb640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9528149610 0x7f9528149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 -- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9528153790 msgr2=0x7f95281ec4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9528153790 0x7f95281ec4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 -- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9528149610 msgr2=0x7f9528149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9528149610 0x7f9528149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 -- 172.21.15.78:0/3350737151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952814d630 con 0x7f95140047c0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d8bb640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9528149610 0x7f9528149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952c8b9640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9528153790 0x7f95281ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.432+0000 7f952d0ba640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f95281ebfa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f952400e480 tx=0x7f952400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.433+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524017070 con 0x7f95140047c0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.433+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95240052e0 con 0x7f95140047c0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.433+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952814e0c0 con 0x7f95140047c0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.433+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524005490 con 0x7f95140047c0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.433+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95281ecc00 con 0x7f95140047c0 2024-02-19T14:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.434+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95281ecd90 con 0x7f95140047c0 2024-02-19T14:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.434+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9524007020 con 0x7f95140047c0 2024-02-19T14:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.435+0000 7f951dffb640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f950c06b840 0x7f950c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.435+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9524089a40 con 0x7f95140047c0 2024-02-19T14:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.435+0000 7f952c8b9640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f950c06b840 0x7f950c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.435+0000 7f952c8b9640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f950c06b840 0x7f950c06dd00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9518034000 tx=0x7f9518009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.443+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f952408be60 con 0x7f95140047c0 2024-02-19T14:52:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.690+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9528074990 con 0x7f95140047c0 2024-02-19T14:52:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.690+0000 7f951dffb640 1 -- 172.21.15.78:0/3350737151 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f95240541b0 con 0x7f95140047c0 2024-02-19T14:52:37.693 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f950c06b840 msgr2=0x7f950c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f950c06b840 0x7f950c06dd00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9518034000 tx=0x7f9518009030 comp rx=0 tx=0).stop 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 msgr2=0x7f95281ebfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f95281ebfa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f952400e480 tx=0x7f952400e940 comp rx=0 tx=0).stop 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 shutdown_connections 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f950c06b840 0x7f950c06dd00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9528149610 0x7f9528149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9528153790 0x7f95281ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 --2- 172.21.15.78:0/3350737151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95140047c0 0x7f95281ebfa0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 >> 172.21.15.78:0/3350737151 conn(0x7f95280c5050 msgr2=0x7f95280a8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 shutdown_connections 2024-02-19T14:52:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.692+0000 7f952f31e640 1 -- 172.21.15.78:0/3350737151 wait complete. 2024-02-19T14:52:37.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.807+0000 7fdf26c12640 1 Processor -- start 2024-02-19T14:52:37.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.807+0000 7fdf26c12640 1 -- start start 2024-02-19T14:52:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf26c12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200c7f50 0x7fdf20160850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf26c12640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200c8380 con 0x7fdf200c7b70 2024-02-19T14:52:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf26c12640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2006db90 con 0x7fdf20155bb0 2024-02-19T14:52:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf26c12640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf20154ec0 con 0x7fdf200c7f50 2024-02-19T14:52:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf249ae640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdf200c7b70 0x7fdf201584b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52626/0 (socket says 172.21.15.78:52626) 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf249ae640 1 -- 172.21.15.78:0/843933904 learned_addr learned my addr 172.21.15.78:0/843933904 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32185935 0 0) 0x7fdf200c8380 con 0x7fdf200c7b70 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf10003520 con 0x7fdf200c7b70 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf251af640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200c7f50 0x7fdf20160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1477134886 0 0) 0x7fdf10003520 con 0x7fdf200c7b70 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200c7f50 msgr2=0x7fdf20160850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200c7f50 0x7fdf20160850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 >> v1:172.21.15.150:6789/0 conn(0x7fdf20155bb0 legacy=0x7fdf2006d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf201550d0 con 0x7fdf200c7b70 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fdf14001690 con 0x7fdf200c7b70 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 >> v1:172.21.15.78:6789/0 conn(0x7fdf200c7b70 legacy=0x7fdf201584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100039a0 0x7fdf10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf100047c0 0x7fdf10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.808+0000 7fdf1f7fe640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf100051d0 0x7fdf1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf20154ec0 con 0x7fdf100051d0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1000c0d0 con 0x7fdf100039a0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1000c290 con 0x7fdf100047c0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200c7f50 0x7fdf20160850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf100051d0 0x7fdf1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf249ae640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100039a0 0x7fdf10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf1ffff640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf100047c0 0x7fdf10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 -- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf100047c0 msgr2=0x7fdf10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf100047c0 0x7fdf10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 -- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100039a0 msgr2=0x7fdf10003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100039a0 0x7fdf10003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf251af640 1 -- 172.21.15.78:0/843933904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10003520 con 0x7fdf100051d0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.809+0000 7fdf1ffff640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf100047c0 0x7fdf10004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf251af640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf100051d0 0x7fdf1000b980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009f00 tx=0x7fdf100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e0676a95ccba9e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1802cc70 con 0x7fdf100051d0 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1802cdd0 con 0x7fdf100051d0 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf1f7fe640 1 -- 172.21.15.78:0/843933904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18003bb0 con 0x7fdf100051d0 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf26c12640 1 -- 172.21.15.78:0/843933904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf100051d0 msgr2=0x7fdf1000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf26c12640 1 --2- 172.21.15.78:0/843933904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf100051d0 0x7fdf1000b980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009f00 tx=0x7fdf100057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf251af640 1 -- 172.21.15.78:0/843933904 reap_dead start 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf26c12640 1 -- 172.21.15.78:0/843933904 shutdown_connections 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf26c12640 1 -- 172.21.15.78:0/843933904 >> 172.21.15.78:0/843933904 conn(0x7fdf200c5050 msgr2=0x7fdf200c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf26c12640 1 -- 172.21.15.78:0/843933904 shutdown_connections 2024-02-19T14:52:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.810+0000 7fdf26c12640 1 -- 172.21.15.78:0/843933904 wait complete. 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.811+0000 7fdf26c12640 1 Processor -- start 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.811+0000 7fdf26c12640 1 -- start start 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf26c12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100051d0 0x7fdf200ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf26c12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200ae170 0x7fdf2014dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf26c12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 0x7fdf201ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf26c12640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2014f140 con 0x7fdf2014e1a0 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf26c12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf20151ec0 con 0x7fdf100051d0 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf26c12640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf20152440 con 0x7fdf200ae170 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf251af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 0x7fdf201ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf251af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 0x7fdf201ee3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41046/0 (socket says 172.21.15.78:41046) 2024-02-19T14:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.812+0000 7fdf251af640 1 -- 172.21.15.78:0/4148984791 learned_addr learned my addr 172.21.15.78:0/4148984791 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf251af640 1 -- 172.21.15.78:0/4148984791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200ae170 msgr2=0x7fdf2014dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf251af640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200ae170 0x7fdf2014dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf251af640 1 -- 172.21.15.78:0/4148984791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100051d0 msgr2=0x7fdf200ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf251af640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100051d0 0x7fdf200ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf251af640 1 -- 172.21.15.78:0/4148984791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2014e650 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf251af640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 0x7fdf201ee3f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdf18007360 tx=0x7fdf18004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1803c070 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf18003e10 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2014ea20 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18004710 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.813+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf200adb40 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.814+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf20074990 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.815+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdf18037460 con 0x7fdf2014e1a0 2024-02-19T14:52:37.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.815+0000 7fdf1d7fa640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf0406b810 0x7fdf0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.815+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdf180abac0 con 0x7fdf2014e1a0 2024-02-19T14:52:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.815+0000 7fdf249ae640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf0406b810 0x7fdf0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.815+0000 7fdf249ae640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf0406b810 0x7fdf0406dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdf14001b60 tx=0x7fdf14001ad0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:37.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.819+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdf18076230 con 0x7fdf2014e1a0 2024-02-19T14:52:37.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.995+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf201ee150 con 0x7fdf2014e1a0 2024-02-19T14:52:37.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.996+0000 7fdf1d7fa640 1 -- 172.21.15.78:0/4148984791 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdf18046020 con 0x7fdf2014e1a0 2024-02-19T14:52:37.998 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf0406b810 msgr2=0x7fdf0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf0406b810 0x7fdf0406dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdf14001b60 tx=0x7fdf14001ad0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 msgr2=0x7fdf201ee3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 0x7fdf201ee3f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdf18007360 tx=0x7fdf18004ab0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 shutdown_connections 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf0406b810 0x7fdf0406dcd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf2014e1a0 0x7fdf201ee3f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf200ae170 0x7fdf2014dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 --2- 172.21.15.78:0/4148984791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf100051d0 0x7fdf200ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 >> 172.21.15.78:0/4148984791 conn(0x7fdf200c5050 msgr2=0x7fdf200c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 shutdown_connections 2024-02-19T14:52:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:37.997+0000 7fdf26c12640 1 -- 172.21.15.78:0/4148984791 wait complete. 2024-02-19T14:52:38.011 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.115+0000 7f90e233e640 1 Processor -- start 2024-02-19T14:52:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.115+0000 7f90e233e640 1 -- start start 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e233e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f90dc150460 0x7f90dc0c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e233e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0721a0 0x7f90dc0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e233e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90dc0c7d90 0x7f90dc0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e233e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc153e80 con 0x7f90dc150460 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e233e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc072c60 con 0x7f90dc0c7d90 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e233e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc1538c0 con 0x7f90dc0721a0 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f90dc150460 0x7f90dc0c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f90dc150460 0x7f90dc0c1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41056/0 (socket says 172.21.15.78:41056) 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 -- 172.21.15.78:0/2750348405 learned_addr learned my addr 172.21.15.78:0/2750348405 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90db7fe640 1 --2- 172.21.15.78:0/2750348405 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0721a0 0x7f90dc0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90e08db640 1 --2- 172.21.15.78:0/2750348405 >> v2:172.21.15.150:3300/0 conn(0x7f90dc0c7d90 0x7f90dc0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 -- 172.21.15.78:0/2750348405 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0721a0 msgr2=0x7f90dc0c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 --2- 172.21.15.78:0/2750348405 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0721a0 0x7f90dc0c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 -- 172.21.15.78:0/2750348405 >> v2:172.21.15.150:3300/0 conn(0x7f90dc0c7d90 msgr2=0x7f90dc0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 --2- 172.21.15.78:0/2750348405 >> v2:172.21.15.150:3300/0 conn(0x7f90dc0c7d90 0x7f90dc0c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.116+0000 7f90dbfff640 1 -- 172.21.15.78:0/2750348405 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc153ba0 con 0x7f90dc150460 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.117+0000 7f90dbfff640 1 --2- 172.21.15.78:0/2750348405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc0c1810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f90dc070410 tx=0x7f90d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8e5a8def7f9436 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.117+0000 7f90daffd640 1 -- 172.21.15.78:0/2750348405 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d402ca70 con 0x7f90dc150460 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.117+0000 7f90daffd640 1 -- 172.21.15.78:0/2750348405 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d402cbd0 con 0x7f90dc150460 2024-02-19T14:52:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.117+0000 7f90daffd640 1 -- 172.21.15.78:0/2750348405 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d40343f0 con 0x7f90dc150460 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 -- 172.21.15.78:0/2750348405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 msgr2=0x7f90dc0c1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 --2- 172.21.15.78:0/2750348405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc0c1810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f90dc070410 tx=0x7f90d402bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 -- 172.21.15.78:0/2750348405 shutdown_connections 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 --2- 172.21.15.78:0/2750348405 >> v2:172.21.15.150:3300/0 conn(0x7f90dc0c7d90 0x7f90dc0c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 --2- 172.21.15.78:0/2750348405 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0721a0 0x7f90dc0c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 --2- 172.21.15.78:0/2750348405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc0c1810 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 -- 172.21.15.78:0/2750348405 >> 172.21.15.78:0/2750348405 conn(0x7f90dc06c170 msgr2=0x7f90dc06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 -- 172.21.15.78:0/2750348405 shutdown_connections 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.118+0000 7f90e233e640 1 -- 172.21.15.78:0/2750348405 wait complete. 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.119+0000 7f90e233e640 1 Processor -- start 2024-02-19T14:52:38.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 -- start start 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90dc0721a0 0x7f90dc0c0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0c7d90 0x7f90dc156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc155dd0 con 0x7f90dc150460 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0c1050 con 0x7f90dc0721a0 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e233e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0c1330 con 0x7f90dc0c7d90 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e08db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e08db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41064/0 (socket says 172.21.15.78:41064) 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90e08db640 1 -- 172.21.15.78:0/948768530 learned_addr learned my addr 172.21.15.78:0/948768530 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90db7fe640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0c7d90 0x7f90dc156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.120+0000 7f90dbfff640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90dc0721a0 0x7f90dc0c0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e08db640 1 -- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0c7d90 msgr2=0x7f90dc156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e08db640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0c7d90 0x7f90dc156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e08db640 1 -- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90dc0721a0 msgr2=0x7f90dc0c0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e08db640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90dc0721a0 0x7f90dc0c0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e08db640 1 -- 172.21.15.78:0/948768530 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4009b30 con 0x7f90dc150460 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e08db640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc149c00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00e7f0 tx=0x7f90cc00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc009510 con 0x7f90dc150460 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc00ee40 con 0x7f90dc150460 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc156740 con 0x7f90dc150460 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00cb80 con 0x7f90dc150460 2024-02-19T14:52:38.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.121+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc14a140 con 0x7f90dc150460 2024-02-19T14:52:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.122+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc070150 con 0x7f90dc150460 2024-02-19T14:52:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.123+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f90cc00cce0 con 0x7f90dc150460 2024-02-19T14:52:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.123+0000 7f90d8ff9640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90c806b810 0x7f90c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.124+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f90cc08a470 con 0x7f90dc150460 2024-02-19T14:52:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.124+0000 7f90dbfff640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90c806b810 0x7f90c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.124+0000 7f90dbfff640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90c806b810 0x7f90c806dcd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f90d4035000 tx=0x7f90d4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.128+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f90cc054c90 con 0x7f90dc150460 2024-02-19T14:52:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.306+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f90dc156c60 con 0x7f90dc150460 2024-02-19T14:52:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.307+0000 7f90d8ff9640 1 -- 172.21.15.78:0/948768530 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f90dc156c60 con 0x7f90dc150460 2024-02-19T14:52:38.310 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90c806b810 msgr2=0x7f90c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90c806b810 0x7f90c806dcd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f90d4035000 tx=0x7f90d4030000 comp rx=0 tx=0).stop 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 msgr2=0x7f90dc149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc149c00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00e7f0 tx=0x7f90cc00ebf0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 shutdown_connections 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90c806b810 0x7f90c806dcd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90dc150460 0x7f90dc149c00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90dc0c7d90 0x7f90dc156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 --2- 172.21.15.78:0/948768530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90dc0721a0 0x7f90dc0c0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 >> 172.21.15.78:0/948768530 conn(0x7f90dc06c170 msgr2=0x7f90dc06c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 shutdown_connections 2024-02-19T14:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.310+0000 7f90e233e640 1 -- 172.21.15.78:0/948768530 wait complete. 2024-02-19T14:52:38.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.429+0000 7f761a72e640 1 Processor -- start 2024-02-19T14:52:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.429+0000 7f761a72e640 1 -- start start 2024-02-19T14:52:38.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f761a72e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f7614155900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f761a72e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f761406da10 0x7f76140723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f761a72e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140c1d80 con 0x7f7614072910 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f761a72e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761406caf0 con 0x7f761406da10 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f761a72e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76140c37e0 con 0x7f76140acb50 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f7618ccb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7614072910 0x7f7614158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52630/0 (socket says 172.21.15.78:52630) 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f7618ccb640 1 -- 172.21.15.78:0/3223865980 learned_addr learned my addr 172.21.15.78:0/3223865980 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f7613fff640 1 --2- 172.21.15.78:0/3223865980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f7614155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f76137fe640 1 --2- 172.21.15.78:0/3223865980 >> v2:172.21.15.150:3300/0 conn(0x7f761406da10 0x7f76140723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f7612ffd640 1 -- 172.21.15.78:0/3223865980 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215756628 0 0) 0x7f76140c1d80 con 0x7f7614072910 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.430+0000 7f7612ffd640 1 -- 172.21.15.78:0/3223865980 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc003520 con 0x7f7614072910 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.431+0000 7f7613fff640 1 -- 172.21.15.78:0/3223865980 >> v2:172.21.15.150:3300/0 conn(0x7f761406da10 msgr2=0x7f76140723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.431+0000 7f7613fff640 1 --2- 172.21.15.78:0/3223865980 >> v2:172.21.15.150:3300/0 conn(0x7f761406da10 0x7f76140723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.431+0000 7f7613fff640 1 -- 172.21.15.78:0/3223865980 >> v1:172.21.15.78:6789/0 conn(0x7f7614072910 legacy=0x7f7614158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.431+0000 7f7613fff640 1 -- 172.21.15.78:0/3223865980 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761406c5a0 con 0x7f76140acb50 2024-02-19T14:52:38.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.431+0000 7f76137fe640 1 --2- 172.21.15.78:0/3223865980 >> v2:172.21.15.150:3300/0 conn(0x7f761406da10 0x7f76140723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.431+0000 7f7613fff640 1 --2- 172.21.15.78:0/3223865980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f7614155900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f76080097a0 tx=0x7f760802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6453b409d58f0e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f7612ffd640 1 -- 172.21.15.78:0/3223865980 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608031780 con 0x7f76140acb50 2024-02-19T14:52:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f7612ffd640 1 -- 172.21.15.78:0/3223865980 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76080318e0 con 0x7f76140acb50 2024-02-19T14:52:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f7612ffd640 1 -- 172.21.15.78:0/3223865980 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608031c70 con 0x7f76140acb50 2024-02-19T14:52:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 -- 172.21.15.78:0/3223865980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 msgr2=0x7f7614155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 --2- 172.21.15.78:0/3223865980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f7614155900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f76080097a0 tx=0x7f760802b820 comp rx=0 tx=0).stop 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 -- 172.21.15.78:0/3223865980 shutdown_connections 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 --2- 172.21.15.78:0/3223865980 >> v2:172.21.15.150:3300/0 conn(0x7f761406da10 0x7f76140723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 --2- 172.21.15.78:0/3223865980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f7614155900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 -- 172.21.15.78:0/3223865980 >> 172.21.15.78:0/3223865980 conn(0x7f76140718e0 msgr2=0x7f7614071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 -- 172.21.15.78:0/3223865980 shutdown_connections 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.432+0000 7f761a72e640 1 -- 172.21.15.78:0/3223865980 wait complete. 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.433+0000 7f761a72e640 1 Processor -- start 2024-02-19T14:52:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 -- start start 2024-02-19T14:52:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761406da10 0x7f761406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 0x7f7614148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f76141494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141569a0 con 0x7f7614072910 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141ea420 con 0x7f761406da10 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f761a72e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76141ea700 con 0x7f76140acb50 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f76137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 0x7f7614148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f76137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 0x7f7614148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41068/0 (socket says 172.21.15.78:41068) 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.434+0000 7f76137fe640 1 -- 172.21.15.78:0/2619603648 learned_addr learned my addr 172.21.15.78:0/2619603648 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f7618ccb640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f76141494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f7613fff640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761406da10 0x7f761406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f76137fe640 1 -- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 msgr2=0x7f76141494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f76137fe640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f76141494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f76137fe640 1 -- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761406da10 msgr2=0x7f761406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f76137fe640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761406da10 0x7f761406ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f76137fe640 1 -- 172.21.15.78:0/2619603648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7604009400 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f76137fe640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 0x7f7614148f60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f760400e960 tx=0x7f760400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760400ca00 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760400cb60 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7608009400 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604013470 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.435+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76141eacc0 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.436+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7614074990 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.437+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f76040135d0 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.437+0000 7f7610ff9640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f760006b810 0x7f760006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.437+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f760408a760 con 0x7f7614072910 2024-02-19T14:52:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.437+0000 7f7613fff640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f760006b810 0x7f760006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.438+0000 7f7613fff640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f760006b810 0x7f760006dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7608036000 tx=0x7f760803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.445+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7604054e50 con 0x7f7614072910 2024-02-19T14:52:38.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.645+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f761406c2e0 con 0x7f7614072910 2024-02-19T14:52:38.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.645+0000 7f7610ff9640 1 -- 172.21.15.78:0/2619603648 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7604058510 con 0x7f7614072910 2024-02-19T14:52:38.648 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f760006b810 msgr2=0x7f760006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f760006b810 0x7f760006dcd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7608036000 tx=0x7f760803d040 comp rx=0 tx=0).stop 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 msgr2=0x7f7614148f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 0x7f7614148f60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f760400e960 tx=0x7f760400ee20 comp rx=0 tx=0).stop 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 shutdown_connections 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f760006b810 0x7f760006dcd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7614072910 0x7f7614148f60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76140acb50 0x7f76141494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 --2- 172.21.15.78:0/2619603648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761406da10 0x7f761406ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 >> 172.21.15.78:0/2619603648 conn(0x7f76140718e0 msgr2=0x7f7614157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 shutdown_connections 2024-02-19T14:52:38.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.647+0000 7f761a72e640 1 -- 172.21.15.78:0/2619603648 wait complete. 2024-02-19T14:52:38.673 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2024-02-19T14:52:38.673 INFO:tasks.cephfs_test_runner:test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:52:38.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:38.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.789+0000 7fbc3953e640 1 Processor -- start 2024-02-19T14:52:38.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.789+0000 7fbc3953e640 1 -- start start 2024-02-19T14:52:38.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc3953e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc3953e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc3406e830 0x7fbc340c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc3953e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3414f730 con 0x7fbc3406ff80 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc3953e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc340c1940 con 0x7fbc3406e830 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc3953e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc34072f70 con 0x7fbc340c6d60 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc3406e830 0x7fbc340c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc3406e830 0x7fbc340c0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41078/0 (socket says 172.21.15.78:41078) 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 -- 172.21.15.78:0/4053376321 learned_addr learned my addr 172.21.15.78:0/4053376321 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc32ffd640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 -- 172.21.15.78:0/4053376321 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 msgr2=0x7fbc34154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 -- 172.21.15.78:0/4053376321 >> v1:172.21.15.150:6789/0 conn(0x7fbc3406ff80 legacy=0x7fbc340c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc327fc640 1 -- 172.21.15.78:0/4053376321 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc340c1b50 con 0x7fbc3406e830 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.790+0000 7fbc32ffd640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.791+0000 7fbc327fc640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406e830 0x7fbc340c0fc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009870 tx=0x7fbc2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3275bff057424414 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.791+0000 7fbc31ffb640 1 -- 172.21.15.78:0/4053376321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28031640 con 0x7fbc3406e830 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.791+0000 7fbc31ffb640 1 -- 172.21.15.78:0/4053376321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc280317a0 con 0x7fbc3406e830 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.791+0000 7fbc31ffb640 1 -- 172.21.15.78:0/4053376321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28031b30 con 0x7fbc3406e830 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 -- 172.21.15.78:0/4053376321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406e830 msgr2=0x7fbc340c0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406e830 0x7fbc340c0fc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009870 tx=0x7fbc2802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 -- 172.21.15.78:0/4053376321 shutdown_connections 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406e830 0x7fbc340c0fc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 --2- 172.21.15.78:0/4053376321 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 -- 172.21.15.78:0/4053376321 >> 172.21.15.78:0/4053376321 conn(0x7fbc3406c170 msgr2=0x7fbc3406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 -- 172.21.15.78:0/4053376321 shutdown_connections 2024-02-19T14:52:38.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.792+0000 7fbc3953e640 1 -- 172.21.15.78:0/4053376321 wait complete. 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.794+0000 7fbc3953e640 1 Processor -- start 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.794+0000 7fbc3953e640 1 -- start start 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc3953e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc3406e830 0x7fbc341514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc3953e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 0x7fbc34151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc3953e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc3953e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3414f8e0 con 0x7fbc3406ff80 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc3953e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3414fb30 con 0x7fbc3406e830 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc3953e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341eb570 con 0x7fbc340c6d60 2024-02-19T14:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 0x7fbc34151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 0x7fbc34151a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41092/0 (socket says 172.21.15.78:41092) 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 -- 172.21.15.78:0/3592404264 learned_addr learned my addr 172.21.15.78:0/3592404264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc32ffd640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc3406e830 0x7fbc341514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc337fe640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 -- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 msgr2=0x7fbc34149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 -- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc3406e830 msgr2=0x7fbc341514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc3406e830 0x7fbc341514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc327fc640 1 -- 172.21.15.78:0/3592404264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc280094d0 con 0x7fbc3406ff80 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.795+0000 7fbc32ffd640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc3406e830 0x7fbc341514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.796+0000 7fbc327fc640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 0x7fbc34151a10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009870 tx=0x7fbc280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.796+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2803ac60 con 0x7fbc3406ff80 2024-02-19T14:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.796+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2803ade0 con 0x7fbc3406ff80 2024-02-19T14:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.796+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc341eb850 con 0x7fbc3406ff80 2024-02-19T14:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.796+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28044630 con 0x7fbc3406ff80 2024-02-19T14:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.796+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc340abd90 con 0x7fbc3406ff80 2024-02-19T14:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.797+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc34074990 con 0x7fbc3406ff80 2024-02-19T14:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.798+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbc2802f050 con 0x7fbc3406ff80 2024-02-19T14:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.798+0000 7fbc1b7fe640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc2006b810 0x7fbc2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.798+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbc280ac240 con 0x7fbc3406ff80 2024-02-19T14:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.798+0000 7fbc32ffd640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc2006b810 0x7fbc2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.799+0000 7fbc32ffd640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc2006b810 0x7fbc2006dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbc24006fd0 tx=0x7fbc24008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:38.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.802+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbc28076a40 con 0x7fbc3406ff80 2024-02-19T14:52:38.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.991+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc340c4e80 con 0x7fbc3406ff80 2024-02-19T14:52:38.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.992+0000 7fbc1b7fe640 1 -- 172.21.15.78:0/3592404264 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18502 (secure 0 0 0) 0x7fbc28031e60 con 0x7fbc3406ff80 2024-02-19T14:52:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:38.995 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:04.685260+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc2006b810 msgr2=0x7fbc2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc2006b810 0x7fbc2006dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbc24006fd0 tx=0x7fbc24008040 comp rx=0 tx=0).stop 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 msgr2=0x7fbc34151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 0x7fbc34151a10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009870 tx=0x7fbc280046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 shutdown_connections 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc2006b810 0x7fbc2006dcd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3406ff80 0x7fbc34151a10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc340c6d60 0x7fbc34149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 --2- 172.21.15.78:0/3592404264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc3406e830 0x7fbc341514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 >> 172.21.15.78:0/3592404264 conn(0x7fbc3406c170 msgr2=0x7fbc34155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 shutdown_connections 2024-02-19T14:52:38.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:38.994+0000 7fbc3953e640 1 -- 172.21.15.78:0/3592404264 wait complete. 2024-02-19T14:52:39.008 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:39.009 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.110+0000 7f3bbbadf640 1 Processor -- start 2024-02-19T14:52:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.110+0000 7f3bbbadf640 1 -- start start 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bbbadf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb40c8e00 0x7f3bb406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bbbadf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3bb406d910 0x7f3bb406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bbbadf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb406cd20 con 0x7f3bb4152700 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bbbadf640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb41551f0 con 0x7f3bb406d910 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bbbadf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4155490 con 0x7f3bb40c8e00 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3bb406d910 0x7f3bb406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3bb406d910 0x7f3bb406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41104/0 (socket says 172.21.15.78:41104) 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 -- 172.21.15.78:0/178783694 learned_addr learned my addr 172.21.15.78:0/178783694 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:39.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bb907a640 1 --2- 172.21.15.78:0/178783694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb40c8e00 0x7f3bb406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 -- 172.21.15.78:0/178783694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb40c8e00 msgr2=0x7f3bb406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 --2- 172.21.15.78:0/178783694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb40c8e00 0x7f3bb406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 -- 172.21.15.78:0/178783694 >> v1:172.21.15.150:6789/0 conn(0x7f3bb4152700 legacy=0x7f3bb40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.111+0000 7f3bba07c640 1 -- 172.21.15.78:0/178783694 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4155770 con 0x7f3bb406d910 2024-02-19T14:52:39.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.112+0000 7f3bba07c640 1 --2- 172.21.15.78:0/178783694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb406d910 0x7f3bb406c7e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009e50 tx=0x7f3bac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec35ab09be0d11ac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.112+0000 7f3bb8879640 1 -- 172.21.15.78:0/178783694 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac02c9e0 con 0x7f3bb406d910 2024-02-19T14:52:39.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.112+0000 7f3bb8879640 1 -- 172.21.15.78:0/178783694 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac02cb40 con 0x7f3bb406d910 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.112+0000 7f3bb8879640 1 -- 172.21.15.78:0/178783694 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0343f0 con 0x7f3bb406d910 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 -- 172.21.15.78:0/178783694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb406d910 msgr2=0x7f3bb406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/178783694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb406d910 0x7f3bb406c7e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009e50 tx=0x7f3bac02be40 comp rx=0 tx=0).stop 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 -- 172.21.15.78:0/178783694 shutdown_connections 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/178783694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb406d910 0x7f3bb406c7e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/178783694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb40c8e00 0x7f3bb406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 -- 172.21.15.78:0/178783694 >> 172.21.15.78:0/178783694 conn(0x7f3bb40cac90 msgr2=0x7f3bb40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 -- 172.21.15.78:0/178783694 shutdown_connections 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.113+0000 7f3bbbadf640 1 -- 172.21.15.78:0/178783694 wait complete. 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.114+0000 7f3bbbadf640 1 Processor -- start 2024-02-19T14:52:39.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.114+0000 7f3bbbadf640 1 -- start start 2024-02-19T14:52:39.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb406d910 0x7f3bb40c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 0x7f3bb40ab590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bb4152700 0x7f3bb41edf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40c1d00 con 0x7f3bb40c8e00 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40ab100 con 0x7f3bb4152700 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40ac460 con 0x7f3bb406d910 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 0x7f3bb40ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 0x7f3bb40ab590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41118/0 (socket says 172.21.15.78:41118) 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 -- 172.21.15.78:0/2045441480 learned_addr learned my addr 172.21.15.78:0/2045441480 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bba07c640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bb4152700 0x7f3bb41edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb987b640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb406d910 0x7f3bb40c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 -- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb406d910 msgr2=0x7f3bb40c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb406d910 0x7f3bb40c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 -- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bb4152700 msgr2=0x7f3bb41edf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bb4152700 0x7f3bb41edf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 -- 172.21.15.78:0/2045441480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac009b30 con 0x7f3bb40c8e00 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bba07c640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bb4152700 0x7f3bb41edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bb907a640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 0x7f3bb40ab590 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000c7f0 tx=0x7f3bb000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0004220 con 0x7f3bb40c8e00 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb00043a0 con 0x7f3bb40c8e00 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb41ee8c0 con 0x7f3bb40c8e00 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0007860 con 0x7f3bb40c8e00 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.115+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb41eeba0 con 0x7f3bb40c8e00 2024-02-19T14:52:39.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.117+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb40ab290 con 0x7f3bb40c8e00 2024-02-19T14:52:39.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.118+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3bb00079c0 con 0x7f3bb40c8e00 2024-02-19T14:52:39.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.118+0000 7f3baa7fc640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b9806b810 0x7f3b9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.118+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3bb008a2c0 con 0x7f3bb40c8e00 2024-02-19T14:52:39.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.118+0000 7f3bb987b640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b9806b810 0x7f3b9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.118+0000 7f3bb987b640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b9806b810 0x7f3b9806dcd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4006fd0 tx=0x7f3ba4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.126+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3bb0054a30 con 0x7f3bb40c8e00 2024-02-19T14:52:39.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.336+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bb4074990 con 0x7f3bb40c8e00 2024-02-19T14:52:39.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.337+0000 7f3baa7fc640 1 -- 172.21.15.78:0/2045441480 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3bb00580f0 con 0x7f3bb40c8e00 2024-02-19T14:52:39.339 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:39.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b9806b810 msgr2=0x7f3b9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b9806b810 0x7f3b9806dcd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4006fd0 tx=0x7f3ba4008040 comp rx=0 tx=0).stop 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 msgr2=0x7f3bb40ab590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 0x7f3bb40ab590 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000c7f0 tx=0x7f3bb000ccb0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 shutdown_connections 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b9806b810 0x7f3b9806dcd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb40c8e00 0x7f3bb40ab590 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bb4152700 0x7f3bb41edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 --2- 172.21.15.78:0/2045441480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3bb406d910 0x7f3bb40c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 >> 172.21.15.78:0/2045441480 conn(0x7f3bb40cac90 msgr2=0x7f3bb414e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 shutdown_connections 2024-02-19T14:52:39.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.339+0000 7f3bbbadf640 1 -- 172.21.15.78:0/2045441480 wait complete. 2024-02-19T14:52:39.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:39.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.460+0000 7fd935793640 1 Processor -- start 2024-02-19T14:52:39.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.460+0000 7fd935793640 1 -- start start 2024-02-19T14:52:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd935793640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9300c7b70 0x7fd9300c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd935793640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd930155bb0 0x7fd9301584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd935793640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd9301589f0 0x7fd93006d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd935793640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd930154ec0 con 0x7fd9301589f0 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd935793640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93006db90 con 0x7fd9300c7b70 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd935793640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9301550d0 con 0x7fd930155bb0 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd92f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd9301589f0 0x7fd93006d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd92f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd9301589f0 0x7fd93006d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41122/0 (socket says 172.21.15.78:41122) 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd92f7fe640 1 -- 172.21.15.78:0/3245378179 learned_addr learned my addr 172.21.15.78:0/3245378179 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd927fff640 1 --2- 172.21.15.78:0/3245378179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd930155bb0 0x7fd9301584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.461+0000 7fd92effd640 1 --2- 172.21.15.78:0/3245378179 >> v2:172.21.15.150:3300/0 conn(0x7fd9300c7b70 0x7fd9300c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.462+0000 7fd92f7fe640 1 -- 172.21.15.78:0/3245378179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd930155bb0 msgr2=0x7fd9301584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.462+0000 7fd92f7fe640 1 --2- 172.21.15.78:0/3245378179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd930155bb0 0x7fd9301584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.462+0000 7fd92f7fe640 1 -- 172.21.15.78:0/3245378179 >> v2:172.21.15.150:3300/0 conn(0x7fd9300c7b70 msgr2=0x7fd9300c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.462+0000 7fd92f7fe640 1 --2- 172.21.15.78:0/3245378179 >> v2:172.21.15.150:3300/0 conn(0x7fd9300c7b70 0x7fd9300c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.462+0000 7fd92f7fe640 1 -- 172.21.15.78:0/3245378179 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9301553b0 con 0x7fd9301589f0 2024-02-19T14:52:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.462+0000 7fd92effd640 1 --2- 172.21.15.78:0/3245378179 >> v2:172.21.15.150:3300/0 conn(0x7fd9300c7b70 0x7fd9300c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.463+0000 7fd92f7fe640 1 --2- 172.21.15.78:0/3245378179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9301589f0 0x7fd93006d040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd93006efc0 tx=0x7fd92802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58883dc64effd04a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.463+0000 7fd92e7fc640 1 -- 172.21.15.78:0/3245378179 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92802c9e0 con 0x7fd9301589f0 2024-02-19T14:52:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.463+0000 7fd92e7fc640 1 -- 172.21.15.78:0/3245378179 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92802cb40 con 0x7fd9301589f0 2024-02-19T14:52:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.463+0000 7fd92e7fc640 1 -- 172.21.15.78:0/3245378179 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280343f0 con 0x7fd9301589f0 2024-02-19T14:52:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 -- 172.21.15.78:0/3245378179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9301589f0 msgr2=0x7fd93006d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 --2- 172.21.15.78:0/3245378179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9301589f0 0x7fd93006d040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd93006efc0 tx=0x7fd92802be40 comp rx=0 tx=0).stop 2024-02-19T14:52:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 -- 172.21.15.78:0/3245378179 shutdown_connections 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 --2- 172.21.15.78:0/3245378179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9301589f0 0x7fd93006d040 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 --2- 172.21.15.78:0/3245378179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd930155bb0 0x7fd9301584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 --2- 172.21.15.78:0/3245378179 >> v2:172.21.15.150:3300/0 conn(0x7fd9300c7b70 0x7fd9300c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 -- 172.21.15.78:0/3245378179 >> 172.21.15.78:0/3245378179 conn(0x7fd9300c5050 msgr2=0x7fd9300c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 -- 172.21.15.78:0/3245378179 shutdown_connections 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.464+0000 7fd935793640 1 -- 172.21.15.78:0/3245378179 wait complete. 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.465+0000 7fd935793640 1 Processor -- start 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.465+0000 7fd935793640 1 -- start start 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd935793640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9300c7b70 0x7fd9300acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd935793640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 0x7fd9300ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd935793640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9301589f0 0x7fd93014f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd935793640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd930151850 con 0x7fd930155bb0 2024-02-19T14:52:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd935793640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300abdd0 con 0x7fd9300c7b70 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd935793640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300ac670 con 0x7fd9301589f0 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd927fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 0x7fd9300ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd927fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 0x7fd9300ad240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41136/0 (socket says 172.21.15.78:41136) 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd927fff640 1 -- 172.21.15.78:0/2837275271 learned_addr learned my addr 172.21.15.78:0/2837275271 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd92f7fe640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9301589f0 0x7fd93014f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.466+0000 7fd92effd640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9300c7b70 0x7fd9300acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd927fff640 1 -- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9301589f0 msgr2=0x7fd93014f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd927fff640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9301589f0 0x7fd93014f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd927fff640 1 -- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9300c7b70 msgr2=0x7fd9300acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd927fff640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9300c7b70 0x7fd9300acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd927fff640 1 -- 172.21.15.78:0/2837275271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd928009b30 con 0x7fd930155bb0 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd92effd640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9300c7b70 0x7fd9300acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd927fff640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 0x7fd9300ad240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd91800e9b0 tx=0x7fd91800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918009670 con 0x7fd930155bb0 2024-02-19T14:52:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91800c9a0 con 0x7fd930155bb0 2024-02-19T14:52:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9301571a0 con 0x7fd930155bb0 2024-02-19T14:52:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91800cd50 con 0x7fd930155bb0 2024-02-19T14:52:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.467+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9300ac020 con 0x7fd930155bb0 2024-02-19T14:52:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.468+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd930074990 con 0x7fd930155bb0 2024-02-19T14:52:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.468+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd918013470 con 0x7fd930155bb0 2024-02-19T14:52:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.469+0000 7fd9277fe640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd91006b810 0x7fd91006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.469+0000 7fd92effd640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd91006b810 0x7fd91006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.469+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd91808a540 con 0x7fd930155bb0 2024-02-19T14:52:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.470+0000 7fd92effd640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd91006b810 0x7fd91006dcd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd920006fd0 tx=0x7fd920008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.473+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd918054d60 con 0x7fd930155bb0 2024-02-19T14:52:39.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.690+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd9300ac7b0 con 0x7fd930155bb0 2024-02-19T14:52:39.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.691+0000 7fd9277fe640 1 -- 172.21.15.78:0/2837275271 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd918058420 con 0x7fd930155bb0 2024-02-19T14:52:39.693 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd91006b810 msgr2=0x7fd91006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd91006b810 0x7fd91006dcd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd920006fd0 tx=0x7fd920008040 comp rx=0 tx=0).stop 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 msgr2=0x7fd9300ad240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 0x7fd9300ad240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd91800e9b0 tx=0x7fd91800ee70 comp rx=0 tx=0).stop 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 shutdown_connections 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd91006b810 0x7fd91006dcd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd930155bb0 0x7fd9300ad240 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9301589f0 0x7fd93014f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 --2- 172.21.15.78:0/2837275271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9300c7b70 0x7fd9300acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 >> 172.21.15.78:0/2837275271 conn(0x7fd9300c5050 msgr2=0x7fd93006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 shutdown_connections 2024-02-19T14:52:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.693+0000 7fd935793640 1 -- 172.21.15.78:0/2837275271 wait complete. 2024-02-19T14:52:39.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.808+0000 7f483d3f7640 1 Processor -- start 2024-02-19T14:52:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.808+0000 7f483d3f7640 1 -- start start 2024-02-19T14:52:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f483d3f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 0x7f48380ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f483d3f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4838158c40 con 0x7f48380c55f0 2024-02-19T14:52:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f483d3f7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48380c5a60 con 0x7f48380bdf20 2024-02-19T14:52:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f483d3f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48380c8de0 con 0x7f48380ac390 2024-02-19T14:52:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48377fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f48380c55f0 0x7f483815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52644/0 (socket says 172.21.15.78:52644) 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48377fe640 1 -- 172.21.15.78:0/1214436188 learned_addr learned my addr 172.21.15.78:0/1214436188 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244194611 0 0) 0x7f4838158c40 con 0x7f48380c55f0 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4824003520 con 0x7f48380c55f0 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f4836ffd640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 0x7f48380ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1122296745 0 0) 0x7f4824003520 con 0x7f48380c55f0 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 msgr2=0x7f48380ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 0x7f48380ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 >> v1:172.21.15.150:6789/0 conn(0x7f48380bdf20 legacy=0x7f4838158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4838155a60 con 0x7f48380c55f0 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f48300019f0 con 0x7f48380c55f0 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 >> v1:172.21.15.78:6789/0 conn(0x7f48380c55f0 legacy=0x7f483815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48240039a0 0x7f4824003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 0x7f48240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48380c8de0 con 0x7f4824004d50 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482400bc00 con 0x7f48240039a0 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.809+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482400bdc0 con 0x7f48380ac390 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f48377fe640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f4836ffd640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48240039a0 0x7f4824003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f482ffff640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 0x7f48240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f48377fe640 1 -- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 msgr2=0x7f48240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f48377fe640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380ac390 0x7f48240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f48377fe640 1 -- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48240039a0 msgr2=0x7f4824003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f48377fe640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48240039a0 0x7f4824003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.810+0000 7f48377fe640 1 -- 172.21.15.78:0/1214436188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824003520 con 0x7f4824004d50 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f48377fe640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48240051d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f483000a800 tx=0x7f48240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e648188975bbc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300072d0 con 0x7f4824004d50 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483002f070 con 0x7f4824004d50 2024-02-19T14:52:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f48367fc640 1 -- 172.21.15.78:0/1214436188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830007690 con 0x7f4824004d50 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f483d3f7640 1 -- 172.21.15.78:0/1214436188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 msgr2=0x7f48240051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f483d3f7640 1 --2- 172.21.15.78:0/1214436188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48240051d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f483000a800 tx=0x7f48240057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f48377fe640 1 -- 172.21.15.78:0/1214436188 reap_dead start 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f483d3f7640 1 -- 172.21.15.78:0/1214436188 shutdown_connections 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f483d3f7640 1 -- 172.21.15.78:0/1214436188 >> 172.21.15.78:0/1214436188 conn(0x7f48380ca890 msgr2=0x7f48380caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f483d3f7640 1 -- 172.21.15.78:0/1214436188 shutdown_connections 2024-02-19T14:52:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.811+0000 7f483d3f7640 1 -- 172.21.15.78:0/1214436188 wait complete. 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.812+0000 7f483d3f7640 1 Processor -- start 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 -- start start 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48381540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380c1340 0x7f48380c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48381ee210 0x7f48381ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48381eddc0 con 0x7f4824004d50 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48380c3420 con 0x7f48381ee210 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f483d3f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48380c36c0 con 0x7f48380c1340 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f4836ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48381540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f4836ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48381540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41140/0 (socket says 172.21.15.78:41140) 2024-02-19T14:52:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f4836ffd640 1 -- 172.21.15.78:0/4226573570 learned_addr learned my addr 172.21.15.78:0/4226573570 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f4836ffd640 1 -- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380c1340 msgr2=0x7f48380c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f482ffff640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380c1340 0x7f48380c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f4836ffd640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380c1340 0x7f48380c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f48377fe640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48381ee210 0x7f48381ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.813+0000 7f4836ffd640 1 -- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48381ee210 msgr2=0x7f48381ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f4836ffd640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48381ee210 0x7f48381ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f4836ffd640 1 -- 172.21.15.78:0/4226573570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48380c39a0 con 0x7f4824004d50 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f4836ffd640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48381540c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f482800b7f0 tx=0x7f482800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828010830 con 0x7f4824004d50 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4828010990 con 0x7f4824004d50 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828010d20 con 0x7f4824004d50 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4838154600 con 0x7f4824004d50 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.814+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48380c18d0 con 0x7f4824004d50 2024-02-19T14:52:39.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.815+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4838074990 con 0x7f4824004d50 2024-02-19T14:52:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.816+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4828014430 con 0x7f4824004d50 2024-02-19T14:52:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.816+0000 7f482f7fe640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f481806b840 0x7f481806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.816+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f482808ac80 con 0x7f4824004d50 2024-02-19T14:52:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.816+0000 7f482ffff640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f481806b840 0x7f481806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.817+0000 7f482ffff640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f481806b840 0x7f481806dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f48380bdc30 tx=0x7f4820009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.821+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f48280553f0 con 0x7f4824004d50 2024-02-19T14:52:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.998+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f483806da00 con 0x7f4824004d50 2024-02-19T14:52:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:39.998+0000 7f482f7fe640 1 -- 172.21.15.78:0/4226573570 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48280220a0 con 0x7f4824004d50 2024-02-19T14:52:40.001 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:40.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f481806b840 msgr2=0x7f481806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f481806b840 0x7f481806dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f48380bdc30 tx=0x7f4820009040 comp rx=0 tx=0).stop 2024-02-19T14:52:40.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 msgr2=0x7f48381540c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48381540c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f482800b7f0 tx=0x7f482800bcb0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 shutdown_connections 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f481806b840 0x7f481806dd00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48381ee210 0x7f48381ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48380c1340 0x7f48380c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 --2- 172.21.15.78:0/4226573570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4824004d50 0x7f48381540c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 >> 172.21.15.78:0/4226573570 conn(0x7f48380ca890 msgr2=0x7f48380aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 shutdown_connections 2024-02-19T14:52:40.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.000+0000 7f483d3f7640 1 -- 172.21.15.78:0/4226573570 wait complete. 2024-02-19T14:52:40.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 Processor -- start 2024-02-19T14:52:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 -- start start 2024-02-19T14:52:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2400add70 0x7fb2400c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2400c2f60 0x7fb24006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2400ae670 con 0x7fb24006f530 2024-02-19T14:52:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2400716b0 con 0x7fb2400add70 2024-02-19T14:52:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.124+0000 7fb24866b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2400c33d0 con 0x7fb2400c2f60 2024-02-19T14:52:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2400add70 0x7fb2400c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2400add70 0x7fb2400c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41146/0 (socket says 172.21.15.78:41146) 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 -- 172.21.15.78:0/2237673416 learned_addr learned my addr 172.21.15.78:0/2237673416 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb245c06640 1 --2- 172.21.15.78:0/2237673416 >> v2:172.21.15.150:3300/0 conn(0x7fb2400c2f60 0x7fb24006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 -- 172.21.15.78:0/2237673416 >> v2:172.21.15.150:3300/0 conn(0x7fb2400c2f60 msgr2=0x7fb24006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 --2- 172.21.15.78:0/2237673416 >> v2:172.21.15.150:3300/0 conn(0x7fb2400c2f60 0x7fb24006eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 -- 172.21.15.78:0/2237673416 >> v1:172.21.15.150:6789/0 conn(0x7fb24006f530 legacy=0x7fb2400705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.125+0000 7fb246407640 1 -- 172.21.15.78:0/2237673416 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240153780 con 0x7fb2400add70 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.126+0000 7fb246407640 1 --2- 172.21.15.78:0/2237673416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400add70 0x7fb2400c4980 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb2400c2a10 tx=0x7fb23002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63779ba8f53c2820 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.126+0000 7fb245405640 1 -- 172.21.15.78:0/2237673416 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300317e0 con 0x7fb2400add70 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.126+0000 7fb245405640 1 -- 172.21.15.78:0/2237673416 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb230031940 con 0x7fb2400add70 2024-02-19T14:52:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.126+0000 7fb245405640 1 -- 172.21.15.78:0/2237673416 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230031cd0 con 0x7fb2400add70 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.126+0000 7fb24866b640 1 -- 172.21.15.78:0/2237673416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400add70 msgr2=0x7fb2400c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.126+0000 7fb24866b640 1 --2- 172.21.15.78:0/2237673416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400add70 0x7fb2400c4980 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb2400c2a10 tx=0x7fb23002b880 comp rx=0 tx=0).stop 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.127+0000 7fb24866b640 1 -- 172.21.15.78:0/2237673416 shutdown_connections 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.127+0000 7fb24866b640 1 --2- 172.21.15.78:0/2237673416 >> v2:172.21.15.150:3300/0 conn(0x7fb2400c2f60 0x7fb24006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.127+0000 7fb24866b640 1 --2- 172.21.15.78:0/2237673416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400add70 0x7fb2400c4980 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.127+0000 7fb24866b640 1 -- 172.21.15.78:0/2237673416 >> 172.21.15.78:0/2237673416 conn(0x7fb2400c85e0 msgr2=0x7fb2400c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.127+0000 7fb24866b640 1 -- 172.21.15.78:0/2237673416 shutdown_connections 2024-02-19T14:52:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.127+0000 7fb24866b640 1 -- 172.21.15.78:0/2237673416 wait complete. 2024-02-19T14:52:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.128+0000 7fb24866b640 1 Processor -- start 2024-02-19T14:52:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.128+0000 7fb24866b640 1 -- start start 2024-02-19T14:52:40.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.129+0000 7fb24866b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24006f530 0x7fb2400abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.129+0000 7fb24866b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2400add70 0x7fb2401ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.129+0000 7fb24866b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 0x7fb240144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.129+0000 7fb24866b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401eeaa0 con 0x7fb2400c2f60 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.129+0000 7fb24866b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb240145220 con 0x7fb24006f530 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.129+0000 7fb24866b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401edaa0 con 0x7fb2400add70 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 0x7fb240144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 0x7fb240144ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41152/0 (socket says 172.21.15.78:41152) 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 -- 172.21.15.78:0/2045972048 learned_addr learned my addr 172.21.15.78:0/2045972048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb245c06640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2400add70 0x7fb2401ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246407640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24006f530 0x7fb2400abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 -- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2400add70 msgr2=0x7fb2401ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2400add70 0x7fb2401ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 -- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24006f530 msgr2=0x7fb2400abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24006f530 0x7fb2400abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 -- 172.21.15.78:0/2045972048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2300094d0 con 0x7fb2400c2f60 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb246c08640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 0x7fb240144ce0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb23800b970 tx=0x7fb23800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23800cb40 con 0x7fb2400c2f60 2024-02-19T14:52:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23800cca0 con 0x7fb2400c2f60 2024-02-19T14:52:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2401543f0 con 0x7fb2400c2f60 2024-02-19T14:52:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238014680 con 0x7fb2400c2f60 2024-02-19T14:52:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.130+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401edd60 con 0x7fb2400c2f60 2024-02-19T14:52:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.131+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2400676a0 con 0x7fb2400c2f60 2024-02-19T14:52:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.132+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb23801d3b0 con 0x7fb2400c2f60 2024-02-19T14:52:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.132+0000 7fb236ffd640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22406b840 0x7fb22406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.132+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb23808a270 con 0x7fb2400c2f60 2024-02-19T14:52:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.132+0000 7fb246407640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22406b840 0x7fb22406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.133+0000 7fb246407640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22406b840 0x7fb22406dd00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb230036000 tx=0x7fb23003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:40.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.137+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2380549e0 con 0x7fb2400c2f60 2024-02-19T14:52:40.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.324+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb2400ae890 con 0x7fb2400c2f60 2024-02-19T14:52:40.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.324+0000 7fb236ffd640 1 -- 172.21.15.78:0/2045972048 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb2380580a0 con 0x7fb2400c2f60 2024-02-19T14:52:40.327 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:40.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22406b840 msgr2=0x7fb22406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22406b840 0x7fb22406dd00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb230036000 tx=0x7fb23003d040 comp rx=0 tx=0).stop 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 msgr2=0x7fb240144ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 0x7fb240144ce0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb23800b970 tx=0x7fb23800be30 comp rx=0 tx=0).stop 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 shutdown_connections 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22406b840 0x7fb22406dd00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2400c2f60 0x7fb240144ce0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2400add70 0x7fb2401ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 --2- 172.21.15.78:0/2045972048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24006f530 0x7fb2400abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 >> 172.21.15.78:0/2045972048 conn(0x7fb2400c85e0 msgr2=0x7fb2400a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 shutdown_connections 2024-02-19T14:52:40.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.326+0000 7fb24866b640 1 -- 172.21.15.78:0/2045972048 wait complete. 2024-02-19T14:52:40.339 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-02-19T14:52:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.438+0000 7fda69455640 1 Processor -- start 2024-02-19T14:52:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.438+0000 7fda69455640 1 -- start start 2024-02-19T14:52:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda69455640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda640c1880 0x7fda64152980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda69455640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda640c1d40 0x7fda6406e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda69455640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6406b710 con 0x7fda64152180 2024-02-19T14:52:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda69455640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda64072430 con 0x7fda640c1880 2024-02-19T14:52:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda69455640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640715e0 con 0x7fda640c1d40 2024-02-19T14:52:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda640c1880 0x7fda64152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda640c1880 0x7fda64152980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41156/0 (socket says 172.21.15.78:41156) 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 -- 172.21.15.78:0/487075187 learned_addr learned my addr 172.21.15.78:0/487075187 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda637fe640 1 --2- 172.21.15.78:0/487075187 >> v2:172.21.15.150:3300/0 conn(0x7fda640c1d40 0x7fda6406e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 -- 172.21.15.78:0/487075187 >> v2:172.21.15.150:3300/0 conn(0x7fda640c1d40 msgr2=0x7fda6406e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 --2- 172.21.15.78:0/487075187 >> v2:172.21.15.150:3300/0 conn(0x7fda640c1d40 0x7fda6406e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 -- 172.21.15.78:0/487075187 >> v1:172.21.15.150:6789/0 conn(0x7fda64152180 legacy=0x7fda64072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.439+0000 7fda5bfff640 1 -- 172.21.15.78:0/487075187 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda64152560 con 0x7fda640c1880 2024-02-19T14:52:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda5bfff640 1 --2- 172.21.15.78:0/487075187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1880 0x7fda64152980 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fda6406d3f0 tx=0x7fda4c02cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a513e868ebf6a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda627fc640 1 -- 172.21.15.78:0/487075187 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c02daf0 con 0x7fda640c1880 2024-02-19T14:52:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda627fc640 1 -- 172.21.15.78:0/487075187 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c02dc50 con 0x7fda640c1880 2024-02-19T14:52:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda627fc640 1 -- 172.21.15.78:0/487075187 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c035440 con 0x7fda640c1880 2024-02-19T14:52:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 -- 172.21.15.78:0/487075187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1880 msgr2=0x7fda64152980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 --2- 172.21.15.78:0/487075187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1880 0x7fda64152980 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fda6406d3f0 tx=0x7fda4c02cd40 comp rx=0 tx=0).stop 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 -- 172.21.15.78:0/487075187 shutdown_connections 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 --2- 172.21.15.78:0/487075187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1880 0x7fda64152980 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 --2- 172.21.15.78:0/487075187 >> v2:172.21.15.150:3300/0 conn(0x7fda640c1d40 0x7fda6406e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 -- 172.21.15.78:0/487075187 >> 172.21.15.78:0/487075187 conn(0x7fda640a8700 msgr2=0x7fda640a8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 -- 172.21.15.78:0/487075187 shutdown_connections 2024-02-19T14:52:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.441+0000 7fda69455640 1 -- 172.21.15.78:0/487075187 wait complete. 2024-02-19T14:52:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.442+0000 7fda69455640 1 Processor -- start 2024-02-19T14:52:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 -- start start 2024-02-19T14:52:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda640c1880 0x7fda640bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 0x7fda640c0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda64152180 0x7fda641e9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640aa570 con 0x7fda640c1d40 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64151b00 con 0x7fda640c1880 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda69455640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640c0860 con 0x7fda64152180 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 0x7fda640c0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 0x7fda640c0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41166/0 (socket says 172.21.15.78:41166) 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda5bfff640 1 -- 172.21.15.78:0/59842967 learned_addr learned my addr 172.21.15.78:0/59842967 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda62ffd640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda640c1880 0x7fda640bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.443+0000 7fda637fe640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda64152180 0x7fda641e9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5bfff640 1 -- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda64152180 msgr2=0x7fda641e9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5bfff640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda64152180 0x7fda641e9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5bfff640 1 -- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda640c1880 msgr2=0x7fda640bfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5bfff640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda640c1880 0x7fda640bfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5bfff640 1 -- 172.21.15.78:0/59842967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c00a670 con 0x7fda640c1d40 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda637fe640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda64152180 0x7fda641e9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5bfff640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 0x7fda640c0320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fda4c02cd80 tx=0x7fda4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c040640 con 0x7fda640c1d40 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c0407c0 con 0x7fda640c1d40 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda64151820 con 0x7fda640c1d40 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c040b70 con 0x7fda640c1d40 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.444+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda640aa190 con 0x7fda640c1d40 2024-02-19T14:52:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.445+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda640c2a30 con 0x7fda640c1d40 2024-02-19T14:52:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.446+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fda4c03c040 con 0x7fda640c1d40 2024-02-19T14:52:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.446+0000 7fda5b7fe640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4406b810 0x7fda4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.447+0000 7fda62ffd640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4406b810 0x7fda4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.447+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fda4c0ad920 con 0x7fda640c1d40 2024-02-19T14:52:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.447+0000 7fda62ffd640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4406b810 0x7fda4406dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fda54001780 tx=0x7fda54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.451+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda4c078090 con 0x7fda640c1d40 2024-02-19T14:52:40.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:40.693+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7fda64074990 con 0x7fda640c1d40 2024-02-19T14:52:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.195+0000 7fda5b7fe640 1 -- 172.21.15.78:0/59842967 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v62) v1 ==== 146+0+0 (secure 0 0 0) 0x7fda4c080e60 con 0x7fda640c1d40 2024-02-19T14:52:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.198+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4406b810 msgr2=0x7fda4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.198+0000 7fda69455640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4406b810 0x7fda4406dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fda54001780 tx=0x7fda54009040 comp rx=0 tx=0).stop 2024-02-19T14:52:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.198+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 msgr2=0x7fda640c0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.198+0000 7fda69455640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 0x7fda640c0320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fda4c02cd80 tx=0x7fda4c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.199+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 shutdown_connections 2024-02-19T14:52:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.199+0000 7fda69455640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4406b810 0x7fda4406dcd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.199+0000 7fda69455640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda640c1d40 0x7fda640c0320 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.199+0000 7fda69455640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda64152180 0x7fda641e9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.199+0000 7fda69455640 1 --2- 172.21.15.78:0/59842967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda640c1880 0x7fda640bfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.199+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 >> 172.21.15.78:0/59842967 conn(0x7fda640a8700 msgr2=0x7fda6406ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.200+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 shutdown_connections 2024-02-19T14:52:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.200+0000 7fda69455640 1 -- 172.21.15.78:0/59842967 wait complete. 2024-02-19T14:52:41.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:52:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.318+0000 7fe5175e2640 1 Processor -- start 2024-02-19T14:52:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.319+0000 7fe5175e2640 1 -- start start 2024-02-19T14:52:41.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.319+0000 7fe5175e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 0x7fe510154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.319+0000 7fe5175e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51014f730 con 0x7fe51006ff80 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.319+0000 7fe5175e2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5100c16d0 con 0x7fe5100c6d60 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.319+0000 7fe5175e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100c1940 con 0x7fe51006e830 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe515b7f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe51006ff80 0x7fe5100c7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52646/0 (socket says 172.21.15.78:52646) 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe515b7f640 1 -- 172.21.15.78:0/2845713606 learned_addr learned my addr 172.21.15.78:0/2845713606 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954419079 0 0) 0x7fe51014f730 con 0x7fe51006ff80 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe514b7d640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 0x7fe510154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc003520 con 0x7fe51006ff80 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564016441 0 0) 0x7fe4fc003520 con 0x7fe51006ff80 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 msgr2=0x7fe510154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 0x7fe510154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 >> v1:172.21.15.150:6789/0 conn(0x7fe5100c6d60 legacy=0x7fe5100c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510072f70 con 0x7fe51006ff80 2024-02-19T14:52:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe5080019f0 con 0x7fe51006ff80 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 >> v1:172.21.15.78:6789/0 conn(0x7fe51006ff80 legacy=0x7fe5100c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe4fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4fc0047c0 0x7fe4fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 0x7fe4fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100c1940 con 0x7fe4fc0039a0 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00bab0 con 0x7fe4fc0047c0 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00bc70 con 0x7fe51006e830 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe51537e640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe4fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe514b7d640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4fc0047c0 0x7fe4fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.320+0000 7fe515b7f640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 0x7fe4fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe51537e640 1 -- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 msgr2=0x7fe4fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe51537e640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe51006e830 0x7fe4fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe51537e640 1 -- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4fc0047c0 msgr2=0x7fe4fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe51537e640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4fc0047c0 0x7fe4fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe51537e640 1 -- 172.21.15.78:0/2845713606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc003520 con 0x7fe4fc0039a0 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe51537e640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe4fc003dc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe50c001750 tx=0x7fe4fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a01ba7226ea41fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0052d0 con 0x7fe4fc0039a0 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50c02e050 con 0x7fe4fc0039a0 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.321+0000 7fe507fff640 1 -- 172.21.15.78:0/2845713606 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0035f0 con 0x7fe4fc0039a0 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe5175e2640 1 -- 172.21.15.78:0/2845713606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 msgr2=0x7fe4fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe5175e2640 1 --2- 172.21.15.78:0/2845713606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe4fc003dc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe50c001750 tx=0x7fe4fc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe515b7f640 1 -- 172.21.15.78:0/2845713606 reap_dead start 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe5175e2640 1 -- 172.21.15.78:0/2845713606 shutdown_connections 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe5175e2640 1 -- 172.21.15.78:0/2845713606 >> 172.21.15.78:0/2845713606 conn(0x7fe51006c170 msgr2=0x7fe51006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe5175e2640 1 -- 172.21.15.78:0/2845713606 shutdown_connections 2024-02-19T14:52:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.322+0000 7fe5175e2640 1 -- 172.21.15.78:0/2845713606 wait complete. 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.324+0000 7fe5175e2640 1 Processor -- start 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.324+0000 7fe5175e2640 1 -- start start 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe5175e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe5101ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe5175e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5101ee370 0x7fe510149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe5175e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe510149a40 0x7fe51014c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe5175e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51014fa40 con 0x7fe4fc0039a0 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe5175e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5101ee940 con 0x7fe5101ee370 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe5175e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51014fc90 con 0x7fe510149a40 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe5101ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe5101ede30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41180/0 (socket says 172.21.15.78:41180) 2024-02-19T14:52:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 -- 172.21.15.78:0/3918645046 learned_addr learned my addr 172.21.15.78:0/3918645046 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 -- 172.21.15.78:0/3918645046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe510149a40 msgr2=0x7fe51014c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe510149a40 0x7fe51014c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 -- 172.21.15.78:0/3918645046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5101ee370 msgr2=0x7fe510149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5101ee370 0x7fe510149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.325+0000 7fe51537e640 1 -- 172.21.15.78:0/3918645046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51014ff70 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.326+0000 7fe51537e640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe5101ede30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe50c008fd0 tx=0x7fe50c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.326+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0076c0 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.326+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50c007a30 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.326+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe510148fe0 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.326+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c006c20 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.326+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe510148ab0 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.327+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe510149170 con 0x7fe4fc0039a0 2024-02-19T14:52:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.327+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe50c0035f0 con 0x7fe4fc0039a0 2024-02-19T14:52:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.328+0000 7fe505ffb640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4f406b840 0x7fe4f406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.328+0000 7fe514b7d640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4f406b840 0x7fe4f406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.328+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fe50c0ac200 con 0x7fe4fc0039a0 2024-02-19T14:52:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.328+0000 7fe514b7d640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4f406b840 0x7fe4f406dd00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe510152b20 tx=0x7fe500006c30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.332+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe50c076850 con 0x7fe4fc0039a0 2024-02-19T14:52:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.509+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe510074990 con 0x7fe4fc0039a0 2024-02-19T14:52:41.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.512+0000 7fe505ffb640 1 -- 172.21.15.78:0/3918645046 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v11)=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe50c079f10 con 0x7fe4fc0039a0 2024-02-19T14:52:41.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4f406b840 msgr2=0x7fe4f406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4f406b840 0x7fe4f406dd00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe510152b20 tx=0x7fe500006c30 comp rx=0 tx=0).stop 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 msgr2=0x7fe5101ede30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe5101ede30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe50c008fd0 tx=0x7fe50c001120 comp rx=0 tx=0).stop 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 shutdown_connections 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4f406b840 0x7fe4f406dd00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe510149a40 0x7fe51014c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5101ee370 0x7fe510149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 --2- 172.21.15.78:0/3918645046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4fc0039a0 0x7fe5101ede30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 >> 172.21.15.78:0/3918645046 conn(0x7fe51006c170 msgr2=0x7fe5100a8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 shutdown_connections 2024-02-19T14:52:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.514+0000 7fe5175e2640 1 -- 172.21.15.78:0/3918645046 wait complete. 2024-02-19T14:52:41.527 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:52:41.527 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:52:41.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:52:41.614 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:52:41.614 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:52:41.614 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:41.629 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:52:41.629 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:41.684 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:52:41.684 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:41.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.877+0000 7faa0e26b640 1 Processor -- start 2024-02-19T14:52:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.877+0000 7faa0e26b640 1 -- start start 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.878+0000 7faa0e26b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faa08152700 0x7faa080c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.878+0000 7faa0e26b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa081551f0 con 0x7faa080c8e00 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.878+0000 7faa0e26b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa08155490 con 0x7faa080c5730 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.878+0000 7faa0e26b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa08155770 con 0x7faa08152700 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.878+0000 7faa0c808640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7faa080c8e00 0x7faa0815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52656/0 (socket says 172.21.15.78:52656) 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.878+0000 7faa0c808640 1 -- 172.21.15.78:0/4205344818 learned_addr learned my addr 172.21.15.78:0/4205344818 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258231613 0 0) 0x7faa081551f0 con 0x7faa080c8e00 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f0003520 con 0x7faa080c8e00 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa07fff640 1 --2- 172.21.15.78:0/4205344818 >> v2:172.21.15.150:3300/0 conn(0x7faa08152700 0x7faa080c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3993692918 0 0) 0x7fa9f0003520 con 0x7faa080c8e00 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 >> v2:172.21.15.150:3300/0 conn(0x7faa08152700 msgr2=0x7faa080c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 --2- 172.21.15.78:0/4205344818 >> v2:172.21.15.150:3300/0 conn(0x7faa08152700 0x7faa080c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 >> v1:172.21.15.150:6789/0 conn(0x7faa080c5730 legacy=0x7faa0806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa08155a50 con 0x7faa080c8e00 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7faa000019f0 con 0x7faa080c8e00 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 >> v1:172.21.15.78:6789/0 conn(0x7faa080c8e00 legacy=0x7faa0815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f0003970 0x7fa9f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9f00047c0 0x7fa9f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9f0005180 0x7fa9f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08155770 con 0x7fa9f00047c0 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f000c120 con 0x7fa9f0003970 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f000c2e0 con 0x7fa9f0005180 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa077fe640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9f00047c0 0x7fa9f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa07fff640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f0003970 0x7fa9f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa0c808640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9f0005180 0x7fa9f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa077fe640 1 -- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9f0005180 msgr2=0x7fa9f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa077fe640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9f0005180 0x7fa9f000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa077fe640 1 -- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f0003970 msgr2=0x7fa9f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa077fe640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f0003970 0x7fa9f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa077fe640 1 -- 172.21.15.78:0/4205344818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0003520 con 0x7fa9f00047c0 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.879+0000 7faa0c808640 1 -- 172.21.15.78:0/4205344818 reap_dead start 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.880+0000 7faa077fe640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9f00047c0 0x7fa9f0004c40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc001750 tx=0x7fa9f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bf12bdc3d30f58 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.880+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc0052d0 con 0x7fa9f00047c0 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.880+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc02e050 con 0x7fa9f00047c0 2024-02-19T14:52:41.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.880+0000 7faa06ffd640 1 -- 172.21.15.78:0/4205344818 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc003830 con 0x7fa9f00047c0 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 -- 172.21.15.78:0/4205344818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9f00047c0 msgr2=0x7fa9f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9f00047c0 0x7fa9f0004c40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc001750 tx=0x7fa9f00057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 -- 172.21.15.78:0/4205344818 shutdown_connections 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 --2- 172.21.15.78:0/4205344818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9f00047c0 0x7fa9f0004c40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 -- 172.21.15.78:0/4205344818 >> 172.21.15.78:0/4205344818 conn(0x7faa080cac90 msgr2=0x7faa080cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 -- 172.21.15.78:0/4205344818 shutdown_connections 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.881+0000 7faa0e26b640 1 -- 172.21.15.78:0/4205344818 wait complete. 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.882+0000 7faa0e26b640 1 Processor -- start 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.882+0000 7faa0e26b640 1 -- start start 2024-02-19T14:52:41.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0e26b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f00047c0 0x7faa081e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0e26b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa080be400 0x7faa081e7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0e26b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 0x7faa081ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0e26b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa081ec440 con 0x7faa081edba0 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0e26b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa081eb830 con 0x7fa9f00047c0 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0e26b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa081ebb10 con 0x7faa080be400 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 0x7faa081ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 0x7faa081ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41204/0 (socket says 172.21.15.78:41204) 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 -- 172.21.15.78:0/169306548 learned_addr learned my addr 172.21.15.78:0/169306548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa077fe640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa080be400 0x7faa081e7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa07fff640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f00047c0 0x7faa081e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 -- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa080be400 msgr2=0x7faa081e7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa080be400 0x7faa081e7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 -- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f00047c0 msgr2=0x7faa081e76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f00047c0 0x7faa081e76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.883+0000 7faa0c808640 1 -- 172.21.15.78:0/169306548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa081ebdf0 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.884+0000 7faa0c808640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 0x7faa081ee000 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faa0808c490 tx=0x7faa00007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.884+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0000ab10 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.884+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0000ac70 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.884+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa081ec0d0 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.884+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000133f0 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.884+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa080abd70 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.885+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa081ec5b0 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.885+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7faa00013550 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.886+0000 7faa04ff9640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9f406b810 0x7fa9f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.886+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7faa0008aeb0 con 0x7faa081edba0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.886+0000 7faa07fff640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9f406b810 0x7fa9f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.886+0000 7faa07fff640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9f406b810 0x7fa9f406dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009b40 tx=0x7fa9f8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:41.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:41.894+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa00055480 con 0x7faa081edba0 2024-02-19T14:52:42.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.073+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa080c49e0 con 0x7faa081edba0 2024-02-19T14:52:42.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.075+0000 7faa04ff9640 1 -- 172.21.15.78:0/169306548 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19031 (secure 0 0 0) 0x7faa00017020 con 0x7faa081edba0 2024-02-19T14:52:42.077 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:42.077 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:41.184968+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":0}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:42.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9f406b810 msgr2=0x7fa9f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9f406b810 0x7fa9f406dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009b40 tx=0x7fa9f8009190 comp rx=0 tx=0).stop 2024-02-19T14:52:42.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 msgr2=0x7faa081ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 0x7faa081ee000 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faa0808c490 tx=0x7faa00007b10 comp rx=0 tx=0).stop 2024-02-19T14:52:42.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 shutdown_connections 2024-02-19T14:52:42.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9f406b810 0x7fa9f406dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa081edba0 0x7faa081ee000 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa080be400 0x7faa081e7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 --2- 172.21.15.78:0/169306548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9f00047c0 0x7faa081e76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 >> 172.21.15.78:0/169306548 conn(0x7faa080cac90 msgr2=0x7faa080c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:42.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 shutdown_connections 2024-02-19T14:52:42.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.077+0000 7faa0e26b640 1 -- 172.21.15.78:0/169306548 wait complete. 2024-02-19T14:52:42.091 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:42.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:42.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.193+0000 7ff0d521b640 1 Processor -- start 2024-02-19T14:52:42.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.193+0000 7ff0d521b640 1 -- start start 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0d521b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0d00c78a0 0x7ff0d0159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0d521b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff0d00c4d80 0x7ff0d00c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0d521b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d006ffa0 con 0x7ff0d00c48f0 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0d521b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d006b720 con 0x7ff0d00c4d80 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0d521b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d006fc70 con 0x7ff0d00c78a0 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff0d00c4d80 0x7ff0d00c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff0d00c4d80 0x7ff0d00c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41216/0 (socket says 172.21.15.78:41216) 2024-02-19T14:52:42.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 -- 172.21.15.78:0/1825560999 learned_addr learned my addr 172.21.15.78:0/1825560999 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:42.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0ced9d640 1 --2- 172.21.15.78:0/1825560999 >> v2:172.21.15.150:3300/0 conn(0x7ff0d00c78a0 0x7ff0d0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 -- 172.21.15.78:0/1825560999 >> v2:172.21.15.150:3300/0 conn(0x7ff0d00c78a0 msgr2=0x7ff0d0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:42.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 --2- 172.21.15.78:0/1825560999 >> v2:172.21.15.150:3300/0 conn(0x7ff0d00c78a0 0x7ff0d0159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 -- 172.21.15.78:0/1825560999 >> v1:172.21.15.150:6789/0 conn(0x7ff0d00c48f0 legacy=0x7ff0d01523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.194+0000 7ff0c7fff640 1 -- 172.21.15.78:0/1825560999 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d006b930 con 0x7ff0d00c4d80 2024-02-19T14:52:42.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.195+0000 7ff0c7fff640 1 --2- 172.21.15.78:0/1825560999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c4d80 0x7ff0d00c43b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00c8420 tx=0x7ff0b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabf33d4f2ce5b07 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0ce59c640 1 -- 172.21.15.78:0/1825560999 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8031870 con 0x7ff0d00c4d80 2024-02-19T14:52:42.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0ce59c640 1 -- 172.21.15.78:0/1825560999 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b80319d0 con 0x7ff0d00c4d80 2024-02-19T14:52:42.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0ce59c640 1 -- 172.21.15.78:0/1825560999 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8031d60 con 0x7ff0d00c4d80 2024-02-19T14:52:42.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 -- 172.21.15.78:0/1825560999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c4d80 msgr2=0x7ff0d00c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 --2- 172.21.15.78:0/1825560999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c4d80 0x7ff0d00c43b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00c8420 tx=0x7ff0b802b910 comp rx=0 tx=0).stop 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 -- 172.21.15.78:0/1825560999 shutdown_connections 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 --2- 172.21.15.78:0/1825560999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c4d80 0x7ff0d00c43b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 --2- 172.21.15.78:0/1825560999 >> v2:172.21.15.150:3300/0 conn(0x7ff0d00c78a0 0x7ff0d0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 -- 172.21.15.78:0/1825560999 >> 172.21.15.78:0/1825560999 conn(0x7ff0d0066e70 msgr2=0x7ff0d00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 -- 172.21.15.78:0/1825560999 shutdown_connections 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.196+0000 7ff0d521b640 1 -- 172.21.15.78:0/1825560999 wait complete. 2024-02-19T14:52:42.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.198+0000 7ff0d521b640 1 Processor -- start 2024-02-19T14:52:42.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.198+0000 7ff0d521b640 1 -- start start 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0d521b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0d00c48f0 0x7ff0d014e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0d521b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0d00c4d80 0x7ff0d01ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0d521b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 0x7ff0d01581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0d521b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d00aae00 con 0x7ff0d00c78a0 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0d521b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d01568f0 con 0x7ff0d00c4d80 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0d521b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d00aa880 con 0x7ff0d00c48f0 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0cf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 0x7ff0d01581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0cf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 0x7ff0d01581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41232/0 (socket says 172.21.15.78:41232) 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0cf59e640 1 -- 172.21.15.78:0/2800727154 learned_addr learned my addr 172.21.15.78:0/2800727154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0c7fff640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0d00c4d80 0x7ff0d01ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.199+0000 7ff0ced9d640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0d00c48f0 0x7ff0d014e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0cf59e640 1 -- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0d00c48f0 msgr2=0x7ff0d014e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0cf59e640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0d00c48f0 0x7ff0d014e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0cf59e640 1 -- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0d00c4d80 msgr2=0x7ff0d01ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0cf59e640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0d00c4d80 0x7ff0d01ee490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0cf59e640 1 -- 172.21.15.78:0/2800727154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b80094d0 con 0x7ff0d00c78a0 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0cf59e640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 0x7ff0d01581e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c7a0 tx=0x7ff0c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8003b70 con 0x7ff0d00c78a0 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c8003cf0 con 0x7ff0d00c78a0 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800aa90 con 0x7ff0d00c78a0 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d014e810 con 0x7ff0d00c78a0 2024-02-19T14:52:42.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.200+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d0156580 con 0x7ff0d00c78a0 2024-02-19T14:52:42.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.201+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d0074990 con 0x7ff0d00c78a0 2024-02-19T14:52:42.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.201+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff0c800abf0 con 0x7ff0d00c78a0 2024-02-19T14:52:42.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.202+0000 7ff0c77fe640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0b006b840 0x7ff0b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.202+0000 7ff0ced9d640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0b006b840 0x7ff0b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.202+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7ff0c808a300 con 0x7ff0d00c78a0 2024-02-19T14:52:42.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.202+0000 7ff0ced9d640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0b006b840 0x7ff0b006dd00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006fd0 tx=0x7ff0c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.206+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff0c8054a00 con 0x7ff0d00c78a0 2024-02-19T14:52:42.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.384+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff0d0156a30 con 0x7ff0d00c78a0 2024-02-19T14:52:42.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.385+0000 7ff0c77fe640 1 -- 172.21.15.78:0/2800727154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff0c8003e60 con 0x7ff0d00c78a0 2024-02-19T14:52:42.387 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.386+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0b006b840 msgr2=0x7ff0b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.386+0000 7ff0d521b640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0b006b840 0x7ff0b006dd00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006fd0 tx=0x7ff0c0008040 comp rx=0 tx=0).stop 2024-02-19T14:52:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.386+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 msgr2=0x7ff0d01581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.386+0000 7ff0d521b640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 0x7ff0d01581e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c7a0 tx=0x7ff0c800cc60 comp rx=0 tx=0).stop 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 shutdown_connections 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0b006b840 0x7ff0b006dd00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0d00c78a0 0x7ff0d01581e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0d00c4d80 0x7ff0d01ee490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 --2- 172.21.15.78:0/2800727154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0d00c48f0 0x7ff0d014e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 >> 172.21.15.78:0/2800727154 conn(0x7ff0d0066e70 msgr2=0x7ff0d00c1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 shutdown_connections 2024-02-19T14:52:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.387+0000 7ff0d521b640 1 -- 172.21.15.78:0/2800727154 wait complete. 2024-02-19T14:52:42.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.517+0000 7f5e157b5640 1 Processor -- start 2024-02-19T14:52:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.517+0000 7f5e157b5640 1 -- start start 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e157b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 0x7f5e10154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e157b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1014f730 con 0x7f5e1006e830 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e157b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e100c16d0 con 0x7f5e100c6d60 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e157b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100c1940 con 0x7f5e1006ff80 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e07fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5e1006e830 0x7f5e100c7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52666/0 (socket says 172.21.15.78:52666) 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e07fff640 1 -- 172.21.15.78:0/84574808 learned_addr learned my addr 172.21.15.78:0/84574808 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e0f7fe640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 0x7f5e10154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389231510 0 0) 0x7f5e1014f730 con 0x7f5e1006e830 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.518+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dfc003520 con 0x7f5e1006e830 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503062801 0 0) 0x7f5e100c16d0 con 0x7f5e100c6d60 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1014f730 con 0x7f5e100c6d60 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089373517 0 0) 0x7f5dfc003520 con 0x7f5e1006e830 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 msgr2=0x7f5e10154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:42.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 0x7f5e10154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 >> v1:172.21.15.150:6789/0 conn(0x7f5e100c6d60 legacy=0x7f5e100c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e10072f70 con 0x7f5e1006e830 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f5df8001670 con 0x7f5e1006e830 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 >> v1:172.21.15.78:6789/0 conn(0x7f5e1006e830 legacy=0x7f5e100c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dfc003b30 0x7f5dfc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 0x7f5dfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dfc004d00 0x7f5dfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100c1940 con 0x7f5dfc004d00 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc00ba60 con 0x7f5dfc003b30 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc00bc20 con 0x7f5e1006ff80 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0f7fe640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dfc004d00 0x7f5dfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0effd640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dfc003b30 0x7f5dfc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e07fff640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 0x7f5dfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0f7fe640 1 -- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 msgr2=0x7f5dfc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0f7fe640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1006ff80 0x7f5dfc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0f7fe640 1 -- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dfc003b30 msgr2=0x7f5dfc003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0f7fe640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dfc003b30 0x7f5dfc003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.519+0000 7f5e0f7fe640 1 -- 172.21.15.78:0/84574808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc003520 con 0x7f5dfc004d00 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.520+0000 7f5e0f7fe640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dfc004d00 0x7f5dfc005180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009fc0 tx=0x7f5dfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50edcecb15c8e8b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.520+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0802cc20 con 0x7f5dfc004d00 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.520+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0800f040 con 0x7f5dfc004d00 2024-02-19T14:52:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.520+0000 7f5e0e7fc640 1 -- 172.21.15.78:0/84574808 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08034440 con 0x7f5dfc004d00 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e157b5640 1 -- 172.21.15.78:0/84574808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dfc004d00 msgr2=0x7f5dfc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e157b5640 1 --2- 172.21.15.78:0/84574808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dfc004d00 0x7f5dfc005180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009fc0 tx=0x7f5dfc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e0f7fe640 1 -- 172.21.15.78:0/84574808 reap_dead start 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e157b5640 1 -- 172.21.15.78:0/84574808 shutdown_connections 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e157b5640 1 -- 172.21.15.78:0/84574808 >> 172.21.15.78:0/84574808 conn(0x7f5e1006c170 msgr2=0x7f5e1006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e157b5640 1 -- 172.21.15.78:0/84574808 shutdown_connections 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.521+0000 7f5e157b5640 1 -- 172.21.15.78:0/84574808 wait complete. 2024-02-19T14:52:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.522+0000 7f5e157b5640 1 Processor -- start 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.522+0000 7f5e157b5640 1 -- start start 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e157b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e101449f0 0x7f5e10144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e157b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 0x7f5e10143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e157b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e101eabd0 0x7f5e101eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e157b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100c3150 con 0x7f5e101438a0 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e157b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e101477b0 con 0x7f5e101eabd0 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e157b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e101467c0 con 0x7f5e101449f0 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.523+0000 7f5e07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 0x7f5e10143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 0x7f5e10143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41258/0 (socket says 172.21.15.78:41258) 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 -- 172.21.15.78:0/3204839864 learned_addr learned my addr 172.21.15.78:0/3204839864 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e0effd640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e101449f0 0x7f5e10144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e0f7fe640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e101eabd0 0x7f5e101eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 -- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e101449f0 msgr2=0x7f5e10144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e101449f0 0x7f5e10144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 -- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e101eabd0 msgr2=0x7f5e101eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e101eabd0 0x7f5e101eb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 -- 172.21.15.78:0/3204839864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e101463c0 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e07fff640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 0x7f5e10143d20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5df8001d50 tx=0x7f5df8008560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8008750 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df800f8f0 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df800fca0 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e10145310 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.524+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e10146cd0 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.525+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e10074990 con 0x7f5e101438a0 2024-02-19T14:52:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.526+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5df80153e0 con 0x7f5e101438a0 2024-02-19T14:52:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.526+0000 7f5e077fe640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5df006b840 0x7f5df006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.526+0000 7f5e0effd640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5df006b840 0x7f5df006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.526+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f5df808a7a0 con 0x7f5e101438a0 2024-02-19T14:52:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.527+0000 7f5e0effd640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5df006b840 0x7f5df006dd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e100a9990 tx=0x7f5e00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.530+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5df8054df0 con 0x7f5e101438a0 2024-02-19T14:52:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.706+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e100c09c0 con 0x7f5e101438a0 2024-02-19T14:52:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.707+0000 7f5e077fe640 1 -- 172.21.15.78:0/3204839864 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5df80584b0 con 0x7f5e101438a0 2024-02-19T14:52:42.710 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5df006b840 msgr2=0x7f5df006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5df006b840 0x7f5df006dd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e100a9990 tx=0x7f5e00009040 comp rx=0 tx=0).stop 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 msgr2=0x7f5e10143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 0x7f5e10143d20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5df8001d50 tx=0x7f5df8008560 comp rx=0 tx=0).stop 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 shutdown_connections 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5df006b840 0x7f5df006dd00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e101438a0 0x7f5e10143d20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e101eabd0 0x7f5e101eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 --2- 172.21.15.78:0/3204839864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e101449f0 0x7f5e10144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 >> 172.21.15.78:0/3204839864 conn(0x7f5e1006c170 msgr2=0x7f5e100a8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 shutdown_connections 2024-02-19T14:52:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.709+0000 7f5e157b5640 1 -- 172.21.15.78:0/3204839864 wait complete. 2024-02-19T14:52:42.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.827+0000 7f0f0f31e640 1 Processor -- start 2024-02-19T14:52:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.827+0000 7f0f0f31e640 1 -- start start 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0f31e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f0814cd50 0x7f0f080c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0f31e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0f081537a0 0x7f0f080c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0f31e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f081563b0 con 0x7f0f080c7b70 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0f31e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f08153cd0 con 0x7f0f081537a0 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0f31e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0806b6e0 con 0x7f0f0814cd50 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0c8b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0f081537a0 0x7f0f080c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0c8b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0f081537a0 0x7f0f080c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41262/0 (socket says 172.21.15.78:41262) 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0d8bb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0f080c7b70 0x7f0f08156860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52678/0 (socket says 172.21.15.78:52678) 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.828+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2989849779 learned_addr learned my addr 172.21.15.78:0/2989849779 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2989849779 >> v2:172.21.15.150:3300/0 conn(0x7f0f0814cd50 msgr2=0x7f0f080c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0f0d0ba640 1 --2- 172.21.15.78:0/2989849779 >> v2:172.21.15.150:3300/0 conn(0x7f0f0814cd50 0x7f0f080c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0f0c8b9640 1 --2- 172.21.15.78:0/2989849779 >> v2:172.21.15.150:3300/0 conn(0x7f0f0814cd50 0x7f0f080c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2989849779 >> v1:172.21.15.78:6789/0 conn(0x7f0f080c7b70 legacy=0x7f0f08156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0efffff640 1 -- 172.21.15.78:0/2989849779 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924676859 0 0) 0x7f0f081563b0 con 0x7f0f080c7b70 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2989849779 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0806b980 con 0x7f0f081537a0 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.829+0000 7f0f0d0ba640 1 --2- 172.21.15.78:0/2989849779 >> v2:172.21.15.150:3300/0 conn(0x7f0f0814cd50 0x7f0f080c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0c8b9640 1 --2- 172.21.15.78:0/2989849779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f081537a0 0x7f0f080c7630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ace20 tx=0x7f0ef802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8e5df32f3e37aa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0efffff640 1 -- 172.21.15.78:0/2989849779 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80317e0 con 0x7f0f081537a0 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0efffff640 1 -- 172.21.15.78:0/2989849779 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8031940 con 0x7f0f081537a0 2024-02-19T14:52:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0efffff640 1 -- 172.21.15.78:0/2989849779 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8031cd0 con 0x7f0f081537a0 2024-02-19T14:52:42.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2989849779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f081537a0 msgr2=0x7f0f080c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2989849779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f081537a0 0x7f0f080c7630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0f080ace20 tx=0x7f0ef802b880 comp rx=0 tx=0).stop 2024-02-19T14:52:42.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2989849779 shutdown_connections 2024-02-19T14:52:42.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2989849779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f081537a0 0x7f0f080c7630 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2989849779 >> v2:172.21.15.150:3300/0 conn(0x7f0f0814cd50 0x7f0f080c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2989849779 >> 172.21.15.78:0/2989849779 conn(0x7f0f08066e70 msgr2=0x7f0f080a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2989849779 shutdown_connections 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.830+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2989849779 wait complete. 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.831+0000 7f0f0f31e640 1 Processor -- start 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.831+0000 7f0f0f31e640 1 -- start start 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0f31e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f080c7b70 0x7f0f080c39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0f31e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 0x7f0f080c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0f31e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f081537a0 0x7f0f0814d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0f31e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c6e00 con 0x7f0f0814cd50 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0f31e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c3520 con 0x7f0f080c7b70 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0f31e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c2f10 con 0x7f0f081537a0 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 0x7f0f080c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 0x7f0f080c6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41264/0 (socket says 172.21.15.78:41264) 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2715246859 learned_addr learned my addr 172.21.15.78:0/2715246859 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0d8bb640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f081537a0 0x7f0f0814d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0d0ba640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f080c7b70 0x7f0f080c39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f081537a0 msgr2=0x7f0f0814d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f081537a0 0x7f0f0814d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f080c7b70 msgr2=0x7f0f080c39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f080c7b70 0x7f0f080c39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.832+0000 7f0f0c8b9640 1 -- 172.21.15.78:0/2715246859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef80094d0 con 0x7f0f0814cd50 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.833+0000 7f0f0c8b9640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 0x7f0f080c6370 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8034040 tx=0x7f0ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.833+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8004230 con 0x7f0f0814cd50 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.833+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef80043b0 con 0x7f0f0814cd50 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.833+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f080c7050 con 0x7f0f0814cd50 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.833+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8004520 con 0x7f0f0814cd50 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.833+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f080c68b0 con 0x7f0f0814cd50 2024-02-19T14:52:42.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.834+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f080c3690 con 0x7f0f0814cd50 2024-02-19T14:52:42.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.835+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0ef8043070 con 0x7f0f0814cd50 2024-02-19T14:52:42.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.835+0000 7f0efdffb640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0eec06b810 0x7f0eec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:42.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.835+0000 7f0f0d0ba640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0eec06b810 0x7f0eec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:42.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.835+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f0ef80ac770 con 0x7f0f0814cd50 2024-02-19T14:52:42.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.836+0000 7f0f0d0ba640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0eec06b810 0x7f0eec06dcd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0f08148680 tx=0x7f0f04008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:42.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:42.840+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0ef8076e60 con 0x7f0f0814cd50 2024-02-19T14:52:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.017+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f08074990 con 0x7f0f0814cd50 2024-02-19T14:52:43.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.018+0000 7f0efdffb640 1 -- 172.21.15.78:0/2715246859 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ef807a520 con 0x7f0f0814cd50 2024-02-19T14:52:43.020 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0eec06b810 msgr2=0x7f0eec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0eec06b810 0x7f0eec06dcd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0f08148680 tx=0x7f0f04008040 comp rx=0 tx=0).stop 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 msgr2=0x7f0f080c6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 0x7f0f080c6370 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8034040 tx=0x7f0ef80046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 shutdown_connections 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0eec06b810 0x7f0eec06dcd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f0814cd50 0x7f0f080c6370 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f081537a0 0x7f0f0814d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 --2- 172.21.15.78:0/2715246859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f080c7b70 0x7f0f080c39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 >> 172.21.15.78:0/2715246859 conn(0x7f0f08066e70 msgr2=0x7f0f08151cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 shutdown_connections 2024-02-19T14:52:43.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.020+0000 7f0f0f31e640 1 -- 172.21.15.78:0/2715246859 wait complete. 2024-02-19T14:52:43.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.139+0000 7f22b4c2a640 1 Processor -- start 2024-02-19T14:52:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.140+0000 7f22b4c2a640 1 -- start start 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22b4c2a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22b00c5bc0 0x7f22b0155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22b4c2a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00ca140 con 0x7f22b01523a0 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22b4c2a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b00c51b0 con 0x7f22b00c8df0 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22b4c2a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b00c5a50 con 0x7f22b00c5bc0 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22ae59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f22b01523a0 0x7f22b006bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52682/0 (socket says 172.21.15.78:52682) 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22ae59c640 1 -- 172.21.15.78:0/670416609 learned_addr learned my addr 172.21.15.78:0/670416609 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22add9b640 1 --2- 172.21.15.78:0/670416609 >> v2:172.21.15.78:3300/0 conn(0x7f22b00c5bc0 0x7f22b0155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22ad59a640 1 -- 172.21.15.78:0/670416609 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784101649 0 0) 0x7f22b00ca140 con 0x7f22b01523a0 2024-02-19T14:52:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22ad59a640 1 -- 172.21.15.78:0/670416609 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2298003520 con 0x7f22b01523a0 2024-02-19T14:52:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.141+0000 7f22add9b640 1 -- 172.21.15.78:0/670416609 >> v1:172.21.15.150:6789/0 conn(0x7f22b00c8df0 legacy=0x7f22b00c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.142+0000 7f22add9b640 1 -- 172.21.15.78:0/670416609 >> v1:172.21.15.78:6789/0 conn(0x7f22b01523a0 legacy=0x7f22b006bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.142+0000 7f22add9b640 1 -- 172.21.15.78:0/670416609 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b00c5490 con 0x7f22b00c5bc0 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.142+0000 7f22add9b640 1 --2- 172.21.15.78:0/670416609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c5bc0 0x7f22b0155950 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f22a40097a0 tx=0x7f22a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b64e6b08d411cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.142+0000 7f22ad59a640 1 -- 172.21.15.78:0/670416609 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031830 con 0x7f22b00c5bc0 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.142+0000 7f22ad59a640 1 -- 172.21.15.78:0/670416609 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a4031990 con 0x7f22b00c5bc0 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.142+0000 7f22ad59a640 1 -- 172.21.15.78:0/670416609 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031d20 con 0x7f22b00c5bc0 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 -- 172.21.15.78:0/670416609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c5bc0 msgr2=0x7f22b0155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/670416609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c5bc0 0x7f22b0155950 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f22a40097a0 tx=0x7f22a402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 -- 172.21.15.78:0/670416609 shutdown_connections 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/670416609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c5bc0 0x7f22b0155950 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 -- 172.21.15.78:0/670416609 >> 172.21.15.78:0/670416609 conn(0x7f22b00ac5a0 msgr2=0x7f22b00ac9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 -- 172.21.15.78:0/670416609 shutdown_connections 2024-02-19T14:52:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.143+0000 7f22b4c2a640 1 -- 172.21.15.78:0/670416609 wait complete. 2024-02-19T14:52:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.145+0000 7f22b4c2a640 1 Processor -- start 2024-02-19T14:52:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.145+0000 7f22b4c2a640 1 -- start start 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22b4c2a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22b00c5bc0 0x7f22b00ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22b4c2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 0x7f22b014bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22b4c2a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b01523a0 0x7f22b014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22b4c2a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00be0a0 con 0x7f22b00c8df0 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22b4c2a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00ae3b0 con 0x7f22b01523a0 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22b4c2a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b01ee990 con 0x7f22b00c5bc0 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22add9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 0x7f22b014bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22add9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 0x7f22b014bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41274/0 (socket says 172.21.15.78:41274) 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.146+0000 7f22add9b640 1 -- 172.21.15.78:0/3348161856 learned_addr learned my addr 172.21.15.78:0/3348161856 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22ae59c640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22b00c5bc0 0x7f22b00ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22aed9d640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b01523a0 0x7f22b014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22add9b640 1 -- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22b00c5bc0 msgr2=0x7f22b00ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22add9b640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22b00c5bc0 0x7f22b00ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22add9b640 1 -- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b01523a0 msgr2=0x7f22b014c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22add9b640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b01523a0 0x7f22b014c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22add9b640 1 -- 172.21.15.78:0/3348161856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4009400 con 0x7f22b00c8df0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22add9b640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 0x7f22b014bc80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22a402bee0 tx=0x7f22a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a403a6f0 con 0x7f22b00c8df0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a403a870 con 0x7f22b00c8df0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b00ae650 con 0x7f22b00c8df0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a403ac20 con 0x7f22b00c8df0 2024-02-19T14:52:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.147+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b00abb60 con 0x7f22b00c8df0 2024-02-19T14:52:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.148+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b0074990 con 0x7f22b00c8df0 2024-02-19T14:52:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.148+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f22a404c3b0 con 0x7f22b00c8df0 2024-02-19T14:52:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.149+0000 7f229effd640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f229006b840 0x7f229006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.149+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f22a40ac200 con 0x7f22b00c8df0 2024-02-19T14:52:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.149+0000 7f22ae59c640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f229006b840 0x7f229006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.149+0000 7f22ae59c640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f229006b840 0x7f229006dd00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f22a8001560 tx=0x7f22a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.157+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f22a40767c0 con 0x7f22b00c8df0 2024-02-19T14:52:43.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.335+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f22b0155cf0 con 0x7f22b00c8df0 2024-02-19T14:52:43.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.336+0000 7f229effd640 1 -- 172.21.15.78:0/3348161856 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f22a4079e80 con 0x7f22b00c8df0 2024-02-19T14:52:43.339 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f229006b840 msgr2=0x7f229006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f229006b840 0x7f229006dd00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f22a8001560 tx=0x7f22a8009040 comp rx=0 tx=0).stop 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 msgr2=0x7f22b014bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 0x7f22b014bc80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f22a402bee0 tx=0x7f22a40046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 shutdown_connections 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f229006b840 0x7f229006dd00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22b00c8df0 0x7f22b014bc80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b01523a0 0x7f22b014c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 --2- 172.21.15.78:0/3348161856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22b00c5bc0 0x7f22b00ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 >> 172.21.15.78:0/3348161856 conn(0x7f22b00ac5a0 msgr2=0x7f22b0150430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 shutdown_connections 2024-02-19T14:52:43.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.339+0000 7f22b4c2a640 1 -- 172.21.15.78:0/3348161856 wait complete. 2024-02-19T14:52:43.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T14:52:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.452+0000 7f56777c9640 1 Processor -- start 2024-02-19T14:52:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.452+0000 7f56777c9640 1 -- start start 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f56777c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f567014f3a0 0x7f56700c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f56777c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56700c12a0 0x7f56700723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f56777c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567006f330 con 0x7f5670072910 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f56777c9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56700a84c0 con 0x7f567014f3a0 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f56777c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56700a8fb0 con 0x7f56700c12a0 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f567014f3a0 0x7f56700c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f567014f3a0 0x7f56700c30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41288/0 (socket says 172.21.15.78:41288) 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 -- 172.21.15.78:0/1093826501 learned_addr learned my addr 172.21.15.78:0/1093826501 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5674d64640 1 --2- 172.21.15.78:0/1093826501 >> v2:172.21.15.150:3300/0 conn(0x7f56700c12a0 0x7f56700723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 -- 172.21.15.78:0/1093826501 >> v2:172.21.15.150:3300/0 conn(0x7f56700c12a0 msgr2=0x7f56700723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 --2- 172.21.15.78:0/1093826501 >> v2:172.21.15.150:3300/0 conn(0x7f56700c12a0 0x7f56700723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 -- 172.21.15.78:0/1093826501 >> v1:172.21.15.150:6789/0 conn(0x7f5670072910 legacy=0x7f5670151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5675565640 1 -- 172.21.15.78:0/1093826501 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56700c35f0 con 0x7f567014f3a0 2024-02-19T14:52:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.453+0000 7f5674d64640 1 --2- 172.21.15.78:0/1093826501 >> v2:172.21.15.150:3300/0 conn(0x7f56700c12a0 0x7f56700723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.454+0000 7f5675565640 1 --2- 172.21.15.78:0/1093826501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567014f3a0 0x7f56700c30b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f567006cb70 tx=0x7f566c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec08c1b6b5ac2680 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.454+0000 7f5667fff640 1 -- 172.21.15.78:0/1093826501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c031750 con 0x7f567014f3a0 2024-02-19T14:52:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.454+0000 7f5667fff640 1 -- 172.21.15.78:0/1093826501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c0318b0 con 0x7f567014f3a0 2024-02-19T14:52:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.454+0000 7f5667fff640 1 -- 172.21.15.78:0/1093826501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c031c40 con 0x7f567014f3a0 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 -- 172.21.15.78:0/1093826501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567014f3a0 msgr2=0x7f56700c30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 --2- 172.21.15.78:0/1093826501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567014f3a0 0x7f56700c30b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f567006cb70 tx=0x7f566c02b7f0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 -- 172.21.15.78:0/1093826501 shutdown_connections 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 --2- 172.21.15.78:0/1093826501 >> v2:172.21.15.150:3300/0 conn(0x7f56700c12a0 0x7f56700723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 --2- 172.21.15.78:0/1093826501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567014f3a0 0x7f56700c30b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 -- 172.21.15.78:0/1093826501 >> 172.21.15.78:0/1093826501 conn(0x7f56700718e0 msgr2=0x7f5670071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 -- 172.21.15.78:0/1093826501 shutdown_connections 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.455+0000 7f56777c9640 1 -- 172.21.15.78:0/1093826501 wait complete. 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.456+0000 7f56777c9640 1 Processor -- start 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.456+0000 7f56777c9640 1 -- start start 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f56777c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5670072910 0x7f56700c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f56777c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 0x7f56701393b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f56777c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567014f3a0 0x7f56701398f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f56777c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5670149e40 con 0x7f56700c12a0 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f56777c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567014bbd0 con 0x7f5670072910 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f56777c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567014b050 con 0x7f567014f3a0 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 0x7f56701393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 0x7f56701393b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41296/0 (socket says 172.21.15.78:41296) 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 -- 172.21.15.78:0/4156549580 learned_addr learned my addr 172.21.15.78:0/4156549580 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5675565640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5670072910 0x7f56700c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5675d66640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567014f3a0 0x7f56701398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 -- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567014f3a0 msgr2=0x7f56701398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567014f3a0 0x7f56701398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 -- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5670072910 msgr2=0x7f56700c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5670072910 0x7f56700c0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 -- 172.21.15.78:0/4156549580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c0094d0 con 0x7f56700c12a0 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5675d66640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567014f3a0 0x7f56701398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5674d64640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 0x7f56701393b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f566000c7f0 tx=0x7f566000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.457+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660007a30 con 0x7f56700c12a0 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.458+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660007b90 con 0x7f56700c12a0 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.458+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567014b330 con 0x7f56700c12a0 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.458+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566000ce40 con 0x7f56700c12a0 2024-02-19T14:52:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.458+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567014b8f0 con 0x7f56700c12a0 2024-02-19T14:52:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.459+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5670074990 con 0x7f56700c12a0 2024-02-19T14:52:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.459+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f56600113f0 con 0x7f56700c12a0 2024-02-19T14:52:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.459+0000 7f5665ffb640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565406b810 0x7f565406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.460+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f566008a700 con 0x7f56700c12a0 2024-02-19T14:52:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.460+0000 7f5675565640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565406b810 0x7f565406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.460+0000 7f5675565640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565406b810 0x7f565406dcd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f566c035000 tx=0x7f566c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.464+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5660054d50 con 0x7f56700c12a0 2024-02-19T14:52:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.653+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f567006f4e0 con 0x7f56700c12a0 2024-02-19T14:52:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.654+0000 7f5665ffb640 1 -- 172.21.15.78:0/4156549580 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5660058410 con 0x7f56700c12a0 2024-02-19T14:52:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:listed 0 entries 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565406b810 msgr2=0x7f565406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565406b810 0x7f565406dcd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f566c035000 tx=0x7f566c008030 comp rx=0 tx=0).stop 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 msgr2=0x7f56701393b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 0x7f56701393b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f566000c7f0 tx=0x7f566000cbf0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 shutdown_connections 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565406b810 0x7f565406dcd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c12a0 0x7f56701393b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567014f3a0 0x7f56701398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 --2- 172.21.15.78:0/4156549580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5670072910 0x7f56700c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 >> 172.21.15.78:0/4156549580 conn(0x7f56700718e0 msgr2=0x7f567006e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 shutdown_connections 2024-02-19T14:52:43.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.656+0000 7f56777c9640 1 -- 172.21.15.78:0/4156549580 wait complete. 2024-02-19T14:52:43.669 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T14:52:43.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:43.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-02-19T14:52:43.697 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T14:52:43.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:43.697 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T14:52:43.753 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:52:43.753 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:43.772 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:52:43.772 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:43.830 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:52:43.831 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:52:43.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:43.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.985+0000 7f5b2a95e640 1 Processor -- start 2024-02-19T14:52:43.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.985+0000 7f5b2a95e640 1 -- start start 2024-02-19T14:52:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b2a95e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5b240719d0 0x7f5b24071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b2a95e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240c15b0 0x7f5b240c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b2a95e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240c1120 con 0x7f5b2414ece0 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b2a95e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240c6080 con 0x7f5b240719d0 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b2a95e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240c57e0 con 0x7f5b240c15b0 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b23fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5b2414ece0 0x7f5b240c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52684/0 (socket says 172.21.15.78:52684) 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b23fff640 1 -- 172.21.15.78:0/424242904 learned_addr learned my addr 172.21.15.78:0/424242904 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.986+0000 7f5b237fe640 1 --2- 172.21.15.78:0/424242904 >> v2:172.21.15.78:3300/0 conn(0x7f5b240719d0 0x7f5b24071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.987+0000 7f5b28efb640 1 --2- 172.21.15.78:0/424242904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240c15b0 0x7f5b240c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.987+0000 7f5b237fe640 1 -- 172.21.15.78:0/424242904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240c15b0 msgr2=0x7f5b240c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.987+0000 7f5b237fe640 1 --2- 172.21.15.78:0/424242904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240c15b0 0x7f5b240c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.987+0000 7f5b237fe640 1 -- 172.21.15.78:0/424242904 >> v1:172.21.15.78:6789/0 conn(0x7f5b2414ece0 legacy=0x7f5b240c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.987+0000 7f5b237fe640 1 -- 172.21.15.78:0/424242904 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240c5ac0 con 0x7f5b240719d0 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.987+0000 7f5b237fe640 1 --2- 172.21.15.78:0/424242904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240719d0 0x7f5b24071e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5b180097a0 tx=0x7f5b1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6802fb19175c1329 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b22ffd640 1 -- 172.21.15.78:0/424242904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18031830 con 0x7f5b240719d0 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b22ffd640 1 -- 172.21.15.78:0/424242904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18031990 con 0x7f5b240719d0 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b22ffd640 1 -- 172.21.15.78:0/424242904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18031d20 con 0x7f5b240719d0 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 -- 172.21.15.78:0/424242904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240719d0 msgr2=0x7f5b24071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/424242904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240719d0 0x7f5b24071e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5b180097a0 tx=0x7f5b1802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 -- 172.21.15.78:0/424242904 shutdown_connections 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/424242904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240719d0 0x7f5b24071e50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/424242904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240c15b0 0x7f5b240c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 -- 172.21.15.78:0/424242904 >> 172.21.15.78:0/424242904 conn(0x7f5b2406b270 msgr2=0x7f5b2406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 -- 172.21.15.78:0/424242904 shutdown_connections 2024-02-19T14:52:43.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.988+0000 7f5b2a95e640 1 -- 172.21.15.78:0/424242904 wait complete. 2024-02-19T14:52:43.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.989+0000 7f5b2a95e640 1 Processor -- start 2024-02-19T14:52:43.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.989+0000 7f5b2a95e640 1 -- start start 2024-02-19T14:52:43.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b2a95e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240719d0 0x7f5b24141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b2a95e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 0x7f5b2414d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b2a95e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2414ece0 0x7f5b241436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b2a95e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2414f2e0 con 0x7f5b240c15b0 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b2a95e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240ac380 con 0x7f5b2414ece0 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b2a95e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2414f5a0 con 0x7f5b240719d0 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 0x7f5b2414d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 0x7f5b2414d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41320/0 (socket says 172.21.15.78:41320) 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b237fe640 1 -- 172.21.15.78:0/1882568561 learned_addr learned my addr 172.21.15.78:0/1882568561 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b23fff640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240719d0 0x7f5b24141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.990+0000 7f5b28efb640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2414ece0 0x7f5b241436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b237fe640 1 -- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240719d0 msgr2=0x7f5b24141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b237fe640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240719d0 0x7f5b24141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b237fe640 1 -- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2414ece0 msgr2=0x7f5b241436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b237fe640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2414ece0 0x7f5b241436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b237fe640 1 -- 172.21.15.78:0/1882568561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18009400 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b237fe640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 0x7f5b2414d240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5b18034040 tx=0x7f5b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1803aa60 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1803abe0 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240ac0a0 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18044400 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.991+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2414d780 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.992+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24074990 con 0x7f5b240c15b0 2024-02-19T14:52:43.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.992+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5b1804d440 con 0x7f5b240c15b0 2024-02-19T14:52:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.993+0000 7f5b20ff9640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b1006b810 0x7f5b1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.993+0000 7f5b23fff640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b1006b810 0x7f5b1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.993+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f5b180ac640 con 0x7f5b240c15b0 2024-02-19T14:52:43.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.993+0000 7f5b23fff640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b1006b810 0x7f5b1006dcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b2415bfd0 tx=0x7f5b14007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:43.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:43.997+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5b18076d40 con 0x7f5b240c15b0 2024-02-19T14:52:44.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.233+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b240c3db0 con 0x7f5b240c15b0 2024-02-19T14:52:44.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.234+0000 7f5b20ff9640 1 -- 172.21.15.78:0/1882568561 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19031 (secure 0 0 0) 0x7f5b1807a400 con 0x7f5b240c15b0 2024-02-19T14:52:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:44.237 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:41.184968+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":0}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:44.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b1006b810 msgr2=0x7f5b1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b1006b810 0x7f5b1006dcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b2415bfd0 tx=0x7f5b14007e80 comp rx=0 tx=0).stop 2024-02-19T14:52:44.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 msgr2=0x7f5b2414d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 0x7f5b2414d240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5b18034040 tx=0x7f5b180046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 shutdown_connections 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b1006b810 0x7f5b1006dcd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b240c15b0 0x7f5b2414d240 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2414ece0 0x7f5b241436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 --2- 172.21.15.78:0/1882568561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b240719d0 0x7f5b24141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 >> 172.21.15.78:0/1882568561 conn(0x7f5b2406b270 msgr2=0x7f5b240c36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 shutdown_connections 2024-02-19T14:52:44.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.237+0000 7f5b2a95e640 1 -- 172.21.15.78:0/1882568561 wait complete. 2024-02-19T14:52:44.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:44.251 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.355+0000 7f93bc93a640 1 Processor -- start 2024-02-19T14:52:44.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.355+0000 7f93bc93a640 1 -- start start 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93bc93a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93b80c5560 0x7f93b80c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93bc93a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 0x7f93b8070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93bc93a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b80c65a0 con 0x7f93b80c7910 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93bc93a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b80c7530 con 0x7f93b80c5560 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93bc93a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b81537b0 con 0x7f93b8070720 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b5d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f93b80c7910 0x7f93b8154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52700/0 (socket says 172.21.15.78:52700) 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b5d9b640 1 -- 172.21.15.78:0/53778763 learned_addr learned my addr 172.21.15.78:0/53778763 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b659c640 1 --2- 172.21.15.78:0/53778763 >> v2:172.21.15.150:3300/0 conn(0x7f93b80c5560 0x7f93b80c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b6d9d640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 0x7f93b8070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696154403 0 0) 0x7f93b80c65a0 con 0x7f93b80c7910 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a0003520 con 0x7f93b80c7910 2024-02-19T14:52:44.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531284855 0 0) 0x7f93a0003520 con 0x7f93b80c7910 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 msgr2=0x7f93b8070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 0x7f93b8070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 >> v2:172.21.15.150:3300/0 conn(0x7f93b80c5560 msgr2=0x7f93b80c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 --2- 172.21.15.78:0/53778763 >> v2:172.21.15.150:3300/0 conn(0x7f93b80c5560 0x7f93b80c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8153a90 con 0x7f93b80c7910 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f93a8001670 con 0x7f93b80c7910 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.356+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 >> v1:172.21.15.78:6789/0 conn(0x7f93b80c7910 legacy=0x7f93b8154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b559a640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a0003970 0x7f93a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b559a640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a00047c0 0x7f93a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b559a640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 0x7f93a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b80c7530 con 0x7f93a0003970 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b81537b0 con 0x7f93a00047c0 2024-02-19T14:52:44.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a000ba00 con 0x7f93b8070720 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b659c640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a0003970 0x7f93a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b5d9b640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a00047c0 0x7f93a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b6d9d640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 0x7f93a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b659c640 1 -- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 msgr2=0x7f93a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b659c640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93b8070720 0x7f93a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b659c640 1 -- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a00047c0 msgr2=0x7f93a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b659c640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a00047c0 0x7f93a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.357+0000 7f93b659c640 1 -- 172.21.15.78:0/53778763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a0003520 con 0x7f93a0003970 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93b659c640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a0003970 0x7f93a0003d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009ac0 tx=0x7f93a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d5b61f61fb27d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac02c710 con 0x7f93a0003970 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac02c870 con 0x7f93a0003970 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93b559a640 1 -- 172.21.15.78:0/53778763 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac02cc00 con 0x7f93a0003970 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93bc93a640 1 -- 172.21.15.78:0/53778763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a0003970 msgr2=0x7f93a0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93bc93a640 1 --2- 172.21.15.78:0/53778763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93a0003970 0x7f93a0003d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009ac0 tx=0x7f93a00057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93b6d9d640 1 -- 172.21.15.78:0/53778763 reap_dead start 2024-02-19T14:52:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93bc93a640 1 -- 172.21.15.78:0/53778763 shutdown_connections 2024-02-19T14:52:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.358+0000 7f93bc93a640 1 -- 172.21.15.78:0/53778763 >> 172.21.15.78:0/53778763 conn(0x7f93b806b4b0 msgr2=0x7f93b80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.359+0000 7f93bc93a640 1 -- 172.21.15.78:0/53778763 shutdown_connections 2024-02-19T14:52:44.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.359+0000 7f93bc93a640 1 -- 172.21.15.78:0/53778763 wait complete. 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 Processor -- start 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 -- start start 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a0003970 0x7f93b80be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 0x7f93b81e8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93b81e8950 0x7f93b81eb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b80a8030 con 0x7f93b80beeb0 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b8151e80 con 0x7f93b81e8950 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93bc93a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b80a9f30 con 0x7f93a0003970 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.360+0000 7f93b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 0x7f93b81e8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 0x7f93b81e8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41336/0 (socket says 172.21.15.78:41336) 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 -- 172.21.15.78:0/70125413 learned_addr learned my addr 172.21.15.78:0/70125413 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b659c640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a0003970 0x7f93b80be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b6d9d640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93b81e8950 0x7f93b81eb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 -- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a0003970 msgr2=0x7f93b80be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a0003970 0x7f93b80be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 -- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93b81e8950 msgr2=0x7f93b81eb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93b81e8950 0x7f93b81eb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 -- 172.21.15.78:0/70125413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b80c5ac0 con 0x7f93b80beeb0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b6d9d640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93b81e8950 0x7f93b81eb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b659c640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a0003970 0x7f93b80be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93b5d9b640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 0x7f93b81e8410 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f93a8001e70 tx=0x7f93a80077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8007e10 con 0x7f93b80beeb0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a800f930 con 0x7f93b80beeb0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b80be270 con 0x7f93b80beeb0 2024-02-19T14:52:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a800fca0 con 0x7f93b80beeb0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.361+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b80aa3c0 con 0x7f93b80beeb0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.362+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b80be790 con 0x7f93b80beeb0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.363+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f93a801d460 con 0x7f93b80beeb0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.363+0000 7f93a6ffd640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f939806b840 0x7f939806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.363+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f93a808a630 con 0x7f93b80beeb0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.363+0000 7f93b659c640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f939806b840 0x7f939806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.364+0000 7f93b659c640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f939806b840 0x7f939806dd00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f93ac036000 tx=0x7f93ac030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.367+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f93a8054c80 con 0x7f93b80beeb0 2024-02-19T14:52:44.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.581+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f93b80a9d20 con 0x7f93b80beeb0 2024-02-19T14:52:44.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.581+0000 7f93a6ffd640 1 -- 172.21.15.78:0/70125413 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f93a8058340 con 0x7f93b80beeb0 2024-02-19T14:52:44.584 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:44.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f939806b840 msgr2=0x7f939806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f939806b840 0x7f939806dd00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f93ac036000 tx=0x7f93ac030000 comp rx=0 tx=0).stop 2024-02-19T14:52:44.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 msgr2=0x7f93b81e8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 0x7f93b81e8410 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f93a8001e70 tx=0x7f93a80077a0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 shutdown_connections 2024-02-19T14:52:44.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f939806b840 0x7f939806dd00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b80beeb0 0x7f93b81e8410 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93b81e8950 0x7f93b81eb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 --2- 172.21.15.78:0/70125413 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a0003970 0x7f93b80be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 >> 172.21.15.78:0/70125413 conn(0x7f93b806b4b0 msgr2=0x7f93b80c3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:44.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 shutdown_connections 2024-02-19T14:52:44.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.583+0000 7f93bc93a640 1 -- 172.21.15.78:0/70125413 wait complete. 2024-02-19T14:52:44.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:44.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.703+0000 7f967405c640 1 Processor -- start 2024-02-19T14:52:44.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.703+0000 7f967405c640 1 -- start start 2024-02-19T14:52:44.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f967405c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f966c1525c0 0x7f966c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f967405c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c06d560 0x7f966c0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f967405c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c0c6460 con 0x7f966c0c6db0 2024-02-19T14:52:44.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f967405c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f966c0c1f50 con 0x7f966c1525c0 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f967405c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c1593a0 con 0x7f966c06d560 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f96715f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c06d560 0x7f966c0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f9671df8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f966c1525c0 0x7f966c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f96715f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c06d560 0x7f966c0c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33092/0 (socket says 172.21.15.78:33092) 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f9671df8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f966c1525c0 0x7f966c06d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:37752/0 (socket says 172.21.15.78:37752) 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f96725f9640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f966c0c6db0 0x7f966c155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:43688/0 (socket says 172.21.15.78:43688) 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.704+0000 7f96715f7640 1 -- 172.21.15.78:0/3210163596 learned_addr learned my addr 172.21.15.78:0/3210163596 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.705+0000 7f9671df8640 1 -- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c06d560 msgr2=0x7f966c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.705+0000 7f9671df8640 1 --2- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c06d560 0x7f966c0c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.705+0000 7f9671df8640 1 -- 172.21.15.78:0/3210163596 >> v1:172.21.15.150:6789/0 conn(0x7f966c0c6db0 legacy=0x7f966c155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.705+0000 7f9671df8640 1 -- 172.21.15.78:0/3210163596 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c158720 con 0x7f966c1525c0 2024-02-19T14:52:44.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.705+0000 7f9671df8640 1 --2- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c1525c0 0x7f966c06d020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96680097d0 tx=0x7f966802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9557c8a1170cc645 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:44.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f9670df6640 1 -- 172.21.15.78:0/3210163596 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668031770 con 0x7f966c1525c0 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f9670df6640 1 -- 172.21.15.78:0/3210163596 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96680318d0 con 0x7f966c1525c0 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f9670df6640 1 -- 172.21.15.78:0/3210163596 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668031c60 con 0x7f966c1525c0 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 -- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c1525c0 msgr2=0x7f966c06d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 --2- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c1525c0 0x7f966c06d020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96680097d0 tx=0x7f966802b810 comp rx=0 tx=0).stop 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 -- 172.21.15.78:0/3210163596 shutdown_connections 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 --2- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c06d560 0x7f966c0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 --2- 172.21.15.78:0/3210163596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c1525c0 0x7f966c06d020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 -- 172.21.15.78:0/3210163596 >> 172.21.15.78:0/3210163596 conn(0x7f966c0cab20 msgr2=0x7f966c0caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 -- 172.21.15.78:0/3210163596 shutdown_connections 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.706+0000 7f967405c640 1 -- 172.21.15.78:0/3210163596 wait complete. 2024-02-19T14:52:44.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.707+0000 7f967405c640 1 Processor -- start 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.707+0000 7f967405c640 1 -- start start 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f967405c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c06d560 0x7f966c1571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f967405c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 0x7f966c1edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f967405c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c1525c0 0x7f966c14a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f967405c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c156460 con 0x7f966c0c6db0 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f967405c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c1545a0 con 0x7f966c06d560 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f967405c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c1ea770 con 0x7f966c1525c0 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f96715f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 0x7f966c1edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f96715f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 0x7f966c1edc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41352/0 (socket says 172.21.15.78:41352) 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f96715f7640 1 -- 172.21.15.78:0/401756859 learned_addr learned my addr 172.21.15.78:0/401756859 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f9671df8640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c06d560 0x7f966c1571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f96725f9640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c1525c0 0x7f966c14a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f96715f7640 1 -- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c1525c0 msgr2=0x7f966c14a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.708+0000 7f96715f7640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c1525c0 0x7f966c14a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96715f7640 1 -- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c06d560 msgr2=0x7f966c1571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96715f7640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c06d560 0x7f966c1571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96715f7640 1 -- 172.21.15.78:0/401756859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9668009400 con 0x7f966c0c6db0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f9671df8640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c06d560 0x7f966c1571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96715f7640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 0x7f966c1edc30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f965c009fd0 tx=0x7f965c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c00c940 con 0x7f966c0c6db0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c1eaa50 con 0x7f966c0c6db0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c00caa0 con 0x7f966c0c6db0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c00ce30 con 0x7f966c0c6db0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.709+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c157730 con 0x7f966c0c6db0 2024-02-19T14:52:44.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.710+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c074990 con 0x7f966c0c6db0 2024-02-19T14:52:44.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.710+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f965c013470 con 0x7f966c0c6db0 2024-02-19T14:52:44.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.711+0000 7f96627fc640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f965006b810 0x7f965006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:44.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.711+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f965c08a840 con 0x7f966c0c6db0 2024-02-19T14:52:44.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.711+0000 7f9671df8640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f965006b810 0x7f965006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:44.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.711+0000 7f9671df8640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f965006b810 0x7f965006dcd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9668036000 tx=0x7f966803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:44.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.719+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f965c054d80 con 0x7f966c0c6db0 2024-02-19T14:52:44.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.896+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f966c0c6650 con 0x7f966c0c6db0 2024-02-19T14:52:44.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.896+0000 7f96627fc640 1 -- 172.21.15.78:0/401756859 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f965c058440 con 0x7f966c0c6db0 2024-02-19T14:52:44.899 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f965006b810 msgr2=0x7f965006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f965006b810 0x7f965006dcd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9668036000 tx=0x7f966803d040 comp rx=0 tx=0).stop 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 msgr2=0x7f966c1edc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 0x7f966c1edc30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f965c009fd0 tx=0x7f965c00ed40 comp rx=0 tx=0).stop 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 shutdown_connections 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f965006b810 0x7f965006dcd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f966c0c6db0 0x7f966c1edc30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f966c1525c0 0x7f966c14a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 --2- 172.21.15.78:0/401756859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f966c06d560 0x7f966c1571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 >> 172.21.15.78:0/401756859 conn(0x7f966c0cab20 msgr2=0x7f966c0caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:44.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 shutdown_connections 2024-02-19T14:52:44.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:44.898+0000 7f967405c640 1 -- 172.21.15.78:0/401756859 wait complete. 2024-02-19T14:52:44.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:45.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.014+0000 7f3eca920640 1 Processor -- start 2024-02-19T14:52:45.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.014+0000 7f3eca920640 1 -- start start 2024-02-19T14:52:45.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3eca920640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 0x7f3ec40c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3eca920640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec406ff90 con 0x7f3ec414d210 2024-02-19T14:52:45.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3eca920640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec4156fc0 con 0x7f3ec40c3d90 2024-02-19T14:52:45.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3eca920640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec4070270 con 0x7f3ec40719e0 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3ec3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3ec414d210 0x7f3ec40c4290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52706/0 (socket says 172.21.15.78:52706) 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3ec3fff640 1 -- 172.21.15.78:0/364690887 learned_addr learned my addr 172.21.15.78:0/364690887 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.015+0000 7f3ec37fe640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 0x7f3ec40c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472402242 0 0) 0x7f3ec406ff90 con 0x7f3ec414d210 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eac003520 con 0x7f3ec414d210 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535704098 0 0) 0x7f3eac003520 con 0x7f3ec414d210 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 msgr2=0x7f3ec40c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 0x7f3ec40c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 >> v1:172.21.15.150:6789/0 conn(0x7f3ec40c3d90 legacy=0x7f3ec415d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec41572a0 con 0x7f3ec414d210 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3eb8001670 con 0x7f3ec414d210 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 >> v1:172.21.15.78:6789/0 conn(0x7f3ec414d210 legacy=0x7f3ec40c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eac0039a0 0x7f3eac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eac0047c0 0x7f3eac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 0x7f3eac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec4070270 con 0x7f3eac0039a0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac00bab0 con 0x7f3eac0047c0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac00bc70 con 0x7f3ec40719e0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eac0039a0 0x7f3eac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec37fe640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eac0047c0 0x7f3eac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec8ebd640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 0x7f3eac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec3fff640 1 -- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 msgr2=0x7f3eac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec40719e0 0x7f3eac0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec3fff640 1 -- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eac0047c0 msgr2=0x7f3eac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eac0047c0 0x7f3eac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec3fff640 1 -- 172.21.15.78:0/364690887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac003520 con 0x7f3eac0039a0 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.016+0000 7f3ec37fe640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eac0047c0 0x7f3eac004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:45.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.017+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eac0039a0 0x7f3eac003dc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800c190 tx=0x7f3eac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcdf642368d4f248 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.017+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8005ad0 con 0x7f3eac0039a0 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.017+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb8005c50 con 0x7f3eac0039a0 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.017+0000 7f3ec2ffd640 1 -- 172.21.15.78:0/364690887 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb802c8a0 con 0x7f3eac0039a0 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3eca920640 1 -- 172.21.15.78:0/364690887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eac0039a0 msgr2=0x7f3eac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3eca920640 1 --2- 172.21.15.78:0/364690887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eac0039a0 0x7f3eac003dc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800c190 tx=0x7f3eac0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3ec8ebd640 1 -- 172.21.15.78:0/364690887 reap_dead start 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3eca920640 1 -- 172.21.15.78:0/364690887 shutdown_connections 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3eca920640 1 -- 172.21.15.78:0/364690887 >> 172.21.15.78:0/364690887 conn(0x7f3ec40c5050 msgr2=0x7f3ec40c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3eca920640 1 -- 172.21.15.78:0/364690887 shutdown_connections 2024-02-19T14:52:45.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.018+0000 7f3eca920640 1 -- 172.21.15.78:0/364690887 wait complete. 2024-02-19T14:52:45.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.019+0000 7f3eca920640 1 Processor -- start 2024-02-19T14:52:45.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.019+0000 7f3eca920640 1 -- start start 2024-02-19T14:52:45.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3eca920640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eac0039a0 0x7f3ec41e6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3eca920640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 0x7f3ec4152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3eca920640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec4153420 0x7f3ec414d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3eca920640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c3970 con 0x7f3ec4150b00 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3eca920640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec414fc80 con 0x7f3ec4153420 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3eca920640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec41510c0 con 0x7f3eac0039a0 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3ec37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 0x7f3ec4152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3ec37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 0x7f3ec4152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41360/0 (socket says 172.21.15.78:41360) 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.020+0000 7f3ec37fe640 1 -- 172.21.15.78:0/2899872304 learned_addr learned my addr 172.21.15.78:0/2899872304 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eac0039a0 0x7f3ec41e6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec8ebd640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec4153420 0x7f3ec414d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec37fe640 1 -- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eac0039a0 msgr2=0x7f3ec41e6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec37fe640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eac0039a0 0x7f3ec41e6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec37fe640 1 -- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec4153420 msgr2=0x7f3ec414d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec37fe640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec4153420 0x7f3ec414d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec37fe640 1 -- 172.21.15.78:0/2899872304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40c3660 con 0x7f3ec4150b00 2024-02-19T14:52:45.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec37fe640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 0x7f3ec4152160 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40115e0 tx=0x7f3eb4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400f8f0 con 0x7f3ec4150b00 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb400fa50 con 0x7f3ec4150b00 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400fdc0 con 0x7f3ec4150b00 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec41513a0 con 0x7f3ec4150b00 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.021+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec4151960 con 0x7f3ec4150b00 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.022+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec41526a0 con 0x7f3ec4150b00 2024-02-19T14:52:45.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.023+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3eb4015460 con 0x7f3ec4150b00 2024-02-19T14:52:45.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.023+0000 7f3ec0ff9640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3eb006b840 0x7f3eb006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.023+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f3eb408ad30 con 0x7f3ec4150b00 2024-02-19T14:52:45.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.023+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3eb006b840 0x7f3eb006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.024+0000 7f3ec3fff640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3eb006b840 0x7f3eb006dd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8034000 tx=0x7f3eb8009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.028+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3eb4055380 con 0x7f3ec4150b00 2024-02-19T14:52:45.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.204+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ec4074990 con 0x7f3ec4150b00 2024-02-19T14:52:45.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.205+0000 7f3ec0ff9640 1 -- 172.21.15.78:0/2899872304 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3eb4058a40 con 0x7f3ec4150b00 2024-02-19T14:52:45.208 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3eb006b840 msgr2=0x7f3eb006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3eb006b840 0x7f3eb006dd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8034000 tx=0x7f3eb8009030 comp rx=0 tx=0).stop 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 msgr2=0x7f3ec4152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 0x7f3ec4152160 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40115e0 tx=0x7f3eb4011aa0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 shutdown_connections 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3eb006b840 0x7f3eb006dd00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec4150b00 0x7f3ec4152160 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec4153420 0x7f3ec414d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 --2- 172.21.15.78:0/2899872304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eac0039a0 0x7f3ec41e6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.207+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 >> 172.21.15.78:0/2899872304 conn(0x7f3ec40c5050 msgr2=0x7f3ec40c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:45.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.208+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 shutdown_connections 2024-02-19T14:52:45.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.208+0000 7f3eca920640 1 -- 172.21.15.78:0/2899872304 wait complete. 2024-02-19T14:52:45.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:45.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.321+0000 7f6c7ef2c640 1 Processor -- start 2024-02-19T14:52:45.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ef2c640 1 -- start start 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ef2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6c78152700 0x7f6c780c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ef2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c780c8e00 0x7f6c7815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ef2c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c781551f0 con 0x7f6c780c5730 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ef2c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c78155490 con 0x7f6c78152700 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ef2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78155770 con 0x7f6c780c8e00 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ccc8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6c78152700 0x7f6c780c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c77fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6c780c5730 0x7f6c7806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52722/0 (socket says 172.21.15.78:52722) 2024-02-19T14:52:45.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c7ccc8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6c78152700 0x7f6c780c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41372/0 (socket says 172.21.15.78:41372) 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.322+0000 7f6c77fff640 1 -- 172.21.15.78:0/2513246797 learned_addr learned my addr 172.21.15.78:0/2513246797 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.323+0000 7f6c7d4c9640 1 --2- 172.21.15.78:0/2513246797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c780c8e00 0x7f6c7815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.323+0000 7f6c7ccc8640 1 -- 172.21.15.78:0/2513246797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c780c8e00 msgr2=0x7f6c7815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.323+0000 7f6c7ccc8640 1 --2- 172.21.15.78:0/2513246797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c780c8e00 0x7f6c7815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.323+0000 7f6c7ccc8640 1 -- 172.21.15.78:0/2513246797 >> v1:172.21.15.78:6789/0 conn(0x7f6c780c5730 legacy=0x7f6c7806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.323+0000 7f6c777fe640 1 -- 172.21.15.78:0/2513246797 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107869452 0 0) 0x7f6c781551f0 con 0x7f6c780c5730 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.323+0000 7f6c7ccc8640 1 -- 172.21.15.78:0/2513246797 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78155a50 con 0x7f6c78152700 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ccc8640 1 --2- 172.21.15.78:0/2513246797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c78152700 0x7f6c780c51f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009d90 tx=0x7f6c7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57115f0e8eab4599 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c777fe640 1 -- 172.21.15.78:0/2513246797 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7002cc50 con 0x7f6c78152700 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c777fe640 1 -- 172.21.15.78:0/2513246797 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7002cdb0 con 0x7f6c78152700 2024-02-19T14:52:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c777fe640 1 -- 172.21.15.78:0/2513246797 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70034660 con 0x7f6c78152700 2024-02-19T14:52:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2513246797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c78152700 msgr2=0x7f6c780c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2513246797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c78152700 0x7f6c780c51f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009d90 tx=0x7f6c7002bea0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2513246797 shutdown_connections 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2513246797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c780c8e00 0x7f6c7815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2513246797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c78152700 0x7f6c780c51f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2513246797 >> 172.21.15.78:0/2513246797 conn(0x7f6c780cac90 msgr2=0x7f6c780cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2513246797 shutdown_connections 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.324+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2513246797 wait complete. 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.326+0000 7f6c7ef2c640 1 Processor -- start 2024-02-19T14:52:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.326+0000 7f6c7ef2c640 1 -- start start 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c7ef2c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c5730 0x7f6c781ea500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c7ef2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 0x7f6c781eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c7ef2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c78152700 0x7f6c78158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c7ef2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c781eaf80 con 0x7f6c780c8e00 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c7ef2c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c78156860 con 0x7f6c780c5730 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c7ef2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c780c1cc0 con 0x7f6c78152700 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.327+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 0x7f6c781eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 0x7f6c781eaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41376/0 (socket says 172.21.15.78:41376) 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 -- 172.21.15.78:0/2773710554 learned_addr learned my addr 172.21.15.78:0/2773710554 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c7d4c9640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c78152700 0x7f6c78158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c7ccc8640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c5730 0x7f6c781ea500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 -- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c78152700 msgr2=0x7f6c78158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c78152700 0x7f6c78158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 -- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c5730 msgr2=0x7f6c781ea500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c5730 0x7f6c781ea500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 -- 172.21.15.78:0/2773710554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c70009a70 con 0x7f6c780c8e00 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c7d4c9640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c78152700 0x7f6c78158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c7ccc8640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c5730 0x7f6c781ea500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c77fff640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 0x7f6c781eaa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00c4d0 tx=0x7f6c6c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0074e0 con 0x7f6c780c8e00 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c007660 con 0x7f6c780c8e00 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c780aa280 con 0x7f6c780c8e00 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c004a20 con 0x7f6c780c8e00 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.328+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c780aa840 con 0x7f6c780c8e00 2024-02-19T14:52:45.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.329+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c78074990 con 0x7f6c780c8e00 2024-02-19T14:52:45.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.330+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c6c004b80 con 0x7f6c780c8e00 2024-02-19T14:52:45.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.330+0000 7f6c757fa640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c6406b810 0x7f6c6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.330+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f6c6c089aa0 con 0x7f6c780c8e00 2024-02-19T14:52:45.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.330+0000 7f6c7ccc8640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c6406b810 0x7f6c6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.331+0000 7f6c7ccc8640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c6406b810 0x7f6c6406dcd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6c70036000 tx=0x7f6c70030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.335+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c6c0540f0 con 0x7f6c780c8e00 2024-02-19T14:52:45.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.511+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6c781569a0 con 0x7f6c780c8e00 2024-02-19T14:52:45.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.511+0000 7f6c757fa640 1 -- 172.21.15.78:0/2773710554 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6c6c0577b0 con 0x7f6c780c8e00 2024-02-19T14:52:45.514 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c6406b810 msgr2=0x7f6c6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c6406b810 0x7f6c6406dcd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6c70036000 tx=0x7f6c70030000 comp rx=0 tx=0).stop 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 msgr2=0x7f6c781eaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 0x7f6c781eaa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00c4d0 tx=0x7f6c6c00c990 comp rx=0 tx=0).stop 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 shutdown_connections 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c6406b810 0x7f6c6406dcd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c780c8e00 0x7f6c781eaa40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c78152700 0x7f6c78158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 --2- 172.21.15.78:0/2773710554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c5730 0x7f6c781ea500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 >> 172.21.15.78:0/2773710554 conn(0x7f6c780cac90 msgr2=0x7f6c7814ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:45.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 shutdown_connections 2024-02-19T14:52:45.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.514+0000 7f6c7ef2c640 1 -- 172.21.15.78:0/2773710554 wait complete. 2024-02-19T14:52:45.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T14:52:45.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.627+0000 7fe4e2439640 1 Processor -- start 2024-02-19T14:52:45.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.627+0000 7fe4e2439640 1 -- start start 2024-02-19T14:52:45.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4e2439640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe4dc0c3d90 0x7fe4dc15e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4e2439640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc06ff90 con 0x7fe4dc0719e0 2024-02-19T14:52:45.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4e2439640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc156fc0 con 0x7fe4dc14d210 2024-02-19T14:52:45.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4e2439640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc070270 con 0x7fe4dc0c3d90 2024-02-19T14:52:45.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4e09d6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe4dc0c3d90 0x7fe4dc15e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4db7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe4dc0719e0 0x7fe4dc0c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52724/0 (socket says 172.21.15.78:52724) 2024-02-19T14:52:45.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4e09d6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe4dc0c3d90 0x7fe4dc15e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41390/0 (socket says 172.21.15.78:41390) 2024-02-19T14:52:45.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.628+0000 7fe4db7fe640 1 -- 172.21.15.78:0/1218933345 learned_addr learned my addr 172.21.15.78:0/1218933345 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:45.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.629+0000 7fe4e09d6640 1 -- 172.21.15.78:0/1218933345 >> v1:172.21.15.150:6789/0 conn(0x7fe4dc14d210 legacy=0x7fe4dc0c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.629+0000 7fe4e09d6640 1 -- 172.21.15.78:0/1218933345 >> v1:172.21.15.78:6789/0 conn(0x7fe4dc0719e0 legacy=0x7fe4dc0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.629+0000 7fe4e09d6640 1 -- 172.21.15.78:0/1218933345 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc1572a0 con 0x7fe4dc0c3d90 2024-02-19T14:52:45.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.629+0000 7fe4daffd640 1 -- 172.21.15.78:0/1218933345 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382611546 0 0) 0x7fe4dc06ff90 con 0x7fe4dc0719e0 2024-02-19T14:52:45.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.629+0000 7fe4e09d6640 1 --2- 172.21.15.78:0/1218933345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0c3d90 0x7fe4dc15e800 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009d90 tx=0x7fe4d402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7fc30b49c01051 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.629+0000 7fe4daffd640 1 -- 172.21.15.78:0/1218933345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d402cc60 con 0x7fe4dc0c3d90 2024-02-19T14:52:45.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4daffd640 1 -- 172.21.15.78:0/1218933345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d402cdc0 con 0x7fe4dc0c3d90 2024-02-19T14:52:45.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4daffd640 1 -- 172.21.15.78:0/1218933345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4034660 con 0x7fe4dc0c3d90 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 -- 172.21.15.78:0/1218933345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0c3d90 msgr2=0x7fe4dc15e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 --2- 172.21.15.78:0/1218933345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0c3d90 0x7fe4dc15e800 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009d90 tx=0x7fe4d402beb0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 -- 172.21.15.78:0/1218933345 shutdown_connections 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 --2- 172.21.15.78:0/1218933345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0c3d90 0x7fe4dc15e800 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 -- 172.21.15.78:0/1218933345 >> 172.21.15.78:0/1218933345 conn(0x7fe4dc0c5050 msgr2=0x7fe4dc0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 -- 172.21.15.78:0/1218933345 shutdown_connections 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.630+0000 7fe4e2439640 1 -- 172.21.15.78:0/1218933345 wait complete. 2024-02-19T14:52:45.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.631+0000 7fe4e2439640 1 Processor -- start 2024-02-19T14:52:45.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.631+0000 7fe4e2439640 1 -- start start 2024-02-19T14:52:45.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e2439640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 0x7fe4dc0ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e2439640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4dc0c3d90 0x7fe4dc1e8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e2439640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4dc14d210 0x7fe4dc1e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e2439640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0c3970 con 0x7fe4dc0719e0 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e2439640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc1ebc40 con 0x7fe4dc14d210 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e2439640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc1ebe90 con 0x7fe4dc0c3d90 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 0x7fe4dc0ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 0x7fe4dc0ad7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41406/0 (socket says 172.21.15.78:41406) 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 -- 172.21.15.78:0/3470717179 learned_addr learned my addr 172.21.15.78:0/3470717179 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e09d6640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4dc14d210 0x7fe4dc1e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4db7fe640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4dc0c3d90 0x7fe4dc1e8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 -- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4dc0c3d90 msgr2=0x7fe4dc1e8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4dc0c3d90 0x7fe4dc1e8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 -- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4dc14d210 msgr2=0x7fe4dc1e8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4dc14d210 0x7fe4dc1e8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4dbfff640 1 -- 172.21.15.78:0/3470717179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d4009a70 con 0x7fe4dc0719e0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4db7fe640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4dc0c3d90 0x7fe4dc1e8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.632+0000 7fe4e09d6640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4dc14d210 0x7fe4dc1e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.633+0000 7fe4dbfff640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 0x7fe4dc0ad7d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000c400 tx=0x7fe4d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.633+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00052d0 con 0x7fe4dc0719e0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.633+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d0005430 con 0x7fe4dc0719e0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.633+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc1ec170 con 0x7fe4dc0719e0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.633+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0003830 con 0x7fe4dc0719e0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.633+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc1ec730 con 0x7fe4dc0719e0 2024-02-19T14:52:45.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.634+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc074990 con 0x7fe4dc0719e0 2024-02-19T14:52:45.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.634+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe4d0003990 con 0x7fe4dc0719e0 2024-02-19T14:52:45.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.635+0000 7fe4d8ff9640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4c806b810 0x7fe4c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:45.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.635+0000 7fe4db7fe640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4c806b810 0x7fe4c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:45.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.635+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fe4d0089b20 con 0x7fe4dc0719e0 2024-02-19T14:52:45.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.635+0000 7fe4db7fe640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4c806b810 0x7fe4c806dcd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc14f9b0 tx=0x7fe4cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:45.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.643+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe4d0054170 con 0x7fe4dc0719e0 2024-02-19T14:52:45.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.831+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe4dc1e86d0 con 0x7fe4dc0719e0 2024-02-19T14:52:45.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.832+0000 7fe4d8ff9640 1 -- 172.21.15.78:0/3470717179 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe4d0057830 con 0x7fe4dc0719e0 2024-02-19T14:52:45.834 INFO:teuthology.orchestra.run.smithi078.stderr:listed 0 entries 2024-02-19T14:52:45.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4c806b810 msgr2=0x7fe4c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4c806b810 0x7fe4c806dcd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc14f9b0 tx=0x7fe4cc009040 comp rx=0 tx=0).stop 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 msgr2=0x7fe4dc0ad7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 0x7fe4dc0ad7d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000c400 tx=0x7fe4d000c8c0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 shutdown_connections 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe4c806b810 0x7fe4c806dcd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4dc14d210 0x7fe4dc1e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.833+0000 7fe4e2439640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4dc0c3d90 0x7fe4dc1e8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.834+0000 7fe4e2439640 1 --2- 172.21.15.78:0/3470717179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4dc0719e0 0x7fe4dc0ad7d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.834+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 >> 172.21.15.78:0/3470717179 conn(0x7fe4dc0c5050 msgr2=0x7fe4dc14a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.834+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 shutdown_connections 2024-02-19T14:52:45.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:45.834+0000 7fe4e2439640 1 -- 172.21.15.78:0/3470717179 wait complete. 2024-02-19T14:52:45.847 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-02-19T14:52:45.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:45.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-02-19T14:52:45.881 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T14:52:45.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T14:52:45.881 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-02-19T14:52:45.953 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:52:45.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:46.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.051+0000 7fb5ed8d4640 1 Processor -- start 2024-02-19T14:52:46.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.051+0000 7fb5ed8d4640 1 -- start start 2024-02-19T14:52:46.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5ed8d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb5e8149790 0x7fb5e806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5ed8d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e806f420 con 0x7fb5e80723d0 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5ed8d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e80ab760 con 0x7fb5e8072800 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5ed8d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e8149b70 con 0x7fb5e8149790 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5e6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb5e8149790 0x7fb5e806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5e67fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb5e80723d0 0x7fb5e8152c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52728/0 (socket says 172.21.15.78:52728) 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.052+0000 7fb5e67fc640 1 -- 172.21.15.78:0/1501554866 learned_addr learned my addr 172.21.15.78:0/1501554866 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e6ffd640 1 -- 172.21.15.78:0/1501554866 >> v1:172.21.15.150:6789/0 conn(0x7fb5e8072800 legacy=0x7fb5e8153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e6ffd640 1 -- 172.21.15.78:0/1501554866 >> v1:172.21.15.78:6789/0 conn(0x7fb5e80723d0 legacy=0x7fb5e8152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e6ffd640 1 -- 172.21.15.78:0/1501554866 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e80aba40 con 0x7fb5e8149790 2024-02-19T14:52:46.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e5ffb640 1 -- 172.21.15.78:0/1501554866 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242346905 0 0) 0x7fb5e806f420 con 0x7fb5e80723d0 2024-02-19T14:52:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e6ffd640 1 --2- 172.21.15.78:0/1501554866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8149790 0x7fb5e806ed40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0097a0 tx=0x7fb5dc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c9bf3df7311684 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e5ffb640 1 -- 172.21.15.78:0/1501554866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc02f050 con 0x7fb5e8149790 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e5ffb640 1 -- 172.21.15.78:0/1501554866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc031700 con 0x7fb5e8149790 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.053+0000 7fb5e5ffb640 1 -- 172.21.15.78:0/1501554866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc031a90 con 0x7fb5e8149790 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/1501554866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8149790 msgr2=0x7fb5e806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/1501554866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8149790 0x7fb5e806ed40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0097a0 tx=0x7fb5dc02b7a0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/1501554866 shutdown_connections 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/1501554866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8149790 0x7fb5e806ed40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/1501554866 >> 172.21.15.78:0/1501554866 conn(0x7fb5e80718e0 msgr2=0x7fb5e8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/1501554866 shutdown_connections 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.054+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/1501554866 wait complete. 2024-02-19T14:52:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.055+0000 7fb5ed8d4640 1 Processor -- start 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.055+0000 7fb5ed8d4640 1 -- start start 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5ed8d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5e80723d0 0x7fb5e8145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5ed8d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 0x7fb5e81dc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5ed8d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e8149790 0x7fb5e81dcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5ed8d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e8150630 con 0x7fb5e8072800 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5ed8d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e81dd2d0 con 0x7fb5e8149790 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5ed8d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e81dc3c0 con 0x7fb5e80723d0 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 0x7fb5e81dc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 0x7fb5e81dc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41410/0 (socket says 172.21.15.78:41410) 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 -- 172.21.15.78:0/12436949 learned_addr learned my addr 172.21.15.78:0/12436949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 -- 172.21.15.78:0/12436949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5e80723d0 msgr2=0x7fb5e8145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5e80723d0 0x7fb5e8145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 -- 172.21.15.78:0/12436949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e8149790 msgr2=0x7fb5e81dcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e8149790 0x7fb5e81dcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 -- 172.21.15.78:0/12436949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc009400 con 0x7fb5e8072800 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.056+0000 7fb5e67fc640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 0x7fb5e81dc850 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c460 tx=0x7fb5d800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.057+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80074e0 con 0x7fb5e8072800 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.057+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8007660 con 0x7fb5e8072800 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.057+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8004a20 con 0x7fb5e8072800 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.057+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e81524b0 con 0x7fb5e8072800 2024-02-19T14:52:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.057+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e8150840 con 0x7fb5e8072800 2024-02-19T14:52:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.058+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e8074990 con 0x7fb5e8072800 2024-02-19T14:52:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.058+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb5d8004b80 con 0x7fb5e8072800 2024-02-19T14:52:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.058+0000 7fb5cffff640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5d406b840 0x7fb5d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.059+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fb5d8089a90 con 0x7fb5e8072800 2024-02-19T14:52:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.059+0000 7fb5e6ffd640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5d406b840 0x7fb5d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.059+0000 7fb5e6ffd640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5d406b840 0x7fb5d406dd00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc035000 tx=0x7fb5dc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.063+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb5d8054050 con 0x7fb5e8072800 2024-02-19T14:52:46.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.272+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5e81dc500 con 0x7fb5e8072800 2024-02-19T14:52:46.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.273+0000 7fb5cffff640 1 -- 172.21.15.78:0/12436949 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7fb5d8057710 con 0x7fb5e8072800 2024-02-19T14:52:46.276 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:46.276 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:46.276 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:46.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5d406b840 msgr2=0x7fb5d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5d406b840 0x7fb5d406dd00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc035000 tx=0x7fb5dc030000 comp rx=0 tx=0).stop 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 msgr2=0x7fb5e81dc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 0x7fb5e81dc850 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c460 tx=0x7fb5d800c920 comp rx=0 tx=0).stop 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 shutdown_connections 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5d406b840 0x7fb5d406dd00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5e8072800 0x7fb5e81dc850 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5e8149790 0x7fb5e81dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 --2- 172.21.15.78:0/12436949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5e80723d0 0x7fb5e8145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 >> 172.21.15.78:0/12436949 conn(0x7fb5e80718e0 msgr2=0x7fb5e80c5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 shutdown_connections 2024-02-19T14:52:46.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.275+0000 7fb5ed8d4640 1 -- 172.21.15.78:0/12436949 wait complete. 2024-02-19T14:52:46.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.388+0000 7ff9833bc640 1 Processor -- start 2024-02-19T14:52:46.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.388+0000 7ff9833bc640 1 -- start start 2024-02-19T14:52:46.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.389+0000 7ff9833bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff97c0c8e00 0x7ff97c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.389+0000 7ff9833bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97c1551f0 con 0x7ff97c06d910 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.389+0000 7ff9833bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97c155490 con 0x7ff97c152700 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.389+0000 7ff9833bc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97c155770 con 0x7ff97c0c8e00 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.389+0000 7ff981959640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff97c06d910 0x7ff97c15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52742/0 (socket says 172.21.15.78:52742) 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.389+0000 7ff981959640 1 -- 172.21.15.78:0/3043145809 learned_addr learned my addr 172.21.15.78:0/3043145809 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff980957640 1 --2- 172.21.15.78:0/3043145809 >> v2:172.21.15.150:3300/0 conn(0x7ff97c0c8e00 0x7ff97c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856992928 0 0) 0x7ff97c1551f0 con 0x7ff97c06d910 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff968003520 con 0x7ff97c06d910 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826607867 0 0) 0x7ff968003520 con 0x7ff97c06d910 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 >> v2:172.21.15.150:3300/0 conn(0x7ff97c0c8e00 msgr2=0x7ff97c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 --2- 172.21.15.78:0/3043145809 >> v2:172.21.15.150:3300/0 conn(0x7ff97c0c8e00 0x7ff97c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 >> v1:172.21.15.150:6789/0 conn(0x7ff97c152700 legacy=0x7ff97c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c155a50 con 0x7ff97c06d910 2024-02-19T14:52:46.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7ff9740019f0 con 0x7ff97c06d910 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 >> v1:172.21.15.78:6789/0 conn(0x7ff97c06d910 legacy=0x7ff97c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff968003970 0x7ff968003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9680047c0 0x7ff968004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff968005180 0x7ff96800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c155770 con 0x7ff968005180 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96800c020 con 0x7ff968003970 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96800c1e0 con 0x7ff9680047c0 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981959640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff968005180 0x7ff96800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981158640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff968003970 0x7ff968003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff980957640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9680047c0 0x7ff968004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981959640 1 -- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9680047c0 msgr2=0x7ff968004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981959640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9680047c0 0x7ff968004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981959640 1 -- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff968003970 msgr2=0x7ff968003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:46.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981959640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff968003970 0x7ff968003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff981959640 1 -- 172.21.15.78:0/3043145809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff968003520 con 0x7ff968005180 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.390+0000 7ff980957640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9680047c0 0x7ff968004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.391+0000 7ff981959640 1 -- 172.21.15.78:0/3043145809 reap_dead start 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.391+0000 7ff981959640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff968005180 0x7ff96800b980 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff97400a820 tx=0x7ff9680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aabc923dd97567b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.391+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9740072d0 con 0x7ff968005180 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.391+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97402f070 con 0x7ff968005180 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.391+0000 7ff973fff640 1 -- 172.21.15.78:0/3043145809 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974007690 con 0x7ff968005180 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 -- 172.21.15.78:0/3043145809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff968005180 msgr2=0x7ff96800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff968005180 0x7ff96800b980 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff97400a820 tx=0x7ff9680057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 -- 172.21.15.78:0/3043145809 shutdown_connections 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 --2- 172.21.15.78:0/3043145809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff968005180 0x7ff96800b980 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 -- 172.21.15.78:0/3043145809 >> 172.21.15.78:0/3043145809 conn(0x7ff97c0cac90 msgr2=0x7ff97c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 -- 172.21.15.78:0/3043145809 shutdown_connections 2024-02-19T14:52:46.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.392+0000 7ff9833bc640 1 -- 172.21.15.78:0/3043145809 wait complete. 2024-02-19T14:52:46.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.393+0000 7ff9833bc640 1 Processor -- start 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.393+0000 7ff9833bc640 1 -- start start 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff968005180 0x7ff97c0c4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff97c0acfa0 0x7ff97c1ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 0x7ff97c1ed260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c1eb950 con 0x7ff97c1eda50 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c0c25b0 con 0x7ff97c0acfa0 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c0c19e0 con 0x7ff968005180 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 0x7ff97c1ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 0x7ff97c1ed260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41432/0 (socket says 172.21.15.78:41432) 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 -- 172.21.15.78:0/1970177811 learned_addr learned my addr 172.21.15.78:0/1970177811 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff980957640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff97c0acfa0 0x7ff97c1ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981158640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff968005180 0x7ff97c0c4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 -- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff968005180 msgr2=0x7ff97c0c4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff968005180 0x7ff97c0c4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 -- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff97c0acfa0 msgr2=0x7ff97c1ecd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff97c0acfa0 0x7ff97c1ecd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 -- 172.21.15.78:0/1970177811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c0c1c80 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff980957640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff97c0acfa0 0x7ff97c1ecd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff981959640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 0x7ff97c1ed260 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff974000f80 tx=0x7ff97c06d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974004d60 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9740070d0 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c0c1f60 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.394+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974007440 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.395+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c1ed7a0 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.396+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97c074990 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.396+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff974031030 con 0x7ff97c1eda50 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.396+0000 7ff971ffb640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff96006b810 0x7ff96006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.397+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7ff9740ac1a0 con 0x7ff97c1eda50 2024-02-19T14:52:46.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.397+0000 7ff981158640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff96006b810 0x7ff96006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.397+0000 7ff981158640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff96006b810 0x7ff96006dcd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff978005510 tx=0x7ff978009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.401+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff974076910 con 0x7ff97c1eda50 2024-02-19T14:52:46.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.612+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff97c0c26f0 con 0x7ff97c1eda50 2024-02-19T14:52:46.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.613+0000 7ff971ffb640 1 -- 172.21.15.78:0/1970177811 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19031 (secure 0 0 0) 0x7ff974079fd0 con 0x7ff97c1eda50 2024-02-19T14:52:46.616 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:46.616 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:41.184968+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":0}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff96006b810 msgr2=0x7ff96006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff96006b810 0x7ff96006dcd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff978005510 tx=0x7ff978009040 comp rx=0 tx=0).stop 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 msgr2=0x7ff97c1ed260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 0x7ff97c1ed260 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff974000f80 tx=0x7ff97c06d6b0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 shutdown_connections 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff96006b810 0x7ff96006dcd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c1eda50 0x7ff97c1ed260 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff97c0acfa0 0x7ff97c1ecd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 --2- 172.21.15.78:0/1970177811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff968005180 0x7ff97c0c4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 >> 172.21.15.78:0/1970177811 conn(0x7ff97c0cac90 msgr2=0x7ff97c1509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 shutdown_connections 2024-02-19T14:52:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.615+0000 7ff9833bc640 1 -- 172.21.15.78:0/1970177811 wait complete. 2024-02-19T14:52:46.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:51:58.054888+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:51:58.551083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-19T14:52:02.276458+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T14:52:46.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:46.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.729+0000 7f2c6cd2e640 1 Processor -- start 2024-02-19T14:52:46.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.729+0000 7f2c6cd2e640 1 -- start start 2024-02-19T14:52:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c6cd2e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c6814cd50 0x7f2c680c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c6cd2e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c68156860 0x7f2c68156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c6cd2e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c681563b0 con 0x7f2c681537a0 2024-02-19T14:52:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c6cd2e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c68153cd0 con 0x7f2c68156860 2024-02-19T14:52:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c6cd2e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6806b6e0 con 0x7f2c6814cd50 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c5dd9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2c681537a0 0x7f2c680c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52750/0 (socket says 172.21.15.78:52750) 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.730+0000 7f2c5dd9b640 1 -- 172.21.15.78:0/3927757531 learned_addr learned my addr 172.21.15.78:0/3927757531 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c66d9d640 1 --2- 172.21.15.78:0/3927757531 >> v2:172.21.15.150:3300/0 conn(0x7f2c68156860 0x7f2c68156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c6659c640 1 --2- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c6814cd50 0x7f2c680c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c65d9b640 1 -- 172.21.15.78:0/3927757531 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925619990 0 0) 0x7f2c681563b0 con 0x7f2c681537a0 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c65d9b640 1 -- 172.21.15.78:0/3927757531 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c54003520 con 0x7f2c681537a0 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c66d9d640 1 -- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c6814cd50 msgr2=0x7f2c680c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c66d9d640 1 --2- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c6814cd50 0x7f2c680c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c66d9d640 1 -- 172.21.15.78:0/3927757531 >> v1:172.21.15.78:6789/0 conn(0x7f2c681537a0 legacy=0x7f2c680c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c65d9b640 1 -- 172.21.15.78:0/3927757531 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338730228 0 0) 0x7f2c54003520 con 0x7f2c681537a0 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.731+0000 7f2c66d9d640 1 -- 172.21.15.78:0/3927757531 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c68157230 con 0x7f2c68156860 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c66d9d640 1 --2- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68156860 0x7f2c68156cf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009d90 tx=0x7f2c6002bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f8a9b7a8ba14490 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c65d9b640 1 -- 172.21.15.78:0/3927757531 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50001670 con 0x7f2c68156860 2024-02-19T14:52:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c65d9b640 1 -- 172.21.15.78:0/3927757531 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6002c780 con 0x7f2c68156860 2024-02-19T14:52:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c65d9b640 1 -- 172.21.15.78:0/3927757531 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6002caf0 con 0x7f2c68156860 2024-02-19T14:52:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68156860 msgr2=0x7f2c68156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68156860 0x7f2c68156cf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009d90 tx=0x7f2c6002bc50 comp rx=0 tx=0).stop 2024-02-19T14:52:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/3927757531 shutdown_connections 2024-02-19T14:52:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68156860 0x7f2c68156cf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/3927757531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c6814cd50 0x7f2c680c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/3927757531 >> 172.21.15.78:0/3927757531 conn(0x7f2c68066e70 msgr2=0x7f2c680a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/3927757531 shutdown_connections 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.732+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/3927757531 wait complete. 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.734+0000 7f2c6cd2e640 1 Processor -- start 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.734+0000 7f2c6cd2e640 1 -- start start 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.734+0000 7f2c6cd2e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6814cd50 0x7f2c681500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 0x7f2c680c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c68156860 0x7f2c680c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6814f620 con 0x7f2c681537a0 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c680c5d10 con 0x7f2c6814cd50 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6814fab0 con 0x7f2c68156860 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 0x7f2c680c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 0x7f2c680c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41448/0 (socket says 172.21.15.78:41448) 2024-02-19T14:52:46.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 -- 172.21.15.78:0/962947233 learned_addr learned my addr 172.21.15.78:0/962947233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6659c640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6814cd50 0x7f2c681500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c66d9d640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c68156860 0x7f2c680c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 -- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c68156860 msgr2=0x7f2c680c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c68156860 0x7f2c680c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 -- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6814cd50 msgr2=0x7f2c681500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6814cd50 0x7f2c681500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 -- 172.21.15.78:0/962947233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60009a70 con 0x7f2c681537a0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5dd9b640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 0x7f2c680c5290 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c750 tx=0x7f2c5000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50005ad0 con 0x7f2c681537a0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c50005c30 con 0x7f2c681537a0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50007070 con 0x7f2c681537a0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6814e730 con 0x7f2c681537a0 2024-02-19T14:52:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.735+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6814ecf0 con 0x7f2c681537a0 2024-02-19T14:52:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.737+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6814f7d0 con 0x7f2c681537a0 2024-02-19T14:52:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.737+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2c500071d0 con 0x7f2c681537a0 2024-02-19T14:52:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.737+0000 7f2c5f7fe640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c4806b810 0x7f2c4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.737+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f2c5008a270 con 0x7f2c681537a0 2024-02-19T14:52:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.737+0000 7f2c6659c640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c4806b810 0x7f2c4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.738+0000 7f2c6659c640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c4806b810 0x7f2c4806dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c68152110 tx=0x7f2c58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:46.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.742+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c6814f7d0 con 0x7f2c681537a0 2024-02-19T14:52:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.920+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c6814f7d0 con 0x7f2c681537a0 2024-02-19T14:52:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.920+0000 7f2c5f7fe640 1 -- 172.21.15.78:0/962947233 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c6814f7d0 con 0x7f2c681537a0 2024-02-19T14:52:46.923 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c4806b810 msgr2=0x7f2c4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c4806b810 0x7f2c4806dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c68152110 tx=0x7f2c58008040 comp rx=0 tx=0).stop 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 msgr2=0x7f2c680c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 0x7f2c680c5290 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c750 tx=0x7f2c5000cc10 comp rx=0 tx=0).stop 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 shutdown_connections 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c4806b810 0x7f2c4806dcd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c681537a0 0x7f2c680c5290 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c68156860 0x7f2c680c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 --2- 172.21.15.78:0/962947233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6814cd50 0x7f2c681500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:46.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 >> 172.21.15.78:0/962947233 conn(0x7f2c68066e70 msgr2=0x7f2c680c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:46.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 shutdown_connections 2024-02-19T14:52:46.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:46.922+0000 7f2c6cd2e640 1 -- 172.21.15.78:0/962947233 wait complete. 2024-02-19T14:52:46.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:52:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.036+0000 7fb8a5325640 1 Processor -- start 2024-02-19T14:52:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb8a5325640 1 -- start start 2024-02-19T14:52:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb8a5325640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8a014ef50 0x7fb8a014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb8a5325640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb8a0072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb8a5325640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a00c35c0 con 0x7fb8a00c78a0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb8a5325640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a00c4bc0 con 0x7fb8a014ef50 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb8a5325640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a014f920 con 0x7fb8a00727d0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb89effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb8a00c78a0 0x7fb8a00c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52764/0 (socket says 172.21.15.78:52764) 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.037+0000 7fb89effd640 1 -- 172.21.15.78:0/2407669544 learned_addr learned my addr 172.21.15.78:0/2407669544 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694921539 0 0) 0x7fb8a00c35c0 con 0x7fb8a00c78a0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb897fff640 1 --2- 172.21.15.78:0/2407669544 >> v2:172.21.15.150:3300/0 conn(0x7fb8a014ef50 0x7fb8a014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb8a0072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c003520 con 0x7fb8a00c78a0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466673197 0 0) 0x7fb88c003520 con 0x7fb8a00c78a0 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 msgr2=0x7fb8a0072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb8a0072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 >> v2:172.21.15.150:3300/0 conn(0x7fb8a014ef50 msgr2=0x7fb8a014f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 --2- 172.21.15.78:0/2407669544 >> v2:172.21.15.150:3300/0 conn(0x7fb8a014ef50 0x7fb8a014f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00c38a0 con 0x7fb8a00c78a0 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb890001670 con 0x7fb8a00c78a0 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 >> v1:172.21.15.78:6789/0 conn(0x7fb8a00c78a0 legacy=0x7fb8a00c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb88c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb88c003f20 0x7fb88c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb88c004d00 0x7fb88c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00c4bc0 con 0x7fb88c004d00 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a014f920 con 0x7fb88c003f20 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c00b980 con 0x7fb8a00727d0 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb88c004d00 0x7fb88c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89effd640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb88c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb897fff640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb88c003f20 0x7fb88c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 -- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 msgr2=0x7fb88c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb88c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 -- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb88c003f20 msgr2=0x7fb88c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb88c003f20 0x7fb88c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89f7fe640 1 -- 172.21.15.78:0/2407669544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c003520 con 0x7fb88c004d00 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb89effd640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a00727d0 0x7fb88c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.038+0000 7fb897fff640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb88c003f20 0x7fb88c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.039+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb88c004d00 0x7fb88c005180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb898009fc0 tx=0x7fb88c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35d309888d1e159 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.039+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89802cc20 con 0x7fb88c004d00 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.039+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89800f040 con 0x7fb88c004d00 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.039+0000 7fb89e7fc640 1 -- 172.21.15.78:0/2407669544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898034440 con 0x7fb88c004d00 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb8a5325640 1 -- 172.21.15.78:0/2407669544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb88c004d00 msgr2=0x7fb88c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb8a5325640 1 --2- 172.21.15.78:0/2407669544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb88c004d00 0x7fb88c005180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb898009fc0 tx=0x7fb88c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb89f7fe640 1 -- 172.21.15.78:0/2407669544 reap_dead start 2024-02-19T14:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb8a5325640 1 -- 172.21.15.78:0/2407669544 shutdown_connections 2024-02-19T14:52:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb8a5325640 1 -- 172.21.15.78:0/2407669544 >> 172.21.15.78:0/2407669544 conn(0x7fb8a0066e70 msgr2=0x7fb8a00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb8a5325640 1 -- 172.21.15.78:0/2407669544 shutdown_connections 2024-02-19T14:52:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.040+0000 7fb8a5325640 1 -- 172.21.15.78:0/2407669544 wait complete. 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.041+0000 7fb8a5325640 1 Processor -- start 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.041+0000 7fb8a5325640 1 -- start start 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8a5325640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a0150360 0x7fb8a01eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8a5325640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a0121460 0x7fb8a0146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8a5325640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 0x7fb8a00aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8a5325640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00aa7f0 con 0x7fb8a0120820 2024-02-19T14:52:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8a5325640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00ac2b0 con 0x7fb8a0150360 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8a5325640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a00aaa60 con 0x7fb8a0121460 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 0x7fb8a00aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 0x7fb8a00aaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41478/0 (socket says 172.21.15.78:41478) 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 -- 172.21.15.78:0/3244563133 learned_addr learned my addr 172.21.15.78:0/3244563133 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89effd640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a0150360 0x7fb8a01eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb897fff640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a0121460 0x7fb8a0146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 -- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a0121460 msgr2=0x7fb8a0146e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a0121460 0x7fb8a0146e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 -- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a0150360 msgr2=0x7fb8a01eaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a0150360 0x7fb8a01eaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 -- 172.21.15.78:0/3244563133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00ac590 con 0x7fb8a0120820 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb897fff640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a0121460 0x7fb8a0146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb89f7fe640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 0x7fb8a00aaf80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb898037040 tx=0x7fb898009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.042+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898043070 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.043+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89802de90 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.043+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898034d90 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.043+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a014b7e0 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.043+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a0150920 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.044+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a0074990 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.044+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb89803e440 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.044+0000 7fb8977fe640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb88006b810 0x7fb88006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.045+0000 7fb89effd640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb88006b810 0x7fb88006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.045+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fb8980ac180 con 0x7fb8a0120820 2024-02-19T14:52:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.045+0000 7fb89effd640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb88006b810 0x7fb88006dcd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb8900017a0 tx=0x7fb890007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.049+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb898076880 con 0x7fb8a0120820 2024-02-19T14:52:47.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.239+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8a014db60 con 0x7fb8a0120820 2024-02-19T14:52:47.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.240+0000 7fb8977fe640 1 -- 172.21.15.78:0/3244563133 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb898079f40 con 0x7fb8a0120820 2024-02-19T14:52:47.243 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb88006b810 msgr2=0x7fb88006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb88006b810 0x7fb88006dcd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb8900017a0 tx=0x7fb890007040 comp rx=0 tx=0).stop 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 msgr2=0x7fb8a00aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 0x7fb8a00aaf80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb898037040 tx=0x7fb898009750 comp rx=0 tx=0).stop 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 shutdown_connections 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb88006b810 0x7fb88006dcd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8a0120820 0x7fb8a00aaf80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a0121460 0x7fb8a0146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 --2- 172.21.15.78:0/3244563133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a0150360 0x7fb8a01eaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 >> 172.21.15.78:0/3244563133 conn(0x7fb8a0066e70 msgr2=0x7fb8a0072bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:47.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 shutdown_connections 2024-02-19T14:52:47.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.242+0000 7fb8a5325640 1 -- 172.21.15.78:0/3244563133 wait complete. 2024-02-19T14:52:47.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:52:47.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.359+0000 7feae4c5c640 1 Processor -- start 2024-02-19T14:52:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.359+0000 7feae4c5c640 1 -- start start 2024-02-19T14:52:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feae4c5c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feae00c7b70 0x7feae00c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feae4c5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae0155bb0 0x7feae01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feae4c5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feae01589f0 0x7feae006d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feae4c5c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae0154ec0 con 0x7feae01589f0 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feae4c5c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae006db90 con 0x7feae00c7b70 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feae4c5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae01550d0 con 0x7feae0155bb0 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feae01589f0 0x7feae006d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feae01589f0 0x7feae006d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41490/0 (socket says 172.21.15.78:41490) 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 -- 172.21.15.78:0/3104966867 learned_addr learned my addr 172.21.15.78:0/3104966867 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feade59c640 1 --2- 172.21.15.78:0/3104966867 >> v2:172.21.15.150:3300/0 conn(0x7feae00c7b70 0x7feae00c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 -- 172.21.15.78:0/3104966867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae0155bb0 msgr2=0x7feae01584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7fead7fff640 1 --2- 172.21.15.78:0/3104966867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae0155bb0 0x7feae01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 --2- 172.21.15.78:0/3104966867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae0155bb0 0x7feae01584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 -- 172.21.15.78:0/3104966867 >> v2:172.21.15.150:3300/0 conn(0x7feae00c7b70 msgr2=0x7feae00c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 --2- 172.21.15.78:0/3104966867 >> v2:172.21.15.150:3300/0 conn(0x7feae00c7b70 0x7feae00c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feaded9d640 1 -- 172.21.15.78:0/3104966867 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae01553b0 con 0x7feae01589f0 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.360+0000 7feade59c640 1 --2- 172.21.15.78:0/3104966867 >> v2:172.21.15.150:3300/0 conn(0x7feae00c7b70 0x7feae00c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.361+0000 7feaded9d640 1 --2- 172.21.15.78:0/3104966867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae006d040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7feae006efc0 tx=0x7fead802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd00862a26d1faf8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.361+0000 7feaddd9b640 1 -- 172.21.15.78:0/3104966867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead802c9e0 con 0x7feae01589f0 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.361+0000 7feaddd9b640 1 -- 172.21.15.78:0/3104966867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead802cb40 con 0x7feae01589f0 2024-02-19T14:52:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.361+0000 7feaddd9b640 1 -- 172.21.15.78:0/3104966867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80343f0 con 0x7feae01589f0 2024-02-19T14:52:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 -- 172.21.15.78:0/3104966867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 msgr2=0x7feae006d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3104966867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae006d040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7feae006efc0 tx=0x7fead802be40 comp rx=0 tx=0).stop 2024-02-19T14:52:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 -- 172.21.15.78:0/3104966867 shutdown_connections 2024-02-19T14:52:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3104966867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae006d040 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3104966867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae0155bb0 0x7feae01584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3104966867 >> v2:172.21.15.150:3300/0 conn(0x7feae00c7b70 0x7feae00c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 -- 172.21.15.78:0/3104966867 >> 172.21.15.78:0/3104966867 conn(0x7feae00c5050 msgr2=0x7feae00c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 -- 172.21.15.78:0/3104966867 shutdown_connections 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.362+0000 7feae4c5c640 1 -- 172.21.15.78:0/3104966867 wait complete. 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.363+0000 7feae4c5c640 1 Processor -- start 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.363+0000 7feae4c5c640 1 -- start start 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feae4c5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae00c7b70 0x7feae0150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feae4c5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae0155bb0 0x7feae0150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feae4c5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae01416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feae4c5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00ab760 con 0x7feae01589f0 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feae4c5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00ab3c0 con 0x7feae0155bb0 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feae4c5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae00ad570 con 0x7feae00c7b70 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae01416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae01416e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41498/0 (socket says 172.21.15.78:41498) 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 -- 172.21.15.78:0/3458217585 learned_addr learned my addr 172.21.15.78:0/3458217585 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feade59c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae00c7b70 0x7feae0150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7fead7fff640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae0155bb0 0x7feae0150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 -- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae00c7b70 msgr2=0x7feae0150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae00c7b70 0x7feae0150f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 -- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae0155bb0 msgr2=0x7feae0150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae0155bb0 0x7feae0150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 -- 172.21.15.78:0/3458217585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8009b30 con 0x7feae01589f0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7fead7fff640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae0155bb0 0x7feae0150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7feaded9d640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae01416e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fead8036040 tx=0x7fead8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.364+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8009510 con 0x7feae01589f0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.365+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead803e450 con 0x7feae01589f0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.365+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae00ad850 con 0x7feae01589f0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.365+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead803e800 con 0x7feae01589f0 2024-02-19T14:52:47.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.365+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae0151470 con 0x7feae01589f0 2024-02-19T14:52:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.365+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae00ab8a0 con 0x7feae01589f0 2024-02-19T14:52:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.366+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fead803e960 con 0x7feae01589f0 2024-02-19T14:52:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.366+0000 7fead6ffd640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feac006b810 0x7feac006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.366+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fead80ad4c0 con 0x7feae01589f0 2024-02-19T14:52:47.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.366+0000 7feade59c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feac006b810 0x7feac006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.367+0000 7feade59c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feac006b810 0x7feac006dcd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fead000b840 tx=0x7fead0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.371+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feae00ab8a0 con 0x7feae01589f0 2024-02-19T14:52:47.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.548+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feae00ab8a0 con 0x7feae01589f0 2024-02-19T14:52:47.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.548+0000 7fead6ffd640 1 -- 172.21.15.78:0/3458217585 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7feae00ab8a0 con 0x7feae01589f0 2024-02-19T14:52:47.551 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feac006b810 msgr2=0x7feac006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feac006b810 0x7feac006dcd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fead000b840 tx=0x7fead0008040 comp rx=0 tx=0).stop 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 msgr2=0x7feae01416e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae01416e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fead8036040 tx=0x7fead8004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 shutdown_connections 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feac006b810 0x7feac006dcd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae01589f0 0x7feae01416e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae0155bb0 0x7feae0150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 --2- 172.21.15.78:0/3458217585 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae00c7b70 0x7feae0150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 >> 172.21.15.78:0/3458217585 conn(0x7feae00c5050 msgr2=0x7feae006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 shutdown_connections 2024-02-19T14:52:47.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.550+0000 7feae4c5c640 1 -- 172.21.15.78:0/3458217585 wait complete. 2024-02-19T14:52:47.563 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-19T14:52:47.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.664+0000 7fe7defd0640 1 Processor -- start 2024-02-19T14:52:47.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.664+0000 7fe7defd0640 1 -- start start 2024-02-19T14:52:47.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.665+0000 7fe7defd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.665+0000 7fe7defd0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806ff90 con 0x7fe7d80719e0 2024-02-19T14:52:47.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.665+0000 7fe7defd0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8156fc0 con 0x7fe7d814d210 2024-02-19T14:52:47.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.665+0000 7fe7defd0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d8070270 con 0x7fe7d80c3d90 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.665+0000 7fe7cffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe7d80719e0 0x7fe7d80c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55896/0 (socket says 172.21.15.78:55896) 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.665+0000 7fe7cffff640 1 -- 172.21.15.78:0/1187850913 learned_addr learned my addr 172.21.15.78:0/1187850913 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.666+0000 7fe7dd56d640 1 --2- 172.21.15.78:0/1187850913 >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.666+0000 7fe7cf7fe640 1 -- 172.21.15.78:0/1187850913 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957494373 0 0) 0x7fe7d806ff90 con 0x7fe7d80719e0 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.666+0000 7fe7cf7fe640 1 -- 172.21.15.78:0/1187850913 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c0003520 con 0x7fe7d80719e0 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.666+0000 7fe7dd56d640 1 -- 172.21.15.78:0/1187850913 >> v1:172.21.15.150:6789/0 conn(0x7fe7d814d210 legacy=0x7fe7d80c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.666+0000 7fe7dd56d640 1 -- 172.21.15.78:0/1187850913 >> v1:172.21.15.78:6789/0 conn(0x7fe7d80719e0 legacy=0x7fe7d80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.666+0000 7fe7dd56d640 1 -- 172.21.15.78:0/1187850913 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d81572a0 con 0x7fe7d80c3d90 2024-02-19T14:52:47.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.667+0000 7fe7dd56d640 1 --2- 172.21.15.78:0/1187850913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d815e800 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009d90 tx=0x7fe7d002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c52bade65a07d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.667+0000 7fe7cf7fe640 1 -- 172.21.15.78:0/1187850913 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d002cc60 con 0x7fe7d80c3d90 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.667+0000 7fe7cf7fe640 1 -- 172.21.15.78:0/1187850913 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d002cdc0 con 0x7fe7d80c3d90 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.667+0000 7fe7cf7fe640 1 -- 172.21.15.78:0/1187850913 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0034640 con 0x7fe7d80c3d90 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 -- 172.21.15.78:0/1187850913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 msgr2=0x7fe7d815e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 --2- 172.21.15.78:0/1187850913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d815e800 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009d90 tx=0x7fe7d002beb0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 -- 172.21.15.78:0/1187850913 shutdown_connections 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 --2- 172.21.15.78:0/1187850913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 -- 172.21.15.78:0/1187850913 >> 172.21.15.78:0/1187850913 conn(0x7fe7d80c5050 msgr2=0x7fe7d80c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 -- 172.21.15.78:0/1187850913 shutdown_connections 2024-02-19T14:52:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.668+0000 7fe7defd0640 1 -- 172.21.15.78:0/1187850913 wait complete. 2024-02-19T14:52:47.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.669+0000 7fe7defd0640 1 Processor -- start 2024-02-19T14:52:47.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.669+0000 7fe7defd0640 1 -- start start 2024-02-19T14:52:47.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7defd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80719e0 0x7fe7d8149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7defd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d80be330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7defd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d81dc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7defd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80c3970 con 0x7fe7d80c3d90 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7defd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80bd890 con 0x7fe7d80719e0 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7defd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d814a2d0 con 0x7fe7d814d210 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d80be330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d80be330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37050/0 (socket says 172.21.15.78:37050) 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 -- 172.21.15.78:0/3275708943 learned_addr learned my addr 172.21.15.78:0/3275708943 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7dcd6c640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80719e0 0x7fe7d8149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7dd56d640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d81dc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 -- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 msgr2=0x7fe7d81dc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d81dc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 -- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80719e0 msgr2=0x7fe7d8149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80719e0 0x7fe7d8149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 -- 172.21.15.78:0/3275708943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d0009a70 con 0x7fe7d80c3d90 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.671+0000 7fe7cffff640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d80be330 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800c4d0 tx=0x7fe7c800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.672+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8005ce0 con 0x7fe7d80c3d90 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.672+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c8005e40 con 0x7fe7d80c3d90 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.672+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d80bddc0 con 0x7fe7d80c3d90 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.672+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8007270 con 0x7fe7d80c3d90 2024-02-19T14:52:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.672+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d81df4d0 con 0x7fe7d80c3d90 2024-02-19T14:52:47.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.673+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d81dd000 con 0x7fe7d80c3d90 2024-02-19T14:52:47.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.673+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe7c80073d0 con 0x7fe7d80c3d90 2024-02-19T14:52:47.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.673+0000 7fe7cd7fa640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7c406b810 0x7fe7c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:47.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.674+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fe7c8089b10 con 0x7fe7d80c3d90 2024-02-19T14:52:47.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.674+0000 7fe7dcd6c640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7c406b810 0x7fe7c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:47.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.674+0000 7fe7dcd6c640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7c406b810 0x7fe7c406dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40016e0 tx=0x7fe7d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.678+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe7c8054170 con 0x7fe7d80c3d90 2024-02-19T14:52:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.987+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe7d8074990 con 0x7fe7d80c3d90 2024-02-19T14:52:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.987+0000 7fe7cd7fa640 1 -- 172.21.15.78:0/3275708943 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe7c8017020 con 0x7fe7d80c3d90 2024-02-19T14:52:47.990 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7c406b810 msgr2=0x7fe7c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7c406b810 0x7fe7c406dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40016e0 tx=0x7fe7d4009040 comp rx=0 tx=0).stop 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 msgr2=0x7fe7d80be330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d80be330 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800c4d0 tx=0x7fe7c800c990 comp rx=0 tx=0).stop 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 shutdown_connections 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7c406b810 0x7fe7c406dcd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d80be330 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d81dc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 --2- 172.21.15.78:0/3275708943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80719e0 0x7fe7d8149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 >> 172.21.15.78:0/3275708943 conn(0x7fe7d80c5050 msgr2=0x7fe7d806f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 shutdown_connections 2024-02-19T14:52:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:47.989+0000 7fe7defd0640 1 -- 172.21.15.78:0/3275708943 wait complete. 2024-02-19T14:52:48.003 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.102+0000 7f11fd9ac640 1 Processor -- start 2024-02-19T14:52:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.102+0000 7f11fd9ac640 1 -- start start 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11fd9ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f11f8155bb0 0x7f11f80c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11fd9ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f806db90 con 0x7f11f80c7b70 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11fd9ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f81550d0 con 0x7f11f806d040 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11fd9ac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f8154ec0 con 0x7f11f8155bb0 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11f6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f11f80c7b70 0x7f11f81584b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55902/0 (socket says 172.21.15.78:55902) 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11f6ffd640 1 -- 172.21.15.78:0/1379025986 learned_addr learned my addr 172.21.15.78:0/1379025986 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11effff640 1 --2- 172.21.15.78:0/1379025986 >> v2:172.21.15.150:3300/0 conn(0x7f11f8155bb0 0x7f11f80c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11f67fc640 1 -- 172.21.15.78:0/1379025986 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956010233 0 0) 0x7f11f806db90 con 0x7f11f80c7b70 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11f67fc640 1 -- 172.21.15.78:0/1379025986 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e4003520 con 0x7f11f80c7b70 2024-02-19T14:52:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11effff640 1 -- 172.21.15.78:0/1379025986 >> v1:172.21.15.150:6789/0 conn(0x7f11f806d040 legacy=0x7f11f815f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11f67fc640 1 -- 172.21.15.78:0/1379025986 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159320369 0 0) 0x7f11f81550d0 con 0x7f11f806d040 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11effff640 1 -- 172.21.15.78:0/1379025986 >> v1:172.21.15.78:6789/0 conn(0x7f11f80c7b70 legacy=0x7f11f81584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.103+0000 7f11effff640 1 -- 172.21.15.78:0/1379025986 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f81553b0 con 0x7f11f8155bb0 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.104+0000 7f11effff640 1 --2- 172.21.15.78:0/1379025986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 0x7f11f80c7f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f11e00097a0 tx=0x7f11e002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e76a12802c249882 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.104+0000 7f11f67fc640 1 -- 172.21.15.78:0/1379025986 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0031830 con 0x7f11f8155bb0 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.104+0000 7f11f67fc640 1 -- 172.21.15.78:0/1379025986 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0031990 con 0x7f11f8155bb0 2024-02-19T14:52:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.104+0000 7f11f67fc640 1 -- 172.21.15.78:0/1379025986 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0031d20 con 0x7f11f8155bb0 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1379025986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 msgr2=0x7f11f80c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1379025986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 0x7f11f80c7f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f11e00097a0 tx=0x7f11e002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1379025986 shutdown_connections 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1379025986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 0x7f11f80c7f50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1379025986 >> 172.21.15.78:0/1379025986 conn(0x7f11f80c5050 msgr2=0x7f11f80c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1379025986 shutdown_connections 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.105+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1379025986 wait complete. 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.106+0000 7f11fd9ac640 1 Processor -- start 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.106+0000 7f11fd9ac640 1 -- start start 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 0x7f11f81eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11f80c7b70 0x7f11f809a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 0x7f11f80ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f81eb130 con 0x7f11f806d040 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f814c560 con 0x7f11f8155bb0 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f8099a60 con 0x7f11f80c7b70 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 0x7f11f81eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 0x7f11f81eabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37052/0 (socket says 172.21.15.78:37052) 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 -- 172.21.15.78:0/1750796772 learned_addr learned my addr 172.21.15.78:0/1750796772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 -- 172.21.15.78:0/1750796772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11f80c7b70 msgr2=0x7f11f809a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11f80c7b70 0x7f11f809a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 -- 172.21.15.78:0/1750796772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 msgr2=0x7f11f80ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 0x7f11f80ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 -- 172.21.15.78:0/1750796772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0009400 con 0x7f11f806d040 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f6ffd640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 0x7f11f81eabf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f11e800c430 tx=0x7f11e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8005ce0 con 0x7f11f806d040 2024-02-19T14:52:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e8005e40 con 0x7f11f806d040 2024-02-19T14:52:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f8099d00 con 0x7f11f806d040 2024-02-19T14:52:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.107+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e80072f0 con 0x7f11f806d040 2024-02-19T14:52:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.108+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f8152de0 con 0x7f11f806d040 2024-02-19T14:52:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.109+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f8152c40 con 0x7f11f806d040 2024-02-19T14:52:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.109+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f11e8007450 con 0x7f11f806d040 2024-02-19T14:52:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.109+0000 7f11f4ff9640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f11d806b810 0x7f11d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.109+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f11e8089d20 con 0x7f11f806d040 2024-02-19T14:52:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.109+0000 7f11effff640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f11d806b810 0x7f11d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.110+0000 7f11effff640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f11d806b810 0x7f11d806dcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11e0036000 tx=0x7f11e0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.114+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f11f8152c40 con 0x7f11f806d040 2024-02-19T14:52:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.329+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11f8152c40 con 0x7f11f806d040 2024-02-19T14:52:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.330+0000 7f11f4ff9640 1 -- 172.21.15.78:0/1750796772 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7f11f8152c40 con 0x7f11f806d040 2024-02-19T14:52:48.333 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:48.333 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f11d806b810 msgr2=0x7f11d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f11d806b810 0x7f11d806dcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11e0036000 tx=0x7f11e0030000 comp rx=0 tx=0).stop 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 msgr2=0x7f11f81eabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 0x7f11f81eabf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f11e800c430 tx=0x7f11e800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 shutdown_connections 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f11d806b810 0x7f11d806dcd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11f8155bb0 0x7f11f80ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11f80c7b70 0x7f11f809a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 --2- 172.21.15.78:0/1750796772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11f806d040 0x7f11f81eabf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 >> 172.21.15.78:0/1750796772 conn(0x7f11f80c5050 msgr2=0x7f11f80c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 shutdown_connections 2024-02-19T14:52:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.332+0000 7f11fd9ac640 1 -- 172.21.15.78:0/1750796772 wait complete. 2024-02-19T14:52:48.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:48.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.444+0000 7f9b70e76640 1 Processor -- start 2024-02-19T14:52:48.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.444+0000 7f9b70e76640 1 -- start start 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b70e76640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b6c0c3d90 0x7f9b6c15e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b70e76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6c06ff90 con 0x7f9b6c14d210 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b70e76640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6c156fc0 con 0x7f9b6c0719e0 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b70e76640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b6c070270 con 0x7f9b6c0c3d90 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b6a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9b6c14d210 0x7f9b6c0c4290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55908/0 (socket says 172.21.15.78:55908) 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b6a59c640 1 -- 172.21.15.78:0/3185160244 learned_addr learned my addr 172.21.15.78:0/3185160244 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b6ad9d640 1 --2- 172.21.15.78:0/3185160244 >> v2:172.21.15.150:3300/0 conn(0x7f9b6c0c3d90 0x7f9b6c15e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704149293 0 0) 0x7f9b6c06ff90 con 0x7f9b6c14d210 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b58003520 con 0x7f9b6c14d210 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3296327055 0 0) 0x7f9b58003520 con 0x7f9b6c14d210 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 >> v2:172.21.15.150:3300/0 conn(0x7f9b6c0c3d90 msgr2=0x7f9b6c15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 --2- 172.21.15.78:0/3185160244 >> v2:172.21.15.150:3300/0 conn(0x7f9b6c0c3d90 0x7f9b6c15e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 >> v1:172.21.15.150:6789/0 conn(0x7f9b6c0719e0 legacy=0x7f9b6c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c1572a0 con 0x7f9b6c14d210 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9b5c001670 con 0x7f9b6c14d210 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 >> v1:172.21.15.78:6789/0 conn(0x7f9b6c14d210 legacy=0x7f9b6c0c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b580047c0 0x7f9b58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c070270 con 0x7f9b58003970 2024-02-19T14:52:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5800c120 con 0x7f9b58005180 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.445+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5800c2e0 con 0x7f9b580047c0 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6ad9d640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b61d9b640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b580047c0 0x7f9b58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 -- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b580047c0 msgr2=0x7f9b58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b580047c0 0x7f9b58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 -- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b58005180 msgr2=0x7f9b5800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6ad9d640 1 -- 172.21.15.78:0/3185160244 reap_dead start 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 -- 172.21.15.78:0/3185160244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b58003520 con 0x7f9b58003970 2024-02-19T14:52:48.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.446+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b58003d90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00c190 tx=0x7f9b580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846073ad1f154a56 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c005ad0 con 0x7f9b58003970 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5c005c30 con 0x7f9b58003970 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b69d9b640 1 -- 172.21.15.78:0/3185160244 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c02ca70 con 0x7f9b58003970 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 -- 172.21.15.78:0/3185160244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 msgr2=0x7f9b58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b58003d90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00c190 tx=0x7f9b580057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 -- 172.21.15.78:0/3185160244 shutdown_connections 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 --2- 172.21.15.78:0/3185160244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b58003d90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 -- 172.21.15.78:0/3185160244 >> 172.21.15.78:0/3185160244 conn(0x7f9b6c0c5050 msgr2=0x7f9b6c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 -- 172.21.15.78:0/3185160244 shutdown_connections 2024-02-19T14:52:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.447+0000 7f9b70e76640 1 -- 172.21.15.78:0/3185160244 wait complete. 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.449+0000 7f9b70e76640 1 Processor -- start 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.449+0000 7f9b70e76640 1 -- start start 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b70e76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b6c144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b70e76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b6c143cc0 0x7f9b6c1ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b70e76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c1ec4d0 0x7f9b6c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b70e76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c0c3970 con 0x7f9b58003970 2024-02-19T14:52:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b70e76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c143790 con 0x7f9b6c1ec4d0 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b70e76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c153aa0 con 0x7f9b6c143cc0 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.450+0000 7f9b6a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b6c144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b6c144470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37076/0 (socket says 172.21.15.78:37076) 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 -- 172.21.15.78:0/2455893845 learned_addr learned my addr 172.21.15.78:0/2455893845 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6ad9d640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c1ec4d0 0x7f9b6c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b61d9b640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b6c143cc0 0x7f9b6c1ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 -- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b6c143cc0 msgr2=0x7f9b6c1ebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b6c143cc0 0x7f9b6c1ebf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 -- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c1ec4d0 msgr2=0x7f9b6c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c1ec4d0 0x7f9b6c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 -- 172.21.15.78:0/2455893845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c14df30 con 0x7f9b58003970 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b6a59c640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b6c144470 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c001d00 tx=0x7f9b5c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c03c040 con 0x7f9b58003970 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5c02cc40 con 0x7f9b58003970 2024-02-19T14:52:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c1439e0 con 0x7f9b58003970 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c007070 con 0x7f9b58003970 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.451+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c14e430 con 0x7f9b58003970 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.452+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c1ecc50 con 0x7f9b58003970 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.453+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9b5c032070 con 0x7f9b58003970 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.453+0000 7f9b637fe640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b4c06b810 0x7f9b4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.453+0000 7f9b61d9b640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b4c06b810 0x7f9b4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.453+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f9b5c0ab7c0 con 0x7f9b58003970 2024-02-19T14:52:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.454+0000 7f9b61d9b640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b4c06b810 0x7f9b4c06dcd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c153780 tx=0x7f9b54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.458+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9b5c075e10 con 0x7f9b58003970 2024-02-19T14:52:48.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.633+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b6c074990 con 0x7f9b58003970 2024-02-19T14:52:48.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.634+0000 7f9b637fe640 1 -- 172.21.15.78:0/2455893845 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19031 (secure 0 0 0) 0x7f9b5c0794d0 con 0x7f9b58003970 2024-02-19T14:52:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:48.637 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:41.184968+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":0}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:48.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b4c06b810 msgr2=0x7f9b4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b4c06b810 0x7f9b4c06dcd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c153780 tx=0x7f9b54009040 comp rx=0 tx=0).stop 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 msgr2=0x7f9b6c144470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b6c144470 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c001d00 tx=0x7f9b5c02e070 comp rx=0 tx=0).stop 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 shutdown_connections 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9b4c06b810 0x7f9b4c06dcd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c1ec4d0 0x7f9b6c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b6c143cc0 0x7f9b6c1ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 --2- 172.21.15.78:0/2455893845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b58003970 0x7f9b6c144470 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 >> 172.21.15.78:0/2455893845 conn(0x7f9b6c0c5050 msgr2=0x7f9b6c0c7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 shutdown_connections 2024-02-19T14:52:48.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.636+0000 7f9b70e76640 1 -- 172.21.15.78:0/2455893845 wait complete. 2024-02-19T14:52:48.649 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:52:48.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:48.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.746+0000 7fd41ab65640 1 Processor -- start 2024-02-19T14:52:48.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd41ab65640 1 -- start start 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd41ab65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4140c6560 0x7fd414159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd41ab65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 0x7fd41406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd41ab65640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4140c6a60 con 0x7fd41406d000 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd41ab65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4141554b0 con 0x7fd4140c6560 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd41ab65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd414155790 con 0x7fd414156d60 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd418901640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd41406d000 0x7fd414156650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55920/0 (socket says 172.21.15.78:55920) 2024-02-19T14:52:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.747+0000 7fd418901640 1 -- 172.21.15.78:0/58243167 learned_addr learned my addr 172.21.15.78:0/58243167 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 --2- 172.21.15.78:0/58243167 >> v2:172.21.15.150:3300/0 conn(0x7fd4140c6560 0x7fd414159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670237917 0 0) 0x7fd4140c6a60 con 0x7fd41406d000 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd400003520 con 0x7fd41406d000 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd419102640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 0x7fd41406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3250643615 0 0) 0x7fd400003520 con 0x7fd41406d000 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 msgr2=0x7fd41406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 0x7fd41406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 >> v2:172.21.15.150:3300/0 conn(0x7fd4140c6560 msgr2=0x7fd414159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 --2- 172.21.15.78:0/58243167 >> v2:172.21.15.150:3300/0 conn(0x7fd4140c6560 0x7fd414159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd414155a70 con 0x7fd41406d000 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fd404001670 con 0x7fd41406d000 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 >> v1:172.21.15.78:6789/0 conn(0x7fd41406d000 legacy=0x7fd414156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400003970 0x7fd400003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4000047c0 0x7fd400004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 0x7fd400005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4141554b0 con 0x7fd4000047c0 2024-02-19T14:52:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd414155790 con 0x7fd400003970 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40000b9d0 con 0x7fd414156d60 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4000047c0 0x7fd400004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd418901640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400003970 0x7fd400003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd419102640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 0x7fd400005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 -- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 msgr2=0x7fd400005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd414156d60 0x7fd400005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 -- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400003970 msgr2=0x7fd400003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400003970 0x7fd400003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.748+0000 7fd413fff640 1 -- 172.21.15.78:0/58243167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd400003520 con 0x7fd4000047c0 2024-02-19T14:52:48.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd413fff640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4000047c0 0x7fd400004c40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009ab0 tx=0x7fd4000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9498ba5d1b85d9d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc005990 con 0x7fd4000047c0 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc005af0 con 0x7fd4000047c0 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd4137fe640 1 -- 172.21.15.78:0/58243167 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc005ea0 con 0x7fd4000047c0 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd41ab65640 1 -- 172.21.15.78:0/58243167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4000047c0 msgr2=0x7fd400004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd41ab65640 1 --2- 172.21.15.78:0/58243167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4000047c0 0x7fd400004c40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009ab0 tx=0x7fd4000057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd419102640 1 -- 172.21.15.78:0/58243167 reap_dead start 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd41ab65640 1 -- 172.21.15.78:0/58243167 shutdown_connections 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd41ab65640 1 -- 172.21.15.78:0/58243167 >> 172.21.15.78:0/58243167 conn(0x7fd4140ca870 msgr2=0x7fd4140cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd41ab65640 1 -- 172.21.15.78:0/58243167 shutdown_connections 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.750+0000 7fd41ab65640 1 -- 172.21.15.78:0/58243167 wait complete. 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.752+0000 7fd41ab65640 1 Processor -- start 2024-02-19T14:52:48.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.752+0000 7fd41ab65640 1 -- start start 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4000047c0 0x7fd41414b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd41414bf70 0x7fd4141eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 0x7fd4140bf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4141eb950 con 0x7fd4141eb4a0 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4141ec430 con 0x7fd41414bf70 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140c12c0 con 0x7fd4000047c0 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 0x7fd4140bf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 0x7fd4140bf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37092/0 (socket says 172.21.15.78:37092) 2024-02-19T14:52:48.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 -- 172.21.15.78:0/2600793406 learned_addr learned my addr 172.21.15.78:0/2600793406 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd413fff640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd41414bf70 0x7fd4141eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd418901640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4000047c0 0x7fd41414b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 -- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4000047c0 msgr2=0x7fd41414b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4000047c0 0x7fd41414b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 -- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd41414bf70 msgr2=0x7fd4141eaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd41414bf70 0x7fd4141eaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 -- 172.21.15.78:0/2600793406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd414156110 con 0x7fd4141eb4a0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd419102640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 0x7fd4140bf6f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd4140c6d40 tx=0x7fd40c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00e040 con 0x7fd4141eb4a0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40c00f040 con 0x7fd4141eb4a0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40c015670 con 0x7fd4141eb4a0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4140c1510 con 0x7fd4141eb4a0 2024-02-19T14:52:48.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.753+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4141ee250 con 0x7fd4141eb4a0 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.754+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4140c1700 con 0x7fd4141eb4a0 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.755+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd40c0157d0 con 0x7fd4141eb4a0 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.755+0000 7fd4117fa640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b840 0x7fd3f406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.755+0000 7fd418901640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b840 0x7fd3f406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.755+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fd40c08ad70 con 0x7fd4141eb4a0 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.756+0000 7fd418901640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b840 0x7fd3f406dd00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd404000fb0 tx=0x7fd404009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:48.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:48.761+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd40c055330 con 0x7fd4141eb4a0 2024-02-19T14:52:49.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.057+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd414074990 con 0x7fd4141eb4a0 2024-02-19T14:52:49.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.058+0000 7fd4117fa640 1 -- 172.21.15.78:0/2600793406 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7fd40c0589f0 con 0x7fd4141eb4a0 2024-02-19T14:52:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:49.061 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:49.061 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.060+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b840 msgr2=0x7fd3f406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.060+0000 7fd41ab65640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b840 0x7fd3f406dd00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd404000fb0 tx=0x7fd404009040 comp rx=0 tx=0).stop 2024-02-19T14:52:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.060+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 msgr2=0x7fd4140bf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.060+0000 7fd41ab65640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 0x7fd4140bf6f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd4140c6d40 tx=0x7fd40c00f580 comp rx=0 tx=0).stop 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 shutdown_connections 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b840 0x7fd3f406dd00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4141eb4a0 0x7fd4140bf6f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd41414bf70 0x7fd4141eaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 --2- 172.21.15.78:0/2600793406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4000047c0 0x7fd41414b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 >> 172.21.15.78:0/2600793406 conn(0x7fd4140ca870 msgr2=0x7fd414156610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 shutdown_connections 2024-02-19T14:52:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.061+0000 7fd41ab65640 1 -- 172.21.15.78:0/2600793406 wait complete. 2024-02-19T14:52:49.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:52:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 Processor -- start 2024-02-19T14:52:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 -- start start 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdf8c0c6d60 0x7fdf8c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdf8c06ff80 0x7fdf8c0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8c14f730 con 0x7fdf8c06e830 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf8c0c1940 con 0x7fdf8c0c6d60 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.207+0000 7fdf9135c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf8c0c16d0 con 0x7fdf8c06ff80 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdf8c0c6d60 0x7fdf8c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdf8c0c6d60 0x7fdf8c154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37102/0 (socket says 172.21.15.78:37102) 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 -- 172.21.15.78:0/943592337 learned_addr learned my addr 172.21.15.78:0/943592337 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8b7fe640 1 --2- 172.21.15.78:0/943592337 >> v2:172.21.15.150:3300/0 conn(0x7fdf8c06ff80 0x7fdf8c0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 -- 172.21.15.78:0/943592337 >> v2:172.21.15.150:3300/0 conn(0x7fdf8c06ff80 msgr2=0x7fdf8c0c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 --2- 172.21.15.78:0/943592337 >> v2:172.21.15.150:3300/0 conn(0x7fdf8c06ff80 0x7fdf8c0c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 -- 172.21.15.78:0/943592337 >> v1:172.21.15.150:6789/0 conn(0x7fdf8c06e830 legacy=0x7fdf8c0c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.208+0000 7fdf8affd640 1 -- 172.21.15.78:0/943592337 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c072f70 con 0x7fdf8c0c6d60 2024-02-19T14:52:49.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.209+0000 7fdf8affd640 1 --2- 172.21.15.78:0/943592337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c0c6d60 0x7fdf8c154f70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c06de00 tx=0x7fdf7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d147b7ba31dee64 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.209+0000 7fdf89ffb640 1 -- 172.21.15.78:0/943592337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c031870 con 0x7fdf8c0c6d60 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.209+0000 7fdf89ffb640 1 -- 172.21.15.78:0/943592337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c0319d0 con 0x7fdf8c0c6d60 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.209+0000 7fdf89ffb640 1 -- 172.21.15.78:0/943592337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c031d60 con 0x7fdf8c0c6d60 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.209+0000 7fdf9135c640 1 -- 172.21.15.78:0/943592337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c0c6d60 msgr2=0x7fdf8c154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.209+0000 7fdf9135c640 1 --2- 172.21.15.78:0/943592337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c0c6d60 0x7fdf8c154f70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c06de00 tx=0x7fdf7c02b910 comp rx=0 tx=0).stop 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.210+0000 7fdf9135c640 1 -- 172.21.15.78:0/943592337 shutdown_connections 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.210+0000 7fdf9135c640 1 --2- 172.21.15.78:0/943592337 >> v2:172.21.15.150:3300/0 conn(0x7fdf8c06ff80 0x7fdf8c0c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.210+0000 7fdf9135c640 1 --2- 172.21.15.78:0/943592337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c0c6d60 0x7fdf8c154f70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.210+0000 7fdf9135c640 1 -- 172.21.15.78:0/943592337 >> 172.21.15.78:0/943592337 conn(0x7fdf8c06c170 msgr2=0x7fdf8c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.210+0000 7fdf9135c640 1 -- 172.21.15.78:0/943592337 shutdown_connections 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.210+0000 7fdf9135c640 1 -- 172.21.15.78:0/943592337 wait complete. 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.211+0000 7fdf9135c640 1 Processor -- start 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.211+0000 7fdf9135c640 1 -- start start 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf8c06e830 0x7fdf8c0a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 0x7fdf8c14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf8c0c6d60 0x7fdf8c1503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c143440 con 0x7fdf8c06ff80 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c0aa100 con 0x7fdf8c06e830 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1eb940 con 0x7fdf8c0c6d60 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 0x7fdf8c14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 0x7fdf8c14feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37110/0 (socket says 172.21.15.78:37110) 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 -- 172.21.15.78:0/360865879 learned_addr learned my addr 172.21.15.78:0/360865879 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8affd640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf8c06e830 0x7fdf8c0a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8b7fe640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf8c0c6d60 0x7fdf8c1503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 -- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf8c0c6d60 msgr2=0x7fdf8c1503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf8c0c6d60 0x7fdf8c1503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 -- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf8c06e830 msgr2=0x7fdf8c0a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf8c06e830 0x7fdf8c0a9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 -- 172.21.15.78:0/360865879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0094d0 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8b7fe640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf8c0c6d60 0x7fdf8c1503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf8a7fc640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 0x7fdf8c14feb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b3c0 tx=0x7fdf8000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80011830 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80011990 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8c146c80 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80011d60 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.212+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c0c4d30 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.213+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c0c2230 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.214+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdf8001b460 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.214+0000 7fdf737fe640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf7806b810 0x7fdf7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.214+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fdf8008a430 con 0x7fdf8c06ff80 2024-02-19T14:52:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.214+0000 7fdf8affd640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf7806b810 0x7fdf7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.215+0000 7fdf8affd640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf7806b810 0x7fdf7806dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c036000 tx=0x7fdf7c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.223+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdf8c0c2230 con 0x7fdf8c06ff80 2024-02-19T14:52:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.482+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf8c0c2230 con 0x7fdf8c06ff80 2024-02-19T14:52:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.483+0000 7fdf737fe640 1 -- 172.21.15.78:0/360865879 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4940 (secure 0 0 0) 0x7fdf8c0c2230 con 0x7fdf8c06ff80 2024-02-19T14:52:49.485 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:49.485 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3967903152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3967903152},{"type":"v1","addr":"172.21.15.150:6834","nonce":3967903152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1449168324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1449168324},{"type":"v1","addr":"172.21.15.150:6835","nonce":1449168324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1565907915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1565907915},{"type":"v1","addr":"172.21.15.78:6835","nonce":1565907915}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:51:59.591061+0000","modified":"2024-02-19T14:52:15.695071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1940908201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1940908201},{"type":"v1","addr":"172.21.15.78:6837","nonce":1940908201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-19T14:52:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf7806b810 msgr2=0x7fdf7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf7806b810 0x7fdf7806dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c036000 tx=0x7fdf7c03c040 comp rx=0 tx=0).stop 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 msgr2=0x7fdf8c14feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 0x7fdf8c14feb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b3c0 tx=0x7fdf8000b880 comp rx=0 tx=0).stop 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 shutdown_connections 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdf7806b810 0x7fdf7806dcd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf8c06ff80 0x7fdf8c14feb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf8c0c6d60 0x7fdf8c1503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 --2- 172.21.15.78:0/360865879 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf8c06e830 0x7fdf8c0a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 >> 172.21.15.78:0/360865879 conn(0x7fdf8c06c170 msgr2=0x7fdf8c1543a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 shutdown_connections 2024-02-19T14:52:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.485+0000 7fdf9135c640 1 -- 172.21.15.78:0/360865879 wait complete. 2024-02-19T14:52:49.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.618+0000 7f73c7adf640 1 Processor -- start 2024-02-19T14:52:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.618+0000 7f73c7adf640 1 -- start start 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c7adf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f73c00ac390 0x7f73c00ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c7adf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f73c00bdf20 0x7f73c00c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c7adf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c00c8de0 con 0x7f73c0158530 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c7adf640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c0155a60 con 0x7f73c00bdf20 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c7adf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c0155d40 con 0x7f73c00ac390 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f73c00bdf20 0x7f73c00c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f73c00bdf20 0x7f73c00c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37118/0 (socket says 172.21.15.78:37118) 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 -- 172.21.15.78:0/3885014357 learned_addr learned my addr 172.21.15.78:0/3885014357 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c587b640 1 --2- 172.21.15.78:0/3885014357 >> v2:172.21.15.150:3300/0 conn(0x7f73c00ac390 0x7f73c00ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 -- 172.21.15.78:0/3885014357 >> v2:172.21.15.150:3300/0 conn(0x7f73c00ac390 msgr2=0x7f73c00ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 --2- 172.21.15.78:0/3885014357 >> v2:172.21.15.150:3300/0 conn(0x7f73c00ac390 0x7f73c00ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 -- 172.21.15.78:0/3885014357 >> v1:172.21.15.150:6789/0 conn(0x7f73c0158530 legacy=0x7f73c006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c507a640 1 -- 172.21.15.78:0/3885014357 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c00c90a0 con 0x7f73c00bdf20 2024-02-19T14:52:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.619+0000 7f73c587b640 1 --2- 172.21.15.78:0/3885014357 >> v2:172.21.15.150:3300/0 conn(0x7f73c00ac390 0x7f73c00ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.620+0000 7f73c507a640 1 --2- 172.21.15.78:0/3885014357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00bdf20 0x7f73c00c55f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f73c00ad2a0 tx=0x7f73b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700ceb7c003e4106 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c4879640 1 -- 172.21.15.78:0/3885014357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b00317e0 con 0x7f73c00bdf20 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c4879640 1 -- 172.21.15.78:0/3885014357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0031940 con 0x7f73c00bdf20 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c4879640 1 -- 172.21.15.78:0/3885014357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0031cd0 con 0x7f73c00bdf20 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 -- 172.21.15.78:0/3885014357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00bdf20 msgr2=0x7f73c00c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 --2- 172.21.15.78:0/3885014357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00bdf20 0x7f73c00c55f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f73c00ad2a0 tx=0x7f73b002b880 comp rx=0 tx=0).stop 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 -- 172.21.15.78:0/3885014357 shutdown_connections 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 --2- 172.21.15.78:0/3885014357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00bdf20 0x7f73c00c55f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 --2- 172.21.15.78:0/3885014357 >> v2:172.21.15.150:3300/0 conn(0x7f73c00ac390 0x7f73c00ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 -- 172.21.15.78:0/3885014357 >> 172.21.15.78:0/3885014357 conn(0x7f73c00ca890 msgr2=0x7f73c00caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 -- 172.21.15.78:0/3885014357 shutdown_connections 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.621+0000 7f73c7adf640 1 -- 172.21.15.78:0/3885014357 wait complete. 2024-02-19T14:52:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.622+0000 7f73c7adf640 1 Processor -- start 2024-02-19T14:52:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.622+0000 7f73c7adf640 1 -- start start 2024-02-19T14:52:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c7adf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 0x7f73c014df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c7adf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73c00bdf20 0x7f73c00ab380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c7adf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73c0158530 0x7f73c00c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c7adf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00c2690 con 0x7f73c00ac390 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c7adf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00c20d0 con 0x7f73c00bdf20 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c7adf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00c23b0 con 0x7f73c0158530 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 0x7f73c014df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 0x7f73c014df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37128/0 (socket says 172.21.15.78:37128) 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 -- 172.21.15.78:0/4146562344 learned_addr learned my addr 172.21.15.78:0/4146562344 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c507a640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73c00bdf20 0x7f73c00ab380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 -- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73c0158530 msgr2=0x7f73c00c3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73c0158530 0x7f73c00c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 -- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73c00bdf20 msgr2=0x7f73c00ab380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73c00bdf20 0x7f73c00ab380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c587b640 1 -- 172.21.15.78:0/4146562344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00094d0 con 0x7f73c00ac390 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.623+0000 7f73c507a640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73c00bdf20 0x7f73c00ab380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.624+0000 7f73c587b640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 0x7f73c014df00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f73b800e7e0 tx=0x7f73b800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.624+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8004030 con 0x7f73c00ac390 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.624+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8004190 con 0x7f73c00ac390 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.624+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c00ad610 con 0x7f73c00ac390 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.624+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8004520 con 0x7f73c00ac390 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.624+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c014e440 con 0x7f73c00ac390 2024-02-19T14:52:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.625+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c0074990 con 0x7f73c00ac390 2024-02-19T14:52:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.626+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f73b800c6d0 con 0x7f73c00ac390 2024-02-19T14:52:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.626+0000 7f73b67fc640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f73a406b810 0x7f73a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.626+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f73b808a310 con 0x7f73c00ac390 2024-02-19T14:52:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.626+0000 7f73c507a640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f73a406b810 0x7f73a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.627+0000 7f73c507a640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f73a406b810 0x7f73a406dcd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f73b0036000 tx=0x7f73b003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.635+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f73b80548d0 con 0x7f73c00ac390 2024-02-19T14:52:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.814+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c006da00 con 0x7f73c00ac390 2024-02-19T14:52:49.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.815+0000 7f73b67fc640 1 -- 172.21.15.78:0/4146562344 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19031 (secure 0 0 0) 0x7f73b8057f90 con 0x7f73c00ac390 2024-02-19T14:52:49.818 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:49.818 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:41.184968+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:51:58.054888+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-19T14:51:58.551083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-19T14:52:02.276458+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":0}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f73a406b810 msgr2=0x7f73a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f73a406b810 0x7f73a406dcd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f73b0036000 tx=0x7f73b003d040 comp rx=0 tx=0).stop 2024-02-19T14:52:49.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 msgr2=0x7f73c014df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 0x7f73c014df00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f73b800e7e0 tx=0x7f73b800eca0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 shutdown_connections 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f73a406b810 0x7f73a406dcd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73c0158530 0x7f73c00c3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73c00bdf20 0x7f73c00ab380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 --2- 172.21.15.78:0/4146562344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73c00ac390 0x7f73c014df00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.817+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 >> 172.21.15.78:0/4146562344 conn(0x7f73c00ca890 msgr2=0x7f73c0151230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.818+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 shutdown_connections 2024-02-19T14:52:49.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.818+0000 7f73c7adf640 1 -- 172.21.15.78:0/4146562344 wait complete. 2024-02-19T14:52:49.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:52:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.931+0000 7f1766743640 1 Processor -- start 2024-02-19T14:52:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.931+0000 7f1766743640 1 -- start start 2024-02-19T14:52:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1766743640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1760152480 0x7f17601559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1766743640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 0x7f176006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1766743640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1760155ef0 con 0x7f176006d350 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1766743640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176006dec0 con 0x7f1760152480 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1766743640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17600c75c0 con 0x7f17600c7070 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1764ce0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f176006d350 0x7f17600c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55928/0 (socket says 172.21.15.78:55928) 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f1764ce0640 1 -- 172.21.15.78:0/2953393942 learned_addr learned my addr 172.21.15.78:0/2953393942 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f175ffff640 1 --2- 172.21.15.78:0/2953393942 >> v2:172.21.15.150:3300/0 conn(0x7f1760152480 0x7f17601559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f175f7fe640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 0x7f176006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.932+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701830195 0 0) 0x7f1758001490 con 0x7f176006d350 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f174c003520 con 0x7f176006d350 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494958539 0 0) 0x7f174c003520 con 0x7f176006d350 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 msgr2=0x7f176006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 0x7f176006ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 >> v2:172.21.15.150:3300/0 conn(0x7f1760152480 msgr2=0x7f17601559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 --2- 172.21.15.78:0/2953393942 >> v2:172.21.15.150:3300/0 conn(0x7f1760152480 0x7f17601559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17600c78a0 con 0x7f176006d350 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1758001b80 con 0x7f176006d350 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 >> v1:172.21.15.78:6789/0 conn(0x7f176006d350 legacy=0x7f17600c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c003970 0x7f174c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c0047c0 0x7f174c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 0x7f174c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176006dec0 con 0x7f174c0047c0 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17600c75c0 con 0x7f174c003970 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c00b980 con 0x7f17600c7070 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175ffff640 1 --2- 172.21.15.78:0/2953393942 >> v2:172.21.15.150:3300/0 conn(0x7f1760152480 0x7f17601559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175f7fe640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c0047c0 0x7f174c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f1764ce0640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 0x7f174c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175ffff640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c003970 0x7f174c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175f7fe640 1 -- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 msgr2=0x7f174c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175f7fe640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17600c7070 0x7f174c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175f7fe640 1 -- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c003970 msgr2=0x7f174c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175f7fe640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c003970 0x7f174c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.933+0000 7f175f7fe640 1 -- 172.21.15.78:0/2953393942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c003520 con 0x7f174c0047c0 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.934+0000 7f175f7fe640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c0047c0 0x7f174c004c40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1750009a90 tx=0x7f174c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0508387ca97b16c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.934+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175002c680 con 0x7f174c0047c0 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.934+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175002c7e0 con 0x7f174c0047c0 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.934+0000 7f175effd640 1 -- 172.21.15.78:0/2953393942 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175002cb90 con 0x7f174c0047c0 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1766743640 1 -- 172.21.15.78:0/2953393942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c0047c0 msgr2=0x7f174c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1766743640 1 --2- 172.21.15.78:0/2953393942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c0047c0 0x7f174c004c40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1750009a90 tx=0x7f174c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1764ce0640 1 -- 172.21.15.78:0/2953393942 reap_dead start 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1766743640 1 -- 172.21.15.78:0/2953393942 shutdown_connections 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1766743640 1 -- 172.21.15.78:0/2953393942 >> 172.21.15.78:0/2953393942 conn(0x7f17600ca8a0 msgr2=0x7f17600cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1766743640 1 -- 172.21.15.78:0/2953393942 shutdown_connections 2024-02-19T14:52:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.935+0000 7f1766743640 1 -- 172.21.15.78:0/2953393942 wait complete. 2024-02-19T14:52:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.936+0000 7f1766743640 1 Processor -- start 2024-02-19T14:52:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.936+0000 7f1766743640 1 -- start start 2024-02-19T14:52:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c0047c0 0x7f1760149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 0x7f17600aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17601590a0 0x7f1760157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17600ab900 con 0x7f1760158c40 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176014a410 con 0x7f174c0047c0 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17600ac2c0 con 0x7f17601590a0 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 0x7f17600aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 0x7f17600aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37140/0 (socket says 172.21.15.78:37140) 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 -- 172.21.15.78:0/3821853987 learned_addr learned my addr 172.21.15.78:0/3821853987 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1764ce0640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17601590a0 0x7f1760157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175ffff640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c0047c0 0x7f1760149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 -- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17601590a0 msgr2=0x7f1760157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17601590a0 0x7f1760157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 -- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c0047c0 msgr2=0x7f1760149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c0047c0 0x7f1760149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 -- 172.21.15.78:0/3821853987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17600ac5a0 con 0x7f1760158c40 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175ffff640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c0047c0 0x7f1760149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175f7fe640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 0x7f17600aa150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1750009bb0 tx=0x7f1750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750034a10 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1750034b70 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17600ab900 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.937+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175003e400 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.938+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17600acb60 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.939+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1760157d40 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.939+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f175003e560 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.939+0000 7f175cff9640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f174006b810 0x7f174006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.940+0000 7f175ffff640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f174006b810 0x7f174006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.940+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f17500ac540 con 0x7f1760158c40 2024-02-19T14:52:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.940+0000 7f175ffff640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f174006b810 0x7f174006dcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1748009b90 tx=0x7f1748008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:49.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:49.944+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1750076bb0 con 0x7f1760158c40 2024-02-19T14:52:50.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.159+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1760074990 con 0x7f1760158c40 2024-02-19T14:52:50.162 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:52:50.159+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:52:50.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.160+0000 7f175cff9640 1 -- 172.21.15.78:0/3821853987 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f175007a270 con 0x7f1760158c40 2024-02-19T14:52:50.163 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:52:50.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f174006b810 msgr2=0x7f174006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:50.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f174006b810 0x7f174006dcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1748009b90 tx=0x7f1748008040 comp rx=0 tx=0).stop 2024-02-19T14:52:50.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 msgr2=0x7f17600aa150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:50.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 0x7f17600aa150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1750009bb0 tx=0x7f1750004730 comp rx=0 tx=0).stop 2024-02-19T14:52:50.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 shutdown_connections 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f174006b810 0x7f174006dcd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1760158c40 0x7f17600aa150 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1750009bb0 tx=0x7f1750004730 comp rx=0 tx=0).stop 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17601590a0 0x7f1760157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 --2- 172.21.15.78:0/3821853987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f174c0047c0 0x7f1760149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.163+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 >> 172.21.15.78:0/3821853987 conn(0x7f17600ca8a0 msgr2=0x7f1760154340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.164+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 shutdown_connections 2024-02-19T14:52:50.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.164+0000 7f1766743640 1 -- 172.21.15.78:0/3821853987 wait complete. 2024-02-19T14:52:50.168 INFO:tasks.ceph.mds.c.smithi078.stderr: -3499> 2024-02-19T14:52:04.656+0000 7f781f2f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:52:50.178 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:52:50.197 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:52:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.279+0000 7f6ca521c640 1 Processor -- start 2024-02-19T14:52:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.279+0000 7f6ca521c640 1 -- start start 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6ca521c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ca00c7b70 0x7f6ca00c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6ca521c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca0155bb0 0x7f6ca01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6ca521c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca006db90 con 0x7f6ca01589f0 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6ca521c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca0154ec0 con 0x7f6ca00c7b70 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6ca521c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca01550d0 con 0x7f6ca0155bb0 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ca00c7b70 0x7f6ca00c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ca00c7b70 0x7f6ca00c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37162/0 (socket says 172.21.15.78:37162) 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 -- 172.21.15.78:0/3985767116 learned_addr learned my addr 172.21.15.78:0/3985767116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 -- 172.21.15.78:0/3985767116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca0155bb0 msgr2=0x7f6ca01584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 --2- 172.21.15.78:0/3985767116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca0155bb0 0x7f6ca01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 -- 172.21.15.78:0/3985767116 >> v1:172.21.15.150:6789/0 conn(0x7f6ca01589f0 legacy=0x7f6ca006d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:50.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.280+0000 7f6c9ed9d640 1 -- 172.21.15.78:0/3985767116 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca01553b0 con 0x7f6ca00c7b70 2024-02-19T14:52:50.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.281+0000 7f6c9ed9d640 1 --2- 172.21.15.78:0/3985767116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00c7f50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009e50 tx=0x7f6c9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3ce7fb54532d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:50.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.281+0000 7f6c9dd9b640 1 -- 172.21.15.78:0/3985767116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9002c9e0 con 0x7f6ca00c7b70 2024-02-19T14:52:50.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.281+0000 7f6c9dd9b640 1 -- 172.21.15.78:0/3985767116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9002cb40 con 0x7f6ca00c7b70 2024-02-19T14:52:50.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.281+0000 7f6c9dd9b640 1 -- 172.21.15.78:0/3985767116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900343f0 con 0x7f6ca00c7b70 2024-02-19T14:52:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 -- 172.21.15.78:0/3985767116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00c7b70 msgr2=0x7f6ca00c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:50.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 --2- 172.21.15.78:0/3985767116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00c7f50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009e50 tx=0x7f6c9002be40 comp rx=0 tx=0).stop 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 -- 172.21.15.78:0/3985767116 shutdown_connections 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 --2- 172.21.15.78:0/3985767116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca0155bb0 0x7f6ca01584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 --2- 172.21.15.78:0/3985767116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00c7f50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 -- 172.21.15.78:0/3985767116 >> 172.21.15.78:0/3985767116 conn(0x7f6ca00c5050 msgr2=0x7f6ca00c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 -- 172.21.15.78:0/3985767116 shutdown_connections 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.282+0000 7f6ca521c640 1 -- 172.21.15.78:0/3985767116 wait complete. 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.283+0000 7f6ca521c640 1 Processor -- start 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.283+0000 7f6ca521c640 1 -- start start 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6ca521c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00706c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6ca521c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 0x7f6ca01eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6ca521c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca01589f0 0x7f6ca01ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6ca521c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00711d0 con 0x7f6ca0155bb0 2024-02-19T14:52:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6ca521c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00ac550 con 0x7f6ca00c7b70 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6ca521c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca01df6f0 con 0x7f6ca01589f0 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 0x7f6ca01eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 0x7f6ca01eddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37176/0 (socket says 172.21.15.78:37176) 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 -- 172.21.15.78:0/1744367714 learned_addr learned my addr 172.21.15.78:0/1744367714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9ed9d640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9f59e640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca01589f0 0x7f6ca01ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 -- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca01589f0 msgr2=0x7f6ca01ee310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca01589f0 0x7f6ca01ee310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 -- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca00c7b70 msgr2=0x7f6ca00706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00706c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9e59c640 1 -- 172.21.15.78:0/1744367714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009b30 con 0x7f6ca0155bb0 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9ed9d640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00706c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:50.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.284+0000 7f6c9f59e640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca01589f0 0x7f6ca01ee310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.285+0000 7f6c9e59c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 0x7f6ca01eddd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400b4a0 tx=0x7f6c9400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.285+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94010610 con 0x7f6ca0155bb0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.285+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c94010770 con 0x7f6ca0155bb0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.285+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca01df9d0 con 0x7f6ca0155bb0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.285+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94010b20 con 0x7f6ca0155bb0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.285+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca00ada90 con 0x7f6ca0155bb0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.286+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca00adfb0 con 0x7f6ca0155bb0 2024-02-19T14:52:50.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.286+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c94010c80 con 0x7f6ca0155bb0 2024-02-19T14:52:50.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.287+0000 7f6c877fe640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8c06b840 0x7f6c8c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:50.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.287+0000 7f6c9ed9d640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8c06b840 0x7f6c8c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:50.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.287+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f6c9408a210 con 0x7f6ca0155bb0 2024-02-19T14:52:50.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.287+0000 7f6c9ed9d640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8c06b840 0x7f6c8c06dd00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c90035000 tx=0x7f6c90030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:50.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.291+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ca00adfb0 con 0x7f6ca0155bb0 2024-02-19T14:52:50.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:50.502+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6ca00adfb0 con 0x7f6ca0155bb0 2024-02-19T14:52:51.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.162+0000 7f6c877fe640 1 -- 172.21.15.78:0/1744367714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ca00adfb0 con 0x7f6ca0155bb0 2024-02-19T14:52:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.164+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8c06b840 msgr2=0x7f6c8c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.164+0000 7f6ca521c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8c06b840 0x7f6c8c06dd00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c90035000 tx=0x7f6c90030000 comp rx=0 tx=0).stop 2024-02-19T14:52:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.164+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 msgr2=0x7f6ca01eddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.164+0000 7f6ca521c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 0x7f6ca01eddd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400b4a0 tx=0x7f6c9400b960 comp rx=0 tx=0).stop 2024-02-19T14:52:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 shutdown_connections 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8c06b840 0x7f6c8c06dd00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0155bb0 0x7f6ca01eddd0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca01589f0 0x7f6ca01ee310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 --2- 172.21.15.78:0/1744367714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca00c7b70 0x7f6ca00706c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 >> 172.21.15.78:0/1744367714 conn(0x7f6ca00c5050 msgr2=0x7f6ca00c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 shutdown_connections 2024-02-19T14:52:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.165+0000 7f6ca521c640 1 -- 172.21.15.78:0/1744367714 wait complete. 2024-02-19T14:52:51.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:52:51.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.331+0000 7f356999a640 1 Processor -- start 2024-02-19T14:52:51.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f356999a640 1 -- start start 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f356999a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35641504d0 0x7f35640c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f356999a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 0x7f356406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f356999a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35641572c0 con 0x7f35640a9630 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f356999a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35641569a0 con 0x7f35641504d0 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f356999a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3564153490 con 0x7f3564156e30 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f3562ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35641504d0 0x7f35640c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f355a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f35640a9630 0x7f3564152d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55944/0 (socket says 172.21.15.78:55944) 2024-02-19T14:52:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.332+0000 7f355a7fc640 1 -- 172.21.15.78:0/91855874 learned_addr learned my addr 172.21.15.78:0/91855874 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.333+0000 7f3562ffd640 1 -- 172.21.15.78:0/91855874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 msgr2=0x7f356406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.333+0000 7f3562ffd640 1 --2- 172.21.15.78:0/91855874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 0x7f356406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.333+0000 7f3562ffd640 1 -- 172.21.15.78:0/91855874 >> v1:172.21.15.78:6789/0 conn(0x7f35640a9630 legacy=0x7f3564152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.333+0000 7f3562ffd640 1 -- 172.21.15.78:0/91855874 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35640c5b90 con 0x7f35641504d0 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.333+0000 7f3562ffd640 1 --2- 172.21.15.78:0/91855874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35641504d0 0x7f35640c55b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f35540097a0 tx=0x7f355402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f5baddf5a7f96d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f35627fc640 1 -- 172.21.15.78:0/91855874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554030710 con 0x7f35641504d0 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f35627fc640 1 -- 172.21.15.78:0/91855874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355402e050 con 0x7f35641504d0 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f35627fc640 1 -- 172.21.15.78:0/91855874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554030a90 con 0x7f35641504d0 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 -- 172.21.15.78:0/91855874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35641504d0 msgr2=0x7f35640c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 --2- 172.21.15.78:0/91855874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35641504d0 0x7f35640c55b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f35540097a0 tx=0x7f355402b710 comp rx=0 tx=0).stop 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 -- 172.21.15.78:0/91855874 shutdown_connections 2024-02-19T14:52:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 --2- 172.21.15.78:0/91855874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 0x7f356406ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 --2- 172.21.15.78:0/91855874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35641504d0 0x7f35640c55b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 -- 172.21.15.78:0/91855874 >> 172.21.15.78:0/91855874 conn(0x7f35640c6fe0 msgr2=0x7f35640a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:51.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 -- 172.21.15.78:0/91855874 shutdown_connections 2024-02-19T14:52:51.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.334+0000 7f356999a640 1 -- 172.21.15.78:0/91855874 wait complete. 2024-02-19T14:52:51.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.335+0000 7f356999a640 1 Processor -- start 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.335+0000 7f356999a640 1 -- start start 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f356999a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 0x7f3564151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f356999a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35641504d0 0x7f3564147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f356999a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 0x7f35641ebef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f356999a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35640a7a30 con 0x7f35640a9630 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f356999a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3564072480 con 0x7f35641504d0 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f356999a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3564072720 con 0x7f3564156e30 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f3562ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 0x7f3564151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f3562ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 0x7f3564151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37194/0 (socket says 172.21.15.78:37194) 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.336+0000 7f3562ffd640 1 -- 172.21.15.78:0/2661917105 learned_addr learned my addr 172.21.15.78:0/2661917105 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f355a7fc640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35641504d0 0x7f3564147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3562ffd640 1 -- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 msgr2=0x7f35641ebef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3562ffd640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 0x7f35641ebef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3562ffd640 1 -- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35641504d0 msgr2=0x7f3564147c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3562ffd640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35641504d0 0x7f3564147c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3562ffd640 1 -- 172.21.15.78:0/2661917105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554009400 con 0x7f35640a9630 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3562ffd640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 0x7f3564151e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35540097a0 tx=0x7f35540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554032910 con 0x7f35640a9630 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3554032a70 con 0x7f35640a9630 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3564072a00 con 0x7f35640a9630 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554032e20 con 0x7f35640a9630 2024-02-19T14:52:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.337+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35641481d0 con 0x7f35640a9630 2024-02-19T14:52:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.338+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35640678b0 con 0x7f35640a9630 2024-02-19T14:52:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.338+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3554039040 con 0x7f35640a9630 2024-02-19T14:52:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.339+0000 7f3560ff9640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f354406b810 0x7f354406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.339+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f35540ac3d0 con 0x7f35640a9630 2024-02-19T14:52:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.339+0000 7f355a7fc640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f354406b810 0x7f354406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.339+0000 7f355a7fc640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f354406b810 0x7f354406dcd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f354c007fd0 tx=0x7f354c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.343+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f35540769d0 con 0x7f35640a9630 2024-02-19T14:52:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:51.513+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3564072f20 con 0x7f35640a9630 2024-02-19T14:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.233+0000 7f3560ff9640 1 -- 172.21.15.78:0/2661917105 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3564072f20 con 0x7f35640a9630 2024-02-19T14:52:52.236 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:52:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f354406b810 msgr2=0x7f354406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f354406b810 0x7f354406dcd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f354c007fd0 tx=0x7f354c009040 comp rx=0 tx=0).stop 2024-02-19T14:52:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 msgr2=0x7f3564151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:52.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 0x7f3564151e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f35540097a0 tx=0x7f35540046c0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 shutdown_connections 2024-02-19T14:52:52.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f354406b810 0x7f354406dcd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3564156e30 0x7f35641ebef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35641504d0 0x7f3564147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 --2- 172.21.15.78:0/2661917105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35640a9630 0x7f3564151e80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.237+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 >> 172.21.15.78:0/2661917105 conn(0x7f35640c6fe0 msgr2=0x7f35640aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:52.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.238+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 shutdown_connections 2024-02-19T14:52:52.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.238+0000 7f356999a640 1 -- 172.21.15.78:0/2661917105 wait complete. 2024-02-19T14:52:52.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:52:52.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 Processor -- start 2024-02-19T14:52:52.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 -- start start 2024-02-19T14:52:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe24c0c4cd0 0x7fe24c0c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c1539e0 con 0x7fe24c0ae050 2024-02-19T14:52:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24c1537d0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe2512d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c0709f0 con 0x7fe24c0c2aa0 2024-02-19T14:52:52.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe24b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe24c0c4cd0 0x7fe24c0c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe24b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe24c0c4cd0 0x7fe24c0c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37210/0 (socket says 172.21.15.78:37210) 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.361+0000 7fe24b7fe640 1 -- 172.21.15.78:0/1645656851 learned_addr learned my addr 172.21.15.78:0/1645656851 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe243fff640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe24b7fe640 1 -- 172.21.15.78:0/1645656851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 msgr2=0x7fe24c0c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe24b7fe640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c0c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe24b7fe640 1 -- 172.21.15.78:0/1645656851 >> v1:172.21.15.150:6789/0 conn(0x7fe24c0ae050 legacy=0x7fe24c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe24b7fe640 1 -- 172.21.15.78:0/1645656851 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c0c3140 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe243fff640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.362+0000 7fe24b7fe640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c0c5140 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe244009e50 tx=0x7fe24402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3425ecf1fb370cbe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe24a7fc640 1 -- 172.21.15.78:0/1645656851 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24402c9e0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe24a7fc640 1 -- 172.21.15.78:0/1645656851 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24402cb40 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe24a7fc640 1 -- 172.21.15.78:0/1645656851 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2440343f0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 -- 172.21.15.78:0/1645656851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 msgr2=0x7fe24c0c5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c0c5140 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe244009e50 tx=0x7fe24402be40 comp rx=0 tx=0).stop 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 -- 172.21.15.78:0/1645656851 shutdown_connections 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c0c5140 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 --2- 172.21.15.78:0/1645656851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 -- 172.21.15.78:0/1645656851 >> 172.21.15.78:0/1645656851 conn(0x7fe24c06f220 msgr2=0x7fe24c06f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 -- 172.21.15.78:0/1645656851 shutdown_connections 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.363+0000 7fe2512d4640 1 -- 172.21.15.78:0/1645656851 wait complete. 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.364+0000 7fe2512d4640 1 Processor -- start 2024-02-19T14:52:52.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.364+0000 7fe2512d4640 1 -- start start 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe2512d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe24c0ae050 0x7fe24c0a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe2512d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe2512d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe2512d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c0a90c0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe2512d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c0a9a90 con 0x7fe24c0ae050 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe2512d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c1543a0 con 0x7fe24c0c2aa0 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37212/0 (socket says 172.21.15.78:37212) 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 -- 172.21.15.78:0/4097414737 learned_addr learned my addr 172.21.15.78:0/4097414737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24affd640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe24c0ae050 0x7fe24c0a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:52.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe243fff640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 -- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 msgr2=0x7fe24c155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 -- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe24c0ae050 msgr2=0x7fe24c0a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe24c0ae050 0x7fe24c0a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24b7fe640 1 -- 172.21.15.78:0/4097414737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244009b30 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.365+0000 7fe24affd640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe24c0ae050 0x7fe24c0a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.366+0000 7fe24b7fe640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c140780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe244036040 tx=0x7fe244004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.366+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24403e450 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.366+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24403e5d0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.366+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24403e980 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.366+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c154680 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.366+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c1557b0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.367+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c0a8bb0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.367+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe24403eae0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.367+0000 7fe2437fe640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe22c06b810 0x7fe22c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.368+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6470+0+0 (secure 0 0 0) 0x7fe2440ad2d0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.368+0000 7fe24affd640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe22c06b810 0x7fe22c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.368+0000 7fe24affd640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe22c06b810 0x7fe22c06dcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe24c14ff10 tx=0x7fe23c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:52.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.372+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe24c0a8bb0 con 0x7fe24c0c4cd0 2024-02-19T14:52:52.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:52.552+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe24c0a8bb0 con 0x7fe24c0c4cd0 2024-02-19T14:52:53.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.235+0000 7fe2437fe640 1 -- 172.21.15.78:0/4097414737 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe24c0a8bb0 con 0x7fe24c0c4cd0 2024-02-19T14:52:53.237 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe22c06b810 msgr2=0x7fe22c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe22c06b810 0x7fe22c06dcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe24c14ff10 tx=0x7fe23c009040 comp rx=0 tx=0).stop 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 msgr2=0x7fe24c140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c140780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe244036040 tx=0x7fe244004a40 comp rx=0 tx=0).stop 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 shutdown_connections 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe22c06b810 0x7fe22c06dcd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe24c0c4cd0 0x7fe24c140780 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe24c0c2aa0 0x7fe24c155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 --2- 172.21.15.78:0/4097414737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe24c0ae050 0x7fe24c0a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 >> 172.21.15.78:0/4097414737 conn(0x7fe24c06f220 msgr2=0x7fe24c074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:53.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 shutdown_connections 2024-02-19T14:52:53.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.237+0000 7fe2512d4640 1 -- 172.21.15.78:0/4097414737 wait complete. 2024-02-19T14:52:53.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-02-19T14:52:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 Processor -- start 2024-02-19T14:52:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 -- start start 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb77c0c5590 0x7fb77c0c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c151d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb77c0c2940 0x7fb77c152270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c0abb60 con 0x7fb77c0c5590 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c06f0a0 con 0x7fb77c0c2940 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.356+0000 7fb780b81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c06e210 con 0x7fb77c06e6a0 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb77c0c5590 0x7fb77c0c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb77c0c5590 0x7fb77c0c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37218/0 (socket says 172.21.15.78:37218) 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 -- 172.21.15.78:0/3007862089 learned_addr learned my addr 172.21.15.78:0/3007862089 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 -- 172.21.15.78:0/3007862089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 msgr2=0x7fb77c151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77ad9d640 1 --2- 172.21.15.78:0/3007862089 >> v2:172.21.15.150:3300/0 conn(0x7fb77c0c2940 0x7fb77c152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb773fff640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 -- 172.21.15.78:0/3007862089 >> v2:172.21.15.150:3300/0 conn(0x7fb77c0c2940 msgr2=0x7fb77c152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 --2- 172.21.15.78:0/3007862089 >> v2:172.21.15.150:3300/0 conn(0x7fb77c0c2940 0x7fb77c152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77a59c640 1 -- 172.21.15.78:0/3007862089 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c0c26f0 con 0x7fb77c0c5590 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb77ad9d640 1 --2- 172.21.15.78:0/3007862089 >> v2:172.21.15.150:3300/0 conn(0x7fb77c0c2940 0x7fb77c152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.357+0000 7fb773fff640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb77a59c640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c5590 0x7fb77c0c5970 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0c0e50 tx=0x7fb76c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440c8037113a814c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb779d9b640 1 -- 172.21.15.78:0/3007862089 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0317e0 con 0x7fb77c0c5590 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb779d9b640 1 -- 172.21.15.78:0/3007862089 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c031940 con 0x7fb77c0c5590 2024-02-19T14:52:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb779d9b640 1 -- 172.21.15.78:0/3007862089 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c031cd0 con 0x7fb77c0c5590 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 -- 172.21.15.78:0/3007862089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c5590 msgr2=0x7fb77c0c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c5590 0x7fb77c0c5970 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0c0e50 tx=0x7fb76c02b880 comp rx=0 tx=0).stop 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 -- 172.21.15.78:0/3007862089 shutdown_connections 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 --2- 172.21.15.78:0/3007862089 >> v2:172.21.15.150:3300/0 conn(0x7fb77c0c2940 0x7fb77c152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 --2- 172.21.15.78:0/3007862089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c5590 0x7fb77c0c5970 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.358+0000 7fb780b81640 1 -- 172.21.15.78:0/3007862089 >> 172.21.15.78:0/3007862089 conn(0x7fb77c0c6fe0 msgr2=0x7fb77c0a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.359+0000 7fb780b81640 1 -- 172.21.15.78:0/3007862089 shutdown_connections 2024-02-19T14:52:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.359+0000 7fb780b81640 1 -- 172.21.15.78:0/3007862089 wait complete. 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 Processor -- start 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 -- start start 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 0x7fb77c140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb77c0c5590 0x7fb77c141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c150e60 con 0x7fb77c0c2940 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c073250 con 0x7fb77c0c5590 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.360+0000 7fb780b81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c072d20 con 0x7fb77c06e6a0 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 0x7fb77c140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 0x7fb77c140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37222/0 (socket says 172.21.15.78:37222) 2024-02-19T14:52:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 -- 172.21.15.78:0/2933162158 learned_addr learned my addr 172.21.15.78:0/2933162158 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb77ad9d640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb77c0c5590 0x7fb77c141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb77a59c640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 -- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 msgr2=0x7fb77c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c0712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 -- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb77c0c5590 msgr2=0x7fb77c141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb77c0c5590 0x7fb77c141160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 -- 172.21.15.78:0/2933162158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c0094d0 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb773fff640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 0x7fb77c140c20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c7f0 tx=0x7fb76400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764004230 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7640043b0 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c072180 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764004520 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.361+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c1472f0 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.362+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c0678b0 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.363+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb764007a70 con 0x7fb77c0c2940 2024-02-19T14:52:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.363+0000 7fb772ffd640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb75c06b810 0x7fb75c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.363+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fb76408a350 con 0x7fb77c0c2940 2024-02-19T14:52:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.363+0000 7fb77a59c640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb75c06b810 0x7fb75c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.363+0000 7fb77a59c640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb75c06b810 0x7fb75c06dcd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb76c036000 tx=0x7fb76c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.367+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb764054dc0 con 0x7fb77c0c2940 2024-02-19T14:52:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:53.538+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb77c074990 con 0x7fb77c0c2940 2024-02-19T14:52:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.249+0000 7fb772ffd640 1 -- 172.21.15.78:0/2933162158 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb764058480 con 0x7fb77c0c2940 2024-02-19T14:52:54.251 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' removed 2024-02-19T14:52:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb75c06b810 msgr2=0x7fb75c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb75c06b810 0x7fb75c06dcd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb76c036000 tx=0x7fb76c03d040 comp rx=0 tx=0).stop 2024-02-19T14:52:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 msgr2=0x7fb77c140c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 0x7fb77c140c20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c7f0 tx=0x7fb76400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 shutdown_connections 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb75c06b810 0x7fb75c06dcd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb77c0c2940 0x7fb77c140c20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb77c0c5590 0x7fb77c141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 --2- 172.21.15.78:0/2933162158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb77c06e6a0 0x7fb77c0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 >> 172.21.15.78:0/2933162158 conn(0x7fb77c0c6fe0 msgr2=0x7fb77c0c3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.252+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 shutdown_connections 2024-02-19T14:52:54.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.253+0000 7fb780b81640 1 -- 172.21.15.78:0/2933162158 wait complete. 2024-02-19T14:52:54.268 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:52:54.268 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:52:54.268 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:52:54.268 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:52:54.268 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:52:54.269 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:52:54.269 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:52:54.269 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:52:54.269 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:52:54.269 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:52:54.269 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:52:54.269 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:52:54.270 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:52:54.270 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:52:54.270 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:52:54.270 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:52:54.278 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:52:54.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:52:54.279 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:52:54.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:52:54.283 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:52:54.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:52:54.284 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:52:54.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:52:54.321 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:52:54.321 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:54.322 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:52:54.322 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:54.326 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:52:54.326 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:54.326 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:52:54.327 DEBUG:teuthology.parallel:result is None 2024-02-19T14:52:54.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:52:54.398 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:52:54.398 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:52:54.407 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:52:54.407 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:52:54.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.431+0000 7f7c9181c640 1 Processor -- start 2024-02-19T14:52:54.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.431+0000 7f7c9181c640 1 -- start start 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c9181c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c8c152700 0x7f7c8c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c9181c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c8c0c8e00 0x7f7c8c15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c9181c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8c1551f0 con 0x7f7c8c0c5730 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c9181c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8c155490 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c9181c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8c155770 con 0x7f7c8c152700 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c8c0c8e00 0x7f7c8c15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c8c0c8e00 0x7f7c8c15f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37254/0 (socket says 172.21.15.78:37254) 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 -- 172.21.15.78:0/2408855431 learned_addr learned my addr 172.21.15.78:0/2408855431 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 -- 172.21.15.78:0/2408855431 >> v2:172.21.15.150:3300/0 conn(0x7f7c8c152700 msgr2=0x7f7c8c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8affd640 1 --2- 172.21.15.78:0/2408855431 >> v2:172.21.15.150:3300/0 conn(0x7f7c8c152700 0x7f7c8c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 --2- 172.21.15.78:0/2408855431 >> v2:172.21.15.150:3300/0 conn(0x7f7c8c152700 0x7f7c8c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 -- 172.21.15.78:0/2408855431 >> v1:172.21.15.150:6789/0 conn(0x7f7c8c0c5730 legacy=0x7f7c8c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:54.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.432+0000 7f7c8b7fe640 1 -- 172.21.15.78:0/2408855431 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c155a50 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c8b7fe640 1 --2- 172.21.15.78:0/2408855431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c15f980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c08bad0 tx=0x7f7c8402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe983f71807d4298 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:54.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c89ffb640 1 -- 172.21.15.78:0/2408855431 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8402c9f0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c89ffb640 1 -- 172.21.15.78:0/2408855431 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8402cb50 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c89ffb640 1 -- 172.21.15.78:0/2408855431 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c840343f0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c9181c640 1 -- 172.21.15.78:0/2408855431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 msgr2=0x7f7c8c15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:54.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c9181c640 1 --2- 172.21.15.78:0/2408855431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c15f980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c08bad0 tx=0x7f7c8402be50 comp rx=0 tx=0).stop 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c9181c640 1 -- 172.21.15.78:0/2408855431 shutdown_connections 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c9181c640 1 --2- 172.21.15.78:0/2408855431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c15f980 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c9181c640 1 --2- 172.21.15.78:0/2408855431 >> v2:172.21.15.150:3300/0 conn(0x7f7c8c152700 0x7f7c8c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.433+0000 7f7c9181c640 1 -- 172.21.15.78:0/2408855431 >> 172.21.15.78:0/2408855431 conn(0x7f7c8c0cac90 msgr2=0x7f7c8c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.434+0000 7f7c9181c640 1 -- 172.21.15.78:0/2408855431 shutdown_connections 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.434+0000 7f7c9181c640 1 -- 172.21.15.78:0/2408855431 wait complete. 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 Processor -- start 2024-02-19T14:52:54.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 -- start start 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c8c0c5730 0x7f7c8c0afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c0c03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c8c152700 0x7f7c8c0be5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0c0db0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0c1090 con 0x7f7c8c152700 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.435+0000 7f7c9181c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0c1370 con 0x7f7c8c0c5730 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c0c03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c0c03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37268/0 (socket says 172.21.15.78:37268) 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 -- 172.21.15.78:0/1707667551 learned_addr learned my addr 172.21.15.78:0/1707667551 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8affd640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c8c0c5730 0x7f7c8c0afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:54.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8b7fe640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c8c152700 0x7f7c8c0be5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 -- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c8c0c5730 msgr2=0x7f7c8c0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c8c0c5730 0x7f7c8c0afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 -- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c8c152700 msgr2=0x7f7c8c0be5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c8c152700 0x7f7c8c0be5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 -- 172.21.15.78:0/1707667551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c84009b30 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c8a7fc640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c0c03d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000c430 tx=0x7f7c8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80011790 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c800118f0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8c0ac5d0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80011ce0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.436+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c0c0910 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.437+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c0acaf0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.438+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7c8001c460 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.438+0000 7f7c73fff640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c7806b810 0x7f7c7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:54.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.438+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f7c80089ec0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.438+0000 7f7c8affd640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c7806b810 0x7f7c7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:54.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.439+0000 7f7c8affd640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c7806b810 0x7f7c7806dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0b0400 tx=0x7f7c7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:54.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.442+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7c8c0acaf0 con 0x7f7c8c0c8e00 2024-02-19T14:52:54.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:54.626+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7c8c0acaf0 con 0x7f7c8c0c8e00 2024-02-19T14:52:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.258+0000 7f7c73fff640 1 -- 172.21.15.78:0/1707667551 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7c8c0acaf0 con 0x7f7c8c0c8e00 2024-02-19T14:52:55.261 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c7806b810 msgr2=0x7f7c7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c7806b810 0x7f7c7806dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0b0400 tx=0x7f7c7c008040 comp rx=0 tx=0).stop 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 msgr2=0x7f7c8c0c03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c0c03d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000c430 tx=0x7f7c8000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 shutdown_connections 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c7806b810 0x7f7c7806dcd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c8c0c8e00 0x7f7c8c0c03d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c8c152700 0x7f7c8c0be5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 --2- 172.21.15.78:0/1707667551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c8c0c5730 0x7f7c8c0afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 >> 172.21.15.78:0/1707667551 conn(0x7f7c8c0cac90 msgr2=0x7f7c8c1513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 shutdown_connections 2024-02-19T14:52:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.261+0000 7f7c9181c640 1 -- 172.21.15.78:0/1707667551 wait complete. 2024-02-19T14:52:55.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:52:55.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 Processor -- start 2024-02-19T14:52:55.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 -- start start 2024-02-19T14:52:55.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3764152a50 0x7f3764071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f37640702d0 0x7f3764158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764071bf0 con 0x7f3764152670 2024-02-19T14:52:55.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3764158a20 con 0x7f3764152a50 2024-02-19T14:52:55.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3769f2c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640c6780 con 0x7f37640702d0 2024-02-19T14:52:55.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.382+0000 7f3762ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3764152a50 0x7f3764071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3764152a50 0x7f3764071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37282/0 (socket says 172.21.15.78:37282) 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 -- 172.21.15.78:0/1127933325 learned_addr learned my addr 172.21.15.78:0/1127933325 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 -- 172.21.15.78:0/1127933325 >> v2:172.21.15.150:3300/0 conn(0x7f37640702d0 msgr2=0x7f3764158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 --2- 172.21.15.78:0/1127933325 >> v2:172.21.15.150:3300/0 conn(0x7f37640702d0 0x7f3764158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 -- 172.21.15.78:0/1127933325 >> v1:172.21.15.150:6789/0 conn(0x7f3764152670 legacy=0x7f3764155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 -- 172.21.15.78:0/1127933325 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3764071630 con 0x7f3764152a50 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.383+0000 7f3762ffd640 1 --2- 172.21.15.78:0/1127933325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152a50 0x7f3764071050 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3764070df0 tx=0x7f375802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4f620f8e231554 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f37627fc640 1 -- 172.21.15.78:0/1127933325 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758030860 con 0x7f3764152a50 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f37627fc640 1 -- 172.21.15.78:0/1127933325 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37580309c0 con 0x7f3764152a50 2024-02-19T14:52:55.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f37627fc640 1 -- 172.21.15.78:0/1127933325 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758030d50 con 0x7f3764152a50 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 -- 172.21.15.78:0/1127933325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152a50 msgr2=0x7f3764071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1127933325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152a50 0x7f3764071050 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3764070df0 tx=0x7f375802b9a0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 -- 172.21.15.78:0/1127933325 shutdown_connections 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1127933325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152a50 0x7f3764071050 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1127933325 >> v2:172.21.15.150:3300/0 conn(0x7f37640702d0 0x7f3764158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 -- 172.21.15.78:0/1127933325 >> 172.21.15.78:0/1127933325 conn(0x7f37640c85e0 msgr2=0x7f37640c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 -- 172.21.15.78:0/1127933325 shutdown_connections 2024-02-19T14:52:55.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.384+0000 7f3769f2c640 1 -- 172.21.15.78:0/1127933325 wait complete. 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.385+0000 7f3769f2c640 1 Processor -- start 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.385+0000 7f3769f2c640 1 -- start start 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3769f2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37640702d0 0x7f37641ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3769f2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 0x7f3764153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3769f2c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3764152a50 0x7f37641ea980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3769f2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37641539c0 con 0x7f3764152670 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3769f2c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640ab980 con 0x7f3764152a50 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3769f2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640ab6a0 con 0x7f37640702d0 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 0x7f3764153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 0x7f3764153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37298/0 (socket says 172.21.15.78:37298) 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f3762ffd640 1 -- 172.21.15.78:0/1349201566 learned_addr learned my addr 172.21.15.78:0/1349201566 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.386+0000 7f37637fe640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37640702d0 0x7f37641ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3763fff640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3764152a50 0x7f37641ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3762ffd640 1 -- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37640702d0 msgr2=0x7f37641ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3762ffd640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37640702d0 0x7f37641ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3762ffd640 1 -- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3764152a50 msgr2=0x7f37641ea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3762ffd640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3764152a50 0x7f37641ea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3762ffd640 1 -- 172.21.15.78:0/1349201566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580094d0 con 0x7f3764152670 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f37637fe640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37640702d0 0x7f37641ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3763fff640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3764152a50 0x7f37641ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3762ffd640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 0x7f3764153e50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3758000c00 tx=0x7f3758004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758039d40 con 0x7f3764152670 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3758043400 con 0x7f3764152670 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758039ea0 con 0x7f3764152670 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37641eea60 con 0x7f3764152670 2024-02-19T14:52:55.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.387+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37641ee6d0 con 0x7f3764152670 2024-02-19T14:52:55.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.388+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37640676a0 con 0x7f3764152670 2024-02-19T14:52:55.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.388+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f37580437b0 con 0x7f3764152670 2024-02-19T14:52:55.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.389+0000 7f374bfff640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375006b840 0x7f375006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.389+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f375803e070 con 0x7f3764152670 2024-02-19T14:52:55.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.389+0000 7f37637fe640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375006b840 0x7f375006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.389+0000 7f37637fe640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375006b840 0x7f375006dd00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f37641555a0 tx=0x7f3754009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:55.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.393+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3758077f60 con 0x7f3764152670 2024-02-19T14:52:55.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:52:55.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:52:55.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.634+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3764074990 con 0x7f3764152670 2024-02-19T14:52:55.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.635+0000 7f374bfff640 1 -- 172.21.15.78:0/1349201566 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6065 (secure 0 0 0) 0x7f375807b620 con 0x7f3764152670 2024-02-19T14:52:55.637 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:52:55.637 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:52:55.638 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:52:55.639 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.640 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.641 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:52:55.642 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.644 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:52:55.645 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:52:55.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.647 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-mds", 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQD3atNlRdfVMBAAoPfGqYnAhmXIeWjW9d29/g==", 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-mds" 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-mgr", 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQD3atNlseHVMBAAYn1CJ4tZVMTU4Xoc6tUQ+A==", 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-mgr" 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.648 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-osd", 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQD3atNlpOrVMBAAADMMuJNyJez7lZxY+18fzw==", 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-osd" 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-rbd", 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQD3atNlxfLVMBAAsZwFKIgzFlPp2G6CIPFQCA==", 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-rbd" 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.649 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQD3atNlu/nVMBAA/jaoNv9UGoGYzlINPrYxxA==", 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.bootstrap-rgw", 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQD3atNlcAHWMBAAIZwWD6byD2ZNLRJaOTteMA==", 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile bootstrap-rgw" 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:52:55.650 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:55.652 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:52:55.652 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.636+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375006b840 msgr2=0x7f375006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.636+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375006b840 0x7f375006dd00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f37641555a0 tx=0x7f3754009040 comp rx=0 tx=0).stop 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 msgr2=0x7f3764153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 0x7f3764153e50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3758000c00 tx=0x7f3758004730 comp rx=0 tx=0).stop 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 shutdown_connections 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375006b840 0x7f375006dd00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3764152670 0x7f3764153e50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3764152a50 0x7f37641ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 --2- 172.21.15.78:0/1349201566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37640702d0 0x7f37641ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 >> 172.21.15.78:0/1349201566 conn(0x7f37640c85e0 msgr2=0x7f3764156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 shutdown_connections 2024-02-19T14:52:55.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.637+0000 7f3769f2c640 1 -- 172.21.15.78:0/1349201566 wait complete. 2024-02-19T14:52:55.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-02-19T14:52:55.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.753+0000 7f1c196d4640 1 Processor -- start 2024-02-19T14:52:55.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.753+0000 7f1c196d4640 1 -- start start 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.753+0000 7f1c196d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 0x7f1c1406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.753+0000 7f1c196d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c141551f0 con 0x7f1c1406d910 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.753+0000 7f1c196d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c14155490 con 0x7f1c14152700 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.753+0000 7f1c196d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14155770 con 0x7f1c140c8e00 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c137fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1c1406d910 0x7f1c1415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55972/0 (socket says 172.21.15.78:55972) 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c137fe640 1 -- 172.21.15.78:0/2547086912 learned_addr learned my addr 172.21.15.78:0/2547086912 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22468486 0 0) 0x7f1c141551f0 con 0x7f1c1406d910 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bfc003520 con 0x7f1c1406d910 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c127fc640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 0x7f1c1406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555170631 0 0) 0x7f1bfc003520 con 0x7f1c1406d910 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 msgr2=0x7f1c1406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 0x7f1c1406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 >> v1:172.21.15.150:6789/0 conn(0x7f1c14152700 legacy=0x7f1c140c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14155a50 con 0x7f1c1406d910 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1c0c0019f0 con 0x7f1c1406d910 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 >> v1:172.21.15.78:6789/0 conn(0x7f1c1406d910 legacy=0x7f1c1415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bfc0039a0 0x7f1bfc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 0x7f1bfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1bfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14155770 con 0x7f1bfc004d00 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc00bab0 con 0x7f1bfc0039a0 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc00bc70 con 0x7f1c140c8e00 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c137fe640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1bfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c12ffd640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bfc0039a0 0x7f1bfc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c127fc640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 0x7f1bfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c137fe640 1 -- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 msgr2=0x7f1bfc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.754+0000 7f1c137fe640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c140c8e00 0x7f1bfc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c137fe640 1 -- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bfc0039a0 msgr2=0x7f1bfc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c137fe640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bfc0039a0 0x7f1bfc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c137fe640 1 -- 172.21.15.78:0/2547086912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc003520 con 0x7f1bfc004d00 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c137fe640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1bfc005180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00a820 tx=0x7f1bfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c9b1c4a047fcaf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0072d0 con 0x7f1bfc004d00 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c02f070 con 0x7f1bfc004d00 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.755+0000 7f1c11ffb640 1 -- 172.21.15.78:0/2547086912 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0076b0 con 0x7f1bfc004d00 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c196d4640 1 -- 172.21.15.78:0/2547086912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 msgr2=0x7f1bfc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2547086912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1bfc005180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00a820 tx=0x7f1bfc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c137fe640 1 -- 172.21.15.78:0/2547086912 reap_dead start 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c196d4640 1 -- 172.21.15.78:0/2547086912 shutdown_connections 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c196d4640 1 -- 172.21.15.78:0/2547086912 >> 172.21.15.78:0/2547086912 conn(0x7f1c140cac90 msgr2=0x7f1c140cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c196d4640 1 -- 172.21.15.78:0/2547086912 shutdown_connections 2024-02-19T14:52:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.756+0000 7f1c196d4640 1 -- 172.21.15.78:0/2547086912 wait complete. 2024-02-19T14:52:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.757+0000 7f1c196d4640 1 Processor -- start 2024-02-19T14:52:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.757+0000 7f1c196d4640 1 -- start start 2024-02-19T14:52:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c196d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1c141edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c196d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c141ee290 0x7f1c14148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c196d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c14148b50 0x7f1c140c2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c196d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140c0660 con 0x7f1bfc004d00 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c196d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140c3b40 con 0x7f1c14148b50 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c196d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c141eea40 con 0x7f1c141ee290 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1c141edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1c141edd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37316/0 (socket says 172.21.15.78:37316) 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 -- 172.21.15.78:0/2720604757 learned_addr learned my addr 172.21.15.78:0/2720604757 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c127fc640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c141ee290 0x7f1c14148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c137fe640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c14148b50 0x7f1c140c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 -- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c141ee290 msgr2=0x7f1c14148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c141ee290 0x7f1c14148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 -- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c14148b50 msgr2=0x7f1c140c2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c14148b50 0x7f1c140c2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 -- 172.21.15.78:0/2720604757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140c0920 con 0x7f1bfc004d00 2024-02-19T14:52:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.758+0000 7f1c12ffd640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1c141edd50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400f820 tx=0x7f1c0400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.759+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c040050d0 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.759+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c04005230 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.759+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c140c0c00 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.759+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400d730 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.759+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c141ee710 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.759+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c14074990 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.760+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1c0400d890 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.760+0000 7f1bfb7fe640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0006b810 0x7f1c0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.760+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1c04089f10 con 0x7f1bfc004d00 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.760+0000 7f1c127fc640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0006b810 0x7f1c0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.761+0000 7f1c127fc640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0006b810 0x7f1c0006dcd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c140c1080 tx=0x7f1c08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:55.765+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1c04054b80 con 0x7f1bfc004d00 2024-02-19T14:52:56.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.002+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1c140c8770 con 0x7f1bfc004d00 2024-02-19T14:52:56.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.004+0000 7f1bfb7fe640 1 -- 172.21.15.78:0/2720604757 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1c04058240 con 0x7f1bfc004d00 2024-02-19T14:52:56.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0006b810 msgr2=0x7f1c0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0006b810 0x7f1c0006dcd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1c140c1080 tx=0x7f1c08008040 comp rx=0 tx=0).stop 2024-02-19T14:52:56.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 msgr2=0x7f1c141edd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1c141edd50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400f820 tx=0x7f1c0400fce0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 shutdown_connections 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1c0006b810 0x7f1c0006dcd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c14148b50 0x7f1c140c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c141ee290 0x7f1c14148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 --2- 172.21.15.78:0/2720604757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bfc004d00 0x7f1c141edd50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.006+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 >> 172.21.15.78:0/2720604757 conn(0x7f1c140cac90 msgr2=0x7f1c1414fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.007+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 shutdown_connections 2024-02-19T14:52:56.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.007+0000 7f1c196d4640 1 -- 172.21.15.78:0/2720604757 wait complete. 2024-02-19T14:52:56.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-02-19T14:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.118+0000 7f232da4e640 1 Processor -- start 2024-02-19T14:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f232da4e640 1 -- start start 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f232da4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23280c42e0 0x7f2328158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f232da4e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23280c34a0 con 0x7f23280c2f50 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f232da4e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232806c8e0 con 0x7f23280c8500 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f232da4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232806c090 con 0x7f23280c42e0 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f23277fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f23280c2f50 0x7f232815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55982/0 (socket says 172.21.15.78:55982) 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.119+0000 7f23277fe640 1 -- 172.21.15.78:0/3845638698 learned_addr learned my addr 172.21.15.78:0/3845638698 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23280c42e0 0x7f2328158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099501205 0 0) 0x7f23280c34a0 con 0x7f23280c2f50 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2310003520 con 0x7f23280c2f50 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411796914 0 0) 0x7f232806c8e0 con 0x7f23280c8500 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23280c34a0 con 0x7f23280c8500 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499612972 0 0) 0x7f2310003520 con 0x7f23280c2f50 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23280c42e0 msgr2=0x7f2328158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23280c42e0 0x7f2328158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 >> v1:172.21.15.150:6789/0 conn(0x7f23280c8500 legacy=0x7f2328158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232806cbc0 con 0x7f23280c2f50 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f23200019f0 con 0x7f23280c2f50 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23280c42e0 0x7f2328158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 >> v1:172.21.15.78:6789/0 conn(0x7f23280c2f50 legacy=0x7f232815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2310003b30 0x7f2310003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23100047c0 0x7f2310004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2310005180 0x7f231000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231000bec0 con 0x7f23100047c0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231000c000 con 0x7f2310003b30 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231000c170 con 0x7f2310005180 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23100047c0 0x7f2310004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f2326ffd640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2310003b30 0x7f2310003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23277fe640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2310005180 0x7f231000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 -- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2310005180 msgr2=0x7f231000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2310005180 0x7f231000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 -- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2310003b30 msgr2=0x7f2310003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2310003b30 0x7f2310003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23267fc640 1 -- 172.21.15.78:0/3845638698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c009400 con 0x7f23100047c0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.120+0000 7f23277fe640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2310005180 0x7f231000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.121+0000 7f23267fc640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23100047c0 0x7f2310004c40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f231c009c90 tx=0x7f23100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fcd58b845ccb31 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.121+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c02c8f0 con 0x7f23100047c0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.121+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c02ca50 con 0x7f23100047c0 2024-02-19T14:52:56.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.121+0000 7f2325ffb640 1 -- 172.21.15.78:0/3845638698 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c02ce00 con 0x7f23100047c0 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f232da4e640 1 -- 172.21.15.78:0/3845638698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23100047c0 msgr2=0x7f2310004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f232da4e640 1 --2- 172.21.15.78:0/3845638698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23100047c0 0x7f2310004c40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f231c009c90 tx=0x7f23100057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f23277fe640 1 -- 172.21.15.78:0/3845638698 reap_dead start 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f232da4e640 1 -- 172.21.15.78:0/3845638698 shutdown_connections 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f232da4e640 1 -- 172.21.15.78:0/3845638698 >> 172.21.15.78:0/3845638698 conn(0x7f23280aa300 msgr2=0x7f23280aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f232da4e640 1 -- 172.21.15.78:0/3845638698 shutdown_connections 2024-02-19T14:52:56.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.122+0000 7f232da4e640 1 -- 172.21.15.78:0/3845638698 wait complete. 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.123+0000 7f232da4e640 1 Processor -- start 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.123+0000 7f232da4e640 1 -- start start 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f232da4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23100047c0 0x7f232814ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f232da4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 0x7f232814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f232da4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2328158890 0x7f2328158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f232da4e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328071500 con 0x7f2328156d60 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f232da4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328156680 con 0x7f2328158890 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f232da4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2328156b10 con 0x7f23100047c0 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 0x7f232814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 0x7f232814b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37338/0 (socket says 172.21.15.78:37338) 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 -- 172.21.15.78:0/2392078280 learned_addr learned my addr 172.21.15.78:0/2392078280 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f2326ffd640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23100047c0 0x7f232814ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23277fe640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2328158890 0x7f2328158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 -- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23100047c0 msgr2=0x7f232814ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23100047c0 0x7f232814ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 -- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2328158890 msgr2=0x7f2328158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2328158890 0x7f2328158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 -- 172.21.15.78:0/2392078280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232814d270 con 0x7f2328156d60 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f2326ffd640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23100047c0 0x7f232814ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.124+0000 7f23267fc640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 0x7f232814b170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f231c037040 tx=0x7f231c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.125+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c006e30 con 0x7f2328156d60 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.125+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c0349e0 con 0x7f2328156d60 2024-02-19T14:52:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.125+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2328071220 con 0x7f2328156d60 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.125+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c034d70 con 0x7f2328156d60 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.125+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232814f410 con 0x7f2328156d60 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.126+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2328071c20 con 0x7f2328156d60 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.126+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f231c040450 con 0x7f2328156d60 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.126+0000 7f230ffff640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f231406b840 0x7f231406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.126+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f231c0abf80 con 0x7f2328156d60 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.126+0000 7f2326ffd640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f231406b840 0x7f231406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.127+0000 7f2326ffd640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f231406b840 0x7f231406dd00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2318000c00 tx=0x7f2318009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.131+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f231c0dfdc0 con 0x7f2328156d60 2024-02-19T14:52:56.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.371+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f23281567f0 con 0x7f2328156d60 2024-02-19T14:52:56.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.373+0000 7f230ffff640 1 -- 172.21.15.78:0/2392078280 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f231c076d10 con 0x7f2328156d60 2024-02-19T14:52:56.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f231406b840 msgr2=0x7f231406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f231406b840 0x7f231406dd00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2318000c00 tx=0x7f2318009040 comp rx=0 tx=0).stop 2024-02-19T14:52:56.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 msgr2=0x7f232814b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 0x7f232814b170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f231c037040 tx=0x7f231c004730 comp rx=0 tx=0).stop 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 shutdown_connections 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f231406b840 0x7f231406dd00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2328156d60 0x7f232814b170 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2328158890 0x7f2328158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 --2- 172.21.15.78:0/2392078280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23100047c0 0x7f232814ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.375+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 >> 172.21.15.78:0/2392078280 conn(0x7f23280aa300 msgr2=0x7f23280c76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.376+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 shutdown_connections 2024-02-19T14:52:56.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.376+0000 7f232da4e640 1 -- 172.21.15.78:0/2392078280 wait complete. 2024-02-19T14:52:56.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-02-19T14:52:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.487+0000 7f8d5149a640 1 Processor -- start 2024-02-19T14:52:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.487+0000 7f8d5149a640 1 -- start start 2024-02-19T14:52:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d5149a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d4c152700 0x7f8d4c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d5149a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 0x7f8d4c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d5149a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4c1551f0 con 0x7f8d4c0c8e00 2024-02-19T14:52:56.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d5149a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4c155490 con 0x7f8d4c152700 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d5149a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c155770 con 0x7f8d4c0c5730 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d4b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8d4c0c8e00 0x7f8d4c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55990/0 (socket says 172.21.15.78:55990) 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d4b7fe640 1 -- 172.21.15.78:0/3888709432 learned_addr learned my addr 172.21.15.78:0/3888709432 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289331895 0 0) 0x7f8d4c1551f0 con 0x7f8d4c0c8e00 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d4affd640 1 --2- 172.21.15.78:0/3888709432 >> v2:172.21.15.150:3300/0 conn(0x7f8d4c152700 0x7f8d4c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 0x7f8d4c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d34003520 con 0x7f8d4c0c8e00 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2203966734 0 0) 0x7f8d34003520 con 0x7f8d4c0c8e00 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 msgr2=0x7f8d4c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 0x7f8d4c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 >> v2:172.21.15.150:3300/0 conn(0x7f8d4c152700 msgr2=0x7f8d4c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 --2- 172.21.15.78:0/3888709432 >> v2:172.21.15.150:3300/0 conn(0x7f8d4c152700 0x7f8d4c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c155a50 con 0x7f8d4c0c8e00 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.488+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f8d440019f0 con 0x7f8d4c0c8e00 2024-02-19T14:52:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 >> v1:172.21.15.78:6789/0 conn(0x7f8d4c0c8e00 legacy=0x7f8d4c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d340047c0 0x7f8d34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 0x7f8d34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c155490 con 0x7f8d34003970 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c155770 con 0x7f8d340047c0 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3400b980 con 0x7f8d4c0c5730 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4affd640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d340047c0 0x7f8d34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4b7fe640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 0x7f8d34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4affd640 1 -- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 msgr2=0x7f8d34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4affd640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c0c5730 0x7f8d34005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4affd640 1 -- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d340047c0 msgr2=0x7f8d34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4affd640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d340047c0 0x7f8d34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4affd640 1 -- 172.21.15.78:0/3888709432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34003520 con 0x7f8d34003970 2024-02-19T14:52:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.489+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d340047c0 0x7f8d34004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d4affd640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d34003d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009ac0 tx=0x7f8d340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482e892b2da5f811 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c02c710 con 0x7f8d34003970 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c02c870 con 0x7f8d34003970 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d49ffb640 1 -- 172.21.15.78:0/3888709432 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c02cc00 con 0x7f8d34003970 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d5149a640 1 -- 172.21.15.78:0/3888709432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 msgr2=0x7f8d34003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d5149a640 1 --2- 172.21.15.78:0/3888709432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d34003d90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009ac0 tx=0x7f8d340057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d4b7fe640 1 -- 172.21.15.78:0/3888709432 reap_dead start 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d5149a640 1 -- 172.21.15.78:0/3888709432 shutdown_connections 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d5149a640 1 -- 172.21.15.78:0/3888709432 >> 172.21.15.78:0/3888709432 conn(0x7f8d4c0cac90 msgr2=0x7f8d4c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d5149a640 1 -- 172.21.15.78:0/3888709432 shutdown_connections 2024-02-19T14:52:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.490+0000 7f8d5149a640 1 -- 172.21.15.78:0/3888709432 wait complete. 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 Processor -- start 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 -- start start 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d4c1ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d4c0ab9a0 0x7f8d4c1ee580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c1eeac0 0x7f8d4c148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0beee0 con 0x7f8d34003970 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c1ec7b0 con 0x7f8d4c0ab9a0 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d5149a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c1ef060 con 0x7f8d4c1eeac0 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d4affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d4c1ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d4affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d4c1ec270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37360/0 (socket says 172.21.15.78:37360) 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.492+0000 7f8d4affd640 1 -- 172.21.15.78:0/4129928506 learned_addr learned my addr 172.21.15.78:0/4129928506 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4b7fe640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c1eeac0 0x7f8d4c148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4affd640 1 -- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c1eeac0 msgr2=0x7f8d4c148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d4c0ab9a0 0x7f8d4c1ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4affd640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c1eeac0 0x7f8d4c148b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4affd640 1 -- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d4c0ab9a0 msgr2=0x7f8d4c1ee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4affd640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d4c0ab9a0 0x7f8d4c1ee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4affd640 1 -- 172.21.15.78:0/4129928506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c159460 con 0x7f8d34003970 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d4c0ab9a0 0x7f8d4c1ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d4affd640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d4c1ec270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c037040 tx=0x7f8d3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c003810 con 0x7f8d34003970 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c003970 con 0x7f8d34003970 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c0ab1a0 con 0x7f8d34003970 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c003d20 con 0x7f8d34003970 2024-02-19T14:52:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.493+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c0ad0d0 con 0x7f8d34003970 2024-02-19T14:52:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.494+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c1eeea0 con 0x7f8d34003970 2024-02-19T14:52:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.494+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8d3c040450 con 0x7f8d34003970 2024-02-19T14:52:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.495+0000 7f8d337fe640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3806b840 0x7f8d3806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.495+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8d3c0abf00 con 0x7f8d34003970 2024-02-19T14:52:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.495+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3806b840 0x7f8d3806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.495+0000 7f8d4a7fc640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3806b840 0x7f8d3806dd00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c1e8c40 tx=0x7f8d40008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.499+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8d3c076ae0 con 0x7f8d34003970 2024-02-19T14:52:56.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.736+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f8d4c074990 con 0x7f8d34003970 2024-02-19T14:52:56.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.738+0000 7f8d337fe640 1 -- 172.21.15.78:0/4129928506 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8d3c07a1a0 con 0x7f8d34003970 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3806b840 msgr2=0x7f8d3806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3806b840 0x7f8d3806dd00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c1e8c40 tx=0x7f8d40008040 comp rx=0 tx=0).stop 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 msgr2=0x7f8d4c1ec270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d4c1ec270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c037040 tx=0x7f8d3c004730 comp rx=0 tx=0).stop 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 shutdown_connections 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3806b840 0x7f8d3806dd00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d4c1eeac0 0x7f8d4c148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d4c0ab9a0 0x7f8d4c1ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 --2- 172.21.15.78:0/4129928506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d34003970 0x7f8d4c1ec270 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 >> 172.21.15.78:0/4129928506 conn(0x7f8d4c0cac90 msgr2=0x7f8d4c14ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:56.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 shutdown_connections 2024-02-19T14:52:56.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.740+0000 7f8d5149a640 1 -- 172.21.15.78:0/4129928506 wait complete. 2024-02-19T14:52:56.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-02-19T14:52:56.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.851+0000 7f3d1aeae640 1 Processor -- start 2024-02-19T14:52:56.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.851+0000 7f3d1aeae640 1 -- start start 2024-02-19T14:52:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d1aeae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 0x7f3d14071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d1aeae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1406fe40 con 0x7f3d140713a0 2024-02-19T14:52:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d1aeae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1406f880 con 0x7f3d140c8b60 2024-02-19T14:52:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d1aeae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d140c32c0 con 0x7f3d140c29e0 2024-02-19T14:52:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d18c4a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3d140713a0 0x7f3d140c64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55998/0 (socket says 172.21.15.78:55998) 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d18c4a640 1 -- 172.21.15.78:0/3232314212 learned_addr learned my addr 172.21.15.78:0/3232314212 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158171622 0 0) 0x7f3d1406fe40 con 0x7f3d140713a0 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d04003520 con 0x7f3d140713a0 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d13fff640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 0x7f3d14071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941255314 0 0) 0x7f3d04003520 con 0x7f3d140713a0 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 msgr2=0x7f3d14071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 0x7f3d14071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 >> v1:172.21.15.150:6789/0 conn(0x7f3d140c8b60 legacy=0x7f3d141536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1406fb60 con 0x7f3d140713a0 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3d08001670 con 0x7f3d140713a0 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 >> v1:172.21.15.78:6789/0 conn(0x7f3d140713a0 legacy=0x7f3d140c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040039a0 0x7f3d04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d040047c0 0x7f3d04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 0x7f3d04005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0400bb10 con 0x7f3d040047c0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0400bc50 con 0x7f3d040039a0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0400be10 con 0x7f3d140c29e0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d13fff640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d040047c0 0x7f3d04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.852+0000 7f3d18c4a640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040039a0 0x7f3d04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d1944b640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 0x7f3d04005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d13fff640 1 -- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 msgr2=0x7f3d04005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d13fff640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d140c29e0 0x7f3d04005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d13fff640 1 -- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040039a0 msgr2=0x7f3d04003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d13fff640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040039a0 0x7f3d04003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d13fff640 1 -- 172.21.15.78:0/3232314212 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d04003520 con 0x7f3d040047c0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d18c4a640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040039a0 0x7f3d04003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d13fff640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d040047c0 0x7f3d04004c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009a90 tx=0x7f3d040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bdae4c2d0957df server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.853+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0002c680 con 0x7f3d040047c0 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0002c7e0 con 0x7f3d040047c0 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d137fe640 1 -- 172.21.15.78:0/3232314212 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0002cb70 con 0x7f3d040047c0 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1aeae640 1 -- 172.21.15.78:0/3232314212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d040047c0 msgr2=0x7f3d04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/3232314212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d040047c0 0x7f3d04004c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009a90 tx=0x7f3d040057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1944b640 1 -- 172.21.15.78:0/3232314212 reap_dead start 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1aeae640 1 -- 172.21.15.78:0/3232314212 shutdown_connections 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1aeae640 1 -- 172.21.15.78:0/3232314212 >> 172.21.15.78:0/3232314212 conn(0x7f3d140aa300 msgr2=0x7f3d140aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1aeae640 1 -- 172.21.15.78:0/3232314212 shutdown_connections 2024-02-19T14:52:56.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.854+0000 7f3d1aeae640 1 -- 172.21.15.78:0/3232314212 wait complete. 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.855+0000 7f3d1aeae640 1 Processor -- start 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.855+0000 7f3d1aeae640 1 -- start start 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d1aeae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040047c0 0x7f3d14154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d1aeae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 0x7f3d140c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d1aeae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d141eb330 0x7f3d1414ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d1aeae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d14156780 con 0x7f3d141ee9a0 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d1aeae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d141eaea0 con 0x7f3d040047c0 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d1aeae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1406d820 con 0x7f3d141eb330 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 0x7f3d140c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 0x7f3d140c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37380/0 (socket says 172.21.15.78:37380) 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 -- 172.21.15.78:0/1820336638 learned_addr learned my addr 172.21.15.78:0/1820336638 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 -- 172.21.15.78:0/1820336638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d141eb330 msgr2=0x7f3d1414ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d141eb330 0x7f3d1414ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 -- 172.21.15.78:0/1820336638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040047c0 msgr2=0x7f3d14154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040047c0 0x7f3d14154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 -- 172.21.15.78:0/1820336638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1406db00 con 0x7f3d141ee9a0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.856+0000 7f3d13fff640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 0x7f3d140c1bf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3d00000c00 tx=0x7f3d00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.857+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00003810 con 0x7f3d141ee9a0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.857+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d00003970 con 0x7f3d141ee9a0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.857+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1406dde0 con 0x7f3d141ee9a0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.857+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00003d20 con 0x7f3d141ee9a0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.857+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d141eee20 con 0x7f3d141ee9a0 2024-02-19T14:52:56.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.858+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d140aac10 con 0x7f3d141ee9a0 2024-02-19T14:52:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.858+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3d00034440 con 0x7f3d141ee9a0 2024-02-19T14:52:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.858+0000 7f3d117fa640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3cf806b840 0x7f3cf806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.859+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3d000ac180 con 0x7f3d141ee9a0 2024-02-19T14:52:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.859+0000 7f3d18c4a640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3cf806b840 0x7f3cf806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.859+0000 7f3d18c4a640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3cf806b840 0x7f3cf806dd00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3d08001bc0 tx=0x7f3d08000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:56.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:56.863+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d00076fc0 con 0x7f3d141ee9a0 2024-02-19T14:52:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.098+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f3d141eb050 con 0x7f3d141ee9a0 2024-02-19T14:52:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.100+0000 7f3d117fa640 1 -- 172.21.15.78:0/1820336638 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3d141eb050 con 0x7f3d141ee9a0 2024-02-19T14:52:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.102+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3cf806b840 msgr2=0x7f3cf806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.102+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3cf806b840 0x7f3cf806dd00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3d08001bc0 tx=0x7f3d08000ed0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.102+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 msgr2=0x7f3d140c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.102+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 0x7f3d140c1bf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3d00000c00 tx=0x7f3d00004730 comp rx=0 tx=0).stop 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 shutdown_connections 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3cf806b840 0x7f3cf806dd00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d141ee9a0 0x7f3d140c1bf0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d141eb330 0x7f3d1414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 --2- 172.21.15.78:0/1820336638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d040047c0 0x7f3d14154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 >> 172.21.15.78:0/1820336638 conn(0x7f3d140aa300 msgr2=0x7f3d14074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 shutdown_connections 2024-02-19T14:52:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.103+0000 7f3d1aeae640 1 -- 172.21.15.78:0/1820336638 wait complete. 2024-02-19T14:52:57.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-02-19T14:52:57.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.214+0000 7fb4b42ae640 1 Processor -- start 2024-02-19T14:52:57.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.214+0000 7fb4b42ae640 1 -- start start 2024-02-19T14:52:57.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b42ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 0x7fb4ac06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b42ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ac1551f0 con 0x7fb4ac06d910 2024-02-19T14:52:57.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b42ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ac155490 con 0x7fb4ac152700 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b42ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac155770 con 0x7fb4ac0c8e00 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b284b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb4ac06d910 0x7fb4ac15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56006/0 (socket says 172.21.15.78:56006) 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b284b640 1 -- 172.21.15.78:0/3385008014 learned_addr learned my addr 172.21.15.78:0/3385008014 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156578819 0 0) 0x7fb4ac1551f0 con 0x7fb4ac06d910 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b1849640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 0x7fb4ac06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.215+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb498003520 con 0x7fb4ac06d910 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567372642 0 0) 0x7fb498003520 con 0x7fb4ac06d910 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 msgr2=0x7fb4ac06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 0x7fb4ac06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 >> v1:172.21.15.150:6789/0 conn(0x7fb4ac152700 legacy=0x7fb4ac0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac155a50 con 0x7fb4ac06d910 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb4a4001670 con 0x7fb4ac06d910 2024-02-19T14:52:57.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 >> v1:172.21.15.78:6789/0 conn(0x7fb4ac06d910 legacy=0x7fb4ac15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 0x7fb4980039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb498004d00 0x7fb498005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac155770 con 0x7fb498003fd0 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49800bbc0 con 0x7fb498004d00 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49800bd80 con 0x7fb4ac0c8e00 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1849640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b204a640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 0x7fb4980039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b284b640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb498004d00 0x7fb498005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1849640 1 -- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 msgr2=0x7fb4980039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1849640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0c8e00 0x7fb4980039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1849640 1 -- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb498004d00 msgr2=0x7fb498005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1849640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb498004d00 0x7fb498005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b1849640 1 -- 172.21.15.78:0/3385008014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb498003520 con 0x7fb498003fd0 2024-02-19T14:52:57.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.216+0000 7fb4b284b640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb498004d00 0x7fb498005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b1849640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4980047c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00ba30 tx=0x7fb4980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fc2d773c2cc66a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c032640 con 0x7fb498003fd0 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c0327a0 con 0x7fb498003fd0 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b1048640 1 -- 172.21.15.78:0/3385008014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c032b30 con 0x7fb498003fd0 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b42ae640 1 -- 172.21.15.78:0/3385008014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 msgr2=0x7fb4980047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/3385008014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4980047c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00ba30 tx=0x7fb4980057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.217+0000 7fb4b284b640 1 -- 172.21.15.78:0/3385008014 reap_dead start 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.218+0000 7fb4b42ae640 1 -- 172.21.15.78:0/3385008014 shutdown_connections 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.218+0000 7fb4b42ae640 1 -- 172.21.15.78:0/3385008014 >> 172.21.15.78:0/3385008014 conn(0x7fb4ac0cac90 msgr2=0x7fb4ac0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.218+0000 7fb4b42ae640 1 -- 172.21.15.78:0/3385008014 shutdown_connections 2024-02-19T14:52:57.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.218+0000 7fb4b42ae640 1 -- 172.21.15.78:0/3385008014 wait complete. 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 Processor -- start 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 -- start start 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4ac154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4ac154a60 0x7fb4ac0aa990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0aaed0 0x7fb4ac148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac1594c0 con 0x7fb498003fd0 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac1eddf0 con 0x7fb4ac154a60 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b42ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac1ee0d0 con 0x7fb4ac0aaed0 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b204a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4ac154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b204a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4ac154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37394/0 (socket says 172.21.15.78:37394) 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.219+0000 7fb4b204a640 1 -- 172.21.15.78:0/1111553884 learned_addr learned my addr 172.21.15.78:0/1111553884 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b204a640 1 -- 172.21.15.78:0/1111553884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0aaed0 msgr2=0x7fb4ac148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b204a640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0aaed0 0x7fb4ac148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b204a640 1 -- 172.21.15.78:0/1111553884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4ac154a60 msgr2=0x7fb4ac0aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b204a640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4ac154a60 0x7fb4ac0aa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b204a640 1 -- 172.21.15.78:0/1111553884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac1ee3b0 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b204a640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4ac154520 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800e8c0 tx=0x7fb4a800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8003b80 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a8003ce0 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac1ee690 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a80052f0 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.220+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac157950 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.221+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ac074990 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.221+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb4a8005450 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.222+0000 7fb4a2ffd640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb49006b840 0x7fb49006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.222+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb4a808a140 con 0x7fb498003fd0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.222+0000 7fb4b1849640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb49006b840 0x7fb49006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.222+0000 7fb4b1849640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb49006b840 0x7fb49006dd00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00ba00 tx=0x7fb49c032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.226+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4a8054db0 con 0x7fb498003fd0 2024-02-19T14:52:57.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.463+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fb4ac0c8770 con 0x7fb498003fd0 2024-02-19T14:52:57.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.465+0000 7fb4a2ffd640 1 -- 172.21.15.78:0/1111553884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb4a8058470 con 0x7fb498003fd0 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb49006b840 msgr2=0x7fb49006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb49006b840 0x7fb49006dd00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00ba00 tx=0x7fb49c032000 comp rx=0 tx=0).stop 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 msgr2=0x7fb4ac154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4ac154520 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800e8c0 tx=0x7fb4a800ed80 comp rx=0 tx=0).stop 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 shutdown_connections 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb49006b840 0x7fb49006dd00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4ac0aaed0 0x7fb4ac148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4ac154a60 0x7fb4ac0aa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 --2- 172.21.15.78:0/1111553884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb498003fd0 0x7fb4ac154520 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 >> 172.21.15.78:0/1111553884 conn(0x7fb4ac0cac90 msgr2=0x7fb4ac0c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 shutdown_connections 2024-02-19T14:52:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.467+0000 7fb4b42ae640 1 -- 172.21.15.78:0/1111553884 wait complete. 2024-02-19T14:52:57.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-02-19T14:52:57.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.579+0000 7fd4eb80e640 1 Processor -- start 2024-02-19T14:52:57.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.579+0000 7fd4eb80e640 1 -- start start 2024-02-19T14:52:57.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.579+0000 7fd4eb80e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4e4152480 0x7fd4e41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4eb80e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4eb80e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd4e406d350 0x7fd4e40c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4eb80e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e4155ef0 con 0x7fd4e406d350 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4eb80e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e406dec0 con 0x7fd4e4152480 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4eb80e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40c75c0 con 0x7fd4e40c7070 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd4e406d350 0x7fd4e40c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd4e406d350 0x7fd4e40c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46872/0 (socket says 172.21.15.78:46872) 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 -- 172.21.15.78:0/3782298657 learned_addr learned my addr 172.21.15.78:0/3782298657 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 -- 172.21.15.78:0/3782298657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 msgr2=0x7fd4e406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 --2- 172.21.15.78:0/3782298657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 -- 172.21.15.78:0/3782298657 >> v2:172.21.15.150:3300/0 conn(0x7fd4e4152480 msgr2=0x7fd4e41559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 --2- 172.21.15.78:0/3782298657 >> v2:172.21.15.150:3300/0 conn(0x7fd4e4152480 0x7fd4e41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.580+0000 7fd4e9dab640 1 -- 172.21.15.78:0/3782298657 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40c78a0 con 0x7fd4e406d350 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4e9dab640 1 --2- 172.21.15.78:0/3782298657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e406d350 0x7fd4e40c5220 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e50 tx=0x7fd4dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c14475fbe648945 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4dbfff640 1 -- 172.21.15.78:0/3782298657 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc02ca70 con 0x7fd4e406d350 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4dbfff640 1 -- 172.21.15.78:0/3782298657 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc02cbd0 con 0x7fd4e406d350 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4dbfff640 1 -- 172.21.15.78:0/3782298657 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc0343f0 con 0x7fd4e406d350 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 -- 172.21.15.78:0/3782298657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e406d350 msgr2=0x7fd4e40c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/3782298657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e406d350 0x7fd4e40c5220 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e50 tx=0x7fd4dc02bed0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 -- 172.21.15.78:0/3782298657 shutdown_connections 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/3782298657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e406d350 0x7fd4e40c5220 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/3782298657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/3782298657 >> v2:172.21.15.150:3300/0 conn(0x7fd4e4152480 0x7fd4e41559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 -- 172.21.15.78:0/3782298657 >> 172.21.15.78:0/3782298657 conn(0x7fd4e40ca8a0 msgr2=0x7fd4e40cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:57.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 -- 172.21.15.78:0/3782298657 shutdown_connections 2024-02-19T14:52:57.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.581+0000 7fd4eb80e640 1 -- 172.21.15.78:0/3782298657 wait complete. 2024-02-19T14:52:57.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.582+0000 7fd4eb80e640 1 Processor -- start 2024-02-19T14:52:57.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 -- start start 2024-02-19T14:52:57.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e40ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 0x7fd4e41ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4e41ea990 0x7fd4e41eea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40add50 con 0x7fd4e4152480 2024-02-19T14:52:57.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41ee0c0 con 0x7fd4e41ea990 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4eb80e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b0430 con 0x7fd4e40c7070 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4e8da9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 0x7fd4e41ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4e8da9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 0x7fd4e41ee550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46874/0 (socket says 172.21.15.78:46874) 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4e8da9640 1 -- 172.21.15.78:0/677496504 learned_addr learned my addr 172.21.15.78:0/677496504 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.583+0000 7fd4e95aa640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e40ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e9dab640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4e41ea990 0x7fd4e41eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e8da9640 1 -- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 msgr2=0x7fd4e40ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e8da9640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e40ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e8da9640 1 -- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4e41ea990 msgr2=0x7fd4e41eea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e8da9640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4e41ea990 0x7fd4e41eea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e8da9640 1 -- 172.21.15.78:0/677496504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc009b30 con 0x7fd4e4152480 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e9dab640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4e41ea990 0x7fd4e41eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4e8da9640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 0x7fd4e41ee550 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b480 tx=0x7fd4d400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4011830 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d4011990 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e40ada70 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4011d40 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.584+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e40acb60 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.585+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e40c17c0 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.585+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd4d401a400 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.586+0000 7fd4d9ffb640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4c806b840 0x7fd4c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.586+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd4d408a010 con 0x7fd4e4152480 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.586+0000 7fd4e95aa640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4c806b840 0x7fd4c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.586+0000 7fd4e95aa640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4c806b840 0x7fd4c806dd00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd4e41553a0 tx=0x7fd4e0009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.590+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4d4054d30 con 0x7fd4e4152480 2024-02-19T14:52:57.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.825+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fd4e40aff90 con 0x7fd4e4152480 2024-02-19T14:52:57.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.827+0000 7fd4d9ffb640 1 -- 172.21.15.78:0/677496504 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd4d40150a0 con 0x7fd4e4152480 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4c806b840 msgr2=0x7fd4c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4c806b840 0x7fd4c806dd00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd4e41553a0 tx=0x7fd4e0009110 comp rx=0 tx=0).stop 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 msgr2=0x7fd4e41ee550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 0x7fd4e41ee550 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b480 tx=0x7fd4d400b880 comp rx=0 tx=0).stop 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 shutdown_connections 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4c806b840 0x7fd4c806dd00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4e4152480 0x7fd4e41ee550 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4e41ea990 0x7fd4e41eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 --2- 172.21.15.78:0/677496504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4e40c7070 0x7fd4e40ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 >> 172.21.15.78:0/677496504 conn(0x7fd4e40ca8a0 msgr2=0x7fd4e40c87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 shutdown_connections 2024-02-19T14:52:57.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.829+0000 7fd4eb80e640 1 -- 172.21.15.78:0/677496504 wait complete. 2024-02-19T14:52:57.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:52:57.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.940+0000 7f3295965640 1 Processor -- start 2024-02-19T14:52:57.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.940+0000 7f3295965640 1 -- start start 2024-02-19T14:52:57.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f3295965640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32900c8500 0x7f32900c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f3295965640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f32900c2f50 0x7f3290158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f3295965640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32900c34a0 con 0x7f32900c42e0 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f3295965640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329006c090 con 0x7f32900c8500 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f3295965640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329006d6a0 con 0x7f32900c2f50 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32900c8500 0x7f32900c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32900c8500 0x7f32900c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46880/0 (socket says 172.21.15.78:46880) 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 -- 172.21.15.78:0/1775696016 learned_addr learned my addr 172.21.15.78:0/1775696016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328f7fe640 1 --2- 172.21.15.78:0/1775696016 >> v2:172.21.15.150:3300/0 conn(0x7f32900c2f50 0x7f3290158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 -- 172.21.15.78:0/1775696016 >> v2:172.21.15.150:3300/0 conn(0x7f32900c2f50 msgr2=0x7f3290158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 --2- 172.21.15.78:0/1775696016 >> v2:172.21.15.150:3300/0 conn(0x7f32900c2f50 0x7f3290158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 -- 172.21.15.78:0/1775696016 >> v1:172.21.15.150:6789/0 conn(0x7f32900c42e0 legacy=0x7f3290158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.941+0000 7f328effd640 1 -- 172.21.15.78:0/1775696016 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329006c8e0 con 0x7f32900c8500 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.942+0000 7f328effd640 1 --2- 172.21.15.78:0/1775696016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32900c8500 0x7f32900c88e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f32900adea0 tx=0x7f328002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208bf74a0800fe13 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.942+0000 7f328dffb640 1 -- 172.21.15.78:0/1775696016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280031870 con 0x7f32900c8500 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.942+0000 7f328dffb640 1 -- 172.21.15.78:0/1775696016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32800319d0 con 0x7f32900c8500 2024-02-19T14:52:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.942+0000 7f328dffb640 1 -- 172.21.15.78:0/1775696016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3280031d60 con 0x7f32900c8500 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 -- 172.21.15.78:0/1775696016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32900c8500 msgr2=0x7f32900c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 --2- 172.21.15.78:0/1775696016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32900c8500 0x7f32900c88e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f32900adea0 tx=0x7f328002b910 comp rx=0 tx=0).stop 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 -- 172.21.15.78:0/1775696016 shutdown_connections 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 --2- 172.21.15.78:0/1775696016 >> v2:172.21.15.150:3300/0 conn(0x7f32900c2f50 0x7f3290158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 --2- 172.21.15.78:0/1775696016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32900c8500 0x7f32900c88e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 -- 172.21.15.78:0/1775696016 >> 172.21.15.78:0/1775696016 conn(0x7f32900aa300 msgr2=0x7f32900aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 -- 172.21.15.78:0/1775696016 shutdown_connections 2024-02-19T14:52:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.943+0000 7f3295965640 1 -- 172.21.15.78:0/1775696016 wait complete. 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.944+0000 7f3295965640 1 Processor -- start 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.944+0000 7f3295965640 1 -- start start 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32900c2f50 0x7f32901eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32900c8500 0x7f32900c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 0x7f3290070560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3290154be0 con 0x7f329014b830 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3290154ec0 con 0x7f32900c8500 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32901ee470 con 0x7f32900c2f50 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 0x7f3290070560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 0x7f3290070560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46892/0 (socket says 172.21.15.78:46892) 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 -- 172.21.15.78:0/567534514 learned_addr learned my addr 172.21.15.78:0/567534514 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328e7fc640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32900c8500 0x7f32900c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328effd640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32900c2f50 0x7f32901eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 -- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32900c2f50 msgr2=0x7f32901eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32900c2f50 0x7f32901eef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 -- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32900c8500 msgr2=0x7f32900c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32900c8500 0x7f32900c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 -- 172.21.15.78:0/567534514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32800094d0 con 0x7f329014b830 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328e7fc640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32900c8500 0x7f32900c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f328f7fe640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 0x7f3290070560 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3288007e40 tx=0x7f328800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3288014620 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328800ee90 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32880149e0 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.945+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32901ee750 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.946+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32900c5720 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.947+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3290071c20 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.947+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3288014b40 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.947+0000 7f3277fff640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327c06b810 0x7f327c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.948+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f328808a9e0 con 0x7f329014b830 2024-02-19T14:52:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.948+0000 7f328effd640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327c06b810 0x7f327c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.948+0000 7f328effd640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327c06b810 0x7f327c06dcd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3280036000 tx=0x7f328003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:57.952+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f32880555c0 con 0x7f329014b830 2024-02-19T14:52:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.124+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32901eec70 con 0x7f329014b830 2024-02-19T14:52:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.125+0000 7f3277fff640 1 -- 172.21.15.78:0/567534514 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+13152 (secure 0 0 0) 0x7f32901eec70 con 0x7f329014b830 2024-02-19T14:52:58.128 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:58.128 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":34,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:52:55.266167+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327c06b810 msgr2=0x7f327c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327c06b810 0x7f327c06dcd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3280036000 tx=0x7f328003c040 comp rx=0 tx=0).stop 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 msgr2=0x7f3290070560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 0x7f3290070560 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3288007e40 tx=0x7f328800e4b0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 shutdown_connections 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327c06b810 0x7f327c06dcd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f329014b830 0x7f3290070560 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32900c8500 0x7f32900c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 --2- 172.21.15.78:0/567534514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32900c2f50 0x7f32901eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 >> 172.21.15.78:0/567534514 conn(0x7f32900aa300 msgr2=0x7f329014e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 shutdown_connections 2024-02-19T14:52:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.127+0000 7f3295965640 1 -- 172.21.15.78:0/567534514 wait complete. 2024-02-19T14:52:58.141 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:52:58.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:52:58.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 Processor -- start 2024-02-19T14:52:58.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 -- start start 2024-02-19T14:52:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbda8152480 0x7fbda81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda8155ef0 con 0x7fbda806d350 2024-02-19T14:52:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda806dec0 con 0x7fbda8152480 2024-02-19T14:52:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.242+0000 7fbdad69a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda80c75c0 con 0x7fbda80c7070 2024-02-19T14:52:58.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbda8152480 0x7fbda81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda67fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbda806d350 0x7fbda80c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57012/0 (socket says 172.21.15.78:57012) 2024-02-19T14:52:58.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbda8152480 0x7fbda81559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:55806/0 (socket says 172.21.15.78:55806) 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda77fe640 1 -- 172.21.15.78:0/673550187 learned_addr learned my addr 172.21.15.78:0/673550187 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda6ffd640 1 -- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 msgr2=0x7fbda806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda6ffd640 1 --2- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda806ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda6ffd640 1 -- 172.21.15.78:0/673550187 >> v1:172.21.15.150:6789/0 conn(0x7fbda806d350 legacy=0x7fbda80c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda6ffd640 1 -- 172.21.15.78:0/673550187 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80c78a0 con 0x7fbda8152480 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.243+0000 7fbda67fc640 1 --2- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda806ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.244+0000 7fbda6ffd640 1 --2- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda81559b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd980097d0 tx=0x7fbd9802b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91164f865b2730bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.244+0000 7fbda5ffb640 1 -- 172.21.15.78:0/673550187 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98031800 con 0x7fbda8152480 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.244+0000 7fbda5ffb640 1 -- 172.21.15.78:0/673550187 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd98031960 con 0x7fbda8152480 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.244+0000 7fbda5ffb640 1 -- 172.21.15.78:0/673550187 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98031cf0 con 0x7fbda8152480 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 -- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 msgr2=0x7fbda81559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 --2- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda81559b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd980097d0 tx=0x7fbd9802b8a0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 -- 172.21.15.78:0/673550187 shutdown_connections 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 --2- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda806ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 --2- 172.21.15.78:0/673550187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda81559b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 -- 172.21.15.78:0/673550187 >> 172.21.15.78:0/673550187 conn(0x7fbda80ca8a0 msgr2=0x7fbda80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 -- 172.21.15.78:0/673550187 shutdown_connections 2024-02-19T14:52:58.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.245+0000 7fbdad69a640 1 -- 172.21.15.78:0/673550187 wait complete. 2024-02-19T14:52:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.246+0000 7fbdad69a640 1 Processor -- start 2024-02-19T14:52:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.246+0000 7fbdad69a640 1 -- start start 2024-02-19T14:52:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbdad69a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbdad69a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda8157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbdad69a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 0x7fbda81eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbdad69a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda81564d0 con 0x7fbda8158530 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbdad69a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda81567b0 con 0x7fbda8152480 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbdad69a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda8156a90 con 0x7fbda80c7070 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 0x7fbda81eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 0x7fbda81eb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46902/0 (socket says 172.21.15.78:46902) 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 -- 172.21.15.78:0/1564613751 learned_addr learned my addr 172.21.15.78:0/1564613751 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda6ffd640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda67fc640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda8157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 -- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 msgr2=0x7fbda814df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda814df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 -- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 msgr2=0x7fbda8157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda8157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 -- 172.21.15.78:0/1564613751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c009400 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda67fc640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda8157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbda77fe640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 0x7fbda81eb070 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbda000cbc0 tx=0x7fbda000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.247+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000ad40 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.248+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda0012400 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.248+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000aea0 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.248+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd98009400 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.248+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda80681a0 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.249+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda8156610 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.249+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbda0007070 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.249+0000 7fbd8f7fe640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9406b810 0x7fbd9406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.249+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbda0089cd0 con 0x7fbda8158530 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.249+0000 7fbda6ffd640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9406b810 0x7fbd9406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.250+0000 7fbda6ffd640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9406b810 0x7fbd9406dcd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd98036000 tx=0x7fbd9803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.254+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbda8156610 con 0x7fbda8158530 2024-02-19T14:52:58.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.424+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbda8155ef0 con 0x7fbda8158530 2024-02-19T14:52:58.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.425+0000 7fbd8f7fe640 1 -- 172.21.15.78:0/1564613751 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbda8155ef0 con 0x7fbda8158530 2024-02-19T14:52:58.427 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.426+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9406b810 msgr2=0x7fbd9406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9406b810 0x7fbd9406dcd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd98036000 tx=0x7fbd9803c040 comp rx=0 tx=0).stop 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 msgr2=0x7fbda81eb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 0x7fbda81eb070 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbda000cbc0 tx=0x7fbda000a430 comp rx=0 tx=0).stop 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 shutdown_connections 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9406b810 0x7fbd9406dcd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbda8158530 0x7fbda81eb070 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbda8152480 0x7fbda8157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 --2- 172.21.15.78:0/1564613751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbda80c7070 0x7fbda814df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 >> 172.21.15.78:0/1564613751 conn(0x7fbda80ca8a0 msgr2=0x7fbda8150630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 shutdown_connections 2024-02-19T14:52:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.427+0000 7fbdad69a640 1 -- 172.21.15.78:0/1564613751 wait complete. 2024-02-19T14:52:58.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:52:58.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8635983640 1 Processor -- start 2024-02-19T14:52:58.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8635983640 1 -- start start 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8635983640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86300c87a0 0x7f86301559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8635983640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630155ef0 con 0x7f86300c7070 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8635983640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863006dec0 con 0x7f8630152480 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8635983640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f863006d520 con 0x7f86300c87a0 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8627fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f86300c7070 0x7f86300c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46398/0 (socket says 172.21.15.78:46398) 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.539+0000 7f8627fff640 1 -- 172.21.15.78:0/848485680 learned_addr learned my addr 172.21.15.78:0/848485680 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862f7fe640 1 --2- 172.21.15.78:0/848485680 >> v2:172.21.15.150:3300/0 conn(0x7f86300c87a0 0x7f86301559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129605382 0 0) 0x7f8618001110 con 0x7f86300c7070 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c003520 con 0x7f86300c7070 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293985562 0 0) 0x7f863006dec0 con 0x7f8630152480 2024-02-19T14:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8618001110 con 0x7f8630152480 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507425785 0 0) 0x7f861c003520 con 0x7f86300c7070 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 >> v2:172.21.15.150:3300/0 conn(0x7f86300c87a0 msgr2=0x7f86301559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 --2- 172.21.15.78:0/848485680 >> v2:172.21.15.150:3300/0 conn(0x7f86300c87a0 0x7f86301559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 >> v1:172.21.15.150:6789/0 conn(0x7f8630152480 legacy=0x7f863006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300c75c0 con 0x7f86300c7070 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f8618001800 con 0x7f86300c7070 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 >> v1:172.21.15.78:6789/0 conn(0x7f86300c7070 legacy=0x7f86300c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f861c003b00 0x7f861c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c0047c0 0x7f861c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f861c005180 0x7f861c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863006d520 con 0x7f861c003b00 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c00c000 con 0x7f861c005180 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c00c1c0 con 0x7f861c0047c0 2024-02-19T14:52:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862effd640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f861c003b00 0x7f861c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862f7fe640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f861c005180 0x7f861c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f8627fff640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c0047c0 0x7f861c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862effd640 1 -- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c0047c0 msgr2=0x7f861c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862effd640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c0047c0 0x7f861c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862effd640 1 -- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f861c005180 msgr2=0x7f861c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862effd640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f861c005180 0x7f861c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862f7fe640 1 -- 172.21.15.78:0/848485680 reap_dead start 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.540+0000 7f862effd640 1 -- 172.21.15.78:0/848485680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c003520 con 0x7f861c003b00 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.541+0000 7f862effd640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f861c003b00 0x7f861c003f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f86200018d0 tx=0x7f861c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9675ce4dc72687 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.541+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86200052d0 con 0x7f861c003b00 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.541+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620005430 con 0x7f861c003b00 2024-02-19T14:52:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.541+0000 7f862e7fc640 1 -- 172.21.15.78:0/848485680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620003810 con 0x7f861c003b00 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 -- 172.21.15.78:0/848485680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f861c003b00 msgr2=0x7f861c003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f861c003b00 0x7f861c003f00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f86200018d0 tx=0x7f861c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 -- 172.21.15.78:0/848485680 shutdown_connections 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 --2- 172.21.15.78:0/848485680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f861c003b00 0x7f861c003f00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 -- 172.21.15.78:0/848485680 >> 172.21.15.78:0/848485680 conn(0x7f86300ca8a0 msgr2=0x7f86300cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 -- 172.21.15.78:0/848485680 shutdown_connections 2024-02-19T14:52:58.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.542+0000 7f8635983640 1 -- 172.21.15.78:0/848485680 wait complete. 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.543+0000 7f8635983640 1 Processor -- start 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.543+0000 7f8635983640 1 -- start start 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c003b00 0x7f86300ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 0x7f86301574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86300ae4a0 0x7f8630157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86300ad750 con 0x7f86301590a0 2024-02-19T14:52:58.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86300ad9c0 con 0x7f86300ae4a0 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86300adca0 con 0x7f861c003b00 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 0x7f86301574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 0x7f86301574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46926/0 (socket says 172.21.15.78:46926) 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 -- 172.21.15.78:0/1795823000 learned_addr learned my addr 172.21.15.78:0/1795823000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862f7fe640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86300ae4a0 0x7f8630157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862effd640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c003b00 0x7f86300ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 -- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c003b00 msgr2=0x7f86300ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c003b00 0x7f86300ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 -- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86300ae4a0 msgr2=0x7f8630157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86300ae4a0 0x7f8630157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 -- 172.21.15.78:0/1795823000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300adf80 con 0x7f86301590a0 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862f7fe640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86300ae4a0 0x7f8630157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862effd640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c003b00 0x7f86300ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:52:58.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8627fff640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 0x7f86301574c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f861800e870 tx=0x7f861800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861800c8d0 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861800ca30 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86300acd10 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861800cda0 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.544+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8630156c80 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.545+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8630074990 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.546+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f861801d460 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.546+0000 7f862cff9640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861006b810 0x7f861006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.546+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8618089fa0 con 0x7f86301590a0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.546+0000 7f862effd640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861006b810 0x7f861006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.546+0000 7f862effd640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861006b810 0x7f861006dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8620034000 tx=0x7f8620031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.550+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f86180bddc0 con 0x7f86301590a0 2024-02-19T14:52:58.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.721+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8630155ef0 con 0x7f86301590a0 2024-02-19T14:52:58.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.722+0000 7f862cff9640 1 -- 172.21.15.78:0/1795823000 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8630155ef0 con 0x7f86301590a0 2024-02-19T14:52:58.724 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:52:58.724 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:52:58.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:52:58.725 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:52:58.725 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:52:58.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:52:58.725 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:52:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861006b810 msgr2=0x7f861006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861006b810 0x7f861006dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8620034000 tx=0x7f8620031000 comp rx=0 tx=0).stop 2024-02-19T14:52:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 msgr2=0x7f86301574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 0x7f86301574c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f861800e870 tx=0x7f861800ed30 comp rx=0 tx=0).stop 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 shutdown_connections 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861006b810 0x7f861006dcd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86301590a0 0x7f86301574c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86300ae4a0 0x7f8630157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.723+0000 7f8635983640 1 --2- 172.21.15.78:0/1795823000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f861c003b00 0x7f86300ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.724+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 >> 172.21.15.78:0/1795823000 conn(0x7f86300ca8a0 msgr2=0x7f86300c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.724+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 shutdown_connections 2024-02-19T14:52:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.724+0000 7f8635983640 1 -- 172.21.15.78:0/1795823000 wait complete. 2024-02-19T14:52:58.737 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:52:58.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:52:58.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.838+0000 7fcc0dc8f640 1 Processor -- start 2024-02-19T14:52:58.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.838+0000 7fcc0dc8f640 1 -- start start 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc0dc8f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc081524e0 0x7fcc080c8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc0dc8f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc080c8db0 0x7fcc080c5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc0dc8f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc080c4d30 con 0x7fcc081559e0 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc0dc8f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc080c3050 con 0x7fcc081524e0 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc0dc8f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc080c3330 con 0x7fcc080c8db0 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc077fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcc081524e0 0x7fcc080c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc06ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcc081559e0 0x7fcc080c5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46408/0 (socket says 172.21.15.78:46408) 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc06ffd640 1 -- 172.21.15.78:0/526381813 learned_addr learned my addr 172.21.15.78:0/526381813 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc077fe640 1 -- 172.21.15.78:0/526381813 >> v2:172.21.15.150:3300/0 conn(0x7fcc080c8db0 msgr2=0x7fcc080c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc077fe640 1 --2- 172.21.15.78:0/526381813 >> v2:172.21.15.150:3300/0 conn(0x7fcc080c8db0 0x7fcc080c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc077fe640 1 -- 172.21.15.78:0/526381813 >> v1:172.21.15.78:6789/0 conn(0x7fcc081559e0 legacy=0x7fcc080c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc077fe640 1 -- 172.21.15.78:0/526381813 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc080c5e60 con 0x7fcc081524e0 2024-02-19T14:52:58.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.839+0000 7fcc067fc640 1 -- 172.21.15.78:0/526381813 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118642694 0 0) 0x7fcc080c4d30 con 0x7fcc081559e0 2024-02-19T14:52:58.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.840+0000 7fcc077fe640 1 --2- 172.21.15.78:0/526381813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081524e0 0x7fcc080c8870 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcc080c3ff0 tx=0x7fcbf802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da432a2b7607c96 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.840+0000 7fcc067fc640 1 -- 172.21.15.78:0/526381813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf802e070 con 0x7fcc081524e0 2024-02-19T14:52:58.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.840+0000 7fcc067fc640 1 -- 172.21.15.78:0/526381813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf80315a0 con 0x7fcc081524e0 2024-02-19T14:52:58.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.840+0000 7fcc067fc640 1 -- 172.21.15.78:0/526381813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8031930 con 0x7fcc081524e0 2024-02-19T14:52:58.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/526381813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081524e0 msgr2=0x7fcc080c8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/526381813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081524e0 0x7fcc080c8870 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcc080c3ff0 tx=0x7fcbf802b840 comp rx=0 tx=0).stop 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/526381813 shutdown_connections 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/526381813 >> v2:172.21.15.150:3300/0 conn(0x7fcc080c8db0 0x7fcc080c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/526381813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081524e0 0x7fcc080c8870 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/526381813 >> 172.21.15.78:0/526381813 conn(0x7fcc0806b060 msgr2=0x7fcc0806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/526381813 shutdown_connections 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.841+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/526381813 wait complete. 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.842+0000 7fcc0dc8f640 1 Processor -- start 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.842+0000 7fcc0dc8f640 1 -- start start 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc0dc8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc080c8db0 0x7fcc08150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc0dc8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc081524e0 0x7fcc080ac830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc0dc8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 0x7fcc081edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc0dc8f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc081585e0 con 0x7fcc081559e0 2024-02-19T14:52:58.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc0dc8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc081545d0 con 0x7fcc081524e0 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc0dc8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc081eb200 con 0x7fcc080c8db0 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 0x7fcc081edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 0x7fcc081edd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46956/0 (socket says 172.21.15.78:46956) 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 -- 172.21.15.78:0/1784550806 learned_addr learned my addr 172.21.15.78:0/1784550806 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc06ffd640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc081524e0 0x7fcc080ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc077fe640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc080c8db0 0x7fcc08150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 -- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc080c8db0 msgr2=0x7fcc08150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc080c8db0 0x7fcc08150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 -- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc081524e0 msgr2=0x7fcc080ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc081524e0 0x7fcc080ac830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:58.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 -- 172.21.15.78:0/1784550806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf8009400 con 0x7fcc081559e0 2024-02-19T14:52:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcc07fff640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 0x7fcc081edd50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000b970 tx=0x7fcc0000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.843+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0000cb40 con 0x7fcc081559e0 2024-02-19T14:52:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.844+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0000ccc0 con 0x7fcc081559e0 2024-02-19T14:52:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.844+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc081547e0 con 0x7fcc081559e0 2024-02-19T14:52:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.844+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00014680 con 0x7fcc081559e0 2024-02-19T14:52:58.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.844+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc081514c0 con 0x7fcc081559e0 2024-02-19T14:52:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.844+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc081eda70 con 0x7fcc081559e0 2024-02-19T14:52:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.845+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcc0001d3b0 con 0x7fcc081559e0 2024-02-19T14:52:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.845+0000 7fcbeffff640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbf406b810 0x7fcbf406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.845+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcc00089ee0 con 0x7fcc081559e0 2024-02-19T14:52:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.845+0000 7fcc077fe640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbf406b810 0x7fcbf406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:58.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.846+0000 7fcc077fe640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbf406b810 0x7fcbf406dcd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8036000 tx=0x7fcbf803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:58.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:58.850+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcc081eda70 con 0x7fcc081559e0 2024-02-19T14:52:59.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.022+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcc081eda70 con 0x7fcc081559e0 2024-02-19T14:52:59.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.830+0000 7fcbeffff640 1 -- 172.21.15.78:0/1784550806 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcc081eda70 con 0x7fcc081559e0 2024-02-19T14:52:59.833 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:52:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.833+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbf406b810 msgr2=0x7fcbf406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.833+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbf406b810 0x7fcbf406dcd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8036000 tx=0x7fcbf803d040 comp rx=0 tx=0).stop 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.833+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 msgr2=0x7fcc081edd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.833+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 0x7fcc081edd50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000b970 tx=0x7fcc0000be30 comp rx=0 tx=0).stop 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 shutdown_connections 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbf406b810 0x7fcbf406dcd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc081559e0 0x7fcc081edd50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc081524e0 0x7fcc080ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 --2- 172.21.15.78:0/1784550806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc080c8db0 0x7fcc08150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 >> 172.21.15.78:0/1784550806 conn(0x7fcc0806b060 msgr2=0x7fcc080a8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:59.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.834+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 shutdown_connections 2024-02-19T14:52:59.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.835+0000 7fcc0dc8f640 1 -- 172.21.15.78:0/1784550806 wait complete. 2024-02-19T14:52:59.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:52:59.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.962+0000 7f4b4fbe4640 1 Processor -- start 2024-02-19T14:52:59.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.963+0000 7f4b4fbe4640 1 -- start start 2024-02-19T14:52:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.963+0000 7f4b4fbe4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b480ace90 0x7f4b480c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.963+0000 7f4b4fbe4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b480c8ee0 0x7f4b480c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.963+0000 7f4b4fbe4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b48152a60 con 0x7f4b48152680 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.963+0000 7f4b4fbe4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b480aa050 con 0x7f4b480c8ee0 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.963+0000 7f4b4fbe4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b481557e0 con 0x7f4b480ace90 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b480c8ee0 0x7f4b480c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b480c8ee0 0x7f4b480c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46972/0 (socket says 172.21.15.78:46972) 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 -- 172.21.15.78:0/670958959 learned_addr learned my addr 172.21.15.78:0/670958959 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4d17f640 1 --2- 172.21.15.78:0/670958959 >> v2:172.21.15.150:3300/0 conn(0x7f4b480ace90 0x7f4b480c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 -- 172.21.15.78:0/670958959 >> v2:172.21.15.150:3300/0 conn(0x7f4b480ace90 msgr2=0x7f4b480c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 --2- 172.21.15.78:0/670958959 >> v2:172.21.15.150:3300/0 conn(0x7f4b480ace90 0x7f4b480c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 -- 172.21.15.78:0/670958959 >> v1:172.21.15.150:6789/0 conn(0x7f4b48152680 legacy=0x7f4b48155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.964+0000 7f4b4e181640 1 -- 172.21.15.78:0/670958959 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b48155ac0 con 0x7f4b480c8ee0 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4e181640 1 --2- 172.21.15.78:0/670958959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b480c8ee0 0x7f4b480c9b80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4b4808bf60 tx=0x7f4b4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859914ad73d7173c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4c97e640 1 -- 172.21.15.78:0/670958959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4002ca80 con 0x7f4b480c8ee0 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4c97e640 1 -- 172.21.15.78:0/670958959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4002cbe0 con 0x7f4b480c8ee0 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4c97e640 1 -- 172.21.15.78:0/670958959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b400343f0 con 0x7f4b480c8ee0 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/670958959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b480c8ee0 msgr2=0x7f4b480c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/670958959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b480c8ee0 0x7f4b480c9b80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4b4808bf60 tx=0x7f4b4002bee0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/670958959 shutdown_connections 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/670958959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b480c8ee0 0x7f4b480c9b80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/670958959 >> v2:172.21.15.150:3300/0 conn(0x7f4b480ace90 0x7f4b480c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.965+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/670958959 >> 172.21.15.78:0/670958959 conn(0x7f4b480ac370 msgr2=0x7f4b480ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:52:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.966+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/670958959 shutdown_connections 2024-02-19T14:52:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.966+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/670958959 wait complete. 2024-02-19T14:52:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 Processor -- start 2024-02-19T14:52:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 -- start start 2024-02-19T14:52:59.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b480ace90 0x7f4b481ea960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b480c8ee0 0x7f4b48148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 0x7f4b48148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48159050 con 0x7f4b48152680 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c1150 con 0x7f4b480ace90 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4fbe4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4814eb40 con 0x7f4b480c8ee0 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4e181640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 0x7f4b48148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4e181640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 0x7f4b48148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46988/0 (socket says 172.21.15.78:46988) 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.967+0000 7f4b4e181640 1 -- 172.21.15.78:0/3900613233 learned_addr learned my addr 172.21.15.78:0/3900613233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4d980640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b480ace90 0x7f4b481ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4e181640 1 -- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b480c8ee0 msgr2=0x7f4b48148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4d17f640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b480c8ee0 0x7f4b48148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4e181640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b480c8ee0 0x7f4b48148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4e181640 1 -- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b480ace90 msgr2=0x7f4b481ea960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4e181640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b480ace90 0x7f4b481ea960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4e181640 1 -- 172.21.15.78:0/3900613233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40009b30 con 0x7f4b48152680 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4d980640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b480ace90 0x7f4b481ea960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4d17f640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b480c8ee0 0x7f4b48148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4e181640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 0x7f4b48148fb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009f40 tx=0x7f4b40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40009510 con 0x7f4b48152680 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4003e450 con 0x7f4b48152680 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4003e800 con 0x7f4b48152680 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b481eb3e0 con 0x7f4b48152680 2024-02-19T14:52:59.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.968+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b480c0de0 con 0x7f4b48152680 2024-02-19T14:52:59.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.969+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b480c12e0 con 0x7f4b48152680 2024-02-19T14:52:59.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.970+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b4002cd50 con 0x7f4b48152680 2024-02-19T14:52:59.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.970+0000 7f4b3e7fc640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b2c06b810 0x7f4b2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:52:59.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.970+0000 7f4b4d980640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b2c06b810 0x7f4b2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:52:59.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.970+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f4b400acbf0 con 0x7f4b48152680 2024-02-19T14:52:59.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.971+0000 7f4b4d980640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b2c06b810 0x7f4b2c06dcd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b440015e0 tx=0x7f4b44009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:52:59.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:52:59.975+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b400776f0 con 0x7f4b48152680 2024-02-19T14:53:00.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.146+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4b480ca450 con 0x7f4b48152680 2024-02-19T14:53:00.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.835+0000 7f4b3e7fc640 1 -- 172.21.15.78:0/3900613233 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4b40043020 con 0x7f4b48152680 2024-02-19T14:53:00.838 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b2c06b810 msgr2=0x7f4b2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b2c06b810 0x7f4b2c06dcd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b440015e0 tx=0x7f4b44009040 comp rx=0 tx=0).stop 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 msgr2=0x7f4b48148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 0x7f4b48148fb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009f40 tx=0x7f4b40004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 shutdown_connections 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b2c06b810 0x7f4b2c06dcd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b48152680 0x7f4b48148fb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b480c8ee0 0x7f4b48148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 --2- 172.21.15.78:0/3900613233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b480ace90 0x7f4b481ea960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 >> 172.21.15.78:0/3900613233 conn(0x7f4b480ac370 msgr2=0x7f4b480c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 shutdown_connections 2024-02-19T14:53:00.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.838+0000 7f4b4fbe4640 1 -- 172.21.15.78:0/3900613233 wait complete. 2024-02-19T14:53:00.865 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:53:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.966+0000 7f6e29ada640 1 Processor -- start 2024-02-19T14:53:00.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.966+0000 7f6e29ada640 1 -- start start 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e29ada640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e240ac390 0x7f6e240ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e29ada640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6e240bdf20 0x7f6e240c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e29ada640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6e24158530 0x7f6e2406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e29ada640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e240c8de0 con 0x7f6e24158530 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e29ada640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e24155a60 con 0x7f6e240bdf20 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e29ada640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24155d40 con 0x7f6e240ac390 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6e24158530 0x7f6e2406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6e24158530 0x7f6e2406c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47002/0 (socket says 172.21.15.78:47002) 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 -- 172.21.15.78:0/4001834339 learned_addr learned my addr 172.21.15.78:0/4001834339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e22ffd640 1 --2- 172.21.15.78:0/4001834339 >> v2:172.21.15.150:3300/0 conn(0x7f6e240bdf20 0x7f6e240c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e237fe640 1 --2- 172.21.15.78:0/4001834339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e240ac390 0x7f6e240ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 -- 172.21.15.78:0/4001834339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e240ac390 msgr2=0x7f6e240ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 --2- 172.21.15.78:0/4001834339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e240ac390 0x7f6e240ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 -- 172.21.15.78:0/4001834339 >> v2:172.21.15.150:3300/0 conn(0x7f6e240bdf20 msgr2=0x7f6e240c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 --2- 172.21.15.78:0/4001834339 >> v2:172.21.15.150:3300/0 conn(0x7f6e240bdf20 0x7f6e240c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.967+0000 7f6e23fff640 1 -- 172.21.15.78:0/4001834339 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240c90a0 con 0x7f6e24158530 2024-02-19T14:53:00.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.968+0000 7f6e23fff640 1 --2- 172.21.15.78:0/4001834339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e24158530 0x7f6e2406c280 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6e240cb890 tx=0x7f6e1c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed91fbe6e01dcb99 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:00.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.968+0000 7f6e227fc640 1 -- 172.21.15.78:0/4001834339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c02c950 con 0x7f6e24158530 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.968+0000 7f6e227fc640 1 -- 172.21.15.78:0/4001834339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c02cab0 con 0x7f6e24158530 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.968+0000 7f6e227fc640 1 -- 172.21.15.78:0/4001834339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c02ce40 con 0x7f6e24158530 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 -- 172.21.15.78:0/4001834339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e24158530 msgr2=0x7f6e2406c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 --2- 172.21.15.78:0/4001834339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e24158530 0x7f6e2406c280 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6e240cb890 tx=0x7f6e1c02bdb0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 -- 172.21.15.78:0/4001834339 shutdown_connections 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 --2- 172.21.15.78:0/4001834339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e24158530 0x7f6e2406c280 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 --2- 172.21.15.78:0/4001834339 >> v2:172.21.15.150:3300/0 conn(0x7f6e240bdf20 0x7f6e240c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 --2- 172.21.15.78:0/4001834339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e240ac390 0x7f6e240ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 -- 172.21.15.78:0/4001834339 >> 172.21.15.78:0/4001834339 conn(0x7f6e240ca890 msgr2=0x7f6e240caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 -- 172.21.15.78:0/4001834339 shutdown_connections 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.969+0000 7f6e29ada640 1 -- 172.21.15.78:0/4001834339 wait complete. 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.970+0000 7f6e29ada640 1 Processor -- start 2024-02-19T14:53:00.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.970+0000 7f6e29ada640 1 -- start start 2024-02-19T14:53:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e29ada640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 0x7f6e24153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e29ada640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e240bdf20 0x7f6e240aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e29ada640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e24158530 0x7f6e240c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e29ada640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e241537b0 con 0x7f6e240ac390 2024-02-19T14:53:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e29ada640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24154580 con 0x7f6e240bdf20 2024-02-19T14:53:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e29ada640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24153fc0 con 0x7f6e24158530 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 0x7f6e24153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 0x7f6e24153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47012/0 (socket says 172.21.15.78:47012) 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 -- 172.21.15.78:0/3869777497 learned_addr learned my addr 172.21.15.78:0/3869777497 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e22ffd640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e240bdf20 0x7f6e240aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e23fff640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e24158530 0x7f6e240c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 -- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e24158530 msgr2=0x7f6e240c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e24158530 0x7f6e240c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 -- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e240bdf20 msgr2=0x7f6e240aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e240bdf20 0x7f6e240aebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 -- 172.21.15.78:0/3869777497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009b30 con 0x7f6e240ac390 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e237fe640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 0x7f6e24153270 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400e890 tx=0x7f6e1400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.971+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400c940 con 0x7f6e240ac390 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.972+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1400caa0 con 0x7f6e240ac390 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.972+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400ce30 con 0x7f6e240ac390 2024-02-19T14:53:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.972+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e241542a0 con 0x7f6e240ac390 2024-02-19T14:53:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.972+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e241eaa70 con 0x7f6e240ac390 2024-02-19T14:53:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.972+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e241ea5f0 con 0x7f6e240ac390 2024-02-19T14:53:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.973+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6e14013470 con 0x7f6e240ac390 2024-02-19T14:53:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.973+0000 7f6e0bfff640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6e1006b810 0x7f6e1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.974+0000 7f6e22ffd640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6e1006b810 0x7f6e1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.974+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f6e1408a490 con 0x7f6e240ac390 2024-02-19T14:53:00.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.974+0000 7f6e22ffd640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6e1006b810 0x7f6e1006dcd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e241580a0 tx=0x7f6e18008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:00.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:00.978+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6e241ea5f0 con 0x7f6e240ac390 2024-02-19T14:53:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.187+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6e241ea5f0 con 0x7f6e240ac390 2024-02-19T14:53:01.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.837+0000 7f6e0bfff640 1 -- 172.21.15.78:0/3869777497 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6e241ea5f0 con 0x7f6e240ac390 2024-02-19T14:53:01.840 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 5 and data pool 6 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6e1006b810 msgr2=0x7f6e1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6e1006b810 0x7f6e1006dcd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e241580a0 tx=0x7f6e18008040 comp rx=0 tx=0).stop 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 msgr2=0x7f6e24153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 0x7f6e24153270 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400e890 tx=0x7f6e1400ed50 comp rx=0 tx=0).stop 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 shutdown_connections 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6e1006b810 0x7f6e1006dcd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e24158530 0x7f6e240c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e240bdf20 0x7f6e240aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 --2- 172.21.15.78:0/3869777497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e240ac390 0x7f6e24153270 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 >> 172.21.15.78:0/3869777497 conn(0x7f6e240ca890 msgr2=0x7f6e24158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 shutdown_connections 2024-02-19T14:53:01.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.840+0000 7f6e29ada640 1 -- 172.21.15.78:0/3869777497 wait complete. 2024-02-19T14:53:01.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:01.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 Processor -- start 2024-02-19T14:53:01.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 -- start start 2024-02-19T14:53:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c0c152830 0x7f8c0c0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c0c0c7990 0x7f8c0c06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c1586a0 con 0x7f8c0c0c5760 2024-02-19T14:53:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c0c6550 con 0x7f8c0c152830 2024-02-19T14:53:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.970+0000 7f8c13d65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c159320 con 0x7f8c0c0c7990 2024-02-19T14:53:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c0c152830 0x7f8c0c0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c0c152830 0x7f8c0c0c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47014/0 (socket says 172.21.15.78:47014) 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 -- 172.21.15.78:0/1076213086 learned_addr learned my addr 172.21.15.78:0/1076213086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c12302640 1 --2- 172.21.15.78:0/1076213086 >> v2:172.21.15.150:3300/0 conn(0x7f8c0c0c7990 0x7f8c0c06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 -- 172.21.15.78:0/1076213086 >> v2:172.21.15.150:3300/0 conn(0x7f8c0c0c7990 msgr2=0x7f8c0c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 --2- 172.21.15.78:0/1076213086 >> v2:172.21.15.150:3300/0 conn(0x7f8c0c0c7990 0x7f8c0c06c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 -- 172.21.15.78:0/1076213086 >> v1:172.21.15.150:6789/0 conn(0x7f8c0c0c5760 legacy=0x7f8c0c0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c11b01640 1 -- 172.21.15.78:0/1076213086 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c158980 con 0x7f8c0c152830 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.971+0000 7f8c12302640 1 --2- 172.21.15.78:0/1076213086 >> v2:172.21.15.150:3300/0 conn(0x7f8c0c0c7990 0x7f8c0c06c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c11b01640 1 --2- 172.21.15.78:0/1076213086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c152830 0x7f8c0c0c5220 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0c5be0 tx=0x7f8c0402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9029cf7d130b78 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c10aff640 1 -- 172.21.15.78:0/1076213086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0402ca80 con 0x7f8c0c152830 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c10aff640 1 -- 172.21.15.78:0/1076213086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0402cbe0 con 0x7f8c0c152830 2024-02-19T14:53:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c10aff640 1 -- 172.21.15.78:0/1076213086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040343f0 con 0x7f8c0c152830 2024-02-19T14:53:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 -- 172.21.15.78:0/1076213086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c152830 msgr2=0x7f8c0c0c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 --2- 172.21.15.78:0/1076213086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c152830 0x7f8c0c0c5220 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0c5be0 tx=0x7f8c0402bee0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 -- 172.21.15.78:0/1076213086 shutdown_connections 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 --2- 172.21.15.78:0/1076213086 >> v2:172.21.15.150:3300/0 conn(0x7f8c0c0c7990 0x7f8c0c06c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 --2- 172.21.15.78:0/1076213086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c152830 0x7f8c0c0c5220 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 -- 172.21.15.78:0/1076213086 >> 172.21.15.78:0/1076213086 conn(0x7f8c0c0ca8a0 msgr2=0x7f8c0c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 -- 172.21.15.78:0/1076213086 shutdown_connections 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.972+0000 7f8c13d65640 1 -- 172.21.15.78:0/1076213086 wait complete. 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.973+0000 7f8c13d65640 1 Processor -- start 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 -- start start 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c0c0c5760 0x7f8c0c1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c0c152830 0x7f8c0c157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 0x7f8c0c0be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1559f0 con 0x7f8c0c1ee850 2024-02-19T14:53:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c158430 con 0x7f8c0c152830 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c13d65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c0ac0c0 con 0x7f8c0c0c5760 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c12302640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 0x7f8c0c0be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c12302640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 0x7f8c0c0be3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47024/0 (socket says 172.21.15.78:47024) 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c12302640 1 -- 172.21.15.78:0/2188428642 learned_addr learned my addr 172.21.15.78:0/2188428642 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c11b01640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c0c0c5760 0x7f8c0c1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.974+0000 7f8c11300640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c0c152830 0x7f8c0c157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c12302640 1 -- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c0c0c5760 msgr2=0x7f8c0c1554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c12302640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c0c0c5760 0x7f8c0c1554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c12302640 1 -- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c0c152830 msgr2=0x7f8c0c157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c12302640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c0c152830 0x7f8c0c157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c12302640 1 -- 172.21.15.78:0/2188428642 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04009b30 con 0x7f8c0c1ee850 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c12302640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 0x7f8c0c0be3d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c820 tx=0x7f8c0800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08007a30 con 0x7f8c0c1ee850 2024-02-19T14:53:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c08007b90 con 0x7f8c0c1ee850 2024-02-19T14:53:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0800ce10 con 0x7f8c0c1ee850 2024-02-19T14:53:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c0ac380 con 0x7f8c0c1ee850 2024-02-19T14:53:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.975+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c0ac940 con 0x7f8c0c1ee850 2024-02-19T14:53:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.976+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c074990 con 0x7f8c0c1ee850 2024-02-19T14:53:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.976+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8c080113f0 con 0x7f8c0c1ee850 2024-02-19T14:53:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.977+0000 7f8c027fc640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8bf006b810 0x7f8bf006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.977+0000 7f8c11b01640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8bf006b810 0x7f8bf006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.977+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8c0808a3a0 con 0x7f8c0c1ee850 2024-02-19T14:53:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.977+0000 7f8c11b01640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8bf006b810 0x7f8bf006dcd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8c04035000 tx=0x7f8c04030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:01.981+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8c08054cf0 con 0x7f8c0c1ee850 2024-02-19T14:53:02.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.152+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c0c0acee0 con 0x7f8c0c1ee850 2024-02-19T14:53:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.154+0000 7f8c027fc640 1 -- 172.21.15.78:0/2188428642 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16943 (secure 0 0 0) 0x7f8c08077070 con 0x7f8c0c1ee850 2024-02-19T14:53:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:02.157 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":37,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:01.188305+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:52:59.040775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:00.165210+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.155+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8bf006b810 msgr2=0x7f8bf006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.155+0000 7f8c13d65640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8bf006b810 0x7f8bf006dcd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8c04035000 tx=0x7f8c04030000 comp rx=0 tx=0).stop 2024-02-19T14:53:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.155+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 msgr2=0x7f8c0c0be3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.155+0000 7f8c13d65640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 0x7f8c0c0be3d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c820 tx=0x7f8c0800cce0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 shutdown_connections 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8bf006b810 0x7f8bf006dcd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c1ee850 0x7f8c0c0be3d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c0c152830 0x7f8c0c157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 --2- 172.21.15.78:0/2188428642 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c0c0c5760 0x7f8c0c1554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 >> 172.21.15.78:0/2188428642 conn(0x7f8c0c0ca8a0 msgr2=0x7f8c0c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 shutdown_connections 2024-02-19T14:53:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.156+0000 7f8c13d65640 1 -- 172.21.15.78:0/2188428642 wait complete. 2024-02-19T14:53:02.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:02.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.267+0000 7fdb20495640 1 Processor -- start 2024-02-19T14:53:02.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.267+0000 7fdb20495640 1 -- start start 2024-02-19T14:53:02.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb20495640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb180713a0 0x7fdb18071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb20495640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c29e0 0x7fdb180c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb20495640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180c69f0 con 0x7fdb180c8b60 2024-02-19T14:53:02.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb20495640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb180c32c0 con 0x7fdb180713a0 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb20495640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb18070500 con 0x7fdb180c29e0 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb180713a0 0x7fdb18071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb180713a0 0x7fdb18071780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47036/0 (socket says 172.21.15.78:47036) 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1ea32640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdb180c8b60 0x7fdb181536b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46424/0 (socket says 172.21.15.78:46424) 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 -- 172.21.15.78:0/2578079810 learned_addr learned my addr 172.21.15.78:0/2578079810 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1da30640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c29e0 0x7fdb180c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 -- 172.21.15.78:0/2578079810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c29e0 msgr2=0x7fdb180c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c29e0 0x7fdb180c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 -- 172.21.15.78:0/2578079810 >> v1:172.21.15.78:6789/0 conn(0x7fdb180c8b60 legacy=0x7fdb181536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1e231640 1 -- 172.21.15.78:0/2578079810 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1806fe40 con 0x7fdb180713a0 2024-02-19T14:53:02.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.268+0000 7fdb1da30640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c29e0 0x7fdb180c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:02.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb1e231640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18071780 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdb140098a0 tx=0x7fdb1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc597078d66bd014 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb1d22f640 1 -- 172.21.15.78:0/2578079810 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14031640 con 0x7fdb180713a0 2024-02-19T14:53:02.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb1d22f640 1 -- 172.21.15.78:0/2578079810 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb140317a0 con 0x7fdb180713a0 2024-02-19T14:53:02.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb1d22f640 1 -- 172.21.15.78:0/2578079810 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14031b30 con 0x7fdb180713a0 2024-02-19T14:53:02.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb20495640 1 -- 172.21.15.78:0/2578079810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 msgr2=0x7fdb18071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb20495640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18071780 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdb140098a0 tx=0x7fdb1402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb20495640 1 -- 172.21.15.78:0/2578079810 shutdown_connections 2024-02-19T14:53:02.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb20495640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c29e0 0x7fdb180c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb20495640 1 --2- 172.21.15.78:0/2578079810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18071780 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.269+0000 7fdb20495640 1 -- 172.21.15.78:0/2578079810 >> 172.21.15.78:0/2578079810 conn(0x7fdb180aa300 msgr2=0x7fdb180aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.270+0000 7fdb20495640 1 -- 172.21.15.78:0/2578079810 shutdown_connections 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.270+0000 7fdb20495640 1 -- 172.21.15.78:0/2578079810 wait complete. 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 Processor -- start 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 -- start start 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb180c29e0 0x7fdb181568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c8b60 0x7fdb181e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb18154070 con 0x7fdb180713a0 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb180c11e0 con 0x7fdb180c29e0 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb20495640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb180bd600 con 0x7fdb180c8b60 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb1e231640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb1e231640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47042/0 (socket says 172.21.15.78:47042) 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.271+0000 7fdb1e231640 1 -- 172.21.15.78:0/3081032307 learned_addr learned my addr 172.21.15.78:0/3081032307 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb1e231640 1 -- 172.21.15.78:0/3081032307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c8b60 msgr2=0x7fdb181e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb1e231640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c8b60 0x7fdb181e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb1e231640 1 -- 172.21.15.78:0/3081032307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb180c29e0 msgr2=0x7fdb181568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb1e231640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb180c29e0 0x7fdb181568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb1e231640 1 -- 172.21.15.78:0/3081032307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb140094d0 con 0x7fdb180713a0 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb1e231640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18156360 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdb140098a0 tx=0x7fdb140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1403ac60 con 0x7fdb180713a0 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1403ade0 con 0x7fdb180713a0 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb181ebb70 con 0x7fdb180713a0 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14044630 con 0x7fdb180713a0 2024-02-19T14:53:02.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.272+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb181eb890 con 0x7fdb180713a0 2024-02-19T14:53:02.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.273+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb18074990 con 0x7fdb180713a0 2024-02-19T14:53:02.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.273+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdb1402f050 con 0x7fdb180713a0 2024-02-19T14:53:02.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.274+0000 7fdb0effd640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdafc06b810 0x7fdafc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.274+0000 7fdb1da30640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdafc06b810 0x7fdafc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.274+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdb140ac480 con 0x7fdb180713a0 2024-02-19T14:53:02.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.274+0000 7fdb1da30640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdafc06b810 0x7fdafc06dcd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdb08006fd0 tx=0x7fdb08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.278+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdb14076d00 con 0x7fdb180713a0 2024-02-19T14:53:02.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.452+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb180c7f80 con 0x7fdb180713a0 2024-02-19T14:53:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.453+0000 7fdb0effd640 1 -- 172.21.15.78:0/3081032307 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16943 (secure 0 0 0) 0x7fdb1407a3c0 con 0x7fdb180713a0 2024-02-19T14:53:02.456 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:02.456 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":37,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:01.188305+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:52:59.040775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:00.165210+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdafc06b810 msgr2=0x7fdafc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdafc06b810 0x7fdafc06dcd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdb08006fd0 tx=0x7fdb08008040 comp rx=0 tx=0).stop 2024-02-19T14:53:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 msgr2=0x7fdb18156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18156360 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdb140098a0 tx=0x7fdb140046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 shutdown_connections 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdafc06b810 0x7fdafc06dcd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb180c8b60 0x7fdb181e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb180c29e0 0x7fdb181568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 --2- 172.21.15.78:0/3081032307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb180713a0 0x7fdb18156360 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 >> 172.21.15.78:0/3081032307 conn(0x7fdb180aa300 msgr2=0x7fdb180c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 shutdown_connections 2024-02-19T14:53:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.455+0000 7fdb20495640 1 -- 172.21.15.78:0/3081032307 wait complete. 2024-02-19T14:53:02.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:53:02.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.570+0000 7fd5826a4640 1 Processor -- start 2024-02-19T14:53:02.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.570+0000 7fd5826a4640 1 -- start start 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd5826a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd57c14f1a0 0x7fd57c072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd5826a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd57c06ea50 0x7fd57c152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd5826a4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0c5970 con 0x7fd57c06fbc0 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd5826a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0c5750 con 0x7fd57c06ea50 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd5826a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0c5bc0 con 0x7fd57c14f1a0 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd57b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd57c06ea50 0x7fd57c152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd57b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd57c06ea50 0x7fd57c152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47050/0 (socket says 172.21.15.78:47050) 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.571+0000 7fd57b7fe640 1 -- 172.21.15.78:0/139354982 learned_addr learned my addr 172.21.15.78:0/139354982 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:02.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57bfff640 1 --2- 172.21.15.78:0/139354982 >> v2:172.21.15.150:3300/0 conn(0x7fd57c14f1a0 0x7fd57c072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57b7fe640 1 -- 172.21.15.78:0/139354982 >> v2:172.21.15.150:3300/0 conn(0x7fd57c14f1a0 msgr2=0x7fd57c072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57b7fe640 1 --2- 172.21.15.78:0/139354982 >> v2:172.21.15.150:3300/0 conn(0x7fd57c14f1a0 0x7fd57c072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57b7fe640 1 -- 172.21.15.78:0/139354982 >> v1:172.21.15.150:6789/0 conn(0x7fd57c06fbc0 legacy=0x7fd57c0c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57b7fe640 1 -- 172.21.15.78:0/139354982 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0c5ea0 con 0x7fd57c06ea50 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57bfff640 1 --2- 172.21.15.78:0/139354982 >> v2:172.21.15.150:3300/0 conn(0x7fd57c14f1a0 0x7fd57c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57b7fe640 1 --2- 172.21.15.78:0/139354982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06ea50 0x7fd57c152680 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0c85a0 tx=0x7fd57002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7bd8733190be44 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.572+0000 7fd57affd640 1 -- 172.21.15.78:0/139354982 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570031750 con 0x7fd57c06ea50 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd57affd640 1 -- 172.21.15.78:0/139354982 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5700318b0 con 0x7fd57c06ea50 2024-02-19T14:53:02.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd57affd640 1 -- 172.21.15.78:0/139354982 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570031c40 con 0x7fd57c06ea50 2024-02-19T14:53:02.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 -- 172.21.15.78:0/139354982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06ea50 msgr2=0x7fd57c152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 --2- 172.21.15.78:0/139354982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06ea50 0x7fd57c152680 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0c85a0 tx=0x7fd57002b7f0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 -- 172.21.15.78:0/139354982 shutdown_connections 2024-02-19T14:53:02.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 --2- 172.21.15.78:0/139354982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06ea50 0x7fd57c152680 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 --2- 172.21.15.78:0/139354982 >> v2:172.21.15.150:3300/0 conn(0x7fd57c14f1a0 0x7fd57c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 -- 172.21.15.78:0/139354982 >> 172.21.15.78:0/139354982 conn(0x7fd57c06c170 msgr2=0x7fd57c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 -- 172.21.15.78:0/139354982 shutdown_connections 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.573+0000 7fd5826a4640 1 -- 172.21.15.78:0/139354982 wait complete. 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.574+0000 7fd5826a4640 1 Processor -- start 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.574+0000 7fd5826a4640 1 -- start start 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd5826a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd57c06ea50 0x7fd57c06ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd5826a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 0x7fd57c154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd5826a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd57c14f1a0 0x7fd57c155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd5826a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1ec990 con 0x7fd57c06fbc0 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd5826a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c06b3c0 con 0x7fd57c06ea50 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd5826a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c155370 con 0x7fd57c14f1a0 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 0x7fd57c154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 0x7fd57c154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47056/0 (socket says 172.21.15.78:47056) 2024-02-19T14:53:02.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 -- 172.21.15.78:0/3355190681 learned_addr learned my addr 172.21.15.78:0/3355190681 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57bfff640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd57c06ea50 0x7fd57c06ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd580c41640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd57c14f1a0 0x7fd57c155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 -- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd57c14f1a0 msgr2=0x7fd57c155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd57c14f1a0 0x7fd57c155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 -- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd57c06ea50 msgr2=0x7fd57c06ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd57c06ea50 0x7fd57c06ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 -- 172.21.15.78:0/3355190681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700094d0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57bfff640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd57c06ea50 0x7fd57c06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd580c41640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd57c14f1a0 0x7fd57c155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.575+0000 7fd57b7fe640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 0x7fd57c154e30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd570000c00 tx=0x7fd5700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.576+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570033c80 con 0x7fd57c06fbc0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.576+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd570033de0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.576+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c06b620 con 0x7fd57c06fbc0 2024-02-19T14:53:02.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.576+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57003c620 con 0x7fd57c06fbc0 2024-02-19T14:53:02.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.576+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c1ec2d0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.577+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c1ecad0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.577+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd57003a040 con 0x7fd57c06fbc0 2024-02-19T14:53:02.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.577+0000 7fd578ff9640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd56806b810 0x7fd56806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.577+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5700ac100 con 0x7fd57c06fbc0 2024-02-19T14:53:02.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.577+0000 7fd57bfff640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd56806b810 0x7fd56806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.578+0000 7fd57bfff640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd56806b810 0x7fd56806dcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd56c006fd0 tx=0x7fd56c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.582+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd57c1ecad0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.788+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd57c1ecad0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.854+0000 7fd578ff9640 1 -- 172.21.15.78:0/3355190681 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd57c1ecad0 con 0x7fd57c06fbc0 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd56806b810 msgr2=0x7fd56806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd56806b810 0x7fd56806dcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd56c006fd0 tx=0x7fd56c008040 comp rx=0 tx=0).stop 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 msgr2=0x7fd57c154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 0x7fd57c154e30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd570000c00 tx=0x7fd5700046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 shutdown_connections 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd56806b810 0x7fd56806dcd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd57c06fbc0 0x7fd57c154e30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd57c14f1a0 0x7fd57c155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 --2- 172.21.15.78:0/3355190681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd57c06ea50 0x7fd57c06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 >> 172.21.15.78:0/3355190681 conn(0x7fd57c06c170 msgr2=0x7fd57c0728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 shutdown_connections 2024-02-19T14:53:02.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.857+0000 7fd5826a4640 1 -- 172.21.15.78:0/3355190681 wait complete. 2024-02-19T14:53:02.870 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:53:02.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:53:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.973+0000 7f5dd8803640 1 Processor -- start 2024-02-19T14:53:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.973+0000 7f5dd8803640 1 -- start start 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd8803640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5dd413b0b0 0x7f5dd4072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd8803640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd406f400 con 0x7f5dd413abe0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd8803640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd40a9f90 con 0x7f5dd406efb0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd8803640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd4152bc0 con 0x7f5dd413b0b0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5dd413abe0 0x7f5dd41524b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46440/0 (socket says 172.21.15.78:46440) 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/134012547 learned_addr learned my addr 172.21.15.78:0/134012547 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161536522 0 0) 0x7f5dd406f400 con 0x7f5dd413abe0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dbc003520 con 0x7f5dd413abe0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd259c640 1 --2- 172.21.15.78:0/134012547 >> v2:172.21.15.150:3300/0 conn(0x7f5dd413b0b0 0x7f5dd4072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502736207 0 0) 0x7f5dbc003520 con 0x7f5dd413abe0 2024-02-19T14:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 >> v2:172.21.15.150:3300/0 conn(0x7f5dd413b0b0 msgr2=0x7f5dd4072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 --2- 172.21.15.78:0/134012547 >> v2:172.21.15.150:3300/0 conn(0x7f5dd413b0b0 0x7f5dd4072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 >> v1:172.21.15.150:6789/0 conn(0x7f5dd406efb0 legacy=0x7f5dd40c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd40a99d0 con 0x7f5dd413abe0 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f5dc4001670 con 0x7f5dd413abe0 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 >> v1:172.21.15.78:6789/0 conn(0x7f5dd413abe0 legacy=0x7f5dd41524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dbc0047c0 0x7f5dbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dbc005180 0x7f5dbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd4152bc0 con 0x7f5dbc003970 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc00c020 con 0x7f5dbc005180 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc00c1e0 con 0x7f5dbc0047c0 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.974+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd359e640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dbc005180 0x7f5dbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd259c640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dbc0047c0 0x7f5dbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dbc0047c0 msgr2=0x7f5dbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dbc0047c0 0x7f5dbc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dbc005180 msgr2=0x7f5dbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dbc005180 0x7f5dbc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/134012547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc003520 con 0x7f5dbc003970 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd359e640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dbc005180 0x7f5dbc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd359e640 1 -- 172.21.15.78:0/134012547 reap_dead start 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.975+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dbc003d90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c240 tx=0x7f5dbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8fc3dfdcce14fa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4005ad0 con 0x7f5dbc003970 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc4005c30 con 0x7f5dbc003970 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd1d9b640 1 -- 172.21.15.78:0/134012547 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc402caf0 con 0x7f5dbc003970 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 -- 172.21.15.78:0/134012547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 msgr2=0x7f5dbc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dbc003d90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c240 tx=0x7f5dbc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 -- 172.21.15.78:0/134012547 shutdown_connections 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 --2- 172.21.15.78:0/134012547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dbc003d90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 -- 172.21.15.78:0/134012547 >> 172.21.15.78:0/134012547 conn(0x7f5dd40c3f20 msgr2=0x7f5dd40c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 -- 172.21.15.78:0/134012547 shutdown_connections 2024-02-19T14:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.976+0000 7f5dd8803640 1 -- 172.21.15.78:0/134012547 wait complete. 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.978+0000 7f5dd8803640 1 Processor -- start 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.978+0000 7f5dd8803640 1 -- start start 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.979+0000 7f5dd8803640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dd4155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.979+0000 7f5dd8803640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dd41e8fc0 0x7f5dd41558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.979+0000 7f5dd8803640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40bc440 0x7f5dd40c06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.979+0000 7f5dd8803640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41ec780 con 0x7f5dbc003970 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.979+0000 7f5dd8803640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41e82d0 con 0x7f5dd40bc440 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.979+0000 7f5dd8803640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41ec250 con 0x7f5dd41e8fc0 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dd4155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dd4155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47066/0 (socket says 172.21.15.78:47066) 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/473477419 learned_addr learned my addr 172.21.15.78:0/473477419 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd259c640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dd41e8fc0 0x7f5dd41558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dd41e8fc0 msgr2=0x7f5dd41558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dd41e8fc0 0x7f5dd41558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40bc440 msgr2=0x7f5dd40c06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40bc440 0x7f5dd40c06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 -- 172.21.15.78:0/473477419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd41ec4a0 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd2d9d640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dd4155390 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000f60 tx=0x7f5dc402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc403c040 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc402ccc0 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd41e84e0 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4007270 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.980+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd41e8aa0 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.981+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd41e8c30 con 0x7f5dbc003970 2024-02-19T14:53:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.982+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5dc4032070 con 0x7f5dbc003970 2024-02-19T14:53:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.982+0000 7f5dbb7fe640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5dc006b810 0x7f5dc006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.982+0000 7f5dd259c640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5dc006b810 0x7f5dc006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.982+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5dc40ab820 con 0x7f5dbc003970 2024-02-19T14:53:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.983+0000 7f5dd259c640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5dc006b810 0x7f5dc006dcd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5dd41501d0 tx=0x7f5dc8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:02.987+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5dc40760b0 con 0x7f5dbc003970 2024-02-19T14:53:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.203+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5dd4151c30 con 0x7f5dbc003970 2024-02-19T14:53:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.861+0000 7f5dbb7fe640 1 -- 172.21.15.78:0/473477419 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5dc4079770 con 0x7f5dbc003970 2024-02-19T14:53:03.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.863+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5dc006b810 msgr2=0x7f5dc006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.863+0000 7f5dd8803640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5dc006b810 0x7f5dc006dcd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5dd41501d0 tx=0x7f5dc8008040 comp rx=0 tx=0).stop 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.863+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 msgr2=0x7f5dd4155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.863+0000 7f5dd8803640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dd4155390 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000f60 tx=0x7f5dc402e070 comp rx=0 tx=0).stop 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 shutdown_connections 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5dc006b810 0x7f5dc006dcd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40bc440 0x7f5dd40c06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5dd41e8fc0 0x7f5dd41558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 --2- 172.21.15.78:0/473477419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc003970 0x7f5dd4155390 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 >> 172.21.15.78:0/473477419 conn(0x7f5dd40c3f20 msgr2=0x7f5dd40c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 shutdown_connections 2024-02-19T14:53:03.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.864+0000 7f5dd8803640 1 -- 172.21.15.78:0/473477419 wait complete. 2024-02-19T14:53:03.877 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:53:03.877 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:03.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.975+0000 7f391f52e640 1 Processor -- start 2024-02-19T14:53:03.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.975+0000 7f391f52e640 1 -- start start 2024-02-19T14:53:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.976+0000 7f391f52e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39180c5560 0x7f39180c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.976+0000 7f391f52e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39180c65a0 con 0x7f39180c7910 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.976+0000 7f391f52e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39180c7530 con 0x7f3918070720 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.976+0000 7f391f52e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39181537b0 con 0x7f39180c5560 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.976+0000 7f391cac9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f39180c7910 0x7f3918154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46444/0 (socket says 172.21.15.78:46444) 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.976+0000 7f391cac9640 1 -- 172.21.15.78:0/2077765019 learned_addr learned my addr 172.21.15.78:0/2077765019 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2077765019 >> v2:172.21.15.150:3300/0 conn(0x7f39180c5560 0x7f39180c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122962560 0 0) 0x7f39180c65a0 con 0x7f39180c7910 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3904003520 con 0x7f39180c7910 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3499207435 0 0) 0x7f3904003520 con 0x7f39180c7910 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 >> v2:172.21.15.150:3300/0 conn(0x7f39180c5560 msgr2=0x7f39180c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 --2- 172.21.15.78:0/2077765019 >> v2:172.21.15.150:3300/0 conn(0x7f39180c5560 0x7f39180c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 >> v1:172.21.15.150:6789/0 conn(0x7f3918070720 legacy=0x7f3918070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918153a90 con 0x7f39180c7910 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3908001670 con 0x7f39180c7910 2024-02-19T14:53:03.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 >> v1:172.21.15.78:6789/0 conn(0x7f39180c7910 legacy=0x7f3918154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f3904003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39040047c0 0x7f3904004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904005180 0x7f390400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39181537b0 con 0x7f3904003970 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390400c0d0 con 0x7f3904005180 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390400c290 con 0x7f39040047c0 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f3904003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f391dacb640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904005180 0x7f390400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.977+0000 7f391cac9640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39040047c0 0x7f3904004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391d2ca640 1 -- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39040047c0 msgr2=0x7f3904004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39040047c0 0x7f3904004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391d2ca640 1 -- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904005180 msgr2=0x7f390400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904005180 0x7f390400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391d2ca640 1 -- 172.21.15.78:0/2077765019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904003520 con 0x7f3904003970 2024-02-19T14:53:03.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391dacb640 1 -- 172.21.15.78:0/2077765019 reap_dead start 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f3904003d50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3914009ac0 tx=0x7f39040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2cdbbc996aa0c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391402c6b0 con 0x7f3904003970 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391402c810 con 0x7f3904003970 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.978+0000 7f390ffff640 1 -- 172.21.15.78:0/2077765019 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391402cba0 con 0x7f3904003970 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 -- 172.21.15.78:0/2077765019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 msgr2=0x7f3904003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f3904003d50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3914009ac0 tx=0x7f39040057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 -- 172.21.15.78:0/2077765019 shutdown_connections 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 --2- 172.21.15.78:0/2077765019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f3904003d50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 -- 172.21.15.78:0/2077765019 >> 172.21.15.78:0/2077765019 conn(0x7f391806b4b0 msgr2=0x7f39180c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 -- 172.21.15.78:0/2077765019 shutdown_connections 2024-02-19T14:53:03.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.979+0000 7f391f52e640 1 -- 172.21.15.78:0/2077765019 wait complete. 2024-02-19T14:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.980+0000 7f391f52e640 1 Processor -- start 2024-02-19T14:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.980+0000 7f391f52e640 1 -- start start 2024-02-19T14:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391f52e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f39180acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391f52e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39180a9c20 0x7f39180aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391f52e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39180a8600 0x7f39180a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391f52e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39180bf7d0 con 0x7f3904003970 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391f52e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39181ec3f0 con 0x7f39180a9c20 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391f52e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39181eb500 con 0x7f39180a8600 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f39180acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f39180acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47084/0 (socket says 172.21.15.78:47084) 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 -- 172.21.15.78:0/2348128213 learned_addr learned my addr 172.21.15.78:0/2348128213 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391dacb640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39180a8600 0x7f39180a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391cac9640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39180a9c20 0x7f39180aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 -- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39180a8600 msgr2=0x7f39180a8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39180a8600 0x7f39180a8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 -- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39180a9c20 msgr2=0x7f39180aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39180a9c20 0x7f39180aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 -- 172.21.15.78:0/2348128213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39180c5ac0 con 0x7f3904003970 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.981+0000 7f391d2ca640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f39180acc20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3914000c00 tx=0x7f3914004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:03.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.982+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914003810 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.982+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3914003970 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.982+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914003d20 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.982+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39181eb7e0 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.982+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39181ebda0 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.983+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3918074990 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.983+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3914034440 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.984+0000 7f390dffb640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38fc06b840 0x7f38fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.984+0000 7f391cac9640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38fc06b840 0x7f38fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.984+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f39140ac290 con 0x7f3904003970 2024-02-19T14:53:03.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.984+0000 7f391cac9640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38fc06b840 0x7f38fc06dd00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f39181531a0 tx=0x7f3908001510 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:03.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:03.991+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3914076d80 con 0x7f3904003970 2024-02-19T14:53:04.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.198+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39180c76f0 con 0x7f3904003970 2024-02-19T14:53:04.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.199+0000 7f390dffb640 1 -- 172.21.15.78:0/2348128213 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f39140044e0 con 0x7f3904003970 2024-02-19T14:53:04.201 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:04.201 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:03.204838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5247":{"gid":5247,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3231075223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3231075223},{"type":"v1","addr":"172.21.15.78:6837","nonce":3231075223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:04.202 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 15 2024-02-19T14:53:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.200+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38fc06b840 msgr2=0x7f38fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.200+0000 7f391f52e640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38fc06b840 0x7f38fc06dd00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f39181531a0 tx=0x7f3908001510 comp rx=0 tx=0).stop 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.200+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 msgr2=0x7f39180acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.200+0000 7f391f52e640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f39180acc20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3914000c00 tx=0x7f3914004730 comp rx=0 tx=0).stop 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 shutdown_connections 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38fc06b840 0x7f38fc06dd00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39180a8600 0x7f39180a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39180a9c20 0x7f39180aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 --2- 172.21.15.78:0/2348128213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904003970 0x7f39180acc20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 >> 172.21.15.78:0/2348128213 conn(0x7f391806b4b0 msgr2=0x7f39180c8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 shutdown_connections 2024-02-19T14:53:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.201+0000 7f391f52e640 1 -- 172.21.15.78:0/2348128213 wait complete. 2024-02-19T14:53:04.214 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:04.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.312+0000 7f470f49a640 1 Processor -- start 2024-02-19T14:53:04.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.312+0000 7f470f49a640 1 -- start start 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470f49a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f470806da10 0x7f47080723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470f49a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080c1d80 con 0x7f47080acb50 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470f49a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470806c5a0 con 0x7f4708072910 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470f49a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470806caf0 con 0x7f470806da10 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470ca35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f470806da10 0x7f47080723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470d236640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f47080acb50 0x7f4708155900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46450/0 (socket says 172.21.15.78:46450) 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470d236640 1 -- 172.21.15.78:0/1174633758 learned_addr learned my addr 172.21.15.78:0/1174633758 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470ca35640 1 -- 172.21.15.78:0/1174633758 >> v1:172.21.15.150:6789/0 conn(0x7f4708072910 legacy=0x7f4708158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470ca35640 1 -- 172.21.15.78:0/1174633758 >> v1:172.21.15.78:6789/0 conn(0x7f47080acb50 legacy=0x7f4708155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f470ca35640 1 -- 172.21.15.78:0/1174633758 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470806c810 con 0x7f470806da10 2024-02-19T14:53:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.313+0000 7f4707fff640 1 -- 172.21.15.78:0/1174633758 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861992508 0 0) 0x7f47080c1d80 con 0x7f47080acb50 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.314+0000 7f470ca35640 1 --2- 172.21.15.78:0/1174633758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470806da10 0x7f47080723d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009d90 tx=0x7f46fc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92199e70d77e0112 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.314+0000 7f4707fff640 1 -- 172.21.15.78:0/1174633758 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc02cbd0 con 0x7f470806da10 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.314+0000 7f4707fff640 1 -- 172.21.15.78:0/1174633758 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc02cd30 con 0x7f470806da10 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.314+0000 7f4707fff640 1 -- 172.21.15.78:0/1174633758 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc02cea0 con 0x7f470806da10 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 -- 172.21.15.78:0/1174633758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470806da10 msgr2=0x7f47080723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 --2- 172.21.15.78:0/1174633758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470806da10 0x7f47080723d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009d90 tx=0x7f46fc02be20 comp rx=0 tx=0).stop 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 -- 172.21.15.78:0/1174633758 shutdown_connections 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 --2- 172.21.15.78:0/1174633758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470806da10 0x7f47080723d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 -- 172.21.15.78:0/1174633758 >> 172.21.15.78:0/1174633758 conn(0x7f47080718e0 msgr2=0x7f4708071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 -- 172.21.15.78:0/1174633758 shutdown_connections 2024-02-19T14:53:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.315+0000 7f470f49a640 1 -- 172.21.15.78:0/1174633758 wait complete. 2024-02-19T14:53:04.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.316+0000 7f470f49a640 1 Processor -- start 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.316+0000 7f470f49a640 1 -- start start 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470806da10 0x7f4708145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4708072910 0x7f47081edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 0x7f47081ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4708154a00 con 0x7f47080acb50 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4708151e20 con 0x7f4708072910 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4708154ef0 con 0x7f470806da10 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 0x7f47081ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 0x7f47081ee3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47110/0 (socket says 172.21.15.78:47110) 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 -- 172.21.15.78:0/2199304801 learned_addr learned my addr 172.21.15.78:0/2199304801 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470ca35640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4708072910 0x7f47081edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470d236640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470806da10 0x7f4708145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 -- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470806da10 msgr2=0x7f4708145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470806da10 0x7f4708145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 -- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4708072910 msgr2=0x7f47081edeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4708072910 0x7f47081edeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 -- 172.21.15.78:0/2199304801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc009a70 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470ca35640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4708072910 0x7f47081edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470da37640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 0x7f47081ee3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f46f800c3c0 tx=0x7f46f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f80035f0 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f8003750 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8003b00 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.317+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4708153e70 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.318+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4708154c10 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.319+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470814f620 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.319+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f46f8003c60 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.319+0000 7f4705ffb640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f406b810 0x7f46f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.319+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f46f8089850 con 0x7f47080acb50 2024-02-19T14:53:04.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.319+0000 7f470d236640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f406b810 0x7f46f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.320+0000 7f470d236640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f406b810 0x7f46f406dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4700000f60 tx=0x7f4700007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:04.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.323+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f470814f620 con 0x7f47080acb50 2024-02-19T14:53:04.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.539+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f470814f620 con 0x7f47080acb50 2024-02-19T14:53:04.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.540+0000 7f4705ffb640 1 -- 172.21.15.78:0/2199304801 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16930 (secure 0 0 0) 0x7f470814f620 con 0x7f47080acb50 2024-02-19T14:53:04.543 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:04.543 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":38,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:02.851349+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:52:59.040775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:00.165210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f406b810 msgr2=0x7f46f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f406b810 0x7f46f406dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4700000f60 tx=0x7f4700007c00 comp rx=0 tx=0).stop 2024-02-19T14:53:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 msgr2=0x7f47081ee3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 0x7f47081ee3f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f46f800c3c0 tx=0x7f46f800c880 comp rx=0 tx=0).stop 2024-02-19T14:53:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 shutdown_connections 2024-02-19T14:53:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f406b810 0x7f46f406dcd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47080acb50 0x7f47081ee3f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4708072910 0x7f47081edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 --2- 172.21.15.78:0/2199304801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470806da10 0x7f4708145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 >> 172.21.15.78:0/2199304801 conn(0x7f47080718e0 msgr2=0x7f4708155640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 shutdown_connections 2024-02-19T14:53:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.542+0000 7f470f49a640 1 -- 172.21.15.78:0/2199304801 wait complete. 2024-02-19T14:53:04.556 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:53:04.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.654+0000 7f7848dd0640 1 Processor -- start 2024-02-19T14:53:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.654+0000 7f7848dd0640 1 -- start start 2024-02-19T14:53:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7848dd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152670 0x7f7844071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7848dd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7844152a50 0x7f7844155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7848dd0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78440702d0 0x7f7844158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7848dd0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7844071bf0 con 0x7f78440702d0 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7848dd0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7844158a20 con 0x7f7844152a50 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7848dd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7844155550 con 0x7f7844152670 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7842d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78440702d0 0x7f7844158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7842d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78440702d0 0x7f7844158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47124/0 (socket says 172.21.15.78:47124) 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.655+0000 7f7842d9d640 1 -- 172.21.15.78:0/2282813589 learned_addr learned my addr 172.21.15.78:0/2282813589 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f784259c640 1 --2- 172.21.15.78:0/2282813589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152670 0x7f7844071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7841d9b640 1 --2- 172.21.15.78:0/2282813589 >> v2:172.21.15.150:3300/0 conn(0x7f7844152a50 0x7f7844155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7842d9d640 1 -- 172.21.15.78:0/2282813589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152670 msgr2=0x7f7844071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7842d9d640 1 --2- 172.21.15.78:0/2282813589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152670 0x7f7844071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7842d9d640 1 -- 172.21.15.78:0/2282813589 >> v2:172.21.15.150:3300/0 conn(0x7f7844152a50 msgr2=0x7f7844155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7842d9d640 1 --2- 172.21.15.78:0/2282813589 >> v2:172.21.15.150:3300/0 conn(0x7f7844152a50 0x7f7844155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7842d9d640 1 -- 172.21.15.78:0/2282813589 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78440c6780 con 0x7f78440702d0 2024-02-19T14:53:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7841d9b640 1 --2- 172.21.15.78:0/2282813589 >> v2:172.21.15.150:3300/0 conn(0x7f7844152a50 0x7f7844155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f7842d9d640 1 --2- 172.21.15.78:0/2282813589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f7844158eb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f784406e680 tx=0x7f783c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01e10ff4b666a33 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f784159a640 1 -- 172.21.15.78:0/2282813589 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c02cb00 con 0x7f78440702d0 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.656+0000 7f784159a640 1 -- 172.21.15.78:0/2282813589 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c02cc60 con 0x7f78440702d0 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f784159a640 1 -- 172.21.15.78:0/2282813589 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c0343f0 con 0x7f78440702d0 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 -- 172.21.15.78:0/2282813589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 msgr2=0x7f7844158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 --2- 172.21.15.78:0/2282813589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f7844158eb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f784406e680 tx=0x7f783c02bf60 comp rx=0 tx=0).stop 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 -- 172.21.15.78:0/2282813589 shutdown_connections 2024-02-19T14:53:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 --2- 172.21.15.78:0/2282813589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f7844158eb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 --2- 172.21.15.78:0/2282813589 >> v2:172.21.15.150:3300/0 conn(0x7f7844152a50 0x7f7844155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 --2- 172.21.15.78:0/2282813589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152670 0x7f7844071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 -- 172.21.15.78:0/2282813589 >> 172.21.15.78:0/2282813589 conn(0x7f78440c85e0 msgr2=0x7f78440c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 -- 172.21.15.78:0/2282813589 shutdown_connections 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.657+0000 7f7848dd0640 1 -- 172.21.15.78:0/2282813589 wait complete. 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.658+0000 7f7848dd0640 1 Processor -- start 2024-02-19T14:53:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.658+0000 7f7848dd0640 1 -- start start 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7848dd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f78440ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7848dd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844152670 0x7f78441eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7848dd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152a50 0x7f7844150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7848dd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78440ad920 con 0x7f78440702d0 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7848dd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78441eb0d0 con 0x7f7844152670 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7848dd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78441ef100 con 0x7f7844152a50 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f784259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f78440ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f784259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f78440ad3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47126/0 (socket says 172.21.15.78:47126) 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f784259c640 1 -- 172.21.15.78:0/1967813397 learned_addr learned my addr 172.21.15.78:0/1967813397 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7842d9d640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152a50 0x7f7844150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.659+0000 7f7841d9b640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844152670 0x7f78441eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f784259c640 1 -- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152a50 msgr2=0x7f7844150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f784259c640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152a50 0x7f7844150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f784259c640 1 -- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844152670 msgr2=0x7f78441eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f784259c640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844152670 0x7f78441eebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f784259c640 1 -- 172.21.15.78:0/1967813397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c009b30 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f7841d9b640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844152670 0x7f78441eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f784259c640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f78440ad3e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f782c00d800 tx=0x7f782c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c0096a0 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782c009800 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78440ab570 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c009b90 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.660+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78440ab810 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.661+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78440676a0 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.661+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f782c009cf0 con 0x7f78440702d0 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.662+0000 7f783affd640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782406b810 0x7f782406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:04.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.662+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f782c08a6f0 con 0x7f78440702d0 2024-02-19T14:53:04.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.662+0000 7f7841d9b640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782406b810 0x7f782406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:04.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.662+0000 7f7841d9b640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782406b810 0x7f782406dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f784414ace0 tx=0x7f7834008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:04.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.666+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f782c054f80 con 0x7f78440702d0 2024-02-19T14:53:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.841+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f78441586b0 con 0x7f782406b810 2024-02-19T14:53:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.849+0000 7f783affd640 1 -- 172.21.15.78:0/1967813397 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289543 (secure 0 0 0) 0x7f783400a7b0 con 0x7f782406b810 2024-02-19T14:53:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782406b810 msgr2=0x7f782406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782406b810 0x7f782406dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f784414ace0 tx=0x7f7834008040 comp rx=0 tx=0).stop 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 msgr2=0x7f78440ad3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f78440ad3e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f782c00d800 tx=0x7f782c00dcc0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 shutdown_connections 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782406b810 0x7f782406dcd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7844152a50 0x7f7844150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844152670 0x7f78441eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 --2- 172.21.15.78:0/1967813397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78440702d0 0x7f78440ad3e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 >> 172.21.15.78:0/1967813397 conn(0x7f78440c85e0 msgr2=0x7f78440c6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 shutdown_connections 2024-02-19T14:53:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:04.852+0000 7f7848dd0640 1 -- 172.21.15.78:0/1967813397 wait complete. 2024-02-19T14:53:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-02-19T14:53:04.073138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220764,"kb_used_data":4564,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753956,"statfs":{"total":773094113280,"available":772868050944,"internally_reserved":0,"allocated":4673536,"data_stored":2663584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.240714"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940411+0000","last_change":"2024-02-19T14:53:01.860296+0000","last_active":"2024-02-19T14:53:02.940411+0000","last_peered":"2024-02-19T14:53:02.940411+0000","last_clean":"2024-02-19T14:53:02.940411+0000","last_became_active":"2024-02-19T14:53:01.860040+0000","last_became_peered":"2024-02-19T14:53:01.860040+0000","last_unstale":"2024-02-19T14:53:02.940411+0000","last_undegraded":"2024-02-19T14:53:02.940411+0000","last_fullsized":"2024-02-19T14:53:02.940411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:26.645451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939864+0000","last_change":"2024-02-19T14:53:01.858367+0000","last_active":"2024-02-19T14:53:02.939864+0000","last_peered":"2024-02-19T14:53:02.939864+0000","last_clean":"2024-02-19T14:53:02.939864+0000","last_became_active":"2024-02-19T14:53:01.858097+0000","last_became_peered":"2024-02-19T14:53:01.858097+0000","last_unstale":"2024-02-19T14:53:02.939864+0000","last_undegraded":"2024-02-19T14:53:02.939864+0000","last_fullsized":"2024-02-19T14:53:02.939864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:43.102502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941123+0000","last_change":"2024-02-19T14:53:01.868554+0000","last_active":"2024-02-19T14:53:02.941123+0000","last_peered":"2024-02-19T14:53:02.941123+0000","last_clean":"2024-02-19T14:53:02.941123+0000","last_became_active":"2024-02-19T14:53:01.868287+0000","last_became_peered":"2024-02-19T14:53:01.868287+0000","last_unstale":"2024-02-19T14:53:02.941123+0000","last_undegraded":"2024-02-19T14:53:02.941123+0000","last_fullsized":"2024-02-19T14:53:02.941123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:12:50.114024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857887+0000","last_change":"2024-02-19T14:53:01.856179+0000","last_active":"2024-02-19T14:53:02.857887+0000","last_peered":"2024-02-19T14:53:02.857887+0000","last_clean":"2024-02-19T14:53:02.857887+0000","last_became_active":"2024-02-19T14:53:01.855913+0000","last_became_peered":"2024-02-19T14:53:01.855913+0000","last_unstale":"2024-02-19T14:53:02.857887+0000","last_undegraded":"2024-02-19T14:53:02.857887+0000","last_fullsized":"2024-02-19T14:53:02.857887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:05.638740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857848+0000","last_change":"2024-02-19T14:53:01.855429+0000","last_active":"2024-02-19T14:53:02.857848+0000","last_peered":"2024-02-19T14:53:02.857848+0000","last_clean":"2024-02-19T14:53:02.857848+0000","last_became_active":"2024-02-19T14:53:01.855159+0000","last_became_peered":"2024-02-19T14:53:01.855159+0000","last_unstale":"2024-02-19T14:53:02.857848+0000","last_undegraded":"2024-02-19T14:53:02.857848+0000","last_fullsized":"2024-02-19T14:53:02.857848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:06.532400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940038+0000","last_change":"2024-02-19T14:53:01.854848+0000","last_active":"2024-02-19T14:53:02.940038+0000","last_peered":"2024-02-19T14:53:02.940038+0000","last_clean":"2024-02-19T14:53:02.940038+0000","last_became_active":"2024-02-19T14:53:01.854597+0000","last_became_peered":"2024-02-19T14:53:01.854597+0000","last_unstale":"2024-02-19T14:53:02.940038+0000","last_undegraded":"2024-02-19T14:53:02.940038+0000","last_fullsized":"2024-02-19T14:53:02.940038+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:54.754384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940126+0000","last_change":"2024-02-19T14:53:01.858729+0000","last_active":"2024-02-19T14:53:02.940126+0000","last_peered":"2024-02-19T14:53:02.940126+0000","last_clean":"2024-02-19T14:53:02.940126+0000","last_became_active":"2024-02-19T14:53:01.858460+0000","last_became_peered":"2024-02-19T14:53:01.858460+0000","last_unstale":"2024-02-19T14:53:02.940126+0000","last_undegraded":"2024-02-19T14:53:02.940126+0000","last_fullsized":"2024-02-19T14:53:02.940126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:43.102502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939767+0000","last_change":"2024-02-19T14:53:01.862373+0000","last_active":"2024-02-19T14:53:02.939767+0000","last_peered":"2024-02-19T14:53:02.939767+0000","last_clean":"2024-02-19T14:53:02.939767+0000","last_became_active":"2024-02-19T14:53:01.862114+0000","last_became_peered":"2024-02-19T14:53:01.862114+0000","last_unstale":"2024-02-19T14:53:02.939767+0000","last_undegraded":"2024-02-19T14:53:02.939767+0000","last_fullsized":"2024-02-19T14:53:02.939767+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:01.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941348+0000","last_change":"2024-02-19T14:53:01.867857+0000","last_active":"2024-02-19T14:53:02.941348+0000","last_peered":"2024-02-19T14:53:02.941348+0000","last_clean":"2024-02-19T14:53:02.941348+0000","last_became_active":"2024-02-19T14:53:01.867609+0000","last_became_peered":"2024-02-19T14:53:01.867609+0000","last_unstale":"2024-02-19T14:53:02.941348+0000","last_undegraded":"2024-02-19T14:53:02.941348+0000","last_fullsized":"2024-02-19T14:53:02.941348+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:02.908488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940771+0000","last_change":"2024-02-19T14:53:01.860504+0000","last_active":"2024-02-19T14:53:02.940771+0000","last_peered":"2024-02-19T14:53:02.940771+0000","last_clean":"2024-02-19T14:53:02.940771+0000","last_became_active":"2024-02-19T14:53:01.860167+0000","last_became_peered":"2024-02-19T14:53:01.860167+0000","last_unstale":"2024-02-19T14:53:02.940771+0000","last_undegraded":"2024-02-19T14:53:02.940771+0000","last_fullsized":"2024-02-19T14:53:02.940771+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:11.625898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940763+0000","last_change":"2024-02-19T14:53:01.853369+0000","last_active":"2024-02-19T14:53:02.940763+0000","last_peered":"2024-02-19T14:53:02.940763+0000","last_clean":"2024-02-19T14:53:02.940763+0000","last_became_active":"2024-02-19T14:53:01.853020+0000","last_became_peered":"2024-02-19T14:53:01.853020+0000","last_unstale":"2024-02-19T14:53:02.940763+0000","last_undegraded":"2024-02-19T14:53:02.940763+0000","last_fullsized":"2024-02-19T14:53:02.940763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:37.403255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940977+0000","last_change":"2024-02-19T14:53:01.859058+0000","last_active":"2024-02-19T14:53:02.940977+0000","last_peered":"2024-02-19T14:53:02.940977+0000","last_clean":"2024-02-19T14:53:02.940977+0000","last_became_active":"2024-02-19T14:53:01.858805+0000","last_became_peered":"2024-02-19T14:53:01.858805+0000","last_unstale":"2024-02-19T14:53:02.940977+0000","last_undegraded":"2024-02-19T14:53:02.940977+0000","last_fullsized":"2024-02-19T14:53:02.940977+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:58:08.710699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940267+0000","last_change":"2024-02-19T14:53:01.861276+0000","last_active":"2024-02-19T14:53:02.940267+0000","last_peered":"2024-02-19T14:53:02.940267+0000","last_clean":"2024-02-19T14:53:02.940267+0000","last_became_active":"2024-02-19T14:53:01.861013+0000","last_became_peered":"2024-02-19T14:53:01.861013+0000","last_unstale":"2024-02-19T14:53:02.940267+0000","last_undegraded":"2024-02-19T14:53:02.940267+0000","last_fullsized":"2024-02-19T14:53:02.940267+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:06.083865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858464+0000","last_change":"2024-02-19T14:53:01.851730+0000","last_active":"2024-02-19T14:53:02.858464+0000","last_peered":"2024-02-19T14:53:02.858464+0000","last_clean":"2024-02-19T14:53:02.858464+0000","last_became_active":"2024-02-19T14:53:01.851476+0000","last_became_peered":"2024-02-19T14:53:01.851476+0000","last_unstale":"2024-02-19T14:53:02.858464+0000","last_undegraded":"2024-02-19T14:53:02.858464+0000","last_fullsized":"2024-02-19T14:53:02.858464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:20.426618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940645+0000","last_change":"2024-02-19T14:53:01.858068+0000","last_active":"2024-02-19T14:53:02.940645+0000","last_peered":"2024-02-19T14:53:02.940645+0000","last_clean":"2024-02-19T14:53:02.940645+0000","last_became_active":"2024-02-19T14:53:01.857745+0000","last_became_peered":"2024-02-19T14:53:01.857745+0000","last_unstale":"2024-02-19T14:53:02.940645+0000","last_undegraded":"2024-02-19T14:53:02.940645+0000","last_fullsized":"2024-02-19T14:53:02.940645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:26.645451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939992+0000","last_change":"2024-02-19T14:53:01.863622+0000","last_active":"2024-02-19T14:53:02.939992+0000","last_peered":"2024-02-19T14:53:02.939992+0000","last_clean":"2024-02-19T14:53:02.939992+0000","last_became_active":"2024-02-19T14:53:01.863054+0000","last_became_peered":"2024-02-19T14:53:01.863054+0000","last_unstale":"2024-02-19T14:53:02.939992+0000","last_undegraded":"2024-02-19T14:53:02.939992+0000","last_fullsized":"2024-02-19T14:53:02.939992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:12:50.114024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940679+0000","last_change":"2024-02-19T14:53:01.867250+0000","last_active":"2024-02-19T14:53:02.940679+0000","last_peered":"2024-02-19T14:53:02.940679+0000","last_clean":"2024-02-19T14:53:02.940679+0000","last_became_active":"2024-02-19T14:53:01.866986+0000","last_became_peered":"2024-02-19T14:53:01.866986+0000","last_unstale":"2024-02-19T14:53:02.940679+0000","last_undegraded":"2024-02-19T14:53:02.940679+0000","last_fullsized":"2024-02-19T14:53:02.940679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:32:37.025792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859030+0000","last_change":"2024-02-19T14:53:01.861574+0000","last_active":"2024-02-19T14:53:02.859030+0000","last_peered":"2024-02-19T14:53:02.859030+0000","last_clean":"2024-02-19T14:53:02.859030+0000","last_became_active":"2024-02-19T14:53:01.861245+0000","last_became_peered":"2024-02-19T14:53:01.861245+0000","last_unstale":"2024-02-19T14:53:02.859030+0000","last_undegraded":"2024-02-19T14:53:02.859030+0000","last_fullsized":"2024-02-19T14:53:02.859030+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:34.269975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857785+0000","last_change":"2024-02-19T14:53:01.857279+0000","last_active":"2024-02-19T14:53:02.857785+0000","last_peered":"2024-02-19T14:53:02.857785+0000","last_clean":"2024-02-19T14:53:02.857785+0000","last_became_active":"2024-02-19T14:53:01.856985+0000","last_became_peered":"2024-02-19T14:53:01.856985+0000","last_unstale":"2024-02-19T14:53:02.857785+0000","last_undegraded":"2024-02-19T14:53:02.857785+0000","last_fullsized":"2024-02-19T14:53:02.857785+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:09.706011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858886+0000","last_change":"2024-02-19T14:53:01.862398+0000","last_active":"2024-02-19T14:53:02.858886+0000","last_peered":"2024-02-19T14:53:02.858886+0000","last_clean":"2024-02-19T14:53:02.858886+0000","last_became_active":"2024-02-19T14:53:01.862089+0000","last_became_peered":"2024-02-19T14:53:01.862089+0000","last_unstale":"2024-02-19T14:53:02.858886+0000","last_undegraded":"2024-02-19T14:53:02.858886+0000","last_fullsized":"2024-02-19T14:53:02.858886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:37.556036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940335+0000","last_change":"2024-02-19T14:53:01.859863+0000","last_active":"2024-02-19T14:53:02.940335+0000","last_peered":"2024-02-19T14:53:02.940335+0000","last_clean":"2024-02-19T14:53:02.940335+0000","last_became_active":"2024-02-19T14:53:01.859615+0000","last_became_peered":"2024-02-19T14:53:01.859615+0000","last_unstale":"2024-02-19T14:53:02.940335+0000","last_undegraded":"2024-02-19T14:53:02.940335+0000","last_fullsized":"2024-02-19T14:53:02.940335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:08.912678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858659+0000","last_change":"2024-02-19T14:53:01.860920+0000","last_active":"2024-02-19T14:53:02.858659+0000","last_peered":"2024-02-19T14:53:02.858659+0000","last_clean":"2024-02-19T14:53:02.858659+0000","last_became_active":"2024-02-19T14:53:01.860651+0000","last_became_peered":"2024-02-19T14:53:01.860651+0000","last_unstale":"2024-02-19T14:53:02.858659+0000","last_undegraded":"2024-02-19T14:53:02.858659+0000","last_fullsized":"2024-02-19T14:53:02.858659+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:34.269975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941082+0000","last_change":"2024-02-19T14:53:01.867092+0000","last_active":"2024-02-19T14:53:02.941082+0000","last_peered":"2024-02-19T14:53:02.941082+0000","last_clean":"2024-02-19T14:53:02.941082+0000","last_became_active":"2024-02-19T14:53:01.866414+0000","last_became_peered":"2024-02-19T14:53:01.866414+0000","last_unstale":"2024-02-19T14:53:02.941082+0000","last_undegraded":"2024-02-19T14:53:02.941082+0000","last_fullsized":"2024-02-19T14:53:02.941082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:01.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857566+0000","last_change":"2024-02-19T14:53:01.858380+0000","last_active":"2024-02-19T14:53:02.857566+0000","last_peered":"2024-02-19T14:53:02.857566+0000","last_clean":"2024-02-19T14:53:02.857566+0000","last_became_active":"2024-02-19T14:53:01.858106+0000","last_became_peered":"2024-02-19T14:53:01.858106+0000","last_unstale":"2024-02-19T14:53:02.857566+0000","last_undegraded":"2024-02-19T14:53:02.857566+0000","last_fullsized":"2024-02-19T14:53:02.857566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:36.025364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858973+0000","last_change":"2024-02-19T14:53:01.858430+0000","last_active":"2024-02-19T14:53:02.858973+0000","last_peered":"2024-02-19T14:53:02.858973+0000","last_clean":"2024-02-19T14:53:02.858973+0000","last_became_active":"2024-02-19T14:53:01.858157+0000","last_became_peered":"2024-02-19T14:53:01.858157+0000","last_unstale":"2024-02-19T14:53:02.858973+0000","last_undegraded":"2024-02-19T14:53:02.858973+0000","last_fullsized":"2024-02-19T14:53:02.858973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.086237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940362+0000","last_change":"2024-02-19T14:53:01.863007+0000","last_active":"2024-02-19T14:53:02.940362+0000","last_peered":"2024-02-19T14:53:02.940362+0000","last_clean":"2024-02-19T14:53:02.940362+0000","last_became_active":"2024-02-19T14:53:01.862744+0000","last_became_peered":"2024-02-19T14:53:01.862744+0000","last_unstale":"2024-02-19T14:53:02.940362+0000","last_undegraded":"2024-02-19T14:53:02.940362+0000","last_fullsized":"2024-02-19T14:53:02.940362+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:02.908488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858371+0000","last_change":"2024-02-19T14:53:01.856728+0000","last_active":"2024-02-19T14:53:02.858371+0000","last_peered":"2024-02-19T14:53:02.858371+0000","last_clean":"2024-02-19T14:53:02.858371+0000","last_became_active":"2024-02-19T14:53:01.856443+0000","last_became_peered":"2024-02-19T14:53:01.856443+0000","last_unstale":"2024-02-19T14:53:02.858371+0000","last_undegraded":"2024-02-19T14:53:02.858371+0000","last_fullsized":"2024-02-19T14:53:02.858371+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:39.871295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940987+0000","last_change":"2024-02-19T14:53:01.852318+0000","last_active":"2024-02-19T14:53:02.940987+0000","last_peered":"2024-02-19T14:53:02.940987+0000","last_clean":"2024-02-19T14:53:02.940987+0000","last_became_active":"2024-02-19T14:53:01.851153+0000","last_became_peered":"2024-02-19T14:53:01.851153+0000","last_unstale":"2024-02-19T14:53:02.940987+0000","last_undegraded":"2024-02-19T14:53:02.940987+0000","last_fullsized":"2024-02-19T14:53:02.940987+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:46:42.210280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858302+0000","last_change":"2024-02-19T14:53:01.860434+0000","last_active":"2024-02-19T14:53:02.858302+0000","last_peered":"2024-02-19T14:53:02.858302+0000","last_clean":"2024-02-19T14:53:02.858302+0000","last_became_active":"2024-02-19T14:53:01.859566+0000","last_became_peered":"2024-02-19T14:53:01.859566+0000","last_unstale":"2024-02-19T14:53:02.858302+0000","last_undegraded":"2024-02-19T14:53:02.858302+0000","last_fullsized":"2024-02-19T14:53:02.858302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:02.469083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858593+0000","last_change":"2024-02-19T14:53:01.855743+0000","last_active":"2024-02-19T14:53:02.858593+0000","last_peered":"2024-02-19T14:53:02.858593+0000","last_clean":"2024-02-19T14:53:02.858593+0000","last_became_active":"2024-02-19T14:53:01.854551+0000","last_became_peered":"2024-02-19T14:53:01.854551+0000","last_unstale":"2024-02-19T14:53:02.858593+0000","last_undegraded":"2024-02-19T14:53:02.858593+0000","last_fullsized":"2024-02-19T14:53:02.858593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:20.426618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940905+0000","last_change":"2024-02-19T14:53:01.867418+0000","last_active":"2024-02-19T14:53:02.940905+0000","last_peered":"2024-02-19T14:53:02.940905+0000","last_clean":"2024-02-19T14:53:02.940905+0000","last_became_active":"2024-02-19T14:53:01.866803+0000","last_became_peered":"2024-02-19T14:53:01.866803+0000","last_unstale":"2024-02-19T14:53:02.940905+0000","last_undegraded":"2024-02-19T14:53:02.940905+0000","last_fullsized":"2024-02-19T14:53:02.940905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:08.515292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940203+0000","last_change":"2024-02-19T14:53:01.856171+0000","last_active":"2024-02-19T14:53:02.940203+0000","last_peered":"2024-02-19T14:53:02.940203+0000","last_clean":"2024-02-19T14:53:02.940203+0000","last_became_active":"2024-02-19T14:53:01.855924+0000","last_became_peered":"2024-02-19T14:53:01.855924+0000","last_unstale":"2024-02-19T14:53:02.940203+0000","last_undegraded":"2024-02-19T14:53:02.940203+0000","last_fullsized":"2024-02-19T14:53:02.940203+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:12:50.114024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941085+0000","last_change":"2024-02-19T14:53:01.868001+0000","last_active":"2024-02-19T14:53:02.941085+0000","last_peered":"2024-02-19T14:53:02.941085+0000","last_clean":"2024-02-19T14:53:02.941085+0000","last_became_active":"2024-02-19T14:53:01.867724+0000","last_became_peered":"2024-02-19T14:53:01.867724+0000","last_unstale":"2024-02-19T14:53:02.941085+0000","last_undegraded":"2024-02-19T14:53:02.941085+0000","last_fullsized":"2024-02-19T14:53:02.941085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:43.102502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859009+0000","last_change":"2024-02-19T14:53:00.851798+0000","last_active":"2024-02-19T14:53:02.859009+0000","last_peered":"2024-02-19T14:53:02.859009+0000","last_clean":"2024-02-19T14:53:02.859009+0000","last_became_active":"2024-02-19T14:53:00.851517+0000","last_became_peered":"2024-02-19T14:53:00.851517+0000","last_unstale":"2024-02-19T14:53:02.859009+0000","last_undegraded":"2024-02-19T14:53:02.859009+0000","last_fullsized":"2024-02-19T14:53:02.859009+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:35.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940129+0000","last_change":"2024-02-19T14:53:01.862922+0000","last_active":"2024-02-19T14:53:02.940129+0000","last_peered":"2024-02-19T14:53:02.940129+0000","last_clean":"2024-02-19T14:53:02.940129+0000","last_became_active":"2024-02-19T14:53:01.862630+0000","last_became_peered":"2024-02-19T14:53:01.862630+0000","last_unstale":"2024-02-19T14:53:02.940129+0000","last_undegraded":"2024-02-19T14:53:02.940129+0000","last_fullsized":"2024-02-19T14:53:02.940129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:58:08.710699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941205+0000","last_change":"2024-02-19T14:53:00.856064+0000","last_active":"2024-02-19T14:53:02.941205+0000","last_peered":"2024-02-19T14:53:02.941205+0000","last_clean":"2024-02-19T14:53:02.941205+0000","last_became_active":"2024-02-19T14:53:00.855799+0000","last_became_peered":"2024-02-19T14:53:00.855799+0000","last_unstale":"2024-02-19T14:53:02.941205+0000","last_undegraded":"2024-02-19T14:53:02.941205+0000","last_fullsized":"2024-02-19T14:53:02.941205+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:35.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858357+0000","last_change":"2024-02-19T14:53:01.852861+0000","last_active":"2024-02-19T14:53:02.858357+0000","last_peered":"2024-02-19T14:53:02.858357+0000","last_clean":"2024-02-19T14:53:02.858357+0000","last_became_active":"2024-02-19T14:53:01.852537+0000","last_became_peered":"2024-02-19T14:53:01.852537+0000","last_unstale":"2024-02-19T14:53:02.858357+0000","last_undegraded":"2024-02-19T14:53:02.858357+0000","last_fullsized":"2024-02-19T14:53:02.858357+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:02.469083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940214+0000","last_change":"2024-02-19T14:53:00.857583+0000","last_active":"2024-02-19T14:53:02.940214+0000","last_peered":"2024-02-19T14:53:02.940214+0000","last_clean":"2024-02-19T14:53:02.940214+0000","last_became_active":"2024-02-19T14:53:00.857297+0000","last_became_peered":"2024-02-19T14:53:00.857297+0000","last_unstale":"2024-02-19T14:53:02.940214+0000","last_undegraded":"2024-02-19T14:53:02.940214+0000","last_fullsized":"2024-02-19T14:53:02.940214+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:52:00.224317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940286+0000","last_change":"2024-02-19T14:53:01.853039+0000","last_active":"2024-02-19T14:53:02.940286+0000","last_peered":"2024-02-19T14:53:02.940286+0000","last_clean":"2024-02-19T14:53:02.940286+0000","last_became_active":"2024-02-19T14:53:01.852533+0000","last_became_peered":"2024-02-19T14:53:01.852533+0000","last_unstale":"2024-02-19T14:53:02.940286+0000","last_undegraded":"2024-02-19T14:53:02.940286+0000","last_fullsized":"2024-02-19T14:53:02.940286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:28.528999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859045+0000","last_change":"2024-02-19T14:53:00.856489+0000","last_active":"2024-02-19T14:53:02.859045+0000","last_peered":"2024-02-19T14:53:02.859045+0000","last_clean":"2024-02-19T14:53:02.859045+0000","last_became_active":"2024-02-19T14:53:00.856111+0000","last_became_peered":"2024-02-19T14:53:00.856111+0000","last_unstale":"2024-02-19T14:53:02.859045+0000","last_undegraded":"2024-02-19T14:53:02.859045+0000","last_fullsized":"2024-02-19T14:53:02.859045+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:45.255899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939701+0000","last_change":"2024-02-19T14:53:01.860741+0000","last_active":"2024-02-19T14:53:02.939701+0000","last_peered":"2024-02-19T14:53:02.939701+0000","last_clean":"2024-02-19T14:53:02.939701+0000","last_became_active":"2024-02-19T14:53:01.860460+0000","last_became_peered":"2024-02-19T14:53:01.860460+0000","last_unstale":"2024-02-19T14:53:02.939701+0000","last_undegraded":"2024-02-19T14:53:02.939701+0000","last_fullsized":"2024-02-19T14:53:02.939701+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:37.556036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940486+0000","last_change":"2024-02-19T14:53:00.860495+0000","last_active":"2024-02-19T14:53:02.940486+0000","last_peered":"2024-02-19T14:53:02.940486+0000","last_clean":"2024-02-19T14:53:02.940486+0000","last_became_active":"2024-02-19T14:53:00.860213+0000","last_became_peered":"2024-02-19T14:53:00.860213+0000","last_unstale":"2024-02-19T14:53:02.940486+0000","last_undegraded":"2024-02-19T14:53:02.940486+0000","last_fullsized":"2024-02-19T14:53:02.940486+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:19.424285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940399+0000","last_change":"2024-02-19T14:53:01.855671+0000","last_active":"2024-02-19T14:53:02.940399+0000","last_peered":"2024-02-19T14:53:02.940399+0000","last_clean":"2024-02-19T14:53:02.940399+0000","last_became_active":"2024-02-19T14:53:01.855386+0000","last_became_peered":"2024-02-19T14:53:01.855386+0000","last_unstale":"2024-02-19T14:53:02.940399+0000","last_undegraded":"2024-02-19T14:53:02.940399+0000","last_fullsized":"2024-02-19T14:53:02.940399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:28.528999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940251+0000","last_change":"2024-02-19T14:53:00.859368+0000","last_active":"2024-02-19T14:53:02.940251+0000","last_peered":"2024-02-19T14:53:02.940251+0000","last_clean":"2024-02-19T14:53:02.940251+0000","last_became_active":"2024-02-19T14:53:00.858991+0000","last_became_peered":"2024-02-19T14:53:00.858991+0000","last_unstale":"2024-02-19T14:53:02.940251+0000","last_undegraded":"2024-02-19T14:53:02.940251+0000","last_fullsized":"2024-02-19T14:53:02.940251+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:05.530067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858794+0000","last_change":"2024-02-19T14:53:01.851134+0000","last_active":"2024-02-19T14:53:02.858794+0000","last_peered":"2024-02-19T14:53:02.858794+0000","last_clean":"2024-02-19T14:53:02.858794+0000","last_became_active":"2024-02-19T14:53:01.850808+0000","last_became_peered":"2024-02-19T14:53:01.850808+0000","last_unstale":"2024-02-19T14:53:02.858794+0000","last_undegraded":"2024-02-19T14:53:02.858794+0000","last_fullsized":"2024-02-19T14:53:02.858794+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:54.754384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940668+0000","last_change":"2024-02-19T14:53:00.858049+0000","last_active":"2024-02-19T14:53:02.940668+0000","last_peered":"2024-02-19T14:53:02.940668+0000","last_clean":"2024-02-19T14:53:02.940668+0000","last_became_active":"2024-02-19T14:53:00.857749+0000","last_became_peered":"2024-02-19T14:53:00.857749+0000","last_unstale":"2024-02-19T14:53:02.940668+0000","last_undegraded":"2024-02-19T14:53:02.940668+0000","last_fullsized":"2024-02-19T14:53:02.940668+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:56.443441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940711+0000","last_change":"2024-02-19T14:53:01.861410+0000","last_active":"2024-02-19T14:53:02.940711+0000","last_peered":"2024-02-19T14:53:02.940711+0000","last_clean":"2024-02-19T14:53:02.940711+0000","last_became_active":"2024-02-19T14:53:01.861170+0000","last_became_peered":"2024-02-19T14:53:01.861170+0000","last_unstale":"2024-02-19T14:53:02.940711+0000","last_undegraded":"2024-02-19T14:53:02.940711+0000","last_fullsized":"2024-02-19T14:53:02.940711+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:37.403255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858080+0000","last_change":"2024-02-19T14:53:00.847884+0000","last_active":"2024-02-19T14:53:02.858080+0000","last_peered":"2024-02-19T14:53:02.858080+0000","last_clean":"2024-02-19T14:53:02.858080+0000","last_became_active":"2024-02-19T14:53:00.847615+0000","last_became_peered":"2024-02-19T14:53:00.847615+0000","last_unstale":"2024-02-19T14:53:02.858080+0000","last_undegraded":"2024-02-19T14:53:02.858080+0000","last_fullsized":"2024-02-19T14:53:02.858080+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:35.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857795+0000","last_change":"2024-02-19T14:53:01.856971+0000","last_active":"2024-02-19T14:53:02.857795+0000","last_peered":"2024-02-19T14:53:02.857795+0000","last_clean":"2024-02-19T14:53:02.857795+0000","last_became_active":"2024-02-19T14:53:01.856654+0000","last_became_peered":"2024-02-19T14:53:01.856654+0000","last_unstale":"2024-02-19T14:53:02.857795+0000","last_undegraded":"2024-02-19T14:53:02.857795+0000","last_fullsized":"2024-02-19T14:53:02.857795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:45.812182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940400+0000","last_change":"2024-02-19T14:53:00.851234+0000","last_active":"2024-02-19T14:53:02.940400+0000","last_peered":"2024-02-19T14:53:02.940400+0000","last_clean":"2024-02-19T14:53:02.940400+0000","last_became_active":"2024-02-19T14:53:00.850983+0000","last_became_peered":"2024-02-19T14:53:00.850983+0000","last_unstale":"2024-02-19T14:53:02.940400+0000","last_undegraded":"2024-02-19T14:53:02.940400+0000","last_fullsized":"2024-02-19T14:53:02.940400+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:28.083904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940339+0000","last_change":"2024-02-19T14:53:01.871178+0000","last_active":"2024-02-19T14:53:02.940339+0000","last_peered":"2024-02-19T14:53:02.940339+0000","last_clean":"2024-02-19T14:53:02.940339+0000","last_became_active":"2024-02-19T14:53:01.870867+0000","last_became_peered":"2024-02-19T14:53:01.870867+0000","last_unstale":"2024-02-19T14:53:02.940339+0000","last_undegraded":"2024-02-19T14:53:02.940339+0000","last_fullsized":"2024-02-19T14:53:02.940339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:02.908488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941162+0000","last_change":"2024-02-19T14:53:00.855894+0000","last_active":"2024-02-19T14:53:02.941162+0000","last_peered":"2024-02-19T14:53:02.941162+0000","last_clean":"2024-02-19T14:53:02.941162+0000","last_became_active":"2024-02-19T14:53:00.855396+0000","last_became_peered":"2024-02-19T14:53:00.855396+0000","last_unstale":"2024-02-19T14:53:02.941162+0000","last_undegraded":"2024-02-19T14:53:02.941162+0000","last_fullsized":"2024-02-19T14:53:02.941162+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:36.553704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941232+0000","last_change":"2024-02-19T14:53:01.865982+0000","last_active":"2024-02-19T14:53:02.941232+0000","last_peered":"2024-02-19T14:53:02.941232+0000","last_clean":"2024-02-19T14:53:02.941232+0000","last_became_active":"2024-02-19T14:53:01.865707+0000","last_became_peered":"2024-02-19T14:53:01.865707+0000","last_unstale":"2024-02-19T14:53:02.941232+0000","last_undegraded":"2024-02-19T14:53:02.941232+0000","last_fullsized":"2024-02-19T14:53:02.941232+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:54:59.322422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940916+0000","last_change":"2024-02-19T14:53:00.860141+0000","last_active":"2024-02-19T14:53:02.940916+0000","last_peered":"2024-02-19T14:53:02.940916+0000","last_clean":"2024-02-19T14:53:02.940916+0000","last_became_active":"2024-02-19T14:53:00.859865+0000","last_became_peered":"2024-02-19T14:53:00.859865+0000","last_unstale":"2024-02-19T14:53:02.940916+0000","last_undegraded":"2024-02-19T14:53:02.940916+0000","last_fullsized":"2024-02-19T14:53:02.940916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:38.868963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858533+0000","last_change":"2024-02-19T14:53:01.861637+0000","last_active":"2024-02-19T14:53:02.858533+0000","last_peered":"2024-02-19T14:53:02.858533+0000","last_clean":"2024-02-19T14:53:02.858533+0000","last_became_active":"2024-02-19T14:53:01.860616+0000","last_became_peered":"2024-02-19T14:53:01.860616+0000","last_unstale":"2024-02-19T14:53:02.858533+0000","last_undegraded":"2024-02-19T14:53:02.858533+0000","last_fullsized":"2024-02-19T14:53:02.858533+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:37.556036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940559+0000","last_change":"2024-02-19T14:53:00.855986+0000","last_active":"2024-02-19T14:53:02.940559+0000","last_peered":"2024-02-19T14:53:02.940559+0000","last_clean":"2024-02-19T14:53:02.940559+0000","last_became_active":"2024-02-19T14:53:00.855720+0000","last_became_peered":"2024-02-19T14:53:00.855720+0000","last_unstale":"2024-02-19T14:53:02.940559+0000","last_undegraded":"2024-02-19T14:53:02.940559+0000","last_fullsized":"2024-02-19T14:53:02.940559+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.267643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939701+0000","last_change":"2024-02-19T14:53:01.868599+0000","last_active":"2024-02-19T14:53:02.939701+0000","last_peered":"2024-02-19T14:53:02.939701+0000","last_clean":"2024-02-19T14:53:02.939701+0000","last_became_active":"2024-02-19T14:53:01.868321+0000","last_became_peered":"2024-02-19T14:53:01.868321+0000","last_unstale":"2024-02-19T14:53:02.939701+0000","last_undegraded":"2024-02-19T14:53:02.939701+0000","last_fullsized":"2024-02-19T14:53:02.939701+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.086237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940851+0000","last_change":"2024-02-19T14:53:00.855971+0000","last_active":"2024-02-19T14:53:02.940851+0000","last_peered":"2024-02-19T14:53:02.940851+0000","last_clean":"2024-02-19T14:53:02.940851+0000","last_became_active":"2024-02-19T14:53:00.855710+0000","last_became_peered":"2024-02-19T14:53:00.855710+0000","last_unstale":"2024-02-19T14:53:02.940851+0000","last_undegraded":"2024-02-19T14:53:02.940851+0000","last_fullsized":"2024-02-19T14:53:02.940851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:36.553704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940286+0000","last_change":"2024-02-19T14:53:01.859675+0000","last_active":"2024-02-19T14:53:02.940286+0000","last_peered":"2024-02-19T14:53:02.940286+0000","last_clean":"2024-02-19T14:53:02.940286+0000","last_became_active":"2024-02-19T14:53:01.859401+0000","last_became_peered":"2024-02-19T14:53:01.859401+0000","last_unstale":"2024-02-19T14:53:02.940286+0000","last_undegraded":"2024-02-19T14:53:02.940286+0000","last_fullsized":"2024-02-19T14:53:02.940286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:08.912678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940026+0000","last_change":"2024-02-19T14:53:00.856625+0000","last_active":"2024-02-19T14:53:02.940026+0000","last_peered":"2024-02-19T14:53:02.940026+0000","last_clean":"2024-02-19T14:53:02.940026+0000","last_became_active":"2024-02-19T14:53:00.856378+0000","last_became_peered":"2024-02-19T14:53:00.856378+0000","last_unstale":"2024-02-19T14:53:02.940026+0000","last_undegraded":"2024-02-19T14:53:02.940026+0000","last_fullsized":"2024-02-19T14:53:02.940026+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:03.270041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941066+0000","last_change":"2024-02-19T14:53:01.867129+0000","last_active":"2024-02-19T14:53:02.941066+0000","last_peered":"2024-02-19T14:53:02.941066+0000","last_clean":"2024-02-19T14:53:02.941066+0000","last_became_active":"2024-02-19T14:53:01.865959+0000","last_became_peered":"2024-02-19T14:53:01.865959+0000","last_unstale":"2024-02-19T14:53:02.941066+0000","last_undegraded":"2024-02-19T14:53:02.941066+0000","last_fullsized":"2024-02-19T14:53:02.941066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:55.254128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858606+0000","last_change":"2024-02-19T14:53:00.858256+0000","last_active":"2024-02-19T14:53:02.858606+0000","last_peered":"2024-02-19T14:53:02.858606+0000","last_clean":"2024-02-19T14:53:02.858606+0000","last_became_active":"2024-02-19T14:53:00.857963+0000","last_became_peered":"2024-02-19T14:53:00.857963+0000","last_unstale":"2024-02-19T14:53:02.858606+0000","last_undegraded":"2024-02-19T14:53:02.858606+0000","last_fullsized":"2024-02-19T14:53:02.858606+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:08.703678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940753+0000","last_change":"2024-02-19T14:53:01.860417+0000","last_active":"2024-02-19T14:53:02.940753+0000","last_peered":"2024-02-19T14:53:02.940753+0000","last_clean":"2024-02-19T14:53:02.940753+0000","last_became_active":"2024-02-19T14:53:01.860128+0000","last_became_peered":"2024-02-19T14:53:01.860128+0000","last_unstale":"2024-02-19T14:53:02.940753+0000","last_undegraded":"2024-02-19T14:53:02.940753+0000","last_fullsized":"2024-02-19T14:53:02.940753+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:11.625898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940033+0000","last_change":"2024-02-19T14:53:00.858837+0000","last_active":"2024-02-19T14:53:02.940033+0000","last_peered":"2024-02-19T14:53:02.940033+0000","last_clean":"2024-02-19T14:53:02.940033+0000","last_became_active":"2024-02-19T14:53:00.858545+0000","last_became_peered":"2024-02-19T14:53:00.858545+0000","last_unstale":"2024-02-19T14:53:02.940033+0000","last_undegraded":"2024-02-19T14:53:02.940033+0000","last_fullsized":"2024-02-19T14:53:02.940033+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:44.809850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940550+0000","last_change":"2024-02-19T14:53:01.853452+0000","last_active":"2024-02-19T14:53:02.940550+0000","last_peered":"2024-02-19T14:53:02.940550+0000","last_clean":"2024-02-19T14:53:02.940550+0000","last_became_active":"2024-02-19T14:53:01.853197+0000","last_became_peered":"2024-02-19T14:53:01.853197+0000","last_unstale":"2024-02-19T14:53:02.940550+0000","last_undegraded":"2024-02-19T14:53:02.940550+0000","last_fullsized":"2024-02-19T14:53:02.940550+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940916+0000","last_change":"2024-02-19T14:53:00.858046+0000","last_active":"2024-02-19T14:53:02.940916+0000","last_peered":"2024-02-19T14:53:02.940916+0000","last_clean":"2024-02-19T14:53:02.940916+0000","last_became_active":"2024-02-19T14:53:00.857764+0000","last_became_peered":"2024-02-19T14:53:00.857764+0000","last_unstale":"2024-02-19T14:53:02.940916+0000","last_undegraded":"2024-02-19T14:53:02.940916+0000","last_fullsized":"2024-02-19T14:53:02.940916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:28.083904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940889+0000","last_change":"2024-02-19T14:53:01.856636+0000","last_active":"2024-02-19T14:53:02.940889+0000","last_peered":"2024-02-19T14:53:02.940889+0000","last_clean":"2024-02-19T14:53:02.940889+0000","last_became_active":"2024-02-19T14:53:01.856071+0000","last_became_peered":"2024-02-19T14:53:01.856071+0000","last_unstale":"2024-02-19T14:53:02.940889+0000","last_undegraded":"2024-02-19T14:53:02.940889+0000","last_fullsized":"2024-02-19T14:53:02.940889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:49.806973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858462+0000","last_change":"2024-02-19T14:53:00.855690+0000","last_active":"2024-02-19T14:53:02.858462+0000","last_peered":"2024-02-19T14:53:02.858462+0000","last_clean":"2024-02-19T14:53:02.858462+0000","last_became_active":"2024-02-19T14:53:00.855418+0000","last_became_peered":"2024-02-19T14:53:00.855418+0000","last_unstale":"2024-02-19T14:53:02.858462+0000","last_undegraded":"2024-02-19T14:53:02.858462+0000","last_fullsized":"2024-02-19T14:53:02.858462+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:08.703678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858463+0000","last_change":"2024-02-19T14:53:01.858809+0000","last_active":"2024-02-19T14:53:02.858463+0000","last_peered":"2024-02-19T14:53:02.858463+0000","last_clean":"2024-02-19T14:53:02.858463+0000","last_became_active":"2024-02-19T14:53:01.858526+0000","last_became_peered":"2024-02-19T14:53:01.858526+0000","last_unstale":"2024-02-19T14:53:02.858463+0000","last_undegraded":"2024-02-19T14:53:02.858463+0000","last_fullsized":"2024-02-19T14:53:02.858463+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:39.871295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857722+0000","last_change":"2024-02-19T14:53:00.851104+0000","last_active":"2024-02-19T14:53:02.857722+0000","last_peered":"2024-02-19T14:53:02.857722+0000","last_clean":"2024-02-19T14:53:02.857722+0000","last_became_active":"2024-02-19T14:53:00.850848+0000","last_became_peered":"2024-02-19T14:53:00.850848+0000","last_unstale":"2024-02-19T14:53:02.857722+0000","last_undegraded":"2024-02-19T14:53:02.857722+0000","last_fullsized":"2024-02-19T14:53:02.857722+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:26.461252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857899+0000","last_change":"2024-02-19T14:53:01.856871+0000","last_active":"2024-02-19T14:53:02.857899+0000","last_peered":"2024-02-19T14:53:02.857899+0000","last_clean":"2024-02-19T14:53:02.857899+0000","last_became_active":"2024-02-19T14:53:01.856504+0000","last_became_peered":"2024-02-19T14:53:01.856504+0000","last_unstale":"2024-02-19T14:53:02.857899+0000","last_undegraded":"2024-02-19T14:53:02.857899+0000","last_fullsized":"2024-02-19T14:53:02.857899+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:57.445773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940303+0000","last_change":"2024-02-19T14:53:00.853953+0000","last_active":"2024-02-19T14:53:02.940303+0000","last_peered":"2024-02-19T14:53:02.940303+0000","last_clean":"2024-02-19T14:53:02.940303+0000","last_became_active":"2024-02-19T14:53:00.853684+0000","last_became_peered":"2024-02-19T14:53:00.853684+0000","last_unstale":"2024-02-19T14:53:02.940303+0000","last_undegraded":"2024-02-19T14:53:02.940303+0000","last_fullsized":"2024-02-19T14:53:02.940303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:40:51.949331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940157+0000","last_change":"2024-02-19T14:53:01.865788+0000","last_active":"2024-02-19T14:53:02.940157+0000","last_peered":"2024-02-19T14:53:02.940157+0000","last_clean":"2024-02-19T14:53:02.940157+0000","last_became_active":"2024-02-19T14:53:01.865436+0000","last_became_peered":"2024-02-19T14:53:01.865436+0000","last_unstale":"2024-02-19T14:53:02.940157+0000","last_undegraded":"2024-02-19T14:53:02.940157+0000","last_fullsized":"2024-02-19T14:53:02.940157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:54:59.322422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857687+0000","last_change":"2024-02-19T14:53:00.856704+0000","last_active":"2024-02-19T14:53:02.857687+0000","last_peered":"2024-02-19T14:53:02.857687+0000","last_clean":"2024-02-19T14:53:02.857687+0000","last_became_active":"2024-02-19T14:53:00.856455+0000","last_became_peered":"2024-02-19T14:53:00.856455+0000","last_unstale":"2024-02-19T14:53:02.857687+0000","last_undegraded":"2024-02-19T14:53:02.857687+0000","last_fullsized":"2024-02-19T14:53:02.857687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:03:13.889687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940857+0000","last_change":"2024-02-19T14:53:01.862116+0000","last_active":"2024-02-19T14:53:02.940857+0000","last_peered":"2024-02-19T14:53:02.940857+0000","last_clean":"2024-02-19T14:53:02.940857+0000","last_became_active":"2024-02-19T14:53:01.861849+0000","last_became_peered":"2024-02-19T14:53:01.861849+0000","last_unstale":"2024-02-19T14:53:02.940857+0000","last_undegraded":"2024-02-19T14:53:02.940857+0000","last_fullsized":"2024-02-19T14:53:02.940857+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:54.754384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857818+0000","last_change":"2024-02-19T14:53:00.851285+0000","last_active":"2024-02-19T14:53:02.857818+0000","last_peered":"2024-02-19T14:53:02.857818+0000","last_clean":"2024-02-19T14:53:02.857818+0000","last_became_active":"2024-02-19T14:53:00.851043+0000","last_became_peered":"2024-02-19T14:53:00.851043+0000","last_unstale":"2024-02-19T14:53:02.857818+0000","last_undegraded":"2024-02-19T14:53:02.857818+0000","last_fullsized":"2024-02-19T14:53:02.857818+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:33.946706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940615+0000","last_change":"2024-02-19T14:53:01.849826+0000","last_active":"2024-02-19T14:53:02.940615+0000","last_peered":"2024-02-19T14:53:02.940615+0000","last_clean":"2024-02-19T14:53:02.940615+0000","last_became_active":"2024-02-19T14:53:01.849521+0000","last_became_peered":"2024-02-19T14:53:01.849521+0000","last_unstale":"2024-02-19T14:53:02.940615+0000","last_undegraded":"2024-02-19T14:53:02.940615+0000","last_fullsized":"2024-02-19T14:53:02.940615+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941008+0000","last_change":"2024-02-19T14:53:00.854167+0000","last_active":"2024-02-19T14:53:02.941008+0000","last_peered":"2024-02-19T14:53:02.941008+0000","last_clean":"2024-02-19T14:53:02.941008+0000","last_became_active":"2024-02-19T14:53:00.853855+0000","last_became_peered":"2024-02-19T14:53:00.853855+0000","last_unstale":"2024-02-19T14:53:02.941008+0000","last_undegraded":"2024-02-19T14:53:02.941008+0000","last_fullsized":"2024-02-19T14:53:02.941008+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:04.636408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940850+0000","last_change":"2024-02-19T14:53:01.859731+0000","last_active":"2024-02-19T14:53:02.940850+0000","last_peered":"2024-02-19T14:53:02.940850+0000","last_clean":"2024-02-19T14:53:02.940850+0000","last_became_active":"2024-02-19T14:53:01.859446+0000","last_became_peered":"2024-02-19T14:53:01.859446+0000","last_unstale":"2024-02-19T14:53:02.940850+0000","last_undegraded":"2024-02-19T14:53:02.940850+0000","last_fullsized":"2024-02-19T14:53:02.940850+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:49.806973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857521+0000","last_change":"2024-02-19T14:53:00.853229+0000","last_active":"2024-02-19T14:53:02.857521+0000","last_peered":"2024-02-19T14:53:02.857521+0000","last_clean":"2024-02-19T14:53:02.857521+0000","last_became_active":"2024-02-19T14:53:00.852967+0000","last_became_peered":"2024-02-19T14:53:00.852967+0000","last_unstale":"2024-02-19T14:53:02.857521+0000","last_undegraded":"2024-02-19T14:53:02.857521+0000","last_fullsized":"2024-02-19T14:53:02.857521+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:29.129278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940218+0000","last_change":"2024-02-19T14:53:01.863580+0000","last_active":"2024-02-19T14:53:02.940218+0000","last_peered":"2024-02-19T14:53:02.940218+0000","last_clean":"2024-02-19T14:53:02.940218+0000","last_became_active":"2024-02-19T14:53:01.863341+0000","last_became_peered":"2024-02-19T14:53:01.863341+0000","last_unstale":"2024-02-19T14:53:02.940218+0000","last_undegraded":"2024-02-19T14:53:02.940218+0000","last_fullsized":"2024-02-19T14:53:02.940218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:54:59.322422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941072+0000","last_change":"2024-02-19T14:53:00.860030+0000","last_active":"2024-02-19T14:53:02.941072+0000","last_peered":"2024-02-19T14:53:02.941072+0000","last_clean":"2024-02-19T14:53:02.941072+0000","last_became_active":"2024-02-19T14:53:00.859735+0000","last_became_peered":"2024-02-19T14:53:00.859735+0000","last_unstale":"2024-02-19T14:53:02.941072+0000","last_undegraded":"2024-02-19T14:53:02.941072+0000","last_fullsized":"2024-02-19T14:53:02.941072+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.267643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940522+0000","last_change":"2024-02-19T14:53:01.871395+0000","last_active":"2024-02-19T14:53:02.940522+0000","last_peered":"2024-02-19T14:53:02.940522+0000","last_clean":"2024-02-19T14:53:02.940522+0000","last_became_active":"2024-02-19T14:53:01.870876+0000","last_became_peered":"2024-02-19T14:53:01.870876+0000","last_unstale":"2024-02-19T14:53:02.940522+0000","last_undegraded":"2024-02-19T14:53:02.940522+0000","last_fullsized":"2024-02-19T14:53:02.940522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:38:54.455007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857760+0000","last_change":"2024-02-19T14:53:00.854721+0000","last_active":"2024-02-19T14:53:02.857760+0000","last_peered":"2024-02-19T14:53:02.857760+0000","last_clean":"2024-02-19T14:53:02.857760+0000","last_became_active":"2024-02-19T14:53:00.854456+0000","last_became_peered":"2024-02-19T14:53:00.854456+0000","last_unstale":"2024-02-19T14:53:02.857760+0000","last_undegraded":"2024-02-19T14:53:02.857760+0000","last_fullsized":"2024-02-19T14:53:02.857760+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:04.170365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939995+0000","last_change":"2024-02-19T14:53:01.857207+0000","last_active":"2024-02-19T14:53:02.939995+0000","last_peered":"2024-02-19T14:53:02.939995+0000","last_clean":"2024-02-19T14:53:02.939995+0000","last_became_active":"2024-02-19T14:53:01.856941+0000","last_became_peered":"2024-02-19T14:53:01.856941+0000","last_unstale":"2024-02-19T14:53:02.939995+0000","last_undegraded":"2024-02-19T14:53:02.939995+0000","last_fullsized":"2024-02-19T14:53:02.939995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:06.083865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941100+0000","last_change":"2024-02-19T14:53:00.853340+0000","last_active":"2024-02-19T14:53:02.941100+0000","last_peered":"2024-02-19T14:53:02.941100+0000","last_clean":"2024-02-19T14:53:02.941100+0000","last_became_active":"2024-02-19T14:53:00.853074+0000","last_became_peered":"2024-02-19T14:53:00.853074+0000","last_unstale":"2024-02-19T14:53:02.941100+0000","last_undegraded":"2024-02-19T14:53:02.941100+0000","last_fullsized":"2024-02-19T14:53:02.941100+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:44.809850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939873+0000","last_change":"2024-02-19T14:53:01.867117+0000","last_active":"2024-02-19T14:53:02.939873+0000","last_peered":"2024-02-19T14:53:02.939873+0000","last_clean":"2024-02-19T14:53:02.939873+0000","last_became_active":"2024-02-19T14:53:01.866851+0000","last_became_peered":"2024-02-19T14:53:01.866851+0000","last_unstale":"2024-02-19T14:53:02.939873+0000","last_undegraded":"2024-02-19T14:53:02.939873+0000","last_fullsized":"2024-02-19T14:53:02.939873+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:34.269975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941068+0000","last_change":"2024-02-19T14:53:00.855423+0000","last_active":"2024-02-19T14:53:02.941068+0000","last_peered":"2024-02-19T14:53:02.941068+0000","last_clean":"2024-02-19T14:53:02.941068+0000","last_became_active":"2024-02-19T14:53:00.854829+0000","last_became_peered":"2024-02-19T14:53:00.854829+0000","last_unstale":"2024-02-19T14:53:02.941068+0000","last_undegraded":"2024-02-19T14:53:02.941068+0000","last_fullsized":"2024-02-19T14:53:02.941068+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:08.703678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940582+0000","last_change":"2024-02-19T14:53:01.863269+0000","last_active":"2024-02-19T14:53:02.940582+0000","last_peered":"2024-02-19T14:53:02.940582+0000","last_clean":"2024-02-19T14:53:02.940582+0000","last_became_active":"2024-02-19T14:53:01.862991+0000","last_became_peered":"2024-02-19T14:53:01.862991+0000","last_unstale":"2024-02-19T14:53:02.940582+0000","last_undegraded":"2024-02-19T14:53:02.940582+0000","last_fullsized":"2024-02-19T14:53:02.940582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:38:54.455007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858716+0000","last_change":"2024-02-19T14:53:00.853321+0000","last_active":"2024-02-19T14:53:02.858716+0000","last_peered":"2024-02-19T14:53:02.858716+0000","last_clean":"2024-02-19T14:53:02.858716+0000","last_became_active":"2024-02-19T14:53:00.853035+0000","last_became_peered":"2024-02-19T14:53:00.853035+0000","last_unstale":"2024-02-19T14:53:02.858716+0000","last_undegraded":"2024-02-19T14:53:02.858716+0000","last_fullsized":"2024-02-19T14:53:02.858716+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:20.391008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858628+0000","last_change":"2024-02-19T14:53:01.853798+0000","last_active":"2024-02-19T14:53:02.858628+0000","last_peered":"2024-02-19T14:53:02.858628+0000","last_clean":"2024-02-19T14:53:02.858628+0000","last_became_active":"2024-02-19T14:53:01.853533+0000","last_became_peered":"2024-02-19T14:53:01.853533+0000","last_unstale":"2024-02-19T14:53:02.858628+0000","last_undegraded":"2024-02-19T14:53:02.858628+0000","last_fullsized":"2024-02-19T14:53:02.858628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.086237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859261+0000","last_change":"2024-02-19T14:53:00.851456+0000","last_active":"2024-02-19T14:53:02.859261+0000","last_peered":"2024-02-19T14:53:02.859261+0000","last_clean":"2024-02-19T14:53:02.859261+0000","last_became_active":"2024-02-19T14:53:00.851165+0000","last_became_peered":"2024-02-19T14:53:00.851165+0000","last_unstale":"2024-02-19T14:53:02.859261+0000","last_undegraded":"2024-02-19T14:53:02.859261+0000","last_fullsized":"2024-02-19T14:53:02.859261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:20.391008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940568+0000","last_change":"2024-02-19T14:53:01.871288+0000","last_active":"2024-02-19T14:53:02.940568+0000","last_peered":"2024-02-19T14:53:02.940568+0000","last_clean":"2024-02-19T14:53:02.940568+0000","last_became_active":"2024-02-19T14:53:01.870958+0000","last_became_peered":"2024-02-19T14:53:01.870958+0000","last_unstale":"2024-02-19T14:53:02.940568+0000","last_undegraded":"2024-02-19T14:53:02.940568+0000","last_fullsized":"2024-02-19T14:53:02.940568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:28.622298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939857+0000","last_change":"2024-02-19T14:53:00.851285+0000","last_active":"2024-02-19T14:53:02.939857+0000","last_peered":"2024-02-19T14:53:02.939857+0000","last_clean":"2024-02-19T14:53:02.939857+0000","last_became_active":"2024-02-19T14:53:00.851031+0000","last_became_peered":"2024-02-19T14:53:00.851031+0000","last_unstale":"2024-02-19T14:53:02.939857+0000","last_undegraded":"2024-02-19T14:53:02.939857+0000","last_fullsized":"2024-02-19T14:53:02.939857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:19.424285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940985+0000","last_change":"2024-02-19T14:53:01.858624+0000","last_active":"2024-02-19T14:53:02.940985+0000","last_peered":"2024-02-19T14:53:02.940985+0000","last_clean":"2024-02-19T14:53:02.940985+0000","last_became_active":"2024-02-19T14:53:01.858279+0000","last_became_peered":"2024-02-19T14:53:01.858279+0000","last_unstale":"2024-02-19T14:53:02.940985+0000","last_undegraded":"2024-02-19T14:53:02.940985+0000","last_fullsized":"2024-02-19T14:53:02.940985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:08.239916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941136+0000","last_change":"2024-02-19T14:53:00.862101+0000","last_active":"2024-02-19T14:53:02.941136+0000","last_peered":"2024-02-19T14:53:02.941136+0000","last_clean":"2024-02-19T14:53:02.941136+0000","last_became_active":"2024-02-19T14:53:00.861896+0000","last_became_peered":"2024-02-19T14:53:00.861896+0000","last_unstale":"2024-02-19T14:53:02.941136+0000","last_undegraded":"2024-02-19T14:53:02.941136+0000","last_fullsized":"2024-02-19T14:53:02.941136+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:56.443441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940190+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:02.940190+0000","last_peered":"2024-02-19T14:53:02.940190+0000","last_clean":"2024-02-19T14:53:02.940190+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:02.940190+0000","last_undegraded":"2024-02-19T14:53:02.940190+0000","last_fullsized":"2024-02-19T14:53:02.940190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.502}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.624}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":184499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.497},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.624}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.66300000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":184029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.503}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.73799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":184521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.504}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.44},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.31}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.377}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":185563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.434}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":183987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82199999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.76900000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.501}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":184007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.224},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.496},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42399999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.44400000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.56699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1024000,"data_stored":774319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.255},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.27400000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.433},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.54700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:04.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:05.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.036+0000 7f9aaac5c640 1 Processor -- start 2024-02-19T14:53:05.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.036+0000 7f9aaac5c640 1 -- start start 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aaac5c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9aa4072180 0x7f9aa4155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aaac5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9aa406f270 0x7f9aa41555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aaac5c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa4155af0 con 0x7f9aa41528d0 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aaac5c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa40734c0 con 0x7f9aa406f270 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aaac5c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa4072e80 con 0x7f9aa4072180 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aa91f9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9aa406f270 0x7f9aa41555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aa91f9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9aa406f270 0x7f9aa41555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47138/0 (socket says 172.21.15.78:47138) 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.037+0000 7f9aa91f9640 1 -- 172.21.15.78:0/1835456821 learned_addr learned my addr 172.21.15.78:0/1835456821 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa3fff640 1 --2- 172.21.15.78:0/1835456821 >> v2:172.21.15.150:3300/0 conn(0x7f9aa4072180 0x7f9aa4155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa91f9640 1 -- 172.21.15.78:0/1835456821 >> v2:172.21.15.150:3300/0 conn(0x7f9aa4072180 msgr2=0x7f9aa4155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa91f9640 1 --2- 172.21.15.78:0/1835456821 >> v2:172.21.15.150:3300/0 conn(0x7f9aa4072180 0x7f9aa4155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa91f9640 1 -- 172.21.15.78:0/1835456821 >> v1:172.21.15.150:6789/0 conn(0x7f9aa41528d0 legacy=0x7f9aa40c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa91f9640 1 -- 172.21.15.78:0/1835456821 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa406e990 con 0x7f9aa406f270 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa3fff640 1 --2- 172.21.15.78:0/1835456821 >> v2:172.21.15.150:3300/0 conn(0x7f9aa4072180 0x7f9aa4155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.038+0000 7f9aa91f9640 1 --2- 172.21.15.78:0/1835456821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41555b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4070310 tx=0x7f9a8c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e2dbc5b40b38f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aa37fe640 1 -- 172.21.15.78:0/1835456821 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0317e0 con 0x7f9aa406f270 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aa37fe640 1 -- 172.21.15.78:0/1835456821 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c031940 con 0x7f9aa406f270 2024-02-19T14:53:05.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aa37fe640 1 -- 172.21.15.78:0/1835456821 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c031cd0 con 0x7f9aa406f270 2024-02-19T14:53:05.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 -- 172.21.15.78:0/1835456821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 msgr2=0x7f9aa41555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/1835456821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41555b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4070310 tx=0x7f9a8c02b880 comp rx=0 tx=0).stop 2024-02-19T14:53:05.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 -- 172.21.15.78:0/1835456821 shutdown_connections 2024-02-19T14:53:05.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/1835456821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41555b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/1835456821 >> v2:172.21.15.150:3300/0 conn(0x7f9aa4072180 0x7f9aa4155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 -- 172.21.15.78:0/1835456821 >> 172.21.15.78:0/1835456821 conn(0x7f9aa40c7590 msgr2=0x7f9aa40c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:05.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 -- 172.21.15.78:0/1835456821 shutdown_connections 2024-02-19T14:53:05.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.039+0000 7f9aaac5c640 1 -- 172.21.15.78:0/1835456821 wait complete. 2024-02-19T14:53:05.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.040+0000 7f9aaac5c640 1 Processor -- start 2024-02-19T14:53:05.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 -- start start 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9aa4072180 0x7f9aa41ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41528d0 0x7f9aa41ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa40a9340 con 0x7f9aa406f270 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa41eab10 con 0x7f9aa41528d0 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aaac5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa40a7a40 con 0x7f9aa4072180 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.041+0000 7f9aa89f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47140/0 (socket says 172.21.15.78:47140) 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 -- 172.21.15.78:0/2208261557 learned_addr learned my addr 172.21.15.78:0/2208261557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa3fff640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9aa4072180 0x7f9aa41ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa91f9640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41528d0 0x7f9aa41ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 -- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9aa4072180 msgr2=0x7f9aa41ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9aa4072180 0x7f9aa41ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 -- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41528d0 msgr2=0x7f9aa41ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41528d0 0x7f9aa41ee8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 -- 172.21.15.78:0/2208261557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c0094d0 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa89f8640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41541a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c7c0 tx=0x7f9a9c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c00aa50 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c00abb0 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa40a7760 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c012420 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.042+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa40a8070 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.043+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa40a7bf0 con 0x7f9aa406f270 2024-02-19T14:53:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.044+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9a9c012580 con 0x7f9aa406f270 2024-02-19T14:53:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.044+0000 7f9aa17fa640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9a8406b810 0x7f9a8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.044+0000 7f9aa3fff640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9a8406b810 0x7f9a8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.044+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9a9c08a580 con 0x7f9aa406f270 2024-02-19T14:53:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.045+0000 7f9aa3fff640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9a8406b810 0x7f9a8406dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4072810 tx=0x7f9a94008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.049+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9aa40a7bf0 con 0x7f9aa406f270 2024-02-19T14:53:05.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.222+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9aa4159400 con 0x7f9a8406b810 2024-02-19T14:53:05.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.229+0000 7f9aa17fa640 1 -- 172.21.15.78:0/2208261557 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289543 (secure 0 0 0) 0x7f9a9400c040 con 0x7f9a8406b810 2024-02-19T14:53:05.235 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:05.236 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9a8406b810 msgr2=0x7f9a8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9a8406b810 0x7f9a8406dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4072810 tx=0x7f9a94008040 comp rx=0 tx=0).stop 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 msgr2=0x7f9aa41541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41541a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c7c0 tx=0x7f9a9c00cc80 comp rx=0 tx=0).stop 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 shutdown_connections 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9a8406b810 0x7f9a8406dcd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41528d0 0x7f9aa41ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9aa4072180 0x7f9aa41ee360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 --2- 172.21.15.78:0/2208261557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9aa406f270 0x7f9aa41541a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 >> 172.21.15.78:0/2208261557 conn(0x7f9aa40c7590 msgr2=0x7f9aa40c1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 shutdown_connections 2024-02-19T14:53:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.233+0000 7f9aaac5c640 1 -- 172.21.15.78:0/2208261557 wait complete. 2024-02-19T14:53:05.246 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-02-19T14:53:04.073138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220764,"kb_used_data":4564,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753956,"statfs":{"total":773094113280,"available":772868050944,"internally_reserved":0,"allocated":4673536,"data_stored":2663584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.240714"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940411+0000","last_change":"2024-02-19T14:53:01.860296+0000","last_active":"2024-02-19T14:53:02.940411+0000","last_peered":"2024-02-19T14:53:02.940411+0000","last_clean":"2024-02-19T14:53:02.940411+0000","last_became_active":"2024-02-19T14:53:01.860040+0000","last_became_peered":"2024-02-19T14:53:01.860040+0000","last_unstale":"2024-02-19T14:53:02.940411+0000","last_undegraded":"2024-02-19T14:53:02.940411+0000","last_fullsized":"2024-02-19T14:53:02.940411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:26.645451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939864+0000","last_change":"2024-02-19T14:53:01.858367+0000","last_active":"2024-02-19T14:53:02.939864+0000","last_peered":"2024-02-19T14:53:02.939864+0000","last_clean":"2024-02-19T14:53:02.939864+0000","last_became_active":"2024-02-19T14:53:01.858097+0000","last_became_peered":"2024-02-19T14:53:01.858097+0000","last_unstale":"2024-02-19T14:53:02.939864+0000","last_undegraded":"2024-02-19T14:53:02.939864+0000","last_fullsized":"2024-02-19T14:53:02.939864+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:43.102502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941123+0000","last_change":"2024-02-19T14:53:01.868554+0000","last_active":"2024-02-19T14:53:02.941123+0000","last_peered":"2024-02-19T14:53:02.941123+0000","last_clean":"2024-02-19T14:53:02.941123+0000","last_became_active":"2024-02-19T14:53:01.868287+0000","last_became_peered":"2024-02-19T14:53:01.868287+0000","last_unstale":"2024-02-19T14:53:02.941123+0000","last_undegraded":"2024-02-19T14:53:02.941123+0000","last_fullsized":"2024-02-19T14:53:02.941123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:12:50.114024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857887+0000","last_change":"2024-02-19T14:53:01.856179+0000","last_active":"2024-02-19T14:53:02.857887+0000","last_peered":"2024-02-19T14:53:02.857887+0000","last_clean":"2024-02-19T14:53:02.857887+0000","last_became_active":"2024-02-19T14:53:01.855913+0000","last_became_peered":"2024-02-19T14:53:01.855913+0000","last_unstale":"2024-02-19T14:53:02.857887+0000","last_undegraded":"2024-02-19T14:53:02.857887+0000","last_fullsized":"2024-02-19T14:53:02.857887+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:05.638740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857848+0000","last_change":"2024-02-19T14:53:01.855429+0000","last_active":"2024-02-19T14:53:02.857848+0000","last_peered":"2024-02-19T14:53:02.857848+0000","last_clean":"2024-02-19T14:53:02.857848+0000","last_became_active":"2024-02-19T14:53:01.855159+0000","last_became_peered":"2024-02-19T14:53:01.855159+0000","last_unstale":"2024-02-19T14:53:02.857848+0000","last_undegraded":"2024-02-19T14:53:02.857848+0000","last_fullsized":"2024-02-19T14:53:02.857848+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:06.532400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940038+0000","last_change":"2024-02-19T14:53:01.854848+0000","last_active":"2024-02-19T14:53:02.940038+0000","last_peered":"2024-02-19T14:53:02.940038+0000","last_clean":"2024-02-19T14:53:02.940038+0000","last_became_active":"2024-02-19T14:53:01.854597+0000","last_became_peered":"2024-02-19T14:53:01.854597+0000","last_unstale":"2024-02-19T14:53:02.940038+0000","last_undegraded":"2024-02-19T14:53:02.940038+0000","last_fullsized":"2024-02-19T14:53:02.940038+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:54.754384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940126+0000","last_change":"2024-02-19T14:53:01.858729+0000","last_active":"2024-02-19T14:53:02.940126+0000","last_peered":"2024-02-19T14:53:02.940126+0000","last_clean":"2024-02-19T14:53:02.940126+0000","last_became_active":"2024-02-19T14:53:01.858460+0000","last_became_peered":"2024-02-19T14:53:01.858460+0000","last_unstale":"2024-02-19T14:53:02.940126+0000","last_undegraded":"2024-02-19T14:53:02.940126+0000","last_fullsized":"2024-02-19T14:53:02.940126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:43.102502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939767+0000","last_change":"2024-02-19T14:53:01.862373+0000","last_active":"2024-02-19T14:53:02.939767+0000","last_peered":"2024-02-19T14:53:02.939767+0000","last_clean":"2024-02-19T14:53:02.939767+0000","last_became_active":"2024-02-19T14:53:01.862114+0000","last_became_peered":"2024-02-19T14:53:01.862114+0000","last_unstale":"2024-02-19T14:53:02.939767+0000","last_undegraded":"2024-02-19T14:53:02.939767+0000","last_fullsized":"2024-02-19T14:53:02.939767+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:01.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941348+0000","last_change":"2024-02-19T14:53:01.867857+0000","last_active":"2024-02-19T14:53:02.941348+0000","last_peered":"2024-02-19T14:53:02.941348+0000","last_clean":"2024-02-19T14:53:02.941348+0000","last_became_active":"2024-02-19T14:53:01.867609+0000","last_became_peered":"2024-02-19T14:53:01.867609+0000","last_unstale":"2024-02-19T14:53:02.941348+0000","last_undegraded":"2024-02-19T14:53:02.941348+0000","last_fullsized":"2024-02-19T14:53:02.941348+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:02.908488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940771+0000","last_change":"2024-02-19T14:53:01.860504+0000","last_active":"2024-02-19T14:53:02.940771+0000","last_peered":"2024-02-19T14:53:02.940771+0000","last_clean":"2024-02-19T14:53:02.940771+0000","last_became_active":"2024-02-19T14:53:01.860167+0000","last_became_peered":"2024-02-19T14:53:01.860167+0000","last_unstale":"2024-02-19T14:53:02.940771+0000","last_undegraded":"2024-02-19T14:53:02.940771+0000","last_fullsized":"2024-02-19T14:53:02.940771+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:11.625898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940763+0000","last_change":"2024-02-19T14:53:01.853369+0000","last_active":"2024-02-19T14:53:02.940763+0000","last_peered":"2024-02-19T14:53:02.940763+0000","last_clean":"2024-02-19T14:53:02.940763+0000","last_became_active":"2024-02-19T14:53:01.853020+0000","last_became_peered":"2024-02-19T14:53:01.853020+0000","last_unstale":"2024-02-19T14:53:02.940763+0000","last_undegraded":"2024-02-19T14:53:02.940763+0000","last_fullsized":"2024-02-19T14:53:02.940763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:37.403255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940977+0000","last_change":"2024-02-19T14:53:01.859058+0000","last_active":"2024-02-19T14:53:02.940977+0000","last_peered":"2024-02-19T14:53:02.940977+0000","last_clean":"2024-02-19T14:53:02.940977+0000","last_became_active":"2024-02-19T14:53:01.858805+0000","last_became_peered":"2024-02-19T14:53:01.858805+0000","last_unstale":"2024-02-19T14:53:02.940977+0000","last_undegraded":"2024-02-19T14:53:02.940977+0000","last_fullsized":"2024-02-19T14:53:02.940977+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:58:08.710699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940267+0000","last_change":"2024-02-19T14:53:01.861276+0000","last_active":"2024-02-19T14:53:02.940267+0000","last_peered":"2024-02-19T14:53:02.940267+0000","last_clean":"2024-02-19T14:53:02.940267+0000","last_became_active":"2024-02-19T14:53:01.861013+0000","last_became_peered":"2024-02-19T14:53:01.861013+0000","last_unstale":"2024-02-19T14:53:02.940267+0000","last_undegraded":"2024-02-19T14:53:02.940267+0000","last_fullsized":"2024-02-19T14:53:02.940267+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:06.083865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858464+0000","last_change":"2024-02-19T14:53:01.851730+0000","last_active":"2024-02-19T14:53:02.858464+0000","last_peered":"2024-02-19T14:53:02.858464+0000","last_clean":"2024-02-19T14:53:02.858464+0000","last_became_active":"2024-02-19T14:53:01.851476+0000","last_became_peered":"2024-02-19T14:53:01.851476+0000","last_unstale":"2024-02-19T14:53:02.858464+0000","last_undegraded":"2024-02-19T14:53:02.858464+0000","last_fullsized":"2024-02-19T14:53:02.858464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:20.426618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940645+0000","last_change":"2024-02-19T14:53:01.858068+0000","last_active":"2024-02-19T14:53:02.940645+0000","last_peered":"2024-02-19T14:53:02.940645+0000","last_clean":"2024-02-19T14:53:02.940645+0000","last_became_active":"2024-02-19T14:53:01.857745+0000","last_became_peered":"2024-02-19T14:53:01.857745+0000","last_unstale":"2024-02-19T14:53:02.940645+0000","last_undegraded":"2024-02-19T14:53:02.940645+0000","last_fullsized":"2024-02-19T14:53:02.940645+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:26.645451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939992+0000","last_change":"2024-02-19T14:53:01.863622+0000","last_active":"2024-02-19T14:53:02.939992+0000","last_peered":"2024-02-19T14:53:02.939992+0000","last_clean":"2024-02-19T14:53:02.939992+0000","last_became_active":"2024-02-19T14:53:01.863054+0000","last_became_peered":"2024-02-19T14:53:01.863054+0000","last_unstale":"2024-02-19T14:53:02.939992+0000","last_undegraded":"2024-02-19T14:53:02.939992+0000","last_fullsized":"2024-02-19T14:53:02.939992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:12:50.114024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940679+0000","last_change":"2024-02-19T14:53:01.867250+0000","last_active":"2024-02-19T14:53:02.940679+0000","last_peered":"2024-02-19T14:53:02.940679+0000","last_clean":"2024-02-19T14:53:02.940679+0000","last_became_active":"2024-02-19T14:53:01.866986+0000","last_became_peered":"2024-02-19T14:53:01.866986+0000","last_unstale":"2024-02-19T14:53:02.940679+0000","last_undegraded":"2024-02-19T14:53:02.940679+0000","last_fullsized":"2024-02-19T14:53:02.940679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:32:37.025792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859030+0000","last_change":"2024-02-19T14:53:01.861574+0000","last_active":"2024-02-19T14:53:02.859030+0000","last_peered":"2024-02-19T14:53:02.859030+0000","last_clean":"2024-02-19T14:53:02.859030+0000","last_became_active":"2024-02-19T14:53:01.861245+0000","last_became_peered":"2024-02-19T14:53:01.861245+0000","last_unstale":"2024-02-19T14:53:02.859030+0000","last_undegraded":"2024-02-19T14:53:02.859030+0000","last_fullsized":"2024-02-19T14:53:02.859030+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:34.269975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857785+0000","last_change":"2024-02-19T14:53:01.857279+0000","last_active":"2024-02-19T14:53:02.857785+0000","last_peered":"2024-02-19T14:53:02.857785+0000","last_clean":"2024-02-19T14:53:02.857785+0000","last_became_active":"2024-02-19T14:53:01.856985+0000","last_became_peered":"2024-02-19T14:53:01.856985+0000","last_unstale":"2024-02-19T14:53:02.857785+0000","last_undegraded":"2024-02-19T14:53:02.857785+0000","last_fullsized":"2024-02-19T14:53:02.857785+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:09.706011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858886+0000","last_change":"2024-02-19T14:53:01.862398+0000","last_active":"2024-02-19T14:53:02.858886+0000","last_peered":"2024-02-19T14:53:02.858886+0000","last_clean":"2024-02-19T14:53:02.858886+0000","last_became_active":"2024-02-19T14:53:01.862089+0000","last_became_peered":"2024-02-19T14:53:01.862089+0000","last_unstale":"2024-02-19T14:53:02.858886+0000","last_undegraded":"2024-02-19T14:53:02.858886+0000","last_fullsized":"2024-02-19T14:53:02.858886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:37.556036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940335+0000","last_change":"2024-02-19T14:53:01.859863+0000","last_active":"2024-02-19T14:53:02.940335+0000","last_peered":"2024-02-19T14:53:02.940335+0000","last_clean":"2024-02-19T14:53:02.940335+0000","last_became_active":"2024-02-19T14:53:01.859615+0000","last_became_peered":"2024-02-19T14:53:01.859615+0000","last_unstale":"2024-02-19T14:53:02.940335+0000","last_undegraded":"2024-02-19T14:53:02.940335+0000","last_fullsized":"2024-02-19T14:53:02.940335+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:08.912678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858659+0000","last_change":"2024-02-19T14:53:01.860920+0000","last_active":"2024-02-19T14:53:02.858659+0000","last_peered":"2024-02-19T14:53:02.858659+0000","last_clean":"2024-02-19T14:53:02.858659+0000","last_became_active":"2024-02-19T14:53:01.860651+0000","last_became_peered":"2024-02-19T14:53:01.860651+0000","last_unstale":"2024-02-19T14:53:02.858659+0000","last_undegraded":"2024-02-19T14:53:02.858659+0000","last_fullsized":"2024-02-19T14:53:02.858659+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:34.269975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941082+0000","last_change":"2024-02-19T14:53:01.867092+0000","last_active":"2024-02-19T14:53:02.941082+0000","last_peered":"2024-02-19T14:53:02.941082+0000","last_clean":"2024-02-19T14:53:02.941082+0000","last_became_active":"2024-02-19T14:53:01.866414+0000","last_became_peered":"2024-02-19T14:53:01.866414+0000","last_unstale":"2024-02-19T14:53:02.941082+0000","last_undegraded":"2024-02-19T14:53:02.941082+0000","last_fullsized":"2024-02-19T14:53:02.941082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:01.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857566+0000","last_change":"2024-02-19T14:53:01.858380+0000","last_active":"2024-02-19T14:53:02.857566+0000","last_peered":"2024-02-19T14:53:02.857566+0000","last_clean":"2024-02-19T14:53:02.857566+0000","last_became_active":"2024-02-19T14:53:01.858106+0000","last_became_peered":"2024-02-19T14:53:01.858106+0000","last_unstale":"2024-02-19T14:53:02.857566+0000","last_undegraded":"2024-02-19T14:53:02.857566+0000","last_fullsized":"2024-02-19T14:53:02.857566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:36.025364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858973+0000","last_change":"2024-02-19T14:53:01.858430+0000","last_active":"2024-02-19T14:53:02.858973+0000","last_peered":"2024-02-19T14:53:02.858973+0000","last_clean":"2024-02-19T14:53:02.858973+0000","last_became_active":"2024-02-19T14:53:01.858157+0000","last_became_peered":"2024-02-19T14:53:01.858157+0000","last_unstale":"2024-02-19T14:53:02.858973+0000","last_undegraded":"2024-02-19T14:53:02.858973+0000","last_fullsized":"2024-02-19T14:53:02.858973+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.086237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940362+0000","last_change":"2024-02-19T14:53:01.863007+0000","last_active":"2024-02-19T14:53:02.940362+0000","last_peered":"2024-02-19T14:53:02.940362+0000","last_clean":"2024-02-19T14:53:02.940362+0000","last_became_active":"2024-02-19T14:53:01.862744+0000","last_became_peered":"2024-02-19T14:53:01.862744+0000","last_unstale":"2024-02-19T14:53:02.940362+0000","last_undegraded":"2024-02-19T14:53:02.940362+0000","last_fullsized":"2024-02-19T14:53:02.940362+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:02.908488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858371+0000","last_change":"2024-02-19T14:53:01.856728+0000","last_active":"2024-02-19T14:53:02.858371+0000","last_peered":"2024-02-19T14:53:02.858371+0000","last_clean":"2024-02-19T14:53:02.858371+0000","last_became_active":"2024-02-19T14:53:01.856443+0000","last_became_peered":"2024-02-19T14:53:01.856443+0000","last_unstale":"2024-02-19T14:53:02.858371+0000","last_undegraded":"2024-02-19T14:53:02.858371+0000","last_fullsized":"2024-02-19T14:53:02.858371+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:39.871295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940987+0000","last_change":"2024-02-19T14:53:01.852318+0000","last_active":"2024-02-19T14:53:02.940987+0000","last_peered":"2024-02-19T14:53:02.940987+0000","last_clean":"2024-02-19T14:53:02.940987+0000","last_became_active":"2024-02-19T14:53:01.851153+0000","last_became_peered":"2024-02-19T14:53:01.851153+0000","last_unstale":"2024-02-19T14:53:02.940987+0000","last_undegraded":"2024-02-19T14:53:02.940987+0000","last_fullsized":"2024-02-19T14:53:02.940987+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:46:42.210280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858302+0000","last_change":"2024-02-19T14:53:01.860434+0000","last_active":"2024-02-19T14:53:02.858302+0000","last_peered":"2024-02-19T14:53:02.858302+0000","last_clean":"2024-02-19T14:53:02.858302+0000","last_became_active":"2024-02-19T14:53:01.859566+0000","last_became_peered":"2024-02-19T14:53:01.859566+0000","last_unstale":"2024-02-19T14:53:02.858302+0000","last_undegraded":"2024-02-19T14:53:02.858302+0000","last_fullsized":"2024-02-19T14:53:02.858302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:02.469083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858593+0000","last_change":"2024-02-19T14:53:01.855743+0000","last_active":"2024-02-19T14:53:02.858593+0000","last_peered":"2024-02-19T14:53:02.858593+0000","last_clean":"2024-02-19T14:53:02.858593+0000","last_became_active":"2024-02-19T14:53:01.854551+0000","last_became_peered":"2024-02-19T14:53:01.854551+0000","last_unstale":"2024-02-19T14:53:02.858593+0000","last_undegraded":"2024-02-19T14:53:02.858593+0000","last_fullsized":"2024-02-19T14:53:02.858593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:20.426618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940905+0000","last_change":"2024-02-19T14:53:01.867418+0000","last_active":"2024-02-19T14:53:02.940905+0000","last_peered":"2024-02-19T14:53:02.940905+0000","last_clean":"2024-02-19T14:53:02.940905+0000","last_became_active":"2024-02-19T14:53:01.866803+0000","last_became_peered":"2024-02-19T14:53:01.866803+0000","last_unstale":"2024-02-19T14:53:02.940905+0000","last_undegraded":"2024-02-19T14:53:02.940905+0000","last_fullsized":"2024-02-19T14:53:02.940905+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:08.515292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940203+0000","last_change":"2024-02-19T14:53:01.856171+0000","last_active":"2024-02-19T14:53:02.940203+0000","last_peered":"2024-02-19T14:53:02.940203+0000","last_clean":"2024-02-19T14:53:02.940203+0000","last_became_active":"2024-02-19T14:53:01.855924+0000","last_became_peered":"2024-02-19T14:53:01.855924+0000","last_unstale":"2024-02-19T14:53:02.940203+0000","last_undegraded":"2024-02-19T14:53:02.940203+0000","last_fullsized":"2024-02-19T14:53:02.940203+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:12:50.114024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941085+0000","last_change":"2024-02-19T14:53:01.868001+0000","last_active":"2024-02-19T14:53:02.941085+0000","last_peered":"2024-02-19T14:53:02.941085+0000","last_clean":"2024-02-19T14:53:02.941085+0000","last_became_active":"2024-02-19T14:53:01.867724+0000","last_became_peered":"2024-02-19T14:53:01.867724+0000","last_unstale":"2024-02-19T14:53:02.941085+0000","last_undegraded":"2024-02-19T14:53:02.941085+0000","last_fullsized":"2024-02-19T14:53:02.941085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:43.102502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859009+0000","last_change":"2024-02-19T14:53:00.851798+0000","last_active":"2024-02-19T14:53:02.859009+0000","last_peered":"2024-02-19T14:53:02.859009+0000","last_clean":"2024-02-19T14:53:02.859009+0000","last_became_active":"2024-02-19T14:53:00.851517+0000","last_became_peered":"2024-02-19T14:53:00.851517+0000","last_unstale":"2024-02-19T14:53:02.859009+0000","last_undegraded":"2024-02-19T14:53:02.859009+0000","last_fullsized":"2024-02-19T14:53:02.859009+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:35.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940129+0000","last_change":"2024-02-19T14:53:01.862922+0000","last_active":"2024-02-19T14:53:02.940129+0000","last_peered":"2024-02-19T14:53:02.940129+0000","last_clean":"2024-02-19T14:53:02.940129+0000","last_became_active":"2024-02-19T14:53:01.862630+0000","last_became_peered":"2024-02-19T14:53:01.862630+0000","last_unstale":"2024-02-19T14:53:02.940129+0000","last_undegraded":"2024-02-19T14:53:02.940129+0000","last_fullsized":"2024-02-19T14:53:02.940129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:58:08.710699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941205+0000","last_change":"2024-02-19T14:53:00.856064+0000","last_active":"2024-02-19T14:53:02.941205+0000","last_peered":"2024-02-19T14:53:02.941205+0000","last_clean":"2024-02-19T14:53:02.941205+0000","last_became_active":"2024-02-19T14:53:00.855799+0000","last_became_peered":"2024-02-19T14:53:00.855799+0000","last_unstale":"2024-02-19T14:53:02.941205+0000","last_undegraded":"2024-02-19T14:53:02.941205+0000","last_fullsized":"2024-02-19T14:53:02.941205+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:35.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858357+0000","last_change":"2024-02-19T14:53:01.852861+0000","last_active":"2024-02-19T14:53:02.858357+0000","last_peered":"2024-02-19T14:53:02.858357+0000","last_clean":"2024-02-19T14:53:02.858357+0000","last_became_active":"2024-02-19T14:53:01.852537+0000","last_became_peered":"2024-02-19T14:53:01.852537+0000","last_unstale":"2024-02-19T14:53:02.858357+0000","last_undegraded":"2024-02-19T14:53:02.858357+0000","last_fullsized":"2024-02-19T14:53:02.858357+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:02.469083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940214+0000","last_change":"2024-02-19T14:53:00.857583+0000","last_active":"2024-02-19T14:53:02.940214+0000","last_peered":"2024-02-19T14:53:02.940214+0000","last_clean":"2024-02-19T14:53:02.940214+0000","last_became_active":"2024-02-19T14:53:00.857297+0000","last_became_peered":"2024-02-19T14:53:00.857297+0000","last_unstale":"2024-02-19T14:53:02.940214+0000","last_undegraded":"2024-02-19T14:53:02.940214+0000","last_fullsized":"2024-02-19T14:53:02.940214+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:52:00.224317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940286+0000","last_change":"2024-02-19T14:53:01.853039+0000","last_active":"2024-02-19T14:53:02.940286+0000","last_peered":"2024-02-19T14:53:02.940286+0000","last_clean":"2024-02-19T14:53:02.940286+0000","last_became_active":"2024-02-19T14:53:01.852533+0000","last_became_peered":"2024-02-19T14:53:01.852533+0000","last_unstale":"2024-02-19T14:53:02.940286+0000","last_undegraded":"2024-02-19T14:53:02.940286+0000","last_fullsized":"2024-02-19T14:53:02.940286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:28.528999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859045+0000","last_change":"2024-02-19T14:53:00.856489+0000","last_active":"2024-02-19T14:53:02.859045+0000","last_peered":"2024-02-19T14:53:02.859045+0000","last_clean":"2024-02-19T14:53:02.859045+0000","last_became_active":"2024-02-19T14:53:00.856111+0000","last_became_peered":"2024-02-19T14:53:00.856111+0000","last_unstale":"2024-02-19T14:53:02.859045+0000","last_undegraded":"2024-02-19T14:53:02.859045+0000","last_fullsized":"2024-02-19T14:53:02.859045+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:45.255899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939701+0000","last_change":"2024-02-19T14:53:01.860741+0000","last_active":"2024-02-19T14:53:02.939701+0000","last_peered":"2024-02-19T14:53:02.939701+0000","last_clean":"2024-02-19T14:53:02.939701+0000","last_became_active":"2024-02-19T14:53:01.860460+0000","last_became_peered":"2024-02-19T14:53:01.860460+0000","last_unstale":"2024-02-19T14:53:02.939701+0000","last_undegraded":"2024-02-19T14:53:02.939701+0000","last_fullsized":"2024-02-19T14:53:02.939701+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:37.556036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940486+0000","last_change":"2024-02-19T14:53:00.860495+0000","last_active":"2024-02-19T14:53:02.940486+0000","last_peered":"2024-02-19T14:53:02.940486+0000","last_clean":"2024-02-19T14:53:02.940486+0000","last_became_active":"2024-02-19T14:53:00.860213+0000","last_became_peered":"2024-02-19T14:53:00.860213+0000","last_unstale":"2024-02-19T14:53:02.940486+0000","last_undegraded":"2024-02-19T14:53:02.940486+0000","last_fullsized":"2024-02-19T14:53:02.940486+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:19.424285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940399+0000","last_change":"2024-02-19T14:53:01.855671+0000","last_active":"2024-02-19T14:53:02.940399+0000","last_peered":"2024-02-19T14:53:02.940399+0000","last_clean":"2024-02-19T14:53:02.940399+0000","last_became_active":"2024-02-19T14:53:01.855386+0000","last_became_peered":"2024-02-19T14:53:01.855386+0000","last_unstale":"2024-02-19T14:53:02.940399+0000","last_undegraded":"2024-02-19T14:53:02.940399+0000","last_fullsized":"2024-02-19T14:53:02.940399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:28.528999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940251+0000","last_change":"2024-02-19T14:53:00.859368+0000","last_active":"2024-02-19T14:53:02.940251+0000","last_peered":"2024-02-19T14:53:02.940251+0000","last_clean":"2024-02-19T14:53:02.940251+0000","last_became_active":"2024-02-19T14:53:00.858991+0000","last_became_peered":"2024-02-19T14:53:00.858991+0000","last_unstale":"2024-02-19T14:53:02.940251+0000","last_undegraded":"2024-02-19T14:53:02.940251+0000","last_fullsized":"2024-02-19T14:53:02.940251+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:05.530067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858794+0000","last_change":"2024-02-19T14:53:01.851134+0000","last_active":"2024-02-19T14:53:02.858794+0000","last_peered":"2024-02-19T14:53:02.858794+0000","last_clean":"2024-02-19T14:53:02.858794+0000","last_became_active":"2024-02-19T14:53:01.850808+0000","last_became_peered":"2024-02-19T14:53:01.850808+0000","last_unstale":"2024-02-19T14:53:02.858794+0000","last_undegraded":"2024-02-19T14:53:02.858794+0000","last_fullsized":"2024-02-19T14:53:02.858794+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:54.754384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940668+0000","last_change":"2024-02-19T14:53:00.858049+0000","last_active":"2024-02-19T14:53:02.940668+0000","last_peered":"2024-02-19T14:53:02.940668+0000","last_clean":"2024-02-19T14:53:02.940668+0000","last_became_active":"2024-02-19T14:53:00.857749+0000","last_became_peered":"2024-02-19T14:53:00.857749+0000","last_unstale":"2024-02-19T14:53:02.940668+0000","last_undegraded":"2024-02-19T14:53:02.940668+0000","last_fullsized":"2024-02-19T14:53:02.940668+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:56.443441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940711+0000","last_change":"2024-02-19T14:53:01.861410+0000","last_active":"2024-02-19T14:53:02.940711+0000","last_peered":"2024-02-19T14:53:02.940711+0000","last_clean":"2024-02-19T14:53:02.940711+0000","last_became_active":"2024-02-19T14:53:01.861170+0000","last_became_peered":"2024-02-19T14:53:01.861170+0000","last_unstale":"2024-02-19T14:53:02.940711+0000","last_undegraded":"2024-02-19T14:53:02.940711+0000","last_fullsized":"2024-02-19T14:53:02.940711+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:37.403255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858080+0000","last_change":"2024-02-19T14:53:00.847884+0000","last_active":"2024-02-19T14:53:02.858080+0000","last_peered":"2024-02-19T14:53:02.858080+0000","last_clean":"2024-02-19T14:53:02.858080+0000","last_became_active":"2024-02-19T14:53:00.847615+0000","last_became_peered":"2024-02-19T14:53:00.847615+0000","last_unstale":"2024-02-19T14:53:02.858080+0000","last_undegraded":"2024-02-19T14:53:02.858080+0000","last_fullsized":"2024-02-19T14:53:02.858080+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:35.023032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857795+0000","last_change":"2024-02-19T14:53:01.856971+0000","last_active":"2024-02-19T14:53:02.857795+0000","last_peered":"2024-02-19T14:53:02.857795+0000","last_clean":"2024-02-19T14:53:02.857795+0000","last_became_active":"2024-02-19T14:53:01.856654+0000","last_became_peered":"2024-02-19T14:53:01.856654+0000","last_unstale":"2024-02-19T14:53:02.857795+0000","last_undegraded":"2024-02-19T14:53:02.857795+0000","last_fullsized":"2024-02-19T14:53:02.857795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:45.812182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940400+0000","last_change":"2024-02-19T14:53:00.851234+0000","last_active":"2024-02-19T14:53:02.940400+0000","last_peered":"2024-02-19T14:53:02.940400+0000","last_clean":"2024-02-19T14:53:02.940400+0000","last_became_active":"2024-02-19T14:53:00.850983+0000","last_became_peered":"2024-02-19T14:53:00.850983+0000","last_unstale":"2024-02-19T14:53:02.940400+0000","last_undegraded":"2024-02-19T14:53:02.940400+0000","last_fullsized":"2024-02-19T14:53:02.940400+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:28.083904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940339+0000","last_change":"2024-02-19T14:53:01.871178+0000","last_active":"2024-02-19T14:53:02.940339+0000","last_peered":"2024-02-19T14:53:02.940339+0000","last_clean":"2024-02-19T14:53:02.940339+0000","last_became_active":"2024-02-19T14:53:01.870867+0000","last_became_peered":"2024-02-19T14:53:01.870867+0000","last_unstale":"2024-02-19T14:53:02.940339+0000","last_undegraded":"2024-02-19T14:53:02.940339+0000","last_fullsized":"2024-02-19T14:53:02.940339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:20:02.908488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941162+0000","last_change":"2024-02-19T14:53:00.855894+0000","last_active":"2024-02-19T14:53:02.941162+0000","last_peered":"2024-02-19T14:53:02.941162+0000","last_clean":"2024-02-19T14:53:02.941162+0000","last_became_active":"2024-02-19T14:53:00.855396+0000","last_became_peered":"2024-02-19T14:53:00.855396+0000","last_unstale":"2024-02-19T14:53:02.941162+0000","last_undegraded":"2024-02-19T14:53:02.941162+0000","last_fullsized":"2024-02-19T14:53:02.941162+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:36.553704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941232+0000","last_change":"2024-02-19T14:53:01.865982+0000","last_active":"2024-02-19T14:53:02.941232+0000","last_peered":"2024-02-19T14:53:02.941232+0000","last_clean":"2024-02-19T14:53:02.941232+0000","last_became_active":"2024-02-19T14:53:01.865707+0000","last_became_peered":"2024-02-19T14:53:01.865707+0000","last_unstale":"2024-02-19T14:53:02.941232+0000","last_undegraded":"2024-02-19T14:53:02.941232+0000","last_fullsized":"2024-02-19T14:53:02.941232+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:54:59.322422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940916+0000","last_change":"2024-02-19T14:53:00.860141+0000","last_active":"2024-02-19T14:53:02.940916+0000","last_peered":"2024-02-19T14:53:02.940916+0000","last_clean":"2024-02-19T14:53:02.940916+0000","last_became_active":"2024-02-19T14:53:00.859865+0000","last_became_peered":"2024-02-19T14:53:00.859865+0000","last_unstale":"2024-02-19T14:53:02.940916+0000","last_undegraded":"2024-02-19T14:53:02.940916+0000","last_fullsized":"2024-02-19T14:53:02.940916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:38.868963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858533+0000","last_change":"2024-02-19T14:53:01.861637+0000","last_active":"2024-02-19T14:53:02.858533+0000","last_peered":"2024-02-19T14:53:02.858533+0000","last_clean":"2024-02-19T14:53:02.858533+0000","last_became_active":"2024-02-19T14:53:01.860616+0000","last_became_peered":"2024-02-19T14:53:01.860616+0000","last_unstale":"2024-02-19T14:53:02.858533+0000","last_undegraded":"2024-02-19T14:53:02.858533+0000","last_fullsized":"2024-02-19T14:53:02.858533+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:37.556036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940559+0000","last_change":"2024-02-19T14:53:00.855986+0000","last_active":"2024-02-19T14:53:02.940559+0000","last_peered":"2024-02-19T14:53:02.940559+0000","last_clean":"2024-02-19T14:53:02.940559+0000","last_became_active":"2024-02-19T14:53:00.855720+0000","last_became_peered":"2024-02-19T14:53:00.855720+0000","last_unstale":"2024-02-19T14:53:02.940559+0000","last_undegraded":"2024-02-19T14:53:02.940559+0000","last_fullsized":"2024-02-19T14:53:02.940559+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.267643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939701+0000","last_change":"2024-02-19T14:53:01.868599+0000","last_active":"2024-02-19T14:53:02.939701+0000","last_peered":"2024-02-19T14:53:02.939701+0000","last_clean":"2024-02-19T14:53:02.939701+0000","last_became_active":"2024-02-19T14:53:01.868321+0000","last_became_peered":"2024-02-19T14:53:01.868321+0000","last_unstale":"2024-02-19T14:53:02.939701+0000","last_undegraded":"2024-02-19T14:53:02.939701+0000","last_fullsized":"2024-02-19T14:53:02.939701+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.086237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940851+0000","last_change":"2024-02-19T14:53:00.855971+0000","last_active":"2024-02-19T14:53:02.940851+0000","last_peered":"2024-02-19T14:53:02.940851+0000","last_clean":"2024-02-19T14:53:02.940851+0000","last_became_active":"2024-02-19T14:53:00.855710+0000","last_became_peered":"2024-02-19T14:53:00.855710+0000","last_unstale":"2024-02-19T14:53:02.940851+0000","last_undegraded":"2024-02-19T14:53:02.940851+0000","last_fullsized":"2024-02-19T14:53:02.940851+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:36.553704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940286+0000","last_change":"2024-02-19T14:53:01.859675+0000","last_active":"2024-02-19T14:53:02.940286+0000","last_peered":"2024-02-19T14:53:02.940286+0000","last_clean":"2024-02-19T14:53:02.940286+0000","last_became_active":"2024-02-19T14:53:01.859401+0000","last_became_peered":"2024-02-19T14:53:01.859401+0000","last_unstale":"2024-02-19T14:53:02.940286+0000","last_undegraded":"2024-02-19T14:53:02.940286+0000","last_fullsized":"2024-02-19T14:53:02.940286+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:08.912678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940026+0000","last_change":"2024-02-19T14:53:00.856625+0000","last_active":"2024-02-19T14:53:02.940026+0000","last_peered":"2024-02-19T14:53:02.940026+0000","last_clean":"2024-02-19T14:53:02.940026+0000","last_became_active":"2024-02-19T14:53:00.856378+0000","last_became_peered":"2024-02-19T14:53:00.856378+0000","last_unstale":"2024-02-19T14:53:02.940026+0000","last_undegraded":"2024-02-19T14:53:02.940026+0000","last_fullsized":"2024-02-19T14:53:02.940026+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:03.270041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941066+0000","last_change":"2024-02-19T14:53:01.867129+0000","last_active":"2024-02-19T14:53:02.941066+0000","last_peered":"2024-02-19T14:53:02.941066+0000","last_clean":"2024-02-19T14:53:02.941066+0000","last_became_active":"2024-02-19T14:53:01.865959+0000","last_became_peered":"2024-02-19T14:53:01.865959+0000","last_unstale":"2024-02-19T14:53:02.941066+0000","last_undegraded":"2024-02-19T14:53:02.941066+0000","last_fullsized":"2024-02-19T14:53:02.941066+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:55.254128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858606+0000","last_change":"2024-02-19T14:53:00.858256+0000","last_active":"2024-02-19T14:53:02.858606+0000","last_peered":"2024-02-19T14:53:02.858606+0000","last_clean":"2024-02-19T14:53:02.858606+0000","last_became_active":"2024-02-19T14:53:00.857963+0000","last_became_peered":"2024-02-19T14:53:00.857963+0000","last_unstale":"2024-02-19T14:53:02.858606+0000","last_undegraded":"2024-02-19T14:53:02.858606+0000","last_fullsized":"2024-02-19T14:53:02.858606+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:08.703678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940753+0000","last_change":"2024-02-19T14:53:01.860417+0000","last_active":"2024-02-19T14:53:02.940753+0000","last_peered":"2024-02-19T14:53:02.940753+0000","last_clean":"2024-02-19T14:53:02.940753+0000","last_became_active":"2024-02-19T14:53:01.860128+0000","last_became_peered":"2024-02-19T14:53:01.860128+0000","last_unstale":"2024-02-19T14:53:02.940753+0000","last_undegraded":"2024-02-19T14:53:02.940753+0000","last_fullsized":"2024-02-19T14:53:02.940753+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:11.625898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940033+0000","last_change":"2024-02-19T14:53:00.858837+0000","last_active":"2024-02-19T14:53:02.940033+0000","last_peered":"2024-02-19T14:53:02.940033+0000","last_clean":"2024-02-19T14:53:02.940033+0000","last_became_active":"2024-02-19T14:53:00.858545+0000","last_became_peered":"2024-02-19T14:53:00.858545+0000","last_unstale":"2024-02-19T14:53:02.940033+0000","last_undegraded":"2024-02-19T14:53:02.940033+0000","last_fullsized":"2024-02-19T14:53:02.940033+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:44.809850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940550+0000","last_change":"2024-02-19T14:53:01.853452+0000","last_active":"2024-02-19T14:53:02.940550+0000","last_peered":"2024-02-19T14:53:02.940550+0000","last_clean":"2024-02-19T14:53:02.940550+0000","last_became_active":"2024-02-19T14:53:01.853197+0000","last_became_peered":"2024-02-19T14:53:01.853197+0000","last_unstale":"2024-02-19T14:53:02.940550+0000","last_undegraded":"2024-02-19T14:53:02.940550+0000","last_fullsized":"2024-02-19T14:53:02.940550+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940916+0000","last_change":"2024-02-19T14:53:00.858046+0000","last_active":"2024-02-19T14:53:02.940916+0000","last_peered":"2024-02-19T14:53:02.940916+0000","last_clean":"2024-02-19T14:53:02.940916+0000","last_became_active":"2024-02-19T14:53:00.857764+0000","last_became_peered":"2024-02-19T14:53:00.857764+0000","last_unstale":"2024-02-19T14:53:02.940916+0000","last_undegraded":"2024-02-19T14:53:02.940916+0000","last_fullsized":"2024-02-19T14:53:02.940916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:28.083904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940889+0000","last_change":"2024-02-19T14:53:01.856636+0000","last_active":"2024-02-19T14:53:02.940889+0000","last_peered":"2024-02-19T14:53:02.940889+0000","last_clean":"2024-02-19T14:53:02.940889+0000","last_became_active":"2024-02-19T14:53:01.856071+0000","last_became_peered":"2024-02-19T14:53:01.856071+0000","last_unstale":"2024-02-19T14:53:02.940889+0000","last_undegraded":"2024-02-19T14:53:02.940889+0000","last_fullsized":"2024-02-19T14:53:02.940889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:49.806973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858462+0000","last_change":"2024-02-19T14:53:00.855690+0000","last_active":"2024-02-19T14:53:02.858462+0000","last_peered":"2024-02-19T14:53:02.858462+0000","last_clean":"2024-02-19T14:53:02.858462+0000","last_became_active":"2024-02-19T14:53:00.855418+0000","last_became_peered":"2024-02-19T14:53:00.855418+0000","last_unstale":"2024-02-19T14:53:02.858462+0000","last_undegraded":"2024-02-19T14:53:02.858462+0000","last_fullsized":"2024-02-19T14:53:02.858462+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:08.703678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858463+0000","last_change":"2024-02-19T14:53:01.858809+0000","last_active":"2024-02-19T14:53:02.858463+0000","last_peered":"2024-02-19T14:53:02.858463+0000","last_clean":"2024-02-19T14:53:02.858463+0000","last_became_active":"2024-02-19T14:53:01.858526+0000","last_became_peered":"2024-02-19T14:53:01.858526+0000","last_unstale":"2024-02-19T14:53:02.858463+0000","last_undegraded":"2024-02-19T14:53:02.858463+0000","last_fullsized":"2024-02-19T14:53:02.858463+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:39.871295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857722+0000","last_change":"2024-02-19T14:53:00.851104+0000","last_active":"2024-02-19T14:53:02.857722+0000","last_peered":"2024-02-19T14:53:02.857722+0000","last_clean":"2024-02-19T14:53:02.857722+0000","last_became_active":"2024-02-19T14:53:00.850848+0000","last_became_peered":"2024-02-19T14:53:00.850848+0000","last_unstale":"2024-02-19T14:53:02.857722+0000","last_undegraded":"2024-02-19T14:53:02.857722+0000","last_fullsized":"2024-02-19T14:53:02.857722+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:26.461252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857899+0000","last_change":"2024-02-19T14:53:01.856871+0000","last_active":"2024-02-19T14:53:02.857899+0000","last_peered":"2024-02-19T14:53:02.857899+0000","last_clean":"2024-02-19T14:53:02.857899+0000","last_became_active":"2024-02-19T14:53:01.856504+0000","last_became_peered":"2024-02-19T14:53:01.856504+0000","last_unstale":"2024-02-19T14:53:02.857899+0000","last_undegraded":"2024-02-19T14:53:02.857899+0000","last_fullsized":"2024-02-19T14:53:02.857899+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:57.445773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940303+0000","last_change":"2024-02-19T14:53:00.853953+0000","last_active":"2024-02-19T14:53:02.940303+0000","last_peered":"2024-02-19T14:53:02.940303+0000","last_clean":"2024-02-19T14:53:02.940303+0000","last_became_active":"2024-02-19T14:53:00.853684+0000","last_became_peered":"2024-02-19T14:53:00.853684+0000","last_unstale":"2024-02-19T14:53:02.940303+0000","last_undegraded":"2024-02-19T14:53:02.940303+0000","last_fullsized":"2024-02-19T14:53:02.940303+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:40:51.949331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940157+0000","last_change":"2024-02-19T14:53:01.865788+0000","last_active":"2024-02-19T14:53:02.940157+0000","last_peered":"2024-02-19T14:53:02.940157+0000","last_clean":"2024-02-19T14:53:02.940157+0000","last_became_active":"2024-02-19T14:53:01.865436+0000","last_became_peered":"2024-02-19T14:53:01.865436+0000","last_unstale":"2024-02-19T14:53:02.940157+0000","last_undegraded":"2024-02-19T14:53:02.940157+0000","last_fullsized":"2024-02-19T14:53:02.940157+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:54:59.322422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857687+0000","last_change":"2024-02-19T14:53:00.856704+0000","last_active":"2024-02-19T14:53:02.857687+0000","last_peered":"2024-02-19T14:53:02.857687+0000","last_clean":"2024-02-19T14:53:02.857687+0000","last_became_active":"2024-02-19T14:53:00.856455+0000","last_became_peered":"2024-02-19T14:53:00.856455+0000","last_unstale":"2024-02-19T14:53:02.857687+0000","last_undegraded":"2024-02-19T14:53:02.857687+0000","last_fullsized":"2024-02-19T14:53:02.857687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:03:13.889687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940857+0000","last_change":"2024-02-19T14:53:01.862116+0000","last_active":"2024-02-19T14:53:02.940857+0000","last_peered":"2024-02-19T14:53:02.940857+0000","last_clean":"2024-02-19T14:53:02.940857+0000","last_became_active":"2024-02-19T14:53:01.861849+0000","last_became_peered":"2024-02-19T14:53:01.861849+0000","last_unstale":"2024-02-19T14:53:02.940857+0000","last_undegraded":"2024-02-19T14:53:02.940857+0000","last_fullsized":"2024-02-19T14:53:02.940857+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:54.754384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857818+0000","last_change":"2024-02-19T14:53:00.851285+0000","last_active":"2024-02-19T14:53:02.857818+0000","last_peered":"2024-02-19T14:53:02.857818+0000","last_clean":"2024-02-19T14:53:02.857818+0000","last_became_active":"2024-02-19T14:53:00.851043+0000","last_became_peered":"2024-02-19T14:53:00.851043+0000","last_unstale":"2024-02-19T14:53:02.857818+0000","last_undegraded":"2024-02-19T14:53:02.857818+0000","last_fullsized":"2024-02-19T14:53:02.857818+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:33.946706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940615+0000","last_change":"2024-02-19T14:53:01.849826+0000","last_active":"2024-02-19T14:53:02.940615+0000","last_peered":"2024-02-19T14:53:02.940615+0000","last_clean":"2024-02-19T14:53:02.940615+0000","last_became_active":"2024-02-19T14:53:01.849521+0000","last_became_peered":"2024-02-19T14:53:01.849521+0000","last_unstale":"2024-02-19T14:53:02.940615+0000","last_undegraded":"2024-02-19T14:53:02.940615+0000","last_fullsized":"2024-02-19T14:53:02.940615+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:44.106308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941008+0000","last_change":"2024-02-19T14:53:00.854167+0000","last_active":"2024-02-19T14:53:02.941008+0000","last_peered":"2024-02-19T14:53:02.941008+0000","last_clean":"2024-02-19T14:53:02.941008+0000","last_became_active":"2024-02-19T14:53:00.853855+0000","last_became_peered":"2024-02-19T14:53:00.853855+0000","last_unstale":"2024-02-19T14:53:02.941008+0000","last_undegraded":"2024-02-19T14:53:02.941008+0000","last_fullsized":"2024-02-19T14:53:02.941008+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:04.636408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940850+0000","last_change":"2024-02-19T14:53:01.859731+0000","last_active":"2024-02-19T14:53:02.940850+0000","last_peered":"2024-02-19T14:53:02.940850+0000","last_clean":"2024-02-19T14:53:02.940850+0000","last_became_active":"2024-02-19T14:53:01.859446+0000","last_became_peered":"2024-02-19T14:53:01.859446+0000","last_unstale":"2024-02-19T14:53:02.940850+0000","last_undegraded":"2024-02-19T14:53:02.940850+0000","last_fullsized":"2024-02-19T14:53:02.940850+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:49.806973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857521+0000","last_change":"2024-02-19T14:53:00.853229+0000","last_active":"2024-02-19T14:53:02.857521+0000","last_peered":"2024-02-19T14:53:02.857521+0000","last_clean":"2024-02-19T14:53:02.857521+0000","last_became_active":"2024-02-19T14:53:00.852967+0000","last_became_peered":"2024-02-19T14:53:00.852967+0000","last_unstale":"2024-02-19T14:53:02.857521+0000","last_undegraded":"2024-02-19T14:53:02.857521+0000","last_fullsized":"2024-02-19T14:53:02.857521+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:29.129278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940218+0000","last_change":"2024-02-19T14:53:01.863580+0000","last_active":"2024-02-19T14:53:02.940218+0000","last_peered":"2024-02-19T14:53:02.940218+0000","last_clean":"2024-02-19T14:53:02.940218+0000","last_became_active":"2024-02-19T14:53:01.863341+0000","last_became_peered":"2024-02-19T14:53:01.863341+0000","last_unstale":"2024-02-19T14:53:02.940218+0000","last_undegraded":"2024-02-19T14:53:02.940218+0000","last_fullsized":"2024-02-19T14:53:02.940218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:54:59.322422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941072+0000","last_change":"2024-02-19T14:53:00.860030+0000","last_active":"2024-02-19T14:53:02.941072+0000","last_peered":"2024-02-19T14:53:02.941072+0000","last_clean":"2024-02-19T14:53:02.941072+0000","last_became_active":"2024-02-19T14:53:00.859735+0000","last_became_peered":"2024-02-19T14:53:00.859735+0000","last_unstale":"2024-02-19T14:53:02.941072+0000","last_undegraded":"2024-02-19T14:53:02.941072+0000","last_fullsized":"2024-02-19T14:53:02.941072+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:33.267643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940522+0000","last_change":"2024-02-19T14:53:01.871395+0000","last_active":"2024-02-19T14:53:02.940522+0000","last_peered":"2024-02-19T14:53:02.940522+0000","last_clean":"2024-02-19T14:53:02.940522+0000","last_became_active":"2024-02-19T14:53:01.870876+0000","last_became_peered":"2024-02-19T14:53:01.870876+0000","last_unstale":"2024-02-19T14:53:02.940522+0000","last_undegraded":"2024-02-19T14:53:02.940522+0000","last_fullsized":"2024-02-19T14:53:02.940522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:38:54.455007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.857760+0000","last_change":"2024-02-19T14:53:00.854721+0000","last_active":"2024-02-19T14:53:02.857760+0000","last_peered":"2024-02-19T14:53:02.857760+0000","last_clean":"2024-02-19T14:53:02.857760+0000","last_became_active":"2024-02-19T14:53:00.854456+0000","last_became_peered":"2024-02-19T14:53:00.854456+0000","last_unstale":"2024-02-19T14:53:02.857760+0000","last_undegraded":"2024-02-19T14:53:02.857760+0000","last_fullsized":"2024-02-19T14:53:02.857760+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:04.170365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939995+0000","last_change":"2024-02-19T14:53:01.857207+0000","last_active":"2024-02-19T14:53:02.939995+0000","last_peered":"2024-02-19T14:53:02.939995+0000","last_clean":"2024-02-19T14:53:02.939995+0000","last_became_active":"2024-02-19T14:53:01.856941+0000","last_became_peered":"2024-02-19T14:53:01.856941+0000","last_unstale":"2024-02-19T14:53:02.939995+0000","last_undegraded":"2024-02-19T14:53:02.939995+0000","last_fullsized":"2024-02-19T14:53:02.939995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:06.083865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941100+0000","last_change":"2024-02-19T14:53:00.853340+0000","last_active":"2024-02-19T14:53:02.941100+0000","last_peered":"2024-02-19T14:53:02.941100+0000","last_clean":"2024-02-19T14:53:02.941100+0000","last_became_active":"2024-02-19T14:53:00.853074+0000","last_became_peered":"2024-02-19T14:53:00.853074+0000","last_unstale":"2024-02-19T14:53:02.941100+0000","last_undegraded":"2024-02-19T14:53:02.941100+0000","last_fullsized":"2024-02-19T14:53:02.941100+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:44.809850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939873+0000","last_change":"2024-02-19T14:53:01.867117+0000","last_active":"2024-02-19T14:53:02.939873+0000","last_peered":"2024-02-19T14:53:02.939873+0000","last_clean":"2024-02-19T14:53:02.939873+0000","last_became_active":"2024-02-19T14:53:01.866851+0000","last_became_peered":"2024-02-19T14:53:01.866851+0000","last_unstale":"2024-02-19T14:53:02.939873+0000","last_undegraded":"2024-02-19T14:53:02.939873+0000","last_fullsized":"2024-02-19T14:53:02.939873+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:34.269975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941068+0000","last_change":"2024-02-19T14:53:00.855423+0000","last_active":"2024-02-19T14:53:02.941068+0000","last_peered":"2024-02-19T14:53:02.941068+0000","last_clean":"2024-02-19T14:53:02.941068+0000","last_became_active":"2024-02-19T14:53:00.854829+0000","last_became_peered":"2024-02-19T14:53:00.854829+0000","last_unstale":"2024-02-19T14:53:02.941068+0000","last_undegraded":"2024-02-19T14:53:02.941068+0000","last_fullsized":"2024-02-19T14:53:02.941068+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:08.703678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940582+0000","last_change":"2024-02-19T14:53:01.863269+0000","last_active":"2024-02-19T14:53:02.940582+0000","last_peered":"2024-02-19T14:53:02.940582+0000","last_clean":"2024-02-19T14:53:02.940582+0000","last_became_active":"2024-02-19T14:53:01.862991+0000","last_became_peered":"2024-02-19T14:53:01.862991+0000","last_unstale":"2024-02-19T14:53:02.940582+0000","last_undegraded":"2024-02-19T14:53:02.940582+0000","last_fullsized":"2024-02-19T14:53:02.940582+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:38:54.455007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858716+0000","last_change":"2024-02-19T14:53:00.853321+0000","last_active":"2024-02-19T14:53:02.858716+0000","last_peered":"2024-02-19T14:53:02.858716+0000","last_clean":"2024-02-19T14:53:02.858716+0000","last_became_active":"2024-02-19T14:53:00.853035+0000","last_became_peered":"2024-02-19T14:53:00.853035+0000","last_unstale":"2024-02-19T14:53:02.858716+0000","last_undegraded":"2024-02-19T14:53:02.858716+0000","last_fullsized":"2024-02-19T14:53:02.858716+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:20.391008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.858628+0000","last_change":"2024-02-19T14:53:01.853798+0000","last_active":"2024-02-19T14:53:02.858628+0000","last_peered":"2024-02-19T14:53:02.858628+0000","last_clean":"2024-02-19T14:53:02.858628+0000","last_became_active":"2024-02-19T14:53:01.853533+0000","last_became_peered":"2024-02-19T14:53:01.853533+0000","last_unstale":"2024-02-19T14:53:02.858628+0000","last_undegraded":"2024-02-19T14:53:02.858628+0000","last_fullsized":"2024-02-19T14:53:02.858628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.086237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.859261+0000","last_change":"2024-02-19T14:53:00.851456+0000","last_active":"2024-02-19T14:53:02.859261+0000","last_peered":"2024-02-19T14:53:02.859261+0000","last_clean":"2024-02-19T14:53:02.859261+0000","last_became_active":"2024-02-19T14:53:00.851165+0000","last_became_peered":"2024-02-19T14:53:00.851165+0000","last_unstale":"2024-02-19T14:53:02.859261+0000","last_undegraded":"2024-02-19T14:53:02.859261+0000","last_fullsized":"2024-02-19T14:53:02.859261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:20.391008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940568+0000","last_change":"2024-02-19T14:53:01.871288+0000","last_active":"2024-02-19T14:53:02.940568+0000","last_peered":"2024-02-19T14:53:02.940568+0000","last_clean":"2024-02-19T14:53:02.940568+0000","last_became_active":"2024-02-19T14:53:01.870958+0000","last_became_peered":"2024-02-19T14:53:01.870958+0000","last_unstale":"2024-02-19T14:53:02.940568+0000","last_undegraded":"2024-02-19T14:53:02.940568+0000","last_fullsized":"2024-02-19T14:53:02.940568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:28.622298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.939857+0000","last_change":"2024-02-19T14:53:00.851285+0000","last_active":"2024-02-19T14:53:02.939857+0000","last_peered":"2024-02-19T14:53:02.939857+0000","last_clean":"2024-02-19T14:53:02.939857+0000","last_became_active":"2024-02-19T14:53:00.851031+0000","last_became_peered":"2024-02-19T14:53:00.851031+0000","last_unstale":"2024-02-19T14:53:02.939857+0000","last_undegraded":"2024-02-19T14:53:02.939857+0000","last_fullsized":"2024-02-19T14:53:02.939857+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:26:19.424285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940985+0000","last_change":"2024-02-19T14:53:01.858624+0000","last_active":"2024-02-19T14:53:02.940985+0000","last_peered":"2024-02-19T14:53:02.940985+0000","last_clean":"2024-02-19T14:53:02.940985+0000","last_became_active":"2024-02-19T14:53:01.858279+0000","last_became_peered":"2024-02-19T14:53:01.858279+0000","last_unstale":"2024-02-19T14:53:02.940985+0000","last_undegraded":"2024-02-19T14:53:02.940985+0000","last_fullsized":"2024-02-19T14:53:02.940985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:00.831735+0000","last_clean_scrub_stamp":"2024-02-19T14:53:00.831735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:08.239916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.941136+0000","last_change":"2024-02-19T14:53:00.862101+0000","last_active":"2024-02-19T14:53:02.941136+0000","last_peered":"2024-02-19T14:53:02.941136+0000","last_clean":"2024-02-19T14:53:02.941136+0000","last_became_active":"2024-02-19T14:53:00.861896+0000","last_became_peered":"2024-02-19T14:53:00.861896+0000","last_unstale":"2024-02-19T14:53:02.941136+0000","last_undegraded":"2024-02-19T14:53:02.941136+0000","last_fullsized":"2024-02-19T14:53:02.941136+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:52:59.829402+0000","last_clean_scrub_stamp":"2024-02-19T14:52:59.829402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:56.443441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-19T14:53:02.940190+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:02.940190+0000","last_peered":"2024-02-19T14:53:02.940190+0000","last_clean":"2024-02-19T14:53:02.940190+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:02.940190+0000","last_undegraded":"2024-02-19T14:53:02.940190+0000","last_fullsized":"2024-02-19T14:53:02.940190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.502}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.624}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":184499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.497},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.624}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.66300000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":184029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.503}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.73799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":184521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.504}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.44},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.31}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.377}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":185563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.434}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":183987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.82199999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.76900000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.501}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":184007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.224},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.496},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42399999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.44400000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.56699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1024000,"data_stored":774319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.255},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.27400000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.433},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.54700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:05.251 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:53:05.251 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:53:05.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.393+0000 7f2f24d0d640 1 Processor -- start 2024-02-19T14:53:05.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.393+0000 7f2f24d0d640 1 -- start start 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f24d0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2f200c1ef0 0x7f2f200723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f24d0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f20072910 0x7f2f2015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f24d0d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20152290 con 0x7f2f2014b260 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f24d0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2006da10 con 0x7f2f200c1ef0 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f24d0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2006dcf0 con 0x7f2f20072910 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f15d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2f200c1ef0 0x7f2f200723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f15d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2f200c1ef0 0x7f2f200723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47152/0 (socket says 172.21.15.78:47152) 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2145610913 learned_addr learned my addr 172.21.15.78:0/2145610913 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f1ed9d640 1 --2- 172.21.15.78:0/2145610913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f20072910 0x7f2f2015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2145610913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f20072910 msgr2=0x7f2f2015bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f15d9b640 1 --2- 172.21.15.78:0/2145610913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f20072910 0x7f2f2015bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.394+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2145610913 >> v1:172.21.15.150:6789/0 conn(0x7f2f2014b260 legacy=0x7f2f2006e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.395+0000 7f2f1dd9b640 1 -- 172.21.15.78:0/2145610913 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518347352 0 0) 0x7f2f20152290 con 0x7f2f2014b260 2024-02-19T14:53:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.395+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2145610913 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2006f180 con 0x7f2f200c1ef0 2024-02-19T14:53:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.395+0000 7f2f15d9b640 1 --2- 172.21.15.78:0/2145610913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f200723d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009870 tx=0x7f2f0802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a3f8416b6fac9e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.395+0000 7f2f1dd9b640 1 -- 172.21.15.78:0/2145610913 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f080317c0 con 0x7f2f200c1ef0 2024-02-19T14:53:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.395+0000 7f2f1dd9b640 1 -- 172.21.15.78:0/2145610913 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08031920 con 0x7f2f200c1ef0 2024-02-19T14:53:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.395+0000 7f2f1dd9b640 1 -- 172.21.15.78:0/2145610913 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08031cb0 con 0x7f2f200c1ef0 2024-02-19T14:53:05.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2145610913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 msgr2=0x7f2f200723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2145610913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f200723d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009870 tx=0x7f2f0802b860 comp rx=0 tx=0).stop 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2145610913 shutdown_connections 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2145610913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f20072910 0x7f2f2015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2145610913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f200723d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2145610913 >> 172.21.15.78:0/2145610913 conn(0x7f2f200718e0 msgr2=0x7f2f20071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2145610913 shutdown_connections 2024-02-19T14:53:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.396+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2145610913 wait complete. 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 Processor -- start 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 -- start start 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f20072910 0x7f2f2014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f20143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f2014b260 0x7f2f201e8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f200c5e40 con 0x7f2f200c1ef0 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201dbc60 con 0x7f2f20072910 2024-02-19T14:53:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f24d0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201ec6f0 con 0x7f2f2014b260 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f15d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f20143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f15d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f20143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47166/0 (socket says 172.21.15.78:47166) 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.398+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2052411199 learned_addr learned my addr 172.21.15.78:0/2052411199 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f1ed9d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f2014b260 0x7f2f201e8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f1e59c640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f20072910 0x7f2f2014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f2014b260 msgr2=0x7f2f201e8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f15d9b640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f2014b260 0x7f2f201e8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f20072910 msgr2=0x7f2f2014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f15d9b640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f20072910 0x7f2f2014e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f15d9b640 1 -- 172.21.15.78:0/2052411199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f080094d0 con 0x7f2f200c1ef0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f15d9b640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f20143960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2f0802b8a0 tx=0x7f2f080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08032860 con 0x7f2f200c1ef0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f080329c0 con 0x7f2f200c1ef0 2024-02-19T14:53:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f201eb930 con 0x7f2f200c1ef0 2024-02-19T14:53:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08032d70 con 0x7f2f200c1ef0 2024-02-19T14:53:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.399+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f201ebef0 con 0x7f2f200c1ef0 2024-02-19T14:53:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.400+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f20074990 con 0x7f2f200c1ef0 2024-02-19T14:53:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.401+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2f0804e450 con 0x7f2f200c1ef0 2024-02-19T14:53:05.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.401+0000 7f2f177fe640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f0006b810 0x7f2f0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.401+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2f080ac520 con 0x7f2f200c1ef0 2024-02-19T14:53:05.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.401+0000 7f2f1e59c640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f0006b810 0x7f2f0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.401+0000 7f2f1e59c640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f0006b810 0x7f2f0006dcd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2f10000f60 tx=0x7f2f10009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.409+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2f08076dc0 con 0x7f2f200c1ef0 2024-02-19T14:53:05.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.655+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2f200a8910 con 0x7f2f200c1ef0 2024-02-19T14:53:05.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.657+0000 7f2f177fe640 1 -- 172.21.15.78:0/2052411199 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2f0807a480 con 0x7f2f200c1ef0 2024-02-19T14:53:05.660 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:53:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:53:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:53:05.660 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:53:05.661 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:53:05.661 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f0006b810 msgr2=0x7f2f0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f0006b810 0x7f2f0006dcd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2f10000f60 tx=0x7f2f10009040 comp rx=0 tx=0).stop 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 msgr2=0x7f2f20143960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f20143960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2f0802b8a0 tx=0x7f2f080046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 shutdown_connections 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f0006b810 0x7f2f0006dcd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f200c1ef0 0x7f2f20143960 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f2014b260 0x7f2f201e8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 --2- 172.21.15.78:0/2052411199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f20072910 0x7f2f2014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 >> 172.21.15.78:0/2052411199 conn(0x7f2f200718e0 msgr2=0x7f2f20144800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 shutdown_connections 2024-02-19T14:53:05.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.660+0000 7f2f24d0d640 1 -- 172.21.15.78:0/2052411199 wait complete. 2024-02-19T14:53:05.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.771+0000 7f31dd61c640 1 Processor -- start 2024-02-19T14:53:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.771+0000 7f31dd61c640 1 -- start start 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.771+0000 7f31dd61c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31d80c7280 0x7f31d80c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.771+0000 7f31dd61c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31d806c570 0x7f31d806c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31dd61c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d81586a0 con 0x7f31d8152830 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31dd61c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d80c6550 con 0x7f31d806c570 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31dd61c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d80c7c40 con 0x7f31d80c7280 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31d806c570 0x7f31d806c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31d806c570 0x7f31d806c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47182/0 (socket says 172.21.15.78:47182) 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 -- 172.21.15.78:0/3862987879 learned_addr learned my addr 172.21.15.78:0/3862987879 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d67fc640 1 --2- 172.21.15.78:0/3862987879 >> v2:172.21.15.150:3300/0 conn(0x7f31d80c7280 0x7f31d80c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 -- 172.21.15.78:0/3862987879 >> v2:172.21.15.150:3300/0 conn(0x7f31d80c7280 msgr2=0x7f31d80c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 --2- 172.21.15.78:0/3862987879 >> v2:172.21.15.150:3300/0 conn(0x7f31d80c7280 0x7f31d80c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 -- 172.21.15.78:0/3862987879 >> v1:172.21.15.150:6789/0 conn(0x7f31d8152830 legacy=0x7f31d80c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.772+0000 7f31d77fe640 1 -- 172.21.15.78:0/3862987879 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d8158980 con 0x7f31d806c570 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.773+0000 7f31d77fe640 1 --2- 172.21.15.78:0/3862987879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d806c570 0x7f31d806c9b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31d80ad3a0 tx=0x7f31d002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14df5f065669034 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.773+0000 7f31d5ffb640 1 -- 172.21.15.78:0/3862987879 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d002c9f0 con 0x7f31d806c570 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.773+0000 7f31d5ffb640 1 -- 172.21.15.78:0/3862987879 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d002cb50 con 0x7f31d806c570 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.773+0000 7f31d5ffb640 1 -- 172.21.15.78:0/3862987879 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00343f0 con 0x7f31d806c570 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 -- 172.21.15.78:0/3862987879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d806c570 msgr2=0x7f31d806c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3862987879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d806c570 0x7f31d806c9b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31d80ad3a0 tx=0x7f31d002be50 comp rx=0 tx=0).stop 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 -- 172.21.15.78:0/3862987879 shutdown_connections 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3862987879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d806c570 0x7f31d806c9b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3862987879 >> v2:172.21.15.150:3300/0 conn(0x7f31d80c7280 0x7f31d80c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 -- 172.21.15.78:0/3862987879 >> 172.21.15.78:0/3862987879 conn(0x7f31d80ca8a0 msgr2=0x7f31d80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 -- 172.21.15.78:0/3862987879 shutdown_connections 2024-02-19T14:53:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.774+0000 7f31dd61c640 1 -- 172.21.15.78:0/3862987879 wait complete. 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.775+0000 7f31dd61c640 1 Processor -- start 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.775+0000 7f31dd61c640 1 -- start start 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d806c570 0x7f31d81ea600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31d80c7280 0x7f31d80ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 0x7f31d80ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8159320 con 0x7f31d8152830 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80aa2c0 con 0x7f31d806c570 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d80ab120 con 0x7f31d80c7280 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 0x7f31d80ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 0x7f31d80ac540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47198/0 (socket says 172.21.15.78:47198) 2024-02-19T14:53:05.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 -- 172.21.15.78:0/3989204025 learned_addr learned my addr 172.21.15.78:0/3989204025 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d67fc640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31d80c7280 0x7f31d80ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d6ffd640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d806c570 0x7f31d81ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 -- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31d80c7280 msgr2=0x7f31d80ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31d80c7280 0x7f31d80ab5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 -- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d806c570 msgr2=0x7f31d81ea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d806c570 0x7f31d81ea600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 -- 172.21.15.78:0/3989204025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0009b30 con 0x7f31d8152830 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d67fc640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31d80c7280 0x7f31d80ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d6ffd640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d806c570 0x7f31d81ea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31d77fe640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 0x7f31d80ac540 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009f40 tx=0x7f31d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0009510 con 0x7f31d8152830 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d003e450 con 0x7f31d8152830 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d814e190 con 0x7f31d8152830 2024-02-19T14:53:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d003e7e0 con 0x7f31d8152830 2024-02-19T14:53:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.776+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d80aa510 con 0x7f31d8152830 2024-02-19T14:53:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.777+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d80ab2b0 con 0x7f31d8152830 2024-02-19T14:53:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.778+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f31d003e940 con 0x7f31d8152830 2024-02-19T14:53:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.778+0000 7f31bf7fe640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c406b810 0x7f31c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.778+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f31d00ad090 con 0x7f31d8152830 2024-02-19T14:53:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.778+0000 7f31d6ffd640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c406b810 0x7f31c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.779+0000 7f31d6ffd640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c406b810 0x7f31c406dcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f31c8006fd0 tx=0x7f31c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:05.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.783+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31d00779b0 con 0x7f31d8152830 2024-02-19T14:53:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.997+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31d8074990 con 0x7f31d8152830 2024-02-19T14:53:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.998+0000 7f31bf7fe640 1 -- 172.21.15.78:0/3989204025 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f31d007b070 con 0x7f31d8152830 2024-02-19T14:53:06.000 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:06.000 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:03.204838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5247":{"gid":5247,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3231075223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3231075223},{"type":"v1","addr":"172.21.15.78:6837","nonce":3231075223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 15 2024-02-19T14:53:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c406b810 msgr2=0x7f31c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c406b810 0x7f31c406dcd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f31c8006fd0 tx=0x7f31c8008040 comp rx=0 tx=0).stop 2024-02-19T14:53:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 msgr2=0x7f31d80ac540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 0x7f31d80ac540 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009f40 tx=0x7f31d0004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 shutdown_connections 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c406b810 0x7f31c406dcd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31d8152830 0x7f31d80ac540 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31d80c7280 0x7f31d80ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:05.999+0000 7f31dd61c640 1 --2- 172.21.15.78:0/3989204025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31d806c570 0x7f31d81ea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.000+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 >> 172.21.15.78:0/3989204025 conn(0x7f31d80ca8a0 msgr2=0x7f31d80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.000+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 shutdown_connections 2024-02-19T14:53:06.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.000+0000 7f31dd61c640 1 -- 172.21.15.78:0/3989204025 wait complete. 2024-02-19T14:53:06.269 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:53:06.267+0000 7f1f6c7af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:06.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:01.188620+0000', 'modified': '2024-02-19T14:53:03.204838+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5247': {'gid': 5247, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3231075223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 3231075223}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3231075223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:06.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:53:06.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:53:06.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.516+0000 7f499496c640 1 Processor -- start 2024-02-19T14:53:06.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.516+0000 7f499496c640 1 -- start start 2024-02-19T14:53:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.517+0000 7f499496c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f49900c78a0 0x7f4990159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.517+0000 7f499496c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4990155010 0x7f49901523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.517+0000 7f499496c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499006ffa0 con 0x7f49900c4d80 2024-02-19T14:53:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.517+0000 7f499496c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499006b720 con 0x7f4990155010 2024-02-19T14:53:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.517+0000 7f499496c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499006fc70 con 0x7f49900c78a0 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4990155010 0x7f49901523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4990155010 0x7f49901523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47204/0 (socket says 172.21.15.78:47204) 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 -- 172.21.15.78:0/537163428 learned_addr learned my addr 172.21.15.78:0/537163428 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 -- 172.21.15.78:0/537163428 >> v2:172.21.15.150:3300/0 conn(0x7f49900c78a0 msgr2=0x7f4990159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498e59c640 1 --2- 172.21.15.78:0/537163428 >> v2:172.21.15.150:3300/0 conn(0x7f49900c78a0 0x7f4990159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 --2- 172.21.15.78:0/537163428 >> v2:172.21.15.150:3300/0 conn(0x7f49900c78a0 0x7f4990159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 -- 172.21.15.78:0/537163428 >> v1:172.21.15.150:6789/0 conn(0x7f49900c4d80 legacy=0x7f49900c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.518+0000 7f498ed9d640 1 -- 172.21.15.78:0/537163428 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499006b930 con 0x7f4990155010 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.519+0000 7f498ed9d640 1 --2- 172.21.15.78:0/537163428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f49901523d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f49900c8420 tx=0x7f498802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8c95271cf9c496 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.519+0000 7f498d59a640 1 -- 172.21.15.78:0/537163428 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498802c9f0 con 0x7f4990155010 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.519+0000 7f498d59a640 1 -- 172.21.15.78:0/537163428 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498802cb50 con 0x7f4990155010 2024-02-19T14:53:06.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.519+0000 7f498d59a640 1 -- 172.21.15.78:0/537163428 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49880343f0 con 0x7f4990155010 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.519+0000 7f499496c640 1 -- 172.21.15.78:0/537163428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 msgr2=0x7f49901523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.519+0000 7f499496c640 1 --2- 172.21.15.78:0/537163428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f49901523d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f49900c8420 tx=0x7f498802be50 comp rx=0 tx=0).stop 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.520+0000 7f499496c640 1 -- 172.21.15.78:0/537163428 shutdown_connections 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.520+0000 7f499496c640 1 --2- 172.21.15.78:0/537163428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f49901523d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.520+0000 7f499496c640 1 --2- 172.21.15.78:0/537163428 >> v2:172.21.15.150:3300/0 conn(0x7f49900c78a0 0x7f4990159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.520+0000 7f499496c640 1 -- 172.21.15.78:0/537163428 >> 172.21.15.78:0/537163428 conn(0x7f4990066e70 msgr2=0x7f49900a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.520+0000 7f499496c640 1 -- 172.21.15.78:0/537163428 shutdown_connections 2024-02-19T14:53:06.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.520+0000 7f499496c640 1 -- 172.21.15.78:0/537163428 wait complete. 2024-02-19T14:53:06.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.521+0000 7f499496c640 1 Processor -- start 2024-02-19T14:53:06.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.522+0000 7f499496c640 1 -- start start 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49900c4d80 0x7f49900ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49900c78a0 0x7f499014bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f4990150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49900c5760 con 0x7f4990155010 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901eae00 con 0x7f49900c78a0 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49901eb0e0 con 0x7f49900c4d80 2024-02-19T14:53:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f4990150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f4990150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47210/0 (socket says 172.21.15.78:47210) 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 -- 172.21.15.78:0/1088460734 learned_addr learned my addr 172.21.15.78:0/1088460734 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498e59c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49900c4d80 0x7f49900ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498dd9b640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49900c78a0 0x7f499014bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 -- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49900c4d80 msgr2=0x7f49900ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49900c4d80 0x7f49900ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 -- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49900c78a0 msgr2=0x7f499014bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49900c78a0 0x7f499014bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 -- 172.21.15.78:0/1088460734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988009b30 con 0x7f4990155010 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498e59c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49900c4d80 0x7f49900ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498dd9b640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49900c78a0 0x7f499014bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f498ed9d640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f4990150eb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4988036040 tx=0x7f4988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f497effd640 1 -- 172.21.15.78:0/1088460734 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498803e450 con 0x7f4990155010 2024-02-19T14:53:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f497effd640 1 -- 172.21.15.78:0/1088460734 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498803e5d0 con 0x7f4990155010 2024-02-19T14:53:06.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f497effd640 1 -- 172.21.15.78:0/1088460734 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498803e980 con 0x7f4990155010 2024-02-19T14:53:06.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.523+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49900ab570 con 0x7f4990155010 2024-02-19T14:53:06.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.524+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49900abb30 con 0x7f4990155010 2024-02-19T14:53:06.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.525+0000 7f497effd640 1 -- 172.21.15.78:0/1088460734 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f498803eae0 con 0x7f4990155010 2024-02-19T14:53:06.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.525+0000 7f497effd640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f497006b810 0x7f497006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.526+0000 7f498e59c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f497006b810 0x7f497006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.526+0000 7f497effd640 1 -- 172.21.15.78:0/1088460734 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49880ad250 con 0x7f4990155010 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.526+0000 7f498e59c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f497006b810 0x7f497006dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f498000a9d0 tx=0x7f4980008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.526+0000 7f4997621740 1 Processor -- start 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 -- start start 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edabae0 0x56356ede5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edaa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ed65cd0 con 0x56356ede88b0 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ed18f20 con 0x56356ede5740 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356ed16450 con 0x56356edabae0 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edaa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edaa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47222/0 (socket says 172.21.15.78:47222) 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f498ed9d640 1 -- 172.21.15.78:0/3081851440 learned_addr learned my addr 172.21.15.78:0/3081851440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f498e59c640 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.527+0000 7f498dd9b640 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edabae0 0x56356ede5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f498ed9d640 1 -- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edabae0 msgr2=0x56356ede5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f498ed9d640 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edabae0 0x56356ede5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f498ed9d640 1 -- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 msgr2=0x56356ede68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f498ed9d640 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f498ed9d640 1 -- 172.21.15.78:0/3081851440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56356ecc6f20 con 0x56356ede88b0 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f498ed9d640 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edaa860 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4988083a10 tx=0x7f498807b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e161f4ea471540 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f496ffff640 1 -- 172.21.15.78:0/3081851440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498807b830 con 0x56356ede88b0 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f496ffff640 1 -- 172.21.15.78:0/3081851440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498807f980 con 0x56356ede88b0 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f496ffff640 1 -- 172.21.15.78:0/3081851440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988083f40 con 0x56356ede88b0 2024-02-19T14:53:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f4997621740 1 -- 172.21.15.78:0/3081851440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 msgr2=0x56356edaa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.528+0000 7f4997621740 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edaa860 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4988083a10 tx=0x7f498807b430 comp rx=0 tx=0).stop 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 -- 172.21.15.78:0/3081851440 shutdown_connections 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edaa860 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edabae0 0x56356ede5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 --2- 172.21.15.78:0/3081851440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 -- 172.21.15.78:0/3081851440 >> 172.21.15.78:0/3081851440 conn(0x56356eda7120 msgr2=0x56356edcbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 -- 172.21.15.78:0/3081851440 shutdown_connections 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 -- 172.21.15.78:0/3081851440 wait complete. 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 -- start start 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.529+0000 7f4997621740 1 Processor -- start 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356eddf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edc9060 0x56356ede1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ed11c30 con 0x56356ede88b0 2024-02-19T14:53:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ecb6d30 con 0x56356ede5740 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356e9e5310 con 0x56356edc9060 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f4997621740 10 client.0 _ll_register_callbacks cb 0x56356ecbed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f498dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356eddf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f498dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356eddf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47232/0 (socket says 172.21.15.78:47232) 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f498dd9b640 1 -- 172.21.15.78:0/3746919117 learned_addr learned my addr 172.21.15.78:0/3746919117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f498ed9d640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edc9060 0x56356ede1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.530+0000 7f498e59c640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498dd9b640 1 -- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edc9060 msgr2=0x56356ede1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498dd9b640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edc9060 0x56356ede1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498dd9b640 1 -- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 msgr2=0x56356ede35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498dd9b640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498dd9b640 1 -- 172.21.15.78:0/3746919117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49900abb30 con 0x56356ede88b0 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498ed9d640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edc9060 0x56356ede1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498e59c640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498dd9b640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356eddf990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4984003f10 tx=0x7f49840014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8b5a3bdb772300 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f496d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f496d7fa640 1 -- 172.21.15.78:0/3746919117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984005460 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f496d7fa640 1 -- 172.21.15.78:0/3746919117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4984014400 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56356ea116c0 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f496d7fa640 1 -- 172.21.15.78:0/3746919117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49840145c0 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 10 client.5415 fetch_fsmap learned FSMap version 15 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x56356ed102b0 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f496d7fa640 1 -- 172.21.15.78:0/3746919117 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56356ea116c0 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f496d7fa640 1 -- 172.21.15.78:0/3746919117 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4984014720 con 0x56356ede88b0 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 10 client.5415 fetch_fsmap finished waiting for FSMap version 15 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 10 client.5415 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 20 client.5415 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356edfa130 0x56356edfa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 4 client.5415 mds_command: new command op to 5247 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 --> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56356ea62270 con 0x56356edfa130 2024-02-19T14:53:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.531+0000 7f498e59c640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356edfa130 0x56356edfa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f498e59c640 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356edfa130 0x56356edfa510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=802bffd310a33b02 server_cookie=c527a2144567ed4 in_seq=0 out_seq=0 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f496d7fa640 10 client.5415 ms_handle_connect on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f496d7fa640 1 -- 172.21.15.78:0/3746919117 <== mds.0 v2:172.21.15.78:6836/3231075223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56356ea62270 con 0x56356edfa130 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f496d7fa640 10 client.5415 handle_command_reply: tid=0 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f4997621740 1 client.5415 shutdown 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 msgr2=0x56356eddf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.532+0000 7f4997621740 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356eddf990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4984003f10 tx=0x7f49840014f0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 shutdown_connections 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356edfa130 0x56356edfa510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356eddf990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356edc9060 0x56356ede1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 --2- 172.21.15.78:0/3746919117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f496d7fa640 0 client.5415 ms_handle_reset on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 >> 172.21.15.78:0/3746919117 conn(0x56356edabae0 msgr2=0x56356edcbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 shutdown_connections 2024-02-19T14:53:06.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 1 -- 172.21.15.78:0/3746919117 wait complete. 2024-02-19T14:53:06.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.533+0000 7f4997621740 20 client.5415 trim_cache size 0 max 16384 2024-02-19T14:53:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.549+0000 7f4997621740 1 Processor -- start 2024-02-19T14:53:06.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.549+0000 7f4997621740 1 -- start start 2024-02-19T14:53:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f4997621740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edc82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356edc8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f4997621740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ec97490 con 0x56356ede88b0 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ede6c00 con 0x56356ede5740 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356eddf0f0 con 0x56356ede7380 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edc82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edc82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47244/0 (socket says 172.21.15.78:47244) 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 -- 172.21.15.78:0/2518209654 learned_addr learned my addr 172.21.15.78:0/2518209654 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498ed9d640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356edc8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 -- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 msgr2=0x56356edc8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498e59c640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356edc8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 -- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 msgr2=0x56356ede6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 -- 172.21.15.78:0/2518209654 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56356ecaca20 con 0x56356ede88b0 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498ed9d640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356edc8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:06.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498e59c640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f498dd9b640 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edc82f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4984005300 tx=0x7f498400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87242be712ee3bf0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f496f7fe640 1 -- 172.21.15.78:0/2518209654 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840019f0 con 0x56356ede88b0 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f496f7fe640 1 -- 172.21.15.78:0/2518209654 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4984014400 con 0x56356ede88b0 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.550+0000 7f496f7fe640 1 -- 172.21.15.78:0/2518209654 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840147b0 con 0x56356ede88b0 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 -- 172.21.15.78:0/2518209654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 msgr2=0x56356edc82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edc82f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4984005300 tx=0x7f498400e000 comp rx=0 tx=0).stop 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 -- 172.21.15.78:0/2518209654 shutdown_connections 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356edc82f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356edc8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 --2- 172.21.15.78:0/2518209654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356ede6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 -- 172.21.15.78:0/2518209654 >> 172.21.15.78:0/2518209654 conn(0x56356edabae0 msgr2=0x56356ede33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 -- 172.21.15.78:0/2518209654 shutdown_connections 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 -- 172.21.15.78:0/2518209654 wait complete. 2024-02-19T14:53:06.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.551+0000 7f4997621740 1 -- start start 2024-02-19T14:53:06.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 Processor -- start 2024-02-19T14:53:06.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356edff580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356ee573d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356ee09410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356edaa390 con 0x56356ede88b0 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356eda62f0 con 0x56356ede5740 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356ee59ee0 con 0x56356ede7380 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.552+0000 7f4997621740 10 client.0 _ll_register_callbacks cb 0x7f4984005050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356ee09410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356ee09410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47260/0 (socket says 172.21.15.78:47260) 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 -- 172.21.15.78:0/975177704 learned_addr learned my addr 172.21.15.78:0/975177704 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498e59c640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356edff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498dd9b640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356ee573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 -- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 msgr2=0x56356ee573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356ee573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 -- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 msgr2=0x56356edff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356edff580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 -- 172.21.15.78:0/975177704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56356ed102b0 con 0x56356ede88b0 2024-02-19T14:53:06.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498dd9b640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356ee573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f498ed9d640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356ee09410 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f498807b530 tx=0x7f49880ad040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae3c263d251c754 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f496d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f496d7fa640 1 -- 172.21.15.78:0/975177704 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498807b830 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f496d7fa640 1 -- 172.21.15.78:0/975177704 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4988083f40 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f498807f980 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f496d7fa640 1 -- 172.21.15.78:0/975177704 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f498809a220 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 10 client.5421 fetch_fsmap learned FSMap version 15 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x56356ecd2780 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f496d7fa640 1 -- 172.21.15.78:0/975177704 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498807f980 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f496d7fa640 1 -- 172.21.15.78:0/975177704 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f498804f020 con 0x56356ede88b0 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 10 client.5421 fetch_fsmap finished waiting for FSMap version 15 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 10 client.5421 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 20 client.5421 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356ede4ed0 0x56356ee09950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.553+0000 7f4997621740 4 client.5421 mds_command: new command op to 5247 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 --> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56356ea4e610 con 0x56356ede4ed0 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f498e59c640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356ede4ed0 0x56356ee09950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f498e59c640 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356ede4ed0 0x56356ee09950 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7889c9ed5a2aab0a server_cookie=442a119fbb2f031f in_seq=0 out_seq=0 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f496d7fa640 10 client.5421 ms_handle_connect on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f496d7fa640 1 -- 172.21.15.78:0/975177704 <== mds.0 v2:172.21.15.78:6836/3231075223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56356ea4e610 con 0x56356ede4ed0 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f496d7fa640 10 client.5421 handle_command_reply: tid=0 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 client.5421 shutdown 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 msgr2=0x56356ee09410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356ee09410 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f498807b530 tx=0x7f49880ad040 comp rx=0 tx=0).stop 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 shutdown_connections 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x56356ede4ed0 0x56356ee09950 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56356ede88b0 0x56356ee09410 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f496d7fa640 0 client.5421 ms_handle_reset on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56356ede7380 0x56356ee573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 --2- 172.21.15.78:0/975177704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56356ede5740 0x56356edff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 >> 172.21.15.78:0/975177704 conn(0x56356edabae0 msgr2=0x56356edc82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 shutdown_connections 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.554+0000 7f4997621740 1 -- 172.21.15.78:0/975177704 wait complete. 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f4997621740 20 client.5421 trim_cache size 0 max 16384 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5247, 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:53:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:53:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.7163532100000003, 2024-02-19T14:53:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 15, 2024-02-19T14:53:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 38, 2024-02-19T14:53:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 38, 2024-02-19T14:53:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 12.157117305 2024-02-19T14:53:06.565 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f497006b810 msgr2=0x7f497006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f497006b810 0x7f497006dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f498000a9d0 tx=0x7f4980008040 comp rx=0 tx=0).stop 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 msgr2=0x7f4990150eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f4990150eb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4988036040 tx=0x7f4988004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 shutdown_connections 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f497006b810 0x7f497006dcd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4990155010 0x7f4990150eb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49900c78a0 0x7f499014bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 --2- 172.21.15.78:0/1088460734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49900c4d80 0x7f49900ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 >> 172.21.15.78:0/1088460734 conn(0x7f4990066e70 msgr2=0x7f49900c2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 shutdown_connections 2024-02-19T14:53:06.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.555+0000 7f499496c640 1 -- 172.21.15.78:0/1088460734 wait complete. 2024-02-19T14:53:06.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 Processor -- start 2024-02-19T14:53:06.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 -- start start 2024-02-19T14:53:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43700c7910 0x7f4370154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700c65a0 con 0x7f4370070720 2024-02-19T14:53:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43700c7530 con 0x7f43700c7910 2024-02-19T14:53:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4376cfa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43701537b0 con 0x7f43700c5560 2024-02-19T14:53:06.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.667+0000 7f4367fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43700c7910 0x7f4370154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4367fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43700c7910 0x7f4370154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47270/0 (socket says 172.21.15.78:47270) 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4367fff640 1 -- 172.21.15.78:0/3246108733 learned_addr learned my addr 172.21.15.78:0/3246108733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4374a96640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4367fff640 1 -- 172.21.15.78:0/3246108733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 msgr2=0x7f43700c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4367fff640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4367fff640 1 -- 172.21.15.78:0/3246108733 >> v1:172.21.15.150:6789/0 conn(0x7f4370070720 legacy=0x7f4370070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4367fff640 1 -- 172.21.15.78:0/3246108733 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370153a90 con 0x7f43700c7910 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.668+0000 7f4374a96640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4367fff640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43700c7910 0x7f4370154180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4360009870 tx=0x7f436002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e42c984ac75827 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f43677fe640 1 -- 172.21.15.78:0/3246108733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43600317c0 con 0x7f43700c7910 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f43677fe640 1 -- 172.21.15.78:0/3246108733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360031920 con 0x7f43700c7910 2024-02-19T14:53:06.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f43677fe640 1 -- 172.21.15.78:0/3246108733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360031cb0 con 0x7f43700c7910 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4376cfa640 1 -- 172.21.15.78:0/3246108733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43700c7910 msgr2=0x7f4370154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4376cfa640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43700c7910 0x7f4370154180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4360009870 tx=0x7f436002b860 comp rx=0 tx=0).stop 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4376cfa640 1 -- 172.21.15.78:0/3246108733 shutdown_connections 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4376cfa640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43700c7910 0x7f4370154180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4376cfa640 1 --2- 172.21.15.78:0/3246108733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.669+0000 7f4376cfa640 1 -- 172.21.15.78:0/3246108733 >> 172.21.15.78:0/3246108733 conn(0x7f437006b4b0 msgr2=0x7f43700c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.670+0000 7f4376cfa640 1 -- 172.21.15.78:0/3246108733 shutdown_connections 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.670+0000 7f4376cfa640 1 -- 172.21.15.78:0/3246108733 wait complete. 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 Processor -- start 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 -- start start 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 0x7f43700aa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43700c7910 0x7f4370148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43700a87c0 con 0x7f4370070720 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43700aad00 con 0x7f43700c7910 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4376cfa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437006c300 con 0x7f43700c5560 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.671+0000 7f4374a96640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 0x7f43700aa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 0x7f43700aa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47276/0 (socket says 172.21.15.78:47276) 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 -- 172.21.15.78:0/1695092241 learned_addr learned my addr 172.21.15.78:0/1695092241 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:06.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4375297640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43700c7910 0x7f4370148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4367fff640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 -- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 msgr2=0x7f43700aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700aa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 -- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43700c7910 msgr2=0x7f4370148850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43700c7910 0x7f4370148850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 -- 172.21.15.78:0/1695092241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600094d0 con 0x7f4370070720 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4367fff640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4374a96640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 0x7f43700aa280 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f436c00d800 tx=0x7f436c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c004030 con 0x7f4370070720 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c004190 con 0x7f4370070720 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43700ab2d0 con 0x7f4370070720 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436c004520 con 0x7f4370070720 2024-02-19T14:53:06.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.672+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43701eaf10 con 0x7f4370070720 2024-02-19T14:53:06.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.673+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43700a8a30 con 0x7f4370070720 2024-02-19T14:53:06.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.674+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f436c009640 con 0x7f4370070720 2024-02-19T14:53:06.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.674+0000 7f43657fa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f435c06b810 0x7f435c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:06.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.674+0000 7f4367fff640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f435c06b810 0x7f435c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:06.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.674+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f436c08a4b0 con 0x7f4370070720 2024-02-19T14:53:06.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.675+0000 7f4367fff640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f435c06b810 0x7f435c06dcd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4360036000 tx=0x7f436003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:06.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.679+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f436c054dd0 con 0x7f4370070720 2024-02-19T14:53:06.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.928+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4370074990 con 0x7f4370070720 2024-02-19T14:53:06.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.928+0000 7f43657fa640 1 -- 172.21.15.78:0/1695092241 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f436c059e60 con 0x7f4370070720 2024-02-19T14:53:06.931 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:06.931 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:03.204838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5247":{"gid":5247,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3231075223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3231075223},{"type":"v1","addr":"172.21.15.78:6837","nonce":3231075223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:06.931 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 15 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f435c06b810 msgr2=0x7f435c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f435c06b810 0x7f435c06dcd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4360036000 tx=0x7f436003d040 comp rx=0 tx=0).stop 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 msgr2=0x7f43700aa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 0x7f43700aa280 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f436c00d800 tx=0x7f436c00dcc0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 shutdown_connections 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f435c06b810 0x7f435c06dcd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43700c7910 0x7f4370148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43700c5560 0x7f43700aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 --2- 172.21.15.78:0/1695092241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4370070720 0x7f43700aa280 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 >> 172.21.15.78:0/1695092241 conn(0x7f437006b4b0 msgr2=0x7f43700c42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 shutdown_connections 2024-02-19T14:53:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:06.930+0000 7f4376cfa640 1 -- 172.21.15.78:0/1695092241 wait complete. 2024-02-19T14:53:06.944 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:53:01.188620+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5247": { "addr": "172.21.15.78:6837/3231075223", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 3231075223, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3231075223, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5247, "incarnation": 13, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-19T14:53:03.204838+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5247 } } } ], "standbys": [ { "addr": "172.21.15.150:6835/4259811019", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 4259811019, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 4259811019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 11, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4766, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6834/3578832669", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3578832669, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3578832669, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5020, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6835/1762664317", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1762664317, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 1762664317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5244, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:53:06.947 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:53:06.947 INFO:tasks.cephfs.test_failover:current = [{'gid': 5244, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/1762664317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1762664317}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 1762664317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5020, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3578832669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3578832669}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3578832669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5247, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3231075223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 3231075223}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3231075223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4766, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 11}] 2024-02-19T14:53:06.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-19T14:53:07.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.050+0000 7f634d1c9640 1 Processor -- start 2024-02-19T14:53:07.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.050+0000 7f634d1c9640 1 -- start start 2024-02-19T14:53:07.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634d1c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f634814cd50 0x7f63480c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634d1c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63481563b0 con 0x7f6348156860 2024-02-19T14:53:07.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634d1c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6348153cd0 con 0x7f63481537a0 2024-02-19T14:53:07.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634d1c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634806b6e0 con 0x7f634814cd50 2024-02-19T14:53:07.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634759e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6348156860 0x7f6348156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46462/0 (socket says 172.21.15.78:46462) 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634759e640 1 -- 172.21.15.78:0/367405264 learned_addr learned my addr 172.21.15.78:0/367405264 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940768042 0 0) 0x7f63481563b0 con 0x7f6348156860 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6334003520 con 0x7f6348156860 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f6346d9d640 1 --2- 172.21.15.78:0/367405264 >> v2:172.21.15.150:3300/0 conn(0x7f634814cd50 0x7f63480c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2036976947 0 0) 0x7f6334003520 con 0x7f6348156860 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 >> v2:172.21.15.150:3300/0 conn(0x7f634814cd50 msgr2=0x7f63480c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 --2- 172.21.15.78:0/367405264 >> v2:172.21.15.150:3300/0 conn(0x7f634814cd50 0x7f63480c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 >> v1:172.21.15.150:6789/0 conn(0x7f63481537a0 legacy=0x7f63480c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634806b980 con 0x7f6348156860 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f63400019f0 con 0x7f6348156860 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 >> v1:172.21.15.78:6789/0 conn(0x7f6348156860 legacy=0x7f6348156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6334003970 0x7f6334003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63340047c0 0x7f6334004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f633400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634806b6e0 con 0x7f6334005380 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633400c210 con 0x7f63340047c0 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63340107e0 con 0x7f6334003970 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.051+0000 7f634759e640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f633400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f633ffff640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63340047c0 0x7f6334004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f6346d9d640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6334003970 0x7f6334003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 -- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6334003970 msgr2=0x7f6334003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6334003970 0x7f6334003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 -- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63340047c0 msgr2=0x7f6334004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63340047c0 0x7f6334004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 -- 172.21.15.78:0/367405264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334003520 con 0x7f6334005380 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f6346d9d640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6334003970 0x7f6334003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:07.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 -- 172.21.15.78:0/367405264 reap_dead start 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.052+0000 7f634759e640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f633400b980 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f634000a830 tx=0x7f63340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c5822b28b5f423 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63400072d0 con 0x7f6334005380 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634002f070 con 0x7f6334005380 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634659c640 1 -- 172.21.15.78:0/367405264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63400076b0 con 0x7f6334005380 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 -- 172.21.15.78:0/367405264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 msgr2=0x7f633400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f633400b980 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f634000a830 tx=0x7f63340057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 -- 172.21.15.78:0/367405264 shutdown_connections 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 --2- 172.21.15.78:0/367405264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f633400b980 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 -- 172.21.15.78:0/367405264 >> 172.21.15.78:0/367405264 conn(0x7f6348066e70 msgr2=0x7f63480a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 -- 172.21.15.78:0/367405264 shutdown_connections 2024-02-19T14:53:07.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.053+0000 7f634d1c9640 1 -- 172.21.15.78:0/367405264 wait complete. 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.054+0000 7f634d1c9640 1 Processor -- start 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.054+0000 7f634d1c9640 1 -- start start 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634d1c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f63480c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634d1c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6348150a30 0x7f63481e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634d1c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63481e7cb0 0x7f634814be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634d1c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63481e6c00 con 0x7f6334005380 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634d1c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63481e8130 con 0x7f6348150a30 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634d1c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63481507c0 con 0x7f63481e7cb0 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f63480c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f63480c5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47300/0 (socket says 172.21.15.78:47300) 2024-02-19T14:53:07.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 -- 172.21.15.78:0/2893174399 learned_addr learned my addr 172.21.15.78:0/2893174399 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f633ffff640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6348150a30 0x7f63481e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f634759e640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63481e7cb0 0x7f634814be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 -- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63481e7cb0 msgr2=0x7f634814be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63481e7cb0 0x7f634814be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 -- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6348150a30 msgr2=0x7f63481e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6348150a30 0x7f63481e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.055+0000 7f6346d9d640 1 -- 172.21.15.78:0/2893174399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63481e6280 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.056+0000 7f6346d9d640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f63480c5d00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f633800f550 tx=0x7f633800fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.056+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338013810 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.056+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6338013970 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.056+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6348150360 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.056+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338013ce0 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.056+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63481e6820 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.057+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6348074990 con 0x7f6334005380 2024-02-19T14:53:07.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.057+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f633801f460 con 0x7f6334005380 2024-02-19T14:53:07.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.058+0000 7f633f7fe640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f632806b810 0x7f632806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.058+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f633808a700 con 0x7f6334005380 2024-02-19T14:53:07.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.058+0000 7f633ffff640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f632806b810 0x7f632806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.058+0000 7f633ffff640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f632806b810 0x7f632806dcd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6348154e70 tx=0x7f6330009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:07.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.066+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6338055020 con 0x7f6334005380 2024-02-19T14:53:07.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.238+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f63480c4230 con 0x7f6334005380 2024-02-19T14:53:07.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.241+0000 7f633f7fe640 1 -- 172.21.15.78:0/2893174399 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v12)=0 v12) v1 ==== 116+0+0 (secure 0 0 0) 0x7f63380586e0 con 0x7f6334005380 2024-02-19T14:53:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.243+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f632806b810 msgr2=0x7f632806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.243+0000 7f634d1c9640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f632806b810 0x7f632806dcd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6348154e70 tx=0x7f6330009040 comp rx=0 tx=0).stop 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.243+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 msgr2=0x7f63480c5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.243+0000 7f634d1c9640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f63480c5d00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f633800f550 tx=0x7f633800fa10 comp rx=0 tx=0).stop 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 shutdown_connections 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f632806b810 0x7f632806dcd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63481e7cb0 0x7f634814be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6348150a30 0x7f63481e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 --2- 172.21.15.78:0/2893174399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6334005380 0x7f63480c5d00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 >> 172.21.15.78:0/2893174399 conn(0x7f6348066e70 msgr2=0x7f6348152d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 shutdown_connections 2024-02-19T14:53:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.244+0000 7f634d1c9640 1 -- 172.21.15.78:0/2893174399 wait complete. 2024-02-19T14:53:07.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:07.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.355+0000 7f664cb8c640 1 Processor -- start 2024-02-19T14:53:07.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.355+0000 7f664cb8c640 1 -- start start 2024-02-19T14:53:07.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664cb8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66480add70 0x7f66480c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664cb8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480c2f60 0x7f664806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664cb8c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66480ae670 con 0x7f664806f530 2024-02-19T14:53:07.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664cb8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66480716b0 con 0x7f66480add70 2024-02-19T14:53:07.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664cb8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66480c33a0 con 0x7f66480c2f60 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66480add70 0x7f66480c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f6645d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480c2f60 0x7f664806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66480add70 0x7f66480c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:56156/0 (socket says 172.21.15.78:56156) 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f6646d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f664806f530 0x7f66480705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57144/0 (socket says 172.21.15.78:57144) 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 -- 172.21.15.78:0/2817408574 learned_addr learned my addr 172.21.15.78:0/2817408574 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 -- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480c2f60 msgr2=0x7f664806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 --2- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480c2f60 0x7f664806eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 -- 172.21.15.78:0/2817408574 >> v1:172.21.15.150:6789/0 conn(0x7f664806f530 legacy=0x7f66480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f664659c640 1 -- 172.21.15.78:0/2817408574 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6648153780 con 0x7f66480add70 2024-02-19T14:53:07.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.357+0000 7f6645d9b640 1 --2- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480c2f60 0x7f664806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:07.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.358+0000 7f664659c640 1 --2- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480add70 0x7f66480c4980 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6638009e50 tx=0x7f663802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47ab4957de0c49e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:07.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.358+0000 7f664559a640 1 -- 172.21.15.78:0/2817408574 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663802ca70 con 0x7f66480add70 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.358+0000 7f664559a640 1 -- 172.21.15.78:0/2817408574 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663802cbd0 con 0x7f66480add70 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.358+0000 7f664559a640 1 -- 172.21.15.78:0/2817408574 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380343f0 con 0x7f66480add70 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 -- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480add70 msgr2=0x7f66480c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 --2- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480add70 0x7f66480c4980 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6638009e50 tx=0x7f663802bed0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 -- 172.21.15.78:0/2817408574 shutdown_connections 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 --2- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480c2f60 0x7f664806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 --2- 172.21.15.78:0/2817408574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480add70 0x7f66480c4980 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 -- 172.21.15.78:0/2817408574 >> 172.21.15.78:0/2817408574 conn(0x7f66480c85e0 msgr2=0x7f66480c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 -- 172.21.15.78:0/2817408574 shutdown_connections 2024-02-19T14:53:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.359+0000 7f664cb8c640 1 -- 172.21.15.78:0/2817408574 wait complete. 2024-02-19T14:53:07.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.361+0000 7f664cb8c640 1 Processor -- start 2024-02-19T14:53:07.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.361+0000 7f664cb8c640 1 -- start start 2024-02-19T14:53:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664cb8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 0x7f66481ebc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664cb8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480add70 0x7f664814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664cb8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480c2f60 0x7f6648156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664cb8c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648151a00 con 0x7f664806f530 2024-02-19T14:53:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664cb8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481514d0 con 0x7f66480c2f60 2024-02-19T14:53:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664cb8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6648156530 con 0x7f66480add70 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 0x7f66481ebc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 0x7f66481ebc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47312/0 (socket says 172.21.15.78:47312) 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f664659c640 1 -- 172.21.15.78:0/478102800 learned_addr learned my addr 172.21.15.78:0/478102800 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f6646d9d640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480c2f60 0x7f6648156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.362+0000 7f6645d9b640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480add70 0x7f664814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664659c640 1 -- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480add70 msgr2=0x7f664814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664659c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480add70 0x7f664814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664659c640 1 -- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480c2f60 msgr2=0x7f6648156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664659c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480c2f60 0x7f6648156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664659c640 1 -- 172.21.15.78:0/478102800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630009400 con 0x7f664806f530 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f6645d9b640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480add70 0x7f664814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664659c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 0x7f66481ebc70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6638036040 tx=0x7f6638004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:07.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638009510 con 0x7f664806f530 2024-02-19T14:53:07.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663803e450 con 0x7f664806f530 2024-02-19T14:53:07.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6638009b30 con 0x7f664806f530 2024-02-19T14:53:07.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663803e800 con 0x7f664806f530 2024-02-19T14:53:07.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.363+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6648152400 con 0x7f664806f530 2024-02-19T14:53:07.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.364+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66480676a0 con 0x7f664806f530 2024-02-19T14:53:07.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.365+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f663803e960 con 0x7f664806f530 2024-02-19T14:53:07.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.365+0000 7f663effd640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f662806b810 0x7f662806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:07.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.365+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f66380acfb0 con 0x7f664806f530 2024-02-19T14:53:07.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.365+0000 7f6645d9b640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f662806b810 0x7f662806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:07.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.365+0000 7f6645d9b640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f662806b810 0x7f662806dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6648154820 tx=0x7f6630008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.369+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f66380e0dc0 con 0x7f664806f530 2024-02-19T14:53:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.582+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6648074990 con 0x7f664806f530 2024-02-19T14:53:07.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.583+0000 7f663effd640 1 -- 172.21.15.78:0/478102800 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f6638077850 con 0x7f664806f530 2024-02-19T14:53:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:07.586 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:03.204838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5247":{"gid":5247,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3231075223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3231075223},{"type":"v1","addr":"172.21.15.78:6837","nonce":3231075223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:07.586 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 15 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f662806b810 msgr2=0x7f662806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f662806b810 0x7f662806dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6648154820 tx=0x7f6630008040 comp rx=0 tx=0).stop 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 msgr2=0x7f66481ebc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 0x7f66481ebc70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6638036040 tx=0x7f6638004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 shutdown_connections 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f662806b810 0x7f662806dcd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66480c2f60 0x7f6648156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66480add70 0x7f664814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 --2- 172.21.15.78:0/478102800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f664806f530 0x7f66481ebc70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 >> 172.21.15.78:0/478102800 conn(0x7f66480c85e0 msgr2=0x7f66480c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:07.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 shutdown_connections 2024-02-19T14:53:07.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:07.585+0000 7f664cb8c640 1 -- 172.21.15.78:0/478102800 wait complete. 2024-02-19T14:53:08.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:01.188620+0000', 'modified': '2024-02-19T14:53:03.204838+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5247': {'gid': 5247, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3231075223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 3231075223}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3231075223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:08.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:53:08.002 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:53:08.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.106+0000 7f458efae640 1 Processor -- start 2024-02-19T14:53:08.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.106+0000 7f458efae640 1 -- start start 2024-02-19T14:53:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458efae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45880c4980 0x7f45880c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458efae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880705b0 0x7f4588070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458efae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45880ae670 con 0x7f45880add70 2024-02-19T14:53:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458efae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45880716b0 con 0x7f45880c4980 2024-02-19T14:53:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458efae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4588153780 con 0x7f45880705b0 2024-02-19T14:53:08.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458d54b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880705b0 0x7f4588070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f4587fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45880c4980 0x7f45880c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458cd4a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f45880add70 0x7f458806eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53184/0 (socket says 172.21.15.78:53184) 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458cd4a640 1 -- 172.21.15.78:0/1493452635 learned_addr learned my addr 172.21.15.78:0/1493452635 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458d54b640 1 -- 172.21.15.78:0/1493452635 >> v2:172.21.15.150:3300/0 conn(0x7f45880c4980 msgr2=0x7f45880c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458d54b640 1 --2- 172.21.15.78:0/1493452635 >> v2:172.21.15.150:3300/0 conn(0x7f45880c4980 0x7f45880c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.107+0000 7f458d54b640 1 -- 172.21.15.78:0/1493452635 >> v1:172.21.15.150:6789/0 conn(0x7f45880add70 legacy=0x7f458806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.108+0000 7f458d54b640 1 -- 172.21.15.78:0/1493452635 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588153a60 con 0x7f45880705b0 2024-02-19T14:53:08.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.108+0000 7f4587fff640 1 --2- 172.21.15.78:0/1493452635 >> v2:172.21.15.150:3300/0 conn(0x7f45880c4980 0x7f45880c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.108+0000 7f458d54b640 1 --2- 172.21.15.78:0/1493452635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880705b0 0x7f4588070a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4580009d90 tx=0x7f458002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5559f37db3cc772 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.108+0000 7f45877fe640 1 -- 172.21.15.78:0/1493452635 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458002cbc0 con 0x7f45880705b0 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f45877fe640 1 -- 172.21.15.78:0/1493452635 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458002cd20 con 0x7f45880705b0 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f45877fe640 1 -- 172.21.15.78:0/1493452635 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580034660 con 0x7f45880705b0 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 -- 172.21.15.78:0/1493452635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880705b0 msgr2=0x7f4588070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 --2- 172.21.15.78:0/1493452635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880705b0 0x7f4588070a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4580009d90 tx=0x7f458002be10 comp rx=0 tx=0).stop 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 -- 172.21.15.78:0/1493452635 shutdown_connections 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 --2- 172.21.15.78:0/1493452635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880705b0 0x7f4588070a00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 --2- 172.21.15.78:0/1493452635 >> v2:172.21.15.150:3300/0 conn(0x7f45880c4980 0x7f45880c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 -- 172.21.15.78:0/1493452635 >> 172.21.15.78:0/1493452635 conn(0x7f45880c85e0 msgr2=0x7f45880c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:08.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 -- 172.21.15.78:0/1493452635 shutdown_connections 2024-02-19T14:53:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.109+0000 7f458efae640 1 -- 172.21.15.78:0/1493452635 wait complete. 2024-02-19T14:53:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.110+0000 7f458efae640 1 Processor -- start 2024-02-19T14:53:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.110+0000 7f458efae640 1 -- start start 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458efae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45880705b0 0x7f45881e1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458efae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880add70 0x7f458814bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458efae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 0x7f458814c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458efae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4588151460 con 0x7f45880c4980 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458efae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45880a9f60 con 0x7f45880705b0 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458efae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45881eea60 con 0x7f45880add70 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 0x7f458814c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 0x7f458814c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38306/0 (socket says 172.21.15.78:38306) 2024-02-19T14:53:08.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 -- 172.21.15.78:0/2449286316 learned_addr learned my addr 172.21.15.78:0/2449286316 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458cd4a640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45880705b0 0x7f45881e1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f4587fff640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880add70 0x7f458814bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 -- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880add70 msgr2=0x7f458814bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880add70 0x7f458814bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 -- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45880705b0 msgr2=0x7f45881e1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45880705b0 0x7f45881e1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.111+0000 7f458d54b640 1 -- 172.21.15.78:0/2449286316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574009400 con 0x7f45880c4980 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.112+0000 7f458d54b640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 0x7f458814c090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4580009d90 tx=0x7f4580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.112+0000 7f45857fa640 1 -- 172.21.15.78:0/2449286316 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580040650 con 0x7f45880c4980 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.112+0000 7f45857fa640 1 -- 172.21.15.78:0/2449286316 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458002ce90 con 0x7f45880c4980 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.112+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4580009a70 con 0x7f45880c4980 2024-02-19T14:53:08.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.112+0000 7f45857fa640 1 -- 172.21.15.78:0/2449286316 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580040a10 con 0x7f45880c4980 2024-02-19T14:53:08.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.112+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45881df030 con 0x7f45880c4980 2024-02-19T14:53:08.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.113+0000 7f45857fa640 1 -- 172.21.15.78:0/2449286316 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4580040b70 con 0x7f45880c4980 2024-02-19T14:53:08.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.114+0000 7f45857fa640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f456c06b840 0x7f456c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.114+0000 7f458cd4a640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f456c06b840 0x7f456c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.114+0000 7f45857fa640 1 -- 172.21.15.78:0/2449286316 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f45800acf60 con 0x7f45880c4980 2024-02-19T14:53:08.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.115+0000 7f458cd4a640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f456c06b840 0x7f456c06dd00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f45880a7eb0 tx=0x7f457c007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.115+0000 7f4591b81740 1 Processor -- start 2024-02-19T14:53:08.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.115+0000 7f4591b81740 1 -- start start 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b9802d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4591b81740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 0x558b9802cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b97ff1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4591b81740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b97faccd0 con 0x558b97ff2ae0 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b97f5ff20 con 0x558b9802c740 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b97f5d450 con 0x558b9802f8b0 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 0x558b9802cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 0x558b9802cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38314/0 (socket says 172.21.15.78:38314) 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 -- 172.21.15.78:0/2112916834 learned_addr learned my addr 172.21.15.78:0/2112916834 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f458cd4a640 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b9802d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 -- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 msgr2=0x558b97ff1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b97ff1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 -- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 msgr2=0x558b9802d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b9802d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 -- 172.21.15.78:0/2112916834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b97f0df20 con 0x558b97ff2ae0 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f458cd4a640 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b9802d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f4587fff640 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 0x558b9802cb20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4574009ce0 tx=0x7f4574011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495a84e8862eb68c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f45727fc640 1 -- 172.21.15.78:0/2112916834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574004030 con 0x558b97ff2ae0 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f45727fc640 1 -- 172.21.15.78:0/2112916834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574004190 con 0x558b97ff2ae0 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.116+0000 7f45727fc640 1 -- 172.21.15.78:0/2112916834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574003610 con 0x558b97ff2ae0 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 -- 172.21.15.78:0/2112916834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 msgr2=0x558b9802cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 0x558b9802cb20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4574009ce0 tx=0x7f4574011690 comp rx=0 tx=0).stop 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 -- 172.21.15.78:0/2112916834 shutdown_connections 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b97ff2ae0 0x558b9802cb20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b97ff1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 --2- 172.21.15.78:0/2112916834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b9802d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 -- 172.21.15.78:0/2112916834 >> 172.21.15.78:0/2112916834 conn(0x558b97fee120 msgr2=0x558b98012b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:08.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 -- 172.21.15.78:0/2112916834 shutdown_connections 2024-02-19T14:53:08.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.117+0000 7f4591b81740 1 -- 172.21.15.78:0/2112916834 wait complete. 2024-02-19T14:53:08.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.118+0000 7f4591b81740 1 -- start start 2024-02-19T14:53:08.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.118+0000 7f4591b81740 1 Processor -- start 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 0x558b98033480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b980342a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802a730 0x558b9800f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b97f58c30 con 0x558b9802c740 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b97efdd30 con 0x558b9802a730 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b97c2c310 con 0x558b9802f8b0 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f458cd4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 0x558b98033480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.119+0000 7f4591b81740 10 client.0 _ll_register_callbacks cb 0x558b97f05d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 0x558b98033480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38330/0 (socket says 172.21.15.78:38330) 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 -- 172.21.15.78:0/1175543353 learned_addr learned my addr 172.21.15.78:0/1175543353 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f4587fff640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b980342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458d54b640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802a730 0x558b9800f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 -- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 msgr2=0x558b980342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b980342a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 -- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802a730 msgr2=0x558b9800f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802a730 0x558b9800f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 -- 172.21.15.78:0/1175543353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4574009910 con 0x558b9802c740 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f458cd4a640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 0x558b98033480 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f457c013450 tx=0x7f457c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374a84191d23b63d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f45677fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f45677fe640 1 -- 172.21.15.78:0/1175543353 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c013e90 con 0x558b9802c740 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f45677fe640 1 -- 172.21.15.78:0/1175543353 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f457c01b040 con 0x558b9802c740 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b97c586c0 con 0x558b9802c740 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f45677fe640 1 -- 172.21.15.78:0/1175543353 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f457c012560 con 0x558b9802c740 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f4591b81740 10 client.5445 fetch_fsmap learned FSMap version 15 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.120+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x558b97f572b0 con 0x558b9802c740 2024-02-19T14:53:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f45677fe640 1 -- 172.21.15.78:0/1175543353 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558b97c586c0 con 0x558b9802c740 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f45677fe640 1 -- 172.21.15.78:0/1175543353 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f457c012070 con 0x558b9802c740 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4591b81740 10 client.5445 fetch_fsmap finished waiting for FSMap version 15 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4591b81740 10 client.5445 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4591b81740 20 client.5445 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4591b81740 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b98041070 0x558b98041450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4591b81740 4 client.5445 mds_command: new command op to 5247 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 --> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b97ca9270 con 0x558b98041070 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4587fff640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b98041070 0x558b98041450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f45677fe640 10 client.5445 ms_handle_connect on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.121+0000 7f4587fff640 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b98041070 0x558b98041450 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa16dee95eb0a7a8 server_cookie=6975514d04859bad in_seq=0 out_seq=0 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f45677fe640 1 -- 172.21.15.78:0/1175543353 <== mds.0 v2:172.21.15.78:6836/3231075223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558b97ca9270 con 0x558b98041070 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f45677fe640 10 client.5445 handle_command_reply: tid=0 2024-02-19T14:53:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 client.5445 shutdown 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 msgr2=0x558b98033480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 0x558b98033480 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f457c013450 tx=0x7f457c000bc0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 shutdown_connections 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b98041070 0x558b98041450 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802a730 0x558b9800f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b980342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f45677fe640 0 client.5445 ms_handle_reset on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 --2- 172.21.15.78:0/1175543353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802c740 0x558b98033480 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 >> 172.21.15.78:0/1175543353 conn(0x558b97ff2ae0 msgr2=0x558b9802d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 shutdown_connections 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 1 -- 172.21.15.78:0/1175543353 wait complete. 2024-02-19T14:53:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.122+0000 7f4591b81740 20 client.5445 trim_cache size 0 max 16384 2024-02-19T14:53:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.139+0000 7f4591b81740 1 Processor -- start 2024-02-19T14:53:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.139+0000 7f4591b81740 1 -- start start 2024-02-19T14:53:08.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4591b81740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b9802e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b98012ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b98012ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4591b81740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b97ede490 con 0x558b9802a730 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b97fe6ce0 con 0x558b9802c740 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b980514f0 con 0x558b9802f8b0 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b9802e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b9802e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38342/0 (socket says 172.21.15.78:38342) 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 -- 172.21.15.78:0/3813090951 learned_addr learned my addr 172.21.15.78:0/3813090951 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f4587fff640 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b98012ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458d54b640 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b98012ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 -- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 msgr2=0x558b98012ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b98012ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 -- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 msgr2=0x558b98012ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b98012ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 -- 172.21.15.78:0/3813090951 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b97ef3a20 con 0x558b9802a730 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.140+0000 7f458cd4a640 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b9802e030 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f457c012880 tx=0x7f457c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36c9b2e50d120b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4571ffb640 1 -- 172.21.15.78:0/3813090951 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c013e90 con 0x558b9802a730 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4571ffb640 1 -- 172.21.15.78:0/3813090951 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c00b070 con 0x558b9802a730 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4571ffb640 1 -- 172.21.15.78:0/3813090951 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0129c0 con 0x558b9802a730 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 -- 172.21.15.78:0/3813090951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 msgr2=0x558b9802e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b9802e030 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f457c012880 tx=0x7f457c01f000 comp rx=0 tx=0).stop 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 -- 172.21.15.78:0/3813090951 shutdown_connections 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802f8b0 0x558b98012ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802c740 0x558b98012ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 --2- 172.21.15.78:0/3813090951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b9802e030 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 -- 172.21.15.78:0/3813090951 >> 172.21.15.78:0/3813090951 conn(0x558b97ff2ae0 msgr2=0x558b98031fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 -- 172.21.15.78:0/3813090951 shutdown_connections 2024-02-19T14:53:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.141+0000 7f4591b81740 1 -- 172.21.15.78:0/3813090951 wait complete. 2024-02-19T14:53:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.142+0000 7f4591b81740 1 -- start start 2024-02-19T14:53:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.142+0000 7f4591b81740 1 Processor -- start 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b98012cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802c740 0x558b9809a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802f8b0 0x558b9809d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b98047850 con 0x558b9802a730 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b98030f90 con 0x558b9802f8b0 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b98030d80 con 0x558b9802c740 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4591b81740 10 client.0 _ll_register_callbacks cb 0x7f457c004b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b98012cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b98012cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38348/0 (socket says 172.21.15.78:38348) 2024-02-19T14:53:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 -- 172.21.15.78:0/71441280 learned_addr learned my addr 172.21.15.78:0/71441280 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f4587fff640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802c740 0x558b9809a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458d54b640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802f8b0 0x558b9809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 -- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802c740 msgr2=0x558b9809a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802c740 0x558b9809a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 -- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802f8b0 msgr2=0x558b9809d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802f8b0 0x558b9809d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 -- 172.21.15.78:0/71441280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b97f572b0 con 0x558b9802a730 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f458cd4a640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b98012cc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f457c013450 tx=0x7f457c00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ac36e80fe46325 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.143+0000 7f45677fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f45677fe640 1 -- 172.21.15.78:0/71441280 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c03a070 con 0x558b9802a730 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f45677fe640 1 -- 172.21.15.78:0/71441280 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f457c037950 con 0x558b9802a730 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b98032d20 con 0x558b9802a730 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f45677fe640 1 -- 172.21.15.78:0/71441280 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f457c013c20 con 0x558b9802a730 2024-02-19T14:53:08.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 10 client.5451 fetch_fsmap learned FSMap version 15 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x558b97f19780 con 0x558b9802a730 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f45677fe640 1 -- 172.21.15.78:0/71441280 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c043070 con 0x558b9802a730 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f45677fe640 1 -- 172.21.15.78:0/71441280 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f457c01b070 con 0x558b9802a730 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 10 client.5451 fetch_fsmap finished waiting for FSMap version 15 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 10 client.5451 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 20 client.5451 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b9809a690 0x558b980a0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 4 client.5451 mds_command: new command op to 5247 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 --> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4574004190 con 0x558b9809a690 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.144+0000 7f4587fff640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b9809a690 0x558b980a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f45677fe640 10 client.5451 ms_handle_connect on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4587fff640 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b9809a690 0x558b980a0300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74838517fb2180b2 server_cookie=b697277d20e9299d in_seq=0 out_seq=0 2024-02-19T14:53:08.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f45677fe640 1 -- 172.21.15.78:0/71441280 <== mds.0 v2:172.21.15.78:6836/3231075223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f4574004190 con 0x558b9809a690 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f45677fe640 10 client.5451 handle_command_reply: tid=0 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 client.5451 shutdown 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 msgr2=0x558b98012cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b98012cc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f457c013450 tx=0x7f457c00a000 comp rx=0 tx=0).stop 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 shutdown_connections 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:6836/3231075223,v1:172.21.15.78:6837/3231075223] conn(0x558b9809a690 0x558b980a0300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558b9802f8b0 0x558b9809d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x558b9802c740 0x558b9809a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 --2- 172.21.15.78:0/71441280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558b9802a730 0x558b98012cc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 >> 172.21.15.78:0/71441280 conn(0x558b97ff2ae0 msgr2=0x558b98031fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f45677fe640 0 client.5451 ms_handle_reset on v2:172.21.15.78:6836/3231075223 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 shutdown_connections 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 1 -- 172.21.15.78:0/71441280 wait complete. 2024-02-19T14:53:08.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.145+0000 7f4591b81740 20 client.5451 trim_cache size 0 max 16384 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5247, 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 6.3070025960000002, 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 15, 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 38, 2024-02-19T14:53:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 38, 2024-02-19T14:53:08.156 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 13.747761083 2024-02-19T14:53:08.156 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:08.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f456c06b840 msgr2=0x7f456c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f456c06b840 0x7f456c06dd00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f45880a7eb0 tx=0x7f457c007e80 comp rx=0 tx=0).stop 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 msgr2=0x7f458814c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 0x7f458814c090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4580009d90 tx=0x7f4580004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 shutdown_connections 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f456c06b840 0x7f456c06dd00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45880c4980 0x7f458814c090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45880add70 0x7f458814bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 --2- 172.21.15.78:0/2449286316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45880705b0 0x7f45881e1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 >> 172.21.15.78:0/2449286316 conn(0x7f45880c85e0 msgr2=0x7f45880c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 shutdown_connections 2024-02-19T14:53:08.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:08.146+0000 7f458efae640 1 -- 172.21.15.78:0/2449286316 wait complete. 2024-02-19T14:53:08.159 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:53:01.188620+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5247": { "addr": "172.21.15.78:6837/3231075223", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 3231075223, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3231075223, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5247, "incarnation": 13, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-19T14:53:03.204838+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5247 } } } ], "standbys": [ { "addr": "172.21.15.150:6835/4259811019", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 4259811019, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 4259811019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 11, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4766, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6834/3578832669", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3578832669, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3578832669, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5020, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6835/1762664317", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1762664317, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 1762664317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5244, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:53:08.160 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 2, 'name': 'd', 'state': 'up:active'}] 2024-02-19T14:53:08.160 INFO:tasks.cephfs.test_failover:current = [{'gid': 5244, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/1762664317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1762664317}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 1762664317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5020, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3578832669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3578832669}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3578832669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5247, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3231075223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 3231075223}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3231075223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4766, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 11}] 2024-02-19T14:53:08.161 DEBUG:tasks.cephfs.test_failover:2 != -1 2024-02-19T14:53:08.161 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:53:10.567 INFO:tasks.ceph.mds.c.smithi078.stderr: -210> 2024-02-19T14:53:06.267+0000 7f1f6c7af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:10.591 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:53:13.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.269+0000 7f51b169d640 1 Processor -- start 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.269+0000 7f51b169d640 1 -- start start 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51b169d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51ac152700 0x7f51ac0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51b169d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c5730 0x7f51ac06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51b169d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ac1551f0 con 0x7f51ac0c8e00 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51b169d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ac155490 con 0x7f51ac152700 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51b169d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac155770 con 0x7f51ac0c5730 2024-02-19T14:53:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51ac152700 0x7f51ac0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f51ac152700 0x7f51ac0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38372/0 (socket says 172.21.15.78:38372) 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 -- 172.21.15.78:0/2147201047 learned_addr learned my addr 172.21.15.78:0/2147201047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aa7fc640 1 --2- 172.21.15.78:0/2147201047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c5730 0x7f51ac06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 -- 172.21.15.78:0/2147201047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c5730 msgr2=0x7f51ac06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 --2- 172.21.15.78:0/2147201047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c5730 0x7f51ac06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 -- 172.21.15.78:0/2147201047 >> v1:172.21.15.150:6789/0 conn(0x7f51ac0c8e00 legacy=0x7f51ac158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.270+0000 7f51aaffd640 1 -- 172.21.15.78:0/2147201047 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ac155a50 con 0x7f51ac152700 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.271+0000 7f51aaffd640 1 --2- 172.21.15.78:0/2147201047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac152700 0x7f51ac0c51f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f519c009870 tx=0x7f519c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283bf39ccefe2d4e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.271+0000 7f51a9ffb640 1 -- 172.21.15.78:0/2147201047 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c031640 con 0x7f51ac152700 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.271+0000 7f51a9ffb640 1 -- 172.21.15.78:0/2147201047 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c0317a0 con 0x7f51ac152700 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.271+0000 7f51a9ffb640 1 -- 172.21.15.78:0/2147201047 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c031b30 con 0x7f51ac152700 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 -- 172.21.15.78:0/2147201047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac152700 msgr2=0x7f51ac0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 --2- 172.21.15.78:0/2147201047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac152700 0x7f51ac0c51f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f519c009870 tx=0x7f519c02b8f0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 -- 172.21.15.78:0/2147201047 shutdown_connections 2024-02-19T14:53:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 --2- 172.21.15.78:0/2147201047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c5730 0x7f51ac06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 --2- 172.21.15.78:0/2147201047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac152700 0x7f51ac0c51f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 -- 172.21.15.78:0/2147201047 >> 172.21.15.78:0/2147201047 conn(0x7f51ac0cac90 msgr2=0x7f51ac0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 -- 172.21.15.78:0/2147201047 shutdown_connections 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.272+0000 7f51b169d640 1 -- 172.21.15.78:0/2147201047 wait complete. 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.273+0000 7f51b169d640 1 Processor -- start 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.273+0000 7f51b169d640 1 -- start start 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51b169d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 0x7f51ac1ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51b169d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c8e00 0x7f51ac1ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51b169d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac152700 0x7f51ac148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51b169d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac1eeab0 con 0x7f51ac0c5730 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51b169d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac1eecd0 con 0x7f51ac152700 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51b169d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac157d10 con 0x7f51ac0c8e00 2024-02-19T14:53:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 0x7f51ac1ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 0x7f51ac1ee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38374/0 (socket says 172.21.15.78:38374) 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 -- 172.21.15.78:0/863966510 learned_addr learned my addr 172.21.15.78:0/863966510 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51ab7fe640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac152700 0x7f51ac148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aa7fc640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c8e00 0x7f51ac1ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 -- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c8e00 msgr2=0x7f51ac1ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c8e00 0x7f51ac1ee570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 -- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac152700 msgr2=0x7f51ac148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac152700 0x7f51ac148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 -- 172.21.15.78:0/863966510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c0094d0 con 0x7f51ac0c5730 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51ab7fe640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac152700 0x7f51ac148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aa7fc640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c8e00 0x7f51ac1ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51aaffd640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 0x7f51ac1ee030 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f519c000c00 tx=0x7f519c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.274+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c03ac60 con 0x7f51ac0c5730 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.275+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c03adc0 con 0x7f51ac0c5730 2024-02-19T14:53:13.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.275+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac1573e0 con 0x7f51ac0c5730 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.275+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c042630 con 0x7f51ac0c5730 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.275+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac1579a0 con 0x7f51ac0c5730 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.276+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac074990 con 0x7f51ac0c5730 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.276+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f519c02f050 con 0x7f51ac0c5730 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.276+0000 7f51937fe640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f519806b810 0x7f519806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.277+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f519c0abec0 con 0x7f51ac0c5730 2024-02-19T14:53:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.277+0000 7f51aa7fc640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f519806b810 0x7f519806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:13.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.277+0000 7f51aa7fc640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f519806b810 0x7f519806dcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0abf20 tx=0x7f51a0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:13.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.281+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f519c076700 con 0x7f51ac0c5730 2024-02-19T14:53:13.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.488+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51ac075150 con 0x7f51ac0c5730 2024-02-19T14:53:13.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.489+0000 7f51937fe640 1 -- 172.21.15.78:0/863966510 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f519c031e60 con 0x7f51ac0c5730 2024-02-19T14:53:13.493 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:13.493 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:11.566882+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:13.493 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 18 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f519806b810 msgr2=0x7f519806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f519806b810 0x7f519806dcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0abf20 tx=0x7f51a0008040 comp rx=0 tx=0).stop 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 msgr2=0x7f51ac1ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 0x7f51ac1ee030 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f519c000c00 tx=0x7f519c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 shutdown_connections 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f519806b810 0x7f519806dcd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac152700 0x7f51ac148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51ac0c8e00 0x7f51ac1ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 --2- 172.21.15.78:0/863966510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ac0c5730 0x7f51ac1ee030 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 >> 172.21.15.78:0/863966510 conn(0x7f51ac0cac90 msgr2=0x7f51ac151240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 shutdown_connections 2024-02-19T14:53:13.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:13.491+0000 7f51b169d640 1 -- 172.21.15.78:0/863966510 wait complete. 2024-02-19T14:53:13.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:01.188620+0000', 'modified': '2024-02-19T14:53:11.566882+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4766}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4766': {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:13.907 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4766:up:rejoin 2024-02-19T14:53:14.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.015+0000 7f7f09ba1640 1 Processor -- start 2024-02-19T14:53:15.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.015+0000 7f7f09ba1640 1 -- start start 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f09ba1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f040c67e0 0x7f7f0406c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f09ba1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f040c7df0 con 0x7f7f04155e00 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f09ba1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f040c77c0 con 0x7f7f0406d460 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f09ba1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f040c4310 con 0x7f7f040c67e0 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f02ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f040c67e0 0x7f7f0406c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f037fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7f04155e00 0x7f7f040c34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46626/0 (socket says 172.21.15.78:46626) 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f037fe640 1 -- 172.21.15.78:0/2127687246 learned_addr learned my addr 172.21.15.78:0/2127687246 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f02ffd640 1 --2- 172.21.15.78:0/2127687246 >> v2:172.21.15.78:3300/0 conn(0x7f7f040c67e0 0x7f7f0406c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38376/0 (socket says 172.21.15.78:38376) 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f02ffd640 1 -- 172.21.15.78:0/2127687246 >> v1:172.21.15.150:6789/0 conn(0x7f7f0406d460 legacy=0x7f7f0415f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f02ffd640 1 -- 172.21.15.78:0/2127687246 >> v1:172.21.15.78:6789/0 conn(0x7f7f04155e00 legacy=0x7f7f040c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f027fc640 1 -- 172.21.15.78:0/2127687246 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365917441 0 0) 0x7f7f040c7df0 con 0x7f7f04155e00 2024-02-19T14:53:15.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.016+0000 7f7f02ffd640 1 -- 172.21.15.78:0/2127687246 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040c70d0 con 0x7f7f040c67e0 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.017+0000 7f7f02ffd640 1 --2- 172.21.15.78:0/2127687246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f040c67e0 0x7f7f0406c390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80097a0 tx=0x7f7ef802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a197d992ea3d8eca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.017+0000 7f7f027fc640 1 -- 172.21.15.78:0/2127687246 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8031830 con 0x7f7f040c67e0 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.017+0000 7f7f027fc640 1 -- 172.21.15.78:0/2127687246 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef8031990 con 0x7f7f040c67e0 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.017+0000 7f7f027fc640 1 -- 172.21.15.78:0/2127687246 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8031d20 con 0x7f7f040c67e0 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 -- 172.21.15.78:0/2127687246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f040c67e0 msgr2=0x7f7f0406c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/2127687246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f040c67e0 0x7f7f0406c390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80097a0 tx=0x7f7ef802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 -- 172.21.15.78:0/2127687246 shutdown_connections 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/2127687246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f040c67e0 0x7f7f0406c390 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 -- 172.21.15.78:0/2127687246 >> 172.21.15.78:0/2127687246 conn(0x7f7f0406b700 msgr2=0x7f7f040aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 -- 172.21.15.78:0/2127687246 shutdown_connections 2024-02-19T14:53:15.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.018+0000 7f7f09ba1640 1 -- 172.21.15.78:0/2127687246 wait complete. 2024-02-19T14:53:15.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.019+0000 7f7f09ba1640 1 Processor -- start 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.019+0000 7f7f09ba1640 1 -- start start 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 0x7f7f0409b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f040c67e0 0x7f7f04158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f04155e00 0x7f7f04145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f04158270 con 0x7f7f0406d460 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f041ee130 con 0x7f7f040c67e0 2024-02-19T14:53:15.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f041ee3f0 con 0x7f7f04155e00 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 0x7f7f0409b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 0x7f7f0409b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38386/0 (socket says 172.21.15.78:38386) 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 -- 172.21.15.78:0/1096206636 learned_addr learned my addr 172.21.15.78:0/1096206636 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f03fff640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f04155e00 0x7f7f04145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f02ffd640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f040c67e0 0x7f7f04158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 -- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f04155e00 msgr2=0x7f7f04145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f04155e00 0x7f7f04145720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 -- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f040c67e0 msgr2=0x7f7f04158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f040c67e0 0x7f7f04158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 -- 172.21.15.78:0/1096206636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8009400 con 0x7f7f0406d460 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f037fe640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 0x7f7f0409b060 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400c4a0 tx=0x7f7ef400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40074e0 con 0x7f7f0406d460 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef4007640 con 0x7f7f0406d460 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f041ee6d0 con 0x7f7f0406d460 2024-02-19T14:53:15.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4004a00 con 0x7f7f0406d460 2024-02-19T14:53:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.020+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f041eec90 con 0x7f7f0406d460 2024-02-19T14:53:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.021+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f041eee80 con 0x7f7f0406d460 2024-02-19T14:53:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.022+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7ef4004b60 con 0x7f7f0406d460 2024-02-19T14:53:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.022+0000 7f7eebfff640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef006b810 0x7f7ef006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.022+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7ef40897a0 con 0x7f7f0406d460 2024-02-19T14:53:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.022+0000 7f7f02ffd640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef006b810 0x7f7ef006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.023+0000 7f7f02ffd640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef006b810 0x7f7ef006dcd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8036000 tx=0x7f7ef803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.028+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7ef4054120 con 0x7f7f0406d460 2024-02-19T14:53:15.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.233+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f040c6cb0 con 0x7f7f0406d460 2024-02-19T14:53:15.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.234+0000 7f7eebfff640 1 -- 172.21.15.78:0/1096206636 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f7ef40577e0 con 0x7f7f0406d460 2024-02-19T14:53:15.238 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:15.238 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:12.570875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:15.238 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 19 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef006b810 msgr2=0x7f7ef006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef006b810 0x7f7ef006dcd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8036000 tx=0x7f7ef803c040 comp rx=0 tx=0).stop 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 msgr2=0x7f7f0409b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 0x7f7f0409b060 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400c4a0 tx=0x7f7ef400c960 comp rx=0 tx=0).stop 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 shutdown_connections 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ef006b810 0x7f7ef006dcd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f04155e00 0x7f7f04145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f040c67e0 0x7f7f04158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 --2- 172.21.15.78:0/1096206636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f0406d460 0x7f7f0409b060 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 >> 172.21.15.78:0/1096206636 conn(0x7f7f0406b700 msgr2=0x7f7f0406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 shutdown_connections 2024-02-19T14:53:15.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.236+0000 7f7f09ba1640 1 -- 172.21.15.78:0/1096206636 wait complete. 2024-02-19T14:53:15.261 INFO:tasks.ceph.mds.d.smithi150.stderr:2024-02-19T14:53:15.258+0000 7efcf809b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:15.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:01.188620+0000', 'modified': '2024-02-19T14:53:12.570875+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4766}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4766': {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:15.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:53:15.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:53:15.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.752+0000 7f5e4fcd4640 1 Processor -- start 2024-02-19T14:53:15.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.752+0000 7f5e4fcd4640 1 -- start start 2024-02-19T14:53:15.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4fcd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4fcd4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e480c6af0 con 0x7f5e48155f90 2024-02-19T14:53:15.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4fcd4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e480c7940 con 0x7f5e481588a0 2024-02-19T14:53:15.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4fcd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480c7270 con 0x7f5e4806d550 2024-02-19T14:53:15.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4da70640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5e48155f90 0x7f5e480c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46634/0 (socket says 172.21.15.78:46634) 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4da70640 1 -- 172.21.15.78:0/734044985 learned_addr learned my addr 172.21.15.78:0/734044985 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/734044985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832308777 0 0) 0x7f5e480c6af0 con 0x7f5e48155f90 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e34003520 con 0x7f5e48155f90 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359882673 0 0) 0x7f5e480c7940 con 0x7f5e481588a0 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e480c6af0 con 0x7f5e481588a0 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4d26f640 1 -- 172.21.15.78:0/734044985 >> v1:172.21.15.150:6789/0 conn(0x7f5e481588a0 legacy=0x7f5e48158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4d26f640 1 -- 172.21.15.78:0/734044985 >> v1:172.21.15.78:6789/0 conn(0x7f5e48155f90 legacy=0x7f5e480c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.753+0000 7f5e4d26f640 1 -- 172.21.15.78:0/734044985 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e480c7c20 con 0x7f5e4806d550 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.754+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/734044985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480c6d30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e380097d0 tx=0x7f5e3802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bca08750abb0109 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.754+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38031800 con 0x7f5e4806d550 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.754+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e38031960 con 0x7f5e4806d550 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.754+0000 7f5e4ca6e640 1 -- 172.21.15.78:0/734044985 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38031cf0 con 0x7f5e4806d550 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/734044985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 msgr2=0x7f5e480c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/734044985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480c6d30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e380097d0 tx=0x7f5e3802b8a0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/734044985 shutdown_connections 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/734044985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480c6d30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/734044985 >> 172.21.15.78:0/734044985 conn(0x7f5e4806b700 msgr2=0x7f5e480aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/734044985 shutdown_connections 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.755+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/734044985 wait complete. 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.756+0000 7f5e4fcd4640 1 Processor -- start 2024-02-19T14:53:15.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.756+0000 7f5e4fcd4640 1 -- start start 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4fcd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4fcd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 0x7f5e4806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4fcd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e481588a0 0x7f5e480a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4fcd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480a9b20 con 0x7f5e48155f90 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4fcd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4809cb30 con 0x7f5e481588a0 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4fcd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e481eb190 con 0x7f5e4806d550 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 0x7f5e4806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 0x7f5e4806cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38396/0 (socket says 172.21.15.78:38396) 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 -- 172.21.15.78:0/1805003202 learned_addr learned my addr 172.21.15.78:0/1805003202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4e271640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e481588a0 0x7f5e480a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 -- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 msgr2=0x7f5e480a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 -- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e481588a0 msgr2=0x7f5e480a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e481588a0 0x7f5e480a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 -- 172.21.15.78:0/1805003202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e38009400 con 0x7f5e48155f90 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4e271640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e481588a0 0x7f5e480a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.757+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 0x7f5e4806cba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5e380098f0 tx=0x7f5e380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.758+0000 7f5e3e7fc640 1 -- 172.21.15.78:0/1805003202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3803bc70 con 0x7f5e48155f90 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.758+0000 7f5e3e7fc640 1 -- 172.21.15.78:0/1805003202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3803be10 con 0x7f5e48155f90 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.758+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4809bc90 con 0x7f5e48155f90 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.758+0000 7f5e3e7fc640 1 -- 172.21.15.78:0/1805003202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38045630 con 0x7f5e48155f90 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.758+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e481eae20 con 0x7f5e48155f90 2024-02-19T14:53:15.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.759+0000 7f5e3e7fc640 1 -- 172.21.15.78:0/1805003202 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5e3804e450 con 0x7f5e48155f90 2024-02-19T14:53:15.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.759+0000 7f5e3e7fc640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e2c06b840 0x7f5e2c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.759+0000 7f5e3e7fc640 1 -- 172.21.15.78:0/1805003202 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5e3802e070 con 0x7f5e48155f90 2024-02-19T14:53:15.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.760+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e2c06b840 0x7f5e2c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.760+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e2c06b840 0x7f5e2c06dd00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5e44000f60 tx=0x7f5e44007d70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.760+0000 7f5e52991740 1 Processor -- start 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.760+0000 7f5e52991740 1 -- start start 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e52991740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 0x55b68296d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b682932ae0 0x55b68296cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296f8b0 0x55b682931860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e52991740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6828eccd0 con 0x55b68296c740 2024-02-19T14:53:15.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68289ff20 con 0x55b682932ae0 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b68289d450 con 0x55b68296f8b0 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4da70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 0x55b68296d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4da70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 0x55b68296d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38398/0 (socket says 172.21.15.78:38398) 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4da70640 1 -- 172.21.15.78:0/2968448566 learned_addr learned my addr 172.21.15.78:0/2968448566 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4e271640 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296f8b0 0x55b682931860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b682932ae0 0x55b68296cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4da70640 1 -- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296f8b0 msgr2=0x55b682931860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4da70640 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296f8b0 0x55b682931860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.761+0000 7f5e4da70640 1 -- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b682932ae0 msgr2=0x55b68296cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e4da70640 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b682932ae0 0x55b68296cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e4da70640 1 -- 172.21.15.78:0/2968448566 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b68284df20 con 0x55b68296c740 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e4da70640 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 0x55b68296d8b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001bb0 tx=0x7f5e44012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8c38bd57920f3f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e2b7fe640 1 -- 172.21.15.78:0/2968448566 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44006240 con 0x55b68296c740 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e2b7fe640 1 -- 172.21.15.78:0/2968448566 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e440063a0 con 0x55b68296c740 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e2b7fe640 1 -- 172.21.15.78:0/2968448566 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44006710 con 0x55b68296c740 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 -- 172.21.15.78:0/2968448566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 msgr2=0x55b68296d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 0x55b68296d8b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001bb0 tx=0x7f5e44012000 comp rx=0 tx=0).stop 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 -- 172.21.15.78:0/2968448566 shutdown_connections 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296f8b0 0x55b682931860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b682932ae0 0x55b68296cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 --2- 172.21.15.78:0/2968448566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296c740 0x55b68296d8b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.762+0000 7f5e52991740 1 -- 172.21.15.78:0/2968448566 >> 172.21.15.78:0/2968448566 conn(0x55b68292e120 msgr2=0x55b682952b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.763+0000 7f5e52991740 1 -- 172.21.15.78:0/2968448566 shutdown_connections 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.763+0000 7f5e52991740 1 -- 172.21.15.78:0/2968448566 wait complete. 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.763+0000 7f5e52991740 1 -- start start 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.763+0000 7f5e52991740 1 Processor -- start 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68294efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682980d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b6829680b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b682898c30 con 0x55b68294f5c0 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68283dd30 con 0x55b68296f8b0 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b68256c310 con 0x55b68296c740 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e52991740 10 client.0 _ll_register_callbacks cb 0x55b682845d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b6829680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4da70640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68294efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b6829680b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38402/0 (socket says 172.21.15.78:38402) 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 -- 172.21.15.78:0/90471778 learned_addr learned my addr 172.21.15.78:0/90471778 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682980d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 -- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 msgr2=0x55b68294efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68294efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 -- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 msgr2=0x55b682980d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682980d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4e271640 1 -- 172.21.15.78:0/90471778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e34003520 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.764+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682980d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e4e271640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b6829680b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5e400129d0 tx=0x7f5e40012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f017da139b80cc8c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e28ff9640 1 -- 172.21.15.78:0/90471778 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4001a030 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e28ff9640 1 -- 172.21.15.78:0/90471778 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5e4001b070 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6825986c0 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e28ff9640 1 -- 172.21.15.78:0/90471778 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e40006e20 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 10 client.5484 fetch_fsmap learned FSMap version 19 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55b6828972b0 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e28ff9640 1 -- 172.21.15.78:0/90471778 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40025020 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e28ff9640 1 -- 172.21.15.78:0/90471778 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5e40003970 con 0x55b68294f5c0 2024-02-19T14:53:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 10 client.5484 fetch_fsmap finished waiting for FSMap version 19 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 10 client.5484 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 20 client.5484 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b682969fc0 0x55b68296a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 4 client.5484 mds_command: new command op to 4766 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 --> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6825e9270 con 0x55b682969fc0 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.765+0000 7f5e4da70640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b682969fc0 0x55b68296a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.766+0000 7f5e28ff9640 10 client.5484 ms_handle_connect on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.766+0000 7f5e4da70640 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b682969fc0 0x55b68296a3a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1de8f382da2c802 server_cookie=e4fe64185ec4f3f6 in_seq=0 out_seq=0 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.766+0000 7f5e28ff9640 1 -- 172.21.15.78:0/90471778 <== mds.0 v2:172.21.15.150:6833/4259811019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b6825e9270 con 0x55b682969fc0 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.766+0000 7f5e28ff9640 10 client.5484 handle_command_reply: tid=0 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.766+0000 7f5e52991740 1 client.5484 shutdown 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.766+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 msgr2=0x55b6829680b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b6829680b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5e400129d0 tx=0x7f5e40012dd0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 shutdown_connections 2024-02-19T14:53:15.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b682969fc0 0x55b68296a3a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b6829680b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e28ff9640 0 client.5484 ms_handle_reset on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682980d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 --2- 172.21.15.78:0/90471778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68294efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 >> 172.21.15.78:0/90471778 conn(0x55b682932ae0 msgr2=0x55b682952ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 shutdown_connections 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 1 -- 172.21.15.78:0/90471778 wait complete. 2024-02-19T14:53:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.767+0000 7f5e52991740 20 client.5484 trim_cache size 0 max 16384 2024-02-19T14:53:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.783+0000 7f5e52991740 1 Processor -- start 2024-02-19T14:53:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.783+0000 7f5e52991740 1 -- start start 2024-02-19T14:53:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e52991740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b68292af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68292b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682967630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e52991740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68281e490 con 0x55b68294f5c0 2024-02-19T14:53:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6829648a0 con 0x55b68296f8b0 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b682963e90 con 0x55b68296c740 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b68292af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b68292af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38416/0 (socket says 172.21.15.78:38416) 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 -- 172.21.15.78:0/2501388144 learned_addr learned my addr 172.21.15.78:0/2501388144 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68292b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4e271640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682967630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 -- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 msgr2=0x55b68292b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68292b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 -- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 msgr2=0x55b682967630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682967630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 -- 172.21.15.78:0/2501388144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b682833a20 con 0x55b68294f5c0 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68292b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4e271640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682967630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e4da70640 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b68292af40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5e44024040 tx=0x7f5e4401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c00ee013213139 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e2affd640 1 -- 172.21.15.78:0/2501388144 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44032070 con 0x55b68294f5c0 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.784+0000 7f5e2affd640 1 -- 172.21.15.78:0/2501388144 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44006730 con 0x55b68294f5c0 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e2affd640 1 -- 172.21.15.78:0/2501388144 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44006240 con 0x55b68294f5c0 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 -- 172.21.15.78:0/2501388144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 msgr2=0x55b68292af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b68292af40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5e44024040 tx=0x7f5e4401e000 comp rx=0 tx=0).stop 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 -- 172.21.15.78:0/2501388144 shutdown_connections 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68296f8b0 0x55b682967630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68292b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 --2- 172.21.15.78:0/2501388144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68294f5c0 0x55b68292af40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 -- 172.21.15.78:0/2501388144 >> 172.21.15.78:0/2501388144 conn(0x55b682932ae0 msgr2=0x55b68296de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 -- 172.21.15.78:0/2501388144 shutdown_connections 2024-02-19T14:53:15.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.785+0000 7f5e52991740 1 -- 172.21.15.78:0/2501388144 wait complete. 2024-02-19T14:53:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 -- start start 2024-02-19T14:53:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 Processor -- start 2024-02-19T14:53:15.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68294f5c0 0x55b682971060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68297a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 0x55b682990410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68294dd00 con 0x55b68296f8b0 2024-02-19T14:53:15.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b682990d60 con 0x55b68294f5c0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.786+0000 7f5e52991740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b68292dc60 con 0x55b68296c740 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e52991740 10 client.0 _ll_register_callbacks cb 0x7f5e40001da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 0x55b682990410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 0x55b682990410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38422/0 (socket says 172.21.15.78:38422) 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 -- 172.21.15.78:0/1988361442 learned_addr learned my addr 172.21.15.78:0/1988361442 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68294f5c0 0x55b682971060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4d26f640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68297a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 -- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 msgr2=0x55b68297a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68297a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 -- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68294f5c0 msgr2=0x55b682971060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68294f5c0 0x55b682971060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 -- 172.21.15.78:0/1988361442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b68284df20 con 0x55b68296f8b0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e4e271640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 0x55b682990410 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e400129d0 tx=0x7f5e400014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af83c0b6a7b5d97a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e28ff9640 1 -- 172.21.15.78:0/1988361442 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4001a040 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e28ff9640 1 -- 172.21.15.78:0/1988361442 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5e40026080 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e440063a0 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e28ff9640 1 -- 172.21.15.78:0/1988361442 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e40005280 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e52991740 10 client.5490 fetch_fsmap learned FSMap version 19 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55b682859780 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.787+0000 7f5e28ff9640 1 -- 172.21.15.78:0/1988361442 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e440063a0 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e28ff9640 1 -- 172.21.15.78:0/1988361442 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5e400053e0 con 0x55b68296f8b0 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 10 client.5490 fetch_fsmap finished waiting for FSMap version 19 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 10 client.5490 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 20 client.5490 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b6829dd970 0x55b6829e05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 4 client.5490 mds_command: new command op to 4766 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:53:15.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 --> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b6825d5610 con 0x55b6829dd970 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b6829dd970 0x55b6829e05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e28ff9640 10 client.5490 ms_handle_connect on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e4da70640 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b6829dd970 0x55b6829e05c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da93f5526551920b server_cookie=950493016a8ea755 in_seq=0 out_seq=0 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e28ff9640 1 -- 172.21.15.78:0/1988361442 <== mds.0 v2:172.21.15.150:6833/4259811019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b6825d5610 con 0x55b6829dd970 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e28ff9640 10 client.5490 handle_command_reply: tid=0 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 client.5490 shutdown 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 msgr2=0x55b682990410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 0x55b682990410 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e400129d0 tx=0x7f5e400014d0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 shutdown_connections 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55b6829dd970 0x55b6829e05c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68296f8b0 0x55b682990410 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b68296c740 0x55b68297a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 --2- 172.21.15.78:0/1988361442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b68294f5c0 0x55b682971060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 >> 172.21.15.78:0/1988361442 conn(0x55b682932ae0 msgr2=0x55b6829816f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.788+0000 7f5e28ff9640 0 client.5490 ms_handle_reset on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 shutdown_connections 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e52991740 1 -- 172.21.15.78:0/1988361442 wait complete. 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e52991740 20 client.5490 trim_cache size 0 max 16384 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4766, 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:53:15.799 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 5.2277415319999996, 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 19, 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 39, 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 39, 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 21.383377694 2024-02-19T14:53:15.800 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e2c06b840 msgr2=0x7f5e2c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e2c06b840 0x7f5e2c06dd00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5e44000f60 tx=0x7f5e44007d70 comp rx=0 tx=0).stop 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 msgr2=0x7f5e4806cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 0x7f5e4806cba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5e380098f0 tx=0x7f5e380046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 shutdown_connections 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e2c06b840 0x7f5e2c06dd00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e48155f90 0x7f5e4806cba0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e481588a0 0x7f5e480a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 --2- 172.21.15.78:0/1805003202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e4806d550 0x7f5e480a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 >> 172.21.15.78:0/1805003202 conn(0x7f5e4806b700 msgr2=0x7f5e48155270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 shutdown_connections 2024-02-19T14:53:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.789+0000 7f5e4fcd4640 1 -- 172.21.15.78:0/1805003202 wait complete. 2024-02-19T14:53:15.803 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:53:01.188620+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4766": { "addr": "172.21.15.150:6835/4259811019", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 4259811019, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 4259811019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4766, "incarnation": 16, "join_fscid": 2, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-19T14:53:12.570875+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4766 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/3578832669", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3578832669, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3578832669, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5020, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6835/1762664317", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1762664317, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 1762664317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5244, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/1556668834", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 1556668834, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 1556668834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 17, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5457, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:53:15.804 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 2, 'name': 'd', 'state': 'up:active'}] 2024-02-19T14:53:15.804 INFO:tasks.cephfs.test_failover:current = [{'gid': 5244, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/1762664317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1762664317}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 1762664317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5020, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3578832669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3578832669}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3578832669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5457, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/1556668834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 1556668834}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1556668834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 17}, {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:53:15.804 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:53:15.804 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:53:15.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.904+0000 7fa15495e640 1 Processor -- start 2024-02-19T14:53:15.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.904+0000 7fa15495e640 1 -- start start 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa15495e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa1500ad890 0x7fa1500c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa15495e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1500ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa15495e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa150152d30 con 0x7fa1501534d0 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa15495e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1500c7860 con 0x7fa1500ad890 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa15495e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1500c7ce0 con 0x7fa1500c49c0 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa1500ad890 0x7fa1500c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa147fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1500ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14ed9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa1501534d0 0x7fa1500c5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53244/0 (socket says 172.21.15.78:53244) 2024-02-19T14:53:15.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14ed9d640 1 -- 172.21.15.78:0/1282102515 learned_addr learned my addr 172.21.15.78:0/1282102515 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:15.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14e59c640 1 -- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 msgr2=0x7fa1500ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:15.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14e59c640 1 --2- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1500ab600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14e59c640 1 -- 172.21.15.78:0/1282102515 >> v1:172.21.15.150:6789/0 conn(0x7fa1501534d0 legacy=0x7fa1500c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14e59c640 1 -- 172.21.15.78:0/1282102515 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1500c7f80 con 0x7fa1500ad890 2024-02-19T14:53:15.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.905+0000 7fa14dd9b640 1 -- 172.21.15.78:0/1282102515 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335776682 0 0) 0x7fa150152d30 con 0x7fa1501534d0 2024-02-19T14:53:15.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.906+0000 7fa14e59c640 1 --2- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1500c64f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa1400097a0 tx=0x7fa14002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f7985e4a8b769d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.906+0000 7fa14dd9b640 1 -- 172.21.15.78:0/1282102515 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14002f050 con 0x7fa1500ad890 2024-02-19T14:53:15.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.906+0000 7fa14dd9b640 1 -- 172.21.15.78:0/1282102515 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140031700 con 0x7fa1500ad890 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.906+0000 7fa14dd9b640 1 -- 172.21.15.78:0/1282102515 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140031a90 con 0x7fa1500ad890 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 -- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 msgr2=0x7fa1500c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 --2- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1500c64f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa1400097a0 tx=0x7fa14002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 -- 172.21.15.78:0/1282102515 shutdown_connections 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 --2- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1500ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 --2- 172.21.15.78:0/1282102515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1500c64f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 -- 172.21.15.78:0/1282102515 >> 172.21.15.78:0/1282102515 conn(0x7fa15006b700 msgr2=0x7fa1500aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 -- 172.21.15.78:0/1282102515 shutdown_connections 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.907+0000 7fa15495e640 1 -- 172.21.15.78:0/1282102515 wait complete. 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.908+0000 7fa15495e640 1 Processor -- start 2024-02-19T14:53:15.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.908+0000 7fa15495e640 1 -- start start 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1501dec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1501dde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 0x7fa15014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1501e9e90 con 0x7fa1501534d0 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa150156070 con 0x7fa1500ad890 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1501e9210 con 0x7fa1500c49c0 2024-02-19T14:53:15.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 0x7fa15014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 0x7fa15014ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38436/0 (socket says 172.21.15.78:38436) 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 -- 172.21.15.78:0/859005258 learned_addr learned my addr 172.21.15.78:0/859005258 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14e59c640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1501dec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa147fff640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1501dde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 -- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 msgr2=0x7fa1501dde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1501dde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 -- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 msgr2=0x7fa1501dec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1501dec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 -- 172.21.15.78:0/859005258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140009400 con 0x7fa1501534d0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14e59c640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1501dec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa14ed9d640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 0x7fa15014ca60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa14800caf0 tx=0x7fa1480021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14800ac00 con 0x7fa1501534d0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14800ad60 con 0x7fa1501534d0 2024-02-19T14:53:15.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1501e94f0 con 0x7fa1501534d0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148012640 con 0x7fa1501534d0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.909+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1501e9ab0 con 0x7fa1501534d0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.910+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1501e9fd0 con 0x7fa1501534d0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.911+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa1480127a0 con 0x7fa1501534d0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.911+0000 7fa146ffd640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa13006b810 0x7fa13006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.911+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa14808a710 con 0x7fa1501534d0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.911+0000 7fa14e59c640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa13006b810 0x7fa13006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:15.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.912+0000 7fa14e59c640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa13006b810 0x7fa13006dcd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa140035000 tx=0x7fa140008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:15.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:15.916+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa1501e9fd0 con 0x7fa1501534d0 2024-02-19T14:53:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.087+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fa1501e9fd0 con 0x7fa1501534d0 2024-02-19T14:53:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.090+0000 7fa146ffd640 1 -- 172.21.15.78:0/859005258 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7fa1501e9fd0 con 0x7fa1501534d0 2024-02-19T14:53:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa13006b810 msgr2=0x7fa13006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa13006b810 0x7fa13006dcd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa140035000 tx=0x7fa140008030 comp rx=0 tx=0).stop 2024-02-19T14:53:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 msgr2=0x7fa15014ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 0x7fa15014ca60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa14800caf0 tx=0x7fa1480021c0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 shutdown_connections 2024-02-19T14:53:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa13006b810 0x7fa13006dcd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1501534d0 0x7fa15014ca60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1500c49c0 0x7fa1501dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 --2- 172.21.15.78:0/859005258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1500ad890 0x7fa1501dec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 >> 172.21.15.78:0/859005258 conn(0x7fa15006b700 msgr2=0x7fa15006ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 shutdown_connections 2024-02-19T14:53:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.092+0000 7fa15495e640 1 -- 172.21.15.78:0/859005258 wait complete. 2024-02-19T14:53:16.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.202+0000 7f6c4bf8c640 1 Processor -- start 2024-02-19T14:53:16.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.202+0000 7f6c4bf8c640 1 -- start start 2024-02-19T14:53:16.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4bf8c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6c440718f0 0x7f6c44071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4bf8c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c44159130 con 0x7f6c44155070 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4bf8c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c440ad030 con 0x7f6c440acc50 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4bf8c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c440c2cf0 con 0x7f6c440718f0 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4a529640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6c440718f0 0x7f6c44071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4a529640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6c440718f0 0x7f6c44071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38444/0 (socket says 172.21.15.78:38444) 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c49527640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6c44155070 0x7f6c44155500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46644/0 (socket says 172.21.15.78:46644) 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4a529640 1 -- 172.21.15.78:0/904921231 learned_addr learned my addr 172.21.15.78:0/904921231 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4a529640 1 -- 172.21.15.78:0/904921231 >> v1:172.21.15.150:6789/0 conn(0x7f6c440acc50 legacy=0x7f6c440c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4a529640 1 -- 172.21.15.78:0/904921231 >> v1:172.21.15.78:6789/0 conn(0x7f6c44155070 legacy=0x7f6c44155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.203+0000 7f6c4a529640 1 -- 172.21.15.78:0/904921231 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c441595c0 con 0x7f6c440718f0 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c4a529640 1 --2- 172.21.15.78:0/904921231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440718f0 0x7f6c44071d60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009e50 tx=0x7f6c3c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931b0d075d7efd1b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c48d26640 1 -- 172.21.15.78:0/904921231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c02cb20 con 0x7f6c440718f0 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c48d26640 1 -- 172.21.15.78:0/904921231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3c02cc80 con 0x7f6c440718f0 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c48d26640 1 -- 172.21.15.78:0/904921231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c0343f0 con 0x7f6c440718f0 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/904921231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440718f0 msgr2=0x7f6c44071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/904921231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440718f0 0x7f6c44071d60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009e50 tx=0x7f6c3c02bf80 comp rx=0 tx=0).stop 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/904921231 shutdown_connections 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/904921231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440718f0 0x7f6c44071d60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.204+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/904921231 >> 172.21.15.78:0/904921231 conn(0x7f6c440c7800 msgr2=0x7f6c440c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.205+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/904921231 shutdown_connections 2024-02-19T14:53:16.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.205+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/904921231 wait complete. 2024-02-19T14:53:16.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 Processor -- start 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 -- start start 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c440718f0 0x7f6c441deda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 0x7f6c44138410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c44155070 0x7f6c44138950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440c2030 con 0x7f6c440acc50 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44157760 con 0x7f6c44155070 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c4bf8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440a89b0 con 0x7f6c440718f0 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c49527640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 0x7f6c44138410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c49527640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 0x7f6c44138410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38458/0 (socket says 172.21.15.78:38458) 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.206+0000 7f6c49527640 1 -- 172.21.15.78:0/1572831154 learned_addr learned my addr 172.21.15.78:0/1572831154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c4a529640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c44155070 0x7f6c44138950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49d28640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c440718f0 0x7f6c441deda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49527640 1 -- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c440718f0 msgr2=0x7f6c441deda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49527640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c440718f0 0x7f6c441deda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49527640 1 -- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c44155070 msgr2=0x7f6c44138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49527640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c44155070 0x7f6c44138950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49527640 1 -- 172.21.15.78:0/1572831154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c009b30 con 0x7f6c440acc50 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c49527640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 0x7f6c44138410 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400c410 tx=0x7f6c3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340074e0 con 0x7f6c440acc50 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c34007640 con 0x7f6c440acc50 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c44138e90 con 0x7f6c440acc50 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340049e0 con 0x7f6c440acc50 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.207+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c44157110 con 0x7f6c440acc50 2024-02-19T14:53:16.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.208+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4406c820 con 0x7f6c440acc50 2024-02-19T14:53:16.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.209+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c34004b40 con 0x7f6c440acc50 2024-02-19T14:53:16.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.209+0000 7f6c3a7fc640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2806b810 0x7f6c2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.209+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6c34089ba0 con 0x7f6c440acc50 2024-02-19T14:53:16.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.209+0000 7f6c49d28640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2806b810 0x7f6c2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.209+0000 7f6c49d28640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2806b810 0x7f6c2806dcd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c440c1d70 tx=0x7f6c40009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.213+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c34054520 con 0x7f6c440acc50 2024-02-19T14:53:16.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.420+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c441578d0 con 0x7f6c440acc50 2024-02-19T14:53:16.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.421+0000 7f6c3a7fc640 1 -- 172.21.15.78:0/1572831154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f6c34057be0 con 0x7f6c440acc50 2024-02-19T14:53:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:16.423 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:12.570875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:16.424 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 19 2024-02-19T14:53:16.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2806b810 msgr2=0x7f6c2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2806b810 0x7f6c2806dcd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c440c1d70 tx=0x7f6c40009040 comp rx=0 tx=0).stop 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 msgr2=0x7f6c44138410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 0x7f6c44138410 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400c410 tx=0x7f6c3400c8d0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 shutdown_connections 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2806b810 0x7f6c2806dcd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c440acc50 0x7f6c44138410 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c44155070 0x7f6c44138950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 --2- 172.21.15.78:0/1572831154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c440718f0 0x7f6c441deda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 >> 172.21.15.78:0/1572831154 conn(0x7f6c440c7800 msgr2=0x7f6c44153c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 shutdown_connections 2024-02-19T14:53:16.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.423+0000 7f6c4bf8c640 1 -- 172.21.15.78:0/1572831154 wait complete. 2024-02-19T14:53:16.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:01.188620+0000', 'modified': '2024-02-19T14:53:12.570875+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4766}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4766': {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:16.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:53:16.838 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:53:16.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.940+0000 7fe18768f640 1 Processor -- start 2024-02-19T14:53:16.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.940+0000 7fe18768f640 1 -- start start 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe18768f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe1800c78a0 0x7fe1800c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe18768f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe180072ee0 0x7fe18014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe18768f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800c35c0 con 0x7fe18014ef50 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe18768f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18014f920 con 0x7fe1800c78a0 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe18768f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800c38a0 con 0x7fe180072ee0 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe184c2a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe18014ef50 0x7fe1800727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46646/0 (socket says 172.21.15.78:46646) 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe184c2a640 1 -- 172.21.15.78:0/4130117556 learned_addr learned my addr 172.21.15.78:0/4130117556 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe185c2c640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe180072ee0 0x7fe18014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.941+0000 7fe18542b640 1 --2- 172.21.15.78:0/4130117556 >> v2:172.21.15.150:3300/0 conn(0x7fe1800c78a0 0x7fe1800c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe17ffff640 1 -- 172.21.15.78:0/4130117556 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676626144 0 0) 0x7fe1800c35c0 con 0x7fe18014ef50 2024-02-19T14:53:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe17ffff640 1 -- 172.21.15.78:0/4130117556 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe168003520 con 0x7fe18014ef50 2024-02-19T14:53:16.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe18542b640 1 -- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe180072ee0 msgr2=0x7fe18014f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe18542b640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe180072ee0 0x7fe18014f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe18542b640 1 -- 172.21.15.78:0/4130117556 >> v1:172.21.15.78:6789/0 conn(0x7fe18014ef50 legacy=0x7fe1800727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe18542b640 1 -- 172.21.15.78:0/4130117556 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800c3b80 con 0x7fe1800c78a0 2024-02-19T14:53:16.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.942+0000 7fe185c2c640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe180072ee0 0x7fe18014f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:16.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18542b640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800c44b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe1740097a0 tx=0x7fe17402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73be3bd54d9d4b2f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe17ffff640 1 -- 172.21.15.78:0/4130117556 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174031830 con 0x7fe1800c78a0 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe17ffff640 1 -- 172.21.15.78:0/4130117556 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe174031990 con 0x7fe1800c78a0 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe17ffff640 1 -- 172.21.15.78:0/4130117556 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174031d20 con 0x7fe1800c78a0 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 -- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 msgr2=0x7fe1800c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800c44b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe1740097a0 tx=0x7fe17402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 -- 172.21.15.78:0/4130117556 shutdown_connections 2024-02-19T14:53:16.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe180072ee0 0x7fe18014f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 --2- 172.21.15.78:0/4130117556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800c44b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 -- 172.21.15.78:0/4130117556 >> 172.21.15.78:0/4130117556 conn(0x7fe180066e70 msgr2=0x7fe1800a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 -- 172.21.15.78:0/4130117556 shutdown_connections 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.943+0000 7fe18768f640 1 -- 172.21.15.78:0/4130117556 wait complete. 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.944+0000 7fe18768f640 1 Processor -- start 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.944+0000 7fe18768f640 1 -- start start 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18768f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 0x7fe18014a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18768f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800be5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18768f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe18014ef50 0x7fe180149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18768f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18014fb40 con 0x7fe180072ee0 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18768f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1801e8c30 con 0x7fe1800c78a0 2024-02-19T14:53:16.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18768f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1801e9160 con 0x7fe18014ef50 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18542b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 0x7fe18014a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18542b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 0x7fe18014a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38462/0 (socket says 172.21.15.78:38462) 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe18542b640 1 -- 172.21.15.78:0/581245129 learned_addr learned my addr 172.21.15.78:0/581245129 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe184c2a640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800be5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.945+0000 7fe185c2c640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe18014ef50 0x7fe180149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18542b640 1 -- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe18014ef50 msgr2=0x7fe180149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18542b640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe18014ef50 0x7fe180149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18542b640 1 -- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 msgr2=0x7fe1800be5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18542b640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800be5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18542b640 1 -- 172.21.15.78:0/581245129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178009400 con 0x7fe180072ee0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18542b640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 0x7fe18014a2f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe174009890 tx=0x7fe1740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe17dffb640 1 -- 172.21.15.78:0/581245129 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17403a830 con 0x7fe180072ee0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe17dffb640 1 -- 172.21.15.78:0/581245129 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17403a9b0 con 0x7fe180072ee0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe174009400 con 0x7fe180072ee0 2024-02-19T14:53:16.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe17dffb640 1 -- 172.21.15.78:0/581245129 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17403ad60 con 0x7fe180072ee0 2024-02-19T14:53:16.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.946+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1801e8e80 con 0x7fe180072ee0 2024-02-19T14:53:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.948+0000 7fe17dffb640 1 -- 172.21.15.78:0/581245129 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe17404d440 con 0x7fe180072ee0 2024-02-19T14:53:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.948+0000 7fe17dffb640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16c06b810 0x7fe16c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.948+0000 7fe17dffb640 1 -- 172.21.15.78:0/581245129 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe1740ac400 con 0x7fe180072ee0 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.948+0000 7fe184c2a640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16c06b810 0x7fe16c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.948+0000 7fe184c2a640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16c06b810 0x7fe16c06dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1800688e0 tx=0x7fe170009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.949+0000 7fe18a272740 1 Processor -- start 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.949+0000 7fe18a272740 1 -- start start 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18a272740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 0x55c09b311720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b311c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18a272740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b290cd0 con 0x55c09b2d1450 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b243f20 con 0x55c09b2d6ae0 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c09b241450 con 0x55c09b2d5c20 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 0x55c09b311720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 0x55c09b311720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38478/0 (socket says 172.21.15.78:38478) 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 -- 172.21.15.78:0/564843197 learned_addr learned my addr 172.21.15.78:0/564843197 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe184c2a640 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b311c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe185c2c640 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 -- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 msgr2=0x55c09b2cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 -- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 msgr2=0x55c09b311c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b311c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 -- 172.21.15.78:0/564843197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c09b1f1f20 con 0x55c09b2d1450 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe184c2a640 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b311c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe18542b640 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 0x55c09b311720 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe174082b90 tx=0x7fe17407a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46e0798d5b4f807 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe166ffd640 1 -- 172.21.15.78:0/564843197 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17407eb00 con 0x55c09b2d1450 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe166ffd640 1 -- 172.21.15.78:0/564843197 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1740830c0 con 0x55c09b2d1450 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.950+0000 7fe166ffd640 1 -- 172.21.15.78:0/564843197 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174077020 con 0x55c09b2d1450 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 -- 172.21.15.78:0/564843197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 msgr2=0x55c09b311720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 0x55c09b311720 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe174082b90 tx=0x7fe17407a3d0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 -- 172.21.15.78:0/564843197 shutdown_connections 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b311c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 --2- 172.21.15.78:0/564843197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d1450 0x55c09b311720 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 -- 172.21.15.78:0/564843197 >> 172.21.15.78:0/564843197 conn(0x55c09b2d2120 msgr2=0x55c09b2f6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 -- 172.21.15.78:0/564843197 shutdown_connections 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.951+0000 7fe18a272740 1 -- 172.21.15.78:0/564843197 wait complete. 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 -- start start 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 Processor -- start 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 0x55c09b2f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b30bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b30c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b23cc30 con 0x55c09b2d5c20 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b1e1d30 con 0x55c09b2d6ae0 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.952+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c09af10310 con 0x55c09b329c70 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18a272740 10 client.0 _ll_register_callbacks cb 0x55c09b1e9d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 0x55c09b2f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 0x55c09b2f97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38492/0 (socket says 172.21.15.78:38492) 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 -- 172.21.15.78:0/1526374647 learned_addr learned my addr 172.21.15.78:0/1526374647 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe185c2c640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b30c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe184c2a640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b30bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 -- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 msgr2=0x55c09b30c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b30c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 -- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 msgr2=0x55c09b30bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b30bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 -- 172.21.15.78:0/1526374647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1801e8e80 con 0x55c09b2d5c20 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18542b640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 0x55c09b2f97b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe17407a6b0 tx=0x7fe174006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502accb1131f16fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe15ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe15ffff640 1 -- 172.21.15.78:0/1526374647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17407eb00 con 0x55c09b2d5c20 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe15ffff640 1 -- 172.21.15.78:0/1526374647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe1740ac0b0 con 0x55c09b2d5c20 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c09af3c6c0 con 0x55c09b2d5c20 2024-02-19T14:53:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe15ffff640 1 -- 172.21.15.78:0/1526374647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1740ac270 con 0x55c09b2d5c20 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18a272740 10 client.5514 fetch_fsmap learned FSMap version 19 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55c09b23b2b0 con 0x55c09b2d5c20 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe15ffff640 1 -- 172.21.15.78:0/1526374647 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c09af3c6c0 con 0x55c09b2d5c20 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.953+0000 7fe15ffff640 1 -- 172.21.15.78:0/1526374647 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe17404b060 con 0x55c09b2d5c20 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe18a272740 10 client.5514 fetch_fsmap finished waiting for FSMap version 19 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe18a272740 10 client.5514 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe18a272740 20 client.5514 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe18a272740 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b2f7030 0x55c09b30e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe18a272740 4 client.5514 mds_command: new command op to 4766 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 --> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c09af8d270 con 0x55c09b2f7030 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe184c2a640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b2f7030 0x55c09b30e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe15ffff640 10 client.5514 ms_handle_connect on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.954+0000 7fe184c2a640 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b2f7030 0x55c09b30e0a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=891b182c7f361a8b server_cookie=8cd43f41d1bb6b14 in_seq=0 out_seq=0 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe15ffff640 1 -- 172.21.15.78:0/1526374647 <== mds.0 v2:172.21.15.150:6833/4259811019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c09af8d270 con 0x55c09b2f7030 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe15ffff640 10 client.5514 handle_command_reply: tid=0 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 client.5514 shutdown 2024-02-19T14:53:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 msgr2=0x55c09b2f97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 0x55c09b2f97b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe17407a6b0 tx=0x7fe174006f40 comp rx=0 tx=0).stop 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 shutdown_connections 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b2f7030 0x55c09b30e0a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b30c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b30bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe15ffff640 0 client.5514 ms_handle_reset on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 --2- 172.21.15.78:0/1526374647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d5c20 0x55c09b2f97b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 >> 172.21.15.78:0/1526374647 conn(0x55c09b2d1450 msgr2=0x55c09b313490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 shutdown_connections 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 1 -- 172.21.15.78:0/1526374647 wait complete. 2024-02-19T14:53:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.955+0000 7fe18a272740 20 client.5514 trim_cache size 0 max 16384 2024-02-19T14:53:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 Processor -- start 2024-02-19T14:53:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 -- start start 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2d50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b380620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 0x55c09b380b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b1c2490 con 0x55c09b329c70 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b317e80 con 0x55c09b2d6ae0 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.972+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c09b2cf5b0 con 0x55c09b2d5c20 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 0x55c09b380b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 0x55c09b380b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38508/0 (socket says 172.21.15.78:38508) 2024-02-19T14:53:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 -- 172.21.15.78:0/870585041 learned_addr learned my addr 172.21.15.78:0/870585041 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe184c2a640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b380620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe18542b640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2d50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 -- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 msgr2=0x55c09b2d50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2d50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 -- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 msgr2=0x55c09b380620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b380620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 -- 172.21.15.78:0/870585041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c09b1d7a20 con 0x55c09b329c70 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe184c2a640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b380620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe18542b640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2d50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe185c2c640 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 0x55c09b380b60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe178009eb0 tx=0x7fe178015420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4050ad33dff4b3f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe1667fc640 1 -- 172.21.15.78:0/870585041 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17801a070 con 0x55c09b329c70 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe1667fc640 1 -- 172.21.15.78:0/870585041 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178015dc0 con 0x55c09b329c70 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.973+0000 7fe1667fc640 1 -- 172.21.15.78:0/870585041 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17801e410 con 0x55c09b329c70 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 -- 172.21.15.78:0/870585041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 msgr2=0x55c09b380b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 0x55c09b380b60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe178009eb0 tx=0x7fe178015420 comp rx=0 tx=0).stop 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 -- 172.21.15.78:0/870585041 shutdown_connections 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b329c70 0x55c09b380b60 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d6ae0 0x55c09b380620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 --2- 172.21.15.78:0/870585041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b2d5c20 0x55c09b2d50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 -- 172.21.15.78:0/870585041 >> 172.21.15.78:0/870585041 conn(0x55c09b2d1450 msgr2=0x55c09b2f8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 -- 172.21.15.78:0/870585041 shutdown_connections 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 -- 172.21.15.78:0/870585041 wait complete. 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 -- start start 2024-02-19T14:53:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.974+0000 7fe18a272740 1 Processor -- start 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d5c20 0x55c09b30e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 0x55c09b389680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b383230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b32ce40 con 0x55c09b2d6ae0 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c09b38a3a0 con 0x55c09b2d5c20 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c09b311740 con 0x55c09b329c70 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18a272740 10 client.0 _ll_register_callbacks cb 0x7fe174091070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 0x55c09b389680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 0x55c09b389680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38514/0 (socket says 172.21.15.78:38514) 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 -- 172.21.15.78:0/1915062721 learned_addr learned my addr 172.21.15.78:0/1915062721 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe18542b640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d5c20 0x55c09b30e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe185c2c640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b383230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 -- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 msgr2=0x55c09b383230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b383230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 -- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d5c20 msgr2=0x55c09b30e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d5c20 0x55c09b30e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.975+0000 7fe184c2a640 1 -- 172.21.15.78:0/1915062721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe178009b30 con 0x55c09b2d6ae0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe184c2a640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 0x55c09b389680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x55c09af10420 tx=0x7fe170000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee90a612fb059d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe15ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe15ffff640 1 -- 172.21.15.78:0/1915062721 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170001ee0 con 0x55c09b2d6ae0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe15ffff640 1 -- 172.21.15.78:0/1915062721 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe170015020 con 0x55c09b2d6ae0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1740830c0 con 0x55c09b2d6ae0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe15ffff640 1 -- 172.21.15.78:0/1915062721 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1700151e0 con 0x55c09b2d6ae0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 10 client.5520 fetch_fsmap learned FSMap version 19 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55c09b1fd780 con 0x55c09b2d6ae0 2024-02-19T14:53:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe15ffff640 1 -- 172.21.15.78:0/1915062721 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1740830c0 con 0x55c09b2d6ae0 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe15ffff640 1 -- 172.21.15.78:0/1915062721 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe170012070 con 0x55c09b2d6ae0 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 10 client.5520 fetch_fsmap finished waiting for FSMap version 19 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 10 client.5520 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 20 client.5520 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b383a40 0x55c09b2cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 4 client.5520 mds_command: new command op to 4766 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 --> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c09af79610 con 0x55c09b383a40 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.976+0000 7fe18542b640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b383a40 0x55c09b2cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18542b640 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b383a40 0x55c09b2cfc90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c58001a097d8f4f8 server_cookie=1a57c48e86f4195d in_seq=0 out_seq=0 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe15ffff640 10 client.5520 ms_handle_connect on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe15ffff640 1 -- 172.21.15.78:0/1915062721 <== mds.0 v2:172.21.15.150:6833/4259811019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55c09af79610 con 0x55c09b383a40 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe15ffff640 10 client.5520 handle_command_reply: tid=0 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 client.5520 shutdown 2024-02-19T14:53:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 msgr2=0x55c09b389680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 0x55c09b389680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x55c09af10420 tx=0x7fe170000f40 comp rx=0 tx=0).stop 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 shutdown_connections 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55c09b383a40 0x55c09b2cfc90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c09b2d6ae0 0x55c09b389680 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c09b329c70 0x55c09b383230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe15ffff640 0 client.5520 ms_handle_reset on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 --2- 172.21.15.78:0/1915062721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c09b2d5c20 0x55c09b30e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 >> 172.21.15.78:0/1915062721 conn(0x55c09b2d1450 msgr2=0x55c09b2d50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 shutdown_connections 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 1 -- 172.21.15.78:0/1915062721 wait complete. 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.977+0000 7fe18a272740 20 client.5520 trim_cache size 0 max 16384 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:16.986 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4766, 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 6.4153772370000004, 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 19, 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 39, 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 39, 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 22.571017887 2024-02-19T14:53:16.987 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:16.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16c06b810 msgr2=0x7fe16c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16c06b810 0x7fe16c06dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1800688e0 tx=0x7fe170009040 comp rx=0 tx=0).stop 2024-02-19T14:53:16.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 msgr2=0x7fe18014a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:16.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 0x7fe18014a2f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe174009890 tx=0x7fe1740046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 shutdown_connections 2024-02-19T14:53:16.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16c06b810 0x7fe16c06dcd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe18014ef50 0x7fe180149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1800c78a0 0x7fe1800be5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 --2- 172.21.15.78:0/581245129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe180072ee0 0x7fe18014a2f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:16.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 >> 172.21.15.78:0/581245129 conn(0x7fe180066e70 msgr2=0x7fe180150720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:16.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 shutdown_connections 2024-02-19T14:53:16.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:16.978+0000 7fe18768f640 1 -- 172.21.15.78:0/581245129 wait complete. 2024-02-19T14:53:16.991 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:53:01.188620+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4766": { "addr": "172.21.15.150:6835/4259811019", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 4259811019, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 4259811019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4766, "incarnation": 16, "join_fscid": 2, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-19T14:53:12.570875+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4766 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/3578832669", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3578832669, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3578832669, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5020, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6835/1762664317", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1762664317, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 1762664317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5244, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/1556668834", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 1556668834, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 1556668834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 17, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5457, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:53:16.991 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-02-19T14:53:16.992 INFO:tasks.cephfs.test_failover:current = [{'gid': 5244, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/1762664317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1762664317}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 1762664317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5020, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3578832669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3578832669}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3578832669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5457, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/1556668834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 1556668834}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1556668834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 17}, {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:53:16.992 DEBUG:tasks.cephfs.test_failover:-1 != 2 2024-02-19T14:53:16.992 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:53:21.993 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:22.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.101+0000 7f5c447f6640 1 Processor -- start 2024-02-19T14:53:22.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.101+0000 7f5c447f6640 1 -- start start 2024-02-19T14:53:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c447f6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5c3c1525c0 0x7f5c3c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c447f6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c3c0c6f80 0x7f5c3c155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c447f6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c0c6460 con 0x7f5c3c06d560 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c447f6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3c0c1f50 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c447f6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3c1593a0 con 0x7f5c3c1525c0 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c42d93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c3c0c6f80 0x7f5c3c155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c42d93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c3c0c6f80 0x7f5c3c155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40262/0 (socket says 172.21.15.78:40262) 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c42d93640 1 -- 172.21.15.78:0/2174708865 learned_addr learned my addr 172.21.15.78:0/2174708865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.102+0000 7f5c42592640 1 --2- 172.21.15.78:0/2174708865 >> v2:172.21.15.150:3300/0 conn(0x7f5c3c1525c0 0x7f5c3c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.103+0000 7f5c42d93640 1 -- 172.21.15.78:0/2174708865 >> v2:172.21.15.150:3300/0 conn(0x7f5c3c1525c0 msgr2=0x7f5c3c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.103+0000 7f5c42d93640 1 --2- 172.21.15.78:0/2174708865 >> v2:172.21.15.150:3300/0 conn(0x7f5c3c1525c0 0x7f5c3c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.103+0000 7f5c42d93640 1 -- 172.21.15.78:0/2174708865 >> v1:172.21.15.150:6789/0 conn(0x7f5c3c06d560 legacy=0x7f5c3c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.103+0000 7f5c42d93640 1 -- 172.21.15.78:0/2174708865 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c158720 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c42d93640 1 --2- 172.21.15.78:0/2174708865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c155070 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c08bd00 tx=0x7f5c3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58421185525ce6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c41590640 1 -- 172.21.15.78:0/2174708865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34031870 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c41590640 1 -- 172.21.15.78:0/2174708865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c340319d0 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c41590640 1 -- 172.21.15.78:0/2174708865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34031d60 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 -- 172.21.15.78:0/2174708865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 msgr2=0x7f5c3c155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 --2- 172.21.15.78:0/2174708865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c155070 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c08bd00 tx=0x7f5c3402b910 comp rx=0 tx=0).stop 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 -- 172.21.15.78:0/2174708865 shutdown_connections 2024-02-19T14:53:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 --2- 172.21.15.78:0/2174708865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c155070 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 --2- 172.21.15.78:0/2174708865 >> v2:172.21.15.150:3300/0 conn(0x7f5c3c1525c0 0x7f5c3c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 -- 172.21.15.78:0/2174708865 >> 172.21.15.78:0/2174708865 conn(0x7f5c3c0cab20 msgr2=0x7f5c3c0caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 -- 172.21.15.78:0/2174708865 shutdown_connections 2024-02-19T14:53:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.104+0000 7f5c447f6640 1 -- 172.21.15.78:0/2174708865 wait complete. 2024-02-19T14:53:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.105+0000 7f5c447f6640 1 Processor -- start 2024-02-19T14:53:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.105+0000 7f5c447f6640 1 -- start start 2024-02-19T14:53:22.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c447f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c3c06d560 0x7f5c3c0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c447f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c14ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c447f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3c1525c0 0x7f5c3c14b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c447f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c158540 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c447f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c1ee730 con 0x7f5c3c1525c0 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c447f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c1eec50 con 0x7f5c3c06d560 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c41d91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c14ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c41d91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c14ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40270/0 (socket says 172.21.15.78:40270) 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c41d91640 1 -- 172.21.15.78:0/230874627 learned_addr learned my addr 172.21.15.78:0/230874627 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c42592640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c3c06d560 0x7f5c3c0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c42d93640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3c1525c0 0x7f5c3c14b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c41d91640 1 -- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c3c06d560 msgr2=0x7f5c3c0c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.106+0000 7f5c41d91640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c3c06d560 0x7f5c3c0c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c41d91640 1 -- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3c1525c0 msgr2=0x7f5c3c14b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c41d91640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3c1525c0 0x7f5c3c14b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c41d91640 1 -- 172.21.15.78:0/230874627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c340094d0 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c41d91640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c14ace0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c400 tx=0x7f5c3000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c300052d0 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c30005450 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c155fa0 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30007260 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.107+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c14ba20 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.108+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c074990 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.108+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5c300073c0 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.109+0000 7f5c3affd640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c2806b810 0x7f5c2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.109+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5c30089690 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.109+0000 7f5c42592640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c2806b810 0x7f5c2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.109+0000 7f5c42592640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c2806b810 0x7f5c2806dcd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c14bfa0 tx=0x7f5c2c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.113+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5c30054010 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.318+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c3c153e30 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.319+0000 7f5c3affd640 1 -- 172.21.15.78:0/230874627 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f5c300576d0 con 0x7f5c3c0c6f80 2024-02-19T14:53:22.322 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:22.322 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:16.569892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 20 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c2806b810 msgr2=0x7f5c2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c2806b810 0x7f5c2806dcd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c14bfa0 tx=0x7f5c2c00b040 comp rx=0 tx=0).stop 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 msgr2=0x7f5c3c14ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c14ace0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c400 tx=0x7f5c3000c8c0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 shutdown_connections 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c2806b810 0x7f5c2806dcd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c0c6f80 0x7f5c3c14ace0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3c1525c0 0x7f5c3c14b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 --2- 172.21.15.78:0/230874627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c3c06d560 0x7f5c3c0c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 >> 172.21.15.78:0/230874627 conn(0x7f5c3c0cab20 msgr2=0x7f5c3c0c6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 shutdown_connections 2024-02-19T14:53:22.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.321+0000 7f5c447f6640 1 -- 172.21.15.78:0/230874627 wait complete. 2024-02-19T14:53:22.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:01.188620+0000', 'modified': '2024-02-19T14:53:16.569892+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4766}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4766': {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:22.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:53:22.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:53:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 Processor -- start 2024-02-19T14:53:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 -- start start 2024-02-19T14:53:22.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47f40713a0 0x7f47f4071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47f40c29e0 0x7f47f40c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f40c69f0 con 0x7f47f40c8b60 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f40c32c0 con 0x7f47f40c29e0 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.837+0000 7f47f89b6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f4070500 con 0x7f47f40713a0 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47f40c29e0 0x7f47f40c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47f40c29e0 0x7f47f40c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40280/0 (socket says 172.21.15.78:40280) 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3102232944 learned_addr learned my addr 172.21.15.78:0/3102232944 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f259c640 1 --2- 172.21.15.78:0/3102232944 >> v2:172.21.15.150:3300/0 conn(0x7f47f40713a0 0x7f47f4071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3102232944 >> v2:172.21.15.150:3300/0 conn(0x7f47f40713a0 msgr2=0x7f47f4071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/3102232944 >> v2:172.21.15.150:3300/0 conn(0x7f47f40713a0 0x7f47f4071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3102232944 >> v1:172.21.15.150:6789/0 conn(0x7f47f40c8b60 legacy=0x7f47f41536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.838+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3102232944 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f406fe40 con 0x7f47f40c29e0 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/3102232944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f40c64b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f47f40c7850 tx=0x7f47e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb3964fd1d72177 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f159a640 1 -- 172.21.15.78:0/3102232944 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8031870 con 0x7f47f40c29e0 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f159a640 1 -- 172.21.15.78:0/3102232944 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e80319d0 con 0x7f47f40c29e0 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f159a640 1 -- 172.21.15.78:0/3102232944 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8031d60 con 0x7f47f40c29e0 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f89b6640 1 -- 172.21.15.78:0/3102232944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 msgr2=0x7f47f40c64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f89b6640 1 --2- 172.21.15.78:0/3102232944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f40c64b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f47f40c7850 tx=0x7f47e802b910 comp rx=0 tx=0).stop 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f89b6640 1 -- 172.21.15.78:0/3102232944 shutdown_connections 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f89b6640 1 --2- 172.21.15.78:0/3102232944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f40c64b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f89b6640 1 --2- 172.21.15.78:0/3102232944 >> v2:172.21.15.150:3300/0 conn(0x7f47f40713a0 0x7f47f4071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.839+0000 7f47f89b6640 1 -- 172.21.15.78:0/3102232944 >> 172.21.15.78:0/3102232944 conn(0x7f47f40aa300 msgr2=0x7f47f40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.840+0000 7f47f89b6640 1 -- 172.21.15.78:0/3102232944 shutdown_connections 2024-02-19T14:53:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.840+0000 7f47f89b6640 1 -- 172.21.15.78:0/3102232944 wait complete. 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 Processor -- start 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 -- start start 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47f40713a0 0x7f47f414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f414a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47f40c8b60 0x7f47f414adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f4157270 con 0x7f47f40c29e0 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40c2650 con 0x7f47f40c8b60 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.841+0000 7f47f89b6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f414b2f0 con 0x7f47f40713a0 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f414a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f414a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40286/0 (socket says 172.21.15.78:40286) 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 -- 172.21.15.78:0/4050002161 learned_addr learned my addr 172.21.15.78:0/4050002161 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f259c640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47f40713a0 0x7f47f414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47f40c8b60 0x7f47f414adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 -- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47f40713a0 msgr2=0x7f47f414e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47f40713a0 0x7f47f414e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 -- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47f40c8b60 msgr2=0x7f47f414adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47f40c8b60 0x7f47f414adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 -- 172.21.15.78:0/4050002161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e80094d0 con 0x7f47f40c29e0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f414a870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f47e8000c00 tx=0x7f47e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47e2ffd640 1 -- 172.21.15.78:0/4050002161 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e803abe0 con 0x7f47f40c29e0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47e2ffd640 1 -- 172.21.15.78:0/4050002161 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e803ad60 con 0x7f47f40c29e0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f4156cd0 con 0x7f47f40c29e0 2024-02-19T14:53:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47e2ffd640 1 -- 172.21.15.78:0/4050002161 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8044630 con 0x7f47f40c29e0 2024-02-19T14:53:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.842+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f40be110 con 0x7f47f40c29e0 2024-02-19T14:53:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.844+0000 7f47e2ffd640 1 -- 172.21.15.78:0/4050002161 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f47e8044790 con 0x7f47f40c29e0 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.844+0000 7f47e2ffd640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f47d406b840 0x7f47d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.844+0000 7f47e2ffd640 1 -- 172.21.15.78:0/4050002161 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f47e802d040 con 0x7f47f40c29e0 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.844+0000 7f47f259c640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f47d406b840 0x7f47d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.844+0000 7f47f259c640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f47d406b840 0x7f47d406dd00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0113e0 tx=0x7f47ec00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.845+0000 7f47fb589740 1 Processor -- start 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.845+0000 7f47fb589740 1 -- start start 2024-02-19T14:53:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47fb589740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 0x55842e435720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3faae0 0x55842e435c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3f9c20 0x55842e3f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47fb589740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e3b4cd0 con 0x55842e3f5450 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e367f20 con 0x55842e3faae0 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55842e365450 con 0x55842e3f9c20 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 0x55842e435720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 0x55842e435720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40288/0 (socket says 172.21.15.78:40288) 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 -- 172.21.15.78:0/3372281907 learned_addr learned my addr 172.21.15.78:0/3372281907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3faae0 0x55842e435c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3f9c20 0x55842e3f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 -- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3f9c20 msgr2=0x55842e3f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3f9c20 0x55842e3f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 -- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3faae0 msgr2=0x55842e435c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3faae0 0x55842e435c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 -- 172.21.15.78:0/3372281907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55842e315f20 con 0x55842e3f5450 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3f9c20 0x55842e3f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47f259c640 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 0x55842e435720 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f47ec003d70 tx=0x7f47ec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5d5d2f65afaeb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47d3fff640 1 -- 172.21.15.78:0/3372281907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec01c040 con 0x55842e3f5450 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47d3fff640 1 -- 172.21.15.78:0/3372281907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec00f040 con 0x55842e3f5450 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.846+0000 7f47d3fff640 1 -- 172.21.15.78:0/3372281907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00e040 con 0x55842e3f5450 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 -- 172.21.15.78:0/3372281907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 msgr2=0x55842e435720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 0x55842e435720 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f47ec003d70 tx=0x7f47ec014000 comp rx=0 tx=0).stop 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 -- 172.21.15.78:0/3372281907 shutdown_connections 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3f9c20 0x55842e3f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3faae0 0x55842e435c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 --2- 172.21.15.78:0/3372281907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3f5450 0x55842e435720 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 -- 172.21.15.78:0/3372281907 >> 172.21.15.78:0/3372281907 conn(0x55842e3f6120 msgr2=0x55842e41ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 -- 172.21.15.78:0/3372281907 shutdown_connections 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.847+0000 7f47fb589740 1 -- 172.21.15.78:0/3372281907 wait complete. 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.848+0000 7f47fb589740 1 -- start start 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.848+0000 7f47fb589740 1 Processor -- start 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e41dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e42fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e41aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e360c30 con 0x55842e3faae0 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e305d30 con 0x55842e3f9c20 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55842e034310 con 0x55842e43a2a0 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 10 client.0 _ll_register_callbacks cb 0x55842e30dd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e42fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e41dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e42fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40290/0 (socket says 172.21.15.78:40290) 2024-02-19T14:53:22.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f2d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e41aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 -- 172.21.15.78:0/2954772582 learned_addr learned my addr 172.21.15.78:0/2954772582 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 -- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 msgr2=0x55842e41aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e41aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 -- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 msgr2=0x55842e41dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e41dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 -- 172.21.15.78:0/2954772582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec003a50 con 0x55842e3faae0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e42fc10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f47e802b950 tx=0x7f47e807af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c140d2c2089988cc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47d17fa640 1 -- 172.21.15.78:0/2954772582 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8083060 con 0x55842e3faae0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47d17fa640 1 -- 172.21.15.78:0/2954772582 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f47e807e070 con 0x55842e3faae0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.849+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55842e0606c0 con 0x55842e3faae0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47d17fa640 1 -- 172.21.15.78:0/2954772582 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e80831c0 con 0x55842e3faae0 2024-02-19T14:53:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 10 client.5538 fetch_fsmap learned FSMap version 20 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55842e35f2b0 con 0x55842e3faae0 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47d17fa640 1 -- 172.21.15.78:0/2954772582 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e8096020 con 0x55842e3faae0 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47d17fa640 1 -- 172.21.15.78:0/2954772582 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 20) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f47e8092b70 con 0x55842e3faae0 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 10 client.5538 fetch_fsmap finished waiting for FSMap version 20 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 10 client.5538 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 20 client.5538 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e44d260 0x55842e44d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 4 client.5538 mds_command: new command op to 4766 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 --> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55842e0b1270 con 0x55842e44d260 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47f259c640 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e44d260 0x55842e44d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47f259c640 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e44d260 0x55842e44d640 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b75aa3fee221adfa server_cookie=c29651a9ab40af95 in_seq=0 out_seq=0 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.850+0000 7f47d17fa640 10 client.5538 ms_handle_connect on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47d17fa640 1 -- 172.21.15.78:0/2954772582 <== mds.0 v2:172.21.15.150:6833/4259811019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55842e0b1270 con 0x55842e44d260 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47d17fa640 10 client.5538 handle_command_reply: tid=0 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 client.5538 shutdown 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 msgr2=0x55842e42fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e42fc10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f47e802b950 tx=0x7f47e807af50 comp rx=0 tx=0).stop 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 shutdown_connections 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e44d260 0x55842e44d640 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e42fc10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e41aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47d17fa640 0 client.5538 ms_handle_reset on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 --2- 172.21.15.78:0/2954772582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e41dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 >> 172.21.15.78:0/2954772582 conn(0x55842e3f5450 msgr2=0x55842e438db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 shutdown_connections 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 1 -- 172.21.15.78:0/2954772582 wait complete. 2024-02-19T14:53:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.851+0000 7f47fb589740 20 client.5538 trim_cache size 0 max 16384 2024-02-19T14:53:22.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.868+0000 7f47fb589740 1 Processor -- start 2024-02-19T14:53:22.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.868+0000 7f47fb589740 1 -- start start 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e453860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3faae0 0x55842e438db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47fb589740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 0x55842e4392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47fb589740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e2e6490 con 0x55842e43a2a0 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e44a1e0 con 0x55842e3f9c20 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55842e314cc0 con 0x55842e3faae0 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 0x55842e4392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 0x55842e4392f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40292/0 (socket says 172.21.15.78:40292) 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 -- 172.21.15.78:0/1745343613 learned_addr learned my addr 172.21.15.78:0/1745343613 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f259c640 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e453860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3faae0 0x55842e438db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 -- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3faae0 msgr2=0x55842e438db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3faae0 0x55842e438db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 -- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 msgr2=0x55842e453860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e453860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 -- 172.21.15.78:0/1745343613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55842e2fba20 con 0x55842e43a2a0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f259c640 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e453860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 0x55842e4392f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f47e4005390 tx=0x7f47e4007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc7cd800dea9b4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47d37fe640 1 -- 172.21.15.78:0/1745343613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4007240 con 0x55842e43a2a0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47d37fe640 1 -- 172.21.15.78:0/1745343613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e40073a0 con 0x55842e43a2a0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.869+0000 7f47d37fe640 1 -- 172.21.15.78:0/1745343613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4007530 con 0x55842e43a2a0 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 -- 172.21.15.78:0/1745343613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 msgr2=0x55842e4392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 0x55842e4392f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f47e4005390 tx=0x7f47e4007db0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 -- 172.21.15.78:0/1745343613 shutdown_connections 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e43a2a0 0x55842e4392f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e3faae0 0x55842e438db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 --2- 172.21.15.78:0/1745343613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e453860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 -- 172.21.15.78:0/1745343613 >> 172.21.15.78:0/1745343613 conn(0x55842e3f5450 msgr2=0x55842e3ef7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 -- 172.21.15.78:0/1745343613 shutdown_connections 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 -- 172.21.15.78:0/1745343613 wait complete. 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 -- start start 2024-02-19T14:53:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.870+0000 7f47fb589740 1 Processor -- start 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e4198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e4aa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e4a7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e4a9f30 con 0x55842e3faae0 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55842e45d7d0 con 0x55842e3f9c20 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55842e4ada80 con 0x55842e43a2a0 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47fb589740 10 client.0 _ll_register_callbacks cb 0x7f47e80091e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e4aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e4aa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40294/0 (socket says 172.21.15.78:40294) 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3142700326 learned_addr learned my addr 172.21.15.78:0/3142700326 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f2d9d640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e4a7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f259c640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e4198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 msgr2=0x55842e4a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e4a7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 msgr2=0x55842e4198c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.871+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e4198c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47f1d9b640 1 -- 172.21.15.78:0/3142700326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f40c69f0 con 0x55842e3faae0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47f1d9b640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e4aa900 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f47e80aabe0 tx=0x7f47e80aaa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead56917967b2a4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47d17fa640 1 -- 172.21.15.78:0/3142700326 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e807a4e0 con 0x55842e3faae0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47d17fa640 1 -- 172.21.15.78:0/3142700326 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f47e8097080 con 0x55842e3faae0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47ec00f040 con 0x55842e3faae0 2024-02-19T14:53:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47d17fa640 1 -- 172.21.15.78:0/3142700326 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e807a640 con 0x55842e3faae0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 10 client.5544 fetch_fsmap learned FSMap version 20 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55842e321780 con 0x55842e3faae0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47d17fa640 1 -- 172.21.15.78:0/3142700326 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec00f040 con 0x55842e3faae0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47d17fa640 1 -- 172.21.15.78:0/3142700326 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 20) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f47e80a4240 con 0x55842e3faae0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 10 client.5544 fetch_fsmap finished waiting for FSMap version 20 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 10 client.5544 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 20 client.5544 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e437fd0 0x55842e4a7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 4 client.5544 mds_command: new command op to 4766 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 --> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55842e09d610 con 0x55842e437fd0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.872+0000 7f47f259c640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e437fd0 0x55842e4a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47d17fa640 10 client.5544 ms_handle_connect on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47f259c640 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e437fd0 0x55842e4a7670 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10dad2a5d90c28ed server_cookie=1711f1da59f311fa in_seq=0 out_seq=0 2024-02-19T14:53:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47d17fa640 1 -- 172.21.15.78:0/3142700326 <== mds.0 v2:172.21.15.150:6833/4259811019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55842e09d610 con 0x55842e437fd0 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47d17fa640 10 client.5544 handle_command_reply: tid=0 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 client.5544 shutdown 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 msgr2=0x55842e4aa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e4aa900 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f47e80aabe0 tx=0x7f47e80aaa60 comp rx=0 tx=0).stop 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 shutdown_connections 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:6833/4259811019,v1:172.21.15.150:6835/4259811019] conn(0x55842e437fd0 0x55842e4a7670 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55842e3faae0 0x55842e4aa900 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55842e43a2a0 0x55842e4a7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 --2- 172.21.15.78:0/3142700326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55842e3f9c20 0x55842e4198c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47d17fa640 0 client.5544 ms_handle_reset on v2:172.21.15.150:6833/4259811019 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 >> 172.21.15.78:0/3142700326 conn(0x55842e3f5450 msgr2=0x55842e447500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 shutdown_connections 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 1 -- 172.21.15.78:0/3142700326 wait complete. 2024-02-19T14:53:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.873+0000 7f47fb589740 20 client.5544 trim_cache size 0 max 16384 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4766, 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 12.311386346000001, 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 20, 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 39, 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 39, 2024-02-19T14:53:22.883 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 28.467024812999998 2024-02-19T14:53:22.884 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f47d406b840 msgr2=0x7f47d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f47d406b840 0x7f47d406dd00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0113e0 tx=0x7f47ec00b040 comp rx=0 tx=0).stop 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 msgr2=0x7f47f414a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f414a870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f47e8000c00 tx=0x7f47e80046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 shutdown_connections 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f47d406b840 0x7f47d406dd00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47f40c29e0 0x7f47f414a870 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47f40c8b60 0x7f47f414adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 --2- 172.21.15.78:0/4050002161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47f40713a0 0x7f47f414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 >> 172.21.15.78:0/4050002161 conn(0x7f47f40aa300 msgr2=0x7f47f40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 shutdown_connections 2024-02-19T14:53:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:22.874+0000 7f47f89b6640 1 -- 172.21.15.78:0/4050002161 wait complete. 2024-02-19T14:53:22.887 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:53:01.188620+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4766": { "addr": "172.21.15.150:6835/4259811019", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 4259811019, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 4259811019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4766, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-19T14:53:16.569892+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4766 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/3578832669", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3578832669, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3578832669, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5020, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6835/1762664317", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1762664317, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 1762664317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5244, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/1556668834", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 1556668834, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 1556668834, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 17, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5457, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:53:22.888 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-02-19T14:53:22.888 INFO:tasks.cephfs.test_failover:current = [{'gid': 5244, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/1762664317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1762664317}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 1762664317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5020, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3578832669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3578832669}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3578832669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5457, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/1556668834', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 1556668834}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1556668834}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 17}, {'gid': 4766, 'name': 'd', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.150:6835/4259811019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 4259811019}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 4259811019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:53:22.888 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:53:22.888 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:53:22.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:53:22.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:53:22.964 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:53:22.964 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:53:22.964 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:53:22.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:53:22.984 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:53:22.984 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:53:23.020 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:53:23.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:23.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.122+0000 7fb6de51e640 1 Processor -- start 2024-02-19T14:53:23.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.122+0000 7fb6de51e640 1 -- start start 2024-02-19T14:53:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6de51e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d80c0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6de51e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6d80c06b0 0x7fb6d80c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6de51e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d81529f0 con 0x7fb6d8152610 2024-02-19T14:53:23.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6de51e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d806da70 con 0x7fb6d80c06b0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6de51e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80bf9b0 con 0x7fb6d806e350 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6d80c06b0 0x7fb6d80c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6d80c06b0 0x7fb6d80c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40310/0 (socket says 172.21.15.78:40310) 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 -- 172.21.15.78:0/1329838392 learned_addr learned my addr 172.21.15.78:0/1329838392 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6d77fe640 1 --2- 172.21.15.78:0/1329838392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d80c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 -- 172.21.15.78:0/1329838392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 msgr2=0x7fb6d80c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 --2- 172.21.15.78:0/1329838392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d80c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 -- 172.21.15.78:0/1329838392 >> v1:172.21.15.150:6789/0 conn(0x7fb6d8152610 legacy=0x7fb6d8158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.123+0000 7fb6dcabb640 1 -- 172.21.15.78:0/1329838392 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d806e8c0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6dcabb640 1 --2- 172.21.15.78:0/1329838392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d80c32c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009e50 tx=0x7fb6d002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf3f15fbd0bbc61 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6d6ffd640 1 -- 172.21.15.78:0/1329838392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d002cb00 con 0x7fb6d80c06b0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6d6ffd640 1 -- 172.21.15.78:0/1329838392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d002cc60 con 0x7fb6d80c06b0 2024-02-19T14:53:23.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6d6ffd640 1 -- 172.21.15.78:0/1329838392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d00343f0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 -- 172.21.15.78:0/1329838392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 msgr2=0x7fb6d80c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 --2- 172.21.15.78:0/1329838392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d80c32c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009e50 tx=0x7fb6d002bf60 comp rx=0 tx=0).stop 2024-02-19T14:53:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 -- 172.21.15.78:0/1329838392 shutdown_connections 2024-02-19T14:53:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 --2- 172.21.15.78:0/1329838392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d80c32c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 --2- 172.21.15.78:0/1329838392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d80c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 -- 172.21.15.78:0/1329838392 >> 172.21.15.78:0/1329838392 conn(0x7fb6d806b270 msgr2=0x7fb6d806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 -- 172.21.15.78:0/1329838392 shutdown_connections 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.124+0000 7fb6de51e640 1 -- 172.21.15.78:0/1329838392 wait complete. 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.125+0000 7fb6de51e640 1 Processor -- start 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 -- start start 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d81edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d814c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6d8152610 0x7fb6d814cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d81ee8a0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d81ee5c0 con 0x7fb6d8152610 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6de51e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80719d0 con 0x7fb6d806e350 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6d77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d814c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6d77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d814c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40312/0 (socket says 172.21.15.78:40312) 2024-02-19T14:53:23.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6d77fe640 1 -- 172.21.15.78:0/163231067 learned_addr learned my addr 172.21.15.78:0/163231067 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6d7fff640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d81edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.126+0000 7fb6dcabb640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6d8152610 0x7fb6d814cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d77fe640 1 -- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 msgr2=0x7fb6d81edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d77fe640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d81edca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d77fe640 1 -- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6d8152610 msgr2=0x7fb6d814cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d77fe640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6d8152610 0x7fb6d814cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d77fe640 1 -- 172.21.15.78:0/163231067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0009b30 con 0x7fb6d80c06b0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d77fe640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d814c670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00e9b0 tx=0x7fb6cc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc009670 con 0x7fb6d80c06b0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc00c9a0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d8071cb0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00cd50 con 0x7fb6d80c06b0 2024-02-19T14:53:23.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.127+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d81e7ff0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.128+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d8074990 con 0x7fb6d80c06b0 2024-02-19T14:53:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.129+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb6cc013470 con 0x7fb6d80c06b0 2024-02-19T14:53:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.129+0000 7fb6d4ff9640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b810 0x7fb6c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.129+0000 7fb6d7fff640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b810 0x7fb6c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.129+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb6cc08a980 con 0x7fb6d80c06b0 2024-02-19T14:53:23.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.129+0000 7fb6d7fff640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b810 0x7fb6c406dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80016e0 tx=0x7fb6c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.133+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb6cc0551e0 con 0x7fb6d80c06b0 2024-02-19T14:53:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.339+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6d8071e40 con 0x7fb6d80c06b0 2024-02-19T14:53:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.339+0000 7fb6d4ff9640 1 -- 172.21.15.78:0/163231067 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4945 (secure 0 0 0) 0x7fb6cc021090 con 0x7fb6d80c06b0 2024-02-19T14:53:23.342 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:23.342 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:16.569892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:23.342 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 20 2024-02-19T14:53:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b810 msgr2=0x7fb6c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b810 0x7fb6c406dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80016e0 tx=0x7fb6c8009040 comp rx=0 tx=0).stop 2024-02-19T14:53:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 msgr2=0x7fb6d814c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d814c670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00e9b0 tx=0x7fb6cc00ee70 comp rx=0 tx=0).stop 2024-02-19T14:53:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 shutdown_connections 2024-02-19T14:53:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b810 0x7fb6c406dcd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6d80c06b0 0x7fb6d814c670 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6d8152610 0x7fb6d814cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 --2- 172.21.15.78:0/163231067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6d806e350 0x7fb6d81edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 >> 172.21.15.78:0/163231067 conn(0x7fb6d806b270 msgr2=0x7fb6d8156930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 shutdown_connections 2024-02-19T14:53:23.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.341+0000 7fb6de51e640 1 -- 172.21.15.78:0/163231067 wait complete. 2024-02-19T14:53:23.355 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.452+0000 7f8d9a9f4640 1 Processor -- start 2024-02-19T14:53:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.452+0000 7f8d9a9f4640 1 -- start start 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d9a9f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c3200 0x7f8d940ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d9a9f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d940c35e0 con 0x7f8d940c4350 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d9a9f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d940acad0 con 0x7f8d94153b60 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d9a9f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9414d3c0 con 0x7f8d940c3200 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d98f91640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8d940c4350 0x7f8d9406ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60814/0 (socket says 172.21.15.78:60814) 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d98f91640 1 -- 172.21.15.78:0/1967167722 learned_addr learned my addr 172.21.15.78:0/1967167722 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d93fff640 1 --2- 172.21.15.78:0/1967167722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c3200 0x7f8d940ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984974768 0 0) 0x7f8d940c35e0 con 0x7f8d940c4350 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c003520 con 0x7f8d940c4350 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778241568 0 0) 0x7f8d940acad0 con 0x7f8d94153b60 2024-02-19T14:53:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d940c35e0 con 0x7f8d94153b60 2024-02-19T14:53:23.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d93fff640 1 -- 172.21.15.78:0/1967167722 >> v1:172.21.15.150:6789/0 conn(0x7f8d94153b60 legacy=0x7f8d940718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d93fff640 1 -- 172.21.15.78:0/1967167722 >> v1:172.21.15.78:6789/0 conn(0x7f8d940c4350 legacy=0x7f8d9406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.453+0000 7f8d93fff640 1 -- 172.21.15.78:0/1967167722 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940ad000 con 0x7f8d940c3200 2024-02-19T14:53:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.454+0000 7f8d93fff640 1 --2- 172.21.15.78:0/1967167722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c3200 0x7f8d940ac2e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d840097d0 tx=0x7f8d8402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9fdb05f349b056 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.454+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84031890 con 0x7f8d940c3200 2024-02-19T14:53:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.454+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d840319f0 con 0x7f8d940c3200 2024-02-19T14:53:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.454+0000 7f8d92ffd640 1 -- 172.21.15.78:0/1967167722 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84031d60 con 0x7f8d940c3200 2024-02-19T14:53:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/1967167722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c3200 msgr2=0x7f8d940ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/1967167722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c3200 0x7f8d940ac2e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d840097d0 tx=0x7f8d8402b930 comp rx=0 tx=0).stop 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/1967167722 shutdown_connections 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/1967167722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c3200 0x7f8d940ac2e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/1967167722 >> 172.21.15.78:0/1967167722 conn(0x7f8d940c7800 msgr2=0x7f8d940c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/1967167722 shutdown_connections 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.455+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/1967167722 wait complete. 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.456+0000 7f8d9a9f4640 1 Processor -- start 2024-02-19T14:53:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.456+0000 7f8d9a9f4640 1 -- start start 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.456+0000 7f8d9a9f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d940c3200 0x7f8d941e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c4350 0x7f8d941ec370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 0x7f8d94156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94156cb0 con 0x7f8d94153b60 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94156ee0 con 0x7f8d940c3200 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d941571c0 con 0x7f8d940c4350 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 0x7f8d94156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 0x7f8d94156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40320/0 (socket says 172.21.15.78:40320) 2024-02-19T14:53:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 -- 172.21.15.78:0/272007991 learned_addr learned my addr 172.21.15.78:0/272007991 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d93fff640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d940c3200 0x7f8d941e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d937fe640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c4350 0x7f8d941ec370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 -- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c4350 msgr2=0x7f8d941ec370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c4350 0x7f8d941ec370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 -- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d940c3200 msgr2=0x7f8d941e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d940c3200 0x7f8d941e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 -- 172.21.15.78:0/272007991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d84009400 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d98f91640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 0x7f8d94156770 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00cba0 tx=0x7f8d8c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00ace0 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00ae40 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d941e8420 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c012620 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.457+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d941e89e0 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.458+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d941e8c00 con 0x7f8d94153b60 2024-02-19T14:53:23.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.459+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8d8c007070 con 0x7f8d94153b60 2024-02-19T14:53:23.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.459+0000 7f8d90ff9640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d8006b810 0x7f8d8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.459+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8d8c08a3a0 con 0x7f8d94153b60 2024-02-19T14:53:23.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.459+0000 7f8d93fff640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d8006b810 0x7f8d8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.459+0000 7f8d93fff640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d8006b810 0x7f8d8006dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8d84036000 tx=0x7f8d8403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.463+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8d8c054c90 con 0x7f8d94153b60 2024-02-19T14:53:23.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.635+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d9406c820 con 0x7f8d94153b60 2024-02-19T14:53:23.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.636+0000 7f8d90ff9640 1 -- 172.21.15.78:0/272007991 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+17059 (secure 0 0 0) 0x7f8d8c058350 con 0x7f8d94153b60 2024-02-19T14:53:23.638 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:23.639 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:10.560315+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:52:59.040775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:00.165210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/3231075223":"2024-02-20T14:53:10.560278+0000","172.21.15.78:6836/3231075223":"2024-02-20T14:53:10.560278+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d8006b810 msgr2=0x7f8d8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d8006b810 0x7f8d8006dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8d84036000 tx=0x7f8d8403d040 comp rx=0 tx=0).stop 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 msgr2=0x7f8d94156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 0x7f8d94156770 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00cba0 tx=0x7f8d8c00a430 comp rx=0 tx=0).stop 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 shutdown_connections 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d8006b810 0x7f8d8006dcd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d94153b60 0x7f8d94156770 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d940c4350 0x7f8d941ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 --2- 172.21.15.78:0/272007991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d940c3200 0x7f8d941e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 >> 172.21.15.78:0/272007991 conn(0x7f8d940c7800 msgr2=0x7f8d940718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 shutdown_connections 2024-02-19T14:53:23.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.638+0000 7f8d9a9f4640 1 -- 172.21.15.78:0/272007991 wait complete. 2024-02-19T14:53:23.651 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:52:59.040775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:53:00.165210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:53:23.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:53:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 Processor -- start 2024-02-19T14:53:23.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 -- start start 2024-02-19T14:53:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06d40c8e00 0x7f06d406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06d406d910 0x7f06d406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d406cd20 con 0x7f06d4152700 2024-02-19T14:53:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d41551f0 con 0x7f06d406d910 2024-02-19T14:53:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.752+0000 7f06dae7b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d4155490 con 0x7f06d40c8e00 2024-02-19T14:53:23.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06d406d910 0x7f06d406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06d406d910 0x7f06d406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40336/0 (socket says 172.21.15.78:40336) 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 -- 172.21.15.78:0/771320443 learned_addr learned my addr 172.21.15.78:0/771320443 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d8c17640 1 --1- 172.21.15.78:0/771320443 >> v1:172.21.15.78:6789/0 conn(0x7f06d4152700 0x7f06d40c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60822/0 (socket says 172.21.15.78:60822) 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d3fff640 1 --2- 172.21.15.78:0/771320443 >> v2:172.21.15.150:3300/0 conn(0x7f06d40c8e00 0x7f06d406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 -- 172.21.15.78:0/771320443 >> v2:172.21.15.150:3300/0 conn(0x7f06d40c8e00 msgr2=0x7f06d406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 --2- 172.21.15.78:0/771320443 >> v2:172.21.15.150:3300/0 conn(0x7f06d40c8e00 0x7f06d406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 -- 172.21.15.78:0/771320443 >> v1:172.21.15.78:6789/0 conn(0x7f06d4152700 legacy=0x7f06d40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d9418640 1 -- 172.21.15.78:0/771320443 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d4155770 con 0x7f06d406d910 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.753+0000 7f06d37fe640 1 -- 172.21.15.78:0/771320443 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093932551 0 0) 0x7f06d406cd20 con 0x7f06d4152700 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06d9418640 1 --2- 172.21.15.78:0/771320443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d406d910 0x7f06d406c7e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f06d40c7620 tx=0x7f06cc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc28be813d472ab9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06d37fe640 1 -- 172.21.15.78:0/771320443 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc02cb20 con 0x7f06d406d910 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06d37fe640 1 -- 172.21.15.78:0/771320443 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc02cc80 con 0x7f06d406d910 2024-02-19T14:53:23.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06d37fe640 1 -- 172.21.15.78:0/771320443 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0343f0 con 0x7f06d406d910 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 -- 172.21.15.78:0/771320443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d406d910 msgr2=0x7f06d406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 --2- 172.21.15.78:0/771320443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d406d910 0x7f06d406c7e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f06d40c7620 tx=0x7f06cc02bf80 comp rx=0 tx=0).stop 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 -- 172.21.15.78:0/771320443 shutdown_connections 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 --2- 172.21.15.78:0/771320443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d406d910 0x7f06d406c7e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 --2- 172.21.15.78:0/771320443 >> v2:172.21.15.150:3300/0 conn(0x7f06d40c8e00 0x7f06d406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 -- 172.21.15.78:0/771320443 >> 172.21.15.78:0/771320443 conn(0x7f06d40cac90 msgr2=0x7f06d40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 -- 172.21.15.78:0/771320443 shutdown_connections 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.754+0000 7f06dae7b640 1 -- 172.21.15.78:0/771320443 wait complete. 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.755+0000 7f06dae7b640 1 Processor -- start 2024-02-19T14:53:23.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 -- start start 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06d406d910 0x7f06d4153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 0x7f06d40c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d4152700 0x7f06d40c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40c4980 con 0x7f06d40c8e00 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d4159540 con 0x7f06d4152700 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06dae7b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40aac90 con 0x7f06d406d910 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 0x7f06d40c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 0x7f06d40c26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40340/0 (socket says 172.21.15.78:40340) 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.756+0000 7f06d3fff640 1 -- 172.21.15.78:0/628305482 learned_addr learned my addr 172.21.15.78:0/628305482 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d8c17640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06d406d910 0x7f06d4153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d9418640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d4152700 0x7f06d40c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d3fff640 1 -- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06d406d910 msgr2=0x7f06d4153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d3fff640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06d406d910 0x7f06d4153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d3fff640 1 -- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d4152700 msgr2=0x7f06d40c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d3fff640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d4152700 0x7f06d40c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d3fff640 1 -- 172.21.15.78:0/628305482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc009b30 con 0x7f06d40c8e00 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d8c17640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06d406d910 0x7f06d4153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d9418640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d4152700 0x7f06d40c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d3fff640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 0x7f06d40c26a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f06c800e9b0 tx=0x7f06c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c8009670 con 0x7f06d40c8e00 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c800c9a0 con 0x7f06d40c8e00 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c800cd50 con 0x7f06d40c8e00 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d40aaf70 con 0x7f06d40c8e00 2024-02-19T14:53:23.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.757+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d40c1dc0 con 0x7f06d40c8e00 2024-02-19T14:53:23.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.758+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d40ab100 con 0x7f06d40c8e00 2024-02-19T14:53:23.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.758+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f06c8013470 con 0x7f06d40c8e00 2024-02-19T14:53:23.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.759+0000 7f06d17fa640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06c006b810 0x7f06c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:23.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.759+0000 7f06d8c17640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06c006b810 0x7f06c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:23.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.759+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f06c808a500 con 0x7f06d40c8e00 2024-02-19T14:53:23.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.759+0000 7f06d8c17640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06c006b810 0x7f06c006dcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f06c4000f60 tx=0x7f06c4007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:23.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.763+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f06c8054dd0 con 0x7f06d40c8e00 2024-02-19T14:53:23.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.937+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06d4074990 con 0x7f06d40c8e00 2024-02-19T14:53:23.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.937+0000 7f06d17fa640 1 -- 172.21.15.78:0/628305482 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06c8058490 con 0x7f06d40c8e00 2024-02-19T14:53:23.940 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06c006b810 msgr2=0x7f06c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06c006b810 0x7f06c006dcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f06c4000f60 tx=0x7f06c4007ca0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 msgr2=0x7f06d40c26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 0x7f06d40c26a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f06c800e9b0 tx=0x7f06c800ee70 comp rx=0 tx=0).stop 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 shutdown_connections 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06c006b810 0x7f06c006dcd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d40c8e00 0x7f06d40c26a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d4152700 0x7f06d40c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 --2- 172.21.15.78:0/628305482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06d406d910 0x7f06d4153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:23.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 >> 172.21.15.78:0/628305482 conn(0x7f06d40cac90 msgr2=0x7f06d40ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:23.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 shutdown_connections 2024-02-19T14:53:23.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:23.939+0000 7f06dae7b640 1 -- 172.21.15.78:0/628305482 wait complete. 2024-02-19T14:53:23.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:53:24.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.053+0000 7f330a891640 1 Processor -- start 2024-02-19T14:53:24.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.053+0000 7f330a891640 1 -- start start 2024-02-19T14:53:24.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.053+0000 7f330a891640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3304152830 0x7f33040c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f330a891640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33040c7990 0x7f330406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f330a891640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33041586a0 con 0x7f33040c5760 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f330a891640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33040c6550 con 0x7f3304152830 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f330a891640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3304159320 con 0x7f33040c7990 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f33037fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f33040c5760 0x7f33040c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60830/0 (socket says 172.21.15.78:60830) 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f33037fe640 1 -- 172.21.15.78:0/3342329475 learned_addr learned my addr 172.21.15.78:0/3342329475 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3303fff640 1 --2- 172.21.15.78:0/3342329475 >> v2:172.21.15.150:3300/0 conn(0x7f3304152830 0x7f33040c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3308e2e640 1 --2- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33040c7990 0x7f330406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3302ffd640 1 -- 172.21.15.78:0/3342329475 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944248795 0 0) 0x7f33041586a0 con 0x7f33040c5760 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3302ffd640 1 -- 172.21.15.78:0/3342329475 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ec003520 con 0x7f33040c5760 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3303fff640 1 -- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33040c7990 msgr2=0x7f330406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3303fff640 1 --2- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33040c7990 0x7f330406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3303fff640 1 -- 172.21.15.78:0/3342329475 >> v1:172.21.15.78:6789/0 conn(0x7f33040c5760 legacy=0x7f33040c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.054+0000 7f3303fff640 1 -- 172.21.15.78:0/3342329475 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3304158980 con 0x7f3304152830 2024-02-19T14:53:24.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.055+0000 7f3303fff640 1 --2- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304152830 0x7f33040c5220 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f32f80097d0 tx=0x7f32f802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aa7af5883356343 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.055+0000 7f3302ffd640 1 -- 172.21.15.78:0/3342329475 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8031770 con 0x7f3304152830 2024-02-19T14:53:24.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.055+0000 7f3302ffd640 1 -- 172.21.15.78:0/3342329475 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f80318d0 con 0x7f3304152830 2024-02-19T14:53:24.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.055+0000 7f3302ffd640 1 -- 172.21.15.78:0/3342329475 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8031c60 con 0x7f3304152830 2024-02-19T14:53:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 -- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304152830 msgr2=0x7f33040c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 --2- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304152830 0x7f33040c5220 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f32f80097d0 tx=0x7f32f802b810 comp rx=0 tx=0).stop 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 -- 172.21.15.78:0/3342329475 shutdown_connections 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 --2- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33040c7990 0x7f330406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 --2- 172.21.15.78:0/3342329475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304152830 0x7f33040c5220 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 -- 172.21.15.78:0/3342329475 >> 172.21.15.78:0/3342329475 conn(0x7f33040ca8a0 msgr2=0x7f33040cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 -- 172.21.15.78:0/3342329475 shutdown_connections 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.056+0000 7f330a891640 1 -- 172.21.15.78:0/3342329475 wait complete. 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 Processor -- start 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 -- start start 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 0x7f33040bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3304152830 0x7f33040bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304154370 0x7f33041e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3304152ee0 con 0x7f33040c5760 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33041534a0 con 0x7f3304154370 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.057+0000 7f330a891640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33041eb0b0 con 0x7f3304152830 2024-02-19T14:53:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 0x7f33040bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 0x7f33040bf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40356/0 (socket says 172.21.15.78:40356) 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 -- 172.21.15.78:0/1181317873 learned_addr learned my addr 172.21.15.78:0/1181317873 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3308e2e640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304154370 0x7f33041e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f33037fe640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3304152830 0x7f33040bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 -- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3304152830 msgr2=0x7f33040bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3304152830 0x7f33040bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 -- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304154370 msgr2=0x7f33041e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304154370 0x7f33041e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 -- 172.21.15.78:0/1181317873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f8009400 con 0x7f33040c5760 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3303fff640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 0x7f33040bf3e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f32f80097d0 tx=0x7f32f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8032610 con 0x7f33040c5760 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f8032790 con 0x7f33040c5760 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33041eb390 con 0x7f33040c5760 2024-02-19T14:53:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8032b40 con 0x7f33040c5760 2024-02-19T14:53:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.058+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33040bdad0 con 0x7f33040c5760 2024-02-19T14:53:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.059+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33041eb520 con 0x7f33040c5760 2024-02-19T14:53:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.060+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f32f8032ca0 con 0x7f33040c5760 2024-02-19T14:53:24.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.060+0000 7f3300ff9640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32f006b810 0x7f32f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.060+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f32f80ac060 con 0x7f33040c5760 2024-02-19T14:53:24.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.060+0000 7f33037fe640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32f006b810 0x7f32f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.061+0000 7f33037fe640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32f006b810 0x7f32f006dcd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f33040c0060 tx=0x7f32f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.065+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f32f80769e0 con 0x7f33040c5760 2024-02-19T14:53:24.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.239+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3304074990 con 0x7f33040c5760 2024-02-19T14:53:24.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.240+0000 7f3300ff9640 1 -- 172.21.15.78:0/1181317873 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7f32f807a0a0 con 0x7f33040c5760 2024-02-19T14:53:24.242 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32f006b810 msgr2=0x7f32f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32f006b810 0x7f32f006dcd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f33040c0060 tx=0x7f32f4009040 comp rx=0 tx=0).stop 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 msgr2=0x7f33040bf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 0x7f33040bf3e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f32f80097d0 tx=0x7f32f80046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 shutdown_connections 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32f006b810 0x7f32f006dcd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3304154370 0x7f33041e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3304152830 0x7f33040bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 --2- 172.21.15.78:0/1181317873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33040c5760 0x7f33040bf3e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 >> 172.21.15.78:0/1181317873 conn(0x7f33040ca8a0 msgr2=0x7f3304150840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 shutdown_connections 2024-02-19T14:53:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.242+0000 7f330a891640 1 -- 172.21.15.78:0/1181317873 wait complete. 2024-02-19T14:53:24.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:53:24.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.355+0000 7f445db20640 1 Processor -- start 2024-02-19T14:53:24.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.355+0000 7f445db20640 1 -- start start 2024-02-19T14:53:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.355+0000 7f445db20640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4458155f90 0x7f445806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.355+0000 7f445db20640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580c6af0 con 0x7f44581588a0 2024-02-19T14:53:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.355+0000 7f445db20640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580c7940 con 0x7f44580c6d30 2024-02-19T14:53:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.355+0000 7f445db20640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580c7c20 con 0x7f4458155f90 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f4457fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f44581588a0 0x7f4458158d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60846/0 (socket says 172.21.15.78:60846) 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f4457fff640 1 -- 172.21.15.78:0/3981360298 learned_addr learned my addr 172.21.15.78:0/3981360298 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44577fe640 1 --2- 172.21.15.78:0/3981360298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4458155f90 0x7f445806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127297321 0 0) 0x7f44580c6af0 con 0x7f44581588a0 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4440003520 con 0x7f44581588a0 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609682074 0 0) 0x7f44580c7940 con 0x7f44580c6d30 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44580c6af0 con 0x7f44580c6d30 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44577fe640 1 -- 172.21.15.78:0/3981360298 >> v1:172.21.15.150:6789/0 conn(0x7f44580c6d30 legacy=0x7f44580c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44577fe640 1 -- 172.21.15.78:0/3981360298 >> v1:172.21.15.78:6789/0 conn(0x7f44581588a0 legacy=0x7f4458158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.356+0000 7f44577fe640 1 -- 172.21.15.78:0/3981360298 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580c7f00 con 0x7f4458155f90 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f44577fe640 1 --2- 172.21.15.78:0/3981360298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4458155f90 0x7f445806d550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097d0 tx=0x7f444c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b84ed7249ee4e9f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c031800 con 0x7f4458155f90 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c031960 con 0x7f4458155f90 2024-02-19T14:53:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f44567fc640 1 -- 172.21.15.78:0/3981360298 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c031cd0 con 0x7f4458155f90 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f445db20640 1 -- 172.21.15.78:0/3981360298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4458155f90 msgr2=0x7f445806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f445db20640 1 --2- 172.21.15.78:0/3981360298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4458155f90 0x7f445806d550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097d0 tx=0x7f444c02b8a0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f445db20640 1 -- 172.21.15.78:0/3981360298 shutdown_connections 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.357+0000 7f445db20640 1 --2- 172.21.15.78:0/3981360298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4458155f90 0x7f445806d550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.358+0000 7f445db20640 1 -- 172.21.15.78:0/3981360298 >> 172.21.15.78:0/3981360298 conn(0x7f445806b700 msgr2=0x7f44580aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.358+0000 7f445db20640 1 -- 172.21.15.78:0/3981360298 shutdown_connections 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.358+0000 7f445db20640 1 -- 172.21.15.78:0/3981360298 wait complete. 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 Processor -- start 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 -- start start 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44580c6d30 0x7f44580ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4458155f90 0x7f445814e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 0x7f445814ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44581eb590 con 0x7f44581588a0 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445814f250 con 0x7f4458155f90 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.359+0000 7f445db20640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44580ac940 con 0x7f44580c6d30 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 0x7f445814ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 0x7f445814ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40370/0 (socket says 172.21.15.78:40370) 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 -- 172.21.15.78:0/978321078 learned_addr learned my addr 172.21.15.78:0/978321078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f44577fe640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44580c6d30 0x7f44580ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4456ffd640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4458155f90 0x7f445814e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 -- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44580c6d30 msgr2=0x7f44580ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44580c6d30 0x7f44580ab660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 -- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4458155f90 msgr2=0x7f445814e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4458155f90 0x7f445814e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 -- 172.21.15.78:0/978321078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c009400 con 0x7f44581588a0 2024-02-19T14:53:24.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f4457fff640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 0x7f445814ed10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f445000caa0 tx=0x7f445000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000ac00 con 0x7f44581588a0 2024-02-19T14:53:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445000ad60 con 0x7f44581588a0 2024-02-19T14:53:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4458099be0 con 0x7f44581588a0 2024-02-19T14:53:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450012600 con 0x7f44581588a0 2024-02-19T14:53:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.360+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44580abba0 con 0x7f44581588a0 2024-02-19T14:53:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.361+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44580acb40 con 0x7f44581588a0 2024-02-19T14:53:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.362+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4450007070 con 0x7f44581588a0 2024-02-19T14:53:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.362+0000 7f443ffff640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444406b810 0x7f444406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.362+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f445008a1f0 con 0x7f44581588a0 2024-02-19T14:53:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.362+0000 7f44577fe640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444406b810 0x7f444406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.363+0000 7f44577fe640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444406b810 0x7f444406dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f444c036000 tx=0x7f444c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.366+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f44580acb40 con 0x7f44581588a0 2024-02-19T14:53:24.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.537+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44580acb40 con 0x7f44581588a0 2024-02-19T14:53:24.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.538+0000 7f443ffff640 1 -- 172.21.15.78:0/978321078 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44580acb40 con 0x7f44581588a0 2024-02-19T14:53:24.540 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:53:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444406b810 msgr2=0x7f444406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444406b810 0x7f444406dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f444c036000 tx=0x7f444c03c040 comp rx=0 tx=0).stop 2024-02-19T14:53:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 msgr2=0x7f445814ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 0x7f445814ed10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f445000caa0 tx=0x7f445000cf60 comp rx=0 tx=0).stop 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 shutdown_connections 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444406b810 0x7f444406dcd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44581588a0 0x7f445814ed10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4458155f90 0x7f445814e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.539+0000 7f445db20640 1 --2- 172.21.15.78:0/978321078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44580c6d30 0x7f44580ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.540+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 >> 172.21.15.78:0/978321078 conn(0x7f445806b700 msgr2=0x7f44580aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.540+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 shutdown_connections 2024-02-19T14:53:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.540+0000 7f445db20640 1 -- 172.21.15.78:0/978321078 wait complete. 2024-02-19T14:53:24.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.651+0000 7f8f56c1b640 1 Processor -- start 2024-02-19T14:53:24.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.651+0000 7f8f56c1b640 1 -- start start 2024-02-19T14:53:24.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f56c1b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8f500ac770 0x7f8f500c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f56c1b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f50158c40 con 0x7f8f500ac390 2024-02-19T14:53:24.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f56c1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f500acbb0 con 0x7f8f500bdf20 2024-02-19T14:53:24.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f56c1b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f500c8de0 con 0x7f8f500ac770 2024-02-19T14:53:24.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f551b8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8f500ac770 0x7f8f500c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f551b8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8f500ac770 0x7f8f500c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40372/0 (socket says 172.21.15.78:40372) 2024-02-19T14:53:24.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f549b7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8f500ac390 0x7f8f50158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60848/0 (socket says 172.21.15.78:60848) 2024-02-19T14:53:24.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f551b8640 1 -- 172.21.15.78:0/395546361 learned_addr learned my addr 172.21.15.78:0/395546361 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:24.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f551b8640 1 -- 172.21.15.78:0/395546361 >> v1:172.21.15.150:6789/0 conn(0x7f8f500bdf20 legacy=0x7f8f5015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f551b8640 1 -- 172.21.15.78:0/395546361 >> v1:172.21.15.78:6789/0 conn(0x7f8f500ac390 legacy=0x7f8f50158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f4f7fe640 1 -- 172.21.15.78:0/395546361 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411074328 0 0) 0x7f8f50158c40 con 0x7f8f500ac390 2024-02-19T14:53:24.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.652+0000 7f8f551b8640 1 -- 172.21.15.78:0/395546361 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f50155a60 con 0x7f8f500ac770 2024-02-19T14:53:24.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.653+0000 7f8f551b8640 1 --2- 172.21.15.78:0/395546361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac770 0x7f8f500c55f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009e50 tx=0x7f8f4802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f1b631d21ba03c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.653+0000 7f8f4f7fe640 1 -- 172.21.15.78:0/395546361 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4802ca80 con 0x7f8f500ac770 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.653+0000 7f8f4f7fe640 1 -- 172.21.15.78:0/395546361 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4802cbe0 con 0x7f8f500ac770 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.653+0000 7f8f4f7fe640 1 -- 172.21.15.78:0/395546361 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f480343f0 con 0x7f8f500ac770 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 -- 172.21.15.78:0/395546361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac770 msgr2=0x7f8f500c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/395546361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac770 0x7f8f500c55f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009e50 tx=0x7f8f4802bee0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 -- 172.21.15.78:0/395546361 shutdown_connections 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/395546361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac770 0x7f8f500c55f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 -- 172.21.15.78:0/395546361 >> 172.21.15.78:0/395546361 conn(0x7f8f500ca890 msgr2=0x7f8f500caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 -- 172.21.15.78:0/395546361 shutdown_connections 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.654+0000 7f8f56c1b640 1 -- 172.21.15.78:0/395546361 wait complete. 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.655+0000 7f8f56c1b640 1 Processor -- start 2024-02-19T14:53:24.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.655+0000 7f8f56c1b640 1 -- start start 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f56c1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 0x7f8f5014f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f56c1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f500ac770 0x7f8f500aac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f56c1b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f500bdf20 0x7f8f5014e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f56c1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f501eb000 con 0x7f8f500ac390 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f56c1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f500c4700 con 0x7f8f500ac770 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f56c1b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f500c3830 con 0x7f8f500bdf20 2024-02-19T14:53:24.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 0x7f8f5014f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 0x7f8f5014f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40374/0 (socket says 172.21.15.78:40374) 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 -- 172.21.15.78:0/4274404504 learned_addr learned my addr 172.21.15.78:0/4274404504 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f4ffff640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f500ac770 0x7f8f500aac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f551b8640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f500bdf20 0x7f8f5014e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 -- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f500bdf20 msgr2=0x7f8f5014e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f500bdf20 0x7f8f5014e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 -- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f500ac770 msgr2=0x7f8f500aac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f500ac770 0x7f8f500aac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 -- 172.21.15.78:0/4274404504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48009b30 con 0x7f8f500ac390 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.656+0000 7f8f549b7640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 0x7f8f5014f2e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000c4a0 tx=0x7f8f4000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.657+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400074e0 con 0x7f8f500ac390 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.657+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f40007640 con 0x7f8f500ac390 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.657+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40004a00 con 0x7f8f500ac390 2024-02-19T14:53:24.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.657+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f501ea380 con 0x7f8f500ac390 2024-02-19T14:53:24.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.657+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f501ea940 con 0x7f8f500ac390 2024-02-19T14:53:24.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.658+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f50074990 con 0x7f8f500ac390 2024-02-19T14:53:24.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.658+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8f40004b60 con 0x7f8f500ac390 2024-02-19T14:53:24.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.658+0000 7f8f4d7fa640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f3c06b840 0x7f8f3c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.659+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8f400893b0 con 0x7f8f500ac390 2024-02-19T14:53:24.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.659+0000 7f8f4ffff640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f3c06b840 0x7f8f3c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.659+0000 7f8f4ffff640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f3c06b840 0x7f8f3c06dd00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8f501586e0 tx=0x7f8f44009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.663+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8f40053d30 con 0x7f8f500ac390 2024-02-19T14:53:24.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.871+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f5006da00 con 0x7f8f500ac390 2024-02-19T14:53:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.871+0000 7f8f4d7fa640 1 -- 172.21.15.78:0/4274404504 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f8f400573f0 con 0x7f8f500ac390 2024-02-19T14:53:24.874 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:24.874 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:16.569892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 20 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f3c06b840 msgr2=0x7f8f3c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f3c06b840 0x7f8f3c06dd00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8f501586e0 tx=0x7f8f44009040 comp rx=0 tx=0).stop 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 msgr2=0x7f8f5014f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 0x7f8f5014f2e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000c4a0 tx=0x7f8f4000c960 comp rx=0 tx=0).stop 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 shutdown_connections 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f3c06b840 0x7f8f3c06dd00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f500bdf20 0x7f8f5014e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f500ac770 0x7f8f500aac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 --2- 172.21.15.78:0/4274404504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f500ac390 0x7f8f5014f2e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 >> 172.21.15.78:0/4274404504 conn(0x7f8f500ca890 msgr2=0x7f8f5006c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 shutdown_connections 2024-02-19T14:53:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.873+0000 7f8f56c1b640 1 -- 172.21.15.78:0/4274404504 wait complete. 2024-02-19T14:53:24.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:24.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.984+0000 7f52fdc50640 1 Processor -- start 2024-02-19T14:53:24.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.984+0000 7f52fdc50640 1 -- start start 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52fdc50640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52f80acc50 0x7f52f80c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52fdc50640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f8159130 con 0x7f52f8155070 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52fdc50640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f80ad030 con 0x7f52f80718f0 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52fdc50640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f80c2cf0 con 0x7f52f80acc50 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52f80acc50 0x7f52f80c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52f80acc50 0x7f52f80c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40384/0 (socket says 172.21.15.78:40384) 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f52f8155070 0x7f52f8155500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60854/0 (socket says 172.21.15.78:60854) 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f77fe640 1 -- 172.21.15.78:0/1781055950 learned_addr learned my addr 172.21.15.78:0/1781055950 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f77fe640 1 -- 172.21.15.78:0/1781055950 >> v1:172.21.15.150:6789/0 conn(0x7f52f80718f0 legacy=0x7f52f8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f77fe640 1 -- 172.21.15.78:0/1781055950 >> v1:172.21.15.78:6789/0 conn(0x7f52f8155070 legacy=0x7f52f8155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.985+0000 7f52f77fe640 1 -- 172.21.15.78:0/1781055950 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f81595c0 con 0x7f52f80acc50 2024-02-19T14:53:24.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.986+0000 7f52f77fe640 1 --2- 172.21.15.78:0/1781055950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f80c17e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7f84b829cadba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.986+0000 7f52f67fc640 1 -- 172.21.15.78:0/1781055950 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8030860 con 0x7f52f80acc50 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.986+0000 7f52f67fc640 1 -- 172.21.15.78:0/1781055950 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e80309c0 con 0x7f52f80acc50 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.986+0000 7f52f67fc640 1 -- 172.21.15.78:0/1781055950 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8030d50 con 0x7f52f80acc50 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 -- 172.21.15.78:0/1781055950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 msgr2=0x7f52f80c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 --2- 172.21.15.78:0/1781055950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f80c17e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e802b9a0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 -- 172.21.15.78:0/1781055950 shutdown_connections 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 --2- 172.21.15.78:0/1781055950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f80c17e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 -- 172.21.15.78:0/1781055950 >> 172.21.15.78:0/1781055950 conn(0x7f52f80c7800 msgr2=0x7f52f80c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 -- 172.21.15.78:0/1781055950 shutdown_connections 2024-02-19T14:53:24.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.987+0000 7f52fdc50640 1 -- 172.21.15.78:0/1781055950 wait complete. 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.988+0000 7f52fdc50640 1 Processor -- start 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.988+0000 7f52fdc50640 1 -- start start 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52f80718f0 0x7f52f81edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f8151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52f8155070 0x7f52f81337e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f80c2030 con 0x7f52f80acc50 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f81341f0 con 0x7f52f80718f0 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f81368d0 con 0x7f52f8155070 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f8151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f8151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40390/0 (socket says 172.21.15.78:40390) 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 -- 172.21.15.78:0/887982476 learned_addr learned my addr 172.21.15.78:0/887982476 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f77fe640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52f80718f0 0x7f52f81edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f7fff640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52f8155070 0x7f52f81337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 -- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52f8155070 msgr2=0x7f52f81337e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52f8155070 0x7f52f81337e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 -- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52f80718f0 msgr2=0x7f52f81edbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52f80718f0 0x7f52f81edbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 -- 172.21.15.78:0/887982476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e80094d0 con 0x7f52f80acc50 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52f6ffd640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f8151710 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00c3e0 tx=0x7f52ec00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0074e0 con 0x7f52f80acc50 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec007640 con 0x7f52f80acc50 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec0049e0 con 0x7f52f80acc50 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f8132ee0 con 0x7f52f80acc50 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.989+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f8132920 con 0x7f52f80acc50 2024-02-19T14:53:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.990+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f806c820 con 0x7f52f80acc50 2024-02-19T14:53:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.991+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f52ec004b40 con 0x7f52f80acc50 2024-02-19T14:53:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.991+0000 7f52dffff640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f52e406b840 0x7f52e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.991+0000 7f52f77fe640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f52e406b840 0x7f52e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.991+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f52ec089ba0 con 0x7f52f80acc50 2024-02-19T14:53:24.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.992+0000 7f52f77fe640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f52e406b840 0x7f52e406dd00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f52e8036000 tx=0x7f52e8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:24.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:24.996+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f52ec054520 con 0x7f52f80acc50 2024-02-19T14:53:25.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.167+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f8074990 con 0x7f52f80acc50 2024-02-19T14:53:25.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.168+0000 7f52dffff640 1 -- 172.21.15.78:0/887982476 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+17059 (secure 0 0 0) 0x7f52ec057be0 con 0x7f52f80acc50 2024-02-19T14:53:25.171 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:25.171 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:10.560315+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:52:59.040775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:00.165210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/3231075223":"2024-02-20T14:53:10.560278+0000","172.21.15.78:6836/3231075223":"2024-02-20T14:53:10.560278+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:25.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f52e406b840 msgr2=0x7f52e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f52e406b840 0x7f52e406dd00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f52e8036000 tx=0x7f52e8008030 comp rx=0 tx=0).stop 2024-02-19T14:53:25.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 msgr2=0x7f52f8151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f8151710 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00c3e0 tx=0x7f52ec00c8a0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 shutdown_connections 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f52e406b840 0x7f52e406dd00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52f80acc50 0x7f52f8151710 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52f8155070 0x7f52f81337e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 --2- 172.21.15.78:0/887982476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52f80718f0 0x7f52f81edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 >> 172.21.15.78:0/887982476 conn(0x7f52f80c7800 msgr2=0x7f52f81548c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 shutdown_connections 2024-02-19T14:53:25.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.170+0000 7f52fdc50640 1 -- 172.21.15.78:0/887982476 wait complete. 2024-02-19T14:53:25.183 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:53:25.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:25.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.281+0000 7f6d94684640 1 Processor -- start 2024-02-19T14:53:25.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.281+0000 7f6d94684640 1 -- start start 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d94684640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0c4980 0x7f6d8c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d94684640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6d8c0705b0 0x7f6d8c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d94684640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0ae670 con 0x7f6d8c0add70 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d94684640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8c0716b0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d94684640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c153780 con 0x7f6d8c0c4980 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d92c21640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6d8c0705b0 0x7f6d8c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d92420640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6d8c0add70 0x7f6d8c06eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60860/0 (socket says 172.21.15.78:60860) 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d92420640 1 -- 172.21.15.78:0/2160950504 learned_addr learned my addr 172.21.15.78:0/2160950504 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.282+0000 7f6d91c1f640 1 --2- 172.21.15.78:0/2160950504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0c4980 0x7f6d8c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d9141e640 1 -- 172.21.15.78:0/2160950504 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86492536 0 0) 0x7f6d8c0ae670 con 0x7f6d8c0add70 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d9141e640 1 -- 172.21.15.78:0/2160950504 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78003520 con 0x7f6d8c0add70 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d92c21640 1 -- 172.21.15.78:0/2160950504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0c4980 msgr2=0x7f6d8c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d92c21640 1 --2- 172.21.15.78:0/2160950504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0c4980 0x7f6d8c0c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d92c21640 1 -- 172.21.15.78:0/2160950504 >> v1:172.21.15.78:6789/0 conn(0x7f6d8c0add70 legacy=0x7f6d8c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d92c21640 1 -- 172.21.15.78:0/2160950504 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c153a60 con 0x7f6d8c0705b0 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.283+0000 7f6d92c21640 1 --2- 172.21.15.78:0/2160950504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0709c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6d840097a0 tx=0x7f6d8402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac83daae57074017 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d9141e640 1 -- 172.21.15.78:0/2160950504 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84031780 con 0x7f6d8c0705b0 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d9141e640 1 -- 172.21.15.78:0/2160950504 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d840318e0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d9141e640 1 -- 172.21.15.78:0/2160950504 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84031c70 con 0x7f6d8c0705b0 2024-02-19T14:53:25.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 -- 172.21.15.78:0/2160950504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 msgr2=0x7f6d8c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 --2- 172.21.15.78:0/2160950504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0709c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6d840097a0 tx=0x7f6d8402b820 comp rx=0 tx=0).stop 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 -- 172.21.15.78:0/2160950504 shutdown_connections 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 --2- 172.21.15.78:0/2160950504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0709c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 --2- 172.21.15.78:0/2160950504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0c4980 0x7f6d8c0c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 -- 172.21.15.78:0/2160950504 >> 172.21.15.78:0/2160950504 conn(0x7f6d8c0c85e0 msgr2=0x7f6d8c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 -- 172.21.15.78:0/2160950504 shutdown_connections 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.284+0000 7f6d94684640 1 -- 172.21.15.78:0/2160950504 wait complete. 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.285+0000 7f6d94684640 1 Processor -- start 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.285+0000 7f6d94684640 1 -- start start 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d94684640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d94684640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0add70 0x7f6d8c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d94684640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c0c4980 0x7f6d8c0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d94684640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1ec6b0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d94684640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c152dc0 con 0x7f6d8c0c4980 2024-02-19T14:53:25.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d94684640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c1ecad0 con 0x7f6d8c0add70 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40406/0 (socket says 172.21.15.78:40406) 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 -- 172.21.15.78:0/2719543706 learned_addr learned my addr 172.21.15.78:0/2719543706 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d91c1f640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0add70 0x7f6d8c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92c21640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c0c4980 0x7f6d8c0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 -- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0add70 msgr2=0x7f6d8c1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0add70 0x7f6d8c1552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 -- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c0c4980 msgr2=0x7f6d8c0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c0c4980 0x7f6d8c0bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.286+0000 7f6d92420640 1 -- 172.21.15.78:0/2719543706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84009400 con 0x7f6d8c0705b0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.287+0000 7f6d92420640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0c1fc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800cb40 tx=0x7f6d8800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.287+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8800aca0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.287+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8800ae00 con 0x7f6d8c0705b0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.287+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d88012620 con 0x7f6d8c0705b0 2024-02-19T14:53:25.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.287+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c1ebd10 con 0x7f6d8c0705b0 2024-02-19T14:53:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.287+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c1ec2d0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.288+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c0676a0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.288+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6d88007070 con 0x7f6d8c0705b0 2024-02-19T14:53:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.288+0000 7f6d82ffd640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d7006b810 0x7f6d7006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.289+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6d8808a2e0 con 0x7f6d8c0705b0 2024-02-19T14:53:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.289+0000 7f6d91c1f640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d7006b810 0x7f6d7006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.289+0000 7f6d91c1f640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d7006b810 0x7f6d7006dcd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c14f710 tx=0x7f6d7c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:25.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.293+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6d88054c60 con 0x7f6d8c0705b0 2024-02-19T14:53:25.486 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:53:25.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:53:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.500+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d8c1ec920 con 0x7f6d8c0705b0 2024-02-19T14:53:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.501+0000 7f6d82ffd640 1 -- 172.21.15.78:0/2719543706 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f6d88001c00 con 0x7f6d8c0705b0 2024-02-19T14:53:25.503 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:25.503 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:16.569892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 20 2024-02-19T14:53:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d7006b810 msgr2=0x7f6d7006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d7006b810 0x7f6d7006dcd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c14f710 tx=0x7f6d7c009190 comp rx=0 tx=0).stop 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 msgr2=0x7f6d8c0c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0c1fc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800cb40 tx=0x7f6d8800a060 comp rx=0 tx=0).stop 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 shutdown_connections 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d7006b810 0x7f6d7006dcd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d8c0c4980 0x7f6d8c0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d8c0add70 0x7f6d8c1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 --2- 172.21.15.78:0/2719543706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d8c0705b0 0x7f6d8c0c1fc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 >> 172.21.15.78:0/2719543706 conn(0x7f6d8c0c85e0 msgr2=0x7f6d8c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 shutdown_connections 2024-02-19T14:53:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.503+0000 7f6d94684640 1 -- 172.21.15.78:0/2719543706 wait complete. 2024-02-19T14:53:25.516 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.674+0000 7f1b3ef77640 1 Processor -- start 2024-02-19T14:53:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.674+0000 7f1b3ef77640 1 -- start start 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3ef77640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b38152610 0x7f1b38158f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3ef77640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b380c1be0 0x7f1b380c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3ef77640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b381529f0 con 0x7f1b3806e350 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3ef77640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3806da70 con 0x7f1b380c1be0 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3ef77640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b380bf9b0 con 0x7f1b38152610 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b380c1be0 0x7f1b380c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b380c1be0 0x7f1b380c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40414/0 (socket says 172.21.15.78:40414) 2024-02-19T14:53:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 -- 172.21.15.78:0/2160728008 learned_addr learned my addr 172.21.15.78:0/2160728008 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3cd13640 1 --2- 172.21.15.78:0/2160728008 >> v2:172.21.15.150:3300/0 conn(0x7f1b38152610 0x7f1b38158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 -- 172.21.15.78:0/2160728008 >> v2:172.21.15.150:3300/0 conn(0x7f1b38152610 msgr2=0x7f1b38158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 --2- 172.21.15.78:0/2160728008 >> v2:172.21.15.150:3300/0 conn(0x7f1b38152610 0x7f1b38158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 -- 172.21.15.78:0/2160728008 >> v1:172.21.15.150:6789/0 conn(0x7f1b3806e350 legacy=0x7f1b380c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.675+0000 7f1b3d514640 1 -- 172.21.15.78:0/2160728008 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38159470 con 0x7f1b380c1be0 2024-02-19T14:53:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3d514640 1 --2- 172.21.15.78:0/2160728008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380c32c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1b3806f5f0 tx=0x7f1b3002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61015ac63bd93afb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b377fe640 1 -- 172.21.15.78:0/2160728008 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3002cb20 con 0x7f1b380c1be0 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b377fe640 1 -- 172.21.15.78:0/2160728008 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3002cc80 con 0x7f1b380c1be0 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b377fe640 1 -- 172.21.15.78:0/2160728008 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b300343f0 con 0x7f1b380c1be0 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2160728008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 msgr2=0x7f1b380c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2160728008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380c32c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1b3806f5f0 tx=0x7f1b3002bf80 comp rx=0 tx=0).stop 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2160728008 shutdown_connections 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2160728008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380c32c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2160728008 >> v2:172.21.15.150:3300/0 conn(0x7f1b38152610 0x7f1b38158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2160728008 >> 172.21.15.78:0/2160728008 conn(0x7f1b3806b270 msgr2=0x7f1b3806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2160728008 shutdown_connections 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.676+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2160728008 wait complete. 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.677+0000 7f1b3ef77640 1 Processor -- start 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 -- start start 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3806e350 0x7f1b380abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b38152610 0x7f1b381551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380a8bb0 con 0x7f1b380c1be0 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b381547f0 con 0x7f1b3806e350 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b3ef77640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380ac2b0 con 0x7f1b38152610 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.678+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380a97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40420/0 (socket says 172.21.15.78:40420) 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 -- 172.21.15.78:0/2372151479 learned_addr learned my addr 172.21.15.78:0/2372151479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b3d514640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b38152610 0x7f1b381551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b3cd13640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3806e350 0x7f1b380abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 -- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b38152610 msgr2=0x7f1b381551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b38152610 0x7f1b381551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 -- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3806e350 msgr2=0x7f1b380abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3806e350 0x7f1b380abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 -- 172.21.15.78:0/2372151479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30009b30 con 0x7f1b380c1be0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b37fff640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380a97e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c3c0 tx=0x7f1b2400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240052d0 con 0x7f1b380c1be0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24005450 con 0x7f1b380c1be0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38154d50 con 0x7f1b380c1be0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24007260 con 0x7f1b380c1be0 2024-02-19T14:53:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.679+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b380a88d0 con 0x7f1b380c1be0 2024-02-19T14:53:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.680+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38154ee0 con 0x7f1b380c1be0 2024-02-19T14:53:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.681+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1b240073c0 con 0x7f1b380c1be0 2024-02-19T14:53:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.681+0000 7f1b357fa640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b1c06b810 0x7f1b1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.681+0000 7f1b3cd13640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b1c06b810 0x7f1b1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.681+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1b24089650 con 0x7f1b380c1be0 2024-02-19T14:53:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.682+0000 7f1b3cd13640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b1c06b810 0x7f1b1c06dcd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009b40 tx=0x7f1b2c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.686+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b24053fd0 con 0x7f1b380c1be0 2024-02-19T14:53:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.895+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b380a9d20 con 0x7f1b380c1be0 2024-02-19T14:53:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.895+0000 7f1b357fa640 1 -- 172.21.15.78:0/2372151479 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f1b24057690 con 0x7f1b380c1be0 2024-02-19T14:53:25.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:25.898 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:01.188620+0000","modified":"2024-02-19T14:53:16.569892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4766},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4766":{"gid":4766,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/4259811019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":4259811019},{"type":"v1","addr":"172.21.15.150:6835","nonce":4259811019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-19T14:53:25.899 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 20 2024-02-19T14:53:25.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.897+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b1c06b810 msgr2=0x7f1b1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.897+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b1c06b810 0x7f1b1c06dcd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009b40 tx=0x7f1b2c009190 comp rx=0 tx=0).stop 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.897+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 msgr2=0x7f1b380a97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.897+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380a97e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c3c0 tx=0x7f1b2400c880 comp rx=0 tx=0).stop 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 shutdown_connections 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b1c06b810 0x7f1b1c06dcd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b380c1be0 0x7f1b380a97e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b38152610 0x7f1b381551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 --2- 172.21.15.78:0/2372151479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b3806e350 0x7f1b380abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 >> 172.21.15.78:0/2372151479 conn(0x7f1b3806b270 msgr2=0x7f1b381503f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 shutdown_connections 2024-02-19T14:53:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:25.898+0000 7f1b3ef77640 1 -- 172.21.15.78:0/2372151479 wait complete. 2024-02-19T14:53:25.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:26.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 Processor -- start 2024-02-19T14:53:26.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 -- start start 2024-02-19T14:53:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1ad8155f90 0x7f1ad806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad80c6d30 0x7f1ad80c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80c6af0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad80c7940 con 0x7f1ad8155f90 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1addf6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80c7c20 con 0x7f1ad80c6d30 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.008+0000 7f1ad77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1ad8155f90 0x7f1ad806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1ad80c3aa0 0x7f1ad81588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60870/0 (socket says 172.21.15.78:60870) 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad7fff640 1 -- 172.21.15.78:0/3418021085 learned_addr learned my addr 172.21.15.78:0/3418021085 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad6ffd640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad80c6d30 0x7f1ad80c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3418021085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad80c6d30 msgr2=0x7f1ad80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad77fe640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad80c6d30 0x7f1ad80c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3418021085 >> v1:172.21.15.78:6789/0 conn(0x7f1ad80c3aa0 legacy=0x7f1ad81588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad67fc640 1 -- 172.21.15.78:0/3418021085 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673776526 0 0) 0x7f1ad80c6af0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3418021085 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80c7f00 con 0x7f1ad8155f90 2024-02-19T14:53:26.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.009+0000 7f1ad6ffd640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad80c6d30 0x7f1ad80c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:26.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1ad77fe640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad8155f90 0x7f1ad806d550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80097a0 tx=0x7f1ac802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdbbf3740628106 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1ad67fc640 1 -- 172.21.15.78:0/3418021085 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac802e070 con 0x7f1ad8155f90 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1ad67fc640 1 -- 172.21.15.78:0/3418021085 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac80315a0 con 0x7f1ad8155f90 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1ad67fc640 1 -- 172.21.15.78:0/3418021085 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8031930 con 0x7f1ad8155f90 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 -- 172.21.15.78:0/3418021085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad8155f90 msgr2=0x7f1ad806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad8155f90 0x7f1ad806d550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80097a0 tx=0x7f1ac802b840 comp rx=0 tx=0).stop 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 -- 172.21.15.78:0/3418021085 shutdown_connections 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad80c6d30 0x7f1ad80c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3418021085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad8155f90 0x7f1ad806d550 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 -- 172.21.15.78:0/3418021085 >> 172.21.15.78:0/3418021085 conn(0x7f1ad806b700 msgr2=0x7f1ad80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 -- 172.21.15.78:0/3418021085 shutdown_connections 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.010+0000 7f1addf6c640 1 -- 172.21.15.78:0/3418021085 wait complete. 2024-02-19T14:53:26.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.011+0000 7f1addf6c640 1 Processor -- start 2024-02-19T14:53:26.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 -- start start 2024-02-19T14:53:26.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 0x7f1ad80a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ad80c6d30 0x7f1ad806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad8155f90 0x7f1ad8152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad809e6e0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad8158be0 con 0x7f1ad80c6d30 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1addf6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad81ea8c0 con 0x7f1ad8155f90 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1ad77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 0x7f1ad80a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1ad77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 0x7f1ad80a9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40446/0 (socket says 172.21.15.78:40446) 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3034939663 learned_addr learned my addr 172.21.15.78:0/3034939663 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1ad7fff640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad8155f90 0x7f1ad8152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.012+0000 7f1ad6ffd640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ad80c6d30 0x7f1ad806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad8155f90 msgr2=0x7f1ad8152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1ad77fe640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad8155f90 0x7f1ad8152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ad80c6d30 msgr2=0x7f1ad806c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1ad77fe640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ad80c6d30 0x7f1ad806c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1ad77fe640 1 -- 172.21.15.78:0/3034939663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac8009400 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1ad77fe640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 0x7f1ad80a9b30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8034040 tx=0x7f1ac80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8032840 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac80329a0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8032d50 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad81eaba0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.013+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad81eb160 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.014+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad8074990 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.014+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1ac804e430 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.015+0000 7f1abffff640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ac406b810 0x7f1ac406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.015+0000 7f1ad6ffd640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ac406b810 0x7f1ac406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.015+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1ac80ac1e0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.015+0000 7f1ad6ffd640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ac406b810 0x7f1ac406dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8157610 tx=0x7f1acc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.019+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ac8076b60 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.194+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ad80c75a0 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.195+0000 7f1abffff640 1 -- 172.21.15.78:0/3034939663 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+17059 (secure 0 0 0) 0x7f1ac807a220 con 0x7f1ad80c3aa0 2024-02-19T14:53:26.198 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:26.198 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:10.560315+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:52:59.040775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:00.165210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/3231075223":"2024-02-20T14:53:10.560278+0000","172.21.15.78:6836/3231075223":"2024-02-20T14:53:10.560278+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ac406b810 msgr2=0x7f1ac406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ac406b810 0x7f1ac406dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8157610 tx=0x7f1acc008040 comp rx=0 tx=0).stop 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 msgr2=0x7f1ad80a9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 0x7f1ad80a9b30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8034040 tx=0x7f1ac80046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 shutdown_connections 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ac406b810 0x7f1ac406dcd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ad8155f90 0x7f1ad8152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ad80c6d30 0x7f1ad806c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 --2- 172.21.15.78:0/3034939663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ad80c3aa0 0x7f1ad80a9b30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 >> 172.21.15.78:0/3034939663 conn(0x7f1ad806b700 msgr2=0x7f1ad80aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 shutdown_connections 2024-02-19T14:53:26.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.197+0000 7f1addf6c640 1 -- 172.21.15.78:0/3034939663 wait complete. 2024-02-19T14:53:26.211 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:53:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.309+0000 7f6131f0c640 1 Processor -- start 2024-02-19T14:53:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.309+0000 7f6131f0c640 1 -- start start 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f6131f0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f6131f0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f612c158c60 0x7f612c15f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f6131f0c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c072ec0 con 0x7f612c0730a0 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f6131f0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c06fa40 con 0x7f612c158c60 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f6131f0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c06f270 con 0x7f612c152610 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f612c158c60 0x7f612c15f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f612c158c60 0x7f612c15f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40456/0 (socket says 172.21.15.78:40456) 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 -- 172.21.15.78:0/2740221646 learned_addr learned my addr 172.21.15.78:0/2740221646 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612b7fe640 1 --2- 172.21.15.78:0/2740221646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 -- 172.21.15.78:0/2740221646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 msgr2=0x7f612c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 --2- 172.21.15.78:0/2740221646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 -- 172.21.15.78:0/2740221646 >> v1:172.21.15.150:6789/0 conn(0x7f612c0730a0 legacy=0x7f612c158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.310+0000 7f612bfff640 1 -- 172.21.15.78:0/2740221646 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c06f4c0 con 0x7f612c158c60 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.311+0000 7f612bfff640 1 --2- 172.21.15.78:0/2740221646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c158c60 0x7f612c15f8a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e50 tx=0x7f612402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4404c7981f91ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.311+0000 7f612a7fc640 1 -- 172.21.15.78:0/2740221646 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612402ca70 con 0x7f612c158c60 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.311+0000 7f612a7fc640 1 -- 172.21.15.78:0/2740221646 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612402cbd0 con 0x7f612c158c60 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.311+0000 7f612a7fc640 1 -- 172.21.15.78:0/2740221646 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61240343f0 con 0x7f612c158c60 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 -- 172.21.15.78:0/2740221646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c158c60 msgr2=0x7f612c15f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2740221646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c158c60 0x7f612c15f8a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e50 tx=0x7f612402bed0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 -- 172.21.15.78:0/2740221646 shutdown_connections 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2740221646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c158c60 0x7f612c15f8a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2740221646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 -- 172.21.15.78:0/2740221646 >> 172.21.15.78:0/2740221646 conn(0x7f612c0c7590 msgr2=0x7f612c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 -- 172.21.15.78:0/2740221646 shutdown_connections 2024-02-19T14:53:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.312+0000 7f6131f0c640 1 -- 172.21.15.78:0/2740221646 wait complete. 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.313+0000 7f6131f0c640 1 Processor -- start 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.313+0000 7f6131f0c640 1 -- start start 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 0x7f612c157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c1ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f612c158c60 0x7f612c1eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c1eb340 con 0x7f612c0730a0 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c1e1d40 con 0x7f612c158c60 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c1ee8e0 con 0x7f612c152610 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 0x7f612c157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 0x7f612c157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40464/0 (socket says 172.21.15.78:40464) 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 -- 172.21.15.78:0/2433442094 learned_addr learned my addr 172.21.15.78:0/2433442094 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612bfff640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f612c158c60 0x7f612c1eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612affd640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c1ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 -- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 msgr2=0x7f612c1ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c1ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 -- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f612c158c60 msgr2=0x7f612c1eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f612c158c60 0x7f612c1eae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 -- 172.21.15.78:0/2433442094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124009b30 con 0x7f612c0730a0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612bfff640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f612c158c60 0x7f612c1eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f612b7fe640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 0x7f612c157b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f611c00e9b0 tx=0x7f611c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c009670 con 0x7f612c0730a0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c00c9a0 con 0x7f612c0730a0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00cd50 con 0x7f612c0730a0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.314+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c0a75a0 con 0x7f612c0730a0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.315+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c158060 con 0x7f612c0730a0 2024-02-19T14:53:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.315+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c0ad680 con 0x7f612c0730a0 2024-02-19T14:53:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.316+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f611c013470 con 0x7f612c0730a0 2024-02-19T14:53:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.316+0000 7f6113fff640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f611806b810 0x7f611806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.316+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f611c08a570 con 0x7f612c0730a0 2024-02-19T14:53:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.316+0000 7f612affd640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f611806b810 0x7f611806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.317+0000 7f612affd640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f611806b810 0x7f611806dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f612c1dfb20 tx=0x7f6120009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.321+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f611c054e40 con 0x7f612c0730a0 2024-02-19T14:53:26.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.528+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f612c074990 con 0x7f612c0730a0 2024-02-19T14:53:26.531 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:53:26.529+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:53:26.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.530+0000 7f6113fff640 1 -- 172.21.15.78:0/2433442094 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7f611c058500 con 0x7f612c0730a0 2024-02-19T14:53:26.532 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:53:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.532+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f611806b810 msgr2=0x7f611806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.532+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f611806b810 0x7f611806dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f612c1dfb20 tx=0x7f6120009040 comp rx=0 tx=0).stop 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.532+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 msgr2=0x7f612c157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.532+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 0x7f612c157b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f611c00e9b0 tx=0x7f611c00ee70 comp rx=0 tx=0).stop 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 shutdown_connections 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f611806b810 0x7f611806dcd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f612c158c60 0x7f612c1eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f612c152610 0x7f612c1ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 --2- 172.21.15.78:0/2433442094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f612c0730a0 0x7f612c157b20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 >> 172.21.15.78:0/2433442094 conn(0x7f612c0c7590 msgr2=0x7f612c0c1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 shutdown_connections 2024-02-19T14:53:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.533+0000 7f6131f0c640 1 -- 172.21.15.78:0/2433442094 wait complete. 2024-02-19T14:53:26.537 INFO:tasks.ceph.mds.d.smithi150.stderr: -478> 2024-02-19T14:53:15.258+0000 7efcf809b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:26.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:53:26.560 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:53:26.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.647+0000 7f7daf89a640 1 Processor -- start 2024-02-19T14:53:26.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.648+0000 7f7daf89a640 1 -- start start 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.648+0000 7f7daf89a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80c64f0 0x7f7da80ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.648+0000 7f7daf89a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da8152d30 con 0x7f7da80c49c0 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.648+0000 7f7daf89a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da80c7860 con 0x7f7da80ad890 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.648+0000 7f7daf89a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da80c7ce0 con 0x7f7da80c64f0 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7dace35640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7da80c49c0 0x7f7da80c5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60886/0 (socket says 172.21.15.78:60886) 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7dace35640 1 -- 172.21.15.78:0/2885698592 learned_addr learned my addr 172.21.15.78:0/2885698592 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7dade37640 1 --2- 172.21.15.78:0/2885698592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80c64f0 0x7f7da80ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005802372 0 0) 0x7f7da8152d30 con 0x7f7da80c49c0 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d94003520 con 0x7f7da80c49c0 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253024248 0 0) 0x7f7da80c7860 con 0x7f7da80ad890 2024-02-19T14:53:26.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da8152d30 con 0x7f7da80ad890 2024-02-19T14:53:26.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7dade37640 1 -- 172.21.15.78:0/2885698592 >> v1:172.21.15.150:6789/0 conn(0x7f7da80ad890 legacy=0x7f7da81534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7dade37640 1 -- 172.21.15.78:0/2885698592 >> v1:172.21.15.78:6789/0 conn(0x7f7da80c49c0 legacy=0x7f7da80c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.649+0000 7f7dade37640 1 -- 172.21.15.78:0/2885698592 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da80c7f80 con 0x7f7da80c64f0 2024-02-19T14:53:26.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7dade37640 1 --2- 172.21.15.78:0/2885698592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80c64f0 0x7f7da80ab600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009dc0 tx=0x7f7da402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bc52dd78d7536fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da402cbb0 con 0x7f7da80c64f0 2024-02-19T14:53:26.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da402cd10 con 0x7f7da80c64f0 2024-02-19T14:53:26.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7d9ffff640 1 -- 172.21.15.78:0/2885698592 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4034660 con 0x7f7da80c64f0 2024-02-19T14:53:26.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7daf89a640 1 -- 172.21.15.78:0/2885698592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80c64f0 msgr2=0x7f7da80ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7daf89a640 1 --2- 172.21.15.78:0/2885698592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80c64f0 0x7f7da80ab600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009dc0 tx=0x7f7da402be00 comp rx=0 tx=0).stop 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7daf89a640 1 -- 172.21.15.78:0/2885698592 shutdown_connections 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7daf89a640 1 --2- 172.21.15.78:0/2885698592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80c64f0 0x7f7da80ab600 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.650+0000 7f7daf89a640 1 -- 172.21.15.78:0/2885698592 >> 172.21.15.78:0/2885698592 conn(0x7f7da806b700 msgr2=0x7f7da80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.651+0000 7f7daf89a640 1 -- 172.21.15.78:0/2885698592 shutdown_connections 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.651+0000 7f7daf89a640 1 -- 172.21.15.78:0/2885698592 wait complete. 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 Processor -- start 2024-02-19T14:53:26.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 -- start start 2024-02-19T14:53:26.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80ad890 0x7f7da80c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 0x7f7da806c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da80c64f0 0x7f7da806cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da806bf90 con 0x7f7da80c49c0 2024-02-19T14:53:26.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da806c270 con 0x7f7da80c64f0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7daf89a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da814bea0 con 0x7f7da80ad890 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7dace35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 0x7f7da806c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7dace35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 0x7f7da806c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40470/0 (socket says 172.21.15.78:40470) 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.652+0000 7f7dace35640 1 -- 172.21.15.78:0/164661495 learned_addr learned my addr 172.21.15.78:0/164661495 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dad636640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80ad890 0x7f7da80c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dade37640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da80c64f0 0x7f7da806cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dace35640 1 -- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80ad890 msgr2=0x7f7da80c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dace35640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80ad890 0x7f7da80c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dace35640 1 -- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da80c64f0 msgr2=0x7f7da806cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dace35640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da80c64f0 0x7f7da806cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dace35640 1 -- 172.21.15.78:0/164661495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4009a70 con 0x7f7da80c49c0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7dace35640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 0x7f7da806c700 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c4c0 tx=0x7f7d9800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98005ce0 con 0x7f7da80c49c0 2024-02-19T14:53:26.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98005e40 con 0x7f7da80c49c0 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da81eb360 con 0x7f7da80c49c0 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98007270 con 0x7f7da80c49c0 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.653+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da80c2130 con 0x7f7da80c49c0 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.654+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da8074990 con 0x7f7da80c49c0 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.655+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7d980073d0 con 0x7f7da80c49c0 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.655+0000 7f7d9dffb640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d8c06b810 0x7f7d8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:26.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.655+0000 7f7dad636640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d8c06b810 0x7f7d8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:26.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.655+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f7d98089810 con 0x7f7da80c49c0 2024-02-19T14:53:26.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.656+0000 7f7dad636640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d8c06b810 0x7f7d8c06dcd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7da0007fd0 tx=0x7f7da0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:26.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.659+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7d98054130 con 0x7f7da80c49c0 2024-02-19T14:53:26.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:26.870+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7da8068120 con 0x7f7da80c49c0 2024-02-19T14:53:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.531+0000 7f7d9dffb640 1 -- 172.21.15.78:0/164661495 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7d980577f0 con 0x7f7da80c49c0 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d8c06b810 msgr2=0x7f7d8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d8c06b810 0x7f7d8c06dcd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7da0007fd0 tx=0x7f7da0009040 comp rx=0 tx=0).stop 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 msgr2=0x7f7da806c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 0x7f7da806c700 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c4c0 tx=0x7f7d9800c980 comp rx=0 tx=0).stop 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 shutdown_connections 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d8c06b810 0x7f7d8c06dcd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da80c49c0 0x7f7da806c700 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da80c64f0 0x7f7da806cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 --2- 172.21.15.78:0/164661495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7da80ad890 0x7f7da80c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 >> 172.21.15.78:0/164661495 conn(0x7f7da806b700 msgr2=0x7f7da81df430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 shutdown_connections 2024-02-19T14:53:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.534+0000 7f7daf89a640 1 -- 172.21.15.78:0/164661495 wait complete. 2024-02-19T14:53:27.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:53:27.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.658+0000 7f9ddc039640 1 Processor -- start 2024-02-19T14:53:27.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 -- start start 2024-02-19T14:53:27.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9dd40c7cd0 0x7f9dd40a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd4152240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9dd4152780 0x7f9dd40727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd4072390 con 0x7f9dd4152780 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd4155df0 con 0x7f9dd40c7cd0 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9ddc039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40c0b90 con 0x7f9dd40a8530 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.659+0000 7f9dda5d6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9dd4152780 0x7f9dd40727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9dd4152780 0x7f9dd40727e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55752/0 (socket says 172.21.15.78:55752) 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 -- 172.21.15.78:0/3390400735 learned_addr learned my addr 172.21.15.78:0/3390400735 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dd95d4640 1 --2- 172.21.15.78:0/3390400735 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd4152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dd9dd5640 1 --2- 172.21.15.78:0/3390400735 >> v2:172.21.15.150:3300/0 conn(0x7f9dd40c7cd0 0x7f9dd40a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 -- 172.21.15.78:0/3390400735 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 msgr2=0x7f9dd4152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 --2- 172.21.15.78:0/3390400735 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd4152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 -- 172.21.15.78:0/3390400735 >> v2:172.21.15.150:3300/0 conn(0x7f9dd40c7cd0 msgr2=0x7f9dd40a9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 --2- 172.21.15.78:0/3390400735 >> v2:172.21.15.150:3300/0 conn(0x7f9dd40c7cd0 0x7f9dd40a9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.660+0000 7f9dda5d6640 1 -- 172.21.15.78:0/3390400735 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40c0e70 con 0x7f9dd4152780 2024-02-19T14:53:27.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9dda5d6640 1 --2- 172.21.15.78:0/3390400735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40727e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40c50d0 tx=0x7f9dcc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181f8ce31e373d03 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9dd8dd3640 1 -- 172.21.15.78:0/3390400735 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc02ca70 con 0x7f9dd4152780 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9dd8dd3640 1 -- 172.21.15.78:0/3390400735 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc02cbd0 con 0x7f9dd4152780 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9dd8dd3640 1 -- 172.21.15.78:0/3390400735 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc0343f0 con 0x7f9dd4152780 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 -- 172.21.15.78:0/3390400735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 msgr2=0x7f9dd40727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 --2- 172.21.15.78:0/3390400735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40727e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40c50d0 tx=0x7f9dcc02bed0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 -- 172.21.15.78:0/3390400735 shutdown_connections 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 --2- 172.21.15.78:0/3390400735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40727e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 --2- 172.21.15.78:0/3390400735 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd4152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 --2- 172.21.15.78:0/3390400735 >> v2:172.21.15.150:3300/0 conn(0x7f9dd40c7cd0 0x7f9dd40a9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 -- 172.21.15.78:0/3390400735 >> 172.21.15.78:0/3390400735 conn(0x7f9dd406f5e0 msgr2=0x7f9dd406f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 -- 172.21.15.78:0/3390400735 shutdown_connections 2024-02-19T14:53:27.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.661+0000 7f9ddc039640 1 -- 172.21.15.78:0/3390400735 wait complete. 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.662+0000 7f9ddc039640 1 Processor -- start 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.662+0000 7f9ddc039640 1 -- start start 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9ddc039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd40c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9ddc039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd40c7cd0 0x7f9dd41eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9ddc039640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9ddc039640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40c4ae0 con 0x7f9dd4152780 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9ddc039640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40c2400 con 0x7f9dd40c7cd0 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9ddc039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40ac410 con 0x7f9dd40a8530 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40c66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55756/0 (socket says 172.21.15.78:55756) 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 -- 172.21.15.78:0/2494254931 learned_addr learned my addr 172.21.15.78:0/2494254931 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dd9dd5640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd40c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dd95d4640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd40c7cd0 0x7f9dd41eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 -- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 msgr2=0x7f9dd40c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd40c1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 -- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd40c7cd0 msgr2=0x7f9dd41eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd40c7cd0 0x7f9dd41eaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.663+0000 7f9dda5d6640 1 -- 172.21.15.78:0/2494254931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc009b30 con 0x7f9dd4152780 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.664+0000 7f9dda5d6640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40c66d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc036040 tx=0x7f9dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.664+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc009510 con 0x7f9dd4152780 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.664+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc03e450 con 0x7f9dd4152780 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.664+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd41eb430 con 0x7f9dd4152780 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.664+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc03e800 con 0x7f9dd4152780 2024-02-19T14:53:27.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.664+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd40ac990 con 0x7f9dd4152780 2024-02-19T14:53:27.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.665+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd40acb80 con 0x7f9dd4152780 2024-02-19T14:53:27.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.665+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9dcc03e960 con 0x7f9dd4152780 2024-02-19T14:53:27.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.666+0000 7f9dca7fc640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9db806b810 0x7f9db806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:27.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.666+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f9dcc0ad100 con 0x7f9dd4152780 2024-02-19T14:53:27.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.666+0000 7f9dd9dd5640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9db806b810 0x7f9db806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:27.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.666+0000 7f9dd9dd5640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9db806b810 0x7f9db806dcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4006fd0 tx=0x7f9dc4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:27.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.670+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9dcc077970 con 0x7f9dd4152780 2024-02-19T14:53:27.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:27.842+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9dd4074990 con 0x7f9dd4152780 2024-02-19T14:53:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.618+0000 7f9dca7fc640 1 -- 172.21.15.78:0/2494254931 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9dcc07b030 con 0x7f9dd4152780 2024-02-19T14:53:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:53:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.621+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9db806b810 msgr2=0x7f9db806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.621+0000 7f9ddc039640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9db806b810 0x7f9db806dcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4006fd0 tx=0x7f9dc4008040 comp rx=0 tx=0).stop 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 msgr2=0x7f9dd40c66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40c66d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc036040 tx=0x7f9dcc004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 shutdown_connections 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9db806b810 0x7f9db806dcd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd4152780 0x7f9dd40c66d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd40c7cd0 0x7f9dd41eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 --2- 172.21.15.78:0/2494254931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd40a8530 0x7f9dd40c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 >> 172.21.15.78:0/2494254931 conn(0x7f9dd406f5e0 msgr2=0x7f9dd40c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 shutdown_connections 2024-02-19T14:53:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.622+0000 7f9ddc039640 1 -- 172.21.15.78:0/2494254931 wait complete. 2024-02-19T14:53:28.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:53:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.744+0000 7fd2c9df7640 1 Processor -- start 2024-02-19T14:53:28.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.744+0000 7fd2c9df7640 1 -- start start 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c9df7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c4152620 0x7fd2c40c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c9df7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2c4155010 0x7fd2c41554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c9df7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4159210 con 0x7fd2c40c6d50 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c9df7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c40c1f00 con 0x7fd2c4155010 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c9df7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40c6420 con 0x7fd2c4152620 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2c4155010 0x7fd2c41554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2c4155010 0x7fd2c41554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55762/0 (socket says 172.21.15.78:55762) 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c3fff640 1 -- 172.21.15.78:0/637133654 learned_addr learned my addr 172.21.15.78:0/637133654 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c37fe640 1 --2- 172.21.15.78:0/637133654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c4152620 0x7fd2c40c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:28.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c3fff640 1 -- 172.21.15.78:0/637133654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c4152620 msgr2=0x7fd2c40c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.745+0000 7fd2c3fff640 1 --2- 172.21.15.78:0/637133654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c4152620 0x7fd2c40c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.746+0000 7fd2c3fff640 1 -- 172.21.15.78:0/637133654 >> v1:172.21.15.150:6789/0 conn(0x7fd2c40c6d50 legacy=0x7fd2c406d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.746+0000 7fd2c3fff640 1 -- 172.21.15.78:0/637133654 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c4158600 con 0x7fd2c4155010 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.746+0000 7fd2c3fff640 1 --2- 172.21.15.78:0/637133654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4155010 0x7fd2c41554a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009870 tx=0x7fd2bc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96933c555e719736 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.746+0000 7fd2c27fc640 1 -- 172.21.15.78:0/637133654 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc0317c0 con 0x7fd2c4155010 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.746+0000 7fd2c27fc640 1 -- 172.21.15.78:0/637133654 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2bc031920 con 0x7fd2c4155010 2024-02-19T14:53:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.746+0000 7fd2c27fc640 1 -- 172.21.15.78:0/637133654 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc031cb0 con 0x7fd2c4155010 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 -- 172.21.15.78:0/637133654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4155010 msgr2=0x7fd2c41554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/637133654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4155010 0x7fd2c41554a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009870 tx=0x7fd2bc02b860 comp rx=0 tx=0).stop 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 -- 172.21.15.78:0/637133654 shutdown_connections 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/637133654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4155010 0x7fd2c41554a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/637133654 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c4152620 0x7fd2c40c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 -- 172.21.15.78:0/637133654 >> 172.21.15.78:0/637133654 conn(0x7fd2c40caae0 msgr2=0x7fd2c40caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 -- 172.21.15.78:0/637133654 shutdown_connections 2024-02-19T14:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.747+0000 7fd2c9df7640 1 -- 172.21.15.78:0/637133654 wait complete. 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.748+0000 7fd2c9df7640 1 Processor -- start 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.748+0000 7fd2c9df7640 1 -- start start 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c9df7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c40c6d50 0x7fd2c4156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c9df7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 0x7fd2c4149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c9df7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c4155010 0x7fd2c4152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c9df7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41567d0 con 0x7fd2c4152620 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c9df7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41ee810 con 0x7fd2c4155010 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c9df7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4154ad0 con 0x7fd2c40c6d50 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 0x7fd2c4149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 0x7fd2c4149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55764/0 (socket says 172.21.15.78:55764) 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 -- 172.21.15.78:0/2386973004 learned_addr learned my addr 172.21.15.78:0/2386973004 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c37fe640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c40c6d50 0x7fd2c4156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c3fff640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c4155010 0x7fd2c4152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 -- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c40c6d50 msgr2=0x7fd2c4156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c40c6d50 0x7fd2c4156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 -- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c4155010 msgr2=0x7fd2c4152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c4155010 0x7fd2c4152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 -- 172.21.15.78:0/2386973004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0094d0 con 0x7fd2c4152620 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c3fff640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c4155010 0x7fd2c4152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.749+0000 7fd2c2ffd640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 0x7fd2c4149d40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800c3c0 tx=0x7fd2b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.750+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b80052d0 con 0x7fd2c4152620 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.750+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8005450 con 0x7fd2c4152620 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.750+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8007260 con 0x7fd2c4152620 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.750+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c41eeab0 con 0x7fd2c4152620 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.750+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c414a280 con 0x7fd2c4152620 2024-02-19T14:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.751+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c41eeca0 con 0x7fd2c4152620 2024-02-19T14:53:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.751+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd2b80073c0 con 0x7fd2c4152620 2024-02-19T14:53:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.752+0000 7fd2abfff640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd2b006b810 0x7fd2b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.752+0000 7fd2c37fe640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd2b006b810 0x7fd2b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.752+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fd2b8089580 con 0x7fd2c4152620 2024-02-19T14:53:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.752+0000 7fd2c37fe640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd2b006b810 0x7fd2b006dcd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40bfcb0 tx=0x7fd2b400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:28.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.756+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd2b8053fb0 con 0x7fd2c4152620 2024-02-19T14:53:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:28.927+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2c4074990 con 0x7fd2c4152620 2024-02-19T14:53:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.620+0000 7fd2abfff640 1 -- 172.21.15.78:0/2386973004 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd2b8057670 con 0x7fd2c4152620 2024-02-19T14:53:29.623 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd2b006b810 msgr2=0x7fd2b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd2b006b810 0x7fd2b006dcd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40bfcb0 tx=0x7fd2b400b040 comp rx=0 tx=0).stop 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 msgr2=0x7fd2c4149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 0x7fd2c4149d40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd2b800c3c0 tx=0x7fd2b800c880 comp rx=0 tx=0).stop 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 shutdown_connections 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd2b006b810 0x7fd2b006dcd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2c4152620 0x7fd2c4149d40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c4155010 0x7fd2c4152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 --2- 172.21.15.78:0/2386973004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2c40c6d50 0x7fd2c4156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 >> 172.21.15.78:0/2386973004 conn(0x7fd2c40caae0 msgr2=0x7fd2c40c4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:29.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.623+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 shutdown_connections 2024-02-19T14:53:29.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.624+0000 7fd2c9df7640 1 -- 172.21.15.78:0/2386973004 wait complete. 2024-02-19T14:53:29.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:53:29.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.743+0000 7f6a99bd0640 1 Processor -- start 2024-02-19T14:53:29.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.743+0000 7f6a99bd0640 1 -- start start 2024-02-19T14:53:29.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a99bd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a94153370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:29.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a99bd0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a940729b0 0x7f6a941538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:29.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a99bd0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a941572b0 con 0x7f6a94150220 2024-02-19T14:53:29.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a99bd0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940c37b0 con 0x7f6a940729b0 2024-02-19T14:53:29.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a99bd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940c31f0 con 0x7f6a94156d40 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a93fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a940729b0 0x7f6a941538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a93fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a940729b0 0x7f6a941538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55774/0 (socket says 172.21.15.78:55774) 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a937fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6a94150220 0x7f6a9406be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59124/0 (socket says 172.21.15.78:59124) 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.744+0000 7f6a93fff640 1 -- 172.21.15.78:0/307822281 learned_addr learned my addr 172.21.15.78:0/307822281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a8bfff640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a94153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a93fff640 1 -- 172.21.15.78:0/307822281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 msgr2=0x7f6a94153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a93fff640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a94153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a92ffd640 1 -- 172.21.15.78:0/307822281 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327794416 0 0) 0x7f6a941572b0 con 0x7f6a94150220 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a93fff640 1 -- 172.21.15.78:0/307822281 >> v1:172.21.15.78:6789/0 conn(0x7f6a94150220 legacy=0x7f6a9406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a93fff640 1 -- 172.21.15.78:0/307822281 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940c34d0 con 0x7f6a940729b0 2024-02-19T14:53:29.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a8bfff640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a94153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.745+0000 7f6a93fff640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940729b0 0x7f6a941538b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009e50 tx=0x7f6a8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f9e14a0c84ba1b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a92ffd640 1 -- 172.21.15.78:0/307822281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c02ca70 con 0x7f6a940729b0 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a92ffd640 1 -- 172.21.15.78:0/307822281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c02cbd0 con 0x7f6a940729b0 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a92ffd640 1 -- 172.21.15.78:0/307822281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0343f0 con 0x7f6a940729b0 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 -- 172.21.15.78:0/307822281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940729b0 msgr2=0x7f6a941538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940729b0 0x7f6a941538b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c009e50 tx=0x7f6a8c02bed0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 -- 172.21.15.78:0/307822281 shutdown_connections 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940729b0 0x7f6a941538b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/307822281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a94153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 -- 172.21.15.78:0/307822281 >> 172.21.15.78:0/307822281 conn(0x7f6a94071920 msgr2=0x7f6a94071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:29.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 -- 172.21.15.78:0/307822281 shutdown_connections 2024-02-19T14:53:29.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.746+0000 7f6a99bd0640 1 -- 172.21.15.78:0/307822281 wait complete. 2024-02-19T14:53:29.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.747+0000 7f6a99bd0640 1 Processor -- start 2024-02-19T14:53:29.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.747+0000 7f6a99bd0640 1 -- start start 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a99bd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a940729b0 0x7f6a94147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a99bd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 0x7f6a9414b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a99bd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a9414aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a99bd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a94152af0 con 0x7f6a94150220 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a99bd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a941481a0 con 0x7f6a940729b0 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a99bd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a94152530 con 0x7f6a94156d40 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 0x7f6a9414b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 0x7f6a9414b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55790/0 (socket says 172.21.15.78:55790) 2024-02-19T14:53:29.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 -- 172.21.15.78:0/1174671603 learned_addr learned my addr 172.21.15.78:0/1174671603 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a93fff640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a9414aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a937fe640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a940729b0 0x7f6a94147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 -- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 msgr2=0x7f6a9414aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a9414aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 -- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a940729b0 msgr2=0x7f6a94147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a940729b0 0x7f6a94147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 -- 172.21.15.78:0/1174671603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c009b30 con 0x7f6a94150220 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a93fff640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a9414aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a937fe640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a940729b0 0x7f6a94147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.748+0000 7f6a8bfff640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 0x7f6a9414b9d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c7f0 tx=0x7f6a7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.749+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c007820 con 0x7f6a94150220 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.749+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c007980 con 0x7f6a94150220 2024-02-19T14:53:29.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.749+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a94152810 con 0x7f6a94150220 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.749+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c007cf0 con 0x7f6a94150220 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.749+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a94152230 con 0x7f6a94150220 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.750+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a94074990 con 0x7f6a94150220 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.750+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6a7c0113f0 con 0x7f6a94150220 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.750+0000 7f6a90ff9640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a7406b840 0x7f6a7406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.750+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f6a7c08a450 con 0x7f6a94150220 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.750+0000 7f6a937fe640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a7406b840 0x7f6a7406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:29.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.751+0000 7f6a937fe640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a7406b840 0x7f6a7406dd00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6a84000f60 tx=0x7f6a84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:29.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.755+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6a7c054f00 con 0x7f6a94150220 2024-02-19T14:53:29.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.926+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f6a94156030 con 0x7f6a94150220 2024-02-19T14:53:29.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.927+0000 7f6a90ff9640 1 -- 172.21.15.78:0/1174671603 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7f6a7c0585c0 con 0x7f6a94150220 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a7406b840 msgr2=0x7f6a7406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a7406b840 0x7f6a7406dd00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6a84000f60 tx=0x7f6a84009040 comp rx=0 tx=0).stop 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 msgr2=0x7f6a9414b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 0x7f6a9414b9d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c7f0 tx=0x7f6a7c00ccb0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 shutdown_connections 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a7406b840 0x7f6a7406dd00 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6a84000f60 tx=0x7f6a84009040 comp rx=0 tx=0).stop 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a94150220 0x7f6a9414b9d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a94156d40 0x7f6a9414aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 --2- 172.21.15.78:0/1174671603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a940729b0 0x7f6a94147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 >> 172.21.15.78:0/1174671603 conn(0x7f6a94071920 msgr2=0x7f6a940abed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:29.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 shutdown_connections 2024-02-19T14:53:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:29.929+0000 7f6a99bd0640 1 -- 172.21.15.78:0/1174671603 wait complete. 2024-02-19T14:53:29.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:53:30.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.042+0000 7f4e05ab5640 1 Processor -- start 2024-02-19T14:53:30.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.042+0000 7f4e05ab5640 1 -- start start 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4e05ab5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e000c3200 0x7f4e000ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4e05ab5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 0x7f4e0006ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4e05ab5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e000c35e0 con 0x7f4e00153b60 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4e05ab5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0014d3c0 con 0x7f4e000c3200 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4e05ab5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000acad0 con 0x7f4e000c4350 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 0x7f4e0006ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dff7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e000c3200 0x7f4e000ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfeffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4e00153b60 0x7f4e000718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:34394/0 (socket says 172.21.15.78:34394) 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfeffd640 1 -- 172.21.15.78:0/1471748831 learned_addr learned my addr 172.21.15.78:0/1471748831 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 0x7f4e0006ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:42294/0 (socket says 172.21.15.78:42294) 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfe7fc640 1 -- 172.21.15.78:0/1471748831 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727957865 0 0) 0x7f4e000c35e0 con 0x7f4e00153b60 2024-02-19T14:53:30.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfe7fc640 1 -- 172.21.15.78:0/1471748831 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de8003520 con 0x7f4e00153b60 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfffff640 1 -- 172.21.15.78:0/1471748831 >> v2:172.21.15.150:3300/0 conn(0x7f4e000c3200 msgr2=0x7f4e000ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfffff640 1 --2- 172.21.15.78:0/1471748831 >> v2:172.21.15.150:3300/0 conn(0x7f4e000c3200 0x7f4e000ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfffff640 1 -- 172.21.15.78:0/1471748831 >> v1:172.21.15.150:6789/0 conn(0x7f4e00153b60 legacy=0x7f4e000718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dfffff640 1 -- 172.21.15.78:0/1471748831 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000ad000 con 0x7f4e000c4350 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.043+0000 7f4dff7fe640 1 --2- 172.21.15.78:0/1471748831 >> v2:172.21.15.150:3300/0 conn(0x7f4e000c3200 0x7f4e000ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.044+0000 7f4dfffff640 1 --2- 172.21.15.78:0/1471748831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 0x7f4e0006ae80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009d90 tx=0x7f4df802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82e8d018fa057a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.044+0000 7f4dfe7fc640 1 -- 172.21.15.78:0/1471748831 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df802cc50 con 0x7f4e000c4350 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.044+0000 7f4dfe7fc640 1 -- 172.21.15.78:0/1471748831 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df802cdb0 con 0x7f4e000c4350 2024-02-19T14:53:30.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.044+0000 7f4dfe7fc640 1 -- 172.21.15.78:0/1471748831 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8034660 con 0x7f4e000c4350 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 -- 172.21.15.78:0/1471748831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 msgr2=0x7f4e0006ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/1471748831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 0x7f4e0006ae80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009d90 tx=0x7f4df802bea0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 -- 172.21.15.78:0/1471748831 shutdown_connections 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/1471748831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c4350 0x7f4e0006ae80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/1471748831 >> v2:172.21.15.150:3300/0 conn(0x7f4e000c3200 0x7f4e000ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 -- 172.21.15.78:0/1471748831 >> 172.21.15.78:0/1471748831 conn(0x7f4e000c7800 msgr2=0x7f4e000c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 -- 172.21.15.78:0/1471748831 shutdown_connections 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.045+0000 7f4e05ab5640 1 -- 172.21.15.78:0/1471748831 wait complete. 2024-02-19T14:53:30.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.046+0000 7f4e05ab5640 1 Processor -- start 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.046+0000 7f4e05ab5640 1 -- start start 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4e05ab5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c3200 0x7f4e00156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4e05ab5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e000c4350 0x7f4e0013a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4e05ab5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 0x7f4e0013a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4e05ab5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00155fe0 con 0x7f4e00153b60 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4e05ab5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0014fb70 con 0x7f4e000c4350 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4e05ab5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0013adb0 con 0x7f4e000c3200 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 0x7f4e0013a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 0x7f4e0013a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55804/0 (socket says 172.21.15.78:55804) 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 -- 172.21.15.78:0/2944420285 learned_addr learned my addr 172.21.15.78:0/2944420285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dff7fe640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c3200 0x7f4e00156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfeffd640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e000c4350 0x7f4e0013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 -- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c3200 msgr2=0x7f4e00156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c3200 0x7f4e00156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 -- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e000c4350 msgr2=0x7f4e0013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e000c4350 0x7f4e0013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 -- 172.21.15.78:0/2944420285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0009400 con 0x7f4e00153b60 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.047+0000 7f4dfffff640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 0x7f4e0013a870 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009d90 tx=0x7f4df8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.048+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8040650 con 0x7f4e00153b60 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.048+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df80407b0 con 0x7f4e00153b60 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.048+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8040b60 con 0x7f4e00153b60 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.048+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df8009a70 con 0x7f4e00153b60 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.048+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e001e97c0 con 0x7f4e00153b60 2024-02-19T14:53:30.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.049+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e001e99b0 con 0x7f4e00153b60 2024-02-19T14:53:30.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.049+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4df8040cc0 con 0x7f4e00153b60 2024-02-19T14:53:30.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.049+0000 7f4de7fff640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dec06b840 0x7f4dec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.049+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4df80aca50 con 0x7f4e00153b60 2024-02-19T14:53:30.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.049+0000 7f4dff7fe640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dec06b840 0x7f4dec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.050+0000 7f4dff7fe640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dec06b840 0x7f4dec06dd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009c60 tx=0x7f4df0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.054+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4df8077620 con 0x7f4e00153b60 2024-02-19T14:53:30.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.233+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4e00074990 con 0x7f4e00153b60 2024-02-19T14:53:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.236+0000 7f4de7fff640 1 -- 172.21.15.78:0/2944420285 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4df8009450 con 0x7f4e00153b60 2024-02-19T14:53:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dec06b840 msgr2=0x7f4dec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dec06b840 0x7f4dec06dd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009c60 tx=0x7f4df0008040 comp rx=0 tx=0).stop 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 msgr2=0x7f4e0013a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 0x7f4e0013a870 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009d90 tx=0x7f4df8004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 shutdown_connections 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dec06b840 0x7f4dec06dd00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e00153b60 0x7f4e0013a870 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e000c4350 0x7f4e0013a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.238+0000 7f4e05ab5640 1 --2- 172.21.15.78:0/2944420285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e000c3200 0x7f4e00156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.239+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 >> 172.21.15.78:0/2944420285 conn(0x7f4e000c7800 msgr2=0x7f4e000718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.239+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 shutdown_connections 2024-02-19T14:53:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.239+0000 7f4e05ab5640 1 -- 172.21.15.78:0/2944420285 wait complete. 2024-02-19T14:53:30.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-02-19T14:53:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 Processor -- start 2024-02-19T14:53:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 -- start start 2024-02-19T14:53:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff62806d550 0x7ff6280c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff628158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280c6af0 con 0x7ff628155f90 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6280c7270 con 0x7ff62806d550 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.350+0000 7ff62ed0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280c7940 con 0x7ff6281588a0 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff627fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff62806d550 0x7ff6280c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff62d2aa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff628158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff62caa9640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff628155f90 0x7ff6280c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:34400/0 (socket says 172.21.15.78:34400) 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff62caa9640 1 -- 172.21.15.78:0/638563388 learned_addr learned my addr 172.21.15.78:0/638563388 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff627fff640 1 -- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 msgr2=0x7ff628158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff627fff640 1 --2- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff628158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff627fff640 1 -- 172.21.15.78:0/638563388 >> v1:172.21.15.150:6789/0 conn(0x7ff628155f90 legacy=0x7ff6280c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff627fff640 1 -- 172.21.15.78:0/638563388 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280c7c20 con 0x7ff62806d550 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff6277fe640 1 -- 172.21.15.78:0/638563388 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156477648 0 0) 0x7ff6280c6af0 con 0x7ff628155f90 2024-02-19T14:53:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.351+0000 7ff62d2aa640 1 --2- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff628158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:30.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.352+0000 7ff627fff640 1 --2- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806d550 0x7ff6280c6d30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff6280c73b0 tx=0x7ff61402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=631dba615cf930f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.352+0000 7ff6277fe640 1 -- 172.21.15.78:0/638563388 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614031770 con 0x7ff62806d550 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.352+0000 7ff6277fe640 1 -- 172.21.15.78:0/638563388 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6140318d0 con 0x7ff62806d550 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.352+0000 7ff6277fe640 1 -- 172.21.15.78:0/638563388 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614031c60 con 0x7ff62806d550 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 -- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806d550 msgr2=0x7ff6280c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806d550 0x7ff6280c6d30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff6280c73b0 tx=0x7ff61402b810 comp rx=0 tx=0).stop 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 -- 172.21.15.78:0/638563388 shutdown_connections 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff628158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/638563388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806d550 0x7ff6280c6d30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 -- 172.21.15.78:0/638563388 >> 172.21.15.78:0/638563388 conn(0x7ff62806b700 msgr2=0x7ff6280aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 -- 172.21.15.78:0/638563388 shutdown_connections 2024-02-19T14:53:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.353+0000 7ff62ed0d640 1 -- 172.21.15.78:0/638563388 wait complete. 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.354+0000 7ff62ed0d640 1 Processor -- start 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.354+0000 7ff62ed0d640 1 -- start start 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 0x7ff628157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff628155f90 0x7ff628151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff6281e0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281586a0 con 0x7ff62806d550 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280ad9f0 con 0x7ff628155f90 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6281ea410 con 0x7ff6281588a0 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 0x7ff628157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 0x7ff628157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55816/0 (socket says 172.21.15.78:55816) 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 -- 172.21.15.78:0/2191467061 learned_addr learned my addr 172.21.15.78:0/2191467061 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff627fff640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff628155f90 0x7ff628151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 -- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 msgr2=0x7ff6281e0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62d2aa640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff6281e0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff6281e0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 -- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff628155f90 msgr2=0x7ff628151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff628155f90 0x7ff628151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 -- 172.21.15.78:0/2191467061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620009a70 con 0x7ff62806d550 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62caa9640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 0x7ff628157f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00c4a0 tx=0x7ff61c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c005ce0 con 0x7ff62806d550 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c005e40 con 0x7ff62806d550 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff614009400 con 0x7ff62806d550 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c0072b0 con 0x7ff62806d550 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.355+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6281df540 con 0x7ff62806d550 2024-02-19T14:53:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.356+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff628074990 con 0x7ff62806d550 2024-02-19T14:53:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.357+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff61c007410 con 0x7ff62806d550 2024-02-19T14:53:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.357+0000 7ff6257fa640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff60c06b810 0x7ff60c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.357+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff61c089630 con 0x7ff62806d550 2024-02-19T14:53:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.357+0000 7ff627fff640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff60c06b810 0x7ff60c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.358+0000 7ff627fff640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff60c06b810 0x7ff60c06dcd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff614036000 tx=0x7ff61403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.362+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff61c054200 con 0x7ff62806d550 2024-02-19T14:53:30.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.599+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7ff6280c75a0 con 0x7ff62806d550 2024-02-19T14:53:30.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.630+0000 7ff6257fa640 1 -- 172.21.15.78:0/2191467061 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v107) v1 ==== 143+0+0 (secure 0 0 0) 0x7ff61c0578c0 con 0x7ff62806d550 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff60c06b810 msgr2=0x7ff60c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff60c06b810 0x7ff60c06dcd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff614036000 tx=0x7ff61403d040 comp rx=0 tx=0).stop 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 msgr2=0x7ff628157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 0x7ff628157f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00c4a0 tx=0x7ff61c00c960 comp rx=0 tx=0).stop 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 shutdown_connections 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff60c06b810 0x7ff60c06dcd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6281588a0 0x7ff6281e0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff628155f90 0x7ff628151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 --2- 172.21.15.78:0/2191467061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff62806d550 0x7ff628157f00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 >> 172.21.15.78:0/2191467061 conn(0x7ff62806b700 msgr2=0x7ff6280aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 shutdown_connections 2024-02-19T14:53:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.632+0000 7ff62ed0d640 1 -- 172.21.15.78:0/2191467061 wait complete. 2024-02-19T14:53:30.645 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. ... ok 2024-02-19T14:53:30.645 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is still used if necessary. 2024-02-19T14:53:30.645 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:53:30.646 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:30.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.744+0000 7f969061b640 1 Processor -- start 2024-02-19T14:53:30.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.744+0000 7f969061b640 1 -- start start 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.745+0000 7f969061b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9688152700 0x7f96880c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.745+0000 7f969061b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96880c8e00 0x7f968815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.745+0000 7f969061b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96881551f0 con 0x7f96880c5730 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.745+0000 7f969061b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9688155490 con 0x7f9688152700 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.745+0000 7f969061b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9688155770 con 0x7f96880c8e00 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.745+0000 7f968e3b7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9688152700 0x7f96880c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968e3b7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9688152700 0x7f96880c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55818/0 (socket says 172.21.15.78:55818) 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968e3b7640 1 -- 172.21.15.78:0/4220613005 learned_addr learned my addr 172.21.15.78:0/4220613005 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968ebb8640 1 --2- 172.21.15.78:0/4220613005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96880c8e00 0x7f968815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968e3b7640 1 -- 172.21.15.78:0/4220613005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96880c8e00 msgr2=0x7f968815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968e3b7640 1 --2- 172.21.15.78:0/4220613005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96880c8e00 0x7f968815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968e3b7640 1 -- 172.21.15.78:0/4220613005 >> v1:172.21.15.150:6789/0 conn(0x7f96880c5730 legacy=0x7f968806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.746+0000 7f968e3b7640 1 -- 172.21.15.78:0/4220613005 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9688155a50 con 0x7f9688152700 2024-02-19T14:53:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f968e3b7640 1 --2- 172.21.15.78:0/4220613005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9688152700 0x7f96880c51f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9680009e50 tx=0x7f968002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698330d3a1ea0c1c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f968d3b5640 1 -- 172.21.15.78:0/4220613005 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968002cb00 con 0x7f9688152700 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f968d3b5640 1 -- 172.21.15.78:0/4220613005 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968002cc60 con 0x7f9688152700 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f968d3b5640 1 -- 172.21.15.78:0/4220613005 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96800343f0 con 0x7f9688152700 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 -- 172.21.15.78:0/4220613005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9688152700 msgr2=0x7f96880c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 --2- 172.21.15.78:0/4220613005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9688152700 0x7f96880c51f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9680009e50 tx=0x7f968002bf60 comp rx=0 tx=0).stop 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 -- 172.21.15.78:0/4220613005 shutdown_connections 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 --2- 172.21.15.78:0/4220613005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96880c8e00 0x7f968815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 --2- 172.21.15.78:0/4220613005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9688152700 0x7f96880c51f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 -- 172.21.15.78:0/4220613005 >> 172.21.15.78:0/4220613005 conn(0x7f96880cac90 msgr2=0x7f96880cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 -- 172.21.15.78:0/4220613005 shutdown_connections 2024-02-19T14:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.747+0000 7f969061b640 1 -- 172.21.15.78:0/4220613005 wait complete. 2024-02-19T14:53:30.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.748+0000 7f969061b640 1 Processor -- start 2024-02-19T14:53:30.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.748+0000 7f969061b640 1 -- start start 2024-02-19T14:53:30.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f969061b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96880c5730 0x7f96880c3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f969061b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 0x7f96880aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f969061b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9688152700 0x7f96880ab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f969061b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96881eb5c0 con 0x7f96880c8e00 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f969061b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880acaf0 con 0x7f96880c5730 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f969061b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96880ab710 con 0x7f9688152700 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 0x7f96880aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 0x7f96880aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55822/0 (socket says 172.21.15.78:55822) 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 -- 172.21.15.78:0/508430538 learned_addr learned my addr 172.21.15.78:0/508430538 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968e3b7640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96880c5730 0x7f96880c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968ebb8640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9688152700 0x7f96880ab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 -- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9688152700 msgr2=0x7f96880ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9688152700 0x7f96880ab1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 -- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96880c5730 msgr2=0x7f96880c3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96880c5730 0x7f96880c3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.749+0000 7f968dbb6640 1 -- 172.21.15.78:0/508430538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9680009b30 con 0x7f96880c8e00 2024-02-19T14:53:30.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.750+0000 7f968dbb6640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 0x7f96880aac90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f967800c480 tx=0x7f967800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.750+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96780052d0 con 0x7f96880c8e00 2024-02-19T14:53:30.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.750+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678005430 con 0x7f96880c8e00 2024-02-19T14:53:30.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.750+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96880acf80 con 0x7f96880c8e00 2024-02-19T14:53:30.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.750+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678012870 con 0x7f96880c8e00 2024-02-19T14:53:30.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.750+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9688159030 con 0x7f96880c8e00 2024-02-19T14:53:30.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.751+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96880acca0 con 0x7f96880c8e00 2024-02-19T14:53:30.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.751+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f96780129d0 con 0x7f96880c8e00 2024-02-19T14:53:30.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.752+0000 7f967effd640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f966c06b810 0x7f966c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:30.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.752+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9678089a90 con 0x7f96880c8e00 2024-02-19T14:53:30.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.752+0000 7f968e3b7640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f966c06b810 0x7f966c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:30.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.752+0000 7f968e3b7640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f966c06b810 0x7f966c06dcd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9680035000 tx=0x7f9680030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:30.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.756+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f96880acca0 con 0x7f96880c8e00 2024-02-19T14:53:30.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.930+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96880acca0 con 0x7f96880c8e00 2024-02-19T14:53:30.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.930+0000 7f967effd640 1 -- 172.21.15.78:0/508430538 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+13413 (secure 0 0 0) 0x7f96880acca0 con 0x7f96880c8e00 2024-02-19T14:53:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:30.933 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":42,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:29.619713+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/4259811019":"2024-02-20T14:53:26.529849+0000","172.21.15.78:6837/3231075223":"2024-02-20T14:53:10.560278+0000","172.21.15.150:6833/4259811019":"2024-02-20T14:53:26.529849+0000","172.21.15.78:6836/3231075223":"2024-02-20T14:53:10.560278+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f966c06b810 msgr2=0x7f966c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f966c06b810 0x7f966c06dcd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9680035000 tx=0x7f9680030000 comp rx=0 tx=0).stop 2024-02-19T14:53:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 msgr2=0x7f96880aac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 0x7f96880aac90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f967800c480 tx=0x7f967800c940 comp rx=0 tx=0).stop 2024-02-19T14:53:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 shutdown_connections 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f966c06b810 0x7f966c06dcd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96880c8e00 0x7f96880aac90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9688152700 0x7f96880ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 --2- 172.21.15.78:0/508430538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96880c5730 0x7f96880c3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 >> 172.21.15.78:0/508430538 conn(0x7f96880cac90 msgr2=0x7f968814d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 shutdown_connections 2024-02-19T14:53:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:30.932+0000 7f969061b640 1 -- 172.21.15.78:0/508430538 wait complete. 2024-02-19T14:53:30.946 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:53:30.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:53:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.046+0000 7f3e45399640 1 Processor -- start 2024-02-19T14:53:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.046+0000 7f3e45399640 1 -- start start 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e45399640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e400c8e00 0x7f3e4006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e45399640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3e4006d910 0x7f3e4006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e45399640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4006cd20 con 0x7f3e40152700 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e45399640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e401551f0 con 0x7f3e4006d910 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e45399640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e40155490 con 0x7f3e400c8e00 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3e4006d910 0x7f3e4006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3e4006d910 0x7f3e4006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55834/0 (socket says 172.21.15.78:55834) 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 -- 172.21.15.78:0/1986918169 learned_addr learned my addr 172.21.15.78:0/1986918169 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e37fff640 1 --2- 172.21.15.78:0/1986918169 >> v2:172.21.15.150:3300/0 conn(0x7f3e400c8e00 0x7f3e4006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 -- 172.21.15.78:0/1986918169 >> v2:172.21.15.150:3300/0 conn(0x7f3e400c8e00 msgr2=0x7f3e4006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 --2- 172.21.15.78:0/1986918169 >> v2:172.21.15.150:3300/0 conn(0x7f3e400c8e00 0x7f3e4006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 -- 172.21.15.78:0/1986918169 >> v1:172.21.15.150:6789/0 conn(0x7f3e40152700 legacy=0x7f3e400c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.047+0000 7f3e3f7fe640 1 -- 172.21.15.78:0/1986918169 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40155770 con 0x7f3e4006d910 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.048+0000 7f3e3f7fe640 1 --2- 172.21.15.78:0/1986918169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e4006c7e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3e400c7620 tx=0x7f3e3802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912a8f553da88faa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.048+0000 7f3e3e7fc640 1 -- 172.21.15.78:0/1986918169 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3802cb20 con 0x7f3e4006d910 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.048+0000 7f3e3e7fc640 1 -- 172.21.15.78:0/1986918169 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3802cc80 con 0x7f3e4006d910 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.048+0000 7f3e3e7fc640 1 -- 172.21.15.78:0/1986918169 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e380343f0 con 0x7f3e4006d910 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.048+0000 7f3e45399640 1 -- 172.21.15.78:0/1986918169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 msgr2=0x7f3e4006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.048+0000 7f3e45399640 1 --2- 172.21.15.78:0/1986918169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e4006c7e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3e400c7620 tx=0x7f3e3802bf80 comp rx=0 tx=0).stop 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.049+0000 7f3e45399640 1 -- 172.21.15.78:0/1986918169 shutdown_connections 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.049+0000 7f3e45399640 1 --2- 172.21.15.78:0/1986918169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e4006c7e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.049+0000 7f3e45399640 1 --2- 172.21.15.78:0/1986918169 >> v2:172.21.15.150:3300/0 conn(0x7f3e400c8e00 0x7f3e4006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.049+0000 7f3e45399640 1 -- 172.21.15.78:0/1986918169 >> 172.21.15.78:0/1986918169 conn(0x7f3e400cac90 msgr2=0x7f3e400cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.049+0000 7f3e45399640 1 -- 172.21.15.78:0/1986918169 shutdown_connections 2024-02-19T14:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.049+0000 7f3e45399640 1 -- 172.21.15.78:0/1986918169 wait complete. 2024-02-19T14:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 Processor -- start 2024-02-19T14:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 -- start start 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e401e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e400c8e00 0x7f3e401e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e40152700 0x7f3e401eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e401497b0 con 0x7f3e4006d910 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40157870 con 0x7f3e40152700 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e45399640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400c41f0 con 0x7f3e400c8e00 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e3effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e401e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e3effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e401e79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55836/0 (socket says 172.21.15.78:55836) 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.050+0000 7f3e3effd640 1 -- 172.21.15.78:0/1613189370 learned_addr learned my addr 172.21.15.78:0/1613189370 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e3effd640 1 -- 172.21.15.78:0/1613189370 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e400c8e00 msgr2=0x7f3e401e7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e3effd640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e400c8e00 0x7f3e401e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e3effd640 1 -- 172.21.15.78:0/1613189370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e40152700 msgr2=0x7f3e401eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e3effd640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e40152700 0x7f3e401eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e3effd640 1 -- 172.21.15.78:0/1613189370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38009b30 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e3effd640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e401e79b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000c3c0 tx=0x7f3e3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e300052d0 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e30005450 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400c44d0 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30007260 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.051+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e40157220 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.052+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e40074990 con 0x7f3e4006d910 2024-02-19T14:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.052+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3e300073c0 con 0x7f3e4006d910 2024-02-19T14:53:31.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.053+0000 7f3e377fe640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e2006b840 0x7f3e2006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.053+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3e30089450 con 0x7f3e4006d910 2024-02-19T14:53:31.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.053+0000 7f3e37fff640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e2006b840 0x7f3e2006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.053+0000 7f3e37fff640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e2006b840 0x7f3e2006dd00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e400aa8f0 tx=0x7f3e28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.057+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3e30053f90 con 0x7f3e4006d910 2024-02-19T14:53:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.227+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e400c8770 con 0x7f3e4006d910 2024-02-19T14:53:31.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.227+0000 7f3e377fe640 1 -- 172.21.15.78:0/1613189370 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e30057650 con 0x7f3e4006d910 2024-02-19T14:53:31.230 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:53:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e2006b840 msgr2=0x7f3e2006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e2006b840 0x7f3e2006dd00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e400aa8f0 tx=0x7f3e28008040 comp rx=0 tx=0).stop 2024-02-19T14:53:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 msgr2=0x7f3e401e79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e401e79b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000c3c0 tx=0x7f3e3000c880 comp rx=0 tx=0).stop 2024-02-19T14:53:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 shutdown_connections 2024-02-19T14:53:31.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e2006b840 0x7f3e2006dd00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e40152700 0x7f3e401eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e400c8e00 0x7f3e401e7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 --2- 172.21.15.78:0/1613189370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e4006d910 0x7f3e401e79b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 >> 172.21.15.78:0/1613189370 conn(0x7f3e400cac90 msgr2=0x7f3e4006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 shutdown_connections 2024-02-19T14:53:31.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.229+0000 7f3e45399640 1 -- 172.21.15.78:0/1613189370 wait complete. 2024-02-19T14:53:31.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-02-19T14:53:31.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.342+0000 7f764bdac640 1 Processor -- start 2024-02-19T14:53:31.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.342+0000 7f764bdac640 1 -- start start 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f764bdac640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76441528d0 0x7f76440c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f764bdac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7644155780 0x7f764415f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f764bdac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7644072e80 con 0x7f76440c19c0 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f764bdac640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76440734c0 con 0x7f76441528d0 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f764bdac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764406e990 con 0x7f7644155780 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76441528d0 0x7f76440c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76441528d0 0x7f76440c1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55844/0 (socket says 172.21.15.78:55844) 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 -- 172.21.15.78:0/1350015597 learned_addr learned my addr 172.21.15.78:0/1350015597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f764a349640 1 --2- 172.21.15.78:0/1350015597 >> v2:172.21.15.150:3300/0 conn(0x7f7644155780 0x7f764415f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 -- 172.21.15.78:0/1350015597 >> v2:172.21.15.150:3300/0 conn(0x7f7644155780 msgr2=0x7f764415f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:31.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 --2- 172.21.15.78:0/1350015597 >> v2:172.21.15.150:3300/0 conn(0x7f7644155780 0x7f764415f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 -- 172.21.15.78:0/1350015597 >> v1:172.21.15.150:6789/0 conn(0x7f76440c19c0 legacy=0x7f7644155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.343+0000 7f7649b48640 1 -- 172.21.15.78:0/1350015597 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440731e0 con 0x7f76441528d0 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.344+0000 7f7649b48640 1 --2- 172.21.15.78:0/1350015597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440c1480 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f76440c8570 tx=0x7f764002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501341d7004365e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.344+0000 7f7648b46640 1 -- 172.21.15.78:0/1350015597 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640031870 con 0x7f76441528d0 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.344+0000 7f7648b46640 1 -- 172.21.15.78:0/1350015597 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76400319d0 con 0x7f76441528d0 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.344+0000 7f7648b46640 1 -- 172.21.15.78:0/1350015597 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640031d60 con 0x7f76441528d0 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 -- 172.21.15.78:0/1350015597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 msgr2=0x7f76440c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 --2- 172.21.15.78:0/1350015597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440c1480 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f76440c8570 tx=0x7f764002b910 comp rx=0 tx=0).stop 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 -- 172.21.15.78:0/1350015597 shutdown_connections 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 --2- 172.21.15.78:0/1350015597 >> v2:172.21.15.150:3300/0 conn(0x7f7644155780 0x7f764415f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 --2- 172.21.15.78:0/1350015597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440c1480 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 -- 172.21.15.78:0/1350015597 >> 172.21.15.78:0/1350015597 conn(0x7f76440c7590 msgr2=0x7f76440c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 -- 172.21.15.78:0/1350015597 shutdown_connections 2024-02-19T14:53:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.345+0000 7f764bdac640 1 -- 172.21.15.78:0/1350015597 wait complete. 2024-02-19T14:53:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 Processor -- start 2024-02-19T14:53:31.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 -- start start 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76440c19c0 0x7f7644157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7644155780 0x7f764414bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76440a9cd0 con 0x7f76441528d0 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76441eee50 con 0x7f76440c19c0 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f764bdac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76440a9a10 con 0x7f7644155780 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.346+0000 7f7649347640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440a7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55860/0 (socket says 172.21.15.78:55860) 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 -- 172.21.15.78:0/3496735797 learned_addr learned my addr 172.21.15.78:0/3496735797 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f764a349640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7644155780 0x7f764414bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649b48640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76440c19c0 0x7f7644157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 -- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7644155780 msgr2=0x7f764414bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7644155780 0x7f764414bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 -- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76440c19c0 msgr2=0x7f7644157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76440c19c0 0x7f7644157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 -- 172.21.15.78:0/3496735797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76400094d0 con 0x7f76441528d0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f7649347640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440a7cc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f763400c3f0 tx=0x7f763400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340052d0 con 0x7f76441528d0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7634005450 con 0x7f76441528d0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76440a88d0 con 0x7f76441528d0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634007260 con 0x7f76441528d0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.347+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76440a77a0 con 0x7f76441528d0 2024-02-19T14:53:31.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.348+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7644068260 con 0x7f76441528d0 2024-02-19T14:53:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.349+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f76340073c0 con 0x7f76441528d0 2024-02-19T14:53:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.349+0000 7f763a7fc640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.349+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f76340890b0 con 0x7f76441528d0 2024-02-19T14:53:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.349+0000 7f7649b48640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.350+0000 7f7649b48640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7640036000 tx=0x7f764003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.354+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7634053c80 con 0x7f76441528d0 2024-02-19T14:53:31.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.589+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7f7644072e80 con 0x7f76441528d0 2024-02-19T14:53:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.631+0000 7f763a7fc640 1 -- 172.21.15.78:0/3496735797 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v108) v1 ==== 153+0+0 (secure 0 0 0) 0x7f7644072e80 con 0x7f76441528d0 2024-02-19T14:53:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 msgr2=0x7f762806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7640036000 tx=0x7f764003c040 comp rx=0 tx=0).stop 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 msgr2=0x7f76440a7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440a7cc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f763400c3f0 tx=0x7f763400c8b0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 shutdown_connections 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76441528d0 0x7f76440a7cc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7644155780 0x7f764414bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 --2- 172.21.15.78:0/3496735797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76440c19c0 0x7f7644157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 >> 172.21.15.78:0/3496735797 conn(0x7f76440c7590 msgr2=0x7f7644153910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 shutdown_connections 2024-02-19T14:53:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.633+0000 7f764bdac640 1 -- 172.21.15.78:0/3496735797 wait complete. 2024-02-19T14:53:31.646 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:53:31.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.747+0000 7ff1e5e09640 1 Processor -- start 2024-02-19T14:53:31.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.747+0000 7ff1e5e09640 1 -- start start 2024-02-19T14:53:31.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1e5e09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1e014ece0 0x7ff1e00c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1e5e09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff1e00c5270 0x7ff1e00719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1e5e09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e0071f10 0x7ff1e00c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1e5e09640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e00c1120 con 0x7ff1e014ece0 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1e5e09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e00c6080 con 0x7ff1e00c5270 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1e5e09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e00c57e0 con 0x7ff1e0071f10 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1e014ece0 0x7ff1e00c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1e014ece0 0x7ff1e00c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55868/0 (socket says 172.21.15.78:55868) 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 -- 172.21.15.78:0/3474015643 learned_addr learned my addr 172.21.15.78:0/3474015643 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1deffd640 1 --2- 172.21.15.78:0/3474015643 >> v2:172.21.15.150:3300/0 conn(0x7ff1e00c5270 0x7ff1e00719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1dffff640 1 --2- 172.21.15.78:0/3474015643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e0071f10 0x7ff1e00c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 -- 172.21.15.78:0/3474015643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e0071f10 msgr2=0x7ff1e00c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 --2- 172.21.15.78:0/3474015643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e0071f10 0x7ff1e00c15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 -- 172.21.15.78:0/3474015643 >> v2:172.21.15.150:3300/0 conn(0x7ff1e00c5270 msgr2=0x7ff1e00719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 --2- 172.21.15.78:0/3474015643 >> v2:172.21.15.150:3300/0 conn(0x7ff1e00c5270 0x7ff1e00719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.748+0000 7ff1df7fe640 1 -- 172.21.15.78:0/3474015643 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00c5ac0 con 0x7ff1e014ece0 2024-02-19T14:53:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.749+0000 7ff1df7fe640 1 --2- 172.21.15.78:0/3474015643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e014ece0 0x7ff1e00c4d30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0155b80 tx=0x7ff1d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cbce855f9abe73 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.749+0000 7ff1de7fc640 1 -- 172.21.15.78:0/3474015643 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0031870 con 0x7ff1e014ece0 2024-02-19T14:53:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.749+0000 7ff1de7fc640 1 -- 172.21.15.78:0/3474015643 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d00319d0 con 0x7ff1e014ece0 2024-02-19T14:53:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.749+0000 7ff1de7fc640 1 -- 172.21.15.78:0/3474015643 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0031d60 con 0x7ff1e014ece0 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 -- 172.21.15.78:0/3474015643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e014ece0 msgr2=0x7ff1e00c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/3474015643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e014ece0 0x7ff1e00c4d30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0155b80 tx=0x7ff1d002b910 comp rx=0 tx=0).stop 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 -- 172.21.15.78:0/3474015643 shutdown_connections 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/3474015643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e0071f10 0x7ff1e00c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/3474015643 >> v2:172.21.15.150:3300/0 conn(0x7ff1e00c5270 0x7ff1e00719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/3474015643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e014ece0 0x7ff1e00c4d30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 -- 172.21.15.78:0/3474015643 >> 172.21.15.78:0/3474015643 conn(0x7ff1e006b270 msgr2=0x7ff1e006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 -- 172.21.15.78:0/3474015643 shutdown_connections 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.750+0000 7ff1e5e09640 1 -- 172.21.15.78:0/3474015643 wait complete. 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.751+0000 7ff1e5e09640 1 Processor -- start 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.751+0000 7ff1e5e09640 1 -- start start 2024-02-19T14:53:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 0x7ff1e0142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1e00c5270 0x7ff1e01422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e014ece0 0x7ff1e00bf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0148440 con 0x7ff1e0071f10 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01eaf00 con 0x7ff1e00c5270 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01eb1e0 con 0x7ff1e014ece0 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 0x7ff1e0142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 0x7ff1e0142c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55876/0 (socket says 172.21.15.78:55876) 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 -- 172.21.15.78:0/466388871 learned_addr learned my addr 172.21.15.78:0/466388871 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1dffff640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e014ece0 0x7ff1e00bf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1deffd640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1e00c5270 0x7ff1e01422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 -- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e014ece0 msgr2=0x7ff1e00bf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e014ece0 0x7ff1e00bf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 -- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1e00c5270 msgr2=0x7ff1e01422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1e00c5270 0x7ff1e01422a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 -- 172.21.15.78:0/466388871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d00094d0 con 0x7ff1e0071f10 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1df7fe640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 0x7ff1e0142c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff1d002b950 tx=0x7ff1d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d003ad30 con 0x7ff1e0071f10 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d00443f0 con 0x7ff1e0071f10 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e0147df0 con 0x7ff1e0071f10 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.752+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00447a0 con 0x7ff1e0071f10 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.753+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e014b9a0 con 0x7ff1e0071f10 2024-02-19T14:53:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.753+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e00c3ca0 con 0x7ff1e0071f10 2024-02-19T14:53:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.754+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff1d004d450 con 0x7ff1e0071f10 2024-02-19T14:53:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.754+0000 7ff1c7fff640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1cc06b810 0x7ff1cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.755+0000 7ff1deffd640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1cc06b810 0x7ff1cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.755+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff1d002d040 con 0x7ff1e0071f10 2024-02-19T14:53:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.755+0000 7ff1deffd640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1cc06b810 0x7ff1cc06dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1e014d300 tx=0x7ff1d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:31.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.759+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff1d0077100 con 0x7ff1e0071f10 2024-02-19T14:53:31.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.929+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff1e0074990 con 0x7ff1e0071f10 2024-02-19T14:53:31.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.932+0000 7ff1c7fff640 1 -- 172.21.15.78:0/466388871 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v15)=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff1d007a7c0 con 0x7ff1e0071f10 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1cc06b810 msgr2=0x7ff1cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1cc06b810 0x7ff1cc06dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1e014d300 tx=0x7ff1d4008040 comp rx=0 tx=0).stop 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 msgr2=0x7ff1e0142c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 0x7ff1e0142c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff1d002b950 tx=0x7ff1d00046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 shutdown_connections 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1cc06b810 0x7ff1cc06dcd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1e014ece0 0x7ff1e00bf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1e00c5270 0x7ff1e01422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 --2- 172.21.15.78:0/466388871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1e0071f10 0x7ff1e0142c20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:31.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 >> 172.21.15.78:0/466388871 conn(0x7ff1e006b270 msgr2=0x7ff1e006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:31.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 shutdown_connections 2024-02-19T14:53:31.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:31.934+0000 7ff1e5e09640 1 -- 172.21.15.78:0/466388871 wait complete. 2024-02-19T14:53:31.947 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:53:31.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:53:32.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:53:32.046 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:53:32.046 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:53:32.046 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:53:32.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:53:32.061 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:53:32.061 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:53:32.078 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:53:32.079 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:53:32.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:32.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.198+0000 7fb284679640 1 Processor -- start 2024-02-19T14:53:32.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.198+0000 7fb284679640 1 -- start start 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb284679640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb27c0730a0 0x7fb27c158550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb284679640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb27c072180 0x7fb27c158a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb284679640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c072ec0 con 0x7fb27c152610 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb284679640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c06fa40 con 0x7fb27c072180 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb284679640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c06f270 con 0x7fb27c0730a0 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb282c16640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb27c072180 0x7fb27c158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb282c16640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb27c072180 0x7fb27c158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55886/0 (socket says 172.21.15.78:55886) 2024-02-19T14:53:32.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb282c16640 1 -- 172.21.15.78:0/3627584685 learned_addr learned my addr 172.21.15.78:0/3627584685 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:32.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.199+0000 7fb281c14640 1 --2- 172.21.15.78:0/3627584685 >> v2:172.21.15.150:3300/0 conn(0x7fb27c0730a0 0x7fb27c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb282c16640 1 -- 172.21.15.78:0/3627584685 >> v2:172.21.15.150:3300/0 conn(0x7fb27c0730a0 msgr2=0x7fb27c158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:32.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb282c16640 1 --2- 172.21.15.78:0/3627584685 >> v2:172.21.15.150:3300/0 conn(0x7fb27c0730a0 0x7fb27c158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb282c16640 1 -- 172.21.15.78:0/3627584685 >> v1:172.21.15.150:6789/0 conn(0x7fb27c152610 legacy=0x7fb27c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb282c16640 1 -- 172.21.15.78:0/3627584685 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c06f4c0 con 0x7fb27c072180 2024-02-19T14:53:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb282c16640 1 --2- 172.21.15.78:0/3627584685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c158a90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb274009e50 tx=0x7fb27402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f72511f96914259 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb281413640 1 -- 172.21.15.78:0/3627584685 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27402ca80 con 0x7fb27c072180 2024-02-19T14:53:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb281413640 1 -- 172.21.15.78:0/3627584685 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27402cbe0 con 0x7fb27c072180 2024-02-19T14:53:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.200+0000 7fb281413640 1 -- 172.21.15.78:0/3627584685 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740343f0 con 0x7fb27c072180 2024-02-19T14:53:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 -- 172.21.15.78:0/3627584685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 msgr2=0x7fb27c158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 --2- 172.21.15.78:0/3627584685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c158a90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb274009e50 tx=0x7fb27402bee0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 -- 172.21.15.78:0/3627584685 shutdown_connections 2024-02-19T14:53:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 --2- 172.21.15.78:0/3627584685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c158a90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 --2- 172.21.15.78:0/3627584685 >> v2:172.21.15.150:3300/0 conn(0x7fb27c0730a0 0x7fb27c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 -- 172.21.15.78:0/3627584685 >> 172.21.15.78:0/3627584685 conn(0x7fb27c0c7590 msgr2=0x7fb27c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 -- 172.21.15.78:0/3627584685 shutdown_connections 2024-02-19T14:53:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.201+0000 7fb284679640 1 -- 172.21.15.78:0/3627584685 wait complete. 2024-02-19T14:53:32.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.202+0000 7fb284679640 1 Processor -- start 2024-02-19T14:53:32.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.202+0000 7fb284679640 1 -- start start 2024-02-19T14:53:32.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb284679640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c0a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb284679640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c0730a0 0x7fb27c0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb284679640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c152610 0x7fb27c0aa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb284679640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0ac560 con 0x7fb27c072180 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb284679640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0a7340 con 0x7fb27c152610 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb284679640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0a7620 con 0x7fb27c0730a0 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c0a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c0a83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55888/0 (socket says 172.21.15.78:55888) 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 -- 172.21.15.78:0/2093957949 learned_addr learned my addr 172.21.15.78:0/2093957949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb281c14640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c0730a0 0x7fb27c0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282c16640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c152610 0x7fb27c0aa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 -- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c0730a0 msgr2=0x7fb27c0a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c0730a0 0x7fb27c0a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 -- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c152610 msgr2=0x7fb27c0aa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c152610 0x7fb27c0aa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 -- 172.21.15.78:0/2093957949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274009b30 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.203+0000 7fb282415640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c0a83e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c3c0 tx=0x7fb27800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.204+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2780052d0 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.204+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb278005450 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.204+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278007260 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.204+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c0a7900 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.204+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c0a7ec0 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.205+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c074990 con 0x7fb27c072180 2024-02-19T14:53:32.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.205+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb2780073c0 con 0x7fb27c072180 2024-02-19T14:53:32.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.206+0000 7fb272ffd640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb26006b810 0x7fb26006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.206+0000 7fb281c14640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb26006b810 0x7fb26006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.206+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb278089320 con 0x7fb27c072180 2024-02-19T14:53:32.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.206+0000 7fb281c14640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb26006b810 0x7fb26006dcd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0c2290 tx=0x7fb26c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:32.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.210+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb278053fa0 con 0x7fb27c072180 2024-02-19T14:53:32.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.419+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb27c154270 con 0x7fb27c072180 2024-02-19T14:53:32.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.419+0000 7fb272ffd640 1 -- 172.21.15.78:0/2093957949 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fb278057660 con 0x7fb27c072180 2024-02-19T14:53:32.422 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:32.422 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2799781974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2799781974},{"type":"v1","addr":"172.21.15.150:6835","nonce":2799781974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5020,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3578832669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3578832669},{"type":"v1","addr":"172.21.15.150:6834","nonce":3578832669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1762664317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1762664317},{"type":"v1","addr":"172.21.15.78:6835","nonce":1762664317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1556668834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1556668834},{"type":"v1","addr":"172.21.15.78:6837","nonce":1556668834}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2024-02-19T14:53:32.423 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 22 2024-02-19T14:53:32.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb26006b810 msgr2=0x7fb26006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb26006b810 0x7fb26006dcd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0c2290 tx=0x7fb26c008040 comp rx=0 tx=0).stop 2024-02-19T14:53:32.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 msgr2=0x7fb27c0a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c0a83e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb27800c3c0 tx=0x7fb27800c880 comp rx=0 tx=0).stop 2024-02-19T14:53:32.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 shutdown_connections 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb26006b810 0x7fb26006dcd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c152610 0x7fb27c0aa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c0730a0 0x7fb27c0a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 --2- 172.21.15.78:0/2093957949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c072180 0x7fb27c0a83e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 >> 172.21.15.78:0/2093957949 conn(0x7fb27c0c7590 msgr2=0x7fb27c154a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 shutdown_connections 2024-02-19T14:53:32.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.421+0000 7fb284679640 1 -- 172.21.15.78:0/2093957949 wait complete. 2024-02-19T14:53:32.435 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:53:32.435 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:53:32.435 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:53:32.435 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:53:32.436 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:53:32.436 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:53:32.436 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:53:32.436 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:53:32.436 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:53:32.436 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:53:32.436 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:53:32.436 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:53:32.437 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:53:32.437 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:53:32.437 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:53:32.437 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:53:32.445 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:53:32.445 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:53:32.446 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:53:32.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:53:32.448 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:53:32.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:53:32.449 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:53:32.449 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:53:32.487 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:53:32.488 DEBUG:teuthology.parallel:result is None 2024-02-19T14:53:32.488 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:53:32.488 DEBUG:teuthology.parallel:result is None 2024-02-19T14:53:32.491 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:53:32.491 DEBUG:teuthology.parallel:result is None 2024-02-19T14:53:32.492 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:53:32.492 DEBUG:teuthology.parallel:result is None 2024-02-19T14:53:32.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:53:32.564 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:53:32.565 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:53:32.574 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:53:32.574 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:53:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 Processor -- start 2024-02-19T14:53:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 -- start start 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4c00c8e00 0x7fa4c006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa4c006d910 0x7fa4c006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c006cd20 con 0x7fa4c0152700 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c01551f0 con 0x7fa4c006d910 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.596+0000 7fa4c594b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c0155490 con 0x7fa4c00c8e00 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4bf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa4c006d910 0x7fa4c006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4beffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa4c0152700 0x7fa4c00c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59140/0 (socket says 172.21.15.78:59140) 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4beffd640 1 -- 172.21.15.78:0/3302228555 learned_addr learned my addr 172.21.15.78:0/3302228555 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4be7fc640 1 --2- 172.21.15.78:0/3302228555 >> v2:172.21.15.150:3300/0 conn(0x7fa4c00c8e00 0x7fa4c006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4bf7fe640 1 -- 172.21.15.78:0/3302228555 >> v2:172.21.15.150:3300/0 conn(0x7fa4c00c8e00 msgr2=0x7fa4c006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4bf7fe640 1 --2- 172.21.15.78:0/3302228555 >> v2:172.21.15.150:3300/0 conn(0x7fa4c00c8e00 0x7fa4c006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4bf7fe640 1 -- 172.21.15.78:0/3302228555 >> v1:172.21.15.78:6789/0 conn(0x7fa4c0152700 legacy=0x7fa4c00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4bdffb640 1 -- 172.21.15.78:0/3302228555 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165839556 0 0) 0x7fa4c006cd20 con 0x7fa4c0152700 2024-02-19T14:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4bf7fe640 1 -- 172.21.15.78:0/3302228555 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0155770 con 0x7fa4c006d910 2024-02-19T14:53:32.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.597+0000 7fa4be7fc640 1 --2- 172.21.15.78:0/3302228555 >> v2:172.21.15.150:3300/0 conn(0x7fa4c00c8e00 0x7fa4c006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4bf7fe640 1 --2- 172.21.15.78:0/3302228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c006d910 0x7fa4c006c7e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00c7620 tx=0x7fa4b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ecfaf8c4ef04ae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4bdffb640 1 -- 172.21.15.78:0/3302228555 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0031830 con 0x7fa4c006d910 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4bdffb640 1 -- 172.21.15.78:0/3302228555 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0031990 con 0x7fa4c006d910 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4bdffb640 1 -- 172.21.15.78:0/3302228555 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0031d20 con 0x7fa4c006d910 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 -- 172.21.15.78:0/3302228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c006d910 msgr2=0x7fa4c006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 --2- 172.21.15.78:0/3302228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c006d910 0x7fa4c006c7e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00c7620 tx=0x7fa4b002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 -- 172.21.15.78:0/3302228555 shutdown_connections 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 --2- 172.21.15.78:0/3302228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c006d910 0x7fa4c006c7e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 --2- 172.21.15.78:0/3302228555 >> v2:172.21.15.150:3300/0 conn(0x7fa4c00c8e00 0x7fa4c006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 -- 172.21.15.78:0/3302228555 >> 172.21.15.78:0/3302228555 conn(0x7fa4c00cac90 msgr2=0x7fa4c00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:32.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 -- 172.21.15.78:0/3302228555 shutdown_connections 2024-02-19T14:53:32.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.598+0000 7fa4c594b640 1 -- 172.21.15.78:0/3302228555 wait complete. 2024-02-19T14:53:32.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.599+0000 7fa4c594b640 1 Processor -- start 2024-02-19T14:53:32.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 -- start start 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c006d910 0x7fa4c00b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 0x7fa4c00ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c0152700 0x7fa4c01ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01eb620 con 0x7fa4c00c8e00 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00ad530 con 0x7fa4c0152700 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4c594b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01eadd0 con 0x7fa4c006d910 2024-02-19T14:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4be7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 0x7fa4c00ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4be7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 0x7fa4c00ac7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55932/0 (socket says 172.21.15.78:55932) 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4be7fc640 1 -- 172.21.15.78:0/2089881163 learned_addr learned my addr 172.21.15.78:0/2089881163 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4bf7fe640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c0152700 0x7fa4c01ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.600+0000 7fa4beffd640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c006d910 0x7fa4c00b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4be7fc640 1 -- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c006d910 msgr2=0x7fa4c00b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4be7fc640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c006d910 0x7fa4c00b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4be7fc640 1 -- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c0152700 msgr2=0x7fa4c01ecaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4be7fc640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c0152700 0x7fa4c01ecaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4be7fc640 1 -- 172.21.15.78:0/2089881163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009400 con 0x7fa4c00c8e00 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4be7fc640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 0x7fa4c00ac7b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009fc0 tx=0x7fa4b400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4003b90 con 0x7fa4c00c8e00 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b400f040 con 0x7fa4c00c8e00 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c00ad7d0 con 0x7fa4c00c8e00 2024-02-19T14:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4004030 con 0x7fa4c00c8e00 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.601+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c01eb0b0 con 0x7fa4c00c8e00 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.602+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c0152ae0 con 0x7fa4c00c8e00 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.602+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa4b4004190 con 0x7fa4c00c8e00 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.603+0000 7fa4a7fff640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ac06b810 0x7fa4ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.603+0000 7fa4beffd640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ac06b810 0x7fa4ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.603+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fa4b408a3b0 con 0x7fa4c00c8e00 2024-02-19T14:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.603+0000 7fa4beffd640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ac06b810 0x7fa4ac06dcd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8007fd0 tx=0x7fa4b8007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.607+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa4b4054fa0 con 0x7fa4c00c8e00 2024-02-19T14:53:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:32.799+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa4c0074990 con 0x7fa4c00c8e00 2024-02-19T14:53:33.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.653+0000 7fa4a7fff640 1 -- 172.21.15.78:0/2089881163 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v47) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa4b4058660 con 0x7fa4c00c8e00 2024-02-19T14:53:33.656 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:53:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.655+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ac06b810 msgr2=0x7fa4ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.655+0000 7fa4c594b640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ac06b810 0x7fa4ac06dcd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8007fd0 tx=0x7fa4b8007de0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.655+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 msgr2=0x7fa4c00ac7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.655+0000 7fa4c594b640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 0x7fa4c00ac7b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009fc0 tx=0x7fa4b400d560 comp rx=0 tx=0).stop 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 shutdown_connections 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ac06b810 0x7fa4ac06dcd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c00c8e00 0x7fa4c00ac7b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c0152700 0x7fa4c01ecaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 --2- 172.21.15.78:0/2089881163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c006d910 0x7fa4c00b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 >> 172.21.15.78:0/2089881163 conn(0x7fa4c00cac90 msgr2=0x7fa4c014ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 shutdown_connections 2024-02-19T14:53:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.656+0000 7fa4c594b640 1 -- 172.21.15.78:0/2089881163 wait complete. 2024-02-19T14:53:33.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:53:33.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.770+0000 7f67762eb640 1 Processor -- start 2024-02-19T14:53:33.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.770+0000 7f67762eb640 1 -- start start 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f67762eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f67700721a0 0x7f67700c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f67762eb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67700c6080 0x7f67700c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f67762eb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6770072c60 con 0x7f6770150460 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f67762eb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6770153e80 con 0x7f67700721a0 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f67762eb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67701538c0 con 0x7f67700c6080 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f676f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f67700721a0 0x7f67700c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:33.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f676f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f67700721a0 0x7f67700c1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55934/0 (socket says 172.21.15.78:55934) 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f676f7fe640 1 -- 172.21.15.78:0/1405644394 learned_addr learned my addr 172.21.15.78:0/1405644394 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.771+0000 7f6774888640 1 --2- 172.21.15.78:0/1405644394 >> v2:172.21.15.150:3300/0 conn(0x7f67700c6080 0x7f67700c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676f7fe640 1 -- 172.21.15.78:0/1405644394 >> v2:172.21.15.150:3300/0 conn(0x7f67700c6080 msgr2=0x7f67700c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676f7fe640 1 --2- 172.21.15.78:0/1405644394 >> v2:172.21.15.150:3300/0 conn(0x7f67700c6080 0x7f67700c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676f7fe640 1 -- 172.21.15.78:0/1405644394 >> v1:172.21.15.150:6789/0 conn(0x7f6770150460 legacy=0x7f67700c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676f7fe640 1 -- 172.21.15.78:0/1405644394 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6770153ba0 con 0x7f67700721a0 2024-02-19T14:53:33.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f6774888640 1 --2- 172.21.15.78:0/1405644394 >> v2:172.21.15.150:3300/0 conn(0x7f67700c6080 0x7f67700c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:33.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676f7fe640 1 --2- 172.21.15.78:0/1405644394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700721a0 0x7f67700c1810 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f677006de00 tx=0x7f676402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699a4ee6fcfcb857 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:33.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676effd640 1 -- 172.21.15.78:0/1405644394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764030860 con 0x7f67700721a0 2024-02-19T14:53:33.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676effd640 1 -- 172.21.15.78:0/1405644394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67640309c0 con 0x7f67700721a0 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.772+0000 7f676effd640 1 -- 172.21.15.78:0/1405644394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764030d50 con 0x7f67700721a0 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 -- 172.21.15.78:0/1405644394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700721a0 msgr2=0x7f67700c1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 --2- 172.21.15.78:0/1405644394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700721a0 0x7f67700c1810 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f677006de00 tx=0x7f676402b9a0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 -- 172.21.15.78:0/1405644394 shutdown_connections 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 --2- 172.21.15.78:0/1405644394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700721a0 0x7f67700c1810 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 --2- 172.21.15.78:0/1405644394 >> v2:172.21.15.150:3300/0 conn(0x7f67700c6080 0x7f67700c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 -- 172.21.15.78:0/1405644394 >> 172.21.15.78:0/1405644394 conn(0x7f677006c170 msgr2=0x7f677006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 -- 172.21.15.78:0/1405644394 shutdown_connections 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.773+0000 7f67762eb640 1 -- 172.21.15.78:0/1405644394 wait complete. 2024-02-19T14:53:33.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.774+0000 7f67762eb640 1 Processor -- start 2024-02-19T14:53:33.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.774+0000 7f67762eb640 1 -- start start 2024-02-19T14:53:33.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f67762eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700721a0 0x7f6770156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:33.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f67762eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c6080 0x7f6770151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:33.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f67762eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 0x7f67701457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:33.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f67762eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67700c27a0 con 0x7f6770150460 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f67762eb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67700c2d60 con 0x7f67700c6080 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f67762eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677014d260 con 0x7f67700721a0 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 0x7f67701457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 0x7f67701457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55942/0 (socket says 172.21.15.78:55942) 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 -- 172.21.15.78:0/1902525342 learned_addr learned my addr 172.21.15.78:0/1902525342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f676ffff640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700721a0 0x7f6770156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f676f7fe640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c6080 0x7f6770151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 -- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700721a0 msgr2=0x7f6770156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700721a0 0x7f6770156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 -- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c6080 msgr2=0x7f6770151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c6080 0x7f6770151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f6774888640 1 -- 172.21.15.78:0/1902525342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640094d0 con 0x7f6770150460 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.775+0000 7f676ffff640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700721a0 0x7f6770156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.776+0000 7f6774888640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 0x7f67701457b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6768007ef0 tx=0x7f676800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.776+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6768003b90 con 0x7f6770150460 2024-02-19T14:53:33.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.776+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676800f040 con 0x7f6770150460 2024-02-19T14:53:33.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.776+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677014d540 con 0x7f6770150460 2024-02-19T14:53:33.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.776+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676800de80 con 0x7f6770150460 2024-02-19T14:53:33.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.776+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6770152440 con 0x7f6770150460 2024-02-19T14:53:33.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.777+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6770070150 con 0x7f6770150460 2024-02-19T14:53:33.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.777+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6768004030 con 0x7f6770150460 2024-02-19T14:53:33.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.777+0000 7f676cff9640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675c06b810 0x7f675c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:33.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.778+0000 7f676ffff640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675c06b810 0x7f675c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:33.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.778+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f676808a1d0 con 0x7f6770150460 2024-02-19T14:53:33.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.778+0000 7f676ffff640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675c06b810 0x7f675c06dcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6760001660 tx=0x7f6760009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:33.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:33.782+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6768054ef0 con 0x7f6770150460 2024-02-19T14:53:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.020+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6770074990 con 0x7f6770150460 2024-02-19T14:53:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.020+0000 7f676cff9640 1 -- 172.21.15.78:0/1902525342 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f67680585b0 con 0x7f6770150460 2024-02-19T14:53:34.022 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:53:34.024 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.026 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:53:34.030 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:53:34.032 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:53:34.034 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675c06b810 msgr2=0x7f675c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675c06b810 0x7f675c06dcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6760001660 tx=0x7f6760009040 comp rx=0 tx=0).stop 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 msgr2=0x7f67701457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 0x7f67701457b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6768007ef0 tx=0x7f676800d4a0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 shutdown_connections 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675c06b810 0x7f675c06dcd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770150460 0x7f67701457b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c6080 0x7f6770151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 --2- 172.21.15.78:0/1902525342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700721a0 0x7f6770156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 >> 172.21.15.78:0/1902525342 conn(0x7f677006c170 msgr2=0x7f677006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 shutdown_connections 2024-02-19T14:53:34.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.022+0000 7f67762eb640 1 -- 172.21.15.78:0/1902525342 wait complete. 2024-02-19T14:53:34.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.133+0000 7fc871c35640 1 Processor -- start 2024-02-19T14:53:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.133+0000 7fc871c35640 1 -- start start 2024-02-19T14:53:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc871c35640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc86c0add70 0x7fc86c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc871c35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc86c0705b0 0x7fc86c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc871c35640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0ae670 con 0x7fc86c0c2f60 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc871c35640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c0716b0 con 0x7fc86c0705b0 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc871c35640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c0c33d0 con 0x7fc86c0add70 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc86bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc86c0705b0 0x7fc86c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc862ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc86c0c2f60 0x7fc86c06eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59146/0 (socket says 172.21.15.78:59146) 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc86bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc86c0705b0 0x7fc86c0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55950/0 (socket says 172.21.15.78:55950) 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.134+0000 7fc862ffd640 1 -- 172.21.15.78:0/3476044634 learned_addr learned my addr 172.21.15.78:0/3476044634 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86b7fe640 1 --2- 172.21.15.78:0/3476044634 >> v2:172.21.15.150:3300/0 conn(0x7fc86c0add70 0x7fc86c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86bfff640 1 -- 172.21.15.78:0/3476044634 >> v2:172.21.15.150:3300/0 conn(0x7fc86c0add70 msgr2=0x7fc86c0c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86bfff640 1 --2- 172.21.15.78:0/3476044634 >> v2:172.21.15.150:3300/0 conn(0x7fc86c0add70 0x7fc86c0c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86bfff640 1 -- 172.21.15.78:0/3476044634 >> v1:172.21.15.78:6789/0 conn(0x7fc86c0c2f60 legacy=0x7fc86c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86bfff640 1 -- 172.21.15.78:0/3476044634 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c153780 con 0x7fc86c0705b0 2024-02-19T14:53:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86b7fe640 1 --2- 172.21.15.78:0/3476044634 >> v2:172.21.15.150:3300/0 conn(0x7fc86c0add70 0x7fc86c0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.135+0000 7fc86bfff640 1 --2- 172.21.15.78:0/3476044634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0705b0 0x7fc86c0709c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0c4fc0 tx=0x7fc86402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42ea7dbb56583bb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc86affd640 1 -- 172.21.15.78:0/3476044634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86402cbf0 con 0x7fc86c0705b0 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc86affd640 1 -- 172.21.15.78:0/3476044634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86402cd50 con 0x7fc86c0705b0 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc86affd640 1 -- 172.21.15.78:0/3476044634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864034660 con 0x7fc86c0705b0 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 -- 172.21.15.78:0/3476044634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0705b0 msgr2=0x7fc86c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 --2- 172.21.15.78:0/3476044634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0705b0 0x7fc86c0709c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0c4fc0 tx=0x7fc86402be40 comp rx=0 tx=0).stop 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 -- 172.21.15.78:0/3476044634 shutdown_connections 2024-02-19T14:53:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 --2- 172.21.15.78:0/3476044634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0705b0 0x7fc86c0709c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 --2- 172.21.15.78:0/3476044634 >> v2:172.21.15.150:3300/0 conn(0x7fc86c0add70 0x7fc86c0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 -- 172.21.15.78:0/3476044634 >> 172.21.15.78:0/3476044634 conn(0x7fc86c0c85e0 msgr2=0x7fc86c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 -- 172.21.15.78:0/3476044634 shutdown_connections 2024-02-19T14:53:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.136+0000 7fc871c35640 1 -- 172.21.15.78:0/3476044634 wait complete. 2024-02-19T14:53:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.137+0000 7fc871c35640 1 Processor -- start 2024-02-19T14:53:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.137+0000 7fc871c35640 1 -- start start 2024-02-19T14:53:34.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc871c35640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c0705b0 0x7fc86c1e9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc871c35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 0x7fc86c1ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc871c35640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc86c0c2f60 0x7fc86c1469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc871c35640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c146510 con 0x7fc86c0add70 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc871c35640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c1476b0 con 0x7fc86c0705b0 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc871c35640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c1ec9f0 con 0x7fc86c0c2f60 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 0x7fc86c1ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 0x7fc86c1ebdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55964/0 (socket says 172.21.15.78:55964) 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 -- 172.21.15.78:0/543955606 learned_addr learned my addr 172.21.15.78:0/543955606 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc86b7fe640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c0705b0 0x7fc86c1e9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc86bfff640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc86c0c2f60 0x7fc86c1469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 -- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc86c0c2f60 msgr2=0x7fc86c1469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc86c0c2f60 0x7fc86c1469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 -- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c0705b0 msgr2=0x7fc86c1e9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c0705b0 0x7fc86c1e9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc862ffd640 1 -- 172.21.15.78:0/543955606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864009a70 con 0x7fc86c0add70 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.138+0000 7fc86b7fe640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c0705b0 0x7fc86c1e9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:34.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.139+0000 7fc862ffd640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 0x7fc86c1ebdd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c400 tx=0x7fc85400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.139+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854005ce0 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.139+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854005e40 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.139+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540072d0 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.139+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c1ec710 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.139+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c1e6120 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.140+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c0676a0 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.140+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc8540074f0 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.141+0000 7fc868ff9640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc84c06b810 0x7fc84c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.141+0000 7fc86b7fe640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc84c06b810 0x7fc84c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.141+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc8540895a0 con 0x7fc86c0add70 2024-02-19T14:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.141+0000 7fc86b7fe640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc84c06b810 0x7fc84c06dcd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00aa30 tx=0x7fc85c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.145+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc854054210 con 0x7fc86c0add70 2024-02-19T14:53:34.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.319+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc86c074990 con 0x7fc86c0add70 2024-02-19T14:53:34.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.320+0000 7fc868ff9640 1 -- 172.21.15.78:0/543955606 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+13152 (secure 0 0 0) 0x7fc8540578d0 con 0x7fc86c0add70 2024-02-19T14:53:34.323 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:34.323 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":47,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:33.651070+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc84c06b810 msgr2=0x7fc84c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc84c06b810 0x7fc84c06dcd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00aa30 tx=0x7fc85c008040 comp rx=0 tx=0).stop 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 msgr2=0x7fc86c1ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 0x7fc86c1ebdd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c400 tx=0x7fc85400c8c0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 shutdown_connections 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc84c06b810 0x7fc84c06dcd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc86c0add70 0x7fc86c1ebdd0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc86c0c2f60 0x7fc86c1469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 --2- 172.21.15.78:0/543955606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc86c0705b0 0x7fc86c1e9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 >> 172.21.15.78:0/543955606 conn(0x7fc86c0c85e0 msgr2=0x7fc86c152390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 shutdown_connections 2024-02-19T14:53:34.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.322+0000 7fc871c35640 1 -- 172.21.15.78:0/543955606 wait complete. 2024-02-19T14:53:34.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:53:34.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:53:34.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.437+0000 7fc5e5039640 1 Processor -- start 2024-02-19T14:53:34.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.437+0000 7fc5e5039640 1 -- start start 2024-02-19T14:53:34.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.437+0000 7fc5e5039640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc5e014ece0 0x7fc5e00c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.437+0000 7fc5e5039640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5e00c5270 0x7fc5e00719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5e5039640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e00c1120 con 0x7fc5e0071f10 2024-02-19T14:53:34.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5e5039640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00c6080 con 0x7fc5e014ece0 2024-02-19T14:53:34.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5e5039640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00c57e0 con 0x7fc5e00c5270 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc5e014ece0 0x7fc5e00c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc5e014ece0 0x7fc5e00c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55976/0 (socket says 172.21.15.78:55976) 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 -- 172.21.15.78:0/1631969210 learned_addr learned my addr 172.21.15.78:0/1631969210 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5d7fff640 1 --2- 172.21.15.78:0/1631969210 >> v2:172.21.15.150:3300/0 conn(0x7fc5e00c5270 0x7fc5e00719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 -- 172.21.15.78:0/1631969210 >> v2:172.21.15.150:3300/0 conn(0x7fc5e00c5270 msgr2=0x7fc5e00719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 --2- 172.21.15.78:0/1631969210 >> v2:172.21.15.150:3300/0 conn(0x7fc5e00c5270 0x7fc5e00719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 -- 172.21.15.78:0/1631969210 >> v1:172.21.15.150:6789/0 conn(0x7fc5e0071f10 legacy=0x7fc5e00c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5ded9d640 1 -- 172.21.15.78:0/1631969210 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00c5ac0 con 0x7fc5e014ece0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.438+0000 7fc5d7fff640 1 --2- 172.21.15.78:0/1631969210 >> v2:172.21.15.150:3300/0 conn(0x7fc5e00c5270 0x7fc5e00719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5ded9d640 1 --2- 172.21.15.78:0/1631969210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e00c4d30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0155b80 tx=0x7fc5d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70fc2e97b4e9cf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5de59c640 1 -- 172.21.15.78:0/1631969210 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0031870 con 0x7fc5e014ece0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5de59c640 1 -- 172.21.15.78:0/1631969210 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d00319d0 con 0x7fc5e014ece0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5de59c640 1 -- 172.21.15.78:0/1631969210 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0031d60 con 0x7fc5e014ece0 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 -- 172.21.15.78:0/1631969210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 msgr2=0x7fc5e00c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 --2- 172.21.15.78:0/1631969210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e00c4d30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0155b80 tx=0x7fc5d002b910 comp rx=0 tx=0).stop 2024-02-19T14:53:34.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 -- 172.21.15.78:0/1631969210 shutdown_connections 2024-02-19T14:53:34.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 --2- 172.21.15.78:0/1631969210 >> v2:172.21.15.150:3300/0 conn(0x7fc5e00c5270 0x7fc5e00719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 --2- 172.21.15.78:0/1631969210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e00c4d30 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 -- 172.21.15.78:0/1631969210 >> 172.21.15.78:0/1631969210 conn(0x7fc5e006b270 msgr2=0x7fc5e006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 -- 172.21.15.78:0/1631969210 shutdown_connections 2024-02-19T14:53:34.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.439+0000 7fc5e5039640 1 -- 172.21.15.78:0/1631969210 wait complete. 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 Processor -- start 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 -- start start 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5e0071f10 0x7fc5e01e5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5e00c5270 0x7fc5e01e55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e01eb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00bdd70 con 0x7fc5e014ece0 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e01e5af0 con 0x7fc5e00c5270 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5e5039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e01e9b00 con 0x7fc5e0071f10 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.441+0000 7fc5df59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e01eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e01eb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55990/0 (socket says 172.21.15.78:55990) 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 -- 172.21.15.78:0/334898647 learned_addr learned my addr 172.21.15.78:0/334898647 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:34.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5d7fff640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5e00c5270 0x7fc5e01e55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5ded9d640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5e0071f10 0x7fc5e01e5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 -- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5e0071f10 msgr2=0x7fc5e01e5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5e0071f10 0x7fc5e01e5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 -- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5e00c5270 msgr2=0x7fc5e01e55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5e00c5270 0x7fc5e01e55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 -- 172.21.15.78:0/334898647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00094d0 con 0x7fc5e014ece0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5df59e640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e01eb680 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c780 tx=0x7fc5d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8004b60 con 0x7fc5e014ece0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d800ce90 con 0x7fc5e014ece0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e01520f0 con 0x7fc5e014ece0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d800a850 con 0x7fc5e014ece0 2024-02-19T14:53:34.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.442+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e00bda90 con 0x7fc5e014ece0 2024-02-19T14:53:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.443+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e0074990 con 0x7fc5e014ece0 2024-02-19T14:53:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.444+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc5d800a9b0 con 0x7fc5e014ece0 2024-02-19T14:53:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.444+0000 7fc5d77fe640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5c006b810 0x7fc5c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.444+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc5d808a240 con 0x7fc5e014ece0 2024-02-19T14:53:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.444+0000 7fc5ded9d640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5c006b810 0x7fc5c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.444+0000 7fc5ded9d640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5c006b810 0x7fc5c006dcd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0036000 tx=0x7fc5d003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.448+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc5d8054e20 con 0x7fc5e014ece0 2024-02-19T14:53:34.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.617+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5e01e9c40 con 0x7fc5e014ece0 2024-02-19T14:53:34.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.618+0000 7fc5d77fe640 1 -- 172.21.15.78:0/334898647 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5d80584e0 con 0x7fc5e014ece0 2024-02-19T14:53:34.621 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5c006b810 msgr2=0x7fc5c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5c006b810 0x7fc5c006dcd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0036000 tx=0x7fc5d003c040 comp rx=0 tx=0).stop 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 msgr2=0x7fc5e01eb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e01eb680 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c780 tx=0x7fc5d800cc40 comp rx=0 tx=0).stop 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 shutdown_connections 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5c006b810 0x7fc5c006dcd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e014ece0 0x7fc5e01eb680 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5e00c5270 0x7fc5e01e55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 --2- 172.21.15.78:0/334898647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5e0071f10 0x7fc5e01e5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 >> 172.21.15.78:0/334898647 conn(0x7fc5e006b270 msgr2=0x7fc5e006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 shutdown_connections 2024-02-19T14:53:34.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.620+0000 7fc5e5039640 1 -- 172.21.15.78:0/334898647 wait complete. 2024-02-19T14:53:34.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:53:34.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.731+0000 7f574fdc9640 1 Processor -- start 2024-02-19T14:53:34.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.731+0000 7f574fdc9640 1 -- start start 2024-02-19T14:53:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574fdc9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f574814ece0 0x7f57480c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574fdc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57480c5270 0x7f57480719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574fdc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480c1120 con 0x7f5748071f10 2024-02-19T14:53:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574fdc9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57480c6080 con 0x7f574814ece0 2024-02-19T14:53:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574fdc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c57e0 con 0x7f57480c5270 2024-02-19T14:53:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574db65640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f574814ece0 0x7f57480c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574e366640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5748071f10 0x7f57480c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59160/0 (socket says 172.21.15.78:59160) 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574e366640 1 -- 172.21.15.78:0/408891316 learned_addr learned my addr 172.21.15.78:0/408891316 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574db65640 1 --2- 172.21.15.78:0/408891316 >> v2:172.21.15.78:3300/0 conn(0x7f574814ece0 0x7f57480c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56002/0 (socket says 172.21.15.78:56002) 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574db65640 1 -- 172.21.15.78:0/408891316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57480c5270 msgr2=0x7f57480719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574db65640 1 --2- 172.21.15.78:0/408891316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57480c5270 0x7f57480719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574db65640 1 -- 172.21.15.78:0/408891316 >> v1:172.21.15.78:6789/0 conn(0x7f5748071f10 legacy=0x7f57480c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.733+0000 7f574db65640 1 -- 172.21.15.78:0/408891316 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480c5ac0 con 0x7f574814ece0 2024-02-19T14:53:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.732+0000 7f574cb63640 1 -- 172.21.15.78:0/408891316 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756302824 0 0) 0x7f57480c1120 con 0x7f5748071f10 2024-02-19T14:53:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.733+0000 7f574db65640 1 --2- 172.21.15.78:0/408891316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574814ece0 0x7f57480c4d30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f574400bfa0 tx=0x7f574402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2d95a416c239c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.733+0000 7f574cb63640 1 -- 172.21.15.78:0/408891316 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574402ea40 con 0x7f574814ece0 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.733+0000 7f574cb63640 1 -- 172.21.15.78:0/408891316 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574402eba0 con 0x7f574814ece0 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.733+0000 7f574cb63640 1 -- 172.21.15.78:0/408891316 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744009540 con 0x7f574814ece0 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 -- 172.21.15.78:0/408891316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574814ece0 msgr2=0x7f57480c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 --2- 172.21.15.78:0/408891316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574814ece0 0x7f57480c4d30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f574400bfa0 tx=0x7f574402e0a0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 -- 172.21.15.78:0/408891316 shutdown_connections 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 --2- 172.21.15.78:0/408891316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57480c5270 0x7f57480719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 --2- 172.21.15.78:0/408891316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574814ece0 0x7f57480c4d30 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 -- 172.21.15.78:0/408891316 >> 172.21.15.78:0/408891316 conn(0x7f574806b270 msgr2=0x7f574806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 -- 172.21.15.78:0/408891316 shutdown_connections 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.734+0000 7f574fdc9640 1 -- 172.21.15.78:0/408891316 wait complete. 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.735+0000 7f574fdc9640 1 Processor -- start 2024-02-19T14:53:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.735+0000 7f574fdc9640 1 -- start start 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 0x7f57480bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c5270 0x7f57481daad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574814ece0 0x7f57481ed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481dc140 con 0x7f5748071f10 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748151d30 con 0x7f57480c5270 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5748155260 con 0x7f574814ece0 2024-02-19T14:53:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 0x7f57480bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 0x7f57480bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56014/0 (socket says 172.21.15.78:56014) 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 -- 172.21.15.78:0/1746603906 learned_addr learned my addr 172.21.15.78:0/1746603906 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 -- 172.21.15.78:0/1746603906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574814ece0 msgr2=0x7f57481ed9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574814ece0 0x7f57481ed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 -- 172.21.15.78:0/1746603906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c5270 msgr2=0x7f57481daad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c5270 0x7f57481daad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 -- 172.21.15.78:0/1746603906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574400bc80 con 0x7f5748071f10 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574db65640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 0x7f57480bfc20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5744000f80 tx=0x7f5744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744036800 con 0x7f5748071f10 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5744036980 con 0x7f5748071f10 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57481eb250 con 0x7f5748071f10 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744036d30 con 0x7f5748071f10 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.736+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57481522b0 con 0x7f5748071f10 2024-02-19T14:53:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.737+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5748074990 con 0x7f5748071f10 2024-02-19T14:53:34.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.738+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f574404a460 con 0x7f5748071f10 2024-02-19T14:53:34.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.738+0000 7f573e7fc640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b810 0x7f572c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:34.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.738+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f57440ad2b0 con 0x7f5748071f10 2024-02-19T14:53:34.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.738+0000 7f574d364640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b810 0x7f572c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:34.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.739+0000 7f574d364640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b810 0x7f572c06dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f57481e82c0 tx=0x7f5738006c90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.743+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5744077f20 con 0x7f5748071f10 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.914+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f57480c4610 con 0x7f5748071f10 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.915+0000 7f573e7fc640 1 -- 172.21.15.78:0/1746603906 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v47) v1 ==== 84+0+65 (secure 0 0 0) 0x7f574407b5e0 con 0x7f5748071f10 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:34.917 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:53:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b810 msgr2=0x7f572c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b810 0x7f572c06dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f57481e82c0 tx=0x7f5738006c90 comp rx=0 tx=0).stop 2024-02-19T14:53:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 msgr2=0x7f57480bfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 0x7f57480bfc20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5744000f80 tx=0x7f5744004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 shutdown_connections 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b810 0x7f572c06dcd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574814ece0 0x7f57481ed9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c5270 0x7f57481daad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 --2- 172.21.15.78:0/1746603906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5748071f10 0x7f57480bfc20 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.916+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 >> 172.21.15.78:0/1746603906 conn(0x7f574806b270 msgr2=0x7f574806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.917+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 shutdown_connections 2024-02-19T14:53:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:34.917+0000 7f574fdc9640 1 -- 172.21.15.78:0/1746603906 wait complete. 2024-02-19T14:53:34.930 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:53:34.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:53:35.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.031+0000 7f5f43f60640 1 Processor -- start 2024-02-19T14:53:35.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.031+0000 7f5f43f60640 1 -- start start 2024-02-19T14:53:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f43f60640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f3c06f170 0x7f5f3c156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f43f60640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3c071050 con 0x7f5f3c156550 2024-02-19T14:53:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f43f60640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3c0c6ea0 con 0x7f5f3c156e70 2024-02-19T14:53:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f43f60640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3c1572c0 con 0x7f5f3c06f170 2024-02-19T14:53:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f414fb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f3c06f170 0x7f5f3c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f41cfc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5f3c156550 0x7f5f3c0ad700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59176/0 (socket says 172.21.15.78:59176) 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f414fb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f3c06f170 0x7f5f3c156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56026/0 (socket says 172.21.15.78:56026) 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.032+0000 7f5f41cfc640 1 -- 172.21.15.78:0/3363915103 learned_addr learned my addr 172.21.15.78:0/3363915103 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f414fb640 1 -- 172.21.15.78:0/3363915103 >> v1:172.21.15.150:6789/0 conn(0x7f5f3c156e70 legacy=0x7f5f3c0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f40cfa640 1 -- 172.21.15.78:0/3363915103 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470160864 0 0) 0x7f5f3c071050 con 0x7f5f3c156550 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f414fb640 1 -- 172.21.15.78:0/3363915103 >> v1:172.21.15.78:6789/0 conn(0x7f5f3c156550 legacy=0x7f5f3c0ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f414fb640 1 -- 172.21.15.78:0/3363915103 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c0c7120 con 0x7f5f3c06f170 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f414fb640 1 --2- 172.21.15.78:0/3363915103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c06f170 0x7f5f3c156930 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0097a0 tx=0x7f5f2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163bc23081a54bfa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f40cfa640 1 -- 172.21.15.78:0/3363915103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c031830 con 0x7f5f3c06f170 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f40cfa640 1 -- 172.21.15.78:0/3363915103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c031990 con 0x7f5f3c06f170 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.033+0000 7f5f40cfa640 1 -- 172.21.15.78:0/3363915103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c031d20 con 0x7f5f3c06f170 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 -- 172.21.15.78:0/3363915103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c06f170 msgr2=0x7f5f3c156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 --2- 172.21.15.78:0/3363915103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c06f170 0x7f5f3c156930 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0097a0 tx=0x7f5f2c02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 -- 172.21.15.78:0/3363915103 shutdown_connections 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 --2- 172.21.15.78:0/3363915103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c06f170 0x7f5f3c156930 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 -- 172.21.15.78:0/3363915103 >> 172.21.15.78:0/3363915103 conn(0x7f5f3c0aa3e0 msgr2=0x7f5f3c0aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 -- 172.21.15.78:0/3363915103 shutdown_connections 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.034+0000 7f5f43f60640 1 -- 172.21.15.78:0/3363915103 wait complete. 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.035+0000 7f5f43f60640 1 Processor -- start 2024-02-19T14:53:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.035+0000 7f5f43f60640 1 -- start start 2024-02-19T14:53:35.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f43f60640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f3c06f170 0x7f5f3c150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f43f60640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c156550 0x7f5f3c1eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f43f60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 0x7f5f3c1eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f43f60640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c151840 con 0x7f5f3c156e70 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f43f60640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c1ee920 con 0x7f5f3c156550 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f43f60640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c1ee010 con 0x7f5f3c06f170 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 0x7f5f3c1eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 0x7f5f3c1eafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56032/0 (socket says 172.21.15.78:56032) 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 -- 172.21.15.78:0/1943415905 learned_addr learned my addr 172.21.15.78:0/1943415905 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f414fb640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c156550 0x7f5f3c1eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f41cfc640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f3c06f170 0x7f5f3c150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 -- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f3c06f170 msgr2=0x7f5f3c150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f3c06f170 0x7f5f3c150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 -- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c156550 msgr2=0x7f5f3c1eaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c156550 0x7f5f3c1eaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 -- 172.21.15.78:0/1943415905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c009400 con 0x7f5f3c156e70 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f414fb640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c156550 0x7f5f3c1eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.036+0000 7f5f424fd640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 0x7f5f3c1eafe0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400c3c0 tx=0x7f5f3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.037+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f340052d0 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.037+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f34005430 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.037+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3c06c130 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.037+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34003810 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.037+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3c1ee540 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.038+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c1eea60 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.038+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5f34003970 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.038+0000 7f5f327fc640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.039+0000 7f5f41cfc640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.039+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5f340894f0 con 0x7f5f3c156e70 2024-02-19T14:53:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.039+0000 7f5f41cfc640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5f380012e0 tx=0x7f5f38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:35.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.043+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5f3c1eea60 con 0x7f5f3c156e70 2024-02-19T14:53:35.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.214+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5f3c1eea60 con 0x7f5f3c156e70 2024-02-19T14:53:35.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.662+0000 7f5f327fc640 1 -- 172.21.15.78:0/1943415905 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v48) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5f3c1eea60 con 0x7f5f3c156e70 2024-02-19T14:53:35.665 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 msgr2=0x7f5f2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5f380012e0 tx=0x7f5f38009040 comp rx=0 tx=0).stop 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 msgr2=0x7f5f3c1eafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 0x7f5f3c1eafe0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400c3c0 tx=0x7f5f3400c880 comp rx=0 tx=0).stop 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 shutdown_connections 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f3c156e70 0x7f5f3c1eafe0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c156550 0x7f5f3c1eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 --2- 172.21.15.78:0/1943415905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f3c06f170 0x7f5f3c150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.665+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 >> 172.21.15.78:0/1943415905 conn(0x7f5f3c0aa3e0 msgr2=0x7f5f3c0a9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.666+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 shutdown_connections 2024-02-19T14:53:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.666+0000 7f5f43f60640 1 -- 172.21.15.78:0/1943415905 wait complete. 2024-02-19T14:53:35.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:53:35.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.787+0000 7f1d148a3640 1 Processor -- start 2024-02-19T14:53:35.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.787+0000 7f1d148a3640 1 -- start start 2024-02-19T14:53:35.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d148a3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d1006c1f0 0x7f1d100c2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d148a3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1d100c2eb0 0x7f1d1006ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d148a3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d10159440 con 0x7f1d1006f2c0 2024-02-19T14:53:35.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d148a3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100c2120 con 0x7f1d100c2eb0 2024-02-19T14:53:35.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d148a3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10158830 con 0x7f1d1006c1f0 2024-02-19T14:53:35.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d0ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1d100c2eb0 0x7f1d1006ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d0ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1d100c2eb0 0x7f1d1006ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56036/0 (socket says 172.21.15.78:56036) 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d0ed9d640 1 -- 172.21.15.78:0/3599185591 learned_addr learned my addr 172.21.15.78:0/3599185591 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.788+0000 7f1d0dd9b640 1 --2- 172.21.15.78:0/3599185591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d1006c1f0 0x7f1d100c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.789+0000 7f1d0ed9d640 1 -- 172.21.15.78:0/3599185591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d1006c1f0 msgr2=0x7f1d100c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.789+0000 7f1d0ed9d640 1 --2- 172.21.15.78:0/3599185591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d1006c1f0 0x7f1d100c2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.789+0000 7f1d0ed9d640 1 -- 172.21.15.78:0/3599185591 >> v1:172.21.15.150:6789/0 conn(0x7f1d1006f2c0 legacy=0x7f1d10155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.789+0000 7f1d0ed9d640 1 -- 172.21.15.78:0/3599185591 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10158b10 con 0x7f1d100c2eb0 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.789+0000 7f1d0ed9d640 1 --2- 172.21.15.78:0/3599185591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d100c2eb0 0x7f1d1006ae80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009870 tx=0x7f1d0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8696bdb20acf83e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.789+0000 7f1d0d59a640 1 -- 172.21.15.78:0/3599185591 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08030840 con 0x7f1d100c2eb0 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d0d59a640 1 -- 172.21.15.78:0/3599185591 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d080309a0 con 0x7f1d100c2eb0 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d0d59a640 1 -- 172.21.15.78:0/3599185591 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08030d30 con 0x7f1d100c2eb0 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 -- 172.21.15.78:0/3599185591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d100c2eb0 msgr2=0x7f1d1006ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 --2- 172.21.15.78:0/3599185591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d100c2eb0 0x7f1d1006ae80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009870 tx=0x7f1d0802b980 comp rx=0 tx=0).stop 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 -- 172.21.15.78:0/3599185591 shutdown_connections 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 --2- 172.21.15.78:0/3599185591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d100c2eb0 0x7f1d1006ae80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 --2- 172.21.15.78:0/3599185591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d1006c1f0 0x7f1d100c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 -- 172.21.15.78:0/3599185591 >> 172.21.15.78:0/3599185591 conn(0x7f1d100c3970 msgr2=0x7f1d100c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 -- 172.21.15.78:0/3599185591 shutdown_connections 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.790+0000 7f1d148a3640 1 -- 172.21.15.78:0/3599185591 wait complete. 2024-02-19T14:53:35.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.791+0000 7f1d148a3640 1 Processor -- start 2024-02-19T14:53:35.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.791+0000 7f1d148a3640 1 -- start start 2024-02-19T14:53:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d148a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1006c1f0 0x7f1d10137b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d148a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 0x7f1d10155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d148a3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d100c2eb0 0x7f1d10156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d148a3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101ec270 con 0x7f1d1006f2c0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d148a3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101384e0 con 0x7f1d1006c1f0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d148a3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1013d0d0 con 0x7f1d100c2eb0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 0x7f1d10155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 0x7f1d10155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56048/0 (socket says 172.21.15.78:56048) 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 -- 172.21.15.78:0/1799674689 learned_addr learned my addr 172.21.15.78:0/1799674689 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0ed9d640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d100c2eb0 0x7f1d10156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0e59c640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1006c1f0 0x7f1d10137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 -- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d100c2eb0 msgr2=0x7f1d10156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d100c2eb0 0x7f1d10156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 -- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1006c1f0 msgr2=0x7f1d10137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1006c1f0 0x7f1d10137b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0dd9b640 1 -- 172.21.15.78:0/1799674689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d080094d0 con 0x7f1d1006f2c0 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0e59c640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1006c1f0 0x7f1d10137b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.792+0000 7f1d0ed9d640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d100c2eb0 0x7f1d10156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.793+0000 7f1d0dd9b640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 0x7f1d10155ec0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400e9e0 tx=0x7f1d0400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.793+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04009670 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.793+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0400c9d0 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.793+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400cd80 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.793+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d101ebbb0 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.793+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d10138040 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.794+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1006c820 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.794+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1d04013470 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.795+0000 7f1cfeffd640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cf006b810 0x7f1cf006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.795+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f1d0408a3b0 con 0x7f1d1006f2c0 2024-02-19T14:53:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.795+0000 7f1d0e59c640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cf006b810 0x7f1cf006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.795+0000 7f1d0e59c640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cf006b810 0x7f1cf006dcd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1d00001e00 tx=0x7f1d00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.800+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1d04054e20 con 0x7f1d1006f2c0 2024-02-19T14:53:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:35.971+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1d10074990 con 0x7f1d1006f2c0 2024-02-19T14:53:36.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.668+0000 7f1cfeffd640 1 -- 172.21.15.78:0/1799674689 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v49) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1d040584e0 con 0x7f1d1006f2c0 2024-02-19T14:53:36.670 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:53:36.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cf006b810 msgr2=0x7f1cf006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:36.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cf006b810 0x7f1cf006dcd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1d00001e00 tx=0x7f1d00009040 comp rx=0 tx=0).stop 2024-02-19T14:53:36.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 msgr2=0x7f1d10155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 0x7f1d10155ec0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400e9e0 tx=0x7f1d0400eea0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 shutdown_connections 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cf006b810 0x7f1cf006dcd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d1006f2c0 0x7f1d10155ec0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d100c2eb0 0x7f1d10156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 --2- 172.21.15.78:0/1799674689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1006c1f0 0x7f1d10137b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.670+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 >> 172.21.15.78:0/1799674689 conn(0x7f1d100c3970 msgr2=0x7f1d1013ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.671+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 shutdown_connections 2024-02-19T14:53:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.671+0000 7f1d148a3640 1 -- 172.21.15.78:0/1799674689 wait complete. 2024-02-19T14:53:36.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:53:36.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.800+0000 7fa516074640 1 Processor -- start 2024-02-19T14:53:36.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.800+0000 7fa516074640 1 -- start start 2024-02-19T14:53:36.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.801+0000 7fa516074640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 0x7fa5100c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.801+0000 7fa516074640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5100c2f60 0x7fa51006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.801+0000 7fa516074640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5100ae670 con 0x7fa51006f530 2024-02-19T14:53:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.801+0000 7fa516074640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5100716b0 con 0x7fa5100c2f60 2024-02-19T14:53:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.801+0000 7fa516074640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100c33a0 con 0x7fa5100add70 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.801+0000 7fa50f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 0x7fa5100c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5100c2f60 0x7fa51006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa51006f530 0x7fa5100705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:34522/0 (socket says 172.21.15.78:34522) 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 0x7fa5100c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:42512/0 (socket says 172.21.15.78:42512) 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50ffff640 1 -- 172.21.15.78:0/931882789 learned_addr learned my addr 172.21.15.78:0/931882789 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50e7fc640 1 -- 172.21.15.78:0/931882789 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938160292 0 0) 0x7fa5100ae670 con 0x7fa51006f530 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50e7fc640 1 -- 172.21.15.78:0/931882789 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4fc003520 con 0x7fa51006f530 2024-02-19T14:53:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50f7fe640 1 -- 172.21.15.78:0/931882789 >> v2:172.21.15.150:3300/0 conn(0x7fa5100c2f60 msgr2=0x7fa51006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50f7fe640 1 --2- 172.21.15.78:0/931882789 >> v2:172.21.15.150:3300/0 conn(0x7fa5100c2f60 0x7fa51006eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50f7fe640 1 -- 172.21.15.78:0/931882789 >> v1:172.21.15.150:6789/0 conn(0x7fa51006f530 legacy=0x7fa5100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50f7fe640 1 -- 172.21.15.78:0/931882789 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510153780 con 0x7fa5100add70 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.802+0000 7fa50effd640 1 --2- 172.21.15.78:0/931882789 >> v2:172.21.15.150:3300/0 conn(0x7fa5100c2f60 0x7fa51006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa50f7fe640 1 --2- 172.21.15.78:0/931882789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 0x7fa5100c4980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa500009d90 tx=0x7fa50002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c030f23a9a437fb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa50e7fc640 1 -- 172.21.15.78:0/931882789 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50002cbc0 con 0x7fa5100add70 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa50e7fc640 1 -- 172.21.15.78:0/931882789 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50002cd20 con 0x7fa5100add70 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa50e7fc640 1 -- 172.21.15.78:0/931882789 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500034640 con 0x7fa5100add70 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa516074640 1 -- 172.21.15.78:0/931882789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 msgr2=0x7fa5100c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa516074640 1 --2- 172.21.15.78:0/931882789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 0x7fa5100c4980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa500009d90 tx=0x7fa50002be10 comp rx=0 tx=0).stop 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa516074640 1 -- 172.21.15.78:0/931882789 shutdown_connections 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa516074640 1 --2- 172.21.15.78:0/931882789 >> v2:172.21.15.150:3300/0 conn(0x7fa5100c2f60 0x7fa51006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa516074640 1 --2- 172.21.15.78:0/931882789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100add70 0x7fa5100c4980 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.803+0000 7fa516074640 1 -- 172.21.15.78:0/931882789 >> 172.21.15.78:0/931882789 conn(0x7fa5100c85e0 msgr2=0x7fa5100c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.804+0000 7fa516074640 1 -- 172.21.15.78:0/931882789 shutdown_connections 2024-02-19T14:53:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.804+0000 7fa516074640 1 -- 172.21.15.78:0/931882789 wait complete. 2024-02-19T14:53:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 Processor -- start 2024-02-19T14:53:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 -- start start 2024-02-19T14:53:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 0x7fa51009a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5100add70 0x7fa5101edae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100c2f60 0x7fa5101519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa510151570 con 0x7fa51006f530 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51009a750 con 0x7fa5100add70 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.805+0000 7fa516074640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101ee560 con 0x7fa5100c2f60 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 0x7fa51009a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 0x7fa51009a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56056/0 (socket says 172.21.15.78:56056) 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 -- 172.21.15.78:0/3262329439 learned_addr learned my addr 172.21.15.78:0/3262329439 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50effd640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5100add70 0x7fa5101edae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50ffff640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100c2f60 0x7fa5101519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 -- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100c2f60 msgr2=0x7fa5101519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100c2f60 0x7fa5101519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 -- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5100add70 msgr2=0x7fa5101edae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5100add70 0x7fa5101edae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 -- 172.21.15.78:0/3262329439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8009400 con 0x7fa51006f530 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa50f7fe640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 0x7fa51009a210 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa500037040 tx=0x7fa500004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500040650 con 0x7fa51006f530 2024-02-19T14:53:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50002ce90 con 0x7fa51006f530 2024-02-19T14:53:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500040a30 con 0x7fa51006f530 2024-02-19T14:53:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa500009a70 con 0x7fa51006f530 2024-02-19T14:53:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.806+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5101544e0 con 0x7fa51006f530 2024-02-19T14:53:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.807+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5100676a0 con 0x7fa51006f530 2024-02-19T14:53:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.808+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa500040b90 con 0x7fa51006f530 2024-02-19T14:53:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.808+0000 7fa507fff640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4f006b840 0x7fa4f006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.808+0000 7fa50effd640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4f006b840 0x7fa4f006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.808+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fa5000ace40 con 0x7fa51006f530 2024-02-19T14:53:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.809+0000 7fa50effd640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4f006b840 0x7fa4f006dd00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa5100a8130 tx=0x7fa4f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:36.813+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa5000777c0 con 0x7fa51006f530 2024-02-19T14:53:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.025+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa510154670 con 0x7fa51006f530 2024-02-19T14:53:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.669+0000 7fa507fff640 1 -- 172.21.15.78:0/3262329439 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v24) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa500009450 con 0x7fa51006f530 2024-02-19T14:53:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 7 and data pool 8 2024-02-19T14:53:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4f006b840 msgr2=0x7fa4f006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4f006b840 0x7fa4f006dd00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa5100a8130 tx=0x7fa4f8008040 comp rx=0 tx=0).stop 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 msgr2=0x7fa51009a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 0x7fa51009a210 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa500037040 tx=0x7fa500004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 shutdown_connections 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4f006b840 0x7fa4f006dd00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5100c2f60 0x7fa5101519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5100add70 0x7fa5101edae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 --2- 172.21.15.78:0/3262329439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51006f530 0x7fa51009a210 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 >> 172.21.15.78:0/3262329439 conn(0x7fa5100c85e0 msgr2=0x7fa5100c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 shutdown_connections 2024-02-19T14:53:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.674+0000 7fa516074640 1 -- 172.21.15.78:0/3262329439 wait complete. 2024-02-19T14:53:37.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:37.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.803+0000 7ffad55a1640 1 Processor -- start 2024-02-19T14:53:37.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.803+0000 7ffad55a1640 1 -- start start 2024-02-19T14:53:37.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffad55a1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffad014f1a0 0x7ffad0072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:37.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffad55a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad0152680 0x7ffad00c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:37.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffad55a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00c5970 con 0x7ffad006ea50 2024-02-19T14:53:37.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffad55a1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad0152af0 con 0x7ffad014f1a0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffad55a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00c5750 con 0x7ffad0152680 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffad014f1a0 0x7ffad0072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffad014f1a0 0x7ffad0072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50770/0 (socket says 172.21.15.78:50770) 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 -- 172.21.15.78:0/3742838268 learned_addr learned my addr 172.21.15.78:0/3742838268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffacf7fe640 1 --2- 172.21.15.78:0/3742838268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad0152680 0x7ffad00c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 -- 172.21.15.78:0/3742838268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad0152680 msgr2=0x7ffad00c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 --2- 172.21.15.78:0/3742838268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad0152680 0x7ffad00c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 -- 172.21.15.78:0/3742838268 >> v1:172.21.15.150:6789/0 conn(0x7ffad006ea50 legacy=0x7ffad006fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.804+0000 7ffaceffd640 1 -- 172.21.15.78:0/3742838268 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00702d0 con 0x7ffad014f1a0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.805+0000 7ffaceffd640 1 --2- 172.21.15.78:0/3742838268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad014f1a0 0x7ffad0072d70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009870 tx=0x7ffac802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b48723c15676fa2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.805+0000 7ffacdffb640 1 -- 172.21.15.78:0/3742838268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80317c0 con 0x7ffad014f1a0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.805+0000 7ffacdffb640 1 -- 172.21.15.78:0/3742838268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac8031920 con 0x7ffad014f1a0 2024-02-19T14:53:37.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.805+0000 7ffacdffb640 1 -- 172.21.15.78:0/3742838268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8031cb0 con 0x7ffad014f1a0 2024-02-19T14:53:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.805+0000 7ffad55a1640 1 -- 172.21.15.78:0/3742838268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad014f1a0 msgr2=0x7ffad0072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.805+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3742838268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad014f1a0 0x7ffad0072d70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009870 tx=0x7ffac802b860 comp rx=0 tx=0).stop 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.806+0000 7ffad55a1640 1 -- 172.21.15.78:0/3742838268 shutdown_connections 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.806+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3742838268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad0152680 0x7ffad00c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.806+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3742838268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad014f1a0 0x7ffad0072d70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.806+0000 7ffad55a1640 1 -- 172.21.15.78:0/3742838268 >> 172.21.15.78:0/3742838268 conn(0x7ffad006c170 msgr2=0x7ffad006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.806+0000 7ffad55a1640 1 -- 172.21.15.78:0/3742838268 shutdown_connections 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.806+0000 7ffad55a1640 1 -- 172.21.15.78:0/3742838268 wait complete. 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 Processor -- start 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 -- start start 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad006ea50 0x7ffad00c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad014f1a0 0x7ffad014fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 0x7ffad0145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00ac590 con 0x7ffad0152680 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0146190 con 0x7ffad006ea50 2024-02-19T14:53:37.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.807+0000 7ffad55a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01ee5a0 con 0x7ffad014f1a0 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 0x7ffad0145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 0x7ffad0145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50784/0 (socket says 172.21.15.78:50784) 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 -- 172.21.15.78:0/3994291467 learned_addr learned my addr 172.21.15.78:0/3994291467 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7fface7fc640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad014f1a0 0x7ffad014fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffaceffd640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad006ea50 0x7ffad00c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 -- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad014f1a0 msgr2=0x7ffad014fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad014f1a0 0x7ffad014fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 -- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad006ea50 msgr2=0x7ffad00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad006ea50 0x7ffad00c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 -- 172.21.15.78:0/3994291467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80094d0 con 0x7ffad0152680 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffacf7fe640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 0x7ffad0145c50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffac000d800 tx=0x7ffac000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac000b6c0 con 0x7ffad0152680 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac000b840 con 0x7ffad0152680 2024-02-19T14:53:37.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac000bbf0 con 0x7ffad0152680 2024-02-19T14:53:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad01ee880 con 0x7ffad0152680 2024-02-19T14:53:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.808+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad0150800 con 0x7ffad0152680 2024-02-19T14:53:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.809+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad0074990 con 0x7ffad0152680 2024-02-19T14:53:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.810+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffac000bd50 con 0x7ffad0152680 2024-02-19T14:53:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.810+0000 7ffab77fe640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffabc06b810 0x7ffabc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.810+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffac008a6c0 con 0x7ffad0152680 2024-02-19T14:53:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.810+0000 7ffaceffd640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffabc06b810 0x7ffabc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.811+0000 7ffaceffd640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffabc06b810 0x7ffabc06dcd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffac8036000 tx=0x7ffac803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.815+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffac0055090 con 0x7ffad0152680 2024-02-19T14:53:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.986+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad006e830 con 0x7ffad0152680 2024-02-19T14:53:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.987+0000 7ffab77fe640 1 -- 172.21.15.78:0/3994291467 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16943 (secure 0 0 0) 0x7ffac0058750 con 0x7ffad0152680 2024-02-19T14:53:37.989 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:37.990 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":50,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:37.026533+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffabc06b810 msgr2=0x7ffabc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffabc06b810 0x7ffabc06dcd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffac8036000 tx=0x7ffac803d040 comp rx=0 tx=0).stop 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 msgr2=0x7ffad0145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 0x7ffad0145c50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffac000d800 tx=0x7ffac000dcc0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 shutdown_connections 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffabc06b810 0x7ffabc06dcd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffad0152680 0x7ffad0145c50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffad014f1a0 0x7ffad014fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 --2- 172.21.15.78:0/3994291467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad006ea50 0x7ffad00c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 >> 172.21.15.78:0/3994291467 conn(0x7ffad006c170 msgr2=0x7ffad00a85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 shutdown_connections 2024-02-19T14:53:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:37.989+0000 7ffad55a1640 1 -- 172.21.15.78:0/3994291467 wait complete. 2024-02-19T14:53:38.003 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.099+0000 7f9045bf7640 1 Processor -- start 2024-02-19T14:53:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.099+0000 7f9045bf7640 1 -- start start 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.100+0000 7f9045bf7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f904014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.100+0000 7f9045bf7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904014f8f0 con 0x7f90400c78a0 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.100+0000 7f9045bf7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90400c35c0 con 0x7f90400727d0 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.100+0000 7f9045bf7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90400c4bc0 con 0x7f904014ef50 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.100+0000 7f903f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f90400c78a0 0x7f90400c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42082/0 (socket says 172.21.15.78:42082) 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.100+0000 7f903f7fe640 1 -- 172.21.15.78:0/3805078066 learned_addr learned my addr 172.21.15.78:0/3805078066 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f904014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923523410 0 0) 0x7f904014f8f0 con 0x7f90400c78a0 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902c003520 con 0x7f90400c78a0 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870428724 0 0) 0x7f902c003520 con 0x7f90400c78a0 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 msgr2=0x7f904014f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f904014f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 >> v1:172.21.15.150:6789/0 conn(0x7f90400727d0 legacy=0x7f9040072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90400c38a0 con 0x7f90400c78a0 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9030001670 con 0x7f90400c78a0 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 >> v1:172.21.15.78:6789/0 conn(0x7f90400c78a0 legacy=0x7f90400c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f902c0039a0 0x7f902c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c004350 0x7f902c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f902c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90400c4bc0 con 0x7f902c004350 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c00b980 con 0x7f902c0039a0 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c00bb40 con 0x7f904014ef50 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c004350 0x7f902c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903f7fe640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f902c0039a0 0x7f902c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903ffff640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f902c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 -- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 msgr2=0x7f902c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f902c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 -- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f902c0039a0 msgr2=0x7f902c003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f902c0039a0 0x7f902c003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f9036ffd640 1 -- 172.21.15.78:0/3805078066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c003520 con 0x7f902c004350 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.101+0000 7f903ffff640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f904014ef50 0x7f902c005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.102+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c004350 0x7f902c004820 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9028009ac0 tx=0x7f902c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15b5295dfac1902 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.102+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902802c6b0 con 0x7f902c004350 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.102+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902802c810 con 0x7f902c004350 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.102+0000 7f903effd640 1 -- 172.21.15.78:0/3805078066 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902802cba0 con 0x7f902c004350 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f9045bf7640 1 -- 172.21.15.78:0/3805078066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c004350 msgr2=0x7f902c004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3805078066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f902c004350 0x7f902c004820 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9028009ac0 tx=0x7f902c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f903ffff640 1 -- 172.21.15.78:0/3805078066 reap_dead start 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f9045bf7640 1 -- 172.21.15.78:0/3805078066 shutdown_connections 2024-02-19T14:53:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f9045bf7640 1 -- 172.21.15.78:0/3805078066 >> 172.21.15.78:0/3805078066 conn(0x7f9040066e70 msgr2=0x7f90400a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f9045bf7640 1 -- 172.21.15.78:0/3805078066 shutdown_connections 2024-02-19T14:53:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.103+0000 7f9045bf7640 1 -- 172.21.15.78:0/3805078066 wait complete. 2024-02-19T14:53:38.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.104+0000 7f9045bf7640 1 Processor -- start 2024-02-19T14:53:38.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.104+0000 7f9045bf7640 1 -- start start 2024-02-19T14:53:38.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f902c004350 0x7f9040120820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 0x7f90400a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9040144dc0 0x7f9040145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9040120f90 con 0x7f9040121e00 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90401e6e30 con 0x7f9040144dc0 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90401e7e00 con 0x7f902c004350 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 0x7f90400a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 0x7f90400a96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50794/0 (socket says 172.21.15.78:50794) 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 -- 172.21.15.78:0/3705537510 learned_addr learned my addr 172.21.15.78:0/3705537510 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 -- 172.21.15.78:0/3705537510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f902c004350 msgr2=0x7f9040120820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f902c004350 0x7f9040120820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 -- 172.21.15.78:0/3705537510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9040144dc0 msgr2=0x7f9040145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9040144dc0 0x7f9040145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 -- 172.21.15.78:0/3705537510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9040120d60 con 0x7f9040121e00 2024-02-19T14:53:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9036ffd640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 0x7f90400a96c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9028009be0 tx=0x7f9028004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028003810 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9028003970 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90401e7040 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028003d20 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.105+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90401e7600 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.106+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9040074990 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.107+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9028047450 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.107+0000 7f903cff9640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f902006b810 0x7f902006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.107+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90280ac250 con 0x7f9040121e00 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.107+0000 7f903f7fe640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f902006b810 0x7f902006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.108+0000 7f903f7fe640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f902006b810 0x7f902006dcd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f90300017a0 tx=0x7f9030001450 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.112+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f902807a7c0 con 0x7f9040121e00 2024-02-19T14:53:38.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.282+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90400aa000 con 0x7f9040121e00 2024-02-19T14:53:38.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.283+0000 7f903cff9640 1 -- 172.21.15.78:0/3705537510 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16943 (secure 0 0 0) 0x7f902807a350 con 0x7f9040121e00 2024-02-19T14:53:38.286 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:38.286 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":50,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:37.026533+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f902006b810 msgr2=0x7f902006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f902006b810 0x7f902006dcd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f90300017a0 tx=0x7f9030001450 comp rx=0 tx=0).stop 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 msgr2=0x7f90400a96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 0x7f90400a96c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9028009be0 tx=0x7f9028004730 comp rx=0 tx=0).stop 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 shutdown_connections 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f902006b810 0x7f902006dcd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9040121e00 0x7f90400a96c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9040144dc0 0x7f9040145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 --2- 172.21.15.78:0/3705537510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f902c004350 0x7f9040120820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 >> 172.21.15.78:0/3705537510 conn(0x7f9040066e70 msgr2=0x7f90400ac760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 shutdown_connections 2024-02-19T14:53:38.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.285+0000 7f9045bf7640 1 -- 172.21.15.78:0/3705537510 wait complete. 2024-02-19T14:53:38.299 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:53:38.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.399+0000 7f5c0b56c640 1 Processor -- start 2024-02-19T14:53:38.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c0b56c640 1 -- start start 2024-02-19T14:53:38.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c0b56c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c78a0 0x7f5c04159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c0b56c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c040c4d80 0x7f5c040c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c0b56c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0406ffa0 con 0x7f5c040c48f0 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c0b56c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0406b720 con 0x7f5c040c4d80 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c0b56c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0406fc70 con 0x7f5c040c78a0 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c08b07640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c040c4d80 0x7f5c040c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c08b07640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c040c4d80 0x7f5c040c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50800/0 (socket says 172.21.15.78:50800) 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.400+0000 7f5c08b07640 1 -- 172.21.15.78:0/1958856177 learned_addr learned my addr 172.21.15.78:0/1958856177 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.401+0000 7f5c08b07640 1 -- 172.21.15.78:0/1958856177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c78a0 msgr2=0x7f5c04159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.401+0000 7f5c08b07640 1 --2- 172.21.15.78:0/1958856177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c78a0 0x7f5c04159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.401+0000 7f5c08b07640 1 -- 172.21.15.78:0/1958856177 >> v1:172.21.15.150:6789/0 conn(0x7f5c040c48f0 legacy=0x7f5c041523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.401+0000 7f5c08b07640 1 -- 172.21.15.78:0/1958856177 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0406b930 con 0x7f5c040c4d80 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.401+0000 7f5c08b07640 1 --2- 172.21.15.78:0/1958856177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c4d80 0x7f5c040c43b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009870 tx=0x7f5c0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6641467af628a77 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5bfbfff640 1 -- 172.21.15.78:0/1958856177 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c000317c0 con 0x7f5c040c4d80 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5bfbfff640 1 -- 172.21.15.78:0/1958856177 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c00031920 con 0x7f5c040c4d80 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5bfbfff640 1 -- 172.21.15.78:0/1958856177 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00031cb0 con 0x7f5c040c4d80 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 -- 172.21.15.78:0/1958856177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c4d80 msgr2=0x7f5c040c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/1958856177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c4d80 0x7f5c040c43b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009870 tx=0x7f5c0002b860 comp rx=0 tx=0).stop 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 -- 172.21.15.78:0/1958856177 shutdown_connections 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/1958856177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c4d80 0x7f5c040c43b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/1958856177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c78a0 0x7f5c04159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 -- 172.21.15.78:0/1958856177 >> 172.21.15.78:0/1958856177 conn(0x7f5c04066e70 msgr2=0x7f5c040a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 -- 172.21.15.78:0/1958856177 shutdown_connections 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.402+0000 7f5c0b56c640 1 -- 172.21.15.78:0/1958856177 wait complete. 2024-02-19T14:53:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.403+0000 7f5c0b56c640 1 Processor -- start 2024-02-19T14:53:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.403+0000 7f5c0b56c640 1 -- start start 2024-02-19T14:53:38.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c0b56c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 0x7f5c040abe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c0b56c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c4d80 0x7f5c0414c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c0b56c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c040c78a0 0x7f5c0414c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c0b56c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c04156fe0 con 0x7f5c040c48f0 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c0b56c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0414cdf0 con 0x7f5c040c78a0 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c0b56c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c04157f80 con 0x7f5c040c4d80 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 0x7f5c040abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 0x7f5c040abe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50812/0 (socket says 172.21.15.78:50812) 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 -- 172.21.15.78:0/3101304607 learned_addr learned my addr 172.21.15.78:0/3101304607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09b09640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c040c78a0 0x7f5c0414c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c08b07640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c4d80 0x7f5c0414c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 -- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c4d80 msgr2=0x7f5c0414c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c4d80 0x7f5c0414c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 -- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c040c78a0 msgr2=0x7f5c0414c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c040c78a0 0x7f5c0414c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09308640 1 -- 172.21.15.78:0/3101304607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000094d0 con 0x7f5c040c48f0 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.404+0000 7f5c09b09640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c040c78a0 0x7f5c0414c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.405+0000 7f5c09308640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 0x7f5c040abe90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400b440 tx=0x7f5bf400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.405+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4011890 con 0x7f5c040c48f0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.405+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf40119f0 con 0x7f5c040c48f0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.405+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c040c6cf0 con 0x7f5c040c48f0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.405+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4011da0 con 0x7f5c040c48f0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.405+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c04159250 con 0x7f5c040c48f0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.406+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c041ea510 con 0x7f5c040c48f0 2024-02-19T14:53:38.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.406+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5bf401a400 con 0x7f5c040c48f0 2024-02-19T14:53:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.407+0000 7f5bf9ffb640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5be806b840 0x7f5be806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.407+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5bf408a620 con 0x7f5c040c48f0 2024-02-19T14:53:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.407+0000 7f5c08b07640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5be806b840 0x7f5be806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.407+0000 7f5c08b07640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5be806b840 0x7f5be806dd00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c00036000 tx=0x7f5c0003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.411+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5bf4054f40 con 0x7f5c040c48f0 2024-02-19T14:53:38.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.619+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5c04074990 con 0x7f5c040c48f0 2024-02-19T14:53:38.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.685+0000 7f5bf9ffb640 1 -- 172.21.15.78:0/3101304607 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5bf4058600 con 0x7f5c040c48f0 2024-02-19T14:53:38.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5be806b840 msgr2=0x7f5be806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5be806b840 0x7f5be806dd00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c00036000 tx=0x7f5c0003d040 comp rx=0 tx=0).stop 2024-02-19T14:53:38.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 msgr2=0x7f5c040abe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 0x7f5c040abe90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400b440 tx=0x7f5bf400b900 comp rx=0 tx=0).stop 2024-02-19T14:53:38.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 shutdown_connections 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5be806b840 0x7f5be806dd00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c040c78a0 0x7f5c0414c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c040c4d80 0x7f5c0414c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 --2- 172.21.15.78:0/3101304607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c040c48f0 0x7f5c040abe90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 >> 172.21.15.78:0/3101304607 conn(0x7f5c04066e70 msgr2=0x7f5c040a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 shutdown_connections 2024-02-19T14:53:38.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.688+0000 7f5c0b56c640 1 -- 172.21.15.78:0/3101304607 wait complete. 2024-02-19T14:53:38.701 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:53:38.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:53:38.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.802+0000 7f43c6ed4640 1 Processor -- start 2024-02-19T14:53:38.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.802+0000 7f43c6ed4640 1 -- start start 2024-02-19T14:53:38.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.803+0000 7f43c6ed4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43c0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.803+0000 7f43c6ed4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c00ae670 con 0x7f43c00add70 2024-02-19T14:53:38.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.803+0000 7f43c6ed4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c00716b0 con 0x7f43c00c4980 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.803+0000 7f43c6ed4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c00c33b0 con 0x7f43c00c2f60 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.803+0000 7f43c4c70640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f43c00add70 0x7f43c006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42094/0 (socket says 172.21.15.78:42094) 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.803+0000 7f43c4c70640 1 -- 172.21.15.78:0/3138865770 learned_addr learned my addr 172.21.15.78:0/3138865770 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c5471640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43c0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423988500 0 0) 0x7f43c00ae670 con 0x7f43c00add70 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b0003520 con 0x7f43c00add70 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3965428919 0 0) 0x7f43b0003520 con 0x7f43c00add70 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 msgr2=0x7f43c0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43c0070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 >> v1:172.21.15.150:6789/0 conn(0x7f43c00c4980 legacy=0x7f43c00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c0153780 con 0x7f43c00add70 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f43b4001670 con 0x7f43c00add70 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 >> v1:172.21.15.78:6789/0 conn(0x7f43c00add70 legacy=0x7f43c006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43b0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0004350 0x7f43b0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43b0005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c00c33b0 con 0x7f43b00039a0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b000ba30 con 0x7f43b0004350 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b000bbf0 con 0x7f43c00c2f60 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c4c70640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43b0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bffff640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0004350 0x7f43b0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c5471640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43b0005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c4c70640 1 -- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 msgr2=0x7f43b0005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c4c70640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43b0005120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c4c70640 1 -- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0004350 msgr2=0x7f43b0004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c4c70640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0004350 0x7f43b0004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c4c70640 1 -- 172.21.15.78:0/3138865770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0003520 con 0x7f43b00039a0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43c5471640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c00c2f60 0x7f43b0005120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.804+0000 7f43bffff640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43b0004350 0x7f43b0004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.805+0000 7f43c4c70640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43b0003d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c210 tx=0x7f43b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6f3f0a996cf62e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.805+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b402cae0 con 0x7f43b00039a0 2024-02-19T14:53:38.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.805+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b402cc60 con 0x7f43b00039a0 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.805+0000 7f43bf7fe640 1 -- 172.21.15.78:0/3138865770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b40058d0 con 0x7f43b00039a0 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c6ed4640 1 -- 172.21.15.78:0/3138865770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 msgr2=0x7f43b0003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/3138865770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43b0003d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c210 tx=0x7f43b00057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c5471640 1 -- 172.21.15.78:0/3138865770 reap_dead start 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c6ed4640 1 -- 172.21.15.78:0/3138865770 shutdown_connections 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c6ed4640 1 -- 172.21.15.78:0/3138865770 >> 172.21.15.78:0/3138865770 conn(0x7f43c00c85e0 msgr2=0x7f43c00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c6ed4640 1 -- 172.21.15.78:0/3138865770 shutdown_connections 2024-02-19T14:53:38.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.806+0000 7f43c6ed4640 1 -- 172.21.15.78:0/3138865770 wait complete. 2024-02-19T14:53:38.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.807+0000 7f43c6ed4640 1 Processor -- start 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.807+0000 7f43c6ed4640 1 -- start start 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43c006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c0156cb0 0x7f43c00c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43c00bef70 0x7f43c00c2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c01ec480 con 0x7f43b00039a0 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c01eb870 con 0x7f43c00bef70 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c01ebb50 con 0x7f43c0156cb0 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43c006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43c006f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50830/0 (socket says 172.21.15.78:50830) 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 -- 172.21.15.78:0/287185635 learned_addr learned my addr 172.21.15.78:0/287185635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43bffff640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c0156cb0 0x7f43c00c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c5471640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43c00bef70 0x7f43c00c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 -- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c0156cb0 msgr2=0x7f43c00c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c0156cb0 0x7f43c00c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 -- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43c00bef70 msgr2=0x7f43c00c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43c00bef70 0x7f43c00c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 -- 172.21.15.78:0/287185635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c01ebe30 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c4c70640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43c006f260 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c210 tx=0x7f43b4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b403c040 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b402cae0 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b4001630 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c01ec110 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.808+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c0154330 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.809+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c0074990 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.810+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f43b4032070 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.810+0000 7f43bd7fa640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43a406b840 0x7f43a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.810+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43b40ab510 con 0x7f43b00039a0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.810+0000 7f43bffff640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43a406b840 0x7f43a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:38.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.811+0000 7f43bffff640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43a406b840 0x7f43a406dd00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f43c00a8020 tx=0x7f43ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:38.815+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f43b4075d90 con 0x7f43b00039a0 2024-02-19T14:53:39.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.025+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f43c0071490 con 0x7f43b00039a0 2024-02-19T14:53:39.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.690+0000 7f43bd7fa640 1 -- 172.21.15.78:0/287185635 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f43b4079450 con 0x7f43b00039a0 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43a406b840 msgr2=0x7f43a406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43a406b840 0x7f43a406dd00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f43c00a8020 tx=0x7f43ac009040 comp rx=0 tx=0).stop 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 msgr2=0x7f43c006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43c006f260 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c210 tx=0x7f43b4001a00 comp rx=0 tx=0).stop 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 shutdown_connections 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43a406b840 0x7f43a406dd00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43c00bef70 0x7f43c00c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43c0156cb0 0x7f43c00c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 --2- 172.21.15.78:0/287185635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43b00039a0 0x7f43c006f260 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 >> 172.21.15.78:0/287185635 conn(0x7f43c00c85e0 msgr2=0x7f43c00c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 shutdown_connections 2024-02-19T14:53:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.692+0000 7f43c6ed4640 1 -- 172.21.15.78:0/287185635 wait complete. 2024-02-19T14:53:39.705 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:53:39.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.807+0000 7f1056fe9640 1 Processor -- start 2024-02-19T14:53:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.807+0000 7f1056fe9640 1 -- start start 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1056fe9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f105014ef50 0x7f105014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1056fe9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10500727d0 0x7f1050072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1056fe9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10500c35c0 con 0x7f10500c78a0 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1056fe9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10500c4bc0 con 0x7f105014ef50 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1056fe9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105014f920 con 0x7f10500727d0 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f105014ef50 0x7f105014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f105014ef50 0x7f105014f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50836/0 (socket says 172.21.15.78:50836) 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 -- 172.21.15.78:0/604767487 learned_addr learned my addr 172.21.15.78:0/604767487 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 -- 172.21.15.78:0/604767487 >> v2:172.21.15.150:3300/0 conn(0x7f10500727d0 msgr2=0x7f1050072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 --2- 172.21.15.78:0/604767487 >> v2:172.21.15.150:3300/0 conn(0x7f10500727d0 0x7f1050072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 -- 172.21.15.78:0/604767487 >> v1:172.21.15.150:6789/0 conn(0x7f10500c78a0 legacy=0x7f10500c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.808+0000 7f1047fff640 1 -- 172.21.15.78:0/604767487 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10500c38a0 con 0x7f105014ef50 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.809+0000 7f1047fff640 1 --2- 172.21.15.78:0/604767487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f105014ef50 0x7f105014f3e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f10500a8ea0 tx=0x7f104002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0da0b0e12e65e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.809+0000 7f10477fe640 1 -- 172.21.15.78:0/604767487 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104002ee90 con 0x7f105014ef50 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.809+0000 7f10477fe640 1 -- 172.21.15.78:0/604767487 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1040036440 con 0x7f105014ef50 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.809+0000 7f10477fe640 1 -- 172.21.15.78:0/604767487 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10400367d0 con 0x7f105014ef50 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 -- 172.21.15.78:0/604767487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f105014ef50 msgr2=0x7f105014f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 --2- 172.21.15.78:0/604767487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f105014ef50 0x7f105014f3e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f10500a8ea0 tx=0x7f104002e0e0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 -- 172.21.15.78:0/604767487 shutdown_connections 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 --2- 172.21.15.78:0/604767487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f105014ef50 0x7f105014f3e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 --2- 172.21.15.78:0/604767487 >> v2:172.21.15.150:3300/0 conn(0x7f10500727d0 0x7f1050072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 -- 172.21.15.78:0/604767487 >> 172.21.15.78:0/604767487 conn(0x7f1050066e70 msgr2=0x7f10500a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 -- 172.21.15.78:0/604767487 shutdown_connections 2024-02-19T14:53:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.810+0000 7f1056fe9640 1 -- 172.21.15.78:0/604767487 wait complete. 2024-02-19T14:53:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.811+0000 7f1056fe9640 1 Processor -- start 2024-02-19T14:53:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.811+0000 7f1056fe9640 1 -- start start 2024-02-19T14:53:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1056fe9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10500727d0 0x7f1050143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1056fe9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 0x7f10500c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1056fe9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f105014ef50 0x7f10501512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1056fe9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1050148660 con 0x7f10500c78a0 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1056fe9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1050148fd0 con 0x7f10500727d0 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1056fe9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10501489c0 con 0x7f105014ef50 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 0x7f10500c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 0x7f10500c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50846/0 (socket says 172.21.15.78:50846) 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 -- 172.21.15.78:0/1421320170 learned_addr learned my addr 172.21.15.78:0/1421320170 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1054d85640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10500727d0 0x7f1050143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1055586640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f105014ef50 0x7f10501512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 -- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f105014ef50 msgr2=0x7f10501512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f105014ef50 0x7f10501512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 -- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10500727d0 msgr2=0x7f1050143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10500727d0 0x7f1050143140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 -- 172.21.15.78:0/1421320170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104000b770 con 0x7f10500c78a0 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1054d85640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10500727d0 0x7f1050143140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f1047fff640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 0x7f10500c5500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f104002e120 tx=0x7f10400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040034950 con 0x7f10500c78a0 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.813+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1040034ab0 con 0x7f10500c78a0 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.814+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1050148380 con 0x7f10500c78a0 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.814+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040034e60 con 0x7f10500c78a0 2024-02-19T14:53:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.814+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1050150a50 con 0x7f10500c78a0 2024-02-19T14:53:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.815+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1050072bf0 con 0x7f10500c78a0 2024-02-19T14:53:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.815+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f104003d040 con 0x7f10500c78a0 2024-02-19T14:53:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.816+0000 7f10457fa640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f103c06b810 0x7f103c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.816+0000 7f1054d85640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f103c06b810 0x7f103c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.816+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f10400aeda0 con 0x7f10500c78a0 2024-02-19T14:53:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.816+0000 7f1054d85640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f103c06b810 0x7f103c06dcd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f10500c6030 tx=0x7f104c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:39.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:39.820+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f10400797e0 con 0x7f10500c78a0 2024-02-19T14:53:40.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.030+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1050074990 con 0x7f10500c78a0 2024-02-19T14:53:40.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.030+0000 7f10457fa640 1 -- 172.21.15.78:0/1421320170 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f104003b090 con 0x7f10500c78a0 2024-02-19T14:53:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:40.033 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5398,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/210065453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":210065453},{"type":"v1","addr":"172.21.15.150:6834","nonce":210065453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:53:39.026813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5703":{"gid":5703,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1917648829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1917648829},{"type":"v1","addr":"172.21.15.78:6837","nonce":1917648829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T14:53:40.033 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 26 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f103c06b810 msgr2=0x7f103c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f103c06b810 0x7f103c06dcd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f10500c6030 tx=0x7f104c009040 comp rx=0 tx=0).stop 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 msgr2=0x7f10500c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 0x7f10500c5500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f104002e120 tx=0x7f10400046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 shutdown_connections 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f103c06b810 0x7f103c06dcd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10500c78a0 0x7f10500c5500 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f105014ef50 0x7f10501512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 --2- 172.21.15.78:0/1421320170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10500727d0 0x7f1050143140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 >> 172.21.15.78:0/1421320170 conn(0x7f1050066e70 msgr2=0x7f105006c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 shutdown_connections 2024-02-19T14:53:40.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.032+0000 7f1056fe9640 1 -- 172.21.15.78:0/1421320170 wait complete. 2024-02-19T14:53:40.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.145+0000 7f8d051bc640 1 Processor -- start 2024-02-19T14:53:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.145+0000 7f8d051bc640 1 -- start start 2024-02-19T14:53:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8d051bc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8d051bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d000ae670 con 0x7f8d000add70 2024-02-19T14:53:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8d051bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d000716b0 con 0x7f8d000705b0 2024-02-19T14:53:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8d051bc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d00153780 con 0x7f8d000c4980 2024-02-19T14:53:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfe59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfe59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50858/0 (socket says 172.21.15.78:50858) 2024-02-19T14:53:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8d000add70 0x7f8d0006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42108/0 (socket says 172.21.15.78:42108) 2024-02-19T14:53:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfe59c640 1 -- 172.21.15.78:0/3212098822 learned_addr learned my addr 172.21.15.78:0/3212098822 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfe59c640 1 -- 172.21.15.78:0/3212098822 >> v1:172.21.15.150:6789/0 conn(0x7f8d000705b0 legacy=0x7f8d000709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfe59c640 1 -- 172.21.15.78:0/3212098822 >> v1:172.21.15.78:6789/0 conn(0x7f8d000add70 legacy=0x7f8d0006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfe59c640 1 -- 172.21.15.78:0/3212098822 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00153a60 con 0x7f8d000c4980 2024-02-19T14:53:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.146+0000 7f8cfdd9b640 1 -- 172.21.15.78:0/3212098822 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973392840 0 0) 0x7f8d000ae670 con 0x7f8d000add70 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.147+0000 7f8cfe59c640 1 --2- 172.21.15.78:0/3212098822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 0x7f8d000c2f60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4009870 tx=0x7f8cf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f36eb4220457795 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.147+0000 7f8cfdd9b640 1 -- 172.21.15.78:0/3212098822 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4031870 con 0x7f8d000c4980 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.147+0000 7f8cfdd9b640 1 -- 172.21.15.78:0/3212098822 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf40319d0 con 0x7f8d000c4980 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.147+0000 7f8cfdd9b640 1 -- 172.21.15.78:0/3212098822 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4031d60 con 0x7f8d000c4980 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 -- 172.21.15.78:0/3212098822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 msgr2=0x7f8d000c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 --2- 172.21.15.78:0/3212098822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 0x7f8d000c2f60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4009870 tx=0x7f8cf402b910 comp rx=0 tx=0).stop 2024-02-19T14:53:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 -- 172.21.15.78:0/3212098822 shutdown_connections 2024-02-19T14:53:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 --2- 172.21.15.78:0/3212098822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 -- 172.21.15.78:0/3212098822 >> 172.21.15.78:0/3212098822 conn(0x7f8d000c85e0 msgr2=0x7f8d000c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 -- 172.21.15.78:0/3212098822 shutdown_connections 2024-02-19T14:53:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.148+0000 7f8d051bc640 1 -- 172.21.15.78:0/3212098822 wait complete. 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.149+0000 7f8d051bc640 1 Processor -- start 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.149+0000 7f8d051bc640 1 -- start start 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8d051bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000705b0 0x7f8d001e8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8d051bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 0x7f8d000c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8d051bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000c4980 0x7f8d00070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8d051bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d001e92e0 con 0x7f8d000add70 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8d051bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d001e6010 con 0x7f8d000705b0 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8d051bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d000bd7f0 con 0x7f8d000c4980 2024-02-19T14:53:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 0x7f8d000c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 0x7f8d000c0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50862/0 (socket says 172.21.15.78:50862) 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 -- 172.21.15.78:0/17047034 learned_addr learned my addr 172.21.15.78:0/17047034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfed9d640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000705b0 0x7f8d001e8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cff59e640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000c4980 0x7f8d00070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 -- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000c4980 msgr2=0x7f8d00070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000c4980 0x7f8d00070c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 -- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000705b0 msgr2=0x7f8d001e8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000705b0 0x7f8d001e8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.150+0000 7f8cfe59c640 1 -- 172.21.15.78:0/17047034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf40094d0 con 0x7f8d000add70 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.151+0000 7f8cfe59c640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 0x7f8d000c0aa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4009870 tx=0x7f8cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.151+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4031820 con 0x7f8d000add70 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.151+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf40443f0 con 0x7f8d000add70 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.151+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d001e8350 con 0x7f8d000add70 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.151+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40447a0 con 0x7f8d000add70 2024-02-19T14:53:40.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.151+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d00156c30 con 0x7f8d000add70 2024-02-19T14:53:40.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.152+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d00074990 con 0x7f8d000add70 2024-02-19T14:53:40.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.153+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8cf404d450 con 0x7f8d000add70 2024-02-19T14:53:40.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.153+0000 7f8ce77fe640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.153+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8cf402d040 con 0x7f8d000add70 2024-02-19T14:53:40.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.153+0000 7f8cfed9d640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.154+0000 7f8cfed9d640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0000f60 tx=0x7f8cf0007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.158+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8cf4077300 con 0x7f8d000add70 2024-02-19T14:53:40.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.332+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d00157180 con 0x7f8d000add70 2024-02-19T14:53:40.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.334+0000 7f8ce77fe640 1 -- 172.21.15.78:0/17047034 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16930 (secure 0 0 0) 0x7f8cf407a9c0 con 0x7f8d000add70 2024-02-19T14:53:40.336 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:40.336 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:38.679120+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:40.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 msgr2=0x7f8cec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0000f60 tx=0x7f8cf0007d50 comp rx=0 tx=0).stop 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 msgr2=0x7f8d000c0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 0x7f8d000c0aa0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4009870 tx=0x7f8cf40046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 shutdown_connections 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000add70 0x7f8d000c0aa0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000c4980 0x7f8d00070c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 --2- 172.21.15.78:0/17047034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000705b0 0x7f8d001e8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 >> 172.21.15.78:0/17047034 conn(0x7f8d000c85e0 msgr2=0x7f8d000c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 shutdown_connections 2024-02-19T14:53:40.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.336+0000 7f8d051bc640 1 -- 172.21.15.78:0/17047034 wait complete. 2024-02-19T14:53:40.349 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:53:40.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.447+0000 7f3eb8b25640 1 Processor -- start 2024-02-19T14:53:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.447+0000 7f3eb8b25640 1 -- start start 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb8b25640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3eb414cd50 0x7f3eb40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb8b25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb40c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb8b25640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3eb40c7b70 0x7f3eb4156860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb8b25640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb41563b0 con 0x7f3eb414cd50 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb8b25640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb4153cd0 con 0x7f3eb40c7b70 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb8b25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb406b6e0 con 0x7f3eb41537a0 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3eb414cd50 0x7f3eb40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3eb414cd50 0x7f3eb40c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50874/0 (socket says 172.21.15.78:50874) 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 -- 172.21.15.78:0/2584932599 learned_addr learned my addr 172.21.15.78:0/2584932599 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb2d9d640 1 --2- 172.21.15.78:0/2584932599 >> v2:172.21.15.150:3300/0 conn(0x7f3eb40c7b70 0x7f3eb4156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb1d9b640 1 --2- 172.21.15.78:0/2584932599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb40c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 -- 172.21.15.78:0/2584932599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 msgr2=0x7f3eb40c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 --2- 172.21.15.78:0/2584932599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb40c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 -- 172.21.15.78:0/2584932599 >> v2:172.21.15.150:3300/0 conn(0x7f3eb40c7b70 msgr2=0x7f3eb4156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 --2- 172.21.15.78:0/2584932599 >> v2:172.21.15.150:3300/0 conn(0x7f3eb40c7b70 0x7f3eb4156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb259c640 1 -- 172.21.15.78:0/2584932599 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb406b980 con 0x7f3eb414cd50 2024-02-19T14:53:40.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.449+0000 7f3eb2d9d640 1 --2- 172.21.15.78:0/2584932599 >> v2:172.21.15.150:3300/0 conn(0x7f3eb40c7b70 0x7f3eb4156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.450+0000 7f3eb259c640 1 --2- 172.21.15.78:0/2584932599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb40c4cd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009870 tx=0x7f3eac02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421e7a6bb575affe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.450+0000 7f3eb159a640 1 -- 172.21.15.78:0/2584932599 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac031740 con 0x7f3eb414cd50 2024-02-19T14:53:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.450+0000 7f3eb159a640 1 -- 172.21.15.78:0/2584932599 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac0318a0 con 0x7f3eb414cd50 2024-02-19T14:53:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.450+0000 7f3eb159a640 1 -- 172.21.15.78:0/2584932599 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac031c30 con 0x7f3eb414cd50 2024-02-19T14:53:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 -- 172.21.15.78:0/2584932599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 msgr2=0x7f3eb40c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/2584932599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb40c4cd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009870 tx=0x7f3eac02b7e0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 -- 172.21.15.78:0/2584932599 shutdown_connections 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/2584932599 >> v2:172.21.15.150:3300/0 conn(0x7f3eb40c7b70 0x7f3eb4156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/2584932599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb40c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/2584932599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb40c4cd0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 -- 172.21.15.78:0/2584932599 >> 172.21.15.78:0/2584932599 conn(0x7f3eb4066e70 msgr2=0x7f3eb40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 -- 172.21.15.78:0/2584932599 shutdown_connections 2024-02-19T14:53:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.451+0000 7f3eb8b25640 1 -- 172.21.15.78:0/2584932599 wait complete. 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.452+0000 7f3eb8b25640 1 Processor -- start 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.452+0000 7f3eb8b25640 1 -- start start 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb8b25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb40c7b70 0x7f3eb414dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb8b25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb41e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb8b25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb41ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb8b25640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40c6d70 con 0x7f3eb414cd50 2024-02-19T14:53:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb8b25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40c61f0 con 0x7f3eb40c7b70 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb8b25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40c64d0 con 0x7f3eb41537a0 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb41e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb41e9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50884/0 (socket says 172.21.15.78:50884) 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 -- 172.21.15.78:0/1688368162 learned_addr learned my addr 172.21.15.78:0/1688368162 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb259c640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb40c7b70 0x7f3eb414dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb2d9d640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb41ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 -- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 msgr2=0x7f3eb41ea4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb41ea4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 -- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb40c7b70 msgr2=0x7f3eb414dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb40c7b70 0x7f3eb414dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.453+0000 7f3eb1d9b640 1 -- 172.21.15.78:0/1688368162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac0094d0 con 0x7f3eb414cd50 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.454+0000 7f3eb1d9b640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb41e9fa0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8007e30 tx=0x7f3ea800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.454+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8014620 con 0x7f3eb414cd50 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.454+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800ee90 con 0x7f3eb414cd50 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.454+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80149c0 con 0x7f3eb414cd50 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.454+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb40c67b0 con 0x7f3eb414cd50 2024-02-19T14:53:40.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.454+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb414e1a0 con 0x7f3eb414cd50 2024-02-19T14:53:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.455+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb4074990 con 0x7f3eb414cd50 2024-02-19T14:53:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.455+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3ea8014b20 con 0x7f3eb414cd50 2024-02-19T14:53:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.456+0000 7f3ea2ffd640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e9406b840 0x7f3e9406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.456+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ea808ab10 con 0x7f3eb414cd50 2024-02-19T14:53:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.456+0000 7f3eb259c640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e9406b840 0x7f3e9406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.456+0000 7f3eb259c640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e9406b840 0x7f3e9406dd00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3eac035000 tx=0x7f3eac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.460+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ea8055480 con 0x7f3eb414cd50 2024-02-19T14:53:40.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.629+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3eb406bc60 con 0x7f3e9406b840 2024-02-19T14:53:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.637+0000 7f3ea2ffd640 1 -- 172.21.15.78:0/1688368162 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289591 (secure 0 0 0) 0x7f3eac041070 con 0x7f3e9406b840 2024-02-19T14:53:40.642 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e9406b840 msgr2=0x7f3e9406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e9406b840 0x7f3e9406dd00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3eac035000 tx=0x7f3eac008030 comp rx=0 tx=0).stop 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 msgr2=0x7f3eb41e9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb41e9fa0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8007e30 tx=0x7f3ea800e4b0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 shutdown_connections 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e9406b840 0x7f3e9406dd00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3eb414cd50 0x7f3eb41e9fa0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3eb41537a0 0x7f3eb41ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 --2- 172.21.15.78:0/1688368162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb40c7b70 0x7f3eb414dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 >> 172.21.15.78:0/1688368162 conn(0x7f3eb4066e70 msgr2=0x7f3eb4154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 shutdown_connections 2024-02-19T14:53:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.640+0000 7f3eb8b25640 1 -- 172.21.15.78:0/1688368162 wait complete. 2024-02-19T14:53:40.653 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-02-19T14:53:40.080235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221944,"kb_used_data":5768,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752776,"statfs":{"total":773094113280,"available":772866842624,"internally_reserved":0,"allocated":5906432,"data_stored":3261045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.415796"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698194+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:37.698194+0000","last_peered":"2024-02-19T14:53:37.698194+0000","last_clean":"2024-02-19T14:53:37.698194+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:37.698194+0000","last_undegraded":"2024-02-19T14:53:37.698194+0000","last_fullsized":"2024-02-19T14:53:37.698194+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698436+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:37.698436+0000","last_peered":"2024-02-19T14:53:37.698436+0000","last_clean":"2024-02-19T14:53:37.698436+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:37.698436+0000","last_undegraded":"2024-02-19T14:53:37.698436+0000","last_fullsized":"2024-02-19T14:53:37.698436+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708593+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:37.708593+0000","last_peered":"2024-02-19T14:53:37.708593+0000","last_clean":"2024-02-19T14:53:37.708593+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:37.708593+0000","last_undegraded":"2024-02-19T14:53:37.708593+0000","last_fullsized":"2024-02-19T14:53:37.708593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700534+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:37.700534+0000","last_peered":"2024-02-19T14:53:37.700534+0000","last_clean":"2024-02-19T14:53:37.700534+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:37.700534+0000","last_undegraded":"2024-02-19T14:53:37.700534+0000","last_fullsized":"2024-02-19T14:53:37.700534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689550+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:37.689550+0000","last_peered":"2024-02-19T14:53:37.689550+0000","last_clean":"2024-02-19T14:53:37.689550+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:37.689550+0000","last_undegraded":"2024-02-19T14:53:37.689550+0000","last_fullsized":"2024-02-19T14:53:37.689550+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:19.938736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689390+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:37.689390+0000","last_peered":"2024-02-19T14:53:37.689390+0000","last_clean":"2024-02-19T14:53:37.689390+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:37.689390+0000","last_undegraded":"2024-02-19T14:53:37.689390+0000","last_fullsized":"2024-02-19T14:53:37.689390+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:18.042708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707125+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:37.707125+0000","last_peered":"2024-02-19T14:53:37.707125+0000","last_clean":"2024-02-19T14:53:37.707125+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:37.707125+0000","last_undegraded":"2024-02-19T14:53:37.707125+0000","last_fullsized":"2024-02-19T14:53:37.707125+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706675+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:37.706675+0000","last_peered":"2024-02-19T14:53:37.706675+0000","last_clean":"2024-02-19T14:53:37.706675+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:37.706675+0000","last_undegraded":"2024-02-19T14:53:37.706675+0000","last_fullsized":"2024-02-19T14:53:37.706675+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.711305+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:37.711305+0000","last_peered":"2024-02-19T14:53:37.711305+0000","last_clean":"2024-02-19T14:53:37.711305+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:37.711305+0000","last_undegraded":"2024-02-19T14:53:37.711305+0000","last_fullsized":"2024-02-19T14:53:37.711305+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689363+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:37.689363+0000","last_peered":"2024-02-19T14:53:37.689363+0000","last_clean":"2024-02-19T14:53:37.689363+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:37.689363+0000","last_undegraded":"2024-02-19T14:53:37.689363+0000","last_fullsized":"2024-02-19T14:53:37.689363+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:25.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.693861+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:37.693861+0000","last_peered":"2024-02-19T14:53:37.693861+0000","last_clean":"2024-02-19T14:53:37.693861+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:37.693861+0000","last_undegraded":"2024-02-19T14:53:37.693861+0000","last_fullsized":"2024-02-19T14:53:37.693861+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:12.858220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692643+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:37.692643+0000","last_peered":"2024-02-19T14:53:37.692643+0000","last_clean":"2024-02-19T14:53:37.692643+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:37.692643+0000","last_undegraded":"2024-02-19T14:53:37.692643+0000","last_fullsized":"2024-02-19T14:53:37.692643+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689360+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:37.689360+0000","last_peered":"2024-02-19T14:53:37.689360+0000","last_clean":"2024-02-19T14:53:37.689360+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:37.689360+0000","last_undegraded":"2024-02-19T14:53:37.689360+0000","last_fullsized":"2024-02-19T14:53:37.689360+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708687+0000","last_change":"2024-02-19T14:53:37.703840+0000","last_active":"2024-02-19T14:53:37.708687+0000","last_peered":"2024-02-19T14:53:37.708687+0000","last_clean":"2024-02-19T14:53:37.708687+0000","last_became_active":"2024-02-19T14:53:37.703424+0000","last_became_peered":"2024-02-19T14:53:37.703424+0000","last_unstale":"2024-02-19T14:53:37.708687+0000","last_undegraded":"2024-02-19T14:53:37.708687+0000","last_fullsized":"2024-02-19T14:53:37.708687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698240+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:37.698240+0000","last_peered":"2024-02-19T14:53:37.698240+0000","last_clean":"2024-02-19T14:53:37.698240+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:37.698240+0000","last_undegraded":"2024-02-19T14:53:37.698240+0000","last_fullsized":"2024-02-19T14:53:37.698240+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689435+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:37.689435+0000","last_peered":"2024-02-19T14:53:37.689435+0000","last_clean":"2024-02-19T14:53:37.689435+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:37.689435+0000","last_undegraded":"2024-02-19T14:53:37.689435+0000","last_fullsized":"2024-02-19T14:53:37.689435+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708676+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:37.708676+0000","last_peered":"2024-02-19T14:53:37.708676+0000","last_clean":"2024-02-19T14:53:37.708676+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:37.708676+0000","last_undegraded":"2024-02-19T14:53:37.708676+0000","last_fullsized":"2024-02-19T14:53:37.708676+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698415+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:37.698415+0000","last_peered":"2024-02-19T14:53:37.698415+0000","last_clean":"2024-02-19T14:53:37.698415+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:37.698415+0000","last_undegraded":"2024-02-19T14:53:37.698415+0000","last_fullsized":"2024-02-19T14:53:37.698415+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707151+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:37.707151+0000","last_peered":"2024-02-19T14:53:37.707151+0000","last_clean":"2024-02-19T14:53:37.707151+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:37.707151+0000","last_undegraded":"2024-02-19T14:53:37.707151+0000","last_fullsized":"2024-02-19T14:53:37.707151+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.691201+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:37.691201+0000","last_peered":"2024-02-19T14:53:37.691201+0000","last_clean":"2024-02-19T14:53:37.691201+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:37.691201+0000","last_undegraded":"2024-02-19T14:53:37.691201+0000","last_fullsized":"2024-02-19T14:53:37.691201+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700407+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:37.700407+0000","last_peered":"2024-02-19T14:53:37.700407+0000","last_clean":"2024-02-19T14:53:37.700407+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:37.700407+0000","last_undegraded":"2024-02-19T14:53:37.700407+0000","last_fullsized":"2024-02-19T14:53:37.700407+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.711288+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:37.711288+0000","last_peered":"2024-02-19T14:53:37.711288+0000","last_clean":"2024-02-19T14:53:37.711288+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:37.711288+0000","last_undegraded":"2024-02-19T14:53:37.711288+0000","last_fullsized":"2024-02-19T14:53:37.711288+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.694199+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:37.694199+0000","last_peered":"2024-02-19T14:53:37.694199+0000","last_clean":"2024-02-19T14:53:37.694199+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:37.694199+0000","last_undegraded":"2024-02-19T14:53:37.694199+0000","last_fullsized":"2024-02-19T14:53:37.694199+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:44.347720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707041+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:37.707041+0000","last_peered":"2024-02-19T14:53:37.707041+0000","last_clean":"2024-02-19T14:53:37.707041+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:37.707041+0000","last_undegraded":"2024-02-19T14:53:37.707041+0000","last_fullsized":"2024-02-19T14:53:37.707041+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707099+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:37.707099+0000","last_peered":"2024-02-19T14:53:37.707099+0000","last_clean":"2024-02-19T14:53:37.707099+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:37.707099+0000","last_undegraded":"2024-02-19T14:53:37.707099+0000","last_fullsized":"2024-02-19T14:53:37.707099+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.701967+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:37.701967+0000","last_peered":"2024-02-19T14:53:37.701967+0000","last_clean":"2024-02-19T14:53:37.701967+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:37.701967+0000","last_undegraded":"2024-02-19T14:53:37.701967+0000","last_fullsized":"2024-02-19T14:53:37.701967+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.705823+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:37.705823+0000","last_peered":"2024-02-19T14:53:37.705823+0000","last_clean":"2024-02-19T14:53:37.705823+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:37.705823+0000","last_undegraded":"2024-02-19T14:53:37.705823+0000","last_fullsized":"2024-02-19T14:53:37.705823+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:56.120959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700583+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:37.700583+0000","last_peered":"2024-02-19T14:53:37.700583+0000","last_clean":"2024-02-19T14:53:37.700583+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:37.700583+0000","last_undegraded":"2024-02-19T14:53:37.700583+0000","last_fullsized":"2024-02-19T14:53:37.700583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:07.107866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.711324+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:37.711324+0000","last_peered":"2024-02-19T14:53:37.711324+0000","last_clean":"2024-02-19T14:53:37.711324+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:37.711324+0000","last_undegraded":"2024-02-19T14:53:37.711324+0000","last_fullsized":"2024-02-19T14:53:37.711324+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698296+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:37.698296+0000","last_peered":"2024-02-19T14:53:37.698296+0000","last_clean":"2024-02-19T14:53:37.698296+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:37.698296+0000","last_undegraded":"2024-02-19T14:53:37.698296+0000","last_fullsized":"2024-02-19T14:53:37.698296+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703181+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:37.703181+0000","last_peered":"2024-02-19T14:53:37.703181+0000","last_clean":"2024-02-19T14:53:37.703181+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:37.703181+0000","last_undegraded":"2024-02-19T14:53:37.703181+0000","last_fullsized":"2024-02-19T14:53:37.703181+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:56.120959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700374+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:37.700374+0000","last_peered":"2024-02-19T14:53:37.700374+0000","last_clean":"2024-02-19T14:53:37.700374+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:37.700374+0000","last_undegraded":"2024-02-19T14:53:37.700374+0000","last_fullsized":"2024-02-19T14:53:37.700374+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698449+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:37.698449+0000","last_peered":"2024-02-19T14:53:37.698449+0000","last_clean":"2024-02-19T14:53:37.698449+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:37.698449+0000","last_undegraded":"2024-02-19T14:53:37.698449+0000","last_fullsized":"2024-02-19T14:53:37.698449+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.701148+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:37.701148+0000","last_peered":"2024-02-19T14:53:37.701148+0000","last_clean":"2024-02-19T14:53:37.701148+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:37.701148+0000","last_undegraded":"2024-02-19T14:53:37.701148+0000","last_fullsized":"2024-02-19T14:53:37.701148+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.690472+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:37.690472+0000","last_peered":"2024-02-19T14:53:37.690472+0000","last_clean":"2024-02-19T14:53:37.690472+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:37.690472+0000","last_undegraded":"2024-02-19T14:53:37.690472+0000","last_fullsized":"2024-02-19T14:53:37.690472+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692809+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:37.692809+0000","last_peered":"2024-02-19T14:53:37.692809+0000","last_clean":"2024-02-19T14:53:37.692809+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:37.692809+0000","last_undegraded":"2024-02-19T14:53:37.692809+0000","last_fullsized":"2024-02-19T14:53:37.692809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700490+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:37.700490+0000","last_peered":"2024-02-19T14:53:37.700490+0000","last_clean":"2024-02-19T14:53:37.700490+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:37.700490+0000","last_undegraded":"2024-02-19T14:53:37.700490+0000","last_fullsized":"2024-02-19T14:53:37.700490+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.690711+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:37.690711+0000","last_peered":"2024-02-19T14:53:37.690711+0000","last_clean":"2024-02-19T14:53:37.690711+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:37.690711+0000","last_undegraded":"2024-02-19T14:53:37.690711+0000","last_fullsized":"2024-02-19T14:53:37.690711+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702116+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:37.702116+0000","last_peered":"2024-02-19T14:53:37.702116+0000","last_clean":"2024-02-19T14:53:37.702116+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:37.702116+0000","last_undegraded":"2024-02-19T14:53:37.702116+0000","last_fullsized":"2024-02-19T14:53:37.702116+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702310+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:37.702310+0000","last_peered":"2024-02-19T14:53:37.702310+0000","last_clean":"2024-02-19T14:53:37.702310+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:37.702310+0000","last_undegraded":"2024-02-19T14:53:37.702310+0000","last_fullsized":"2024-02-19T14:53:37.702310+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706165+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:37.706165+0000","last_peered":"2024-02-19T14:53:37.706165+0000","last_clean":"2024-02-19T14:53:37.706165+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:37.706165+0000","last_undegraded":"2024-02-19T14:53:37.706165+0000","last_fullsized":"2024-02-19T14:53:37.706165+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:25.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702380+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:37.702380+0000","last_peered":"2024-02-19T14:53:37.702380+0000","last_clean":"2024-02-19T14:53:37.702380+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:37.702380+0000","last_undegraded":"2024-02-19T14:53:37.702380+0000","last_fullsized":"2024-02-19T14:53:37.702380+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698380+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:37.698380+0000","last_peered":"2024-02-19T14:53:37.698380+0000","last_clean":"2024-02-19T14:53:37.698380+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:37.698380+0000","last_undegraded":"2024-02-19T14:53:37.698380+0000","last_fullsized":"2024-02-19T14:53:37.698380+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697911+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:37.697911+0000","last_peered":"2024-02-19T14:53:37.697911+0000","last_clean":"2024-02-19T14:53:37.697911+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:37.697911+0000","last_undegraded":"2024-02-19T14:53:37.697911+0000","last_fullsized":"2024-02-19T14:53:37.697911+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692542+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:37.692542+0000","last_peered":"2024-02-19T14:53:37.692542+0000","last_clean":"2024-02-19T14:53:37.692542+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:37.692542+0000","last_undegraded":"2024-02-19T14:53:37.692542+0000","last_fullsized":"2024-02-19T14:53:37.692542+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.681553+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:37.681553+0000","last_peered":"2024-02-19T14:53:37.681553+0000","last_clean":"2024-02-19T14:53:37.681553+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:37.681553+0000","last_undegraded":"2024-02-19T14:53:37.681553+0000","last_fullsized":"2024-02-19T14:53:37.681553+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706985+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:37.706985+0000","last_peered":"2024-02-19T14:53:37.706985+0000","last_clean":"2024-02-19T14:53:37.706985+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:37.706985+0000","last_undegraded":"2024-02-19T14:53:37.706985+0000","last_fullsized":"2024-02-19T14:53:37.706985+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.687872+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:37.687872+0000","last_peered":"2024-02-19T14:53:37.687872+0000","last_clean":"2024-02-19T14:53:37.687872+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:37.687872+0000","last_undegraded":"2024-02-19T14:53:37.687872+0000","last_fullsized":"2024-02-19T14:53:37.687872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697553+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:37.697553+0000","last_peered":"2024-02-19T14:53:37.697553+0000","last_clean":"2024-02-19T14:53:37.697553+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:37.697553+0000","last_undegraded":"2024-02-19T14:53:37.697553+0000","last_fullsized":"2024-02-19T14:53:37.697553+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:13.235683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.681686+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:37.681686+0000","last_peered":"2024-02-19T14:53:37.681686+0000","last_clean":"2024-02-19T14:53:37.681686+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:37.681686+0000","last_undegraded":"2024-02-19T14:53:37.681686+0000","last_fullsized":"2024-02-19T14:53:37.681686+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697675+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:37.697675+0000","last_peered":"2024-02-19T14:53:37.697675+0000","last_clean":"2024-02-19T14:53:37.697675+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:37.697675+0000","last_undegraded":"2024-02-19T14:53:37.697675+0000","last_fullsized":"2024-02-19T14:53:37.697675+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.673417+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:37.673417+0000","last_peered":"2024-02-19T14:53:37.673417+0000","last_clean":"2024-02-19T14:53:37.673417+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:37.673417+0000","last_undegraded":"2024-02-19T14:53:37.673417+0000","last_fullsized":"2024-02-19T14:53:37.673417+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708576+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:37.708576+0000","last_peered":"2024-02-19T14:53:37.708576+0000","last_clean":"2024-02-19T14:53:37.708576+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:37.708576+0000","last_undegraded":"2024-02-19T14:53:37.708576+0000","last_fullsized":"2024-02-19T14:53:37.708576+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.699136+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:37.699136+0000","last_peered":"2024-02-19T14:53:37.699136+0000","last_clean":"2024-02-19T14:53:37.699136+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:37.699136+0000","last_undegraded":"2024-02-19T14:53:37.699136+0000","last_fullsized":"2024-02-19T14:53:37.699136+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700373+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:37.700373+0000","last_peered":"2024-02-19T14:53:37.700373+0000","last_clean":"2024-02-19T14:53:37.700373+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:37.700373+0000","last_undegraded":"2024-02-19T14:53:37.700373+0000","last_fullsized":"2024-02-19T14:53:37.700373+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:47.458326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708901+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:37.708901+0000","last_peered":"2024-02-19T14:53:37.708901+0000","last_clean":"2024-02-19T14:53:37.708901+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:37.708901+0000","last_undegraded":"2024-02-19T14:53:37.708901+0000","last_fullsized":"2024-02-19T14:53:37.708901+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697707+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:37.697707+0000","last_peered":"2024-02-19T14:53:37.697707+0000","last_clean":"2024-02-19T14:53:37.697707+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:37.697707+0000","last_undegraded":"2024-02-19T14:53:37.697707+0000","last_fullsized":"2024-02-19T14:53:37.697707+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702767+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:37.702767+0000","last_peered":"2024-02-19T14:53:37.702767+0000","last_clean":"2024-02-19T14:53:37.702767+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:37.702767+0000","last_undegraded":"2024-02-19T14:53:37.702767+0000","last_fullsized":"2024-02-19T14:53:37.702767+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703703+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:37.703703+0000","last_peered":"2024-02-19T14:53:37.703703+0000","last_clean":"2024-02-19T14:53:37.703703+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:37.703703+0000","last_undegraded":"2024-02-19T14:53:37.703703+0000","last_fullsized":"2024-02-19T14:53:37.703703+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:07.107866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.681151+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:37.681151+0000","last_peered":"2024-02-19T14:53:37.681151+0000","last_clean":"2024-02-19T14:53:37.681151+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:37.681151+0000","last_undegraded":"2024-02-19T14:53:37.681151+0000","last_fullsized":"2024-02-19T14:53:37.681151+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692563+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:37.692563+0000","last_peered":"2024-02-19T14:53:37.692563+0000","last_clean":"2024-02-19T14:53:37.692563+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:37.692563+0000","last_undegraded":"2024-02-19T14:53:37.692563+0000","last_fullsized":"2024-02-19T14:53:37.692563+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.680817+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:37.680817+0000","last_peered":"2024-02-19T14:53:37.680817+0000","last_clean":"2024-02-19T14:53:37.680817+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:37.680817+0000","last_undegraded":"2024-02-19T14:53:37.680817+0000","last_fullsized":"2024-02-19T14:53:37.680817+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708655+0000","last_change":"2024-02-19T14:53:37.703805+0000","last_active":"2024-02-19T14:53:37.708655+0000","last_peered":"2024-02-19T14:53:37.708655+0000","last_clean":"2024-02-19T14:53:37.708655+0000","last_became_active":"2024-02-19T14:53:37.703436+0000","last_became_peered":"2024-02-19T14:53:37.703436+0000","last_unstale":"2024-02-19T14:53:37.708655+0000","last_undegraded":"2024-02-19T14:53:37.708655+0000","last_fullsized":"2024-02-19T14:53:37.708655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700043+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:37.700043+0000","last_peered":"2024-02-19T14:53:37.700043+0000","last_clean":"2024-02-19T14:53:37.700043+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:37.700043+0000","last_undegraded":"2024-02-19T14:53:37.700043+0000","last_fullsized":"2024-02-19T14:53:37.700043+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.695143+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:37.695143+0000","last_peered":"2024-02-19T14:53:37.695143+0000","last_clean":"2024-02-19T14:53:37.695143+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:37.695143+0000","last_undegraded":"2024-02-19T14:53:37.695143+0000","last_fullsized":"2024-02-19T14:53:37.695143+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703275+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:37.703275+0000","last_peered":"2024-02-19T14:53:37.703275+0000","last_clean":"2024-02-19T14:53:37.703275+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:37.703275+0000","last_undegraded":"2024-02-19T14:53:37.703275+0000","last_fullsized":"2024-02-19T14:53:37.703275+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702141+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:37.702141+0000","last_peered":"2024-02-19T14:53:37.702141+0000","last_clean":"2024-02-19T14:53:37.702141+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:37.702141+0000","last_undegraded":"2024-02-19T14:53:37.702141+0000","last_fullsized":"2024-02-19T14:53:37.702141+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:07.107866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703010+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:37.703010+0000","last_peered":"2024-02-19T14:53:37.703010+0000","last_clean":"2024-02-19T14:53:37.703010+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:37.703010+0000","last_undegraded":"2024-02-19T14:53:37.703010+0000","last_fullsized":"2024-02-19T14:53:37.703010+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706185+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:37.706185+0000","last_peered":"2024-02-19T14:53:37.706185+0000","last_clean":"2024-02-19T14:53:37.706185+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:37.706185+0000","last_undegraded":"2024-02-19T14:53:37.706185+0000","last_fullsized":"2024-02-19T14:53:37.706185+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:18.042708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700653+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:37.700653+0000","last_peered":"2024-02-19T14:53:37.700653+0000","last_clean":"2024-02-19T14:53:37.700653+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:37.700653+0000","last_undegraded":"2024-02-19T14:53:37.700653+0000","last_fullsized":"2024-02-19T14:53:37.700653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689980+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:37.689980+0000","last_peered":"2024-02-19T14:53:37.689980+0000","last_clean":"2024-02-19T14:53:37.689980+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:37.689980+0000","last_undegraded":"2024-02-19T14:53:37.689980+0000","last_fullsized":"2024-02-19T14:53:37.689980+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.680848+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:37.680848+0000","last_peered":"2024-02-19T14:53:37.680848+0000","last_clean":"2024-02-19T14:53:37.680848+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:37.680848+0000","last_undegraded":"2024-02-19T14:53:37.680848+0000","last_fullsized":"2024-02-19T14:53:37.680848+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697730+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:37.697730+0000","last_peered":"2024-02-19T14:53:37.697730+0000","last_clean":"2024-02-19T14:53:37.697730+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:37.697730+0000","last_undegraded":"2024-02-19T14:53:37.697730+0000","last_fullsized":"2024-02-19T14:53:37.697730+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.694511+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:37.694511+0000","last_peered":"2024-02-19T14:53:37.694511+0000","last_clean":"2024-02-19T14:53:37.694511+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:37.694511+0000","last_undegraded":"2024-02-19T14:53:37.694511+0000","last_fullsized":"2024-02-19T14:53:37.694511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697565+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:37.697565+0000","last_peered":"2024-02-19T14:53:37.697565+0000","last_clean":"2024-02-19T14:53:37.697565+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:37.697565+0000","last_undegraded":"2024-02-19T14:53:37.697565+0000","last_fullsized":"2024-02-19T14:53:37.697565+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.672664+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:37.672664+0000","last_peered":"2024-02-19T14:53:37.672664+0000","last_clean":"2024-02-19T14:53:37.672664+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:37.672664+0000","last_undegraded":"2024-02-19T14:53:37.672664+0000","last_fullsized":"2024-02-19T14:53:37.672664+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703209+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:37.703209+0000","last_peered":"2024-02-19T14:53:37.703209+0000","last_clean":"2024-02-19T14:53:37.703209+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:37.703209+0000","last_undegraded":"2024-02-19T14:53:37.703209+0000","last_fullsized":"2024-02-19T14:53:37.703209+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689485+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:37.689485+0000","last_peered":"2024-02-19T14:53:37.689485+0000","last_clean":"2024-02-19T14:53:37.689485+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:37.689485+0000","last_undegraded":"2024-02-19T14:53:37.689485+0000","last_fullsized":"2024-02-19T14:53:37.689485+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689477+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:37.689477+0000","last_peered":"2024-02-19T14:53:37.689477+0000","last_clean":"2024-02-19T14:53:37.689477+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:37.689477+0000","last_undegraded":"2024-02-19T14:53:37.689477+0000","last_fullsized":"2024-02-19T14:53:37.689477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.679673+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:37.679673+0000","last_peered":"2024-02-19T14:53:37.679673+0000","last_clean":"2024-02-19T14:53:37.679673+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:37.679673+0000","last_undegraded":"2024-02-19T14:53:37.679673+0000","last_fullsized":"2024-02-19T14:53:37.679673+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700352+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:37.700352+0000","last_peered":"2024-02-19T14:53:37.700352+0000","last_clean":"2024-02-19T14:53:37.700352+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:37.700352+0000","last_undegraded":"2024-02-19T14:53:37.700352+0000","last_fullsized":"2024-02-19T14:53:37.700352+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.704078+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:37.704078+0000","last_peered":"2024-02-19T14:53:37.704078+0000","last_clean":"2024-02-19T14:53:37.704078+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:37.704078+0000","last_undegraded":"2024-02-19T14:53:37.704078+0000","last_fullsized":"2024-02-19T14:53:37.704078+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698254+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:37.698254+0000","last_peered":"2024-02-19T14:53:37.698254+0000","last_clean":"2024-02-19T14:53:37.698254+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:37.698254+0000","last_undegraded":"2024-02-19T14:53:37.698254+0000","last_fullsized":"2024-02-19T14:53:37.698254+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:57.477164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.683514+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:37.683514+0000","last_peered":"2024-02-19T14:53:37.683514+0000","last_clean":"2024-02-19T14:53:37.683514+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:37.683514+0000","last_undegraded":"2024-02-19T14:53:37.683514+0000","last_fullsized":"2024-02-19T14:53:37.683514+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692667+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:37.692667+0000","last_peered":"2024-02-19T14:53:37.692667+0000","last_clean":"2024-02-19T14:53:37.692667+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:37.692667+0000","last_undegraded":"2024-02-19T14:53:37.692667+0000","last_fullsized":"2024-02-19T14:53:37.692667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.688134+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:37.688134+0000","last_peered":"2024-02-19T14:53:37.688134+0000","last_clean":"2024-02-19T14:53:37.688134+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:37.688134+0000","last_undegraded":"2024-02-19T14:53:37.688134+0000","last_fullsized":"2024-02-19T14:53:37.688134+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707086+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:37.707086+0000","last_peered":"2024-02-19T14:53:37.707086+0000","last_clean":"2024-02-19T14:53:37.707086+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:37.707086+0000","last_undegraded":"2024-02-19T14:53:37.707086+0000","last_fullsized":"2024-02-19T14:53:37.707086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.674888+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:37.674888+0000","last_peered":"2024-02-19T14:53:37.674888+0000","last_clean":"2024-02-19T14:53:37.674888+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:37.674888+0000","last_undegraded":"2024-02-19T14:53:37.674888+0000","last_fullsized":"2024-02-19T14:53:37.674888+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697573+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:37.697573+0000","last_peered":"2024-02-19T14:53:37.697573+0000","last_clean":"2024-02-19T14:53:37.697573+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:37.697573+0000","last_undegraded":"2024-02-19T14:53:37.697573+0000","last_fullsized":"2024-02-19T14:53:37.697573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.672789+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:37.672789+0000","last_peered":"2024-02-19T14:53:37.672789+0000","last_clean":"2024-02-19T14:53:37.672789+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:37.672789+0000","last_undegraded":"2024-02-19T14:53:37.672789+0000","last_fullsized":"2024-02-19T14:53:37.672789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707024+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:37.707024+0000","last_peered":"2024-02-19T14:53:37.707024+0000","last_clean":"2024-02-19T14:53:37.707024+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:37.707024+0000","last_undegraded":"2024-02-19T14:53:37.707024+0000","last_fullsized":"2024-02-19T14:53:37.707024+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.673418+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:37.673418+0000","last_peered":"2024-02-19T14:53:37.673418+0000","last_clean":"2024-02-19T14:53:37.673418+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:37.673418+0000","last_undegraded":"2024-02-19T14:53:37.673418+0000","last_fullsized":"2024-02-19T14:53:37.673418+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692514+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:37.692514+0000","last_peered":"2024-02-19T14:53:37.692514+0000","last_clean":"2024-02-19T14:53:37.692514+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:37.692514+0000","last_undegraded":"2024-02-19T14:53:37.692514+0000","last_fullsized":"2024-02-19T14:53:37.692514+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.677290+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:37.677290+0000","last_peered":"2024-02-19T14:53:37.677290+0000","last_clean":"2024-02-19T14:53:37.677290+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:37.677290+0000","last_undegraded":"2024-02-19T14:53:37.677290+0000","last_fullsized":"2024-02-19T14:53:37.677290+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700547+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:37.700547+0000","last_peered":"2024-02-19T14:53:37.700547+0000","last_clean":"2024-02-19T14:53:37.700547+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:37.700547+0000","last_undegraded":"2024-02-19T14:53:37.700547+0000","last_fullsized":"2024-02-19T14:53:37.700547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:57.477164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.687364+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:37.687364+0000","last_peered":"2024-02-19T14:53:37.687364+0000","last_clean":"2024-02-19T14:53:37.687364+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:37.687364+0000","last_undegraded":"2024-02-19T14:53:37.687364+0000","last_fullsized":"2024-02-19T14:53:37.687364+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":226,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.680604+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:37.680604+0000","last_peered":"2024-02-19T14:53:37.680604+0000","last_clean":"2024-02-19T14:53:37.680604+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:37.680604+0000","last_undegraded":"2024-02-19T14:53:37.680604+0000","last_fullsized":"2024-02-19T14:53:37.680604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1196032,"data_stored":854773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.505},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.432}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.624},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57299999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.49099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.435}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.36799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61699999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.59299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.66400000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.78000000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.86899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.626},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.61699999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.57399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":265893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.254}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.215},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.94199999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":264441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":847482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.27100000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.314}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.501},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:40.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:40.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.794+0000 7fad9c2ed640 1 Processor -- start 2024-02-19T14:53:40.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.794+0000 7fad9c2ed640 1 -- start start 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9c2ed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 0x7fad940c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9c2ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad9406bb00 0x7fad940c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9c2ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94156e80 con 0x7fad940721a0 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9c2ed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9406ffc0 con 0x7fad94150560 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9c2ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940c6cd0 con 0x7fad9406bb00 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9a089640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 0x7fad940c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad99888640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad9406bb00 0x7fad940c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9a88a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fad940721a0 0x7fad940c65c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51988/0 (socket says 172.21.15.78:51988) 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9a089640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 0x7fad940c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:36550/0 (socket says 172.21.15.78:36550) 2024-02-19T14:53:40.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9a88a640 1 -- 172.21.15.78:0/4280641322 learned_addr learned my addr 172.21.15.78:0/4280641322 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad99888640 1 -- 172.21.15.78:0/4280641322 >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 msgr2=0x7fad940c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad99888640 1 --2- 172.21.15.78:0/4280641322 >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 0x7fad940c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad99888640 1 -- 172.21.15.78:0/4280641322 >> v1:172.21.15.150:6789/0 conn(0x7fad940721a0 legacy=0x7fad940c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad99888640 1 -- 172.21.15.78:0/4280641322 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94072d70 con 0x7fad9406bb00 2024-02-19T14:53:40.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.795+0000 7fad9a089640 1 --2- 172.21.15.78:0/4280641322 >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 0x7fad940c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.796+0000 7fad99888640 1 --2- 172.21.15.78:0/4280641322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad9406bb00 0x7fad940c6080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fad840097a0 tx=0x7fad8402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e07ce9e8d3863 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.796+0000 7fad99087640 1 -- 172.21.15.78:0/4280641322 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8402f050 con 0x7fad9406bb00 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.796+0000 7fad99087640 1 -- 172.21.15.78:0/4280641322 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad84031700 con 0x7fad9406bb00 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.796+0000 7fad99087640 1 -- 172.21.15.78:0/4280641322 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84031a90 con 0x7fad9406bb00 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4280641322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad9406bb00 msgr2=0x7fad940c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4280641322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad9406bb00 0x7fad940c6080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fad840097a0 tx=0x7fad8402b7a0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4280641322 shutdown_connections 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4280641322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad9406bb00 0x7fad940c6080 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4280641322 >> v2:172.21.15.150:3300/0 conn(0x7fad94150560 0x7fad940c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4280641322 >> 172.21.15.78:0/4280641322 conn(0x7fad9406c170 msgr2=0x7fad9406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4280641322 shutdown_connections 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.797+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4280641322 wait complete. 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.798+0000 7fad9c2ed640 1 Processor -- start 2024-02-19T14:53:40.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.798+0000 7fad9c2ed640 1 -- start start 2024-02-19T14:53:40.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad9406bb00 0x7fad94152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 0x7fad941e5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad94150560 0x7fad941e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94150d70 con 0x7fad940721a0 2024-02-19T14:53:40.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94156680 con 0x7fad9406bb00 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940aa0b0 con 0x7fad94150560 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 0x7fad941e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 0x7fad941e5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50898/0 (socket says 172.21.15.78:50898) 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 -- 172.21.15.78:0/4293946022 learned_addr learned my addr 172.21.15.78:0/4293946022 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9a88a640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad94150560 0x7fad941e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9a089640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad9406bb00 0x7fad94152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 -- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad94150560 msgr2=0x7fad941e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad94150560 0x7fad941e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 -- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad9406bb00 msgr2=0x7fad94152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad9406bb00 0x7fad94152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 -- 172.21.15.78:0/4293946022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c009400 con 0x7fad940721a0 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9a88a640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad94150560 0x7fad941e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9a089640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad9406bb00 0x7fad94152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad99888640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 0x7fad941e5600 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fad84036040 tx=0x7fad840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad840339b0 con 0x7fad940721a0 2024-02-19T14:53:40.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad84033b30 con 0x7fad940721a0 2024-02-19T14:53:40.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad84009400 con 0x7fad940721a0 2024-02-19T14:53:40.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8403c400 con 0x7fad940721a0 2024-02-19T14:53:40.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.800+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad941ec1d0 con 0x7fad940721a0 2024-02-19T14:53:40.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.801+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad94150ee0 con 0x7fad940721a0 2024-02-19T14:53:40.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.802+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fad8403a040 con 0x7fad940721a0 2024-02-19T14:53:40.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.802+0000 7fad8affd640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad7806b810 0x7fad7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:40.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.802+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fad840ab790 con 0x7fad940721a0 2024-02-19T14:53:40.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.802+0000 7fad9a089640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad7806b810 0x7fad7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:40.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.803+0000 7fad9a089640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad7806b810 0x7fad7806dcd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009d20 tx=0x7fad8c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:40.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.807+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fad94150ee0 con 0x7fad940721a0 2024-02-19T14:53:40.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.981+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad94150ee0 con 0x7fad7806b810 2024-02-19T14:53:40.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.988+0000 7fad8affd640 1 -- 172.21.15.78:0/4293946022 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289591 (secure 0 0 0) 0x7fad94150ee0 con 0x7fad7806b810 2024-02-19T14:53:40.993 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:40.995 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad7806b810 msgr2=0x7fad7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad7806b810 0x7fad7806dcd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009d20 tx=0x7fad8c008040 comp rx=0 tx=0).stop 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 msgr2=0x7fad941e5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 0x7fad941e5600 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fad84036040 tx=0x7fad840046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 shutdown_connections 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad7806b810 0x7fad7806dcd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad940721a0 0x7fad941e5600 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad94150560 0x7fad941e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 --2- 172.21.15.78:0/4293946022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad9406bb00 0x7fad94152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 >> 172.21.15.78:0/4293946022 conn(0x7fad9406c170 msgr2=0x7fad9406ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 shutdown_connections 2024-02-19T14:53:40.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:40.992+0000 7fad9c2ed640 1 -- 172.21.15.78:0/4293946022 wait complete. 2024-02-19T14:53:41.009 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-02-19T14:53:40.695997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221944,"kb_used_data":5768,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752776,"statfs":{"total":773094113280,"available":772866842624,"internally_reserved":0,"allocated":5906432,"data_stored":3261045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.616516"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698194+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:37.698194+0000","last_peered":"2024-02-19T14:53:37.698194+0000","last_clean":"2024-02-19T14:53:37.698194+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:37.698194+0000","last_undegraded":"2024-02-19T14:53:37.698194+0000","last_fullsized":"2024-02-19T14:53:37.698194+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698436+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:37.698436+0000","last_peered":"2024-02-19T14:53:37.698436+0000","last_clean":"2024-02-19T14:53:37.698436+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:37.698436+0000","last_undegraded":"2024-02-19T14:53:37.698436+0000","last_fullsized":"2024-02-19T14:53:37.698436+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708593+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:37.708593+0000","last_peered":"2024-02-19T14:53:37.708593+0000","last_clean":"2024-02-19T14:53:37.708593+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:37.708593+0000","last_undegraded":"2024-02-19T14:53:37.708593+0000","last_fullsized":"2024-02-19T14:53:37.708593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700534+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:37.700534+0000","last_peered":"2024-02-19T14:53:37.700534+0000","last_clean":"2024-02-19T14:53:37.700534+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:37.700534+0000","last_undegraded":"2024-02-19T14:53:37.700534+0000","last_fullsized":"2024-02-19T14:53:37.700534+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689550+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:37.689550+0000","last_peered":"2024-02-19T14:53:37.689550+0000","last_clean":"2024-02-19T14:53:37.689550+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:37.689550+0000","last_undegraded":"2024-02-19T14:53:37.689550+0000","last_fullsized":"2024-02-19T14:53:37.689550+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:19.938736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689390+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:37.689390+0000","last_peered":"2024-02-19T14:53:37.689390+0000","last_clean":"2024-02-19T14:53:37.689390+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:37.689390+0000","last_undegraded":"2024-02-19T14:53:37.689390+0000","last_fullsized":"2024-02-19T14:53:37.689390+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:18.042708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707125+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:37.707125+0000","last_peered":"2024-02-19T14:53:37.707125+0000","last_clean":"2024-02-19T14:53:37.707125+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:37.707125+0000","last_undegraded":"2024-02-19T14:53:37.707125+0000","last_fullsized":"2024-02-19T14:53:37.707125+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706675+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:37.706675+0000","last_peered":"2024-02-19T14:53:37.706675+0000","last_clean":"2024-02-19T14:53:37.706675+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:37.706675+0000","last_undegraded":"2024-02-19T14:53:37.706675+0000","last_fullsized":"2024-02-19T14:53:37.706675+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.711305+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:37.711305+0000","last_peered":"2024-02-19T14:53:37.711305+0000","last_clean":"2024-02-19T14:53:37.711305+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:37.711305+0000","last_undegraded":"2024-02-19T14:53:37.711305+0000","last_fullsized":"2024-02-19T14:53:37.711305+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689363+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:37.689363+0000","last_peered":"2024-02-19T14:53:37.689363+0000","last_clean":"2024-02-19T14:53:37.689363+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:37.689363+0000","last_undegraded":"2024-02-19T14:53:37.689363+0000","last_fullsized":"2024-02-19T14:53:37.689363+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:25.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.693861+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:37.693861+0000","last_peered":"2024-02-19T14:53:37.693861+0000","last_clean":"2024-02-19T14:53:37.693861+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:37.693861+0000","last_undegraded":"2024-02-19T14:53:37.693861+0000","last_fullsized":"2024-02-19T14:53:37.693861+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:12.858220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692643+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:37.692643+0000","last_peered":"2024-02-19T14:53:37.692643+0000","last_clean":"2024-02-19T14:53:37.692643+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:37.692643+0000","last_undegraded":"2024-02-19T14:53:37.692643+0000","last_fullsized":"2024-02-19T14:53:37.692643+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689360+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:37.689360+0000","last_peered":"2024-02-19T14:53:37.689360+0000","last_clean":"2024-02-19T14:53:37.689360+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:37.689360+0000","last_undegraded":"2024-02-19T14:53:37.689360+0000","last_fullsized":"2024-02-19T14:53:37.689360+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708687+0000","last_change":"2024-02-19T14:53:37.703840+0000","last_active":"2024-02-19T14:53:37.708687+0000","last_peered":"2024-02-19T14:53:37.708687+0000","last_clean":"2024-02-19T14:53:37.708687+0000","last_became_active":"2024-02-19T14:53:37.703424+0000","last_became_peered":"2024-02-19T14:53:37.703424+0000","last_unstale":"2024-02-19T14:53:37.708687+0000","last_undegraded":"2024-02-19T14:53:37.708687+0000","last_fullsized":"2024-02-19T14:53:37.708687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698240+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:37.698240+0000","last_peered":"2024-02-19T14:53:37.698240+0000","last_clean":"2024-02-19T14:53:37.698240+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:37.698240+0000","last_undegraded":"2024-02-19T14:53:37.698240+0000","last_fullsized":"2024-02-19T14:53:37.698240+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689435+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:37.689435+0000","last_peered":"2024-02-19T14:53:37.689435+0000","last_clean":"2024-02-19T14:53:37.689435+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:37.689435+0000","last_undegraded":"2024-02-19T14:53:37.689435+0000","last_fullsized":"2024-02-19T14:53:37.689435+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708676+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:37.708676+0000","last_peered":"2024-02-19T14:53:37.708676+0000","last_clean":"2024-02-19T14:53:37.708676+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:37.708676+0000","last_undegraded":"2024-02-19T14:53:37.708676+0000","last_fullsized":"2024-02-19T14:53:37.708676+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698415+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:37.698415+0000","last_peered":"2024-02-19T14:53:37.698415+0000","last_clean":"2024-02-19T14:53:37.698415+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:37.698415+0000","last_undegraded":"2024-02-19T14:53:37.698415+0000","last_fullsized":"2024-02-19T14:53:37.698415+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707151+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:37.707151+0000","last_peered":"2024-02-19T14:53:37.707151+0000","last_clean":"2024-02-19T14:53:37.707151+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:37.707151+0000","last_undegraded":"2024-02-19T14:53:37.707151+0000","last_fullsized":"2024-02-19T14:53:37.707151+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.691201+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:37.691201+0000","last_peered":"2024-02-19T14:53:37.691201+0000","last_clean":"2024-02-19T14:53:37.691201+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:37.691201+0000","last_undegraded":"2024-02-19T14:53:37.691201+0000","last_fullsized":"2024-02-19T14:53:37.691201+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700407+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:37.700407+0000","last_peered":"2024-02-19T14:53:37.700407+0000","last_clean":"2024-02-19T14:53:37.700407+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:37.700407+0000","last_undegraded":"2024-02-19T14:53:37.700407+0000","last_fullsized":"2024-02-19T14:53:37.700407+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.711288+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:37.711288+0000","last_peered":"2024-02-19T14:53:37.711288+0000","last_clean":"2024-02-19T14:53:37.711288+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:37.711288+0000","last_undegraded":"2024-02-19T14:53:37.711288+0000","last_fullsized":"2024-02-19T14:53:37.711288+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.694199+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:37.694199+0000","last_peered":"2024-02-19T14:53:37.694199+0000","last_clean":"2024-02-19T14:53:37.694199+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:37.694199+0000","last_undegraded":"2024-02-19T14:53:37.694199+0000","last_fullsized":"2024-02-19T14:53:37.694199+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:49:44.347720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707041+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:37.707041+0000","last_peered":"2024-02-19T14:53:37.707041+0000","last_clean":"2024-02-19T14:53:37.707041+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:37.707041+0000","last_undegraded":"2024-02-19T14:53:37.707041+0000","last_fullsized":"2024-02-19T14:53:37.707041+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707099+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:37.707099+0000","last_peered":"2024-02-19T14:53:37.707099+0000","last_clean":"2024-02-19T14:53:37.707099+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:37.707099+0000","last_undegraded":"2024-02-19T14:53:37.707099+0000","last_fullsized":"2024-02-19T14:53:37.707099+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.701967+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:37.701967+0000","last_peered":"2024-02-19T14:53:37.701967+0000","last_clean":"2024-02-19T14:53:37.701967+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:37.701967+0000","last_undegraded":"2024-02-19T14:53:37.701967+0000","last_fullsized":"2024-02-19T14:53:37.701967+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.705823+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:37.705823+0000","last_peered":"2024-02-19T14:53:37.705823+0000","last_clean":"2024-02-19T14:53:37.705823+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:37.705823+0000","last_undegraded":"2024-02-19T14:53:37.705823+0000","last_fullsized":"2024-02-19T14:53:37.705823+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:56.120959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700583+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:37.700583+0000","last_peered":"2024-02-19T14:53:37.700583+0000","last_clean":"2024-02-19T14:53:37.700583+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:37.700583+0000","last_undegraded":"2024-02-19T14:53:37.700583+0000","last_fullsized":"2024-02-19T14:53:37.700583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:07.107866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.711324+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:37.711324+0000","last_peered":"2024-02-19T14:53:37.711324+0000","last_clean":"2024-02-19T14:53:37.711324+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:37.711324+0000","last_undegraded":"2024-02-19T14:53:37.711324+0000","last_fullsized":"2024-02-19T14:53:37.711324+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698296+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:37.698296+0000","last_peered":"2024-02-19T14:53:37.698296+0000","last_clean":"2024-02-19T14:53:37.698296+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:37.698296+0000","last_undegraded":"2024-02-19T14:53:37.698296+0000","last_fullsized":"2024-02-19T14:53:37.698296+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703181+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:37.703181+0000","last_peered":"2024-02-19T14:53:37.703181+0000","last_clean":"2024-02-19T14:53:37.703181+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:37.703181+0000","last_undegraded":"2024-02-19T14:53:37.703181+0000","last_fullsized":"2024-02-19T14:53:37.703181+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:56.120959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700374+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:37.700374+0000","last_peered":"2024-02-19T14:53:37.700374+0000","last_clean":"2024-02-19T14:53:37.700374+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:37.700374+0000","last_undegraded":"2024-02-19T14:53:37.700374+0000","last_fullsized":"2024-02-19T14:53:37.700374+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698449+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:37.698449+0000","last_peered":"2024-02-19T14:53:37.698449+0000","last_clean":"2024-02-19T14:53:37.698449+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:37.698449+0000","last_undegraded":"2024-02-19T14:53:37.698449+0000","last_fullsized":"2024-02-19T14:53:37.698449+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.701148+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:37.701148+0000","last_peered":"2024-02-19T14:53:37.701148+0000","last_clean":"2024-02-19T14:53:37.701148+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:37.701148+0000","last_undegraded":"2024-02-19T14:53:37.701148+0000","last_fullsized":"2024-02-19T14:53:37.701148+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.690472+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:37.690472+0000","last_peered":"2024-02-19T14:53:37.690472+0000","last_clean":"2024-02-19T14:53:37.690472+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:37.690472+0000","last_undegraded":"2024-02-19T14:53:37.690472+0000","last_fullsized":"2024-02-19T14:53:37.690472+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692809+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:37.692809+0000","last_peered":"2024-02-19T14:53:37.692809+0000","last_clean":"2024-02-19T14:53:37.692809+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:37.692809+0000","last_undegraded":"2024-02-19T14:53:37.692809+0000","last_fullsized":"2024-02-19T14:53:37.692809+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700490+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:37.700490+0000","last_peered":"2024-02-19T14:53:37.700490+0000","last_clean":"2024-02-19T14:53:37.700490+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:37.700490+0000","last_undegraded":"2024-02-19T14:53:37.700490+0000","last_fullsized":"2024-02-19T14:53:37.700490+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.690711+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:37.690711+0000","last_peered":"2024-02-19T14:53:37.690711+0000","last_clean":"2024-02-19T14:53:37.690711+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:37.690711+0000","last_undegraded":"2024-02-19T14:53:37.690711+0000","last_fullsized":"2024-02-19T14:53:37.690711+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702116+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:37.702116+0000","last_peered":"2024-02-19T14:53:37.702116+0000","last_clean":"2024-02-19T14:53:37.702116+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:37.702116+0000","last_undegraded":"2024-02-19T14:53:37.702116+0000","last_fullsized":"2024-02-19T14:53:37.702116+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702310+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:37.702310+0000","last_peered":"2024-02-19T14:53:37.702310+0000","last_clean":"2024-02-19T14:53:37.702310+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:37.702310+0000","last_undegraded":"2024-02-19T14:53:37.702310+0000","last_fullsized":"2024-02-19T14:53:37.702310+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706165+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:37.706165+0000","last_peered":"2024-02-19T14:53:37.706165+0000","last_clean":"2024-02-19T14:53:37.706165+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:37.706165+0000","last_undegraded":"2024-02-19T14:53:37.706165+0000","last_fullsized":"2024-02-19T14:53:37.706165+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:25.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702380+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:37.702380+0000","last_peered":"2024-02-19T14:53:37.702380+0000","last_clean":"2024-02-19T14:53:37.702380+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:37.702380+0000","last_undegraded":"2024-02-19T14:53:37.702380+0000","last_fullsized":"2024-02-19T14:53:37.702380+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698380+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:37.698380+0000","last_peered":"2024-02-19T14:53:37.698380+0000","last_clean":"2024-02-19T14:53:37.698380+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:37.698380+0000","last_undegraded":"2024-02-19T14:53:37.698380+0000","last_fullsized":"2024-02-19T14:53:37.698380+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697911+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:37.697911+0000","last_peered":"2024-02-19T14:53:37.697911+0000","last_clean":"2024-02-19T14:53:37.697911+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:37.697911+0000","last_undegraded":"2024-02-19T14:53:37.697911+0000","last_fullsized":"2024-02-19T14:53:37.697911+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692542+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:37.692542+0000","last_peered":"2024-02-19T14:53:37.692542+0000","last_clean":"2024-02-19T14:53:37.692542+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:37.692542+0000","last_undegraded":"2024-02-19T14:53:37.692542+0000","last_fullsized":"2024-02-19T14:53:37.692542+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.681553+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:37.681553+0000","last_peered":"2024-02-19T14:53:37.681553+0000","last_clean":"2024-02-19T14:53:37.681553+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:37.681553+0000","last_undegraded":"2024-02-19T14:53:37.681553+0000","last_fullsized":"2024-02-19T14:53:37.681553+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706985+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:37.706985+0000","last_peered":"2024-02-19T14:53:37.706985+0000","last_clean":"2024-02-19T14:53:37.706985+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:37.706985+0000","last_undegraded":"2024-02-19T14:53:37.706985+0000","last_fullsized":"2024-02-19T14:53:37.706985+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.687872+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:37.687872+0000","last_peered":"2024-02-19T14:53:37.687872+0000","last_clean":"2024-02-19T14:53:37.687872+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:37.687872+0000","last_undegraded":"2024-02-19T14:53:37.687872+0000","last_fullsized":"2024-02-19T14:53:37.687872+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697553+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:37.697553+0000","last_peered":"2024-02-19T14:53:37.697553+0000","last_clean":"2024-02-19T14:53:37.697553+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:37.697553+0000","last_undegraded":"2024-02-19T14:53:37.697553+0000","last_fullsized":"2024-02-19T14:53:37.697553+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:13.235683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.681686+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:37.681686+0000","last_peered":"2024-02-19T14:53:37.681686+0000","last_clean":"2024-02-19T14:53:37.681686+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:37.681686+0000","last_undegraded":"2024-02-19T14:53:37.681686+0000","last_fullsized":"2024-02-19T14:53:37.681686+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697675+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:37.697675+0000","last_peered":"2024-02-19T14:53:37.697675+0000","last_clean":"2024-02-19T14:53:37.697675+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:37.697675+0000","last_undegraded":"2024-02-19T14:53:37.697675+0000","last_fullsized":"2024-02-19T14:53:37.697675+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.673417+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:37.673417+0000","last_peered":"2024-02-19T14:53:37.673417+0000","last_clean":"2024-02-19T14:53:37.673417+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:37.673417+0000","last_undegraded":"2024-02-19T14:53:37.673417+0000","last_fullsized":"2024-02-19T14:53:37.673417+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708576+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:37.708576+0000","last_peered":"2024-02-19T14:53:37.708576+0000","last_clean":"2024-02-19T14:53:37.708576+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:37.708576+0000","last_undegraded":"2024-02-19T14:53:37.708576+0000","last_fullsized":"2024-02-19T14:53:37.708576+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.699136+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:37.699136+0000","last_peered":"2024-02-19T14:53:37.699136+0000","last_clean":"2024-02-19T14:53:37.699136+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:37.699136+0000","last_undegraded":"2024-02-19T14:53:37.699136+0000","last_fullsized":"2024-02-19T14:53:37.699136+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700373+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:37.700373+0000","last_peered":"2024-02-19T14:53:37.700373+0000","last_clean":"2024-02-19T14:53:37.700373+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:37.700373+0000","last_undegraded":"2024-02-19T14:53:37.700373+0000","last_fullsized":"2024-02-19T14:53:37.700373+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:47.458326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708901+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:37.708901+0000","last_peered":"2024-02-19T14:53:37.708901+0000","last_clean":"2024-02-19T14:53:37.708901+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:37.708901+0000","last_undegraded":"2024-02-19T14:53:37.708901+0000","last_fullsized":"2024-02-19T14:53:37.708901+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697707+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:37.697707+0000","last_peered":"2024-02-19T14:53:37.697707+0000","last_clean":"2024-02-19T14:53:37.697707+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:37.697707+0000","last_undegraded":"2024-02-19T14:53:37.697707+0000","last_fullsized":"2024-02-19T14:53:37.697707+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702767+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:37.702767+0000","last_peered":"2024-02-19T14:53:37.702767+0000","last_clean":"2024-02-19T14:53:37.702767+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:37.702767+0000","last_undegraded":"2024-02-19T14:53:37.702767+0000","last_fullsized":"2024-02-19T14:53:37.702767+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703703+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:37.703703+0000","last_peered":"2024-02-19T14:53:37.703703+0000","last_clean":"2024-02-19T14:53:37.703703+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:37.703703+0000","last_undegraded":"2024-02-19T14:53:37.703703+0000","last_fullsized":"2024-02-19T14:53:37.703703+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:07.107866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.681151+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:37.681151+0000","last_peered":"2024-02-19T14:53:37.681151+0000","last_clean":"2024-02-19T14:53:37.681151+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:37.681151+0000","last_undegraded":"2024-02-19T14:53:37.681151+0000","last_fullsized":"2024-02-19T14:53:37.681151+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692563+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:37.692563+0000","last_peered":"2024-02-19T14:53:37.692563+0000","last_clean":"2024-02-19T14:53:37.692563+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:37.692563+0000","last_undegraded":"2024-02-19T14:53:37.692563+0000","last_fullsized":"2024-02-19T14:53:37.692563+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.680817+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:37.680817+0000","last_peered":"2024-02-19T14:53:37.680817+0000","last_clean":"2024-02-19T14:53:37.680817+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:37.680817+0000","last_undegraded":"2024-02-19T14:53:37.680817+0000","last_fullsized":"2024-02-19T14:53:37.680817+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.708655+0000","last_change":"2024-02-19T14:53:37.703805+0000","last_active":"2024-02-19T14:53:37.708655+0000","last_peered":"2024-02-19T14:53:37.708655+0000","last_clean":"2024-02-19T14:53:37.708655+0000","last_became_active":"2024-02-19T14:53:37.703436+0000","last_became_peered":"2024-02-19T14:53:37.703436+0000","last_unstale":"2024-02-19T14:53:37.708655+0000","last_undegraded":"2024-02-19T14:53:37.708655+0000","last_fullsized":"2024-02-19T14:53:37.708655+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700043+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:37.700043+0000","last_peered":"2024-02-19T14:53:37.700043+0000","last_clean":"2024-02-19T14:53:37.700043+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:37.700043+0000","last_undegraded":"2024-02-19T14:53:37.700043+0000","last_fullsized":"2024-02-19T14:53:37.700043+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.695143+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:37.695143+0000","last_peered":"2024-02-19T14:53:37.695143+0000","last_clean":"2024-02-19T14:53:37.695143+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:37.695143+0000","last_undegraded":"2024-02-19T14:53:37.695143+0000","last_fullsized":"2024-02-19T14:53:37.695143+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703275+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:37.703275+0000","last_peered":"2024-02-19T14:53:37.703275+0000","last_clean":"2024-02-19T14:53:37.703275+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:37.703275+0000","last_undegraded":"2024-02-19T14:53:37.703275+0000","last_fullsized":"2024-02-19T14:53:37.703275+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.702141+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:37.702141+0000","last_peered":"2024-02-19T14:53:37.702141+0000","last_clean":"2024-02-19T14:53:37.702141+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:37.702141+0000","last_undegraded":"2024-02-19T14:53:37.702141+0000","last_fullsized":"2024-02-19T14:53:37.702141+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:07.107866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703010+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:37.703010+0000","last_peered":"2024-02-19T14:53:37.703010+0000","last_clean":"2024-02-19T14:53:37.703010+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:37.703010+0000","last_undegraded":"2024-02-19T14:53:37.703010+0000","last_fullsized":"2024-02-19T14:53:37.703010+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.706185+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:37.706185+0000","last_peered":"2024-02-19T14:53:37.706185+0000","last_clean":"2024-02-19T14:53:37.706185+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:37.706185+0000","last_undegraded":"2024-02-19T14:53:37.706185+0000","last_fullsized":"2024-02-19T14:53:37.706185+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:18.042708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700653+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:37.700653+0000","last_peered":"2024-02-19T14:53:37.700653+0000","last_clean":"2024-02-19T14:53:37.700653+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:37.700653+0000","last_undegraded":"2024-02-19T14:53:37.700653+0000","last_fullsized":"2024-02-19T14:53:37.700653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689980+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:37.689980+0000","last_peered":"2024-02-19T14:53:37.689980+0000","last_clean":"2024-02-19T14:53:37.689980+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:37.689980+0000","last_undegraded":"2024-02-19T14:53:37.689980+0000","last_fullsized":"2024-02-19T14:53:37.689980+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.680848+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:37.680848+0000","last_peered":"2024-02-19T14:53:37.680848+0000","last_clean":"2024-02-19T14:53:37.680848+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:37.680848+0000","last_undegraded":"2024-02-19T14:53:37.680848+0000","last_fullsized":"2024-02-19T14:53:37.680848+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697730+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:37.697730+0000","last_peered":"2024-02-19T14:53:37.697730+0000","last_clean":"2024-02-19T14:53:37.697730+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:37.697730+0000","last_undegraded":"2024-02-19T14:53:37.697730+0000","last_fullsized":"2024-02-19T14:53:37.697730+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.694511+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:37.694511+0000","last_peered":"2024-02-19T14:53:37.694511+0000","last_clean":"2024-02-19T14:53:37.694511+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:37.694511+0000","last_undegraded":"2024-02-19T14:53:37.694511+0000","last_fullsized":"2024-02-19T14:53:37.694511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697565+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:37.697565+0000","last_peered":"2024-02-19T14:53:37.697565+0000","last_clean":"2024-02-19T14:53:37.697565+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:37.697565+0000","last_undegraded":"2024-02-19T14:53:37.697565+0000","last_fullsized":"2024-02-19T14:53:37.697565+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.672664+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:37.672664+0000","last_peered":"2024-02-19T14:53:37.672664+0000","last_clean":"2024-02-19T14:53:37.672664+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:37.672664+0000","last_undegraded":"2024-02-19T14:53:37.672664+0000","last_fullsized":"2024-02-19T14:53:37.672664+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.703209+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:37.703209+0000","last_peered":"2024-02-19T14:53:37.703209+0000","last_clean":"2024-02-19T14:53:37.703209+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:37.703209+0000","last_undegraded":"2024-02-19T14:53:37.703209+0000","last_fullsized":"2024-02-19T14:53:37.703209+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689485+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:37.689485+0000","last_peered":"2024-02-19T14:53:37.689485+0000","last_clean":"2024-02-19T14:53:37.689485+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:37.689485+0000","last_undegraded":"2024-02-19T14:53:37.689485+0000","last_fullsized":"2024-02-19T14:53:37.689485+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.689477+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:37.689477+0000","last_peered":"2024-02-19T14:53:37.689477+0000","last_clean":"2024-02-19T14:53:37.689477+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:37.689477+0000","last_undegraded":"2024-02-19T14:53:37.689477+0000","last_fullsized":"2024-02-19T14:53:37.689477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.679673+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:37.679673+0000","last_peered":"2024-02-19T14:53:37.679673+0000","last_clean":"2024-02-19T14:53:37.679673+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:37.679673+0000","last_undegraded":"2024-02-19T14:53:37.679673+0000","last_fullsized":"2024-02-19T14:53:37.679673+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700352+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:37.700352+0000","last_peered":"2024-02-19T14:53:37.700352+0000","last_clean":"2024-02-19T14:53:37.700352+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:37.700352+0000","last_undegraded":"2024-02-19T14:53:37.700352+0000","last_fullsized":"2024-02-19T14:53:37.700352+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.704078+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:37.704078+0000","last_peered":"2024-02-19T14:53:37.704078+0000","last_clean":"2024-02-19T14:53:37.704078+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:37.704078+0000","last_undegraded":"2024-02-19T14:53:37.704078+0000","last_fullsized":"2024-02-19T14:53:37.704078+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.698254+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:37.698254+0000","last_peered":"2024-02-19T14:53:37.698254+0000","last_clean":"2024-02-19T14:53:37.698254+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:37.698254+0000","last_undegraded":"2024-02-19T14:53:37.698254+0000","last_fullsized":"2024-02-19T14:53:37.698254+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:57.477164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.683514+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:37.683514+0000","last_peered":"2024-02-19T14:53:37.683514+0000","last_clean":"2024-02-19T14:53:37.683514+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:37.683514+0000","last_undegraded":"2024-02-19T14:53:37.683514+0000","last_fullsized":"2024-02-19T14:53:37.683514+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692667+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:37.692667+0000","last_peered":"2024-02-19T14:53:37.692667+0000","last_clean":"2024-02-19T14:53:37.692667+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:37.692667+0000","last_undegraded":"2024-02-19T14:53:37.692667+0000","last_fullsized":"2024-02-19T14:53:37.692667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.688134+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:37.688134+0000","last_peered":"2024-02-19T14:53:37.688134+0000","last_clean":"2024-02-19T14:53:37.688134+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:37.688134+0000","last_undegraded":"2024-02-19T14:53:37.688134+0000","last_fullsized":"2024-02-19T14:53:37.688134+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707086+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:37.707086+0000","last_peered":"2024-02-19T14:53:37.707086+0000","last_clean":"2024-02-19T14:53:37.707086+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:37.707086+0000","last_undegraded":"2024-02-19T14:53:37.707086+0000","last_fullsized":"2024-02-19T14:53:37.707086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.674888+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:37.674888+0000","last_peered":"2024-02-19T14:53:37.674888+0000","last_clean":"2024-02-19T14:53:37.674888+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:37.674888+0000","last_undegraded":"2024-02-19T14:53:37.674888+0000","last_fullsized":"2024-02-19T14:53:37.674888+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.697573+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:37.697573+0000","last_peered":"2024-02-19T14:53:37.697573+0000","last_clean":"2024-02-19T14:53:37.697573+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:37.697573+0000","last_undegraded":"2024-02-19T14:53:37.697573+0000","last_fullsized":"2024-02-19T14:53:37.697573+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.672789+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:37.672789+0000","last_peered":"2024-02-19T14:53:37.672789+0000","last_clean":"2024-02-19T14:53:37.672789+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:37.672789+0000","last_undegraded":"2024-02-19T14:53:37.672789+0000","last_fullsized":"2024-02-19T14:53:37.672789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.707024+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:37.707024+0000","last_peered":"2024-02-19T14:53:37.707024+0000","last_clean":"2024-02-19T14:53:37.707024+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:37.707024+0000","last_undegraded":"2024-02-19T14:53:37.707024+0000","last_fullsized":"2024-02-19T14:53:37.707024+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.673418+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:37.673418+0000","last_peered":"2024-02-19T14:53:37.673418+0000","last_clean":"2024-02-19T14:53:37.673418+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:37.673418+0000","last_undegraded":"2024-02-19T14:53:37.673418+0000","last_fullsized":"2024-02-19T14:53:37.673418+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.692514+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:37.692514+0000","last_peered":"2024-02-19T14:53:37.692514+0000","last_clean":"2024-02-19T14:53:37.692514+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:37.692514+0000","last_undegraded":"2024-02-19T14:53:37.692514+0000","last_fullsized":"2024-02-19T14:53:37.692514+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.677290+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:37.677290+0000","last_peered":"2024-02-19T14:53:37.677290+0000","last_clean":"2024-02-19T14:53:37.677290+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:37.677290+0000","last_undegraded":"2024-02-19T14:53:37.677290+0000","last_fullsized":"2024-02-19T14:53:37.677290+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.700547+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:37.700547+0000","last_peered":"2024-02-19T14:53:37.700547+0000","last_clean":"2024-02-19T14:53:37.700547+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:37.700547+0000","last_undegraded":"2024-02-19T14:53:37.700547+0000","last_fullsized":"2024-02-19T14:53:37.700547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:57.477164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.687364+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:37.687364+0000","last_peered":"2024-02-19T14:53:37.687364+0000","last_clean":"2024-02-19T14:53:37.687364+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:37.687364+0000","last_undegraded":"2024-02-19T14:53:37.687364+0000","last_fullsized":"2024-02-19T14:53:37.687364+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":226,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-02-19T14:53:37.680604+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:37.680604+0000","last_peered":"2024-02-19T14:53:37.680604+0000","last_clean":"2024-02-19T14:53:37.680604+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:37.680604+0000","last_undegraded":"2024-02-19T14:53:37.680604+0000","last_fullsized":"2024-02-19T14:53:37.680604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1196032,"data_stored":854773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.505},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.432}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.624},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57299999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.49099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.435}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.36799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61699999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.59299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.66400000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.78000000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.81499999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.86899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":257114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.626},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.61699999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.57399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":265893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.254}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.215},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.94199999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":264441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":847482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.27100000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.314}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.501},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:41.013 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:53:41.014 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:53:41.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.154+0000 7f63d1c12640 1 Processor -- start 2024-02-19T14:53:41.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.154+0000 7f63d1c12640 1 -- start start 2024-02-19T14:53:41.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63d1c12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f63cc0c25e0 0x7f63cc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63d1c12640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f63cc0c5be0 0x7f63cc06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63d1c12640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc072250 con 0x7f63cc0c29c0 2024-02-19T14:53:41.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63d1c12640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc152c90 con 0x7f63cc0c5be0 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63d1c12640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc072480 con 0x7f63cc0c25e0 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f63cc0c5be0 0x7f63cc06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f63cc0c5be0 0x7f63cc06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50906/0 (socket says 172.21.15.78:50906) 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 -- 172.21.15.78:0/3208194792 learned_addr learned my addr 172.21.15.78:0/3208194792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cb7fe640 1 --2- 172.21.15.78:0/3208194792 >> v2:172.21.15.150:3300/0 conn(0x7f63cc0c25e0 0x7f63cc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 -- 172.21.15.78:0/3208194792 >> v2:172.21.15.150:3300/0 conn(0x7f63cc0c25e0 msgr2=0x7f63cc1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 --2- 172.21.15.78:0/3208194792 >> v2:172.21.15.150:3300/0 conn(0x7f63cc0c25e0 0x7f63cc1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 -- 172.21.15.78:0/3208194792 >> v1:172.21.15.150:6789/0 conn(0x7f63cc0c29c0 legacy=0x7f63cc0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cbfff640 1 -- 172.21.15.78:0/3208194792 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0c4b90 con 0x7f63cc0c5be0 2024-02-19T14:53:41.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.155+0000 7f63cb7fe640 1 --2- 172.21.15.78:0/3208194792 >> v2:172.21.15.150:3300/0 conn(0x7f63cc0c25e0 0x7f63cc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:41.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63cbfff640 1 --2- 172.21.15.78:0/3208194792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c5be0 0x7f63cc06e420 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f63cc06f120 tx=0x7f63c402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963fddef9d0a1b64 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63ca7fc640 1 -- 172.21.15.78:0/3208194792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c402ea00 con 0x7f63cc0c5be0 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63ca7fc640 1 -- 172.21.15.78:0/3208194792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c402eb60 con 0x7f63cc0c5be0 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63ca7fc640 1 -- 172.21.15.78:0/3208194792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4009600 con 0x7f63cc0c5be0 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 -- 172.21.15.78:0/3208194792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c5be0 msgr2=0x7f63cc06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 --2- 172.21.15.78:0/3208194792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c5be0 0x7f63cc06e420 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f63cc06f120 tx=0x7f63c402e060 comp rx=0 tx=0).stop 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 -- 172.21.15.78:0/3208194792 shutdown_connections 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 --2- 172.21.15.78:0/3208194792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c5be0 0x7f63cc06e420 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 --2- 172.21.15.78:0/3208194792 >> v2:172.21.15.150:3300/0 conn(0x7f63cc0c25e0 0x7f63cc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 -- 172.21.15.78:0/3208194792 >> 172.21.15.78:0/3208194792 conn(0x7f63cc06b270 msgr2=0x7f63cc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 -- 172.21.15.78:0/3208194792 shutdown_connections 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.157+0000 7f63d1c12640 1 -- 172.21.15.78:0/3208194792 wait complete. 2024-02-19T14:53:41.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.158+0000 7f63d1c12640 1 Processor -- start 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 -- start start 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 0x7f63cc1ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63cc0c29c0 0x7f63cc09a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63cc0c5be0 0x7f63cc14d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1ede20 con 0x7f63cc0c25e0 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1ead90 con 0x7f63cc0c29c0 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63d1c12640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1eb070 con 0x7f63cc0c5be0 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 0x7f63cc1ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 0x7f63cc1ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50918/0 (socket says 172.21.15.78:50918) 2024-02-19T14:53:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.159+0000 7f63cb7fe640 1 -- 172.21.15.78:0/347740031 learned_addr learned my addr 172.21.15.78:0/347740031 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63caffd640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63cc0c29c0 0x7f63cc09a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cbfff640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63cc0c5be0 0x7f63cc14d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cb7fe640 1 -- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63cc0c5be0 msgr2=0x7f63cc14d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cb7fe640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63cc0c5be0 0x7f63cc14d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cb7fe640 1 -- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63cc0c29c0 msgr2=0x7f63cc09a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cb7fe640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63cc0c29c0 0x7f63cc09a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cb7fe640 1 -- 172.21.15.78:0/347740031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c400bd40 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63cb7fe640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 0x7f63cc1ee2b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00c7f0 tx=0x7f63bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc007820 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc007980 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc007cf0 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc1ed7d0 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.160+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc14ddc0 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.161+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc074990 con 0x7f63cc0c25e0 2024-02-19T14:53:41.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.162+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f63bc0113f0 con 0x7f63cc0c25e0 2024-02-19T14:53:41.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.162+0000 7f63b3fff640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f63b806b810 0x7f63b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.162+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f63bc08a680 con 0x7f63cc0c25e0 2024-02-19T14:53:41.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.162+0000 7f63caffd640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f63b806b810 0x7f63b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.163+0000 7f63caffd640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f63b806b810 0x7f63b806dcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0a93b0 tx=0x7f63c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:41.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.167+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63bc054f70 con 0x7f63cc0c25e0 2024-02-19T14:53:41.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.406+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f63cc0c0bd0 con 0x7f63cc0c25e0 2024-02-19T14:53:41.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.408+0000 7f63b3fff640 1 -- 172.21.15.78:0/347740031 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f63bc058630 con 0x7f63cc0c25e0 2024-02-19T14:53:41.411 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:53:41.411 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:53:41.411 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:53:41.411 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:53:41.411 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:53:41.411 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f63b806b810 msgr2=0x7f63b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f63b806b810 0x7f63b806dcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0a93b0 tx=0x7f63c0009040 comp rx=0 tx=0).stop 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 msgr2=0x7f63cc1ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 0x7f63cc1ee2b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00c7f0 tx=0x7f63bc00ccb0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 shutdown_connections 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f63b806b810 0x7f63b806dcd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63cc0c5be0 0x7f63cc14d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63cc0c29c0 0x7f63cc09a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 --2- 172.21.15.78:0/347740031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63cc0c25e0 0x7f63cc1ee2b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 >> 172.21.15.78:0/347740031 conn(0x7f63cc06b270 msgr2=0x7f63cc06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:41.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 shutdown_connections 2024-02-19T14:53:41.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.410+0000 7f63d1c12640 1 -- 172.21.15.78:0/347740031 wait complete. 2024-02-19T14:53:41.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.525+0000 7f4b5bb0d640 1 Processor -- start 2024-02-19T14:53:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.525+0000 7f4b5bb0d640 1 -- start start 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5bb0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b5406d000 0x7f4b54159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5bb0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b54156d60 0x7f4b5406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5bb0d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b540c6a60 con 0x7f4b540c6560 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5bb0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b541554b0 con 0x7f4b54156d60 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5bb0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b54155790 con 0x7f4b5406d000 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5a0aa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b54156d60 0x7f4b5406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b590a8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4b540c6560 0x7f4b54156650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42116/0 (socket says 172.21.15.78:42116) 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b590a8640 1 -- 172.21.15.78:0/1947085043 learned_addr learned my addr 172.21.15.78:0/1947085043 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b598a9640 1 --2- 172.21.15.78:0/1947085043 >> v2:172.21.15.150:3300/0 conn(0x7f4b5406d000 0x7f4b54159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b588a7640 1 -- 172.21.15.78:0/1947085043 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842831993 0 0) 0x7f4b540c6a60 con 0x7f4b540c6560 2024-02-19T14:53:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b588a7640 1 -- 172.21.15.78:0/1947085043 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b40003520 con 0x7f4b540c6560 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/1947085043 >> v2:172.21.15.150:3300/0 conn(0x7f4b5406d000 msgr2=0x7f4b54159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5a0aa640 1 --2- 172.21.15.78:0/1947085043 >> v2:172.21.15.150:3300/0 conn(0x7f4b5406d000 0x7f4b54159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/1947085043 >> v1:172.21.15.78:6789/0 conn(0x7f4b540c6560 legacy=0x7f4b54156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/1947085043 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b54155a70 con 0x7f4b54156d60 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.526+0000 7f4b598a9640 1 --2- 172.21.15.78:0/1947085043 >> v2:172.21.15.150:3300/0 conn(0x7f4b5406d000 0x7f4b54159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.527+0000 7f4b5a0aa640 1 --2- 172.21.15.78:0/1947085043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5406d3e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4b5408bdd0 tx=0x7f4b4c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b456aae77a84a397 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.527+0000 7f4b588a7640 1 -- 172.21.15.78:0/1947085043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c02cbf0 con 0x7f4b54156d60 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.527+0000 7f4b588a7640 1 -- 172.21.15.78:0/1947085043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4c02cd50 con 0x7f4b54156d60 2024-02-19T14:53:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.527+0000 7f4b588a7640 1 -- 172.21.15.78:0/1947085043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c034660 con 0x7f4b54156d60 2024-02-19T14:53:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/1947085043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 msgr2=0x7f4b5406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/1947085043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5406d3e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4b5408bdd0 tx=0x7f4b4c02be40 comp rx=0 tx=0).stop 2024-02-19T14:53:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/1947085043 shutdown_connections 2024-02-19T14:53:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/1947085043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5406d3e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/1947085043 >> v2:172.21.15.150:3300/0 conn(0x7f4b5406d000 0x7f4b54159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/1947085043 >> 172.21.15.78:0/1947085043 conn(0x7f4b540ca870 msgr2=0x7f4b540cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/1947085043 shutdown_connections 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.528+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/1947085043 wait complete. 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.529+0000 7f4b5bb0d640 1 Processor -- start 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.529+0000 7f4b5bb0d640 1 -- start start 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5bb0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5406d000 0x7f4b540afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5bb0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b540c6560 0x7f4b540c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5bb0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5414ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5bb0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b540c1580 con 0x7f4b54156d60 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5bb0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5414e1d0 con 0x7f4b5406d000 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5bb0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5414e4b0 con 0x7f4b540c6560 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5414ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5414ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50942/0 (socket says 172.21.15.78:50942) 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/2030416987 learned_addr learned my addr 172.21.15.78:0/2030416987 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b598a9640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5406d000 0x7f4b540afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b590a8640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b540c6560 0x7f4b540c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b540c6560 msgr2=0x7f4b540c0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b540c6560 0x7f4b540c0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5406d000 msgr2=0x7f4b540afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5406d000 0x7f4b540afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 -- 172.21.15.78:0/2030416987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c009a70 con 0x7f4b54156d60 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b590a8640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b540c6560 0x7f4b540c0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.530+0000 7f4b5a0aa640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5414ba70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009e80 tx=0x7f4b4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.531+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c040650 con 0x7f4b54156d60 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.531+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4c0407b0 con 0x7f4b54156d60 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.531+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b54156110 con 0x7f4b54156d60 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.531+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c040b60 con 0x7f4b54156d60 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.531+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5414ea70 con 0x7f4b54156d60 2024-02-19T14:53:41.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.532+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b54074990 con 0x7f4b54156d60 2024-02-19T14:53:41.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.533+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b4c040cc0 con 0x7f4b54156d60 2024-02-19T14:53:41.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.533+0000 7f4b4a7fc640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b3806b810 0x7f4b3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:41.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.533+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4b4c0ad100 con 0x7f4b54156d60 2024-02-19T14:53:41.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.533+0000 7f4b598a9640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b3806b810 0x7f4b3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:41.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.533+0000 7f4b598a9640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b3806b810 0x7f4b3806dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:41.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.537+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b4c077a80 con 0x7f4b54156d60 2024-02-19T14:53:41.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.754+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b540b0960 con 0x7f4b54156d60 2024-02-19T14:53:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.754+0000 7f4b4a7fc640 1 -- 172.21.15.78:0/2030416987 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f4b4c009450 con 0x7f4b54156d60 2024-02-19T14:53:41.757 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:41.757 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5398,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/210065453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":210065453},{"type":"v1","addr":"172.21.15.150:6834","nonce":210065453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:53:39.026813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5703":{"gid":5703,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1917648829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1917648829},{"type":"v1","addr":"172.21.15.78:6837","nonce":1917648829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T14:53:41.757 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 26 2024-02-19T14:53:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.756+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b3806b810 msgr2=0x7f4b3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.756+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b3806b810 0x7f4b3806dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44008040 comp rx=0 tx=0).stop 2024-02-19T14:53:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.756+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 msgr2=0x7f4b5414ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.756+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5414ba70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009e80 tx=0x7f4b4c004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 shutdown_connections 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b3806b810 0x7f4b3806dcd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b54156d60 0x7f4b5414ba70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b540c6560 0x7f4b540c0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 --2- 172.21.15.78:0/2030416987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b5406d000 0x7f4b540afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 >> 172.21.15.78:0/2030416987 conn(0x7f4b540ca870 msgr2=0x7f4b54156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 shutdown_connections 2024-02-19T14:53:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:41.757+0000 7f4b5bb0d640 1 -- 172.21.15.78:0/2030416987 wait complete. 2024-02-19T14:53:42.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:53:37.026854+0000', 'modified': '2024-02-19T14:53:39.026813+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5703}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5703': {'gid': 5703, 'name': 'a', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/1917648829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 1917648829}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1917648829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:53:42.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:53:42.173 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T14:53:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.281+0000 7f26dc55c640 1 Processor -- start 2024-02-19T14:53:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.281+0000 7f26dc55c640 1 -- start start 2024-02-19T14:53:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26dc55c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26d4072ee0 0x7f26d414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26dc55c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40c35c0 con 0x7f26d40c78a0 2024-02-19T14:53:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26dc55c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40c38a0 con 0x7f26d414ef50 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26dc55c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d40c4bc0 con 0x7f26d4072ee0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26da2f8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f26d40c78a0 0x7f26d40c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42130/0 (socket says 172.21.15.78:42130) 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26da2f8640 1 -- 172.21.15.78:0/1851322109 learned_addr learned my addr 172.21.15.78:0/1851322109 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26daaf9640 1 --2- 172.21.15.78:0/1851322109 >> v2:172.21.15.150:3300/0 conn(0x7f26d4072ee0 0x7f26d414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040678184 0 0) 0x7f26d40c35c0 con 0x7f26d40c78a0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0003520 con 0x7f26d40c78a0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1391653315 0 0) 0x7f26c0003520 con 0x7f26d40c78a0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 >> v2:172.21.15.150:3300/0 conn(0x7f26d4072ee0 msgr2=0x7f26d414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 --2- 172.21.15.78:0/1851322109 >> v2:172.21.15.150:3300/0 conn(0x7f26d4072ee0 0x7f26d414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 >> v1:172.21.15.150:6789/0 conn(0x7f26d414ef50 legacy=0x7f26d40727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40c3b80 con 0x7f26d40c78a0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f26d0001670 con 0x7f26d40c78a0 2024-02-19T14:53:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 >> v1:172.21.15.78:6789/0 conn(0x7f26d40c78a0 legacy=0x7f26d40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26c0003970 0x7f26c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c00047c0 0x7f26c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0005180 0x7f26c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d40c4bc0 con 0x7f26c0003970 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c000c020 con 0x7f26c00047c0 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.282+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c000c1e0 con 0x7f26c0005180 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26da2f8640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26c0003970 0x7f26c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26d9af7640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c00047c0 0x7f26c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26daaf9640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0005180 0x7f26c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26da2f8640 1 -- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0005180 msgr2=0x7f26c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26da2f8640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0005180 0x7f26c000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26da2f8640 1 -- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c00047c0 msgr2=0x7f26c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26da2f8640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c00047c0 0x7f26c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26da2f8640 1 -- 172.21.15.78:0/1851322109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0003520 con 0x7f26c0003970 2024-02-19T14:53:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26daaf9640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0005180 0x7f26c000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.283+0000 7f26daaf9640 1 -- 172.21.15.78:0/1851322109 reap_dead start 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.284+0000 7f26da2f8640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26c0003970 0x7f26c0003d90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f26d000c240 tx=0x7f26c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2935e7067ef5053 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.284+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0005ad0 con 0x7f26c0003970 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.284+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d0005c30 con 0x7f26c0003970 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.284+0000 7f26d92f6640 1 -- 172.21.15.78:0/1851322109 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d002c850 con 0x7f26c0003970 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 -- 172.21.15.78:0/1851322109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26c0003970 msgr2=0x7f26c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26c0003970 0x7f26c0003d90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f26d000c240 tx=0x7f26c00057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 -- 172.21.15.78:0/1851322109 shutdown_connections 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 --2- 172.21.15.78:0/1851322109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26c0003970 0x7f26c0003d90 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 -- 172.21.15.78:0/1851322109 >> 172.21.15.78:0/1851322109 conn(0x7f26d4066e70 msgr2=0x7f26d40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 -- 172.21.15.78:0/1851322109 shutdown_connections 2024-02-19T14:53:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.285+0000 7f26dc55c640 1 -- 172.21.15.78:0/1851322109 wait complete. 2024-02-19T14:53:42.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.286+0000 7f26dc55c640 1 Processor -- start 2024-02-19T14:53:42.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.286+0000 7f26dc55c640 1 -- start start 2024-02-19T14:53:42.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.286+0000 7f26dc55c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0003970 0x7f26d41eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.286+0000 7f26dc55c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 0x7f26d41edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.286+0000 7f26dc55c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26d41ee490 0x7f26d4148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26dc55c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d41edb30 con 0x7f26d40c0d40 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26dc55c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d4125c10 con 0x7f26d41ee490 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26dc55c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d41eb490 con 0x7f26c0003970 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 0x7f26d41edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 0x7f26d41edf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50950/0 (socket says 172.21.15.78:50950) 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 -- 172.21.15.78:0/2462138934 learned_addr learned my addr 172.21.15.78:0/2462138934 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0003970 0x7f26d41eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26daaf9640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26d41ee490 0x7f26d4148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 -- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0003970 msgr2=0x7f26d41eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0003970 0x7f26d41eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 -- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26d41ee490 msgr2=0x7f26d4148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26d41ee490 0x7f26d4148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 -- 172.21.15.78:0/2462138934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d4122d80 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26daaf9640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26d41ee490 0x7f26d4148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26d9af7640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 0x7f26d41edf50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f26c400f650 tx=0x7f26c400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26caffd640 1 -- 172.21.15.78:0/2462138934 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c40050d0 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26caffd640 1 -- 172.21.15.78:0/2462138934 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c4005230 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d4121bc0 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26caffd640 1 -- 172.21.15.78:0/2462138934 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c400d730 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.287+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d4122180 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.289+0000 7f26caffd640 1 -- 172.21.15.78:0/2462138934 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f26c400d890 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.289+0000 7f26caffd640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26b806b840 0x7f26b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.289+0000 7f26caffd640 1 -- 172.21.15.78:0/2462138934 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f26c408a250 con 0x7f26d40c0d40 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.289+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26b806b840 0x7f26b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.290+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26b806b840 0x7f26b806dd00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f26d0034000 tx=0x7f26d0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.291+0000 7f26df207740 1 Processor -- start 2024-02-19T14:53:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.291+0000 7f26df207740 1 -- start start 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.291+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99eb740 0x55b2d99ec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.291+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99b1ae0 0x55b2d99ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.291+0000 7f26df207740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 0x55b2d99b0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26df207740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d996bcd0 con 0x55b2d99ee8b0 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d991ef20 con 0x55b2d99b1ae0 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d991c450 con 0x55b2d99eb740 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 0x55b2d99b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 0x55b2d99b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50966/0 (socket says 172.21.15.78:50966) 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 -- 172.21.15.78:0/386288960 learned_addr learned my addr 172.21.15.78:0/386288960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26da2f8640 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99eb740 0x55b2d99ec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26d9af7640 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99b1ae0 0x55b2d99ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 -- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99eb740 msgr2=0x55b2d99ec8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99eb740 0x55b2d99ec8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 -- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99b1ae0 msgr2=0x55b2d99ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99b1ae0 0x55b2d99ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 -- 172.21.15.78:0/386288960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2d98ccf20 con 0x55b2d99ee8b0 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26daaf9640 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 0x55b2d99b0860 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f26d40aa4f0 tx=0x7f26cc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95b550c50eec66f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26b7fff640 1 -- 172.21.15.78:0/386288960 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc00e040 con 0x55b2d99ee8b0 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26b7fff640 1 -- 172.21.15.78:0/386288960 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc017660 con 0x55b2d99ee8b0 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.292+0000 7f26b7fff640 1 -- 172.21.15.78:0/386288960 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc0179f0 con 0x55b2d99ee8b0 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 -- 172.21.15.78:0/386288960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 msgr2=0x55b2d99b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 0x55b2d99b0860 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f26d40aa4f0 tx=0x7f26cc009750 comp rx=0 tx=0).stop 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 -- 172.21.15.78:0/386288960 shutdown_connections 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99ee8b0 0x55b2d99b0860 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99b1ae0 0x55b2d99ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 --2- 172.21.15.78:0/386288960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99eb740 0x55b2d99ec8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 -- 172.21.15.78:0/386288960 >> 172.21.15.78:0/386288960 conn(0x55b2d99ad120 msgr2=0x55b2d99d1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 -- 172.21.15.78:0/386288960 shutdown_connections 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.293+0000 7f26df207740 1 -- 172.21.15.78:0/386288960 wait complete. 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.294+0000 7f26df207740 1 -- start start 2024-02-19T14:53:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.294+0000 7f26df207740 1 Processor -- start 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d9917c30 con 0x55b2d99e3a70 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d98bcd30 con 0x55b2d99eb740 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d95eb310 con 0x55b2d99ee8b0 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 10 client.0 _ll_register_callbacks cb 0x55b2d98c4d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99c3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50978/0 (socket says 172.21.15.78:50978) 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 -- 172.21.15.78:0/699175538 learned_addr learned my addr 172.21.15.78:0/699175538 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26da2f8640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26d9af7640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 -- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 msgr2=0x55b2d99fd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99fd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 -- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 msgr2=0x55b2d99b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 -- 172.21.15.78:0/699175538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc009ca0 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26daaf9640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99c3550 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f26cc01a040 tx=0x7f26cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3383dfa1629011 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26b57fa640 1 -- 172.21.15.78:0/699175538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26cc00e050 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26b57fa640 1 -- 172.21.15.78:0/699175538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f26cc017d90 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b2d96176c0 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.296+0000 7f26b57fa640 1 -- 172.21.15.78:0/699175538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26cc004040 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 10 client.5850 fetch_fsmap learned FSMap version 26 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55b2d99162b0 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26b57fa640 1 -- 172.21.15.78:0/699175538 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b2d96176c0 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26b57fa640 1 -- 172.21.15.78:0/699175538 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f26cc0041a0 con 0x55b2d99e3a70 2024-02-19T14:53:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 10 client.5850 fetch_fsmap finished waiting for FSMap version 26 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 10 client.5850 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 20 client.5850 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99e7920 0x55b2d99e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 4 client.5850 mds_command: new command op to 5703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 --> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b2d9668270 con 0x55b2d99e7920 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26da2f8640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99e7920 0x55b2d99e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26b57fa640 10 client.5850 ms_handle_connect on v2:172.21.15.78:6835/1917648829 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.297+0000 7f26da2f8640 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99e7920 0x55b2d99e7d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20abb6e5d03af3a6 server_cookie=d827d7645ec01ec3 in_seq=0 out_seq=0 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26b57fa640 1 -- 172.21.15.78:0/699175538 <== mds.0 v2:172.21.15.78:6835/1917648829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b2d9668270 con 0x55b2d99e7920 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26b57fa640 10 client.5850 handle_command_reply: tid=0 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 client.5850 shutdown 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 msgr2=0x55b2d99c3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99c3550 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f26cc01a040 tx=0x7f26cc004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:42.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 shutdown_connections 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99e7920 0x55b2d99e7d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99c3550 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99fd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 --2- 172.21.15.78:0/699175538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 >> 172.21.15.78:0/699175538 conn(0x55b2d99b1ae0 msgr2=0x55b2d99d1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26b57fa640 0 client.5850 ms_handle_reset on v2:172.21.15.78:6835/1917648829 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 shutdown_connections 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 1 -- 172.21.15.78:0/699175538 wait complete. 2024-02-19T14:53:42.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.298+0000 7f26df207740 20 client.5850 trim_cache size 0 max 16384 2024-02-19T14:53:42.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.315+0000 7f26df207740 1 Processor -- start 2024-02-19T14:53:42.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.315+0000 7f26df207740 1 -- start start 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.315+0000 7f26df207740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99ecc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.315+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99e7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26df207740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d989d490 con 0x55b2d99e3a70 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d99e8ed0 con 0x55b2d99eb740 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d99ce8a0 con 0x55b2d99ee8b0 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99ecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99ecc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50980/0 (socket says 172.21.15.78:50980) 2024-02-19T14:53:42.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 -- 172.21.15.78:0/3824437893 learned_addr learned my addr 172.21.15.78:0/3824437893 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 -- 172.21.15.78:0/3824437893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 msgr2=0x55b2d99e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 -- 172.21.15.78:0/3824437893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 msgr2=0x55b2d99e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 -- 172.21.15.78:0/3824437893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2d99eaaf0 con 0x55b2d99e3a70 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26da2f8640 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99ecc90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f26d0007f30 tx=0x7f26d0007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152be7d79654a229 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26b77fe640 1 -- 172.21.15.78:0/3824437893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0001470 con 0x55b2d99e3a70 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26b77fe640 1 -- 172.21.15.78:0/3824437893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d003d040 con 0x55b2d99e3a70 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.316+0000 7f26b77fe640 1 -- 172.21.15.78:0/3824437893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d002cce0 con 0x55b2d99e3a70 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 -- 172.21.15.78:0/3824437893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 msgr2=0x55b2d99ecc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99ecc90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f26d0007f30 tx=0x7f26d0007000 comp rx=0 tx=0).stop 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 -- 172.21.15.78:0/3824437893 shutdown_connections 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99eb740 0x55b2d99e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 --2- 172.21.15.78:0/3824437893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99e3a70 0x55b2d99ecc90 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 -- 172.21.15.78:0/3824437893 >> 172.21.15.78:0/3824437893 conn(0x55b2d99b1ae0 msgr2=0x55b2d99e31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 -- 172.21.15.78:0/3824437893 shutdown_connections 2024-02-19T14:53:42.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.317+0000 7f26df207740 1 -- 172.21.15.78:0/3824437893 wait complete. 2024-02-19T14:53:42.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.318+0000 7f26df207740 1 -- start start 2024-02-19T14:53:42.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.318+0000 7f26df207740 1 Processor -- start 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99e3a70 0x55b2d99ed090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 0x55b2d9a5f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d9a5fbe0 con 0x55b2d99eb740 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2d99af930 con 0x55b2d99e3a70 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2d99afb80 con 0x55b2d99ee8b0 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26df207740 10 client.0 _ll_register_callbacks cb 0x7f26cc009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:53:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 0x55b2d9a5f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 0x55b2d9a5f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50988/0 (socket says 172.21.15.78:50988) 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 -- 172.21.15.78:0/2765733775 learned_addr learned my addr 172.21.15.78:0/2765733775 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99e3a70 0x55b2d99ed090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26daaf9640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 -- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 msgr2=0x55b2d99c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 -- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99e3a70 msgr2=0x55b2d99ed090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99e3a70 0x55b2d99ed090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26d9af7640 1 -- 172.21.15.78:0/2765733775 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d004a040 con 0x55b2d99eb740 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.319+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99e3a70 0x55b2d99ed090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26d9af7640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 0x55b2d9a5f160 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f26c4058790 tx=0x7f26c4058220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c93dcc42cb71fbd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26b57fa640 1 -- 172.21.15.78:0/2765733775 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c4060f10 con 0x55b2d99eb740 2024-02-19T14:53:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26b57fa640 1 -- 172.21.15.78:0/2765733775 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f26c40701d0 con 0x55b2d99eb740 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26cc017660 con 0x55b2d99eb740 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26b57fa640 1 -- 172.21.15.78:0/2765733775 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26c4070390 con 0x55b2d99eb740 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 10 client.5856 fetch_fsmap learned FSMap version 26 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55b2d98b2a20 con 0x55b2d99eb740 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26b57fa640 1 -- 172.21.15.78:0/2765733775 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc017660 con 0x55b2d99eb740 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26b57fa640 1 -- 172.21.15.78:0/2765733775 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f26c4012070 con 0x55b2d99eb740 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 10 client.5856 fetch_fsmap finished waiting for FSMap version 26 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 10 client.5856 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 20 client.5856 populate_metadata read hostname 'smithi078' 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99c2a20 0x55b2d9a58a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 4 client.5856 mds_command: new command op to 5703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 --> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b2d9654610 con 0x55b2d99c2a20 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.320+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99c2a20 0x55b2d9a58a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26b57fa640 10 client.5856 ms_handle_connect on v2:172.21.15.78:6835/1917648829 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26da2f8640 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99c2a20 0x55b2d9a58a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd63f54ee5702be3 server_cookie=2e3f08d69fbf043d in_seq=0 out_seq=0 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26b57fa640 1 -- 172.21.15.78:0/2765733775 <== mds.0 v2:172.21.15.78:6835/1917648829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55b2d9654610 con 0x55b2d99c2a20 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26b57fa640 10 client.5856 handle_command_reply: tid=0 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 client.5856 shutdown 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 msgr2=0x55b2d9a5f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 0x55b2d9a5f160 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f26c4058790 tx=0x7f26c4058220 comp rx=0 tx=0).stop 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 shutdown_connections 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:6835/1917648829,v1:172.21.15.78:6837/1917648829] conn(0x55b2d99c2a20 0x55b2d9a58a20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b2d99eb740 0x55b2d9a5f160 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b2d99ee8b0 0x55b2d99c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26b57fa640 0 client.5856 ms_handle_reset on v2:172.21.15.78:6835/1917648829 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 --2- 172.21.15.78:0/2765733775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b2d99e3a70 0x55b2d99ed090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 >> 172.21.15.78:0/2765733775 conn(0x55b2d99b1ae0 msgr2=0x55b2d99e2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 shutdown_connections 2024-02-19T14:53:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 1 -- 172.21.15.78:0/2765733775 wait complete. 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.321+0000 7f26df207740 20 client.5856 trim_cache size 0 max 16384 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5703, 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:53:42.330 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.6513477840000004, 2024-02-19T14:53:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 26, 2024-02-19T14:53:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 53, 2024-02-19T14:53:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 51, 2024-02-19T14:53:42.331 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 9.7575530189999995 2024-02-19T14:53:42.331 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26b806b840 msgr2=0x7f26b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26b806b840 0x7f26b806dd00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f26d0034000 tx=0x7f26d0009030 comp rx=0 tx=0).stop 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 msgr2=0x7f26d41edf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 0x7f26d41edf50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f26c400f650 tx=0x7f26c400fb10 comp rx=0 tx=0).stop 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 shutdown_connections 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26b806b840 0x7f26b806dd00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d40c0d40 0x7f26d41edf50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26d41ee490 0x7f26d4148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 --2- 172.21.15.78:0/2462138934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c0003970 0x7f26d41eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 >> 172.21.15.78:0/2462138934 conn(0x7f26d4066e70 msgr2=0x7f26d406b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 shutdown_connections 2024-02-19T14:53:42.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.322+0000 7f26dc55c640 1 -- 172.21.15.78:0/2462138934 wait complete. 2024-02-19T14:53:42.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:42.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.433+0000 7f07f9e1c640 1 Processor -- start 2024-02-19T14:53:42.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.433+0000 7f07f9e1c640 1 -- start start 2024-02-19T14:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f9e1c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07f40c5130 0x7f07f40c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f9e1c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f07f40c5510 0x7f07f41559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f9e1c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f40c3cf0 con 0x7f07f4158f80 2024-02-19T14:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f9e1c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f4158b60 con 0x7f07f40c5510 2024-02-19T14:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f9e1c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f4155f00 con 0x7f07f40c5130 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f07f40c5510 0x7f07f41559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f07f40c5510 0x7f07f41559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50998/0 (socket says 172.21.15.78:50998) 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 -- 172.21.15.78:0/3121542507 learned_addr learned my addr 172.21.15.78:0/3121542507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f37fe640 1 --2- 172.21.15.78:0/3121542507 >> v2:172.21.15.150:3300/0 conn(0x7f07f40c5130 0x7f07f40c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 -- 172.21.15.78:0/3121542507 >> v2:172.21.15.150:3300/0 conn(0x7f07f40c5130 msgr2=0x7f07f40c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 --2- 172.21.15.78:0/3121542507 >> v2:172.21.15.150:3300/0 conn(0x7f07f40c5130 0x7f07f40c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 -- 172.21.15.78:0/3121542507 >> v1:172.21.15.150:6789/0 conn(0x7f07f4158f80 legacy=0x7f07f40c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f3fff640 1 -- 172.21.15.78:0/3121542507 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f4159400 con 0x7f07f40c5510 2024-02-19T14:53:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.434+0000 7f07f37fe640 1 --2- 172.21.15.78:0/3121542507 >> v2:172.21.15.150:3300/0 conn(0x7f07f40c5130 0x7f07f40c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.435+0000 7f07f3fff640 1 --2- 172.21.15.78:0/3121542507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5510 0x7f07f41559c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f07f40c9130 tx=0x7f07ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed7abd90748bd22 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.435+0000 7f07f2ffd640 1 -- 172.21.15.78:0/3121542507 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec02ca80 con 0x7f07f40c5510 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f2ffd640 1 -- 172.21.15.78:0/3121542507 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ec02cbe0 con 0x7f07f40c5510 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f2ffd640 1 -- 172.21.15.78:0/3121542507 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0343f0 con 0x7f07f40c5510 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f9e1c640 1 -- 172.21.15.78:0/3121542507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5510 msgr2=0x7f07f41559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/3121542507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5510 0x7f07f41559c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f07f40c9130 tx=0x7f07ec02bee0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f9e1c640 1 -- 172.21.15.78:0/3121542507 shutdown_connections 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/3121542507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5510 0x7f07f41559c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/3121542507 >> v2:172.21.15.150:3300/0 conn(0x7f07f40c5130 0x7f07f40c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.436+0000 7f07f9e1c640 1 -- 172.21.15.78:0/3121542507 >> 172.21.15.78:0/3121542507 conn(0x7f07f406b4b0 msgr2=0x7f07f40c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.437+0000 7f07f9e1c640 1 -- 172.21.15.78:0/3121542507 shutdown_connections 2024-02-19T14:53:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.437+0000 7f07f9e1c640 1 -- 172.21.15.78:0/3121542507 wait complete. 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.438+0000 7f07f9e1c640 1 Processor -- start 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.438+0000 7f07f9e1c640 1 -- start start 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.438+0000 7f07f9e1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 0x7f07f414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.438+0000 7f07f9e1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f40c5510 0x7f07f4144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.438+0000 7f07f9e1c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f4158f80 0x7f07f4144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.438+0000 7f07f9e1c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f414ee10 con 0x7f07f40c5130 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f9e1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f414e200 con 0x7f07f40c5510 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f9e1c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f414e4e0 con 0x7f07f4158f80 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 0x7f07f414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 0x7f07f414c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51000/0 (socket says 172.21.15.78:51000) 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 -- 172.21.15.78:0/506840765 learned_addr learned my addr 172.21.15.78:0/506840765 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f3fff640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f4158f80 0x7f07f4144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07ebfff640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f40c5510 0x7f07f4144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 -- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f4158f80 msgr2=0x7f07f4144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f4158f80 0x7f07f4144e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 -- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f40c5510 msgr2=0x7f07f4144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f40c5510 0x7f07f4144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 -- 172.21.15.78:0/506840765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec009b30 con 0x7f07f40c5130 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f37fe640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 0x7f07f414c2d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f07e400c7f0 tx=0x7f07e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e4007a30 con 0x7f07f40c5130 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e4007b90 con 0x7f07f40c5130 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f414e7c0 con 0x7f07f40c5130 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e400ce40 con 0x7f07f40c5130 2024-02-19T14:53:42.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.439+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f406dce0 con 0x7f07f40c5130 2024-02-19T14:53:42.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.440+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f4074990 con 0x7f07f40c5130 2024-02-19T14:53:42.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.441+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f07e40113f0 con 0x7f07f40c5130 2024-02-19T14:53:42.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.441+0000 7f07f0ff9640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07d406b840 0x7f07d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.441+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f07e408a460 con 0x7f07f40c5130 2024-02-19T14:53:42.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.441+0000 7f07ebfff640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07d406b840 0x7f07d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.442+0000 7f07ebfff640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07d406b840 0x7f07d406dd00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f07dc006fd0 tx=0x7f07dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.446+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f07e4054d50 con 0x7f07f40c5130 2024-02-19T14:53:42.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.661+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f40c9400 con 0x7f07f40c5130 2024-02-19T14:53:42.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.661+0000 7f07f0ff9640 1 -- 172.21.15.78:0/506840765 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f07e4058410 con 0x7f07f40c5130 2024-02-19T14:53:42.664 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:42.664 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5398,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/210065453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":210065453},{"type":"v1","addr":"172.21.15.150:6834","nonce":210065453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5700,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:53:39.026813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5703":{"gid":5703,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1917648829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1917648829},{"type":"v1","addr":"172.21.15.78:6837","nonce":1917648829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-19T14:53:42.665 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 26 2024-02-19T14:53:42.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.663+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07d406b840 msgr2=0x7f07d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.663+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07d406b840 0x7f07d406dd00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f07dc006fd0 tx=0x7f07dc008040 comp rx=0 tx=0).stop 2024-02-19T14:53:42.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.663+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 msgr2=0x7f07f414c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.663+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 0x7f07f414c2d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f07e400c7f0 tx=0x7f07e400cbf0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 shutdown_connections 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07d406b840 0x7f07d406dd00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f4158f80 0x7f07f4144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f40c5510 0x7f07f4144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 --2- 172.21.15.78:0/506840765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f40c5130 0x7f07f414c2d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 >> 172.21.15.78:0/506840765 conn(0x7f07f406b4b0 msgr2=0x7f07f40c7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 shutdown_connections 2024-02-19T14:53:42.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.664+0000 7f07f9e1c640 1 -- 172.21.15.78:0/506840765 wait complete. 2024-02-19T14:53:42.677 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 3, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:53:37.026854+0000", "damaged": [], "data_pools": [ 8 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5703": { "addr": "172.21.15.78:6837/1917648829", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 1917648829, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 1917648829, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5703, "incarnation": 24, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-02-19T14:53:39.026813+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5703 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/210065453", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 210065453, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 210065453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 23, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5398, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/2194753931", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 2194753931, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2194753931, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 23, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5401, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/3678473057", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3678473057, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3678473057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 23, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5700, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:53:42.678 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:53:42.678 INFO:tasks.cephfs.test_failover:current = [{'gid': 5703, 'name': 'a', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/1917648829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 1917648829}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1917648829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5401, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2194753931', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2194753931}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2194753931}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 23}, {'gid': 5700, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/3678473057', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3678473057}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3678473057}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 23}, {'gid': 5398, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/210065453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 210065453}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 210065453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 23}] 2024-02-19T14:53:42.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-02-19T14:53:42.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.818+0000 7f78e190c640 1 Processor -- start 2024-02-19T14:53:42.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.818+0000 7f78e190c640 1 -- start start 2024-02-19T14:53:42.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78e190c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 0x7f78dc0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78e190c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc1551f0 con 0x7f78dc0c5730 2024-02-19T14:53:42.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78e190c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc155490 con 0x7f78dc0c8e00 2024-02-19T14:53:42.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78e190c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc155770 con 0x7f78dc152700 2024-02-19T14:53:42.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78d3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f78dc0c5730 0x7f78dc06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42132/0 (socket says 172.21.15.78:42132) 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78d3fff640 1 -- 172.21.15.78:0/1586263549 learned_addr learned my addr 172.21.15.78:0/1586263549 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78daffd640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 0x7f78dc0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563869619 0 0) 0x7f78dc1551f0 con 0x7f78dc0c5730 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c8003520 con 0x7f78dc0c5730 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785826713 0 0) 0x7f78c8003520 con 0x7f78dc0c5730 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 msgr2=0x7f78dc0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 0x7f78dc0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 >> v1:172.21.15.150:6789/0 conn(0x7f78dc0c8e00 legacy=0x7f78dc15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc155a50 con 0x7f78dc0c5730 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f78c4001670 con 0x7f78dc0c5730 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 >> v1:172.21.15.78:6789/0 conn(0x7f78dc0c5730 legacy=0x7f78dc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c80039a0 0x7f78c8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78c8004350 0x7f78c8004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 0x7f78c8005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc155770 con 0x7f78c8004350 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c800b980 con 0x7f78c80039a0 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.819+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c800bb40 con 0x7f78dc152700 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78d3fff640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78c8004350 0x7f78c8004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78daffd640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c80039a0 0x7f78c8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78db7fe640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 0x7f78c8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78d3fff640 1 -- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 msgr2=0x7f78c8005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78d3fff640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78dc152700 0x7f78c8005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78d3fff640 1 -- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c80039a0 msgr2=0x7f78c8003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78d3fff640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c80039a0 0x7f78c8003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.820+0000 7f78d3fff640 1 -- 172.21.15.78:0/1586263549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8003520 con 0x7f78c8004350 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78d3fff640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78c8004350 0x7f78c8004820 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f78c400c240 tx=0x7f78c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44285942d958213b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4005ad0 con 0x7f78c8004350 2024-02-19T14:53:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4005c30 con 0x7f78c8004350 2024-02-19T14:53:42.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78da7fc640 1 -- 172.21.15.78:0/1586263549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c402c8a0 con 0x7f78c8004350 2024-02-19T14:53:42.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78e190c640 1 -- 172.21.15.78:0/1586263549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78c8004350 msgr2=0x7f78c8004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78e190c640 1 --2- 172.21.15.78:0/1586263549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78c8004350 0x7f78c8004820 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f78c400c240 tx=0x7f78c80057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78db7fe640 1 -- 172.21.15.78:0/1586263549 reap_dead start 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78e190c640 1 -- 172.21.15.78:0/1586263549 shutdown_connections 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78e190c640 1 -- 172.21.15.78:0/1586263549 >> 172.21.15.78:0/1586263549 conn(0x7f78dc0cac90 msgr2=0x7f78dc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78e190c640 1 -- 172.21.15.78:0/1586263549 shutdown_connections 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.821+0000 7f78e190c640 1 -- 172.21.15.78:0/1586263549 wait complete. 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.823+0000 7f78e190c640 1 Processor -- start 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.823+0000 7f78e190c640 1 -- start start 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78e190c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78c8004350 0x7f78dc148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78e190c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78dc0bdc50 0x7f78dc1491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78e190c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 0x7f78dc158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78e190c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc0acc10 con 0x7f78dc158750 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78e190c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc1eaff0 con 0x7f78dc0bdc50 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78e190c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc156470 con 0x7f78c8004350 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 0x7f78dc158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 0x7f78dc158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51014/0 (socket says 172.21.15.78:51014) 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 -- 172.21.15.78:0/3844540990 learned_addr learned my addr 172.21.15.78:0/3844540990 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78daffd640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78c8004350 0x7f78dc148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78d3fff640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78dc0bdc50 0x7f78dc1491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 -- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78c8004350 msgr2=0x7f78dc148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78c8004350 0x7f78dc148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 -- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78dc0bdc50 msgr2=0x7f78dc1491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78dc0bdc50 0x7f78dc1491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 -- 172.21.15.78:0/3844540990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc156750 con 0x7f78dc158750 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78d3fff640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78dc0bdc50 0x7f78dc1491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78db7fe640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 0x7f78dc158bd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0aa260 tx=0x7f78d400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.824+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4003b80 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.825+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4003ce0 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.825+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4005310 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.825+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78dc156a30 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.825+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78dc159110 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.826+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc1562d0 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.826+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f78d4005470 con 0x7f78dc158750 2024-02-19T14:53:42.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.826+0000 7f78d8ff9640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78bc06b810 0x7f78bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.827+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f78d408a490 con 0x7f78dc158750 2024-02-19T14:53:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.827+0000 7f78daffd640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78bc06b810 0x7f78bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:42.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.827+0000 7f78daffd640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78bc06b810 0x7f78bc06dcd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009a80 tx=0x7f78cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:42.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:42.831+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78dc1562d0 con 0x7f78dc158750 2024-02-19T14:53:42.946 INFO:tasks.ceph.mds.a.smithi078.stderr:2024-02-19T14:53:42.944+0000 7f8f5e0a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:43.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.020+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f78dc1562d0 con 0x7f78dc158750 2024-02-19T14:53:43.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.023+0000 7f78d8ff9640 1 -- 172.21.15.78:0/3844540990 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v16)=0 v16) v1 ==== 117+0+0 (secure 0 0 0) 0x7f78dc1562d0 con 0x7f78dc158750 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78bc06b810 msgr2=0x7f78bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78bc06b810 0x7f78bc06dcd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009a80 tx=0x7f78cc008040 comp rx=0 tx=0).stop 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 msgr2=0x7f78dc158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 0x7f78dc158bd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0aa260 tx=0x7f78d400ff70 comp rx=0 tx=0).stop 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 shutdown_connections 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78bc06b810 0x7f78bc06dcd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78dc158750 0x7f78dc158bd0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78dc0bdc50 0x7f78dc1491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 --2- 172.21.15.78:0/3844540990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78c8004350 0x7f78dc148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 >> 172.21.15.78:0/3844540990 conn(0x7f78dc0cac90 msgr2=0x7f78dc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 shutdown_connections 2024-02-19T14:53:43.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.025+0000 7f78e190c640 1 -- 172.21.15.78:0/3844540990 wait complete. 2024-02-19T14:53:43.039 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-02-19T14:53:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.142+0000 7efc6efbc640 1 Processor -- start 2024-02-19T14:53:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.142+0000 7efc6efbc640 1 -- start start 2024-02-19T14:53:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6efbc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc680bdf20 0x7efc680ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6efbc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc680c55f0 0x7efc6815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6efbc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc68158c40 con 0x7efc680ac390 2024-02-19T14:53:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6efbc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc680c8de0 con 0x7efc680bdf20 2024-02-19T14:53:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6efbc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc680c5a30 con 0x7efc680c55f0 2024-02-19T14:53:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc67fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc680bdf20 0x7efc680ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6cd58640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efc680ac390 0x7efc68158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42140/0 (socket says 172.21.15.78:42140) 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc67fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efc680bdf20 0x7efc680ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51026/0 (socket says 172.21.15.78:51026) 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6cd58640 1 -- 172.21.15.78:0/3429152807 learned_addr learned my addr 172.21.15.78:0/3429152807 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc67fff640 1 -- 172.21.15.78:0/3429152807 >> v2:172.21.15.150:3300/0 conn(0x7efc680c55f0 msgr2=0x7efc6815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc6d559640 1 --2- 172.21.15.78:0/3429152807 >> v2:172.21.15.150:3300/0 conn(0x7efc680c55f0 0x7efc6815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc677fe640 1 -- 172.21.15.78:0/3429152807 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884864511 0 0) 0x7efc68158c40 con 0x7efc680ac390 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc67fff640 1 --2- 172.21.15.78:0/3429152807 >> v2:172.21.15.150:3300/0 conn(0x7efc680c55f0 0x7efc6815f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc67fff640 1 -- 172.21.15.78:0/3429152807 >> v1:172.21.15.78:6789/0 conn(0x7efc680ac390 legacy=0x7efc68158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.143+0000 7efc67fff640 1 -- 172.21.15.78:0/3429152807 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc68155a60 con 0x7efc680bdf20 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.144+0000 7efc67fff640 1 --2- 172.21.15.78:0/3429152807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680bdf20 0x7efc680ac770 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efc6806d9a0 tx=0x7efc5402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5390c21a3b2d077a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.144+0000 7efc677fe640 1 -- 172.21.15.78:0/3429152807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54031710 con 0x7efc680bdf20 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.144+0000 7efc677fe640 1 -- 172.21.15.78:0/3429152807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5402f050 con 0x7efc680bdf20 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.144+0000 7efc677fe640 1 -- 172.21.15.78:0/3429152807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54031a90 con 0x7efc680bdf20 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 -- 172.21.15.78:0/3429152807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680bdf20 msgr2=0x7efc680ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 --2- 172.21.15.78:0/3429152807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680bdf20 0x7efc680ac770 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efc6806d9a0 tx=0x7efc5402b7b0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 -- 172.21.15.78:0/3429152807 shutdown_connections 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 --2- 172.21.15.78:0/3429152807 >> v2:172.21.15.150:3300/0 conn(0x7efc680c55f0 0x7efc6815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 --2- 172.21.15.78:0/3429152807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680bdf20 0x7efc680ac770 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 -- 172.21.15.78:0/3429152807 >> 172.21.15.78:0/3429152807 conn(0x7efc680ca890 msgr2=0x7efc680caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 -- 172.21.15.78:0/3429152807 shutdown_connections 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.145+0000 7efc6efbc640 1 -- 172.21.15.78:0/3429152807 wait complete. 2024-02-19T14:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.146+0000 7efc6efbc640 1 Processor -- start 2024-02-19T14:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.146+0000 7efc6efbc640 1 -- start start 2024-02-19T14:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 0x7efc680ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc680bdf20 0x7efc681518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc680c55f0 0x7efc68151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc681ee710 con 0x7efc680ac390 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc681eec40 con 0x7efc680c55f0 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc681ee960 con 0x7efc680bdf20 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 0x7efc680ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 0x7efc680ae840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51034/0 (socket says 172.21.15.78:51034) 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 -- 172.21.15.78:0/1217228756 learned_addr learned my addr 172.21.15.78:0/1217228756 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 -- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc680bdf20 msgr2=0x7efc681518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6d559640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc680c55f0 0x7efc68151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc680bdf20 0x7efc681518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 -- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc680c55f0 msgr2=0x7efc68151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc680c55f0 0x7efc68151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 -- 172.21.15.78:0/1217228756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54009400 con 0x7efc680ac390 2024-02-19T14:53:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6cd58640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 0x7efc680ae840 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00c500 tx=0x7efc5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c007560 con 0x7efc680ac390 2024-02-19T14:53:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c0047e0 con 0x7efc680ac390 2024-02-19T14:53:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc680c3760 con 0x7efc680ac390 2024-02-19T14:53:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c004b90 con 0x7efc680ac390 2024-02-19T14:53:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.148+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc680c2a10 con 0x7efc680ac390 2024-02-19T14:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.149+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc68074990 con 0x7efc680ac390 2024-02-19T14:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.150+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efc5c004cf0 con 0x7efc680ac390 2024-02-19T14:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.151+0000 7efc657fa640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4c06b810 0x7efc4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.151+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efc5c089800 con 0x7efc680ac390 2024-02-19T14:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.151+0000 7efc67fff640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4c06b810 0x7efc4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.151+0000 7efc67fff640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4c06b810 0x7efc4c06dcd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efc54035000 tx=0x7efc54030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.155+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efc5c0540f0 con 0x7efc680ac390 2024-02-19T14:53:43.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.328+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7efc680c49c0 con 0x7efc680ac390 2024-02-19T14:53:43.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.331+0000 7efc657fa640 1 -- 172.21.15.78:0/1217228756 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v17)=0 v17) v1 ==== 117+0+0 (secure 0 0 0) 0x7efc5c0577b0 con 0x7efc680ac390 2024-02-19T14:53:43.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4c06b810 msgr2=0x7efc4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4c06b810 0x7efc4c06dcd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efc54035000 tx=0x7efc54030000 comp rx=0 tx=0).stop 2024-02-19T14:53:43.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 msgr2=0x7efc680ae840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 0x7efc680ae840 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00c500 tx=0x7efc5c00c9c0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 shutdown_connections 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc4c06b810 0x7efc4c06dcd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.333+0000 7efc6efbc640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc680c55f0 0x7efc68151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.334+0000 7efc6efbc640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc680bdf20 0x7efc681518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.334+0000 7efc6efbc640 1 --2- 172.21.15.78:0/1217228756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc680ac390 0x7efc680ae840 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.334+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 >> 172.21.15.78:0/1217228756 conn(0x7efc680ca890 msgr2=0x7efc68153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.334+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 shutdown_connections 2024-02-19T14:53:43.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.334+0000 7efc6efbc640 1 -- 172.21.15.78:0/1217228756 wait complete. 2024-02-19T14:53:43.347 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-02-19T14:53:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.447+0000 7fccdad01640 1 Processor -- start 2024-02-19T14:53:43.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.447+0000 7fccdad01640 1 -- start start 2024-02-19T14:53:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.448+0000 7fccdad01640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccd40c6ff0 0x7fccd4070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.448+0000 7fccdad01640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd40c65a0 con 0x7fccd40c7910 2024-02-19T14:53:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.448+0000 7fccdad01640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd41537b0 con 0x7fccd40c5560 2024-02-19T14:53:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.448+0000 7fccdad01640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd4153a90 con 0x7fccd40c6ff0 2024-02-19T14:53:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.448+0000 7fccd3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fccd40c7910 0x7fccd4070720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42156/0 (socket says 172.21.15.78:42156) 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.448+0000 7fccd3fff640 1 -- 172.21.15.78:0/1716913613 learned_addr learned my addr 172.21.15.78:0/1716913613 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353815724 0 0) 0x7fccd40c65a0 con 0x7fccd40c7910 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc4003520 con 0x7fccd40c7910 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 --2- 172.21.15.78:0/1716913613 >> v2:172.21.15.150:3300/0 conn(0x7fccd40c6ff0 0x7fccd4070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665193650 0 0) 0x7fccc4003520 con 0x7fccd40c7910 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 >> v2:172.21.15.150:3300/0 conn(0x7fccd40c6ff0 msgr2=0x7fccd4070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 --2- 172.21.15.78:0/1716913613 >> v2:172.21.15.150:3300/0 conn(0x7fccd40c6ff0 0x7fccd4070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 >> v1:172.21.15.150:6789/0 conn(0x7fccd40c5560 legacy=0x7fccd4154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40c67f0 con 0x7fccd40c7910 2024-02-19T14:53:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fccc0001670 con 0x7fccd40c7910 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 >> v1:172.21.15.78:6789/0 conn(0x7fccd40c7910 legacy=0x7fccd4070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc40052a0 0x7fccc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd4153a90 con 0x7fccc40052a0 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc400c170 con 0x7fccc4003970 2024-02-19T14:53:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc400c330 con 0x7fccc40047c0 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc40052a0 0x7fccc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd8a9d640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd3fff640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 -- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 msgr2=0x7fccc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 -- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 msgr2=0x7fccc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd929e640 1 -- 172.21.15.78:0/1716913613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4003520 con 0x7fccc40052a0 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.449+0000 7fccd3fff640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.450+0000 7fccd929e640 1 -- 172.21.15.78:0/1716913613 reap_dead start 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.450+0000 7fccd929e640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc40052a0 0x7fccc400b980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009f90 tx=0x7fccc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06a149ec73b1bb4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.450+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc02cc20 con 0x7fccc40052a0 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.450+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc00f040 con 0x7fccc40052a0 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.450+0000 7fccd37fe640 1 -- 172.21.15.78:0/1716913613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc003990 con 0x7fccc40052a0 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 -- 172.21.15.78:0/1716913613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc40052a0 msgr2=0x7fccc400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc40052a0 0x7fccc400b980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009f90 tx=0x7fccc40057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 -- 172.21.15.78:0/1716913613 shutdown_connections 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 --2- 172.21.15.78:0/1716913613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc40052a0 0x7fccc400b980 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 -- 172.21.15.78:0/1716913613 >> 172.21.15.78:0/1716913613 conn(0x7fccd406b4b0 msgr2=0x7fccd40c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 -- 172.21.15.78:0/1716913613 shutdown_connections 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.451+0000 7fccdad01640 1 -- 172.21.15.78:0/1716913613 wait complete. 2024-02-19T14:53:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.452+0000 7fccdad01640 1 Processor -- start 2024-02-19T14:53:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.452+0000 7fccdad01640 1 -- start start 2024-02-19T14:53:43.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccdad01640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40052a0 0x7fccd40bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccdad01640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 0x7fccd40a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccdad01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccd41ebea0 0x7fccd41ec320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccdad01640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd40a7f60 con 0x7fccd40bd6c0 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccdad01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd40a8240 con 0x7fccd41ebea0 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccdad01640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd40a8520 con 0x7fccc40052a0 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 0x7fccd40a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 0x7fccd40a9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51050/0 (socket says 172.21.15.78:51050) 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 -- 172.21.15.78:0/2573268788 learned_addr learned my addr 172.21.15.78:0/2573268788 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd8a9d640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40052a0 0x7fccd40bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd929e640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccd41ebea0 0x7fccd41ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 -- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40052a0 msgr2=0x7fccd40bfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40052a0 0x7fccd40bfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 -- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccd41ebea0 msgr2=0x7fccd41ec320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccd41ebea0 0x7fccd41ec320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 -- 172.21.15.78:0/2573268788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40c5ac0 con 0x7fccd40bd6c0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.453+0000 7fccd3fff640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 0x7fccd40a9000 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000ff0 tx=0x7fccc000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.454+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc000c660 con 0x7fccd40bd6c0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.454+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc000c7c0 con 0x7fccd40bd6c0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.454+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd40a8800 con 0x7fccd40bd6c0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.454+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc000cb50 con 0x7fccd40bd6c0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.454+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd40a9540 con 0x7fccd40bd6c0 2024-02-19T14:53:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.455+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd40a8d20 con 0x7fccd40bd6c0 2024-02-19T14:53:43.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.455+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fccc001d460 con 0x7fccd40bd6c0 2024-02-19T14:53:43.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.456+0000 7fccd17fa640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccb806b810 0x7fccb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.456+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fccc008a2a0 con 0x7fccd40bd6c0 2024-02-19T14:53:43.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.456+0000 7fccd8a9d640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccb806b810 0x7fccb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.456+0000 7fccd8a9d640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccb806b810 0x7fccb806dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fccc8001760 tx=0x7fccc8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.460+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fccd40a8d20 con 0x7fccd40bd6c0 2024-02-19T14:53:43.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.637+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fccd40a8d20 con 0x7fccd40bd6c0 2024-02-19T14:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.640+0000 7fccd17fa640 1 -- 172.21.15.78:0/2573268788 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v18)=0 v18) v1 ==== 117+0+0 (secure 0 0 0) 0x7fccd40a8d20 con 0x7fccd40bd6c0 2024-02-19T14:53:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccb806b810 msgr2=0x7fccb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccb806b810 0x7fccb806dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fccc8001760 tx=0x7fccc8009040 comp rx=0 tx=0).stop 2024-02-19T14:53:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 msgr2=0x7fccd40a9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 0x7fccd40a9000 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fccc0000ff0 tx=0x7fccc000eb90 comp rx=0 tx=0).stop 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 shutdown_connections 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccb806b810 0x7fccb806dcd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd40bd6c0 0x7fccd40a9000 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccd41ebea0 0x7fccd41ec320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 --2- 172.21.15.78:0/2573268788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40052a0 0x7fccd40bfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.642+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 >> 172.21.15.78:0/2573268788 conn(0x7fccd406b4b0 msgr2=0x7fccd4156aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.643+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 shutdown_connections 2024-02-19T14:53:43.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.643+0000 7fccdad01640 1 -- 172.21.15.78:0/2573268788 wait complete. 2024-02-19T14:53:43.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:43.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.757+0000 7f16d644e640 1 Processor -- start 2024-02-19T14:53:43.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.758+0000 7f16d644e640 1 -- start start 2024-02-19T14:53:43.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16d644e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16d0072d70 0x7f16d0152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16d644e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d00c5970 con 0x7f16d006ea50 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16d644e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d00c5750 con 0x7f16d014f1a0 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16d644e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d00702d0 con 0x7f16d0072d70 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16cf7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f16d006ea50 0x7f16d00c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42170/0 (socket says 172.21.15.78:42170) 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16cf7fe640 1 -- 172.21.15.78:0/4085135717 learned_addr learned my addr 172.21.15.78:0/4085135717 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060101378 0 0) 0x7f16d00c5970 con 0x7f16d006ea50 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b8003520 con 0x7f16d006ea50 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16d49eb640 1 --2- 172.21.15.78:0/4085135717 >> v2:172.21.15.150:3300/0 conn(0x7f16d0072d70 0x7f16d0152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681430231 0 0) 0x7f16b8003520 con 0x7f16d006ea50 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 >> v2:172.21.15.150:3300/0 conn(0x7f16d0072d70 msgr2=0x7f16d0152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 --2- 172.21.15.78:0/4085135717 >> v2:172.21.15.150:3300/0 conn(0x7f16d0072d70 0x7f16d0152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 >> v1:172.21.15.150:6789/0 conn(0x7f16d014f1a0 legacy=0x7f16d006fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d00c5bc0 con 0x7f16d006ea50 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.759+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f16c4001670 con 0x7f16d006ea50 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 >> v1:172.21.15.78:6789/0 conn(0x7f16d006ea50 legacy=0x7f16d00c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b8003970 0x7f16b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16b80052a0 0x7f16b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d00702d0 con 0x7f16b80047c0 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b800c170 con 0x7f16b8003970 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b800c330 con 0x7f16b80052a0 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cf7fe640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cffff640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b8003970 0x7f16b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16d49eb640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16b80052a0 0x7f16b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cf7fe640 1 -- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16b80052a0 msgr2=0x7f16b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cf7fe640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16b80052a0 0x7f16b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cf7fe640 1 -- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b8003970 msgr2=0x7f16b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cf7fe640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b8003970 0x7f16b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16cf7fe640 1 -- 172.21.15.78:0/4085135717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8003520 con 0x7f16b80047c0 2024-02-19T14:53:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.760+0000 7f16d49eb640 1 -- 172.21.15.78:0/4085135717 reap_dead start 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16cf7fe640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f16c4007fd0 tx=0x7f16b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb89e7f847e408 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4005ad0 con 0x7f16b80047c0 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c4005c30 con 0x7f16b80047c0 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16ceffd640 1 -- 172.21.15.78:0/4085135717 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c402ca10 con 0x7f16b80047c0 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 -- 172.21.15.78:0/4085135717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b80047c0 msgr2=0x7f16b8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f16c4007fd0 tx=0x7f16b80057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 -- 172.21.15.78:0/4085135717 shutdown_connections 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 --2- 172.21.15.78:0/4085135717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 -- 172.21.15.78:0/4085135717 >> 172.21.15.78:0/4085135717 conn(0x7f16d006c170 msgr2=0x7f16d006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 -- 172.21.15.78:0/4085135717 shutdown_connections 2024-02-19T14:53:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.761+0000 7f16d644e640 1 -- 172.21.15.78:0/4085135717 wait complete. 2024-02-19T14:53:43.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.763+0000 7f16d644e640 1 Processor -- start 2024-02-19T14:53:43.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.763+0000 7f16d644e640 1 -- start start 2024-02-19T14:53:43.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.764+0000 7f16d644e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b80047c0 0x7f16d014ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.764+0000 7f16d644e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16d0154030 0x7f16d00c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.764+0000 7f16d644e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 0x7f16d01ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.764+0000 7f16d644e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d00bc8f0 con 0x7f16d00c4a80 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.764+0000 7f16d644e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d0153330 con 0x7f16b80047c0 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.764+0000 7f16d644e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01eb630 con 0x7f16d0154030 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 0x7f16d01ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 0x7f16d01ea6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51064/0 (socket says 172.21.15.78:51064) 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 -- 172.21.15.78:0/2758840811 learned_addr learned my addr 172.21.15.78:0/2758840811 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16cffff640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b80047c0 0x7f16d014ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16cf7fe640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16d0154030 0x7f16d00c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 -- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16d0154030 msgr2=0x7f16d00c4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16d0154030 0x7f16d00c4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 -- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b80047c0 msgr2=0x7f16d014ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b80047c0 0x7f16d014ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 -- 172.21.15.78:0/2758840811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d014b0c0 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d49eb640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 0x7f16d01ea6f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f16d0071f60 tx=0x7f16c80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c800e070 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c8011e40 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d01eb350 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8003990 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.765+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d01537f0 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.766+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d0074990 con 0x7f16d00c4a80 2024-02-19T14:53:43.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.767+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f16c8003af0 con 0x7f16d00c4a80 2024-02-19T14:53:43.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.768+0000 7f16ccff9640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16bc06b840 0x7f16bc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:43.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.768+0000 7f16cffff640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16bc06b840 0x7f16bc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:43.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.768+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f16c808a960 con 0x7f16d00c4a80 2024-02-19T14:53:43.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.768+0000 7f16cffff640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16bc06b840 0x7f16bc06dd00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f16d00ac640 tx=0x7f16c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:43.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.772+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f16c8055250 con 0x7f16d00c4a80 2024-02-19T14:53:43.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.945+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d006e830 con 0x7f16d00c4a80 2024-02-19T14:53:43.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.946+0000 7f16ccff9640 1 -- 172.21.15.78:0/2758840811 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17026 (secure 0 0 0) 0x7f16c8058910 con 0x7f16d00c4a80 2024-02-19T14:53:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:43.949 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":53,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:41.695517+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:43.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16bc06b840 msgr2=0x7f16bc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16bc06b840 0x7f16bc06dd00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f16d00ac640 tx=0x7f16c0009040 comp rx=0 tx=0).stop 2024-02-19T14:53:43.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 msgr2=0x7f16d01ea6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 0x7f16d01ea6f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f16d0071f60 tx=0x7f16c80115a0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 shutdown_connections 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16bc06b840 0x7f16bc06dd00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16d00c4a80 0x7f16d01ea6f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16d0154030 0x7f16d00c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 --2- 172.21.15.78:0/2758840811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16b80047c0 0x7f16d014ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 >> 172.21.15.78:0/2758840811 conn(0x7f16d006c170 msgr2=0x7f16d006fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 shutdown_connections 2024-02-19T14:53:43.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:43.948+0000 7f16d644e640 1 -- 172.21.15.78:0/2758840811 wait complete. 2024-02-19T14:53:43.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:53:35.233332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:53:35.989674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:53:43.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:53:44.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.064+0000 7fcc8e81c640 1 Processor -- start 2024-02-19T14:53:44.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.064+0000 7fcc8e81c640 1 -- start start 2024-02-19T14:53:44.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8e81c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc880c2f60 0x7fcc88070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8e81c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc880ae670 con 0x7fcc880c4980 2024-02-19T14:53:44.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8e81c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc880716b0 con 0x7fcc880add70 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8e81c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc880c3390 con 0x7fcc880c2f60 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc877fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcc880c4980 0x7fcc880705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42174/0 (socket says 172.21.15.78:42174) 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc877fe640 1 -- 172.21.15.78:0/2651100436 learned_addr learned my addr 172.21.15.78:0/2651100436 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8cdb9640 1 --2- 172.21.15.78:0/2651100436 >> v2:172.21.15.150:3300/0 conn(0x7fcc880c2f60 0x7fcc88070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612255215 0 0) 0x7fcc880ae670 con 0x7fcc880c4980 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc70003520 con 0x7fcc880c4980 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777606647 0 0) 0x7fcc880716b0 con 0x7fcc880add70 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc880ae670 con 0x7fcc880add70 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306899702 0 0) 0x7fcc70003520 con 0x7fcc880c4980 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8cdb9640 1 -- 172.21.15.78:0/2651100436 >> v1:172.21.15.150:6789/0 conn(0x7fcc880add70 legacy=0x7fcc8806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8cdb9640 1 -- 172.21.15.78:0/2651100436 >> v1:172.21.15.78:6789/0 conn(0x7fcc880c4980 legacy=0x7fcc880705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.065+0000 7fcc8cdb9640 1 -- 172.21.15.78:0/2651100436 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88153780 con 0x7fcc880c2f60 2024-02-19T14:53:44.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.066+0000 7fcc8cdb9640 1 --2- 172.21.15.78:0/2651100436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880c2f60 0x7fcc88070cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc8002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d8f25726b8c2339 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.066+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c001670 con 0x7fcc880c2f60 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.066+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8002c8b0 con 0x7fcc880c2f60 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.066+0000 7fcc86ffd640 1 -- 172.21.15.78:0/2651100436 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8002cc40 con 0x7fcc880c2f60 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 -- 172.21.15.78:0/2651100436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880c2f60 msgr2=0x7fcc88070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/2651100436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880c2f60 0x7fcc88070cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009d90 tx=0x7fcc8002bd80 comp rx=0 tx=0).stop 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 -- 172.21.15.78:0/2651100436 shutdown_connections 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/2651100436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880c2f60 0x7fcc88070cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 -- 172.21.15.78:0/2651100436 >> 172.21.15.78:0/2651100436 conn(0x7fcc880c85e0 msgr2=0x7fcc880c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 -- 172.21.15.78:0/2651100436 shutdown_connections 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.067+0000 7fcc8e81c640 1 -- 172.21.15.78:0/2651100436 wait complete. 2024-02-19T14:53:44.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.068+0000 7fcc8e81c640 1 Processor -- start 2024-02-19T14:53:44.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.068+0000 7fcc8e81c640 1 -- start start 2024-02-19T14:53:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8e81c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880add70 0x7fcc880c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8e81c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 0x7fcc8806d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8e81c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc880c4980 0x7fcc881553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8e81c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880c2370 con 0x7fcc880c2f60 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8e81c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8806c9a0 con 0x7fcc880add70 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8e81c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8806cc80 con 0x7fcc880c4980 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 0x7fcc8806d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 0x7fcc8806d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51080/0 (socket says 172.21.15.78:51080) 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 -- 172.21.15.78:0/1677384959 learned_addr learned my addr 172.21.15.78:0/1677384959 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc87fff640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880add70 0x7fcc880c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8cdb9640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc880c4980 0x7fcc881553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 -- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc880c4980 msgr2=0x7fcc881553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc880c4980 0x7fcc881553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 -- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880add70 msgr2=0x7fcc880c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880add70 0x7fcc880c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 -- 172.21.15.78:0/1677384959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc80009a70 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc8cdb9640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc880c4980 0x7fcc881553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.069+0000 7fcc877fe640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 0x7fcc8806d940 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00c750 tx=0x7fcc7c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.070+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c005ad0 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.070+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7c005c30 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.070+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8806cf60 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.070+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c007070 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.070+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc880c5580 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.071+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc881550c0 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.071+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcc7c0071d0 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.072+0000 7fcc84ff9640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcc7406b840 0x7fcc7406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.072+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcc7c089ff0 con 0x7fcc880c2f60 2024-02-19T14:53:44.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.072+0000 7fcc87fff640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcc7406b840 0x7fcc7406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.072+0000 7fcc87fff640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcc7406b840 0x7fcc7406dd00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcc78001650 tx=0x7fcc78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.076+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcc881550c0 con 0x7fcc880c2f60 2024-02-19T14:53:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.247+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc881550c0 con 0x7fcc880c2f60 2024-02-19T14:53:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.248+0000 7fcc84ff9640 1 -- 172.21.15.78:0/1677384959 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc881550c0 con 0x7fcc880c2f60 2024-02-19T14:53:44.250 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcc7406b840 msgr2=0x7fcc7406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcc7406b840 0x7fcc7406dd00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcc78001650 tx=0x7fcc78009040 comp rx=0 tx=0).stop 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 msgr2=0x7fcc8806d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 0x7fcc8806d940 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00c750 tx=0x7fcc7c00cc10 comp rx=0 tx=0).stop 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 shutdown_connections 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcc7406b840 0x7fcc7406dd00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc880c2f60 0x7fcc8806d940 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc880c4980 0x7fcc881553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 --2- 172.21.15.78:0/1677384959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc880add70 0x7fcc880c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 >> 172.21.15.78:0/1677384959 conn(0x7fcc880c85e0 msgr2=0x7fcc880c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:44.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 shutdown_connections 2024-02-19T14:53:44.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.250+0000 7fcc8e81c640 1 -- 172.21.15.78:0/1677384959 wait complete. 2024-02-19T14:53:44.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:53:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.364+0000 7f85fce76640 1 Processor -- start 2024-02-19T14:53:44.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.364+0000 7f85fce76640 1 -- start start 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85fce76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85fce76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f85f80bdf20 0x7f85f80c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85fce76640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80c8de0 con 0x7f85f8158530 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85fce76640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f8155a60 con 0x7f85f80bdf20 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85fce76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8155d40 con 0x7f85f80ac390 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85f5d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f85f80bdf20 0x7f85f80c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85f5d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f85f80bdf20 0x7f85f80c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51090/0 (socket says 172.21.15.78:51090) 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85f5d9b640 1 -- 172.21.15.78:0/1087364269 learned_addr learned my addr 172.21.15.78:0/1087364269 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85f659c640 1 --2- 172.21.15.78:0/1087364269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85f5d9b640 1 -- 172.21.15.78:0/1087364269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 msgr2=0x7f85f80ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:44.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.365+0000 7f85f5d9b640 1 --2- 172.21.15.78:0/1087364269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.366+0000 7f85f5d9b640 1 -- 172.21.15.78:0/1087364269 >> v1:172.21.15.150:6789/0 conn(0x7f85f8158530 legacy=0x7f85f806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.366+0000 7f85f5d9b640 1 -- 172.21.15.78:0/1087364269 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80c90a0 con 0x7f85f80bdf20 2024-02-19T14:53:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.366+0000 7f85f5d9b640 1 --2- 172.21.15.78:0/1087364269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c55f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009870 tx=0x7f85e802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e777674f82654b0b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.366+0000 7f85f559a640 1 -- 172.21.15.78:0/1087364269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e80317c0 con 0x7f85f80bdf20 2024-02-19T14:53:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.366+0000 7f85f559a640 1 -- 172.21.15.78:0/1087364269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e8031920 con 0x7f85f80bdf20 2024-02-19T14:53:44.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.366+0000 7f85f559a640 1 -- 172.21.15.78:0/1087364269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8031cb0 con 0x7f85f80bdf20 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 -- 172.21.15.78:0/1087364269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 msgr2=0x7f85f80c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 --2- 172.21.15.78:0/1087364269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c55f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009870 tx=0x7f85e802b860 comp rx=0 tx=0).stop 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 -- 172.21.15.78:0/1087364269 shutdown_connections 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 --2- 172.21.15.78:0/1087364269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c55f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 --2- 172.21.15.78:0/1087364269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 -- 172.21.15.78:0/1087364269 >> 172.21.15.78:0/1087364269 conn(0x7f85f80ca890 msgr2=0x7f85f80caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:44.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 -- 172.21.15.78:0/1087364269 shutdown_connections 2024-02-19T14:53:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.367+0000 7f85fce76640 1 -- 172.21.15.78:0/1087364269 wait complete. 2024-02-19T14:53:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.369+0000 7f85fce76640 1 Processor -- start 2024-02-19T14:53:44.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.369+0000 7f85fce76640 1 -- start start 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85fce76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85fce76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85fce76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f8158530 0x7f85f8147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85fce76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f80c18d0 con 0x7f85f80bdf20 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85fce76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f80c2730 con 0x7f85f8158530 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85fce76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8154d70 con 0x7f85f80ac390 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51104/0 (socket says 172.21.15.78:51104) 2024-02-19T14:53:44.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 -- 172.21.15.78:0/2668543231 learned_addr learned my addr 172.21.15.78:0/2668543231 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f659c640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f6d9d640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f8158530 0x7f85f8147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 -- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 msgr2=0x7f85f80c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 -- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f8158530 msgr2=0x7f85f8147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f8158530 0x7f85f8147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 -- 172.21.15.78:0/2668543231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80094d0 con 0x7f85f80bdf20 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.370+0000 7f85f5d9b640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c2bc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f85e802b8a0 tx=0x7f85e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.371+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8032860 con 0x7f85f80bdf20 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.371+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e8032a00 con 0x7f85f80bdf20 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.371+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8032db0 con 0x7f85f80bdf20 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.371+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f80c1b20 con 0x7f85f80bdf20 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.371+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f80c20e0 con 0x7f85f80bdf20 2024-02-19T14:53:44.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.372+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f8074990 con 0x7f85f80bdf20 2024-02-19T14:53:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.372+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f85e804e450 con 0x7f85f80bdf20 2024-02-19T14:53:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.372+0000 7f85e6ffd640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f85d806b810 0x7f85d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.372+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f85e80ac5f0 con 0x7f85f80bdf20 2024-02-19T14:53:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.373+0000 7f85f659c640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f85d806b810 0x7f85d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.373+0000 7f85f659c640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f85d806b810 0x7f85d806dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f85ec006fd0 tx=0x7f85ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.377+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f85e8076e80 con 0x7f85f80bdf20 2024-02-19T14:53:44.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.690+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85f80ca480 con 0x7f85f80bdf20 2024-02-19T14:53:44.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.690+0000 7f85e6ffd640 1 -- 172.21.15.78:0/2668543231 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f85e807a540 con 0x7f85f80bdf20 2024-02-19T14:53:44.693 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f85d806b810 msgr2=0x7f85d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f85d806b810 0x7f85d806dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f85ec006fd0 tx=0x7f85ec008040 comp rx=0 tx=0).stop 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 msgr2=0x7f85f80c2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c2bc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f85e802b8a0 tx=0x7f85e80046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 shutdown_connections 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f85d806b810 0x7f85d806dcd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85f80bdf20 0x7f85f80c2bc0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f8158530 0x7f85f8147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 --2- 172.21.15.78:0/2668543231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85f80ac390 0x7f85f80c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 >> 172.21.15.78:0/2668543231 conn(0x7f85f80ca890 msgr2=0x7f85f80c9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:44.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 shutdown_connections 2024-02-19T14:53:44.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.692+0000 7f85fce76640 1 -- 172.21.15.78:0/2668543231 wait complete. 2024-02-19T14:53:44.706 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:53:44.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.808+0000 7f445835c640 1 Processor -- start 2024-02-19T14:53:44.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.808+0000 7f445835c640 1 -- start start 2024-02-19T14:53:44.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f445835c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f44500c8e00 0x7f445006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f445835c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44501551f0 con 0x7f445006d910 2024-02-19T14:53:44.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f445835c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4450155490 con 0x7f4450152700 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f445835c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4450155770 con 0x7f44500c8e00 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f44558f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f44500c8e00 0x7f445006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f44558f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f44500c8e00 0x7f445006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51108/0 (socket says 172.21.15.78:51108) 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f44568f9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f445006d910 0x7f445015f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42180/0 (socket says 172.21.15.78:42180) 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.809+0000 7f44558f7640 1 -- 172.21.15.78:0/1146250699 learned_addr learned my addr 172.21.15.78:0/1146250699 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.810+0000 7f44558f7640 1 -- 172.21.15.78:0/1146250699 >> v1:172.21.15.150:6789/0 conn(0x7f4450152700 legacy=0x7f44500c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.810+0000 7f44558f7640 1 -- 172.21.15.78:0/1146250699 >> v1:172.21.15.78:6789/0 conn(0x7f445006d910 legacy=0x7f445015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.810+0000 7f44550f6640 1 -- 172.21.15.78:0/1146250699 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929019264 0 0) 0x7f44501551f0 con 0x7f445006d910 2024-02-19T14:53:44.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.810+0000 7f44558f7640 1 -- 172.21.15.78:0/1146250699 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450155a50 con 0x7f44500c8e00 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.810+0000 7f44558f7640 1 --2- 172.21.15.78:0/1146250699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44500c8e00 0x7f445006c2a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f444c009870 tx=0x7f444c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a603679a3b0277fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.810+0000 7f44550f6640 1 -- 172.21.15.78:0/1146250699 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c030860 con 0x7f44500c8e00 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f44550f6640 1 -- 172.21.15.78:0/1146250699 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c0309c0 con 0x7f44500c8e00 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f44550f6640 1 -- 172.21.15.78:0/1146250699 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c030d50 con 0x7f44500c8e00 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 -- 172.21.15.78:0/1146250699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44500c8e00 msgr2=0x7f445006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 --2- 172.21.15.78:0/1146250699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44500c8e00 0x7f445006c2a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f444c009870 tx=0x7f444c02b9a0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 -- 172.21.15.78:0/1146250699 shutdown_connections 2024-02-19T14:53:44.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 --2- 172.21.15.78:0/1146250699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44500c8e00 0x7f445006c2a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 -- 172.21.15.78:0/1146250699 >> 172.21.15.78:0/1146250699 conn(0x7f44500cac90 msgr2=0x7f44500cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:44.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 -- 172.21.15.78:0/1146250699 shutdown_connections 2024-02-19T14:53:44.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.811+0000 7f445835c640 1 -- 172.21.15.78:0/1146250699 wait complete. 2024-02-19T14:53:44.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.812+0000 7f445835c640 1 Processor -- start 2024-02-19T14:53:44.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.812+0000 7f445835c640 1 -- start start 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f445835c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445006d910 0x7f44500aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f445835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44500c8e00 0x7f4450154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f445835c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 0x7f44500aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f445835c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44501eacf0 con 0x7f4450152700 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f445835c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44500aa280 con 0x7f445006d910 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f445835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44500aa800 con 0x7f44500c8e00 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 0x7f44500aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 0x7f44500aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51118/0 (socket says 172.21.15.78:51118) 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 -- 172.21.15.78:0/1621845543 learned_addr learned my addr 172.21.15.78:0/1621845543 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44558f7640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44500c8e00 0x7f4450154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44560f8640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445006d910 0x7f44500aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 -- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44500c8e00 msgr2=0x7f4450154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44500c8e00 0x7f4450154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 -- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445006d910 msgr2=0x7f44500aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445006d910 0x7f44500aca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 -- 172.21.15.78:0/1621845543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c0094d0 con 0x7f4450152700 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.813+0000 7f44568f9640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 0x7f44500aac90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f444800c7d0 tx=0x7f444800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.814+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444800a640 con 0x7f4450152700 2024-02-19T14:53:44.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.814+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444800a7a0 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.814+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445014e7e0 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.814+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444800ab90 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.814+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44500aa490 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.815+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44500ab990 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.815+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f444800acf0 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.816+0000 7f4446ffd640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f443406b810 0x7f443406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.816+0000 7f44560f8640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f443406b810 0x7f443406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.816+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4448089b60 con 0x7f4450152700 2024-02-19T14:53:44.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.816+0000 7f44560f8640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f443406b810 0x7f443406dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4440006fd0 tx=0x7f4440008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:44.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:44.820+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4448054450 con 0x7f4450152700 2024-02-19T14:53:45.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.103+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4450074990 con 0x7f4450152700 2024-02-19T14:53:45.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.104+0000 7f4446ffd640 1 -- 172.21.15.78:0/1621845543 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4448018020 con 0x7f4450152700 2024-02-19T14:53:45.106 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:53:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f443406b810 msgr2=0x7f443406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f443406b810 0x7f443406dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4440006fd0 tx=0x7f4440008040 comp rx=0 tx=0).stop 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 msgr2=0x7f44500aac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 0x7f44500aac90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f444800c7d0 tx=0x7f444800cc90 comp rx=0 tx=0).stop 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 shutdown_connections 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f443406b810 0x7f443406dcd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4450152700 0x7f44500aac90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44500c8e00 0x7f4450154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 --2- 172.21.15.78:0/1621845543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445006d910 0x7f44500aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 >> 172.21.15.78:0/1621845543 conn(0x7f44500cac90 msgr2=0x7f445014a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 shutdown_connections 2024-02-19T14:53:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.106+0000 7f445835c640 1 -- 172.21.15.78:0/1621845543 wait complete. 2024-02-19T14:53:45.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:53:45.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.217+0000 7fe56629a640 1 Processor -- start 2024-02-19T14:53:45.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.217+0000 7fe56629a640 1 -- start start 2024-02-19T14:53:45.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe56629a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600add70 0x7fe5600c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe56629a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe5600705b0 0x7fe5600709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe56629a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5600ae670 con 0x7fe5600c2f60 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe56629a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5600716b0 con 0x7fe5600705b0 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe56629a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600c33a0 con 0x7fe5600add70 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe564837640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe5600705b0 0x7fe5600709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe55f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe5600c2f60 0x7fe56006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42182/0 (socket says 172.21.15.78:42182) 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe55f7fe640 1 -- 172.21.15.78:0/318679545 learned_addr learned my addr 172.21.15.78:0/318679545 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe564837640 1 -- 172.21.15.78:0/318679545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600add70 msgr2=0x7fe5600c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe564837640 1 --2- 172.21.15.78:0/318679545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600add70 0x7fe5600c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe564837640 1 -- 172.21.15.78:0/318679545 >> v1:172.21.15.78:6789/0 conn(0x7fe5600c2f60 legacy=0x7fe56006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe564837640 1 -- 172.21.15.78:0/318679545 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560153780 con 0x7fe5600705b0 2024-02-19T14:53:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.219+0000 7fe55effd640 1 -- 172.21.15.78:0/318679545 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501792444 0 0) 0x7fe5600ae670 con 0x7fe5600c2f60 2024-02-19T14:53:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.220+0000 7fe564837640 1 --2- 172.21.15.78:0/318679545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600705b0 0x7fe5600709c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe558009d90 tx=0x7fe55802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69b2818ee61ced9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.220+0000 7fe55effd640 1 -- 172.21.15.78:0/318679545 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55802cbc0 con 0x7fe5600705b0 2024-02-19T14:53:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.220+0000 7fe55effd640 1 -- 172.21.15.78:0/318679545 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55802cd20 con 0x7fe5600705b0 2024-02-19T14:53:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.220+0000 7fe55effd640 1 -- 172.21.15.78:0/318679545 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558034660 con 0x7fe5600705b0 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.220+0000 7fe56629a640 1 -- 172.21.15.78:0/318679545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600705b0 msgr2=0x7fe5600709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 --2- 172.21.15.78:0/318679545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600705b0 0x7fe5600709c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe558009d90 tx=0x7fe55802be10 comp rx=0 tx=0).stop 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 -- 172.21.15.78:0/318679545 shutdown_connections 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 --2- 172.21.15.78:0/318679545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600705b0 0x7fe5600709c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 --2- 172.21.15.78:0/318679545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600add70 0x7fe5600c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 -- 172.21.15.78:0/318679545 >> 172.21.15.78:0/318679545 conn(0x7fe5600c85e0 msgr2=0x7fe5600c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 -- 172.21.15.78:0/318679545 shutdown_connections 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.221+0000 7fe56629a640 1 -- 172.21.15.78:0/318679545 wait complete. 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 Processor -- start 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 -- start start 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600705b0 0x7fe5600accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 0x7fe560144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600c2f60 0x7fe5601452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56006d1e0 con 0x7fe5600add70 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe560156680 con 0x7fe5600705b0 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.222+0000 7fe56629a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe560156f00 con 0x7fe5600c2f60 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 0x7fe560144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 0x7fe560144d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51148/0 (socket says 172.21.15.78:51148) 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 -- 172.21.15.78:0/1643857823 learned_addr learned my addr 172.21.15.78:0/1643857823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe564837640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600c2f60 0x7fe5601452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 -- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600c2f60 msgr2=0x7fe5601452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600c2f60 0x7fe5601452d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 -- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600705b0 msgr2=0x7fe5600accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55ffff640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600705b0 0x7fe5600accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600705b0 0x7fe5600accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 -- 172.21.15.78:0/1643857823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558009a70 con 0x7fe5600add70 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55ffff640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600705b0 0x7fe5600accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55f7fe640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 0x7fe560144d90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe55400c460 tx=0x7fe55400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540074e0 con 0x7fe5600add70 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe554007660 con 0x7fe5600add70 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe554004a20 con 0x7fe5600add70 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56006ccb0 con 0x7fe5600add70 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.223+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe560145810 con 0x7fe5600add70 2024-02-19T14:53:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.224+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5601570b0 con 0x7fe5600add70 2024-02-19T14:53:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.225+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe554004b80 con 0x7fe5600add70 2024-02-19T14:53:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.225+0000 7fe55cff9640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe54c06b810 0x7fe54c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.225+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe554089760 con 0x7fe5600add70 2024-02-19T14:53:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.225+0000 7fe55ffff640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe54c06b810 0x7fe54c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.226+0000 7fe55ffff640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe54c06b810 0x7fe54c06dcd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe550006fd0 tx=0x7fe550008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.234+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe554054050 con 0x7fe5600add70 2024-02-19T14:53:45.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.452+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe560074990 con 0x7fe5600add70 2024-02-19T14:53:45.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.452+0000 7fe55cff9640 1 -- 172.21.15.78:0/1643857823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) v1 ==== 84+0+207 (secure 0 0 0) 0x7fe554057710 con 0x7fe5600add70 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 7, 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": "cephfs_metadata" 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:53:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:53:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 8, 2024-02-19T14:53:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": "cephfs_data" 2024-02-19T14:53:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:53:45.456 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe54c06b810 msgr2=0x7fe54c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe54c06b810 0x7fe54c06dcd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe550006fd0 tx=0x7fe550008040 comp rx=0 tx=0).stop 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 msgr2=0x7fe560144d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 0x7fe560144d90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe55400c460 tx=0x7fe55400c920 comp rx=0 tx=0).stop 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 shutdown_connections 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe54c06b810 0x7fe54c06dcd0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5600add70 0x7fe560144d90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5600c2f60 0x7fe5601452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 --2- 172.21.15.78:0/1643857823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600705b0 0x7fe5600accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 >> 172.21.15.78:0/1643857823 conn(0x7fe5600c85e0 msgr2=0x7fe5600c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 shutdown_connections 2024-02-19T14:53:45.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.454+0000 7fe56629a640 1 -- 172.21.15.78:0/1643857823 wait complete. 2024-02-19T14:53:45.468 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-02-19T14:53:45.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-02-19T14:53:45.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.569+0000 7f71f8237640 1 Processor -- start 2024-02-19T14:53:45.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f8237640 1 -- start start 2024-02-19T14:53:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f8237640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71f0148860 0x7f71f0152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f8237640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f71f00c20a0 0x7f71f00c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f8237640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f00c8270 con 0x7f71f0155870 2024-02-19T14:53:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f8237640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f0152bc0 con 0x7f71f0148860 2024-02-19T14:53:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f8237640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00c79d0 con 0x7f71f00c20a0 2024-02-19T14:53:45.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f5fd3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71f0148860 0x7f71f0152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f67d4640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f71f0155870 0x7f71f00c40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42190/0 (socket says 172.21.15.78:42190) 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f67d4640 1 -- 172.21.15.78:0/1386050330 learned_addr learned my addr 172.21.15.78:0/1386050330 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.570+0000 7f71f5fd3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71f0148860 0x7f71f0152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51150/0 (socket says 172.21.15.78:51150) 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f5fd3640 1 -- 172.21.15.78:0/1386050330 >> v2:172.21.15.150:3300/0 conn(0x7f71f00c20a0 msgr2=0x7f71f00c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f57d2640 1 --2- 172.21.15.78:0/1386050330 >> v2:172.21.15.150:3300/0 conn(0x7f71f00c20a0 0x7f71f00c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f4fd1640 1 -- 172.21.15.78:0/1386050330 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165587421 0 0) 0x7f71f00c8270 con 0x7f71f0155870 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f5fd3640 1 --2- 172.21.15.78:0/1386050330 >> v2:172.21.15.150:3300/0 conn(0x7f71f00c20a0 0x7f71f00c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f5fd3640 1 -- 172.21.15.78:0/1386050330 >> v1:172.21.15.78:6789/0 conn(0x7f71f0155870 legacy=0x7f71f00c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f5fd3640 1 -- 172.21.15.78:0/1386050330 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f00c7cb0 con 0x7f71f0148860 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f57d2640 1 --2- 172.21.15.78:0/1386050330 >> v2:172.21.15.150:3300/0 conn(0x7f71f00c20a0 0x7f71f00c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.571+0000 7f71f5fd3640 1 --2- 172.21.15.78:0/1386050330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0148860 0x7f71f0152680 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f71f00c5240 tx=0x7f71ec02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c40333c64c92065 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f4fd1640 1 -- 172.21.15.78:0/1386050330 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec031710 con 0x7f71f0148860 2024-02-19T14:53:45.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f4fd1640 1 -- 172.21.15.78:0/1386050330 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec02f050 con 0x7f71f0148860 2024-02-19T14:53:45.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f4fd1640 1 -- 172.21.15.78:0/1386050330 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec031a90 con 0x7f71f0148860 2024-02-19T14:53:45.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 -- 172.21.15.78:0/1386050330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0148860 msgr2=0x7f71f0152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 --2- 172.21.15.78:0/1386050330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0148860 0x7f71f0152680 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f71f00c5240 tx=0x7f71ec02b7b0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 -- 172.21.15.78:0/1386050330 shutdown_connections 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 --2- 172.21.15.78:0/1386050330 >> v2:172.21.15.150:3300/0 conn(0x7f71f00c20a0 0x7f71f00c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 --2- 172.21.15.78:0/1386050330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0148860 0x7f71f0152680 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 -- 172.21.15.78:0/1386050330 >> 172.21.15.78:0/1386050330 conn(0x7f71f006bf90 msgr2=0x7f71f006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 -- 172.21.15.78:0/1386050330 shutdown_connections 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.572+0000 7f71f8237640 1 -- 172.21.15.78:0/1386050330 wait complete. 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.573+0000 7f71f8237640 1 Processor -- start 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 -- start start 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f00c20a0 0x7f71f00bc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f0148860 0x7f71f00ac410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 0x7f71f0146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0145b90 con 0x7f71f0155870 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01e93f0 con 0x7f71f0148860 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f8237640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0145da0 con 0x7f71f00c20a0 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f67d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 0x7f71f0146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f67d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 0x7f71f0146880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51152/0 (socket says 172.21.15.78:51152) 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.574+0000 7f71f67d4640 1 -- 172.21.15.78:0/1384161011 learned_addr learned my addr 172.21.15.78:0/1384161011 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f67d4640 1 -- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f00c20a0 msgr2=0x7f71f00bc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f57d2640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f0148860 0x7f71f00ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f5fd3640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f00c20a0 0x7f71f00bc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f67d4640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f00c20a0 0x7f71f00bc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f67d4640 1 -- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f0148860 msgr2=0x7f71f00ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f67d4640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f0148860 0x7f71f00ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f67d4640 1 -- 172.21.15.78:0/1384161011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec009400 con 0x7f71f0155870 2024-02-19T14:53:45.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f57d2640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f0148860 0x7f71f00ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f67d4640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 0x7f71f0146880 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f71e800c7a0 tx=0x7f71e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e800a640 con 0x7f71f0155870 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e800a7a0 con 0x7f71f0155870 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f0146080 con 0x7f71f0155870 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e800abd0 con 0x7f71f0155870 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.575+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f00ab610 con 0x7f71f0155870 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.576+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f01465a0 con 0x7f71f0155870 2024-02-19T14:53:45.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.576+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f71e800ad30 con 0x7f71f0155870 2024-02-19T14:53:45.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.577+0000 7f71e67fc640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:45.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.577+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f71e8089b80 con 0x7f71f0155870 2024-02-19T14:53:45.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.577+0000 7f71f5fd3640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:45.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.577+0000 7f71f5fd3640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f71ec035000 tx=0x7f71ec008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.581+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f71f01465a0 con 0x7f71f0155870 2024-02-19T14:53:45.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:45.778+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f71f01465a0 con 0x7f71f0155870 2024-02-19T14:53:46.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.715+0000 7f71e67fc640 1 -- 172.21.15.78:0/1384161011 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v54) v1 ==== 140+0+0 (secure 0 0 0) 0x7f71f01465a0 con 0x7f71f0155870 2024-02-19T14:53:46.718 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_metadata' created 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.717+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 msgr2=0x7f71d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.717+0000 7f71f8237640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f71ec035000 tx=0x7f71ec008030 comp rx=0 tx=0).stop 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.717+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 msgr2=0x7f71f0146880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.717+0000 7f71f8237640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 0x7f71f0146880 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f71e800c7a0 tx=0x7f71e800cc60 comp rx=0 tx=0).stop 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 shutdown_connections 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0155870 0x7f71f0146880 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f0148860 0x7f71f00ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 --2- 172.21.15.78:0/1384161011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f00c20a0 0x7f71f00bc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 >> 172.21.15.78:0/1384161011 conn(0x7f71f006bf90 msgr2=0x7f71f01511f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 shutdown_connections 2024-02-19T14:53:46.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.718+0000 7f71f8237640 1 -- 172.21.15.78:0/1384161011 wait complete. 2024-02-19T14:53:46.736 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:53:46.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.862+0000 7f8784010640 1 Processor -- start 2024-02-19T14:53:46.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.862+0000 7f8784010640 1 -- start start 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f8784010640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f877c06cf60 0x7f877c160940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f8784010640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c06dfd0 con 0x7f877c0c87c0 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f8784010640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877c155d80 con 0x7f877c14bb30 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f8784010640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f877c0ca610 con 0x7f877c06cf60 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87825ad640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f877c06cf60 0x7f877c160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87825ad640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f877c06cf60 0x7f877c160940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51156/0 (socket says 172.21.15.78:51156) 2024-02-19T14:53:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87815ab640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f877c0c87c0 0x7f877c06c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42206/0 (socket says 172.21.15.78:42206) 2024-02-19T14:53:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87825ad640 1 -- 172.21.15.78:0/815323128 learned_addr learned my addr 172.21.15.78:0/815323128 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87825ad640 1 -- 172.21.15.78:0/815323128 >> v1:172.21.15.150:6789/0 conn(0x7f877c14bb30 legacy=0x7f877c0c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87825ad640 1 -- 172.21.15.78:0/815323128 >> v1:172.21.15.78:6789/0 conn(0x7f877c0c87c0 legacy=0x7f877c06c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:46.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.863+0000 7f87825ad640 1 -- 172.21.15.78:0/815323128 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c0c5240 con 0x7f877c06cf60 2024-02-19T14:53:46.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.864+0000 7f87825ad640 1 --2- 172.21.15.78:0/815323128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c06cf60 0x7f877c160940 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8774009e50 tx=0x7f877402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd1f05dacb005d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.864+0000 7f8780daa640 1 -- 172.21.15.78:0/815323128 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877402cb20 con 0x7f877c06cf60 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.864+0000 7f8780daa640 1 -- 172.21.15.78:0/815323128 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877402cc80 con 0x7f877c06cf60 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.864+0000 7f8780daa640 1 -- 172.21.15.78:0/815323128 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87740343f0 con 0x7f877c06cf60 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.864+0000 7f8784010640 1 -- 172.21.15.78:0/815323128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c06cf60 msgr2=0x7f877c160940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.865+0000 7f8784010640 1 --2- 172.21.15.78:0/815323128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c06cf60 0x7f877c160940 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8774009e50 tx=0x7f877402bf80 comp rx=0 tx=0).stop 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.865+0000 7f8784010640 1 -- 172.21.15.78:0/815323128 shutdown_connections 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.865+0000 7f8784010640 1 --2- 172.21.15.78:0/815323128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c06cf60 0x7f877c160940 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.865+0000 7f8784010640 1 -- 172.21.15.78:0/815323128 >> 172.21.15.78:0/815323128 conn(0x7f877c0ac3f0 msgr2=0x7f877c0ac7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.865+0000 7f8784010640 1 -- 172.21.15.78:0/815323128 shutdown_connections 2024-02-19T14:53:46.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.865+0000 7f8784010640 1 -- 172.21.15.78:0/815323128 wait complete. 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 Processor -- start 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 -- start start 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877c06cf60 0x7f877c146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f877c0c87c0 0x7f877c1ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 0x7f877c1ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c159430 con 0x7f877c14bb30 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c159150 con 0x7f877c0c87c0 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f8784010640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c14ebe0 con 0x7f877c06cf60 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.866+0000 7f87825ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 0x7f877c1ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 0x7f877c1ee9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51162/0 (socket says 172.21.15.78:51162) 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 -- 172.21.15.78:0/578637528 learned_addr learned my addr 172.21.15.78:0/578637528 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87815ab640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f877c0c87c0 0x7f877c1ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:46.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f8781dac640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877c06cf60 0x7f877c146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 -- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877c06cf60 msgr2=0x7f877c146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877c06cf60 0x7f877c146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 -- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f877c0c87c0 msgr2=0x7f877c1ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f877c0c87c0 0x7f877c1ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 -- 172.21.15.78:0/578637528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8774009b30 con 0x7f877c14bb30 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f8781dac640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877c06cf60 0x7f877c146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87825ad640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 0x7f877c1ee9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8774036040 tx=0x7f8774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774009510 con 0x7f877c14bb30 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8774007e50 con 0x7f877c14bb30 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c1edf60 con 0x7f877c14bb30 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877403e690 con 0x7f877c14bb30 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.867+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c146a30 con 0x7f877c14bb30 2024-02-19T14:53:46.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.868+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c1ee150 con 0x7f877c14bb30 2024-02-19T14:53:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.869+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f877403e7f0 con 0x7f877c14bb30 2024-02-19T14:53:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.869+0000 7f87727fc640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f876006b840 0x7f876006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.869+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6530+0+0 (secure 0 0 0) 0x7f87740acf10 con 0x7f877c14bb30 2024-02-19T14:53:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.869+0000 7f8781dac640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f876006b840 0x7f876006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.870+0000 7f8781dac640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f876006b840 0x7f876006dd00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8778007fd0 tx=0x7f8778009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:46.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:46.874+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f87740775a0 con 0x7f877c14bb30 2024-02-19T14:53:47.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.058+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f877c074990 con 0x7f877c14bb30 2024-02-19T14:53:47.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.720+0000 7f87727fc640 1 -- 172.21.15.78:0/578637528 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v55) v1 ==== 172+0+0 (secure 0 0 0) 0x7f877407ac60 con 0x7f877c14bb30 2024-02-19T14:53:47.723 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_data' created 2024-02-19T14:53:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.723+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f876006b840 msgr2=0x7f876006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.723+0000 7f8784010640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f876006b840 0x7f876006dd00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8778007fd0 tx=0x7f8778009040 comp rx=0 tx=0).stop 2024-02-19T14:53:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 msgr2=0x7f877c1ee9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:47.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 0x7f877c1ee9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8774036040 tx=0x7f8774004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 shutdown_connections 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f876006b840 0x7f876006dd00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f877c14bb30 0x7f877c1ee9c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f877c0c87c0 0x7f877c1ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 --2- 172.21.15.78:0/578637528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877c06cf60 0x7f877c146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 >> 172.21.15.78:0/578637528 conn(0x7f877c0ac3f0 msgr2=0x7f877c0ac7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 shutdown_connections 2024-02-19T14:53:47.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.724+0000 7f8784010640 1 -- 172.21.15.78:0/578637528 wait complete. 2024-02-19T14:53:47.746 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-02-19T14:53:47.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.888+0000 7f62fa85c640 1 Processor -- start 2024-02-19T14:53:47.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.888+0000 7f62fa85c640 1 -- start start 2024-02-19T14:53:47.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.888+0000 7f62fa85c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62f40713a0 0x7f62f4071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:47.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.888+0000 7f62fa85c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62f40c29e0 0x7f62f40c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.888+0000 7f62fa85c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40c69f0 con 0x7f62f40c8b60 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62fa85c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f40c32c0 con 0x7f62f40713a0 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62fa85c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f4070500 con 0x7f62f40c29e0 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62f40713a0 0x7f62f4071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f8df9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f62f40c8b60 0x7f62f41536b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60298/0 (socket says 172.21.15.78:60298) 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f8df9640 1 -- 172.21.15.78:0/3928395273 learned_addr learned my addr 172.21.15.78:0/3928395273 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f3fff640 1 -- 172.21.15.78:0/3928395273 >> v2:172.21.15.150:3300/0 conn(0x7f62f40c29e0 msgr2=0x7f62f40c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f3fff640 1 --2- 172.21.15.78:0/3928395273 >> v2:172.21.15.150:3300/0 conn(0x7f62f40c29e0 0x7f62f40c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f3fff640 1 -- 172.21.15.78:0/3928395273 >> v1:172.21.15.78:6789/0 conn(0x7f62f40c8b60 legacy=0x7f62f41536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f3fff640 1 -- 172.21.15.78:0/3928395273 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f406fe40 con 0x7f62f40713a0 2024-02-19T14:53:47.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.889+0000 7f62f2ffd640 1 -- 172.21.15.78:0/3928395273 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890673366 0 0) 0x7f62f40c69f0 con 0x7f62f40c8b60 2024-02-19T14:53:47.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.890+0000 7f62f3fff640 1 --2- 172.21.15.78:0/3928395273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40713a0 0x7f62f4071780 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f62f40c3070 tx=0x7f62e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d2b08c15417323 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:47.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.890+0000 7f62f2ffd640 1 -- 172.21.15.78:0/3928395273 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e402e070 con 0x7f62f40713a0 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.890+0000 7f62f2ffd640 1 -- 172.21.15.78:0/3928395273 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e40315a0 con 0x7f62f40713a0 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.890+0000 7f62f2ffd640 1 -- 172.21.15.78:0/3928395273 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4031930 con 0x7f62f40713a0 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 -- 172.21.15.78:0/3928395273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40713a0 msgr2=0x7f62f4071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 --2- 172.21.15.78:0/3928395273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40713a0 0x7f62f4071780 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f62f40c3070 tx=0x7f62e402b840 comp rx=0 tx=0).stop 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 -- 172.21.15.78:0/3928395273 shutdown_connections 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 --2- 172.21.15.78:0/3928395273 >> v2:172.21.15.150:3300/0 conn(0x7f62f40c29e0 0x7f62f40c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 --2- 172.21.15.78:0/3928395273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40713a0 0x7f62f4071780 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 -- 172.21.15.78:0/3928395273 >> 172.21.15.78:0/3928395273 conn(0x7f62f40aa300 msgr2=0x7f62f40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 -- 172.21.15.78:0/3928395273 shutdown_connections 2024-02-19T14:53:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.891+0000 7f62fa85c640 1 -- 172.21.15.78:0/3928395273 wait complete. 2024-02-19T14:53:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.892+0000 7f62fa85c640 1 Processor -- start 2024-02-19T14:53:47.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.892+0000 7f62fa85c640 1 -- start start 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62fa85c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f40713a0 0x7f62f41eb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62fa85c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 0x7f62f41ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62fa85c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f40c8b60 0x7f62f40ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62fa85c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41ec2b0 con 0x7f62f40c29e0 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62fa85c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40ac200 con 0x7f62f40713a0 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62fa85c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41ec490 con 0x7f62f40c8b60 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 0x7f62f41ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 0x7f62f41ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53242/0 (socket says 172.21.15.78:53242) 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 -- 172.21.15.78:0/1450353030 learned_addr learned my addr 172.21.15.78:0/1450353030 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f3fff640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f40713a0 0x7f62f41eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f8df9640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f40c8b60 0x7f62f40ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 -- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f40c8b60 msgr2=0x7f62f40ac690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f40c8b60 0x7f62f40ac690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 -- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f40713a0 msgr2=0x7f62f41eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f40713a0 0x7f62f41eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 -- 172.21.15.78:0/1450353030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4009400 con 0x7f62f40c29e0 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f3fff640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f40713a0 0x7f62f41eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f37fe640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 0x7f62f41ebd70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f62e800b450 tx=0x7f62e800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8011890 con 0x7f62f40c29e0 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.893+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e80119f0 con 0x7f62f40c29e0 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.894+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f4154170 con 0x7f62f40c29e0 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.894+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8011da0 con 0x7f62f40c29e0 2024-02-19T14:53:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.894+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41546f0 con 0x7f62f40c29e0 2024-02-19T14:53:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.895+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f40ac370 con 0x7f62f40c29e0 2024-02-19T14:53:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.895+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f62e801a400 con 0x7f62f40c29e0 2024-02-19T14:53:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.895+0000 7f62f0ff9640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62e006b810 0x7f62e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.896+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6913+0+0 (secure 0 0 0) 0x7f62e808a9a0 con 0x7f62f40c29e0 2024-02-19T14:53:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.896+0000 7f62f3fff640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62e006b810 0x7f62e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:47.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.896+0000 7f62f3fff640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62e006b810 0x7f62e006dcd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62e4036000 tx=0x7f62e403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:47.900+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62e8054f90 con 0x7f62f40c29e0 2024-02-19T14:53:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.211+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f62f4074990 con 0x7f62f40c29e0 2024-02-19T14:53:48.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.733+0000 7f62f0ff9640 1 -- 172.21.15.78:0/1450353030 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 9 and data pool 10 v27) v1 ==== 176+0+0 (secure 0 0 0) 0x7f62e8009280 con 0x7f62f40c29e0 2024-02-19T14:53:48.737 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 9 and data pool 10 2024-02-19T14:53:48.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.740+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62e006b810 msgr2=0x7f62e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:48.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.740+0000 7f62fa85c640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62e006b810 0x7f62e006dcd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62e4036000 tx=0x7f62e403d040 comp rx=0 tx=0).stop 2024-02-19T14:53:48.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.740+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 msgr2=0x7f62f41ebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.740+0000 7f62fa85c640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 0x7f62f41ebd70 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f62e800b450 tx=0x7f62e800b910 comp rx=0 tx=0).stop 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.741+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 shutdown_connections 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.741+0000 7f62fa85c640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62e006b810 0x7f62e006dcd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.741+0000 7f62fa85c640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f40c29e0 0x7f62f41ebd70 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.741+0000 7f62fa85c640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f40c8b60 0x7f62f40ac690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.741+0000 7f62fa85c640 1 --2- 172.21.15.78:0/1450353030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f40713a0 0x7f62f41eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.741+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 >> 172.21.15.78:0/1450353030 conn(0x7f62f40aa300 msgr2=0x7f62f41536b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:48.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.742+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 shutdown_connections 2024-02-19T14:53:48.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.743+0000 7f62fa85c640 1 -- 172.21.15.78:0/1450353030 wait complete. 2024-02-19T14:53:48.763 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.862+0000 7fb50381c640 1 Processor -- start 2024-02-19T14:53:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.863+0000 7fb50381c640 1 -- start start 2024-02-19T14:53:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb50381c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4fc06ea50 0x7fb4fc072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb50381c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 0x7fb4fc0c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb50381c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc0c5970 con 0x7fb4fc14f1a0 2024-02-19T14:53:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb50381c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c5750 con 0x7fb4fc06ea50 2024-02-19T14:53:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb50381c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0702d0 con 0x7fb4fc152680 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb5015b8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb4fc14f1a0 0x7fb4fc06fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60308/0 (socket says 172.21.15.78:60308) 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb5015b8640 1 -- 172.21.15.78:0/3299652515 learned_addr learned my addr 172.21.15.78:0/3299652515 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb500db7640 1 --2- 172.21.15.78:0/3299652515 >> v2:172.21.15.150:3300/0 conn(0x7fb4fc06ea50 0x7fb4fc072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098868510 0 0) 0x7fb4fc0c5970 con 0x7fb4fc14f1a0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e8003520 con 0x7fb4fc14f1a0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb501db9640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 0x7fb4fc0c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3733258407 0 0) 0x7fb4e8003520 con 0x7fb4fc14f1a0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 msgr2=0x7fb4fc0c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 0x7fb4fc0c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 >> v2:172.21.15.150:3300/0 conn(0x7fb4fc06ea50 msgr2=0x7fb4fc072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 --2- 172.21.15.78:0/3299652515 >> v2:172.21.15.150:3300/0 conn(0x7fb4fc06ea50 0x7fb4fc072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0c5bc0 con 0x7fb4fc14f1a0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.864+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb4f80019f0 con 0x7fb4fc14f1a0 2024-02-19T14:53:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 >> v1:172.21.15.78:6789/0 conn(0x7fb4fc14f1a0 legacy=0x7fb4fc06fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 0x7fb4e8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4e8004d00 0x7fb4e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c5750 con 0x7fb4e8003ef0 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0702d0 con 0x7fb4e8004d00 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e800ba60 con 0x7fb4fc152680 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb500db7640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb5015b8640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 0x7fb4e8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb501db9640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4e8004d00 0x7fb4e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb500db7640 1 -- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 msgr2=0x7fb4e8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb500db7640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc152680 0x7fb4e8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb500db7640 1 -- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4e8004d00 msgr2=0x7fb4e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb500db7640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4e8004d00 0x7fb4e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.865+0000 7fb500db7640 1 -- 172.21.15.78:0/3299652515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e8003520 con 0x7fb4e8003ef0 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb500db7640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4e80047c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009ac0 tx=0x7fb4e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672a860b14e4123f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec02c710 con 0x7fb4e8003ef0 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec02c870 con 0x7fb4e8003ef0 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb4f3fff640 1 -- 172.21.15.78:0/3299652515 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec02cc00 con 0x7fb4e8003ef0 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb50381c640 1 -- 172.21.15.78:0/3299652515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 msgr2=0x7fb4e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb50381c640 1 --2- 172.21.15.78:0/3299652515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4e80047c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009ac0 tx=0x7fb4e80057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb501db9640 1 -- 172.21.15.78:0/3299652515 reap_dead start 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb50381c640 1 -- 172.21.15.78:0/3299652515 shutdown_connections 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb50381c640 1 -- 172.21.15.78:0/3299652515 >> 172.21.15.78:0/3299652515 conn(0x7fb4fc06c170 msgr2=0x7fb4fc06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.866+0000 7fb50381c640 1 -- 172.21.15.78:0/3299652515 shutdown_connections 2024-02-19T14:53:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.867+0000 7fb50381c640 1 -- 172.21.15.78:0/3299652515 wait complete. 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.868+0000 7fb50381c640 1 Processor -- start 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.868+0000 7fb50381c640 1 -- start start 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb50381c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4fc0ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb50381c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4fc0c2020 0x7fb4fc1468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb50381c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc151ac0 0x7fb4fc146e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb50381c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c3870 con 0x7fb4e8003ef0 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb50381c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc14c320 con 0x7fb4fc0c2020 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb50381c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c2750 con 0x7fb4fc151ac0 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb5015b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4fc0ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb5015b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4fc0ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53262/0 (socket says 172.21.15.78:53262) 2024-02-19T14:53:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.869+0000 7fb5015b8640 1 -- 172.21.15.78:0/3522008124 learned_addr learned my addr 172.21.15.78:0/3522008124 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb500db7640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4fc0c2020 0x7fb4fc1468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb501db9640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc151ac0 0x7fb4fc146e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb5015b8640 1 -- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc151ac0 msgr2=0x7fb4fc146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb5015b8640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc151ac0 0x7fb4fc146e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb5015b8640 1 -- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4fc0c2020 msgr2=0x7fb4fc1468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb5015b8640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4fc0c2020 0x7fb4fc1468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb5015b8640 1 -- 172.21.15.78:0/3522008124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc0c4a80 con 0x7fb4e8003ef0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb501db9640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc151ac0 0x7fb4fc146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb5015b8640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4fc0ab6b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80079f0 tx=0x7fb4f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8001a30 con 0x7fb4e8003ef0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f80133f0 con 0x7fb4e8003ef0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80137a0 con 0x7fb4e8003ef0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc0c4d60 con 0x7fb4e8003ef0 2024-02-19T14:53:48.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.870+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc0c2450 con 0x7fb4e8003ef0 2024-02-19T14:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.871+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc14c460 con 0x7fb4e8003ef0 2024-02-19T14:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.872+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb4f8013900 con 0x7fb4e8003ef0 2024-02-19T14:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.872+0000 7fb4f1ffb640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4e006b7b0 0x7fb4e006dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.872+0000 7fb500db7640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4e006b7b0 0x7fb4e006dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.872+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7015+0+0 (secure 0 0 0) 0x7fb4f808b330 con 0x7fb4e8003ef0 2024-02-19T14:53:48.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.873+0000 7fb500db7640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4e006b7b0 0x7fb4e006dc70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec036000 tx=0x7fb4ec030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:48.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:48.880+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4fc14c460 con 0x7fb4e8003ef0 2024-02-19T14:53:49.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.056+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4fc14c460 con 0x7fb4e8003ef0 2024-02-19T14:53:49.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.058+0000 7fb4f1ffb640 1 -- 172.21.15.78:0/3522008124 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+20763 (secure 0 0 0) 0x7fb4fc14c460 con 0x7fb4e8003ef0 2024-02-19T14:53:49.061 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:49.061 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":56,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:48.212884+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4e006b7b0 msgr2=0x7fb4e006dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4e006b7b0 0x7fb4e006dc70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec036000 tx=0x7fb4ec030000 comp rx=0 tx=0).stop 2024-02-19T14:53:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 msgr2=0x7fb4fc0ab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4fc0ab6b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80079f0 tx=0x7fb4f800a710 comp rx=0 tx=0).stop 2024-02-19T14:53:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 shutdown_connections 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4e006b7b0 0x7fb4e006dc70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4fc151ac0 0x7fb4fc146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4fc0c2020 0x7fb4fc1468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 --2- 172.21.15.78:0/3522008124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4e8003ef0 0x7fb4fc0ab6b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 >> 172.21.15.78:0/3522008124 conn(0x7fb4fc06c170 msgr2=0x7fb4fc06ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 shutdown_connections 2024-02-19T14:53:49.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.060+0000 7fb50381c640 1 -- 172.21.15.78:0/3522008124 wait complete. 2024-02-19T14:53:49.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:49.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.184+0000 7fa4aa476640 1 Processor -- start 2024-02-19T14:53:49.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.184+0000 7fa4aa476640 1 -- start start 2024-02-19T14:53:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4aa476640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4a40c2f60 0x7fa4a4070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4aa476640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40ae670 con 0x7fa4a40add70 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4aa476640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40716b0 con 0x7fa4a40c4980 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4aa476640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40c3390 con 0x7fa4a40c2f60 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4a3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa4a40add70 0x7fa4a406eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60310/0 (socket says 172.21.15.78:60310) 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4a3fff640 1 -- 172.21.15.78:0/3967706487 learned_addr learned my addr 172.21.15.78:0/3967706487 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.185+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236049520 0 0) 0x7fa4a40ae670 con 0x7fa4a40add70 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c003520 con 0x7fa4a40add70 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a8a13640 1 --2- 172.21.15.78:0/3967706487 >> v2:172.21.15.150:3300/0 conn(0x7fa4a40c2f60 0x7fa4a4070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 134626357 0 0) 0x7fa48c003520 con 0x7fa4a40add70 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 >> v2:172.21.15.150:3300/0 conn(0x7fa4a40c2f60 msgr2=0x7fa4a4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 --2- 172.21.15.78:0/3967706487 >> v2:172.21.15.150:3300/0 conn(0x7fa4a40c2f60 0x7fa4a4070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 >> v1:172.21.15.150:6789/0 conn(0x7fa4a40c4980 legacy=0x7fa4a40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4153780 con 0x7fa4a40add70 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fa494001670 con 0x7fa4a40add70 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 >> v1:172.21.15.78:6789/0 conn(0x7fa4a40add70 legacy=0x7fa4a406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48c003970 0x7fa48c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa48c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48c005180 0x7fa48c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40c3390 con 0x7fa48c0047c0 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c00c120 con 0x7fa48c005180 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c00c2e0 con 0x7fa48c003970 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa48c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a8a13640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48c005180 0x7fa48c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a3fff640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48c003970 0x7fa48c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a37fe640 1 -- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48c003970 msgr2=0x7fa48c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48c003970 0x7fa48c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a37fe640 1 -- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48c005180 msgr2=0x7fa48c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48c005180 0x7fa48c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a8a13640 1 -- 172.21.15.78:0/3967706487 reap_dead start 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.186+0000 7fa4a37fe640 1 -- 172.21.15.78:0/3967706487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c003520 con 0x7fa48c0047c0 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.187+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa48c004c40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa498001750 tx=0x7fa48c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef2a2b05b641c63 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.187+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4980052d0 con 0x7fa48c0047c0 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.187+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49802e050 con 0x7fa48c0047c0 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.187+0000 7fa4a2ffd640 1 -- 172.21.15.78:0/3967706487 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498003830 con 0x7fa48c0047c0 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 -- 172.21.15.78:0/3967706487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 msgr2=0x7fa48c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa48c004c40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa498001750 tx=0x7fa48c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 -- 172.21.15.78:0/3967706487 shutdown_connections 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 --2- 172.21.15.78:0/3967706487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa48c004c40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 -- 172.21.15.78:0/3967706487 >> 172.21.15.78:0/3967706487 conn(0x7fa4a40c85e0 msgr2=0x7fa4a40c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 -- 172.21.15.78:0/3967706487 shutdown_connections 2024-02-19T14:53:49.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.188+0000 7fa4aa476640 1 -- 172.21.15.78:0/3967706487 wait complete. 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.189+0000 7fa4aa476640 1 Processor -- start 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.189+0000 7fa4aa476640 1 -- start start 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4aa476640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa4a40ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4aa476640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4a41551f0 0x7fa4a4145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4aa476640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4a4145c90 0x7fa4a41ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4aa476640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a40acb40 con 0x7fa48c0047c0 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4aa476640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41545c0 con 0x7fa4a41551f0 2024-02-19T14:53:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4aa476640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4146110 con 0x7fa4a4145c90 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa4a40ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa4a40ac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53282/0 (socket says 172.21.15.78:53282) 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 -- 172.21.15.78:0/2602669122 learned_addr learned my addr 172.21.15.78:0/2602669122 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4a41551f0 0x7fa4a4145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a8a13640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4a4145c90 0x7fa4a41ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 -- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4a4145c90 msgr2=0x7fa4a41ea4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4a4145c90 0x7fa4a41ea4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 -- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4a41551f0 msgr2=0x7fa4a4145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4a41551f0 0x7fa4a4145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 -- 172.21.15.78:0/2602669122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4152170 con 0x7fa48c0047c0 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4a41551f0 0x7fa4a4145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a8a13640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4a4145c90 0x7fa4a41ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.190+0000 7fa4a3fff640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa4a40ac600 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa49400e780 tx=0x7fa49400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.191+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49400c660 con 0x7fa48c0047c0 2024-02-19T14:53:49.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.191+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49400c800 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.191+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4154000 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.191+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49400cbb0 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.191+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a40abe00 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.191+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4074990 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.192+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa49400cd10 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.193+0000 7fa4a0ff9640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa49006b810 0x7fa49006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.193+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa49006b810 0x7fa49006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.193+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7015+0+0 (secure 0 0 0) 0x7fa49408afe0 con 0x7fa48c0047c0 2024-02-19T14:53:49.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.193+0000 7fa4a37fe640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa49006b810 0x7fa49006dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa498033000 tx=0x7fa498009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.196+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa494055650 con 0x7fa48c0047c0 2024-02-19T14:53:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.370+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4a4071490 con 0x7fa48c0047c0 2024-02-19T14:53:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.371+0000 7fa4a0ff9640 1 -- 172.21.15.78:0/2602669122 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+20763 (secure 0 0 0) 0x7fa494058d10 con 0x7fa48c0047c0 2024-02-19T14:53:49.374 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:49.374 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":56,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:48.212884+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa49006b810 msgr2=0x7fa49006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa49006b810 0x7fa49006dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa498033000 tx=0x7fa498009030 comp rx=0 tx=0).stop 2024-02-19T14:53:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 msgr2=0x7fa4a40ac600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa4a40ac600 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa49400e780 tx=0x7fa49400ec40 comp rx=0 tx=0).stop 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 shutdown_connections 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa49006b810 0x7fa49006dcd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4a4145c90 0x7fa4a41ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4a41551f0 0x7fa4a4145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 --2- 172.21.15.78:0/2602669122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa48c0047c0 0x7fa4a40ac600 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.373+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 >> 172.21.15.78:0/2602669122 conn(0x7fa4a40c85e0 msgr2=0x7fa4a40a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.374+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 shutdown_connections 2024-02-19T14:53:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.374+0000 7fa4aa476640 1 -- 172.21.15.78:0/2602669122 wait complete. 2024-02-19T14:53:49.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-02-19T14:53:49.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.493+0000 7fee46dc7640 1 Processor -- start 2024-02-19T14:53:49.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.493+0000 7fee46dc7640 1 -- start start 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee46dc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400719d0 0x7fee40071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee46dc7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee400c15b0 0x7fee400c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee46dc7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee400c1120 con 0x7fee4014ece0 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee46dc7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee400c6080 con 0x7fee400c15b0 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee46dc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee400c57e0 con 0x7fee400719d0 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee45364640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee400c15b0 0x7fee400c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee45364640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee400c15b0 0x7fee400c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53294/0 (socket says 172.21.15.78:53294) 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee44b63640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee4014ece0 0x7fee400c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60318/0 (socket says 172.21.15.78:60318) 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.494+0000 7fee45364640 1 -- 172.21.15.78:0/2160175271 learned_addr learned my addr 172.21.15.78:0/2160175271 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee3ffff640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400719d0 0x7fee40071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee45364640 1 -- 172.21.15.78:0/2160175271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400719d0 msgr2=0x7fee40071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee45364640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400719d0 0x7fee40071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee45364640 1 -- 172.21.15.78:0/2160175271 >> v1:172.21.15.78:6789/0 conn(0x7fee4014ece0 legacy=0x7fee400c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee3f7fe640 1 -- 172.21.15.78:0/2160175271 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570033698 0 0) 0x7fee400c1120 con 0x7fee4014ece0 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee45364640 1 -- 172.21.15.78:0/2160175271 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400c5ac0 con 0x7fee400c15b0 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee3ffff640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400719d0 0x7fee40071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee45364640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400c15b0 0x7fee400c1a40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fee38009e50 tx=0x7fee3802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c78e742197e0ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.495+0000 7fee3f7fe640 1 -- 172.21.15.78:0/2160175271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3802cb00 con 0x7fee400c15b0 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee3f7fe640 1 -- 172.21.15.78:0/2160175271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3802cc60 con 0x7fee400c15b0 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee3f7fe640 1 -- 172.21.15.78:0/2160175271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee380343f0 con 0x7fee400c15b0 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 -- 172.21.15.78:0/2160175271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400c15b0 msgr2=0x7fee400c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400c15b0 0x7fee400c1a40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fee38009e50 tx=0x7fee3802bf60 comp rx=0 tx=0).stop 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 -- 172.21.15.78:0/2160175271 shutdown_connections 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400c15b0 0x7fee400c1a40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 --2- 172.21.15.78:0/2160175271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400719d0 0x7fee40071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 -- 172.21.15.78:0/2160175271 >> 172.21.15.78:0/2160175271 conn(0x7fee4006b270 msgr2=0x7fee4006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:49.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 -- 172.21.15.78:0/2160175271 shutdown_connections 2024-02-19T14:53:49.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.496+0000 7fee46dc7640 1 -- 172.21.15.78:0/2160175271 wait complete. 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.497+0000 7fee46dc7640 1 Processor -- start 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.497+0000 7fee46dc7640 1 -- start start 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.498+0000 7fee46dc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 0x7fee4014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.498+0000 7fee46dc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400c15b0 0x7fee401dbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.498+0000 7fee46dc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee4014ece0 0x7fee401dc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.498+0000 7fee46dc7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee401488f0 con 0x7fee400719d0 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.498+0000 7fee46dc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee401494e0 con 0x7fee4014ece0 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.498+0000 7fee46dc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee40151e10 con 0x7fee400c15b0 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 0x7fee4014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 0x7fee4014f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53306/0 (socket says 172.21.15.78:53306) 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 -- 172.21.15.78:0/135771883 learned_addr learned my addr 172.21.15.78:0/135771883 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:49.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 -- 172.21.15.78:0/135771883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400c15b0 msgr2=0x7fee401dbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400c15b0 0x7fee401dbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 -- 172.21.15.78:0/135771883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee4014ece0 msgr2=0x7fee401dc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee4014ece0 0x7fee401dc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 -- 172.21.15.78:0/135771883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee38009b30 con 0x7fee400719d0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee44b63640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 0x7fee4014f240 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fee3400c490 tx=0x7fee3400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee340074e0 con 0x7fee400719d0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee34007660 con 0x7fee400719d0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee401ee870 con 0x7fee400719d0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee34004a20 con 0x7fee400719d0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.499+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee40152360 con 0x7fee400719d0 2024-02-19T14:53:49.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.500+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee400c2920 con 0x7fee400719d0 2024-02-19T14:53:49.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.501+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fee34004b80 con 0x7fee400719d0 2024-02-19T14:53:49.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.501+0000 7fee3d7fa640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee2406b810 0x7fee2406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.501+0000 7fee3ffff640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee2406b810 0x7fee2406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.501+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7015+0+0 (secure 0 0 0) 0x7fee34089ab0 con 0x7fee400719d0 2024-02-19T14:53:49.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.502+0000 7fee3ffff640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee2406b810 0x7fee2406dcd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006fd0 tx=0x7fee2c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.506+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee34054040 con 0x7fee400719d0 2024-02-19T14:53:49.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.718+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fee40074990 con 0x7fee400719d0 2024-02-19T14:53:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.738+0000 7fee3d7fa640 1 -- 172.21.15.78:0/135771883 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v28) v1 ==== 119+0+0 (secure 0 0 0) 0x7fee34057700 con 0x7fee400719d0 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee2406b810 msgr2=0x7fee2406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee2406b810 0x7fee2406dcd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006fd0 tx=0x7fee2c008040 comp rx=0 tx=0).stop 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 msgr2=0x7fee4014f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 0x7fee4014f240 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fee3400c490 tx=0x7fee3400c950 comp rx=0 tx=0).stop 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 shutdown_connections 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee2406b810 0x7fee2406dcd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee4014ece0 0x7fee401dc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee400c15b0 0x7fee401dbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 --2- 172.21.15.78:0/135771883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee400719d0 0x7fee4014f240 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 >> 172.21.15.78:0/135771883 conn(0x7fee4006b270 msgr2=0x7fee40071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 shutdown_connections 2024-02-19T14:53:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.741+0000 7fee46dc7640 1 -- 172.21.15.78:0/135771883 wait complete. 2024-02-19T14:53:49.754 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:53:49.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-02-19T14:53:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.863+0000 7fe693a07640 1 Processor -- start 2024-02-19T14:53:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.863+0000 7fe693a07640 1 -- start start 2024-02-19T14:53:49.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.863+0000 7fe693a07640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe68c0c4d80 0x7fe68c159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe693a07640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c06ffa0 con 0x7fe68c155010 2024-02-19T14:53:49.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe693a07640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c06fc70 con 0x7fe68c0c78a0 2024-02-19T14:53:49.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe693a07640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe68c06b720 con 0x7fe68c0c4d80 2024-02-19T14:53:49.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe691fa4640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe68c155010 0x7fe68c1523d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60326/0 (socket says 172.21.15.78:60326) 2024-02-19T14:53:49.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe691fa4640 1 -- 172.21.15.78:0/3730918771 learned_addr learned my addr 172.21.15.78:0/3730918771 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe690fa2640 1 --2- 172.21.15.78:0/3730918771 >> v2:172.21.15.150:3300/0 conn(0x7fe68c0c4d80 0x7fe68c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331929614 0 0) 0x7fe68c06ffa0 con 0x7fe68c155010 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe678003520 con 0x7fe68c155010 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692582198 0 0) 0x7fe678003520 con 0x7fe68c155010 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 >> v2:172.21.15.150:3300/0 conn(0x7fe68c0c4d80 msgr2=0x7fe68c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 --2- 172.21.15.78:0/3730918771 >> v2:172.21.15.150:3300/0 conn(0x7fe68c0c4d80 0x7fe68c159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 >> v1:172.21.15.150:6789/0 conn(0x7fe68c0c78a0 legacy=0x7fe68c0c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c06b930 con 0x7fe68c155010 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe6840019f0 con 0x7fe68c155010 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 >> v1:172.21.15.78:6789/0 conn(0x7fe68c155010 legacy=0x7fe68c1523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe678003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6780047c0 0x7fe678004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe678005180 0x7fe67800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c06b720 con 0x7fe678003970 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67800c120 con 0x7fe678005180 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67800c2e0 con 0x7fe6780047c0 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.864+0000 7fe6917a3640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe678003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe691fa4640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe678005180 0x7fe67800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe690fa2640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6780047c0 0x7fe678004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe6917a3640 1 -- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6780047c0 msgr2=0x7fe678004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe6917a3640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6780047c0 0x7fe678004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe6917a3640 1 -- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe678005180 msgr2=0x7fe67800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe6917a3640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe678005180 0x7fe67800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe691fa4640 1 -- 172.21.15.78:0/3730918771 reap_dead start 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe6917a3640 1 -- 172.21.15.78:0/3730918771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe678003520 con 0x7fe678003970 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.865+0000 7fe6917a3640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe678003d90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe688001750 tx=0x7fe6780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09715053b715af4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6880052d0 con 0x7fe678003970 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68802e050 con 0x7fe678003970 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe683fff640 1 -- 172.21.15.78:0/3730918771 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688003830 con 0x7fe678003970 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe693a07640 1 -- 172.21.15.78:0/3730918771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 msgr2=0x7fe678003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe693a07640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe678003d90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe688001750 tx=0x7fe6780057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe693a07640 1 -- 172.21.15.78:0/3730918771 shutdown_connections 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe693a07640 1 --2- 172.21.15.78:0/3730918771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe678003d90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.866+0000 7fe693a07640 1 -- 172.21.15.78:0/3730918771 >> 172.21.15.78:0/3730918771 conn(0x7fe68c066e70 msgr2=0x7fe68c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.867+0000 7fe693a07640 1 -- 172.21.15.78:0/3730918771 shutdown_connections 2024-02-19T14:53:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.867+0000 7fe693a07640 1 -- 172.21.15.78:0/3730918771 wait complete. 2024-02-19T14:53:49.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.868+0000 7fe693a07640 1 Processor -- start 2024-02-19T14:53:49.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.868+0000 7fe693a07640 1 -- start start 2024-02-19T14:53:49.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe693a07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe68c0c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe693a07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe68c14f5c0 0x7fe68c0abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe693a07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe68c14f000 0x7fe68c0c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe693a07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c121110 con 0x7fe678003970 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe693a07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c1563c0 con 0x7fe68c14f000 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe693a07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c1566a0 con 0x7fe68c14f5c0 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe6917a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe68c0c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.869+0000 7fe6917a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe68c0c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53316/0 (socket says 172.21.15.78:53316) 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 -- 172.21.15.78:0/2195868822 learned_addr learned my addr 172.21.15.78:0/2195868822 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 -- 172.21.15.78:0/2195868822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe68c14f5c0 msgr2=0x7fe68c0abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe68c14f5c0 0x7fe68c0abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 -- 172.21.15.78:0/2195868822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe68c14f000 msgr2=0x7fe68c0c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe68c14f000 0x7fe68c0c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 -- 172.21.15.78:0/2195868822 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c0c57d0 con 0x7fe678003970 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe6917a3640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe68c0c5290 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe688000c00 tx=0x7fe688001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688007a30 con 0x7fe678003970 2024-02-19T14:53:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688007640 con 0x7fe678003970 2024-02-19T14:53:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c1eb310 con 0x7fe678003970 2024-02-19T14:53:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688006c00 con 0x7fe678003970 2024-02-19T14:53:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.870+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c0ac4f0 con 0x7fe678003970 2024-02-19T14:53:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.871+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c074990 con 0x7fe678003970 2024-02-19T14:53:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.872+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe6880035f0 con 0x7fe678003970 2024-02-19T14:53:49.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.872+0000 7fe681ffb640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe67006b840 0x7fe67006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:49.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.872+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7fe6880ac260 con 0x7fe678003970 2024-02-19T14:53:49.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.872+0000 7fe690fa2640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe67006b840 0x7fe67006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:49.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.873+0000 7fe690fa2640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe67006b840 0x7fe67006dd00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe68c151470 tx=0x7fe67c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:49.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:49.877+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe688076760 con 0x7fe678003970 2024-02-19T14:53:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.147+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe68c152f80 con 0x7fe678003970 2024-02-19T14:53:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.743+0000 7fe681ffb640 1 -- 172.21.15.78:0/2195868822 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v29) v1 ==== 123+0+0 (secure 0 0 0) 0x7fe688079e20 con 0x7fe678003970 2024-02-19T14:53:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe67006b840 msgr2=0x7fe67006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe67006b840 0x7fe67006dd00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe68c151470 tx=0x7fe67c008040 comp rx=0 tx=0).stop 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 msgr2=0x7fe68c0c5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe68c0c5290 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe688000c00 tx=0x7fe688001120 comp rx=0 tx=0).stop 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 shutdown_connections 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe67006b840 0x7fe67006dd00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe68c14f000 0x7fe68c0c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe68c14f5c0 0x7fe68c0abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 --2- 172.21.15.78:0/2195868822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe678003970 0x7fe68c0c5290 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.745+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 >> 172.21.15.78:0/2195868822 conn(0x7fe68c066e70 msgr2=0x7fe68c153970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.746+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 shutdown_connections 2024-02-19T14:53:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.746+0000 7fe693a07640 1 -- 172.21.15.78:0/2195868822 wait complete. 2024-02-19T14:53:50.759 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-02-19T14:53:50.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.863+0000 7f0456b25640 1 Processor -- start 2024-02-19T14:53:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.863+0000 7f0456b25640 1 -- start start 2024-02-19T14:53:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f0456b25640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04500c2f60 0x7f0450070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f0456b25640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04500ae670 con 0x7f04500add70 2024-02-19T14:53:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f0456b25640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04500716b0 con 0x7f04500c4980 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f0456b25640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04500c3390 con 0x7f04500c2f60 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f04548c1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f04500add70 0x7f045006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60328/0 (socket says 172.21.15.78:60328) 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f04548c1640 1 -- 172.21.15.78:0/3462055482 learned_addr learned my addr 172.21.15.78:0/3462055482 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f04550c2640 1 --2- 172.21.15.78:0/3462055482 >> v2:172.21.15.150:3300/0 conn(0x7f04500c2f60 0x7f0450070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221032711 0 0) 0x7f04500ae670 con 0x7f04500add70 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0440003520 con 0x7f04500add70 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756284912 0 0) 0x7f0440003520 con 0x7f04500add70 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 >> v2:172.21.15.150:3300/0 conn(0x7f04500c2f60 msgr2=0x7f0450070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 --2- 172.21.15.78:0/3462055482 >> v2:172.21.15.150:3300/0 conn(0x7f04500c2f60 0x7f0450070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 >> v1:172.21.15.150:6789/0 conn(0x7f04500c4980 legacy=0x7f04500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450153780 con 0x7f04500add70 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.864+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0444001670 con 0x7f04500add70 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 >> v1:172.21.15.78:6789/0 conn(0x7f04500add70 legacy=0x7f045006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0440003970 0x7f0440003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04400047c0 0x7f0440004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0440005180 0x7f044000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04500c3390 con 0x7f0440005180 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044000c120 con 0x7f0440003970 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044000c2e0 con 0x7f04400047c0 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04550c2640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0440005180 0x7f044000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04548c1640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0440003970 0x7f0440003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f044ffff640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04400047c0 0x7f0440004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04550c2640 1 -- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04400047c0 msgr2=0x7f0440004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04550c2640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04400047c0 0x7f0440004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04550c2640 1 -- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0440003970 msgr2=0x7f0440003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04550c2640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0440003970 0x7f0440003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.865+0000 7f04550c2640 1 -- 172.21.15.78:0/3462055482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440003520 con 0x7f0440005180 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.866+0000 7f04550c2640 1 -- 172.21.15.78:0/3462055482 reap_dead start 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.866+0000 7f04550c2640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0440005180 0x7f044000b980 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0448009fc0 tx=0x7f04400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c546f45c586ae5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.866+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044802cc60 con 0x7f0440005180 2024-02-19T14:53:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.866+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044800f040 con 0x7f0440005180 2024-02-19T14:53:50.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.866+0000 7f044f7fe640 1 -- 172.21.15.78:0/3462055482 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448003990 con 0x7f0440005180 2024-02-19T14:53:50.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 -- 172.21.15.78:0/3462055482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0440005180 msgr2=0x7f044000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0440005180 0x7f044000b980 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0448009fc0 tx=0x7f04400057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 -- 172.21.15.78:0/3462055482 shutdown_connections 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 --2- 172.21.15.78:0/3462055482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0440005180 0x7f044000b980 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 -- 172.21.15.78:0/3462055482 >> 172.21.15.78:0/3462055482 conn(0x7f04500c85e0 msgr2=0x7f04500c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 -- 172.21.15.78:0/3462055482 shutdown_connections 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.867+0000 7f0456b25640 1 -- 172.21.15.78:0/3462055482 wait complete. 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.868+0000 7f0456b25640 1 Processor -- start 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.868+0000 7f0456b25640 1 -- start start 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f0456b25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440005180 0x7f04501e8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f0456b25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04501ebce0 0x7f04500ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f0456b25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 0x7f0450154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f0456b25640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04500acf30 con 0x7f0450154710 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f0456b25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04501e8cb0 con 0x7f04501ebce0 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f0456b25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04501e8ee0 con 0x7f0440005180 2024-02-19T14:53:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 0x7f0450154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 0x7f0450154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53344/0 (socket says 172.21.15.78:53344) 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 -- 172.21.15.78:0/2822854881 learned_addr learned my addr 172.21.15.78:0/2822854881 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04548c1640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440005180 0x7f04501e8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f044ffff640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04501ebce0 0x7f04500ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 -- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440005180 msgr2=0x7f04501e8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440005180 0x7f04501e8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 -- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04501ebce0 msgr2=0x7f04500ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04501ebce0 0x7f04500ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 -- 172.21.15.78:0/2822854881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04501e91c0 con 0x7f0450154710 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.869+0000 7f04550c2640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 0x7f0450154b90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f044800e200 tx=0x7f0448009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.870+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448035df0 con 0x7f0450154710 2024-02-19T14:53:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.870+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0448003990 con 0x7f0450154710 2024-02-19T14:53:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.870+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04501e94a0 con 0x7f0450154710 2024-02-19T14:53:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.870+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448003d40 con 0x7f0450154710 2024-02-19T14:53:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.870+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04500acac0 con 0x7f0450154710 2024-02-19T14:53:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.871+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0450154050 con 0x7f0450154710 2024-02-19T14:53:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.872+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0448004030 con 0x7f0450154710 2024-02-19T14:53:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.872+0000 7f044d7fa640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f043406b810 0x7f043406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.872+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f04480ad340 con 0x7f0450154710 2024-02-19T14:53:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.872+0000 7f04548c1640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f043406b810 0x7f043406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.872+0000 7f04548c1640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f043406b810 0x7f043406dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0444001050 tx=0x7f0444009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:50.877+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0450154050 con 0x7f0450154710 2024-02-19T14:53:51.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.090+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0450154050 con 0x7f0450154710 2024-02-19T14:53:51.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.091+0000 7f044d7fa640 1 -- 172.21.15.78:0/2822854881 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6183 (secure 0 0 0) 0x7f0450154050 con 0x7f0450154710 2024-02-19T14:53:51.093 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:51.093 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5398,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/210065453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":210065453},{"type":"v1","addr":"172.21.15.150:6834","nonce":210065453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:53:39.026813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5703},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5703":{"gid":5703,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/1917648829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1917648829},{"type":"v1","addr":"172.21.15.78:6837","nonce":1917648829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:53:51.094 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 29 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f043406b810 msgr2=0x7f043406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f043406b810 0x7f043406dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0444001050 tx=0x7f0444009040 comp rx=0 tx=0).stop 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 msgr2=0x7f0450154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 0x7f0450154b90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f044800e200 tx=0x7f0448009750 comp rx=0 tx=0).stop 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 shutdown_connections 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f043406b810 0x7f043406dcd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0450154710 0x7f0450154b90 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04501ebce0 0x7f04500ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 --2- 172.21.15.78:0/2822854881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440005180 0x7f04501e8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 >> 172.21.15.78:0/2822854881 conn(0x7f04500c85e0 msgr2=0x7f04500a9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 shutdown_connections 2024-02-19T14:53:51.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.093+0000 7f0456b25640 1 -- 172.21.15.78:0/2822854881 wait complete. 2024-02-19T14:53:51.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:53:51.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.210+0000 7f6777165640 1 Processor -- start 2024-02-19T14:53:51.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.210+0000 7f6777165640 1 -- start start 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6777165640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6770152700 0x7f67700c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6777165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700c8e00 0x7f677015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6777165640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67701551f0 con 0x7f67700c5730 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6777165640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6770155490 con 0x7f6770152700 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6777165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6770155770 con 0x7f67700c8e00 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6770152700 0x7f67700c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6770152700 0x7f67700c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53350/0 (socket says 172.21.15.78:53350) 2024-02-19T14:53:51.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 -- 172.21.15.78:0/2007162649 learned_addr learned my addr 172.21.15.78:0/2007162649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:51.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6775702640 1 --2- 172.21.15.78:0/2007162649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700c8e00 0x7f677015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 -- 172.21.15.78:0/2007162649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700c8e00 msgr2=0x7f677015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:51.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 --2- 172.21.15.78:0/2007162649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700c8e00 0x7f677015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 -- 172.21.15.78:0/2007162649 >> v1:172.21.15.150:6789/0 conn(0x7f67700c5730 legacy=0x7f677006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.212+0000 7f6774f01640 1 -- 172.21.15.78:0/2007162649 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6770155a50 con 0x7f6770152700 2024-02-19T14:53:51.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.213+0000 7f6774f01640 1 --2- 172.21.15.78:0/2007162649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770152700 0x7f67700c51f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6764009870 tx=0x7f676402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878c91ef1d51d2ab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.213+0000 7f676f7fe640 1 -- 172.21.15.78:0/2007162649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764030840 con 0x7f6770152700 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.213+0000 7f676f7fe640 1 -- 172.21.15.78:0/2007162649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67640309a0 con 0x7f6770152700 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.213+0000 7f676f7fe640 1 -- 172.21.15.78:0/2007162649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764030d30 con 0x7f6770152700 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 -- 172.21.15.78:0/2007162649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770152700 msgr2=0x7f67700c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 --2- 172.21.15.78:0/2007162649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770152700 0x7f67700c51f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6764009870 tx=0x7f676402b980 comp rx=0 tx=0).stop 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 -- 172.21.15.78:0/2007162649 shutdown_connections 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 --2- 172.21.15.78:0/2007162649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67700c8e00 0x7f677015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 --2- 172.21.15.78:0/2007162649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6770152700 0x7f67700c51f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 -- 172.21.15.78:0/2007162649 >> 172.21.15.78:0/2007162649 conn(0x7f67700cac90 msgr2=0x7f67700cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 -- 172.21.15.78:0/2007162649 shutdown_connections 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.214+0000 7f6777165640 1 -- 172.21.15.78:0/2007162649 wait complete. 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.215+0000 7f6777165640 1 Processor -- start 2024-02-19T14:53:51.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.215+0000 7f6777165640 1 -- start start 2024-02-19T14:53:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.216+0000 7f6777165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c5730 0x7f67701ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.216+0000 7f6777165640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 0x7f67701ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.216+0000 7f6777165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6770152700 0x7f67700ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.216+0000 7f6777165640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67701499e0 con 0x7f67700c8e00 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.216+0000 7f6777165640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67701e8480 con 0x7f67700c5730 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.216+0000 7f6777165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67701eebd0 con 0x7f6770152700 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 0x7f67701ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 0x7f67701ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53362/0 (socket says 172.21.15.78:53362) 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 -- 172.21.15.78:0/3305896824 learned_addr learned my addr 172.21.15.78:0/3305896824 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f6775702640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6770152700 0x7f67700ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f6774f01640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c5730 0x7f67701ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 -- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6770152700 msgr2=0x7f67700ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6770152700 0x7f67700ac4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 -- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c5730 msgr2=0x7f67701ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c5730 0x7f67701ee050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 -- 172.21.15.78:0/3305896824 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640094d0 con 0x7f67700c8e00 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676ffff640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 0x7f67701ee590 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f675c00c400 tx=0x7f675c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0052d0 con 0x7f67700c8e00 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c005450 con 0x7f67700c8e00 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67701ebd80 con 0x7f67700c8e00 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c007260 con 0x7f67700c8e00 2024-02-19T14:53:51.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.217+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67701ec340 con 0x7f67700c8e00 2024-02-19T14:53:51.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.219+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67700c4b20 con 0x7f67700c8e00 2024-02-19T14:53:51.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.219+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f675c0073c0 con 0x7f67700c8e00 2024-02-19T14:53:51.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.219+0000 7f676d7fa640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675406b810 0x7f675406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.219+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f675c089a90 con 0x7f67700c8e00 2024-02-19T14:53:51.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.219+0000 7f6774f01640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675406b810 0x7f675406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.220+0000 7f6774f01640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675406b810 0x7f675406dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6764036000 tx=0x7f676403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:51.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.224+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f675c054020 con 0x7f67700c8e00 2024-02-19T14:53:51.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.462+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6770074990 con 0x7f67700c8e00 2024-02-19T14:53:51.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.463+0000 7f676d7fa640 1 -- 172.21.15.78:0/3305896824 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+20750 (secure 0 0 0) 0x7f675c0576e0 con 0x7f67700c8e00 2024-02-19T14:53:51.466 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:51.466 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":57,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:53:49.733119+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675406b810 msgr2=0x7f675406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675406b810 0x7f675406dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6764036000 tx=0x7f676403b040 comp rx=0 tx=0).stop 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 msgr2=0x7f67701ee590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 0x7f67701ee590 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f675c00c400 tx=0x7f675c00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 shutdown_connections 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675406b810 0x7f675406dcd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67700c8e00 0x7f67701ee590 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6770152700 0x7f67700ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 --2- 172.21.15.78:0/3305896824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67700c5730 0x7f67701ee050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 >> 172.21.15.78:0/3305896824 conn(0x7f67700cac90 msgr2=0x7f677014d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:51.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 shutdown_connections 2024-02-19T14:53:51.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.465+0000 7f6777165640 1 -- 172.21.15.78:0/3305896824 wait complete. 2024-02-19T14:53:51.482 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:53:51.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:51.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.592+0000 7f59cfdb5640 1 Processor -- start 2024-02-19T14:53:51.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.593+0000 7f59cfdb5640 1 -- start start 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cfdb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c51b0 0x7f59c80c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cfdb5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59c80c5590 0x7f59c80c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cfdb5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f59c80c59f0 0x7f59c80ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cfdb5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c80ca6d0 con 0x7f59c80c59f0 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cfdb5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c80c9200 con 0x7f59c80c5590 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cfdb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80cb720 con 0x7f59c80c51b0 2024-02-19T14:53:51.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f59c80c59f0 0x7f59c80ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f59c80c59f0 0x7f59c80ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53378/0 (socket says 172.21.15.78:53378) 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 -- 172.21.15.78:0/3554442876 learned_addr learned my addr 172.21.15.78:0/3554442876 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cd350640 1 --2- 172.21.15.78:0/3554442876 >> v2:172.21.15.150:3300/0 conn(0x7f59c80c5590 0x7f59c80c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cdb51640 1 --2- 172.21.15.78:0/3554442876 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c51b0 0x7f59c80c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 -- 172.21.15.78:0/3554442876 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c51b0 msgr2=0x7f59c80c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 --2- 172.21.15.78:0/3554442876 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c51b0 0x7f59c80c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 -- 172.21.15.78:0/3554442876 >> v2:172.21.15.150:3300/0 conn(0x7f59c80c5590 msgr2=0x7f59c80c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 --2- 172.21.15.78:0/3554442876 >> v2:172.21.15.150:3300/0 conn(0x7f59c80c5590 0x7f59c80c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59ce352640 1 -- 172.21.15.78:0/3554442876 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8158ff0 con 0x7f59c80c59f0 2024-02-19T14:53:51.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.594+0000 7f59cd350640 1 --2- 172.21.15.78:0/3554442876 >> v2:172.21.15.150:3300/0 conn(0x7f59c80c5590 0x7f59c80c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:51.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.595+0000 7f59ce352640 1 --2- 172.21.15.78:0/3554442876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c59f0 0x7f59c80ca190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f59c808bfd0 tx=0x7f59c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb5c079ac0a80a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:51.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59ccb4f640 1 -- 172.21.15.78:0/3554442876 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c002ca70 con 0x7f59c80c59f0 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59ccb4f640 1 -- 172.21.15.78:0/3554442876 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c002cbd0 con 0x7f59c80c59f0 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59ccb4f640 1 -- 172.21.15.78:0/3554442876 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00343f0 con 0x7f59c80c59f0 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 -- 172.21.15.78:0/3554442876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c59f0 msgr2=0x7f59c80ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/3554442876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c59f0 0x7f59c80ca190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f59c808bfd0 tx=0x7f59c002bed0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 -- 172.21.15.78:0/3554442876 shutdown_connections 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/3554442876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c59f0 0x7f59c80ca190 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/3554442876 >> v2:172.21.15.150:3300/0 conn(0x7f59c80c5590 0x7f59c80c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/3554442876 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c51b0 0x7f59c80c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 -- 172.21.15.78:0/3554442876 >> 172.21.15.78:0/3554442876 conn(0x7f59c80ca830 msgr2=0x7f59c80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 -- 172.21.15.78:0/3554442876 shutdown_connections 2024-02-19T14:53:51.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.596+0000 7f59cfdb5640 1 -- 172.21.15.78:0/3554442876 wait complete. 2024-02-19T14:53:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.598+0000 7f59cfdb5640 1 Processor -- start 2024-02-19T14:53:51.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.598+0000 7f59cfdb5640 1 -- start start 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cfdb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 0x7f59c80be360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cfdb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5590 0x7f59c80c3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cfdb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c59f0 0x7f59c81eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cfdb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80c46f0 con 0x7f59c80c51b0 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cfdb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c814ae50 con 0x7f59c80c5590 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cfdb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c8149460 con 0x7f59c80c59f0 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cdb51640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 0x7f59c80be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cdb51640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 0x7f59c80be360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53392/0 (socket says 172.21.15.78:53392) 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cdb51640 1 -- 172.21.15.78:0/2694197083 learned_addr learned my addr 172.21.15.78:0/2694197083 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59cd350640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5590 0x7f59c80c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.599+0000 7f59ce352640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c59f0 0x7f59c81eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cdb51640 1 -- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c59f0 msgr2=0x7f59c81eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cdb51640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c59f0 0x7f59c81eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cdb51640 1 -- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5590 msgr2=0x7f59c80c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cdb51640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5590 0x7f59c80c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cdb51640 1 -- 172.21.15.78:0/2694197083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0009b30 con 0x7f59c80c51b0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cdb51640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 0x7f59c80be360 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c7f0 tx=0x7f59c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4004220 con 0x7f59c80c51b0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4004380 con 0x7f59c80c51b0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4003830 con 0x7f59c80c51b0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c81ea880 con 0x7f59c80c51b0 2024-02-19T14:53:51.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.600+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c8149180 con 0x7f59c80c51b0 2024-02-19T14:53:51.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.601+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c8074990 con 0x7f59c80c51b0 2024-02-19T14:53:51.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.601+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f59c4003ba0 con 0x7f59c80c51b0 2024-02-19T14:53:51.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.602+0000 7f59be7fc640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59ac06b840 0x7f59ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:51.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.602+0000 7f59cd350640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59ac06b840 0x7f59ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:51.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.602+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f59c408a3b0 con 0x7f59c80c51b0 2024-02-19T14:53:51.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.603+0000 7f59cd350640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59ac06b840 0x7f59ac06dd00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f59b800aa40 tx=0x7f59b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.606+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f59c40549f0 con 0x7f59c80c51b0 2024-02-19T14:53:51.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.848+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59c80c9730 con 0x7f59ac06b840 2024-02-19T14:53:51.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.860+0000 7f59be7fc640 1 -- 172.21.15.78:0/2694197083 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+522875 (secure 0 0 0) 0x7f59b800c040 con 0x7f59ac06b840 2024-02-19T14:53:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59ac06b840 msgr2=0x7f59ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59ac06b840 0x7f59ac06dd00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f59b800aa40 tx=0x7f59b8008040 comp rx=0 tx=0).stop 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 msgr2=0x7f59c80be360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 0x7f59c80be360 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c7f0 tx=0x7f59c400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 shutdown_connections 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59ac06b840 0x7f59ac06dd00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.864+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c59f0 0x7f59c81eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.865+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5590 0x7f59c80c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.865+0000 7f59cfdb5640 1 --2- 172.21.15.78:0/2694197083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c80c51b0 0x7f59c80be360 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.865+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 >> 172.21.15.78:0/2694197083 conn(0x7f59c80ca830 msgr2=0x7f59c80cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:51.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.865+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 shutdown_connections 2024-02-19T14:53:51.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:51.865+0000 7f59cfdb5640 1 -- 172.21.15.78:0/2694197083 wait complete. 2024-02-19T14:53:51.878 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-02-19T14:53:50.081943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222440,"kb_used_data":6272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752280,"statfs":{"total":773094113280,"available":772866334720,"internally_reserved":0,"allocated":6422528,"data_stored":3527724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000647"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730654+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:47.730654+0000","last_peered":"2024-02-19T14:53:47.730654+0000","last_clean":"2024-02-19T14:53:47.730654+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:47.730654+0000","last_undegraded":"2024-02-19T14:53:47.730654+0000","last_fullsized":"2024-02-19T14:53:47.730654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731535+0000","last_change":"2024-02-19T14:53:47.731535+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731535+0000","last_undegraded":"2024-02-19T14:53:47.731535+0000","last_fullsized":"2024-02-19T14:53:47.731535+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730461+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:47.730461+0000","last_peered":"2024-02-19T14:53:47.730461+0000","last_clean":"2024-02-19T14:53:47.730461+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:47.730461+0000","last_undegraded":"2024-02-19T14:53:47.730461+0000","last_fullsized":"2024-02-19T14:53:47.730461+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728965+0000","last_change":"2024-02-19T14:53:47.728965+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728965+0000","last_undegraded":"2024-02-19T14:53:47.728965+0000","last_fullsized":"2024-02-19T14:53:47.728965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732216+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:47.732216+0000","last_peered":"2024-02-19T14:53:47.732216+0000","last_clean":"2024-02-19T14:53:47.732216+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:47.732216+0000","last_undegraded":"2024-02-19T14:53:47.732216+0000","last_fullsized":"2024-02-19T14:53:47.732216+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726570+0000","last_change":"2024-02-19T14:53:47.726570+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726570+0000","last_undegraded":"2024-02-19T14:53:47.726570+0000","last_fullsized":"2024-02-19T14:53:47.726570+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729626+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:47.729626+0000","last_peered":"2024-02-19T14:53:47.729626+0000","last_clean":"2024-02-19T14:53:47.729626+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:47.729626+0000","last_undegraded":"2024-02-19T14:53:47.729626+0000","last_fullsized":"2024-02-19T14:53:47.729626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730942+0000","last_change":"2024-02-19T14:53:47.730942+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730942+0000","last_undegraded":"2024-02-19T14:53:47.730942+0000","last_fullsized":"2024-02-19T14:53:47.730942+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731084+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:47.731084+0000","last_peered":"2024-02-19T14:53:47.731084+0000","last_clean":"2024-02-19T14:53:47.731084+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:47.731084+0000","last_undegraded":"2024-02-19T14:53:47.731084+0000","last_fullsized":"2024-02-19T14:53:47.731084+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729130+0000","last_change":"2024-02-19T14:53:47.729130+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729130+0000","last_undegraded":"2024-02-19T14:53:47.729130+0000","last_fullsized":"2024-02-19T14:53:47.729130+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729373+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:47.729373+0000","last_peered":"2024-02-19T14:53:47.729373+0000","last_clean":"2024-02-19T14:53:47.729373+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:47.729373+0000","last_undegraded":"2024-02-19T14:53:47.729373+0000","last_fullsized":"2024-02-19T14:53:47.729373+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731517+0000","last_change":"2024-02-19T14:53:47.731517+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731517+0000","last_undegraded":"2024-02-19T14:53:47.731517+0000","last_fullsized":"2024-02-19T14:53:47.731517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731295+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:47.731295+0000","last_peered":"2024-02-19T14:53:47.731295+0000","last_clean":"2024-02-19T14:53:47.731295+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:47.731295+0000","last_undegraded":"2024-02-19T14:53:47.731295+0000","last_fullsized":"2024-02-19T14:53:47.731295+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727983+0000","last_change":"2024-02-19T14:53:47.727983+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727983+0000","last_undegraded":"2024-02-19T14:53:47.727983+0000","last_fullsized":"2024-02-19T14:53:47.727983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723628+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:47.723628+0000","last_peered":"2024-02-19T14:53:47.723628+0000","last_clean":"2024-02-19T14:53:47.723628+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:47.723628+0000","last_undegraded":"2024-02-19T14:53:47.723628+0000","last_fullsized":"2024-02-19T14:53:47.723628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733825+0000","last_change":"2024-02-19T14:53:47.733825+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733825+0000","last_undegraded":"2024-02-19T14:53:47.733825+0000","last_fullsized":"2024-02-19T14:53:47.733825+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724256+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:47.724256+0000","last_peered":"2024-02-19T14:53:47.724256+0000","last_clean":"2024-02-19T14:53:47.724256+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:47.724256+0000","last_undegraded":"2024-02-19T14:53:47.724256+0000","last_fullsized":"2024-02-19T14:53:47.724256+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:50.707228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733022+0000","last_change":"2024-02-19T14:53:47.733022+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733022+0000","last_undegraded":"2024-02-19T14:53:47.733022+0000","last_fullsized":"2024-02-19T14:53:47.733022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730312+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:47.730312+0000","last_peered":"2024-02-19T14:53:47.730312+0000","last_clean":"2024-02-19T14:53:47.730312+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:47.730312+0000","last_undegraded":"2024-02-19T14:53:47.730312+0000","last_fullsized":"2024-02-19T14:53:47.730312+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733611+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:47.733611+0000","last_peered":"2024-02-19T14:53:47.733611+0000","last_clean":"2024-02-19T14:53:47.733611+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:47.733611+0000","last_undegraded":"2024-02-19T14:53:47.733611+0000","last_fullsized":"2024-02-19T14:53:47.733611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:37.193943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726799+0000","last_change":"2024-02-19T14:53:47.726799+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726799+0000","last_undegraded":"2024-02-19T14:53:47.726799+0000","last_fullsized":"2024-02-19T14:53:47.726799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724742+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:47.724742+0000","last_peered":"2024-02-19T14:53:47.724742+0000","last_clean":"2024-02-19T14:53:47.724742+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:47.724742+0000","last_undegraded":"2024-02-19T14:53:47.724742+0000","last_fullsized":"2024-02-19T14:53:47.724742+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:01.639745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726754+0000","last_change":"2024-02-19T14:53:47.726754+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726754+0000","last_undegraded":"2024-02-19T14:53:47.726754+0000","last_fullsized":"2024-02-19T14:53:47.726754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729060+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:47.729060+0000","last_peered":"2024-02-19T14:53:47.729060+0000","last_clean":"2024-02-19T14:53:47.729060+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:47.729060+0000","last_undegraded":"2024-02-19T14:53:47.729060+0000","last_fullsized":"2024-02-19T14:53:47.729060+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731147+0000","last_change":"2024-02-19T14:53:47.731147+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731147+0000","last_undegraded":"2024-02-19T14:53:47.731147+0000","last_fullsized":"2024-02-19T14:53:47.731147+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739485+0000","last_change":"2024-02-19T14:53:42.136272+0000","last_active":"2024-02-19T14:53:47.739485+0000","last_peered":"2024-02-19T14:53:47.739485+0000","last_clean":"2024-02-19T14:53:47.739485+0000","last_became_active":"2024-02-19T14:53:42.135998+0000","last_became_peered":"2024-02-19T14:53:42.135998+0000","last_unstale":"2024-02-19T14:53:47.739485+0000","last_undegraded":"2024-02-19T14:53:47.739485+0000","last_fullsized":"2024-02-19T14:53:47.739485+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734360+0000","last_change":"2024-02-19T14:53:47.734360+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734360+0000","last_undegraded":"2024-02-19T14:53:47.734360+0000","last_fullsized":"2024-02-19T14:53:47.734360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729068+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:47.729068+0000","last_peered":"2024-02-19T14:53:47.729068+0000","last_clean":"2024-02-19T14:53:47.729068+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:47.729068+0000","last_undegraded":"2024-02-19T14:53:47.729068+0000","last_fullsized":"2024-02-19T14:53:47.729068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726289+0000","last_change":"2024-02-19T14:53:47.726289+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726289+0000","last_undegraded":"2024-02-19T14:53:47.726289+0000","last_fullsized":"2024-02-19T14:53:47.726289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727510+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:47.727510+0000","last_peered":"2024-02-19T14:53:47.727510+0000","last_clean":"2024-02-19T14:53:47.727510+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:47.727510+0000","last_undegraded":"2024-02-19T14:53:47.727510+0000","last_fullsized":"2024-02-19T14:53:47.727510+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731901+0000","last_change":"2024-02-19T14:53:47.731901+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731901+0000","last_undegraded":"2024-02-19T14:53:47.731901+0000","last_fullsized":"2024-02-19T14:53:47.731901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734311+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:47.734311+0000","last_peered":"2024-02-19T14:53:47.734311+0000","last_clean":"2024-02-19T14:53:47.734311+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:47.734311+0000","last_undegraded":"2024-02-19T14:53:47.734311+0000","last_fullsized":"2024-02-19T14:53:47.734311+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726614+0000","last_change":"2024-02-19T14:53:47.726614+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726614+0000","last_undegraded":"2024-02-19T14:53:47.726614+0000","last_fullsized":"2024-02-19T14:53:47.726614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730365+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:47.730365+0000","last_peered":"2024-02-19T14:53:47.730365+0000","last_clean":"2024-02-19T14:53:47.730365+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:47.730365+0000","last_undegraded":"2024-02-19T14:53:47.730365+0000","last_fullsized":"2024-02-19T14:53:47.730365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728581+0000","last_change":"2024-02-19T14:53:47.728581+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728581+0000","last_undegraded":"2024-02-19T14:53:47.728581+0000","last_fullsized":"2024-02-19T14:53:47.728581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732357+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:47.732357+0000","last_peered":"2024-02-19T14:53:47.732357+0000","last_clean":"2024-02-19T14:53:47.732357+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:47.732357+0000","last_undegraded":"2024-02-19T14:53:47.732357+0000","last_fullsized":"2024-02-19T14:53:47.732357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735616+0000","last_change":"2024-02-19T14:53:47.735616+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735616+0000","last_undegraded":"2024-02-19T14:53:47.735616+0000","last_fullsized":"2024-02-19T14:53:47.735616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737614+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:47.737614+0000","last_peered":"2024-02-19T14:53:47.737614+0000","last_clean":"2024-02-19T14:53:47.737614+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:47.737614+0000","last_undegraded":"2024-02-19T14:53:47.737614+0000","last_fullsized":"2024-02-19T14:53:47.737614+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726307+0000","last_change":"2024-02-19T14:53:47.726307+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726307+0000","last_undegraded":"2024-02-19T14:53:47.726307+0000","last_fullsized":"2024-02-19T14:53:47.726307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739540+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:47.739540+0000","last_peered":"2024-02-19T14:53:47.739540+0000","last_clean":"2024-02-19T14:53:47.739540+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:47.739540+0000","last_undegraded":"2024-02-19T14:53:47.739540+0000","last_fullsized":"2024-02-19T14:53:47.739540+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725966+0000","last_change":"2024-02-19T14:53:47.725966+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725966+0000","last_undegraded":"2024-02-19T14:53:47.725966+0000","last_fullsized":"2024-02-19T14:53:47.725966+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723966+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:47.723966+0000","last_peered":"2024-02-19T14:53:47.723966+0000","last_clean":"2024-02-19T14:53:47.723966+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:47.723966+0000","last_undegraded":"2024-02-19T14:53:47.723966+0000","last_fullsized":"2024-02-19T14:53:47.723966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:59.053283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729507+0000","last_change":"2024-02-19T14:53:47.729507+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729507+0000","last_undegraded":"2024-02-19T14:53:47.729507+0000","last_fullsized":"2024-02-19T14:53:47.729507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732727+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:47.732727+0000","last_peered":"2024-02-19T14:53:47.732727+0000","last_clean":"2024-02-19T14:53:47.732727+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:47.732727+0000","last_undegraded":"2024-02-19T14:53:47.732727+0000","last_fullsized":"2024-02-19T14:53:47.732727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:38:18.934930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726592+0000","last_change":"2024-02-19T14:53:47.726592+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726592+0000","last_undegraded":"2024-02-19T14:53:47.726592+0000","last_fullsized":"2024-02-19T14:53:47.726592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730738+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:47.730738+0000","last_peered":"2024-02-19T14:53:47.730738+0000","last_clean":"2024-02-19T14:53:47.730738+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:47.730738+0000","last_undegraded":"2024-02-19T14:53:47.730738+0000","last_fullsized":"2024-02-19T14:53:47.730738+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734650+0000","last_change":"2024-02-19T14:53:47.734650+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734650+0000","last_undegraded":"2024-02-19T14:53:47.734650+0000","last_fullsized":"2024-02-19T14:53:47.734650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734454+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:47.734454+0000","last_peered":"2024-02-19T14:53:47.734454+0000","last_clean":"2024-02-19T14:53:47.734454+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:47.734454+0000","last_undegraded":"2024-02-19T14:53:47.734454+0000","last_fullsized":"2024-02-19T14:53:47.734454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729138+0000","last_change":"2024-02-19T14:53:47.729138+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729138+0000","last_undegraded":"2024-02-19T14:53:47.729138+0000","last_fullsized":"2024-02-19T14:53:47.729138+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734486+0000","last_change":"2024-02-19T14:53:47.734486+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734486+0000","last_undegraded":"2024-02-19T14:53:47.734486+0000","last_fullsized":"2024-02-19T14:53:47.734486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729945+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:47.729945+0000","last_peered":"2024-02-19T14:53:47.729945+0000","last_clean":"2024-02-19T14:53:47.729945+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:47.729945+0000","last_undegraded":"2024-02-19T14:53:47.729945+0000","last_fullsized":"2024-02-19T14:53:47.729945+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733413+0000","last_change":"2024-02-19T14:53:47.733413+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733413+0000","last_undegraded":"2024-02-19T14:53:47.733413+0000","last_fullsized":"2024-02-19T14:53:47.733413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737727+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:47.737727+0000","last_peered":"2024-02-19T14:53:47.737727+0000","last_clean":"2024-02-19T14:53:47.737727+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:47.737727+0000","last_undegraded":"2024-02-19T14:53:47.737727+0000","last_fullsized":"2024-02-19T14:53:47.737727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729994+0000","last_change":"2024-02-19T14:53:47.729994+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729994+0000","last_undegraded":"2024-02-19T14:53:47.729994+0000","last_fullsized":"2024-02-19T14:53:47.729994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732428+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:47.732428+0000","last_peered":"2024-02-19T14:53:47.732428+0000","last_clean":"2024-02-19T14:53:47.732428+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:47.732428+0000","last_undegraded":"2024-02-19T14:53:47.732428+0000","last_fullsized":"2024-02-19T14:53:47.732428+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728586+0000","last_change":"2024-02-19T14:53:47.728586+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728586+0000","last_undegraded":"2024-02-19T14:53:47.728586+0000","last_fullsized":"2024-02-19T14:53:47.728586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729908+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:47.729908+0000","last_peered":"2024-02-19T14:53:47.729908+0000","last_clean":"2024-02-19T14:53:47.729908+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:47.729908+0000","last_undegraded":"2024-02-19T14:53:47.729908+0000","last_fullsized":"2024-02-19T14:53:47.729908+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726173+0000","last_change":"2024-02-19T14:53:47.726173+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726173+0000","last_undegraded":"2024-02-19T14:53:47.726173+0000","last_fullsized":"2024-02-19T14:53:47.726173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731441+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:47.731441+0000","last_peered":"2024-02-19T14:53:47.731441+0000","last_clean":"2024-02-19T14:53:47.731441+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:47.731441+0000","last_undegraded":"2024-02-19T14:53:47.731441+0000","last_fullsized":"2024-02-19T14:53:47.731441+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727599+0000","last_change":"2024-02-19T14:53:47.727599+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727599+0000","last_undegraded":"2024-02-19T14:53:47.727599+0000","last_fullsized":"2024-02-19T14:53:47.727599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734161+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:47.734161+0000","last_peered":"2024-02-19T14:53:47.734161+0000","last_clean":"2024-02-19T14:53:47.734161+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:47.734161+0000","last_undegraded":"2024-02-19T14:53:47.734161+0000","last_fullsized":"2024-02-19T14:53:47.734161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735808+0000","last_change":"2024-02-19T14:53:47.735808+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735808+0000","last_undegraded":"2024-02-19T14:53:47.735808+0000","last_fullsized":"2024-02-19T14:53:47.735808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746858+0000","last_change":"2024-02-19T14:53:47.740946+0000","last_active":"2024-02-19T14:53:47.746858+0000","last_peered":"2024-02-19T14:53:47.746858+0000","last_clean":"2024-02-19T14:53:47.746858+0000","last_became_active":"2024-02-19T14:53:47.740550+0000","last_became_peered":"2024-02-19T14:53:47.740550+0000","last_unstale":"2024-02-19T14:53:47.746858+0000","last_undegraded":"2024-02-19T14:53:47.746858+0000","last_fullsized":"2024-02-19T14:53:47.746858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:09.101397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728386+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:47.728386+0000","last_peered":"2024-02-19T14:53:47.728386+0000","last_clean":"2024-02-19T14:53:47.728386+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:47.728386+0000","last_undegraded":"2024-02-19T14:53:47.728386+0000","last_fullsized":"2024-02-19T14:53:47.728386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":26,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739615+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:47.739615+0000","last_peered":"2024-02-19T14:53:47.739615+0000","last_clean":"2024-02-19T14:53:47.739615+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:47.739615+0000","last_undegraded":"2024-02-19T14:53:47.739615+0000","last_fullsized":"2024-02-19T14:53:47.739615+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727391+0000","last_change":"2024-02-19T14:53:47.727391+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727391+0000","last_undegraded":"2024-02-19T14:53:47.727391+0000","last_fullsized":"2024-02-19T14:53:47.727391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746433+0000","last_change":"2024-02-19T14:53:47.742941+0000","last_active":"2024-02-19T14:53:47.746433+0000","last_peered":"2024-02-19T14:53:47.746433+0000","last_clean":"2024-02-19T14:53:47.746433+0000","last_became_active":"2024-02-19T14:53:47.742590+0000","last_became_peered":"2024-02-19T14:53:47.742590+0000","last_unstale":"2024-02-19T14:53:47.746433+0000","last_undegraded":"2024-02-19T14:53:47.746433+0000","last_fullsized":"2024-02-19T14:53:47.746433+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:35.779780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729200+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:47.729200+0000","last_peered":"2024-02-19T14:53:47.729200+0000","last_clean":"2024-02-19T14:53:47.729200+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:47.729200+0000","last_undegraded":"2024-02-19T14:53:47.729200+0000","last_fullsized":"2024-02-19T14:53:47.729200+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730410+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:47.730410+0000","last_peered":"2024-02-19T14:53:47.730410+0000","last_clean":"2024-02-19T14:53:47.730410+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:47.730410+0000","last_undegraded":"2024-02-19T14:53:47.730410+0000","last_fullsized":"2024-02-19T14:53:47.730410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729346+0000","last_change":"2024-02-19T14:53:47.729346+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729346+0000","last_undegraded":"2024-02-19T14:53:47.729346+0000","last_fullsized":"2024-02-19T14:53:47.729346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747720+0000","last_change":"2024-02-19T14:53:47.746326+0000","last_active":"2024-02-19T14:53:47.747720+0000","last_peered":"2024-02-19T14:53:47.747720+0000","last_clean":"2024-02-19T14:53:47.747720+0000","last_became_active":"2024-02-19T14:53:47.745753+0000","last_became_peered":"2024-02-19T14:53:47.745753+0000","last_unstale":"2024-02-19T14:53:47.747720+0000","last_undegraded":"2024-02-19T14:53:47.747720+0000","last_fullsized":"2024-02-19T14:53:47.747720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739624+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:47.739624+0000","last_peered":"2024-02-19T14:53:47.739624+0000","last_clean":"2024-02-19T14:53:47.739624+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:47.739624+0000","last_undegraded":"2024-02-19T14:53:47.739624+0000","last_fullsized":"2024-02-19T14:53:47.739624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730034+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:47.730034+0000","last_peered":"2024-02-19T14:53:47.730034+0000","last_clean":"2024-02-19T14:53:47.730034+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:47.730034+0000","last_undegraded":"2024-02-19T14:53:47.730034+0000","last_fullsized":"2024-02-19T14:53:47.730034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726858+0000","last_change":"2024-02-19T14:53:47.726858+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726858+0000","last_undegraded":"2024-02-19T14:53:47.726858+0000","last_fullsized":"2024-02-19T14:53:47.726858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747935+0000","last_change":"2024-02-19T14:53:47.740910+0000","last_active":"2024-02-19T14:53:47.747935+0000","last_peered":"2024-02-19T14:53:47.747935+0000","last_clean":"2024-02-19T14:53:47.747935+0000","last_became_active":"2024-02-19T14:53:47.740626+0000","last_became_peered":"2024-02-19T14:53:47.740626+0000","last_unstale":"2024-02-19T14:53:47.747935+0000","last_undegraded":"2024-02-19T14:53:47.747935+0000","last_fullsized":"2024-02-19T14:53:47.747935+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723809+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:47.723809+0000","last_peered":"2024-02-19T14:53:47.723809+0000","last_clean":"2024-02-19T14:53:47.723809+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:47.723809+0000","last_undegraded":"2024-02-19T14:53:47.723809+0000","last_fullsized":"2024-02-19T14:53:47.723809+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:36.026355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733496+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:47.733496+0000","last_peered":"2024-02-19T14:53:47.733496+0000","last_clean":"2024-02-19T14:53:47.733496+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:47.733496+0000","last_undegraded":"2024-02-19T14:53:47.733496+0000","last_fullsized":"2024-02-19T14:53:47.733496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732000+0000","last_change":"2024-02-19T14:53:47.732000+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732000+0000","last_undegraded":"2024-02-19T14:53:47.732000+0000","last_fullsized":"2024-02-19T14:53:47.732000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747004+0000","last_change":"2024-02-19T14:53:47.745151+0000","last_active":"2024-02-19T14:53:47.747004+0000","last_peered":"2024-02-19T14:53:47.747004+0000","last_clean":"2024-02-19T14:53:47.747004+0000","last_became_active":"2024-02-19T14:53:47.744887+0000","last_became_peered":"2024-02-19T14:53:47.744887+0000","last_unstale":"2024-02-19T14:53:47.747004+0000","last_undegraded":"2024-02-19T14:53:47.747004+0000","last_fullsized":"2024-02-19T14:53:47.747004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732432+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:47.732432+0000","last_peered":"2024-02-19T14:53:47.732432+0000","last_clean":"2024-02-19T14:53:47.732432+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:47.732432+0000","last_undegraded":"2024-02-19T14:53:47.732432+0000","last_fullsized":"2024-02-19T14:53:47.732432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732599+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:47.732599+0000","last_peered":"2024-02-19T14:53:47.732599+0000","last_clean":"2024-02-19T14:53:47.732599+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:47.732599+0000","last_undegraded":"2024-02-19T14:53:47.732599+0000","last_fullsized":"2024-02-19T14:53:47.732599+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.736522+0000","last_change":"2024-02-19T14:53:47.736522+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.736522+0000","last_undegraded":"2024-02-19T14:53:47.736522+0000","last_fullsized":"2024-02-19T14:53:47.736522+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746094+0000","last_change":"2024-02-19T14:53:47.743381+0000","last_active":"2024-02-19T14:53:47.746094+0000","last_peered":"2024-02-19T14:53:47.746094+0000","last_clean":"2024-02-19T14:53:47.746094+0000","last_became_active":"2024-02-19T14:53:47.743135+0000","last_became_peered":"2024-02-19T14:53:47.743135+0000","last_unstale":"2024-02-19T14:53:47.746094+0000","last_undegraded":"2024-02-19T14:53:47.746094+0000","last_fullsized":"2024-02-19T14:53:47.746094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:30.540219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730104+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:47.730104+0000","last_peered":"2024-02-19T14:53:47.730104+0000","last_clean":"2024-02-19T14:53:47.730104+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:47.730104+0000","last_undegraded":"2024-02-19T14:53:47.730104+0000","last_fullsized":"2024-02-19T14:53:47.730104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728795+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:47.728795+0000","last_peered":"2024-02-19T14:53:47.728795+0000","last_clean":"2024-02-19T14:53:47.728795+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:47.728795+0000","last_undegraded":"2024-02-19T14:53:47.728795+0000","last_fullsized":"2024-02-19T14:53:47.728795+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728755+0000","last_change":"2024-02-19T14:53:47.728755+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728755+0000","last_undegraded":"2024-02-19T14:53:47.728755+0000","last_fullsized":"2024-02-19T14:53:47.728755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746985+0000","last_change":"2024-02-19T14:53:47.744368+0000","last_active":"2024-02-19T14:53:47.746985+0000","last_peered":"2024-02-19T14:53:47.746985+0000","last_clean":"2024-02-19T14:53:47.746985+0000","last_became_active":"2024-02-19T14:53:47.743379+0000","last_became_peered":"2024-02-19T14:53:47.743379+0000","last_unstale":"2024-02-19T14:53:47.746985+0000","last_undegraded":"2024-02-19T14:53:47.746985+0000","last_fullsized":"2024-02-19T14:53:47.746985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724152+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:47.724152+0000","last_peered":"2024-02-19T14:53:47.724152+0000","last_clean":"2024-02-19T14:53:47.724152+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:47.724152+0000","last_undegraded":"2024-02-19T14:53:47.724152+0000","last_fullsized":"2024-02-19T14:53:47.724152+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:41.307928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737668+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:47.737668+0000","last_peered":"2024-02-19T14:53:47.737668+0000","last_clean":"2024-02-19T14:53:47.737668+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:47.737668+0000","last_undegraded":"2024-02-19T14:53:47.737668+0000","last_fullsized":"2024-02-19T14:53:47.737668+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726342+0000","last_change":"2024-02-19T14:53:47.726342+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726342+0000","last_undegraded":"2024-02-19T14:53:47.726342+0000","last_fullsized":"2024-02-19T14:53:47.726342+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747509+0000","last_change":"2024-02-19T14:53:47.744422+0000","last_active":"2024-02-19T14:53:47.747509+0000","last_peered":"2024-02-19T14:53:47.747509+0000","last_clean":"2024-02-19T14:53:47.747509+0000","last_became_active":"2024-02-19T14:53:47.744122+0000","last_became_peered":"2024-02-19T14:53:47.744122+0000","last_unstale":"2024-02-19T14:53:47.747509+0000","last_undegraded":"2024-02-19T14:53:47.747509+0000","last_fullsized":"2024-02-19T14:53:47.747509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:50.594221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730639+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:47.730639+0000","last_peered":"2024-02-19T14:53:47.730639+0000","last_clean":"2024-02-19T14:53:47.730639+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:47.730639+0000","last_undegraded":"2024-02-19T14:53:47.730639+0000","last_fullsized":"2024-02-19T14:53:47.730639+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730829+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:47.730829+0000","last_peered":"2024-02-19T14:53:47.730829+0000","last_clean":"2024-02-19T14:53:47.730829+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:47.730829+0000","last_undegraded":"2024-02-19T14:53:47.730829+0000","last_fullsized":"2024-02-19T14:53:47.730829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728855+0000","last_change":"2024-02-19T14:53:47.728855+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728855+0000","last_undegraded":"2024-02-19T14:53:47.728855+0000","last_fullsized":"2024-02-19T14:53:47.728855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746360+0000","last_change":"2024-02-19T14:53:47.745353+0000","last_active":"2024-02-19T14:53:47.746360+0000","last_peered":"2024-02-19T14:53:47.746360+0000","last_clean":"2024-02-19T14:53:47.746360+0000","last_became_active":"2024-02-19T14:53:47.745008+0000","last_became_peered":"2024-02-19T14:53:47.745008+0000","last_unstale":"2024-02-19T14:53:47.746360+0000","last_undegraded":"2024-02-19T14:53:47.746360+0000","last_fullsized":"2024-02-19T14:53:47.746360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:29.660730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729669+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:47.729669+0000","last_peered":"2024-02-19T14:53:47.729669+0000","last_clean":"2024-02-19T14:53:47.729669+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:47.729669+0000","last_undegraded":"2024-02-19T14:53:47.729669+0000","last_fullsized":"2024-02-19T14:53:47.729669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730746+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:47.730746+0000","last_peered":"2024-02-19T14:53:47.730746+0000","last_clean":"2024-02-19T14:53:47.730746+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:47.730746+0000","last_undegraded":"2024-02-19T14:53:47.730746+0000","last_fullsized":"2024-02-19T14:53:47.730746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726359+0000","last_change":"2024-02-19T14:53:47.726359+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726359+0000","last_undegraded":"2024-02-19T14:53:47.726359+0000","last_fullsized":"2024-02-19T14:53:47.726359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746332+0000","last_change":"2024-02-19T14:53:47.742114+0000","last_active":"2024-02-19T14:53:47.746332+0000","last_peered":"2024-02-19T14:53:47.746332+0000","last_clean":"2024-02-19T14:53:47.746332+0000","last_became_active":"2024-02-19T14:53:47.741840+0000","last_became_peered":"2024-02-19T14:53:47.741840+0000","last_unstale":"2024-02-19T14:53:47.746332+0000","last_undegraded":"2024-02-19T14:53:47.746332+0000","last_fullsized":"2024-02-19T14:53:47.746332+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:20.795346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727709+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:47.727709+0000","last_peered":"2024-02-19T14:53:47.727709+0000","last_clean":"2024-02-19T14:53:47.727709+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:47.727709+0000","last_undegraded":"2024-02-19T14:53:47.727709+0000","last_fullsized":"2024-02-19T14:53:47.727709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.736179+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:47.736179+0000","last_peered":"2024-02-19T14:53:47.736179+0000","last_clean":"2024-02-19T14:53:47.736179+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:47.736179+0000","last_undegraded":"2024-02-19T14:53:47.736179+0000","last_fullsized":"2024-02-19T14:53:47.736179+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731814+0000","last_change":"2024-02-19T14:53:47.731814+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731814+0000","last_undegraded":"2024-02-19T14:53:47.731814+0000","last_fullsized":"2024-02-19T14:53:47.731814+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737453+0000","last_change":"2024-02-19T14:53:47.734753+0000","last_active":"2024-02-19T14:53:47.737453+0000","last_peered":"2024-02-19T14:53:47.737453+0000","last_clean":"2024-02-19T14:53:47.737453+0000","last_became_active":"2024-02-19T14:53:47.734515+0000","last_became_peered":"2024-02-19T14:53:47.734515+0000","last_unstale":"2024-02-19T14:53:47.737453+0000","last_undegraded":"2024-02-19T14:53:47.737453+0000","last_fullsized":"2024-02-19T14:53:47.737453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:02:56.724991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730223+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:47.730223+0000","last_peered":"2024-02-19T14:53:47.730223+0000","last_clean":"2024-02-19T14:53:47.730223+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:47.730223+0000","last_undegraded":"2024-02-19T14:53:47.730223+0000","last_fullsized":"2024-02-19T14:53:47.730223+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":25,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731281+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:47.731281+0000","last_peered":"2024-02-19T14:53:47.731281+0000","last_clean":"2024-02-19T14:53:47.731281+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:47.731281+0000","last_undegraded":"2024-02-19T14:53:47.731281+0000","last_fullsized":"2024-02-19T14:53:47.731281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727760+0000","last_change":"2024-02-19T14:53:47.727760+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727760+0000","last_undegraded":"2024-02-19T14:53:47.727760+0000","last_fullsized":"2024-02-19T14:53:47.727760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737793+0000","last_change":"2024-02-19T14:53:47.735759+0000","last_active":"2024-02-19T14:53:47.737793+0000","last_peered":"2024-02-19T14:53:47.737793+0000","last_clean":"2024-02-19T14:53:47.737793+0000","last_became_active":"2024-02-19T14:53:47.735503+0000","last_became_peered":"2024-02-19T14:53:47.735503+0000","last_unstale":"2024-02-19T14:53:47.737793+0000","last_undegraded":"2024-02-19T14:53:47.737793+0000","last_fullsized":"2024-02-19T14:53:47.737793+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:35.189687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731195+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:47.731195+0000","last_peered":"2024-02-19T14:53:47.731195+0000","last_clean":"2024-02-19T14:53:47.731195+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:47.731195+0000","last_undegraded":"2024-02-19T14:53:47.731195+0000","last_fullsized":"2024-02-19T14:53:47.731195+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":28,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724746+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:47.724746+0000","last_peered":"2024-02-19T14:53:47.724746+0000","last_clean":"2024-02-19T14:53:47.724746+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:47.724746+0000","last_undegraded":"2024-02-19T14:53:47.724746+0000","last_fullsized":"2024-02-19T14:53:47.724746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731071+0000","last_change":"2024-02-19T14:53:47.731071+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731071+0000","last_undegraded":"2024-02-19T14:53:47.731071+0000","last_fullsized":"2024-02-19T14:53:47.731071+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746268+0000","last_change":"2024-02-19T14:53:47.742801+0000","last_active":"2024-02-19T14:53:47.746268+0000","last_peered":"2024-02-19T14:53:47.746268+0000","last_clean":"2024-02-19T14:53:47.746268+0000","last_became_active":"2024-02-19T14:53:47.742534+0000","last_became_peered":"2024-02-19T14:53:47.742534+0000","last_unstale":"2024-02-19T14:53:47.746268+0000","last_undegraded":"2024-02-19T14:53:47.746268+0000","last_fullsized":"2024-02-19T14:53:47.746268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:59.453875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733273+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:47.733273+0000","last_peered":"2024-02-19T14:53:47.733273+0000","last_clean":"2024-02-19T14:53:47.733273+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:47.733273+0000","last_undegraded":"2024-02-19T14:53:47.733273+0000","last_fullsized":"2024-02-19T14:53:47.733273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732699+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:47.732699+0000","last_peered":"2024-02-19T14:53:47.732699+0000","last_clean":"2024-02-19T14:53:47.732699+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:47.732699+0000","last_undegraded":"2024-02-19T14:53:47.732699+0000","last_fullsized":"2024-02-19T14:53:47.732699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726717+0000","last_change":"2024-02-19T14:53:47.726717+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726717+0000","last_undegraded":"2024-02-19T14:53:47.726717+0000","last_fullsized":"2024-02-19T14:53:47.726717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748938+0000","last_change":"2024-02-19T14:53:47.744212+0000","last_active":"2024-02-19T14:53:47.748938+0000","last_peered":"2024-02-19T14:53:47.748938+0000","last_clean":"2024-02-19T14:53:47.748938+0000","last_became_active":"2024-02-19T14:53:47.743916+0000","last_became_peered":"2024-02-19T14:53:47.743916+0000","last_unstale":"2024-02-19T14:53:47.748938+0000","last_undegraded":"2024-02-19T14:53:47.748938+0000","last_fullsized":"2024-02-19T14:53:47.748938+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732440+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:47.732440+0000","last_peered":"2024-02-19T14:53:47.732440+0000","last_clean":"2024-02-19T14:53:47.732440+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:47.732440+0000","last_undegraded":"2024-02-19T14:53:47.732440+0000","last_fullsized":"2024-02-19T14:53:47.732440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739733+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:47.739733+0000","last_peered":"2024-02-19T14:53:47.739733+0000","last_clean":"2024-02-19T14:53:47.739733+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:47.739733+0000","last_undegraded":"2024-02-19T14:53:47.739733+0000","last_fullsized":"2024-02-19T14:53:47.739733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731538+0000","last_change":"2024-02-19T14:53:47.731538+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731538+0000","last_undegraded":"2024-02-19T14:53:47.731538+0000","last_fullsized":"2024-02-19T14:53:47.731538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.744120+0000","last_change":"2024-02-19T14:53:47.741970+0000","last_active":"2024-02-19T14:53:47.744120+0000","last_peered":"2024-02-19T14:53:47.744120+0000","last_clean":"2024-02-19T14:53:47.744120+0000","last_became_active":"2024-02-19T14:53:47.741707+0000","last_became_peered":"2024-02-19T14:53:47.741707+0000","last_unstale":"2024-02-19T14:53:47.744120+0000","last_undegraded":"2024-02-19T14:53:47.744120+0000","last_fullsized":"2024-02-19T14:53:47.744120+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:26.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724087+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:47.724087+0000","last_peered":"2024-02-19T14:53:47.724087+0000","last_clean":"2024-02-19T14:53:47.724087+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:47.724087+0000","last_undegraded":"2024-02-19T14:53:47.724087+0000","last_fullsized":"2024-02-19T14:53:47.724087+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729186+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:47.729186+0000","last_peered":"2024-02-19T14:53:47.729186+0000","last_clean":"2024-02-19T14:53:47.729186+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:47.729186+0000","last_undegraded":"2024-02-19T14:53:47.729186+0000","last_fullsized":"2024-02-19T14:53:47.729186+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734244+0000","last_change":"2024-02-19T14:53:47.734244+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734244+0000","last_undegraded":"2024-02-19T14:53:47.734244+0000","last_fullsized":"2024-02-19T14:53:47.734244+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743671+0000","last_change":"2024-02-19T14:53:47.742140+0000","last_active":"2024-02-19T14:53:47.743671+0000","last_peered":"2024-02-19T14:53:47.743671+0000","last_clean":"2024-02-19T14:53:47.743671+0000","last_became_active":"2024-02-19T14:53:47.741838+0000","last_became_peered":"2024-02-19T14:53:47.741838+0000","last_unstale":"2024-02-19T14:53:47.743671+0000","last_undegraded":"2024-02-19T14:53:47.743671+0000","last_fullsized":"2024-02-19T14:53:47.743671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:55.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737818+0000","last_change":"2024-02-19T14:53:41.706063+0000","last_active":"2024-02-19T14:53:47.737818+0000","last_peered":"2024-02-19T14:53:47.737818+0000","last_clean":"2024-02-19T14:53:47.737818+0000","last_became_active":"2024-02-19T14:53:41.705766+0000","last_became_peered":"2024-02-19T14:53:41.705766+0000","last_unstale":"2024-02-19T14:53:47.737818+0000","last_undegraded":"2024-02-19T14:53:47.737818+0000","last_fullsized":"2024-02-19T14:53:47.737818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":30,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723730+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:47.723730+0000","last_peered":"2024-02-19T14:53:47.723730+0000","last_clean":"2024-02-19T14:53:47.723730+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:47.723730+0000","last_undegraded":"2024-02-19T14:53:47.723730+0000","last_fullsized":"2024-02-19T14:53:47.723730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727027+0000","last_change":"2024-02-19T14:53:47.727027+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727027+0000","last_undegraded":"2024-02-19T14:53:47.727027+0000","last_fullsized":"2024-02-19T14:53:47.727027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743702+0000","last_change":"2024-02-19T14:53:47.742196+0000","last_active":"2024-02-19T14:53:47.743702+0000","last_peered":"2024-02-19T14:53:47.743702+0000","last_clean":"2024-02-19T14:53:47.743702+0000","last_became_active":"2024-02-19T14:53:47.741852+0000","last_became_peered":"2024-02-19T14:53:47.741852+0000","last_unstale":"2024-02-19T14:53:47.743702+0000","last_undegraded":"2024-02-19T14:53:47.743702+0000","last_fullsized":"2024-02-19T14:53:47.743702+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:31.942230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724625+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:47.724625+0000","last_peered":"2024-02-19T14:53:47.724625+0000","last_clean":"2024-02-19T14:53:47.724625+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:47.724625+0000","last_undegraded":"2024-02-19T14:53:47.724625+0000","last_fullsized":"2024-02-19T14:53:47.724625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734529+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:47.734529+0000","last_peered":"2024-02-19T14:53:47.734529+0000","last_clean":"2024-02-19T14:53:47.734529+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:47.734529+0000","last_undegraded":"2024-02-19T14:53:47.734529+0000","last_fullsized":"2024-02-19T14:53:47.734529+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730482+0000","last_change":"2024-02-19T14:53:47.730482+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730482+0000","last_undegraded":"2024-02-19T14:53:47.730482+0000","last_fullsized":"2024-02-19T14:53:47.730482+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.749226+0000","last_change":"2024-02-19T14:53:47.745519+0000","last_active":"2024-02-19T14:53:47.749226+0000","last_peered":"2024-02-19T14:53:47.749226+0000","last_clean":"2024-02-19T14:53:47.749226+0000","last_became_active":"2024-02-19T14:53:47.745263+0000","last_became_peered":"2024-02-19T14:53:47.745263+0000","last_unstale":"2024-02-19T14:53:47.749226+0000","last_undegraded":"2024-02-19T14:53:47.749226+0000","last_fullsized":"2024-02-19T14:53:47.749226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:51.861192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728027+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:47.728027+0000","last_peered":"2024-02-19T14:53:47.728027+0000","last_clean":"2024-02-19T14:53:47.728027+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:47.728027+0000","last_undegraded":"2024-02-19T14:53:47.728027+0000","last_fullsized":"2024-02-19T14:53:47.728027+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733299+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:47.733299+0000","last_peered":"2024-02-19T14:53:47.733299+0000","last_clean":"2024-02-19T14:53:47.733299+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:47.733299+0000","last_undegraded":"2024-02-19T14:53:47.733299+0000","last_fullsized":"2024-02-19T14:53:47.733299+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728440+0000","last_change":"2024-02-19T14:53:47.728440+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728440+0000","last_undegraded":"2024-02-19T14:53:47.728440+0000","last_fullsized":"2024-02-19T14:53:47.728440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.741263+0000","last_change":"2024-02-19T14:53:47.736640+0000","last_active":"2024-02-19T14:53:47.741263+0000","last_peered":"2024-02-19T14:53:47.741263+0000","last_clean":"2024-02-19T14:53:47.741263+0000","last_became_active":"2024-02-19T14:53:47.735150+0000","last_became_peered":"2024-02-19T14:53:47.735150+0000","last_unstale":"2024-02-19T14:53:47.741263+0000","last_undegraded":"2024-02-19T14:53:47.741263+0000","last_fullsized":"2024-02-19T14:53:47.741263+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:02.655336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728667+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:47.728667+0000","last_peered":"2024-02-19T14:53:47.728667+0000","last_clean":"2024-02-19T14:53:47.728667+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:47.728667+0000","last_undegraded":"2024-02-19T14:53:47.728667+0000","last_fullsized":"2024-02-19T14:53:47.728667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737687+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:47.737687+0000","last_peered":"2024-02-19T14:53:47.737687+0000","last_clean":"2024-02-19T14:53:47.737687+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:47.737687+0000","last_undegraded":"2024-02-19T14:53:47.737687+0000","last_fullsized":"2024-02-19T14:53:47.737687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730447+0000","last_change":"2024-02-19T14:53:47.730447+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730447+0000","last_undegraded":"2024-02-19T14:53:47.730447+0000","last_fullsized":"2024-02-19T14:53:47.730447+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734187+0000","last_change":"2024-02-19T14:53:47.730313+0000","last_active":"2024-02-19T14:53:47.734187+0000","last_peered":"2024-02-19T14:53:47.734187+0000","last_clean":"2024-02-19T14:53:47.734187+0000","last_became_active":"2024-02-19T14:53:47.730053+0000","last_became_peered":"2024-02-19T14:53:47.730053+0000","last_unstale":"2024-02-19T14:53:47.734187+0000","last_undegraded":"2024-02-19T14:53:47.734187+0000","last_fullsized":"2024-02-19T14:53:47.734187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729153+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:47.729153+0000","last_peered":"2024-02-19T14:53:47.729153+0000","last_clean":"2024-02-19T14:53:47.729153+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:47.729153+0000","last_undegraded":"2024-02-19T14:53:47.729153+0000","last_fullsized":"2024-02-19T14:53:47.729153+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737838+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:47.737838+0000","last_peered":"2024-02-19T14:53:47.737838+0000","last_clean":"2024-02-19T14:53:47.737838+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:47.737838+0000","last_undegraded":"2024-02-19T14:53:47.737838+0000","last_fullsized":"2024-02-19T14:53:47.737838+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728393+0000","last_change":"2024-02-19T14:53:47.728393+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728393+0000","last_undegraded":"2024-02-19T14:53:47.728393+0000","last_fullsized":"2024-02-19T14:53:47.728393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746290+0000","last_change":"2024-02-19T14:53:47.742841+0000","last_active":"2024-02-19T14:53:47.746290+0000","last_peered":"2024-02-19T14:53:47.746290+0000","last_clean":"2024-02-19T14:53:47.746290+0000","last_became_active":"2024-02-19T14:53:47.742581+0000","last_became_peered":"2024-02-19T14:53:47.742581+0000","last_unstale":"2024-02-19T14:53:47.746290+0000","last_undegraded":"2024-02-19T14:53:47.746290+0000","last_fullsized":"2024-02-19T14:53:47.746290+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:57.701790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733388+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:47.733388+0000","last_peered":"2024-02-19T14:53:47.733388+0000","last_clean":"2024-02-19T14:53:47.733388+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:47.733388+0000","last_undegraded":"2024-02-19T14:53:47.733388+0000","last_fullsized":"2024-02-19T14:53:47.733388+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729246+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:47.729246+0000","last_peered":"2024-02-19T14:53:47.729246+0000","last_clean":"2024-02-19T14:53:47.729246+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:47.729246+0000","last_undegraded":"2024-02-19T14:53:47.729246+0000","last_fullsized":"2024-02-19T14:53:47.729246+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731908+0000","last_change":"2024-02-19T14:53:47.731908+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731908+0000","last_undegraded":"2024-02-19T14:53:47.731908+0000","last_fullsized":"2024-02-19T14:53:47.731908+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746114+0000","last_change":"2024-02-19T14:53:47.743491+0000","last_active":"2024-02-19T14:53:47.746114+0000","last_peered":"2024-02-19T14:53:47.746114+0000","last_clean":"2024-02-19T14:53:47.746114+0000","last_became_active":"2024-02-19T14:53:47.743209+0000","last_became_peered":"2024-02-19T14:53:47.743209+0000","last_unstale":"2024-02-19T14:53:47.746114+0000","last_undegraded":"2024-02-19T14:53:47.746114+0000","last_fullsized":"2024-02-19T14:53:47.746114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:51.151311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737504+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:47.737504+0000","last_peered":"2024-02-19T14:53:47.737504+0000","last_clean":"2024-02-19T14:53:47.737504+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:47.737504+0000","last_undegraded":"2024-02-19T14:53:47.737504+0000","last_fullsized":"2024-02-19T14:53:47.737504+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724055+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:47.724055+0000","last_peered":"2024-02-19T14:53:47.724055+0000","last_clean":"2024-02-19T14:53:47.724055+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:47.724055+0000","last_undegraded":"2024-02-19T14:53:47.724055+0000","last_fullsized":"2024-02-19T14:53:47.724055+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728116+0000","last_change":"2024-02-19T14:53:47.728116+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728116+0000","last_undegraded":"2024-02-19T14:53:47.728116+0000","last_fullsized":"2024-02-19T14:53:47.728116+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747934+0000","last_change":"2024-02-19T14:53:47.740831+0000","last_active":"2024-02-19T14:53:47.747934+0000","last_peered":"2024-02-19T14:53:47.747934+0000","last_clean":"2024-02-19T14:53:47.747934+0000","last_became_active":"2024-02-19T14:53:47.740540+0000","last_became_peered":"2024-02-19T14:53:47.740540+0000","last_unstale":"2024-02-19T14:53:47.747934+0000","last_undegraded":"2024-02-19T14:53:47.747934+0000","last_fullsized":"2024-02-19T14:53:47.747934+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733979+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:47.733979+0000","last_peered":"2024-02-19T14:53:47.733979+0000","last_clean":"2024-02-19T14:53:47.733979+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:47.733979+0000","last_undegraded":"2024-02-19T14:53:47.733979+0000","last_fullsized":"2024-02-19T14:53:47.733979+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731528+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:47.731528+0000","last_peered":"2024-02-19T14:53:47.731528+0000","last_clean":"2024-02-19T14:53:47.731528+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:47.731528+0000","last_undegraded":"2024-02-19T14:53:47.731528+0000","last_fullsized":"2024-02-19T14:53:47.731528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731324+0000","last_change":"2024-02-19T14:53:47.731324+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731324+0000","last_undegraded":"2024-02-19T14:53:47.731324+0000","last_fullsized":"2024-02-19T14:53:47.731324+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.740896+0000","last_change":"2024-02-19T14:53:47.736026+0000","last_active":"2024-02-19T14:53:47.740896+0000","last_peered":"2024-02-19T14:53:47.740896+0000","last_clean":"2024-02-19T14:53:47.740896+0000","last_became_active":"2024-02-19T14:53:47.735680+0000","last_became_peered":"2024-02-19T14:53:47.735680+0000","last_unstale":"2024-02-19T14:53:47.740896+0000","last_undegraded":"2024-02-19T14:53:47.740896+0000","last_fullsized":"2024-02-19T14:53:47.740896+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730749+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:47.730749+0000","last_peered":"2024-02-19T14:53:47.730749+0000","last_clean":"2024-02-19T14:53:47.730749+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:47.730749+0000","last_undegraded":"2024-02-19T14:53:47.730749+0000","last_fullsized":"2024-02-19T14:53:47.730749+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732455+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:47.732455+0000","last_peered":"2024-02-19T14:53:47.732455+0000","last_clean":"2024-02-19T14:53:47.732455+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:47.732455+0000","last_undegraded":"2024-02-19T14:53:47.732455+0000","last_fullsized":"2024-02-19T14:53:47.732455+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731805+0000","last_change":"2024-02-19T14:53:47.731805+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731805+0000","last_undegraded":"2024-02-19T14:53:47.731805+0000","last_fullsized":"2024-02-19T14:53:47.731805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743744+0000","last_change":"2024-02-19T14:53:47.737977+0000","last_active":"2024-02-19T14:53:47.743744+0000","last_peered":"2024-02-19T14:53:47.743744+0000","last_clean":"2024-02-19T14:53:47.743744+0000","last_became_active":"2024-02-19T14:53:47.737709+0000","last_became_peered":"2024-02-19T14:53:47.737709+0000","last_unstale":"2024-02-19T14:53:47.743744+0000","last_undegraded":"2024-02-19T14:53:47.743744+0000","last_fullsized":"2024-02-19T14:53:47.743744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:00.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734277+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:47.734277+0000","last_peered":"2024-02-19T14:53:47.734277+0000","last_clean":"2024-02-19T14:53:47.734277+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:47.734277+0000","last_undegraded":"2024-02-19T14:53:47.734277+0000","last_fullsized":"2024-02-19T14:53:47.734277+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739764+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:47.739764+0000","last_peered":"2024-02-19T14:53:47.739764+0000","last_clean":"2024-02-19T14:53:47.739764+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:47.739764+0000","last_undegraded":"2024-02-19T14:53:47.739764+0000","last_fullsized":"2024-02-19T14:53:47.739764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728207+0000","last_change":"2024-02-19T14:53:47.728207+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728207+0000","last_undegraded":"2024-02-19T14:53:47.728207+0000","last_fullsized":"2024-02-19T14:53:47.728207+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748964+0000","last_change":"2024-02-19T14:53:47.745205+0000","last_active":"2024-02-19T14:53:47.748964+0000","last_peered":"2024-02-19T14:53:47.748964+0000","last_clean":"2024-02-19T14:53:47.748964+0000","last_became_active":"2024-02-19T14:53:47.744958+0000","last_became_peered":"2024-02-19T14:53:47.744958+0000","last_unstale":"2024-02-19T14:53:47.748964+0000","last_undegraded":"2024-02-19T14:53:47.748964+0000","last_fullsized":"2024-02-19T14:53:47.748964+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729468+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:47.729468+0000","last_peered":"2024-02-19T14:53:47.729468+0000","last_clean":"2024-02-19T14:53:47.729468+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:47.729468+0000","last_undegraded":"2024-02-19T14:53:47.729468+0000","last_fullsized":"2024-02-19T14:53:47.729468+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731445+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:47.731445+0000","last_peered":"2024-02-19T14:53:47.731445+0000","last_clean":"2024-02-19T14:53:47.731445+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:47.731445+0000","last_undegraded":"2024-02-19T14:53:47.731445+0000","last_fullsized":"2024-02-19T14:53:47.731445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727406+0000","last_change":"2024-02-19T14:53:47.727406+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727406+0000","last_undegraded":"2024-02-19T14:53:47.727406+0000","last_fullsized":"2024-02-19T14:53:47.727406+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747521+0000","last_change":"2024-02-19T14:53:47.740210+0000","last_active":"2024-02-19T14:53:47.747521+0000","last_peered":"2024-02-19T14:53:47.747521+0000","last_clean":"2024-02-19T14:53:47.747521+0000","last_became_active":"2024-02-19T14:53:47.739913+0000","last_became_peered":"2024-02-19T14:53:47.739913+0000","last_unstale":"2024-02-19T14:53:47.747521+0000","last_undegraded":"2024-02-19T14:53:47.747521+0000","last_fullsized":"2024-02-19T14:53:47.747521+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723813+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:47.723813+0000","last_peered":"2024-02-19T14:53:47.723813+0000","last_clean":"2024-02-19T14:53:47.723813+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:47.723813+0000","last_undegraded":"2024-02-19T14:53:47.723813+0000","last_fullsized":"2024-02-19T14:53:47.723813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:10.864244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732089+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:47.732089+0000","last_peered":"2024-02-19T14:53:47.732089+0000","last_clean":"2024-02-19T14:53:47.732089+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:47.732089+0000","last_undegraded":"2024-02-19T14:53:47.732089+0000","last_fullsized":"2024-02-19T14:53:47.732089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735192+0000","last_change":"2024-02-19T14:53:47.735192+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735192+0000","last_undegraded":"2024-02-19T14:53:47.735192+0000","last_fullsized":"2024-02-19T14:53:47.735192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.745944+0000","last_change":"2024-02-19T14:53:47.738871+0000","last_active":"2024-02-19T14:53:47.745944+0000","last_peered":"2024-02-19T14:53:47.745944+0000","last_clean":"2024-02-19T14:53:47.745944+0000","last_became_active":"2024-02-19T14:53:47.738585+0000","last_became_peered":"2024-02-19T14:53:47.738585+0000","last_unstale":"2024-02-19T14:53:47.745944+0000","last_undegraded":"2024-02-19T14:53:47.745944+0000","last_fullsized":"2024-02-19T14:53:47.745944+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:02.790030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731936+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:47.731936+0000","last_peered":"2024-02-19T14:53:47.731936+0000","last_clean":"2024-02-19T14:53:47.731936+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:47.731936+0000","last_undegraded":"2024-02-19T14:53:47.731936+0000","last_fullsized":"2024-02-19T14:53:47.731936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732414+0000","last_change":"2024-02-19T14:53:47.732414+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732414+0000","last_undegraded":"2024-02-19T14:53:47.732414+0000","last_fullsized":"2024-02-19T14:53:47.732414+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743039+0000","last_change":"2024-02-19T14:53:47.738834+0000","last_active":"2024-02-19T14:53:47.743039+0000","last_peered":"2024-02-19T14:53:47.743039+0000","last_clean":"2024-02-19T14:53:47.743039+0000","last_became_active":"2024-02-19T14:53:47.738567+0000","last_became_peered":"2024-02-19T14:53:47.738567+0000","last_unstale":"2024-02-19T14:53:47.743039+0000","last_undegraded":"2024-02-19T14:53:47.743039+0000","last_fullsized":"2024-02-19T14:53:47.743039+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:50.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727552+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:47.727552+0000","last_peered":"2024-02-19T14:53:47.727552+0000","last_clean":"2024-02-19T14:53:47.727552+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:47.727552+0000","last_undegraded":"2024-02-19T14:53:47.727552+0000","last_fullsized":"2024-02-19T14:53:47.727552+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724124+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:47.724124+0000","last_peered":"2024-02-19T14:53:47.724124+0000","last_clean":"2024-02-19T14:53:47.724124+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:47.724124+0000","last_undegraded":"2024-02-19T14:53:47.724124+0000","last_fullsized":"2024-02-19T14:53:47.724124+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728013+0000","last_change":"2024-02-19T14:53:47.728013+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728013+0000","last_undegraded":"2024-02-19T14:53:47.728013+0000","last_fullsized":"2024-02-19T14:53:47.728013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747961+0000","last_change":"2024-02-19T14:53:47.744007+0000","last_active":"2024-02-19T14:53:47.747961+0000","last_peered":"2024-02-19T14:53:47.747961+0000","last_clean":"2024-02-19T14:53:47.747961+0000","last_became_active":"2024-02-19T14:53:47.743726+0000","last_became_peered":"2024-02-19T14:53:47.743726+0000","last_unstale":"2024-02-19T14:53:47.747961+0000","last_undegraded":"2024-02-19T14:53:47.747961+0000","last_fullsized":"2024-02-19T14:53:47.747961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:26.617672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733834+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:47.733834+0000","last_peered":"2024-02-19T14:53:47.733834+0000","last_clean":"2024-02-19T14:53:47.733834+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:47.733834+0000","last_undegraded":"2024-02-19T14:53:47.733834+0000","last_fullsized":"2024-02-19T14:53:47.733834+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730733+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:47.730733+0000","last_peered":"2024-02-19T14:53:47.730733+0000","last_clean":"2024-02-19T14:53:47.730733+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:47.730733+0000","last_undegraded":"2024-02-19T14:53:47.730733+0000","last_fullsized":"2024-02-19T14:53:47.730733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732525+0000","last_change":"2024-02-19T14:53:47.732525+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732525+0000","last_undegraded":"2024-02-19T14:53:47.732525+0000","last_fullsized":"2024-02-19T14:53:47.732525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746917+0000","last_change":"2024-02-19T14:53:47.742015+0000","last_active":"2024-02-19T14:53:47.746917+0000","last_peered":"2024-02-19T14:53:47.746917+0000","last_clean":"2024-02-19T14:53:47.746917+0000","last_became_active":"2024-02-19T14:53:47.741742+0000","last_became_peered":"2024-02-19T14:53:47.741742+0000","last_unstale":"2024-02-19T14:53:47.746917+0000","last_undegraded":"2024-02-19T14:53:47.746917+0000","last_fullsized":"2024-02-19T14:53:47.746917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724348+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:47.724348+0000","last_peered":"2024-02-19T14:53:47.724348+0000","last_clean":"2024-02-19T14:53:47.724348+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:47.724348+0000","last_undegraded":"2024-02-19T14:53:47.724348+0000","last_fullsized":"2024-02-19T14:53:47.724348+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:45.189873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734386+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:47.734386+0000","last_peered":"2024-02-19T14:53:47.734386+0000","last_clean":"2024-02-19T14:53:47.734386+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:47.734386+0000","last_undegraded":"2024-02-19T14:53:47.734386+0000","last_fullsized":"2024-02-19T14:53:47.734386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728949+0000","last_change":"2024-02-19T14:53:47.728949+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728949+0000","last_undegraded":"2024-02-19T14:53:47.728949+0000","last_fullsized":"2024-02-19T14:53:47.728949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737489+0000","last_change":"2024-02-19T14:53:47.734524+0000","last_active":"2024-02-19T14:53:47.737489+0000","last_peered":"2024-02-19T14:53:47.737489+0000","last_clean":"2024-02-19T14:53:47.737489+0000","last_became_active":"2024-02-19T14:53:47.734132+0000","last_became_peered":"2024-02-19T14:53:47.734132+0000","last_unstale":"2024-02-19T14:53:47.737489+0000","last_undegraded":"2024-02-19T14:53:47.737489+0000","last_fullsized":"2024-02-19T14:53:47.737489+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737553+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:47.737553+0000","last_peered":"2024-02-19T14:53:47.737553+0000","last_clean":"2024-02-19T14:53:47.737553+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:47.737553+0000","last_undegraded":"2024-02-19T14:53:47.737553+0000","last_fullsized":"2024-02-19T14:53:47.737553+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734066+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:47.734066+0000","last_peered":"2024-02-19T14:53:47.734066+0000","last_clean":"2024-02-19T14:53:47.734066+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:47.734066+0000","last_undegraded":"2024-02-19T14:53:47.734066+0000","last_fullsized":"2024-02-19T14:53:47.734066+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":236,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729312+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:47.729312+0000","last_peered":"2024-02-19T14:53:47.729312+0000","last_clean":"2024-02-19T14:53:47.729312+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:47.729312+0000","last_undegraded":"2024-02-19T14:53:47.729312+0000","last_fullsized":"2024-02-19T14:53:47.729312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.502},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":291530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.502}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.626},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.311},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":299032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.624},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":292512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.245}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.372}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.621}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":292490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.31}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.5}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":298610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.247},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.20000000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.217},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.435},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.36399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.42399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:51.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:52.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.017+0000 7f3a65bc2640 1 Processor -- start 2024-02-19T14:53:52.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.017+0000 7f3a65bc2640 1 -- start start 2024-02-19T14:53:52.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a65bc2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a60152700 0x7f3a600c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a65bc2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c8e00 0x7f3a6015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a65bc2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a601551f0 con 0x7f3a600c5730 2024-02-19T14:53:52.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a65bc2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a60155490 con 0x7f3a60152700 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a65bc2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a60155770 con 0x7f3a600c8e00 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a60152700 0x7f3a600c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a60152700 0x7f3a600c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53396/0 (socket says 172.21.15.78:53396) 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 -- 172.21.15.78:0/718691625 learned_addr learned my addr 172.21.15.78:0/718691625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5ffff640 1 --2- 172.21.15.78:0/718691625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c8e00 0x7f3a6015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 -- 172.21.15.78:0/718691625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c8e00 msgr2=0x7f3a6015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 --2- 172.21.15.78:0/718691625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c8e00 0x7f3a6015f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 -- 172.21.15.78:0/718691625 >> v1:172.21.15.150:6789/0 conn(0x7f3a600c5730 legacy=0x7f3a6006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:52.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.018+0000 7f3a5f7fe640 1 -- 172.21.15.78:0/718691625 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60155a50 con 0x7f3a60152700 2024-02-19T14:53:52.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.019+0000 7f3a5f7fe640 1 --2- 172.21.15.78:0/718691625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a600c51f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009870 tx=0x7f3a5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2aa58935093d998 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a5effd640 1 -- 172.21.15.78:0/718691625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50031640 con 0x7f3a60152700 2024-02-19T14:53:52.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a5effd640 1 -- 172.21.15.78:0/718691625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a500317a0 con 0x7f3a60152700 2024-02-19T14:53:52.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a5effd640 1 -- 172.21.15.78:0/718691625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50031b30 con 0x7f3a60152700 2024-02-19T14:53:52.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 -- 172.21.15.78:0/718691625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 msgr2=0x7f3a600c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/718691625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a600c51f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009870 tx=0x7f3a5002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 -- 172.21.15.78:0/718691625 shutdown_connections 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/718691625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c8e00 0x7f3a6015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/718691625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a600c51f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 -- 172.21.15.78:0/718691625 >> 172.21.15.78:0/718691625 conn(0x7f3a600cac90 msgr2=0x7f3a600cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 -- 172.21.15.78:0/718691625 shutdown_connections 2024-02-19T14:53:52.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.020+0000 7f3a65bc2640 1 -- 172.21.15.78:0/718691625 wait complete. 2024-02-19T14:53:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.021+0000 7f3a65bc2640 1 Processor -- start 2024-02-19T14:53:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.021+0000 7f3a65bc2640 1 -- start start 2024-02-19T14:53:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a65bc2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c5730 0x7f3a60158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a65bc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a600c8e00 0x7f3a60156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a65bc2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a60148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a65bc2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600c2610 con 0x7f3a60152700 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a65bc2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a60157f50 con 0x7f3a600c8e00 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a65bc2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a601576b0 con 0x7f3a600c5730 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a5ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a60148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a5ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a60148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53398/0 (socket says 172.21.15.78:53398) 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a5ffff640 1 -- 172.21.15.78:0/2863379681 learned_addr learned my addr 172.21.15.78:0/2863379681 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a5f7fe640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c5730 0x7f3a60158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.022+0000 7f3a56ffd640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a600c8e00 0x7f3a60156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5ffff640 1 -- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c5730 msgr2=0x7f3a60158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5ffff640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c5730 0x7f3a60158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5ffff640 1 -- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a600c8e00 msgr2=0x7f3a60156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5ffff640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a600c8e00 0x7f3a60156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5ffff640 1 -- 172.21.15.78:0/2863379681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a500094d0 con 0x7f3a60152700 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5ffff640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a60148c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800cd70 tx=0x7f3a5800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58003b90 con 0x7f3a60152700 2024-02-19T14:53:52.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58003cf0 con 0x7f3a60152700 2024-02-19T14:53:52.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a60157990 con 0x7f3a60152700 2024-02-19T14:53:52.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58004240 con 0x7f3a60152700 2024-02-19T14:53:52.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.023+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a60159240 con 0x7f3a60152700 2024-02-19T14:53:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.024+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a600c2430 con 0x7f3a60152700 2024-02-19T14:53:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.025+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3a580043a0 con 0x7f3a60152700 2024-02-19T14:53:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.025+0000 7f3a5cff9640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a4006b840 0x7f3a4006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.025+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f3a58012070 con 0x7f3a60152700 2024-02-19T14:53:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.025+0000 7f3a5f7fe640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a4006b840 0x7f3a4006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.026+0000 7f3a5f7fe640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a4006b840 0x7f3a4006dd00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a50036000 tx=0x7f3a5003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.033+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3a580555c0 con 0x7f3a60152700 2024-02-19T14:53:52.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.236+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a6006ded0 con 0x7f3a4006b840 2024-02-19T14:53:52.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.261+0000 7f3a5cff9640 1 -- 172.21.15.78:0/2863379681 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+522875 (secure 0 0 0) 0x7f3a500315e0 con 0x7f3a4006b840 2024-02-19T14:53:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:52.273 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a4006b840 msgr2=0x7f3a4006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a4006b840 0x7f3a4006dd00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a50036000 tx=0x7f3a5003c040 comp rx=0 tx=0).stop 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 msgr2=0x7f3a60148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a60148c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800cd70 tx=0x7f3a5800a660 comp rx=0 tx=0).stop 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 shutdown_connections 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a4006b840 0x7f3a4006dd00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a60152700 0x7f3a60148c90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a600c8e00 0x7f3a60156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 --2- 172.21.15.78:0/2863379681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a600c5730 0x7f3a60158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 >> 172.21.15.78:0/2863379681 conn(0x7f3a600cac90 msgr2=0x7f3a601517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:52.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 shutdown_connections 2024-02-19T14:53:52.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.266+0000 7f3a65bc2640 1 -- 172.21.15.78:0/2863379681 wait complete. 2024-02-19T14:53:52.280 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-02-19T14:53:52.082264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222440,"kb_used_data":6272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752280,"statfs":{"total":773094113280,"available":772866334720,"internally_reserved":0,"allocated":6422528,"data_stored":3527724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.364855"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730654+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:47.730654+0000","last_peered":"2024-02-19T14:53:47.730654+0000","last_clean":"2024-02-19T14:53:47.730654+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:47.730654+0000","last_undegraded":"2024-02-19T14:53:47.730654+0000","last_fullsized":"2024-02-19T14:53:47.730654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731535+0000","last_change":"2024-02-19T14:53:47.731535+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731535+0000","last_undegraded":"2024-02-19T14:53:47.731535+0000","last_fullsized":"2024-02-19T14:53:47.731535+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730461+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:47.730461+0000","last_peered":"2024-02-19T14:53:47.730461+0000","last_clean":"2024-02-19T14:53:47.730461+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:47.730461+0000","last_undegraded":"2024-02-19T14:53:47.730461+0000","last_fullsized":"2024-02-19T14:53:47.730461+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728965+0000","last_change":"2024-02-19T14:53:47.728965+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728965+0000","last_undegraded":"2024-02-19T14:53:47.728965+0000","last_fullsized":"2024-02-19T14:53:47.728965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732216+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:47.732216+0000","last_peered":"2024-02-19T14:53:47.732216+0000","last_clean":"2024-02-19T14:53:47.732216+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:47.732216+0000","last_undegraded":"2024-02-19T14:53:47.732216+0000","last_fullsized":"2024-02-19T14:53:47.732216+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726570+0000","last_change":"2024-02-19T14:53:47.726570+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726570+0000","last_undegraded":"2024-02-19T14:53:47.726570+0000","last_fullsized":"2024-02-19T14:53:47.726570+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729626+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:47.729626+0000","last_peered":"2024-02-19T14:53:47.729626+0000","last_clean":"2024-02-19T14:53:47.729626+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:47.729626+0000","last_undegraded":"2024-02-19T14:53:47.729626+0000","last_fullsized":"2024-02-19T14:53:47.729626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730942+0000","last_change":"2024-02-19T14:53:47.730942+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730942+0000","last_undegraded":"2024-02-19T14:53:47.730942+0000","last_fullsized":"2024-02-19T14:53:47.730942+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731084+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:47.731084+0000","last_peered":"2024-02-19T14:53:47.731084+0000","last_clean":"2024-02-19T14:53:47.731084+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:47.731084+0000","last_undegraded":"2024-02-19T14:53:47.731084+0000","last_fullsized":"2024-02-19T14:53:47.731084+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729130+0000","last_change":"2024-02-19T14:53:47.729130+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729130+0000","last_undegraded":"2024-02-19T14:53:47.729130+0000","last_fullsized":"2024-02-19T14:53:47.729130+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729373+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:47.729373+0000","last_peered":"2024-02-19T14:53:47.729373+0000","last_clean":"2024-02-19T14:53:47.729373+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:47.729373+0000","last_undegraded":"2024-02-19T14:53:47.729373+0000","last_fullsized":"2024-02-19T14:53:47.729373+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731517+0000","last_change":"2024-02-19T14:53:47.731517+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731517+0000","last_undegraded":"2024-02-19T14:53:47.731517+0000","last_fullsized":"2024-02-19T14:53:47.731517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731295+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:47.731295+0000","last_peered":"2024-02-19T14:53:47.731295+0000","last_clean":"2024-02-19T14:53:47.731295+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:47.731295+0000","last_undegraded":"2024-02-19T14:53:47.731295+0000","last_fullsized":"2024-02-19T14:53:47.731295+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727983+0000","last_change":"2024-02-19T14:53:47.727983+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727983+0000","last_undegraded":"2024-02-19T14:53:47.727983+0000","last_fullsized":"2024-02-19T14:53:47.727983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723628+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:47.723628+0000","last_peered":"2024-02-19T14:53:47.723628+0000","last_clean":"2024-02-19T14:53:47.723628+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:47.723628+0000","last_undegraded":"2024-02-19T14:53:47.723628+0000","last_fullsized":"2024-02-19T14:53:47.723628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733825+0000","last_change":"2024-02-19T14:53:47.733825+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733825+0000","last_undegraded":"2024-02-19T14:53:47.733825+0000","last_fullsized":"2024-02-19T14:53:47.733825+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724256+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:47.724256+0000","last_peered":"2024-02-19T14:53:47.724256+0000","last_clean":"2024-02-19T14:53:47.724256+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:47.724256+0000","last_undegraded":"2024-02-19T14:53:47.724256+0000","last_fullsized":"2024-02-19T14:53:47.724256+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:50.707228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733022+0000","last_change":"2024-02-19T14:53:47.733022+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733022+0000","last_undegraded":"2024-02-19T14:53:47.733022+0000","last_fullsized":"2024-02-19T14:53:47.733022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730312+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:47.730312+0000","last_peered":"2024-02-19T14:53:47.730312+0000","last_clean":"2024-02-19T14:53:47.730312+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:47.730312+0000","last_undegraded":"2024-02-19T14:53:47.730312+0000","last_fullsized":"2024-02-19T14:53:47.730312+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733611+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:47.733611+0000","last_peered":"2024-02-19T14:53:47.733611+0000","last_clean":"2024-02-19T14:53:47.733611+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:47.733611+0000","last_undegraded":"2024-02-19T14:53:47.733611+0000","last_fullsized":"2024-02-19T14:53:47.733611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:37.193943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726799+0000","last_change":"2024-02-19T14:53:47.726799+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726799+0000","last_undegraded":"2024-02-19T14:53:47.726799+0000","last_fullsized":"2024-02-19T14:53:47.726799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724742+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:47.724742+0000","last_peered":"2024-02-19T14:53:47.724742+0000","last_clean":"2024-02-19T14:53:47.724742+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:47.724742+0000","last_undegraded":"2024-02-19T14:53:47.724742+0000","last_fullsized":"2024-02-19T14:53:47.724742+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:01.639745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726754+0000","last_change":"2024-02-19T14:53:47.726754+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726754+0000","last_undegraded":"2024-02-19T14:53:47.726754+0000","last_fullsized":"2024-02-19T14:53:47.726754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729060+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:47.729060+0000","last_peered":"2024-02-19T14:53:47.729060+0000","last_clean":"2024-02-19T14:53:47.729060+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:47.729060+0000","last_undegraded":"2024-02-19T14:53:47.729060+0000","last_fullsized":"2024-02-19T14:53:47.729060+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731147+0000","last_change":"2024-02-19T14:53:47.731147+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731147+0000","last_undegraded":"2024-02-19T14:53:47.731147+0000","last_fullsized":"2024-02-19T14:53:47.731147+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739485+0000","last_change":"2024-02-19T14:53:42.136272+0000","last_active":"2024-02-19T14:53:47.739485+0000","last_peered":"2024-02-19T14:53:47.739485+0000","last_clean":"2024-02-19T14:53:47.739485+0000","last_became_active":"2024-02-19T14:53:42.135998+0000","last_became_peered":"2024-02-19T14:53:42.135998+0000","last_unstale":"2024-02-19T14:53:47.739485+0000","last_undegraded":"2024-02-19T14:53:47.739485+0000","last_fullsized":"2024-02-19T14:53:47.739485+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734360+0000","last_change":"2024-02-19T14:53:47.734360+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734360+0000","last_undegraded":"2024-02-19T14:53:47.734360+0000","last_fullsized":"2024-02-19T14:53:47.734360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729068+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:47.729068+0000","last_peered":"2024-02-19T14:53:47.729068+0000","last_clean":"2024-02-19T14:53:47.729068+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:47.729068+0000","last_undegraded":"2024-02-19T14:53:47.729068+0000","last_fullsized":"2024-02-19T14:53:47.729068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726289+0000","last_change":"2024-02-19T14:53:47.726289+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726289+0000","last_undegraded":"2024-02-19T14:53:47.726289+0000","last_fullsized":"2024-02-19T14:53:47.726289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727510+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:47.727510+0000","last_peered":"2024-02-19T14:53:47.727510+0000","last_clean":"2024-02-19T14:53:47.727510+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:47.727510+0000","last_undegraded":"2024-02-19T14:53:47.727510+0000","last_fullsized":"2024-02-19T14:53:47.727510+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731901+0000","last_change":"2024-02-19T14:53:47.731901+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731901+0000","last_undegraded":"2024-02-19T14:53:47.731901+0000","last_fullsized":"2024-02-19T14:53:47.731901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734311+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:47.734311+0000","last_peered":"2024-02-19T14:53:47.734311+0000","last_clean":"2024-02-19T14:53:47.734311+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:47.734311+0000","last_undegraded":"2024-02-19T14:53:47.734311+0000","last_fullsized":"2024-02-19T14:53:47.734311+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726614+0000","last_change":"2024-02-19T14:53:47.726614+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726614+0000","last_undegraded":"2024-02-19T14:53:47.726614+0000","last_fullsized":"2024-02-19T14:53:47.726614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730365+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:47.730365+0000","last_peered":"2024-02-19T14:53:47.730365+0000","last_clean":"2024-02-19T14:53:47.730365+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:47.730365+0000","last_undegraded":"2024-02-19T14:53:47.730365+0000","last_fullsized":"2024-02-19T14:53:47.730365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728581+0000","last_change":"2024-02-19T14:53:47.728581+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728581+0000","last_undegraded":"2024-02-19T14:53:47.728581+0000","last_fullsized":"2024-02-19T14:53:47.728581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732357+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:47.732357+0000","last_peered":"2024-02-19T14:53:47.732357+0000","last_clean":"2024-02-19T14:53:47.732357+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:47.732357+0000","last_undegraded":"2024-02-19T14:53:47.732357+0000","last_fullsized":"2024-02-19T14:53:47.732357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735616+0000","last_change":"2024-02-19T14:53:47.735616+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735616+0000","last_undegraded":"2024-02-19T14:53:47.735616+0000","last_fullsized":"2024-02-19T14:53:47.735616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737614+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:47.737614+0000","last_peered":"2024-02-19T14:53:47.737614+0000","last_clean":"2024-02-19T14:53:47.737614+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:47.737614+0000","last_undegraded":"2024-02-19T14:53:47.737614+0000","last_fullsized":"2024-02-19T14:53:47.737614+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726307+0000","last_change":"2024-02-19T14:53:47.726307+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726307+0000","last_undegraded":"2024-02-19T14:53:47.726307+0000","last_fullsized":"2024-02-19T14:53:47.726307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739540+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:47.739540+0000","last_peered":"2024-02-19T14:53:47.739540+0000","last_clean":"2024-02-19T14:53:47.739540+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:47.739540+0000","last_undegraded":"2024-02-19T14:53:47.739540+0000","last_fullsized":"2024-02-19T14:53:47.739540+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725966+0000","last_change":"2024-02-19T14:53:47.725966+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725966+0000","last_undegraded":"2024-02-19T14:53:47.725966+0000","last_fullsized":"2024-02-19T14:53:47.725966+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723966+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:47.723966+0000","last_peered":"2024-02-19T14:53:47.723966+0000","last_clean":"2024-02-19T14:53:47.723966+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:47.723966+0000","last_undegraded":"2024-02-19T14:53:47.723966+0000","last_fullsized":"2024-02-19T14:53:47.723966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:59.053283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729507+0000","last_change":"2024-02-19T14:53:47.729507+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729507+0000","last_undegraded":"2024-02-19T14:53:47.729507+0000","last_fullsized":"2024-02-19T14:53:47.729507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732727+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:47.732727+0000","last_peered":"2024-02-19T14:53:47.732727+0000","last_clean":"2024-02-19T14:53:47.732727+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:47.732727+0000","last_undegraded":"2024-02-19T14:53:47.732727+0000","last_fullsized":"2024-02-19T14:53:47.732727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:38:18.934930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726592+0000","last_change":"2024-02-19T14:53:47.726592+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726592+0000","last_undegraded":"2024-02-19T14:53:47.726592+0000","last_fullsized":"2024-02-19T14:53:47.726592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730738+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:47.730738+0000","last_peered":"2024-02-19T14:53:47.730738+0000","last_clean":"2024-02-19T14:53:47.730738+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:47.730738+0000","last_undegraded":"2024-02-19T14:53:47.730738+0000","last_fullsized":"2024-02-19T14:53:47.730738+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734650+0000","last_change":"2024-02-19T14:53:47.734650+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734650+0000","last_undegraded":"2024-02-19T14:53:47.734650+0000","last_fullsized":"2024-02-19T14:53:47.734650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734454+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:47.734454+0000","last_peered":"2024-02-19T14:53:47.734454+0000","last_clean":"2024-02-19T14:53:47.734454+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:47.734454+0000","last_undegraded":"2024-02-19T14:53:47.734454+0000","last_fullsized":"2024-02-19T14:53:47.734454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729138+0000","last_change":"2024-02-19T14:53:47.729138+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729138+0000","last_undegraded":"2024-02-19T14:53:47.729138+0000","last_fullsized":"2024-02-19T14:53:47.729138+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734486+0000","last_change":"2024-02-19T14:53:47.734486+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734486+0000","last_undegraded":"2024-02-19T14:53:47.734486+0000","last_fullsized":"2024-02-19T14:53:47.734486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729945+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:47.729945+0000","last_peered":"2024-02-19T14:53:47.729945+0000","last_clean":"2024-02-19T14:53:47.729945+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:47.729945+0000","last_undegraded":"2024-02-19T14:53:47.729945+0000","last_fullsized":"2024-02-19T14:53:47.729945+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733413+0000","last_change":"2024-02-19T14:53:47.733413+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733413+0000","last_undegraded":"2024-02-19T14:53:47.733413+0000","last_fullsized":"2024-02-19T14:53:47.733413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737727+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:47.737727+0000","last_peered":"2024-02-19T14:53:47.737727+0000","last_clean":"2024-02-19T14:53:47.737727+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:47.737727+0000","last_undegraded":"2024-02-19T14:53:47.737727+0000","last_fullsized":"2024-02-19T14:53:47.737727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729994+0000","last_change":"2024-02-19T14:53:47.729994+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729994+0000","last_undegraded":"2024-02-19T14:53:47.729994+0000","last_fullsized":"2024-02-19T14:53:47.729994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732428+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:47.732428+0000","last_peered":"2024-02-19T14:53:47.732428+0000","last_clean":"2024-02-19T14:53:47.732428+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:47.732428+0000","last_undegraded":"2024-02-19T14:53:47.732428+0000","last_fullsized":"2024-02-19T14:53:47.732428+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728586+0000","last_change":"2024-02-19T14:53:47.728586+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728586+0000","last_undegraded":"2024-02-19T14:53:47.728586+0000","last_fullsized":"2024-02-19T14:53:47.728586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729908+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:47.729908+0000","last_peered":"2024-02-19T14:53:47.729908+0000","last_clean":"2024-02-19T14:53:47.729908+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:47.729908+0000","last_undegraded":"2024-02-19T14:53:47.729908+0000","last_fullsized":"2024-02-19T14:53:47.729908+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726173+0000","last_change":"2024-02-19T14:53:47.726173+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726173+0000","last_undegraded":"2024-02-19T14:53:47.726173+0000","last_fullsized":"2024-02-19T14:53:47.726173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731441+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:47.731441+0000","last_peered":"2024-02-19T14:53:47.731441+0000","last_clean":"2024-02-19T14:53:47.731441+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:47.731441+0000","last_undegraded":"2024-02-19T14:53:47.731441+0000","last_fullsized":"2024-02-19T14:53:47.731441+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727599+0000","last_change":"2024-02-19T14:53:47.727599+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727599+0000","last_undegraded":"2024-02-19T14:53:47.727599+0000","last_fullsized":"2024-02-19T14:53:47.727599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734161+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:47.734161+0000","last_peered":"2024-02-19T14:53:47.734161+0000","last_clean":"2024-02-19T14:53:47.734161+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:47.734161+0000","last_undegraded":"2024-02-19T14:53:47.734161+0000","last_fullsized":"2024-02-19T14:53:47.734161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735808+0000","last_change":"2024-02-19T14:53:47.735808+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735808+0000","last_undegraded":"2024-02-19T14:53:47.735808+0000","last_fullsized":"2024-02-19T14:53:47.735808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746858+0000","last_change":"2024-02-19T14:53:47.740946+0000","last_active":"2024-02-19T14:53:47.746858+0000","last_peered":"2024-02-19T14:53:47.746858+0000","last_clean":"2024-02-19T14:53:47.746858+0000","last_became_active":"2024-02-19T14:53:47.740550+0000","last_became_peered":"2024-02-19T14:53:47.740550+0000","last_unstale":"2024-02-19T14:53:47.746858+0000","last_undegraded":"2024-02-19T14:53:47.746858+0000","last_fullsized":"2024-02-19T14:53:47.746858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:09.101397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728386+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:47.728386+0000","last_peered":"2024-02-19T14:53:47.728386+0000","last_clean":"2024-02-19T14:53:47.728386+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:47.728386+0000","last_undegraded":"2024-02-19T14:53:47.728386+0000","last_fullsized":"2024-02-19T14:53:47.728386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":26,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739615+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:47.739615+0000","last_peered":"2024-02-19T14:53:47.739615+0000","last_clean":"2024-02-19T14:53:47.739615+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:47.739615+0000","last_undegraded":"2024-02-19T14:53:47.739615+0000","last_fullsized":"2024-02-19T14:53:47.739615+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727391+0000","last_change":"2024-02-19T14:53:47.727391+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727391+0000","last_undegraded":"2024-02-19T14:53:47.727391+0000","last_fullsized":"2024-02-19T14:53:47.727391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746433+0000","last_change":"2024-02-19T14:53:47.742941+0000","last_active":"2024-02-19T14:53:47.746433+0000","last_peered":"2024-02-19T14:53:47.746433+0000","last_clean":"2024-02-19T14:53:47.746433+0000","last_became_active":"2024-02-19T14:53:47.742590+0000","last_became_peered":"2024-02-19T14:53:47.742590+0000","last_unstale":"2024-02-19T14:53:47.746433+0000","last_undegraded":"2024-02-19T14:53:47.746433+0000","last_fullsized":"2024-02-19T14:53:47.746433+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:35.779780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729200+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:47.729200+0000","last_peered":"2024-02-19T14:53:47.729200+0000","last_clean":"2024-02-19T14:53:47.729200+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:47.729200+0000","last_undegraded":"2024-02-19T14:53:47.729200+0000","last_fullsized":"2024-02-19T14:53:47.729200+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730410+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:47.730410+0000","last_peered":"2024-02-19T14:53:47.730410+0000","last_clean":"2024-02-19T14:53:47.730410+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:47.730410+0000","last_undegraded":"2024-02-19T14:53:47.730410+0000","last_fullsized":"2024-02-19T14:53:47.730410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729346+0000","last_change":"2024-02-19T14:53:47.729346+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729346+0000","last_undegraded":"2024-02-19T14:53:47.729346+0000","last_fullsized":"2024-02-19T14:53:47.729346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747720+0000","last_change":"2024-02-19T14:53:47.746326+0000","last_active":"2024-02-19T14:53:47.747720+0000","last_peered":"2024-02-19T14:53:47.747720+0000","last_clean":"2024-02-19T14:53:47.747720+0000","last_became_active":"2024-02-19T14:53:47.745753+0000","last_became_peered":"2024-02-19T14:53:47.745753+0000","last_unstale":"2024-02-19T14:53:47.747720+0000","last_undegraded":"2024-02-19T14:53:47.747720+0000","last_fullsized":"2024-02-19T14:53:47.747720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739624+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:47.739624+0000","last_peered":"2024-02-19T14:53:47.739624+0000","last_clean":"2024-02-19T14:53:47.739624+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:47.739624+0000","last_undegraded":"2024-02-19T14:53:47.739624+0000","last_fullsized":"2024-02-19T14:53:47.739624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730034+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:47.730034+0000","last_peered":"2024-02-19T14:53:47.730034+0000","last_clean":"2024-02-19T14:53:47.730034+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:47.730034+0000","last_undegraded":"2024-02-19T14:53:47.730034+0000","last_fullsized":"2024-02-19T14:53:47.730034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726858+0000","last_change":"2024-02-19T14:53:47.726858+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726858+0000","last_undegraded":"2024-02-19T14:53:47.726858+0000","last_fullsized":"2024-02-19T14:53:47.726858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747935+0000","last_change":"2024-02-19T14:53:47.740910+0000","last_active":"2024-02-19T14:53:47.747935+0000","last_peered":"2024-02-19T14:53:47.747935+0000","last_clean":"2024-02-19T14:53:47.747935+0000","last_became_active":"2024-02-19T14:53:47.740626+0000","last_became_peered":"2024-02-19T14:53:47.740626+0000","last_unstale":"2024-02-19T14:53:47.747935+0000","last_undegraded":"2024-02-19T14:53:47.747935+0000","last_fullsized":"2024-02-19T14:53:47.747935+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723809+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:47.723809+0000","last_peered":"2024-02-19T14:53:47.723809+0000","last_clean":"2024-02-19T14:53:47.723809+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:47.723809+0000","last_undegraded":"2024-02-19T14:53:47.723809+0000","last_fullsized":"2024-02-19T14:53:47.723809+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:36.026355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733496+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:47.733496+0000","last_peered":"2024-02-19T14:53:47.733496+0000","last_clean":"2024-02-19T14:53:47.733496+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:47.733496+0000","last_undegraded":"2024-02-19T14:53:47.733496+0000","last_fullsized":"2024-02-19T14:53:47.733496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732000+0000","last_change":"2024-02-19T14:53:47.732000+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732000+0000","last_undegraded":"2024-02-19T14:53:47.732000+0000","last_fullsized":"2024-02-19T14:53:47.732000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747004+0000","last_change":"2024-02-19T14:53:47.745151+0000","last_active":"2024-02-19T14:53:47.747004+0000","last_peered":"2024-02-19T14:53:47.747004+0000","last_clean":"2024-02-19T14:53:47.747004+0000","last_became_active":"2024-02-19T14:53:47.744887+0000","last_became_peered":"2024-02-19T14:53:47.744887+0000","last_unstale":"2024-02-19T14:53:47.747004+0000","last_undegraded":"2024-02-19T14:53:47.747004+0000","last_fullsized":"2024-02-19T14:53:47.747004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732432+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:47.732432+0000","last_peered":"2024-02-19T14:53:47.732432+0000","last_clean":"2024-02-19T14:53:47.732432+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:47.732432+0000","last_undegraded":"2024-02-19T14:53:47.732432+0000","last_fullsized":"2024-02-19T14:53:47.732432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732599+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:47.732599+0000","last_peered":"2024-02-19T14:53:47.732599+0000","last_clean":"2024-02-19T14:53:47.732599+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:47.732599+0000","last_undegraded":"2024-02-19T14:53:47.732599+0000","last_fullsized":"2024-02-19T14:53:47.732599+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.736522+0000","last_change":"2024-02-19T14:53:47.736522+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.736522+0000","last_undegraded":"2024-02-19T14:53:47.736522+0000","last_fullsized":"2024-02-19T14:53:47.736522+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746094+0000","last_change":"2024-02-19T14:53:47.743381+0000","last_active":"2024-02-19T14:53:47.746094+0000","last_peered":"2024-02-19T14:53:47.746094+0000","last_clean":"2024-02-19T14:53:47.746094+0000","last_became_active":"2024-02-19T14:53:47.743135+0000","last_became_peered":"2024-02-19T14:53:47.743135+0000","last_unstale":"2024-02-19T14:53:47.746094+0000","last_undegraded":"2024-02-19T14:53:47.746094+0000","last_fullsized":"2024-02-19T14:53:47.746094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:30.540219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730104+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:47.730104+0000","last_peered":"2024-02-19T14:53:47.730104+0000","last_clean":"2024-02-19T14:53:47.730104+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:47.730104+0000","last_undegraded":"2024-02-19T14:53:47.730104+0000","last_fullsized":"2024-02-19T14:53:47.730104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728795+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:47.728795+0000","last_peered":"2024-02-19T14:53:47.728795+0000","last_clean":"2024-02-19T14:53:47.728795+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:47.728795+0000","last_undegraded":"2024-02-19T14:53:47.728795+0000","last_fullsized":"2024-02-19T14:53:47.728795+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728755+0000","last_change":"2024-02-19T14:53:47.728755+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728755+0000","last_undegraded":"2024-02-19T14:53:47.728755+0000","last_fullsized":"2024-02-19T14:53:47.728755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746985+0000","last_change":"2024-02-19T14:53:47.744368+0000","last_active":"2024-02-19T14:53:47.746985+0000","last_peered":"2024-02-19T14:53:47.746985+0000","last_clean":"2024-02-19T14:53:47.746985+0000","last_became_active":"2024-02-19T14:53:47.743379+0000","last_became_peered":"2024-02-19T14:53:47.743379+0000","last_unstale":"2024-02-19T14:53:47.746985+0000","last_undegraded":"2024-02-19T14:53:47.746985+0000","last_fullsized":"2024-02-19T14:53:47.746985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724152+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:47.724152+0000","last_peered":"2024-02-19T14:53:47.724152+0000","last_clean":"2024-02-19T14:53:47.724152+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:47.724152+0000","last_undegraded":"2024-02-19T14:53:47.724152+0000","last_fullsized":"2024-02-19T14:53:47.724152+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:41.307928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737668+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:47.737668+0000","last_peered":"2024-02-19T14:53:47.737668+0000","last_clean":"2024-02-19T14:53:47.737668+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:47.737668+0000","last_undegraded":"2024-02-19T14:53:47.737668+0000","last_fullsized":"2024-02-19T14:53:47.737668+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726342+0000","last_change":"2024-02-19T14:53:47.726342+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726342+0000","last_undegraded":"2024-02-19T14:53:47.726342+0000","last_fullsized":"2024-02-19T14:53:47.726342+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747509+0000","last_change":"2024-02-19T14:53:47.744422+0000","last_active":"2024-02-19T14:53:47.747509+0000","last_peered":"2024-02-19T14:53:47.747509+0000","last_clean":"2024-02-19T14:53:47.747509+0000","last_became_active":"2024-02-19T14:53:47.744122+0000","last_became_peered":"2024-02-19T14:53:47.744122+0000","last_unstale":"2024-02-19T14:53:47.747509+0000","last_undegraded":"2024-02-19T14:53:47.747509+0000","last_fullsized":"2024-02-19T14:53:47.747509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:50.594221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730639+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:47.730639+0000","last_peered":"2024-02-19T14:53:47.730639+0000","last_clean":"2024-02-19T14:53:47.730639+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:47.730639+0000","last_undegraded":"2024-02-19T14:53:47.730639+0000","last_fullsized":"2024-02-19T14:53:47.730639+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730829+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:47.730829+0000","last_peered":"2024-02-19T14:53:47.730829+0000","last_clean":"2024-02-19T14:53:47.730829+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:47.730829+0000","last_undegraded":"2024-02-19T14:53:47.730829+0000","last_fullsized":"2024-02-19T14:53:47.730829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728855+0000","last_change":"2024-02-19T14:53:47.728855+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728855+0000","last_undegraded":"2024-02-19T14:53:47.728855+0000","last_fullsized":"2024-02-19T14:53:47.728855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746360+0000","last_change":"2024-02-19T14:53:47.745353+0000","last_active":"2024-02-19T14:53:47.746360+0000","last_peered":"2024-02-19T14:53:47.746360+0000","last_clean":"2024-02-19T14:53:47.746360+0000","last_became_active":"2024-02-19T14:53:47.745008+0000","last_became_peered":"2024-02-19T14:53:47.745008+0000","last_unstale":"2024-02-19T14:53:47.746360+0000","last_undegraded":"2024-02-19T14:53:47.746360+0000","last_fullsized":"2024-02-19T14:53:47.746360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:29.660730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729669+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:47.729669+0000","last_peered":"2024-02-19T14:53:47.729669+0000","last_clean":"2024-02-19T14:53:47.729669+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:47.729669+0000","last_undegraded":"2024-02-19T14:53:47.729669+0000","last_fullsized":"2024-02-19T14:53:47.729669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730746+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:47.730746+0000","last_peered":"2024-02-19T14:53:47.730746+0000","last_clean":"2024-02-19T14:53:47.730746+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:47.730746+0000","last_undegraded":"2024-02-19T14:53:47.730746+0000","last_fullsized":"2024-02-19T14:53:47.730746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726359+0000","last_change":"2024-02-19T14:53:47.726359+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726359+0000","last_undegraded":"2024-02-19T14:53:47.726359+0000","last_fullsized":"2024-02-19T14:53:47.726359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746332+0000","last_change":"2024-02-19T14:53:47.742114+0000","last_active":"2024-02-19T14:53:47.746332+0000","last_peered":"2024-02-19T14:53:47.746332+0000","last_clean":"2024-02-19T14:53:47.746332+0000","last_became_active":"2024-02-19T14:53:47.741840+0000","last_became_peered":"2024-02-19T14:53:47.741840+0000","last_unstale":"2024-02-19T14:53:47.746332+0000","last_undegraded":"2024-02-19T14:53:47.746332+0000","last_fullsized":"2024-02-19T14:53:47.746332+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:20.795346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727709+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:47.727709+0000","last_peered":"2024-02-19T14:53:47.727709+0000","last_clean":"2024-02-19T14:53:47.727709+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:47.727709+0000","last_undegraded":"2024-02-19T14:53:47.727709+0000","last_fullsized":"2024-02-19T14:53:47.727709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.736179+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:47.736179+0000","last_peered":"2024-02-19T14:53:47.736179+0000","last_clean":"2024-02-19T14:53:47.736179+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:47.736179+0000","last_undegraded":"2024-02-19T14:53:47.736179+0000","last_fullsized":"2024-02-19T14:53:47.736179+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731814+0000","last_change":"2024-02-19T14:53:47.731814+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731814+0000","last_undegraded":"2024-02-19T14:53:47.731814+0000","last_fullsized":"2024-02-19T14:53:47.731814+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737453+0000","last_change":"2024-02-19T14:53:47.734753+0000","last_active":"2024-02-19T14:53:47.737453+0000","last_peered":"2024-02-19T14:53:47.737453+0000","last_clean":"2024-02-19T14:53:47.737453+0000","last_became_active":"2024-02-19T14:53:47.734515+0000","last_became_peered":"2024-02-19T14:53:47.734515+0000","last_unstale":"2024-02-19T14:53:47.737453+0000","last_undegraded":"2024-02-19T14:53:47.737453+0000","last_fullsized":"2024-02-19T14:53:47.737453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:02:56.724991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730223+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:47.730223+0000","last_peered":"2024-02-19T14:53:47.730223+0000","last_clean":"2024-02-19T14:53:47.730223+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:47.730223+0000","last_undegraded":"2024-02-19T14:53:47.730223+0000","last_fullsized":"2024-02-19T14:53:47.730223+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":25,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731281+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:47.731281+0000","last_peered":"2024-02-19T14:53:47.731281+0000","last_clean":"2024-02-19T14:53:47.731281+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:47.731281+0000","last_undegraded":"2024-02-19T14:53:47.731281+0000","last_fullsized":"2024-02-19T14:53:47.731281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727760+0000","last_change":"2024-02-19T14:53:47.727760+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727760+0000","last_undegraded":"2024-02-19T14:53:47.727760+0000","last_fullsized":"2024-02-19T14:53:47.727760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737793+0000","last_change":"2024-02-19T14:53:47.735759+0000","last_active":"2024-02-19T14:53:47.737793+0000","last_peered":"2024-02-19T14:53:47.737793+0000","last_clean":"2024-02-19T14:53:47.737793+0000","last_became_active":"2024-02-19T14:53:47.735503+0000","last_became_peered":"2024-02-19T14:53:47.735503+0000","last_unstale":"2024-02-19T14:53:47.737793+0000","last_undegraded":"2024-02-19T14:53:47.737793+0000","last_fullsized":"2024-02-19T14:53:47.737793+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:35.189687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731195+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:47.731195+0000","last_peered":"2024-02-19T14:53:47.731195+0000","last_clean":"2024-02-19T14:53:47.731195+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:47.731195+0000","last_undegraded":"2024-02-19T14:53:47.731195+0000","last_fullsized":"2024-02-19T14:53:47.731195+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":28,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724746+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:47.724746+0000","last_peered":"2024-02-19T14:53:47.724746+0000","last_clean":"2024-02-19T14:53:47.724746+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:47.724746+0000","last_undegraded":"2024-02-19T14:53:47.724746+0000","last_fullsized":"2024-02-19T14:53:47.724746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731071+0000","last_change":"2024-02-19T14:53:47.731071+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731071+0000","last_undegraded":"2024-02-19T14:53:47.731071+0000","last_fullsized":"2024-02-19T14:53:47.731071+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746268+0000","last_change":"2024-02-19T14:53:47.742801+0000","last_active":"2024-02-19T14:53:47.746268+0000","last_peered":"2024-02-19T14:53:47.746268+0000","last_clean":"2024-02-19T14:53:47.746268+0000","last_became_active":"2024-02-19T14:53:47.742534+0000","last_became_peered":"2024-02-19T14:53:47.742534+0000","last_unstale":"2024-02-19T14:53:47.746268+0000","last_undegraded":"2024-02-19T14:53:47.746268+0000","last_fullsized":"2024-02-19T14:53:47.746268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:59.453875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733273+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:47.733273+0000","last_peered":"2024-02-19T14:53:47.733273+0000","last_clean":"2024-02-19T14:53:47.733273+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:47.733273+0000","last_undegraded":"2024-02-19T14:53:47.733273+0000","last_fullsized":"2024-02-19T14:53:47.733273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732699+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:47.732699+0000","last_peered":"2024-02-19T14:53:47.732699+0000","last_clean":"2024-02-19T14:53:47.732699+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:47.732699+0000","last_undegraded":"2024-02-19T14:53:47.732699+0000","last_fullsized":"2024-02-19T14:53:47.732699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726717+0000","last_change":"2024-02-19T14:53:47.726717+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726717+0000","last_undegraded":"2024-02-19T14:53:47.726717+0000","last_fullsized":"2024-02-19T14:53:47.726717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748938+0000","last_change":"2024-02-19T14:53:47.744212+0000","last_active":"2024-02-19T14:53:47.748938+0000","last_peered":"2024-02-19T14:53:47.748938+0000","last_clean":"2024-02-19T14:53:47.748938+0000","last_became_active":"2024-02-19T14:53:47.743916+0000","last_became_peered":"2024-02-19T14:53:47.743916+0000","last_unstale":"2024-02-19T14:53:47.748938+0000","last_undegraded":"2024-02-19T14:53:47.748938+0000","last_fullsized":"2024-02-19T14:53:47.748938+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732440+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:47.732440+0000","last_peered":"2024-02-19T14:53:47.732440+0000","last_clean":"2024-02-19T14:53:47.732440+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:47.732440+0000","last_undegraded":"2024-02-19T14:53:47.732440+0000","last_fullsized":"2024-02-19T14:53:47.732440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739733+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:47.739733+0000","last_peered":"2024-02-19T14:53:47.739733+0000","last_clean":"2024-02-19T14:53:47.739733+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:47.739733+0000","last_undegraded":"2024-02-19T14:53:47.739733+0000","last_fullsized":"2024-02-19T14:53:47.739733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731538+0000","last_change":"2024-02-19T14:53:47.731538+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731538+0000","last_undegraded":"2024-02-19T14:53:47.731538+0000","last_fullsized":"2024-02-19T14:53:47.731538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.744120+0000","last_change":"2024-02-19T14:53:47.741970+0000","last_active":"2024-02-19T14:53:47.744120+0000","last_peered":"2024-02-19T14:53:47.744120+0000","last_clean":"2024-02-19T14:53:47.744120+0000","last_became_active":"2024-02-19T14:53:47.741707+0000","last_became_peered":"2024-02-19T14:53:47.741707+0000","last_unstale":"2024-02-19T14:53:47.744120+0000","last_undegraded":"2024-02-19T14:53:47.744120+0000","last_fullsized":"2024-02-19T14:53:47.744120+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:26.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724087+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:47.724087+0000","last_peered":"2024-02-19T14:53:47.724087+0000","last_clean":"2024-02-19T14:53:47.724087+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:47.724087+0000","last_undegraded":"2024-02-19T14:53:47.724087+0000","last_fullsized":"2024-02-19T14:53:47.724087+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729186+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:47.729186+0000","last_peered":"2024-02-19T14:53:47.729186+0000","last_clean":"2024-02-19T14:53:47.729186+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:47.729186+0000","last_undegraded":"2024-02-19T14:53:47.729186+0000","last_fullsized":"2024-02-19T14:53:47.729186+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734244+0000","last_change":"2024-02-19T14:53:47.734244+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734244+0000","last_undegraded":"2024-02-19T14:53:47.734244+0000","last_fullsized":"2024-02-19T14:53:47.734244+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743671+0000","last_change":"2024-02-19T14:53:47.742140+0000","last_active":"2024-02-19T14:53:47.743671+0000","last_peered":"2024-02-19T14:53:47.743671+0000","last_clean":"2024-02-19T14:53:47.743671+0000","last_became_active":"2024-02-19T14:53:47.741838+0000","last_became_peered":"2024-02-19T14:53:47.741838+0000","last_unstale":"2024-02-19T14:53:47.743671+0000","last_undegraded":"2024-02-19T14:53:47.743671+0000","last_fullsized":"2024-02-19T14:53:47.743671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:55.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737818+0000","last_change":"2024-02-19T14:53:41.706063+0000","last_active":"2024-02-19T14:53:47.737818+0000","last_peered":"2024-02-19T14:53:47.737818+0000","last_clean":"2024-02-19T14:53:47.737818+0000","last_became_active":"2024-02-19T14:53:41.705766+0000","last_became_peered":"2024-02-19T14:53:41.705766+0000","last_unstale":"2024-02-19T14:53:47.737818+0000","last_undegraded":"2024-02-19T14:53:47.737818+0000","last_fullsized":"2024-02-19T14:53:47.737818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":30,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723730+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:47.723730+0000","last_peered":"2024-02-19T14:53:47.723730+0000","last_clean":"2024-02-19T14:53:47.723730+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:47.723730+0000","last_undegraded":"2024-02-19T14:53:47.723730+0000","last_fullsized":"2024-02-19T14:53:47.723730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727027+0000","last_change":"2024-02-19T14:53:47.727027+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727027+0000","last_undegraded":"2024-02-19T14:53:47.727027+0000","last_fullsized":"2024-02-19T14:53:47.727027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743702+0000","last_change":"2024-02-19T14:53:47.742196+0000","last_active":"2024-02-19T14:53:47.743702+0000","last_peered":"2024-02-19T14:53:47.743702+0000","last_clean":"2024-02-19T14:53:47.743702+0000","last_became_active":"2024-02-19T14:53:47.741852+0000","last_became_peered":"2024-02-19T14:53:47.741852+0000","last_unstale":"2024-02-19T14:53:47.743702+0000","last_undegraded":"2024-02-19T14:53:47.743702+0000","last_fullsized":"2024-02-19T14:53:47.743702+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:31.942230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724625+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:47.724625+0000","last_peered":"2024-02-19T14:53:47.724625+0000","last_clean":"2024-02-19T14:53:47.724625+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:47.724625+0000","last_undegraded":"2024-02-19T14:53:47.724625+0000","last_fullsized":"2024-02-19T14:53:47.724625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734529+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:47.734529+0000","last_peered":"2024-02-19T14:53:47.734529+0000","last_clean":"2024-02-19T14:53:47.734529+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:47.734529+0000","last_undegraded":"2024-02-19T14:53:47.734529+0000","last_fullsized":"2024-02-19T14:53:47.734529+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730482+0000","last_change":"2024-02-19T14:53:47.730482+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730482+0000","last_undegraded":"2024-02-19T14:53:47.730482+0000","last_fullsized":"2024-02-19T14:53:47.730482+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.749226+0000","last_change":"2024-02-19T14:53:47.745519+0000","last_active":"2024-02-19T14:53:47.749226+0000","last_peered":"2024-02-19T14:53:47.749226+0000","last_clean":"2024-02-19T14:53:47.749226+0000","last_became_active":"2024-02-19T14:53:47.745263+0000","last_became_peered":"2024-02-19T14:53:47.745263+0000","last_unstale":"2024-02-19T14:53:47.749226+0000","last_undegraded":"2024-02-19T14:53:47.749226+0000","last_fullsized":"2024-02-19T14:53:47.749226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:51.861192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728027+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:47.728027+0000","last_peered":"2024-02-19T14:53:47.728027+0000","last_clean":"2024-02-19T14:53:47.728027+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:47.728027+0000","last_undegraded":"2024-02-19T14:53:47.728027+0000","last_fullsized":"2024-02-19T14:53:47.728027+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733299+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:47.733299+0000","last_peered":"2024-02-19T14:53:47.733299+0000","last_clean":"2024-02-19T14:53:47.733299+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:47.733299+0000","last_undegraded":"2024-02-19T14:53:47.733299+0000","last_fullsized":"2024-02-19T14:53:47.733299+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728440+0000","last_change":"2024-02-19T14:53:47.728440+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728440+0000","last_undegraded":"2024-02-19T14:53:47.728440+0000","last_fullsized":"2024-02-19T14:53:47.728440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.741263+0000","last_change":"2024-02-19T14:53:47.736640+0000","last_active":"2024-02-19T14:53:47.741263+0000","last_peered":"2024-02-19T14:53:47.741263+0000","last_clean":"2024-02-19T14:53:47.741263+0000","last_became_active":"2024-02-19T14:53:47.735150+0000","last_became_peered":"2024-02-19T14:53:47.735150+0000","last_unstale":"2024-02-19T14:53:47.741263+0000","last_undegraded":"2024-02-19T14:53:47.741263+0000","last_fullsized":"2024-02-19T14:53:47.741263+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:02.655336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728667+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:47.728667+0000","last_peered":"2024-02-19T14:53:47.728667+0000","last_clean":"2024-02-19T14:53:47.728667+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:47.728667+0000","last_undegraded":"2024-02-19T14:53:47.728667+0000","last_fullsized":"2024-02-19T14:53:47.728667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737687+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:47.737687+0000","last_peered":"2024-02-19T14:53:47.737687+0000","last_clean":"2024-02-19T14:53:47.737687+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:47.737687+0000","last_undegraded":"2024-02-19T14:53:47.737687+0000","last_fullsized":"2024-02-19T14:53:47.737687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730447+0000","last_change":"2024-02-19T14:53:47.730447+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730447+0000","last_undegraded":"2024-02-19T14:53:47.730447+0000","last_fullsized":"2024-02-19T14:53:47.730447+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734187+0000","last_change":"2024-02-19T14:53:47.730313+0000","last_active":"2024-02-19T14:53:47.734187+0000","last_peered":"2024-02-19T14:53:47.734187+0000","last_clean":"2024-02-19T14:53:47.734187+0000","last_became_active":"2024-02-19T14:53:47.730053+0000","last_became_peered":"2024-02-19T14:53:47.730053+0000","last_unstale":"2024-02-19T14:53:47.734187+0000","last_undegraded":"2024-02-19T14:53:47.734187+0000","last_fullsized":"2024-02-19T14:53:47.734187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729153+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:47.729153+0000","last_peered":"2024-02-19T14:53:47.729153+0000","last_clean":"2024-02-19T14:53:47.729153+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:47.729153+0000","last_undegraded":"2024-02-19T14:53:47.729153+0000","last_fullsized":"2024-02-19T14:53:47.729153+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737838+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:47.737838+0000","last_peered":"2024-02-19T14:53:47.737838+0000","last_clean":"2024-02-19T14:53:47.737838+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:47.737838+0000","last_undegraded":"2024-02-19T14:53:47.737838+0000","last_fullsized":"2024-02-19T14:53:47.737838+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728393+0000","last_change":"2024-02-19T14:53:47.728393+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728393+0000","last_undegraded":"2024-02-19T14:53:47.728393+0000","last_fullsized":"2024-02-19T14:53:47.728393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746290+0000","last_change":"2024-02-19T14:53:47.742841+0000","last_active":"2024-02-19T14:53:47.746290+0000","last_peered":"2024-02-19T14:53:47.746290+0000","last_clean":"2024-02-19T14:53:47.746290+0000","last_became_active":"2024-02-19T14:53:47.742581+0000","last_became_peered":"2024-02-19T14:53:47.742581+0000","last_unstale":"2024-02-19T14:53:47.746290+0000","last_undegraded":"2024-02-19T14:53:47.746290+0000","last_fullsized":"2024-02-19T14:53:47.746290+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:57.701790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733388+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:47.733388+0000","last_peered":"2024-02-19T14:53:47.733388+0000","last_clean":"2024-02-19T14:53:47.733388+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:47.733388+0000","last_undegraded":"2024-02-19T14:53:47.733388+0000","last_fullsized":"2024-02-19T14:53:47.733388+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729246+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:47.729246+0000","last_peered":"2024-02-19T14:53:47.729246+0000","last_clean":"2024-02-19T14:53:47.729246+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:47.729246+0000","last_undegraded":"2024-02-19T14:53:47.729246+0000","last_fullsized":"2024-02-19T14:53:47.729246+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731908+0000","last_change":"2024-02-19T14:53:47.731908+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731908+0000","last_undegraded":"2024-02-19T14:53:47.731908+0000","last_fullsized":"2024-02-19T14:53:47.731908+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746114+0000","last_change":"2024-02-19T14:53:47.743491+0000","last_active":"2024-02-19T14:53:47.746114+0000","last_peered":"2024-02-19T14:53:47.746114+0000","last_clean":"2024-02-19T14:53:47.746114+0000","last_became_active":"2024-02-19T14:53:47.743209+0000","last_became_peered":"2024-02-19T14:53:47.743209+0000","last_unstale":"2024-02-19T14:53:47.746114+0000","last_undegraded":"2024-02-19T14:53:47.746114+0000","last_fullsized":"2024-02-19T14:53:47.746114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:51.151311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737504+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:47.737504+0000","last_peered":"2024-02-19T14:53:47.737504+0000","last_clean":"2024-02-19T14:53:47.737504+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:47.737504+0000","last_undegraded":"2024-02-19T14:53:47.737504+0000","last_fullsized":"2024-02-19T14:53:47.737504+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724055+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:47.724055+0000","last_peered":"2024-02-19T14:53:47.724055+0000","last_clean":"2024-02-19T14:53:47.724055+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:47.724055+0000","last_undegraded":"2024-02-19T14:53:47.724055+0000","last_fullsized":"2024-02-19T14:53:47.724055+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728116+0000","last_change":"2024-02-19T14:53:47.728116+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728116+0000","last_undegraded":"2024-02-19T14:53:47.728116+0000","last_fullsized":"2024-02-19T14:53:47.728116+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747934+0000","last_change":"2024-02-19T14:53:47.740831+0000","last_active":"2024-02-19T14:53:47.747934+0000","last_peered":"2024-02-19T14:53:47.747934+0000","last_clean":"2024-02-19T14:53:47.747934+0000","last_became_active":"2024-02-19T14:53:47.740540+0000","last_became_peered":"2024-02-19T14:53:47.740540+0000","last_unstale":"2024-02-19T14:53:47.747934+0000","last_undegraded":"2024-02-19T14:53:47.747934+0000","last_fullsized":"2024-02-19T14:53:47.747934+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733979+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:47.733979+0000","last_peered":"2024-02-19T14:53:47.733979+0000","last_clean":"2024-02-19T14:53:47.733979+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:47.733979+0000","last_undegraded":"2024-02-19T14:53:47.733979+0000","last_fullsized":"2024-02-19T14:53:47.733979+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731528+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:47.731528+0000","last_peered":"2024-02-19T14:53:47.731528+0000","last_clean":"2024-02-19T14:53:47.731528+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:47.731528+0000","last_undegraded":"2024-02-19T14:53:47.731528+0000","last_fullsized":"2024-02-19T14:53:47.731528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731324+0000","last_change":"2024-02-19T14:53:47.731324+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731324+0000","last_undegraded":"2024-02-19T14:53:47.731324+0000","last_fullsized":"2024-02-19T14:53:47.731324+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.740896+0000","last_change":"2024-02-19T14:53:47.736026+0000","last_active":"2024-02-19T14:53:47.740896+0000","last_peered":"2024-02-19T14:53:47.740896+0000","last_clean":"2024-02-19T14:53:47.740896+0000","last_became_active":"2024-02-19T14:53:47.735680+0000","last_became_peered":"2024-02-19T14:53:47.735680+0000","last_unstale":"2024-02-19T14:53:47.740896+0000","last_undegraded":"2024-02-19T14:53:47.740896+0000","last_fullsized":"2024-02-19T14:53:47.740896+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730749+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:47.730749+0000","last_peered":"2024-02-19T14:53:47.730749+0000","last_clean":"2024-02-19T14:53:47.730749+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:47.730749+0000","last_undegraded":"2024-02-19T14:53:47.730749+0000","last_fullsized":"2024-02-19T14:53:47.730749+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732455+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:47.732455+0000","last_peered":"2024-02-19T14:53:47.732455+0000","last_clean":"2024-02-19T14:53:47.732455+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:47.732455+0000","last_undegraded":"2024-02-19T14:53:47.732455+0000","last_fullsized":"2024-02-19T14:53:47.732455+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731805+0000","last_change":"2024-02-19T14:53:47.731805+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731805+0000","last_undegraded":"2024-02-19T14:53:47.731805+0000","last_fullsized":"2024-02-19T14:53:47.731805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743744+0000","last_change":"2024-02-19T14:53:47.737977+0000","last_active":"2024-02-19T14:53:47.743744+0000","last_peered":"2024-02-19T14:53:47.743744+0000","last_clean":"2024-02-19T14:53:47.743744+0000","last_became_active":"2024-02-19T14:53:47.737709+0000","last_became_peered":"2024-02-19T14:53:47.737709+0000","last_unstale":"2024-02-19T14:53:47.743744+0000","last_undegraded":"2024-02-19T14:53:47.743744+0000","last_fullsized":"2024-02-19T14:53:47.743744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:00.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734277+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:47.734277+0000","last_peered":"2024-02-19T14:53:47.734277+0000","last_clean":"2024-02-19T14:53:47.734277+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:47.734277+0000","last_undegraded":"2024-02-19T14:53:47.734277+0000","last_fullsized":"2024-02-19T14:53:47.734277+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739764+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:47.739764+0000","last_peered":"2024-02-19T14:53:47.739764+0000","last_clean":"2024-02-19T14:53:47.739764+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:47.739764+0000","last_undegraded":"2024-02-19T14:53:47.739764+0000","last_fullsized":"2024-02-19T14:53:47.739764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728207+0000","last_change":"2024-02-19T14:53:47.728207+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728207+0000","last_undegraded":"2024-02-19T14:53:47.728207+0000","last_fullsized":"2024-02-19T14:53:47.728207+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748964+0000","last_change":"2024-02-19T14:53:47.745205+0000","last_active":"2024-02-19T14:53:47.748964+0000","last_peered":"2024-02-19T14:53:47.748964+0000","last_clean":"2024-02-19T14:53:47.748964+0000","last_became_active":"2024-02-19T14:53:47.744958+0000","last_became_peered":"2024-02-19T14:53:47.744958+0000","last_unstale":"2024-02-19T14:53:47.748964+0000","last_undegraded":"2024-02-19T14:53:47.748964+0000","last_fullsized":"2024-02-19T14:53:47.748964+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729468+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:47.729468+0000","last_peered":"2024-02-19T14:53:47.729468+0000","last_clean":"2024-02-19T14:53:47.729468+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:47.729468+0000","last_undegraded":"2024-02-19T14:53:47.729468+0000","last_fullsized":"2024-02-19T14:53:47.729468+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731445+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:47.731445+0000","last_peered":"2024-02-19T14:53:47.731445+0000","last_clean":"2024-02-19T14:53:47.731445+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:47.731445+0000","last_undegraded":"2024-02-19T14:53:47.731445+0000","last_fullsized":"2024-02-19T14:53:47.731445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727406+0000","last_change":"2024-02-19T14:53:47.727406+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727406+0000","last_undegraded":"2024-02-19T14:53:47.727406+0000","last_fullsized":"2024-02-19T14:53:47.727406+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747521+0000","last_change":"2024-02-19T14:53:47.740210+0000","last_active":"2024-02-19T14:53:47.747521+0000","last_peered":"2024-02-19T14:53:47.747521+0000","last_clean":"2024-02-19T14:53:47.747521+0000","last_became_active":"2024-02-19T14:53:47.739913+0000","last_became_peered":"2024-02-19T14:53:47.739913+0000","last_unstale":"2024-02-19T14:53:47.747521+0000","last_undegraded":"2024-02-19T14:53:47.747521+0000","last_fullsized":"2024-02-19T14:53:47.747521+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723813+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:47.723813+0000","last_peered":"2024-02-19T14:53:47.723813+0000","last_clean":"2024-02-19T14:53:47.723813+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:47.723813+0000","last_undegraded":"2024-02-19T14:53:47.723813+0000","last_fullsized":"2024-02-19T14:53:47.723813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:10.864244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732089+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:47.732089+0000","last_peered":"2024-02-19T14:53:47.732089+0000","last_clean":"2024-02-19T14:53:47.732089+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:47.732089+0000","last_undegraded":"2024-02-19T14:53:47.732089+0000","last_fullsized":"2024-02-19T14:53:47.732089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735192+0000","last_change":"2024-02-19T14:53:47.735192+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735192+0000","last_undegraded":"2024-02-19T14:53:47.735192+0000","last_fullsized":"2024-02-19T14:53:47.735192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.745944+0000","last_change":"2024-02-19T14:53:47.738871+0000","last_active":"2024-02-19T14:53:47.745944+0000","last_peered":"2024-02-19T14:53:47.745944+0000","last_clean":"2024-02-19T14:53:47.745944+0000","last_became_active":"2024-02-19T14:53:47.738585+0000","last_became_peered":"2024-02-19T14:53:47.738585+0000","last_unstale":"2024-02-19T14:53:47.745944+0000","last_undegraded":"2024-02-19T14:53:47.745944+0000","last_fullsized":"2024-02-19T14:53:47.745944+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:02.790030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731936+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:47.731936+0000","last_peered":"2024-02-19T14:53:47.731936+0000","last_clean":"2024-02-19T14:53:47.731936+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:47.731936+0000","last_undegraded":"2024-02-19T14:53:47.731936+0000","last_fullsized":"2024-02-19T14:53:47.731936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732414+0000","last_change":"2024-02-19T14:53:47.732414+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732414+0000","last_undegraded":"2024-02-19T14:53:47.732414+0000","last_fullsized":"2024-02-19T14:53:47.732414+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743039+0000","last_change":"2024-02-19T14:53:47.738834+0000","last_active":"2024-02-19T14:53:47.743039+0000","last_peered":"2024-02-19T14:53:47.743039+0000","last_clean":"2024-02-19T14:53:47.743039+0000","last_became_active":"2024-02-19T14:53:47.738567+0000","last_became_peered":"2024-02-19T14:53:47.738567+0000","last_unstale":"2024-02-19T14:53:47.743039+0000","last_undegraded":"2024-02-19T14:53:47.743039+0000","last_fullsized":"2024-02-19T14:53:47.743039+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:50.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727552+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:47.727552+0000","last_peered":"2024-02-19T14:53:47.727552+0000","last_clean":"2024-02-19T14:53:47.727552+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:47.727552+0000","last_undegraded":"2024-02-19T14:53:47.727552+0000","last_fullsized":"2024-02-19T14:53:47.727552+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724124+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:47.724124+0000","last_peered":"2024-02-19T14:53:47.724124+0000","last_clean":"2024-02-19T14:53:47.724124+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:47.724124+0000","last_undegraded":"2024-02-19T14:53:47.724124+0000","last_fullsized":"2024-02-19T14:53:47.724124+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728013+0000","last_change":"2024-02-19T14:53:47.728013+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728013+0000","last_undegraded":"2024-02-19T14:53:47.728013+0000","last_fullsized":"2024-02-19T14:53:47.728013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747961+0000","last_change":"2024-02-19T14:53:47.744007+0000","last_active":"2024-02-19T14:53:47.747961+0000","last_peered":"2024-02-19T14:53:47.747961+0000","last_clean":"2024-02-19T14:53:47.747961+0000","last_became_active":"2024-02-19T14:53:47.743726+0000","last_became_peered":"2024-02-19T14:53:47.743726+0000","last_unstale":"2024-02-19T14:53:47.747961+0000","last_undegraded":"2024-02-19T14:53:47.747961+0000","last_fullsized":"2024-02-19T14:53:47.747961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:26.617672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733834+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:47.733834+0000","last_peered":"2024-02-19T14:53:47.733834+0000","last_clean":"2024-02-19T14:53:47.733834+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:47.733834+0000","last_undegraded":"2024-02-19T14:53:47.733834+0000","last_fullsized":"2024-02-19T14:53:47.733834+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730733+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:47.730733+0000","last_peered":"2024-02-19T14:53:47.730733+0000","last_clean":"2024-02-19T14:53:47.730733+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:47.730733+0000","last_undegraded":"2024-02-19T14:53:47.730733+0000","last_fullsized":"2024-02-19T14:53:47.730733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732525+0000","last_change":"2024-02-19T14:53:47.732525+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732525+0000","last_undegraded":"2024-02-19T14:53:47.732525+0000","last_fullsized":"2024-02-19T14:53:47.732525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746917+0000","last_change":"2024-02-19T14:53:47.742015+0000","last_active":"2024-02-19T14:53:47.746917+0000","last_peered":"2024-02-19T14:53:47.746917+0000","last_clean":"2024-02-19T14:53:47.746917+0000","last_became_active":"2024-02-19T14:53:47.741742+0000","last_became_peered":"2024-02-19T14:53:47.741742+0000","last_unstale":"2024-02-19T14:53:47.746917+0000","last_undegraded":"2024-02-19T14:53:47.746917+0000","last_fullsized":"2024-02-19T14:53:47.746917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724348+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:47.724348+0000","last_peered":"2024-02-19T14:53:47.724348+0000","last_clean":"2024-02-19T14:53:47.724348+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:47.724348+0000","last_undegraded":"2024-02-19T14:53:47.724348+0000","last_fullsized":"2024-02-19T14:53:47.724348+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:45.189873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734386+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:47.734386+0000","last_peered":"2024-02-19T14:53:47.734386+0000","last_clean":"2024-02-19T14:53:47.734386+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:47.734386+0000","last_undegraded":"2024-02-19T14:53:47.734386+0000","last_fullsized":"2024-02-19T14:53:47.734386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728949+0000","last_change":"2024-02-19T14:53:47.728949+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728949+0000","last_undegraded":"2024-02-19T14:53:47.728949+0000","last_fullsized":"2024-02-19T14:53:47.728949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737489+0000","last_change":"2024-02-19T14:53:47.734524+0000","last_active":"2024-02-19T14:53:47.737489+0000","last_peered":"2024-02-19T14:53:47.737489+0000","last_clean":"2024-02-19T14:53:47.737489+0000","last_became_active":"2024-02-19T14:53:47.734132+0000","last_became_peered":"2024-02-19T14:53:47.734132+0000","last_unstale":"2024-02-19T14:53:47.737489+0000","last_undegraded":"2024-02-19T14:53:47.737489+0000","last_fullsized":"2024-02-19T14:53:47.737489+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737553+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:47.737553+0000","last_peered":"2024-02-19T14:53:47.737553+0000","last_clean":"2024-02-19T14:53:47.737553+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:47.737553+0000","last_undegraded":"2024-02-19T14:53:47.737553+0000","last_fullsized":"2024-02-19T14:53:47.737553+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734066+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:47.734066+0000","last_peered":"2024-02-19T14:53:47.734066+0000","last_clean":"2024-02-19T14:53:47.734066+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:47.734066+0000","last_undegraded":"2024-02-19T14:53:47.734066+0000","last_fullsized":"2024-02-19T14:53:47.734066+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":236,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729312+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:47.729312+0000","last_peered":"2024-02-19T14:53:47.729312+0000","last_clean":"2024-02-19T14:53:47.729312+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:47.729312+0000","last_undegraded":"2024-02-19T14:53:47.729312+0000","last_fullsized":"2024-02-19T14:53:47.729312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.502},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":291530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.502}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.626},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.311},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":299032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.624},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":292512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.245}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.372}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.621}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":292490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.31}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.5}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":298610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.247},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.20000000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.217},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.435},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.36399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.42399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:52.288 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:52.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.418+0000 7f335641c640 1 Processor -- start 2024-02-19T14:53:52.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.418+0000 7f335641c640 1 -- start start 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f335641c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33500add70 0x7f33500c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f335641c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500705b0 0x7f3350070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f335641c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33500ae670 con 0x7f33500c2f60 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f335641c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33500716b0 con 0x7f33500add70 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f335641c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33500c33d0 con 0x7f33500705b0 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f334ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33500add70 0x7f33500c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f334f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f33500c2f60 0x7f335006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60334/0 (socket says 172.21.15.78:60334) 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f334f7fe640 1 -- 172.21.15.78:0/3259672082 learned_addr learned my addr 172.21.15.78:0/3259672082 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.419+0000 7f33549b9640 1 --2- 172.21.15.78:0/3259672082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500705b0 0x7f3350070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.420+0000 7f334ffff640 1 -- 172.21.15.78:0/3259672082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500705b0 msgr2=0x7f3350070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.420+0000 7f334ffff640 1 --2- 172.21.15.78:0/3259672082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500705b0 0x7f3350070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.420+0000 7f334ffff640 1 -- 172.21.15.78:0/3259672082 >> v1:172.21.15.78:6789/0 conn(0x7f33500c2f60 legacy=0x7f335006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.420+0000 7f334ffff640 1 -- 172.21.15.78:0/3259672082 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3350153780 con 0x7f33500add70 2024-02-19T14:53:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.420+0000 7f334effd640 1 -- 172.21.15.78:0/3259672082 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324046899 0 0) 0x7f33500ae670 con 0x7f33500c2f60 2024-02-19T14:53:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f334ffff640 1 --2- 172.21.15.78:0/3259672082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500add70 0x7f33500c4980 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f33440097a0 tx=0x7f334402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abe178f6102913f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f334effd640 1 -- 172.21.15.78:0/3259672082 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334402e070 con 0x7f33500add70 2024-02-19T14:53:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f334effd640 1 -- 172.21.15.78:0/3259672082 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33440315a0 con 0x7f33500add70 2024-02-19T14:53:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f334effd640 1 -- 172.21.15.78:0/3259672082 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3344031930 con 0x7f33500add70 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 -- 172.21.15.78:0/3259672082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500add70 msgr2=0x7f33500c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 --2- 172.21.15.78:0/3259672082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500add70 0x7f33500c4980 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f33440097a0 tx=0x7f334402b840 comp rx=0 tx=0).stop 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 -- 172.21.15.78:0/3259672082 shutdown_connections 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 --2- 172.21.15.78:0/3259672082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500705b0 0x7f3350070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 --2- 172.21.15.78:0/3259672082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500add70 0x7f33500c4980 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 -- 172.21.15.78:0/3259672082 >> 172.21.15.78:0/3259672082 conn(0x7f33500c85e0 msgr2=0x7f33500c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 -- 172.21.15.78:0/3259672082 shutdown_connections 2024-02-19T14:53:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.421+0000 7f335641c640 1 -- 172.21.15.78:0/3259672082 wait complete. 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.422+0000 7f335641c640 1 Processor -- start 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 -- start start 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 0x7f33500a81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500add70 0x7f3350156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33500c2f60 0x7f3350156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33500bdbd0 con 0x7f33500705b0 2024-02-19T14:53:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3350155180 con 0x7f33500c2f60 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f335641c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3350154210 con 0x7f33500add70 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f334ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 0x7f33500a81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f334ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 0x7f33500a81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53422/0 (socket says 172.21.15.78:53422) 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.423+0000 7f334ffff640 1 -- 172.21.15.78:0/3083109935 learned_addr learned my addr 172.21.15.78:0/3083109935 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334f7fe640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500add70 0x7f3350156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f33549b9640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33500c2f60 0x7f3350156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334ffff640 1 -- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500add70 msgr2=0x7f3350156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334ffff640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500add70 0x7f3350156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334ffff640 1 -- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33500c2f60 msgr2=0x7f3350156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334ffff640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33500c2f60 0x7f3350156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334ffff640 1 -- 172.21.15.78:0/3083109935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009400 con 0x7f33500705b0 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334ffff640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 0x7f33500a81e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f33440098c0 tx=0x7f33440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3344032840 con 0x7f33500705b0 2024-02-19T14:53:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33440329c0 con 0x7f33500705b0 2024-02-19T14:53:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33500bd790 con 0x7f33500705b0 2024-02-19T14:53:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3344032d70 con 0x7f33500705b0 2024-02-19T14:53:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.424+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33501544d0 con 0x7f33500705b0 2024-02-19T14:53:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.425+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33500676a0 con 0x7f33500705b0 2024-02-19T14:53:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.426+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f334403b3f0 con 0x7f33500705b0 2024-02-19T14:53:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.427+0000 7f334cff9640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.427+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f33440ac620 con 0x7f33500705b0 2024-02-19T14:53:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.427+0000 7f334f7fe640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.427+0000 7f334f7fe640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f33500c23c0 tx=0x7f3340007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.431+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3344076bb0 con 0x7f33500705b0 2024-02-19T14:53:52.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.605+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3350071900 con 0x7f333c06b810 2024-02-19T14:53:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.617+0000 7f334cff9640 1 -- 172.21.15.78:0/3083109935 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+522875 (secure 0 0 0) 0x7f33400071f0 con 0x7f333c06b810 2024-02-19T14:53:52.625 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:52.629 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 msgr2=0x7f333c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f33500c23c0 tx=0x7f3340007b70 comp rx=0 tx=0).stop 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 msgr2=0x7f33500a81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 0x7f33500a81e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f33440098c0 tx=0x7f33440046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 shutdown_connections 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33500c2f60 0x7f3350156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33500add70 0x7f3350156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 --2- 172.21.15.78:0/3083109935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33500705b0 0x7f33500a81e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.621+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 >> 172.21.15.78:0/3083109935 conn(0x7f33500c85e0 msgr2=0x7f335014fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:52.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.622+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 shutdown_connections 2024-02-19T14:53:52.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.622+0000 7f335641c640 1 -- 172.21.15.78:0/3083109935 wait complete. 2024-02-19T14:53:52.635 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-02-19T14:53:52.082264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222440,"kb_used_data":6272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752280,"statfs":{"total":773094113280,"available":772866334720,"internally_reserved":0,"allocated":6422528,"data_stored":3527724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.364855"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730654+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:47.730654+0000","last_peered":"2024-02-19T14:53:47.730654+0000","last_clean":"2024-02-19T14:53:47.730654+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:47.730654+0000","last_undegraded":"2024-02-19T14:53:47.730654+0000","last_fullsized":"2024-02-19T14:53:47.730654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731535+0000","last_change":"2024-02-19T14:53:47.731535+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731535+0000","last_undegraded":"2024-02-19T14:53:47.731535+0000","last_fullsized":"2024-02-19T14:53:47.731535+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730461+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:47.730461+0000","last_peered":"2024-02-19T14:53:47.730461+0000","last_clean":"2024-02-19T14:53:47.730461+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:47.730461+0000","last_undegraded":"2024-02-19T14:53:47.730461+0000","last_fullsized":"2024-02-19T14:53:47.730461+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728965+0000","last_change":"2024-02-19T14:53:47.728965+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728965+0000","last_undegraded":"2024-02-19T14:53:47.728965+0000","last_fullsized":"2024-02-19T14:53:47.728965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732216+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:47.732216+0000","last_peered":"2024-02-19T14:53:47.732216+0000","last_clean":"2024-02-19T14:53:47.732216+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:47.732216+0000","last_undegraded":"2024-02-19T14:53:47.732216+0000","last_fullsized":"2024-02-19T14:53:47.732216+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726570+0000","last_change":"2024-02-19T14:53:47.726570+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726570+0000","last_undegraded":"2024-02-19T14:53:47.726570+0000","last_fullsized":"2024-02-19T14:53:47.726570+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729626+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:47.729626+0000","last_peered":"2024-02-19T14:53:47.729626+0000","last_clean":"2024-02-19T14:53:47.729626+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:47.729626+0000","last_undegraded":"2024-02-19T14:53:47.729626+0000","last_fullsized":"2024-02-19T14:53:47.729626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730942+0000","last_change":"2024-02-19T14:53:47.730942+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730942+0000","last_undegraded":"2024-02-19T14:53:47.730942+0000","last_fullsized":"2024-02-19T14:53:47.730942+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731084+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:47.731084+0000","last_peered":"2024-02-19T14:53:47.731084+0000","last_clean":"2024-02-19T14:53:47.731084+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:47.731084+0000","last_undegraded":"2024-02-19T14:53:47.731084+0000","last_fullsized":"2024-02-19T14:53:47.731084+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729130+0000","last_change":"2024-02-19T14:53:47.729130+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729130+0000","last_undegraded":"2024-02-19T14:53:47.729130+0000","last_fullsized":"2024-02-19T14:53:47.729130+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729373+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:47.729373+0000","last_peered":"2024-02-19T14:53:47.729373+0000","last_clean":"2024-02-19T14:53:47.729373+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:47.729373+0000","last_undegraded":"2024-02-19T14:53:47.729373+0000","last_fullsized":"2024-02-19T14:53:47.729373+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731517+0000","last_change":"2024-02-19T14:53:47.731517+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731517+0000","last_undegraded":"2024-02-19T14:53:47.731517+0000","last_fullsized":"2024-02-19T14:53:47.731517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731295+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:47.731295+0000","last_peered":"2024-02-19T14:53:47.731295+0000","last_clean":"2024-02-19T14:53:47.731295+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:47.731295+0000","last_undegraded":"2024-02-19T14:53:47.731295+0000","last_fullsized":"2024-02-19T14:53:47.731295+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727983+0000","last_change":"2024-02-19T14:53:47.727983+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727983+0000","last_undegraded":"2024-02-19T14:53:47.727983+0000","last_fullsized":"2024-02-19T14:53:47.727983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723628+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:47.723628+0000","last_peered":"2024-02-19T14:53:47.723628+0000","last_clean":"2024-02-19T14:53:47.723628+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:47.723628+0000","last_undegraded":"2024-02-19T14:53:47.723628+0000","last_fullsized":"2024-02-19T14:53:47.723628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733825+0000","last_change":"2024-02-19T14:53:47.733825+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733825+0000","last_undegraded":"2024-02-19T14:53:47.733825+0000","last_fullsized":"2024-02-19T14:53:47.733825+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724256+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:47.724256+0000","last_peered":"2024-02-19T14:53:47.724256+0000","last_clean":"2024-02-19T14:53:47.724256+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:47.724256+0000","last_undegraded":"2024-02-19T14:53:47.724256+0000","last_fullsized":"2024-02-19T14:53:47.724256+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:50.707228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733022+0000","last_change":"2024-02-19T14:53:47.733022+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733022+0000","last_undegraded":"2024-02-19T14:53:47.733022+0000","last_fullsized":"2024-02-19T14:53:47.733022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730312+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:47.730312+0000","last_peered":"2024-02-19T14:53:47.730312+0000","last_clean":"2024-02-19T14:53:47.730312+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:47.730312+0000","last_undegraded":"2024-02-19T14:53:47.730312+0000","last_fullsized":"2024-02-19T14:53:47.730312+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733611+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:47.733611+0000","last_peered":"2024-02-19T14:53:47.733611+0000","last_clean":"2024-02-19T14:53:47.733611+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:47.733611+0000","last_undegraded":"2024-02-19T14:53:47.733611+0000","last_fullsized":"2024-02-19T14:53:47.733611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:37.193943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726799+0000","last_change":"2024-02-19T14:53:47.726799+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726799+0000","last_undegraded":"2024-02-19T14:53:47.726799+0000","last_fullsized":"2024-02-19T14:53:47.726799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724742+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:47.724742+0000","last_peered":"2024-02-19T14:53:47.724742+0000","last_clean":"2024-02-19T14:53:47.724742+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:47.724742+0000","last_undegraded":"2024-02-19T14:53:47.724742+0000","last_fullsized":"2024-02-19T14:53:47.724742+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:01.639745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726754+0000","last_change":"2024-02-19T14:53:47.726754+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726754+0000","last_undegraded":"2024-02-19T14:53:47.726754+0000","last_fullsized":"2024-02-19T14:53:47.726754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729060+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:47.729060+0000","last_peered":"2024-02-19T14:53:47.729060+0000","last_clean":"2024-02-19T14:53:47.729060+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:47.729060+0000","last_undegraded":"2024-02-19T14:53:47.729060+0000","last_fullsized":"2024-02-19T14:53:47.729060+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731147+0000","last_change":"2024-02-19T14:53:47.731147+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731147+0000","last_undegraded":"2024-02-19T14:53:47.731147+0000","last_fullsized":"2024-02-19T14:53:47.731147+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739485+0000","last_change":"2024-02-19T14:53:42.136272+0000","last_active":"2024-02-19T14:53:47.739485+0000","last_peered":"2024-02-19T14:53:47.739485+0000","last_clean":"2024-02-19T14:53:47.739485+0000","last_became_active":"2024-02-19T14:53:42.135998+0000","last_became_peered":"2024-02-19T14:53:42.135998+0000","last_unstale":"2024-02-19T14:53:47.739485+0000","last_undegraded":"2024-02-19T14:53:47.739485+0000","last_fullsized":"2024-02-19T14:53:47.739485+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734360+0000","last_change":"2024-02-19T14:53:47.734360+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734360+0000","last_undegraded":"2024-02-19T14:53:47.734360+0000","last_fullsized":"2024-02-19T14:53:47.734360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729068+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:47.729068+0000","last_peered":"2024-02-19T14:53:47.729068+0000","last_clean":"2024-02-19T14:53:47.729068+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:47.729068+0000","last_undegraded":"2024-02-19T14:53:47.729068+0000","last_fullsized":"2024-02-19T14:53:47.729068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726289+0000","last_change":"2024-02-19T14:53:47.726289+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726289+0000","last_undegraded":"2024-02-19T14:53:47.726289+0000","last_fullsized":"2024-02-19T14:53:47.726289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727510+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:47.727510+0000","last_peered":"2024-02-19T14:53:47.727510+0000","last_clean":"2024-02-19T14:53:47.727510+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:47.727510+0000","last_undegraded":"2024-02-19T14:53:47.727510+0000","last_fullsized":"2024-02-19T14:53:47.727510+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731901+0000","last_change":"2024-02-19T14:53:47.731901+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731901+0000","last_undegraded":"2024-02-19T14:53:47.731901+0000","last_fullsized":"2024-02-19T14:53:47.731901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734311+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:47.734311+0000","last_peered":"2024-02-19T14:53:47.734311+0000","last_clean":"2024-02-19T14:53:47.734311+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:47.734311+0000","last_undegraded":"2024-02-19T14:53:47.734311+0000","last_fullsized":"2024-02-19T14:53:47.734311+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726614+0000","last_change":"2024-02-19T14:53:47.726614+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726614+0000","last_undegraded":"2024-02-19T14:53:47.726614+0000","last_fullsized":"2024-02-19T14:53:47.726614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730365+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:47.730365+0000","last_peered":"2024-02-19T14:53:47.730365+0000","last_clean":"2024-02-19T14:53:47.730365+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:47.730365+0000","last_undegraded":"2024-02-19T14:53:47.730365+0000","last_fullsized":"2024-02-19T14:53:47.730365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728581+0000","last_change":"2024-02-19T14:53:47.728581+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728581+0000","last_undegraded":"2024-02-19T14:53:47.728581+0000","last_fullsized":"2024-02-19T14:53:47.728581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732357+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:47.732357+0000","last_peered":"2024-02-19T14:53:47.732357+0000","last_clean":"2024-02-19T14:53:47.732357+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:47.732357+0000","last_undegraded":"2024-02-19T14:53:47.732357+0000","last_fullsized":"2024-02-19T14:53:47.732357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735616+0000","last_change":"2024-02-19T14:53:47.735616+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735616+0000","last_undegraded":"2024-02-19T14:53:47.735616+0000","last_fullsized":"2024-02-19T14:53:47.735616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737614+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:47.737614+0000","last_peered":"2024-02-19T14:53:47.737614+0000","last_clean":"2024-02-19T14:53:47.737614+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:47.737614+0000","last_undegraded":"2024-02-19T14:53:47.737614+0000","last_fullsized":"2024-02-19T14:53:47.737614+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726307+0000","last_change":"2024-02-19T14:53:47.726307+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726307+0000","last_undegraded":"2024-02-19T14:53:47.726307+0000","last_fullsized":"2024-02-19T14:53:47.726307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739540+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:47.739540+0000","last_peered":"2024-02-19T14:53:47.739540+0000","last_clean":"2024-02-19T14:53:47.739540+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:47.739540+0000","last_undegraded":"2024-02-19T14:53:47.739540+0000","last_fullsized":"2024-02-19T14:53:47.739540+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725966+0000","last_change":"2024-02-19T14:53:47.725966+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725966+0000","last_undegraded":"2024-02-19T14:53:47.725966+0000","last_fullsized":"2024-02-19T14:53:47.725966+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723966+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:47.723966+0000","last_peered":"2024-02-19T14:53:47.723966+0000","last_clean":"2024-02-19T14:53:47.723966+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:47.723966+0000","last_undegraded":"2024-02-19T14:53:47.723966+0000","last_fullsized":"2024-02-19T14:53:47.723966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:59.053283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729507+0000","last_change":"2024-02-19T14:53:47.729507+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729507+0000","last_undegraded":"2024-02-19T14:53:47.729507+0000","last_fullsized":"2024-02-19T14:53:47.729507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732727+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:47.732727+0000","last_peered":"2024-02-19T14:53:47.732727+0000","last_clean":"2024-02-19T14:53:47.732727+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:47.732727+0000","last_undegraded":"2024-02-19T14:53:47.732727+0000","last_fullsized":"2024-02-19T14:53:47.732727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:38:18.934930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726592+0000","last_change":"2024-02-19T14:53:47.726592+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726592+0000","last_undegraded":"2024-02-19T14:53:47.726592+0000","last_fullsized":"2024-02-19T14:53:47.726592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730738+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:47.730738+0000","last_peered":"2024-02-19T14:53:47.730738+0000","last_clean":"2024-02-19T14:53:47.730738+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:47.730738+0000","last_undegraded":"2024-02-19T14:53:47.730738+0000","last_fullsized":"2024-02-19T14:53:47.730738+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734650+0000","last_change":"2024-02-19T14:53:47.734650+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734650+0000","last_undegraded":"2024-02-19T14:53:47.734650+0000","last_fullsized":"2024-02-19T14:53:47.734650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734454+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:47.734454+0000","last_peered":"2024-02-19T14:53:47.734454+0000","last_clean":"2024-02-19T14:53:47.734454+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:47.734454+0000","last_undegraded":"2024-02-19T14:53:47.734454+0000","last_fullsized":"2024-02-19T14:53:47.734454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729138+0000","last_change":"2024-02-19T14:53:47.729138+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729138+0000","last_undegraded":"2024-02-19T14:53:47.729138+0000","last_fullsized":"2024-02-19T14:53:47.729138+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734486+0000","last_change":"2024-02-19T14:53:47.734486+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734486+0000","last_undegraded":"2024-02-19T14:53:47.734486+0000","last_fullsized":"2024-02-19T14:53:47.734486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729945+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:47.729945+0000","last_peered":"2024-02-19T14:53:47.729945+0000","last_clean":"2024-02-19T14:53:47.729945+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:47.729945+0000","last_undegraded":"2024-02-19T14:53:47.729945+0000","last_fullsized":"2024-02-19T14:53:47.729945+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733413+0000","last_change":"2024-02-19T14:53:47.733413+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733413+0000","last_undegraded":"2024-02-19T14:53:47.733413+0000","last_fullsized":"2024-02-19T14:53:47.733413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737727+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:47.737727+0000","last_peered":"2024-02-19T14:53:47.737727+0000","last_clean":"2024-02-19T14:53:47.737727+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:47.737727+0000","last_undegraded":"2024-02-19T14:53:47.737727+0000","last_fullsized":"2024-02-19T14:53:47.737727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729994+0000","last_change":"2024-02-19T14:53:47.729994+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729994+0000","last_undegraded":"2024-02-19T14:53:47.729994+0000","last_fullsized":"2024-02-19T14:53:47.729994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732428+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:47.732428+0000","last_peered":"2024-02-19T14:53:47.732428+0000","last_clean":"2024-02-19T14:53:47.732428+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:47.732428+0000","last_undegraded":"2024-02-19T14:53:47.732428+0000","last_fullsized":"2024-02-19T14:53:47.732428+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728586+0000","last_change":"2024-02-19T14:53:47.728586+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728586+0000","last_undegraded":"2024-02-19T14:53:47.728586+0000","last_fullsized":"2024-02-19T14:53:47.728586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729908+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:47.729908+0000","last_peered":"2024-02-19T14:53:47.729908+0000","last_clean":"2024-02-19T14:53:47.729908+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:47.729908+0000","last_undegraded":"2024-02-19T14:53:47.729908+0000","last_fullsized":"2024-02-19T14:53:47.729908+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726173+0000","last_change":"2024-02-19T14:53:47.726173+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726173+0000","last_undegraded":"2024-02-19T14:53:47.726173+0000","last_fullsized":"2024-02-19T14:53:47.726173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731441+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:47.731441+0000","last_peered":"2024-02-19T14:53:47.731441+0000","last_clean":"2024-02-19T14:53:47.731441+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:47.731441+0000","last_undegraded":"2024-02-19T14:53:47.731441+0000","last_fullsized":"2024-02-19T14:53:47.731441+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727599+0000","last_change":"2024-02-19T14:53:47.727599+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727599+0000","last_undegraded":"2024-02-19T14:53:47.727599+0000","last_fullsized":"2024-02-19T14:53:47.727599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734161+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:47.734161+0000","last_peered":"2024-02-19T14:53:47.734161+0000","last_clean":"2024-02-19T14:53:47.734161+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:47.734161+0000","last_undegraded":"2024-02-19T14:53:47.734161+0000","last_fullsized":"2024-02-19T14:53:47.734161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735808+0000","last_change":"2024-02-19T14:53:47.735808+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735808+0000","last_undegraded":"2024-02-19T14:53:47.735808+0000","last_fullsized":"2024-02-19T14:53:47.735808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746858+0000","last_change":"2024-02-19T14:53:47.740946+0000","last_active":"2024-02-19T14:53:47.746858+0000","last_peered":"2024-02-19T14:53:47.746858+0000","last_clean":"2024-02-19T14:53:47.746858+0000","last_became_active":"2024-02-19T14:53:47.740550+0000","last_became_peered":"2024-02-19T14:53:47.740550+0000","last_unstale":"2024-02-19T14:53:47.746858+0000","last_undegraded":"2024-02-19T14:53:47.746858+0000","last_fullsized":"2024-02-19T14:53:47.746858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:09.101397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728386+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:47.728386+0000","last_peered":"2024-02-19T14:53:47.728386+0000","last_clean":"2024-02-19T14:53:47.728386+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:47.728386+0000","last_undegraded":"2024-02-19T14:53:47.728386+0000","last_fullsized":"2024-02-19T14:53:47.728386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":26,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739615+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:47.739615+0000","last_peered":"2024-02-19T14:53:47.739615+0000","last_clean":"2024-02-19T14:53:47.739615+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:47.739615+0000","last_undegraded":"2024-02-19T14:53:47.739615+0000","last_fullsized":"2024-02-19T14:53:47.739615+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727391+0000","last_change":"2024-02-19T14:53:47.727391+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727391+0000","last_undegraded":"2024-02-19T14:53:47.727391+0000","last_fullsized":"2024-02-19T14:53:47.727391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746433+0000","last_change":"2024-02-19T14:53:47.742941+0000","last_active":"2024-02-19T14:53:47.746433+0000","last_peered":"2024-02-19T14:53:47.746433+0000","last_clean":"2024-02-19T14:53:47.746433+0000","last_became_active":"2024-02-19T14:53:47.742590+0000","last_became_peered":"2024-02-19T14:53:47.742590+0000","last_unstale":"2024-02-19T14:53:47.746433+0000","last_undegraded":"2024-02-19T14:53:47.746433+0000","last_fullsized":"2024-02-19T14:53:47.746433+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:35.779780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729200+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:47.729200+0000","last_peered":"2024-02-19T14:53:47.729200+0000","last_clean":"2024-02-19T14:53:47.729200+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:47.729200+0000","last_undegraded":"2024-02-19T14:53:47.729200+0000","last_fullsized":"2024-02-19T14:53:47.729200+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730410+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:47.730410+0000","last_peered":"2024-02-19T14:53:47.730410+0000","last_clean":"2024-02-19T14:53:47.730410+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:47.730410+0000","last_undegraded":"2024-02-19T14:53:47.730410+0000","last_fullsized":"2024-02-19T14:53:47.730410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729346+0000","last_change":"2024-02-19T14:53:47.729346+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729346+0000","last_undegraded":"2024-02-19T14:53:47.729346+0000","last_fullsized":"2024-02-19T14:53:47.729346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747720+0000","last_change":"2024-02-19T14:53:47.746326+0000","last_active":"2024-02-19T14:53:47.747720+0000","last_peered":"2024-02-19T14:53:47.747720+0000","last_clean":"2024-02-19T14:53:47.747720+0000","last_became_active":"2024-02-19T14:53:47.745753+0000","last_became_peered":"2024-02-19T14:53:47.745753+0000","last_unstale":"2024-02-19T14:53:47.747720+0000","last_undegraded":"2024-02-19T14:53:47.747720+0000","last_fullsized":"2024-02-19T14:53:47.747720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739624+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:47.739624+0000","last_peered":"2024-02-19T14:53:47.739624+0000","last_clean":"2024-02-19T14:53:47.739624+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:47.739624+0000","last_undegraded":"2024-02-19T14:53:47.739624+0000","last_fullsized":"2024-02-19T14:53:47.739624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730034+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:47.730034+0000","last_peered":"2024-02-19T14:53:47.730034+0000","last_clean":"2024-02-19T14:53:47.730034+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:47.730034+0000","last_undegraded":"2024-02-19T14:53:47.730034+0000","last_fullsized":"2024-02-19T14:53:47.730034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726858+0000","last_change":"2024-02-19T14:53:47.726858+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726858+0000","last_undegraded":"2024-02-19T14:53:47.726858+0000","last_fullsized":"2024-02-19T14:53:47.726858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747935+0000","last_change":"2024-02-19T14:53:47.740910+0000","last_active":"2024-02-19T14:53:47.747935+0000","last_peered":"2024-02-19T14:53:47.747935+0000","last_clean":"2024-02-19T14:53:47.747935+0000","last_became_active":"2024-02-19T14:53:47.740626+0000","last_became_peered":"2024-02-19T14:53:47.740626+0000","last_unstale":"2024-02-19T14:53:47.747935+0000","last_undegraded":"2024-02-19T14:53:47.747935+0000","last_fullsized":"2024-02-19T14:53:47.747935+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723809+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:47.723809+0000","last_peered":"2024-02-19T14:53:47.723809+0000","last_clean":"2024-02-19T14:53:47.723809+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:47.723809+0000","last_undegraded":"2024-02-19T14:53:47.723809+0000","last_fullsized":"2024-02-19T14:53:47.723809+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:36.026355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733496+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:47.733496+0000","last_peered":"2024-02-19T14:53:47.733496+0000","last_clean":"2024-02-19T14:53:47.733496+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:47.733496+0000","last_undegraded":"2024-02-19T14:53:47.733496+0000","last_fullsized":"2024-02-19T14:53:47.733496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732000+0000","last_change":"2024-02-19T14:53:47.732000+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732000+0000","last_undegraded":"2024-02-19T14:53:47.732000+0000","last_fullsized":"2024-02-19T14:53:47.732000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747004+0000","last_change":"2024-02-19T14:53:47.745151+0000","last_active":"2024-02-19T14:53:47.747004+0000","last_peered":"2024-02-19T14:53:47.747004+0000","last_clean":"2024-02-19T14:53:47.747004+0000","last_became_active":"2024-02-19T14:53:47.744887+0000","last_became_peered":"2024-02-19T14:53:47.744887+0000","last_unstale":"2024-02-19T14:53:47.747004+0000","last_undegraded":"2024-02-19T14:53:47.747004+0000","last_fullsized":"2024-02-19T14:53:47.747004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732432+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:47.732432+0000","last_peered":"2024-02-19T14:53:47.732432+0000","last_clean":"2024-02-19T14:53:47.732432+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:47.732432+0000","last_undegraded":"2024-02-19T14:53:47.732432+0000","last_fullsized":"2024-02-19T14:53:47.732432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732599+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:47.732599+0000","last_peered":"2024-02-19T14:53:47.732599+0000","last_clean":"2024-02-19T14:53:47.732599+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:47.732599+0000","last_undegraded":"2024-02-19T14:53:47.732599+0000","last_fullsized":"2024-02-19T14:53:47.732599+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.736522+0000","last_change":"2024-02-19T14:53:47.736522+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.736522+0000","last_undegraded":"2024-02-19T14:53:47.736522+0000","last_fullsized":"2024-02-19T14:53:47.736522+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746094+0000","last_change":"2024-02-19T14:53:47.743381+0000","last_active":"2024-02-19T14:53:47.746094+0000","last_peered":"2024-02-19T14:53:47.746094+0000","last_clean":"2024-02-19T14:53:47.746094+0000","last_became_active":"2024-02-19T14:53:47.743135+0000","last_became_peered":"2024-02-19T14:53:47.743135+0000","last_unstale":"2024-02-19T14:53:47.746094+0000","last_undegraded":"2024-02-19T14:53:47.746094+0000","last_fullsized":"2024-02-19T14:53:47.746094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:30.540219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730104+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:47.730104+0000","last_peered":"2024-02-19T14:53:47.730104+0000","last_clean":"2024-02-19T14:53:47.730104+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:47.730104+0000","last_undegraded":"2024-02-19T14:53:47.730104+0000","last_fullsized":"2024-02-19T14:53:47.730104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728795+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:47.728795+0000","last_peered":"2024-02-19T14:53:47.728795+0000","last_clean":"2024-02-19T14:53:47.728795+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:47.728795+0000","last_undegraded":"2024-02-19T14:53:47.728795+0000","last_fullsized":"2024-02-19T14:53:47.728795+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728755+0000","last_change":"2024-02-19T14:53:47.728755+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728755+0000","last_undegraded":"2024-02-19T14:53:47.728755+0000","last_fullsized":"2024-02-19T14:53:47.728755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746985+0000","last_change":"2024-02-19T14:53:47.744368+0000","last_active":"2024-02-19T14:53:47.746985+0000","last_peered":"2024-02-19T14:53:47.746985+0000","last_clean":"2024-02-19T14:53:47.746985+0000","last_became_active":"2024-02-19T14:53:47.743379+0000","last_became_peered":"2024-02-19T14:53:47.743379+0000","last_unstale":"2024-02-19T14:53:47.746985+0000","last_undegraded":"2024-02-19T14:53:47.746985+0000","last_fullsized":"2024-02-19T14:53:47.746985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724152+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:47.724152+0000","last_peered":"2024-02-19T14:53:47.724152+0000","last_clean":"2024-02-19T14:53:47.724152+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:47.724152+0000","last_undegraded":"2024-02-19T14:53:47.724152+0000","last_fullsized":"2024-02-19T14:53:47.724152+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:41.307928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737668+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:47.737668+0000","last_peered":"2024-02-19T14:53:47.737668+0000","last_clean":"2024-02-19T14:53:47.737668+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:47.737668+0000","last_undegraded":"2024-02-19T14:53:47.737668+0000","last_fullsized":"2024-02-19T14:53:47.737668+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726342+0000","last_change":"2024-02-19T14:53:47.726342+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726342+0000","last_undegraded":"2024-02-19T14:53:47.726342+0000","last_fullsized":"2024-02-19T14:53:47.726342+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747509+0000","last_change":"2024-02-19T14:53:47.744422+0000","last_active":"2024-02-19T14:53:47.747509+0000","last_peered":"2024-02-19T14:53:47.747509+0000","last_clean":"2024-02-19T14:53:47.747509+0000","last_became_active":"2024-02-19T14:53:47.744122+0000","last_became_peered":"2024-02-19T14:53:47.744122+0000","last_unstale":"2024-02-19T14:53:47.747509+0000","last_undegraded":"2024-02-19T14:53:47.747509+0000","last_fullsized":"2024-02-19T14:53:47.747509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:50.594221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730639+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:47.730639+0000","last_peered":"2024-02-19T14:53:47.730639+0000","last_clean":"2024-02-19T14:53:47.730639+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:47.730639+0000","last_undegraded":"2024-02-19T14:53:47.730639+0000","last_fullsized":"2024-02-19T14:53:47.730639+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730829+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:47.730829+0000","last_peered":"2024-02-19T14:53:47.730829+0000","last_clean":"2024-02-19T14:53:47.730829+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:47.730829+0000","last_undegraded":"2024-02-19T14:53:47.730829+0000","last_fullsized":"2024-02-19T14:53:47.730829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728855+0000","last_change":"2024-02-19T14:53:47.728855+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728855+0000","last_undegraded":"2024-02-19T14:53:47.728855+0000","last_fullsized":"2024-02-19T14:53:47.728855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746360+0000","last_change":"2024-02-19T14:53:47.745353+0000","last_active":"2024-02-19T14:53:47.746360+0000","last_peered":"2024-02-19T14:53:47.746360+0000","last_clean":"2024-02-19T14:53:47.746360+0000","last_became_active":"2024-02-19T14:53:47.745008+0000","last_became_peered":"2024-02-19T14:53:47.745008+0000","last_unstale":"2024-02-19T14:53:47.746360+0000","last_undegraded":"2024-02-19T14:53:47.746360+0000","last_fullsized":"2024-02-19T14:53:47.746360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:29.660730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729669+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:47.729669+0000","last_peered":"2024-02-19T14:53:47.729669+0000","last_clean":"2024-02-19T14:53:47.729669+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:47.729669+0000","last_undegraded":"2024-02-19T14:53:47.729669+0000","last_fullsized":"2024-02-19T14:53:47.729669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730746+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:47.730746+0000","last_peered":"2024-02-19T14:53:47.730746+0000","last_clean":"2024-02-19T14:53:47.730746+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:47.730746+0000","last_undegraded":"2024-02-19T14:53:47.730746+0000","last_fullsized":"2024-02-19T14:53:47.730746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726359+0000","last_change":"2024-02-19T14:53:47.726359+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726359+0000","last_undegraded":"2024-02-19T14:53:47.726359+0000","last_fullsized":"2024-02-19T14:53:47.726359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746332+0000","last_change":"2024-02-19T14:53:47.742114+0000","last_active":"2024-02-19T14:53:47.746332+0000","last_peered":"2024-02-19T14:53:47.746332+0000","last_clean":"2024-02-19T14:53:47.746332+0000","last_became_active":"2024-02-19T14:53:47.741840+0000","last_became_peered":"2024-02-19T14:53:47.741840+0000","last_unstale":"2024-02-19T14:53:47.746332+0000","last_undegraded":"2024-02-19T14:53:47.746332+0000","last_fullsized":"2024-02-19T14:53:47.746332+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:20.795346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727709+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:47.727709+0000","last_peered":"2024-02-19T14:53:47.727709+0000","last_clean":"2024-02-19T14:53:47.727709+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:47.727709+0000","last_undegraded":"2024-02-19T14:53:47.727709+0000","last_fullsized":"2024-02-19T14:53:47.727709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.736179+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:47.736179+0000","last_peered":"2024-02-19T14:53:47.736179+0000","last_clean":"2024-02-19T14:53:47.736179+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:47.736179+0000","last_undegraded":"2024-02-19T14:53:47.736179+0000","last_fullsized":"2024-02-19T14:53:47.736179+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731814+0000","last_change":"2024-02-19T14:53:47.731814+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731814+0000","last_undegraded":"2024-02-19T14:53:47.731814+0000","last_fullsized":"2024-02-19T14:53:47.731814+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737453+0000","last_change":"2024-02-19T14:53:47.734753+0000","last_active":"2024-02-19T14:53:47.737453+0000","last_peered":"2024-02-19T14:53:47.737453+0000","last_clean":"2024-02-19T14:53:47.737453+0000","last_became_active":"2024-02-19T14:53:47.734515+0000","last_became_peered":"2024-02-19T14:53:47.734515+0000","last_unstale":"2024-02-19T14:53:47.737453+0000","last_undegraded":"2024-02-19T14:53:47.737453+0000","last_fullsized":"2024-02-19T14:53:47.737453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:02:56.724991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730223+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:47.730223+0000","last_peered":"2024-02-19T14:53:47.730223+0000","last_clean":"2024-02-19T14:53:47.730223+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:47.730223+0000","last_undegraded":"2024-02-19T14:53:47.730223+0000","last_fullsized":"2024-02-19T14:53:47.730223+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":25,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731281+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:47.731281+0000","last_peered":"2024-02-19T14:53:47.731281+0000","last_clean":"2024-02-19T14:53:47.731281+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:47.731281+0000","last_undegraded":"2024-02-19T14:53:47.731281+0000","last_fullsized":"2024-02-19T14:53:47.731281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727760+0000","last_change":"2024-02-19T14:53:47.727760+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727760+0000","last_undegraded":"2024-02-19T14:53:47.727760+0000","last_fullsized":"2024-02-19T14:53:47.727760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737793+0000","last_change":"2024-02-19T14:53:47.735759+0000","last_active":"2024-02-19T14:53:47.737793+0000","last_peered":"2024-02-19T14:53:47.737793+0000","last_clean":"2024-02-19T14:53:47.737793+0000","last_became_active":"2024-02-19T14:53:47.735503+0000","last_became_peered":"2024-02-19T14:53:47.735503+0000","last_unstale":"2024-02-19T14:53:47.737793+0000","last_undegraded":"2024-02-19T14:53:47.737793+0000","last_fullsized":"2024-02-19T14:53:47.737793+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:35.189687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731195+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:47.731195+0000","last_peered":"2024-02-19T14:53:47.731195+0000","last_clean":"2024-02-19T14:53:47.731195+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:47.731195+0000","last_undegraded":"2024-02-19T14:53:47.731195+0000","last_fullsized":"2024-02-19T14:53:47.731195+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":28,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724746+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:47.724746+0000","last_peered":"2024-02-19T14:53:47.724746+0000","last_clean":"2024-02-19T14:53:47.724746+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:47.724746+0000","last_undegraded":"2024-02-19T14:53:47.724746+0000","last_fullsized":"2024-02-19T14:53:47.724746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731071+0000","last_change":"2024-02-19T14:53:47.731071+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731071+0000","last_undegraded":"2024-02-19T14:53:47.731071+0000","last_fullsized":"2024-02-19T14:53:47.731071+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746268+0000","last_change":"2024-02-19T14:53:47.742801+0000","last_active":"2024-02-19T14:53:47.746268+0000","last_peered":"2024-02-19T14:53:47.746268+0000","last_clean":"2024-02-19T14:53:47.746268+0000","last_became_active":"2024-02-19T14:53:47.742534+0000","last_became_peered":"2024-02-19T14:53:47.742534+0000","last_unstale":"2024-02-19T14:53:47.746268+0000","last_undegraded":"2024-02-19T14:53:47.746268+0000","last_fullsized":"2024-02-19T14:53:47.746268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:59.453875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733273+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:47.733273+0000","last_peered":"2024-02-19T14:53:47.733273+0000","last_clean":"2024-02-19T14:53:47.733273+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:47.733273+0000","last_undegraded":"2024-02-19T14:53:47.733273+0000","last_fullsized":"2024-02-19T14:53:47.733273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732699+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:47.732699+0000","last_peered":"2024-02-19T14:53:47.732699+0000","last_clean":"2024-02-19T14:53:47.732699+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:47.732699+0000","last_undegraded":"2024-02-19T14:53:47.732699+0000","last_fullsized":"2024-02-19T14:53:47.732699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726717+0000","last_change":"2024-02-19T14:53:47.726717+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726717+0000","last_undegraded":"2024-02-19T14:53:47.726717+0000","last_fullsized":"2024-02-19T14:53:47.726717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748938+0000","last_change":"2024-02-19T14:53:47.744212+0000","last_active":"2024-02-19T14:53:47.748938+0000","last_peered":"2024-02-19T14:53:47.748938+0000","last_clean":"2024-02-19T14:53:47.748938+0000","last_became_active":"2024-02-19T14:53:47.743916+0000","last_became_peered":"2024-02-19T14:53:47.743916+0000","last_unstale":"2024-02-19T14:53:47.748938+0000","last_undegraded":"2024-02-19T14:53:47.748938+0000","last_fullsized":"2024-02-19T14:53:47.748938+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732440+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:47.732440+0000","last_peered":"2024-02-19T14:53:47.732440+0000","last_clean":"2024-02-19T14:53:47.732440+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:47.732440+0000","last_undegraded":"2024-02-19T14:53:47.732440+0000","last_fullsized":"2024-02-19T14:53:47.732440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739733+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:47.739733+0000","last_peered":"2024-02-19T14:53:47.739733+0000","last_clean":"2024-02-19T14:53:47.739733+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:47.739733+0000","last_undegraded":"2024-02-19T14:53:47.739733+0000","last_fullsized":"2024-02-19T14:53:47.739733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731538+0000","last_change":"2024-02-19T14:53:47.731538+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731538+0000","last_undegraded":"2024-02-19T14:53:47.731538+0000","last_fullsized":"2024-02-19T14:53:47.731538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.744120+0000","last_change":"2024-02-19T14:53:47.741970+0000","last_active":"2024-02-19T14:53:47.744120+0000","last_peered":"2024-02-19T14:53:47.744120+0000","last_clean":"2024-02-19T14:53:47.744120+0000","last_became_active":"2024-02-19T14:53:47.741707+0000","last_became_peered":"2024-02-19T14:53:47.741707+0000","last_unstale":"2024-02-19T14:53:47.744120+0000","last_undegraded":"2024-02-19T14:53:47.744120+0000","last_fullsized":"2024-02-19T14:53:47.744120+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:26.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724087+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:47.724087+0000","last_peered":"2024-02-19T14:53:47.724087+0000","last_clean":"2024-02-19T14:53:47.724087+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:47.724087+0000","last_undegraded":"2024-02-19T14:53:47.724087+0000","last_fullsized":"2024-02-19T14:53:47.724087+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729186+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:47.729186+0000","last_peered":"2024-02-19T14:53:47.729186+0000","last_clean":"2024-02-19T14:53:47.729186+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:47.729186+0000","last_undegraded":"2024-02-19T14:53:47.729186+0000","last_fullsized":"2024-02-19T14:53:47.729186+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734244+0000","last_change":"2024-02-19T14:53:47.734244+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734244+0000","last_undegraded":"2024-02-19T14:53:47.734244+0000","last_fullsized":"2024-02-19T14:53:47.734244+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743671+0000","last_change":"2024-02-19T14:53:47.742140+0000","last_active":"2024-02-19T14:53:47.743671+0000","last_peered":"2024-02-19T14:53:47.743671+0000","last_clean":"2024-02-19T14:53:47.743671+0000","last_became_active":"2024-02-19T14:53:47.741838+0000","last_became_peered":"2024-02-19T14:53:47.741838+0000","last_unstale":"2024-02-19T14:53:47.743671+0000","last_undegraded":"2024-02-19T14:53:47.743671+0000","last_fullsized":"2024-02-19T14:53:47.743671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:55.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737818+0000","last_change":"2024-02-19T14:53:41.706063+0000","last_active":"2024-02-19T14:53:47.737818+0000","last_peered":"2024-02-19T14:53:47.737818+0000","last_clean":"2024-02-19T14:53:47.737818+0000","last_became_active":"2024-02-19T14:53:41.705766+0000","last_became_peered":"2024-02-19T14:53:41.705766+0000","last_unstale":"2024-02-19T14:53:47.737818+0000","last_undegraded":"2024-02-19T14:53:47.737818+0000","last_fullsized":"2024-02-19T14:53:47.737818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":30,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723730+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:47.723730+0000","last_peered":"2024-02-19T14:53:47.723730+0000","last_clean":"2024-02-19T14:53:47.723730+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:47.723730+0000","last_undegraded":"2024-02-19T14:53:47.723730+0000","last_fullsized":"2024-02-19T14:53:47.723730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727027+0000","last_change":"2024-02-19T14:53:47.727027+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727027+0000","last_undegraded":"2024-02-19T14:53:47.727027+0000","last_fullsized":"2024-02-19T14:53:47.727027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743702+0000","last_change":"2024-02-19T14:53:47.742196+0000","last_active":"2024-02-19T14:53:47.743702+0000","last_peered":"2024-02-19T14:53:47.743702+0000","last_clean":"2024-02-19T14:53:47.743702+0000","last_became_active":"2024-02-19T14:53:47.741852+0000","last_became_peered":"2024-02-19T14:53:47.741852+0000","last_unstale":"2024-02-19T14:53:47.743702+0000","last_undegraded":"2024-02-19T14:53:47.743702+0000","last_fullsized":"2024-02-19T14:53:47.743702+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:31.942230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724625+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:47.724625+0000","last_peered":"2024-02-19T14:53:47.724625+0000","last_clean":"2024-02-19T14:53:47.724625+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:47.724625+0000","last_undegraded":"2024-02-19T14:53:47.724625+0000","last_fullsized":"2024-02-19T14:53:47.724625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734529+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:47.734529+0000","last_peered":"2024-02-19T14:53:47.734529+0000","last_clean":"2024-02-19T14:53:47.734529+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:47.734529+0000","last_undegraded":"2024-02-19T14:53:47.734529+0000","last_fullsized":"2024-02-19T14:53:47.734529+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730482+0000","last_change":"2024-02-19T14:53:47.730482+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730482+0000","last_undegraded":"2024-02-19T14:53:47.730482+0000","last_fullsized":"2024-02-19T14:53:47.730482+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.749226+0000","last_change":"2024-02-19T14:53:47.745519+0000","last_active":"2024-02-19T14:53:47.749226+0000","last_peered":"2024-02-19T14:53:47.749226+0000","last_clean":"2024-02-19T14:53:47.749226+0000","last_became_active":"2024-02-19T14:53:47.745263+0000","last_became_peered":"2024-02-19T14:53:47.745263+0000","last_unstale":"2024-02-19T14:53:47.749226+0000","last_undegraded":"2024-02-19T14:53:47.749226+0000","last_fullsized":"2024-02-19T14:53:47.749226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:51.861192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728027+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:47.728027+0000","last_peered":"2024-02-19T14:53:47.728027+0000","last_clean":"2024-02-19T14:53:47.728027+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:47.728027+0000","last_undegraded":"2024-02-19T14:53:47.728027+0000","last_fullsized":"2024-02-19T14:53:47.728027+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733299+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:47.733299+0000","last_peered":"2024-02-19T14:53:47.733299+0000","last_clean":"2024-02-19T14:53:47.733299+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:47.733299+0000","last_undegraded":"2024-02-19T14:53:47.733299+0000","last_fullsized":"2024-02-19T14:53:47.733299+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728440+0000","last_change":"2024-02-19T14:53:47.728440+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728440+0000","last_undegraded":"2024-02-19T14:53:47.728440+0000","last_fullsized":"2024-02-19T14:53:47.728440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.741263+0000","last_change":"2024-02-19T14:53:47.736640+0000","last_active":"2024-02-19T14:53:47.741263+0000","last_peered":"2024-02-19T14:53:47.741263+0000","last_clean":"2024-02-19T14:53:47.741263+0000","last_became_active":"2024-02-19T14:53:47.735150+0000","last_became_peered":"2024-02-19T14:53:47.735150+0000","last_unstale":"2024-02-19T14:53:47.741263+0000","last_undegraded":"2024-02-19T14:53:47.741263+0000","last_fullsized":"2024-02-19T14:53:47.741263+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:02.655336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728667+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:47.728667+0000","last_peered":"2024-02-19T14:53:47.728667+0000","last_clean":"2024-02-19T14:53:47.728667+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:47.728667+0000","last_undegraded":"2024-02-19T14:53:47.728667+0000","last_fullsized":"2024-02-19T14:53:47.728667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737687+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:47.737687+0000","last_peered":"2024-02-19T14:53:47.737687+0000","last_clean":"2024-02-19T14:53:47.737687+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:47.737687+0000","last_undegraded":"2024-02-19T14:53:47.737687+0000","last_fullsized":"2024-02-19T14:53:47.737687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730447+0000","last_change":"2024-02-19T14:53:47.730447+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730447+0000","last_undegraded":"2024-02-19T14:53:47.730447+0000","last_fullsized":"2024-02-19T14:53:47.730447+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734187+0000","last_change":"2024-02-19T14:53:47.730313+0000","last_active":"2024-02-19T14:53:47.734187+0000","last_peered":"2024-02-19T14:53:47.734187+0000","last_clean":"2024-02-19T14:53:47.734187+0000","last_became_active":"2024-02-19T14:53:47.730053+0000","last_became_peered":"2024-02-19T14:53:47.730053+0000","last_unstale":"2024-02-19T14:53:47.734187+0000","last_undegraded":"2024-02-19T14:53:47.734187+0000","last_fullsized":"2024-02-19T14:53:47.734187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729153+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:47.729153+0000","last_peered":"2024-02-19T14:53:47.729153+0000","last_clean":"2024-02-19T14:53:47.729153+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:47.729153+0000","last_undegraded":"2024-02-19T14:53:47.729153+0000","last_fullsized":"2024-02-19T14:53:47.729153+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737838+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:47.737838+0000","last_peered":"2024-02-19T14:53:47.737838+0000","last_clean":"2024-02-19T14:53:47.737838+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:47.737838+0000","last_undegraded":"2024-02-19T14:53:47.737838+0000","last_fullsized":"2024-02-19T14:53:47.737838+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728393+0000","last_change":"2024-02-19T14:53:47.728393+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728393+0000","last_undegraded":"2024-02-19T14:53:47.728393+0000","last_fullsized":"2024-02-19T14:53:47.728393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746290+0000","last_change":"2024-02-19T14:53:47.742841+0000","last_active":"2024-02-19T14:53:47.746290+0000","last_peered":"2024-02-19T14:53:47.746290+0000","last_clean":"2024-02-19T14:53:47.746290+0000","last_became_active":"2024-02-19T14:53:47.742581+0000","last_became_peered":"2024-02-19T14:53:47.742581+0000","last_unstale":"2024-02-19T14:53:47.746290+0000","last_undegraded":"2024-02-19T14:53:47.746290+0000","last_fullsized":"2024-02-19T14:53:47.746290+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:57.701790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733388+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:47.733388+0000","last_peered":"2024-02-19T14:53:47.733388+0000","last_clean":"2024-02-19T14:53:47.733388+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:47.733388+0000","last_undegraded":"2024-02-19T14:53:47.733388+0000","last_fullsized":"2024-02-19T14:53:47.733388+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729246+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:47.729246+0000","last_peered":"2024-02-19T14:53:47.729246+0000","last_clean":"2024-02-19T14:53:47.729246+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:47.729246+0000","last_undegraded":"2024-02-19T14:53:47.729246+0000","last_fullsized":"2024-02-19T14:53:47.729246+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731908+0000","last_change":"2024-02-19T14:53:47.731908+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731908+0000","last_undegraded":"2024-02-19T14:53:47.731908+0000","last_fullsized":"2024-02-19T14:53:47.731908+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746114+0000","last_change":"2024-02-19T14:53:47.743491+0000","last_active":"2024-02-19T14:53:47.746114+0000","last_peered":"2024-02-19T14:53:47.746114+0000","last_clean":"2024-02-19T14:53:47.746114+0000","last_became_active":"2024-02-19T14:53:47.743209+0000","last_became_peered":"2024-02-19T14:53:47.743209+0000","last_unstale":"2024-02-19T14:53:47.746114+0000","last_undegraded":"2024-02-19T14:53:47.746114+0000","last_fullsized":"2024-02-19T14:53:47.746114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:51.151311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737504+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:47.737504+0000","last_peered":"2024-02-19T14:53:47.737504+0000","last_clean":"2024-02-19T14:53:47.737504+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:47.737504+0000","last_undegraded":"2024-02-19T14:53:47.737504+0000","last_fullsized":"2024-02-19T14:53:47.737504+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724055+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:47.724055+0000","last_peered":"2024-02-19T14:53:47.724055+0000","last_clean":"2024-02-19T14:53:47.724055+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:47.724055+0000","last_undegraded":"2024-02-19T14:53:47.724055+0000","last_fullsized":"2024-02-19T14:53:47.724055+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728116+0000","last_change":"2024-02-19T14:53:47.728116+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728116+0000","last_undegraded":"2024-02-19T14:53:47.728116+0000","last_fullsized":"2024-02-19T14:53:47.728116+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747934+0000","last_change":"2024-02-19T14:53:47.740831+0000","last_active":"2024-02-19T14:53:47.747934+0000","last_peered":"2024-02-19T14:53:47.747934+0000","last_clean":"2024-02-19T14:53:47.747934+0000","last_became_active":"2024-02-19T14:53:47.740540+0000","last_became_peered":"2024-02-19T14:53:47.740540+0000","last_unstale":"2024-02-19T14:53:47.747934+0000","last_undegraded":"2024-02-19T14:53:47.747934+0000","last_fullsized":"2024-02-19T14:53:47.747934+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733979+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:47.733979+0000","last_peered":"2024-02-19T14:53:47.733979+0000","last_clean":"2024-02-19T14:53:47.733979+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:47.733979+0000","last_undegraded":"2024-02-19T14:53:47.733979+0000","last_fullsized":"2024-02-19T14:53:47.733979+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731528+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:47.731528+0000","last_peered":"2024-02-19T14:53:47.731528+0000","last_clean":"2024-02-19T14:53:47.731528+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:47.731528+0000","last_undegraded":"2024-02-19T14:53:47.731528+0000","last_fullsized":"2024-02-19T14:53:47.731528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731324+0000","last_change":"2024-02-19T14:53:47.731324+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731324+0000","last_undegraded":"2024-02-19T14:53:47.731324+0000","last_fullsized":"2024-02-19T14:53:47.731324+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.740896+0000","last_change":"2024-02-19T14:53:47.736026+0000","last_active":"2024-02-19T14:53:47.740896+0000","last_peered":"2024-02-19T14:53:47.740896+0000","last_clean":"2024-02-19T14:53:47.740896+0000","last_became_active":"2024-02-19T14:53:47.735680+0000","last_became_peered":"2024-02-19T14:53:47.735680+0000","last_unstale":"2024-02-19T14:53:47.740896+0000","last_undegraded":"2024-02-19T14:53:47.740896+0000","last_fullsized":"2024-02-19T14:53:47.740896+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730749+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:47.730749+0000","last_peered":"2024-02-19T14:53:47.730749+0000","last_clean":"2024-02-19T14:53:47.730749+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:47.730749+0000","last_undegraded":"2024-02-19T14:53:47.730749+0000","last_fullsized":"2024-02-19T14:53:47.730749+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732455+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:47.732455+0000","last_peered":"2024-02-19T14:53:47.732455+0000","last_clean":"2024-02-19T14:53:47.732455+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:47.732455+0000","last_undegraded":"2024-02-19T14:53:47.732455+0000","last_fullsized":"2024-02-19T14:53:47.732455+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731805+0000","last_change":"2024-02-19T14:53:47.731805+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731805+0000","last_undegraded":"2024-02-19T14:53:47.731805+0000","last_fullsized":"2024-02-19T14:53:47.731805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743744+0000","last_change":"2024-02-19T14:53:47.737977+0000","last_active":"2024-02-19T14:53:47.743744+0000","last_peered":"2024-02-19T14:53:47.743744+0000","last_clean":"2024-02-19T14:53:47.743744+0000","last_became_active":"2024-02-19T14:53:47.737709+0000","last_became_peered":"2024-02-19T14:53:47.737709+0000","last_unstale":"2024-02-19T14:53:47.743744+0000","last_undegraded":"2024-02-19T14:53:47.743744+0000","last_fullsized":"2024-02-19T14:53:47.743744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:00.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734277+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:47.734277+0000","last_peered":"2024-02-19T14:53:47.734277+0000","last_clean":"2024-02-19T14:53:47.734277+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:47.734277+0000","last_undegraded":"2024-02-19T14:53:47.734277+0000","last_fullsized":"2024-02-19T14:53:47.734277+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739764+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:47.739764+0000","last_peered":"2024-02-19T14:53:47.739764+0000","last_clean":"2024-02-19T14:53:47.739764+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:47.739764+0000","last_undegraded":"2024-02-19T14:53:47.739764+0000","last_fullsized":"2024-02-19T14:53:47.739764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728207+0000","last_change":"2024-02-19T14:53:47.728207+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728207+0000","last_undegraded":"2024-02-19T14:53:47.728207+0000","last_fullsized":"2024-02-19T14:53:47.728207+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748964+0000","last_change":"2024-02-19T14:53:47.745205+0000","last_active":"2024-02-19T14:53:47.748964+0000","last_peered":"2024-02-19T14:53:47.748964+0000","last_clean":"2024-02-19T14:53:47.748964+0000","last_became_active":"2024-02-19T14:53:47.744958+0000","last_became_peered":"2024-02-19T14:53:47.744958+0000","last_unstale":"2024-02-19T14:53:47.748964+0000","last_undegraded":"2024-02-19T14:53:47.748964+0000","last_fullsized":"2024-02-19T14:53:47.748964+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729468+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:47.729468+0000","last_peered":"2024-02-19T14:53:47.729468+0000","last_clean":"2024-02-19T14:53:47.729468+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:47.729468+0000","last_undegraded":"2024-02-19T14:53:47.729468+0000","last_fullsized":"2024-02-19T14:53:47.729468+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731445+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:47.731445+0000","last_peered":"2024-02-19T14:53:47.731445+0000","last_clean":"2024-02-19T14:53:47.731445+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:47.731445+0000","last_undegraded":"2024-02-19T14:53:47.731445+0000","last_fullsized":"2024-02-19T14:53:47.731445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727406+0000","last_change":"2024-02-19T14:53:47.727406+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727406+0000","last_undegraded":"2024-02-19T14:53:47.727406+0000","last_fullsized":"2024-02-19T14:53:47.727406+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747521+0000","last_change":"2024-02-19T14:53:47.740210+0000","last_active":"2024-02-19T14:53:47.747521+0000","last_peered":"2024-02-19T14:53:47.747521+0000","last_clean":"2024-02-19T14:53:47.747521+0000","last_became_active":"2024-02-19T14:53:47.739913+0000","last_became_peered":"2024-02-19T14:53:47.739913+0000","last_unstale":"2024-02-19T14:53:47.747521+0000","last_undegraded":"2024-02-19T14:53:47.747521+0000","last_fullsized":"2024-02-19T14:53:47.747521+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723813+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:47.723813+0000","last_peered":"2024-02-19T14:53:47.723813+0000","last_clean":"2024-02-19T14:53:47.723813+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:47.723813+0000","last_undegraded":"2024-02-19T14:53:47.723813+0000","last_fullsized":"2024-02-19T14:53:47.723813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:10.864244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732089+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:47.732089+0000","last_peered":"2024-02-19T14:53:47.732089+0000","last_clean":"2024-02-19T14:53:47.732089+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:47.732089+0000","last_undegraded":"2024-02-19T14:53:47.732089+0000","last_fullsized":"2024-02-19T14:53:47.732089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735192+0000","last_change":"2024-02-19T14:53:47.735192+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735192+0000","last_undegraded":"2024-02-19T14:53:47.735192+0000","last_fullsized":"2024-02-19T14:53:47.735192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.745944+0000","last_change":"2024-02-19T14:53:47.738871+0000","last_active":"2024-02-19T14:53:47.745944+0000","last_peered":"2024-02-19T14:53:47.745944+0000","last_clean":"2024-02-19T14:53:47.745944+0000","last_became_active":"2024-02-19T14:53:47.738585+0000","last_became_peered":"2024-02-19T14:53:47.738585+0000","last_unstale":"2024-02-19T14:53:47.745944+0000","last_undegraded":"2024-02-19T14:53:47.745944+0000","last_fullsized":"2024-02-19T14:53:47.745944+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:02.790030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731936+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:47.731936+0000","last_peered":"2024-02-19T14:53:47.731936+0000","last_clean":"2024-02-19T14:53:47.731936+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:47.731936+0000","last_undegraded":"2024-02-19T14:53:47.731936+0000","last_fullsized":"2024-02-19T14:53:47.731936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732414+0000","last_change":"2024-02-19T14:53:47.732414+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732414+0000","last_undegraded":"2024-02-19T14:53:47.732414+0000","last_fullsized":"2024-02-19T14:53:47.732414+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743039+0000","last_change":"2024-02-19T14:53:47.738834+0000","last_active":"2024-02-19T14:53:47.743039+0000","last_peered":"2024-02-19T14:53:47.743039+0000","last_clean":"2024-02-19T14:53:47.743039+0000","last_became_active":"2024-02-19T14:53:47.738567+0000","last_became_peered":"2024-02-19T14:53:47.738567+0000","last_unstale":"2024-02-19T14:53:47.743039+0000","last_undegraded":"2024-02-19T14:53:47.743039+0000","last_fullsized":"2024-02-19T14:53:47.743039+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:50.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727552+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:47.727552+0000","last_peered":"2024-02-19T14:53:47.727552+0000","last_clean":"2024-02-19T14:53:47.727552+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:47.727552+0000","last_undegraded":"2024-02-19T14:53:47.727552+0000","last_fullsized":"2024-02-19T14:53:47.727552+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724124+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:47.724124+0000","last_peered":"2024-02-19T14:53:47.724124+0000","last_clean":"2024-02-19T14:53:47.724124+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:47.724124+0000","last_undegraded":"2024-02-19T14:53:47.724124+0000","last_fullsized":"2024-02-19T14:53:47.724124+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728013+0000","last_change":"2024-02-19T14:53:47.728013+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728013+0000","last_undegraded":"2024-02-19T14:53:47.728013+0000","last_fullsized":"2024-02-19T14:53:47.728013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747961+0000","last_change":"2024-02-19T14:53:47.744007+0000","last_active":"2024-02-19T14:53:47.747961+0000","last_peered":"2024-02-19T14:53:47.747961+0000","last_clean":"2024-02-19T14:53:47.747961+0000","last_became_active":"2024-02-19T14:53:47.743726+0000","last_became_peered":"2024-02-19T14:53:47.743726+0000","last_unstale":"2024-02-19T14:53:47.747961+0000","last_undegraded":"2024-02-19T14:53:47.747961+0000","last_fullsized":"2024-02-19T14:53:47.747961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:26.617672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733834+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:47.733834+0000","last_peered":"2024-02-19T14:53:47.733834+0000","last_clean":"2024-02-19T14:53:47.733834+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:47.733834+0000","last_undegraded":"2024-02-19T14:53:47.733834+0000","last_fullsized":"2024-02-19T14:53:47.733834+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730733+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:47.730733+0000","last_peered":"2024-02-19T14:53:47.730733+0000","last_clean":"2024-02-19T14:53:47.730733+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:47.730733+0000","last_undegraded":"2024-02-19T14:53:47.730733+0000","last_fullsized":"2024-02-19T14:53:47.730733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732525+0000","last_change":"2024-02-19T14:53:47.732525+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732525+0000","last_undegraded":"2024-02-19T14:53:47.732525+0000","last_fullsized":"2024-02-19T14:53:47.732525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746917+0000","last_change":"2024-02-19T14:53:47.742015+0000","last_active":"2024-02-19T14:53:47.746917+0000","last_peered":"2024-02-19T14:53:47.746917+0000","last_clean":"2024-02-19T14:53:47.746917+0000","last_became_active":"2024-02-19T14:53:47.741742+0000","last_became_peered":"2024-02-19T14:53:47.741742+0000","last_unstale":"2024-02-19T14:53:47.746917+0000","last_undegraded":"2024-02-19T14:53:47.746917+0000","last_fullsized":"2024-02-19T14:53:47.746917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724348+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:47.724348+0000","last_peered":"2024-02-19T14:53:47.724348+0000","last_clean":"2024-02-19T14:53:47.724348+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:47.724348+0000","last_undegraded":"2024-02-19T14:53:47.724348+0000","last_fullsized":"2024-02-19T14:53:47.724348+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:45.189873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734386+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:47.734386+0000","last_peered":"2024-02-19T14:53:47.734386+0000","last_clean":"2024-02-19T14:53:47.734386+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:47.734386+0000","last_undegraded":"2024-02-19T14:53:47.734386+0000","last_fullsized":"2024-02-19T14:53:47.734386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728949+0000","last_change":"2024-02-19T14:53:47.728949+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728949+0000","last_undegraded":"2024-02-19T14:53:47.728949+0000","last_fullsized":"2024-02-19T14:53:47.728949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737489+0000","last_change":"2024-02-19T14:53:47.734524+0000","last_active":"2024-02-19T14:53:47.737489+0000","last_peered":"2024-02-19T14:53:47.737489+0000","last_clean":"2024-02-19T14:53:47.737489+0000","last_became_active":"2024-02-19T14:53:47.734132+0000","last_became_peered":"2024-02-19T14:53:47.734132+0000","last_unstale":"2024-02-19T14:53:47.737489+0000","last_undegraded":"2024-02-19T14:53:47.737489+0000","last_fullsized":"2024-02-19T14:53:47.737489+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737553+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:47.737553+0000","last_peered":"2024-02-19T14:53:47.737553+0000","last_clean":"2024-02-19T14:53:47.737553+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:47.737553+0000","last_undegraded":"2024-02-19T14:53:47.737553+0000","last_fullsized":"2024-02-19T14:53:47.737553+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734066+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:47.734066+0000","last_peered":"2024-02-19T14:53:47.734066+0000","last_clean":"2024-02-19T14:53:47.734066+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:47.734066+0000","last_undegraded":"2024-02-19T14:53:47.734066+0000","last_fullsized":"2024-02-19T14:53:47.734066+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":236,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729312+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:47.729312+0000","last_peered":"2024-02-19T14:53:47.729312+0000","last_clean":"2024-02-19T14:53:47.729312+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:47.729312+0000","last_undegraded":"2024-02-19T14:53:47.729312+0000","last_fullsized":"2024-02-19T14:53:47.729312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.502},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":291530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.502}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.626},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.311},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":299032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.624},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":292512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.245}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.372}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.621}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":292490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.31}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.5}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":298610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.247},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.20000000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.217},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.435},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.36399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.42399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.725964+0000', 'last_change': '2024-02-19T14:53:47.725964+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.725964+0000', 'last_undegraded': '2024-02-19T14:53:47.725964+0000', 'last_fullsized': '2024-02-19T14:53:47.725964+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3c is not active+clean 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731535+0000', 'last_change': '2024-02-19T14:53:47.731535+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731535+0000', 'last_undegraded': '2024-02-19T14:53:47.731535+0000', 'last_fullsized': '2024-02-19T14:53:47.731535+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3f is not active+clean 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728965+0000', 'last_change': '2024-02-19T14:53:47.728965+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728965+0000', 'last_undegraded': '2024-02-19T14:53:47.728965+0000', 'last_fullsized': '2024-02-19T14:53:47.728965+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3e is not active+clean 2024-02-19T14:53:52.643 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726570+0000', 'last_change': '2024-02-19T14:53:47.726570+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726570+0000', 'last_undegraded': '2024-02-19T14:53:47.726570+0000', 'last_fullsized': '2024-02-19T14:53:47.726570+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.39 is not active+clean 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.730942+0000', 'last_change': '2024-02-19T14:53:47.730942+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.730942+0000', 'last_undegraded': '2024-02-19T14:53:47.730942+0000', 'last_fullsized': '2024-02-19T14:53:47.730942+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.38 is not active+clean 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.729130+0000', 'last_change': '2024-02-19T14:53:47.729130+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.729130+0000', 'last_undegraded': '2024-02-19T14:53:47.729130+0000', 'last_fullsized': '2024-02-19T14:53:47.729130+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3b is not active+clean 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731517+0000', 'last_change': '2024-02-19T14:53:47.731517+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731517+0000', 'last_undegraded': '2024-02-19T14:53:47.731517+0000', 'last_fullsized': '2024-02-19T14:53:47.731517+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3a is not active+clean 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.727983+0000', 'last_change': '2024-02-19T14:53:47.727983+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.727983+0000', 'last_undegraded': '2024-02-19T14:53:47.727983+0000', 'last_fullsized': '2024-02-19T14:53:47.727983+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.35 is not active+clean 2024-02-19T14:53:52.644 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.733825+0000', 'last_change': '2024-02-19T14:53:47.733825+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.733825+0000', 'last_undegraded': '2024-02-19T14:53:47.733825+0000', 'last_fullsized': '2024-02-19T14:53:47.733825+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.34 is not active+clean 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.733022+0000', 'last_change': '2024-02-19T14:53:47.733022+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.733022+0000', 'last_undegraded': '2024-02-19T14:53:47.733022+0000', 'last_fullsized': '2024-02-19T14:53:47.733022+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.725964+0000', 'last_change': '2024-02-19T14:53:47.725964+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.725964+0000', 'last_undegraded': '2024-02-19T14:53:47.725964+0000', 'last_fullsized': '2024-02-19T14:53:47.725964+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726799+0000', 'last_change': '2024-02-19T14:53:47.726799+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726799+0000', 'last_undegraded': '2024-02-19T14:53:47.726799+0000', 'last_fullsized': '2024-02-19T14:53:47.726799+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726754+0000', 'last_change': '2024-02-19T14:53:47.726754+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726754+0000', 'last_undegraded': '2024-02-19T14:53:47.726754+0000', 'last_fullsized': '2024-02-19T14:53:47.726754+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.30 is not active+clean 2024-02-19T14:53:52.645 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731147+0000', 'last_change': '2024-02-19T14:53:47.731147+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731147+0000', 'last_undegraded': '2024-02-19T14:53:47.731147+0000', 'last_fullsized': '2024-02-19T14:53:47.731147+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.33 is not active+clean 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.734360+0000', 'last_change': '2024-02-19T14:53:47.734360+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.734360+0000', 'last_undegraded': '2024-02-19T14:53:47.734360+0000', 'last_fullsized': '2024-02-19T14:53:47.734360+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726289+0000', 'last_change': '2024-02-19T14:53:47.726289+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726289+0000', 'last_undegraded': '2024-02-19T14:53:47.726289+0000', 'last_fullsized': '2024-02-19T14:53:47.726289+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2d is not active+clean 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731901+0000', 'last_change': '2024-02-19T14:53:47.731901+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731901+0000', 'last_undegraded': '2024-02-19T14:53:47.731901+0000', 'last_fullsized': '2024-02-19T14:53:47.731901+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2c is not active+clean 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726614+0000', 'last_change': '2024-02-19T14:53:47.726614+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726614+0000', 'last_undegraded': '2024-02-19T14:53:47.726614+0000', 'last_fullsized': '2024-02-19T14:53:47.726614+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2f is not active+clean 2024-02-19T14:53:52.646 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728581+0000', 'last_change': '2024-02-19T14:53:47.728581+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728581+0000', 'last_undegraded': '2024-02-19T14:53:47.728581+0000', 'last_fullsized': '2024-02-19T14:53:47.728581+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2e is not active+clean 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.735616+0000', 'last_change': '2024-02-19T14:53:47.735616+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.735616+0000', 'last_undegraded': '2024-02-19T14:53:47.735616+0000', 'last_fullsized': '2024-02-19T14:53:47.735616+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.29 is not active+clean 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726307+0000', 'last_change': '2024-02-19T14:53:47.726307+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726307+0000', 'last_undegraded': '2024-02-19T14:53:47.726307+0000', 'last_fullsized': '2024-02-19T14:53:47.726307+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.725966+0000', 'last_change': '2024-02-19T14:53:47.725966+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.725966+0000', 'last_undegraded': '2024-02-19T14:53:47.725966+0000', 'last_fullsized': '2024-02-19T14:53:47.725966+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2b is not active+clean 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.729507+0000', 'last_change': '2024-02-19T14:53:47.729507+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.729507+0000', 'last_undegraded': '2024-02-19T14:53:47.729507+0000', 'last_fullsized': '2024-02-19T14:53:47.729507+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2a is not active+clean 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726592+0000', 'last_change': '2024-02-19T14:53:47.726592+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726592+0000', 'last_undegraded': '2024-02-19T14:53:47.726592+0000', 'last_fullsized': '2024-02-19T14:53:47.726592+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.25 is not active+clean 2024-02-19T14:53:52.647 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.734650+0000', 'last_change': '2024-02-19T14:53:47.734650+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.734650+0000', 'last_undegraded': '2024-02-19T14:53:47.734650+0000', 'last_fullsized': '2024-02-19T14:53:47.734650+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.24 is not active+clean 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.729138+0000', 'last_change': '2024-02-19T14:53:47.729138+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.729138+0000', 'last_undegraded': '2024-02-19T14:53:47.729138+0000', 'last_fullsized': '2024-02-19T14:53:47.729138+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.27 is not active+clean 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.734486+0000', 'last_change': '2024-02-19T14:53:47.734486+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.734486+0000', 'last_undegraded': '2024-02-19T14:53:47.734486+0000', 'last_fullsized': '2024-02-19T14:53:47.734486+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.26 is not active+clean 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.733413+0000', 'last_change': '2024-02-19T14:53:47.733413+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.733413+0000', 'last_undegraded': '2024-02-19T14:53:47.733413+0000', 'last_fullsized': '2024-02-19T14:53:47.733413+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.21 is not active+clean 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.729994+0000', 'last_change': '2024-02-19T14:53:47.729994+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.729994+0000', 'last_undegraded': '2024-02-19T14:53:47.729994+0000', 'last_fullsized': '2024-02-19T14:53:47.729994+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.20 is not active+clean 2024-02-19T14:53:52.648 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728586+0000', 'last_change': '2024-02-19T14:53:47.728586+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728586+0000', 'last_undegraded': '2024-02-19T14:53:47.728586+0000', 'last_fullsized': '2024-02-19T14:53:47.728586+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726173+0000', 'last_change': '2024-02-19T14:53:47.726173+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726173+0000', 'last_undegraded': '2024-02-19T14:53:47.726173+0000', 'last_fullsized': '2024-02-19T14:53:47.726173+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.22 is not active+clean 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.727599+0000', 'last_change': '2024-02-19T14:53:47.727599+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.727599+0000', 'last_undegraded': '2024-02-19T14:53:47.727599+0000', 'last_fullsized': '2024-02-19T14:53:47.727599+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.735808+0000', 'last_change': '2024-02-19T14:53:47.735808+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.735808+0000', 'last_undegraded': '2024-02-19T14:53:47.735808+0000', 'last_fullsized': '2024-02-19T14:53:47.735808+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.13 is not active+clean 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.727391+0000', 'last_change': '2024-02-19T14:53:47.727391+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.727391+0000', 'last_undegraded': '2024-02-19T14:53:47.727391+0000', 'last_fullsized': '2024-02-19T14:53:47.727391+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.729346+0000', 'last_change': '2024-02-19T14:53:47.729346+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.729346+0000', 'last_undegraded': '2024-02-19T14:53:47.729346+0000', 'last_fullsized': '2024-02-19T14:53:47.729346+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2024-02-19T14:53:52.649 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726858+0000', 'last_change': '2024-02-19T14:53:47.726858+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726858+0000', 'last_undegraded': '2024-02-19T14:53:47.726858+0000', 'last_fullsized': '2024-02-19T14:53:47.726858+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.732000+0000', 'last_change': '2024-02-19T14:53:47.732000+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.732000+0000', 'last_undegraded': '2024-02-19T14:53:47.732000+0000', 'last_fullsized': '2024-02-19T14:53:47.732000+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.736522+0000', 'last_change': '2024-02-19T14:53:47.736522+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.736522+0000', 'last_undegraded': '2024-02-19T14:53:47.736522+0000', 'last_fullsized': '2024-02-19T14:53:47.736522+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.14 is not active+clean 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728755+0000', 'last_change': '2024-02-19T14:53:47.728755+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728755+0000', 'last_undegraded': '2024-02-19T14:53:47.728755+0000', 'last_fullsized': '2024-02-19T14:53:47.728755+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726342+0000', 'last_change': '2024-02-19T14:53:47.726342+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726342+0000', 'last_undegraded': '2024-02-19T14:53:47.726342+0000', 'last_fullsized': '2024-02-19T14:53:47.726342+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2024-02-19T14:53:52.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728855+0000', 'last_change': '2024-02-19T14:53:47.728855+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728855+0000', 'last_undegraded': '2024-02-19T14:53:47.728855+0000', 'last_fullsized': '2024-02-19T14:53:47.728855+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726359+0000', 'last_change': '2024-02-19T14:53:47.726359+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726359+0000', 'last_undegraded': '2024-02-19T14:53:47.726359+0000', 'last_fullsized': '2024-02-19T14:53:47.726359+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.18 is not active+clean 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731814+0000', 'last_change': '2024-02-19T14:53:47.731814+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731814+0000', 'last_undegraded': '2024-02-19T14:53:47.731814+0000', 'last_fullsized': '2024-02-19T14:53:47.731814+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.19 is not active+clean 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.727760+0000', 'last_change': '2024-02-19T14:53:47.727760+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.727760+0000', 'last_undegraded': '2024-02-19T14:53:47.727760+0000', 'last_fullsized': '2024-02-19T14:53:47.727760+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731071+0000', 'last_change': '2024-02-19T14:53:47.731071+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731071+0000', 'last_undegraded': '2024-02-19T14:53:47.731071+0000', 'last_fullsized': '2024-02-19T14:53:47.731071+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.726717+0000', 'last_change': '2024-02-19T14:53:47.726717+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.726717+0000', 'last_undegraded': '2024-02-19T14:53:47.726717+0000', 'last_fullsized': '2024-02-19T14:53:47.726717+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731538+0000', 'last_change': '2024-02-19T14:53:47.731538+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731538+0000', 'last_undegraded': '2024-02-19T14:53:47.731538+0000', 'last_fullsized': '2024-02-19T14:53:47.731538+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.734244+0000', 'last_change': '2024-02-19T14:53:47.734244+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.734244+0000', 'last_undegraded': '2024-02-19T14:53:47.734244+0000', 'last_fullsized': '2024-02-19T14:53:47.734244+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2 is not active+clean 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.727027+0000', 'last_change': '2024-02-19T14:53:47.727027+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.727027+0000', 'last_undegraded': '2024-02-19T14:53:47.727027+0000', 'last_fullsized': '2024-02-19T14:53:47.727027+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3 is not active+clean 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.730482+0000', 'last_change': '2024-02-19T14:53:47.730482+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.730482+0000', 'last_undegraded': '2024-02-19T14:53:47.730482+0000', 'last_fullsized': '2024-02-19T14:53:47.730482+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728440+0000', 'last_change': '2024-02-19T14:53:47.728440+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728440+0000', 'last_undegraded': '2024-02-19T14:53:47.728440+0000', 'last_fullsized': '2024-02-19T14:53:47.728440+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2024-02-19T14:53:52.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.730447+0000', 'last_change': '2024-02-19T14:53:47.730447+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.730447+0000', 'last_undegraded': '2024-02-19T14:53:47.730447+0000', 'last_fullsized': '2024-02-19T14:53:47.730447+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728393+0000', 'last_change': '2024-02-19T14:53:47.728393+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728393+0000', 'last_undegraded': '2024-02-19T14:53:47.728393+0000', 'last_fullsized': '2024-02-19T14:53:47.728393+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731908+0000', 'last_change': '2024-02-19T14:53:47.731908+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731908+0000', 'last_undegraded': '2024-02-19T14:53:47.731908+0000', 'last_fullsized': '2024-02-19T14:53:47.731908+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728116+0000', 'last_change': '2024-02-19T14:53:47.728116+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728116+0000', 'last_undegraded': '2024-02-19T14:53:47.728116+0000', 'last_fullsized': '2024-02-19T14:53:47.728116+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.5 is not active+clean 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731324+0000', 'last_change': '2024-02-19T14:53:47.731324+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731324+0000', 'last_undegraded': '2024-02-19T14:53:47.731324+0000', 'last_fullsized': '2024-02-19T14:53:47.731324+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.731805+0000', 'last_change': '2024-02-19T14:53:47.731805+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.731805+0000', 'last_undegraded': '2024-02-19T14:53:47.731805+0000', 'last_fullsized': '2024-02-19T14:53:47.731805+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-02-19T14:53:52.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728207+0000', 'last_change': '2024-02-19T14:53:47.728207+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728207+0000', 'last_undegraded': '2024-02-19T14:53:47.728207+0000', 'last_fullsized': '2024-02-19T14:53:47.728207+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.727406+0000', 'last_change': '2024-02-19T14:53:47.727406+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.727406+0000', 'last_undegraded': '2024-02-19T14:53:47.727406+0000', 'last_fullsized': '2024-02-19T14:53:47.727406+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.735192+0000', 'last_change': '2024-02-19T14:53:47.735192+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.735192+0000', 'last_undegraded': '2024-02-19T14:53:47.735192+0000', 'last_fullsized': '2024-02-19T14:53:47.735192+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.732414+0000', 'last_change': '2024-02-19T14:53:47.732414+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.732414+0000', 'last_undegraded': '2024-02-19T14:53:47.732414+0000', 'last_fullsized': '2024-02-19T14:53:47.732414+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-02-19T14:53:52.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728013+0000', 'last_change': '2024-02-19T14:53:47.728013+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728013+0000', 'last_undegraded': '2024-02-19T14:53:47.728013+0000', 'last_fullsized': '2024-02-19T14:53:47.728013+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:53:52.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2024-02-19T14:53:52.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.732525+0000', 'last_change': '2024-02-19T14:53:47.732525+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.732525+0000', 'last_undegraded': '2024-02-19T14:53:47.732525+0000', 'last_fullsized': '2024-02-19T14:53:47.732525+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2024-02-19T14:53:52.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 55, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:53:47.728949+0000', 'last_change': '2024-02-19T14:53:47.728949+0000', 'last_active': '2024-02-19T14:53:47.716914+0000', 'last_peered': '2024-02-19T14:53:47.716914+0000', 'last_clean': '2024-02-19T14:53:47.716914+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:53:47.728949+0000', 'last_undegraded': '2024-02-19T14:53:47.728949+0000', 'last_fullsized': '2024-02-19T14:53:47.728949+0000', 'mapping_epoch': 55, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 55, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'last_clean_scrub_stamp': '2024-02-19T14:53:47.716914+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:53:52.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T14:53:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.780+0000 7f764e01c640 1 Processor -- start 2024-02-19T14:53:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.780+0000 7f764e01c640 1 -- start start 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f764e01c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7648150460 0x7f76480c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f764e01c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76480721a0 0x7f76480c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f764e01c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7648072c60 con 0x7f76480c7d90 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f764e01c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7648153e80 con 0x7f76480721a0 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f764e01c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76481538c0 con 0x7f7648150460 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f7646ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76480721a0 0x7f76480c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f7646ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76480721a0 0x7f76480c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53424/0 (socket says 172.21.15.78:53424) 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.781+0000 7f7646ffd640 1 -- 172.21.15.78:0/407621271 learned_addr learned my addr 172.21.15.78:0/407621271 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f76477fe640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7648150460 0x7f76480c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f7646ffd640 1 -- 172.21.15.78:0/407621271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7648150460 msgr2=0x7f76480c1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f7646ffd640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7648150460 0x7f76480c1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f7646ffd640 1 -- 172.21.15.78:0/407621271 >> v1:172.21.15.78:6789/0 conn(0x7f76480c7d90 legacy=0x7f76480c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f76467fc640 1 -- 172.21.15.78:0/407621271 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928383686 0 0) 0x7f7648072c60 con 0x7f76480c7d90 2024-02-19T14:53:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f7646ffd640 1 -- 172.21.15.78:0/407621271 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7648153ba0 con 0x7f76480721a0 2024-02-19T14:53:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.782+0000 7f76477fe640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7648150460 0x7f76480c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.783+0000 7f7646ffd640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76480721a0 0x7f76480c7850 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7630009870 tx=0x7f763002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1688aaea73214b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.783+0000 7f76467fc640 1 -- 172.21.15.78:0/407621271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630030840 con 0x7f76480721a0 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.783+0000 7f76467fc640 1 -- 172.21.15.78:0/407621271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76300309a0 con 0x7f76480721a0 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.783+0000 7f76467fc640 1 -- 172.21.15.78:0/407621271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630030d30 con 0x7f76480721a0 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.783+0000 7f764e01c640 1 -- 172.21.15.78:0/407621271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76480721a0 msgr2=0x7f76480c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.783+0000 7f764e01c640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76480721a0 0x7f76480c7850 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7630009870 tx=0x7f763002b980 comp rx=0 tx=0).stop 2024-02-19T14:53:52.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.784+0000 7f764e01c640 1 -- 172.21.15.78:0/407621271 shutdown_connections 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.784+0000 7f764e01c640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76480721a0 0x7f76480c7850 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.784+0000 7f764e01c640 1 --2- 172.21.15.78:0/407621271 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7648150460 0x7f76480c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.784+0000 7f764e01c640 1 -- 172.21.15.78:0/407621271 >> 172.21.15.78:0/407621271 conn(0x7f764806c170 msgr2=0x7f764806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.784+0000 7f764e01c640 1 -- 172.21.15.78:0/407621271 shutdown_connections 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.784+0000 7f764e01c640 1 -- 172.21.15.78:0/407621271 wait complete. 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.785+0000 7f764e01c640 1 Processor -- start 2024-02-19T14:53:52.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.785+0000 7f764e01c640 1 -- start start 2024-02-19T14:53:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f764e01c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76480721a0 0x7f76481e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f764e01c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76480c7d90 0x7f76480c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f764e01c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 0x7f7648149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f764e01c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480bdb20 con 0x7f7648150460 2024-02-19T14:53:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f764e01c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481e9310 con 0x7f76480721a0 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f764e01c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480bd440 con 0x7f76480c7d90 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 0x7f7648149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 0x7f7648149900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53440/0 (socket says 172.21.15.78:53440) 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 -- 172.21.15.78:0/568017479 learned_addr learned my addr 172.21.15.78:0/568017479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f76477fe640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76480721a0 0x7f76481e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7646ffd640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76480c7d90 0x7f76480c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 -- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76480c7d90 msgr2=0x7f76480c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76480c7d90 0x7f76480c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 -- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76480721a0 msgr2=0x7f76481e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76480721a0 0x7f76481e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.786+0000 7f7647fff640 1 -- 172.21.15.78:0/568017479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76300094d0 con 0x7f7648150460 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.787+0000 7f7647fff640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 0x7f7648149900 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f764000c480 tx=0x7f764000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.787+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640005480 con 0x7f7648150460 2024-02-19T14:53:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.787+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7648155440 con 0x7f7648150460 2024-02-19T14:53:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.787+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76400035f0 con 0x7f7648150460 2024-02-19T14:53:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.787+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76400039a0 con 0x7f7648150460 2024-02-19T14:53:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.787+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7648155a00 con 0x7f7648150460 2024-02-19T14:53:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.788+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648070150 con 0x7f7648150460 2024-02-19T14:53:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.788+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7640003b00 con 0x7f7648150460 2024-02-19T14:53:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.789+0000 7f763ffff640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.789+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f764008a160 con 0x7f7648150460 2024-02-19T14:53:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.789+0000 7f76477fe640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.789+0000 7f76477fe640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f76480c1bf0 tx=0x7f7638009890 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:52.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:52.793+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7640058250 con 0x7f7648150460 2024-02-19T14:53:52.946 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:53:52.944+0000 7ff8f32cd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.039+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7648074990 con 0x7f7648150460 2024-02-19T14:53:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.039+0000 7f763ffff640 1 -- 172.21.15.78:0/568017479 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1428 (secure 0 0 0) 0x7f7640057de0 con 0x7f7648150460 2024-02-19T14:53:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:53.042 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":137,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1708354307,"num_in_osds":8,"osd_in_since":1708354298,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129},{"state_name":"creating+peering","count":64}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":592778,"bytes_used":227778560,"bytes_avail":772866334720,"bytes_total":773094113280,"inactive_pgs_ratio":0.33160620927810669},"fsmap":{"epoch":30,"id":3,"up":1,"in":1,"max":1,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":5700},{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5703}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-02-19T14:52:56.071883+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 msgr2=0x7f762806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f76480c1bf0 tx=0x7f7638009890 comp rx=0 tx=0).stop 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 msgr2=0x7f7648149900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 0x7f7648149900 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f764000c480 tx=0x7f764000c940 comp rx=0 tx=0).stop 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 shutdown_connections 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f762806b810 0x7f762806dcd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7648150460 0x7f7648149900 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76480c7d90 0x7f76480c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 --2- 172.21.15.78:0/568017479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76480721a0 0x7f76481e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 >> 172.21.15.78:0/568017479 conn(0x7f764806c170 msgr2=0x7f764806f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 shutdown_connections 2024-02-19T14:53:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.041+0000 7f764e01c640 1 -- 172.21.15.78:0/568017479 wait complete. 2024-02-19T14:53:53.055 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T14:53:53.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.163+0000 7f53d2b01640 1 Processor -- start 2024-02-19T14:53:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.163+0000 7f53d2b01640 1 -- start start 2024-02-19T14:53:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.164+0000 7f53d2b01640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.164+0000 7f53d2b01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53cc06d910 0x7f53cc06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.164+0000 7f53d2b01640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc06cd20 con 0x7f53cc152700 2024-02-19T14:53:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.164+0000 7f53d2b01640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53cc1551f0 con 0x7f53cc06d910 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.164+0000 7f53d2b01640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc155490 con 0x7f53cc0c8e00 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d109e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53cc06d910 0x7f53cc06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d089d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f53cc152700 0x7f53cc0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60346/0 (socket says 172.21.15.78:60346) 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d089d640 1 -- 172.21.15.78:0/3382931342 learned_addr learned my addr 172.21.15.78:0/3382931342 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d109e640 1 -- 172.21.15.78:0/3382931342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 msgr2=0x7f53cc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d109e640 1 --2- 172.21.15.78:0/3382931342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d109e640 1 -- 172.21.15.78:0/3382931342 >> v1:172.21.15.78:6789/0 conn(0x7f53cc152700 legacy=0x7f53cc0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.165+0000 7f53d109e640 1 -- 172.21.15.78:0/3382931342 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc155770 con 0x7f53cc06d910 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d109e640 1 --2- 172.21.15.78:0/3382931342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc06d910 0x7f53cc06c7e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009d90 tx=0x7f53c402bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ba520d8bd9fce2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53cb7fe640 1 -- 172.21.15.78:0/3382931342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c402c980 con 0x7f53cc06d910 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53cb7fe640 1 -- 172.21.15.78:0/3382931342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c402cae0 con 0x7f53cc06d910 2024-02-19T14:53:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53cb7fe640 1 -- 172.21.15.78:0/3382931342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c402ce70 con 0x7f53cc06d910 2024-02-19T14:53:53.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d2b01640 1 -- 172.21.15.78:0/3382931342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc06d910 msgr2=0x7f53cc06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d2b01640 1 --2- 172.21.15.78:0/3382931342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc06d910 0x7f53cc06c7e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009d90 tx=0x7f53c402bc50 comp rx=0 tx=0).stop 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d2b01640 1 -- 172.21.15.78:0/3382931342 shutdown_connections 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d2b01640 1 --2- 172.21.15.78:0/3382931342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc06d910 0x7f53cc06c7e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d2b01640 1 --2- 172.21.15.78:0/3382931342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.166+0000 7f53d2b01640 1 -- 172.21.15.78:0/3382931342 >> 172.21.15.78:0/3382931342 conn(0x7f53cc0cac90 msgr2=0x7f53cc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.167+0000 7f53d2b01640 1 -- 172.21.15.78:0/3382931342 shutdown_connections 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.167+0000 7f53d2b01640 1 -- 172.21.15.78:0/3382931342 wait complete. 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 Processor -- start 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 -- start start 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53cc06d910 0x7f53cc1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc0c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 0x7f53cc0aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc0ada90 con 0x7f53cc152700 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc1499e0 con 0x7f53cc06d910 2024-02-19T14:53:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.168+0000 7f53d2b01640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc0aaef0 con 0x7f53cc0c8e00 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 0x7f53cc0aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 0x7f53cc0aec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53454/0 (socket says 172.21.15.78:53454) 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 -- 172.21.15.78:0/2935283215 learned_addr learned my addr 172.21.15.78:0/2935283215 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d089d640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53cc06d910 0x7f53cc1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53cbfff640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc0c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 -- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 msgr2=0x7f53cc0c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc0c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 -- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53cc06d910 msgr2=0x7f53cc1ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53cc06d910 0x7f53cc1ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 -- 172.21.15.78:0/2935283215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4009a70 con 0x7f53cc152700 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d089d640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53cc06d910 0x7f53cc1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53cbfff640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc0c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d109e640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 0x7f53cc0aec80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f53c4037040 tx=0x7f53c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c402c900 con 0x7f53cc152700 2024-02-19T14:53:53.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c4040450 con 0x7f53cc152700 2024-02-19T14:53:53.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc0ab4b0 con 0x7f53cc152700 2024-02-19T14:53:53.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4040800 con 0x7f53cc152700 2024-02-19T14:53:53.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.169+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc0ab140 con 0x7f53cc152700 2024-02-19T14:53:53.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.170+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc0ab640 con 0x7f53cc152700 2024-02-19T14:53:53.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.171+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f53c4040960 con 0x7f53cc152700 2024-02-19T14:53:53.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.171+0000 7f53c97fa640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53b006b810 0x7f53b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:53.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.171+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f53c40ad330 con 0x7f53cc152700 2024-02-19T14:53:53.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.171+0000 7f53d089d640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53b006b810 0x7f53b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:53.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.172+0000 7f53d089d640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53b006b810 0x7f53b006dcd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53c0000f60 tx=0x7f53c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.176+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f53c40778f0 con 0x7f53cc152700 2024-02-19T14:53:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.460+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53cc0c5ad0 con 0x7f53b006b810 2024-02-19T14:53:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.472+0000 7f53c97fa640 1 -- 172.21.15.78:0/2935283215 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+522875 (secure 0 0 0) 0x7f53cc0c5ad0 con 0x7f53b006b810 2024-02-19T14:53:53.480 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:53.484 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53b006b810 msgr2=0x7f53b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53b006b810 0x7f53b006dcd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53c0000f60 tx=0x7f53c0009040 comp rx=0 tx=0).stop 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 msgr2=0x7f53cc0aec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 0x7f53cc0aec80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f53c4037040 tx=0x7f53c4004a40 comp rx=0 tx=0).stop 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 shutdown_connections 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53b006b810 0x7f53b006dcd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53cc152700 0x7f53cc0aec80 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53cc0c8e00 0x7f53cc0c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 --2- 172.21.15.78:0/2935283215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53cc06d910 0x7f53cc1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 >> 172.21.15.78:0/2935283215 conn(0x7f53cc0cac90 msgr2=0x7f53cc14e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:53.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 shutdown_connections 2024-02-19T14:53:53.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:53.477+0000 7f53d2b01640 1 -- 172.21.15.78:0/2935283215 wait complete. 2024-02-19T14:53:53.491 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-02-19T14:53:52.082264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222440,"kb_used_data":6272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752280,"statfs":{"total":773094113280,"available":772866334720,"internally_reserved":0,"allocated":6422528,"data_stored":3527724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.364855"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730654+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:47.730654+0000","last_peered":"2024-02-19T14:53:47.730654+0000","last_clean":"2024-02-19T14:53:47.730654+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:47.730654+0000","last_undegraded":"2024-02-19T14:53:47.730654+0000","last_fullsized":"2024-02-19T14:53:47.730654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731535+0000","last_change":"2024-02-19T14:53:47.731535+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731535+0000","last_undegraded":"2024-02-19T14:53:47.731535+0000","last_fullsized":"2024-02-19T14:53:47.731535+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730461+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:47.730461+0000","last_peered":"2024-02-19T14:53:47.730461+0000","last_clean":"2024-02-19T14:53:47.730461+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:47.730461+0000","last_undegraded":"2024-02-19T14:53:47.730461+0000","last_fullsized":"2024-02-19T14:53:47.730461+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728965+0000","last_change":"2024-02-19T14:53:47.728965+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728965+0000","last_undegraded":"2024-02-19T14:53:47.728965+0000","last_fullsized":"2024-02-19T14:53:47.728965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732216+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:47.732216+0000","last_peered":"2024-02-19T14:53:47.732216+0000","last_clean":"2024-02-19T14:53:47.732216+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:47.732216+0000","last_undegraded":"2024-02-19T14:53:47.732216+0000","last_fullsized":"2024-02-19T14:53:47.732216+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726570+0000","last_change":"2024-02-19T14:53:47.726570+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726570+0000","last_undegraded":"2024-02-19T14:53:47.726570+0000","last_fullsized":"2024-02-19T14:53:47.726570+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729626+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:47.729626+0000","last_peered":"2024-02-19T14:53:47.729626+0000","last_clean":"2024-02-19T14:53:47.729626+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:47.729626+0000","last_undegraded":"2024-02-19T14:53:47.729626+0000","last_fullsized":"2024-02-19T14:53:47.729626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730942+0000","last_change":"2024-02-19T14:53:47.730942+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730942+0000","last_undegraded":"2024-02-19T14:53:47.730942+0000","last_fullsized":"2024-02-19T14:53:47.730942+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731084+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:47.731084+0000","last_peered":"2024-02-19T14:53:47.731084+0000","last_clean":"2024-02-19T14:53:47.731084+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:47.731084+0000","last_undegraded":"2024-02-19T14:53:47.731084+0000","last_fullsized":"2024-02-19T14:53:47.731084+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729130+0000","last_change":"2024-02-19T14:53:47.729130+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729130+0000","last_undegraded":"2024-02-19T14:53:47.729130+0000","last_fullsized":"2024-02-19T14:53:47.729130+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729373+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:47.729373+0000","last_peered":"2024-02-19T14:53:47.729373+0000","last_clean":"2024-02-19T14:53:47.729373+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:47.729373+0000","last_undegraded":"2024-02-19T14:53:47.729373+0000","last_fullsized":"2024-02-19T14:53:47.729373+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731517+0000","last_change":"2024-02-19T14:53:47.731517+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731517+0000","last_undegraded":"2024-02-19T14:53:47.731517+0000","last_fullsized":"2024-02-19T14:53:47.731517+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731295+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:47.731295+0000","last_peered":"2024-02-19T14:53:47.731295+0000","last_clean":"2024-02-19T14:53:47.731295+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:47.731295+0000","last_undegraded":"2024-02-19T14:53:47.731295+0000","last_fullsized":"2024-02-19T14:53:47.731295+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727983+0000","last_change":"2024-02-19T14:53:47.727983+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727983+0000","last_undegraded":"2024-02-19T14:53:47.727983+0000","last_fullsized":"2024-02-19T14:53:47.727983+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723628+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:47.723628+0000","last_peered":"2024-02-19T14:53:47.723628+0000","last_clean":"2024-02-19T14:53:47.723628+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:47.723628+0000","last_undegraded":"2024-02-19T14:53:47.723628+0000","last_fullsized":"2024-02-19T14:53:47.723628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733825+0000","last_change":"2024-02-19T14:53:47.733825+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733825+0000","last_undegraded":"2024-02-19T14:53:47.733825+0000","last_fullsized":"2024-02-19T14:53:47.733825+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724256+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:47.724256+0000","last_peered":"2024-02-19T14:53:47.724256+0000","last_clean":"2024-02-19T14:53:47.724256+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:47.724256+0000","last_undegraded":"2024-02-19T14:53:47.724256+0000","last_fullsized":"2024-02-19T14:53:47.724256+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:50.707228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733022+0000","last_change":"2024-02-19T14:53:47.733022+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733022+0000","last_undegraded":"2024-02-19T14:53:47.733022+0000","last_fullsized":"2024-02-19T14:53:47.733022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730312+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:47.730312+0000","last_peered":"2024-02-19T14:53:47.730312+0000","last_clean":"2024-02-19T14:53:47.730312+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:47.730312+0000","last_undegraded":"2024-02-19T14:53:47.730312+0000","last_fullsized":"2024-02-19T14:53:47.730312+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725964+0000","last_change":"2024-02-19T14:53:47.725964+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725964+0000","last_undegraded":"2024-02-19T14:53:47.725964+0000","last_fullsized":"2024-02-19T14:53:47.725964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733611+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:47.733611+0000","last_peered":"2024-02-19T14:53:47.733611+0000","last_clean":"2024-02-19T14:53:47.733611+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:47.733611+0000","last_undegraded":"2024-02-19T14:53:47.733611+0000","last_fullsized":"2024-02-19T14:53:47.733611+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:37.193943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726799+0000","last_change":"2024-02-19T14:53:47.726799+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726799+0000","last_undegraded":"2024-02-19T14:53:47.726799+0000","last_fullsized":"2024-02-19T14:53:47.726799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724742+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:47.724742+0000","last_peered":"2024-02-19T14:53:47.724742+0000","last_clean":"2024-02-19T14:53:47.724742+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:47.724742+0000","last_undegraded":"2024-02-19T14:53:47.724742+0000","last_fullsized":"2024-02-19T14:53:47.724742+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:01.639745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726754+0000","last_change":"2024-02-19T14:53:47.726754+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726754+0000","last_undegraded":"2024-02-19T14:53:47.726754+0000","last_fullsized":"2024-02-19T14:53:47.726754+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729060+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:47.729060+0000","last_peered":"2024-02-19T14:53:47.729060+0000","last_clean":"2024-02-19T14:53:47.729060+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:47.729060+0000","last_undegraded":"2024-02-19T14:53:47.729060+0000","last_fullsized":"2024-02-19T14:53:47.729060+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731147+0000","last_change":"2024-02-19T14:53:47.731147+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731147+0000","last_undegraded":"2024-02-19T14:53:47.731147+0000","last_fullsized":"2024-02-19T14:53:47.731147+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739485+0000","last_change":"2024-02-19T14:53:42.136272+0000","last_active":"2024-02-19T14:53:47.739485+0000","last_peered":"2024-02-19T14:53:47.739485+0000","last_clean":"2024-02-19T14:53:47.739485+0000","last_became_active":"2024-02-19T14:53:42.135998+0000","last_became_peered":"2024-02-19T14:53:42.135998+0000","last_unstale":"2024-02-19T14:53:47.739485+0000","last_undegraded":"2024-02-19T14:53:47.739485+0000","last_fullsized":"2024-02-19T14:53:47.739485+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734360+0000","last_change":"2024-02-19T14:53:47.734360+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734360+0000","last_undegraded":"2024-02-19T14:53:47.734360+0000","last_fullsized":"2024-02-19T14:53:47.734360+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729068+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:47.729068+0000","last_peered":"2024-02-19T14:53:47.729068+0000","last_clean":"2024-02-19T14:53:47.729068+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:47.729068+0000","last_undegraded":"2024-02-19T14:53:47.729068+0000","last_fullsized":"2024-02-19T14:53:47.729068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726289+0000","last_change":"2024-02-19T14:53:47.726289+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726289+0000","last_undegraded":"2024-02-19T14:53:47.726289+0000","last_fullsized":"2024-02-19T14:53:47.726289+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727510+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:47.727510+0000","last_peered":"2024-02-19T14:53:47.727510+0000","last_clean":"2024-02-19T14:53:47.727510+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:47.727510+0000","last_undegraded":"2024-02-19T14:53:47.727510+0000","last_fullsized":"2024-02-19T14:53:47.727510+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731901+0000","last_change":"2024-02-19T14:53:47.731901+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731901+0000","last_undegraded":"2024-02-19T14:53:47.731901+0000","last_fullsized":"2024-02-19T14:53:47.731901+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734311+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:47.734311+0000","last_peered":"2024-02-19T14:53:47.734311+0000","last_clean":"2024-02-19T14:53:47.734311+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:47.734311+0000","last_undegraded":"2024-02-19T14:53:47.734311+0000","last_fullsized":"2024-02-19T14:53:47.734311+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726614+0000","last_change":"2024-02-19T14:53:47.726614+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726614+0000","last_undegraded":"2024-02-19T14:53:47.726614+0000","last_fullsized":"2024-02-19T14:53:47.726614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730365+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:47.730365+0000","last_peered":"2024-02-19T14:53:47.730365+0000","last_clean":"2024-02-19T14:53:47.730365+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:47.730365+0000","last_undegraded":"2024-02-19T14:53:47.730365+0000","last_fullsized":"2024-02-19T14:53:47.730365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728581+0000","last_change":"2024-02-19T14:53:47.728581+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728581+0000","last_undegraded":"2024-02-19T14:53:47.728581+0000","last_fullsized":"2024-02-19T14:53:47.728581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732357+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:47.732357+0000","last_peered":"2024-02-19T14:53:47.732357+0000","last_clean":"2024-02-19T14:53:47.732357+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:47.732357+0000","last_undegraded":"2024-02-19T14:53:47.732357+0000","last_fullsized":"2024-02-19T14:53:47.732357+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735616+0000","last_change":"2024-02-19T14:53:47.735616+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735616+0000","last_undegraded":"2024-02-19T14:53:47.735616+0000","last_fullsized":"2024-02-19T14:53:47.735616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737614+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:47.737614+0000","last_peered":"2024-02-19T14:53:47.737614+0000","last_clean":"2024-02-19T14:53:47.737614+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:47.737614+0000","last_undegraded":"2024-02-19T14:53:47.737614+0000","last_fullsized":"2024-02-19T14:53:47.737614+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726307+0000","last_change":"2024-02-19T14:53:47.726307+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726307+0000","last_undegraded":"2024-02-19T14:53:47.726307+0000","last_fullsized":"2024-02-19T14:53:47.726307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739540+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:47.739540+0000","last_peered":"2024-02-19T14:53:47.739540+0000","last_clean":"2024-02-19T14:53:47.739540+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:47.739540+0000","last_undegraded":"2024-02-19T14:53:47.739540+0000","last_fullsized":"2024-02-19T14:53:47.739540+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.725966+0000","last_change":"2024-02-19T14:53:47.725966+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.725966+0000","last_undegraded":"2024-02-19T14:53:47.725966+0000","last_fullsized":"2024-02-19T14:53:47.725966+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723966+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:47.723966+0000","last_peered":"2024-02-19T14:53:47.723966+0000","last_clean":"2024-02-19T14:53:47.723966+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:47.723966+0000","last_undegraded":"2024-02-19T14:53:47.723966+0000","last_fullsized":"2024-02-19T14:53:47.723966+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:59.053283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729507+0000","last_change":"2024-02-19T14:53:47.729507+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729507+0000","last_undegraded":"2024-02-19T14:53:47.729507+0000","last_fullsized":"2024-02-19T14:53:47.729507+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732727+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:47.732727+0000","last_peered":"2024-02-19T14:53:47.732727+0000","last_clean":"2024-02-19T14:53:47.732727+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:47.732727+0000","last_undegraded":"2024-02-19T14:53:47.732727+0000","last_fullsized":"2024-02-19T14:53:47.732727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:38:18.934930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726592+0000","last_change":"2024-02-19T14:53:47.726592+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726592+0000","last_undegraded":"2024-02-19T14:53:47.726592+0000","last_fullsized":"2024-02-19T14:53:47.726592+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730738+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:47.730738+0000","last_peered":"2024-02-19T14:53:47.730738+0000","last_clean":"2024-02-19T14:53:47.730738+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:47.730738+0000","last_undegraded":"2024-02-19T14:53:47.730738+0000","last_fullsized":"2024-02-19T14:53:47.730738+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734650+0000","last_change":"2024-02-19T14:53:47.734650+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734650+0000","last_undegraded":"2024-02-19T14:53:47.734650+0000","last_fullsized":"2024-02-19T14:53:47.734650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734454+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:47.734454+0000","last_peered":"2024-02-19T14:53:47.734454+0000","last_clean":"2024-02-19T14:53:47.734454+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:47.734454+0000","last_undegraded":"2024-02-19T14:53:47.734454+0000","last_fullsized":"2024-02-19T14:53:47.734454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729138+0000","last_change":"2024-02-19T14:53:47.729138+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729138+0000","last_undegraded":"2024-02-19T14:53:47.729138+0000","last_fullsized":"2024-02-19T14:53:47.729138+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734486+0000","last_change":"2024-02-19T14:53:47.734486+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734486+0000","last_undegraded":"2024-02-19T14:53:47.734486+0000","last_fullsized":"2024-02-19T14:53:47.734486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729945+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:47.729945+0000","last_peered":"2024-02-19T14:53:47.729945+0000","last_clean":"2024-02-19T14:53:47.729945+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:47.729945+0000","last_undegraded":"2024-02-19T14:53:47.729945+0000","last_fullsized":"2024-02-19T14:53:47.729945+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.733413+0000","last_change":"2024-02-19T14:53:47.733413+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.733413+0000","last_undegraded":"2024-02-19T14:53:47.733413+0000","last_fullsized":"2024-02-19T14:53:47.733413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737727+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:47.737727+0000","last_peered":"2024-02-19T14:53:47.737727+0000","last_clean":"2024-02-19T14:53:47.737727+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:47.737727+0000","last_undegraded":"2024-02-19T14:53:47.737727+0000","last_fullsized":"2024-02-19T14:53:47.737727+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729994+0000","last_change":"2024-02-19T14:53:47.729994+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729994+0000","last_undegraded":"2024-02-19T14:53:47.729994+0000","last_fullsized":"2024-02-19T14:53:47.729994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732428+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:47.732428+0000","last_peered":"2024-02-19T14:53:47.732428+0000","last_clean":"2024-02-19T14:53:47.732428+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:47.732428+0000","last_undegraded":"2024-02-19T14:53:47.732428+0000","last_fullsized":"2024-02-19T14:53:47.732428+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728586+0000","last_change":"2024-02-19T14:53:47.728586+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728586+0000","last_undegraded":"2024-02-19T14:53:47.728586+0000","last_fullsized":"2024-02-19T14:53:47.728586+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729908+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:47.729908+0000","last_peered":"2024-02-19T14:53:47.729908+0000","last_clean":"2024-02-19T14:53:47.729908+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:47.729908+0000","last_undegraded":"2024-02-19T14:53:47.729908+0000","last_fullsized":"2024-02-19T14:53:47.729908+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726173+0000","last_change":"2024-02-19T14:53:47.726173+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726173+0000","last_undegraded":"2024-02-19T14:53:47.726173+0000","last_fullsized":"2024-02-19T14:53:47.726173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731441+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:47.731441+0000","last_peered":"2024-02-19T14:53:47.731441+0000","last_clean":"2024-02-19T14:53:47.731441+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:47.731441+0000","last_undegraded":"2024-02-19T14:53:47.731441+0000","last_fullsized":"2024-02-19T14:53:47.731441+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727599+0000","last_change":"2024-02-19T14:53:47.727599+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727599+0000","last_undegraded":"2024-02-19T14:53:47.727599+0000","last_fullsized":"2024-02-19T14:53:47.727599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734161+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:47.734161+0000","last_peered":"2024-02-19T14:53:47.734161+0000","last_clean":"2024-02-19T14:53:47.734161+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:47.734161+0000","last_undegraded":"2024-02-19T14:53:47.734161+0000","last_fullsized":"2024-02-19T14:53:47.734161+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735808+0000","last_change":"2024-02-19T14:53:47.735808+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735808+0000","last_undegraded":"2024-02-19T14:53:47.735808+0000","last_fullsized":"2024-02-19T14:53:47.735808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746858+0000","last_change":"2024-02-19T14:53:47.740946+0000","last_active":"2024-02-19T14:53:47.746858+0000","last_peered":"2024-02-19T14:53:47.746858+0000","last_clean":"2024-02-19T14:53:47.746858+0000","last_became_active":"2024-02-19T14:53:47.740550+0000","last_became_peered":"2024-02-19T14:53:47.740550+0000","last_unstale":"2024-02-19T14:53:47.746858+0000","last_undegraded":"2024-02-19T14:53:47.746858+0000","last_fullsized":"2024-02-19T14:53:47.746858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:09.101397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728386+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:47.728386+0000","last_peered":"2024-02-19T14:53:47.728386+0000","last_clean":"2024-02-19T14:53:47.728386+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:47.728386+0000","last_undegraded":"2024-02-19T14:53:47.728386+0000","last_fullsized":"2024-02-19T14:53:47.728386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":26,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739615+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:47.739615+0000","last_peered":"2024-02-19T14:53:47.739615+0000","last_clean":"2024-02-19T14:53:47.739615+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:47.739615+0000","last_undegraded":"2024-02-19T14:53:47.739615+0000","last_fullsized":"2024-02-19T14:53:47.739615+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727391+0000","last_change":"2024-02-19T14:53:47.727391+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727391+0000","last_undegraded":"2024-02-19T14:53:47.727391+0000","last_fullsized":"2024-02-19T14:53:47.727391+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746433+0000","last_change":"2024-02-19T14:53:47.742941+0000","last_active":"2024-02-19T14:53:47.746433+0000","last_peered":"2024-02-19T14:53:47.746433+0000","last_clean":"2024-02-19T14:53:47.746433+0000","last_became_active":"2024-02-19T14:53:47.742590+0000","last_became_peered":"2024-02-19T14:53:47.742590+0000","last_unstale":"2024-02-19T14:53:47.746433+0000","last_undegraded":"2024-02-19T14:53:47.746433+0000","last_fullsized":"2024-02-19T14:53:47.746433+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:35.779780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729200+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:47.729200+0000","last_peered":"2024-02-19T14:53:47.729200+0000","last_clean":"2024-02-19T14:53:47.729200+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:47.729200+0000","last_undegraded":"2024-02-19T14:53:47.729200+0000","last_fullsized":"2024-02-19T14:53:47.729200+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730410+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:47.730410+0000","last_peered":"2024-02-19T14:53:47.730410+0000","last_clean":"2024-02-19T14:53:47.730410+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:47.730410+0000","last_undegraded":"2024-02-19T14:53:47.730410+0000","last_fullsized":"2024-02-19T14:53:47.730410+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.729346+0000","last_change":"2024-02-19T14:53:47.729346+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.729346+0000","last_undegraded":"2024-02-19T14:53:47.729346+0000","last_fullsized":"2024-02-19T14:53:47.729346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747720+0000","last_change":"2024-02-19T14:53:47.746326+0000","last_active":"2024-02-19T14:53:47.747720+0000","last_peered":"2024-02-19T14:53:47.747720+0000","last_clean":"2024-02-19T14:53:47.747720+0000","last_became_active":"2024-02-19T14:53:47.745753+0000","last_became_peered":"2024-02-19T14:53:47.745753+0000","last_unstale":"2024-02-19T14:53:47.747720+0000","last_undegraded":"2024-02-19T14:53:47.747720+0000","last_fullsized":"2024-02-19T14:53:47.747720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739624+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:47.739624+0000","last_peered":"2024-02-19T14:53:47.739624+0000","last_clean":"2024-02-19T14:53:47.739624+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:47.739624+0000","last_undegraded":"2024-02-19T14:53:47.739624+0000","last_fullsized":"2024-02-19T14:53:47.739624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730034+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:47.730034+0000","last_peered":"2024-02-19T14:53:47.730034+0000","last_clean":"2024-02-19T14:53:47.730034+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:47.730034+0000","last_undegraded":"2024-02-19T14:53:47.730034+0000","last_fullsized":"2024-02-19T14:53:47.730034+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726858+0000","last_change":"2024-02-19T14:53:47.726858+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726858+0000","last_undegraded":"2024-02-19T14:53:47.726858+0000","last_fullsized":"2024-02-19T14:53:47.726858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747935+0000","last_change":"2024-02-19T14:53:47.740910+0000","last_active":"2024-02-19T14:53:47.747935+0000","last_peered":"2024-02-19T14:53:47.747935+0000","last_clean":"2024-02-19T14:53:47.747935+0000","last_became_active":"2024-02-19T14:53:47.740626+0000","last_became_peered":"2024-02-19T14:53:47.740626+0000","last_unstale":"2024-02-19T14:53:47.747935+0000","last_undegraded":"2024-02-19T14:53:47.747935+0000","last_fullsized":"2024-02-19T14:53:47.747935+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:20.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723809+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:47.723809+0000","last_peered":"2024-02-19T14:53:47.723809+0000","last_clean":"2024-02-19T14:53:47.723809+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:47.723809+0000","last_undegraded":"2024-02-19T14:53:47.723809+0000","last_fullsized":"2024-02-19T14:53:47.723809+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:36.026355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733496+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:47.733496+0000","last_peered":"2024-02-19T14:53:47.733496+0000","last_clean":"2024-02-19T14:53:47.733496+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:47.733496+0000","last_undegraded":"2024-02-19T14:53:47.733496+0000","last_fullsized":"2024-02-19T14:53:47.733496+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732000+0000","last_change":"2024-02-19T14:53:47.732000+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732000+0000","last_undegraded":"2024-02-19T14:53:47.732000+0000","last_fullsized":"2024-02-19T14:53:47.732000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747004+0000","last_change":"2024-02-19T14:53:47.745151+0000","last_active":"2024-02-19T14:53:47.747004+0000","last_peered":"2024-02-19T14:53:47.747004+0000","last_clean":"2024-02-19T14:53:47.747004+0000","last_became_active":"2024-02-19T14:53:47.744887+0000","last_became_peered":"2024-02-19T14:53:47.744887+0000","last_unstale":"2024-02-19T14:53:47.747004+0000","last_undegraded":"2024-02-19T14:53:47.747004+0000","last_fullsized":"2024-02-19T14:53:47.747004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732432+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:47.732432+0000","last_peered":"2024-02-19T14:53:47.732432+0000","last_clean":"2024-02-19T14:53:47.732432+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:47.732432+0000","last_undegraded":"2024-02-19T14:53:47.732432+0000","last_fullsized":"2024-02-19T14:53:47.732432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732599+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:47.732599+0000","last_peered":"2024-02-19T14:53:47.732599+0000","last_clean":"2024-02-19T14:53:47.732599+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:47.732599+0000","last_undegraded":"2024-02-19T14:53:47.732599+0000","last_fullsized":"2024-02-19T14:53:47.732599+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.736522+0000","last_change":"2024-02-19T14:53:47.736522+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.736522+0000","last_undegraded":"2024-02-19T14:53:47.736522+0000","last_fullsized":"2024-02-19T14:53:47.736522+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746094+0000","last_change":"2024-02-19T14:53:47.743381+0000","last_active":"2024-02-19T14:53:47.746094+0000","last_peered":"2024-02-19T14:53:47.746094+0000","last_clean":"2024-02-19T14:53:47.746094+0000","last_became_active":"2024-02-19T14:53:47.743135+0000","last_became_peered":"2024-02-19T14:53:47.743135+0000","last_unstale":"2024-02-19T14:53:47.746094+0000","last_undegraded":"2024-02-19T14:53:47.746094+0000","last_fullsized":"2024-02-19T14:53:47.746094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:30.540219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730104+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:47.730104+0000","last_peered":"2024-02-19T14:53:47.730104+0000","last_clean":"2024-02-19T14:53:47.730104+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:47.730104+0000","last_undegraded":"2024-02-19T14:53:47.730104+0000","last_fullsized":"2024-02-19T14:53:47.730104+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728795+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:47.728795+0000","last_peered":"2024-02-19T14:53:47.728795+0000","last_clean":"2024-02-19T14:53:47.728795+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:47.728795+0000","last_undegraded":"2024-02-19T14:53:47.728795+0000","last_fullsized":"2024-02-19T14:53:47.728795+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728755+0000","last_change":"2024-02-19T14:53:47.728755+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728755+0000","last_undegraded":"2024-02-19T14:53:47.728755+0000","last_fullsized":"2024-02-19T14:53:47.728755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746985+0000","last_change":"2024-02-19T14:53:47.744368+0000","last_active":"2024-02-19T14:53:47.746985+0000","last_peered":"2024-02-19T14:53:47.746985+0000","last_clean":"2024-02-19T14:53:47.746985+0000","last_became_active":"2024-02-19T14:53:47.743379+0000","last_became_peered":"2024-02-19T14:53:47.743379+0000","last_unstale":"2024-02-19T14:53:47.746985+0000","last_undegraded":"2024-02-19T14:53:47.746985+0000","last_fullsized":"2024-02-19T14:53:47.746985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724152+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:47.724152+0000","last_peered":"2024-02-19T14:53:47.724152+0000","last_clean":"2024-02-19T14:53:47.724152+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:47.724152+0000","last_undegraded":"2024-02-19T14:53:47.724152+0000","last_fullsized":"2024-02-19T14:53:47.724152+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:41.307928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737668+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:47.737668+0000","last_peered":"2024-02-19T14:53:47.737668+0000","last_clean":"2024-02-19T14:53:47.737668+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:47.737668+0000","last_undegraded":"2024-02-19T14:53:47.737668+0000","last_fullsized":"2024-02-19T14:53:47.737668+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726342+0000","last_change":"2024-02-19T14:53:47.726342+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726342+0000","last_undegraded":"2024-02-19T14:53:47.726342+0000","last_fullsized":"2024-02-19T14:53:47.726342+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747509+0000","last_change":"2024-02-19T14:53:47.744422+0000","last_active":"2024-02-19T14:53:47.747509+0000","last_peered":"2024-02-19T14:53:47.747509+0000","last_clean":"2024-02-19T14:53:47.747509+0000","last_became_active":"2024-02-19T14:53:47.744122+0000","last_became_peered":"2024-02-19T14:53:47.744122+0000","last_unstale":"2024-02-19T14:53:47.747509+0000","last_undegraded":"2024-02-19T14:53:47.747509+0000","last_fullsized":"2024-02-19T14:53:47.747509+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:50.594221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730639+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:47.730639+0000","last_peered":"2024-02-19T14:53:47.730639+0000","last_clean":"2024-02-19T14:53:47.730639+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:47.730639+0000","last_undegraded":"2024-02-19T14:53:47.730639+0000","last_fullsized":"2024-02-19T14:53:47.730639+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730829+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:47.730829+0000","last_peered":"2024-02-19T14:53:47.730829+0000","last_clean":"2024-02-19T14:53:47.730829+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:47.730829+0000","last_undegraded":"2024-02-19T14:53:47.730829+0000","last_fullsized":"2024-02-19T14:53:47.730829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728855+0000","last_change":"2024-02-19T14:53:47.728855+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728855+0000","last_undegraded":"2024-02-19T14:53:47.728855+0000","last_fullsized":"2024-02-19T14:53:47.728855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746360+0000","last_change":"2024-02-19T14:53:47.745353+0000","last_active":"2024-02-19T14:53:47.746360+0000","last_peered":"2024-02-19T14:53:47.746360+0000","last_clean":"2024-02-19T14:53:47.746360+0000","last_became_active":"2024-02-19T14:53:47.745008+0000","last_became_peered":"2024-02-19T14:53:47.745008+0000","last_unstale":"2024-02-19T14:53:47.746360+0000","last_undegraded":"2024-02-19T14:53:47.746360+0000","last_fullsized":"2024-02-19T14:53:47.746360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:29.660730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729669+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:47.729669+0000","last_peered":"2024-02-19T14:53:47.729669+0000","last_clean":"2024-02-19T14:53:47.729669+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:47.729669+0000","last_undegraded":"2024-02-19T14:53:47.729669+0000","last_fullsized":"2024-02-19T14:53:47.729669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730746+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:47.730746+0000","last_peered":"2024-02-19T14:53:47.730746+0000","last_clean":"2024-02-19T14:53:47.730746+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:47.730746+0000","last_undegraded":"2024-02-19T14:53:47.730746+0000","last_fullsized":"2024-02-19T14:53:47.730746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726359+0000","last_change":"2024-02-19T14:53:47.726359+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726359+0000","last_undegraded":"2024-02-19T14:53:47.726359+0000","last_fullsized":"2024-02-19T14:53:47.726359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746332+0000","last_change":"2024-02-19T14:53:47.742114+0000","last_active":"2024-02-19T14:53:47.746332+0000","last_peered":"2024-02-19T14:53:47.746332+0000","last_clean":"2024-02-19T14:53:47.746332+0000","last_became_active":"2024-02-19T14:53:47.741840+0000","last_became_peered":"2024-02-19T14:53:47.741840+0000","last_unstale":"2024-02-19T14:53:47.746332+0000","last_undegraded":"2024-02-19T14:53:47.746332+0000","last_fullsized":"2024-02-19T14:53:47.746332+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:20.795346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727709+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:47.727709+0000","last_peered":"2024-02-19T14:53:47.727709+0000","last_clean":"2024-02-19T14:53:47.727709+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:47.727709+0000","last_undegraded":"2024-02-19T14:53:47.727709+0000","last_fullsized":"2024-02-19T14:53:47.727709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.736179+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:47.736179+0000","last_peered":"2024-02-19T14:53:47.736179+0000","last_clean":"2024-02-19T14:53:47.736179+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:47.736179+0000","last_undegraded":"2024-02-19T14:53:47.736179+0000","last_fullsized":"2024-02-19T14:53:47.736179+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731814+0000","last_change":"2024-02-19T14:53:47.731814+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731814+0000","last_undegraded":"2024-02-19T14:53:47.731814+0000","last_fullsized":"2024-02-19T14:53:47.731814+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737453+0000","last_change":"2024-02-19T14:53:47.734753+0000","last_active":"2024-02-19T14:53:47.737453+0000","last_peered":"2024-02-19T14:53:47.737453+0000","last_clean":"2024-02-19T14:53:47.737453+0000","last_became_active":"2024-02-19T14:53:47.734515+0000","last_became_peered":"2024-02-19T14:53:47.734515+0000","last_unstale":"2024-02-19T14:53:47.737453+0000","last_undegraded":"2024-02-19T14:53:47.737453+0000","last_fullsized":"2024-02-19T14:53:47.737453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:02:56.724991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730223+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:47.730223+0000","last_peered":"2024-02-19T14:53:47.730223+0000","last_clean":"2024-02-19T14:53:47.730223+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:47.730223+0000","last_undegraded":"2024-02-19T14:53:47.730223+0000","last_fullsized":"2024-02-19T14:53:47.730223+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":25,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731281+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:47.731281+0000","last_peered":"2024-02-19T14:53:47.731281+0000","last_clean":"2024-02-19T14:53:47.731281+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:47.731281+0000","last_undegraded":"2024-02-19T14:53:47.731281+0000","last_fullsized":"2024-02-19T14:53:47.731281+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727760+0000","last_change":"2024-02-19T14:53:47.727760+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727760+0000","last_undegraded":"2024-02-19T14:53:47.727760+0000","last_fullsized":"2024-02-19T14:53:47.727760+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737793+0000","last_change":"2024-02-19T14:53:47.735759+0000","last_active":"2024-02-19T14:53:47.737793+0000","last_peered":"2024-02-19T14:53:47.737793+0000","last_clean":"2024-02-19T14:53:47.737793+0000","last_became_active":"2024-02-19T14:53:47.735503+0000","last_became_peered":"2024-02-19T14:53:47.735503+0000","last_unstale":"2024-02-19T14:53:47.737793+0000","last_undegraded":"2024-02-19T14:53:47.737793+0000","last_fullsized":"2024-02-19T14:53:47.737793+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:35.189687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731195+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:47.731195+0000","last_peered":"2024-02-19T14:53:47.731195+0000","last_clean":"2024-02-19T14:53:47.731195+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:47.731195+0000","last_undegraded":"2024-02-19T14:53:47.731195+0000","last_fullsized":"2024-02-19T14:53:47.731195+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":28,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724746+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:47.724746+0000","last_peered":"2024-02-19T14:53:47.724746+0000","last_clean":"2024-02-19T14:53:47.724746+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:47.724746+0000","last_undegraded":"2024-02-19T14:53:47.724746+0000","last_fullsized":"2024-02-19T14:53:47.724746+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731071+0000","last_change":"2024-02-19T14:53:47.731071+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731071+0000","last_undegraded":"2024-02-19T14:53:47.731071+0000","last_fullsized":"2024-02-19T14:53:47.731071+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746268+0000","last_change":"2024-02-19T14:53:47.742801+0000","last_active":"2024-02-19T14:53:47.746268+0000","last_peered":"2024-02-19T14:53:47.746268+0000","last_clean":"2024-02-19T14:53:47.746268+0000","last_became_active":"2024-02-19T14:53:47.742534+0000","last_became_peered":"2024-02-19T14:53:47.742534+0000","last_unstale":"2024-02-19T14:53:47.746268+0000","last_undegraded":"2024-02-19T14:53:47.746268+0000","last_fullsized":"2024-02-19T14:53:47.746268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:59.453875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733273+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:47.733273+0000","last_peered":"2024-02-19T14:53:47.733273+0000","last_clean":"2024-02-19T14:53:47.733273+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:47.733273+0000","last_undegraded":"2024-02-19T14:53:47.733273+0000","last_fullsized":"2024-02-19T14:53:47.733273+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732699+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:47.732699+0000","last_peered":"2024-02-19T14:53:47.732699+0000","last_clean":"2024-02-19T14:53:47.732699+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:47.732699+0000","last_undegraded":"2024-02-19T14:53:47.732699+0000","last_fullsized":"2024-02-19T14:53:47.732699+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.726717+0000","last_change":"2024-02-19T14:53:47.726717+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.726717+0000","last_undegraded":"2024-02-19T14:53:47.726717+0000","last_fullsized":"2024-02-19T14:53:47.726717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748938+0000","last_change":"2024-02-19T14:53:47.744212+0000","last_active":"2024-02-19T14:53:47.748938+0000","last_peered":"2024-02-19T14:53:47.748938+0000","last_clean":"2024-02-19T14:53:47.748938+0000","last_became_active":"2024-02-19T14:53:47.743916+0000","last_became_peered":"2024-02-19T14:53:47.743916+0000","last_unstale":"2024-02-19T14:53:47.748938+0000","last_undegraded":"2024-02-19T14:53:47.748938+0000","last_fullsized":"2024-02-19T14:53:47.748938+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732440+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:47.732440+0000","last_peered":"2024-02-19T14:53:47.732440+0000","last_clean":"2024-02-19T14:53:47.732440+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:47.732440+0000","last_undegraded":"2024-02-19T14:53:47.732440+0000","last_fullsized":"2024-02-19T14:53:47.732440+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739733+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:47.739733+0000","last_peered":"2024-02-19T14:53:47.739733+0000","last_clean":"2024-02-19T14:53:47.739733+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:47.739733+0000","last_undegraded":"2024-02-19T14:53:47.739733+0000","last_fullsized":"2024-02-19T14:53:47.739733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731538+0000","last_change":"2024-02-19T14:53:47.731538+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731538+0000","last_undegraded":"2024-02-19T14:53:47.731538+0000","last_fullsized":"2024-02-19T14:53:47.731538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.744120+0000","last_change":"2024-02-19T14:53:47.741970+0000","last_active":"2024-02-19T14:53:47.744120+0000","last_peered":"2024-02-19T14:53:47.744120+0000","last_clean":"2024-02-19T14:53:47.744120+0000","last_became_active":"2024-02-19T14:53:47.741707+0000","last_became_peered":"2024-02-19T14:53:47.741707+0000","last_unstale":"2024-02-19T14:53:47.744120+0000","last_undegraded":"2024-02-19T14:53:47.744120+0000","last_fullsized":"2024-02-19T14:53:47.744120+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:26.047456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724087+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:47.724087+0000","last_peered":"2024-02-19T14:53:47.724087+0000","last_clean":"2024-02-19T14:53:47.724087+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:47.724087+0000","last_undegraded":"2024-02-19T14:53:47.724087+0000","last_fullsized":"2024-02-19T14:53:47.724087+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729186+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:47.729186+0000","last_peered":"2024-02-19T14:53:47.729186+0000","last_clean":"2024-02-19T14:53:47.729186+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:47.729186+0000","last_undegraded":"2024-02-19T14:53:47.729186+0000","last_fullsized":"2024-02-19T14:53:47.729186+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.734244+0000","last_change":"2024-02-19T14:53:47.734244+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.734244+0000","last_undegraded":"2024-02-19T14:53:47.734244+0000","last_fullsized":"2024-02-19T14:53:47.734244+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743671+0000","last_change":"2024-02-19T14:53:47.742140+0000","last_active":"2024-02-19T14:53:47.743671+0000","last_peered":"2024-02-19T14:53:47.743671+0000","last_clean":"2024-02-19T14:53:47.743671+0000","last_became_active":"2024-02-19T14:53:47.741838+0000","last_became_peered":"2024-02-19T14:53:47.741838+0000","last_unstale":"2024-02-19T14:53:47.743671+0000","last_undegraded":"2024-02-19T14:53:47.743671+0000","last_fullsized":"2024-02-19T14:53:47.743671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:55.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737818+0000","last_change":"2024-02-19T14:53:41.706063+0000","last_active":"2024-02-19T14:53:47.737818+0000","last_peered":"2024-02-19T14:53:47.737818+0000","last_clean":"2024-02-19T14:53:47.737818+0000","last_became_active":"2024-02-19T14:53:41.705766+0000","last_became_peered":"2024-02-19T14:53:41.705766+0000","last_unstale":"2024-02-19T14:53:47.737818+0000","last_undegraded":"2024-02-19T14:53:47.737818+0000","last_fullsized":"2024-02-19T14:53:47.737818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":30,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723730+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:47.723730+0000","last_peered":"2024-02-19T14:53:47.723730+0000","last_clean":"2024-02-19T14:53:47.723730+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:47.723730+0000","last_undegraded":"2024-02-19T14:53:47.723730+0000","last_fullsized":"2024-02-19T14:53:47.723730+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727027+0000","last_change":"2024-02-19T14:53:47.727027+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727027+0000","last_undegraded":"2024-02-19T14:53:47.727027+0000","last_fullsized":"2024-02-19T14:53:47.727027+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743702+0000","last_change":"2024-02-19T14:53:47.742196+0000","last_active":"2024-02-19T14:53:47.743702+0000","last_peered":"2024-02-19T14:53:47.743702+0000","last_clean":"2024-02-19T14:53:47.743702+0000","last_became_active":"2024-02-19T14:53:47.741852+0000","last_became_peered":"2024-02-19T14:53:47.741852+0000","last_unstale":"2024-02-19T14:53:47.743702+0000","last_undegraded":"2024-02-19T14:53:47.743702+0000","last_fullsized":"2024-02-19T14:53:47.743702+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:31.942230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724625+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:47.724625+0000","last_peered":"2024-02-19T14:53:47.724625+0000","last_clean":"2024-02-19T14:53:47.724625+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:47.724625+0000","last_undegraded":"2024-02-19T14:53:47.724625+0000","last_fullsized":"2024-02-19T14:53:47.724625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734529+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:47.734529+0000","last_peered":"2024-02-19T14:53:47.734529+0000","last_clean":"2024-02-19T14:53:47.734529+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:47.734529+0000","last_undegraded":"2024-02-19T14:53:47.734529+0000","last_fullsized":"2024-02-19T14:53:47.734529+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730482+0000","last_change":"2024-02-19T14:53:47.730482+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730482+0000","last_undegraded":"2024-02-19T14:53:47.730482+0000","last_fullsized":"2024-02-19T14:53:47.730482+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.749226+0000","last_change":"2024-02-19T14:53:47.745519+0000","last_active":"2024-02-19T14:53:47.749226+0000","last_peered":"2024-02-19T14:53:47.749226+0000","last_clean":"2024-02-19T14:53:47.749226+0000","last_became_active":"2024-02-19T14:53:47.745263+0000","last_became_peered":"2024-02-19T14:53:47.745263+0000","last_unstale":"2024-02-19T14:53:47.749226+0000","last_undegraded":"2024-02-19T14:53:47.749226+0000","last_fullsized":"2024-02-19T14:53:47.749226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:51.861192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728027+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:47.728027+0000","last_peered":"2024-02-19T14:53:47.728027+0000","last_clean":"2024-02-19T14:53:47.728027+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:47.728027+0000","last_undegraded":"2024-02-19T14:53:47.728027+0000","last_fullsized":"2024-02-19T14:53:47.728027+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733299+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:47.733299+0000","last_peered":"2024-02-19T14:53:47.733299+0000","last_clean":"2024-02-19T14:53:47.733299+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:47.733299+0000","last_undegraded":"2024-02-19T14:53:47.733299+0000","last_fullsized":"2024-02-19T14:53:47.733299+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728440+0000","last_change":"2024-02-19T14:53:47.728440+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728440+0000","last_undegraded":"2024-02-19T14:53:47.728440+0000","last_fullsized":"2024-02-19T14:53:47.728440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.741263+0000","last_change":"2024-02-19T14:53:47.736640+0000","last_active":"2024-02-19T14:53:47.741263+0000","last_peered":"2024-02-19T14:53:47.741263+0000","last_clean":"2024-02-19T14:53:47.741263+0000","last_became_active":"2024-02-19T14:53:47.735150+0000","last_became_peered":"2024-02-19T14:53:47.735150+0000","last_unstale":"2024-02-19T14:53:47.741263+0000","last_undegraded":"2024-02-19T14:53:47.741263+0000","last_fullsized":"2024-02-19T14:53:47.741263+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:02.655336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.728667+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:47.728667+0000","last_peered":"2024-02-19T14:53:47.728667+0000","last_clean":"2024-02-19T14:53:47.728667+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:47.728667+0000","last_undegraded":"2024-02-19T14:53:47.728667+0000","last_fullsized":"2024-02-19T14:53:47.728667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737687+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:47.737687+0000","last_peered":"2024-02-19T14:53:47.737687+0000","last_clean":"2024-02-19T14:53:47.737687+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:47.737687+0000","last_undegraded":"2024-02-19T14:53:47.737687+0000","last_fullsized":"2024-02-19T14:53:47.737687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.730447+0000","last_change":"2024-02-19T14:53:47.730447+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.730447+0000","last_undegraded":"2024-02-19T14:53:47.730447+0000","last_fullsized":"2024-02-19T14:53:47.730447+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734187+0000","last_change":"2024-02-19T14:53:47.730313+0000","last_active":"2024-02-19T14:53:47.734187+0000","last_peered":"2024-02-19T14:53:47.734187+0000","last_clean":"2024-02-19T14:53:47.734187+0000","last_became_active":"2024-02-19T14:53:47.730053+0000","last_became_peered":"2024-02-19T14:53:47.730053+0000","last_unstale":"2024-02-19T14:53:47.734187+0000","last_undegraded":"2024-02-19T14:53:47.734187+0000","last_fullsized":"2024-02-19T14:53:47.734187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729153+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:47.729153+0000","last_peered":"2024-02-19T14:53:47.729153+0000","last_clean":"2024-02-19T14:53:47.729153+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:47.729153+0000","last_undegraded":"2024-02-19T14:53:47.729153+0000","last_fullsized":"2024-02-19T14:53:47.729153+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737838+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:47.737838+0000","last_peered":"2024-02-19T14:53:47.737838+0000","last_clean":"2024-02-19T14:53:47.737838+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:47.737838+0000","last_undegraded":"2024-02-19T14:53:47.737838+0000","last_fullsized":"2024-02-19T14:53:47.737838+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728393+0000","last_change":"2024-02-19T14:53:47.728393+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728393+0000","last_undegraded":"2024-02-19T14:53:47.728393+0000","last_fullsized":"2024-02-19T14:53:47.728393+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746290+0000","last_change":"2024-02-19T14:53:47.742841+0000","last_active":"2024-02-19T14:53:47.746290+0000","last_peered":"2024-02-19T14:53:47.746290+0000","last_clean":"2024-02-19T14:53:47.746290+0000","last_became_active":"2024-02-19T14:53:47.742581+0000","last_became_peered":"2024-02-19T14:53:47.742581+0000","last_unstale":"2024-02-19T14:53:47.746290+0000","last_undegraded":"2024-02-19T14:53:47.746290+0000","last_fullsized":"2024-02-19T14:53:47.746290+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:57.701790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733388+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:47.733388+0000","last_peered":"2024-02-19T14:53:47.733388+0000","last_clean":"2024-02-19T14:53:47.733388+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:47.733388+0000","last_undegraded":"2024-02-19T14:53:47.733388+0000","last_fullsized":"2024-02-19T14:53:47.733388+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729246+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:47.729246+0000","last_peered":"2024-02-19T14:53:47.729246+0000","last_clean":"2024-02-19T14:53:47.729246+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:47.729246+0000","last_undegraded":"2024-02-19T14:53:47.729246+0000","last_fullsized":"2024-02-19T14:53:47.729246+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731908+0000","last_change":"2024-02-19T14:53:47.731908+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731908+0000","last_undegraded":"2024-02-19T14:53:47.731908+0000","last_fullsized":"2024-02-19T14:53:47.731908+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746114+0000","last_change":"2024-02-19T14:53:47.743491+0000","last_active":"2024-02-19T14:53:47.746114+0000","last_peered":"2024-02-19T14:53:47.746114+0000","last_clean":"2024-02-19T14:53:47.746114+0000","last_became_active":"2024-02-19T14:53:47.743209+0000","last_became_peered":"2024-02-19T14:53:47.743209+0000","last_unstale":"2024-02-19T14:53:47.746114+0000","last_undegraded":"2024-02-19T14:53:47.746114+0000","last_fullsized":"2024-02-19T14:53:47.746114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:51.151311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737504+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:47.737504+0000","last_peered":"2024-02-19T14:53:47.737504+0000","last_clean":"2024-02-19T14:53:47.737504+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:47.737504+0000","last_undegraded":"2024-02-19T14:53:47.737504+0000","last_fullsized":"2024-02-19T14:53:47.737504+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724055+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:47.724055+0000","last_peered":"2024-02-19T14:53:47.724055+0000","last_clean":"2024-02-19T14:53:47.724055+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:47.724055+0000","last_undegraded":"2024-02-19T14:53:47.724055+0000","last_fullsized":"2024-02-19T14:53:47.724055+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728116+0000","last_change":"2024-02-19T14:53:47.728116+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728116+0000","last_undegraded":"2024-02-19T14:53:47.728116+0000","last_fullsized":"2024-02-19T14:53:47.728116+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747934+0000","last_change":"2024-02-19T14:53:47.740831+0000","last_active":"2024-02-19T14:53:47.747934+0000","last_peered":"2024-02-19T14:53:47.747934+0000","last_clean":"2024-02-19T14:53:47.747934+0000","last_became_active":"2024-02-19T14:53:47.740540+0000","last_became_peered":"2024-02-19T14:53:47.740540+0000","last_unstale":"2024-02-19T14:53:47.747934+0000","last_undegraded":"2024-02-19T14:53:47.747934+0000","last_fullsized":"2024-02-19T14:53:47.747934+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:50.874837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733979+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:47.733979+0000","last_peered":"2024-02-19T14:53:47.733979+0000","last_clean":"2024-02-19T14:53:47.733979+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:47.733979+0000","last_undegraded":"2024-02-19T14:53:47.733979+0000","last_fullsized":"2024-02-19T14:53:47.733979+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731528+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:47.731528+0000","last_peered":"2024-02-19T14:53:47.731528+0000","last_clean":"2024-02-19T14:53:47.731528+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:47.731528+0000","last_undegraded":"2024-02-19T14:53:47.731528+0000","last_fullsized":"2024-02-19T14:53:47.731528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731324+0000","last_change":"2024-02-19T14:53:47.731324+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731324+0000","last_undegraded":"2024-02-19T14:53:47.731324+0000","last_fullsized":"2024-02-19T14:53:47.731324+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.740896+0000","last_change":"2024-02-19T14:53:47.736026+0000","last_active":"2024-02-19T14:53:47.740896+0000","last_peered":"2024-02-19T14:53:47.740896+0000","last_clean":"2024-02-19T14:53:47.740896+0000","last_became_active":"2024-02-19T14:53:47.735680+0000","last_became_peered":"2024-02-19T14:53:47.735680+0000","last_unstale":"2024-02-19T14:53:47.740896+0000","last_undegraded":"2024-02-19T14:53:47.740896+0000","last_fullsized":"2024-02-19T14:53:47.740896+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730749+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:47.730749+0000","last_peered":"2024-02-19T14:53:47.730749+0000","last_clean":"2024-02-19T14:53:47.730749+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:47.730749+0000","last_undegraded":"2024-02-19T14:53:47.730749+0000","last_fullsized":"2024-02-19T14:53:47.730749+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732455+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:47.732455+0000","last_peered":"2024-02-19T14:53:47.732455+0000","last_clean":"2024-02-19T14:53:47.732455+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:47.732455+0000","last_undegraded":"2024-02-19T14:53:47.732455+0000","last_fullsized":"2024-02-19T14:53:47.732455+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.731805+0000","last_change":"2024-02-19T14:53:47.731805+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.731805+0000","last_undegraded":"2024-02-19T14:53:47.731805+0000","last_fullsized":"2024-02-19T14:53:47.731805+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743744+0000","last_change":"2024-02-19T14:53:47.737977+0000","last_active":"2024-02-19T14:53:47.743744+0000","last_peered":"2024-02-19T14:53:47.743744+0000","last_clean":"2024-02-19T14:53:47.743744+0000","last_became_active":"2024-02-19T14:53:47.737709+0000","last_became_peered":"2024-02-19T14:53:47.737709+0000","last_unstale":"2024-02-19T14:53:47.743744+0000","last_undegraded":"2024-02-19T14:53:47.743744+0000","last_fullsized":"2024-02-19T14:53:47.743744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:00.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734277+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:47.734277+0000","last_peered":"2024-02-19T14:53:47.734277+0000","last_clean":"2024-02-19T14:53:47.734277+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:47.734277+0000","last_undegraded":"2024-02-19T14:53:47.734277+0000","last_fullsized":"2024-02-19T14:53:47.734277+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.739764+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:47.739764+0000","last_peered":"2024-02-19T14:53:47.739764+0000","last_clean":"2024-02-19T14:53:47.739764+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:47.739764+0000","last_undegraded":"2024-02-19T14:53:47.739764+0000","last_fullsized":"2024-02-19T14:53:47.739764+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728207+0000","last_change":"2024-02-19T14:53:47.728207+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728207+0000","last_undegraded":"2024-02-19T14:53:47.728207+0000","last_fullsized":"2024-02-19T14:53:47.728207+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.748964+0000","last_change":"2024-02-19T14:53:47.745205+0000","last_active":"2024-02-19T14:53:47.748964+0000","last_peered":"2024-02-19T14:53:47.748964+0000","last_clean":"2024-02-19T14:53:47.748964+0000","last_became_active":"2024-02-19T14:53:47.744958+0000","last_became_peered":"2024-02-19T14:53:47.744958+0000","last_unstale":"2024-02-19T14:53:47.748964+0000","last_undegraded":"2024-02-19T14:53:47.748964+0000","last_fullsized":"2024-02-19T14:53:47.748964+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729468+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:47.729468+0000","last_peered":"2024-02-19T14:53:47.729468+0000","last_clean":"2024-02-19T14:53:47.729468+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:47.729468+0000","last_undegraded":"2024-02-19T14:53:47.729468+0000","last_fullsized":"2024-02-19T14:53:47.729468+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731445+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:47.731445+0000","last_peered":"2024-02-19T14:53:47.731445+0000","last_clean":"2024-02-19T14:53:47.731445+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:47.731445+0000","last_undegraded":"2024-02-19T14:53:47.731445+0000","last_fullsized":"2024-02-19T14:53:47.731445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.727406+0000","last_change":"2024-02-19T14:53:47.727406+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.727406+0000","last_undegraded":"2024-02-19T14:53:47.727406+0000","last_fullsized":"2024-02-19T14:53:47.727406+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747521+0000","last_change":"2024-02-19T14:53:47.740210+0000","last_active":"2024-02-19T14:53:47.747521+0000","last_peered":"2024-02-19T14:53:47.747521+0000","last_clean":"2024-02-19T14:53:47.747521+0000","last_became_active":"2024-02-19T14:53:47.739913+0000","last_became_peered":"2024-02-19T14:53:47.739913+0000","last_unstale":"2024-02-19T14:53:47.747521+0000","last_undegraded":"2024-02-19T14:53:47.747521+0000","last_fullsized":"2024-02-19T14:53:47.747521+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:43.885177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723813+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:47.723813+0000","last_peered":"2024-02-19T14:53:47.723813+0000","last_clean":"2024-02-19T14:53:47.723813+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:47.723813+0000","last_undegraded":"2024-02-19T14:53:47.723813+0000","last_fullsized":"2024-02-19T14:53:47.723813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:10.864244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.732089+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:47.732089+0000","last_peered":"2024-02-19T14:53:47.732089+0000","last_clean":"2024-02-19T14:53:47.732089+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:47.732089+0000","last_undegraded":"2024-02-19T14:53:47.732089+0000","last_fullsized":"2024-02-19T14:53:47.732089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.735192+0000","last_change":"2024-02-19T14:53:47.735192+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.735192+0000","last_undegraded":"2024-02-19T14:53:47.735192+0000","last_fullsized":"2024-02-19T14:53:47.735192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.745944+0000","last_change":"2024-02-19T14:53:47.738871+0000","last_active":"2024-02-19T14:53:47.745944+0000","last_peered":"2024-02-19T14:53:47.745944+0000","last_clean":"2024-02-19T14:53:47.745944+0000","last_became_active":"2024-02-19T14:53:47.738585+0000","last_became_peered":"2024-02-19T14:53:47.738585+0000","last_unstale":"2024-02-19T14:53:47.745944+0000","last_undegraded":"2024-02-19T14:53:47.745944+0000","last_fullsized":"2024-02-19T14:53:47.745944+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:02.790030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.731936+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:47.731936+0000","last_peered":"2024-02-19T14:53:47.731936+0000","last_clean":"2024-02-19T14:53:47.731936+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:47.731936+0000","last_undegraded":"2024-02-19T14:53:47.731936+0000","last_fullsized":"2024-02-19T14:53:47.731936+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.723997+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:47.723997+0000","last_peered":"2024-02-19T14:53:47.723997+0000","last_clean":"2024-02-19T14:53:47.723997+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:47.723997+0000","last_undegraded":"2024-02-19T14:53:47.723997+0000","last_fullsized":"2024-02-19T14:53:47.723997+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732414+0000","last_change":"2024-02-19T14:53:47.732414+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732414+0000","last_undegraded":"2024-02-19T14:53:47.732414+0000","last_fullsized":"2024-02-19T14:53:47.732414+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.743039+0000","last_change":"2024-02-19T14:53:47.738834+0000","last_active":"2024-02-19T14:53:47.743039+0000","last_peered":"2024-02-19T14:53:47.743039+0000","last_clean":"2024-02-19T14:53:47.743039+0000","last_became_active":"2024-02-19T14:53:47.738567+0000","last_became_peered":"2024-02-19T14:53:47.738567+0000","last_unstale":"2024-02-19T14:53:47.743039+0000","last_undegraded":"2024-02-19T14:53:47.743039+0000","last_fullsized":"2024-02-19T14:53:47.743039+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:50.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.727552+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:47.727552+0000","last_peered":"2024-02-19T14:53:47.727552+0000","last_clean":"2024-02-19T14:53:47.727552+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:47.727552+0000","last_undegraded":"2024-02-19T14:53:47.727552+0000","last_fullsized":"2024-02-19T14:53:47.727552+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724124+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:47.724124+0000","last_peered":"2024-02-19T14:53:47.724124+0000","last_clean":"2024-02-19T14:53:47.724124+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:47.724124+0000","last_undegraded":"2024-02-19T14:53:47.724124+0000","last_fullsized":"2024-02-19T14:53:47.724124+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728013+0000","last_change":"2024-02-19T14:53:47.728013+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728013+0000","last_undegraded":"2024-02-19T14:53:47.728013+0000","last_fullsized":"2024-02-19T14:53:47.728013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.747961+0000","last_change":"2024-02-19T14:53:47.744007+0000","last_active":"2024-02-19T14:53:47.747961+0000","last_peered":"2024-02-19T14:53:47.747961+0000","last_clean":"2024-02-19T14:53:47.747961+0000","last_became_active":"2024-02-19T14:53:47.743726+0000","last_became_peered":"2024-02-19T14:53:47.743726+0000","last_unstale":"2024-02-19T14:53:47.747961+0000","last_undegraded":"2024-02-19T14:53:47.747961+0000","last_fullsized":"2024-02-19T14:53:47.747961+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:26.617672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.733834+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:47.733834+0000","last_peered":"2024-02-19T14:53:47.733834+0000","last_clean":"2024-02-19T14:53:47.733834+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:47.733834+0000","last_undegraded":"2024-02-19T14:53:47.733834+0000","last_fullsized":"2024-02-19T14:53:47.733834+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.730733+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:47.730733+0000","last_peered":"2024-02-19T14:53:47.730733+0000","last_clean":"2024-02-19T14:53:47.730733+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:47.730733+0000","last_undegraded":"2024-02-19T14:53:47.730733+0000","last_fullsized":"2024-02-19T14:53:47.730733+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.732525+0000","last_change":"2024-02-19T14:53:47.732525+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.732525+0000","last_undegraded":"2024-02-19T14:53:47.732525+0000","last_fullsized":"2024-02-19T14:53:47.732525+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.746917+0000","last_change":"2024-02-19T14:53:47.742015+0000","last_active":"2024-02-19T14:53:47.746917+0000","last_peered":"2024-02-19T14:53:47.746917+0000","last_clean":"2024-02-19T14:53:47.746917+0000","last_became_active":"2024-02-19T14:53:47.741742+0000","last_became_peered":"2024-02-19T14:53:47.741742+0000","last_unstale":"2024-02-19T14:53:47.746917+0000","last_undegraded":"2024-02-19T14:53:47.746917+0000","last_fullsized":"2024-02-19T14:53:47.746917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:10.063517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.724348+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:47.724348+0000","last_peered":"2024-02-19T14:53:47.724348+0000","last_clean":"2024-02-19T14:53:47.724348+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:47.724348+0000","last_undegraded":"2024-02-19T14:53:47.724348+0000","last_fullsized":"2024-02-19T14:53:47.724348+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:45.189873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734386+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:47.734386+0000","last_peered":"2024-02-19T14:53:47.734386+0000","last_clean":"2024-02-19T14:53:47.734386+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:47.734386+0000","last_undegraded":"2024-02-19T14:53:47.734386+0000","last_fullsized":"2024-02-19T14:53:47.734386+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-02-19T14:53:47.728949+0000","last_change":"2024-02-19T14:53:47.728949+0000","last_active":"2024-02-19T14:53:47.716914+0000","last_peered":"2024-02-19T14:53:47.716914+0000","last_clean":"2024-02-19T14:53:47.716914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:53:47.728949+0000","last_undegraded":"2024-02-19T14:53:47.728949+0000","last_fullsized":"2024-02-19T14:53:47.728949+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737489+0000","last_change":"2024-02-19T14:53:47.734524+0000","last_active":"2024-02-19T14:53:47.737489+0000","last_peered":"2024-02-19T14:53:47.737489+0000","last_clean":"2024-02-19T14:53:47.737489+0000","last_became_active":"2024-02-19T14:53:47.734132+0000","last_became_peered":"2024-02-19T14:53:47.734132+0000","last_unstale":"2024-02-19T14:53:47.737489+0000","last_undegraded":"2024-02-19T14:53:47.737489+0000","last_fullsized":"2024-02-19T14:53:47.737489+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:11.323989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.737553+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:47.737553+0000","last_peered":"2024-02-19T14:53:47.737553+0000","last_clean":"2024-02-19T14:53:47.737553+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:47.737553+0000","last_undegraded":"2024-02-19T14:53:47.737553+0000","last_fullsized":"2024-02-19T14:53:47.737553+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":23,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.734066+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:47.734066+0000","last_peered":"2024-02-19T14:53:47.734066+0000","last_clean":"2024-02-19T14:53:47.734066+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:47.734066+0000","last_undegraded":"2024-02-19T14:53:47.734066+0000","last_fullsized":"2024-02-19T14:53:47.734066+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":236,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-02-19T14:53:47.729312+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:47.729312+0000","last_peered":"2024-02-19T14:53:47.729312+0000","last_clean":"2024-02-19T14:53:47.729312+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:47.729312+0000","last_undegraded":"2024-02-19T14:53:47.729312+0000","last_fullsized":"2024-02-19T14:53:47.729312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.502},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52500000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":651264,"data_stored":291530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.502}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.626},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":290878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.311},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.504}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47799999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.53700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":659456,"data_stored":299032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.624},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":292512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"last":0.245}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.372}]},{"osd":2,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.621}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":292490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.31}]},{"osd":3,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:52:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.5}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":298610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.247},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.72699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1245184,"data_stored":881302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.20000000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.217},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.435},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.36399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.42399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:52:49 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:55.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:53:55.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:53:56.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:53:56.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.599+0000 7f7ffef72640 1 Processor -- start 2024-02-19T14:53:56.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.599+0000 7f7ffef72640 1 -- start start 2024-02-19T14:53:56.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffef72640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7ff80c6d60 0x7f7ff8154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:56.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffef72640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ff806e830 0x7f7ff80c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:56.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffef72640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff814f730 con 0x7f7ff806ff80 2024-02-19T14:53:56.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffef72640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff80c1940 con 0x7f7ff80c6d60 2024-02-19T14:53:56.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffef72640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff8072f70 con 0x7f7ff806e830 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7ff80c6d60 0x7f7ff8154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7ff80c6d60 0x7f7ff8154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53456/0 (socket says 172.21.15.78:53456) 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 -- 172.21.15.78:0/4056761778 learned_addr learned my addr 172.21.15.78:0/4056761778 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ff7fff640 1 --2- 172.21.15.78:0/4056761778 >> v2:172.21.15.150:3300/0 conn(0x7f7ff806e830 0x7f7ff80c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 -- 172.21.15.78:0/4056761778 >> v2:172.21.15.150:3300/0 conn(0x7f7ff806e830 msgr2=0x7f7ff80c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 --2- 172.21.15.78:0/4056761778 >> v2:172.21.15.150:3300/0 conn(0x7f7ff806e830 0x7f7ff80c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 -- 172.21.15.78:0/4056761778 >> v1:172.21.15.150:6789/0 conn(0x7f7ff806ff80 legacy=0x7f7ff80c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:56.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.600+0000 7f7ffcd0e640 1 -- 172.21.15.78:0/4056761778 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80c1b50 con 0x7f7ff80c6d60 2024-02-19T14:53:56.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.601+0000 7f7ffcd0e640 1 --2- 172.21.15.78:0/4056761778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff80c6d60 0x7f7ff8154f70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7ff806cfc0 tx=0x7f7fec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e85a5fa769542d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ff77fe640 1 -- 172.21.15.78:0/4056761778 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec031870 con 0x7f7ff80c6d60 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ff77fe640 1 -- 172.21.15.78:0/4056761778 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec0319d0 con 0x7f7ff80c6d60 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ff77fe640 1 -- 172.21.15.78:0/4056761778 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec031d60 con 0x7f7ff80c6d60 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 -- 172.21.15.78:0/4056761778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff80c6d60 msgr2=0x7f7ff8154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 --2- 172.21.15.78:0/4056761778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff80c6d60 0x7f7ff8154f70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7ff806cfc0 tx=0x7f7fec02b910 comp rx=0 tx=0).stop 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 -- 172.21.15.78:0/4056761778 shutdown_connections 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 --2- 172.21.15.78:0/4056761778 >> v2:172.21.15.150:3300/0 conn(0x7f7ff806e830 0x7f7ff80c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 --2- 172.21.15.78:0/4056761778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff80c6d60 0x7f7ff8154f70 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 -- 172.21.15.78:0/4056761778 >> 172.21.15.78:0/4056761778 conn(0x7f7ff806c170 msgr2=0x7f7ff806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 -- 172.21.15.78:0/4056761778 shutdown_connections 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.602+0000 7f7ffef72640 1 -- 172.21.15.78:0/4056761778 wait complete. 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.603+0000 7f7ffef72640 1 Processor -- start 2024-02-19T14:53:56.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.603+0000 7f7ffef72640 1 -- start start 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffef72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff806e830 0x7f7ff8149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffef72640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 0x7f7ff8149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffef72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff80c6d60 0x7f7ff81ee060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffef72640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80a7840 con 0x7f7ff806ff80 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffef72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80a7c60 con 0x7f7ff806e830 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffef72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80a74a0 con 0x7f7ff80c6d60 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ff7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 0x7f7ff8149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ff7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 0x7f7ff8149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53468/0 (socket says 172.21.15.78:53468) 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ff7fff640 1 -- 172.21.15.78:0/2768843617 learned_addr learned my addr 172.21.15.78:0/2768843617 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffcd0e640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff806e830 0x7f7ff8149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.604+0000 7f7ffd50f640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff80c6d60 0x7f7ff81ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff7fff640 1 -- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff80c6d60 msgr2=0x7f7ff81ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff7fff640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff80c6d60 0x7f7ff81ee060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff7fff640 1 -- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff806e830 msgr2=0x7f7ff8149310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff7fff640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff806e830 0x7f7ff8149310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff7fff640 1 -- 172.21.15.78:0/2768843617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0094d0 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff7fff640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 0x7f7ff8149850 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400d5c0 tx=0x7f7fe400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40125f0 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe4012750 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4012b00 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff81eebf0 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.605+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff81eeed0 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.606+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff8074990 con 0x7f7ff806ff80 2024-02-19T14:53:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.606+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7fe4012c60 con 0x7f7ff806ff80 2024-02-19T14:53:56.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.607+0000 7f7ff57fa640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7fdc06b810 0x7f7fdc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:56.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.607+0000 7f7ffcd0e640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7fdc06b810 0x7f7fdc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:56.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.607+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f7fe408ab30 con 0x7f7ff806ff80 2024-02-19T14:53:56.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.607+0000 7f7ffcd0e640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7fdc06b810 0x7f7fdc06dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7fec036000 tx=0x7f7fec03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:56.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.611+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7fe40550c0 con 0x7f7ff806ff80 2024-02-19T14:53:56.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.817+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ff80c7d10 con 0x7f7fdc06b810 2024-02-19T14:53:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.843+0000 7f7ff57fa640 1 -- 172.21.15.78:0/2768843617 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+527955 (secure 0 0 0) 0x7f7ff80c7d10 con 0x7f7fdc06b810 2024-02-19T14:53:56.851 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:56.854 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7fdc06b810 msgr2=0x7f7fdc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7fdc06b810 0x7f7fdc06dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7fec036000 tx=0x7f7fec03c040 comp rx=0 tx=0).stop 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 msgr2=0x7f7ff8149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 0x7f7ff8149850 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400d5c0 tx=0x7f7fe400da80 comp rx=0 tx=0).stop 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 shutdown_connections 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7fdc06b810 0x7f7fdc06dcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ff806ff80 0x7f7ff8149850 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff80c6d60 0x7f7ff81ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 --2- 172.21.15.78:0/2768843617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff806e830 0x7f7ff8149310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 >> 172.21.15.78:0/2768843617 conn(0x7f7ff806c170 msgr2=0x7f7ff80726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 shutdown_connections 2024-02-19T14:53:56.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:56.848+0000 7f7ffef72640 1 -- 172.21.15.78:0/2768843617 wait complete. 2024-02-19T14:53:56.861 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-02-19T14:53:56.083106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222796,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751924,"statfs":{"total":773094113280,"available":772865970176,"internally_reserved":0,"allocated":6758400,"data_stored":3703496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001513"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102047+0000","last_change":"2024-02-19T14:53:48.757941+0000","last_active":"2024-02-19T14:53:50.102047+0000","last_peered":"2024-02-19T14:53:50.102047+0000","last_clean":"2024-02-19T14:53:50.102047+0000","last_became_active":"2024-02-19T14:53:48.757562+0000","last_became_peered":"2024-02-19T14:53:48.757562+0000","last_unstale":"2024-02-19T14:53:50.102047+0000","last_undegraded":"2024-02-19T14:53:50.102047+0000","last_fullsized":"2024-02-19T14:53:50.102047+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:52.856768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737338+0000","last_change":"2024-02-19T14:53:37.690208+0000","last_active":"2024-02-19T14:53:49.737338+0000","last_peered":"2024-02-19T14:53:49.737338+0000","last_clean":"2024-02-19T14:53:49.737338+0000","last_became_active":"2024-02-19T14:53:37.689948+0000","last_became_peered":"2024-02-19T14:53:37.689948+0000","last_unstale":"2024-02-19T14:53:49.737338+0000","last_undegraded":"2024-02-19T14:53:49.737338+0000","last_fullsized":"2024-02-19T14:53:49.737338+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738588+0000","last_change":"2024-02-19T14:53:48.754214+0000","last_active":"2024-02-19T14:53:49.738588+0000","last_peered":"2024-02-19T14:53:49.738588+0000","last_clean":"2024-02-19T14:53:49.738588+0000","last_became_active":"2024-02-19T14:53:48.753910+0000","last_became_peered":"2024-02-19T14:53:48.753910+0000","last_unstale":"2024-02-19T14:53:49.738588+0000","last_undegraded":"2024-02-19T14:53:49.738588+0000","last_fullsized":"2024-02-19T14:53:49.738588+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:52.865830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102088+0000","last_change":"2024-02-19T14:53:37.694456+0000","last_active":"2024-02-19T14:53:50.102088+0000","last_peered":"2024-02-19T14:53:50.102088+0000","last_clean":"2024-02-19T14:53:50.102088+0000","last_became_active":"2024-02-19T14:53:37.694185+0000","last_became_peered":"2024-02-19T14:53:37.694185+0000","last_unstale":"2024-02-19T14:53:50.102088+0000","last_undegraded":"2024-02-19T14:53:50.102088+0000","last_fullsized":"2024-02-19T14:53:50.102088+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737257+0000","last_change":"2024-02-19T14:53:48.745324+0000","last_active":"2024-02-19T14:53:49.737257+0000","last_peered":"2024-02-19T14:53:49.737257+0000","last_clean":"2024-02-19T14:53:49.737257+0000","last_became_active":"2024-02-19T14:53:48.745062+0000","last_became_peered":"2024-02-19T14:53:48.745062+0000","last_unstale":"2024-02-19T14:53:49.737257+0000","last_undegraded":"2024-02-19T14:53:49.737257+0000","last_fullsized":"2024-02-19T14:53:49.737257+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:12.328626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103398+0000","last_change":"2024-02-19T14:53:37.703581+0000","last_active":"2024-02-19T14:53:50.103398+0000","last_peered":"2024-02-19T14:53:50.103398+0000","last_clean":"2024-02-19T14:53:50.103398+0000","last_became_active":"2024-02-19T14:53:37.702703+0000","last_became_peered":"2024-02-19T14:53:37.702703+0000","last_unstale":"2024-02-19T14:53:50.103398+0000","last_undegraded":"2024-02-19T14:53:50.103398+0000","last_fullsized":"2024-02-19T14:53:50.103398+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737137+0000","last_change":"2024-02-19T14:53:48.736718+0000","last_active":"2024-02-19T14:53:49.737137+0000","last_peered":"2024-02-19T14:53:49.737137+0000","last_clean":"2024-02-19T14:53:49.737137+0000","last_became_active":"2024-02-19T14:53:48.736462+0000","last_became_peered":"2024-02-19T14:53:48.736462+0000","last_unstale":"2024-02-19T14:53:49.737137+0000","last_undegraded":"2024-02-19T14:53:49.737137+0000","last_fullsized":"2024-02-19T14:53:49.737137+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:49.406859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101878+0000","last_change":"2024-02-19T14:53:37.695736+0000","last_active":"2024-02-19T14:53:50.101878+0000","last_peered":"2024-02-19T14:53:50.101878+0000","last_clean":"2024-02-19T14:53:50.101878+0000","last_became_active":"2024-02-19T14:53:37.695408+0000","last_became_peered":"2024-02-19T14:53:37.695408+0000","last_unstale":"2024-02-19T14:53:50.101878+0000","last_undegraded":"2024-02-19T14:53:50.101878+0000","last_fullsized":"2024-02-19T14:53:50.101878+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103212+0000","last_change":"2024-02-19T14:53:48.743188+0000","last_active":"2024-02-19T14:53:50.103212+0000","last_peered":"2024-02-19T14:53:50.103212+0000","last_clean":"2024-02-19T14:53:50.103212+0000","last_became_active":"2024-02-19T14:53:48.742913+0000","last_became_peered":"2024-02-19T14:53:48.742913+0000","last_unstale":"2024-02-19T14:53:50.103212+0000","last_undegraded":"2024-02-19T14:53:50.103212+0000","last_fullsized":"2024-02-19T14:53:50.103212+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:29.490375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737819+0000","last_change":"2024-02-19T14:53:37.685381+0000","last_active":"2024-02-19T14:53:49.737819+0000","last_peered":"2024-02-19T14:53:49.737819+0000","last_clean":"2024-02-19T14:53:49.737819+0000","last_became_active":"2024-02-19T14:53:37.685087+0000","last_became_peered":"2024-02-19T14:53:37.685087+0000","last_unstale":"2024-02-19T14:53:49.737819+0000","last_undegraded":"2024-02-19T14:53:49.737819+0000","last_fullsized":"2024-02-19T14:53:49.737819+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:17.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737251+0000","last_change":"2024-02-19T14:53:48.752515+0000","last_active":"2024-02-19T14:53:49.737251+0000","last_peered":"2024-02-19T14:53:49.737251+0000","last_clean":"2024-02-19T14:53:49.737251+0000","last_became_active":"2024-02-19T14:53:48.752253+0000","last_became_peered":"2024-02-19T14:53:48.752253+0000","last_unstale":"2024-02-19T14:53:49.737251+0000","last_undegraded":"2024-02-19T14:53:49.737251+0000","last_fullsized":"2024-02-19T14:53:49.737251+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:19.624345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736384+0000","last_change":"2024-02-19T14:53:37.685701+0000","last_active":"2024-02-19T14:53:49.736384+0000","last_peered":"2024-02-19T14:53:49.736384+0000","last_clean":"2024-02-19T14:53:49.736384+0000","last_became_active":"2024-02-19T14:53:37.685457+0000","last_became_peered":"2024-02-19T14:53:37.685457+0000","last_unstale":"2024-02-19T14:53:49.736384+0000","last_undegraded":"2024-02-19T14:53:49.736384+0000","last_fullsized":"2024-02-19T14:53:49.736384+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101400+0000","last_change":"2024-02-19T14:53:48.750539+0000","last_active":"2024-02-19T14:53:50.101400+0000","last_peered":"2024-02-19T14:53:50.101400+0000","last_clean":"2024-02-19T14:53:50.101400+0000","last_became_active":"2024-02-19T14:53:48.750206+0000","last_became_peered":"2024-02-19T14:53:48.750206+0000","last_unstale":"2024-02-19T14:53:50.101400+0000","last_undegraded":"2024-02-19T14:53:50.101400+0000","last_fullsized":"2024-02-19T14:53:50.101400+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:01.759980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102027+0000","last_change":"2024-02-19T14:53:37.703011+0000","last_active":"2024-02-19T14:53:50.102027+0000","last_peered":"2024-02-19T14:53:50.102027+0000","last_clean":"2024-02-19T14:53:50.102027+0000","last_became_active":"2024-02-19T14:53:37.702430+0000","last_became_peered":"2024-02-19T14:53:37.702430+0000","last_unstale":"2024-02-19T14:53:50.102027+0000","last_undegraded":"2024-02-19T14:53:50.102027+0000","last_fullsized":"2024-02-19T14:53:50.102027+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101042+0000","last_change":"2024-02-19T14:53:48.738876+0000","last_active":"2024-02-19T14:53:50.101042+0000","last_peered":"2024-02-19T14:53:50.101042+0000","last_clean":"2024-02-19T14:53:50.101042+0000","last_became_active":"2024-02-19T14:53:48.738625+0000","last_became_peered":"2024-02-19T14:53:48.738625+0000","last_unstale":"2024-02-19T14:53:50.101042+0000","last_undegraded":"2024-02-19T14:53:50.101042+0000","last_fullsized":"2024-02-19T14:53:50.101042+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:10.556428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101728+0000","last_change":"2024-02-19T14:53:37.704147+0000","last_active":"2024-02-19T14:53:50.101728+0000","last_peered":"2024-02-19T14:53:50.101728+0000","last_clean":"2024-02-19T14:53:50.101728+0000","last_became_active":"2024-02-19T14:53:37.703914+0000","last_became_peered":"2024-02-19T14:53:37.703914+0000","last_unstale":"2024-02-19T14:53:50.101728+0000","last_undegraded":"2024-02-19T14:53:50.101728+0000","last_fullsized":"2024-02-19T14:53:50.101728+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739362+0000","last_change":"2024-02-19T14:53:48.740132+0000","last_active":"2024-02-19T14:53:49.739362+0000","last_peered":"2024-02-19T14:53:49.739362+0000","last_clean":"2024-02-19T14:53:49.739362+0000","last_became_active":"2024-02-19T14:53:48.739841+0000","last_became_peered":"2024-02-19T14:53:48.739841+0000","last_unstale":"2024-02-19T14:53:49.739362+0000","last_undegraded":"2024-02-19T14:53:49.739362+0000","last_fullsized":"2024-02-19T14:53:49.739362+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:08:10.106034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102174+0000","last_change":"2024-02-19T14:53:37.706570+0000","last_active":"2024-02-19T14:53:50.102174+0000","last_peered":"2024-02-19T14:53:50.102174+0000","last_clean":"2024-02-19T14:53:50.102174+0000","last_became_active":"2024-02-19T14:53:37.706277+0000","last_became_peered":"2024-02-19T14:53:37.706277+0000","last_unstale":"2024-02-19T14:53:50.102174+0000","last_undegraded":"2024-02-19T14:53:50.102174+0000","last_fullsized":"2024-02-19T14:53:50.102174+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:55:50.707228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738080+0000","last_change":"2024-02-19T14:53:48.753605+0000","last_active":"2024-02-19T14:53:49.738080+0000","last_peered":"2024-02-19T14:53:49.738080+0000","last_clean":"2024-02-19T14:53:49.738080+0000","last_became_active":"2024-02-19T14:53:48.752982+0000","last_became_peered":"2024-02-19T14:53:48.752982+0000","last_unstale":"2024-02-19T14:53:49.738080+0000","last_undegraded":"2024-02-19T14:53:49.738080+0000","last_fullsized":"2024-02-19T14:53:49.738080+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:11.846193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737152+0000","last_change":"2024-02-19T14:53:37.685139+0000","last_active":"2024-02-19T14:53:49.737152+0000","last_peered":"2024-02-19T14:53:49.737152+0000","last_clean":"2024-02-19T14:53:49.737152+0000","last_became_active":"2024-02-19T14:53:37.684824+0000","last_became_peered":"2024-02-19T14:53:37.684824+0000","last_unstale":"2024-02-19T14:53:49.737152+0000","last_undegraded":"2024-02-19T14:53:49.737152+0000","last_fullsized":"2024-02-19T14:53:49.737152+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102567+0000","last_change":"2024-02-19T14:53:48.759862+0000","last_active":"2024-02-19T14:53:50.102567+0000","last_peered":"2024-02-19T14:53:50.102567+0000","last_clean":"2024-02-19T14:53:50.102567+0000","last_became_active":"2024-02-19T14:53:48.759562+0000","last_became_peered":"2024-02-19T14:53:48.759562+0000","last_unstale":"2024-02-19T14:53:50.102567+0000","last_undegraded":"2024-02-19T14:53:50.102567+0000","last_fullsized":"2024-02-19T14:53:50.102567+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:15.245868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101298+0000","last_change":"2024-02-19T14:53:37.686436+0000","last_active":"2024-02-19T14:53:50.101298+0000","last_peered":"2024-02-19T14:53:50.101298+0000","last_clean":"2024-02-19T14:53:50.101298+0000","last_became_active":"2024-02-19T14:53:37.686135+0000","last_became_peered":"2024-02-19T14:53:37.686135+0000","last_unstale":"2024-02-19T14:53:50.101298+0000","last_undegraded":"2024-02-19T14:53:50.101298+0000","last_fullsized":"2024-02-19T14:53:50.101298+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:01:37.193943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101906+0000","last_change":"2024-02-19T14:53:48.757577+0000","last_active":"2024-02-19T14:53:50.101906+0000","last_peered":"2024-02-19T14:53:50.101906+0000","last_clean":"2024-02-19T14:53:50.101906+0000","last_became_active":"2024-02-19T14:53:48.757177+0000","last_became_peered":"2024-02-19T14:53:48.757177+0000","last_unstale":"2024-02-19T14:53:50.101906+0000","last_undegraded":"2024-02-19T14:53:50.101906+0000","last_fullsized":"2024-02-19T14:53:50.101906+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:19.624345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102785+0000","last_change":"2024-02-19T14:53:37.688052+0000","last_active":"2024-02-19T14:53:50.102785+0000","last_peered":"2024-02-19T14:53:50.102785+0000","last_clean":"2024-02-19T14:53:50.102785+0000","last_became_active":"2024-02-19T14:53:37.687767+0000","last_became_peered":"2024-02-19T14:53:37.687767+0000","last_unstale":"2024-02-19T14:53:50.102785+0000","last_undegraded":"2024-02-19T14:53:50.102785+0000","last_fullsized":"2024-02-19T14:53:50.102785+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:01.639745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102302+0000","last_change":"2024-02-19T14:53:48.756714+0000","last_active":"2024-02-19T14:53:50.102302+0000","last_peered":"2024-02-19T14:53:50.102302+0000","last_clean":"2024-02-19T14:53:50.102302+0000","last_became_active":"2024-02-19T14:53:48.756444+0000","last_became_peered":"2024-02-19T14:53:48.756444+0000","last_unstale":"2024-02-19T14:53:50.102302+0000","last_undegraded":"2024-02-19T14:53:50.102302+0000","last_fullsized":"2024-02-19T14:53:50.102302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:26.499487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736593+0000","last_change":"2024-02-19T14:53:37.686388+0000","last_active":"2024-02-19T14:53:49.736593+0000","last_peered":"2024-02-19T14:53:49.736593+0000","last_clean":"2024-02-19T14:53:49.736593+0000","last_became_active":"2024-02-19T14:53:37.686084+0000","last_became_peered":"2024-02-19T14:53:37.686084+0000","last_unstale":"2024-02-19T14:53:49.736593+0000","last_undegraded":"2024-02-19T14:53:49.736593+0000","last_fullsized":"2024-02-19T14:53:49.736593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100708+0000","last_change":"2024-02-19T14:53:48.742258+0000","last_active":"2024-02-19T14:53:50.100708+0000","last_peered":"2024-02-19T14:53:50.100708+0000","last_clean":"2024-02-19T14:53:50.100708+0000","last_became_active":"2024-02-19T14:53:48.742003+0000","last_became_peered":"2024-02-19T14:53:48.742003+0000","last_unstale":"2024-02-19T14:53:50.100708+0000","last_undegraded":"2024-02-19T14:53:50.100708+0000","last_fullsized":"2024-02-19T14:53:50.100708+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:29.095460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100748+0000","last_change":"2024-02-19T14:53:42.136272+0000","last_active":"2024-02-19T14:53:50.100748+0000","last_peered":"2024-02-19T14:53:50.100748+0000","last_clean":"2024-02-19T14:53:50.100748+0000","last_became_active":"2024-02-19T14:53:42.135998+0000","last_became_peered":"2024-02-19T14:53:42.135998+0000","last_unstale":"2024-02-19T14:53:50.100748+0000","last_undegraded":"2024-02-19T14:53:50.100748+0000","last_fullsized":"2024-02-19T14:53:50.100748+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737693+0000","last_change":"2024-02-19T14:53:48.741292+0000","last_active":"2024-02-19T14:53:49.737693+0000","last_peered":"2024-02-19T14:53:49.737693+0000","last_clean":"2024-02-19T14:53:49.737693+0000","last_became_active":"2024-02-19T14:53:48.740774+0000","last_became_peered":"2024-02-19T14:53:48.740774+0000","last_unstale":"2024-02-19T14:53:49.737693+0000","last_undegraded":"2024-02-19T14:53:49.737693+0000","last_fullsized":"2024-02-19T14:53:49.737693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:05.407532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102574+0000","last_change":"2024-02-19T14:53:37.690867+0000","last_active":"2024-02-19T14:53:50.102574+0000","last_peered":"2024-02-19T14:53:50.102574+0000","last_clean":"2024-02-19T14:53:50.102574+0000","last_became_active":"2024-02-19T14:53:37.690550+0000","last_became_peered":"2024-02-19T14:53:37.690550+0000","last_unstale":"2024-02-19T14:53:50.102574+0000","last_undegraded":"2024-02-19T14:53:50.102574+0000","last_fullsized":"2024-02-19T14:53:50.102574+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101480+0000","last_change":"2024-02-19T14:53:48.757750+0000","last_active":"2024-02-19T14:53:50.101480+0000","last_peered":"2024-02-19T14:53:50.101480+0000","last_clean":"2024-02-19T14:53:50.101480+0000","last_became_active":"2024-02-19T14:53:48.757442+0000","last_became_peered":"2024-02-19T14:53:48.757442+0000","last_unstale":"2024-02-19T14:53:50.101480+0000","last_undegraded":"2024-02-19T14:53:50.101480+0000","last_fullsized":"2024-02-19T14:53:50.101480+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:11.846193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736241+0000","last_change":"2024-02-19T14:53:37.687426+0000","last_active":"2024-02-19T14:53:49.736241+0000","last_peered":"2024-02-19T14:53:49.736241+0000","last_clean":"2024-02-19T14:53:49.736241+0000","last_became_active":"2024-02-19T14:53:37.687148+0000","last_became_peered":"2024-02-19T14:53:37.687148+0000","last_unstale":"2024-02-19T14:53:49.736241+0000","last_undegraded":"2024-02-19T14:53:49.736241+0000","last_fullsized":"2024-02-19T14:53:49.736241+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101599+0000","last_change":"2024-02-19T14:53:48.746454+0000","last_active":"2024-02-19T14:53:50.101599+0000","last_peered":"2024-02-19T14:53:50.101599+0000","last_clean":"2024-02-19T14:53:50.101599+0000","last_became_active":"2024-02-19T14:53:48.746160+0000","last_became_peered":"2024-02-19T14:53:48.746160+0000","last_unstale":"2024-02-19T14:53:50.101599+0000","last_undegraded":"2024-02-19T14:53:50.101599+0000","last_fullsized":"2024-02-19T14:53:50.101599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:56.242624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102501+0000","last_change":"2024-02-19T14:53:37.703923+0000","last_active":"2024-02-19T14:53:50.102501+0000","last_peered":"2024-02-19T14:53:50.102501+0000","last_clean":"2024-02-19T14:53:50.102501+0000","last_became_active":"2024-02-19T14:53:37.703529+0000","last_became_peered":"2024-02-19T14:53:37.703529+0000","last_unstale":"2024-02-19T14:53:50.102501+0000","last_undegraded":"2024-02-19T14:53:50.102501+0000","last_fullsized":"2024-02-19T14:53:50.102501+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102407+0000","last_change":"2024-02-19T14:53:48.743347+0000","last_active":"2024-02-19T14:53:50.102407+0000","last_peered":"2024-02-19T14:53:50.102407+0000","last_clean":"2024-02-19T14:53:50.102407+0000","last_became_active":"2024-02-19T14:53:48.743102+0000","last_became_peered":"2024-02-19T14:53:48.743102+0000","last_unstale":"2024-02-19T14:53:50.102407+0000","last_undegraded":"2024-02-19T14:53:50.102407+0000","last_fullsized":"2024-02-19T14:53:50.102407+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:44.384011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102117+0000","last_change":"2024-02-19T14:53:37.694066+0000","last_active":"2024-02-19T14:53:50.102117+0000","last_peered":"2024-02-19T14:53:50.102117+0000","last_clean":"2024-02-19T14:53:50.102117+0000","last_became_active":"2024-02-19T14:53:37.693787+0000","last_became_peered":"2024-02-19T14:53:37.693787+0000","last_unstale":"2024-02-19T14:53:50.102117+0000","last_undegraded":"2024-02-19T14:53:50.102117+0000","last_fullsized":"2024-02-19T14:53:50.102117+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737273+0000","last_change":"2024-02-19T14:53:48.733236+0000","last_active":"2024-02-19T14:53:49.737273+0000","last_peered":"2024-02-19T14:53:49.737273+0000","last_clean":"2024-02-19T14:53:49.737273+0000","last_became_active":"2024-02-19T14:53:48.732981+0000","last_became_peered":"2024-02-19T14:53:48.732981+0000","last_unstale":"2024-02-19T14:53:49.737273+0000","last_undegraded":"2024-02-19T14:53:49.737273+0000","last_fullsized":"2024-02-19T14:53:49.737273+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:02:57.729629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103474+0000","last_change":"2024-02-19T14:53:37.702984+0000","last_active":"2024-02-19T14:53:50.103474+0000","last_peered":"2024-02-19T14:53:50.103474+0000","last_clean":"2024-02-19T14:53:50.103474+0000","last_became_active":"2024-02-19T14:53:37.702267+0000","last_became_peered":"2024-02-19T14:53:37.702267+0000","last_unstale":"2024-02-19T14:53:50.103474+0000","last_undegraded":"2024-02-19T14:53:50.103474+0000","last_fullsized":"2024-02-19T14:53:50.103474+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:52.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739364+0000","last_change":"2024-02-19T14:53:48.752667+0000","last_active":"2024-02-19T14:53:49.739364+0000","last_peered":"2024-02-19T14:53:49.739364+0000","last_clean":"2024-02-19T14:53:49.739364+0000","last_became_active":"2024-02-19T14:53:48.752413+0000","last_became_peered":"2024-02-19T14:53:48.752413+0000","last_unstale":"2024-02-19T14:53:49.739364+0000","last_undegraded":"2024-02-19T14:53:49.739364+0000","last_fullsized":"2024-02-19T14:53:49.739364+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:03.794668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101249+0000","last_change":"2024-02-19T14:53:37.686678+0000","last_active":"2024-02-19T14:53:50.101249+0000","last_peered":"2024-02-19T14:53:50.101249+0000","last_clean":"2024-02-19T14:53:50.101249+0000","last_became_active":"2024-02-19T14:53:37.686342+0000","last_became_peered":"2024-02-19T14:53:37.686342+0000","last_unstale":"2024-02-19T14:53:50.101249+0000","last_undegraded":"2024-02-19T14:53:50.101249+0000","last_fullsized":"2024-02-19T14:53:50.101249+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.828377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100649+0000","last_change":"2024-02-19T14:53:48.743985+0000","last_active":"2024-02-19T14:53:50.100649+0000","last_peered":"2024-02-19T14:53:50.100649+0000","last_clean":"2024-02-19T14:53:50.100649+0000","last_became_active":"2024-02-19T14:53:48.743748+0000","last_became_peered":"2024-02-19T14:53:48.743748+0000","last_unstale":"2024-02-19T14:53:50.100649+0000","last_undegraded":"2024-02-19T14:53:50.100649+0000","last_fullsized":"2024-02-19T14:53:50.100649+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:20:32.874979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101500+0000","last_change":"2024-02-19T14:53:37.691731+0000","last_active":"2024-02-19T14:53:50.101500+0000","last_peered":"2024-02-19T14:53:50.101500+0000","last_clean":"2024-02-19T14:53:50.101500+0000","last_became_active":"2024-02-19T14:53:37.691411+0000","last_became_peered":"2024-02-19T14:53:37.691411+0000","last_unstale":"2024-02-19T14:53:50.101500+0000","last_undegraded":"2024-02-19T14:53:50.101500+0000","last_fullsized":"2024-02-19T14:53:50.101500+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:17:40.546107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101062+0000","last_change":"2024-02-19T14:53:48.757928+0000","last_active":"2024-02-19T14:53:50.101062+0000","last_peered":"2024-02-19T14:53:50.101062+0000","last_clean":"2024-02-19T14:53:50.101062+0000","last_became_active":"2024-02-19T14:53:48.757541+0000","last_became_peered":"2024-02-19T14:53:48.757541+0000","last_unstale":"2024-02-19T14:53:50.101062+0000","last_undegraded":"2024-02-19T14:53:50.101062+0000","last_fullsized":"2024-02-19T14:53:50.101062+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:05.407532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101317+0000","last_change":"2024-02-19T14:53:37.706178+0000","last_active":"2024-02-19T14:53:50.101317+0000","last_peered":"2024-02-19T14:53:50.101317+0000","last_clean":"2024-02-19T14:53:50.101317+0000","last_became_active":"2024-02-19T14:53:37.705870+0000","last_became_peered":"2024-02-19T14:53:37.705870+0000","last_unstale":"2024-02-19T14:53:50.101317+0000","last_undegraded":"2024-02-19T14:53:50.101317+0000","last_fullsized":"2024-02-19T14:53:50.101317+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:59.053283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737839+0000","last_change":"2024-02-19T14:53:48.741070+0000","last_active":"2024-02-19T14:53:49.737839+0000","last_peered":"2024-02-19T14:53:49.737839+0000","last_clean":"2024-02-19T14:53:49.737839+0000","last_became_active":"2024-02-19T14:53:48.740473+0000","last_became_peered":"2024-02-19T14:53:48.740473+0000","last_unstale":"2024-02-19T14:53:49.737839+0000","last_undegraded":"2024-02-19T14:53:49.737839+0000","last_fullsized":"2024-02-19T14:53:49.737839+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:52.521850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103395+0000","last_change":"2024-02-19T14:53:37.692007+0000","last_active":"2024-02-19T14:53:50.103395+0000","last_peered":"2024-02-19T14:53:50.103395+0000","last_clean":"2024-02-19T14:53:50.103395+0000","last_became_active":"2024-02-19T14:53:37.691713+0000","last_became_peered":"2024-02-19T14:53:37.691713+0000","last_unstale":"2024-02-19T14:53:50.103395+0000","last_undegraded":"2024-02-19T14:53:50.103395+0000","last_fullsized":"2024-02-19T14:53:50.103395+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:38:18.934930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101052+0000","last_change":"2024-02-19T14:53:48.743972+0000","last_active":"2024-02-19T14:53:50.101052+0000","last_peered":"2024-02-19T14:53:50.101052+0000","last_clean":"2024-02-19T14:53:50.101052+0000","last_became_active":"2024-02-19T14:53:48.743707+0000","last_became_peered":"2024-02-19T14:53:48.743707+0000","last_unstale":"2024-02-19T14:53:50.101052+0000","last_undegraded":"2024-02-19T14:53:50.101052+0000","last_fullsized":"2024-02-19T14:53:50.101052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:19.408429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103068+0000","last_change":"2024-02-19T14:53:37.702633+0000","last_active":"2024-02-19T14:53:50.103068+0000","last_peered":"2024-02-19T14:53:50.103068+0000","last_clean":"2024-02-19T14:53:50.103068+0000","last_became_active":"2024-02-19T14:53:37.702334+0000","last_became_peered":"2024-02-19T14:53:37.702334+0000","last_unstale":"2024-02-19T14:53:50.103068+0000","last_undegraded":"2024-02-19T14:53:50.103068+0000","last_fullsized":"2024-02-19T14:53:50.103068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:13.405953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739121+0000","last_change":"2024-02-19T14:53:48.736067+0000","last_active":"2024-02-19T14:53:49.739121+0000","last_peered":"2024-02-19T14:53:49.739121+0000","last_clean":"2024-02-19T14:53:49.739121+0000","last_became_active":"2024-02-19T14:53:48.735758+0000","last_became_peered":"2024-02-19T14:53:48.735758+0000","last_unstale":"2024-02-19T14:53:49.739121+0000","last_undegraded":"2024-02-19T14:53:49.739121+0000","last_fullsized":"2024-02-19T14:53:49.739121+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:14.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102626+0000","last_change":"2024-02-19T14:53:37.703105+0000","last_active":"2024-02-19T14:53:50.102626+0000","last_peered":"2024-02-19T14:53:50.102626+0000","last_clean":"2024-02-19T14:53:50.102626+0000","last_became_active":"2024-02-19T14:53:37.702175+0000","last_became_peered":"2024-02-19T14:53:37.702175+0000","last_unstale":"2024-02-19T14:53:50.102626+0000","last_undegraded":"2024-02-19T14:53:50.102626+0000","last_fullsized":"2024-02-19T14:53:50.102626+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736893+0000","last_change":"2024-02-19T14:53:48.742934+0000","last_active":"2024-02-19T14:53:49.736893+0000","last_peered":"2024-02-19T14:53:49.736893+0000","last_clean":"2024-02-19T14:53:49.736893+0000","last_became_active":"2024-02-19T14:53:48.742667+0000","last_became_peered":"2024-02-19T14:53:48.742667+0000","last_unstale":"2024-02-19T14:53:49.736893+0000","last_undegraded":"2024-02-19T14:53:49.736893+0000","last_fullsized":"2024-02-19T14:53:49.736893+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:00.458513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100768+0000","last_change":"2024-02-19T14:53:37.697144+0000","last_active":"2024-02-19T14:53:50.100768+0000","last_peered":"2024-02-19T14:53:50.100768+0000","last_clean":"2024-02-19T14:53:50.100768+0000","last_became_active":"2024-02-19T14:53:37.696863+0000","last_became_peered":"2024-02-19T14:53:37.696863+0000","last_unstale":"2024-02-19T14:53:50.100768+0000","last_undegraded":"2024-02-19T14:53:50.100768+0000","last_fullsized":"2024-02-19T14:53:50.100768+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:40.826723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738160+0000","last_change":"2024-02-19T14:53:48.745344+0000","last_active":"2024-02-19T14:53:49.738160+0000","last_peered":"2024-02-19T14:53:49.738160+0000","last_clean":"2024-02-19T14:53:49.738160+0000","last_became_active":"2024-02-19T14:53:48.745084+0000","last_became_peered":"2024-02-19T14:53:48.745084+0000","last_unstale":"2024-02-19T14:53:49.738160+0000","last_undegraded":"2024-02-19T14:53:49.738160+0000","last_fullsized":"2024-02-19T14:53:49.738160+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:10.556428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736955+0000","last_change":"2024-02-19T14:53:37.704883+0000","last_active":"2024-02-19T14:53:49.736955+0000","last_peered":"2024-02-19T14:53:49.736955+0000","last_clean":"2024-02-19T14:53:49.736955+0000","last_became_active":"2024-02-19T14:53:37.704599+0000","last_became_peered":"2024-02-19T14:53:37.704599+0000","last_unstale":"2024-02-19T14:53:49.736955+0000","last_undegraded":"2024-02-19T14:53:49.736955+0000","last_fullsized":"2024-02-19T14:53:49.736955+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:31.135186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739061+0000","last_change":"2024-02-19T14:53:48.746703+0000","last_active":"2024-02-19T14:53:49.739061+0000","last_peered":"2024-02-19T14:53:49.739061+0000","last_clean":"2024-02-19T14:53:49.739061+0000","last_became_active":"2024-02-19T14:53:48.746001+0000","last_became_peered":"2024-02-19T14:53:48.746001+0000","last_unstale":"2024-02-19T14:53:49.739061+0000","last_undegraded":"2024-02-19T14:53:49.739061+0000","last_fullsized":"2024-02-19T14:53:49.739061+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:15.245868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101388+0000","last_change":"2024-02-19T14:53:37.695075+0000","last_active":"2024-02-19T14:53:50.101388+0000","last_peered":"2024-02-19T14:53:50.101388+0000","last_clean":"2024-02-19T14:53:50.101388+0000","last_became_active":"2024-02-19T14:53:37.694637+0000","last_became_peered":"2024-02-19T14:53:37.694637+0000","last_unstale":"2024-02-19T14:53:50.101388+0000","last_undegraded":"2024-02-19T14:53:50.101388+0000","last_fullsized":"2024-02-19T14:53:50.101388+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:48.534276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738313+0000","last_change":"2024-02-19T14:53:48.734953+0000","last_active":"2024-02-19T14:53:49.738313+0000","last_peered":"2024-02-19T14:53:49.738313+0000","last_clean":"2024-02-19T14:53:49.738313+0000","last_became_active":"2024-02-19T14:53:48.734647+0000","last_became_peered":"2024-02-19T14:53:48.734647+0000","last_unstale":"2024-02-19T14:53:49.738313+0000","last_undegraded":"2024-02-19T14:53:49.738313+0000","last_fullsized":"2024-02-19T14:53:49.738313+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:52.155949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103116+0000","last_change":"2024-02-19T14:53:37.706738+0000","last_active":"2024-02-19T14:53:50.103116+0000","last_peered":"2024-02-19T14:53:50.103116+0000","last_clean":"2024-02-19T14:53:50.103116+0000","last_became_active":"2024-02-19T14:53:37.704011+0000","last_became_peered":"2024-02-19T14:53:37.704011+0000","last_unstale":"2024-02-19T14:53:50.103116+0000","last_undegraded":"2024-02-19T14:53:50.103116+0000","last_fullsized":"2024-02-19T14:53:50.103116+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102855+0000","last_change":"2024-02-19T14:53:48.737588+0000","last_active":"2024-02-19T14:53:50.102855+0000","last_peered":"2024-02-19T14:53:50.102855+0000","last_clean":"2024-02-19T14:53:50.102855+0000","last_became_active":"2024-02-19T14:53:48.737340+0000","last_became_peered":"2024-02-19T14:53:48.737340+0000","last_unstale":"2024-02-19T14:53:50.102855+0000","last_undegraded":"2024-02-19T14:53:50.102855+0000","last_fullsized":"2024-02-19T14:53:50.102855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:26.499487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101588+0000","last_change":"2024-02-19T14:53:37.690755+0000","last_active":"2024-02-19T14:53:50.101588+0000","last_peered":"2024-02-19T14:53:50.101588+0000","last_clean":"2024-02-19T14:53:50.101588+0000","last_became_active":"2024-02-19T14:53:37.690412+0000","last_became_peered":"2024-02-19T14:53:37.690412+0000","last_unstale":"2024-02-19T14:53:50.101588+0000","last_undegraded":"2024-02-19T14:53:50.101588+0000","last_fullsized":"2024-02-19T14:53:50.101588+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:12.411182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101200+0000","last_change":"2024-02-19T14:53:48.760297+0000","last_active":"2024-02-19T14:53:50.101200+0000","last_peered":"2024-02-19T14:53:50.101200+0000","last_clean":"2024-02-19T14:53:50.101200+0000","last_became_active":"2024-02-19T14:53:48.760037+0000","last_became_peered":"2024-02-19T14:53:48.760037+0000","last_unstale":"2024-02-19T14:53:50.101200+0000","last_undegraded":"2024-02-19T14:53:50.101200+0000","last_fullsized":"2024-02-19T14:53:50.101200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:52.521850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102157+0000","last_change":"2024-02-19T14:53:37.696173+0000","last_active":"2024-02-19T14:53:50.102157+0000","last_peered":"2024-02-19T14:53:50.102157+0000","last_clean":"2024-02-19T14:53:50.102157+0000","last_became_active":"2024-02-19T14:53:37.695887+0000","last_became_peered":"2024-02-19T14:53:37.695887+0000","last_unstale":"2024-02-19T14:53:50.102157+0000","last_undegraded":"2024-02-19T14:53:50.102157+0000","last_fullsized":"2024-02-19T14:53:50.102157+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737660+0000","last_change":"2024-02-19T14:53:48.754317+0000","last_active":"2024-02-19T14:53:49.737660+0000","last_peered":"2024-02-19T14:53:49.737660+0000","last_clean":"2024-02-19T14:53:49.737660+0000","last_became_active":"2024-02-19T14:53:48.754033+0000","last_became_peered":"2024-02-19T14:53:48.754033+0000","last_unstale":"2024-02-19T14:53:49.737660+0000","last_undegraded":"2024-02-19T14:53:49.737660+0000","last_fullsized":"2024-02-19T14:53:49.737660+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:07.407139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737293+0000","last_change":"2024-02-19T14:53:37.692820+0000","last_active":"2024-02-19T14:53:49.737293+0000","last_peered":"2024-02-19T14:53:49.737293+0000","last_clean":"2024-02-19T14:53:49.737293+0000","last_became_active":"2024-02-19T14:53:37.692560+0000","last_became_peered":"2024-02-19T14:53:37.692560+0000","last_unstale":"2024-02-19T14:53:49.737293+0000","last_undegraded":"2024-02-19T14:53:49.737293+0000","last_fullsized":"2024-02-19T14:53:49.737293+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100853+0000","last_change":"2024-02-19T14:53:48.751395+0000","last_active":"2024-02-19T14:53:50.100853+0000","last_peered":"2024-02-19T14:53:50.100853+0000","last_clean":"2024-02-19T14:53:50.100853+0000","last_became_active":"2024-02-19T14:53:48.750167+0000","last_became_peered":"2024-02-19T14:53:48.750167+0000","last_unstale":"2024-02-19T14:53:50.100853+0000","last_undegraded":"2024-02-19T14:53:50.100853+0000","last_fullsized":"2024-02-19T14:53:50.100853+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:21.916996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102225+0000","last_change":"2024-02-19T14:53:47.740946+0000","last_active":"2024-02-19T14:53:50.102225+0000","last_peered":"2024-02-19T14:53:50.102225+0000","last_clean":"2024-02-19T14:53:50.102225+0000","last_became_active":"2024-02-19T14:53:47.740550+0000","last_became_peered":"2024-02-19T14:53:47.740550+0000","last_unstale":"2024-02-19T14:53:50.102225+0000","last_undegraded":"2024-02-19T14:53:50.102225+0000","last_fullsized":"2024-02-19T14:53:50.102225+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:00.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101285+0000","last_change":"2024-02-19T14:53:37.694878+0000","last_active":"2024-02-19T14:53:50.101285+0000","last_peered":"2024-02-19T14:53:50.101285+0000","last_clean":"2024-02-19T14:53:50.101285+0000","last_became_active":"2024-02-19T14:53:37.694620+0000","last_became_peered":"2024-02-19T14:53:37.694620+0000","last_unstale":"2024-02-19T14:53:50.101285+0000","last_undegraded":"2024-02-19T14:53:50.101285+0000","last_fullsized":"2024-02-19T14:53:50.101285+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"50'4","reported_seq":30,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101913+0000","last_change":"2024-02-19T14:53:36.688634+0000","last_active":"2024-02-19T14:53:50.101913+0000","last_peered":"2024-02-19T14:53:50.101913+0000","last_clean":"2024-02-19T14:53:50.101913+0000","last_became_active":"2024-02-19T14:53:36.688320+0000","last_became_peered":"2024-02-19T14:53:36.688320+0000","last_unstale":"2024-02-19T14:53:50.101913+0000","last_undegraded":"2024-02-19T14:53:50.101913+0000","last_fullsized":"2024-02-19T14:53:50.101913+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736755+0000","last_change":"2024-02-19T14:53:48.748281+0000","last_active":"2024-02-19T14:53:49.736755+0000","last_peered":"2024-02-19T14:53:49.736755+0000","last_clean":"2024-02-19T14:53:49.736755+0000","last_became_active":"2024-02-19T14:53:48.748033+0000","last_became_peered":"2024-02-19T14:53:48.748033+0000","last_unstale":"2024-02-19T14:53:49.736755+0000","last_undegraded":"2024-02-19T14:53:49.736755+0000","last_fullsized":"2024-02-19T14:53:49.736755+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:56.242624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737365+0000","last_change":"2024-02-19T14:53:47.742941+0000","last_active":"2024-02-19T14:53:49.737365+0000","last_peered":"2024-02-19T14:53:49.737365+0000","last_clean":"2024-02-19T14:53:49.737365+0000","last_became_active":"2024-02-19T14:53:47.742590+0000","last_became_peered":"2024-02-19T14:53:47.742590+0000","last_unstale":"2024-02-19T14:53:49.737365+0000","last_undegraded":"2024-02-19T14:53:49.737365+0000","last_fullsized":"2024-02-19T14:53:49.737365+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:00.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738992+0000","last_change":"2024-02-19T14:53:37.685659+0000","last_active":"2024-02-19T14:53:49.738992+0000","last_peered":"2024-02-19T14:53:49.738992+0000","last_clean":"2024-02-19T14:53:49.738992+0000","last_became_active":"2024-02-19T14:53:37.685264+0000","last_became_peered":"2024-02-19T14:53:37.685264+0000","last_unstale":"2024-02-19T14:53:49.738992+0000","last_undegraded":"2024-02-19T14:53:49.738992+0000","last_fullsized":"2024-02-19T14:53:49.738992+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"50'1","reported_seq":27,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737228+0000","last_change":"2024-02-19T14:53:36.681159+0000","last_active":"2024-02-19T14:53:49.737228+0000","last_peered":"2024-02-19T14:53:49.737228+0000","last_clean":"2024-02-19T14:53:49.737228+0000","last_became_active":"2024-02-19T14:53:36.680827+0000","last_became_peered":"2024-02-19T14:53:36.680827+0000","last_unstale":"2024-02-19T14:53:49.737228+0000","last_undegraded":"2024-02-19T14:53:49.737228+0000","last_fullsized":"2024-02-19T14:53:49.737228+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:25:18.933918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737253+0000","last_change":"2024-02-19T14:53:48.744278+0000","last_active":"2024-02-19T14:53:49.737253+0000","last_peered":"2024-02-19T14:53:49.737253+0000","last_clean":"2024-02-19T14:53:49.737253+0000","last_became_active":"2024-02-19T14:53:48.742661+0000","last_became_peered":"2024-02-19T14:53:48.742661+0000","last_unstale":"2024-02-19T14:53:49.737253+0000","last_undegraded":"2024-02-19T14:53:49.737253+0000","last_fullsized":"2024-02-19T14:53:49.737253+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:27.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101454+0000","last_change":"2024-02-19T14:53:47.746326+0000","last_active":"2024-02-19T14:53:50.101454+0000","last_peered":"2024-02-19T14:53:50.101454+0000","last_clean":"2024-02-19T14:53:50.101454+0000","last_became_active":"2024-02-19T14:53:47.745753+0000","last_became_peered":"2024-02-19T14:53:47.745753+0000","last_unstale":"2024-02-19T14:53:50.101454+0000","last_undegraded":"2024-02-19T14:53:50.101454+0000","last_fullsized":"2024-02-19T14:53:50.101454+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:46.074469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101150+0000","last_change":"2024-02-19T14:53:37.692591+0000","last_active":"2024-02-19T14:53:50.101150+0000","last_peered":"2024-02-19T14:53:50.101150+0000","last_clean":"2024-02-19T14:53:50.101150+0000","last_became_active":"2024-02-19T14:53:37.691775+0000","last_became_peered":"2024-02-19T14:53:37.691775+0000","last_unstale":"2024-02-19T14:53:50.101150+0000","last_undegraded":"2024-02-19T14:53:50.101150+0000","last_fullsized":"2024-02-19T14:53:50.101150+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:33.610610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"50'2","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737017+0000","last_change":"2024-02-19T14:53:36.688842+0000","last_active":"2024-02-19T14:53:49.737017+0000","last_peered":"2024-02-19T14:53:49.737017+0000","last_clean":"2024-02-19T14:53:49.737017+0000","last_became_active":"2024-02-19T14:53:36.688583+0000","last_became_peered":"2024-02-19T14:53:36.688583+0000","last_unstale":"2024-02-19T14:53:49.737017+0000","last_undegraded":"2024-02-19T14:53:49.737017+0000","last_fullsized":"2024-02-19T14:53:49.737017+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738962+0000","last_change":"2024-02-19T14:53:48.750809+0000","last_active":"2024-02-19T14:53:49.738962+0000","last_peered":"2024-02-19T14:53:49.738962+0000","last_clean":"2024-02-19T14:53:49.738962+0000","last_became_active":"2024-02-19T14:53:48.750155+0000","last_became_peered":"2024-02-19T14:53:48.750155+0000","last_unstale":"2024-02-19T14:53:49.738962+0000","last_undegraded":"2024-02-19T14:53:49.738962+0000","last_fullsized":"2024-02-19T14:53:49.738962+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:35.702079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737763+0000","last_change":"2024-02-19T14:53:47.740910+0000","last_active":"2024-02-19T14:53:49.737763+0000","last_peered":"2024-02-19T14:53:49.737763+0000","last_clean":"2024-02-19T14:53:49.737763+0000","last_became_active":"2024-02-19T14:53:47.740626+0000","last_became_peered":"2024-02-19T14:53:47.740626+0000","last_unstale":"2024-02-19T14:53:49.737763+0000","last_undegraded":"2024-02-19T14:53:49.737763+0000","last_fullsized":"2024-02-19T14:53:49.737763+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:51.356042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101387+0000","last_change":"2024-02-19T14:53:37.697482+0000","last_active":"2024-02-19T14:53:50.101387+0000","last_peered":"2024-02-19T14:53:50.101387+0000","last_clean":"2024-02-19T14:53:50.101387+0000","last_became_active":"2024-02-19T14:53:37.696981+0000","last_became_peered":"2024-02-19T14:53:37.696981+0000","last_unstale":"2024-02-19T14:53:50.101387+0000","last_undegraded":"2024-02-19T14:53:50.101387+0000","last_fullsized":"2024-02-19T14:53:50.101387+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:36.026355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"50'2","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738418+0000","last_change":"2024-02-19T14:53:36.686084+0000","last_active":"2024-02-19T14:53:49.738418+0000","last_peered":"2024-02-19T14:53:49.738418+0000","last_clean":"2024-02-19T14:53:49.738418+0000","last_became_active":"2024-02-19T14:53:36.685765+0000","last_became_peered":"2024-02-19T14:53:36.685765+0000","last_unstale":"2024-02-19T14:53:49.738418+0000","last_undegraded":"2024-02-19T14:53:49.738418+0000","last_fullsized":"2024-02-19T14:53:49.738418+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101840+0000","last_change":"2024-02-19T14:53:48.745926+0000","last_active":"2024-02-19T14:53:50.101840+0000","last_peered":"2024-02-19T14:53:50.101840+0000","last_clean":"2024-02-19T14:53:50.101840+0000","last_became_active":"2024-02-19T14:53:48.742589+0000","last_became_peered":"2024-02-19T14:53:48.742589+0000","last_unstale":"2024-02-19T14:53:50.101840+0000","last_undegraded":"2024-02-19T14:53:50.101840+0000","last_fullsized":"2024-02-19T14:53:50.101840+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:00.458513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102356+0000","last_change":"2024-02-19T14:53:47.745151+0000","last_active":"2024-02-19T14:53:50.102356+0000","last_peered":"2024-02-19T14:53:50.102356+0000","last_clean":"2024-02-19T14:53:50.102356+0000","last_became_active":"2024-02-19T14:53:47.744887+0000","last_became_peered":"2024-02-19T14:53:47.744887+0000","last_unstale":"2024-02-19T14:53:50.102356+0000","last_undegraded":"2024-02-19T14:53:50.102356+0000","last_fullsized":"2024-02-19T14:53:50.102356+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:48.402221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738692+0000","last_change":"2024-02-19T14:53:37.704748+0000","last_active":"2024-02-19T14:53:49.738692+0000","last_peered":"2024-02-19T14:53:49.738692+0000","last_clean":"2024-02-19T14:53:49.738692+0000","last_became_active":"2024-02-19T14:53:37.704409+0000","last_became_peered":"2024-02-19T14:53:37.704409+0000","last_unstale":"2024-02-19T14:53:49.738692+0000","last_undegraded":"2024-02-19T14:53:49.738692+0000","last_fullsized":"2024-02-19T14:53:49.738692+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:19.396678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"50'1","reported_seq":27,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103314+0000","last_change":"2024-02-19T14:53:36.695047+0000","last_active":"2024-02-19T14:53:50.103314+0000","last_peered":"2024-02-19T14:53:50.103314+0000","last_clean":"2024-02-19T14:53:50.103314+0000","last_became_active":"2024-02-19T14:53:36.694750+0000","last_became_peered":"2024-02-19T14:53:36.694750+0000","last_unstale":"2024-02-19T14:53:50.103314+0000","last_undegraded":"2024-02-19T14:53:50.103314+0000","last_fullsized":"2024-02-19T14:53:50.103314+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:46.508186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739346+0000","last_change":"2024-02-19T14:53:48.740019+0000","last_active":"2024-02-19T14:53:49.739346+0000","last_peered":"2024-02-19T14:53:49.739346+0000","last_clean":"2024-02-19T14:53:49.739346+0000","last_became_active":"2024-02-19T14:53:48.739241+0000","last_became_peered":"2024-02-19T14:53:48.739241+0000","last_unstale":"2024-02-19T14:53:49.739346+0000","last_undegraded":"2024-02-19T14:53:49.739346+0000","last_fullsized":"2024-02-19T14:53:49.739346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:03.659973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737001+0000","last_change":"2024-02-19T14:53:47.743381+0000","last_active":"2024-02-19T14:53:49.737001+0000","last_peered":"2024-02-19T14:53:49.737001+0000","last_clean":"2024-02-19T14:53:49.737001+0000","last_became_active":"2024-02-19T14:53:47.743135+0000","last_became_peered":"2024-02-19T14:53:47.743135+0000","last_unstale":"2024-02-19T14:53:49.737001+0000","last_undegraded":"2024-02-19T14:53:49.737001+0000","last_fullsized":"2024-02-19T14:53:49.737001+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:27.722851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102152+0000","last_change":"2024-02-19T14:53:37.692933+0000","last_active":"2024-02-19T14:53:50.102152+0000","last_peered":"2024-02-19T14:53:50.102152+0000","last_clean":"2024-02-19T14:53:50.102152+0000","last_became_active":"2024-02-19T14:53:37.692669+0000","last_became_peered":"2024-02-19T14:53:37.692669+0000","last_unstale":"2024-02-19T14:53:50.102152+0000","last_undegraded":"2024-02-19T14:53:50.102152+0000","last_fullsized":"2024-02-19T14:53:50.102152+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"50'1","reported_seq":27,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737812+0000","last_change":"2024-02-19T14:53:36.688713+0000","last_active":"2024-02-19T14:53:49.737812+0000","last_peered":"2024-02-19T14:53:49.737812+0000","last_clean":"2024-02-19T14:53:49.737812+0000","last_became_active":"2024-02-19T14:53:36.688458+0000","last_became_peered":"2024-02-19T14:53:36.688458+0000","last_unstale":"2024-02-19T14:53:49.737812+0000","last_undegraded":"2024-02-19T14:53:49.737812+0000","last_fullsized":"2024-02-19T14:53:49.737812+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:01.473061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736929+0000","last_change":"2024-02-19T14:53:48.734080+0000","last_active":"2024-02-19T14:53:49.736929+0000","last_peered":"2024-02-19T14:53:49.736929+0000","last_clean":"2024-02-19T14:53:49.736929+0000","last_became_active":"2024-02-19T14:53:48.733815+0000","last_became_peered":"2024-02-19T14:53:48.733815+0000","last_unstale":"2024-02-19T14:53:49.736929+0000","last_undegraded":"2024-02-19T14:53:49.736929+0000","last_fullsized":"2024-02-19T14:53:49.736929+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:58.706427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102546+0000","last_change":"2024-02-19T14:53:47.744368+0000","last_active":"2024-02-19T14:53:50.102546+0000","last_peered":"2024-02-19T14:53:50.102546+0000","last_clean":"2024-02-19T14:53:50.102546+0000","last_became_active":"2024-02-19T14:53:47.743379+0000","last_became_peered":"2024-02-19T14:53:47.743379+0000","last_unstale":"2024-02-19T14:53:50.102546+0000","last_undegraded":"2024-02-19T14:53:50.102546+0000","last_fullsized":"2024-02-19T14:53:50.102546+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:55.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100963+0000","last_change":"2024-02-19T14:53:37.682166+0000","last_active":"2024-02-19T14:53:50.100963+0000","last_peered":"2024-02-19T14:53:50.100963+0000","last_clean":"2024-02-19T14:53:50.100963+0000","last_became_active":"2024-02-19T14:53:37.681920+0000","last_became_peered":"2024-02-19T14:53:37.681920+0000","last_unstale":"2024-02-19T14:53:50.100963+0000","last_undegraded":"2024-02-19T14:53:50.100963+0000","last_fullsized":"2024-02-19T14:53:50.100963+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:41.307928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101080+0000","last_change":"2024-02-19T14:53:36.689130+0000","last_active":"2024-02-19T14:53:50.101080+0000","last_peered":"2024-02-19T14:53:50.101080+0000","last_clean":"2024-02-19T14:53:50.101080+0000","last_became_active":"2024-02-19T14:53:36.688861+0000","last_became_peered":"2024-02-19T14:53:36.688861+0000","last_unstale":"2024-02-19T14:53:50.101080+0000","last_undegraded":"2024-02-19T14:53:50.101080+0000","last_fullsized":"2024-02-19T14:53:50.101080+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101195+0000","last_change":"2024-02-19T14:53:48.742690+0000","last_active":"2024-02-19T14:53:50.101195+0000","last_peered":"2024-02-19T14:53:50.101195+0000","last_clean":"2024-02-19T14:53:50.101195+0000","last_became_active":"2024-02-19T14:53:48.742408+0000","last_became_peered":"2024-02-19T14:53:48.742408+0000","last_unstale":"2024-02-19T14:53:50.101195+0000","last_undegraded":"2024-02-19T14:53:50.101195+0000","last_fullsized":"2024-02-19T14:53:50.101195+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:40:24.050809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739177+0000","last_change":"2024-02-19T14:53:47.744422+0000","last_active":"2024-02-19T14:53:49.739177+0000","last_peered":"2024-02-19T14:53:49.739177+0000","last_clean":"2024-02-19T14:53:49.739177+0000","last_became_active":"2024-02-19T14:53:47.744122+0000","last_became_peered":"2024-02-19T14:53:47.744122+0000","last_unstale":"2024-02-19T14:53:49.739177+0000","last_undegraded":"2024-02-19T14:53:49.739177+0000","last_fullsized":"2024-02-19T14:53:49.739177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:43.658724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102650+0000","last_change":"2024-02-19T14:53:37.705832+0000","last_active":"2024-02-19T14:53:50.102650+0000","last_peered":"2024-02-19T14:53:50.102650+0000","last_clean":"2024-02-19T14:53:50.102650+0000","last_became_active":"2024-02-19T14:53:37.705526+0000","last_became_peered":"2024-02-19T14:53:37.705526+0000","last_unstale":"2024-02-19T14:53:50.102650+0000","last_undegraded":"2024-02-19T14:53:50.102650+0000","last_fullsized":"2024-02-19T14:53:50.102650+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:10.747232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103150+0000","last_change":"2024-02-19T14:53:36.688536+0000","last_active":"2024-02-19T14:53:50.103150+0000","last_peered":"2024-02-19T14:53:50.103150+0000","last_clean":"2024-02-19T14:53:50.103150+0000","last_became_active":"2024-02-19T14:53:36.688285+0000","last_became_peered":"2024-02-19T14:53:36.688285+0000","last_unstale":"2024-02-19T14:53:50.103150+0000","last_undegraded":"2024-02-19T14:53:50.103150+0000","last_fullsized":"2024-02-19T14:53:50.103150+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101332+0000","last_change":"2024-02-19T14:53:48.746581+0000","last_active":"2024-02-19T14:53:50.101332+0000","last_peered":"2024-02-19T14:53:50.101332+0000","last_clean":"2024-02-19T14:53:50.101332+0000","last_became_active":"2024-02-19T14:53:48.746260+0000","last_became_peered":"2024-02-19T14:53:48.746260+0000","last_unstale":"2024-02-19T14:53:50.101332+0000","last_undegraded":"2024-02-19T14:53:50.101332+0000","last_fullsized":"2024-02-19T14:53:50.101332+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:47.079107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100737+0000","last_change":"2024-02-19T14:53:47.745353+0000","last_active":"2024-02-19T14:53:50.100737+0000","last_peered":"2024-02-19T14:53:50.100737+0000","last_clean":"2024-02-19T14:53:50.100737+0000","last_became_active":"2024-02-19T14:53:47.745008+0000","last_became_peered":"2024-02-19T14:53:47.745008+0000","last_unstale":"2024-02-19T14:53:50.100737+0000","last_undegraded":"2024-02-19T14:53:50.100737+0000","last_fullsized":"2024-02-19T14:53:50.100737+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:32:10.841556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737533+0000","last_change":"2024-02-19T14:53:37.691622+0000","last_active":"2024-02-19T14:53:49.737533+0000","last_peered":"2024-02-19T14:53:49.737533+0000","last_clean":"2024-02-19T14:53:49.737533+0000","last_became_active":"2024-02-19T14:53:37.691352+0000","last_became_peered":"2024-02-19T14:53:37.691352+0000","last_unstale":"2024-02-19T14:53:49.737533+0000","last_undegraded":"2024-02-19T14:53:49.737533+0000","last_fullsized":"2024-02-19T14:53:49.737533+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:44.072344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102669+0000","last_change":"2024-02-19T14:53:36.684443+0000","last_active":"2024-02-19T14:53:50.102669+0000","last_peered":"2024-02-19T14:53:50.102669+0000","last_clean":"2024-02-19T14:53:50.102669+0000","last_became_active":"2024-02-19T14:53:36.684124+0000","last_became_peered":"2024-02-19T14:53:36.684124+0000","last_unstale":"2024-02-19T14:53:50.102669+0000","last_undegraded":"2024-02-19T14:53:50.102669+0000","last_fullsized":"2024-02-19T14:53:50.102669+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:05:06.103049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101611+0000","last_change":"2024-02-19T14:53:48.748973+0000","last_active":"2024-02-19T14:53:50.101611+0000","last_peered":"2024-02-19T14:53:50.101611+0000","last_clean":"2024-02-19T14:53:50.101611+0000","last_became_active":"2024-02-19T14:53:48.748712+0000","last_became_peered":"2024-02-19T14:53:48.748712+0000","last_unstale":"2024-02-19T14:53:50.101611+0000","last_undegraded":"2024-02-19T14:53:50.101611+0000","last_fullsized":"2024-02-19T14:53:50.101611+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:12.780355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100677+0000","last_change":"2024-02-19T14:53:47.742114+0000","last_active":"2024-02-19T14:53:50.100677+0000","last_peered":"2024-02-19T14:53:50.100677+0000","last_clean":"2024-02-19T14:53:50.100677+0000","last_became_active":"2024-02-19T14:53:47.741840+0000","last_became_peered":"2024-02-19T14:53:47.741840+0000","last_unstale":"2024-02-19T14:53:50.100677+0000","last_undegraded":"2024-02-19T14:53:50.100677+0000","last_fullsized":"2024-02-19T14:53:50.100677+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:22.459296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738876+0000","last_change":"2024-02-19T14:53:37.688206+0000","last_active":"2024-02-19T14:53:49.738876+0000","last_peered":"2024-02-19T14:53:49.738876+0000","last_clean":"2024-02-19T14:53:49.738876+0000","last_became_active":"2024-02-19T14:53:37.687546+0000","last_became_peered":"2024-02-19T14:53:37.687546+0000","last_unstale":"2024-02-19T14:53:49.738876+0000","last_undegraded":"2024-02-19T14:53:49.738876+0000","last_fullsized":"2024-02-19T14:53:49.738876+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739479+0000","last_change":"2024-02-19T14:53:36.690460+0000","last_active":"2024-02-19T14:53:49.739479+0000","last_peered":"2024-02-19T14:53:49.739479+0000","last_clean":"2024-02-19T14:53:49.739479+0000","last_became_active":"2024-02-19T14:53:36.690155+0000","last_became_peered":"2024-02-19T14:53:36.690155+0000","last_unstale":"2024-02-19T14:53:49.739479+0000","last_undegraded":"2024-02-19T14:53:49.739479+0000","last_fullsized":"2024-02-19T14:53:49.739479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.740289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100563+0000","last_change":"2024-02-19T14:53:48.733671+0000","last_active":"2024-02-19T14:53:50.100563+0000","last_peered":"2024-02-19T14:53:50.100563+0000","last_clean":"2024-02-19T14:53:50.100563+0000","last_became_active":"2024-02-19T14:53:48.733332+0000","last_became_peered":"2024-02-19T14:53:48.733332+0000","last_unstale":"2024-02-19T14:53:50.100563+0000","last_undegraded":"2024-02-19T14:53:50.100563+0000","last_fullsized":"2024-02-19T14:53:50.100563+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:36.692152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101289+0000","last_change":"2024-02-19T14:53:47.734753+0000","last_active":"2024-02-19T14:53:50.101289+0000","last_peered":"2024-02-19T14:53:50.101289+0000","last_clean":"2024-02-19T14:53:50.101289+0000","last_became_active":"2024-02-19T14:53:47.734515+0000","last_became_peered":"2024-02-19T14:53:47.734515+0000","last_unstale":"2024-02-19T14:53:50.101289+0000","last_undegraded":"2024-02-19T14:53:50.101289+0000","last_fullsized":"2024-02-19T14:53:50.101289+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:06.402501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101236+0000","last_change":"2024-02-19T14:53:37.703269+0000","last_active":"2024-02-19T14:53:50.101236+0000","last_peered":"2024-02-19T14:53:50.101236+0000","last_clean":"2024-02-19T14:53:50.101236+0000","last_became_active":"2024-02-19T14:53:37.702986+0000","last_became_peered":"2024-02-19T14:53:37.702986+0000","last_unstale":"2024-02-19T14:53:50.101236+0000","last_undegraded":"2024-02-19T14:53:50.101236+0000","last_fullsized":"2024-02-19T14:53:50.101236+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:00.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"50'3","reported_seq":29,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102261+0000","last_change":"2024-02-19T14:53:36.687247+0000","last_active":"2024-02-19T14:53:50.102261+0000","last_peered":"2024-02-19T14:53:50.102261+0000","last_clean":"2024-02-19T14:53:50.102261+0000","last_became_active":"2024-02-19T14:53:36.686985+0000","last_became_peered":"2024-02-19T14:53:36.686985+0000","last_unstale":"2024-02-19T14:53:50.102261+0000","last_undegraded":"2024-02-19T14:53:50.102261+0000","last_fullsized":"2024-02-19T14:53:50.102261+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:50:55.116142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736225+0000","last_change":"2024-02-19T14:53:48.733001+0000","last_active":"2024-02-19T14:53:49.736225+0000","last_peered":"2024-02-19T14:53:49.736225+0000","last_clean":"2024-02-19T14:53:49.736225+0000","last_became_active":"2024-02-19T14:53:48.732743+0000","last_became_peered":"2024-02-19T14:53:48.732743+0000","last_unstale":"2024-02-19T14:53:49.736225+0000","last_undegraded":"2024-02-19T14:53:49.736225+0000","last_fullsized":"2024-02-19T14:53:49.736225+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:47.079107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101140+0000","last_change":"2024-02-19T14:53:47.735759+0000","last_active":"2024-02-19T14:53:50.101140+0000","last_peered":"2024-02-19T14:53:50.101140+0000","last_clean":"2024-02-19T14:53:50.101140+0000","last_became_active":"2024-02-19T14:53:47.735503+0000","last_became_peered":"2024-02-19T14:53:47.735503+0000","last_unstale":"2024-02-19T14:53:50.101140+0000","last_undegraded":"2024-02-19T14:53:50.101140+0000","last_fullsized":"2024-02-19T14:53:50.101140+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:07.689250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737925+0000","last_change":"2024-02-19T14:53:37.696249+0000","last_active":"2024-02-19T14:53:49.737925+0000","last_peered":"2024-02-19T14:53:49.737925+0000","last_clean":"2024-02-19T14:53:49.737925+0000","last_became_active":"2024-02-19T14:53:37.695979+0000","last_became_peered":"2024-02-19T14:53:37.695979+0000","last_unstale":"2024-02-19T14:53:49.737925+0000","last_undegraded":"2024-02-19T14:53:49.737925+0000","last_fullsized":"2024-02-19T14:53:49.737925+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:18.437623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"50'6","reported_seq":32,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102871+0000","last_change":"2024-02-19T14:53:36.684685+0000","last_active":"2024-02-19T14:53:50.102871+0000","last_peered":"2024-02-19T14:53:50.102871+0000","last_clean":"2024-02-19T14:53:50.102871+0000","last_became_active":"2024-02-19T14:53:36.683202+0000","last_became_peered":"2024-02-19T14:53:36.683202+0000","last_unstale":"2024-02-19T14:53:50.102871+0000","last_undegraded":"2024-02-19T14:53:50.102871+0000","last_fullsized":"2024-02-19T14:53:50.102871+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:19:30.130369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102622+0000","last_change":"2024-02-19T14:53:48.736972+0000","last_active":"2024-02-19T14:53:50.102622+0000","last_peered":"2024-02-19T14:53:50.102622+0000","last_clean":"2024-02-19T14:53:50.102622+0000","last_became_active":"2024-02-19T14:53:48.736237+0000","last_became_peered":"2024-02-19T14:53:48.736237+0000","last_unstale":"2024-02-19T14:53:50.102622+0000","last_undegraded":"2024-02-19T14:53:50.102622+0000","last_fullsized":"2024-02-19T14:53:50.102622+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:52.856768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103229+0000","last_change":"2024-02-19T14:53:47.742801+0000","last_active":"2024-02-19T14:53:50.103229+0000","last_peered":"2024-02-19T14:53:50.103229+0000","last_clean":"2024-02-19T14:53:50.103229+0000","last_became_active":"2024-02-19T14:53:47.742534+0000","last_became_peered":"2024-02-19T14:53:47.742534+0000","last_unstale":"2024-02-19T14:53:50.103229+0000","last_undegraded":"2024-02-19T14:53:50.103229+0000","last_fullsized":"2024-02-19T14:53:50.103229+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:35.189687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738186+0000","last_change":"2024-02-19T14:53:37.688201+0000","last_active":"2024-02-19T14:53:49.738186+0000","last_peered":"2024-02-19T14:53:49.738186+0000","last_clean":"2024-02-19T14:53:49.738186+0000","last_became_active":"2024-02-19T14:53:37.687920+0000","last_became_peered":"2024-02-19T14:53:37.687920+0000","last_unstale":"2024-02-19T14:53:49.738186+0000","last_undegraded":"2024-02-19T14:53:49.738186+0000","last_fullsized":"2024-02-19T14:53:49.738186+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"50'2","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103361+0000","last_change":"2024-02-19T14:53:36.689251+0000","last_active":"2024-02-19T14:53:50.103361+0000","last_peered":"2024-02-19T14:53:50.103361+0000","last_clean":"2024-02-19T14:53:50.103361+0000","last_became_active":"2024-02-19T14:53:36.688967+0000","last_became_peered":"2024-02-19T14:53:36.688967+0000","last_unstale":"2024-02-19T14:53:50.103361+0000","last_undegraded":"2024-02-19T14:53:50.103361+0000","last_fullsized":"2024-02-19T14:53:50.103361+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102324+0000","last_change":"2024-02-19T14:53:48.757981+0000","last_active":"2024-02-19T14:53:50.102324+0000","last_peered":"2024-02-19T14:53:50.102324+0000","last_clean":"2024-02-19T14:53:50.102324+0000","last_became_active":"2024-02-19T14:53:48.757604+0000","last_became_peered":"2024-02-19T14:53:48.757604+0000","last_unstale":"2024-02-19T14:53:50.102324+0000","last_undegraded":"2024-02-19T14:53:50.102324+0000","last_fullsized":"2024-02-19T14:53:50.102324+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:35.702079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101533+0000","last_change":"2024-02-19T14:53:47.744212+0000","last_active":"2024-02-19T14:53:50.101533+0000","last_peered":"2024-02-19T14:53:50.101533+0000","last_clean":"2024-02-19T14:53:50.101533+0000","last_became_active":"2024-02-19T14:53:47.743916+0000","last_became_peered":"2024-02-19T14:53:47.743916+0000","last_unstale":"2024-02-19T14:53:50.101533+0000","last_undegraded":"2024-02-19T14:53:50.101533+0000","last_fullsized":"2024-02-19T14:53:50.101533+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:29.660730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103519+0000","last_change":"2024-02-19T14:53:37.696225+0000","last_active":"2024-02-19T14:53:50.103519+0000","last_peered":"2024-02-19T14:53:50.103519+0000","last_clean":"2024-02-19T14:53:50.103519+0000","last_became_active":"2024-02-19T14:53:37.695962+0000","last_became_peered":"2024-02-19T14:53:37.695962+0000","last_unstale":"2024-02-19T14:53:50.103519+0000","last_undegraded":"2024-02-19T14:53:50.103519+0000","last_fullsized":"2024-02-19T14:53:50.103519+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:52.607222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101369+0000","last_change":"2024-02-19T14:53:36.693462+0000","last_active":"2024-02-19T14:53:50.101369+0000","last_peered":"2024-02-19T14:53:50.101369+0000","last_clean":"2024-02-19T14:53:50.101369+0000","last_became_active":"2024-02-19T14:53:36.693160+0000","last_became_peered":"2024-02-19T14:53:36.693160+0000","last_unstale":"2024-02-19T14:53:50.101369+0000","last_undegraded":"2024-02-19T14:53:50.101369+0000","last_fullsized":"2024-02-19T14:53:50.101369+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:24.634583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101046+0000","last_change":"2024-02-19T14:53:48.752738+0000","last_active":"2024-02-19T14:53:50.101046+0000","last_peered":"2024-02-19T14:53:50.101046+0000","last_clean":"2024-02-19T14:53:50.101046+0000","last_became_active":"2024-02-19T14:53:48.750930+0000","last_became_peered":"2024-02-19T14:53:48.750930+0000","last_unstale":"2024-02-19T14:53:50.101046+0000","last_undegraded":"2024-02-19T14:53:50.101046+0000","last_fullsized":"2024-02-19T14:53:50.101046+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:12.692496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737331+0000","last_change":"2024-02-19T14:53:47.741970+0000","last_active":"2024-02-19T14:53:49.737331+0000","last_peered":"2024-02-19T14:53:49.737331+0000","last_clean":"2024-02-19T14:53:49.737331+0000","last_became_active":"2024-02-19T14:53:47.741707+0000","last_became_peered":"2024-02-19T14:53:47.741707+0000","last_unstale":"2024-02-19T14:53:49.737331+0000","last_undegraded":"2024-02-19T14:53:49.737331+0000","last_fullsized":"2024-02-19T14:53:49.737331+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:02.790030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100869+0000","last_change":"2024-02-19T14:53:37.686072+0000","last_active":"2024-02-19T14:53:50.100869+0000","last_peered":"2024-02-19T14:53:50.100869+0000","last_clean":"2024-02-19T14:53:50.100869+0000","last_became_active":"2024-02-19T14:53:37.685840+0000","last_became_peered":"2024-02-19T14:53:37.685840+0000","last_unstale":"2024-02-19T14:53:50.100869+0000","last_undegraded":"2024-02-19T14:53:50.100869+0000","last_fullsized":"2024-02-19T14:53:50.100869+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:16.569558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102679+0000","last_change":"2024-02-19T14:53:36.693777+0000","last_active":"2024-02-19T14:53:50.102679+0000","last_peered":"2024-02-19T14:53:50.102679+0000","last_clean":"2024-02-19T14:53:50.102679+0000","last_became_active":"2024-02-19T14:53:36.693458+0000","last_became_peered":"2024-02-19T14:53:36.693458+0000","last_unstale":"2024-02-19T14:53:50.102679+0000","last_undegraded":"2024-02-19T14:53:50.102679+0000","last_fullsized":"2024-02-19T14:53:50.102679+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:56.472347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739155+0000","last_change":"2024-02-19T14:53:48.747038+0000","last_active":"2024-02-19T14:53:49.739155+0000","last_peered":"2024-02-19T14:53:49.739155+0000","last_clean":"2024-02-19T14:53:49.739155+0000","last_became_active":"2024-02-19T14:53:48.746463+0000","last_became_peered":"2024-02-19T14:53:48.746463+0000","last_unstale":"2024-02-19T14:53:49.739155+0000","last_undegraded":"2024-02-19T14:53:49.739155+0000","last_fullsized":"2024-02-19T14:53:49.739155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:28.727489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102765+0000","last_change":"2024-02-19T14:53:47.742140+0000","last_active":"2024-02-19T14:53:50.102765+0000","last_peered":"2024-02-19T14:53:50.102765+0000","last_clean":"2024-02-19T14:53:50.102765+0000","last_became_active":"2024-02-19T14:53:47.741838+0000","last_became_peered":"2024-02-19T14:53:47.741838+0000","last_unstale":"2024-02-19T14:53:50.102765+0000","last_undegraded":"2024-02-19T14:53:50.102765+0000","last_fullsized":"2024-02-19T14:53:50.102765+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:28:25.067735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101823+0000","last_change":"2024-02-19T14:53:41.706063+0000","last_active":"2024-02-19T14:53:50.101823+0000","last_peered":"2024-02-19T14:53:50.101823+0000","last_clean":"2024-02-19T14:53:50.101823+0000","last_became_active":"2024-02-19T14:53:41.705766+0000","last_became_peered":"2024-02-19T14:53:41.705766+0000","last_unstale":"2024-02-19T14:53:50.101823+0000","last_undegraded":"2024-02-19T14:53:50.101823+0000","last_fullsized":"2024-02-19T14:53:50.101823+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:35.687364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.10","version":"50'8","reported_seq":34,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101900+0000","last_change":"2024-02-19T14:53:36.683829+0000","last_active":"2024-02-19T14:53:50.101900+0000","last_peered":"2024-02-19T14:53:50.101900+0000","last_clean":"2024-02-19T14:53:50.101900+0000","last_became_active":"2024-02-19T14:53:36.682854+0000","last_became_peered":"2024-02-19T14:53:36.682854+0000","last_unstale":"2024-02-19T14:53:50.101900+0000","last_undegraded":"2024-02-19T14:53:50.101900+0000","last_fullsized":"2024-02-19T14:53:50.101900+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:15.320022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736276+0000","last_change":"2024-02-19T14:53:48.741431+0000","last_active":"2024-02-19T14:53:49.736276+0000","last_peered":"2024-02-19T14:53:49.736276+0000","last_clean":"2024-02-19T14:53:49.736276+0000","last_became_active":"2024-02-19T14:53:48.741151+0000","last_became_peered":"2024-02-19T14:53:48.741151+0000","last_unstale":"2024-02-19T14:53:49.736276+0000","last_undegraded":"2024-02-19T14:53:49.736276+0000","last_fullsized":"2024-02-19T14:53:49.736276+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:01.759980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100830+0000","last_change":"2024-02-19T14:53:47.742196+0000","last_active":"2024-02-19T14:53:50.100830+0000","last_peered":"2024-02-19T14:53:50.100830+0000","last_clean":"2024-02-19T14:53:50.100830+0000","last_became_active":"2024-02-19T14:53:47.741852+0000","last_became_peered":"2024-02-19T14:53:47.741852+0000","last_unstale":"2024-02-19T14:53:50.100830+0000","last_undegraded":"2024-02-19T14:53:50.100830+0000","last_fullsized":"2024-02-19T14:53:50.100830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:09.551790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101813+0000","last_change":"2024-02-19T14:53:37.691397+0000","last_active":"2024-02-19T14:53:50.101813+0000","last_peered":"2024-02-19T14:53:50.101813+0000","last_clean":"2024-02-19T14:53:50.101813+0000","last_became_active":"2024-02-19T14:53:37.691105+0000","last_became_peered":"2024-02-19T14:53:37.691105+0000","last_unstale":"2024-02-19T14:53:50.101813+0000","last_undegraded":"2024-02-19T14:53:50.101813+0000","last_fullsized":"2024-02-19T14:53:50.101813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:33.837063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"50'1","reported_seq":27,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102705+0000","last_change":"2024-02-19T14:53:36.689862+0000","last_active":"2024-02-19T14:53:50.102705+0000","last_peered":"2024-02-19T14:53:50.102705+0000","last_clean":"2024-02-19T14:53:50.102705+0000","last_became_active":"2024-02-19T14:53:36.689587+0000","last_became_peered":"2024-02-19T14:53:36.689587+0000","last_unstale":"2024-02-19T14:53:50.102705+0000","last_undegraded":"2024-02-19T14:53:50.102705+0000","last_fullsized":"2024-02-19T14:53:50.102705+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:16.754512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100837+0000","last_change":"2024-02-19T14:53:48.742851+0000","last_active":"2024-02-19T14:53:50.100837+0000","last_peered":"2024-02-19T14:53:50.100837+0000","last_clean":"2024-02-19T14:53:50.100837+0000","last_became_active":"2024-02-19T14:53:48.742613+0000","last_became_peered":"2024-02-19T14:53:48.742613+0000","last_unstale":"2024-02-19T14:53:50.100837+0000","last_undegraded":"2024-02-19T14:53:50.100837+0000","last_fullsized":"2024-02-19T14:53:50.100837+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:55.125096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736178+0000","last_change":"2024-02-19T14:53:47.745519+0000","last_active":"2024-02-19T14:53:49.736178+0000","last_peered":"2024-02-19T14:53:49.736178+0000","last_clean":"2024-02-19T14:53:49.736178+0000","last_became_active":"2024-02-19T14:53:47.745263+0000","last_became_peered":"2024-02-19T14:53:47.745263+0000","last_unstale":"2024-02-19T14:53:49.736178+0000","last_undegraded":"2024-02-19T14:53:49.736178+0000","last_fullsized":"2024-02-19T14:53:49.736178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:13.814772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736268+0000","last_change":"2024-02-19T14:53:37.696725+0000","last_active":"2024-02-19T14:53:49.736268+0000","last_peered":"2024-02-19T14:53:49.736268+0000","last_clean":"2024-02-19T14:53:49.736268+0000","last_became_active":"2024-02-19T14:53:37.696468+0000","last_became_peered":"2024-02-19T14:53:37.696468+0000","last_unstale":"2024-02-19T14:53:49.736268+0000","last_undegraded":"2024-02-19T14:53:49.736268+0000","last_fullsized":"2024-02-19T14:53:49.736268+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:23.451649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"50'2","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101215+0000","last_change":"2024-02-19T14:53:36.682544+0000","last_active":"2024-02-19T14:53:50.101215+0000","last_peered":"2024-02-19T14:53:50.101215+0000","last_clean":"2024-02-19T14:53:50.101215+0000","last_became_active":"2024-02-19T14:53:36.682317+0000","last_became_peered":"2024-02-19T14:53:36.682317+0000","last_unstale":"2024-02-19T14:53:50.101215+0000","last_undegraded":"2024-02-19T14:53:50.101215+0000","last_fullsized":"2024-02-19T14:53:50.101215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:29.282617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100735+0000","last_change":"2024-02-19T14:53:48.745439+0000","last_active":"2024-02-19T14:53:50.100735+0000","last_peered":"2024-02-19T14:53:50.100735+0000","last_clean":"2024-02-19T14:53:50.100735+0000","last_became_active":"2024-02-19T14:53:48.742371+0000","last_became_peered":"2024-02-19T14:53:48.742371+0000","last_unstale":"2024-02-19T14:53:50.100735+0000","last_undegraded":"2024-02-19T14:53:50.100735+0000","last_fullsized":"2024-02-19T14:53:50.100735+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:51.879475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101775+0000","last_change":"2024-02-19T14:53:47.736640+0000","last_active":"2024-02-19T14:53:50.101775+0000","last_peered":"2024-02-19T14:53:50.101775+0000","last_clean":"2024-02-19T14:53:50.101775+0000","last_became_active":"2024-02-19T14:53:47.735150+0000","last_became_peered":"2024-02-19T14:53:47.735150+0000","last_unstale":"2024-02-19T14:53:50.101775+0000","last_undegraded":"2024-02-19T14:53:50.101775+0000","last_fullsized":"2024-02-19T14:53:50.101775+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:59.453875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737706+0000","last_change":"2024-02-19T14:53:37.704759+0000","last_active":"2024-02-19T14:53:49.737706+0000","last_peered":"2024-02-19T14:53:49.737706+0000","last_clean":"2024-02-19T14:53:49.737706+0000","last_became_active":"2024-02-19T14:53:37.704480+0000","last_became_peered":"2024-02-19T14:53:37.704480+0000","last_unstale":"2024-02-19T14:53:49.737706+0000","last_undegraded":"2024-02-19T14:53:49.737706+0000","last_fullsized":"2024-02-19T14:53:49.737706+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:47.513003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"50'1","reported_seq":27,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102430+0000","last_change":"2024-02-19T14:53:36.688651+0000","last_active":"2024-02-19T14:53:50.102430+0000","last_peered":"2024-02-19T14:53:50.102430+0000","last_clean":"2024-02-19T14:53:50.102430+0000","last_became_active":"2024-02-19T14:53:36.688346+0000","last_became_peered":"2024-02-19T14:53:36.688346+0000","last_unstale":"2024-02-19T14:53:50.102430+0000","last_undegraded":"2024-02-19T14:53:50.102430+0000","last_fullsized":"2024-02-19T14:53:50.102430+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:43.067527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738127+0000","last_change":"2024-02-19T14:53:48.745274+0000","last_active":"2024-02-19T14:53:49.738127+0000","last_peered":"2024-02-19T14:53:49.738127+0000","last_clean":"2024-02-19T14:53:49.738127+0000","last_became_active":"2024-02-19T14:53:48.744753+0000","last_became_peered":"2024-02-19T14:53:48.744753+0000","last_unstale":"2024-02-19T14:53:49.738127+0000","last_undegraded":"2024-02-19T14:53:49.738127+0000","last_fullsized":"2024-02-19T14:53:49.738127+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:31.544857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101235+0000","last_change":"2024-02-19T14:53:47.730313+0000","last_active":"2024-02-19T14:53:50.101235+0000","last_peered":"2024-02-19T14:53:50.101235+0000","last_clean":"2024-02-19T14:53:50.101235+0000","last_became_active":"2024-02-19T14:53:47.730053+0000","last_became_peered":"2024-02-19T14:53:47.730053+0000","last_unstale":"2024-02-19T14:53:50.101235+0000","last_undegraded":"2024-02-19T14:53:50.101235+0000","last_fullsized":"2024-02-19T14:53:50.101235+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:04.402894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736623+0000","last_change":"2024-02-19T14:53:37.688877+0000","last_active":"2024-02-19T14:53:49.736623+0000","last_peered":"2024-02-19T14:53:49.736623+0000","last_clean":"2024-02-19T14:53:49.736623+0000","last_became_active":"2024-02-19T14:53:37.688582+0000","last_became_peered":"2024-02-19T14:53:37.688582+0000","last_unstale":"2024-02-19T14:53:49.736623+0000","last_undegraded":"2024-02-19T14:53:49.736623+0000","last_fullsized":"2024-02-19T14:53:49.736623+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101498+0000","last_change":"2024-02-19T14:53:36.688830+0000","last_active":"2024-02-19T14:53:50.101498+0000","last_peered":"2024-02-19T14:53:50.101498+0000","last_clean":"2024-02-19T14:53:50.101498+0000","last_became_active":"2024-02-19T14:53:36.688544+0000","last_became_peered":"2024-02-19T14:53:36.688544+0000","last_unstale":"2024-02-19T14:53:50.101498+0000","last_undegraded":"2024-02-19T14:53:50.101498+0000","last_fullsized":"2024-02-19T14:53:50.101498+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738662+0000","last_change":"2024-02-19T14:53:48.751967+0000","last_active":"2024-02-19T14:53:49.738662+0000","last_peered":"2024-02-19T14:53:49.738662+0000","last_clean":"2024-02-19T14:53:49.738662+0000","last_became_active":"2024-02-19T14:53:48.751694+0000","last_became_peered":"2024-02-19T14:53:48.751694+0000","last_unstale":"2024-02-19T14:53:49.738662+0000","last_undegraded":"2024-02-19T14:53:49.738662+0000","last_fullsized":"2024-02-19T14:53:49.738662+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:52.856768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103317+0000","last_change":"2024-02-19T14:53:47.742841+0000","last_active":"2024-02-19T14:53:50.103317+0000","last_peered":"2024-02-19T14:53:50.103317+0000","last_clean":"2024-02-19T14:53:50.103317+0000","last_became_active":"2024-02-19T14:53:47.742581+0000","last_became_peered":"2024-02-19T14:53:47.742581+0000","last_unstale":"2024-02-19T14:53:50.103317+0000","last_undegraded":"2024-02-19T14:53:50.103317+0000","last_fullsized":"2024-02-19T14:53:50.103317+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:50.984645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738306+0000","last_change":"2024-02-19T14:53:37.688744+0000","last_active":"2024-02-19T14:53:49.738306+0000","last_peered":"2024-02-19T14:53:49.738306+0000","last_clean":"2024-02-19T14:53:49.738306+0000","last_became_active":"2024-02-19T14:53:37.688486+0000","last_became_peered":"2024-02-19T14:53:37.688486+0000","last_unstale":"2024-02-19T14:53:49.738306+0000","last_undegraded":"2024-02-19T14:53:49.738306+0000","last_fullsized":"2024-02-19T14:53:49.738306+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:17.674737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"50'2","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736709+0000","last_change":"2024-02-19T14:53:36.681337+0000","last_active":"2024-02-19T14:53:49.736709+0000","last_peered":"2024-02-19T14:53:49.736709+0000","last_clean":"2024-02-19T14:53:49.736709+0000","last_became_active":"2024-02-19T14:53:36.681077+0000","last_became_peered":"2024-02-19T14:53:36.681077+0000","last_unstale":"2024-02-19T14:53:49.736709+0000","last_undegraded":"2024-02-19T14:53:49.736709+0000","last_fullsized":"2024-02-19T14:53:49.736709+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:12.230865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102069+0000","last_change":"2024-02-19T14:53:48.746082+0000","last_active":"2024-02-19T14:53:50.102069+0000","last_peered":"2024-02-19T14:53:50.102069+0000","last_clean":"2024-02-19T14:53:50.102069+0000","last_became_active":"2024-02-19T14:53:48.745700+0000","last_became_peered":"2024-02-19T14:53:48.745700+0000","last_unstale":"2024-02-19T14:53:50.102069+0000","last_undegraded":"2024-02-19T14:53:50.102069+0000","last_fullsized":"2024-02-19T14:53:50.102069+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:49.406859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737024+0000","last_change":"2024-02-19T14:53:47.743491+0000","last_active":"2024-02-19T14:53:49.737024+0000","last_peered":"2024-02-19T14:53:49.737024+0000","last_clean":"2024-02-19T14:53:49.737024+0000","last_became_active":"2024-02-19T14:53:47.743209+0000","last_became_peered":"2024-02-19T14:53:47.743209+0000","last_unstale":"2024-02-19T14:53:49.737024+0000","last_undegraded":"2024-02-19T14:53:49.737024+0000","last_fullsized":"2024-02-19T14:53:49.737024+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:52:02.655336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102164+0000","last_change":"2024-02-19T14:53:37.688559+0000","last_active":"2024-02-19T14:53:50.102164+0000","last_peered":"2024-02-19T14:53:50.102164+0000","last_clean":"2024-02-19T14:53:50.102164+0000","last_became_active":"2024-02-19T14:53:37.688239+0000","last_became_peered":"2024-02-19T14:53:37.688239+0000","last_unstale":"2024-02-19T14:53:50.102164+0000","last_undegraded":"2024-02-19T14:53:50.102164+0000","last_fullsized":"2024-02-19T14:53:50.102164+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:47:50.043589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101399+0000","last_change":"2024-02-19T14:53:36.684453+0000","last_active":"2024-02-19T14:53:50.101399+0000","last_peered":"2024-02-19T14:53:50.101399+0000","last_clean":"2024-02-19T14:53:50.101399+0000","last_became_active":"2024-02-19T14:53:36.682802+0000","last_became_peered":"2024-02-19T14:53:36.682802+0000","last_unstale":"2024-02-19T14:53:50.101399+0000","last_undegraded":"2024-02-19T14:53:50.101399+0000","last_fullsized":"2024-02-19T14:53:50.101399+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:10:34.682547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101450+0000","last_change":"2024-02-19T14:53:48.745426+0000","last_active":"2024-02-19T14:53:50.101450+0000","last_peered":"2024-02-19T14:53:50.101450+0000","last_clean":"2024-02-19T14:53:50.101450+0000","last_became_active":"2024-02-19T14:53:48.745155+0000","last_became_peered":"2024-02-19T14:53:48.745155+0000","last_unstale":"2024-02-19T14:53:50.101450+0000","last_undegraded":"2024-02-19T14:53:50.101450+0000","last_fullsized":"2024-02-19T14:53:50.101450+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:19.624345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738465+0000","last_change":"2024-02-19T14:53:47.740831+0000","last_active":"2024-02-19T14:53:49.738465+0000","last_peered":"2024-02-19T14:53:49.738465+0000","last_clean":"2024-02-19T14:53:49.738465+0000","last_became_active":"2024-02-19T14:53:47.740540+0000","last_became_peered":"2024-02-19T14:53:47.740540+0000","last_unstale":"2024-02-19T14:53:49.738465+0000","last_undegraded":"2024-02-19T14:53:49.738465+0000","last_fullsized":"2024-02-19T14:53:49.738465+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:55.237987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103018+0000","last_change":"2024-02-19T14:53:37.696555+0000","last_active":"2024-02-19T14:53:50.103018+0000","last_peered":"2024-02-19T14:53:50.103018+0000","last_clean":"2024-02-19T14:53:50.103018+0000","last_became_active":"2024-02-19T14:53:37.696315+0000","last_became_peered":"2024-02-19T14:53:37.696315+0000","last_unstale":"2024-02-19T14:53:50.103018+0000","last_undegraded":"2024-02-19T14:53:50.103018+0000","last_fullsized":"2024-02-19T14:53:50.103018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:26:03.766658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102305+0000","last_change":"2024-02-19T14:53:36.687655+0000","last_active":"2024-02-19T14:53:50.102305+0000","last_peered":"2024-02-19T14:53:50.102305+0000","last_clean":"2024-02-19T14:53:50.102305+0000","last_became_active":"2024-02-19T14:53:36.687359+0000","last_became_peered":"2024-02-19T14:53:36.687359+0000","last_unstale":"2024-02-19T14:53:50.102305+0000","last_undegraded":"2024-02-19T14:53:50.102305+0000","last_fullsized":"2024-02-19T14:53:50.102305+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:17:18.391861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101165+0000","last_change":"2024-02-19T14:53:48.746428+0000","last_active":"2024-02-19T14:53:50.101165+0000","last_peered":"2024-02-19T14:53:50.101165+0000","last_clean":"2024-02-19T14:53:50.101165+0000","last_became_active":"2024-02-19T14:53:48.746052+0000","last_became_peered":"2024-02-19T14:53:48.746052+0000","last_unstale":"2024-02-19T14:53:50.101165+0000","last_undegraded":"2024-02-19T14:53:50.101165+0000","last_fullsized":"2024-02-19T14:53:50.101165+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:28.812081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738967+0000","last_change":"2024-02-19T14:53:47.736026+0000","last_active":"2024-02-19T14:53:49.738967+0000","last_peered":"2024-02-19T14:53:49.738967+0000","last_clean":"2024-02-19T14:53:49.738967+0000","last_became_active":"2024-02-19T14:53:47.735680+0000","last_became_peered":"2024-02-19T14:53:47.735680+0000","last_unstale":"2024-02-19T14:53:49.738967+0000","last_undegraded":"2024-02-19T14:53:49.738967+0000","last_fullsized":"2024-02-19T14:53:49.738967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:00.755342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737384+0000","last_change":"2024-02-19T14:53:37.688334+0000","last_active":"2024-02-19T14:53:49.737384+0000","last_peered":"2024-02-19T14:53:49.737384+0000","last_clean":"2024-02-19T14:53:49.737384+0000","last_became_active":"2024-02-19T14:53:37.688067+0000","last_became_peered":"2024-02-19T14:53:37.688067+0000","last_unstale":"2024-02-19T14:53:49.737384+0000","last_undegraded":"2024-02-19T14:53:49.737384+0000","last_fullsized":"2024-02-19T14:53:49.737384+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:16.324839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100966+0000","last_change":"2024-02-19T14:53:36.683139+0000","last_active":"2024-02-19T14:53:50.100966+0000","last_peered":"2024-02-19T14:53:50.100966+0000","last_clean":"2024-02-19T14:53:50.100966+0000","last_became_active":"2024-02-19T14:53:36.682880+0000","last_became_peered":"2024-02-19T14:53:36.682880+0000","last_unstale":"2024-02-19T14:53:50.100966+0000","last_undegraded":"2024-02-19T14:53:50.100966+0000","last_fullsized":"2024-02-19T14:53:50.100966+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:03.449908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101045+0000","last_change":"2024-02-19T14:53:48.747311+0000","last_active":"2024-02-19T14:53:50.101045+0000","last_peered":"2024-02-19T14:53:50.101045+0000","last_clean":"2024-02-19T14:53:50.101045+0000","last_became_active":"2024-02-19T14:53:48.746975+0000","last_became_peered":"2024-02-19T14:53:48.746975+0000","last_unstale":"2024-02-19T14:53:50.101045+0000","last_undegraded":"2024-02-19T14:53:50.101045+0000","last_fullsized":"2024-02-19T14:53:50.101045+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:31:27.622310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101945+0000","last_change":"2024-02-19T14:53:47.737977+0000","last_active":"2024-02-19T14:53:50.101945+0000","last_peered":"2024-02-19T14:53:50.101945+0000","last_clean":"2024-02-19T14:53:50.101945+0000","last_became_active":"2024-02-19T14:53:47.737709+0000","last_became_peered":"2024-02-19T14:53:47.737709+0000","last_unstale":"2024-02-19T14:53:50.101945+0000","last_undegraded":"2024-02-19T14:53:50.101945+0000","last_fullsized":"2024-02-19T14:53:50.101945+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:02:56.724991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737389+0000","last_change":"2024-02-19T14:53:37.691501+0000","last_active":"2024-02-19T14:53:49.737389+0000","last_peered":"2024-02-19T14:53:49.737389+0000","last_clean":"2024-02-19T14:53:49.737389+0000","last_became_active":"2024-02-19T14:53:37.691225+0000","last_became_peered":"2024-02-19T14:53:37.691225+0000","last_unstale":"2024-02-19T14:53:49.737389+0000","last_undegraded":"2024-02-19T14:53:49.737389+0000","last_fullsized":"2024-02-19T14:53:49.737389+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:54:20.492105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"50'2","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102055+0000","last_change":"2024-02-19T14:53:36.689772+0000","last_active":"2024-02-19T14:53:50.102055+0000","last_peered":"2024-02-19T14:53:50.102055+0000","last_clean":"2024-02-19T14:53:50.102055+0000","last_became_active":"2024-02-19T14:53:36.689518+0000","last_became_peered":"2024-02-19T14:53:36.689518+0000","last_unstale":"2024-02-19T14:53:50.102055+0000","last_undegraded":"2024-02-19T14:53:50.102055+0000","last_fullsized":"2024-02-19T14:53:50.102055+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:46.453509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102985+0000","last_change":"2024-02-19T14:53:48.735142+0000","last_active":"2024-02-19T14:53:50.102985+0000","last_peered":"2024-02-19T14:53:50.102985+0000","last_clean":"2024-02-19T14:53:50.102985+0000","last_became_active":"2024-02-19T14:53:48.734855+0000","last_became_peered":"2024-02-19T14:53:48.734855+0000","last_unstale":"2024-02-19T14:53:50.102985+0000","last_undegraded":"2024-02-19T14:53:50.102985+0000","last_fullsized":"2024-02-19T14:53:50.102985+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:35.702079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103214+0000","last_change":"2024-02-19T14:53:47.745205+0000","last_active":"2024-02-19T14:53:50.103214+0000","last_peered":"2024-02-19T14:53:50.103214+0000","last_clean":"2024-02-19T14:53:50.103214+0000","last_became_active":"2024-02-19T14:53:47.744958+0000","last_became_peered":"2024-02-19T14:53:47.744958+0000","last_unstale":"2024-02-19T14:53:50.103214+0000","last_undegraded":"2024-02-19T14:53:50.103214+0000","last_fullsized":"2024-02-19T14:53:50.103214+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:31.942230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.736432+0000","last_change":"2024-02-19T14:53:37.692851+0000","last_active":"2024-02-19T14:53:49.736432+0000","last_peered":"2024-02-19T14:53:49.736432+0000","last_clean":"2024-02-19T14:53:49.736432+0000","last_became_active":"2024-02-19T14:53:37.692610+0000","last_became_peered":"2024-02-19T14:53:37.692610+0000","last_unstale":"2024-02-19T14:53:49.736432+0000","last_undegraded":"2024-02-19T14:53:49.736432+0000","last_fullsized":"2024-02-19T14:53:49.736432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:11.881422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102394+0000","last_change":"2024-02-19T14:53:36.684548+0000","last_active":"2024-02-19T14:53:50.102394+0000","last_peered":"2024-02-19T14:53:50.102394+0000","last_clean":"2024-02-19T14:53:50.102394+0000","last_became_active":"2024-02-19T14:53:36.684229+0000","last_became_peered":"2024-02-19T14:53:36.684229+0000","last_unstale":"2024-02-19T14:53:50.102394+0000","last_undegraded":"2024-02-19T14:53:50.102394+0000","last_fullsized":"2024-02-19T14:53:50.102394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:24.726849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738640+0000","last_change":"2024-02-19T14:53:48.744070+0000","last_active":"2024-02-19T14:53:49.738640+0000","last_peered":"2024-02-19T14:53:49.738640+0000","last_clean":"2024-02-19T14:53:49.738640+0000","last_became_active":"2024-02-19T14:53:48.743776+0000","last_became_peered":"2024-02-19T14:53:48.743776+0000","last_unstale":"2024-02-19T14:53:49.738640+0000","last_undegraded":"2024-02-19T14:53:49.738640+0000","last_fullsized":"2024-02-19T14:53:49.738640+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:49:26.499487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737870+0000","last_change":"2024-02-19T14:53:47.740210+0000","last_active":"2024-02-19T14:53:49.737870+0000","last_peered":"2024-02-19T14:53:49.737870+0000","last_clean":"2024-02-19T14:53:49.737870+0000","last_became_active":"2024-02-19T14:53:47.739913+0000","last_became_peered":"2024-02-19T14:53:47.739913+0000","last_unstale":"2024-02-19T14:53:49.737870+0000","last_undegraded":"2024-02-19T14:53:49.737870+0000","last_fullsized":"2024-02-19T14:53:49.737870+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:46.074469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101472+0000","last_change":"2024-02-19T14:53:37.684948+0000","last_active":"2024-02-19T14:53:50.101472+0000","last_peered":"2024-02-19T14:53:50.101472+0000","last_clean":"2024-02-19T14:53:50.101472+0000","last_became_active":"2024-02-19T14:53:37.684635+0000","last_became_peered":"2024-02-19T14:53:37.684635+0000","last_unstale":"2024-02-19T14:53:50.101472+0000","last_undegraded":"2024-02-19T14:53:50.101472+0000","last_fullsized":"2024-02-19T14:53:50.101472+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:10.864244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101846+0000","last_change":"2024-02-19T14:53:36.691589+0000","last_active":"2024-02-19T14:53:50.101846+0000","last_peered":"2024-02-19T14:53:50.101846+0000","last_clean":"2024-02-19T14:53:50.101846+0000","last_became_active":"2024-02-19T14:53:36.691343+0000","last_became_peered":"2024-02-19T14:53:36.691343+0000","last_unstale":"2024-02-19T14:53:50.101846+0000","last_undegraded":"2024-02-19T14:53:50.101846+0000","last_fullsized":"2024-02-19T14:53:50.101846+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:46:17.432806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.739444+0000","last_change":"2024-02-19T14:53:48.748006+0000","last_active":"2024-02-19T14:53:49.739444+0000","last_peered":"2024-02-19T14:53:49.739444+0000","last_clean":"2024-02-19T14:53:49.739444+0000","last_became_active":"2024-02-19T14:53:48.747662+0000","last_became_peered":"2024-02-19T14:53:48.747662+0000","last_unstale":"2024-02-19T14:53:49.739444+0000","last_undegraded":"2024-02-19T14:53:49.739444+0000","last_fullsized":"2024-02-19T14:53:49.739444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:11.068155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101671+0000","last_change":"2024-02-19T14:53:47.738871+0000","last_active":"2024-02-19T14:53:50.101671+0000","last_peered":"2024-02-19T14:53:50.101671+0000","last_clean":"2024-02-19T14:53:50.101671+0000","last_became_active":"2024-02-19T14:53:47.738585+0000","last_became_peered":"2024-02-19T14:53:47.738585+0000","last_unstale":"2024-02-19T14:53:50.101671+0000","last_undegraded":"2024-02-19T14:53:50.101671+0000","last_fullsized":"2024-02-19T14:53:50.101671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:57.701790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101635+0000","last_change":"2024-02-19T14:53:37.702976+0000","last_active":"2024-02-19T14:53:50.101635+0000","last_peered":"2024-02-19T14:53:50.101635+0000","last_clean":"2024-02-19T14:53:50.101635+0000","last_became_active":"2024-02-19T14:53:37.702241+0000","last_became_peered":"2024-02-19T14:53:37.702241+0000","last_unstale":"2024-02-19T14:53:50.101635+0000","last_undegraded":"2024-02-19T14:53:50.101635+0000","last_fullsized":"2024-02-19T14:53:50.101635+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:15:15.999342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101692+0000","last_change":"2024-02-19T14:53:36.682487+0000","last_active":"2024-02-19T14:53:50.101692+0000","last_peered":"2024-02-19T14:53:50.101692+0000","last_clean":"2024-02-19T14:53:50.101692+0000","last_became_active":"2024-02-19T14:53:36.682231+0000","last_became_peered":"2024-02-19T14:53:36.682231+0000","last_unstale":"2024-02-19T14:53:50.101692+0000","last_undegraded":"2024-02-19T14:53:50.101692+0000","last_fullsized":"2024-02-19T14:53:50.101692+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:10.876605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101780+0000","last_change":"2024-02-19T14:53:48.749668+0000","last_active":"2024-02-19T14:53:50.101780+0000","last_peered":"2024-02-19T14:53:50.101780+0000","last_clean":"2024-02-19T14:53:50.101780+0000","last_became_active":"2024-02-19T14:53:48.749338+0000","last_became_peered":"2024-02-19T14:53:48.749338+0000","last_unstale":"2024-02-19T14:53:50.101780+0000","last_undegraded":"2024-02-19T14:53:50.101780+0000","last_fullsized":"2024-02-19T14:53:50.101780+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:58.706427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102490+0000","last_change":"2024-02-19T14:53:47.738834+0000","last_active":"2024-02-19T14:53:50.102490+0000","last_peered":"2024-02-19T14:53:50.102490+0000","last_clean":"2024-02-19T14:53:50.102490+0000","last_became_active":"2024-02-19T14:53:47.738567+0000","last_became_peered":"2024-02-19T14:53:47.738567+0000","last_unstale":"2024-02-19T14:53:50.102490+0000","last_undegraded":"2024-02-19T14:53:50.102490+0000","last_fullsized":"2024-02-19T14:53:50.102490+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:18.619708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737854+0000","last_change":"2024-02-19T14:53:37.687974+0000","last_active":"2024-02-19T14:53:49.737854+0000","last_peered":"2024-02-19T14:53:49.737854+0000","last_clean":"2024-02-19T14:53:49.737854+0000","last_became_active":"2024-02-19T14:53:37.687624+0000","last_became_peered":"2024-02-19T14:53:37.687624+0000","last_unstale":"2024-02-19T14:53:49.737854+0000","last_undegraded":"2024-02-19T14:53:49.737854+0000","last_fullsized":"2024-02-19T14:53:49.737854+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:41.103197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101481+0000","last_change":"2024-02-19T14:53:36.681002+0000","last_active":"2024-02-19T14:53:50.101481+0000","last_peered":"2024-02-19T14:53:50.101481+0000","last_clean":"2024-02-19T14:53:50.101481+0000","last_became_active":"2024-02-19T14:53:36.680621+0000","last_became_peered":"2024-02-19T14:53:36.680621+0000","last_unstale":"2024-02-19T14:53:50.101481+0000","last_undegraded":"2024-02-19T14:53:50.101481+0000","last_fullsized":"2024-02-19T14:53:50.101481+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:23:22.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101542+0000","last_change":"2024-02-19T14:53:48.759798+0000","last_active":"2024-02-19T14:53:50.101542+0000","last_peered":"2024-02-19T14:53:50.101542+0000","last_clean":"2024-02-19T14:53:50.101542+0000","last_became_active":"2024-02-19T14:53:48.759486+0000","last_became_peered":"2024-02-19T14:53:48.759486+0000","last_unstale":"2024-02-19T14:53:50.101542+0000","last_undegraded":"2024-02-19T14:53:50.101542+0000","last_fullsized":"2024-02-19T14:53:50.101542+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:10.556428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738265+0000","last_change":"2024-02-19T14:53:47.744007+0000","last_active":"2024-02-19T14:53:49.738265+0000","last_peered":"2024-02-19T14:53:49.738265+0000","last_clean":"2024-02-19T14:53:49.738265+0000","last_became_active":"2024-02-19T14:53:47.743726+0000","last_became_peered":"2024-02-19T14:53:47.743726+0000","last_unstale":"2024-02-19T14:53:49.738265+0000","last_undegraded":"2024-02-19T14:53:49.738265+0000","last_fullsized":"2024-02-19T14:53:49.738265+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:59.453875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102851+0000","last_change":"2024-02-19T14:53:37.702735+0000","last_active":"2024-02-19T14:53:50.102851+0000","last_peered":"2024-02-19T14:53:50.102851+0000","last_clean":"2024-02-19T14:53:50.102851+0000","last_became_active":"2024-02-19T14:53:37.702346+0000","last_became_peered":"2024-02-19T14:53:37.702346+0000","last_unstale":"2024-02-19T14:53:50.102851+0000","last_undegraded":"2024-02-19T14:53:50.102851+0000","last_fullsized":"2024-02-19T14:53:50.102851+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:41.813078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.738616+0000","last_change":"2024-02-19T14:53:36.685823+0000","last_active":"2024-02-19T14:53:49.738616+0000","last_peered":"2024-02-19T14:53:49.738616+0000","last_clean":"2024-02-19T14:53:49.738616+0000","last_became_active":"2024-02-19T14:53:36.685494+0000","last_became_peered":"2024-02-19T14:53:36.685494+0000","last_unstale":"2024-02-19T14:53:49.738616+0000","last_undegraded":"2024-02-19T14:53:49.738616+0000","last_fullsized":"2024-02-19T14:53:49.738616+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:03.356609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102553+0000","last_change":"2024-02-19T14:53:48.755650+0000","last_active":"2024-02-19T14:53:50.102553+0000","last_peered":"2024-02-19T14:53:50.102553+0000","last_clean":"2024-02-19T14:53:50.102553+0000","last_became_active":"2024-02-19T14:53:48.755366+0000","last_became_peered":"2024-02-19T14:53:48.755366+0000","last_unstale":"2024-02-19T14:53:50.102553+0000","last_undegraded":"2024-02-19T14:53:50.102553+0000","last_fullsized":"2024-02-19T14:53:50.102553+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:19:12.328626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101880+0000","last_change":"2024-02-19T14:53:47.742015+0000","last_active":"2024-02-19T14:53:50.101880+0000","last_peered":"2024-02-19T14:53:50.101880+0000","last_clean":"2024-02-19T14:53:50.101880+0000","last_became_active":"2024-02-19T14:53:47.741742+0000","last_became_peered":"2024-02-19T14:53:47.741742+0000","last_unstale":"2024-02-19T14:53:50.101880+0000","last_undegraded":"2024-02-19T14:53:50.101880+0000","last_fullsized":"2024-02-19T14:53:50.101880+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:51.356042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.102257+0000","last_change":"2024-02-19T14:53:37.682770+0000","last_active":"2024-02-19T14:53:50.102257+0000","last_peered":"2024-02-19T14:53:50.102257+0000","last_clean":"2024-02-19T14:53:50.102257+0000","last_became_active":"2024-02-19T14:53:37.682533+0000","last_became_peered":"2024-02-19T14:53:37.682533+0000","last_unstale":"2024-02-19T14:53:50.102257+0000","last_undegraded":"2024-02-19T14:53:50.102257+0000","last_fullsized":"2024-02-19T14:53:50.102257+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:43:45.189873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:49.737482+0000","last_change":"2024-02-19T14:53:36.683756+0000","last_active":"2024-02-19T14:53:49.737482+0000","last_peered":"2024-02-19T14:53:49.737482+0000","last_clean":"2024-02-19T14:53:49.737482+0000","last_became_active":"2024-02-19T14:53:36.683489+0000","last_became_peered":"2024-02-19T14:53:36.683489+0000","last_unstale":"2024-02-19T14:53:49.737482+0000","last_undegraded":"2024-02-19T14:53:49.737482+0000","last_fullsized":"2024-02-19T14:53:49.737482+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:47.529459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101190+0000","last_change":"2024-02-19T14:53:48.748430+0000","last_active":"2024-02-19T14:53:50.101190+0000","last_peered":"2024-02-19T14:53:50.101190+0000","last_clean":"2024-02-19T14:53:50.101190+0000","last_became_active":"2024-02-19T14:53:48.746994+0000","last_became_peered":"2024-02-19T14:53:48.746994+0000","last_unstale":"2024-02-19T14:53:50.101190+0000","last_undegraded":"2024-02-19T14:53:50.101190+0000","last_fullsized":"2024-02-19T14:53:50.101190+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:47.716914+0000","last_clean_scrub_stamp":"2024-02-19T14:53:47.716914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:19:52.360680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101989+0000","last_change":"2024-02-19T14:53:47.734524+0000","last_active":"2024-02-19T14:53:50.101989+0000","last_peered":"2024-02-19T14:53:50.101989+0000","last_clean":"2024-02-19T14:53:50.101989+0000","last_became_active":"2024-02-19T14:53:47.734132+0000","last_became_peered":"2024-02-19T14:53:47.734132+0000","last_unstale":"2024-02-19T14:53:50.101989+0000","last_undegraded":"2024-02-19T14:53:50.101989+0000","last_fullsized":"2024-02-19T14:53:50.101989+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:46.712277+0000","last_clean_scrub_stamp":"2024-02-19T14:53:46.712277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:51.517212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.100956+0000","last_change":"2024-02-19T14:53:37.694125+0000","last_active":"2024-02-19T14:53:50.100956+0000","last_peered":"2024-02-19T14:53:50.100956+0000","last_clean":"2024-02-19T14:53:50.100956+0000","last_became_active":"2024-02-19T14:53:37.693839+0000","last_became_peered":"2024-02-19T14:53:37.693839+0000","last_unstale":"2024-02-19T14:53:50.100956+0000","last_undegraded":"2024-02-19T14:53:50.100956+0000","last_fullsized":"2024-02-19T14:53:50.100956+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:36.664163+0000","last_clean_scrub_stamp":"2024-02-19T14:53:36.664163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:38:25.731666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"51'1","reported_seq":27,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.103023+0000","last_change":"2024-02-19T14:53:36.689182+0000","last_active":"2024-02-19T14:53:50.103023+0000","last_peered":"2024-02-19T14:53:50.103023+0000","last_clean":"2024-02-19T14:53:50.103023+0000","last_became_active":"2024-02-19T14:53:36.688901+0000","last_became_peered":"2024-02-19T14:53:36.688901+0000","last_unstale":"2024-02-19T14:53:50.103023+0000","last_undegraded":"2024-02-19T14:53:50.103023+0000","last_fullsized":"2024-02-19T14:53:50.103023+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:53:35.659345+0000","last_clean_scrub_stamp":"2024-02-19T14:53:35.659345+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:47:17.037891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":241,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-19T14:53:50.101920+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:53:50.101920+0000","last_peered":"2024-02-19T14:53:50.101920+0000","last_clean":"2024-02-19T14:53:50.101920+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:53:50.101920+0000","last_undegraded":"2024-02-19T14:53:50.101920+0000","last_fullsized":"2024-02-19T14:53:50.101920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28292,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343548,"statfs":{"total":96636764160,"available":96607793152,"internally_reserved":0,"allocated":1298432,"data_stored":905408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.626},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.433},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.497,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.67900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":314826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.496}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.61699999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":314162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.497},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.441},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55800000000000005,"15min":0.56899999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46500000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.48199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":316334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.747,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.59699999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.64400000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95799999999999996,"5min":1.038,"15min":1.038},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70999999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.748}]},{"osd":7,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.76800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":315774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29499999999999998},"min":{"1min":0.17899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.309,"15min":0.30499999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.31},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.374},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.35799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.374,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49199999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.28299999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69099999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68700000000000006,"5min":1.133,"15min":1.133},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.66100000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82999999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.628}]}]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":315752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.254},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.437,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.42799999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.373}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.375,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79000000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84199999999999997,"5min":0.878,"15min":0.878},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69999999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94199999999999995,"5min":1.236,"15min":1.236},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.995,"15min":0.995},"last":0.62}]}]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":316676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33500000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.497,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33100000000000002,"15min":0.33800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.253}]},{"osd":2,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.307,"15min":0.309},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.371,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57999999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.622}]},{"osd":5,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.85799999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44400000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.746}]},{"osd":7,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.69299999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":904564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.38100000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.315,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32300000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42599999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34899999999999998,"15min":0.35199999999999998},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.42699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.41199999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.436},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.317}]},{"osd":4,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57599999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.59999999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.504},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68700000000000006,"5min":0.754,"15min":0.754},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72999999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:53:56.869 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:53:56.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 3:0 2024-02-19T14:53:57.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.000+0000 7f2d3ea88640 1 Processor -- start 2024-02-19T14:53:57.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.000+0000 7f2d3ea88640 1 -- start start 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.001+0000 7f2d3ea88640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d38155010 0x7f2d38159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.001+0000 7f2d3ea88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3806ffa0 con 0x7f2d380c4d80 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.001+0000 7f2d3ea88640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3806fc70 con 0x7f2d380c78a0 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.001+0000 7f2d3ea88640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3806b720 con 0x7f2d38155010 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.001+0000 7f2d37fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2d380c4d80 0x7f2d381523d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60354/0 (socket says 172.21.15.78:60354) 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.001+0000 7f2d37fff640 1 -- 172.21.15.78:0/3992621897 learned_addr learned my addr 172.21.15.78:0/3992621897 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d3d025640 1 --2- 172.21.15.78:0/3992621897 >> v2:172.21.15.150:3300/0 conn(0x7f2d38155010 0x7f2d38159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821379735 0 0) 0x7f2d3806ffa0 con 0x7f2d380c4d80 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d28003520 con 0x7f2d380c4d80 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426327160 0 0) 0x7f2d28003520 con 0x7f2d380c4d80 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 >> v2:172.21.15.150:3300/0 conn(0x7f2d38155010 msgr2=0x7f2d38159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 --2- 172.21.15.78:0/3992621897 >> v2:172.21.15.150:3300/0 conn(0x7f2d38155010 0x7f2d38159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 >> v1:172.21.15.150:6789/0 conn(0x7f2d380c78a0 legacy=0x7f2d380c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3806b930 con 0x7f2d380c4d80 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2d24001670 con 0x7f2d380c4d80 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 >> v1:172.21.15.78:6789/0 conn(0x7f2d380c4d80 legacy=0x7f2d381523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d28003970 0x7f2d28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28005180 0x7f2d2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3806b720 con 0x7f2d280047c0 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2800c120 con 0x7f2d28005180 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2800c2e0 con 0x7f2d28003970 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d37fff640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d3d025640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28005180 0x7f2d2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d3c824640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d28003970 0x7f2d28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d37fff640 1 -- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d28003970 msgr2=0x7f2d28003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d37fff640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d28003970 0x7f2d28003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d37fff640 1 -- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28005180 msgr2=0x7f2d2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d37fff640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28005180 0x7f2d2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d3d025640 1 -- 172.21.15.78:0/3992621897 reap_dead start 2024-02-19T14:53:57.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.002+0000 7f2d37fff640 1 -- 172.21.15.78:0/3992621897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28003520 con 0x7f2d280047c0 2024-02-19T14:53:57.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.003+0000 7f2d37fff640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d28004c40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c240 tx=0x7f2d280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1115e83b2761af server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.003+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24005ad0 con 0x7f2d280047c0 2024-02-19T14:53:57.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.003+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24005c30 con 0x7f2d280047c0 2024-02-19T14:53:57.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.003+0000 7f2d377fe640 1 -- 172.21.15.78:0/3992621897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2402ca90 con 0x7f2d280047c0 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 -- 172.21.15.78:0/3992621897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 msgr2=0x7f2d28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d28004c40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c240 tx=0x7f2d280057f0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 -- 172.21.15.78:0/3992621897 shutdown_connections 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/3992621897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d28004c40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 -- 172.21.15.78:0/3992621897 >> 172.21.15.78:0/3992621897 conn(0x7f2d38066e70 msgr2=0x7f2d380a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 -- 172.21.15.78:0/3992621897 shutdown_connections 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.004+0000 7f2d3ea88640 1 -- 172.21.15.78:0/3992621897 wait complete. 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.005+0000 7f2d3ea88640 1 Processor -- start 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.005+0000 7f2d3ea88640 1 -- start start 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3ea88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d380ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3ea88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d380ab7e0 0x7f2d3814cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3ea88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d38156a10 0x7f2d381edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3ea88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d380ab350 con 0x7f2d280047c0 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3ea88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d38157be0 con 0x7f2d38156a10 2024-02-19T14:53:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3ea88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d38157ec0 con 0x7f2d380ab7e0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d380ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d380ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53478/0 (socket says 172.21.15.78:53478) 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 -- 172.21.15.78:0/4001209754 learned_addr learned my addr 172.21.15.78:0/4001209754 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3d025640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d38156a10 0x7f2d381edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d37fff640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d380ab7e0 0x7f2d3814cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 -- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d380ab7e0 msgr2=0x7f2d3814cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d380ab7e0 0x7f2d3814cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 -- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d38156a10 msgr2=0x7f2d381edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d38156a10 0x7f2d381edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 -- 172.21.15.78:0/4001209754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d381581a0 con 0x7f2d280047c0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3d025640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d38156a10 0x7f2d381edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d37fff640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d380ab7e0 0x7f2d3814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.006+0000 7f2d3c824640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d380ac300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00f5f0 tx=0x7f2d2c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.007+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c0052e0 con 0x7f2d280047c0 2024-02-19T14:53:57.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.007+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c005440 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.007+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d381ee8f0 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.007+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c003810 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.007+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3814d5c0 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.008+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d38074990 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.008+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2d2c003970 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.009+0000 7f2d357fa640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1c06b810 0x7f2d1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.009+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7015+0+0 (secure 0 0 0) 0x7f2d2c08a510 con 0x7f2d280047c0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.009+0000 7f2d37fff640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1c06b810 0x7f2d1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.009+0000 7f2d37fff640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1c06b810 0x7f2d1c06dcd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d24034000 tx=0x7f2d24009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.013+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d2c054aa0 con 0x7f2d280047c0 2024-02-19T14:53:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.264+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "3:0"} v 0) v1 -- 0x7f2d380a87a0 con 0x7f2d280047c0 2024-02-19T14:53:57.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:53:57.265+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:53:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.266+0000 7f2d357fa640 1 -- 172.21.15.78:0/4001209754 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "3:0"}]=0 failed mds gid 5703 v31) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2d2c058160 con 0x7f2d280047c0 2024-02-19T14:53:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:failed mds gid 5703 2024-02-19T14:53:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.269+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1c06b810 msgr2=0x7f2d1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.269+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1c06b810 0x7f2d1c06dcd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d24034000 tx=0x7f2d24009030 comp rx=0 tx=0).stop 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.269+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 msgr2=0x7f2d380ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.269+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d380ac300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00f5f0 tx=0x7f2d2c00fab0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 shutdown_connections 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1c06b810 0x7f2d1c06dcd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d38156a10 0x7f2d381edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d380ab7e0 0x7f2d3814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 --2- 172.21.15.78:0/4001209754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d280047c0 0x7f2d380ac300 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 >> 172.21.15.78:0/4001209754 conn(0x7f2d38066e70 msgr2=0x7f2d380a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 shutdown_connections 2024-02-19T14:53:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.270+0000 7f2d3ea88640 1 -- 172.21.15.78:0/4001209754 wait complete. 2024-02-19T14:53:57.273 INFO:tasks.ceph.mds.a.smithi078.stderr: -445> 2024-02-19T14:53:42.944+0000 7f8f5e0a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:53:57.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:57.302 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:53:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.388+0000 7fc32f5e9640 1 Processor -- start 2024-02-19T14:53:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.389+0000 7fc32f5e9640 1 -- start start 2024-02-19T14:53:57.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.389+0000 7fc32f5e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3280c7070 0x7fc3281559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.389+0000 7fc32f5e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.389+0000 7fc32f5e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc328155ef0 con 0x7fc328152480 2024-02-19T14:53:57.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.389+0000 7fc32f5e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32806dec0 con 0x7fc3280c7070 2024-02-19T14:53:57.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.389+0000 7fc32f5e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32806d520 con 0x7fc3280c87a0 2024-02-19T14:53:57.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32cb84640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3280c7070 0x7fc3281559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32db86640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32d385640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc328152480 0x7fc32806ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:37212/0 (socket says 172.21.15.78:37212) 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32d385640 1 -- 172.21.15.78:0/3260049542 learned_addr learned my addr 172.21.15.78:0/3260049542 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32cb84640 1 -- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 msgr2=0x7fc3280c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32cb84640 1 --2- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32cb84640 1 -- 172.21.15.78:0/3260049542 >> v1:172.21.15.150:6789/0 conn(0x7fc328152480 legacy=0x7fc32806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc32cb84640 1 -- 172.21.15.78:0/3260049542 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280c75c0 con 0x7fc3280c7070 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.390+0000 7fc31ffff640 1 -- 172.21.15.78:0/3260049542 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102594074 0 0) 0x7fc324001110 con 0x7fc328152480 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.391+0000 7fc32cb84640 1 --2- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3281559b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc3180097a0 tx=0x7fc31802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d415640040a390 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.391+0000 7fc31ffff640 1 -- 172.21.15.78:0/3260049542 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318031830 con 0x7fc3280c7070 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.391+0000 7fc31ffff640 1 -- 172.21.15.78:0/3260049542 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318031990 con 0x7fc3280c7070 2024-02-19T14:53:57.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.391+0000 7fc31ffff640 1 -- 172.21.15.78:0/3260049542 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318031d20 con 0x7fc3280c7070 2024-02-19T14:53:57.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 -- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 msgr2=0x7fc3281559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3281559b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc3180097a0 tx=0x7fc31802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 -- 172.21.15.78:0/3260049542 shutdown_connections 2024-02-19T14:53:57.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/3260049542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3281559b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 -- 172.21.15.78:0/3260049542 >> 172.21.15.78:0/3260049542 conn(0x7fc3280ca8a0 msgr2=0x7fc3280cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:57.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 -- 172.21.15.78:0/3260049542 shutdown_connections 2024-02-19T14:53:57.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.392+0000 7fc32f5e9640 1 -- 172.21.15.78:0/3260049542 wait complete. 2024-02-19T14:53:57.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 Processor -- start 2024-02-19T14:53:57.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 -- start start 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3280ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 0x7fc3281ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280c1970 con 0x7fc328152480 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280af110 con 0x7fc3280c7070 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.393+0000 7fc32f5e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280c4870 con 0x7fc3280c87a0 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 0x7fc3281ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 0x7fc3281ee0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53516/0 (socket says 172.21.15.78:53516) 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 -- 172.21.15.78:0/618184398 learned_addr learned my addr 172.21.15.78:0/618184398 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32d385640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3280ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32cb84640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 -- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 msgr2=0x7fc3280af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 -- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 msgr2=0x7fc3280ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3280ab320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 -- 172.21.15.78:0/618184398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc318009400 con 0x7fc328152480 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32d385640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3280ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32db86640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 0x7fc3281ee0e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc32000efc0 tx=0x7fc32000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320014480 con 0x7fc328152480 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000f040 con 0x7fc328152480 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3280c1bc0 con 0x7fc328152480 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.394+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320014820 con 0x7fc328152480 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.395+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3280afae0 con 0x7fc328152480 2024-02-19T14:53:57.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.395+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3281eeda0 con 0x7fc328152480 2024-02-19T14:53:57.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.396+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc320014980 con 0x7fc328152480 2024-02-19T14:53:57.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.397+0000 7fc31dffb640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc30c06b840 0x7fc30c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.397+0000 7fc32d385640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc30c06b840 0x7fc30c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.397+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7101+0+0 (secure 0 0 0) 0x7fc32008b220 con 0x7fc328152480 2024-02-19T14:53:57.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.397+0000 7fc32d385640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc30c06b840 0x7fc30c06dd00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc324000f60 tx=0x7fc324009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.401+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc320055760 con 0x7fc328152480 2024-02-19T14:53:57.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.610+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3280af280 con 0x7fc328152480 2024-02-19T14:53:57.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.610+0000 7fc31dffb640 1 -- 172.21.15.78:0/618184398 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5391 (secure 0 0 0) 0x7fc320058e20 con 0x7fc328152480 2024-02-19T14:53:57.613 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:57.613 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:53:57.270753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5398":{"gid":5398,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":6,"addr":"172.21.15.150:6834/210065453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":210065453},{"type":"v1","addr":"172.21.15.150:6834","nonce":210065453}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:53:57.613 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc30c06b840 msgr2=0x7fc30c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc30c06b840 0x7fc30c06dd00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc324000f60 tx=0x7fc324009040 comp rx=0 tx=0).stop 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 msgr2=0x7fc3281ee0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 0x7fc3281ee0e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc32000efc0 tx=0x7fc32000c490 comp rx=0 tx=0).stop 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 shutdown_connections 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc30c06b840 0x7fc30c06dd00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc328152480 0x7fc3281ee0e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3280c87a0 0x7fc3280af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 --2- 172.21.15.78:0/618184398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3280c7070 0x7fc3280ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.612+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 >> 172.21.15.78:0/618184398 conn(0x7fc3280ca8a0 msgr2=0x7fc328153560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.613+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 shutdown_connections 2024-02-19T14:53:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.613+0000 7fc32f5e9640 1 -- 172.21.15.78:0/618184398 wait complete. 2024-02-19T14:53:57.626 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-02-19T14:53:57.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-02-19T14:53:57.781 INFO:teuthology.orchestra.run.smithi078.stdout:{"mds_beacon_grace":"15.000000"} 2024-02-19T14:53:57.781 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-02-19T14:53:57.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:53:57.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.880+0000 7f5f516eb640 1 Processor -- start 2024-02-19T14:53:57.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.880+0000 7f5f516eb640 1 -- start start 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f516eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f516eb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f4c0c42e0 0x7f5f4c158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f516eb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0c34a0 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f516eb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c06c090 con 0x7f5f4c0c42e0 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f516eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c06d6a0 con 0x7f5f4c0c8500 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f4b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5f4c0c2f50 0x7f5f4c158b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47876/0 (socket says 172.21.15.78:47876) 2024-02-19T14:53:57.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f4b7fe640 1 -- 172.21.15.78:0/944926254 learned_addr learned my addr 172.21.15.78:0/944926254 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f4a7fc640 1 --2- 172.21.15.78:0/944926254 >> v2:172.21.15.150:3300/0 conn(0x7f5f4c0c42e0 0x7f5f4c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f4affd640 1 --2- 172.21.15.78:0/944926254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f49ffb640 1 -- 172.21.15.78:0/944926254 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248699924 0 0) 0x7f5f4c0c34a0 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f49ffb640 1 -- 172.21.15.78:0/944926254 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34003520 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.881+0000 7f5f49ffb640 1 -- 172.21.15.78:0/944926254 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926239337 0 0) 0x7f5f34003520 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.882+0000 7f5f4affd640 1 -- 172.21.15.78:0/944926254 >> v2:172.21.15.150:3300/0 conn(0x7f5f4c0c42e0 msgr2=0x7f5f4c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.882+0000 7f5f4affd640 1 --2- 172.21.15.78:0/944926254 >> v2:172.21.15.150:3300/0 conn(0x7f5f4c0c42e0 0x7f5f4c158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.882+0000 7f5f4affd640 1 -- 172.21.15.78:0/944926254 >> v1:172.21.15.78:6789/0 conn(0x7f5f4c0c2f50 legacy=0x7f5f4c158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.882+0000 7f5f4affd640 1 -- 172.21.15.78:0/944926254 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c06c8e0 con 0x7f5f4c0c8500 2024-02-19T14:53:57.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.882+0000 7f5f4a7fc640 1 --2- 172.21.15.78:0/944926254 >> v2:172.21.15.150:3300/0 conn(0x7f5f4c0c42e0 0x7f5f4c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f4affd640 1 --2- 172.21.15.78:0/944926254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c0c88e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5f400097a0 tx=0x7f5f4002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43c6c83acfed1c87 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f49ffb640 1 -- 172.21.15.78:0/944926254 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440019f0 con 0x7f5f4c0c8500 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f49ffb640 1 -- 172.21.15.78:0/944926254 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4002e050 con 0x7f5f4c0c8500 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f49ffb640 1 -- 172.21.15.78:0/944926254 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40030710 con 0x7f5f4c0c8500 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 -- 172.21.15.78:0/944926254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 msgr2=0x7f5f4c0c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 --2- 172.21.15.78:0/944926254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c0c88e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5f400097a0 tx=0x7f5f4002b710 comp rx=0 tx=0).stop 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 -- 172.21.15.78:0/944926254 shutdown_connections 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 --2- 172.21.15.78:0/944926254 >> v2:172.21.15.150:3300/0 conn(0x7f5f4c0c42e0 0x7f5f4c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 --2- 172.21.15.78:0/944926254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c0c88e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 -- 172.21.15.78:0/944926254 >> 172.21.15.78:0/944926254 conn(0x7f5f4c0aa300 msgr2=0x7f5f4c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:57.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 -- 172.21.15.78:0/944926254 shutdown_connections 2024-02-19T14:53:57.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.883+0000 7f5f516eb640 1 -- 172.21.15.78:0/944926254 wait complete. 2024-02-19T14:53:57.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.884+0000 7f5f516eb640 1 Processor -- start 2024-02-19T14:53:57.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 -- start start 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 0x7f5f4c1edc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c1ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f4c159250 0x7f5f4c14aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c1493b0 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c14aa90 con 0x7f5f4c159250 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f516eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c156bb0 con 0x7f5f4c0c8500 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 0x7f5f4c1edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f4affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 0x7f5f4c1edc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56000/0 (socket says 172.21.15.78:56000) 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f4affd640 1 -- 172.21.15.78:0/1774227786 learned_addr learned my addr 172.21.15.78:0/1774227786 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f4b7fe640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f4c159250 0x7f5f4c14aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.885+0000 7f5f4a7fc640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c1ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f4affd640 1 -- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 msgr2=0x7f5f4c1ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f4affd640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c1ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f4affd640 1 -- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f4c159250 msgr2=0x7f5f4c14aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f4affd640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f4c159250 0x7f5f4c14aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f4affd640 1 -- 172.21.15.78:0/1774227786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c009400 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f4affd640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 0x7f5f4c1edc50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5f40000c00 tx=0x7f5f400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4003b810 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4003b970 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40009400 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4003bd20 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.886+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c1ee770 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.887+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c071c20 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.888+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5f40039040 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.889+0000 7f5f337fe640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:53:57.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.889+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7101+0+0 (secure 0 0 0) 0x7f5f400abe00 con 0x7f5f4c0c2f50 2024-02-19T14:53:57.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.889+0000 7f5f4a7fc640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:53:57.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.889+0000 7f5f4a7fc640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c153ba0 tx=0x7f5f3c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:53:57.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:57.893+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5f40076510 con 0x7f5f4c0c2f50 2024-02-19T14:53:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.109+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4c14ad00 con 0x7f5f4c0c2f50 2024-02-19T14:53:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.109+0000 7f5f337fe640 1 -- 172.21.15.78:0/1774227786 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5391 (secure 0 0 0) 0x7f5f4004b090 con 0x7f5f4c0c2f50 2024-02-19T14:53:58.112 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:53:58.112 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:53:57.270753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5398":{"gid":5398,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":6,"addr":"172.21.15.150:6834/210065453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":210065453},{"type":"v1","addr":"172.21.15.150:6834","nonce":210065453}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:53:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 msgr2=0x7f5f3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c153ba0 tx=0x7f5f3c009190 comp rx=0 tx=0).stop 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 msgr2=0x7f5f4c1edc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 0x7f5f4c1edc50 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5f40000c00 tx=0x7f5f400046c0 comp rx=0 tx=0).stop 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 shutdown_connections 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f4c159250 0x7f5f4c14aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f4c0c8500 0x7f5f4c1ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 --2- 172.21.15.78:0/1774227786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f4c0c2f50 0x7f5f4c1edc50 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:53:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 >> 172.21.15.78:0/1774227786 conn(0x7f5f4c0aa300 msgr2=0x7f5f4c0c72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:53:58.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 shutdown_connections 2024-02-19T14:53:58.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:53:58.111+0000 7f5f516eb640 1 -- 172.21.15.78:0/1774227786 wait complete. 2024-02-19T14:53:58.124 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-02-19T14:54:00.315 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:54:03.125 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.223+0000 7f72235e9640 1 Processor -- start 2024-02-19T14:54:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.223+0000 7f72235e9640 1 -- start start 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f72235e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f721c0c8e00 0x7f721c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f72235e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c1551f0 con 0x7f721c152700 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f72235e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c155490 con 0x7f721c06d910 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f72235e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c155770 con 0x7f721c0c8e00 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f7221385640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f721c152700 0x7f721c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47882/0 (socket says 172.21.15.78:47882) 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f7221385640 1 -- 172.21.15.78:0/2090408386 learned_addr learned my addr 172.21.15.78:0/2090408386 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755862867 0 0) 0x7f721c1551f0 con 0x7f721c152700 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f7220b84640 1 --2- 172.21.15.78:0/2090408386 >> v2:172.21.15.150:3300/0 conn(0x7f721c0c8e00 0x7f721c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.224+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7204003520 con 0x7f721c152700 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4038462183 0 0) 0x7f7204003520 con 0x7f721c152700 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 >> v2:172.21.15.150:3300/0 conn(0x7f721c0c8e00 msgr2=0x7f721c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 --2- 172.21.15.78:0/2090408386 >> v2:172.21.15.150:3300/0 conn(0x7f721c0c8e00 0x7f721c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 >> v1:172.21.15.150:6789/0 conn(0x7f721c06d910 legacy=0x7f721c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c155a50 con 0x7f721c152700 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f720c001670 con 0x7f721c152700 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 >> v1:172.21.15.78:6789/0 conn(0x7f721c152700 legacy=0x7f721c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7204003970 0x7f7204003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72040047c0 0x7f7204004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7204005180 0x7f720400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c155770 con 0x7f72040047c0 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720400c020 con 0x7f7204005180 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720400c1e0 con 0x7f7204003970 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7220b84640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72040047c0 0x7f7204004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7221385640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7204003970 0x7f7204003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7221b86640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7204005180 0x7f720400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7220b84640 1 -- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7204003970 msgr2=0x7f7204003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7220b84640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7204003970 0x7f7204003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7220b84640 1 -- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7204005180 msgr2=0x7f720400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7220b84640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7204005180 0x7f720400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7220b84640 1 -- 172.21.15.78:0/2090408386 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7204003520 con 0x7f72040047c0 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7221b86640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7204005180 0x7f720400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7221385640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7204003970 0x7f7204003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.225+0000 7f7221b86640 1 -- 172.21.15.78:0/2090408386 reap_dead start 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.226+0000 7f7220b84640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72040047c0 0x7f7204004c40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7210009ac0 tx=0x7f72040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d32dca2f79b8fb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.226+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721002c6b0 con 0x7f72040047c0 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.226+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721002c810 con 0x7f72040047c0 2024-02-19T14:54:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.226+0000 7f721bfff640 1 -- 172.21.15.78:0/2090408386 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721002cba0 con 0x7f72040047c0 2024-02-19T14:54:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 -- 172.21.15.78:0/2090408386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72040047c0 msgr2=0x7f7204004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72040047c0 0x7f7204004c40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7210009ac0 tx=0x7f72040057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 -- 172.21.15.78:0/2090408386 shutdown_connections 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 --2- 172.21.15.78:0/2090408386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72040047c0 0x7f7204004c40 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 -- 172.21.15.78:0/2090408386 >> 172.21.15.78:0/2090408386 conn(0x7f721c0cac90 msgr2=0x7f721c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 -- 172.21.15.78:0/2090408386 shutdown_connections 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.227+0000 7f72235e9640 1 -- 172.21.15.78:0/2090408386 wait complete. 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.229+0000 7f72235e9640 1 Processor -- start 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.229+0000 7f72235e9640 1 -- start start 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f72235e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72040047c0 0x7f721c1ea930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f72235e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c0be370 0x7f721c0bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f72235e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 0x7f721c1ebe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f72235e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0bf460 con 0x7f721c0c0b40 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f72235e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0be850 con 0x7f721c0be370 2024-02-19T14:54:03.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f72235e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0beb30 con 0x7f72040047c0 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 0x7f721c1ebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 0x7f721c1ebe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56012/0 (socket says 172.21.15.78:56012) 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 -- 172.21.15.78:0/3245961782 learned_addr learned my addr 172.21.15.78:0/3245961782 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221385640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72040047c0 0x7f721c1ea930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7220b84640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c0be370 0x7f721c0bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 -- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72040047c0 msgr2=0x7f721c1ea930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72040047c0 0x7f721c1ea930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 -- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c0be370 msgr2=0x7f721c0bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c0be370 0x7f721c0bf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 -- 172.21.15.78:0/3245961782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c0bee10 con 0x7f721c0c0b40 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7221b86640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 0x7f721c1ebe20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f721c155120 tx=0x7f721400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.230+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214003b80 con 0x7f721c0c0b40 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.231+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7214003ce0 con 0x7f721c0c0b40 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.231+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c0bf0f0 con 0x7f721c0c0b40 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.231+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214005310 con 0x7f721c0c0b40 2024-02-19T14:54:03.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.231+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c1ec360 con 0x7f721c0c0b40 2024-02-19T14:54:03.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.232+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c0bf5d0 con 0x7f721c0c0b40 2024-02-19T14:54:03.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.232+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7214005470 con 0x7f721c0c0b40 2024-02-19T14:54:03.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.232+0000 7f7219ffb640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f720806b810 0x7f720806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.233+0000 7f7221385640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f720806b810 0x7f720806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.233+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f721408a450 con 0x7f721c0c0b40 2024-02-19T14:54:03.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.233+0000 7f7221385640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f720806b810 0x7f720806dcd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f721c069910 tx=0x7f720c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:03.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.237+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7214054930 con 0x7f721c0c0b40 2024-02-19T14:54:03.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.466+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f721c074990 con 0x7f721c0c0b40 2024-02-19T14:54:03.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.466+0000 7f7219ffb640 1 -- 172.21.15.78:0/3245961782 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6185 (secure 0 0 0) 0x7f7214057ff0 con 0x7f721c0c0b40 2024-02-19T14:54:03.469 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:03.469 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:54:02.288421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6042},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6042":{"gid":6042,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1226290552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1226290552},{"type":"v1","addr":"172.21.15.78:6837","nonce":1226290552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:03.469 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 39 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f720806b810 msgr2=0x7f720806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f720806b810 0x7f720806dcd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f721c069910 tx=0x7f720c009040 comp rx=0 tx=0).stop 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 msgr2=0x7f721c1ebe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 0x7f721c1ebe20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f721c155120 tx=0x7f721400e960 comp rx=0 tx=0).stop 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 shutdown_connections 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f720806b810 0x7f720806dcd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f721c0c0b40 0x7f721c1ebe20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c0be370 0x7f721c0bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 --2- 172.21.15.78:0/3245961782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72040047c0 0x7f721c1ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 >> 172.21.15.78:0/3245961782 conn(0x7f721c0cac90 msgr2=0x7f721c1514e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:03.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 shutdown_connections 2024-02-19T14:54:03.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.468+0000 7f72235e9640 1 -- 172.21.15.78:0/3245961782 wait complete. 2024-02-19T14:54:03.481 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:54:03.482 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:54:03.482 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:54:03.522 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:54:03.567 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:54:03.567 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:54:03.568 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:54:03.583 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:54:03.583 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:54:03.583 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:54:03.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:54:03.600 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:03.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.720+0000 7f6dbadd7640 1 Processor -- start 2024-02-19T14:54:03.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6dbadd7640 1 -- start start 2024-02-19T14:54:03.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6dbadd7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6db40c78a0 0x7f6db40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6dbadd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db40c35c0 con 0x7f6db414ef50 2024-02-19T14:54:03.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6dbadd7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db40c38a0 con 0x7f6db4072ee0 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6dbadd7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db40c3b80 con 0x7f6db40c78a0 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6db8b73640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6db40c78a0 0x7f6db40c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6db3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6db414ef50 0x7f6db40727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47890/0 (socket says 172.21.15.78:47890) 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.721+0000 7f6db3fff640 1 -- 172.21.15.78:0/2241541992 learned_addr learned my addr 172.21.15.78:0/2241541992 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.722+0000 7f6db8b73640 1 -- 172.21.15.78:0/2241541992 >> v1:172.21.15.150:6789/0 conn(0x7f6db4072ee0 legacy=0x7f6db415c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.722+0000 7f6db8b73640 1 -- 172.21.15.78:0/2241541992 >> v1:172.21.15.78:6789/0 conn(0x7f6db414ef50 legacy=0x7f6db40727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.722+0000 7f6db8b73640 1 -- 172.21.15.78:0/2241541992 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db40c3e60 con 0x7f6db40c78a0 2024-02-19T14:54:03.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.722+0000 7f6db37fe640 1 -- 172.21.15.78:0/2241541992 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886370630 0 0) 0x7f6db40c35c0 con 0x7f6db414ef50 2024-02-19T14:54:03.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6db8b73640 1 --2- 172.21.15.78:0/2241541992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db40c78a0 0x7f6db40c44b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6da80097a0 tx=0x7f6da802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f028ec6e5ced4a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:03.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6db37fe640 1 -- 172.21.15.78:0/2241541992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8030610 con 0x7f6db40c78a0 2024-02-19T14:54:03.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6db37fe640 1 -- 172.21.15.78:0/2241541992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8030770 con 0x7f6db40c78a0 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6db37fe640 1 -- 172.21.15.78:0/2241541992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8030b00 con 0x7f6db40c78a0 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2241541992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db40c78a0 msgr2=0x7f6db40c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2241541992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db40c78a0 0x7f6db40c44b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6da80097a0 tx=0x7f6da802b960 comp rx=0 tx=0).stop 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2241541992 shutdown_connections 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2241541992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db40c78a0 0x7f6db40c44b0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2241541992 >> 172.21.15.78:0/2241541992 conn(0x7f6db4066e70 msgr2=0x7f6db40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2241541992 shutdown_connections 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.723+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2241541992 wait complete. 2024-02-19T14:54:03.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.725+0000 7f6dbadd7640 1 Processor -- start 2024-02-19T14:54:03.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.725+0000 7f6dbadd7640 1 -- start start 2024-02-19T14:54:03.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6dbadd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 0x7f6db40aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6dbadd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6db40c78a0 0x7f6db414bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6dbadd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db414ef50 0x7f6db41e7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6dbadd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40be240 con 0x7f6db4072ee0 2024-02-19T14:54:03.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6dbadd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40bd170 con 0x7f6db414ef50 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6dbadd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40bd450 con 0x7f6db40c78a0 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 0x7f6db40aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 0x7f6db40aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56018/0 (socket says 172.21.15.78:56018) 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 -- 172.21.15.78:0/2196680380 learned_addr learned my addr 172.21.15.78:0/2196680380 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db9374640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db414ef50 0x7f6db41e7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db3fff640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6db40c78a0 0x7f6db414bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 -- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6db40c78a0 msgr2=0x7f6db414bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6db40c78a0 0x7f6db414bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 -- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db414ef50 msgr2=0x7f6db41e7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db414ef50 0x7f6db41e7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db8b73640 1 -- 172.21.15.78:0/2196680380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da8009400 con 0x7f6db4072ee0 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.726+0000 7f6db9374640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db414ef50 0x7f6db41e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.727+0000 7f6db8b73640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 0x7f6db40aba10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6da80098c0 tx=0x7f6da8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:03.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.727+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da803e040 con 0x7f6db4072ee0 2024-02-19T14:54:03.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.727+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8039cb0 con 0x7f6db4072ee0 2024-02-19T14:54:03.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.727+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8044630 con 0x7f6db4072ee0 2024-02-19T14:54:03.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.727+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db40bded0 con 0x7f6db4072ee0 2024-02-19T14:54:03.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.727+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db40ac860 con 0x7f6db4072ee0 2024-02-19T14:54:03.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.728+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db40bdd30 con 0x7f6db4072ee0 2024-02-19T14:54:03.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.728+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6da804d3b0 con 0x7f6db4072ee0 2024-02-19T14:54:03.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.729+0000 7f6db17fa640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d9806b810 0x7f6d9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:03.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.729+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f6da80acef0 con 0x7f6db4072ee0 2024-02-19T14:54:03.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.729+0000 7f6db3fff640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d9806b810 0x7f6d9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:03.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.729+0000 7f6db3fff640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d9806b810 0x7f6d9806dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6db41e7250 tx=0x7f6da0007d00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:03.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.733+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6db40bdd30 con 0x7f6db4072ee0 2024-02-19T14:54:03.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.954+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db40bdd30 con 0x7f6db4072ee0 2024-02-19T14:54:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.955+0000 7f6db17fa640 1 -- 172.21.15.78:0/2196680380 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6185 (secure 0 0 0) 0x7f6db40bdd30 con 0x7f6db4072ee0 2024-02-19T14:54:03.958 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:03.958 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:54:02.288421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6042},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6042":{"gid":6042,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1226290552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1226290552},{"type":"v1","addr":"172.21.15.78:6837","nonce":1226290552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:03.959 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 39 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d9806b810 msgr2=0x7f6d9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d9806b810 0x7f6d9806dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6db41e7250 tx=0x7f6da0007d00 comp rx=0 tx=0).stop 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 msgr2=0x7f6db40aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 0x7f6db40aba10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6da80098c0 tx=0x7f6da8004730 comp rx=0 tx=0).stop 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 shutdown_connections 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d9806b810 0x7f6d9806dcd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db414ef50 0x7f6db41e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6db40c78a0 0x7f6db414bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 --2- 172.21.15.78:0/2196680380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6db4072ee0 0x7f6db40aba10 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 >> 172.21.15.78:0/2196680380 conn(0x7f6db4066e70 msgr2=0x7f6db40c6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 shutdown_connections 2024-02-19T14:54:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:03.957+0000 7f6dbadd7640 1 -- 172.21.15.78:0/2196680380 wait complete. 2024-02-19T14:54:03.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:04.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.072+0000 7f6f2ec83640 1 Processor -- start 2024-02-19T14:54:04.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.072+0000 7f6f2ec83640 1 -- start start 2024-02-19T14:54:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2ec83640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6f280c6d60 0x7f6f28154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2ec83640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6f2806ff80 0x7f6f280c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2ec83640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2814f730 con 0x7f6f2806e830 2024-02-19T14:54:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2ec83640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f280c1940 con 0x7f6f280c6d60 2024-02-19T14:54:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2ec83640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f280c16d0 con 0x7f6f2806ff80 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2ca1f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6f280c6d60 0x7f6f28154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f27fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6f2806e830 0x7f6f280c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47898/0 (socket says 172.21.15.78:47898) 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f27fff640 1 -- 172.21.15.78:0/2024150741 learned_addr learned my addr 172.21.15.78:0/2024150741 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.073+0000 7f6f2d220640 1 --2- 172.21.15.78:0/2024150741 >> v2:172.21.15.150:3300/0 conn(0x7f6f2806ff80 0x7f6f280c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.074+0000 7f6f2ca1f640 1 -- 172.21.15.78:0/2024150741 >> v2:172.21.15.150:3300/0 conn(0x7f6f2806ff80 msgr2=0x7f6f280c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.074+0000 7f6f2ca1f640 1 --2- 172.21.15.78:0/2024150741 >> v2:172.21.15.150:3300/0 conn(0x7f6f2806ff80 0x7f6f280c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.074+0000 7f6f2ca1f640 1 -- 172.21.15.78:0/2024150741 >> v1:172.21.15.78:6789/0 conn(0x7f6f2806e830 legacy=0x7f6f280c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.074+0000 7f6f277fe640 1 -- 172.21.15.78:0/2024150741 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885421545 0 0) 0x7f6f2814f730 con 0x7f6f2806e830 2024-02-19T14:54:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.074+0000 7f6f2ca1f640 1 -- 172.21.15.78:0/2024150741 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28072f70 con 0x7f6f280c6d60 2024-02-19T14:54:04.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ca1f640 1 --2- 172.21.15.78:0/2024150741 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f280c6d60 0x7f6f28154f70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6f2806de00 tx=0x7f6f1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f94070b40660267 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f277fe640 1 -- 172.21.15.78:0/2024150741 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18031830 con 0x7f6f280c6d60 2024-02-19T14:54:04.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f277fe640 1 -- 172.21.15.78:0/2024150741 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f18031990 con 0x7f6f280c6d60 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f277fe640 1 -- 172.21.15.78:0/2024150741 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18031d20 con 0x7f6f280c6d60 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 -- 172.21.15.78:0/2024150741 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f280c6d60 msgr2=0x7f6f28154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/2024150741 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f280c6d60 0x7f6f28154f70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6f2806de00 tx=0x7f6f1802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 -- 172.21.15.78:0/2024150741 shutdown_connections 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/2024150741 >> v2:172.21.15.150:3300/0 conn(0x7f6f2806ff80 0x7f6f280c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/2024150741 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f280c6d60 0x7f6f28154f70 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 -- 172.21.15.78:0/2024150741 >> 172.21.15.78:0/2024150741 conn(0x7f6f2806c170 msgr2=0x7f6f2806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.075+0000 7f6f2ec83640 1 -- 172.21.15.78:0/2024150741 shutdown_connections 2024-02-19T14:54:04.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.076+0000 7f6f2ec83640 1 -- 172.21.15.78:0/2024150741 wait complete. 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.076+0000 7f6f2ec83640 1 Processor -- start 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 -- start start 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f2806e830 0x7f6f28153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 0x7f6f28149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f280c6d60 0x7f6f28149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280a7ef0 con 0x7f6f2806ff80 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280c3a50 con 0x7f6f280c6d60 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f2ec83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280c3d30 con 0x7f6f2806e830 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 0x7f6f28149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 0x7f6f28149760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56050/0 (socket says 172.21.15.78:56050) 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.077+0000 7f6f27fff640 1 -- 172.21.15.78:0/624892673 learned_addr learned my addr 172.21.15.78:0/624892673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f2d220640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f280c6d60 0x7f6f28149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f2ca1f640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f2806e830 0x7f6f28153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f27fff640 1 -- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f2806e830 msgr2=0x7f6f28153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f27fff640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f2806e830 0x7f6f28153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f27fff640 1 -- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f280c6d60 msgr2=0x7f6f28149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f27fff640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f280c6d60 0x7f6f28149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f27fff640 1 -- 172.21.15.78:0/624892673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18009400 con 0x7f6f2806ff80 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f2d220640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f280c6d60 0x7f6f28149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f27fff640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 0x7f6f28149760 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00c410 tx=0x7f6f1c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c0074e0 con 0x7f6f2806ff80 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1c007660 con 0x7f6f2806ff80 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c004a20 con 0x7f6f2806ff80 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f280a8100 con 0x7f6f2806ff80 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.078+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f28154240 con 0x7f6f2806ff80 2024-02-19T14:54:04.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.079+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f280a8620 con 0x7f6f2806ff80 2024-02-19T14:54:04.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.080+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6f1c004b80 con 0x7f6f2806ff80 2024-02-19T14:54:04.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.080+0000 7f6f257fa640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f1406b810 0x7f6f1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.080+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f6f1c08a5e0 con 0x7f6f2806ff80 2024-02-19T14:54:04.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.080+0000 7f6f2ca1f640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f1406b810 0x7f6f1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.081+0000 7f6f2ca1f640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f1406b810 0x7f6f1406dcd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6f18036000 tx=0x7f6f1803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.089+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6f280a8620 con 0x7f6f2806ff80 2024-02-19T14:54:04.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.347+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f280a8620 con 0x7f6f2806ff80 2024-02-19T14:54:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.348+0000 7f6f257fa640 1 -- 172.21.15.78:0/624892673 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+21009 (secure 0 0 0) 0x7f6f280a8620 con 0x7f6f2806ff80 2024-02-19T14:54:04.352 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:04.352 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":59,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:00.278788+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f1406b810 msgr2=0x7f6f1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f1406b810 0x7f6f1406dcd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6f18036000 tx=0x7f6f1803c040 comp rx=0 tx=0).stop 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 msgr2=0x7f6f28149760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 0x7f6f28149760 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00c410 tx=0x7f6f1c00c8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 shutdown_connections 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f1406b810 0x7f6f1406dcd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f2806ff80 0x7f6f28149760 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f280c6d60 0x7f6f28149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 --2- 172.21.15.78:0/624892673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f2806e830 0x7f6f28153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 >> 172.21.15.78:0/624892673 conn(0x7f6f2806c170 msgr2=0x7f6f28155090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 shutdown_connections 2024-02-19T14:54:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.351+0000 7f6f2ec83640 1 -- 172.21.15.78:0/624892673 wait complete. 2024-02-19T14:54:04.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:53:35.233332+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:53:35.989674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-19T14:53:45.797141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-19T14:53:47.077376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:04.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:04.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.469+0000 7f5f2d69a640 1 Processor -- start 2024-02-19T14:54:04.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.469+0000 7f5f2d69a640 1 -- start start 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f2d69a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f280c12a0 0x7f5f280723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f2d69a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f2d69a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2806f330 con 0x7f5f2814f3a0 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f2d69a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f280a84c0 con 0x7f5f280c12a0 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f2d69a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280a8fb0 con 0x7f5f28072910 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f267fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f280c12a0 0x7f5f280723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f26ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5f2814f3a0 0x7f5f280c30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47912/0 (socket says 172.21.15.78:47912) 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f26ffd640 1 -- 172.21.15.78:0/485198795 learned_addr learned my addr 172.21.15.78:0/485198795 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:04.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f267fc640 1 -- 172.21.15.78:0/485198795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 msgr2=0x7f5f28151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:04.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f277fe640 1 --2- 172.21.15.78:0/485198795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f25ffb640 1 -- 172.21.15.78:0/485198795 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001237642 0 0) 0x7f5f2806f330 con 0x7f5f2814f3a0 2024-02-19T14:54:04.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f267fc640 1 --2- 172.21.15.78:0/485198795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f267fc640 1 -- 172.21.15.78:0/485198795 >> v1:172.21.15.78:6789/0 conn(0x7f5f2814f3a0 legacy=0x7f5f280c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.471+0000 7f5f267fc640 1 -- 172.21.15.78:0/485198795 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f280a8710 con 0x7f5f280c12a0 2024-02-19T14:54:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.472+0000 7f5f267fc640 1 --2- 172.21.15.78:0/485198795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f280723d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800ba40 tx=0x7f5f1802df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5fab1234335e8f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.472+0000 7f5f25ffb640 1 -- 172.21.15.78:0/485198795 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1802ed30 con 0x7f5f280c12a0 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.472+0000 7f5f25ffb640 1 -- 172.21.15.78:0/485198795 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1802ee90 con 0x7f5f280c12a0 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.472+0000 7f5f25ffb640 1 -- 172.21.15.78:0/485198795 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18036660 con 0x7f5f280c12a0 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 -- 172.21.15.78:0/485198795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 msgr2=0x7f5f280723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/485198795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f280723d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800ba40 tx=0x7f5f1802df80 comp rx=0 tx=0).stop 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 -- 172.21.15.78:0/485198795 shutdown_connections 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/485198795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f280723d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/485198795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 -- 172.21.15.78:0/485198795 >> 172.21.15.78:0/485198795 conn(0x7f5f280718e0 msgr2=0x7f5f28071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 -- 172.21.15.78:0/485198795 shutdown_connections 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.473+0000 7f5f2d69a640 1 -- 172.21.15.78:0/485198795 wait complete. 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.474+0000 7f5f2d69a640 1 Processor -- start 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.474+0000 7f5f2d69a640 1 -- start start 2024-02-19T14:54:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f2d69a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f2d69a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f281e9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f2d69a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f2814f3a0 0x7f5f280ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f2d69a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280aafc0 con 0x7f5f280c12a0 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f2d69a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f281454b0 con 0x7f5f2814f3a0 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f2d69a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f281459b0 con 0x7f5f28072910 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f281e9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f281e9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56068/0 (socket says 172.21.15.78:56068) 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 -- 172.21.15.78:0/576736067 learned_addr learned my addr 172.21.15.78:0/576736067 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f277fe640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f2814f3a0 0x7f5f280ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f26ffd640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 -- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 msgr2=0x7f5f28144f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28144f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 -- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f2814f3a0 msgr2=0x7f5f280ab4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f2814f3a0 0x7f5f280ab4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 -- 172.21.15.78:0/576736067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1800b6a0 con 0x7f5f280c12a0 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f277fe640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f2814f3a0 0x7f5f280ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f267fc640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f281e9a70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5f18039040 tx=0x7f5f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.475+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1802ed30 con 0x7f5f280c12a0 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.476+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f18040840 con 0x7f5f280c12a0 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.476+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f281456d0 con 0x7f5f280c12a0 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.476+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18040bf0 con 0x7f5f280c12a0 2024-02-19T14:54:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.476+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f281ea800 con 0x7f5f280c12a0 2024-02-19T14:54:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.477+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f28074990 con 0x7f5f280c12a0 2024-02-19T14:54:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.477+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5f1803d050 con 0x7f5f280c12a0 2024-02-19T14:54:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.477+0000 7f5f0f7fe640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f1406b810 0x7f5f1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.478+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f5f180af210 con 0x7f5f280c12a0 2024-02-19T14:54:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.478+0000 7f5f26ffd640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f1406b810 0x7f5f1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.478+0000 7f5f26ffd640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f1406b810 0x7f5f1406dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000f60 tx=0x7f5f1c007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.482+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5f18079660 con 0x7f5f280c12a0 2024-02-19T14:54:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.717+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f280c6c50 con 0x7f5f280c12a0 2024-02-19T14:54:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.717+0000 7f5f0f7fe640 1 -- 172.21.15.78:0/576736067 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f180500a0 con 0x7f5f280c12a0 2024-02-19T14:54:04.721 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f1406b810 msgr2=0x7f5f1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f1406b810 0x7f5f1406dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000f60 tx=0x7f5f1c007c00 comp rx=0 tx=0).stop 2024-02-19T14:54:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 msgr2=0x7f5f281e9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f281e9a70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5f18039040 tx=0x7f5f180046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 shutdown_connections 2024-02-19T14:54:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f1406b810 0x7f5f1406dcd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f280c12a0 0x7f5f281e9a70 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f2814f3a0 0x7f5f280ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 --2- 172.21.15.78:0/576736067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f28072910 0x7f5f28144f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 >> 172.21.15.78:0/576736067 conn(0x7f5f280718e0 msgr2=0x7f5f28146b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 shutdown_connections 2024-02-19T14:54:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.719+0000 7f5f2d69a640 1 -- 172.21.15.78:0/576736067 wait complete. 2024-02-19T14:54:04.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:54:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.842+0000 7f06843e2640 1 Processor -- start 2024-02-19T14:54:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.842+0000 7f06843e2640 1 -- start start 2024-02-19T14:54:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f06843e2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f067c0c4980 0x7f067c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f06843e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 0x7f067c070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f06843e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c0ae670 con 0x7f067c0add70 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f06843e2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c0716b0 con 0x7f067c0c4980 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f06843e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c153780 con 0x7f067c0705b0 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068217e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f067c0add70 0x7f067c06eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47922/0 (socket says 172.21.15.78:47922) 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068217e640 1 -- 172.21.15.78:0/2925786416 learned_addr learned my addr 172.21.15.78:0/2925786416 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068197d640 1 --2- 172.21.15.78:0/2925786416 >> v2:172.21.15.150:3300/0 conn(0x7f067c0c4980 0x7f067c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068297f640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 0x7f067c070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854035508 0 0) 0x7f067c0ae670 con 0x7f067c0add70 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0668003520 con 0x7f067c0add70 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.844+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3204405773 0 0) 0x7f0668003520 con 0x7f067c0add70 2024-02-19T14:54:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 msgr2=0x7f067c070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 0x7f067c070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 >> v2:172.21.15.150:3300/0 conn(0x7f067c0c4980 msgr2=0x7f067c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 --2- 172.21.15.78:0/2925786416 >> v2:172.21.15.150:3300/0 conn(0x7f067c0c4980 0x7f067c0c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c153a60 con 0x7f067c0add70 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0678001670 con 0x7f067c0add70 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 >> v1:172.21.15.78:6789/0 conn(0x7f067c0add70 legacy=0x7f067c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 0x7f0668003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0668003ef0 0x7f06680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668004d00 0x7f0668005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c0716b0 con 0x7f0668003ef0 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c153780 con 0x7f0668004d00 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066800b9d0 con 0x7f067c0705b0 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068197d640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0668003ef0 0x7f06680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068217e640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 0x7f0668003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068297f640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668004d00 0x7f0668005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068197d640 1 -- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 msgr2=0x7f0668003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068197d640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0705b0 0x7f0668003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068197d640 1 -- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668004d00 msgr2=0x7f0668005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068197d640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668004d00 0x7f0668005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.845+0000 7f068197d640 1 -- 172.21.15.78:0/2925786416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0668003520 con 0x7f0668003ef0 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.846+0000 7f068197d640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0668003ef0 0x7f06680047c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f066c009af0 tx=0x7f06680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbea5c313eefcf59 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.846+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c02c710 con 0x7f0668003ef0 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.846+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c02c870 con 0x7f0668003ef0 2024-02-19T14:54:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.846+0000 7f068117c640 1 -- 172.21.15.78:0/2925786416 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c02cc00 con 0x7f0668003ef0 2024-02-19T14:54:04.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f06843e2640 1 -- 172.21.15.78:0/2925786416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0668003ef0 msgr2=0x7f06680047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f06843e2640 1 --2- 172.21.15.78:0/2925786416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0668003ef0 0x7f06680047c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f066c009af0 tx=0x7f06680057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f068297f640 1 -- 172.21.15.78:0/2925786416 reap_dead start 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f06843e2640 1 -- 172.21.15.78:0/2925786416 shutdown_connections 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f06843e2640 1 -- 172.21.15.78:0/2925786416 >> 172.21.15.78:0/2925786416 conn(0x7f067c0c85e0 msgr2=0x7f067c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f06843e2640 1 -- 172.21.15.78:0/2925786416 shutdown_connections 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.847+0000 7f06843e2640 1 -- 172.21.15.78:0/2925786416 wait complete. 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.848+0000 7f06843e2640 1 Processor -- start 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.848+0000 7f06843e2640 1 -- start start 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.848+0000 7f06843e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668003ef0 0x7f067c1eca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.848+0000 7f06843e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0c1960 0x7f067c14e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f06843e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 0x7f067c0c0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f06843e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c0c1de0 con 0x7f067c1e8950 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f06843e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c154b10 con 0x7f0668003ef0 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f06843e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c0c2020 con 0x7f067c0c1960 2024-02-19T14:54:04.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 0x7f067c0c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 0x7f067c0c0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56082/0 (socket says 172.21.15.78:56082) 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 -- 172.21.15.78:0/528912916 learned_addr learned my addr 172.21.15.78:0/528912916 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068217e640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668003ef0 0x7f067c1eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068197d640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0c1960 0x7f067c14e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 -- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0c1960 msgr2=0x7f067c14e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0c1960 0x7f067c14e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 -- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668003ef0 msgr2=0x7f067c1eca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668003ef0 0x7f067c1eca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 -- 172.21.15.78:0/528912916 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c1ebf60 con 0x7f067c1e8950 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068217e640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668003ef0 0x7f067c1eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f068297f640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 0x7f067c0c0780 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f067c0c2a10 tx=0x7f067400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067400e040 con 0x7f067c1e8950 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067400f040 con 0x7f067c1e8950 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c1ec240 con 0x7f067c1e8950 2024-02-19T14:54:04.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674015430 con 0x7f067c1e8950 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.849+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c14ee50 con 0x7f067c1e8950 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.851+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c1ec760 con 0x7f067c1e8950 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.851+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0674015590 con 0x7f067c1e8950 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.851+0000 7f0672ffd640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f066006b840 0x7f066006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.852+0000 7f068217e640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f066006b840 0x7f066006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.852+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f067408b0b0 con 0x7f067c1e8950 2024-02-19T14:54:04.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.852+0000 7f068217e640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f066006b840 0x7f066006dd00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f067c157260 tx=0x7f0678009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:04.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:04.856+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f067c1ec760 con 0x7f067c1e8950 2024-02-19T14:54:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.119+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f067c1ec760 con 0x7f067c1e8950 2024-02-19T14:54:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.120+0000 7f0672ffd640 1 -- 172.21.15.78:0/528912916 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f067c1ec760 con 0x7f067c1e8950 2024-02-19T14:54:05.123 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f066006b840 msgr2=0x7f066006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f066006b840 0x7f066006dd00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f067c157260 tx=0x7f0678009040 comp rx=0 tx=0).stop 2024-02-19T14:54:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 msgr2=0x7f067c0c0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 0x7f067c0c0780 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f067c0c2a10 tx=0x7f067400f580 comp rx=0 tx=0).stop 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 shutdown_connections 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f066006b840 0x7f066006dd00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c1e8950 0x7f067c0c0780 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f067c0c1960 0x7f067c14e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 --2- 172.21.15.78:0/528912916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0668003ef0 0x7f067c1eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.121+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 >> 172.21.15.78:0/528912916 conn(0x7f067c0c85e0 msgr2=0x7f067c0a91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.122+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 shutdown_connections 2024-02-19T14:54:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.122+0000 7f06843e2640 1 -- 172.21.15.78:0/528912916 wait complete. 2024-02-19T14:54:05.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:54:05.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.236+0000 7fd17741c640 1 Processor -- start 2024-02-19T14:54:05.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.236+0000 7fd17741c640 1 -- start start 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd17741c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd170152700 0x7fd1700c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd17741c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1701551f0 con 0x7fd1700c5730 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd17741c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd170155490 con 0x7fd1700c8e00 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd17741c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd170155770 con 0x7fd170152700 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd1749b7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd1700c5730 0x7fd17006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47938/0 (socket says 172.21.15.78:47938) 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd1749b7640 1 -- 172.21.15.78:0/1219366211 learned_addr learned my addr 172.21.15.78:0/1219366211 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd1751b8640 1 --2- 172.21.15.78:0/1219366211 >> v2:172.21.15.150:3300/0 conn(0x7fd170152700 0x7fd1700c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd167fff640 1 -- 172.21.15.78:0/1219366211 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182631457 0 0) 0x7fd1701551f0 con 0x7fd1700c5730 2024-02-19T14:54:05.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd167fff640 1 -- 172.21.15.78:0/1219366211 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c003520 con 0x7fd1700c5730 2024-02-19T14:54:05.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd167fff640 1 -- 172.21.15.78:0/1219366211 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848321619 0 0) 0x7fd15c003520 con 0x7fd1700c5730 2024-02-19T14:54:05.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd1751b8640 1 -- 172.21.15.78:0/1219366211 >> v1:172.21.15.150:6789/0 conn(0x7fd1700c8e00 legacy=0x7fd17015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd1751b8640 1 -- 172.21.15.78:0/1219366211 >> v1:172.21.15.78:6789/0 conn(0x7fd1700c5730 legacy=0x7fd17006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.238+0000 7fd1751b8640 1 -- 172.21.15.78:0/1219366211 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170155a50 con 0x7fd170152700 2024-02-19T14:54:05.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.239+0000 7fd1751b8640 1 --2- 172.21.15.78:0/1219366211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd170152700 0x7fd1700c51f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0097a0 tx=0x7fd16c02b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb878ccf32400218 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:05.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.239+0000 7fd167fff640 1 -- 172.21.15.78:0/1219366211 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160001670 con 0x7fd170152700 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.239+0000 7fd167fff640 1 -- 172.21.15.78:0/1219366211 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c02e050 con 0x7fd170152700 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.239+0000 7fd167fff640 1 -- 172.21.15.78:0/1219366211 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c030510 con 0x7fd170152700 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 -- 172.21.15.78:0/1219366211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd170152700 msgr2=0x7fd1700c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 --2- 172.21.15.78:0/1219366211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd170152700 0x7fd1700c51f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0097a0 tx=0x7fd16c02b600 comp rx=0 tx=0).stop 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 -- 172.21.15.78:0/1219366211 shutdown_connections 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 --2- 172.21.15.78:0/1219366211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd170152700 0x7fd1700c51f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 -- 172.21.15.78:0/1219366211 >> 172.21.15.78:0/1219366211 conn(0x7fd1700cac90 msgr2=0x7fd1700cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 -- 172.21.15.78:0/1219366211 shutdown_connections 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.240+0000 7fd17741c640 1 -- 172.21.15.78:0/1219366211 wait complete. 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.241+0000 7fd17741c640 1 Processor -- start 2024-02-19T14:54:05.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.241+0000 7fd17741c640 1 -- start start 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd17741c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1700c5730 0x7fd1700ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd17741c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1700c8e00 0x7fd1701ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd17741c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 0x7fd1700c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd17741c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd170151dc0 con 0x7fd170152700 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd17741c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700accd0 con 0x7fd1700c5730 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd17741c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700adea0 con 0x7fd1700c8e00 2024-02-19T14:54:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1759b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 0x7fd1700c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1759b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 0x7fd1700c2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56086/0 (socket says 172.21.15.78:56086) 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1759b9640 1 -- 172.21.15.78:0/926877153 learned_addr learned my addr 172.21.15.78:0/926877153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1749b7640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1700c8e00 0x7fd1701ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1751b8640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1700c5730 0x7fd1700ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1759b9640 1 -- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1700c8e00 msgr2=0x7fd1701ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.242+0000 7fd1759b9640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1700c8e00 0x7fd1701ee360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd1759b9640 1 -- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1700c5730 msgr2=0x7fd1700ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd1759b9640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1700c5730 0x7fd1700ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd1759b9640 1 -- 172.21.15.78:0/926877153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009400 con 0x7fd170152700 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd1759b9640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 0x7fd1700c2670 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd16800c8e0 tx=0x7fd16800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168003da0 con 0x7fd170152700 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd168005070 con 0x7fd170152700 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1701eeba0 con 0x7fd170152700 2024-02-19T14:54:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168005440 con 0x7fd170152700 2024-02-19T14:54:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.243+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1700ae0f0 con 0x7fd170152700 2024-02-19T14:54:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.244+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd170074990 con 0x7fd170152700 2024-02-19T14:54:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.245+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd1680055a0 con 0x7fd170152700 2024-02-19T14:54:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.245+0000 7fd165ffb640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd15406b810 0x7fd15406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.245+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7fd16808a1d0 con 0x7fd170152700 2024-02-19T14:54:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.245+0000 7fd1751b8640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd15406b810 0x7fd15406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.246+0000 7fd1751b8640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd15406b810 0x7fd15406dcd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd16c034000 tx=0x7fd16c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.254+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1680546d0 con 0x7fd170152700 2024-02-19T14:54:05.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.564+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1700b04c0 con 0x7fd170152700 2024-02-19T14:54:05.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.565+0000 7fd165ffb640 1 -- 172.21.15.78:0/926877153 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd1680170a0 con 0x7fd170152700 2024-02-19T14:54:05.568 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd15406b810 msgr2=0x7fd15406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd15406b810 0x7fd15406dcd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd16c034000 tx=0x7fd16c008030 comp rx=0 tx=0).stop 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 msgr2=0x7fd1700c2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 0x7fd1700c2670 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd16800c8e0 tx=0x7fd16800cda0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 shutdown_connections 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd15406b810 0x7fd15406dcd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd170152700 0x7fd1700c2670 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1700c8e00 0x7fd1701ee360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 --2- 172.21.15.78:0/926877153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1700c5730 0x7fd1700ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 >> 172.21.15.78:0/926877153 conn(0x7fd1700cac90 msgr2=0x7fd17014fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:05.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 shutdown_connections 2024-02-19T14:54:05.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.567+0000 7fd17741c640 1 -- 172.21.15.78:0/926877153 wait complete. 2024-02-19T14:54:05.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-19T14:54:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.687+0000 7f41bdf0c640 1 Processor -- start 2024-02-19T14:54:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.687+0000 7f41bdf0c640 1 -- start start 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41bdf0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 0x7f41b80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41bdf0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41b80c8e00 0x7f41b815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41bdf0c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b81551f0 con 0x7f41b80c5730 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41bdf0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b8155490 con 0x7f41b80c8e00 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41bdf0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8155770 con 0x7f41b8152700 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41b6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f41b80c5730 0x7f41b806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47940/0 (socket says 172.21.15.78:47940) 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.688+0000 7f41b6ffd640 1 -- 172.21.15.78:0/3333633740 learned_addr learned my addr 172.21.15.78:0/3333633740 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284593598 0 0) 0x7f41b81551f0 con 0x7f41b80c5730 2024-02-19T14:54:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b7fff640 1 --2- 172.21.15.78:0/3333633740 >> v2:172.21.15.150:3300/0 conn(0x7f41b80c8e00 0x7f41b815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b77fe640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 0x7f41b80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0003520 con 0x7f41b80c5730 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2536542888 0 0) 0x7f41a0003520 con 0x7f41b80c5730 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 msgr2=0x7f41b80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 0x7f41b80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 >> v2:172.21.15.150:3300/0 conn(0x7f41b80c8e00 msgr2=0x7f41b815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 --2- 172.21.15.78:0/3333633740 >> v2:172.21.15.150:3300/0 conn(0x7f41b80c8e00 0x7f41b815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b8155a50 con 0x7f41b80c5730 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f41ac001670 con 0x7f41b80c5730 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 >> v1:172.21.15.78:6789/0 conn(0x7f41b80c5730 legacy=0x7f41b806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a0003970 0x7f41a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 0x7f41a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8155490 con 0x7f41a0004d00 2024-02-19T14:54:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8155770 con 0x7f41a0003970 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a000b980 con 0x7f41b8152700 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b6ffd640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 0x7f41a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b7fff640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b77fe640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a0003970 0x7f41a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b7fff640 1 -- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 msgr2=0x7f41a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.689+0000 7f41b7fff640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b8152700 0x7f41a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b7fff640 1 -- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a0003970 msgr2=0x7f41a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b7fff640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a0003970 0x7f41a0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b7fff640 1 -- 172.21.15.78:0/3333633740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0003520 con 0x7f41a0004d00 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b7fff640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009fc0 tx=0x7f41a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff476c11d6f64b31 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b002cc60 con 0x7f41a0004d00 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b000f040 con 0x7f41a0004d00 2024-02-19T14:54:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.690+0000 7f41b67fc640 1 -- 172.21.15.78:0/3333633740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0034440 con 0x7f41a0004d00 2024-02-19T14:54:05.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41bdf0c640 1 -- 172.21.15.78:0/3333633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41a0004d00 msgr2=0x7f41a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/3333633740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009fc0 tx=0x7f41a00057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41b7fff640 1 -- 172.21.15.78:0/3333633740 reap_dead start 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41bdf0c640 1 -- 172.21.15.78:0/3333633740 shutdown_connections 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41bdf0c640 1 -- 172.21.15.78:0/3333633740 >> 172.21.15.78:0/3333633740 conn(0x7f41b80cac90 msgr2=0x7f41b80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41bdf0c640 1 -- 172.21.15.78:0/3333633740 shutdown_connections 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.691+0000 7f41bdf0c640 1 -- 172.21.15.78:0/3333633740 wait complete. 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.692+0000 7f41bdf0c640 1 Processor -- start 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 -- start start 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41b80c3420 0x7f41b8158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 0x7f41b81e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b81e7f60 0x7f41b81567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b81e83e0 con 0x7f41b80bdcb0 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80c2f80 con 0x7f41b80c3420 2024-02-19T14:54:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41bdf0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b80c26e0 con 0x7f41b81e7f60 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.693+0000 7f41b6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 0x7f41b81e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 0x7f41b81e7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56108/0 (socket says 172.21.15.78:56108) 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 -- 172.21.15.78:0/2022198823 learned_addr learned my addr 172.21.15.78:0/2022198823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b77fe640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41b80c3420 0x7f41b8158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b7fff640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b81e7f60 0x7f41b81567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 -- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b81e7f60 msgr2=0x7f41b81567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b81e7f60 0x7f41b81567c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 -- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41b80c3420 msgr2=0x7f41b8158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41b80c3420 0x7f41b8158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 -- 172.21.15.78:0/2022198823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80c29c0 con 0x7f41b80bdcb0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b77fe640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41b80c3420 0x7f41b8158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41b6ffd640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 0x7f41b81e7a20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f41ac007290 tx=0x7f41ac007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00f940 con 0x7f41b80bdcb0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac00faa0 con 0x7f41b80bdcb0 2024-02-19T14:54:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b80c2ca0 con 0x7f41b80bdcb0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00fe50 con 0x7f41b80bdcb0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.694+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b80c3800 con 0x7f41b80bdcb0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.695+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b80c3a20 con 0x7f41b80bdcb0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.696+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f41ac01d460 con 0x7f41b80bdcb0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.696+0000 7f419ffff640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41a406b810 0x7f41a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.696+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f41ac08a8f0 con 0x7f41b80bdcb0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.696+0000 7f41b77fe640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41a406b810 0x7f41a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.696+0000 7f41b77fe640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41a406b810 0x7f41a406dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009bc0 tx=0x7f41a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.701+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f41b80c3a20 con 0x7f41b80bdcb0 2024-02-19T14:54:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.917+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f41b80c3a20 con 0x7f41b80bdcb0 2024-02-19T14:54:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.918+0000 7f419ffff640 1 -- 172.21.15.78:0/2022198823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v59) v1 ==== 105+0+11 (secure 0 0 0) 0x7f41b80c3a20 con 0x7f41b80bdcb0 2024-02-19T14:54:05.921 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41a406b810 msgr2=0x7f41a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41a406b810 0x7f41a406dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009bc0 tx=0x7f41a8008040 comp rx=0 tx=0).stop 2024-02-19T14:54:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 msgr2=0x7f41b81e7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 0x7f41b81e7a20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f41ac007290 tx=0x7f41ac007750 comp rx=0 tx=0).stop 2024-02-19T14:54:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 shutdown_connections 2024-02-19T14:54:05.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f41a406b810 0x7f41a406dcd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41b80bdcb0 0x7f41b81e7a20 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41b81e7f60 0x7f41b81567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 --2- 172.21.15.78:0/2022198823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41b80c3420 0x7f41b8158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.919+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 >> 172.21.15.78:0/2022198823 conn(0x7f41b80cac90 msgr2=0x7f41b80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.920+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 shutdown_connections 2024-02-19T14:54:05.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:05.920+0000 7f41bdf0c640 1 -- 172.21.15.78:0/2022198823 wait complete. 2024-02-19T14:54:05.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-19T14:54:06.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.075+0000 7f6a26748640 1 Processor -- start 2024-02-19T14:54:06.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.075+0000 7f6a26748640 1 -- start start 2024-02-19T14:54:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a26748640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a2006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a26748640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a2006d910 0x7f6a2006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a26748640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2006cd20 con 0x7f6a20152700 2024-02-19T14:54:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a26748640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a201551f0 con 0x7f6a2006d910 2024-02-19T14:54:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a26748640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20155490 con 0x7f6a200c8e00 2024-02-19T14:54:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a24ce5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a2006d910 0x7f6a2006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a1ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6a20152700 0x7f6a200c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47946/0 (socket says 172.21.15.78:47946) 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a1ffff640 1 -- 172.21.15.78:0/3045520528 learned_addr learned my addr 172.21.15.78:0/3045520528 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a24ce5640 1 -- 172.21.15.78:0/3045520528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 msgr2=0x7f6a2006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a1f7fe640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a2006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a24ce5640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a2006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a1effd640 1 -- 172.21.15.78:0/3045520528 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793576015 0 0) 0x7f6a2006cd20 con 0x7f6a20152700 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a24ce5640 1 -- 172.21.15.78:0/3045520528 >> v1:172.21.15.78:6789/0 conn(0x7f6a20152700 legacy=0x7f6a200c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.076+0000 7f6a24ce5640 1 -- 172.21.15.78:0/3045520528 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20155770 con 0x7f6a2006d910 2024-02-19T14:54:06.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.077+0000 7f6a1f7fe640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a2006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:06.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.077+0000 7f6a24ce5640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a2006c7e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009d90 tx=0x7f6a1802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739627d8bfcd4e89 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.077+0000 7f6a1effd640 1 -- 172.21.15.78:0/3045520528 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1802cbc0 con 0x7f6a2006d910 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.077+0000 7f6a1effd640 1 -- 172.21.15.78:0/3045520528 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1802cd20 con 0x7f6a2006d910 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.077+0000 7f6a1effd640 1 -- 172.21.15.78:0/3045520528 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18034660 con 0x7f6a2006d910 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 -- 172.21.15.78:0/3045520528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 msgr2=0x7f6a2006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a2006c7e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009d90 tx=0x7f6a1802be10 comp rx=0 tx=0).stop 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 -- 172.21.15.78:0/3045520528 shutdown_connections 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a2006c7e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 --2- 172.21.15.78:0/3045520528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a2006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 -- 172.21.15.78:0/3045520528 >> 172.21.15.78:0/3045520528 conn(0x7f6a200cac90 msgr2=0x7f6a200cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 -- 172.21.15.78:0/3045520528 shutdown_connections 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.078+0000 7f6a26748640 1 -- 172.21.15.78:0/3045520528 wait complete. 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.079+0000 7f6a26748640 1 Processor -- start 2024-02-19T14:54:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.079+0000 7f6a26748640 1 -- start start 2024-02-19T14:54:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a26748640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a200c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a26748640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a200ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a26748640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a20152700 0x7f6a20156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a26748640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201511b0 con 0x7f6a2006d910 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a26748640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201ee4e0 con 0x7f6a20152700 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a26748640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201eea10 con 0x7f6a200c8e00 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a200c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a200c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56128/0 (socket says 172.21.15.78:56128) 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 -- 172.21.15.78:0/3696555009 learned_addr learned my addr 172.21.15.78:0/3696555009 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1f7fe640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a200ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a24ce5640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a20152700 0x7f6a20156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 -- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 msgr2=0x7f6a200ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a200ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 -- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a20152700 msgr2=0x7f6a20156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a20152700 0x7f6a20156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.080+0000 7f6a1ffff640 1 -- 172.21.15.78:0/3696555009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009a70 con 0x7f6a2006d910 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.081+0000 7f6a1ffff640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a200c3530 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000c430 tx=0x7f6a1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.081+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100074e0 con 0x7f6a2006d910 2024-02-19T14:54:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.081+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a10007660 con 0x7f6a2006d910 2024-02-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.081+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10004a20 con 0x7f6a2006d910 2024-02-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.081+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a201ee730 con 0x7f6a2006d910 2024-02-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.081+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a20159250 con 0x7f6a2006d910 2024-02-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.082+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a20152ae0 con 0x7f6a2006d910 2024-02-19T14:54:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.083+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6a10004b80 con 0x7f6a2006d910 2024-02-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.083+0000 7f6a1cff9640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a0c06b840 0x7f6a0c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.083+0000 7f6a1f7fe640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a0c06b840 0x7f6a0c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.083+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f6a1008a390 con 0x7f6a2006d910 2024-02-19T14:54:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.084+0000 7f6a1f7fe640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a0c06b840 0x7f6a0c06dd00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a200ca100 tx=0x7f6a14008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.088+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6a20152ae0 con 0x7f6a2006d910 2024-02-19T14:54:06.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.389+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f6a20152ae0 con 0x7f6a2006d910 2024-02-19T14:54:06.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.390+0000 7f6a1cff9640 1 -- 172.21.15.78:0/3696555009 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v59) v1 ==== 101+0+11 (secure 0 0 0) 0x7f6a20152ae0 con 0x7f6a2006d910 2024-02-19T14:54:06.393 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a0c06b840 msgr2=0x7f6a0c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a0c06b840 0x7f6a0c06dd00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6a200ca100 tx=0x7f6a14008040 comp rx=0 tx=0).stop 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 msgr2=0x7f6a200c3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a200c3530 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000c430 tx=0x7f6a1000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 shutdown_connections 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a0c06b840 0x7f6a0c06dd00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a20152700 0x7f6a20156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a200c8e00 0x7f6a200ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 --2- 172.21.15.78:0/3696555009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a2006d910 0x7f6a200c3530 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 >> 172.21.15.78:0/3696555009 conn(0x7f6a200cac90 msgr2=0x7f6a20150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 shutdown_connections 2024-02-19T14:54:06.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.392+0000 7f6a26748640 1 -- 172.21.15.78:0/3696555009 wait complete. 2024-02-19T14:54:06.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:06.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.526+0000 7f4b2d7a2640 1 Processor -- start 2024-02-19T14:54:06.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.526+0000 7f4b2d7a2640 1 -- start start 2024-02-19T14:54:06.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2d7a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2806d740 0x7f4b281559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2d7a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 0x7f4b280c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2d7a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2806db80 con 0x7f4b281524b0 2024-02-19T14:54:06.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2d7a2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b280c8d50 con 0x7f4b280c7150 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2d7a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280c7580 con 0x7f4b2806d740 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2806d740 0x7f4b281559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2cfa1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 0x7f4b280c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b27fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4b281524b0 0x7f4b280c8640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53248/0 (socket says 172.21.15.78:53248) 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b27fff640 1 -- 172.21.15.78:0/2984895404 learned_addr learned my addr 172.21.15.78:0/2984895404 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.527+0000 7f4b2cfa1640 1 --2- 172.21.15.78:0/2984895404 >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 0x7f4b280c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:44260/0 (socket says 172.21.15.78:44260) 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.528+0000 7f4b277fe640 1 -- 172.21.15.78:0/2984895404 >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 msgr2=0x7f4b280c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.528+0000 7f4b277fe640 1 --2- 172.21.15.78:0/2984895404 >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 0x7f4b280c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.528+0000 7f4b277fe640 1 -- 172.21.15.78:0/2984895404 >> v1:172.21.15.150:6789/0 conn(0x7f4b281524b0 legacy=0x7f4b280c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.528+0000 7f4b277fe640 1 -- 172.21.15.78:0/2984895404 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280c7850 con 0x7f4b2806d740 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.528+0000 7f4b26ffd640 1 -- 172.21.15.78:0/2984895404 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056453062 0 0) 0x7f4b2806db80 con 0x7f4b281524b0 2024-02-19T14:54:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.528+0000 7f4b2cfa1640 1 --2- 172.21.15.78:0/2984895404 >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 0x7f4b280c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.529+0000 7f4b277fe640 1 --2- 172.21.15.78:0/2984895404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2806d740 0x7f4b281559c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0097a0 tx=0x7f4b1c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9cf4fb82bef3ee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.529+0000 7f4b26ffd640 1 -- 172.21.15.78:0/2984895404 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c02e070 con 0x7f4b2806d740 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.529+0000 7f4b26ffd640 1 -- 172.21.15.78:0/2984895404 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c0315a0 con 0x7f4b2806d740 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.529+0000 7f4b26ffd640 1 -- 172.21.15.78:0/2984895404 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c031930 con 0x7f4b2806d740 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/2984895404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2806d740 msgr2=0x7f4b281559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/2984895404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2806d740 0x7f4b281559c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0097a0 tx=0x7f4b1c02b840 comp rx=0 tx=0).stop 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/2984895404 shutdown_connections 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/2984895404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2806d740 0x7f4b281559c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/2984895404 >> v2:172.21.15.150:3300/0 conn(0x7f4b280c7150 0x7f4b280c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/2984895404 >> 172.21.15.78:0/2984895404 conn(0x7f4b280ac250 msgr2=0x7f4b280ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/2984895404 shutdown_connections 2024-02-19T14:54:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.530+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/2984895404 wait complete. 2024-02-19T14:54:06.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.531+0000 7f4b2d7a2640 1 Processor -- start 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.531+0000 7f4b2d7a2640 1 -- start start 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.531+0000 7f4b2d7a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 0x7f4b28154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.531+0000 7f4b2d7a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b280c7150 0x7f4b280be860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.531+0000 7f4b2d7a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b281524b0 0x7f4b2814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.531+0000 7f4b2d7a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280746c0 con 0x7f4b2806d740 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b2d7a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280c39b0 con 0x7f4b281524b0 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b2d7a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280c3c90 con 0x7f4b280c7150 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 0x7f4b28154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 0x7f4b28154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56144/0 (socket says 172.21.15.78:56144) 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 -- 172.21.15.78:0/4036022359 learned_addr learned my addr 172.21.15.78:0/4036022359 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b2cfa1640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b281524b0 0x7f4b2814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b277fe640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b280c7150 0x7f4b280be860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 -- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b280c7150 msgr2=0x7f4b280be860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b280c7150 0x7f4b280be860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 -- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b281524b0 msgr2=0x7f4b2814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b281524b0 0x7f4b2814a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 -- 172.21.15.78:0/4036022359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b20009a70 con 0x7f4b2806d740 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b2cfa1640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b281524b0 0x7f4b2814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b277fe640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b280c7150 0x7f4b280be860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b27fff640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 0x7f4b28154070 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c430 tx=0x7f4b1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b180074e0 con 0x7f4b2806d740 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18007660 con 0x7f4b2806d740 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c009400 con 0x7f4b2806d740 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18004a20 con 0x7f4b2806d740 2024-02-19T14:54:06.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.532+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b280c4250 con 0x7f4b2806d740 2024-02-19T14:54:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.534+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b28074990 con 0x7f4b2806d740 2024-02-19T14:54:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.534+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b18004b80 con 0x7f4b2806d740 2024-02-19T14:54:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.534+0000 7f4b24ff9640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b810 0x7f4b1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.534+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f4b1808a5c0 con 0x7f4b2806d740 2024-02-19T14:54:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.534+0000 7f4b277fe640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b810 0x7f4b1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.535+0000 7f4b277fe640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b810 0x7f4b1406dcd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036000 tx=0x7f4b1c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.539+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b18054a20 con 0x7f4b2806d740 2024-02-19T14:54:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.753+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b280c7df0 con 0x7f4b2806d740 2024-02-19T14:54:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.754+0000 7f4b24ff9640 1 -- 172.21.15.78:0/4036022359 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6185 (secure 0 0 0) 0x7f4b180580e0 con 0x7f4b2806d740 2024-02-19T14:54:06.758 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:06.758 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:54:02.288421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6042},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6042":{"gid":6042,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1226290552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1226290552},{"type":"v1","addr":"172.21.15.78:6837","nonce":1226290552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 39 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b810 msgr2=0x7f4b1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b810 0x7f4b1406dcd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036000 tx=0x7f4b1c008030 comp rx=0 tx=0).stop 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 msgr2=0x7f4b28154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 0x7f4b28154070 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c430 tx=0x7f4b1800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 shutdown_connections 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b810 0x7f4b1406dcd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b281524b0 0x7f4b2814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b280c7150 0x7f4b280be860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 --2- 172.21.15.78:0/4036022359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2806d740 0x7f4b28154070 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 >> 172.21.15.78:0/4036022359 conn(0x7f4b280ac250 msgr2=0x7f4b28150150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:06.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 shutdown_connections 2024-02-19T14:54:06.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.756+0000 7f4b2d7a2640 1 -- 172.21.15.78:0/4036022359 wait complete. 2024-02-19T14:54:06.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:06.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 Processor -- start 2024-02-19T14:54:06.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 -- start start 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c4d80 0x7f98ac159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98ac155010 0x7f98ac1523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac06ffa0 con 0x7f98ac0c78a0 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac06b720 con 0x7f98ac155010 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98b1d19640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac06fc70 con 0x7f98ac0c4d80 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.869+0000 7f98abfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98ac155010 0x7f98ac1523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98abfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98ac155010 0x7f98ac1523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56160/0 (socket says 172.21.15.78:56160) 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98abfff640 1 -- 172.21.15.78:0/1736317738 learned_addr learned my addr 172.21.15.78:0/1736317738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98aaffd640 1 --2- 172.21.15.78:0/1736317738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c4d80 0x7f98ac159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98abfff640 1 -- 172.21.15.78:0/1736317738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c4d80 msgr2=0x7f98ac159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98abfff640 1 --2- 172.21.15.78:0/1736317738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c4d80 0x7f98ac159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98abfff640 1 -- 172.21.15.78:0/1736317738 >> v1:172.21.15.150:6789/0 conn(0x7f98ac0c78a0 legacy=0x7f98ac0c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.870+0000 7f98abfff640 1 -- 172.21.15.78:0/1736317738 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac06b930 con 0x7f98ac155010 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98abfff640 1 --2- 172.21.15.78:0/1736317738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1523d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009e50 tx=0x7f98a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a05f70c5347eae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98aa7fc640 1 -- 172.21.15.78:0/1736317738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a402ca70 con 0x7f98ac155010 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98aa7fc640 1 -- 172.21.15.78:0/1736317738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a402cbd0 con 0x7f98ac155010 2024-02-19T14:54:06.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98aa7fc640 1 -- 172.21.15.78:0/1736317738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a40343f0 con 0x7f98ac155010 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 -- 172.21.15.78:0/1736317738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 msgr2=0x7f98ac1523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 --2- 172.21.15.78:0/1736317738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1523d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009e50 tx=0x7f98a402bed0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 -- 172.21.15.78:0/1736317738 shutdown_connections 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 --2- 172.21.15.78:0/1736317738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1523d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 --2- 172.21.15.78:0/1736317738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c4d80 0x7f98ac159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 -- 172.21.15.78:0/1736317738 >> 172.21.15.78:0/1736317738 conn(0x7f98ac066e70 msgr2=0x7f98ac0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 -- 172.21.15.78:0/1736317738 shutdown_connections 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.871+0000 7f98b1d19640 1 -- 172.21.15.78:0/1736317738 wait complete. 2024-02-19T14:54:06.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.872+0000 7f98b1d19640 1 Processor -- start 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 -- start start 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98ac0c4d80 0x7f98ac0c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c78a0 0x7f98ac141e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0c2660 con 0x7f98ac155010 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac158470 con 0x7f98ac0c4d80 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98b1d19640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0c21d0 con 0x7f98ac0c78a0 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98abfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98abfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56162/0 (socket says 172.21.15.78:56162) 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.873+0000 7f98abfff640 1 -- 172.21.15.78:0/688932849 learned_addr learned my addr 172.21.15.78:0/688932849 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98ab7fe640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98ac0c4d80 0x7f98ac0c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98aaffd640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c78a0 0x7f98ac141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98abfff640 1 -- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c78a0 msgr2=0x7f98ac141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98abfff640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c78a0 0x7f98ac141e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98abfff640 1 -- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98ac0c4d80 msgr2=0x7f98ac0c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98abfff640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98ac0c4d80 0x7f98ac0c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98abfff640 1 -- 172.21.15.78:0/688932849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4009b30 con 0x7f98ac155010 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98abfff640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1423c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009e50 tx=0x7f98a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4009510 con 0x7f98ac155010 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a403e450 con 0x7f98ac155010 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac0aae40 con 0x7f98ac155010 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a403e7e0 con 0x7f98ac155010 2024-02-19T14:54:06.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.874+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac0aab60 con 0x7f98ac155010 2024-02-19T14:54:06.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.875+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac074990 con 0x7f98ac155010 2024-02-19T14:54:06.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.875+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f98a403e940 con 0x7f98ac155010 2024-02-19T14:54:06.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.876+0000 7f9893fff640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f989806b810 0x7f989806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:06.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.876+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f98a40ad7f0 con 0x7f98ac155010 2024-02-19T14:54:06.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.876+0000 7f98ab7fe640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f989806b810 0x7f989806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:06.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.876+0000 7f98ab7fe640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f989806b810 0x7f989806dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f989c007fd0 tx=0x7f989c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:06.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:06.884+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98a4077cd0 con 0x7f98ac155010 2024-02-19T14:54:07.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.063+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98ac0c1fa0 con 0x7f98ac155010 2024-02-19T14:54:07.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.064+0000 7f9893fff640 1 -- 172.21.15.78:0/688932849 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+21009 (secure 0 0 0) 0x7f98a407b390 con 0x7f98ac155010 2024-02-19T14:54:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:07.068 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":59,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:00.278788+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:07.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f989806b810 msgr2=0x7f989806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f989806b810 0x7f989806dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f989c007fd0 tx=0x7f989c009040 comp rx=0 tx=0).stop 2024-02-19T14:54:07.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 msgr2=0x7f98ac1423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1423c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009e50 tx=0x7f98a4004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 shutdown_connections 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f989806b810 0x7f989806dcd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98ac155010 0x7f98ac1423c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98ac0c78a0 0x7f98ac141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 --2- 172.21.15.78:0/688932849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98ac0c4d80 0x7f98ac0c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.066+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 >> 172.21.15.78:0/688932849 conn(0x7f98ac066e70 msgr2=0x7f98ac1523c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.067+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 shutdown_connections 2024-02-19T14:54:07.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.067+0000 7f98b1d19640 1 -- 172.21.15.78:0/688932849 wait complete. 2024-02-19T14:54:07.085 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:54:07.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:07.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.182+0000 7f16e169a640 1 Processor -- start 2024-02-19T14:54:07.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.182+0000 7f16e169a640 1 -- start start 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.183+0000 7f16e169a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16dc14f1a0 0x7f16dc072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.183+0000 7f16e169a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06ea50 0x7f16dc152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.183+0000 7f16e169a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16dc0c5970 con 0x7f16dc06fbc0 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.183+0000 7f16e169a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16dc0c5750 con 0x7f16dc14f1a0 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.183+0000 7f16e169a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0c5bc0 con 0x7f16dc06ea50 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16daffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16dc14f1a0 0x7f16dc072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16da7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06ea50 0x7f16dc152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16daffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16dc14f1a0 0x7f16dc072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:44294/0 (socket says 172.21.15.78:44294) 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16db7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f16dc06fbc0 0x7f16dc0c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53258/0 (socket says 172.21.15.78:53258) 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16daffd640 1 -- 172.21.15.78:0/499619716 learned_addr learned my addr 172.21.15.78:0/499619716 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16da7fc640 1 -- 172.21.15.78:0/499619716 >> v2:172.21.15.150:3300/0 conn(0x7f16dc14f1a0 msgr2=0x7f16dc072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16da7fc640 1 --2- 172.21.15.78:0/499619716 >> v2:172.21.15.150:3300/0 conn(0x7f16dc14f1a0 0x7f16dc072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16da7fc640 1 -- 172.21.15.78:0/499619716 >> v1:172.21.15.150:6789/0 conn(0x7f16dc06fbc0 legacy=0x7f16dc0c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.184+0000 7f16da7fc640 1 -- 172.21.15.78:0/499619716 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc0c5ea0 con 0x7f16dc06ea50 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16da7fc640 1 --2- 172.21.15.78:0/499619716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06ea50 0x7f16dc152680 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16d00097a0 tx=0x7f16d002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d082b6c129678e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16d9ffb640 1 -- 172.21.15.78:0/499619716 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d002f050 con 0x7f16dc06ea50 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16d9ffb640 1 -- 172.21.15.78:0/499619716 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d0031700 con 0x7f16dc06ea50 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16d9ffb640 1 -- 172.21.15.78:0/499619716 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0031a90 con 0x7f16dc06ea50 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16e169a640 1 -- 172.21.15.78:0/499619716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06ea50 msgr2=0x7f16dc152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16e169a640 1 --2- 172.21.15.78:0/499619716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06ea50 0x7f16dc152680 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f16d00097a0 tx=0x7f16d002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16e169a640 1 -- 172.21.15.78:0/499619716 shutdown_connections 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.185+0000 7f16e169a640 1 --2- 172.21.15.78:0/499619716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06ea50 0x7f16dc152680 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.186+0000 7f16e169a640 1 --2- 172.21.15.78:0/499619716 >> v2:172.21.15.150:3300/0 conn(0x7f16dc14f1a0 0x7f16dc072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.186+0000 7f16e169a640 1 -- 172.21.15.78:0/499619716 >> 172.21.15.78:0/499619716 conn(0x7f16dc06c170 msgr2=0x7f16dc06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:07.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.186+0000 7f16e169a640 1 -- 172.21.15.78:0/499619716 shutdown_connections 2024-02-19T14:54:07.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.186+0000 7f16e169a640 1 -- 172.21.15.78:0/499619716 wait complete. 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 Processor -- start 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 -- start start 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 0x7f16dc1e98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06fbc0 0x7f16dc0c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16dc14f1a0 0x7f16dc0c11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0c29e0 con 0x7f16dc06ea50 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.187+0000 7f16e169a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc153390 con 0x7f16dc14f1a0 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16e169a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc0c4060 con 0x7f16dc06fbc0 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 0x7f16dc1e98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 0x7f16dc1e98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:56168/0 (socket says 172.21.15.78:56168) 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 -- 172.21.15.78:0/1562435400 learned_addr learned my addr 172.21.15.78:0/1562435400 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16db7fe640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16dc14f1a0 0x7f16dc0c11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16da7fc640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06fbc0 0x7f16dc0c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 -- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06fbc0 msgr2=0x7f16dc0c0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06fbc0 0x7f16dc0c0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 -- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16dc14f1a0 msgr2=0x7f16dc0c11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16dc14f1a0 0x7f16dc0c11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 -- 172.21.15.78:0/1562435400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d0009400 con 0x7f16dc06ea50 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16da7fc640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06fbc0 0x7f16dc0c0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16daffd640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 0x7f16dc1e98e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009fd0 tx=0x7f16cc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc00c940 con 0x7f16dc06ea50 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc00caa0 con 0x7f16dc06ea50 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc00ce30 con 0x7f16dc06ea50 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc0c4340 con 0x7f16dc06ea50 2024-02-19T14:54:07.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.188+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc1ea2d0 con 0x7f16dc06ea50 2024-02-19T14:54:07.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.189+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16dc074990 con 0x7f16dc06ea50 2024-02-19T14:54:07.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.190+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f16cc013470 con 0x7f16dc06ea50 2024-02-19T14:54:07.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.191+0000 7f16c37fe640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c806b840 0x7f16c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.191+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f16cc08a8f0 con 0x7f16dc06ea50 2024-02-19T14:54:07.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.191+0000 7f16da7fc640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c806b840 0x7f16c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.192+0000 7f16da7fc640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c806b840 0x7f16c806dd00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f16d0035000 tx=0x7f16d0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.195+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f16cc054d50 con 0x7f16dc06ea50 2024-02-19T14:54:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.412+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16dc0a9a70 con 0x7f16dc06ea50 2024-02-19T14:54:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.413+0000 7f16c37fe640 1 -- 172.21.15.78:0/1562435400 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6185 (secure 0 0 0) 0x7f16cc058410 con 0x7f16dc06ea50 2024-02-19T14:54:07.417 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:07.417 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:54:02.288421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6042},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6042":{"gid":6042,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1226290552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1226290552},{"type":"v1","addr":"172.21.15.78:6837","nonce":1226290552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:07.417 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 39 2024-02-19T14:54:07.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c806b840 msgr2=0x7f16c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c806b840 0x7f16c806dd00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f16d0035000 tx=0x7f16d0008030 comp rx=0 tx=0).stop 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 msgr2=0x7f16dc1e98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 0x7f16dc1e98e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009fd0 tx=0x7f16cc00ed40 comp rx=0 tx=0).stop 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 shutdown_connections 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c806b840 0x7f16c806dd00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16dc14f1a0 0x7f16dc0c11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16dc06fbc0 0x7f16dc0c0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 --2- 172.21.15.78:0/1562435400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16dc06ea50 0x7f16dc1e98e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 >> 172.21.15.78:0/1562435400 conn(0x7f16dc06c170 msgr2=0x7f16dc14e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:07.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 shutdown_connections 2024-02-19T14:54:07.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.415+0000 7f16e169a640 1 -- 172.21.15.78:0/1562435400 wait complete. 2024-02-19T14:54:07.433 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:07.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.536+0000 7f3194a55640 1 Processor -- start 2024-02-19T14:54:07.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.536+0000 7f3194a55640 1 -- start start 2024-02-19T14:54:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.537+0000 7f3194a55640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31900c12a0 0x7f31900723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.537+0000 7f3194a55640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3190072910 0x7f3190151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.537+0000 7f3194a55640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319006f330 con 0x7f319014f3a0 2024-02-19T14:54:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.537+0000 7f3194a55640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31900a84c0 con 0x7f31900c12a0 2024-02-19T14:54:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.537+0000 7f3194a55640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31900a8fb0 con 0x7f3190072910 2024-02-19T14:54:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31900c12a0 0x7f31900723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31900c12a0 0x7f31900723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38244/0 (socket says 172.21.15.78:38244) 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 -- 172.21.15.78:0/1054966222 learned_addr learned my addr 172.21.15.78:0/1054966222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318ed9d640 1 --2- 172.21.15.78:0/1054966222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3190072910 0x7f3190151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 -- 172.21.15.78:0/1054966222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3190072910 msgr2=0x7f3190151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 --2- 172.21.15.78:0/1054966222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3190072910 0x7f3190151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 -- 172.21.15.78:0/1054966222 >> v1:172.21.15.150:6789/0 conn(0x7f319014f3a0 legacy=0x7f31900c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.538+0000 7f318dd9b640 1 -- 172.21.15.78:0/1054966222 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31900a8710 con 0x7f31900c12a0 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f318dd9b640 1 --2- 172.21.15.78:0/1054966222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31900c12a0 0x7f31900723d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3178009870 tx=0x7f317802da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c822553e6b8efd58 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f318d59a640 1 -- 172.21.15.78:0/1054966222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317802f830 con 0x7f31900c12a0 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f318d59a640 1 -- 172.21.15.78:0/1054966222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317802f990 con 0x7f31900c12a0 2024-02-19T14:54:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f318d59a640 1 -- 172.21.15.78:0/1054966222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317802fd20 con 0x7f31900c12a0 2024-02-19T14:54:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 -- 172.21.15.78:0/1054966222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31900c12a0 msgr2=0x7f31900723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 --2- 172.21.15.78:0/1054966222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31900c12a0 0x7f31900723d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3178009870 tx=0x7f317802da70 comp rx=0 tx=0).stop 2024-02-19T14:54:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 -- 172.21.15.78:0/1054966222 shutdown_connections 2024-02-19T14:54:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 --2- 172.21.15.78:0/1054966222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3190072910 0x7f3190151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 --2- 172.21.15.78:0/1054966222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31900c12a0 0x7f31900723d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 -- 172.21.15.78:0/1054966222 >> 172.21.15.78:0/1054966222 conn(0x7f31900718e0 msgr2=0x7f3190071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 -- 172.21.15.78:0/1054966222 shutdown_connections 2024-02-19T14:54:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.539+0000 7f3194a55640 1 -- 172.21.15.78:0/1054966222 wait complete. 2024-02-19T14:54:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 Processor -- start 2024-02-19T14:54:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 -- start start 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3190072910 0x7f31901512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31900c12a0 0x7f31901e7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 0x7f3190145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31901e76a0 con 0x7f319014f3a0 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319014f9e0 con 0x7f3190072910 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f3194a55640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3190150280 con 0x7f31900c12a0 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f318ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 0x7f3190145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f318ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 0x7f3190145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38258/0 (socket says 172.21.15.78:38258) 2024-02-19T14:54:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.541+0000 7f318ed9d640 1 -- 172.21.15.78:0/1075271780 learned_addr learned my addr 172.21.15.78:0/1075271780 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318e59c640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3190072910 0x7f31901512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318dd9b640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31900c12a0 0x7f31901e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318ed9d640 1 -- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31900c12a0 msgr2=0x7f31901e7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318ed9d640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31900c12a0 0x7f31901e7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318ed9d640 1 -- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3190072910 msgr2=0x7f31901512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318ed9d640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3190072910 0x7f31901512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318ed9d640 1 -- 172.21.15.78:0/1075271780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780094d0 con 0x7f319014f3a0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318dd9b640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31900c12a0 0x7f31901e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f318ed9d640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 0x7f3190145820 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f318800d800 tx=0x7f318800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318800b660 con 0x7f319014f3a0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318800b7e0 con 0x7f319014f3a0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3190151920 con 0x7f319014f3a0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318800bb90 con 0x7f319014f3a0 2024-02-19T14:54:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.542+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319014ff10 con 0x7f319014f3a0 2024-02-19T14:54:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.543+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3190074990 con 0x7f319014f3a0 2024-02-19T14:54:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.544+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f318800bcf0 con 0x7f319014f3a0 2024-02-19T14:54:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.544+0000 7f3186ffd640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f317006b810 0x7f317006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.544+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f318808acf0 con 0x7f319014f3a0 2024-02-19T14:54:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.544+0000 7f318e59c640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f317006b810 0x7f317006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.545+0000 7f318e59c640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f317006b810 0x7f317006dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3180001ca0 tx=0x7f3180009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.552+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f318808f070 con 0x7f319014f3a0 2024-02-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.766+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31901503c0 con 0x7f319014f3a0 2024-02-19T14:54:07.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.767+0000 7f3186ffd640 1 -- 172.21.15.78:0/1075271780 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6185 (secure 0 0 0) 0x7f3188055150 con 0x7f319014f3a0 2024-02-19T14:54:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:07.771 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:37.026854+0000","modified":"2024-02-19T14:54:02.288421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6042},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6042":{"gid":6042,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1226290552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1226290552},{"type":"v1","addr":"172.21.15.78:6837","nonce":1226290552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:53:50.148330+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5700},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5700":{"gid":5700,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6836/3678473057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3678473057},{"type":"v1","addr":"172.21.15.78:6836","nonce":3678473057}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:07.771 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 39 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f317006b810 msgr2=0x7f317006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f317006b810 0x7f317006dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3180001ca0 tx=0x7f3180009040 comp rx=0 tx=0).stop 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 msgr2=0x7f3190145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 0x7f3190145820 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f318800d800 tx=0x7f318800dcc0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 shutdown_connections 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f317006b810 0x7f317006dcd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f319014f3a0 0x7f3190145820 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31900c12a0 0x7f31901e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 --2- 172.21.15.78:0/1075271780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3190072910 0x7f31901512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 >> 172.21.15.78:0/1075271780 conn(0x7f31900718e0 msgr2=0x7f31900c1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:07.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 shutdown_connections 2024-02-19T14:54:07.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.770+0000 7f3194a55640 1 -- 172.21.15.78:0/1075271780 wait complete. 2024-02-19T14:54:07.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:07.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.915+0000 7f823e977640 1 Processor -- start 2024-02-19T14:54:07.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.915+0000 7f823e977640 1 -- start start 2024-02-19T14:54:07.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f823e977640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8238152670 0x7f8238071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f823e977640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8238152a50 0x7f8238155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f823e977640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8238071bf0 con 0x7f82380702d0 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f823e977640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8238158a20 con 0x7f8238152a50 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f823e977640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8238155550 con 0x7f8238152670 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f82377fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8238152a50 0x7f8238155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f82377fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8238152a50 0x7f8238155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38264/0 (socket says 172.21.15.78:38264) 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f82377fe640 1 -- 172.21.15.78:0/2612453682 learned_addr learned my addr 172.21.15.78:0/2612453682 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.916+0000 7f8237fff640 1 --2- 172.21.15.78:0/2612453682 >> v2:172.21.15.150:3300/0 conn(0x7f8238152670 0x7f8238071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.917+0000 7f82377fe640 1 -- 172.21.15.78:0/2612453682 >> v2:172.21.15.150:3300/0 conn(0x7f8238152670 msgr2=0x7f8238071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.917+0000 7f82377fe640 1 --2- 172.21.15.78:0/2612453682 >> v2:172.21.15.150:3300/0 conn(0x7f8238152670 0x7f8238071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.917+0000 7f82377fe640 1 -- 172.21.15.78:0/2612453682 >> v1:172.21.15.150:6789/0 conn(0x7f82380702d0 legacy=0x7f8238158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.917+0000 7f82377fe640 1 -- 172.21.15.78:0/2612453682 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380c6780 con 0x7f8238152a50 2024-02-19T14:54:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.918+0000 7f82377fe640 1 --2- 172.21.15.78:0/2612453682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152a50 0x7f8238155010 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f823806e600 tx=0x7f822c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7daefec2e17090 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.918+0000 7f8236ffd640 1 -- 172.21.15.78:0/2612453682 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c02f850 con 0x7f8238152a50 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.918+0000 7f8236ffd640 1 -- 172.21.15.78:0/2612453682 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c02f9b0 con 0x7f8238152a50 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.918+0000 7f8236ffd640 1 -- 172.21.15.78:0/2612453682 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c02fd40 con 0x7f8238152a50 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 -- 172.21.15.78:0/2612453682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152a50 msgr2=0x7f8238155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 --2- 172.21.15.78:0/2612453682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152a50 0x7f8238155010 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f823806e600 tx=0x7f822c02ba30 comp rx=0 tx=0).stop 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 -- 172.21.15.78:0/2612453682 shutdown_connections 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 --2- 172.21.15.78:0/2612453682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152a50 0x7f8238155010 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 --2- 172.21.15.78:0/2612453682 >> v2:172.21.15.150:3300/0 conn(0x7f8238152670 0x7f8238071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 -- 172.21.15.78:0/2612453682 >> 172.21.15.78:0/2612453682 conn(0x7f82380c85e0 msgr2=0x7f82380c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 -- 172.21.15.78:0/2612453682 shutdown_connections 2024-02-19T14:54:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.919+0000 7f823e977640 1 -- 172.21.15.78:0/2612453682 wait complete. 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.920+0000 7f823e977640 1 Processor -- start 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.920+0000 7f823e977640 1 -- start start 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82380702d0 0x7f82380ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 0x7f8238157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8238152a50 0x7f82381eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82381eab00 con 0x7f8238152670 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82380aba10 con 0x7f82380702d0 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82380ab730 con 0x7f8238152a50 2024-02-19T14:54:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 0x7f8238157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 0x7f8238157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38266/0 (socket says 172.21.15.78:38266) 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 -- 172.21.15.78:0/3807769932 learned_addr learned my addr 172.21.15.78:0/3807769932 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f8237fff640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82380702d0 0x7f82380ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823cf14640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8238152a50 0x7f82381eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 -- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8238152a50 msgr2=0x7f82381eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8238152a50 0x7f82381eb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 -- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82380702d0 msgr2=0x7f82380ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82380702d0 0x7f82380ac4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 -- 172.21.15.78:0/3807769932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c0094d0 con 0x7f8238152670 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823cf14640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8238152a50 0x7f82381eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f8237fff640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82380702d0 0x7f82380ac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f82377fe640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 0x7f8238157640 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f822c02ba70 tx=0x7f822c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c037d10 con 0x7f8238152670 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c037e90 con 0x7f8238152670 2024-02-19T14:54:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.921+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82381eeb20 con 0x7f8238152670 2024-02-19T14:54:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.922+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c041640 con 0x7f8238152670 2024-02-19T14:54:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.922+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82381eadc0 con 0x7f8238152670 2024-02-19T14:54:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.923+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82380676a0 con 0x7f8238152670 2024-02-19T14:54:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.923+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f822c0417a0 con 0x7f8238152670 2024-02-19T14:54:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.924+0000 7f8234ff9640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822406b810 0x7f822406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.924+0000 7f8237fff640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822406b810 0x7f822406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.924+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f822c03c070 con 0x7f8238152670 2024-02-19T14:54:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.924+0000 7f8237fff640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822406b810 0x7f822406dcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8228006fd0 tx=0x7f8228008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:07.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:07.928+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f822c077a70 con 0x7f8238152670 2024-02-19T14:54:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.125+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82380c4e60 con 0x7f8238152670 2024-02-19T14:54:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.128+0000 7f8234ff9640 1 -- 172.21.15.78:0/3807769932 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+21009 (secure 0 0 0) 0x7f822c07b130 con 0x7f8238152670 2024-02-19T14:54:08.131 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:08.132 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":59,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:00.278788+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:53:35.233332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-19T14:53:35.989674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1f","mappings":[{"from":0,"to":6}]},{"pgid":"8.32","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822406b810 msgr2=0x7f822406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822406b810 0x7f822406dcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8228006fd0 tx=0x7f8228008040 comp rx=0 tx=0).stop 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 msgr2=0x7f8238157640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 0x7f8238157640 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f822c02ba70 tx=0x7f822c004730 comp rx=0 tx=0).stop 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 shutdown_connections 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822406b810 0x7f822406dcd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8238152670 0x7f8238157640 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8238152a50 0x7f82381eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 --2- 172.21.15.78:0/3807769932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82380702d0 0x7f82380ac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 >> 172.21.15.78:0/3807769932 conn(0x7f82380c85e0 msgr2=0x7f823806d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 shutdown_connections 2024-02-19T14:54:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.130+0000 7f823e977640 1 -- 172.21.15.78:0/3807769932 wait complete. 2024-02-19T14:54:08.145 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:54:08.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.243+0000 7f397350d640 1 Processor -- start 2024-02-19T14:54:08.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.244+0000 7f397350d640 1 -- start start 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f397350d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c152700 0x7f396c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f397350d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f396c0c8e00 0x7f396c15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f397350d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c1551f0 con 0x7f396c0c5730 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f397350d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c155490 con 0x7f396c0c8e00 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f397350d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c155770 con 0x7f396c152700 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f396c0c8e00 0x7f396c15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f396c0c8e00 0x7f396c15f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38282/0 (socket says 172.21.15.78:38282) 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3970aa8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f396c0c5730 0x7f396c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38828/0 (socket says 172.21.15.78:38828) 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 -- 172.21.15.78:0/3439190517 learned_addr learned my addr 172.21.15.78:0/3439190517 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f39712a9640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c152700 0x7f396c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 -- 172.21.15.78:0/3439190517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c152700 msgr2=0x7f396c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c152700 0x7f396c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 -- 172.21.15.78:0/3439190517 >> v1:172.21.15.78:6789/0 conn(0x7f396c0c5730 legacy=0x7f396c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3963fff640 1 -- 172.21.15.78:0/3439190517 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946932954 0 0) 0x7f396c1551f0 con 0x7f396c0c5730 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f3971aaa640 1 -- 172.21.15.78:0/3439190517 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c155a50 con 0x7f396c0c8e00 2024-02-19T14:54:08.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.245+0000 7f39712a9640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c152700 0x7f396c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:08.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.246+0000 7f3971aaa640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c8e00 0x7f396c15f980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3964009e50 tx=0x7f396402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe15e4cc266c9f61 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:08.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f3963fff640 1 -- 172.21.15.78:0/3439190517 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396402cb00 con 0x7f396c0c8e00 2024-02-19T14:54:08.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f3963fff640 1 -- 172.21.15.78:0/3439190517 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396402cc60 con 0x7f396c0c8e00 2024-02-19T14:54:08.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f3963fff640 1 -- 172.21.15.78:0/3439190517 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39640343f0 con 0x7f396c0c8e00 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 -- 172.21.15.78:0/3439190517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c8e00 msgr2=0x7f396c15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c8e00 0x7f396c15f980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3964009e50 tx=0x7f396402bf60 comp rx=0 tx=0).stop 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 -- 172.21.15.78:0/3439190517 shutdown_connections 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c8e00 0x7f396c15f980 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 --2- 172.21.15.78:0/3439190517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c152700 0x7f396c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 -- 172.21.15.78:0/3439190517 >> 172.21.15.78:0/3439190517 conn(0x7f396c0cac90 msgr2=0x7f396c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 -- 172.21.15.78:0/3439190517 shutdown_connections 2024-02-19T14:54:08.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.247+0000 7f397350d640 1 -- 172.21.15.78:0/3439190517 wait complete. 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.248+0000 7f397350d640 1 Processor -- start 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.248+0000 7f397350d640 1 -- start start 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f397350d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 0x7f396c0c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f397350d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c0c8e00 0x7f396c1ed620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f397350d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f396c152700 0x7f396c154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f397350d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1ea470 con 0x7f396c0c5730 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f397350d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1edb60 con 0x7f396c152700 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f397350d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1ea0b0 con 0x7f396c0c8e00 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 0x7f396c0c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 0x7f396c0c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38294/0 (socket says 172.21.15.78:38294) 2024-02-19T14:54:08.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 -- 172.21.15.78:0/3318186636 learned_addr learned my addr 172.21.15.78:0/3318186636 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f3971aaa640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f396c152700 0x7f396c154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f3970aa8640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c0c8e00 0x7f396c1ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 -- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c0c8e00 msgr2=0x7f396c1ed620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c0c8e00 0x7f396c1ed620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 -- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f396c152700 msgr2=0x7f396c154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f396c152700 0x7f396c154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 -- 172.21.15.78:0/3318186636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009b30 con 0x7f396c0c5730 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.249+0000 7f39712a9640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 0x7f396c0c1c70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f396800c7f0 tx=0x7f396800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.250+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968007820 con 0x7f396c0c5730 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.250+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3968007980 con 0x7f396c0c5730 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.250+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c1ede00 con 0x7f396c0c5730 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.250+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968007cf0 con 0x7f396c0c5730 2024-02-19T14:54:08.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.250+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c1ee3c0 con 0x7f396c0c5730 2024-02-19T14:54:08.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.251+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c074990 con 0x7f396c0c5730 2024-02-19T14:54:08.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.251+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f39680113f0 con 0x7f396c0c5730 2024-02-19T14:54:08.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.252+0000 7f3961ffb640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f395006b810 0x7f395006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.252+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7187+0+0 (secure 0 0 0) 0x7f396808a9f0 con 0x7f396c0c5730 2024-02-19T14:54:08.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.252+0000 7f3970aa8640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f395006b810 0x7f395006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.252+0000 7f3970aa8640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f395006b810 0x7f395006dcd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f396c1503e0 tx=0x7f395c007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:08.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.256+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3968054ed0 con 0x7f396c0c5730 2024-02-19T14:54:08.280 INFO:tasks.ceph.mds.a.smithi078.stderr:2024-02-19T14:54:08.277+0000 7f9f19fa7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.496+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f396c1ea6e0 con 0x7f396c0c5730 2024-02-19T14:54:08.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:54:08.497+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:54:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.498+0000 7f3961ffb640 1 -- 172.21.15.78:0/3318186636 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v40) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3968058590 con 0x7f396c0c5730 2024-02-19T14:54:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:54:08.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.501+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f395006b810 msgr2=0x7f395006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.501+0000 7f397350d640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f395006b810 0x7f395006dcd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f396c1503e0 tx=0x7f395c007b50 comp rx=0 tx=0).stop 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.501+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 msgr2=0x7f396c0c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.501+0000 7f397350d640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 0x7f396c0c1c70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f396800c7f0 tx=0x7f396800ccb0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 shutdown_connections 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f395006b810 0x7f395006dcd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f396c152700 0x7f396c154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f396c0c8e00 0x7f396c1ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 --2- 172.21.15.78:0/3318186636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f396c0c5730 0x7f396c0c1c70 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 >> 172.21.15.78:0/3318186636 conn(0x7f396c0cac90 msgr2=0x7f396c0cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 shutdown_connections 2024-02-19T14:54:08.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.502+0000 7f397350d640 1 -- 172.21.15.78:0/3318186636 wait complete. 2024-02-19T14:54:08.507 INFO:tasks.ceph.mds.a.smithi078.stderr: -40> 2024-02-19T14:54:08.277+0000 7f9f19fa7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:08.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:54:08.531 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:54:08.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.620+0000 7f68568d4640 1 Processor -- start 2024-02-19T14:54:08.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.620+0000 7f68568d4640 1 -- start start 2024-02-19T14:54:08.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f68568d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f68500c78a0 0x7f68500c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f68568d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f685014ef50 0x7f685014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f68568d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68500c35c0 con 0x7f68500727d0 2024-02-19T14:54:08.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f68568d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685014f920 con 0x7f68500c78a0 2024-02-19T14:54:08.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f68568d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500c38a0 con 0x7f685014ef50 2024-02-19T14:54:08.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f68500c78a0 0x7f68500c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f68500c78a0 0x7f68500c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38316/0 (socket says 172.21.15.78:38316) 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 -- 172.21.15.78:0/2295993950 learned_addr learned my addr 172.21.15.78:0/2295993950 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f6854e71640 1 --1- 172.21.15.78:0/2295993950 >> v1:172.21.15.78:6789/0 conn(0x7f68500727d0 0x7f6850072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38846/0 (socket says 172.21.15.78:38846) 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684f7fe640 1 --2- 172.21.15.78:0/2295993950 >> v2:172.21.15.150:3300/0 conn(0x7f685014ef50 0x7f685014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 -- 172.21.15.78:0/2295993950 >> v2:172.21.15.150:3300/0 conn(0x7f685014ef50 msgr2=0x7f685014f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 --2- 172.21.15.78:0/2295993950 >> v2:172.21.15.150:3300/0 conn(0x7f685014ef50 0x7f685014f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 -- 172.21.15.78:0/2295993950 >> v1:172.21.15.78:6789/0 conn(0x7f68500727d0 legacy=0x7f6850072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684ffff640 1 -- 172.21.15.78:0/2295993950 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500c3b80 con 0x7f68500c78a0 2024-02-19T14:54:08.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.621+0000 7f684effd640 1 -- 172.21.15.78:0/2295993950 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352714756 0 0) 0x7f68500c35c0 con 0x7f68500727d0 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.622+0000 7f684ffff640 1 --2- 172.21.15.78:0/2295993950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500c78a0 0x7f68500c44b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f68500c1eb0 tx=0x7f684002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d641892378271a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.622+0000 7f684effd640 1 -- 172.21.15.78:0/2295993950 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840030860 con 0x7f68500c78a0 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.622+0000 7f684effd640 1 -- 172.21.15.78:0/2295993950 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68400309c0 con 0x7f68500c78a0 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.622+0000 7f684effd640 1 -- 172.21.15.78:0/2295993950 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840030d50 con 0x7f68500c78a0 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 -- 172.21.15.78:0/2295993950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500c78a0 msgr2=0x7f68500c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 --2- 172.21.15.78:0/2295993950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500c78a0 0x7f68500c44b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f68500c1eb0 tx=0x7f684002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 -- 172.21.15.78:0/2295993950 shutdown_connections 2024-02-19T14:54:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 --2- 172.21.15.78:0/2295993950 >> v2:172.21.15.150:3300/0 conn(0x7f685014ef50 0x7f685014f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 --2- 172.21.15.78:0/2295993950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500c78a0 0x7f68500c44b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 -- 172.21.15.78:0/2295993950 >> 172.21.15.78:0/2295993950 conn(0x7f6850066e70 msgr2=0x7f68500a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 -- 172.21.15.78:0/2295993950 shutdown_connections 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.623+0000 7f68568d4640 1 -- 172.21.15.78:0/2295993950 wait complete. 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.624+0000 7f68568d4640 1 Processor -- start 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.624+0000 7f68568d4640 1 -- start start 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 0x7f6850120920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68500c78a0 0x7f68501ec4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014ef50 0x7f685006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68501e8930 con 0x7f68500727d0 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68501228a0 con 0x7f68500c78a0 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68501ec9e0 con 0x7f685014ef50 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 0x7f6850120920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 0x7f6850120920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38326/0 (socket says 172.21.15.78:38326) 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 -- 172.21.15.78:0/1747976705 learned_addr learned my addr 172.21.15.78:0/1747976705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684f7fe640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68500c78a0 0x7f68501ec4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f6854e71640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014ef50 0x7f685006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 -- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014ef50 msgr2=0x7f685006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014ef50 0x7f685006b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 -- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68500c78a0 msgr2=0x7f68501ec4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68500c78a0 0x7f68501ec4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 -- 172.21.15.78:0/1747976705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68400094d0 con 0x7f68500727d0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f6854e71640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014ef50 0x7f685006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684ffff640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 0x7f6850120920 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6840000c00 tx=0x7f6840004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684003e040 con 0x7f68500727d0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6840039c90 con 0x7f68500727d0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68501e8570 con 0x7f68500727d0 2024-02-19T14:54:08.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840004030 con 0x7f68500727d0 2024-02-19T14:54:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.625+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68500c1290 con 0x7f68500727d0 2024-02-19T14:54:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.626+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6850074990 con 0x7f68500727d0 2024-02-19T14:54:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.627+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6840004190 con 0x7f68500727d0 2024-02-19T14:54:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.627+0000 7f684cff9640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683c06b810 0x7f683c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.627+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7273+0+0 (secure 0 0 0) 0x7f68400acfc0 con 0x7f68500727d0 2024-02-19T14:54:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.627+0000 7f684f7fe640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683c06b810 0x7f683c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:08.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.628+0000 7f684f7fe640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683c06b810 0x7f683c06dcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6844006fd0 tx=0x7f6844008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.636+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68400e0dc0 con 0x7f68500727d0 2024-02-19T14:54:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:08.851+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6850151f30 con 0x7f68500727d0 2024-02-19T14:54:09.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.500+0000 7f684cff9640 1 -- 172.21.15.78:0/1747976705 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7f684007aef0 con 0x7f68500727d0 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683c06b810 msgr2=0x7f683c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683c06b810 0x7f683c06dcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6844006fd0 tx=0x7f6844008040 comp rx=0 tx=0).stop 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 msgr2=0x7f6850120920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 0x7f6850120920 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6840000c00 tx=0x7f6840004730 comp rx=0 tx=0).stop 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 shutdown_connections 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683c06b810 0x7f683c06dcd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014ef50 0x7f685006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68500c78a0 0x7f68501ec4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 --2- 172.21.15.78:0/1747976705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68500727d0 0x7f6850120920 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 >> 172.21.15.78:0/1747976705 conn(0x7f6850066e70 msgr2=0x7f68500c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:09.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 shutdown_connections 2024-02-19T14:54:09.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.502+0000 7f68568d4640 1 -- 172.21.15.78:0/1747976705 wait complete. 2024-02-19T14:54:09.510 INFO:tasks.ceph.mds.c.smithi078.stderr: -526> 2024-02-19T14:53:52.944+0000 7ff8f32cd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:09.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:54:09.538 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:54:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.638+0000 7fbcb3720640 1 Processor -- start 2024-02-19T14:54:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb3720640 1 -- start start 2024-02-19T14:54:09.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb3720640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbcac0ae050 0x7fbcac0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:09.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb3720640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbcac0c4cd0 0x7fbcac0c5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:09.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb3720640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcac1539e0 con 0x7fbcac0c2aa0 2024-02-19T14:54:09.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb3720640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcac1537d0 con 0x7fbcac0c4cd0 2024-02-19T14:54:09.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb3720640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcac0709f0 con 0x7fbcac0ae050 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.639+0000 7fbcb1cbd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbcac0c4cd0 0x7fbcac0c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb0cbb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbcac0c2aa0 0x7fbcac0702e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38876/0 (socket says 172.21.15.78:38876) 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb0cbb640 1 -- 172.21.15.78:0/1354266201 learned_addr learned my addr 172.21.15.78:0/1354266201 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb1cbd640 1 --2- 172.21.15.78:0/1354266201 >> v2:172.21.15.78:3300/0 conn(0x7fbcac0c4cd0 0x7fbcac0c5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38356/0 (socket says 172.21.15.78:38356) 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb14bc640 1 --2- 172.21.15.78:0/1354266201 >> v2:172.21.15.150:3300/0 conn(0x7fbcac0ae050 0x7fbcac0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb1cbd640 1 -- 172.21.15.78:0/1354266201 >> v2:172.21.15.150:3300/0 conn(0x7fbcac0ae050 msgr2=0x7fbcac0c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb1cbd640 1 --2- 172.21.15.78:0/1354266201 >> v2:172.21.15.150:3300/0 conn(0x7fbcac0ae050 0x7fbcac0c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb1cbd640 1 -- 172.21.15.78:0/1354266201 >> v1:172.21.15.78:6789/0 conn(0x7fbcac0c2aa0 legacy=0x7fbcac0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb1cbd640 1 -- 172.21.15.78:0/1354266201 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac0c3140 con 0x7fbcac0c4cd0 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbca3fff640 1 -- 172.21.15.78:0/1354266201 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219036194 0 0) 0x7fbcac1539e0 con 0x7fbcac0c2aa0 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.640+0000 7fbcb14bc640 1 --2- 172.21.15.78:0/1354266201 >> v2:172.21.15.150:3300/0 conn(0x7fbcac0ae050 0x7fbcac0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:09.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb1cbd640 1 --2- 172.21.15.78:0/1354266201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0c4cd0 0x7fbcac0c5120 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0678c0 tx=0x7fbca402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b623ab78e7bf22 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:09.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbca3fff640 1 -- 172.21.15.78:0/1354266201 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca402cbf0 con 0x7fbcac0c4cd0 2024-02-19T14:54:09.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbca3fff640 1 -- 172.21.15.78:0/1354266201 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca402cd50 con 0x7fbcac0c4cd0 2024-02-19T14:54:09.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbca3fff640 1 -- 172.21.15.78:0/1354266201 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca4034660 con 0x7fbcac0c4cd0 2024-02-19T14:54:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 -- 172.21.15.78:0/1354266201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0c4cd0 msgr2=0x7fbcac0c5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 --2- 172.21.15.78:0/1354266201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0c4cd0 0x7fbcac0c5120 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0678c0 tx=0x7fbca402be40 comp rx=0 tx=0).stop 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 -- 172.21.15.78:0/1354266201 shutdown_connections 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 --2- 172.21.15.78:0/1354266201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0c4cd0 0x7fbcac0c5120 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 --2- 172.21.15.78:0/1354266201 >> v2:172.21.15.150:3300/0 conn(0x7fbcac0ae050 0x7fbcac0c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 -- 172.21.15.78:0/1354266201 >> 172.21.15.78:0/1354266201 conn(0x7fbcac06f220 msgr2=0x7fbcac06f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 -- 172.21.15.78:0/1354266201 shutdown_connections 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.641+0000 7fbcb3720640 1 -- 172.21.15.78:0/1354266201 wait complete. 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.642+0000 7fbcb3720640 1 Processor -- start 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.642+0000 7fbcb3720640 1 -- start start 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb3720640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 0x7fbcac0a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb3720640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcac0c2aa0 0x7fbcac156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb3720640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcac0c4cd0 0x7fbcac156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb3720640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac14d620 con 0x7fbcac0ae050 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb3720640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac155610 con 0x7fbcac0c2aa0 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb3720640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac155860 con 0x7fbcac0c4cd0 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 0x7fbcac0a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 0x7fbcac0a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38360/0 (socket says 172.21.15.78:38360) 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 -- 172.21.15.78:0/3507351868 learned_addr learned my addr 172.21.15.78:0/3507351868 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb0cbb640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcac0c2aa0 0x7fbcac156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 -- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcac0c4cd0 msgr2=0x7fbcac156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcac0c4cd0 0x7fbcac156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 -- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcac0c2aa0 msgr2=0x7fbcac156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcac0c2aa0 0x7fbcac156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb14bc640 1 -- 172.21.15.78:0/3507351868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4009a70 con 0x7fbcac0ae050 2024-02-19T14:54:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.643+0000 7fbcb0cbb640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcac0c2aa0 0x7fbcac156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.644+0000 7fbcb14bc640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 0x7fbcac0a90d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7f0 tx=0x7fbca800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.644+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8007a30 con 0x7fbcac0ae050 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.644+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8007b90 con 0x7fbcac0ae050 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.644+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac155b40 con 0x7fbcac0ae050 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.644+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca800ce40 con 0x7fbcac0ae050 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.644+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac156100 con 0x7fbcac0ae050 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.645+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac074990 con 0x7fbcac0ae050 2024-02-19T14:54:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.645+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbca80113f0 con 0x7fbcac0ae050 2024-02-19T14:54:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.646+0000 7fbca1ffb640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc9006b810 0x7fbc9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.646+0000 7fbcb0cbb640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc9006b810 0x7fbc9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.646+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7359+0+0 (secure 0 0 0) 0x7fbca808a960 con 0x7fbcac0ae050 2024-02-19T14:54:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.646+0000 7fbcb0cbb640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc9006b810 0x7fbc9006dcd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0ad120 tx=0x7fbc9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:09.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.650+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbca8054da0 con 0x7fbcac0ae050 2024-02-19T14:54:09.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:09.825+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbcac071720 con 0x7fbcac0ae050 2024-02-19T14:54:10.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.509+0000 7fbca1ffb640 1 -- 172.21.15.78:0/3507351868 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v62) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbca8058460 con 0x7fbcac0ae050 2024-02-19T14:54:10.512 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc9006b810 msgr2=0x7fbc9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc9006b810 0x7fbc9006dcd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0ad120 tx=0x7fbc9c009040 comp rx=0 tx=0).stop 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 msgr2=0x7fbcac0a90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 0x7fbcac0a90d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7f0 tx=0x7fbca800cbf0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 shutdown_connections 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc9006b810 0x7fbc9006dcd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcac0c4cd0 0x7fbcac156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcac0c2aa0 0x7fbcac156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 --2- 172.21.15.78:0/3507351868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcac0ae050 0x7fbcac0a90d0 secure :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c7f0 tx=0x7fbca800cbf0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.512+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 >> 172.21.15.78:0/3507351868 conn(0x7fbcac06f220 msgr2=0x7fbcac0702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.513+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 shutdown_connections 2024-02-19T14:54:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.513+0000 7fbcb3720640 1 -- 172.21.15.78:0/3507351868 wait complete. 2024-02-19T14:54:10.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:54:10.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.637+0000 7f0bedebb640 1 Processor -- start 2024-02-19T14:54:10.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.637+0000 7f0bedebb640 1 -- start start 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.637+0000 7f0bedebb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0be80ae050 0x7f0be80c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.637+0000 7f0bedebb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be81539e0 con 0x7f0be80c4cd0 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.637+0000 7f0bedebb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be80c3140 con 0x7f0be80c2aa0 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.637+0000 7f0bedebb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be81537d0 con 0x7f0be80ae050 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0be80c4cd0 0x7f0be80c5120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38884/0 (socket says 172.21.15.78:38884) 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be7fff640 1 -- 172.21.15.78:0/3204790188 learned_addr learned my addr 172.21.15.78:0/3204790188 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813488128 0 0) 0x7f0be81539e0 con 0x7f0be80c4cd0 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd4003520 con 0x7f0be80c4cd0 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be77fe640 1 --2- 172.21.15.78:0/3204790188 >> v2:172.21.15.150:3300/0 conn(0x7f0be80ae050 0x7f0be80c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3182479101 0 0) 0x7f0bd4003520 con 0x7f0be80c4cd0 2024-02-19T14:54:10.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 >> v2:172.21.15.150:3300/0 conn(0x7f0be80ae050 msgr2=0x7f0be80c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 --2- 172.21.15.78:0/3204790188 >> v2:172.21.15.150:3300/0 conn(0x7f0be80ae050 0x7f0be80c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 >> v1:172.21.15.150:6789/0 conn(0x7f0be80c2aa0 legacy=0x7f0be80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80c3420 con 0x7f0be80c4cd0 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0be00019f0 con 0x7f0be80c4cd0 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 >> v1:172.21.15.78:6789/0 conn(0x7f0be80c4cd0 legacy=0x7f0be80c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd4003970 0x7f0bd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bd40047c0 0x7f0bd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd4005180 0x7f0bd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be81537d0 con 0x7f0bd40047c0 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd400c120 con 0x7f0bd4005180 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd400c2e0 con 0x7f0bd4003970 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0bdffff640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bd40047c0 0x7f0bd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be7fff640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd4005180 0x7f0bd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.638+0000 7f0be77fe640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd4003970 0x7f0bd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0bdffff640 1 -- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd4003970 msgr2=0x7f0bd4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0bdffff640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd4003970 0x7f0bd4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0bdffff640 1 -- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd4005180 msgr2=0x7f0bd400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0bdffff640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd4005180 0x7f0bd400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0be7fff640 1 -- 172.21.15.78:0/3204790188 reap_dead start 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0bdffff640 1 -- 172.21.15.78:0/3204790188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd4003520 con 0x7f0bd40047c0 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0be77fe640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd4003970 0x7f0bd4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.639+0000 7f0bdffff640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bd40047c0 0x7f0bd4004c40 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0001750 tx=0x7f0bd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec36e9c8e1853389 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00052d0 con 0x7f0bd40047c0 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd002e050 con 0x7f0bd40047c0 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0be6ffd640 1 -- 172.21.15.78:0/3204790188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0003830 con 0x7f0bd40047c0 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 -- 172.21.15.78:0/3204790188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bd40047c0 msgr2=0x7f0bd4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bd40047c0 0x7f0bd4004c40 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0001750 tx=0x7f0bd40057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 -- 172.21.15.78:0/3204790188 shutdown_connections 2024-02-19T14:54:10.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 --2- 172.21.15.78:0/3204790188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bd40047c0 0x7f0bd4004c40 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 -- 172.21.15.78:0/3204790188 >> 172.21.15.78:0/3204790188 conn(0x7f0be806f220 msgr2=0x7f0be806f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:10.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 -- 172.21.15.78:0/3204790188 shutdown_connections 2024-02-19T14:54:10.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.640+0000 7f0bedebb640 1 -- 172.21.15.78:0/3204790188 wait complete. 2024-02-19T14:54:10.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.641+0000 7f0bedebb640 1 Processor -- start 2024-02-19T14:54:10.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.641+0000 7f0bedebb640 1 -- start start 2024-02-19T14:54:10.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bedebb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd40047c0 0x7f0be80c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bedebb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0be80a99e0 0x7f0be8148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bedebb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 0x7f0be8149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bedebb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80a8110 con 0x7f0be80a9e60 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bedebb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be81e84e0 con 0x7f0be80a99e0 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bedebb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be80a8b50 con 0x7f0bd40047c0 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 0x7f0be8149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 0x7f0be8149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38370/0 (socket says 172.21.15.78:38370) 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 -- 172.21.15.78:0/2390360894 learned_addr learned my addr 172.21.15.78:0/2390360894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0bdffff640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0be80a99e0 0x7f0be8148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be77fe640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd40047c0 0x7f0be80c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 -- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd40047c0 msgr2=0x7f0be80c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd40047c0 0x7f0be80c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 -- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0be80a99e0 msgr2=0x7f0be8148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0be80a99e0 0x7f0be8148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 -- 172.21.15.78:0/2390360894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80a82f0 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.642+0000 7f0be7fff640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 0x7f0be8149070 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0be80c6180 tx=0x7f0be0007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.643+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be000ab70 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.643+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be000acf0 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.643+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0013630 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.643+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be80a85d0 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.643+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be814ed20 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.644+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be8074990 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.644+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0be0013790 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.644+0000 7f0be4ff9640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0bc806b810 0x7f0bc806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.644+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f0be008aed0 con 0x7f0be80a9e60 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.645+0000 7f0be77fe640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0bc806b810 0x7f0bc806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:10.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.645+0000 7f0be77fe640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0bc806b810 0x7f0bc806dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0be8069910 tx=0x7f0bd8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:10.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.649+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0be0055570 con 0x7f0be80a9e60 2024-02-19T14:54:10.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:10.820+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0be80a8d10 con 0x7f0be80a9e60 2024-02-19T14:54:11.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.520+0000 7f0be4ff9640 1 -- 172.21.15.78:0/2390360894 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v63) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0be0017070 con 0x7f0be80a9e60 2024-02-19T14:54:11.522 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:54:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0bc806b810 msgr2=0x7f0bc806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0bc806b810 0x7f0bc806dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0be8069910 tx=0x7f0bd8008040 comp rx=0 tx=0).stop 2024-02-19T14:54:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 msgr2=0x7f0be8149070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 0x7f0be8149070 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0be80c6180 tx=0x7f0be0007bd0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 shutdown_connections 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0bc806b810 0x7f0bc806dcd0 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0be8069910 tx=0x7f0bd8008040 comp rx=0 tx=0).stop 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0be80a9e60 0x7f0be8149070 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0be80a99e0 0x7f0be8148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 --2- 172.21.15.78:0/2390360894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bd40047c0 0x7f0be80c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 >> 172.21.15.78:0/2390360894 conn(0x7f0be806f220 msgr2=0x7f0be80abaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 shutdown_connections 2024-02-19T14:54:11.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.523+0000 7f0bedebb640 1 -- 172.21.15.78:0/2390360894 wait complete. 2024-02-19T14:54:11.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:11.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.639+0000 7f1a1535e640 1 Processor -- start 2024-02-19T14:54:11.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.639+0000 7f1a1535e640 1 -- start start 2024-02-19T14:54:11.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a1535e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1a100c8e00 0x7f1a1006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a1535e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1a1006d910 0x7f1a1006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a1535e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1006cd20 con 0x7f1a10152700 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a1535e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a101551f0 con 0x7f1a100c8e00 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a1535e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a10155490 con 0x7f1a1006d910 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1a100c8e00 0x7f1a1006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1a100c8e00 0x7f1a1006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38378/0 (socket says 172.21.15.78:38378) 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 -- 172.21.15.78:0/502985839 learned_addr learned my addr 172.21.15.78:0/502985839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a0effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1a10152700 0x7f1a100c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38900/0 (socket says 172.21.15.78:38900) 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a0f7fe640 1 --2- 172.21.15.78:0/502985839 >> v2:172.21.15.150:3300/0 conn(0x7f1a1006d910 0x7f1a1006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 -- 172.21.15.78:0/502985839 >> v2:172.21.15.150:3300/0 conn(0x7f1a1006d910 msgr2=0x7f1a1006c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 --2- 172.21.15.78:0/502985839 >> v2:172.21.15.150:3300/0 conn(0x7f1a1006d910 0x7f1a1006c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 -- 172.21.15.78:0/502985839 >> v1:172.21.15.78:6789/0 conn(0x7f1a10152700 legacy=0x7f1a100c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a0e7fc640 1 -- 172.21.15.78:0/502985839 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909921949 0 0) 0x7f1a1006cd20 con 0x7f1a10152700 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.640+0000 7f1a07fff640 1 -- 172.21.15.78:0/502985839 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10155770 con 0x7f1a100c8e00 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.641+0000 7f1a07fff640 1 --2- 172.21.15.78:0/502985839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a100c8e00 0x7f1a1006c2a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1a101596f0 tx=0x7f19f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc8883fe5b0e31f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:11.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.641+0000 7f1a0e7fc640 1 -- 172.21.15.78:0/502985839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8030860 con 0x7f1a100c8e00 2024-02-19T14:54:11.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.641+0000 7f1a0e7fc640 1 -- 172.21.15.78:0/502985839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f80309c0 con 0x7f1a100c8e00 2024-02-19T14:54:11.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.641+0000 7f1a0e7fc640 1 -- 172.21.15.78:0/502985839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8030d50 con 0x7f1a100c8e00 2024-02-19T14:54:11.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.641+0000 7f1a1535e640 1 -- 172.21.15.78:0/502985839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a100c8e00 msgr2=0x7f1a1006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.641+0000 7f1a1535e640 1 --2- 172.21.15.78:0/502985839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a100c8e00 0x7f1a1006c2a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1a101596f0 tx=0x7f19f802b9a0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.642+0000 7f1a1535e640 1 -- 172.21.15.78:0/502985839 shutdown_connections 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.642+0000 7f1a1535e640 1 --2- 172.21.15.78:0/502985839 >> v2:172.21.15.150:3300/0 conn(0x7f1a1006d910 0x7f1a1006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.642+0000 7f1a1535e640 1 --2- 172.21.15.78:0/502985839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a100c8e00 0x7f1a1006c2a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.642+0000 7f1a1535e640 1 -- 172.21.15.78:0/502985839 >> 172.21.15.78:0/502985839 conn(0x7f1a100cac90 msgr2=0x7f1a100cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.642+0000 7f1a1535e640 1 -- 172.21.15.78:0/502985839 shutdown_connections 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.642+0000 7f1a1535e640 1 -- 172.21.15.78:0/502985839 wait complete. 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 Processor -- start 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 -- start start 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 0x7f1a101edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a100c8e00 0x7f1a100c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a10152700 0x7f1a10158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10154cc0 con 0x7f1a1006d910 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100c48f0 con 0x7f1a100c8e00 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a1535e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10157d10 con 0x7f1a10152700 2024-02-19T14:54:11.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 0x7f1a101edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 0x7f1a101edb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38380/0 (socket says 172.21.15.78:38380) 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.643+0000 7f1a0effd640 1 -- 172.21.15.78:0/389830104 learned_addr learned my addr 172.21.15.78:0/389830104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a07fff640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a100c8e00 0x7f1a100c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0effd640 1 -- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a10152700 msgr2=0x7f1a10158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0f7fe640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a10152700 0x7f1a10158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0effd640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a10152700 0x7f1a10158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0effd640 1 -- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a100c8e00 msgr2=0x7f1a100c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0effd640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a100c8e00 0x7f1a100c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0effd640 1 -- 172.21.15.78:0/389830104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80094d0 con 0x7f1a1006d910 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0f7fe640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a10152700 0x7f1a10158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a0effd640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 0x7f1a101edb40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000c430 tx=0x7f1a0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000074e0 con 0x7f1a1006d910 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a00007660 con 0x7f1a1006d910 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00004a20 con 0x7f1a1006d910 2024-02-19T14:54:11.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a10157ff0 con 0x7f1a1006d910 2024-02-19T14:54:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.644+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a101587c0 con 0x7f1a1006d910 2024-02-19T14:54:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.645+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a101589e0 con 0x7f1a1006d910 2024-02-19T14:54:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.645+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1a00004b80 con 0x7f1a1006d910 2024-02-19T14:54:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.646+0000 7f1a077fe640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19f006b810 0x7f19f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.646+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f1a0008a2a0 con 0x7f1a1006d910 2024-02-19T14:54:11.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.646+0000 7f1a07fff640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19f006b810 0x7f19f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.646+0000 7f1a07fff640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19f006b810 0x7f19f006dcd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f19f8036000 tx=0x7f19f8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:11.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.650+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a00054a60 con 0x7f1a1006d910 2024-02-19T14:54:11.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.825+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a10074990 con 0x7f1a1006d910 2024-02-19T14:54:11.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.826+0000 7f1a077fe640 1 -- 172.21.15.78:0/389830104 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+17395 (secure 0 0 0) 0x7f1a00058120 con 0x7f1a1006d910 2024-02-19T14:54:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:11.829 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":63,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:11.510672+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6837/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.78:6834/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6835/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19f006b810 msgr2=0x7f19f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19f006b810 0x7f19f006dcd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f19f8036000 tx=0x7f19f8008030 comp rx=0 tx=0).stop 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 msgr2=0x7f1a101edb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 0x7f1a101edb40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000c430 tx=0x7f1a0000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 shutdown_connections 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19f006b810 0x7f19f006dcd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a10152700 0x7f1a10158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a100c8e00 0x7f1a100c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 --2- 172.21.15.78:0/389830104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a1006d910 0x7f1a101edb40 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 >> 172.21.15.78:0/389830104 conn(0x7f1a100cac90 msgr2=0x7f1a100ca110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 shutdown_connections 2024-02-19T14:54:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.828+0000 7f1a1535e640 1 -- 172.21.15.78:0/389830104 wait complete. 2024-02-19T14:54:11.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-19T14:53:45.797141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-19T14:53:47.077376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:11.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:11.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.942+0000 7f9d58a03640 1 Processor -- start 2024-02-19T14:54:11.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.942+0000 7f9d58a03640 1 -- start start 2024-02-19T14:54:11.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d58a03640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9d540c8ec0 0x7f9d5415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d58a03640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d540c2d20 con 0x7f9d540c7a20 2024-02-19T14:54:11.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d58a03640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d54159580 con 0x7f9d540c4940 2024-02-19T14:54:11.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d58a03640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5415a100 con 0x7f9d540c8ec0 2024-02-19T14:54:11.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d5359e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9d540c7a20 0x7f9d5415d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38910/0 (socket says 172.21.15.78:38910) 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d5359e640 1 -- 172.21.15.78:0/2200624329 learned_addr learned my addr 172.21.15.78:0/2200624329 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d5259c640 1 --2- 172.21.15.78:0/2200624329 >> v2:172.21.15.150:3300/0 conn(0x7f9d540c8ec0 0x7f9d5415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471020501 0 0) 0x7f9d540c2d20 con 0x7f9d540c7a20 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d3c003520 con 0x7f9d540c7a20 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103232458 0 0) 0x7f9d3c003520 con 0x7f9d540c7a20 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 >> v2:172.21.15.150:3300/0 conn(0x7f9d540c8ec0 msgr2=0x7f9d5415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 --2- 172.21.15.78:0/2200624329 >> v2:172.21.15.150:3300/0 conn(0x7f9d540c8ec0 0x7f9d5415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 >> v1:172.21.15.150:6789/0 conn(0x7f9d540c4940 legacy=0x7f9d540bb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d54159860 con 0x7f9d540c7a20 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9d44001670 con 0x7f9d540c7a20 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.944+0000 7f9d5259c640 1 --2- 172.21.15.78:0/2200624329 >> v2:172.21.15.150:3300/0 conn(0x7f9d540c8ec0 0x7f9d5415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 >> v1:172.21.15.78:6789/0 conn(0x7f9d540c7a20 legacy=0x7f9d5415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c003970 0x7f9d3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c0047c0 0x7f9d3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d3c0052a0 0x7f9d3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5415a100 con 0x7f9d3c0047c0 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c00c170 con 0x7f9d3c003970 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c00c330 con 0x7f9d3c0052a0 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5259c640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c0047c0 0x7f9d3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d52d9d640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c003970 0x7f9d3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5359e640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d3c0052a0 0x7f9d3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5259c640 1 -- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d3c0052a0 msgr2=0x7f9d3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5259c640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d3c0052a0 0x7f9d3c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5259c640 1 -- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c003970 msgr2=0x7f9d3c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5259c640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c003970 0x7f9d3c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5259c640 1 -- 172.21.15.78:0/2200624329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c003520 con 0x7f9d3c0047c0 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.945+0000 7f9d5359e640 1 -- 172.21.15.78:0/2200624329 reap_dead start 2024-02-19T14:54:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.946+0000 7f9d5259c640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c0047c0 0x7f9d3c004c40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d540c92c0 tx=0x7f9d3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8b1dd9a47ea701 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.946+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c02cd00 con 0x7f9d3c0047c0 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.946+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c00f040 con 0x7f9d3c0047c0 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.946+0000 7f9d51d9b640 1 -- 172.21.15.78:0/2200624329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c02cea0 con 0x7f9d3c0047c0 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 -- 172.21.15.78:0/2200624329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c0047c0 msgr2=0x7f9d3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c0047c0 0x7f9d3c004c40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d540c92c0 tx=0x7f9d3c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 -- 172.21.15.78:0/2200624329 shutdown_connections 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 --2- 172.21.15.78:0/2200624329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c0047c0 0x7f9d3c004c40 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 -- 172.21.15.78:0/2200624329 >> 172.21.15.78:0/2200624329 conn(0x7f9d540c83a0 msgr2=0x7f9d540c87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 -- 172.21.15.78:0/2200624329 shutdown_connections 2024-02-19T14:54:11.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.947+0000 7f9d58a03640 1 -- 172.21.15.78:0/2200624329 wait complete. 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.948+0000 7f9d58a03640 1 Processor -- start 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.948+0000 7f9d58a03640 1 -- start start 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d58a03640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c0047c0 0x7f9d540ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d58a03640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 0x7f9d540ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d58a03640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d540bc070 0x7f9d541f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d58a03640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d540be380 con 0x7f9d540ad3c0 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d58a03640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d540aca90 con 0x7f9d3c0047c0 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d58a03640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d540adf50 con 0x7f9d540bc070 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 0x7f9d540ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 0x7f9d540ada10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38400/0 (socket says 172.21.15.78:38400) 2024-02-19T14:54:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 -- 172.21.15.78:0/1311343436 learned_addr learned my addr 172.21.15.78:0/1311343436 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d52d9d640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c0047c0 0x7f9d540ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5359e640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d540bc070 0x7f9d541f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 -- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d540bc070 msgr2=0x7f9d541f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d540bc070 0x7f9d541f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 -- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c0047c0 msgr2=0x7f9d540ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c0047c0 0x7f9d540ac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 -- 172.21.15.78:0/1311343436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d540ac0c0 con 0x7f9d540ad3c0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.949+0000 7f9d5259c640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 0x7f9d540ada10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00e3e0 tx=0x7f9d4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.950+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c034a60 con 0x7f9d540ad3c0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.950+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c034bc0 con 0x7f9d540ad3c0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.950+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c02d030 con 0x7f9d540ad3c0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.950+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d541f34c0 con 0x7f9d540ad3c0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.950+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d540aba70 con 0x7f9d540ad3c0 2024-02-19T14:54:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.951+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d540ac250 con 0x7f9d540ad3c0 2024-02-19T14:54:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.951+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9d4c02d190 con 0x7f9d540ad3c0 2024-02-19T14:54:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.952+0000 7f9d3b7fe640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d4006b840 0x7f9d4006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.952+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f9d4c0ad5a0 con 0x7f9d540ad3c0 2024-02-19T14:54:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.952+0000 7f9d52d9d640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d4006b840 0x7f9d4006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.952+0000 7f9d52d9d640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d4006b840 0x7f9d4006dd00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9d48000c00 tx=0x7f9d48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:11.956+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9d4c077f30 con 0x7f9d540ad3c0 2024-02-19T14:54:12.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.126+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d5415c710 con 0x7f9d540ad3c0 2024-02-19T14:54:12.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.127+0000 7f9d3b7fe640 1 -- 172.21.15.78:0/1311343436 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d4c009de0 con 0x7f9d540ad3c0 2024-02-19T14:54:12.129 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d4006b840 msgr2=0x7f9d4006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d4006b840 0x7f9d4006dd00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9d48000c00 tx=0x7f9d48009040 comp rx=0 tx=0).stop 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 msgr2=0x7f9d540ada10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 0x7f9d540ada10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00e3e0 tx=0x7f9d4c009750 comp rx=0 tx=0).stop 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 shutdown_connections 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d4006b840 0x7f9d4006dd00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d540ad3c0 0x7f9d540ada10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d540bc070 0x7f9d541f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 --2- 172.21.15.78:0/1311343436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3c0047c0 0x7f9d540ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 >> 172.21.15.78:0/1311343436 conn(0x7f9d540c83a0 msgr2=0x7f9d540c33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 shutdown_connections 2024-02-19T14:54:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.129+0000 7f9d58a03640 1 -- 172.21.15.78:0/1311343436 wait complete. 2024-02-19T14:54:12.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-19T14:54:12.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.242+0000 7f0dd20c7640 1 Processor -- start 2024-02-19T14:54:12.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.242+0000 7f0dd20c7640 1 -- start start 2024-02-19T14:54:12.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dd20c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0dcc0c29c0 0x7f0dcc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dd20c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc072250 con 0x7f0dcc0c5be0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dd20c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc072480 con 0x7f0dcc0c25e0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dd20c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc152c90 con 0x7f0dcc0c29c0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dcbfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0dcc0c5be0 0x7f0dcc06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38916/0 (socket says 172.21.15.78:38916) 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dcbfff640 1 -- 172.21.15.78:0/1499741811 learned_addr learned my addr 172.21.15.78:0/1499741811 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281489083 0 0) 0x7f0dcc072250 con 0x7f0dcc0c5be0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4003520 con 0x7f0dcc0c5be0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.243+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/1499741811 >> v2:172.21.15.150:3300/0 conn(0x7f0dcc0c29c0 0x7f0dcc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446016404 0 0) 0x7f0db4003520 con 0x7f0dcc0c5be0 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 >> v2:172.21.15.150:3300/0 conn(0x7f0dcc0c29c0 msgr2=0x7f0dcc1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 --2- 172.21.15.78:0/1499741811 >> v2:172.21.15.150:3300/0 conn(0x7f0dcc0c29c0 0x7f0dcc1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 >> v1:172.21.15.150:6789/0 conn(0x7f0dcc0c25e0 legacy=0x7f0dcc0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc0c4b90 con 0x7f0dcc0c5be0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0dc40019f0 con 0x7f0dcc0c5be0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/1499741811 >> v2:172.21.15.150:3300/0 conn(0x7f0dcc0c29c0 0x7f0dcc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 >> v1:172.21.15.78:6789/0 conn(0x7f0dcc0c5be0 legacy=0x7f0dcc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0db4003970 0x7f0db4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0db4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0db40052a0 0x7f0db400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc152c90 con 0x7f0db40047c0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db400c170 con 0x7f0db4003970 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db400e540 con 0x7f0db40052a0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0db4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcb7fe640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0db4003970 0x7f0db4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcbfff640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0db40052a0 0x7f0db400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 -- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0db40052a0 msgr2=0x7f0db400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0db40052a0 0x7f0db400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 -- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0db4003970 msgr2=0x7f0db4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:12.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0db4003970 0x7f0db4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcaffd640 1 -- 172.21.15.78:0/1499741811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4003520 con 0x7f0db40047c0 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcbfff640 1 -- 172.21.15.78:0/1499741811 reap_dead start 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.244+0000 7f0dcb7fe640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0db4003970 0x7f0db4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0db4004c40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0006fd0 tx=0x7f0db40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f113bf94cad6abc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0004230 con 0x7f0db40047c0 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0004390 con 0x7f0db40047c0 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dca7fc640 1 -- 172.21.15.78:0/1499741811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0003830 con 0x7f0db40047c0 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dd20c7640 1 -- 172.21.15.78:0/1499741811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 msgr2=0x7f0db4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0db4004c40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0006fd0 tx=0x7f0db40057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dd20c7640 1 -- 172.21.15.78:0/1499741811 shutdown_connections 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.245+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/1499741811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0db4004c40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.246+0000 7f0dd20c7640 1 -- 172.21.15.78:0/1499741811 >> 172.21.15.78:0/1499741811 conn(0x7f0dcc06b270 msgr2=0x7f0dcc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.246+0000 7f0dd20c7640 1 -- 172.21.15.78:0/1499741811 shutdown_connections 2024-02-19T14:54:12.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.246+0000 7f0dd20c7640 1 -- 172.21.15.78:0/1499741811 wait complete. 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 Processor -- start 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 -- start start 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0dcc1e9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dcc0a9630 0x7f0dcc154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0dcc1e9ec0 0x7f0dcc155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc1559c0 con 0x7f0db40047c0 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0a67b0 con 0x7f0dcc0a9630 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.247+0000 7f0dd20c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc0a6fc0 con 0x7f0dcc1e9ec0 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0dcc1e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0dcc1e9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38424/0 (socket says 172.21.15.78:38424) 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 -- 172.21.15.78:0/3862908586 learned_addr learned my addr 172.21.15.78:0/3862908586 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:12.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcbfff640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0dcc1e9ec0 0x7f0dcc155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dcc0a9630 0x7f0dcc154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 -- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0dcc1e9ec0 msgr2=0x7f0dcc155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0dcc1e9ec0 0x7f0dcc155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 -- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dcc0a9630 msgr2=0x7f0dcc154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dcc0a9630 0x7f0dcc154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 -- 172.21.15.78:0/3862908586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc0a9060 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dcb7fe640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0dcc1e9980 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00e480 tx=0x7f0dbc00e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc017070 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc007e90 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc1ea490 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0052e0 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.248+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc0a6ce0 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.249+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc074990 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.250+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0dbc007020 con 0x7f0db40047c0 2024-02-19T14:54:12.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.250+0000 7f0db3fff640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0db806b840 0x7f0db806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.250+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0db806b840 0x7f0db806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.250+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f0dbc089ab0 con 0x7f0db40047c0 2024-02-19T14:54:12.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.250+0000 7f0dcaffd640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0db806b840 0x7f0db806dd00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0035000 tx=0x7f0dc003a040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.254+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0dbc054270 con 0x7f0db40047c0 2024-02-19T14:54:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.429+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0dcc14d080 con 0x7f0db40047c0 2024-02-19T14:54:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.429+0000 7f0db3fff640 1 -- 172.21.15.78:0/3862908586 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v63) v1 ==== 105+0+11 (secure 0 0 0) 0x7f0dbc057930 con 0x7f0db40047c0 2024-02-19T14:54:12.432 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0db806b840 msgr2=0x7f0db806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0db806b840 0x7f0db806dd00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0035000 tx=0x7f0dc003a040 comp rx=0 tx=0).stop 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 msgr2=0x7f0dcc1e9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0dcc1e9980 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00e480 tx=0x7f0dbc00e940 comp rx=0 tx=0).stop 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 shutdown_connections 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0db806b840 0x7f0db806dd00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0dcc1e9ec0 0x7f0dcc155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dcc0a9630 0x7f0dcc154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 --2- 172.21.15.78:0/3862908586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0db40047c0 0x7f0dcc1e9980 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 >> 172.21.15.78:0/3862908586 conn(0x7f0dcc06b270 msgr2=0x7f0dcc150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 shutdown_connections 2024-02-19T14:54:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.431+0000 7f0dd20c7640 1 -- 172.21.15.78:0/3862908586 wait complete. 2024-02-19T14:54:12.445 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-19T14:54:12.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.547+0000 7f1a1930c640 1 Processor -- start 2024-02-19T14:54:12.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.547+0000 7f1a1930c640 1 -- start start 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a1930c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1a140ac390 0x7f1a140ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a1930c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a14158c40 con 0x7f1a140c55f0 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a1930c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140c8de0 con 0x7f1a140bdf20 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a1930c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a140c5a30 con 0x7f1a140ac390 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a12ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1a140ac390 0x7f1a140ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a137fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1a140c55f0 0x7f1a1415f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38920/0 (socket says 172.21.15.78:38920) 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a12ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1a140ac390 0x7f1a140ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38432/0 (socket says 172.21.15.78:38432) 2024-02-19T14:54:12.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a137fe640 1 -- 172.21.15.78:0/1071352666 learned_addr learned my addr 172.21.15.78:0/1071352666 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:12.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a12ffd640 1 -- 172.21.15.78:0/1071352666 >> v1:172.21.15.150:6789/0 conn(0x7f1a140bdf20 legacy=0x7f1a14158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a12ffd640 1 -- 172.21.15.78:0/1071352666 >> v1:172.21.15.78:6789/0 conn(0x7f1a140c55f0 legacy=0x7f1a1415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a12ffd640 1 -- 172.21.15.78:0/1071352666 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a14155a60 con 0x7f1a140ac390 2024-02-19T14:54:12.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.548+0000 7f1a11ffb640 1 -- 172.21.15.78:0/1071352666 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082398858 0 0) 0x7f1a14158c40 con 0x7f1a140c55f0 2024-02-19T14:54:12.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.549+0000 7f1a12ffd640 1 --2- 172.21.15.78:0/1071352666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ac770 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1a040097a0 tx=0x7f1a0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc9ebdf3c13a664 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.549+0000 7f1a11ffb640 1 -- 172.21.15.78:0/1071352666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0402e070 con 0x7f1a140ac390 2024-02-19T14:54:12.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.549+0000 7f1a11ffb640 1 -- 172.21.15.78:0/1071352666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a040315a0 con 0x7f1a140ac390 2024-02-19T14:54:12.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.549+0000 7f1a11ffb640 1 -- 172.21.15.78:0/1071352666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04031930 con 0x7f1a140ac390 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 -- 172.21.15.78:0/1071352666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 msgr2=0x7f1a140ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 --2- 172.21.15.78:0/1071352666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ac770 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1a040097a0 tx=0x7f1a0402b840 comp rx=0 tx=0).stop 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 -- 172.21.15.78:0/1071352666 shutdown_connections 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 --2- 172.21.15.78:0/1071352666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ac770 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 -- 172.21.15.78:0/1071352666 >> 172.21.15.78:0/1071352666 conn(0x7f1a140ca890 msgr2=0x7f1a140caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 -- 172.21.15.78:0/1071352666 shutdown_connections 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.550+0000 7f1a1930c640 1 -- 172.21.15.78:0/1071352666 wait complete. 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.551+0000 7f1a1930c640 1 Processor -- start 2024-02-19T14:54:12.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.551+0000 7f1a1930c640 1 -- start start 2024-02-19T14:54:12.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a140bdf20 0x7f1a140bf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a140c55f0 0x7f1a140bf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140c16d0 con 0x7f1a140ac390 2024-02-19T14:54:12.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140bd2d0 con 0x7f1a140c55f0 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140cb170 con 0x7f1a140bdf20 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38434/0 (socket says 172.21.15.78:38434) 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 -- 172.21.15.78:0/3482858770 learned_addr learned my addr 172.21.15.78:0/3482858770 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a137fe640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a140c55f0 0x7f1a140bf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a127fc640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a140bdf20 0x7f1a140bf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 -- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a140bdf20 msgr2=0x7f1a140bf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a140bdf20 0x7f1a140bf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 -- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a140c55f0 msgr2=0x7f1a140bf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a140c55f0 0x7f1a140bf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 -- 172.21.15.78:0/3482858770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04009400 con 0x7f1a140ac390 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a12ffd640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ae770 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1a04034040 tx=0x7f1a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04032980 con 0x7f1a140ac390 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04032ae0 con 0x7f1a140ac390 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04032e90 con 0x7f1a140ac390 2024-02-19T14:54:12.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a141eae00 con 0x7f1a140ac390 2024-02-19T14:54:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.552+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a140bd9c0 con 0x7f1a140ac390 2024-02-19T14:54:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.553+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a14074990 con 0x7f1a140ac390 2024-02-19T14:54:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.554+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1a0403b3f0 con 0x7f1a140ac390 2024-02-19T14:54:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.554+0000 7f19fb7fe640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a0006b810 0x7f1a0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.554+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f1a040ac650 con 0x7f1a140ac390 2024-02-19T14:54:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.554+0000 7f1a127fc640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a0006b810 0x7f1a0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.555+0000 7f1a127fc640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a0006b810 0x7f1a0006dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a14154400 tx=0x7f1a08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.559+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a04076d80 con 0x7f1a140ac390 2024-02-19T14:54:12.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.734+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f1a1406da00 con 0x7f1a140ac390 2024-02-19T14:54:12.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.734+0000 7f19fb7fe640 1 -- 172.21.15.78:0/3482858770 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v63) v1 ==== 101+0+11 (secure 0 0 0) 0x7f1a0407a440 con 0x7f1a140ac390 2024-02-19T14:54:12.737 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a0006b810 msgr2=0x7f1a0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a0006b810 0x7f1a0006dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a14154400 tx=0x7f1a08009040 comp rx=0 tx=0).stop 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 msgr2=0x7f1a140ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ae770 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1a04034040 tx=0x7f1a040046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 shutdown_connections 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a0006b810 0x7f1a0006dcd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a140c55f0 0x7f1a140bf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a140bdf20 0x7f1a140bf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 --2- 172.21.15.78:0/3482858770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a140ac390 0x7f1a140ae770 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 >> 172.21.15.78:0/3482858770 conn(0x7f1a140ca890 msgr2=0x7f1a1406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 shutdown_connections 2024-02-19T14:54:12.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.736+0000 7f1a1930c640 1 -- 172.21.15.78:0/3482858770 wait complete. 2024-02-19T14:54:12.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:12.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.848+0000 7f44c99c7640 1 Processor -- start 2024-02-19T14:54:12.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.848+0000 7f44c99c7640 1 -- start start 2024-02-19T14:54:12.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c99c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c51b0 0x7f44c40c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c99c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f44c40c59f0 0x7f44c415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c99c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c40c9200 con 0x7f44c40c5590 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c99c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c40cb720 con 0x7f44c40c59f0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c99c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4158ff0 con 0x7f44c40c51b0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f44c40c59f0 0x7f44c415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f44c40c59f0 0x7f44c415f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38438/0 (socket says 172.21.15.78:38438) 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 -- 172.21.15.78:0/630645804 learned_addr learned my addr 172.21.15.78:0/630645804 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c2ffd640 1 --2- 172.21.15.78:0/630645804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c51b0 0x7f44c40c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 -- 172.21.15.78:0/630645804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c51b0 msgr2=0x7f44c40c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 --2- 172.21.15.78:0/630645804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c51b0 0x7f44c40c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 -- 172.21.15.78:0/630645804 >> v1:172.21.15.150:6789/0 conn(0x7f44c40c5590 legacy=0x7f44c40c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.849+0000 7f44c37fe640 1 -- 172.21.15.78:0/630645804 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c41592d0 con 0x7f44c40c59f0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.850+0000 7f44c37fe640 1 --2- 172.21.15.78:0/630645804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c59f0 0x7f44c415f8d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009e50 tx=0x7f44bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ec243762cc3c40 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.850+0000 7f44c1ffb640 1 -- 172.21.15.78:0/630645804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc02cb00 con 0x7f44c40c59f0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.850+0000 7f44c1ffb640 1 -- 172.21.15.78:0/630645804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc02cc60 con 0x7f44c40c59f0 2024-02-19T14:54:12.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.850+0000 7f44c1ffb640 1 -- 172.21.15.78:0/630645804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0343f0 con 0x7f44c40c59f0 2024-02-19T14:54:12.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 -- 172.21.15.78:0/630645804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c59f0 msgr2=0x7f44c415f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 --2- 172.21.15.78:0/630645804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c59f0 0x7f44c415f8d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009e50 tx=0x7f44bc02bf60 comp rx=0 tx=0).stop 2024-02-19T14:54:12.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 -- 172.21.15.78:0/630645804 shutdown_connections 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 --2- 172.21.15.78:0/630645804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c59f0 0x7f44c415f8d0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 --2- 172.21.15.78:0/630645804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c51b0 0x7f44c40c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 -- 172.21.15.78:0/630645804 >> 172.21.15.78:0/630645804 conn(0x7f44c40ca830 msgr2=0x7f44c40cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 -- 172.21.15.78:0/630645804 shutdown_connections 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.851+0000 7f44c99c7640 1 -- 172.21.15.78:0/630645804 wait complete. 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.852+0000 7f44c99c7640 1 Processor -- start 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.852+0000 7f44c99c7640 1 -- start start 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 0x7f44c41e0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c5590 0x7f44c4156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40c59f0 0x7f44c41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4149360 con 0x7f44c40c51b0 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c414a6f0 con 0x7f44c40c59f0 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40aaf10 con 0x7f44c40c5590 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 0x7f44c41e0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 0x7f44c41e0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38448/0 (socket says 172.21.15.78:38448) 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 -- 172.21.15.78:0/1258539049 learned_addr learned my addr 172.21.15.78:0/1258539049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c27fc640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c5590 0x7f44c4156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c37fe640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40c59f0 0x7f44c41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 -- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c5590 msgr2=0x7f44c4156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c5590 0x7f44c4156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 -- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40c59f0 msgr2=0x7f44c41570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40c59f0 0x7f44c41570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 -- 172.21.15.78:0/1258539049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc009b30 con 0x7f44c40c51b0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c2ffd640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 0x7f44c41e0d00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f44b400e870 tx=0x7f44b400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b40094f0 con 0x7f44c40c51b0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b400c940 con 0x7f44c40c51b0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c4149080 con 0x7f44c40c51b0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b400ccf0 con 0x7f44c40c51b0 2024-02-19T14:54:12.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.853+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c414a380 con 0x7f44c40c51b0 2024-02-19T14:54:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.854+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c414a540 con 0x7f44c40c51b0 2024-02-19T14:54:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.855+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f44b4013470 con 0x7f44c40c51b0 2024-02-19T14:54:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.855+0000 7f44abfff640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f44b006b840 0x7f44b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:12.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.855+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f44b408a400 con 0x7f44c40c51b0 2024-02-19T14:54:12.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.855+0000 7f44c27fc640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f44b006b840 0x7f44b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:12.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.856+0000 7f44c27fc640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f44b006b840 0x7f44b006dd00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f44c40c16b0 tx=0x7f44b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:12.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:12.860+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f44b4054c70 con 0x7f44c40c51b0 2024-02-19T14:54:12.947 INFO:tasks.ceph.mds.b.smithi150.stderr:2024-02-19T14:54:12.945+0000 7f85fa383640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:13.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.067+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44c4149ed0 con 0x7f44c40c51b0 2024-02-19T14:54:13.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.068+0000 7f44abfff640 1 -- 172.21.15.78:0/1258539049 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f44b4058330 con 0x7f44c40c51b0 2024-02-19T14:54:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:13.070 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1284379174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1284379174},{"type":"v1","addr":"172.21.15.78:6837","nonce":1284379174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6171,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/4156997217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4156997217},{"type":"v1","addr":"172.21.15.78:6836","nonce":4156997217}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:54:11.528053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5401":{"gid":5401,"name":"b","rank":0,"incarnation":42,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:13.071 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 45 2024-02-19T14:54:13.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f44b006b840 msgr2=0x7f44b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f44b006b840 0x7f44b006dd00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f44c40c16b0 tx=0x7f44b8009040 comp rx=0 tx=0).stop 2024-02-19T14:54:13.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 msgr2=0x7f44c41e0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 0x7f44c41e0d00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f44b400e870 tx=0x7f44b400ed30 comp rx=0 tx=0).stop 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 shutdown_connections 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f44b006b840 0x7f44b006dd00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40c59f0 0x7f44c41570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44c40c5590 0x7f44c4156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.069+0000 7f44c99c7640 1 --2- 172.21.15.78:0/1258539049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44c40c51b0 0x7f44c41e0d00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.070+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 >> 172.21.15.78:0/1258539049 conn(0x7f44c40ca830 msgr2=0x7f44c40c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.070+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 shutdown_connections 2024-02-19T14:54:13.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.070+0000 7f44c99c7640 1 -- 172.21.15.78:0/1258539049 wait complete. 2024-02-19T14:54:13.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:13.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.180+0000 7fbd2829a640 1 Processor -- start 2024-02-19T14:54:13.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.180+0000 7fbd2829a640 1 -- start start 2024-02-19T14:54:13.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd2829a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c4d80 0x7fbd20159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd2829a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd20155010 0x7fbd201523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd2829a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2006ffa0 con 0x7fbd200c78a0 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd2829a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2006b720 con 0x7fbd20155010 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd2829a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2006fc70 con 0x7fbd200c4d80 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd26036640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbd200c78a0 0x7fbd200c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38934/0 (socket says 172.21.15.78:38934) 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd26036640 1 -- 172.21.15.78:0/4121107087 learned_addr learned my addr 172.21.15.78:0/4121107087 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd25835640 1 --2- 172.21.15.78:0/4121107087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c4d80 0x7fbd20159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd26837640 1 --2- 172.21.15.78:0/4121107087 >> v2:172.21.15.150:3300/0 conn(0x7fbd20155010 0x7fbd201523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd25034640 1 -- 172.21.15.78:0/4121107087 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957319668 0 0) 0x7fbd2006ffa0 con 0x7fbd200c78a0 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.181+0000 7fbd25034640 1 -- 172.21.15.78:0/4121107087 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c003520 con 0x7fbd200c78a0 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.182+0000 7fbd25835640 1 -- 172.21.15.78:0/4121107087 >> v2:172.21.15.150:3300/0 conn(0x7fbd20155010 msgr2=0x7fbd201523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.182+0000 7fbd25835640 1 --2- 172.21.15.78:0/4121107087 >> v2:172.21.15.150:3300/0 conn(0x7fbd20155010 0x7fbd201523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.182+0000 7fbd25835640 1 -- 172.21.15.78:0/4121107087 >> v1:172.21.15.78:6789/0 conn(0x7fbd200c78a0 legacy=0x7fbd200c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.182+0000 7fbd25835640 1 -- 172.21.15.78:0/4121107087 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2006b930 con 0x7fbd200c4d80 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.182+0000 7fbd25835640 1 --2- 172.21.15.78:0/4121107087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c4d80 0x7fbd20159010 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbd100097a0 tx=0x7fbd1002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=706ca029955d440f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.182+0000 7fbd25034640 1 -- 172.21.15.78:0/4121107087 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1002e070 con 0x7fbd200c4d80 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd25034640 1 -- 172.21.15.78:0/4121107087 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd100315a0 con 0x7fbd200c4d80 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd25034640 1 -- 172.21.15.78:0/4121107087 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10031910 con 0x7fbd200c4d80 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 -- 172.21.15.78:0/4121107087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c4d80 msgr2=0x7fbd20159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 --2- 172.21.15.78:0/4121107087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c4d80 0x7fbd20159010 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbd100097a0 tx=0x7fbd1002b840 comp rx=0 tx=0).stop 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 -- 172.21.15.78:0/4121107087 shutdown_connections 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 --2- 172.21.15.78:0/4121107087 >> v2:172.21.15.150:3300/0 conn(0x7fbd20155010 0x7fbd201523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 --2- 172.21.15.78:0/4121107087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c4d80 0x7fbd20159010 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 -- 172.21.15.78:0/4121107087 >> 172.21.15.78:0/4121107087 conn(0x7fbd20066e70 msgr2=0x7fbd200a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 -- 172.21.15.78:0/4121107087 shutdown_connections 2024-02-19T14:54:13.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.183+0000 7fbd2829a640 1 -- 172.21.15.78:0/4121107087 wait complete. 2024-02-19T14:54:13.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.184+0000 7fbd2829a640 1 Processor -- start 2024-02-19T14:54:13.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.184+0000 7fbd2829a640 1 -- start start 2024-02-19T14:54:13.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd2829a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 0x7fbd200ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd2829a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c78a0 0x7fbd201eb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd2829a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd20155010 0x7fbd2014e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd2829a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd20150a40 con 0x7fbd200c4d80 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd2829a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd201ebc60 con 0x7fbd20155010 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd2829a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd201eaca0 con 0x7fbd200c78a0 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 0x7fbd200ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 0x7fbd200ac210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38454/0 (socket says 172.21.15.78:38454) 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 -- 172.21.15.78:0/2685317797 learned_addr learned my addr 172.21.15.78:0/2685317797 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd25835640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c78a0 0x7fbd201eb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26837640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd20155010 0x7fbd2014e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 -- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c78a0 msgr2=0x7fbd201eb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c78a0 0x7fbd201eb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 -- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd20155010 msgr2=0x7fbd2014e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd20155010 0x7fbd2014e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 -- 172.21.15.78:0/2685317797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd10009400 con 0x7fbd200c4d80 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd26036640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 0x7fbd200ac210 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00caa0 tx=0x7fbd1c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.185+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c00ac00 con 0x7fbd200c4d80 2024-02-19T14:54:13.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.186+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c00ad60 con 0x7fbd200c4d80 2024-02-19T14:54:13.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.186+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd200abbf0 con 0x7fbd200c4d80 2024-02-19T14:54:13.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.186+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c012620 con 0x7fbd200c4d80 2024-02-19T14:54:13.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.186+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd201eaf20 con 0x7fbd200c4d80 2024-02-19T14:54:13.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.186+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd200c2b70 con 0x7fbd200c4d80 2024-02-19T14:54:13.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.187+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbd1c007070 con 0x7fbd200c4d80 2024-02-19T14:54:13.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.187+0000 7fbd16ffd640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd0406b810 0x7fbd0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.187+0000 7fbd25835640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd0406b810 0x7fbd0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.187+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fbd1c08a3c0 con 0x7fbd200c4d80 2024-02-19T14:54:13.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.188+0000 7fbd25835640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd0406b810 0x7fbd0406dcd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbd10036000 tx=0x7fbd1003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.192+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbd1c054bb0 con 0x7fbd200c4d80 2024-02-19T14:54:13.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.364+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd20074990 con 0x7fbd200c4d80 2024-02-19T14:54:13.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.366+0000 7fbd16ffd640 1 -- 172.21.15.78:0/2685317797 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+17395 (secure 0 0 0) 0x7fbd1c001b60 con 0x7fbd200c4d80 2024-02-19T14:54:13.368 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:13.369 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":63,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:11.510672+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6837/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.78:6834/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6835/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:13.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.367+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd0406b810 msgr2=0x7fbd0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.367+0000 7fbd2829a640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd0406b810 0x7fbd0406dcd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbd10036000 tx=0x7fbd1003d040 comp rx=0 tx=0).stop 2024-02-19T14:54:13.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 msgr2=0x7fbd200ac210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 0x7fbd200ac210 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00caa0 tx=0x7fbd1c00cf60 comp rx=0 tx=0).stop 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 shutdown_connections 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd0406b810 0x7fbd0406dcd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd20155010 0x7fbd2014e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd200c78a0 0x7fbd201eb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 --2- 172.21.15.78:0/2685317797 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd200c4d80 0x7fbd200ac210 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 >> 172.21.15.78:0/2685317797 conn(0x7fbd20066e70 msgr2=0x7fbd2006f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 shutdown_connections 2024-02-19T14:54:13.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.368+0000 7fbd2829a640 1 -- 172.21.15.78:0/2685317797 wait complete. 2024-02-19T14:54:13.381 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-02-19T14:54:13.381 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:13.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.478+0000 7ff762fd7640 1 Processor -- start 2024-02-19T14:54:13.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.478+0000 7ff762fd7640 1 -- start start 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff762fd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c155780 0x7ff75c160920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff762fd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c072e80 con 0x7ff75c072180 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff762fd7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75c06e990 con 0x7ff75c1528d0 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff762fd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c0734c0 con 0x7ff75c155780 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff753fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff75c072180 0x7ff75c155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38936/0 (socket says 172.21.15.78:38936) 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff753fff640 1 -- 172.21.15.78:0/421994528 learned_addr learned my addr 172.21.15.78:0/421994528 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff761574640 1 --2- 172.21.15.78:0/421994528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c155780 0x7ff75c160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438984007 0 0) 0x7ff74c001110 con 0x7ff75c072180 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff744003520 con 0x7ff75c072180 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368385855 0 0) 0x7ff75c06e990 con 0x7ff75c1528d0 2024-02-19T14:54:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74c001110 con 0x7ff75c1528d0 2024-02-19T14:54:13.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff761574640 1 -- 172.21.15.78:0/421994528 >> v1:172.21.15.150:6789/0 conn(0x7ff75c1528d0 legacy=0x7ff75c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff761574640 1 -- 172.21.15.78:0/421994528 >> v1:172.21.15.78:6789/0 conn(0x7ff75c072180 legacy=0x7ff75c155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.479+0000 7ff761574640 1 -- 172.21.15.78:0/421994528 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c0731e0 con 0x7ff75c155780 2024-02-19T14:54:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.480+0000 7ff761574640 1 --2- 172.21.15.78:0/421994528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c155780 0x7ff75c160920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff754009dc0 tx=0x7ff75402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2bdc8018837fd97 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.480+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75402cc40 con 0x7ff75c155780 2024-02-19T14:54:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.480+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75402cda0 con 0x7ff75c155780 2024-02-19T14:54:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.480+0000 7ff7537fe640 1 -- 172.21.15.78:0/421994528 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754034660 con 0x7ff75c155780 2024-02-19T14:54:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 -- 172.21.15.78:0/421994528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c155780 msgr2=0x7ff75c160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 --2- 172.21.15.78:0/421994528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c155780 0x7ff75c160920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff754009dc0 tx=0x7ff75402be90 comp rx=0 tx=0).stop 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 -- 172.21.15.78:0/421994528 shutdown_connections 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 --2- 172.21.15.78:0/421994528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c155780 0x7ff75c160920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 -- 172.21.15.78:0/421994528 >> 172.21.15.78:0/421994528 conn(0x7ff75c0c7590 msgr2=0x7ff75c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 -- 172.21.15.78:0/421994528 shutdown_connections 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.481+0000 7ff762fd7640 1 -- 172.21.15.78:0/421994528 wait complete. 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.482+0000 7ff762fd7640 1 Processor -- start 2024-02-19T14:54:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.482+0000 7ff762fd7640 1 -- start start 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c072180 0x7ff75c1eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 0x7ff75c154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff75c155780 0x7ff75c1df640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c0ad390 con 0x7ff75c1528d0 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c0a8c70 con 0x7ff75c155780 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c1ebce0 con 0x7ff75c072180 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 0x7ff75c154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 0x7ff75c154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38470/0 (socket says 172.21.15.78:38470) 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 -- 172.21.15.78:0/3828470130 learned_addr learned my addr 172.21.15.78:0/3828470130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:13.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff761574640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff75c155780 0x7ff75c1df640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff760d73640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c072180 0x7ff75c1eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 -- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c072180 msgr2=0x7ff75c1eb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c072180 0x7ff75c1eb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 -- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff75c155780 msgr2=0x7ff75c1df640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff75c155780 0x7ff75c1df640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 -- 172.21.15.78:0/3828470130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff754009a70 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff753fff640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 0x7ff75c154270 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c730 tx=0x7ff74c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c005ab0 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c005c10 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c007070 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c1e1490 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.483+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c1e2110 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.484+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75c06ee20 con 0x7ff75c1528d0 2024-02-19T14:54:13.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.485+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff74c0073e0 con 0x7ff75c1528d0 2024-02-19T14:54:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.485+0000 7ff7517fa640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff74806b810 0x7ff74806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.485+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7ff74c089a60 con 0x7ff75c1528d0 2024-02-19T14:54:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.485+0000 7ff760d73640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff74806b810 0x7ff74806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.486+0000 7ff760d73640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff74806b810 0x7ff74806dcd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff758007fd0 tx=0x7ff758009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.489+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff74c0542d0 con 0x7ff75c1528d0 2024-02-19T14:54:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.695+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff75c072e80 con 0x7ff75c1528d0 2024-02-19T14:54:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.696+0000 7ff7517fa640 1 -- 172.21.15.78:0/3828470130 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4946 (secure 0 0 0) 0x7ff75c072e80 con 0x7ff75c1528d0 2024-02-19T14:54:13.699 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:13.699 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1284379174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1284379174},{"type":"v1","addr":"172.21.15.78:6837","nonce":1284379174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6171,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/4156997217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4156997217},{"type":"v1","addr":"172.21.15.78:6836","nonce":4156997217}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:54:11.528053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5401":{"gid":5401,"name":"b","rank":0,"incarnation":42,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:13.699 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 45 2024-02-19T14:54:13.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff74806b810 msgr2=0x7ff74806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff74806b810 0x7ff74806dcd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff758007fd0 tx=0x7ff758009040 comp rx=0 tx=0).stop 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 msgr2=0x7ff75c154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 0x7ff75c154270 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c730 tx=0x7ff74c00cbf0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 shutdown_connections 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff74806b810 0x7ff74806dcd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff75c1528d0 0x7ff75c154270 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff75c155780 0x7ff75c1df640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 --2- 172.21.15.78:0/3828470130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff75c072180 0x7ff75c1eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 >> 172.21.15.78:0/3828470130 conn(0x7ff75c0c7590 msgr2=0x7ff75c1583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 shutdown_connections 2024-02-19T14:54:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.698+0000 7ff762fd7640 1 -- 172.21.15.78:0/3828470130 wait complete. 2024-02-19T14:54:13.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.808+0000 7f8d495f4640 1 Processor -- start 2024-02-19T14:54:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.808+0000 7f8d495f4640 1 -- start start 2024-02-19T14:54:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d495f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d440c29e0 0x7f8d44071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d495f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 0x7f8d441536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d495f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4406fe40 con 0x7f8d440713a0 2024-02-19T14:54:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d495f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d440c32c0 con 0x7f8d440c29e0 2024-02-19T14:54:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d495f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44070500 con 0x7f8d440c8b60 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d427fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d440c29e0 0x7f8d44071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d437fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 0x7f8d441536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d42ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8d440713a0 0x7f8d440c64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:39760/0 (socket says 172.21.15.78:39760) 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d437fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 0x7f8d441536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33042/0 (socket says 172.21.15.78:33042) 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d42ffd640 1 -- 172.21.15.78:0/3680434409 learned_addr learned my addr 172.21.15.78:0/3680434409 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d427fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d440c29e0 0x7f8d44071780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:54326/0 (socket says 172.21.15.78:54326) 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d427fc640 1 -- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 msgr2=0x7f8d441536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d427fc640 1 --2- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 0x7f8d441536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d427fc640 1 -- 172.21.15.78:0/3680434409 >> v1:172.21.15.150:6789/0 conn(0x7f8d440713a0 legacy=0x7f8d440c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d41ffb640 1 -- 172.21.15.78:0/3680434409 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543157212 0 0) 0x7f8d4406fe40 con 0x7f8d440713a0 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d427fc640 1 -- 172.21.15.78:0/3680434409 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d44153bf0 con 0x7f8d440c29e0 2024-02-19T14:54:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.809+0000 7f8d437fe640 1 --2- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 0x7f8d441536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.810+0000 7f8d427fc640 1 --2- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c29e0 0x7f8d44071780 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d380097a0 tx=0x7f8d3802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8151813816ec0be server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.810+0000 7f8d41ffb640 1 -- 172.21.15.78:0/3680434409 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3802e070 con 0x7f8d440c29e0 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.810+0000 7f8d41ffb640 1 -- 172.21.15.78:0/3680434409 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d380315a0 con 0x7f8d440c29e0 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.810+0000 7f8d41ffb640 1 -- 172.21.15.78:0/3680434409 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d38031930 con 0x7f8d440c29e0 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 -- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c29e0 msgr2=0x7f8d44071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 --2- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c29e0 0x7f8d44071780 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d380097a0 tx=0x7f8d3802b840 comp rx=0 tx=0).stop 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 -- 172.21.15.78:0/3680434409 shutdown_connections 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 --2- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440c8b60 0x7f8d441536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 --2- 172.21.15.78:0/3680434409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c29e0 0x7f8d44071780 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 -- 172.21.15.78:0/3680434409 >> 172.21.15.78:0/3680434409 conn(0x7f8d440aa300 msgr2=0x7f8d440aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 -- 172.21.15.78:0/3680434409 shutdown_connections 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.811+0000 7f8d495f4640 1 -- 172.21.15.78:0/3680434409 wait complete. 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.812+0000 7f8d495f4640 1 Processor -- start 2024-02-19T14:54:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.812+0000 7f8d495f4640 1 -- start start 2024-02-19T14:54:13.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d495f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 0x7f8d441487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d495f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d440c29e0 0x7f8d44148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d495f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c8b60 0x7f8d441565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d495f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44154ad0 con 0x7f8d440c29e0 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d495f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d440be390 con 0x7f8d440c8b60 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d495f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d441542c0 con 0x7f8d440713a0 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d427fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d440c29e0 0x7f8d44148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 0x7f8d441487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 0x7f8d441487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33048/0 (socket says 172.21.15.78:33048) 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d427fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d440c29e0 0x7f8d44148d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38476/0 (socket says 172.21.15.78:38476) 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d437fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c8b60 0x7f8d441565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 -- 172.21.15.78:0/4072702234 learned_addr learned my addr 172.21.15.78:0/4072702234 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 -- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c8b60 msgr2=0x7f8d441565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c8b60 0x7f8d441565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 -- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d440c29e0 msgr2=0x7f8d44148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:13.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d440c29e0 0x7f8d44148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.813+0000 7f8d42ffd640 1 -- 172.21.15.78:0/4072702234 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c009400 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.814+0000 7f8d42ffd640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 0x7f8d441487c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c890 tx=0x7f8d3400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.814+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3400a640 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.814+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3400a7c0 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.814+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d38009400 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.814+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3400ab70 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.814+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d44156120 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.815+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d44149240 con 0x7f8d440713a0 2024-02-19T14:54:13.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.815+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8d3400acd0 con 0x7f8d440713a0 2024-02-19T14:54:13.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.816+0000 7f8d2b7fe640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3006b840 0x7f8d3006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:13.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.816+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f8d34089b60 con 0x7f8d440713a0 2024-02-19T14:54:13.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.816+0000 7f8d427fc640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3006b840 0x7f8d3006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:13.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.816+0000 7f8d427fc640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3006b840 0x7f8d3006dd00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8d38036000 tx=0x7f8d3803e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:13.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:13.820+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8d340543d0 con 0x7f8d440713a0 2024-02-19T14:54:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.027+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d440add90 con 0x7f8d440713a0 2024-02-19T14:54:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.027+0000 7f8d2b7fe640 1 -- 172.21.15.78:0/4072702234 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f8d34018020 con 0x7f8d440713a0 2024-02-19T14:54:14.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:14.030 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":45,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1284379174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1284379174},{"type":"v1","addr":"172.21.15.78:6837","nonce":1284379174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6171,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/4156997217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4156997217},{"type":"v1","addr":"172.21.15.78:6836","nonce":4156997217}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:53:48.213287+0000","modified":"2024-02-19T14:54:11.528053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5401":{"gid":5401,"name":"b","rank":0,"incarnation":42,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/2194753931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194753931},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194753931}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-19T14:54:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 45 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3006b840 msgr2=0x7f8d3006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3006b840 0x7f8d3006dd00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8d38036000 tx=0x7f8d3803e040 comp rx=0 tx=0).stop 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 msgr2=0x7f8d441487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 0x7f8d441487c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c890 tx=0x7f8d3400cc90 comp rx=0 tx=0).stop 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 shutdown_connections 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8d3006b840 0x7f8d3006dd00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d440c29e0 0x7f8d44148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d440c8b60 0x7f8d441565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 --2- 172.21.15.78:0/4072702234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d440713a0 0x7f8d441487c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 >> 172.21.15.78:0/4072702234 conn(0x7f8d440aa300 msgr2=0x7f8d44152540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 shutdown_connections 2024-02-19T14:54:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.029+0000 7f8d495f4640 1 -- 172.21.15.78:0/4072702234 wait complete. 2024-02-19T14:54:14.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.139+0000 7f0b6028f640 1 Processor -- start 2024-02-19T14:54:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.139+0000 7f0b6028f640 1 -- start start 2024-02-19T14:54:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b6028f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b580c8e00 0x7f0b5806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b6028f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b5806d910 0x7f0b5806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b6028f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5806cd20 con 0x7f0b58152700 2024-02-19T14:54:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b6028f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b581551f0 con 0x7f0b580c8e00 2024-02-19T14:54:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b6028f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b58155490 con 0x7f0b5806d910 2024-02-19T14:54:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b5d82a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b580c8e00 0x7f0b5806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b5e02b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0b58152700 0x7f0b580c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38946/0 (socket says 172.21.15.78:38946) 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b5e02b640 1 -- 172.21.15.78:0/1860600859 learned_addr learned my addr 172.21.15.78:0/1860600859 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.140+0000 7f0b5e82c640 1 --2- 172.21.15.78:0/1860600859 >> v2:172.21.15.150:3300/0 conn(0x7f0b5806d910 0x7f0b5806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.141+0000 7f0b5d82a640 1 -- 172.21.15.78:0/1860600859 >> v2:172.21.15.150:3300/0 conn(0x7f0b5806d910 msgr2=0x7f0b5806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.141+0000 7f0b5d82a640 1 --2- 172.21.15.78:0/1860600859 >> v2:172.21.15.150:3300/0 conn(0x7f0b5806d910 0x7f0b5806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.141+0000 7f0b5d82a640 1 -- 172.21.15.78:0/1860600859 >> v1:172.21.15.78:6789/0 conn(0x7f0b58152700 legacy=0x7f0b580c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.141+0000 7f0b5d82a640 1 -- 172.21.15.78:0/1860600859 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58155770 con 0x7f0b580c8e00 2024-02-19T14:54:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.141+0000 7f0b5d029640 1 -- 172.21.15.78:0/1860600859 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506648938 0 0) 0x7f0b5806cd20 con 0x7f0b58152700 2024-02-19T14:54:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.141+0000 7f0b5d82a640 1 --2- 172.21.15.78:0/1860600859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b580c8e00 0x7f0b5806c2a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0b581596f0 tx=0x7f0b4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9774e7971a4f9d8c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b5d029640 1 -- 172.21.15.78:0/1860600859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48031830 con 0x7f0b580c8e00 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b5d029640 1 -- 172.21.15.78:0/1860600859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b48031990 con 0x7f0b580c8e00 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b5d029640 1 -- 172.21.15.78:0/1860600859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48031d20 con 0x7f0b580c8e00 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 -- 172.21.15.78:0/1860600859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b580c8e00 msgr2=0x7f0b5806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 --2- 172.21.15.78:0/1860600859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b580c8e00 0x7f0b5806c2a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0b581596f0 tx=0x7f0b4802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 -- 172.21.15.78:0/1860600859 shutdown_connections 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 --2- 172.21.15.78:0/1860600859 >> v2:172.21.15.150:3300/0 conn(0x7f0b5806d910 0x7f0b5806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 --2- 172.21.15.78:0/1860600859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b580c8e00 0x7f0b5806c2a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 -- 172.21.15.78:0/1860600859 >> 172.21.15.78:0/1860600859 conn(0x7f0b580cac90 msgr2=0x7f0b580cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 -- 172.21.15.78:0/1860600859 shutdown_connections 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.142+0000 7f0b6028f640 1 -- 172.21.15.78:0/1860600859 wait complete. 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.143+0000 7f0b6028f640 1 Processor -- start 2024-02-19T14:54:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.143+0000 7f0b6028f640 1 -- start start 2024-02-19T14:54:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b6028f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b5806d910 0x7f0b581edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b6028f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8e00 0x7f0b580c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b6028f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 0x7f0b58158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b6028f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58154cc0 con 0x7f0b58152700 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b6028f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580c48f0 con 0x7f0b5806d910 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b6028f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58157d10 con 0x7f0b580c8e00 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 0x7f0b58158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 0x7f0b58158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38496/0 (socket says 172.21.15.78:38496) 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 -- 172.21.15.78:0/4214600039 learned_addr learned my addr 172.21.15.78:0/4214600039 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e02b640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b5806d910 0x7f0b581edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5d82a640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8e00 0x7f0b580c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 -- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8e00 msgr2=0x7f0b580c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8e00 0x7f0b580c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 -- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b5806d910 msgr2=0x7f0b581edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b5806d910 0x7f0b581edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 -- 172.21.15.78:0/4214600039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b48009400 con 0x7f0b58152700 2024-02-19T14:54:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.144+0000 7f0b5e82c640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 0x7f0b58158ce0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000efc0 tx=0x7f0b5000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.145+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50014480 con 0x7f0b58152700 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.145+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5000f040 con 0x7f0b58152700 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.145+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50014820 con 0x7f0b58152700 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.145+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b58157ff0 con 0x7f0b58152700 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.145+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b581587c0 con 0x7f0b58152700 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.146+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b581589e0 con 0x7f0b58152700 2024-02-19T14:54:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.146+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0b50014980 con 0x7f0b58152700 2024-02-19T14:54:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.146+0000 7f0b4effd640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b3c06b810 0x7f0b3c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.147+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f0b5008aef0 con 0x7f0b58152700 2024-02-19T14:54:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.147+0000 7f0b5e02b640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b3c06b810 0x7f0b3c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.147+0000 7f0b5e02b640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b3c06b810 0x7f0b3c06dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007fd0 tx=0x7f0b54007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.151+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0b50055760 con 0x7f0b58152700 2024-02-19T14:54:14.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.321+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b58074990 con 0x7f0b58152700 2024-02-19T14:54:14.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.322+0000 7f0b4effd640 1 -- 172.21.15.78:0/4214600039 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+17395 (secure 0 0 0) 0x7f0b50018020 con 0x7f0b58152700 2024-02-19T14:54:14.325 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:14.325 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":63,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:11.510672+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:53:45.797141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:53:47.077376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6837/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.78:6834/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6835/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b3c06b810 msgr2=0x7f0b3c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b3c06b810 0x7f0b3c06dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007fd0 tx=0x7f0b54007de0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 msgr2=0x7f0b58158ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 0x7f0b58158ce0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000efc0 tx=0x7f0b5000c490 comp rx=0 tx=0).stop 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 shutdown_connections 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b3c06b810 0x7f0b3c06dcd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b58152700 0x7f0b58158ce0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8e00 0x7f0b580c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 --2- 172.21.15.78:0/4214600039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b5806d910 0x7f0b581edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 >> 172.21.15.78:0/4214600039 conn(0x7f0b580cac90 msgr2=0x7f0b580ca110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 shutdown_connections 2024-02-19T14:54:14.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.324+0000 7f0b6028f640 1 -- 172.21.15.78:0/4214600039 wait complete. 2024-02-19T14:54:14.338 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-02-19T14:54:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.435+0000 7f53b838a640 1 Processor -- start 2024-02-19T14:54:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.435+0000 7f53b838a640 1 -- start start 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b838a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53b00c42e0 0x7f53b0158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b838a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53b00c2f50 0x7f53b00c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b838a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b00c34a0 con 0x7f53b00c8500 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b838a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b006c090 con 0x7f53b00c42e0 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b838a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b006d6a0 con 0x7f53b00c2f50 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53b00c42e0 0x7f53b0158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53b00c42e0 0x7f53b0158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38506/0 (socket says 172.21.15.78:38506) 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 -- 172.21.15.78:0/276873298 learned_addr learned my addr 172.21.15.78:0/276873298 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b6927640 1 --2- 172.21.15.78:0/276873298 >> v2:172.21.15.150:3300/0 conn(0x7f53b00c2f50 0x7f53b00c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 -- 172.21.15.78:0/276873298 >> v2:172.21.15.150:3300/0 conn(0x7f53b00c2f50 msgr2=0x7f53b00c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 --2- 172.21.15.78:0/276873298 >> v2:172.21.15.150:3300/0 conn(0x7f53b00c2f50 0x7f53b00c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 -- 172.21.15.78:0/276873298 >> v1:172.21.15.150:6789/0 conn(0x7f53b00c8500 legacy=0x7f53b0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5124640 1 -- 172.21.15.78:0/276873298 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888347346 0 0) 0x7f53b00c34a0 con 0x7f53b00c8500 2024-02-19T14:54:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.436+0000 7f53b5925640 1 -- 172.21.15.78:0/276873298 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b006c8e0 con 0x7f53b00c42e0 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.437+0000 7f53b5925640 1 --2- 172.21.15.78:0/276873298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c42e0 0x7f53b0158d60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f53b00adea0 tx=0x7f53a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7575a1032594386 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.437+0000 7f53b5124640 1 -- 172.21.15.78:0/276873298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0031870 con 0x7f53b00c42e0 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.437+0000 7f53b5124640 1 -- 172.21.15.78:0/276873298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a00319d0 con 0x7f53b00c42e0 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.437+0000 7f53b5124640 1 -- 172.21.15.78:0/276873298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0031d60 con 0x7f53b00c42e0 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 -- 172.21.15.78:0/276873298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c42e0 msgr2=0x7f53b0158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 --2- 172.21.15.78:0/276873298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c42e0 0x7f53b0158d60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f53b00adea0 tx=0x7f53a002b910 comp rx=0 tx=0).stop 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 -- 172.21.15.78:0/276873298 shutdown_connections 2024-02-19T14:54:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 --2- 172.21.15.78:0/276873298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c42e0 0x7f53b0158d60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 --2- 172.21.15.78:0/276873298 >> v2:172.21.15.150:3300/0 conn(0x7f53b00c2f50 0x7f53b00c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 -- 172.21.15.78:0/276873298 >> 172.21.15.78:0/276873298 conn(0x7f53b00aa300 msgr2=0x7f53b00aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 -- 172.21.15.78:0/276873298 shutdown_connections 2024-02-19T14:54:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.438+0000 7f53b838a640 1 -- 172.21.15.78:0/276873298 wait complete. 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.439+0000 7f53b838a640 1 Processor -- start 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.439+0000 7f53b838a640 1 -- start start 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b838a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53b00c2f50 0x7f53b0154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b838a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 0x7f53b0157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b838a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53b00711f0 0x7f53b0158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b838a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b014b760 con 0x7f53b00c8500 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b838a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b014be40 con 0x7f53b00c2f50 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b838a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0150460 con 0x7f53b00711f0 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 0x7f53b0157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 0x7f53b0157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38510/0 (socket says 172.21.15.78:38510) 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 -- 172.21.15.78:0/1593526401 learned_addr learned my addr 172.21.15.78:0/1593526401 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b6927640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53b00711f0 0x7f53b0158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b6126640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53b00c2f50 0x7f53b0154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 -- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53b00711f0 msgr2=0x7f53b0158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53b00711f0 0x7f53b0158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 -- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53b00c2f50 msgr2=0x7f53b0154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53b00c2f50 0x7f53b0154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 -- 172.21.15.78:0/1593526401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a00094d0 con 0x7f53b00c8500 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b6927640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53b00711f0 0x7f53b0158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53b5925640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 0x7f53b0157af0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a0004230 con 0x7f53b00c8500 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.440+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0004390 con 0x7f53b00c8500 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.441+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a003baa0 con 0x7f53b00c8500 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.441+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b01ee290 con 0x7f53b00c8500 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.441+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b01ee550 con 0x7f53b00c8500 2024-02-19T14:54:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.442+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b0071c20 con 0x7f53b00c8500 2024-02-19T14:54:14.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.442+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f53a0043070 con 0x7f53b00c8500 2024-02-19T14:54:14.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.442+0000 7f53a6ffd640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f539406b810 0x7f539406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.442+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f53a002d040 con 0x7f53b00c8500 2024-02-19T14:54:14.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.442+0000 7f53b6126640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f539406b810 0x7f539406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.443+0000 7f53b6126640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f539406b810 0x7f539406dcd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f53ac000f60 tx=0x7f53ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.446+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f53a0076e70 con 0x7f53b00c8500 2024-02-19T14:54:14.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.656+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f53b0074990 con 0x7f53b00c8500 2024-02-19T14:54:14.659 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:54:14.657+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:54:14.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.658+0000 7f53a6ffd640 1 -- 172.21.15.78:0/1593526401 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) v1 ==== 163+0+0 (secure 0 0 0) 0x7f53a007a530 con 0x7f53b00c8500 2024-02-19T14:54:14.661 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:54:14.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.660+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f539406b810 msgr2=0x7f539406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.660+0000 7f53b838a640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f539406b810 0x7f539406dcd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f53ac000f60 tx=0x7f53ac009040 comp rx=0 tx=0).stop 2024-02-19T14:54:14.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.660+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 msgr2=0x7f53b0157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.660+0000 7f53b838a640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 0x7f53b0157af0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a00046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.660+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 shutdown_connections 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.660+0000 7f53b838a640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f539406b810 0x7f539406dcd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.661+0000 7f53b838a640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53b00c8500 0x7f53b0157af0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.661+0000 7f53b838a640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53b00711f0 0x7f53b0158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.661+0000 7f53b838a640 1 --2- 172.21.15.78:0/1593526401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53b00c2f50 0x7f53b0154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.661+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 >> 172.21.15.78:0/1593526401 conn(0x7f53b00aa300 msgr2=0x7f53b00aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.661+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 shutdown_connections 2024-02-19T14:54:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.661+0000 7f53b838a640 1 -- 172.21.15.78:0/1593526401 wait complete. 2024-02-19T14:54:14.664 INFO:tasks.ceph.mds.b.smithi150.stderr: -67> 2024-02-19T14:54:12.945+0000 7f85fa383640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:14.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-02-19T14:54:14.687 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:54:14.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.774+0000 7f982d207640 1 Processor -- start 2024-02-19T14:54:14.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.774+0000 7f982d207640 1 -- start start 2024-02-19T14:54:14.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.774+0000 7f982d207640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98280ad890 0x7f98280c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.774+0000 7f982d207640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98280aba60 con 0x7f98280c49c0 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.774+0000 7f982d207640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9828152d30 con 0x7f98280ab600 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.774+0000 7f982d207640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98280c7860 con 0x7f98280ad890 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f9826d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98280ad890 0x7f98280c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f981ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f98280c49c0 0x7f98281534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38950/0 (socket says 172.21.15.78:38950) 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f9826d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98280ad890 0x7f98280c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38526/0 (socket says 172.21.15.78:38526) 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f981ffff640 1 -- 172.21.15.78:0/1274297479 learned_addr learned my addr 172.21.15.78:0/1274297479 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f9826d9d640 1 -- 172.21.15.78:0/1274297479 >> v1:172.21.15.150:6789/0 conn(0x7f98280ab600 legacy=0x7f98280c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f9826d9d640 1 -- 172.21.15.78:0/1274297479 >> v1:172.21.15.78:6789/0 conn(0x7f98280c49c0 legacy=0x7f98281534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f9826d9d640 1 -- 172.21.15.78:0/1274297479 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98280c7ce0 con 0x7f98280ad890 2024-02-19T14:54:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.775+0000 7f982659c640 1 -- 172.21.15.78:0/1274297479 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041897631 0 0) 0x7f98280aba60 con 0x7f98280c49c0 2024-02-19T14:54:14.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f9826d9d640 1 --2- 172.21.15.78:0/1274297479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280ad890 0x7f98280c64f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f98180097a0 tx=0x7f981802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebb8ce5fb532ce8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982659c640 1 -- 172.21.15.78:0/1274297479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981802f050 con 0x7f98280ad890 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982659c640 1 -- 172.21.15.78:0/1274297479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818031700 con 0x7f98280ad890 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982659c640 1 -- 172.21.15.78:0/1274297479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818031a90 con 0x7f98280ad890 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 -- 172.21.15.78:0/1274297479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280ad890 msgr2=0x7f98280c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 --2- 172.21.15.78:0/1274297479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280ad890 0x7f98280c64f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f98180097a0 tx=0x7f981802b7a0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 -- 172.21.15.78:0/1274297479 shutdown_connections 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 --2- 172.21.15.78:0/1274297479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280ad890 0x7f98280c64f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 -- 172.21.15.78:0/1274297479 >> 172.21.15.78:0/1274297479 conn(0x7f982806b700 msgr2=0x7f98280aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 -- 172.21.15.78:0/1274297479 shutdown_connections 2024-02-19T14:54:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.776+0000 7f982d207640 1 -- 172.21.15.78:0/1274297479 wait complete. 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 Processor -- start 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 -- start start 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98280ab600 0x7f98281edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98280ad890 0x7f98280c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 0x7f98280c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98281545a0 con 0x7f98280c49c0 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98281ee2d0 con 0x7f98280ad890 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982d207640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982806c130 con 0x7f98280ab600 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 0x7f98280c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 0x7f98280c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38528/0 (socket says 172.21.15.78:38528) 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f982759e640 1 -- 172.21.15.78:0/1987061520 learned_addr learned my addr 172.21.15.78:0/1987061520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f9826d9d640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98280ab600 0x7f98281edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.778+0000 7f981ffff640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98280ad890 0x7f98280c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982759e640 1 -- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98280ab600 msgr2=0x7f98281edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982759e640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98280ab600 0x7f98281edd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982759e640 1 -- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98280ad890 msgr2=0x7f98280c2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982759e640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98280ad890 0x7f98280c2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982759e640 1 -- 172.21.15.78:0/1987061520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818009400 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982759e640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 0x7f98280c1bf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f982000c7b0 tx=0x7f982000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820004b60 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9820004cc0 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000aa90 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98281541c0 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.779+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98280c2130 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.780+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982806e160 con 0x7f98280c49c0 2024-02-19T14:54:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.780+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f982000abf0 con 0x7f98280c49c0 2024-02-19T14:54:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.781+0000 7f981f7fe640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f980806b810 0x7f980806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.781+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f982008a520 con 0x7f98280c49c0 2024-02-19T14:54:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.781+0000 7f9826d9d640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f980806b810 0x7f980806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.781+0000 7f9826d9d640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f980806b810 0x7f980806dcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9818035000 tx=0x7f9818008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:14.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.785+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9820054d40 con 0x7f98280c49c0 2024-02-19T14:54:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:14.993+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9828074990 con 0x7f98280c49c0 2024-02-19T14:54:15.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.660+0000 7f981f7fe640 1 -- 172.21.15.78:0/1987061520 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9820058400 con 0x7f98280c49c0 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f980806b810 msgr2=0x7f980806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f980806b810 0x7f980806dcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9818035000 tx=0x7f9818008030 comp rx=0 tx=0).stop 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 msgr2=0x7f98280c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 0x7f98280c1bf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f982000c7b0 tx=0x7f982000cc70 comp rx=0 tx=0).stop 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 shutdown_connections 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f980806b810 0x7f980806dcd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98280c49c0 0x7f98280c1bf0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98280ad890 0x7f98280c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 --2- 172.21.15.78:0/1987061520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98280ab600 0x7f98281edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 >> 172.21.15.78:0/1987061520 conn(0x7f982806b700 msgr2=0x7f98280a9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:15.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 shutdown_connections 2024-02-19T14:54:15.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.662+0000 7f982d207640 1 -- 172.21.15.78:0/1987061520 wait complete. 2024-02-19T14:54:15.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-02-19T14:54:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.781+0000 7f7217aed640 1 Processor -- start 2024-02-19T14:54:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7217aed640 1 -- start start 2024-02-19T14:54:15.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7217aed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721006d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7217aed640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7210155010 0x7f72101554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7217aed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7210159210 con 0x7f7210152620 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7217aed640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100c1f00 con 0x7f7210155010 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7217aed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721006d980 con 0x7f721006d000 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f721608a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7210155010 0x7f72101554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f721608a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7210155010 0x7f72101554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38540/0 (socket says 172.21.15.78:38540) 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.782+0000 7f7215889640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7210152620 0x7f72100c6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38954/0 (socket says 172.21.15.78:38954) 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f721608a640 1 -- 172.21.15.78:0/1012501499 learned_addr learned my addr 172.21.15.78:0/1012501499 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f7215088640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f721608a640 1 -- 172.21.15.78:0/1012501499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 msgr2=0x7f721006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f721608a640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721006d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f721608a640 1 -- 172.21.15.78:0/1012501499 >> v1:172.21.15.78:6789/0 conn(0x7f7210152620 legacy=0x7f72100c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f721608a640 1 -- 172.21.15.78:0/1012501499 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100c6420 con 0x7f7210155010 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f7214887640 1 -- 172.21.15.78:0/1012501499 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029387740 0 0) 0x7f7210159210 con 0x7f7210152620 2024-02-19T14:54:15.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.783+0000 7f7215088640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721006d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f721608a640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210155010 0x7f72101554a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7208009e50 tx=0x7f720802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c6f7cfc6552645 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7214887640 1 -- 172.21.15.78:0/1012501499 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720802ca70 con 0x7f7210155010 2024-02-19T14:54:15.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7214887640 1 -- 172.21.15.78:0/1012501499 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720802cbd0 con 0x7f7210155010 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7214887640 1 -- 172.21.15.78:0/1012501499 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72080343f0 con 0x7f7210155010 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 -- 172.21.15.78:0/1012501499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210155010 msgr2=0x7f72101554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210155010 0x7f72101554a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7208009e50 tx=0x7f720802bed0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 -- 172.21.15.78:0/1012501499 shutdown_connections 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210155010 0x7f72101554a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 --2- 172.21.15.78:0/1012501499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721006d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 -- 172.21.15.78:0/1012501499 >> 172.21.15.78:0/1012501499 conn(0x7f72100caae0 msgr2=0x7f72100caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 -- 172.21.15.78:0/1012501499 shutdown_connections 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.784+0000 7f7217aed640 1 -- 172.21.15.78:0/1012501499 wait complete. 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.785+0000 7f7217aed640 1 Processor -- start 2024-02-19T14:54:15.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.785+0000 7f7217aed640 1 -- start start 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7217aed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721014c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7217aed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 0x7f7210157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7217aed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7210155010 0x7f7210157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7217aed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210158450 con 0x7f7210152620 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7217aed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210157d50 con 0x7f7210155010 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7217aed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c09a0 con 0x7f721006d000 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 0x7f7210157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 0x7f7210157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38550/0 (socket says 172.21.15.78:38550) 2024-02-19T14:54:15.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 -- 172.21.15.78:0/3477299577 learned_addr learned my addr 172.21.15.78:0/3477299577 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215889640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721014c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 -- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 msgr2=0x7f721014c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721014c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f721608a640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7210155010 0x7f7210157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 -- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7210155010 msgr2=0x7f7210157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7210155010 0x7f7210157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215088640 1 -- 172.21.15.78:0/3477299577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7208009b30 con 0x7f7210152620 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.786+0000 7f7215889640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721014c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.787+0000 7f7215088640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 0x7f7210157f10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f720000c820 tx=0x7f720000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.787+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200007a30 con 0x7f7210152620 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.787+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7200007b90 con 0x7f7210152620 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.787+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720000ce70 con 0x7f7210152620 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.787+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72100afbd0 con 0x7f7210152620 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.787+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100afe90 con 0x7f7210152620 2024-02-19T14:54:15.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.788+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72100b00b0 con 0x7f7210152620 2024-02-19T14:54:15.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.788+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f72000113f0 con 0x7f7210152620 2024-02-19T14:54:15.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.788+0000 7f72067fc640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71f406b810 0x7f71f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:15.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.789+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f720008a6c0 con 0x7f7210152620 2024-02-19T14:54:15.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.789+0000 7f7215889640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71f406b810 0x7f71f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:15.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.789+0000 7f7215889640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71f406b810 0x7f71f406dcd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f720c000f60 tx=0x7f720c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:15.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.793+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7200054da0 con 0x7f7210152620 2024-02-19T14:54:15.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:15.962+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7210074990 con 0x7f7210152620 2024-02-19T14:54:16.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.664+0000 7f72067fc640 1 -- 172.21.15.78:0/3477299577 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v65) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7200058460 con 0x7f7210152620 2024-02-19T14:54:16.667 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_metadata' removed 2024-02-19T14:54:16.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71f406b810 msgr2=0x7f71f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:16.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71f406b810 0x7f71f406dcd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f720c000f60 tx=0x7f720c009040 comp rx=0 tx=0).stop 2024-02-19T14:54:16.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 msgr2=0x7f7210157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:16.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 0x7f7210157f10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f720000c820 tx=0x7f720000cc20 comp rx=0 tx=0).stop 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 shutdown_connections 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71f406b810 0x7f71f406dcd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210152620 0x7f7210157f10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7210155010 0x7f7210157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 --2- 172.21.15.78:0/3477299577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f721006d000 0x7f721014c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 >> 172.21.15.78:0/3477299577 conn(0x7f72100caae0 msgr2=0x7f72100c1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.668+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 shutdown_connections 2024-02-19T14:54:16.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.669+0000 7f7217aed640 1 -- 172.21.15.78:0/3477299577 wait complete. 2024-02-19T14:54:16.684 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-02-19T14:54:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 Processor -- start 2024-02-19T14:54:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 -- start start 2024-02-19T14:54:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97200becb0 0x7f97200c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 0x7f9720152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9720070f10 con 0x7f97200c5360 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97200bf540 con 0x7f97200becb0 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.791+0000 7f9726239640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9720071420 con 0x7f9720157050 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 0x7f9720152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97200becb0 0x7f97200c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f97200c5360 0x7f97200c6600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:39792/0 (socket says 172.21.15.78:39792) 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971ffff640 1 -- 172.21.15.78:0/2954819398 learned_addr learned my addr 172.21.15.78:0/2954819398 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971effd640 1 --2- 172.21.15.78:0/2954819398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 0x7f9720152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33100/0 (socket says 172.21.15.78:33100) 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971effd640 1 -- 172.21.15.78:0/2954819398 >> v2:172.21.15.150:3300/0 conn(0x7f97200becb0 msgr2=0x7f97200c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971effd640 1 --2- 172.21.15.78:0/2954819398 >> v2:172.21.15.150:3300/0 conn(0x7f97200becb0 0x7f97200c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971effd640 1 -- 172.21.15.78:0/2954819398 >> v1:172.21.15.150:6789/0 conn(0x7f97200c5360 legacy=0x7f97200c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971effd640 1 -- 172.21.15.78:0/2954819398 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97200c5970 con 0x7f9720157050 2024-02-19T14:54:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.792+0000 7f971f7fe640 1 --2- 172.21.15.78:0/2954819398 >> v2:172.21.15.150:3300/0 conn(0x7f97200becb0 0x7f97200c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.793+0000 7f971effd640 1 --2- 172.21.15.78:0/2954819398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 0x7f9720152d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f97140097a0 tx=0x7f971402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96ade5538aa06abd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.793+0000 7f971e7fc640 1 -- 172.21.15.78:0/2954819398 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714031830 con 0x7f9720157050 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.793+0000 7f971e7fc640 1 -- 172.21.15.78:0/2954819398 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9714031990 con 0x7f9720157050 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.793+0000 7f971e7fc640 1 -- 172.21.15.78:0/2954819398 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714031d20 con 0x7f9720157050 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 -- 172.21.15.78:0/2954819398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 msgr2=0x7f9720152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 --2- 172.21.15.78:0/2954819398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 0x7f9720152d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f97140097a0 tx=0x7f971402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 -- 172.21.15.78:0/2954819398 shutdown_connections 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 --2- 172.21.15.78:0/2954819398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9720157050 0x7f9720152d70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 --2- 172.21.15.78:0/2954819398 >> v2:172.21.15.150:3300/0 conn(0x7f97200becb0 0x7f97200c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 -- 172.21.15.78:0/2954819398 >> 172.21.15.78:0/2954819398 conn(0x7f97200aa370 msgr2=0x7f97200aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 -- 172.21.15.78:0/2954819398 shutdown_connections 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.794+0000 7f9726239640 1 -- 172.21.15.78:0/2954819398 wait complete. 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.795+0000 7f9726239640 1 Processor -- start 2024-02-19T14:54:16.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.795+0000 7f9726239640 1 -- start start 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97200becb0 0x7f9720152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 0x7f972014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9720157050 0x7f972014c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97200c1a30 con 0x7f97200c5360 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97201e91a0 con 0x7f9720157050 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97200bfd00 con 0x7f97200becb0 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 0x7f972014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 0x7f972014be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38566/0 (socket says 172.21.15.78:38566) 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 -- 172.21.15.78:0/3562276937 learned_addr learned my addr 172.21.15.78:0/3562276937 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971f7fe640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97200becb0 0x7f9720152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971ffff640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9720157050 0x7f972014c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:16.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 -- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97200becb0 msgr2=0x7f9720152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97200becb0 0x7f9720152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 -- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9720157050 msgr2=0x7f972014c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9720157050 0x7f972014c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 -- 172.21.15.78:0/3562276937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9710009400 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971ffff640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9720157050 0x7f972014c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f971effd640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 0x7f972014be20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9714000c00 tx=0x7f97140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971403f860 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9714043070 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9714009400 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.796+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971403fc20 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.797+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97200c02c0 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.798+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97201eb930 con 0x7f97200c5360 2024-02-19T14:54:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.798+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9714052450 con 0x7f97200c5360 2024-02-19T14:54:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.798+0000 7f9707fff640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f970c06b840 0x7f970c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.798+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6095+0+0 (secure 0 0 0) 0x7f97140b03c0 con 0x7f97200c5360 2024-02-19T14:54:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.798+0000 7f971f7fe640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f970c06b840 0x7f970c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.799+0000 7f971f7fe640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f970c06b840 0x7f970c06dd00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9710009c40 tx=0x7f9710008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.803+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f971407abe0 con 0x7f97200c5360 2024-02-19T14:54:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:16.974+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9720074990 con 0x7f97200c5360 2024-02-19T14:54:17.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.666+0000 7f9707fff640 1 -- 172.21.15.78:0/3562276937 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v66) v1 ==== 172+0+0 (secure 0 0 0) 0x7f971407e2a0 con 0x7f97200c5360 2024-02-19T14:54:17.669 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_data' removed 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f970c06b840 msgr2=0x7f970c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f970c06b840 0x7f970c06dd00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9710009c40 tx=0x7f9710008040 comp rx=0 tx=0).stop 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 msgr2=0x7f972014be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 0x7f972014be20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9714000c00 tx=0x7f97140046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 shutdown_connections 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f970c06b840 0x7f970c06dd00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97200c5360 0x7f972014be20 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9720157050 0x7f972014c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 --2- 172.21.15.78:0/3562276937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97200becb0 0x7f9720152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.670+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 >> 172.21.15.78:0/3562276937 conn(0x7f97200aa370 msgr2=0x7f9720156160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.671+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 shutdown_connections 2024-02-19T14:54:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.671+0000 7f9726239640 1 -- 172.21.15.78:0/3562276937 wait complete. 2024-02-19T14:54:17.687 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-19T14:54:17.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.789+0000 7f77d631e640 1 Processor -- start 2024-02-19T14:54:17.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.789+0000 7f77d631e640 1 -- start start 2024-02-19T14:54:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d631e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77d00c50a0 0x7f77d00c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d631e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77d0071790 0x7f77d00c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d631e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d0071c20 con 0x7f77d00c5480 2024-02-19T14:54:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d631e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d00c7fe0 con 0x7f77d00c50a0 2024-02-19T14:54:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d631e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00c82c0 con 0x7f77d0071790 2024-02-19T14:54:17.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d48bb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f77d00c5480 0x7f77d006cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59832/0 (socket says 172.21.15.78:59832) 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77d48bb640 1 -- 172.21.15.78:0/1541847965 learned_addr learned my addr 172.21.15.78:0/1541847965 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77cffff640 1 --2- 172.21.15.78:0/1541847965 >> v2:172.21.15.150:3300/0 conn(0x7f77d00c50a0 0x7f77d00c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218503968 0 0) 0x7f77d0071c20 con 0x7f77d00c5480 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b8003520 con 0x7f77d00c5480 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77cf7fe640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77d0071790 0x7f77d00c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2213398526 0 0) 0x7f77b8003520 con 0x7f77d00c5480 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77d0071790 msgr2=0x7f77d00c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77d0071790 0x7f77d00c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 >> v2:172.21.15.150:3300/0 conn(0x7f77d00c50a0 msgr2=0x7f77d00c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 --2- 172.21.15.78:0/1541847965 >> v2:172.21.15.150:3300/0 conn(0x7f77d00c50a0 0x7f77d00c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d006c700 con 0x7f77d00c5480 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f77c80019f0 con 0x7f77d00c5480 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 >> v1:172.21.15.78:6789/0 conn(0x7f77d00c5480 legacy=0x7f77d006cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8003970 0x7f77b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b80047c0 0x7f77b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b8005180 0x7f77b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00c7fe0 con 0x7f77b80047c0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00c82c0 con 0x7f77b8003970 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.790+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b800bec0 con 0x7f77b8005180 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cf7fe640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b80047c0 0x7f77b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cffff640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8003970 0x7f77b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77d48bb640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b8005180 0x7f77b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cf7fe640 1 -- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b8005180 msgr2=0x7f77b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cf7fe640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b8005180 0x7f77b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cf7fe640 1 -- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8003970 msgr2=0x7f77b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cf7fe640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8003970 0x7f77b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.791+0000 7f77cf7fe640 1 -- 172.21.15.78:0/1541847965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b8003520 con 0x7f77b80047c0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77cf7fe640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b80047c0 0x7f77b8004c40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009ac0 tx=0x7f77b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a122f40d795774 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c402c6b0 con 0x7f77b80047c0 2024-02-19T14:54:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c402c810 con 0x7f77b80047c0 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77ceffd640 1 -- 172.21.15.78:0/1541847965 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c402cba0 con 0x7f77b80047c0 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d631e640 1 -- 172.21.15.78:0/1541847965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b80047c0 msgr2=0x7f77b8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d631e640 1 --2- 172.21.15.78:0/1541847965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b80047c0 0x7f77b8004c40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009ac0 tx=0x7f77b80057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d48bb640 1 -- 172.21.15.78:0/1541847965 reap_dead start 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d631e640 1 -- 172.21.15.78:0/1541847965 shutdown_connections 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d631e640 1 -- 172.21.15.78:0/1541847965 >> 172.21.15.78:0/1541847965 conn(0x7f77d006b700 msgr2=0x7f77d00aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d631e640 1 -- 172.21.15.78:0/1541847965 shutdown_connections 2024-02-19T14:54:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.792+0000 7f77d631e640 1 -- 172.21.15.78:0/1541847965 wait complete. 2024-02-19T14:54:17.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.793+0000 7f77d631e640 1 Processor -- start 2024-02-19T14:54:17.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 -- start start 2024-02-19T14:54:17.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b80047c0 0x7f77d01edb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77d00a8b30 0x7f77d00a8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 0x7f77d009b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d009a080 con 0x7f77d009b160 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00a9e10 con 0x7f77d00a8b30 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d631e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d0099da0 con 0x7f77b80047c0 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d48bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 0x7f77d009b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d48bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 0x7f77d009b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35078/0 (socket says 172.21.15.78:35078) 2024-02-19T14:54:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77d48bb640 1 -- 172.21.15.78:0/2066277352 learned_addr learned my addr 172.21.15.78:0/2066277352 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:17.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77cffff640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b80047c0 0x7f77d01edb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.794+0000 7f77cf7fe640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77d00a8b30 0x7f77d00a8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d48bb640 1 -- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b80047c0 msgr2=0x7f77d01edb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d48bb640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b80047c0 0x7f77d01edb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d48bb640 1 -- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77d00a8b30 msgr2=0x7f77d00a8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d48bb640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77d00a8b30 0x7f77d00a8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d48bb640 1 -- 172.21.15.78:0/2066277352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d01ee440 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d48bb640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 0x7f77d009b5e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f77c8001330 tx=0x7f77c8007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c800ea40 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c800eba0 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d0158310 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8013420 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.795+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d009d290 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.796+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d0074990 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.796+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f77c8013580 con 0x7f77d009b160 2024-02-19T14:54:17.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.797+0000 7f77ccff9640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77bc06b840 0x7f77bc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.797+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f77c808a2e0 con 0x7f77d009b160 2024-02-19T14:54:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.797+0000 7f77cffff640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77bc06b840 0x7f77bc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.797+0000 7f77cffff640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77bc06b840 0x7f77bc06dd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f77d01eb4c0 tx=0x7f77c0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:17.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.801+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f77c8054db0 con 0x7f77d009b160 2024-02-19T14:54:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.973+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f77d00c33b0 con 0x7f77d009b160 2024-02-19T14:54:17.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.976+0000 7f77ccff9640 1 -- 172.21.15.78:0/2066277352 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v19) v1 ==== 96+0+0 (secure 0 0 0) 0x7f77c8058470 con 0x7f77d009b160 2024-02-19T14:54:17.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77bc06b840 msgr2=0x7f77bc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77bc06b840 0x7f77bc06dd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f77d01eb4c0 tx=0x7f77c0009190 comp rx=0 tx=0).stop 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 msgr2=0x7f77d009b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 0x7f77d009b5e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f77c8001330 tx=0x7f77c8007b00 comp rx=0 tx=0).stop 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 shutdown_connections 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77bc06b840 0x7f77bc06dd00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d009b160 0x7f77d009b5e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77d00a8b30 0x7f77d00a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 --2- 172.21.15.78:0/2066277352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77b80047c0 0x7f77d01edb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 >> 172.21.15.78:0/2066277352 conn(0x7f77d006b700 msgr2=0x7f77d00c89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 shutdown_connections 2024-02-19T14:54:17.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:17.978+0000 7f77d631e640 1 -- 172.21.15.78:0/2066277352 wait complete. 2024-02-19T14:54:17.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-19T14:54:18.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.090+0000 7f16f4a07640 1 Processor -- start 2024-02-19T14:54:18.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.090+0000 7f16f4a07640 1 -- start start 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16f4a07640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16f00c2f70 0x7f16f006c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16f4a07640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16f0156050 0x7f16f006cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16f4a07640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00717b0 con 0x7f16f00c6a50 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16f4a07640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f00c7560 con 0x7f16f0156050 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16f4a07640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f0071a00 con 0x7f16f00c2f70 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16eed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16f0156050 0x7f16f006cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16ee59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f16f00c6a50 0x7f16f0155940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59848/0 (socket says 172.21.15.78:59848) 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16ee59c640 1 -- 172.21.15.78:0/3233316279 learned_addr learned my addr 172.21.15.78:0/3233316279 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16edd9b640 1 --2- 172.21.15.78:0/3233316279 >> v2:172.21.15.150:3300/0 conn(0x7f16f00c2f70 0x7f16f006c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16eed9d640 1 -- 172.21.15.78:0/3233316279 >> v2:172.21.15.150:3300/0 conn(0x7f16f00c2f70 msgr2=0x7f16f006c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:18.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16eed9d640 1 --2- 172.21.15.78:0/3233316279 >> v2:172.21.15.150:3300/0 conn(0x7f16f00c2f70 0x7f16f006c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16eed9d640 1 -- 172.21.15.78:0/3233316279 >> v1:172.21.15.78:6789/0 conn(0x7f16f00c6a50 legacy=0x7f16f0155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16eed9d640 1 -- 172.21.15.78:0/3233316279 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00c8030 con 0x7f16f0156050 2024-02-19T14:54:18.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.091+0000 7f16ed59a640 1 -- 172.21.15.78:0/3233316279 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443372121 0 0) 0x7f16f00717b0 con 0x7f16f00c6a50 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.092+0000 7f16eed9d640 1 --2- 172.21.15.78:0/3233316279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f0156050 0x7f16f006cf20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f16f00c2c80 tx=0x7f16e802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa595487141756f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.092+0000 7f16ed59a640 1 -- 172.21.15.78:0/3233316279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e802cbf0 con 0x7f16f0156050 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.092+0000 7f16ed59a640 1 -- 172.21.15.78:0/3233316279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e802cd50 con 0x7f16f0156050 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.092+0000 7f16ed59a640 1 -- 172.21.15.78:0/3233316279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8034660 con 0x7f16f0156050 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 -- 172.21.15.78:0/3233316279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f0156050 msgr2=0x7f16f006cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 --2- 172.21.15.78:0/3233316279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f0156050 0x7f16f006cf20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f16f00c2c80 tx=0x7f16e802be40 comp rx=0 tx=0).stop 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 -- 172.21.15.78:0/3233316279 shutdown_connections 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 --2- 172.21.15.78:0/3233316279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f0156050 0x7f16f006cf20 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 --2- 172.21.15.78:0/3233316279 >> v2:172.21.15.150:3300/0 conn(0x7f16f00c2f70 0x7f16f006c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 -- 172.21.15.78:0/3233316279 >> 172.21.15.78:0/3233316279 conn(0x7f16f006b700 msgr2=0x7f16f00aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:18.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 -- 172.21.15.78:0/3233316279 shutdown_connections 2024-02-19T14:54:18.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.093+0000 7f16f4a07640 1 -- 172.21.15.78:0/3233316279 wait complete. 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.094+0000 7f16f4a07640 1 Processor -- start 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.094+0000 7f16f4a07640 1 -- start start 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f00c2f70 0x7f16f00aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 0x7f16f00acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16f0156050 0x7f16f01eb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00a9290 con 0x7f16f00c6a50 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f01eb160 con 0x7f16f00c2f70 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00a9550 con 0x7f16f0156050 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 0x7f16f00acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 0x7f16f00acb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35096/0 (socket says 172.21.15.78:35096) 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 -- 172.21.15.78:0/2665092293 learned_addr learned my addr 172.21.15.78:0/2665092293 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16ee59c640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f00c2f70 0x7f16f00aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16eed9d640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16f0156050 0x7f16f01eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 -- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16f0156050 msgr2=0x7f16f01eb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16f0156050 0x7f16f01eb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 -- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f00c2f70 msgr2=0x7f16f00aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f00c2f70 0x7f16f00aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 -- 172.21.15.78:0/2665092293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e8009a70 con 0x7f16f00c6a50 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16edd9b640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 0x7f16f00acb90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f16e400d800 tx=0x7f16e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400b6c0 con 0x7f16f00c6a50 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400b820 con 0x7f16f00c6a50 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f00a9830 con 0x7f16f00c6a50 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400bbd0 con 0x7f16f00c6a50 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.095+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f00ad0d0 con 0x7f16f00c6a50 2024-02-19T14:54:18.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.096+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f0074990 con 0x7f16f00c6a50 2024-02-19T14:54:18.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.097+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f16e400bd30 con 0x7f16f00c6a50 2024-02-19T14:54:18.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.097+0000 7f16deffd640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16d006b810 0x7f16d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.097+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f16e408a730 con 0x7f16f00c6a50 2024-02-19T14:54:18.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.097+0000 7f16ee59c640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16d006b810 0x7f16d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.098+0000 7f16ee59c640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16d006b810 0x7f16d006dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000f60 tx=0x7f16e0007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.102+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f16e4055170 con 0x7f16f00c6a50 2024-02-19T14:54:18.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.271+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f16f00c65c0 con 0x7f16f00c6a50 2024-02-19T14:54:18.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.275+0000 7f16deffd640 1 -- 172.21.15.78:0/2665092293 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v20) v1 ==== 96+0+0 (secure 0 0 0) 0x7f16e4058830 con 0x7f16f00c6a50 2024-02-19T14:54:18.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16d006b810 msgr2=0x7f16d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16d006b810 0x7f16d006dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000f60 tx=0x7f16e0007b50 comp rx=0 tx=0).stop 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 msgr2=0x7f16f00acb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 0x7f16f00acb90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f16e400d800 tx=0x7f16e400dcc0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 shutdown_connections 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16d006b810 0x7f16d006dcd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f00c6a50 0x7f16f00acb90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16f0156050 0x7f16f01eb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 --2- 172.21.15.78:0/2665092293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f00c2f70 0x7f16f00aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 >> 172.21.15.78:0/2665092293 conn(0x7f16f006b700 msgr2=0x7f16f006fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 shutdown_connections 2024-02-19T14:54:18.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.277+0000 7f16f4a07640 1 -- 172.21.15.78:0/2665092293 wait complete. 2024-02-19T14:54:18.290 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:54:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.389+0000 7f8a3b78c640 1 Processor -- start 2024-02-19T14:54:18.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.389+0000 7f8a3b78c640 1 -- start start 2024-02-19T14:54:18.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a3b78c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8a340c8e00 0x7f8a3406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a3b78c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8a3406d910 0x7f8a3406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a3b78c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3406cd20 con 0x7f8a34152700 2024-02-19T14:54:18.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a3b78c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a341551f0 con 0x7f8a3406d910 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a3b78c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a34155490 con 0x7f8a340c8e00 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a39d29640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8a3406d910 0x7f8a3406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a39d29640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8a3406d910 0x7f8a3406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35102/0 (socket says 172.21.15.78:35102) 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.390+0000 7f8a39d29640 1 -- 172.21.15.78:0/1084806916 learned_addr learned my addr 172.21.15.78:0/1084806916 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.391+0000 7f8a38d27640 1 --2- 172.21.15.78:0/1084806916 >> v2:172.21.15.150:3300/0 conn(0x7f8a340c8e00 0x7f8a3406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.391+0000 7f8a39d29640 1 -- 172.21.15.78:0/1084806916 >> v2:172.21.15.150:3300/0 conn(0x7f8a340c8e00 msgr2=0x7f8a3406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.391+0000 7f8a39d29640 1 --2- 172.21.15.78:0/1084806916 >> v2:172.21.15.150:3300/0 conn(0x7f8a340c8e00 0x7f8a3406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.391+0000 7f8a39d29640 1 -- 172.21.15.78:0/1084806916 >> v1:172.21.15.150:6789/0 conn(0x7f8a34152700 legacy=0x7f8a340c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:18.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.391+0000 7f8a39d29640 1 -- 172.21.15.78:0/1084806916 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34155770 con 0x7f8a3406d910 2024-02-19T14:54:18.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.391+0000 7f8a39d29640 1 --2- 172.21.15.78:0/1084806916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a3406d910 0x7f8a3406c7e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8a340c7620 tx=0x7f8a2c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a527080e366a34a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a2bfff640 1 -- 172.21.15.78:0/1084806916 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c031870 con 0x7f8a3406d910 2024-02-19T14:54:18.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a2bfff640 1 -- 172.21.15.78:0/1084806916 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c0319d0 con 0x7f8a3406d910 2024-02-19T14:54:18.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a2bfff640 1 -- 172.21.15.78:0/1084806916 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c031d60 con 0x7f8a3406d910 2024-02-19T14:54:18.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 -- 172.21.15.78:0/1084806916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a3406d910 msgr2=0x7f8a3406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/1084806916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a3406d910 0x7f8a3406c7e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8a340c7620 tx=0x7f8a2c02b910 comp rx=0 tx=0).stop 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 -- 172.21.15.78:0/1084806916 shutdown_connections 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/1084806916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a3406d910 0x7f8a3406c7e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/1084806916 >> v2:172.21.15.150:3300/0 conn(0x7f8a340c8e00 0x7f8a3406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 -- 172.21.15.78:0/1084806916 >> 172.21.15.78:0/1084806916 conn(0x7f8a340cac90 msgr2=0x7f8a340cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 -- 172.21.15.78:0/1084806916 shutdown_connections 2024-02-19T14:54:18.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.392+0000 7f8a3b78c640 1 -- 172.21.15.78:0/1084806916 wait complete. 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.393+0000 7f8a3b78c640 1 Processor -- start 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.393+0000 7f8a3b78c640 1 -- start start 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a3b78c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a3406d910 0x7f8a340c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a3b78c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a340c8e00 0x7f8a341ea860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a3b78c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 0x7f8a340ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a3b78c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340aac90 con 0x7f8a34152700 2024-02-19T14:54:18.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a3b78c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a341eb2e0 con 0x7f8a340c8e00 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a3b78c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340acd10 con 0x7f8a3406d910 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 0x7f8a340ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 0x7f8a340ab190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35118/0 (socket says 172.21.15.78:35118) 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 -- 172.21.15.78:0/2183484888 learned_addr learned my addr 172.21.15.78:0/2183484888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39528640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a3406d910 0x7f8a340c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a38d27640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a340c8e00 0x7f8a341ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 -- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a3406d910 msgr2=0x7f8a340c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a3406d910 0x7f8a340c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 -- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a340c8e00 msgr2=0x7f8a341ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a340c8e00 0x7f8a341ea860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 -- 172.21.15.78:0/2183484888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c0094d0 con 0x7f8a34152700 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39528640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a3406d910 0x7f8a340c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.394+0000 7f8a39d29640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 0x7f8a340ab190 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c02b950 tx=0x7f8a2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.395+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0443f0 con 0x7f8a34152700 2024-02-19T14:54:18.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.395+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c044570 con 0x7f8a34152700 2024-02-19T14:54:18.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.395+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a340acff0 con 0x7f8a34152700 2024-02-19T14:54:18.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.395+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c044920 con 0x7f8a34152700 2024-02-19T14:54:18.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.395+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a340c2780 con 0x7f8a34152700 2024-02-19T14:54:18.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.396+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a340aae60 con 0x7f8a34152700 2024-02-19T14:54:18.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.396+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8a2c044a80 con 0x7f8a34152700 2024-02-19T14:54:18.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.397+0000 7f8a29ffb640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a1806b840 0x7f8a1806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.397+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f8a2c02d040 con 0x7f8a34152700 2024-02-19T14:54:18.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.397+0000 7f8a39528640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a1806b840 0x7f8a1806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.397+0000 7f8a39528640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a1806b840 0x7f8a1806dd00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a30007e90 tx=0x7f8a300073a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.401+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8a2c0773f0 con 0x7f8a34152700 2024-02-19T14:54:18.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.573+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f8a34074990 con 0x7f8a34152700 2024-02-19T14:54:18.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.577+0000 7f8a29ffb640 1 -- 172.21.15.78:0/2183484888 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v21) v1 ==== 96+0+0 (secure 0 0 0) 0x7f8a2c07aab0 con 0x7f8a34152700 2024-02-19T14:54:18.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a1806b840 msgr2=0x7f8a1806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a1806b840 0x7f8a1806dd00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8a30007e90 tx=0x7f8a300073a0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 msgr2=0x7f8a340ab190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 0x7f8a340ab190 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c02b950 tx=0x7f8a2c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 shutdown_connections 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a1806b840 0x7f8a1806dd00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a34152700 0x7f8a340ab190 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a340c8e00 0x7f8a341ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 --2- 172.21.15.78:0/2183484888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a3406d910 0x7f8a340c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 >> 172.21.15.78:0/2183484888 conn(0x7f8a340cac90 msgr2=0x7f8a340cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 shutdown_connections 2024-02-19T14:54:18.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.579+0000 7f8a3b78c640 1 -- 172.21.15.78:0/2183484888 wait complete. 2024-02-19T14:54:18.592 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:54:18.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.692+0000 7f9ffb872640 1 Processor -- start 2024-02-19T14:54:18.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.692+0000 7f9ffb872640 1 -- start start 2024-02-19T14:54:18.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ffb872640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9ff40c6d60 0x7f9ff4154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ffb872640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff414f730 con 0x7f9ff406ff80 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ffb872640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40c16d0 con 0x7f9ff406e830 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ffb872640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff40c1940 con 0x7f9ff40c6d60 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ff9e0f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9ff406ff80 0x7f9ff40c7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59850/0 (socket says 172.21.15.78:59850) 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ff9e0f640 1 -- 172.21.15.78:0/1890685495 learned_addr learned my addr 172.21.15.78:0/1890685495 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.693+0000 7f9ff960e640 1 --2- 172.21.15.78:0/1890685495 >> v2:172.21.15.150:3300/0 conn(0x7f9ff40c6d60 0x7f9ff4154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644492467 0 0) 0x7f9ff414f730 con 0x7f9ff406ff80 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe0003520 con 0x7f9ff406ff80 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2918288579 0 0) 0x7f9fe0003520 con 0x7f9ff406ff80 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 >> v2:172.21.15.150:3300/0 conn(0x7f9ff40c6d60 msgr2=0x7f9ff4154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 --2- 172.21.15.78:0/1890685495 >> v2:172.21.15.150:3300/0 conn(0x7f9ff40c6d60 0x7f9ff4154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 >> v1:172.21.15.150:6789/0 conn(0x7f9ff406e830 legacy=0x7f9ff40c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4072f70 con 0x7f9ff406ff80 2024-02-19T14:54:18.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9fec001670 con 0x7f9ff406ff80 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 >> v1:172.21.15.78:6789/0 conn(0x7f9ff406ff80 legacy=0x7f9ff40c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fe0003970 0x7f9fe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe00047c0 0x7f9fe0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0005180 0x7f9fe000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40c1940 con 0x7f9fe0003970 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe000c020 con 0x7f9fe00047c0 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe000c1e0 con 0x7f9fe0005180 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff960e640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fe0003970 0x7f9fe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff8e0d640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe00047c0 0x7f9fe0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff9e0f640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0005180 0x7f9fe000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff960e640 1 -- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0005180 msgr2=0x7f9fe000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff960e640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0005180 0x7f9fe000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff960e640 1 -- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe00047c0 msgr2=0x7f9fe0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff960e640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe00047c0 0x7f9fe0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff960e640 1 -- 172.21.15.78:0/1890685495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0003520 con 0x7f9fe0003970 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff9e0f640 1 -- 172.21.15.78:0/1890685495 reap_dead start 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.694+0000 7f9ff8e0d640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fe00047c0 0x7f9fe0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.695+0000 7f9ff960e640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fe0003970 0x7f9fe0003d90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009fc0 tx=0x7f9fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a21476cf52cbf25 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.695+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff002cc20 con 0x7f9fe0003970 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.695+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff000f040 con 0x7f9fe0003970 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.695+0000 7f9febfff640 1 -- 172.21.15.78:0/1890685495 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0034440 con 0x7f9fe0003970 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 -- 172.21.15.78:0/1890685495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fe0003970 msgr2=0x7f9fe0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fe0003970 0x7f9fe0003d90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009fc0 tx=0x7f9fe00057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 -- 172.21.15.78:0/1890685495 shutdown_connections 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 --2- 172.21.15.78:0/1890685495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fe0003970 0x7f9fe0003d90 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 -- 172.21.15.78:0/1890685495 >> 172.21.15.78:0/1890685495 conn(0x7f9ff406c170 msgr2=0x7f9ff406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 -- 172.21.15.78:0/1890685495 shutdown_connections 2024-02-19T14:54:18.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.696+0000 7f9ffb872640 1 -- 172.21.15.78:0/1890685495 wait complete. 2024-02-19T14:54:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 Processor -- start 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 -- start start 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0003970 0x7f9ff4158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff4158f80 0x7f9ff4158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 0x7f9ff40c3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4156890 con 0x7f9ff40c30c0 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4158bb0 con 0x7f9ff4158f80 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.697+0000 7f9ffb872640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff41eb7d0 con 0x7f9fe0003970 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 0x7f9ff40c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 0x7f9ff40c3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35140/0 (socket says 172.21.15.78:35140) 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 -- 172.21.15.78:0/2855538279 learned_addr learned my addr 172.21.15.78:0/2855538279 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff960e640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0003970 0x7f9ff4158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff8e0d640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff4158f80 0x7f9ff4158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 -- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0003970 msgr2=0x7f9ff4158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0003970 0x7f9ff4158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 -- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff4158f80 msgr2=0x7f9ff4158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff4158f80 0x7f9ff4158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 -- 172.21.15.78:0/2855538279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff41562d0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff8e0d640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff4158f80 0x7f9ff4158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff960e640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0003970 0x7f9ff4158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ff9e0f640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 0x7f9ff40c3540 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40aa3b0 tx=0x7f9fec00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00eed0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec00c630 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00c9c0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff41565b0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.698+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff41ebab0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.699+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff4074990 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.700+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9fec00cb20 con 0x7f9ff40c30c0 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.700+0000 7f9fe9ffb640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:18.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.700+0000 7f9ff960e640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.700+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f9fec089cd0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.700+0000 7f9ff960e640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0036000 tx=0x7f9ff002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.704+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9fec0547a0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.879+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9ff40c09c0 con 0x7f9ff40c30c0 2024-02-19T14:54:18.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.882+0000 7f9fe9ffb640 1 -- 172.21.15.78:0/2855538279 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9fec057e60 con 0x7f9ff40c30c0 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 msgr2=0x7f9fd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0036000 tx=0x7f9ff002e000 comp rx=0 tx=0).stop 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 msgr2=0x7f9ff40c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 0x7f9ff40c3540 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40aa3b0 tx=0x7f9fec00eb90 comp rx=0 tx=0).stop 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 shutdown_connections 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff40c30c0 0x7f9ff40c3540 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff4158f80 0x7f9ff4158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 --2- 172.21.15.78:0/2855538279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fe0003970 0x7f9ff4158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 >> 172.21.15.78:0/2855538279 conn(0x7f9ff406c170 msgr2=0x7f9ff4072230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 shutdown_connections 2024-02-19T14:54:18.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.885+0000 7f9ffb872640 1 -- 172.21.15.78:0/2855538279 wait complete. 2024-02-19T14:54:18.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-02-19T14:54:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.997+0000 7f14e55e4640 1 Processor -- start 2024-02-19T14:54:19.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.997+0000 7f14e55e4640 1 -- start start 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14e55e4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14e006ff80 0x7f14e0154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14e55e4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e014f730 con 0x7f14e006e830 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14e55e4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e00c16d0 con 0x7f14e00c6d60 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14e55e4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e00c1940 con 0x7f14e006ff80 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14df7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14e006ff80 0x7f14e0154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14d7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f14e006e830 0x7f14e00c7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59854/0 (socket says 172.21.15.78:59854) 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14d7fff640 1 -- 172.21.15.78:0/2868008298 learned_addr learned my addr 172.21.15.78:0/2868008298 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14df7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14e006ff80 0x7f14e0154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35154/0 (socket says 172.21.15.78:35154) 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14df7fe640 1 -- 172.21.15.78:0/2868008298 >> v1:172.21.15.150:6789/0 conn(0x7f14e00c6d60 legacy=0x7f14e00c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14df7fe640 1 -- 172.21.15.78:0/2868008298 >> v1:172.21.15.78:6789/0 conn(0x7f14e006e830 legacy=0x7f14e00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14df7fe640 1 -- 172.21.15.78:0/2868008298 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e0072f70 con 0x7f14e006ff80 2024-02-19T14:54:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.998+0000 7f14de7fc640 1 -- 172.21.15.78:0/2868008298 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158886828 0 0) 0x7f14e014f730 con 0x7f14e006e830 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.999+0000 7f14df7fe640 1 --2- 172.21.15.78:0/2868008298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006ff80 0x7f14e0154f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009d90 tx=0x7f14d802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3f7911794ebd4d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.999+0000 7f14de7fc640 1 -- 172.21.15.78:0/2868008298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d802cc60 con 0x7f14e006ff80 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.999+0000 7f14de7fc640 1 -- 172.21.15.78:0/2868008298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d802cdc0 con 0x7f14e006ff80 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:18.999+0000 7f14de7fc640 1 -- 172.21.15.78:0/2868008298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8034660 con 0x7f14e006ff80 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 -- 172.21.15.78:0/2868008298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006ff80 msgr2=0x7f14e0154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2868008298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006ff80 0x7f14e0154f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009d90 tx=0x7f14d802beb0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 -- 172.21.15.78:0/2868008298 shutdown_connections 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2868008298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006ff80 0x7f14e0154f70 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 -- 172.21.15.78:0/2868008298 >> 172.21.15.78:0/2868008298 conn(0x7f14e006c170 msgr2=0x7f14e006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 -- 172.21.15.78:0/2868008298 shutdown_connections 2024-02-19T14:54:19.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.000+0000 7f14e55e4640 1 -- 172.21.15.78:0/2868008298 wait complete. 2024-02-19T14:54:19.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 Processor -- start 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 -- start start 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 0x7f14e00c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e006ff80 0x7f14e01eab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14e00c6d60 0x7f14e01522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00c38d0 con 0x7f14e006e830 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e01eb060 con 0x7f14e006ff80 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14e55e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0150670 con 0x7f14e00c6d60 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.001+0000 7f14deffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 0x7f14e00c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 0x7f14e00c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35170/0 (socket says 172.21.15.78:35170) 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 -- 172.21.15.78:0/2581996172 learned_addr learned my addr 172.21.15.78:0/2581996172 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 -- 172.21.15.78:0/2581996172 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14e00c6d60 msgr2=0x7f14e01522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14e00c6d60 0x7f14e01522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 -- 172.21.15.78:0/2581996172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e006ff80 msgr2=0x7f14e01eab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e006ff80 0x7f14e01eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 -- 172.21.15.78:0/2581996172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8009a70 con 0x7f14e006e830 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14deffd640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 0x7f14e00c3390 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f14d000c430 tx=0x7f14d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0011790 con 0x7f14e006e830 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d00118f0 con 0x7f14e006e830 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e014ff00 con 0x7f14e006e830 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0011ce0 con 0x7f14e006e830 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.002+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e014fc20 con 0x7f14e006e830 2024-02-19T14:54:19.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.003+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e0074990 con 0x7f14e006e830 2024-02-19T14:54:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.004+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f14d001c460 con 0x7f14e006e830 2024-02-19T14:54:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.004+0000 7f14d77fe640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14c006b810 0x7f14c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.004+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f14d008a110 con 0x7f14e006e830 2024-02-19T14:54:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.004+0000 7f14d7fff640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14c006b810 0x7f14c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.004+0000 7f14d7fff640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14c006b810 0x7f14c006dcd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f14c8007fd0 tx=0x7f14c8007960 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:19.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.008+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f14d0054be0 con 0x7f14e006e830 2024-02-19T14:54:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.254+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7f14e00751b0 con 0x7f14e006e830 2024-02-19T14:54:19.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.677+0000 7f14d77fe640 1 -- 172.21.15.78:0/2581996172 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v154) v1 ==== 150+0+0 (secure 0 0 0) 0x7f14d00582a0 con 0x7f14e006e830 2024-02-19T14:54:19.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14c006b810 msgr2=0x7f14c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14c006b810 0x7f14c006dcd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f14c8007fd0 tx=0x7f14c8007960 comp rx=0 tx=0).stop 2024-02-19T14:54:19.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 msgr2=0x7f14e00c3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 0x7f14e00c3390 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f14d000c430 tx=0x7f14d000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 shutdown_connections 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14c006b810 0x7f14c006dcd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14e00c6d60 0x7f14e01522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e006ff80 0x7f14e01eab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 --2- 172.21.15.78:0/2581996172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14e006e830 0x7f14e00c3390 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 >> 172.21.15.78:0/2581996172 conn(0x7f14e006c170 msgr2=0x7f14e00a75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 shutdown_connections 2024-02-19T14:54:19.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.679+0000 7f14e55e4640 1 -- 172.21.15.78:0/2581996172 wait complete. 2024-02-19T14:54:19.694 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is still used if necessary. ... ok 2024-02-19T14:54:19.694 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. 2024-02-19T14:54:19.694 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:54:19.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:19.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.794+0000 7ff0c704e640 1 Processor -- start 2024-02-19T14:54:19.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.794+0000 7ff0c704e640 1 -- start start 2024-02-19T14:54:19.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c704e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff0c00ad890 0x7ff0c00c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c704e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0c00c49c0 0x7ff0c00ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c704e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c0152d30 con 0x7ff0c01534d0 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c704e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c00c7860 con 0x7ff0c00ad890 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c704e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c00c7ce0 con 0x7ff0c00c49c0 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff0c00ad890 0x7ff0c00c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff0c00ad890 0x7ff0c00c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35184/0 (socket says 172.21.15.78:35184) 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 -- 172.21.15.78:0/596776265 learned_addr learned my addr 172.21.15.78:0/596776265 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c55eb640 1 --1- 172.21.15.78:0/596776265 >> v1:172.21.15.78:6789/0 conn(0x7ff0c01534d0 0x7ff0c00c5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59866/0 (socket says 172.21.15.78:59866) 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0bffff640 1 --2- 172.21.15.78:0/596776265 >> v2:172.21.15.150:3300/0 conn(0x7ff0c00c49c0 0x7ff0c00ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 -- 172.21.15.78:0/596776265 >> v2:172.21.15.150:3300/0 conn(0x7ff0c00c49c0 msgr2=0x7ff0c00ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 --2- 172.21.15.78:0/596776265 >> v2:172.21.15.150:3300/0 conn(0x7ff0c00c49c0 0x7ff0c00ab600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 -- 172.21.15.78:0/596776265 >> v1:172.21.15.78:6789/0 conn(0x7ff0c01534d0 legacy=0x7ff0c00c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0bf7fe640 1 -- 172.21.15.78:0/596776265 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137116023 0 0) 0x7ff0c0152d30 con 0x7ff0c01534d0 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0c4dea640 1 -- 172.21.15.78:0/596776265 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00c7f80 con 0x7ff0c00ad890 2024-02-19T14:54:19.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.795+0000 7ff0bffff640 1 --2- 172.21.15.78:0/596776265 >> v2:172.21.15.150:3300/0 conn(0x7ff0c00c49c0 0x7ff0c00ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.796+0000 7ff0c4dea640 1 --2- 172.21.15.78:0/596776265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c00ad890 0x7ff0c00c64f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00aaf00 tx=0x7ff0b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bdde0d5d857b7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.796+0000 7ff0bf7fe640 1 -- 172.21.15.78:0/596776265 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40317e0 con 0x7ff0c00ad890 2024-02-19T14:54:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.796+0000 7ff0bf7fe640 1 -- 172.21.15.78:0/596776265 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b4031940 con 0x7ff0c00ad890 2024-02-19T14:54:19.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.796+0000 7ff0bf7fe640 1 -- 172.21.15.78:0/596776265 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4031cd0 con 0x7ff0c00ad890 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 -- 172.21.15.78:0/596776265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c00ad890 msgr2=0x7ff0c00c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 --2- 172.21.15.78:0/596776265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c00ad890 0x7ff0c00c64f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00aaf00 tx=0x7ff0b402b880 comp rx=0 tx=0).stop 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 -- 172.21.15.78:0/596776265 shutdown_connections 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 --2- 172.21.15.78:0/596776265 >> v2:172.21.15.150:3300/0 conn(0x7ff0c00c49c0 0x7ff0c00ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 --2- 172.21.15.78:0/596776265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c00ad890 0x7ff0c00c64f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 -- 172.21.15.78:0/596776265 >> 172.21.15.78:0/596776265 conn(0x7ff0c006b700 msgr2=0x7ff0c00aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 -- 172.21.15.78:0/596776265 shutdown_connections 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.797+0000 7ff0c704e640 1 -- 172.21.15.78:0/596776265 wait complete. 2024-02-19T14:54:19.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.798+0000 7ff0c704e640 1 Processor -- start 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.798+0000 7ff0c704e640 1 -- start start 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c00ad890 0x7ff0c0150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c00c49c0 0x7ff0c01ea360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 0x7ff0c01ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0154dd0 con 0x7ff0c01534d0 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01554f0 con 0x7ff0c00c49c0 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01557d0 con 0x7ff0c00ad890 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 0x7ff0c01ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 0x7ff0c01ea8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35198/0 (socket says 172.21.15.78:35198) 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 -- 172.21.15.78:0/939782470 learned_addr learned my addr 172.21.15.78:0/939782470 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 -- 172.21.15.78:0/939782470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c00ad890 msgr2=0x7ff0c0150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c00ad890 0x7ff0c0150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 -- 172.21.15.78:0/939782470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c00c49c0 msgr2=0x7ff0c01ea360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c00c49c0 0x7ff0c01ea360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 -- 172.21.15.78:0/939782470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b40094d0 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c55eb640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 0x7ff0c01ea8a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800c700 tx=0x7ff0b800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b800ce10 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b800a640 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b800aa70 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c0155ab0 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.799+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c0151350 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.800+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c0155fd0 con 0x7ff0c01534d0 2024-02-19T14:54:19.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.801+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff0b800abd0 con 0x7ff0c01534d0 2024-02-19T14:54:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.801+0000 7ff0bd7fa640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0a406b810 0x7ff0a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.801+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7ff0b80898b0 con 0x7ff0c01534d0 2024-02-19T14:54:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.801+0000 7ff0c4dea640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0a406b810 0x7ff0a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:19.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.802+0000 7ff0c4dea640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0a406b810 0x7ff0a406dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4036000 tx=0x7ff0b4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.806+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff0b8054380 con 0x7ff0c01534d0 2024-02-19T14:54:19.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.977+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0c0074990 con 0x7ff0c01534d0 2024-02-19T14:54:19.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.978+0000 7ff0bd7fa640 1 -- 172.21.15.78:0/939782470 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13804 (secure 0 0 0) 0x7ff0b8018020 con 0x7ff0c01534d0 2024-02-19T14:54:19.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:19.981 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":66,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:17.665776+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/2194753931":"2024-02-20T14:54:14.658138+0000","172.21.15.150:6833/2194753931":"2024-02-20T14:54:14.658138+0000","172.21.15.78:6836/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6837/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.78:6834/3678473057":"2024-02-20T14:54:09.503426+0000","172.21.15.78:6835/1226290552":"2024-02-20T14:54:08.498684+0000","172.21.15.150:6832/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6837/1917648829":"2024-02-20T14:53:57.265990+0000","172.21.15.150:6834/210065453":"2024-02-20T14:54:00.278764+0000","172.21.15.78:6835/1917648829":"2024-02-20T14:53:57.265990+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0a406b810 msgr2=0x7ff0a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0a406b810 0x7ff0a406dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4036000 tx=0x7ff0b4030000 comp rx=0 tx=0).stop 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 msgr2=0x7ff0c01ea8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 0x7ff0c01ea8a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800c700 tx=0x7ff0b800cbc0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 shutdown_connections 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff0a406b810 0x7ff0a406dcd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0c01534d0 0x7ff0c01ea8a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c00c49c0 0x7ff0c01ea360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 --2- 172.21.15.78:0/939782470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c00ad890 0x7ff0c0150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 >> 172.21.15.78:0/939782470 conn(0x7ff0c006b700 msgr2=0x7ff0c0152750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 shutdown_connections 2024-02-19T14:54:19.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:19.980+0000 7ff0c704e640 1 -- 172.21.15.78:0/939782470 wait complete. 2024-02-19T14:54:19.994 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:19.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.094+0000 7f706990d640 1 Processor -- start 2024-02-19T14:54:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.094+0000 7f706990d640 1 -- start start 2024-02-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f706990d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f706414ef50 0x7f706414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f706990d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70640727d0 0x7f7064072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f706990d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70640c35c0 con 0x7f70640c78a0 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f706990d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70640c4bc0 con 0x7f706414ef50 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f706990d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706414f920 con 0x7f70640727d0 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f706414ef50 0x7f706414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f706414ef50 0x7f706414f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35206/0 (socket says 172.21.15.78:35206) 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 -- 172.21.15.78:0/1812521107 learned_addr learned my addr 172.21.15.78:0/1812521107 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70637fe640 1 --2- 172.21.15.78:0/1812521107 >> v2:172.21.15.150:3300/0 conn(0x7f70640727d0 0x7f7064072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 -- 172.21.15.78:0/1812521107 >> v2:172.21.15.150:3300/0 conn(0x7f70640727d0 msgr2=0x7f7064072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 --2- 172.21.15.78:0/1812521107 >> v2:172.21.15.150:3300/0 conn(0x7f70640727d0 0x7f7064072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 -- 172.21.15.78:0/1812521107 >> v1:172.21.15.150:6789/0 conn(0x7f70640c78a0 legacy=0x7f70640c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.095+0000 7f70627fc640 1 -- 172.21.15.78:0/1812521107 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70640c38a0 con 0x7f706414ef50 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.096+0000 7f70627fc640 1 --2- 172.21.15.78:0/1812521107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f706414ef50 0x7f706414f3e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f70640a8ea0 tx=0x7f705802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948c3f9f351b6686 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.096+0000 7f7061ffb640 1 -- 172.21.15.78:0/1812521107 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058031870 con 0x7f706414ef50 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.096+0000 7f7061ffb640 1 -- 172.21.15.78:0/1812521107 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70580319d0 con 0x7f706414ef50 2024-02-19T14:54:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.096+0000 7f7061ffb640 1 -- 172.21.15.78:0/1812521107 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058031d60 con 0x7f706414ef50 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 -- 172.21.15.78:0/1812521107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f706414ef50 msgr2=0x7f706414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 --2- 172.21.15.78:0/1812521107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f706414ef50 0x7f706414f3e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f70640a8ea0 tx=0x7f705802b910 comp rx=0 tx=0).stop 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 -- 172.21.15.78:0/1812521107 shutdown_connections 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 --2- 172.21.15.78:0/1812521107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f706414ef50 0x7f706414f3e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 --2- 172.21.15.78:0/1812521107 >> v2:172.21.15.150:3300/0 conn(0x7f70640727d0 0x7f7064072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 -- 172.21.15.78:0/1812521107 >> 172.21.15.78:0/1812521107 conn(0x7f7064066e70 msgr2=0x7f70640a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 -- 172.21.15.78:0/1812521107 shutdown_connections 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.097+0000 7f706990d640 1 -- 172.21.15.78:0/1812521107 wait complete. 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.098+0000 7f706990d640 1 Processor -- start 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.098+0000 7f706990d640 1 -- start start 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.098+0000 7f706990d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 0x7f70640c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70640c78a0 0x7f706414d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f706414ef50 0x7f706414da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064151ab0 con 0x7f70640727d0 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7064150f40 con 0x7f70640c78a0 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70640ac5d0 con 0x7f706414ef50 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 0x7f70640c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 0x7f70640c0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35214/0 (socket says 172.21.15.78:35214) 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 -- 172.21.15.78:0/2661989373 learned_addr learned my addr 172.21.15.78:0/2661989373 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 -- 172.21.15.78:0/2661989373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f706414ef50 msgr2=0x7f706414da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f706414ef50 0x7f706414da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 -- 172.21.15.78:0/2661989373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70640c78a0 msgr2=0x7f706414d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70640c78a0 0x7f706414d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 -- 172.21.15.78:0/2661989373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580094d0 con 0x7f70640727d0 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f7062ffd640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 0x7f70640c0d40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f705400c7f0 tx=0x7f705400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705400aa50 con 0x7f70640727d0 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705400abb0 con 0x7f70640727d0 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70641516d0 con 0x7f70640727d0 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054012420 con 0x7f70640727d0 2024-02-19T14:54:20.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.099+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70640ac2c0 con 0x7f70640727d0 2024-02-19T14:54:20.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.100+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7064151bf0 con 0x7f70640727d0 2024-02-19T14:54:20.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.101+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7054012580 con 0x7f70640727d0 2024-02-19T14:54:20.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.101+0000 7f704bfff640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705006b810 0x7f705006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.101+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f705408a070 con 0x7f70640727d0 2024-02-19T14:54:20.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.101+0000 7f70627fc640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705006b810 0x7f705006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.101+0000 7f70627fc640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705006b810 0x7f705006dcd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7058036000 tx=0x7f7058030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.105+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7064151bf0 con 0x7f70640727d0 2024-02-19T14:54:20.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.274+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7064151bf0 con 0x7f70640727d0 2024-02-19T14:54:20.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.275+0000 7f704bfff640 1 -- 172.21.15.78:0/2661989373 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7064151bf0 con 0x7f70640727d0 2024-02-19T14:54:20.277 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.276+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705006b810 msgr2=0x7f705006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.276+0000 7f706990d640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705006b810 0x7f705006dcd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7058036000 tx=0x7f7058030000 comp rx=0 tx=0).stop 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.276+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 msgr2=0x7f70640c0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.276+0000 7f706990d640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 0x7f70640c0d40 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f705400c7f0 tx=0x7f705400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 shutdown_connections 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705006b810 0x7f705006dcd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f706414ef50 0x7f706414da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70640c78a0 0x7f706414d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 --2- 172.21.15.78:0/2661989373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70640727d0 0x7f70640c0d40 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 >> 172.21.15.78:0/2661989373 conn(0x7f7064066e70 msgr2=0x7f70640a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 shutdown_connections 2024-02-19T14:54:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.277+0000 7f706990d640 1 -- 172.21.15.78:0/2661989373 wait complete. 2024-02-19T14:54:20.290 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-02-19T14:54:20.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 Processor -- start 2024-02-19T14:54:20.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 -- start start 2024-02-19T14:54:20.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f308806ae80 0x7f308806b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f308806f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3088072180 con 0x7f30880c19f0 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308806fdd0 con 0x7f308806ae80 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.388+0000 7f308df0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3088073440 con 0x7f3088155af0 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f30877fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f308806ae80 0x7f308806b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f3086ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f308806f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f3087fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f30880c19f0 0x7f30881585e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:45002/0 (socket says 172.21.15.78:45002) 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f3087fff640 1 -- 172.21.15.78:0/1102059075 learned_addr learned my addr 172.21.15.78:0/1102059075 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f30877fe640 1 -- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 msgr2=0x7f308806f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f30877fe640 1 --2- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f308806f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f30877fe640 1 -- 172.21.15.78:0/1102059075 >> v1:172.21.15.150:6789/0 conn(0x7f30880c19f0 legacy=0x7f30881585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.389+0000 7f30877fe640 1 -- 172.21.15.78:0/1102059075 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308814f810 con 0x7f308806ae80 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.390+0000 7f30877fe640 1 --2- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f308806ae80 0x7f308806b260 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f30780097a0 tx=0x7f307802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81244f63e9212d45 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.390+0000 7f30867fc640 1 -- 172.21.15.78:0/1102059075 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307802f050 con 0x7f308806ae80 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.390+0000 7f30867fc640 1 -- 172.21.15.78:0/1102059075 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078031700 con 0x7f308806ae80 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.390+0000 7f30867fc640 1 -- 172.21.15.78:0/1102059075 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078031a90 con 0x7f308806ae80 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 -- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f308806ae80 msgr2=0x7f308806b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 --2- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f308806ae80 0x7f308806b260 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f30780097a0 tx=0x7f307802b7a0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 -- 172.21.15.78:0/1102059075 shutdown_connections 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 --2- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f308806f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 --2- 172.21.15.78:0/1102059075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f308806ae80 0x7f308806b260 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 -- 172.21.15.78:0/1102059075 >> 172.21.15.78:0/1102059075 conn(0x7f30880c7590 msgr2=0x7f30880c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 -- 172.21.15.78:0/1102059075 shutdown_connections 2024-02-19T14:54:20.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.391+0000 7f308df0f640 1 -- 172.21.15.78:0/1102059075 wait complete. 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.392+0000 7f308df0f640 1 Processor -- start 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.392+0000 7f308df0f640 1 -- start start 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.392+0000 7f308df0f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 0x7f3088158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30880c19f0 0x7f30881ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f3088156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088148490 con 0x7f308806ae80 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088148e30 con 0x7f30880c19f0 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30881486a0 con 0x7f3088155af0 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 0x7f3088158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 0x7f3088158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35226/0 (socket says 172.21.15.78:35226) 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 -- 172.21.15.78:0/2001722435 learned_addr learned my addr 172.21.15.78:0/2001722435 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f3087fff640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f3088156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f3086ffd640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30880c19f0 0x7f30881ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 -- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 msgr2=0x7f3088156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f3088156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 -- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30880c19f0 msgr2=0x7f30881ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30880c19f0 0x7f30881ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 -- 172.21.15.78:0/2001722435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078009400 con 0x7f308806ae80 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f30877fe640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 0x7f3088158d90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f30780097a0 tx=0x7f30780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30780339b0 con 0x7f308806ae80 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078033b10 con 0x7f308806ae80 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307803d470 con 0x7f308806ae80 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30881eea70 con 0x7f308806ae80 2024-02-19T14:54:20.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.393+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30881eed50 con 0x7f308806ae80 2024-02-19T14:54:20.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.395+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3088158ab0 con 0x7f308806ae80 2024-02-19T14:54:20.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.395+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f307803a040 con 0x7f308806ae80 2024-02-19T14:54:20.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.395+0000 7f306ffff640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f307406b810 0x7f307406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.395+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f30780abc30 con 0x7f308806ae80 2024-02-19T14:54:20.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.395+0000 7f3086ffd640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f307406b810 0x7f307406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.396+0000 7f3086ffd640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f307406b810 0x7f307406dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f308806f250 tx=0x7f307c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.400+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3088158ab0 con 0x7f308806ae80 2024-02-19T14:54:20.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.636+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7f3088158ab0 con 0x7f308806ae80 2024-02-19T14:54:20.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.681+0000 7f306ffff640 1 -- 172.21.15.78:0/2001722435 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v155) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3088158ab0 con 0x7f308806ae80 2024-02-19T14:54:20.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f307406b810 msgr2=0x7f307406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f307406b810 0x7f307406dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f308806f250 tx=0x7f307c009190 comp rx=0 tx=0).stop 2024-02-19T14:54:20.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 msgr2=0x7f3088158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 0x7f3088158d90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f30780097a0 tx=0x7f30780046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 shutdown_connections 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f307406b810 0x7f307406dcd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3088155af0 0x7f3088156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30880c19f0 0x7f30881ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 --2- 172.21.15.78:0/2001722435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f308806ae80 0x7f3088158d90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.683+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 >> 172.21.15.78:0/2001722435 conn(0x7f30880c7590 msgr2=0x7f30880c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.684+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 shutdown_connections 2024-02-19T14:54:20.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.684+0000 7f308df0f640 1 -- 172.21.15.78:0/2001722435 wait complete. 2024-02-19T14:54:20.697 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:54:20.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.797+0000 7fd67ed2c640 1 Processor -- start 2024-02-19T14:54:20.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.797+0000 7fd67ed2c640 1 -- start start 2024-02-19T14:54:20.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd67ed2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd67814f1a0 0x7fd678072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd67ed2c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd678152680 0x7fd6780c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd67ed2c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd678152ab0 con 0x7fd67806ea50 2024-02-19T14:54:20.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd67ed2c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6780c5970 con 0x7fd67814f1a0 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd67ed2c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6780c5750 con 0x7fd678152680 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd67cac8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd67814f1a0 0x7fd678072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd677fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd67806ea50 0x7fd67806fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59876/0 (socket says 172.21.15.78:59876) 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.798+0000 7fd677fff640 1 -- 172.21.15.78:0/1541372932 learned_addr learned my addr 172.21.15.78:0/1541372932 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd67d2c9640 1 --2- 172.21.15.78:0/1541372932 >> v2:172.21.15.150:3300/0 conn(0x7fd678152680 0x7fd6780c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd67cac8640 1 -- 172.21.15.78:0/1541372932 >> v2:172.21.15.150:3300/0 conn(0x7fd678152680 msgr2=0x7fd6780c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd67cac8640 1 --2- 172.21.15.78:0/1541372932 >> v2:172.21.15.150:3300/0 conn(0x7fd678152680 0x7fd6780c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd67cac8640 1 -- 172.21.15.78:0/1541372932 >> v1:172.21.15.78:6789/0 conn(0x7fd67806ea50 legacy=0x7fd67806fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd67cac8640 1 -- 172.21.15.78:0/1541372932 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780702d0 con 0x7fd67814f1a0 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd67cac8640 1 --2- 172.21.15.78:0/1541372932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67814f1a0 0x7fd678072d70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd67806de00 tx=0x7fd66802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6673754632646991 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.799+0000 7fd6777fe640 1 -- 172.21.15.78:0/1541372932 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd668031710 con 0x7fd67814f1a0 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd6777fe640 1 -- 172.21.15.78:0/1541372932 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66802f050 con 0x7fd67814f1a0 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd6777fe640 1 -- 172.21.15.78:0/1541372932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd668031a90 con 0x7fd67814f1a0 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 -- 172.21.15.78:0/1541372932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67814f1a0 msgr2=0x7fd678072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/1541372932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67814f1a0 0x7fd678072d70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd67806de00 tx=0x7fd66802b7b0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 -- 172.21.15.78:0/1541372932 shutdown_connections 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/1541372932 >> v2:172.21.15.150:3300/0 conn(0x7fd678152680 0x7fd6780c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/1541372932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67814f1a0 0x7fd678072d70 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 -- 172.21.15.78:0/1541372932 >> 172.21.15.78:0/1541372932 conn(0x7fd67806c170 msgr2=0x7fd67806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 -- 172.21.15.78:0/1541372932 shutdown_connections 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.800+0000 7fd67ed2c640 1 -- 172.21.15.78:0/1541372932 wait complete. 2024-02-19T14:54:20.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.801+0000 7fd67ed2c640 1 Processor -- start 2024-02-19T14:54:20.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.801+0000 7fd67ed2c640 1 -- start start 2024-02-19T14:54:20.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67ed2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 0x7fd6780a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67ed2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67814f1a0 0x7fd678148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67ed2c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd678152680 0x7fd678148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67ed2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6781ebad0 con 0x7fd67806ea50 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67ed2c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678152ed0 con 0x7fd678152680 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67ed2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780c0c40 con 0x7fd67814f1a0 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 0x7fd6780a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 0x7fd6780a8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35258/0 (socket says 172.21.15.78:35258) 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 -- 172.21.15.78:0/4256940955 learned_addr learned my addr 172.21.15.78:0/4256940955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd677fff640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67814f1a0 0x7fd678148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67d2c9640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd678152680 0x7fd678148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 -- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67814f1a0 msgr2=0x7fd678148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67814f1a0 0x7fd678148530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 -- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd678152680 msgr2=0x7fd678148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.802+0000 7fd67cac8640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd678152680 0x7fd678148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd67cac8640 1 -- 172.21.15.78:0/4256940955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668009400 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd67d2c9640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd678152680 0x7fd678148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd67cac8640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 0x7fd6780a8a30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd668000c00 tx=0x7fd6680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd668033910 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd668033a70 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd668033e20 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6780c0f20 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.803+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6780c14e0 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.804+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6780c61e0 con 0x7fd67806ea50 2024-02-19T14:54:20.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.804+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd66803a040 con 0x7fd67806ea50 2024-02-19T14:54:20.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.805+0000 7fd6757fa640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd66406b840 0x7fd66406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:20.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.805+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fd6680ab740 con 0x7fd67806ea50 2024-02-19T14:54:20.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.805+0000 7fd677fff640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd66406b840 0x7fd66406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:20.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.805+0000 7fd677fff640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd66406b840 0x7fd66406dd00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd6780a9230 tx=0x7fd66c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:20.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.809+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd668076210 con 0x7fd67806ea50 2024-02-19T14:54:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.981+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd67814de30 con 0x7fd67806ea50 2024-02-19T14:54:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.984+0000 7fd6757fa640 1 -- 172.21.15.78:0/4256940955 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v23)=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd6680798d0 con 0x7fd67806ea50 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd66406b840 msgr2=0x7fd66406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd66406b840 0x7fd66406dd00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd6780a9230 tx=0x7fd66c009040 comp rx=0 tx=0).stop 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 msgr2=0x7fd6780a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 0x7fd6780a8a30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd668000c00 tx=0x7fd6680046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 shutdown_connections 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd66406b840 0x7fd66406dd00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd678152680 0x7fd678148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67814f1a0 0x7fd678148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 --2- 172.21.15.78:0/4256940955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd67806ea50 0x7fd6780a8a30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.986+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 >> 172.21.15.78:0/4256940955 conn(0x7fd67806c170 msgr2=0x7fd67806fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.987+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 shutdown_connections 2024-02-19T14:54:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:20.987+0000 7fd67ed2c640 1 -- 172.21.15.78:0/4256940955 wait complete. 2024-02-19T14:54:21.000 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:54:21.000 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:54:21.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:54:21.093 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:54:21.094 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:54:21.094 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:54:21.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:54:21.109 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:54:21.109 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:54:21.127 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:54:21.127 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:54:21.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:21.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.246+0000 7f0b6c35c640 1 Processor -- start 2024-02-19T14:54:21.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.246+0000 7f0b6c35c640 1 -- start start 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6c35c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b640c25e0 0x7f0b641526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6c35c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b640c5be0 0x7f0b6406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6c35c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b64072250 con 0x7f0b640c29c0 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6c35c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b64152c90 con 0x7f0b640c25e0 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6c35c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b64072480 con 0x7f0b640c5be0 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b640c25e0 0x7f0b641526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b640c25e0 0x7f0b641526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35262/0 (socket says 172.21.15.78:35262) 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/3177965491 learned_addr learned my addr 172.21.15.78:0/3177965491 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/3177965491 >> v2:172.21.15.150:3300/0 conn(0x7f0b640c5be0 msgr2=0x7f0b6406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:21.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 --2- 172.21.15.78:0/3177965491 >> v2:172.21.15.150:3300/0 conn(0x7f0b640c5be0 0x7f0b6406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/3177965491 >> v1:172.21.15.150:6789/0 conn(0x7f0b640c29c0 legacy=0x7f0b640c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:21.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.247+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/3177965491 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640c4b90 con 0x7f0b640c25e0 2024-02-19T14:54:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.248+0000 7f0b6a0f8640 1 --2- 172.21.15.78:0/3177965491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b641526a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0b64151c50 tx=0x7f0b6002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715885948ff356ef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.248+0000 7f0b690f6640 1 -- 172.21.15.78:0/3177965491 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b600317e0 con 0x7f0b640c25e0 2024-02-19T14:54:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.248+0000 7f0b690f6640 1 -- 172.21.15.78:0/3177965491 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60031940 con 0x7f0b640c25e0 2024-02-19T14:54:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.248+0000 7f0b690f6640 1 -- 172.21.15.78:0/3177965491 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60031cd0 con 0x7f0b640c25e0 2024-02-19T14:54:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.248+0000 7f0b6c35c640 1 -- 172.21.15.78:0/3177965491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 msgr2=0x7f0b641526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.248+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/3177965491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b641526a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0b64151c50 tx=0x7f0b6002b880 comp rx=0 tx=0).stop 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.249+0000 7f0b6c35c640 1 -- 172.21.15.78:0/3177965491 shutdown_connections 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.249+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/3177965491 >> v2:172.21.15.150:3300/0 conn(0x7f0b640c5be0 0x7f0b6406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.249+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/3177965491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b641526a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.249+0000 7f0b6c35c640 1 -- 172.21.15.78:0/3177965491 >> 172.21.15.78:0/3177965491 conn(0x7f0b6406b270 msgr2=0x7f0b6406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.249+0000 7f0b6c35c640 1 -- 172.21.15.78:0/3177965491 shutdown_connections 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.249+0000 7f0b6c35c640 1 -- 172.21.15.78:0/3177965491 wait complete. 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 Processor -- start 2024-02-19T14:54:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 -- start start 2024-02-19T14:54:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b6414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b640c29c0 0x7f0b640a8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b640c5be0 0x7f0b640a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b641eb690 con 0x7f0b640c25e0 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6414d1b0 con 0x7f0b640c29c0 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6c35c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b64149630 con 0x7f0b640c5be0 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6a0f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b6414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6a0f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b6414c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35270/0 (socket says 172.21.15.78:35270) 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.250+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/372110507 learned_addr learned my addr 172.21.15.78:0/372110507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/372110507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b640c5be0 msgr2=0x7f0b640a8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6a0f8640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b640c5be0 0x7f0b640a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/372110507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b640c29c0 msgr2=0x7f0b640a8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6a0f8640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b640c29c0 0x7f0b640a8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6a0f8640 1 -- 172.21.15.78:0/372110507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600094d0 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6a0f8640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b6414c800 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0b6002b8c0 tx=0x7f0b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60032940 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60032ac0 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b641e8f50 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60032e70 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.251+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b640a6c30 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.252+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b641eb7e0 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.253+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0b6004d3f0 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.253+0000 7f0b5affd640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b4806b810 0x7f0b4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.253+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f0b600ac010 con 0x7f0b640c25e0 2024-02-19T14:54:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.253+0000 7f0b698f7640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b4806b810 0x7f0b4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.253+0000 7f0b698f7640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b4806b810 0x7f0b4806dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007fd0 tx=0x7f0b54007980 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:21.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.257+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0b641eb7e0 con 0x7f0b640c25e0 2024-02-19T14:54:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.465+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b641eb7e0 con 0x7f0b640c25e0 2024-02-19T14:54:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.465+0000 7f0b5affd640 1 -- 172.21.15.78:0/372110507 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f0b641eb7e0 con 0x7f0b640c25e0 2024-02-19T14:54:21.468 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:21.468 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2194703685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2194703685},{"type":"v1","addr":"172.21.15.150:6835","nonce":2194703685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5635,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/2334690423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2334690423},{"type":"v1","addr":"172.21.15.150:6834","nonce":2334690423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6156,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1284379174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1284379174},{"type":"v1","addr":"172.21.15.78:6837","nonce":1284379174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6171,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/4156997217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4156997217},{"type":"v1","addr":"172.21.15.78:6836","nonce":4156997217}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[]} 2024-02-19T14:54:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 47 2024-02-19T14:54:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b4806b810 msgr2=0x7f0b4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b4806b810 0x7f0b4806dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007fd0 tx=0x7f0b54007980 comp rx=0 tx=0).stop 2024-02-19T14:54:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 msgr2=0x7f0b6414c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b6414c800 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0b6002b8c0 tx=0x7f0b600046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 shutdown_connections 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b4806b810 0x7f0b4806dcd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b640c5be0 0x7f0b640a8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b640c29c0 0x7f0b640a8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 --2- 172.21.15.78:0/372110507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b640c25e0 0x7f0b6414c800 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 >> 172.21.15.78:0/372110507 conn(0x7f0b6406b270 msgr2=0x7f0b6406e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 shutdown_connections 2024-02-19T14:54:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.467+0000 7f0b6c35c640 1 -- 172.21.15.78:0/372110507 wait complete. 2024-02-19T14:54:21.481 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:54:21.481 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:54:21.481 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:54:21.481 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:54:21.482 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:54:21.482 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:54:21.482 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:54:21.482 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:54:21.482 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:54:21.482 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:54:21.483 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:54:21.483 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:54:21.483 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:54:21.483 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:54:21.483 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:54:21.483 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:54:21.491 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:54:21.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:54:21.492 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:54:21.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:54:21.498 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:54:21.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:54:21.499 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:54:21.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:54:21.534 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:54:21.534 DEBUG:teuthology.parallel:result is None 2024-02-19T14:54:21.534 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:54:21.534 DEBUG:teuthology.parallel:result is None 2024-02-19T14:54:21.541 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:54:21.541 DEBUG:teuthology.parallel:result is None 2024-02-19T14:54:21.542 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:54:21.542 DEBUG:teuthology.parallel:result is None 2024-02-19T14:54:21.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:54:21.609 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:54:21.610 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:54:21.626 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:54:21.626 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:54:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.645+0000 7fe871bf4640 1 Processor -- start 2024-02-19T14:54:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.645+0000 7fe871bf4640 1 -- start start 2024-02-19T14:54:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe871bf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c0c8e00 0x7fe86c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe871bf4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe86c06d910 0x7fe86c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe871bf4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c06cd20 con 0x7fe86c152700 2024-02-19T14:54:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe871bf4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86c1551f0 con 0x7fe86c06d910 2024-02-19T14:54:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe871bf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c155490 con 0x7fe86c0c8e00 2024-02-19T14:54:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe86bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe86c06d910 0x7fe86c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe86bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe86c06d910 0x7fe86c06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:51184/0 (socket says 172.21.15.78:51184) 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe86bfff640 1 -- 172.21.15.78:0/545389292 learned_addr learned my addr 172.21.15.78:0/545389292 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe86b7fe640 1 --1- 172.21.15.78:0/545389292 >> v1:172.21.15.150:6789/0 conn(0x7fe86c152700 0x7fe86c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:45022/0 (socket says 172.21.15.78:45022) 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.646+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c0c8e00 0x7fe86c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.647+0000 7fe86bfff640 1 -- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c0c8e00 msgr2=0x7fe86c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.647+0000 7fe86bfff640 1 --2- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c0c8e00 0x7fe86c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.647+0000 7fe86bfff640 1 -- 172.21.15.78:0/545389292 >> v1:172.21.15.150:6789/0 conn(0x7fe86c152700 legacy=0x7fe86c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.647+0000 7fe86affd640 1 -- 172.21.15.78:0/545389292 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246863 0 0) 0x7fe86c06cd20 con 0x7fe86c152700 2024-02-19T14:54:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.647+0000 7fe86bfff640 1 -- 172.21.15.78:0/545389292 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c155770 con 0x7fe86c06d910 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.647+0000 7fe86bfff640 1 --2- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c06d910 0x7fe86c06c7e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe864009e50 tx=0x7fe86402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34836faa704c896 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe86affd640 1 -- 172.21.15.78:0/545389292 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86402ca70 con 0x7fe86c06d910 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe86affd640 1 -- 172.21.15.78:0/545389292 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86402cbd0 con 0x7fe86c06d910 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe86affd640 1 -- 172.21.15.78:0/545389292 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640343f0 con 0x7fe86c06d910 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 -- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c06d910 msgr2=0x7fe86c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 --2- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c06d910 0x7fe86c06c7e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe864009e50 tx=0x7fe86402bed0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 -- 172.21.15.78:0/545389292 shutdown_connections 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 --2- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c0c8e00 0x7fe86c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 --2- 172.21.15.78:0/545389292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c06d910 0x7fe86c06c7e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 -- 172.21.15.78:0/545389292 >> 172.21.15.78:0/545389292 conn(0x7fe86c0cac90 msgr2=0x7fe86c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 -- 172.21.15.78:0/545389292 shutdown_connections 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.648+0000 7fe871bf4640 1 -- 172.21.15.78:0/545389292 wait complete. 2024-02-19T14:54:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.649+0000 7fe871bf4640 1 Processor -- start 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 -- start start 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c06d910 0x7fe86c154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 0x7fe86c1ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c152700 0x7fe86c1eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c154f40 con 0x7fe86c0c8e00 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c1ee5b0 con 0x7fe86c152700 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.650+0000 7fe871bf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c158b90 con 0x7fe86c06d910 2024-02-19T14:54:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 0x7fe86c1ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 0x7fe86c1ea6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35320/0 (socket says 172.21.15.78:35320) 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 -- 172.21.15.78:0/316384121 learned_addr learned my addr 172.21.15.78:0/316384121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe86b7fe640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c06d910 0x7fe86c154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe86bfff640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c152700 0x7fe86c1eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 -- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c06d910 msgr2=0x7fe86c154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c06d910 0x7fe86c154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 -- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c152700 msgr2=0x7fe86c1eabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c152700 0x7fe86c1eabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 -- 172.21.15.78:0/316384121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe864009b30 con 0x7fe86c0c8e00 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe86b7fe640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c06d910 0x7fe86c154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe862ffd640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 0x7fe86c1ea6a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe85400e9b0 tx=0x7fe85400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854009670 con 0x7fe86c0c8e00 2024-02-19T14:54:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85400c9a0 con 0x7fe86c0c8e00 2024-02-19T14:54:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85400cd50 con 0x7fe86c0c8e00 2024-02-19T14:54:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c147790 con 0x7fe86c0c8e00 2024-02-19T14:54:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.651+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c1ee7a0 con 0x7fe86c0c8e00 2024-02-19T14:54:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.653+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c158d00 con 0x7fe86c0c8e00 2024-02-19T14:54:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.653+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe854013470 con 0x7fe86c0c8e00 2024-02-19T14:54:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.653+0000 7fe868ff9640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe85006b810 0x7fe85006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.654+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe85408a280 con 0x7fe86c0c8e00 2024-02-19T14:54:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.654+0000 7fe86b7fe640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe85006b810 0x7fe85006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.654+0000 7fe86b7fe640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe85006b810 0x7fe85006dcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe85c000f60 tx=0x7fe85c007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.658+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe86c158d00 con 0x7fe86c0c8e00 2024-02-19T14:54:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:21.843+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe86c158d00 con 0x7fe86c0c8e00 2024-02-19T14:54:22.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.706+0000 7fe868ff9640 1 -- 172.21.15.78:0/316384121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v71) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe86c158d00 con 0x7fe86c0c8e00 2024-02-19T14:54:22.709 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:54:22.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe85006b810 msgr2=0x7fe85006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:22.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe85006b810 0x7fe85006dcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe85c000f60 tx=0x7fe85c007ca0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 msgr2=0x7fe86c1ea6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:22.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 0x7fe86c1ea6a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe85400e9b0 tx=0x7fe85400ee70 comp rx=0 tx=0).stop 2024-02-19T14:54:22.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 shutdown_connections 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe85006b810 0x7fe85006dcd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c0c8e00 0x7fe86c1ea6a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c152700 0x7fe86c1eabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 --2- 172.21.15.78:0/316384121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe86c06d910 0x7fe86c154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 >> 172.21.15.78:0/316384121 conn(0x7fe86c0cac90 msgr2=0x7fe86c14a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 shutdown_connections 2024-02-19T14:54:22.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.709+0000 7fe871bf4640 1 -- 172.21.15.78:0/316384121 wait complete. 2024-02-19T14:54:22.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:54:22.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.829+0000 7ff9ef01c640 1 Processor -- start 2024-02-19T14:54:22.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.829+0000 7ff9ef01c640 1 -- start start 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.829+0000 7ff9ef01c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9e8155f90 0x7ff9e806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ef01c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9e81588a0 0x7ff9e8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ef01c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80c6af0 con 0x7ff9e80c6d30 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ef01c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80c7940 con 0x7ff9e81588a0 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ef01c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80c7c20 con 0x7ff9e8155f90 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9e81588a0 0x7ff9e8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9e81588a0 0x7ff9e8158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35334/0 (socket says 172.21.15.78:35334) 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1453316811 learned_addr learned my addr 172.21.15.78:0/1453316811 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1453316811 >> v2:172.21.15.150:3300/0 conn(0x7ff9e8155f90 msgr2=0x7ff9e806d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 --2- 172.21.15.78:0/1453316811 >> v2:172.21.15.150:3300/0 conn(0x7ff9e8155f90 0x7ff9e806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1453316811 >> v1:172.21.15.150:6789/0 conn(0x7ff9e80c6d30 legacy=0x7ff9e80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:22.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.830+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1453316811 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80c7f00 con 0x7ff9e81588a0 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ed5b9640 1 --2- 172.21.15.78:0/1453316811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8158ca0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80c2a40 tx=0x7ff9e002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561a59bb5f8a419e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9e77fe640 1 -- 172.21.15.78:0/1453316811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e002c9f0 con 0x7ff9e81588a0 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9e77fe640 1 -- 172.21.15.78:0/1453316811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e002cb50 con 0x7ff9e81588a0 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9e77fe640 1 -- 172.21.15.78:0/1453316811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00343f0 con 0x7ff9e81588a0 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1453316811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 msgr2=0x7ff9e8158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1453316811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8158ca0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80c2a40 tx=0x7ff9e002be50 comp rx=0 tx=0).stop 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1453316811 shutdown_connections 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1453316811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8158ca0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1453316811 >> v2:172.21.15.150:3300/0 conn(0x7ff9e8155f90 0x7ff9e806d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1453316811 >> 172.21.15.78:0/1453316811 conn(0x7ff9e806b700 msgr2=0x7ff9e80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1453316811 shutdown_connections 2024-02-19T14:54:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.831+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1453316811 wait complete. 2024-02-19T14:54:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 Processor -- start 2024-02-19T14:54:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 -- start start 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e80c6d30 0x7ff9e81e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9e8155f90 0x7ff9e81ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81de7b0 con 0x7ff9e81588a0 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81e2110 con 0x7ff9e80c6d30 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ef01c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80a9910 con 0x7ff9e8155f90 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.833+0000 7ff9ed5b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35338/0 (socket says 172.21.15.78:35338) 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1965923931 learned_addr learned my addr 172.21.15.78:0/1965923931 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9e7fff640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9e8155f90 0x7ff9e81ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ecdb8640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e80c6d30 0x7ff9e81e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9e8155f90 msgr2=0x7ff9e81ee2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9e8155f90 0x7ff9e81ee2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e80c6d30 msgr2=0x7ff9e81e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e80c6d30 0x7ff9e81e1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 -- 172.21.15.78:0/1965923931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0009b30 con 0x7ff9e81588a0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ed5b9640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8150660 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009f40 tx=0x7ff9e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e003e450 con 0x7ff9e81588a0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e003e5d0 con 0x7ff9e81588a0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e81eed80 con 0x7ff9e81588a0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e003e980 con 0x7ff9e81588a0 2024-02-19T14:54:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.834+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e81ec370 con 0x7ff9e81588a0 2024-02-19T14:54:22.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.835+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80c7170 con 0x7ff9e81588a0 2024-02-19T14:54:22.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.836+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff9e003eae0 con 0x7ff9e81588a0 2024-02-19T14:54:22.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.836+0000 7ff9e57fa640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9cc06b840 0x7ff9cc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:22.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.836+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff9e00acbc0 con 0x7ff9e81588a0 2024-02-19T14:54:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.836+0000 7ff9ecdb8640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9cc06b840 0x7ff9cc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:22.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.836+0000 7ff9ecdb8640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9cc06b840 0x7ff9cc06dd00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009890 tx=0x7ff9dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:22.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:22.840+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff9e80c7170 con 0x7ff9e81588a0 2024-02-19T14:54:23.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.080+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff9e80c7170 con 0x7ff9e81588a0 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.080+0000 7ff9e57fa640 1 -- 172.21.15.78:0/1965923931 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+4724 (secure 0 0 0) 0x7ff9e80c7170 con 0x7ff9e81588a0 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:54:23.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.084 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:54:23.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.086 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:54:23.088 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.089 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:54:23.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:54:23.094 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9cc06b840 msgr2=0x7ff9cc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9cc06b840 0x7ff9cc06dd00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009890 tx=0x7ff9dc008040 comp rx=0 tx=0).stop 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 msgr2=0x7ff9e8150660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8150660 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009f40 tx=0x7ff9e0004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 shutdown_connections 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9cc06b840 0x7ff9cc06dd00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9e81588a0 0x7ff9e8150660 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9e8155f90 0x7ff9e81ee2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 --2- 172.21.15.78:0/1965923931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e80c6d30 0x7ff9e81e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 >> 172.21.15.78:0/1965923931 conn(0x7ff9e806b700 msgr2=0x7ff9e806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 shutdown_connections 2024-02-19T14:54:23.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.082+0000 7ff9ef01c640 1 -- 172.21.15.78:0/1965923931 wait complete. 2024-02-19T14:54:23.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.193+0000 7f252cdbc640 1 Processor -- start 2024-02-19T14:54:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.193+0000 7f252cdbc640 1 -- start start 2024-02-19T14:54:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.194+0000 7f252cdbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 0x7f2528155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.194+0000 7f252cdbc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528155fd0 con 0x7f252813add0 2024-02-19T14:54:23.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.194+0000 7f252cdbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280c4190 con 0x7f2528159320 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.194+0000 7f252cdbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25280c2030 con 0x7f25281555e0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f2525d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f252813add0 0x7f25280718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59916/0 (socket says 172.21.15.78:59916) 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f2525d9b640 1 -- 172.21.15.78:0/421295432 learned_addr learned my addr 172.21.15.78:0/421295432 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037927321 0 0) 0x7f2528155fd0 con 0x7f252813add0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2510003520 con 0x7f252813add0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f2526d9d640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 0x7f2528155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088962184 0 0) 0x7f2510003520 con 0x7f252813add0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 msgr2=0x7f2528155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 0x7f2528155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 >> v1:172.21.15.150:6789/0 conn(0x7f2528159320 legacy=0x7f25280c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25280c4470 con 0x7f252813add0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2518001670 con 0x7f252813add0 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 >> v1:172.21.15.78:6789/0 conn(0x7f252813add0 legacy=0x7f25280718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25100039a0 0x7f2510003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 0x7f25100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2510004d00 0x7f2510005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25280c2030 con 0x7f2510004d00 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251000bb90 con 0x7f25100039a0 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251000bd50 con 0x7f25281555e0 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f2526d9d640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2510004d00 0x7f2510005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f252659c640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25100039a0 0x7f2510003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.195+0000 7f2525d9b640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 0x7f25100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f2526d9d640 1 -- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 msgr2=0x7f25100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f2526d9d640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25281555e0 0x7f25100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f2526d9d640 1 -- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25100039a0 msgr2=0x7f2510003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f2526d9d640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25100039a0 0x7f2510003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f2526d9d640 1 -- 172.21.15.78:0/421295432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2510003520 con 0x7f2510004d00 2024-02-19T14:54:23.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f2526d9d640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2510004d00 0x7f2510005180 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2520007360 tx=0x7f25100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7636d1b8392bca56 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252002ccd0 con 0x7f2510004d00 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252002ce30 con 0x7f2510004d00 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.196+0000 7f252559a640 1 -- 172.21.15.78:0/421295432 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520034660 con 0x7f2510004d00 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f252cdbc640 1 -- 172.21.15.78:0/421295432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2510004d00 msgr2=0x7f2510005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f252cdbc640 1 --2- 172.21.15.78:0/421295432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2510004d00 0x7f2510005180 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2520007360 tx=0x7f25100057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f2526d9d640 1 -- 172.21.15.78:0/421295432 reap_dead start 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f252cdbc640 1 -- 172.21.15.78:0/421295432 shutdown_connections 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f252cdbc640 1 -- 172.21.15.78:0/421295432 >> 172.21.15.78:0/421295432 conn(0x7f25280c7800 msgr2=0x7f25280c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f252cdbc640 1 -- 172.21.15.78:0/421295432 shutdown_connections 2024-02-19T14:54:23.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.197+0000 7f252cdbc640 1 -- 172.21.15.78:0/421295432 wait complete. 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.198+0000 7f252cdbc640 1 Processor -- start 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.198+0000 7f252cdbc640 1 -- start start 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252cdbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2510004d00 0x7f252806b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252cdbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 0x7f25281387d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252cdbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2528138d10 0x7f25281eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252cdbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2528139740 con 0x7f25281398b0 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252cdbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25280ab7c0 con 0x7f2510004d00 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252cdbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2528139190 con 0x7f2528138d10 2024-02-19T14:54:23.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 0x7f25281387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 0x7f25281387d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35358/0 (socket says 172.21.15.78:35358) 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 -- 172.21.15.78:0/1793595125 learned_addr learned my addr 172.21.15.78:0/1793595125 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252659c640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2510004d00 0x7f252806b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 -- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2528138d10 msgr2=0x7f25281eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2528138d10 0x7f25281eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 -- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2510004d00 msgr2=0x7f252806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2510004d00 0x7f252806b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f2525d9b640 1 -- 172.21.15.78:0/1793595125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25281e19b0 con 0x7f25281398b0 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.199+0000 7f252659c640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2510004d00 0x7f252806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.200+0000 7f2525d9b640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 0x7f25281387d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2518007290 tx=0x7f2518007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.200+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251800f940 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.200+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251800fac0 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.200+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251800fe70 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.200+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25280a8560 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.200+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25280a8800 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.201+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25280c25b0 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.201+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f251801d460 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.201+0000 7f2516ffd640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f250806b810 0x7f250806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.201+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f251808a180 con 0x7f25281398b0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.201+0000 7f252659c640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f250806b810 0x7f250806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.202+0000 7f252659c640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f250806b810 0x7f250806dcd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f251c000c00 tx=0x7f251c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.206+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2518054d60 con 0x7f25281398b0 2024-02-19T14:54:23.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.377+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2528158560 con 0x7f25281398b0 2024-02-19T14:54:23.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.377+0000 7f2516ffd640 1 -- 172.21.15.78:0/1793595125 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+13153 (secure 0 0 0) 0x7f2518058420 con 0x7f25281398b0 2024-02-19T14:54:23.380 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:23.380 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":71,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:22.705970+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f250806b810 msgr2=0x7f250806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f250806b810 0x7f250806dcd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f251c000c00 tx=0x7f251c009040 comp rx=0 tx=0).stop 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 msgr2=0x7f25281387d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 0x7f25281387d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2518007290 tx=0x7f2518007750 comp rx=0 tx=0).stop 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 shutdown_connections 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f250806b810 0x7f250806dcd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25281398b0 0x7f25281387d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2528138d10 0x7f25281eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 --2- 172.21.15.78:0/1793595125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2510004d00 0x7f252806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.379+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 >> 172.21.15.78:0/1793595125 conn(0x7f25280c7800 msgr2=0x7f25280718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.380+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 shutdown_connections 2024-02-19T14:54:23.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.380+0000 7f252cdbc640 1 -- 172.21.15.78:0/1793595125 wait complete. 2024-02-19T14:54:23.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:23.394 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:23.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.493+0000 7f571bef4640 1 Processor -- start 2024-02-19T14:54:23.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571bef4640 1 -- start start 2024-02-19T14:54:23.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571bef4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57140c8e00 0x7f571406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571bef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f571406d910 0x7f571406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571bef4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571406cd20 con 0x7f5714152700 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571bef4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57141551f0 con 0x7f57140c8e00 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571bef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5714155490 con 0x7f571406d910 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571948f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57140c8e00 0x7f571406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571948f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57140c8e00 0x7f571406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35360/0 (socket says 172.21.15.78:35360) 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.494+0000 7f571948f640 1 -- 172.21.15.78:0/155969907 learned_addr learned my addr 172.21.15.78:0/155969907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571a491640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f571406d910 0x7f571406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571948f640 1 -- 172.21.15.78:0/155969907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f571406d910 msgr2=0x7f571406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571948f640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f571406d910 0x7f571406c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571948f640 1 -- 172.21.15.78:0/155969907 >> v1:172.21.15.150:6789/0 conn(0x7f5714152700 legacy=0x7f57140c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571948f640 1 -- 172.21.15.78:0/155969907 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714155770 con 0x7f57140c8e00 2024-02-19T14:54:23.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571a491640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f571406d910 0x7f571406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.495+0000 7f571948f640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f571406c2a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5704009870 tx=0x7f570402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec1b67c8463ea78 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f5718c8e640 1 -- 172.21.15.78:0/155969907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704031640 con 0x7f57140c8e00 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f5718c8e640 1 -- 172.21.15.78:0/155969907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57040317a0 con 0x7f57140c8e00 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f5718c8e640 1 -- 172.21.15.78:0/155969907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704031b30 con 0x7f57140c8e00 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 -- 172.21.15.78:0/155969907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 msgr2=0x7f571406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f571406c2a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5704009870 tx=0x7f570402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 -- 172.21.15.78:0/155969907 shutdown_connections 2024-02-19T14:54:23.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f571406d910 0x7f571406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 --2- 172.21.15.78:0/155969907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f571406c2a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 -- 172.21.15.78:0/155969907 >> 172.21.15.78:0/155969907 conn(0x7f57140cac90 msgr2=0x7f57140cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 -- 172.21.15.78:0/155969907 shutdown_connections 2024-02-19T14:54:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.496+0000 7f571bef4640 1 -- 172.21.15.78:0/155969907 wait complete. 2024-02-19T14:54:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.497+0000 7f571bef4640 1 Processor -- start 2024-02-19T14:54:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.497+0000 7f571bef4640 1 -- start start 2024-02-19T14:54:23.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571bef4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f571406d910 0x7f57140aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571bef4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f57140c1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571bef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5714152700 0x7f5714158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571bef4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5714156650 con 0x7f57140c8e00 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571bef4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57141ebbf0 con 0x7f571406d910 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571bef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57140aaec0 con 0x7f5714152700 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f57140c1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f57140c1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35376/0 (socket says 172.21.15.78:35376) 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 -- 172.21.15.78:0/2690775235 learned_addr learned my addr 172.21.15.78:0/2690775235 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571a491640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5714152700 0x7f5714158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f5719c90640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f571406d910 0x7f57140aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 -- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5714152700 msgr2=0x7f5714158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5714152700 0x7f5714158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 -- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f571406d910 msgr2=0x7f57140aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f571406d910 0x7f57140aa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571948f640 1 -- 172.21.15.78:0/2690775235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57040094d0 con 0x7f57140c8e00 2024-02-19T14:54:23.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f5719c90640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f571406d910 0x7f57140aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.498+0000 7f571a491640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5714152700 0x7f5714158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.499+0000 7f571948f640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f57140c1070 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5704000c00 tx=0x7f57040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.499+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570403ac60 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.499+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570403adc0 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.499+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5714157610 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.499+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704044630 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.499+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57140c1d10 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.500+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5714067ed0 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.500+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f570402f050 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.500+0000 7f570a7fc640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f56f806b810 0x7f56f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.501+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f57040abe60 con 0x7f57140c8e00 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.501+0000 7f5719c90640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f56f806b810 0x7f56f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.501+0000 7f5719c90640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f56f806b810 0x7f56f806dcd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57141e8c40 tx=0x7f5710009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.505+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5704076ad0 con 0x7f57140c8e00 2024-02-19T14:54:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.674+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5714074990 con 0x7f57140c8e00 2024-02-19T14:54:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.675+0000 7f570a7fc640 1 -- 172.21.15.78:0/2690775235 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57040350a0 con 0x7f57140c8e00 2024-02-19T14:54:23.677 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.676+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f56f806b810 msgr2=0x7f56f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.676+0000 7f571bef4640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f56f806b810 0x7f56f806dcd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57141e8c40 tx=0x7f5710009040 comp rx=0 tx=0).stop 2024-02-19T14:54:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.676+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 msgr2=0x7f57140c1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.676+0000 7f571bef4640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f57140c1070 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5704000c00 tx=0x7f57040046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 shutdown_connections 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f56f806b810 0x7f56f806dcd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57140c8e00 0x7f57140c1070 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5714152700 0x7f5714158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 --2- 172.21.15.78:0/2690775235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f571406d910 0x7f57140aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 >> 172.21.15.78:0/2690775235 conn(0x7f57140cac90 msgr2=0x7f57140cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 shutdown_connections 2024-02-19T14:54:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.677+0000 7f571bef4640 1 -- 172.21.15.78:0/2690775235 wait complete. 2024-02-19T14:54:23.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:54:23.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.788+0000 7f6819560640 1 Processor -- start 2024-02-19T14:54:23.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.788+0000 7f6819560640 1 -- start start 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f6819560640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68140c7910 0x7f68140c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f6819560640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6814070720 0x7f6814070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f6819560640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68140c65a0 con 0x7f68140c5560 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f6819560640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68140c7530 con 0x7f6814070720 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f6819560640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68141537b0 con 0x7f68140c7910 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f68137fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6814070720 0x7f6814070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f68137fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6814070720 0x7f6814070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35382/0 (socket says 172.21.15.78:35382) 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.789+0000 7f68137fe640 1 -- 172.21.15.78:0/1308013576 learned_addr learned my addr 172.21.15.78:0/1308013576 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f68127fc640 1 --2- 172.21.15.78:0/1308013576 >> v2:172.21.15.150:3300/0 conn(0x7f68140c7910 0x7f68140c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f68137fe640 1 -- 172.21.15.78:0/1308013576 >> v2:172.21.15.150:3300/0 conn(0x7f68140c7910 msgr2=0x7f68140c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:23.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f68137fe640 1 --2- 172.21.15.78:0/1308013576 >> v2:172.21.15.150:3300/0 conn(0x7f68140c7910 0x7f68140c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f68137fe640 1 -- 172.21.15.78:0/1308013576 >> v1:172.21.15.150:6789/0 conn(0x7f68140c5560 legacy=0x7f6814154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f68137fe640 1 -- 172.21.15.78:0/1308013576 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814153a90 con 0x7f6814070720 2024-02-19T14:54:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f68137fe640 1 --2- 172.21.15.78:0/1308013576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6814070720 0x7f6814070b70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f68140c91b0 tx=0x7f680c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee87a37e5bf6423 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f6811ffb640 1 -- 172.21.15.78:0/1308013576 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c02c960 con 0x7f6814070720 2024-02-19T14:54:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f6811ffb640 1 -- 172.21.15.78:0/1308013576 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c02cac0 con 0x7f6814070720 2024-02-19T14:54:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.790+0000 7f6811ffb640 1 -- 172.21.15.78:0/1308013576 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c02ce50 con 0x7f6814070720 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 -- 172.21.15.78:0/1308013576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6814070720 msgr2=0x7f6814070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 --2- 172.21.15.78:0/1308013576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6814070720 0x7f6814070b70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f68140c91b0 tx=0x7f680c02bdc0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 -- 172.21.15.78:0/1308013576 shutdown_connections 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 --2- 172.21.15.78:0/1308013576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6814070720 0x7f6814070b70 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 --2- 172.21.15.78:0/1308013576 >> v2:172.21.15.150:3300/0 conn(0x7f68140c7910 0x7f68140c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 -- 172.21.15.78:0/1308013576 >> 172.21.15.78:0/1308013576 conn(0x7f681406b4b0 msgr2=0x7f68140c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 -- 172.21.15.78:0/1308013576 shutdown_connections 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.791+0000 7f6819560640 1 -- 172.21.15.78:0/1308013576 wait complete. 2024-02-19T14:54:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.792+0000 7f6819560640 1 Processor -- start 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.792+0000 7f6819560640 1 -- start start 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6819560640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6814070720 0x7f68140a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6819560640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140c5560 0x7f68141dc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6819560640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 0x7f68141dccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6819560640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681414cbb0 con 0x7f68140c7910 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6819560640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68141de7c0 con 0x7f68140c5560 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6819560640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68141dc2f0 con 0x7f6814070720 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 0x7f68141dccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 0x7f68141dccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35388/0 (socket says 172.21.15.78:35388) 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 -- 172.21.15.78:0/4188630500 learned_addr learned my addr 172.21.15.78:0/4188630500 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68127fc640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140c5560 0x7f68141dc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6812ffd640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6814070720 0x7f68140a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 -- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6814070720 msgr2=0x7f68140a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6814070720 0x7f68140a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 -- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140c5560 msgr2=0x7f68141dc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140c5560 0x7f68141dc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f68137fe640 1 -- 172.21.15.78:0/4188630500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c009b30 con 0x7f68140c7910 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.793+0000 7f6812ffd640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6814070720 0x7f68140a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.794+0000 7f68137fe640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 0x7f68141dccc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f680c009f40 tx=0x7f680c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.794+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c03e450 con 0x7f68140c7910 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.794+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c03e5d0 con 0x7f68140c7910 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.794+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68140c5ac0 con 0x7f68140c7910 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.794+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c03e980 con 0x7f68140c7910 2024-02-19T14:54:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.794+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68141df0c0 con 0x7f68140c7910 2024-02-19T14:54:23.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.795+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68141dc460 con 0x7f68140c7910 2024-02-19T14:54:23.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.795+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f680c03eae0 con 0x7f68140c7910 2024-02-19T14:54:23.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.796+0000 7f67fb7fe640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f680006b840 0x7f680006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:23.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.796+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f680c0ad130 con 0x7f68140c7910 2024-02-19T14:54:23.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.796+0000 7f6812ffd640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f680006b840 0x7f680006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:23.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.796+0000 7f6812ffd640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f680006b840 0x7f680006dd00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f680400a830 tx=0x7f6804008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:23.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.800+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f680c077ec0 con 0x7f68140c7910 2024-02-19T14:54:23.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.971+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6814074990 con 0x7f68140c7910 2024-02-19T14:54:23.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.971+0000 7f67fb7fe640 1 -- 172.21.15.78:0/4188630500 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) v1 ==== 84+0+65 (secure 0 0 0) 0x7f680c07b580 con 0x7f68140c7910 2024-02-19T14:54:23.974 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:23.974 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:54:23.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:23.974 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:54:23.974 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:54:23.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:23.975 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:54:23.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f680006b840 msgr2=0x7f680006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f680006b840 0x7f680006dd00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f680400a830 tx=0x7f6804008040 comp rx=0 tx=0).stop 2024-02-19T14:54:23.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 msgr2=0x7f68141dccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 0x7f68141dccc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f680c009f40 tx=0x7f680c004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 shutdown_connections 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f680006b840 0x7f680006dd00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68140c7910 0x7f68141dccc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68140c5560 0x7f68141dc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 --2- 172.21.15.78:0/4188630500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6814070720 0x7f68140a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.973+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 >> 172.21.15.78:0/4188630500 conn(0x7f681406b4b0 msgr2=0x7f6814151850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.974+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 shutdown_connections 2024-02-19T14:54:23.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:23.974+0000 7f6819560640 1 -- 172.21.15.78:0/4188630500 wait complete. 2024-02-19T14:54:23.987 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:54:23.987 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:54:24.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.087+0000 7fc68ebe9640 1 Processor -- start 2024-02-19T14:54:24.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68ebe9640 1 -- start start 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68ebe9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc688152720 0x7fc6880bd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68ebe9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc6880c6600 0x7fc6880c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68ebe9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6880c8560 con 0x7fc688070ff0 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68ebe9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68806fa90 con 0x7fc688152720 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68ebe9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc688155ea0 con 0x7fc6880c6600 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68c985640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc688152720 0x7fc6880bd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68c985640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc688152720 0x7fc6880bd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35392/0 (socket says 172.21.15.78:35392) 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc68c985640 1 -- 172.21.15.78:0/1484154130 learned_addr learned my addr 172.21.15.78:0/1484154130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.088+0000 7fc687fff640 1 --1- 172.21.15.78:0/1484154130 >> v1:172.21.15.78:6789/0 conn(0x7fc688070ff0 0x7fc6880bdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59924/0 (socket says 172.21.15.78:59924) 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68d186640 1 --2- 172.21.15.78:0/1484154130 >> v2:172.21.15.150:3300/0 conn(0x7fc6880c6600 0x7fc6880c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68c985640 1 -- 172.21.15.78:0/1484154130 >> v2:172.21.15.150:3300/0 conn(0x7fc6880c6600 msgr2=0x7fc6880c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68c985640 1 --2- 172.21.15.78:0/1484154130 >> v2:172.21.15.150:3300/0 conn(0x7fc6880c6600 0x7fc6880c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68c985640 1 -- 172.21.15.78:0/1484154130 >> v1:172.21.15.78:6789/0 conn(0x7fc688070ff0 legacy=0x7fc6880bdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68c985640 1 -- 172.21.15.78:0/1484154130 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6880c80d0 con 0x7fc688152720 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc6877fe640 1 -- 172.21.15.78:0/1484154130 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599582838 0 0) 0x7fc674001110 con 0x7fc688070ff0 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68d186640 1 --2- 172.21.15.78:0/1484154130 >> v2:172.21.15.150:3300/0 conn(0x7fc6880c6600 0x7fc6880c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.089+0000 7fc68c985640 1 --2- 172.21.15.78:0/1484154130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc6880bd5e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc6880bf510 tx=0x7fc67c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1415592dbacea78d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc6877fe640 1 -- 172.21.15.78:0/1484154130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c031870 con 0x7fc688152720 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc6877fe640 1 -- 172.21.15.78:0/1484154130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67c0319d0 con 0x7fc688152720 2024-02-19T14:54:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc6877fe640 1 -- 172.21.15.78:0/1484154130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67c031d60 con 0x7fc688152720 2024-02-19T14:54:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 -- 172.21.15.78:0/1484154130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 msgr2=0x7fc6880bd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/1484154130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc6880bd5e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc6880bf510 tx=0x7fc67c02b910 comp rx=0 tx=0).stop 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 -- 172.21.15.78:0/1484154130 shutdown_connections 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/1484154130 >> v2:172.21.15.150:3300/0 conn(0x7fc6880c6600 0x7fc6880c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/1484154130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc6880bd5e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 -- 172.21.15.78:0/1484154130 >> 172.21.15.78:0/1484154130 conn(0x7fc6880aa370 msgr2=0x7fc6880aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 -- 172.21.15.78:0/1484154130 shutdown_connections 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.090+0000 7fc68ebe9640 1 -- 172.21.15.78:0/1484154130 wait complete. 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.091+0000 7fc68ebe9640 1 Processor -- start 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.091+0000 7fc68ebe9640 1 -- start start 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68ebe9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc688070ff0 0x7fc6880c5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68ebe9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6880c6600 0x7fc6880a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68ebe9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc68814b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68ebe9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880c2620 con 0x7fc688152720 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68ebe9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688156370 con 0x7fc6880c6600 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68ebe9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880c2870 con 0x7fc688070ff0 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc68814b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc68814b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35398/0 (socket says 172.21.15.78:35398) 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 -- 172.21.15.78:0/2784216804 learned_addr learned my addr 172.21.15.78:0/2784216804 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc687fff640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6880c6600 0x7fc6880a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68c985640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc688070ff0 0x7fc6880c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 -- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc688070ff0 msgr2=0x7fc6880c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc688070ff0 0x7fc6880c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 -- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6880c6600 msgr2=0x7fc6880a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6880c6600 0x7fc6880a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc68d186640 1 -- 172.21.15.78:0/2784216804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67c0094d0 con 0x7fc688152720 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.092+0000 7fc687fff640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6880c6600 0x7fc6880a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.093+0000 7fc68d186640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc68814b8b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc68000cef0 tx=0x7fc68000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.093+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68000d840 con 0x7fc688152720 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.093+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68000d9c0 con 0x7fc688152720 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.093+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc688158d50 con 0x7fc688152720 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.093+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68000dd70 con 0x7fc688152720 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.093+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6880aa0b0 con 0x7fc688152720 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.094+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc688159270 con 0x7fc688152720 2024-02-19T14:54:24.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.094+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc68001e3b0 con 0x7fc688152720 2024-02-19T14:54:24.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.095+0000 7fc6857fa640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66c06b810 0x7fc66c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.095+0000 7fc68c985640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66c06b810 0x7fc66c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.095+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc68008aab0 con 0x7fc688152720 2024-02-19T14:54:24.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.095+0000 7fc68c985640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66c06b810 0x7fc66c06dcd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc67c036000 tx=0x7fc67c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:24.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.099+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc680055750 con 0x7fc688152720 2024-02-19T14:54:24.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.272+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc688155920 con 0x7fc688152720 2024-02-19T14:54:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.714+0000 7fc6857fa640 1 -- 172.21.15.78:0/2784216804 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc68001a020 con 0x7fc688152720 2024-02-19T14:54:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:54:24.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66c06b810 msgr2=0x7fc66c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66c06b810 0x7fc66c06dcd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc67c036000 tx=0x7fc67c03c040 comp rx=0 tx=0).stop 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 msgr2=0x7fc68814b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc68814b8b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc68000cef0 tx=0x7fc68000a710 comp rx=0 tx=0).stop 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 shutdown_connections 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66c06b810 0x7fc66c06dcd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc688152720 0x7fc68814b8b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6880c6600 0x7fc6880a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 --2- 172.21.15.78:0/2784216804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc688070ff0 0x7fc6880c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 >> 172.21.15.78:0/2784216804 conn(0x7fc6880aa370 msgr2=0x7fc68814d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 shutdown_connections 2024-02-19T14:54:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.717+0000 7fc68ebe9640 1 -- 172.21.15.78:0/2784216804 wait complete. 2024-02-19T14:54:24.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:54:24.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.838+0000 7f21e73bc640 1 Processor -- start 2024-02-19T14:54:24.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.838+0000 7f21e73bc640 1 -- start start 2024-02-19T14:54:24.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e73bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e006e420 0x7f21e006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e73bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21e00c5930 0x7f21e00c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e73bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e00c48e0 con 0x7f21e0155540 2024-02-19T14:54:24.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e73bc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e00c62d0 con 0x7f21e00c5930 2024-02-19T14:54:24.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e73bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00c4b30 con 0x7f21e006e420 2024-02-19T14:54:24.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e4957640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21e00c5930 0x7f21e00c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.839+0000 7f21e5158640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e006e420 0x7f21e006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e5959640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f21e0155540 0x7f21e01559b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:45066/0 (socket says 172.21.15.78:45066) 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e5959640 1 -- 172.21.15.78:0/2107664353 learned_addr learned my addr 172.21.15.78:0/2107664353 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e5158640 1 -- 172.21.15.78:0/2107664353 >> v2:172.21.15.150:3300/0 conn(0x7f21e00c5930 msgr2=0x7f21e00c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e5158640 1 --2- 172.21.15.78:0/2107664353 >> v2:172.21.15.150:3300/0 conn(0x7f21e00c5930 0x7f21e00c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e5158640 1 -- 172.21.15.78:0/2107664353 >> v1:172.21.15.150:6789/0 conn(0x7f21e0155540 legacy=0x7f21e01559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e5158640 1 -- 172.21.15.78:0/2107664353 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e00c4e10 con 0x7f21e006e420 2024-02-19T14:54:24.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.840+0000 7f21e4957640 1 --2- 172.21.15.78:0/2107664353 >> v2:172.21.15.150:3300/0 conn(0x7f21e00c5930 0x7f21e00c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21e5158640 1 --2- 172.21.15.78:0/2107664353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e006e420 0x7f21e006e800 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0097a0 tx=0x7f21dc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f619728628fbca3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21d7fff640 1 -- 172.21.15.78:0/2107664353 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc02e070 con 0x7f21e006e420 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21d7fff640 1 -- 172.21.15.78:0/2107664353 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc0315a0 con 0x7f21e006e420 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21d7fff640 1 -- 172.21.15.78:0/2107664353 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc031930 con 0x7f21e006e420 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21e73bc640 1 -- 172.21.15.78:0/2107664353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e006e420 msgr2=0x7f21e006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2107664353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e006e420 0x7f21e006e800 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0097a0 tx=0x7f21dc02b840 comp rx=0 tx=0).stop 2024-02-19T14:54:24.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.841+0000 7f21e73bc640 1 -- 172.21.15.78:0/2107664353 shutdown_connections 2024-02-19T14:54:24.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.842+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2107664353 >> v2:172.21.15.150:3300/0 conn(0x7f21e00c5930 0x7f21e00c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.842+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2107664353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e006e420 0x7f21e006e800 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.842+0000 7f21e73bc640 1 -- 172.21.15.78:0/2107664353 >> 172.21.15.78:0/2107664353 conn(0x7f21e00c6d20 msgr2=0x7f21e006ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:24.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.842+0000 7f21e73bc640 1 -- 172.21.15.78:0/2107664353 shutdown_connections 2024-02-19T14:54:24.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.842+0000 7f21e73bc640 1 -- 172.21.15.78:0/2107664353 wait complete. 2024-02-19T14:54:24.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 Processor -- start 2024-02-19T14:54:24.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 -- start start 2024-02-19T14:54:24.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 0x7f21e00bf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e00c5930 0x7f21e01dc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0155540 0x7f21e01dc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e01e9710 con 0x7f21e006e420 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e01dcd80 con 0x7f21e0155540 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e73bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e01dee50 con 0x7f21e00c5930 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e5158640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 0x7f21e00bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e5158640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 0x7f21e00bf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35414/0 (socket says 172.21.15.78:35414) 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.843+0000 7f21e5158640 1 -- 172.21.15.78:0/2659259048 learned_addr learned my addr 172.21.15.78:0/2659259048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e4957640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e00c5930 0x7f21e01dc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5959640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0155540 0x7f21e01dc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5158640 1 -- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e00c5930 msgr2=0x7f21e01dc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5158640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e00c5930 0x7f21e01dc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5158640 1 -- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0155540 msgr2=0x7f21e01dc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:24.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5158640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0155540 0x7f21e01dc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5158640 1 -- 172.21.15.78:0/2659259048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0009400 con 0x7f21e006e420 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5959640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0155540 0x7f21e01dc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e5158640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 0x7f21e00bf600 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0098c0 tx=0x7f21dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc032840 con 0x7f21e006e420 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc031f00 con 0x7f21e006e420 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc009400 con 0x7f21e006e420 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc032be0 con 0x7f21e006e420 2024-02-19T14:54:24.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.844+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e00c0120 con 0x7f21e006e420 2024-02-19T14:54:24.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.845+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e00c2c20 con 0x7f21e006e420 2024-02-19T14:54:24.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.846+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f21dc032d40 con 0x7f21e006e420 2024-02-19T14:54:24.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.846+0000 7f21d5ffb640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b810 0x7f21c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:24.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.846+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f21dc0ac120 con 0x7f21e006e420 2024-02-19T14:54:24.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.847+0000 7f21e4957640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b810 0x7f21c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:24.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.847+0000 7f21e4957640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b810 0x7f21c406dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f21e00c3870 tx=0x7f21d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:24.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:24.851+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f21dc076b80 con 0x7f21e006e420 2024-02-19T14:54:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.023+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f21e0074990 con 0x7f21e006e420 2024-02-19T14:54:25.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.716+0000 7f21d5ffb640 1 -- 172.21.15.78:0/2659259048 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v73) v1 ==== 170+0+0 (secure 0 0 0) 0x7f21dc07a240 con 0x7f21e006e420 2024-02-19T14:54:25.719 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:54:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b810 msgr2=0x7f21c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b810 0x7f21c406dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f21e00c3870 tx=0x7f21d0008040 comp rx=0 tx=0).stop 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 msgr2=0x7f21e00bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 0x7f21e00bf600 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0098c0 tx=0x7f21dc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 shutdown_connections 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b810 0x7f21c406dcd0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21e0155540 0x7f21e01dc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21e00c5930 0x7f21e01dc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 --2- 172.21.15.78:0/2659259048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e006e420 0x7f21e00bf600 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 >> 172.21.15.78:0/2659259048 conn(0x7f21e00c6d20 msgr2=0x7f21e006dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 shutdown_connections 2024-02-19T14:54:25.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.719+0000 7f21e73bc640 1 -- 172.21.15.78:0/2659259048 wait complete. 2024-02-19T14:54:25.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:54:25.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:54:25.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:54:25.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.848+0000 7f0174d1e640 1 Processor -- start 2024-02-19T14:54:25.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.848+0000 7f0174d1e640 1 -- start start 2024-02-19T14:54:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.848+0000 7f0174d1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f01700c2f60 0x7f0170070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.848+0000 7f0174d1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01700ae670 con 0x7f01700add70 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.848+0000 7f0174d1e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01700716b0 con 0x7f01700c4980 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.848+0000 7f0174d1e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01700c3390 con 0x7f01700c2f60 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f01700c2f60 0x7f0170070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f01700c2f60 0x7f0170070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35420/0 (socket says 172.21.15.78:35420) 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f01700add70 0x7f017006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59928/0 (socket says 172.21.15.78:59928) 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016ed9d640 1 -- 172.21.15.78:0/3921346917 learned_addr learned my addr 172.21.15.78:0/3921346917 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016ed9d640 1 -- 172.21.15.78:0/3921346917 >> v1:172.21.15.150:6789/0 conn(0x7f01700c4980 legacy=0x7f01700705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016ed9d640 1 -- 172.21.15.78:0/3921346917 >> v1:172.21.15.78:6789/0 conn(0x7f01700add70 legacy=0x7f017006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016ed9d640 1 -- 172.21.15.78:0/3921346917 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0170153780 con 0x7f01700c2f60 2024-02-19T14:54:25.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.849+0000 7f016dd9b640 1 -- 172.21.15.78:0/3921346917 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768630571 0 0) 0x7f01700ae670 con 0x7f01700add70 2024-02-19T14:54:25.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f016ed9d640 1 --2- 172.21.15.78:0/3921346917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c2f60 0x7f0170070cc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0168009e50 tx=0x7f016802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe3c4295d3bc502 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:25.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f016dd9b640 1 -- 172.21.15.78:0/3921346917 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016802cb20 con 0x7f01700c2f60 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f016dd9b640 1 -- 172.21.15.78:0/3921346917 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016802cc80 con 0x7f01700c2f60 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f016dd9b640 1 -- 172.21.15.78:0/3921346917 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01680343f0 con 0x7f01700c2f60 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 -- 172.21.15.78:0/3921346917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c2f60 msgr2=0x7f0170070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 --2- 172.21.15.78:0/3921346917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c2f60 0x7f0170070cc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0168009e50 tx=0x7f016802bf80 comp rx=0 tx=0).stop 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 -- 172.21.15.78:0/3921346917 shutdown_connections 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 --2- 172.21.15.78:0/3921346917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c2f60 0x7f0170070cc0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 -- 172.21.15.78:0/3921346917 >> 172.21.15.78:0/3921346917 conn(0x7f01700c85e0 msgr2=0x7f01700c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 -- 172.21.15.78:0/3921346917 shutdown_connections 2024-02-19T14:54:25.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.850+0000 7f0174d1e640 1 -- 172.21.15.78:0/3921346917 wait complete. 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 Processor -- start 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 -- start start 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01700add70 0x7f01701e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01700c2f60 0x7f01700c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 0x7f01701418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01700bde40 con 0x7f01700c4980 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01701e9380 con 0x7f01700c2f60 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f0174d1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01700bd880 con 0x7f01700add70 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f016ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 0x7f01701418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:25.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f016ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 0x7f01701418b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35428/0 (socket says 172.21.15.78:35428) 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.852+0000 7f016ed9d640 1 -- 172.21.15.78:0/744375116 learned_addr learned my addr 172.21.15.78:0/744375116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0167fff640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01700c2f60 0x7f01700c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016e59c640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01700add70 0x7f01701e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016ed9d640 1 -- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01700add70 msgr2=0x7f01701e8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016ed9d640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01700add70 0x7f01701e8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016ed9d640 1 -- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01700c2f60 msgr2=0x7f01700c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016ed9d640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01700c2f60 0x7f01700c1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016ed9d640 1 -- 172.21.15.78:0/744375116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0168009b30 con 0x7f01700c4980 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0167fff640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01700c2f60 0x7f01700c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f016ed9d640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 0x7f01701418b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0168009e50 tx=0x7f0168004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0168007e70 con 0x7f01700c4980 2024-02-19T14:54:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0168009510 con 0x7f01700c4980 2024-02-19T14:54:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01700bdb60 con 0x7f01700c4980 2024-02-19T14:54:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016803e690 con 0x7f01700c4980 2024-02-19T14:54:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.853+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0170155460 con 0x7f01700c4980 2024-02-19T14:54:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.854+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0170155680 con 0x7f01700c4980 2024-02-19T14:54:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.855+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f016803e7f0 con 0x7f01700c4980 2024-02-19T14:54:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.855+0000 7f0166ffd640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f015006b810 0x7f015006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.855+0000 7f016e59c640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f015006b810 0x7f015006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.855+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f01680ace90 con 0x7f01700c4980 2024-02-19T14:54:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.855+0000 7f016e59c640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f015006b810 0x7f015006dcd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0160007fd0 tx=0x7f0160009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:25.860+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0168077810 con 0x7f01700c4980 2024-02-19T14:54:26.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.068+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0170074990 con 0x7f01700c4980 2024-02-19T14:54:26.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.734+0000 7f0166ffd640 1 -- 172.21.15.78:0/744375116 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v49) v1 ==== 174+0+0 (secure 0 0 0) 0x7f016807aed0 con 0x7f01700c4980 2024-02-19T14:54:26.738 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 11 and data pool 12 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.748+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f015006b810 msgr2=0x7f015006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.748+0000 7f0174d1e640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f015006b810 0x7f015006dcd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0160007fd0 tx=0x7f0160009040 comp rx=0 tx=0).stop 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.748+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 msgr2=0x7f01701418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.748+0000 7f0174d1e640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 0x7f01701418b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0168009e50 tx=0x7f0168004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.749+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 shutdown_connections 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.749+0000 7f0174d1e640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f015006b810 0x7f015006dcd0 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0160007fd0 tx=0x7f0160009040 comp rx=0 tx=0).stop 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.749+0000 7f0174d1e640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01700c4980 0x7f01701418b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.749+0000 7f0174d1e640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01700c2f60 0x7f01700c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.749+0000 7f0174d1e640 1 --2- 172.21.15.78:0/744375116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01700add70 0x7f01701e8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.749+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 >> 172.21.15.78:0/744375116 conn(0x7f01700c85e0 msgr2=0x7f0170150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.750+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 shutdown_connections 2024-02-19T14:54:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.750+0000 7f0174d1e640 1 -- 172.21.15.78:0/744375116 wait complete. 2024-02-19T14:54:26.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:26.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.871+0000 7f190cef4640 1 Processor -- start 2024-02-19T14:54:26.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.871+0000 7f190cef4640 1 -- start start 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190cef4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19081528d0 0x7f19080c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190cef4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19080c19c0 0x7f1908155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190cef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f19081555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190cef4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19080734c0 con 0x7f19081528d0 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190cef4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1908155af0 con 0x7f19080c19c0 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190cef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1908072e80 con 0x7f1908072180 2024-02-19T14:54:26.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19081528d0 0x7f19080c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19081528d0 0x7f19080c1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35430/0 (socket says 172.21.15.78:35430) 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 -- 172.21.15.78:0/423615274 learned_addr learned my addr 172.21.15.78:0/423615274 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 -- 172.21.15.78:0/423615274 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 msgr2=0x7f19081555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 --2- 172.21.15.78:0/423615274 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f19081555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 -- 172.21.15.78:0/423615274 >> v2:172.21.15.150:3300/0 conn(0x7f19080c19c0 msgr2=0x7f1908155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 --2- 172.21.15.78:0/423615274 >> v2:172.21.15.150:3300/0 conn(0x7f19080c19c0 0x7f1908155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.872+0000 7f190659c640 1 -- 172.21.15.78:0/423615274 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190806e990 con 0x7f19081528d0 2024-02-19T14:54:26.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.873+0000 7f190659c640 1 --2- 172.21.15.78:0/423615274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080c1480 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1908070310 tx=0x7f18f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27a070048deb008 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:26.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.873+0000 7f190559a640 1 -- 172.21.15.78:0/423615274 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f802c9e0 con 0x7f19081528d0 2024-02-19T14:54:26.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.873+0000 7f190559a640 1 -- 172.21.15.78:0/423615274 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f802cb40 con 0x7f19081528d0 2024-02-19T14:54:26.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.873+0000 7f190559a640 1 -- 172.21.15.78:0/423615274 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f80343f0 con 0x7f19081528d0 2024-02-19T14:54:26.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 -- 172.21.15.78:0/423615274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 msgr2=0x7f19080c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:26.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 --2- 172.21.15.78:0/423615274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080c1480 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1908070310 tx=0x7f18f802be40 comp rx=0 tx=0).stop 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 -- 172.21.15.78:0/423615274 shutdown_connections 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 --2- 172.21.15.78:0/423615274 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f19081555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 --2- 172.21.15.78:0/423615274 >> v2:172.21.15.150:3300/0 conn(0x7f19080c19c0 0x7f1908155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 --2- 172.21.15.78:0/423615274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080c1480 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 -- 172.21.15.78:0/423615274 >> 172.21.15.78:0/423615274 conn(0x7f19080c7590 msgr2=0x7f19080c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 -- 172.21.15.78:0/423615274 shutdown_connections 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.874+0000 7f190cef4640 1 -- 172.21.15.78:0/423615274 wait complete. 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.875+0000 7f190cef4640 1 Processor -- start 2024-02-19T14:54:26.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.875+0000 7f190cef4640 1 -- start start 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190cef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f1908154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190cef4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19080c19c0 0x7f19080a8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190cef4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190cef4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19081ea6b0 con 0x7f19081528d0 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190cef4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19080a82c0 con 0x7f19080c19c0 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190cef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19080ad330 con 0x7f1908072180 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:26.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080a94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35436/0 (socket says 172.21.15.78:35436) 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 -- 172.21.15.78:0/2284654924 learned_addr learned my addr 172.21.15.78:0/2284654924 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f190659c640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f1908154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1905d9b640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19080c19c0 0x7f19080a8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 -- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 msgr2=0x7f1908154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f1908154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 -- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19080c19c0 msgr2=0x7f19080a8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19080c19c0 0x7f19080a8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 -- 172.21.15.78:0/2284654924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f8009b30 con 0x7f19081528d0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1905d9b640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19080c19c0 0x7f19080a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.876+0000 7f1906d9d640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080a94f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f190000b410 tx=0x7f190000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.877+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1900011830 con 0x7f19081528d0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.877+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19000119b0 con 0x7f19081528d0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.877+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19081ea8c0 con 0x7f19081528d0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.877+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1900011d60 con 0x7f19081528d0 2024-02-19T14:54:26.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.877+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19080a87b0 con 0x7f19081528d0 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.878+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190806ee00 con 0x7f19081528d0 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.878+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f190001a400 con 0x7f19081528d0 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.878+0000 7f18f6ffd640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e806b840 0x7f18e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.878+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f190008a3b0 con 0x7f19081528d0 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.878+0000 7f190659c640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e806b840 0x7f18e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.879+0000 7f190659c640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e806b840 0x7f18e806dd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f18f8035000 tx=0x7f18f8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:26.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:26.883+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f190806ee00 con 0x7f19081528d0 2024-02-19T14:54:27.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.054+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f190806ee00 con 0x7f19081528d0 2024-02-19T14:54:27.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.055+0000 7f18f6ffd640 1 -- 172.21.15.78:0/2284654924 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16946 (secure 0 0 0) 0x7f190806ee00 con 0x7f19081528d0 2024-02-19T14:54:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:27.058 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":74,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:26.069756+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e806b840 msgr2=0x7f18e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e806b840 0x7f18e806dd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f18f8035000 tx=0x7f18f8030000 comp rx=0 tx=0).stop 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 msgr2=0x7f19080a94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080a94f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f190000b410 tx=0x7f190000b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 shutdown_connections 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e806b840 0x7f18e806dd00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19081528d0 0x7f19080a94f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19080c19c0 0x7f19080a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 --2- 172.21.15.78:0/2284654924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1908072180 0x7f1908154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 >> 172.21.15.78:0/2284654924 conn(0x7f19080c7590 msgr2=0x7f19080c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 shutdown_connections 2024-02-19T14:54:27.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.057+0000 7f190cef4640 1 -- 172.21.15.78:0/2284654924 wait complete. 2024-02-19T14:54:27.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:27.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.168+0000 7fbe1273e640 1 Processor -- start 2024-02-19T14:54:27.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.168+0000 7fbe1273e640 1 -- start start 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe1273e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe0c0c3200 0x7fbe0c0ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe1273e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0c4350 0x7fbe0c06ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe1273e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0c35e0 con 0x7fbe0c153b60 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe1273e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c14d3c0 con 0x7fbe0c0c4350 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe1273e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c0acad0 con 0x7fbe0c0c3200 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0c4350 0x7fbe0c06ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0c4350 0x7fbe0c06ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35440/0 (socket says 172.21.15.78:35440) 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 -- 172.21.15.78:0/1147782130 learned_addr learned my addr 172.21.15.78:0/1147782130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe0bfff640 1 --2- 172.21.15.78:0/1147782130 >> v2:172.21.15.150:3300/0 conn(0x7fbe0c0c3200 0x7fbe0c0ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 -- 172.21.15.78:0/1147782130 >> v2:172.21.15.150:3300/0 conn(0x7fbe0c0c3200 msgr2=0x7fbe0c0ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 --2- 172.21.15.78:0/1147782130 >> v2:172.21.15.150:3300/0 conn(0x7fbe0c0c3200 0x7fbe0c0ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 -- 172.21.15.78:0/1147782130 >> v1:172.21.15.150:6789/0 conn(0x7fbe0c153b60 legacy=0x7fbe0c0718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.169+0000 7fbe10cdb640 1 -- 172.21.15.78:0/1147782130 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0ad000 con 0x7fbe0c0c4350 2024-02-19T14:54:27.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.170+0000 7fbe10cdb640 1 --2- 172.21.15.78:0/1147782130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c06ae80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c072ca0 tx=0x7fbe0402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92690f461534f37 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.170+0000 7fbe0affd640 1 -- 172.21.15.78:0/1147782130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0402cb20 con 0x7fbe0c0c4350 2024-02-19T14:54:27.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.170+0000 7fbe0affd640 1 -- 172.21.15.78:0/1147782130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0402cc80 con 0x7fbe0c0c4350 2024-02-19T14:54:27.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.170+0000 7fbe0affd640 1 -- 172.21.15.78:0/1147782130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe040343f0 con 0x7fbe0c0c4350 2024-02-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 -- 172.21.15.78:0/1147782130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c4350 msgr2=0x7fbe0c06ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 --2- 172.21.15.78:0/1147782130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c06ae80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c072ca0 tx=0x7fbe0402bf80 comp rx=0 tx=0).stop 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 -- 172.21.15.78:0/1147782130 shutdown_connections 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 --2- 172.21.15.78:0/1147782130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c06ae80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 --2- 172.21.15.78:0/1147782130 >> v2:172.21.15.150:3300/0 conn(0x7fbe0c0c3200 0x7fbe0c0ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 -- 172.21.15.78:0/1147782130 >> 172.21.15.78:0/1147782130 conn(0x7fbe0c0c7800 msgr2=0x7fbe0c0c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 -- 172.21.15.78:0/1147782130 shutdown_connections 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.171+0000 7fbe1273e640 1 -- 172.21.15.78:0/1147782130 wait complete. 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.172+0000 7fbe1273e640 1 Processor -- start 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.172+0000 7fbe1273e640 1 -- start start 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe1273e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 0x7fbe0c154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe1273e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe1273e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe0c153b60 0x7fbe0c1e9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe1273e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c1e8bc0 con 0x7fbe0c0c3200 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe1273e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c1562f0 con 0x7fbe0c0c4350 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe1273e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c155d30 con 0x7fbe0c153b60 2024-02-19T14:54:27.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 0x7fbe0c154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 0x7fbe0c154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35444/0 (socket says 172.21.15.78:35444) 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 -- 172.21.15.78:0/3515733247 learned_addr learned my addr 172.21.15.78:0/3515733247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0b7fe640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe10cdb640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe0c153b60 0x7fbe0c1e9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 -- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe0c153b60 msgr2=0x7fbe0c1e9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe0c153b60 0x7fbe0c1e9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 -- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0c4350 msgr2=0x7fbe0c147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 -- 172.21.15.78:0/3515733247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04009b30 con 0x7fbe0c0c3200 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe10cdb640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe0c153b60 0x7fbe0c1e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0b7fe640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.173+0000 7fbe0bfff640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 0x7fbe0c154110 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00d5f0 tx=0x7fbdfc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.174+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc0125f0 con 0x7fbe0c0c3200 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.174+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdfc012750 con 0x7fbe0c0c3200 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.174+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c156010 con 0x7fbe0c0c3200 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.174+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc012b00 con 0x7fbe0c0c3200 2024-02-19T14:54:27.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.174+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c1483b0 con 0x7fbe0c0c3200 2024-02-19T14:54:27.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.175+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c06c850 con 0x7fbe0c0c3200 2024-02-19T14:54:27.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.175+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbdfc012c60 con 0x7fbe0c0c3200 2024-02-19T14:54:27.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.175+0000 7fbe08ff9640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdf806b810 0x7fbdf806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.175+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbdfc08a7b0 con 0x7fbe0c0c3200 2024-02-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.175+0000 7fbe0b7fe640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdf806b810 0x7fbdf806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.176+0000 7fbe0b7fe640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdf806b810 0x7fbdf806dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0c5a20 tx=0x7fbe00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.180+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbe0c06c850 con 0x7fbe0c0c3200 2024-02-19T14:54:27.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.352+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe0c06c850 con 0x7fbe0c0c3200 2024-02-19T14:54:27.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.353+0000 7fbe08ff9640 1 -- 172.21.15.78:0/3515733247 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16946 (secure 0 0 0) 0x7fbe0c06c850 con 0x7fbe0c0c3200 2024-02-19T14:54:27.356 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:27.356 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":74,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:26.069756+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdf806b810 msgr2=0x7fbdf806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdf806b810 0x7fbdf806dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0c5a20 tx=0x7fbe00009040 comp rx=0 tx=0).stop 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 msgr2=0x7fbe0c154110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 0x7fbe0c154110 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00d5f0 tx=0x7fbdfc00dab0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 shutdown_connections 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdf806b810 0x7fbdf806dcd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe0c153b60 0x7fbe0c1e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0c4350 0x7fbe0c147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 --2- 172.21.15.78:0/3515733247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0c3200 0x7fbe0c154110 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 >> 172.21.15.78:0/3515733247 conn(0x7fbe0c0c7800 msgr2=0x7fbe0c0718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:27.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 shutdown_connections 2024-02-19T14:54:27.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.355+0000 7fbe1273e640 1 -- 172.21.15.78:0/3515733247 wait complete. 2024-02-19T14:54:27.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:54:27.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.469+0000 7fd48d2cd640 1 Processor -- start 2024-02-19T14:54:27.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.469+0000 7fd48d2cd640 1 -- start start 2024-02-19T14:54:27.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd48d2cd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd488152700 0x7fd4880c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd48d2cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4880c5730 0x7fd48806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd48d2cd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4881551f0 con 0x7fd4880c8e00 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd48d2cd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd488155490 con 0x7fd488152700 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd48d2cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd488155770 con 0x7fd4880c5730 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd488152700 0x7fd4880c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd488152700 0x7fd4880c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38840/0 (socket says 172.21.15.78:38840) 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 -- 172.21.15.78:0/3948351460 learned_addr learned my addr 172.21.15.78:0/3948351460 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 -- 172.21.15.78:0/3948351460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4880c5730 msgr2=0x7fd48806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 --2- 172.21.15.78:0/3948351460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4880c5730 0x7fd48806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 -- 172.21.15.78:0/3948351460 >> v1:172.21.15.150:6789/0 conn(0x7fd4880c8e00 legacy=0x7fd488158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:27.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.470+0000 7fd486ffd640 1 -- 172.21.15.78:0/3948351460 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd488155a50 con 0x7fd488152700 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.471+0000 7fd486ffd640 1 --2- 172.21.15.78:0/3948351460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd488152700 0x7fd4880c51f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd478009870 tx=0x7fd47802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f634a15119c6e318 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.471+0000 7fd4867fc640 1 -- 172.21.15.78:0/3948351460 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478031640 con 0x7fd488152700 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.471+0000 7fd4867fc640 1 -- 172.21.15.78:0/3948351460 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4780317a0 con 0x7fd488152700 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.471+0000 7fd4867fc640 1 -- 172.21.15.78:0/3948351460 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd478031b30 con 0x7fd488152700 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.471+0000 7fd48d2cd640 1 -- 172.21.15.78:0/3948351460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd488152700 msgr2=0x7fd4880c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.471+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/3948351460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd488152700 0x7fd4880c51f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd478009870 tx=0x7fd47802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.472+0000 7fd48d2cd640 1 -- 172.21.15.78:0/3948351460 shutdown_connections 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.472+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/3948351460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4880c5730 0x7fd48806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.472+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/3948351460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd488152700 0x7fd4880c51f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.472+0000 7fd48d2cd640 1 -- 172.21.15.78:0/3948351460 >> 172.21.15.78:0/3948351460 conn(0x7fd4880cac90 msgr2=0x7fd4880cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.472+0000 7fd48d2cd640 1 -- 172.21.15.78:0/3948351460 shutdown_connections 2024-02-19T14:54:27.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.472+0000 7fd48d2cd640 1 -- 172.21.15.78:0/3948351460 wait complete. 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 Processor -- start 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 -- start start 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4880c5730 0x7fd4881566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 0x7fd4880ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd488152700 0x7fd4880ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880c1fc0 con 0x7fd4880c8e00 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880ab020 con 0x7fd4880c5730 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd48d2cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880aa6f0 con 0x7fd488152700 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd47ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 0x7fd4880ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd47ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 0x7fd4880ab4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38850/0 (socket says 172.21.15.78:38850) 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.473+0000 7fd47ffff640 1 -- 172.21.15.78:0/1183415048 learned_addr learned my addr 172.21.15.78:0/1183415048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd4877fe640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd488152700 0x7fd4880ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd486ffd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4880c5730 0x7fd4881566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47ffff640 1 -- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd488152700 msgr2=0x7fd4880ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47ffff640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd488152700 0x7fd4880ab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47ffff640 1 -- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4880c5730 msgr2=0x7fd4881566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47ffff640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4880c5730 0x7fd4881566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47ffff640 1 -- 172.21.15.78:0/1183415048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780094d0 con 0x7fd4880c8e00 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47ffff640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 0x7fd4880ab4b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd47000b4a0 tx=0x7fd47000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470010610 con 0x7fd4880c8e00 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd470010770 con 0x7fd4880c8e00 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4880aa9d0 con 0x7fd4880c8e00 2024-02-19T14:54:27.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470010b20 con 0x7fd4880c8e00 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.474+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488156c00 con 0x7fd4880c8e00 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.475+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4880ab1b0 con 0x7fd4880c8e00 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.475+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd470010c80 con 0x7fd4880c8e00 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.476+0000 7fd47f7fe640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd46806b810 0x7fd46806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.476+0000 7fd486ffd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd46806b810 0x7fd46806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.476+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd470089f90 con 0x7fd4880c8e00 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.476+0000 7fd486ffd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd46806b810 0x7fd46806dcd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd478036000 tx=0x7fd47803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.480+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4700bddc0 con 0x7fd4880c8e00 2024-02-19T14:54:27.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.689+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd488074990 con 0x7fd4880c8e00 2024-02-19T14:54:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.743+0000 7fd47f7fe640 1 -- 172.21.15.78:0/1183415048 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd47001e0b0 con 0x7fd4880c8e00 2024-02-19T14:54:27.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd46806b810 msgr2=0x7fd46806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd46806b810 0x7fd46806dcd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd478036000 tx=0x7fd47803c040 comp rx=0 tx=0).stop 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 msgr2=0x7fd4880ab4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 0x7fd4880ab4b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd47000b4a0 tx=0x7fd47000b960 comp rx=0 tx=0).stop 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 shutdown_connections 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd46806b810 0x7fd46806dcd0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4880c8e00 0x7fd4880ab4b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd488152700 0x7fd4880ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 --2- 172.21.15.78:0/1183415048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4880c5730 0x7fd4881566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 >> 172.21.15.78:0/1183415048 conn(0x7fd4880cac90 msgr2=0x7fd4881516f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 shutdown_connections 2024-02-19T14:54:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.746+0000 7fd48d2cd640 1 -- 172.21.15.78:0/1183415048 wait complete. 2024-02-19T14:54:27.759 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:54:27.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:54:27.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 Processor -- start 2024-02-19T14:54:27.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 -- start start 2024-02-19T14:54:27.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb048152480 0x7fb0481559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb04806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb048155ef0 con 0x7fb04806d350 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04806dec0 con 0x7fb048152480 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.869+0000 7fb04e6d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480c75c0 con 0x7fb0480c7070 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb048152480 0x7fb0481559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb048152480 0x7fb0481559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38864/0 (socket says 172.21.15.78:38864) 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 -- 172.21.15.78:0/724053056 learned_addr learned my addr 172.21.15.78:0/724053056 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 -- 172.21.15.78:0/724053056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 msgr2=0x7fb04806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb0477fe640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb04806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb04806ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 -- 172.21.15.78:0/724053056 >> v1:172.21.15.150:6789/0 conn(0x7fb04806d350 legacy=0x7fb0480c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb047fff640 1 -- 172.21.15.78:0/724053056 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480c78a0 con 0x7fb048152480 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.870+0000 7fb0477fe640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb04806ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb047fff640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481559b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb0380098a0 tx=0x7fb03802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d05fb4c535525d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb046ffd640 1 -- 172.21.15.78:0/724053056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038030840 con 0x7fb048152480 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb046ffd640 1 -- 172.21.15.78:0/724053056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0380309a0 con 0x7fb048152480 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb046ffd640 1 -- 172.21.15.78:0/724053056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038030d30 con 0x7fb048152480 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 -- 172.21.15.78:0/724053056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 msgr2=0x7fb0481559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481559b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb0380098a0 tx=0x7fb03802b980 comp rx=0 tx=0).stop 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 -- 172.21.15.78:0/724053056 shutdown_connections 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb04806ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/724053056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481559b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 -- 172.21.15.78:0/724053056 >> 172.21.15.78:0/724053056 conn(0x7fb0480ca8a0 msgr2=0x7fb0480cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 -- 172.21.15.78:0/724053056 shutdown_connections 2024-02-19T14:54:27.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.871+0000 7fb04e6d4640 1 -- 172.21.15.78:0/724053056 wait complete. 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 Processor -- start 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 -- start start 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb0481eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481ee980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0480c3550 0x7fb048149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480c2a30 con 0x7fb048152480 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0481eeec0 con 0x7fb0480c3550 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb04e6d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0480c20d0 con 0x7fb0480c7070 2024-02-19T14:54:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.873+0000 7fb0477fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481ee980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38876/0 (socket says 172.21.15.78:38876) 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 -- 172.21.15.78:0/2478126972 learned_addr learned my addr 172.21.15.78:0/2478126972 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb047fff640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb0481eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb04cc71640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0480c3550 0x7fb048149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 -- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 msgr2=0x7fb0481eb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb0481eb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 -- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0480c3550 msgr2=0x7fb048149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0480c3550 0x7fb048149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 -- 172.21.15.78:0/2478126972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0380094d0 con 0x7fb048152480 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb047fff640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb0481eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb0477fe640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481ee980 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00c7f0 tx=0x7fb03c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00cea0 con 0x7fb048152480 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c004230 con 0x7fb048152480 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0481eb030 con 0x7fb048152480 2024-02-19T14:54:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c007830 con 0x7fb048152480 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.874+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0480c1b10 con 0x7fb048152480 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.875+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0481eb1c0 con 0x7fb048152480 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.876+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb03c007990 con 0x7fb048152480 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.876+0000 7fb044ff9640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b840 0x7fb03406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.877+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb03c08a200 con 0x7fb048152480 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.877+0000 7fb047fff640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b840 0x7fb03406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.877+0000 7fb047fff640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b840 0x7fb03406dd00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb038036000 tx=0x7fb03803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:27.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:27.881+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb03c054b20 con 0x7fb048152480 2024-02-19T14:54:28.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.096+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb048074990 con 0x7fb048152480 2024-02-19T14:54:28.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.748+0000 7fb044ff9640 1 -- 172.21.15.78:0/2478126972 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb03c0581e0 con 0x7fb048152480 2024-02-19T14:54:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b840 msgr2=0x7fb03406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b840 0x7fb03406dd00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb038036000 tx=0x7fb03803b040 comp rx=0 tx=0).stop 2024-02-19T14:54:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 msgr2=0x7fb0481ee980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481ee980 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00c7f0 tx=0x7fb03c00ccb0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 shutdown_connections 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03406b840 0x7fb03406dd00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb048152480 0x7fb0481ee980 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0480c3550 0x7fb048149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 --2- 172.21.15.78:0/2478126972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0480c7070 0x7fb0481eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 >> 172.21.15.78:0/2478126972 conn(0x7fb0480ca8a0 msgr2=0x7fb048153940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 shutdown_connections 2024-02-19T14:54:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.750+0000 7fb04e6d4640 1 -- 172.21.15.78:0/2478126972 wait complete. 2024-02-19T14:54:28.764 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:54:28.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.866+0000 7f734462a640 1 Processor -- start 2024-02-19T14:54:28.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.866+0000 7f734462a640 1 -- start start 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f734462a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c0c2f60 0x7f733c070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f734462a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c0ae670 con 0x7f733c0c4980 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f734462a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c0716b0 con 0x7f733c0add70 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f734462a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0c33b0 con 0x7f733c0c2f60 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f7341bc5640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f733c0c4980 0x7f733c0705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39638/0 (socket says 172.21.15.78:39638) 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f7341bc5640 1 -- 172.21.15.78:0/4213136184 learned_addr learned my addr 172.21.15.78:0/4213136184 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f7342bc7640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c0c2f60 0x7f733c070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514492946 0 0) 0x7f733c0ae670 con 0x7f733c0c4980 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.867+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7328003520 con 0x7f733c0c4980 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4226040608 0 0) 0x7f7328003520 con 0x7f733c0c4980 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c0c2f60 msgr2=0x7f733c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c0c2f60 0x7f733c070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 >> v1:172.21.15.150:6789/0 conn(0x7f733c0add70 legacy=0x7f733c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c153780 con 0x7f733c0c4980 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f732c001670 con 0x7f733c0c4980 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 >> v1:172.21.15.78:6789/0 conn(0x7f733c0c4980 legacy=0x7f733c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73280039a0 0x7f7328003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7328004350 0x7f7328004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73280050c0 0x7f732800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0c33b0 con 0x7f73280039a0 2024-02-19T14:54:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732800bfa0 con 0x7f7328004350 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732800c160 con 0x7f73280050c0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f7342bc7640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c0c2f60 0x7f733c070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73423c6640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73280039a0 0x7f7328003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f7341bc5640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7328004350 0x7f7328004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f7342bc7640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73280050c0 0x7f732800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73423c6640 1 -- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73280050c0 msgr2=0x7f732800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73423c6640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73280050c0 0x7f732800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73423c6640 1 -- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7328004350 msgr2=0x7f7328004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73423c6640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7328004350 0x7f7328004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.868+0000 7f73423c6640 1 -- 172.21.15.78:0/4213136184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328003520 con 0x7f73280039a0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f73423c6640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73280039a0 0x7f7328003d80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7338001860 tx=0x7f73280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f68381f88d90b88 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338033630 con 0x7f73280039a0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73380337b0 con 0x7f73280039a0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f73413c4640 1 -- 172.21.15.78:0/4213136184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338033b60 con 0x7f73280039a0 2024-02-19T14:54:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f734462a640 1 -- 172.21.15.78:0/4213136184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73280039a0 msgr2=0x7f7328003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f734462a640 1 --2- 172.21.15.78:0/4213136184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73280039a0 0x7f7328003d80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7338001860 tx=0x7f73280057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.869+0000 7f7342bc7640 1 -- 172.21.15.78:0/4213136184 reap_dead start 2024-02-19T14:54:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.870+0000 7f734462a640 1 -- 172.21.15.78:0/4213136184 shutdown_connections 2024-02-19T14:54:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.870+0000 7f734462a640 1 -- 172.21.15.78:0/4213136184 >> 172.21.15.78:0/4213136184 conn(0x7f733c0c85e0 msgr2=0x7f733c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.870+0000 7f734462a640 1 -- 172.21.15.78:0/4213136184 shutdown_connections 2024-02-19T14:54:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.870+0000 7f734462a640 1 -- 172.21.15.78:0/4213136184 wait complete. 2024-02-19T14:54:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 Processor -- start 2024-02-19T14:54:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 -- start start 2024-02-19T14:54:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73280039a0 0x7f733c0c1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c06db30 0x7f733c1eaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 0x7f733c1edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1550f0 con 0x7f733c1eafb0 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c156ef0 con 0x7f73280039a0 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f734462a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1ea410 con 0x7f733c06db30 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f7342bc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 0x7f733c1edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f7342bc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 0x7f733c1edcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38900/0 (socket says 172.21.15.78:38900) 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.871+0000 7f7342bc7640 1 -- 172.21.15.78:0/3164608226 learned_addr learned my addr 172.21.15.78:0/3164608226 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7341bc5640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c06db30 0x7f733c1eaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f73423c6640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73280039a0 0x7f733c0c1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7342bc7640 1 -- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c06db30 msgr2=0x7f733c1eaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7342bc7640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c06db30 0x7f733c1eaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7342bc7640 1 -- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73280039a0 msgr2=0x7f733c0c1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:28.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7342bc7640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73280039a0 0x7f733c0c1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7342bc7640 1 -- 172.21.15.78:0/3164608226 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c1562e0 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7341bc5640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c06db30 0x7f733c1eaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f73423c6640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73280039a0 0x7f733c0c1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7342bc7640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 0x7f733c1edcb0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f733c06e600 tx=0x7f7334011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733400e070 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7334003990 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c1565c0 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7334003b40 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.872+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c156b80 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.873+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c06e140 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.873+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7334003ca0 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.874+0000 7f7332ffd640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f732006b840 0x7f732006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.874+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f733408ab60 con 0x7f733c1eafb0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.874+0000 7f73423c6640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f732006b840 0x7f732006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:28.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.874+0000 7f73423c6640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f732006b840 0x7f732006dd00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7338036000 tx=0x7f7338031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:28.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:28.882+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7334055480 con 0x7f733c1eafb0 2024-02-19T14:54:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.095+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f733c074990 con 0x7f733c1eafb0 2024-02-19T14:54:29.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.096+0000 7f7332ffd640 1 -- 172.21.15.78:0/3164608226 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f7334058b40 con 0x7f733c1eafb0 2024-02-19T14:54:29.098 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:29.099 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:28.097183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2137063883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":2137063883},{"type":"v1","addr":"172.21.15.78:6837","nonce":2137063883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T14:54:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 51 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f732006b840 msgr2=0x7f732006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f732006b840 0x7f732006dd00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7338036000 tx=0x7f7338031000 comp rx=0 tx=0).stop 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 msgr2=0x7f733c1edcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 0x7f733c1edcb0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f733c06e600 tx=0x7f7334011890 comp rx=0 tx=0).stop 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 shutdown_connections 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f732006b840 0x7f732006dd00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f733c1eafb0 0x7f733c1edcb0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c06db30 0x7f733c1eaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 --2- 172.21.15.78:0/3164608226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73280039a0 0x7f733c0c1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 >> 172.21.15.78:0/3164608226 conn(0x7f733c0c85e0 msgr2=0x7f733c0c7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:29.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 shutdown_connections 2024-02-19T14:54:29.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.098+0000 7f734462a640 1 -- 172.21.15.78:0/3164608226 wait complete. 2024-02-19T14:54:29.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.210+0000 7fe7478cd640 1 Processor -- start 2024-02-19T14:54:29.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.210+0000 7fe7478cd640 1 -- start start 2024-02-19T14:54:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe7478cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7400c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe7478cd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7400c8e00 0x7fe74015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe7478cd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7401551f0 con 0x7fe7400c5730 2024-02-19T14:54:29.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe7478cd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe740155490 con 0x7fe7400c8e00 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe7478cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe740155770 con 0x7fe740152700 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe745e6a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7400c8e00 0x7fe74015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe745e6a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7400c8e00 0x7fe74015f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38908/0 (socket says 172.21.15.78:38908) 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe744e68640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe7400c5730 0x7fe74006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39650/0 (socket says 172.21.15.78:39650) 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.211+0000 7fe745e6a640 1 -- 172.21.15.78:0/485376632 learned_addr learned my addr 172.21.15.78:0/485376632 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe745669640 1 --2- 172.21.15.78:0/485376632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7400c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe745e6a640 1 -- 172.21.15.78:0/485376632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 msgr2=0x7fe7400c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe745e6a640 1 --2- 172.21.15.78:0/485376632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7400c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe745e6a640 1 -- 172.21.15.78:0/485376632 >> v1:172.21.15.78:6789/0 conn(0x7fe7400c5730 legacy=0x7fe74006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe737fff640 1 -- 172.21.15.78:0/485376632 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869804218 0 0) 0x7fe7401551f0 con 0x7fe7400c5730 2024-02-19T14:54:29.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe745e6a640 1 -- 172.21.15.78:0/485376632 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740155a50 con 0x7fe7400c8e00 2024-02-19T14:54:29.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.212+0000 7fe745e6a640 1 --2- 172.21.15.78:0/485376632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c8e00 0x7fe74015f980 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe738009e50 tx=0x7fe73802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b539760eca22bc9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe737fff640 1 -- 172.21.15.78:0/485376632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73802cb00 con 0x7fe7400c8e00 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe737fff640 1 -- 172.21.15.78:0/485376632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73802cc60 con 0x7fe7400c8e00 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe737fff640 1 -- 172.21.15.78:0/485376632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7380343f0 con 0x7fe7400c8e00 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 -- 172.21.15.78:0/485376632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c8e00 msgr2=0x7fe74015f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 --2- 172.21.15.78:0/485376632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c8e00 0x7fe74015f980 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe738009e50 tx=0x7fe73802bf60 comp rx=0 tx=0).stop 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 -- 172.21.15.78:0/485376632 shutdown_connections 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 --2- 172.21.15.78:0/485376632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c8e00 0x7fe74015f980 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 --2- 172.21.15.78:0/485376632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 -- 172.21.15.78:0/485376632 >> 172.21.15.78:0/485376632 conn(0x7fe7400cac90 msgr2=0x7fe7400cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 -- 172.21.15.78:0/485376632 shutdown_connections 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.213+0000 7fe7478cd640 1 -- 172.21.15.78:0/485376632 wait complete. 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.214+0000 7fe7478cd640 1 Processor -- start 2024-02-19T14:54:29.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.214+0000 7fe7478cd640 1 -- start start 2024-02-19T14:54:29.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe7478cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 0x7fe7400c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe7478cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7400c8e00 0x7fe7400aa750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe7478cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7401e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe7478cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7401ee950 con 0x7fe7400c5730 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe7478cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7400c28d0 con 0x7fe7400c8e00 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe7478cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7401ee670 con 0x7fe740152700 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 0x7fe7400c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 0x7fe7400c2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38910/0 (socket says 172.21.15.78:38910) 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 -- 172.21.15.78:0/1830186190 learned_addr learned my addr 172.21.15.78:0/1830186190 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745e6a640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7401e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe744e68640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7400c8e00 0x7fe7400aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 -- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 msgr2=0x7fe7401e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7401e7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 -- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7400c8e00 msgr2=0x7fe7400aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7400c8e00 0x7fe7400aa750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe745669640 1 -- 172.21.15.78:0/1830186190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738009b30 con 0x7fe7400c5730 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.215+0000 7fe744e68640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7400c8e00 0x7fe7400aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.216+0000 7fe745669640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 0x7fe7400c2cf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00d5d0 tx=0x7fe73c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.216+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c0125f0 con 0x7fe7400c5730 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.216+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c012770 con 0x7fe7400c5730 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.216+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7400c24f0 con 0x7fe7400c5730 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.216+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c012b20 con 0x7fe7400c5730 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.216+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7400c3ea0 con 0x7fe7400c5730 2024-02-19T14:54:29.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.217+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7400c2350 con 0x7fe7400c5730 2024-02-19T14:54:29.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.218+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe73c012c80 con 0x7fe7400c5730 2024-02-19T14:54:29.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.218+0000 7fe735ffb640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe72406b810 0x7fe72406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.218+0000 7fe744e68640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe72406b810 0x7fe72406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.218+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe73c08a8a0 con 0x7fe7400c5730 2024-02-19T14:54:29.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.219+0000 7fe744e68640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe72406b810 0x7fe72406dcd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe740151440 tx=0x7fe730009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.222+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe7400c2350 con 0x7fe7400c5730 2024-02-19T14:54:29.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.396+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7400c2350 con 0x7fe7400c5730 2024-02-19T14:54:29.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.397+0000 7fe735ffb640 1 -- 172.21.15.78:0/1830186190 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16933 (secure 0 0 0) 0x7fe7400c2350 con 0x7fe7400c5730 2024-02-19T14:54:29.400 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:29.400 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":75,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:27.734957+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe72406b810 msgr2=0x7fe72406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe72406b810 0x7fe72406dcd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe740151440 tx=0x7fe730009040 comp rx=0 tx=0).stop 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 msgr2=0x7fe7400c2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 0x7fe7400c2cf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00d5d0 tx=0x7fe73c00da90 comp rx=0 tx=0).stop 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 shutdown_connections 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe72406b810 0x7fe72406dcd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740152700 0x7fe7401e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7400c8e00 0x7fe7400aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 --2- 172.21.15.78:0/1830186190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7400c5730 0x7fe7400c2cf0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 >> 172.21.15.78:0/1830186190 conn(0x7fe7400cac90 msgr2=0x7fe74014e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:29.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 shutdown_connections 2024-02-19T14:54:29.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.399+0000 7fe7478cd640 1 -- 172.21.15.78:0/1830186190 wait complete. 2024-02-19T14:54:29.413 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:54:29.413 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:54:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.511+0000 7f998705c640 1 Processor -- start 2024-02-19T14:54:29.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.511+0000 7f998705c640 1 -- start start 2024-02-19T14:54:29.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f998705c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99801525c0 0x7f998006d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f998705c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9980155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f998705c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99800c6460 con 0x7f998006d560 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f998705c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99800c1f50 con 0x7f99801525c0 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f998705c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99801593a0 con 0x7f99800c6f80 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f9977fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f998006d560 0x7f99800c6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39652/0 (socket says 172.21.15.78:39652) 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f9977fff640 1 -- 172.21.15.78:0/148773081 learned_addr learned my addr 172.21.15.78:0/148773081 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337809653 0 0) 0x7f99800c6460 con 0x7f998006d560 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9968003520 con 0x7f998006d560 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f9984df8640 1 --2- 172.21.15.78:0/148773081 >> v2:172.21.15.150:3300/0 conn(0x7f99801525c0 0x7f998006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99855f9640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9980155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3962332750 0 0) 0x7f9968003520 con 0x7f998006d560 2024-02-19T14:54:29.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 msgr2=0x7f9980155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9980155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 >> v2:172.21.15.150:3300/0 conn(0x7f99801525c0 msgr2=0x7f998006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 --2- 172.21.15.78:0/148773081 >> v2:172.21.15.150:3300/0 conn(0x7f99801525c0 0x7f998006d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980158720 con 0x7f998006d560 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9970001670 con 0x7f998006d560 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f9984df8640 1 --2- 172.21.15.78:0/148773081 >> v2:172.21.15.150:3300/0 conn(0x7f99801525c0 0x7f998006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 >> v1:172.21.15.78:6789/0 conn(0x7f998006d560 legacy=0x7f99800c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9968003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9968003fa0 0x7f99680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9968004d00 0x7f9968005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.512+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99800c1f50 con 0x7f9968004d00 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801593a0 con 0x7f9968003fa0 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996800bb40 con 0x7f99800c6f80 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99855f9640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9968004d00 0x7f9968005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f9977fff640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9968003fa0 0x7f99680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f9984df8640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9968003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99855f9640 1 -- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 msgr2=0x7f9968003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99855f9640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9968003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99855f9640 1 -- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9968003fa0 msgr2=0x7f99680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99855f9640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9968003fa0 0x7f99680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f99855f9640 1 -- 172.21.15.78:0/148773081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968003520 con 0x7f9968004d00 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.513+0000 7f9984df8640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99800c6f80 0x7f9968003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.514+0000 7f99855f9640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9968004d00 0x7f9968005180 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9978007330 tx=0x7f99680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9159f8ea8e6453ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.514+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997802cca0 con 0x7f9968004d00 2024-02-19T14:54:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.514+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997802ce00 con 0x7f9968004d00 2024-02-19T14:54:29.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.514+0000 7f99777fe640 1 -- 172.21.15.78:0/148773081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978034660 con 0x7f9968004d00 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f998705c640 1 -- 172.21.15.78:0/148773081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9968004d00 msgr2=0x7f9968005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f998705c640 1 --2- 172.21.15.78:0/148773081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9968004d00 0x7f9968005180 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9978007330 tx=0x7f99680057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f99855f9640 1 -- 172.21.15.78:0/148773081 reap_dead start 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f998705c640 1 -- 172.21.15.78:0/148773081 shutdown_connections 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f998705c640 1 -- 172.21.15.78:0/148773081 >> 172.21.15.78:0/148773081 conn(0x7f99800cab20 msgr2=0x7f99800caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f998705c640 1 -- 172.21.15.78:0/148773081 shutdown_connections 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.515+0000 7f998705c640 1 -- 172.21.15.78:0/148773081 wait complete. 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.516+0000 7f998705c640 1 Processor -- start 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.516+0000 7f998705c640 1 -- start start 2024-02-19T14:54:29.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f998014d400 0x7f99801eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9980152f50 0x7f998014b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 0x7f998014b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99800c8740 con 0x7f99801ebe20 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99800c92c0 con 0x7f9980152f50 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800c8990 con 0x7f998014d400 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 0x7f998014b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 0x7f998014b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38932/0 (socket says 172.21.15.78:38932) 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 -- 172.21.15.78:0/1947607819 learned_addr learned my addr 172.21.15.78:0/1947607819 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f9984df8640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f998014d400 0x7f99801eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f9977fff640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9980152f50 0x7f998014b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 -- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f998014d400 msgr2=0x7f99801eb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f998014d400 0x7f99801eb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 -- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9980152f50 msgr2=0x7f998014b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9980152f50 0x7f998014b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 -- 172.21.15.78:0/1947607819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800c8c70 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99855f9640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 0x7f998014b560 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f997802c710 tx=0x7f9978004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978003b90 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9978003d10 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99800c8f50 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978004260 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.517+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801edbc0 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.518+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9980074990 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.519+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f99780043c0 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.519+0000 7f99757fa640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f996c06b810 0x7f996c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.519+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f99780ac400 con 0x7f99801ebe20 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.519+0000 7f9984df8640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f996c06b810 0x7f996c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.520+0000 7f9984df8640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f996c06b810 0x7f996c06dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f997c006fd0 tx=0x7f997c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.523+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9978076dd0 con 0x7f99801ebe20 2024-02-19T14:54:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.697+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9980158fc0 con 0x7f996c06b810 2024-02-19T14:54:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.704+0000 7f99757fa640 1 -- 172.21.15.78:0/1947607819 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276974 (secure 0 0 0) 0x7f997c009560 con 0x7f996c06b810 2024-02-19T14:54:29.709 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:29.711 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:54:29.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f996c06b810 msgr2=0x7f996c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f996c06b810 0x7f996c06dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f997c006fd0 tx=0x7f997c008040 comp rx=0 tx=0).stop 2024-02-19T14:54:29.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 msgr2=0x7f998014b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 0x7f998014b560 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f997802c710 tx=0x7f9978004ab0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 shutdown_connections 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f996c06b810 0x7f996c06dcd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99801ebe20 0x7f998014b560 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9980152f50 0x7f998014b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 --2- 172.21.15.78:0/1947607819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f998014d400 0x7f99801eb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 >> 172.21.15.78:0/1947607819 conn(0x7f99800cab20 msgr2=0x7f99800c3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 shutdown_connections 2024-02-19T14:54:29.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.707+0000 7f998705c640 1 -- 172.21.15.78:0/1947607819 wait complete. 2024-02-19T14:54:29.720 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-02-19T14:54:28.089308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":168,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":224124,"kb_used_data":7920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754750596,"statfs":{"total":773094113280,"available":772864610304,"internally_reserved":0,"allocated":8110080,"data_stored":4431191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":742,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000599"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842470+0000","last_change":"2024-02-19T14:54:26.747816+0000","last_active":"2024-02-19T14:54:27.842470+0000","last_peered":"2024-02-19T14:54:27.842470+0000","last_clean":"2024-02-19T14:54:27.842470+0000","last_became_active":"2024-02-19T14:54:26.747520+0000","last_became_peered":"2024-02-19T14:54:26.747520+0000","last_unstale":"2024-02-19T14:54:27.842470+0000","last_undegraded":"2024-02-19T14:54:27.842470+0000","last_fullsized":"2024-02-19T14:54:27.842470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:36.704748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841055+0000","last_change":"2024-02-19T14:54:26.748066+0000","last_active":"2024-02-19T14:54:27.841055+0000","last_peered":"2024-02-19T14:54:27.841055+0000","last_clean":"2024-02-19T14:54:27.841055+0000","last_became_active":"2024-02-19T14:54:26.747792+0000","last_became_peered":"2024-02-19T14:54:26.747792+0000","last_unstale":"2024-02-19T14:54:27.841055+0000","last_undegraded":"2024-02-19T14:54:27.841055+0000","last_fullsized":"2024-02-19T14:54:27.841055+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:05.620631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842521+0000","last_change":"2024-02-19T14:54:26.746983+0000","last_active":"2024-02-19T14:54:27.842521+0000","last_peered":"2024-02-19T14:54:27.842521+0000","last_clean":"2024-02-19T14:54:27.842521+0000","last_became_active":"2024-02-19T14:54:26.746664+0000","last_became_peered":"2024-02-19T14:54:26.746664+0000","last_unstale":"2024-02-19T14:54:27.842521+0000","last_undegraded":"2024-02-19T14:54:27.842521+0000","last_fullsized":"2024-02-19T14:54:27.842521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:27.405179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841511+0000","last_change":"2024-02-19T14:54:26.747599+0000","last_active":"2024-02-19T14:54:27.841511+0000","last_peered":"2024-02-19T14:54:27.841511+0000","last_clean":"2024-02-19T14:54:27.841511+0000","last_became_active":"2024-02-19T14:54:26.747345+0000","last_became_peered":"2024-02-19T14:54:26.747345+0000","last_unstale":"2024-02-19T14:54:27.841511+0000","last_undegraded":"2024-02-19T14:54:27.841511+0000","last_fullsized":"2024-02-19T14:54:27.841511+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:20:30.359001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842150+0000","last_change":"2024-02-19T14:54:26.747148+0000","last_active":"2024-02-19T14:54:27.842150+0000","last_peered":"2024-02-19T14:54:27.842150+0000","last_clean":"2024-02-19T14:54:27.842150+0000","last_became_active":"2024-02-19T14:54:26.746685+0000","last_became_peered":"2024-02-19T14:54:26.746685+0000","last_unstale":"2024-02-19T14:54:27.842150+0000","last_undegraded":"2024-02-19T14:54:27.842150+0000","last_fullsized":"2024-02-19T14:54:27.842150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:44:34.240945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841845+0000","last_change":"2024-02-19T14:54:26.752317+0000","last_active":"2024-02-19T14:54:27.841845+0000","last_peered":"2024-02-19T14:54:27.841845+0000","last_clean":"2024-02-19T14:54:27.841845+0000","last_became_active":"2024-02-19T14:54:26.752031+0000","last_became_peered":"2024-02-19T14:54:26.752031+0000","last_unstale":"2024-02-19T14:54:27.841845+0000","last_undegraded":"2024-02-19T14:54:27.841845+0000","last_fullsized":"2024-02-19T14:54:27.841845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:39.758301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842275+0000","last_change":"2024-02-19T14:54:26.746821+0000","last_active":"2024-02-19T14:54:27.842275+0000","last_peered":"2024-02-19T14:54:27.842275+0000","last_clean":"2024-02-19T14:54:27.842275+0000","last_became_active":"2024-02-19T14:54:26.746408+0000","last_became_peered":"2024-02-19T14:54:26.746408+0000","last_unstale":"2024-02-19T14:54:27.842275+0000","last_undegraded":"2024-02-19T14:54:27.842275+0000","last_fullsized":"2024-02-19T14:54:27.842275+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:38.456834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"74'4","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841489+0000","last_change":"2024-02-19T14:54:25.736715+0000","last_active":"2024-02-19T14:54:27.841489+0000","last_peered":"2024-02-19T14:54:27.841489+0000","last_clean":"2024-02-19T14:54:27.841489+0000","last_became_active":"2024-02-19T14:54:25.736266+0000","last_became_peered":"2024-02-19T14:54:25.736266+0000","last_unstale":"2024-02-19T14:54:27.841489+0000","last_undegraded":"2024-02-19T14:54:27.841489+0000","last_fullsized":"2024-02-19T14:54:27.841489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:29.148205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842270+0000","last_change":"2024-02-19T14:54:25.736306+0000","last_active":"2024-02-19T14:54:27.842270+0000","last_peered":"2024-02-19T14:54:27.842270+0000","last_clean":"2024-02-19T14:54:27.842270+0000","last_became_active":"2024-02-19T14:54:25.736029+0000","last_became_peered":"2024-02-19T14:54:25.736029+0000","last_unstale":"2024-02-19T14:54:27.842270+0000","last_undegraded":"2024-02-19T14:54:27.842270+0000","last_fullsized":"2024-02-19T14:54:27.842270+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:59:00.456190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"74'3","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842740+0000","last_change":"2024-02-19T14:54:25.732654+0000","last_active":"2024-02-19T14:54:27.842740+0000","last_peered":"2024-02-19T14:54:27.842740+0000","last_clean":"2024-02-19T14:54:27.842740+0000","last_became_active":"2024-02-19T14:54:25.732388+0000","last_became_peered":"2024-02-19T14:54:25.732388+0000","last_unstale":"2024-02-19T14:54:27.842740+0000","last_undegraded":"2024-02-19T14:54:27.842740+0000","last_fullsized":"2024-02-19T14:54:27.842740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:04.044350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841519+0000","last_change":"2024-02-19T14:54:25.736476+0000","last_active":"2024-02-19T14:54:27.841519+0000","last_peered":"2024-02-19T14:54:27.841519+0000","last_clean":"2024-02-19T14:54:27.841519+0000","last_became_active":"2024-02-19T14:54:25.736202+0000","last_became_peered":"2024-02-19T14:54:25.736202+0000","last_unstale":"2024-02-19T14:54:27.841519+0000","last_undegraded":"2024-02-19T14:54:27.841519+0000","last_fullsized":"2024-02-19T14:54:27.841519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:01.450961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841426+0000","last_change":"2024-02-19T14:54:26.747952+0000","last_active":"2024-02-19T14:54:27.841426+0000","last_peered":"2024-02-19T14:54:27.841426+0000","last_clean":"2024-02-19T14:54:27.841426+0000","last_became_active":"2024-02-19T14:54:26.747656+0000","last_became_peered":"2024-02-19T14:54:26.747656+0000","last_unstale":"2024-02-19T14:54:27.841426+0000","last_undegraded":"2024-02-19T14:54:27.841426+0000","last_fullsized":"2024-02-19T14:54:27.841426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:50.690817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841660+0000","last_change":"2024-02-19T14:54:26.756834+0000","last_active":"2024-02-19T14:54:27.841660+0000","last_peered":"2024-02-19T14:54:27.841660+0000","last_clean":"2024-02-19T14:54:27.841660+0000","last_became_active":"2024-02-19T14:54:26.756548+0000","last_became_peered":"2024-02-19T14:54:26.756548+0000","last_unstale":"2024-02-19T14:54:27.841660+0000","last_undegraded":"2024-02-19T14:54:27.841660+0000","last_fullsized":"2024-02-19T14:54:27.841660+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:25.077428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"74'1","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842724+0000","last_change":"2024-02-19T14:54:25.735865+0000","last_active":"2024-02-19T14:54:27.842724+0000","last_peered":"2024-02-19T14:54:27.842724+0000","last_clean":"2024-02-19T14:54:27.842724+0000","last_became_active":"2024-02-19T14:54:25.735569+0000","last_became_peered":"2024-02-19T14:54:25.735569+0000","last_unstale":"2024-02-19T14:54:27.842724+0000","last_undegraded":"2024-02-19T14:54:27.842724+0000","last_fullsized":"2024-02-19T14:54:27.842724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:55:08.537113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841940+0000","last_change":"2024-02-19T14:54:25.738928+0000","last_active":"2024-02-19T14:54:27.841940+0000","last_peered":"2024-02-19T14:54:27.841940+0000","last_clean":"2024-02-19T14:54:27.841940+0000","last_became_active":"2024-02-19T14:54:25.738674+0000","last_became_peered":"2024-02-19T14:54:25.738674+0000","last_unstale":"2024-02-19T14:54:27.841940+0000","last_undegraded":"2024-02-19T14:54:27.841940+0000","last_fullsized":"2024-02-19T14:54:27.841940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:58.792240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:25.715235+0000","last_change":"2024-02-19T14:54:25.715235+0000","last_active":"2024-02-19T14:54:25.715235+0000","last_peered":"2024-02-19T14:54:25.715235+0000","last_clean":"2024-02-19T14:54:25.715235+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:25.715235+0000","last_undegraded":"2024-02-19T14:54:25.715235+0000","last_fullsized":"2024-02-19T14:54:25.715235+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:54:24.709171+0000","last_change":"2024-02-19T14:54:24.709171+0000","last_active":"2024-02-19T14:54:24.709171+0000","last_peered":"2024-02-19T14:54:24.709171+0000","last_clean":"2024-02-19T14:54:24.709171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:54:24.709171+0000","last_undegraded":"2024-02-19T14:54:24.709171+0000","last_fullsized":"2024-02-19T14:54:24.709171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":264,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-02-19T14:54:22.040189+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:54:22.040189+0000","last_peered":"2024-02-19T14:54:22.040189+0000","last_clean":"2024-02-19T14:54:22.040189+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:54:22.040189+0000","last_undegraded":"2024-02-19T14:54:22.040189+0000","last_fullsized":"2024-02-19T14:54:22.040189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":742,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":12,"acting":12,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28436,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343404,"statfs":{"total":96636764160,"available":96607645696,"internally_reserved":0,"allocated":1449984,"data_stored":992627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.625},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.76200000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.497,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.77900000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.377}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.441},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57799999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.50600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27864,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":856064,"data_stored":402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46999999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.432}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.625}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55800000000000005,"15min":0.56899999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.76200000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.66000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.71599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27868,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":856064,"data_stored":402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.747,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.59699999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.746}]},{"osd":1,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.877},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.71499999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95799999999999996,"5min":1.038,"15min":1.038},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.79000000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.84499999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.80400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":402259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29499999999999998},"min":{"1min":0.17899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.309,"15min":0.30499999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.31},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.374,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.441},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49199999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69099999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.625}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68700000000000006,"5min":1.133,"15min":1.133},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82999999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.51300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":428843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.437,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.42799999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.375,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79000000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84199999999999997,"5min":0.878,"15min":0.878},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94199999999999995,"5min":1.236,"15min":1.236},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.629},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.995,"15min":0.995},"last":0.56599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27876,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":868352,"data_stored":408058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33500000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.497,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.246},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33100000000000002,"15min":0.33800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.373}]},{"osd":2,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.307,"15min":0.309},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.371,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57999999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.73899999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.625},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"last":0.755},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44400000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.69299999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28436,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343404,"statfs":{"total":96636764160,"available":96607645696,"internally_reserved":0,"allocated":1449984,"data_stored":992627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.38100000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32300000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42599999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34899999999999998,"15min":0.35199999999999998},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.42699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.41199999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57599999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.497}]},{"osd":5,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.59999999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68700000000000006,"5min":0.754,"15min":0.754},"last":0.439}]},{"osd":7,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72999999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:54:29.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:54:29.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.858+0000 7f528bf27640 1 Processor -- start 2024-02-19T14:54:29.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.858+0000 7f528bf27640 1 -- start start 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528bf27640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f52840c4980 0x7f52840c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528bf27640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52840705b0 0x7f52840709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528bf27640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52840ae670 con 0x7f52840add70 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528bf27640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52840716b0 con 0x7f52840705b0 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528bf27640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5284153780 con 0x7f52840c4980 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52840705b0 0x7f52840709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52840705b0 0x7f52840709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38934/0 (socket says 172.21.15.78:38934) 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 -- 172.21.15.78:0/2212804045 learned_addr learned my addr 172.21.15.78:0/2212804045 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 -- 172.21.15.78:0/2212804045 >> v2:172.21.15.150:3300/0 conn(0x7f52840c4980 msgr2=0x7f52840c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f52894c2640 1 --2- 172.21.15.78:0/2212804045 >> v2:172.21.15.150:3300/0 conn(0x7f52840c4980 0x7f52840c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 --2- 172.21.15.78:0/2212804045 >> v2:172.21.15.150:3300/0 conn(0x7f52840c4980 0x7f52840c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 -- 172.21.15.78:0/2212804045 >> v1:172.21.15.150:6789/0 conn(0x7f52840add70 legacy=0x7f528406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:29.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.859+0000 7f528a4c4640 1 -- 172.21.15.78:0/2212804045 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5284153a60 con 0x7f52840705b0 2024-02-19T14:54:29.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.860+0000 7f528a4c4640 1 --2- 172.21.15.78:0/2212804045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f52840709e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5284073050 tx=0x7f528002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7265bdf5a19f470d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.860+0000 7f5288cc1640 1 -- 172.21.15.78:0/2212804045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52800317e0 con 0x7f52840705b0 2024-02-19T14:54:29.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.860+0000 7f5288cc1640 1 -- 172.21.15.78:0/2212804045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280031940 con 0x7f52840705b0 2024-02-19T14:54:29.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.860+0000 7f5288cc1640 1 -- 172.21.15.78:0/2212804045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280031cd0 con 0x7f52840705b0 2024-02-19T14:54:29.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 -- 172.21.15.78:0/2212804045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 msgr2=0x7f52840709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 --2- 172.21.15.78:0/2212804045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f52840709e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5284073050 tx=0x7f528002b880 comp rx=0 tx=0).stop 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 -- 172.21.15.78:0/2212804045 shutdown_connections 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 --2- 172.21.15.78:0/2212804045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f52840709e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 --2- 172.21.15.78:0/2212804045 >> v2:172.21.15.150:3300/0 conn(0x7f52840c4980 0x7f52840c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 -- 172.21.15.78:0/2212804045 >> 172.21.15.78:0/2212804045 conn(0x7f52840c85e0 msgr2=0x7f52840c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 -- 172.21.15.78:0/2212804045 shutdown_connections 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.861+0000 7f528bf27640 1 -- 172.21.15.78:0/2212804045 wait complete. 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 Processor -- start 2024-02-19T14:54:29.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 -- start start 2024-02-19T14:54:29.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f5284154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52840add70 0x7f52840c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52840c4980 0x7f52841e4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5284155940 con 0x7f52840705b0 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5284155540 con 0x7f52840c4980 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.862+0000 7f528bf27640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52841eb710 con 0x7f52840add70 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f5284154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f5284154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38946/0 (socket says 172.21.15.78:38946) 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 -- 172.21.15.78:0/187820103 learned_addr learned my addr 172.21.15.78:0/187820103 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 -- 172.21.15.78:0/187820103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52840add70 msgr2=0x7f52840c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52840add70 0x7f52840c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 -- 172.21.15.78:0/187820103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52840c4980 msgr2=0x7f52841e4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52840c4980 0x7f52841e4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 -- 172.21.15.78:0/187820103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800094d0 con 0x7f52840705b0 2024-02-19T14:54:29.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f5289cc3640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f5284154050 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f527c00c7d0 tx=0x7f527c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c003d80 con 0x7f52840705b0 2024-02-19T14:54:29.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527c005070 con 0x7f52840705b0 2024-02-19T14:54:29.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c005420 con 0x7f52840705b0 2024-02-19T14:54:29.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52841eab90 con 0x7f52840705b0 2024-02-19T14:54:29.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.863+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52841eb150 con 0x7f52840705b0 2024-02-19T14:54:29.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.864+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f527c005580 con 0x7f52840705b0 2024-02-19T14:54:29.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.864+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52840676a0 con 0x7f52840705b0 2024-02-19T14:54:29.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.865+0000 7f52767fc640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f526806b810 0x7f526806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:29.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.865+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f527c089c10 con 0x7f52840705b0 2024-02-19T14:54:29.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.865+0000 7f52894c2640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f526806b810 0x7f526806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:29.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.865+0000 7f52894c2640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f526806b810 0x7f526806dcd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f52840a7e60 tx=0x7f5278008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:29.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:29.869+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f527c054530 con 0x7f52840705b0 2024-02-19T14:54:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.110+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52840ae150 con 0x7f526806b810 2024-02-19T14:54:30.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.118+0000 7f52767fc640 1 -- 172.21.15.78:0/187820103 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289575 (secure 0 0 0) 0x7f527800adc0 con 0x7f526806b810 2024-02-19T14:54:30.123 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:30.125 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f526806b810 msgr2=0x7f526806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f526806b810 0x7f526806dcd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f52840a7e60 tx=0x7f5278008040 comp rx=0 tx=0).stop 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 msgr2=0x7f5284154050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f5284154050 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f527c00c7d0 tx=0x7f527c00cc90 comp rx=0 tx=0).stop 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 shutdown_connections 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f526806b810 0x7f526806dcd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52840c4980 0x7f52841e4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52840add70 0x7f52840c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 --2- 172.21.15.78:0/187820103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52840705b0 0x7f5284154050 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 >> 172.21.15.78:0/187820103 conn(0x7f52840c85e0 msgr2=0x7f52840c74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 shutdown_connections 2024-02-19T14:54:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.121+0000 7f528bf27640 1 -- 172.21.15.78:0/187820103 wait complete. 2024-02-19T14:54:30.135 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-02-19T14:54:30.089618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224520,"kb_used_data":8308,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754750200,"statfs":{"total":773094113280,"available":772864204800,"internally_reserved":0,"allocated":8507392,"data_stored":4617056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.368067"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741979+0000","last_change":"2024-02-19T14:54:26.743766+0000","last_active":"2024-02-19T14:54:27.741979+0000","last_peered":"2024-02-19T14:54:27.741979+0000","last_clean":"2024-02-19T14:54:27.741979+0000","last_became_active":"2024-02-19T14:54:26.743402+0000","last_became_peered":"2024-02-19T14:54:26.743402+0000","last_unstale":"2024-02-19T14:54:27.741979+0000","last_undegraded":"2024-02-19T14:54:27.741979+0000","last_fullsized":"2024-02-19T14:54:27.741979+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841730+0000","last_change":"2024-02-19T14:54:26.751673+0000","last_active":"2024-02-19T14:54:27.841730+0000","last_peered":"2024-02-19T14:54:27.841730+0000","last_clean":"2024-02-19T14:54:27.841730+0000","last_became_active":"2024-02-19T14:54:26.751360+0000","last_became_peered":"2024-02-19T14:54:26.751360+0000","last_unstale":"2024-02-19T14:54:27.841730+0000","last_undegraded":"2024-02-19T14:54:27.841730+0000","last_fullsized":"2024-02-19T14:54:27.841730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841480+0000","last_change":"2024-02-19T14:54:26.750603+0000","last_active":"2024-02-19T14:54:27.841480+0000","last_peered":"2024-02-19T14:54:27.841480+0000","last_clean":"2024-02-19T14:54:27.841480+0000","last_became_active":"2024-02-19T14:54:26.750088+0000","last_became_peered":"2024-02-19T14:54:26.750088+0000","last_unstale":"2024-02-19T14:54:27.841480+0000","last_undegraded":"2024-02-19T14:54:27.841480+0000","last_fullsized":"2024-02-19T14:54:27.841480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:47.414910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841876+0000","last_change":"2024-02-19T14:54:26.733014+0000","last_active":"2024-02-19T14:54:27.841876+0000","last_peered":"2024-02-19T14:54:27.841876+0000","last_clean":"2024-02-19T14:54:27.841876+0000","last_became_active":"2024-02-19T14:54:26.732738+0000","last_became_peered":"2024-02-19T14:54:26.732738+0000","last_unstale":"2024-02-19T14:54:27.841876+0000","last_undegraded":"2024-02-19T14:54:27.841876+0000","last_fullsized":"2024-02-19T14:54:27.841876+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:37.293731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842470+0000","last_change":"2024-02-19T14:54:26.747816+0000","last_active":"2024-02-19T14:54:27.842470+0000","last_peered":"2024-02-19T14:54:27.842470+0000","last_clean":"2024-02-19T14:54:27.842470+0000","last_became_active":"2024-02-19T14:54:26.747520+0000","last_became_peered":"2024-02-19T14:54:26.747520+0000","last_unstale":"2024-02-19T14:54:27.842470+0000","last_undegraded":"2024-02-19T14:54:27.842470+0000","last_fullsized":"2024-02-19T14:54:27.842470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:36.704748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738237+0000","last_change":"2024-02-19T14:54:26.746700+0000","last_active":"2024-02-19T14:54:27.738237+0000","last_peered":"2024-02-19T14:54:27.738237+0000","last_clean":"2024-02-19T14:54:27.738237+0000","last_became_active":"2024-02-19T14:54:26.746436+0000","last_became_peered":"2024-02-19T14:54:26.746436+0000","last_unstale":"2024-02-19T14:54:27.738237+0000","last_undegraded":"2024-02-19T14:54:27.738237+0000","last_fullsized":"2024-02-19T14:54:27.738237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841611+0000","last_change":"2024-02-19T14:54:26.751787+0000","last_active":"2024-02-19T14:54:27.841611+0000","last_peered":"2024-02-19T14:54:27.841611+0000","last_clean":"2024-02-19T14:54:27.841611+0000","last_became_active":"2024-02-19T14:54:26.751414+0000","last_became_peered":"2024-02-19T14:54:26.751414+0000","last_unstale":"2024-02-19T14:54:27.841611+0000","last_undegraded":"2024-02-19T14:54:27.841611+0000","last_fullsized":"2024-02-19T14:54:27.841611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:31.766567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841055+0000","last_change":"2024-02-19T14:54:26.748066+0000","last_active":"2024-02-19T14:54:27.841055+0000","last_peered":"2024-02-19T14:54:27.841055+0000","last_clean":"2024-02-19T14:54:27.841055+0000","last_became_active":"2024-02-19T14:54:26.747792+0000","last_became_peered":"2024-02-19T14:54:26.747792+0000","last_unstale":"2024-02-19T14:54:27.841055+0000","last_undegraded":"2024-02-19T14:54:27.841055+0000","last_fullsized":"2024-02-19T14:54:27.841055+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:05.620631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743143+0000","last_change":"2024-02-19T14:54:26.741238+0000","last_active":"2024-02-19T14:54:27.743143+0000","last_peered":"2024-02-19T14:54:27.743143+0000","last_clean":"2024-02-19T14:54:27.743143+0000","last_became_active":"2024-02-19T14:54:26.740986+0000","last_became_peered":"2024-02-19T14:54:26.740986+0000","last_unstale":"2024-02-19T14:54:27.743143+0000","last_undegraded":"2024-02-19T14:54:27.743143+0000","last_fullsized":"2024-02-19T14:54:27.743143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:30.855089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841344+0000","last_change":"2024-02-19T14:54:26.741087+0000","last_active":"2024-02-19T14:54:27.841344+0000","last_peered":"2024-02-19T14:54:27.841344+0000","last_clean":"2024-02-19T14:54:27.841344+0000","last_became_active":"2024-02-19T14:54:26.740795+0000","last_became_peered":"2024-02-19T14:54:26.740795+0000","last_unstale":"2024-02-19T14:54:27.841344+0000","last_undegraded":"2024-02-19T14:54:27.841344+0000","last_fullsized":"2024-02-19T14:54:27.841344+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:50.652136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841660+0000","last_change":"2024-02-19T14:54:26.753214+0000","last_active":"2024-02-19T14:54:27.841660+0000","last_peered":"2024-02-19T14:54:27.841660+0000","last_clean":"2024-02-19T14:54:27.841660+0000","last_became_active":"2024-02-19T14:54:26.752817+0000","last_became_peered":"2024-02-19T14:54:26.752817+0000","last_unstale":"2024-02-19T14:54:27.841660+0000","last_undegraded":"2024-02-19T14:54:27.841660+0000","last_fullsized":"2024-02-19T14:54:27.841660+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841884+0000","last_change":"2024-02-19T14:54:26.752439+0000","last_active":"2024-02-19T14:54:27.841884+0000","last_peered":"2024-02-19T14:54:27.841884+0000","last_clean":"2024-02-19T14:54:27.841884+0000","last_became_active":"2024-02-19T14:54:26.752104+0000","last_became_peered":"2024-02-19T14:54:26.752104+0000","last_unstale":"2024-02-19T14:54:27.841884+0000","last_undegraded":"2024-02-19T14:54:27.841884+0000","last_fullsized":"2024-02-19T14:54:27.841884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:29.434729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842521+0000","last_change":"2024-02-19T14:54:26.746983+0000","last_active":"2024-02-19T14:54:27.842521+0000","last_peered":"2024-02-19T14:54:27.842521+0000","last_clean":"2024-02-19T14:54:27.842521+0000","last_became_active":"2024-02-19T14:54:26.746664+0000","last_became_peered":"2024-02-19T14:54:26.746664+0000","last_unstale":"2024-02-19T14:54:27.842521+0000","last_undegraded":"2024-02-19T14:54:27.842521+0000","last_fullsized":"2024-02-19T14:54:27.842521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:27.405179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738370+0000","last_change":"2024-02-19T14:54:26.744541+0000","last_active":"2024-02-19T14:54:27.738370+0000","last_peered":"2024-02-19T14:54:27.738370+0000","last_clean":"2024-02-19T14:54:27.738370+0000","last_became_active":"2024-02-19T14:54:26.744242+0000","last_became_peered":"2024-02-19T14:54:26.744242+0000","last_unstale":"2024-02-19T14:54:27.738370+0000","last_undegraded":"2024-02-19T14:54:27.738370+0000","last_fullsized":"2024-02-19T14:54:27.738370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840724+0000","last_change":"2024-02-19T14:54:26.745146+0000","last_active":"2024-02-19T14:54:27.840724+0000","last_peered":"2024-02-19T14:54:27.840724+0000","last_clean":"2024-02-19T14:54:27.840724+0000","last_became_active":"2024-02-19T14:54:26.744342+0000","last_became_peered":"2024-02-19T14:54:26.744342+0000","last_unstale":"2024-02-19T14:54:27.840724+0000","last_undegraded":"2024-02-19T14:54:27.840724+0000","last_fullsized":"2024-02-19T14:54:27.840724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741654+0000","last_change":"2024-02-19T14:54:26.743966+0000","last_active":"2024-02-19T14:54:27.741654+0000","last_peered":"2024-02-19T14:54:27.741654+0000","last_clean":"2024-02-19T14:54:27.741654+0000","last_became_active":"2024-02-19T14:54:26.743693+0000","last_became_peered":"2024-02-19T14:54:26.743693+0000","last_unstale":"2024-02-19T14:54:27.741654+0000","last_undegraded":"2024-02-19T14:54:27.741654+0000","last_fullsized":"2024-02-19T14:54:27.741654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841511+0000","last_change":"2024-02-19T14:54:26.747599+0000","last_active":"2024-02-19T14:54:27.841511+0000","last_peered":"2024-02-19T14:54:27.841511+0000","last_clean":"2024-02-19T14:54:27.841511+0000","last_became_active":"2024-02-19T14:54:26.747345+0000","last_became_peered":"2024-02-19T14:54:26.747345+0000","last_unstale":"2024-02-19T14:54:27.841511+0000","last_undegraded":"2024-02-19T14:54:27.841511+0000","last_fullsized":"2024-02-19T14:54:27.841511+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:20:30.359001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841160+0000","last_change":"2024-02-19T14:54:26.736611+0000","last_active":"2024-02-19T14:54:27.841160+0000","last_peered":"2024-02-19T14:54:27.841160+0000","last_clean":"2024-02-19T14:54:27.841160+0000","last_became_active":"2024-02-19T14:54:26.736284+0000","last_became_peered":"2024-02-19T14:54:26.736284+0000","last_unstale":"2024-02-19T14:54:27.841160+0000","last_undegraded":"2024-02-19T14:54:27.841160+0000","last_fullsized":"2024-02-19T14:54:27.841160+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:15.218606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738368+0000","last_change":"2024-02-19T14:54:26.749665+0000","last_active":"2024-02-19T14:54:27.738368+0000","last_peered":"2024-02-19T14:54:27.738368+0000","last_clean":"2024-02-19T14:54:27.738368+0000","last_became_active":"2024-02-19T14:54:26.749397+0000","last_became_peered":"2024-02-19T14:54:26.749397+0000","last_unstale":"2024-02-19T14:54:27.738368+0000","last_undegraded":"2024-02-19T14:54:27.738368+0000","last_fullsized":"2024-02-19T14:54:27.738368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842150+0000","last_change":"2024-02-19T14:54:26.747148+0000","last_active":"2024-02-19T14:54:27.842150+0000","last_peered":"2024-02-19T14:54:27.842150+0000","last_clean":"2024-02-19T14:54:27.842150+0000","last_became_active":"2024-02-19T14:54:26.746685+0000","last_became_peered":"2024-02-19T14:54:26.746685+0000","last_unstale":"2024-02-19T14:54:27.842150+0000","last_undegraded":"2024-02-19T14:54:27.842150+0000","last_fullsized":"2024-02-19T14:54:27.842150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:44:34.240945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743324+0000","last_change":"2024-02-19T14:54:26.738144+0000","last_active":"2024-02-19T14:54:27.743324+0000","last_peered":"2024-02-19T14:54:27.743324+0000","last_clean":"2024-02-19T14:54:27.743324+0000","last_became_active":"2024-02-19T14:54:26.737816+0000","last_became_peered":"2024-02-19T14:54:26.737816+0000","last_unstale":"2024-02-19T14:54:27.743324+0000","last_undegraded":"2024-02-19T14:54:27.743324+0000","last_fullsized":"2024-02-19T14:54:27.743324+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:53.244189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841845+0000","last_change":"2024-02-19T14:54:26.752317+0000","last_active":"2024-02-19T14:54:27.841845+0000","last_peered":"2024-02-19T14:54:27.841845+0000","last_clean":"2024-02-19T14:54:27.841845+0000","last_became_active":"2024-02-19T14:54:26.752031+0000","last_became_peered":"2024-02-19T14:54:26.752031+0000","last_unstale":"2024-02-19T14:54:27.841845+0000","last_undegraded":"2024-02-19T14:54:27.841845+0000","last_fullsized":"2024-02-19T14:54:27.841845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:39.758301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743190+0000","last_change":"2024-02-19T14:54:26.733087+0000","last_active":"2024-02-19T14:54:27.743190+0000","last_peered":"2024-02-19T14:54:27.743190+0000","last_clean":"2024-02-19T14:54:27.743190+0000","last_became_active":"2024-02-19T14:54:26.732793+0000","last_became_peered":"2024-02-19T14:54:26.732793+0000","last_unstale":"2024-02-19T14:54:27.743190+0000","last_undegraded":"2024-02-19T14:54:27.743190+0000","last_fullsized":"2024-02-19T14:54:27.743190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:13.700400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840664+0000","last_change":"2024-02-19T14:54:26.743065+0000","last_active":"2024-02-19T14:54:27.840664+0000","last_peered":"2024-02-19T14:54:27.840664+0000","last_clean":"2024-02-19T14:54:27.840664+0000","last_became_active":"2024-02-19T14:54:26.742623+0000","last_became_peered":"2024-02-19T14:54:26.742623+0000","last_unstale":"2024-02-19T14:54:27.840664+0000","last_undegraded":"2024-02-19T14:54:27.840664+0000","last_fullsized":"2024-02-19T14:54:27.840664+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:29.434729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841775+0000","last_change":"2024-02-19T14:54:26.752403+0000","last_active":"2024-02-19T14:54:27.841775+0000","last_peered":"2024-02-19T14:54:27.841775+0000","last_clean":"2024-02-19T14:54:27.841775+0000","last_became_active":"2024-02-19T14:54:26.752116+0000","last_became_peered":"2024-02-19T14:54:26.752116+0000","last_unstale":"2024-02-19T14:54:27.841775+0000","last_undegraded":"2024-02-19T14:54:27.841775+0000","last_fullsized":"2024-02-19T14:54:27.841775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:34:33.181295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841487+0000","last_change":"2024-02-19T14:54:26.752358+0000","last_active":"2024-02-19T14:54:27.841487+0000","last_peered":"2024-02-19T14:54:27.841487+0000","last_clean":"2024-02-19T14:54:27.841487+0000","last_became_active":"2024-02-19T14:54:26.752076+0000","last_became_peered":"2024-02-19T14:54:26.752076+0000","last_unstale":"2024-02-19T14:54:27.841487+0000","last_undegraded":"2024-02-19T14:54:27.841487+0000","last_fullsized":"2024-02-19T14:54:27.841487+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:40.346858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741861+0000","last_change":"2024-02-19T14:54:26.743117+0000","last_active":"2024-02-19T14:54:27.741861+0000","last_peered":"2024-02-19T14:54:27.741861+0000","last_clean":"2024-02-19T14:54:27.741861+0000","last_became_active":"2024-02-19T14:54:26.742845+0000","last_became_peered":"2024-02-19T14:54:26.742845+0000","last_unstale":"2024-02-19T14:54:27.741861+0000","last_undegraded":"2024-02-19T14:54:27.741861+0000","last_fullsized":"2024-02-19T14:54:27.741861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842275+0000","last_change":"2024-02-19T14:54:26.746821+0000","last_active":"2024-02-19T14:54:27.842275+0000","last_peered":"2024-02-19T14:54:27.842275+0000","last_clean":"2024-02-19T14:54:27.842275+0000","last_became_active":"2024-02-19T14:54:26.746408+0000","last_became_peered":"2024-02-19T14:54:26.746408+0000","last_unstale":"2024-02-19T14:54:27.842275+0000","last_undegraded":"2024-02-19T14:54:27.842275+0000","last_fullsized":"2024-02-19T14:54:27.842275+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:38.456834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841273+0000","last_change":"2024-02-19T14:54:26.745851+0000","last_active":"2024-02-19T14:54:27.841273+0000","last_peered":"2024-02-19T14:54:27.841273+0000","last_clean":"2024-02-19T14:54:27.841273+0000","last_became_active":"2024-02-19T14:54:26.745587+0000","last_became_peered":"2024-02-19T14:54:26.745587+0000","last_unstale":"2024-02-19T14:54:27.841273+0000","last_undegraded":"2024-02-19T14:54:27.841273+0000","last_fullsized":"2024-02-19T14:54:27.841273+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741997+0000","last_change":"2024-02-19T14:54:26.758958+0000","last_active":"2024-02-19T14:54:27.741997+0000","last_peered":"2024-02-19T14:54:27.741997+0000","last_clean":"2024-02-19T14:54:27.741997+0000","last_became_active":"2024-02-19T14:54:26.758649+0000","last_became_peered":"2024-02-19T14:54:26.758649+0000","last_unstale":"2024-02-19T14:54:27.741997+0000","last_undegraded":"2024-02-19T14:54:27.741997+0000","last_fullsized":"2024-02-19T14:54:27.741997+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:38.553508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840704+0000","last_change":"2024-02-19T14:54:26.742392+0000","last_active":"2024-02-19T14:54:27.840704+0000","last_peered":"2024-02-19T14:54:27.840704+0000","last_clean":"2024-02-19T14:54:27.840704+0000","last_became_active":"2024-02-19T14:54:26.742129+0000","last_became_peered":"2024-02-19T14:54:26.742129+0000","last_unstale":"2024-02-19T14:54:27.840704+0000","last_undegraded":"2024-02-19T14:54:27.840704+0000","last_fullsized":"2024-02-19T14:54:27.840704+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:50.140868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741876+0000","last_change":"2024-02-19T14:54:26.747016+0000","last_active":"2024-02-19T14:54:27.741876+0000","last_peered":"2024-02-19T14:54:27.741876+0000","last_clean":"2024-02-19T14:54:27.741876+0000","last_became_active":"2024-02-19T14:54:26.746761+0000","last_became_peered":"2024-02-19T14:54:26.746761+0000","last_unstale":"2024-02-19T14:54:27.741876+0000","last_undegraded":"2024-02-19T14:54:27.741876+0000","last_fullsized":"2024-02-19T14:54:27.741876+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841662+0000","last_change":"2024-02-19T14:54:26.736520+0000","last_active":"2024-02-19T14:54:27.841662+0000","last_peered":"2024-02-19T14:54:27.841662+0000","last_clean":"2024-02-19T14:54:27.841662+0000","last_became_active":"2024-02-19T14:54:26.736235+0000","last_became_peered":"2024-02-19T14:54:26.736235+0000","last_unstale":"2024-02-19T14:54:27.841662+0000","last_undegraded":"2024-02-19T14:54:27.841662+0000","last_fullsized":"2024-02-19T14:54:27.841662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:16.908530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"74'4","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841489+0000","last_change":"2024-02-19T14:54:25.736715+0000","last_active":"2024-02-19T14:54:27.841489+0000","last_peered":"2024-02-19T14:54:27.841489+0000","last_clean":"2024-02-19T14:54:27.841489+0000","last_became_active":"2024-02-19T14:54:25.736266+0000","last_became_peered":"2024-02-19T14:54:25.736266+0000","last_unstale":"2024-02-19T14:54:27.841489+0000","last_undegraded":"2024-02-19T14:54:27.841489+0000","last_fullsized":"2024-02-19T14:54:27.841489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:29.148205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840859+0000","last_change":"2024-02-19T14:54:26.745742+0000","last_active":"2024-02-19T14:54:27.840859+0000","last_peered":"2024-02-19T14:54:27.840859+0000","last_clean":"2024-02-19T14:54:27.840859+0000","last_became_active":"2024-02-19T14:54:26.745478+0000","last_became_peered":"2024-02-19T14:54:26.745478+0000","last_unstale":"2024-02-19T14:54:27.840859+0000","last_undegraded":"2024-02-19T14:54:27.840859+0000","last_fullsized":"2024-02-19T14:54:27.840859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"74'1","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741813+0000","last_change":"2024-02-19T14:54:25.737577+0000","last_active":"2024-02-19T14:54:27.741813+0000","last_peered":"2024-02-19T14:54:27.741813+0000","last_clean":"2024-02-19T14:54:27.741813+0000","last_became_active":"2024-02-19T14:54:25.737301+0000","last_became_peered":"2024-02-19T14:54:25.737301+0000","last_unstale":"2024-02-19T14:54:27.741813+0000","last_undegraded":"2024-02-19T14:54:27.741813+0000","last_fullsized":"2024-02-19T14:54:27.741813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:47.328330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743567+0000","last_change":"2024-02-19T14:54:26.744292+0000","last_active":"2024-02-19T14:54:27.743567+0000","last_peered":"2024-02-19T14:54:27.743567+0000","last_clean":"2024-02-19T14:54:27.743567+0000","last_became_active":"2024-02-19T14:54:26.743866+0000","last_became_peered":"2024-02-19T14:54:26.743866+0000","last_unstale":"2024-02-19T14:54:27.743567+0000","last_undegraded":"2024-02-19T14:54:27.743567+0000","last_fullsized":"2024-02-19T14:54:27.743567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:21:10.873300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"74'2","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741776+0000","last_change":"2024-02-19T14:54:25.739812+0000","last_active":"2024-02-19T14:54:27.741776+0000","last_peered":"2024-02-19T14:54:27.741776+0000","last_clean":"2024-02-19T14:54:27.741776+0000","last_became_active":"2024-02-19T14:54:25.739538+0000","last_became_peered":"2024-02-19T14:54:25.739538+0000","last_unstale":"2024-02-19T14:54:27.741776+0000","last_undegraded":"2024-02-19T14:54:27.741776+0000","last_fullsized":"2024-02-19T14:54:27.741776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:02.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743522+0000","last_change":"2024-02-19T14:54:26.738473+0000","last_active":"2024-02-19T14:54:27.743522+0000","last_peered":"2024-02-19T14:54:27.743522+0000","last_clean":"2024-02-19T14:54:27.743522+0000","last_became_active":"2024-02-19T14:54:26.738077+0000","last_became_peered":"2024-02-19T14:54:26.738077+0000","last_unstale":"2024-02-19T14:54:27.743522+0000","last_undegraded":"2024-02-19T14:54:27.743522+0000","last_fullsized":"2024-02-19T14:54:27.743522+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:26:27.849069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"74'2","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841346+0000","last_change":"2024-02-19T14:54:25.733935+0000","last_active":"2024-02-19T14:54:27.841346+0000","last_peered":"2024-02-19T14:54:27.841346+0000","last_clean":"2024-02-19T14:54:27.841346+0000","last_became_active":"2024-02-19T14:54:25.733653+0000","last_became_peered":"2024-02-19T14:54:25.733653+0000","last_unstale":"2024-02-19T14:54:27.841346+0000","last_undegraded":"2024-02-19T14:54:27.841346+0000","last_fullsized":"2024-02-19T14:54:27.841346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:52.238124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841847+0000","last_change":"2024-02-19T14:54:26.754225+0000","last_active":"2024-02-19T14:54:27.841847+0000","last_peered":"2024-02-19T14:54:27.841847+0000","last_clean":"2024-02-19T14:54:27.841847+0000","last_became_active":"2024-02-19T14:54:26.753892+0000","last_became_peered":"2024-02-19T14:54:26.753892+0000","last_unstale":"2024-02-19T14:54:27.841847+0000","last_undegraded":"2024-02-19T14:54:27.841847+0000","last_fullsized":"2024-02-19T14:54:27.841847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:02.979120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"74'1","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743495+0000","last_change":"2024-02-19T14:54:25.733786+0000","last_active":"2024-02-19T14:54:27.743495+0000","last_peered":"2024-02-19T14:54:27.743495+0000","last_clean":"2024-02-19T14:54:27.743495+0000","last_became_active":"2024-02-19T14:54:25.733327+0000","last_became_peered":"2024-02-19T14:54:25.733327+0000","last_unstale":"2024-02-19T14:54:27.743495+0000","last_undegraded":"2024-02-19T14:54:27.743495+0000","last_fullsized":"2024-02-19T14:54:27.743495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:13.186581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841789+0000","last_change":"2024-02-19T14:54:26.753347+0000","last_active":"2024-02-19T14:54:27.841789+0000","last_peered":"2024-02-19T14:54:27.841789+0000","last_clean":"2024-02-19T14:54:27.841789+0000","last_became_active":"2024-02-19T14:54:26.752855+0000","last_became_peered":"2024-02-19T14:54:26.752855+0000","last_unstale":"2024-02-19T14:54:27.841789+0000","last_undegraded":"2024-02-19T14:54:27.841789+0000","last_fullsized":"2024-02-19T14:54:27.841789+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:48.880118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"74'1","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840885+0000","last_change":"2024-02-19T14:54:25.730846+0000","last_active":"2024-02-19T14:54:27.840885+0000","last_peered":"2024-02-19T14:54:27.840885+0000","last_clean":"2024-02-19T14:54:27.840885+0000","last_became_active":"2024-02-19T14:54:25.730539+0000","last_became_peered":"2024-02-19T14:54:25.730539+0000","last_unstale":"2024-02-19T14:54:27.840885+0000","last_undegraded":"2024-02-19T14:54:27.840885+0000","last_fullsized":"2024-02-19T14:54:27.840885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:56.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.742944+0000","last_change":"2024-02-19T14:54:26.741073+0000","last_active":"2024-02-19T14:54:27.742944+0000","last_peered":"2024-02-19T14:54:27.742944+0000","last_clean":"2024-02-19T14:54:27.742944+0000","last_became_active":"2024-02-19T14:54:26.740765+0000","last_became_peered":"2024-02-19T14:54:26.740765+0000","last_unstale":"2024-02-19T14:54:27.742944+0000","last_undegraded":"2024-02-19T14:54:27.742944+0000","last_fullsized":"2024-02-19T14:54:27.742944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:57.622666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840955+0000","last_change":"2024-02-19T14:54:25.737703+0000","last_active":"2024-02-19T14:54:27.840955+0000","last_peered":"2024-02-19T14:54:27.840955+0000","last_clean":"2024-02-19T14:54:27.840955+0000","last_became_active":"2024-02-19T14:54:25.737444+0000","last_became_peered":"2024-02-19T14:54:25.737444+0000","last_unstale":"2024-02-19T14:54:27.840955+0000","last_undegraded":"2024-02-19T14:54:27.840955+0000","last_fullsized":"2024-02-19T14:54:27.840955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:26.488134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841564+0000","last_change":"2024-02-19T14:54:26.751885+0000","last_active":"2024-02-19T14:54:27.841564+0000","last_peered":"2024-02-19T14:54:27.841564+0000","last_clean":"2024-02-19T14:54:27.841564+0000","last_became_active":"2024-02-19T14:54:26.751496+0000","last_became_peered":"2024-02-19T14:54:26.751496+0000","last_unstale":"2024-02-19T14:54:27.841564+0000","last_undegraded":"2024-02-19T14:54:27.841564+0000","last_fullsized":"2024-02-19T14:54:27.841564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:57.427605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842270+0000","last_change":"2024-02-19T14:54:25.736306+0000","last_active":"2024-02-19T14:54:27.842270+0000","last_peered":"2024-02-19T14:54:27.842270+0000","last_clean":"2024-02-19T14:54:27.842270+0000","last_became_active":"2024-02-19T14:54:25.736029+0000","last_became_peered":"2024-02-19T14:54:25.736029+0000","last_unstale":"2024-02-19T14:54:27.842270+0000","last_undegraded":"2024-02-19T14:54:27.842270+0000","last_fullsized":"2024-02-19T14:54:27.842270+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:59:00.456190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841963+0000","last_change":"2024-02-19T14:54:26.743807+0000","last_active":"2024-02-19T14:54:27.841963+0000","last_peered":"2024-02-19T14:54:27.841963+0000","last_clean":"2024-02-19T14:54:27.841963+0000","last_became_active":"2024-02-19T14:54:26.743493+0000","last_became_peered":"2024-02-19T14:54:26.743493+0000","last_unstale":"2024-02-19T14:54:27.841963+0000","last_undegraded":"2024-02-19T14:54:27.841963+0000","last_fullsized":"2024-02-19T14:54:27.841963+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:38.383508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.742077+0000","last_change":"2024-02-19T14:54:25.737199+0000","last_active":"2024-02-19T14:54:27.742077+0000","last_peered":"2024-02-19T14:54:27.742077+0000","last_clean":"2024-02-19T14:54:27.742077+0000","last_became_active":"2024-02-19T14:54:25.736941+0000","last_became_peered":"2024-02-19T14:54:25.736941+0000","last_unstale":"2024-02-19T14:54:27.742077+0000","last_undegraded":"2024-02-19T14:54:27.742077+0000","last_fullsized":"2024-02-19T14:54:27.742077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:26.806558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841040+0000","last_change":"2024-02-19T14:54:26.733706+0000","last_active":"2024-02-19T14:54:27.841040+0000","last_peered":"2024-02-19T14:54:27.841040+0000","last_clean":"2024-02-19T14:54:27.841040+0000","last_became_active":"2024-02-19T14:54:26.733434+0000","last_became_peered":"2024-02-19T14:54:26.733434+0000","last_unstale":"2024-02-19T14:54:27.841040+0000","last_undegraded":"2024-02-19T14:54:27.841040+0000","last_fullsized":"2024-02-19T14:54:27.841040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:32.153897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841580+0000","last_change":"2024-02-19T14:54:25.729258+0000","last_active":"2024-02-19T14:54:27.841580+0000","last_peered":"2024-02-19T14:54:27.841580+0000","last_clean":"2024-02-19T14:54:27.841580+0000","last_became_active":"2024-02-19T14:54:25.728966+0000","last_became_peered":"2024-02-19T14:54:25.728966+0000","last_unstale":"2024-02-19T14:54:27.841580+0000","last_undegraded":"2024-02-19T14:54:27.841580+0000","last_fullsized":"2024-02-19T14:54:27.841580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:41.651494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738419+0000","last_change":"2024-02-19T14:54:26.741110+0000","last_active":"2024-02-19T14:54:27.738419+0000","last_peered":"2024-02-19T14:54:27.738419+0000","last_clean":"2024-02-19T14:54:27.738419+0000","last_became_active":"2024-02-19T14:54:26.740831+0000","last_became_peered":"2024-02-19T14:54:26.740831+0000","last_unstale":"2024-02-19T14:54:27.738419+0000","last_undegraded":"2024-02-19T14:54:27.738419+0000","last_fullsized":"2024-02-19T14:54:27.738419+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"74'3","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842740+0000","last_change":"2024-02-19T14:54:25.732654+0000","last_active":"2024-02-19T14:54:27.842740+0000","last_peered":"2024-02-19T14:54:27.842740+0000","last_clean":"2024-02-19T14:54:27.842740+0000","last_became_active":"2024-02-19T14:54:25.732388+0000","last_became_peered":"2024-02-19T14:54:25.732388+0000","last_unstale":"2024-02-19T14:54:27.842740+0000","last_undegraded":"2024-02-19T14:54:27.842740+0000","last_fullsized":"2024-02-19T14:54:27.842740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:04.044350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841008+0000","last_change":"2024-02-19T14:54:26.749111+0000","last_active":"2024-02-19T14:54:27.841008+0000","last_peered":"2024-02-19T14:54:27.841008+0000","last_clean":"2024-02-19T14:54:27.841008+0000","last_became_active":"2024-02-19T14:54:26.748830+0000","last_became_peered":"2024-02-19T14:54:26.748830+0000","last_unstale":"2024-02-19T14:54:27.841008+0000","last_undegraded":"2024-02-19T14:54:27.841008+0000","last_fullsized":"2024-02-19T14:54:27.841008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:36.425397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"74'6","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841556+0000","last_change":"2024-02-19T14:54:25.733916+0000","last_active":"2024-02-19T14:54:27.841556+0000","last_peered":"2024-02-19T14:54:27.841556+0000","last_clean":"2024-02-19T14:54:27.841556+0000","last_became_active":"2024-02-19T14:54:25.733638+0000","last_became_peered":"2024-02-19T14:54:25.733638+0000","last_unstale":"2024-02-19T14:54:27.841556+0000","last_undegraded":"2024-02-19T14:54:27.841556+0000","last_fullsized":"2024-02-19T14:54:27.841556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:26:26.843004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841994+0000","last_change":"2024-02-19T14:54:26.741929+0000","last_active":"2024-02-19T14:54:27.841994+0000","last_peered":"2024-02-19T14:54:27.841994+0000","last_clean":"2024-02-19T14:54:27.841994+0000","last_became_active":"2024-02-19T14:54:26.741662+0000","last_became_peered":"2024-02-19T14:54:26.741662+0000","last_unstale":"2024-02-19T14:54:27.841994+0000","last_undegraded":"2024-02-19T14:54:27.841994+0000","last_fullsized":"2024-02-19T14:54:27.841994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"74'2","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840909+0000","last_change":"2024-02-19T14:54:25.731407+0000","last_active":"2024-02-19T14:54:27.840909+0000","last_peered":"2024-02-19T14:54:27.840909+0000","last_clean":"2024-02-19T14:54:27.840909+0000","last_became_active":"2024-02-19T14:54:25.731062+0000","last_became_peered":"2024-02-19T14:54:25.731062+0000","last_unstale":"2024-02-19T14:54:27.840909+0000","last_undegraded":"2024-02-19T14:54:27.840909+0000","last_fullsized":"2024-02-19T14:54:27.840909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:30.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841263+0000","last_change":"2024-02-19T14:54:26.739942+0000","last_active":"2024-02-19T14:54:27.841263+0000","last_peered":"2024-02-19T14:54:27.841263+0000","last_clean":"2024-02-19T14:54:27.841263+0000","last_became_active":"2024-02-19T14:54:26.739642+0000","last_became_peered":"2024-02-19T14:54:26.739642+0000","last_unstale":"2024-02-19T14:54:27.841263+0000","last_undegraded":"2024-02-19T14:54:27.841263+0000","last_fullsized":"2024-02-19T14:54:27.841263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:16.908530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741713+0000","last_change":"2024-02-19T14:54:25.731469+0000","last_active":"2024-02-19T14:54:27.741713+0000","last_peered":"2024-02-19T14:54:27.741713+0000","last_clean":"2024-02-19T14:54:27.741713+0000","last_became_active":"2024-02-19T14:54:25.731139+0000","last_became_peered":"2024-02-19T14:54:25.731139+0000","last_unstale":"2024-02-19T14:54:27.741713+0000","last_undegraded":"2024-02-19T14:54:27.741713+0000","last_fullsized":"2024-02-19T14:54:27.741713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:09.115941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841569+0000","last_change":"2024-02-19T14:54:26.746776+0000","last_active":"2024-02-19T14:54:27.841569+0000","last_peered":"2024-02-19T14:54:27.841569+0000","last_clean":"2024-02-19T14:54:27.841569+0000","last_became_active":"2024-02-19T14:54:26.746452+0000","last_became_peered":"2024-02-19T14:54:26.746452+0000","last_unstale":"2024-02-19T14:54:27.841569+0000","last_undegraded":"2024-02-19T14:54:27.841569+0000","last_fullsized":"2024-02-19T14:54:27.841569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:05.385960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841519+0000","last_change":"2024-02-19T14:54:25.736476+0000","last_active":"2024-02-19T14:54:27.841519+0000","last_peered":"2024-02-19T14:54:27.841519+0000","last_clean":"2024-02-19T14:54:27.841519+0000","last_became_active":"2024-02-19T14:54:25.736202+0000","last_became_peered":"2024-02-19T14:54:25.736202+0000","last_unstale":"2024-02-19T14:54:27.841519+0000","last_undegraded":"2024-02-19T14:54:27.841519+0000","last_fullsized":"2024-02-19T14:54:27.841519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:01.450961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841685+0000","last_change":"2024-02-19T14:54:26.742073+0000","last_active":"2024-02-19T14:54:27.841685+0000","last_peered":"2024-02-19T14:54:27.841685+0000","last_clean":"2024-02-19T14:54:27.841685+0000","last_became_active":"2024-02-19T14:54:26.741791+0000","last_became_peered":"2024-02-19T14:54:26.741791+0000","last_unstale":"2024-02-19T14:54:27.841685+0000","last_undegraded":"2024-02-19T14:54:27.841685+0000","last_fullsized":"2024-02-19T14:54:27.841685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:42.657558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"74'8","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.840770+0000","last_change":"2024-02-19T14:54:25.737048+0000","last_active":"2024-02-19T14:54:27.840770+0000","last_peered":"2024-02-19T14:54:27.840770+0000","last_clean":"2024-02-19T14:54:27.840770+0000","last_became_active":"2024-02-19T14:54:25.736643+0000","last_became_peered":"2024-02-19T14:54:25.736643+0000","last_unstale":"2024-02-19T14:54:27.840770+0000","last_undegraded":"2024-02-19T14:54:27.840770+0000","last_fullsized":"2024-02-19T14:54:27.840770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:15.902465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738065+0000","last_change":"2024-02-19T14:54:26.751085+0000","last_active":"2024-02-19T14:54:27.738065+0000","last_peered":"2024-02-19T14:54:27.738065+0000","last_clean":"2024-02-19T14:54:27.738065+0000","last_became_active":"2024-02-19T14:54:26.750808+0000","last_became_peered":"2024-02-19T14:54:26.750808+0000","last_unstale":"2024-02-19T14:54:27.738065+0000","last_undegraded":"2024-02-19T14:54:27.738065+0000","last_fullsized":"2024-02-19T14:54:27.738065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"74'1","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.742176+0000","last_change":"2024-02-19T14:54:25.732423+0000","last_active":"2024-02-19T14:54:27.742176+0000","last_peered":"2024-02-19T14:54:27.742176+0000","last_clean":"2024-02-19T14:54:27.742176+0000","last_became_active":"2024-02-19T14:54:25.732159+0000","last_became_peered":"2024-02-19T14:54:25.732159+0000","last_unstale":"2024-02-19T14:54:27.742176+0000","last_undegraded":"2024-02-19T14:54:27.742176+0000","last_fullsized":"2024-02-19T14:54:27.742176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:07.433362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841426+0000","last_change":"2024-02-19T14:54:26.747952+0000","last_active":"2024-02-19T14:54:27.841426+0000","last_peered":"2024-02-19T14:54:27.841426+0000","last_clean":"2024-02-19T14:54:27.841426+0000","last_became_active":"2024-02-19T14:54:26.747656+0000","last_became_peered":"2024-02-19T14:54:26.747656+0000","last_unstale":"2024-02-19T14:54:27.841426+0000","last_undegraded":"2024-02-19T14:54:27.841426+0000","last_fullsized":"2024-02-19T14:54:27.841426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:50.690817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"74'2","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743412+0000","last_change":"2024-02-19T14:54:25.733175+0000","last_active":"2024-02-19T14:54:27.743412+0000","last_peered":"2024-02-19T14:54:27.743412+0000","last_clean":"2024-02-19T14:54:27.743412+0000","last_became_active":"2024-02-19T14:54:25.732709+0000","last_became_peered":"2024-02-19T14:54:25.732709+0000","last_unstale":"2024-02-19T14:54:27.743412+0000","last_undegraded":"2024-02-19T14:54:27.743412+0000","last_fullsized":"2024-02-19T14:54:27.743412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:28.981539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841660+0000","last_change":"2024-02-19T14:54:26.756834+0000","last_active":"2024-02-19T14:54:27.841660+0000","last_peered":"2024-02-19T14:54:27.841660+0000","last_clean":"2024-02-19T14:54:27.841660+0000","last_became_active":"2024-02-19T14:54:26.756548+0000","last_became_peered":"2024-02-19T14:54:26.756548+0000","last_unstale":"2024-02-19T14:54:27.841660+0000","last_undegraded":"2024-02-19T14:54:27.841660+0000","last_fullsized":"2024-02-19T14:54:27.841660+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:25.077428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"74'1","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842724+0000","last_change":"2024-02-19T14:54:25.735865+0000","last_active":"2024-02-19T14:54:27.842724+0000","last_peered":"2024-02-19T14:54:27.842724+0000","last_clean":"2024-02-19T14:54:27.842724+0000","last_became_active":"2024-02-19T14:54:25.735569+0000","last_became_peered":"2024-02-19T14:54:25.735569+0000","last_unstale":"2024-02-19T14:54:27.842724+0000","last_undegraded":"2024-02-19T14:54:27.842724+0000","last_fullsized":"2024-02-19T14:54:27.842724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:55:08.537113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738324+0000","last_change":"2024-02-19T14:54:26.750291+0000","last_active":"2024-02-19T14:54:27.738324+0000","last_peered":"2024-02-19T14:54:27.738324+0000","last_clean":"2024-02-19T14:54:27.738324+0000","last_became_active":"2024-02-19T14:54:26.750040+0000","last_became_peered":"2024-02-19T14:54:26.750040+0000","last_unstale":"2024-02-19T14:54:27.738324+0000","last_undegraded":"2024-02-19T14:54:27.738324+0000","last_fullsized":"2024-02-19T14:54:27.738324+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841101+0000","last_change":"2024-02-19T14:54:25.731254+0000","last_active":"2024-02-19T14:54:27.841101+0000","last_peered":"2024-02-19T14:54:27.841101+0000","last_clean":"2024-02-19T14:54:27.841101+0000","last_became_active":"2024-02-19T14:54:25.730971+0000","last_became_peered":"2024-02-19T14:54:25.730971+0000","last_unstale":"2024-02-19T14:54:27.841101+0000","last_undegraded":"2024-02-19T14:54:27.841101+0000","last_fullsized":"2024-02-19T14:54:27.841101+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:52.238124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841746+0000","last_change":"2024-02-19T14:54:26.746449+0000","last_active":"2024-02-19T14:54:27.841746+0000","last_peered":"2024-02-19T14:54:27.841746+0000","last_clean":"2024-02-19T14:54:27.841746+0000","last_became_active":"2024-02-19T14:54:26.746208+0000","last_became_peered":"2024-02-19T14:54:26.746208+0000","last_unstale":"2024-02-19T14:54:27.841746+0000","last_undegraded":"2024-02-19T14:54:27.841746+0000","last_fullsized":"2024-02-19T14:54:27.841746+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:38.553508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"74'2","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738285+0000","last_change":"2024-02-19T14:54:25.734037+0000","last_active":"2024-02-19T14:54:27.738285+0000","last_peered":"2024-02-19T14:54:27.738285+0000","last_clean":"2024-02-19T14:54:27.738285+0000","last_became_active":"2024-02-19T14:54:25.733736+0000","last_became_peered":"2024-02-19T14:54:25.733736+0000","last_unstale":"2024-02-19T14:54:27.738285+0000","last_undegraded":"2024-02-19T14:54:27.738285+0000","last_fullsized":"2024-02-19T14:54:27.738285+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:02.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841889+0000","last_change":"2024-02-19T14:54:26.749731+0000","last_active":"2024-02-19T14:54:27.841889+0000","last_peered":"2024-02-19T14:54:27.841889+0000","last_clean":"2024-02-19T14:54:27.841889+0000","last_became_active":"2024-02-19T14:54:26.749338+0000","last_became_peered":"2024-02-19T14:54:26.749338+0000","last_unstale":"2024-02-19T14:54:27.841889+0000","last_undegraded":"2024-02-19T14:54:27.841889+0000","last_fullsized":"2024-02-19T14:54:27.841889+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:14:43.891437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738158+0000","last_change":"2024-02-19T14:54:25.737967+0000","last_active":"2024-02-19T14:54:27.738158+0000","last_peered":"2024-02-19T14:54:27.738158+0000","last_clean":"2024-02-19T14:54:27.738158+0000","last_became_active":"2024-02-19T14:54:25.737526+0000","last_became_peered":"2024-02-19T14:54:25.737526+0000","last_unstale":"2024-02-19T14:54:27.738158+0000","last_undegraded":"2024-02-19T14:54:27.738158+0000","last_fullsized":"2024-02-19T14:54:27.738158+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:09.115941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841731+0000","last_change":"2024-02-19T14:54:26.743403+0000","last_active":"2024-02-19T14:54:27.841731+0000","last_peered":"2024-02-19T14:54:27.841731+0000","last_clean":"2024-02-19T14:54:27.841731+0000","last_became_active":"2024-02-19T14:54:26.742380+0000","last_became_peered":"2024-02-19T14:54:26.742380+0000","last_unstale":"2024-02-19T14:54:27.841731+0000","last_undegraded":"2024-02-19T14:54:27.841731+0000","last_fullsized":"2024-02-19T14:54:27.841731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:48.334394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738118+0000","last_change":"2024-02-19T14:54:25.737893+0000","last_active":"2024-02-19T14:54:27.738118+0000","last_peered":"2024-02-19T14:54:27.738118+0000","last_clean":"2024-02-19T14:54:27.738118+0000","last_became_active":"2024-02-19T14:54:25.737503+0000","last_became_peered":"2024-02-19T14:54:25.737503+0000","last_unstale":"2024-02-19T14:54:27.738118+0000","last_undegraded":"2024-02-19T14:54:27.738118+0000","last_fullsized":"2024-02-19T14:54:27.738118+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:47.328330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.742766+0000","last_change":"2024-02-19T14:54:26.743767+0000","last_active":"2024-02-19T14:54:27.742766+0000","last_peered":"2024-02-19T14:54:27.742766+0000","last_clean":"2024-02-19T14:54:27.742766+0000","last_became_active":"2024-02-19T14:54:26.743477+0000","last_became_peered":"2024-02-19T14:54:26.743477+0000","last_unstale":"2024-02-19T14:54:27.742766+0000","last_undegraded":"2024-02-19T14:54:27.742766+0000","last_fullsized":"2024-02-19T14:54:27.742766+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:48.554748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"74'2","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841179+0000","last_change":"2024-02-19T14:54:25.729454+0000","last_active":"2024-02-19T14:54:27.841179+0000","last_peered":"2024-02-19T14:54:27.841179+0000","last_clean":"2024-02-19T14:54:27.841179+0000","last_became_active":"2024-02-19T14:54:25.729052+0000","last_became_peered":"2024-02-19T14:54:25.729052+0000","last_unstale":"2024-02-19T14:54:27.841179+0000","last_undegraded":"2024-02-19T14:54:27.841179+0000","last_fullsized":"2024-02-19T14:54:27.841179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:04.379896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841768+0000","last_change":"2024-02-19T14:54:26.742043+0000","last_active":"2024-02-19T14:54:27.841768+0000","last_peered":"2024-02-19T14:54:27.841768+0000","last_clean":"2024-02-19T14:54:27.841768+0000","last_became_active":"2024-02-19T14:54:26.741725+0000","last_became_peered":"2024-02-19T14:54:26.741725+0000","last_unstale":"2024-02-19T14:54:27.841768+0000","last_undegraded":"2024-02-19T14:54:27.841768+0000","last_fullsized":"2024-02-19T14:54:27.841768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:10.122005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841839+0000","last_change":"2024-02-19T14:54:25.737062+0000","last_active":"2024-02-19T14:54:27.841839+0000","last_peered":"2024-02-19T14:54:27.841839+0000","last_clean":"2024-02-19T14:54:27.841839+0000","last_became_active":"2024-02-19T14:54:25.736783+0000","last_became_peered":"2024-02-19T14:54:25.736783+0000","last_unstale":"2024-02-19T14:54:27.841839+0000","last_undegraded":"2024-02-19T14:54:27.841839+0000","last_fullsized":"2024-02-19T14:54:27.841839+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:56.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.743164+0000","last_change":"2024-02-19T14:54:26.744132+0000","last_active":"2024-02-19T14:54:27.743164+0000","last_peered":"2024-02-19T14:54:27.743164+0000","last_clean":"2024-02-19T14:54:27.743164+0000","last_became_active":"2024-02-19T14:54:26.741010+0000","last_became_peered":"2024-02-19T14:54:26.741010+0000","last_unstale":"2024-02-19T14:54:27.743164+0000","last_undegraded":"2024-02-19T14:54:27.743164+0000","last_fullsized":"2024-02-19T14:54:27.743164+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:04.497808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741977+0000","last_change":"2024-02-19T14:54:25.736615+0000","last_active":"2024-02-19T14:54:27.741977+0000","last_peered":"2024-02-19T14:54:27.741977+0000","last_clean":"2024-02-19T14:54:27.741977+0000","last_became_active":"2024-02-19T14:54:25.736328+0000","last_became_peered":"2024-02-19T14:54:25.736328+0000","last_unstale":"2024-02-19T14:54:27.741977+0000","last_undegraded":"2024-02-19T14:54:27.741977+0000","last_fullsized":"2024-02-19T14:54:27.741977+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:07.018554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841422+0000","last_change":"2024-02-19T14:54:26.743310+0000","last_active":"2024-02-19T14:54:27.841422+0000","last_peered":"2024-02-19T14:54:27.841422+0000","last_clean":"2024-02-19T14:54:27.841422+0000","last_became_active":"2024-02-19T14:54:26.742887+0000","last_became_peered":"2024-02-19T14:54:26.742887+0000","last_unstale":"2024-02-19T14:54:27.841422+0000","last_undegraded":"2024-02-19T14:54:27.841422+0000","last_fullsized":"2024-02-19T14:54:27.841422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741669+0000","last_change":"2024-02-19T14:54:25.736615+0000","last_active":"2024-02-19T14:54:27.741669+0000","last_peered":"2024-02-19T14:54:27.741669+0000","last_clean":"2024-02-19T14:54:27.741669+0000","last_became_active":"2024-02-19T14:54:25.736314+0000","last_became_peered":"2024-02-19T14:54:25.736314+0000","last_unstale":"2024-02-19T14:54:27.741669+0000","last_undegraded":"2024-02-19T14:54:27.741669+0000","last_fullsized":"2024-02-19T14:54:27.741669+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:41.651494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841236+0000","last_change":"2024-02-19T14:54:26.742237+0000","last_active":"2024-02-19T14:54:27.841236+0000","last_peered":"2024-02-19T14:54:27.841236+0000","last_clean":"2024-02-19T14:54:27.841236+0000","last_became_active":"2024-02-19T14:54:26.741929+0000","last_became_peered":"2024-02-19T14:54:26.741929+0000","last_unstale":"2024-02-19T14:54:27.841236+0000","last_undegraded":"2024-02-19T14:54:27.841236+0000","last_fullsized":"2024-02-19T14:54:27.841236+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:47.414910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841133+0000","last_change":"2024-02-19T14:54:25.734486+0000","last_active":"2024-02-19T14:54:27.841133+0000","last_peered":"2024-02-19T14:54:27.841133+0000","last_clean":"2024-02-19T14:54:27.841133+0000","last_became_active":"2024-02-19T14:54:25.734200+0000","last_became_peered":"2024-02-19T14:54:25.734200+0000","last_unstale":"2024-02-19T14:54:27.841133+0000","last_undegraded":"2024-02-19T14:54:27.841133+0000","last_fullsized":"2024-02-19T14:54:27.841133+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:03.491743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841264+0000","last_change":"2024-02-19T14:54:26.751887+0000","last_active":"2024-02-19T14:54:27.841264+0000","last_peered":"2024-02-19T14:54:27.841264+0000","last_clean":"2024-02-19T14:54:27.841264+0000","last_became_active":"2024-02-19T14:54:26.751432+0000","last_became_peered":"2024-02-19T14:54:26.751432+0000","last_unstale":"2024-02-19T14:54:27.841264+0000","last_undegraded":"2024-02-19T14:54:27.841264+0000","last_fullsized":"2024-02-19T14:54:27.841264+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:27.494199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841940+0000","last_change":"2024-02-19T14:54:25.738928+0000","last_active":"2024-02-19T14:54:27.841940+0000","last_peered":"2024-02-19T14:54:27.841940+0000","last_clean":"2024-02-19T14:54:27.841940+0000","last_became_active":"2024-02-19T14:54:25.738674+0000","last_became_peered":"2024-02-19T14:54:25.738674+0000","last_unstale":"2024-02-19T14:54:27.841940+0000","last_undegraded":"2024-02-19T14:54:27.841940+0000","last_fullsized":"2024-02-19T14:54:27.841940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:58.792240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841242+0000","last_change":"2024-02-19T14:54:26.754376+0000","last_active":"2024-02-19T14:54:27.841242+0000","last_peered":"2024-02-19T14:54:27.841242+0000","last_clean":"2024-02-19T14:54:27.841242+0000","last_became_active":"2024-02-19T14:54:26.754102+0000","last_became_peered":"2024-02-19T14:54:26.754102+0000","last_unstale":"2024-02-19T14:54:27.841242+0000","last_undegraded":"2024-02-19T14:54:27.841242+0000","last_fullsized":"2024-02-19T14:54:27.841242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:02.882466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841789+0000","last_change":"2024-02-19T14:54:25.735756+0000","last_active":"2024-02-19T14:54:27.841789+0000","last_peered":"2024-02-19T14:54:27.841789+0000","last_clean":"2024-02-19T14:54:27.841789+0000","last_became_active":"2024-02-19T14:54:25.735486+0000","last_became_peered":"2024-02-19T14:54:25.735486+0000","last_unstale":"2024-02-19T14:54:27.841789+0000","last_undegraded":"2024-02-19T14:54:27.841789+0000","last_fullsized":"2024-02-19T14:54:27.841789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:12.694335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.842151+0000","last_change":"2024-02-19T14:54:26.743441+0000","last_active":"2024-02-19T14:54:27.842151+0000","last_peered":"2024-02-19T14:54:27.842151+0000","last_clean":"2024-02-19T14:54:27.842151+0000","last_became_active":"2024-02-19T14:54:26.743151+0000","last_became_peered":"2024-02-19T14:54:26.743151+0000","last_unstale":"2024-02-19T14:54:27.842151+0000","last_undegraded":"2024-02-19T14:54:27.842151+0000","last_fullsized":"2024-02-19T14:54:27.842151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:27.494199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.841661+0000","last_change":"2024-02-19T14:54:25.728807+0000","last_active":"2024-02-19T14:54:27.841661+0000","last_peered":"2024-02-19T14:54:27.841661+0000","last_clean":"2024-02-19T14:54:27.841661+0000","last_became_active":"2024-02-19T14:54:25.728541+0000","last_became_peered":"2024-02-19T14:54:25.728541+0000","last_unstale":"2024-02-19T14:54:27.841661+0000","last_undegraded":"2024-02-19T14:54:27.841661+0000","last_fullsized":"2024-02-19T14:54:27.841661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:12.694335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741958+0000","last_change":"2024-02-19T14:54:26.742776+0000","last_active":"2024-02-19T14:54:27.741958+0000","last_peered":"2024-02-19T14:54:27.741958+0000","last_clean":"2024-02-19T14:54:27.741958+0000","last_became_active":"2024-02-19T14:54:26.742433+0000","last_became_peered":"2024-02-19T14:54:26.742433+0000","last_unstale":"2024-02-19T14:54:27.741958+0000","last_undegraded":"2024-02-19T14:54:27.741958+0000","last_fullsized":"2024-02-19T14:54:27.741958+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.738493+0000","last_change":"2024-02-19T14:54:25.738298+0000","last_active":"2024-02-19T14:54:27.738493+0000","last_peered":"2024-02-19T14:54:27.738493+0000","last_clean":"2024-02-19T14:54:27.738493+0000","last_became_active":"2024-02-19T14:54:25.737717+0000","last_became_peered":"2024-02-19T14:54:25.737717+0000","last_unstale":"2024-02-19T14:54:27.738493+0000","last_undegraded":"2024-02-19T14:54:27.738493+0000","last_fullsized":"2024-02-19T14:54:27.738493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:07.018554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":273,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-02-19T14:54:27.741870+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:54:27.741870+0000","last_peered":"2024-02-19T14:54:27.741870+0000","last_clean":"2024-02-19T14:54:27.741870+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:54:27.741870+0000","last_undegraded":"2024-02-19T14:54:27.741870+0000","last_fullsized":"2024-02-19T14:54:27.741870+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28496,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343344,"statfs":{"total":96636764160,"available":96607584256,"internally_reserved":0,"allocated":1503232,"data_stored":1018491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.496}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.436},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.497,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.44400000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":917504,"data_stored":429713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.434},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.621},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56100000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":909312,"data_stored":428135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.498},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55800000000000005,"15min":0.56899999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36499999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.52300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":925696,"data_stored":429817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.747,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.59699999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.745}]},{"osd":1,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.627},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69399999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95799999999999996,"5min":1.038,"15min":1.038},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65700000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.83199999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.996}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":909312,"data_stored":428683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29499999999999998},"min":{"1min":0.17899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":3.694},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.309,"15min":0.30499999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":4.835}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.31},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":6.3810000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.374,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":3.6320000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49199999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":4.9889999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69099999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":4.9539999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":3.6669999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68700000000000006,"5min":1.133,"15min":1.133},"last":4.9130000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":3.7189999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":4.8799999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82999999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":3.0760000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":917504,"data_stored":428843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.437,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.42799999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.375,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79000000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84199999999999997,"5min":0.878,"15min":0.878},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94199999999999995,"5min":1.236,"15min":1.236},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.629},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.995,"15min":0.995},"last":0.56599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":925696,"data_stored":434905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33500000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.497,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33100000000000002,"15min":0.33800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.307,"15min":0.309},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.371,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.30299999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57999999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.34000000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.442},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.40000000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44400000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.497},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.42799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28484,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343356,"statfs":{"total":96636764160,"available":96607596544,"internally_reserved":0,"allocated":1499136,"data_stored":1018469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.38100000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.315,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32300000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42599999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34899999999999998,"15min":0.35199999999999998},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.42699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.32800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.41199999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57599999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.59999999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.752},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68700000000000006,"5min":0.754,"15min":0.754},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72999999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:54:30.139 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:54:30.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:54:30.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.307+0000 7fc117d08640 1 Processor -- start 2024-02-19T14:54:30.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.307+0000 7fc117d08640 1 -- start start 2024-02-19T14:54:30.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc117d08640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc110152670 0x7fc110071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc117d08640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc110071bf0 con 0x7fc1100702d0 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc117d08640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc110158a20 con 0x7fc110152a50 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc117d08640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1100c6780 con 0x7fc110152670 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc115aa4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc110152670 0x7fc110071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc115aa4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc110152670 0x7fc110071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38948/0 (socket says 172.21.15.78:38948) 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.308+0000 7fc115aa4640 1 -- 172.21.15.78:0/2070097860 learned_addr learned my addr 172.21.15.78:0/2070097860 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.309+0000 7fc115aa4640 1 -- 172.21.15.78:0/2070097860 >> v1:172.21.15.150:6789/0 conn(0x7fc110152a50 legacy=0x7fc110155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.309+0000 7fc115aa4640 1 -- 172.21.15.78:0/2070097860 >> v1:172.21.15.78:6789/0 conn(0x7fc1100702d0 legacy=0x7fc110158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.309+0000 7fc115aa4640 1 -- 172.21.15.78:0/2070097860 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc110071630 con 0x7fc110152670 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.309+0000 7fc115aa4640 1 --2- 172.21.15.78:0/2070097860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc110071050 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc104009870 tx=0x7fc10402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceda524503387b9d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc114aa2640 1 -- 172.21.15.78:0/2070097860 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10402f850 con 0x7fc110152670 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc114aa2640 1 -- 172.21.15.78:0/2070097860 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10402f9b0 con 0x7fc110152670 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc114aa2640 1 -- 172.21.15.78:0/2070097860 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10402fd40 con 0x7fc110152670 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 -- 172.21.15.78:0/2070097860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 msgr2=0x7fc110071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 --2- 172.21.15.78:0/2070097860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc110071050 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc104009870 tx=0x7fc10402ba30 comp rx=0 tx=0).stop 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 -- 172.21.15.78:0/2070097860 shutdown_connections 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 --2- 172.21.15.78:0/2070097860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc110071050 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 -- 172.21.15.78:0/2070097860 >> 172.21.15.78:0/2070097860 conn(0x7fc1100c85e0 msgr2=0x7fc1100c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 -- 172.21.15.78:0/2070097860 shutdown_connections 2024-02-19T14:54:30.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.310+0000 7fc117d08640 1 -- 172.21.15.78:0/2070097860 wait complete. 2024-02-19T14:54:30.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.311+0000 7fc117d08640 1 Processor -- start 2024-02-19T14:54:30.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.311+0000 7fc117d08640 1 -- start start 2024-02-19T14:54:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc117d08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1100702d0 0x7fc11009a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc117d08640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc1101ee370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc117d08640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc110152a50 0x7fc110150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc117d08640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1101ef040 con 0x7fc110152670 2024-02-19T14:54:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc117d08640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1101eed60 con 0x7fc1100702d0 2024-02-19T14:54:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc117d08640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1101de4b0 con 0x7fc110152a50 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc1101ee370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc1101ee370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38954/0 (socket says 172.21.15.78:38954) 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 -- 172.21.15.78:0/410838394 learned_addr learned my addr 172.21.15.78:0/410838394 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 -- 172.21.15.78:0/410838394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc110152a50 msgr2=0x7fc110150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc110152a50 0x7fc110150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 -- 172.21.15.78:0/410838394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1100702d0 msgr2=0x7fc11009a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.312+0000 7fc1152a3640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1100702d0 0x7fc11009a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc1152a3640 1 -- 172.21.15.78:0/410838394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1040094d0 con 0x7fc110152670 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc1152a3640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc1101ee370 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00c430 tx=0x7fc10c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c011790 con 0x7fc110152670 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c0118f0 con 0x7fc110152670 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1101de790 con 0x7fc110152670 2024-02-19T14:54:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c011ce0 con 0x7fc110152670 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.313+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1100ae0b0 con 0x7fc110152670 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.314+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1100676a0 con 0x7fc110152670 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.314+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc10c01c460 con 0x7fc110152670 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.315+0000 7fc1027fc640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc0f406b810 0x7fc0f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.315+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc10c08a2e0 con 0x7fc110152670 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.315+0000 7fc115aa4640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc0f406b810 0x7fc0f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.315+0000 7fc115aa4640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc0f406b810 0x7fc0f406dcd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc104036000 tx=0x7fc104039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:30.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.319+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc10c054c00 con 0x7fc110152670 2024-02-19T14:54:30.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.612+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc110074990 con 0x7fc110152670 2024-02-19T14:54:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.614+0000 7fc1027fc640 1 -- 172.21.15.78:0/410838394 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc10c0582c0 con 0x7fc110152670 2024-02-19T14:54:30.617 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:54:30.617 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:54:30.617 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:54:30.617 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:54:30.617 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:54:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:54:30.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc0f406b810 msgr2=0x7fc0f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc0f406b810 0x7fc0f406dcd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc104036000 tx=0x7fc104039040 comp rx=0 tx=0).stop 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 msgr2=0x7fc1101ee370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc1101ee370 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00c430 tx=0x7fc10c00c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 shutdown_connections 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc0f406b810 0x7fc0f406dcd0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110152670 0x7fc1101ee370 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc110152a50 0x7fc110150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 --2- 172.21.15.78:0/410838394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1100702d0 0x7fc11009a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 >> 172.21.15.78:0/410838394 conn(0x7fc1100c85e0 msgr2=0x7fc110154e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 shutdown_connections 2024-02-19T14:54:30.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.617+0000 7fc117d08640 1 -- 172.21.15.78:0/410838394 wait complete. 2024-02-19T14:54:30.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.728+0000 7f6c1530d640 1 Processor -- start 2024-02-19T14:54:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.728+0000 7f6c1530d640 1 -- start start 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c1530d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c10152700 0x7f6c100c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c1530d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 0x7f6c1015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c1530d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c101551f0 con 0x7f6c100c5730 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c1530d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c10155490 con 0x7f6c10152700 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c1530d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10155770 con 0x7f6c100c8e00 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c0e7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6c100c5730 0x7f6c1006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39672/0 (socket says 172.21.15.78:39672) 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c0e7fc640 1 -- 172.21.15.78:0/2012056197 learned_addr learned my addr 172.21.15.78:0/2012056197 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c0effd640 1 --2- 172.21.15.78:0/2012056197 >> v2:172.21.15.150:3300/0 conn(0x7f6c10152700 0x7f6c100c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 0x7f6c1015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186170959 0 0) 0x7f6c101551f0 con 0x7f6c100c5730 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.729+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf8003520 con 0x7f6c100c5730 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377739530 0 0) 0x7f6bf8003520 con 0x7f6c100c5730 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 msgr2=0x7f6c1015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 0x7f6c1015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 >> v2:172.21.15.150:3300/0 conn(0x7f6c10152700 msgr2=0x7f6c100c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 --2- 172.21.15.78:0/2012056197 >> v2:172.21.15.150:3300/0 conn(0x7f6c10152700 0x7f6c100c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c10155a50 con 0x7f6c100c5730 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f6c00001670 con 0x7f6c100c5730 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 >> v1:172.21.15.78:6789/0 conn(0x7f6c100c5730 legacy=0x7f6c1006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 0x7f6bf8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8003ef0 0x7f6bf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bf8004d00 0x7f6bf8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10155490 con 0x7f6bf8004d00 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10155770 con 0x7f6bf8003ef0 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf800b980 con 0x7f6c100c8e00 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bf8004d00 0x7f6bf8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0effd640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 0x7f6bf8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0e7fc640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8003ef0 0x7f6bf80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 msgr2=0x7f6bf8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c8e00 0x7f6bf8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8003ef0 msgr2=0x7f6bf80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8003ef0 0x7f6bf80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.730+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/2012056197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8003520 con 0x7f6bf8004d00 2024-02-19T14:54:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bf8004d00 0x7f6bf8005180 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6c08009fc0 tx=0x7f6bf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd60c51dc2db2ab6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0802cc00 con 0x7f6bf8004d00 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0800f040 con 0x7f6bf8004d00 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c0dffb640 1 -- 172.21.15.78:0/2012056197 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08034440 con 0x7f6bf8004d00 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c1530d640 1 -- 172.21.15.78:0/2012056197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bf8004d00 msgr2=0x7f6bf8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c1530d640 1 --2- 172.21.15.78:0/2012056197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bf8004d00 0x7f6bf8005180 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6c08009fc0 tx=0x7f6bf80057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.731+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/2012056197 reap_dead start 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.732+0000 7f6c1530d640 1 -- 172.21.15.78:0/2012056197 shutdown_connections 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.732+0000 7f6c1530d640 1 -- 172.21.15.78:0/2012056197 >> 172.21.15.78:0/2012056197 conn(0x7f6c100cac90 msgr2=0x7f6c100cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.732+0000 7f6c1530d640 1 -- 172.21.15.78:0/2012056197 shutdown_connections 2024-02-19T14:54:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.732+0000 7f6c1530d640 1 -- 172.21.15.78:0/2012056197 wait complete. 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.733+0000 7f6c1530d640 1 Processor -- start 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.733+0000 7f6c1530d640 1 -- start start 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.733+0000 7f6c1530d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8004d00 0x7f6c100ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.733+0000 7f6c1530d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c3730 0x7f6c10148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.733+0000 7f6c1530d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 0x7f6c101490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c1530d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c101ee9e0 con 0x7f6c100c3bb0 2024-02-19T14:54:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c1530d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c101ede10 con 0x7f6bf8004d00 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c1530d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c101ee0b0 con 0x7f6c100c3730 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 0x7f6c101490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 0x7f6c101490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38968/0 (socket says 172.21.15.78:38968) 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/289875935 learned_addr learned my addr 172.21.15.78:0/289875935 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/289875935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c3730 msgr2=0x7f6c10148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c3730 0x7f6c10148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/289875935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8004d00 msgr2=0x7f6c100ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8004d00 0x7f6c100ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 -- 172.21.15.78:0/289875935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c101ee390 con 0x7f6c100c3bb0 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c0f7fe640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 0x7f6c101490a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6c08037040 tx=0x7f6c08009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08043070 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c08034b50 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c08003990 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c101ee670 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.734+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c100c4810 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.735+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c10074990 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.736+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c0803e440 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.736+0000 7f6bf77fe640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bfc06b840 0x7f6bfc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.736+0000 7f6c0effd640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bfc06b840 0x7f6bfc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.736+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c080abbd0 con 0x7f6c100c3bb0 2024-02-19T14:54:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.737+0000 7f6c0effd640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bfc06b840 0x7f6bfc06dd00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6c04006e80 tx=0x7f6c04006df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.741+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c080764f0 con 0x7f6c100c3bb0 2024-02-19T14:54:30.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.960+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c100c8770 con 0x7f6c100c3bb0 2024-02-19T14:54:30.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.961+0000 7f6bf77fe640 1 -- 172.21.15.78:0/289875935 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f6c08039020 con 0x7f6c100c3bb0 2024-02-19T14:54:30.964 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:30.964 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:28.097183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2137063883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":2137063883},{"type":"v1","addr":"172.21.15.78:6837","nonce":2137063883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T14:54:30.964 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 51 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bfc06b840 msgr2=0x7f6bfc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bfc06b840 0x7f6bfc06dd00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6c04006e80 tx=0x7f6c04006df0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 msgr2=0x7f6c101490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 0x7f6c101490a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f6c08037040 tx=0x7f6c08009750 comp rx=0 tx=0).stop 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 shutdown_connections 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bfc06b840 0x7f6bfc06dd00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c100c3bb0 0x7f6c101490a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c100c3730 0x7f6c10148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 --2- 172.21.15.78:0/289875935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bf8004d00 0x7f6c100ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:30.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 >> 172.21.15.78:0/289875935 conn(0x7f6c100cac90 msgr2=0x7f6c101517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:30.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 shutdown_connections 2024-02-19T14:54:30.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:30.963+0000 7f6c1530d640 1 -- 172.21.15.78:0/289875935 wait complete. 2024-02-19T14:54:31.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:54:26.070082+0000', 'modified': '2024-02-19T14:54:28.097183+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6363}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6363': {'gid': 6363, 'name': 'c', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/2137063883', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 2137063883}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2137063883}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:54:31.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:54:31.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:54:31.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 Processor -- start 2024-02-19T14:54:31.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 -- start start 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0c8e00 0x7ff4ec06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff4ec06d910 0x7ff4ec06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec06cd20 con 0x7ff4ec152700 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec1551f0 con 0x7ff4ec06d910 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.483+0000 7ff4f3639640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec155490 con 0x7ff4ec0c8e00 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff4ec06d910 0x7ff4ec06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff4ec06d910 0x7ff4ec06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38978/0 (socket says 172.21.15.78:38978) 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/3528349597 learned_addr learned my addr 172.21.15.78:0/3528349597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/3528349597 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0c8e00 0x7ff4ec06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/3528349597 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0c8e00 msgr2=0x7ff4ec06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/3528349597 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0c8e00 0x7ff4ec06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/3528349597 >> v1:172.21.15.150:6789/0 conn(0x7ff4ec152700 legacy=0x7ff4ec0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.484+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/3528349597 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec155770 con 0x7ff4ec06d910 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.485+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/3528349597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec06d910 0x7ff4ec06c7e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0c7620 tx=0x7ff4e402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ea9157482f39f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.485+0000 7ff4e3fff640 1 -- 172.21.15.78:0/3528349597 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e402cb20 con 0x7ff4ec06d910 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.485+0000 7ff4e3fff640 1 -- 172.21.15.78:0/3528349597 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e402cc80 con 0x7ff4ec06d910 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.485+0000 7ff4e3fff640 1 -- 172.21.15.78:0/3528349597 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e40343f0 con 0x7ff4ec06d910 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 -- 172.21.15.78:0/3528349597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec06d910 msgr2=0x7ff4ec06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 --2- 172.21.15.78:0/3528349597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec06d910 0x7ff4ec06c7e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0c7620 tx=0x7ff4e402bf80 comp rx=0 tx=0).stop 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 -- 172.21.15.78:0/3528349597 shutdown_connections 2024-02-19T14:54:31.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 --2- 172.21.15.78:0/3528349597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec06d910 0x7ff4ec06c7e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 --2- 172.21.15.78:0/3528349597 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0c8e00 0x7ff4ec06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 -- 172.21.15.78:0/3528349597 >> 172.21.15.78:0/3528349597 conn(0x7ff4ec0cac90 msgr2=0x7ff4ec0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 -- 172.21.15.78:0/3528349597 shutdown_connections 2024-02-19T14:54:31.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.486+0000 7ff4f3639640 1 -- 172.21.15.78:0/3528349597 wait complete. 2024-02-19T14:54:31.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.487+0000 7ff4f3639640 1 Processor -- start 2024-02-19T14:54:31.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.487+0000 7ff4f3639640 1 -- start start 2024-02-19T14:54:31.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f3639640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec06d910 0x7ff4ec0ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f3639640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec0c8e00 0x7ff4ec1eb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f3639640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 0x7ff4ec0c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f3639640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec1496b0 con 0x7ff4ec152700 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f3639640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0adea0 con 0x7ff4ec0c8e00 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f3639640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0aeff0 con 0x7ff4ec06d910 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 0x7ff4ec0c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 0x7ff4ec0c0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38994/0 (socket says 172.21.15.78:38994) 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/1698841412 learned_addr learned my addr 172.21.15.78:0/1698841412 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec0c8e00 0x7ff4ec1eb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec06d910 0x7ff4ec0ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec06d910 msgr2=0x7ff4ec0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec06d910 0x7ff4ec0ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec0c8e00 msgr2=0x7ff4ec1eb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec0c8e00 0x7ff4ec1eb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/1698841412 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4009b30 con 0x7ff4ec152700 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec06d910 0x7ff4ec0ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 0x7ff4ec0c0ce0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009f40 tx=0x7ff4e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4e1ffb640 1 -- 172.21.15.78:0/1698841412 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4009510 con 0x7ff4ec152700 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.488+0000 7ff4e1ffb640 1 -- 172.21.15.78:0/1698841412 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e403e450 con 0x7ff4ec152700 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.489+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec0aff40 con 0x7ff4ec152700 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.489+0000 7ff4e1ffb640 1 -- 172.21.15.78:0/1698841412 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e403e800 con 0x7ff4ec152700 2024-02-19T14:54:31.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.489+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec0ae0f0 con 0x7ff4ec152700 2024-02-19T14:54:31.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.490+0000 7ff4e1ffb640 1 -- 172.21.15.78:0/1698841412 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff4e403e960 con 0x7ff4ec152700 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.491+0000 7ff4e1ffb640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4d006b810 0x7ff4d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.491+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4d006b810 0x7ff4d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.491+0000 7ff4e1ffb640 1 -- 172.21.15.78:0/1698841412 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4e40ad0d0 con 0x7ff4ec152700 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.491+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4d006b810 0x7ff4d006dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8001660 tx=0x7ff4e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.491+0000 7ff4f6215740 1 Processor -- start 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.491+0000 7ff4f6215740 1 -- start start 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff31740 0x55ab9ff328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 0x55ab9ff31b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff348b0 0x55ab9fef6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f6215740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9feb1cd0 con 0x55ab9fef7ae0 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fe64f20 con 0x55ab9ff348b0 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fe62450 con 0x55ab9ff31740 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 0x55ab9ff31b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 0x55ab9ff31b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39008/0 (socket says 172.21.15.78:39008) 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/854486296 learned_addr learned my addr 172.21.15.78:0/854486296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff31740 0x55ab9ff328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff348b0 0x55ab9fef6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff31740 msgr2=0x55ab9ff328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff31740 0x55ab9ff328b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff348b0 msgr2=0x55ab9fef6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff348b0 0x55ab9fef6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.492+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/854486296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab9fe12f20 con 0x55ab9fef7ae0 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.493+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 0x55ab9ff31b20 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc007960 tx=0x7ff4dc007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da6af0e7dc4be69 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.493+0000 7ff4d6ffd640 1 -- 172.21.15.78:0/854486296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc004300 con 0x55ab9fef7ae0 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.493+0000 7ff4d6ffd640 1 -- 172.21.15.78:0/854486296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc004460 con 0x55ab9fef7ae0 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.493+0000 7ff4d6ffd640 1 -- 172.21.15.78:0/854486296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc010af0 con 0x55ab9fef7ae0 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.493+0000 7ff4f6215740 1 -- 172.21.15.78:0/854486296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 msgr2=0x55ab9ff31b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.493+0000 7ff4f6215740 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 0x55ab9ff31b20 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc007960 tx=0x7ff4dc007e20 comp rx=0 tx=0).stop 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 -- 172.21.15.78:0/854486296 shutdown_connections 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9fef7ae0 0x55ab9ff31b20 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff348b0 0x55ab9fef6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 --2- 172.21.15.78:0/854486296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff31740 0x55ab9ff328b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 -- 172.21.15.78:0/854486296 >> 172.21.15.78:0/854486296 conn(0x55ab9fef3120 msgr2=0x55ab9ff17b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 -- 172.21.15.78:0/854486296 shutdown_connections 2024-02-19T14:54:31.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 -- 172.21.15.78:0/854486296 wait complete. 2024-02-19T14:54:31.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.494+0000 7ff4f6215740 1 -- start start 2024-02-19T14:54:31.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.495+0000 7ff4f6215740 1 Processor -- start 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff2f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9fef6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fe5dc30 con 0x55ab9ff31740 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fe02d30 con 0x55ab9ff14e30 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fb31310 con 0x55ab9ff348b0 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f6215740 10 client.0 _ll_register_callbacks cb 0x55ab9fe0ad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:54:31.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39022/0 (socket says 172.21.15.78:39022) 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 -- 172.21.15.78:0/560370036 learned_addr learned my addr 172.21.15.78:0/560370036 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff2f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9fef6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 -- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 msgr2=0x55ab9ff2f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff2f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 -- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 msgr2=0x55ab9fef6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9fef6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 -- 172.21.15.78:0/560370036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc00e040 con 0x55ab9ff31740 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef53d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8001e50 tx=0x7ff4e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f117682aab15a538 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.496+0000 7ff4cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4cbfff640 1 -- 172.21.15.78:0/560370036 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8003a40 con 0x55ab9ff31740 2024-02-19T14:54:31.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4cbfff640 1 -- 172.21.15.78:0/560370036 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4e8022c20 con 0x55ab9ff31740 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab9fb5d6c0 con 0x55ab9ff31740 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4cbfff640 1 -- 172.21.15.78:0/560370036 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4e8022de0 con 0x55ab9ff31740 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 10 client.6504 fetch_fsmap learned FSMap version 51 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55ab9fe5c2b0 con 0x55ab9ff31740 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4cbfff640 1 -- 172.21.15.78:0/560370036 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ab9fb5d6c0 con 0x55ab9ff31740 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4cbfff640 1 -- 172.21.15.78:0/560370036 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff4e8003ba0 con 0x55ab9ff31740 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 10 client.6504 fetch_fsmap finished waiting for FSMap version 51 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 10 client.6504 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 20 client.6504 populate_metadata read hostname 'smithi078' 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff46150 0x55ab9ff46530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 4 client.6504 mds_command: new command op to 6363 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 --> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab9fbae270 con 0x55ab9ff46150 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff46150 0x55ab9ff46530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.497+0000 7ff4cbfff640 10 client.6504 ms_handle_connect on v2:172.21.15.78:6836/2137063883 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff46150 0x55ab9ff46530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=967631a179926310 server_cookie=67953a4e4dfa770a in_seq=0 out_seq=0 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4cbfff640 1 -- 172.21.15.78:0/560370036 <== mds.0 v2:172.21.15.78:6836/2137063883 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ab9fbae270 con 0x55ab9ff46150 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4cbfff640 10 client.6504 handle_command_reply: tid=0 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 client.6504 shutdown 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 msgr2=0x55ab9fef53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef53d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8001e50 tx=0x7ff4e8000bc0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 shutdown_connections 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff46150 0x55ab9ff46530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9fef6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4cbfff640 0 client.6504 ms_handle_reset on v2:172.21.15.78:6836/2137063883 2024-02-19T14:54:31.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff2f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 --2- 172.21.15.78:0/560370036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef53d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 >> 172.21.15.78:0/560370036 conn(0x55ab9fef7ae0 msgr2=0x55ab9ff32500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 shutdown_connections 2024-02-19T14:54:31.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.498+0000 7ff4f6215740 1 -- 172.21.15.78:0/560370036 wait complete. 2024-02-19T14:54:31.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.499+0000 7ff4f6215740 20 client.6504 trim_cache size 0 max 16384 2024-02-19T14:54:31.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.515+0000 7ff4f6215740 1 Processor -- start 2024-02-19T14:54:31.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.515+0000 7ff4f6215740 1 -- start start 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.515+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff14e30 0x55ab9ff2c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.515+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff31740 0x55ab9ff32e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 0x55ab9fef1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f6215740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fde3490 con 0x55ab9ff348b0 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9ff301e0 con 0x55ab9ff31740 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ab9ff30b10 con 0x55ab9ff14e30 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 0x55ab9fef1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 0x55ab9fef1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39034/0 (socket says 172.21.15.78:39034) 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/94852530 learned_addr learned my addr 172.21.15.78:0/94852530 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff31740 0x55ab9ff32e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff14e30 0x55ab9ff2c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff14e30 msgr2=0x55ab9ff2c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff14e30 0x55ab9ff2c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff31740 msgr2=0x55ab9ff32e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff31740 0x55ab9ff32e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 -- 172.21.15.78:0/94852530 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab9fdf8a20 con 0x55ab9ff348b0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff31740 0x55ab9ff32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff14e30 0x55ab9ff2c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 0x55ab9fef1a60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff4e407b800 tx=0x7ff4e4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78885da41f5802ef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4d67fc640 1 -- 172.21.15.78:0/94852530 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e407b870 con 0x55ab9ff348b0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4d67fc640 1 -- 172.21.15.78:0/94852530 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e40a7020 con 0x55ab9ff348b0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.516+0000 7ff4d67fc640 1 -- 172.21.15.78:0/94852530 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e407f9c0 con 0x55ab9ff348b0 2024-02-19T14:54:31.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 -- 172.21.15.78:0/94852530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 msgr2=0x55ab9fef1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 0x55ab9fef1a60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff4e407b800 tx=0x7ff4e4009040 comp rx=0 tx=0).stop 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 -- 172.21.15.78:0/94852530 shutdown_connections 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff348b0 0x55ab9fef1a60 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff31740 0x55ab9ff32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 --2- 172.21.15.78:0/94852530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff14e30 0x55ab9ff2c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 -- 172.21.15.78:0/94852530 >> 172.21.15.78:0/94852530 conn(0x55ab9fef7ae0 msgr2=0x55ab9ff2cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 -- 172.21.15.78:0/94852530 shutdown_connections 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 -- 172.21.15.78:0/94852530 wait complete. 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 -- start start 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.517+0000 7ff4f6215740 1 Processor -- start 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9ff2ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff9f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9ff2bfd0 con 0x55ab9ff31740 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab9ff2b540 con 0x55ab9ff14e30 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ab9fef0400 con 0x55ab9ff348b0 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f6215740 10 client.0 _ll_register_callbacks cb 0x7ff4e80013a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f0bd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f0bd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39050/0 (socket says 172.21.15.78:39050) 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/4156692615 learned_addr learned my addr 172.21.15.78:0/4156692615 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f1bd6640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff9f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.518+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9ff2ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 msgr2=0x55ab9ff9f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff9f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 msgr2=0x55ab9ff2ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9ff2ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f0bd4640 1 -- 172.21.15.78:0/4156692615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec0ae0f0 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f0bd4640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef12f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc017040 tx=0x7ff4dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d347a9b6c263b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4cbfff640 1 -- 172.21.15.78:0/4156692615 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc024040 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4cbfff640 1 -- 172.21.15.78:0/4156692615 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4dc02a030 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4dc004460 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4cbfff640 1 -- 172.21.15.78:0/4156692615 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4dc02a1f0 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 10 client.6510 fetch_fsmap learned FSMap version 51 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55ab9fe1e780 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4cbfff640 1 -- 172.21.15.78:0/4156692615 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc004460 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4cbfff640 1 -- 172.21.15.78:0/4156692615 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff4dc010b80 con 0x55ab9ff31740 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 10 client.6510 fetch_fsmap finished waiting for FSMap version 51 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 10 client.6510 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:54:31.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 20 client.6510 populate_metadata read hostname 'smithi078' 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff9f650 0x55ab9ff28510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 4 client.6510 mds_command: new command op to 6363 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.519+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 --> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ab9fb9a610 con 0x55ab9ff9f650 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff9f650 0x55ab9ff28510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4cbfff640 10 client.6510 ms_handle_connect on v2:172.21.15.78:6836/2137063883 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f13d5640 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff9f650 0x55ab9ff28510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be7e304e2334f2c8 server_cookie=fe7c932b4f55a440 in_seq=0 out_seq=0 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4cbfff640 1 -- 172.21.15.78:0/4156692615 <== mds.0 v2:172.21.15.78:6836/2137063883 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ab9fb9a610 con 0x55ab9ff9f650 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4cbfff640 10 client.6510 handle_command_reply: tid=0 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 client.6510 shutdown 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 msgr2=0x55ab9fef12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef12f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc017040 tx=0x7ff4dc004730 comp rx=0 tx=0).stop 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 shutdown_connections 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:6836/2137063883,v1:172.21.15.78:6837/2137063883] conn(0x55ab9ff9f650 0x55ab9ff28510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ab9ff31740 0x55ab9fef12f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4cbfff640 0 client.6510 ms_handle_reset on v2:172.21.15.78:6836/2137063883 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55ab9ff348b0 0x55ab9ff9f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 --2- 172.21.15.78:0/4156692615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab9ff14e30 0x55ab9ff2ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 >> 172.21.15.78:0/4156692615 conn(0x55ab9fef7ae0 msgr2=0x55ab9ff29f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 shutdown_connections 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.520+0000 7ff4f6215740 1 -- 172.21.15.78:0/4156692615 wait complete. 2024-02-19T14:54:31.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f6215740 20 client.6510 trim_cache size 0 max 16384 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 6363, 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.7847637680000004, 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 51, 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 75, 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 75, 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 9.9106819589999997 2024-02-19T14:54:31.532 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:54:31.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4d006b810 msgr2=0x7ff4d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4d006b810 0x7ff4d006dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8001660 tx=0x7ff4e8009040 comp rx=0 tx=0).stop 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 msgr2=0x7ff4ec0c0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 0x7ff4ec0c0ce0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009f40 tx=0x7ff4e4004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 shutdown_connections 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4d006b810 0x7ff4d006dcd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4ec152700 0x7ff4ec0c0ce0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec0c8e00 0x7ff4ec1eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 --2- 172.21.15.78:0/1698841412 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec06d910 0x7ff4ec0ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 >> 172.21.15.78:0/1698841412 conn(0x7ff4ec0cac90 msgr2=0x7ff4ec0c53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 shutdown_connections 2024-02-19T14:54:31.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.521+0000 7ff4f3639640 1 -- 172.21.15.78:0/1698841412 wait complete. 2024-02-19T14:54:31.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.634+0000 7ffa65f1e640 1 Processor -- start 2024-02-19T14:54:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.634+0000 7ffa65f1e640 1 -- start start 2024-02-19T14:54:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa65f1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffa600c5560 0x7ffa600c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa65f1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa65f1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa600c65a0 con 0x7ffa600c7910 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa65f1e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa600c7530 con 0x7ffa600c5560 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa65f1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa601537b0 con 0x7ffa60070720 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffa600c5560 0x7ffa600c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffa600c5560 0x7ffa600c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39054/0 (socket says 172.21.15.78:39054) 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 -- 172.21.15.78:0/1871931104 learned_addr learned my addr 172.21.15.78:0/1871931104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5effd640 1 --1- 172.21.15.78:0/1871931104 >> v1:172.21.15.78:6789/0 conn(0x7ffa600c7910 0x7ffa60154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39688/0 (socket says 172.21.15.78:39688) 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5ffff640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 -- 172.21.15.78:0/1871931104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 msgr2=0x7ffa60070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 -- 172.21.15.78:0/1871931104 >> v1:172.21.15.78:6789/0 conn(0x7ffa600c7910 legacy=0x7ffa60154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5f7fe640 1 -- 172.21.15.78:0/1871931104 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60153a90 con 0x7ffa600c5560 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5e7fc640 1 -- 172.21.15.78:0/1871931104 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264461781 0 0) 0x7ffa600c65a0 con 0x7ffa600c7910 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.635+0000 7ffa5ffff640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.636+0000 7ffa5f7fe640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c5560 0x7ffa600c6ff0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009870 tx=0x7ffa5002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14082759f6a890f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.636+0000 7ffa5e7fc640 1 -- 172.21.15.78:0/1871931104 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa500317c0 con 0x7ffa600c5560 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.636+0000 7ffa5e7fc640 1 -- 172.21.15.78:0/1871931104 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa50031920 con 0x7ffa600c5560 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.636+0000 7ffa5e7fc640 1 -- 172.21.15.78:0/1871931104 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50031cb0 con 0x7ffa600c5560 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.636+0000 7ffa65f1e640 1 -- 172.21.15.78:0/1871931104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c5560 msgr2=0x7ffa600c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.636+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c5560 0x7ffa600c6ff0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009870 tx=0x7ffa5002b860 comp rx=0 tx=0).stop 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.637+0000 7ffa65f1e640 1 -- 172.21.15.78:0/1871931104 shutdown_connections 2024-02-19T14:54:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.637+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.637+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/1871931104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c5560 0x7ffa600c6ff0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.637+0000 7ffa65f1e640 1 -- 172.21.15.78:0/1871931104 >> 172.21.15.78:0/1871931104 conn(0x7ffa6006b4b0 msgr2=0x7ffa600c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.637+0000 7ffa65f1e640 1 -- 172.21.15.78:0/1871931104 shutdown_connections 2024-02-19T14:54:31.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.637+0000 7ffa65f1e640 1 -- 172.21.15.78:0/1871931104 wait complete. 2024-02-19T14:54:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.638+0000 7ffa65f1e640 1 Processor -- start 2024-02-19T14:54:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.638+0000 7ffa65f1e640 1 -- start start 2024-02-19T14:54:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.638+0000 7ffa65f1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.638+0000 7ffa65f1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa600c5560 0x7ffa60146e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.638+0000 7ffa65f1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 0x7ffa60147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.638+0000 7ffa65f1e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa601ee660 con 0x7ffa600c7910 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa65f1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa601eeb90 con 0x7ffa600c5560 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa65f1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa601ee8b0 con 0x7ffa60070720 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 0x7ffa60147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 0x7ffa60147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39070/0 (socket says 172.21.15.78:39070) 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 -- 172.21.15.78:0/3121611456 learned_addr learned my addr 172.21.15.78:0/3121611456 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5f7fe640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5effd640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa600c5560 0x7ffa60146e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 -- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 msgr2=0x7ffa60154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 -- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa600c5560 msgr2=0x7ffa60146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa600c5560 0x7ffa60146e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 -- 172.21.15.78:0/3121611456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500094d0 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5f7fe640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5effd640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa600c5560 0x7ffa60146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa5ffff640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 0x7ffa60147350 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800cd50 tx=0x7ffa5800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800f680 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5800f800 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800fbb0 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa600c5ac0 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.639+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa600a8770 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.640+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa60074990 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.641+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffa5800b040 con 0x7ffa600c7910 2024-02-19T14:54:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.641+0000 7ffa47fff640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffa4c06b810 0x7ffa4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.641+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa5808ad30 con 0x7ffa600c7910 2024-02-19T14:54:31.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.641+0000 7ffa5f7fe640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffa4c06b810 0x7ffa4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.642+0000 7ffa5f7fe640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffa4c06b810 0x7ffa4c06dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ffa50036000 tx=0x7ffa5003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.646+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffa58055650 con 0x7ffa600c7910 2024-02-19T14:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.861+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa600c76f0 con 0x7ffa600c7910 2024-02-19T14:54:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.861+0000 7ffa47fff640 1 -- 172.21.15.78:0/3121611456 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4940 (secure 0 0 0) 0x7ffa58058d10 con 0x7ffa600c7910 2024-02-19T14:54:31.864 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:31.864 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":51,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6360,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:28.097183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2137063883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":2137063883},{"type":"v1","addr":"172.21.15.78:6837","nonce":2137063883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-19T14:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 51 2024-02-19T14:54:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffa4c06b810 msgr2=0x7ffa4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffa4c06b810 0x7ffa4c06dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ffa50036000 tx=0x7ffa5003d040 comp rx=0 tx=0).stop 2024-02-19T14:54:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 msgr2=0x7ffa60147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 0x7ffa60147350 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800cd50 tx=0x7ffa5800a650 comp rx=0 tx=0).stop 2024-02-19T14:54:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 shutdown_connections 2024-02-19T14:54:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffa4c06b810 0x7ffa4c06dcd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa600c7910 0x7ffa60147350 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa600c5560 0x7ffa60146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 --2- 172.21.15.78:0/3121611456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa60070720 0x7ffa60154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 >> 172.21.15.78:0/3121611456 conn(0x7ffa6006b4b0 msgr2=0x7ffa600c4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.863+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 shutdown_connections 2024-02-19T14:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.864+0000 7ffa65f1e640 1 -- 172.21.15.78:0/3121611456 wait complete. 2024-02-19T14:54:31.877 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 5, "epoch": 51, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 5, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:54:26.070082+0000", "damaged": [], "data_pools": [ 12 ], "enabled": true, "epoch": 51, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6363": { "addr": "172.21.15.78:6837/2137063883", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 2137063883, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2137063883, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6363, "incarnation": 49, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 11, "modified": "2024-02-19T14:54:28.097183+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6363 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/20921361", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 20921361, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 20921361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 48, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5354, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/1628693681", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1628693681, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1628693681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 48, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5357, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6835/4201159351", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 4201159351, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 4201159351, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 48, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6360, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:54:31.878 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:54:31.878 INFO:tasks.cephfs.test_failover:current = [{'gid': 6360, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/4201159351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 4201159351}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 4201159351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 48}, {'gid': 5357, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1628693681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1628693681}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1628693681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 48}, {'gid': 6363, 'name': 'c', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/2137063883', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 2137063883}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2137063883}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5354, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/20921361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 20921361}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 20921361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 48}] 2024-02-19T14:54:31.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-02-19T14:54:31.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.979+0000 7fc9450b5640 1 Processor -- start 2024-02-19T14:54:31.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.979+0000 7fc9450b5640 1 -- start start 2024-02-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc9450b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc940152700 0x7fc9400c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc9450b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc94006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc9450b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9400c8e00 0x7fc94006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc9450b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94006cd20 con 0x7fc9400c8e00 2024-02-19T14:54:31.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc9450b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9401551f0 con 0x7fc940152700 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc9450b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc940155490 con 0x7fc9400c5730 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc93f59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9400c8e00 0x7fc94006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc93f59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9400c8e00 0x7fc94006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39086/0 (socket says 172.21.15.78:39086) 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.980+0000 7fc93f59e640 1 -- 172.21.15.78:0/3571652322 learned_addr learned my addr 172.21.15.78:0/3571652322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93ed9d640 1 --2- 172.21.15.78:0/3571652322 >> v2:172.21.15.150:3300/0 conn(0x7fc940152700 0x7fc9400c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc937fff640 1 --2- 172.21.15.78:0/3571652322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc94006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93f59e640 1 -- 172.21.15.78:0/3571652322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 msgr2=0x7fc94006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93f59e640 1 --2- 172.21.15.78:0/3571652322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc94006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93f59e640 1 -- 172.21.15.78:0/3571652322 >> v2:172.21.15.150:3300/0 conn(0x7fc940152700 msgr2=0x7fc9400c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93f59e640 1 --2- 172.21.15.78:0/3571652322 >> v2:172.21.15.150:3300/0 conn(0x7fc940152700 0x7fc9400c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93f59e640 1 -- 172.21.15.78:0/3571652322 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc940155770 con 0x7fc9400c8e00 2024-02-19T14:54:31.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93ed9d640 1 --2- 172.21.15.78:0/3571652322 >> v2:172.21.15.150:3300/0 conn(0x7fc940152700 0x7fc9400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:31.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.981+0000 7fc93f59e640 1 --2- 172.21.15.78:0/3571652322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9400c8e00 0x7fc94006c7e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc94008bad0 tx=0x7fc93802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f03e4ca5554e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc93e59c640 1 -- 172.21.15.78:0/3571652322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93802ca80 con 0x7fc9400c8e00 2024-02-19T14:54:31.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc93e59c640 1 -- 172.21.15.78:0/3571652322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93802cbe0 con 0x7fc9400c8e00 2024-02-19T14:54:31.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc93e59c640 1 -- 172.21.15.78:0/3571652322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9380343f0 con 0x7fc9400c8e00 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 -- 172.21.15.78:0/3571652322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9400c8e00 msgr2=0x7fc94006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 --2- 172.21.15.78:0/3571652322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9400c8e00 0x7fc94006c7e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc94008bad0 tx=0x7fc93802bee0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 -- 172.21.15.78:0/3571652322 shutdown_connections 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 --2- 172.21.15.78:0/3571652322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9400c8e00 0x7fc94006c7e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 --2- 172.21.15.78:0/3571652322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc94006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 --2- 172.21.15.78:0/3571652322 >> v2:172.21.15.150:3300/0 conn(0x7fc940152700 0x7fc9400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 -- 172.21.15.78:0/3571652322 >> 172.21.15.78:0/3571652322 conn(0x7fc9400cac90 msgr2=0x7fc9400cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 -- 172.21.15.78:0/3571652322 shutdown_connections 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.982+0000 7fc9450b5640 1 -- 172.21.15.78:0/3571652322 wait complete. 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 Processor -- start 2024-02-19T14:54:31.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 -- start start 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc9401edd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9400c8e00 0x7fc9400c36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 0x7fc9400c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400c4120 con 0x7fc940152700 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400c4930 con 0x7fc9400c8e00 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc9450b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400acf50 con 0x7fc9400c5730 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc93f59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 0x7fc9400c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc93f59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 0x7fc9400c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39092/0 (socket says 172.21.15.78:39092) 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.984+0000 7fc93f59e640 1 -- 172.21.15.78:0/2720072794 learned_addr learned my addr 172.21.15.78:0/2720072794 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93ed9d640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc9401edd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc937fff640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9400c8e00 0x7fc9400c36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93f59e640 1 -- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 msgr2=0x7fc9401edd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93f59e640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc9401edd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93f59e640 1 -- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9400c8e00 msgr2=0x7fc9400c36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93f59e640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9400c8e00 0x7fc9400c36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93f59e640 1 -- 172.21.15.78:0/2720072794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938009b30 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc93f59e640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 0x7fc9400c3be0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc938036040 tx=0x7fc938004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938009510 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93803e450 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93803e7e0 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9401eea40 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.985+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9400acbe0 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.986+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc940074990 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.986+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc93802cd50 con 0x7fc940152700 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.987+0000 7fc9377fe640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc92006b840 0x7fc92006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.987+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc9380acb70 con 0x7fc940152700 2024-02-19T14:54:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.987+0000 7fc93ed9d640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc92006b840 0x7fc92006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.987+0000 7fc93ed9d640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc92006b840 0x7fc92006dd00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc930006fd0 tx=0x7fc930008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:31.993 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:54:31.989+0000 7f4264c23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:31.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:31.991+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc938077490 con 0x7fc940152700 2024-02-19T14:54:32.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.170+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fc9400c8770 con 0x7fc940152700 2024-02-19T14:54:32.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.173+0000 7fc9377fe640 1 -- 172.21.15.78:0/2720072794 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v24)=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7fc93807ab50 con 0x7fc940152700 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc92006b840 msgr2=0x7fc92006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc92006b840 0x7fc92006dd00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc930006fd0 tx=0x7fc930008040 comp rx=0 tx=0).stop 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 msgr2=0x7fc9400c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 0x7fc9400c3be0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc938036040 tx=0x7fc938004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 shutdown_connections 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc92006b840 0x7fc92006dd00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc940152700 0x7fc9400c3be0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9400c8e00 0x7fc9400c36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 --2- 172.21.15.78:0/2720072794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9400c5730 0x7fc9401edd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 >> 172.21.15.78:0/2720072794 conn(0x7fc9400cac90 msgr2=0x7fc94006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 shutdown_connections 2024-02-19T14:54:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.176+0000 7fc9450b5640 1 -- 172.21.15.78:0/2720072794 wait complete. 2024-02-19T14:54:32.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-02-19T14:54:32.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.292+0000 7f313593e640 1 Processor -- start 2024-02-19T14:54:32.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.292+0000 7f313593e640 1 -- start start 2024-02-19T14:54:32.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f313593e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f313014cd50 0x7f31300c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f313593e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31301563b0 con 0x7f31301537a0 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f313593e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3130153cd0 con 0x7f3130156860 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f313593e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313006b6e0 con 0x7f313014cd50 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f312effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f313014cd50 0x7f31300c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f3127fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f31301537a0 0x7f31300c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39700/0 (socket says 172.21.15.78:39700) 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f3127fff640 1 -- 172.21.15.78:0/3800046812 learned_addr learned my addr 172.21.15.78:0/3800046812 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f312effd640 1 -- 172.21.15.78:0/3800046812 >> v1:172.21.15.150:6789/0 conn(0x7f3130156860 legacy=0x7f3130156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f312effd640 1 -- 172.21.15.78:0/3800046812 >> v1:172.21.15.78:6789/0 conn(0x7f31301537a0 legacy=0x7f31300c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f312effd640 1 -- 172.21.15.78:0/3800046812 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313006b980 con 0x7f313014cd50 2024-02-19T14:54:32.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.293+0000 7f312e7fc640 1 -- 172.21.15.78:0/3800046812 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530956821 0 0) 0x7f31301563b0 con 0x7f31301537a0 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.294+0000 7f312effd640 1 --2- 172.21.15.78:0/3800046812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300c4cd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f31200097a0 tx=0x7f312002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae3e2a2c2d9fde3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.294+0000 7f312e7fc640 1 -- 172.21.15.78:0/3800046812 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312002f050 con 0x7f313014cd50 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.294+0000 7f312e7fc640 1 -- 172.21.15.78:0/3800046812 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3120031700 con 0x7f313014cd50 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.294+0000 7f312e7fc640 1 -- 172.21.15.78:0/3800046812 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120031a90 con 0x7f313014cd50 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.294+0000 7f313593e640 1 -- 172.21.15.78:0/3800046812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 msgr2=0x7f31300c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.294+0000 7f313593e640 1 --2- 172.21.15.78:0/3800046812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300c4cd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f31200097a0 tx=0x7f312002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.295+0000 7f313593e640 1 -- 172.21.15.78:0/3800046812 shutdown_connections 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.295+0000 7f313593e640 1 --2- 172.21.15.78:0/3800046812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300c4cd0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.295+0000 7f313593e640 1 -- 172.21.15.78:0/3800046812 >> 172.21.15.78:0/3800046812 conn(0x7f3130066e70 msgr2=0x7f31300a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.295+0000 7f313593e640 1 -- 172.21.15.78:0/3800046812 shutdown_connections 2024-02-19T14:54:32.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.295+0000 7f313593e640 1 -- 172.21.15.78:0/3800046812 wait complete. 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 Processor -- start 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 -- start start 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300aa850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31301537a0 0x7f31300c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3130156860 0x7f313014bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31301504c0 con 0x7f313014cd50 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130150910 con 0x7f31301537a0 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f313593e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313014ae70 con 0x7f3130156860 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f312effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f312effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300aa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39098/0 (socket says 172.21.15.78:39098) 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.296+0000 7f312effd640 1 -- 172.21.15.78:0/962570050 learned_addr learned my addr 172.21.15.78:0/962570050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312f7fe640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3130156860 0x7f313014bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f3127fff640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31301537a0 0x7f31300c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312effd640 1 -- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3130156860 msgr2=0x7f313014bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312effd640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3130156860 0x7f313014bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312effd640 1 -- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31301537a0 msgr2=0x7f31300c0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312effd640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31301537a0 0x7f31300c0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312effd640 1 -- 172.21.15.78:0/962570050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120009400 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312f7fe640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3130156860 0x7f313014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f3127fff640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31301537a0 0x7f31300c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312effd640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300aa850 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3120000c00 tx=0x7f31200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120033c30 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3120033d90 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313014b150 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312003d6b0 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.297+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313014b710 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.298+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130074990 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.299+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f312003a040 con 0x7f313014cd50 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.299+0000 7f312cff9640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f311006b840 0x7f311006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.299+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f31200ac0c0 con 0x7f313014cd50 2024-02-19T14:54:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.299+0000 7f3127fff640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f311006b840 0x7f311006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.299+0000 7f3127fff640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f311006b840 0x7f311006dd00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f31301ebd70 tx=0x7f3118009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.303+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31200769e0 con 0x7f313014cd50 2024-02-19T14:54:32.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.483+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f31300c4230 con 0x7f313014cd50 2024-02-19T14:54:32.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.486+0000 7f312cff9640 1 -- 172.21.15.78:0/962570050 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v25)=0 v25) v1 ==== 117+0+0 (secure 0 0 0) 0x7f312007a0a0 con 0x7f313014cd50 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f311006b840 msgr2=0x7f311006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f311006b840 0x7f311006dd00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f31301ebd70 tx=0x7f3118009040 comp rx=0 tx=0).stop 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 msgr2=0x7f31300aa850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300aa850 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3120000c00 tx=0x7f31200046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 shutdown_connections 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f311006b840 0x7f311006dd00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3130156860 0x7f313014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31301537a0 0x7f31300c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 --2- 172.21.15.78:0/962570050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f313014cd50 0x7f31300aa850 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 >> 172.21.15.78:0/962570050 conn(0x7f3130066e70 msgr2=0x7f31300a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 shutdown_connections 2024-02-19T14:54:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.488+0000 7f313593e640 1 -- 172.21.15.78:0/962570050 wait complete. 2024-02-19T14:54:32.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-02-19T14:54:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.602+0000 7f874c9c2640 1 Processor -- start 2024-02-19T14:54:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.602+0000 7f874c9c2640 1 -- start start 2024-02-19T14:54:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f874c9c2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87480c8e00 0x7f874806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f874c9c2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f874806d910 0x7f874806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f874c9c2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874806cd20 con 0x7f8748152700 2024-02-19T14:54:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f874c9c2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87481551f0 con 0x7f874806d910 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f874c9c2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8748155490 con 0x7f87480c8e00 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f8746d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f874806d910 0x7f874806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f8746d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f874806d910 0x7f874806c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39106/0 (socket says 172.21.15.78:39106) 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.603+0000 7f8746d9d640 1 -- 172.21.15.78:0/2334737852 learned_addr learned my addr 172.21.15.78:0/2334737852 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8746d9d640 1 -- 172.21.15.78:0/2334737852 >> v2:172.21.15.150:3300/0 conn(0x7f87480c8e00 msgr2=0x7f874806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8745d9b640 1 --2- 172.21.15.78:0/2334737852 >> v2:172.21.15.150:3300/0 conn(0x7f87480c8e00 0x7f874806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8746d9d640 1 --2- 172.21.15.78:0/2334737852 >> v2:172.21.15.150:3300/0 conn(0x7f87480c8e00 0x7f874806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8746d9d640 1 -- 172.21.15.78:0/2334737852 >> v1:172.21.15.150:6789/0 conn(0x7f8748152700 legacy=0x7f87480c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8746d9d640 1 -- 172.21.15.78:0/2334737852 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748155770 con 0x7f874806d910 2024-02-19T14:54:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8745d9b640 1 --2- 172.21.15.78:0/2334737852 >> v2:172.21.15.150:3300/0 conn(0x7f87480c8e00 0x7f874806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:32.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.604+0000 7f8746d9d640 1 --2- 172.21.15.78:0/2334737852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f874806d910 0x7f874806c7e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f87480c7620 tx=0x7f873802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d573f2ae08c05a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874559a640 1 -- 172.21.15.78:0/2334737852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738031870 con 0x7f874806d910 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874559a640 1 -- 172.21.15.78:0/2334737852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87380319d0 con 0x7f874806d910 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874559a640 1 -- 172.21.15.78:0/2334737852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738031d60 con 0x7f874806d910 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 -- 172.21.15.78:0/2334737852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f874806d910 msgr2=0x7f874806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 --2- 172.21.15.78:0/2334737852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f874806d910 0x7f874806c7e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f87480c7620 tx=0x7f873802b910 comp rx=0 tx=0).stop 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 -- 172.21.15.78:0/2334737852 shutdown_connections 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 --2- 172.21.15.78:0/2334737852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f874806d910 0x7f874806c7e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 --2- 172.21.15.78:0/2334737852 >> v2:172.21.15.150:3300/0 conn(0x7f87480c8e00 0x7f874806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 -- 172.21.15.78:0/2334737852 >> 172.21.15.78:0/2334737852 conn(0x7f87480cac90 msgr2=0x7f87480cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 -- 172.21.15.78:0/2334737852 shutdown_connections 2024-02-19T14:54:32.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.605+0000 7f874c9c2640 1 -- 172.21.15.78:0/2334737852 wait complete. 2024-02-19T14:54:32.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.606+0000 7f874c9c2640 1 Processor -- start 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 -- start start 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f874806d910 0x7f87481eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 0x7f87480ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8748152700 0x7f87480c3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480c1940 con 0x7f87480c8e00 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480aa280 con 0x7f874806d910 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874c9c2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480c1b50 con 0x7f8748152700 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f8745d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 0x7f87480ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f8745d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 0x7f87480ac9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39122/0 (socket says 172.21.15.78:39122) 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f8745d9b640 1 -- 172.21.15.78:0/741626146 learned_addr learned my addr 172.21.15.78:0/741626146 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f8746d9d640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8748152700 0x7f87480c3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.607+0000 7f874659c640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f874806d910 0x7f87481eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f8745d9b640 1 -- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8748152700 msgr2=0x7f87480c3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f8745d9b640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8748152700 0x7f87480c3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f8745d9b640 1 -- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f874806d910 msgr2=0x7f87481eabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f8745d9b640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f874806d910 0x7f87481eabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f8745d9b640 1 -- 172.21.15.78:0/741626146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380094d0 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f8745d9b640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 0x7f87480ac9a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f873000c7f0 tx=0x7f873000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730007a30 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730007b90 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87480c1e30 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000ce40 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.608+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87480acee0 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.609+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87480aa780 con 0x7f87480c8e00 2024-02-19T14:54:32.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.609+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f87300113f0 con 0x7f87480c8e00 2024-02-19T14:54:32.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.610+0000 7f873effd640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f872806b840 0x7f872806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.610+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f873008a400 con 0x7f87480c8e00 2024-02-19T14:54:32.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.610+0000 7f874659c640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f872806b840 0x7f872806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.610+0000 7f874659c640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f872806b840 0x7f872806dd00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f87480c5590 tx=0x7f874000d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.614+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f87480aa780 con 0x7f87480c8e00 2024-02-19T14:54:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.793+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f87480aa780 con 0x7f87480c8e00 2024-02-19T14:54:32.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.795+0000 7f873effd640 1 -- 172.21.15.78:0/741626146 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v26)=0 v26) v1 ==== 117+0+0 (secure 0 0 0) 0x7f87480aa780 con 0x7f87480c8e00 2024-02-19T14:54:32.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f872806b840 msgr2=0x7f872806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f872806b840 0x7f872806dd00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f87480c5590 tx=0x7f874000d040 comp rx=0 tx=0).stop 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 msgr2=0x7f87480ac9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 0x7f87480ac9a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f873000c7f0 tx=0x7f873000cbf0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 shutdown_connections 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f872806b840 0x7f872806dd00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87480c8e00 0x7f87480ac9a0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8748152700 0x7f87480c3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 --2- 172.21.15.78:0/741626146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f874806d910 0x7f87481eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 >> 172.21.15.78:0/741626146 conn(0x7f87480cac90 msgr2=0x7f874814e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 shutdown_connections 2024-02-19T14:54:32.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.797+0000 7f874c9c2640 1 -- 172.21.15.78:0/741626146 wait complete. 2024-02-19T14:54:32.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:32.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.909+0000 7f429990d640 1 Processor -- start 2024-02-19T14:54:32.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.909+0000 7f429990d640 1 -- start start 2024-02-19T14:54:32.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f429990d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f42940c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f429990d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f42940c7910 0x7f4294154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f429990d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42940c65a0 con 0x7f4294070720 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f429990d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940c7530 con 0x7f42940c7910 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f429990d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42941537b0 con 0x7f42940c5560 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f42940c7910 0x7f4294154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f42940c7910 0x7f4294154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39126/0 (socket says 172.21.15.78:39126) 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 -- 172.21.15.78:0/3847470985 learned_addr learned my addr 172.21.15.78:0/3847470985 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f4292ffd640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f42940c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 -- 172.21.15.78:0/3847470985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 msgr2=0x7f42940c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f42940c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 -- 172.21.15.78:0/3847470985 >> v1:172.21.15.150:6789/0 conn(0x7f4294070720 legacy=0x7f4294070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f42927fc640 1 -- 172.21.15.78:0/3847470985 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294153a90 con 0x7f42940c7910 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.910+0000 7f4292ffd640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f42940c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.911+0000 7f42927fc640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42940c7910 0x7f4294154180 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4284009870 tx=0x7f428402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf23e9a440f3da5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.911+0000 7f4291ffb640 1 -- 172.21.15.78:0/3847470985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284031740 con 0x7f42940c7910 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.911+0000 7f4291ffb640 1 -- 172.21.15.78:0/3847470985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42840318a0 con 0x7f42940c7910 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.911+0000 7f4291ffb640 1 -- 172.21.15.78:0/3847470985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284031c30 con 0x7f42940c7910 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 -- 172.21.15.78:0/3847470985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42940c7910 msgr2=0x7f4294154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42940c7910 0x7f4294154180 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4284009870 tx=0x7f428402b7e0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 -- 172.21.15.78:0/3847470985 shutdown_connections 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42940c7910 0x7f4294154180 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 --2- 172.21.15.78:0/3847470985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f42940c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 -- 172.21.15.78:0/3847470985 >> 172.21.15.78:0/3847470985 conn(0x7f429406b4b0 msgr2=0x7f42940c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 -- 172.21.15.78:0/3847470985 shutdown_connections 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.912+0000 7f429990d640 1 -- 172.21.15.78:0/3847470985 wait complete. 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.913+0000 7f429990d640 1 Processor -- start 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.913+0000 7f429990d640 1 -- start start 2024-02-19T14:54:32.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 0x7f42941523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f429413d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42940c7910 0x7f429413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940aa370 con 0x7f4294070720 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429406bf90 con 0x7f42940c7910 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42941eb2a0 con 0x7f42940c5560 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 0x7f42941523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 0x7f42941523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39134/0 (socket says 172.21.15.78:39134) 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 -- 172.21.15.78:0/1194824668 learned_addr learned my addr 172.21.15.78:0/1194824668 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 -- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 msgr2=0x7f429413d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f42927fc640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f429413d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f429413d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 -- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42940c7910 msgr2=0x7f429413dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42940c7910 0x7f429413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:32.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 -- 172.21.15.78:0/1194824668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840094d0 con 0x7f4294070720 2024-02-19T14:54:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f4292ffd640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 0x7f42941523b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f428800c7f0 tx=0x7f428800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288007820 con 0x7f4294070720 2024-02-19T14:54:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4288007980 con 0x7f4294070720 2024-02-19T14:54:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42941eb580 con 0x7f4294070720 2024-02-19T14:54:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288007cf0 con 0x7f4294070720 2024-02-19T14:54:32.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.914+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42941ebb40 con 0x7f4294070720 2024-02-19T14:54:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.915+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42941570a0 con 0x7f4294070720 2024-02-19T14:54:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.916+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f42880113f0 con 0x7f4294070720 2024-02-19T14:54:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.916+0000 7f427bfff640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f428006b810 0x7f428006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.916+0000 7f42927fc640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f428006b810 0x7f428006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.916+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f428808a5b0 con 0x7f4294070720 2024-02-19T14:54:32.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.917+0000 7f42927fc640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f428006b810 0x7f428006dcd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4284035000 tx=0x7f4284008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:32.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:32.921+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4288054ed0 con 0x7f4294070720 2024-02-19T14:54:33.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.106+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4294074990 con 0x7f4294070720 2024-02-19T14:54:33.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.107+0000 7f427bfff640 1 -- 172.21.15.78:0/1194824668 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16933 (secure 0 0 0) 0x7f4288058590 con 0x7f4294070720 2024-02-19T14:54:33.110 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:33.110 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":75,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:27.734957+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:33.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f428006b810 msgr2=0x7f428006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f428006b810 0x7f428006dcd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4284035000 tx=0x7f4284008030 comp rx=0 tx=0).stop 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 msgr2=0x7f42941523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 0x7f42941523b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f428800c7f0 tx=0x7f428800ccb0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 shutdown_connections 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f428006b810 0x7f428006dcd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42940c7910 0x7f429413dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42940c5560 0x7f429413d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 --2- 172.21.15.78:0/1194824668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4294070720 0x7f42941523b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 >> 172.21.15.78:0/1194824668 conn(0x7f429406b4b0 msgr2=0x7f4294070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 shutdown_connections 2024-02-19T14:54:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.110+0000 7f429990d640 1 -- 172.21.15.78:0/1194824668 wait complete. 2024-02-19T14:54:33.123 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:54:24.290728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:54:25.042750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:33.124 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:33.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.225+0000 7f2b8b379640 1 Processor -- start 2024-02-19T14:54:33.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.225+0000 7f2b8b379640 1 -- start start 2024-02-19T14:54:33.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.226+0000 7f2b8b379640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b840c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.226+0000 7f2b8b379640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2b84156860 0x7f2b84156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.226+0000 7f2b8b379640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b841563b0 con 0x7f2b841537a0 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.226+0000 7f2b8b379640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b84153cd0 con 0x7f2b84156860 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.226+0000 7f2b8b379640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8406b6e0 con 0x7f2b8414cd50 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.226+0000 7f2b89115640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b840c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89916640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2b84156860 0x7f2b84156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89115640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b840c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:49014/0 (socket says 172.21.15.78:49014) 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89115640 1 -- 172.21.15.78:0/3373930879 learned_addr learned my addr 172.21.15.78:0/3373930879 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89115640 1 -- 172.21.15.78:0/3373930879 >> v2:172.21.15.150:3300/0 conn(0x7f2b84156860 msgr2=0x7f2b84156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89115640 1 --2- 172.21.15.78:0/3373930879 >> v2:172.21.15.150:3300/0 conn(0x7f2b84156860 0x7f2b84156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89115640 1 -- 172.21.15.78:0/3373930879 >> v1:172.21.15.150:6789/0 conn(0x7f2b841537a0 legacy=0x7f2b840c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89115640 1 -- 172.21.15.78:0/3373930879 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84157230 con 0x7f2b8414cd50 2024-02-19T14:54:33.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.227+0000 7f2b89916640 1 --2- 172.21.15.78:0/3373930879 >> v2:172.21.15.150:3300/0 conn(0x7f2b84156860 0x7f2b84156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b89115640 1 --2- 172.21.15.78:0/3373930879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b840c4cd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009870 tx=0x7f2b8002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa5c7f97940a243b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b7bfff640 1 -- 172.21.15.78:0/3373930879 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b800317c0 con 0x7f2b8414cd50 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b7bfff640 1 -- 172.21.15.78:0/3373930879 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b80031920 con 0x7f2b8414cd50 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b7bfff640 1 -- 172.21.15.78:0/3373930879 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80031cb0 con 0x7f2b8414cd50 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b8b379640 1 -- 172.21.15.78:0/3373930879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 msgr2=0x7f2b840c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b8b379640 1 --2- 172.21.15.78:0/3373930879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b840c4cd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009870 tx=0x7f2b8002b860 comp rx=0 tx=0).stop 2024-02-19T14:54:33.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b8b379640 1 -- 172.21.15.78:0/3373930879 shutdown_connections 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b8b379640 1 --2- 172.21.15.78:0/3373930879 >> v2:172.21.15.150:3300/0 conn(0x7f2b84156860 0x7f2b84156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b8b379640 1 --2- 172.21.15.78:0/3373930879 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b840c4cd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.228+0000 7f2b8b379640 1 -- 172.21.15.78:0/3373930879 >> 172.21.15.78:0/3373930879 conn(0x7f2b84066e70 msgr2=0x7f2b840a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.229+0000 7f2b8b379640 1 -- 172.21.15.78:0/3373930879 shutdown_connections 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.229+0000 7f2b8b379640 1 -- 172.21.15.78:0/3373930879 wait complete. 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 Processor -- start 2024-02-19T14:54:33.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 -- start start 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b84154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 0x7f2b841457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b84156860 0x7f2b84145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84146220 con 0x7f2b841537a0 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b841dcc00 con 0x7f2b84156860 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b8b379640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840c5330 con 0x7f2b8414cd50 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b88914640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 0x7f2b841457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b88914640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 0x7f2b841457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39144/0 (socket says 172.21.15.78:39144) 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.230+0000 7f2b88914640 1 -- 172.21.15.78:0/1243195734 learned_addr learned my addr 172.21.15.78:0/1243195734 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:33.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b89115640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b84154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b89916640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b84156860 0x7f2b84145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b88914640 1 -- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 msgr2=0x7f2b84154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b88914640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b84154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b88914640 1 -- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b84156860 msgr2=0x7f2b84145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b88914640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b84156860 0x7f2b84145ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b88914640 1 -- 172.21.15.78:0/1243195734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c009a70 con 0x7f2b841537a0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b88914640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 0x7f2b841457a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400c470 tx=0x7f2b7400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740052d0 con 0x7f2b841537a0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b74005450 con 0x7f2b841537a0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b800094d0 con 0x7f2b841537a0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74007260 con 0x7f2b841537a0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.231+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b840c55d0 con 0x7f2b841537a0 2024-02-19T14:54:33.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.232+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b84152d90 con 0x7f2b841537a0 2024-02-19T14:54:33.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.233+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2b740073c0 con 0x7f2b841537a0 2024-02-19T14:54:33.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.233+0000 7f2b79ffb640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b6806b810 0x7f2b6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.233+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2b740895c0 con 0x7f2b841537a0 2024-02-19T14:54:33.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.233+0000 7f2b89115640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b6806b810 0x7f2b6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.234+0000 7f2b89115640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b6806b810 0x7f2b6806dcd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2b80036000 tx=0x7f2b8003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.241+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2b74053f00 con 0x7f2b841537a0 2024-02-19T14:54:33.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.436+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b84074990 con 0x7f2b841537a0 2024-02-19T14:54:33.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.437+0000 7f2b79ffb640 1 -- 172.21.15.78:0/1243195734 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b740575c0 con 0x7f2b841537a0 2024-02-19T14:54:33.440 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b6806b810 msgr2=0x7f2b6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b6806b810 0x7f2b6806dcd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2b80036000 tx=0x7f2b8003d040 comp rx=0 tx=0).stop 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 msgr2=0x7f2b841457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 0x7f2b841457a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400c470 tx=0x7f2b7400c870 comp rx=0 tx=0).stop 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 shutdown_connections 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b6806b810 0x7f2b6806dcd0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b841537a0 0x7f2b841457a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b84156860 0x7f2b84145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 --2- 172.21.15.78:0/1243195734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b8414cd50 0x7f2b84154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 >> 172.21.15.78:0/1243195734 conn(0x7f2b84066e70 msgr2=0x7f2b84152150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:33.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 shutdown_connections 2024-02-19T14:54:33.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.439+0000 7f2b8b379640 1 -- 172.21.15.78:0/1243195734 wait complete. 2024-02-19T14:54:33.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.554+0000 7f5bf50c3640 1 Processor -- start 2024-02-19T14:54:33.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.554+0000 7f5bf50c3640 1 -- start start 2024-02-19T14:54:33.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.555+0000 7f5bf50c3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5bf0152700 0x7f5bf00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.555+0000 7f5bf50c3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.555+0000 7f5bf50c3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf01551f0 con 0x7f5bf00c5730 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.555+0000 7f5bf50c3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf0155490 con 0x7f5bf0152700 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.555+0000 7f5bf50c3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf0155770 con 0x7f5bf00c8e00 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5bef59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5beed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5bf0152700 0x7f5bf00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5be7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5bf00c5730 0x7f5bf006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:59044/0 (socket says 172.21.15.78:59044) 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5be7fff640 1 -- 172.21.15.78:0/73640638 learned_addr learned my addr 172.21.15.78:0/73640638 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5beed9d640 1 -- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 msgr2=0x7f5bf015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5beed9d640 1 --2- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5beed9d640 1 -- 172.21.15.78:0/73640638 >> v1:172.21.15.150:6789/0 conn(0x7f5bf00c5730 legacy=0x7f5bf006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.556+0000 7f5beed9d640 1 -- 172.21.15.78:0/73640638 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0155a50 con 0x7f5bf0152700 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5beed9d640 1 --2- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00c51f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5be00097a0 tx=0x7f5be002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31832b9c6f6eeba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bee59c640 1 -- 172.21.15.78:0/73640638 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0031830 con 0x7f5bf0152700 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bee59c640 1 -- 172.21.15.78:0/73640638 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be0031990 con 0x7f5bf0152700 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bee59c640 1 -- 172.21.15.78:0/73640638 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0031d20 con 0x7f5bf0152700 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 -- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 msgr2=0x7f5bf00c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00c51f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5be00097a0 tx=0x7f5be002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 -- 172.21.15.78:0/73640638 shutdown_connections 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/73640638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00c51f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 -- 172.21.15.78:0/73640638 >> 172.21.15.78:0/73640638 conn(0x7f5bf00cac90 msgr2=0x7f5bf00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:33.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 -- 172.21.15.78:0/73640638 shutdown_connections 2024-02-19T14:54:33.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.557+0000 7f5bf50c3640 1 -- 172.21.15.78:0/73640638 wait complete. 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 Processor -- start 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 -- start start 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 0x7f5bf00c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf00aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00ab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf01497b0 con 0x7f5bf00c5730 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00c0b90 con 0x7f5bf0152700 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5bf50c3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00abf10 con 0x7f5bf00c8e00 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5beed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 0x7f5bf00c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.559+0000 7f5beed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 0x7f5bf00c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39152/0 (socket says 172.21.15.78:39152) 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 -- 172.21.15.78:0/386050175 learned_addr learned my addr 172.21.15.78:0/386050175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5be7fff640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf00aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 -- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 msgr2=0x7f5bf00aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5bef59e640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00ab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf00aac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 -- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 msgr2=0x7f5bf00ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00ab1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 -- 172.21.15.78:0/386050175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be0009400 con 0x7f5bf00c5730 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5beed9d640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 0x7f5bf00c0650 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5be0034040 tx=0x7f5be00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be003a8f0 con 0x7f5bf00c5730 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be003aa70 con 0x7f5bf00c5730 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf00acb40 con 0x7f5bf00c5730 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be003ae20 con 0x7f5bf00c5730 2024-02-19T14:54:33.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.560+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf00be4e0 con 0x7f5bf00c5730 2024-02-19T14:54:33.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.561+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf00ad020 con 0x7f5bf00c5730 2024-02-19T14:54:33.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.562+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5be004d440 con 0x7f5bf00c5730 2024-02-19T14:54:33.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.562+0000 7f5be77fe640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5bd006b810 0x7f5bd006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.562+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5be00ac380 con 0x7f5bf00c5730 2024-02-19T14:54:33.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.562+0000 7f5be7fff640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5bd006b810 0x7f5bd006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.562+0000 7f5be7fff640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5bd006b810 0x7f5bd006dcd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5bf014e860 tx=0x7f5bd8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.566+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5bf00ad020 con 0x7f5bf00c5730 2024-02-19T14:54:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.748+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5bf00be340 con 0x7f5bf00c5730 2024-02-19T14:54:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.748+0000 7f5be77fe640 1 -- 172.21.15.78:0/386050175 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v75) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5bf00be340 con 0x7f5bf00c5730 2024-02-19T14:54:33.751 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.750+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5bd006b810 msgr2=0x7f5bd006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.750+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5bd006b810 0x7f5bd006dcd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5bf014e860 tx=0x7f5bd8009040 comp rx=0 tx=0).stop 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.750+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 msgr2=0x7f5bf00c0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.750+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 0x7f5bf00c0650 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5be0034040 tx=0x7f5be00046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 shutdown_connections 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5bd006b810 0x7f5bd006dcd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0152700 0x7f5bf00ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5bf00c8e00 0x7f5bf00aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 --2- 172.21.15.78:0/386050175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bf00c5730 0x7f5bf00c0650 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 >> 172.21.15.78:0/386050175 conn(0x7f5bf00cac90 msgr2=0x7f5bf00cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 shutdown_connections 2024-02-19T14:54:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.751+0000 7f5bf50c3640 1 -- 172.21.15.78:0/386050175 wait complete. 2024-02-19T14:54:33.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:54:33.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.865+0000 7fb431725640 1 Processor -- start 2024-02-19T14:54:33.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.865+0000 7fb431725640 1 -- start start 2024-02-19T14:54:33.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb431725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c14d110 0x7fb42c0ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb431725640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb42c073220 0x7fb42c06ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb431725640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c06b620 con 0x7fb42c06bae0 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb431725640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c06ea00 con 0x7fb42c073220 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb431725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c157230 con 0x7fb42c14d110 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb42a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb42c073220 0x7fb42c06ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb42a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb42c073220 0x7fb42c06ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39158/0 (socket says 172.21.15.78:39158) 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.866+0000 7fb42a7fc640 1 -- 172.21.15.78:0/3907994769 learned_addr learned my addr 172.21.15.78:0/3907994769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.867+0000 7fb42affd640 1 --2- 172.21.15.78:0/3907994769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c14d110 0x7fb42c0ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.867+0000 7fb42a7fc640 1 -- 172.21.15.78:0/3907994769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c14d110 msgr2=0x7fb42c0ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.867+0000 7fb42a7fc640 1 --2- 172.21.15.78:0/3907994769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c14d110 0x7fb42c0ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.867+0000 7fb42a7fc640 1 -- 172.21.15.78:0/3907994769 >> v1:172.21.15.150:6789/0 conn(0x7fb42c06bae0 legacy=0x7fb42c06f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.867+0000 7fb42a7fc640 1 -- 172.21.15.78:0/3907994769 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c072180 con 0x7fb42c073220 2024-02-19T14:54:33.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.867+0000 7fb42a7fc640 1 --2- 172.21.15.78:0/3907994769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c06ee20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb420009870 tx=0x7fb42002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1a98d859e8c2b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb429ffb640 1 -- 172.21.15.78:0/3907994769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420031640 con 0x7fb42c073220 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb429ffb640 1 -- 172.21.15.78:0/3907994769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4200317a0 con 0x7fb42c073220 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb429ffb640 1 -- 172.21.15.78:0/3907994769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420031b30 con 0x7fb42c073220 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 -- 172.21.15.78:0/3907994769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 msgr2=0x7fb42c06ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 --2- 172.21.15.78:0/3907994769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c06ee20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb420009870 tx=0x7fb42002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 -- 172.21.15.78:0/3907994769 shutdown_connections 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 --2- 172.21.15.78:0/3907994769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c06ee20 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 --2- 172.21.15.78:0/3907994769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c14d110 0x7fb42c0ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 -- 172.21.15.78:0/3907994769 >> 172.21.15.78:0/3907994769 conn(0x7fb42c0c7590 msgr2=0x7fb42c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 -- 172.21.15.78:0/3907994769 shutdown_connections 2024-02-19T14:54:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.868+0000 7fb431725640 1 -- 172.21.15.78:0/3907994769 wait complete. 2024-02-19T14:54:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 Processor -- start 2024-02-19T14:54:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 -- start start 2024-02-19T14:54:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c06bae0 0x7fb42c153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42c14d110 0x7fb42c145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c150d20 con 0x7fb42c073220 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c146400 con 0x7fb42c14d110 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb431725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1505a0 con 0x7fb42c06bae0 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb42a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb42a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c151de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39174/0 (socket says 172.21.15.78:39174) 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.870+0000 7fb42a7fc640 1 -- 172.21.15.78:0/2791585900 learned_addr learned my addr 172.21.15.78:0/2791585900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42a7fc640 1 -- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c06bae0 msgr2=0x7fb42c153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42a7fc640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c06bae0 0x7fb42c153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42a7fc640 1 -- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42c14d110 msgr2=0x7fb42c145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42b7fe640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42c14d110 0x7fb42c145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42a7fc640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42c14d110 0x7fb42c145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42a7fc640 1 -- 172.21.15.78:0/2791585900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200094d0 con 0x7fb42c073220 2024-02-19T14:54:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb42a7fc640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c151de0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb420000c00 tx=0x7fb4200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42003a930 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42003aa90 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c150f70 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42003ae40 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.871+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c152320 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.872+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c150ad0 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.872+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb42002f050 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.873+0000 7fb4137fe640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb41806b840 0x7fb41806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.873+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4200ac0d0 con 0x7fb42c073220 2024-02-19T14:54:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.873+0000 7fb42affd640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb41806b840 0x7fb41806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.873+0000 7fb42affd640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb41806b840 0x7fb41806dd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006a90 tx=0x7fb41c006a20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:33.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:33.877+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb420076970 con 0x7fb42c073220 2024-02-19T14:54:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.054+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb42c074990 con 0x7fb42c073220 2024-02-19T14:54:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.054+0000 7fb4137fe640 1 -- 172.21.15.78:0/2791585900 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v75) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb42007a030 con 0x7fb42c073220 2024-02-19T14:54:34.057 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb41806b840 msgr2=0x7fb41806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb41806b840 0x7fb41806dd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006a90 tx=0x7fb41c006a20 comp rx=0 tx=0).stop 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 msgr2=0x7fb42c151de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c151de0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb420000c00 tx=0x7fb4200046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 shutdown_connections 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb41806b840 0x7fb41806dd00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb42c073220 0x7fb42c151de0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb42c14d110 0x7fb42c145820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 --2- 172.21.15.78:0/2791585900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb42c06bae0 0x7fb42c153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 >> 172.21.15.78:0/2791585900 conn(0x7fb42c0c7590 msgr2=0x7fb42c14bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 shutdown_connections 2024-02-19T14:54:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.056+0000 7fb431725640 1 -- 172.21.15.78:0/2791585900 wait complete. 2024-02-19T14:54:34.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:54:34.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.193+0000 7f819d039640 1 Processor -- start 2024-02-19T14:54:34.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.193+0000 7f819d039640 1 -- start start 2024-02-19T14:54:34.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f819d039640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81980c29c0 0x7f81981526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f819d039640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81980c5be0 0x7f819806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f819d039640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8198072250 con 0x7f81980c25e0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f819d039640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8198152c90 con 0x7f81980c29c0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f819d039640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8198072480 con 0x7f81980c5be0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f819659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81980c29c0 0x7f81981526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f8196d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f81980c25e0 0x7f81980c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39704/0 (socket says 172.21.15.78:39704) 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.194+0000 7f8196d9d640 1 -- 172.21.15.78:0/2236105013 learned_addr learned my addr 172.21.15.78:0/2236105013 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f819759e640 1 --2- 172.21.15.78:0/2236105013 >> v2:172.21.15.150:3300/0 conn(0x7f81980c5be0 0x7f819806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f819659c640 1 -- 172.21.15.78:0/2236105013 >> v2:172.21.15.150:3300/0 conn(0x7f81980c5be0 msgr2=0x7f819806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f819659c640 1 --2- 172.21.15.78:0/2236105013 >> v2:172.21.15.150:3300/0 conn(0x7f81980c5be0 0x7f819806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f819659c640 1 -- 172.21.15.78:0/2236105013 >> v1:172.21.15.78:6789/0 conn(0x7f81980c25e0 legacy=0x7f81980c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f819659c640 1 -- 172.21.15.78:0/2236105013 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980c4b90 con 0x7f81980c29c0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f8195d9b640 1 -- 172.21.15.78:0/2236105013 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200301250 0 0) 0x7f8198072250 con 0x7f81980c25e0 2024-02-19T14:54:34.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.195+0000 7f819759e640 1 --2- 172.21.15.78:0/2236105013 >> v2:172.21.15.150:3300/0 conn(0x7f81980c5be0 0x7f819806e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:34.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819659c640 1 --2- 172.21.15.78:0/2236105013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c29c0 0x7f81981526a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8198151c50 tx=0x7f818c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7e804e0e3b5b2d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f8195d9b640 1 -- 172.21.15.78:0/2236105013 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c02e070 con 0x7f81980c29c0 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f8195d9b640 1 -- 172.21.15.78:0/2236105013 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c0315a0 con 0x7f81980c29c0 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f8195d9b640 1 -- 172.21.15.78:0/2236105013 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c031930 con 0x7f81980c29c0 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 -- 172.21.15.78:0/2236105013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c29c0 msgr2=0x7f81981526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 --2- 172.21.15.78:0/2236105013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c29c0 0x7f81981526a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8198151c50 tx=0x7f818c02b840 comp rx=0 tx=0).stop 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 -- 172.21.15.78:0/2236105013 shutdown_connections 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 --2- 172.21.15.78:0/2236105013 >> v2:172.21.15.150:3300/0 conn(0x7f81980c5be0 0x7f819806e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 --2- 172.21.15.78:0/2236105013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c29c0 0x7f81981526a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 -- 172.21.15.78:0/2236105013 >> 172.21.15.78:0/2236105013 conn(0x7f819806b270 msgr2=0x7f819806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 -- 172.21.15.78:0/2236105013 shutdown_connections 2024-02-19T14:54:34.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.196+0000 7f819d039640 1 -- 172.21.15.78:0/2236105013 wait complete. 2024-02-19T14:54:34.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 Processor -- start 2024-02-19T14:54:34.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 -- start start 2024-02-19T14:54:34.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81980c25e0 0x7f81981eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81980c29c0 0x7f81981ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 0x7f8198149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81981dbf50 con 0x7f81980c5be0 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81981ec2c0 con 0x7f81980c25e0 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819d039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81981ec560 con 0x7f81980c29c0 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 0x7f8198149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 0x7f8198149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39192/0 (socket says 172.21.15.78:39192) 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.198+0000 7f819759e640 1 -- 172.21.15.78:0/3265370671 learned_addr learned my addr 172.21.15.78:0/3265370671 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819659c640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81980c29c0 0x7f81981ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f8196d9d640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81980c25e0 0x7f81981eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819759e640 1 -- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81980c29c0 msgr2=0x7f81981ebcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819759e640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81980c29c0 0x7f81981ebcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819759e640 1 -- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81980c25e0 msgr2=0x7f81981eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819759e640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81980c25e0 0x7f81981eb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819759e640 1 -- 172.21.15.78:0/3265370671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c009400 con 0x7f81980c5be0 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819759e640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 0x7f8198149310 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8190007ef0 tx=0x7f819000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190003b90 con 0x7f81980c5be0 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819000f040 con 0x7f81980c5be0 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81980a60a0 con 0x7f81980c5be0 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819000de80 con 0x7f81980c5be0 2024-02-19T14:54:34.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.199+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81980a6660 con 0x7f81980c5be0 2024-02-19T14:54:34.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.200+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81981dbd40 con 0x7f81980c5be0 2024-02-19T14:54:34.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.200+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8190004030 con 0x7f81980c5be0 2024-02-19T14:54:34.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.201+0000 7f817f7fe640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f818406b810 0x7f818406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.201+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f819008a5c0 con 0x7f81980c5be0 2024-02-19T14:54:34.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.201+0000 7f8196d9d640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f818406b810 0x7f818406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.201+0000 7f8196d9d640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f818406b810 0x7f818406dcd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8188000f60 tx=0x7f8188007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:34.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.205+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81981dbd40 con 0x7f81980c5be0 2024-02-19T14:54:34.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.381+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f81981e9680 con 0x7f81980c5be0 2024-02-19T14:54:34.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.382+0000 7f817f7fe640 1 -- 172.21.15.78:0/3265370671 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) v1 ==== 84+0+209 (secure 0 0 0) 0x7f81981e9680 con 0x7f81980c5be0 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:34.385 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 11, 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": "cephfs_metadata" 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 12, 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": "cephfs_data" 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:54:34.386 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:54:34.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.384+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f818406b810 msgr2=0x7f818406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f818406b810 0x7f818406dcd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8188000f60 tx=0x7f8188007c00 comp rx=0 tx=0).stop 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 msgr2=0x7f8198149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 0x7f8198149310 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8190007ef0 tx=0x7f819000d4a0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 shutdown_connections 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f818406b810 0x7f818406dcd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81980c5be0 0x7f8198149310 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81980c29c0 0x7f81981ebcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 --2- 172.21.15.78:0/3265370671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81980c25e0 0x7f81981eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 >> 172.21.15.78:0/3265370671 conn(0x7f819806b270 msgr2=0x7f819806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 shutdown_connections 2024-02-19T14:54:34.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.385+0000 7f819d039640 1 -- 172.21.15.78:0/3265370671 wait complete. 2024-02-19T14:54:34.401 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-02-19T14:54:34.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-02-19T14:54:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.540+0000 7fbbf2a23640 1 Processor -- start 2024-02-19T14:54:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.541+0000 7fbbf2a23640 1 -- start start 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf2a23640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf2a23640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbec0c2aa0 0x7fbbec0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf2a23640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec1539e0 con 0x7fbbec156340 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf2a23640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbec1537d0 con 0x7fbbec0c2aa0 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf2a23640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec0c3140 con 0x7fbbec0ae050 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbeb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbec0c2aa0 0x7fbbec0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbebfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf0fc0640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbbec156340 0x7fbbec156790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:59056/0 (socket says 172.21.15.78:59056) 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbf0fc0640 1 -- 172.21.15.78:0/3831847295 learned_addr learned my addr 172.21.15.78:0/3831847295 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbebfff640 1 -- 172.21.15.78:0/3831847295 >> v2:172.21.15.150:3300/0 conn(0x7fbbec0c2aa0 msgr2=0x7fbbec0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbebfff640 1 --2- 172.21.15.78:0/3831847295 >> v2:172.21.15.150:3300/0 conn(0x7fbbec0c2aa0 0x7fbbec0702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbeaffd640 1 -- 172.21.15.78:0/3831847295 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449352502 0 0) 0x7fbbec1539e0 con 0x7fbbec156340 2024-02-19T14:54:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbebfff640 1 -- 172.21.15.78:0/3831847295 >> v1:172.21.15.150:6789/0 conn(0x7fbbec156340 legacy=0x7fbbec156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.542+0000 7fbbebfff640 1 -- 172.21.15.78:0/3831847295 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec0c3420 con 0x7fbbec0ae050 2024-02-19T14:54:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.543+0000 7fbbebfff640 1 --2- 172.21.15.78:0/3831847295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0c6880 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097a0 tx=0x7fbbdc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=731014d5fd514621 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.543+0000 7fbbeaffd640 1 -- 172.21.15.78:0/3831847295 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc02f050 con 0x7fbbec0ae050 2024-02-19T14:54:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.543+0000 7fbbeaffd640 1 -- 172.21.15.78:0/3831847295 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc031700 con 0x7fbbec0ae050 2024-02-19T14:54:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.543+0000 7fbbeaffd640 1 -- 172.21.15.78:0/3831847295 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc031a90 con 0x7fbbec0ae050 2024-02-19T14:54:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 -- 172.21.15.78:0/3831847295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 msgr2=0x7fbbec0c6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/3831847295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0c6880 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097a0 tx=0x7fbbdc02b7a0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 -- 172.21.15.78:0/3831847295 shutdown_connections 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/3831847295 >> v2:172.21.15.150:3300/0 conn(0x7fbbec0c2aa0 0x7fbbec0702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/3831847295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0c6880 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 -- 172.21.15.78:0/3831847295 >> 172.21.15.78:0/3831847295 conn(0x7fbbec06f220 msgr2=0x7fbbec06f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 -- 172.21.15.78:0/3831847295 shutdown_connections 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.544+0000 7fbbf2a23640 1 -- 172.21.15.78:0/3831847295 wait complete. 2024-02-19T14:54:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.545+0000 7fbbf2a23640 1 Processor -- start 2024-02-19T14:54:34.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.545+0000 7fbbf2a23640 1 -- start start 2024-02-19T14:54:34.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf2a23640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf2a23640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbec0c2aa0 0x7fbbec1e5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf2a23640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 0x7fbbec1e5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf2a23640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec1e6280 con 0x7fbbec156340 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf2a23640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec14d760 con 0x7fbbec0c2aa0 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf2a23640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec1e8550 con 0x7fbbec0ae050 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 0x7fbbec1e5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 0x7fbbec1e5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39198/0 (socket says 172.21.15.78:39198) 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 -- 172.21.15.78:0/4071197272 learned_addr learned my addr 172.21.15.78:0/4071197272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbebfff640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbeb7fe640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbec0c2aa0 0x7fbbec1e5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 -- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 msgr2=0x7fbbec0ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0ac9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 -- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbec0c2aa0 msgr2=0x7fbbec1e5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbec0c2aa0 0x7fbbec1e5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.546+0000 7fbbf0fc0640 1 -- 172.21.15.78:0/4071197272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc009400 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.547+0000 7fbbf0fc0640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 0x7fbbec1e5d40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c700 tx=0x7fbbe400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.547+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe400ce10 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.547+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe400a640 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.547+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe400aa70 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.547+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec0aada0 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.547+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec14b7f0 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.548+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbec1e8710 con 0x7fbbec156340 2024-02-19T14:54:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.548+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbbe400abd0 con 0x7fbbec156340 2024-02-19T14:54:34.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.549+0000 7fbbe8ff9640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbbd806b810 0x7fbbd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:34.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.549+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbbe40897c0 con 0x7fbbec156340 2024-02-19T14:54:34.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.549+0000 7fbbebfff640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbbd806b810 0x7fbbd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:34.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.549+0000 7fbbebfff640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbbd806b810 0x7fbbd806dcd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc035000 tx=0x7fbbdc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:34.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.553+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbbe4054110 con 0x7fbbec156340 2024-02-19T14:54:34.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:34.822+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbbec074990 con 0x7fbbec156340 2024-02-19T14:54:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.804+0000 7fbbe8ff9640 1 -- 172.21.15.78:0/4071197272 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v76) v1 ==== 140+0+0 (secure 0 0 0) 0x7fbbe40577d0 con 0x7fbbec156340 2024-02-19T14:54:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_metadata' created 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbbd806b810 msgr2=0x7fbbd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbbd806b810 0x7fbbd806dcd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc035000 tx=0x7fbbdc008030 comp rx=0 tx=0).stop 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 msgr2=0x7fbbec1e5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 0x7fbbec1e5d40 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c700 tx=0x7fbbe400cbc0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 shutdown_connections 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbbd806b810 0x7fbbd806dcd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbbec156340 0x7fbbec1e5d40 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbec0c2aa0 0x7fbbec1e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 --2- 172.21.15.78:0/4071197272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbec0ae050 0x7fbbec0ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 >> 172.21.15.78:0/4071197272 conn(0x7fbbec06f220 msgr2=0x7fbbec151500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:35.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 shutdown_connections 2024-02-19T14:54:35.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.807+0000 7fbbf2a23640 1 -- 172.21.15.78:0/4071197272 wait complete. 2024-02-19T14:54:35.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:54:35.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.930+0000 7fe1a8ea4640 1 Processor -- start 2024-02-19T14:54:35.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.930+0000 7fe1a8ea4640 1 -- start start 2024-02-19T14:54:35.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a8ea4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe1a40c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a8ea4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe1a4158f30 0x7fe1a415f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a8ea4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a4158aa0 con 0x7fe1a4155a80 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a8ea4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a4152aa0 con 0x7fe1a4158f30 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a8ea4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40cafd0 con 0x7fe1a40c5220 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe19bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe1a4155a80 0x7fe1a4156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39708/0 (socket says 172.21.15.78:39708) 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe19bfff640 1 -- 172.21.15.78:0/4290897720 learned_addr learned my addr 172.21.15.78:0/4290897720 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437605148 0 0) 0x7fe1a4158aa0 con 0x7fe1a4155a80 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe190003520 con 0x7fe1a4155a80 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a259c640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe1a40c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a2d9d640 1 --2- 172.21.15.78:0/4290897720 >> v2:172.21.15.150:3300/0 conn(0x7fe1a4158f30 0x7fe1a415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3807081076 0 0) 0x7fe190003520 con 0x7fe1a4155a80 2024-02-19T14:54:35.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 msgr2=0x7fe1a40c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe1a40c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 >> v2:172.21.15.150:3300/0 conn(0x7fe1a4158f30 msgr2=0x7fe1a415f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 --2- 172.21.15.78:0/4290897720 >> v2:172.21.15.150:3300/0 conn(0x7fe1a4158f30 0x7fe1a415f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a40cb2b0 con 0x7fe1a4155a80 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe18c001670 con 0x7fe1a4155a80 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 >> v1:172.21.15.78:6789/0 conn(0x7fe1a4155a80 legacy=0x7fe1a4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe190003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe190004d00 0x7fe190005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4152aa0 con 0x7fe190003ef0 2024-02-19T14:54:35.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40cafd0 con 0x7fe190004d00 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.931+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19000b9d0 con 0x7fe1a40c5220 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe19bfff640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe1a259c640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe190003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe1a2d9d640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe190004d00 0x7fe190005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe19bfff640 1 -- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 msgr2=0x7fe190003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe19bfff640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe190003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe19bfff640 1 -- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe190004d00 msgr2=0x7fe190005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe19bfff640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe190004d00 0x7fe190005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe19bfff640 1 -- 172.21.15.78:0/4290897720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe190003520 con 0x7fe190003ef0 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe1a259c640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a40c5220 0x7fe190003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.932+0000 7fe1a2d9d640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe190004d00 0x7fe190005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe19bfff640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1900047c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c2c0 tx=0x7fe1900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7103db03a45ef7d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:35.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c005ad0 con 0x7fe190003ef0 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c005c30 con 0x7fe190003ef0 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a1d9b640 1 -- 172.21.15.78:0/4290897720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c02c8b0 con 0x7fe190003ef0 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/4290897720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 msgr2=0x7fe1900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/4290897720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1900047c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c2c0 tx=0x7fe1900057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a2d9d640 1 -- 172.21.15.78:0/4290897720 reap_dead start 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/4290897720 shutdown_connections 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/4290897720 >> 172.21.15.78:0/4290897720 conn(0x7fe1a4072790 msgr2=0x7fe1a4072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/4290897720 shutdown_connections 2024-02-19T14:54:35.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.933+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/4290897720 wait complete. 2024-02-19T14:54:35.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.935+0000 7fe1a8ea4640 1 Processor -- start 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.935+0000 7fe1a8ea4640 1 -- start start 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a8ea4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1a40aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a8ea4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1a40c0de0 0x7fe1a41edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a8ea4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a41ee090 0x7fe1a40be390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a8ea4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40c02c0 con 0x7fe190003ef0 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a8ea4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40bfd00 con 0x7fe1a40c0de0 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a8ea4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41eaf80 con 0x7fe1a41ee090 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1a40aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1a40aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39224/0 (socket says 172.21.15.78:39224) 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 -- 172.21.15.78:0/1964678953 learned_addr learned my addr 172.21.15.78:0/1964678953 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:35.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe19bfff640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1a40c0de0 0x7fe1a41edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a2d9d640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a41ee090 0x7fe1a40be390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 -- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a41ee090 msgr2=0x7fe1a40be390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a41ee090 0x7fe1a40be390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 -- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1a40c0de0 msgr2=0x7fe1a41edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1a40c0de0 0x7fe1a41edb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe1a259c640 1 -- 172.21.15.78:0/1964678953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a41eb260 con 0x7fe190003ef0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.936+0000 7fe19bfff640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1a40c0de0 0x7fe1a41edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.937+0000 7fe1a259c640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1a40aecc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe194011600 tx=0x7fe194011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.937+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400f920 con 0x7fe190003ef0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.937+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19400fa80 con 0x7fe190003ef0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.937+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a40bff50 con 0x7fe190003ef0 2024-02-19T14:54:35.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.937+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400fe10 con 0x7fe190003ef0 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.937+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a40af7c0 con 0x7fe190003ef0 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.938+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a4074990 con 0x7fe190003ef0 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.938+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe194015460 con 0x7fe190003ef0 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.939+0000 7fe19affd640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe18406b840 0x7fe18406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.939+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6472+0+0 (secure 0 0 0) 0x7fe19408abb0 con 0x7fe190003ef0 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.939+0000 7fe19bfff640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe18406b840 0x7fe18406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:35.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.939+0000 7fe19bfff640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe18406b840 0x7fe18406dd00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe18c034000 tx=0x7fe18c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:35.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:35.948+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe194055360 con 0x7fe190003ef0 2024-02-19T14:54:36.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.233+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe1a406d980 con 0x7fe190003ef0 2024-02-19T14:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.809+0000 7fe19affd640 1 -- 172.21.15.78:0/1964678953 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v77) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe194058a20 con 0x7fe190003ef0 2024-02-19T14:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_data' created 2024-02-19T14:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe18406b840 msgr2=0x7fe18406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe18406b840 0x7fe18406dd00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe18c034000 tx=0x7fe18c007000 comp rx=0 tx=0).stop 2024-02-19T14:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 msgr2=0x7fe1a40aecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1a40aecc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe194011600 tx=0x7fe194011ac0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 shutdown_connections 2024-02-19T14:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe18406b840 0x7fe18406dd00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1a41ee090 0x7fe1a40be390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1a40c0de0 0x7fe1a41edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 --2- 172.21.15.78:0/1964678953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe190003ef0 0x7fe1a40aecc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 >> 172.21.15.78:0/1964678953 conn(0x7fe1a4072790 msgr2=0x7fe1a40c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 shutdown_connections 2024-02-19T14:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.812+0000 7fe1a8ea4640 1 -- 172.21.15.78:0/1964678953 wait complete. 2024-02-19T14:54:36.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-02-19T14:54:36.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 Processor -- start 2024-02-19T14:54:36.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 -- start start 2024-02-19T14:54:36.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f66fc071790 0x7f66fc0c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:36.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66fc0c5480 0x7f66fc0c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:36.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc071c20 con 0x7f66fc0c50a0 2024-02-19T14:54:36.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc0c7fe0 con 0x7f66fc071790 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.974+0000 7f6701e39640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc0c82c0 con 0x7f66fc0c5480 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f66fc071790 0x7f66fc0c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f66fc071790 0x7f66fc0c73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39228/0 (socket says 172.21.15.78:39228) 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66fb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f66fc0c50a0 0x7f66fc06cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39718/0 (socket says 172.21.15.78:39718) 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 -- 172.21.15.78:0/147986234 learned_addr learned my addr 172.21.15.78:0/147986234 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66fbfff640 1 --2- 172.21.15.78:0/147986234 >> v2:172.21.15.150:3300/0 conn(0x7f66fc0c5480 0x7f66fc0c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 -- 172.21.15.78:0/147986234 >> v2:172.21.15.150:3300/0 conn(0x7f66fc0c5480 msgr2=0x7f66fc0c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 --2- 172.21.15.78:0/147986234 >> v2:172.21.15.150:3300/0 conn(0x7f66fc0c5480 0x7f66fc0c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 -- 172.21.15.78:0/147986234 >> v1:172.21.15.78:6789/0 conn(0x7f66fc0c50a0 legacy=0x7f66fc06cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66faffd640 1 -- 172.21.15.78:0/147986234 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc06c700 con 0x7f66fc071790 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.975+0000 7f66fa7fc640 1 -- 172.21.15.78:0/147986234 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097575187 0 0) 0x7f66fc071c20 con 0x7f66fc0c50a0 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f66faffd640 1 --2- 172.21.15.78:0/147986234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc071790 0x7f66fc0c73e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0c4b90 tx=0x7f66f002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9dddb9c41b75f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f66fa7fc640 1 -- 172.21.15.78:0/147986234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f00317e0 con 0x7f66fc071790 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f66fa7fc640 1 -- 172.21.15.78:0/147986234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0031940 con 0x7f66fc071790 2024-02-19T14:54:36.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f66fa7fc640 1 -- 172.21.15.78:0/147986234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0031cd0 con 0x7f66fc071790 2024-02-19T14:54:36.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f6701e39640 1 -- 172.21.15.78:0/147986234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc071790 msgr2=0x7f66fc0c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:36.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f6701e39640 1 --2- 172.21.15.78:0/147986234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc071790 0x7f66fc0c73e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0c4b90 tx=0x7f66f002b880 comp rx=0 tx=0).stop 2024-02-19T14:54:36.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f6701e39640 1 -- 172.21.15.78:0/147986234 shutdown_connections 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f6701e39640 1 --2- 172.21.15.78:0/147986234 >> v2:172.21.15.150:3300/0 conn(0x7f66fc0c5480 0x7f66fc0c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f6701e39640 1 --2- 172.21.15.78:0/147986234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc071790 0x7f66fc0c73e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.976+0000 7f6701e39640 1 -- 172.21.15.78:0/147986234 >> 172.21.15.78:0/147986234 conn(0x7f66fc06b700 msgr2=0x7f66fc0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.977+0000 7f6701e39640 1 -- 172.21.15.78:0/147986234 shutdown_connections 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.977+0000 7f6701e39640 1 -- 172.21.15.78:0/147986234 wait complete. 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 Processor -- start 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 -- start start 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66fc0c50a0 0x7f66fc154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 0x7f66fc14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66fc0a9cf0 0x7f66fc14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1eae10 con 0x7f66fc0c5480 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1eb340 con 0x7f66fc0c50a0 2024-02-19T14:54:36.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.978+0000 7f6701e39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1ed940 con 0x7f66fc0a9cf0 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 0x7f66fc14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 0x7f66fc14fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39242/0 (socket says 172.21.15.78:39242) 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 -- 172.21.15.78:0/2719560972 learned_addr learned my addr 172.21.15.78:0/2719560972 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66fbfff640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66fc0a9cf0 0x7f66fc14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66fb7fe640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66fc0c50a0 0x7f66fc154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 -- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66fc0a9cf0 msgr2=0x7f66fc14d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66fc0a9cf0 0x7f66fc14d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 -- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66fc0c50a0 msgr2=0x7f66fc154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66fc0c50a0 0x7f66fc154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 -- 172.21.15.78:0/2719560972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00094d0 con 0x7f66fc0c5480 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66faffd640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 0x7f66fc14fb30 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f66f002b8c0 tx=0x7f66f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0032880 con 0x7f66fc0c5480 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0032a00 con 0x7f66fc0c5480 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc1571c0 con 0x7f66fc0c5480 2024-02-19T14:54:36.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0032db0 con 0x7f66fc0c5480 2024-02-19T14:54:36.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.979+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc1565a0 con 0x7f66fc0c5480 2024-02-19T14:54:36.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.980+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc074990 con 0x7f66fc0c5480 2024-02-19T14:54:36.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.981+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f66f00453f0 con 0x7f66fc0c5480 2024-02-19T14:54:36.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.981+0000 7f66e3fff640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f66e806b840 0x7f66e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:36.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.981+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f66f00ac760 con 0x7f66fc0c5480 2024-02-19T14:54:36.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.981+0000 7f66fb7fe640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f66e806b840 0x7f66e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:36.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.982+0000 7f66fb7fe640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f66e806b840 0x7f66e806dd00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f66ec000f60 tx=0x7f66ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:36.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:36.986+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f66f0076eb0 con 0x7f66fc0c5480 2024-02-19T14:54:37.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.201+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f66fc1eda80 con 0x7f66fc0c5480 2024-02-19T14:54:37.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.811+0000 7f66e3fff640 1 -- 172.21.15.78:0/2719560972 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 13 and data pool 14 v52) v1 ==== 177+0+0 (secure 0 0 0) 0x7f66f007a570 con 0x7f66fc0c5480 2024-02-19T14:54:37.816 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 13 and data pool 14 2024-02-19T14:54:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.816+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f66e806b840 msgr2=0x7f66e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.816+0000 7f6701e39640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f66e806b840 0x7f66e806dd00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f66ec000f60 tx=0x7f66ec009040 comp rx=0 tx=0).stop 2024-02-19T14:54:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.816+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 msgr2=0x7f66fc14fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.816+0000 7f6701e39640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 0x7f66fc14fb30 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f66f002b8c0 tx=0x7f66f00046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.818+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 shutdown_connections 2024-02-19T14:54:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.818+0000 7f6701e39640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f66e806b840 0x7f66e806dd00 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.818+0000 7f6701e39640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66fc0c5480 0x7f66fc14fb30 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.818+0000 7f6701e39640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66fc0a9cf0 0x7f66fc14d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.818+0000 7f6701e39640 1 --2- 172.21.15.78:0/2719560972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66fc0c50a0 0x7f66fc154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.818+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 >> 172.21.15.78:0/2719560972 conn(0x7f66fc06b700 msgr2=0x7f66fc0c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:37.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.819+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 shutdown_connections 2024-02-19T14:54:37.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.821+0000 7f6701e39640 1 -- 172.21.15.78:0/2719560972 wait complete. 2024-02-19T14:54:37.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.947+0000 7f6724920640 1 Processor -- start 2024-02-19T14:54:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.947+0000 7f6724920640 1 -- start start 2024-02-19T14:54:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f6724920640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 0x7f67200c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f6724920640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672006f330 con 0x7f6720151ec0 2024-02-19T14:54:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f6724920640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67200a84c0 con 0x7f67200c12a0 2024-02-19T14:54:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f6724920640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67200a8fb0 con 0x7f672014f3a0 2024-02-19T14:54:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6720151ec0 0x7f67201522f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54792/0 (socket says 172.21.15.78:54792) 2024-02-19T14:54:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671ed9d640 1 -- 172.21.15.78:0/1550378199 learned_addr learned my addr 172.21.15.78:0/1550378199 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671e59c640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 0x7f67200c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283571581 0 0) 0x7f672006f330 con 0x7f6720151ec0 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6708003520 con 0x7f6720151ec0 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 451828829 0 0) 0x7f6708003520 con 0x7f6720151ec0 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 msgr2=0x7f67200c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.949+0000 7f671d59a640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 0x7f67200c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 >> v1:172.21.15.150:6789/0 conn(0x7f67200c12a0 legacy=0x7f67200723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200c35f0 con 0x7f6720151ec0 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f67180019f0 con 0x7f6720151ec0 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 >> v1:172.21.15.78:6789/0 conn(0x7f6720151ec0 legacy=0x7f67201522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 0x7f67080039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6708003f20 0x7f67080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708004d00 0x7f6708005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200a8fb0 con 0x7f6708003f20 2024-02-19T14:54:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670800bbc0 con 0x7f6708004d00 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670800bd80 con 0x7f672014f3a0 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671dd9b640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6708003f20 0x7f67080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671e59c640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 0x7f67080039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671ed9d640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708004d00 0x7f6708005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671dd9b640 1 -- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 msgr2=0x7f67080039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671dd9b640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f672014f3a0 0x7f67080039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671dd9b640 1 -- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708004d00 msgr2=0x7f6708005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671dd9b640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708004d00 0x7f6708005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.950+0000 7f671dd9b640 1 -- 172.21.15.78:0/1550378199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708003520 con 0x7f6708003f20 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f671dd9b640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6708003f20 0x7f67080047c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6714001750 tx=0x7f67080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d6ffefa31e3f20 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140052d0 con 0x7f6708003f20 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671402e050 con 0x7f6708003f20 2024-02-19T14:54:37.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f671d59a640 1 -- 172.21.15.78:0/1550378199 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714003830 con 0x7f6708003f20 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f6724920640 1 -- 172.21.15.78:0/1550378199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6708003f20 msgr2=0x7f67080047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f6724920640 1 --2- 172.21.15.78:0/1550378199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6708003f20 0x7f67080047c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6714001750 tx=0x7f67080057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.951+0000 7f671ed9d640 1 -- 172.21.15.78:0/1550378199 reap_dead start 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.952+0000 7f6724920640 1 -- 172.21.15.78:0/1550378199 shutdown_connections 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.952+0000 7f6724920640 1 -- 172.21.15.78:0/1550378199 >> 172.21.15.78:0/1550378199 conn(0x7f67200718e0 msgr2=0x7f6720071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.952+0000 7f6724920640 1 -- 172.21.15.78:0/1550378199 shutdown_connections 2024-02-19T14:54:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.952+0000 7f6724920640 1 -- 172.21.15.78:0/1550378199 wait complete. 2024-02-19T14:54:37.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.953+0000 7f6724920640 1 Processor -- start 2024-02-19T14:54:37.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.953+0000 7f6724920640 1 -- start start 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.953+0000 7f6724920640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708003f20 0x7f67200c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.953+0000 7f6724920640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6720155900 0x7f6720153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.953+0000 7f6724920640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 0x7f6720144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f6724920640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200c6370 con 0x7f6720144710 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f6724920640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200c6090 con 0x7f6708003f20 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f6724920640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6720152fa0 con 0x7f6720155900 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 0x7f6720144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 0x7f6720144b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32886/0 (socket says 172.21.15.78:32886) 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 -- 172.21.15.78:0/2197943338 learned_addr learned my addr 172.21.15.78:0/2197943338 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671dd9b640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6720155900 0x7f6720153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671e59c640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708003f20 0x7f67200c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 -- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6720155900 msgr2=0x7f6720153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6720155900 0x7f6720153db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 -- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708003f20 msgr2=0x7f67200c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708003f20 0x7f67200c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 -- 172.21.15.78:0/2197943338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6720153280 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f671ed9d640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 0x7f6720144b90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6720066d50 tx=0x7f6718007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671800ab10 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671800ac90 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6720153560 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67180133f0 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.954+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6720150970 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.955+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6720153750 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.956+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6718013550 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.956+0000 7f670effd640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f670006b840 0x7f670006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.956+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f671808ada0 con 0x7f6720144710 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.956+0000 7f671e59c640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f670006b840 0x7f670006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.957+0000 7f671e59c640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f670006b840 0x7f670006dd00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6710009bc0 tx=0x7f6710009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:37.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:37.961+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6718055370 con 0x7f6720144710 2024-02-19T14:54:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.141+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6720074990 con 0x7f6720144710 2024-02-19T14:54:38.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.144+0000 7f670effd640 1 -- 172.21.15.78:0/2197943338 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+20670 (secure 0 0 0) 0x7f6718058a30 con 0x7f6720144710 2024-02-19T14:54:38.147 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:38.147 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:37.203028+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f670006b840 msgr2=0x7f670006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f670006b840 0x7f670006dd00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6710009bc0 tx=0x7f6710009190 comp rx=0 tx=0).stop 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 msgr2=0x7f6720144b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 0x7f6720144b90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6720066d50 tx=0x7f6718007a40 comp rx=0 tx=0).stop 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 shutdown_connections 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f670006b840 0x7f670006dd00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6720144710 0x7f6720144b90 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6720155900 0x7f6720153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 --2- 172.21.15.78:0/2197943338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708003f20 0x7f67200c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 >> 172.21.15.78:0/2197943338 conn(0x7f67200718e0 msgr2=0x7f6720150f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 shutdown_connections 2024-02-19T14:54:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.146+0000 7f6724920640 1 -- 172.21.15.78:0/2197943338 wait complete. 2024-02-19T14:54:38.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:38.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.257+0000 7fbea0eae640 1 Processor -- start 2024-02-19T14:54:38.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.257+0000 7fbea0eae640 1 -- start start 2024-02-19T14:54:38.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbea0eae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 0x7fbe9c06d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbea0eae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbea0eae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0c6af0 con 0x7fbe9c0c6d30 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbea0eae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c7940 con 0x7fbe9c155f90 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbea0eae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c7c20 con 0x7fbe9c1588a0 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 0x7fbe9c06d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 0x7fbe9c06d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:37160/0 (socket says 172.21.15.78:37160) 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9ad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe93fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbe9c0c6d30 0x7fbe9c0c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:35816/0 (socket says 172.21.15.78:35816) 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9a59c640 1 -- 172.21.15.78:0/4268313725 learned_addr learned my addr 172.21.15.78:0/4268313725 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9ad9d640 1 -- 172.21.15.78:0/4268313725 >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 msgr2=0x7fbe9c06d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9ad9d640 1 --2- 172.21.15.78:0/4268313725 >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 0x7fbe9c06d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9ad9d640 1 -- 172.21.15.78:0/4268313725 >> v1:172.21.15.150:6789/0 conn(0x7fbe9c0c6d30 legacy=0x7fbe9c0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9ad9d640 1 -- 172.21.15.78:0/4268313725 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0c7f00 con 0x7fbe9c1588a0 2024-02-19T14:54:38.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.258+0000 7fbe9a59c640 1 --2- 172.21.15.78:0/4268313725 >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 0x7fbe9c06d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:38.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbe9ad9d640 1 --2- 172.21.15.78:0/4268313725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c158cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009d90 tx=0x7fbe9402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f569cf141590e5a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbe99d9b640 1 -- 172.21.15.78:0/4268313725 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9402cbc0 con 0x7fbe9c1588a0 2024-02-19T14:54:38.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbe99d9b640 1 -- 172.21.15.78:0/4268313725 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9402cd20 con 0x7fbe9c1588a0 2024-02-19T14:54:38.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbe99d9b640 1 -- 172.21.15.78:0/4268313725 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94034660 con 0x7fbe9c1588a0 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 -- 172.21.15.78:0/4268313725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 msgr2=0x7fbe9c158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 --2- 172.21.15.78:0/4268313725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c158cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009d90 tx=0x7fbe9402be10 comp rx=0 tx=0).stop 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 -- 172.21.15.78:0/4268313725 shutdown_connections 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 --2- 172.21.15.78:0/4268313725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c158cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 --2- 172.21.15.78:0/4268313725 >> v2:172.21.15.150:3300/0 conn(0x7fbe9c155f90 0x7fbe9c06d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 -- 172.21.15.78:0/4268313725 >> 172.21.15.78:0/4268313725 conn(0x7fbe9c06b700 msgr2=0x7fbe9c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 -- 172.21.15.78:0/4268313725 shutdown_connections 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.260+0000 7fbea0eae640 1 -- 172.21.15.78:0/4268313725 wait complete. 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 Processor -- start 2024-02-19T14:54:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 -- start start 2024-02-19T14:54:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe9c0c6d30 0x7fbe9c0ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 0x7fbe9c06f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c1eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0ac600 con 0x7fbe9c155f90 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c156370 con 0x7fbe9c0c6d30 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbea0eae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c1eb0c0 con 0x7fbe9c1588a0 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 0x7fbe9c06f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 0x7fbe9c06f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32894/0 (socket says 172.21.15.78:32894) 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.262+0000 7fbe93fff640 1 -- 172.21.15.78:0/207122450 learned_addr learned my addr 172.21.15.78:0/207122450 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe9a59c640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe9c0c6d30 0x7fbe9c0ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe9ad9d640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c1eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe93fff640 1 -- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 msgr2=0x7fbe9c1eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe93fff640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c1eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe93fff640 1 -- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe9c0c6d30 msgr2=0x7fbe9c0ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe93fff640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe9c0c6d30 0x7fbe9c0ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe93fff640 1 -- 172.21.15.78:0/207122450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c0094d0 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe93fff640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 0x7fbe9c06f860 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c400 tx=0x7fbe8400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe840074e0 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84007680 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe94009a70 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84004a00 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.263+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c1ef0e0 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.264+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c074990 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.265+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbe84004b60 con 0x7fbe9c155f90 2024-02-19T14:54:38.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.265+0000 7fbe92ffd640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.265+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fbe8408a100 con 0x7fbe9c155f90 2024-02-19T14:54:38.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.265+0000 7fbe9a59c640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.266+0000 7fbe9a59c640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009cf0 tx=0x7fbe8c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.269+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbe84054650 con 0x7fbe9c155f90 2024-02-19T14:54:38.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.444+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe9c0aa2e0 con 0x7fbe9c155f90 2024-02-19T14:54:38.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.445+0000 7fbe92ffd640 1 -- 172.21.15.78:0/207122450 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+20670 (secure 0 0 0) 0x7fbe84057d10 con 0x7fbe9c155f90 2024-02-19T14:54:38.448 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:38.448 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:37.203028+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe7c06b810 msgr2=0x7fbe7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009cf0 tx=0x7fbe8c008040 comp rx=0 tx=0).stop 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 msgr2=0x7fbe9c06f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 0x7fbe9c06f860 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c400 tx=0x7fbe8400c8c0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 shutdown_connections 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe7c06b810 0x7fbe7c06dcd0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe9c155f90 0x7fbe9c06f860 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe9c1588a0 0x7fbe9c1eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 --2- 172.21.15.78:0/207122450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe9c0c6d30 0x7fbe9c0ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 >> 172.21.15.78:0/207122450 conn(0x7fbe9c06b700 msgr2=0x7fbe9c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:38.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 shutdown_connections 2024-02-19T14:54:38.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.447+0000 7fbea0eae640 1 -- 172.21.15.78:0/207122450 wait complete. 2024-02-19T14:54:38.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-02-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 Processor -- start 2024-02-19T14:54:38.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 -- start start 2024-02-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fed48155f90 0x7fed4806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed480c6d30 0x7fed480c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c3aa0 0x7fed481588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed480c6af0 con 0x7fed48155f90 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed480c7940 con 0x7fed480c6d30 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4f24e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed480c7c20 con 0x7fed480c3aa0 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4cfea640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fed48155f90 0x7fed4806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4cfea640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fed48155f90 0x7fed4806d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32898/0 (socket says 172.21.15.78:32898) 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.566+0000 7fed4cfea640 1 -- 172.21.15.78:0/2728731000 learned_addr learned my addr 172.21.15.78:0/2728731000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed3ffff640 1 --2- 172.21.15.78:0/2728731000 >> v2:172.21.15.150:3300/0 conn(0x7fed480c6d30 0x7fed480c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4d7eb640 1 --2- 172.21.15.78:0/2728731000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c3aa0 0x7fed481588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4cfea640 1 -- 172.21.15.78:0/2728731000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c3aa0 msgr2=0x7fed481588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4cfea640 1 --2- 172.21.15.78:0/2728731000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c3aa0 0x7fed481588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4cfea640 1 -- 172.21.15.78:0/2728731000 >> v2:172.21.15.150:3300/0 conn(0x7fed480c6d30 msgr2=0x7fed480c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4cfea640 1 --2- 172.21.15.78:0/2728731000 >> v2:172.21.15.150:3300/0 conn(0x7fed480c6d30 0x7fed480c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4cfea640 1 -- 172.21.15.78:0/2728731000 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed480c7f00 con 0x7fed48155f90 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.567+0000 7fed4cfea640 1 --2- 172.21.15.78:0/2728731000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4806d550 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fed480c2b60 tx=0x7fed4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae2b5f788aa1386 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed3f7fe640 1 -- 172.21.15.78:0/2728731000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed440317e0 con 0x7fed48155f90 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed3f7fe640 1 -- 172.21.15.78:0/2728731000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44031940 con 0x7fed48155f90 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed3f7fe640 1 -- 172.21.15.78:0/2728731000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44031cd0 con 0x7fed48155f90 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 -- 172.21.15.78:0/2728731000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 msgr2=0x7fed4806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2728731000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4806d550 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fed480c2b60 tx=0x7fed4402b880 comp rx=0 tx=0).stop 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 -- 172.21.15.78:0/2728731000 shutdown_connections 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2728731000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c3aa0 0x7fed481588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2728731000 >> v2:172.21.15.150:3300/0 conn(0x7fed480c6d30 0x7fed480c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2728731000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4806d550 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 -- 172.21.15.78:0/2728731000 >> 172.21.15.78:0/2728731000 conn(0x7fed4806b700 msgr2=0x7fed480aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 -- 172.21.15.78:0/2728731000 shutdown_connections 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.568+0000 7fed4f24e640 1 -- 172.21.15.78:0/2728731000 wait complete. 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.569+0000 7fed4f24e640 1 Processor -- start 2024-02-19T14:54:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.569+0000 7fed4f24e640 1 -- start start 2024-02-19T14:54:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4f24e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed480c3aa0 0x7fed48151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4f24e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c6d30 0x7fed4814f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4f24e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4814eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4f24e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4806f2e0 con 0x7fed48155f90 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4f24e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed481e2640 con 0x7fed480c3aa0 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4f24e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed481e1ac0 con 0x7fed480c6d30 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4814eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4814eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32914/0 (socket says 172.21.15.78:32914) 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 -- 172.21.15.78:0/2516333834 learned_addr learned my addr 172.21.15.78:0/2516333834 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4cfea640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed480c3aa0 0x7fed48151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed3ffff640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c6d30 0x7fed4814f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 -- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c6d30 msgr2=0x7fed4814f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c6d30 0x7fed4814f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 -- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed480c3aa0 msgr2=0x7fed48151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.570+0000 7fed4d7eb640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed480c3aa0 0x7fed48151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed4d7eb640 1 -- 172.21.15.78:0/2516333834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed440094d0 con 0x7fed48155f90 2024-02-19T14:54:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed4d7eb640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4814eeb0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed40007e60 tx=0x7fed4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40014620 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4000ee90 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed481e1da0 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400149e0 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.571+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed481e2360 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.572+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48074990 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.572+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fed40014b40 con 0x7fed48155f90 2024-02-19T14:54:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.573+0000 7fed3d7fa640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed3406b810 0x7fed3406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.573+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fed4008af20 con 0x7fed48155f90 2024-02-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.573+0000 7fed4cfea640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed3406b810 0x7fed3406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.573+0000 7fed4cfea640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed3406b810 0x7fed3406dcd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fed44036000 tx=0x7fed4403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.581+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fed400554f0 con 0x7fed48155f90 2024-02-19T14:54:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.792+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fed480c75a0 con 0x7fed48155f90 2024-02-19T14:54:38.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.826+0000 7fed3d7fa640 1 -- 172.21.15.78:0/2516333834 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v53) v1 ==== 119+0+0 (secure 0 0 0) 0x7fed40058bb0 con 0x7fed48155f90 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed3406b810 msgr2=0x7fed3406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed3406b810 0x7fed3406dcd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fed44036000 tx=0x7fed4403d040 comp rx=0 tx=0).stop 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 msgr2=0x7fed4814eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4814eeb0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed40007e60 tx=0x7fed4000e4b0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 shutdown_connections 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed3406b810 0x7fed3406dcd0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed48155f90 0x7fed4814eeb0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed480c6d30 0x7fed4814f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 --2- 172.21.15.78:0/2516333834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed480c3aa0 0x7fed48151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.829+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 >> 172.21.15.78:0/2516333834 conn(0x7fed4806b700 msgr2=0x7fed481584e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.830+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 shutdown_connections 2024-02-19T14:54:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.830+0000 7fed4f24e640 1 -- 172.21.15.78:0/2516333834 wait complete. 2024-02-19T14:54:38.843 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:54:38.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-02-19T14:54:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.944+0000 7f7c4fd6b640 1 Processor -- start 2024-02-19T14:54:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4fd6b640 1 -- start start 2024-02-19T14:54:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4fd6b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c480c8e00 0x7f7c4806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4fd6b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c481551f0 con 0x7f7c48152700 2024-02-19T14:54:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4fd6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48155490 con 0x7f7c4806d910 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4fd6b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c48155770 con 0x7f7c480c8e00 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4d306640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c480c8e00 0x7f7c4806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4db07640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7c48152700 0x7f7c480c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54794/0 (socket says 172.21.15.78:54794) 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4d306640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c480c8e00 0x7f7c4806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32924/0 (socket says 172.21.15.78:32924) 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.945+0000 7f7c4db07640 1 -- 172.21.15.78:0/3555984413 learned_addr learned my addr 172.21.15.78:0/3555984413 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.946+0000 7f7c4d306640 1 -- 172.21.15.78:0/3555984413 >> v1:172.21.15.150:6789/0 conn(0x7f7c4806d910 legacy=0x7f7c4815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.946+0000 7f7c4d306640 1 -- 172.21.15.78:0/3555984413 >> v1:172.21.15.78:6789/0 conn(0x7f7c48152700 legacy=0x7f7c480c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.946+0000 7f7c4d306640 1 -- 172.21.15.78:0/3555984413 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c48155a50 con 0x7f7c480c8e00 2024-02-19T14:54:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.946+0000 7f7c4cb05640 1 -- 172.21.15.78:0/3555984413 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804157764 0 0) 0x7f7c481551f0 con 0x7f7c48152700 2024-02-19T14:54:38.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.947+0000 7f7c4d306640 1 --2- 172.21.15.78:0/3555984413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c480c8e00 0x7f7c4806c2a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0097a0 tx=0x7f7c3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569d8c0da837057e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.947+0000 7f7c4cb05640 1 -- 172.21.15.78:0/3555984413 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c031830 con 0x7f7c480c8e00 2024-02-19T14:54:38.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.947+0000 7f7c4cb05640 1 -- 172.21.15.78:0/3555984413 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3c031990 con 0x7f7c480c8e00 2024-02-19T14:54:38.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.947+0000 7f7c4cb05640 1 -- 172.21.15.78:0/3555984413 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c031d20 con 0x7f7c480c8e00 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3555984413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c480c8e00 msgr2=0x7f7c4806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3555984413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c480c8e00 0x7f7c4806c2a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0097a0 tx=0x7f7c3c02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3555984413 shutdown_connections 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3555984413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c480c8e00 0x7f7c4806c2a0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3555984413 >> 172.21.15.78:0/3555984413 conn(0x7f7c480cac90 msgr2=0x7f7c480cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3555984413 shutdown_connections 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.948+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3555984413 wait complete. 2024-02-19T14:54:38.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.949+0000 7f7c4fd6b640 1 Processor -- start 2024-02-19T14:54:38.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.949+0000 7f7c4fd6b640 1 -- start start 2024-02-19T14:54:38.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c4806d910 0x7f7c48146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c480c8e00 0x7f7c480ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 0x7f7c480aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480ad070 con 0x7f7c48152700 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48154c00 con 0x7f7c480c8e00 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c48154ee0 con 0x7f7c4806d910 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 0x7f7c480aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 0x7f7c480aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32932/0 (socket says 172.21.15.78:32932) 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 -- 172.21.15.78:0/3803840513 learned_addr learned my addr 172.21.15.78:0/3803840513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4d306640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c480c8e00 0x7f7c480ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4db07640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c4806d910 0x7f7c48146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 -- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c4806d910 msgr2=0x7f7c48146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c4806d910 0x7f7c48146190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 -- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c480c8e00 msgr2=0x7f7c480ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:38.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c480c8e00 0x7f7c480ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 -- 172.21.15.78:0/3803840513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c009400 con 0x7f7c48152700 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4e308640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 0x7f7c480aae00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c780 tx=0x7f7c4000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40004b60 con 0x7f7c48152700 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4000ce90 con 0x7f7c48152700 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.950+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c480aa320 con 0x7f7c48152700 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.951+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000a850 con 0x7f7c48152700 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.951+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c480aa8e0 con 0x7f7c48152700 2024-02-19T14:54:38.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.951+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c480aaad0 con 0x7f7c48152700 2024-02-19T14:54:38.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.953+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7c40006b50 con 0x7f7c48152700 2024-02-19T14:54:38.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.953+0000 7f7c3a7fc640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c2c06b810 0x7f7c2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:38.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.953+0000 7f7c4db07640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c2c06b810 0x7f7c2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:38.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.953+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7c4008b200 con 0x7f7c48152700 2024-02-19T14:54:38.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.954+0000 7f7c4db07640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c2c06b810 0x7f7c2c06dcd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000f60 tx=0x7f7c44007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:38.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:38.959+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7c400558b0 con 0x7f7c48152700 2024-02-19T14:54:39.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.176+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7c48074990 con 0x7f7c48152700 2024-02-19T14:54:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.830+0000 7f7c3a7fc640 1 -- 172.21.15.78:0/3803840513 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v54) v1 ==== 123+0+0 (secure 0 0 0) 0x7f7c40058f70 con 0x7f7c48152700 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c2c06b810 msgr2=0x7f7c2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c2c06b810 0x7f7c2c06dcd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000f60 tx=0x7f7c44007c00 comp rx=0 tx=0).stop 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 msgr2=0x7f7c480aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 0x7f7c480aae00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c780 tx=0x7f7c4000cc40 comp rx=0 tx=0).stop 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 shutdown_connections 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7c2c06b810 0x7f7c2c06dcd0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c48152700 0x7f7c480aae00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c480c8e00 0x7f7c480ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 --2- 172.21.15.78:0/3803840513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c4806d910 0x7f7c48146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 >> 172.21.15.78:0/3803840513 conn(0x7f7c480cac90 msgr2=0x7f7c4814c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 shutdown_connections 2024-02-19T14:54:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.833+0000 7f7c4fd6b640 1 -- 172.21.15.78:0/3803840513 wait complete. 2024-02-19T14:54:39.846 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-02-19T14:54:39.847 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 Processor -- start 2024-02-19T14:54:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 -- start start 2024-02-19T14:54:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf500c19f0 0x7fbf50156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 0x7fbf500c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf50155a50 con 0x7fbf500c7570 2024-02-19T14:54:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf50152af0 con 0x7fbf500c19f0 2024-02-19T14:54:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.965+0000 7fbf5590c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50155f80 con 0x7fbf50071410 2024-02-19T14:54:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 0x7fbf500c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf4effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf500c19f0 0x7fbf50156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 0x7fbf500c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:59544/0 (socket says 172.21.15.78:59544) 2024-02-19T14:54:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 -- 172.21.15.78:0/1377602245 learned_addr learned my addr 172.21.15.78:0/1377602245 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf4f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbf500c7570 0x7fbf500c1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:35824/0 (socket says 172.21.15.78:35824) 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 -- 172.21.15.78:0/1377602245 >> v2:172.21.15.150:3300/0 conn(0x7fbf500c19f0 msgr2=0x7fbf50156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 --2- 172.21.15.78:0/1377602245 >> v2:172.21.15.150:3300/0 conn(0x7fbf500c19f0 0x7fbf50156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 -- 172.21.15.78:0/1377602245 >> v1:172.21.15.150:6789/0 conn(0x7fbf500c7570 legacy=0x7fbf500c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf47fff640 1 -- 172.21.15.78:0/1377602245 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50070840 con 0x7fbf50071410 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf4e7fc640 1 -- 172.21.15.78:0/1377602245 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009135832 0 0) 0x7fbf50155a50 con 0x7fbf500c7570 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.966+0000 7fbf4effd640 1 --2- 172.21.15.78:0/1377602245 >> v2:172.21.15.150:3300/0 conn(0x7fbf500c19f0 0x7fbf50156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.967+0000 7fbf47fff640 1 --2- 172.21.15.78:0/1377602245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 0x7fbf500c7030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009870 tx=0x7fbf3802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fba4a683feec6f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf4e7fc640 1 -- 172.21.15.78:0/1377602245 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380317c0 con 0x7fbf50071410 2024-02-19T14:54:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf4e7fc640 1 -- 172.21.15.78:0/1377602245 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38031920 con 0x7fbf50071410 2024-02-19T14:54:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf4e7fc640 1 -- 172.21.15.78:0/1377602245 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38031cb0 con 0x7fbf50071410 2024-02-19T14:54:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 -- 172.21.15.78:0/1377602245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 msgr2=0x7fbf500c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1377602245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 0x7fbf500c7030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009870 tx=0x7fbf3802b860 comp rx=0 tx=0).stop 2024-02-19T14:54:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 -- 172.21.15.78:0/1377602245 shutdown_connections 2024-02-19T14:54:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1377602245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf50071410 0x7fbf500c7030 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1377602245 >> v2:172.21.15.150:3300/0 conn(0x7fbf500c19f0 0x7fbf50156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 -- 172.21.15.78:0/1377602245 >> 172.21.15.78:0/1377602245 conn(0x7fbf500aa370 msgr2=0x7fbf500aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 -- 172.21.15.78:0/1377602245 shutdown_connections 2024-02-19T14:54:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.968+0000 7fbf5590c640 1 -- 172.21.15.78:0/1377602245 wait complete. 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.969+0000 7fbf5590c640 1 Processor -- start 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.969+0000 7fbf5590c640 1 -- start start 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf5590c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 0x7fbf501e1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf5590c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf500c19f0 0x7fbf500adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf5590c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf500c7570 0x7fbf500ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf5590c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501e0af0 con 0x7fbf50071410 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf5590c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501dea30 con 0x7fbf500c19f0 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf5590c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501e23b0 con 0x7fbf500c7570 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf4effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 0x7fbf501e1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf4effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 0x7fbf501e1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32934/0 (socket says 172.21.15.78:32934) 2024-02-19T14:54:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf4effd640 1 -- 172.21.15.78:0/1837701133 learned_addr learned my addr 172.21.15.78:0/1837701133 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf47fff640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf500c19f0 0x7fbf500adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.970+0000 7fbf4f7fe640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf500c7570 0x7fbf500ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4effd640 1 -- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf500c7570 msgr2=0x7fbf500ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4effd640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf500c7570 0x7fbf500ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4effd640 1 -- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf500c19f0 msgr2=0x7fbf500adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4effd640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf500c19f0 0x7fbf500adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4effd640 1 -- 172.21.15.78:0/1837701133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40009400 con 0x7fbf50071410 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4effd640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 0x7fbf501e1ba0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000ec10 tx=0x7fbf4000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40009760 con 0x7fbf50071410 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4000cc30 con 0x7fbf50071410 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40013470 con 0x7fbf50071410 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf380094d0 con 0x7fbf50071410 2024-02-19T14:54:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.971+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501dec80 con 0x7fbf50071410 2024-02-19T14:54:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.972+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5006e4f0 con 0x7fbf50071410 2024-02-19T14:54:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.973+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbf400135d0 con 0x7fbf50071410 2024-02-19T14:54:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.973+0000 7fbf4cff9640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf3006b840 0x7fbf3006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.973+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fbf4008a8e0 con 0x7fbf50071410 2024-02-19T14:54:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.973+0000 7fbf47fff640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf3006b840 0x7fbf3006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.974+0000 7fbf47fff640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf3006b840 0x7fbf3006dd00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf38036000 tx=0x7fbf3803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:39.978+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbf40054ec0 con 0x7fbf50071410 2024-02-19T14:54:40.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.235+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf500c9460 con 0x7fbf50071410 2024-02-19T14:54:40.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.236+0000 7fbf4cff9640 1 -- 172.21.15.78:0/1837701133 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6183 (secure 0 0 0) 0x7fbf40058580 con 0x7fbf50071410 2024-02-19T14:54:40.238 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:40.239 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":54,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:28.097183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"c","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2137063883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":2137063883},{"type":"v1","addr":"172.21.15.78:6837","nonce":2137063883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:40.239 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 54 2024-02-19T14:54:40.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf3006b840 msgr2=0x7fbf3006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf3006b840 0x7fbf3006dd00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbf38036000 tx=0x7fbf3803d040 comp rx=0 tx=0).stop 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 msgr2=0x7fbf501e1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 0x7fbf501e1ba0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000ec10 tx=0x7fbf4000c520 comp rx=0 tx=0).stop 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 shutdown_connections 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbf3006b840 0x7fbf3006dd00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf500c7570 0x7fbf500ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf500c19f0 0x7fbf500adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 --2- 172.21.15.78:0/1837701133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf50071410 0x7fbf501e1ba0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 >> 172.21.15.78:0/1837701133 conn(0x7fbf500aa370 msgr2=0x7fbf50158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.238+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 shutdown_connections 2024-02-19T14:54:40.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.239+0000 7fbf5590c640 1 -- 172.21.15.78:0/1837701133 wait complete. 2024-02-19T14:54:40.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:40.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.354+0000 7f8389116640 1 Processor -- start 2024-02-19T14:54:40.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.354+0000 7f8389116640 1 -- start start 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.354+0000 7f8389116640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f838414f1a0 0x7f8384072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8389116640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8384152ab0 con 0x7f8384152680 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8389116640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83840c5970 con 0x7f838406ea50 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8389116640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83840c5750 con 0x7f838414f1a0 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8382d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f838414f1a0 0x7f8384072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f838359e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8384152680 0x7f83840c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54806/0 (socket says 172.21.15.78:54806) 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8382d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f838414f1a0 0x7f8384072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32942/0 (socket says 172.21.15.78:32942) 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f838359e640 1 -- 172.21.15.78:0/1624508236 learned_addr learned my addr 172.21.15.78:0/1624508236 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8382d9d640 1 -- 172.21.15.78:0/1624508236 >> v1:172.21.15.150:6789/0 conn(0x7f838406ea50 legacy=0x7f838406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8382d9d640 1 -- 172.21.15.78:0/1624508236 >> v1:172.21.15.78:6789/0 conn(0x7f8384152680 legacy=0x7f83840c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8382d9d640 1 -- 172.21.15.78:0/1624508236 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840702d0 con 0x7f838414f1a0 2024-02-19T14:54:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.355+0000 7f8381d9b640 1 -- 172.21.15.78:0/1624508236 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030128643 0 0) 0x7f8384152ab0 con 0x7f8384152680 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8382d9d640 1 --2- 172.21.15.78:0/1624508236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f838414f1a0 0x7f8384072d70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f83740097a0 tx=0x7f837402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b140ab1155ef4b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8381d9b640 1 -- 172.21.15.78:0/1624508236 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837402e070 con 0x7f838414f1a0 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8381d9b640 1 -- 172.21.15.78:0/1624508236 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83740315a0 con 0x7f838414f1a0 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8381d9b640 1 -- 172.21.15.78:0/1624508236 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374031930 con 0x7f838414f1a0 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8389116640 1 -- 172.21.15.78:0/1624508236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f838414f1a0 msgr2=0x7f8384072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8389116640 1 --2- 172.21.15.78:0/1624508236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f838414f1a0 0x7f8384072d70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f83740097a0 tx=0x7f837402b840 comp rx=0 tx=0).stop 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8389116640 1 -- 172.21.15.78:0/1624508236 shutdown_connections 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.356+0000 7f8389116640 1 --2- 172.21.15.78:0/1624508236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f838414f1a0 0x7f8384072d70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.357+0000 7f8389116640 1 -- 172.21.15.78:0/1624508236 >> 172.21.15.78:0/1624508236 conn(0x7f838406c170 msgr2=0x7f838406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.357+0000 7f8389116640 1 -- 172.21.15.78:0/1624508236 shutdown_connections 2024-02-19T14:54:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.357+0000 7f8389116640 1 -- 172.21.15.78:0/1624508236 wait complete. 2024-02-19T14:54:40.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.358+0000 7f8389116640 1 Processor -- start 2024-02-19T14:54:40.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.358+0000 7f8389116640 1 -- start start 2024-02-19T14:54:40.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f838406ea50 0x7f83841e9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f838414f1a0 0x7f838414e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 0x7f83841458c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83841537f0 con 0x7f8384152680 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83841e99d0 con 0x7f838414f1a0 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83840ab130 con 0x7f838406ea50 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 0x7f83841458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 0x7f83841458c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32956/0 (socket says 172.21.15.78:32956) 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 -- 172.21.15.78:0/1106409045 learned_addr learned my addr 172.21.15.78:0/1106409045 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8382d9d640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f838406ea50 0x7f83841e9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838259c640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f838414f1a0 0x7f838414e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 -- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f838406ea50 msgr2=0x7f83841e9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f838406ea50 0x7f83841e9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 -- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f838414f1a0 msgr2=0x7f838414e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f838414f1a0 0x7f838414e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 -- 172.21.15.78:0/1106409045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374009400 con 0x7f8384152680 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f838359e640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 0x7f83841458c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f837c00c7a0 tx=0x7f837c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c00a5e0 con 0x7f8384152680 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c00a740 con 0x7f8384152680 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83840aa520 con 0x7f8384152680 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.359+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c00ab70 con 0x7f8384152680 2024-02-19T14:54:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.360+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83840aaae0 con 0x7f8384152680 2024-02-19T14:54:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.360+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83840a7b50 con 0x7f8384152680 2024-02-19T14:54:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.361+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f837c00acd0 con 0x7f8384152680 2024-02-19T14:54:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.362+0000 7f836b7fe640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837006b810 0x7f837006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.362+0000 7f8382d9d640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837006b810 0x7f837006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.362+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f837c089e60 con 0x7f8384152680 2024-02-19T14:54:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.363+0000 7f8382d9d640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837006b810 0x7f837006dcd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8374036000 tx=0x7f837403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:40.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.370+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f837c054430 con 0x7f8384152680 2024-02-19T14:54:40.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.551+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8384074990 con 0x7f8384152680 2024-02-19T14:54:40.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.552+0000 7f836b7fe640 1 -- 172.21.15.78:0/1106409045 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20657 (secure 0 0 0) 0x7f837c018020 con 0x7f8384152680 2024-02-19T14:54:40.555 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:40.555 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":79,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:38.820692+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:40.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837006b810 msgr2=0x7f837006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837006b810 0x7f837006dcd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8374036000 tx=0x7f837403d040 comp rx=0 tx=0).stop 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 msgr2=0x7f83841458c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 0x7f83841458c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f837c00c7a0 tx=0x7f837c00cc60 comp rx=0 tx=0).stop 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 shutdown_connections 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837006b810 0x7f837006dcd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8384152680 0x7f83841458c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f838414f1a0 0x7f838414e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 --2- 172.21.15.78:0/1106409045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f838406ea50 0x7f83841e9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.554+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 >> 172.21.15.78:0/1106409045 conn(0x7f838406c170 msgr2=0x7f83840a8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.555+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 shutdown_connections 2024-02-19T14:54:40.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.555+0000 7f8389116640 1 -- 172.21.15.78:0/1106409045 wait complete. 2024-02-19T14:54:40.572 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:54:40.572 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.673+0000 7f630df6c640 1 Processor -- start 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.673+0000 7f630df6c640 1 -- start start 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f630df6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6308152700 0x7f63080c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f630df6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 0x7f630806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f630df6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63081551f0 con 0x7f63080c8e00 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f630df6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6308155490 con 0x7f6308152700 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f630df6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6308155770 con 0x7f63080c5730 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f6307fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f63080c8e00 0x7f6308158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54816/0 (socket says 172.21.15.78:54816) 2024-02-19T14:54:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f6307fff640 1 -- 172.21.15.78:0/4091256385 learned_addr learned my addr 172.21.15.78:0/4091256385 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f6306ffd640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 0x7f630806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f63077fe640 1 --2- 172.21.15.78:0/4091256385 >> v2:172.21.15.150:3300/0 conn(0x7f6308152700 0x7f63080c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058902262 0 0) 0x7f63081551f0 con 0x7f63080c8e00 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.674+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f0003520 con 0x7f63080c8e00 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 750318857 0 0) 0x7f62f0003520 con 0x7f63080c8e00 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 msgr2=0x7f630806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 0x7f630806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 >> v2:172.21.15.150:3300/0 conn(0x7f6308152700 msgr2=0x7f63080c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 --2- 172.21.15.78:0/4091256385 >> v2:172.21.15.150:3300/0 conn(0x7f6308152700 0x7f63080c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6308155a50 con 0x7f63080c8e00 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f63000019f0 con 0x7f63080c8e00 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 >> v1:172.21.15.78:6789/0 conn(0x7f63080c8e00 legacy=0x7f6308158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 0x7f62f0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f62f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d00 0x7f62f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308155490 con 0x7f62f0003ef0 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308155770 con 0x7f62f0004d00 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f000b980 con 0x7f63080c5730 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6306ffd640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f62f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f63077fe640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 0x7f62f0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6307fff640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d00 0x7f62f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6306ffd640 1 -- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 msgr2=0x7f62f0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6306ffd640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c5730 0x7f62f0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6306ffd640 1 -- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d00 msgr2=0x7f62f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6306ffd640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d00 0x7f62f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.675+0000 7f6306ffd640 1 -- 172.21.15.78:0/4091256385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0003520 con 0x7f62f0003ef0 2024-02-19T14:54:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.676+0000 7f6306ffd640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f62f00047c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009ac0 tx=0x7f62f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f2eb6c14b2a108 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.676+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc02c6b0 con 0x7f62f0003ef0 2024-02-19T14:54:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.676+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc02c810 con 0x7f62f0003ef0 2024-02-19T14:54:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.676+0000 7f63067fc640 1 -- 172.21.15.78:0/4091256385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc02cba0 con 0x7f62f0003ef0 2024-02-19T14:54:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.676+0000 7f630df6c640 1 -- 172.21.15.78:0/4091256385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 msgr2=0x7f62f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.677+0000 7f630df6c640 1 --2- 172.21.15.78:0/4091256385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f62f00047c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009ac0 tx=0x7f62f00057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.677+0000 7f6307fff640 1 -- 172.21.15.78:0/4091256385 reap_dead start 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.677+0000 7f630df6c640 1 -- 172.21.15.78:0/4091256385 shutdown_connections 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.677+0000 7f630df6c640 1 -- 172.21.15.78:0/4091256385 >> 172.21.15.78:0/4091256385 conn(0x7f63080cac90 msgr2=0x7f63080cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.677+0000 7f630df6c640 1 -- 172.21.15.78:0/4091256385 shutdown_connections 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.677+0000 7f630df6c640 1 -- 172.21.15.78:0/4091256385 wait complete. 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.678+0000 7f630df6c640 1 Processor -- start 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.679+0000 7f630df6c640 1 -- start start 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f6308154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63081ee3b0 0x7f63080be930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080bee70 0x7f63081ea430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080be510 con 0x7f62f0003ef0 2024-02-19T14:54:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080ac550 con 0x7f63081ee3b0 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63081ee970 con 0x7f63080bee70 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f6308154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f6308154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32970/0 (socket says 172.21.15.78:32970) 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 -- 172.21.15.78:0/2989162808 learned_addr learned my addr 172.21.15.78:0/2989162808 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f6306ffd640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63081ee3b0 0x7f63080be930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f6307fff640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080bee70 0x7f63081ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 -- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080bee70 msgr2=0x7f63081ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080bee70 0x7f63081ea430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 -- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63081ee3b0 msgr2=0x7f63080be930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63081ee3b0 0x7f63080be930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 -- 172.21.15.78:0/2989162808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080aa2b0 con 0x7f62f0003ef0 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f6307fff640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080bee70 0x7f63081ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f63077fe640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f6308154280 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f62f8011a70 tx=0x7f62f8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8009680 con 0x7f62f0003ef0 2024-02-19T14:54:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f800fb50 con 0x7f62f0003ef0 2024-02-19T14:54:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f800fee0 con 0x7f62f0003ef0 2024-02-19T14:54:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.680+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080c0120 con 0x7f62f0003ef0 2024-02-19T14:54:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.681+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080c06e0 con 0x7f62f0003ef0 2024-02-19T14:54:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.682+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308074990 con 0x7f62f0003ef0 2024-02-19T14:54:40.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.683+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f62f8014440 con 0x7f62f0003ef0 2024-02-19T14:54:40.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.683+0000 7f62effff640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62f406b810 0x7f62f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:40.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.683+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f62f808aee0 con 0x7f62f0003ef0 2024-02-19T14:54:40.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.683+0000 7f6306ffd640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62f406b810 0x7f62f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:40.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.684+0000 7f6306ffd640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62f406b810 0x7f62f406dcd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f62fc036000 tx=0x7f62fc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:40.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.688+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62f80554b0 con 0x7f62f0003ef0 2024-02-19T14:54:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.893+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f630806ded0 con 0x7f62f406b810 2024-02-19T14:54:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.905+0000 7f62effff640 1 -- 172.21.15.78:0/2989162808 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528014 (secure 0 0 0) 0x7f62fc034440 con 0x7f62f406b810 2024-02-19T14:54:40.913 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:40.917 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:54:40.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62f406b810 msgr2=0x7f62f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62f406b810 0x7f62f406dcd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f62fc036000 tx=0x7f62fc030000 comp rx=0 tx=0).stop 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 msgr2=0x7f6308154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f6308154280 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f62f8011a70 tx=0x7f62f8011f30 comp rx=0 tx=0).stop 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 shutdown_connections 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62f406b810 0x7f62f406dcd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080bee70 0x7f63081ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63081ee3b0 0x7f63080be930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 --2- 172.21.15.78:0/2989162808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f0003ef0 0x7f6308154280 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 >> 172.21.15.78:0/2989162808 conn(0x7f63080cac90 msgr2=0x7f6308150820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 shutdown_connections 2024-02-19T14:54:40.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:40.910+0000 7f630df6c640 1 -- 172.21.15.78:0/2989162808 wait complete. 2024-02-19T14:54:40.924 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-02-19T14:54:40.842732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":261,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224784,"kb_used_data":8612,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754749936,"statfs":{"total":773094113280,"available":772863934464,"internally_reserved":0,"allocated":8818688,"data_stored":4796700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.751745"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842491+0000","last_change":"2024-02-19T14:54:37.837507+0000","last_active":"2024-02-19T14:54:37.842491+0000","last_peered":"2024-02-19T14:54:37.842491+0000","last_clean":"2024-02-19T14:54:37.842491+0000","last_became_active":"2024-02-19T14:54:37.837268+0000","last_became_peered":"2024-02-19T14:54:37.837268+0000","last_unstale":"2024-02-19T14:54:37.842491+0000","last_undegraded":"2024-02-19T14:54:37.842491+0000","last_fullsized":"2024-02-19T14:54:37.842491+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:05.336408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823128+0000","last_change":"2024-02-19T14:54:26.743766+0000","last_active":"2024-02-19T14:54:37.823128+0000","last_peered":"2024-02-19T14:54:37.823128+0000","last_clean":"2024-02-19T14:54:37.823128+0000","last_became_active":"2024-02-19T14:54:26.743402+0000","last_became_peered":"2024-02-19T14:54:26.743402+0000","last_unstale":"2024-02-19T14:54:37.823128+0000","last_undegraded":"2024-02-19T14:54:37.823128+0000","last_fullsized":"2024-02-19T14:54:37.823128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846266+0000","last_change":"2024-02-19T14:54:37.842884+0000","last_active":"2024-02-19T14:54:37.846266+0000","last_peered":"2024-02-19T14:54:37.846266+0000","last_clean":"2024-02-19T14:54:37.846266+0000","last_became_active":"2024-02-19T14:54:37.842649+0000","last_became_peered":"2024-02-19T14:54:37.842649+0000","last_unstale":"2024-02-19T14:54:37.846266+0000","last_undegraded":"2024-02-19T14:54:37.846266+0000","last_fullsized":"2024-02-19T14:54:37.846266+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:18.993081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.831094+0000","last_change":"2024-02-19T14:54:26.751673+0000","last_active":"2024-02-19T14:54:37.831094+0000","last_peered":"2024-02-19T14:54:37.831094+0000","last_clean":"2024-02-19T14:54:37.831094+0000","last_became_active":"2024-02-19T14:54:26.751360+0000","last_became_peered":"2024-02-19T14:54:26.751360+0000","last_unstale":"2024-02-19T14:54:37.831094+0000","last_undegraded":"2024-02-19T14:54:37.831094+0000","last_fullsized":"2024-02-19T14:54:37.831094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844010+0000","last_change":"2024-02-19T14:54:37.843084+0000","last_active":"2024-02-19T14:54:37.844010+0000","last_peered":"2024-02-19T14:54:37.844010+0000","last_clean":"2024-02-19T14:54:37.844010+0000","last_became_active":"2024-02-19T14:54:37.842512+0000","last_became_peered":"2024-02-19T14:54:37.842512+0000","last_unstale":"2024-02-19T14:54:37.844010+0000","last_undegraded":"2024-02-19T14:54:37.844010+0000","last_fullsized":"2024-02-19T14:54:37.844010+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:36:42.957311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821285+0000","last_change":"2024-02-19T14:54:26.750603+0000","last_active":"2024-02-19T14:54:37.821285+0000","last_peered":"2024-02-19T14:54:37.821285+0000","last_clean":"2024-02-19T14:54:37.821285+0000","last_became_active":"2024-02-19T14:54:26.750088+0000","last_became_peered":"2024-02-19T14:54:26.750088+0000","last_unstale":"2024-02-19T14:54:37.821285+0000","last_undegraded":"2024-02-19T14:54:37.821285+0000","last_fullsized":"2024-02-19T14:54:37.821285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:47.414910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841260+0000","last_change":"2024-02-19T14:54:37.838099+0000","last_active":"2024-02-19T14:54:37.841260+0000","last_peered":"2024-02-19T14:54:37.841260+0000","last_clean":"2024-02-19T14:54:37.841260+0000","last_became_active":"2024-02-19T14:54:37.837824+0000","last_became_peered":"2024-02-19T14:54:37.837824+0000","last_unstale":"2024-02-19T14:54:37.841260+0000","last_undegraded":"2024-02-19T14:54:37.841260+0000","last_fullsized":"2024-02-19T14:54:37.841260+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:48.394105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822248+0000","last_change":"2024-02-19T14:54:26.733014+0000","last_active":"2024-02-19T14:54:37.822248+0000","last_peered":"2024-02-19T14:54:37.822248+0000","last_clean":"2024-02-19T14:54:37.822248+0000","last_became_active":"2024-02-19T14:54:26.732738+0000","last_became_peered":"2024-02-19T14:54:26.732738+0000","last_unstale":"2024-02-19T14:54:37.822248+0000","last_undegraded":"2024-02-19T14:54:37.822248+0000","last_fullsized":"2024-02-19T14:54:37.822248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:37.293731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835059+0000","last_change":"2024-02-19T14:54:37.833022+0000","last_active":"2024-02-19T14:54:37.835059+0000","last_peered":"2024-02-19T14:54:37.835059+0000","last_clean":"2024-02-19T14:54:37.835059+0000","last_became_active":"2024-02-19T14:54:37.831958+0000","last_became_peered":"2024-02-19T14:54:37.831958+0000","last_unstale":"2024-02-19T14:54:37.835059+0000","last_undegraded":"2024-02-19T14:54:37.835059+0000","last_fullsized":"2024-02-19T14:54:37.835059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:46:45.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815053+0000","last_change":"2024-02-19T14:54:26.747816+0000","last_active":"2024-02-19T14:54:37.815053+0000","last_peered":"2024-02-19T14:54:37.815053+0000","last_clean":"2024-02-19T14:54:37.815053+0000","last_became_active":"2024-02-19T14:54:26.747520+0000","last_became_peered":"2024-02-19T14:54:26.747520+0000","last_unstale":"2024-02-19T14:54:37.815053+0000","last_undegraded":"2024-02-19T14:54:37.815053+0000","last_fullsized":"2024-02-19T14:54:37.815053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:36.704748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840622+0000","last_change":"2024-02-19T14:54:37.836146+0000","last_active":"2024-02-19T14:54:37.840622+0000","last_peered":"2024-02-19T14:54:37.840622+0000","last_clean":"2024-02-19T14:54:37.840622+0000","last_became_active":"2024-02-19T14:54:37.835654+0000","last_became_peered":"2024-02-19T14:54:37.835654+0000","last_unstale":"2024-02-19T14:54:37.840622+0000","last_undegraded":"2024-02-19T14:54:37.840622+0000","last_fullsized":"2024-02-19T14:54:37.840622+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:24.580690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819866+0000","last_change":"2024-02-19T14:54:26.746700+0000","last_active":"2024-02-19T14:54:37.819866+0000","last_peered":"2024-02-19T14:54:37.819866+0000","last_clean":"2024-02-19T14:54:37.819866+0000","last_became_active":"2024-02-19T14:54:26.746436+0000","last_became_peered":"2024-02-19T14:54:26.746436+0000","last_unstale":"2024-02-19T14:54:37.819866+0000","last_undegraded":"2024-02-19T14:54:37.819866+0000","last_fullsized":"2024-02-19T14:54:37.819866+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843709+0000","last_change":"2024-02-19T14:54:37.837088+0000","last_active":"2024-02-19T14:54:37.843709+0000","last_peered":"2024-02-19T14:54:37.843709+0000","last_clean":"2024-02-19T14:54:37.843709+0000","last_became_active":"2024-02-19T14:54:37.835520+0000","last_became_peered":"2024-02-19T14:54:37.835520+0000","last_unstale":"2024-02-19T14:54:37.843709+0000","last_undegraded":"2024-02-19T14:54:37.843709+0000","last_fullsized":"2024-02-19T14:54:37.843709+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:51.437501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817399+0000","last_change":"2024-02-19T14:54:26.751787+0000","last_active":"2024-02-19T14:54:37.817399+0000","last_peered":"2024-02-19T14:54:37.817399+0000","last_clean":"2024-02-19T14:54:37.817399+0000","last_became_active":"2024-02-19T14:54:26.751414+0000","last_became_peered":"2024-02-19T14:54:26.751414+0000","last_unstale":"2024-02-19T14:54:37.817399+0000","last_undegraded":"2024-02-19T14:54:37.817399+0000","last_fullsized":"2024-02-19T14:54:37.817399+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:31.766567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.848706+0000","last_change":"2024-02-19T14:54:37.847071+0000","last_active":"2024-02-19T14:54:37.848706+0000","last_peered":"2024-02-19T14:54:37.848706+0000","last_clean":"2024-02-19T14:54:37.848706+0000","last_became_active":"2024-02-19T14:54:37.846827+0000","last_became_peered":"2024-02-19T14:54:37.846827+0000","last_unstale":"2024-02-19T14:54:37.848706+0000","last_undegraded":"2024-02-19T14:54:37.848706+0000","last_fullsized":"2024-02-19T14:54:37.848706+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:31.003078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814652+0000","last_change":"2024-02-19T14:54:26.748066+0000","last_active":"2024-02-19T14:54:37.814652+0000","last_peered":"2024-02-19T14:54:37.814652+0000","last_clean":"2024-02-19T14:54:37.814652+0000","last_became_active":"2024-02-19T14:54:26.747792+0000","last_became_peered":"2024-02-19T14:54:26.747792+0000","last_unstale":"2024-02-19T14:54:37.814652+0000","last_undegraded":"2024-02-19T14:54:37.814652+0000","last_fullsized":"2024-02-19T14:54:37.814652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:05.620631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839543+0000","last_change":"2024-02-19T14:54:37.832366+0000","last_active":"2024-02-19T14:54:37.839543+0000","last_peered":"2024-02-19T14:54:37.839543+0000","last_clean":"2024-02-19T14:54:37.839543+0000","last_became_active":"2024-02-19T14:54:37.832111+0000","last_became_peered":"2024-02-19T14:54:37.832111+0000","last_unstale":"2024-02-19T14:54:37.839543+0000","last_undegraded":"2024-02-19T14:54:37.839543+0000","last_fullsized":"2024-02-19T14:54:37.839543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:59.425037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819364+0000","last_change":"2024-02-19T14:54:26.741238+0000","last_active":"2024-02-19T14:54:37.819364+0000","last_peered":"2024-02-19T14:54:37.819364+0000","last_clean":"2024-02-19T14:54:37.819364+0000","last_became_active":"2024-02-19T14:54:26.740986+0000","last_became_peered":"2024-02-19T14:54:26.740986+0000","last_unstale":"2024-02-19T14:54:37.819364+0000","last_undegraded":"2024-02-19T14:54:37.819364+0000","last_fullsized":"2024-02-19T14:54:37.819364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:30.855089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836454+0000","last_change":"2024-02-19T14:54:37.831523+0000","last_active":"2024-02-19T14:54:37.836454+0000","last_peered":"2024-02-19T14:54:37.836454+0000","last_clean":"2024-02-19T14:54:37.836454+0000","last_became_active":"2024-02-19T14:54:37.831272+0000","last_became_peered":"2024-02-19T14:54:37.831272+0000","last_unstale":"2024-02-19T14:54:37.836454+0000","last_undegraded":"2024-02-19T14:54:37.836454+0000","last_fullsized":"2024-02-19T14:54:37.836454+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:54.496496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820662+0000","last_change":"2024-02-19T14:54:26.741087+0000","last_active":"2024-02-19T14:54:37.820662+0000","last_peered":"2024-02-19T14:54:37.820662+0000","last_clean":"2024-02-19T14:54:37.820662+0000","last_became_active":"2024-02-19T14:54:26.740795+0000","last_became_peered":"2024-02-19T14:54:26.740795+0000","last_unstale":"2024-02-19T14:54:37.820662+0000","last_undegraded":"2024-02-19T14:54:37.820662+0000","last_fullsized":"2024-02-19T14:54:37.820662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:50.652136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844390+0000","last_change":"2024-02-19T14:54:37.839043+0000","last_active":"2024-02-19T14:54:37.844390+0000","last_peered":"2024-02-19T14:54:37.844390+0000","last_clean":"2024-02-19T14:54:37.844390+0000","last_became_active":"2024-02-19T14:54:37.838128+0000","last_became_peered":"2024-02-19T14:54:37.838128+0000","last_unstale":"2024-02-19T14:54:37.844390+0000","last_undegraded":"2024-02-19T14:54:37.844390+0000","last_fullsized":"2024-02-19T14:54:37.844390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:48.384374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.829864+0000","last_change":"2024-02-19T14:54:26.753214+0000","last_active":"2024-02-19T14:54:37.829864+0000","last_peered":"2024-02-19T14:54:37.829864+0000","last_clean":"2024-02-19T14:54:37.829864+0000","last_became_active":"2024-02-19T14:54:26.752817+0000","last_became_peered":"2024-02-19T14:54:26.752817+0000","last_unstale":"2024-02-19T14:54:37.829864+0000","last_undegraded":"2024-02-19T14:54:37.829864+0000","last_fullsized":"2024-02-19T14:54:37.829864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836972+0000","last_change":"2024-02-19T14:54:37.831625+0000","last_active":"2024-02-19T14:54:37.836972+0000","last_peered":"2024-02-19T14:54:37.836972+0000","last_clean":"2024-02-19T14:54:37.836972+0000","last_became_active":"2024-02-19T14:54:37.831376+0000","last_became_peered":"2024-02-19T14:54:37.831376+0000","last_unstale":"2024-02-19T14:54:37.836972+0000","last_undegraded":"2024-02-19T14:54:37.836972+0000","last_fullsized":"2024-02-19T14:54:37.836972+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:41.610814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822342+0000","last_change":"2024-02-19T14:54:26.752439+0000","last_active":"2024-02-19T14:54:37.822342+0000","last_peered":"2024-02-19T14:54:37.822342+0000","last_clean":"2024-02-19T14:54:37.822342+0000","last_became_active":"2024-02-19T14:54:26.752104+0000","last_became_peered":"2024-02-19T14:54:26.752104+0000","last_unstale":"2024-02-19T14:54:37.822342+0000","last_undegraded":"2024-02-19T14:54:37.822342+0000","last_fullsized":"2024-02-19T14:54:37.822342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:29.434729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838842+0000","last_change":"2024-02-19T14:54:37.833202+0000","last_active":"2024-02-19T14:54:37.838842+0000","last_peered":"2024-02-19T14:54:37.838842+0000","last_clean":"2024-02-19T14:54:37.838842+0000","last_became_active":"2024-02-19T14:54:37.832943+0000","last_became_peered":"2024-02-19T14:54:37.832943+0000","last_unstale":"2024-02-19T14:54:37.838842+0000","last_undegraded":"2024-02-19T14:54:37.838842+0000","last_fullsized":"2024-02-19T14:54:37.838842+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:51.437501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815465+0000","last_change":"2024-02-19T14:54:26.746983+0000","last_active":"2024-02-19T14:54:37.815465+0000","last_peered":"2024-02-19T14:54:37.815465+0000","last_clean":"2024-02-19T14:54:37.815465+0000","last_became_active":"2024-02-19T14:54:26.746664+0000","last_became_peered":"2024-02-19T14:54:26.746664+0000","last_unstale":"2024-02-19T14:54:37.815465+0000","last_undegraded":"2024-02-19T14:54:37.815465+0000","last_fullsized":"2024-02-19T14:54:37.815465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:27.405179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845627+0000","last_change":"2024-02-19T14:54:37.837657+0000","last_active":"2024-02-19T14:54:37.845627+0000","last_peered":"2024-02-19T14:54:37.845627+0000","last_clean":"2024-02-19T14:54:37.845627+0000","last_became_active":"2024-02-19T14:54:37.837050+0000","last_became_peered":"2024-02-19T14:54:37.837050+0000","last_unstale":"2024-02-19T14:54:37.845627+0000","last_undegraded":"2024-02-19T14:54:37.845627+0000","last_fullsized":"2024-02-19T14:54:37.845627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:46:45.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819956+0000","last_change":"2024-02-19T14:54:26.744541+0000","last_active":"2024-02-19T14:54:37.819956+0000","last_peered":"2024-02-19T14:54:37.819956+0000","last_clean":"2024-02-19T14:54:37.819956+0000","last_became_active":"2024-02-19T14:54:26.744242+0000","last_became_peered":"2024-02-19T14:54:26.744242+0000","last_unstale":"2024-02-19T14:54:37.819956+0000","last_undegraded":"2024-02-19T14:54:37.819956+0000","last_fullsized":"2024-02-19T14:54:37.819956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840428+0000","last_change":"2024-02-19T14:54:37.834913+0000","last_active":"2024-02-19T14:54:37.840428+0000","last_peered":"2024-02-19T14:54:37.840428+0000","last_clean":"2024-02-19T14:54:37.840428+0000","last_became_active":"2024-02-19T14:54:37.834674+0000","last_became_peered":"2024-02-19T14:54:37.834674+0000","last_unstale":"2024-02-19T14:54:37.840428+0000","last_undegraded":"2024-02-19T14:54:37.840428+0000","last_fullsized":"2024-02-19T14:54:37.840428+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:15.913328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826352+0000","last_change":"2024-02-19T14:54:26.745146+0000","last_active":"2024-02-19T14:54:37.826352+0000","last_peered":"2024-02-19T14:54:37.826352+0000","last_clean":"2024-02-19T14:54:37.826352+0000","last_became_active":"2024-02-19T14:54:26.744342+0000","last_became_peered":"2024-02-19T14:54:26.744342+0000","last_unstale":"2024-02-19T14:54:37.826352+0000","last_undegraded":"2024-02-19T14:54:37.826352+0000","last_fullsized":"2024-02-19T14:54:37.826352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843692+0000","last_change":"2024-02-19T14:54:37.836998+0000","last_active":"2024-02-19T14:54:37.843692+0000","last_peered":"2024-02-19T14:54:37.843692+0000","last_clean":"2024-02-19T14:54:37.843692+0000","last_became_active":"2024-02-19T14:54:37.835908+0000","last_became_peered":"2024-02-19T14:54:37.835908+0000","last_unstale":"2024-02-19T14:54:37.843692+0000","last_undegraded":"2024-02-19T14:54:37.843692+0000","last_fullsized":"2024-02-19T14:54:37.843692+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:43.244541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821197+0000","last_change":"2024-02-19T14:54:26.743966+0000","last_active":"2024-02-19T14:54:37.821197+0000","last_peered":"2024-02-19T14:54:37.821197+0000","last_clean":"2024-02-19T14:54:37.821197+0000","last_became_active":"2024-02-19T14:54:26.743693+0000","last_became_peered":"2024-02-19T14:54:26.743693+0000","last_unstale":"2024-02-19T14:54:37.821197+0000","last_undegraded":"2024-02-19T14:54:37.821197+0000","last_fullsized":"2024-02-19T14:54:37.821197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839625+0000","last_change":"2024-02-19T14:54:37.833579+0000","last_active":"2024-02-19T14:54:37.839625+0000","last_peered":"2024-02-19T14:54:37.839625+0000","last_clean":"2024-02-19T14:54:37.839625+0000","last_became_active":"2024-02-19T14:54:37.833293+0000","last_became_peered":"2024-02-19T14:54:37.833293+0000","last_unstale":"2024-02-19T14:54:37.839625+0000","last_undegraded":"2024-02-19T14:54:37.839625+0000","last_fullsized":"2024-02-19T14:54:37.839625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:14.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814856+0000","last_change":"2024-02-19T14:54:26.747599+0000","last_active":"2024-02-19T14:54:37.814856+0000","last_peered":"2024-02-19T14:54:37.814856+0000","last_clean":"2024-02-19T14:54:37.814856+0000","last_became_active":"2024-02-19T14:54:26.747345+0000","last_became_peered":"2024-02-19T14:54:26.747345+0000","last_unstale":"2024-02-19T14:54:37.814856+0000","last_undegraded":"2024-02-19T14:54:37.814856+0000","last_fullsized":"2024-02-19T14:54:37.814856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:20:30.359001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840582+0000","last_change":"2024-02-19T14:54:37.833641+0000","last_active":"2024-02-19T14:54:37.840582+0000","last_peered":"2024-02-19T14:54:37.840582+0000","last_clean":"2024-02-19T14:54:37.840582+0000","last_became_active":"2024-02-19T14:54:37.833394+0000","last_became_peered":"2024-02-19T14:54:37.833394+0000","last_unstale":"2024-02-19T14:54:37.840582+0000","last_undegraded":"2024-02-19T14:54:37.840582+0000","last_fullsized":"2024-02-19T14:54:37.840582+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:40.525372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822151+0000","last_change":"2024-02-19T14:54:26.736611+0000","last_active":"2024-02-19T14:54:37.822151+0000","last_peered":"2024-02-19T14:54:37.822151+0000","last_clean":"2024-02-19T14:54:37.822151+0000","last_became_active":"2024-02-19T14:54:26.736284+0000","last_became_peered":"2024-02-19T14:54:26.736284+0000","last_unstale":"2024-02-19T14:54:37.822151+0000","last_undegraded":"2024-02-19T14:54:37.822151+0000","last_fullsized":"2024-02-19T14:54:37.822151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:15.218606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845610+0000","last_change":"2024-02-19T14:54:37.837888+0000","last_active":"2024-02-19T14:54:37.845610+0000","last_peered":"2024-02-19T14:54:37.845610+0000","last_clean":"2024-02-19T14:54:37.845610+0000","last_became_active":"2024-02-19T14:54:37.836846+0000","last_became_peered":"2024-02-19T14:54:37.836846+0000","last_unstale":"2024-02-19T14:54:37.845610+0000","last_undegraded":"2024-02-19T14:54:37.845610+0000","last_fullsized":"2024-02-19T14:54:37.845610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:56:31.705336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819882+0000","last_change":"2024-02-19T14:54:26.749665+0000","last_active":"2024-02-19T14:54:37.819882+0000","last_peered":"2024-02-19T14:54:37.819882+0000","last_clean":"2024-02-19T14:54:37.819882+0000","last_became_active":"2024-02-19T14:54:26.749397+0000","last_became_peered":"2024-02-19T14:54:26.749397+0000","last_unstale":"2024-02-19T14:54:37.819882+0000","last_undegraded":"2024-02-19T14:54:37.819882+0000","last_fullsized":"2024-02-19T14:54:37.819882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842456+0000","last_change":"2024-02-19T14:54:37.836287+0000","last_active":"2024-02-19T14:54:37.842456+0000","last_peered":"2024-02-19T14:54:37.842456+0000","last_clean":"2024-02-19T14:54:37.842456+0000","last_became_active":"2024-02-19T14:54:37.835722+0000","last_became_peered":"2024-02-19T14:54:37.835722+0000","last_unstale":"2024-02-19T14:54:37.842456+0000","last_undegraded":"2024-02-19T14:54:37.842456+0000","last_fullsized":"2024-02-19T14:54:37.842456+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:20.338152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815123+0000","last_change":"2024-02-19T14:54:26.747148+0000","last_active":"2024-02-19T14:54:37.815123+0000","last_peered":"2024-02-19T14:54:37.815123+0000","last_clean":"2024-02-19T14:54:37.815123+0000","last_became_active":"2024-02-19T14:54:26.746685+0000","last_became_peered":"2024-02-19T14:54:26.746685+0000","last_unstale":"2024-02-19T14:54:37.815123+0000","last_undegraded":"2024-02-19T14:54:37.815123+0000","last_fullsized":"2024-02-19T14:54:37.815123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:44:34.240945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841117+0000","last_change":"2024-02-19T14:54:37.836254+0000","last_active":"2024-02-19T14:54:37.841117+0000","last_peered":"2024-02-19T14:54:37.841117+0000","last_clean":"2024-02-19T14:54:37.841117+0000","last_became_active":"2024-02-19T14:54:37.835938+0000","last_became_peered":"2024-02-19T14:54:37.835938+0000","last_unstale":"2024-02-19T14:54:37.841117+0000","last_undegraded":"2024-02-19T14:54:37.841117+0000","last_fullsized":"2024-02-19T14:54:37.841117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:49.644151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827142+0000","last_change":"2024-02-19T14:54:26.738144+0000","last_active":"2024-02-19T14:54:37.827142+0000","last_peered":"2024-02-19T14:54:37.827142+0000","last_clean":"2024-02-19T14:54:37.827142+0000","last_became_active":"2024-02-19T14:54:26.737816+0000","last_became_peered":"2024-02-19T14:54:26.737816+0000","last_unstale":"2024-02-19T14:54:37.827142+0000","last_undegraded":"2024-02-19T14:54:37.827142+0000","last_fullsized":"2024-02-19T14:54:37.827142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:53.244189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.834020+0000","last_change":"2024-02-19T14:54:37.830128+0000","last_active":"2024-02-19T14:54:37.834020+0000","last_peered":"2024-02-19T14:54:37.834020+0000","last_clean":"2024-02-19T14:54:37.834020+0000","last_became_active":"2024-02-19T14:54:37.829171+0000","last_became_peered":"2024-02-19T14:54:37.829171+0000","last_unstale":"2024-02-19T14:54:37.834020+0000","last_undegraded":"2024-02-19T14:54:37.834020+0000","last_fullsized":"2024-02-19T14:54:37.834020+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:52:53.015938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815123+0000","last_change":"2024-02-19T14:54:26.752317+0000","last_active":"2024-02-19T14:54:37.815123+0000","last_peered":"2024-02-19T14:54:37.815123+0000","last_clean":"2024-02-19T14:54:37.815123+0000","last_became_active":"2024-02-19T14:54:26.752031+0000","last_became_peered":"2024-02-19T14:54:26.752031+0000","last_unstale":"2024-02-19T14:54:37.815123+0000","last_undegraded":"2024-02-19T14:54:37.815123+0000","last_fullsized":"2024-02-19T14:54:37.815123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:39.758301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843570+0000","last_change":"2024-02-19T14:54:37.835043+0000","last_active":"2024-02-19T14:54:37.843570+0000","last_peered":"2024-02-19T14:54:37.843570+0000","last_clean":"2024-02-19T14:54:37.843570+0000","last_became_active":"2024-02-19T14:54:37.834163+0000","last_became_peered":"2024-02-19T14:54:37.834163+0000","last_unstale":"2024-02-19T14:54:37.843570+0000","last_undegraded":"2024-02-19T14:54:37.843570+0000","last_fullsized":"2024-02-19T14:54:37.843570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:38.903266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822655+0000","last_change":"2024-02-19T14:54:26.733087+0000","last_active":"2024-02-19T14:54:37.822655+0000","last_peered":"2024-02-19T14:54:37.822655+0000","last_clean":"2024-02-19T14:54:37.822655+0000","last_became_active":"2024-02-19T14:54:26.732793+0000","last_became_peered":"2024-02-19T14:54:26.732793+0000","last_unstale":"2024-02-19T14:54:37.822655+0000","last_undegraded":"2024-02-19T14:54:37.822655+0000","last_fullsized":"2024-02-19T14:54:37.822655+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:13.700400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840039+0000","last_change":"2024-02-19T14:54:37.834374+0000","last_active":"2024-02-19T14:54:37.840039+0000","last_peered":"2024-02-19T14:54:37.840039+0000","last_clean":"2024-02-19T14:54:37.840039+0000","last_became_active":"2024-02-19T14:54:37.834083+0000","last_became_peered":"2024-02-19T14:54:37.834083+0000","last_unstale":"2024-02-19T14:54:37.840039+0000","last_undegraded":"2024-02-19T14:54:37.840039+0000","last_fullsized":"2024-02-19T14:54:37.840039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:19.115262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826474+0000","last_change":"2024-02-19T14:54:26.743065+0000","last_active":"2024-02-19T14:54:37.826474+0000","last_peered":"2024-02-19T14:54:37.826474+0000","last_clean":"2024-02-19T14:54:37.826474+0000","last_became_active":"2024-02-19T14:54:26.742623+0000","last_became_peered":"2024-02-19T14:54:26.742623+0000","last_unstale":"2024-02-19T14:54:37.826474+0000","last_undegraded":"2024-02-19T14:54:37.826474+0000","last_fullsized":"2024-02-19T14:54:37.826474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:29.434729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836805+0000","last_change":"2024-02-19T14:54:37.830200+0000","last_active":"2024-02-19T14:54:37.836805+0000","last_peered":"2024-02-19T14:54:37.836805+0000","last_clean":"2024-02-19T14:54:37.836805+0000","last_became_active":"2024-02-19T14:54:37.829933+0000","last_became_peered":"2024-02-19T14:54:37.829933+0000","last_unstale":"2024-02-19T14:54:37.836805+0000","last_undegraded":"2024-02-19T14:54:37.836805+0000","last_fullsized":"2024-02-19T14:54:37.836805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:56.496103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.829041+0000","last_change":"2024-02-19T14:54:26.752403+0000","last_active":"2024-02-19T14:54:37.829041+0000","last_peered":"2024-02-19T14:54:37.829041+0000","last_clean":"2024-02-19T14:54:37.829041+0000","last_became_active":"2024-02-19T14:54:26.752116+0000","last_became_peered":"2024-02-19T14:54:26.752116+0000","last_unstale":"2024-02-19T14:54:37.829041+0000","last_undegraded":"2024-02-19T14:54:37.829041+0000","last_fullsized":"2024-02-19T14:54:37.829041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:34:33.181295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838823+0000","last_change":"2024-02-19T14:54:37.833103+0000","last_active":"2024-02-19T14:54:37.838823+0000","last_peered":"2024-02-19T14:54:37.838823+0000","last_clean":"2024-02-19T14:54:37.838823+0000","last_became_active":"2024-02-19T14:54:37.832498+0000","last_became_peered":"2024-02-19T14:54:37.832498+0000","last_unstale":"2024-02-19T14:54:37.838823+0000","last_undegraded":"2024-02-19T14:54:37.838823+0000","last_fullsized":"2024-02-19T14:54:37.838823+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:01.742780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820829+0000","last_change":"2024-02-19T14:54:26.752358+0000","last_active":"2024-02-19T14:54:37.820829+0000","last_peered":"2024-02-19T14:54:37.820829+0000","last_clean":"2024-02-19T14:54:37.820829+0000","last_became_active":"2024-02-19T14:54:26.752076+0000","last_became_peered":"2024-02-19T14:54:26.752076+0000","last_unstale":"2024-02-19T14:54:37.820829+0000","last_undegraded":"2024-02-19T14:54:37.820829+0000","last_fullsized":"2024-02-19T14:54:37.820829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:40.346858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.837267+0000","last_change":"2024-02-19T14:54:37.830152+0000","last_active":"2024-02-19T14:54:37.837267+0000","last_peered":"2024-02-19T14:54:37.837267+0000","last_clean":"2024-02-19T14:54:37.837267+0000","last_became_active":"2024-02-19T14:54:37.829865+0000","last_became_peered":"2024-02-19T14:54:37.829865+0000","last_unstale":"2024-02-19T14:54:37.837267+0000","last_undegraded":"2024-02-19T14:54:37.837267+0000","last_fullsized":"2024-02-19T14:54:37.837267+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:16.681755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821887+0000","last_change":"2024-02-19T14:54:26.743117+0000","last_active":"2024-02-19T14:54:37.821887+0000","last_peered":"2024-02-19T14:54:37.821887+0000","last_clean":"2024-02-19T14:54:37.821887+0000","last_became_active":"2024-02-19T14:54:26.742845+0000","last_became_peered":"2024-02-19T14:54:26.742845+0000","last_unstale":"2024-02-19T14:54:37.821887+0000","last_undegraded":"2024-02-19T14:54:37.821887+0000","last_fullsized":"2024-02-19T14:54:37.821887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843627+0000","last_change":"2024-02-19T14:54:37.835305+0000","last_active":"2024-02-19T14:54:37.843627+0000","last_peered":"2024-02-19T14:54:37.843627+0000","last_clean":"2024-02-19T14:54:37.843627+0000","last_became_active":"2024-02-19T14:54:37.834979+0000","last_became_peered":"2024-02-19T14:54:37.834979+0000","last_unstale":"2024-02-19T14:54:37.843627+0000","last_undegraded":"2024-02-19T14:54:37.843627+0000","last_fullsized":"2024-02-19T14:54:37.843627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:19.530070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814983+0000","last_change":"2024-02-19T14:54:26.746821+0000","last_active":"2024-02-19T14:54:37.814983+0000","last_peered":"2024-02-19T14:54:37.814983+0000","last_clean":"2024-02-19T14:54:37.814983+0000","last_became_active":"2024-02-19T14:54:26.746408+0000","last_became_peered":"2024-02-19T14:54:26.746408+0000","last_unstale":"2024-02-19T14:54:37.814983+0000","last_undegraded":"2024-02-19T14:54:37.814983+0000","last_fullsized":"2024-02-19T14:54:37.814983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:38.456834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835347+0000","last_change":"2024-02-19T14:54:37.829118+0000","last_active":"2024-02-19T14:54:37.835347+0000","last_peered":"2024-02-19T14:54:37.835347+0000","last_clean":"2024-02-19T14:54:37.835347+0000","last_became_active":"2024-02-19T14:54:37.827474+0000","last_became_peered":"2024-02-19T14:54:37.827474+0000","last_unstale":"2024-02-19T14:54:37.835347+0000","last_undegraded":"2024-02-19T14:54:37.835347+0000","last_fullsized":"2024-02-19T14:54:37.835347+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:18:51.236995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827682+0000","last_change":"2024-02-19T14:54:26.745851+0000","last_active":"2024-02-19T14:54:37.827682+0000","last_peered":"2024-02-19T14:54:37.827682+0000","last_clean":"2024-02-19T14:54:37.827682+0000","last_became_active":"2024-02-19T14:54:26.745587+0000","last_became_peered":"2024-02-19T14:54:26.745587+0000","last_unstale":"2024-02-19T14:54:37.827682+0000","last_undegraded":"2024-02-19T14:54:37.827682+0000","last_fullsized":"2024-02-19T14:54:37.827682+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845645+0000","last_change":"2024-02-19T14:54:37.837776+0000","last_active":"2024-02-19T14:54:37.845645+0000","last_peered":"2024-02-19T14:54:37.845645+0000","last_clean":"2024-02-19T14:54:37.845645+0000","last_became_active":"2024-02-19T14:54:37.837082+0000","last_became_peered":"2024-02-19T14:54:37.837082+0000","last_unstale":"2024-02-19T14:54:37.845645+0000","last_undegraded":"2024-02-19T14:54:37.845645+0000","last_fullsized":"2024-02-19T14:54:37.845645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:25.979876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821464+0000","last_change":"2024-02-19T14:54:26.758958+0000","last_active":"2024-02-19T14:54:37.821464+0000","last_peered":"2024-02-19T14:54:37.821464+0000","last_clean":"2024-02-19T14:54:37.821464+0000","last_became_active":"2024-02-19T14:54:26.758649+0000","last_became_peered":"2024-02-19T14:54:26.758649+0000","last_unstale":"2024-02-19T14:54:37.821464+0000","last_undegraded":"2024-02-19T14:54:37.821464+0000","last_fullsized":"2024-02-19T14:54:37.821464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:38.553508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835089+0000","last_change":"2024-02-19T14:54:37.833083+0000","last_active":"2024-02-19T14:54:37.835089+0000","last_peered":"2024-02-19T14:54:37.835089+0000","last_clean":"2024-02-19T14:54:37.835089+0000","last_became_active":"2024-02-19T14:54:37.832783+0000","last_became_peered":"2024-02-19T14:54:37.832783+0000","last_unstale":"2024-02-19T14:54:37.835089+0000","last_undegraded":"2024-02-19T14:54:37.835089+0000","last_fullsized":"2024-02-19T14:54:37.835089+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:25.266694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820135+0000","last_change":"2024-02-19T14:54:26.742392+0000","last_active":"2024-02-19T14:54:37.820135+0000","last_peered":"2024-02-19T14:54:37.820135+0000","last_clean":"2024-02-19T14:54:37.820135+0000","last_became_active":"2024-02-19T14:54:26.742129+0000","last_became_peered":"2024-02-19T14:54:26.742129+0000","last_unstale":"2024-02-19T14:54:37.820135+0000","last_undegraded":"2024-02-19T14:54:37.820135+0000","last_fullsized":"2024-02-19T14:54:37.820135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:50.140868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843726+0000","last_change":"2024-02-19T14:54:37.836933+0000","last_active":"2024-02-19T14:54:37.843726+0000","last_peered":"2024-02-19T14:54:37.843726+0000","last_clean":"2024-02-19T14:54:37.843726+0000","last_became_active":"2024-02-19T14:54:37.835463+0000","last_became_peered":"2024-02-19T14:54:37.835463+0000","last_unstale":"2024-02-19T14:54:37.843726+0000","last_undegraded":"2024-02-19T14:54:37.843726+0000","last_fullsized":"2024-02-19T14:54:37.843726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:01.742780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821307+0000","last_change":"2024-02-19T14:54:26.747016+0000","last_active":"2024-02-19T14:54:37.821307+0000","last_peered":"2024-02-19T14:54:37.821307+0000","last_clean":"2024-02-19T14:54:37.821307+0000","last_became_active":"2024-02-19T14:54:26.746761+0000","last_became_peered":"2024-02-19T14:54:26.746761+0000","last_unstale":"2024-02-19T14:54:37.821307+0000","last_undegraded":"2024-02-19T14:54:37.821307+0000","last_fullsized":"2024-02-19T14:54:37.821307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.847341+0000","last_change":"2024-02-19T14:54:37.846531+0000","last_active":"2024-02-19T14:54:37.847341+0000","last_peered":"2024-02-19T14:54:37.847341+0000","last_clean":"2024-02-19T14:54:37.847341+0000","last_became_active":"2024-02-19T14:54:37.846257+0000","last_became_peered":"2024-02-19T14:54:37.846257+0000","last_unstale":"2024-02-19T14:54:37.847341+0000","last_undegraded":"2024-02-19T14:54:37.847341+0000","last_fullsized":"2024-02-19T14:54:37.847341+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:20.081130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819416+0000","last_change":"2024-02-19T14:54:36.826134+0000","last_active":"2024-02-19T14:54:37.819416+0000","last_peered":"2024-02-19T14:54:37.819416+0000","last_clean":"2024-02-19T14:54:37.819416+0000","last_became_active":"2024-02-19T14:54:36.825871+0000","last_became_peered":"2024-02-19T14:54:36.825871+0000","last_unstale":"2024-02-19T14:54:37.819416+0000","last_undegraded":"2024-02-19T14:54:37.819416+0000","last_fullsized":"2024-02-19T14:54:37.819416+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:15.677220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819026+0000","last_change":"2024-02-19T14:54:26.736520+0000","last_active":"2024-02-19T14:54:37.819026+0000","last_peered":"2024-02-19T14:54:37.819026+0000","last_clean":"2024-02-19T14:54:37.819026+0000","last_became_active":"2024-02-19T14:54:26.736235+0000","last_became_peered":"2024-02-19T14:54:26.736235+0000","last_unstale":"2024-02-19T14:54:37.819026+0000","last_undegraded":"2024-02-19T14:54:37.819026+0000","last_fullsized":"2024-02-19T14:54:37.819026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:16.908530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"74'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815144+0000","last_change":"2024-02-19T14:54:25.736715+0000","last_active":"2024-02-19T14:54:37.815144+0000","last_peered":"2024-02-19T14:54:37.815144+0000","last_clean":"2024-02-19T14:54:37.815144+0000","last_became_active":"2024-02-19T14:54:25.736266+0000","last_became_peered":"2024-02-19T14:54:25.736266+0000","last_unstale":"2024-02-19T14:54:37.815144+0000","last_undegraded":"2024-02-19T14:54:37.815144+0000","last_fullsized":"2024-02-19T14:54:37.815144+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:29.148205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840407+0000","last_change":"2024-02-19T14:54:37.834149+0000","last_active":"2024-02-19T14:54:37.840407+0000","last_peered":"2024-02-19T14:54:37.840407+0000","last_clean":"2024-02-19T14:54:37.840407+0000","last_became_active":"2024-02-19T14:54:37.833917+0000","last_became_peered":"2024-02-19T14:54:37.833917+0000","last_unstale":"2024-02-19T14:54:37.840407+0000","last_undegraded":"2024-02-19T14:54:37.840407+0000","last_fullsized":"2024-02-19T14:54:37.840407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:39.422646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821303+0000","last_change":"2024-02-19T14:54:36.833581+0000","last_active":"2024-02-19T14:54:37.821303+0000","last_peered":"2024-02-19T14:54:37.821303+0000","last_clean":"2024-02-19T14:54:37.821303+0000","last_became_active":"2024-02-19T14:54:36.833276+0000","last_became_peered":"2024-02-19T14:54:36.833276+0000","last_unstale":"2024-02-19T14:54:37.821303+0000","last_undegraded":"2024-02-19T14:54:37.821303+0000","last_fullsized":"2024-02-19T14:54:37.821303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:58.062735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827040+0000","last_change":"2024-02-19T14:54:26.745742+0000","last_active":"2024-02-19T14:54:37.827040+0000","last_peered":"2024-02-19T14:54:37.827040+0000","last_clean":"2024-02-19T14:54:37.827040+0000","last_became_active":"2024-02-19T14:54:26.745478+0000","last_became_peered":"2024-02-19T14:54:26.745478+0000","last_unstale":"2024-02-19T14:54:37.827040+0000","last_undegraded":"2024-02-19T14:54:37.827040+0000","last_fullsized":"2024-02-19T14:54:37.827040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822319+0000","last_change":"2024-02-19T14:54:25.737577+0000","last_active":"2024-02-19T14:54:37.822319+0000","last_peered":"2024-02-19T14:54:37.822319+0000","last_clean":"2024-02-19T14:54:37.822319+0000","last_became_active":"2024-02-19T14:54:25.737301+0000","last_became_peered":"2024-02-19T14:54:25.737301+0000","last_unstale":"2024-02-19T14:54:37.822319+0000","last_undegraded":"2024-02-19T14:54:37.822319+0000","last_fullsized":"2024-02-19T14:54:37.822319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:47.328330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839625+0000","last_change":"2024-02-19T14:54:37.833409+0000","last_active":"2024-02-19T14:54:37.839625+0000","last_peered":"2024-02-19T14:54:37.839625+0000","last_clean":"2024-02-19T14:54:37.839625+0000","last_became_active":"2024-02-19T14:54:37.833141+0000","last_became_peered":"2024-02-19T14:54:37.833141+0000","last_unstale":"2024-02-19T14:54:37.839625+0000","last_undegraded":"2024-02-19T14:54:37.839625+0000","last_fullsized":"2024-02-19T14:54:37.839625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:48.384374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840195+0000","last_change":"2024-02-19T14:54:36.825186+0000","last_active":"2024-02-19T14:54:37.840195+0000","last_peered":"2024-02-19T14:54:37.840195+0000","last_clean":"2024-02-19T14:54:37.840195+0000","last_became_active":"2024-02-19T14:54:36.824892+0000","last_became_peered":"2024-02-19T14:54:36.824892+0000","last_unstale":"2024-02-19T14:54:37.840195+0000","last_undegraded":"2024-02-19T14:54:37.840195+0000","last_fullsized":"2024-02-19T14:54:37.840195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:41.852675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825860+0000","last_change":"2024-02-19T14:54:26.744292+0000","last_active":"2024-02-19T14:54:37.825860+0000","last_peered":"2024-02-19T14:54:37.825860+0000","last_clean":"2024-02-19T14:54:37.825860+0000","last_became_active":"2024-02-19T14:54:26.743866+0000","last_became_peered":"2024-02-19T14:54:26.743866+0000","last_unstale":"2024-02-19T14:54:37.825860+0000","last_undegraded":"2024-02-19T14:54:37.825860+0000","last_fullsized":"2024-02-19T14:54:37.825860+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:21:10.873300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821993+0000","last_change":"2024-02-19T14:54:25.739812+0000","last_active":"2024-02-19T14:54:37.821993+0000","last_peered":"2024-02-19T14:54:37.821993+0000","last_clean":"2024-02-19T14:54:37.821993+0000","last_became_active":"2024-02-19T14:54:25.739538+0000","last_became_peered":"2024-02-19T14:54:25.739538+0000","last_unstale":"2024-02-19T14:54:37.821993+0000","last_undegraded":"2024-02-19T14:54:37.821993+0000","last_fullsized":"2024-02-19T14:54:37.821993+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:02.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843641+0000","last_change":"2024-02-19T14:54:37.838138+0000","last_active":"2024-02-19T14:54:37.843641+0000","last_peered":"2024-02-19T14:54:37.843641+0000","last_clean":"2024-02-19T14:54:37.843641+0000","last_became_active":"2024-02-19T14:54:37.837860+0000","last_became_peered":"2024-02-19T14:54:37.837860+0000","last_unstale":"2024-02-19T14:54:37.843641+0000","last_undegraded":"2024-02-19T14:54:37.843641+0000","last_fullsized":"2024-02-19T14:54:37.843641+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:14.069764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840466+0000","last_change":"2024-02-19T14:54:36.826883+0000","last_active":"2024-02-19T14:54:37.840466+0000","last_peered":"2024-02-19T14:54:37.840466+0000","last_clean":"2024-02-19T14:54:37.840466+0000","last_became_active":"2024-02-19T14:54:36.826635+0000","last_became_peered":"2024-02-19T14:54:36.826635+0000","last_unstale":"2024-02-19T14:54:37.840466+0000","last_undegraded":"2024-02-19T14:54:37.840466+0000","last_fullsized":"2024-02-19T14:54:37.840466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:42.240006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827253+0000","last_change":"2024-02-19T14:54:26.738473+0000","last_active":"2024-02-19T14:54:37.827253+0000","last_peered":"2024-02-19T14:54:37.827253+0000","last_clean":"2024-02-19T14:54:37.827253+0000","last_became_active":"2024-02-19T14:54:26.738077+0000","last_became_peered":"2024-02-19T14:54:26.738077+0000","last_unstale":"2024-02-19T14:54:37.827253+0000","last_undegraded":"2024-02-19T14:54:37.827253+0000","last_fullsized":"2024-02-19T14:54:37.827253+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:26:27.849069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819592+0000","last_change":"2024-02-19T14:54:25.733935+0000","last_active":"2024-02-19T14:54:37.819592+0000","last_peered":"2024-02-19T14:54:37.819592+0000","last_clean":"2024-02-19T14:54:37.819592+0000","last_became_active":"2024-02-19T14:54:25.733653+0000","last_became_peered":"2024-02-19T14:54:25.733653+0000","last_unstale":"2024-02-19T14:54:37.819592+0000","last_undegraded":"2024-02-19T14:54:37.819592+0000","last_fullsized":"2024-02-19T14:54:37.819592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:52.238124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836108+0000","last_change":"2024-02-19T14:54:37.833743+0000","last_active":"2024-02-19T14:54:37.836108+0000","last_peered":"2024-02-19T14:54:37.836108+0000","last_clean":"2024-02-19T14:54:37.836108+0000","last_became_active":"2024-02-19T14:54:37.833437+0000","last_became_peered":"2024-02-19T14:54:37.833437+0000","last_unstale":"2024-02-19T14:54:37.836108+0000","last_undegraded":"2024-02-19T14:54:37.836108+0000","last_fullsized":"2024-02-19T14:54:37.836108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:25.979876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838077+0000","last_change":"2024-02-19T14:54:36.828034+0000","last_active":"2024-02-19T14:54:37.838077+0000","last_peered":"2024-02-19T14:54:37.838077+0000","last_clean":"2024-02-19T14:54:37.838077+0000","last_became_active":"2024-02-19T14:54:36.827707+0000","last_became_peered":"2024-02-19T14:54:36.827707+0000","last_unstale":"2024-02-19T14:54:37.838077+0000","last_undegraded":"2024-02-19T14:54:37.838077+0000","last_fullsized":"2024-02-19T14:54:37.838077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:36:41.952775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.828858+0000","last_change":"2024-02-19T14:54:26.754225+0000","last_active":"2024-02-19T14:54:37.828858+0000","last_peered":"2024-02-19T14:54:37.828858+0000","last_clean":"2024-02-19T14:54:37.828858+0000","last_became_active":"2024-02-19T14:54:26.753892+0000","last_became_peered":"2024-02-19T14:54:26.753892+0000","last_unstale":"2024-02-19T14:54:37.828858+0000","last_undegraded":"2024-02-19T14:54:37.828858+0000","last_fullsized":"2024-02-19T14:54:37.828858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:02.979120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827443+0000","last_change":"2024-02-19T14:54:25.733786+0000","last_active":"2024-02-19T14:54:37.827443+0000","last_peered":"2024-02-19T14:54:37.827443+0000","last_clean":"2024-02-19T14:54:37.827443+0000","last_became_active":"2024-02-19T14:54:25.733327+0000","last_became_peered":"2024-02-19T14:54:25.733327+0000","last_unstale":"2024-02-19T14:54:37.827443+0000","last_undegraded":"2024-02-19T14:54:37.827443+0000","last_fullsized":"2024-02-19T14:54:37.827443+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:13.186581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844546+0000","last_change":"2024-02-19T14:54:37.834983+0000","last_active":"2024-02-19T14:54:37.844546+0000","last_peered":"2024-02-19T14:54:37.844546+0000","last_clean":"2024-02-19T14:54:37.844546+0000","last_became_active":"2024-02-19T14:54:37.833764+0000","last_became_peered":"2024-02-19T14:54:37.833764+0000","last_unstale":"2024-02-19T14:54:37.844546+0000","last_undegraded":"2024-02-19T14:54:37.844546+0000","last_fullsized":"2024-02-19T14:54:37.844546+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:59:43.474020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842738+0000","last_change":"2024-02-19T14:54:36.826847+0000","last_active":"2024-02-19T14:54:37.842738+0000","last_peered":"2024-02-19T14:54:37.842738+0000","last_clean":"2024-02-19T14:54:37.842738+0000","last_became_active":"2024-02-19T14:54:36.826575+0000","last_became_peered":"2024-02-19T14:54:36.826575+0000","last_unstale":"2024-02-19T14:54:37.842738+0000","last_undegraded":"2024-02-19T14:54:37.842738+0000","last_fullsized":"2024-02-19T14:54:37.842738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:14.908793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.829978+0000","last_change":"2024-02-19T14:54:26.753347+0000","last_active":"2024-02-19T14:54:37.829978+0000","last_peered":"2024-02-19T14:54:37.829978+0000","last_clean":"2024-02-19T14:54:37.829978+0000","last_became_active":"2024-02-19T14:54:26.752855+0000","last_became_peered":"2024-02-19T14:54:26.752855+0000","last_unstale":"2024-02-19T14:54:37.829978+0000","last_undegraded":"2024-02-19T14:54:37.829978+0000","last_fullsized":"2024-02-19T14:54:37.829978+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:48.880118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823925+0000","last_change":"2024-02-19T14:54:25.730846+0000","last_active":"2024-02-19T14:54:37.823925+0000","last_peered":"2024-02-19T14:54:37.823925+0000","last_clean":"2024-02-19T14:54:37.823925+0000","last_became_active":"2024-02-19T14:54:25.730539+0000","last_became_peered":"2024-02-19T14:54:25.730539+0000","last_unstale":"2024-02-19T14:54:37.823925+0000","last_undegraded":"2024-02-19T14:54:37.823925+0000","last_fullsized":"2024-02-19T14:54:37.823925+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:56.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846501+0000","last_change":"2024-02-19T14:54:37.845595+0000","last_active":"2024-02-19T14:54:37.846501+0000","last_peered":"2024-02-19T14:54:37.846501+0000","last_clean":"2024-02-19T14:54:37.846501+0000","last_became_active":"2024-02-19T14:54:37.844986+0000","last_became_peered":"2024-02-19T14:54:37.844986+0000","last_unstale":"2024-02-19T14:54:37.846501+0000","last_undegraded":"2024-02-19T14:54:37.846501+0000","last_fullsized":"2024-02-19T14:54:37.846501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:22.391301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"78'4","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.847633+0000","last_change":"2024-02-19T14:54:36.835389+0000","last_active":"2024-02-19T14:54:37.847633+0000","last_peered":"2024-02-19T14:54:37.847633+0000","last_clean":"2024-02-19T14:54:37.847633+0000","last_became_active":"2024-02-19T14:54:36.835045+0000","last_became_peered":"2024-02-19T14:54:36.835045+0000","last_unstale":"2024-02-19T14:54:37.847633+0000","last_undegraded":"2024-02-19T14:54:37.847633+0000","last_fullsized":"2024-02-19T14:54:37.847633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:23.576155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826951+0000","last_change":"2024-02-19T14:54:26.741073+0000","last_active":"2024-02-19T14:54:37.826951+0000","last_peered":"2024-02-19T14:54:37.826951+0000","last_clean":"2024-02-19T14:54:37.826951+0000","last_became_active":"2024-02-19T14:54:26.740765+0000","last_became_peered":"2024-02-19T14:54:26.740765+0000","last_unstale":"2024-02-19T14:54:37.826951+0000","last_undegraded":"2024-02-19T14:54:37.826951+0000","last_fullsized":"2024-02-19T14:54:37.826951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:57.622666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827147+0000","last_change":"2024-02-19T14:54:25.737703+0000","last_active":"2024-02-19T14:54:37.827147+0000","last_peered":"2024-02-19T14:54:37.827147+0000","last_clean":"2024-02-19T14:54:37.827147+0000","last_became_active":"2024-02-19T14:54:25.737444+0000","last_became_peered":"2024-02-19T14:54:25.737444+0000","last_unstale":"2024-02-19T14:54:37.827147+0000","last_undegraded":"2024-02-19T14:54:37.827147+0000","last_fullsized":"2024-02-19T14:54:37.827147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:26.488134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845117+0000","last_change":"2024-02-19T14:54:37.837242+0000","last_active":"2024-02-19T14:54:37.845117+0000","last_peered":"2024-02-19T14:54:37.845117+0000","last_clean":"2024-02-19T14:54:37.845117+0000","last_became_active":"2024-02-19T14:54:37.836969+0000","last_became_peered":"2024-02-19T14:54:37.836969+0000","last_unstale":"2024-02-19T14:54:37.845117+0000","last_undegraded":"2024-02-19T14:54:37.845117+0000","last_fullsized":"2024-02-19T14:54:37.845117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:25.266694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839556+0000","last_change":"2024-02-19T14:54:36.827013+0000","last_active":"2024-02-19T14:54:37.839556+0000","last_peered":"2024-02-19T14:54:37.839556+0000","last_clean":"2024-02-19T14:54:37.839556+0000","last_became_active":"2024-02-19T14:54:36.825537+0000","last_became_peered":"2024-02-19T14:54:36.825537+0000","last_unstale":"2024-02-19T14:54:37.839556+0000","last_undegraded":"2024-02-19T14:54:37.839556+0000","last_fullsized":"2024-02-19T14:54:37.839556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:36:07.513713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820081+0000","last_change":"2024-02-19T14:54:26.751885+0000","last_active":"2024-02-19T14:54:37.820081+0000","last_peered":"2024-02-19T14:54:37.820081+0000","last_clean":"2024-02-19T14:54:37.820081+0000","last_became_active":"2024-02-19T14:54:26.751496+0000","last_became_peered":"2024-02-19T14:54:26.751496+0000","last_unstale":"2024-02-19T14:54:37.820081+0000","last_undegraded":"2024-02-19T14:54:37.820081+0000","last_fullsized":"2024-02-19T14:54:37.820081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:57.427605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814741+0000","last_change":"2024-02-19T14:54:25.736306+0000","last_active":"2024-02-19T14:54:37.814741+0000","last_peered":"2024-02-19T14:54:37.814741+0000","last_clean":"2024-02-19T14:54:37.814741+0000","last_became_active":"2024-02-19T14:54:25.736029+0000","last_became_peered":"2024-02-19T14:54:25.736029+0000","last_unstale":"2024-02-19T14:54:37.814741+0000","last_undegraded":"2024-02-19T14:54:37.814741+0000","last_fullsized":"2024-02-19T14:54:37.814741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:59:00.456190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836255+0000","last_change":"2024-02-19T14:54:37.829194+0000","last_active":"2024-02-19T14:54:37.836255+0000","last_peered":"2024-02-19T14:54:37.836255+0000","last_clean":"2024-02-19T14:54:37.836255+0000","last_became_active":"2024-02-19T14:54:37.828948+0000","last_became_peered":"2024-02-19T14:54:37.828948+0000","last_unstale":"2024-02-19T14:54:37.836255+0000","last_undegraded":"2024-02-19T14:54:37.836255+0000","last_fullsized":"2024-02-19T14:54:37.836255+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:57.782852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820033+0000","last_change":"2024-02-19T14:54:36.824354+0000","last_active":"2024-02-19T14:54:37.820033+0000","last_peered":"2024-02-19T14:54:37.820033+0000","last_clean":"2024-02-19T14:54:37.820033+0000","last_became_active":"2024-02-19T14:54:36.824099+0000","last_became_peered":"2024-02-19T14:54:36.824099+0000","last_unstale":"2024-02-19T14:54:37.820033+0000","last_undegraded":"2024-02-19T14:54:37.820033+0000","last_fullsized":"2024-02-19T14:54:37.820033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:37.898731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826564+0000","last_change":"2024-02-19T14:54:26.743807+0000","last_active":"2024-02-19T14:54:37.826564+0000","last_peered":"2024-02-19T14:54:37.826564+0000","last_clean":"2024-02-19T14:54:37.826564+0000","last_became_active":"2024-02-19T14:54:26.743493+0000","last_became_peered":"2024-02-19T14:54:26.743493+0000","last_unstale":"2024-02-19T14:54:37.826564+0000","last_undegraded":"2024-02-19T14:54:37.826564+0000","last_fullsized":"2024-02-19T14:54:37.826564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:38.383508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823296+0000","last_change":"2024-02-19T14:54:25.737199+0000","last_active":"2024-02-19T14:54:37.823296+0000","last_peered":"2024-02-19T14:54:37.823296+0000","last_clean":"2024-02-19T14:54:37.823296+0000","last_became_active":"2024-02-19T14:54:25.736941+0000","last_became_peered":"2024-02-19T14:54:25.736941+0000","last_unstale":"2024-02-19T14:54:37.823296+0000","last_undegraded":"2024-02-19T14:54:37.823296+0000","last_fullsized":"2024-02-19T14:54:37.823296+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:26.806558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841043+0000","last_change":"2024-02-19T14:54:37.836750+0000","last_active":"2024-02-19T14:54:37.841043+0000","last_peered":"2024-02-19T14:54:37.841043+0000","last_clean":"2024-02-19T14:54:37.841043+0000","last_became_active":"2024-02-19T14:54:37.836439+0000","last_became_peered":"2024-02-19T14:54:37.836439+0000","last_unstale":"2024-02-19T14:54:37.841043+0000","last_undegraded":"2024-02-19T14:54:37.841043+0000","last_fullsized":"2024-02-19T14:54:37.841043+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:59.645392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"78'8","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845053+0000","last_change":"2024-02-19T14:54:36.825524+0000","last_active":"2024-02-19T14:54:37.845053+0000","last_peered":"2024-02-19T14:54:37.845053+0000","last_clean":"2024-02-19T14:54:37.845053+0000","last_became_active":"2024-02-19T14:54:36.825254+0000","last_became_peered":"2024-02-19T14:54:36.825254+0000","last_unstale":"2024-02-19T14:54:37.845053+0000","last_undegraded":"2024-02-19T14:54:37.845053+0000","last_fullsized":"2024-02-19T14:54:37.845053+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:18.110727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820337+0000","last_change":"2024-02-19T14:54:26.733706+0000","last_active":"2024-02-19T14:54:37.820337+0000","last_peered":"2024-02-19T14:54:37.820337+0000","last_clean":"2024-02-19T14:54:37.820337+0000","last_became_active":"2024-02-19T14:54:26.733434+0000","last_became_peered":"2024-02-19T14:54:26.733434+0000","last_unstale":"2024-02-19T14:54:37.820337+0000","last_undegraded":"2024-02-19T14:54:37.820337+0000","last_fullsized":"2024-02-19T14:54:37.820337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:32.153897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820923+0000","last_change":"2024-02-19T14:54:25.729258+0000","last_active":"2024-02-19T14:54:37.820923+0000","last_peered":"2024-02-19T14:54:37.820923+0000","last_clean":"2024-02-19T14:54:37.820923+0000","last_became_active":"2024-02-19T14:54:25.728966+0000","last_became_peered":"2024-02-19T14:54:25.728966+0000","last_unstale":"2024-02-19T14:54:37.820923+0000","last_undegraded":"2024-02-19T14:54:37.820923+0000","last_fullsized":"2024-02-19T14:54:37.820923+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:41.651494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840605+0000","last_change":"2024-02-19T14:54:37.835912+0000","last_active":"2024-02-19T14:54:37.840605+0000","last_peered":"2024-02-19T14:54:37.840605+0000","last_clean":"2024-02-19T14:54:37.840605+0000","last_became_active":"2024-02-19T14:54:37.835316+0000","last_became_peered":"2024-02-19T14:54:37.835316+0000","last_unstale":"2024-02-19T14:54:37.840605+0000","last_undegraded":"2024-02-19T14:54:37.840605+0000","last_fullsized":"2024-02-19T14:54:37.840605+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:42.659206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846048+0000","last_change":"2024-02-19T14:54:36.834272+0000","last_active":"2024-02-19T14:54:37.846048+0000","last_peered":"2024-02-19T14:54:37.846048+0000","last_clean":"2024-02-19T14:54:37.846048+0000","last_became_active":"2024-02-19T14:54:36.833964+0000","last_became_peered":"2024-02-19T14:54:36.833964+0000","last_unstale":"2024-02-19T14:54:37.846048+0000","last_undegraded":"2024-02-19T14:54:37.846048+0000","last_fullsized":"2024-02-19T14:54:37.846048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:04.331873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820041+0000","last_change":"2024-02-19T14:54:26.741110+0000","last_active":"2024-02-19T14:54:37.820041+0000","last_peered":"2024-02-19T14:54:37.820041+0000","last_clean":"2024-02-19T14:54:37.820041+0000","last_became_active":"2024-02-19T14:54:26.740831+0000","last_became_peered":"2024-02-19T14:54:26.740831+0000","last_unstale":"2024-02-19T14:54:37.820041+0000","last_undegraded":"2024-02-19T14:54:37.820041+0000","last_fullsized":"2024-02-19T14:54:37.820041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"74'3","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815884+0000","last_change":"2024-02-19T14:54:25.732654+0000","last_active":"2024-02-19T14:54:37.815884+0000","last_peered":"2024-02-19T14:54:37.815884+0000","last_clean":"2024-02-19T14:54:37.815884+0000","last_became_active":"2024-02-19T14:54:25.732388+0000","last_became_peered":"2024-02-19T14:54:25.732388+0000","last_unstale":"2024-02-19T14:54:37.815884+0000","last_undegraded":"2024-02-19T14:54:37.815884+0000","last_fullsized":"2024-02-19T14:54:37.815884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:04.044350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839503+0000","last_change":"2024-02-19T14:54:37.831656+0000","last_active":"2024-02-19T14:54:37.839503+0000","last_peered":"2024-02-19T14:54:37.839503+0000","last_clean":"2024-02-19T14:54:37.839503+0000","last_became_active":"2024-02-19T14:54:37.831331+0000","last_became_peered":"2024-02-19T14:54:37.831331+0000","last_unstale":"2024-02-19T14:54:37.839503+0000","last_undegraded":"2024-02-19T14:54:37.839503+0000","last_fullsized":"2024-02-19T14:54:37.839503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:53.748202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.828306+0000","last_change":"2024-02-19T14:54:36.833090+0000","last_active":"2024-02-19T14:54:37.828306+0000","last_peered":"2024-02-19T14:54:37.828306+0000","last_clean":"2024-02-19T14:54:37.828306+0000","last_became_active":"2024-02-19T14:54:36.832830+0000","last_became_peered":"2024-02-19T14:54:36.832830+0000","last_unstale":"2024-02-19T14:54:37.828306+0000","last_undegraded":"2024-02-19T14:54:37.828306+0000","last_fullsized":"2024-02-19T14:54:37.828306+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:04.331873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821189+0000","last_change":"2024-02-19T14:54:26.749111+0000","last_active":"2024-02-19T14:54:37.821189+0000","last_peered":"2024-02-19T14:54:37.821189+0000","last_clean":"2024-02-19T14:54:37.821189+0000","last_became_active":"2024-02-19T14:54:26.748830+0000","last_became_peered":"2024-02-19T14:54:26.748830+0000","last_unstale":"2024-02-19T14:54:37.821189+0000","last_undegraded":"2024-02-19T14:54:37.821189+0000","last_fullsized":"2024-02-19T14:54:37.821189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:36.425397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"74'6","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819500+0000","last_change":"2024-02-19T14:54:25.733916+0000","last_active":"2024-02-19T14:54:37.819500+0000","last_peered":"2024-02-19T14:54:37.819500+0000","last_clean":"2024-02-19T14:54:37.819500+0000","last_became_active":"2024-02-19T14:54:25.733638+0000","last_became_peered":"2024-02-19T14:54:25.733638+0000","last_unstale":"2024-02-19T14:54:37.819500+0000","last_undegraded":"2024-02-19T14:54:37.819500+0000","last_fullsized":"2024-02-19T14:54:37.819500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:26:26.843004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842514+0000","last_change":"2024-02-19T14:54:37.836488+0000","last_active":"2024-02-19T14:54:37.842514+0000","last_peered":"2024-02-19T14:54:37.842514+0000","last_clean":"2024-02-19T14:54:37.842514+0000","last_became_active":"2024-02-19T14:54:37.836045+0000","last_became_peered":"2024-02-19T14:54:37.836045+0000","last_unstale":"2024-02-19T14:54:37.842514+0000","last_undegraded":"2024-02-19T14:54:37.842514+0000","last_fullsized":"2024-02-19T14:54:37.842514+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:59.067270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"78'3","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845056+0000","last_change":"2024-02-19T14:54:36.833202+0000","last_active":"2024-02-19T14:54:37.845056+0000","last_peered":"2024-02-19T14:54:37.845056+0000","last_clean":"2024-02-19T14:54:37.845056+0000","last_became_active":"2024-02-19T14:54:36.832869+0000","last_became_peered":"2024-02-19T14:54:36.832869+0000","last_unstale":"2024-02-19T14:54:37.845056+0000","last_undegraded":"2024-02-19T14:54:37.845056+0000","last_fullsized":"2024-02-19T14:54:37.845056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:57.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822435+0000","last_change":"2024-02-19T14:54:26.741929+0000","last_active":"2024-02-19T14:54:37.822435+0000","last_peered":"2024-02-19T14:54:37.822435+0000","last_clean":"2024-02-19T14:54:37.822435+0000","last_became_active":"2024-02-19T14:54:26.741662+0000","last_became_peered":"2024-02-19T14:54:26.741662+0000","last_unstale":"2024-02-19T14:54:37.822435+0000","last_undegraded":"2024-02-19T14:54:37.822435+0000","last_fullsized":"2024-02-19T14:54:37.822435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821343+0000","last_change":"2024-02-19T14:54:25.731407+0000","last_active":"2024-02-19T14:54:37.821343+0000","last_peered":"2024-02-19T14:54:37.821343+0000","last_clean":"2024-02-19T14:54:37.821343+0000","last_became_active":"2024-02-19T14:54:25.731062+0000","last_became_peered":"2024-02-19T14:54:25.731062+0000","last_unstale":"2024-02-19T14:54:37.821343+0000","last_undegraded":"2024-02-19T14:54:37.821343+0000","last_fullsized":"2024-02-19T14:54:37.821343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:30.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840554+0000","last_change":"2024-02-19T14:54:37.836496+0000","last_active":"2024-02-19T14:54:37.840554+0000","last_peered":"2024-02-19T14:54:37.840554+0000","last_clean":"2024-02-19T14:54:37.840554+0000","last_became_active":"2024-02-19T14:54:37.836206+0000","last_became_peered":"2024-02-19T14:54:37.836206+0000","last_unstale":"2024-02-19T14:54:37.840554+0000","last_undegraded":"2024-02-19T14:54:37.840554+0000","last_fullsized":"2024-02-19T14:54:37.840554+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:47.604862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"78'6","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.849065+0000","last_change":"2024-02-19T14:54:36.833069+0000","last_active":"2024-02-19T14:54:37.849065+0000","last_peered":"2024-02-19T14:54:37.849065+0000","last_clean":"2024-02-19T14:54:37.849065+0000","last_became_active":"2024-02-19T14:54:36.832764+0000","last_became_peered":"2024-02-19T14:54:36.832764+0000","last_unstale":"2024-02-19T14:54:37.849065+0000","last_undegraded":"2024-02-19T14:54:37.849065+0000","last_fullsized":"2024-02-19T14:54:37.849065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:36:41.952775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819099+0000","last_change":"2024-02-19T14:54:26.739942+0000","last_active":"2024-02-19T14:54:37.819099+0000","last_peered":"2024-02-19T14:54:37.819099+0000","last_clean":"2024-02-19T14:54:37.819099+0000","last_became_active":"2024-02-19T14:54:26.739642+0000","last_became_peered":"2024-02-19T14:54:26.739642+0000","last_unstale":"2024-02-19T14:54:37.819099+0000","last_undegraded":"2024-02-19T14:54:37.819099+0000","last_fullsized":"2024-02-19T14:54:37.819099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:16.908530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821501+0000","last_change":"2024-02-19T14:54:25.731469+0000","last_active":"2024-02-19T14:54:37.821501+0000","last_peered":"2024-02-19T14:54:37.821501+0000","last_clean":"2024-02-19T14:54:37.821501+0000","last_became_active":"2024-02-19T14:54:25.731139+0000","last_became_peered":"2024-02-19T14:54:25.731139+0000","last_unstale":"2024-02-19T14:54:37.821501+0000","last_undegraded":"2024-02-19T14:54:37.821501+0000","last_fullsized":"2024-02-19T14:54:37.821501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:09.115941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843573+0000","last_change":"2024-02-19T14:54:37.833273+0000","last_active":"2024-02-19T14:54:37.843573+0000","last_peered":"2024-02-19T14:54:37.843573+0000","last_clean":"2024-02-19T14:54:37.843573+0000","last_became_active":"2024-02-19T14:54:37.832642+0000","last_became_peered":"2024-02-19T14:54:37.832642+0000","last_unstale":"2024-02-19T14:54:37.843573+0000","last_undegraded":"2024-02-19T14:54:37.843573+0000","last_fullsized":"2024-02-19T14:54:37.843573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:21.212649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821801+0000","last_change":"2024-02-19T14:54:36.829199+0000","last_active":"2024-02-19T14:54:37.821801+0000","last_peered":"2024-02-19T14:54:37.821801+0000","last_clean":"2024-02-19T14:54:37.821801+0000","last_became_active":"2024-02-19T14:54:36.828927+0000","last_became_peered":"2024-02-19T14:54:36.828927+0000","last_unstale":"2024-02-19T14:54:37.821801+0000","last_undegraded":"2024-02-19T14:54:37.821801+0000","last_fullsized":"2024-02-19T14:54:37.821801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:14:53.977545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819347+0000","last_change":"2024-02-19T14:54:26.746776+0000","last_active":"2024-02-19T14:54:37.819347+0000","last_peered":"2024-02-19T14:54:37.819347+0000","last_clean":"2024-02-19T14:54:37.819347+0000","last_became_active":"2024-02-19T14:54:26.746452+0000","last_became_peered":"2024-02-19T14:54:26.746452+0000","last_unstale":"2024-02-19T14:54:37.819347+0000","last_undegraded":"2024-02-19T14:54:37.819347+0000","last_fullsized":"2024-02-19T14:54:37.819347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:05.385960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814479+0000","last_change":"2024-02-19T14:54:25.736476+0000","last_active":"2024-02-19T14:54:37.814479+0000","last_peered":"2024-02-19T14:54:37.814479+0000","last_clean":"2024-02-19T14:54:37.814479+0000","last_became_active":"2024-02-19T14:54:25.736202+0000","last_became_peered":"2024-02-19T14:54:25.736202+0000","last_unstale":"2024-02-19T14:54:37.814479+0000","last_undegraded":"2024-02-19T14:54:37.814479+0000","last_fullsized":"2024-02-19T14:54:37.814479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:01.450961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836143+0000","last_change":"2024-02-19T14:54:37.834352+0000","last_active":"2024-02-19T14:54:37.836143+0000","last_peered":"2024-02-19T14:54:37.836143+0000","last_clean":"2024-02-19T14:54:37.836143+0000","last_became_active":"2024-02-19T14:54:37.833482+0000","last_became_peered":"2024-02-19T14:54:37.833482+0000","last_unstale":"2024-02-19T14:54:37.836143+0000","last_undegraded":"2024-02-19T14:54:37.836143+0000","last_fullsized":"2024-02-19T14:54:37.836143+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:56:31.705336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820419+0000","last_change":"2024-02-19T14:54:36.824408+0000","last_active":"2024-02-19T14:54:37.820419+0000","last_peered":"2024-02-19T14:54:37.820419+0000","last_clean":"2024-02-19T14:54:37.820419+0000","last_became_active":"2024-02-19T14:54:36.824166+0000","last_became_peered":"2024-02-19T14:54:36.824166+0000","last_unstale":"2024-02-19T14:54:37.820419+0000","last_undegraded":"2024-02-19T14:54:37.820419+0000","last_fullsized":"2024-02-19T14:54:37.820419+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:39.662911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820687+0000","last_change":"2024-02-19T14:54:26.742073+0000","last_active":"2024-02-19T14:54:37.820687+0000","last_peered":"2024-02-19T14:54:37.820687+0000","last_clean":"2024-02-19T14:54:37.820687+0000","last_became_active":"2024-02-19T14:54:26.741791+0000","last_became_peered":"2024-02-19T14:54:26.741791+0000","last_unstale":"2024-02-19T14:54:37.820687+0000","last_undegraded":"2024-02-19T14:54:37.820687+0000","last_fullsized":"2024-02-19T14:54:37.820687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:42.657558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"74'8","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826661+0000","last_change":"2024-02-19T14:54:25.737048+0000","last_active":"2024-02-19T14:54:37.826661+0000","last_peered":"2024-02-19T14:54:37.826661+0000","last_clean":"2024-02-19T14:54:37.826661+0000","last_became_active":"2024-02-19T14:54:25.736643+0000","last_became_peered":"2024-02-19T14:54:25.736643+0000","last_unstale":"2024-02-19T14:54:37.826661+0000","last_undegraded":"2024-02-19T14:54:37.826661+0000","last_fullsized":"2024-02-19T14:54:37.826661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:15.902465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835318+0000","last_change":"2024-02-19T14:54:37.830345+0000","last_active":"2024-02-19T14:54:37.835318+0000","last_peered":"2024-02-19T14:54:37.835318+0000","last_clean":"2024-02-19T14:54:37.835318+0000","last_became_active":"2024-02-19T14:54:37.830044+0000","last_became_peered":"2024-02-19T14:54:37.830044+0000","last_unstale":"2024-02-19T14:54:37.835318+0000","last_undegraded":"2024-02-19T14:54:37.835318+0000","last_fullsized":"2024-02-19T14:54:37.835318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.667447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820543+0000","last_change":"2024-02-19T14:54:36.835659+0000","last_active":"2024-02-19T14:54:37.820543+0000","last_peered":"2024-02-19T14:54:37.820543+0000","last_clean":"2024-02-19T14:54:37.820543+0000","last_became_active":"2024-02-19T14:54:36.835365+0000","last_became_peered":"2024-02-19T14:54:36.835365+0000","last_unstale":"2024-02-19T14:54:37.820543+0000","last_undegraded":"2024-02-19T14:54:37.820543+0000","last_fullsized":"2024-02-19T14:54:37.820543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:58.966226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819309+0000","last_change":"2024-02-19T14:54:26.751085+0000","last_active":"2024-02-19T14:54:37.819309+0000","last_peered":"2024-02-19T14:54:37.819309+0000","last_clean":"2024-02-19T14:54:37.819309+0000","last_became_active":"2024-02-19T14:54:26.750808+0000","last_became_peered":"2024-02-19T14:54:26.750808+0000","last_unstale":"2024-02-19T14:54:37.819309+0000","last_undegraded":"2024-02-19T14:54:37.819309+0000","last_fullsized":"2024-02-19T14:54:37.819309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823389+0000","last_change":"2024-02-19T14:54:25.732423+0000","last_active":"2024-02-19T14:54:37.823389+0000","last_peered":"2024-02-19T14:54:37.823389+0000","last_clean":"2024-02-19T14:54:37.823389+0000","last_became_active":"2024-02-19T14:54:25.732159+0000","last_became_peered":"2024-02-19T14:54:25.732159+0000","last_unstale":"2024-02-19T14:54:37.823389+0000","last_undegraded":"2024-02-19T14:54:37.823389+0000","last_fullsized":"2024-02-19T14:54:37.823389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:07.433362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845611+0000","last_change":"2024-02-19T14:54:37.838527+0000","last_active":"2024-02-19T14:54:37.845611+0000","last_peered":"2024-02-19T14:54:37.845611+0000","last_clean":"2024-02-19T14:54:37.845611+0000","last_became_active":"2024-02-19T14:54:37.838279+0000","last_became_peered":"2024-02-19T14:54:37.838279+0000","last_unstale":"2024-02-19T14:54:37.845611+0000","last_undegraded":"2024-02-19T14:54:37.845611+0000","last_fullsized":"2024-02-19T14:54:37.845611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:10.508329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815377+0000","last_change":"2024-02-19T14:54:36.830894+0000","last_active":"2024-02-19T14:54:37.815377+0000","last_peered":"2024-02-19T14:54:37.815377+0000","last_clean":"2024-02-19T14:54:37.815377+0000","last_became_active":"2024-02-19T14:54:36.830485+0000","last_became_peered":"2024-02-19T14:54:36.830485+0000","last_unstale":"2024-02-19T14:54:37.815377+0000","last_undegraded":"2024-02-19T14:54:37.815377+0000","last_fullsized":"2024-02-19T14:54:37.815377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:29:14.156802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814386+0000","last_change":"2024-02-19T14:54:26.747952+0000","last_active":"2024-02-19T14:54:37.814386+0000","last_peered":"2024-02-19T14:54:37.814386+0000","last_clean":"2024-02-19T14:54:37.814386+0000","last_became_active":"2024-02-19T14:54:26.747656+0000","last_became_peered":"2024-02-19T14:54:26.747656+0000","last_unstale":"2024-02-19T14:54:37.814386+0000","last_undegraded":"2024-02-19T14:54:37.814386+0000","last_fullsized":"2024-02-19T14:54:37.814386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:50.690817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827056+0000","last_change":"2024-02-19T14:54:25.733175+0000","last_active":"2024-02-19T14:54:37.827056+0000","last_peered":"2024-02-19T14:54:37.827056+0000","last_clean":"2024-02-19T14:54:37.827056+0000","last_became_active":"2024-02-19T14:54:25.732709+0000","last_became_peered":"2024-02-19T14:54:25.732709+0000","last_unstale":"2024-02-19T14:54:37.827056+0000","last_undegraded":"2024-02-19T14:54:37.827056+0000","last_fullsized":"2024-02-19T14:54:37.827056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:28.981539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846606+0000","last_change":"2024-02-19T14:54:37.842201+0000","last_active":"2024-02-19T14:54:37.846606+0000","last_peered":"2024-02-19T14:54:37.846606+0000","last_clean":"2024-02-19T14:54:37.846606+0000","last_became_active":"2024-02-19T14:54:37.841982+0000","last_became_peered":"2024-02-19T14:54:37.841982+0000","last_unstale":"2024-02-19T14:54:37.846606+0000","last_undegraded":"2024-02-19T14:54:37.846606+0000","last_fullsized":"2024-02-19T14:54:37.846606+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:43.283852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842919+0000","last_change":"2024-02-19T14:54:36.829399+0000","last_active":"2024-02-19T14:54:37.842919+0000","last_peered":"2024-02-19T14:54:37.842919+0000","last_clean":"2024-02-19T14:54:37.842919+0000","last_became_active":"2024-02-19T14:54:36.829111+0000","last_became_peered":"2024-02-19T14:54:36.829111+0000","last_unstale":"2024-02-19T14:54:37.842919+0000","last_undegraded":"2024-02-19T14:54:37.842919+0000","last_fullsized":"2024-02-19T14:54:37.842919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:13.065228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814956+0000","last_change":"2024-02-19T14:54:26.756834+0000","last_active":"2024-02-19T14:54:37.814956+0000","last_peered":"2024-02-19T14:54:37.814956+0000","last_clean":"2024-02-19T14:54:37.814956+0000","last_became_active":"2024-02-19T14:54:26.756548+0000","last_became_peered":"2024-02-19T14:54:26.756548+0000","last_unstale":"2024-02-19T14:54:37.814956+0000","last_undegraded":"2024-02-19T14:54:37.814956+0000","last_fullsized":"2024-02-19T14:54:37.814956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:25.077428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815986+0000","last_change":"2024-02-19T14:54:25.735865+0000","last_active":"2024-02-19T14:54:37.815986+0000","last_peered":"2024-02-19T14:54:37.815986+0000","last_clean":"2024-02-19T14:54:37.815986+0000","last_became_active":"2024-02-19T14:54:25.735569+0000","last_became_peered":"2024-02-19T14:54:25.735569+0000","last_unstale":"2024-02-19T14:54:37.815986+0000","last_undegraded":"2024-02-19T14:54:37.815986+0000","last_fullsized":"2024-02-19T14:54:37.815986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:55:08.537113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844182+0000","last_change":"2024-02-19T14:54:37.838803+0000","last_active":"2024-02-19T14:54:37.844182+0000","last_peered":"2024-02-19T14:54:37.844182+0000","last_clean":"2024-02-19T14:54:37.844182+0000","last_became_active":"2024-02-19T14:54:37.838530+0000","last_became_peered":"2024-02-19T14:54:37.838530+0000","last_unstale":"2024-02-19T14:54:37.844182+0000","last_undegraded":"2024-02-19T14:54:37.844182+0000","last_fullsized":"2024-02-19T14:54:37.844182+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:20.338152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814891+0000","last_change":"2024-02-19T14:54:36.827017+0000","last_active":"2024-02-19T14:54:37.814891+0000","last_peered":"2024-02-19T14:54:37.814891+0000","last_clean":"2024-02-19T14:54:37.814891+0000","last_became_active":"2024-02-19T14:54:36.826763+0000","last_became_peered":"2024-02-19T14:54:36.826763+0000","last_unstale":"2024-02-19T14:54:37.814891+0000","last_undegraded":"2024-02-19T14:54:37.814891+0000","last_fullsized":"2024-02-19T14:54:37.814891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:18.749797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819439+0000","last_change":"2024-02-19T14:54:26.750291+0000","last_active":"2024-02-19T14:54:37.819439+0000","last_peered":"2024-02-19T14:54:37.819439+0000","last_clean":"2024-02-19T14:54:37.819439+0000","last_became_active":"2024-02-19T14:54:26.750040+0000","last_became_peered":"2024-02-19T14:54:26.750040+0000","last_unstale":"2024-02-19T14:54:37.819439+0000","last_undegraded":"2024-02-19T14:54:37.819439+0000","last_fullsized":"2024-02-19T14:54:37.819439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822288+0000","last_change":"2024-02-19T14:54:25.731254+0000","last_active":"2024-02-19T14:54:37.822288+0000","last_peered":"2024-02-19T14:54:37.822288+0000","last_clean":"2024-02-19T14:54:37.822288+0000","last_became_active":"2024-02-19T14:54:25.730971+0000","last_became_peered":"2024-02-19T14:54:25.730971+0000","last_unstale":"2024-02-19T14:54:37.822288+0000","last_undegraded":"2024-02-19T14:54:37.822288+0000","last_fullsized":"2024-02-19T14:54:37.822288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:52.238124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844446+0000","last_change":"2024-02-19T14:54:37.841599+0000","last_active":"2024-02-19T14:54:37.844446+0000","last_peered":"2024-02-19T14:54:37.844446+0000","last_clean":"2024-02-19T14:54:37.844446+0000","last_became_active":"2024-02-19T14:54:37.840362+0000","last_became_peered":"2024-02-19T14:54:37.840362+0000","last_unstale":"2024-02-19T14:54:37.844446+0000","last_undegraded":"2024-02-19T14:54:37.844446+0000","last_fullsized":"2024-02-19T14:54:37.844446+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:40.525372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842579+0000","last_change":"2024-02-19T14:54:36.818412+0000","last_active":"2024-02-19T14:54:37.842579+0000","last_peered":"2024-02-19T14:54:37.842579+0000","last_clean":"2024-02-19T14:54:37.842579+0000","last_became_active":"2024-02-19T14:54:36.818117+0000","last_became_peered":"2024-02-19T14:54:36.818117+0000","last_unstale":"2024-02-19T14:54:37.842579+0000","last_undegraded":"2024-02-19T14:54:37.842579+0000","last_fullsized":"2024-02-19T14:54:37.842579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:26.994638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.828964+0000","last_change":"2024-02-19T14:54:26.746449+0000","last_active":"2024-02-19T14:54:37.828964+0000","last_peered":"2024-02-19T14:54:37.828964+0000","last_clean":"2024-02-19T14:54:37.828964+0000","last_became_active":"2024-02-19T14:54:26.746208+0000","last_became_peered":"2024-02-19T14:54:26.746208+0000","last_unstale":"2024-02-19T14:54:37.828964+0000","last_undegraded":"2024-02-19T14:54:37.828964+0000","last_fullsized":"2024-02-19T14:54:37.828964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:38.553508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819531+0000","last_change":"2024-02-19T14:54:25.734037+0000","last_active":"2024-02-19T14:54:37.819531+0000","last_peered":"2024-02-19T14:54:37.819531+0000","last_clean":"2024-02-19T14:54:37.819531+0000","last_became_active":"2024-02-19T14:54:25.733736+0000","last_became_peered":"2024-02-19T14:54:25.733736+0000","last_unstale":"2024-02-19T14:54:37.819531+0000","last_undegraded":"2024-02-19T14:54:37.819531+0000","last_fullsized":"2024-02-19T14:54:37.819531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:02.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846479+0000","last_change":"2024-02-19T14:54:37.845443+0000","last_active":"2024-02-19T14:54:37.846479+0000","last_peered":"2024-02-19T14:54:37.846479+0000","last_clean":"2024-02-19T14:54:37.846479+0000","last_became_active":"2024-02-19T14:54:37.844776+0000","last_became_peered":"2024-02-19T14:54:37.844776+0000","last_unstale":"2024-02-19T14:54:37.846479+0000","last_undegraded":"2024-02-19T14:54:37.846479+0000","last_fullsized":"2024-02-19T14:54:37.846479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:14.227823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839429+0000","last_change":"2024-02-19T14:54:36.827091+0000","last_active":"2024-02-19T14:54:37.839429+0000","last_peered":"2024-02-19T14:54:37.839429+0000","last_clean":"2024-02-19T14:54:37.839429+0000","last_became_active":"2024-02-19T14:54:36.826794+0000","last_became_peered":"2024-02-19T14:54:36.826794+0000","last_unstale":"2024-02-19T14:54:37.839429+0000","last_undegraded":"2024-02-19T14:54:37.839429+0000","last_fullsized":"2024-02-19T14:54:37.839429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:46.600327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.831192+0000","last_change":"2024-02-19T14:54:26.749731+0000","last_active":"2024-02-19T14:54:37.831192+0000","last_peered":"2024-02-19T14:54:37.831192+0000","last_clean":"2024-02-19T14:54:37.831192+0000","last_became_active":"2024-02-19T14:54:26.749338+0000","last_became_peered":"2024-02-19T14:54:26.749338+0000","last_unstale":"2024-02-19T14:54:37.831192+0000","last_undegraded":"2024-02-19T14:54:37.831192+0000","last_fullsized":"2024-02-19T14:54:37.831192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:14:43.891437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819640+0000","last_change":"2024-02-19T14:54:25.737967+0000","last_active":"2024-02-19T14:54:37.819640+0000","last_peered":"2024-02-19T14:54:37.819640+0000","last_clean":"2024-02-19T14:54:37.819640+0000","last_became_active":"2024-02-19T14:54:25.737526+0000","last_became_peered":"2024-02-19T14:54:25.737526+0000","last_unstale":"2024-02-19T14:54:37.819640+0000","last_undegraded":"2024-02-19T14:54:37.819640+0000","last_fullsized":"2024-02-19T14:54:37.819640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:09.115941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835844+0000","last_change":"2024-02-19T14:54:37.830541+0000","last_active":"2024-02-19T14:54:37.835844+0000","last_peered":"2024-02-19T14:54:37.835844+0000","last_clean":"2024-02-19T14:54:37.835844+0000","last_became_active":"2024-02-19T14:54:37.830250+0000","last_became_peered":"2024-02-19T14:54:37.830250+0000","last_unstale":"2024-02-19T14:54:37.835844+0000","last_undegraded":"2024-02-19T14:54:37.835844+0000","last_fullsized":"2024-02-19T14:54:37.835844+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:18.993081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844414+0000","last_change":"2024-02-19T14:54:36.830901+0000","last_active":"2024-02-19T14:54:37.844414+0000","last_peered":"2024-02-19T14:54:37.844414+0000","last_clean":"2024-02-19T14:54:37.844414+0000","last_became_active":"2024-02-19T14:54:36.830625+0000","last_became_peered":"2024-02-19T14:54:36.830625+0000","last_unstale":"2024-02-19T14:54:37.844414+0000","last_undegraded":"2024-02-19T14:54:37.844414+0000","last_fullsized":"2024-02-19T14:54:37.844414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:59:42.469485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820290+0000","last_change":"2024-02-19T14:54:26.743403+0000","last_active":"2024-02-19T14:54:37.820290+0000","last_peered":"2024-02-19T14:54:37.820290+0000","last_clean":"2024-02-19T14:54:37.820290+0000","last_became_active":"2024-02-19T14:54:26.742380+0000","last_became_peered":"2024-02-19T14:54:26.742380+0000","last_unstale":"2024-02-19T14:54:37.820290+0000","last_undegraded":"2024-02-19T14:54:37.820290+0000","last_fullsized":"2024-02-19T14:54:37.820290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:48.334394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819693+0000","last_change":"2024-02-19T14:54:25.737893+0000","last_active":"2024-02-19T14:54:37.819693+0000","last_peered":"2024-02-19T14:54:37.819693+0000","last_clean":"2024-02-19T14:54:37.819693+0000","last_became_active":"2024-02-19T14:54:25.737503+0000","last_became_peered":"2024-02-19T14:54:25.737503+0000","last_unstale":"2024-02-19T14:54:37.819693+0000","last_undegraded":"2024-02-19T14:54:37.819693+0000","last_fullsized":"2024-02-19T14:54:37.819693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:47.328330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841233+0000","last_change":"2024-02-19T14:54:37.837307+0000","last_active":"2024-02-19T14:54:37.841233+0000","last_peered":"2024-02-19T14:54:37.841233+0000","last_clean":"2024-02-19T14:54:37.841233+0000","last_became_active":"2024-02-19T14:54:37.836597+0000","last_became_peered":"2024-02-19T14:54:37.836597+0000","last_unstale":"2024-02-19T14:54:37.841233+0000","last_undegraded":"2024-02-19T14:54:37.841233+0000","last_fullsized":"2024-02-19T14:54:37.841233+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:59.970761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817173+0000","last_change":"2024-02-19T14:54:36.827233+0000","last_active":"2024-02-19T14:54:37.817173+0000","last_peered":"2024-02-19T14:54:37.817173+0000","last_clean":"2024-02-19T14:54:37.817173+0000","last_became_active":"2024-02-19T14:54:36.826930+0000","last_became_peered":"2024-02-19T14:54:36.826930+0000","last_unstale":"2024-02-19T14:54:37.817173+0000","last_undegraded":"2024-02-19T14:54:37.817173+0000","last_fullsized":"2024-02-19T14:54:37.817173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:38.418111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819106+0000","last_change":"2024-02-19T14:54:26.743767+0000","last_active":"2024-02-19T14:54:37.819106+0000","last_peered":"2024-02-19T14:54:37.819106+0000","last_clean":"2024-02-19T14:54:37.819106+0000","last_became_active":"2024-02-19T14:54:26.743477+0000","last_became_peered":"2024-02-19T14:54:26.743477+0000","last_unstale":"2024-02-19T14:54:37.819106+0000","last_undegraded":"2024-02-19T14:54:37.819106+0000","last_fullsized":"2024-02-19T14:54:37.819106+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:48.554748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821916+0000","last_change":"2024-02-19T14:54:25.729454+0000","last_active":"2024-02-19T14:54:37.821916+0000","last_peered":"2024-02-19T14:54:37.821916+0000","last_clean":"2024-02-19T14:54:37.821916+0000","last_became_active":"2024-02-19T14:54:25.729052+0000","last_became_peered":"2024-02-19T14:54:25.729052+0000","last_unstale":"2024-02-19T14:54:37.821916+0000","last_undegraded":"2024-02-19T14:54:37.821916+0000","last_fullsized":"2024-02-19T14:54:37.821916+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:04.379896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844503+0000","last_change":"2024-02-19T14:54:37.836034+0000","last_active":"2024-02-19T14:54:37.844503+0000","last_peered":"2024-02-19T14:54:37.844503+0000","last_clean":"2024-02-19T14:54:37.844503+0000","last_became_active":"2024-02-19T14:54:37.835762+0000","last_became_peered":"2024-02-19T14:54:37.835762+0000","last_unstale":"2024-02-19T14:54:37.844503+0000","last_undegraded":"2024-02-19T14:54:37.844503+0000","last_fullsized":"2024-02-19T14:54:37.844503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:43.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821032+0000","last_change":"2024-02-19T14:54:36.834792+0000","last_active":"2024-02-19T14:54:37.821032+0000","last_peered":"2024-02-19T14:54:37.821032+0000","last_clean":"2024-02-19T14:54:37.821032+0000","last_became_active":"2024-02-19T14:54:36.834516+0000","last_became_peered":"2024-02-19T14:54:36.834516+0000","last_unstale":"2024-02-19T14:54:37.821032+0000","last_undegraded":"2024-02-19T14:54:37.821032+0000","last_fullsized":"2024-02-19T14:54:37.821032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:19.333617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817540+0000","last_change":"2024-02-19T14:54:26.742043+0000","last_active":"2024-02-19T14:54:37.817540+0000","last_peered":"2024-02-19T14:54:37.817540+0000","last_clean":"2024-02-19T14:54:37.817540+0000","last_became_active":"2024-02-19T14:54:26.741725+0000","last_became_peered":"2024-02-19T14:54:26.741725+0000","last_unstale":"2024-02-19T14:54:37.817540+0000","last_undegraded":"2024-02-19T14:54:37.817540+0000","last_fullsized":"2024-02-19T14:54:37.817540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:10.122005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.832069+0000","last_change":"2024-02-19T14:54:25.737062+0000","last_active":"2024-02-19T14:54:37.832069+0000","last_peered":"2024-02-19T14:54:37.832069+0000","last_clean":"2024-02-19T14:54:37.832069+0000","last_became_active":"2024-02-19T14:54:25.736783+0000","last_became_peered":"2024-02-19T14:54:25.736783+0000","last_unstale":"2024-02-19T14:54:37.832069+0000","last_undegraded":"2024-02-19T14:54:37.832069+0000","last_fullsized":"2024-02-19T14:54:37.832069+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:56.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836483+0000","last_change":"2024-02-19T14:54:37.833286+0000","last_active":"2024-02-19T14:54:37.836483+0000","last_peered":"2024-02-19T14:54:37.836483+0000","last_clean":"2024-02-19T14:54:37.836483+0000","last_became_active":"2024-02-19T14:54:37.833025+0000","last_became_peered":"2024-02-19T14:54:37.833025+0000","last_unstale":"2024-02-19T14:54:37.836483+0000","last_undegraded":"2024-02-19T14:54:37.836483+0000","last_fullsized":"2024-02-19T14:54:37.836483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:00.935158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827550+0000","last_change":"2024-02-19T14:54:36.834415+0000","last_active":"2024-02-19T14:54:37.827550+0000","last_peered":"2024-02-19T14:54:37.827550+0000","last_clean":"2024-02-19T14:54:37.827550+0000","last_became_active":"2024-02-19T14:54:36.833594+0000","last_became_peered":"2024-02-19T14:54:36.833594+0000","last_unstale":"2024-02-19T14:54:37.827550+0000","last_undegraded":"2024-02-19T14:54:37.827550+0000","last_fullsized":"2024-02-19T14:54:37.827550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:58.062735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825987+0000","last_change":"2024-02-19T14:54:26.744132+0000","last_active":"2024-02-19T14:54:37.825987+0000","last_peered":"2024-02-19T14:54:37.825987+0000","last_clean":"2024-02-19T14:54:37.825987+0000","last_became_active":"2024-02-19T14:54:26.741010+0000","last_became_peered":"2024-02-19T14:54:26.741010+0000","last_unstale":"2024-02-19T14:54:37.825987+0000","last_undegraded":"2024-02-19T14:54:37.825987+0000","last_fullsized":"2024-02-19T14:54:37.825987+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:04.497808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825519+0000","last_change":"2024-02-19T14:54:25.736615+0000","last_active":"2024-02-19T14:54:37.825519+0000","last_peered":"2024-02-19T14:54:37.825519+0000","last_clean":"2024-02-19T14:54:37.825519+0000","last_became_active":"2024-02-19T14:54:25.736328+0000","last_became_peered":"2024-02-19T14:54:25.736328+0000","last_unstale":"2024-02-19T14:54:37.825519+0000","last_undegraded":"2024-02-19T14:54:37.825519+0000","last_fullsized":"2024-02-19T14:54:37.825519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:07.018554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842345+0000","last_change":"2024-02-19T14:54:37.834319+0000","last_active":"2024-02-19T14:54:37.842345+0000","last_peered":"2024-02-19T14:54:37.842345+0000","last_clean":"2024-02-19T14:54:37.842345+0000","last_became_active":"2024-02-19T14:54:37.834070+0000","last_became_peered":"2024-02-19T14:54:37.834070+0000","last_unstale":"2024-02-19T14:54:37.842345+0000","last_undegraded":"2024-02-19T14:54:37.842345+0000","last_fullsized":"2024-02-19T14:54:37.842345+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:48.394105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823853+0000","last_change":"2024-02-19T14:54:36.830086+0000","last_active":"2024-02-19T14:54:37.823853+0000","last_peered":"2024-02-19T14:54:37.823853+0000","last_clean":"2024-02-19T14:54:37.823853+0000","last_became_active":"2024-02-19T14:54:36.829795+0000","last_became_peered":"2024-02-19T14:54:36.829795+0000","last_unstale":"2024-02-19T14:54:37.823853+0000","last_undegraded":"2024-02-19T14:54:37.823853+0000","last_fullsized":"2024-02-19T14:54:37.823853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:48.469617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827768+0000","last_change":"2024-02-19T14:54:26.743310+0000","last_active":"2024-02-19T14:54:37.827768+0000","last_peered":"2024-02-19T14:54:37.827768+0000","last_clean":"2024-02-19T14:54:37.827768+0000","last_became_active":"2024-02-19T14:54:26.742887+0000","last_became_peered":"2024-02-19T14:54:26.742887+0000","last_unstale":"2024-02-19T14:54:37.827768+0000","last_undegraded":"2024-02-19T14:54:37.827768+0000","last_fullsized":"2024-02-19T14:54:37.827768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821737+0000","last_change":"2024-02-19T14:54:25.736615+0000","last_active":"2024-02-19T14:54:37.821737+0000","last_peered":"2024-02-19T14:54:37.821737+0000","last_clean":"2024-02-19T14:54:37.821737+0000","last_became_active":"2024-02-19T14:54:25.736314+0000","last_became_peered":"2024-02-19T14:54:25.736314+0000","last_unstale":"2024-02-19T14:54:37.821737+0000","last_undegraded":"2024-02-19T14:54:37.821737+0000","last_fullsized":"2024-02-19T14:54:37.821737+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:41.651494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845963+0000","last_change":"2024-02-19T14:54:37.841997+0000","last_active":"2024-02-19T14:54:37.845963+0000","last_peered":"2024-02-19T14:54:37.845963+0000","last_clean":"2024-02-19T14:54:37.845963+0000","last_became_active":"2024-02-19T14:54:37.841668+0000","last_became_peered":"2024-02-19T14:54:37.841668+0000","last_unstale":"2024-02-19T14:54:37.845963+0000","last_undegraded":"2024-02-19T14:54:37.845963+0000","last_fullsized":"2024-02-19T14:54:37.845963+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:40.611498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.824452+0000","last_change":"2024-02-19T14:54:36.833066+0000","last_active":"2024-02-19T14:54:37.824452+0000","last_peered":"2024-02-19T14:54:37.824452+0000","last_clean":"2024-02-19T14:54:37.824452+0000","last_became_active":"2024-02-19T14:54:36.832815+0000","last_became_peered":"2024-02-19T14:54:36.832815+0000","last_unstale":"2024-02-19T14:54:37.824452+0000","last_undegraded":"2024-02-19T14:54:37.824452+0000","last_fullsized":"2024-02-19T14:54:37.824452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:37.580307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819373+0000","last_change":"2024-02-19T14:54:26.742237+0000","last_active":"2024-02-19T14:54:37.819373+0000","last_peered":"2024-02-19T14:54:37.819373+0000","last_clean":"2024-02-19T14:54:37.819373+0000","last_became_active":"2024-02-19T14:54:26.741929+0000","last_became_peered":"2024-02-19T14:54:26.741929+0000","last_unstale":"2024-02-19T14:54:37.819373+0000","last_undegraded":"2024-02-19T14:54:37.819373+0000","last_fullsized":"2024-02-19T14:54:37.819373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:47.414910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.818743+0000","last_change":"2024-02-19T14:54:25.734486+0000","last_active":"2024-02-19T14:54:37.818743+0000","last_peered":"2024-02-19T14:54:37.818743+0000","last_clean":"2024-02-19T14:54:37.818743+0000","last_became_active":"2024-02-19T14:54:25.734200+0000","last_became_peered":"2024-02-19T14:54:25.734200+0000","last_unstale":"2024-02-19T14:54:37.818743+0000","last_undegraded":"2024-02-19T14:54:37.818743+0000","last_fullsized":"2024-02-19T14:54:37.818743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:03.491743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.837289+0000","last_change":"2024-02-19T14:54:37.831915+0000","last_active":"2024-02-19T14:54:37.837289+0000","last_peered":"2024-02-19T14:54:37.837289+0000","last_clean":"2024-02-19T14:54:37.837289+0000","last_became_active":"2024-02-19T14:54:37.831649+0000","last_became_peered":"2024-02-19T14:54:37.831649+0000","last_unstale":"2024-02-19T14:54:37.837289+0000","last_undegraded":"2024-02-19T14:54:37.837289+0000","last_fullsized":"2024-02-19T14:54:37.837289+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:52:53.015938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819439+0000","last_change":"2024-02-19T14:54:36.828188+0000","last_active":"2024-02-19T14:54:37.819439+0000","last_peered":"2024-02-19T14:54:37.819439+0000","last_clean":"2024-02-19T14:54:37.819439+0000","last_became_active":"2024-02-19T14:54:36.827920+0000","last_became_peered":"2024-02-19T14:54:36.827920+0000","last_unstale":"2024-02-19T14:54:37.819439+0000","last_undegraded":"2024-02-19T14:54:37.819439+0000","last_fullsized":"2024-02-19T14:54:37.819439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:41.654671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819507+0000","last_change":"2024-02-19T14:54:26.751887+0000","last_active":"2024-02-19T14:54:37.819507+0000","last_peered":"2024-02-19T14:54:37.819507+0000","last_clean":"2024-02-19T14:54:37.819507+0000","last_became_active":"2024-02-19T14:54:26.751432+0000","last_became_peered":"2024-02-19T14:54:26.751432+0000","last_unstale":"2024-02-19T14:54:37.819507+0000","last_undegraded":"2024-02-19T14:54:37.819507+0000","last_fullsized":"2024-02-19T14:54:37.819507+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:27.494199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815220+0000","last_change":"2024-02-19T14:54:25.738928+0000","last_active":"2024-02-19T14:54:37.815220+0000","last_peered":"2024-02-19T14:54:37.815220+0000","last_clean":"2024-02-19T14:54:37.815220+0000","last_became_active":"2024-02-19T14:54:25.738674+0000","last_became_peered":"2024-02-19T14:54:25.738674+0000","last_unstale":"2024-02-19T14:54:37.815220+0000","last_undegraded":"2024-02-19T14:54:37.815220+0000","last_fullsized":"2024-02-19T14:54:37.815220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:58.792240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841292+0000","last_change":"2024-02-19T14:54:37.836690+0000","last_active":"2024-02-19T14:54:37.841292+0000","last_peered":"2024-02-19T14:54:37.841292+0000","last_clean":"2024-02-19T14:54:37.841292+0000","last_became_active":"2024-02-19T14:54:37.835735+0000","last_became_peered":"2024-02-19T14:54:37.835735+0000","last_unstale":"2024-02-19T14:54:37.841292+0000","last_undegraded":"2024-02-19T14:54:37.841292+0000","last_fullsized":"2024-02-19T14:54:37.841292+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:34:44.271939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841428+0000","last_change":"2024-02-19T14:54:36.830155+0000","last_active":"2024-02-19T14:54:37.841428+0000","last_peered":"2024-02-19T14:54:37.841428+0000","last_clean":"2024-02-19T14:54:37.841428+0000","last_became_active":"2024-02-19T14:54:36.829822+0000","last_became_peered":"2024-02-19T14:54:36.829822+0000","last_unstale":"2024-02-19T14:54:37.841428+0000","last_undegraded":"2024-02-19T14:54:37.841428+0000","last_fullsized":"2024-02-19T14:54:37.841428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:42.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822240+0000","last_change":"2024-02-19T14:54:26.754376+0000","last_active":"2024-02-19T14:54:37.822240+0000","last_peered":"2024-02-19T14:54:37.822240+0000","last_clean":"2024-02-19T14:54:37.822240+0000","last_became_active":"2024-02-19T14:54:26.754102+0000","last_became_peered":"2024-02-19T14:54:26.754102+0000","last_unstale":"2024-02-19T14:54:37.822240+0000","last_undegraded":"2024-02-19T14:54:37.822240+0000","last_fullsized":"2024-02-19T14:54:37.822240+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:02.882466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817664+0000","last_change":"2024-02-19T14:54:25.735756+0000","last_active":"2024-02-19T14:54:37.817664+0000","last_peered":"2024-02-19T14:54:37.817664+0000","last_clean":"2024-02-19T14:54:37.817664+0000","last_became_active":"2024-02-19T14:54:25.735486+0000","last_became_peered":"2024-02-19T14:54:25.735486+0000","last_unstale":"2024-02-19T14:54:37.817664+0000","last_undegraded":"2024-02-19T14:54:37.817664+0000","last_fullsized":"2024-02-19T14:54:37.817664+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:12.694335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846488+0000","last_change":"2024-02-19T14:54:37.845487+0000","last_active":"2024-02-19T14:54:37.846488+0000","last_peered":"2024-02-19T14:54:37.846488+0000","last_clean":"2024-02-19T14:54:37.846488+0000","last_became_active":"2024-02-19T14:54:37.845242+0000","last_became_peered":"2024-02-19T14:54:37.845242+0000","last_unstale":"2024-02-19T14:54:37.846488+0000","last_undegraded":"2024-02-19T14:54:37.846488+0000","last_fullsized":"2024-02-19T14:54:37.846488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:43.228266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826689+0000","last_change":"2024-02-19T14:54:36.825202+0000","last_active":"2024-02-19T14:54:37.826689+0000","last_peered":"2024-02-19T14:54:37.826689+0000","last_clean":"2024-02-19T14:54:37.826689+0000","last_became_active":"2024-02-19T14:54:36.824918+0000","last_became_peered":"2024-02-19T14:54:36.824918+0000","last_unstale":"2024-02-19T14:54:37.826689+0000","last_undegraded":"2024-02-19T14:54:37.826689+0000","last_fullsized":"2024-02-19T14:54:37.826689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:15.472069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821787+0000","last_change":"2024-02-19T14:54:26.743441+0000","last_active":"2024-02-19T14:54:37.821787+0000","last_peered":"2024-02-19T14:54:37.821787+0000","last_clean":"2024-02-19T14:54:37.821787+0000","last_became_active":"2024-02-19T14:54:26.743151+0000","last_became_peered":"2024-02-19T14:54:26.743151+0000","last_unstale":"2024-02-19T14:54:37.821787+0000","last_undegraded":"2024-02-19T14:54:37.821787+0000","last_fullsized":"2024-02-19T14:54:37.821787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:27.494199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819120+0000","last_change":"2024-02-19T14:54:25.728807+0000","last_active":"2024-02-19T14:54:37.819120+0000","last_peered":"2024-02-19T14:54:37.819120+0000","last_clean":"2024-02-19T14:54:37.819120+0000","last_became_active":"2024-02-19T14:54:25.728541+0000","last_became_peered":"2024-02-19T14:54:25.728541+0000","last_unstale":"2024-02-19T14:54:37.819120+0000","last_undegraded":"2024-02-19T14:54:37.819120+0000","last_fullsized":"2024-02-19T14:54:37.819120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:12.694335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838481+0000","last_change":"2024-02-19T14:54:37.835628+0000","last_active":"2024-02-19T14:54:37.838481+0000","last_peered":"2024-02-19T14:54:37.838481+0000","last_clean":"2024-02-19T14:54:37.838481+0000","last_became_active":"2024-02-19T14:54:37.833937+0000","last_became_peered":"2024-02-19T14:54:37.833937+0000","last_unstale":"2024-02-19T14:54:37.838481+0000","last_undegraded":"2024-02-19T14:54:37.838481+0000","last_fullsized":"2024-02-19T14:54:37.838481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:10.508329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822177+0000","last_change":"2024-02-19T14:54:36.825196+0000","last_active":"2024-02-19T14:54:37.822177+0000","last_peered":"2024-02-19T14:54:37.822177+0000","last_clean":"2024-02-19T14:54:37.822177+0000","last_became_active":"2024-02-19T14:54:36.824941+0000","last_became_peered":"2024-02-19T14:54:36.824941+0000","last_unstale":"2024-02-19T14:54:37.822177+0000","last_undegraded":"2024-02-19T14:54:37.822177+0000","last_fullsized":"2024-02-19T14:54:37.822177+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:18.525535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825384+0000","last_change":"2024-02-19T14:54:26.742776+0000","last_active":"2024-02-19T14:54:37.825384+0000","last_peered":"2024-02-19T14:54:37.825384+0000","last_clean":"2024-02-19T14:54:37.825384+0000","last_became_active":"2024-02-19T14:54:26.742433+0000","last_became_peered":"2024-02-19T14:54:26.742433+0000","last_unstale":"2024-02-19T14:54:37.825384+0000","last_undegraded":"2024-02-19T14:54:37.825384+0000","last_fullsized":"2024-02-19T14:54:37.825384+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820247+0000","last_change":"2024-02-19T14:54:25.738298+0000","last_active":"2024-02-19T14:54:37.820247+0000","last_peered":"2024-02-19T14:54:37.820247+0000","last_clean":"2024-02-19T14:54:37.820247+0000","last_became_active":"2024-02-19T14:54:25.737717+0000","last_became_peered":"2024-02-19T14:54:25.737717+0000","last_unstale":"2024-02-19T14:54:37.820247+0000","last_undegraded":"2024-02-19T14:54:37.820247+0000","last_fullsized":"2024-02-19T14:54:37.820247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:07.018554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":279,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821363+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:54:37.821363+0000","last_peered":"2024-02-19T14:54:37.821363+0000","last_clean":"2024-02-19T14:54:37.821363+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:54:37.821363+0000","last_undegraded":"2024-02-19T14:54:37.821363+0000","last_fullsized":"2024-02-19T14:54:37.821363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1540096,"data_stored":1039768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.497,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.5},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":450990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.62},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42299999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":449412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.496},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56299999999999994}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55800000000000005,"15min":0.56899999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":451094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.747,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.59699999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95799999999999996,"5min":1.038,"15min":1.038},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.64200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":457985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29499999999999998},"min":{"1min":0.17899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.309,"15min":0.30499999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.245}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.31},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.374,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49199999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69099999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.42799999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68700000000000006,"5min":1.133,"15min":1.133},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.38},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.502}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82999999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.51200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":450120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.222},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.437,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.42799999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.20699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.253},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.375,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.309}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79000000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.433},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84199999999999997,"5min":0.878,"15min":0.878},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94199999999999995,"5min":1.236,"15min":1.236},"last":0.54000000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.995,"15min":0.995},"last":0.49199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":457585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33500000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.497,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33100000000000002,"15min":0.33800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.307,"15min":0.309},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.371,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.23699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57999999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.438}]},{"osd":6,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44400000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.55100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1536000,"data_stored":1039746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.38100000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.193},{"interface":"front","average":{"1min":0.315,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32300000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42599999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34899999999999998,"15min":0.35199999999999998},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.42699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34499999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.41199999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.317}]},{"osd":4,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57599999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.504}]},{"osd":5,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.59999999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68700000000000006,"5min":0.754,"15min":0.754},"last":0.63100000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72999999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:54:40.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:54:41.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 Processor -- start 2024-02-19T14:54:41.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 -- start start 2024-02-19T14:54:41.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7640c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7640c12a0 0x7fc7640723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76406f330 con 0x7fc764072910 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7640a84c0 con 0x7fc7640c12a0 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc76962a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640a8fb0 con 0x7fc76414f3a0 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.086+0000 7fc75a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7640c12a0 0x7fc7640723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc75a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7640c12a0 0x7fc7640723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32986/0 (socket says 172.21.15.78:32986) 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3752194002 learned_addr learned my addr 172.21.15.78:0/3752194002 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc762ffd640 1 --2- 172.21.15.78:0/3752194002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7640c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3752194002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 msgr2=0x7fc7640c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc75a7fc640 1 --2- 172.21.15.78:0/3752194002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7640c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3752194002 >> v1:172.21.15.150:6789/0 conn(0x7fc764072910 legacy=0x7fc764151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:41.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.087+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3752194002 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640c35f0 con 0x7fc7640c12a0 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc75a7fc640 1 --2- 172.21.15.78:0/3752194002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7640723d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009870 tx=0x7fc74c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd59c5f3ac62b70 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc7627fc640 1 -- 172.21.15.78:0/3752194002 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c031740 con 0x7fc7640c12a0 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc7627fc640 1 -- 172.21.15.78:0/3752194002 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c0318a0 con 0x7fc7640c12a0 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc7627fc640 1 -- 172.21.15.78:0/3752194002 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c031c30 con 0x7fc7640c12a0 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 -- 172.21.15.78:0/3752194002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 msgr2=0x7fc7640723d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 --2- 172.21.15.78:0/3752194002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7640723d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009870 tx=0x7fc74c02b7e0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 -- 172.21.15.78:0/3752194002 shutdown_connections 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 --2- 172.21.15.78:0/3752194002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7640723d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 --2- 172.21.15.78:0/3752194002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7640c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 -- 172.21.15.78:0/3752194002 >> 172.21.15.78:0/3752194002 conn(0x7fc7640718e0 msgr2=0x7fc764071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 -- 172.21.15.78:0/3752194002 shutdown_connections 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.088+0000 7fc76962a640 1 -- 172.21.15.78:0/3752194002 wait complete. 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.089+0000 7fc76962a640 1 Processor -- start 2024-02-19T14:54:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 -- start start 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc764072910 0x7fc7640c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7641e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7641e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc764151910 con 0x7fc7640c12a0 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7641513e0 con 0x7fc764072910 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc76962a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc764151630 con 0x7fc76414f3a0 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc75a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7641e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc75a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7641e8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32990/0 (socket says 172.21.15.78:32990) 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.090+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3762857969 learned_addr learned my addr 172.21.15.78:0/3762857969 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:41.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc7637fe640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7641e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc762ffd640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc764072910 0x7fc7640c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 msgr2=0x7fc7641e8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75a7fc640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7641e8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc764072910 msgr2=0x7fc7640c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75a7fc640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc764072910 0x7fc7640c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75a7fc640 1 -- 172.21.15.78:0/3762857969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c0094d0 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75a7fc640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7641e8a40 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009870 tx=0x7fc74c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c033c70 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c033dd0 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c03c640 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7641ec850 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.091+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7640c6260 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.092+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7641ec6b0 con 0x7fc7640c12a0 2024-02-19T14:54:41.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.093+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc74c03a040 con 0x7fc7640c12a0 2024-02-19T14:54:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.093+0000 7fc75bfff640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc74406b810 0x7fc74406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.093+0000 7fc762ffd640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc74406b810 0x7fc74406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.093+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7073+0+0 (secure 0 0 0) 0x7fc74c0ac150 con 0x7fc7640c12a0 2024-02-19T14:54:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.094+0000 7fc762ffd640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc74406b810 0x7fc74406dcd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc754006fd0 tx=0x7fc754008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.097+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc7641ec6b0 con 0x7fc7640c12a0 2024-02-19T14:54:41.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.303+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7640a89f0 con 0x7fc74406b810 2024-02-19T14:54:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.328+0000 7fc75bfff640 1 -- 172.21.15.78:0/3762857969 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528014 (secure 0 0 0) 0x7fc75400e4b0 con 0x7fc74406b810 2024-02-19T14:54:41.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc74406b810 msgr2=0x7fc74406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc74406b810 0x7fc74406dcd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc754006fd0 tx=0x7fc754008040 comp rx=0 tx=0).stop 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 msgr2=0x7fc7641e8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7641e8a40 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009870 tx=0x7fc74c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 shutdown_connections 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc74406b810 0x7fc74406dcd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7640c12a0 0x7fc7641e8a40 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc76414f3a0 0x7fc7641e8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 --2- 172.21.15.78:0/3762857969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc764072910 0x7fc7640c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 >> 172.21.15.78:0/3762857969 conn(0x7fc7640718e0 msgr2=0x7fc76414cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 shutdown_connections 2024-02-19T14:54:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.332+0000 7fc76962a640 1 -- 172.21.15.78:0/3762857969 wait complete. 2024-02-19T14:54:41.346 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-02-19T14:54:40.842732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":261,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224784,"kb_used_data":8612,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754749936,"statfs":{"total":773094113280,"available":772863934464,"internally_reserved":0,"allocated":8818688,"data_stored":4796700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.751745"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842491+0000","last_change":"2024-02-19T14:54:37.837507+0000","last_active":"2024-02-19T14:54:37.842491+0000","last_peered":"2024-02-19T14:54:37.842491+0000","last_clean":"2024-02-19T14:54:37.842491+0000","last_became_active":"2024-02-19T14:54:37.837268+0000","last_became_peered":"2024-02-19T14:54:37.837268+0000","last_unstale":"2024-02-19T14:54:37.842491+0000","last_undegraded":"2024-02-19T14:54:37.842491+0000","last_fullsized":"2024-02-19T14:54:37.842491+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:05.336408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823128+0000","last_change":"2024-02-19T14:54:26.743766+0000","last_active":"2024-02-19T14:54:37.823128+0000","last_peered":"2024-02-19T14:54:37.823128+0000","last_clean":"2024-02-19T14:54:37.823128+0000","last_became_active":"2024-02-19T14:54:26.743402+0000","last_became_peered":"2024-02-19T14:54:26.743402+0000","last_unstale":"2024-02-19T14:54:37.823128+0000","last_undegraded":"2024-02-19T14:54:37.823128+0000","last_fullsized":"2024-02-19T14:54:37.823128+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846266+0000","last_change":"2024-02-19T14:54:37.842884+0000","last_active":"2024-02-19T14:54:37.846266+0000","last_peered":"2024-02-19T14:54:37.846266+0000","last_clean":"2024-02-19T14:54:37.846266+0000","last_became_active":"2024-02-19T14:54:37.842649+0000","last_became_peered":"2024-02-19T14:54:37.842649+0000","last_unstale":"2024-02-19T14:54:37.846266+0000","last_undegraded":"2024-02-19T14:54:37.846266+0000","last_fullsized":"2024-02-19T14:54:37.846266+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:18.993081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.831094+0000","last_change":"2024-02-19T14:54:26.751673+0000","last_active":"2024-02-19T14:54:37.831094+0000","last_peered":"2024-02-19T14:54:37.831094+0000","last_clean":"2024-02-19T14:54:37.831094+0000","last_became_active":"2024-02-19T14:54:26.751360+0000","last_became_peered":"2024-02-19T14:54:26.751360+0000","last_unstale":"2024-02-19T14:54:37.831094+0000","last_undegraded":"2024-02-19T14:54:37.831094+0000","last_fullsized":"2024-02-19T14:54:37.831094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844010+0000","last_change":"2024-02-19T14:54:37.843084+0000","last_active":"2024-02-19T14:54:37.844010+0000","last_peered":"2024-02-19T14:54:37.844010+0000","last_clean":"2024-02-19T14:54:37.844010+0000","last_became_active":"2024-02-19T14:54:37.842512+0000","last_became_peered":"2024-02-19T14:54:37.842512+0000","last_unstale":"2024-02-19T14:54:37.844010+0000","last_undegraded":"2024-02-19T14:54:37.844010+0000","last_fullsized":"2024-02-19T14:54:37.844010+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:36:42.957311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821285+0000","last_change":"2024-02-19T14:54:26.750603+0000","last_active":"2024-02-19T14:54:37.821285+0000","last_peered":"2024-02-19T14:54:37.821285+0000","last_clean":"2024-02-19T14:54:37.821285+0000","last_became_active":"2024-02-19T14:54:26.750088+0000","last_became_peered":"2024-02-19T14:54:26.750088+0000","last_unstale":"2024-02-19T14:54:37.821285+0000","last_undegraded":"2024-02-19T14:54:37.821285+0000","last_fullsized":"2024-02-19T14:54:37.821285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:47.414910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841260+0000","last_change":"2024-02-19T14:54:37.838099+0000","last_active":"2024-02-19T14:54:37.841260+0000","last_peered":"2024-02-19T14:54:37.841260+0000","last_clean":"2024-02-19T14:54:37.841260+0000","last_became_active":"2024-02-19T14:54:37.837824+0000","last_became_peered":"2024-02-19T14:54:37.837824+0000","last_unstale":"2024-02-19T14:54:37.841260+0000","last_undegraded":"2024-02-19T14:54:37.841260+0000","last_fullsized":"2024-02-19T14:54:37.841260+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:48.394105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822248+0000","last_change":"2024-02-19T14:54:26.733014+0000","last_active":"2024-02-19T14:54:37.822248+0000","last_peered":"2024-02-19T14:54:37.822248+0000","last_clean":"2024-02-19T14:54:37.822248+0000","last_became_active":"2024-02-19T14:54:26.732738+0000","last_became_peered":"2024-02-19T14:54:26.732738+0000","last_unstale":"2024-02-19T14:54:37.822248+0000","last_undegraded":"2024-02-19T14:54:37.822248+0000","last_fullsized":"2024-02-19T14:54:37.822248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:37.293731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835059+0000","last_change":"2024-02-19T14:54:37.833022+0000","last_active":"2024-02-19T14:54:37.835059+0000","last_peered":"2024-02-19T14:54:37.835059+0000","last_clean":"2024-02-19T14:54:37.835059+0000","last_became_active":"2024-02-19T14:54:37.831958+0000","last_became_peered":"2024-02-19T14:54:37.831958+0000","last_unstale":"2024-02-19T14:54:37.835059+0000","last_undegraded":"2024-02-19T14:54:37.835059+0000","last_fullsized":"2024-02-19T14:54:37.835059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:46:45.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815053+0000","last_change":"2024-02-19T14:54:26.747816+0000","last_active":"2024-02-19T14:54:37.815053+0000","last_peered":"2024-02-19T14:54:37.815053+0000","last_clean":"2024-02-19T14:54:37.815053+0000","last_became_active":"2024-02-19T14:54:26.747520+0000","last_became_peered":"2024-02-19T14:54:26.747520+0000","last_unstale":"2024-02-19T14:54:37.815053+0000","last_undegraded":"2024-02-19T14:54:37.815053+0000","last_fullsized":"2024-02-19T14:54:37.815053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:36.704748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840622+0000","last_change":"2024-02-19T14:54:37.836146+0000","last_active":"2024-02-19T14:54:37.840622+0000","last_peered":"2024-02-19T14:54:37.840622+0000","last_clean":"2024-02-19T14:54:37.840622+0000","last_became_active":"2024-02-19T14:54:37.835654+0000","last_became_peered":"2024-02-19T14:54:37.835654+0000","last_unstale":"2024-02-19T14:54:37.840622+0000","last_undegraded":"2024-02-19T14:54:37.840622+0000","last_fullsized":"2024-02-19T14:54:37.840622+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:24.580690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819866+0000","last_change":"2024-02-19T14:54:26.746700+0000","last_active":"2024-02-19T14:54:37.819866+0000","last_peered":"2024-02-19T14:54:37.819866+0000","last_clean":"2024-02-19T14:54:37.819866+0000","last_became_active":"2024-02-19T14:54:26.746436+0000","last_became_peered":"2024-02-19T14:54:26.746436+0000","last_unstale":"2024-02-19T14:54:37.819866+0000","last_undegraded":"2024-02-19T14:54:37.819866+0000","last_fullsized":"2024-02-19T14:54:37.819866+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843709+0000","last_change":"2024-02-19T14:54:37.837088+0000","last_active":"2024-02-19T14:54:37.843709+0000","last_peered":"2024-02-19T14:54:37.843709+0000","last_clean":"2024-02-19T14:54:37.843709+0000","last_became_active":"2024-02-19T14:54:37.835520+0000","last_became_peered":"2024-02-19T14:54:37.835520+0000","last_unstale":"2024-02-19T14:54:37.843709+0000","last_undegraded":"2024-02-19T14:54:37.843709+0000","last_fullsized":"2024-02-19T14:54:37.843709+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:51.437501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817399+0000","last_change":"2024-02-19T14:54:26.751787+0000","last_active":"2024-02-19T14:54:37.817399+0000","last_peered":"2024-02-19T14:54:37.817399+0000","last_clean":"2024-02-19T14:54:37.817399+0000","last_became_active":"2024-02-19T14:54:26.751414+0000","last_became_peered":"2024-02-19T14:54:26.751414+0000","last_unstale":"2024-02-19T14:54:37.817399+0000","last_undegraded":"2024-02-19T14:54:37.817399+0000","last_fullsized":"2024-02-19T14:54:37.817399+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:31.766567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.848706+0000","last_change":"2024-02-19T14:54:37.847071+0000","last_active":"2024-02-19T14:54:37.848706+0000","last_peered":"2024-02-19T14:54:37.848706+0000","last_clean":"2024-02-19T14:54:37.848706+0000","last_became_active":"2024-02-19T14:54:37.846827+0000","last_became_peered":"2024-02-19T14:54:37.846827+0000","last_unstale":"2024-02-19T14:54:37.848706+0000","last_undegraded":"2024-02-19T14:54:37.848706+0000","last_fullsized":"2024-02-19T14:54:37.848706+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:31.003078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814652+0000","last_change":"2024-02-19T14:54:26.748066+0000","last_active":"2024-02-19T14:54:37.814652+0000","last_peered":"2024-02-19T14:54:37.814652+0000","last_clean":"2024-02-19T14:54:37.814652+0000","last_became_active":"2024-02-19T14:54:26.747792+0000","last_became_peered":"2024-02-19T14:54:26.747792+0000","last_unstale":"2024-02-19T14:54:37.814652+0000","last_undegraded":"2024-02-19T14:54:37.814652+0000","last_fullsized":"2024-02-19T14:54:37.814652+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:05.620631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839543+0000","last_change":"2024-02-19T14:54:37.832366+0000","last_active":"2024-02-19T14:54:37.839543+0000","last_peered":"2024-02-19T14:54:37.839543+0000","last_clean":"2024-02-19T14:54:37.839543+0000","last_became_active":"2024-02-19T14:54:37.832111+0000","last_became_peered":"2024-02-19T14:54:37.832111+0000","last_unstale":"2024-02-19T14:54:37.839543+0000","last_undegraded":"2024-02-19T14:54:37.839543+0000","last_fullsized":"2024-02-19T14:54:37.839543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:59.425037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819364+0000","last_change":"2024-02-19T14:54:26.741238+0000","last_active":"2024-02-19T14:54:37.819364+0000","last_peered":"2024-02-19T14:54:37.819364+0000","last_clean":"2024-02-19T14:54:37.819364+0000","last_became_active":"2024-02-19T14:54:26.740986+0000","last_became_peered":"2024-02-19T14:54:26.740986+0000","last_unstale":"2024-02-19T14:54:37.819364+0000","last_undegraded":"2024-02-19T14:54:37.819364+0000","last_fullsized":"2024-02-19T14:54:37.819364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:53:30.855089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836454+0000","last_change":"2024-02-19T14:54:37.831523+0000","last_active":"2024-02-19T14:54:37.836454+0000","last_peered":"2024-02-19T14:54:37.836454+0000","last_clean":"2024-02-19T14:54:37.836454+0000","last_became_active":"2024-02-19T14:54:37.831272+0000","last_became_peered":"2024-02-19T14:54:37.831272+0000","last_unstale":"2024-02-19T14:54:37.836454+0000","last_undegraded":"2024-02-19T14:54:37.836454+0000","last_fullsized":"2024-02-19T14:54:37.836454+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:54.496496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820662+0000","last_change":"2024-02-19T14:54:26.741087+0000","last_active":"2024-02-19T14:54:37.820662+0000","last_peered":"2024-02-19T14:54:37.820662+0000","last_clean":"2024-02-19T14:54:37.820662+0000","last_became_active":"2024-02-19T14:54:26.740795+0000","last_became_peered":"2024-02-19T14:54:26.740795+0000","last_unstale":"2024-02-19T14:54:37.820662+0000","last_undegraded":"2024-02-19T14:54:37.820662+0000","last_fullsized":"2024-02-19T14:54:37.820662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:38:50.652136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844390+0000","last_change":"2024-02-19T14:54:37.839043+0000","last_active":"2024-02-19T14:54:37.844390+0000","last_peered":"2024-02-19T14:54:37.844390+0000","last_clean":"2024-02-19T14:54:37.844390+0000","last_became_active":"2024-02-19T14:54:37.838128+0000","last_became_peered":"2024-02-19T14:54:37.838128+0000","last_unstale":"2024-02-19T14:54:37.844390+0000","last_undegraded":"2024-02-19T14:54:37.844390+0000","last_fullsized":"2024-02-19T14:54:37.844390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:48.384374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.829864+0000","last_change":"2024-02-19T14:54:26.753214+0000","last_active":"2024-02-19T14:54:37.829864+0000","last_peered":"2024-02-19T14:54:37.829864+0000","last_clean":"2024-02-19T14:54:37.829864+0000","last_became_active":"2024-02-19T14:54:26.752817+0000","last_became_peered":"2024-02-19T14:54:26.752817+0000","last_unstale":"2024-02-19T14:54:37.829864+0000","last_undegraded":"2024-02-19T14:54:37.829864+0000","last_fullsized":"2024-02-19T14:54:37.829864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836972+0000","last_change":"2024-02-19T14:54:37.831625+0000","last_active":"2024-02-19T14:54:37.836972+0000","last_peered":"2024-02-19T14:54:37.836972+0000","last_clean":"2024-02-19T14:54:37.836972+0000","last_became_active":"2024-02-19T14:54:37.831376+0000","last_became_peered":"2024-02-19T14:54:37.831376+0000","last_unstale":"2024-02-19T14:54:37.836972+0000","last_undegraded":"2024-02-19T14:54:37.836972+0000","last_fullsized":"2024-02-19T14:54:37.836972+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:41.610814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822342+0000","last_change":"2024-02-19T14:54:26.752439+0000","last_active":"2024-02-19T14:54:37.822342+0000","last_peered":"2024-02-19T14:54:37.822342+0000","last_clean":"2024-02-19T14:54:37.822342+0000","last_became_active":"2024-02-19T14:54:26.752104+0000","last_became_peered":"2024-02-19T14:54:26.752104+0000","last_unstale":"2024-02-19T14:54:37.822342+0000","last_undegraded":"2024-02-19T14:54:37.822342+0000","last_fullsized":"2024-02-19T14:54:37.822342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:29.434729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838842+0000","last_change":"2024-02-19T14:54:37.833202+0000","last_active":"2024-02-19T14:54:37.838842+0000","last_peered":"2024-02-19T14:54:37.838842+0000","last_clean":"2024-02-19T14:54:37.838842+0000","last_became_active":"2024-02-19T14:54:37.832943+0000","last_became_peered":"2024-02-19T14:54:37.832943+0000","last_unstale":"2024-02-19T14:54:37.838842+0000","last_undegraded":"2024-02-19T14:54:37.838842+0000","last_fullsized":"2024-02-19T14:54:37.838842+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:51.437501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815465+0000","last_change":"2024-02-19T14:54:26.746983+0000","last_active":"2024-02-19T14:54:37.815465+0000","last_peered":"2024-02-19T14:54:37.815465+0000","last_clean":"2024-02-19T14:54:37.815465+0000","last_became_active":"2024-02-19T14:54:26.746664+0000","last_became_peered":"2024-02-19T14:54:26.746664+0000","last_unstale":"2024-02-19T14:54:37.815465+0000","last_undegraded":"2024-02-19T14:54:37.815465+0000","last_fullsized":"2024-02-19T14:54:37.815465+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:27.405179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845627+0000","last_change":"2024-02-19T14:54:37.837657+0000","last_active":"2024-02-19T14:54:37.845627+0000","last_peered":"2024-02-19T14:54:37.845627+0000","last_clean":"2024-02-19T14:54:37.845627+0000","last_became_active":"2024-02-19T14:54:37.837050+0000","last_became_peered":"2024-02-19T14:54:37.837050+0000","last_unstale":"2024-02-19T14:54:37.845627+0000","last_undegraded":"2024-02-19T14:54:37.845627+0000","last_fullsized":"2024-02-19T14:54:37.845627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:46:45.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819956+0000","last_change":"2024-02-19T14:54:26.744541+0000","last_active":"2024-02-19T14:54:37.819956+0000","last_peered":"2024-02-19T14:54:37.819956+0000","last_clean":"2024-02-19T14:54:37.819956+0000","last_became_active":"2024-02-19T14:54:26.744242+0000","last_became_peered":"2024-02-19T14:54:26.744242+0000","last_unstale":"2024-02-19T14:54:37.819956+0000","last_undegraded":"2024-02-19T14:54:37.819956+0000","last_fullsized":"2024-02-19T14:54:37.819956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840428+0000","last_change":"2024-02-19T14:54:37.834913+0000","last_active":"2024-02-19T14:54:37.840428+0000","last_peered":"2024-02-19T14:54:37.840428+0000","last_clean":"2024-02-19T14:54:37.840428+0000","last_became_active":"2024-02-19T14:54:37.834674+0000","last_became_peered":"2024-02-19T14:54:37.834674+0000","last_unstale":"2024-02-19T14:54:37.840428+0000","last_undegraded":"2024-02-19T14:54:37.840428+0000","last_fullsized":"2024-02-19T14:54:37.840428+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:15.913328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826352+0000","last_change":"2024-02-19T14:54:26.745146+0000","last_active":"2024-02-19T14:54:37.826352+0000","last_peered":"2024-02-19T14:54:37.826352+0000","last_clean":"2024-02-19T14:54:37.826352+0000","last_became_active":"2024-02-19T14:54:26.744342+0000","last_became_peered":"2024-02-19T14:54:26.744342+0000","last_unstale":"2024-02-19T14:54:37.826352+0000","last_undegraded":"2024-02-19T14:54:37.826352+0000","last_fullsized":"2024-02-19T14:54:37.826352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843692+0000","last_change":"2024-02-19T14:54:37.836998+0000","last_active":"2024-02-19T14:54:37.843692+0000","last_peered":"2024-02-19T14:54:37.843692+0000","last_clean":"2024-02-19T14:54:37.843692+0000","last_became_active":"2024-02-19T14:54:37.835908+0000","last_became_peered":"2024-02-19T14:54:37.835908+0000","last_unstale":"2024-02-19T14:54:37.843692+0000","last_undegraded":"2024-02-19T14:54:37.843692+0000","last_fullsized":"2024-02-19T14:54:37.843692+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:43.244541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821197+0000","last_change":"2024-02-19T14:54:26.743966+0000","last_active":"2024-02-19T14:54:37.821197+0000","last_peered":"2024-02-19T14:54:37.821197+0000","last_clean":"2024-02-19T14:54:37.821197+0000","last_became_active":"2024-02-19T14:54:26.743693+0000","last_became_peered":"2024-02-19T14:54:26.743693+0000","last_unstale":"2024-02-19T14:54:37.821197+0000","last_undegraded":"2024-02-19T14:54:37.821197+0000","last_fullsized":"2024-02-19T14:54:37.821197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839625+0000","last_change":"2024-02-19T14:54:37.833579+0000","last_active":"2024-02-19T14:54:37.839625+0000","last_peered":"2024-02-19T14:54:37.839625+0000","last_clean":"2024-02-19T14:54:37.839625+0000","last_became_active":"2024-02-19T14:54:37.833293+0000","last_became_peered":"2024-02-19T14:54:37.833293+0000","last_unstale":"2024-02-19T14:54:37.839625+0000","last_undegraded":"2024-02-19T14:54:37.839625+0000","last_fullsized":"2024-02-19T14:54:37.839625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:14.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814856+0000","last_change":"2024-02-19T14:54:26.747599+0000","last_active":"2024-02-19T14:54:37.814856+0000","last_peered":"2024-02-19T14:54:37.814856+0000","last_clean":"2024-02-19T14:54:37.814856+0000","last_became_active":"2024-02-19T14:54:26.747345+0000","last_became_peered":"2024-02-19T14:54:26.747345+0000","last_unstale":"2024-02-19T14:54:37.814856+0000","last_undegraded":"2024-02-19T14:54:37.814856+0000","last_fullsized":"2024-02-19T14:54:37.814856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:20:30.359001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840582+0000","last_change":"2024-02-19T14:54:37.833641+0000","last_active":"2024-02-19T14:54:37.840582+0000","last_peered":"2024-02-19T14:54:37.840582+0000","last_clean":"2024-02-19T14:54:37.840582+0000","last_became_active":"2024-02-19T14:54:37.833394+0000","last_became_peered":"2024-02-19T14:54:37.833394+0000","last_unstale":"2024-02-19T14:54:37.840582+0000","last_undegraded":"2024-02-19T14:54:37.840582+0000","last_fullsized":"2024-02-19T14:54:37.840582+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:40.525372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822151+0000","last_change":"2024-02-19T14:54:26.736611+0000","last_active":"2024-02-19T14:54:37.822151+0000","last_peered":"2024-02-19T14:54:37.822151+0000","last_clean":"2024-02-19T14:54:37.822151+0000","last_became_active":"2024-02-19T14:54:26.736284+0000","last_became_peered":"2024-02-19T14:54:26.736284+0000","last_unstale":"2024-02-19T14:54:37.822151+0000","last_undegraded":"2024-02-19T14:54:37.822151+0000","last_fullsized":"2024-02-19T14:54:37.822151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:15.218606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845610+0000","last_change":"2024-02-19T14:54:37.837888+0000","last_active":"2024-02-19T14:54:37.845610+0000","last_peered":"2024-02-19T14:54:37.845610+0000","last_clean":"2024-02-19T14:54:37.845610+0000","last_became_active":"2024-02-19T14:54:37.836846+0000","last_became_peered":"2024-02-19T14:54:37.836846+0000","last_unstale":"2024-02-19T14:54:37.845610+0000","last_undegraded":"2024-02-19T14:54:37.845610+0000","last_fullsized":"2024-02-19T14:54:37.845610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:56:31.705336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819882+0000","last_change":"2024-02-19T14:54:26.749665+0000","last_active":"2024-02-19T14:54:37.819882+0000","last_peered":"2024-02-19T14:54:37.819882+0000","last_clean":"2024-02-19T14:54:37.819882+0000","last_became_active":"2024-02-19T14:54:26.749397+0000","last_became_peered":"2024-02-19T14:54:26.749397+0000","last_unstale":"2024-02-19T14:54:37.819882+0000","last_undegraded":"2024-02-19T14:54:37.819882+0000","last_fullsized":"2024-02-19T14:54:37.819882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842456+0000","last_change":"2024-02-19T14:54:37.836287+0000","last_active":"2024-02-19T14:54:37.842456+0000","last_peered":"2024-02-19T14:54:37.842456+0000","last_clean":"2024-02-19T14:54:37.842456+0000","last_became_active":"2024-02-19T14:54:37.835722+0000","last_became_peered":"2024-02-19T14:54:37.835722+0000","last_unstale":"2024-02-19T14:54:37.842456+0000","last_undegraded":"2024-02-19T14:54:37.842456+0000","last_fullsized":"2024-02-19T14:54:37.842456+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:20.338152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815123+0000","last_change":"2024-02-19T14:54:26.747148+0000","last_active":"2024-02-19T14:54:37.815123+0000","last_peered":"2024-02-19T14:54:37.815123+0000","last_clean":"2024-02-19T14:54:37.815123+0000","last_became_active":"2024-02-19T14:54:26.746685+0000","last_became_peered":"2024-02-19T14:54:26.746685+0000","last_unstale":"2024-02-19T14:54:37.815123+0000","last_undegraded":"2024-02-19T14:54:37.815123+0000","last_fullsized":"2024-02-19T14:54:37.815123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:44:34.240945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841117+0000","last_change":"2024-02-19T14:54:37.836254+0000","last_active":"2024-02-19T14:54:37.841117+0000","last_peered":"2024-02-19T14:54:37.841117+0000","last_clean":"2024-02-19T14:54:37.841117+0000","last_became_active":"2024-02-19T14:54:37.835938+0000","last_became_peered":"2024-02-19T14:54:37.835938+0000","last_unstale":"2024-02-19T14:54:37.841117+0000","last_undegraded":"2024-02-19T14:54:37.841117+0000","last_fullsized":"2024-02-19T14:54:37.841117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:49.644151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827142+0000","last_change":"2024-02-19T14:54:26.738144+0000","last_active":"2024-02-19T14:54:37.827142+0000","last_peered":"2024-02-19T14:54:37.827142+0000","last_clean":"2024-02-19T14:54:37.827142+0000","last_became_active":"2024-02-19T14:54:26.737816+0000","last_became_peered":"2024-02-19T14:54:26.737816+0000","last_unstale":"2024-02-19T14:54:37.827142+0000","last_undegraded":"2024-02-19T14:54:37.827142+0000","last_fullsized":"2024-02-19T14:54:37.827142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:53.244189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.834020+0000","last_change":"2024-02-19T14:54:37.830128+0000","last_active":"2024-02-19T14:54:37.834020+0000","last_peered":"2024-02-19T14:54:37.834020+0000","last_clean":"2024-02-19T14:54:37.834020+0000","last_became_active":"2024-02-19T14:54:37.829171+0000","last_became_peered":"2024-02-19T14:54:37.829171+0000","last_unstale":"2024-02-19T14:54:37.834020+0000","last_undegraded":"2024-02-19T14:54:37.834020+0000","last_fullsized":"2024-02-19T14:54:37.834020+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:52:53.015938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815123+0000","last_change":"2024-02-19T14:54:26.752317+0000","last_active":"2024-02-19T14:54:37.815123+0000","last_peered":"2024-02-19T14:54:37.815123+0000","last_clean":"2024-02-19T14:54:37.815123+0000","last_became_active":"2024-02-19T14:54:26.752031+0000","last_became_peered":"2024-02-19T14:54:26.752031+0000","last_unstale":"2024-02-19T14:54:37.815123+0000","last_undegraded":"2024-02-19T14:54:37.815123+0000","last_fullsized":"2024-02-19T14:54:37.815123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:56:39.758301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843570+0000","last_change":"2024-02-19T14:54:37.835043+0000","last_active":"2024-02-19T14:54:37.843570+0000","last_peered":"2024-02-19T14:54:37.843570+0000","last_clean":"2024-02-19T14:54:37.843570+0000","last_became_active":"2024-02-19T14:54:37.834163+0000","last_became_peered":"2024-02-19T14:54:37.834163+0000","last_unstale":"2024-02-19T14:54:37.843570+0000","last_undegraded":"2024-02-19T14:54:37.843570+0000","last_fullsized":"2024-02-19T14:54:37.843570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:38.903266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822655+0000","last_change":"2024-02-19T14:54:26.733087+0000","last_active":"2024-02-19T14:54:37.822655+0000","last_peered":"2024-02-19T14:54:37.822655+0000","last_clean":"2024-02-19T14:54:37.822655+0000","last_became_active":"2024-02-19T14:54:26.732793+0000","last_became_peered":"2024-02-19T14:54:26.732793+0000","last_unstale":"2024-02-19T14:54:37.822655+0000","last_undegraded":"2024-02-19T14:54:37.822655+0000","last_fullsized":"2024-02-19T14:54:37.822655+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:13.700400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840039+0000","last_change":"2024-02-19T14:54:37.834374+0000","last_active":"2024-02-19T14:54:37.840039+0000","last_peered":"2024-02-19T14:54:37.840039+0000","last_clean":"2024-02-19T14:54:37.840039+0000","last_became_active":"2024-02-19T14:54:37.834083+0000","last_became_peered":"2024-02-19T14:54:37.834083+0000","last_unstale":"2024-02-19T14:54:37.840039+0000","last_undegraded":"2024-02-19T14:54:37.840039+0000","last_fullsized":"2024-02-19T14:54:37.840039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:19.115262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826474+0000","last_change":"2024-02-19T14:54:26.743065+0000","last_active":"2024-02-19T14:54:37.826474+0000","last_peered":"2024-02-19T14:54:37.826474+0000","last_clean":"2024-02-19T14:54:37.826474+0000","last_became_active":"2024-02-19T14:54:26.742623+0000","last_became_peered":"2024-02-19T14:54:26.742623+0000","last_unstale":"2024-02-19T14:54:37.826474+0000","last_undegraded":"2024-02-19T14:54:37.826474+0000","last_fullsized":"2024-02-19T14:54:37.826474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:29.434729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836805+0000","last_change":"2024-02-19T14:54:37.830200+0000","last_active":"2024-02-19T14:54:37.836805+0000","last_peered":"2024-02-19T14:54:37.836805+0000","last_clean":"2024-02-19T14:54:37.836805+0000","last_became_active":"2024-02-19T14:54:37.829933+0000","last_became_peered":"2024-02-19T14:54:37.829933+0000","last_unstale":"2024-02-19T14:54:37.836805+0000","last_undegraded":"2024-02-19T14:54:37.836805+0000","last_fullsized":"2024-02-19T14:54:37.836805+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:56.496103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.829041+0000","last_change":"2024-02-19T14:54:26.752403+0000","last_active":"2024-02-19T14:54:37.829041+0000","last_peered":"2024-02-19T14:54:37.829041+0000","last_clean":"2024-02-19T14:54:37.829041+0000","last_became_active":"2024-02-19T14:54:26.752116+0000","last_became_peered":"2024-02-19T14:54:26.752116+0000","last_unstale":"2024-02-19T14:54:37.829041+0000","last_undegraded":"2024-02-19T14:54:37.829041+0000","last_fullsized":"2024-02-19T14:54:37.829041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:34:33.181295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838823+0000","last_change":"2024-02-19T14:54:37.833103+0000","last_active":"2024-02-19T14:54:37.838823+0000","last_peered":"2024-02-19T14:54:37.838823+0000","last_clean":"2024-02-19T14:54:37.838823+0000","last_became_active":"2024-02-19T14:54:37.832498+0000","last_became_peered":"2024-02-19T14:54:37.832498+0000","last_unstale":"2024-02-19T14:54:37.838823+0000","last_undegraded":"2024-02-19T14:54:37.838823+0000","last_fullsized":"2024-02-19T14:54:37.838823+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:01.742780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820829+0000","last_change":"2024-02-19T14:54:26.752358+0000","last_active":"2024-02-19T14:54:37.820829+0000","last_peered":"2024-02-19T14:54:37.820829+0000","last_clean":"2024-02-19T14:54:37.820829+0000","last_became_active":"2024-02-19T14:54:26.752076+0000","last_became_peered":"2024-02-19T14:54:26.752076+0000","last_unstale":"2024-02-19T14:54:37.820829+0000","last_undegraded":"2024-02-19T14:54:37.820829+0000","last_fullsized":"2024-02-19T14:54:37.820829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:40.346858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.837267+0000","last_change":"2024-02-19T14:54:37.830152+0000","last_active":"2024-02-19T14:54:37.837267+0000","last_peered":"2024-02-19T14:54:37.837267+0000","last_clean":"2024-02-19T14:54:37.837267+0000","last_became_active":"2024-02-19T14:54:37.829865+0000","last_became_peered":"2024-02-19T14:54:37.829865+0000","last_unstale":"2024-02-19T14:54:37.837267+0000","last_undegraded":"2024-02-19T14:54:37.837267+0000","last_fullsized":"2024-02-19T14:54:37.837267+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:16.681755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821887+0000","last_change":"2024-02-19T14:54:26.743117+0000","last_active":"2024-02-19T14:54:37.821887+0000","last_peered":"2024-02-19T14:54:37.821887+0000","last_clean":"2024-02-19T14:54:37.821887+0000","last_became_active":"2024-02-19T14:54:26.742845+0000","last_became_peered":"2024-02-19T14:54:26.742845+0000","last_unstale":"2024-02-19T14:54:37.821887+0000","last_undegraded":"2024-02-19T14:54:37.821887+0000","last_fullsized":"2024-02-19T14:54:37.821887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843627+0000","last_change":"2024-02-19T14:54:37.835305+0000","last_active":"2024-02-19T14:54:37.843627+0000","last_peered":"2024-02-19T14:54:37.843627+0000","last_clean":"2024-02-19T14:54:37.843627+0000","last_became_active":"2024-02-19T14:54:37.834979+0000","last_became_peered":"2024-02-19T14:54:37.834979+0000","last_unstale":"2024-02-19T14:54:37.843627+0000","last_undegraded":"2024-02-19T14:54:37.843627+0000","last_fullsized":"2024-02-19T14:54:37.843627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:19.530070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814983+0000","last_change":"2024-02-19T14:54:26.746821+0000","last_active":"2024-02-19T14:54:37.814983+0000","last_peered":"2024-02-19T14:54:37.814983+0000","last_clean":"2024-02-19T14:54:37.814983+0000","last_became_active":"2024-02-19T14:54:26.746408+0000","last_became_peered":"2024-02-19T14:54:26.746408+0000","last_unstale":"2024-02-19T14:54:37.814983+0000","last_undegraded":"2024-02-19T14:54:37.814983+0000","last_fullsized":"2024-02-19T14:54:37.814983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:23:38.456834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835347+0000","last_change":"2024-02-19T14:54:37.829118+0000","last_active":"2024-02-19T14:54:37.835347+0000","last_peered":"2024-02-19T14:54:37.835347+0000","last_clean":"2024-02-19T14:54:37.835347+0000","last_became_active":"2024-02-19T14:54:37.827474+0000","last_became_peered":"2024-02-19T14:54:37.827474+0000","last_unstale":"2024-02-19T14:54:37.835347+0000","last_undegraded":"2024-02-19T14:54:37.835347+0000","last_fullsized":"2024-02-19T14:54:37.835347+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:18:51.236995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827682+0000","last_change":"2024-02-19T14:54:26.745851+0000","last_active":"2024-02-19T14:54:37.827682+0000","last_peered":"2024-02-19T14:54:37.827682+0000","last_clean":"2024-02-19T14:54:37.827682+0000","last_became_active":"2024-02-19T14:54:26.745587+0000","last_became_peered":"2024-02-19T14:54:26.745587+0000","last_unstale":"2024-02-19T14:54:37.827682+0000","last_undegraded":"2024-02-19T14:54:37.827682+0000","last_fullsized":"2024-02-19T14:54:37.827682+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845645+0000","last_change":"2024-02-19T14:54:37.837776+0000","last_active":"2024-02-19T14:54:37.845645+0000","last_peered":"2024-02-19T14:54:37.845645+0000","last_clean":"2024-02-19T14:54:37.845645+0000","last_became_active":"2024-02-19T14:54:37.837082+0000","last_became_peered":"2024-02-19T14:54:37.837082+0000","last_unstale":"2024-02-19T14:54:37.845645+0000","last_undegraded":"2024-02-19T14:54:37.845645+0000","last_fullsized":"2024-02-19T14:54:37.845645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:25.979876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821464+0000","last_change":"2024-02-19T14:54:26.758958+0000","last_active":"2024-02-19T14:54:37.821464+0000","last_peered":"2024-02-19T14:54:37.821464+0000","last_clean":"2024-02-19T14:54:37.821464+0000","last_became_active":"2024-02-19T14:54:26.758649+0000","last_became_peered":"2024-02-19T14:54:26.758649+0000","last_unstale":"2024-02-19T14:54:37.821464+0000","last_undegraded":"2024-02-19T14:54:37.821464+0000","last_fullsized":"2024-02-19T14:54:37.821464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:38.553508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835089+0000","last_change":"2024-02-19T14:54:37.833083+0000","last_active":"2024-02-19T14:54:37.835089+0000","last_peered":"2024-02-19T14:54:37.835089+0000","last_clean":"2024-02-19T14:54:37.835089+0000","last_became_active":"2024-02-19T14:54:37.832783+0000","last_became_peered":"2024-02-19T14:54:37.832783+0000","last_unstale":"2024-02-19T14:54:37.835089+0000","last_undegraded":"2024-02-19T14:54:37.835089+0000","last_fullsized":"2024-02-19T14:54:37.835089+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:25.266694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820135+0000","last_change":"2024-02-19T14:54:26.742392+0000","last_active":"2024-02-19T14:54:37.820135+0000","last_peered":"2024-02-19T14:54:37.820135+0000","last_clean":"2024-02-19T14:54:37.820135+0000","last_became_active":"2024-02-19T14:54:26.742129+0000","last_became_peered":"2024-02-19T14:54:26.742129+0000","last_unstale":"2024-02-19T14:54:37.820135+0000","last_undegraded":"2024-02-19T14:54:37.820135+0000","last_fullsized":"2024-02-19T14:54:37.820135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:50.140868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843726+0000","last_change":"2024-02-19T14:54:37.836933+0000","last_active":"2024-02-19T14:54:37.843726+0000","last_peered":"2024-02-19T14:54:37.843726+0000","last_clean":"2024-02-19T14:54:37.843726+0000","last_became_active":"2024-02-19T14:54:37.835463+0000","last_became_peered":"2024-02-19T14:54:37.835463+0000","last_unstale":"2024-02-19T14:54:37.843726+0000","last_undegraded":"2024-02-19T14:54:37.843726+0000","last_fullsized":"2024-02-19T14:54:37.843726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:01.742780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821307+0000","last_change":"2024-02-19T14:54:26.747016+0000","last_active":"2024-02-19T14:54:37.821307+0000","last_peered":"2024-02-19T14:54:37.821307+0000","last_clean":"2024-02-19T14:54:37.821307+0000","last_became_active":"2024-02-19T14:54:26.746761+0000","last_became_peered":"2024-02-19T14:54:26.746761+0000","last_unstale":"2024-02-19T14:54:37.821307+0000","last_undegraded":"2024-02-19T14:54:37.821307+0000","last_fullsized":"2024-02-19T14:54:37.821307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.847341+0000","last_change":"2024-02-19T14:54:37.846531+0000","last_active":"2024-02-19T14:54:37.847341+0000","last_peered":"2024-02-19T14:54:37.847341+0000","last_clean":"2024-02-19T14:54:37.847341+0000","last_became_active":"2024-02-19T14:54:37.846257+0000","last_became_peered":"2024-02-19T14:54:37.846257+0000","last_unstale":"2024-02-19T14:54:37.847341+0000","last_undegraded":"2024-02-19T14:54:37.847341+0000","last_fullsized":"2024-02-19T14:54:37.847341+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:20.081130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819416+0000","last_change":"2024-02-19T14:54:36.826134+0000","last_active":"2024-02-19T14:54:37.819416+0000","last_peered":"2024-02-19T14:54:37.819416+0000","last_clean":"2024-02-19T14:54:37.819416+0000","last_became_active":"2024-02-19T14:54:36.825871+0000","last_became_peered":"2024-02-19T14:54:36.825871+0000","last_unstale":"2024-02-19T14:54:37.819416+0000","last_undegraded":"2024-02-19T14:54:37.819416+0000","last_fullsized":"2024-02-19T14:54:37.819416+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:15.677220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819026+0000","last_change":"2024-02-19T14:54:26.736520+0000","last_active":"2024-02-19T14:54:37.819026+0000","last_peered":"2024-02-19T14:54:37.819026+0000","last_clean":"2024-02-19T14:54:37.819026+0000","last_became_active":"2024-02-19T14:54:26.736235+0000","last_became_peered":"2024-02-19T14:54:26.736235+0000","last_unstale":"2024-02-19T14:54:37.819026+0000","last_undegraded":"2024-02-19T14:54:37.819026+0000","last_fullsized":"2024-02-19T14:54:37.819026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:16.908530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"74'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815144+0000","last_change":"2024-02-19T14:54:25.736715+0000","last_active":"2024-02-19T14:54:37.815144+0000","last_peered":"2024-02-19T14:54:37.815144+0000","last_clean":"2024-02-19T14:54:37.815144+0000","last_became_active":"2024-02-19T14:54:25.736266+0000","last_became_peered":"2024-02-19T14:54:25.736266+0000","last_unstale":"2024-02-19T14:54:37.815144+0000","last_undegraded":"2024-02-19T14:54:37.815144+0000","last_fullsized":"2024-02-19T14:54:37.815144+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:29.148205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840407+0000","last_change":"2024-02-19T14:54:37.834149+0000","last_active":"2024-02-19T14:54:37.840407+0000","last_peered":"2024-02-19T14:54:37.840407+0000","last_clean":"2024-02-19T14:54:37.840407+0000","last_became_active":"2024-02-19T14:54:37.833917+0000","last_became_peered":"2024-02-19T14:54:37.833917+0000","last_unstale":"2024-02-19T14:54:37.840407+0000","last_undegraded":"2024-02-19T14:54:37.840407+0000","last_fullsized":"2024-02-19T14:54:37.840407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:39.422646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821303+0000","last_change":"2024-02-19T14:54:36.833581+0000","last_active":"2024-02-19T14:54:37.821303+0000","last_peered":"2024-02-19T14:54:37.821303+0000","last_clean":"2024-02-19T14:54:37.821303+0000","last_became_active":"2024-02-19T14:54:36.833276+0000","last_became_peered":"2024-02-19T14:54:36.833276+0000","last_unstale":"2024-02-19T14:54:37.821303+0000","last_undegraded":"2024-02-19T14:54:37.821303+0000","last_fullsized":"2024-02-19T14:54:37.821303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:58.062735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827040+0000","last_change":"2024-02-19T14:54:26.745742+0000","last_active":"2024-02-19T14:54:37.827040+0000","last_peered":"2024-02-19T14:54:37.827040+0000","last_clean":"2024-02-19T14:54:37.827040+0000","last_became_active":"2024-02-19T14:54:26.745478+0000","last_became_peered":"2024-02-19T14:54:26.745478+0000","last_unstale":"2024-02-19T14:54:37.827040+0000","last_undegraded":"2024-02-19T14:54:37.827040+0000","last_fullsized":"2024-02-19T14:54:37.827040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:04.822685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822319+0000","last_change":"2024-02-19T14:54:25.737577+0000","last_active":"2024-02-19T14:54:37.822319+0000","last_peered":"2024-02-19T14:54:37.822319+0000","last_clean":"2024-02-19T14:54:37.822319+0000","last_became_active":"2024-02-19T14:54:25.737301+0000","last_became_peered":"2024-02-19T14:54:25.737301+0000","last_unstale":"2024-02-19T14:54:37.822319+0000","last_undegraded":"2024-02-19T14:54:37.822319+0000","last_fullsized":"2024-02-19T14:54:37.822319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:47.328330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839625+0000","last_change":"2024-02-19T14:54:37.833409+0000","last_active":"2024-02-19T14:54:37.839625+0000","last_peered":"2024-02-19T14:54:37.839625+0000","last_clean":"2024-02-19T14:54:37.839625+0000","last_became_active":"2024-02-19T14:54:37.833141+0000","last_became_peered":"2024-02-19T14:54:37.833141+0000","last_unstale":"2024-02-19T14:54:37.839625+0000","last_undegraded":"2024-02-19T14:54:37.839625+0000","last_fullsized":"2024-02-19T14:54:37.839625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:48.384374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840195+0000","last_change":"2024-02-19T14:54:36.825186+0000","last_active":"2024-02-19T14:54:37.840195+0000","last_peered":"2024-02-19T14:54:37.840195+0000","last_clean":"2024-02-19T14:54:37.840195+0000","last_became_active":"2024-02-19T14:54:36.824892+0000","last_became_peered":"2024-02-19T14:54:36.824892+0000","last_unstale":"2024-02-19T14:54:37.840195+0000","last_undegraded":"2024-02-19T14:54:37.840195+0000","last_fullsized":"2024-02-19T14:54:37.840195+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:41.852675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825860+0000","last_change":"2024-02-19T14:54:26.744292+0000","last_active":"2024-02-19T14:54:37.825860+0000","last_peered":"2024-02-19T14:54:37.825860+0000","last_clean":"2024-02-19T14:54:37.825860+0000","last_became_active":"2024-02-19T14:54:26.743866+0000","last_became_peered":"2024-02-19T14:54:26.743866+0000","last_unstale":"2024-02-19T14:54:37.825860+0000","last_undegraded":"2024-02-19T14:54:37.825860+0000","last_fullsized":"2024-02-19T14:54:37.825860+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:21:10.873300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821993+0000","last_change":"2024-02-19T14:54:25.739812+0000","last_active":"2024-02-19T14:54:37.821993+0000","last_peered":"2024-02-19T14:54:37.821993+0000","last_clean":"2024-02-19T14:54:37.821993+0000","last_became_active":"2024-02-19T14:54:25.739538+0000","last_became_peered":"2024-02-19T14:54:25.739538+0000","last_unstale":"2024-02-19T14:54:37.821993+0000","last_undegraded":"2024-02-19T14:54:37.821993+0000","last_fullsized":"2024-02-19T14:54:37.821993+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:02.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843641+0000","last_change":"2024-02-19T14:54:37.838138+0000","last_active":"2024-02-19T14:54:37.843641+0000","last_peered":"2024-02-19T14:54:37.843641+0000","last_clean":"2024-02-19T14:54:37.843641+0000","last_became_active":"2024-02-19T14:54:37.837860+0000","last_became_peered":"2024-02-19T14:54:37.837860+0000","last_unstale":"2024-02-19T14:54:37.843641+0000","last_undegraded":"2024-02-19T14:54:37.843641+0000","last_fullsized":"2024-02-19T14:54:37.843641+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:14.069764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840466+0000","last_change":"2024-02-19T14:54:36.826883+0000","last_active":"2024-02-19T14:54:37.840466+0000","last_peered":"2024-02-19T14:54:37.840466+0000","last_clean":"2024-02-19T14:54:37.840466+0000","last_became_active":"2024-02-19T14:54:36.826635+0000","last_became_peered":"2024-02-19T14:54:36.826635+0000","last_unstale":"2024-02-19T14:54:37.840466+0000","last_undegraded":"2024-02-19T14:54:37.840466+0000","last_fullsized":"2024-02-19T14:54:37.840466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:42.240006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827253+0000","last_change":"2024-02-19T14:54:26.738473+0000","last_active":"2024-02-19T14:54:37.827253+0000","last_peered":"2024-02-19T14:54:37.827253+0000","last_clean":"2024-02-19T14:54:37.827253+0000","last_became_active":"2024-02-19T14:54:26.738077+0000","last_became_peered":"2024-02-19T14:54:26.738077+0000","last_unstale":"2024-02-19T14:54:37.827253+0000","last_undegraded":"2024-02-19T14:54:37.827253+0000","last_fullsized":"2024-02-19T14:54:37.827253+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:26:27.849069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819592+0000","last_change":"2024-02-19T14:54:25.733935+0000","last_active":"2024-02-19T14:54:37.819592+0000","last_peered":"2024-02-19T14:54:37.819592+0000","last_clean":"2024-02-19T14:54:37.819592+0000","last_became_active":"2024-02-19T14:54:25.733653+0000","last_became_peered":"2024-02-19T14:54:25.733653+0000","last_unstale":"2024-02-19T14:54:37.819592+0000","last_undegraded":"2024-02-19T14:54:37.819592+0000","last_fullsized":"2024-02-19T14:54:37.819592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:52.238124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836108+0000","last_change":"2024-02-19T14:54:37.833743+0000","last_active":"2024-02-19T14:54:37.836108+0000","last_peered":"2024-02-19T14:54:37.836108+0000","last_clean":"2024-02-19T14:54:37.836108+0000","last_became_active":"2024-02-19T14:54:37.833437+0000","last_became_peered":"2024-02-19T14:54:37.833437+0000","last_unstale":"2024-02-19T14:54:37.836108+0000","last_undegraded":"2024-02-19T14:54:37.836108+0000","last_fullsized":"2024-02-19T14:54:37.836108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:22:25.979876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838077+0000","last_change":"2024-02-19T14:54:36.828034+0000","last_active":"2024-02-19T14:54:37.838077+0000","last_peered":"2024-02-19T14:54:37.838077+0000","last_clean":"2024-02-19T14:54:37.838077+0000","last_became_active":"2024-02-19T14:54:36.827707+0000","last_became_peered":"2024-02-19T14:54:36.827707+0000","last_unstale":"2024-02-19T14:54:37.838077+0000","last_undegraded":"2024-02-19T14:54:37.838077+0000","last_fullsized":"2024-02-19T14:54:37.838077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:36:41.952775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.828858+0000","last_change":"2024-02-19T14:54:26.754225+0000","last_active":"2024-02-19T14:54:37.828858+0000","last_peered":"2024-02-19T14:54:37.828858+0000","last_clean":"2024-02-19T14:54:37.828858+0000","last_became_active":"2024-02-19T14:54:26.753892+0000","last_became_peered":"2024-02-19T14:54:26.753892+0000","last_unstale":"2024-02-19T14:54:37.828858+0000","last_undegraded":"2024-02-19T14:54:37.828858+0000","last_fullsized":"2024-02-19T14:54:37.828858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:02.979120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827443+0000","last_change":"2024-02-19T14:54:25.733786+0000","last_active":"2024-02-19T14:54:37.827443+0000","last_peered":"2024-02-19T14:54:37.827443+0000","last_clean":"2024-02-19T14:54:37.827443+0000","last_became_active":"2024-02-19T14:54:25.733327+0000","last_became_peered":"2024-02-19T14:54:25.733327+0000","last_unstale":"2024-02-19T14:54:37.827443+0000","last_undegraded":"2024-02-19T14:54:37.827443+0000","last_fullsized":"2024-02-19T14:54:37.827443+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:58:13.186581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844546+0000","last_change":"2024-02-19T14:54:37.834983+0000","last_active":"2024-02-19T14:54:37.844546+0000","last_peered":"2024-02-19T14:54:37.844546+0000","last_clean":"2024-02-19T14:54:37.844546+0000","last_became_active":"2024-02-19T14:54:37.833764+0000","last_became_peered":"2024-02-19T14:54:37.833764+0000","last_unstale":"2024-02-19T14:54:37.844546+0000","last_undegraded":"2024-02-19T14:54:37.844546+0000","last_fullsized":"2024-02-19T14:54:37.844546+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:59:43.474020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842738+0000","last_change":"2024-02-19T14:54:36.826847+0000","last_active":"2024-02-19T14:54:37.842738+0000","last_peered":"2024-02-19T14:54:37.842738+0000","last_clean":"2024-02-19T14:54:37.842738+0000","last_became_active":"2024-02-19T14:54:36.826575+0000","last_became_peered":"2024-02-19T14:54:36.826575+0000","last_unstale":"2024-02-19T14:54:37.842738+0000","last_undegraded":"2024-02-19T14:54:37.842738+0000","last_fullsized":"2024-02-19T14:54:37.842738+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:14.908793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.829978+0000","last_change":"2024-02-19T14:54:26.753347+0000","last_active":"2024-02-19T14:54:37.829978+0000","last_peered":"2024-02-19T14:54:37.829978+0000","last_clean":"2024-02-19T14:54:37.829978+0000","last_became_active":"2024-02-19T14:54:26.752855+0000","last_became_peered":"2024-02-19T14:54:26.752855+0000","last_unstale":"2024-02-19T14:54:37.829978+0000","last_undegraded":"2024-02-19T14:54:37.829978+0000","last_fullsized":"2024-02-19T14:54:37.829978+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:48.880118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823925+0000","last_change":"2024-02-19T14:54:25.730846+0000","last_active":"2024-02-19T14:54:37.823925+0000","last_peered":"2024-02-19T14:54:37.823925+0000","last_clean":"2024-02-19T14:54:37.823925+0000","last_became_active":"2024-02-19T14:54:25.730539+0000","last_became_peered":"2024-02-19T14:54:25.730539+0000","last_unstale":"2024-02-19T14:54:37.823925+0000","last_undegraded":"2024-02-19T14:54:37.823925+0000","last_fullsized":"2024-02-19T14:54:37.823925+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:56.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846501+0000","last_change":"2024-02-19T14:54:37.845595+0000","last_active":"2024-02-19T14:54:37.846501+0000","last_peered":"2024-02-19T14:54:37.846501+0000","last_clean":"2024-02-19T14:54:37.846501+0000","last_became_active":"2024-02-19T14:54:37.844986+0000","last_became_peered":"2024-02-19T14:54:37.844986+0000","last_unstale":"2024-02-19T14:54:37.846501+0000","last_undegraded":"2024-02-19T14:54:37.846501+0000","last_fullsized":"2024-02-19T14:54:37.846501+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:22.391301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"78'4","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.847633+0000","last_change":"2024-02-19T14:54:36.835389+0000","last_active":"2024-02-19T14:54:37.847633+0000","last_peered":"2024-02-19T14:54:37.847633+0000","last_clean":"2024-02-19T14:54:37.847633+0000","last_became_active":"2024-02-19T14:54:36.835045+0000","last_became_peered":"2024-02-19T14:54:36.835045+0000","last_unstale":"2024-02-19T14:54:37.847633+0000","last_undegraded":"2024-02-19T14:54:37.847633+0000","last_fullsized":"2024-02-19T14:54:37.847633+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:23.576155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826951+0000","last_change":"2024-02-19T14:54:26.741073+0000","last_active":"2024-02-19T14:54:37.826951+0000","last_peered":"2024-02-19T14:54:37.826951+0000","last_clean":"2024-02-19T14:54:37.826951+0000","last_became_active":"2024-02-19T14:54:26.740765+0000","last_became_peered":"2024-02-19T14:54:26.740765+0000","last_unstale":"2024-02-19T14:54:37.826951+0000","last_undegraded":"2024-02-19T14:54:37.826951+0000","last_fullsized":"2024-02-19T14:54:37.826951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:57.622666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827147+0000","last_change":"2024-02-19T14:54:25.737703+0000","last_active":"2024-02-19T14:54:37.827147+0000","last_peered":"2024-02-19T14:54:37.827147+0000","last_clean":"2024-02-19T14:54:37.827147+0000","last_became_active":"2024-02-19T14:54:25.737444+0000","last_became_peered":"2024-02-19T14:54:25.737444+0000","last_unstale":"2024-02-19T14:54:37.827147+0000","last_undegraded":"2024-02-19T14:54:37.827147+0000","last_fullsized":"2024-02-19T14:54:37.827147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:26.488134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845117+0000","last_change":"2024-02-19T14:54:37.837242+0000","last_active":"2024-02-19T14:54:37.845117+0000","last_peered":"2024-02-19T14:54:37.845117+0000","last_clean":"2024-02-19T14:54:37.845117+0000","last_became_active":"2024-02-19T14:54:37.836969+0000","last_became_peered":"2024-02-19T14:54:37.836969+0000","last_unstale":"2024-02-19T14:54:37.845117+0000","last_undegraded":"2024-02-19T14:54:37.845117+0000","last_fullsized":"2024-02-19T14:54:37.845117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:25.266694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839556+0000","last_change":"2024-02-19T14:54:36.827013+0000","last_active":"2024-02-19T14:54:37.839556+0000","last_peered":"2024-02-19T14:54:37.839556+0000","last_clean":"2024-02-19T14:54:37.839556+0000","last_became_active":"2024-02-19T14:54:36.825537+0000","last_became_peered":"2024-02-19T14:54:36.825537+0000","last_unstale":"2024-02-19T14:54:37.839556+0000","last_undegraded":"2024-02-19T14:54:37.839556+0000","last_fullsized":"2024-02-19T14:54:37.839556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:36:07.513713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820081+0000","last_change":"2024-02-19T14:54:26.751885+0000","last_active":"2024-02-19T14:54:37.820081+0000","last_peered":"2024-02-19T14:54:37.820081+0000","last_clean":"2024-02-19T14:54:37.820081+0000","last_became_active":"2024-02-19T14:54:26.751496+0000","last_became_peered":"2024-02-19T14:54:26.751496+0000","last_unstale":"2024-02-19T14:54:37.820081+0000","last_undegraded":"2024-02-19T14:54:37.820081+0000","last_fullsized":"2024-02-19T14:54:37.820081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:57.427605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814741+0000","last_change":"2024-02-19T14:54:25.736306+0000","last_active":"2024-02-19T14:54:37.814741+0000","last_peered":"2024-02-19T14:54:37.814741+0000","last_clean":"2024-02-19T14:54:37.814741+0000","last_became_active":"2024-02-19T14:54:25.736029+0000","last_became_peered":"2024-02-19T14:54:25.736029+0000","last_unstale":"2024-02-19T14:54:37.814741+0000","last_undegraded":"2024-02-19T14:54:37.814741+0000","last_fullsized":"2024-02-19T14:54:37.814741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:59:00.456190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836255+0000","last_change":"2024-02-19T14:54:37.829194+0000","last_active":"2024-02-19T14:54:37.836255+0000","last_peered":"2024-02-19T14:54:37.836255+0000","last_clean":"2024-02-19T14:54:37.836255+0000","last_became_active":"2024-02-19T14:54:37.828948+0000","last_became_peered":"2024-02-19T14:54:37.828948+0000","last_unstale":"2024-02-19T14:54:37.836255+0000","last_undegraded":"2024-02-19T14:54:37.836255+0000","last_fullsized":"2024-02-19T14:54:37.836255+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:04:57.782852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820033+0000","last_change":"2024-02-19T14:54:36.824354+0000","last_active":"2024-02-19T14:54:37.820033+0000","last_peered":"2024-02-19T14:54:37.820033+0000","last_clean":"2024-02-19T14:54:37.820033+0000","last_became_active":"2024-02-19T14:54:36.824099+0000","last_became_peered":"2024-02-19T14:54:36.824099+0000","last_unstale":"2024-02-19T14:54:37.820033+0000","last_undegraded":"2024-02-19T14:54:37.820033+0000","last_fullsized":"2024-02-19T14:54:37.820033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:37.898731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826564+0000","last_change":"2024-02-19T14:54:26.743807+0000","last_active":"2024-02-19T14:54:37.826564+0000","last_peered":"2024-02-19T14:54:37.826564+0000","last_clean":"2024-02-19T14:54:37.826564+0000","last_became_active":"2024-02-19T14:54:26.743493+0000","last_became_peered":"2024-02-19T14:54:26.743493+0000","last_unstale":"2024-02-19T14:54:37.826564+0000","last_undegraded":"2024-02-19T14:54:37.826564+0000","last_fullsized":"2024-02-19T14:54:37.826564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:38.383508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823296+0000","last_change":"2024-02-19T14:54:25.737199+0000","last_active":"2024-02-19T14:54:37.823296+0000","last_peered":"2024-02-19T14:54:37.823296+0000","last_clean":"2024-02-19T14:54:37.823296+0000","last_became_active":"2024-02-19T14:54:25.736941+0000","last_became_peered":"2024-02-19T14:54:25.736941+0000","last_unstale":"2024-02-19T14:54:37.823296+0000","last_undegraded":"2024-02-19T14:54:37.823296+0000","last_fullsized":"2024-02-19T14:54:37.823296+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:41:26.806558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841043+0000","last_change":"2024-02-19T14:54:37.836750+0000","last_active":"2024-02-19T14:54:37.841043+0000","last_peered":"2024-02-19T14:54:37.841043+0000","last_clean":"2024-02-19T14:54:37.841043+0000","last_became_active":"2024-02-19T14:54:37.836439+0000","last_became_peered":"2024-02-19T14:54:37.836439+0000","last_unstale":"2024-02-19T14:54:37.841043+0000","last_undegraded":"2024-02-19T14:54:37.841043+0000","last_fullsized":"2024-02-19T14:54:37.841043+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:59.645392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"78'8","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845053+0000","last_change":"2024-02-19T14:54:36.825524+0000","last_active":"2024-02-19T14:54:37.845053+0000","last_peered":"2024-02-19T14:54:37.845053+0000","last_clean":"2024-02-19T14:54:37.845053+0000","last_became_active":"2024-02-19T14:54:36.825254+0000","last_became_peered":"2024-02-19T14:54:36.825254+0000","last_unstale":"2024-02-19T14:54:37.845053+0000","last_undegraded":"2024-02-19T14:54:37.845053+0000","last_fullsized":"2024-02-19T14:54:37.845053+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:18.110727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820337+0000","last_change":"2024-02-19T14:54:26.733706+0000","last_active":"2024-02-19T14:54:37.820337+0000","last_peered":"2024-02-19T14:54:37.820337+0000","last_clean":"2024-02-19T14:54:37.820337+0000","last_became_active":"2024-02-19T14:54:26.733434+0000","last_became_peered":"2024-02-19T14:54:26.733434+0000","last_unstale":"2024-02-19T14:54:37.820337+0000","last_undegraded":"2024-02-19T14:54:37.820337+0000","last_fullsized":"2024-02-19T14:54:37.820337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:32.153897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820923+0000","last_change":"2024-02-19T14:54:25.729258+0000","last_active":"2024-02-19T14:54:37.820923+0000","last_peered":"2024-02-19T14:54:37.820923+0000","last_clean":"2024-02-19T14:54:37.820923+0000","last_became_active":"2024-02-19T14:54:25.728966+0000","last_became_peered":"2024-02-19T14:54:25.728966+0000","last_unstale":"2024-02-19T14:54:37.820923+0000","last_undegraded":"2024-02-19T14:54:37.820923+0000","last_fullsized":"2024-02-19T14:54:37.820923+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:41.651494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840605+0000","last_change":"2024-02-19T14:54:37.835912+0000","last_active":"2024-02-19T14:54:37.840605+0000","last_peered":"2024-02-19T14:54:37.840605+0000","last_clean":"2024-02-19T14:54:37.840605+0000","last_became_active":"2024-02-19T14:54:37.835316+0000","last_became_peered":"2024-02-19T14:54:37.835316+0000","last_unstale":"2024-02-19T14:54:37.840605+0000","last_undegraded":"2024-02-19T14:54:37.840605+0000","last_fullsized":"2024-02-19T14:54:37.840605+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:42.659206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846048+0000","last_change":"2024-02-19T14:54:36.834272+0000","last_active":"2024-02-19T14:54:37.846048+0000","last_peered":"2024-02-19T14:54:37.846048+0000","last_clean":"2024-02-19T14:54:37.846048+0000","last_became_active":"2024-02-19T14:54:36.833964+0000","last_became_peered":"2024-02-19T14:54:36.833964+0000","last_unstale":"2024-02-19T14:54:37.846048+0000","last_undegraded":"2024-02-19T14:54:37.846048+0000","last_fullsized":"2024-02-19T14:54:37.846048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:04.331873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820041+0000","last_change":"2024-02-19T14:54:26.741110+0000","last_active":"2024-02-19T14:54:37.820041+0000","last_peered":"2024-02-19T14:54:37.820041+0000","last_clean":"2024-02-19T14:54:37.820041+0000","last_became_active":"2024-02-19T14:54:26.740831+0000","last_became_peered":"2024-02-19T14:54:26.740831+0000","last_unstale":"2024-02-19T14:54:37.820041+0000","last_undegraded":"2024-02-19T14:54:37.820041+0000","last_fullsized":"2024-02-19T14:54:37.820041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:07.902437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"74'3","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815884+0000","last_change":"2024-02-19T14:54:25.732654+0000","last_active":"2024-02-19T14:54:37.815884+0000","last_peered":"2024-02-19T14:54:37.815884+0000","last_clean":"2024-02-19T14:54:37.815884+0000","last_became_active":"2024-02-19T14:54:25.732388+0000","last_became_peered":"2024-02-19T14:54:25.732388+0000","last_unstale":"2024-02-19T14:54:37.815884+0000","last_undegraded":"2024-02-19T14:54:37.815884+0000","last_fullsized":"2024-02-19T14:54:37.815884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:16:04.044350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839503+0000","last_change":"2024-02-19T14:54:37.831656+0000","last_active":"2024-02-19T14:54:37.839503+0000","last_peered":"2024-02-19T14:54:37.839503+0000","last_clean":"2024-02-19T14:54:37.839503+0000","last_became_active":"2024-02-19T14:54:37.831331+0000","last_became_peered":"2024-02-19T14:54:37.831331+0000","last_unstale":"2024-02-19T14:54:37.839503+0000","last_undegraded":"2024-02-19T14:54:37.839503+0000","last_fullsized":"2024-02-19T14:54:37.839503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:53.748202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.828306+0000","last_change":"2024-02-19T14:54:36.833090+0000","last_active":"2024-02-19T14:54:37.828306+0000","last_peered":"2024-02-19T14:54:37.828306+0000","last_clean":"2024-02-19T14:54:37.828306+0000","last_became_active":"2024-02-19T14:54:36.832830+0000","last_became_peered":"2024-02-19T14:54:36.832830+0000","last_unstale":"2024-02-19T14:54:37.828306+0000","last_undegraded":"2024-02-19T14:54:37.828306+0000","last_fullsized":"2024-02-19T14:54:37.828306+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:04.331873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821189+0000","last_change":"2024-02-19T14:54:26.749111+0000","last_active":"2024-02-19T14:54:37.821189+0000","last_peered":"2024-02-19T14:54:37.821189+0000","last_clean":"2024-02-19T14:54:37.821189+0000","last_became_active":"2024-02-19T14:54:26.748830+0000","last_became_peered":"2024-02-19T14:54:26.748830+0000","last_unstale":"2024-02-19T14:54:37.821189+0000","last_undegraded":"2024-02-19T14:54:37.821189+0000","last_fullsized":"2024-02-19T14:54:37.821189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:36.425397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"74'6","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819500+0000","last_change":"2024-02-19T14:54:25.733916+0000","last_active":"2024-02-19T14:54:37.819500+0000","last_peered":"2024-02-19T14:54:37.819500+0000","last_clean":"2024-02-19T14:54:37.819500+0000","last_became_active":"2024-02-19T14:54:25.733638+0000","last_became_peered":"2024-02-19T14:54:25.733638+0000","last_unstale":"2024-02-19T14:54:37.819500+0000","last_undegraded":"2024-02-19T14:54:37.819500+0000","last_fullsized":"2024-02-19T14:54:37.819500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:26:26.843004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842514+0000","last_change":"2024-02-19T14:54:37.836488+0000","last_active":"2024-02-19T14:54:37.842514+0000","last_peered":"2024-02-19T14:54:37.842514+0000","last_clean":"2024-02-19T14:54:37.842514+0000","last_became_active":"2024-02-19T14:54:37.836045+0000","last_became_peered":"2024-02-19T14:54:37.836045+0000","last_unstale":"2024-02-19T14:54:37.842514+0000","last_undegraded":"2024-02-19T14:54:37.842514+0000","last_fullsized":"2024-02-19T14:54:37.842514+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:59.067270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"78'3","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845056+0000","last_change":"2024-02-19T14:54:36.833202+0000","last_active":"2024-02-19T14:54:37.845056+0000","last_peered":"2024-02-19T14:54:37.845056+0000","last_clean":"2024-02-19T14:54:37.845056+0000","last_became_active":"2024-02-19T14:54:36.832869+0000","last_became_peered":"2024-02-19T14:54:36.832869+0000","last_unstale":"2024-02-19T14:54:37.845056+0000","last_undegraded":"2024-02-19T14:54:37.845056+0000","last_fullsized":"2024-02-19T14:54:37.845056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:57.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822435+0000","last_change":"2024-02-19T14:54:26.741929+0000","last_active":"2024-02-19T14:54:37.822435+0000","last_peered":"2024-02-19T14:54:37.822435+0000","last_clean":"2024-02-19T14:54:37.822435+0000","last_became_active":"2024-02-19T14:54:26.741662+0000","last_became_peered":"2024-02-19T14:54:26.741662+0000","last_unstale":"2024-02-19T14:54:37.822435+0000","last_undegraded":"2024-02-19T14:54:37.822435+0000","last_fullsized":"2024-02-19T14:54:37.822435+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821343+0000","last_change":"2024-02-19T14:54:25.731407+0000","last_active":"2024-02-19T14:54:37.821343+0000","last_peered":"2024-02-19T14:54:37.821343+0000","last_clean":"2024-02-19T14:54:37.821343+0000","last_became_active":"2024-02-19T14:54:25.731062+0000","last_became_peered":"2024-02-19T14:54:25.731062+0000","last_unstale":"2024-02-19T14:54:37.821343+0000","last_undegraded":"2024-02-19T14:54:37.821343+0000","last_fullsized":"2024-02-19T14:54:37.821343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:10:30.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.840554+0000","last_change":"2024-02-19T14:54:37.836496+0000","last_active":"2024-02-19T14:54:37.840554+0000","last_peered":"2024-02-19T14:54:37.840554+0000","last_clean":"2024-02-19T14:54:37.840554+0000","last_became_active":"2024-02-19T14:54:37.836206+0000","last_became_peered":"2024-02-19T14:54:37.836206+0000","last_unstale":"2024-02-19T14:54:37.840554+0000","last_undegraded":"2024-02-19T14:54:37.840554+0000","last_fullsized":"2024-02-19T14:54:37.840554+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:47.604862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"78'6","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.849065+0000","last_change":"2024-02-19T14:54:36.833069+0000","last_active":"2024-02-19T14:54:37.849065+0000","last_peered":"2024-02-19T14:54:37.849065+0000","last_clean":"2024-02-19T14:54:37.849065+0000","last_became_active":"2024-02-19T14:54:36.832764+0000","last_became_peered":"2024-02-19T14:54:36.832764+0000","last_unstale":"2024-02-19T14:54:37.849065+0000","last_undegraded":"2024-02-19T14:54:37.849065+0000","last_fullsized":"2024-02-19T14:54:37.849065+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:36:41.952775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819099+0000","last_change":"2024-02-19T14:54:26.739942+0000","last_active":"2024-02-19T14:54:37.819099+0000","last_peered":"2024-02-19T14:54:37.819099+0000","last_clean":"2024-02-19T14:54:37.819099+0000","last_became_active":"2024-02-19T14:54:26.739642+0000","last_became_peered":"2024-02-19T14:54:26.739642+0000","last_unstale":"2024-02-19T14:54:37.819099+0000","last_undegraded":"2024-02-19T14:54:37.819099+0000","last_fullsized":"2024-02-19T14:54:37.819099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:16.908530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821501+0000","last_change":"2024-02-19T14:54:25.731469+0000","last_active":"2024-02-19T14:54:37.821501+0000","last_peered":"2024-02-19T14:54:37.821501+0000","last_clean":"2024-02-19T14:54:37.821501+0000","last_became_active":"2024-02-19T14:54:25.731139+0000","last_became_peered":"2024-02-19T14:54:25.731139+0000","last_unstale":"2024-02-19T14:54:37.821501+0000","last_undegraded":"2024-02-19T14:54:37.821501+0000","last_fullsized":"2024-02-19T14:54:37.821501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:09.115941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.843573+0000","last_change":"2024-02-19T14:54:37.833273+0000","last_active":"2024-02-19T14:54:37.843573+0000","last_peered":"2024-02-19T14:54:37.843573+0000","last_clean":"2024-02-19T14:54:37.843573+0000","last_became_active":"2024-02-19T14:54:37.832642+0000","last_became_peered":"2024-02-19T14:54:37.832642+0000","last_unstale":"2024-02-19T14:54:37.843573+0000","last_undegraded":"2024-02-19T14:54:37.843573+0000","last_fullsized":"2024-02-19T14:54:37.843573+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:21.212649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821801+0000","last_change":"2024-02-19T14:54:36.829199+0000","last_active":"2024-02-19T14:54:37.821801+0000","last_peered":"2024-02-19T14:54:37.821801+0000","last_clean":"2024-02-19T14:54:37.821801+0000","last_became_active":"2024-02-19T14:54:36.828927+0000","last_became_peered":"2024-02-19T14:54:36.828927+0000","last_unstale":"2024-02-19T14:54:37.821801+0000","last_undegraded":"2024-02-19T14:54:37.821801+0000","last_fullsized":"2024-02-19T14:54:37.821801+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:14:53.977545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819347+0000","last_change":"2024-02-19T14:54:26.746776+0000","last_active":"2024-02-19T14:54:37.819347+0000","last_peered":"2024-02-19T14:54:37.819347+0000","last_clean":"2024-02-19T14:54:37.819347+0000","last_became_active":"2024-02-19T14:54:26.746452+0000","last_became_peered":"2024-02-19T14:54:26.746452+0000","last_unstale":"2024-02-19T14:54:37.819347+0000","last_undegraded":"2024-02-19T14:54:37.819347+0000","last_fullsized":"2024-02-19T14:54:37.819347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:05.385960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814479+0000","last_change":"2024-02-19T14:54:25.736476+0000","last_active":"2024-02-19T14:54:37.814479+0000","last_peered":"2024-02-19T14:54:37.814479+0000","last_clean":"2024-02-19T14:54:37.814479+0000","last_became_active":"2024-02-19T14:54:25.736202+0000","last_became_peered":"2024-02-19T14:54:25.736202+0000","last_unstale":"2024-02-19T14:54:37.814479+0000","last_undegraded":"2024-02-19T14:54:37.814479+0000","last_fullsized":"2024-02-19T14:54:37.814479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:05:01.450961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836143+0000","last_change":"2024-02-19T14:54:37.834352+0000","last_active":"2024-02-19T14:54:37.836143+0000","last_peered":"2024-02-19T14:54:37.836143+0000","last_clean":"2024-02-19T14:54:37.836143+0000","last_became_active":"2024-02-19T14:54:37.833482+0000","last_became_peered":"2024-02-19T14:54:37.833482+0000","last_unstale":"2024-02-19T14:54:37.836143+0000","last_undegraded":"2024-02-19T14:54:37.836143+0000","last_fullsized":"2024-02-19T14:54:37.836143+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:56:31.705336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820419+0000","last_change":"2024-02-19T14:54:36.824408+0000","last_active":"2024-02-19T14:54:37.820419+0000","last_peered":"2024-02-19T14:54:37.820419+0000","last_clean":"2024-02-19T14:54:37.820419+0000","last_became_active":"2024-02-19T14:54:36.824166+0000","last_became_peered":"2024-02-19T14:54:36.824166+0000","last_unstale":"2024-02-19T14:54:37.820419+0000","last_undegraded":"2024-02-19T14:54:37.820419+0000","last_fullsized":"2024-02-19T14:54:37.820419+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:39.662911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820687+0000","last_change":"2024-02-19T14:54:26.742073+0000","last_active":"2024-02-19T14:54:37.820687+0000","last_peered":"2024-02-19T14:54:37.820687+0000","last_clean":"2024-02-19T14:54:37.820687+0000","last_became_active":"2024-02-19T14:54:26.741791+0000","last_became_peered":"2024-02-19T14:54:26.741791+0000","last_unstale":"2024-02-19T14:54:37.820687+0000","last_undegraded":"2024-02-19T14:54:37.820687+0000","last_fullsized":"2024-02-19T14:54:37.820687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:42.657558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"74'8","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826661+0000","last_change":"2024-02-19T14:54:25.737048+0000","last_active":"2024-02-19T14:54:37.826661+0000","last_peered":"2024-02-19T14:54:37.826661+0000","last_clean":"2024-02-19T14:54:37.826661+0000","last_became_active":"2024-02-19T14:54:25.736643+0000","last_became_peered":"2024-02-19T14:54:25.736643+0000","last_unstale":"2024-02-19T14:54:37.826661+0000","last_undegraded":"2024-02-19T14:54:37.826661+0000","last_fullsized":"2024-02-19T14:54:37.826661+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:15.902465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835318+0000","last_change":"2024-02-19T14:54:37.830345+0000","last_active":"2024-02-19T14:54:37.835318+0000","last_peered":"2024-02-19T14:54:37.835318+0000","last_clean":"2024-02-19T14:54:37.835318+0000","last_became_active":"2024-02-19T14:54:37.830044+0000","last_became_peered":"2024-02-19T14:54:37.830044+0000","last_unstale":"2024-02-19T14:54:37.835318+0000","last_undegraded":"2024-02-19T14:54:37.835318+0000","last_fullsized":"2024-02-19T14:54:37.835318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.667447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820543+0000","last_change":"2024-02-19T14:54:36.835659+0000","last_active":"2024-02-19T14:54:37.820543+0000","last_peered":"2024-02-19T14:54:37.820543+0000","last_clean":"2024-02-19T14:54:37.820543+0000","last_became_active":"2024-02-19T14:54:36.835365+0000","last_became_peered":"2024-02-19T14:54:36.835365+0000","last_unstale":"2024-02-19T14:54:37.820543+0000","last_undegraded":"2024-02-19T14:54:37.820543+0000","last_fullsized":"2024-02-19T14:54:37.820543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:58.966226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819309+0000","last_change":"2024-02-19T14:54:26.751085+0000","last_active":"2024-02-19T14:54:37.819309+0000","last_peered":"2024-02-19T14:54:37.819309+0000","last_clean":"2024-02-19T14:54:37.819309+0000","last_became_active":"2024-02-19T14:54:26.750808+0000","last_became_peered":"2024-02-19T14:54:26.750808+0000","last_unstale":"2024-02-19T14:54:37.819309+0000","last_undegraded":"2024-02-19T14:54:37.819309+0000","last_fullsized":"2024-02-19T14:54:37.819309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:28.332002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823389+0000","last_change":"2024-02-19T14:54:25.732423+0000","last_active":"2024-02-19T14:54:37.823389+0000","last_peered":"2024-02-19T14:54:37.823389+0000","last_clean":"2024-02-19T14:54:37.823389+0000","last_became_active":"2024-02-19T14:54:25.732159+0000","last_became_peered":"2024-02-19T14:54:25.732159+0000","last_unstale":"2024-02-19T14:54:37.823389+0000","last_undegraded":"2024-02-19T14:54:37.823389+0000","last_fullsized":"2024-02-19T14:54:37.823389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:07.433362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845611+0000","last_change":"2024-02-19T14:54:37.838527+0000","last_active":"2024-02-19T14:54:37.845611+0000","last_peered":"2024-02-19T14:54:37.845611+0000","last_clean":"2024-02-19T14:54:37.845611+0000","last_became_active":"2024-02-19T14:54:37.838279+0000","last_became_peered":"2024-02-19T14:54:37.838279+0000","last_unstale":"2024-02-19T14:54:37.845611+0000","last_undegraded":"2024-02-19T14:54:37.845611+0000","last_fullsized":"2024-02-19T14:54:37.845611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:10.508329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815377+0000","last_change":"2024-02-19T14:54:36.830894+0000","last_active":"2024-02-19T14:54:37.815377+0000","last_peered":"2024-02-19T14:54:37.815377+0000","last_clean":"2024-02-19T14:54:37.815377+0000","last_became_active":"2024-02-19T14:54:36.830485+0000","last_became_peered":"2024-02-19T14:54:36.830485+0000","last_unstale":"2024-02-19T14:54:37.815377+0000","last_undegraded":"2024-02-19T14:54:37.815377+0000","last_fullsized":"2024-02-19T14:54:37.815377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:29:14.156802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814386+0000","last_change":"2024-02-19T14:54:26.747952+0000","last_active":"2024-02-19T14:54:37.814386+0000","last_peered":"2024-02-19T14:54:37.814386+0000","last_clean":"2024-02-19T14:54:37.814386+0000","last_became_active":"2024-02-19T14:54:26.747656+0000","last_became_peered":"2024-02-19T14:54:26.747656+0000","last_unstale":"2024-02-19T14:54:37.814386+0000","last_undegraded":"2024-02-19T14:54:37.814386+0000","last_fullsized":"2024-02-19T14:54:37.814386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:50.690817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827056+0000","last_change":"2024-02-19T14:54:25.733175+0000","last_active":"2024-02-19T14:54:37.827056+0000","last_peered":"2024-02-19T14:54:37.827056+0000","last_clean":"2024-02-19T14:54:37.827056+0000","last_became_active":"2024-02-19T14:54:25.732709+0000","last_became_peered":"2024-02-19T14:54:25.732709+0000","last_unstale":"2024-02-19T14:54:37.827056+0000","last_undegraded":"2024-02-19T14:54:37.827056+0000","last_fullsized":"2024-02-19T14:54:37.827056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:28.981539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846606+0000","last_change":"2024-02-19T14:54:37.842201+0000","last_active":"2024-02-19T14:54:37.846606+0000","last_peered":"2024-02-19T14:54:37.846606+0000","last_clean":"2024-02-19T14:54:37.846606+0000","last_became_active":"2024-02-19T14:54:37.841982+0000","last_became_peered":"2024-02-19T14:54:37.841982+0000","last_unstale":"2024-02-19T14:54:37.846606+0000","last_undegraded":"2024-02-19T14:54:37.846606+0000","last_fullsized":"2024-02-19T14:54:37.846606+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:24:43.283852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842919+0000","last_change":"2024-02-19T14:54:36.829399+0000","last_active":"2024-02-19T14:54:37.842919+0000","last_peered":"2024-02-19T14:54:37.842919+0000","last_clean":"2024-02-19T14:54:37.842919+0000","last_became_active":"2024-02-19T14:54:36.829111+0000","last_became_peered":"2024-02-19T14:54:36.829111+0000","last_unstale":"2024-02-19T14:54:37.842919+0000","last_undegraded":"2024-02-19T14:54:37.842919+0000","last_fullsized":"2024-02-19T14:54:37.842919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:13.065228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814956+0000","last_change":"2024-02-19T14:54:26.756834+0000","last_active":"2024-02-19T14:54:37.814956+0000","last_peered":"2024-02-19T14:54:37.814956+0000","last_clean":"2024-02-19T14:54:37.814956+0000","last_became_active":"2024-02-19T14:54:26.756548+0000","last_became_peered":"2024-02-19T14:54:26.756548+0000","last_unstale":"2024-02-19T14:54:37.814956+0000","last_undegraded":"2024-02-19T14:54:37.814956+0000","last_fullsized":"2024-02-19T14:54:37.814956+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:25.077428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"74'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815986+0000","last_change":"2024-02-19T14:54:25.735865+0000","last_active":"2024-02-19T14:54:37.815986+0000","last_peered":"2024-02-19T14:54:37.815986+0000","last_clean":"2024-02-19T14:54:37.815986+0000","last_became_active":"2024-02-19T14:54:25.735569+0000","last_became_peered":"2024-02-19T14:54:25.735569+0000","last_unstale":"2024-02-19T14:54:37.815986+0000","last_undegraded":"2024-02-19T14:54:37.815986+0000","last_fullsized":"2024-02-19T14:54:37.815986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:55:08.537113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844182+0000","last_change":"2024-02-19T14:54:37.838803+0000","last_active":"2024-02-19T14:54:37.844182+0000","last_peered":"2024-02-19T14:54:37.844182+0000","last_clean":"2024-02-19T14:54:37.844182+0000","last_became_active":"2024-02-19T14:54:37.838530+0000","last_became_peered":"2024-02-19T14:54:37.838530+0000","last_unstale":"2024-02-19T14:54:37.844182+0000","last_undegraded":"2024-02-19T14:54:37.844182+0000","last_fullsized":"2024-02-19T14:54:37.844182+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:20.338152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.814891+0000","last_change":"2024-02-19T14:54:36.827017+0000","last_active":"2024-02-19T14:54:37.814891+0000","last_peered":"2024-02-19T14:54:37.814891+0000","last_clean":"2024-02-19T14:54:37.814891+0000","last_became_active":"2024-02-19T14:54:36.826763+0000","last_became_peered":"2024-02-19T14:54:36.826763+0000","last_unstale":"2024-02-19T14:54:37.814891+0000","last_undegraded":"2024-02-19T14:54:37.814891+0000","last_fullsized":"2024-02-19T14:54:37.814891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:18.749797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819439+0000","last_change":"2024-02-19T14:54:26.750291+0000","last_active":"2024-02-19T14:54:37.819439+0000","last_peered":"2024-02-19T14:54:37.819439+0000","last_clean":"2024-02-19T14:54:37.819439+0000","last_became_active":"2024-02-19T14:54:26.750040+0000","last_became_peered":"2024-02-19T14:54:26.750040+0000","last_unstale":"2024-02-19T14:54:37.819439+0000","last_undegraded":"2024-02-19T14:54:37.819439+0000","last_fullsized":"2024-02-19T14:54:37.819439+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:31.568562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822288+0000","last_change":"2024-02-19T14:54:25.731254+0000","last_active":"2024-02-19T14:54:37.822288+0000","last_peered":"2024-02-19T14:54:37.822288+0000","last_clean":"2024-02-19T14:54:37.822288+0000","last_became_active":"2024-02-19T14:54:25.730971+0000","last_became_peered":"2024-02-19T14:54:25.730971+0000","last_unstale":"2024-02-19T14:54:37.822288+0000","last_undegraded":"2024-02-19T14:54:37.822288+0000","last_fullsized":"2024-02-19T14:54:37.822288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:07:52.238124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844446+0000","last_change":"2024-02-19T14:54:37.841599+0000","last_active":"2024-02-19T14:54:37.844446+0000","last_peered":"2024-02-19T14:54:37.844446+0000","last_clean":"2024-02-19T14:54:37.844446+0000","last_became_active":"2024-02-19T14:54:37.840362+0000","last_became_peered":"2024-02-19T14:54:37.840362+0000","last_unstale":"2024-02-19T14:54:37.844446+0000","last_undegraded":"2024-02-19T14:54:37.844446+0000","last_fullsized":"2024-02-19T14:54:37.844446+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:34:40.525372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842579+0000","last_change":"2024-02-19T14:54:36.818412+0000","last_active":"2024-02-19T14:54:37.842579+0000","last_peered":"2024-02-19T14:54:37.842579+0000","last_clean":"2024-02-19T14:54:37.842579+0000","last_became_active":"2024-02-19T14:54:36.818117+0000","last_became_peered":"2024-02-19T14:54:36.818117+0000","last_unstale":"2024-02-19T14:54:37.842579+0000","last_undegraded":"2024-02-19T14:54:37.842579+0000","last_fullsized":"2024-02-19T14:54:37.842579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:31:26.994638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.828964+0000","last_change":"2024-02-19T14:54:26.746449+0000","last_active":"2024-02-19T14:54:37.828964+0000","last_peered":"2024-02-19T14:54:37.828964+0000","last_clean":"2024-02-19T14:54:37.828964+0000","last_became_active":"2024-02-19T14:54:26.746208+0000","last_became_peered":"2024-02-19T14:54:26.746208+0000","last_unstale":"2024-02-19T14:54:37.828964+0000","last_undegraded":"2024-02-19T14:54:37.828964+0000","last_fullsized":"2024-02-19T14:54:37.828964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:30:38.553508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819531+0000","last_change":"2024-02-19T14:54:25.734037+0000","last_active":"2024-02-19T14:54:37.819531+0000","last_peered":"2024-02-19T14:54:37.819531+0000","last_clean":"2024-02-19T14:54:37.819531+0000","last_became_active":"2024-02-19T14:54:25.733736+0000","last_became_peered":"2024-02-19T14:54:25.733736+0000","last_unstale":"2024-02-19T14:54:37.819531+0000","last_undegraded":"2024-02-19T14:54:37.819531+0000","last_fullsized":"2024-02-19T14:54:37.819531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:02.628447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846479+0000","last_change":"2024-02-19T14:54:37.845443+0000","last_active":"2024-02-19T14:54:37.846479+0000","last_peered":"2024-02-19T14:54:37.846479+0000","last_clean":"2024-02-19T14:54:37.846479+0000","last_became_active":"2024-02-19T14:54:37.844776+0000","last_became_peered":"2024-02-19T14:54:37.844776+0000","last_unstale":"2024-02-19T14:54:37.846479+0000","last_undegraded":"2024-02-19T14:54:37.846479+0000","last_fullsized":"2024-02-19T14:54:37.846479+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:14.227823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"78'1","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.839429+0000","last_change":"2024-02-19T14:54:36.827091+0000","last_active":"2024-02-19T14:54:37.839429+0000","last_peered":"2024-02-19T14:54:37.839429+0000","last_clean":"2024-02-19T14:54:37.839429+0000","last_became_active":"2024-02-19T14:54:36.826794+0000","last_became_peered":"2024-02-19T14:54:36.826794+0000","last_unstale":"2024-02-19T14:54:37.839429+0000","last_undegraded":"2024-02-19T14:54:37.839429+0000","last_fullsized":"2024-02-19T14:54:37.839429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:46.600327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.831192+0000","last_change":"2024-02-19T14:54:26.749731+0000","last_active":"2024-02-19T14:54:37.831192+0000","last_peered":"2024-02-19T14:54:37.831192+0000","last_clean":"2024-02-19T14:54:37.831192+0000","last_became_active":"2024-02-19T14:54:26.749338+0000","last_became_peered":"2024-02-19T14:54:26.749338+0000","last_unstale":"2024-02-19T14:54:37.831192+0000","last_undegraded":"2024-02-19T14:54:37.831192+0000","last_fullsized":"2024-02-19T14:54:37.831192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:14:43.891437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819640+0000","last_change":"2024-02-19T14:54:25.737967+0000","last_active":"2024-02-19T14:54:37.819640+0000","last_peered":"2024-02-19T14:54:37.819640+0000","last_clean":"2024-02-19T14:54:37.819640+0000","last_became_active":"2024-02-19T14:54:25.737526+0000","last_became_peered":"2024-02-19T14:54:25.737526+0000","last_unstale":"2024-02-19T14:54:37.819640+0000","last_undegraded":"2024-02-19T14:54:37.819640+0000","last_fullsized":"2024-02-19T14:54:37.819640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:09.115941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.835844+0000","last_change":"2024-02-19T14:54:37.830541+0000","last_active":"2024-02-19T14:54:37.835844+0000","last_peered":"2024-02-19T14:54:37.835844+0000","last_clean":"2024-02-19T14:54:37.835844+0000","last_became_active":"2024-02-19T14:54:37.830250+0000","last_became_peered":"2024-02-19T14:54:37.830250+0000","last_unstale":"2024-02-19T14:54:37.835844+0000","last_undegraded":"2024-02-19T14:54:37.835844+0000","last_fullsized":"2024-02-19T14:54:37.835844+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:18.993081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844414+0000","last_change":"2024-02-19T14:54:36.830901+0000","last_active":"2024-02-19T14:54:37.844414+0000","last_peered":"2024-02-19T14:54:37.844414+0000","last_clean":"2024-02-19T14:54:37.844414+0000","last_became_active":"2024-02-19T14:54:36.830625+0000","last_became_peered":"2024-02-19T14:54:36.830625+0000","last_unstale":"2024-02-19T14:54:37.844414+0000","last_undegraded":"2024-02-19T14:54:37.844414+0000","last_fullsized":"2024-02-19T14:54:37.844414+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:59:42.469485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820290+0000","last_change":"2024-02-19T14:54:26.743403+0000","last_active":"2024-02-19T14:54:37.820290+0000","last_peered":"2024-02-19T14:54:37.820290+0000","last_clean":"2024-02-19T14:54:37.820290+0000","last_became_active":"2024-02-19T14:54:26.742380+0000","last_became_peered":"2024-02-19T14:54:26.742380+0000","last_unstale":"2024-02-19T14:54:37.820290+0000","last_undegraded":"2024-02-19T14:54:37.820290+0000","last_fullsized":"2024-02-19T14:54:37.820290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:48.334394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819693+0000","last_change":"2024-02-19T14:54:25.737893+0000","last_active":"2024-02-19T14:54:37.819693+0000","last_peered":"2024-02-19T14:54:37.819693+0000","last_clean":"2024-02-19T14:54:37.819693+0000","last_became_active":"2024-02-19T14:54:25.737503+0000","last_became_peered":"2024-02-19T14:54:25.737503+0000","last_unstale":"2024-02-19T14:54:37.819693+0000","last_undegraded":"2024-02-19T14:54:37.819693+0000","last_fullsized":"2024-02-19T14:54:37.819693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:04:47.328330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841233+0000","last_change":"2024-02-19T14:54:37.837307+0000","last_active":"2024-02-19T14:54:37.841233+0000","last_peered":"2024-02-19T14:54:37.841233+0000","last_clean":"2024-02-19T14:54:37.841233+0000","last_became_active":"2024-02-19T14:54:37.836597+0000","last_became_peered":"2024-02-19T14:54:37.836597+0000","last_unstale":"2024-02-19T14:54:37.841233+0000","last_undegraded":"2024-02-19T14:54:37.841233+0000","last_fullsized":"2024-02-19T14:54:37.841233+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:05:59.970761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817173+0000","last_change":"2024-02-19T14:54:36.827233+0000","last_active":"2024-02-19T14:54:37.817173+0000","last_peered":"2024-02-19T14:54:37.817173+0000","last_clean":"2024-02-19T14:54:37.817173+0000","last_became_active":"2024-02-19T14:54:36.826930+0000","last_became_peered":"2024-02-19T14:54:36.826930+0000","last_unstale":"2024-02-19T14:54:37.817173+0000","last_undegraded":"2024-02-19T14:54:37.817173+0000","last_fullsized":"2024-02-19T14:54:37.817173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:38.418111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819106+0000","last_change":"2024-02-19T14:54:26.743767+0000","last_active":"2024-02-19T14:54:37.819106+0000","last_peered":"2024-02-19T14:54:37.819106+0000","last_clean":"2024-02-19T14:54:37.819106+0000","last_became_active":"2024-02-19T14:54:26.743477+0000","last_became_peered":"2024-02-19T14:54:26.743477+0000","last_unstale":"2024-02-19T14:54:37.819106+0000","last_undegraded":"2024-02-19T14:54:37.819106+0000","last_fullsized":"2024-02-19T14:54:37.819106+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:51:48.554748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"74'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821916+0000","last_change":"2024-02-19T14:54:25.729454+0000","last_active":"2024-02-19T14:54:37.821916+0000","last_peered":"2024-02-19T14:54:37.821916+0000","last_clean":"2024-02-19T14:54:37.821916+0000","last_became_active":"2024-02-19T14:54:25.729052+0000","last_became_peered":"2024-02-19T14:54:25.729052+0000","last_unstale":"2024-02-19T14:54:37.821916+0000","last_undegraded":"2024-02-19T14:54:37.821916+0000","last_fullsized":"2024-02-19T14:54:37.821916+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:04.379896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.844503+0000","last_change":"2024-02-19T14:54:37.836034+0000","last_active":"2024-02-19T14:54:37.844503+0000","last_peered":"2024-02-19T14:54:37.844503+0000","last_clean":"2024-02-19T14:54:37.844503+0000","last_became_active":"2024-02-19T14:54:37.835762+0000","last_became_peered":"2024-02-19T14:54:37.835762+0000","last_unstale":"2024-02-19T14:54:37.844503+0000","last_undegraded":"2024-02-19T14:54:37.844503+0000","last_fullsized":"2024-02-19T14:54:37.844503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:43.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821032+0000","last_change":"2024-02-19T14:54:36.834792+0000","last_active":"2024-02-19T14:54:37.821032+0000","last_peered":"2024-02-19T14:54:37.821032+0000","last_clean":"2024-02-19T14:54:37.821032+0000","last_became_active":"2024-02-19T14:54:36.834516+0000","last_became_peered":"2024-02-19T14:54:36.834516+0000","last_unstale":"2024-02-19T14:54:37.821032+0000","last_undegraded":"2024-02-19T14:54:37.821032+0000","last_fullsized":"2024-02-19T14:54:37.821032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:19.333617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817540+0000","last_change":"2024-02-19T14:54:26.742043+0000","last_active":"2024-02-19T14:54:37.817540+0000","last_peered":"2024-02-19T14:54:37.817540+0000","last_clean":"2024-02-19T14:54:37.817540+0000","last_became_active":"2024-02-19T14:54:26.741725+0000","last_became_peered":"2024-02-19T14:54:26.741725+0000","last_unstale":"2024-02-19T14:54:37.817540+0000","last_undegraded":"2024-02-19T14:54:37.817540+0000","last_fullsized":"2024-02-19T14:54:37.817540+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:10.122005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.832069+0000","last_change":"2024-02-19T14:54:25.737062+0000","last_active":"2024-02-19T14:54:37.832069+0000","last_peered":"2024-02-19T14:54:37.832069+0000","last_clean":"2024-02-19T14:54:37.832069+0000","last_became_active":"2024-02-19T14:54:25.736783+0000","last_became_peered":"2024-02-19T14:54:25.736783+0000","last_unstale":"2024-02-19T14:54:37.832069+0000","last_undegraded":"2024-02-19T14:54:37.832069+0000","last_fullsized":"2024-02-19T14:54:37.832069+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:35:56.421540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.836483+0000","last_change":"2024-02-19T14:54:37.833286+0000","last_active":"2024-02-19T14:54:37.836483+0000","last_peered":"2024-02-19T14:54:37.836483+0000","last_clean":"2024-02-19T14:54:37.836483+0000","last_became_active":"2024-02-19T14:54:37.833025+0000","last_became_peered":"2024-02-19T14:54:37.833025+0000","last_unstale":"2024-02-19T14:54:37.836483+0000","last_undegraded":"2024-02-19T14:54:37.836483+0000","last_fullsized":"2024-02-19T14:54:37.836483+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:00.935158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827550+0000","last_change":"2024-02-19T14:54:36.834415+0000","last_active":"2024-02-19T14:54:37.827550+0000","last_peered":"2024-02-19T14:54:37.827550+0000","last_clean":"2024-02-19T14:54:37.827550+0000","last_became_active":"2024-02-19T14:54:36.833594+0000","last_became_peered":"2024-02-19T14:54:36.833594+0000","last_unstale":"2024-02-19T14:54:37.827550+0000","last_undegraded":"2024-02-19T14:54:37.827550+0000","last_fullsized":"2024-02-19T14:54:37.827550+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:08:58.062735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825987+0000","last_change":"2024-02-19T14:54:26.744132+0000","last_active":"2024-02-19T14:54:37.825987+0000","last_peered":"2024-02-19T14:54:37.825987+0000","last_clean":"2024-02-19T14:54:37.825987+0000","last_became_active":"2024-02-19T14:54:26.741010+0000","last_became_peered":"2024-02-19T14:54:26.741010+0000","last_unstale":"2024-02-19T14:54:37.825987+0000","last_undegraded":"2024-02-19T14:54:37.825987+0000","last_fullsized":"2024-02-19T14:54:37.825987+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:04.497808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825519+0000","last_change":"2024-02-19T14:54:25.736615+0000","last_active":"2024-02-19T14:54:37.825519+0000","last_peered":"2024-02-19T14:54:37.825519+0000","last_clean":"2024-02-19T14:54:37.825519+0000","last_became_active":"2024-02-19T14:54:25.736328+0000","last_became_peered":"2024-02-19T14:54:25.736328+0000","last_unstale":"2024-02-19T14:54:37.825519+0000","last_undegraded":"2024-02-19T14:54:37.825519+0000","last_fullsized":"2024-02-19T14:54:37.825519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:07.018554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.842345+0000","last_change":"2024-02-19T14:54:37.834319+0000","last_active":"2024-02-19T14:54:37.842345+0000","last_peered":"2024-02-19T14:54:37.842345+0000","last_clean":"2024-02-19T14:54:37.842345+0000","last_became_active":"2024-02-19T14:54:37.834070+0000","last_became_peered":"2024-02-19T14:54:37.834070+0000","last_unstale":"2024-02-19T14:54:37.842345+0000","last_undegraded":"2024-02-19T14:54:37.842345+0000","last_fullsized":"2024-02-19T14:54:37.842345+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:48.394105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.823853+0000","last_change":"2024-02-19T14:54:36.830086+0000","last_active":"2024-02-19T14:54:37.823853+0000","last_peered":"2024-02-19T14:54:37.823853+0000","last_clean":"2024-02-19T14:54:37.823853+0000","last_became_active":"2024-02-19T14:54:36.829795+0000","last_became_peered":"2024-02-19T14:54:36.829795+0000","last_unstale":"2024-02-19T14:54:37.823853+0000","last_undegraded":"2024-02-19T14:54:37.823853+0000","last_fullsized":"2024-02-19T14:54:37.823853+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:48.469617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.827768+0000","last_change":"2024-02-19T14:54:26.743310+0000","last_active":"2024-02-19T14:54:37.827768+0000","last_peered":"2024-02-19T14:54:37.827768+0000","last_clean":"2024-02-19T14:54:37.827768+0000","last_became_active":"2024-02-19T14:54:26.742887+0000","last_became_peered":"2024-02-19T14:54:26.742887+0000","last_unstale":"2024-02-19T14:54:37.827768+0000","last_undegraded":"2024-02-19T14:54:37.827768+0000","last_fullsized":"2024-02-19T14:54:37.827768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:13.490047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821737+0000","last_change":"2024-02-19T14:54:25.736615+0000","last_active":"2024-02-19T14:54:37.821737+0000","last_peered":"2024-02-19T14:54:37.821737+0000","last_clean":"2024-02-19T14:54:37.821737+0000","last_became_active":"2024-02-19T14:54:25.736314+0000","last_became_peered":"2024-02-19T14:54:25.736314+0000","last_unstale":"2024-02-19T14:54:37.821737+0000","last_undegraded":"2024-02-19T14:54:37.821737+0000","last_fullsized":"2024-02-19T14:54:37.821737+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:41.651494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.845963+0000","last_change":"2024-02-19T14:54:37.841997+0000","last_active":"2024-02-19T14:54:37.845963+0000","last_peered":"2024-02-19T14:54:37.845963+0000","last_clean":"2024-02-19T14:54:37.845963+0000","last_became_active":"2024-02-19T14:54:37.841668+0000","last_became_peered":"2024-02-19T14:54:37.841668+0000","last_unstale":"2024-02-19T14:54:37.845963+0000","last_undegraded":"2024-02-19T14:54:37.845963+0000","last_fullsized":"2024-02-19T14:54:37.845963+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:14:40.611498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.824452+0000","last_change":"2024-02-19T14:54:36.833066+0000","last_active":"2024-02-19T14:54:37.824452+0000","last_peered":"2024-02-19T14:54:37.824452+0000","last_clean":"2024-02-19T14:54:37.824452+0000","last_became_active":"2024-02-19T14:54:36.832815+0000","last_became_peered":"2024-02-19T14:54:36.832815+0000","last_unstale":"2024-02-19T14:54:37.824452+0000","last_undegraded":"2024-02-19T14:54:37.824452+0000","last_fullsized":"2024-02-19T14:54:37.824452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:37.580307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819373+0000","last_change":"2024-02-19T14:54:26.742237+0000","last_active":"2024-02-19T14:54:37.819373+0000","last_peered":"2024-02-19T14:54:37.819373+0000","last_clean":"2024-02-19T14:54:37.819373+0000","last_became_active":"2024-02-19T14:54:26.741929+0000","last_became_peered":"2024-02-19T14:54:26.741929+0000","last_unstale":"2024-02-19T14:54:37.819373+0000","last_undegraded":"2024-02-19T14:54:37.819373+0000","last_fullsized":"2024-02-19T14:54:37.819373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:47.414910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.818743+0000","last_change":"2024-02-19T14:54:25.734486+0000","last_active":"2024-02-19T14:54:37.818743+0000","last_peered":"2024-02-19T14:54:37.818743+0000","last_clean":"2024-02-19T14:54:37.818743+0000","last_became_active":"2024-02-19T14:54:25.734200+0000","last_became_peered":"2024-02-19T14:54:25.734200+0000","last_unstale":"2024-02-19T14:54:37.818743+0000","last_undegraded":"2024-02-19T14:54:37.818743+0000","last_fullsized":"2024-02-19T14:54:37.818743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:50:03.491743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.837289+0000","last_change":"2024-02-19T14:54:37.831915+0000","last_active":"2024-02-19T14:54:37.837289+0000","last_peered":"2024-02-19T14:54:37.837289+0000","last_clean":"2024-02-19T14:54:37.837289+0000","last_became_active":"2024-02-19T14:54:37.831649+0000","last_became_peered":"2024-02-19T14:54:37.831649+0000","last_unstale":"2024-02-19T14:54:37.837289+0000","last_undegraded":"2024-02-19T14:54:37.837289+0000","last_fullsized":"2024-02-19T14:54:37.837289+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:52:53.015938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819439+0000","last_change":"2024-02-19T14:54:36.828188+0000","last_active":"2024-02-19T14:54:37.819439+0000","last_peered":"2024-02-19T14:54:37.819439+0000","last_clean":"2024-02-19T14:54:37.819439+0000","last_became_active":"2024-02-19T14:54:36.827920+0000","last_became_peered":"2024-02-19T14:54:36.827920+0000","last_unstale":"2024-02-19T14:54:37.819439+0000","last_undegraded":"2024-02-19T14:54:37.819439+0000","last_fullsized":"2024-02-19T14:54:37.819439+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:41.654671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819507+0000","last_change":"2024-02-19T14:54:26.751887+0000","last_active":"2024-02-19T14:54:37.819507+0000","last_peered":"2024-02-19T14:54:37.819507+0000","last_clean":"2024-02-19T14:54:37.819507+0000","last_became_active":"2024-02-19T14:54:26.751432+0000","last_became_peered":"2024-02-19T14:54:26.751432+0000","last_unstale":"2024-02-19T14:54:37.819507+0000","last_undegraded":"2024-02-19T14:54:37.819507+0000","last_fullsized":"2024-02-19T14:54:37.819507+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:27.494199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.815220+0000","last_change":"2024-02-19T14:54:25.738928+0000","last_active":"2024-02-19T14:54:37.815220+0000","last_peered":"2024-02-19T14:54:37.815220+0000","last_clean":"2024-02-19T14:54:37.815220+0000","last_became_active":"2024-02-19T14:54:25.738674+0000","last_became_peered":"2024-02-19T14:54:25.738674+0000","last_unstale":"2024-02-19T14:54:37.815220+0000","last_undegraded":"2024-02-19T14:54:37.815220+0000","last_fullsized":"2024-02-19T14:54:37.815220+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:46:58.792240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841292+0000","last_change":"2024-02-19T14:54:37.836690+0000","last_active":"2024-02-19T14:54:37.841292+0000","last_peered":"2024-02-19T14:54:37.841292+0000","last_clean":"2024-02-19T14:54:37.841292+0000","last_became_active":"2024-02-19T14:54:37.835735+0000","last_became_peered":"2024-02-19T14:54:37.835735+0000","last_unstale":"2024-02-19T14:54:37.841292+0000","last_undegraded":"2024-02-19T14:54:37.841292+0000","last_fullsized":"2024-02-19T14:54:37.841292+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:34:44.271939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"78'2","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.841428+0000","last_change":"2024-02-19T14:54:36.830155+0000","last_active":"2024-02-19T14:54:37.841428+0000","last_peered":"2024-02-19T14:54:37.841428+0000","last_clean":"2024-02-19T14:54:37.841428+0000","last_became_active":"2024-02-19T14:54:36.829822+0000","last_became_peered":"2024-02-19T14:54:36.829822+0000","last_unstale":"2024-02-19T14:54:37.841428+0000","last_undegraded":"2024-02-19T14:54:37.841428+0000","last_fullsized":"2024-02-19T14:54:37.841428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:42.251169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822240+0000","last_change":"2024-02-19T14:54:26.754376+0000","last_active":"2024-02-19T14:54:37.822240+0000","last_peered":"2024-02-19T14:54:37.822240+0000","last_clean":"2024-02-19T14:54:37.822240+0000","last_became_active":"2024-02-19T14:54:26.754102+0000","last_became_peered":"2024-02-19T14:54:26.754102+0000","last_unstale":"2024-02-19T14:54:37.822240+0000","last_undegraded":"2024-02-19T14:54:37.822240+0000","last_fullsized":"2024-02-19T14:54:37.822240+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:02.882466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.817664+0000","last_change":"2024-02-19T14:54:25.735756+0000","last_active":"2024-02-19T14:54:37.817664+0000","last_peered":"2024-02-19T14:54:37.817664+0000","last_clean":"2024-02-19T14:54:37.817664+0000","last_became_active":"2024-02-19T14:54:25.735486+0000","last_became_peered":"2024-02-19T14:54:25.735486+0000","last_unstale":"2024-02-19T14:54:37.817664+0000","last_undegraded":"2024-02-19T14:54:37.817664+0000","last_fullsized":"2024-02-19T14:54:37.817664+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:12.694335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.846488+0000","last_change":"2024-02-19T14:54:37.845487+0000","last_active":"2024-02-19T14:54:37.846488+0000","last_peered":"2024-02-19T14:54:37.846488+0000","last_clean":"2024-02-19T14:54:37.846488+0000","last_became_active":"2024-02-19T14:54:37.845242+0000","last_became_peered":"2024-02-19T14:54:37.845242+0000","last_unstale":"2024-02-19T14:54:37.846488+0000","last_undegraded":"2024-02-19T14:54:37.846488+0000","last_fullsized":"2024-02-19T14:54:37.846488+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:43.228266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.826689+0000","last_change":"2024-02-19T14:54:36.825202+0000","last_active":"2024-02-19T14:54:37.826689+0000","last_peered":"2024-02-19T14:54:37.826689+0000","last_clean":"2024-02-19T14:54:37.826689+0000","last_became_active":"2024-02-19T14:54:36.824918+0000","last_became_peered":"2024-02-19T14:54:36.824918+0000","last_unstale":"2024-02-19T14:54:37.826689+0000","last_undegraded":"2024-02-19T14:54:37.826689+0000","last_fullsized":"2024-02-19T14:54:37.826689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:15.472069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821787+0000","last_change":"2024-02-19T14:54:26.743441+0000","last_active":"2024-02-19T14:54:37.821787+0000","last_peered":"2024-02-19T14:54:37.821787+0000","last_clean":"2024-02-19T14:54:37.821787+0000","last_became_active":"2024-02-19T14:54:26.743151+0000","last_became_peered":"2024-02-19T14:54:26.743151+0000","last_unstale":"2024-02-19T14:54:37.821787+0000","last_undegraded":"2024-02-19T14:54:37.821787+0000","last_fullsized":"2024-02-19T14:54:37.821787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:27.494199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.819120+0000","last_change":"2024-02-19T14:54:25.728807+0000","last_active":"2024-02-19T14:54:37.819120+0000","last_peered":"2024-02-19T14:54:37.819120+0000","last_clean":"2024-02-19T14:54:37.819120+0000","last_became_active":"2024-02-19T14:54:25.728541+0000","last_became_peered":"2024-02-19T14:54:25.728541+0000","last_unstale":"2024-02-19T14:54:37.819120+0000","last_undegraded":"2024-02-19T14:54:37.819120+0000","last_fullsized":"2024-02-19T14:54:37.819120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:12.694335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.838481+0000","last_change":"2024-02-19T14:54:37.835628+0000","last_active":"2024-02-19T14:54:37.838481+0000","last_peered":"2024-02-19T14:54:37.838481+0000","last_clean":"2024-02-19T14:54:37.838481+0000","last_became_active":"2024-02-19T14:54:37.833937+0000","last_became_peered":"2024-02-19T14:54:37.833937+0000","last_unstale":"2024-02-19T14:54:37.838481+0000","last_undegraded":"2024-02-19T14:54:37.838481+0000","last_fullsized":"2024-02-19T14:54:37.838481+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:36.805879+0000","last_clean_scrub_stamp":"2024-02-19T14:54:36.805879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:10.508329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.822177+0000","last_change":"2024-02-19T14:54:36.825196+0000","last_active":"2024-02-19T14:54:37.822177+0000","last_peered":"2024-02-19T14:54:37.822177+0000","last_clean":"2024-02-19T14:54:37.822177+0000","last_became_active":"2024-02-19T14:54:36.824941+0000","last_became_peered":"2024-02-19T14:54:36.824941+0000","last_unstale":"2024-02-19T14:54:37.822177+0000","last_undegraded":"2024-02-19T14:54:37.822177+0000","last_fullsized":"2024-02-19T14:54:37.822177+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:35.801343+0000","last_clean_scrub_stamp":"2024-02-19T14:54:35.801343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:18.525535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.825384+0000","last_change":"2024-02-19T14:54:26.742776+0000","last_active":"2024-02-19T14:54:37.825384+0000","last_peered":"2024-02-19T14:54:37.825384+0000","last_clean":"2024-02-19T14:54:37.825384+0000","last_became_active":"2024-02-19T14:54:26.742433+0000","last_became_peered":"2024-02-19T14:54:26.742433+0000","last_unstale":"2024-02-19T14:54:37.825384+0000","last_undegraded":"2024-02-19T14:54:37.825384+0000","last_fullsized":"2024-02-19T14:54:37.825384+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:25.715235+0000","last_clean_scrub_stamp":"2024-02-19T14:54:25.715235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:56:36.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.820247+0000","last_change":"2024-02-19T14:54:25.738298+0000","last_active":"2024-02-19T14:54:37.820247+0000","last_peered":"2024-02-19T14:54:37.820247+0000","last_clean":"2024-02-19T14:54:37.820247+0000","last_became_active":"2024-02-19T14:54:25.737717+0000","last_became_peered":"2024-02-19T14:54:25.737717+0000","last_unstale":"2024-02-19T14:54:37.820247+0000","last_undegraded":"2024-02-19T14:54:37.820247+0000","last_fullsized":"2024-02-19T14:54:37.820247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:54:24.709171+0000","last_clean_scrub_stamp":"2024-02-19T14:54:24.709171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:07.018554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":279,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-19T14:54:37.821363+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:54:37.821363+0000","last_peered":"2024-02-19T14:54:37.821363+0000","last_clean":"2024-02-19T14:54:37.821363+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:54:37.821363+0000","last_undegraded":"2024-02-19T14:54:37.821363+0000","last_fullsized":"2024-02-19T14:54:37.821363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1540096,"data_stored":1039768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.60499999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.46000000000000002,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.502,"15min":0.499},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.497,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.441,"15min":0.44400000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.5},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":450990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70799999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.498,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.65800000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.52900000000000003,"15min":0.51900000000000002},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71199999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.62},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.46200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78100000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42299999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61399999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70999999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54600000000000004,"15min":0.55500000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63200000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.754,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":449412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.496},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48699999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.38}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56299999999999994}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55800000000000005,"15min":0.56899999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.59799999999999998,"15min":0.60799999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73099999999999998,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":451094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.747,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.59699999999999998,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68600000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.95799999999999996,"5min":1.038,"15min":1.038},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.495,"15min":0.48799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.64200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":457985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29599999999999999,"15min":0.29499999999999998},"min":{"1min":0.17899999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.309,"15min":0.30499999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.245}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.31},"min":{"1min":0.22,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.374,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.32900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49199999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69099999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.42799999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68700000000000006,"5min":1.133,"15min":1.133},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.38},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77500000000000002,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.502}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73799999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82999999999999996,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.51200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":954368,"data_stored":450120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.17999999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.222},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"min":{"1min":0.19900000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.437,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.42799999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.54100000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.20699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.215,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.253},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.375,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.309}]},{"osd":4,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79000000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.433},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84199999999999997,"5min":0.878,"15min":0.878},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94199999999999995,"5min":1.236,"15min":1.236},"last":0.54000000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:53:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71299999999999997,"5min":0.995,"15min":0.995},"last":0.49199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":457585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.33500000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.497,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.33100000000000002,"15min":0.33800000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.307,"15min":0.309},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.371,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54600000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.23699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.57999999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.53300000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55300000000000005,"15min":0.55500000000000005},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70199999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.438}]},{"osd":6,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95499999999999996,"5min":1.026,"15min":1.026},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44400000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.55100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28524,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343316,"statfs":{"total":96636764160,"available":96607555584,"internally_reserved":0,"allocated":1536000,"data_stored":1039746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.38100000000000001,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.193},{"interface":"front","average":{"1min":0.315,"5min":0.30299999999999999,"15min":0.30099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32300000000000001,"15min":0.32400000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42599999999999999,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.34899999999999998,"15min":0.35199999999999998},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.42699999999999999,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34499999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34300000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.41199999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.317}]},{"osd":4,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57599999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.504}]},{"osd":5,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.60299999999999998,"15min":0.61699999999999999},"min":{"1min":0.42099999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.59999999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58299999999999996,"15min":0.58999999999999997},"min":{"1min":0.39600000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.69999999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.36399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68700000000000006,"5min":0.754,"15min":0.754},"last":0.63100000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.434,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72999999999999998,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.622}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:54:41.354 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:54:41.354 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs refuse_standby_for_another_fs true 2024-02-19T14:54:41.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.491+0000 7f478ad77640 1 Processor -- start 2024-02-19T14:54:41.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.491+0000 7f478ad77640 1 -- start start 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f478ad77640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 0x7f47840c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f478ad77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47840c4ed0 con 0x7f47840c7c50 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f478ad77640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4784155e70 con 0x7f47840c18f0 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f478ad77640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478406c510 con 0x7f47840c7660 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f4789314640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f47840c7c50 0x7f478415f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54818/0 (socket says 172.21.15.78:54818) 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f4789314640 1 -- 172.21.15.78:0/543637635 learned_addr learned my addr 172.21.15.78:0/543637635 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911374444 0 0) 0x7f47840c4ed0 con 0x7f47840c7c50 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774003520 con 0x7f47840c7c50 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f4783fff640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 0x7f47840c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290147646 0 0) 0x7f4774003520 con 0x7f47840c7c50 2024-02-19T14:54:41.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 msgr2=0x7f47840c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 0x7f47840c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 >> v1:172.21.15.150:6789/0 conn(0x7f47840c18f0 legacy=0x7f47840c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478406d040 con 0x7f47840c7c50 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f477c0019f0 con 0x7f47840c7c50 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.492+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 >> v1:172.21.15.78:6789/0 conn(0x7f47840c7c50 legacy=0x7f478415f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f47837fe640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47740039a0 0x7f4774003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f47837fe640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47740047c0 0x7f4774004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f47837fe640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 0x7f4774005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477400ba30 con 0x7f47740039a0 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477400bba0 con 0x7f47740047c0 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477400bd60 con 0x7f47840c7660 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4788b13640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47740039a0 0x7f4774003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4783fff640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47740047c0 0x7f4774004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4789314640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 0x7f4774005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4788b13640 1 -- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 msgr2=0x7f4774005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4788b13640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47840c7660 0x7f4774005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4788b13640 1 -- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47740047c0 msgr2=0x7f4774004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:41.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4788b13640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47740047c0 0x7f4774004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.493+0000 7f4788b13640 1 -- 172.21.15.78:0/543637635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774003520 con 0x7f47740039a0 2024-02-19T14:54:41.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.494+0000 7f4788b13640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47740039a0 0x7f4774003dc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f4778001720 tx=0x7f47740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6edea5ca01608 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.494+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780052d0 con 0x7f47740039a0 2024-02-19T14:54:41.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.494+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477802e050 con 0x7f47740039a0 2024-02-19T14:54:41.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.494+0000 7f47837fe640 1 -- 172.21.15.78:0/543637635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47780035f0 con 0x7f47740039a0 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f478ad77640 1 -- 172.21.15.78:0/543637635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47740039a0 msgr2=0x7f4774003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f478ad77640 1 --2- 172.21.15.78:0/543637635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47740039a0 0x7f4774003dc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f4778001720 tx=0x7f47740057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f4789314640 1 -- 172.21.15.78:0/543637635 reap_dead start 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f478ad77640 1 -- 172.21.15.78:0/543637635 shutdown_connections 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f478ad77640 1 -- 172.21.15.78:0/543637635 >> 172.21.15.78:0/543637635 conn(0x7f47840c85e0 msgr2=0x7f47840c89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f478ad77640 1 -- 172.21.15.78:0/543637635 shutdown_connections 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.495+0000 7f478ad77640 1 -- 172.21.15.78:0/543637635 wait complete. 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.496+0000 7f478ad77640 1 Processor -- start 2024-02-19T14:54:41.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.496+0000 7f478ad77640 1 -- start start 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f478ad77640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47740039a0 0x7f47840c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f478ad77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784158b40 0x7f47840a7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f478ad77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 0x7f47840a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f478ad77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4784158420 con 0x7f4784159130 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f478ad77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4784157da0 con 0x7f4784158b40 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f478ad77640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47841eaff0 con 0x7f47740039a0 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f4789314640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 0x7f47840a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f4789314640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 0x7f47840a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33002/0 (socket says 172.21.15.78:33002) 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.497+0000 7f4789314640 1 -- 172.21.15.78:0/3952036127 learned_addr learned my addr 172.21.15.78:0/3952036127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4788b13640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47740039a0 0x7f47840c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4783fff640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784158b40 0x7f47840a7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4789314640 1 -- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47740039a0 msgr2=0x7f47840c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4789314640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47740039a0 0x7f47840c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4789314640 1 -- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784158b40 msgr2=0x7f47840a7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4789314640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784158b40 0x7f47840a7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4789314640 1 -- 172.21.15.78:0/3952036127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784158040 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4788b13640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47740039a0 0x7f47840c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4783fff640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784158b40 0x7f47840a7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f4789314640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 0x7f47840a75c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f47840c8030 tx=0x7f477c007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c00ac30 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477c00ad90 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c013610 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47841eb2d0 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.498+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47840ae320 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.499+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47840c2930 con 0x7f4784159130 2024-02-19T14:54:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.500+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f477c013770 con 0x7f4784159130 2024-02-19T14:54:41.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.500+0000 7f47817fa640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476806b810 0x7f476806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.500+0000 7f4788b13640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476806b810 0x7f476806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.500+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7073+0+0 (secure 0 0 0) 0x7f477c08aeb0 con 0x7f4784159130 2024-02-19T14:54:41.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.501+0000 7f4788b13640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476806b810 0x7f476806dcd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4778033000 tx=0x7f4778009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.504+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f477c055370 con 0x7f4784159130 2024-02-19T14:54:41.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.720+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"} v 0) v1 -- 0x7f4784074970 con 0x7f4784159130 2024-02-19T14:54:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.848+0000 7f47817fa640 1 -- 172.21.15.78:0/3952036127 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"}]=0 set to refuse standby for another fs v55) v1 ==== 166+0+0 (secure 0 0 0) 0x7f477c058a30 con 0x7f4784159130 2024-02-19T14:54:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:set to refuse standby for another fs 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476806b810 msgr2=0x7f476806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476806b810 0x7f476806dcd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4778033000 tx=0x7f4778009030 comp rx=0 tx=0).stop 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 msgr2=0x7f47840a75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 0x7f47840a75c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f47840c8030 tx=0x7f477c007bd0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 shutdown_connections 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476806b810 0x7f476806dcd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4784159130 0x7f47840a75c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784158b40 0x7f47840a7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 --2- 172.21.15.78:0/3952036127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47740039a0 0x7f47840c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 >> 172.21.15.78:0/3952036127 conn(0x7f47840c85e0 msgr2=0x7f4784153b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 shutdown_connections 2024-02-19T14:54:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.851+0000 7f478ad77640 1 -- 172.21.15.78:0/3952036127 wait complete. 2024-02-19T14:54:41.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 5:0 2024-02-19T14:54:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.966+0000 7f88f1a04640 1 Processor -- start 2024-02-19T14:54:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.966+0000 7f88f1a04640 1 -- start start 2024-02-19T14:54:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88f1a04640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88ec0c1e70 0x7f88ec066c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88f1a04640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec0a6220 con 0x7f88ec0671d0 2024-02-19T14:54:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88f1a04640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec072ee0 con 0x7f88ec14f210 2024-02-19T14:54:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88f1a04640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0c2e60 con 0x7f88ec0c1e70 2024-02-19T14:54:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88ea7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88ec0c1e70 0x7f88ec066c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88eb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f88ec0671d0 0x7f88ec15c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54824/0 (socket says 172.21.15.78:54824) 2024-02-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.967+0000 7f88eb7fe640 1 -- 172.21.15.78:0/3688481764 learned_addr learned my addr 172.21.15.78:0/3688481764 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.968+0000 7f88ea7fc640 1 -- 172.21.15.78:0/3688481764 >> v1:172.21.15.150:6789/0 conn(0x7f88ec14f210 legacy=0x7f88ec0727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.968+0000 7f88e9ffb640 1 -- 172.21.15.78:0/3688481764 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066847059 0 0) 0x7f88ec0a6220 con 0x7f88ec0671d0 2024-02-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.968+0000 7f88ea7fc640 1 -- 172.21.15.78:0/3688481764 >> v1:172.21.15.78:6789/0 conn(0x7f88ec0671d0 legacy=0x7f88ec15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.968+0000 7f88ea7fc640 1 -- 172.21.15.78:0/3688481764 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec0a6470 con 0x7f88ec0c1e70 2024-02-19T14:54:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.968+0000 7f88ea7fc640 1 --2- 172.21.15.78:0/3688481764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec0c1e70 0x7f88ec066c90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f88e00097a0 tx=0x7f88e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cc333d0d40b438 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88e9ffb640 1 -- 172.21.15.78:0/3688481764 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e002e070 con 0x7f88ec0c1e70 2024-02-19T14:54:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88e9ffb640 1 -- 172.21.15.78:0/3688481764 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e00315a0 con 0x7f88ec0c1e70 2024-02-19T14:54:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88e9ffb640 1 -- 172.21.15.78:0/3688481764 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0031930 con 0x7f88ec0c1e70 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 -- 172.21.15.78:0/3688481764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec0c1e70 msgr2=0x7f88ec066c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3688481764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec0c1e70 0x7f88ec066c90 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f88e00097a0 tx=0x7f88e002b840 comp rx=0 tx=0).stop 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 -- 172.21.15.78:0/3688481764 shutdown_connections 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3688481764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec0c1e70 0x7f88ec066c90 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 -- 172.21.15.78:0/3688481764 >> 172.21.15.78:0/3688481764 conn(0x7f88ec0c3660 msgr2=0x7f88ec0c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 -- 172.21.15.78:0/3688481764 shutdown_connections 2024-02-19T14:54:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.969+0000 7f88f1a04640 1 -- 172.21.15.78:0/3688481764 wait complete. 2024-02-19T14:54:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.970+0000 7f88f1a04640 1 Processor -- start 2024-02-19T14:54:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 -- start start 2024-02-19T14:54:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88ec0671d0 0x7f88ec1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88ec0c1e70 0x7f88ec1dbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 0x7f88ec1dc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1e82d0 con 0x7f88ec14f210 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1e7ff0 con 0x7f88ec0671d0 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88f1a04640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1db410 con 0x7f88ec0c1e70 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88eb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 0x7f88ec1dc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88eb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 0x7f88ec1dc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33014/0 (socket says 172.21.15.78:33014) 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.971+0000 7f88eb7fe640 1 -- 172.21.15.78:0/3453238229 learned_addr learned my addr 172.21.15.78:0/3453238229 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88ea7fc640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88ec0c1e70 0x7f88ec1dbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eaffd640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88ec0671d0 0x7f88ec1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eb7fe640 1 -- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88ec0c1e70 msgr2=0x7f88ec1dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eb7fe640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88ec0c1e70 0x7f88ec1dbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eb7fe640 1 -- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88ec0671d0 msgr2=0x7f88ec1559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eb7fe640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88ec0671d0 0x7f88ec1559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eb7fe640 1 -- 172.21.15.78:0/3453238229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e0009400 con 0x7f88ec14f210 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88ea7fc640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88ec0c1e70 0x7f88ec1dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88eb7fe640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 0x7f88ec1dc430 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f88e400c7a0 tx=0x7f88e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e400a640 con 0x7f88ec14f210 2024-02-19T14:54:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e400a7a0 con 0x7f88ec14f210 2024-02-19T14:54:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e400abd0 con 0x7f88ec14f210 2024-02-19T14:54:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec1db6f0 con 0x7f88ec14f210 2024-02-19T14:54:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.972+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec155ef0 con 0x7f88ec14f210 2024-02-19T14:54:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.973+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec1db270 con 0x7f88ec14f210 2024-02-19T14:54:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.974+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f88e400ad30 con 0x7f88ec14f210 2024-02-19T14:54:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.974+0000 7f88d3fff640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f88d806b810 0x7f88d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.974+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7073+0+0 (secure 0 0 0) 0x7f88e408a010 con 0x7f88ec14f210 2024-02-19T14:54:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.974+0000 7f88eaffd640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f88d806b810 0x7f88d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.974+0000 7f88eaffd640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f88d806b810 0x7f88d806dcd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f88dc006fd0 tx=0x7f88dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:41.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:41.982+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f88ec1db270 con 0x7f88ec14f210 2024-02-19T14:54:41.992 INFO:tasks.ceph.mds.a.smithi078.stderr:2024-02-19T14:54:41.990+0000 7fb177223640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:42.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.306+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "5:0"} v 0) v1 -- 0x7f88ec1dbc10 con 0x7f88ec14f210 2024-02-19T14:54:42.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:54:42.307+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:54:42.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.308+0000 7f88d3fff640 1 -- 172.21.15.78:0/3453238229 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "5:0"}]=0 failed mds gid 6363 v56) v1 ==== 97+0+0 (secure 0 0 0) 0x7f88ec1dbc10 con 0x7f88ec14f210 2024-02-19T14:54:42.311 INFO:teuthology.orchestra.run.smithi078.stderr:failed mds gid 6363 2024-02-19T14:54:42.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f88d806b810 msgr2=0x7f88d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f88d806b810 0x7f88d806dcd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f88dc006fd0 tx=0x7f88dc008040 comp rx=0 tx=0).stop 2024-02-19T14:54:42.315 INFO:tasks.ceph.mds.c.smithi078.stderr: -372> 2024-02-19T14:54:31.989+0000 7f4264c23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 msgr2=0x7f88ec1dc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 0x7f88ec1dc430 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f88e400c7a0 tx=0x7f88e400cc60 comp rx=0 tx=0).stop 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 shutdown_connections 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f88d806b810 0x7f88d806dcd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec14f210 0x7f88ec1dc430 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88ec0c1e70 0x7f88ec1dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 --2- 172.21.15.78:0/3453238229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88ec0671d0 0x7f88ec1559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 >> 172.21.15.78:0/3453238229 conn(0x7f88ec0c3660 msgr2=0x7f88ec0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:42.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 shutdown_connections 2024-02-19T14:54:42.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.312+0000 7f88f1a04640 1 -- 172.21.15.78:0/3453238229 wait complete. 2024-02-19T14:54:42.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:42.343 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:54:42.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.429+0000 7efd90725640 1 Processor -- start 2024-02-19T14:54:42.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.429+0000 7efd90725640 1 -- start start 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd90725640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efd88152700 0x7efd880c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd90725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd8806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd90725640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd881551f0 con 0x7efd880c8e00 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd90725640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd88155490 con 0x7efd88152700 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd90725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd88155770 con 0x7efd880c5730 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8ecc2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efd880c8e00 0x7efd88158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54840/0 (socket says 172.21.15.78:54840) 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8ecc2640 1 -- 172.21.15.78:0/941033205 learned_addr learned my addr 172.21.15.78:0/941033205 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8e4c1640 1 --2- 172.21.15.78:0/941033205 >> v2:172.21.15.150:3300/0 conn(0x7efd88152700 0x7efd880c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8dcc0640 1 --2- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd8806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8d4bf640 1 -- 172.21.15.78:0/941033205 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18536637 0 0) 0x7efd881551f0 con 0x7efd880c8e00 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8d4bf640 1 -- 172.21.15.78:0/941033205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd74003520 con 0x7efd880c8e00 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8e4c1640 1 -- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 msgr2=0x7efd8806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8e4c1640 1 --2- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd8806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8e4c1640 1 -- 172.21.15.78:0/941033205 >> v1:172.21.15.78:6789/0 conn(0x7efd880c8e00 legacy=0x7efd88158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8d4bf640 1 -- 172.21.15.78:0/941033205 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940718486 0 0) 0x7efd74003520 con 0x7efd880c8e00 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.430+0000 7efd8e4c1640 1 -- 172.21.15.78:0/941033205 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88155a50 con 0x7efd88152700 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.431+0000 7efd8e4c1640 1 --2- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd88152700 0x7efd880c51f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efd840097a0 tx=0x7efd8402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c6f7d24e544cc93 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.431+0000 7efd8d4bf640 1 -- 172.21.15.78:0/941033205 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800019f0 con 0x7efd88152700 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.431+0000 7efd8d4bf640 1 -- 172.21.15.78:0/941033205 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8402e050 con 0x7efd88152700 2024-02-19T14:54:42.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.431+0000 7efd8d4bf640 1 -- 172.21.15.78:0/941033205 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84030710 con 0x7efd88152700 2024-02-19T14:54:42.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 -- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd88152700 msgr2=0x7efd880c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 --2- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd88152700 0x7efd880c51f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efd840097a0 tx=0x7efd8402b790 comp rx=0 tx=0).stop 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 -- 172.21.15.78:0/941033205 shutdown_connections 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 --2- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd8806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 --2- 172.21.15.78:0/941033205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd88152700 0x7efd880c51f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 -- 172.21.15.78:0/941033205 >> 172.21.15.78:0/941033205 conn(0x7efd880cac90 msgr2=0x7efd880cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 -- 172.21.15.78:0/941033205 shutdown_connections 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.432+0000 7efd90725640 1 -- 172.21.15.78:0/941033205 wait complete. 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.433+0000 7efd90725640 1 Processor -- start 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.433+0000 7efd90725640 1 -- start start 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd880c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd880c8e00 0x7efd880c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 0x7efd880c2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880c1de0 con 0x7efd88152700 2024-02-19T14:54:42.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd881eb770 con 0x7efd880c8e00 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd881ebca0 con 0x7efd880c5730 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 0x7efd880c2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 0x7efd880c2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33038/0 (socket says 172.21.15.78:33038) 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 -- 172.21.15.78:0/1179052508 learned_addr learned my addr 172.21.15.78:0/1179052508 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8dcc0640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd880c8e00 0x7efd880c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8e4c1640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd880c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 -- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 msgr2=0x7efd880c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd880c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 -- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd880c8e00 msgr2=0x7efd880c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd880c8e00 0x7efd880c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 -- 172.21.15.78:0/1179052508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84009400 con 0x7efd88152700 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd8ecc2640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 0x7efd880c2270 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efd80001860 tx=0x7efd8000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80012400 con 0x7efd88152700 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd80012560 con 0x7efd88152700 2024-02-19T14:54:42.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.434+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd880ace00 con 0x7efd88152700 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.435+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80012910 con 0x7efd88152700 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.435+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd880c1a70 con 0x7efd88152700 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.435+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd880c1f90 con 0x7efd88152700 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.436+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efd80012a70 con 0x7efd88152700 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.436+0000 7efd7effd640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6c06b840 0x7efd6c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.436+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7efd8008af50 con 0x7efd88152700 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.436+0000 7efd8e4c1640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6c06b840 0x7efd6c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.437+0000 7efd8e4c1640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6c06b840 0x7efd6c06dd00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efd84034000 tx=0x7efd8402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:42.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.441+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efd880c1f90 con 0x7efd88152700 2024-02-19T14:54:42.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.652+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd880c1f90 con 0x7efd88152700 2024-02-19T14:54:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.652+0000 7efd7effd640 1 -- 172.21.15.78:0/1179052508 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5382 (secure 0 0 0) 0x7efd880c1f90 con 0x7efd88152700 2024-02-19T14:54:42.655 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:42.655 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":56,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:42.308169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:42.656 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 56 2024-02-19T14:54:42.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6c06b840 msgr2=0x7efd6c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6c06b840 0x7efd6c06dd00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efd84034000 tx=0x7efd8402f000 comp rx=0 tx=0).stop 2024-02-19T14:54:42.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 msgr2=0x7efd880c2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 0x7efd880c2270 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efd80001860 tx=0x7efd8000a430 comp rx=0 tx=0).stop 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 shutdown_connections 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6c06b840 0x7efd6c06dd00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd88152700 0x7efd880c2270 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd880c8e00 0x7efd880c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 --2- 172.21.15.78:0/1179052508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd880c5730 0x7efd880c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 >> 172.21.15.78:0/1179052508 conn(0x7efd880cac90 msgr2=0x7efd880cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 shutdown_connections 2024-02-19T14:54:42.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.655+0000 7efd90725640 1 -- 172.21.15.78:0/1179052508 wait complete. 2024-02-19T14:54:42.669 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-02-19T14:54:42.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-02-19T14:54:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:{"mds_beacon_grace":"15.000000"} 2024-02-19T14:54:42.820 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-02-19T14:54:42.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:42.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.919+0000 7fd03b264640 1 Processor -- start 2024-02-19T14:54:42.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.919+0000 7fd03b264640 1 -- start start 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd03b264640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd03414f1a0 0x7fd034072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd03b264640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0340c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd03b264640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0340c5970 con 0x7fd03406ea50 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd03b264640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd034152af0 con 0x7fd03414f1a0 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd03b264640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340c5750 con 0x7fd034152680 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd039000640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd03414f1a0 0x7fd034072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd039801640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0340c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd033fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd03406ea50 0x7fd03406fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:35882/0 (socket says 172.21.15.78:35882) 2024-02-19T14:54:42.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd033fff640 1 -- 172.21.15.78:0/3455208978 learned_addr learned my addr 172.21.15.78:0/3455208978 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:42.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.920+0000 7fd039000640 1 --2- 172.21.15.78:0/3455208978 >> v2:172.21.15.150:3300/0 conn(0x7fd03414f1a0 0x7fd034072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:37292/0 (socket says 172.21.15.78:37292) 2024-02-19T14:54:42.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd039000640 1 -- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 msgr2=0x7fd0340c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:42.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd039000640 1 --2- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0340c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd039000640 1 -- 172.21.15.78:0/3455208978 >> v1:172.21.15.150:6789/0 conn(0x7fd03406ea50 legacy=0x7fd03406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd039000640 1 -- 172.21.15.78:0/3455208978 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0340702d0 con 0x7fd03414f1a0 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd039000640 1 --2- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03414f1a0 0x7fd034072d70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd028009d90 tx=0x7fd02802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f903039a85baebb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd0337fe640 1 -- 172.21.15.78:0/3455208978 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02802cb30 con 0x7fd03414f1a0 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.921+0000 7fd0337fe640 1 -- 172.21.15.78:0/3455208978 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02802cc90 con 0x7fd03414f1a0 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd0337fe640 1 -- 172.21.15.78:0/3455208978 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028034440 con 0x7fd03414f1a0 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 -- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03414f1a0 msgr2=0x7fd034072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 --2- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03414f1a0 0x7fd034072d70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd028009d90 tx=0x7fd02802bd80 comp rx=0 tx=0).stop 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 -- 172.21.15.78:0/3455208978 shutdown_connections 2024-02-19T14:54:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 --2- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0340c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 --2- 172.21.15.78:0/3455208978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03414f1a0 0x7fd034072d70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 -- 172.21.15.78:0/3455208978 >> 172.21.15.78:0/3455208978 conn(0x7fd03406c170 msgr2=0x7fd03406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 -- 172.21.15.78:0/3455208978 shutdown_connections 2024-02-19T14:54:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.922+0000 7fd03b264640 1 -- 172.21.15.78:0/3455208978 wait complete. 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.923+0000 7fd03b264640 1 Processor -- start 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.923+0000 7fd03b264640 1 -- start start 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd03b264640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03406ea50 0x7fd03414cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd03b264640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 0x7fd034149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd03b264640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0341edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd03b264640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd034151000 con 0x7fd03414f1a0 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd03b264640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd034150420 con 0x7fd03406ea50 2024-02-19T14:54:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd03b264640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd034150700 con 0x7fd034152680 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd033fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 0x7fd034149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd033fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 0x7fd034149b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33046/0 (socket says 172.21.15.78:33046) 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd033fff640 1 -- 172.21.15.78:0/2255803121 learned_addr learned my addr 172.21.15.78:0/2255803121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd039801640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0341edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.924+0000 7fd039000640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03406ea50 0x7fd03414cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd033fff640 1 -- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 msgr2=0x7fd0341edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd033fff640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0341edb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd033fff640 1 -- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03406ea50 msgr2=0x7fd03414cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd033fff640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03406ea50 0x7fd03414cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd033fff640 1 -- 172.21.15.78:0/2255803121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd028009a70 con 0x7fd03414f1a0 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd039000640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03406ea50 0x7fd03414cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd039801640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0341edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd033fff640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 0x7fd034149b10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd02000c3c0 tx=0x7fd02000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd020005ce0 con 0x7fd03414f1a0 2024-02-19T14:54:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd020005e40 con 0x7fd03414f1a0 2024-02-19T14:54:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0200072d0 con 0x7fd03414f1a0 2024-02-19T14:54:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0341509e0 con 0x7fd03414f1a0 2024-02-19T14:54:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.925+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03414ff50 con 0x7fd03414f1a0 2024-02-19T14:54:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.926+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd034074990 con 0x7fd03414f1a0 2024-02-19T14:54:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.926+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd020007430 con 0x7fd03414f1a0 2024-02-19T14:54:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.927+0000 7fd0317fa640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd01806b840 0x7fd01806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.927+0000 7fd039000640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd01806b840 0x7fd01806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.927+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fd020089c50 con 0x7fd03414f1a0 2024-02-19T14:54:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.927+0000 7fd039000640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd01806b840 0x7fd01806dd00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd028036000 tx=0x7fd028030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:42.931+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd02008f020 con 0x7fd03414f1a0 2024-02-19T14:54:43.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.200+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd03406e830 con 0x7fd03414f1a0 2024-02-19T14:54:43.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.200+0000 7fd0317fa640 1 -- 172.21.15.78:0/2255803121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5382 (secure 0 0 0) 0x7fd020017090 con 0x7fd03414f1a0 2024-02-19T14:54:43.203 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:43.203 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":56,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:42.308169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:43.203 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 56 2024-02-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd01806b840 msgr2=0x7fd01806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd01806b840 0x7fd01806dd00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd028036000 tx=0x7fd028030000 comp rx=0 tx=0).stop 2024-02-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 msgr2=0x7fd034149b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 0x7fd034149b10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd02000c3c0 tx=0x7fd02000c880 comp rx=0 tx=0).stop 2024-02-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 shutdown_connections 2024-02-19T14:54:43.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd01806b840 0x7fd01806dd00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd03414f1a0 0x7fd034149b10 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd034152680 0x7fd0341edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 --2- 172.21.15.78:0/2255803121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd03406ea50 0x7fd03414cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 >> 172.21.15.78:0/2255803121 conn(0x7fd03406c170 msgr2=0x7fd03406c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 shutdown_connections 2024-02-19T14:54:43.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:43.202+0000 7fd03b264640 1 -- 172.21.15.78:0/2255803121 wait complete. 2024-02-19T14:54:43.216 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-02-19T14:54:48.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:48.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.320+0000 7f4223e1b640 1 Processor -- start 2024-02-19T14:54:48.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.320+0000 7f4223e1b640 1 -- start start 2024-02-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4223e1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f421c0c2f60 0x7f421c070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4223e1b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0ae670 con 0x7f421c0add70 2024-02-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4223e1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0716b0 con 0x7f421c0c4980 2024-02-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4223e1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421c0c3390 con 0x7f421c0c2f60 2024-02-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4221bb7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f421c0add70 0x7f421c06eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33666/0 (socket says 172.21.15.78:33666) 2024-02-19T14:54:48.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4221bb7640 1 -- 172.21.15.78:0/666485267 learned_addr learned my addr 172.21.15.78:0/666485267 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f42223b8640 1 --2- 172.21.15.78:0/666485267 >> v2:172.21.15.150:3300/0 conn(0x7f421c0c2f60 0x7f421c070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4220bb5640 1 -- 172.21.15.78:0/666485267 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023149438 0 0) 0x7f421c0ae670 con 0x7f421c0add70 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.321+0000 7f4220bb5640 1 -- 172.21.15.78:0/666485267 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4208003520 con 0x7f421c0add70 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.322+0000 7f4220bb5640 1 -- 172.21.15.78:0/666485267 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1246440792 0 0) 0x7f4208003520 con 0x7f421c0add70 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.322+0000 7f42223b8640 1 -- 172.21.15.78:0/666485267 >> v1:172.21.15.150:6789/0 conn(0x7f421c0c4980 legacy=0x7f421c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.322+0000 7f42223b8640 1 -- 172.21.15.78:0/666485267 >> v1:172.21.15.78:6789/0 conn(0x7f421c0add70 legacy=0x7f421c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.322+0000 7f42223b8640 1 -- 172.21.15.78:0/666485267 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c153780 con 0x7f421c0c2f60 2024-02-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.323+0000 7f42223b8640 1 --2- 172.21.15.78:0/666485267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c2f60 0x7f421c070cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4214009d90 tx=0x7f421402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74d705ebce313a50 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.323+0000 7f4220bb5640 1 -- 172.21.15.78:0/666485267 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c001670 con 0x7f421c0c2f60 2024-02-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.323+0000 7f4220bb5640 1 -- 172.21.15.78:0/666485267 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421402c8b0 con 0x7f421c0c2f60 2024-02-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.323+0000 7f4220bb5640 1 -- 172.21.15.78:0/666485267 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421402cc40 con 0x7f421c0c2f60 2024-02-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.323+0000 7f4223e1b640 1 -- 172.21.15.78:0/666485267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c2f60 msgr2=0x7f421c070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.323+0000 7f4223e1b640 1 --2- 172.21.15.78:0/666485267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c2f60 0x7f421c070cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4214009d90 tx=0x7f421402bd80 comp rx=0 tx=0).stop 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.324+0000 7f4223e1b640 1 -- 172.21.15.78:0/666485267 shutdown_connections 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.324+0000 7f4223e1b640 1 --2- 172.21.15.78:0/666485267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c2f60 0x7f421c070cc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.324+0000 7f4223e1b640 1 -- 172.21.15.78:0/666485267 >> 172.21.15.78:0/666485267 conn(0x7f421c0c85e0 msgr2=0x7f421c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.324+0000 7f4223e1b640 1 -- 172.21.15.78:0/666485267 shutdown_connections 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.324+0000 7f4223e1b640 1 -- 172.21.15.78:0/666485267 wait complete. 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 Processor -- start 2024-02-19T14:54:48.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 -- start start 2024-02-19T14:54:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 0x7f421c0c1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f421c0c2f60 0x7f421c0c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c4980 0x7f421c0bd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c156140 con 0x7f421c0add70 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c155830 con 0x7f421c0c4980 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4223e1b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421c0c2250 con 0x7f421c0c2f60 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.325+0000 7f4221bb7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 0x7f421c0c1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 0x7f421c0c1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55308/0 (socket says 172.21.15.78:55308) 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 -- 172.21.15.78:0/711468292 learned_addr learned my addr 172.21.15.78:0/711468292 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f42213b6640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f421c0c2f60 0x7f421c0c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f42223b8640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c4980 0x7f421c0bd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 -- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f421c0c2f60 msgr2=0x7f421c0c0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f421c0c2f60 0x7f421c0c0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 -- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c4980 msgr2=0x7f421c0bd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c4980 0x7f421c0bd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 -- 172.21.15.78:0/711468292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4214009a70 con 0x7f421c0add70 2024-02-19T14:54:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4221bb7640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 0x7f421c0c1a20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f420c0011e0 tx=0x7f420c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c005ad0 con 0x7f421c0add70 2024-02-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c005c50 con 0x7f421c0add70 2024-02-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c155a80 con 0x7f421c0add70 2024-02-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c0072b0 con 0x7f421c0add70 2024-02-19T14:54:48.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.326+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c0c1f60 con 0x7f421c0add70 2024-02-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.327+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c0c94a0 con 0x7f421c0add70 2024-02-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.328+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f420c007410 con 0x7f421c0add70 2024-02-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.328+0000 7f42127fc640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f420006b840 0x7f420006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.328+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f420c089de0 con 0x7f421c0add70 2024-02-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.328+0000 7f42213b6640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f420006b840 0x7f420006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.328+0000 7f42213b6640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f420006b840 0x7f420006dd00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f421c0c5e10 tx=0x7f4218009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.332+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f420c0542e0 con 0x7f421c0add70 2024-02-19T14:54:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.593+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f421c074990 con 0x7f421c0add70 2024-02-19T14:54:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.594+0000 7f42127fc640 1 -- 172.21.15.78:0/711468292 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6182 (secure 0 0 0) 0x7f420c017020 con 0x7f421c0add70 2024-02-19T14:54:48.597 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:48.597 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:45.326993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6657":{"gid":6657,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1047936932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1047936932},{"type":"v1","addr":"172.21.15.78:6837","nonce":1047936932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:48.597 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 61 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f420006b840 msgr2=0x7f420006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f420006b840 0x7f420006dd00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f421c0c5e10 tx=0x7f4218009040 comp rx=0 tx=0).stop 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 msgr2=0x7f421c0c1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 0x7f421c0c1a20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f420c0011e0 tx=0x7f420c00cbe0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 shutdown_connections 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f420006b840 0x7f420006dd00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f421c0c4980 0x7f421c0bd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f421c0c2f60 0x7f421c0c0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 --2- 172.21.15.78:0/711468292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f421c0add70 0x7f421c0c1a20 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 >> 172.21.15.78:0/711468292 conn(0x7f421c0c85e0 msgr2=0x7f421c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 shutdown_connections 2024-02-19T14:54:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.596+0000 7f4223e1b640 1 -- 172.21.15.78:0/711468292 wait complete. 2024-02-19T14:54:48.611 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:54:48.611 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:54:48.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:54:48.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:54:48.709 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:54:48.709 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:54:48.709 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:54:48.724 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:54:48.724 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:54:48.724 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:54:48.742 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:54:48.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.867+0000 7f89ee1ac640 1 Processor -- start 2024-02-19T14:54:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.867+0000 7f89ee1ac640 1 -- start start 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89ee1ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f89e80c7910 0x7f89e80c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89ee1ac640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f89e8070720 0x7f89e8070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89ee1ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80c65a0 con 0x7f89e80c5560 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89ee1ac640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e80c7530 con 0x7f89e8070720 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89ee1ac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e81537b0 con 0x7f89e80c7910 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f89e8070720 0x7f89e8070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f89e80c5560 0x7f89e8154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33670/0 (socket says 172.21.15.78:33670) 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f89e8070720 0x7f89e8070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55316/0 (socket says 172.21.15.78:55316) 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e77fe640 1 -- 172.21.15.78:0/3969372487 learned_addr learned my addr 172.21.15.78:0/3969372487 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e6ffd640 1 --2- 172.21.15.78:0/3969372487 >> v2:172.21.15.150:3300/0 conn(0x7f89e80c7910 0x7f89e80c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e7fff640 1 -- 172.21.15.78:0/3969372487 >> v2:172.21.15.150:3300/0 conn(0x7f89e80c7910 msgr2=0x7f89e80c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e7fff640 1 --2- 172.21.15.78:0/3969372487 >> v2:172.21.15.150:3300/0 conn(0x7f89e80c7910 0x7f89e80c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e7fff640 1 -- 172.21.15.78:0/3969372487 >> v1:172.21.15.78:6789/0 conn(0x7f89e80c5560 legacy=0x7f89e8154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e7fff640 1 -- 172.21.15.78:0/3969372487 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e8153a90 con 0x7f89e8070720 2024-02-19T14:54:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.869+0000 7f89e67fc640 1 -- 172.21.15.78:0/3969372487 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287450288 0 0) 0x7f89e80c65a0 con 0x7f89e80c5560 2024-02-19T14:54:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.870+0000 7f89e7fff640 1 --2- 172.21.15.78:0/3969372487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e8070720 0x7f89e8070b70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f89e80c91b0 tx=0x7f89e002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24df6740fa6dd8c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.870+0000 7f89e67fc640 1 -- 172.21.15.78:0/3969372487 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e002cb60 con 0x7f89e8070720 2024-02-19T14:54:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.870+0000 7f89e67fc640 1 -- 172.21.15.78:0/3969372487 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e002ccc0 con 0x7f89e8070720 2024-02-19T14:54:48.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.870+0000 7f89e67fc640 1 -- 172.21.15.78:0/3969372487 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e0034660 con 0x7f89e8070720 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 -- 172.21.15.78:0/3969372487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e8070720 msgr2=0x7f89e8070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/3969372487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e8070720 0x7f89e8070b70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f89e80c91b0 tx=0x7f89e002bdb0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 -- 172.21.15.78:0/3969372487 shutdown_connections 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/3969372487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e8070720 0x7f89e8070b70 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/3969372487 >> v2:172.21.15.150:3300/0 conn(0x7f89e80c7910 0x7f89e80c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 -- 172.21.15.78:0/3969372487 >> 172.21.15.78:0/3969372487 conn(0x7f89e806b4b0 msgr2=0x7f89e80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 -- 172.21.15.78:0/3969372487 shutdown_connections 2024-02-19T14:54:48.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.871+0000 7f89ee1ac640 1 -- 172.21.15.78:0/3969372487 wait complete. 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.872+0000 7f89ee1ac640 1 Processor -- start 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 -- start start 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89e8070720 0x7f89e8142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 0x7f89e8142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89e80c7910 0x7f89e81ebe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8151fb0 con 0x7f89e80c5560 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8143300 con 0x7f89e80c7910 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89ee1ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e80aa560 con 0x7f89e8070720 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 0x7f89e8142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 0x7f89e8142dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55318/0 (socket says 172.21.15.78:55318) 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.873+0000 7f89e6ffd640 1 -- 172.21.15.78:0/4143170390 learned_addr learned my addr 172.21.15.78:0/4143170390 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e77fe640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89e8070720 0x7f89e8142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e7fff640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89e80c7910 0x7f89e81ebe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e6ffd640 1 -- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89e8070720 msgr2=0x7f89e8142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e6ffd640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89e8070720 0x7f89e8142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e6ffd640 1 -- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89e80c7910 msgr2=0x7f89e81ebe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e6ffd640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89e80c7910 0x7f89e81ebe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e6ffd640 1 -- 172.21.15.78:0/4143170390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e0009a70 con 0x7f89e80c5560 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89e6ffd640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 0x7f89e8142dc0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00d5f0 tx=0x7f89dc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0125f0 con 0x7f89e80c5560 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc012750 con 0x7f89e80c5560 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e80c5ac0 con 0x7f89e80c5560 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc012b00 con 0x7f89e80c5560 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.874+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e80bd6c0 con 0x7f89e80c5560 2024-02-19T14:54:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.875+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e8074990 con 0x7f89e80c5560 2024-02-19T14:54:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.875+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f89dc012c60 con 0x7f89e80c5560 2024-02-19T14:54:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.876+0000 7f89cffff640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f89d406b810 0x7f89d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.876+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f89dc08aca0 con 0x7f89e80c5560 2024-02-19T14:54:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.876+0000 7f89e77fe640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f89d406b810 0x7f89d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.876+0000 7f89e77fe640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f89d406b810 0x7f89d406dcd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000f60 tx=0x7f89d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:48.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:48.884+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f89dc055120 con 0x7f89e80c5560 2024-02-19T14:54:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.135+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e8151ca0 con 0x7f89e80c5560 2024-02-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.136+0000 7f89cffff640 1 -- 172.21.15.78:0/4143170390 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6182 (secure 0 0 0) 0x7f89dc0587e0 con 0x7f89e80c5560 2024-02-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:45.326993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6657":{"gid":6657,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1047936932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1047936932},{"type":"v1","addr":"172.21.15.78:6837","nonce":1047936932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 61 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f89d406b810 msgr2=0x7f89d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f89d406b810 0x7f89d406dcd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000f60 tx=0x7f89d8009040 comp rx=0 tx=0).stop 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 msgr2=0x7f89e8142dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 0x7f89e8142dc0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00d5f0 tx=0x7f89dc00dab0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 shutdown_connections 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f89d406b810 0x7f89d406dcd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89e80c5560 0x7f89e8142dc0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89e80c7910 0x7f89e81ebe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 --2- 172.21.15.78:0/4143170390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89e8070720 0x7f89e8142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 >> 172.21.15.78:0/4143170390 conn(0x7f89e806b4b0 msgr2=0x7f89e8152440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 shutdown_connections 2024-02-19T14:54:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.138+0000 7f89ee1ac640 1 -- 172.21.15.78:0/4143170390 wait complete. 2024-02-19T14:54:49.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:49.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.250+0000 7f55003bc640 1 Processor -- start 2024-02-19T14:54:49.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.250+0000 7f55003bc640 1 -- start start 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f55003bc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54f80c4980 0x7f54f80c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f55003bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f80ae670 con 0x7f54f80705b0 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f55003bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f80716b0 con 0x7f54f80add70 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f55003bc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f8153780 con 0x7f54f80c4980 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f54fd957640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54f80c4980 0x7f54f80c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f54fe959640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f54f80705b0 0x7f54f80709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33674/0 (socket says 172.21.15.78:33674) 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f54fe959640 1 -- 172.21.15.78:0/3205834588 learned_addr learned my addr 172.21.15.78:0/3205834588 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f54fd957640 1 -- 172.21.15.78:0/3205834588 >> v1:172.21.15.150:6789/0 conn(0x7f54f80add70 legacy=0x7f54f806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.251+0000 7f54fd957640 1 -- 172.21.15.78:0/3205834588 >> v1:172.21.15.78:6789/0 conn(0x7f54f80705b0 legacy=0x7f54f80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.252+0000 7f54fd156640 1 -- 172.21.15.78:0/3205834588 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785283096 0 0) 0x7f54f80ae670 con 0x7f54f80705b0 2024-02-19T14:54:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.252+0000 7f54fd957640 1 -- 172.21.15.78:0/3205834588 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f8153a60 con 0x7f54f80c4980 2024-02-19T14:54:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.252+0000 7f54fd957640 1 --2- 172.21.15.78:0/3205834588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c4980 0x7f54f80c2f60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f54f40097a0 tx=0x7f54f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5a066b21514c93 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.252+0000 7f54fd156640 1 -- 172.21.15.78:0/3205834588 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f402e070 con 0x7f54f80c4980 2024-02-19T14:54:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.252+0000 7f54fd156640 1 -- 172.21.15.78:0/3205834588 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f40315a0 con 0x7f54f80c4980 2024-02-19T14:54:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.252+0000 7f54fd156640 1 -- 172.21.15.78:0/3205834588 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4031930 con 0x7f54f80c4980 2024-02-19T14:54:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 -- 172.21.15.78:0/3205834588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c4980 msgr2=0x7f54f80c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 --2- 172.21.15.78:0/3205834588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c4980 0x7f54f80c2f60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f54f40097a0 tx=0x7f54f402b840 comp rx=0 tx=0).stop 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 -- 172.21.15.78:0/3205834588 shutdown_connections 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 --2- 172.21.15.78:0/3205834588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c4980 0x7f54f80c2f60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 -- 172.21.15.78:0/3205834588 >> 172.21.15.78:0/3205834588 conn(0x7f54f80c85e0 msgr2=0x7f54f80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 -- 172.21.15.78:0/3205834588 shutdown_connections 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.253+0000 7f55003bc640 1 -- 172.21.15.78:0/3205834588 wait complete. 2024-02-19T14:54:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.255+0000 7f55003bc640 1 Processor -- start 2024-02-19T14:54:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.255+0000 7f55003bc640 1 -- start start 2024-02-19T14:54:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f55003bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80705b0 0x7f54f80abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f55003bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 0x7f54f81dd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f55003bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f80c4980 0x7f54f81ddf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f55003bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80bf980 con 0x7f54f80add70 2024-02-19T14:54:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f55003bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80beb00 con 0x7f54f80c4980 2024-02-19T14:54:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f55003bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80c8c20 con 0x7f54f80705b0 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 0x7f54f81dd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 0x7f54f81dd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55338/0 (socket says 172.21.15.78:55338) 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 -- 172.21.15.78:0/935533584 learned_addr learned my addr 172.21.15.78:0/935533584 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fe158640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80705b0 0x7f54f80abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fe959640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f80c4980 0x7f54f81ddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 -- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80705b0 msgr2=0x7f54f80abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80705b0 0x7f54f80abcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 -- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f80c4980 msgr2=0x7f54f81ddf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f80c4980 0x7f54f81ddf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fd957640 1 -- 172.21.15.78:0/935533584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f4009400 con 0x7f54f80add70 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fe158640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80705b0 0x7f54f80abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.256+0000 7f54fe959640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f80c4980 0x7f54f81ddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.257+0000 7f54fd957640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 0x7f54f81dd9d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f54f4034040 tx=0x7f54f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.257+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4032900 con 0x7f54f80add70 2024-02-19T14:54:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.257+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f4032a80 con 0x7f54f80add70 2024-02-19T14:54:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.257+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f81de450 con 0x7f54f80add70 2024-02-19T14:54:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.257+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4032e30 con 0x7f54f80add70 2024-02-19T14:54:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.257+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f80be260 con 0x7f54f80add70 2024-02-19T14:54:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.258+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f8074990 con 0x7f54f80add70 2024-02-19T14:54:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.258+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f54f403b3f0 con 0x7f54f80add70 2024-02-19T14:54:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.259+0000 7f54eeffd640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b840 0x7f54dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.259+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f54f40ac730 con 0x7f54f80add70 2024-02-19T14:54:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.259+0000 7f54fe158640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b840 0x7f54dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.259+0000 7f54fe158640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b840 0x7f54dc06dd00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f54e8003eb0 tx=0x7f54e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.263+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f54f4076c30 con 0x7f54f80add70 2024-02-19T14:54:49.317 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:54:49.314+0000 7f90b49af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.477+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f8071490 con 0x7f54f80add70 2024-02-19T14:54:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.479+0000 7f54eeffd640 1 -- 172.21.15.78:0/935533584 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+20980 (secure 0 0 0) 0x7f54f407a2f0 con 0x7f54f80add70 2024-02-19T14:54:49.482 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:49.483 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":82,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:42.308181+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b840 msgr2=0x7f54dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b840 0x7f54dc06dd00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f54e8003eb0 tx=0x7f54e8009040 comp rx=0 tx=0).stop 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 msgr2=0x7f54f81dd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 0x7f54f81dd9d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f54f4034040 tx=0x7f54f40046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 shutdown_connections 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b840 0x7f54dc06dd00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80add70 0x7f54f81dd9d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f80c4980 0x7f54f81ddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 --2- 172.21.15.78:0/935533584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80705b0 0x7f54f80abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.481+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 >> 172.21.15.78:0/935533584 conn(0x7f54f80c85e0 msgr2=0x7f54f8151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.482+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 shutdown_connections 2024-02-19T14:54:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.482+0000 7f55003bc640 1 -- 172.21.15.78:0/935533584 wait complete. 2024-02-19T14:54:49.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:54:24.290728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:54:25.042750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-19T14:54:34.841768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-19T14:54:36.252782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:49.496 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:49.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.605+0000 7f5565f7a640 1 Processor -- start 2024-02-19T14:54:49.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.605+0000 7f5565f7a640 1 -- start start 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f5565f7a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5560152700 0x7f55600c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f5565f7a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55600c5730 0x7f556006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f5565f7a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c8e00 0x7f556006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f5565f7a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556006cd20 con 0x7f55600c5730 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f5565f7a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55601551f0 con 0x7f5560152700 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f5565f7a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5560155490 con 0x7f55600c8e00 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55600c5730 0x7f556006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55600c5730 0x7f556006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55340/0 (socket says 172.21.15.78:55340) 2024-02-19T14:54:49.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 -- 172.21.15.78:0/3633266754 learned_addr learned my addr 172.21.15.78:0/3633266754 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555ffff640 1 --2- 172.21.15.78:0/3633266754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c8e00 0x7f556006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555f7fe640 1 --2- 172.21.15.78:0/3633266754 >> v2:172.21.15.150:3300/0 conn(0x7f5560152700 0x7f55600c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 -- 172.21.15.78:0/3633266754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c8e00 msgr2=0x7f556006c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 --2- 172.21.15.78:0/3633266754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c8e00 0x7f556006c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 -- 172.21.15.78:0/3633266754 >> v2:172.21.15.150:3300/0 conn(0x7f5560152700 msgr2=0x7f55600c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 --2- 172.21.15.78:0/3633266754 >> v2:172.21.15.150:3300/0 conn(0x7f5560152700 0x7f55600c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555effd640 1 -- 172.21.15.78:0/3633266754 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560155770 con 0x7f55600c5730 2024-02-19T14:54:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.607+0000 7f555f7fe640 1 --2- 172.21.15.78:0/3633266754 >> v2:172.21.15.150:3300/0 conn(0x7f5560152700 0x7f55600c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.608+0000 7f555effd640 1 --2- 172.21.15.78:0/3633266754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c5730 0x7f556006c2a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f556008bad0 tx=0x7f555402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8513d0c19bc37d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.608+0000 7f555e7fc640 1 -- 172.21.15.78:0/3633266754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554031870 con 0x7f55600c5730 2024-02-19T14:54:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.608+0000 7f555e7fc640 1 -- 172.21.15.78:0/3633266754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55540319d0 con 0x7f55600c5730 2024-02-19T14:54:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.608+0000 7f555e7fc640 1 -- 172.21.15.78:0/3633266754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554031d60 con 0x7f55600c5730 2024-02-19T14:54:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 -- 172.21.15.78:0/3633266754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c5730 msgr2=0x7f556006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 --2- 172.21.15.78:0/3633266754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c5730 0x7f556006c2a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f556008bad0 tx=0x7f555402b910 comp rx=0 tx=0).stop 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 -- 172.21.15.78:0/3633266754 shutdown_connections 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 --2- 172.21.15.78:0/3633266754 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c8e00 0x7f556006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 --2- 172.21.15.78:0/3633266754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c5730 0x7f556006c2a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 --2- 172.21.15.78:0/3633266754 >> v2:172.21.15.150:3300/0 conn(0x7f5560152700 0x7f55600c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 -- 172.21.15.78:0/3633266754 >> 172.21.15.78:0/3633266754 conn(0x7f55600cac90 msgr2=0x7f55600cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 -- 172.21.15.78:0/3633266754 shutdown_connections 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.609+0000 7f5565f7a640 1 -- 172.21.15.78:0/3633266754 wait complete. 2024-02-19T14:54:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.611+0000 7f5565f7a640 1 Processor -- start 2024-02-19T14:54:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.611+0000 7f5565f7a640 1 -- start start 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f5565f7a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c5730 0x7f5560153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f5565f7a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 0x7f55601ee010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f5565f7a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5560152700 0x7f55601ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f5565f7a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600ad010 con 0x7f55600c8e00 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f5565f7a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c4740 con 0x7f5560152700 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f5565f7a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600ac6e0 con 0x7f55600c5730 2024-02-19T14:54:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 0x7f55601ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 0x7f55601ee010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55352/0 (socket says 172.21.15.78:55352) 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 -- 172.21.15.78:0/384379119 learned_addr learned my addr 172.21.15.78:0/384379119 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555f7fe640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c5730 0x7f5560153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555ffff640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5560152700 0x7f55601ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 -- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c5730 msgr2=0x7f5560153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c5730 0x7f5560153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 -- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5560152700 msgr2=0x7f55601ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5560152700 0x7f55601ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.612+0000 7f555effd640 1 -- 172.21.15.78:0/384379119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55540094d0 con 0x7f55600c8e00 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.613+0000 7f555effd640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 0x7f55601ee010 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f555402b950 tx=0x7f55540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.613+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555403ad30 con 0x7f55600c8e00 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.613+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55540443f0 con 0x7f55600c8e00 2024-02-19T14:54:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.613+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55600ac9c0 con 0x7f55600c8e00 2024-02-19T14:54:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.613+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55540447a0 con 0x7f55600c8e00 2024-02-19T14:54:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.613+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5560154460 con 0x7f55600c8e00 2024-02-19T14:54:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.614+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55600ad1c0 con 0x7f55600c8e00 2024-02-19T14:54:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.614+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f555404d450 con 0x7f55600c8e00 2024-02-19T14:54:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.615+0000 7f5547fff640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f554c06b840 0x7f554c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.615+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f555402d040 con 0x7f55600c8e00 2024-02-19T14:54:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.615+0000 7f555f7fe640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f554c06b840 0x7f554c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.615+0000 7f555f7fe640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f554c06b840 0x7f554c06dd00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5550006e80 tx=0x7f5550008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.619+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f55600ad1c0 con 0x7f55600c8e00 2024-02-19T14:54:49.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.869+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5560074990 con 0x7f55600c8e00 2024-02-19T14:54:49.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.869+0000 7f5547fff640 1 -- 172.21.15.78:0/384379119 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5554077130 con 0x7f55600c8e00 2024-02-19T14:54:49.872 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f554c06b840 msgr2=0x7f554c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f554c06b840 0x7f554c06dd00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5550006e80 tx=0x7f5550008040 comp rx=0 tx=0).stop 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 msgr2=0x7f55601ee010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 0x7f55601ee010 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f555402b950 tx=0x7f55540046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 shutdown_connections 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f554c06b840 0x7f554c06dd00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55600c8e00 0x7f55601ee010 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5560152700 0x7f55601ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 --2- 172.21.15.78:0/384379119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55600c5730 0x7f5560153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 >> 172.21.15.78:0/384379119 conn(0x7f55600cac90 msgr2=0x7f556006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 shutdown_connections 2024-02-19T14:54:49.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.872+0000 7f5565f7a640 1 -- 172.21.15.78:0/384379119 wait complete. 2024-02-19T14:54:49.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:54:49.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.989+0000 7fc80b6b5640 1 Processor -- start 2024-02-19T14:54:49.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.989+0000 7fc80b6b5640 1 -- start start 2024-02-19T14:54:49.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc80b6b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8040713a0 0x7fc804071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc80b6b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc8040c8b60 0x7fc8041536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:49.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc80b6b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80406fe40 con 0x7fc8040c29e0 2024-02-19T14:54:49.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc80b6b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8040c32c0 con 0x7fc8040c8b60 2024-02-19T14:54:49.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc80b6b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc804070500 con 0x7fc8040713a0 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc809c52640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc8040c8b60 0x7fc8041536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc809c52640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc8040c8b60 0x7fc8041536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55354/0 (socket says 172.21.15.78:55354) 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc809c52640 1 -- 172.21.15.78:0/2748760871 learned_addr learned my addr 172.21.15.78:0/2748760871 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.990+0000 7fc809451640 1 --2- 172.21.15.78:0/2748760871 >> v2:172.21.15.150:3300/0 conn(0x7fc8040713a0 0x7fc804071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.991+0000 7fc809c52640 1 -- 172.21.15.78:0/2748760871 >> v2:172.21.15.150:3300/0 conn(0x7fc8040713a0 msgr2=0x7fc804071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.991+0000 7fc809c52640 1 --2- 172.21.15.78:0/2748760871 >> v2:172.21.15.150:3300/0 conn(0x7fc8040713a0 0x7fc804071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.991+0000 7fc809c52640 1 -- 172.21.15.78:0/2748760871 >> v1:172.21.15.150:6789/0 conn(0x7fc8040c29e0 legacy=0x7fc8040c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.991+0000 7fc809c52640 1 -- 172.21.15.78:0/2748760871 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804153bf0 con 0x7fc8040c8b60 2024-02-19T14:54:49.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.991+0000 7fc809451640 1 --2- 172.21.15.78:0/2748760871 >> v2:172.21.15.150:3300/0 conn(0x7fc8040713a0 0x7fc804071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:49.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.992+0000 7fc809c52640 1 --2- 172.21.15.78:0/2748760871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8041536b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc8040c7850 tx=0x7fc7fc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be3bada18733784 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:49.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.992+0000 7fc7fbfff640 1 -- 172.21.15.78:0/2748760871 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc02c9f0 con 0x7fc8040c8b60 2024-02-19T14:54:49.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.992+0000 7fc7fbfff640 1 -- 172.21.15.78:0/2748760871 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc02cb50 con 0x7fc8040c8b60 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.992+0000 7fc7fbfff640 1 -- 172.21.15.78:0/2748760871 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc0343f0 con 0x7fc8040c8b60 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 -- 172.21.15.78:0/2748760871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 msgr2=0x7fc8041536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/2748760871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8041536b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc8040c7850 tx=0x7fc7fc02be50 comp rx=0 tx=0).stop 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 -- 172.21.15.78:0/2748760871 shutdown_connections 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/2748760871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8041536b0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/2748760871 >> v2:172.21.15.150:3300/0 conn(0x7fc8040713a0 0x7fc804071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 -- 172.21.15.78:0/2748760871 >> 172.21.15.78:0/2748760871 conn(0x7fc8040aa300 msgr2=0x7fc8040aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 -- 172.21.15.78:0/2748760871 shutdown_connections 2024-02-19T14:54:49.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.993+0000 7fc80b6b5640 1 -- 172.21.15.78:0/2748760871 wait complete. 2024-02-19T14:54:49.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.994+0000 7fc80b6b5640 1 Processor -- start 2024-02-19T14:54:49.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.994+0000 7fc80b6b5640 1 -- start start 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.995+0000 7fc80b6b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040713a0 0x7fc804154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.995+0000 7fc80b6b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8040c29e0 0x7fc8040c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc80b6b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8040c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc80b6b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804154670 con 0x7fc8040c8b60 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc80b6b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8040ac4f0 con 0x7fc8040713a0 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc80b6b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8040ac7d0 con 0x7fc8040c29e0 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8040c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8040c23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55362/0 (socket says 172.21.15.78:55362) 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 -- 172.21.15.78:0/3803505894 learned_addr learned my addr 172.21.15.78:0/3803505894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809451640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040713a0 0x7fc804154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc808c50640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8040c29e0 0x7fc8040c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 -- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8040c29e0 msgr2=0x7fc8040c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8040c29e0 0x7fc8040c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 -- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040713a0 msgr2=0x7fc804154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040713a0 0x7fc804154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 -- 172.21.15.78:0/3803505894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc009b30 con 0x7fc8040c8b60 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809451640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040713a0 0x7fc804154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc809c52640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8040c23f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc036040 tx=0x7fc7fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc03e450 con 0x7fc8040c8b60 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc03e5d0 con 0x7fc8040c8b60 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8040acab0 con 0x7fc8040c8b60 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.996+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc03e980 con 0x7fc8040c8b60 2024-02-19T14:54:50.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.997+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80414b480 con 0x7fc8040c8b60 2024-02-19T14:54:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.998+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8040acfd0 con 0x7fc8040c8b60 2024-02-19T14:54:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.998+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc7fc03eae0 con 0x7fc8040c8b60 2024-02-19T14:54:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.998+0000 7fc7f9ffb640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc7e806b810 0x7fc7e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.999+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fc7fc0ad5a0 con 0x7fc8040c8b60 2024-02-19T14:54:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.999+0000 7fc809451640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc7e806b810 0x7fc7e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:49.999+0000 7fc809451640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc7e806b810 0x7fc7e806dcd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc800006fd0 tx=0x7fc800008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.007+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc8040acfd0 con 0x7fc8040c8b60 2024-02-19T14:54:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.283+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc8040acfd0 con 0x7fc8040c8b60 2024-02-19T14:54:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.284+0000 7fc7f9ffb640 1 -- 172.21.15.78:0/3803505894 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc8040acfd0 con 0x7fc8040c8b60 2024-02-19T14:54:50.287 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc7e806b810 msgr2=0x7fc7e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc7e806b810 0x7fc7e806dcd0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc800006fd0 tx=0x7fc800008040 comp rx=0 tx=0).stop 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 msgr2=0x7fc8040c23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8040c23f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc036040 tx=0x7fc7fc004a40 comp rx=0 tx=0).stop 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 shutdown_connections 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc7e806b810 0x7fc7e806dcd0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8040c8b60 0x7fc8040c23f0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8040c29e0 0x7fc8040c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 --2- 172.21.15.78:0/3803505894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040713a0 0x7fc804154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 >> 172.21.15.78:0/3803505894 conn(0x7fc8040aa300 msgr2=0x7fc8040aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 shutdown_connections 2024-02-19T14:54:50.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.286+0000 7fc80b6b5640 1 -- 172.21.15.78:0/3803505894 wait complete. 2024-02-19T14:54:50.299 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:54:50.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 Processor -- start 2024-02-19T14:54:50.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 -- start start 2024-02-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1be00c7070 0x7f1be006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1be006d350 0x7f1be00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be0155ef0 con 0x7f1be00c7070 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be006dec0 con 0x7f1be006d350 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.402+0000 7f1be5965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00c75c0 con 0x7f1be0152480 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1be00c7070 0x7f1be006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1be00c7070 0x7f1be006ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55366/0 (socket says 172.21.15.78:55366) 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 -- 172.21.15.78:0/40513161 learned_addr learned my addr 172.21.15.78:0/40513161 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bdeffd640 1 --2- 172.21.15.78:0/40513161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bdf7fe640 1 --2- 172.21.15.78:0/40513161 >> v2:172.21.15.150:3300/0 conn(0x7f1be006d350 0x7f1be00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 -- 172.21.15.78:0/40513161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 msgr2=0x7f1be01559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 --2- 172.21.15.78:0/40513161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be01559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 -- 172.21.15.78:0/40513161 >> v2:172.21.15.150:3300/0 conn(0x7f1be006d350 msgr2=0x7f1be00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 --2- 172.21.15.78:0/40513161 >> v2:172.21.15.150:3300/0 conn(0x7f1be006d350 0x7f1be00c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.403+0000 7f1bde7fc640 1 -- 172.21.15.78:0/40513161 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be00c78a0 con 0x7f1be00c7070 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1bde7fc640 1 --2- 172.21.15.78:0/40513161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be006ce10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009870 tx=0x7f1bd402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d80e154fa920eac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1bddffb640 1 -- 172.21.15.78:0/40513161 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4031640 con 0x7f1be00c7070 2024-02-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1bddffb640 1 -- 172.21.15.78:0/40513161 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd40317a0 con 0x7f1be00c7070 2024-02-19T14:54:50.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1bddffb640 1 -- 172.21.15.78:0/40513161 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4031b30 con 0x7f1be00c7070 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 -- 172.21.15.78:0/40513161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 msgr2=0x7f1be006ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 --2- 172.21.15.78:0/40513161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be006ce10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009870 tx=0x7f1bd402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 -- 172.21.15.78:0/40513161 shutdown_connections 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 --2- 172.21.15.78:0/40513161 >> v2:172.21.15.150:3300/0 conn(0x7f1be006d350 0x7f1be00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 --2- 172.21.15.78:0/40513161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be006ce10 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 --2- 172.21.15.78:0/40513161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be01559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 -- 172.21.15.78:0/40513161 >> 172.21.15.78:0/40513161 conn(0x7f1be00ca8a0 msgr2=0x7f1be00cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 -- 172.21.15.78:0/40513161 shutdown_connections 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.404+0000 7f1be5965640 1 -- 172.21.15.78:0/40513161 wait complete. 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.405+0000 7f1be5965640 1 Processor -- start 2024-02-19T14:54:50.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 -- start start 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be00bee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be00af700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1be00ab250 0x7f1be00ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00bf3a0 con 0x7f1be00c7070 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00aa320 con 0x7f1be00ab250 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1be5965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00aa570 con 0x7f1be0152480 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1bdeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be00bee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1bdeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be00bee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55372/0 (socket says 172.21.15.78:55372) 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.406+0000 7f1bdeffd640 1 -- 172.21.15.78:0/3019497549 learned_addr learned my addr 172.21.15.78:0/3019497549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bde7fc640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be00af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdf7fe640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1be00ab250 0x7f1be00ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdeffd640 1 -- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 msgr2=0x7f1be00af700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdeffd640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be00af700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdeffd640 1 -- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1be00ab250 msgr2=0x7f1be00ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdeffd640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1be00ab250 0x7f1be00ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdeffd640 1 -- 172.21.15.78:0/3019497549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40094d0 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdf7fe640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1be00ab250 0x7f1be00ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bdeffd640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be00bee60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d5f0 tx=0x7f1bd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00125f0 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0012750 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be00aa850 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0012b00 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.407+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be00af1e0 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.408+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be00af370 con 0x7f1be00c7070 2024-02-19T14:54:50.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.409+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1bd0012c60 con 0x7f1be00c7070 2024-02-19T14:54:50.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.409+0000 7f1bc7fff640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1bcc06b810 0x7f1bcc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.409+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f1bd008ac00 con 0x7f1be00c7070 2024-02-19T14:54:50.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.409+0000 7f1bde7fc640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1bcc06b810 0x7f1bcc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.409+0000 7f1bde7fc640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1bcc06b810 0x7f1bcc06dcd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4036000 tx=0x7f1bd403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.413+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1bd0091080 con 0x7f1be00c7070 2024-02-19T14:54:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.587+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1be0074990 con 0x7f1be00c7070 2024-02-19T14:54:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.588+0000 7f1bc7fff640 1 -- 172.21.15.78:0/3019497549 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1bd0055100 con 0x7f1be00c7070 2024-02-19T14:54:50.590 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.589+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1bcc06b810 msgr2=0x7f1bcc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.589+0000 7f1be5965640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1bcc06b810 0x7f1bcc06dcd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4036000 tx=0x7f1bd403c040 comp rx=0 tx=0).stop 2024-02-19T14:54:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.589+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 msgr2=0x7f1be00bee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.589+0000 7f1be5965640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be00bee60 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d5f0 tx=0x7f1bd000dab0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 shutdown_connections 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1bcc06b810 0x7f1bcc06dcd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1be00ab250 0x7f1be00ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1be0152480 0x7f1be00af700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 --2- 172.21.15.78:0/3019497549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00c7070 0x7f1be00bee60 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 >> 172.21.15.78:0/3019497549 conn(0x7f1be00ca8a0 msgr2=0x7f1be014d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 shutdown_connections 2024-02-19T14:54:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.590+0000 7f1be5965640 1 -- 172.21.15.78:0/3019497549 wait complete. 2024-02-19T14:54:50.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-19T14:54:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.704+0000 7f465287b640 1 Processor -- start 2024-02-19T14:54:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.704+0000 7f465287b640 1 -- start start 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.705+0000 7f465287b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f464c149790 0x7f464c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f465287b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0723d0 0x7f464c072820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f465287b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c06f420 con 0x7f464c152c30 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f465287b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c149b70 con 0x7f464c149790 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f465287b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0ab760 con 0x7f464c0723d0 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f464c149790 0x7f464c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f464c149790 0x7f464c06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55378/0 (socket says 172.21.15.78:55378) 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 -- 172.21.15.78:0/1045550152 learned_addr learned my addr 172.21.15.78:0/1045550152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 -- 172.21.15.78:0/1045550152 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0723d0 msgr2=0x7f464c072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 --2- 172.21.15.78:0/1045550152 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0723d0 0x7f464c072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 -- 172.21.15.78:0/1045550152 >> v1:172.21.15.150:6789/0 conn(0x7f464c152c30 legacy=0x7f464c153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.706+0000 7f464bfff640 1 -- 172.21.15.78:0/1045550152 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0aba40 con 0x7f464c149790 2024-02-19T14:54:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f464bfff640 1 --2- 172.21.15.78:0/1045550152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c149790 0x7f464c06ed40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4640009870 tx=0x7f464002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193045ae7b9ccb6e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f464affd640 1 -- 172.21.15.78:0/1045550152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640031740 con 0x7f464c149790 2024-02-19T14:54:50.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f464affd640 1 -- 172.21.15.78:0/1045550152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46400318a0 con 0x7f464c149790 2024-02-19T14:54:50.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f464affd640 1 -- 172.21.15.78:0/1045550152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640031c30 con 0x7f464c149790 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f465287b640 1 -- 172.21.15.78:0/1045550152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c149790 msgr2=0x7f464c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f465287b640 1 --2- 172.21.15.78:0/1045550152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c149790 0x7f464c06ed40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4640009870 tx=0x7f464002b7e0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f465287b640 1 -- 172.21.15.78:0/1045550152 shutdown_connections 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f465287b640 1 --2- 172.21.15.78:0/1045550152 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0723d0 0x7f464c072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f465287b640 1 --2- 172.21.15.78:0/1045550152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c149790 0x7f464c06ed40 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.707+0000 7f465287b640 1 -- 172.21.15.78:0/1045550152 >> 172.21.15.78:0/1045550152 conn(0x7f464c0718e0 msgr2=0x7f464c071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.708+0000 7f465287b640 1 -- 172.21.15.78:0/1045550152 shutdown_connections 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.708+0000 7f465287b640 1 -- 172.21.15.78:0/1045550152 wait complete. 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 Processor -- start 2024-02-19T14:54:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 -- start start 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0723d0 0x7f464c150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c149790 0x7f464c1deb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 0x7f464c1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1515a0 con 0x7f464c152c30 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1512c0 con 0x7f464c0723d0 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.709+0000 7f465287b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0a97f0 con 0x7f464c149790 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 0x7f464c1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 0x7f464c1ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55394/0 (socket says 172.21.15.78:55394) 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 -- 172.21.15.78:0/3591834053 learned_addr learned my addr 172.21.15.78:0/3591834053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f464bfff640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0723d0 0x7f464c150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f464b7fe640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c149790 0x7f464c1deb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 -- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c149790 msgr2=0x7f464c1deb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c149790 0x7f464c1deb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 -- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0723d0 msgr2=0x7f464c150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0723d0 0x7f464c150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 -- 172.21.15.78:0/3591834053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46400094d0 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4650e18640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 0x7f464c1ee810 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f464400c780 tx=0x7f464400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644003d80 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.710+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464400ce90 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.711+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c0a9ad0 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.711+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440052b0 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.711+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c1df070 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.711+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c0a9ff0 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.712+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4644005410 con 0x7f464c152c30 2024-02-19T14:54:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.712+0000 7f4648ff9640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f463806b810 0x7f463806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:50.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.712+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f4644089ef0 con 0x7f464c152c30 2024-02-19T14:54:50.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.713+0000 7f464bfff640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f463806b810 0x7f463806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:50.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.713+0000 7f464bfff640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f463806b810 0x7f463806dcd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4640035000 tx=0x7f4640008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:50.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.721+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f464c0a9ff0 con 0x7f464c152c30 2024-02-19T14:54:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.959+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f464c0a9ff0 con 0x7f464c152c30 2024-02-19T14:54:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.959+0000 7f4648ff9640 1 -- 172.21.15.78:0/3591834053 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v82) v1 ==== 105+0+11 (secure 0 0 0) 0x7f464c0a9ff0 con 0x7f464c152c30 2024-02-19T14:54:50.962 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f463806b810 msgr2=0x7f463806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f463806b810 0x7f463806dcd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4640035000 tx=0x7f4640008030 comp rx=0 tx=0).stop 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 msgr2=0x7f464c1ee810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 0x7f464c1ee810 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f464400c780 tx=0x7f464400cc40 comp rx=0 tx=0).stop 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 shutdown_connections 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f463806b810 0x7f463806dcd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f464c152c30 0x7f464c1ee810 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c149790 0x7f464c1deb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 --2- 172.21.15.78:0/3591834053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0723d0 0x7f464c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:50.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 >> 172.21.15.78:0/3591834053 conn(0x7f464c0718e0 msgr2=0x7f464c06ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:50.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 shutdown_connections 2024-02-19T14:54:50.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:50.962+0000 7f465287b640 1 -- 172.21.15.78:0/3591834053 wait complete. 2024-02-19T14:54:50.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-19T14:54:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.113+0000 7f0417360640 1 Processor -- start 2024-02-19T14:54:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.113+0000 7f0417360640 1 -- start start 2024-02-19T14:54:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f0417360640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f041014f1a0 0x7f0410072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f0417360640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f041006ea50 0x7f0410152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f0417360640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04100c5970 con 0x7f041006fbc0 2024-02-19T14:54:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f0417360640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04100c5750 con 0x7f041014f1a0 2024-02-19T14:54:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f0417360640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04100c5bc0 con 0x7f041006ea50 2024-02-19T14:54:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f041014f1a0 0x7f0410072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f041014f1a0 0x7f0410072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55404/0 (socket says 172.21.15.78:55404) 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 -- 172.21.15.78:0/2264613371 learned_addr learned my addr 172.21.15.78:0/2264613371 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04148fb640 1 --2- 172.21.15.78:0/2264613371 >> v2:172.21.15.150:3300/0 conn(0x7f041006ea50 0x7f0410152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 -- 172.21.15.78:0/2264613371 >> v2:172.21.15.150:3300/0 conn(0x7f041006ea50 msgr2=0x7f0410152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 --2- 172.21.15.78:0/2264613371 >> v2:172.21.15.150:3300/0 conn(0x7f041006ea50 0x7f0410152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 -- 172.21.15.78:0/2264613371 >> v1:172.21.15.150:6789/0 conn(0x7f041006fbc0 legacy=0x7f04100c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.114+0000 7f04150fc640 1 -- 172.21.15.78:0/2264613371 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100c5ea0 con 0x7f041014f1a0 2024-02-19T14:54:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.115+0000 7f04150fc640 1 --2- 172.21.15.78:0/2264613371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041014f1a0 0x7f0410072d70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f041006f090 tx=0x7f040c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2622a7f2c1f64e03 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0407fff640 1 -- 172.21.15.78:0/2264613371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c031750 con 0x7f041014f1a0 2024-02-19T14:54:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0407fff640 1 -- 172.21.15.78:0/2264613371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c0318b0 con 0x7f041014f1a0 2024-02-19T14:54:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0407fff640 1 -- 172.21.15.78:0/2264613371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c031c40 con 0x7f041014f1a0 2024-02-19T14:54:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 -- 172.21.15.78:0/2264613371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041014f1a0 msgr2=0x7f0410072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 --2- 172.21.15.78:0/2264613371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041014f1a0 0x7f0410072d70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f041006f090 tx=0x7f040c02b7f0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 -- 172.21.15.78:0/2264613371 shutdown_connections 2024-02-19T14:54:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 --2- 172.21.15.78:0/2264613371 >> v2:172.21.15.150:3300/0 conn(0x7f041006ea50 0x7f0410152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 --2- 172.21.15.78:0/2264613371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041014f1a0 0x7f0410072d70 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 -- 172.21.15.78:0/2264613371 >> 172.21.15.78:0/2264613371 conn(0x7f041006c170 msgr2=0x7f041006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 -- 172.21.15.78:0/2264613371 shutdown_connections 2024-02-19T14:54:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.116+0000 7f0417360640 1 -- 172.21.15.78:0/2264613371 wait complete. 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.117+0000 7f0417360640 1 Processor -- start 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.117+0000 7f0417360640 1 -- start start 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f0417360640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 0x7f04100c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f0417360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f041006fbc0 0x7f04100aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f0417360640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f041014f1a0 0x7f04101ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f0417360640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04101ee250 con 0x7f041006ea50 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f0417360640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04100c22e0 con 0x7f041006fbc0 2024-02-19T14:54:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f0417360640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04101edc40 con 0x7f041014f1a0 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 0x7f04100c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 0x7f04100c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55410/0 (socket says 172.21.15.78:55410) 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 -- 172.21.15.78:0/3237450759 learned_addr learned my addr 172.21.15.78:0/3237450759 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 -- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f041014f1a0 msgr2=0x7f04101ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04148fb640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f041006fbc0 0x7f04100aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f041014f1a0 0x7f04101ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 -- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f041006fbc0 msgr2=0x7f04100aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f041006fbc0 0x7f04100aa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.118+0000 7f04150fc640 1 -- 172.21.15.78:0/3237450759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c0094d0 con 0x7f041006ea50 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.119+0000 7f04150fc640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 0x7f04100c4b20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f040c000c00 tx=0x7f040c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.119+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c033c80 con 0x7f041006ea50 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.119+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c033de0 con 0x7f041006ea50 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.119+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04100aab70 con 0x7f041006ea50 2024-02-19T14:54:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.119+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c03c640 con 0x7f041006ea50 2024-02-19T14:54:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.119+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04101edee0 con 0x7f041006ea50 2024-02-19T14:54:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.120+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04101ee070 con 0x7f041006ea50 2024-02-19T14:54:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.121+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f040c03a040 con 0x7f041006ea50 2024-02-19T14:54:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.121+0000 7f0405ffb640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03f406b810 0x7f03f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.121+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f040c0ac350 con 0x7f041006ea50 2024-02-19T14:54:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.121+0000 7f04148fb640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03f406b810 0x7f03f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.122+0000 7f04148fb640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03f406b810 0x7f03f406dcd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f04100a77a0 tx=0x7f0400008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.126+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f040c0767b0 con 0x7f041006ea50 2024-02-19T14:54:51.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.402+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f0410074990 con 0x7f041006ea50 2024-02-19T14:54:51.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.403+0000 7f0405ffb640 1 -- 172.21.15.78:0/3237450759 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v82) v1 ==== 101+0+11 (secure 0 0 0) 0x7f040c079e70 con 0x7f041006ea50 2024-02-19T14:54:51.405 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03f406b810 msgr2=0x7f03f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03f406b810 0x7f03f406dcd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f04100a77a0 tx=0x7f0400008040 comp rx=0 tx=0).stop 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 msgr2=0x7f04100c4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 0x7f04100c4b20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f040c000c00 tx=0x7f040c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 shutdown_connections 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03f406b810 0x7f03f406dcd0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f041014f1a0 0x7f04101ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f041006fbc0 0x7f04100aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 --2- 172.21.15.78:0/3237450759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f041006ea50 0x7f04100c4b20 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 >> 172.21.15.78:0/3237450759 conn(0x7f041006c170 msgr2=0x7f04100c09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:51.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 shutdown_connections 2024-02-19T14:54:51.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.405+0000 7f0417360640 1 -- 172.21.15.78:0/3237450759 wait complete. 2024-02-19T14:54:51.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.519+0000 7fca5821c640 1 Processor -- start 2024-02-19T14:54:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.519+0000 7fca5821c640 1 -- start start 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.520+0000 7fca5821c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500add70 0x7fca500c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.520+0000 7fca5821c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca500c2f60 0x7fca5006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.520+0000 7fca5821c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500ae670 con 0x7fca5006f530 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.520+0000 7fca5821c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500716b0 con 0x7fca500c2f60 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.520+0000 7fca5821c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500c33a0 con 0x7fca500add70 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca557b7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca500c2f60 0x7fca5006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca567b9640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fca5006f530 0x7fca500705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:56906/0 (socket says 172.21.15.78:56906) 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca55fb8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500add70 0x7fca500c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca557b7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca500c2f60 0x7fca5006eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:42658/0 (socket says 172.21.15.78:42658) 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca567b9640 1 -- 172.21.15.78:0/884523775 learned_addr learned my addr 172.21.15.78:0/884523775 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca557b7640 1 -- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500add70 msgr2=0x7fca500c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca557b7640 1 --2- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500add70 0x7fca500c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca557b7640 1 -- 172.21.15.78:0/884523775 >> v1:172.21.15.150:6789/0 conn(0x7fca5006f530 legacy=0x7fca500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca54fb6640 1 -- 172.21.15.78:0/884523775 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131742223 0 0) 0x7fca500ae670 con 0x7fca5006f530 2024-02-19T14:54:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca557b7640 1 -- 172.21.15.78:0/884523775 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca50153780 con 0x7fca500c2f60 2024-02-19T14:54:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.521+0000 7fca55fb8640 1 --2- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500add70 0x7fca500c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.522+0000 7fca557b7640 1 --2- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca500c2f60 0x7fca5006eff0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fca400097a0 tx=0x7fca4002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82a35c9adb9e25b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.522+0000 7fca54fb6640 1 -- 172.21.15.78:0/884523775 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4002e070 con 0x7fca500c2f60 2024-02-19T14:54:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.522+0000 7fca54fb6640 1 -- 172.21.15.78:0/884523775 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca400315a0 con 0x7fca500c2f60 2024-02-19T14:54:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.522+0000 7fca54fb6640 1 -- 172.21.15.78:0/884523775 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40031930 con 0x7fca500c2f60 2024-02-19T14:54:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 -- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca500c2f60 msgr2=0x7fca5006eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 --2- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca500c2f60 0x7fca5006eff0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fca400097a0 tx=0x7fca4002b840 comp rx=0 tx=0).stop 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 -- 172.21.15.78:0/884523775 shutdown_connections 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 --2- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca500c2f60 0x7fca5006eff0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 --2- 172.21.15.78:0/884523775 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500add70 0x7fca500c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 -- 172.21.15.78:0/884523775 >> 172.21.15.78:0/884523775 conn(0x7fca500c85e0 msgr2=0x7fca500c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 -- 172.21.15.78:0/884523775 shutdown_connections 2024-02-19T14:54:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.523+0000 7fca5821c640 1 -- 172.21.15.78:0/884523775 wait complete. 2024-02-19T14:54:51.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.524+0000 7fca5821c640 1 Processor -- start 2024-02-19T14:54:51.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.524+0000 7fca5821c640 1 -- start start 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca5821c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca5006f530 0x7fca501dd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca5821c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 0x7fca5014dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca5821c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500c2f60 0x7fca5014e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca5821c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501e8c60 con 0x7fca500add70 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca5821c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501e8f40 con 0x7fca5006f530 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca5821c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca501e9220 con 0x7fca500c2f60 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 0x7fca5014dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 0x7fca5014dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55422/0 (socket says 172.21.15.78:55422) 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 -- 172.21.15.78:0/3144796845 learned_addr learned my addr 172.21.15.78:0/3144796845 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca567b9640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500c2f60 0x7fca5014e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca55fb8640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca5006f530 0x7fca501dd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 -- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500c2f60 msgr2=0x7fca5014e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500c2f60 0x7fca5014e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 -- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca5006f530 msgr2=0x7fca501dd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca5006f530 0x7fca501dd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 -- 172.21.15.78:0/3144796845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c009400 con 0x7fca500add70 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.526+0000 7fca557b7640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 0x7fca5014dd90 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fca400097a0 tx=0x7fca400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.527+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40004230 con 0x7fca500add70 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.527+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca40004390 con 0x7fca500add70 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.527+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca40009400 con 0x7fca500add70 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.527+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40003850 con 0x7fca500add70 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.527+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca501e97e0 con 0x7fca500add70 2024-02-19T14:54:51.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.527+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca500676a0 con 0x7fca500add70 2024-02-19T14:54:51.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.529+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fca400039b0 con 0x7fca500add70 2024-02-19T14:54:51.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.529+0000 7fca467fc640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca3406b840 0x7fca3406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.529+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fca400ac620 con 0x7fca500add70 2024-02-19T14:54:51.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.529+0000 7fca55fb8640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca3406b840 0x7fca3406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.530+0000 7fca55fb8640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca3406b840 0x7fca3406dd00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009cd0 tx=0x7fca4c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.538+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fca40076cf0 con 0x7fca500add70 2024-02-19T14:54:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.819+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca501ebe80 con 0x7fca500add70 2024-02-19T14:54:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.820+0000 7fca467fc640 1 -- 172.21.15.78:0/3144796845 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6182 (secure 0 0 0) 0x7fca501ebe80 con 0x7fca500add70 2024-02-19T14:54:51.822 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:51.822 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:45.326993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6657":{"gid":6657,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1047936932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1047936932},{"type":"v1","addr":"172.21.15.78:6837","nonce":1047936932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 61 2024-02-19T14:54:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca3406b840 msgr2=0x7fca3406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca3406b840 0x7fca3406dd00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009cd0 tx=0x7fca4c009190 comp rx=0 tx=0).stop 2024-02-19T14:54:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 msgr2=0x7fca5014dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 0x7fca5014dd90 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fca400097a0 tx=0x7fca400046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 shutdown_connections 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca3406b840 0x7fca3406dd00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500add70 0x7fca5014dd90 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca500c2f60 0x7fca5014e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 --2- 172.21.15.78:0/3144796845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca5006f530 0x7fca501dd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 >> 172.21.15.78:0/3144796845 conn(0x7fca500c85e0 msgr2=0x7fca500acb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 shutdown_connections 2024-02-19T14:54:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.822+0000 7fca5821c640 1 -- 172.21.15.78:0/3144796845 wait complete. 2024-02-19T14:54:51.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:51.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 Processor -- start 2024-02-19T14:54:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 -- start start 2024-02-19T14:54:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71840c6d60 0x7f7184154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718406ff80 0x7f71840c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718414f730 con 0x7f718406e830 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71840c1940 con 0x7f71840c6d60 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.934+0000 7f718afe2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840c16d0 con 0x7f718406ff80 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7188d7e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71840c6d60 0x7f7184154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7183fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f718406e830 0x7f71840c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33678/0 (socket says 172.21.15.78:33678) 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7183fff640 1 -- 172.21.15.78:0/4182161352 learned_addr learned my addr 172.21.15.78:0/4182161352 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f718957f640 1 --2- 172.21.15.78:0/4182161352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718406ff80 0x7f71840c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7188d7e640 1 -- 172.21.15.78:0/4182161352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718406ff80 msgr2=0x7f71840c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7188d7e640 1 --2- 172.21.15.78:0/4182161352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718406ff80 0x7f71840c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f71837fe640 1 -- 172.21.15.78:0/4182161352 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447022784 0 0) 0x7f718414f730 con 0x7f718406e830 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7188d7e640 1 -- 172.21.15.78:0/4182161352 >> v1:172.21.15.78:6789/0 conn(0x7f718406e830 legacy=0x7f71840c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.935+0000 7f7188d7e640 1 -- 172.21.15.78:0/4182161352 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184072f70 con 0x7f71840c6d60 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f7188d7e640 1 --2- 172.21.15.78:0/4182161352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71840c6d60 0x7f7184154f70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f71780097a0 tx=0x7f717802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751e6cfdec933bcd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f71837fe640 1 -- 172.21.15.78:0/4182161352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802e070 con 0x7f71840c6d60 2024-02-19T14:54:51.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f71837fe640 1 -- 172.21.15.78:0/4182161352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71780315a0 con 0x7f71840c6d60 2024-02-19T14:54:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f71837fe640 1 -- 172.21.15.78:0/4182161352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7178031930 con 0x7f71840c6d60 2024-02-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 -- 172.21.15.78:0/4182161352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71840c6d60 msgr2=0x7f7184154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 --2- 172.21.15.78:0/4182161352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71840c6d60 0x7f7184154f70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f71780097a0 tx=0x7f717802b840 comp rx=0 tx=0).stop 2024-02-19T14:54:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 -- 172.21.15.78:0/4182161352 shutdown_connections 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 --2- 172.21.15.78:0/4182161352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718406ff80 0x7f71840c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 --2- 172.21.15.78:0/4182161352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71840c6d60 0x7f7184154f70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 -- 172.21.15.78:0/4182161352 >> 172.21.15.78:0/4182161352 conn(0x7f718406c170 msgr2=0x7f718406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 -- 172.21.15.78:0/4182161352 shutdown_connections 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.936+0000 7f718afe2640 1 -- 172.21.15.78:0/4182161352 wait complete. 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.937+0000 7f718afe2640 1 Processor -- start 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 -- start start 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718406e830 0x7f71841573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 0x7f7184146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71840c6d60 0x7f718414f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7184146af0 con 0x7f718406ff80 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7184152710 con 0x7f718406e830 2024-02-19T14:54:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f718afe2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71841eb720 con 0x7f71840c6d60 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f7183fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 0x7f7184146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f7183fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 0x7f7184146dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55440/0 (socket says 172.21.15.78:55440) 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.938+0000 7f7183fff640 1 -- 172.21.15.78:0/4293211869 learned_addr learned my addr 172.21.15.78:0/4293211869 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f718957f640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71840c6d60 0x7f718414f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7183fff640 1 -- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71840c6d60 msgr2=0x7f718414f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7188d7e640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718406e830 0x7f71841573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7183fff640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71840c6d60 0x7f718414f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7183fff640 1 -- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718406e830 msgr2=0x7f71841573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7183fff640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718406e830 0x7f71841573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7183fff640 1 -- 172.21.15.78:0/4293211869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178009400 con 0x7f718406ff80 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f7183fff640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 0x7f7184146dd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f717000c410 tx=0x7f717000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71700074e0 con 0x7f718406ff80 2024-02-19T14:54:51.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170007660 con 0x7f718406ff80 2024-02-19T14:54:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170004a20 con 0x7f718406ff80 2024-02-19T14:54:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71840c1ec0 con 0x7f718406ff80 2024-02-19T14:54:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.939+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7184145e30 con 0x7f718406ff80 2024-02-19T14:54:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.940+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71841eb570 con 0x7f718406ff80 2024-02-19T14:54:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.941+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7170004b80 con 0x7f718406ff80 2024-02-19T14:54:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.942+0000 7f71817fa640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f716806b840 0x7f716806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.942+0000 7f7188d7e640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f716806b840 0x7f716806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.942+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f717008a4d0 con 0x7f718406ff80 2024-02-19T14:54:51.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.942+0000 7f7188d7e640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f716806b840 0x7f716806dd00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7178036000 tx=0x7f717803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:51.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:51.946+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7170058500 con 0x7f718406ff80 2024-02-19T14:54:52.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.129+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7184074990 con 0x7f718406ff80 2024-02-19T14:54:52.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.130+0000 7f71817fa640 1 -- 172.21.15.78:0/4293211869 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+20980 (secure 0 0 0) 0x7f7170058090 con 0x7f718406ff80 2024-02-19T14:54:52.133 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:52.133 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":82,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:42.308181+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f716806b840 msgr2=0x7f716806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f716806b840 0x7f716806dd00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7178036000 tx=0x7f717803d040 comp rx=0 tx=0).stop 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 msgr2=0x7f7184146dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 0x7f7184146dd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f717000c410 tx=0x7f717000c8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 shutdown_connections 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f716806b840 0x7f716806dd00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f718406ff80 0x7f7184146dd0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71840c6d60 0x7f718414f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 --2- 172.21.15.78:0/4293211869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718406e830 0x7f71841573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 >> 172.21.15.78:0/4293211869 conn(0x7f718406c170 msgr2=0x7f71840721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 shutdown_connections 2024-02-19T14:54:52.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.133+0000 7f718afe2640 1 -- 172.21.15.78:0/4293211869 wait complete. 2024-02-19T14:54:52.147 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:54:52.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:52.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 Processor -- start 2024-02-19T14:54:52.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 -- start start 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c25e0 0x7f1aa81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1aa80c29c0 0x7f1aa80c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa8072250 con 0x7f1aa80c49c0 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa8152c90 con 0x7f1aa80c29c0 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aae6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa8072480 con 0x7f1aa80c25e0 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aa77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1aa80c29c0 0x7f1aa80c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aa77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1aa80c29c0 0x7f1aa80c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55444/0 (socket says 172.21.15.78:55444) 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.245+0000 7f1aa77fe640 1 -- 172.21.15.78:0/2431226069 learned_addr learned my addr 172.21.15.78:0/2431226069 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.246+0000 7f1aa7fff640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c25e0 0x7f1aa81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.246+0000 7f1aa77fe640 1 -- 172.21.15.78:0/2431226069 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c25e0 msgr2=0x7f1aa81526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.246+0000 7f1aa77fe640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c25e0 0x7f1aa81526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.246+0000 7f1aa77fe640 1 -- 172.21.15.78:0/2431226069 >> v1:172.21.15.150:6789/0 conn(0x7f1aa80c49c0 legacy=0x7f1aa806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.246+0000 7f1aa77fe640 1 -- 172.21.15.78:0/2431226069 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa80c3eb0 con 0x7f1aa80c29c0 2024-02-19T14:54:52.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.246+0000 7f1aa7fff640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c25e0 0x7f1aa81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:52.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.247+0000 7f1aa77fe640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c29c0 0x7f1aa80c4480 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009870 tx=0x7f1a9c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c7a768006364bd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.247+0000 7f1aa6ffd640 1 -- 172.21.15.78:0/2431226069 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c0317c0 con 0x7f1aa80c29c0 2024-02-19T14:54:52.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.247+0000 7f1aa6ffd640 1 -- 172.21.15.78:0/2431226069 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c031920 con 0x7f1aa80c29c0 2024-02-19T14:54:52.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.247+0000 7f1aa6ffd640 1 -- 172.21.15.78:0/2431226069 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c031cb0 con 0x7f1aa80c29c0 2024-02-19T14:54:52.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 -- 172.21.15.78:0/2431226069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c29c0 msgr2=0x7f1aa80c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c29c0 0x7f1aa80c4480 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009870 tx=0x7f1a9c02b860 comp rx=0 tx=0).stop 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 -- 172.21.15.78:0/2431226069 shutdown_connections 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c29c0 0x7f1aa80c4480 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/2431226069 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c25e0 0x7f1aa81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 -- 172.21.15.78:0/2431226069 >> 172.21.15.78:0/2431226069 conn(0x7f1aa806b270 msgr2=0x7f1aa806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 -- 172.21.15.78:0/2431226069 shutdown_connections 2024-02-19T14:54:52.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.248+0000 7f1aae6c9640 1 -- 172.21.15.78:0/2431226069 wait complete. 2024-02-19T14:54:52.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.249+0000 7f1aae6c9640 1 Processor -- start 2024-02-19T14:54:52.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.250+0000 7f1aae6c9640 1 -- start start 2024-02-19T14:54:52.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 0x7f1aa814ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c29c0 0x7f1aa80bf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1aa8150b10 0x7f1aa80bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa81517b0 con 0x7f1aa80c25e0 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80c0390 con 0x7f1aa8150b10 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80c0800 con 0x7f1aa80c29c0 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 0x7f1aa814ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 0x7f1aa814ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55450/0 (socket says 172.21.15.78:55450) 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 -- 172.21.15.78:0/275834019 learned_addr learned my addr 172.21.15.78:0/275834019 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa77fe640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c29c0 0x7f1aa80bf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aacc66640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1aa8150b10 0x7f1aa80bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 -- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c29c0 msgr2=0x7f1aa80bf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c29c0 0x7f1aa80bf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 -- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1aa8150b10 msgr2=0x7f1aa80bfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1aa8150b10 0x7f1aa80bfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 -- 172.21.15.78:0/275834019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0094d0 con 0x7f1aa80c25e0 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa77fe640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c29c0 0x7f1aa80bf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:52.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa7fff640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 0x7f1aa814ca30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800c820 tx=0x7f1a9800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98007a30 con 0x7f1aa80c25e0 2024-02-19T14:54:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a98007b90 con 0x7f1aa80c25e0 2024-02-19T14:54:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800ce10 con 0x7f1aa80c25e0 2024-02-19T14:54:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.251+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa81ec710 con 0x7f1aa80c25e0 2024-02-19T14:54:52.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.252+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa81513d0 con 0x7f1aa80c25e0 2024-02-19T14:54:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.253+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa81ecc30 con 0x7f1aa80c25e0 2024-02-19T14:54:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.254+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1a980113f0 con 0x7f1aa80c25e0 2024-02-19T14:54:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.254+0000 7f1aa4ff9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a9406b810 0x7f1a9406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.254+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f1a9808a840 con 0x7f1aa80c25e0 2024-02-19T14:54:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.254+0000 7f1aa77fe640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a9406b810 0x7f1a9406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.254+0000 7f1aa77fe640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a9406b810 0x7f1a9406dcd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c036000 tx=0x7f1a9c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.262+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a98054d40 con 0x7f1aa80c25e0 2024-02-19T14:54:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.480+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aa8074990 con 0x7f1aa80c25e0 2024-02-19T14:54:52.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.481+0000 7f1aa4ff9640 1 -- 172.21.15.78:0/275834019 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6182 (secure 0 0 0) 0x7f1a98058400 con 0x7f1aa80c25e0 2024-02-19T14:54:52.483 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:52.483 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:45.326993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6657":{"gid":6657,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1047936932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1047936932},{"type":"v1","addr":"172.21.15.78:6837","nonce":1047936932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:52.484 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 61 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.482+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a9406b810 msgr2=0x7f1a9406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.482+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a9406b810 0x7f1a9406dcd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c036000 tx=0x7f1a9c03d040 comp rx=0 tx=0).stop 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 msgr2=0x7f1aa814ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 0x7f1aa814ca30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800c820 tx=0x7f1a9800cce0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 shutdown_connections 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a9406b810 0x7f1a9406dcd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1aa8150b10 0x7f1aa80bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1aa80c29c0 0x7f1aa80bf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 --2- 172.21.15.78:0/275834019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1aa80c25e0 0x7f1aa814ca30 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 >> 172.21.15.78:0/275834019 conn(0x7f1aa806b270 msgr2=0x7f1aa806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 shutdown_connections 2024-02-19T14:54:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.483+0000 7f1aae6c9640 1 -- 172.21.15.78:0/275834019 wait complete. 2024-02-19T14:54:52.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:52.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.609+0000 7fa12b593640 1 Processor -- start 2024-02-19T14:54:52.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.609+0000 7fa12b593640 1 -- start start 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa12b593640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c78a0 0x7fa124159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa12b593640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa124155010 0x7fa1241523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa12b593640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12406ffa0 con 0x7fa1240c4d80 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa12b593640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12406b720 con 0x7fa124155010 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa12b593640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12406fc70 con 0x7fa1240c78a0 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa129b30640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa124155010 0x7fa1241523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa129b30640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa124155010 0x7fa1241523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55462/0 (socket says 172.21.15.78:55462) 2024-02-19T14:54:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.610+0000 7fa129b30640 1 -- 172.21.15.78:0/2571978855 learned_addr learned my addr 172.21.15.78:0/2571978855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa12932f640 1 --2- 172.21.15.78:0/2571978855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c78a0 0x7fa124159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa129b30640 1 -- 172.21.15.78:0/2571978855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c78a0 msgr2=0x7fa124159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa129b30640 1 --2- 172.21.15.78:0/2571978855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c78a0 0x7fa124159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa129b30640 1 -- 172.21.15.78:0/2571978855 >> v1:172.21.15.150:6789/0 conn(0x7fa1240c4d80 legacy=0x7fa1240c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa129b30640 1 -- 172.21.15.78:0/2571978855 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12406b930 con 0x7fa124155010 2024-02-19T14:54:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa129b30640 1 --2- 172.21.15.78:0/2571978855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa124155010 0x7fa1241523d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009e50 tx=0x7fa11c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab54e4c2f79ca64d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.611+0000 7fa11bfff640 1 -- 172.21.15.78:0/2571978855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c02c9e0 con 0x7fa124155010 2024-02-19T14:54:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa11bfff640 1 -- 172.21.15.78:0/2571978855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c02cb40 con 0x7fa124155010 2024-02-19T14:54:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa11bfff640 1 -- 172.21.15.78:0/2571978855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0343f0 con 0x7fa124155010 2024-02-19T14:54:52.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 -- 172.21.15.78:0/2571978855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa124155010 msgr2=0x7fa1241523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 --2- 172.21.15.78:0/2571978855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa124155010 0x7fa1241523d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009e50 tx=0x7fa11c02be40 comp rx=0 tx=0).stop 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 -- 172.21.15.78:0/2571978855 shutdown_connections 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 --2- 172.21.15.78:0/2571978855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa124155010 0x7fa1241523d0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 --2- 172.21.15.78:0/2571978855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c78a0 0x7fa124159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 -- 172.21.15.78:0/2571978855 >> 172.21.15.78:0/2571978855 conn(0x7fa124066e70 msgr2=0x7fa1240a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 -- 172.21.15.78:0/2571978855 shutdown_connections 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.612+0000 7fa12b593640 1 -- 172.21.15.78:0/2571978855 wait complete. 2024-02-19T14:54:52.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.614+0000 7fa12b593640 1 Processor -- start 2024-02-19T14:54:52.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.614+0000 7fa12b593640 1 -- start start 2024-02-19T14:54:52.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c4d80 0x7fa124157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 0x7fa1241516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa124155010 0x7fa124158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa124158360 con 0x7fa1240c78a0 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1241eb900 con 0x7fa124155010 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa124151bf0 con 0x7fa1240c4d80 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 0x7fa1241516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 0x7fa1241516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55478/0 (socket says 172.21.15.78:55478) 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 -- 172.21.15.78:0/1932471157 learned_addr learned my addr 172.21.15.78:0/1932471157 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa129b30640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa124155010 0x7fa124158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12932f640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c4d80 0x7fa124157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 -- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c4d80 msgr2=0x7fa124157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c4d80 0x7fa124157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 -- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa124155010 msgr2=0x7fa124158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa124155010 0x7fa124158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 -- 172.21.15.78:0/1932471157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c009b30 con 0x7fa1240c78a0 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa129b30640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa124155010 0x7fa124158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12932f640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c4d80 0x7fa124157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa128b2e640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 0x7fa1241516b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b3f0 tx=0x7fa11400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114011830 con 0x7fa1240c78a0 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa114011990 con 0x7fa1240c78a0 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.615+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1241eafd0 con 0x7fa1240c78a0 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.616+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114011d40 con 0x7fa1240c78a0 2024-02-19T14:54:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.616+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1241eb590 con 0x7fa1240c78a0 2024-02-19T14:54:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.617+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1240c1fa0 con 0x7fa1240c78a0 2024-02-19T14:54:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.617+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa11401a400 con 0x7fa1240c78a0 2024-02-19T14:54:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.618+0000 7fa119ffb640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa10806b810 0x7fa10806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.618+0000 7fa12932f640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa10806b810 0x7fa10806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.618+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fa11408a5f0 con 0x7fa1240c78a0 2024-02-19T14:54:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.618+0000 7fa12932f640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa10806b810 0x7fa10806dcd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa12000b820 tx=0x7fa120008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.626+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa114054af0 con 0x7fa1240c78a0 2024-02-19T14:54:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.851+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa124074990 con 0x7fa1240c78a0 2024-02-19T14:54:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.852+0000 7fa119ffb640 1 -- 172.21.15.78:0/1932471157 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6182 (secure 0 0 0) 0x7fa1140581b0 con 0x7fa1240c78a0 2024-02-19T14:54:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:52.855 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5354,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:26.070082+0000","modified":"2024-02-19T14:54:45.326993+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6657":{"gid":6657,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/1047936932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1047936932},{"type":"v1","addr":"172.21.15.78:6837","nonce":1047936932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:39.178036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6360":{"gid":6360,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6835/4201159351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":4201159351},{"type":"v1","addr":"172.21.15.78:6835","nonce":4201159351}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 61 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa10806b810 msgr2=0x7fa10806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa10806b810 0x7fa10806dcd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa12000b820 tx=0x7fa120008040 comp rx=0 tx=0).stop 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 msgr2=0x7fa1241516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 0x7fa1241516b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b3f0 tx=0x7fa11400b8b0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 shutdown_connections 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa10806b810 0x7fa10806dcd0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1240c78a0 0x7fa1241516b0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa124155010 0x7fa124158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 --2- 172.21.15.78:0/1932471157 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1240c4d80 0x7fa124157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 >> 172.21.15.78:0/1932471157 conn(0x7fa124066e70 msgr2=0x7fa124155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 shutdown_connections 2024-02-19T14:54:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.854+0000 7fa12b593640 1 -- 172.21.15.78:0/1932471157 wait complete. 2024-02-19T14:54:52.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:52.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.967+0000 7efe88ac2640 1 Processor -- start 2024-02-19T14:54:52.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.967+0000 7efe88ac2640 1 -- start start 2024-02-19T14:54:52.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe88ac2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe840c6cd0 0x7efe84155780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe88ac2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe840c6460 con 0x7efe841525c0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe88ac2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe84158720 con 0x7efe840c6870 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe88ac2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe840c1f50 con 0x7efe840c6cd0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe8259c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efe841525c0 0x7efe8406d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33692/0 (socket says 172.21.15.78:33692) 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe8259c640 1 -- 172.21.15.78:0/3589376427 learned_addr learned my addr 172.21.15.78:0/3589376427 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe82d9d640 1 --2- 172.21.15.78:0/3589376427 >> v2:172.21.15.150:3300/0 conn(0x7efe840c6cd0 0x7efe84155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19079118 0 0) 0x7efe840c6460 con 0x7efe841525c0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe70003520 con 0x7efe841525c0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.968+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163099936 0 0) 0x7efe70003520 con 0x7efe841525c0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 >> v2:172.21.15.150:3300/0 conn(0x7efe840c6cd0 msgr2=0x7efe84155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 --2- 172.21.15.78:0/3589376427 >> v2:172.21.15.150:3300/0 conn(0x7efe840c6cd0 0x7efe84155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 >> v1:172.21.15.150:6789/0 conn(0x7efe840c6870 legacy=0x7efe84155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84158a00 con 0x7efe841525c0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7efe74001670 con 0x7efe841525c0 2024-02-19T14:54:52.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 >> v1:172.21.15.78:6789/0 conn(0x7efe841525c0 legacy=0x7efe8406d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe70003970 0x7efe70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe700047c0 0x7efe70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe70005180 0x7efe7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840c1f50 con 0x7efe700047c0 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7000c120 con 0x7efe70003970 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7000c2e0 con 0x7efe70005180 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe82d9d640 1 --2- 172.21.15.78:0/3589376427 >> v2:172.21.15.150:3300/0 conn(0x7efe840c6cd0 0x7efe84155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe700047c0 0x7efe70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe8259c640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe70003970 0x7efe70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe82d9d640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe70005180 0x7efe7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe79d9b640 1 -- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe70005180 msgr2=0x7efe7000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe70005180 0x7efe7000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe79d9b640 1 -- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe70003970 msgr2=0x7efe70003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:52.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe70003970 0x7efe70003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe79d9b640 1 -- 172.21.15.78:0/3589376427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70003520 con 0x7efe700047c0 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.969+0000 7efe82d9d640 1 -- 172.21.15.78:0/3589376427 reap_dead start 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.970+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe700047c0 0x7efe70004c40 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efe6c001750 tx=0x7efe700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf3b5a3b465dd2b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.970+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0052d0 con 0x7efe700047c0 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.970+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c02e050 con 0x7efe700047c0 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.970+0000 7efe81d9b640 1 -- 172.21.15.78:0/3589376427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c003830 con 0x7efe700047c0 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 -- 172.21.15.78:0/3589376427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe700047c0 msgr2=0x7efe70004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe700047c0 0x7efe70004c40 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efe6c001750 tx=0x7efe700057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 -- 172.21.15.78:0/3589376427 shutdown_connections 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3589376427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe700047c0 0x7efe70004c40 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 -- 172.21.15.78:0/3589376427 >> 172.21.15.78:0/3589376427 conn(0x7efe840cab20 msgr2=0x7efe840caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 -- 172.21.15.78:0/3589376427 shutdown_connections 2024-02-19T14:54:52.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.971+0000 7efe88ac2640 1 -- 172.21.15.78:0/3589376427 wait complete. 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.973+0000 7efe88ac2640 1 Processor -- start 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.973+0000 7efe88ac2640 1 -- start start 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe88ac2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe700047c0 0x7efe841400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe88ac2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 0x7efe840c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe88ac2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe841e7a90 0x7efe841e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe88ac2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe841593a0 con 0x7efe84140650 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe88ac2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe84155e60 con 0x7efe700047c0 2024-02-19T14:54:52.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe88ac2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe841e8480 con 0x7efe841e7a90 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 0x7efe840c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 0x7efe840c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55498/0 (socket says 172.21.15.78:55498) 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 -- 172.21.15.78:0/3297092411 learned_addr learned my addr 172.21.15.78:0/3297092411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe82d9d640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe841e7a90 0x7efe841e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe8259c640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe700047c0 0x7efe841400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 -- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe841e7a90 msgr2=0x7efe841e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe841e7a90 0x7efe841e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 -- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe700047c0 msgr2=0x7efe841400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe700047c0 0x7efe841400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.974+0000 7efe79d9b640 1 -- 172.21.15.78:0/3297092411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84140ae0 con 0x7efe84140650 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.975+0000 7efe79d9b640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 0x7efe840c9a00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efe6c000c00 tx=0x7efe6c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.975+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c007a30 con 0x7efe84140650 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.975+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c007640 con 0x7efe84140650 2024-02-19T14:54:52.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.975+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe841eebf0 con 0x7efe84140650 2024-02-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.975+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c006c00 con 0x7efe84140650 2024-02-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.975+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe841ee110 con 0x7efe84140650 2024-02-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.976+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe84074990 con 0x7efe84140650 2024-02-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.977+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efe6c0035f0 con 0x7efe84140650 2024-02-19T14:54:52.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.977+0000 7efe7b7fe640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe6406b810 0x7efe6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:52.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.977+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7efe6c0ac180 con 0x7efe84140650 2024-02-19T14:54:52.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.977+0000 7efe8259c640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe6406b810 0x7efe6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:52.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.978+0000 7efe8259c640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe6406b810 0x7efe6406dcd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efe74001c10 tx=0x7efe74001b20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:52.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:52.985+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efe6c0766b0 con 0x7efe84140650 2024-02-19T14:54:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.166+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe840c51c0 con 0x7efe84140650 2024-02-19T14:54:53.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.168+0000 7efe7b7fe640 1 -- 172.21.15.78:0/3297092411 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+20980 (secure 0 0 0) 0x7efe6c079d70 con 0x7efe84140650 2024-02-19T14:54:53.171 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:53.171 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":82,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:42.308181+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:54:24.290728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-19T14:54:25.042750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe6406b810 msgr2=0x7efe6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe6406b810 0x7efe6406dcd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efe74001c10 tx=0x7efe74001b20 comp rx=0 tx=0).stop 2024-02-19T14:54:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 msgr2=0x7efe840c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 0x7efe840c9a00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efe6c000c00 tx=0x7efe6c001120 comp rx=0 tx=0).stop 2024-02-19T14:54:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 shutdown_connections 2024-02-19T14:54:53.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe6406b810 0x7efe6406dcd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe84140650 0x7efe840c9a00 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe841e7a90 0x7efe841e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 --2- 172.21.15.78:0/3297092411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe700047c0 0x7efe841400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 >> 172.21.15.78:0/3297092411 conn(0x7efe840cab20 msgr2=0x7efe840bf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 shutdown_connections 2024-02-19T14:54:53.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.170+0000 7efe88ac2640 1 -- 172.21.15.78:0/3297092411 wait complete. 2024-02-19T14:54:53.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:54:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.287+0000 7f12c4c07640 1 Processor -- start 2024-02-19T14:54:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.288+0000 7f12c4c07640 1 -- start start 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12c4c07640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12c014d110 0x7f12c00ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12c4c07640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c006b620 con 0x7f12c006bae0 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12c4c07640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c0157230 con 0x7f12c0073220 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12c4c07640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c006ea00 con 0x7f12c014d110 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12be59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12c014d110 0x7f12c00ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12be59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12c014d110 0x7f12c00ac950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55510/0 (socket says 172.21.15.78:55510) 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12be59c640 1 -- 172.21.15.78:0/1545770188 learned_addr learned my addr 172.21.15.78:0/1545770188 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12bed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f12c006bae0 0x7f12c006f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33694/0 (socket says 172.21.15.78:33694) 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12be59c640 1 -- 172.21.15.78:0/1545770188 >> v1:172.21.15.150:6789/0 conn(0x7f12c0073220 legacy=0x7f12c006ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12be59c640 1 -- 172.21.15.78:0/1545770188 >> v1:172.21.15.78:6789/0 conn(0x7f12c006bae0 legacy=0x7f12c006f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.289+0000 7f12bd59a640 1 -- 172.21.15.78:0/1545770188 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803287547 0 0) 0x7f12c006b620 con 0x7f12c006bae0 2024-02-19T14:54:53.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.290+0000 7f12be59c640 1 -- 172.21.15.78:0/1545770188 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0072180 con 0x7f12c014d110 2024-02-19T14:54:53.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.290+0000 7f12be59c640 1 --2- 172.21.15.78:0/1545770188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c00ac950 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009e50 tx=0x7f12b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b470410619bba402 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:53.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.290+0000 7f12bd59a640 1 -- 172.21.15.78:0/1545770188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b802ca80 con 0x7f12c014d110 2024-02-19T14:54:53.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.290+0000 7f12bd59a640 1 -- 172.21.15.78:0/1545770188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b802cbe0 con 0x7f12c014d110 2024-02-19T14:54:53.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.290+0000 7f12bd59a640 1 -- 172.21.15.78:0/1545770188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b80343f0 con 0x7f12c014d110 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 -- 172.21.15.78:0/1545770188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 msgr2=0x7f12c00ac950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 --2- 172.21.15.78:0/1545770188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c00ac950 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009e50 tx=0x7f12b802bee0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 -- 172.21.15.78:0/1545770188 shutdown_connections 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 --2- 172.21.15.78:0/1545770188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c00ac950 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 -- 172.21.15.78:0/1545770188 >> 172.21.15.78:0/1545770188 conn(0x7f12c00c7590 msgr2=0x7f12c00c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 -- 172.21.15.78:0/1545770188 shutdown_connections 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.291+0000 7f12c4c07640 1 -- 172.21.15.78:0/1545770188 wait complete. 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.292+0000 7f12c4c07640 1 Processor -- start 2024-02-19T14:54:53.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.292+0000 7f12c4c07640 1 -- start start 2024-02-19T14:54:53.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12c4c07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c006bae0 0x7f12c0147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12c4c07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c0073220 0x7f12c0153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12c4c07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12c4c07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0148980 con 0x7f12c014d110 2024-02-19T14:54:53.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12c4c07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00c4670 con 0x7f12c006bae0 2024-02-19T14:54:53.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12c4c07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0150120 con 0x7f12c0073220 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12bed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12bed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c014e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55526/0 (socket says 172.21.15.78:55526) 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12bed9d640 1 -- 172.21.15.78:0/354743849 learned_addr learned my addr 172.21.15.78:0/354743849 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12bdd9b640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c0073220 0x7f12c0153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12be59c640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c006bae0 0x7f12c0147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.293+0000 7f12bed9d640 1 -- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c0073220 msgr2=0x7f12c0153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12bed9d640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c0073220 0x7f12c0153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12bed9d640 1 -- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c006bae0 msgr2=0x7f12c0147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12bed9d640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c006bae0 0x7f12c0147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12bed9d640 1 -- 172.21.15.78:0/354743849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b8009b30 con 0x7f12c014d110 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12bed9d640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c014e540 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f12b000c440 tx=0x7f12b000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b00074e0 con 0x7f12c014d110 2024-02-19T14:54:53.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b0007640 con 0x7f12c014d110 2024-02-19T14:54:53.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0004a20 con 0x7f12c014d110 2024-02-19T14:54:53.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c01486a0 con 0x7f12c014d110 2024-02-19T14:54:53.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.294+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c01e94e0 con 0x7f12c014d110 2024-02-19T14:54:53.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.295+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c014fca0 con 0x7f12c014d110 2024-02-19T14:54:53.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.295+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f12b0004b80 con 0x7f12c014d110 2024-02-19T14:54:53.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.296+0000 7f12b6ffd640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a006b810 0x7f12a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.296+0000 7f12be59c640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a006b810 0x7f12a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.296+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f12b008a300 con 0x7f12c014d110 2024-02-19T14:54:53.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.296+0000 7f12be59c640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a006b810 0x7f12a006dcd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f12b8035000 tx=0x7f12b8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:53.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.300+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f12c014fca0 con 0x7f12c014d110 2024-02-19T14:54:53.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.606+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f12c014fca0 con 0x7f12c014d110 2024-02-19T14:54:53.609 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:54:53.607+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:54:53.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.608+0000 7f12b6ffd640 1 -- 172.21.15.78:0/354743849 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) v1 ==== 161+0+0 (secure 0 0 0) 0x7f12c014fca0 con 0x7f12c014d110 2024-02-19T14:54:53.611 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a006b810 msgr2=0x7f12a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a006b810 0x7f12a006dcd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f12b8035000 tx=0x7f12b8030000 comp rx=0 tx=0).stop 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 msgr2=0x7f12c014e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c014e540 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f12b000c440 tx=0x7f12b000c900 comp rx=0 tx=0).stop 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 shutdown_connections 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a006b810 0x7f12a006dcd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c014d110 0x7f12c014e540 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c0073220 0x7f12c0153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 --2- 172.21.15.78:0/354743849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c006bae0 0x7f12c0147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 >> 172.21.15.78:0/354743849 conn(0x7f12c00c7590 msgr2=0x7f12c00c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:53.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 shutdown_connections 2024-02-19T14:54:53.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.611+0000 7f12c4c07640 1 -- 172.21.15.78:0/354743849 wait complete. 2024-02-19T14:54:53.615 INFO:tasks.ceph.mds.c.smithi078.stderr: -161> 2024-02-19T14:54:49.314+0000 7f90b49af640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:53.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:54:53.641 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:54:53.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.728+0000 7ffbae083640 1 Processor -- start 2024-02-19T14:54:53.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.728+0000 7ffbae083640 1 -- start start 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffbae083640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffba80c19f0 0x7ffba8156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffbae083640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffba80c1dd0 0x7ffba80c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffbae083640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba8155a50 con 0x7ffba8071410 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffbae083640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba8152af0 con 0x7ffba80c1dd0 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffbae083640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba8155f80 con 0x7ffba80c19f0 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffba80c1dd0 0x7ffba80c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffba80c1dd0 0x7ffba80c21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55552/0 (socket says 172.21.15.78:55552) 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 -- 172.21.15.78:0/2148185363 learned_addr learned my addr 172.21.15.78:0/2148185363 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba77fe640 1 --2- 172.21.15.78:0/2148185363 >> v2:172.21.15.150:3300/0 conn(0x7ffba80c19f0 0x7ffba8156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 -- 172.21.15.78:0/2148185363 >> v2:172.21.15.150:3300/0 conn(0x7ffba80c19f0 msgr2=0x7ffba8156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:53.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 --2- 172.21.15.78:0/2148185363 >> v2:172.21.15.150:3300/0 conn(0x7ffba80c19f0 0x7ffba8156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 -- 172.21.15.78:0/2148185363 >> v1:172.21.15.150:6789/0 conn(0x7ffba8071410 legacy=0x7ffba80c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:53.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.729+0000 7ffba7fff640 1 -- 172.21.15.78:0/2148185363 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8070840 con 0x7ffba80c1dd0 2024-02-19T14:54:53.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.730+0000 7ffba77fe640 1 --2- 172.21.15.78:0/2148185363 >> v2:172.21.15.150:3300/0 conn(0x7ffba80c19f0 0x7ffba8156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:53.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffba7fff640 1 --2- 172.21.15.78:0/2148185363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c1dd0 0x7ffba80c21e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ffba80c2ca0 tx=0x7ffba002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1369fd7659573d52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:53.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffba6ffd640 1 -- 172.21.15.78:0/2148185363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba002c960 con 0x7ffba80c1dd0 2024-02-19T14:54:53.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffba6ffd640 1 -- 172.21.15.78:0/2148185363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba002cac0 con 0x7ffba80c1dd0 2024-02-19T14:54:53.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffba6ffd640 1 -- 172.21.15.78:0/2148185363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba002ce50 con 0x7ffba80c1dd0 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 -- 172.21.15.78:0/2148185363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c1dd0 msgr2=0x7ffba80c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 --2- 172.21.15.78:0/2148185363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c1dd0 0x7ffba80c21e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ffba80c2ca0 tx=0x7ffba002bdc0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 -- 172.21.15.78:0/2148185363 shutdown_connections 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 --2- 172.21.15.78:0/2148185363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c1dd0 0x7ffba80c21e0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 --2- 172.21.15.78:0/2148185363 >> v2:172.21.15.150:3300/0 conn(0x7ffba80c19f0 0x7ffba8156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 -- 172.21.15.78:0/2148185363 >> 172.21.15.78:0/2148185363 conn(0x7ffba80aa370 msgr2=0x7ffba80aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 -- 172.21.15.78:0/2148185363 shutdown_connections 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.731+0000 7ffbae083640 1 -- 172.21.15.78:0/2148185363 wait complete. 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.732+0000 7ffbae083640 1 Processor -- start 2024-02-19T14:54:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 -- start start 2024-02-19T14:54:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba8071410 0x7ffba81eafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 0x7ffba81eb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c1dd0 0x7ffba80ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba81eab10 con 0x7ffba80c19f0 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80c5580 con 0x7ffba8071410 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffbae083640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80c5860 con 0x7ffba80c1dd0 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffb9effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 0x7ffba81eb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffb9effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 0x7ffba81eb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55566/0 (socket says 172.21.15.78:55566) 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.733+0000 7ffb9effd640 1 -- 172.21.15.78:0/3888314388 learned_addr learned my addr 172.21.15.78:0/3888314388 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba77fe640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba8071410 0x7ffba81eafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba7fff640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c1dd0 0x7ffba80ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffb9effd640 1 -- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c1dd0 msgr2=0x7ffba80ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffb9effd640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c1dd0 0x7ffba80ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffb9effd640 1 -- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba8071410 msgr2=0x7ffba81eafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffb9effd640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba8071410 0x7ffba81eafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffb9effd640 1 -- 172.21.15.78:0/3888314388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0009b30 con 0x7ffba80c19f0 2024-02-19T14:54:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba7fff640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c1dd0 0x7ffba80ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba77fe640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba8071410 0x7ffba81eafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffb9effd640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 0x7ffba81eb4e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000b3f0 tx=0x7ffb9000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90011830 con 0x7ffba80c19f0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90011990 con 0x7ffba80c19f0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90011d40 con 0x7ffba80c19f0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba80c5b40 con 0x7ffba80c19f0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.734+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba814ddf0 con 0x7ffba80c19f0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.735+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba80c8320 con 0x7ffba80c19f0 2024-02-19T14:54:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.736+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffb9001a400 con 0x7ffba80c19f0 2024-02-19T14:54:53.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.736+0000 7ffba4ff9640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb8806b810 0x7ffb8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:53.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.736+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7245+0+0 (secure 0 0 0) 0x7ffb9008a580 con 0x7ffba80c19f0 2024-02-19T14:54:53.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.736+0000 7ffba77fe640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb8806b810 0x7ffb8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:53.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.737+0000 7ffba77fe640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb8806b810 0x7ffb8806dcd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffba80a8120 tx=0x7ffb98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:53.741+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffb90054a30 con 0x7ffba80c19f0 2024-02-19T14:54:54.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.066+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffba8074990 con 0x7ffba80c19f0 2024-02-19T14:54:54.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.612+0000 7ffba4ff9640 1 -- 172.21.15.78:0/3888314388 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffb900580f0 con 0x7ffba80c19f0 2024-02-19T14:54:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb8806b810 msgr2=0x7ffb8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb8806b810 0x7ffb8806dcd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffba80a8120 tx=0x7ffb98008040 comp rx=0 tx=0).stop 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 msgr2=0x7ffba81eb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 0x7ffba81eb4e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000b3f0 tx=0x7ffb9000b8b0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 shutdown_connections 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb8806b810 0x7ffb8806dcd0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c19f0 0x7ffba81eb4e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c1dd0 0x7ffba80ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 --2- 172.21.15.78:0/3888314388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba8071410 0x7ffba81eafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 >> 172.21.15.78:0/3888314388 conn(0x7ffba80aa370 msgr2=0x7ffba80aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 shutdown_connections 2024-02-19T14:54:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.615+0000 7ffbae083640 1 -- 172.21.15.78:0/3888314388 wait complete. 2024-02-19T14:54:54.623 INFO:tasks.ceph.mds.a.smithi078.stderr: -410> 2024-02-19T14:54:41.990+0000 7fb177223640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:54:54.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:54:54.648 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:54:54.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.749+0000 7fac85d6c640 1 Processor -- start 2024-02-19T14:54:54.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.749+0000 7fac85d6c640 1 -- start start 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac85d6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac800c37e0 0x7fac800c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac85d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800a9e00 0x7fac80158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac85d6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac80159320 con 0x7fac800c7fc0 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac85d6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac800c0f00 con 0x7fac800c37e0 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac85d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac800aa300 con 0x7fac800a9e00 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800a9e00 0x7fac80158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac800c37e0 0x7fac800c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fac800c7fc0 0x7fac800c4050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57008/0 (socket says 172.21.15.78:57008) 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7ffff640 1 -- 172.21.15.78:0/1767584099 learned_addr learned my addr 172.21.15.78:0/1767584099 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7f7fe640 1 -- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800a9e00 msgr2=0x7fac80158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7f7fe640 1 --2- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800a9e00 0x7fac80158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7f7fe640 1 -- 172.21.15.78:0/1767584099 >> v1:172.21.15.150:6789/0 conn(0x7fac800c7fc0 legacy=0x7fac800c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7f7fe640 1 -- 172.21.15.78:0/1767584099 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac80155ea0 con 0x7fac800c37e0 2024-02-19T14:54:54.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.750+0000 7fac7effd640 1 --2- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800a9e00 0x7fac80158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.751+0000 7fac7f7fe640 1 --2- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac800c7a80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fac700097a0 tx=0x7fac7002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=724ac1d74acf0c46 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.751+0000 7fac7e7fc640 1 -- 172.21.15.78:0/1767584099 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7002f050 con 0x7fac800c37e0 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.751+0000 7fac7e7fc640 1 -- 172.21.15.78:0/1767584099 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac70031700 con 0x7fac800c37e0 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.751+0000 7fac7e7fc640 1 -- 172.21.15.78:0/1767584099 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70031a90 con 0x7fac800c37e0 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 -- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 msgr2=0x7fac800c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 --2- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac800c7a80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fac700097a0 tx=0x7fac7002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 -- 172.21.15.78:0/1767584099 shutdown_connections 2024-02-19T14:54:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 --2- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800a9e00 0x7fac80158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 --2- 172.21.15.78:0/1767584099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac800c7a80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 -- 172.21.15.78:0/1767584099 >> 172.21.15.78:0/1767584099 conn(0x7fac8006f5e0 msgr2=0x7fac8006f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 -- 172.21.15.78:0/1767584099 shutdown_connections 2024-02-19T14:54:54.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.752+0000 7fac85d6c640 1 -- 172.21.15.78:0/1767584099 wait complete. 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.753+0000 7fac85d6c640 1 Processor -- start 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.753+0000 7fac85d6c640 1 -- start start 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac85d6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 0x7fac80157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac85d6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac80151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac85d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800c7fc0 0x7fac80152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac85d6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac800c7010 con 0x7fac800a9e00 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac85d6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac801eb640 con 0x7fac800c37e0 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac85d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac800c6a50 con 0x7fac800c7fc0 2024-02-19T14:54:54.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 0x7fac80157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 0x7fac80157650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55600/0 (socket says 172.21.15.78:55600) 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 -- 172.21.15.78:0/2786644959 learned_addr learned my addr 172.21.15.78:0/2786644959 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7ffff640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800c7fc0 0x7fac80152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7effd640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac80151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 -- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800c7fc0 msgr2=0x7fac80152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800c7fc0 0x7fac80152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 -- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 msgr2=0x7fac80151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac80151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7f7fe640 1 -- 172.21.15.78:0/2786644959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74009400 con 0x7fac800a9e00 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.754+0000 7fac7effd640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac80151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.755+0000 7fac7f7fe640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 0x7fac80157650 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fac70036040 tx=0x7fac700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.755+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70033930 con 0x7fac800a9e00 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.755+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac70033ab0 con 0x7fac800a9e00 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.755+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac70009400 con 0x7fac800a9e00 2024-02-19T14:54:54.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.755+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70033e60 con 0x7fac800a9e00 2024-02-19T14:54:54.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.755+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac800c6760 con 0x7fac800a9e00 2024-02-19T14:54:54.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.756+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac80074990 con 0x7fac800a9e00 2024-02-19T14:54:54.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.756+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fac7003a040 con 0x7fac800a9e00 2024-02-19T14:54:54.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.757+0000 7fac67fff640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fac6c06b810 0x7fac6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:54.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.757+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7331+0+0 (secure 0 0 0) 0x7fac700ab9c0 con 0x7fac800a9e00 2024-02-19T14:54:54.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.757+0000 7fac7effd640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fac6c06b810 0x7fac6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:54.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.757+0000 7fac7effd640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fac6c06b810 0x7fac6c06dcd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fac800c65f0 tx=0x7fac74008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:54.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.761+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fac70075e10 con 0x7fac800a9e00 2024-02-19T14:54:54.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:54.931+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac80072790 con 0x7fac800a9e00 2024-02-19T14:54:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.618+0000 7fac67fff640 1 -- 172.21.15.78:0/2786644959 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 181+0+0 (secure 0 0 0) 0x7fac700400d0 con 0x7fac800a9e00 2024-02-19T14:54:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:54:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.621+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fac6c06b810 msgr2=0x7fac6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.621+0000 7fac85d6c640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fac6c06b810 0x7fac6c06dcd0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fac800c65f0 tx=0x7fac74008040 comp rx=0 tx=0).stop 2024-02-19T14:54:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.621+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 msgr2=0x7fac80157650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.621+0000 7fac85d6c640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 0x7fac80157650 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fac70036040 tx=0x7fac700046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 shutdown_connections 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fac6c06b810 0x7fac6c06dcd0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac800c7fc0 0x7fac80152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac800c37e0 0x7fac80151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 --2- 172.21.15.78:0/2786644959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800a9e00 0x7fac80157650 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 >> 172.21.15.78:0/2786644959 conn(0x7fac8006f5e0 msgr2=0x7fac80072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 shutdown_connections 2024-02-19T14:54:55.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.622+0000 7fac85d6c640 1 -- 172.21.15.78:0/2786644959 wait complete. 2024-02-19T14:54:55.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:54:55.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.743+0000 7f18446a4640 1 Processor -- start 2024-02-19T14:54:55.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.743+0000 7f18446a4640 1 -- start start 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f18446a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f183c0c5590 0x7f183c0c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f18446a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f183c0c2940 0x7f183c15c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f18446a4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183c06e210 con 0x7f183c06e6a0 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f18446a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c0abb60 con 0x7f183c0c5590 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f18446a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c06f0a0 con 0x7f183c0c2940 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f183c0c5590 0x7f183c0c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f183c0c5590 0x7f183c0c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55616/0 (socket says 172.21.15.78:55616) 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 -- 172.21.15.78:0/3691012759 learned_addr learned my addr 172.21.15.78:0/3691012759 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 -- 172.21.15.78:0/3691012759 >> v2:172.21.15.150:3300/0 conn(0x7f183c0c2940 msgr2=0x7f183c15c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 --2- 172.21.15.78:0/3691012759 >> v2:172.21.15.150:3300/0 conn(0x7f183c0c2940 0x7f183c15c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 -- 172.21.15.78:0/3691012759 >> v1:172.21.15.150:6789/0 conn(0x7f183c06e6a0 legacy=0x7f183c151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:55.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.744+0000 7f1842440640 1 -- 172.21.15.78:0/3691012759 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c0c26f0 con 0x7f183c0c5590 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.745+0000 7f1842440640 1 --2- 172.21.15.78:0/3691012759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c5590 0x7f183c0c5970 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f183c0c6030 tx=0x7f183802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8dfa3dee1844ce1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.745+0000 7f184143e640 1 -- 172.21.15.78:0/3691012759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183802ee00 con 0x7f183c0c5590 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.745+0000 7f184143e640 1 -- 172.21.15.78:0/3691012759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838036440 con 0x7f183c0c5590 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.745+0000 7f184143e640 1 -- 172.21.15.78:0/3691012759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18380367d0 con 0x7f183c0c5590 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 -- 172.21.15.78:0/3691012759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c5590 msgr2=0x7f183c0c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 --2- 172.21.15.78:0/3691012759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c5590 0x7f183c0c5970 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f183c0c6030 tx=0x7f183802e050 comp rx=0 tx=0).stop 2024-02-19T14:54:55.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 -- 172.21.15.78:0/3691012759 shutdown_connections 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 --2- 172.21.15.78:0/3691012759 >> v2:172.21.15.150:3300/0 conn(0x7f183c0c2940 0x7f183c15c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 --2- 172.21.15.78:0/3691012759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c5590 0x7f183c0c5970 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 -- 172.21.15.78:0/3691012759 >> 172.21.15.78:0/3691012759 conn(0x7f183c0c6fe0 msgr2=0x7f183c0a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 -- 172.21.15.78:0/3691012759 shutdown_connections 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.746+0000 7f18446a4640 1 -- 172.21.15.78:0/3691012759 wait complete. 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 Processor -- start 2024-02-19T14:54:55.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 -- start start 2024-02-19T14:54:55.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c06e6a0 0x7f183c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 0x7f183c153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f183c0c5590 0x7f183c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c1532b0 con 0x7f183c0c2940 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c150730 con 0x7f183c06e6a0 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.747+0000 7f18446a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183c150a10 con 0x7f183c0c5590 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 0x7f183c153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 0x7f183c153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55632/0 (socket says 172.21.15.78:55632) 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 -- 172.21.15.78:0/2078232119 learned_addr learned my addr 172.21.15.78:0/2078232119 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1842440640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c06e6a0 0x7f183c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1842c41640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f183c0c5590 0x7f183c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 -- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f183c0c5590 msgr2=0x7f183c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f183c0c5590 0x7f183c1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 -- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c06e6a0 msgr2=0x7f183c155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:55.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c06e6a0 0x7f183c155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 -- 172.21.15.78:0/2078232119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183800b770 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1842c41640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f183c0c5590 0x7f183c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1841c3f640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 0x7f183c153b60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f182c00c400 tx=0x7f182c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0035f0 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c003750 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c150cf0 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c003b40 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.748+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c1512b0 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.749+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c0678b0 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.750+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f182c003ca0 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.750+0000 7f1832ffd640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f182006b810 0x7f182006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.750+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6893+0+0 (secure 0 0 0) 0x7f182c089bf0 con 0x7f183c0c2940 2024-02-19T14:54:55.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.750+0000 7f1842440640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f182006b810 0x7f182006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:55.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.750+0000 7f1842440640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f182006b810 0x7f182006dcd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1838038000 tx=0x7f1838032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:55.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.754+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f182c054220 con 0x7f183c0c2940 2024-02-19T14:54:55.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:54:55.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:54:55.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:55.927+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f183c074990 con 0x7f183c0c2940 2024-02-19T14:54:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.629+0000 7f1832ffd640 1 -- 172.21.15.78:0/2078232119 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f182c0578e0 con 0x7f183c0c2940 2024-02-19T14:54:56.632 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:54:56.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f182006b810 msgr2=0x7f182006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f182006b810 0x7f182006dcd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1838038000 tx=0x7f1838032000 comp rx=0 tx=0).stop 2024-02-19T14:54:56.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 msgr2=0x7f183c153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 0x7f183c153b60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f182c00c400 tx=0x7f182c00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 shutdown_connections 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f182006b810 0x7f182006dcd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f183c0c2940 0x7f183c153b60 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f183c0c5590 0x7f183c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 --2- 172.21.15.78:0/2078232119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c06e6a0 0x7f183c155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.633+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 >> 172.21.15.78:0/2078232119 conn(0x7f183c0c6fe0 msgr2=0x7f183c14c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.634+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 shutdown_connections 2024-02-19T14:54:56.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.634+0000 7f18446a4640 1 -- 172.21.15.78:0/2078232119 wait complete. 2024-02-19T14:54:56.650 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:56.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.747+0000 7f8d05f83640 1 Processor -- start 2024-02-19T14:54:56.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8d05f83640 1 -- start start 2024-02-19T14:54:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8d05f83640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8d05f83640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d000705b0 0x7f8d000709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8d05f83640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d000ae670 con 0x7f8d000add70 2024-02-19T14:54:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8d05f83640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d000716b0 con 0x7f8d000c4980 2024-02-19T14:54:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8d05f83640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d00153780 con 0x7f8d000705b0 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8cfeffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8cfeffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55640/0 (socket says 172.21.15.78:55640) 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.748+0000 7f8cfeffd640 1 -- 172.21.15.78:0/2883728181 learned_addr learned my addr 172.21.15.78:0/2883728181 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfffff640 1 --2- 172.21.15.78:0/2883728181 >> v2:172.21.15.150:3300/0 conn(0x7f8d000705b0 0x7f8d000709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfeffd640 1 -- 172.21.15.78:0/2883728181 >> v2:172.21.15.150:3300/0 conn(0x7f8d000705b0 msgr2=0x7f8d000709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfeffd640 1 --2- 172.21.15.78:0/2883728181 >> v2:172.21.15.150:3300/0 conn(0x7f8d000705b0 0x7f8d000709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfeffd640 1 -- 172.21.15.78:0/2883728181 >> v1:172.21.15.150:6789/0 conn(0x7f8d000add70 legacy=0x7f8d0006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfeffd640 1 -- 172.21.15.78:0/2883728181 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00153a60 con 0x7f8d000c4980 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfffff640 1 --2- 172.21.15.78:0/2883728181 >> v2:172.21.15.150:3300/0 conn(0x7f8d000705b0 0x7f8d000709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.749+0000 7f8cfeffd640 1 --2- 172.21.15.78:0/2883728181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 0x7f8d000c2f60 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8d0006ff70 tx=0x7f8cf002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ec5a7936a061c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/2883728181 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf00317e0 con 0x7f8d000c4980 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/2883728181 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf0031940 con 0x7f8d000c4980 2024-02-19T14:54:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/2883728181 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0031cd0 con 0x7f8d000c4980 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 -- 172.21.15.78:0/2883728181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 msgr2=0x7f8d000c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2883728181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 0x7f8d000c2f60 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8d0006ff70 tx=0x7f8cf002b880 comp rx=0 tx=0).stop 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 -- 172.21.15.78:0/2883728181 shutdown_connections 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2883728181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000c4980 0x7f8d000c2f60 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2883728181 >> v2:172.21.15.150:3300/0 conn(0x7f8d000705b0 0x7f8d000709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 -- 172.21.15.78:0/2883728181 >> 172.21.15.78:0/2883728181 conn(0x7f8d000c85e0 msgr2=0x7f8d000c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 -- 172.21.15.78:0/2883728181 shutdown_connections 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.750+0000 7f8d05f83640 1 -- 172.21.15.78:0/2883728181 wait complete. 2024-02-19T14:54:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.751+0000 7f8d05f83640 1 Processor -- start 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 -- start start 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 0x7f8d001e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000add70 0x7f8d000bd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000c4980 0x7f8d0014b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d001e9130 con 0x7f8d000705b0 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d000accf0 con 0x7f8d000c4980 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8d05f83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d00155e90 con 0x7f8d000add70 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8cff7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 0x7f8d001e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8cff7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 0x7f8d001e8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55644/0 (socket says 172.21.15.78:55644) 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.752+0000 7f8cff7fe640 1 -- 172.21.15.78:0/2105067776 learned_addr learned my addr 172.21.15.78:0/2105067776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:56.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cfffff640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000c4980 0x7f8d0014b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cfeffd640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000add70 0x7f8d000bd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cff7fe640 1 -- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000add70 msgr2=0x7f8d000bd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cff7fe640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000add70 0x7f8d000bd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cff7fe640 1 -- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000c4980 msgr2=0x7f8d0014b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cff7fe640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000c4980 0x7f8d0014b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cff7fe640 1 -- 172.21.15.78:0/2105067776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf00094d0 con 0x7f8d000705b0 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cfffff640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000c4980 0x7f8d0014b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cfeffd640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000add70 0x7f8d000bd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8cff7fe640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 0x7f8d001e8bf0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400c460 tx=0x7f8cf400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4012830 con 0x7f8d000705b0 2024-02-19T14:54:56.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf40129d0 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d000ac450 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4012d80 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.753+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d000aca10 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.754+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d000676a0 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.754+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8cf401a3b0 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.755+0000 7f8ce7fff640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.755+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f8cf408a3c0 con 0x7f8d000705b0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.755+0000 7f8cfeffd640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.755+0000 7f8cfeffd640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0036000 tx=0x7f8cf003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.759+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8cf4054b70 con 0x7f8d000705b0 2024-02-19T14:54:56.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.929+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d00074990 con 0x7f8d000705b0 2024-02-19T14:54:56.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.931+0000 7f8ce7fff640 1 -- 172.21.15.78:0/2105067776 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17460 (secure 0 0 0) 0x7f8cf4058230 con 0x7f8d000705b0 2024-02-19T14:54:56.933 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:56.934 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":86,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:56.626216+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6835/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6834/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6836/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:56.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.932+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 msgr2=0x7f8cec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.932+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0036000 tx=0x7f8cf003d040 comp rx=0 tx=0).stop 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.932+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 msgr2=0x7f8d001e8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.932+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 0x7f8d001e8bf0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400c460 tx=0x7f8cf400c920 comp rx=0 tx=0).stop 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 shutdown_connections 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8cec06b810 0x7f8cec06dcd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d000c4980 0x7f8d0014b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d000add70 0x7f8d000bd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 --2- 172.21.15.78:0/2105067776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d000705b0 0x7f8d001e8bf0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 >> 172.21.15.78:0/2105067776 conn(0x7f8d000c85e0 msgr2=0x7f8d000c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 shutdown_connections 2024-02-19T14:54:56.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:56.933+0000 7f8d05f83640 1 -- 172.21.15.78:0/2105067776 wait complete. 2024-02-19T14:54:56.947 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-19T14:54:34.841768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-19T14:54:36.252782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:54:56.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:54:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 Processor -- start 2024-02-19T14:54:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 -- start start 2024-02-19T14:54:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa1040c8e00 0x7fa10406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa10406d910 0x7fa10406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10406cd20 con 0x7fa104152700 2024-02-19T14:54:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1041551f0 con 0x7fa1040c8e00 2024-02-19T14:54:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa10b0b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa104155490 con 0x7fa10406d910 2024-02-19T14:54:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa103fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa1040c8e00 0x7fa10406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa108e51640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa104152700 0x7fa1040c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33702/0 (socket says 172.21.15.78:33702) 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa103fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa1040c8e00 0x7fa10406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55658/0 (socket says 172.21.15.78:55658) 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.048+0000 7fa108e51640 1 -- 172.21.15.78:0/1276231098 learned_addr learned my addr 172.21.15.78:0/1276231098 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.049+0000 7fa109652640 1 --2- 172.21.15.78:0/1276231098 >> v2:172.21.15.150:3300/0 conn(0x7fa10406d910 0x7fa10406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.049+0000 7fa103fff640 1 -- 172.21.15.78:0/1276231098 >> v2:172.21.15.150:3300/0 conn(0x7fa10406d910 msgr2=0x7fa10406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.049+0000 7fa103fff640 1 --2- 172.21.15.78:0/1276231098 >> v2:172.21.15.150:3300/0 conn(0x7fa10406d910 0x7fa10406c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.049+0000 7fa103fff640 1 -- 172.21.15.78:0/1276231098 >> v1:172.21.15.78:6789/0 conn(0x7fa104152700 legacy=0x7fa1040c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.049+0000 7fa1037fe640 1 -- 172.21.15.78:0/1276231098 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596766476 0 0) 0x7fa10406cd20 con 0x7fa104152700 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.049+0000 7fa103fff640 1 -- 172.21.15.78:0/1276231098 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104155770 con 0x7fa1040c8e00 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa103fff640 1 --2- 172.21.15.78:0/1276231098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1040c8e00 0x7fa10406c2a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa1041596f0 tx=0x7fa0f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b226976abea649dc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa1037fe640 1 -- 172.21.15.78:0/1276231098 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0031830 con 0x7fa1040c8e00 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa1037fe640 1 -- 172.21.15.78:0/1276231098 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f0031990 con 0x7fa1040c8e00 2024-02-19T14:54:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa1037fe640 1 -- 172.21.15.78:0/1276231098 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0031d20 con 0x7fa1040c8e00 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 -- 172.21.15.78:0/1276231098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1040c8e00 msgr2=0x7fa10406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/1276231098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1040c8e00 0x7fa10406c2a0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa1041596f0 tx=0x7fa0f002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 -- 172.21.15.78:0/1276231098 shutdown_connections 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/1276231098 >> v2:172.21.15.150:3300/0 conn(0x7fa10406d910 0x7fa10406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/1276231098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa1040c8e00 0x7fa10406c2a0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 -- 172.21.15.78:0/1276231098 >> 172.21.15.78:0/1276231098 conn(0x7fa1040cac90 msgr2=0x7fa1040cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 -- 172.21.15.78:0/1276231098 shutdown_connections 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.050+0000 7fa10b0b5640 1 -- 172.21.15.78:0/1276231098 wait complete. 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.051+0000 7fa10b0b5640 1 Processor -- start 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.051+0000 7fa10b0b5640 1 -- start start 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa10b0b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 0x7fa10406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa10b0b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1040c8e00 0x7fa1040c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa10b0b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa104152700 0x7fa1041edc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa10b0b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040c49b0 con 0x7fa10406d910 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa10b0b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040c35f0 con 0x7fa1040c8e00 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa10b0b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040c2cc0 con 0x7fa104152700 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 0x7fa10406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 0x7fa10406ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55674/0 (socket says 172.21.15.78:55674) 2024-02-19T14:54:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 -- 172.21.15.78:0/904383788 learned_addr learned my addr 172.21.15.78:0/904383788 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 -- 172.21.15.78:0/904383788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa104152700 msgr2=0x7fa1041edc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa104152700 0x7fa1041edc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 -- 172.21.15.78:0/904383788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1040c8e00 msgr2=0x7fa1040c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1040c8e00 0x7fa1040c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.052+0000 7fa108e51640 1 -- 172.21.15.78:0/904383788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0009400 con 0x7fa10406d910 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.053+0000 7fa108e51640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 0x7fa10406ccd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800c430 tx=0x7fa0f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.053+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80074e0 con 0x7fa10406d910 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.053+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f8007660 con 0x7fa10406d910 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.053+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8004a20 con 0x7fa10406d910 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.053+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1040c2fa0 con 0x7fa10406d910 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.053+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10406d210 con 0x7fa10406d910 2024-02-19T14:54:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.054+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104074990 con 0x7fa10406d910 2024-02-19T14:54:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.054+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa0f8004b80 con 0x7fa10406d910 2024-02-19T14:54:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.054+0000 7fa1017fa640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa0e806b840 0x7fa0e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.055+0000 7fa103fff640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa0e806b840 0x7fa0e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.055+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7fa0f808a2e0 con 0x7fa10406d910 2024-02-19T14:54:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.055+0000 7fa103fff640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa0e806b840 0x7fa0e806dd00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0036000 tx=0x7fa0f0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.059+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa0f8054a00 con 0x7fa10406d910 2024-02-19T14:54:57.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.230+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1040c8770 con 0x7fa10406d910 2024-02-19T14:54:57.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.230+0000 7fa1017fa640 1 -- 172.21.15.78:0/904383788 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0f80580c0 con 0x7fa10406d910 2024-02-19T14:54:57.233 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa0e806b840 msgr2=0x7fa0e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa0e806b840 0x7fa0e806dd00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0036000 tx=0x7fa0f0030000 comp rx=0 tx=0).stop 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 msgr2=0x7fa10406ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 0x7fa10406ccd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800c430 tx=0x7fa0f800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 shutdown_connections 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa0e806b840 0x7fa0e806dd00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa104152700 0x7fa1041edc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1040c8e00 0x7fa1040c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 --2- 172.21.15.78:0/904383788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa10406d910 0x7fa10406ccd0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 >> 172.21.15.78:0/904383788 conn(0x7fa1040cac90 msgr2=0x7fa1041509a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 shutdown_connections 2024-02-19T14:54:57.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.232+0000 7fa10b0b5640 1 -- 172.21.15.78:0/904383788 wait complete. 2024-02-19T14:54:57.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-19T14:54:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.346+0000 7f0139502640 1 Processor -- start 2024-02-19T14:54:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.346+0000 7f0139502640 1 -- start start 2024-02-19T14:54:57.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0139502640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f013406d920 0x7f013415e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0139502640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0134150820 con 0x7f0134150440 2024-02-19T14:54:57.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0139502640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01341537d0 con 0x7f013406ed20 2024-02-19T14:54:57.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0139502640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01340c1b60 con 0x7f013406d920 2024-02-19T14:54:57.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0132ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0134150440 0x7f01340c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:33712/0 (socket says 172.21.15.78:33712) 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0132ffd640 1 -- 172.21.15.78:0/4149990899 learned_addr learned my addr 172.21.15.78:0/4149990899 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f01337fe640 1 --2- 172.21.15.78:0/4149990899 >> v2:172.21.15.150:3300/0 conn(0x7f013406d920 0x7f013415e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632575522 0 0) 0x7f0134150820 con 0x7f0134150440 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c003520 con 0x7f0134150440 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123431919 0 0) 0x7f011c003520 con 0x7f0134150440 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 >> v2:172.21.15.150:3300/0 conn(0x7f013406d920 msgr2=0x7f013415e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 --2- 172.21.15.78:0/4149990899 >> v2:172.21.15.150:3300/0 conn(0x7f013406d920 0x7f013415e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 >> v1:172.21.15.150:6789/0 conn(0x7f013406ed20 legacy=0x7f013406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0134153d10 con 0x7f0134150440 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.347+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0124001670 con 0x7f0134150440 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 >> v1:172.21.15.78:6789/0 conn(0x7f0134150440 legacy=0x7f01340c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f011c003970 0x7f011c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f011c0047c0 0x7f011c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c005180 0x7f011c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c1b60 con 0x7f011c003970 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c00c030 con 0x7f011c005180 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c00c1f0 con 0x7f011c0047c0 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0132ffd640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f011c003970 0x7f011c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f01337fe640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c005180 0x7f011c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f01327fc640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f011c0047c0 0x7f011c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0132ffd640 1 -- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f011c0047c0 msgr2=0x7f011c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0132ffd640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f011c0047c0 0x7f011c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0132ffd640 1 -- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c005180 msgr2=0x7f011c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0132ffd640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c005180 0x7f011c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f0132ffd640 1 -- 172.21.15.78:0/4149990899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c003520 con 0x7f011c003970 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.348+0000 7f01337fe640 1 -- 172.21.15.78:0/4149990899 reap_dead start 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0132ffd640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f011c003970 0x7f011c003db0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f012400c190 tx=0x7f011c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f80efd77638507 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124005ad0 con 0x7f011c003970 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124005c30 con 0x7f011c003970 2024-02-19T14:54:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0131ffb640 1 -- 172.21.15.78:0/4149990899 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012402cab0 con 0x7f011c003970 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 -- 172.21.15.78:0/4149990899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f011c003970 msgr2=0x7f011c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f011c003970 0x7f011c003db0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f012400c190 tx=0x7f011c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 -- 172.21.15.78:0/4149990899 shutdown_connections 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 --2- 172.21.15.78:0/4149990899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f011c003970 0x7f011c003db0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 -- 172.21.15.78:0/4149990899 >> 172.21.15.78:0/4149990899 conn(0x7f013406b270 msgr2=0x7f013406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 -- 172.21.15.78:0/4149990899 shutdown_connections 2024-02-19T14:54:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.349+0000 7f0139502640 1 -- 172.21.15.78:0/4149990899 wait complete. 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 Processor -- start 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 -- start start 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c003970 0x7f01340a80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 0x7f0134146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340ac290 0x7f0134147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340a6990 con 0x7f01340abe50 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340a77b0 con 0x7f011c003970 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f0139502640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01341479a0 con 0x7f01340ac290 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f01327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 0x7f0134146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f01327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 0x7f0134146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55690/0 (socket says 172.21.15.78:55690) 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.351+0000 7f01327fc640 1 -- 172.21.15.78:0/549037358 learned_addr learned my addr 172.21.15.78:0/549037358 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01337fe640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340ac290 0x7f0134147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f0132ffd640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c003970 0x7f01340a80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01327fc640 1 -- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340ac290 msgr2=0x7f0134147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01327fc640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340ac290 0x7f0134147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01327fc640 1 -- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c003970 msgr2=0x7f01340a80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01327fc640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c003970 0x7f01340a80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01327fc640 1 -- 172.21.15.78:0/549037358 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340a6ba0 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f01327fc640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 0x7f0134146f20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f012800e4b0 tx=0x7f012800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01280052e0 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0128005440 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01340a6e80 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128003830 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.352+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01340a7440 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.353+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01340c1540 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.353+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0128003990 con 0x7f01340abe50 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.354+0000 7f011b7fe640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f012006b840 0x7f012006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.354+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f012808a1d0 con 0x7f01340abe50 2024-02-19T14:54:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.354+0000 7f0132ffd640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f012006b840 0x7f012006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.354+0000 7f0132ffd640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f012006b840 0x7f012006dd00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0124034000 tx=0x7f0124009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.358+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f01280548f0 con 0x7f01340abe50 2024-02-19T14:54:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.528+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01340c4140 con 0x7f01340abe50 2024-02-19T14:54:57.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.529+0000 7f011b7fe640 1 -- 172.21.15.78:0/549037358 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v86) v1 ==== 105+0+11 (secure 0 0 0) 0x7f01340c4140 con 0x7f01340abe50 2024-02-19T14:54:57.531 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:54:57.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.530+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f012006b840 msgr2=0x7f012006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.530+0000 7f0139502640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f012006b840 0x7f012006dd00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0124034000 tx=0x7f0124009030 comp rx=0 tx=0).stop 2024-02-19T14:54:57.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.530+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 msgr2=0x7f0134146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.530+0000 7f0139502640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 0x7f0134146f20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f012800e4b0 tx=0x7f012800e970 comp rx=0 tx=0).stop 2024-02-19T14:54:57.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.530+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 shutdown_connections 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.530+0000 7f0139502640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f012006b840 0x7f012006dd00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.531+0000 7f0139502640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01340abe50 0x7f0134146f20 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.531+0000 7f0139502640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340ac290 0x7f0134147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.531+0000 7f0139502640 1 --2- 172.21.15.78:0/549037358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011c003970 0x7f01340a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.531+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 >> 172.21.15.78:0/549037358 conn(0x7f013406b270 msgr2=0x7f013406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.531+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 shutdown_connections 2024-02-19T14:54:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.531+0000 7f0139502640 1 -- 172.21.15.78:0/549037358 wait complete. 2024-02-19T14:54:57.544 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-19T14:54:57.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.644+0000 7f5831725640 1 Processor -- start 2024-02-19T14:54:57.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f5831725640 1 -- start start 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f5831725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f5831725640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 0x7f582c0c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f5831725640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582c14f730 con 0x7f582c06ff80 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f5831725640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c0c1940 con 0x7f582c06e830 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f5831725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c072f70 con 0x7f582c0c6d60 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f582a7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 0x7f582c0c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f582affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f582a7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 0x7f582c0c0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:41626/0 (socket says 172.21.15.78:41626) 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f582affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:44994/0 (socket says 172.21.15.78:44994) 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f582a7fc640 1 -- 172.21.15.78:0/2359461906 learned_addr learned my addr 172.21.15.78:0/2359461906 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.645+0000 7f582b7fe640 1 --1- 172.21.15.78:0/2359461906 >> v1:172.21.15.150:6789/0 conn(0x7f582c06ff80 0x7f582c0c7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:39062/0 (socket says 172.21.15.78:39062) 2024-02-19T14:54:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.646+0000 7f582affd640 1 -- 172.21.15.78:0/2359461906 >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 msgr2=0x7f582c0c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:57.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.646+0000 7f582affd640 1 --2- 172.21.15.78:0/2359461906 >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 0x7f582c0c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.646+0000 7f582affd640 1 -- 172.21.15.78:0/2359461906 >> v1:172.21.15.150:6789/0 conn(0x7f582c06ff80 legacy=0x7f582c0c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.646+0000 7f582affd640 1 -- 172.21.15.78:0/2359461906 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0c1b50 con 0x7f582c0c6d60 2024-02-19T14:54:57.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.646+0000 7f582a7fc640 1 --2- 172.21.15.78:0/2359461906 >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 0x7f582c0c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.647+0000 7f582affd640 1 --2- 172.21.15.78:0/2359461906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c154f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece566ce5def2853 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.647+0000 7f5829ffb640 1 -- 172.21.15.78:0/2359461906 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c02e070 con 0x7f582c0c6d60 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.647+0000 7f5829ffb640 1 -- 172.21.15.78:0/2359461906 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0315a0 con 0x7f582c0c6d60 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.647+0000 7f5829ffb640 1 -- 172.21.15.78:0/2359461906 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c031930 con 0x7f582c0c6d60 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 -- 172.21.15.78:0/2359461906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 msgr2=0x7f582c154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 --2- 172.21.15.78:0/2359461906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c154f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c02b840 comp rx=0 tx=0).stop 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 -- 172.21.15.78:0/2359461906 shutdown_connections 2024-02-19T14:54:57.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 --2- 172.21.15.78:0/2359461906 >> v2:172.21.15.150:3300/0 conn(0x7f582c06e830 0x7f582c0c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 --2- 172.21.15.78:0/2359461906 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c154f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 -- 172.21.15.78:0/2359461906 >> 172.21.15.78:0/2359461906 conn(0x7f582c06c170 msgr2=0x7f582c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 -- 172.21.15.78:0/2359461906 shutdown_connections 2024-02-19T14:54:57.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.648+0000 7f5831725640 1 -- 172.21.15.78:0/2359461906 wait complete. 2024-02-19T14:54:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.649+0000 7f5831725640 1 Processor -- start 2024-02-19T14:54:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.649+0000 7f5831725640 1 -- start start 2024-02-19T14:54:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f5831725640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 0x7f582c0a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f5831725640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f582c06ff80 0x7f582c149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f5831725640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c0c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f5831725640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c1eb270 con 0x7f582c06e830 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f5831725640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c0aa1f0 con 0x7f582c06ff80 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f5831725640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c0a9f10 con 0x7f582c0c6d60 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 0x7f582c0a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 0x7f582c0a9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58148/0 (socket says 172.21.15.78:58148) 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 -- 172.21.15.78:0/287728407 learned_addr learned my addr 172.21.15.78:0/287728407 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582b7fe640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c0c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582a7fc640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f582c06ff80 0x7f582c149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 -- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 msgr2=0x7f582c0c1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c0c1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 -- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f582c06ff80 msgr2=0x7f582c149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f582c06ff80 0x7f582c149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 -- 172.21.15.78:0/287728407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58200094d0 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.650+0000 7f582affd640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 0x7f582c0a9930 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f581c034040 tx=0x7f581c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.651+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c032640 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.651+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0327e0 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.651+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c009400 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.651+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c032b90 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.651+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c1eaf00 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.652+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c1eaa80 con 0x7f582c06e830 2024-02-19T14:54:57.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.652+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f581c032cf0 con 0x7f582c06e830 2024-02-19T14:54:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.652+0000 7f58137fe640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f581806b810 0x7f581806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.652+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f581c0abba0 con 0x7f582c06e830 2024-02-19T14:54:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.652+0000 7f582a7fc640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f581806b810 0x7f581806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.653+0000 7f582a7fc640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f581806b810 0x7f581806dcd0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f582c0c4ae0 tx=0x7f5820009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.657+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f582c1eaa80 con 0x7f582c06e830 2024-02-19T14:54:57.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.828+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f582c1eaa80 con 0x7f582c06e830 2024-02-19T14:54:57.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.828+0000 7f58137fe640 1 -- 172.21.15.78:0/287728407 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v86) v1 ==== 101+0+11 (secure 0 0 0) 0x7f582c1eaa80 con 0x7f582c06e830 2024-02-19T14:54:57.832 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:54:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f581806b810 msgr2=0x7f581806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f581806b810 0x7f581806dcd0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f582c0c4ae0 tx=0x7f5820009240 comp rx=0 tx=0).stop 2024-02-19T14:54:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 msgr2=0x7f582c0a9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 0x7f582c0a9930 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f581c034040 tx=0x7f581c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 shutdown_connections 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f581806b810 0x7f581806dcd0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f582c0c6d60 0x7f582c0c1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f582c06ff80 0x7f582c149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 --2- 172.21.15.78:0/287728407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f582c06e830 0x7f582c0a9930 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 >> 172.21.15.78:0/287728407 conn(0x7f582c06c170 msgr2=0x7f582c0c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 shutdown_connections 2024-02-19T14:54:57.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.830+0000 7f5831725640 1 -- 172.21.15.78:0/287728407 wait complete. 2024-02-19T14:54:57.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:57.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.941+0000 7f3916d48640 1 Processor -- start 2024-02-19T14:54:57.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.941+0000 7f3916d48640 1 -- start start 2024-02-19T14:54:57.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.941+0000 7f3916d48640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c8500 0x7f39100c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3916d48640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39100c34a0 con 0x7f39100c42e0 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3916d48640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f391006c8e0 con 0x7f39100c2f50 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3916d48640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391006c090 con 0x7f39100c8500 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f39100c42e0 0x7f3910158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36520/0 (socket says 172.21.15.78:36520) 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390ffff640 1 -- 172.21.15.78:0/1063715207 learned_addr learned my addr 172.21.15.78:0/1063715207 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3914ae4640 1 --2- 172.21.15.78:0/1063715207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c8500 0x7f39100c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443581451 0 0) 0x7f39100c34a0 con 0x7f39100c42e0 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3900003520 con 0x7f39100c42e0 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362194665 0 0) 0x7f391006c8e0 con 0x7f39100c2f50 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39100c34a0 con 0x7f39100c2f50 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3914ae4640 1 -- 172.21.15.78:0/1063715207 >> v1:172.21.15.150:6789/0 conn(0x7f39100c2f50 legacy=0x7f391015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3914ae4640 1 -- 172.21.15.78:0/1063715207 >> v1:172.21.15.78:6789/0 conn(0x7f39100c42e0 legacy=0x7f3910158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675293038 0 0) 0x7f3900003520 con 0x7f39100c42e0 2024-02-19T14:54:57.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.942+0000 7f3914ae4640 1 -- 172.21.15.78:0/1063715207 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391006cbc0 con 0x7f39100c8500 2024-02-19T14:54:57.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.943+0000 7f3914ae4640 1 --2- 172.21.15.78:0/1063715207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c8500 0x7f39100c88e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f39040097d0 tx=0x7f390402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bbc0ac6bddf50a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.943+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc001670 con 0x7f39100c8500 2024-02-19T14:54:57.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.943+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390402e050 con 0x7f39100c8500 2024-02-19T14:54:57.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.943+0000 7f390f7fe640 1 -- 172.21.15.78:0/1063715207 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3904030730 con 0x7f39100c8500 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 -- 172.21.15.78:0/1063715207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c8500 msgr2=0x7f39100c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 --2- 172.21.15.78:0/1063715207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c8500 0x7f39100c88e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f39040097d0 tx=0x7f390402b770 comp rx=0 tx=0).stop 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 -- 172.21.15.78:0/1063715207 shutdown_connections 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 --2- 172.21.15.78:0/1063715207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c8500 0x7f39100c88e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 -- 172.21.15.78:0/1063715207 >> 172.21.15.78:0/1063715207 conn(0x7f39100aa300 msgr2=0x7f39100aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 -- 172.21.15.78:0/1063715207 shutdown_connections 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.944+0000 7f3916d48640 1 -- 172.21.15.78:0/1063715207 wait complete. 2024-02-19T14:54:57.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.945+0000 7f3916d48640 1 Processor -- start 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.945+0000 7f3916d48640 1 -- start start 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c2f50 0x7f391014dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 0x7f391014ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f391014b760 0x7f391014b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3910070730 con 0x7f39100c8500 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39101eee80 con 0x7f391014b760 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39100709f0 con 0x7f39100c2f50 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 0x7f391014ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 0x7f391014ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58162/0 (socket says 172.21.15.78:58162) 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 -- 172.21.15.78:0/2524065374 learned_addr learned my addr 172.21.15.78:0/2524065374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3914ae4640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c2f50 0x7f391014dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f39152e5640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f391014b760 0x7f391014b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 -- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c2f50 msgr2=0x7f391014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c2f50 0x7f391014dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 -- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f391014b760 msgr2=0x7f391014b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f391014b760 0x7f391014b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 -- 172.21.15.78:0/2524065374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904009400 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390ffff640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 0x7f391014ac10 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00c750 tx=0x7f38fc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc005ad0 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc005c30 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391014e100 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc007070 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.946+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39100c5850 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.947+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3910070c00 con 0x7f39100c8500 2024-02-19T14:54:57.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.948+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f38fc0071d0 con 0x7f39100c8500 2024-02-19T14:54:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.948+0000 7f390d7fa640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f406b810 0x7f38f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.948+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f38fc089a40 con 0x7f39100c8500 2024-02-19T14:54:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.948+0000 7f3914ae4640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f406b810 0x7f38f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:57.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.948+0000 7f3914ae4640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f406b810 0x7f38f406dcd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3904034000 tx=0x7f3904008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:57.952+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f38fc08be60 con 0x7f39100c8500 2024-02-19T14:54:58.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.163+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3910074990 con 0x7f39100c8500 2024-02-19T14:54:58.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.164+0000 7f390d7fa640 1 -- 172.21.15.78:0/2524065374 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f38fc0542a0 con 0x7f39100c8500 2024-02-19T14:54:58.167 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:58.168 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6768,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3549382195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3549382195},{"type":"v1","addr":"172.21.15.78:6837","nonce":3549382195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2858964398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2858964398},{"type":"v1","addr":"172.21.15.78:6835","nonce":2858964398}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:56.633799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5354":{"gid":5354,"name":"d","rank":0,"incarnation":64,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:58.168 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 67 2024-02-19T14:54:58.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.165+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f406b810 msgr2=0x7f38f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.165+0000 7f3916d48640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f406b810 0x7f38f406dcd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3904034000 tx=0x7f3904008030 comp rx=0 tx=0).stop 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.165+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 msgr2=0x7f391014ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.165+0000 7f3916d48640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 0x7f391014ac10 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00c750 tx=0x7f38fc00cc10 comp rx=0 tx=0).stop 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 shutdown_connections 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f406b810 0x7f38f406dcd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39100c8500 0x7f391014ac10 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f391014b760 0x7f391014b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 --2- 172.21.15.78:0/2524065374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39100c2f50 0x7f391014dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 >> 172.21.15.78:0/2524065374 conn(0x7f39100aa300 msgr2=0x7f39100ad4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 shutdown_connections 2024-02-19T14:54:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.166+0000 7f3916d48640 1 -- 172.21.15.78:0/2524065374 wait complete. 2024-02-19T14:54:58.180 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.276+0000 7f10dc8f4640 1 Processor -- start 2024-02-19T14:54:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.276+0000 7f10dc8f4640 1 -- start start 2024-02-19T14:54:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10dc8f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f10d8071e50 0x7f10d80c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10dc8f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d80c1120 con 0x7f10d80719d0 2024-02-19T14:54:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10dc8f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d80c6080 con 0x7f10d814ece0 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10dc8f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d80722a0 con 0x7f10d8071e50 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10d6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f10d8071e50 0x7f10d80c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10d5d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f10d80719d0 0x7f10d80c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36526/0 (socket says 172.21.15.78:36526) 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.277+0000 7f10d5d9b640 1 -- 172.21.15.78:0/2543383373 learned_addr learned my addr 172.21.15.78:0/2543383373 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.278+0000 7f10d6d9d640 1 -- 172.21.15.78:0/2543383373 >> v1:172.21.15.150:6789/0 conn(0x7f10d814ece0 legacy=0x7f10d80c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.278+0000 7f10d6d9d640 1 -- 172.21.15.78:0/2543383373 >> v1:172.21.15.78:6789/0 conn(0x7f10d80719d0 legacy=0x7f10d80c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.278+0000 7f10d559a640 1 -- 172.21.15.78:0/2543383373 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721994270 0 0) 0x7f10d80c1120 con 0x7f10d80719d0 2024-02-19T14:54:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.278+0000 7f10d6d9d640 1 -- 172.21.15.78:0/2543383373 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d80c57e0 con 0x7f10d8071e50 2024-02-19T14:54:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.278+0000 7f10d6d9d640 1 --2- 172.21.15.78:0/2543383373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d80c1cc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009720 tx=0x7f10c802b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c1bf7810fd1c2c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10d559a640 1 -- 172.21.15.78:0/2543383373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c8030710 con 0x7f10d8071e50 2024-02-19T14:54:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10d559a640 1 -- 172.21.15.78:0/2543383373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c802e050 con 0x7f10d8071e50 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10d559a640 1 -- 172.21.15.78:0/2543383373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c8030a90 con 0x7f10d8071e50 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 -- 172.21.15.78:0/2543383373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 msgr2=0x7f10d80c1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/2543383373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d80c1cc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009720 tx=0x7f10c802b730 comp rx=0 tx=0).stop 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 -- 172.21.15.78:0/2543383373 shutdown_connections 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/2543383373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d80c1cc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 -- 172.21.15.78:0/2543383373 >> 172.21.15.78:0/2543383373 conn(0x7f10d806b270 msgr2=0x7f10d806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 -- 172.21.15.78:0/2543383373 shutdown_connections 2024-02-19T14:54:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.279+0000 7f10dc8f4640 1 -- 172.21.15.78:0/2543383373 wait complete. 2024-02-19T14:54:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.280+0000 7f10dc8f4640 1 Processor -- start 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 -- start start 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10d80719d0 0x7f10d80bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d8150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10d814ece0 0x7f10d8151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8147740 con 0x7f10d8071e50 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8146b30 con 0x7f10d80719d0 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10dc8f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8146e10 con 0x7f10d814ece0 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10d5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d8150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10d5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d8150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58186/0 (socket says 172.21.15.78:58186) 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10d5d9b640 1 -- 172.21.15.78:0/3242581615 learned_addr learned my addr 172.21.15.78:0/3242581615 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10d6d9d640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10d814ece0 0x7f10d8151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.281+0000 7f10d659c640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10d80719d0 0x7f10d80bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10d5d9b640 1 -- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10d814ece0 msgr2=0x7f10d8151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10d5d9b640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10d814ece0 0x7f10d8151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10d5d9b640 1 -- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10d80719d0 msgr2=0x7f10d80bb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10d5d9b640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10d80719d0 0x7f10d80bb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10d5d9b640 1 -- 172.21.15.78:0/3242581615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8009400 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10d5d9b640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d8150440 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c8c0 tx=0x7f10cc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00a640 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc00a7c0 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d81470f0 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00ab70 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.282+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d8150980 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.283+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d81478f0 con 0x7f10d8071e50 2024-02-19T14:54:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.283+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f10cc00acd0 con 0x7f10d8071e50 2024-02-19T14:54:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.284+0000 7f10c6ffd640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f10b806b840 0x7f10b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.284+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f10cc089c40 con 0x7f10d8071e50 2024-02-19T14:54:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.284+0000 7f10d659c640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f10b806b840 0x7f10b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.284+0000 7f10d659c640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f10b806b840 0x7f10b806dd00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000f60 tx=0x7f10d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.288+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f10cc0544a0 con 0x7f10d8071e50 2024-02-19T14:54:58.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.460+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10d8074990 con 0x7f10d8071e50 2024-02-19T14:54:58.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.461+0000 7f10c6ffd640 1 -- 172.21.15.78:0/3242581615 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17460 (secure 0 0 0) 0x7f10cc018020 con 0x7f10d8071e50 2024-02-19T14:54:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:58.465 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":86,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:56.626216+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6835/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6834/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6836/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f10b806b840 msgr2=0x7f10b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f10b806b840 0x7f10b806dd00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000f60 tx=0x7f10d0009040 comp rx=0 tx=0).stop 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 msgr2=0x7f10d8150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d8150440 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c8c0 tx=0x7f10cc00cd80 comp rx=0 tx=0).stop 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 shutdown_connections 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f10b806b840 0x7f10b806dd00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8071e50 0x7f10d8150440 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10d814ece0 0x7f10d8151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 --2- 172.21.15.78:0/3242581615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10d80719d0 0x7f10d80bb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 >> 172.21.15.78:0/3242581615 conn(0x7f10d806b270 msgr2=0x7f10d8154bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 shutdown_connections 2024-02-19T14:54:58.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.463+0000 7f10dc8f4640 1 -- 172.21.15.78:0/3242581615 wait complete. 2024-02-19T14:54:58.478 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-02-19T14:54:58.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:58.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.574+0000 7f786b1a0640 1 Processor -- start 2024-02-19T14:54:58.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.574+0000 7f786b1a0640 1 -- start start 2024-02-19T14:54:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786b1a0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f786406d740 0x7f78641559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786b1a0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786406db80 con 0x7f78640c7150 2024-02-19T14:54:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786b1a0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78640c7580 con 0x7f78641524b0 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786b1a0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78640c8d50 con 0x7f786406d740 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786a99f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f78640c7150 0x7f78640c50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36536/0 (socket says 172.21.15.78:36536) 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786a99f640 1 -- 172.21.15.78:0/644839553 learned_addr learned my addr 172.21.15.78:0/644839553 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786999d640 1 --2- 172.21.15.78:0/644839553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f786406d740 0x7f78641559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985927966 0 0) 0x7f786406db80 con 0x7f78640c7150 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7850003520 con 0x7f78640c7150 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.575+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377215813 0 0) 0x7f78640c7580 con 0x7f78641524b0 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.576+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f786406db80 con 0x7f78641524b0 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.576+0000 7f786999d640 1 -- 172.21.15.78:0/644839553 >> v1:172.21.15.150:6789/0 conn(0x7f78641524b0 legacy=0x7f78640c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.576+0000 7f786999d640 1 -- 172.21.15.78:0/644839553 >> v1:172.21.15.78:6789/0 conn(0x7f78640c7150 legacy=0x7f78640c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.576+0000 7f786999d640 1 -- 172.21.15.78:0/644839553 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78640c7850 con 0x7f786406d740 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.576+0000 7f786999d640 1 --2- 172.21.15.78:0/644839553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f786406d740 0x7f78641559c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78540097d0 tx=0x7f785402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf76f042b5568df4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854031800 con 0x7f786406d740 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854031960 con 0x7f786406d740 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786919c640 1 -- 172.21.15.78:0/644839553 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854031cf0 con 0x7f786406d740 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 -- 172.21.15.78:0/644839553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f786406d740 msgr2=0x7f78641559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 --2- 172.21.15.78:0/644839553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f786406d740 0x7f78641559c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78540097d0 tx=0x7f785402b8a0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 -- 172.21.15.78:0/644839553 shutdown_connections 2024-02-19T14:54:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 --2- 172.21.15.78:0/644839553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f786406d740 0x7f78641559c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 -- 172.21.15.78:0/644839553 >> 172.21.15.78:0/644839553 conn(0x7f78640ac250 msgr2=0x7f78640ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 -- 172.21.15.78:0/644839553 shutdown_connections 2024-02-19T14:54:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.577+0000 7f786b1a0640 1 -- 172.21.15.78:0/644839553 wait complete. 2024-02-19T14:54:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.578+0000 7f786b1a0640 1 Processor -- start 2024-02-19T14:54:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.578+0000 7f786b1a0640 1 -- start start 2024-02-19T14:54:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786b1a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 0x7f78641eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786b1a0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78640c7150 0x7f786414c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786b1a0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78641524b0 0x7f786414d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786b1a0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786414f1f0 con 0x7f786406d740 2024-02-19T14:54:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786b1a0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786414f4d0 con 0x7f78641524b0 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786b1a0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786414f7b0 con 0x7f78640c7150 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 0x7f78641eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 0x7f78641eb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58192/0 (socket says 172.21.15.78:58192) 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 -- 172.21.15.78:0/4076430142 learned_addr learned my addr 172.21.15.78:0/4076430142 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a99f640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78641524b0 0x7f786414d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786999d640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78640c7150 0x7f786414c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 -- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78640c7150 msgr2=0x7f786414c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78640c7150 0x7f786414c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 -- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78641524b0 msgr2=0x7f786414d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78641524b0 0x7f786414d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.579+0000 7f786a19e640 1 -- 172.21.15.78:0/4076430142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854009400 con 0x7f786406d740 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.580+0000 7f786a19e640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 0x7f78641eb4d0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f786000c430 tx=0x7f786000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.580+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78600052d0 con 0x7f786406d740 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.580+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7860005450 con 0x7f786406d740 2024-02-19T14:54:58.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.580+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860003830 con 0x7f786406d740 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.580+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78641ea9f0 con 0x7f786406d740 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.580+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78641eafb0 con 0x7f786406d740 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.581+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7864074990 con 0x7f786406d740 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.581+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7860003990 con 0x7f786406d740 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.582+0000 7f785affd640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f784806b840 0x7f784806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.582+0000 7f786999d640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f784806b840 0x7f784806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.582+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f7860089a80 con 0x7f786406d740 2024-02-19T14:54:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.582+0000 7f786999d640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f784806b840 0x7f784806dd00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7854036000 tx=0x7f785403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.586+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78600541d0 con 0x7f786406d740 2024-02-19T14:54:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.795+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78640c7df0 con 0x7f786406d740 2024-02-19T14:54:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.796+0000 7f785affd640 1 -- 172.21.15.78:0/4076430142 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f7860057890 con 0x7f786406d740 2024-02-19T14:54:58.800 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:58.800 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6768,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3549382195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3549382195},{"type":"v1","addr":"172.21.15.78:6837","nonce":3549382195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2858964398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2858964398},{"type":"v1","addr":"172.21.15.78:6835","nonce":2858964398}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:56.633799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5354":{"gid":5354,"name":"d","rank":0,"incarnation":64,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:58.800 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 67 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f784806b840 msgr2=0x7f784806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f784806b840 0x7f784806dd00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7854036000 tx=0x7f785403c040 comp rx=0 tx=0).stop 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 msgr2=0x7f78641eb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 0x7f78641eb4d0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f786000c430 tx=0x7f786000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 shutdown_connections 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f784806b840 0x7f784806dd00 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78641524b0 0x7f786414d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78640c7150 0x7f786414c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 --2- 172.21.15.78:0/4076430142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f786406d740 0x7f78641eb4d0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 >> 172.21.15.78:0/4076430142 conn(0x7f78640ac250 msgr2=0x7f786414bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 shutdown_connections 2024-02-19T14:54:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.798+0000 7f786b1a0640 1 -- 172.21.15.78:0/4076430142 wait complete. 2024-02-19T14:54:58.812 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:54:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.909+0000 7f62f87a1640 1 Processor -- start 2024-02-19T14:54:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.909+0000 7f62f87a1640 1 -- start start 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f87a1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62f00c29c0 0x7f62f01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f87a1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f0072250 con 0x7f62f00c5be0 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f87a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f0072480 con 0x7f62f00c25e0 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f87a1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f0152c90 con 0x7f62f00c29c0 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f5d3c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62f00c29c0 0x7f62f01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f5d3c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62f00c29c0 0x7f62f01526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58204/0 (socket says 172.21.15.78:58204) 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f6d3e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f62f00c5be0 0x7f62f006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36552/0 (socket says 172.21.15.78:36552) 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2703490829 learned_addr learned my addr 172.21.15.78:0/2703490829 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2703490829 >> v1:172.21.15.150:6789/0 conn(0x7f62f00c25e0 legacy=0x7f62f00c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2703490829 >> v1:172.21.15.78:6789/0 conn(0x7f62f00c5be0 legacy=0x7f62f006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2703490829 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f00c4b90 con 0x7f62f00c29c0 2024-02-19T14:54:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.910+0000 7f62f553b640 1 -- 172.21.15.78:0/2703490829 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228722169 0 0) 0x7f62f0072250 con 0x7f62f00c5be0 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.911+0000 7f62f5d3c640 1 --2- 172.21.15.78:0/2703490829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01526a0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009870 tx=0x7f62e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4aa47475bb7469 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.911+0000 7f62f553b640 1 -- 172.21.15.78:0/2703490829 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0030860 con 0x7f62f00c29c0 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.911+0000 7f62f553b640 1 -- 172.21.15.78:0/2703490829 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e00309c0 con 0x7f62f00c29c0 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.911+0000 7f62f553b640 1 -- 172.21.15.78:0/2703490829 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0030d50 con 0x7f62f00c29c0 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 -- 172.21.15.78:0/2703490829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 msgr2=0x7f62f01526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2703490829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01526a0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009870 tx=0x7f62e002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 -- 172.21.15.78:0/2703490829 shutdown_connections 2024-02-19T14:54:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2703490829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01526a0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 -- 172.21.15.78:0/2703490829 >> 172.21.15.78:0/2703490829 conn(0x7f62f006b270 msgr2=0x7f62f006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 -- 172.21.15.78:0/2703490829 shutdown_connections 2024-02-19T14:54:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.912+0000 7f62f87a1640 1 -- 172.21.15.78:0/2703490829 wait complete. 2024-02-19T14:54:58.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.913+0000 7f62f87a1640 1 Processor -- start 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.913+0000 7f62f87a1640 1 -- start start 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f00c25e0 0x7f62f00ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f00c5be0 0x7f62f01ea570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0153320 con 0x7f62f00c29c0 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f01edf30 con 0x7f62f00c5be0 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0149ef0 con 0x7f62f00c25e0 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01ea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58216/0 (socket says 172.21.15.78:58216) 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2833364093 learned_addr learned my addr 172.21.15.78:0/2833364093 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f6d3e640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f00c5be0 0x7f62f01ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f653d640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f00c25e0 0x7f62f00ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f00c25e0 msgr2=0x7f62f00ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f00c25e0 0x7f62f00ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f00c5be0 msgr2=0x7f62f01ea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f00c5be0 0x7f62f01ea570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 -- 172.21.15.78:0/2833364093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e00094d0 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f653d640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f00c25e0 0x7f62f00ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f5d3c640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01ea030 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009870 tx=0x7f62e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0030810 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e0043400 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f0149c10 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e00437b0 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.914+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f00c0910 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.915+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f0074990 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.916+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f62e0043910 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.916+0000 7f62e6ffd640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62d406b810 0x7f62d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.916+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f62e003e070 con 0x7f62f00c29c0 2024-02-19T14:54:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.916+0000 7f62f653d640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62d406b810 0x7f62d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.917+0000 7f62f653d640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62d406b810 0x7f62d406dcd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f62f006e3b0 tx=0x7f62ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:58.920+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62e0077eb0 con 0x7f62f00c29c0 2024-02-19T14:54:59.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.129+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62f00abf00 con 0x7f62f00c29c0 2024-02-19T14:54:59.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.130+0000 7f62e6ffd640 1 -- 172.21.15.78:0/2833364093 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f62e007b570 con 0x7f62f00c29c0 2024-02-19T14:54:59.134 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:59.134 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6768,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3549382195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3549382195},{"type":"v1","addr":"172.21.15.78:6837","nonce":3549382195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2858964398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2858964398},{"type":"v1","addr":"172.21.15.78:6835","nonce":2858964398}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:54:37.203340+0000","modified":"2024-02-19T14:54:56.633799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5354":{"gid":5354,"name":"d","rank":0,"incarnation":64,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6834/20921361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":20921361},{"type":"v1","addr":"172.21.15.150:6834","nonce":20921361}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-19T14:54:59.134 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 67 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62d406b810 msgr2=0x7f62d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62d406b810 0x7f62d406dcd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f62f006e3b0 tx=0x7f62ec009040 comp rx=0 tx=0).stop 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 msgr2=0x7f62f01ea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01ea030 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009870 tx=0x7f62e0004730 comp rx=0 tx=0).stop 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 shutdown_connections 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62d406b810 0x7f62d406dcd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62f00c29c0 0x7f62f01ea030 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f00c5be0 0x7f62f01ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 --2- 172.21.15.78:0/2833364093 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62f00c25e0 0x7f62f00ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 >> 172.21.15.78:0/2833364093 conn(0x7f62f006b270 msgr2=0x7f62f006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 shutdown_connections 2024-02-19T14:54:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.132+0000 7f62f87a1640 1 -- 172.21.15.78:0/2833364093 wait complete. 2024-02-19T14:54:59.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:54:59.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.243+0000 7f5e99a4e640 1 Processor -- start 2024-02-19T14:54:59.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.243+0000 7f5e99a4e640 1 -- start start 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.243+0000 7f5e99a4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e99a4e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e941586a0 con 0x7f5e9406c570 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e99a4e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e94158980 con 0x7f5e94152830 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e99a4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e940c6550 con 0x7f5e940c7280 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e937fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5e9406c570 0x7f5e9406ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36554/0 (socket says 172.21.15.78:36554) 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e937fe640 1 -- 172.21.15.78:0/4131327202 learned_addr learned my addr 172.21.15.78:0/4131327202 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e8bfff640 1 --2- 172.21.15.78:0/4131327202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370012821 0 0) 0x7f5e941586a0 con 0x7f5e9406c570 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e80003520 con 0x7f5e9406c570 2024-02-19T14:54:59.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199400766 0 0) 0x7f5e94158980 con 0x7f5e94152830 2024-02-19T14:54:59.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e941586a0 con 0x7f5e94152830 2024-02-19T14:54:59.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e8bfff640 1 -- 172.21.15.78:0/4131327202 >> v1:172.21.15.150:6789/0 conn(0x7f5e94152830 legacy=0x7f5e940c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e8bfff640 1 -- 172.21.15.78:0/4131327202 >> v1:172.21.15.78:6789/0 conn(0x7f5e9406c570 legacy=0x7f5e9406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.244+0000 7f5e8bfff640 1 -- 172.21.15.78:0/4131327202 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e94158c60 con 0x7f5e940c7280 2024-02-19T14:54:59.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.245+0000 7f5e8bfff640 1 --2- 172.21.15.78:0/4131327202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940c7700 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0097d0 tx=0x7f5e7c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d12e388b1e4ee03c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.245+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c031800 con 0x7f5e940c7280 2024-02-19T14:54:59.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.245+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c031960 con 0x7f5e940c7280 2024-02-19T14:54:59.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.245+0000 7f5e927fc640 1 -- 172.21.15.78:0/4131327202 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c031cd0 con 0x7f5e940c7280 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 -- 172.21.15.78:0/4131327202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 msgr2=0x7f5e940c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/4131327202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940c7700 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c0097d0 tx=0x7f5e7c02b8a0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 -- 172.21.15.78:0/4131327202 shutdown_connections 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/4131327202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940c7700 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 -- 172.21.15.78:0/4131327202 >> 172.21.15.78:0/4131327202 conn(0x7f5e940ca8a0 msgr2=0x7f5e940cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 -- 172.21.15.78:0/4131327202 shutdown_connections 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.246+0000 7f5e99a4e640 1 -- 172.21.15.78:0/4131327202 wait complete. 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 Processor -- start 2024-02-19T14:54:59.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 -- start start 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e9406c570 0x7f5e941eb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 0x7f5e940beb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e940c7c40 con 0x7f5e94152830 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94159320 con 0x7f5e9406c570 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e99a4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e941eb970 con 0x7f5e940c7280 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.247+0000 7f5e937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 0x7f5e940beb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 0x7f5e940beb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58232/0 (socket says 172.21.15.78:58232) 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 -- 172.21.15.78:0/566459440 learned_addr learned my addr 172.21.15.78:0/566459440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:59.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e92ffd640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e9406c570 0x7f5e941eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e8bfff640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 -- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 msgr2=0x7f5e940ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 -- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e9406c570 msgr2=0x7f5e941eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e9406c570 0x7f5e941eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 -- 172.21.15.78:0/566459440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c009400 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e937fe640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 0x7f5e940beb20 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00caf0 tx=0x7f5e8c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c00ac40 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c00ada0 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e940be3b0 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c012620 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.248+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e940be690 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.249+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e940be820 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.249+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5e8c007070 con 0x7f5e94152830 2024-02-19T14:54:59.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.250+0000 7f5e90ff9640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e7406b810 0x7f5e7406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.250+0000 7f5e92ffd640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e7406b810 0x7f5e7406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.250+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f5e8c08a330 con 0x7f5e94152830 2024-02-19T14:54:59.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.250+0000 7f5e92ffd640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e7406b810 0x7f5e7406dcd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5e84007fd0 tx=0x7f5e84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.254+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5e8c054b90 con 0x7f5e94152830 2024-02-19T14:54:59.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.427+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e940ca580 con 0x7f5e94152830 2024-02-19T14:54:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.428+0000 7f5e90ff9640 1 -- 172.21.15.78:0/566459440 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+17460 (secure 0 0 0) 0x7f5e8c058250 con 0x7f5e94152830 2024-02-19T14:54:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:54:59.432 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":86,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:54:56.626216+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:54:34.841768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:54:36.252782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.33","mappings":[{"from":1,"to":2}]},{"pgid":"14.38","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6835/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6834/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6836/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e7406b810 msgr2=0x7f5e7406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e7406b810 0x7f5e7406dcd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5e84007fd0 tx=0x7f5e84009040 comp rx=0 tx=0).stop 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 msgr2=0x7f5e940beb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 0x7f5e940beb20 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00caf0 tx=0x7f5e8c00a060 comp rx=0 tx=0).stop 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 shutdown_connections 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e7406b810 0x7f5e7406dcd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e94152830 0x7f5e940beb20 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e940c7280 0x7f5e940ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 --2- 172.21.15.78:0/566459440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e9406c570 0x7f5e941eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 >> 172.21.15.78:0/566459440 conn(0x7f5e940ca8a0 msgr2=0x7f5e9414d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 shutdown_connections 2024-02-19T14:54:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.430+0000 7f5e99a4e640 1 -- 172.21.15.78:0/566459440 wait complete. 2024-02-19T14:54:59.445 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-02-19T14:54:59.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.543+0000 7f044741c640 1 Processor -- start 2024-02-19T14:54:59.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.543+0000 7f044741c640 1 -- start start 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f044741c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04400c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f044741c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04400c5760 0x7f04400c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f044741c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f04400c77c0 0x7f044006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f044741c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04400c6550 con 0x7f04400c77c0 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f044741c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0440159320 con 0x7f04400c5760 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f044741c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04401586a0 con 0x7f0440152830 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f04400c77c0 0x7f044006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04451b8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04400c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f04400c77c0 0x7f044006c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58248/0 (socket says 172.21.15.78:58248) 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04449b7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04400c5760 0x7f04400c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04451b8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04400c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:45084/0 (socket says 172.21.15.78:45084) 2024-02-19T14:54:59.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 -- 172.21.15.78:0/3054183936 learned_addr learned my addr 172.21.15.78:0/3054183936 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 -- 172.21.15.78:0/3054183936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 msgr2=0x7f04400c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 --2- 172.21.15.78:0/3054183936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04400c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 -- 172.21.15.78:0/3054183936 >> v2:172.21.15.150:3300/0 conn(0x7f04400c5760 msgr2=0x7f04400c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 --2- 172.21.15.78:0/3054183936 >> v2:172.21.15.150:3300/0 conn(0x7f04400c5760 0x7f04400c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.544+0000 7f04459b9640 1 -- 172.21.15.78:0/3054183936 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440158980 con 0x7f04400c77c0 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.545+0000 7f04459b9640 1 --2- 172.21.15.78:0/3054183936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f044006c570 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f04400ad420 tx=0x7f043802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af64eb7f57b11275 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.545+0000 7f0437fff640 1 -- 172.21.15.78:0/3054183936 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043802c9e0 con 0x7f04400c77c0 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.545+0000 7f0437fff640 1 -- 172.21.15.78:0/3054183936 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043802cb40 con 0x7f04400c77c0 2024-02-19T14:54:59.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.545+0000 7f0437fff640 1 -- 172.21.15.78:0/3054183936 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380343f0 con 0x7f04400c77c0 2024-02-19T14:54:59.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 -- 172.21.15.78:0/3054183936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 msgr2=0x7f044006c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 --2- 172.21.15.78:0/3054183936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f044006c570 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f04400ad420 tx=0x7f043802be40 comp rx=0 tx=0).stop 2024-02-19T14:54:59.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 -- 172.21.15.78:0/3054183936 shutdown_connections 2024-02-19T14:54:59.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 --2- 172.21.15.78:0/3054183936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f044006c570 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 --2- 172.21.15.78:0/3054183936 >> v2:172.21.15.150:3300/0 conn(0x7f04400c5760 0x7f04400c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 --2- 172.21.15.78:0/3054183936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04400c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 -- 172.21.15.78:0/3054183936 >> 172.21.15.78:0/3054183936 conn(0x7f04400ca8a0 msgr2=0x7f04400cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 -- 172.21.15.78:0/3054183936 shutdown_connections 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.546+0000 7f044741c640 1 -- 172.21.15.78:0/3054183936 wait complete. 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.547+0000 7f044741c640 1 Processor -- start 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.547+0000 7f044741c640 1 -- start start 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04400c5760 0x7f04400abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f04401ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04401502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400c3ba0 con 0x7f04400c77c0 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400c3520 con 0x7f04400c5760 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04400aca00 con 0x7f0440152830 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f04401ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f04401ee240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58260/0 (socket says 172.21.15.78:58260) 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 -- 172.21.15.78:0/1725874791 learned_addr learned my addr 172.21.15.78:0/1725874791 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04451b8640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04400c5760 0x7f04400abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04459b9640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04401502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 -- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 msgr2=0x7f04401502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04401502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 -- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04400c5760 msgr2=0x7f04400abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04400c5760 0x7f04400abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 -- 172.21.15.78:0/1725874791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438009b30 con 0x7f04400c77c0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f04449b7640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f04401ee240 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f043000e890 tx=0x7f043000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043000c8e0 con 0x7f04400c77c0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043000ca40 con 0x7f04400c77c0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04400acca0 con 0x7f04400c77c0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043000cdd0 con 0x7f04400c77c0 2024-02-19T14:54:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.548+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04400ac4d0 con 0x7f04400c77c0 2024-02-19T14:54:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.550+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0440074990 con 0x7f04400c77c0 2024-02-19T14:54:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.550+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0430013470 con 0x7f04400c77c0 2024-02-19T14:54:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.550+0000 7f0435ffb640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f042406b840 0x7f042406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.550+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f043008a480 con 0x7f04400c77c0 2024-02-19T14:54:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.550+0000 7f04451b8640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f042406b840 0x7f042406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.551+0000 7f04451b8640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f042406b840 0x7f042406dd00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f043c009aa0 tx=0x7f043c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.555+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0430054ce0 con 0x7f04400c77c0 2024-02-19T14:54:59.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.763+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f044006dae0 con 0x7f04400c77c0 2024-02-19T14:54:59.767 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:54:59.764+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:54:59.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.765+0000 7f0435ffb640 1 -- 172.21.15.78:0/1725874791 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 163+0+0 (secure 0 0 0) 0x7f04300583a0 con 0x7f04400c77c0 2024-02-19T14:54:59.768 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f042406b840 msgr2=0x7f042406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f042406b840 0x7f042406dd00 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f043c009aa0 tx=0x7f043c009190 comp rx=0 tx=0).stop 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 msgr2=0x7f04401ee240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f04401ee240 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f043000e890 tx=0x7f043000ed50 comp rx=0 tx=0).stop 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 shutdown_connections 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f042406b840 0x7f042406dd00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04400c77c0 0x7f04401ee240 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0440152830 0x7f04401502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 --2- 172.21.15.78:0/1725874791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04400c5760 0x7f04400abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 >> 172.21.15.78:0/1725874791 conn(0x7f04400ca8a0 msgr2=0x7f04400cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:59.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 shutdown_connections 2024-02-19T14:54:59.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.767+0000 7f044741c640 1 -- 172.21.15.78:0/1725874791 wait complete. 2024-02-19T14:54:59.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-02-19T14:54:59.796 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:54:59.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.880+0000 7f289cb8a640 1 Processor -- start 2024-02-19T14:54:59.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.880+0000 7f289cb8a640 1 -- start start 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f289cb8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c42e0 0x7f2898158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f289cb8a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f28980c2f50 0x7f28980c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f289cb8a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28980c34a0 con 0x7f28980c8500 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f289cb8a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f289806c090 con 0x7f28980c2f50 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f289cb8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f289806d6a0 con 0x7f28980c42e0 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f28980c2f50 0x7f28980c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f28980c2f50 0x7f28980c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58276/0 (socket says 172.21.15.78:58276) 2024-02-19T14:54:59.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 -- 172.21.15.78:0/938859676 learned_addr learned my addr 172.21.15.78:0/938859676 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:59.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2895d9b640 1 --2- 172.21.15.78:0/938859676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c42e0 0x7f2898158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 -- 172.21.15.78:0/938859676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c42e0 msgr2=0x7f2898158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:54:59.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 --2- 172.21.15.78:0/938859676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c42e0 0x7f2898158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 -- 172.21.15.78:0/938859676 >> v1:172.21.15.150:6789/0 conn(0x7f28980c8500 legacy=0x7f2898158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.881+0000 7f2896d9d640 1 -- 172.21.15.78:0/938859676 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289806c8e0 con 0x7f28980c2f50 2024-02-19T14:54:59.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.882+0000 7f2896d9d640 1 --2- 172.21.15.78:0/938859676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28980c2f50 0x7f28980c88e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2890009e50 tx=0x7f289002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c59512126565963 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.882+0000 7f289559a640 1 -- 172.21.15.78:0/938859676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289002ca70 con 0x7f28980c2f50 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.882+0000 7f289559a640 1 -- 172.21.15.78:0/938859676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289002cbd0 con 0x7f28980c2f50 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.882+0000 7f289559a640 1 -- 172.21.15.78:0/938859676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28900343f0 con 0x7f28980c2f50 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 -- 172.21.15.78:0/938859676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28980c2f50 msgr2=0x7f28980c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 --2- 172.21.15.78:0/938859676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28980c2f50 0x7f28980c88e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2890009e50 tx=0x7f289002bed0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 -- 172.21.15.78:0/938859676 shutdown_connections 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 --2- 172.21.15.78:0/938859676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28980c2f50 0x7f28980c88e0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 --2- 172.21.15.78:0/938859676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c42e0 0x7f2898158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 -- 172.21.15.78:0/938859676 >> 172.21.15.78:0/938859676 conn(0x7f28980aa300 msgr2=0x7f28980aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 -- 172.21.15.78:0/938859676 shutdown_connections 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.883+0000 7f289cb8a640 1 -- 172.21.15.78:0/938859676 wait complete. 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.884+0000 7f289cb8a640 1 Processor -- start 2024-02-19T14:54:59.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.884+0000 7f289cb8a640 1 -- start start 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289cb8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28980c2f50 0x7f28981ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289cb8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c8500 0x7f28980acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289cb8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 0x7f289814c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289cb8a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289814d120 con 0x7f2898156450 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289cb8a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2898156e20 con 0x7f28980c2f50 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289cb8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28981eaee0 con 0x7f28980c8500 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 0x7f289814c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 0x7f289814c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58278/0 (socket says 172.21.15.78:58278) 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 -- 172.21.15.78:0/3009547997 learned_addr learned my addr 172.21.15.78:0/3009547997 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2895d9b640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c8500 0x7f28980acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f289659c640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28980c2f50 0x7f28981ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 -- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c8500 msgr2=0x7f28980acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c8500 0x7f28980acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 -- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28980c2f50 msgr2=0x7f28981ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28980c2f50 0x7f28981ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 -- 172.21.15.78:0/3009547997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2890009b30 con 0x7f2898156450 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2896d9d640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 0x7f289814c3b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2890036040 tx=0x7f2890004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890009510 con 0x7f2898156450 2024-02-19T14:54:59.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289003e450 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.885+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289003e800 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.886+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289814d5e0 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.886+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28981eeb30 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.886+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2898071c20 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.887+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f289003e960 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.887+0000 7f2886ffd640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287806b840 0x7f287806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.887+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6565+0+0 (secure 0 0 0) 0x7f28900acd60 con 0x7f2898156450 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.887+0000 7f289659c640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287806b840 0x7f287806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:54:59.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.888+0000 7f289659c640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287806b840 0x7f287806dd00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f28880016e0 tx=0x7f2888009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:54:59.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:54:59.892+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2890077560 con 0x7f2898156450 2024-02-19T14:55:00.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.099+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f28981eecc0 con 0x7f2898156450 2024-02-19T14:55:00.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.769+0000 7f2886ffd640 1 -- 172.21.15.78:0/3009547997 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 105+0+0 (secure 0 0 0) 0x7f289007ac20 con 0x7f2898156450 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287806b840 msgr2=0x7f287806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287806b840 0x7f287806dd00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f28880016e0 tx=0x7f2888009040 comp rx=0 tx=0).stop 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 msgr2=0x7f289814c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 0x7f289814c3b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2890036040 tx=0x7f2890004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 shutdown_connections 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287806b840 0x7f287806dd00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2898156450 0x7f289814c3b0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28980c8500 0x7f28980acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 --2- 172.21.15.78:0/3009547997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28980c2f50 0x7f28981ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 >> 172.21.15.78:0/3009547997 conn(0x7f28980aa300 msgr2=0x7f28981541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:00.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 shutdown_connections 2024-02-19T14:55:00.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.771+0000 7f289cb8a640 1 -- 172.21.15.78:0/3009547997 wait complete. 2024-02-19T14:55:00.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-02-19T14:55:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.889+0000 7fe75c00e640 1 Processor -- start 2024-02-19T14:55:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe75c00e640 1 -- start start 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe75c00e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 0x7fe754155720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe75c00e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe754159210 con 0x7fe75406d000 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe75c00e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7540c6420 con 0x7fe754152620 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe75c00e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7540c1f00 con 0x7fe75406d440 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe7595a9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe75406d000 0x7fe754155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36568/0 (socket says 172.21.15.78:36568) 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.890+0000 7fe7595a9640 1 -- 172.21.15.78:0/3582428245 learned_addr learned my addr 172.21.15.78:0/3582428245 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604175893 0 0) 0x7fe754159210 con 0x7fe75406d000 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe740003520 con 0x7fe75406d000 2024-02-19T14:55:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 0x7fe754155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719161117 0 0) 0x7fe740003520 con 0x7fe75406d000 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 msgr2=0x7fe754155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 0x7fe754155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 >> v1:172.21.15.150:6789/0 conn(0x7fe754152620 legacy=0x7fe7540c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75406d8d0 con 0x7fe75406d000 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe744001670 con 0x7fe75406d000 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 >> v1:172.21.15.78:6789/0 conn(0x7fe75406d000 legacy=0x7fe754155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 0x7fe7400039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe740003f20 0x7fe7400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe740004d00 0x7fe740005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74000bb10 con 0x7fe740004d00 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74000bc50 con 0x7fe740003f20 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74000be10 con 0x7fe75406d440 2024-02-19T14:55:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe740004d00 0x7fe740005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe759daa640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 0x7fe7400039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe7595a9640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe740003f20 0x7fe7400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 -- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 msgr2=0x7fe7400039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe75406d440 0x7fe7400039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 -- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe740003f20 msgr2=0x7fe7400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe740003f20 0x7fe7400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.891+0000 7fe75a5ab640 1 -- 172.21.15.78:0/3582428245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740003520 con 0x7fe740004d00 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.892+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe740004d00 0x7fe740005180 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009fc0 tx=0x7fe7400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653d9b9d4a720279 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.892+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c02cc20 con 0x7fe740004d00 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.892+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c00f040 con 0x7fe740004d00 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.892+0000 7fe758da8640 1 -- 172.21.15.78:0/3582428245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c034440 con 0x7fe740004d00 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75c00e640 1 -- 172.21.15.78:0/3582428245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe740004d00 msgr2=0x7fe740005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75c00e640 1 --2- 172.21.15.78:0/3582428245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe740004d00 0x7fe740005180 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009fc0 tx=0x7fe7400057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75a5ab640 1 -- 172.21.15.78:0/3582428245 reap_dead start 2024-02-19T14:55:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75c00e640 1 -- 172.21.15.78:0/3582428245 shutdown_connections 2024-02-19T14:55:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75c00e640 1 -- 172.21.15.78:0/3582428245 >> 172.21.15.78:0/3582428245 conn(0x7fe7540caae0 msgr2=0x7fe7540caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75c00e640 1 -- 172.21.15.78:0/3582428245 shutdown_connections 2024-02-19T14:55:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.893+0000 7fe75c00e640 1 -- 172.21.15.78:0/3582428245 wait complete. 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.894+0000 7fe75c00e640 1 Processor -- start 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.894+0000 7fe75c00e640 1 -- start start 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740004d00 0x7fe7541eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7541506c0 0x7fe754154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 0x7fe75414c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541ee360 con 0x7fe754150b10 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541eab60 con 0x7fe7541506c0 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541ea550 con 0x7fe740004d00 2024-02-19T14:55:00.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 0x7fe75414c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 0x7fe75414c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58302/0 (socket says 172.21.15.78:58302) 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 -- 172.21.15.78:0/2130634251 learned_addr learned my addr 172.21.15.78:0/2130634251 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe7595a9640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7541506c0 0x7fe754154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe759daa640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740004d00 0x7fe7541eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 -- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740004d00 msgr2=0x7fe7541eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740004d00 0x7fe7541eaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 -- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7541506c0 msgr2=0x7fe754154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7541506c0 0x7fe754154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 -- 172.21.15.78:0/2130634251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7541ee560 con 0x7fe754150b10 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe7595a9640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7541506c0 0x7fe754154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe759daa640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740004d00 0x7fe7541eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75a5ab640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 0x7fe75414c040 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe74c02c690 tx=0x7fe74c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c044070 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c034ad0 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c034e80 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7541ea7f0 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.895+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe754154680 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.896+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe754074990 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.897+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe74c03b070 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.897+0000 7fe74a7fc640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe73806b840 0x7fe73806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.897+0000 7fe759daa640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe73806b840 0x7fe73806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.897+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6565+0+0 (secure 0 0 0) 0x7fe74c0abaf0 con 0x7fe754150b10 2024-02-19T14:55:00.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.898+0000 7fe759daa640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe73806b840 0x7fe73806dd00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe7500035f0 tx=0x7fe750009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:00.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:00.902+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe74c076270 con 0x7fe754150b10 2024-02-19T14:55:01.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.072+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe7541eaca0 con 0x7fe754150b10 2024-02-19T14:55:01.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.771+0000 7fe74a7fc640 1 -- 172.21.15.78:0/2130634251 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v88) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe74c039020 con 0x7fe754150b10 2024-02-19T14:55:01.774 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_metadata' removed 2024-02-19T14:55:01.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.774+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe73806b840 msgr2=0x7fe73806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:01.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.774+0000 7fe75c00e640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe73806b840 0x7fe73806dd00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe7500035f0 tx=0x7fe750009040 comp rx=0 tx=0).stop 2024-02-19T14:55:01.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.774+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 msgr2=0x7fe75414c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:01.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.774+0000 7fe75c00e640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 0x7fe75414c040 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe74c02c690 tx=0x7fe74c009750 comp rx=0 tx=0).stop 2024-02-19T14:55:01.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 shutdown_connections 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe73806b840 0x7fe73806dd00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe754150b10 0x7fe75414c040 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7541506c0 0x7fe754154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 --2- 172.21.15.78:0/2130634251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe740004d00 0x7fe7541eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 >> 172.21.15.78:0/2130634251 conn(0x7fe7540caae0 msgr2=0x7fe754155720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 shutdown_connections 2024-02-19T14:55:01.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.775+0000 7fe75c00e640 1 -- 172.21.15.78:0/2130634251 wait complete. 2024-02-19T14:55:01.792 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-02-19T14:55:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.900+0000 7f54ffb0c640 1 Processor -- start 2024-02-19T14:55:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.900+0000 7f54ffb0c640 1 -- start start 2024-02-19T14:55:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54ffb0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 0x7f54f80a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54ffb0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54f80a8530 0x7f54f8152240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54ffb0c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f8155df0 con 0x7f54f8152780 2024-02-19T14:55:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54ffb0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f8072390 con 0x7f54f80a8530 2024-02-19T14:55:01.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54ffb0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80c0b90 con 0x7f54f80c7cd0 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54fd8a8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 0x7f54f80a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54fd0a7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54f80a8530 0x7f54f8152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54fd8a8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 0x7f54f80a9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:45150/0 (socket says 172.21.15.78:45150) 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54fd8a8640 1 -- 172.21.15.78:0/4171950690 learned_addr learned my addr 172.21.15.78:0/4171950690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.901+0000 7f54fe0a9640 1 --1- 172.21.15.78:0/4171950690 >> v1:172.21.15.150:6789/0 conn(0x7f54f8152780 0x7f54f80727e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:39136/0 (socket says 172.21.15.78:39136) 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.902+0000 7f54fd8a8640 1 -- 172.21.15.78:0/4171950690 >> v2:172.21.15.150:3300/0 conn(0x7f54f80a8530 msgr2=0x7f54f8152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.902+0000 7f54fd8a8640 1 --2- 172.21.15.78:0/4171950690 >> v2:172.21.15.150:3300/0 conn(0x7f54f80a8530 0x7f54f8152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.902+0000 7f54fd8a8640 1 -- 172.21.15.78:0/4171950690 >> v1:172.21.15.150:6789/0 conn(0x7f54f8152780 legacy=0x7f54f80727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.902+0000 7f54fd8a8640 1 -- 172.21.15.78:0/4171950690 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f80c0e70 con 0x7f54f80c7cd0 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.902+0000 7f54fd0a7640 1 --2- 172.21.15.78:0/4171950690 >> v2:172.21.15.150:3300/0 conn(0x7f54f80a8530 0x7f54f8152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.902+0000 7f54fd8a8640 1 --2- 172.21.15.78:0/4171950690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 0x7f54f80a9e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009870 tx=0x7f54f402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2bf928a53605224 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54fc8a6640 1 -- 172.21.15.78:0/4171950690 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4030840 con 0x7f54f80c7cd0 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54fc8a6640 1 -- 172.21.15.78:0/4171950690 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f40309a0 con 0x7f54f80c7cd0 2024-02-19T14:55:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54fc8a6640 1 -- 172.21.15.78:0/4171950690 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4030d30 con 0x7f54f80c7cd0 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4171950690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 msgr2=0x7f54f80a9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4171950690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 0x7f54f80a9e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009870 tx=0x7f54f402b980 comp rx=0 tx=0).stop 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4171950690 shutdown_connections 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4171950690 >> v2:172.21.15.150:3300/0 conn(0x7f54f80a8530 0x7f54f8152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4171950690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80c7cd0 0x7f54f80a9e00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4171950690 >> 172.21.15.78:0/4171950690 conn(0x7f54f806f5e0 msgr2=0x7f54f806f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4171950690 shutdown_connections 2024-02-19T14:55:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.903+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4171950690 wait complete. 2024-02-19T14:55:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.904+0000 7f54ffb0c640 1 Processor -- start 2024-02-19T14:55:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.904+0000 7f54ffb0c640 1 -- start start 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54ffb0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80a8530 0x7f54f81e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54ffb0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 0x7f54f81e7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54ffb0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f8152780 0x7f54f80ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54ffb0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80ad050 con 0x7f54f80c7cd0 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54ffb0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f80ad2f0 con 0x7f54f8152780 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54ffb0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f8154bb0 con 0x7f54f80a8530 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 0x7f54f81e7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:01.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 0x7f54f81e7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58306/0 (socket says 172.21.15.78:58306) 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 -- 172.21.15.78:0/4220151874 learned_addr learned my addr 172.21.15.78:0/4220151874 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fe0a9640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f8152780 0x7f54f80ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd8a8640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80a8530 0x7f54f81e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 -- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80a8530 msgr2=0x7f54f81e7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80a8530 0x7f54f81e7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 -- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f8152780 msgr2=0x7f54f80ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f8152780 0x7f54f80ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd0a7640 1 -- 172.21.15.78:0/4220151874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8009400 con 0x7f54f80c7cd0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.905+0000 7f54fd8a8640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80a8530 0x7f54f81e7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.906+0000 7f54fd0a7640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 0x7f54f81e7f50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f54e800e9b0 tx=0x7f54e800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.906+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e800ca00 con 0x7f54f80c7cd0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.906+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e800cb60 con 0x7f54f80c7cd0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.906+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f40094d0 con 0x7f54f80c7cd0 2024-02-19T14:55:01.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.906+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e8013470 con 0x7f54f80c7cd0 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.906+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f80c1f60 con 0x7f54f80c7cd0 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.907+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f80ad4a0 con 0x7f54f80c7cd0 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.907+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f54e80135d0 con 0x7f54f80c7cd0 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.908+0000 7f54ee7fc640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b810 0x7f54dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.908+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f54e808a580 con 0x7f54f80c7cd0 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.908+0000 7f54fd8a8640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b810 0x7f54dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:01.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.908+0000 7f54fd8a8640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b810 0x7f54dc06dcd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54f4036000 tx=0x7f54f4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:01.912+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f54e8054f40 con 0x7f54f80c7cd0 2024-02-19T14:55:02.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.089+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54f8074990 con 0x7f54f80c7cd0 2024-02-19T14:55:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.776+0000 7f54ee7fc640 1 -- 172.21.15.78:0/4220151874 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v89) v1 ==== 172+0+0 (secure 0 0 0) 0x7f54e8058600 con 0x7f54f80c7cd0 2024-02-19T14:55:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_data' removed 2024-02-19T14:55:02.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b810 msgr2=0x7f54dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b810 0x7f54dc06dcd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54f4036000 tx=0x7f54f4008030 comp rx=0 tx=0).stop 2024-02-19T14:55:02.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 msgr2=0x7f54f81e7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 0x7f54f81e7f50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f54e800e9b0 tx=0x7f54e800ee70 comp rx=0 tx=0).stop 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 shutdown_connections 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f54dc06b810 0x7f54dc06dcd0 secure :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54f4036000 tx=0x7f54f4008030 comp rx=0 tx=0).stop 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54f80c7cd0 0x7f54f81e7f50 secure :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f54e800e9b0 tx=0x7f54e800ee70 comp rx=0 tx=0).stop 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54f8152780 0x7f54f80ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 --2- 172.21.15.78:0/4220151874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54f80a8530 0x7f54f81e7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 >> 172.21.15.78:0/4220151874 conn(0x7f54f806f5e0 msgr2=0x7f54f806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.779+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 shutdown_connections 2024-02-19T14:55:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.780+0000 7f54ffb0c640 1 -- 172.21.15.78:0/4220151874 wait complete. 2024-02-19T14:55:02.797 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-19T14:55:02.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.897+0000 7f275444e640 1 Processor -- start 2024-02-19T14:55:02.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.897+0000 7f275444e640 1 -- start start 2024-02-19T14:55:02.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.898+0000 7f275444e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f274c0c4980 0x7f274c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.898+0000 7f275444e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0ae670 con 0x7f274c0add70 2024-02-19T14:55:02.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.898+0000 7f275444e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0716b0 con 0x7f274c0705b0 2024-02-19T14:55:02.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.898+0000 7f275444e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c153780 con 0x7f274c0c4980 2024-02-19T14:55:02.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.898+0000 7f27521ea640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f274c0add70 0x7f274c06eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36580/0 (socket says 172.21.15.78:36580) 2024-02-19T14:55:02.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.898+0000 7f27521ea640 1 -- 172.21.15.78:0/3754907234 learned_addr learned my addr 172.21.15.78:0/3754907234 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:02.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 --2- 172.21.15.78:0/3754907234 >> v2:172.21.15.150:3300/0 conn(0x7f274c0c4980 0x7f274c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654415673 0 0) 0x7f274c0ae670 con 0x7f274c0add70 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2738003520 con 0x7f274c0add70 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251909391 0 0) 0x7f2738003520 con 0x7f274c0add70 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 >> v2:172.21.15.150:3300/0 conn(0x7f274c0c4980 msgr2=0x7f274c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 --2- 172.21.15.78:0/3754907234 >> v2:172.21.15.150:3300/0 conn(0x7f274c0c4980 0x7f274c0c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 >> v1:172.21.15.150:6789/0 conn(0x7f274c0705b0 legacy=0x7f274c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c153a60 con 0x7f274c0add70 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f27480019f0 con 0x7f274c0add70 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 >> v1:172.21.15.78:6789/0 conn(0x7f274c0add70 legacy=0x7f274c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2738003970 0x7f2738003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27380047c0 0x7f2738004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2738005180 0x7f273800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c153780 con 0x7f27380047c0 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273800c120 con 0x7f2738003970 2024-02-19T14:55:02.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273800c2e0 con 0x7f2738005180 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27380047c0 0x7f2738004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27521ea640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2738003970 0x7f2738003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27529eb640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2738005180 0x7f273800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 -- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2738005180 msgr2=0x7f273800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2738005180 0x7f273800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 -- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2738003970 msgr2=0x7f2738003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2738003970 0x7f2738003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27519e9640 1 -- 172.21.15.78:0/3754907234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2738003520 con 0x7f27380047c0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.899+0000 7f27529eb640 1 -- 172.21.15.78:0/3754907234 reap_dead start 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.900+0000 7f27519e9640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27380047c0 0x7f2738004c40 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f273c009ac0 tx=0x7f27380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2017a9e862c97947 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.900+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c02c6d0 con 0x7f27380047c0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.900+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c02c830 con 0x7f27380047c0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.900+0000 7f27511e8640 1 -- 172.21.15.78:0/3754907234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c02cbc0 con 0x7f27380047c0 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 -- 172.21.15.78:0/3754907234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27380047c0 msgr2=0x7f2738004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27380047c0 0x7f2738004c40 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f273c009ac0 tx=0x7f27380057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 -- 172.21.15.78:0/3754907234 shutdown_connections 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 --2- 172.21.15.78:0/3754907234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27380047c0 0x7f2738004c40 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 -- 172.21.15.78:0/3754907234 >> 172.21.15.78:0/3754907234 conn(0x7f274c0c85e0 msgr2=0x7f274c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 -- 172.21.15.78:0/3754907234 shutdown_connections 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.901+0000 7f275444e640 1 -- 172.21.15.78:0/3754907234 wait complete. 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.902+0000 7f275444e640 1 Processor -- start 2024-02-19T14:55:02.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.902+0000 7f275444e640 1 -- start start 2024-02-19T14:55:02.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27380047c0 0x7f274c0c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f274c1ee960 0x7f274c147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 0x7f274c147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c154430 con 0x7f274c06bf90 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c1eee60 con 0x7f27380047c0 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0ace80 con 0x7f274c1ee960 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 0x7f274c147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 0x7f274c147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58328/0 (socket says 172.21.15.78:58328) 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 -- 172.21.15.78:0/2297364638 learned_addr learned my addr 172.21.15.78:0/2297364638 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27521ea640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27380047c0 0x7f274c0c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27519e9640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f274c1ee960 0x7f274c147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 -- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f274c1ee960 msgr2=0x7f274c147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f274c1ee960 0x7f274c147650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 -- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27380047c0 msgr2=0x7f274c0c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27380047c0 0x7f274c0c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 -- 172.21.15.78:0/2297364638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c0ad160 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27519e9640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f274c1ee960 0x7f274c147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f27529eb640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 0x7f274c147b90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f274c152c60 tx=0x7f274400e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27440050d0 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2744005250 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c154050 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27440035f0 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.903+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c0a7e70 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.904+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c074990 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.905+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2744003750 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.905+0000 7f2742ffd640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f273006b840 0x7f273006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.905+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2744089480 con 0x7f274c06bf90 2024-02-19T14:55:02.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.905+0000 7f27521ea640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f273006b840 0x7f273006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:02.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.906+0000 7f27521ea640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f273006b840 0x7f273006dd00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2748004fd0 tx=0x7f2748006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:02.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:02.909+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2744054050 con 0x7f274c06bf90 2024-02-19T14:55:03.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.081+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f274c0ae3e0 con 0x7f274c06bf90 2024-02-19T14:55:03.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.083+0000 7f2742ffd640 1 -- 172.21.15.78:0/2297364638 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v27) v1 ==== 96+0+0 (secure 0 0 0) 0x7f274c0ae3e0 con 0x7f274c06bf90 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f273006b840 msgr2=0x7f273006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f273006b840 0x7f273006dd00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2748004fd0 tx=0x7f2748006040 comp rx=0 tx=0).stop 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 msgr2=0x7f274c147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 0x7f274c147b90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f274c152c60 tx=0x7f274400e5f0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 shutdown_connections 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f273006b840 0x7f273006dd00 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f274c06bf90 0x7f274c147b90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f274c1ee960 0x7f274c147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 --2- 172.21.15.78:0/2297364638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27380047c0 0x7f274c0c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 >> 172.21.15.78:0/2297364638 conn(0x7f274c0c85e0 msgr2=0x7f274c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 shutdown_connections 2024-02-19T14:55:03.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.085+0000 7f275444e640 1 -- 172.21.15.78:0/2297364638 wait complete. 2024-02-19T14:55:03.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:55:03.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 Processor -- start 2024-02-19T14:55:03.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 -- start start 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd728152670 0x7fd728071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7280702d0 0x7fd728158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd728071bf0 con 0x7fd728152a50 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd728158a20 con 0x7fd728152670 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72fac0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7280c6780 con 0x7fd7280702d0 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72d85c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd728152670 0x7fd728071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72d85c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd728152670 0x7fd728071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58338/0 (socket says 172.21.15.78:58338) 2024-02-19T14:55:03.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.198+0000 7fd72d85c640 1 -- 172.21.15.78:0/2633999056 learned_addr learned my addr 172.21.15.78:0/2633999056 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.199+0000 7fd72e05d640 1 --2- 172.21.15.78:0/2633999056 >> v2:172.21.15.150:3300/0 conn(0x7fd7280702d0 0x7fd728158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.199+0000 7fd72d85c640 1 -- 172.21.15.78:0/2633999056 >> v2:172.21.15.150:3300/0 conn(0x7fd7280702d0 msgr2=0x7fd728158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.199+0000 7fd72d85c640 1 --2- 172.21.15.78:0/2633999056 >> v2:172.21.15.150:3300/0 conn(0x7fd7280702d0 0x7fd728158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.199+0000 7fd72d85c640 1 -- 172.21.15.78:0/2633999056 >> v1:172.21.15.150:6789/0 conn(0x7fd728152a50 legacy=0x7fd728155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.199+0000 7fd72d85c640 1 -- 172.21.15.78:0/2633999056 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd728071630 con 0x7fd728152670 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.199+0000 7fd72d85c640 1 --2- 172.21.15.78:0/2633999056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd728152670 0x7fd728071050 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd728070df0 tx=0x7fd72402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4f8e3930c8a0e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72c85a640 1 -- 172.21.15.78:0/2633999056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724030860 con 0x7fd728152670 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72c85a640 1 -- 172.21.15.78:0/2633999056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7240309c0 con 0x7fd728152670 2024-02-19T14:55:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72c85a640 1 -- 172.21.15.78:0/2633999056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724030d50 con 0x7fd728152670 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 -- 172.21.15.78:0/2633999056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd728152670 msgr2=0x7fd728071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 --2- 172.21.15.78:0/2633999056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd728152670 0x7fd728071050 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd728070df0 tx=0x7fd72402b9a0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 -- 172.21.15.78:0/2633999056 shutdown_connections 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 --2- 172.21.15.78:0/2633999056 >> v2:172.21.15.150:3300/0 conn(0x7fd7280702d0 0x7fd728158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 --2- 172.21.15.78:0/2633999056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd728152670 0x7fd728071050 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 -- 172.21.15.78:0/2633999056 >> 172.21.15.78:0/2633999056 conn(0x7fd7280c85e0 msgr2=0x7fd7280c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 -- 172.21.15.78:0/2633999056 shutdown_connections 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.200+0000 7fd72fac0640 1 -- 172.21.15.78:0/2633999056 wait complete. 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.201+0000 7fd72fac0640 1 Processor -- start 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.201+0000 7fd72fac0640 1 -- start start 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72fac0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 0x7fd7280ad820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72fac0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd728152670 0x7fd7280add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72fac0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd728152a50 0x7fd7281edea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72fac0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280ad400 con 0x7fd7280702d0 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72fac0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280ae2a0 con 0x7fd728152a50 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72fac0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7281e86a0 con 0x7fd728152670 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 0x7fd7280ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 0x7fd7280ad820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58348/0 (socket says 172.21.15.78:58348) 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 -- 172.21.15.78:0/742701077 learned_addr learned my addr 172.21.15.78:0/742701077 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72e05d640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd728152a50 0x7fd7281edea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d05b640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd728152670 0x7fd7280add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 -- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd728152670 msgr2=0x7fd7280add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd728152670 0x7fd7280add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 -- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd728152a50 msgr2=0x7fd7281edea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd728152a50 0x7fd7281edea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.202+0000 7fd72d85c640 1 -- 172.21.15.78:0/742701077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7240094d0 con 0x7fd7280702d0 2024-02-19T14:55:03.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.203+0000 7fd72d85c640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 0x7fd7280ad820 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd72402b9e0 tx=0x7fd724004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.203+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724039d60 con 0x7fd7280702d0 2024-02-19T14:55:03.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.203+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd724043400 con 0x7fd7280702d0 2024-02-19T14:55:03.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.203+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7281521f0 con 0x7fd7280702d0 2024-02-19T14:55:03.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.203+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7240437b0 con 0x7fd7280702d0 2024-02-19T14:55:03.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.203+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7281df060 con 0x7fd7280702d0 2024-02-19T14:55:03.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.204+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7280676a0 con 0x7fd7280702d0 2024-02-19T14:55:03.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.204+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd724043910 con 0x7fd7280702d0 2024-02-19T14:55:03.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.205+0000 7fd71a7fc640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd70c06b810 0x7fd70c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.205+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fd72403e070 con 0x7fd7280702d0 2024-02-19T14:55:03.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.205+0000 7fd72d05b640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd70c06b810 0x7fd70c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.205+0000 7fd72d05b640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd70c06b810 0x7fd70c06dcd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd728156430 tx=0x7fd71c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.209+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd724077da0 con 0x7fd7280702d0 2024-02-19T14:55:03.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.381+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fd728070120 con 0x7fd7280702d0 2024-02-19T14:55:03.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.384+0000 7fd71a7fc640 1 -- 172.21.15.78:0/742701077 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v28) v1 ==== 96+0+0 (secure 0 0 0) 0x7fd72407b460 con 0x7fd7280702d0 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd70c06b810 msgr2=0x7fd70c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd70c06b810 0x7fd70c06dcd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd728156430 tx=0x7fd71c009040 comp rx=0 tx=0).stop 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 msgr2=0x7fd7280ad820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 0x7fd7280ad820 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd72402b9e0 tx=0x7fd724004730 comp rx=0 tx=0).stop 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 shutdown_connections 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd70c06b810 0x7fd70c06dcd0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd728152a50 0x7fd7281edea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd728152670 0x7fd7280add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 --2- 172.21.15.78:0/742701077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7280702d0 0x7fd7280ad820 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 >> 172.21.15.78:0/742701077 conn(0x7fd7280c85e0 msgr2=0x7fd728157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 shutdown_connections 2024-02-19T14:55:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.386+0000 7fd72fac0640 1 -- 172.21.15.78:0/742701077 wait complete. 2024-02-19T14:55:03.399 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:55:03.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab662ae640 1 Processor -- start 2024-02-19T14:55:03.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab662ae640 1 -- start start 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab662ae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fab6006c9b0 0x7fab60160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab662ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab601551f0 con 0x7fab600c8e00 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab662ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab60155490 con 0x7fab60152700 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab662ae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab60155770 con 0x7fab6006c9b0 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab6484b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fab6006c9b0 0x7fab60160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab5f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fab600c8e00 0x7fab6006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36586/0 (socket says 172.21.15.78:36586) 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab5f7fe640 1 -- 172.21.15.78:0/3298059696 learned_addr learned my addr 172.21.15.78:0/3298059696 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.499+0000 7fab6484b640 1 --2- 172.21.15.78:0/3298059696 >> v2:172.21.15.78:3300/0 conn(0x7fab6006c9b0 0x7fab60160990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58354/0 (socket says 172.21.15.78:58354) 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.500+0000 7fab6484b640 1 -- 172.21.15.78:0/3298059696 >> v1:172.21.15.150:6789/0 conn(0x7fab60152700 legacy=0x7fab600c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.500+0000 7fab6484b640 1 -- 172.21.15.78:0/3298059696 >> v1:172.21.15.78:6789/0 conn(0x7fab600c8e00 legacy=0x7fab6006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.500+0000 7fab6484b640 1 -- 172.21.15.78:0/3298059696 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60155a50 con 0x7fab6006c9b0 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.500+0000 7fab5effd640 1 -- 172.21.15.78:0/3298059696 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116938292 0 0) 0x7fab601551f0 con 0x7fab600c8e00 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.500+0000 7fab6484b640 1 --2- 172.21.15.78:0/3298059696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab6006c9b0 0x7fab60160990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fab58009d90 tx=0x7fab5802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c166b05373f2ef7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab5effd640 1 -- 172.21.15.78:0/3298059696 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5802cc60 con 0x7fab6006c9b0 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab5effd640 1 -- 172.21.15.78:0/3298059696 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5802cdc0 con 0x7fab6006c9b0 2024-02-19T14:55:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab5effd640 1 -- 172.21.15.78:0/3298059696 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58034660 con 0x7fab6006c9b0 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 -- 172.21.15.78:0/3298059696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab6006c9b0 msgr2=0x7fab60160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 --2- 172.21.15.78:0/3298059696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab6006c9b0 0x7fab60160990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fab58009d90 tx=0x7fab5802beb0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 -- 172.21.15.78:0/3298059696 shutdown_connections 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 --2- 172.21.15.78:0/3298059696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab6006c9b0 0x7fab60160990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 -- 172.21.15.78:0/3298059696 >> 172.21.15.78:0/3298059696 conn(0x7fab600cac90 msgr2=0x7fab600cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 -- 172.21.15.78:0/3298059696 shutdown_connections 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.501+0000 7fab662ae640 1 -- 172.21.15.78:0/3298059696 wait complete. 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.502+0000 7fab662ae640 1 Processor -- start 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.502+0000 7fab662ae640 1 -- start start 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab662ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6006c9b0 0x7fab600c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab662ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 0x7fab601ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab662ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab60152700 0x7fab601eac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab662ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601ee530 con 0x7fab600c8e00 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab662ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601edc90 con 0x7fab6006c9b0 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab662ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600aa900 con 0x7fab60152700 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 0x7fab601ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 0x7fab601ea720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58368/0 (socket says 172.21.15.78:58368) 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 -- 172.21.15.78:0/2035207572 learned_addr learned my addr 172.21.15.78:0/2035207572 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5ffff640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6006c9b0 0x7fab600c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab6484b640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab60152700 0x7fab601eac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 -- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab60152700 msgr2=0x7fab601eac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab60152700 0x7fab601eac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 -- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6006c9b0 msgr2=0x7fab600c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6006c9b0 0x7fab600c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.503+0000 7fab5f7fe640 1 -- 172.21.15.78:0/2035207572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58009a70 con 0x7fab600c8e00 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.504+0000 7fab5f7fe640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 0x7fab601ea720 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fab5400c4d0 tx=0x7fab5400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.504+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab540074e0 con 0x7fab600c8e00 2024-02-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.504+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab54007660 con 0x7fab600c8e00 2024-02-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.504+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab601edee0 con 0x7fab600c8e00 2024-02-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.504+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54004a20 con 0x7fab600c8e00 2024-02-19T14:55:03.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.504+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab600ca160 con 0x7fab600c8e00 2024-02-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.505+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab601ee6e0 con 0x7fab600c8e00 2024-02-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.505+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fab54004b80 con 0x7fab600c8e00 2024-02-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.505+0000 7fab5cff9640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab4c06b810 0x7fab4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.506+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fab54089660 con 0x7fab600c8e00 2024-02-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.506+0000 7fab5ffff640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab4c06b810 0x7fab4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.506+0000 7fab5ffff640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab4c06b810 0x7fab4c06dcd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fab50006fd0 tx=0x7fab50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.510+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fab601ee6e0 con 0x7fab600c8e00 2024-02-19T14:55:03.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.681+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fab601ee6e0 con 0x7fab600c8e00 2024-02-19T14:55:03.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.684+0000 7fab5cff9640 1 -- 172.21.15.78:0/2035207572 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab601ee6e0 con 0x7fab600c8e00 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab4c06b810 msgr2=0x7fab4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab4c06b810 0x7fab4c06dcd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fab50006fd0 tx=0x7fab50008040 comp rx=0 tx=0).stop 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 msgr2=0x7fab601ea720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 0x7fab601ea720 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fab5400c4d0 tx=0x7fab5400c990 comp rx=0 tx=0).stop 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 shutdown_connections 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab4c06b810 0x7fab4c06dcd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab600c8e00 0x7fab601ea720 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab60152700 0x7fab601eac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 --2- 172.21.15.78:0/2035207572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6006c9b0 0x7fab600c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 >> 172.21.15.78:0/2035207572 conn(0x7fab600cac90 msgr2=0x7fab600cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 shutdown_connections 2024-02-19T14:55:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.686+0000 7fab662ae640 1 -- 172.21.15.78:0/2035207572 wait complete. 2024-02-19T14:55:03.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-02-19T14:55:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.799+0000 7fc4c26ec640 1 Processor -- start 2024-02-19T14:55:03.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.799+0000 7fc4c26ec640 1 -- start start 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c26ec640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ad890 0x7fc4bc0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c26ec640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 0x7fc4bc0c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c26ec640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc152d30 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c26ec640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc0aba90 con 0x7fc4bc0ab600 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c26ec640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0c7860 con 0x7fc4bc0ad890 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c0c89640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 0x7fc4bc0c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bbfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ad890 0x7fc4bc0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bb7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc4bc0c49c0 0x7fc4bc1534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:39186/0 (socket says 172.21.15.78:39186) 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c0c89640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 0x7fc4bc0c5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:41808/0 (socket says 172.21.15.78:41808) 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bb7fe640 1 -- 172.21.15.78:0/730205652 learned_addr learned my addr 172.21.15.78:0/730205652 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bbfff640 1 -- 172.21.15.78:0/730205652 >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 msgr2=0x7fc4bc0c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bbfff640 1 --2- 172.21.15.78:0/730205652 >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 0x7fc4bc0c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bbfff640 1 -- 172.21.15.78:0/730205652 >> v1:172.21.15.150:6789/0 conn(0x7fc4bc0c49c0 legacy=0x7fc4bc1534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4bbfff640 1 -- 172.21.15.78:0/730205652 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0c7ce0 con 0x7fc4bc0ad890 2024-02-19T14:55:03.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.800+0000 7fc4c0c89640 1 --2- 172.21.15.78:0/730205652 >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 0x7fc4bc0c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.801+0000 7fc4bbfff640 1 --2- 172.21.15.78:0/730205652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ad890 0x7fc4bc0c64f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0097a0 tx=0x7fc4ac02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56de153d4111fbfb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.801+0000 7fc4baffd640 1 -- 172.21.15.78:0/730205652 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac02f050 con 0x7fc4bc0ad890 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.801+0000 7fc4baffd640 1 -- 172.21.15.78:0/730205652 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac031700 con 0x7fc4bc0ad890 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.801+0000 7fc4baffd640 1 -- 172.21.15.78:0/730205652 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac031a90 con 0x7fc4bc0ad890 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 -- 172.21.15.78:0/730205652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ad890 msgr2=0x7fc4bc0c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/730205652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ad890 0x7fc4bc0c64f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0097a0 tx=0x7fc4ac02b7a0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 -- 172.21.15.78:0/730205652 shutdown_connections 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/730205652 >> v2:172.21.15.150:3300/0 conn(0x7fc4bc0ab600 0x7fc4bc0c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/730205652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ad890 0x7fc4bc0c64f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 -- 172.21.15.78:0/730205652 >> 172.21.15.78:0/730205652 conn(0x7fc4bc06b700 msgr2=0x7fc4bc0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 -- 172.21.15.78:0/730205652 shutdown_connections 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.802+0000 7fc4c26ec640 1 -- 172.21.15.78:0/730205652 wait complete. 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.803+0000 7fc4c26ec640 1 Processor -- start 2024-02-19T14:55:03.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.803+0000 7fc4c26ec640 1 -- start start 2024-02-19T14:55:03.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c26ec640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ab600 0x7fc4bc14c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c26ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4bc0ad890 0x7fc4bc156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c26ec640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 0x7fc4bc156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c26ec640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc14cda0 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c26ec640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc157310 con 0x7fc4bc0ad890 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c26ec640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc1527f0 con 0x7fc4bc0ab600 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 0x7fc4bc156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 0x7fc4bc156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58372/0 (socket says 172.21.15.78:58372) 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 -- 172.21.15.78:0/793752474 learned_addr learned my addr 172.21.15.78:0/793752474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4bb7fe640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4bc0ad890 0x7fc4bc156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4bbfff640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ab600 0x7fc4bc14c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 -- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ab600 msgr2=0x7fc4bc14c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ab600 0x7fc4bc14c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 -- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4bc0ad890 msgr2=0x7fc4bc156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4bc0ad890 0x7fc4bc156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.804+0000 7fc4c0c89640 1 -- 172.21.15.78:0/793752474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4009a70 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.805+0000 7fc4c0c89640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 0x7fc4bc156dd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400efc0 tx=0x7fc4b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.805+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4009cd0 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.805+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b400f040 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.805+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac009400 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.805+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4003990 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.805+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc1e93d0 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.806+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc1e8f20 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.806+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc4b4014480 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.806+0000 7fc4b8ff9640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc4a806b810 0x7fc4a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.807+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fc4b408aba0 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.807+0000 7fc4bbfff640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc4a806b810 0x7fc4a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:03.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.807+0000 7fc4bbfff640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc4a806b810 0x7fc4a806dcd0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac035000 tx=0x7fc4ac008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:03.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.811+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc4b4055770 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.990+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7fc4bc153ee0 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.992+0000 7fc4b8ff9640 1 -- 172.21.15.78:0/793752474 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v30) v1 ==== 96+0+0 (secure 0 0 0) 0x7fc4b4058e30 con 0x7fc4bc0c49c0 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.994+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc4a806b810 msgr2=0x7fc4a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.994+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc4a806b810 0x7fc4a806dcd0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac035000 tx=0x7fc4ac008030 comp rx=0 tx=0).stop 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 msgr2=0x7fc4bc156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 0x7fc4bc156dd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400efc0 tx=0x7fc4b400c490 comp rx=0 tx=0).stop 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 shutdown_connections 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc4a806b810 0x7fc4a806dcd0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4bc0c49c0 0x7fc4bc156dd0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4bc0ad890 0x7fc4bc156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 --2- 172.21.15.78:0/793752474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4bc0ab600 0x7fc4bc14c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 >> 172.21.15.78:0/793752474 conn(0x7fc4bc06b700 msgr2=0x7fc4bc06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 shutdown_connections 2024-02-19T14:55:03.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:03.995+0000 7fc4c26ec640 1 -- 172.21.15.78:0/793752474 wait complete. 2024-02-19T14:55:04.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-02-19T14:55:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.105+0000 7f50883c9640 1 Processor -- start 2024-02-19T14:55:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.105+0000 7f50883c9640 1 -- start start 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f50883c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5080155e00 0x7f50800c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f50883c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50800c67e0 0x7f508006c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f50883c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800c7df0 con 0x7f50800c39e0 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f50883c9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800c4310 con 0x7f5080155e00 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f50883c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800c77c0 con 0x7f50800c67e0 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5080155e00 0x7f50800c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5080155e00 0x7f50800c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58376/0 (socket says 172.21.15.78:58376) 2024-02-19T14:55:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 -- 172.21.15.78:0/688888443 learned_addr learned my addr 172.21.15.78:0/688888443 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5085964640 1 --2- 172.21.15.78:0/688888443 >> v2:172.21.15.150:3300/0 conn(0x7f50800c67e0 0x7f508006c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 -- 172.21.15.78:0/688888443 >> v2:172.21.15.150:3300/0 conn(0x7f50800c67e0 msgr2=0x7f508006c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 --2- 172.21.15.78:0/688888443 >> v2:172.21.15.150:3300/0 conn(0x7f50800c67e0 0x7f508006c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 -- 172.21.15.78:0/688888443 >> v1:172.21.15.150:6789/0 conn(0x7f50800c39e0 legacy=0x7f50800c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.106+0000 7f5086165640 1 -- 172.21.15.78:0/688888443 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800c70d0 con 0x7f5080155e00 2024-02-19T14:55:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.107+0000 7f5086165640 1 --2- 172.21.15.78:0/688888443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5080155e00 0x7f50800c34a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5080071b10 tx=0x7f507c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56db721d66d574b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.107+0000 7f5085163640 1 -- 172.21.15.78:0/688888443 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c031870 con 0x7f5080155e00 2024-02-19T14:55:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.107+0000 7f5085163640 1 -- 172.21.15.78:0/688888443 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c0319d0 con 0x7f5080155e00 2024-02-19T14:55:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.107+0000 7f5085163640 1 -- 172.21.15.78:0/688888443 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c031d60 con 0x7f5080155e00 2024-02-19T14:55:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 -- 172.21.15.78:0/688888443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5080155e00 msgr2=0x7f50800c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 --2- 172.21.15.78:0/688888443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5080155e00 0x7f50800c34a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5080071b10 tx=0x7f507c02b910 comp rx=0 tx=0).stop 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 -- 172.21.15.78:0/688888443 shutdown_connections 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 --2- 172.21.15.78:0/688888443 >> v2:172.21.15.150:3300/0 conn(0x7f50800c67e0 0x7f508006c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 --2- 172.21.15.78:0/688888443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5080155e00 0x7f50800c34a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 -- 172.21.15.78:0/688888443 >> 172.21.15.78:0/688888443 conn(0x7f508006b700 msgr2=0x7f50800aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 -- 172.21.15.78:0/688888443 shutdown_connections 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.108+0000 7f50883c9640 1 -- 172.21.15.78:0/688888443 wait complete. 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.109+0000 7f50883c9640 1 Processor -- start 2024-02-19T14:55:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.109+0000 7f50883c9640 1 -- start start 2024-02-19T14:55:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50800c39e0 0x7f50801eb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 0x7f50801e02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5080155e00 0x7f50801dec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50801eab80 con 0x7f50800c67e0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50801e2040 con 0x7f5080155e00 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50801e2320 con 0x7f50800c39e0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 0x7f50801e02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 0x7f50801e02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58384/0 (socket says 172.21.15.78:58384) 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 -- 172.21.15.78:0/898161720 learned_addr learned my addr 172.21.15.78:0/898161720 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 -- 172.21.15.78:0/898161720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50800c39e0 msgr2=0x7f50801eb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50800c39e0 0x7f50801eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 -- 172.21.15.78:0/898161720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5080155e00 msgr2=0x7f50801dec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5080155e00 0x7f50801dec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 -- 172.21.15.78:0/898161720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0094d0 con 0x7f50800c67e0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5085964640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 0x7f50801e02d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f507000d800 tx=0x7f507000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507000b6c0 con 0x7f50800c67e0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507000b820 con 0x7f50800c67e0 2024-02-19T14:55:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507000bbb0 con 0x7f50800c67e0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50801eae40 con 0x7f50800c67e0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.110+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50801e0810 con 0x7f50800c67e0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.111+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50800c6cb0 con 0x7f50800c67e0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.112+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f507000bd10 con 0x7f50800c67e0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.112+0000 7f5076ffd640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f506406b810 0x7f506406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.112+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f507008a670 con 0x7f50800c67e0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.112+0000 7f5086165640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f506406b810 0x7f506406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.113+0000 7f5086165640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f506406b810 0x7f506406dcd0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f507c036000 tx=0x7f507c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.116+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5070055190 con 0x7f50800c67e0 2024-02-19T14:55:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.352+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7f5080074990 con 0x7f50800c67e0 2024-02-19T14:55:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.780+0000 7f5076ffd640 1 -- 172.21.15.78:0/898161720 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v197) v1 ==== 158+0+0 (secure 0 0 0) 0x7f5070058850 con 0x7f50800c67e0 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f506406b810 msgr2=0x7f506406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f506406b810 0x7f506406dcd0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f507c036000 tx=0x7f507c030000 comp rx=0 tx=0).stop 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 msgr2=0x7f50801e02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 0x7f50801e02d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f507000d800 tx=0x7f507000dcc0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 shutdown_connections 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f506406b810 0x7f506406dcd0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50800c67e0 0x7f50801e02d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5080155e00 0x7f50801dec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 --2- 172.21.15.78:0/898161720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50800c39e0 0x7f50801eb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 >> 172.21.15.78:0/898161720 conn(0x7f508006b700 msgr2=0x7f5080074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 shutdown_connections 2024-02-19T14:55:04.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.782+0000 7f50883c9640 1 -- 172.21.15.78:0/898161720 wait complete. 2024-02-19T14:55:04.796 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. ... ok 2024-02-19T14:55:04.797 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) That setting mds_join_fs at runtime affects the cluster layout. 2024-02-19T14:55:04.797 INFO:tasks.cephfs_test_runner:test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:55:04.797 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.896+0000 7f4dbc9a1640 1 Processor -- start 2024-02-19T14:55:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.896+0000 7f4dbc9a1640 1 -- start start 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4dbc9a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4db81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4dbc9a1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db8155ef0 con 0x7f4db80c7070 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4dbc9a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db806dec0 con 0x7f4db80c87a0 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4dbc9a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806d520 con 0x7f4db8152480 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db5d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4db80c7070 0x7f4db806ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36598/0 (socket says 172.21.15.78:36598) 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db5d9b640 1 -- 172.21.15.78:0/3417380444 learned_addr learned my addr 172.21.15.78:0/3417380444 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438519483 0 0) 0x7f4dac001110 con 0x7f4db80c7070 2024-02-19T14:55:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da0003520 con 0x7f4db80c7070 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db659c640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4db81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279574137 0 0) 0x7f4da0003520 con 0x7f4db80c7070 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 msgr2=0x7f4db81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4db81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 >> v1:172.21.15.150:6789/0 conn(0x7f4db80c87a0 legacy=0x7f4db80c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80c75c0 con 0x7f4db80c7070 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4dac001800 con 0x7f4db80c7070 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 >> v1:172.21.15.78:6789/0 conn(0x7f4db80c7070 legacy=0x7f4db806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da00039a0 0x7f4da0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.897+0000 7f4db559a640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4da0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db559a640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4da0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db806d520 con 0x7f4da0004350 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da000b980 con 0x7f4da00039a0 2024-02-19T14:55:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da000bb40 con 0x7f4db8152480 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4da0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db659c640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da00039a0 0x7f4da0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db6d9d640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4da0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db5d9b640 1 -- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 msgr2=0x7f4da0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4da0005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db5d9b640 1 -- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da00039a0 msgr2=0x7f4da0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da00039a0 0x7f4da0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db5d9b640 1 -- 172.21.15.78:0/3417380444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0003520 con 0x7f4da0004350 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db659c640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da00039a0 0x7f4da0003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.898+0000 7f4db6d9d640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db8152480 0x7f4da0005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4da0004820 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c210 tx=0x7f4da00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc9ca151ebe2d65 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac005ad0 con 0x7f4da0004350 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac005c30 con 0x7f4da0004350 2024-02-19T14:55:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4db559a640 1 -- 172.21.15.78:0/3417380444 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac02c870 con 0x7f4da0004350 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/3417380444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 msgr2=0x7f4da0004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/3417380444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4da0004820 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c210 tx=0x7f4da00057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4db6d9d640 1 -- 172.21.15.78:0/3417380444 reap_dead start 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/3417380444 shutdown_connections 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/3417380444 >> 172.21.15.78:0/3417380444 conn(0x7f4db80ca8a0 msgr2=0x7f4db80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/3417380444 shutdown_connections 2024-02-19T14:55:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.899+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/3417380444 wait complete. 2024-02-19T14:55:04.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 Processor -- start 2024-02-19T14:55:04.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 -- start start 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4db8156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db80be380 0x7f4db80ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db8158f90 0x7f4db80c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8157f60 con 0x7f4da0004350 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8157250 con 0x7f4db8158f90 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4dbc9a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8158d40 con 0x7f4db80be380 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4db659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4db8156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4db659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4db8156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58396/0 (socket says 172.21.15.78:58396) 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.901+0000 7f4db659c640 1 -- 172.21.15.78:0/2544359547 learned_addr learned my addr 172.21.15.78:0/2544359547 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db6d9d640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db8158f90 0x7f4db80c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db80be380 0x7f4db80ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db659c640 1 -- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db80be380 msgr2=0x7f4db80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db659c640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db80be380 0x7f4db80ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db659c640 1 -- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db8158f90 msgr2=0x7f4db80c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db659c640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db8158f90 0x7f4db80c0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db659c640 1 -- 172.21.15.78:0/2544359547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8158780 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4db659c640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4db8156d10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4da8011700 tx=0x7f4da8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da800f9c0 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da800fb20 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db8158a60 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da800fed0 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.902+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db80ac790 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.903+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db8074990 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.903+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4da8015460 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.904+0000 7f4da6ffd640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9806b840 0x7f4d9806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.904+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f4da808aa90 con 0x7f4da0004350 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.904+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9806b840 0x7f4d9806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:04.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.904+0000 7f4db5d9b640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9806b840 0x7f4d9806dd00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c210 tx=0x7f4dac009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:04.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:04.908+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4da8055520 con 0x7f4da0004350 2024-02-19T14:55:05.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.081+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db806dae0 con 0x7f4da0004350 2024-02-19T14:55:05.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.082+0000 7f4da6ffd640 1 -- 172.21.15.78:0/2544359547 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+13670 (secure 0 0 0) 0x7f4da8058be0 con 0x7f4da0004350 2024-02-19T14:55:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:05.085 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:02.773119+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6834/20921361":"2024-02-20T14:54:59.765772+0000","172.21.15.150:6832/20921361":"2024-02-20T14:54:59.765772+0000","172.21.15.78:6835/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6834/4201159351":"2024-02-20T14:54:54.615583+0000","172.21.15.78:6836/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/1047936932":"2024-02-20T14:54:53.607744+0000","172.21.15.78:6837/2137063883":"2024-02-20T14:54:42.308159+0000","172.21.15.78:6836/2137063883":"2024-02-20T14:54:42.308159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:05.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9806b840 msgr2=0x7f4d9806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9806b840 0x7f4d9806dd00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00c210 tx=0x7f4dac009030 comp rx=0 tx=0).stop 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 msgr2=0x7f4db8156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4db8156d10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4da8011700 tx=0x7f4da8011bc0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 shutdown_connections 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9806b840 0x7f4d9806dd00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db8158f90 0x7f4db80c0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db80be380 0x7f4db80ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 --2- 172.21.15.78:0/2544359547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4da0004350 0x7f4db8156d10 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 >> 172.21.15.78:0/2544359547 conn(0x7f4db80ca8a0 msgr2=0x7f4db80af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 shutdown_connections 2024-02-19T14:55:05.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.084+0000 7f4dbc9a1640 1 -- 172.21.15.78:0/2544359547 wait complete. 2024-02-19T14:55:05.097 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:55:05.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:55:05.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.198+0000 7fe893fff640 1 Processor -- start 2024-02-19T14:55:05.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.198+0000 7fe893fff640 1 -- start start 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe893fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe8940c10e0 0x7fe8940c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe893fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe894158320 0x7fe894158770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe893fff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894155800 con 0x7fe89406c4e0 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe893fff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89406d220 con 0x7fe894158320 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe893fff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe894155c90 con 0x7fe8940c10e0 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe894158320 0x7fe894158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe894158320 0x7fe894158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58410/0 (socket says 172.21.15.78:58410) 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 -- 172.21.15.78:0/357539273 learned_addr learned my addr 172.21.15.78:0/357539273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8927fc640 1 --2- 172.21.15.78:0/357539273 >> v2:172.21.15.150:3300/0 conn(0x7fe8940c10e0 0x7fe8940c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 -- 172.21.15.78:0/357539273 >> v2:172.21.15.150:3300/0 conn(0x7fe8940c10e0 msgr2=0x7fe8940c1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 --2- 172.21.15.78:0/357539273 >> v2:172.21.15.150:3300/0 conn(0x7fe8940c10e0 0x7fe8940c1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 -- 172.21.15.78:0/357539273 >> v1:172.21.15.150:6789/0 conn(0x7fe89406c4e0 legacy=0x7fe8940c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:05.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.199+0000 7fe8937fe640 1 -- 172.21.15.78:0/357539273 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894154ed0 con 0x7fe894158320 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.200+0000 7fe8937fe640 1 --2- 172.21.15.78:0/357539273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe894158320 0x7fe894158770 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe8940700b0 tx=0x7fe88402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedfbca0ea365ea4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.200+0000 7fe891ffb640 1 -- 172.21.15.78:0/357539273 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88402ca80 con 0x7fe894158320 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.200+0000 7fe891ffb640 1 -- 172.21.15.78:0/357539273 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88402cbe0 con 0x7fe894158320 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.200+0000 7fe891ffb640 1 -- 172.21.15.78:0/357539273 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8840343f0 con 0x7fe894158320 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 -- 172.21.15.78:0/357539273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe894158320 msgr2=0x7fe894158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 --2- 172.21.15.78:0/357539273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe894158320 0x7fe894158770 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe8940700b0 tx=0x7fe88402bee0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 -- 172.21.15.78:0/357539273 shutdown_connections 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 --2- 172.21.15.78:0/357539273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe894158320 0x7fe894158770 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 --2- 172.21.15.78:0/357539273 >> v2:172.21.15.150:3300/0 conn(0x7fe8940c10e0 0x7fe8940c1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 -- 172.21.15.78:0/357539273 >> 172.21.15.78:0/357539273 conn(0x7fe8940c7490 msgr2=0x7fe8940c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 -- 172.21.15.78:0/357539273 shutdown_connections 2024-02-19T14:55:05.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.201+0000 7fe893fff640 1 -- 172.21.15.78:0/357539273 wait complete. 2024-02-19T14:55:05.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.202+0000 7fe893fff640 1 Processor -- start 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.202+0000 7fe893fff640 1 -- start start 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 0x7fe8940c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8940c10e0 0x7fe8940c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe894158320 0x7fe8941eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c5bc0 con 0x7fe89406c4e0 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8941ee570 con 0x7fe894158320 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940aaf70 con 0x7fe8940c10e0 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 0x7fe8940c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 0x7fe8940c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58416/0 (socket says 172.21.15.78:58416) 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 -- 172.21.15.78:0/3757395794 learned_addr learned my addr 172.21.15.78:0/3757395794 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe8937fe640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe894158320 0x7fe8941eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe8927fc640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8940c10e0 0x7fe8940c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 -- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8940c10e0 msgr2=0x7fe8940c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8940c10e0 0x7fe8940c6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 -- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe894158320 msgr2=0x7fe8941eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe894158320 0x7fe8941eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 -- 172.21.15.78:0/3757395794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe884009b30 con 0x7fe89406c4e0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe8937fe640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe894158320 0x7fe8941eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe892ffd640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 0x7fe8940c6050 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00c400 tx=0x7fe88c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0052d0 con 0x7fe89406c4e0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c005450 con 0x7fe89406c4e0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8940aa9b0 con 0x7fe89406c4e0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c007260 con 0x7fe89406c4e0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.203+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8940c6c90 con 0x7fe89406c4e0 2024-02-19T14:55:05.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.204+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe894070780 con 0x7fe89406c4e0 2024-02-19T14:55:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.205+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe88c0073c0 con 0x7fe89406c4e0 2024-02-19T14:55:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.205+0000 7fe87b7fe640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe88006b840 0x7fe88006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.205+0000 7fe8927fc640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe88006b840 0x7fe88006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.205+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fe88c0894d0 con 0x7fe89406c4e0 2024-02-19T14:55:05.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.206+0000 7fe8927fc640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe88006b840 0x7fe88006dd00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe894154db0 tx=0x7fe888008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.210+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe88c053ff0 con 0x7fe89406c4e0 2024-02-19T14:55:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.382+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe894073360 con 0x7fe89406c4e0 2024-02-19T14:55:05.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.382+0000 7fe87b7fe640 1 -- 172.21.15.78:0/3757395794 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe88c0576b0 con 0x7fe89406c4e0 2024-02-19T14:55:05.385 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe88006b840 msgr2=0x7fe88006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe88006b840 0x7fe88006dd00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe894154db0 tx=0x7fe888008040 comp rx=0 tx=0).stop 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 msgr2=0x7fe8940c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 0x7fe8940c6050 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00c400 tx=0x7fe88c00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 shutdown_connections 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe88006b840 0x7fe88006dd00 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe894158320 0x7fe8941eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8940c10e0 0x7fe8940c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 --2- 172.21.15.78:0/3757395794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89406c4e0 0x7fe8940c6050 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 >> 172.21.15.78:0/3757395794 conn(0x7fe8940c7490 msgr2=0x7fe8940c1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:05.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 shutdown_connections 2024-02-19T14:55:05.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.384+0000 7fe893fff640 1 -- 172.21.15.78:0/3757395794 wait complete. 2024-02-19T14:55:05.398 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-02-19T14:55:05.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.495+0000 7fe30597a640 1 Processor -- start 2024-02-19T14:55:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.495+0000 7fe30597a640 1 -- start start 2024-02-19T14:55:05.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe30597a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe300152480 0x7fe3001559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe30597a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe300155ef0 con 0x7fe3000c87a0 2024-02-19T14:55:05.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe30597a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30006dec0 con 0x7fe3000c7070 2024-02-19T14:55:05.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe30597a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30006d520 con 0x7fe300152480 2024-02-19T14:55:05.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2ff7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe3000c87a0 0x7fe3000c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36600/0 (socket says 172.21.15.78:36600) 2024-02-19T14:55:05.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2ff7fe640 1 -- 172.21.15.78:0/2228749899 learned_addr learned my addr 172.21.15.78:0/2228749899 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574339078 0 0) 0x7fe2f8001490 con 0x7fe3000c87a0 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e8003520 con 0x7fe3000c87a0 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2feffd640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe300152480 0x7fe3001559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2564860577 0 0) 0x7fe2e8003520 con 0x7fe3000c87a0 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe300152480 msgr2=0x7fe3001559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe300152480 0x7fe3001559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 >> v1:172.21.15.150:6789/0 conn(0x7fe3000c7070 legacy=0x7fe30006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000c75c0 con 0x7fe3000c87a0 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.496+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe2f8001b80 con 0x7fe3000c87a0 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 >> v1:172.21.15.78:6789/0 conn(0x7fe3000c87a0 legacy=0x7fe3000c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2e80039a0 0x7fe2e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2e80047c0 0x7fe2e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e8005180 0x7fe2e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30006d520 con 0x7fe2e8005180 2024-02-19T14:55:05.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e800c050 con 0x7fe2e80039a0 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e800c210 con 0x7fe2e80047c0 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2feffd640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe300152480 0x7fe3001559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2ff7fe640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e8005180 0x7fe2e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2fe7fc640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2e80047c0 0x7fe2e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2feffd640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2e80039a0 0x7fe2e8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2ff7fe640 1 -- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2e80047c0 msgr2=0x7fe2e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2ff7fe640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2e80047c0 0x7fe2e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2ff7fe640 1 -- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2e80039a0 msgr2=0x7fe2e8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2ff7fe640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2e80039a0 0x7fe2e8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.497+0000 7fe2ff7fe640 1 -- 172.21.15.78:0/2228749899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8003520 con 0x7fe2e8005180 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe2ff7fe640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e8005180 0x7fe2e800b980 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800a7e0 tx=0x7fe2e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9178508c9e37ec6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8034630 con 0x7fe2e8005180 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f802f070 con 0x7fe2e8005180 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe2fdffb640 1 -- 172.21.15.78:0/2228749899 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80349f0 con 0x7fe2e8005180 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe30597a640 1 -- 172.21.15.78:0/2228749899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e8005180 msgr2=0x7fe2e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe30597a640 1 --2- 172.21.15.78:0/2228749899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e8005180 0x7fe2e800b980 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800a7e0 tx=0x7fe2e80057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe2ff7fe640 1 -- 172.21.15.78:0/2228749899 reap_dead start 2024-02-19T14:55:05.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe30597a640 1 -- 172.21.15.78:0/2228749899 shutdown_connections 2024-02-19T14:55:05.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe30597a640 1 -- 172.21.15.78:0/2228749899 >> 172.21.15.78:0/2228749899 conn(0x7fe3000ca8a0 msgr2=0x7fe3000cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:05.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe30597a640 1 -- 172.21.15.78:0/2228749899 shutdown_connections 2024-02-19T14:55:05.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.498+0000 7fe30597a640 1 -- 172.21.15.78:0/2228749899 wait complete. 2024-02-19T14:55:05.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.499+0000 7fe30597a640 1 Processor -- start 2024-02-19T14:55:05.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 -- start start 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe30014d020 0x7fe3001ed950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 0x7fe3001ede90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30014ac90 0x7fe3000af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300158d80 con 0x7fe3001590b0 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30014bc20 con 0x7fe30014ac90 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe30597a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000ae5e0 con 0x7fe30014d020 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe2fe7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 0x7fe3001ede90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe2fe7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 0x7fe3001ede90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58440/0 (socket says 172.21.15.78:58440) 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.500+0000 7fe2fe7fc640 1 -- 172.21.15.78:0/2656019606 learned_addr learned my addr 172.21.15.78:0/2656019606 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2feffd640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe30014d020 0x7fe3001ed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2ff7fe640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30014ac90 0x7fe3000af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2fe7fc640 1 -- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe30014d020 msgr2=0x7fe3001ed950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2fe7fc640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe30014d020 0x7fe3001ed950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2fe7fc640 1 -- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30014ac90 msgr2=0x7fe3000af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2fe7fc640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30014ac90 0x7fe3000af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2fe7fc640 1 -- 172.21.15.78:0/2656019606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000ae8c0 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2fe7fc640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 0x7fe3001ede90 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400f7f0 tx=0x7fe2f400fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f40050d0 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4005230 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400d730 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3000aeba0 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.501+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3000ae2e0 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.502+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300074990 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.502+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe2f400d890 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.503+0000 7fe2e7fff640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2ec06b810 0x7fe2ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.503+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fe2f408a010 con 0x7fe3001590b0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.503+0000 7fe2feffd640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2ec06b810 0x7fe2ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.503+0000 7fe2feffd640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2ec06b810 0x7fe2ec06dcd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0006fd0 tx=0x7fe2f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.507+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2f4054b30 con 0x7fe3001590b0 2024-02-19T14:55:05.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.743+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7fe3000ca0e0 con 0x7fe3001590b0 2024-02-19T14:55:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.781+0000 7fe2e7fff640 1 -- 172.21.15.78:0/2656019606 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v198) v1 ==== 149+0+0 (secure 0 0 0) 0x7fe2f40581f0 con 0x7fe3001590b0 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2ec06b810 msgr2=0x7fe2ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2ec06b810 0x7fe2ec06dcd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0006fd0 tx=0x7fe2f0008040 comp rx=0 tx=0).stop 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 msgr2=0x7fe3001ede90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 0x7fe3001ede90 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400f7f0 tx=0x7fe2f400fcb0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 shutdown_connections 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2ec06b810 0x7fe2ec06dcd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3001590b0 0x7fe3001ede90 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30014ac90 0x7fe3000af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 --2- 172.21.15.78:0/2656019606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe30014d020 0x7fe3001ed950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 >> 172.21.15.78:0/2656019606 conn(0x7fe3000ca8a0 msgr2=0x7fe3000c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:05.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 shutdown_connections 2024-02-19T14:55:05.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.783+0000 7fe30597a640 1 -- 172.21.15.78:0/2656019606 wait complete. 2024-02-19T14:55:05.797 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:55:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.897+0000 7f8894d25640 1 Processor -- start 2024-02-19T14:55:05.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.897+0000 7f8894d25640 1 -- start start 2024-02-19T14:55:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f8894d25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c78a0 0x7f88900c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f8894d25640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f889014ef50 0x7f889014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f8894d25640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889014f8f0 con 0x7f88900727d0 2024-02-19T14:55:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f8894d25640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88900c35c0 con 0x7f889014ef50 2024-02-19T14:55:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f8894d25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88900c38a0 con 0x7f88900c78a0 2024-02-19T14:55:05.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f888dd9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f889014ef50 0x7f889014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f888ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f88900727d0 0x7f8890072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36602/0 (socket says 172.21.15.78:36602) 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.898+0000 7f888ed9d640 1 -- 172.21.15.78:0/3231824140 learned_addr learned my addr 172.21.15.78:0/3231824140 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888dd9b640 1 -- 172.21.15.78:0/3231824140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c78a0 msgr2=0x7f88900c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888dd9b640 1 --2- 172.21.15.78:0/3231824140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c78a0 0x7f88900c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888dd9b640 1 -- 172.21.15.78:0/3231824140 >> v1:172.21.15.78:6789/0 conn(0x7f88900727d0 legacy=0x7f8890072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888d59a640 1 -- 172.21.15.78:0/3231824140 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856149846 0 0) 0x7f889014f8f0 con 0x7f88900727d0 2024-02-19T14:55:05.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888dd9b640 1 -- 172.21.15.78:0/3231824140 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900c3b80 con 0x7f889014ef50 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888dd9b640 1 --2- 172.21.15.78:0/3231824140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889014f3b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f88840097a0 tx=0x7f888402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1198867e243f01 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888d59a640 1 -- 172.21.15.78:0/3231824140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884031830 con 0x7f889014ef50 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888d59a640 1 -- 172.21.15.78:0/3231824140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8884031990 con 0x7f889014ef50 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.899+0000 7f888d59a640 1 -- 172.21.15.78:0/3231824140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884031d20 con 0x7f889014ef50 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 -- 172.21.15.78:0/3231824140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 msgr2=0x7f889014f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 --2- 172.21.15.78:0/3231824140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889014f3b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f88840097a0 tx=0x7f888402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 -- 172.21.15.78:0/3231824140 shutdown_connections 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 --2- 172.21.15.78:0/3231824140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889014f3b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 --2- 172.21.15.78:0/3231824140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c78a0 0x7f88900c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 -- 172.21.15.78:0/3231824140 >> 172.21.15.78:0/3231824140 conn(0x7f8890066e70 msgr2=0x7f88900a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:05.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 -- 172.21.15.78:0/3231824140 shutdown_connections 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.900+0000 7f8894d25640 1 -- 172.21.15.78:0/3231824140 wait complete. 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.901+0000 7f8894d25640 1 Processor -- start 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.901+0000 7f8894d25640 1 -- start start 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f8894d25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900727d0 0x7f889014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f8894d25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c78a0 0x7f88901ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f8894d25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889006b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f8894d25640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900bf690 con 0x7f889014ef50 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f8894d25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900bf970 con 0x7f88900c78a0 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f8894d25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88901ebaa0 con 0x7f88900727d0 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889006b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58450/0 (socket says 172.21.15.78:58450) 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 -- 172.21.15.78:0/424939416 learned_addr learned my addr 172.21.15.78:0/424939416 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888e59c640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900727d0 0x7f889014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888dd9b640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c78a0 0x7f88901ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 -- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900727d0 msgr2=0x7f889014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900727d0 0x7f889014b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 -- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c78a0 msgr2=0x7f88901ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c78a0 0x7f88901ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888ed9d640 1 -- 172.21.15.78:0/424939416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884009400 con 0x7f889014ef50 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.902+0000 7f888dd9b640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c78a0 0x7f88901ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.903+0000 7f888ed9d640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889006b960 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f888800c450 tx=0x7f888800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.903+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88880074e0 con 0x7f889014ef50 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.903+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8888007660 con 0x7f889014ef50 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.903+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888004a20 con 0x7f889014ef50 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.903+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88901ebd80 con 0x7f889014ef50 2024-02-19T14:55:05.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.903+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88900c02b0 con 0x7f889014ef50 2024-02-19T14:55:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.904+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88901ec2a0 con 0x7f889014ef50 2024-02-19T14:55:05.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.904+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8888004b80 con 0x7f889014ef50 2024-02-19T14:55:05.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.904+0000 7f887effd640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887006b840 0x7f887006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:05.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.905+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f88880893f0 con 0x7f889014ef50 2024-02-19T14:55:05.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.905+0000 7f888e59c640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887006b840 0x7f887006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:05.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.905+0000 7f888e59c640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887006b840 0x7f887006dd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f889006c1f0 tx=0x7f888000d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:05.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:05.909+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8888053fc0 con 0x7f889014ef50 2024-02-19T14:55:06.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.078+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8890074990 con 0x7f889014ef50 2024-02-19T14:55:06.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.081+0000 7f887effd640 1 -- 172.21.15.78:0/424939416 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v31)=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8888057680 con 0x7f889014ef50 2024-02-19T14:55:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887006b840 msgr2=0x7f887006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887006b840 0x7f887006dd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f889006c1f0 tx=0x7f888000d040 comp rx=0 tx=0).stop 2024-02-19T14:55:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 msgr2=0x7f889006b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889006b960 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f888800c450 tx=0x7f888800c910 comp rx=0 tx=0).stop 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 shutdown_connections 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887006b840 0x7f887006dd00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f889014ef50 0x7f889006b960 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c78a0 0x7f88901ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 --2- 172.21.15.78:0/424939416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900727d0 0x7f889014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 >> 172.21.15.78:0/424939416 conn(0x7f8890066e70 msgr2=0x7f889006ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 shutdown_connections 2024-02-19T14:55:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.083+0000 7f8894d25640 1 -- 172.21.15.78:0/424939416 wait complete. 2024-02-19T14:55:06.097 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:55:06.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:55:06.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:55:06.179 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:55:06.179 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:55:06.179 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:55:06.194 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:55:06.195 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:55:06.195 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:55:06.238 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:55:06.238 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:55:06.238 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.335+0000 7f8a262f4640 1 Processor -- start 2024-02-19T14:55:06.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.335+0000 7f8a262f4640 1 -- start start 2024-02-19T14:55:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a262f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8a200c2f60 0x7f8a20070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a262f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a200ae670 con 0x7f8a200add70 2024-02-19T14:55:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a262f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a200716b0 con 0x7f8a200c4980 2024-02-19T14:55:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a262f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a200c3390 con 0x7f8a200c2f60 2024-02-19T14:55:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a24891640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8a200c2f60 0x7f8a20070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a24891640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8a200c2f60 0x7f8a20070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58462/0 (socket says 172.21.15.78:58462) 2024-02-19T14:55:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a24891640 1 -- 172.21.15.78:0/2602055623 learned_addr learned my addr 172.21.15.78:0/2602055623 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.336+0000 7f8a1ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8a200add70 0x7f8a2006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36610/0 (socket says 172.21.15.78:36610) 2024-02-19T14:55:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a24891640 1 -- 172.21.15.78:0/2602055623 >> v1:172.21.15.150:6789/0 conn(0x7f8a200c4980 legacy=0x7f8a200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a24891640 1 -- 172.21.15.78:0/2602055623 >> v1:172.21.15.78:6789/0 conn(0x7f8a200add70 legacy=0x7f8a2006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a24891640 1 -- 172.21.15.78:0/2602055623 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a20153780 con 0x7f8a200c2f60 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a24891640 1 --2- 172.21.15.78:0/2602055623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20070cc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009e50 tx=0x7f8a1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d2ebe949b32195 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a1effd640 1 -- 172.21.15.78:0/2602055623 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1802ca80 con 0x7f8a200c2f60 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a1effd640 1 -- 172.21.15.78:0/2602055623 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1802cbe0 con 0x7f8a200c2f60 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.337+0000 7f8a1effd640 1 -- 172.21.15.78:0/2602055623 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a180343f0 con 0x7f8a200c2f60 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 -- 172.21.15.78:0/2602055623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 msgr2=0x7f8a20070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 --2- 172.21.15.78:0/2602055623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20070cc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009e50 tx=0x7f8a1802bee0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 -- 172.21.15.78:0/2602055623 shutdown_connections 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 --2- 172.21.15.78:0/2602055623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20070cc0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 -- 172.21.15.78:0/2602055623 >> 172.21.15.78:0/2602055623 conn(0x7f8a200c85e0 msgr2=0x7f8a200c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:06.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 -- 172.21.15.78:0/2602055623 shutdown_connections 2024-02-19T14:55:06.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.338+0000 7f8a262f4640 1 -- 172.21.15.78:0/2602055623 wait complete. 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.339+0000 7f8a262f4640 1 Processor -- start 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.339+0000 7f8a262f4640 1 -- start start 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a262f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a200add70 0x7f8a20144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a262f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a262f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a200c4980 0x7f8a20154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a262f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a20154870 con 0x7f8a200c2f60 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a262f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a201e88e0 con 0x7f8a200add70 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a262f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a201447d0 con 0x7f8a200c4980 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58466/0 (socket says 172.21.15.78:58466) 2024-02-19T14:55:06.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 -- 172.21.15.78:0/1879267810 learned_addr learned my addr 172.21.15.78:0/1879267810 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1ffff640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a200add70 0x7f8a20144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a24891640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a200c4980 0x7f8a20154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 -- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a200c4980 msgr2=0x7f8a20154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a200c4980 0x7f8a20154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 -- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a200add70 msgr2=0x7f8a20144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a200add70 0x7f8a20144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 -- 172.21.15.78:0/1879267810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18009b30 con 0x7f8a200c2f60 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.340+0000 7f8a1f7fe640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20156690 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400b3f0 tx=0x7f8a1400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.341+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14011830 con 0x7f8a200c2f60 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.341+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14011990 con 0x7f8a200c2f60 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.341+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a20153f40 con 0x7f8a200c2f60 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.341+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14011d40 con 0x7f8a200c2f60 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.341+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a20154500 con 0x7f8a200c2f60 2024-02-19T14:55:06.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.342+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a20074990 con 0x7f8a200c2f60 2024-02-19T14:55:06.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.342+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8a1401a400 con 0x7f8a200c2f60 2024-02-19T14:55:06.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.342+0000 7f8a1cff9640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a0c06b810 0x7f8a0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.342+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8a1408a0f0 con 0x7f8a200c2f60 2024-02-19T14:55:06.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.342+0000 7f8a1ffff640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a0c06b810 0x7f8a0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.343+0000 7f8a1ffff640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a0c06b810 0x7f8a0c06dcd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007fd0 tx=0x7f8a10007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:06.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.347+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8a14054cc0 con 0x7f8a200c2f60 2024-02-19T14:55:06.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.557+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a20071490 con 0x7f8a200c2f60 2024-02-19T14:55:06.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.557+0000 7f8a1cff9640 1 -- 172.21.15.78:0/1879267810 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f8a14015e90 con 0x7f8a200c2f60 2024-02-19T14:55:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:06.560 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.150:6835/1628693681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1628693681},{"type":"v1","addr":"172.21.15.150:6835","nonce":1628693681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6211,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1367972860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1367972860},{"type":"v1","addr":"172.21.15.150:6834","nonce":1367972860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6768,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3549382195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3549382195},{"type":"v1","addr":"172.21.15.78:6837","nonce":3549382195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6780,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2858964398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2858964398},{"type":"v1","addr":"172.21.15.78:6835","nonce":2858964398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[]} 2024-02-19T14:55:06.560 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 69 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a0c06b810 msgr2=0x7f8a0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a0c06b810 0x7f8a0c06dcd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8a10007fd0 tx=0x7f8a10007e80 comp rx=0 tx=0).stop 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 msgr2=0x7f8a20156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20156690 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400b3f0 tx=0x7f8a1400b8b0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 shutdown_connections 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a0c06b810 0x7f8a0c06dcd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a200c2f60 0x7f8a20156690 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a200c4980 0x7f8a20154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 --2- 172.21.15.78:0/1879267810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a200add70 0x7f8a20144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 >> 172.21.15.78:0/1879267810 conn(0x7f8a200c85e0 msgr2=0x7f8a2014eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 shutdown_connections 2024-02-19T14:55:06.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.559+0000 7f8a262f4640 1 -- 172.21.15.78:0/1879267810 wait complete. 2024-02-19T14:55:06.573 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:55:06.573 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:55:06.573 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:55:06.573 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:06.573 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:55:06.573 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:55:06.574 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:55:06.574 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:06.574 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:55:06.574 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:55:06.574 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:55:06.574 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:06.574 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:55:06.574 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:55:06.575 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:55:06.575 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:06.583 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:55:06.583 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:55:06.583 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:55:06.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:55:06.586 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:55:06.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:55:06.587 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:55:06.588 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:55:06.626 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:55:06.626 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:06.626 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:55:06.627 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:06.629 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:55:06.629 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:06.630 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:55:06.630 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:06.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:55:06.701 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:55:06.702 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:55:06.712 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:55:06.712 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:55:06.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.733+0000 7f5f55125640 1 Processor -- start 2024-02-19T14:55:06.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.733+0000 7f5f55125640 1 -- start start 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f55125640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f50152670 0x7f5f50071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f55125640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f50152a50 0x7f5f50155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f55125640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f50071bf0 con 0x7f5f500702d0 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f55125640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f50158a20 con 0x7f5f50152a50 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f55125640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f50155550 con 0x7f5f50152670 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f50152a50 0x7f5f50155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5f50152a50 0x7f5f50155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58500/0 (socket says 172.21.15.78:58500) 2024-02-19T14:55:06.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 -- 172.21.15.78:0/2191360074 learned_addr learned my addr 172.21.15.78:0/2191360074 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:06.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 -- 172.21.15.78:0/2191360074 >> v2:172.21.15.150:3300/0 conn(0x7f5f50152670 msgr2=0x7f5f50071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:06.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f4ed9d640 1 --2- 172.21.15.78:0/2191360074 >> v2:172.21.15.150:3300/0 conn(0x7f5f50152670 0x7f5f50071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 --2- 172.21.15.78:0/2191360074 >> v2:172.21.15.150:3300/0 conn(0x7f5f50152670 0x7f5f50071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 -- 172.21.15.78:0/2191360074 >> v1:172.21.15.150:6789/0 conn(0x7f5f500702d0 legacy=0x7f5f50158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:06.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.734+0000 7f5f47fff640 1 -- 172.21.15.78:0/2191360074 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f500c6780 con 0x7f5f50152a50 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.735+0000 7f5f47fff640 1 --2- 172.21.15.78:0/2191360074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50152a50 0x7f5f50155010 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5f5006e600 tx=0x7f5f3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63009b45eda32062 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.735+0000 7f5f4e59c640 1 -- 172.21.15.78:0/2191360074 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38030860 con 0x7f5f50152a50 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.735+0000 7f5f4e59c640 1 -- 172.21.15.78:0/2191360074 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f380309c0 con 0x7f5f50152a50 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.735+0000 7f5f4e59c640 1 -- 172.21.15.78:0/2191360074 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38030d50 con 0x7f5f50152a50 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 -- 172.21.15.78:0/2191360074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50152a50 msgr2=0x7f5f50155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 --2- 172.21.15.78:0/2191360074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50152a50 0x7f5f50155010 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5f5006e600 tx=0x7f5f3802b9a0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 -- 172.21.15.78:0/2191360074 shutdown_connections 2024-02-19T14:55:06.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 --2- 172.21.15.78:0/2191360074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f50152a50 0x7f5f50155010 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 --2- 172.21.15.78:0/2191360074 >> v2:172.21.15.150:3300/0 conn(0x7f5f50152670 0x7f5f50071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 -- 172.21.15.78:0/2191360074 >> 172.21.15.78:0/2191360074 conn(0x7f5f500c85e0 msgr2=0x7f5f500c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:06.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 -- 172.21.15.78:0/2191360074 shutdown_connections 2024-02-19T14:55:06.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.736+0000 7f5f55125640 1 -- 172.21.15.78:0/2191360074 wait complete. 2024-02-19T14:55:06.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.737+0000 7f5f55125640 1 Processor -- start 2024-02-19T14:55:06.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.737+0000 7f5f55125640 1 -- start start 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f55125640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 0x7f5f501eab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f55125640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f50152670 0x7f5f5014fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f55125640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f50150240 0x7f5f5006c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f55125640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5006c130 con 0x7f5f500702d0 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f55125640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f501eb150 con 0x7f5f50150240 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f55125640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f501eb430 con 0x7f5f50152670 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 0x7f5f501eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 0x7f5f501eab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58516/0 (socket says 172.21.15.78:58516) 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 -- 172.21.15.78:0/390041766 learned_addr learned my addr 172.21.15.78:0/390041766 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f47fff640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f50152670 0x7f5f5014fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 -- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f50152670 msgr2=0x7f5f5014fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4f59e640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f50150240 0x7f5f5006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f50152670 0x7f5f5014fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 -- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f50150240 msgr2=0x7f5f5006c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:06.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f50150240 0x7f5f5006c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 -- 172.21.15.78:0/390041766 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380094d0 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f47fff640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f50152670 0x7f5f5014fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4f59e640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f50150240 0x7f5f5006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.738+0000 7f5f4ed9d640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 0x7f5f501eab00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000c7f0 tx=0x7f5f4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.739+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40004220 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.739+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f400043a0 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.739+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f501eb710 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.739+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40007860 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.739+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f50150f70 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.740+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f500676a0 con 0x7f5f500702d0 2024-02-19T14:55:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.740+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5f400079c0 con 0x7f5f500702d0 2024-02-19T14:55:06.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.741+0000 7f5f477fe640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3006b810 0x7f5f3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:06.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.741+0000 7f5f47fff640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3006b810 0x7f5f3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:06.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.741+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f5f40089f40 con 0x7f5f500702d0 2024-02-19T14:55:06.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.741+0000 7f5f47fff640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3006b810 0x7f5f3006dcd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5f38036000 tx=0x7f5f3803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:06.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.745+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5f40054a60 con 0x7f5f500702d0 2024-02-19T14:55:06.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:06.932+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5f50070120 con 0x7f5f500702d0 2024-02-19T14:55:07.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.796+0000 7f5f477fe640 1 -- 172.21.15.78:0/390041766 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v94) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5f4001e080 con 0x7f5f500702d0 2024-02-19T14:55:07.799 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3006b810 msgr2=0x7f5f3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3006b810 0x7f5f3006dcd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5f38036000 tx=0x7f5f3803b040 comp rx=0 tx=0).stop 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 msgr2=0x7f5f501eab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 0x7f5f501eab00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000c7f0 tx=0x7f5f4000ccb0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 shutdown_connections 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3006b810 0x7f5f3006dcd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f50150240 0x7f5f5006c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f50152670 0x7f5f5014fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 --2- 172.21.15.78:0/390041766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f500702d0 0x7f5f501eab00 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 >> 172.21.15.78:0/390041766 conn(0x7f5f500c85e0 msgr2=0x7f5f500a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 shutdown_connections 2024-02-19T14:55:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.799+0000 7f5f55125640 1 -- 172.21.15.78:0/390041766 wait complete. 2024-02-19T14:55:07.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:55:07.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.917+0000 7fb2f05b5640 1 Processor -- start 2024-02-19T14:55:07.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.917+0000 7fb2f05b5640 1 -- start start 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2f05b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2e80c25e0 0x7fb2e81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2f05b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e80c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2f05b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e8072250 con 0x7fb2e80c49c0 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2f05b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e8152c90 con 0x7fb2e80c25e0 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2f05b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e8072480 con 0x7fb2e80c29c0 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2ee351640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2e80c25e0 0x7fb2e81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2eeb52640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb2e80c49c0 0x7fb2e806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48118/0 (socket says 172.21.15.78:48118) 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2eeb52640 1 -- 172.21.15.78:0/46993376 learned_addr learned my addr 172.21.15.78:0/46993376 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:07.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.918+0000 7fb2ee351640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2e80c25e0 0x7fb2e81526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59968/0 (socket says 172.21.15.78:59968) 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2ee351640 1 -- 172.21.15.78:0/46993376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 msgr2=0x7fb2e80c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2edb50640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e80c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2ed34f640 1 -- 172.21.15.78:0/46993376 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729386951 0 0) 0x7fb2e8072250 con 0x7fb2e80c49c0 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2ee351640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e80c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2ee351640 1 -- 172.21.15.78:0/46993376 >> v1:172.21.15.78:6789/0 conn(0x7fb2e80c49c0 legacy=0x7fb2e806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2ee351640 1 -- 172.21.15.78:0/46993376 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80c3eb0 con 0x7fb2e80c25e0 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.919+0000 7fb2edb50640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e80c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2ee351640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e81526a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000bfa0 tx=0x7fb2e002e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d1bee65b522a98 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2ed34f640 1 -- 172.21.15.78:0/46993376 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e002e9b0 con 0x7fb2e80c25e0 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2ed34f640 1 -- 172.21.15.78:0/46993376 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e002eb10 con 0x7fb2e80c25e0 2024-02-19T14:55:07.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2ed34f640 1 -- 172.21.15.78:0/46993376 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e002eea0 con 0x7fb2e80c25e0 2024-02-19T14:55:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 -- 172.21.15.78:0/46993376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 msgr2=0x7fb2e81526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e81526a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000bfa0 tx=0x7fb2e002e010 comp rx=0 tx=0).stop 2024-02-19T14:55:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 -- 172.21.15.78:0/46993376 shutdown_connections 2024-02-19T14:55:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e80c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/46993376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e81526a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 -- 172.21.15.78:0/46993376 >> 172.21.15.78:0/46993376 conn(0x7fb2e806b270 msgr2=0x7fb2e806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 -- 172.21.15.78:0/46993376 shutdown_connections 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.920+0000 7fb2f05b5640 1 -- 172.21.15.78:0/46993376 wait complete. 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.921+0000 7fb2f05b5640 1 Processor -- start 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.921+0000 7fb2f05b5640 1 -- start start 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2f05b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e8148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2f05b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e8145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2f05b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2e8145e00 0x7fb2e8145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2f05b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e8145bc0 con 0x7fb2e80c25e0 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2f05b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81e9170 con 0x7fb2e8145e00 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2f05b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81e9450 con 0x7fb2e80c29c0 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e8148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e8148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59978/0 (socket says 172.21.15.78:59978) 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 -- 172.21.15.78:0/1676151718 learned_addr learned my addr 172.21.15.78:0/1676151718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2edb50640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e8145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:07.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2eeb52640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2e8145e00 0x7fb2e8145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 -- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 msgr2=0x7fb2e8145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e8145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 -- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2e8145e00 msgr2=0x7fb2e8145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2e8145e00 0x7fb2e8145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.922+0000 7fb2ee351640 1 -- 172.21.15.78:0/1676151718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e000bc80 con 0x7fb2e80c25e0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.923+0000 7fb2ee351640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e8148740 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0038040 tx=0x7fb2e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.923+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0040660 con 0x7fb2e80c25e0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.923+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e00407e0 con 0x7fb2e80c25e0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.923+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0040b90 con 0x7fb2e80c25e0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.923+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e81e9730 con 0x7fb2e80c25e0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.923+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e8148c80 con 0x7fb2e80c25e0 2024-02-19T14:55:07.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.924+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e8074990 con 0x7fb2e80c25e0 2024-02-19T14:55:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.924+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb2e0040cf0 con 0x7fb2e80c25e0 2024-02-19T14:55:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.925+0000 7fb2deffd640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2cc06b810 0x7fb2cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.925+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb2e00af200 con 0x7fb2e80c25e0 2024-02-19T14:55:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.925+0000 7fb2edb50640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2cc06b810 0x7fb2cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:07.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.925+0000 7fb2edb50640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2cc06b810 0x7fb2cc06dcd0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8153a40 tx=0x7fb2d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:07.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:07.929+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2e0079de0 con 0x7fb2e80c25e0 2024-02-19T14:55:08.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.166+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb2e8151790 con 0x7fb2e80c25e0 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.166+0000 7fb2deffd640 1 -- 172.21.15.78:0/1676151718 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fb2e007d4a0 con 0x7fb2e80c25e0 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:08.169 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:55:08.171 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.173 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:55:08.175 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.177 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:55:08.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:55:08.180 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2cc06b810 msgr2=0x7fb2cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2cc06b810 0x7fb2cc06dcd0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8153a40 tx=0x7fb2d8008040 comp rx=0 tx=0).stop 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 msgr2=0x7fb2e8148740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e8148740 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0038040 tx=0x7fb2e0004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 shutdown_connections 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2cc06b810 0x7fb2cc06dcd0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2e8145e00 0x7fb2e8145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2e80c29c0 0x7fb2e8145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 --2- 172.21.15.78:0/1676151718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2e80c25e0 0x7fb2e8148740 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 >> 172.21.15.78:0/1676151718 conn(0x7fb2e806b270 msgr2=0x7fb2e8150930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:08.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 shutdown_connections 2024-02-19T14:55:08.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.168+0000 7fb2f05b5640 1 -- 172.21.15.78:0/1676151718 wait complete. 2024-02-19T14:55:08.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.281+0000 7f5a1586b640 1 Processor -- start 2024-02-19T14:55:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.281+0000 7f5a1586b640 1 -- start start 2024-02-19T14:55:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a1586b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a100c5560 0x7f5a100c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a1586b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100c65a0 con 0x7f5a10070720 2024-02-19T14:55:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a1586b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100c7530 con 0x7f5a100c7910 2024-02-19T14:55:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a1586b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a101537b0 con 0x7f5a100c5560 2024-02-19T14:55:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a100c5560 0x7f5a100c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5a10070720 0x7f5a10070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48128/0 (socket says 172.21.15.78:48128) 2024-02-19T14:55:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a100c5560 0x7f5a100c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59990/0 (socket says 172.21.15.78:59990) 2024-02-19T14:55:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0f7fe640 1 -- 172.21.15.78:0/1948662799 learned_addr learned my addr 172.21.15.78:0/1948662799 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0effd640 1 -- 172.21.15.78:0/1948662799 >> v1:172.21.15.150:6789/0 conn(0x7f5a100c7910 legacy=0x7f5a10154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0effd640 1 -- 172.21.15.78:0/1948662799 >> v1:172.21.15.78:6789/0 conn(0x7f5a10070720 legacy=0x7f5a10070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.282+0000 7f5a0effd640 1 -- 172.21.15.78:0/1948662799 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10153a90 con 0x7f5a100c5560 2024-02-19T14:55:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.283+0000 7f5a0effd640 1 --2- 172.21.15.78:0/1948662799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c5560 0x7f5a100c6ff0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009d90 tx=0x7f5a0002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5e1bf60785c377 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.283+0000 7f5a0dffb640 1 -- 172.21.15.78:0/1948662799 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0002cb40 con 0x7f5a100c5560 2024-02-19T14:55:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.283+0000 7f5a0dffb640 1 -- 172.21.15.78:0/1948662799 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0002cca0 con 0x7f5a100c5560 2024-02-19T14:55:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.283+0000 7f5a0dffb640 1 -- 172.21.15.78:0/1948662799 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00034660 con 0x7f5a100c5560 2024-02-19T14:55:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 -- 172.21.15.78:0/1948662799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c5560 msgr2=0x7f5a100c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1948662799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c5560 0x7f5a100c6ff0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009d90 tx=0x7f5a0002bd90 comp rx=0 tx=0).stop 2024-02-19T14:55:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 -- 172.21.15.78:0/1948662799 shutdown_connections 2024-02-19T14:55:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1948662799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c5560 0x7f5a100c6ff0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 -- 172.21.15.78:0/1948662799 >> 172.21.15.78:0/1948662799 conn(0x7f5a1006b4b0 msgr2=0x7f5a100c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 -- 172.21.15.78:0/1948662799 shutdown_connections 2024-02-19T14:55:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.284+0000 7f5a1586b640 1 -- 172.21.15.78:0/1948662799 wait complete. 2024-02-19T14:55:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.285+0000 7f5a1586b640 1 Processor -- start 2024-02-19T14:55:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.285+0000 7f5a1586b640 1 -- start start 2024-02-19T14:55:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a1586b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a10070720 0x7f5a10149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a1586b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a100c5560 0x7f5a100aa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a1586b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 0x7f5a10148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a1586b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a100a8760 con 0x7f5a100c7910 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a1586b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a100a92e0 con 0x7f5a10070720 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a1586b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1014d690 con 0x7f5a100c5560 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 0x7f5a10148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 0x7f5a10148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59992/0 (socket says 172.21.15.78:59992) 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 -- 172.21.15.78:0/1666902000 learned_addr learned my addr 172.21.15.78:0/1666902000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0effd640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a10070720 0x7f5a10149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0e7fc640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a100c5560 0x7f5a100aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 -- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a100c5560 msgr2=0x7f5a100aa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a100c5560 0x7f5a100aa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 -- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a10070720 msgr2=0x7f5a10149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a10070720 0x7f5a10149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 -- 172.21.15.78:0/1666902000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00009a70 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.286+0000 7f5a0f7fe640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 0x7f5a10148ba0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c3a0 tx=0x7f5a0800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.287+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08005ce0 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.287+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08005e40 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.287+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080072d0 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.287+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a100c5ac0 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.287+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a100a8c90 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.288+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a10074990 con 0x7f5a100c7910 2024-02-19T14:55:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.288+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5a08007430 con 0x7f5a100c7910 2024-02-19T14:55:08.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.288+0000 7f59f7fff640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59fc06b840 0x7f59fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.288+0000 7f5a0effd640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59fc06b840 0x7f59fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.288+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5a08089300 con 0x7f5a100c7910 2024-02-19T14:55:08.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.289+0000 7f5a0effd640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59fc06b840 0x7f59fc06dd00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5a00036000 tx=0x7f5a00030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.293+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5a08054020 con 0x7f5a100c7910 2024-02-19T14:55:08.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.464+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a100c76f0 con 0x7f5a100c7910 2024-02-19T14:55:08.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.465+0000 7f59f7fff640 1 -- 172.21.15.78:0/1666902000 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13153 (secure 0 0 0) 0x7f5a080576e0 con 0x7f5a100c7910 2024-02-19T14:55:08.468 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:08.468 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:07.793998+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:08.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59fc06b840 msgr2=0x7f59fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59fc06b840 0x7f59fc06dd00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5a00036000 tx=0x7f5a00030000 comp rx=0 tx=0).stop 2024-02-19T14:55:08.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 msgr2=0x7f5a10148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 0x7f5a10148ba0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800c3a0 tx=0x7f5a0800c860 comp rx=0 tx=0).stop 2024-02-19T14:55:08.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 shutdown_connections 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f59fc06b840 0x7f59fc06dd00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a100c7910 0x7f5a10148ba0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a100c5560 0x7f5a100aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 --2- 172.21.15.78:0/1666902000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a10070720 0x7f5a10149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.467+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 >> 172.21.15.78:0/1666902000 conn(0x7f5a1006b4b0 msgr2=0x7f5a100c2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.468+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 shutdown_connections 2024-02-19T14:55:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.468+0000 7f5a1586b640 1 -- 172.21.15.78:0/1666902000 wait complete. 2024-02-19T14:55:08.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:55:08.481 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:55:08.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.581+0000 7fd5629e2640 1 Processor -- start 2024-02-19T14:55:08.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.581+0000 7fd5629e2640 1 -- start start 2024-02-19T14:55:08.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd5629e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 0x7fd55c0c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd5629e2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd55c06ed20 0x7fd55c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd5629e2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c150820 con 0x7fd55c0c3800 2024-02-19T14:55:08.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd5629e2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55c0c1b60 con 0x7fd55c06ed20 2024-02-19T14:55:08.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd5629e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c1537d0 con 0x7fd55c150440 2024-02-19T14:55:08.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd55bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 0x7fd55c0c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd55b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd55c06ed20 0x7fd55c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd55bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 0x7fd55c0c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:39176/0 (socket says 172.21.15.78:39176) 2024-02-19T14:55:08.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd55b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd55c06ed20 0x7fd55c06d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:44224/0 (socket says 172.21.15.78:44224) 2024-02-19T14:55:08.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd55bfff640 1 -- 172.21.15.78:0/2644043641 learned_addr learned my addr 172.21.15.78:0/2644043641 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:08.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.582+0000 7fd560f7f640 1 --1- 172.21.15.78:0/2644043641 >> v1:172.21.15.150:6789/0 conn(0x7fd55c0c3800 0x7fd55c06d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58914/0 (socket says 172.21.15.78:58914) 2024-02-19T14:55:08.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.583+0000 7fd55bfff640 1 -- 172.21.15.78:0/2644043641 >> v2:172.21.15.150:3300/0 conn(0x7fd55c06ed20 msgr2=0x7fd55c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:08.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.583+0000 7fd55bfff640 1 --2- 172.21.15.78:0/2644043641 >> v2:172.21.15.150:3300/0 conn(0x7fd55c06ed20 0x7fd55c06d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.583+0000 7fd55bfff640 1 -- 172.21.15.78:0/2644043641 >> v1:172.21.15.150:6789/0 conn(0x7fd55c0c3800 legacy=0x7fd55c06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.583+0000 7fd55bfff640 1 -- 172.21.15.78:0/2644043641 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c153d10 con 0x7fd55c150440 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.583+0000 7fd55bfff640 1 --2- 172.21.15.78:0/2644043641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 0x7fd55c0c32c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009870 tx=0x7fd54c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4689405a9ffb5d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd55affd640 1 -- 172.21.15.78:0/2644043641 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c02f840 con 0x7fd55c150440 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd55affd640 1 -- 172.21.15.78:0/2644043641 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c02f9a0 con 0x7fd55c150440 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd55affd640 1 -- 172.21.15.78:0/2644043641 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c02fd30 con 0x7fd55c150440 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 -- 172.21.15.78:0/2644043641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 msgr2=0x7fd55c0c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2644043641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 0x7fd55c0c32c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009870 tx=0x7fd54c02ba20 comp rx=0 tx=0).stop 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 -- 172.21.15.78:0/2644043641 shutdown_connections 2024-02-19T14:55:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2644043641 >> v2:172.21.15.150:3300/0 conn(0x7fd55c06ed20 0x7fd55c06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2644043641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c150440 0x7fd55c0c32c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 -- 172.21.15.78:0/2644043641 >> 172.21.15.78:0/2644043641 conn(0x7fd55c06b270 msgr2=0x7fd55c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:08.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 -- 172.21.15.78:0/2644043641 shutdown_connections 2024-02-19T14:55:08.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.584+0000 7fd5629e2640 1 -- 172.21.15.78:0/2644043641 wait complete. 2024-02-19T14:55:08.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.585+0000 7fd5629e2640 1 Processor -- start 2024-02-19T14:55:08.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 -- start start 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c06ed20 0x7fd55c147380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 0x7fd55c1e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55c150440 0x7fd55c156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c154190 con 0x7fd55c0c3800 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c157230 con 0x7fd55c150440 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd5629e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c1e9910 con 0x7fd55c06ed20 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd55b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 0x7fd55c1e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd55b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 0x7fd55c1e93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60008/0 (socket says 172.21.15.78:60008) 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.586+0000 7fd55b7fe640 1 -- 172.21.15.78:0/2503640834 learned_addr learned my addr 172.21.15.78:0/2503640834 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55bfff640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c06ed20 0x7fd55c147380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd560f7f640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55c150440 0x7fd55c156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55b7fe640 1 -- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c06ed20 msgr2=0x7fd55c147380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55b7fe640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c06ed20 0x7fd55c147380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55b7fe640 1 -- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55c150440 msgr2=0x7fd55c156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55b7fe640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55c150440 0x7fd55c156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55b7fe640 1 -- 172.21.15.78:0/2503640834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0094d0 con 0x7fd55c0c3800 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55bfff640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c06ed20 0x7fd55c147380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd560f7f640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55c150440 0x7fd55c156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd55b7fe640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 0x7fd55c1e93d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd55000e9e0 tx=0x7fd55000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550009670 con 0x7fd55c0c3800 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55000c970 con 0x7fd55c0c3800 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c0a8ad0 con 0x7fd55c0c3800 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55000cd20 con 0x7fd55c0c3800 2024-02-19T14:55:08.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.587+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c0a9090 con 0x7fd55c0c3800 2024-02-19T14:55:08.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.588+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c074990 con 0x7fd55c0c3800 2024-02-19T14:55:08.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.588+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd550013470 con 0x7fd55c0c3800 2024-02-19T14:55:08.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.589+0000 7fd558ff9640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd54806b810 0x7fd54806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.589+0000 7fd55bfff640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd54806b810 0x7fd54806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.589+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd55008a1f0 con 0x7fd55c0c3800 2024-02-19T14:55:08.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.589+0000 7fd55bfff640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd54806b810 0x7fd54806dcd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd54c036000 tx=0x7fd54c039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.593+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd550054de0 con 0x7fd55c0c3800 2024-02-19T14:55:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.763+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd55c0c4a90 con 0x7fd55c0c3800 2024-02-19T14:55:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.763+0000 7fd558ff9640 1 -- 172.21.15.78:0/2503640834 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5500584a0 con 0x7fd55c0c3800 2024-02-19T14:55:08.765 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd54806b810 msgr2=0x7fd54806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd54806b810 0x7fd54806dcd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd54c036000 tx=0x7fd54c039040 comp rx=0 tx=0).stop 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 msgr2=0x7fd55c1e93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 0x7fd55c1e93d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd55000e9e0 tx=0x7fd55000eea0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 shutdown_connections 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd54806b810 0x7fd54806dcd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0c3800 0x7fd55c1e93d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd55c150440 0x7fd55c156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 --2- 172.21.15.78:0/2503640834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd55c06ed20 0x7fd55c147380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 >> 172.21.15.78:0/2503640834 conn(0x7fd55c06b270 msgr2=0x7fd55c14dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 shutdown_connections 2024-02-19T14:55:08.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.765+0000 7fd5629e2640 1 -- 172.21.15.78:0/2503640834 wait complete. 2024-02-19T14:55:08.778 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:55:08.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.876+0000 7faffc5bf640 1 Processor -- start 2024-02-19T14:55:08.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.876+0000 7faffc5bf640 1 -- start start 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.876+0000 7faffc5bf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faff4155f90 0x7faff406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffc5bf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faff40c6d30 0x7faff40c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffc5bf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff40c6af0 con 0x7faff40c3aa0 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffc5bf640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff40c7940 con 0x7faff4155f90 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffc5bf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff40c7c20 con 0x7faff40c6d30 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faff4155f90 0x7faff406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faff4155f90 0x7faff406d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60022/0 (socket says 172.21.15.78:60022) 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 -- 172.21.15.78:0/630944671 learned_addr learned my addr 172.21.15.78:0/630944671 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 -- 172.21.15.78:0/630944671 >> v2:172.21.15.150:3300/0 conn(0x7faff40c6d30 msgr2=0x7faff40c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 --2- 172.21.15.78:0/630944671 >> v2:172.21.15.150:3300/0 conn(0x7faff40c6d30 0x7faff40c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 -- 172.21.15.78:0/630944671 >> v1:172.21.15.150:6789/0 conn(0x7faff40c3aa0 legacy=0x7faff41588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:08.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.877+0000 7faffa35b640 1 -- 172.21.15.78:0/630944671 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40c7f00 con 0x7faff4155f90 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffa35b640 1 --2- 172.21.15.78:0/630944671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff4155f90 0x7faff406d550 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7faff40c2b60 tx=0x7faff002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c4b2e8706cb35 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faff9359640 1 -- 172.21.15.78:0/630944671 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff00317e0 con 0x7faff4155f90 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faff9359640 1 -- 172.21.15.78:0/630944671 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0031940 con 0x7faff4155f90 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faff9359640 1 -- 172.21.15.78:0/630944671 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0031cd0 con 0x7faff4155f90 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffc5bf640 1 -- 172.21.15.78:0/630944671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff4155f90 msgr2=0x7faff406d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffc5bf640 1 --2- 172.21.15.78:0/630944671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff4155f90 0x7faff406d550 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7faff40c2b60 tx=0x7faff002b880 comp rx=0 tx=0).stop 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffc5bf640 1 -- 172.21.15.78:0/630944671 shutdown_connections 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffc5bf640 1 --2- 172.21.15.78:0/630944671 >> v2:172.21.15.150:3300/0 conn(0x7faff40c6d30 0x7faff40c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffc5bf640 1 --2- 172.21.15.78:0/630944671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff4155f90 0x7faff406d550 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.878+0000 7faffc5bf640 1 -- 172.21.15.78:0/630944671 >> 172.21.15.78:0/630944671 conn(0x7faff406b700 msgr2=0x7faff40aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:08.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.879+0000 7faffc5bf640 1 -- 172.21.15.78:0/630944671 shutdown_connections 2024-02-19T14:55:08.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.879+0000 7faffc5bf640 1 -- 172.21.15.78:0/630944671 wait complete. 2024-02-19T14:55:08.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 Processor -- start 2024-02-19T14:55:08.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 -- start start 2024-02-19T14:55:08.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faff40c3aa0 0x7faff4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 0x7faff414ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faff4155f90 0x7faff414f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff41e0620 con 0x7faff40c6d30 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff41e2600 con 0x7faff40c3aa0 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffc5bf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff41e19f0 con 0x7faff4155f90 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faff9b5a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 0x7faff414ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faff9b5a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 0x7faff414ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60030/0 (socket says 172.21.15.78:60030) 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faff9b5a640 1 -- 172.21.15.78:0/1422898862 learned_addr learned my addr 172.21.15.78:0/1422898862 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffa35b640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faff40c3aa0 0x7faff4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.880+0000 7faffab5c640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faff4155f90 0x7faff414f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faff9b5a640 1 -- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faff4155f90 msgr2=0x7faff414f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faff9b5a640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faff4155f90 0x7faff414f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faff9b5a640 1 -- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faff40c3aa0 msgr2=0x7faff4070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faff9b5a640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faff40c3aa0 0x7faff4070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faff9b5a640 1 -- 172.21.15.78:0/1422898862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff00094d0 con 0x7faff40c6d30 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faffa35b640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faff40c3aa0 0x7faff4070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faff9b5a640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 0x7faff414ec20 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fafe400b4a0 tx=0x7fafe400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4010610 con 0x7faff40c6d30 2024-02-19T14:55:08.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4010770 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff41e1cd0 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4010b20 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.881+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff41e2290 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.882+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff4074990 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.882+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fafe4010c80 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.883+0000 7fafeaffd640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fafd806b810 0x7fafd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.883+0000 7faffa35b640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fafd806b810 0x7fafd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.883+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fafe4089b70 con 0x7faff40c6d30 2024-02-19T14:55:08.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.883+0000 7faffa35b640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fafd806b810 0x7fafd806dcd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7faff0036000 tx=0x7faff003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:08.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:08.887+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fafe40547e0 con 0x7faff40c6d30 2024-02-19T14:55:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.059+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7faff40c75a0 con 0x7faff40c6d30 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.059+0000 7fafeaffd640 1 -- 172.21.15.78:0/1422898862 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+65 (secure 0 0 0) 0x7fafe401e090 con 0x7faff40c6d30 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:09.062 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fafd806b810 msgr2=0x7fafd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fafd806b810 0x7fafd806dcd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7faff0036000 tx=0x7faff003d040 comp rx=0 tx=0).stop 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 msgr2=0x7faff414ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 0x7faff414ec20 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fafe400b4a0 tx=0x7fafe400b960 comp rx=0 tx=0).stop 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 shutdown_connections 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fafd806b810 0x7fafd806dcd0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faff40c6d30 0x7faff414ec20 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faff4155f90 0x7faff414f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 --2- 172.21.15.78:0/1422898862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faff40c3aa0 0x7faff4070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 >> 172.21.15.78:0/1422898862 conn(0x7faff406b700 msgr2=0x7faff40aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:09.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 shutdown_connections 2024-02-19T14:55:09.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.061+0000 7faffc5bf640 1 -- 172.21.15.78:0/1422898862 wait complete. 2024-02-19T14:55:09.074 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:55:09.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:55:09.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 Processor -- start 2024-02-19T14:55:09.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 -- start start 2024-02-19T14:55:09.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 0x7f8fd80c8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8fd81559e0 0x7f8fd80c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80c4d30 con 0x7f8fd80c8db0 2024-02-19T14:55:09.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd80c3050 con 0x7f8fd81559e0 2024-02-19T14:55:09.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.176+0000 7f8fdcf6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd80c3330 con 0x7f8fd81524e0 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 0x7f8fd80c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fcffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8fd81559e0 0x7f8fd80c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 0x7f8fd80c8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:39202/0 (socket says 172.21.15.78:39202) 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 -- 172.21.15.78:0/2949787999 learned_addr learned my addr 172.21.15.78:0/2949787999 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 -- 172.21.15.78:0/2949787999 >> v2:172.21.15.150:3300/0 conn(0x7f8fd81559e0 msgr2=0x7f8fd80c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 --2- 172.21.15.78:0/2949787999 >> v2:172.21.15.150:3300/0 conn(0x7f8fd81559e0 0x7f8fd80c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 -- 172.21.15.78:0/2949787999 >> v1:172.21.15.150:6789/0 conn(0x7f8fd80c8db0 legacy=0x7f8fd80c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.177+0000 7f8fd659c640 1 -- 172.21.15.78:0/2949787999 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd80c5e60 con 0x7f8fd81524e0 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.178+0000 7f8fd659c640 1 --2- 172.21.15.78:0/2949787999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 0x7f8fd80c8870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd4aaedd6f14b3e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.178+0000 7f8fd5d9b640 1 -- 172.21.15.78:0/2949787999 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8031640 con 0x7f8fd81524e0 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.178+0000 7f8fd5d9b640 1 -- 172.21.15.78:0/2949787999 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc80317a0 con 0x7f8fd81524e0 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.178+0000 7f8fd5d9b640 1 -- 172.21.15.78:0/2949787999 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8031b30 con 0x7f8fd81524e0 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/2949787999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 msgr2=0x7f8fd80c8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/2949787999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 0x7f8fd80c8870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009870 tx=0x7f8fc802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/2949787999 shutdown_connections 2024-02-19T14:55:09.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/2949787999 >> v2:172.21.15.150:3300/0 conn(0x7f8fd81559e0 0x7f8fd80c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/2949787999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81524e0 0x7f8fd80c8870 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/2949787999 >> 172.21.15.78:0/2949787999 conn(0x7f8fd806b060 msgr2=0x7f8fd806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:09.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/2949787999 shutdown_connections 2024-02-19T14:55:09.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.179+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/2949787999 wait complete. 2024-02-19T14:55:09.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.180+0000 7f8fdcf6c640 1 Processor -- start 2024-02-19T14:55:09.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.180+0000 7f8fdcf6c640 1 -- start start 2024-02-19T14:55:09.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fdcf6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 0x7f8fd80ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fdcf6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fd81524e0 0x7f8fd8143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fdcf6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81559e0 0x7f8fd81512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fdcf6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd80ab640 con 0x7f8fd80c8db0 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fdcf6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd81ee1e0 con 0x7f8fd81524e0 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fdcf6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd8151b30 con 0x7f8fd81559e0 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 0x7f8fd80ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 0x7f8fd80ad0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60042/0 (socket says 172.21.15.78:60042) 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 -- 172.21.15.78:0/1842123121 learned_addr learned my addr 172.21.15.78:0/1842123121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd6d9d640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81559e0 0x7f8fd81512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fcffff640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fd81524e0 0x7f8fd8143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 -- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81559e0 msgr2=0x7f8fd81512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81559e0 0x7f8fd81512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 -- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fd81524e0 msgr2=0x7f8fd8143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fd81524e0 0x7f8fd8143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 -- 172.21.15.78:0/1842123121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc80094d0 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.181+0000 7f8fd659c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 0x7f8fd80ad0c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8000c00 tx=0x7f8fc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.182+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc803a930 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.182+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc803aa90 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.182+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd81ed8b0 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.182+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc803ae40 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.182+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd81ede70 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.183+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd8074990 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.183+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8fc802f050 con 0x7f8fd80c8db0 2024-02-19T14:55:09.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.183+0000 7f8fceffd640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fb806b810 0x7f8fb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.183+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8fc80abc50 con 0x7f8fd80c8db0 2024-02-19T14:55:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.183+0000 7f8fcffff640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fb806b810 0x7f8fb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.184+0000 7f8fcffff640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fb806b810 0x7f8fb806dcd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8fd806dc00 tx=0x7f8fc0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:09.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.188+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8fc80b1020 con 0x7f8fd80c8db0 2024-02-19T14:55:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.359+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8fd80c2770 con 0x7f8fd80c8db0 2024-02-19T14:55:09.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.808+0000 7f8fceffd640 1 -- 172.21.15.78:0/1842123121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8fc80350b0 con 0x7f8fd80c8db0 2024-02-19T14:55:09.811 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:55:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fb806b810 msgr2=0x7f8fb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fb806b810 0x7f8fb806dcd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8fd806dc00 tx=0x7f8fc0009190 comp rx=0 tx=0).stop 2024-02-19T14:55:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 msgr2=0x7f8fd80ad0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 0x7f8fd80ad0c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8000c00 tx=0x7f8fc80046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 shutdown_connections 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fb806b810 0x7f8fb806dcd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8fd81559e0 0x7f8fd81512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8fd81524e0 0x7f8fd8143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 --2- 172.21.15.78:0/1842123121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8fd80c8db0 0x7f8fd80ad0c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.810+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 >> 172.21.15.78:0/1842123121 conn(0x7f8fd806b060 msgr2=0x7f8fd814a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.811+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 shutdown_connections 2024-02-19T14:55:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.811+0000 7f8fdcf6c640 1 -- 172.21.15.78:0/1842123121 wait complete. 2024-02-19T14:55:09.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:55:09.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 Processor -- start 2024-02-19T14:55:09.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 -- start start 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 0x7f83940c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83940c8ee0 0x7f83940c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394152a60 con 0x7f8394152680 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83940aa050 con 0x7f83940c8ee0 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.934+0000 7f839a155640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83941557e0 con 0x7f83940ace90 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 0x7f83940c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f8393fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83940c8ee0 0x7f83940c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 0x7f83940c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:39218/0 (socket says 172.21.15.78:39218) 2024-02-19T14:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 -- 172.21.15.78:0/1801317140 learned_addr learned my addr 172.21.15.78:0/1801317140 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f83937fe640 1 --1- 172.21.15.78:0/1801317140 >> v1:172.21.15.150:6789/0 conn(0x7f8394152680 0x7f8394155000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58928/0 (socket says 172.21.15.78:58928) 2024-02-19T14:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f8392ffd640 1 -- 172.21.15.78:0/1801317140 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830230230 0 0) 0x7f8394152a60 con 0x7f8394152680 2024-02-19T14:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f8392ffd640 1 -- 172.21.15.78:0/1801317140 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8380003520 con 0x7f8394152680 2024-02-19T14:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 -- 172.21.15.78:0/1801317140 >> v2:172.21.15.150:3300/0 conn(0x7f83940c8ee0 msgr2=0x7f83940c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 --2- 172.21.15.78:0/1801317140 >> v2:172.21.15.150:3300/0 conn(0x7f83940c8ee0 0x7f83940c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 -- 172.21.15.78:0/1801317140 >> v1:172.21.15.150:6789/0 conn(0x7f8394152680 legacy=0x7f8394155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f838bfff640 1 -- 172.21.15.78:0/1801317140 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394155ac0 con 0x7f83940ace90 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.935+0000 7f8393fff640 1 --2- 172.21.15.78:0/1801317140 >> v2:172.21.15.150:3300/0 conn(0x7f83940c8ee0 0x7f83940c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.936+0000 7f838bfff640 1 --2- 172.21.15.78:0/1801317140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 0x7f83940c89a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f837c009870 tx=0x7f837c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb5aa22cc966a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.936+0000 7f8392ffd640 1 -- 172.21.15.78:0/1801317140 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c031640 con 0x7f83940ace90 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.936+0000 7f8392ffd640 1 -- 172.21.15.78:0/1801317140 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c0317a0 con 0x7f83940ace90 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.936+0000 7f8392ffd640 1 -- 172.21.15.78:0/1801317140 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c031b10 con 0x7f83940ace90 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 -- 172.21.15.78:0/1801317140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 msgr2=0x7f83940c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 --2- 172.21.15.78:0/1801317140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 0x7f83940c89a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f837c009870 tx=0x7f837c02b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 -- 172.21.15.78:0/1801317140 shutdown_connections 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 --2- 172.21.15.78:0/1801317140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83940ace90 0x7f83940c89a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 --2- 172.21.15.78:0/1801317140 >> v2:172.21.15.150:3300/0 conn(0x7f83940c8ee0 0x7f83940c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 -- 172.21.15.78:0/1801317140 >> 172.21.15.78:0/1801317140 conn(0x7f83940ac370 msgr2=0x7f83940ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 -- 172.21.15.78:0/1801317140 shutdown_connections 2024-02-19T14:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.937+0000 7f839a155640 1 -- 172.21.15.78:0/1801317140 wait complete. 2024-02-19T14:55:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.938+0000 7f839a155640 1 Processor -- start 2024-02-19T14:55:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.938+0000 7f839a155640 1 -- start start 2024-02-19T14:55:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.938+0000 7f839a155640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 0x7f83940c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.938+0000 7f839a155640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83940c8ee0 0x7f8394151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f839a155640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8394152680 0x7f8394148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f839a155640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83941eb070 con 0x7f83940ace90 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f839a155640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8394154140 con 0x7f83940c8ee0 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f839a155640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83940c4bf0 con 0x7f8394152680 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 0x7f83940c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 0x7f83940c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60048/0 (socket says 172.21.15.78:60048) 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 -- 172.21.15.78:0/2261981653 learned_addr learned my addr 172.21.15.78:0/2261981653 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f8393fff640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8394152680 0x7f8394148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f838bfff640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83940c8ee0 0x7f8394151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 -- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8394152680 msgr2=0x7f8394148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8394152680 0x7f8394148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 -- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83940c8ee0 msgr2=0x7f8394151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83940c8ee0 0x7f8394151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 -- 172.21.15.78:0/2261981653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c009a70 con 0x7f83940ace90 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f83937fe640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 0x7f83940c46b0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f838400c3c0 tx=0x7f838400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:09.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840074e0 con 0x7f83940ace90 2024-02-19T14:55:09.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8384007660 con 0x7f83940ace90 2024-02-19T14:55:09.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c0094d0 con 0x7f83940ace90 2024-02-19T14:55:09.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384004a20 con 0x7f83940ace90 2024-02-19T14:55:09.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.939+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83941ea740 con 0x7f83940ace90 2024-02-19T14:55:09.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.940+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8394074990 con 0x7f83940ace90 2024-02-19T14:55:09.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.941+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8384004b80 con 0x7f83940ace90 2024-02-19T14:55:09.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.941+0000 7f8390ff9640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837406b810 0x7f837406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:09.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.941+0000 7f838bfff640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837406b810 0x7f837406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:09.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.941+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f83840899a0 con 0x7f83940ace90 2024-02-19T14:55:09.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.942+0000 7f838bfff640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837406b810 0x7f837406dcd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f837c036000 tx=0x7f837c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:09.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:09.946+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8384054610 con 0x7f83940ace90 2024-02-19T14:55:10.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.118+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f83940c8780 con 0x7f83940ace90 2024-02-19T14:55:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.814+0000 7f8390ff9640 1 -- 172.21.15.78:0/2261981653 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8384057cd0 con 0x7f83940ace90 2024-02-19T14:55:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837406b810 msgr2=0x7f837406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837406b810 0x7f837406dcd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f837c036000 tx=0x7f837c03c040 comp rx=0 tx=0).stop 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 msgr2=0x7f83940c46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 0x7f83940c46b0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f838400c3c0 tx=0x7f838400c880 comp rx=0 tx=0).stop 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 shutdown_connections 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f837406b810 0x7f837406dcd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8394152680 0x7f8394148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83940c8ee0 0x7f8394151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 --2- 172.21.15.78:0/2261981653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83940ace90 0x7f83940c46b0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 >> 172.21.15.78:0/2261981653 conn(0x7f83940ac370 msgr2=0x7f839406d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 shutdown_connections 2024-02-19T14:55:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.816+0000 7f839a155640 1 -- 172.21.15.78:0/2261981653 wait complete. 2024-02-19T14:55:10.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:55:10.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.942+0000 7f09b5688640 1 Processor -- start 2024-02-19T14:55:10.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.942+0000 7f09b5688640 1 -- start start 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.942+0000 7f09b5688640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 0x7f09b00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.942+0000 7f09b5688640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09b00c5730 0x7f09b006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.942+0000 7f09b5688640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b01551f0 con 0x7f09b00c8e00 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09b5688640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b0155490 con 0x7f09b00c5730 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09b5688640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0155770 con 0x7f09b0152700 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 0x7f09b00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09a7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09b00c5730 0x7f09b006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 0x7f09b00c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:39244/0 (socket says 172.21.15.78:39244) 2024-02-19T14:55:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 -- 172.21.15.78:0/3177416646 learned_addr learned my addr 172.21.15.78:0/3177416646 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 -- 172.21.15.78:0/3177416646 >> v2:172.21.15.150:3300/0 conn(0x7f09b00c5730 msgr2=0x7f09b006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 --2- 172.21.15.78:0/3177416646 >> v2:172.21.15.150:3300/0 conn(0x7f09b00c5730 0x7f09b006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 -- 172.21.15.78:0/3177416646 >> v1:172.21.15.150:6789/0 conn(0x7f09b00c8e00 legacy=0x7f09b0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.943+0000 7f09aeffd640 1 -- 172.21.15.78:0/3177416646 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0155a50 con 0x7f09b0152700 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.944+0000 7f09aeffd640 1 --2- 172.21.15.78:0/3177416646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 0x7f09b00c51f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009870 tx=0x7f09a002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dac3c4eb6a5edf76 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.944+0000 7f09ae7fc640 1 -- 172.21.15.78:0/3177416646 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0030840 con 0x7f09b0152700 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.944+0000 7f09ae7fc640 1 -- 172.21.15.78:0/3177416646 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a00309a0 con 0x7f09b0152700 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.944+0000 7f09ae7fc640 1 -- 172.21.15.78:0/3177416646 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0030d30 con 0x7f09b0152700 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 -- 172.21.15.78:0/3177416646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 msgr2=0x7f09b00c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 --2- 172.21.15.78:0/3177416646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 0x7f09b00c51f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009870 tx=0x7f09a002b980 comp rx=0 tx=0).stop 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 -- 172.21.15.78:0/3177416646 shutdown_connections 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 --2- 172.21.15.78:0/3177416646 >> v2:172.21.15.150:3300/0 conn(0x7f09b00c5730 0x7f09b006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 --2- 172.21.15.78:0/3177416646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b0152700 0x7f09b00c51f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 -- 172.21.15.78:0/3177416646 >> 172.21.15.78:0/3177416646 conn(0x7f09b00cac90 msgr2=0x7f09b00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 -- 172.21.15.78:0/3177416646 shutdown_connections 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.945+0000 7f09b5688640 1 -- 172.21.15.78:0/3177416646 wait complete. 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.946+0000 7f09b5688640 1 Processor -- start 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.946+0000 7f09b5688640 1 -- start start 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09b00c5730 0x7f09b00c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b00c8e00 0x7f09b00c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 0x7f09b0148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00ac320 con 0x7f09b0152700 2024-02-19T14:55:10.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b01eb7d0 con 0x7f09b00c5730 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0149400 con 0x7f09b00c8e00 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 0x7f09b0148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 0x7f09b0148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60058/0 (socket says 172.21.15.78:60058) 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 -- 172.21.15.78:0/2179824553 learned_addr learned my addr 172.21.15.78:0/2179824553 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b00c8e00 0x7f09b00c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09aeffd640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09b00c5730 0x7f09b00c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 -- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b00c8e00 msgr2=0x7f09b00c20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b00c8e00 0x7f09b00c20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 -- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09b00c5730 msgr2=0x7f09b00c1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09b00c5730 0x7f09b00c1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 -- 172.21.15.78:0/2179824553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00094d0 con 0x7f09b0152700 2024-02-19T14:55:10.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09af7fe640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 0x7f09b0148610 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f09a800c7a0 tx=0x7f09a800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8003b70 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8003cf0 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b00acfe0 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800aa90 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.947+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0159330 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.948+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b01595b0 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.949+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f09a800abf0 con 0x7f09b0152700 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.949+0000 7f09a77fe640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f099006b810 0x7f099006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:10.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.949+0000 7f09aeffd640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f099006b810 0x7f099006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:10.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.949+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f09a8089fd0 con 0x7f09b0152700 2024-02-19T14:55:10.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.950+0000 7f09aeffd640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f099006b810 0x7f099006dcd0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f09a0036000 tx=0x7f09a0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:10.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:10.954+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f09a8054980 con 0x7f09b0152700 2024-02-19T14:55:11.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.161+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f09b00ad220 con 0x7f09b0152700 2024-02-19T14:55:11.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.815+0000 7f09a77fe640 1 -- 172.21.15.78:0/2179824553 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f09b00ad220 con 0x7f09b0152700 2024-02-19T14:55:11.818 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 15 and data pool 16 2024-02-19T14:55:11.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.825+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f099006b810 msgr2=0x7f099006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:11.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.825+0000 7f09b5688640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f099006b810 0x7f099006dcd0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f09a0036000 tx=0x7f09a0008030 comp rx=0 tx=0).stop 2024-02-19T14:55:11.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.825+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 msgr2=0x7f09b0148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:11.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.825+0000 7f09b5688640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 0x7f09b0148610 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f09a800c7a0 tx=0x7f09a800cc60 comp rx=0 tx=0).stop 2024-02-19T14:55:11.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.828+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 shutdown_connections 2024-02-19T14:55:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.829+0000 7f09b5688640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f099006b810 0x7f099006dcd0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.829+0000 7f09b5688640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09b0152700 0x7f09b0148610 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.829+0000 7f09b5688640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09b00c8e00 0x7f09b00c20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.829+0000 7f09b5688640 1 --2- 172.21.15.78:0/2179824553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09b00c5730 0x7f09b00c1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.829+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 >> 172.21.15.78:0/2179824553 conn(0x7f09b00cac90 msgr2=0x7f09b00cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:11.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.833+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 shutdown_connections 2024-02-19T14:55:11.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.834+0000 7f09b5688640 1 -- 172.21.15.78:0/2179824553 wait complete. 2024-02-19T14:55:11.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:11.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 Processor -- start 2024-02-19T14:55:11.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 -- start start 2024-02-19T14:55:11.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f837014cd50 0x7f83700c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83701537a0 0x7f83700c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83700c7b70 0x7f8370156860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83701563b0 con 0x7f83700c7b70 2024-02-19T14:55:11.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8370153cd0 con 0x7f837014cd50 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f837684e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837006b6e0 con 0x7f83701537a0 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.959+0000 7f8374deb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83700c7b70 0x7f8370156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83700c7b70 0x7f8370156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60070/0 (socket says 172.21.15.78:60070) 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 -- 172.21.15.78:0/2142923197 learned_addr learned my addr 172.21.15.78:0/2142923197 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f836f7fe640 1 --2- 172.21.15.78:0/2142923197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83701537a0 0x7f83700c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f836ffff640 1 --2- 172.21.15.78:0/2142923197 >> v2:172.21.15.150:3300/0 conn(0x7f837014cd50 0x7f83700c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 -- 172.21.15.78:0/2142923197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83701537a0 msgr2=0x7f83700c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 --2- 172.21.15.78:0/2142923197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83701537a0 0x7f83700c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 -- 172.21.15.78:0/2142923197 >> v2:172.21.15.150:3300/0 conn(0x7f837014cd50 msgr2=0x7f83700c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 --2- 172.21.15.78:0/2142923197 >> v2:172.21.15.150:3300/0 conn(0x7f837014cd50 0x7f83700c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.960+0000 7f8374deb640 1 -- 172.21.15.78:0/2142923197 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837006b980 con 0x7f83700c7b70 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f8374deb640 1 --2- 172.21.15.78:0/2142923197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83700c7b70 0x7f8370156860 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f8368009e50 tx=0x7f836802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312c5a29e8f643e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f836effd640 1 -- 172.21.15.78:0/2142923197 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836802c950 con 0x7f83700c7b70 2024-02-19T14:55:11.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f836effd640 1 -- 172.21.15.78:0/2142923197 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836802cab0 con 0x7f83700c7b70 2024-02-19T14:55:11.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f836effd640 1 -- 172.21.15.78:0/2142923197 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836802ce40 con 0x7f83700c7b70 2024-02-19T14:55:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 -- 172.21.15.78:0/2142923197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83700c7b70 msgr2=0x7f8370156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:11.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 --2- 172.21.15.78:0/2142923197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83700c7b70 0x7f8370156860 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f8368009e50 tx=0x7f836802bdb0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 -- 172.21.15.78:0/2142923197 shutdown_connections 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 --2- 172.21.15.78:0/2142923197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83700c7b70 0x7f8370156860 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 --2- 172.21.15.78:0/2142923197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83701537a0 0x7f83700c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 --2- 172.21.15.78:0/2142923197 >> v2:172.21.15.150:3300/0 conn(0x7f837014cd50 0x7f83700c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 -- 172.21.15.78:0/2142923197 >> 172.21.15.78:0/2142923197 conn(0x7f8370066e70 msgr2=0x7f83700a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 -- 172.21.15.78:0/2142923197 shutdown_connections 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.961+0000 7f837684e640 1 -- 172.21.15.78:0/2142923197 wait complete. 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.962+0000 7f837684e640 1 Processor -- start 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 -- start start 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83700c7b70 0x7f83701e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f837014cd50 0x7f83701dfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 0x7f837014fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837014da00 con 0x7f83701537a0 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837014e1d0 con 0x7f83700c7b70 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f837684e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83701dd110 con 0x7f837014cd50 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f8374deb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 0x7f837014fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f8374deb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 0x7f837014fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60080/0 (socket says 172.21.15.78:60080) 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.963+0000 7f8374deb640 1 -- 172.21.15.78:0/3047619479 learned_addr learned my addr 172.21.15.78:0/3047619479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f836f7fe640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f837014cd50 0x7f83701dfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f836ffff640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83700c7b70 0x7f83701e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f8374deb640 1 -- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f837014cd50 msgr2=0x7f83701dfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f8374deb640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f837014cd50 0x7f83701dfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f8374deb640 1 -- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83700c7b70 msgr2=0x7f83701e95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f8374deb640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83700c7b70 0x7f83701e95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f8374deb640 1 -- 172.21.15.78:0/3047619479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8368009b30 con 0x7f83701537a0 2024-02-19T14:55:11.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f8374deb640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 0x7f837014fd90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f8368009e50 tx=0x7f8368004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8368009510 con 0x7f83701537a0 2024-02-19T14:55:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836803e450 con 0x7f83701537a0 2024-02-19T14:55:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83701e9b20 con 0x7f83701537a0 2024-02-19T14:55:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836803e800 con 0x7f83701537a0 2024-02-19T14:55:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.964+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837014def0 con 0x7f83701537a0 2024-02-19T14:55:11.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.965+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83700c2f90 con 0x7f83701537a0 2024-02-19T14:55:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.965+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f836803e960 con 0x7f83701537a0 2024-02-19T14:55:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.966+0000 7f836cff9640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f835c06b840 0x7f835c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.966+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f83680ad050 con 0x7f83701537a0 2024-02-19T14:55:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.966+0000 7f836ffff640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f835c06b840 0x7f835c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:11.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.966+0000 7f836ffff640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f835c06b840 0x7f835c06dd00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8364006fd0 tx=0x7f8364008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:11.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:11.970+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f83680778f0 con 0x7f83701537a0 2024-02-19T14:55:12.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.144+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8370072b20 con 0x7f83701537a0 2024-02-19T14:55:12.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.145+0000 7f836cff9640 1 -- 172.21.15.78:0/3047619479 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16874 (secure 0 0 0) 0x7f836807afb0 con 0x7f83701537a0 2024-02-19T14:55:12.148 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:12.149 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":97,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:11.163204+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:09.377887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:10.136982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f835c06b840 msgr2=0x7f835c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f835c06b840 0x7f835c06dd00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8364006fd0 tx=0x7f8364008040 comp rx=0 tx=0).stop 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 msgr2=0x7f837014fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 0x7f837014fd90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f8368009e50 tx=0x7f8368004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 shutdown_connections 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f835c06b840 0x7f835c06dd00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83701537a0 0x7f837014fd90 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f837014cd50 0x7f83701dfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 --2- 172.21.15.78:0/3047619479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83700c7b70 0x7f83701e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 >> 172.21.15.78:0/3047619479 conn(0x7f8370066e70 msgr2=0x7f837006f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:12.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 shutdown_connections 2024-02-19T14:55:12.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.148+0000 7f837684e640 1 -- 172.21.15.78:0/3047619479 wait complete. 2024-02-19T14:55:12.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:12.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.258+0000 7f21ddbe2640 1 Processor -- start 2024-02-19T14:55:12.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.258+0000 7f21ddbe2640 1 -- start start 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21ddbe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f21d8071e50 0x7f21d80c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21ddbe2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80c1120 con 0x7f21d80719d0 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21ddbe2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80c6080 con 0x7f21d814ece0 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21ddbe2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80722a0 con 0x7f21d8071e50 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f21d8071e50 0x7f21d80c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f21d8071e50 0x7f21d80c1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60096/0 (socket says 172.21.15.78:60096) 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f21d80719d0 0x7f21d80c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48144/0 (socket says 172.21.15.78:48144) 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d7fff640 1 -- 172.21.15.78:0/2915984253 learned_addr learned my addr 172.21.15.78:0/2915984253 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d7fff640 1 -- 172.21.15.78:0/2915984253 >> v1:172.21.15.150:6789/0 conn(0x7f21d814ece0 legacy=0x7f21d80c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d7fff640 1 -- 172.21.15.78:0/2915984253 >> v1:172.21.15.78:6789/0 conn(0x7f21d80719d0 legacy=0x7f21d80c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d67fc640 1 -- 172.21.15.78:0/2915984253 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958327120 0 0) 0x7f21d80c1120 con 0x7f21d80719d0 2024-02-19T14:55:12.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.259+0000 7f21d7fff640 1 -- 172.21.15.78:0/2915984253 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80c57e0 con 0x7f21d8071e50 2024-02-19T14:55:12.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.260+0000 7f21d7fff640 1 --2- 172.21.15.78:0/2915984253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d8071e50 0x7f21d80c1cc0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009e50 tx=0x7f21d002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb3c824c16490d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.260+0000 7f21d67fc640 1 -- 172.21.15.78:0/2915984253 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d002cbb0 con 0x7f21d8071e50 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.260+0000 7f21d67fc640 1 -- 172.21.15.78:0/2915984253 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d002cd10 con 0x7f21d8071e50 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.260+0000 7f21d67fc640 1 -- 172.21.15.78:0/2915984253 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0034610 con 0x7f21d8071e50 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 -- 172.21.15.78:0/2915984253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d8071e50 msgr2=0x7f21d80c1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/2915984253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d8071e50 0x7f21d80c1cc0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009e50 tx=0x7f21d002c010 comp rx=0 tx=0).stop 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 -- 172.21.15.78:0/2915984253 shutdown_connections 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/2915984253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d8071e50 0x7f21d80c1cc0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 -- 172.21.15.78:0/2915984253 >> 172.21.15.78:0/2915984253 conn(0x7f21d806b270 msgr2=0x7f21d806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 -- 172.21.15.78:0/2915984253 shutdown_connections 2024-02-19T14:55:12.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.261+0000 7f21ddbe2640 1 -- 172.21.15.78:0/2915984253 wait complete. 2024-02-19T14:55:12.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.262+0000 7f21ddbe2640 1 Processor -- start 2024-02-19T14:55:12.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.262+0000 7f21ddbe2640 1 -- start start 2024-02-19T14:55:12.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d80719d0 0x7f21d814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21d8071e50 0x7f21d8145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 0x7f21d8145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d81dcdc0 con 0x7f21d814ece0 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d81e8d90 con 0x7f21d80719d0 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d81462a0 con 0x7f21d8071e50 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 0x7f21d8145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 0x7f21d8145d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60108/0 (socket says 172.21.15.78:60108) 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 -- 172.21.15.78:0/1582464219 learned_addr learned my addr 172.21.15.78:0/1582464219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d6ffd640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21d8071e50 0x7f21d8145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 -- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21d8071e50 msgr2=0x7f21d8145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d77fe640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d80719d0 0x7f21d814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21d8071e50 0x7f21d8145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 -- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d80719d0 msgr2=0x7f21d814c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d80719d0 0x7f21d814c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 -- 172.21.15.78:0/1582464219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0009b30 con 0x7f21d814ece0 2024-02-19T14:55:12.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21d7fff640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 0x7f21d8145d60 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009e50 tx=0x7f21d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d003e450 con 0x7f21d814ece0 2024-02-19T14:55:12.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d003e5d0 con 0x7f21d814ece0 2024-02-19T14:55:12.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.263+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d81e81a0 con 0x7f21d814ece0 2024-02-19T14:55:12.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.264+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d003e980 con 0x7f21d814ece0 2024-02-19T14:55:12.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.264+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d81e8740 con 0x7f21d814ece0 2024-02-19T14:55:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.264+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d81e8f40 con 0x7f21d814ece0 2024-02-19T14:55:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.265+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f21d003eae0 con 0x7f21d814ece0 2024-02-19T14:55:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.265+0000 7f21bffff640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b840 0x7f21c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.265+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f21d00ace90 con 0x7f21d814ece0 2024-02-19T14:55:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.265+0000 7f21d77fe640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b840 0x7f21c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.266+0000 7f21d77fe640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b840 0x7f21c406dd00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000f60 tx=0x7f21c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.270+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f21d00777b0 con 0x7f21d814ece0 2024-02-19T14:55:12.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.441+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21d8074990 con 0x7f21d814ece0 2024-02-19T14:55:12.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.442+0000 7f21bffff640 1 -- 172.21.15.78:0/1582464219 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16874 (secure 0 0 0) 0x7f21d00a4e60 con 0x7f21d814ece0 2024-02-19T14:55:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:12.445 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":97,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:11.163204+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:09.377887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:10.136982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b840 msgr2=0x7f21c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b840 0x7f21c406dd00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000f60 tx=0x7f21c8009040 comp rx=0 tx=0).stop 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 msgr2=0x7f21d8145d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 0x7f21d8145d60 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009e50 tx=0x7f21d0004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 shutdown_connections 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f21c406b840 0x7f21c406dd00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21d814ece0 0x7f21d8145d60 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21d8071e50 0x7f21d8145820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 --2- 172.21.15.78:0/1582464219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21d80719d0 0x7f21d814c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 >> 172.21.15.78:0/1582464219 conn(0x7f21d806b270 msgr2=0x7f21d806e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:12.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 shutdown_connections 2024-02-19T14:55:12.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.444+0000 7f21ddbe2640 1 -- 172.21.15.78:0/1582464219 wait complete. 2024-02-19T14:55:12.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:55:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.558+0000 7f78432ff640 1 Processor -- start 2024-02-19T14:55:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f78432ff640 1 -- start start 2024-02-19T14:55:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f78432ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f783c0c4980 0x7f783c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f78432ff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f783c0705b0 0x7f783c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f78432ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c0ae670 con 0x7f783c0add70 2024-02-19T14:55:12.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f78432ff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f783c0716b0 con 0x7f783c0705b0 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f78432ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f783c153780 con 0x7f783c0c4980 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f784189c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f783c0705b0 0x7f783c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f784189c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f783c0705b0 0x7f783c0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60112/0 (socket says 172.21.15.78:60112) 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.559+0000 7f784189c640 1 -- 172.21.15.78:0/2093189720 learned_addr learned my addr 172.21.15.78:0/2093189720 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.560+0000 7f784089a640 1 --2- 172.21.15.78:0/2093189720 >> v2:172.21.15.150:3300/0 conn(0x7f783c0c4980 0x7f783c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.560+0000 7f784189c640 1 -- 172.21.15.78:0/2093189720 >> v2:172.21.15.150:3300/0 conn(0x7f783c0c4980 msgr2=0x7f783c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.560+0000 7f784189c640 1 --2- 172.21.15.78:0/2093189720 >> v2:172.21.15.150:3300/0 conn(0x7f783c0c4980 0x7f783c0c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.560+0000 7f784189c640 1 -- 172.21.15.78:0/2093189720 >> v1:172.21.15.150:6789/0 conn(0x7f783c0add70 legacy=0x7f783c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:12.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.560+0000 7f784189c640 1 -- 172.21.15.78:0/2093189720 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c153a60 con 0x7f783c0705b0 2024-02-19T14:55:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.560+0000 7f784189c640 1 --2- 172.21.15.78:0/2093189720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0705b0 0x7f783c0709e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f783c073050 tx=0x7f783802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e8dbe0fdd510c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f7833fff640 1 -- 172.21.15.78:0/2093189720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783802c9f0 con 0x7f783c0705b0 2024-02-19T14:55:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f7833fff640 1 -- 172.21.15.78:0/2093189720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783802cb50 con 0x7f783c0705b0 2024-02-19T14:55:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f7833fff640 1 -- 172.21.15.78:0/2093189720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380343f0 con 0x7f783c0705b0 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 -- 172.21.15.78:0/2093189720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0705b0 msgr2=0x7f783c0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 --2- 172.21.15.78:0/2093189720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0705b0 0x7f783c0709e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f783c073050 tx=0x7f783802be50 comp rx=0 tx=0).stop 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 -- 172.21.15.78:0/2093189720 shutdown_connections 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 --2- 172.21.15.78:0/2093189720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0705b0 0x7f783c0709e0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 --2- 172.21.15.78:0/2093189720 >> v2:172.21.15.150:3300/0 conn(0x7f783c0c4980 0x7f783c0c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 -- 172.21.15.78:0/2093189720 >> 172.21.15.78:0/2093189720 conn(0x7f783c0c85e0 msgr2=0x7f783c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 -- 172.21.15.78:0/2093189720 shutdown_connections 2024-02-19T14:55:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.561+0000 7f78432ff640 1 -- 172.21.15.78:0/2093189720 wait complete. 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.562+0000 7f78432ff640 1 Processor -- start 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.562+0000 7f78432ff640 1 -- start start 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f78432ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783c0705b0 0x7f783c0c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f78432ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f783c0add70 0x7f783c0c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f78432ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 0x7f783c0bf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f78432ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c0c8d10 con 0x7f783c0c4980 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f78432ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c0bd4e0 con 0x7f783c0705b0 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f78432ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783c155970 con 0x7f783c0add70 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 0x7f783c0bf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 0x7f783c0bf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60126/0 (socket says 172.21.15.78:60126) 2024-02-19T14:55:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 -- 172.21.15.78:0/660026078 learned_addr learned my addr 172.21.15.78:0/660026078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784089a640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f783c0add70 0x7f783c0c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784109b640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783c0705b0 0x7f783c0c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 -- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f783c0add70 msgr2=0x7f783c0c1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f783c0add70 0x7f783c0c1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 -- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783c0705b0 msgr2=0x7f783c0c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783c0705b0 0x7f783c0c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 -- 172.21.15.78:0/660026078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838009b30 con 0x7f783c0c4980 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.563+0000 7f784189c640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 0x7f783c0bf610 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7838009f40 tx=0x7f7838004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.564+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783803e450 con 0x7f783c0c4980 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.564+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783803e5d0 con 0x7f783c0c4980 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.564+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783c1e9060 con 0x7f783c0c4980 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.564+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783803e980 con 0x7f783c0c4980 2024-02-19T14:55:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.564+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783c152e20 con 0x7f783c0c4980 2024-02-19T14:55:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.565+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783c0676a0 con 0x7f783c0c4980 2024-02-19T14:55:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.565+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f783803eae0 con 0x7f783c0c4980 2024-02-19T14:55:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.566+0000 7f7831ffb640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782006b840 0x7f782006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.566+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78380ad480 con 0x7f783c0c4980 2024-02-19T14:55:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.566+0000 7f784109b640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782006b840 0x7f782006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.566+0000 7f784109b640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782006b840 0x7f782006dd00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7834004ea0 tx=0x7f7834009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.570+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7838077ec0 con 0x7f783c0c4980 2024-02-19T14:55:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.777+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f783c06f680 con 0x7f783c0c4980 2024-02-19T14:55:12.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.831+0000 7f7831ffb640 1 -- 172.21.15.78:0/660026078 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f783807b580 con 0x7f783c0c4980 2024-02-19T14:55:12.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782006b840 msgr2=0x7f782006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782006b840 0x7f782006dd00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7834004ea0 tx=0x7f7834009040 comp rx=0 tx=0).stop 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 msgr2=0x7f783c0bf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 0x7f783c0bf610 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7838009f40 tx=0x7f7838004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 shutdown_connections 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f782006b840 0x7f782006dd00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f783c0c4980 0x7f783c0bf610 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f783c0add70 0x7f783c0c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 --2- 172.21.15.78:0/660026078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783c0705b0 0x7f783c0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 >> 172.21.15.78:0/660026078 conn(0x7f783c0c85e0 msgr2=0x7f783c14eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 shutdown_connections 2024-02-19T14:55:12.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.833+0000 7f78432ff640 1 -- 172.21.15.78:0/660026078 wait complete. 2024-02-19T14:55:12.848 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:55:12.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:55:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.950+0000 7f32ce639640 1 Processor -- start 2024-02-19T14:55:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.950+0000 7f32ce639640 1 -- start start 2024-02-19T14:55:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32ce639640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32c80c78a0 0x7f32c80c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32ce639640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c8072ee0 0x7f32c814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32ce639640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80c35c0 con 0x7f32c814ef50 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32ce639640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c814f920 con 0x7f32c80c78a0 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32ce639640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80c38a0 con 0x7f32c8072ee0 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32c7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32c80c78a0 0x7f32c80c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32c7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32c80c78a0 0x7f32c80c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60132/0 (socket says 172.21.15.78:60132) 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.951+0000 7f32c7fff640 1 -- 172.21.15.78:0/1894190676 learned_addr learned my addr 172.21.15.78:0/1894190676 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32ccbd6640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c8072ee0 0x7f32c814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32c7fff640 1 -- 172.21.15.78:0/1894190676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c8072ee0 msgr2=0x7f32c814f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32c7fff640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c8072ee0 0x7f32c814f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32c7fff640 1 -- 172.21.15.78:0/1894190676 >> v1:172.21.15.150:6789/0 conn(0x7f32c814ef50 legacy=0x7f32c80727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32c7fff640 1 -- 172.21.15.78:0/1894190676 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c80c3b80 con 0x7f32c80c78a0 2024-02-19T14:55:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32ccbd6640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c8072ee0 0x7f32c814f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.952+0000 7f32c7fff640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c80c78a0 0x7f32c80c44b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009870 tx=0x7f32b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720ecbbf0a8b5f73 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32c6ffd640 1 -- 172.21.15.78:0/1894190676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8031640 con 0x7f32c80c78a0 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32c6ffd640 1 -- 172.21.15.78:0/1894190676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b80317a0 con 0x7f32c80c78a0 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32c6ffd640 1 -- 172.21.15.78:0/1894190676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8031b30 con 0x7f32c80c78a0 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 -- 172.21.15.78:0/1894190676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c80c78a0 msgr2=0x7f32c80c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c80c78a0 0x7f32c80c44b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009870 tx=0x7f32b802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 -- 172.21.15.78:0/1894190676 shutdown_connections 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c8072ee0 0x7f32c814f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 --2- 172.21.15.78:0/1894190676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c80c78a0 0x7f32c80c44b0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 -- 172.21.15.78:0/1894190676 >> 172.21.15.78:0/1894190676 conn(0x7f32c8066e70 msgr2=0x7f32c80a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 -- 172.21.15.78:0/1894190676 shutdown_connections 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.953+0000 7f32ce639640 1 -- 172.21.15.78:0/1894190676 wait complete. 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.954+0000 7f32ce639640 1 Processor -- start 2024-02-19T14:55:12.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.954+0000 7f32ce639640 1 -- start start 2024-02-19T14:55:12.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ce639640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 0x7f32c80ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ce639640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32c80c78a0 0x7f32c8148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ce639640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c814ef50 0x7f32c80c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ce639640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80aa750 con 0x7f32c8072ee0 2024-02-19T14:55:12.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ce639640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80ab6e0 con 0x7f32c80c78a0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ce639640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c8151f90 con 0x7f32c814ef50 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 0x7f32c80ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 0x7f32c80ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60138/0 (socket says 172.21.15.78:60138) 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 -- 172.21.15.78:0/3658727939 learned_addr learned my addr 172.21.15.78:0/3658727939 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c77fe640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32c80c78a0 0x7f32c8148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32ccbd6640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c814ef50 0x7f32c80c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 -- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c814ef50 msgr2=0x7f32c80c0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c814ef50 0x7f32c80c0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 -- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32c80c78a0 msgr2=0x7f32c8148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32c80c78a0 0x7f32c8148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c7fff640 1 -- 172.21.15.78:0/3658727939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b80094d0 con 0x7f32c8072ee0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.955+0000 7f32c77fe640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32c80c78a0 0x7f32c8148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.956+0000 7f32c7fff640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 0x7f32c80ab1a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009870 tx=0x7f32b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.956+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b803ac60 con 0x7f32c8072ee0 2024-02-19T14:55:12.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.956+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b803adc0 con 0x7f32c8072ee0 2024-02-19T14:55:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.956+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8044630 con 0x7f32c8072ee0 2024-02-19T14:55:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.956+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c81eb150 con 0x7f32c8072ee0 2024-02-19T14:55:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.956+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c80aa9a0 con 0x7f32c8072ee0 2024-02-19T14:55:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.957+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c8074990 con 0x7f32c8072ee0 2024-02-19T14:55:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.957+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f32b802f050 con 0x7f32c8072ee0 2024-02-19T14:55:12.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.958+0000 7f32c4ff9640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32b406b810 0x7f32b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:12.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.958+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32b80ac1d0 con 0x7f32c8072ee0 2024-02-19T14:55:12.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.958+0000 7f32c77fe640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32b406b810 0x7f32b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:12.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.958+0000 7f32c77fe640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32b406b810 0x7f32b406dcd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f32bc006fd0 tx=0x7f32bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:12.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:12.962+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f32b8076af0 con 0x7f32c8072ee0 2024-02-19T14:55:13.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.178+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f32c81487d0 con 0x7f32c8072ee0 2024-02-19T14:55:13.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.838+0000 7f32c4ff9640 1 -- 172.21.15.78:0/3658727939 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32b8031e60 con 0x7f32c8072ee0 2024-02-19T14:55:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32b406b810 msgr2=0x7f32b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32b406b810 0x7f32b406dcd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f32bc006fd0 tx=0x7f32bc008040 comp rx=0 tx=0).stop 2024-02-19T14:55:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 msgr2=0x7f32c80ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 0x7f32c80ab1a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009870 tx=0x7f32b80046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 shutdown_connections 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f32b406b810 0x7f32b406dcd0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f32c814ef50 0x7f32c80c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32c80c78a0 0x7f32c8148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 --2- 172.21.15.78:0/3658727939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32c8072ee0 0x7f32c80ab1a0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 >> 172.21.15.78:0/3658727939 conn(0x7f32c8066e70 msgr2=0x7f32c806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 shutdown_connections 2024-02-19T14:55:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.840+0000 7f32ce639640 1 -- 172.21.15.78:0/3658727939 wait complete. 2024-02-19T14:55:13.853 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:55:13.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.952+0000 7fda6bf8a640 1 Processor -- start 2024-02-19T14:55:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.952+0000 7fda6bf8a640 1 -- start start 2024-02-19T14:55:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda6bf8a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda640c6d30 0x7fda64158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda6bf8a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda640c6af0 con 0x7fda6406d550 2024-02-19T14:55:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda6bf8a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda640c7940 con 0x7fda64155f90 2024-02-19T14:55:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda6bf8a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda640c7c20 con 0x7fda640c6d30 2024-02-19T14:55:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda69525640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fda6406d550 0x7fda641588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48158/0 (socket says 172.21.15.78:48158) 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda69525640 1 -- 172.21.15.78:0/2153041490 learned_addr learned my addr 172.21.15.78:0/2153041490 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda6a527640 1 --2- 172.21.15.78:0/2153041490 >> v2:172.21.15.150:3300/0 conn(0x7fda640c6d30 0x7fda64158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148582622 0 0) 0x7fda640c6af0 con 0x7fda6406d550 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda50003520 con 0x7fda6406d550 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452611309 0 0) 0x7fda50003520 con 0x7fda6406d550 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 >> v2:172.21.15.150:3300/0 conn(0x7fda640c6d30 msgr2=0x7fda64158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 --2- 172.21.15.78:0/2153041490 >> v2:172.21.15.150:3300/0 conn(0x7fda640c6d30 0x7fda64158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 >> v1:172.21.15.150:6789/0 conn(0x7fda64155f90 legacy=0x7fda640c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640c7f00 con 0x7fda6406d550 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fda54001670 con 0x7fda6406d550 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 >> v1:172.21.15.78:6789/0 conn(0x7fda6406d550 legacy=0x7fda641588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda50003970 0x7fda50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda500047c0 0x7fda50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda5000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640c7c20 con 0x7fda50005180 2024-02-19T14:55:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5000c120 con 0x7fda500047c0 2024-02-19T14:55:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.953+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5000c2e0 con 0x7fda50003970 2024-02-19T14:55:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda5000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda69d26640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda50003970 0x7fda50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda69525640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda500047c0 0x7fda50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 -- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda50003970 msgr2=0x7fda50003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda50003970 0x7fda50003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 -- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda500047c0 msgr2=0x7fda50004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda500047c0 0x7fda50004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 -- 172.21.15.78:0/2153041490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda50003520 con 0x7fda50005180 2024-02-19T14:55:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda69525640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda500047c0 0x7fda50004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.954+0000 7fda6a527640 1 -- 172.21.15.78:0/2153041490 reap_dead start 2024-02-19T14:55:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6a527640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda5000b980 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fda5c009fc0 tx=0x7fda500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10dd37ad136ff1f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c02cc60 con 0x7fda50005180 2024-02-19T14:55:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c00f040 con 0x7fda50005180 2024-02-19T14:55:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda68d24640 1 -- 172.21.15.78:0/2153041490 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c003990 con 0x7fda50005180 2024-02-19T14:55:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 -- 172.21.15.78:0/2153041490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 msgr2=0x7fda5000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda5000b980 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fda5c009fc0 tx=0x7fda500057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 -- 172.21.15.78:0/2153041490 shutdown_connections 2024-02-19T14:55:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/2153041490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda5000b980 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 -- 172.21.15.78:0/2153041490 >> 172.21.15.78:0/2153041490 conn(0x7fda6406b700 msgr2=0x7fda640aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 -- 172.21.15.78:0/2153041490 shutdown_connections 2024-02-19T14:55:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.955+0000 7fda6bf8a640 1 -- 172.21.15.78:0/2153041490 wait complete. 2024-02-19T14:55:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.956+0000 7fda6bf8a640 1 Processor -- start 2024-02-19T14:55:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.956+0000 7fda6bf8a640 1 -- start start 2024-02-19T14:55:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6bf8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda640a8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6bf8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda640710f0 0x7fda6414e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6bf8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda64071530 0x7fda6414eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6bf8a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640a7ac0 con 0x7fda50005180 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6bf8a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64157b70 con 0x7fda64071530 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6bf8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6414f2e0 con 0x7fda640710f0 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda69d26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda640a8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda69d26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda640a8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60168/0 (socket says 172.21.15.78:60168) 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda69d26640 1 -- 172.21.15.78:0/3138519768 learned_addr learned my addr 172.21.15.78:0/3138519768 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda69525640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda640710f0 0x7fda6414e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.957+0000 7fda6a527640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda64071530 0x7fda6414eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda69d26640 1 -- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda640710f0 msgr2=0x7fda6414e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda69d26640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda640710f0 0x7fda6414e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda69d26640 1 -- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda64071530 msgr2=0x7fda6414eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda69d26640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda64071530 0x7fda6414eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda69d26640 1 -- 172.21.15.78:0/3138519768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640aa5e0 con 0x7fda50005180 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda69d26640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda640a8040 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fda6000f5f0 tx=0x7fda6000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda600052e0 con 0x7fda50005180 2024-02-19T14:55:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda60005440 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60003810 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda640a7420 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.958+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda64070310 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.959+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda64074990 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.959+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fda60003970 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.960+0000 7fda5a7fc640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4806b810 0x7fda4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.960+0000 7fda69525640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4806b810 0x7fda4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.960+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fda6008a150 con 0x7fda50005180 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.960+0000 7fda69525640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4806b810 0x7fda4806dcd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fda54001200 tx=0x7fda54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:13.964+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda60054a70 con 0x7fda50005180 2024-02-19T14:55:14.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.177+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda640c75a0 con 0x7fda50005180 2024-02-19T14:55:14.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.178+0000 7fda5a7fc640 1 -- 172.21.15.78:0/3138519768 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4940 (secure 0 0 0) 0x7fda60058130 con 0x7fda50005180 2024-02-19T14:55:14.181 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:14.181 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5672,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:13.179911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6836/3610724432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3610724432},{"type":"v1","addr":"172.21.15.78:6836","nonce":3610724432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:14.181 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 73 2024-02-19T14:55:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4806b810 msgr2=0x7fda4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4806b810 0x7fda4806dcd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fda54001200 tx=0x7fda54009040 comp rx=0 tx=0).stop 2024-02-19T14:55:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 msgr2=0x7fda640a8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda640a8040 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fda6000f5f0 tx=0x7fda6000fab0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 shutdown_connections 2024-02-19T14:55:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda4806b810 0x7fda4806dcd0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda64071530 0x7fda6414eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda640710f0 0x7fda6414e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 --2- 172.21.15.78:0/3138519768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda50005180 0x7fda640a8040 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 >> 172.21.15.78:0/3138519768 conn(0x7fda6406b700 msgr2=0x7fda6406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 shutdown_connections 2024-02-19T14:55:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.180+0000 7fda6bf8a640 1 -- 172.21.15.78:0/3138519768 wait complete. 2024-02-19T14:55:14.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.291+0000 7fe9e59d0640 1 Processor -- start 2024-02-19T14:55:14.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.291+0000 7fe9e59d0640 1 -- start start 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9e59d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9e00c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9e59d0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e006db90 con 0x7fe9e006d040 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9e59d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e01550d0 con 0x7fe9e0155bb0 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9e59d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0154ec0 con 0x7fe9e00c7b70 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9df7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe9e006d040 0x7fe9e015f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48162/0 (socket says 172.21.15.78:48162) 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9df7fe640 1 -- 172.21.15.78:0/2724905375 learned_addr learned my addr 172.21.15.78:0/2724905375 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9deffd640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9e00c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659365104 0 0) 0x7fe9e006db90 con 0x7fe9e006d040 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.292+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c8003520 con 0x7fe9e006d040 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588171733 0 0) 0x7fe9e01550d0 con 0x7fe9e0155bb0 2024-02-19T14:55:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e006db90 con 0x7fe9e0155bb0 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740605416 0 0) 0x7fe9c8003520 con 0x7fe9e006d040 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 msgr2=0x7fe9e00c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9e00c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 >> v1:172.21.15.150:6789/0 conn(0x7fe9e0155bb0 legacy=0x7fe9e01584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e01553b0 con 0x7fe9e006d040 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe9d80019f0 con 0x7fe9e006d040 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9deffd640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9e00c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 >> v1:172.21.15.78:6789/0 conn(0x7fe9e006d040 legacy=0x7fe9e015f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9c8003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c8004070 0x7fe9c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c8004d00 0x7fe9c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0154ec0 con 0x7fe9c8004070 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c800bac0 con 0x7fe9c8004d00 2024-02-19T14:55:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c800bc80 con 0x7fe9e00c7b70 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c8004070 0x7fe9c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9deffd640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9c8003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9df7fe640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c8004d00 0x7fe9c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9de7fc640 1 -- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 msgr2=0x7fe9c8003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e00c7b70 0x7fe9c8003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9de7fc640 1 -- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c8004d00 msgr2=0x7fe9c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9c8004d00 0x7fe9c8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.293+0000 7fe9de7fc640 1 -- 172.21.15.78:0/2724905375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0009400 con 0x7fe9c8004070 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.294+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c8004070 0x7fe9c80047c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40018d0 tx=0x7fe9c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ba08e17c5340ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.294+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d40052d0 con 0x7fe9c8004070 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.294+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d4005430 con 0x7fe9c8004070 2024-02-19T14:55:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.294+0000 7fe9ddffb640 1 -- 172.21.15.78:0/2724905375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4003810 con 0x7fe9c8004070 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9e59d0640 1 -- 172.21.15.78:0/2724905375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c8004070 msgr2=0x7fe9c80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/2724905375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9c8004070 0x7fe9c80047c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40018d0 tx=0x7fe9c80057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9df7fe640 1 -- 172.21.15.78:0/2724905375 reap_dead start 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9e59d0640 1 -- 172.21.15.78:0/2724905375 shutdown_connections 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9e59d0640 1 -- 172.21.15.78:0/2724905375 >> 172.21.15.78:0/2724905375 conn(0x7fe9e00c5050 msgr2=0x7fe9e00c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9e59d0640 1 -- 172.21.15.78:0/2724905375 shutdown_connections 2024-02-19T14:55:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.295+0000 7fe9e59d0640 1 -- 172.21.15.78:0/2724905375 wait complete. 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.296+0000 7fe9e59d0640 1 Processor -- start 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.296+0000 7fe9e59d0640 1 -- start start 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9c8004070 0x7fe9e00c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 0x7fe9e00ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e009a3c0 0x7fe9e00ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01ee140 con 0x7fe9e00ab930 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01ede60 con 0x7fe9e009a3c0 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00ac680 con 0x7fe9c8004070 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 0x7fe9e00ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 0x7fe9e00ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60194/0 (socket says 172.21.15.78:60194) 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 -- 172.21.15.78:0/1156668159 learned_addr learned my addr 172.21.15.78:0/1156668159 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9df7fe640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e009a3c0 0x7fe9e00ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9deffd640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9c8004070 0x7fe9e00c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 -- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9c8004070 msgr2=0x7fe9e00c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9c8004070 0x7fe9e00c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 -- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e009a3c0 msgr2=0x7fe9e00ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e009a3c0 0x7fe9e00ad6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 -- 172.21.15.78:0/1156668159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00ac960 con 0x7fe9e00ab930 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9de7fc640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 0x7fe9e00ad160 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4035040 tx=0x7fe9d40012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4006ea0 con 0x7fe9e00ab930 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d4007450 con 0x7fe9e00ab930 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d40035f0 con 0x7fe9e00ab930 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.297+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e00acc40 con 0x7fe9e00ab930 2024-02-19T14:55:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.298+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e0151810 con 0x7fe9e00ab930 2024-02-19T14:55:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.298+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e0071c60 con 0x7fe9e00ab930 2024-02-19T14:55:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.299+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe9d4003750 con 0x7fe9e00ab930 2024-02-19T14:55:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.300+0000 7fe9c7fff640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9cc06b810 0x7fe9cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.300+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe9d40ac190 con 0x7fe9e00ab930 2024-02-19T14:55:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.300+0000 7fe9deffd640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9cc06b810 0x7fe9cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.300+0000 7fe9deffd640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9cc06b810 0x7fe9cc06dcd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009c60 tx=0x7fe9d0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.307+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9d4076ad0 con 0x7fe9e00ab930 2024-02-19T14:55:14.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.480+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9e00acdd0 con 0x7fe9e00ab930 2024-02-19T14:55:14.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.482+0000 7fe9c7fff640 1 -- 172.21.15.78:0/1156668159 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16861 (secure 0 0 0) 0x7fe9d407a190 con 0x7fe9e00ab930 2024-02-19T14:55:14.485 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:14.485 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":98,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:12.827114+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:09.377887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:10.136982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9cc06b810 msgr2=0x7fe9cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9cc06b810 0x7fe9cc06dcd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009c60 tx=0x7fe9d0009190 comp rx=0 tx=0).stop 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 msgr2=0x7fe9e00ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 0x7fe9e00ad160 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4035040 tx=0x7fe9d40012d0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 shutdown_connections 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9cc06b810 0x7fe9cc06dcd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e00ab930 0x7fe9e00ad160 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e009a3c0 0x7fe9e00ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 --2- 172.21.15.78:0/1156668159 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9c8004070 0x7fe9e00c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 >> 172.21.15.78:0/1156668159 conn(0x7fe9e00c5050 msgr2=0x7fe9e014e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 shutdown_connections 2024-02-19T14:55:14.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.485+0000 7fe9e59d0640 1 -- 172.21.15.78:0/1156668159 wait complete. 2024-02-19T14:55:14.499 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:55:14.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:14.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.597+0000 7f49bfbc7640 1 Processor -- start 2024-02-19T14:55:14.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.597+0000 7f49bfbc7640 1 -- start start 2024-02-19T14:55:14.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.598+0000 7f49bfbc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.598+0000 7f49bfbc7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49b80c3d90 0x7f49b815d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.598+0000 7f49bfbc7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b806ff90 con 0x7f49b80719e0 2024-02-19T14:55:14.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.598+0000 7f49bfbc7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b8070270 con 0x7f49b80c3d90 2024-02-19T14:55:14.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.598+0000 7f49bfbc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b80c3970 con 0x7f49b814d210 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49b80c3d90 0x7f49b815d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49b80c3d90 0x7f49b815d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60210/0 (socket says 172.21.15.78:60210) 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 -- 172.21.15.78:0/2090013830 learned_addr learned my addr 172.21.15.78:0/2090013830 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49bd162640 1 --1- 172.21.15.78:0/2090013830 >> v1:172.21.15.78:6789/0 conn(0x7f49b80719e0 0x7f49b80c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48164/0 (socket says 172.21.15.78:48164) 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49bd963640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 -- 172.21.15.78:0/2090013830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 msgr2=0x7f49b80c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 -- 172.21.15.78:0/2090013830 >> v1:172.21.15.78:6789/0 conn(0x7f49b80719e0 legacy=0x7f49b80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49bc961640 1 -- 172.21.15.78:0/2090013830 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236435617 0 0) 0x7f49b806ff90 con 0x7f49b80719e0 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49be164640 1 -- 172.21.15.78:0/2090013830 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b8156fc0 con 0x7f49b80c3d90 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.599+0000 7f49bd963640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49be164640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80c3d90 0x7f49b815d7f0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009e50 tx=0x7f49b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d148fa23d71c2192 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bc961640 1 -- 172.21.15.78:0/2090013830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b002ca70 con 0x7f49b80c3d90 2024-02-19T14:55:14.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bc961640 1 -- 172.21.15.78:0/2090013830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b002cbd0 con 0x7f49b80c3d90 2024-02-19T14:55:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bc961640 1 -- 172.21.15.78:0/2090013830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b00343f0 con 0x7f49b80c3d90 2024-02-19T14:55:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 -- 172.21.15.78:0/2090013830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80c3d90 msgr2=0x7f49b815d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80c3d90 0x7f49b815d7f0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009e50 tx=0x7f49b002bed0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 -- 172.21.15.78:0/2090013830 shutdown_connections 2024-02-19T14:55:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80c3d90 0x7f49b815d7f0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/2090013830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 -- 172.21.15.78:0/2090013830 >> 172.21.15.78:0/2090013830 conn(0x7f49b80c5050 msgr2=0x7f49b80c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.600+0000 7f49bfbc7640 1 -- 172.21.15.78:0/2090013830 shutdown_connections 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.601+0000 7f49bfbc7640 1 -- 172.21.15.78:0/2090013830 wait complete. 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.601+0000 7f49bfbc7640 1 Processor -- start 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 -- start start 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 0x7f49b81ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b80c3d90 0x7f49b80c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b80adc60 con 0x7f49b80719e0 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b80ae200 con 0x7f49b80c3d90 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bfbc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b80c2500 con 0x7f49b814d210 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bd963640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 0x7f49b81ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bd963640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 0x7f49b81ea5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60218/0 (socket says 172.21.15.78:60218) 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bd963640 1 -- 172.21.15.78:0/1880322662 learned_addr learned my addr 172.21.15.78:0/1880322662 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bd963640 1 -- 172.21.15.78:0/1880322662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 msgr2=0x7f49b80abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bd963640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.602+0000 7f49bd963640 1 -- 172.21.15.78:0/1880322662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b80c3d90 msgr2=0x7f49b80c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49bd963640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b80c3d90 0x7f49b80c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49bd963640 1 -- 172.21.15.78:0/1880322662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0009b30 con 0x7f49b80719e0 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49bd963640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 0x7f49b81ea5f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00c7f0 tx=0x7f49ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac007820 con 0x7f49b80719e0 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ac007980 con 0x7f49b80719e0 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b81ee610 con 0x7f49b80719e0 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac007cf0 con 0x7f49b80719e0 2024-02-19T14:55:14.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.603+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b80ade90 con 0x7f49b80719e0 2024-02-19T14:55:14.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.604+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f49ac0113f0 con 0x7f49b80719e0 2024-02-19T14:55:14.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.604+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b8074990 con 0x7f49b80719e0 2024-02-19T14:55:14.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.604+0000 7f49aa7fc640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499c06b840 0x7f499c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.605+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49ac08a710 con 0x7f49b80719e0 2024-02-19T14:55:14.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.605+0000 7f49bd162640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499c06b840 0x7f499c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.605+0000 7f49bd162640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499c06b840 0x7f499c06dd00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f49b4007fd0 tx=0x7f49b4007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.609+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f49ac054f10 con 0x7f49b80719e0 2024-02-19T14:55:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.778+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49b80ae340 con 0x7f499c06b840 2024-02-19T14:55:14.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.785+0000 7f49aa7fc640 1 -- 172.21.15.78:0/1880322662 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289451 (secure 0 0 0) 0x7f49b40070d0 con 0x7f499c06b840 2024-02-19T14:55:14.791 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499c06b840 msgr2=0x7f499c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499c06b840 0x7f499c06dd00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f49b4007fd0 tx=0x7f49b4007920 comp rx=0 tx=0).stop 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 msgr2=0x7f49b81ea5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 0x7f49b81ea5f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00c7f0 tx=0x7f49ac00ccb0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 shutdown_connections 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499c06b840 0x7f499c06dd00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49b814d210 0x7f49b80abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49b80c3d90 0x7f49b80c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 --2- 172.21.15.78:0/1880322662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49b80719e0 0x7f49b81ea5f0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 >> 172.21.15.78:0/1880322662 conn(0x7f49b80c5050 msgr2=0x7f49b814c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 shutdown_connections 2024-02-19T14:55:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.789+0000 7f49bfbc7640 1 -- 172.21.15.78:0/1880322662 wait complete. 2024-02-19T14:55:14.802 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-02-19T14:55:14.097667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":247668,"kb_used_data":10692,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754727052,"statfs":{"total":773094113280,"available":772840501248,"internally_reserved":0,"allocated":10948608,"data_stored":5904808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.287754"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844130+0000","last_change":"2024-02-19T14:55:11.838771+0000","last_active":"2024-02-19T14:55:12.844130+0000","last_peered":"2024-02-19T14:55:12.844130+0000","last_clean":"2024-02-19T14:55:12.844130+0000","last_became_active":"2024-02-19T14:55:11.838434+0000","last_became_peered":"2024-02-19T14:55:11.838434+0000","last_unstale":"2024-02-19T14:55:12.844130+0000","last_undegraded":"2024-02-19T14:55:12.844130+0000","last_fullsized":"2024-02-19T14:55:12.844130+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:00.750078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846615+0000","last_change":"2024-02-19T14:55:11.840091+0000","last_active":"2024-02-19T14:55:12.846615+0000","last_peered":"2024-02-19T14:55:12.846615+0000","last_clean":"2024-02-19T14:55:12.846615+0000","last_became_active":"2024-02-19T14:55:11.839802+0000","last_became_peered":"2024-02-19T14:55:11.839802+0000","last_unstale":"2024-02-19T14:55:12.846615+0000","last_undegraded":"2024-02-19T14:55:12.846615+0000","last_fullsized":"2024-02-19T14:55:12.846615+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:48.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844455+0000","last_change":"2024-02-19T14:55:11.836219+0000","last_active":"2024-02-19T14:55:12.844455+0000","last_peered":"2024-02-19T14:55:12.844455+0000","last_clean":"2024-02-19T14:55:12.844455+0000","last_became_active":"2024-02-19T14:55:11.835947+0000","last_became_peered":"2024-02-19T14:55:11.835947+0000","last_unstale":"2024-02-19T14:55:12.844455+0000","last_undegraded":"2024-02-19T14:55:12.844455+0000","last_fullsized":"2024-02-19T14:55:12.844455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:26.866265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838474+0000","last_change":"2024-02-19T14:55:11.844578+0000","last_active":"2024-02-19T14:55:12.838474+0000","last_peered":"2024-02-19T14:55:12.838474+0000","last_clean":"2024-02-19T14:55:12.838474+0000","last_became_active":"2024-02-19T14:55:11.844300+0000","last_became_peered":"2024-02-19T14:55:11.844300+0000","last_unstale":"2024-02-19T14:55:12.838474+0000","last_undegraded":"2024-02-19T14:55:12.838474+0000","last_fullsized":"2024-02-19T14:55:12.838474+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:52.830832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845810+0000","last_change":"2024-02-19T14:55:11.826254+0000","last_active":"2024-02-19T14:55:12.845810+0000","last_peered":"2024-02-19T14:55:12.845810+0000","last_clean":"2024-02-19T14:55:12.845810+0000","last_became_active":"2024-02-19T14:55:11.825905+0000","last_became_peered":"2024-02-19T14:55:11.825905+0000","last_unstale":"2024-02-19T14:55:12.845810+0000","last_undegraded":"2024-02-19T14:55:12.845810+0000","last_fullsized":"2024-02-19T14:55:12.845810+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:09.540005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846121+0000","last_change":"2024-02-19T14:55:11.837587+0000","last_active":"2024-02-19T14:55:12.846121+0000","last_peered":"2024-02-19T14:55:12.846121+0000","last_clean":"2024-02-19T14:55:12.846121+0000","last_became_active":"2024-02-19T14:55:11.837213+0000","last_became_peered":"2024-02-19T14:55:11.837213+0000","last_unstale":"2024-02-19T14:55:12.846121+0000","last_undegraded":"2024-02-19T14:55:12.846121+0000","last_fullsized":"2024-02-19T14:55:12.846121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:12.712057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846372+0000","last_change":"2024-02-19T14:55:11.831652+0000","last_active":"2024-02-19T14:55:12.846372+0000","last_peered":"2024-02-19T14:55:12.846372+0000","last_clean":"2024-02-19T14:55:12.846372+0000","last_became_active":"2024-02-19T14:55:11.831331+0000","last_became_peered":"2024-02-19T14:55:11.831331+0000","last_unstale":"2024-02-19T14:55:12.846372+0000","last_undegraded":"2024-02-19T14:55:12.846372+0000","last_fullsized":"2024-02-19T14:55:12.846372+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:24.300283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846694+0000","last_change":"2024-02-19T14:55:11.829608+0000","last_active":"2024-02-19T14:55:12.846694+0000","last_peered":"2024-02-19T14:55:12.846694+0000","last_clean":"2024-02-19T14:55:12.846694+0000","last_became_active":"2024-02-19T14:55:11.829302+0000","last_became_peered":"2024-02-19T14:55:11.829302+0000","last_unstale":"2024-02-19T14:55:12.846694+0000","last_undegraded":"2024-02-19T14:55:12.846694+0000","last_fullsized":"2024-02-19T14:55:12.846694+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:54.341805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846741+0000","last_change":"2024-02-19T14:55:11.839292+0000","last_active":"2024-02-19T14:55:12.846741+0000","last_peered":"2024-02-19T14:55:12.846741+0000","last_clean":"2024-02-19T14:55:12.846741+0000","last_became_active":"2024-02-19T14:55:11.838884+0000","last_became_peered":"2024-02-19T14:55:11.838884+0000","last_unstale":"2024-02-19T14:55:12.846741+0000","last_undegraded":"2024-02-19T14:55:12.846741+0000","last_fullsized":"2024-02-19T14:55:12.846741+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:52.830832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845977+0000","last_change":"2024-02-19T14:55:11.848002+0000","last_active":"2024-02-19T14:55:12.845977+0000","last_peered":"2024-02-19T14:55:12.845977+0000","last_clean":"2024-02-19T14:55:12.845977+0000","last_became_active":"2024-02-19T14:55:11.847712+0000","last_became_peered":"2024-02-19T14:55:11.847712+0000","last_unstale":"2024-02-19T14:55:12.845977+0000","last_undegraded":"2024-02-19T14:55:12.845977+0000","last_fullsized":"2024-02-19T14:55:12.845977+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:15.966669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835016+0000","last_change":"2024-02-19T14:55:11.839665+0000","last_active":"2024-02-19T14:55:12.835016+0000","last_peered":"2024-02-19T14:55:12.835016+0000","last_clean":"2024-02-19T14:55:12.835016+0000","last_became_active":"2024-02-19T14:55:11.838822+0000","last_became_peered":"2024-02-19T14:55:11.838822+0000","last_unstale":"2024-02-19T14:55:12.835016+0000","last_undegraded":"2024-02-19T14:55:12.835016+0000","last_fullsized":"2024-02-19T14:55:12.835016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:59.498962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835909+0000","last_change":"2024-02-19T14:55:11.831880+0000","last_active":"2024-02-19T14:55:12.835909+0000","last_peered":"2024-02-19T14:55:12.835909+0000","last_clean":"2024-02-19T14:55:12.835909+0000","last_became_active":"2024-02-19T14:55:11.831577+0000","last_became_peered":"2024-02-19T14:55:11.831577+0000","last_unstale":"2024-02-19T14:55:12.835909+0000","last_undegraded":"2024-02-19T14:55:12.835909+0000","last_fullsized":"2024-02-19T14:55:12.835909+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:12.105121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838390+0000","last_change":"2024-02-19T14:55:11.841580+0000","last_active":"2024-02-19T14:55:12.838390+0000","last_peered":"2024-02-19T14:55:12.838390+0000","last_clean":"2024-02-19T14:55:12.838390+0000","last_became_active":"2024-02-19T14:55:11.840946+0000","last_became_peered":"2024-02-19T14:55:11.840946+0000","last_unstale":"2024-02-19T14:55:12.838390+0000","last_undegraded":"2024-02-19T14:55:12.838390+0000","last_fullsized":"2024-02-19T14:55:12.838390+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:48.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846192+0000","last_change":"2024-02-19T14:55:11.842613+0000","last_active":"2024-02-19T14:55:12.846192+0000","last_peered":"2024-02-19T14:55:12.846192+0000","last_clean":"2024-02-19T14:55:12.846192+0000","last_became_active":"2024-02-19T14:55:11.842333+0000","last_became_peered":"2024-02-19T14:55:11.842333+0000","last_unstale":"2024-02-19T14:55:12.846192+0000","last_undegraded":"2024-02-19T14:55:12.846192+0000","last_fullsized":"2024-02-19T14:55:12.846192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:27:49.010666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845926+0000","last_change":"2024-02-19T14:55:11.841647+0000","last_active":"2024-02-19T14:55:12.845926+0000","last_peered":"2024-02-19T14:55:12.845926+0000","last_clean":"2024-02-19T14:55:12.845926+0000","last_became_active":"2024-02-19T14:55:11.841138+0000","last_became_peered":"2024-02-19T14:55:11.841138+0000","last_unstale":"2024-02-19T14:55:12.845926+0000","last_undegraded":"2024-02-19T14:55:12.845926+0000","last_fullsized":"2024-02-19T14:55:12.845926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:03.445127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846000+0000","last_change":"2024-02-19T14:55:11.835906+0000","last_active":"2024-02-19T14:55:12.846000+0000","last_peered":"2024-02-19T14:55:12.846000+0000","last_clean":"2024-02-19T14:55:12.846000+0000","last_became_active":"2024-02-19T14:55:11.835621+0000","last_became_peered":"2024-02-19T14:55:11.835621+0000","last_unstale":"2024-02-19T14:55:12.846000+0000","last_undegraded":"2024-02-19T14:55:12.846000+0000","last_fullsized":"2024-02-19T14:55:12.846000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:44:48.454635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844172+0000","last_change":"2024-02-19T14:55:11.839649+0000","last_active":"2024-02-19T14:55:12.844172+0000","last_peered":"2024-02-19T14:55:12.844172+0000","last_clean":"2024-02-19T14:55:12.844172+0000","last_became_active":"2024-02-19T14:55:11.839356+0000","last_became_peered":"2024-02-19T14:55:11.839356+0000","last_unstale":"2024-02-19T14:55:12.844172+0000","last_undegraded":"2024-02-19T14:55:12.844172+0000","last_fullsized":"2024-02-19T14:55:12.844172+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:18.957582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846905+0000","last_change":"2024-02-19T14:55:11.847744+0000","last_active":"2024-02-19T14:55:12.846905+0000","last_peered":"2024-02-19T14:55:12.846905+0000","last_clean":"2024-02-19T14:55:12.846905+0000","last_became_active":"2024-02-19T14:55:11.847431+0000","last_became_peered":"2024-02-19T14:55:11.847431+0000","last_unstale":"2024-02-19T14:55:12.846905+0000","last_undegraded":"2024-02-19T14:55:12.846905+0000","last_fullsized":"2024-02-19T14:55:12.846905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:54.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844590+0000","last_change":"2024-02-19T14:55:11.842771+0000","last_active":"2024-02-19T14:55:12.844590+0000","last_peered":"2024-02-19T14:55:12.844590+0000","last_clean":"2024-02-19T14:55:12.844590+0000","last_became_active":"2024-02-19T14:55:11.842497+0000","last_became_peered":"2024-02-19T14:55:11.842497+0000","last_unstale":"2024-02-19T14:55:12.844590+0000","last_undegraded":"2024-02-19T14:55:12.844590+0000","last_fullsized":"2024-02-19T14:55:12.844590+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:23.420269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836072+0000","last_change":"2024-02-19T14:55:11.843573+0000","last_active":"2024-02-19T14:55:12.836072+0000","last_peered":"2024-02-19T14:55:12.836072+0000","last_clean":"2024-02-19T14:55:12.836072+0000","last_became_active":"2024-02-19T14:55:11.842128+0000","last_became_peered":"2024-02-19T14:55:11.842128+0000","last_unstale":"2024-02-19T14:55:12.836072+0000","last_undegraded":"2024-02-19T14:55:12.836072+0000","last_fullsized":"2024-02-19T14:55:12.836072+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:12.712057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846677+0000","last_change":"2024-02-19T14:55:11.840274+0000","last_active":"2024-02-19T14:55:12.846677+0000","last_peered":"2024-02-19T14:55:12.846677+0000","last_clean":"2024-02-19T14:55:12.846677+0000","last_became_active":"2024-02-19T14:55:11.840011+0000","last_became_peered":"2024-02-19T14:55:11.840011+0000","last_unstale":"2024-02-19T14:55:12.846677+0000","last_undegraded":"2024-02-19T14:55:12.846677+0000","last_fullsized":"2024-02-19T14:55:12.846677+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:22.397758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845783+0000","last_change":"2024-02-19T14:55:11.836957+0000","last_active":"2024-02-19T14:55:12.845783+0000","last_peered":"2024-02-19T14:55:12.845783+0000","last_clean":"2024-02-19T14:55:12.845783+0000","last_became_active":"2024-02-19T14:55:11.836643+0000","last_became_peered":"2024-02-19T14:55:11.836643+0000","last_unstale":"2024-02-19T14:55:12.845783+0000","last_undegraded":"2024-02-19T14:55:12.845783+0000","last_fullsized":"2024-02-19T14:55:12.845783+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:12.105121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843775+0000","last_change":"2024-02-19T14:55:11.836924+0000","last_active":"2024-02-19T14:55:12.843775+0000","last_peered":"2024-02-19T14:55:12.843775+0000","last_clean":"2024-02-19T14:55:12.843775+0000","last_became_active":"2024-02-19T14:55:11.836666+0000","last_became_peered":"2024-02-19T14:55:11.836666+0000","last_unstale":"2024-02-19T14:55:12.843775+0000","last_undegraded":"2024-02-19T14:55:12.843775+0000","last_fullsized":"2024-02-19T14:55:12.843775+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:07.076656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845588+0000","last_change":"2024-02-19T14:55:11.836820+0000","last_active":"2024-02-19T14:55:12.845588+0000","last_peered":"2024-02-19T14:55:12.845588+0000","last_clean":"2024-02-19T14:55:12.845588+0000","last_became_active":"2024-02-19T14:55:11.836539+0000","last_became_peered":"2024-02-19T14:55:11.836539+0000","last_unstale":"2024-02-19T14:55:12.845588+0000","last_undegraded":"2024-02-19T14:55:12.845588+0000","last_fullsized":"2024-02-19T14:55:12.845588+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:55.108868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835499+0000","last_change":"2024-02-19T14:55:11.845753+0000","last_active":"2024-02-19T14:55:12.835499+0000","last_peered":"2024-02-19T14:55:12.835499+0000","last_clean":"2024-02-19T14:55:12.835499+0000","last_became_active":"2024-02-19T14:55:11.844140+0000","last_became_peered":"2024-02-19T14:55:11.844140+0000","last_unstale":"2024-02-19T14:55:12.835499+0000","last_undegraded":"2024-02-19T14:55:12.835499+0000","last_fullsized":"2024-02-19T14:55:12.835499+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:32.836180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836038+0000","last_change":"2024-02-19T14:55:11.842508+0000","last_active":"2024-02-19T14:55:12.836038+0000","last_peered":"2024-02-19T14:55:12.836038+0000","last_clean":"2024-02-19T14:55:12.836038+0000","last_became_active":"2024-02-19T14:55:11.842240+0000","last_became_peered":"2024-02-19T14:55:11.842240+0000","last_unstale":"2024-02-19T14:55:12.836038+0000","last_undegraded":"2024-02-19T14:55:12.836038+0000","last_fullsized":"2024-02-19T14:55:12.836038+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:09.540005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835171+0000","last_change":"2024-02-19T14:55:11.839624+0000","last_active":"2024-02-19T14:55:12.835171+0000","last_peered":"2024-02-19T14:55:12.835171+0000","last_clean":"2024-02-19T14:55:12.835171+0000","last_became_active":"2024-02-19T14:55:11.838344+0000","last_became_peered":"2024-02-19T14:55:11.838344+0000","last_unstale":"2024-02-19T14:55:12.835171+0000","last_undegraded":"2024-02-19T14:55:12.835171+0000","last_fullsized":"2024-02-19T14:55:12.835171+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:48.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846633+0000","last_change":"2024-02-19T14:55:11.840286+0000","last_active":"2024-02-19T14:55:12.846633+0000","last_peered":"2024-02-19T14:55:12.846633+0000","last_clean":"2024-02-19T14:55:12.846633+0000","last_became_active":"2024-02-19T14:55:11.838188+0000","last_became_peered":"2024-02-19T14:55:11.838188+0000","last_unstale":"2024-02-19T14:55:12.846633+0000","last_undegraded":"2024-02-19T14:55:12.846633+0000","last_fullsized":"2024-02-19T14:55:12.846633+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:59.498962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846816+0000","last_change":"2024-02-19T14:55:11.840600+0000","last_active":"2024-02-19T14:55:12.846816+0000","last_peered":"2024-02-19T14:55:12.846816+0000","last_clean":"2024-02-19T14:55:12.846816+0000","last_became_active":"2024-02-19T14:55:11.840332+0000","last_became_peered":"2024-02-19T14:55:11.840332+0000","last_unstale":"2024-02-19T14:55:12.846816+0000","last_undegraded":"2024-02-19T14:55:12.846816+0000","last_fullsized":"2024-02-19T14:55:12.846816+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:39.340061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835307+0000","last_change":"2024-02-19T14:55:11.845336+0000","last_active":"2024-02-19T14:55:12.835307+0000","last_peered":"2024-02-19T14:55:12.835307+0000","last_clean":"2024-02-19T14:55:12.835307+0000","last_became_active":"2024-02-19T14:55:11.844969+0000","last_became_peered":"2024-02-19T14:55:11.844969+0000","last_unstale":"2024-02-19T14:55:12.835307+0000","last_undegraded":"2024-02-19T14:55:12.835307+0000","last_fullsized":"2024-02-19T14:55:12.835307+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:59.436868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846256+0000","last_change":"2024-02-19T14:55:11.842304+0000","last_active":"2024-02-19T14:55:12.846256+0000","last_peered":"2024-02-19T14:55:12.846256+0000","last_clean":"2024-02-19T14:55:12.846256+0000","last_became_active":"2024-02-19T14:55:11.841995+0000","last_became_peered":"2024-02-19T14:55:11.841995+0000","last_unstale":"2024-02-19T14:55:12.846256+0000","last_undegraded":"2024-02-19T14:55:12.846256+0000","last_fullsized":"2024-02-19T14:55:12.846256+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:15.966669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846368+0000","last_change":"2024-02-19T14:55:11.844220+0000","last_active":"2024-02-19T14:55:12.846368+0000","last_peered":"2024-02-19T14:55:12.846368+0000","last_clean":"2024-02-19T14:55:12.846368+0000","last_became_active":"2024-02-19T14:55:11.843960+0000","last_became_peered":"2024-02-19T14:55:11.843960+0000","last_unstale":"2024-02-19T14:55:12.846368+0000","last_undegraded":"2024-02-19T14:55:12.846368+0000","last_fullsized":"2024-02-19T14:55:12.846368+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:15:28.985734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843943+0000","last_change":"2024-02-19T14:55:11.835128+0000","last_active":"2024-02-19T14:55:12.843943+0000","last_peered":"2024-02-19T14:55:12.843943+0000","last_clean":"2024-02-19T14:55:12.843943+0000","last_became_active":"2024-02-19T14:55:11.834211+0000","last_became_peered":"2024-02-19T14:55:11.834211+0000","last_unstale":"2024-02-19T14:55:12.843943+0000","last_undegraded":"2024-02-19T14:55:12.843943+0000","last_fullsized":"2024-02-19T14:55:12.843943+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:38:35.087537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"97'4","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838659+0000","last_change":"2024-02-19T14:55:10.837716+0000","last_active":"2024-02-19T14:55:12.838659+0000","last_peered":"2024-02-19T14:55:12.838659+0000","last_clean":"2024-02-19T14:55:12.838659+0000","last_became_active":"2024-02-19T14:55:10.837476+0000","last_became_peered":"2024-02-19T14:55:10.837476+0000","last_unstale":"2024-02-19T14:55:12.838659+0000","last_undegraded":"2024-02-19T14:55:12.838659+0000","last_fullsized":"2024-02-19T14:55:12.838659+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:11.097688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846420+0000","last_change":"2024-02-19T14:55:11.837039+0000","last_active":"2024-02-19T14:55:12.846420+0000","last_peered":"2024-02-19T14:55:12.846420+0000","last_clean":"2024-02-19T14:55:12.846420+0000","last_became_active":"2024-02-19T14:55:11.836777+0000","last_became_peered":"2024-02-19T14:55:11.836777+0000","last_unstale":"2024-02-19T14:55:12.846420+0000","last_undegraded":"2024-02-19T14:55:12.846420+0000","last_fullsized":"2024-02-19T14:55:12.846420+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:19:25.240648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846444+0000","last_change":"2024-02-19T14:55:10.828355+0000","last_active":"2024-02-19T14:55:12.846444+0000","last_peered":"2024-02-19T14:55:12.846444+0000","last_clean":"2024-02-19T14:55:12.846444+0000","last_became_active":"2024-02-19T14:55:10.828091+0000","last_became_peered":"2024-02-19T14:55:10.828091+0000","last_unstale":"2024-02-19T14:55:12.846444+0000","last_undegraded":"2024-02-19T14:55:12.846444+0000","last_fullsized":"2024-02-19T14:55:12.846444+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:11.899487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835473+0000","last_change":"2024-02-19T14:55:11.829400+0000","last_active":"2024-02-19T14:55:12.835473+0000","last_peered":"2024-02-19T14:55:12.835473+0000","last_clean":"2024-02-19T14:55:12.835473+0000","last_became_active":"2024-02-19T14:55:11.829112+0000","last_became_peered":"2024-02-19T14:55:11.829112+0000","last_unstale":"2024-02-19T14:55:12.835473+0000","last_undegraded":"2024-02-19T14:55:12.835473+0000","last_fullsized":"2024-02-19T14:55:12.835473+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:55.108868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836102+0000","last_change":"2024-02-19T14:55:10.828576+0000","last_active":"2024-02-19T14:55:12.836102+0000","last_peered":"2024-02-19T14:55:12.836102+0000","last_clean":"2024-02-19T14:55:12.836102+0000","last_became_active":"2024-02-19T14:55:10.828275+0000","last_became_peered":"2024-02-19T14:55:10.828275+0000","last_unstale":"2024-02-19T14:55:12.836102+0000","last_undegraded":"2024-02-19T14:55:12.836102+0000","last_fullsized":"2024-02-19T14:55:12.836102+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:47.224043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835827+0000","last_change":"2024-02-19T14:55:11.844773+0000","last_active":"2024-02-19T14:55:12.835827+0000","last_peered":"2024-02-19T14:55:12.835827+0000","last_clean":"2024-02-19T14:55:12.835827+0000","last_became_active":"2024-02-19T14:55:11.844496+0000","last_became_peered":"2024-02-19T14:55:11.844496+0000","last_unstale":"2024-02-19T14:55:12.835827+0000","last_undegraded":"2024-02-19T14:55:12.835827+0000","last_fullsized":"2024-02-19T14:55:12.835827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:54.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845742+0000","last_change":"2024-02-19T14:55:10.822667+0000","last_active":"2024-02-19T14:55:12.845742+0000","last_peered":"2024-02-19T14:55:12.845742+0000","last_clean":"2024-02-19T14:55:12.845742+0000","last_became_active":"2024-02-19T14:55:10.822426+0000","last_became_peered":"2024-02-19T14:55:10.822426+0000","last_unstale":"2024-02-19T14:55:12.845742+0000","last_undegraded":"2024-02-19T14:55:12.845742+0000","last_fullsized":"2024-02-19T14:55:12.845742+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:16.280072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838489+0000","last_change":"2024-02-19T14:55:11.842436+0000","last_active":"2024-02-19T14:55:12.838489+0000","last_peered":"2024-02-19T14:55:12.838489+0000","last_clean":"2024-02-19T14:55:12.838489+0000","last_became_active":"2024-02-19T14:55:11.842147+0000","last_became_peered":"2024-02-19T14:55:11.842147+0000","last_unstale":"2024-02-19T14:55:12.838489+0000","last_undegraded":"2024-02-19T14:55:12.838489+0000","last_fullsized":"2024-02-19T14:55:12.838489+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:05.650374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846363+0000","last_change":"2024-02-19T14:55:10.836425+0000","last_active":"2024-02-19T14:55:12.846363+0000","last_peered":"2024-02-19T14:55:12.846363+0000","last_clean":"2024-02-19T14:55:12.846363+0000","last_became_active":"2024-02-19T14:55:10.836138+0000","last_became_peered":"2024-02-19T14:55:10.836138+0000","last_unstale":"2024-02-19T14:55:12.846363+0000","last_undegraded":"2024-02-19T14:55:12.846363+0000","last_fullsized":"2024-02-19T14:55:12.846363+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:23.292850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844307+0000","last_change":"2024-02-19T14:55:11.842687+0000","last_active":"2024-02-19T14:55:12.844307+0000","last_peered":"2024-02-19T14:55:12.844307+0000","last_clean":"2024-02-19T14:55:12.844307+0000","last_became_active":"2024-02-19T14:55:11.842163+0000","last_became_peered":"2024-02-19T14:55:11.842163+0000","last_unstale":"2024-02-19T14:55:12.844307+0000","last_undegraded":"2024-02-19T14:55:12.844307+0000","last_fullsized":"2024-02-19T14:55:12.844307+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:45.219155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.834898+0000","last_change":"2024-02-19T14:55:10.828619+0000","last_active":"2024-02-19T14:55:12.834898+0000","last_peered":"2024-02-19T14:55:12.834898+0000","last_clean":"2024-02-19T14:55:12.834898+0000","last_became_active":"2024-02-19T14:55:10.828346+0000","last_became_peered":"2024-02-19T14:55:10.828346+0000","last_unstale":"2024-02-19T14:55:12.834898+0000","last_undegraded":"2024-02-19T14:55:12.834898+0000","last_fullsized":"2024-02-19T14:55:12.834898+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:58.429435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846905+0000","last_change":"2024-02-19T14:55:11.839309+0000","last_active":"2024-02-19T14:55:12.846905+0000","last_peered":"2024-02-19T14:55:12.846905+0000","last_clean":"2024-02-19T14:55:12.846905+0000","last_became_active":"2024-02-19T14:55:11.838322+0000","last_became_peered":"2024-02-19T14:55:11.838322+0000","last_unstale":"2024-02-19T14:55:12.846905+0000","last_undegraded":"2024-02-19T14:55:12.846905+0000","last_fullsized":"2024-02-19T14:55:12.846905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:59.280637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835233+0000","last_change":"2024-02-19T14:55:10.824800+0000","last_active":"2024-02-19T14:55:12.835233+0000","last_peered":"2024-02-19T14:55:12.835233+0000","last_clean":"2024-02-19T14:55:12.835233+0000","last_became_active":"2024-02-19T14:55:10.824497+0000","last_became_peered":"2024-02-19T14:55:10.824497+0000","last_unstale":"2024-02-19T14:55:12.835233+0000","last_undegraded":"2024-02-19T14:55:12.835233+0000","last_fullsized":"2024-02-19T14:55:12.835233+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:43.504550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838670+0000","last_change":"2024-02-19T14:55:11.841254+0000","last_active":"2024-02-19T14:55:12.838670+0000","last_peered":"2024-02-19T14:55:12.838670+0000","last_clean":"2024-02-19T14:55:12.838670+0000","last_became_active":"2024-02-19T14:55:11.840971+0000","last_became_peered":"2024-02-19T14:55:11.840971+0000","last_unstale":"2024-02-19T14:55:12.838670+0000","last_undegraded":"2024-02-19T14:55:12.838670+0000","last_fullsized":"2024-02-19T14:55:12.838670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:54.111189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838307+0000","last_change":"2024-02-19T14:55:10.827791+0000","last_active":"2024-02-19T14:55:12.838307+0000","last_peered":"2024-02-19T14:55:12.838307+0000","last_clean":"2024-02-19T14:55:12.838307+0000","last_became_active":"2024-02-19T14:55:10.827515+0000","last_became_peered":"2024-02-19T14:55:10.827515+0000","last_unstale":"2024-02-19T14:55:12.838307+0000","last_undegraded":"2024-02-19T14:55:12.838307+0000","last_fullsized":"2024-02-19T14:55:12.838307+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:54.101435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843768+0000","last_change":"2024-02-19T14:55:11.838975+0000","last_active":"2024-02-19T14:55:12.843768+0000","last_peered":"2024-02-19T14:55:12.843768+0000","last_clean":"2024-02-19T14:55:12.843768+0000","last_became_active":"2024-02-19T14:55:11.838686+0000","last_became_peered":"2024-02-19T14:55:11.838686+0000","last_unstale":"2024-02-19T14:55:12.843768+0000","last_undegraded":"2024-02-19T14:55:12.843768+0000","last_fullsized":"2024-02-19T14:55:12.843768+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:20:18.728778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835608+0000","last_change":"2024-02-19T14:55:10.830723+0000","last_active":"2024-02-19T14:55:12.835608+0000","last_peered":"2024-02-19T14:55:12.835608+0000","last_clean":"2024-02-19T14:55:12.835608+0000","last_became_active":"2024-02-19T14:55:10.830430+0000","last_became_peered":"2024-02-19T14:55:10.830430+0000","last_unstale":"2024-02-19T14:55:12.835608+0000","last_undegraded":"2024-02-19T14:55:12.835608+0000","last_fullsized":"2024-02-19T14:55:12.835608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:13.607719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845870+0000","last_change":"2024-02-19T14:55:11.830911+0000","last_active":"2024-02-19T14:55:12.845870+0000","last_peered":"2024-02-19T14:55:12.845870+0000","last_clean":"2024-02-19T14:55:12.845870+0000","last_became_active":"2024-02-19T14:55:11.830627+0000","last_became_peered":"2024-02-19T14:55:11.830627+0000","last_unstale":"2024-02-19T14:55:12.845870+0000","last_undegraded":"2024-02-19T14:55:12.845870+0000","last_fullsized":"2024-02-19T14:55:12.845870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:32.836180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843767+0000","last_change":"2024-02-19T14:55:10.835017+0000","last_active":"2024-02-19T14:55:12.843767+0000","last_peered":"2024-02-19T14:55:12.843767+0000","last_clean":"2024-02-19T14:55:12.843767+0000","last_became_active":"2024-02-19T14:55:10.834765+0000","last_became_peered":"2024-02-19T14:55:10.834765+0000","last_unstale":"2024-02-19T14:55:12.843767+0000","last_undegraded":"2024-02-19T14:55:12.843767+0000","last_fullsized":"2024-02-19T14:55:12.843767+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:29:57.675228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846530+0000","last_change":"2024-02-19T14:55:11.839278+0000","last_active":"2024-02-19T14:55:12.846530+0000","last_peered":"2024-02-19T14:55:12.846530+0000","last_clean":"2024-02-19T14:55:12.846530+0000","last_became_active":"2024-02-19T14:55:11.839015+0000","last_became_peered":"2024-02-19T14:55:11.839015+0000","last_unstale":"2024-02-19T14:55:12.846530+0000","last_undegraded":"2024-02-19T14:55:12.846530+0000","last_fullsized":"2024-02-19T14:55:12.846530+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:50.685409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"97'3","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846280+0000","last_change":"2024-02-19T14:55:10.833409+0000","last_active":"2024-02-19T14:55:12.846280+0000","last_peered":"2024-02-19T14:55:12.846280+0000","last_clean":"2024-02-19T14:55:12.846280+0000","last_became_active":"2024-02-19T14:55:10.833090+0000","last_became_peered":"2024-02-19T14:55:10.833090+0000","last_unstale":"2024-02-19T14:55:12.846280+0000","last_undegraded":"2024-02-19T14:55:12.846280+0000","last_fullsized":"2024-02-19T14:55:12.846280+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:58.491529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846004+0000","last_change":"2024-02-19T14:55:11.830719+0000","last_active":"2024-02-19T14:55:12.846004+0000","last_peered":"2024-02-19T14:55:12.846004+0000","last_clean":"2024-02-19T14:55:12.846004+0000","last_became_active":"2024-02-19T14:55:11.830470+0000","last_became_peered":"2024-02-19T14:55:11.830470+0000","last_unstale":"2024-02-19T14:55:12.846004+0000","last_undegraded":"2024-02-19T14:55:12.846004+0000","last_fullsized":"2024-02-19T14:55:12.846004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:06:25.750044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"97'6","reported_seq":20,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846902+0000","last_change":"2024-02-19T14:55:10.834962+0000","last_active":"2024-02-19T14:55:12.846902+0000","last_peered":"2024-02-19T14:55:12.846902+0000","last_clean":"2024-02-19T14:55:12.846902+0000","last_became_active":"2024-02-19T14:55:10.834567+0000","last_became_peered":"2024-02-19T14:55:10.834567+0000","last_unstale":"2024-02-19T14:55:12.846902+0000","last_undegraded":"2024-02-19T14:55:12.846902+0000","last_fullsized":"2024-02-19T14:55:12.846902+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:16.224486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836039+0000","last_change":"2024-02-19T14:55:11.834810+0000","last_active":"2024-02-19T14:55:12.836039+0000","last_peered":"2024-02-19T14:55:12.836039+0000","last_clean":"2024-02-19T14:55:12.836039+0000","last_became_active":"2024-02-19T14:55:11.834552+0000","last_became_peered":"2024-02-19T14:55:11.834552+0000","last_unstale":"2024-02-19T14:55:12.836039+0000","last_undegraded":"2024-02-19T14:55:12.836039+0000","last_fullsized":"2024-02-19T14:55:12.836039+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:59.280637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845942+0000","last_change":"2024-02-19T14:55:10.828427+0000","last_active":"2024-02-19T14:55:12.845942+0000","last_peered":"2024-02-19T14:55:12.845942+0000","last_clean":"2024-02-19T14:55:12.845942+0000","last_became_active":"2024-02-19T14:55:10.828177+0000","last_became_peered":"2024-02-19T14:55:10.828177+0000","last_unstale":"2024-02-19T14:55:12.845942+0000","last_undegraded":"2024-02-19T14:55:12.845942+0000","last_fullsized":"2024-02-19T14:55:12.845942+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:55.387522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846770+0000","last_change":"2024-02-19T14:55:11.836340+0000","last_active":"2024-02-19T14:55:12.846770+0000","last_peered":"2024-02-19T14:55:12.846770+0000","last_clean":"2024-02-19T14:55:12.846770+0000","last_became_active":"2024-02-19T14:55:11.836039+0000","last_became_peered":"2024-02-19T14:55:11.836039+0000","last_unstale":"2024-02-19T14:55:12.846770+0000","last_undegraded":"2024-02-19T14:55:12.846770+0000","last_fullsized":"2024-02-19T14:55:12.846770+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:05.650374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838608+0000","last_change":"2024-02-19T14:55:10.830725+0000","last_active":"2024-02-19T14:55:12.838608+0000","last_peered":"2024-02-19T14:55:12.838608+0000","last_clean":"2024-02-19T14:55:12.838608+0000","last_became_active":"2024-02-19T14:55:10.830107+0000","last_became_peered":"2024-02-19T14:55:10.830107+0000","last_unstale":"2024-02-19T14:55:12.838608+0000","last_undegraded":"2024-02-19T14:55:12.838608+0000","last_fullsized":"2024-02-19T14:55:12.838608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:31.828747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846672+0000","last_change":"2024-02-19T14:55:11.844286+0000","last_active":"2024-02-19T14:55:12.846672+0000","last_peered":"2024-02-19T14:55:12.846672+0000","last_clean":"2024-02-19T14:55:12.846672+0000","last_became_active":"2024-02-19T14:55:11.844007+0000","last_became_peered":"2024-02-19T14:55:11.844007+0000","last_unstale":"2024-02-19T14:55:12.846672+0000","last_undegraded":"2024-02-19T14:55:12.846672+0000","last_fullsized":"2024-02-19T14:55:12.846672+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:27.019592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846442+0000","last_change":"2024-02-19T14:55:10.830156+0000","last_active":"2024-02-19T14:55:12.846442+0000","last_peered":"2024-02-19T14:55:12.846442+0000","last_clean":"2024-02-19T14:55:12.846442+0000","last_became_active":"2024-02-19T14:55:10.829915+0000","last_became_peered":"2024-02-19T14:55:10.829915+0000","last_unstale":"2024-02-19T14:55:12.846442+0000","last_undegraded":"2024-02-19T14:55:12.846442+0000","last_fullsized":"2024-02-19T14:55:12.846442+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:52.111482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844310+0000","last_change":"2024-02-19T14:55:11.840359+0000","last_active":"2024-02-19T14:55:12.844310+0000","last_peered":"2024-02-19T14:55:12.844310+0000","last_clean":"2024-02-19T14:55:12.844310+0000","last_became_active":"2024-02-19T14:55:11.840093+0000","last_became_peered":"2024-02-19T14:55:11.840093+0000","last_unstale":"2024-02-19T14:55:12.844310+0000","last_undegraded":"2024-02-19T14:55:12.844310+0000","last_fullsized":"2024-02-19T14:55:12.844310+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:11.593198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"97'8","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835077+0000","last_change":"2024-02-19T14:55:10.824809+0000","last_active":"2024-02-19T14:55:12.835077+0000","last_peered":"2024-02-19T14:55:12.835077+0000","last_clean":"2024-02-19T14:55:12.835077+0000","last_became_active":"2024-02-19T14:55:10.824588+0000","last_became_peered":"2024-02-19T14:55:10.824588+0000","last_unstale":"2024-02-19T14:55:12.835077+0000","last_undegraded":"2024-02-19T14:55:12.835077+0000","last_fullsized":"2024-02-19T14:55:12.835077+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:16.280072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846394+0000","last_change":"2024-02-19T14:55:11.835061+0000","last_active":"2024-02-19T14:55:12.846394+0000","last_peered":"2024-02-19T14:55:12.846394+0000","last_clean":"2024-02-19T14:55:12.846394+0000","last_became_active":"2024-02-19T14:55:11.834743+0000","last_became_peered":"2024-02-19T14:55:11.834743+0000","last_unstale":"2024-02-19T14:55:12.846394+0000","last_undegraded":"2024-02-19T14:55:12.846394+0000","last_fullsized":"2024-02-19T14:55:12.846394+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:09:23.477805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846881+0000","last_change":"2024-02-19T14:55:10.835158+0000","last_active":"2024-02-19T14:55:12.846881+0000","last_peered":"2024-02-19T14:55:12.846881+0000","last_clean":"2024-02-19T14:55:12.846881+0000","last_became_active":"2024-02-19T14:55:10.834868+0000","last_became_peered":"2024-02-19T14:55:10.834868+0000","last_unstale":"2024-02-19T14:55:12.846881+0000","last_undegraded":"2024-02-19T14:55:12.846881+0000","last_fullsized":"2024-02-19T14:55:12.846881+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:53.077350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846295+0000","last_change":"2024-02-19T14:55:11.847670+0000","last_active":"2024-02-19T14:55:12.846295+0000","last_peered":"2024-02-19T14:55:12.846295+0000","last_clean":"2024-02-19T14:55:12.846295+0000","last_became_active":"2024-02-19T14:55:11.847382+0000","last_became_peered":"2024-02-19T14:55:11.847382+0000","last_unstale":"2024-02-19T14:55:12.846295+0000","last_undegraded":"2024-02-19T14:55:12.846295+0000","last_fullsized":"2024-02-19T14:55:12.846295+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:06:25.750044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846350+0000","last_change":"2024-02-19T14:55:10.830990+0000","last_active":"2024-02-19T14:55:12.846350+0000","last_peered":"2024-02-19T14:55:12.846350+0000","last_clean":"2024-02-19T14:55:12.846350+0000","last_became_active":"2024-02-19T14:55:10.830666+0000","last_became_peered":"2024-02-19T14:55:10.830666+0000","last_unstale":"2024-02-19T14:55:12.846350+0000","last_undegraded":"2024-02-19T14:55:12.846350+0000","last_fullsized":"2024-02-19T14:55:12.846350+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:47.004003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845821+0000","last_change":"2024-02-19T14:55:11.838641+0000","last_active":"2024-02-19T14:55:12.845821+0000","last_peered":"2024-02-19T14:55:12.845821+0000","last_clean":"2024-02-19T14:55:12.845821+0000","last_became_active":"2024-02-19T14:55:11.838352+0000","last_became_peered":"2024-02-19T14:55:11.838352+0000","last_unstale":"2024-02-19T14:55:12.845821+0000","last_undegraded":"2024-02-19T14:55:12.845821+0000","last_fullsized":"2024-02-19T14:55:12.845821+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:12.712057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836035+0000","last_change":"2024-02-19T14:55:10.836108+0000","last_active":"2024-02-19T14:55:12.836035+0000","last_peered":"2024-02-19T14:55:12.836035+0000","last_clean":"2024-02-19T14:55:12.836035+0000","last_became_active":"2024-02-19T14:55:10.835828+0000","last_became_peered":"2024-02-19T14:55:10.835828+0000","last_unstale":"2024-02-19T14:55:12.836035+0000","last_undegraded":"2024-02-19T14:55:12.836035+0000","last_fullsized":"2024-02-19T14:55:12.836035+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:13.652545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844256+0000","last_change":"2024-02-19T14:55:11.832238+0000","last_active":"2024-02-19T14:55:12.844256+0000","last_peered":"2024-02-19T14:55:12.844256+0000","last_clean":"2024-02-19T14:55:12.844256+0000","last_became_active":"2024-02-19T14:55:11.831685+0000","last_became_peered":"2024-02-19T14:55:11.831685+0000","last_unstale":"2024-02-19T14:55:12.844256+0000","last_undegraded":"2024-02-19T14:55:12.844256+0000","last_fullsized":"2024-02-19T14:55:12.844256+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:44:49.221183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846871+0000","last_change":"2024-02-19T14:55:10.832745+0000","last_active":"2024-02-19T14:55:12.846871+0000","last_peered":"2024-02-19T14:55:12.846871+0000","last_clean":"2024-02-19T14:55:12.846871+0000","last_became_active":"2024-02-19T14:55:10.832486+0000","last_became_peered":"2024-02-19T14:55:10.832486+0000","last_unstale":"2024-02-19T14:55:12.846871+0000","last_undegraded":"2024-02-19T14:55:12.846871+0000","last_fullsized":"2024-02-19T14:55:12.846871+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:16.240761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844159+0000","last_change":"2024-02-19T14:55:11.831504+0000","last_active":"2024-02-19T14:55:12.844159+0000","last_peered":"2024-02-19T14:55:12.844159+0000","last_clean":"2024-02-19T14:55:12.844159+0000","last_became_active":"2024-02-19T14:55:11.831195+0000","last_became_peered":"2024-02-19T14:55:11.831195+0000","last_unstale":"2024-02-19T14:55:12.844159+0000","last_undegraded":"2024-02-19T14:55:12.844159+0000","last_fullsized":"2024-02-19T14:55:12.844159+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:01.917498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846363+0000","last_change":"2024-02-19T14:55:10.833643+0000","last_active":"2024-02-19T14:55:12.846363+0000","last_peered":"2024-02-19T14:55:12.846363+0000","last_clean":"2024-02-19T14:55:12.846363+0000","last_became_active":"2024-02-19T14:55:10.833354+0000","last_became_peered":"2024-02-19T14:55:10.833354+0000","last_unstale":"2024-02-19T14:55:12.846363+0000","last_undegraded":"2024-02-19T14:55:12.846363+0000","last_fullsized":"2024-02-19T14:55:12.846363+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:34.739000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838630+0000","last_change":"2024-02-19T14:55:11.840593+0000","last_active":"2024-02-19T14:55:12.838630+0000","last_peered":"2024-02-19T14:55:12.838630+0000","last_clean":"2024-02-19T14:55:12.838630+0000","last_became_active":"2024-02-19T14:55:11.840334+0000","last_became_peered":"2024-02-19T14:55:11.840334+0000","last_unstale":"2024-02-19T14:55:12.838630+0000","last_undegraded":"2024-02-19T14:55:12.838630+0000","last_fullsized":"2024-02-19T14:55:12.838630+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:59.280637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846383+0000","last_change":"2024-02-19T14:55:10.833759+0000","last_active":"2024-02-19T14:55:12.846383+0000","last_peered":"2024-02-19T14:55:12.846383+0000","last_clean":"2024-02-19T14:55:12.846383+0000","last_became_active":"2024-02-19T14:55:10.833445+0000","last_became_peered":"2024-02-19T14:55:10.833445+0000","last_unstale":"2024-02-19T14:55:12.846383+0000","last_undegraded":"2024-02-19T14:55:12.846383+0000","last_fullsized":"2024-02-19T14:55:12.846383+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:16.280072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846534+0000","last_change":"2024-02-19T14:55:11.838671+0000","last_active":"2024-02-19T14:55:12.846534+0000","last_peered":"2024-02-19T14:55:12.846534+0000","last_clean":"2024-02-19T14:55:12.846534+0000","last_became_active":"2024-02-19T14:55:11.838431+0000","last_became_peered":"2024-02-19T14:55:11.838431+0000","last_unstale":"2024-02-19T14:55:12.846534+0000","last_undegraded":"2024-02-19T14:55:12.846534+0000","last_fullsized":"2024-02-19T14:55:12.846534+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:46:14.671100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846481+0000","last_change":"2024-02-19T14:55:10.834293+0000","last_active":"2024-02-19T14:55:12.846481+0000","last_peered":"2024-02-19T14:55:12.846481+0000","last_clean":"2024-02-19T14:55:12.846481+0000","last_became_active":"2024-02-19T14:55:10.833575+0000","last_became_peered":"2024-02-19T14:55:10.833575+0000","last_unstale":"2024-02-19T14:55:12.846481+0000","last_undegraded":"2024-02-19T14:55:12.846481+0000","last_fullsized":"2024-02-19T14:55:12.846481+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:54.208869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843751+0000","last_change":"2024-02-19T14:55:11.832477+0000","last_active":"2024-02-19T14:55:12.843751+0000","last_peered":"2024-02-19T14:55:12.843751+0000","last_clean":"2024-02-19T14:55:12.843751+0000","last_became_active":"2024-02-19T14:55:11.832094+0000","last_became_peered":"2024-02-19T14:55:11.832094+0000","last_unstale":"2024-02-19T14:55:12.843751+0000","last_undegraded":"2024-02-19T14:55:12.843751+0000","last_fullsized":"2024-02-19T14:55:12.843751+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:12:57.885633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845970+0000","last_change":"2024-02-19T14:55:10.827943+0000","last_active":"2024-02-19T14:55:12.845970+0000","last_peered":"2024-02-19T14:55:12.845970+0000","last_clean":"2024-02-19T14:55:12.845970+0000","last_became_active":"2024-02-19T14:55:10.827672+0000","last_became_peered":"2024-02-19T14:55:10.827672+0000","last_unstale":"2024-02-19T14:55:12.845970+0000","last_undegraded":"2024-02-19T14:55:12.845970+0000","last_fullsized":"2024-02-19T14:55:12.845970+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:47.224043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835117+0000","last_change":"2024-02-19T14:55:11.832420+0000","last_active":"2024-02-19T14:55:12.835117+0000","last_peered":"2024-02-19T14:55:12.835117+0000","last_clean":"2024-02-19T14:55:12.835117+0000","last_became_active":"2024-02-19T14:55:11.831650+0000","last_became_peered":"2024-02-19T14:55:11.831650+0000","last_unstale":"2024-02-19T14:55:12.835117+0000","last_undegraded":"2024-02-19T14:55:12.835117+0000","last_fullsized":"2024-02-19T14:55:12.835117+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:14.659978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843879+0000","last_change":"2024-02-19T14:55:10.833463+0000","last_active":"2024-02-19T14:55:12.843879+0000","last_peered":"2024-02-19T14:55:12.843879+0000","last_clean":"2024-02-19T14:55:12.843879+0000","last_became_active":"2024-02-19T14:55:10.833215+0000","last_became_peered":"2024-02-19T14:55:10.833215+0000","last_unstale":"2024-02-19T14:55:12.843879+0000","last_undegraded":"2024-02-19T14:55:12.843879+0000","last_fullsized":"2024-02-19T14:55:12.843879+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:40.066589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838593+0000","last_change":"2024-02-19T14:55:11.841227+0000","last_active":"2024-02-19T14:55:12.838593+0000","last_peered":"2024-02-19T14:55:12.838593+0000","last_clean":"2024-02-19T14:55:12.838593+0000","last_became_active":"2024-02-19T14:55:11.840620+0000","last_became_peered":"2024-02-19T14:55:11.840620+0000","last_unstale":"2024-02-19T14:55:12.838593+0000","last_undegraded":"2024-02-19T14:55:12.838593+0000","last_fullsized":"2024-02-19T14:55:12.838593+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:54.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846815+0000","last_change":"2024-02-19T14:55:10.834182+0000","last_active":"2024-02-19T14:55:12.846815+0000","last_peered":"2024-02-19T14:55:12.846815+0000","last_clean":"2024-02-19T14:55:12.846815+0000","last_became_active":"2024-02-19T14:55:10.833892+0000","last_became_peered":"2024-02-19T14:55:10.833892+0000","last_unstale":"2024-02-19T14:55:12.846815+0000","last_undegraded":"2024-02-19T14:55:12.846815+0000","last_fullsized":"2024-02-19T14:55:12.846815+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:55.387522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835655+0000","last_change":"2024-02-19T14:55:11.839500+0000","last_active":"2024-02-19T14:55:12.835655+0000","last_peered":"2024-02-19T14:55:12.835655+0000","last_clean":"2024-02-19T14:55:12.835655+0000","last_became_active":"2024-02-19T14:55:11.839198+0000","last_became_peered":"2024-02-19T14:55:11.839198+0000","last_unstale":"2024-02-19T14:55:12.835655+0000","last_undegraded":"2024-02-19T14:55:12.835655+0000","last_fullsized":"2024-02-19T14:55:12.835655+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:52.830832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845838+0000","last_change":"2024-02-19T14:55:10.837379+0000","last_active":"2024-02-19T14:55:12.845838+0000","last_peered":"2024-02-19T14:55:12.845838+0000","last_clean":"2024-02-19T14:55:12.845838+0000","last_became_active":"2024-02-19T14:55:10.837090+0000","last_became_peered":"2024-02-19T14:55:10.837090+0000","last_unstale":"2024-02-19T14:55:12.845838+0000","last_undegraded":"2024-02-19T14:55:12.845838+0000","last_fullsized":"2024-02-19T14:55:12.845838+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:16.224486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846777+0000","last_change":"2024-02-19T14:55:11.837770+0000","last_active":"2024-02-19T14:55:12.846777+0000","last_peered":"2024-02-19T14:55:12.846777+0000","last_clean":"2024-02-19T14:55:12.846777+0000","last_became_active":"2024-02-19T14:55:11.837497+0000","last_became_peered":"2024-02-19T14:55:11.837497+0000","last_unstale":"2024-02-19T14:55:12.846777+0000","last_undegraded":"2024-02-19T14:55:12.846777+0000","last_fullsized":"2024-02-19T14:55:12.846777+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:33.070924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835909+0000","last_change":"2024-02-19T14:55:10.835073+0000","last_active":"2024-02-19T14:55:12.835909+0000","last_peered":"2024-02-19T14:55:12.835909+0000","last_clean":"2024-02-19T14:55:12.835909+0000","last_became_active":"2024-02-19T14:55:10.834627+0000","last_became_peered":"2024-02-19T14:55:10.834627+0000","last_unstale":"2024-02-19T14:55:12.835909+0000","last_undegraded":"2024-02-19T14:55:12.835909+0000","last_fullsized":"2024-02-19T14:55:12.835909+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:57:04.701556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846914+0000","last_change":"2024-02-19T14:55:11.835889+0000","last_active":"2024-02-19T14:55:12.846914+0000","last_peered":"2024-02-19T14:55:12.846914+0000","last_clean":"2024-02-19T14:55:12.846914+0000","last_became_active":"2024-02-19T14:55:11.835635+0000","last_became_peered":"2024-02-19T14:55:11.835635+0000","last_unstale":"2024-02-19T14:55:12.846914+0000","last_undegraded":"2024-02-19T14:55:12.846914+0000","last_fullsized":"2024-02-19T14:55:12.846914+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:37:16.960964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846617+0000","last_change":"2024-02-19T14:55:10.835087+0000","last_active":"2024-02-19T14:55:12.846617+0000","last_peered":"2024-02-19T14:55:12.846617+0000","last_clean":"2024-02-19T14:55:12.846617+0000","last_became_active":"2024-02-19T14:55:10.834861+0000","last_became_peered":"2024-02-19T14:55:10.834861+0000","last_unstale":"2024-02-19T14:55:12.846617+0000","last_undegraded":"2024-02-19T14:55:12.846617+0000","last_fullsized":"2024-02-19T14:55:12.846617+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:24.433722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846298+0000","last_change":"2024-02-19T14:55:11.841244+0000","last_active":"2024-02-19T14:55:12.846298+0000","last_peered":"2024-02-19T14:55:12.846298+0000","last_clean":"2024-02-19T14:55:12.846298+0000","last_became_active":"2024-02-19T14:55:11.840964+0000","last_became_peered":"2024-02-19T14:55:11.840964+0000","last_unstale":"2024-02-19T14:55:12.846298+0000","last_undegraded":"2024-02-19T14:55:12.846298+0000","last_fullsized":"2024-02-19T14:55:12.846298+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:37.147578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846485+0000","last_change":"2024-02-19T14:55:10.836070+0000","last_active":"2024-02-19T14:55:12.846485+0000","last_peered":"2024-02-19T14:55:12.846485+0000","last_clean":"2024-02-19T14:55:12.846485+0000","last_became_active":"2024-02-19T14:55:10.835489+0000","last_became_peered":"2024-02-19T14:55:10.835489+0000","last_unstale":"2024-02-19T14:55:12.846485+0000","last_undegraded":"2024-02-19T14:55:12.846485+0000","last_fullsized":"2024-02-19T14:55:12.846485+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:47.224043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835667+0000","last_change":"2024-02-19T14:55:11.837369+0000","last_active":"2024-02-19T14:55:12.835667+0000","last_peered":"2024-02-19T14:55:12.835667+0000","last_clean":"2024-02-19T14:55:12.835667+0000","last_became_active":"2024-02-19T14:55:11.837065+0000","last_became_peered":"2024-02-19T14:55:11.837065+0000","last_unstale":"2024-02-19T14:55:12.835667+0000","last_undegraded":"2024-02-19T14:55:12.835667+0000","last_fullsized":"2024-02-19T14:55:12.835667+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:05.650374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846567+0000","last_change":"2024-02-19T14:55:10.830880+0000","last_active":"2024-02-19T14:55:12.846567+0000","last_peered":"2024-02-19T14:55:12.846567+0000","last_clean":"2024-02-19T14:55:12.846567+0000","last_became_active":"2024-02-19T14:55:10.830515+0000","last_became_peered":"2024-02-19T14:55:10.830515+0000","last_unstale":"2024-02-19T14:55:12.846567+0000","last_undegraded":"2024-02-19T14:55:12.846567+0000","last_fullsized":"2024-02-19T14:55:12.846567+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:47.721375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.834929+0000","last_change":"2024-02-19T14:55:11.834498+0000","last_active":"2024-02-19T14:55:12.834929+0000","last_peered":"2024-02-19T14:55:12.834929+0000","last_clean":"2024-02-19T14:55:12.834929+0000","last_became_active":"2024-02-19T14:55:11.834162+0000","last_became_peered":"2024-02-19T14:55:11.834162+0000","last_unstale":"2024-02-19T14:55:12.834929+0000","last_undegraded":"2024-02-19T14:55:12.834929+0000","last_fullsized":"2024-02-19T14:55:12.834929+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:24.300283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846547+0000","last_change":"2024-02-19T14:55:10.827941+0000","last_active":"2024-02-19T14:55:12.846547+0000","last_peered":"2024-02-19T14:55:12.846547+0000","last_clean":"2024-02-19T14:55:12.846547+0000","last_became_active":"2024-02-19T14:55:10.827293+0000","last_became_peered":"2024-02-19T14:55:10.827293+0000","last_unstale":"2024-02-19T14:55:12.846547+0000","last_undegraded":"2024-02-19T14:55:12.846547+0000","last_fullsized":"2024-02-19T14:55:12.846547+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:52.526291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":317,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845802+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:12.845802+0000","last_peered":"2024-02-19T14:55:12.845802+0000","last_clean":"2024-02-19T14:55:12.845802+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:12.845802+0000","last_undegraded":"2024-02-19T14:55:12.845802+0000","last_fullsized":"2024-02-19T14:55:12.845802+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1806336,"data_stored":1176252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.63400000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1236992,"data_stored":594632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48199999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.499},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.374},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.29399999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44}]}]},{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1216512,"data_stored":592416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.628}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.64200000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1224704,"data_stored":592508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40200000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.83099999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38100000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.624}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.442},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1208320,"data_stored":586306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.34300000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.60399999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.52800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1220608,"data_stored":592450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.371}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.318}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.47999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":592450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.376},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.314}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.497},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.41699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1814528,"data_stored":1177794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.60699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:14.808 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 Processor -- start 2024-02-19T14:55:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 -- start start 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45a80c30a0 0x7f45a8158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 0x7f45a81559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a80c3cf0 con 0x7f45a80c5130 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a8158b60 con 0x7f45a80c30a0 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.942+0000 7f45adfd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80c3520 con 0x7f45a80c5510 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 0x7f45a81559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45a80c30a0 0x7f45a8158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45a80c30a0 0x7f45a8158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:44420/0 (socket says 172.21.15.78:44420) 2024-02-19T14:55:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 0x7f45a81559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:39384/0 (socket says 172.21.15.78:39384) 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 -- 172.21.15.78:0/799614791 learned_addr learned my addr 172.21.15.78:0/799614791 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 -- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 msgr2=0x7f45a81559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 --2- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 0x7f45a81559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 -- 172.21.15.78:0/799614791 >> v1:172.21.15.150:6789/0 conn(0x7f45a80c5130 legacy=0x7f45a80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a67fc640 1 -- 172.21.15.78:0/799614791 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742459914 0 0) 0x7f45a80c3cf0 con 0x7f45a80c5130 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a6ffd640 1 -- 172.21.15.78:0/799614791 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a80c59a0 con 0x7f45a80c30a0 2024-02-19T14:55:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.943+0000 7f45a7fff640 1 --2- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 0x7f45a81559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.944+0000 7f45a6ffd640 1 --2- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c30a0 0x7f45a8158f80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e885ed35bf36dac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.944+0000 7f45a67fc640 1 -- 172.21.15.78:0/799614791 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c030840 con 0x7f45a80c30a0 2024-02-19T14:55:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.944+0000 7f45a67fc640 1 -- 172.21.15.78:0/799614791 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459c0309a0 con 0x7f45a80c30a0 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.944+0000 7f45a67fc640 1 -- 172.21.15.78:0/799614791 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c030d30 con 0x7f45a80c30a0 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 -- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c30a0 msgr2=0x7f45a8158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 --2- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c30a0 0x7f45a8158f80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c02b980 comp rx=0 tx=0).stop 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 -- 172.21.15.78:0/799614791 shutdown_connections 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 --2- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5510 0x7f45a81559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 --2- 172.21.15.78:0/799614791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c30a0 0x7f45a8158f80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 -- 172.21.15.78:0/799614791 >> 172.21.15.78:0/799614791 conn(0x7f45a806b4b0 msgr2=0x7f45a80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 -- 172.21.15.78:0/799614791 shutdown_connections 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.945+0000 7f45adfd0640 1 -- 172.21.15.78:0/799614791 wait complete. 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 Processor -- start 2024-02-19T14:55:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 -- start start 2024-02-19T14:55:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 0x7f45a80ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5130 0x7f45a80a8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c5510 0x7f45a80a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80ad900 con 0x7f45a80c30a0 2024-02-19T14:55:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80a8e00 con 0x7f45a80c5510 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.946+0000 7f45adfd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a81eb490 con 0x7f45a80c5130 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 0x7f45a80ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 0x7f45a80ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60228/0 (socket says 172.21.15.78:60228) 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 -- 172.21.15.78:0/2592001263 learned_addr learned my addr 172.21.15.78:0/2592001263 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a6ffd640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5130 0x7f45a80a8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a7fff640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c5510 0x7f45a80a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 -- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5130 msgr2=0x7f45a80a8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5130 0x7f45a80a8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 -- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c5510 msgr2=0x7f45a80a88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c5510 0x7f45a80a88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 -- 172.21.15.78:0/2592001263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0009a70 con 0x7f45a80c30a0 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45a77fe640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 0x7f45a80ab6b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f459800c4b0 tx=0x7f459800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598005480 con 0x7f45a80c30a0 2024-02-19T14:55:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45980035f0 con 0x7f45a80c30a0 2024-02-19T14:55:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45980039e0 con 0x7f45a80c30a0 2024-02-19T14:55:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c0094d0 con 0x7f45a80c30a0 2024-02-19T14:55:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.947+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a81ea9b0 con 0x7f45a80c30a0 2024-02-19T14:55:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.948+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a80c9400 con 0x7f45a80c30a0 2024-02-19T14:55:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.949+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4598003b40 con 0x7f45a80c30a0 2024-02-19T14:55:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.950+0000 7f458ffff640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f459406b840 0x7f459406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.950+0000 7f45a6ffd640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f459406b840 0x7f459406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.950+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4598089860 con 0x7f45a80c30a0 2024-02-19T14:55:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.950+0000 7f45a6ffd640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f459406b840 0x7f459406dd00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f459c036000 tx=0x7f459c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:14.954+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f45980541a0 con 0x7f45a80c30a0 2024-02-19T14:55:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.128+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45a8158530 con 0x7f459406b840 2024-02-19T14:55:15.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.142+0000 7f458ffff640 1 -- 172.21.15.78:0/2592001263 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289451 (secure 0 0 0) 0x7f459c03f040 con 0x7f459406b840 2024-02-19T14:55:15.148 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:15.149 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f459406b840 msgr2=0x7f459406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f459406b840 0x7f459406dd00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f459c036000 tx=0x7f459c008030 comp rx=0 tx=0).stop 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 msgr2=0x7f45a80ab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 0x7f45a80ab6b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f459800c4b0 tx=0x7f459800c970 comp rx=0 tx=0).stop 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 shutdown_connections 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f459406b840 0x7f459406dd00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a80c5510 0x7f45a80a88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a80c5130 0x7f45a80a8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 --2- 172.21.15.78:0/2592001263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a80c30a0 0x7f45a80ab6b0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 >> 172.21.15.78:0/2592001263 conn(0x7f45a806b4b0 msgr2=0x7f45a80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:15.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 shutdown_connections 2024-02-19T14:55:15.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.145+0000 7f45adfd0640 1 -- 172.21.15.78:0/2592001263 wait complete. 2024-02-19T14:55:15.159 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-02-19T14:55:14.097667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":247668,"kb_used_data":10692,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754727052,"statfs":{"total":773094113280,"available":772840501248,"internally_reserved":0,"allocated":10948608,"data_stored":5904808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.287754"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844130+0000","last_change":"2024-02-19T14:55:11.838771+0000","last_active":"2024-02-19T14:55:12.844130+0000","last_peered":"2024-02-19T14:55:12.844130+0000","last_clean":"2024-02-19T14:55:12.844130+0000","last_became_active":"2024-02-19T14:55:11.838434+0000","last_became_peered":"2024-02-19T14:55:11.838434+0000","last_unstale":"2024-02-19T14:55:12.844130+0000","last_undegraded":"2024-02-19T14:55:12.844130+0000","last_fullsized":"2024-02-19T14:55:12.844130+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:00.750078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846615+0000","last_change":"2024-02-19T14:55:11.840091+0000","last_active":"2024-02-19T14:55:12.846615+0000","last_peered":"2024-02-19T14:55:12.846615+0000","last_clean":"2024-02-19T14:55:12.846615+0000","last_became_active":"2024-02-19T14:55:11.839802+0000","last_became_peered":"2024-02-19T14:55:11.839802+0000","last_unstale":"2024-02-19T14:55:12.846615+0000","last_undegraded":"2024-02-19T14:55:12.846615+0000","last_fullsized":"2024-02-19T14:55:12.846615+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:48.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844455+0000","last_change":"2024-02-19T14:55:11.836219+0000","last_active":"2024-02-19T14:55:12.844455+0000","last_peered":"2024-02-19T14:55:12.844455+0000","last_clean":"2024-02-19T14:55:12.844455+0000","last_became_active":"2024-02-19T14:55:11.835947+0000","last_became_peered":"2024-02-19T14:55:11.835947+0000","last_unstale":"2024-02-19T14:55:12.844455+0000","last_undegraded":"2024-02-19T14:55:12.844455+0000","last_fullsized":"2024-02-19T14:55:12.844455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:26.866265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838474+0000","last_change":"2024-02-19T14:55:11.844578+0000","last_active":"2024-02-19T14:55:12.838474+0000","last_peered":"2024-02-19T14:55:12.838474+0000","last_clean":"2024-02-19T14:55:12.838474+0000","last_became_active":"2024-02-19T14:55:11.844300+0000","last_became_peered":"2024-02-19T14:55:11.844300+0000","last_unstale":"2024-02-19T14:55:12.838474+0000","last_undegraded":"2024-02-19T14:55:12.838474+0000","last_fullsized":"2024-02-19T14:55:12.838474+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:52.830832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845810+0000","last_change":"2024-02-19T14:55:11.826254+0000","last_active":"2024-02-19T14:55:12.845810+0000","last_peered":"2024-02-19T14:55:12.845810+0000","last_clean":"2024-02-19T14:55:12.845810+0000","last_became_active":"2024-02-19T14:55:11.825905+0000","last_became_peered":"2024-02-19T14:55:11.825905+0000","last_unstale":"2024-02-19T14:55:12.845810+0000","last_undegraded":"2024-02-19T14:55:12.845810+0000","last_fullsized":"2024-02-19T14:55:12.845810+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:09.540005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846121+0000","last_change":"2024-02-19T14:55:11.837587+0000","last_active":"2024-02-19T14:55:12.846121+0000","last_peered":"2024-02-19T14:55:12.846121+0000","last_clean":"2024-02-19T14:55:12.846121+0000","last_became_active":"2024-02-19T14:55:11.837213+0000","last_became_peered":"2024-02-19T14:55:11.837213+0000","last_unstale":"2024-02-19T14:55:12.846121+0000","last_undegraded":"2024-02-19T14:55:12.846121+0000","last_fullsized":"2024-02-19T14:55:12.846121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:12.712057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846372+0000","last_change":"2024-02-19T14:55:11.831652+0000","last_active":"2024-02-19T14:55:12.846372+0000","last_peered":"2024-02-19T14:55:12.846372+0000","last_clean":"2024-02-19T14:55:12.846372+0000","last_became_active":"2024-02-19T14:55:11.831331+0000","last_became_peered":"2024-02-19T14:55:11.831331+0000","last_unstale":"2024-02-19T14:55:12.846372+0000","last_undegraded":"2024-02-19T14:55:12.846372+0000","last_fullsized":"2024-02-19T14:55:12.846372+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:24.300283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846694+0000","last_change":"2024-02-19T14:55:11.829608+0000","last_active":"2024-02-19T14:55:12.846694+0000","last_peered":"2024-02-19T14:55:12.846694+0000","last_clean":"2024-02-19T14:55:12.846694+0000","last_became_active":"2024-02-19T14:55:11.829302+0000","last_became_peered":"2024-02-19T14:55:11.829302+0000","last_unstale":"2024-02-19T14:55:12.846694+0000","last_undegraded":"2024-02-19T14:55:12.846694+0000","last_fullsized":"2024-02-19T14:55:12.846694+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:54.341805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846741+0000","last_change":"2024-02-19T14:55:11.839292+0000","last_active":"2024-02-19T14:55:12.846741+0000","last_peered":"2024-02-19T14:55:12.846741+0000","last_clean":"2024-02-19T14:55:12.846741+0000","last_became_active":"2024-02-19T14:55:11.838884+0000","last_became_peered":"2024-02-19T14:55:11.838884+0000","last_unstale":"2024-02-19T14:55:12.846741+0000","last_undegraded":"2024-02-19T14:55:12.846741+0000","last_fullsized":"2024-02-19T14:55:12.846741+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:52.830832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845977+0000","last_change":"2024-02-19T14:55:11.848002+0000","last_active":"2024-02-19T14:55:12.845977+0000","last_peered":"2024-02-19T14:55:12.845977+0000","last_clean":"2024-02-19T14:55:12.845977+0000","last_became_active":"2024-02-19T14:55:11.847712+0000","last_became_peered":"2024-02-19T14:55:11.847712+0000","last_unstale":"2024-02-19T14:55:12.845977+0000","last_undegraded":"2024-02-19T14:55:12.845977+0000","last_fullsized":"2024-02-19T14:55:12.845977+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:15.966669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835016+0000","last_change":"2024-02-19T14:55:11.839665+0000","last_active":"2024-02-19T14:55:12.835016+0000","last_peered":"2024-02-19T14:55:12.835016+0000","last_clean":"2024-02-19T14:55:12.835016+0000","last_became_active":"2024-02-19T14:55:11.838822+0000","last_became_peered":"2024-02-19T14:55:11.838822+0000","last_unstale":"2024-02-19T14:55:12.835016+0000","last_undegraded":"2024-02-19T14:55:12.835016+0000","last_fullsized":"2024-02-19T14:55:12.835016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:59.498962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835909+0000","last_change":"2024-02-19T14:55:11.831880+0000","last_active":"2024-02-19T14:55:12.835909+0000","last_peered":"2024-02-19T14:55:12.835909+0000","last_clean":"2024-02-19T14:55:12.835909+0000","last_became_active":"2024-02-19T14:55:11.831577+0000","last_became_peered":"2024-02-19T14:55:11.831577+0000","last_unstale":"2024-02-19T14:55:12.835909+0000","last_undegraded":"2024-02-19T14:55:12.835909+0000","last_fullsized":"2024-02-19T14:55:12.835909+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:12.105121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838390+0000","last_change":"2024-02-19T14:55:11.841580+0000","last_active":"2024-02-19T14:55:12.838390+0000","last_peered":"2024-02-19T14:55:12.838390+0000","last_clean":"2024-02-19T14:55:12.838390+0000","last_became_active":"2024-02-19T14:55:11.840946+0000","last_became_peered":"2024-02-19T14:55:11.840946+0000","last_unstale":"2024-02-19T14:55:12.838390+0000","last_undegraded":"2024-02-19T14:55:12.838390+0000","last_fullsized":"2024-02-19T14:55:12.838390+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:48.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846192+0000","last_change":"2024-02-19T14:55:11.842613+0000","last_active":"2024-02-19T14:55:12.846192+0000","last_peered":"2024-02-19T14:55:12.846192+0000","last_clean":"2024-02-19T14:55:12.846192+0000","last_became_active":"2024-02-19T14:55:11.842333+0000","last_became_peered":"2024-02-19T14:55:11.842333+0000","last_unstale":"2024-02-19T14:55:12.846192+0000","last_undegraded":"2024-02-19T14:55:12.846192+0000","last_fullsized":"2024-02-19T14:55:12.846192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:27:49.010666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845926+0000","last_change":"2024-02-19T14:55:11.841647+0000","last_active":"2024-02-19T14:55:12.845926+0000","last_peered":"2024-02-19T14:55:12.845926+0000","last_clean":"2024-02-19T14:55:12.845926+0000","last_became_active":"2024-02-19T14:55:11.841138+0000","last_became_peered":"2024-02-19T14:55:11.841138+0000","last_unstale":"2024-02-19T14:55:12.845926+0000","last_undegraded":"2024-02-19T14:55:12.845926+0000","last_fullsized":"2024-02-19T14:55:12.845926+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:03.445127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846000+0000","last_change":"2024-02-19T14:55:11.835906+0000","last_active":"2024-02-19T14:55:12.846000+0000","last_peered":"2024-02-19T14:55:12.846000+0000","last_clean":"2024-02-19T14:55:12.846000+0000","last_became_active":"2024-02-19T14:55:11.835621+0000","last_became_peered":"2024-02-19T14:55:11.835621+0000","last_unstale":"2024-02-19T14:55:12.846000+0000","last_undegraded":"2024-02-19T14:55:12.846000+0000","last_fullsized":"2024-02-19T14:55:12.846000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:44:48.454635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844172+0000","last_change":"2024-02-19T14:55:11.839649+0000","last_active":"2024-02-19T14:55:12.844172+0000","last_peered":"2024-02-19T14:55:12.844172+0000","last_clean":"2024-02-19T14:55:12.844172+0000","last_became_active":"2024-02-19T14:55:11.839356+0000","last_became_peered":"2024-02-19T14:55:11.839356+0000","last_unstale":"2024-02-19T14:55:12.844172+0000","last_undegraded":"2024-02-19T14:55:12.844172+0000","last_fullsized":"2024-02-19T14:55:12.844172+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:18.957582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846905+0000","last_change":"2024-02-19T14:55:11.847744+0000","last_active":"2024-02-19T14:55:12.846905+0000","last_peered":"2024-02-19T14:55:12.846905+0000","last_clean":"2024-02-19T14:55:12.846905+0000","last_became_active":"2024-02-19T14:55:11.847431+0000","last_became_peered":"2024-02-19T14:55:11.847431+0000","last_unstale":"2024-02-19T14:55:12.846905+0000","last_undegraded":"2024-02-19T14:55:12.846905+0000","last_fullsized":"2024-02-19T14:55:12.846905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:54.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844590+0000","last_change":"2024-02-19T14:55:11.842771+0000","last_active":"2024-02-19T14:55:12.844590+0000","last_peered":"2024-02-19T14:55:12.844590+0000","last_clean":"2024-02-19T14:55:12.844590+0000","last_became_active":"2024-02-19T14:55:11.842497+0000","last_became_peered":"2024-02-19T14:55:11.842497+0000","last_unstale":"2024-02-19T14:55:12.844590+0000","last_undegraded":"2024-02-19T14:55:12.844590+0000","last_fullsized":"2024-02-19T14:55:12.844590+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:23.420269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836072+0000","last_change":"2024-02-19T14:55:11.843573+0000","last_active":"2024-02-19T14:55:12.836072+0000","last_peered":"2024-02-19T14:55:12.836072+0000","last_clean":"2024-02-19T14:55:12.836072+0000","last_became_active":"2024-02-19T14:55:11.842128+0000","last_became_peered":"2024-02-19T14:55:11.842128+0000","last_unstale":"2024-02-19T14:55:12.836072+0000","last_undegraded":"2024-02-19T14:55:12.836072+0000","last_fullsized":"2024-02-19T14:55:12.836072+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:12.712057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846677+0000","last_change":"2024-02-19T14:55:11.840274+0000","last_active":"2024-02-19T14:55:12.846677+0000","last_peered":"2024-02-19T14:55:12.846677+0000","last_clean":"2024-02-19T14:55:12.846677+0000","last_became_active":"2024-02-19T14:55:11.840011+0000","last_became_peered":"2024-02-19T14:55:11.840011+0000","last_unstale":"2024-02-19T14:55:12.846677+0000","last_undegraded":"2024-02-19T14:55:12.846677+0000","last_fullsized":"2024-02-19T14:55:12.846677+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:22.397758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845783+0000","last_change":"2024-02-19T14:55:11.836957+0000","last_active":"2024-02-19T14:55:12.845783+0000","last_peered":"2024-02-19T14:55:12.845783+0000","last_clean":"2024-02-19T14:55:12.845783+0000","last_became_active":"2024-02-19T14:55:11.836643+0000","last_became_peered":"2024-02-19T14:55:11.836643+0000","last_unstale":"2024-02-19T14:55:12.845783+0000","last_undegraded":"2024-02-19T14:55:12.845783+0000","last_fullsized":"2024-02-19T14:55:12.845783+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:12.105121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843775+0000","last_change":"2024-02-19T14:55:11.836924+0000","last_active":"2024-02-19T14:55:12.843775+0000","last_peered":"2024-02-19T14:55:12.843775+0000","last_clean":"2024-02-19T14:55:12.843775+0000","last_became_active":"2024-02-19T14:55:11.836666+0000","last_became_peered":"2024-02-19T14:55:11.836666+0000","last_unstale":"2024-02-19T14:55:12.843775+0000","last_undegraded":"2024-02-19T14:55:12.843775+0000","last_fullsized":"2024-02-19T14:55:12.843775+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:07.076656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845588+0000","last_change":"2024-02-19T14:55:11.836820+0000","last_active":"2024-02-19T14:55:12.845588+0000","last_peered":"2024-02-19T14:55:12.845588+0000","last_clean":"2024-02-19T14:55:12.845588+0000","last_became_active":"2024-02-19T14:55:11.836539+0000","last_became_peered":"2024-02-19T14:55:11.836539+0000","last_unstale":"2024-02-19T14:55:12.845588+0000","last_undegraded":"2024-02-19T14:55:12.845588+0000","last_fullsized":"2024-02-19T14:55:12.845588+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:55.108868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835499+0000","last_change":"2024-02-19T14:55:11.845753+0000","last_active":"2024-02-19T14:55:12.835499+0000","last_peered":"2024-02-19T14:55:12.835499+0000","last_clean":"2024-02-19T14:55:12.835499+0000","last_became_active":"2024-02-19T14:55:11.844140+0000","last_became_peered":"2024-02-19T14:55:11.844140+0000","last_unstale":"2024-02-19T14:55:12.835499+0000","last_undegraded":"2024-02-19T14:55:12.835499+0000","last_fullsized":"2024-02-19T14:55:12.835499+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:32.836180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836038+0000","last_change":"2024-02-19T14:55:11.842508+0000","last_active":"2024-02-19T14:55:12.836038+0000","last_peered":"2024-02-19T14:55:12.836038+0000","last_clean":"2024-02-19T14:55:12.836038+0000","last_became_active":"2024-02-19T14:55:11.842240+0000","last_became_peered":"2024-02-19T14:55:11.842240+0000","last_unstale":"2024-02-19T14:55:12.836038+0000","last_undegraded":"2024-02-19T14:55:12.836038+0000","last_fullsized":"2024-02-19T14:55:12.836038+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:09.540005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835171+0000","last_change":"2024-02-19T14:55:11.839624+0000","last_active":"2024-02-19T14:55:12.835171+0000","last_peered":"2024-02-19T14:55:12.835171+0000","last_clean":"2024-02-19T14:55:12.835171+0000","last_became_active":"2024-02-19T14:55:11.838344+0000","last_became_peered":"2024-02-19T14:55:11.838344+0000","last_unstale":"2024-02-19T14:55:12.835171+0000","last_undegraded":"2024-02-19T14:55:12.835171+0000","last_fullsized":"2024-02-19T14:55:12.835171+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:48.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846633+0000","last_change":"2024-02-19T14:55:11.840286+0000","last_active":"2024-02-19T14:55:12.846633+0000","last_peered":"2024-02-19T14:55:12.846633+0000","last_clean":"2024-02-19T14:55:12.846633+0000","last_became_active":"2024-02-19T14:55:11.838188+0000","last_became_peered":"2024-02-19T14:55:11.838188+0000","last_unstale":"2024-02-19T14:55:12.846633+0000","last_undegraded":"2024-02-19T14:55:12.846633+0000","last_fullsized":"2024-02-19T14:55:12.846633+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:59.498962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846816+0000","last_change":"2024-02-19T14:55:11.840600+0000","last_active":"2024-02-19T14:55:12.846816+0000","last_peered":"2024-02-19T14:55:12.846816+0000","last_clean":"2024-02-19T14:55:12.846816+0000","last_became_active":"2024-02-19T14:55:11.840332+0000","last_became_peered":"2024-02-19T14:55:11.840332+0000","last_unstale":"2024-02-19T14:55:12.846816+0000","last_undegraded":"2024-02-19T14:55:12.846816+0000","last_fullsized":"2024-02-19T14:55:12.846816+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:16:39.340061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835307+0000","last_change":"2024-02-19T14:55:11.845336+0000","last_active":"2024-02-19T14:55:12.835307+0000","last_peered":"2024-02-19T14:55:12.835307+0000","last_clean":"2024-02-19T14:55:12.835307+0000","last_became_active":"2024-02-19T14:55:11.844969+0000","last_became_peered":"2024-02-19T14:55:11.844969+0000","last_unstale":"2024-02-19T14:55:12.835307+0000","last_undegraded":"2024-02-19T14:55:12.835307+0000","last_fullsized":"2024-02-19T14:55:12.835307+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:59.436868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846256+0000","last_change":"2024-02-19T14:55:11.842304+0000","last_active":"2024-02-19T14:55:12.846256+0000","last_peered":"2024-02-19T14:55:12.846256+0000","last_clean":"2024-02-19T14:55:12.846256+0000","last_became_active":"2024-02-19T14:55:11.841995+0000","last_became_peered":"2024-02-19T14:55:11.841995+0000","last_unstale":"2024-02-19T14:55:12.846256+0000","last_undegraded":"2024-02-19T14:55:12.846256+0000","last_fullsized":"2024-02-19T14:55:12.846256+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:39:15.966669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846368+0000","last_change":"2024-02-19T14:55:11.844220+0000","last_active":"2024-02-19T14:55:12.846368+0000","last_peered":"2024-02-19T14:55:12.846368+0000","last_clean":"2024-02-19T14:55:12.846368+0000","last_became_active":"2024-02-19T14:55:11.843960+0000","last_became_peered":"2024-02-19T14:55:11.843960+0000","last_unstale":"2024-02-19T14:55:12.846368+0000","last_undegraded":"2024-02-19T14:55:12.846368+0000","last_fullsized":"2024-02-19T14:55:12.846368+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:15:28.985734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843943+0000","last_change":"2024-02-19T14:55:11.835128+0000","last_active":"2024-02-19T14:55:12.843943+0000","last_peered":"2024-02-19T14:55:12.843943+0000","last_clean":"2024-02-19T14:55:12.843943+0000","last_became_active":"2024-02-19T14:55:11.834211+0000","last_became_peered":"2024-02-19T14:55:11.834211+0000","last_unstale":"2024-02-19T14:55:12.843943+0000","last_undegraded":"2024-02-19T14:55:12.843943+0000","last_fullsized":"2024-02-19T14:55:12.843943+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:38:35.087537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"97'4","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838659+0000","last_change":"2024-02-19T14:55:10.837716+0000","last_active":"2024-02-19T14:55:12.838659+0000","last_peered":"2024-02-19T14:55:12.838659+0000","last_clean":"2024-02-19T14:55:12.838659+0000","last_became_active":"2024-02-19T14:55:10.837476+0000","last_became_peered":"2024-02-19T14:55:10.837476+0000","last_unstale":"2024-02-19T14:55:12.838659+0000","last_undegraded":"2024-02-19T14:55:12.838659+0000","last_fullsized":"2024-02-19T14:55:12.838659+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:11.097688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846420+0000","last_change":"2024-02-19T14:55:11.837039+0000","last_active":"2024-02-19T14:55:12.846420+0000","last_peered":"2024-02-19T14:55:12.846420+0000","last_clean":"2024-02-19T14:55:12.846420+0000","last_became_active":"2024-02-19T14:55:11.836777+0000","last_became_peered":"2024-02-19T14:55:11.836777+0000","last_unstale":"2024-02-19T14:55:12.846420+0000","last_undegraded":"2024-02-19T14:55:12.846420+0000","last_fullsized":"2024-02-19T14:55:12.846420+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:19:25.240648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846444+0000","last_change":"2024-02-19T14:55:10.828355+0000","last_active":"2024-02-19T14:55:12.846444+0000","last_peered":"2024-02-19T14:55:12.846444+0000","last_clean":"2024-02-19T14:55:12.846444+0000","last_became_active":"2024-02-19T14:55:10.828091+0000","last_became_peered":"2024-02-19T14:55:10.828091+0000","last_unstale":"2024-02-19T14:55:12.846444+0000","last_undegraded":"2024-02-19T14:55:12.846444+0000","last_fullsized":"2024-02-19T14:55:12.846444+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:42:11.899487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835473+0000","last_change":"2024-02-19T14:55:11.829400+0000","last_active":"2024-02-19T14:55:12.835473+0000","last_peered":"2024-02-19T14:55:12.835473+0000","last_clean":"2024-02-19T14:55:12.835473+0000","last_became_active":"2024-02-19T14:55:11.829112+0000","last_became_peered":"2024-02-19T14:55:11.829112+0000","last_unstale":"2024-02-19T14:55:12.835473+0000","last_undegraded":"2024-02-19T14:55:12.835473+0000","last_fullsized":"2024-02-19T14:55:12.835473+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:55.108868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836102+0000","last_change":"2024-02-19T14:55:10.828576+0000","last_active":"2024-02-19T14:55:12.836102+0000","last_peered":"2024-02-19T14:55:12.836102+0000","last_clean":"2024-02-19T14:55:12.836102+0000","last_became_active":"2024-02-19T14:55:10.828275+0000","last_became_peered":"2024-02-19T14:55:10.828275+0000","last_unstale":"2024-02-19T14:55:12.836102+0000","last_undegraded":"2024-02-19T14:55:12.836102+0000","last_fullsized":"2024-02-19T14:55:12.836102+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:47.224043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835827+0000","last_change":"2024-02-19T14:55:11.844773+0000","last_active":"2024-02-19T14:55:12.835827+0000","last_peered":"2024-02-19T14:55:12.835827+0000","last_clean":"2024-02-19T14:55:12.835827+0000","last_became_active":"2024-02-19T14:55:11.844496+0000","last_became_peered":"2024-02-19T14:55:11.844496+0000","last_unstale":"2024-02-19T14:55:12.835827+0000","last_undegraded":"2024-02-19T14:55:12.835827+0000","last_fullsized":"2024-02-19T14:55:12.835827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:54.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845742+0000","last_change":"2024-02-19T14:55:10.822667+0000","last_active":"2024-02-19T14:55:12.845742+0000","last_peered":"2024-02-19T14:55:12.845742+0000","last_clean":"2024-02-19T14:55:12.845742+0000","last_became_active":"2024-02-19T14:55:10.822426+0000","last_became_peered":"2024-02-19T14:55:10.822426+0000","last_unstale":"2024-02-19T14:55:12.845742+0000","last_undegraded":"2024-02-19T14:55:12.845742+0000","last_fullsized":"2024-02-19T14:55:12.845742+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:16.280072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838489+0000","last_change":"2024-02-19T14:55:11.842436+0000","last_active":"2024-02-19T14:55:12.838489+0000","last_peered":"2024-02-19T14:55:12.838489+0000","last_clean":"2024-02-19T14:55:12.838489+0000","last_became_active":"2024-02-19T14:55:11.842147+0000","last_became_peered":"2024-02-19T14:55:11.842147+0000","last_unstale":"2024-02-19T14:55:12.838489+0000","last_undegraded":"2024-02-19T14:55:12.838489+0000","last_fullsized":"2024-02-19T14:55:12.838489+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:05.650374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846363+0000","last_change":"2024-02-19T14:55:10.836425+0000","last_active":"2024-02-19T14:55:12.846363+0000","last_peered":"2024-02-19T14:55:12.846363+0000","last_clean":"2024-02-19T14:55:12.846363+0000","last_became_active":"2024-02-19T14:55:10.836138+0000","last_became_peered":"2024-02-19T14:55:10.836138+0000","last_unstale":"2024-02-19T14:55:12.846363+0000","last_undegraded":"2024-02-19T14:55:12.846363+0000","last_fullsized":"2024-02-19T14:55:12.846363+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:23.292850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844307+0000","last_change":"2024-02-19T14:55:11.842687+0000","last_active":"2024-02-19T14:55:12.844307+0000","last_peered":"2024-02-19T14:55:12.844307+0000","last_clean":"2024-02-19T14:55:12.844307+0000","last_became_active":"2024-02-19T14:55:11.842163+0000","last_became_peered":"2024-02-19T14:55:11.842163+0000","last_unstale":"2024-02-19T14:55:12.844307+0000","last_undegraded":"2024-02-19T14:55:12.844307+0000","last_fullsized":"2024-02-19T14:55:12.844307+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:07:45.219155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.834898+0000","last_change":"2024-02-19T14:55:10.828619+0000","last_active":"2024-02-19T14:55:12.834898+0000","last_peered":"2024-02-19T14:55:12.834898+0000","last_clean":"2024-02-19T14:55:12.834898+0000","last_became_active":"2024-02-19T14:55:10.828346+0000","last_became_peered":"2024-02-19T14:55:10.828346+0000","last_unstale":"2024-02-19T14:55:12.834898+0000","last_undegraded":"2024-02-19T14:55:12.834898+0000","last_fullsized":"2024-02-19T14:55:12.834898+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:33:58.429435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846905+0000","last_change":"2024-02-19T14:55:11.839309+0000","last_active":"2024-02-19T14:55:12.846905+0000","last_peered":"2024-02-19T14:55:12.846905+0000","last_clean":"2024-02-19T14:55:12.846905+0000","last_became_active":"2024-02-19T14:55:11.838322+0000","last_became_peered":"2024-02-19T14:55:11.838322+0000","last_unstale":"2024-02-19T14:55:12.846905+0000","last_undegraded":"2024-02-19T14:55:12.846905+0000","last_fullsized":"2024-02-19T14:55:12.846905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:59.280637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835233+0000","last_change":"2024-02-19T14:55:10.824800+0000","last_active":"2024-02-19T14:55:12.835233+0000","last_peered":"2024-02-19T14:55:12.835233+0000","last_clean":"2024-02-19T14:55:12.835233+0000","last_became_active":"2024-02-19T14:55:10.824497+0000","last_became_peered":"2024-02-19T14:55:10.824497+0000","last_unstale":"2024-02-19T14:55:12.835233+0000","last_undegraded":"2024-02-19T14:55:12.835233+0000","last_fullsized":"2024-02-19T14:55:12.835233+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:31:43.504550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838670+0000","last_change":"2024-02-19T14:55:11.841254+0000","last_active":"2024-02-19T14:55:12.838670+0000","last_peered":"2024-02-19T14:55:12.838670+0000","last_clean":"2024-02-19T14:55:12.838670+0000","last_became_active":"2024-02-19T14:55:11.840971+0000","last_became_peered":"2024-02-19T14:55:11.840971+0000","last_unstale":"2024-02-19T14:55:12.838670+0000","last_undegraded":"2024-02-19T14:55:12.838670+0000","last_fullsized":"2024-02-19T14:55:12.838670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:54.111189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838307+0000","last_change":"2024-02-19T14:55:10.827791+0000","last_active":"2024-02-19T14:55:12.838307+0000","last_peered":"2024-02-19T14:55:12.838307+0000","last_clean":"2024-02-19T14:55:12.838307+0000","last_became_active":"2024-02-19T14:55:10.827515+0000","last_became_peered":"2024-02-19T14:55:10.827515+0000","last_unstale":"2024-02-19T14:55:12.838307+0000","last_undegraded":"2024-02-19T14:55:12.838307+0000","last_fullsized":"2024-02-19T14:55:12.838307+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:43:54.101435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843768+0000","last_change":"2024-02-19T14:55:11.838975+0000","last_active":"2024-02-19T14:55:12.843768+0000","last_peered":"2024-02-19T14:55:12.843768+0000","last_clean":"2024-02-19T14:55:12.843768+0000","last_became_active":"2024-02-19T14:55:11.838686+0000","last_became_peered":"2024-02-19T14:55:11.838686+0000","last_unstale":"2024-02-19T14:55:12.843768+0000","last_undegraded":"2024-02-19T14:55:12.843768+0000","last_fullsized":"2024-02-19T14:55:12.843768+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:20:18.728778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835608+0000","last_change":"2024-02-19T14:55:10.830723+0000","last_active":"2024-02-19T14:55:12.835608+0000","last_peered":"2024-02-19T14:55:12.835608+0000","last_clean":"2024-02-19T14:55:12.835608+0000","last_became_active":"2024-02-19T14:55:10.830430+0000","last_became_peered":"2024-02-19T14:55:10.830430+0000","last_unstale":"2024-02-19T14:55:12.835608+0000","last_undegraded":"2024-02-19T14:55:12.835608+0000","last_fullsized":"2024-02-19T14:55:12.835608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:15:13.607719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845870+0000","last_change":"2024-02-19T14:55:11.830911+0000","last_active":"2024-02-19T14:55:12.845870+0000","last_peered":"2024-02-19T14:55:12.845870+0000","last_clean":"2024-02-19T14:55:12.845870+0000","last_became_active":"2024-02-19T14:55:11.830627+0000","last_became_peered":"2024-02-19T14:55:11.830627+0000","last_unstale":"2024-02-19T14:55:12.845870+0000","last_undegraded":"2024-02-19T14:55:12.845870+0000","last_fullsized":"2024-02-19T14:55:12.845870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:32.836180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843767+0000","last_change":"2024-02-19T14:55:10.835017+0000","last_active":"2024-02-19T14:55:12.843767+0000","last_peered":"2024-02-19T14:55:12.843767+0000","last_clean":"2024-02-19T14:55:12.843767+0000","last_became_active":"2024-02-19T14:55:10.834765+0000","last_became_peered":"2024-02-19T14:55:10.834765+0000","last_unstale":"2024-02-19T14:55:12.843767+0000","last_undegraded":"2024-02-19T14:55:12.843767+0000","last_fullsized":"2024-02-19T14:55:12.843767+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:29:57.675228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846530+0000","last_change":"2024-02-19T14:55:11.839278+0000","last_active":"2024-02-19T14:55:12.846530+0000","last_peered":"2024-02-19T14:55:12.846530+0000","last_clean":"2024-02-19T14:55:12.846530+0000","last_became_active":"2024-02-19T14:55:11.839015+0000","last_became_peered":"2024-02-19T14:55:11.839015+0000","last_unstale":"2024-02-19T14:55:12.846530+0000","last_undegraded":"2024-02-19T14:55:12.846530+0000","last_fullsized":"2024-02-19T14:55:12.846530+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:47:50.685409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"97'3","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846280+0000","last_change":"2024-02-19T14:55:10.833409+0000","last_active":"2024-02-19T14:55:12.846280+0000","last_peered":"2024-02-19T14:55:12.846280+0000","last_clean":"2024-02-19T14:55:12.846280+0000","last_became_active":"2024-02-19T14:55:10.833090+0000","last_became_peered":"2024-02-19T14:55:10.833090+0000","last_unstale":"2024-02-19T14:55:12.846280+0000","last_undegraded":"2024-02-19T14:55:12.846280+0000","last_fullsized":"2024-02-19T14:55:12.846280+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:58.491529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846004+0000","last_change":"2024-02-19T14:55:11.830719+0000","last_active":"2024-02-19T14:55:12.846004+0000","last_peered":"2024-02-19T14:55:12.846004+0000","last_clean":"2024-02-19T14:55:12.846004+0000","last_became_active":"2024-02-19T14:55:11.830470+0000","last_became_peered":"2024-02-19T14:55:11.830470+0000","last_unstale":"2024-02-19T14:55:12.846004+0000","last_undegraded":"2024-02-19T14:55:12.846004+0000","last_fullsized":"2024-02-19T14:55:12.846004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:06:25.750044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"97'6","reported_seq":20,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846902+0000","last_change":"2024-02-19T14:55:10.834962+0000","last_active":"2024-02-19T14:55:12.846902+0000","last_peered":"2024-02-19T14:55:12.846902+0000","last_clean":"2024-02-19T14:55:12.846902+0000","last_became_active":"2024-02-19T14:55:10.834567+0000","last_became_peered":"2024-02-19T14:55:10.834567+0000","last_unstale":"2024-02-19T14:55:12.846902+0000","last_undegraded":"2024-02-19T14:55:12.846902+0000","last_fullsized":"2024-02-19T14:55:12.846902+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:16.224486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836039+0000","last_change":"2024-02-19T14:55:11.834810+0000","last_active":"2024-02-19T14:55:12.836039+0000","last_peered":"2024-02-19T14:55:12.836039+0000","last_clean":"2024-02-19T14:55:12.836039+0000","last_became_active":"2024-02-19T14:55:11.834552+0000","last_became_peered":"2024-02-19T14:55:11.834552+0000","last_unstale":"2024-02-19T14:55:12.836039+0000","last_undegraded":"2024-02-19T14:55:12.836039+0000","last_fullsized":"2024-02-19T14:55:12.836039+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:59.280637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845942+0000","last_change":"2024-02-19T14:55:10.828427+0000","last_active":"2024-02-19T14:55:12.845942+0000","last_peered":"2024-02-19T14:55:12.845942+0000","last_clean":"2024-02-19T14:55:12.845942+0000","last_became_active":"2024-02-19T14:55:10.828177+0000","last_became_peered":"2024-02-19T14:55:10.828177+0000","last_unstale":"2024-02-19T14:55:12.845942+0000","last_undegraded":"2024-02-19T14:55:12.845942+0000","last_fullsized":"2024-02-19T14:55:12.845942+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:55.387522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846770+0000","last_change":"2024-02-19T14:55:11.836340+0000","last_active":"2024-02-19T14:55:12.846770+0000","last_peered":"2024-02-19T14:55:12.846770+0000","last_clean":"2024-02-19T14:55:12.846770+0000","last_became_active":"2024-02-19T14:55:11.836039+0000","last_became_peered":"2024-02-19T14:55:11.836039+0000","last_unstale":"2024-02-19T14:55:12.846770+0000","last_undegraded":"2024-02-19T14:55:12.846770+0000","last_fullsized":"2024-02-19T14:55:12.846770+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:05.650374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838608+0000","last_change":"2024-02-19T14:55:10.830725+0000","last_active":"2024-02-19T14:55:12.838608+0000","last_peered":"2024-02-19T14:55:12.838608+0000","last_clean":"2024-02-19T14:55:12.838608+0000","last_became_active":"2024-02-19T14:55:10.830107+0000","last_became_peered":"2024-02-19T14:55:10.830107+0000","last_unstale":"2024-02-19T14:55:12.838608+0000","last_undegraded":"2024-02-19T14:55:12.838608+0000","last_fullsized":"2024-02-19T14:55:12.838608+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:31.828747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846672+0000","last_change":"2024-02-19T14:55:11.844286+0000","last_active":"2024-02-19T14:55:12.846672+0000","last_peered":"2024-02-19T14:55:12.846672+0000","last_clean":"2024-02-19T14:55:12.846672+0000","last_became_active":"2024-02-19T14:55:11.844007+0000","last_became_peered":"2024-02-19T14:55:11.844007+0000","last_unstale":"2024-02-19T14:55:12.846672+0000","last_undegraded":"2024-02-19T14:55:12.846672+0000","last_fullsized":"2024-02-19T14:55:12.846672+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:53:27.019592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846442+0000","last_change":"2024-02-19T14:55:10.830156+0000","last_active":"2024-02-19T14:55:12.846442+0000","last_peered":"2024-02-19T14:55:12.846442+0000","last_clean":"2024-02-19T14:55:12.846442+0000","last_became_active":"2024-02-19T14:55:10.829915+0000","last_became_peered":"2024-02-19T14:55:10.829915+0000","last_unstale":"2024-02-19T14:55:12.846442+0000","last_undegraded":"2024-02-19T14:55:12.846442+0000","last_fullsized":"2024-02-19T14:55:12.846442+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:52.111482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844310+0000","last_change":"2024-02-19T14:55:11.840359+0000","last_active":"2024-02-19T14:55:12.844310+0000","last_peered":"2024-02-19T14:55:12.844310+0000","last_clean":"2024-02-19T14:55:12.844310+0000","last_became_active":"2024-02-19T14:55:11.840093+0000","last_became_peered":"2024-02-19T14:55:11.840093+0000","last_unstale":"2024-02-19T14:55:12.844310+0000","last_undegraded":"2024-02-19T14:55:12.844310+0000","last_fullsized":"2024-02-19T14:55:12.844310+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:11.593198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"97'8","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835077+0000","last_change":"2024-02-19T14:55:10.824809+0000","last_active":"2024-02-19T14:55:12.835077+0000","last_peered":"2024-02-19T14:55:12.835077+0000","last_clean":"2024-02-19T14:55:12.835077+0000","last_became_active":"2024-02-19T14:55:10.824588+0000","last_became_peered":"2024-02-19T14:55:10.824588+0000","last_unstale":"2024-02-19T14:55:12.835077+0000","last_undegraded":"2024-02-19T14:55:12.835077+0000","last_fullsized":"2024-02-19T14:55:12.835077+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:16.280072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846394+0000","last_change":"2024-02-19T14:55:11.835061+0000","last_active":"2024-02-19T14:55:12.846394+0000","last_peered":"2024-02-19T14:55:12.846394+0000","last_clean":"2024-02-19T14:55:12.846394+0000","last_became_active":"2024-02-19T14:55:11.834743+0000","last_became_peered":"2024-02-19T14:55:11.834743+0000","last_unstale":"2024-02-19T14:55:12.846394+0000","last_undegraded":"2024-02-19T14:55:12.846394+0000","last_fullsized":"2024-02-19T14:55:12.846394+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:09:23.477805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846881+0000","last_change":"2024-02-19T14:55:10.835158+0000","last_active":"2024-02-19T14:55:12.846881+0000","last_peered":"2024-02-19T14:55:12.846881+0000","last_clean":"2024-02-19T14:55:12.846881+0000","last_became_active":"2024-02-19T14:55:10.834868+0000","last_became_peered":"2024-02-19T14:55:10.834868+0000","last_unstale":"2024-02-19T14:55:12.846881+0000","last_undegraded":"2024-02-19T14:55:12.846881+0000","last_fullsized":"2024-02-19T14:55:12.846881+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:23:53.077350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846295+0000","last_change":"2024-02-19T14:55:11.847670+0000","last_active":"2024-02-19T14:55:12.846295+0000","last_peered":"2024-02-19T14:55:12.846295+0000","last_clean":"2024-02-19T14:55:12.846295+0000","last_became_active":"2024-02-19T14:55:11.847382+0000","last_became_peered":"2024-02-19T14:55:11.847382+0000","last_unstale":"2024-02-19T14:55:12.846295+0000","last_undegraded":"2024-02-19T14:55:12.846295+0000","last_fullsized":"2024-02-19T14:55:12.846295+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:06:25.750044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"97'1","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846350+0000","last_change":"2024-02-19T14:55:10.830990+0000","last_active":"2024-02-19T14:55:12.846350+0000","last_peered":"2024-02-19T14:55:12.846350+0000","last_clean":"2024-02-19T14:55:12.846350+0000","last_became_active":"2024-02-19T14:55:10.830666+0000","last_became_peered":"2024-02-19T14:55:10.830666+0000","last_unstale":"2024-02-19T14:55:12.846350+0000","last_undegraded":"2024-02-19T14:55:12.846350+0000","last_fullsized":"2024-02-19T14:55:12.846350+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:47.004003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845821+0000","last_change":"2024-02-19T14:55:11.838641+0000","last_active":"2024-02-19T14:55:12.845821+0000","last_peered":"2024-02-19T14:55:12.845821+0000","last_clean":"2024-02-19T14:55:12.845821+0000","last_became_active":"2024-02-19T14:55:11.838352+0000","last_became_peered":"2024-02-19T14:55:11.838352+0000","last_unstale":"2024-02-19T14:55:12.845821+0000","last_undegraded":"2024-02-19T14:55:12.845821+0000","last_fullsized":"2024-02-19T14:55:12.845821+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:12.712057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.836035+0000","last_change":"2024-02-19T14:55:10.836108+0000","last_active":"2024-02-19T14:55:12.836035+0000","last_peered":"2024-02-19T14:55:12.836035+0000","last_clean":"2024-02-19T14:55:12.836035+0000","last_became_active":"2024-02-19T14:55:10.835828+0000","last_became_peered":"2024-02-19T14:55:10.835828+0000","last_unstale":"2024-02-19T14:55:12.836035+0000","last_undegraded":"2024-02-19T14:55:12.836035+0000","last_fullsized":"2024-02-19T14:55:12.836035+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:13.652545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844256+0000","last_change":"2024-02-19T14:55:11.832238+0000","last_active":"2024-02-19T14:55:12.844256+0000","last_peered":"2024-02-19T14:55:12.844256+0000","last_clean":"2024-02-19T14:55:12.844256+0000","last_became_active":"2024-02-19T14:55:11.831685+0000","last_became_peered":"2024-02-19T14:55:11.831685+0000","last_unstale":"2024-02-19T14:55:12.844256+0000","last_undegraded":"2024-02-19T14:55:12.844256+0000","last_fullsized":"2024-02-19T14:55:12.844256+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:44:49.221183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846871+0000","last_change":"2024-02-19T14:55:10.832745+0000","last_active":"2024-02-19T14:55:12.846871+0000","last_peered":"2024-02-19T14:55:12.846871+0000","last_clean":"2024-02-19T14:55:12.846871+0000","last_became_active":"2024-02-19T14:55:10.832486+0000","last_became_peered":"2024-02-19T14:55:10.832486+0000","last_unstale":"2024-02-19T14:55:12.846871+0000","last_undegraded":"2024-02-19T14:55:12.846871+0000","last_fullsized":"2024-02-19T14:55:12.846871+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:16.240761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.844159+0000","last_change":"2024-02-19T14:55:11.831504+0000","last_active":"2024-02-19T14:55:12.844159+0000","last_peered":"2024-02-19T14:55:12.844159+0000","last_clean":"2024-02-19T14:55:12.844159+0000","last_became_active":"2024-02-19T14:55:11.831195+0000","last_became_peered":"2024-02-19T14:55:11.831195+0000","last_unstale":"2024-02-19T14:55:12.844159+0000","last_undegraded":"2024-02-19T14:55:12.844159+0000","last_fullsized":"2024-02-19T14:55:12.844159+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:01.917498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846363+0000","last_change":"2024-02-19T14:55:10.833643+0000","last_active":"2024-02-19T14:55:12.846363+0000","last_peered":"2024-02-19T14:55:12.846363+0000","last_clean":"2024-02-19T14:55:12.846363+0000","last_became_active":"2024-02-19T14:55:10.833354+0000","last_became_peered":"2024-02-19T14:55:10.833354+0000","last_unstale":"2024-02-19T14:55:12.846363+0000","last_undegraded":"2024-02-19T14:55:12.846363+0000","last_fullsized":"2024-02-19T14:55:12.846363+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:39:34.739000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838630+0000","last_change":"2024-02-19T14:55:11.840593+0000","last_active":"2024-02-19T14:55:12.838630+0000","last_peered":"2024-02-19T14:55:12.838630+0000","last_clean":"2024-02-19T14:55:12.838630+0000","last_became_active":"2024-02-19T14:55:11.840334+0000","last_became_peered":"2024-02-19T14:55:11.840334+0000","last_unstale":"2024-02-19T14:55:12.838630+0000","last_undegraded":"2024-02-19T14:55:12.838630+0000","last_fullsized":"2024-02-19T14:55:12.838630+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:16:59.280637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846383+0000","last_change":"2024-02-19T14:55:10.833759+0000","last_active":"2024-02-19T14:55:12.846383+0000","last_peered":"2024-02-19T14:55:12.846383+0000","last_clean":"2024-02-19T14:55:12.846383+0000","last_became_active":"2024-02-19T14:55:10.833445+0000","last_became_peered":"2024-02-19T14:55:10.833445+0000","last_unstale":"2024-02-19T14:55:12.846383+0000","last_undegraded":"2024-02-19T14:55:12.846383+0000","last_fullsized":"2024-02-19T14:55:12.846383+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:16.280072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846534+0000","last_change":"2024-02-19T14:55:11.838671+0000","last_active":"2024-02-19T14:55:12.846534+0000","last_peered":"2024-02-19T14:55:12.846534+0000","last_clean":"2024-02-19T14:55:12.846534+0000","last_became_active":"2024-02-19T14:55:11.838431+0000","last_became_peered":"2024-02-19T14:55:11.838431+0000","last_unstale":"2024-02-19T14:55:12.846534+0000","last_undegraded":"2024-02-19T14:55:12.846534+0000","last_fullsized":"2024-02-19T14:55:12.846534+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:46:14.671100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846481+0000","last_change":"2024-02-19T14:55:10.834293+0000","last_active":"2024-02-19T14:55:12.846481+0000","last_peered":"2024-02-19T14:55:12.846481+0000","last_clean":"2024-02-19T14:55:12.846481+0000","last_became_active":"2024-02-19T14:55:10.833575+0000","last_became_peered":"2024-02-19T14:55:10.833575+0000","last_unstale":"2024-02-19T14:55:12.846481+0000","last_undegraded":"2024-02-19T14:55:12.846481+0000","last_fullsized":"2024-02-19T14:55:12.846481+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:54.208869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843751+0000","last_change":"2024-02-19T14:55:11.832477+0000","last_active":"2024-02-19T14:55:12.843751+0000","last_peered":"2024-02-19T14:55:12.843751+0000","last_clean":"2024-02-19T14:55:12.843751+0000","last_became_active":"2024-02-19T14:55:11.832094+0000","last_became_peered":"2024-02-19T14:55:11.832094+0000","last_unstale":"2024-02-19T14:55:12.843751+0000","last_undegraded":"2024-02-19T14:55:12.843751+0000","last_fullsized":"2024-02-19T14:55:12.843751+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:12:57.885633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845970+0000","last_change":"2024-02-19T14:55:10.827943+0000","last_active":"2024-02-19T14:55:12.845970+0000","last_peered":"2024-02-19T14:55:12.845970+0000","last_clean":"2024-02-19T14:55:12.845970+0000","last_became_active":"2024-02-19T14:55:10.827672+0000","last_became_peered":"2024-02-19T14:55:10.827672+0000","last_unstale":"2024-02-19T14:55:12.845970+0000","last_undegraded":"2024-02-19T14:55:12.845970+0000","last_fullsized":"2024-02-19T14:55:12.845970+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:47.224043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835117+0000","last_change":"2024-02-19T14:55:11.832420+0000","last_active":"2024-02-19T14:55:12.835117+0000","last_peered":"2024-02-19T14:55:12.835117+0000","last_clean":"2024-02-19T14:55:12.835117+0000","last_became_active":"2024-02-19T14:55:11.831650+0000","last_became_peered":"2024-02-19T14:55:11.831650+0000","last_unstale":"2024-02-19T14:55:12.835117+0000","last_undegraded":"2024-02-19T14:55:12.835117+0000","last_fullsized":"2024-02-19T14:55:12.835117+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:14.659978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.843879+0000","last_change":"2024-02-19T14:55:10.833463+0000","last_active":"2024-02-19T14:55:12.843879+0000","last_peered":"2024-02-19T14:55:12.843879+0000","last_clean":"2024-02-19T14:55:12.843879+0000","last_became_active":"2024-02-19T14:55:10.833215+0000","last_became_peered":"2024-02-19T14:55:10.833215+0000","last_unstale":"2024-02-19T14:55:12.843879+0000","last_undegraded":"2024-02-19T14:55:12.843879+0000","last_fullsized":"2024-02-19T14:55:12.843879+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:40.066589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.838593+0000","last_change":"2024-02-19T14:55:11.841227+0000","last_active":"2024-02-19T14:55:12.838593+0000","last_peered":"2024-02-19T14:55:12.838593+0000","last_clean":"2024-02-19T14:55:12.838593+0000","last_became_active":"2024-02-19T14:55:11.840620+0000","last_became_peered":"2024-02-19T14:55:11.840620+0000","last_unstale":"2024-02-19T14:55:12.838593+0000","last_undegraded":"2024-02-19T14:55:12.838593+0000","last_fullsized":"2024-02-19T14:55:12.838593+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:54.679598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846815+0000","last_change":"2024-02-19T14:55:10.834182+0000","last_active":"2024-02-19T14:55:12.846815+0000","last_peered":"2024-02-19T14:55:12.846815+0000","last_clean":"2024-02-19T14:55:12.846815+0000","last_became_active":"2024-02-19T14:55:10.833892+0000","last_became_peered":"2024-02-19T14:55:10.833892+0000","last_unstale":"2024-02-19T14:55:12.846815+0000","last_undegraded":"2024-02-19T14:55:12.846815+0000","last_fullsized":"2024-02-19T14:55:12.846815+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:55.387522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835655+0000","last_change":"2024-02-19T14:55:11.839500+0000","last_active":"2024-02-19T14:55:12.835655+0000","last_peered":"2024-02-19T14:55:12.835655+0000","last_clean":"2024-02-19T14:55:12.835655+0000","last_became_active":"2024-02-19T14:55:11.839198+0000","last_became_peered":"2024-02-19T14:55:11.839198+0000","last_unstale":"2024-02-19T14:55:12.835655+0000","last_undegraded":"2024-02-19T14:55:12.835655+0000","last_fullsized":"2024-02-19T14:55:12.835655+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:52.830832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845838+0000","last_change":"2024-02-19T14:55:10.837379+0000","last_active":"2024-02-19T14:55:12.845838+0000","last_peered":"2024-02-19T14:55:12.845838+0000","last_clean":"2024-02-19T14:55:12.845838+0000","last_became_active":"2024-02-19T14:55:10.837090+0000","last_became_peered":"2024-02-19T14:55:10.837090+0000","last_unstale":"2024-02-19T14:55:12.845838+0000","last_undegraded":"2024-02-19T14:55:12.845838+0000","last_fullsized":"2024-02-19T14:55:12.845838+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:16.224486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846777+0000","last_change":"2024-02-19T14:55:11.837770+0000","last_active":"2024-02-19T14:55:12.846777+0000","last_peered":"2024-02-19T14:55:12.846777+0000","last_clean":"2024-02-19T14:55:12.846777+0000","last_became_active":"2024-02-19T14:55:11.837497+0000","last_became_peered":"2024-02-19T14:55:11.837497+0000","last_unstale":"2024-02-19T14:55:12.846777+0000","last_undegraded":"2024-02-19T14:55:12.846777+0000","last_fullsized":"2024-02-19T14:55:12.846777+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:33.070924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835909+0000","last_change":"2024-02-19T14:55:10.835073+0000","last_active":"2024-02-19T14:55:12.835909+0000","last_peered":"2024-02-19T14:55:12.835909+0000","last_clean":"2024-02-19T14:55:12.835909+0000","last_became_active":"2024-02-19T14:55:10.834627+0000","last_became_peered":"2024-02-19T14:55:10.834627+0000","last_unstale":"2024-02-19T14:55:12.835909+0000","last_undegraded":"2024-02-19T14:55:12.835909+0000","last_fullsized":"2024-02-19T14:55:12.835909+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:57:04.701556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846914+0000","last_change":"2024-02-19T14:55:11.835889+0000","last_active":"2024-02-19T14:55:12.846914+0000","last_peered":"2024-02-19T14:55:12.846914+0000","last_clean":"2024-02-19T14:55:12.846914+0000","last_became_active":"2024-02-19T14:55:11.835635+0000","last_became_peered":"2024-02-19T14:55:11.835635+0000","last_unstale":"2024-02-19T14:55:12.846914+0000","last_undegraded":"2024-02-19T14:55:12.846914+0000","last_fullsized":"2024-02-19T14:55:12.846914+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:37:16.960964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846617+0000","last_change":"2024-02-19T14:55:10.835087+0000","last_active":"2024-02-19T14:55:12.846617+0000","last_peered":"2024-02-19T14:55:12.846617+0000","last_clean":"2024-02-19T14:55:12.846617+0000","last_became_active":"2024-02-19T14:55:10.834861+0000","last_became_peered":"2024-02-19T14:55:10.834861+0000","last_unstale":"2024-02-19T14:55:12.846617+0000","last_undegraded":"2024-02-19T14:55:12.846617+0000","last_fullsized":"2024-02-19T14:55:12.846617+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:47:24.433722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846298+0000","last_change":"2024-02-19T14:55:11.841244+0000","last_active":"2024-02-19T14:55:12.846298+0000","last_peered":"2024-02-19T14:55:12.846298+0000","last_clean":"2024-02-19T14:55:12.846298+0000","last_became_active":"2024-02-19T14:55:11.840964+0000","last_became_peered":"2024-02-19T14:55:11.840964+0000","last_unstale":"2024-02-19T14:55:12.846298+0000","last_undegraded":"2024-02-19T14:55:12.846298+0000","last_fullsized":"2024-02-19T14:55:12.846298+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:37.147578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846485+0000","last_change":"2024-02-19T14:55:10.836070+0000","last_active":"2024-02-19T14:55:12.846485+0000","last_peered":"2024-02-19T14:55:12.846485+0000","last_clean":"2024-02-19T14:55:12.846485+0000","last_became_active":"2024-02-19T14:55:10.835489+0000","last_became_peered":"2024-02-19T14:55:10.835489+0000","last_unstale":"2024-02-19T14:55:12.846485+0000","last_undegraded":"2024-02-19T14:55:12.846485+0000","last_fullsized":"2024-02-19T14:55:12.846485+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:47.224043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.835667+0000","last_change":"2024-02-19T14:55:11.837369+0000","last_active":"2024-02-19T14:55:12.835667+0000","last_peered":"2024-02-19T14:55:12.835667+0000","last_clean":"2024-02-19T14:55:12.835667+0000","last_became_active":"2024-02-19T14:55:11.837065+0000","last_became_peered":"2024-02-19T14:55:11.837065+0000","last_unstale":"2024-02-19T14:55:12.835667+0000","last_undegraded":"2024-02-19T14:55:12.835667+0000","last_fullsized":"2024-02-19T14:55:12.835667+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:53:05.650374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846567+0000","last_change":"2024-02-19T14:55:10.830880+0000","last_active":"2024-02-19T14:55:12.846567+0000","last_peered":"2024-02-19T14:55:12.846567+0000","last_clean":"2024-02-19T14:55:12.846567+0000","last_became_active":"2024-02-19T14:55:10.830515+0000","last_became_peered":"2024-02-19T14:55:10.830515+0000","last_unstale":"2024-02-19T14:55:12.846567+0000","last_undegraded":"2024-02-19T14:55:12.846567+0000","last_fullsized":"2024-02-19T14:55:12.846567+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:47.721375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.834929+0000","last_change":"2024-02-19T14:55:11.834498+0000","last_active":"2024-02-19T14:55:12.834929+0000","last_peered":"2024-02-19T14:55:12.834929+0000","last_clean":"2024-02-19T14:55:12.834929+0000","last_became_active":"2024-02-19T14:55:11.834162+0000","last_became_peered":"2024-02-19T14:55:11.834162+0000","last_unstale":"2024-02-19T14:55:12.834929+0000","last_undegraded":"2024-02-19T14:55:12.834929+0000","last_fullsized":"2024-02-19T14:55:12.834929+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:10.809532+0000","last_clean_scrub_stamp":"2024-02-19T14:55:10.809532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:24.300283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.846547+0000","last_change":"2024-02-19T14:55:10.827941+0000","last_active":"2024-02-19T14:55:12.846547+0000","last_peered":"2024-02-19T14:55:12.846547+0000","last_clean":"2024-02-19T14:55:12.846547+0000","last_became_active":"2024-02-19T14:55:10.827293+0000","last_became_peered":"2024-02-19T14:55:10.827293+0000","last_unstale":"2024-02-19T14:55:12.846547+0000","last_undegraded":"2024-02-19T14:55:12.846547+0000","last_fullsized":"2024-02-19T14:55:12.846547+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:09.802099+0000","last_clean_scrub_stamp":"2024-02-19T14:55:09.802099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:52.526291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":317,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-19T14:55:12.845802+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:12.845802+0000","last_peered":"2024-02-19T14:55:12.845802+0000","last_clean":"2024-02-19T14:55:12.845802+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:12.845802+0000","last_undegraded":"2024-02-19T14:55:12.845802+0000","last_fullsized":"2024-02-19T14:55:12.845802+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1806336,"data_stored":1176252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.64700000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.63400000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1236992,"data_stored":594632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48199999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.499},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.374},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.29399999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.44}]}]},{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1216512,"data_stored":592416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.628}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.64200000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.58199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1224704,"data_stored":592508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40200000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.83099999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38100000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.624}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.442},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28208,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343632,"statfs":{"total":96636764160,"available":96607879168,"internally_reserved":0,"allocated":1208320,"data_stored":586306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.34300000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.60399999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.52800000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1220608,"data_stored":592450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.371}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.318}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.51300000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.47999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":592450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.376},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.314}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.497},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.41699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1814528,"data_stored":1177794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.60699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:15.164 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:55:15.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:55:15.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.308+0000 7f6ffeaff640 1 Processor -- start 2024-02-19T14:55:15.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.308+0000 7f6ffeaff640 1 -- start start 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffeaff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ff8152700 0x7f6ff80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffeaff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffeaff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff81551f0 con 0x7f6ff80c8e00 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffeaff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff8155490 con 0x7f6ff8152700 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffeaff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8155770 con 0x7f6ff80c5730 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ff8152700 0x7f6ff80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6ff8152700 0x7f6ff80c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60240/0 (socket says 172.21.15.78:60240) 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 -- 172.21.15.78:0/3109545556 learned_addr learned my addr 172.21.15.78:0/3109545556 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:15.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ff7fff640 1 --2- 172.21.15.78:0/3109545556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 -- 172.21.15.78:0/3109545556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 msgr2=0x7f6ff806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 --2- 172.21.15.78:0/3109545556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 -- 172.21.15.78:0/3109545556 >> v1:172.21.15.150:6789/0 conn(0x7f6ff80c8e00 legacy=0x7f6ff8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.309+0000 7f6ffc89b640 1 -- 172.21.15.78:0/3109545556 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff8155a50 con 0x7f6ff8152700 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.310+0000 7f6ffc89b640 1 --2- 172.21.15.78:0/3109545556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80c51f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1211803c2dafe52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.310+0000 7f6ff77fe640 1 -- 172.21.15.78:0/3109545556 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec030840 con 0x7f6ff8152700 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.310+0000 7f6ff77fe640 1 -- 172.21.15.78:0/3109545556 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec0309a0 con 0x7f6ff8152700 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.310+0000 7f6ff77fe640 1 -- 172.21.15.78:0/3109545556 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec030d30 con 0x7f6ff8152700 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3109545556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 msgr2=0x7f6ff80c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3109545556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80c51f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec02b980 comp rx=0 tx=0).stop 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3109545556 shutdown_connections 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3109545556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3109545556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80c51f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3109545556 >> 172.21.15.78:0/3109545556 conn(0x7f6ff80cac90 msgr2=0x7f6ff80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3109545556 shutdown_connections 2024-02-19T14:55:15.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.311+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3109545556 wait complete. 2024-02-19T14:55:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.312+0000 7f6ffeaff640 1 Processor -- start 2024-02-19T14:55:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.312+0000 7f6ffeaff640 1 -- start start 2024-02-19T14:55:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffeaff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff81eba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffeaff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff80c8e00 0x7f6ff81ea860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffeaff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffeaff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80c1ee0 con 0x7f6ff8152700 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffeaff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80aacc0 con 0x7f6ff80c8e00 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffeaff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80acde0 con 0x7f6ff80c5730 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60242/0 (socket says 172.21.15.78:60242) 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 -- 172.21.15.78:0/3472412625 learned_addr learned my addr 172.21.15.78:0/3472412625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffc89b640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff81eba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ff7fff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff80c8e00 0x7f6ff81ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 -- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 msgr2=0x7f6ff81eba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff81eba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 -- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff80c8e00 msgr2=0x7f6ff81ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff80c8e00 0x7f6ff81ea860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 -- 172.21.15.78:0/3472412625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0094d0 con 0x7f6ff8152700 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffc89b640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff81eba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ff7fff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff80c8e00 0x7f6ff81ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.314+0000 7f6ffd09c640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80ab430 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000c7a0 tx=0x7f6ff000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.315+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0003b70 con 0x7f6ff8152700 2024-02-19T14:55:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.315+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff0003cf0 con 0x7f6ff8152700 2024-02-19T14:55:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.315+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0003ea0 con 0x7f6ff8152700 2024-02-19T14:55:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.315+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff80acb00 con 0x7f6ff8152700 2024-02-19T14:55:15.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.315+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff81eb380 con 0x7f6ff8152700 2024-02-19T14:55:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.316+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8074990 con 0x7f6ff8152700 2024-02-19T14:55:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.316+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6ff000aa90 con 0x7f6ff8152700 2024-02-19T14:55:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.317+0000 7f6ff57fa640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6fdc06b810 0x7f6fdc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.317+0000 7f6ffc89b640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6fdc06b810 0x7f6fdc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.317+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6ff008a1b0 con 0x7f6ff8152700 2024-02-19T14:55:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.317+0000 7f6ffc89b640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6fdc06b810 0x7f6fdc06dcd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6fec036000 tx=0x7f6fec03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:15.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.321+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ff0054ad0 con 0x7f6ff8152700 2024-02-19T14:55:15.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.563+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6ff80c8770 con 0x7f6ff8152700 2024-02-19T14:55:15.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.565+0000 7f6ff57fa640 1 -- 172.21.15.78:0/3472412625 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6ff0058190 con 0x7f6ff8152700 2024-02-19T14:55:15.568 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:55:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:55:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:55:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:55:15.568 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:55:15.568 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6fdc06b810 msgr2=0x7f6fdc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6fdc06b810 0x7f6fdc06dcd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6fec036000 tx=0x7f6fec03b040 comp rx=0 tx=0).stop 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 msgr2=0x7f6ff80ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80ab430 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000c7a0 tx=0x7f6ff000cc60 comp rx=0 tx=0).stop 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 shutdown_connections 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6fdc06b810 0x7f6fdc06dcd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8152700 0x7f6ff80ab430 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff80c8e00 0x7f6ff81ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 --2- 172.21.15.78:0/3472412625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff80c5730 0x7f6ff81eba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 >> 172.21.15.78:0/3472412625 conn(0x7f6ff80cac90 msgr2=0x7f6ff814fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:15.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 shutdown_connections 2024-02-19T14:55:15.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.567+0000 7f6ffeaff640 1 -- 172.21.15.78:0/3472412625 wait complete. 2024-02-19T14:55:15.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:15.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.678+0000 7f9549f0c640 1 Processor -- start 2024-02-19T14:55:15.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.678+0000 7f9549f0c640 1 -- start start 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9549f0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95440719d0 0x7f9544071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9549f0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95440c15b0 0x7f95440c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9549f0c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95440c1120 con 0x7f954414ece0 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9549f0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95440c6080 con 0x7f95440c15b0 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9549f0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95440c57e0 con 0x7f95440719d0 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f95437fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f954414ece0 0x7f95440c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48172/0 (socket says 172.21.15.78:48172) 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f95437fe640 1 -- 172.21.15.78:0/4209177955 learned_addr learned my addr 172.21.15.78:0/4209177955 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9543fff640 1 --2- 172.21.15.78:0/4209177955 >> v2:172.21.15.78:3300/0 conn(0x7f95440c15b0 0x7f95440c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f953affd640 1 --2- 172.21.15.78:0/4209177955 >> v2:172.21.15.150:3300/0 conn(0x7f95440719d0 0x7f9544071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9543fff640 1 -- 172.21.15.78:0/4209177955 >> v2:172.21.15.150:3300/0 conn(0x7f95440719d0 msgr2=0x7f9544071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9543fff640 1 --2- 172.21.15.78:0/4209177955 >> v2:172.21.15.150:3300/0 conn(0x7f95440719d0 0x7f9544071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9543fff640 1 -- 172.21.15.78:0/4209177955 >> v1:172.21.15.78:6789/0 conn(0x7f954414ece0 legacy=0x7f95440c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9543fff640 1 -- 172.21.15.78:0/4209177955 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440c5ac0 con 0x7f95440c15b0 2024-02-19T14:55:15.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.679+0000 7f9542ffd640 1 -- 172.21.15.78:0/4209177955 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502202050 0 0) 0x7f95440c1120 con 0x7f954414ece0 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.680+0000 7f9543fff640 1 --2- 172.21.15.78:0/4209177955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95440c15b0 0x7f95440c1a00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9544155b80 tx=0x7f953c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f366cc917669625d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.680+0000 7f9542ffd640 1 -- 172.21.15.78:0/4209177955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c02cc80 con 0x7f95440c15b0 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.680+0000 7f9542ffd640 1 -- 172.21.15.78:0/4209177955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c02cde0 con 0x7f95440c15b0 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.680+0000 7f9542ffd640 1 -- 172.21.15.78:0/4209177955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c034660 con 0x7f95440c15b0 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 -- 172.21.15.78:0/4209177955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95440c15b0 msgr2=0x7f95440c1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 --2- 172.21.15.78:0/4209177955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95440c15b0 0x7f95440c1a00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9544155b80 tx=0x7f953c02bed0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 -- 172.21.15.78:0/4209177955 shutdown_connections 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 --2- 172.21.15.78:0/4209177955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95440c15b0 0x7f95440c1a00 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 --2- 172.21.15.78:0/4209177955 >> v2:172.21.15.150:3300/0 conn(0x7f95440719d0 0x7f9544071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 -- 172.21.15.78:0/4209177955 >> 172.21.15.78:0/4209177955 conn(0x7f954406b270 msgr2=0x7f954406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:15.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 -- 172.21.15.78:0/4209177955 shutdown_connections 2024-02-19T14:55:15.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.681+0000 7f9549f0c640 1 -- 172.21.15.78:0/4209177955 wait complete. 2024-02-19T14:55:15.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.682+0000 7f9549f0c640 1 Processor -- start 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.682+0000 7f9549f0c640 1 -- start start 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9549f0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95440719d0 0x7f95441ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9549f0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95440c15b0 0x7f9544151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9549f0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 0x7f95441eab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9549f0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95441ea1c0 con 0x7f954414ece0 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9549f0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95441ea700 con 0x7f95440c15b0 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9549f0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95441edcf0 con 0x7f95440719d0 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 0x7f95441eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 0x7f95441eab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60262/0 (socket says 172.21.15.78:60262) 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 -- 172.21.15.78:0/1287236139 learned_addr learned my addr 172.21.15.78:0/1287236139 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f95437fe640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95440719d0 0x7f95441ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 -- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95440719d0 msgr2=0x7f95441ed7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f953affd640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95440c15b0 0x7f9544151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95440719d0 0x7f95441ed7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 -- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95440c15b0 msgr2=0x7f9544151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95440c15b0 0x7f9544151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.683+0000 7f9543fff640 1 -- 172.21.15.78:0/1287236139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c009a70 con 0x7f954414ece0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.684+0000 7f9543fff640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 0x7f95441eab90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f953c02c4e0 tx=0x7f953c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.684+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c040650 con 0x7f954414ece0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.684+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c0407b0 con 0x7f954414ece0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.684+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9544152240 con 0x7f954414ece0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.684+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c040b60 con 0x7f954414ece0 2024-02-19T14:55:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.684+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95440bfd70 con 0x7f954414ece0 2024-02-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.685+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95441ea8b0 con 0x7f954414ece0 2024-02-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.685+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f953c040cc0 con 0x7f954414ece0 2024-02-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.685+0000 7f9540ff9640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f952406b810 0x7f952406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.686+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f953c0ad370 con 0x7f954414ece0 2024-02-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.686+0000 7f95437fe640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f952406b810 0x7f952406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:15.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.686+0000 7f95437fe640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f952406b810 0x7f952406dcd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9534000f60 tx=0x7f9534007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:15.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.690+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f95441ea8b0 con 0x7f954414ece0 2024-02-19T14:55:15.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.952+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95441ea8b0 con 0x7f954414ece0 2024-02-19T14:55:15.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.952+0000 7f9540ff9640 1 -- 172.21.15.78:0/1287236139 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f95441ea8b0 con 0x7f954414ece0 2024-02-19T14:55:15.955 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:15.955 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5672,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:13.179911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6836/3610724432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3610724432},{"type":"v1","addr":"172.21.15.78:6836","nonce":3610724432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:15.955 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 73 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f952406b810 msgr2=0x7f952406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f952406b810 0x7f952406dcd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9534000f60 tx=0x7f9534007c00 comp rx=0 tx=0).stop 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 msgr2=0x7f95441eab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 0x7f95441eab90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f953c02c4e0 tx=0x7f953c004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 shutdown_connections 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f952406b810 0x7f952406dcd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f954414ece0 0x7f95441eab90 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95440c15b0 0x7f9544151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 --2- 172.21.15.78:0/1287236139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95440719d0 0x7f95441ed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:15.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 >> 172.21.15.78:0/1287236139 conn(0x7f954406b270 msgr2=0x7f95440c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:15.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 shutdown_connections 2024-02-19T14:55:15.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:15.954+0000 7f9549f0c640 1 -- 172.21.15.78:0/1287236139 wait complete. 2024-02-19T14:55:16.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:55:11.163509+0000', 'modified': '2024-02-19T14:55:13.179911+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6948}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6948': {'gid': 6948, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6836/3610724432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3610724432}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3610724432}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:16.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:16.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T14:55:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.473+0000 7f70437d0640 1 Processor -- start 2024-02-19T14:55:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.473+0000 7f70437d0640 1 -- start start 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.473+0000 7f70437d0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f703c0add70 0x7f703c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f70437d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f70437d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c0ae670 con 0x7f703c0c2f60 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f70437d0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703c0716b0 con 0x7f703c0add70 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f70437d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0c33d0 con 0x7f703c0705b0 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f703c0add70 0x7f703c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f703c0add70 0x7f703c0c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60274/0 (socket says 172.21.15.78:60274) 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 -- 172.21.15.78:0/2825741150 learned_addr learned my addr 172.21.15.78:0/2825741150 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f7041d6d640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 -- 172.21.15.78:0/2825741150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 msgr2=0x7f703c070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 -- 172.21.15.78:0/2825741150 >> v1:172.21.15.150:6789/0 conn(0x7f703c0c2f60 legacy=0x7f703c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f704156c640 1 -- 172.21.15.78:0/2825741150 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c153780 con 0x7f703c0add70 2024-02-19T14:55:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.474+0000 7f7041d6d640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f704156c640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c0c4980 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7038009870 tx=0x7f703802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75247ff48a08da94 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f7033fff640 1 -- 172.21.15.78:0/2825741150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031640 con 0x7f703c0add70 2024-02-19T14:55:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f7033fff640 1 -- 172.21.15.78:0/2825741150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70380317a0 con 0x7f703c0add70 2024-02-19T14:55:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f7033fff640 1 -- 172.21.15.78:0/2825741150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031b30 con 0x7f703c0add70 2024-02-19T14:55:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f70437d0640 1 -- 172.21.15.78:0/2825741150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 msgr2=0x7f703c0c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f70437d0640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c0c4980 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7038009870 tx=0x7f703802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f70437d0640 1 -- 172.21.15.78:0/2825741150 shutdown_connections 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f70437d0640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.475+0000 7f70437d0640 1 --2- 172.21.15.78:0/2825741150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c0c4980 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.476+0000 7f70437d0640 1 -- 172.21.15.78:0/2825741150 >> 172.21.15.78:0/2825741150 conn(0x7f703c0c85e0 msgr2=0x7f703c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.476+0000 7f70437d0640 1 -- 172.21.15.78:0/2825741150 shutdown_connections 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.476+0000 7f70437d0640 1 -- 172.21.15.78:0/2825741150 wait complete. 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 Processor -- start 2024-02-19T14:55:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 -- start start 2024-02-19T14:55:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c0c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703c0c2f60 0x7f703c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c1eb180 con 0x7f703c0add70 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c06bf90 con 0x7f703c0c2f60 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f70437d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c06c270 con 0x7f703c0705b0 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.477+0000 7f7040d6b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c153fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60290/0 (socket says 172.21.15.78:60290) 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 -- 172.21.15.78:0/135489961 learned_addr learned my addr 172.21.15.78:0/135489961 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f704156c640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c0c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7041d6d640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703c0c2f60 0x7f703c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 -- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 msgr2=0x7f703c0c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c0c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 -- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703c0c2f60 msgr2=0x7f703c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703c0c2f60 0x7f703c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 -- 172.21.15.78:0/135489961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380094d0 con 0x7f703c0add70 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7041d6d640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703c0c2f60 0x7f703c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7040d6b640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c153fb0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c400 tx=0x7f702c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7031ffb640 1 -- 172.21.15.78:0/135489961 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c0052d0 con 0x7f703c0add70 2024-02-19T14:55:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7031ffb640 1 -- 172.21.15.78:0/135489961 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c005450 con 0x7f703c0add70 2024-02-19T14:55:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f7031ffb640 1 -- 172.21.15.78:0/135489961 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c007260 con 0x7f703c0add70 2024-02-19T14:55:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c06c550 con 0x7f703c0add70 2024-02-19T14:55:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.478+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c0c20a0 con 0x7f703c0add70 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.480+0000 7f7031ffb640 1 -- 172.21.15.78:0/135489961 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f702c0073c0 con 0x7f703c0add70 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.481+0000 7f7031ffb640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f702006b810 0x7f702006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.481+0000 7f7031ffb640 1 -- 172.21.15.78:0/135489961 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f702c089640 con 0x7f703c0add70 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.481+0000 7f704156c640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f702006b810 0x7f702006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.481+0000 7f704156c640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f702006b810 0x7f702006dcd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7038036000 tx=0x7f703803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.481+0000 7f70463af740 1 Processor -- start 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.481+0000 7f70463af740 1 -- start start 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c265018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264c6ae0 0x559c26500b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f70463af740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c264c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f70463af740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c26480cd0 con 0x559c265038b0 2024-02-19T14:55:16.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c26433f20 con 0x559c26500740 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c26431450 con 0x559c264c6ae0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c264c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c264c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60296/0 (socket says 172.21.15.78:60296) 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 -- 172.21.15.78:0/1374657292 learned_addr learned my addr 172.21.15.78:0/1374657292 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f704156c640 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c265018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 -- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264c6ae0 msgr2=0x559c26500b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264c6ae0 0x559c26500b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 -- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 msgr2=0x559c265018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c265018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.482+0000 7f7041d6d640 1 -- 172.21.15.78:0/1374657292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c263e1f20 con 0x559c265038b0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f7041d6d640 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c264c5860 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7034009b30 tx=0x7f7034004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378cc0f9957030e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f701effd640 1 -- 172.21.15.78:0/1374657292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034013780 con 0x559c265038b0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f701effd640 1 -- 172.21.15.78:0/1374657292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703400f050 con 0x559c265038b0 2024-02-19T14:55:16.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f701effd640 1 -- 172.21.15.78:0/1374657292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034013b00 con 0x559c265038b0 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 -- 172.21.15.78:0/1374657292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 msgr2=0x559c264c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c264c5860 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7034009b30 tx=0x7f7034004040 comp rx=0 tx=0).stop 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 -- 172.21.15.78:0/1374657292 shutdown_connections 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c264c5860 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264c6ae0 0x559c26500b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 --2- 172.21.15.78:0/1374657292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c265018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 -- 172.21.15.78:0/1374657292 >> 172.21.15.78:0/1374657292 conn(0x559c264c2120 msgr2=0x559c264e6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 -- 172.21.15.78:0/1374657292 shutdown_connections 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.483+0000 7f70463af740 1 -- 172.21.15.78:0/1374657292 wait complete. 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.484+0000 7f70463af740 1 -- start start 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.484+0000 7f70463af740 1 Processor -- start 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 0x559c264c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c265038b0 0x559c265073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264fa900 0x559c264e3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2642cc30 con 0x559c26500740 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c263d1d30 con 0x559c265038b0 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c26100310 con 0x559c264fa900 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f70463af740 10 client.0 _ll_register_callbacks cb 0x559c263d9d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 0x559c264c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 0x559c264c4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60298/0 (socket says 172.21.15.78:60298) 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 -- 172.21.15.78:0/3260312319 learned_addr learned my addr 172.21.15.78:0/3260312319 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f7041d6d640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264fa900 0x559c264e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c265038b0 0x559c265073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 -- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264fa900 msgr2=0x559c264e3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264fa900 0x559c264e3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 -- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c265038b0 msgr2=0x559c265073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c265038b0 0x559c265073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 -- 172.21.15.78:0/3260312319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7034009730 con 0x559c26500740 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f7041d6d640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264fa900 0x559c264e3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c265038b0 0x559c265073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f704156c640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 0x559c264c4c20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f703802bf00 tx=0x7f7038004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d13eb2a19191944 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.485+0000 7f7017fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7017fff640 1 -- 172.21.15.78:0/3260312319 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703804a070 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7017fff640 1 -- 172.21.15.78:0/3260312319 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f703803f070 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c2612c6c0 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7017fff640 1 -- 172.21.15.78:0/3260312319 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f703803adf0 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 10 client.7080 fetch_fsmap learned FSMap version 73 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x559c2642b2b0 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7017fff640 1 -- 172.21.15.78:0/3260312319 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c2612c6c0 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7017fff640 1 -- 172.21.15.78:0/3260312319 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f703804f470 con 0x559c26500740 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 10 client.7080 fetch_fsmap finished waiting for FSMap version 73 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 10 client.7080 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 20 client.7080 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c26506680 0x559c26506a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 4 client.7080 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 --> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c2617d270 con 0x559c26506680 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c26506680 0x559c26506a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.486+0000 7f7017fff640 10 client.7080 ms_handle_connect on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.487+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c26506680 0x559c26506a60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=772648948836bae3 server_cookie=a308fe883adbdfbc in_seq=0 out_seq=0 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.487+0000 7f7017fff640 1 -- 172.21.15.78:0/3260312319 <== mds.0 v2:172.21.15.78:6834/3610724432 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559c2617d270 con 0x559c26506680 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.487+0000 7f7017fff640 10 client.7080 handle_command_reply: tid=0 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 client.7080 shutdown 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 msgr2=0x559c264c4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 0x559c264c4c20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f703802bf00 tx=0x7f7038004730 comp rx=0 tx=0).stop 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 shutdown_connections 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c26506680 0x559c26506a60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c264fa900 0x559c264e3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c265038b0 0x559c265073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f7017fff640 0 client.7080 ms_handle_reset on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:16.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 --2- 172.21.15.78:0/3260312319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c26500740 0x559c264c4c20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 >> 172.21.15.78:0/3260312319 conn(0x559c264c6ae0 msgr2=0x559c264e6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 shutdown_connections 2024-02-19T14:55:16.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 1 -- 172.21.15.78:0/3260312319 wait complete. 2024-02-19T14:55:16.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.488+0000 7f70463af740 20 client.7080 trim_cache size 0 max 16384 2024-02-19T14:55:16.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.504+0000 7f70463af740 1 Processor -- start 2024-02-19T14:55:16.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.504+0000 7f70463af740 1 -- start start 2024-02-19T14:55:16.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f70463af740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 0x559c26501e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c264c0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c265038b0 0x559c264c1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f70463af740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c263b2490 con 0x559c264fa900 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c264c2f50 con 0x559c26500740 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c264e4610 con 0x559c265038b0 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 0x559c26501e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 0x559c26501e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60308/0 (socket says 172.21.15.78:60308) 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 -- 172.21.15.78:0/3647238407 learned_addr learned my addr 172.21.15.78:0/3647238407 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f7041d6d640 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c265038b0 0x559c264c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c264c0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 -- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c265038b0 msgr2=0x559c264c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c265038b0 0x559c264c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 -- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 msgr2=0x559c264c0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c264c0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f704156c640 1 -- 172.21.15.78:0/3647238407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c263c7a20 con 0x559c264fa900 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.505+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c264c0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:16.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f704156c640 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 0x559c26501e40 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f7038009990 tx=0x7f7038047070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03c418b46f91d62 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f701e7fc640 1 -- 172.21.15.78:0/3647238407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703804f470 con 0x559c264fa900 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f701e7fc640 1 -- 172.21.15.78:0/3647238407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703804f5d0 con 0x559c264fa900 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f701e7fc640 1 -- 172.21.15.78:0/3647238407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703804fb30 con 0x559c264fa900 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 -- 172.21.15.78:0/3647238407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 msgr2=0x559c26501e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 0x559c26501e40 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f7038009990 tx=0x7f7038047070 comp rx=0 tx=0).stop 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 -- 172.21.15.78:0/3647238407 shutdown_connections 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c265038b0 0x559c264c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c26500740 0x559c264c0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 --2- 172.21.15.78:0/3647238407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c264fa900 0x559c26501e40 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 -- 172.21.15.78:0/3647238407 >> 172.21.15.78:0/3647238407 conn(0x559c264c6ae0 msgr2=0x559c264e34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 -- 172.21.15.78:0/3647238407 shutdown_connections 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.506+0000 7f70463af740 1 -- 172.21.15.78:0/3647238407 wait complete. 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.507+0000 7f70463af740 1 -- start start 2024-02-19T14:55:16.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.507+0000 7f70463af740 1 Processor -- start 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c264fa900 0x559c264c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c26500740 0x559c264f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c2656e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c264ffa50 con 0x559c265038b0 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c264f8320 con 0x559c264fa900 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c26524fd0 con 0x559c26500740 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f70463af740 10 client.0 _ll_register_callbacks cb 0x7f7038031440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c2656e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c2656e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60310/0 (socket says 172.21.15.78:60310) 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 -- 172.21.15.78:0/3331993128 learned_addr learned my addr 172.21.15.78:0/3331993128 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7040d6b640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c26500740 0x559c264f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f704156c640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c264fa900 0x559c264c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 -- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c26500740 msgr2=0x559c264f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c26500740 0x559c264f7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 -- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c264fa900 msgr2=0x559c264c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c264fa900 0x559c264c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.508+0000 7f7041d6d640 1 -- 172.21.15.78:0/3331993128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c2642b2b0 con 0x559c265038b0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7041d6d640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c2656e430 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x559c265013f0 tx=0x7f7034012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa54d86ec74b3000 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7017fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7017fff640 1 -- 172.21.15.78:0/3331993128 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034023040 con 0x559c265038b0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7017fff640 1 -- 172.21.15.78:0/3331993128 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7034003e00 con 0x559c265038b0 2024-02-19T14:55:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c26169610 con 0x559c265038b0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7017fff640 1 -- 172.21.15.78:0/3331993128 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7034003af0 con 0x559c265038b0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 10 client.7086 fetch_fsmap learned FSMap version 73 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x559c263ed780 con 0x559c265038b0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7017fff640 1 -- 172.21.15.78:0/3331993128 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c26169610 con 0x559c265038b0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f7017fff640 1 -- 172.21.15.78:0/3331993128 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7034026030 con 0x559c265038b0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 10 client.7086 fetch_fsmap finished waiting for FSMap version 73 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 10 client.7086 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 20 client.7086 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c264c30e0 0x559c264c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 4 client.7086 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 --> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559c260d85e0 con 0x559c264c30e0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.509+0000 7f704156c640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c264c30e0 0x559c264c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f7017fff640 10 client.7086 ms_handle_connect on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f704156c640 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c264c30e0 0x559c264c34c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98e8bf0c4d0d0781 server_cookie=aeea869512abe6ae in_seq=0 out_seq=0 2024-02-19T14:55:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f7017fff640 1 -- 172.21.15.78:0/3331993128 <== mds.0 v2:172.21.15.78:6834/3610724432 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559c260d85e0 con 0x559c264c30e0 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f7017fff640 10 client.7086 handle_command_reply: tid=0 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 client.7086 shutdown 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 msgr2=0x559c2656e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c2656e430 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x559c265013f0 tx=0x7f7034012000 comp rx=0 tx=0).stop 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 shutdown_connections 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x559c264c30e0 0x559c264c34c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c265038b0 0x559c2656e430 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559c26500740 0x559c264f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 --2- 172.21.15.78:0/3331993128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559c264fa900 0x559c264c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f7017fff640 0 client.7086 ms_handle_reset on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 >> 172.21.15.78:0/3331993128 conn(0x559c264c6ae0 msgr2=0x559c26501e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 shutdown_connections 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 1 -- 172.21.15.78:0/3331993128 wait complete. 2024-02-19T14:55:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.510+0000 7f70463af740 20 client.7086 trim_cache size 0 max 16384 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 6948, 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.6941784159999997, 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 73, 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 98, 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 98, 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 9.8093545609999993 2024-02-19T14:55:16.519 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f702006b810 msgr2=0x7f702006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f702006b810 0x7f702006dcd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7038036000 tx=0x7f703803c040 comp rx=0 tx=0).stop 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 msgr2=0x7f703c153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c153fb0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c400 tx=0x7f702c00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 shutdown_connections 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f702006b810 0x7f702006dcd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f703c0add70 0x7f703c153fb0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703c0c2f60 0x7f703c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 --2- 172.21.15.78:0/135489961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f703c0705b0 0x7f703c0c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 >> 172.21.15.78:0/135489961 conn(0x7f703c0c85e0 msgr2=0x7f703c0a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 shutdown_connections 2024-02-19T14:55:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.511+0000 7f70437d0640 1 -- 172.21.15.78:0/135489961 wait complete. 2024-02-19T14:55:16.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:16.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.622+0000 7f2364da1640 1 Processor -- start 2024-02-19T14:55:16.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.622+0000 7f2364da1640 1 -- start start 2024-02-19T14:55:16.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f2364da1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2360072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f2364da1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2360152680 0x7f23600c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f2364da1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600c5970 con 0x7f236006ea50 2024-02-19T14:55:16.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f2364da1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2360152af0 con 0x7f2360152680 2024-02-19T14:55:16.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f2364da1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23600c5750 con 0x7f236014f1a0 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f235dd9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f236006ea50 0x7f236006fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48184/0 (socket says 172.21.15.78:48184) 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.623+0000 7f235dd9b640 1 -- 172.21.15.78:0/1689480893 learned_addr learned my addr 172.21.15.78:0/1689480893 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235e59c640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2360072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236337033 0 0) 0x7f23600c5970 con 0x7f236006ea50 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2348003520 con 0x7f236006ea50 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235ed9d640 1 --2- 172.21.15.78:0/1689480893 >> v2:172.21.15.150:3300/0 conn(0x7f2360152680 0x7f23600c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1760619172 0 0) 0x7f2348003520 con 0x7f236006ea50 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 msgr2=0x7f2360072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2360072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 >> v2:172.21.15.150:3300/0 conn(0x7f2360152680 msgr2=0x7f23600c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 --2- 172.21.15.78:0/1689480893 >> v2:172.21.15.150:3300/0 conn(0x7f2360152680 0x7f23600c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23600702d0 con 0x7f236006ea50 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2354001670 con 0x7f236006ea50 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 >> v1:172.21.15.78:6789/0 conn(0x7f236006ea50 legacy=0x7f236006fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2348003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2348004d00 0x7f2348005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23600c5750 con 0x7f2348003f40 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234800ba30 con 0x7f2348004d00 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234800bbf0 con 0x7f236014f1a0 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235e59c640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2348003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235ed9d640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2348004d00 0x7f2348005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235dd9b640 1 -- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 msgr2=0x7f2348003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.624+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2348003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.625+0000 7f235dd9b640 1 -- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2348004d00 msgr2=0x7f2348005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.625+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2348004d00 0x7f2348005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.625+0000 7f235dd9b640 1 -- 172.21.15.78:0/1689480893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2348003520 con 0x7f2348003f40 2024-02-19T14:55:16.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.625+0000 7f235e59c640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f236014f1a0 0x7f2348003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:16.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23480047c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f235400c0e0 tx=0x7f23480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f785c50303e5e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354005ad0 con 0x7f2348003f40 2024-02-19T14:55:16.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2354005c50 con 0x7f2348003f40 2024-02-19T14:55:16.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f235d59a640 1 -- 172.21.15.78:0/1689480893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235402c810 con 0x7f2348003f40 2024-02-19T14:55:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f2364da1640 1 -- 172.21.15.78:0/1689480893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 msgr2=0x7f23480047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f2364da1640 1 --2- 172.21.15.78:0/1689480893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23480047c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f235400c0e0 tx=0x7f23480057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f235ed9d640 1 -- 172.21.15.78:0/1689480893 reap_dead start 2024-02-19T14:55:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f2364da1640 1 -- 172.21.15.78:0/1689480893 shutdown_connections 2024-02-19T14:55:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f2364da1640 1 -- 172.21.15.78:0/1689480893 >> 172.21.15.78:0/1689480893 conn(0x7f236006c170 msgr2=0x7f236006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f2364da1640 1 -- 172.21.15.78:0/1689480893 shutdown_connections 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.626+0000 7f2364da1640 1 -- 172.21.15.78:0/1689480893 wait complete. 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.628+0000 7f2364da1640 1 Processor -- start 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.628+0000 7f2364da1640 1 -- start start 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23601e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23601eb530 0x7f23601505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2360154400 0x7f236014a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601e8530 con 0x7f2348003f40 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23600c45d0 con 0x7f23601eb530 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23601eaae0 con 0x7f2360154400 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23601e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23601e7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60336/0 (socket says 172.21.15.78:60336) 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 -- 172.21.15.78:0/1593458935 learned_addr learned my addr 172.21.15.78:0/1593458935 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23601eb530 0x7f23601505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 -- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2360154400 msgr2=0x7f236014a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2360154400 0x7f236014a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 -- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23601eb530 msgr2=0x7f23601505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23601eb530 0x7f23601505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 -- 172.21.15.78:0/1593458935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2360153050 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f235e59c640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23601e7ff0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2350011670 tx=0x7f2350011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000e040 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000f040 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2360153330 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350003bb0 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.629+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23601ead30 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.630+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2360074990 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.631+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2350003d10 con 0x7f2348003f40 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.631+0000 7f234effd640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f234006b840 0x7f234006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.631+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f235008c6c0 con 0x7f2348003f40 2024-02-19T14:55:16.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.631+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f234006b840 0x7f234006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.632+0000 7f235dd9b640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f234006b840 0x7f234006dd00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2354034000 tx=0x7f2354007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.636+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2350056fe0 con 0x7f2348003f40 2024-02-19T14:55:16.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.852+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f236006e830 con 0x7f2348003f40 2024-02-19T14:55:16.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.852+0000 7f234effd640 1 -- 172.21.15.78:0/1593458935 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f235005a6a0 con 0x7f2348003f40 2024-02-19T14:55:16.855 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:16.856 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5672,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:13.179911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6836/3610724432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3610724432},{"type":"v1","addr":"172.21.15.78:6836","nonce":3610724432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:16.856 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 73 2024-02-19T14:55:16.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f234006b840 msgr2=0x7f234006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f234006b840 0x7f234006dd00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2354034000 tx=0x7f2354007000 comp rx=0 tx=0).stop 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 msgr2=0x7f23601e7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23601e7ff0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2350011670 tx=0x7f2350011b30 comp rx=0 tx=0).stop 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 shutdown_connections 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f234006b840 0x7f234006dd00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2360154400 0x7f236014a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23601eb530 0x7f23601505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 --2- 172.21.15.78:0/1593458935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2348003f40 0x7f23601e7ff0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 >> 172.21.15.78:0/1593458935 conn(0x7f236006c170 msgr2=0x7f236006fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 shutdown_connections 2024-02-19T14:55:16.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.855+0000 7f2364da1640 1 -- 172.21.15.78:0/1593458935 wait complete. 2024-02-19T14:55:16.870 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:11.163509+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6948": { "addr": "172.21.15.78:6836/3610724432", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3610724432, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3610724432, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6948, "incarnation": 71, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-02-19T14:55:13.179911+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6948 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/408965047", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 408965047, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 408965047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5672, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/829046094", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 829046094, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 829046094, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6310, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/2386100646", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2386100646, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2386100646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6945, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:16.871 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:55:16.871 INFO:tasks.cephfs.test_failover:current = [{'gid': 6948, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6836/3610724432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3610724432}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3610724432}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6310, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/829046094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 829046094}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 829046094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}, {'gid': 6945, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/2386100646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2386100646}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2386100646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}, {'gid': 5672, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/408965047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 408965047}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 408965047}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}] 2024-02-19T14:55:16.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-19T14:55:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.971+0000 7ff1d08a4640 1 Processor -- start 2024-02-19T14:55:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.971+0000 7ff1d08a4640 1 -- start start 2024-02-19T14:55:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.972+0000 7ff1d08a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff1cc14d210 0x7ff1cc0c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.972+0000 7ff1d08a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1cc0719e0 0x7ff1cc0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.972+0000 7ff1d08a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc06ff90 con 0x7ff1cc0c3d90 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.972+0000 7ff1d08a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc070270 con 0x7ff1cc0719e0 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.972+0000 7ff1d08a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0c3970 con 0x7ff1cc14d210 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1c9d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1cc0719e0 0x7ff1cc0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1cad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff1cc0c3d90 0x7ff1cc152ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48194/0 (socket says 172.21.15.78:48194) 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1cad9d640 1 -- 172.21.15.78:0/503074080 learned_addr learned my addr 172.21.15.78:0/503074080 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1c9d9b640 1 -- 172.21.15.78:0/503074080 >> v2:172.21.15.150:3300/0 conn(0x7ff1cc14d210 msgr2=0x7ff1cc0c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1c9d9b640 1 --2- 172.21.15.78:0/503074080 >> v2:172.21.15.150:3300/0 conn(0x7ff1cc14d210 0x7ff1cc0c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1c9d9b640 1 -- 172.21.15.78:0/503074080 >> v1:172.21.15.78:6789/0 conn(0x7ff1cc0c3d90 legacy=0x7ff1cc152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1c9d9b640 1 -- 172.21.15.78:0/503074080 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc156fc0 con 0x7ff1cc0719e0 2024-02-19T14:55:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.973+0000 7ff1c959a640 1 -- 172.21.15.78:0/503074080 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505434875 0 0) 0x7ff1cc06ff90 con 0x7ff1cc0c3d90 2024-02-19T14:55:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1c9d9b640 1 --2- 172.21.15.78:0/503074080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc0719e0 0x7ff1cc0c64d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0c5de0 tx=0x7ff1bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82be0799090cfe8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1c959a640 1 -- 172.21.15.78:0/503074080 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc02e070 con 0x7ff1cc0719e0 2024-02-19T14:55:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1c959a640 1 -- 172.21.15.78:0/503074080 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0315a0 con 0x7ff1cc0719e0 2024-02-19T14:55:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1c959a640 1 -- 172.21.15.78:0/503074080 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc031930 con 0x7ff1cc0719e0 2024-02-19T14:55:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 -- 172.21.15.78:0/503074080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc0719e0 msgr2=0x7ff1cc0c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/503074080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc0719e0 0x7ff1cc0c64d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0c5de0 tx=0x7ff1bc02b840 comp rx=0 tx=0).stop 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 -- 172.21.15.78:0/503074080 shutdown_connections 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/503074080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc0719e0 0x7ff1cc0c64d0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/503074080 >> v2:172.21.15.150:3300/0 conn(0x7ff1cc14d210 0x7ff1cc0c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 -- 172.21.15.78:0/503074080 >> 172.21.15.78:0/503074080 conn(0x7ff1cc0c5050 msgr2=0x7ff1cc0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 -- 172.21.15.78:0/503074080 shutdown_connections 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.974+0000 7ff1d08a4640 1 -- 172.21.15.78:0/503074080 wait complete. 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 Processor -- start 2024-02-19T14:55:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 -- start start 2024-02-19T14:55:16.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1cc0719e0 0x7ff1cc153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc0c3d90 0x7ff1cc156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 0x7ff1cc0ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1ebd10 con 0x7ff1cc14d210 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1e94d0 con 0x7ff1cc0c3d90 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1d08a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1ebef0 con 0x7ff1cc0719e0 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1cad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 0x7ff1cc0ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1cad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 0x7ff1cc0ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60346/0 (socket says 172.21.15.78:60346) 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.976+0000 7ff1cad9d640 1 -- 172.21.15.78:0/2754150739 learned_addr learned my addr 172.21.15.78:0/2754150739 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1ca59c640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1cc0719e0 0x7ff1cc153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1c9d9b640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc0c3d90 0x7ff1cc156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1cad9d640 1 -- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1cc0719e0 msgr2=0x7ff1cc153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1cad9d640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1cc0719e0 0x7ff1cc153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1cad9d640 1 -- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc0c3d90 msgr2=0x7ff1cc156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1cad9d640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc0c3d90 0x7ff1cc156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1cad9d640 1 -- 172.21.15.78:0/2754150739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc009400 con 0x7ff1cc14d210 2024-02-19T14:55:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1ca59c640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1cc0719e0 0x7ff1cc153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1cad9d640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 0x7ff1cc0ad2a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400c780 tx=0x7ff1c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400ce90 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c400a640 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc0c3660 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400aa70 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.977+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1ec4b0 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.978+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc1ec6a0 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.979+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff1c400abd0 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.979+0000 7ff1baffd640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1ac06b840 0x7ff1ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.979+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff1c4089b30 con 0x7ff1cc14d210 2024-02-19T14:55:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.980+0000 7ff1ca59c640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1ac06b840 0x7ff1ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.980+0000 7ff1ca59c640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1ac06b840 0x7ff1ac06dd00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000a870 tx=0x7ff1c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:16.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:16.984+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff1c4054450 con 0x7ff1cc14d210 2024-02-19T14:55:17.093 INFO:tasks.ceph.mds.a.smithi078.stderr:2024-02-19T14:55:17.091+0000 7f0578a1b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.322+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7ff1cc074990 con 0x7ff1cc14d210 2024-02-19T14:55:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.326+0000 7ff1baffd640 1 -- 172.21.15.78:0/2754150739 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v32)=0 v32) v1 ==== 116+0+0 (secure 0 0 0) 0x7ff1c4018020 con 0x7ff1cc14d210 2024-02-19T14:55:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1ac06b840 msgr2=0x7ff1ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1ac06b840 0x7ff1ac06dd00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000a870 tx=0x7ff1c0008040 comp rx=0 tx=0).stop 2024-02-19T14:55:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 msgr2=0x7ff1cc0ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 0x7ff1cc0ad2a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400c780 tx=0x7ff1c400cc40 comp rx=0 tx=0).stop 2024-02-19T14:55:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 shutdown_connections 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1ac06b840 0x7ff1ac06dd00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1cc14d210 0x7ff1cc0ad2a0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc0c3d90 0x7ff1cc156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 --2- 172.21.15.78:0/2754150739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1cc0719e0 0x7ff1cc153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 >> 172.21.15.78:0/2754150739 conn(0x7ff1cc0c5050 msgr2=0x7ff1cc14b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 shutdown_connections 2024-02-19T14:55:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.328+0000 7ff1d08a4640 1 -- 172.21.15.78:0/2754150739 wait complete. 2024-02-19T14:55:17.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.443+0000 7f9572c37640 1 Processor -- start 2024-02-19T14:55:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.443+0000 7f9572c37640 1 -- start start 2024-02-19T14:55:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f9572c37640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c0c8e00 0x7f956c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f9572c37640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f956c06d910 0x7f956c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f9572c37640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c06cd20 con 0x7f956c152700 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f9572c37640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956c1551f0 con 0x7f956c06d910 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f9572c37640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c155490 con 0x7f956c0c8e00 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f956bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c0c8e00 0x7f956c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f95711d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f956c06d910 0x7f956c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f95709d3640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f956c152700 0x7f956c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58992/0 (socket says 172.21.15.78:58992) 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.444+0000 7f95709d3640 1 -- 172.21.15.78:0/694499873 learned_addr learned my addr 172.21.15.78:0/694499873 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.445+0000 7f95711d4640 1 -- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c0c8e00 msgr2=0x7f956c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.445+0000 7f95711d4640 1 --2- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c0c8e00 0x7f956c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.445+0000 7f95711d4640 1 -- 172.21.15.78:0/694499873 >> v1:172.21.15.150:6789/0 conn(0x7f956c152700 legacy=0x7f956c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.445+0000 7f95711d4640 1 -- 172.21.15.78:0/694499873 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c155770 con 0x7f956c06d910 2024-02-19T14:55:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.445+0000 7f95711d4640 1 --2- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c06d910 0x7f956c06c7e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9564009d90 tx=0x7f956402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29be56e2777bb317 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f956b7fe640 1 -- 172.21.15.78:0/694499873 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956402cbc0 con 0x7f956c06d910 2024-02-19T14:55:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f956b7fe640 1 -- 172.21.15.78:0/694499873 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956402cd20 con 0x7f956c06d910 2024-02-19T14:55:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f956b7fe640 1 -- 172.21.15.78:0/694499873 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564034660 con 0x7f956c06d910 2024-02-19T14:55:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 -- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c06d910 msgr2=0x7f956c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 --2- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c06d910 0x7f956c06c7e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9564009d90 tx=0x7f956402be10 comp rx=0 tx=0).stop 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 -- 172.21.15.78:0/694499873 shutdown_connections 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 --2- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c0c8e00 0x7f956c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 --2- 172.21.15.78:0/694499873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c06d910 0x7f956c06c7e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 -- 172.21.15.78:0/694499873 >> 172.21.15.78:0/694499873 conn(0x7f956c0cac90 msgr2=0x7f956c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 -- 172.21.15.78:0/694499873 shutdown_connections 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.446+0000 7f9572c37640 1 -- 172.21.15.78:0/694499873 wait complete. 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.447+0000 7f9572c37640 1 Processor -- start 2024-02-19T14:55:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.447+0000 7f9572c37640 1 -- start start 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f9572c37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 0x7f956c0c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f9572c37640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c0c8e00 0x7f956c0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f9572c37640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c152700 0x7f956c0abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f9572c37640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c14b7f0 con 0x7f956c06d910 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f9572c37640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c1594c0 con 0x7f956c0c8e00 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f9572c37640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c156a40 con 0x7f956c152700 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f95709d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 0x7f956c0c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f95709d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 0x7f956c0c3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60358/0 (socket says 172.21.15.78:60358) 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f95709d3640 1 -- 172.21.15.78:0/1699441283 learned_addr learned my addr 172.21.15.78:0/1699441283 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f956bfff640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c0c8e00 0x7f956c0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.448+0000 7f95711d4640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c152700 0x7f956c0abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95709d3640 1 -- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c152700 msgr2=0x7f956c0abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95709d3640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c152700 0x7f956c0abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95709d3640 1 -- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c0c8e00 msgr2=0x7f956c0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95709d3640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c0c8e00 0x7f956c0ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95709d3640 1 -- 172.21.15.78:0/1699441283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564009a70 con 0x7f956c06d910 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95709d3640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 0x7f956c0c3d80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f956000c3c0 tx=0x7f956000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560005ce0 con 0x7f956c06d910 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560005e40 con 0x7f956c06d910 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c158f00 con 0x7f956c06d910 2024-02-19T14:55:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95600072f0 con 0x7f956c06d910 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.449+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c0c42c0 con 0x7f956c06d910 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.450+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c074990 con 0x7f956c06d910 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.450+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9560007450 con 0x7f956c06d910 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.451+0000 7f95697fa640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f955006b840 0x7f955006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.451+0000 7f956bfff640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f955006b840 0x7f955006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.451+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95600897e0 con 0x7f956c06d910 2024-02-19T14:55:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.451+0000 7f956bfff640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f955006b840 0x7f955006dd00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f956c0ad340 tx=0x7f9558009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.455+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9560054130 con 0x7f956c06d910 2024-02-19T14:55:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.668+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f956c0c8770 con 0x7f956c06d910 2024-02-19T14:55:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.669+0000 7f95697fa640 1 -- 172.21.15.78:0/1699441283 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f95600577f0 con 0x7f956c06d910 2024-02-19T14:55:17.672 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:17.672 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5672,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:13.179911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6836/3610724432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3610724432},{"type":"v1","addr":"172.21.15.78:6836","nonce":3610724432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:17.672 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 73 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f955006b840 msgr2=0x7f955006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f955006b840 0x7f955006dd00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f956c0ad340 tx=0x7f9558009190 comp rx=0 tx=0).stop 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 msgr2=0x7f956c0c3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 0x7f956c0c3d80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f956000c3c0 tx=0x7f956000c880 comp rx=0 tx=0).stop 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 shutdown_connections 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f955006b840 0x7f955006dd00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f956c152700 0x7f956c0abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f956c0c8e00 0x7f956c0ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 --2- 172.21.15.78:0/1699441283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f956c06d910 0x7f956c0c3d80 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 >> 172.21.15.78:0/1699441283 conn(0x7f956c0cac90 msgr2=0x7f956c1512f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 shutdown_connections 2024-02-19T14:55:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:17.671+0000 7f9572c37640 1 -- 172.21.15.78:0/1699441283 wait complete. 2024-02-19T14:55:18.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:55:11.163509+0000', 'modified': '2024-02-19T14:55:13.179911+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6948}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6948': {'gid': 6948, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6836/3610724432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3610724432}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3610724432}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:18.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:18.087 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T14:55:18.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.187+0000 7f80140df640 1 Processor -- start 2024-02-19T14:55:18.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.187+0000 7f80140df640 1 -- start start 2024-02-19T14:55:18.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f80140df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c0c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f80140df640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f800c0c19c0 0x7f800c155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f80140df640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f800c072180 0x7f800c1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f80140df640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800c0734c0 con 0x7f800c0c19c0 2024-02-19T14:55:18.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f80140df640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800c155af0 con 0x7f800c072180 2024-02-19T14:55:18.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f80140df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c072e80 con 0x7f800c1528d0 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f800c0c19c0 0x7f800c155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f800c0c19c0 0x7f800c155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44848/0 (socket says 172.21.15.78:44848) 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 -- 172.21.15.78:0/125323308 learned_addr learned my addr 172.21.15.78:0/125323308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f8011e7b640 1 --2- 172.21.15.78:0/125323308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c0c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801267c640 1 --2- 172.21.15.78:0/125323308 >> v2:172.21.15.150:3300/0 conn(0x7f800c072180 0x7f800c1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 -- 172.21.15.78:0/125323308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 msgr2=0x7f800c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 --2- 172.21.15.78:0/125323308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c0c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 -- 172.21.15.78:0/125323308 >> v2:172.21.15.150:3300/0 conn(0x7f800c072180 msgr2=0x7f800c1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 --2- 172.21.15.78:0/125323308 >> v2:172.21.15.150:3300/0 conn(0x7f800c072180 0x7f800c1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801167a640 1 -- 172.21.15.78:0/125323308 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c06e990 con 0x7f800c0c19c0 2024-02-19T14:55:18.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.188+0000 7f801267c640 1 --2- 172.21.15.78:0/125323308 >> v2:172.21.15.150:3300/0 conn(0x7f800c072180 0x7f800c1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.189+0000 7f801167a640 1 --2- 172.21.15.78:0/125323308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c0c19c0 0x7f800c155070 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f800c0c1230 tx=0x7f7ffc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9198d5994d24cc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.189+0000 7f8010e79640 1 -- 172.21.15.78:0/125323308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0317e0 con 0x7f800c0c19c0 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.189+0000 7f8010e79640 1 -- 172.21.15.78:0/125323308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc031940 con 0x7f800c0c19c0 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.189+0000 7f8010e79640 1 -- 172.21.15.78:0/125323308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc031cd0 con 0x7f800c0c19c0 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 -- 172.21.15.78:0/125323308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c0c19c0 msgr2=0x7f800c155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 --2- 172.21.15.78:0/125323308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c0c19c0 0x7f800c155070 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f800c0c1230 tx=0x7f7ffc02b880 comp rx=0 tx=0).stop 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 -- 172.21.15.78:0/125323308 shutdown_connections 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 --2- 172.21.15.78:0/125323308 >> v2:172.21.15.150:3300/0 conn(0x7f800c072180 0x7f800c1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 --2- 172.21.15.78:0/125323308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c0c19c0 0x7f800c155070 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 --2- 172.21.15.78:0/125323308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c0c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 -- 172.21.15.78:0/125323308 >> 172.21.15.78:0/125323308 conn(0x7f800c0c7590 msgr2=0x7f800c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:18.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 -- 172.21.15.78:0/125323308 shutdown_connections 2024-02-19T14:55:18.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.190+0000 7f80140df640 1 -- 172.21.15.78:0/125323308 wait complete. 2024-02-19T14:55:18.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.191+0000 7f80140df640 1 Processor -- start 2024-02-19T14:55:18.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.191+0000 7f80140df640 1 -- start start 2024-02-19T14:55:18.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 0x7f800c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f800c0c19c0 0x7f800c1eaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c1eaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0a8fc0 con 0x7f800c072180 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0a97d0 con 0x7f800c0c19c0 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0acf20 con 0x7f800c1528d0 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 0x7f800c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 0x7f800c0ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44856/0 (socket says 172.21.15.78:44856) 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 -- 172.21.15.78:0/4265451114 learned_addr learned my addr 172.21.15.78:0/4265451114 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f801167a640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f800c0c19c0 0x7f800c1eaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f801267c640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c1eaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 -- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 msgr2=0x7f800c1eaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c1eaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 -- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f800c0c19c0 msgr2=0x7f800c1eaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f800c0c19c0 0x7f800c1eaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 -- 172.21.15.78:0/4265451114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc0094d0 con 0x7f800c072180 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f8011e7b640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 0x7f800c0ad5b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f800400d5f0 tx=0x7f800400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80027fc640 1 -- 172.21.15.78:0/4265451114 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80040125f0 con 0x7f800c072180 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80027fc640 1 -- 172.21.15.78:0/4265451114 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004012750 con 0x7f800c072180 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c0a9210 con 0x7f800c072180 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.192+0000 7f80027fc640 1 -- 172.21.15.78:0/4265451114 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004012b00 con 0x7f800c072180 2024-02-19T14:55:18.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.193+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c0a7cf0 con 0x7f800c072180 2024-02-19T14:55:18.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.194+0000 7f80027fc640 1 -- 172.21.15.78:0/4265451114 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8004012c60 con 0x7f800c072180 2024-02-19T14:55:18.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.195+0000 7f80027fc640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ff006b810 0x7f7ff006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.195+0000 7f80027fc640 1 -- 172.21.15.78:0/4265451114 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f800408a7c0 con 0x7f800c072180 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.195+0000 7f801167a640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ff006b810 0x7f7ff006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.195+0000 7f8016d97740 1 Processor -- start 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.196+0000 7f8016d97740 1 -- start start 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.196+0000 7f801167a640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ff006b810 0x7f7ff006dcd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc036000 tx=0x7f7ffc03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8016d97740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 0x563f24bf78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bbcae0 0x563f24bf6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bbb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8016d97740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24b76cd0 con 0x563f24bf6740 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24b29f20 con 0x563f24bbcae0 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f24b27450 con 0x563f24bf98b0 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 0x563f24bf78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 0x563f24bf78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44864/0 (socket says 172.21.15.78:44864) 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 -- 172.21.15.78:0/567096952 learned_addr learned my addr 172.21.15.78:0/567096952 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f801267c640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bbb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 -- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 msgr2=0x563f24bbb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bbb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f801167a640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bbcae0 0x563f24bf6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 -- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bbcae0 msgr2=0x563f24bf6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bbcae0 0x563f24bf6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 -- 172.21.15.78:0/567096952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f24ad7f20 con 0x563f24bf6740 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f801267c640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bbb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f801167a640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bbcae0 0x563f24bf6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f8011e7b640 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 0x563f24bf78b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f8004061020 tx=0x7f8004058860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29deaa970ba942e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f7fef7fe640 1 -- 172.21.15.78:0/567096952 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800405cf90 con 0x563f24bf6740 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f7fef7fe640 1 -- 172.21.15.78:0/567096952 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004061550 con 0x563f24bf6740 2024-02-19T14:55:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.197+0000 7f7fef7fe640 1 -- 172.21.15.78:0/567096952 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004057140 con 0x563f24bf6740 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 -- 172.21.15.78:0/567096952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 msgr2=0x563f24bf78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 0x563f24bf78b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f8004061020 tx=0x7f8004058860 comp rx=0 tx=0).stop 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 -- 172.21.15.78:0/567096952 shutdown_connections 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bbb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bbcae0 0x563f24bf6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 --2- 172.21.15.78:0/567096952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf6740 0x563f24bf78b0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 -- 172.21.15.78:0/567096952 >> 172.21.15.78:0/567096952 conn(0x563f24bb8120 msgr2=0x563f24bdcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 -- 172.21.15.78:0/567096952 shutdown_connections 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 -- 172.21.15.78:0/567096952 wait complete. 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.198+0000 7f8016d97740 1 -- start start 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 Processor -- start 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24c08510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bf2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bd95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24b22c30 con 0x563f24bf45c0 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24ac7d30 con 0x563f24bf6740 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f247f6310 con 0x563f24bf98b0 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.199+0000 7f8016d97740 10 client.0 _ll_register_callbacks cb 0x563f24acfd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bd95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bd95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44874/0 (socket says 172.21.15.78:44874) 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 -- 172.21.15.78:0/851191786 learned_addr learned my addr 172.21.15.78:0/851191786 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801167a640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bf2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f8011e7b640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24c08510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 -- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 msgr2=0x563f24bf2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bf2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 -- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 msgr2=0x563f24c08510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24c08510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 -- 172.21.15.78:0/851191786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c0a7cf0 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f801267c640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bd95e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f800800fb00 tx=0x7f800800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368f5d74a2445c70 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f7fecff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f7fecff9640 1 -- 172.21.15.78:0/851191786 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008019470 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f7fecff9640 1 -- 172.21.15.78:0/851191786 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f80080195d0 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f248226c0 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f7fecff9640 1 -- 172.21.15.78:0/851191786 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8008019790 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f8016d97740 10 client.7116 fetch_fsmap learned FSMap version 73 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x563f24b212b0 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.200+0000 7f7fecff9640 1 -- 172.21.15.78:0/851191786 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563f248226c0 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f7fecff9640 1 -- 172.21.15.78:0/851191786 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8008010400 con 0x563f24bf45c0 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8016d97740 10 client.7116 fetch_fsmap finished waiting for FSMap version 73 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8016d97740 10 client.7116 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:55:18.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8016d97740 20 client.7116 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8016d97740 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24bce600 0x563f24bce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8016d97740 4 client.7116 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 --> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f24873270 con 0x563f24bce600 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8011e7b640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24bce600 0x563f24bce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f7fecff9640 10 client.7116 ms_handle_connect on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.201+0000 7f8011e7b640 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24bce600 0x563f24bce9e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2658fd18a2f379 server_cookie=7dc940bcd84f36e7 in_seq=0 out_seq=0 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f7fecff9640 1 -- 172.21.15.78:0/851191786 <== mds.0 v2:172.21.15.78:6834/3610724432 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563f24873270 con 0x563f24bce600 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f7fecff9640 10 client.7116 handle_command_reply: tid=0 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 client.7116 shutdown 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 msgr2=0x563f24bd95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bd95e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f800800fb00 tx=0x7f800800a340 comp rx=0 tx=0).stop 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 shutdown_connections 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24bce600 0x563f24bce9e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bd95e0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bf2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f7fecff9640 0 client.7116 ms_handle_reset on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 --2- 172.21.15.78:0/851191786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24c08510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 >> 172.21.15.78:0/851191786 conn(0x563f24bbcae0 msgr2=0x563f24bdcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 shutdown_connections 2024-02-19T14:55:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 1 -- 172.21.15.78:0/851191786 wait complete. 2024-02-19T14:55:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.202+0000 7f8016d97740 20 client.7116 trim_cache size 0 max 16384 2024-02-19T14:55:18.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 Processor -- start 2024-02-19T14:55:18.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 -- start start 2024-02-19T14:55:18.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bba770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bb72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24aa8490 con 0x563f24bf45c0 2024-02-19T14:55:18.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24bf3250 con 0x563f24bf6740 2024-02-19T14:55:18.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f24c13010 con 0x563f24bf98b0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.219+0000 7f8011e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bba770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bba770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44880/0 (socket says 172.21.15.78:44880) 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 -- 172.21.15.78:0/1019433523 learned_addr learned my addr 172.21.15.78:0/1019433523 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f801267c640 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bb72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f801167a640 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 -- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 msgr2=0x563f24bb72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bb72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 -- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 msgr2=0x563f24bb6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 -- 172.21.15.78:0/1019433523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f24bf5af0 con 0x563f24bf45c0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f8011e7b640 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bba770 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f80040552a0 tx=0x7f80040588d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f0c6fbe96c7ce2f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f7feeffd640 1 -- 172.21.15.78:0/1019433523 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004088040 con 0x563f24bf45c0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f7feeffd640 1 -- 172.21.15.78:0/1019433523 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80040550e0 con 0x563f24bf45c0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.220+0000 7f7feeffd640 1 -- 172.21.15.78:0/1019433523 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004056f60 con 0x563f24bf45c0 2024-02-19T14:55:18.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 -- 172.21.15.78:0/1019433523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 msgr2=0x563f24bba770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bba770 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f80040552a0 tx=0x7f80040588d0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 -- 172.21.15.78:0/1019433523 shutdown_connections 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf98b0 0x563f24bb72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 --2- 172.21.15.78:0/1019433523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf45c0 0x563f24bba770 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 -- 172.21.15.78:0/1019433523 >> 172.21.15.78:0/1019433523 conn(0x563f24bbcae0 msgr2=0x563f24bf7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 -- 172.21.15.78:0/1019433523 shutdown_connections 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 -- 172.21.15.78:0/1019433523 wait complete. 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 -- start start 2024-02-19T14:55:18.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.221+0000 7f8016d97740 1 Processor -- start 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf45c0 0x563f24bbab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 0x563f24c646a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24bb6b90 con 0x563f24bf98b0 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f24bf2690 con 0x563f24bf6740 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f24bfaf20 con 0x563f24bf45c0 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8016d97740 10 client.0 _ll_register_callbacks cb 0x7f80080076b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f801267c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 0x563f24c646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f801267c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 0x563f24c646a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44894/0 (socket says 172.21.15.78:44894) 2024-02-19T14:55:18.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f801267c640 1 -- 172.21.15.78:0/721590691 learned_addr learned my addr 172.21.15.78:0/721590691 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f8011e7b640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf45c0 0x563f24bbab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.222+0000 7f801167a640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f801267c640 1 -- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf45c0 msgr2=0x563f24bbab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f801267c640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf45c0 0x563f24bbab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f801267c640 1 -- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 msgr2=0x563f24bb5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f801267c640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f801267c640 1 -- 172.21.15.78:0/721590691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f24ad7f20 con 0x563f24bf98b0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f801267c640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 0x563f24c646a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x563f24bbb880 tx=0x7f8008004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4a5b311a2bd134 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f7fecff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f7fecff9640 1 -- 172.21.15.78:0/721590691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800802a070 con 0x563f24bf98b0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f7fecff9640 1 -- 172.21.15.78:0/721590691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f800800f610 con 0x563f24bf98b0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8004061550 con 0x563f24bf98b0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f7fecff9640 1 -- 172.21.15.78:0/721590691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f800800a840 con 0x563f24bf98b0 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 10 client.7122 fetch_fsmap learned FSMap version 73 2024-02-19T14:55:18.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x563f24abda20 con 0x563f24bf98b0 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f7fecff9640 1 -- 172.21.15.78:0/721590691 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004061550 con 0x563f24bf98b0 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f7fecff9640 1 -- 172.21.15.78:0/721590691 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8008003990 con 0x563f24bf98b0 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 10 client.7122 fetch_fsmap finished waiting for FSMap version 73 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 10 client.7122 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 20 client.7122 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24c64c50 0x563f24bcd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 4 client.7122 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 --> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563f2485f610 con 0x563f24c64c50 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.223+0000 7f8011e7b640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24c64c50 0x563f24bcd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f7fecff9640 10 client.7122 ms_handle_connect on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8011e7b640 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24c64c50 0x563f24bcd190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52ee6c47fa89d0d server_cookie=cf087c73b02678bd in_seq=0 out_seq=0 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f7fecff9640 1 -- 172.21.15.78:0/721590691 <== mds.0 v2:172.21.15.78:6834/3610724432 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x563f2485f610 con 0x563f24c64c50 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f7fecff9640 10 client.7122 handle_command_reply: tid=0 2024-02-19T14:55:18.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 client.7122 shutdown 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 msgr2=0x563f24c646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 0x563f24c646a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x563f24bbb880 tx=0x7f8008004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 shutdown_connections 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:6834/3610724432,v1:172.21.15.78:6836/3610724432] conn(0x563f24c64c50 0x563f24bcd190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f24bf98b0 0x563f24c646a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f24bf6740 0x563f24bb5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f7fecff9640 0 client.7122 ms_handle_reset on v2:172.21.15.78:6834/3610724432 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 --2- 172.21.15.78:0/721590691 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563f24bf45c0 0x563f24bbab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 >> 172.21.15.78:0/721590691 conn(0x563f24bbcae0 msgr2=0x563f24bb1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 shutdown_connections 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.224+0000 7f8016d97740 1 -- 172.21.15.78:0/721590691 wait complete. 2024-02-19T14:55:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f8016d97740 20 client.7122 trim_cache size 0 max 16384 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 6948, 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 6.4084428969999996, 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 73, 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 98, 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 98, 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 11.523623337 2024-02-19T14:55:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:18.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ff006b810 msgr2=0x7f7ff006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ff006b810 0x7f7ff006dcd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc036000 tx=0x7f7ffc03d040 comp rx=0 tx=0).stop 2024-02-19T14:55:18.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 msgr2=0x7f800c0ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:18.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 0x7f800c0ad5b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f800400d5f0 tx=0x7f800400dab0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 shutdown_connections 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ff006b810 0x7f7ff006dcd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f800c1528d0 0x7f800c1eaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f800c0c19c0 0x7f800c1eaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 --2- 172.21.15.78:0/4265451114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f800c072180 0x7f800c0ad5b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 >> 172.21.15.78:0/4265451114 conn(0x7f800c0c7590 msgr2=0x7f800c0c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 shutdown_connections 2024-02-19T14:55:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:18.225+0000 7f80140df640 1 -- 172.21.15.78:0/4265451114 wait complete. 2024-02-19T14:55:18.238 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:11.163509+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6948": { "addr": "172.21.15.78:6836/3610724432", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3610724432, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3610724432, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6948, "incarnation": 71, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-02-19T14:55:13.179911+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6948 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/408965047", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 408965047, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 408965047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5672, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/829046094", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 829046094, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 829046094, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6310, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/2386100646", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2386100646, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2386100646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6945, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:18.239 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 7, 'name': 'd', 'state': 'up:active'}] 2024-02-19T14:55:18.239 INFO:tasks.cephfs.test_failover:current = [{'gid': 6948, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6836/3610724432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3610724432}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3610724432}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6310, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/829046094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 829046094}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 829046094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}, {'gid': 6945, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/2386100646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2386100646}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2386100646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}, {'gid': 5672, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/408965047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 408965047}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 408965047}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}] 2024-02-19T14:55:18.239 DEBUG:tasks.cephfs.test_failover:7 != -1 2024-02-19T14:55:18.240 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:55:18.865 INFO:tasks.ceph.mds.a.smithi078.stderr: -104> 2024-02-19T14:55:17.091+0000 7f0578a1b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:18.890 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:55:22.094 INFO:tasks.ceph.mds.d.smithi150.stderr:2024-02-19T14:55:22.091+0000 7fb60f4d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:23.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:23.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.352+0000 7f4e3581c640 1 Processor -- start 2024-02-19T14:55:23.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.352+0000 7f4e3581c640 1 -- start start 2024-02-19T14:55:23.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e3581c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 0x7f4e3006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:23.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e3581c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e3006d750 0x7f4e3015d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:23.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e3581c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e30150820 con 0x7f4e30150440 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e3581c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e300c1b60 con 0x7f4e3006d750 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e3581c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e301537d0 con 0x7f4e3006ed20 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2e7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 0x7f4e3006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e3006d750 0x7f4e3015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2e7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 0x7f4e3006d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:35508/0 (socket says 172.21.15.78:35508) 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2e7fc640 1 -- 172.21.15.78:0/3831922669 learned_addr learned my addr 172.21.15.78:0/3831922669 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2effd640 1 --1- 172.21.15.78:0/3831922669 >> v1:172.21.15.150:6789/0 conn(0x7f4e30150440 0x7f4e300c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57842/0 (socket says 172.21.15.78:57842) 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2e7fc640 1 -- 172.21.15.78:0/3831922669 >> v2:172.21.15.150:3300/0 conn(0x7f4e3006d750 msgr2=0x7f4e3015d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2e7fc640 1 --2- 172.21.15.78:0/3831922669 >> v2:172.21.15.150:3300/0 conn(0x7f4e3006d750 0x7f4e3015d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.353+0000 7f4e2e7fc640 1 -- 172.21.15.78:0/3831922669 >> v1:172.21.15.150:6789/0 conn(0x7f4e30150440 legacy=0x7f4e300c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.354+0000 7f4e2e7fc640 1 -- 172.21.15.78:0/3831922669 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e30153d10 con 0x7f4e3006ed20 2024-02-19T14:55:23.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.354+0000 7f4e2f7fe640 1 --2- 172.21.15.78:0/3831922669 >> v2:172.21.15.150:3300/0 conn(0x7f4e3006d750 0x7f4e3015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.354+0000 7f4e2e7fc640 1 --2- 172.21.15.78:0/3831922669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 0x7f4e3006d210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009870 tx=0x7f4e2402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4761d0feb039f4e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.354+0000 7f4e2dffb640 1 -- 172.21.15.78:0/3831922669 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2402f830 con 0x7f4e3006ed20 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e2dffb640 1 -- 172.21.15.78:0/3831922669 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2402f990 con 0x7f4e3006ed20 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e2dffb640 1 -- 172.21.15.78:0/3831922669 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2402fd20 con 0x7f4e3006ed20 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 -- 172.21.15.78:0/3831922669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 msgr2=0x7f4e3006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 --2- 172.21.15.78:0/3831922669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 0x7f4e3006d210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009870 tx=0x7f4e2402ba10 comp rx=0 tx=0).stop 2024-02-19T14:55:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 -- 172.21.15.78:0/3831922669 shutdown_connections 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 --2- 172.21.15.78:0/3831922669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e3006ed20 0x7f4e3006d210 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 --2- 172.21.15.78:0/3831922669 >> v2:172.21.15.150:3300/0 conn(0x7f4e3006d750 0x7f4e3015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 -- 172.21.15.78:0/3831922669 >> 172.21.15.78:0/3831922669 conn(0x7f4e3006b270 msgr2=0x7f4e3006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 -- 172.21.15.78:0/3831922669 shutdown_connections 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.355+0000 7f4e3581c640 1 -- 172.21.15.78:0/3831922669 wait complete. 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.356+0000 7f4e3581c640 1 Processor -- start 2024-02-19T14:55:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 -- start start 2024-02-19T14:55:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e3006d750 0x7f4e301e1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 0x7f4e301ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e30150440 0x7f4e301eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3014b260 con 0x7f4e3006ed20 2024-02-19T14:55:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3014a6e0 con 0x7f4e3006d750 2024-02-19T14:55:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e3581c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3014a9c0 con 0x7f4e30150440 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e2e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 0x7f4e301ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e2e7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 0x7f4e301ee6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44930/0 (socket says 172.21.15.78:44930) 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e2f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e30150440 0x7f4e301eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e2e7fc640 1 -- 172.21.15.78:0/1293036047 learned_addr learned my addr 172.21.15.78:0/1293036047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.357+0000 7f4e2effd640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e3006d750 0x7f4e301e1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2f7fe640 1 -- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e3006d750 msgr2=0x7f4e301e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2f7fe640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e3006d750 0x7f4e301e1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2f7fe640 1 -- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 msgr2=0x7f4e301ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2f7fe640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 0x7f4e301ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2f7fe640 1 -- 172.21.15.78:0/1293036047 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28009a70 con 0x7f4e30150440 2024-02-19T14:55:23.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2e7fc640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 0x7f4e301ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2effd640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e3006d750 0x7f4e301e1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e2f7fe640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e30150440 0x7f4e301eebf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800efc0 tx=0x7f4e2800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800cd30 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2800f040 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e240094d0 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28003bd0 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.358+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3014af80 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.359+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e30074990 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.360+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4e28014480 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.360+0000 7f4e17fff640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4e1c06b810 0x7f4e1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.360+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4e2808adb0 con 0x7f4e30150440 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.360+0000 7f4e2effd640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4e1c06b810 0x7f4e1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.360+0000 7f4e2effd640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4e1c06b810 0x7f4e1c06dcd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4e20007fd0 tx=0x7f4e20007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.365+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4e28055730 con 0x7f4e30150440 2024-02-19T14:55:23.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.569+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e300c19c0 con 0x7f4e30150440 2024-02-19T14:55:23.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.570+0000 7f4e17fff640 1 -- 172.21.15.78:0/1293036047 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f4e300c19c0 con 0x7f4e30150440 2024-02-19T14:55:23.572 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:23.573 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7128,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1361734331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1361734331},{"type":"v1","addr":"172.21.15.78:6836","nonce":1361734331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:20.871770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5672":{"gid":5672,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:23.573 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 78 2024-02-19T14:55:23.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4e1c06b810 msgr2=0x7f4e1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4e1c06b810 0x7f4e1c06dcd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4e20007fd0 tx=0x7f4e20007e80 comp rx=0 tx=0).stop 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e30150440 msgr2=0x7f4e301eebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e30150440 0x7f4e301eebf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800efc0 tx=0x7f4e2800c490 comp rx=0 tx=0).stop 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 shutdown_connections 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4e1c06b810 0x7f4e1c06dcd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e3006ed20 0x7f4e301ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e30150440 0x7f4e301eebf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 --2- 172.21.15.78:0/1293036047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e3006d750 0x7f4e301e1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.571+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 >> 172.21.15.78:0/1293036047 conn(0x7f4e3006b270 msgr2=0x7f4e3006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.572+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 shutdown_connections 2024-02-19T14:55:23.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:23.572+0000 7f4e3581c640 1 -- 172.21.15.78:0/1293036047 wait complete. 2024-02-19T14:55:23.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:55:11.163509+0000', 'modified': '2024-02-19T14:55:20.871770+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5672}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5672': {'gid': 5672, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.150:6834/408965047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 408965047}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 408965047}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:23.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:23.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:55:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.086+0000 7ff9af2c2640 1 Processor -- start 2024-02-19T14:55:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9af2c2640 1 -- start start 2024-02-19T14:55:24.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9af2c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a8152700 0x7ff9a80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9af2c2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a81551f0 con 0x7ff9a80c8e00 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9af2c2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a8155490 con 0x7ff9a80c5730 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9af2c2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a8155770 con 0x7ff9a8152700 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9ad85f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff9a80c8e00 0x7ff9a815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47070/0 (socket says 172.21.15.78:47070) 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9ad85f640 1 -- 172.21.15.78:0/3085307070 learned_addr learned my addr 172.21.15.78:0/3085307070 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.087+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/3085307070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a8152700 0x7ff9a80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356749786 0 0) 0x7ff9a81551f0 con 0x7ff9a80c8e00 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff994003520 con 0x7ff9a80c8e00 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723048634 0 0) 0x7ff9a8155490 con 0x7ff9a80c5730 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a81551f0 con 0x7ff9a80c5730 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff9ad05e640 1 -- 172.21.15.78:0/3085307070 >> v1:172.21.15.150:6789/0 conn(0x7ff9a80c5730 legacy=0x7ff9a806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff9ad05e640 1 -- 172.21.15.78:0/3085307070 >> v1:172.21.15.78:6789/0 conn(0x7ff9a80c8e00 legacy=0x7ff9a815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.088+0000 7ff9ad05e640 1 -- 172.21.15.78:0/3085307070 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8155a50 con 0x7ff9a8152700 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/3085307070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a8152700 0x7ff9a80c51f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40097d0 tx=0x7ff9a402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17f4952915077313 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4031890 con 0x7ff9a8152700 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a40319f0 con 0x7ff9a8152700 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff99bfff640 1 -- 172.21.15.78:0/3085307070 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4031d60 con 0x7ff9a8152700 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 -- 172.21.15.78:0/3085307070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a8152700 msgr2=0x7ff9a80c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/3085307070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a8152700 0x7ff9a80c51f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40097d0 tx=0x7ff9a402b930 comp rx=0 tx=0).stop 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 -- 172.21.15.78:0/3085307070 shutdown_connections 2024-02-19T14:55:24.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/3085307070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a8152700 0x7ff9a80c51f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 -- 172.21.15.78:0/3085307070 >> 172.21.15.78:0/3085307070 conn(0x7ff9a80cac90 msgr2=0x7ff9a80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 -- 172.21.15.78:0/3085307070 shutdown_connections 2024-02-19T14:55:24.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.089+0000 7ff9af2c2640 1 -- 172.21.15.78:0/3085307070 wait complete. 2024-02-19T14:55:24.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.090+0000 7ff9af2c2640 1 Processor -- start 2024-02-19T14:55:24.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.090+0000 7ff9af2c2640 1 -- start start 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9af2c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a80c5730 0x7ff9a81ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9af2c2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 0x7ff9a8148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9af2c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8152700 0x7ff9a80c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9af2c2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a80c4530 con 0x7ff9a80c8e00 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9af2c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a80bdf10 con 0x7ff9a8152700 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9af2c2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a80c4780 con 0x7ff9a80c5730 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 0x7ff9a8148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 0x7ff9a8148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44942/0 (socket says 172.21.15.78:44942) 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2500383940 learned_addr learned my addr 172.21.15.78:0/2500383940 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a80c5730 0x7ff9a81ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ad85f640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8152700 0x7ff9a80c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a80c5730 msgr2=0x7ff9a81ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a80c5730 0x7ff9a81ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8152700 msgr2=0x7ff9a80c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8152700 0x7ff9a80c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.091+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2500383940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4009400 con 0x7ff9a80c8e00 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff9ad85f640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8152700 0x7ff9a80c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 0x7ff9a8148610 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00c500 tx=0x7ff99c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff999ffb640 1 -- 172.21.15.78:0/2500383940 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0052d0 con 0x7ff9a80c8e00 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff999ffb640 1 -- 172.21.15.78:0/2500383940 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c005450 con 0x7ff9a80c8e00 2024-02-19T14:55:24.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a81eed70 con 0x7ff9a80c8e00 2024-02-19T14:55:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff999ffb640 1 -- 172.21.15.78:0/2500383940 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c012870 con 0x7ff9a80c8e00 2024-02-19T14:55:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.092+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a80ace90 con 0x7ff9a80c8e00 2024-02-19T14:55:24.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.094+0000 7ff999ffb640 1 -- 172.21.15.78:0/2500383940 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff99c0129d0 con 0x7ff9a80c8e00 2024-02-19T14:55:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.094+0000 7ff999ffb640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff98c06b810 0x7ff98c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.094+0000 7ff999ffb640 1 -- 172.21.15.78:0/2500383940 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff99c089fd0 con 0x7ff9a80c8e00 2024-02-19T14:55:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.094+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff98c06b810 0x7ff98c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.094+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff98c06b810 0x7ff98c06dcd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4036000 tx=0x7ff9a403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.095+0000 7ff9b1f70740 1 Processor -- start 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.095+0000 7ff9b1f70740 1 -- start start 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4c740 0x5617fcd4d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 0x5617fcd4cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd11860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcccccd0 con 0x5617fcd12ae0 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcc7ff20 con 0x5617fcd4f8b0 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617fcc7d450 con 0x5617fcd4c740 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 0x5617fcd4cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 0x5617fcd4cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44952/0 (socket says 172.21.15.78:44952) 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2309499187 learned_addr learned my addr 172.21.15.78:0/2309499187 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4c740 0x5617fcd4d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ad85f640 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd11860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4c740 msgr2=0x5617fcd4d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4c740 0x5617fcd4d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 msgr2=0x5617fcd11860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd11860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 -- 172.21.15.78:0/2309499187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617fcc2df20 con 0x5617fcd12ae0 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 0x5617fcd4cb20 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff99c005620 tx=0x7ff99c058d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557cf56927b509a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff992ffd640 1 -- 172.21.15.78:0/2309499187 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c060c90 con 0x5617fcd12ae0 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff992ffd640 1 -- 172.21.15.78:0/2309499187 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c056880 con 0x5617fcd12ae0 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.096+0000 7ff992ffd640 1 -- 172.21.15.78:0/2309499187 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0770d0 con 0x5617fcd12ae0 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 -- 172.21.15.78:0/2309499187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 msgr2=0x5617fcd4cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 0x5617fcd4cb20 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff99c005620 tx=0x7ff99c058d40 comp rx=0 tx=0).stop 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 -- 172.21.15.78:0/2309499187 shutdown_connections 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd12ae0 0x5617fcd4cb20 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd11860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/2309499187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4c740 0x5617fcd4d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 -- 172.21.15.78:0/2309499187 >> 172.21.15.78:0/2309499187 conn(0x5617fcd0e120 msgr2=0x5617fcd32b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 -- 172.21.15.78:0/2309499187 shutdown_connections 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.097+0000 7ff9b1f70740 1 -- 172.21.15.78:0/2309499187 wait complete. 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 -- start start 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 Processor -- start 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd112b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd44d90 0x5617fcd451f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcc78c30 con 0x5617fcd4c740 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcc1dd30 con 0x5617fcd4f8b0 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.098+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617fc94c310 con 0x5617fcd44d90 2024-02-19T14:55:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9b1f70740 10 client.0 _ll_register_callbacks cb 0x5617fcc25d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd112b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44960/0 (socket says 172.21.15.78:44960) 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 -- 172.21.15.78:0/76206309 learned_addr learned my addr 172.21.15.78:0/76206309 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad85f640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd44d90 0x5617fcd451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 -- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd44d90 msgr2=0x5617fcd451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd44d90 0x5617fcd451f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 -- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 msgr2=0x5617fcd462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 -- 172.21.15.78:0/76206309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a80ace90 con 0x5617fcd4c740 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd112b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40098c0 tx=0x7ff9a402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d3725118a2df7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff987fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff987fff640 1 -- 172.21.15.78:0/76206309 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a404b040 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff987fff640 1 -- 172.21.15.78:0/76206309 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff9a403f070 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.099+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617fc9786c0 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff987fff640 1 -- 172.21.15.78:0/76206309 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9a4044890 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 10 client.7143 fetch_fsmap learned FSMap version 78 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5617fcc772b0 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff987fff640 1 -- 172.21.15.78:0/76206309 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617fc9786c0 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff987fff640 1 -- 172.21.15.78:0/76206309 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff9a40449f0 con 0x5617fcd4c740 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 10 client.7143 fetch_fsmap finished waiting for FSMap version 78 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 10 client.7143 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 20 client.7143 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd60ef0 0x5617fcd612d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 4 client.7143 mds_command: new command op to 5672 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 --> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617fc9c9270 con 0x5617fcd60ef0 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd60ef0 0x5617fcd612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff987fff640 10 client.7143 ms_handle_connect on v2:172.21.15.150:6832/408965047 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.100+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd60ef0 0x5617fcd612d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79d6c33d9dfeef25 server_cookie=140b48361136d7d2 in_seq=0 out_seq=0 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff987fff640 1 -- 172.21.15.78:0/76206309 <== mds.0 v2:172.21.15.150:6832/408965047 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5617fc9c9270 con 0x5617fcd60ef0 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff987fff640 10 client.7143 handle_command_reply: tid=0 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 client.7143 shutdown 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 msgr2=0x5617fcd112b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd112b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40098c0 tx=0x7ff9a402e000 comp rx=0 tx=0).stop 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 shutdown_connections 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd60ef0 0x5617fcd612d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd44d90 0x5617fcd451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4f8b0 0x5617fcd462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/76206309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd112b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff987fff640 0 client.7143 ms_handle_reset on v2:172.21.15.150:6832/408965047 2024-02-19T14:55:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.101+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 >> 172.21.15.78:0/76206309 conn(0x5617fcd12ae0 msgr2=0x5617fcd0af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.102+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 shutdown_connections 2024-02-19T14:55:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.102+0000 7ff9b1f70740 1 -- 172.21.15.78:0/76206309 wait complete. 2024-02-19T14:55:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.102+0000 7ff9b1f70740 20 client.7143 trim_cache size 0 max 16384 2024-02-19T14:55:24.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.118+0000 7ff9b1f70740 1 Processor -- start 2024-02-19T14:55:24.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.118+0000 7ff9b1f70740 1 -- start start 2024-02-19T14:55:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd44d90 0x5617fcd4cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcd10910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcbfe490 con 0x5617fcd4c740 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fccaa380 con 0x5617fcd44d90 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617fcd4a0a0 con 0x5617fcd4f8b0 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd103d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44964/0 (socket says 172.21.15.78:44964) 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 -- 172.21.15.78:0/330773940 learned_addr learned my addr 172.21.15.78:0/330773940 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ad85f640 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcd10910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd44d90 0x5617fcd4cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 -- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 msgr2=0x5617fcd10910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcd10910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 -- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd44d90 msgr2=0x5617fcd4cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd44d90 0x5617fcd4cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 -- 172.21.15.78:0/330773940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617fcc13a20 con 0x5617fcd4c740 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.119+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd103d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ff99c058740 tx=0x7ff99c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3df864d408e03d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9927fc640 1 -- 172.21.15.78:0/330773940 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c060c90 con 0x5617fcd4c740 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9927fc640 1 -- 172.21.15.78:0/330773940 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c0430e0 con 0x5617fcd4c740 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9927fc640 1 -- 172.21.15.78:0/330773940 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c054bf0 con 0x5617fcd4c740 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 -- 172.21.15.78:0/330773940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 msgr2=0x5617fcd103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd103d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ff99c058740 tx=0x7ff99c01c000 comp rx=0 tx=0).stop 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 -- 172.21.15.78:0/330773940 shutdown_connections 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd4c740 0x5617fcd103d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcd10910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/330773940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd44d90 0x5617fcd4cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 -- 172.21.15.78:0/330773940 >> 172.21.15.78:0/330773940 conn(0x5617fcd12ae0 msgr2=0x5617fcd112b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 -- 172.21.15.78:0/330773940 shutdown_connections 2024-02-19T14:55:24.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.120+0000 7ff9b1f70740 1 -- 172.21.15.78:0/330773940 wait complete. 2024-02-19T14:55:24.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.121+0000 7ff9b1f70740 1 -- start start 2024-02-19T14:55:24.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.121+0000 7ff9b1f70740 1 Processor -- start 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 0x5617fcd5e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4c740 0x5617fcd0fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcdc0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcd421a0 con 0x5617fcd44d90 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fcd4cb20 con 0x5617fcd4c740 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617fcd07860 con 0x5617fcd4f8b0 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 10 client.0 _ll_register_callbacks cb 0x7ff9a4009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 0x5617fcd5e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 0x5617fcd5e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44970/0 (socket says 172.21.15.78:44970) 2024-02-19T14:55:24.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 -- 172.21.15.78:0/53977417 learned_addr learned my addr 172.21.15.78:0/53977417 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4c740 0x5617fcd0fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad85f640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcdc0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 -- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 msgr2=0x5617fcdc0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcdc0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 -- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4c740 msgr2=0x5617fcd0fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4c740 0x5617fcd0fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 -- 172.21.15.78:0/53977417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c070040 con 0x5617fcd44d90 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9ad05e640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 0x5617fcd5e220 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ff9a402bf40 tx=0x7ff9a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f32f0d01608b01 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff987fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff987fff640 1 -- 172.21.15.78:0/53977417 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a404b040 con 0x5617fcd44d90 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff987fff640 1 -- 172.21.15.78:0/53977417 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff9a403f040 con 0x5617fcd44d90 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff99c056880 con 0x5617fcd44d90 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff987fff640 1 -- 172.21.15.78:0/53977417 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9a4044aa0 con 0x5617fcd44d90 2024-02-19T14:55:24.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 10 client.7149 fetch_fsmap learned FSMap version 78 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.122+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5617fcc39780 con 0x5617fcd44d90 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff987fff640 1 -- 172.21.15.78:0/53977417 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c056880 con 0x5617fcd44d90 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff987fff640 1 -- 172.21.15.78:0/53977417 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff9a404f510 con 0x5617fcd44d90 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 10 client.7149 fetch_fsmap finished waiting for FSMap version 78 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 10 client.7149 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 20 client.7149 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd2bc90 0x5617fcd2c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 4 client.7149 mds_command: new command op to 5672 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 --> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5617fc9b5610 con 0x5617fcd2bc90 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd2bc90 0x5617fcd2c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff987fff640 10 client.7149 ms_handle_connect on v2:172.21.15.150:6832/408965047 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9ac85d640 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd2bc90 0x5617fcd2c070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd98f08b1150c117 server_cookie=1575e8978a1b19f5 in_seq=0 out_seq=0 2024-02-19T14:55:24.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff987fff640 1 -- 172.21.15.78:0/53977417 <== mds.0 v2:172.21.15.150:6832/408965047 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5617fc9b5610 con 0x5617fcd2bc90 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff987fff640 10 client.7149 handle_command_reply: tid=0 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.123+0000 7ff9b1f70740 1 client.7149 shutdown 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 msgr2=0x5617fcd5e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 0x5617fcd5e220 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ff9a402bf40 tx=0x7ff9a4004730 comp rx=0 tx=0).stop 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 shutdown_connections 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:6832/408965047,v1:172.21.15.150:6834/408965047] conn(0x5617fcd2bc90 0x5617fcd2c070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5617fcd4f8b0 0x5617fcdc0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617fcd4c740 0x5617fcd0fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff987fff640 0 client.7149 ms_handle_reset on v2:172.21.15.150:6832/408965047 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 --2- 172.21.15.78:0/53977417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5617fcd44d90 0x5617fcd5e220 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 >> 172.21.15.78:0/53977417 conn(0x5617fcd12ae0 msgr2=0x5617fcd4ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 shutdown_connections 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 1 -- 172.21.15.78:0/53977417 wait complete. 2024-02-19T14:55:24.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9b1f70740 20 client.7149 trim_cache size 0 max 16384 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5672, 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 5.2626588520000004, 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 78, 2024-02-19T14:55:24.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 99, 2024-02-19T14:55:24.135 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 99, 2024-02-19T14:55:24.135 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17.412300448 2024-02-19T14:55:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff98c06b810 msgr2=0x7ff98c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff98c06b810 0x7ff98c06dcd0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4036000 tx=0x7ff9a403c040 comp rx=0 tx=0).stop 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 msgr2=0x7ff9a8148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 0x7ff9a8148610 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00c500 tx=0x7ff99c00c9c0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 shutdown_connections 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff98c06b810 0x7ff98c06dcd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9a80c8e00 0x7ff9a8148610 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a8152700 0x7ff9a80c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 --2- 172.21.15.78:0/2500383940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9a80c5730 0x7ff9a81ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 >> 172.21.15.78:0/2500383940 conn(0x7ff9a80cac90 msgr2=0x7ff9a8150990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 shutdown_connections 2024-02-19T14:55:24.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.124+0000 7ff9af2c2640 1 -- 172.21.15.78:0/2500383940 wait complete. 2024-02-19T14:55:24.137 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 78, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:11.163509+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 78, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5672": { "addr": "172.21.15.150:6834/408965047", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 408965047, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 408965047, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5672, "incarnation": 75, "join_fscid": 7, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 99, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-02-19T14:55:20.871770+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5672 } } } ], "standbys": [ { "addr": "172.21.15.150:6835/829046094", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 829046094, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 829046094, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6310, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/2386100646", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2386100646, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2386100646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 70, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6945, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1361734331", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1361734331, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1361734331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 76, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7128, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:24.138 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 7, 'name': 'd', 'state': 'up:active'}] 2024-02-19T14:55:24.139 INFO:tasks.cephfs.test_failover:current = [{'gid': 7128, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/1361734331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1361734331}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1361734331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 76}, {'gid': 6310, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/829046094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 829046094}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 829046094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}, {'gid': 6945, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/2386100646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2386100646}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2386100646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 70}, {'gid': 5672, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.150:6834/408965047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 408965047}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 408965047}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:55:24.139 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:55:24.139 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:55:24.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:55:24.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:55:24.218 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:55:24.218 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:55:24.218 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:55:24.234 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:55:24.234 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:55:24.234 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:55:24.275 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:55:24.275 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:24.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.371+0000 7f55a9de4640 1 Processor -- start 2024-02-19T14:55:24.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.371+0000 7f55a9de4640 1 -- start start 2024-02-19T14:55:24.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a9de4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c25e0 0x7f55a41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a9de4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55a40c29c0 0x7f55a40c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a9de4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55a40c49c0 0x7f55a406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a9de4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a4152c90 con 0x7f55a40c49c0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a9de4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a4072250 con 0x7f55a40c29c0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a9de4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a4072480 con 0x7f55a40c25e0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55a40c49c0 0x7f55a406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55a40c49c0 0x7f55a406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44972/0 (socket says 172.21.15.78:44972) 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 -- 172.21.15.78:0/3684258366 learned_addr learned my addr 172.21.15.78:0/3684258366 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a37fe640 1 --2- 172.21.15.78:0/3684258366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c25e0 0x7f55a41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a2ffd640 1 --2- 172.21.15.78:0/3684258366 >> v2:172.21.15.150:3300/0 conn(0x7f55a40c29c0 0x7f55a40c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 -- 172.21.15.78:0/3684258366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c25e0 msgr2=0x7f55a41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 --2- 172.21.15.78:0/3684258366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c25e0 0x7f55a41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 -- 172.21.15.78:0/3684258366 >> v2:172.21.15.150:3300/0 conn(0x7f55a40c29c0 msgr2=0x7f55a40c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 --2- 172.21.15.78:0/3684258366 >> v2:172.21.15.150:3300/0 conn(0x7f55a40c29c0 0x7f55a40c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.372+0000 7f55a3fff640 1 -- 172.21.15.78:0/3684258366 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a40c3eb0 con 0x7f55a40c49c0 2024-02-19T14:55:24.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.373+0000 7f55a3fff640 1 --2- 172.21.15.78:0/3684258366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40c49c0 0x7f55a406e420 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f55a4072650 tx=0x7f559c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd62c4e469513ab7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.373+0000 7f55a27fc640 1 -- 172.21.15.78:0/3684258366 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c02c9c0 con 0x7f55a40c49c0 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.373+0000 7f55a27fc640 1 -- 172.21.15.78:0/3684258366 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c02cb20 con 0x7f55a40c49c0 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.373+0000 7f55a27fc640 1 -- 172.21.15.78:0/3684258366 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c0343f0 con 0x7f55a40c49c0 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 -- 172.21.15.78:0/3684258366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40c49c0 msgr2=0x7f55a406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 --2- 172.21.15.78:0/3684258366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40c49c0 0x7f55a406e420 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f55a4072650 tx=0x7f559c02be20 comp rx=0 tx=0).stop 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 -- 172.21.15.78:0/3684258366 shutdown_connections 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 --2- 172.21.15.78:0/3684258366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40c49c0 0x7f55a406e420 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 --2- 172.21.15.78:0/3684258366 >> v2:172.21.15.150:3300/0 conn(0x7f55a40c29c0 0x7f55a40c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 --2- 172.21.15.78:0/3684258366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c25e0 0x7f55a41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 -- 172.21.15.78:0/3684258366 >> 172.21.15.78:0/3684258366 conn(0x7f55a406b270 msgr2=0x7f55a406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 -- 172.21.15.78:0/3684258366 shutdown_connections 2024-02-19T14:55:24.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.374+0000 7f55a9de4640 1 -- 172.21.15.78:0/3684258366 wait complete. 2024-02-19T14:55:24.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.375+0000 7f55a9de4640 1 Processor -- start 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.375+0000 7f55a9de4640 1 -- start start 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55a40c25e0 0x7f55a40a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c29c0 0x7f55a41eade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 0x7f55a4154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a4155240 con 0x7f55a40bfc20 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a4155a30 con 0x7f55a40c25e0 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a41544e0 con 0x7f55a40c29c0 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 0x7f55a4154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 0x7f55a4154970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44978/0 (socket says 172.21.15.78:44978) 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 -- 172.21.15.78:0/2914343240 learned_addr learned my addr 172.21.15.78:0/2914343240 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a2ffd640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c29c0 0x7f55a41eade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a37fe640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55a40c25e0 0x7f55a40a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 -- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c29c0 msgr2=0x7f55a41eade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c29c0 0x7f55a41eade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 -- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55a40c25e0 msgr2=0x7f55a40a9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55a40c25e0 0x7f55a40a9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 -- 172.21.15.78:0/2914343240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c009b30 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a3fff640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 0x7f55a4154970 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f559c036040 tx=0x7f559c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c009510 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c03e450 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a41eb320 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c03e800 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.376+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a4155470 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.377+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a4074990 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.378+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f559c03e960 con 0x7f55a40bfc20 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.378+0000 7f558bfff640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f559006b810 0x7f559006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.378+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f559c0acdf0 con 0x7f55a40bfc20 2024-02-19T14:55:24.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.378+0000 7f55a37fe640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f559006b810 0x7f559006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.379+0000 7f55a37fe640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f559006b810 0x7f559006dcd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f559400b240 tx=0x7f559400b3a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.383+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f559c077770 con 0x7f55a40bfc20 2024-02-19T14:55:24.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.589+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55a40c43e0 con 0x7f55a40bfc20 2024-02-19T14:55:24.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.590+0000 7f558bfff640 1 -- 172.21.15.78:0/2914343240 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f559c07ae30 con 0x7f55a40bfc20 2024-02-19T14:55:24.593 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:24.593 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7128,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1361734331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1361734331},{"type":"v1","addr":"172.21.15.78:6836","nonce":1361734331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:20.871770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5672":{"gid":5672,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:24.593 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 78 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f559006b810 msgr2=0x7f559006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f559006b810 0x7f559006dcd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f559400b240 tx=0x7f559400b3a0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 msgr2=0x7f55a4154970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 0x7f55a4154970 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f559c036040 tx=0x7f559c004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 shutdown_connections 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f559006b810 0x7f559006dcd0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55a40bfc20 0x7f55a4154970 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55a40c29c0 0x7f55a41eade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 --2- 172.21.15.78:0/2914343240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55a40c25e0 0x7f55a40a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 >> 172.21.15.78:0/2914343240 conn(0x7f55a406b270 msgr2=0x7f55a406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 shutdown_connections 2024-02-19T14:55:24.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.592+0000 7f55a9de4640 1 -- 172.21.15.78:0/2914343240 wait complete. 2024-02-19T14:55:24.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:24.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.703+0000 7f19d5ac9640 1 Processor -- start 2024-02-19T14:55:24.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.703+0000 7f19d5ac9640 1 -- start start 2024-02-19T14:55:24.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19d5ac9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19d00c2f50 0x7f19d0158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19d5ac9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d00c34a0 con 0x7f19d00c8500 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19d5ac9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d006c8e0 con 0x7f19d00c42e0 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19d5ac9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d006c090 con 0x7f19d00c2f50 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19cf7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f19d00c8500 0x7f19d0158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47072/0 (socket says 172.21.15.78:47072) 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19cf7fe640 1 -- 172.21.15.78:0/3820271679 learned_addr learned my addr 172.21.15.78:0/3820271679 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19cffff640 1 --2- 172.21.15.78:0/3820271679 >> v2:172.21.15.150:3300/0 conn(0x7f19d00c2f50 0x7f19d0158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19ce7fc640 1 -- 172.21.15.78:0/3820271679 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043141242 0 0) 0x7f19d00c34a0 con 0x7f19d00c8500 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19ce7fc640 1 -- 172.21.15.78:0/3820271679 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b8003520 con 0x7f19d00c8500 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19cffff640 1 -- 172.21.15.78:0/3820271679 >> v1:172.21.15.150:6789/0 conn(0x7f19d00c42e0 legacy=0x7f19d015d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19cffff640 1 -- 172.21.15.78:0/3820271679 >> v1:172.21.15.78:6789/0 conn(0x7f19d00c8500 legacy=0x7f19d0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.704+0000 7f19cffff640 1 -- 172.21.15.78:0/3820271679 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d006cbc0 con 0x7f19d00c2f50 2024-02-19T14:55:24.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.705+0000 7f19cffff640 1 --2- 172.21.15.78:0/3820271679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c2f50 0x7f19d0158d60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009d90 tx=0x7f19c802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c80070d989e34903 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.705+0000 7f19ce7fc640 1 -- 172.21.15.78:0/3820271679 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c802cc60 con 0x7f19d00c2f50 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.705+0000 7f19ce7fc640 1 -- 172.21.15.78:0/3820271679 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c802cdc0 con 0x7f19d00c2f50 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.705+0000 7f19ce7fc640 1 -- 172.21.15.78:0/3820271679 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c8034640 con 0x7f19d00c2f50 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 -- 172.21.15.78:0/3820271679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c2f50 msgr2=0x7f19d0158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/3820271679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c2f50 0x7f19d0158d60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009d90 tx=0x7f19c802beb0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 -- 172.21.15.78:0/3820271679 shutdown_connections 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/3820271679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c2f50 0x7f19d0158d60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 -- 172.21.15.78:0/3820271679 >> 172.21.15.78:0/3820271679 conn(0x7f19d00aa300 msgr2=0x7f19d00aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 -- 172.21.15.78:0/3820271679 shutdown_connections 2024-02-19T14:55:24.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.706+0000 7f19d5ac9640 1 -- 172.21.15.78:0/3820271679 wait complete. 2024-02-19T14:55:24.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.707+0000 7f19d5ac9640 1 Processor -- start 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.707+0000 7f19d5ac9640 1 -- start start 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.707+0000 7f19d5ac9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 0x7f19d01582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.707+0000 7f19d5ac9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c8500 0x7f19d01587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19d5ac9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d006f290 0x7f19d006f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19d5ac9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0156b10 con 0x7f19d00c2f50 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19d5ac9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d014d4c0 con 0x7f19d00c8500 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19d5ac9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0156680 con 0x7f19d006f290 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 0x7f19d01582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 0x7f19d01582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44986/0 (socket says 172.21.15.78:44986) 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 -- 172.21.15.78:0/2277413992 learned_addr learned my addr 172.21.15.78:0/2277413992 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19ceffd640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c8500 0x7f19d01587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cffff640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d006f290 0x7f19d006f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 -- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d006f290 msgr2=0x7f19d006f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d006f290 0x7f19d006f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 -- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c8500 msgr2=0x7f19d01587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c8500 0x7f19d01587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 -- 172.21.15.78:0/2277413992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8009a70 con 0x7f19d00c2f50 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19cf7fe640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 0x7f19d01582b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c440 tx=0x7f19c000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0005ce0 con 0x7f19d00c2f50 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c0005e40 con 0x7f19d00c2f50 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c00072b0 con 0x7f19d00c2f50 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d0158d30 con 0x7f19d00c2f50 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.708+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d01eb9b0 con 0x7f19d00c2f50 2024-02-19T14:55:24.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.709+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d0071c20 con 0x7f19d00c2f50 2024-02-19T14:55:24.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.710+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f19c0007410 con 0x7f19d00c2f50 2024-02-19T14:55:24.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.710+0000 7f19b7fff640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19bc06b810 0x7f19bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:24.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.710+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f19c0089800 con 0x7f19d00c2f50 2024-02-19T14:55:24.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.710+0000 7f19ceffd640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19bc06b810 0x7f19bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:24.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.711+0000 7f19ceffd640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19bc06b810 0x7f19bc06dcd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f19d00c8360 tx=0x7f19c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.714+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f19c0054240 con 0x7f19d00c2f50 2024-02-19T14:55:24.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.887+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d0074990 con 0x7f19d00c2f50 2024-02-19T14:55:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.888+0000 7f19b7fff640 1 -- 172.21.15.78:0/2277413992 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16990 (secure 0 0 0) 0x7f19c0057900 con 0x7f19d00c2f50 2024-02-19T14:55:24.891 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:24.891 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":99,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:18.858198+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:09.377887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:10.136982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3610724432":"2024-02-20T14:55:18.858155+0000","172.21.15.78:6834/3610724432":"2024-02-20T14:55:18.858155+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19bc06b810 msgr2=0x7f19bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19bc06b810 0x7f19bc06dcd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f19d00c8360 tx=0x7f19c4009040 comp rx=0 tx=0).stop 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 msgr2=0x7f19d01582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 0x7f19d01582b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c440 tx=0x7f19c000c900 comp rx=0 tx=0).stop 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 shutdown_connections 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f19bc06b810 0x7f19bc06dcd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d006f290 0x7f19d006f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d00c8500 0x7f19d01587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 --2- 172.21.15.78:0/2277413992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19d00c2f50 0x7f19d01582b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 >> 172.21.15.78:0/2277413992 conn(0x7f19d00aa300 msgr2=0x7f19d0154330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 shutdown_connections 2024-02-19T14:55:24.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:24.891+0000 7f19d5ac9640 1 -- 172.21.15.78:0/2277413992 wait complete. 2024-02-19T14:55:24.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:55:09.377887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:55:10.136982+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:55:24.905 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:55:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 Processor -- start 2024-02-19T14:55:25.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 -- start start 2024-02-19T14:55:25.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7eb4152700 0x7f7eb40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c8e00 0x7f7eb415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb41551f0 con 0x7f7eb40c5730 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb4155490 con 0x7f7eb4152700 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eba87b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4155770 con 0x7f7eb40c8e00 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eb37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7eb40c5730 0x7f7eb406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47088/0 (socket says 172.21.15.78:47088) 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.005+0000 7f7eb37fe640 1 -- 172.21.15.78:0/807024210 learned_addr learned my addr 172.21.15.78:0/807024210 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb3fff640 1 --2- 172.21.15.78:0/807024210 >> v2:172.21.15.150:3300/0 conn(0x7f7eb4152700 0x7f7eb40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb8e18640 1 --2- 172.21.15.78:0/807024210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c8e00 0x7f7eb415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb2ffd640 1 -- 172.21.15.78:0/807024210 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135972507 0 0) 0x7f7eb41551f0 con 0x7f7eb40c5730 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb2ffd640 1 -- 172.21.15.78:0/807024210 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c003520 con 0x7f7eb40c5730 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb8e18640 1 -- 172.21.15.78:0/807024210 >> v2:172.21.15.150:3300/0 conn(0x7f7eb4152700 msgr2=0x7f7eb40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb8e18640 1 --2- 172.21.15.78:0/807024210 >> v2:172.21.15.150:3300/0 conn(0x7f7eb4152700 0x7f7eb40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb8e18640 1 -- 172.21.15.78:0/807024210 >> v1:172.21.15.78:6789/0 conn(0x7f7eb40c5730 legacy=0x7f7eb406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb2ffd640 1 -- 172.21.15.78:0/807024210 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208071412 0 0) 0x7f7e9c003520 con 0x7f7eb40c5730 2024-02-19T14:55:25.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb8e18640 1 -- 172.21.15.78:0/807024210 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4155a50 con 0x7f7eb40c8e00 2024-02-19T14:55:25.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.006+0000 7f7eb3fff640 1 --2- 172.21.15.78:0/807024210 >> v2:172.21.15.150:3300/0 conn(0x7f7eb4152700 0x7f7eb40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eb8e18640 1 --2- 172.21.15.78:0/807024210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c8e00 0x7f7eb415f980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009d90 tx=0x7f7eac02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575e05a85e143c6c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eb2ffd640 1 -- 172.21.15.78:0/807024210 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac02cc50 con 0x7f7eb40c8e00 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eb2ffd640 1 -- 172.21.15.78:0/807024210 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac02cdb0 con 0x7f7eb40c8e00 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eb2ffd640 1 -- 172.21.15.78:0/807024210 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac034660 con 0x7f7eb40c8e00 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 -- 172.21.15.78:0/807024210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c8e00 msgr2=0x7f7eb415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 --2- 172.21.15.78:0/807024210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c8e00 0x7f7eb415f980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009d90 tx=0x7f7eac02bea0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 -- 172.21.15.78:0/807024210 shutdown_connections 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 --2- 172.21.15.78:0/807024210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c8e00 0x7f7eb415f980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 --2- 172.21.15.78:0/807024210 >> v2:172.21.15.150:3300/0 conn(0x7f7eb4152700 0x7f7eb40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 -- 172.21.15.78:0/807024210 >> 172.21.15.78:0/807024210 conn(0x7f7eb40cac90 msgr2=0x7f7eb40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.007+0000 7f7eba87b640 1 -- 172.21.15.78:0/807024210 shutdown_connections 2024-02-19T14:55:25.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.008+0000 7f7eba87b640 1 -- 172.21.15.78:0/807024210 wait complete. 2024-02-19T14:55:25.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.008+0000 7f7eba87b640 1 Processor -- start 2024-02-19T14:55:25.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 -- start start 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c5730 0x7f7eb40aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb40c8e00 0x7f7eb4154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 0x7f7eb40abf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40c10f0 con 0x7f7eb4152700 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40acbb0 con 0x7f7eb40c8e00 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eba87b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb41ee7d0 con 0x7f7eb40c5730 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eb8e18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 0x7f7eb40abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eb8e18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 0x7f7eb40abf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45000/0 (socket says 172.21.15.78:45000) 2024-02-19T14:55:25.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.009+0000 7f7eb8e18640 1 -- 172.21.15.78:0/3613377945 learned_addr learned my addr 172.21.15.78:0/3613377945 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb3fff640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c5730 0x7f7eb40aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb37fe640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb40c8e00 0x7f7eb4154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb8e18640 1 -- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c5730 msgr2=0x7f7eb40aad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb8e18640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c5730 0x7f7eb40aad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb8e18640 1 -- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb40c8e00 msgr2=0x7f7eb4154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb8e18640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb40c8e00 0x7f7eb4154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb8e18640 1 -- 172.21.15.78:0/3613377945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea4009400 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb8e18640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 0x7f7eb40abf20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009d90 tx=0x7f7eac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac040650 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac0407b0 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac040b60 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac009a70 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.010+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb41eeab0 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.011+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb40ad090 con 0x7f7eb4152700 2024-02-19T14:55:25.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.012+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7eac040cc0 con 0x7f7eb4152700 2024-02-19T14:55:25.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.012+0000 7f7eb0ff9640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ea006b840 0x7f7ea006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.012+0000 7f7eb3fff640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ea006b840 0x7f7ea006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.012+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7eac0acbd0 con 0x7f7eb4152700 2024-02-19T14:55:25.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.012+0000 7f7eb3fff640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ea006b840 0x7f7ea006dd00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7eb41512a0 tx=0x7f7ea4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.016+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7eb40ad090 con 0x7f7eb4152700 2024-02-19T14:55:25.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.188+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7eb40ad090 con 0x7f7eb4152700 2024-02-19T14:55:25.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.188+0000 7f7eb0ff9640 1 -- 172.21.15.78:0/3613377945 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7eb40ad090 con 0x7f7eb4152700 2024-02-19T14:55:25.191 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:55:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.190+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ea006b840 msgr2=0x7f7ea006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.190+0000 7f7eba87b640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ea006b840 0x7f7ea006dd00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7eb41512a0 tx=0x7f7ea4009190 comp rx=0 tx=0).stop 2024-02-19T14:55:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 msgr2=0x7f7eb40abf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 0x7f7eb40abf20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009d90 tx=0x7f7eac004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 shutdown_connections 2024-02-19T14:55:25.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ea006b840 0x7f7ea006dd00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7eb4152700 0x7f7eb40abf20 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb40c8e00 0x7f7eb4154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 --2- 172.21.15.78:0/3613377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40c5730 0x7f7eb40aad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 >> 172.21.15.78:0/3613377945 conn(0x7f7eb40cac90 msgr2=0x7f7eb414cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 shutdown_connections 2024-02-19T14:55:25.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.191+0000 7f7eba87b640 1 -- 172.21.15.78:0/3613377945 wait complete. 2024-02-19T14:55:25.204 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:55:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.303+0000 7fd38bd83640 1 Processor -- start 2024-02-19T14:55:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd38bd83640 1 -- start start 2024-02-19T14:55:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd38bd83640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd3840ad890 0x7fd3840c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd38bd83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840ab600 0x7fd3840c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd38bd83640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3840aba60 con 0x7fd3840c49c0 2024-02-19T14:55:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd38bd83640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd384152d30 con 0x7fd3840ad890 2024-02-19T14:55:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd38bd83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840c7860 con 0x7fd3840ab600 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd389b1f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd3840ad890 0x7fd3840c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.304+0000 7fd389b1f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd3840ad890 0x7fd3840c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45010/0 (socket says 172.21.15.78:45010) 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd389b1f640 1 -- 172.21.15.78:0/892159649 learned_addr learned my addr 172.21.15.78:0/892159649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd38a320640 1 --2- 172.21.15.78:0/892159649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840ab600 0x7fd3840c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd389b1f640 1 -- 172.21.15.78:0/892159649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840ab600 msgr2=0x7fd3840c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd389b1f640 1 --2- 172.21.15.78:0/892159649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840ab600 0x7fd3840c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd389b1f640 1 -- 172.21.15.78:0/892159649 >> v1:172.21.15.150:6789/0 conn(0x7fd3840c49c0 legacy=0x7fd3841534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd389b1f640 1 -- 172.21.15.78:0/892159649 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3840c7ce0 con 0x7fd3840ad890 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.305+0000 7fd389b1f640 1 --2- 172.21.15.78:0/892159649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3840c64f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd3840c96d0 tx=0x7fd38002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e17d836868915d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd388b1d640 1 -- 172.21.15.78:0/892159649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380031520 con 0x7fd3840ad890 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd388b1d640 1 -- 172.21.15.78:0/892159649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd380031680 con 0x7fd3840ad890 2024-02-19T14:55:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd388b1d640 1 -- 172.21.15.78:0/892159649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd380031a10 con 0x7fd3840ad890 2024-02-19T14:55:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 -- 172.21.15.78:0/892159649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 msgr2=0x7fd3840c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 --2- 172.21.15.78:0/892159649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3840c64f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd3840c96d0 tx=0x7fd38002b7c0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 -- 172.21.15.78:0/892159649 shutdown_connections 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 --2- 172.21.15.78:0/892159649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840ab600 0x7fd3840c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 --2- 172.21.15.78:0/892159649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3840c64f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 -- 172.21.15.78:0/892159649 >> 172.21.15.78:0/892159649 conn(0x7fd38406b700 msgr2=0x7fd3840aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 -- 172.21.15.78:0/892159649 shutdown_connections 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.306+0000 7fd38bd83640 1 -- 172.21.15.78:0/892159649 wait complete. 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.307+0000 7fd38bd83640 1 Processor -- start 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.307+0000 7fd38bd83640 1 -- start start 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38bd83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3840ab600 0x7fd3840705a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38bd83640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3841e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38bd83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840c49c0 0x7fd3841e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38bd83640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3841dd410 con 0x7fd3840ad890 2024-02-19T14:55:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38bd83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3841dd130 con 0x7fd3840ab600 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38bd83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3841ec4c0 con 0x7fd3840c49c0 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3841e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3841e8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45020/0 (socket says 172.21.15.78:45020) 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 -- 172.21.15.78:0/1498885447 learned_addr learned my addr 172.21.15.78:0/1498885447 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38a320640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840c49c0 0x7fd3841e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd389b1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3840ab600 0x7fd3840705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 -- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840c49c0 msgr2=0x7fd3841e92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840c49c0 0x7fd3841e92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 -- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3840ab600 msgr2=0x7fd3840705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3840ab600 0x7fd3840705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd38931e640 1 -- 172.21.15.78:0/1498885447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800094d0 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.308+0000 7fd389b1f640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3840ab600 0x7fd3840705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.309+0000 7fd38931e640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3841e8db0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd37400c410 tx=0x7fd37400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.309+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3740035f0 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.309+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd374003750 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.309+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3841ec7a0 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.309+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd374003b40 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.309+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd384070ae0 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.310+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3841ec320 con 0x7fd3840ad890 2024-02-19T14:55:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.310+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd374003ca0 con 0x7fd3840ad890 2024-02-19T14:55:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.310+0000 7fd37a7fc640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd36806b810 0x7fd36806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.311+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd374089940 con 0x7fd3840ad890 2024-02-19T14:55:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.311+0000 7fd389b1f640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd36806b810 0x7fd36806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.311+0000 7fd389b1f640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd36806b810 0x7fd36806dcd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd380035000 tx=0x7fd380030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.315+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd3841ec320 con 0x7fd3840ad890 2024-02-19T14:55:25.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.484+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3841eccc0 con 0x7fd3840ad890 2024-02-19T14:55:25.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.484+0000 7fd37a7fc640 1 -- 172.21.15.78:0/1498885447 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3841eccc0 con 0x7fd3840ad890 2024-02-19T14:55:25.487 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:55:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd36806b810 msgr2=0x7fd36806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd36806b810 0x7fd36806dcd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd380035000 tx=0x7fd380030000 comp rx=0 tx=0).stop 2024-02-19T14:55:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 msgr2=0x7fd3841e8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3841e8db0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fd37400c410 tx=0x7fd37400c8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 shutdown_connections 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd36806b810 0x7fd36806dcd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3840ad890 0x7fd3841e8db0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3840c49c0 0x7fd3841e92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 --2- 172.21.15.78:0/1498885447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3840ab600 0x7fd3840705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 >> 172.21.15.78:0/1498885447 conn(0x7fd38406b700 msgr2=0x7fd3840a8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 shutdown_connections 2024-02-19T14:55:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.486+0000 7fd38bd83640 1 -- 172.21.15.78:0/1498885447 wait complete. 2024-02-19T14:55:25.500 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:55:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f814311e640 1 Processor -- start 2024-02-19T14:55:25.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f814311e640 1 -- start start 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f814311e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f814311e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c06fe40 con 0x7f813c0c29e0 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f814311e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c06f880 con 0x7f813c0c8b60 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f814311e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0c32c0 con 0x7f813c0713a0 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.601+0000 7f813bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f813c0c29e0 0x7f813c0c64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47100/0 (socket says 172.21.15.78:47100) 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f813bfff640 1 -- 172.21.15.78:0/3966042331 learned_addr learned my addr 172.21.15.78:0/3966042331 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f8140eba640 1 --2- 172.21.15.78:0/3966042331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f813b7fe640 1 -- 172.21.15.78:0/3966042331 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716303091 0 0) 0x7f813c06fe40 con 0x7f813c0c29e0 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f813b7fe640 1 -- 172.21.15.78:0/3966042331 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c003520 con 0x7f813c0c29e0 2024-02-19T14:55:25.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f8140eba640 1 -- 172.21.15.78:0/3966042331 >> v1:172.21.15.150:6789/0 conn(0x7f813c0c8b60 legacy=0x7f813c1536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f813b7fe640 1 -- 172.21.15.78:0/3966042331 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895921774 0 0) 0x7f813c06f880 con 0x7f813c0c8b60 2024-02-19T14:55:25.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f8140eba640 1 -- 172.21.15.78:0/3966042331 >> v1:172.21.15.78:6789/0 conn(0x7f813c0c29e0 legacy=0x7f813c0c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.602+0000 7f8140eba640 1 -- 172.21.15.78:0/3966042331 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c06fb60 con 0x7f813c0713a0 2024-02-19T14:55:25.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f8140eba640 1 --2- 172.21.15.78:0/3966042331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c071780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81300097d0 tx=0x7f813002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=983450254424c302 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f813b7fe640 1 -- 172.21.15.78:0/3966042331 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130031800 con 0x7f813c0713a0 2024-02-19T14:55:25.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f813b7fe640 1 -- 172.21.15.78:0/3966042331 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8130031960 con 0x7f813c0713a0 2024-02-19T14:55:25.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f813b7fe640 1 -- 172.21.15.78:0/3966042331 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130031cf0 con 0x7f813c0713a0 2024-02-19T14:55:25.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 -- 172.21.15.78:0/3966042331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 msgr2=0x7f813c071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 --2- 172.21.15.78:0/3966042331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c071780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f81300097d0 tx=0x7f813002b8a0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 -- 172.21.15.78:0/3966042331 shutdown_connections 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 --2- 172.21.15.78:0/3966042331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c071780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 -- 172.21.15.78:0/3966042331 >> 172.21.15.78:0/3966042331 conn(0x7f813c0aa300 msgr2=0x7f813c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 -- 172.21.15.78:0/3966042331 shutdown_connections 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.603+0000 7f814311e640 1 -- 172.21.15.78:0/3966042331 wait complete. 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.604+0000 7f814311e640 1 Processor -- start 2024-02-19T14:55:25.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.604+0000 7f814311e640 1 -- start start 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f814311e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c0c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f814311e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f813c0c29e0 0x7f813c14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f814311e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 0x7f813c0beea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f814311e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0bfe60 con 0x7f813c0c8b60 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f814311e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0c24e0 con 0x7f813c0c29e0 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f814311e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0c26f0 con 0x7f813c0713a0 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f81416bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 0x7f813c0beea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f81416bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 0x7f813c0beea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45024/0 (socket says 172.21.15.78:45024) 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f81416bb640 1 -- 172.21.15.78:0/616610746 learned_addr learned my addr 172.21.15.78:0/616610746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f8140eba640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c0c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.605+0000 7f813bfff640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f813c0c29e0 0x7f813c14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81416bb640 1 -- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 msgr2=0x7f813c0c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81416bb640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c0c1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81416bb640 1 -- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f813c0c29e0 msgr2=0x7f813c14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81416bb640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f813c0c29e0 0x7f813c14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81416bb640 1 -- 172.21.15.78:0/616610746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130009400 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81416bb640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 0x7f813c0beea0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f813400c7b0 tx=0x7f813400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8134003ba0 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8134003d00 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c1ec480 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8134005270 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.606+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c1ec910 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.607+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c1ec610 con 0x7f813c0c8b60 2024-02-19T14:55:25.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.607+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81340053d0 con 0x7f813c0c8b60 2024-02-19T14:55:25.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.608+0000 7f81397fa640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f812006b810 0x7f812006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.608+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8134089f50 con 0x7f813c0c8b60 2024-02-19T14:55:25.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.608+0000 7f8140eba640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f812006b810 0x7f812006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.608+0000 7f8140eba640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f812006b810 0x7f812006dcd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8130036000 tx=0x7f813003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.612+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81340548d0 con 0x7f813c0c8b60 2024-02-19T14:55:25.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.782+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f813c074990 con 0x7f813c0c8b60 2024-02-19T14:55:25.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.782+0000 7f81397fa640 1 -- 172.21.15.78:0/616610746 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8134057f90 con 0x7f813c0c8b60 2024-02-19T14:55:25.785 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:55:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f812006b810 msgr2=0x7f812006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f812006b810 0x7f812006dcd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8130036000 tx=0x7f813003d040 comp rx=0 tx=0).stop 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 msgr2=0x7f813c0beea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 0x7f813c0beea0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f813400c7b0 tx=0x7f813400cbb0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 shutdown_connections 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f812006b810 0x7f812006dcd0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f813c0c8b60 0x7f813c0beea0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f813c0c29e0 0x7f813c14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 --2- 172.21.15.78:0/616610746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f813c0713a0 0x7f813c0c1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.784+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 >> 172.21.15.78:0/616610746 conn(0x7f813c0aa300 msgr2=0x7f813c1536b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.785+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 shutdown_connections 2024-02-19T14:55:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.785+0000 7f814311e640 1 -- 172.21.15.78:0/616610746 wait complete. 2024-02-19T14:55:25.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.895+0000 7f172592e640 1 Processor -- start 2024-02-19T14:55:25.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.895+0000 7f172592e640 1 -- start start 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f172592e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1720152480 0x7f17201559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f172592e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17200c87a0 0x7f17200c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f172592e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1720155ef0 con 0x7f17200c7070 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f172592e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172006dec0 con 0x7f1720152480 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f172592e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172006d520 con 0x7f17200c87a0 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17200c87a0 0x7f17200c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171e7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f17200c7070 0x7f172006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57882/0 (socket says 172.21.15.78:57882) 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1720152480 0x7f17201559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171e7fc640 1 -- 172.21.15.78:0/96136772 learned_addr learned my addr 172.21.15.78:0/96136772 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171dffb640 1 -- 172.21.15.78:0/96136772 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132798420 0 0) 0x7f1714001110 con 0x7f17200c7070 2024-02-19T14:55:25.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171dffb640 1 -- 172.21.15.78:0/96136772 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1708003520 con 0x7f17200c7070 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171f7fe640 1 -- 172.21.15.78:0/96136772 >> v2:172.21.15.150:3300/0 conn(0x7f1720152480 msgr2=0x7f17201559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171f7fe640 1 --2- 172.21.15.78:0/96136772 >> v2:172.21.15.150:3300/0 conn(0x7f1720152480 0x7f17201559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171f7fe640 1 -- 172.21.15.78:0/96136772 >> v1:172.21.15.150:6789/0 conn(0x7f17200c7070 legacy=0x7f172006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171f7fe640 1 -- 172.21.15.78:0/96136772 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17200c75c0 con 0x7f17200c87a0 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.896+0000 7f171effd640 1 --2- 172.21.15.78:0/96136772 >> v2:172.21.15.150:3300/0 conn(0x7f1720152480 0x7f17201559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.897+0000 7f171f7fe640 1 --2- 172.21.15.78:0/96136772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17200c87a0 0x7f17200c5220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1718009d90 tx=0x7f171802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55c4947862f7bfa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.897+0000 7f171dffb640 1 -- 172.21.15.78:0/96136772 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171802cc50 con 0x7f17200c87a0 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.897+0000 7f171dffb640 1 -- 172.21.15.78:0/96136772 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171802cdb0 con 0x7f17200c87a0 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.897+0000 7f171dffb640 1 -- 172.21.15.78:0/96136772 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1718034660 con 0x7f17200c87a0 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 -- 172.21.15.78:0/96136772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17200c87a0 msgr2=0x7f17200c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 --2- 172.21.15.78:0/96136772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17200c87a0 0x7f17200c5220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1718009d90 tx=0x7f171802bea0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 -- 172.21.15.78:0/96136772 shutdown_connections 2024-02-19T14:55:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 --2- 172.21.15.78:0/96136772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17200c87a0 0x7f17200c5220 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 --2- 172.21.15.78:0/96136772 >> v2:172.21.15.150:3300/0 conn(0x7f1720152480 0x7f17201559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 -- 172.21.15.78:0/96136772 >> 172.21.15.78:0/96136772 conn(0x7f17200ca8a0 msgr2=0x7f17200cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 -- 172.21.15.78:0/96136772 shutdown_connections 2024-02-19T14:55:25.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.898+0000 7f172592e640 1 -- 172.21.15.78:0/96136772 wait complete. 2024-02-19T14:55:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.899+0000 7f172592e640 1 Processor -- start 2024-02-19T14:55:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.899+0000 7f172592e640 1 -- start start 2024-02-19T14:55:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f172592e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 0x7f17200ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f172592e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17200c87a0 0x7f17201ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f172592e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1720152480 0x7f17201ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f172592e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1720157ed0 con 0x7f17200c7070 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f172592e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17201eaba0 con 0x7f17200c87a0 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f172592e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17200af080 con 0x7f1720152480 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 0x7f17200ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 0x7f17200ac5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45038/0 (socket says 172.21.15.78:45038) 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 -- 172.21.15.78:0/2251032127 learned_addr learned my addr 172.21.15.78:0/2251032127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17200c87a0 0x7f17201ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171f7fe640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1720152480 0x7f17201ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 -- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1720152480 msgr2=0x7f17201ee930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1720152480 0x7f17201ee930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 -- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17200c87a0 msgr2=0x7f17201ee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17200c87a0 0x7f17201ee3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.900+0000 7f171effd640 1 -- 172.21.15.78:0/2251032127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1710009400 con 0x7f17200c7070 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.901+0000 7f171effd640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 0x7f17200ac5f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f171000e990 tx=0x7f171000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.901+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171000c9e0 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.901+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171000cb40 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.901+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1718009a70 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.901+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171000ced0 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.901+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17200ad640 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.902+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1720074990 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.902+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1710013470 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.902+0000 7f1707fff640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170c06b810 0x7f170c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.903+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f171008a5c0 con 0x7f17200c7070 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.903+0000 7f171e7fc640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170c06b810 0x7f170c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:25.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.903+0000 7f171e7fc640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170c06b810 0x7f170c06dcd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f17201562f0 tx=0x7f1714007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:25.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:25.907+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1710054eb0 con 0x7f17200c7070 2024-02-19T14:55:26.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:55:26.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:55:26.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.117+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17200ad4a0 con 0x7f17200c7070 2024-02-19T14:55:26.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.118+0000 7f1707fff640 1 -- 172.21.15.78:0/2251032127 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f17200ad4a0 con 0x7f17200c7070 2024-02-19T14:55:26.120 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:26.120 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7128,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1361734331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1361734331},{"type":"v1","addr":"172.21.15.78:6836","nonce":1361734331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:20.871770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5672":{"gid":5672,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:26.121 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 78 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170c06b810 msgr2=0x7f170c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170c06b810 0x7f170c06dcd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f17201562f0 tx=0x7f1714007b70 comp rx=0 tx=0).stop 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 msgr2=0x7f17200ac5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 0x7f17200ac5f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f171000e990 tx=0x7f171000ee50 comp rx=0 tx=0).stop 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 shutdown_connections 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170c06b810 0x7f170c06dcd0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1720152480 0x7f17201ee930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17200c87a0 0x7f17201ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 --2- 172.21.15.78:0/2251032127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17200c7070 0x7f17200ac5f0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 >> 172.21.15.78:0/2251032127 conn(0x7f17200ca8a0 msgr2=0x7f1720152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 shutdown_connections 2024-02-19T14:55:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.120+0000 7f172592e640 1 -- 172.21.15.78:0/2251032127 wait complete. 2024-02-19T14:55:26.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:26.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.230+0000 7f18e6a76640 1 Processor -- start 2024-02-19T14:55:26.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.230+0000 7f18e6a76640 1 -- start start 2024-02-19T14:55:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e6a76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e0153b60 0x7f18e00ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e6a76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f18e00c4350 0x7f18e006ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e6a76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e00c35e0 con 0x7f18e00c3200 2024-02-19T14:55:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e6a76640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e014d3c0 con 0x7f18e00c4350 2024-02-19T14:55:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e6a76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00acad0 con 0x7f18e0153b60 2024-02-19T14:55:26.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f18e00c4350 0x7f18e006ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f18e00c4350 0x7f18e006ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45054/0 (socket says 172.21.15.78:45054) 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e4812640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f18e00c3200 0x7f18e00718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47110/0 (socket says 172.21.15.78:47110) 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 -- 172.21.15.78:0/3985019934 learned_addr learned my addr 172.21.15.78:0/3985019934 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18dffff640 1 --2- 172.21.15.78:0/3985019934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e0153b60 0x7f18e00ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 -- 172.21.15.78:0/3985019934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e0153b60 msgr2=0x7f18e00ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 --2- 172.21.15.78:0/3985019934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e0153b60 0x7f18e00ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 -- 172.21.15.78:0/3985019934 >> v1:172.21.15.78:6789/0 conn(0x7f18e00c3200 legacy=0x7f18e00718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18e5013640 1 -- 172.21.15.78:0/3985019934 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00ad000 con 0x7f18e00c4350 2024-02-19T14:55:26.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.231+0000 7f18df7fe640 1 -- 172.21.15.78:0/3985019934 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296162129 0 0) 0x7f18e00c35e0 con 0x7f18e00c3200 2024-02-19T14:55:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.232+0000 7f18e5013640 1 --2- 172.21.15.78:0/3985019934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e00c4350 0x7f18e006ae80 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009e50 tx=0x7f18d802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd269e26104e58f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.232+0000 7f18df7fe640 1 -- 172.21.15.78:0/3985019934 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d802cb00 con 0x7f18e00c4350 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.232+0000 7f18df7fe640 1 -- 172.21.15.78:0/3985019934 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d802cc60 con 0x7f18e00c4350 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.232+0000 7f18df7fe640 1 -- 172.21.15.78:0/3985019934 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d80343f0 con 0x7f18e00c4350 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 -- 172.21.15.78:0/3985019934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e00c4350 msgr2=0x7f18e006ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3985019934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e00c4350 0x7f18e006ae80 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f18d8009e50 tx=0x7f18d802bf60 comp rx=0 tx=0).stop 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 -- 172.21.15.78:0/3985019934 shutdown_connections 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3985019934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e00c4350 0x7f18e006ae80 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3985019934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e0153b60 0x7f18e00ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 -- 172.21.15.78:0/3985019934 >> 172.21.15.78:0/3985019934 conn(0x7f18e00c7800 msgr2=0x7f18e00c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 -- 172.21.15.78:0/3985019934 shutdown_connections 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.233+0000 7f18e6a76640 1 -- 172.21.15.78:0/3985019934 wait complete. 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.234+0000 7f18e6a76640 1 Processor -- start 2024-02-19T14:55:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.234+0000 7f18e6a76640 1 -- start start 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e6a76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e00c3200 0x7f18e00c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e6a76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e00c4350 0x7f18e01445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e6a76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 0x7f18e0144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e6a76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e0145070 con 0x7f18e0153b60 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e6a76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e00c6880 con 0x7f18e00c4350 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e6a76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e01ee490 con 0x7f18e00c3200 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 0x7f18e0144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 0x7f18e0144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45064/0 (socket says 172.21.15.78:45064) 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 -- 172.21.15.78:0/3972029168 learned_addr learned my addr 172.21.15.78:0/3972029168 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e4812640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e00c3200 0x7f18e00c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18dffff640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e00c4350 0x7f18e01445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 -- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e00c3200 msgr2=0x7f18e00c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e00c3200 0x7f18e00c0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 -- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e00c4350 msgr2=0x7f18e01445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e00c4350 0x7f18e01445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 -- 172.21.15.78:0/3972029168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8009b30 con 0x7f18e0153b60 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18dffff640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e00c4350 0x7f18e01445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18e5013640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 0x7f18e0144b30 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f18d8036040 tx=0x7f18d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.235+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8009510 con 0x7f18e0153b60 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.236+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d803e450 con 0x7f18e0153b60 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.236+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e01ee770 con 0x7f18e0153b60 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.236+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d803e7e0 con 0x7f18e0153b60 2024-02-19T14:55:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.236+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e00a8840 con 0x7f18e0153b60 2024-02-19T14:55:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.237+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e0074990 con 0x7f18e0153b60 2024-02-19T14:55:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.237+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f18d803e940 con 0x7f18e0153b60 2024-02-19T14:55:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.237+0000 7f18dd7fa640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18cc06b810 0x7f18cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.237+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f18d80acf70 con 0x7f18e0153b60 2024-02-19T14:55:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.237+0000 7f18e4812640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18cc06b810 0x7f18cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.238+0000 7f18e4812640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18cc06b810 0x7f18cc06dcd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f18d0000f60 tx=0x7f18d0007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.242+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f18d80e0dc0 con 0x7f18e0153b60 2024-02-19T14:55:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.414+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18e00c4130 con 0x7f18e0153b60 2024-02-19T14:55:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.415+0000 7f18dd7fa640 1 -- 172.21.15.78:0/3972029168 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16990 (secure 0 0 0) 0x7f18d8077840 con 0x7f18e0153b60 2024-02-19T14:55:26.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:26.420 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":99,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:18.858198+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:09.377887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:10.136982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3610724432":"2024-02-20T14:55:18.858155+0000","172.21.15.78:6834/3610724432":"2024-02-20T14:55:18.858155+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:26.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18cc06b810 msgr2=0x7f18cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18cc06b810 0x7f18cc06dcd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f18d0000f60 tx=0x7f18d0007d50 comp rx=0 tx=0).stop 2024-02-19T14:55:26.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 msgr2=0x7f18e0144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 0x7f18e0144b30 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f18d8036040 tx=0x7f18d8004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:26.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 shutdown_connections 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18cc06b810 0x7f18cc06dcd0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18e0153b60 0x7f18e0144b30 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e00c4350 0x7f18e01445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 --2- 172.21.15.78:0/3972029168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18e00c3200 0x7f18e00c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 >> 172.21.15.78:0/3972029168 conn(0x7f18e00c7800 msgr2=0x7f18e014ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 shutdown_connections 2024-02-19T14:55:26.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.417+0000 7f18e6a76640 1 -- 172.21.15.78:0/3972029168 wait complete. 2024-02-19T14:55:26.430 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:55:26.431 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:26.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.527+0000 7fcba735e640 1 Processor -- start 2024-02-19T14:55:26.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 -- start start 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba0152700 0x7fcba00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcba00c5730 0x7fcba006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcba00c8e00 0x7fcba006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba006cd20 con 0x7fcba00c5730 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba01551f0 con 0x7fcba00c8e00 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba735e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba0155490 con 0x7fcba0152700 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba48f9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcba00c5730 0x7fcba006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba48f9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcba00c5730 0x7fcba006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45070/0 (socket says 172.21.15.78:45070) 2024-02-19T14:55:26.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba48f9640 1 -- 172.21.15.78:0/3971372387 learned_addr learned my addr 172.21.15.78:0/3971372387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba50fa640 1 --2- 172.21.15.78:0/3971372387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba0152700 0x7fcba00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.528+0000 7fcba58fb640 1 --2- 172.21.15.78:0/3971372387 >> v2:172.21.15.150:3300/0 conn(0x7fcba00c8e00 0x7fcba006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.529+0000 7fcba48f9640 1 -- 172.21.15.78:0/3971372387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba0152700 msgr2=0x7fcba00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.529+0000 7fcba48f9640 1 --2- 172.21.15.78:0/3971372387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba0152700 0x7fcba00c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.529+0000 7fcba48f9640 1 -- 172.21.15.78:0/3971372387 >> v2:172.21.15.150:3300/0 conn(0x7fcba00c8e00 msgr2=0x7fcba006c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.529+0000 7fcba48f9640 1 --2- 172.21.15.78:0/3971372387 >> v2:172.21.15.150:3300/0 conn(0x7fcba00c8e00 0x7fcba006c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.529+0000 7fcba48f9640 1 -- 172.21.15.78:0/3971372387 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0155770 con 0x7fcba00c5730 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.529+0000 7fcba48f9640 1 --2- 172.21.15.78:0/3971372387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c5730 0x7fcba006c2a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fcba00c8bc0 tx=0x7fcb9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcfa476ff2f21b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcb97fff640 1 -- 172.21.15.78:0/3971372387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90031870 con 0x7fcba00c5730 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcb97fff640 1 -- 172.21.15.78:0/3971372387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb900319d0 con 0x7fcba00c5730 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcb97fff640 1 -- 172.21.15.78:0/3971372387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90031d60 con 0x7fcba00c5730 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 -- 172.21.15.78:0/3971372387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c5730 msgr2=0x7fcba006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 --2- 172.21.15.78:0/3971372387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c5730 0x7fcba006c2a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fcba00c8bc0 tx=0x7fcb9002b910 comp rx=0 tx=0).stop 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 -- 172.21.15.78:0/3971372387 shutdown_connections 2024-02-19T14:55:26.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 --2- 172.21.15.78:0/3971372387 >> v2:172.21.15.150:3300/0 conn(0x7fcba00c8e00 0x7fcba006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 --2- 172.21.15.78:0/3971372387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c5730 0x7fcba006c2a0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 --2- 172.21.15.78:0/3971372387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba0152700 0x7fcba00c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 -- 172.21.15.78:0/3971372387 >> 172.21.15.78:0/3971372387 conn(0x7fcba00cac90 msgr2=0x7fcba00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 -- 172.21.15.78:0/3971372387 shutdown_connections 2024-02-19T14:55:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.530+0000 7fcba735e640 1 -- 172.21.15.78:0/3971372387 wait complete. 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.531+0000 7fcba735e640 1 Processor -- start 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.531+0000 7fcba735e640 1 -- start start 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba735e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba00c5730 0x7fcba00c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba735e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 0x7fcba00b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba735e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcba0152700 0x7fcba01ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba735e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00b0600 con 0x7fcba00c8e00 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba735e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba01ea5d0 con 0x7fcba0152700 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba735e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c3d20 con 0x7fcba00c5730 2024-02-19T14:55:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 0x7fcba00b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 0x7fcba00b00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45072/0 (socket says 172.21.15.78:45072) 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba50fa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba00c5730 0x7fcba00c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 -- 172.21.15.78:0/1650936516 learned_addr learned my addr 172.21.15.78:0/1650936516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba58fb640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcba0152700 0x7fcba01ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 -- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba00c5730 msgr2=0x7fcba00c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba00c5730 0x7fcba00c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 -- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcba0152700 msgr2=0x7fcba01ee430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcba0152700 0x7fcba01ee430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.532+0000 7fcba48f9640 1 -- 172.21.15.78:0/1650936516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900094d0 con 0x7fcba00c8e00 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.533+0000 7fcba48f9640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 0x7fcba00b00c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.533+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9003ac60 con 0x7fcba00c8e00 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.533+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9003adc0 con 0x7fcba00c8e00 2024-02-19T14:55:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.533+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba00ac750 con 0x7fcba00c8e00 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.533+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90044630 con 0x7fcba00c8e00 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.533+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba00acfb0 con 0x7fcba00c8e00 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.534+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba01ea760 con 0x7fcba00c8e00 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.534+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcb9004d450 con 0x7fcba00c8e00 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.535+0000 7fcb95ffb640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb8406b840 0x7fcb8406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.535+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcb9002d040 con 0x7fcba00c8e00 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.535+0000 7fcba50fa640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb8406b840 0x7fcb8406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.535+0000 7fcba50fa640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb8406b840 0x7fcb8406dd00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c011340 tx=0x7fcb9c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.539+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcb90077040 con 0x7fcba00c8e00 2024-02-19T14:55:26.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.744+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcba0074990 con 0x7fcba00c8e00 2024-02-19T14:55:26.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.745+0000 7fcb95ffb640 1 -- 172.21.15.78:0/1650936516 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7fcb9007a700 con 0x7fcba00c8e00 2024-02-19T14:55:26.748 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:26.748 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7128,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1361734331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1361734331},{"type":"v1","addr":"172.21.15.78:6836","nonce":1361734331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:20.871770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5672":{"gid":5672,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:26.748 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 78 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb8406b840 msgr2=0x7fcb8406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb8406b840 0x7fcb8406dd00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c011340 tx=0x7fcb9c00b040 comp rx=0 tx=0).stop 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 msgr2=0x7fcba00b00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 0x7fcba00b00c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb900046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 shutdown_connections 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb8406b840 0x7fcb8406dd00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcba00c8e00 0x7fcba00b00c0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcba0152700 0x7fcba01ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 --2- 172.21.15.78:0/1650936516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcba00c5730 0x7fcba00c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 >> 172.21.15.78:0/1650936516 conn(0x7fcba00cac90 msgr2=0x7fcba0150930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 shutdown_connections 2024-02-19T14:55:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.747+0000 7fcba735e640 1 -- 172.21.15.78:0/1650936516 wait complete. 2024-02-19T14:55:26.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:26.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.859+0000 7f2d16c5c640 1 Processor -- start 2024-02-19T14:55:26.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.859+0000 7f2d16c5c640 1 -- start start 2024-02-19T14:55:26.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.859+0000 7f2d16c5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2d10155070 0x7f2d10155500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.859+0000 7f2d16c5c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d10159130 con 0x7f2d100718f0 2024-02-19T14:55:26.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d16c5c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d100ad030 con 0x7f2d100acc50 2024-02-19T14:55:26.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d16c5c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d100c2cf0 con 0x7f2d10155070 2024-02-19T14:55:26.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d0ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2d10155070 0x7f2d10155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d151f9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2d100718f0 0x7f2d10071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47122/0 (socket says 172.21.15.78:47122) 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d0ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2d10155070 0x7f2d10155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45074/0 (socket says 172.21.15.78:45074) 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d151f9640 1 -- 172.21.15.78:0/2485867165 learned_addr learned my addr 172.21.15.78:0/2485867165 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d0ffff640 1 -- 172.21.15.78:0/2485867165 >> v1:172.21.15.150:6789/0 conn(0x7f2d100acc50 legacy=0x7f2d100c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d0ffff640 1 -- 172.21.15.78:0/2485867165 >> v1:172.21.15.78:6789/0 conn(0x7f2d100718f0 legacy=0x7f2d10071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/2485867165 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537992725 0 0) 0x7f2d10159130 con 0x7f2d100718f0 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.860+0000 7f2d0ffff640 1 -- 172.21.15.78:0/2485867165 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d101595c0 con 0x7f2d10155070 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d0ffff640 1 --2- 172.21.15.78:0/2485867165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10155070 0x7f2d10155500 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0097a0 tx=0x7f2cfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64872327b60f0f47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/2485867165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc031830 con 0x7f2d10155070 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/2485867165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc031990 con 0x7f2d10155070 2024-02-19T14:55:26.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/2485867165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc031d20 con 0x7f2d10155070 2024-02-19T14:55:26.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 -- 172.21.15.78:0/2485867165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10155070 msgr2=0x7f2d10155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/2485867165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10155070 0x7f2d10155500 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0097a0 tx=0x7f2cfc02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 -- 172.21.15.78:0/2485867165 shutdown_connections 2024-02-19T14:55:26.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/2485867165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10155070 0x7f2d10155500 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 -- 172.21.15.78:0/2485867165 >> 172.21.15.78:0/2485867165 conn(0x7f2d100c7800 msgr2=0x7f2d100c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 -- 172.21.15.78:0/2485867165 shutdown_connections 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.861+0000 7f2d16c5c640 1 -- 172.21.15.78:0/2485867165 wait complete. 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.862+0000 7f2d16c5c640 1 Processor -- start 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 -- start start 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 0x7f2d101ee110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d100acc50 0x7f2d10134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10155070 0x7f2d10134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100c2030 con 0x7f2d100718f0 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10134190 con 0x7f2d10155070 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d16c5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d101ee9c0 con 0x7f2d100acc50 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d149f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 0x7f2d101ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d149f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 0x7f2d101ee110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45078/0 (socket says 172.21.15.78:45078) 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.863+0000 7f2d149f8640 1 -- 172.21.15.78:0/1835328806 learned_addr learned my addr 172.21.15.78:0/1835328806 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d0ffff640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d100acc50 0x7f2d10134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d151f9640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10155070 0x7f2d10134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d149f8640 1 -- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d100acc50 msgr2=0x7f2d10134a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d149f8640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d100acc50 0x7f2d10134a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d149f8640 1 -- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10155070 msgr2=0x7f2d10134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d149f8640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10155070 0x7f2d10134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d149f8640 1 -- 172.21.15.78:0/1835328806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc009400 con 0x7f2d100718f0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d149f8640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 0x7f2d101ee110 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400c450 tx=0x7f2d0400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d040052d0 con 0x7f2d100718f0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d04005430 con 0x7f2d100718f0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d101eeca0 con 0x7f2d100718f0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d04012870 con 0x7f2d100718f0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.864+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d10137000 con 0x7f2d100718f0 2024-02-19T14:55:26.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.865+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1006c820 con 0x7f2d100718f0 2024-02-19T14:55:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.865+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2d040129d0 con 0x7f2d100718f0 2024-02-19T14:55:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.866+0000 7f2d0d7fa640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2cf406b810 0x7f2cf406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.866+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2d04089d10 con 0x7f2d100718f0 2024-02-19T14:55:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.866+0000 7f2d0ffff640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2cf406b810 0x7f2cf406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:26.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.866+0000 7f2d0ffff640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2cf406b810 0x7f2cf406dcd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc036000 tx=0x7f2cfc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:26.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:26.870+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d04054690 con 0x7f2d100718f0 2024-02-19T14:55:27.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.078+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d10074990 con 0x7f2d100718f0 2024-02-19T14:55:27.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.078+0000 7f2d0d7fa640 1 -- 172.21.15.78:0/1835328806 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f2d04057d50 con 0x7f2d100718f0 2024-02-19T14:55:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:27.081 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7128,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1361734331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1361734331},{"type":"v1","addr":"172.21.15.78:6836","nonce":1361734331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:11.163509+0000","modified":"2024-02-19T14:55:20.871770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5672":{"gid":5672,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6834/408965047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":408965047},{"type":"v1","addr":"172.21.15.150:6834","nonce":408965047}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-19T14:55:27.081 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 78 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2cf406b810 msgr2=0x7f2cf406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2cf406b810 0x7f2cf406dcd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc036000 tx=0x7f2cfc03c040 comp rx=0 tx=0).stop 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 msgr2=0x7f2d101ee110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 0x7f2d101ee110 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400c450 tx=0x7f2d0400c910 comp rx=0 tx=0).stop 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 shutdown_connections 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2cf406b810 0x7f2cf406dcd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10155070 0x7f2d10134f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d100acc50 0x7f2d10134a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 --2- 172.21.15.78:0/1835328806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d100718f0 0x7f2d101ee110 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 >> 172.21.15.78:0/1835328806 conn(0x7f2d100c7800 msgr2=0x7f2d10153150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:27.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 shutdown_connections 2024-02-19T14:55:27.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.080+0000 7f2d16c5c640 1 -- 172.21.15.78:0/1835328806 wait complete. 2024-02-19T14:55:27.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 Processor -- start 2024-02-19T14:55:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 -- start start 2024-02-19T14:55:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf00c67e0 0x7feaf006c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf00c7df0 con 0x7feaf0155e00 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf00c4310 con 0x7feaf00c67e0 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.191+0000 7feaf835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf006ce10 con 0x7feaf006d460 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf68f9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf006c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf58f7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf00c67e0 0x7feaf006c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf60f8640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7feaf0155e00 0x7feaf00c34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57906/0 (socket says 172.21.15.78:57906) 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf60f8640 1 -- 172.21.15.78:0/2650562889 learned_addr learned my addr 172.21.15.78:0/2650562889 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf68f9640 1 -- 172.21.15.78:0/2650562889 >> v2:172.21.15.150:3300/0 conn(0x7feaf00c67e0 msgr2=0x7feaf006c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf68f9640 1 --2- 172.21.15.78:0/2650562889 >> v2:172.21.15.150:3300/0 conn(0x7feaf00c67e0 0x7feaf006c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf68f9640 1 -- 172.21.15.78:0/2650562889 >> v1:172.21.15.150:6789/0 conn(0x7feaf0155e00 legacy=0x7feaf00c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf68f9640 1 -- 172.21.15.78:0/2650562889 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf00c77c0 con 0x7feaf006d460 2024-02-19T14:55:27.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.192+0000 7feaf58f7640 1 --2- 172.21.15.78:0/2650562889 >> v2:172.21.15.150:3300/0 conn(0x7feaf00c67e0 0x7feaf006c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.193+0000 7feaf68f9640 1 --2- 172.21.15.78:0/2650562889 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf006c8d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feae8009d90 tx=0x7feae802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=945c8c993eb59499 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.193+0000 7feaf50f6640 1 -- 172.21.15.78:0/2650562889 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae802cb30 con 0x7feaf006d460 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.193+0000 7feaf50f6640 1 -- 172.21.15.78:0/2650562889 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae802cc90 con 0x7feaf006d460 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.193+0000 7feaf50f6640 1 -- 172.21.15.78:0/2650562889 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8034440 con 0x7feaf006d460 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.193+0000 7feaf835c640 1 -- 172.21.15.78:0/2650562889 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 msgr2=0x7feaf006c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.193+0000 7feaf835c640 1 --2- 172.21.15.78:0/2650562889 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf006c8d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feae8009d90 tx=0x7feae802bd80 comp rx=0 tx=0).stop 2024-02-19T14:55:27.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.194+0000 7feaf835c640 1 -- 172.21.15.78:0/2650562889 shutdown_connections 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.194+0000 7feaf835c640 1 --2- 172.21.15.78:0/2650562889 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf006c8d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.194+0000 7feaf835c640 1 --2- 172.21.15.78:0/2650562889 >> v2:172.21.15.150:3300/0 conn(0x7feaf00c67e0 0x7feaf006c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.194+0000 7feaf835c640 1 -- 172.21.15.78:0/2650562889 >> 172.21.15.78:0/2650562889 conn(0x7feaf006b700 msgr2=0x7feaf00aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.194+0000 7feaf835c640 1 -- 172.21.15.78:0/2650562889 shutdown_connections 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.194+0000 7feaf835c640 1 -- 172.21.15.78:0/2650562889 wait complete. 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 Processor -- start 2024-02-19T14:55:27.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 -- start start 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf01582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 0x7feaf01572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0155e00 0x7feaf0157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01e03d0 con 0x7feaf00c67e0 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0156e60 con 0x7feaf0155e00 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01e13a0 con 0x7feaf006d460 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf58f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 0x7feaf01572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf58f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 0x7feaf01572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45092/0 (socket says 172.21.15.78:45092) 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf58f7640 1 -- 172.21.15.78:0/2641045081 learned_addr learned my addr 172.21.15.78:0/2641045081 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf60f8640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf01582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.195+0000 7feaf68f9640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0155e00 0x7feaf0157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf58f7640 1 -- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 msgr2=0x7feaf01582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf58f7640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf01582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf58f7640 1 -- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0155e00 msgr2=0x7feaf0157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf58f7640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0155e00 0x7feaf0157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf58f7640 1 -- 172.21.15.78:0/2641045081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0009400 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf68f9640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0155e00 0x7feaf0157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf58f7640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 0x7feaf01572f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7feae000e990 tx=0x7feae000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae000c9e0 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae000cb40 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae8009a70 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae000ced0 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.196+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf0156530 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.197+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf0156ff0 con 0x7feaf00c67e0 2024-02-19T14:55:27.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.197+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feae0013470 con 0x7feaf00c67e0 2024-02-19T14:55:27.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.198+0000 7feae6ffd640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.198+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7feae008a4a0 con 0x7feaf00c67e0 2024-02-19T14:55:27.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.198+0000 7feaf60f8640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.198+0000 7feaf60f8640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7feaf006ddf0 tx=0x7feaec007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.202+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feae0054e20 con 0x7feaf00c67e0 2024-02-19T14:55:27.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.373+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaf0074990 con 0x7feaf00c67e0 2024-02-19T14:55:27.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.374+0000 7feae6ffd640 1 -- 172.21.15.78:0/2641045081 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16990 (secure 0 0 0) 0x7feae00584e0 con 0x7feaf00c67e0 2024-02-19T14:55:27.377 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:27.377 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":99,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:18.858198+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:09.377887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:10.136982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3610724432":"2024-02-20T14:55:18.858155+0000","172.21.15.78:6834/3610724432":"2024-02-20T14:55:18.858155+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 msgr2=0x7fead406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7feaf006ddf0 tx=0x7feaec007e80 comp rx=0 tx=0).stop 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 msgr2=0x7feaf01572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 0x7feaf01572f0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7feae000e990 tx=0x7feae000ee50 comp rx=0 tx=0).stop 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 shutdown_connections 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaf00c67e0 0x7feaf01572f0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0155e00 0x7feaf0157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 --2- 172.21.15.78:0/2641045081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf006d460 0x7feaf01582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 >> 172.21.15.78:0/2641045081 conn(0x7feaf006b700 msgr2=0x7feaf006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:27.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 shutdown_connections 2024-02-19T14:55:27.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.376+0000 7feaf835c640 1 -- 172.21.15.78:0/2641045081 wait complete. 2024-02-19T14:55:27.390 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:55:27.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.488+0000 7f26ecc15640 1 Processor -- start 2024-02-19T14:55:27.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.488+0000 7f26ecc15640 1 -- start start 2024-02-19T14:55:27.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26ecc15640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26e806ed20 0x7f26e806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26ecc15640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e8150820 con 0x7f26e8150440 2024-02-19T14:55:27.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26ecc15640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e81537d0 con 0x7f26e806d750 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26ecc15640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80c1b60 con 0x7f26e806ed20 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26dffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26e806ed20 0x7f26e806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26e659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f26e8150440 0x7f26e80c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34266/0 (socket says 172.21.15.78:34266) 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26e659c640 1 -- 172.21.15.78:0/995007956 learned_addr learned my addr 172.21.15.78:0/995007956 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26dffff640 1 -- 172.21.15.78:0/995007956 >> v1:172.21.15.150:6789/0 conn(0x7f26e806d750 legacy=0x7f26e815d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26dffff640 1 -- 172.21.15.78:0/995007956 >> v1:172.21.15.78:6789/0 conn(0x7f26e8150440 legacy=0x7f26e80c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26e5d9b640 1 -- 172.21.15.78:0/995007956 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578745125 0 0) 0x7f26e8150820 con 0x7f26e8150440 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.489+0000 7f26dffff640 1 -- 172.21.15.78:0/995007956 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8153d10 con 0x7f26e806ed20 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26dffff640 1 --2- 172.21.15.78:0/995007956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e806ed20 0x7f26e806d210 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f26e8153960 tx=0x7f26d002b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7259fa7c19319962 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26e5d9b640 1 -- 172.21.15.78:0/995007956 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0030890 con 0x7f26e806ed20 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26e5d9b640 1 -- 172.21.15.78:0/995007956 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d00309f0 con 0x7f26e806ed20 2024-02-19T14:55:27.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26e5d9b640 1 -- 172.21.15.78:0/995007956 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0030d80 con 0x7f26e806ed20 2024-02-19T14:55:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26ecc15640 1 -- 172.21.15.78:0/995007956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e806ed20 msgr2=0x7f26e806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26ecc15640 1 --2- 172.21.15.78:0/995007956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e806ed20 0x7f26e806d210 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f26e8153960 tx=0x7f26d002b9d0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26ecc15640 1 -- 172.21.15.78:0/995007956 shutdown_connections 2024-02-19T14:55:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26ecc15640 1 --2- 172.21.15.78:0/995007956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e806ed20 0x7f26e806d210 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.490+0000 7f26ecc15640 1 -- 172.21.15.78:0/995007956 >> 172.21.15.78:0/995007956 conn(0x7f26e806b270 msgr2=0x7f26e806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.491+0000 7f26ecc15640 1 -- 172.21.15.78:0/995007956 shutdown_connections 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.491+0000 7f26ecc15640 1 -- 172.21.15.78:0/995007956 wait complete. 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 Processor -- start 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 -- start start 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e806d750 0x7f26e8142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e806ed20 0x7f26e81ea860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 0x7f26e80c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e81ee140 con 0x7f26e8150440 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e81525d0 con 0x7f26e806ed20 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26ecc15640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e81528b0 con 0x7f26e806d750 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.492+0000 7f26e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 0x7f26e80c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 0x7f26e80c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51264/0 (socket says 172.21.15.78:51264) 2024-02-19T14:55:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 -- 172.21.15.78:0/3535551606 learned_addr learned my addr 172.21.15.78:0/3535551606 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e659c640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e806d750 0x7f26e8142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26dffff640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e806ed20 0x7f26e81ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 -- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e806d750 msgr2=0x7f26e8142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e806d750 0x7f26e8142f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 -- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e806ed20 msgr2=0x7f26e81ea860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e806ed20 0x7f26e81ea860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 -- 172.21.15.78:0/3535551606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d0009400 con 0x7f26e8150440 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26e6d9d640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 0x7f26e80c58c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c7a0 tx=0x7f26e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0003b70 con 0x7f26e8150440 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e0003cf0 con 0x7f26e8150440 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0003ea0 con 0x7f26e8150440 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e8152b90 con 0x7f26e8150440 2024-02-19T14:55:27.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.493+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e80c5e00 con 0x7f26e8150440 2024-02-19T14:55:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.494+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e814a4b0 con 0x7f26e8150440 2024-02-19T14:55:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.494+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f26e000aa90 con 0x7f26e8150440 2024-02-19T14:55:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.495+0000 7f26deffd640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c806b810 0x7f26c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.495+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f26e008a0a0 con 0x7f26e8150440 2024-02-19T14:55:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.495+0000 7f26e659c640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c806b810 0x7f26c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.495+0000 7f26e659c640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c806b810 0x7f26c806dcd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f60 tx=0x7f26d8007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.499+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f26e0054990 con 0x7f26e8150440 2024-02-19T14:55:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.705+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f26e8074990 con 0x7f26e8150440 2024-02-19T14:55:27.708 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:55:27.705+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:55:27.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.707+0000 7f26deffd640 1 -- 172.21.15.78:0/3535551606 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7f26e0058050 con 0x7f26e8150440 2024-02-19T14:55:27.709 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c806b810 msgr2=0x7f26c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c806b810 0x7f26c806dcd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f60 tx=0x7f26d8007e40 comp rx=0 tx=0).stop 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 msgr2=0x7f26e80c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 0x7f26e80c58c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c7a0 tx=0x7f26e000cc60 comp rx=0 tx=0).stop 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 shutdown_connections 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c806b810 0x7f26c806dcd0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e8150440 0x7f26e80c58c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e806ed20 0x7f26e81ea860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 --2- 172.21.15.78:0/3535551606 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e806d750 0x7f26e8142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 >> 172.21.15.78:0/3535551606 conn(0x7f26e806b270 msgr2=0x7f26e81487c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 shutdown_connections 2024-02-19T14:55:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.709+0000 7f26ecc15640 1 -- 172.21.15.78:0/3535551606 wait complete. 2024-02-19T14:55:27.713 INFO:tasks.ceph.mds.d.smithi150.stderr: -202> 2024-02-19T14:55:22.091+0000 7fb60f4d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:27.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:55:27.736 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:55:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.822+0000 7feba8cc7640 1 Processor -- start 2024-02-19T14:55:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.822+0000 7feba8cc7640 1 -- start start 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba8cc7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feba40c25e0 0x7feba41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba8cc7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feba40c29c0 0x7feba40c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba8cc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba40c49c0 0x7feba406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba8cc7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba4152c90 con 0x7feba40c25e0 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba8cc7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba4072250 con 0x7feba40c29c0 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba8cc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba4072480 con 0x7feba40c49c0 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feba40c25e0 0x7feba41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feba40c25e0 0x7feba41526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51272/0 (socket says 172.21.15.78:51272) 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba259c640 1 -- 172.21.15.78:0/3238509632 learned_addr learned my addr 172.21.15.78:0/3238509632 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba1d9b640 1 --2- 172.21.15.78:0/3238509632 >> v2:172.21.15.150:3300/0 conn(0x7feba40c29c0 0x7feba40c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.823+0000 7feba2d9d640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba40c49c0 0x7feba406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba259c640 1 -- 172.21.15.78:0/3238509632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba40c49c0 msgr2=0x7feba406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba259c640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba40c49c0 0x7feba406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba259c640 1 -- 172.21.15.78:0/3238509632 >> v2:172.21.15.150:3300/0 conn(0x7feba40c29c0 msgr2=0x7feba40c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba259c640 1 --2- 172.21.15.78:0/3238509632 >> v2:172.21.15.150:3300/0 conn(0x7feba40c29c0 0x7feba40c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba259c640 1 -- 172.21.15.78:0/3238509632 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40c3eb0 con 0x7feba40c25e0 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba2d9d640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba40c49c0 0x7feba406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba259c640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba41526a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7feba40c59a0 tx=0x7feb9402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a070892dfab463 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba159a640 1 -- 172.21.15.78:0/3238509632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9402c9e0 con 0x7feba40c25e0 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba159a640 1 -- 172.21.15.78:0/3238509632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9402cb40 con 0x7feba40c25e0 2024-02-19T14:55:27.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.824+0000 7feba159a640 1 -- 172.21.15.78:0/3238509632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940343f0 con 0x7feba40c25e0 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 -- 172.21.15.78:0/3238509632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 msgr2=0x7feba41526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba41526a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7feba40c59a0 tx=0x7feb9402be40 comp rx=0 tx=0).stop 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 -- 172.21.15.78:0/3238509632 shutdown_connections 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba40c49c0 0x7feba406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 --2- 172.21.15.78:0/3238509632 >> v2:172.21.15.150:3300/0 conn(0x7feba40c29c0 0x7feba40c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 --2- 172.21.15.78:0/3238509632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba41526a0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 -- 172.21.15.78:0/3238509632 >> 172.21.15.78:0/3238509632 conn(0x7feba406b270 msgr2=0x7feba406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 -- 172.21.15.78:0/3238509632 shutdown_connections 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.825+0000 7feba8cc7640 1 -- 172.21.15.78:0/3238509632 wait complete. 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.826+0000 7feba8cc7640 1 Processor -- start 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.826+0000 7feba8cc7640 1 -- start start 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba8cc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba40bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba8cc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40c29c0 0x7feba41ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba8cc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba41ea7e0 0x7feba41ee850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba8cc7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba40a6420 con 0x7feba40c25e0 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba8cc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4153630 con 0x7feba40c29c0 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba8cc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba4153910 con 0x7feba41ea7e0 2024-02-19T14:55:27.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba40bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba40bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51274/0 (socket says 172.21.15.78:51274) 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 -- 172.21.15.78:0/4024890977 learned_addr learned my addr 172.21.15.78:0/4024890977 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba1d9b640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40c29c0 0x7feba41ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba2d9d640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba41ea7e0 0x7feba41ee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 -- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba41ea7e0 msgr2=0x7feba41ee850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba41ea7e0 0x7feba41ee850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 -- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40c29c0 msgr2=0x7feba41ee310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40c29c0 0x7feba41ee310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 -- 172.21.15.78:0/4024890977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94009b30 con 0x7feba40c25e0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.827+0000 7feba259c640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba40bfc20 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7feb94000f80 tx=0x7feb94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.828+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94009510 con 0x7feba40c25e0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.828+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9403e450 con 0x7feba40c25e0 2024-02-19T14:55:27.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.828+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9403e800 con 0x7feba40c25e0 2024-02-19T14:55:27.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.828+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4153bf0 con 0x7feba40c25e0 2024-02-19T14:55:27.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.828+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba40c0160 con 0x7feba40c25e0 2024-02-19T14:55:27.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.829+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba41ee030 con 0x7feba40c25e0 2024-02-19T14:55:27.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.829+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feb9403e960 con 0x7feba40c25e0 2024-02-19T14:55:27.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.829+0000 7feb92ffd640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb8406b840 0x7feb8406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:27.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.830+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6273+0+0 (secure 0 0 0) 0x7feb940acf70 con 0x7feba40c25e0 2024-02-19T14:55:27.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.830+0000 7feba1d9b640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb8406b840 0x7feb8406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:27.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.830+0000 7feba1d9b640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb8406b840 0x7feb8406dd00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7feba4156200 tx=0x7feb98009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:27.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:27.834+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb940e0dc0 con 0x7feba40c25e0 2024-02-19T14:55:28.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.042+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feba4074990 con 0x7feba40c25e0 2024-02-19T14:55:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.708+0000 7feb92ffd640 1 -- 172.21.15.78:0/4024890977 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7feba41ee030 con 0x7feba40c25e0 2024-02-19T14:55:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb8406b840 msgr2=0x7feb8406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb8406b840 0x7feb8406dd00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7feba4156200 tx=0x7feb98009190 comp rx=0 tx=0).stop 2024-02-19T14:55:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 msgr2=0x7feba40bfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba40bfc20 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7feb94000f80 tx=0x7feb94004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 shutdown_connections 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb8406b840 0x7feb8406dd00 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feba41ea7e0 0x7feba41ee850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40c29c0 0x7feba41ee310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 --2- 172.21.15.78:0/4024890977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba40c25e0 0x7feba40bfc20 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 >> 172.21.15.78:0/4024890977 conn(0x7feba406b270 msgr2=0x7feba4150b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 shutdown_connections 2024-02-19T14:55:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.710+0000 7feba8cc7640 1 -- 172.21.15.78:0/4024890977 wait complete. 2024-02-19T14:55:28.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:55:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.828+0000 7fe6a8d8c640 1 Processor -- start 2024-02-19T14:55:28.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.828+0000 7fe6a8d8c640 1 -- start start 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a8d8c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe6a4070f10 0x7fe6a4159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a8d8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe6a40c7f10 0x7fe6a40c5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a8d8c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a40c37c0 con 0x7fe6a40c4e20 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a8d8c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a406f4d0 con 0x7fe6a4070f10 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a8d8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a4155a10 con 0x7fe6a40c7f10 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe6a4070f10 0x7fe6a4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe6a4070f10 0x7fe6a4159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51286/0 (socket says 172.21.15.78:51286) 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 -- 172.21.15.78:0/3244754086 learned_addr learned my addr 172.21.15.78:0/3244754086 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a2d9d640 1 --2- 172.21.15.78:0/3244754086 >> v2:172.21.15.150:3300/0 conn(0x7fe6a40c7f10 0x7fe6a40c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 -- 172.21.15.78:0/3244754086 >> v2:172.21.15.150:3300/0 conn(0x7fe6a40c7f10 msgr2=0x7fe6a40c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 --2- 172.21.15.78:0/3244754086 >> v2:172.21.15.150:3300/0 conn(0x7fe6a40c7f10 0x7fe6a40c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 -- 172.21.15.78:0/3244754086 >> v1:172.21.15.78:6789/0 conn(0x7fe6a40c4e20 legacy=0x7fe6a41588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe69bfff640 1 -- 172.21.15.78:0/3244754086 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a4155cf0 con 0x7fe6a4070f10 2024-02-19T14:55:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.829+0000 7fe6a1d9b640 1 -- 172.21.15.78:0/3244754086 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441328628 0 0) 0x7fe6a40c37c0 con 0x7fe6a40c4e20 2024-02-19T14:55:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.830+0000 7fe69bfff640 1 --2- 172.21.15.78:0/3244754086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4159000 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009870 tx=0x7fe68c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3477b214eabaf230 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.830+0000 7fe6a1d9b640 1 -- 172.21.15.78:0/3244754086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c031870 con 0x7fe6a4070f10 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.830+0000 7fe6a1d9b640 1 -- 172.21.15.78:0/3244754086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c0319d0 con 0x7fe6a4070f10 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.830+0000 7fe6a1d9b640 1 -- 172.21.15.78:0/3244754086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68c031d60 con 0x7fe6a4070f10 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.830+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/3244754086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 msgr2=0x7fe6a4159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.830+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/3244754086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4159000 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009870 tx=0x7fe68c02b910 comp rx=0 tx=0).stop 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.831+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/3244754086 shutdown_connections 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.831+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/3244754086 >> v2:172.21.15.150:3300/0 conn(0x7fe6a40c7f10 0x7fe6a40c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.831+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/3244754086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4159000 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.831+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/3244754086 >> 172.21.15.78:0/3244754086 conn(0x7fe6a40aa370 msgr2=0x7fe6a40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.831+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/3244754086 shutdown_connections 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.831+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/3244754086 wait complete. 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 Processor -- start 2024-02-19T14:55:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 -- start start 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6a40c4e20 0x7fe6a40bda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6a40c7f10 0x7fe6a41571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a4154d10 con 0x7fe6a4070f10 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a406ce20 con 0x7fe6a40c7f10 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.832+0000 7fe6a8d8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a41eb8c0 con 0x7fe6a40c4e20 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4144480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51298/0 (socket says 172.21.15.78:51298) 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 -- 172.21.15.78:0/1231064909 learned_addr learned my addr 172.21.15.78:0/1231064909 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe69bfff640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6a40c4e20 0x7fe6a40bda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a2d9d640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6a40c7f10 0x7fe6a41571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 -- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6a40c4e20 msgr2=0x7fe6a40bda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6a40c4e20 0x7fe6a40bda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 -- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6a40c7f10 msgr2=0x7fe6a41571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6a40c7f10 0x7fe6a41571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 -- 172.21.15.78:0/1231064909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c0094d0 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe69bfff640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6a40c4e20 0x7fe6a40bda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a259c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4144480 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe69400c4b0 tx=0x7fe69400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694005480 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6940035f0 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6940039e0 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a406d070 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.833+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a406d630 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.834+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a406e120 con 0x7fe6a4070f10 2024-02-19T14:55:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.835+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe694003b40 con 0x7fe6a4070f10 2024-02-19T14:55:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.835+0000 7fe69affd640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe68406b810 0x7fe68406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.835+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe6940898f0 con 0x7fe6a4070f10 2024-02-19T14:55:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.835+0000 7fe69bfff640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe68406b810 0x7fe68406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.836+0000 7fe69bfff640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe68406b810 0x7fe68406dcd0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe68c036000 tx=0x7fe68c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:28.839+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe694054180 con 0x7fe6a4070f10 2024-02-19T14:55:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.011+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe6a4074990 con 0x7fe6a4070f10 2024-02-19T14:55:29.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.894+0000 7fe69affd640 1 -- 172.21.15.78:0/1231064909 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe6940170d0 con 0x7fe6a4070f10 2024-02-19T14:55:29.897 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe68406b810 msgr2=0x7fe68406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe68406b810 0x7fe68406dcd0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe68c036000 tx=0x7fe68c03c040 comp rx=0 tx=0).stop 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 msgr2=0x7fe6a4144480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4144480 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fe69400c4b0 tx=0x7fe69400c970 comp rx=0 tx=0).stop 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 shutdown_connections 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe68406b810 0x7fe68406dcd0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6a40c7f10 0x7fe6a41571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6a40c4e20 0x7fe6a40bda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:29.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 --2- 172.21.15.78:0/1231064909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4070f10 0x7fe6a4144480 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:29.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 >> 172.21.15.78:0/1231064909 conn(0x7fe6a40aa370 msgr2=0x7fe6a40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:29.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 shutdown_connections 2024-02-19T14:55:29.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:29.897+0000 7fe6a8d8c640 1 -- 172.21.15.78:0/1231064909 wait complete. 2024-02-19T14:55:29.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:55:30.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.013+0000 7f698a313640 1 Processor -- start 2024-02-19T14:55:30.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.013+0000 7f698a313640 1 -- start start 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f698a313640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6984152670 0x7f698406d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f698a313640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f698406dc70 con 0x7f6984154ea0 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f698a313640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6984158ef0 con 0x7f69840c8730 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f698a313640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6984155a40 con 0x7f6984152670 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f6983fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6984152670 0x7f698406d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f69888b0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6984154ea0 0x7f6984155330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34282/0 (socket says 172.21.15.78:34282) 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f69888b0640 1 -- 172.21.15.78:0/1769193378 learned_addr learned my addr 172.21.15.78:0/1769193378 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:30.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f6983fff640 1 -- 172.21.15.78:0/1769193378 >> v1:172.21.15.150:6789/0 conn(0x7f69840c8730 legacy=0x7f69840c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:30.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f6983fff640 1 -- 172.21.15.78:0/1769193378 >> v1:172.21.15.78:6789/0 conn(0x7f6984154ea0 legacy=0x7f6984155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:30.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f6983fff640 1 -- 172.21.15.78:0/1769193378 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69840be190 con 0x7f6984152670 2024-02-19T14:55:30.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.014+0000 7f6982ffd640 1 -- 172.21.15.78:0/1769193378 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994770405 0 0) 0x7f698406dc70 con 0x7f6984154ea0 2024-02-19T14:55:30.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.015+0000 7f6983fff640 1 --2- 172.21.15.78:0/1769193378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f698406d730 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f69740097a0 tx=0x7f697402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c58a1e14c8f95c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:30.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.015+0000 7f6982ffd640 1 -- 172.21.15.78:0/1769193378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697402f050 con 0x7f6984152670 2024-02-19T14:55:30.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.015+0000 7f6982ffd640 1 -- 172.21.15.78:0/1769193378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6974031700 con 0x7f6984152670 2024-02-19T14:55:30.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.015+0000 7f6982ffd640 1 -- 172.21.15.78:0/1769193378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974031a90 con 0x7f6984152670 2024-02-19T14:55:30.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 -- 172.21.15.78:0/1769193378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 msgr2=0x7f698406d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:30.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 --2- 172.21.15.78:0/1769193378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f698406d730 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f69740097a0 tx=0x7f697402b7a0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 -- 172.21.15.78:0/1769193378 shutdown_connections 2024-02-19T14:55:30.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 --2- 172.21.15.78:0/1769193378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f698406d730 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 -- 172.21.15.78:0/1769193378 >> 172.21.15.78:0/1769193378 conn(0x7f6984072790 msgr2=0x7f6984072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:30.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 -- 172.21.15.78:0/1769193378 shutdown_connections 2024-02-19T14:55:30.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.016+0000 7f698a313640 1 -- 172.21.15.78:0/1769193378 wait complete. 2024-02-19T14:55:30.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.017+0000 7f698a313640 1 Processor -- start 2024-02-19T14:55:30.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.017+0000 7f698a313640 1 -- start start 2024-02-19T14:55:30.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69840c8730 0x7f69841eb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:30.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f69840c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6984154ea0 0x7f6984145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841eb8f0 con 0x7f6984152670 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69840bf860 con 0x7f69840c8730 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69840be3c0 con 0x7f6984154ea0 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f69840c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f69840c0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51316/0 (socket says 172.21.15.78:51316) 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 -- 172.21.15.78:0/3979192694 learned_addr learned my addr 172.21.15.78:0/3979192694 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 -- 172.21.15.78:0/3979192694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6984154ea0 msgr2=0x7f6984145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6984154ea0 0x7f6984145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 -- 172.21.15.78:0/3979192694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69840c8730 msgr2=0x7f69841eb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69840c8730 0x7f69841eb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 -- 172.21.15.78:0/3979192694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974009400 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f69837fe640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f69840c0d90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f697800b410 tx=0x7f697800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978011830 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6978011990 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69840be6a0 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978011d40 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.018+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69840bec60 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.019+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6984159360 con 0x7f6984152670 2024-02-19T14:55:30.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.020+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f697801a400 con 0x7f6984152670 2024-02-19T14:55:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.020+0000 7f6980ff9640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f697006b810 0x7f697006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.020+0000 7f6983fff640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f697006b810 0x7f697006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.020+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f697808a2c0 con 0x7f6984152670 2024-02-19T14:55:30.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.021+0000 7f6983fff640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f697006b810 0x7f697006dcd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6974035000 tx=0x7f6974030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:30.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.025+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6984159360 con 0x7f6984152670 2024-02-19T14:55:30.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.196+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6984159360 con 0x7f6984152670 2024-02-19T14:55:30.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.901+0000 7f6980ff9640 1 -- 172.21.15.78:0/3979192694 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6984159360 con 0x7f6984152670 2024-02-19T14:55:30.904 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f697006b810 msgr2=0x7f697006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f697006b810 0x7f697006dcd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6974035000 tx=0x7f6974030000 comp rx=0 tx=0).stop 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 msgr2=0x7f69840c0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f69840c0d90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f697800b410 tx=0x7f697800b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 shutdown_connections 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f697006b810 0x7f697006dcd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6984152670 0x7f69840c0d90 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6984154ea0 0x7f6984145960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 --2- 172.21.15.78:0/3979192694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69840c8730 0x7f69841eb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 >> 172.21.15.78:0/3979192694 conn(0x7f6984072790 msgr2=0x7f69840ab200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:30.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 shutdown_connections 2024-02-19T14:55:30.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:30.903+0000 7f698a313640 1 -- 172.21.15.78:0/3979192694 wait complete. 2024-02-19T14:55:30.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:55:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.024+0000 7f390f02d640 1 Processor -- start 2024-02-19T14:55:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.024+0000 7f390f02d640 1 -- start start 2024-02-19T14:55:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390f02d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3908154e00 0x7f39080c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390f02d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39080bfc10 0x7f39080c16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390f02d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3908155300 con 0x7f390813e0f0 2024-02-19T14:55:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390f02d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3908071710 con 0x7f39080bfc10 2024-02-19T14:55:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390f02d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39081555e0 con 0x7f3908154e00 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390d82a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3908154e00 0x7f39080c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390e82c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39080bfc10 0x7f39080c16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390e82c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39080bfc10 0x7f39080c16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:35030/0 (socket says 172.21.15.78:35030) 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390e82c640 1 -- 172.21.15.78:0/2178038833 learned_addr learned my addr 172.21.15.78:0/2178038833 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390e02b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f390813e0f0 0x7f39080c2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:49520/0 (socket says 172.21.15.78:49520) 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.025+0000 7f390d82a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3908154e00 0x7f39080c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:55062/0 (socket says 172.21.15.78:55062) 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.026+0000 7f390e82c640 1 -- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3908154e00 msgr2=0x7f39080c11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.026+0000 7f390e82c640 1 --2- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3908154e00 0x7f39080c11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.026+0000 7f390e82c640 1 -- 172.21.15.78:0/2178038833 >> v1:172.21.15.150:6789/0 conn(0x7f390813e0f0 legacy=0x7f39080c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.026+0000 7f390e82c640 1 -- 172.21.15.78:0/2178038833 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39081558c0 con 0x7f39080bfc10 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.027+0000 7f390e82c640 1 --2- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39080bfc10 0x7f39080c16f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3900009e50 tx=0x7f390002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3acb87412541725 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.027+0000 7f390d029640 1 -- 172.21.15.78:0/2178038833 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390002c9f0 con 0x7f39080bfc10 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.027+0000 7f390d029640 1 -- 172.21.15.78:0/2178038833 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390002cb50 con 0x7f39080bfc10 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.027+0000 7f390d029640 1 -- 172.21.15.78:0/2178038833 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000343f0 con 0x7f39080bfc10 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 -- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39080bfc10 msgr2=0x7f39080c16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 --2- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39080bfc10 0x7f39080c16f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3900009e50 tx=0x7f390002be50 comp rx=0 tx=0).stop 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 -- 172.21.15.78:0/2178038833 shutdown_connections 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 --2- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3908154e00 0x7f39080c11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 --2- 172.21.15.78:0/2178038833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39080bfc10 0x7f39080c16f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 -- 172.21.15.78:0/2178038833 >> 172.21.15.78:0/2178038833 conn(0x7f39080c6410 msgr2=0x7f39080c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 -- 172.21.15.78:0/2178038833 shutdown_connections 2024-02-19T14:55:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.028+0000 7f390f02d640 1 -- 172.21.15.78:0/2178038833 wait complete. 2024-02-19T14:55:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.029+0000 7f390f02d640 1 Processor -- start 2024-02-19T14:55:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.029+0000 7f390f02d640 1 -- start start 2024-02-19T14:55:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390f02d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39080bfc10 0x7f39081e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390f02d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f390813e0f0 0x7f39080c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390f02d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 0x7f39080c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390f02d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080aa210 con 0x7f3908154e00 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390f02d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080ab1e0 con 0x7f390813e0f0 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390f02d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39081e8060 con 0x7f39080bfc10 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 0x7f39080c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 0x7f39080c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51326/0 (socket says 172.21.15.78:51326) 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 -- 172.21.15.78:0/4226893170 learned_addr learned my addr 172.21.15.78:0/4226893170 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e02b640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39080bfc10 0x7f39081e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390d82a640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f390813e0f0 0x7f39080c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 -- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39080bfc10 msgr2=0x7f39081e7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39080bfc10 0x7f39081e7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 -- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f390813e0f0 msgr2=0x7f39080c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f390813e0f0 0x7f39080c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 -- 172.21.15.78:0/4226893170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3900009b30 con 0x7f3908154e00 2024-02-19T14:55:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f390e82c640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 0x7f39080c42d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f3900009e50 tx=0x7f3900004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390003e450 con 0x7f3908154e00 2024-02-19T14:55:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.030+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390003e5b0 con 0x7f3908154e00 2024-02-19T14:55:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.031+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39081eb270 con 0x7f3908154e00 2024-02-19T14:55:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.031+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390003e960 con 0x7f3908154e00 2024-02-19T14:55:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.031+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39081edce0 con 0x7f3908154e00 2024-02-19T14:55:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.032+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3908074990 con 0x7f3908154e00 2024-02-19T14:55:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.032+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f390003eac0 con 0x7f3908154e00 2024-02-19T14:55:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.032+0000 7f38feffd640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38ec06b840 0x7f38ec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.033+0000 7f390e02b640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38ec06b840 0x7f38ec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.033+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f39000aceb0 con 0x7f3908154e00 2024-02-19T14:55:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.033+0000 7f390e02b640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38ec06b840 0x7f38ec06dd00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f39080c5650 tx=0x7f3904007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.037+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3900077a80 con 0x7f3908154e00 2024-02-19T14:55:31.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.208+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f39080c2260 con 0x7f3908154e00 2024-02-19T14:55:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.211+0000 7f38feffd640 1 -- 172.21.15.78:0/4226893170 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v33) v1 ==== 96+0+0 (secure 0 0 0) 0x7f390007b140 con 0x7f3908154e00 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38ec06b840 msgr2=0x7f38ec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38ec06b840 0x7f38ec06dd00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f39080c5650 tx=0x7f3904007ce0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 msgr2=0x7f39080c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 0x7f39080c42d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f3900009e50 tx=0x7f3900004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 shutdown_connections 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38ec06b840 0x7f38ec06dd00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3908154e00 0x7f39080c42d0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f390813e0f0 0x7f39080c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 --2- 172.21.15.78:0/4226893170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39080bfc10 0x7f39081e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 >> 172.21.15.78:0/4226893170 conn(0x7f39080c6410 msgr2=0x7f390813c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:31.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 shutdown_connections 2024-02-19T14:55:31.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.213+0000 7f390f02d640 1 -- 172.21.15.78:0/4226893170 wait complete. 2024-02-19T14:55:31.226 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:55:31.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.325+0000 7f1cd7ea4640 1 Processor -- start 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.325+0000 7f1cd7ea4640 1 -- start start 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd7ea4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 0x7f1cd00ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd7ea4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd7ea4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd0158c40 con 0x7f1cd00bdf20 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd7ea4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd00c8de0 con 0x7f1cd00ac390 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd7ea4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd00c5a30 con 0x7f1cd00c55f0 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd5c40640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 0x7f1cd00ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd6441640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd5c40640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 0x7f1cd00ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:35060/0 (socket says 172.21.15.78:35060) 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd5c40640 1 -- 172.21.15.78:0/2144379666 learned_addr learned my addr 172.21.15.78:0/2144379666 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd6441640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd015f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:55080/0 (socket says 172.21.15.78:55080) 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd6441640 1 -- 172.21.15.78:0/2144379666 >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 msgr2=0x7f1cd00ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd6441640 1 --2- 172.21.15.78:0/2144379666 >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 0x7f1cd00ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd6441640 1 -- 172.21.15.78:0/2144379666 >> v1:172.21.15.150:6789/0 conn(0x7f1cd00bdf20 legacy=0x7f1cd0158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd6441640 1 -- 172.21.15.78:0/2144379666 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd0155a60 con 0x7f1cd00c55f0 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.326+0000 7f1cd5c40640 1 --2- 172.21.15.78:0/2144379666 >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 0x7f1cd00ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.327+0000 7f1cd6441640 1 --2- 172.21.15.78:0/2144379666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd015f8c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009d90 tx=0x7f1cc802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114e4cb291e1d680 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.327+0000 7f1cd4c3e640 1 -- 172.21.15.78:0/2144379666 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc802cb30 con 0x7f1cd00c55f0 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.327+0000 7f1cd4c3e640 1 -- 172.21.15.78:0/2144379666 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc802cc90 con 0x7f1cd00c55f0 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.327+0000 7f1cd4c3e640 1 -- 172.21.15.78:0/2144379666 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8034440 con 0x7f1cd00c55f0 2024-02-19T14:55:31.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/2144379666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 msgr2=0x7f1cd015f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/2144379666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd015f8c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009d90 tx=0x7f1cc802bd80 comp rx=0 tx=0).stop 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/2144379666 shutdown_connections 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/2144379666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd015f8c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/2144379666 >> v2:172.21.15.150:3300/0 conn(0x7f1cd00ac390 0x7f1cd00ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/2144379666 >> 172.21.15.78:0/2144379666 conn(0x7f1cd00ca890 msgr2=0x7f1cd00caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/2144379666 shutdown_connections 2024-02-19T14:55:31.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.328+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/2144379666 wait complete. 2024-02-19T14:55:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.329+0000 7f1cd7ea4640 1 Processor -- start 2024-02-19T14:55:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.329+0000 7f1cd7ea4640 1 -- start start 2024-02-19T14:55:31.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd7ea4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 0x7f1cd00be6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd7ea4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cd00bdf20 0x7f1cd00c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd7ea4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd0151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd7ea4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd00ab960 con 0x7f1cd00ac390 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd7ea4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd00c39a0 con 0x7f1cd00bdf20 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd7ea4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd00c3c80 con 0x7f1cd00c55f0 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 0x7f1cd00be6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 0x7f1cd00be6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51332/0 (socket says 172.21.15.78:51332) 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 -- 172.21.15.78:0/219747518 learned_addr learned my addr 172.21.15.78:0/219747518 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd543f640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cd00bdf20 0x7f1cd00c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd6441640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd0151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 -- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 msgr2=0x7f1cd0151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd0151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 -- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cd00bdf20 msgr2=0x7f1cd00c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cd00bdf20 0x7f1cd00c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 -- 172.21.15.78:0/219747518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc0094d0 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd543f640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cd00bdf20 0x7f1cd00c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cd5c40640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 0x7f1cd00be6b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9d0 tx=0x7f1ccc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.330+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00ca60 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.331+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc00cbc0 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.331+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc013470 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.331+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc8009a70 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.331+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd00c2660 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.331+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd0074990 con 0x7f1cd00ac390 2024-02-19T14:55:31.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.332+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1ccc0135d0 con 0x7f1cd00ac390 2024-02-19T14:55:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.332+0000 7f1cc67fc640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cb406b810 0x7f1cb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.332+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1ccc08a430 con 0x7f1cd00ac390 2024-02-19T14:55:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.332+0000 7f1cd543f640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cb406b810 0x7f1cb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.333+0000 7f1cd543f640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cb406b810 0x7f1cb406dcd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0075330 tx=0x7f1cc0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.337+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ccc054f80 con 0x7f1cd00ac390 2024-02-19T14:55:31.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.508+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1cd006da00 con 0x7f1cd00ac390 2024-02-19T14:55:31.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.514+0000 7f1cc67fc640 1 -- 172.21.15.78:0/219747518 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1ccc058640 con 0x7f1cd00ac390 2024-02-19T14:55:31.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cb406b810 msgr2=0x7f1cb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cb406b810 0x7f1cb406dcd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0075330 tx=0x7f1cc0009040 comp rx=0 tx=0).stop 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 msgr2=0x7f1cd00be6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 0x7f1cd00be6b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9d0 tx=0x7f1ccc00ee90 comp rx=0 tx=0).stop 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 shutdown_connections 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cb406b810 0x7f1cb406dcd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cd00c55f0 0x7f1cd0151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cd00bdf20 0x7f1cd00c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 --2- 172.21.15.78:0/219747518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cd00ac390 0x7f1cd00be6b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 >> 172.21.15.78:0/219747518 conn(0x7f1cd00ca890 msgr2=0x7f1cd00aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 shutdown_connections 2024-02-19T14:55:31.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.516+0000 7f1cd7ea4640 1 -- 172.21.15.78:0/219747518 wait complete. 2024-02-19T14:55:31.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-02-19T14:55:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 Processor -- start 2024-02-19T14:55:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 -- start start 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f678806d550 0x7f67880c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 0x7f6788158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67880c6af0 con 0x7f6788155f90 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67880c7270 con 0x7f678806d550 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.626+0000 7f678e508640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880c7940 con 0x7f67881588a0 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f678caa5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 0x7f6788158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f67877fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f678806d550 0x7f67880c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f6787fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6788155f90 0x7f67880c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:49532/0 (socket says 172.21.15.78:49532) 2024-02-19T14:55:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f678caa5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 0x7f6788158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:55094/0 (socket says 172.21.15.78:55094) 2024-02-19T14:55:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f6787fff640 1 -- 172.21.15.78:0/235487737 learned_addr learned my addr 172.21.15.78:0/235487737 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f678caa5640 1 -- 172.21.15.78:0/235487737 >> v2:172.21.15.150:3300/0 conn(0x7f678806d550 msgr2=0x7f67880c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f678caa5640 1 --2- 172.21.15.78:0/235487737 >> v2:172.21.15.150:3300/0 conn(0x7f678806d550 0x7f67880c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f678caa5640 1 -- 172.21.15.78:0/235487737 >> v1:172.21.15.150:6789/0 conn(0x7f6788155f90 legacy=0x7f67880c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.627+0000 7f678caa5640 1 -- 172.21.15.78:0/235487737 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67880c7c20 con 0x7f67881588a0 2024-02-19T14:55:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.628+0000 7f678caa5640 1 --2- 172.21.15.78:0/235487737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 0x7f6788158cf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f67880c73b0 tx=0x7f678002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d66957bde25d0358 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.628+0000 7f6786ffd640 1 -- 172.21.15.78:0/235487737 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678002cbb0 con 0x7f67881588a0 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.628+0000 7f6786ffd640 1 -- 172.21.15.78:0/235487737 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678002cd10 con 0x7f67881588a0 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.628+0000 7f6786ffd640 1 -- 172.21.15.78:0/235487737 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780034660 con 0x7f67881588a0 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 -- 172.21.15.78:0/235487737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 msgr2=0x7f6788158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 --2- 172.21.15.78:0/235487737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 0x7f6788158cf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f67880c73b0 tx=0x7f678002be00 comp rx=0 tx=0).stop 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 -- 172.21.15.78:0/235487737 shutdown_connections 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 --2- 172.21.15.78:0/235487737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67881588a0 0x7f6788158cf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 --2- 172.21.15.78:0/235487737 >> v2:172.21.15.150:3300/0 conn(0x7f678806d550 0x7f67880c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 -- 172.21.15.78:0/235487737 >> 172.21.15.78:0/235487737 conn(0x7f678806b700 msgr2=0x7f67880aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 -- 172.21.15.78:0/235487737 shutdown_connections 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.629+0000 7f678e508640 1 -- 172.21.15.78:0/235487737 wait complete. 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.630+0000 7f678e508640 1 Processor -- start 2024-02-19T14:55:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.630+0000 7f678e508640 1 -- start start 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678e508640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f678806d550 0x7f67881eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678e508640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6788155f90 0x7f67881dea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678e508640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 0x7f67881df770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678e508640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67880704d0 con 0x7f67881588a0 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678e508640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67881e1a00 con 0x7f6788155f90 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678e508640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678806f850 con 0x7f678806d550 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 0x7f67881df770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 0x7f67881df770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51346/0 (socket says 172.21.15.78:51346) 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 -- 172.21.15.78:0/1740298714 learned_addr learned my addr 172.21.15.78:0/1740298714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f6787fff640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f678806d550 0x7f67881eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f67877fe640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6788155f90 0x7f67881dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 -- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f678806d550 msgr2=0x7f67881eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f678806d550 0x7f67881eb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 -- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6788155f90 msgr2=0x7f67881dea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6788155f90 0x7f67881dea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 -- 172.21.15.78:0/1740298714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780009a70 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.631+0000 7f678caa5640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 0x7f67881df770 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f678002c410 tx=0x7f6780004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.632+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780040650 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.632+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67800407b0 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.632+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678806fb30 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.632+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780040b60 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.632+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67880700f0 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.632+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6788070610 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.633+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6780040cc0 con 0x7f67881588a0 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.633+0000 7f6784ff9640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f677406b810 0x7f677406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.633+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f67800acd70 con 0x7f67881588a0 2024-02-19T14:55:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.633+0000 7f6787fff640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f677406b810 0x7f677406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.634+0000 7f6787fff640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f677406b810 0x7f677406dcd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6778000f60 tx=0x7f6778009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.638+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6788070610 con 0x7f67881588a0 2024-02-19T14:55:31.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.875+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7f6788070610 con 0x7f67881588a0 2024-02-19T14:55:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.904+0000 7f6784ff9640 1 -- 172.21.15.78:0/1740298714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v223) v1 ==== 146+0+0 (secure 0 0 0) 0x7f6788070610 con 0x7f67881588a0 2024-02-19T14:55:31.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f677406b810 msgr2=0x7f677406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f677406b810 0x7f677406dcd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6778000f60 tx=0x7f6778009040 comp rx=0 tx=0).stop 2024-02-19T14:55:31.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 msgr2=0x7f67881df770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:31.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 0x7f67881df770 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f678002c410 tx=0x7f6780004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:31.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 shutdown_connections 2024-02-19T14:55:31.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f677406b810 0x7f677406dcd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67881588a0 0x7f67881df770 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6788155f90 0x7f67881dea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 --2- 172.21.15.78:0/1740298714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f678806d550 0x7f67881eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:31.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 >> 172.21.15.78:0/1740298714 conn(0x7f678806b700 msgr2=0x7f67881537b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:31.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 shutdown_connections 2024-02-19T14:55:31.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:31.906+0000 7f678e508640 1 -- 172.21.15.78:0/1740298714 wait complete. 2024-02-19T14:55:31.920 INFO:tasks.cephfs_test_runner:That setting mds_join_fs at runtime affects the cluster layout. ... ok 2024-02-19T14:55:31.920 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) That a standby-replay daemon with weak affinity is replaced by a stronger one. 2024-02-19T14:55:31.920 INFO:tasks.cephfs_test_runner:test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:55:31.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:32.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.017+0000 7fb6e9495640 1 Processor -- start 2024-02-19T14:55:32.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.017+0000 7fb6e9495640 1 -- start start 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e9495640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6e41496b0 0x7fb6e4153900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e9495640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e40c4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e9495640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6e40c4870 0x7fb6e406f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e9495640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e40a9de0 con 0x7fb6e40c4870 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e9495640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e40a9480 con 0x7fb6e41496b0 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e9495640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e40a9720 con 0x7fb6e40c7bf0 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6e40c4870 0x7fb6e406f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6e40c4870 0x7fb6e406f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51358/0 (socket says 172.21.15.78:51358) 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 -- 172.21.15.78:0/2190308968 learned_addr learned my addr 172.21.15.78:0/2190308968 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:32.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e2ffd640 1 --2- 172.21.15.78:0/2190308968 >> v2:172.21.15.150:3300/0 conn(0x7fb6e41496b0 0x7fb6e4153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6dbfff640 1 --2- 172.21.15.78:0/2190308968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e40c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 -- 172.21.15.78:0/2190308968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 msgr2=0x7fb6e40c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 --2- 172.21.15.78:0/2190308968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e40c4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 -- 172.21.15.78:0/2190308968 >> v2:172.21.15.150:3300/0 conn(0x7fb6e41496b0 msgr2=0x7fb6e4153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 --2- 172.21.15.78:0/2190308968 >> v2:172.21.15.150:3300/0 conn(0x7fb6e41496b0 0x7fb6e4153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e37fe640 1 -- 172.21.15.78:0/2190308968 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e40a9a00 con 0x7fb6e40c4870 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.018+0000 7fb6e2ffd640 1 --2- 172.21.15.78:0/2190308968 >> v2:172.21.15.150:3300/0 conn(0x7fb6e41496b0 0x7fb6e4153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.019+0000 7fb6e37fe640 1 --2- 172.21.15.78:0/2190308968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e40c4870 0x7fb6e406f380 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40c42c0 tx=0x7fb6dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0899c91d2e8686b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.019+0000 7fb6e27fc640 1 -- 172.21.15.78:0/2190308968 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc02c9e0 con 0x7fb6e40c4870 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.019+0000 7fb6e27fc640 1 -- 172.21.15.78:0/2190308968 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc02cb40 con 0x7fb6e40c4870 2024-02-19T14:55:32.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.019+0000 7fb6e27fc640 1 -- 172.21.15.78:0/2190308968 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0343f0 con 0x7fb6e40c4870 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 -- 172.21.15.78:0/2190308968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e40c4870 msgr2=0x7fb6e406f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 --2- 172.21.15.78:0/2190308968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e40c4870 0x7fb6e406f380 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40c42c0 tx=0x7fb6dc02be40 comp rx=0 tx=0).stop 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 -- 172.21.15.78:0/2190308968 shutdown_connections 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 --2- 172.21.15.78:0/2190308968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e40c4870 0x7fb6e406f380 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 --2- 172.21.15.78:0/2190308968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e40c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 --2- 172.21.15.78:0/2190308968 >> v2:172.21.15.150:3300/0 conn(0x7fb6e41496b0 0x7fb6e4153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 -- 172.21.15.78:0/2190308968 >> 172.21.15.78:0/2190308968 conn(0x7fb6e406ea10 msgr2=0x7fb6e40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 -- 172.21.15.78:0/2190308968 shutdown_connections 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.020+0000 7fb6e9495640 1 -- 172.21.15.78:0/2190308968 wait complete. 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.021+0000 7fb6e9495640 1 Processor -- start 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.021+0000 7fb6e9495640 1 -- start start 2024-02-19T14:55:32.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e40c4870 0x7fb6e41eba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e413f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 0x7fb6e413fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4156c50 con 0x7fb6e41496b0 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4140270 con 0x7fb6e40c4870 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e41553e0 con 0x7fb6e40c7bf0 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 0x7fb6e413fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 0x7fb6e413fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51364/0 (socket says 172.21.15.78:51364) 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 -- 172.21.15.78:0/1921836557 learned_addr learned my addr 172.21.15.78:0/1921836557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e2ffd640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e40c4870 0x7fb6e41eba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6dbfff640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e413f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 -- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 msgr2=0x7fb6e413f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e413f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 -- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e40c4870 msgr2=0x7fb6e41eba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e40c4870 0x7fb6e41eba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 -- 172.21.15.78:0/1921836557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc009b30 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e37fe640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 0x7fb6e413fd30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009f40 tx=0x7fb6dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc009510 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc03e450 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc03e800 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e41ec770 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.022+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e4154dd0 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.023+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e4155590 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.024+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb6dc03e960 con 0x7fb6e41496b0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.024+0000 7fb6db7fe640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b840 0x7fb6c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.024+0000 7fb6e2ffd640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b840 0x7fb6c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.024+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb6dc0ad2b0 con 0x7fb6e41496b0 2024-02-19T14:55:32.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.025+0000 7fb6e2ffd640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b840 0x7fb6c406dd00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400a990 tx=0x7fb6d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.029+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb6e4155590 con 0x7fb6e41496b0 2024-02-19T14:55:32.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.202+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6e4155550 con 0x7fb6e41496b0 2024-02-19T14:55:32.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.202+0000 7fb6db7fe640 1 -- 172.21.15.78:0/1921836557 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+13413 (secure 0 0 0) 0x7fb6dc078050 con 0x7fb6e41496b0 2024-02-19T14:55:32.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:32.205 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":102,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:30.895621+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6832/408965047":"2024-02-20T14:55:27.706219+0000","172.21.15.78:6836/3610724432":"2024-02-20T14:55:18.858155+0000","172.21.15.150:6834/408965047":"2024-02-20T14:55:27.706219+0000","172.21.15.78:6834/3610724432":"2024-02-20T14:55:18.858155+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:32.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.204+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b840 msgr2=0x7fb6c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.204+0000 7fb6e9495640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b840 0x7fb6c406dd00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400a990 tx=0x7fb6d4008040 comp rx=0 tx=0).stop 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.204+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 msgr2=0x7fb6e413fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.204+0000 7fb6e9495640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 0x7fb6e413fd30 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009f40 tx=0x7fb6dc004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.204+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 shutdown_connections 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb6c406b840 0x7fb6c406dd00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6e41496b0 0x7fb6e413fd30 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6e40c7bf0 0x7fb6e413f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 --2- 172.21.15.78:0/1921836557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6e40c4870 0x7fb6e41eba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 >> 172.21.15.78:0/1921836557 conn(0x7fb6e406ea10 msgr2=0x7fb6e40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 shutdown_connections 2024-02-19T14:55:32.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.205+0000 7fb6e9495640 1 -- 172.21.15.78:0/1921836557 wait complete. 2024-02-19T14:55:32.218 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:55:32.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:55:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 Processor -- start 2024-02-19T14:55:32.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 -- start start 2024-02-19T14:55:32.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52201526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52200c5be0 0x7f522006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5220072250 con 0x7f52200c25e0 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5220152c90 con 0x7f52200c5be0 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f522691e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5220072480 con 0x7f52200c29c0 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f5224ebb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52200c5be0 0x7f522006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f521ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f52200c25e0 0x7f52200c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34292/0 (socket says 172.21.15.78:34292) 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f5224ebb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f52200c5be0 0x7f522006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51380/0 (socket says 172.21.15.78:51380) 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.319+0000 7f521ffff640 1 -- 172.21.15.78:0/1717162634 learned_addr learned my addr 172.21.15.78:0/1717162634 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.320+0000 7f521f7fe640 1 --2- 172.21.15.78:0/1717162634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52201526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.320+0000 7f5224ebb640 1 -- 172.21.15.78:0/1717162634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 msgr2=0x7f52201526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.320+0000 7f5224ebb640 1 --2- 172.21.15.78:0/1717162634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52201526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.320+0000 7f5224ebb640 1 -- 172.21.15.78:0/1717162634 >> v1:172.21.15.78:6789/0 conn(0x7f52200c25e0 legacy=0x7f52200c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.320+0000 7f5224ebb640 1 -- 172.21.15.78:0/1717162634 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200c4b90 con 0x7f52200c5be0 2024-02-19T14:55:32.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.320+0000 7f521effd640 1 -- 172.21.15.78:0/1717162634 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620817054 0 0) 0x7f5220072250 con 0x7f52200c25e0 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f5224ebb640 1 --2- 172.21.15.78:0/1717162634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f522006e420 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5218009dc0 tx=0x7f521802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75be6ff67c943cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f521effd640 1 -- 172.21.15.78:0/1717162634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521802cbb0 con 0x7f52200c5be0 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f521effd640 1 -- 172.21.15.78:0/1717162634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521802cd10 con 0x7f52200c5be0 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f521effd640 1 -- 172.21.15.78:0/1717162634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218034660 con 0x7f52200c5be0 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 -- 172.21.15.78:0/1717162634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 msgr2=0x7f522006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 --2- 172.21.15.78:0/1717162634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f522006e420 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5218009dc0 tx=0x7f521802be00 comp rx=0 tx=0).stop 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 -- 172.21.15.78:0/1717162634 shutdown_connections 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 --2- 172.21.15.78:0/1717162634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f522006e420 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 --2- 172.21.15.78:0/1717162634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52201526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 -- 172.21.15.78:0/1717162634 >> 172.21.15.78:0/1717162634 conn(0x7f522006b270 msgr2=0x7f522006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 -- 172.21.15.78:0/1717162634 shutdown_connections 2024-02-19T14:55:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.321+0000 7f522691e640 1 -- 172.21.15.78:0/1717162634 wait complete. 2024-02-19T14:55:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.322+0000 7f522691e640 1 Processor -- start 2024-02-19T14:55:32.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 -- start start 2024-02-19T14:55:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52200c25e0 0x7f52201eab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52200bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f52201e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52201eb090 con 0x7f52200c5be0 2024-02-19T14:55:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522014a850 con 0x7f52200c25e0 2024-02-19T14:55:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f522691e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52200a8700 con 0x7f52200c29c0 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f5224ebb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f52201e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f5224ebb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f52201e7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51382/0 (socket says 172.21.15.78:51382) 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f5224ebb640 1 -- 172.21.15.78:0/1650961163 learned_addr learned my addr 172.21.15.78:0/1650961163 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f521ffff640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52200c25e0 0x7f52201eab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.323+0000 7f521f7fe640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52200bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f5224ebb640 1 -- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 msgr2=0x7f52200bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f5224ebb640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52200bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f5224ebb640 1 -- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52200c25e0 msgr2=0x7f52201eab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f5224ebb640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52200c25e0 0x7f52201eab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f5224ebb640 1 -- 172.21.15.78:0/1650961163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5218009a70 con 0x7f52200c5be0 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f521ffff640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52200c25e0 0x7f52201eab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f5224ebb640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f52201e7240 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5218037040 tx=0x7f5218004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218040650 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52180407b0 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52200a89e0 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218040b60 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.324+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52200a8fa0 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.325+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5220074990 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.325+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5218040cc0 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.326+0000 7f521cff9640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520c06b810 0x7f520c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.326+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f52180ace40 con 0x7f52200c5be0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.326+0000 7f521ffff640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520c06b810 0x7f520c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.326+0000 7f521ffff640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520c06b810 0x7f520c06dcd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5210000f60 tx=0x7f5210009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.330+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5218077a10 con 0x7f52200c5be0 2024-02-19T14:55:32.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.500+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52200c53e0 con 0x7f52200c5be0 2024-02-19T14:55:32.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.501+0000 7f521cff9640 1 -- 172.21.15.78:0/1650961163 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5218044020 con 0x7f52200c5be0 2024-02-19T14:55:32.504 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520c06b810 msgr2=0x7f520c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520c06b810 0x7f520c06dcd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5210000f60 tx=0x7f5210009040 comp rx=0 tx=0).stop 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 msgr2=0x7f52201e7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f52201e7240 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5218037040 tx=0x7f5218004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 shutdown_connections 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520c06b810 0x7f520c06dcd0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52200c5be0 0x7f52201e7240 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52200c29c0 0x7f52200bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 --2- 172.21.15.78:0/1650961163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52200c25e0 0x7f52201eab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 >> 172.21.15.78:0/1650961163 conn(0x7f522006b270 msgr2=0x7f52200ac370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:32.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 shutdown_connections 2024-02-19T14:55:32.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.503+0000 7f522691e640 1 -- 172.21.15.78:0/1650961163 wait complete. 2024-02-19T14:55:32.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-02-19T14:55:32.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.613+0000 7f824108f640 1 Processor -- start 2024-02-19T14:55:32.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.613+0000 7f824108f640 1 -- start start 2024-02-19T14:55:32.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f824108f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f823c0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f824108f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f823c0c15b0 0x7f823c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f824108f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823c0c1120 con 0x7f823c0c5270 2024-02-19T14:55:32.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f824108f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f823c0c6080 con 0x7f823c0c15b0 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f824108f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0c57e0 con 0x7f823c14ece0 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f823a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f823c0c5270 0x7f823c0719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34306/0 (socket says 172.21.15.78:34306) 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f823a59c640 1 -- 172.21.15.78:0/3922682867 learned_addr learned my addr 172.21.15.78:0/3922682867 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789964670 0 0) 0x7f823c0c1120 con 0x7f823c0c5270 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8224004630 con 0x7f823c0c5270 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f823b59e640 1 --2- 172.21.15.78:0/3922682867 >> v2:172.21.15.150:3300/0 conn(0x7f823c0c15b0 0x7f823c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f823c0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718290472 0 0) 0x7f8224004630 con 0x7f823c0c5270 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 msgr2=0x7f823c0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f823c0c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 >> v2:172.21.15.150:3300/0 conn(0x7f823c0c15b0 msgr2=0x7f823c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 --2- 172.21.15.78:0/3922682867 >> v2:172.21.15.150:3300/0 conn(0x7f823c0c15b0 0x7f823c0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c0c5ac0 con 0x7f823c0c5270 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f8230001670 con 0x7f823c0c5270 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 >> v1:172.21.15.78:6789/0 conn(0x7f823c0c5270 legacy=0x7f823c0719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f8224004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240058d0 0x7f8224005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f8224006290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0c6080 con 0x7f8224004a80 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0c57e0 con 0x7f82240058d0 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.614+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822400cb40 con 0x7f823c14ece0 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f8224004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823a59c640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240058d0 0x7f8224005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823b59e640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f8224006290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823ad9d640 1 -- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 msgr2=0x7f8224006290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f8224006290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823ad9d640 1 -- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240058d0 msgr2=0x7f8224005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:32.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240058d0 0x7f8224005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823ad9d640 1 -- 172.21.15.78:0/3922682867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8224004630 con 0x7f8224004a80 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823b59e640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c14ece0 0x7f8224006290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.615+0000 7f823a59c640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240058d0 0x7f8224005d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f8224004ea0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f8234007330 tx=0x7f8224006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8850bd3fb8308a39 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823402cc70 con 0x7f8224004a80 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823402cdd0 con 0x7f8224004a80 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f8239d9b640 1 -- 172.21.15.78:0/3922682867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234034660 con 0x7f8224004a80 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f824108f640 1 -- 172.21.15.78:0/3922682867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 msgr2=0x7f8224004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f824108f640 1 --2- 172.21.15.78:0/3922682867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f8224004ea0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f8234007330 tx=0x7f8224006900 comp rx=0 tx=0).stop 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f823b59e640 1 -- 172.21.15.78:0/3922682867 reap_dead start 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f824108f640 1 -- 172.21.15.78:0/3922682867 shutdown_connections 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f824108f640 1 -- 172.21.15.78:0/3922682867 >> 172.21.15.78:0/3922682867 conn(0x7f823c06b270 msgr2=0x7f823c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f824108f640 1 -- 172.21.15.78:0/3922682867 shutdown_connections 2024-02-19T14:55:32.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.616+0000 7f824108f640 1 -- 172.21.15.78:0/3922682867 wait complete. 2024-02-19T14:55:32.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 Processor -- start 2024-02-19T14:55:32.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 -- start start 2024-02-19T14:55:32.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f823c147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c146430 0x7f823c1509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c148180 0x7f823c150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c1ebe50 con 0x7f8224004a80 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c1e8570 con 0x7f823c146430 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f824108f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c1e9540 con 0x7f823c148180 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f823ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f823c147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f823ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f823c147c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51388/0 (socket says 172.21.15.78:51388) 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.618+0000 7f823ad9d640 1 -- 172.21.15.78:0/3363067601 learned_addr learned my addr 172.21.15.78:0/3363067601 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823a59c640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c146430 0x7f823c1509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823b59e640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c148180 0x7f823c150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823ad9d640 1 -- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c148180 msgr2=0x7f823c150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c148180 0x7f823c150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823ad9d640 1 -- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c146430 msgr2=0x7f823c1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c146430 0x7f823c1509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823ad9d640 1 -- 172.21.15.78:0/3363067601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c1e8780 con 0x7f8224004a80 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f823ad9d640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f823c147c40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f8234007450 tx=0x7f8234004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234003b90 con 0x7f8224004a80 2024-02-19T14:55:32.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8234003d10 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823c1e8a60 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234004260 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.619+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823c1e9020 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.620+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823c1e91e0 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.620+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f82340043c0 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.621+0000 7f82237fe640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822806b840 0x7f822806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.621+0000 7f823a59c640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822806b840 0x7f822806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.621+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f82340ac220 con 0x7f8224004a80 2024-02-19T14:55:32.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.621+0000 7f823a59c640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822806b840 0x7f822806dd00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f823c1ecba0 tx=0x7f8230009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:32.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.625+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8234076ed0 con 0x7f8224004a80 2024-02-19T14:55:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.865+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7f823c074990 con 0x7f8224004a80 2024-02-19T14:55:32.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.910+0000 7f82237fe640 1 -- 172.21.15.78:0/3363067601 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v224) v1 ==== 156+0+0 (secure 0 0 0) 0x7f8234003e80 con 0x7f8224004a80 2024-02-19T14:55:32.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822806b840 msgr2=0x7f822806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822806b840 0x7f822806dd00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f823c1ecba0 tx=0x7f8230009040 comp rx=0 tx=0).stop 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 msgr2=0x7f823c147c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f823c147c40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f8234007450 tx=0x7f8234004ab0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 shutdown_connections 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f822806b840 0x7f822806dd00 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f823c148180 0x7f823c150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823c146430 0x7f823c1509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 --2- 172.21.15.78:0/3363067601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224004a80 0x7f823c147c40 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 >> 172.21.15.78:0/3363067601 conn(0x7f823c06b270 msgr2=0x7f823c0c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 shutdown_connections 2024-02-19T14:55:32.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:32.912+0000 7f824108f640 1 -- 172.21.15.78:0/3363067601 wait complete. 2024-02-19T14:55:32.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:55:33.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.026+0000 7fd8862ae640 1 Processor -- start 2024-02-19T14:55:33.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd8862ae640 1 -- start start 2024-02-19T14:55:33.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd8862ae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd88006e830 0x7fd880154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd8862ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd88006ff80 0x7fd8800c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd8862ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88014f730 con 0x7fd8800c6d60 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd8862ae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800c1940 con 0x7fd88006e830 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd8862ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800c16d0 con 0x7fd88006ff80 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd87f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd88006e830 0x7fd880154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd87ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd8800c6d60 0x7fd8800c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34310/0 (socket says 172.21.15.78:34310) 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.027+0000 7fd87ffff640 1 -- 172.21.15.78:0/1089257395 learned_addr learned my addr 172.21.15.78:0/1089257395 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd88484b640 1 --2- 172.21.15.78:0/1089257395 >> v2:172.21.15.150:3300/0 conn(0x7fd88006ff80 0x7fd8800c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1089257395 >> v2:172.21.15.150:3300/0 conn(0x7fd88006ff80 msgr2=0x7fd8800c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd87f7fe640 1 --2- 172.21.15.78:0/1089257395 >> v2:172.21.15.150:3300/0 conn(0x7fd88006ff80 0x7fd8800c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1089257395 >> v1:172.21.15.78:6789/0 conn(0x7fd8800c6d60 legacy=0x7fd8800c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd87effd640 1 -- 172.21.15.78:0/1089257395 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97828630 0 0) 0x7fd88014f730 con 0x7fd8800c6d60 2024-02-19T14:55:33.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1089257395 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880072f70 con 0x7fd88006e830 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.028+0000 7fd87f7fe640 1 --2- 172.21.15.78:0/1089257395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006e830 0x7fd880154f70 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd88006de00 tx=0x7fd87402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90756a09d736249d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd87effd640 1 -- 172.21.15.78:0/1089257395 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874031830 con 0x7fd88006e830 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd87effd640 1 -- 172.21.15.78:0/1089257395 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd874031990 con 0x7fd88006e830 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd87effd640 1 -- 172.21.15.78:0/1089257395 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874031d20 con 0x7fd88006e830 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 -- 172.21.15.78:0/1089257395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006e830 msgr2=0x7fd880154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1089257395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006e830 0x7fd880154f70 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd88006de00 tx=0x7fd87402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 -- 172.21.15.78:0/1089257395 shutdown_connections 2024-02-19T14:55:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1089257395 >> v2:172.21.15.150:3300/0 conn(0x7fd88006ff80 0x7fd8800c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1089257395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006e830 0x7fd880154f70 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 -- 172.21.15.78:0/1089257395 >> 172.21.15.78:0/1089257395 conn(0x7fd88006c170 msgr2=0x7fd88006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 -- 172.21.15.78:0/1089257395 shutdown_connections 2024-02-19T14:55:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.029+0000 7fd8862ae640 1 -- 172.21.15.78:0/1089257395 wait complete. 2024-02-19T14:55:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.030+0000 7fd8862ae640 1 Processor -- start 2024-02-19T14:55:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.030+0000 7fd8862ae640 1 -- start start 2024-02-19T14:55:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd8862ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88006e830 0x7fd8800c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd8862ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 0x7fd8800a7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd8862ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8800c6d60 0x7fd8800c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd8862ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800c43f0 con 0x7fd88006ff80 2024-02-19T14:55:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd8862ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8801eeba0 con 0x7fd88006e830 2024-02-19T14:55:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd8862ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88014aff0 con 0x7fd8800c6d60 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 0x7fd8800a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 0x7fd8800a7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51406/0 (socket says 172.21.15.78:51406) 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1076082153 learned_addr learned my addr 172.21.15.78:0/1076082153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd87ffff640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88006e830 0x7fd8800c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd88484b640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8800c6d60 0x7fd8800c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.031+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8800c6d60 msgr2=0x7fd8800c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87f7fe640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8800c6d60 0x7fd8800c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88006e830 msgr2=0x7fd8800c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87f7fe640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88006e830 0x7fd8800c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87f7fe640 1 -- 172.21.15.78:0/1076082153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd874009400 con 0x7fd88006ff80 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87f7fe640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 0x7fd8800a7730 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd874000c00 tx=0x7fd8740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87403aa60 con 0x7fd88006ff80 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87403abe0 con 0x7fd88006ff80 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800c4110 con 0x7fd88006ff80 2024-02-19T14:55:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740443f0 con 0x7fd88006ff80 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.032+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800c2e10 con 0x7fd88006ff80 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.033+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800c2990 con 0x7fd88006ff80 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.033+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd874044550 con 0x7fd88006ff80 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.034+0000 7fd87cff9640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd86c06b810 0x7fd86c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.034+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd8740ac240 con 0x7fd88006ff80 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.034+0000 7fd87ffff640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd86c06b810 0x7fd86c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.034+0000 7fd87ffff640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd86c06b810 0x7fd86c06dcd0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd870000f60 tx=0x7fd870007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.038+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd8800c2990 con 0x7fd88006ff80 2024-02-19T14:55:33.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.211+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd8800c2990 con 0x7fd88006ff80 2024-02-19T14:55:33.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.214+0000 7fd87cff9640 1 -- 172.21.15.78:0/1076082153 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd8800c2990 con 0x7fd88006ff80 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd86c06b810 msgr2=0x7fd86c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd86c06b810 0x7fd86c06dcd0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd870000f60 tx=0x7fd870007b70 comp rx=0 tx=0).stop 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 msgr2=0x7fd8800a7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 0x7fd8800a7730 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd874000c00 tx=0x7fd8740046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 shutdown_connections 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd86c06b810 0x7fd86c06dcd0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd88006ff80 0x7fd8800a7730 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8800c6d60 0x7fd8800c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 --2- 172.21.15.78:0/1076082153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd88006e830 0x7fd8800c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 >> 172.21.15.78:0/1076082153 conn(0x7fd88006c170 msgr2=0x7fd8800c0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:33.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 shutdown_connections 2024-02-19T14:55:33.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.216+0000 7fd8862ae640 1 -- 172.21.15.78:0/1076082153 wait complete. 2024-02-19T14:55:33.229 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:55:33.229 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:55:33.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:55:33.319 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:55:33.319 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:55:33.319 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:55:33.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:55:33.335 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:55:33.335 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:55:33.353 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:55:33.353 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:55:33.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:33.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 Processor -- start 2024-02-19T14:55:33.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 -- start start 2024-02-19T14:55:33.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ddc0c8500 0x7f4ddc0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c42e0 0x7f4ddc158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ddc0c2f50 0x7f4ddc158b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc06c090 con 0x7f4ddc0c2f50 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc06d6a0 con 0x7f4ddc0c8500 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.473+0000 7f4de2f1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0c34a0 con 0x7f4ddc0c42e0 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ddc0c2f50 0x7f4ddc158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ddc0c2f50 0x7f4ddc158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51412/0 (socket says 172.21.15.78:51412) 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 -- 172.21.15.78:0/335426930 learned_addr learned my addr 172.21.15.78:0/335426930 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/335426930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c42e0 0x7f4ddc158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de0cba640 1 --2- 172.21.15.78:0/335426930 >> v2:172.21.15.150:3300/0 conn(0x7f4ddc0c8500 0x7f4ddc0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 -- 172.21.15.78:0/335426930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c42e0 msgr2=0x7f4ddc158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 --2- 172.21.15.78:0/335426930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c42e0 0x7f4ddc158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 -- 172.21.15.78:0/335426930 >> v2:172.21.15.150:3300/0 conn(0x7f4ddc0c8500 msgr2=0x7f4ddc0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:33.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 --2- 172.21.15.78:0/335426930 >> v2:172.21.15.150:3300/0 conn(0x7f4ddc0c8500 0x7f4ddc0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.474+0000 7f4de14bb640 1 -- 172.21.15.78:0/335426930 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc06c8e0 con 0x7f4ddc0c2f50 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de14bb640 1 --2- 172.21.15.78:0/335426930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc158b90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0c9350 tx=0x7f4dcc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe895cbd5e68580 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/335426930 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0317e0 con 0x7f4ddc0c2f50 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/335426930 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc031940 con 0x7f4ddc0c2f50 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/335426930 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc031cd0 con 0x7f4ddc0c2f50 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 -- 172.21.15.78:0/335426930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc0c2f50 msgr2=0x7f4ddc158b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/335426930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc158b90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0c9350 tx=0x7f4dcc02b880 comp rx=0 tx=0).stop 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 -- 172.21.15.78:0/335426930 shutdown_connections 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/335426930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc158b90 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/335426930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c42e0 0x7f4ddc158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/335426930 >> v2:172.21.15.150:3300/0 conn(0x7f4ddc0c8500 0x7f4ddc0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 -- 172.21.15.78:0/335426930 >> 172.21.15.78:0/335426930 conn(0x7f4ddc0aa300 msgr2=0x7f4ddc0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 -- 172.21.15.78:0/335426930 shutdown_connections 2024-02-19T14:55:33.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.475+0000 7f4de2f1e640 1 -- 172.21.15.78:0/335426930 wait complete. 2024-02-19T14:55:33.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.476+0000 7f4de2f1e640 1 Processor -- start 2024-02-19T14:55:33.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.476+0000 7f4de2f1e640 1 -- start start 2024-02-19T14:55:33.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de2f1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc1503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de2f1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c8500 0x7f4ddc14ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de2f1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 0x7f4ddc14d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de2f1e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc14d620 con 0x7f4ddc14cc50 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de2f1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc151280 con 0x7f4ddc0c2f50 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de2f1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc06fe70 con 0x7f4ddc0c8500 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de14bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 0x7f4ddc14d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de14bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 0x7f4ddc14d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51420/0 (socket says 172.21.15.78:51420) 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.477+0000 7f4de14bb640 1 -- 172.21.15.78:0/1684079435 learned_addr learned my addr 172.21.15.78:0/1684079435 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c8500 0x7f4ddc14ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de0cba640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc1503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de14bb640 1 -- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c8500 msgr2=0x7f4ddc14ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de14bb640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c8500 0x7f4ddc14ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de14bb640 1 -- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ddc0c2f50 msgr2=0x7f4ddc1503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de14bb640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc1503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de14bb640 1 -- 172.21.15.78:0/1684079435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0094d0 con 0x7f4ddc14cc50 2024-02-19T14:55:33.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de0cba640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc1503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c8500 0x7f4ddc14ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de14bb640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 0x7f4ddc14d0e0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0090b0 tx=0x7f4dcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc032880 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc032a00 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc032db0 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc0abe00 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.478+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc150930 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.479+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc071c20 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.479+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4dcc0453f0 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.480+0000 7f4dd97fa640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dbc06b840 0x7f4dbc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.480+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4dcc0ac590 con 0x7f4ddc14cc50 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.480+0000 7f4de0cba640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dbc06b840 0x7f4dbc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.480+0000 7f4de0cba640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dbc06b840 0x7f4dbc06dd00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009a00 tx=0x7f4dd4003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.484+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4dcc0772c0 con 0x7f4ddc14cc50 2024-02-19T14:55:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.689+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ddc06ffe0 con 0x7f4ddc14cc50 2024-02-19T14:55:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.689+0000 7f4dd97fa640 1 -- 172.21.15.78:0/1684079435 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f4ddc06ffe0 con 0x7f4ddc14cc50 2024-02-19T14:55:33.692 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:33.692 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":80,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6310,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/829046094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":829046094},{"type":"v1","addr":"172.21.15.150:6835","nonce":829046094}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":6538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1812635560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1812635560},{"type":"v1","addr":"172.21.15.150:6834","nonce":1812635560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2386100646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2386100646},{"type":"v1","addr":"172.21.15.78:6837","nonce":2386100646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":7128,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1361734331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1361734331},{"type":"v1","addr":"172.21.15.78:6836","nonce":1361734331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[]} 2024-02-19T14:55:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 80 2024-02-19T14:55:33.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dbc06b840 msgr2=0x7f4dbc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dbc06b840 0x7f4dbc06dd00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009a00 tx=0x7f4dd4003990 comp rx=0 tx=0).stop 2024-02-19T14:55:33.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 msgr2=0x7f4ddc14d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 0x7f4ddc14d0e0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0090b0 tx=0x7f4dcc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 shutdown_connections 2024-02-19T14:55:33.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dbc06b840 0x7f4dbc06dd00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ddc14cc50 0x7f4ddc14d0e0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ddc0c8500 0x7f4ddc14ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 --2- 172.21.15.78:0/1684079435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ddc0c2f50 0x7f4ddc1503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 >> 172.21.15.78:0/1684079435 conn(0x7f4ddc0aa300 msgr2=0x7f4ddc0c7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 shutdown_connections 2024-02-19T14:55:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.691+0000 7f4de2f1e640 1 -- 172.21.15.78:0/1684079435 wait complete. 2024-02-19T14:55:33.705 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:55:33.705 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:55:33.705 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:55:33.705 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:33.705 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:55:33.705 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:55:33.706 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:55:33.706 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:33.706 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:55:33.706 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:55:33.706 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:55:33.706 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:33.706 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:55:33.707 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:55:33.707 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:55:33.707 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:55:33.715 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:55:33.715 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:55:33.716 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:55:33.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:55:33.718 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:55:33.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:55:33.719 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:55:33.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:55:33.760 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:55:33.761 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:55:33.761 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:33.761 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:33.763 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:55:33.763 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:33.764 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:55:33.764 DEBUG:teuthology.parallel:result is None 2024-02-19T14:55:33.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:55:33.834 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:55:33.834 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:55:33.849 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:55:33.850 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:55:33.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.866+0000 7f80fe3d7640 1 Processor -- start 2024-02-19T14:55:33.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.866+0000 7f80fe3d7640 1 -- start start 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.867+0000 7f80fe3d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80f80add70 0x7f80f80c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.867+0000 7f80fe3d7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80f80c2f60 0x7f80f806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.867+0000 7f80fe3d7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f80ae670 con 0x7f80f806f530 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.867+0000 7f80f77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80f80c2f60 0x7f80f806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.867+0000 7f80f77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80f80c2f60 0x7f80f806eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51468/0 (socket says 172.21.15.78:51468) 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.867+0000 7f80f77fe640 1 -- 172.21.15.78:0/413440365 learned_addr learned my addr 172.21.15.78:0/413440365 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f80716b0 con 0x7f80f80c2f60 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f80c33d0 con 0x7f80f80add70 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80f7fff640 1 --2- 172.21.15.78:0/413440365 >> v2:172.21.15.150:3300/0 conn(0x7f80f80add70 0x7f80f80c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80f77fe640 1 -- 172.21.15.78:0/413440365 >> v2:172.21.15.150:3300/0 conn(0x7f80f80add70 msgr2=0x7f80f80c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80f77fe640 1 --2- 172.21.15.78:0/413440365 >> v2:172.21.15.150:3300/0 conn(0x7f80f80add70 0x7f80f80c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80f77fe640 1 -- 172.21.15.78:0/413440365 >> v1:172.21.15.150:6789/0 conn(0x7f80f806f530 legacy=0x7f80f80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80f77fe640 1 -- 172.21.15.78:0/413440365 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f8153780 con 0x7f80f80c2f60 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.868+0000 7f80f77fe640 1 --2- 172.21.15.78:0/413440365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80c2f60 0x7f80f806eff0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f80f80c4fc0 tx=0x7f80ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92274ca45b96b683 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80f6ffd640 1 -- 172.21.15.78:0/413440365 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0317e0 con 0x7f80f80c2f60 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80f6ffd640 1 -- 172.21.15.78:0/413440365 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ec031940 con 0x7f80f80c2f60 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80f6ffd640 1 -- 172.21.15.78:0/413440365 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec031cd0 con 0x7f80f80c2f60 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80c2f60 msgr2=0x7f80f806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/413440365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80c2f60 0x7f80f806eff0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f80f80c4fc0 tx=0x7f80ec02b880 comp rx=0 tx=0).stop 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 shutdown_connections 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/413440365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80c2f60 0x7f80f806eff0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/413440365 >> v2:172.21.15.150:3300/0 conn(0x7f80f80add70 0x7f80f80c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 >> 172.21.15.78:0/413440365 conn(0x7f80f80c85e0 msgr2=0x7f80f80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 shutdown_connections 2024-02-19T14:55:33.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.869+0000 7f80fe3d7640 1 -- 172.21.15.78:0/413440365 wait complete. 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.870+0000 7f80fe3d7640 1 Processor -- start 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.870+0000 7f80fe3d7640 1 -- start start 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fe3d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80f806f530 0x7f80f81ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fe3d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 0x7f80f8154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fe3d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80f80c2f60 0x7f80f814d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fe3d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f8067ed0 con 0x7f80f80add70 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fe3d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80681b0 con 0x7f80f806f530 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fe3d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f81eb260 con 0x7f80f80c2f60 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 0x7f80f8154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 0x7f80f8154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51482/0 (socket says 172.21.15.78:51482) 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f77fe640 1 -- 172.21.15.78:0/867173750 learned_addr learned my addr 172.21.15.78:0/867173750 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80fc974640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80f80c2f60 0x7f80f814d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f7fff640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80f806f530 0x7f80f81ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f77fe640 1 -- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80f80c2f60 msgr2=0x7f80f814d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f77fe640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80f80c2f60 0x7f80f814d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.871+0000 7f80f77fe640 1 -- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80f806f530 msgr2=0x7f80f81ead20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80f77fe640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80f806f530 0x7f80f81ead20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80f77fe640 1 -- 172.21.15.78:0/867173750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0094d0 con 0x7f80f80add70 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80f77fe640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 0x7f80f8154bc0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f80ec000c00 tx=0x7f80ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec032880 con 0x7f80f80add70 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ec0329e0 con 0x7f80f80add70 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f8155100 con 0x7f80f80add70 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec032d90 con 0x7f80f80add70 2024-02-19T14:55:33.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.872+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f80c2200 con 0x7f80f80add70 2024-02-19T14:55:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.873+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f80676a0 con 0x7f80f80add70 2024-02-19T14:55:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.873+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f80ec0453f0 con 0x7f80f80add70 2024-02-19T14:55:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.874+0000 7f80f4ff9640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80e406b810 0x7f80e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.874+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f80ec0ac170 con 0x7f80f80add70 2024-02-19T14:55:33.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.874+0000 7f80f7fff640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80e406b810 0x7f80e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:33.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.874+0000 7f80f7fff640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80e406b810 0x7f80e406dcd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f80e8006fd0 tx=0x7f80e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:33.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:33.878+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f80ec076d40 con 0x7f80f80add70 2024-02-19T14:55:34.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.066+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f80f8153200 con 0x7f80f80add70 2024-02-19T14:55:34.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.940+0000 7f80f4ff9640 1 -- 172.21.15.78:0/867173750 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f80ec07a400 con 0x7f80f80add70 2024-02-19T14:55:34.943 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:55:34.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80e406b810 msgr2=0x7f80e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:34.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80e406b810 0x7f80e406dcd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f80e8006fd0 tx=0x7f80e8008040 comp rx=0 tx=0).stop 2024-02-19T14:55:34.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 msgr2=0x7f80f8154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:34.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 0x7f80f8154bc0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f80ec000c00 tx=0x7f80ec0046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:34.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 shutdown_connections 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f80e406b810 0x7f80e406dcd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80f80add70 0x7f80f8154bc0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80f80c2f60 0x7f80f814d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 --2- 172.21.15.78:0/867173750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80f806f530 0x7f80f81ead20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.942+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 >> 172.21.15.78:0/867173750 conn(0x7f80f80c85e0 msgr2=0x7f80f814e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.943+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 shutdown_connections 2024-02-19T14:55:34.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:34.943+0000 7f80fe3d7640 1 -- 172.21.15.78:0/867173750 wait complete. 2024-02-19T14:55:34.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:55:35.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.054+0000 7fba7ccc0640 1 Processor -- start 2024-02-19T14:55:35.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.054+0000 7fba7ccc0640 1 -- start start 2024-02-19T14:55:35.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.055+0000 7fba7ccc0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 0x7fba78160920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.055+0000 7fba7ccc0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba78072e80 con 0x7fba781528d0 2024-02-19T14:55:35.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.055+0000 7fba7ccc0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7806e990 con 0x7fba78072180 2024-02-19T14:55:35.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.055+0000 7fba7ccc0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba780734c0 con 0x7fba78155780 2024-02-19T14:55:35.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.055+0000 7fba7659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fba781528d0 0x7fba780c1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34314/0 (socket says 172.21.15.78:34314) 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.055+0000 7fba7659c640 1 -- 172.21.15.78:0/107887643 learned_addr learned my addr 172.21.15.78:0/107887643 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144616100 0 0) 0x7fba68001110 con 0x7fba781528d0 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba76d9d640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 0x7fba78160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba64003520 con 0x7fba781528d0 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 951471936 0 0) 0x7fba64003520 con 0x7fba781528d0 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 msgr2=0x7fba78160920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 0x7fba78160920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 >> v1:172.21.15.150:6789/0 conn(0x7fba78072180 legacy=0x7fba78155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780731e0 con 0x7fba781528d0 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fba68001800 con 0x7fba781528d0 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 >> v1:172.21.15.78:6789/0 conn(0x7fba781528d0 legacy=0x7fba780c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba640039a0 0x7fba64003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 0x7fba640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004d00 0x7fba64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba780734c0 con 0x7fba640039a0 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6400bab0 con 0x7fba64004d00 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6400bc70 con 0x7fba78155780 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba7659c640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba640039a0 0x7fba64003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba76d9d640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004d00 0x7fba64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba6ffff640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 0x7fba640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba7659c640 1 -- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 msgr2=0x7fba640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba7659c640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba78155780 0x7fba640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba7659c640 1 -- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004d00 msgr2=0x7fba64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:35.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba7659c640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004d00 0x7fba64005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba7659c640 1 -- 172.21.15.78:0/107887643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64003520 con 0x7fba640039a0 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.056+0000 7fba76d9d640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004d00 0x7fba64005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.057+0000 7fba7659c640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba640039a0 0x7fba64003dc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c240 tx=0x7fba640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5066cc49d8424476 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.057+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68005ad0 con 0x7fba640039a0 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.057+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba68005c50 con 0x7fba640039a0 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.057+0000 7fba75d9b640 1 -- 172.21.15.78:0/107887643 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6802c810 con 0x7fba640039a0 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba7ccc0640 1 -- 172.21.15.78:0/107887643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba640039a0 msgr2=0x7fba64003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/107887643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba640039a0 0x7fba64003dc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fba6800c240 tx=0x7fba640057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba76d9d640 1 -- 172.21.15.78:0/107887643 reap_dead start 2024-02-19T14:55:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba7ccc0640 1 -- 172.21.15.78:0/107887643 shutdown_connections 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba7ccc0640 1 -- 172.21.15.78:0/107887643 >> 172.21.15.78:0/107887643 conn(0x7fba780c7590 msgr2=0x7fba780c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba7ccc0640 1 -- 172.21.15.78:0/107887643 shutdown_connections 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.058+0000 7fba7ccc0640 1 -- 172.21.15.78:0/107887643 wait complete. 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.059+0000 7fba7ccc0640 1 Processor -- start 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.059+0000 7fba7ccc0640 1 -- start start 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7ccc0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba640039a0 0x7fba7814ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7ccc0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 0x7fba781edd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7ccc0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba781ee2b0 0x7fba781e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7ccc0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba780a8790 con 0x7fba7814f1c0 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7ccc0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba780c39a0 con 0x7fba781ee2b0 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7ccc0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba781e0910 con 0x7fba640039a0 2024-02-19T14:55:35.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 0x7fba781edd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 0x7fba781edd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51498/0 (socket says 172.21.15.78:51498) 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 -- 172.21.15.78:0/1586275479 learned_addr learned my addr 172.21.15.78:0/1586275479 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba7659c640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba640039a0 0x7fba7814ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba76d9d640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba781ee2b0 0x7fba781e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 -- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba640039a0 msgr2=0x7fba7814ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba640039a0 0x7fba7814ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 -- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba781ee2b0 msgr2=0x7fba781e0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba781ee2b0 0x7fba781e0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 -- 172.21.15.78:0/1586275479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780a7c80 con 0x7fba7814f1c0 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba76d9d640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba781ee2b0 0x7fba781e0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:35.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6ffff640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 0x7fba781edd70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e550 tx=0x7fba6000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.060+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba600052e0 con 0x7fba7814f1c0 2024-02-19T14:55:35.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.061+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60005440 con 0x7fba7814f1c0 2024-02-19T14:55:35.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.061+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba781eebb0 con 0x7fba7814f1c0 2024-02-19T14:55:35.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.061+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60003810 con 0x7fba7814f1c0 2024-02-19T14:55:35.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.061+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba780a8220 con 0x7fba7814f1c0 2024-02-19T14:55:35.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.062+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba78074990 con 0x7fba7814f1c0 2024-02-19T14:55:35.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.062+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fba60003970 con 0x7fba7814f1c0 2024-02-19T14:55:35.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.062+0000 7fba6effd640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba5806b840 0x7fba5806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.063+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fba60089ba0 con 0x7fba7814f1c0 2024-02-19T14:55:35.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.063+0000 7fba7659c640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba5806b840 0x7fba5806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.063+0000 7fba7659c640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba5806b840 0x7fba5806dd00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba68034000 tx=0x7fba68009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.067+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba60054810 con 0x7fba7814f1c0 2024-02-19T14:55:35.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.300+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fba781e0a50 con 0x7fba7814f1c0 2024-02-19T14:55:35.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.301+0000 7fba6effd640 1 -- 172.21.15.78:0/1586275479 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fba60057ed0 con 0x7fba7814f1c0 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:55:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:55:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:55:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:55:35.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:55:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:55:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:55:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:55:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:55:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:55:35.315 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:35.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba5806b840 msgr2=0x7fba5806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba5806b840 0x7fba5806dd00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba68034000 tx=0x7fba68009030 comp rx=0 tx=0).stop 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 msgr2=0x7fba781edd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 0x7fba781edd70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e550 tx=0x7fba6000ea10 comp rx=0 tx=0).stop 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 shutdown_connections 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba5806b840 0x7fba5806dd00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba7814f1c0 0x7fba781edd70 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba781ee2b0 0x7fba781e0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 --2- 172.21.15.78:0/1586275479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba640039a0 0x7fba7814ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 >> 172.21.15.78:0/1586275479 conn(0x7fba780c7590 msgr2=0x7fba78153a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 shutdown_connections 2024-02-19T14:55:35.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.303+0000 7fba7ccc0640 1 -- 172.21.15.78:0/1586275479 wait complete. 2024-02-19T14:55:35.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.415+0000 7f9fb0e4e640 1 Processor -- start 2024-02-19T14:55:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.415+0000 7f9fb0e4e640 1 -- start start 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fb0e4e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9fac0ad890 0x7f9fac0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fb0e4e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fac0ab600 0x7f9fac0c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fb0e4e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fac0aba60 con 0x7f9fac0c49c0 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fb0e4e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fac152d30 con 0x7f9fac0ad890 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fb0e4e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fac0c7860 con 0x7f9fac0ab600 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9faa59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9fac0ad890 0x7f9fac0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fa9d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9fac0c49c0 0x7f9fac1534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34324/0 (socket says 172.21.15.78:34324) 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9fa9d9b640 1 -- 172.21.15.78:0/224811222 learned_addr learned my addr 172.21.15.78:0/224811222 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.416+0000 7f9faad9d640 1 --2- 172.21.15.78:0/224811222 >> v2:172.21.15.150:3300/0 conn(0x7f9fac0ab600 0x7f9fac0c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9faa59c640 1 -- 172.21.15.78:0/224811222 >> v2:172.21.15.150:3300/0 conn(0x7f9fac0ab600 msgr2=0x7f9fac0c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9faa59c640 1 --2- 172.21.15.78:0/224811222 >> v2:172.21.15.150:3300/0 conn(0x7f9fac0ab600 0x7f9fac0c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9faa59c640 1 -- 172.21.15.78:0/224811222 >> v1:172.21.15.78:6789/0 conn(0x7f9fac0c49c0 legacy=0x7f9fac1534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9faa59c640 1 -- 172.21.15.78:0/224811222 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac0c7ce0 con 0x7f9fac0ad890 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9fa959a640 1 -- 172.21.15.78:0/224811222 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333155871 0 0) 0x7f9fac0aba60 con 0x7f9fac0c49c0 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9faa59c640 1 --2- 172.21.15.78:0/224811222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0ad890 0x7f9fac0c64f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f9fac071a80 tx=0x7f9fa402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b378185c55466e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9fa959a640 1 -- 172.21.15.78:0/224811222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa402cb60 con 0x7f9fac0ad890 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9fa959a640 1 -- 172.21.15.78:0/224811222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa402ccc0 con 0x7f9fac0ad890 2024-02-19T14:55:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.417+0000 7f9fa959a640 1 -- 172.21.15.78:0/224811222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4034660 con 0x7f9fac0ad890 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/224811222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0ad890 msgr2=0x7f9fac0c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/224811222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0ad890 0x7f9fac0c64f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f9fac071a80 tx=0x7f9fa402bdb0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/224811222 shutdown_connections 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/224811222 >> v2:172.21.15.150:3300/0 conn(0x7f9fac0ab600 0x7f9fac0c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/224811222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0ad890 0x7f9fac0c64f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/224811222 >> 172.21.15.78:0/224811222 conn(0x7f9fac06b700 msgr2=0x7f9fac0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/224811222 shutdown_connections 2024-02-19T14:55:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.418+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/224811222 wait complete. 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.419+0000 7f9fb0e4e640 1 Processor -- start 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.419+0000 7f9fb0e4e640 1 -- start start 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fb0e4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fac0ab600 0x7f9fac1ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fb0e4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fac0ad890 0x7f9fac1ec480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fb0e4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 0x7f9fac156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fb0e4e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac1dc9f0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fb0e4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac1dc430 con 0x7f9fac0ab600 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fb0e4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac1dc710 con 0x7f9fac0ad890 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 0x7f9fac156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 0x7f9fac156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51512/0 (socket says 172.21.15.78:51512) 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 -- 172.21.15.78:0/2216034007 learned_addr learned my addr 172.21.15.78:0/2216034007 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9fa9d9b640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fac0ad890 0x7f9fac1ec480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 -- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fac0ad890 msgr2=0x7f9fac1ec480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fac0ad890 0x7f9fac1ec480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 -- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fac0ab600 msgr2=0x7f9fac1ebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fac0ab600 0x7f9fac1ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.420+0000 7f9faad9d640 1 -- 172.21.15.78:0/2216034007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4009a70 con 0x7f9fac0c49c0 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.421+0000 7f9faad9d640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 0x7f9fac156460 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00d5d0 tx=0x7f9f9c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.421+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0125f0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.421+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c012770 con 0x7f9fac0c49c0 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.421+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac154bc0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.421+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c012b20 con 0x7f9fac0c49c0 2024-02-19T14:55:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.421+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fac1ec9c0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.422+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac154a20 con 0x7f9fac0c49c0 2024-02-19T14:55:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.422+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9f9c012c80 con 0x7f9fac0c49c0 2024-02-19T14:55:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.422+0000 7f9fa2ffd640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9f8c06b810 0x7f9f8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.422+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9f9c08a4d0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.422+0000 7f9faa59c640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9f8c06b810 0x7f9f8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.423+0000 7f9faa59c640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9f8c06b810 0x7f9f8c06dcd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4036000 tx=0x7f9fa402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.427+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9fac154a20 con 0x7f9fac0c49c0 2024-02-19T14:55:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.601+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fac1550e0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.602+0000 7f9fa2ffd640 1 -- 172.21.15.78:0/2216034007 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13154 (secure 0 0 0) 0x7f9fac1550e0 con 0x7f9fac0c49c0 2024-02-19T14:55:35.604 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:35.604 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":107,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:34.938056+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:35.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9f8c06b810 msgr2=0x7f9f8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9f8c06b810 0x7f9f8c06dcd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4036000 tx=0x7f9fa402e000 comp rx=0 tx=0).stop 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 msgr2=0x7f9fac156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 0x7f9fac156460 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00d5d0 tx=0x7f9f9c00da90 comp rx=0 tx=0).stop 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 shutdown_connections 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9f8c06b810 0x7f9f8c06dcd0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fac0c49c0 0x7f9fac156460 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fac0ad890 0x7f9fac1ec480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 --2- 172.21.15.78:0/2216034007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fac0ab600 0x7f9fac1ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 >> 172.21.15.78:0/2216034007 conn(0x7f9fac06b700 msgr2=0x7f9fac0a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 shutdown_connections 2024-02-19T14:55:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.604+0000 7f9fb0e4e640 1 -- 172.21.15.78:0/2216034007 wait complete. 2024-02-19T14:55:35.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:55:35.618 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:55:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 Processor -- start 2024-02-19T14:55:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 -- start start 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f170406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f170406bae0 0x7f170406f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170406b620 con 0x7f170414d110 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170406ea00 con 0x7f170406bae0 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.718+0000 7f1709b8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1704157230 con 0x7f1704073220 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f170406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1703fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f170406bae0 0x7f170406f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f170406ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:55270/0 (socket says 172.21.15.78:55270) 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 -- 172.21.15.78:0/270606545 learned_addr learned my addr 172.21.15.78:0/270606545 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 -- 172.21.15.78:0/270606545 >> v2:172.21.15.150:3300/0 conn(0x7f170406bae0 msgr2=0x7f170406f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 --2- 172.21.15.78:0/270606545 >> v2:172.21.15.150:3300/0 conn(0x7f170406bae0 0x7f170406f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 -- 172.21.15.78:0/270606545 >> v1:172.21.15.150:6789/0 conn(0x7f170414d110 legacy=0x7f17040ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.719+0000 7f1702ffd640 1 -- 172.21.15.78:0/270606545 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704072180 con 0x7f1704073220 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1702ffd640 1 --2- 172.21.15.78:0/270606545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f170406ee20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009870 tx=0x7f16f802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72363f2ac60300d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f17027fc640 1 -- 172.21.15.78:0/270606545 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8031640 con 0x7f1704073220 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f17027fc640 1 -- 172.21.15.78:0/270606545 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f80317a0 con 0x7f1704073220 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f17027fc640 1 -- 172.21.15.78:0/270606545 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8031b30 con 0x7f1704073220 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 -- 172.21.15.78:0/270606545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 msgr2=0x7f170406ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 --2- 172.21.15.78:0/270606545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f170406ee20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009870 tx=0x7f16f802b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 -- 172.21.15.78:0/270606545 shutdown_connections 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 --2- 172.21.15.78:0/270606545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f170406ee20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 --2- 172.21.15.78:0/270606545 >> v2:172.21.15.150:3300/0 conn(0x7f170406bae0 0x7f170406f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 -- 172.21.15.78:0/270606545 >> 172.21.15.78:0/270606545 conn(0x7f17040c7590 msgr2=0x7f17040c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.720+0000 7f1709b8c640 1 -- 172.21.15.78:0/270606545 shutdown_connections 2024-02-19T14:55:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.721+0000 7f1709b8c640 1 -- 172.21.15.78:0/270606545 wait complete. 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 Processor -- start 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 -- start start 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 0x7f17041470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f17041462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f170414d110 0x7f1704148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1704153bd0 con 0x7f170406bae0 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17040c4a80 con 0x7f170414d110 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.722+0000 7f1709b8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17041ec7d0 con 0x7f1704073220 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 0x7f17041470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 0x7f17041470e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51514/0 (socket says 172.21.15.78:51514) 2024-02-19T14:55:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 -- 172.21.15.78:0/729644895 learned_addr learned my addr 172.21.15.78:0/729644895 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f1702ffd640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f17041462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f1703fff640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f170414d110 0x7f1704148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 -- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 msgr2=0x7f17041462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f17041462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 -- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f170414d110 msgr2=0x7f1704148c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f170414d110 0x7f1704148c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 -- 172.21.15.78:0/729644895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80094d0 con 0x7f170406bae0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f1702ffd640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f17041462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f1703fff640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f170414d110 0x7f1704148c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f17037fe640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 0x7f17041470e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c430 tx=0x7f16f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f40052d0 con 0x7f170406bae0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4005450 con 0x7f170406bae0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4007260 con 0x7f170406bae0 2024-02-19T14:55:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17041eccf0 con 0x7f170406bae0 2024-02-19T14:55:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.723+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17041eca10 con 0x7f170406bae0 2024-02-19T14:55:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.724+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1704074990 con 0x7f170406bae0 2024-02-19T14:55:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.725+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f16f40073c0 con 0x7f170406bae0 2024-02-19T14:55:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.725+0000 7f16ebfff640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16f006b810 0x7f16f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.725+0000 7f1702ffd640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16f006b810 0x7f16f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:35.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.725+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f16f4089360 con 0x7f170406bae0 2024-02-19T14:55:35.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.726+0000 7f1702ffd640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16f006b810 0x7f16f006dcd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f16f8036000 tx=0x7f16f803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:35.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.730+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f16f4053fd0 con 0x7f170406bae0 2024-02-19T14:55:35.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.902+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f170406b7f0 con 0x7f170406bae0 2024-02-19T14:55:35.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.902+0000 7f16ebfff640 1 -- 172.21.15.78:0/729644895 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f16f4057690 con 0x7f170406bae0 2024-02-19T14:55:35.905 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:55:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16f006b810 msgr2=0x7f16f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16f006b810 0x7f16f006dcd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f16f8036000 tx=0x7f16f803c040 comp rx=0 tx=0).stop 2024-02-19T14:55:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 msgr2=0x7f17041470e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 0x7f17041470e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c430 tx=0x7f16f400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 shutdown_connections 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16f006b810 0x7f16f006dcd0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f170414d110 0x7f1704148c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1704073220 0x7f17041462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 --2- 172.21.15.78:0/729644895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f170406bae0 0x7f17041470e0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 >> 172.21.15.78:0/729644895 conn(0x7f17040c7590 msgr2=0x7f17040a9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 shutdown_connections 2024-02-19T14:55:35.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:35.904+0000 7f1709b8c640 1 -- 172.21.15.78:0/729644895 wait complete. 2024-02-19T14:55:35.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:55:36.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.015+0000 7f863bec7640 1 Processor -- start 2024-02-19T14:55:36.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.015+0000 7f863bec7640 1 -- start start 2024-02-19T14:55:36.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f863bec7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86340c42e0 0x7f8634158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f863bec7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86340c2f50 0x7f86340c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f863bec7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86340c34a0 con 0x7f86340c8500 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f863bec7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f863406c090 con 0x7f86340c42e0 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f863bec7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f863406d6a0 con 0x7f86340c2f50 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639c63640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f86340c8500 0x7f8634158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34334/0 (socket says 172.21.15.78:34334) 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639c63640 1 -- 172.21.15.78:0/2855027979 learned_addr learned my addr 172.21.15.78:0/2855027979 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639462640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86340c42e0 0x7f8634158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f863a464640 1 --2- 172.21.15.78:0/2855027979 >> v2:172.21.15.150:3300/0 conn(0x7f86340c2f50 0x7f86340c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639462640 1 -- 172.21.15.78:0/2855027979 >> v2:172.21.15.150:3300/0 conn(0x7f86340c2f50 msgr2=0x7f86340c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639462640 1 --2- 172.21.15.78:0/2855027979 >> v2:172.21.15.150:3300/0 conn(0x7f86340c2f50 0x7f86340c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639462640 1 -- 172.21.15.78:0/2855027979 >> v1:172.21.15.78:6789/0 conn(0x7f86340c8500 legacy=0x7f8634158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8639462640 1 -- 172.21.15.78:0/2855027979 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863406c8e0 con 0x7f86340c42e0 2024-02-19T14:55:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.016+0000 7f8638c61640 1 -- 172.21.15.78:0/2855027979 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069259250 0 0) 0x7f86340c34a0 con 0x7f86340c8500 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.017+0000 7f8639462640 1 --2- 172.21.15.78:0/2855027979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86340c42e0 0x7f8634158d60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f86340adea0 tx=0x7f862402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c5c50a8702f5b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.017+0000 7f8638c61640 1 -- 172.21.15.78:0/2855027979 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862402e070 con 0x7f86340c42e0 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.017+0000 7f8638c61640 1 -- 172.21.15.78:0/2855027979 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86240315a0 con 0x7f86340c42e0 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.017+0000 7f8638c61640 1 -- 172.21.15.78:0/2855027979 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624031930 con 0x7f86340c42e0 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 -- 172.21.15.78:0/2855027979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86340c42e0 msgr2=0x7f8634158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 --2- 172.21.15.78:0/2855027979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86340c42e0 0x7f8634158d60 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f86340adea0 tx=0x7f862402b840 comp rx=0 tx=0).stop 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 -- 172.21.15.78:0/2855027979 shutdown_connections 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 --2- 172.21.15.78:0/2855027979 >> v2:172.21.15.150:3300/0 conn(0x7f86340c2f50 0x7f86340c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 --2- 172.21.15.78:0/2855027979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86340c42e0 0x7f8634158d60 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 -- 172.21.15.78:0/2855027979 >> 172.21.15.78:0/2855027979 conn(0x7f86340aa300 msgr2=0x7f86340aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 -- 172.21.15.78:0/2855027979 shutdown_connections 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.018+0000 7f863bec7640 1 -- 172.21.15.78:0/2855027979 wait complete. 2024-02-19T14:55:36.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.019+0000 7f863bec7640 1 Processor -- start 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.019+0000 7f863bec7640 1 -- start start 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.019+0000 7f863bec7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86340c2f50 0x7f8634156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86340c8500 0x7f8634155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 0x7f86341ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8634158da0 con 0x7f8634159230 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86340c65b0 con 0x7f86340c2f50 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8634156d60 con 0x7f86340c8500 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 0x7f86341ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 0x7f86341ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51528/0 (socket says 172.21.15.78:51528) 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 -- 172.21.15.78:0/3514294757 learned_addr learned my addr 172.21.15.78:0/3514294757 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 -- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86340c8500 msgr2=0x7f8634155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f8639c63640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86340c2f50 0x7f8634156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f8639462640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86340c8500 0x7f8634155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86340c8500 0x7f8634155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 -- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86340c2f50 msgr2=0x7f8634156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86340c2f50 0x7f8634156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 -- 172.21.15.78:0/3514294757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8624009400 con 0x7f8634159230 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f8639c63640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86340c2f50 0x7f8634156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f8639462640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86340c8500 0x7f8634155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863a464640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 0x7f86341ee0b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f862c007e40 tx=0x7f862c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c014620 con 0x7f8634159230 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c00ee90 con 0x7f8634159230 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8634158750 con 0x7f8634159230 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c0149e0 con 0x7f8634159230 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.020+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8634157040 con 0x7f8634159230 2024-02-19T14:55:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.021+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8634071c20 con 0x7f8634159230 2024-02-19T14:55:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.022+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f862c014b40 con 0x7f8634159230 2024-02-19T14:55:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.022+0000 7f862a7fc640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861806b840 0x7f861806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.022+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f862c08a8f0 con 0x7f8634159230 2024-02-19T14:55:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.022+0000 7f8639c63640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861806b840 0x7f861806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.023+0000 7f8639c63640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861806b840 0x7f861806dd00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8630000f60 tx=0x7f8630009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.026+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f862c055560 con 0x7f8634159230 2024-02-19T14:55:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.202+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8634158590 con 0x7f8634159230 2024-02-19T14:55:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.202+0000 7f862a7fc640 1 -- 172.21.15.78:0/3514294757 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7f862c058c20 con 0x7f8634159230 2024-02-19T14:55:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:36.205 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:55:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:55:36.205 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:55:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:55:36.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:55:36.206 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:55:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861806b840 msgr2=0x7f861806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861806b840 0x7f861806dd00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8630000f60 tx=0x7f8630009040 comp rx=0 tx=0).stop 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 msgr2=0x7f86341ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 0x7f86341ee0b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f862c007e40 tx=0x7f862c00e4b0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 shutdown_connections 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f861806b840 0x7f861806dd00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8634159230 0x7f86341ee0b0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86340c8500 0x7f8634155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 --2- 172.21.15.78:0/3514294757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86340c2f50 0x7f8634156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.204+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 >> 172.21.15.78:0/3514294757 conn(0x7f86340aa300 msgr2=0x7f86340c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.205+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 shutdown_connections 2024-02-19T14:55:36.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.205+0000 7f863bec7640 1 -- 172.21.15.78:0/3514294757 wait complete. 2024-02-19T14:55:36.218 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:55:36.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:55:36.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.319+0000 7f5ebd839640 1 Processor -- start 2024-02-19T14:55:36.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.319+0000 7f5ebd839640 1 -- start start 2024-02-19T14:55:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5ebd839640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5eb8152620 0x7f5eb80c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5ebd839640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5eb8155010 0x7f5eb8155460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5ebd839640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8159210 con 0x7f5eb80c6d50 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5ebd839640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb80c1f00 con 0x7f5eb8155010 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5ebd839640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb80c6420 con 0x7f5eb8152620 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5eb8155010 0x7f5eb8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5eb8155010 0x7f5eb8155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51534/0 (socket says 172.21.15.78:51534) 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb67fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5eb80c6d50 0x7f5eb806d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34342/0 (socket says 172.21.15.78:34342) 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 -- 172.21.15.78:0/1010201399 learned_addr learned my addr 172.21.15.78:0/1010201399 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb6ffd640 1 --2- 172.21.15.78:0/1010201399 >> v2:172.21.15.150:3300/0 conn(0x7f5eb8152620 0x7f5eb80c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 -- 172.21.15.78:0/1010201399 >> v2:172.21.15.150:3300/0 conn(0x7f5eb8152620 msgr2=0x7f5eb80c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 --2- 172.21.15.78:0/1010201399 >> v2:172.21.15.150:3300/0 conn(0x7f5eb8152620 0x7f5eb80c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 -- 172.21.15.78:0/1010201399 >> v1:172.21.15.78:6789/0 conn(0x7f5eb80c6d50 legacy=0x7f5eb806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb5ffb640 1 -- 172.21.15.78:0/1010201399 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943107350 0 0) 0x7f5eb8159210 con 0x7f5eb80c6d50 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb77fe640 1 -- 172.21.15.78:0/1010201399 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8158600 con 0x7f5eb8155010 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.320+0000 7f5eb6ffd640 1 --2- 172.21.15.78:0/1010201399 >> v2:172.21.15.150:3300/0 conn(0x7f5eb8152620 0x7f5eb80c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.321+0000 7f5eb77fe640 1 --2- 172.21.15.78:0/1010201399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb8155010 0x7f5eb8155460 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5eb808c5b0 tx=0x7f5eb002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cea62bd4c4f941 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.321+0000 7f5eb5ffb640 1 -- 172.21.15.78:0/1010201399 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb00317e0 con 0x7f5eb8155010 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.321+0000 7f5eb5ffb640 1 -- 172.21.15.78:0/1010201399 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb0031940 con 0x7f5eb8155010 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.321+0000 7f5eb5ffb640 1 -- 172.21.15.78:0/1010201399 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0031cd0 con 0x7f5eb8155010 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 -- 172.21.15.78:0/1010201399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb8155010 msgr2=0x7f5eb8155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 --2- 172.21.15.78:0/1010201399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb8155010 0x7f5eb8155460 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5eb808c5b0 tx=0x7f5eb002b880 comp rx=0 tx=0).stop 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 -- 172.21.15.78:0/1010201399 shutdown_connections 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 --2- 172.21.15.78:0/1010201399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb8155010 0x7f5eb8155460 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 --2- 172.21.15.78:0/1010201399 >> v2:172.21.15.150:3300/0 conn(0x7f5eb8152620 0x7f5eb80c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 -- 172.21.15.78:0/1010201399 >> 172.21.15.78:0/1010201399 conn(0x7f5eb80caae0 msgr2=0x7f5eb80caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 -- 172.21.15.78:0/1010201399 shutdown_connections 2024-02-19T14:55:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.322+0000 7f5ebd839640 1 -- 172.21.15.78:0/1010201399 wait complete. 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.323+0000 7f5ebd839640 1 Processor -- start 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.323+0000 7f5ebd839640 1 -- start start 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 0x7f5eb80c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5eb8152620 0x7f5eb8150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5eb8155010 0x7f5eb81561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb814cf50 con 0x7f5eb80c6d50 2024-02-19T14:55:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8158220 con 0x7f5eb8155010 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb814cbe0 con 0x7f5eb8152620 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 0x7f5eb80c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 0x7f5eb80c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51550/0 (socket says 172.21.15.78:51550) 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 -- 172.21.15.78:0/2661137008 learned_addr learned my addr 172.21.15.78:0/2661137008 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 -- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5eb8152620 msgr2=0x7f5eb8150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5eb8152620 0x7f5eb8150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 -- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5eb8155010 msgr2=0x7f5eb81561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb77fe640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5eb8155010 0x7f5eb81561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5eb8155010 0x7f5eb81561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 -- 172.21.15.78:0/2661137008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00094d0 con 0x7f5eb80c6d50 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5eb6ffd640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 0x7f5eb80c8850 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800cf60 tx=0x7f5ea800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea800d840 con 0x7f5eb80c6d50 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea800d9a0 con 0x7f5eb80c6d50 2024-02-19T14:55:36.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb8150aa0 con 0x7f5eb80c6d50 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea800dd30 con 0x7f5eb80c6d50 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.324+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb80afc90 con 0x7f5eb80c6d50 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.325+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb814ca40 con 0x7f5eb80c6d50 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.326+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5ea8015440 con 0x7f5eb80c6d50 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.326+0000 7f5e9ffff640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ea406b810 0x7f5ea406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.326+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5ea808a980 con 0x7f5eb80c6d50 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.326+0000 7f5eb67fc640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ea406b810 0x7f5ea406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:36.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.327+0000 7f5eb67fc640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ea406b810 0x7f5ea406dcd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5eac007fd0 tx=0x7f5eac007960 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:36.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.331+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5eb814ca40 con 0x7f5eb80c6d50 2024-02-19T14:55:36.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.502+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5eb814cda0 con 0x7f5eb80c6d50 2024-02-19T14:55:36.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.945+0000 7f5e9ffff640 1 -- 172.21.15.78:0/2661137008 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5ea801a020 con 0x7f5eb80c6d50 2024-02-19T14:55:36.948 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ea406b810 msgr2=0x7f5ea406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ea406b810 0x7f5ea406dcd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5eac007fd0 tx=0x7f5eac007960 comp rx=0 tx=0).stop 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 msgr2=0x7f5eb80c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 0x7f5eb80c8850 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800cf60 tx=0x7f5ea800cf90 comp rx=0 tx=0).stop 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 shutdown_connections 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ea406b810 0x7f5ea406dcd0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5eb8155010 0x7f5eb81561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5eb8152620 0x7f5eb8150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 --2- 172.21.15.78:0/2661137008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5eb80c6d50 0x7f5eb80c8850 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 >> 172.21.15.78:0/2661137008 conn(0x7f5eb80caae0 msgr2=0x7f5eb806d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 shutdown_connections 2024-02-19T14:55:36.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:36.948+0000 7f5ebd839640 1 -- 172.21.15.78:0/2661137008 wait complete. 2024-02-19T14:55:36.965 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:55:37.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.070+0000 7fcf21839640 1 Processor -- start 2024-02-19T14:55:37.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.070+0000 7fcf21839640 1 -- start start 2024-02-19T14:55:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf21839640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c155e90 0x7fcf1c0c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf21839640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcf1c0c53f0 0x7fcf1c06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf21839640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0c7850 con 0x7fcf1c06dc80 2024-02-19T14:55:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf21839640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c071580 con 0x7fcf1c0c53f0 2024-02-19T14:55:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf21839640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0c6c80 con 0x7fcf1c155e90 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcf1c0c53f0 0x7fcf1c06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcf1c06dc80 0x7fcf1c0c4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34350/0 (socket says 172.21.15.78:34350) 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1a7fc640 1 -- 172.21.15.78:0/2389587885 learned_addr learned my addr 172.21.15.78:0/2389587885 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1b7fe640 1 --2- 172.21.15.78:0/2389587885 >> v2:172.21.15.78:3300/0 conn(0x7fcf1c0c53f0 0x7fcf1c06c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51554/0 (socket says 172.21.15.78:51554) 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1b7fe640 1 -- 172.21.15.78:0/2389587885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c155e90 msgr2=0x7fcf1c0c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1b7fe640 1 --2- 172.21.15.78:0/2389587885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c155e90 0x7fcf1c0c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.071+0000 7fcf1b7fe640 1 -- 172.21.15.78:0/2389587885 >> v1:172.21.15.78:6789/0 conn(0x7fcf1c06dc80 legacy=0x7fcf1c0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.072+0000 7fcf19ffb640 1 -- 172.21.15.78:0/2389587885 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877606819 0 0) 0x7fcf1c0c7850 con 0x7fcf1c06dc80 2024-02-19T14:55:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.072+0000 7fcf1b7fe640 1 -- 172.21.15.78:0/2389587885 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0c7ce0 con 0x7fcf1c0c53f0 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.072+0000 7fcf1b7fe640 1 --2- 172.21.15.78:0/2389587885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c0c53f0 0x7fcf1c06c7d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcf140097a0 tx=0x7fcf1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84d3eaacd696f43 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.072+0000 7fcf19ffb640 1 -- 172.21.15.78:0/2389587885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14031830 con 0x7fcf1c0c53f0 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.072+0000 7fcf19ffb640 1 -- 172.21.15.78:0/2389587885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf14031990 con 0x7fcf1c0c53f0 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.072+0000 7fcf19ffb640 1 -- 172.21.15.78:0/2389587885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14031d20 con 0x7fcf1c0c53f0 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 -- 172.21.15.78:0/2389587885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c0c53f0 msgr2=0x7fcf1c06c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 --2- 172.21.15.78:0/2389587885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c0c53f0 0x7fcf1c06c7d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcf140097a0 tx=0x7fcf1402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 -- 172.21.15.78:0/2389587885 shutdown_connections 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 --2- 172.21.15.78:0/2389587885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c0c53f0 0x7fcf1c06c7d0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 --2- 172.21.15.78:0/2389587885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c155e90 0x7fcf1c0c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 -- 172.21.15.78:0/2389587885 >> 172.21.15.78:0/2389587885 conn(0x7fcf1c0aa300 msgr2=0x7fcf1c0aa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 -- 172.21.15.78:0/2389587885 shutdown_connections 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.073+0000 7fcf21839640 1 -- 172.21.15.78:0/2389587885 wait complete. 2024-02-19T14:55:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.074+0000 7fcf21839640 1 Processor -- start 2024-02-19T14:55:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.074+0000 7fcf21839640 1 -- start start 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf21839640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 0x7fcf1c1edbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf21839640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c0c53f0 0x7fcf1c158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf21839640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c155e90 0x7fcf1c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf21839640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c154340 con 0x7fcf1c06dc80 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf21839640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c14ca70 con 0x7fcf1c155e90 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf21839640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1574c0 con 0x7fcf1c0c53f0 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 0x7fcf1c1edbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 0x7fcf1c1edbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51566/0 (socket says 172.21.15.78:51566) 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 -- 172.21.15.78:0/1008320354 learned_addr learned my addr 172.21.15.78:0/1008320354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1a7fc640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c0c53f0 0x7fcf1c158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1b7fe640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c155e90 0x7fcf1c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 -- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c0c53f0 msgr2=0x7fcf1c158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c0c53f0 0x7fcf1c158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 -- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c155e90 msgr2=0x7fcf1c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c155e90 0x7fcf1c158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 -- 172.21.15.78:0/1008320354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf14009400 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.075+0000 7fcf1affd640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 0x7fcf1c1edbd0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00ba80 tx=0x7fcf0c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.076+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c00cc00 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.076+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c00cd80 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.076+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c14c1d0 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.076+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c014680 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.076+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c14c790 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.077+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c074990 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.077+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcf0c01d3b0 con 0x7fcf1c06dc80 2024-02-19T14:55:37.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.077+0000 7fcf03fff640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf0806b810 0x7fcf0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.077+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fcf0c08a080 con 0x7fcf1c06dc80 2024-02-19T14:55:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.078+0000 7fcf1a7fc640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf0806b810 0x7fcf0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.078+0000 7fcf1a7fc640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf0806b810 0x7fcf0806dcd0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0aa200 tx=0x7fcf10007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:37.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.082+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf0c054b80 con 0x7fcf1c06dc80 2024-02-19T14:55:37.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.252+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcf1c0c69c0 con 0x7fcf1c06dc80 2024-02-19T14:55:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.951+0000 7fcf03fff640 1 -- 172.21.15.78:0/1008320354 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcf0c0190d0 con 0x7fcf1c06dc80 2024-02-19T14:55:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:55:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf0806b810 msgr2=0x7fcf0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf0806b810 0x7fcf0806dcd0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0aa200 tx=0x7fcf10007b50 comp rx=0 tx=0).stop 2024-02-19T14:55:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 msgr2=0x7fcf1c1edbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:37.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 0x7fcf1c1edbd0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00ba80 tx=0x7fcf0c00bf40 comp rx=0 tx=0).stop 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 shutdown_connections 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcf0806b810 0x7fcf0806dcd0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c155e90 0x7fcf1c158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf1c0c53f0 0x7fcf1c158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 --2- 172.21.15.78:0/1008320354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf1c06dc80 0x7fcf1c1edbd0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.954+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 >> 172.21.15.78:0/1008320354 conn(0x7fcf1c0aa300 msgr2=0x7fcf1c0c3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.955+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 shutdown_connections 2024-02-19T14:55:37.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:37.956+0000 7fcf21839640 1 -- 172.21.15.78:0/1008320354 wait complete. 2024-02-19T14:55:37.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:55:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.087+0000 7f130a688640 1 Processor -- start 2024-02-19T14:55:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.087+0000 7f130a688640 1 -- start start 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f130a688640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1304155f90 0x7f130406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f130a688640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13040c6d30 0x7f13040c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f130a688640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13040c6af0 con 0x7f13040c3aa0 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f130a688640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040c7940 con 0x7f13040c6d30 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f130a688640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040c7c20 con 0x7f1304155f90 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f13037fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13040c6d30 0x7f13040c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f1308c25640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f13040c3aa0 0x7f13041588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60580/0 (socket says 172.21.15.78:60580) 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f1308c25640 1 -- 172.21.15.78:0/1052424460 learned_addr learned my addr 172.21.15.78:0/1052424460 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f13037fe640 1 -- 172.21.15.78:0/1052424460 >> v2:172.21.15.150:3300/0 conn(0x7f1304155f90 msgr2=0x7f130406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f1303fff640 1 --2- 172.21.15.78:0/1052424460 >> v2:172.21.15.150:3300/0 conn(0x7f1304155f90 0x7f130406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f13037fe640 1 --2- 172.21.15.78:0/1052424460 >> v2:172.21.15.150:3300/0 conn(0x7f1304155f90 0x7f130406d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f1302ffd640 1 -- 172.21.15.78:0/1052424460 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521661074 0 0) 0x7f13040c6af0 con 0x7f13040c3aa0 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f13037fe640 1 -- 172.21.15.78:0/1052424460 >> v1:172.21.15.78:6789/0 conn(0x7f13040c3aa0 legacy=0x7f13041588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:38.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.088+0000 7f13037fe640 1 -- 172.21.15.78:0/1052424460 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040c7f00 con 0x7f13040c6d30 2024-02-19T14:55:38.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.089+0000 7f13037fe640 1 --2- 172.21.15.78:0/1052424460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13040c6d30 0x7f13040c3560 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f13040c2a40 tx=0x7f12f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc63a28430fa3a5b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.089+0000 7f1302ffd640 1 -- 172.21.15.78:0/1052424460 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f802e070 con 0x7f13040c6d30 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.089+0000 7f1302ffd640 1 -- 172.21.15.78:0/1052424460 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f80315a0 con 0x7f13040c6d30 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.089+0000 7f1302ffd640 1 -- 172.21.15.78:0/1052424460 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8031930 con 0x7f13040c6d30 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 -- 172.21.15.78:0/1052424460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13040c6d30 msgr2=0x7f13040c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 --2- 172.21.15.78:0/1052424460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13040c6d30 0x7f13040c3560 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f13040c2a40 tx=0x7f12f802b840 comp rx=0 tx=0).stop 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 -- 172.21.15.78:0/1052424460 shutdown_connections 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 --2- 172.21.15.78:0/1052424460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13040c6d30 0x7f13040c3560 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 --2- 172.21.15.78:0/1052424460 >> v2:172.21.15.150:3300/0 conn(0x7f1304155f90 0x7f130406d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 -- 172.21.15.78:0/1052424460 >> 172.21.15.78:0/1052424460 conn(0x7f130406b700 msgr2=0x7f13040aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 -- 172.21.15.78:0/1052424460 shutdown_connections 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.090+0000 7f130a688640 1 -- 172.21.15.78:0/1052424460 wait complete. 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.091+0000 7f130a688640 1 Processor -- start 2024-02-19T14:55:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.091+0000 7f130a688640 1 -- start start 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f130a688640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13040c3aa0 0x7f13041e1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f130a688640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13040c6d30 0x7f13041dfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f130a688640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 0x7f13041e0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f130a688640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13041e0860 con 0x7f1304155f90 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f130a688640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040ad9b0 con 0x7f13040c6d30 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f130a688640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040a74a0 con 0x7f13040c3aa0 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 0x7f13041e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 0x7f13041e0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40354/0 (socket says 172.21.15.78:40354) 2024-02-19T14:55:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 -- 172.21.15.78:0/1970139923 learned_addr learned my addr 172.21.15.78:0/1970139923 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f13037fe640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13040c6d30 0x7f13041dfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1303fff640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13040c3aa0 0x7f13041e1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 -- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13040c3aa0 msgr2=0x7f13041e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13040c3aa0 0x7f13041e1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 -- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13040c6d30 msgr2=0x7f13041dfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13040c6d30 0x7f13041dfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 -- 172.21.15.78:0/1970139923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8009400 con 0x7f1304155f90 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1308c25640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 0x7f13041e0320 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00c7a0 tx=0x7f12fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.092+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00a640 con 0x7f1304155f90 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.093+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc00a7a0 con 0x7f1304155f90 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.093+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00abd0 con 0x7f1304155f90 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.093+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13040a7780 con 0x7f1304155f90 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.093+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13041df8c0 con 0x7f1304155f90 2024-02-19T14:55:38.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.094+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13041dfab0 con 0x7f1304155f90 2024-02-19T14:55:38.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.094+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f12fc00ad30 con 0x7f1304155f90 2024-02-19T14:55:38.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.094+0000 7f1300ff9640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12f006b810 0x7f12f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:38.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.095+0000 7f1303fff640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12f006b810 0x7f12f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:38.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.095+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f12fc089a20 con 0x7f1304155f90 2024-02-19T14:55:38.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.095+0000 7f1303fff640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12f006b810 0x7f12f006dcd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f13041edd90 tx=0x7f12f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.099+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f12fc054450 con 0x7f1304155f90 2024-02-19T14:55:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.310+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1304074990 con 0x7f1304155f90 2024-02-19T14:55:38.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.964+0000 7f1300ff9640 1 -- 172.21.15.78:0/1970139923 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v82) v1 ==== 174+0+0 (secure 0 0 0) 0x7f12fc018020 con 0x7f1304155f90 2024-02-19T14:55:38.967 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 17 and data pool 18 2024-02-19T14:55:38.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.973+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12f006b810 msgr2=0x7f12f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:38.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.973+0000 7f130a688640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12f006b810 0x7f12f006dcd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f13041edd90 tx=0x7f12f4008040 comp rx=0 tx=0).stop 2024-02-19T14:55:38.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.973+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 msgr2=0x7f13041e0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:38.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.973+0000 7f130a688640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 0x7f13041e0320 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00c7a0 tx=0x7f12fc00cc60 comp rx=0 tx=0).stop 2024-02-19T14:55:38.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.975+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 shutdown_connections 2024-02-19T14:55:38.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.975+0000 7f130a688640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12f006b810 0x7f12f006dcd0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.975+0000 7f130a688640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1304155f90 0x7f13041e0320 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.975+0000 7f130a688640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13040c6d30 0x7f13041dfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.975+0000 7f130a688640 1 --2- 172.21.15.78:0/1970139923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13040c3aa0 0x7f13041e1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:38.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.975+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 >> 172.21.15.78:0/1970139923 conn(0x7f130406b700 msgr2=0x7f1304155220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:38.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.977+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 shutdown_connections 2024-02-19T14:55:38.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:38.977+0000 7f130a688640 1 -- 172.21.15.78:0/1970139923 wait complete. 2024-02-19T14:55:38.998 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:39.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.096+0000 7f4c4e75e640 1 Processor -- start 2024-02-19T14:55:39.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.096+0000 7f4c4e75e640 1 -- start start 2024-02-19T14:55:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c4e75e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4c4814ef50 0x7f4c4814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c4e75e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c4e75e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c480c35c0 con 0x7f4c480c78a0 2024-02-19T14:55:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c4e75e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c480c4bc0 con 0x7f4c4814ef50 2024-02-19T14:55:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c4e75e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4814f920 con 0x7f4c480727d0 2024-02-19T14:55:39.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c47fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4c480c78a0 0x7f4c480c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60594/0 (socket says 172.21.15.78:60594) 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c47fff640 1 -- 172.21.15.78:0/3302763035 learned_addr learned my addr 172.21.15.78:0/3302763035 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c477fe640 1 --2- 172.21.15.78:0/3302763035 >> v2:172.21.15.78:3300/0 conn(0x7f4c4814ef50 0x7f4c4814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c4ccfb640 1 --2- 172.21.15.78:0/3302763035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c46ffd640 1 -- 172.21.15.78:0/3302763035 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685079087 0 0) 0x7f4c480c35c0 con 0x7f4c480c78a0 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.097+0000 7f4c46ffd640 1 -- 172.21.15.78:0/3302763035 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c30005730 con 0x7f4c480c78a0 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c477fe640 1 -- 172.21.15.78:0/3302763035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 msgr2=0x7f4c48072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c477fe640 1 --2- 172.21.15.78:0/3302763035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c477fe640 1 -- 172.21.15.78:0/3302763035 >> v1:172.21.15.78:6789/0 conn(0x7f4c480c78a0 legacy=0x7f4c480c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c477fe640 1 -- 172.21.15.78:0/3302763035 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c480c38a0 con 0x7f4c4814ef50 2024-02-19T14:55:39.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c477fe640 1 --2- 172.21.15.78:0/3302763035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c4814f3e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c0097d0 tx=0x7f4c3c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ce093b3249ddc2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c46ffd640 1 -- 172.21.15.78:0/3302763035 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c031800 con 0x7f4c4814ef50 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c46ffd640 1 -- 172.21.15.78:0/3302763035 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c031960 con 0x7f4c4814ef50 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.098+0000 7f4c46ffd640 1 -- 172.21.15.78:0/3302763035 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c031cf0 con 0x7f4c4814ef50 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3302763035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 msgr2=0x7f4c4814f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3302763035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c4814f3e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c0097d0 tx=0x7f4c3c02b8a0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3302763035 shutdown_connections 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3302763035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c4814f3e0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3302763035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3302763035 >> 172.21.15.78:0/3302763035 conn(0x7f4c48066e70 msgr2=0x7f4c480a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3302763035 shutdown_connections 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.099+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3302763035 wait complete. 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.100+0000 7f4c4e75e640 1 Processor -- start 2024-02-19T14:55:39.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.100+0000 7f4c4e75e640 1 -- start start 2024-02-19T14:55:39.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4e75e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4e75e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c480c78a0 0x7f4c481e6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4e75e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c481e6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4e75e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4814bbb0 con 0x7f4c4814ef50 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4e75e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c481dd660 con 0x7f4c480c78a0 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4e75e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c481dd940 con 0x7f4c480727d0 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c481e6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c481e6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40360/0 (socket says 172.21.15.78:40360) 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 -- 172.21.15.78:0/3615954784 learned_addr learned my addr 172.21.15.78:0/3615954784 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c47fff640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c477fe640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c480c78a0 0x7f4c481e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 -- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 msgr2=0x7f4c48150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 -- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c480c78a0 msgr2=0x7f4c481e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c480c78a0 0x7f4c481e6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 -- 172.21.15.78:0/3615954784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c009400 con 0x7f4c4814ef50 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c47fff640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:39.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.101+0000 7f4c4ccfb640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c481e6ea0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f4c40007e40 tx=0x7f4c4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.102+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40014620 con 0x7f4c4814ef50 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.102+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4000ee90 con 0x7f4c4814ef50 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.102+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c400149e0 con 0x7f4c4814ef50 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.102+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c481ddc20 con 0x7f4c4814ef50 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.102+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c481512a0 con 0x7f4c4814ef50 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.102+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c480c3270 con 0x7f4c4814ef50 2024-02-19T14:55:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.103+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4c40014b40 con 0x7f4c4814ef50 2024-02-19T14:55:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.103+0000 7f4c44ff9640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c3406b840 0x7f4c3406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.103+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c4008ac50 con 0x7f4c4814ef50 2024-02-19T14:55:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.104+0000 7f4c47fff640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c3406b840 0x7f4c3406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.104+0000 7f4c47fff640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c3406b840 0x7f4c3406dd00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4c38007fd0 tx=0x7f4c38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.108+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4c480c3270 con 0x7f4c4814ef50 2024-02-19T14:55:39.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.280+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c480c3270 con 0x7f4c4814ef50 2024-02-19T14:55:39.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.281+0000 7f4c44ff9640 1 -- 172.21.15.78:0/3615954784 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16953 (secure 0 0 0) 0x7f4c480c3270 con 0x7f4c4814ef50 2024-02-19T14:55:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:39.284 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:38.311477+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:36.521408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:37.271805+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c3406b840 msgr2=0x7f4c3406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c3406b840 0x7f4c3406dd00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4c38007fd0 tx=0x7f4c38009040 comp rx=0 tx=0).stop 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 msgr2=0x7f4c481e6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c481e6ea0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f4c40007e40 tx=0x7f4c4000e4b0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 shutdown_connections 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c3406b840 0x7f4c3406dd00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c4814ef50 0x7f4c481e6ea0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c480c78a0 0x7f4c481e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 --2- 172.21.15.78:0/3615954784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c480727d0 0x7f4c48150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 >> 172.21.15.78:0/3615954784 conn(0x7f4c48066e70 msgr2=0x7f4c4814ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:39.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 shutdown_connections 2024-02-19T14:55:39.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.283+0000 7f4c4e75e640 1 -- 172.21.15.78:0/3615954784 wait complete. 2024-02-19T14:55:39.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:39.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.394+0000 7f0308209640 1 Processor -- start 2024-02-19T14:55:39.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.394+0000 7f0308209640 1 -- start start 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0308209640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03000c6d30 0x7f0300158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0308209640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000c6af0 con 0x7f0300155f90 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0308209640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000c7940 con 0x7f030006d550 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0308209640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03000c7c20 con 0x7f03000c6d30 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0305fa5640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0300155f90 0x7f03000c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60606/0 (socket says 172.21.15.78:60606) 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0305fa5640 1 -- 172.21.15.78:0/1343683748 learned_addr learned my addr 172.21.15.78:0/1343683748 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f03067a6640 1 --2- 172.21.15.78:0/1343683748 >> v2:172.21.15.150:3300/0 conn(0x7f03000c6d30 0x7f0300158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0304fa3640 1 -- 172.21.15.78:0/1343683748 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441882402 0 0) 0x7f03000c6af0 con 0x7f0300155f90 2024-02-19T14:55:39.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f0304fa3640 1 -- 172.21.15.78:0/1343683748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec003520 con 0x7f0300155f90 2024-02-19T14:55:39.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.395+0000 7f03067a6640 1 -- 172.21.15.78:0/1343683748 >> v1:172.21.15.150:6789/0 conn(0x7f030006d550 legacy=0x7f03001588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.396+0000 7f03067a6640 1 -- 172.21.15.78:0/1343683748 >> v1:172.21.15.78:6789/0 conn(0x7f0300155f90 legacy=0x7f03000c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.396+0000 7f0304fa3640 1 -- 172.21.15.78:0/1343683748 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 508907677 0 0) 0x7f02ec003520 con 0x7f0300155f90 2024-02-19T14:55:39.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.396+0000 7f03067a6640 1 -- 172.21.15.78:0/1343683748 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000c7f00 con 0x7f03000c6d30 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.396+0000 7f03067a6640 1 --2- 172.21.15.78:0/1343683748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f0300158fb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009d90 tx=0x7f02f802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce2dd25d8be80b81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0304fa3640 1 -- 172.21.15.78:0/1343683748 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc001670 con 0x7f03000c6d30 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0304fa3640 1 -- 172.21.15.78:0/1343683748 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f802c8b0 con 0x7f03000c6d30 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0304fa3640 1 -- 172.21.15.78:0/1343683748 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f802cc20 con 0x7f03000c6d30 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 -- 172.21.15.78:0/1343683748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 msgr2=0x7f0300158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 --2- 172.21.15.78:0/1343683748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f0300158fb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009d90 tx=0x7f02f802bd80 comp rx=0 tx=0).stop 2024-02-19T14:55:39.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 -- 172.21.15.78:0/1343683748 shutdown_connections 2024-02-19T14:55:39.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 --2- 172.21.15.78:0/1343683748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f0300158fb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 -- 172.21.15.78:0/1343683748 >> 172.21.15.78:0/1343683748 conn(0x7f030006b700 msgr2=0x7f03000aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:39.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 -- 172.21.15.78:0/1343683748 shutdown_connections 2024-02-19T14:55:39.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.397+0000 7f0308209640 1 -- 172.21.15.78:0/1343683748 wait complete. 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.398+0000 7f0308209640 1 Processor -- start 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.398+0000 7f0308209640 1 -- start start 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0308209640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f030006d550 0x7f03000a8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0308209640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f03000a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0308209640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 0x7f0300152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0308209640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001edc40 con 0x7f0300155f90 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0308209640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001edf20 con 0x7f03000c6d30 2024-02-19T14:55:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0308209640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03001ee200 con 0x7f030006d550 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 0x7f0300152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 0x7f0300152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40372/0 (socket says 172.21.15.78:40372) 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 -- 172.21.15.78:0/969836415 learned_addr learned my addr 172.21.15.78:0/969836415 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03057a4640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f03000a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0305fa5640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f030006d550 0x7f03000a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 -- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f030006d550 msgr2=0x7f03000a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f030006d550 0x7f03000a8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 -- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 msgr2=0x7f03000a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f03000a7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 -- 172.21.15.78:0/969836415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f8009a70 con 0x7f0300155f90 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03057a4640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f03000a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f0305fa5640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f030006d550 0x7f03000a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.399+0000 7f03067a6640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 0x7f0300152810 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009d90 tx=0x7f02f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.400+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8004450 con 0x7f0300155f90 2024-02-19T14:55:39.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.400+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f80045d0 con 0x7f0300155f90 2024-02-19T14:55:39.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.400+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03001ee4e0 con 0x7f0300155f90 2024-02-19T14:55:39.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.400+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8009450 con 0x7f0300155f90 2024-02-19T14:55:39.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.400+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03000a8e70 con 0x7f0300155f90 2024-02-19T14:55:39.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.401+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0300074990 con 0x7f0300155f90 2024-02-19T14:55:39.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.401+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f02f803b050 con 0x7f0300155f90 2024-02-19T14:55:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.402+0000 7f02f67fc640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f02e406b810 0x7f02e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.402+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02f80ac500 con 0x7f0300155f90 2024-02-19T14:55:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.402+0000 7f0305fa5640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f02e406b810 0x7f02e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.402+0000 7f0305fa5640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f02e406b810 0x7f02e406dcd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005880 tx=0x7f02fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.406+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f02f8076f40 con 0x7f0300155f90 2024-02-19T14:55:39.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.579+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03000c75a0 con 0x7f0300155f90 2024-02-19T14:55:39.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.580+0000 7f02f67fc640 1 -- 172.21.15.78:0/969836415 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16953 (secure 0 0 0) 0x7f02f807a600 con 0x7f0300155f90 2024-02-19T14:55:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:39.583 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:38.311477+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:36.521408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:37.271805+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:39.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f02e406b810 msgr2=0x7f02e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f02e406b810 0x7f02e406dcd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f02fc005880 tx=0x7f02fc009040 comp rx=0 tx=0).stop 2024-02-19T14:55:39.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 msgr2=0x7f0300152810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 0x7f0300152810 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009d90 tx=0x7f02f8004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:39.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 shutdown_connections 2024-02-19T14:55:39.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f02e406b810 0x7f02e406dcd0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0300155f90 0x7f0300152810 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03000c6d30 0x7f03000a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 --2- 172.21.15.78:0/969836415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f030006d550 0x7f03000a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 >> 172.21.15.78:0/969836415 conn(0x7f030006b700 msgr2=0x7f0300154730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:39.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 shutdown_connections 2024-02-19T14:55:39.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.582+0000 7f0308209640 1 -- 172.21.15.78:0/969836415 wait complete. 2024-02-19T14:55:39.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:55:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.696+0000 7f724cd81640 1 Processor -- start 2024-02-19T14:55:39.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.696+0000 7f724cd81640 1 -- start start 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724cd81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724cd81640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72480c49c0 0x7f72480ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724cd81640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7248152d30 con 0x7f72481534d0 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724cd81640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72480c7860 con 0x7f72480c49c0 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724cd81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72480c7ce0 con 0x7f72480ad890 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f723ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72480c49c0 0x7f72480ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f7246d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f72481534d0 0x7f72480c5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42070/0 (socket says 172.21.15.78:42070) 2024-02-19T14:55:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f7246d9d640 1 -- 172.21.15.78:0/2812654229 learned_addr learned my addr 172.21.15.78:0/2812654229 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:43984/0 (socket says 172.21.15.78:43984) 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f7245d9b640 1 -- 172.21.15.78:0/2812654229 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10161278 0 0) 0x7f7248152d30 con 0x7f72481534d0 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f7245d9b640 1 -- 172.21.15.78:0/2812654229 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234003520 con 0x7f72481534d0 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f723ffff640 1 -- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 msgr2=0x7f72480c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f723ffff640 1 --2- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f723ffff640 1 -- 172.21.15.78:0/2812654229 >> v1:172.21.15.150:6789/0 conn(0x7f72481534d0 legacy=0x7f72480c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f723ffff640 1 -- 172.21.15.78:0/2812654229 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480c7f80 con 0x7f72480c49c0 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.697+0000 7f724659c640 1 --2- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.698+0000 7f723ffff640 1 --2- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72480ab600 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72300097a0 tx=0x7f723002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad53e20b06edb729 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.698+0000 7f7245d9b640 1 -- 172.21.15.78:0/2812654229 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723002f050 con 0x7f72480c49c0 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.698+0000 7f7245d9b640 1 -- 172.21.15.78:0/2812654229 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230031700 con 0x7f72480c49c0 2024-02-19T14:55:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.698+0000 7f7245d9b640 1 -- 172.21.15.78:0/2812654229 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7230031a90 con 0x7f72480c49c0 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 -- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 msgr2=0x7f72480ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 --2- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72480ab600 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72300097a0 tx=0x7f723002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 -- 172.21.15.78:0/2812654229 shutdown_connections 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 --2- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72480ab600 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 --2- 172.21.15.78:0/2812654229 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 -- 172.21.15.78:0/2812654229 >> 172.21.15.78:0/2812654229 conn(0x7f724806b700 msgr2=0x7f72480aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 -- 172.21.15.78:0/2812654229 shutdown_connections 2024-02-19T14:55:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.699+0000 7f724cd81640 1 -- 172.21.15.78:0/2812654229 wait complete. 2024-02-19T14:55:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.700+0000 7f724cd81640 1 Processor -- start 2024-02-19T14:55:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.700+0000 7f724cd81640 1 -- start start 2024-02-19T14:55:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724cd81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480a9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724cd81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72481e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724cd81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 0x7f72481e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724cd81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72480bdd80 con 0x7f72481534d0 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724cd81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248151560 con 0x7f72480c49c0 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724cd81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72481e8bd0 con 0x7f72480ad890 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 0x7f72481e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 0x7f72481e5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40384/0 (socket says 172.21.15.78:40384) 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 -- 172.21.15.78:0/3516570003 learned_addr learned my addr 172.21.15.78:0/3516570003 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f723ffff640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72481e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f724659c640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 -- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 msgr2=0x7f72480a9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480a9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 -- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 msgr2=0x7f72481e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72481e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 -- 172.21.15.78:0/3516570003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7238009400 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.701+0000 7f7246d9d640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 0x7f72481e5b90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f724000cb30 tx=0x7f724000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.702+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000aca0 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.702+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000ae00 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.702+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7230009400 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.702+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240012600 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.702+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72481e9150 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.703+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72480bdec0 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.703+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7240012760 con 0x7f72481534d0 2024-02-19T14:55:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.703+0000 7f723effd640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f722806b810 0x7f722806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.703+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f724008a1c0 con 0x7f72481534d0 2024-02-19T14:55:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.703+0000 7f724659c640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f722806b810 0x7f722806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.704+0000 7f724659c640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f722806b810 0x7f722806dcd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7238009c30 tx=0x7f7238009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.708+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f72480bdec0 con 0x7f72481534d0 2024-02-19T14:55:39.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.913+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f72480bdec0 con 0x7f72481534d0 2024-02-19T14:55:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.965+0000 7f723effd640 1 -- 172.21.15.78:0/3516570003 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v83) v1 ==== 118+0+0 (secure 0 0 0) 0x7f72480bdec0 con 0x7f72481534d0 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f722806b810 msgr2=0x7f722806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f722806b810 0x7f722806dcd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7238009c30 tx=0x7f7238009190 comp rx=0 tx=0).stop 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 msgr2=0x7f72481e5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 0x7f72481e5b90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f724000cb30 tx=0x7f724000a060 comp rx=0 tx=0).stop 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 shutdown_connections 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f722806b810 0x7f722806dcd0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72481534d0 0x7f72481e5b90 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72480c49c0 0x7f72481e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 --2- 172.21.15.78:0/3516570003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72480ad890 0x7f72480a9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 >> 172.21.15.78:0/3516570003 conn(0x7f724806b700 msgr2=0x7f724814f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 shutdown_connections 2024-02-19T14:55:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:39.967+0000 7f724cd81640 1 -- 172.21.15.78:0/3516570003 wait complete. 2024-02-19T14:55:39.981 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:55:39.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:55:40.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.082+0000 7fa02fbe2640 1 Processor -- start 2024-02-19T14:55:40.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.083+0000 7fa02fbe2640 1 -- start start 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02fbe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0280719e0 0x7fa0280c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02fbe2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0280c3d90 0x7fa0280c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02fbe2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02806ff90 con 0x7fa02814d210 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02fbe2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028070270 con 0x7fa0280719e0 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02fbe2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0280c3970 con 0x7fa0280c3d90 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0280719e0 0x7fa0280c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0280719e0 0x7fa0280c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40390/0 (socket says 172.21.15.78:40390) 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 -- 172.21.15.78:0/4160387349 learned_addr learned my addr 172.21.15.78:0/4160387349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02e17f640 1 --2- 172.21.15.78:0/4160387349 >> v2:172.21.15.150:3300/0 conn(0x7fa0280c3d90 0x7fa0280c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:40.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 -- 172.21.15.78:0/4160387349 >> v2:172.21.15.150:3300/0 conn(0x7fa0280c3d90 msgr2=0x7fa0280c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 --2- 172.21.15.78:0/4160387349 >> v2:172.21.15.150:3300/0 conn(0x7fa0280c3d90 0x7fa0280c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 -- 172.21.15.78:0/4160387349 >> v1:172.21.15.150:6789/0 conn(0x7fa02814d210 legacy=0x7fa0280c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.084+0000 7fa02d17d640 1 -- 172.21.15.78:0/4160387349 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028156fc0 con 0x7fa0280719e0 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.085+0000 7fa02d17d640 1 --2- 172.21.15.78:0/4160387349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0280719e0 0x7fa0280c64d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fa02806eb30 tx=0x7fa02402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4aeb5684158024c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.085+0000 7fa02c97c640 1 -- 172.21.15.78:0/4160387349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0240317e0 con 0x7fa0280719e0 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.085+0000 7fa02c97c640 1 -- 172.21.15.78:0/4160387349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa024031940 con 0x7fa0280719e0 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.085+0000 7fa02c97c640 1 -- 172.21.15.78:0/4160387349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa024031cd0 con 0x7fa0280719e0 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 -- 172.21.15.78:0/4160387349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0280719e0 msgr2=0x7fa0280c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/4160387349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0280719e0 0x7fa0280c64d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fa02806eb30 tx=0x7fa02402b880 comp rx=0 tx=0).stop 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 -- 172.21.15.78:0/4160387349 shutdown_connections 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/4160387349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0280719e0 0x7fa0280c64d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/4160387349 >> v2:172.21.15.150:3300/0 conn(0x7fa0280c3d90 0x7fa0280c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 -- 172.21.15.78:0/4160387349 >> 172.21.15.78:0/4160387349 conn(0x7fa0280c5050 msgr2=0x7fa0280c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 -- 172.21.15.78:0/4160387349 shutdown_connections 2024-02-19T14:55:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.086+0000 7fa02fbe2640 1 -- 172.21.15.78:0/4160387349 wait complete. 2024-02-19T14:55:40.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.087+0000 7fa02fbe2640 1 Processor -- start 2024-02-19T14:55:40.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.087+0000 7fa02fbe2640 1 -- start start 2024-02-19T14:55:40.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280719e0 0x7fa0280c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280c3d90 0x7fa028145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 0x7fa028145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281eab30 con 0x7fa02814d210 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa028145aa0 con 0x7fa0280719e0 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281eea70 con 0x7fa0280c3d90 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 0x7fa028145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 0x7fa028145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40402/0 (socket says 172.21.15.78:40402) 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 -- 172.21.15.78:0/445817597 learned_addr learned my addr 172.21.15.78:0/445817597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02d17d640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280c3d90 0x7fa028145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02d97e640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280719e0 0x7fa0280c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 -- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280c3d90 msgr2=0x7fa028145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280c3d90 0x7fa028145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 -- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280719e0 msgr2=0x7fa0280c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280719e0 0x7fa0280c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 -- 172.21.15.78:0/445817597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0240094d0 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02e17f640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 0x7fa028145560 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fa02000cd60 tx=0x7fa02000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa020003b90 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa020003d10 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0280ab370 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa020004260 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.088+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa028145c70 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.089+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0281eac70 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.090+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa0200043c0 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.090+0000 7fa01e7fc640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b810 0x7fa00c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.090+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa020012070 con 0x7fa02814d210 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.090+0000 7fa02d97e640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b810 0x7fa00c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:40.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.091+0000 7fa02d97e640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b810 0x7fa00c06dcd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa01800a730 tx=0x7fa018008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:40.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.095+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa0281eac70 con 0x7fa02814d210 2024-02-19T14:55:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.367+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa0281eac70 con 0x7fa02814d210 2024-02-19T14:55:40.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.973+0000 7fa01e7fc640 1 -- 172.21.15.78:0/445817597 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v84) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa0281eac70 con 0x7fa02814d210 2024-02-19T14:55:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b810 msgr2=0x7fa00c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b810 0x7fa00c06dcd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa01800a730 tx=0x7fa018008040 comp rx=0 tx=0).stop 2024-02-19T14:55:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 msgr2=0x7fa028145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 0x7fa028145560 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fa02000cd60 tx=0x7fa02000a660 comp rx=0 tx=0).stop 2024-02-19T14:55:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 shutdown_connections 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa00c06b810 0x7fa00c06dcd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa02814d210 0x7fa028145560 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0280c3d90 0x7fa028145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 --2- 172.21.15.78:0/445817597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0280719e0 0x7fa0280c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.975+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 >> 172.21.15.78:0/445817597 conn(0x7fa0280c5050 msgr2=0x7fa0280c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.976+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 shutdown_connections 2024-02-19T14:55:40.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:40.976+0000 7fa02fbe2640 1 -- 172.21.15.78:0/445817597 wait complete. 2024-02-19T14:55:40.989 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:55:40.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:41.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.085+0000 7f2d2d579640 1 Processor -- start 2024-02-19T14:55:41.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.085+0000 7f2d2d579640 1 -- start start 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.086+0000 7f2d2d579640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d2806d920 0x7f2d2815e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.086+0000 7f2d2d579640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d28150820 con 0x7f2d2806ed20 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.086+0000 7f2d2d579640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d281537d0 con 0x7f2d28150440 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.086+0000 7f2d2d579640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d280c1b60 con 0x7f2d2806d920 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.086+0000 7f2d267fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2d2806ed20 0x7f2d2806d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60618/0 (socket says 172.21.15.78:60618) 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.086+0000 7f2d267fc640 1 -- 172.21.15.78:0/2675985612 learned_addr learned my addr 172.21.15.78:0/2675985612 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:41.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 --2- 172.21.15.78:0/2675985612 >> v2:172.21.15.150:3300/0 conn(0x7f2d2806d920 0x7f2d2815e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043361368 0 0) 0x7f2d28150820 con 0x7f2d2806ed20 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10003520 con 0x7f2d2806ed20 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 874793337 0 0) 0x7f2d10003520 con 0x7f2d2806ed20 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 >> v2:172.21.15.150:3300/0 conn(0x7f2d2806d920 msgr2=0x7f2d2815e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 --2- 172.21.15.78:0/2675985612 >> v2:172.21.15.150:3300/0 conn(0x7f2d2806d920 0x7f2d2815e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 >> v1:172.21.15.150:6789/0 conn(0x7f2d28150440 legacy=0x7f2d280c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28153d10 con 0x7f2d2806ed20 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2d1c001670 con 0x7f2d2806ed20 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 >> v1:172.21.15.78:6789/0 conn(0x7f2d2806ed20 legacy=0x7f2d2806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10003970 0x7f2d10003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d100047c0 0x7f2d10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10005180 0x7f2d1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d280c1b60 con 0x7f2d10005180 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1000c030 con 0x7f2d100047c0 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1000c1f0 con 0x7f2d10003970 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10005180 0x7f2d1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d267fc640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d100047c0 0x7f2d10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d26ffd640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10003970 0x7f2d10003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 -- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10003970 msgr2=0x7f2d10003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10003970 0x7f2d10003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 -- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d100047c0 msgr2=0x7f2d10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d100047c0 0x7f2d10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d277fe640 1 -- 172.21.15.78:0/2675985612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10003520 con 0x7f2d10005180 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.087+0000 7f2d26ffd640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10003970 0x7f2d10003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.088+0000 7f2d277fe640 1 -- 172.21.15.78:0/2675985612 reap_dead start 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.088+0000 7f2d277fe640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10005180 0x7f2d1000b980 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009fc0 tx=0x7f2d100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c636ad8746c07a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.088+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2002cc60 con 0x7f2d10005180 2024-02-19T14:55:41.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.088+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2000f040 con 0x7f2d10005180 2024-02-19T14:55:41.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.088+0000 7f2d25ffb640 1 -- 172.21.15.78:0/2675985612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20003990 con 0x7f2d10005180 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 -- 172.21.15.78:0/2675985612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10005180 msgr2=0x7f2d1000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10005180 0x7f2d1000b980 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009fc0 tx=0x7f2d100057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 -- 172.21.15.78:0/2675985612 shutdown_connections 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 --2- 172.21.15.78:0/2675985612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d10005180 0x7f2d1000b980 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 -- 172.21.15.78:0/2675985612 >> 172.21.15.78:0/2675985612 conn(0x7f2d2806b270 msgr2=0x7f2d2806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 -- 172.21.15.78:0/2675985612 shutdown_connections 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.089+0000 7f2d2d579640 1 -- 172.21.15.78:0/2675985612 wait complete. 2024-02-19T14:55:41.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.090+0000 7f2d2d579640 1 Processor -- start 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.090+0000 7f2d2d579640 1 -- start start 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d2d579640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10005180 0x7f2d28146e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d2d579640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 0x7f2d281dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d2d579640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28154da0 0x7f2d28155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d2d579640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d281473a0 con 0x7f2d2814a5e0 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d2d579640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28146b80 con 0x7f2d28154da0 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d2d579640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d281542c0 con 0x7f2d10005180 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 0x7f2d281dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 0x7f2d281dfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40412/0 (socket says 172.21.15.78:40412) 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 -- 172.21.15.78:0/1024928044 learned_addr learned my addr 172.21.15.78:0/1024928044 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d26ffd640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10005180 0x7f2d28146e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d277fe640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28154da0 0x7f2d28155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 -- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10005180 msgr2=0x7f2d28146e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10005180 0x7f2d28146e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 -- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28154da0 msgr2=0x7f2d28155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28154da0 0x7f2d28155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d267fc640 1 -- 172.21.15.78:0/1024928044 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d281545a0 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d26ffd640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10005180 0x7f2d28146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.091+0000 7f2d277fe640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28154da0 0x7f2d28155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.092+0000 7f2d267fc640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 0x7f2d281dfd50 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c004f70 tx=0x7f2d1c0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.092+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00f750 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.092+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00f8d0 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.092+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d28154880 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.092+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00fc80 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.092+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d28152e30 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.093+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d28154aa0 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.093+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2d1c01d460 con 0x7f2d2814a5e0 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.094+0000 7f2d0f7fe640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1406b810 0x7f2d1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.094+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2d1c08a400 con 0x7f2d2814a5e0 2024-02-19T14:55:41.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.094+0000 7f2d26ffd640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1406b810 0x7f2d1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.094+0000 7f2d26ffd640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1406b810 0x7f2d1406dcd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2d18000c00 tx=0x7f2d18009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.102+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d1c054d20 con 0x7f2d2814a5e0 2024-02-19T14:55:41.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.315+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d28074990 con 0x7f2d2814a5e0 2024-02-19T14:55:41.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.316+0000 7f2d0f7fe640 1 -- 172.21.15.78:0/1024928044 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f2d1c0583e0 con 0x7f2d2814a5e0 2024-02-19T14:55:41.319 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:41.319 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5927,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:40.368516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2978708219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2978708219},{"type":"v1","addr":"172.21.15.78:6837","nonce":2978708219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:41.319 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 84 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1406b810 msgr2=0x7f2d1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1406b810 0x7f2d1406dcd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2d18000c00 tx=0x7f2d18009040 comp rx=0 tx=0).stop 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 msgr2=0x7f2d281dfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 0x7f2d281dfd50 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c004f70 tx=0x7f2d1c0074f0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 shutdown_connections 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2d1406b810 0x7f2d1406dcd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d2814a5e0 0x7f2d281dfd50 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28154da0 0x7f2d28155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 --2- 172.21.15.78:0/1024928044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d10005180 0x7f2d28146e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 >> 172.21.15.78:0/1024928044 conn(0x7f2d2806b270 msgr2=0x7f2d2806d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 shutdown_connections 2024-02-19T14:55:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.318+0000 7f2d2d579640 1 -- 172.21.15.78:0/1024928044 wait complete. 2024-02-19T14:55:41.332 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:55:41.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 Processor -- start 2024-02-19T14:55:41.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 -- start start 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6cb4155f90 0x7f6cb406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 0x7f6cb4158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb40c6af0 con 0x7f6cb40c6d30 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb40c7940 con 0x7f6cb4155f90 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb97b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40c7c20 con 0x7f6cb41588a0 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.429+0000 7f6cb27fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6cb40c6d30 0x7f6cb40c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60622/0 (socket says 172.21.15.78:60622) 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb27fc640 1 -- 172.21.15.78:0/3335716761 learned_addr learned my addr 172.21.15.78:0/3335716761 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 0x7f6cb4158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb2ffd640 1 --2- 172.21.15.78:0/3335716761 >> v2:172.21.15.150:3300/0 conn(0x7f6cb4155f90 0x7f6cb406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119237699 0 0) 0x7f6cb40c6af0 con 0x7f6cb40c6d30 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c003520 con 0x7f6cb40c6d30 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971128653 0 0) 0x7f6c9c003520 con 0x7f6cb40c6d30 2024-02-19T14:55:41.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 msgr2=0x7f6cb4158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 0x7f6cb4158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 >> v2:172.21.15.150:3300/0 conn(0x7f6cb4155f90 msgr2=0x7f6cb406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 --2- 172.21.15.78:0/3335716761 >> v2:172.21.15.150:3300/0 conn(0x7f6cb4155f90 0x7f6cb406d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb40c7f00 con 0x7f6cb40c6d30 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f6ca8001690 con 0x7f6cb40c6d30 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 >> v1:172.21.15.78:6789/0 conn(0x7f6cb40c6d30 legacy=0x7f6cb40c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c9c003970 0x7f6c9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 0x7f6c9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c004d00 0x7f6c9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40c7940 con 0x7f6c9c004d00 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40c7c20 con 0x7f6c9c003970 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c00b980 con 0x7f6cb41588a0 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c004d00 0x7f6c9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb2ffd640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c9c003970 0x7f6c9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.430+0000 7f6cb27fc640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 0x7f6c9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb37fe640 1 -- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 msgr2=0x7f6c9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb41588a0 0x7f6c9c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb37fe640 1 -- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c9c003970 msgr2=0x7f6c9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c9c003970 0x7f6c9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb37fe640 1 -- 172.21.15.78:0/3335716761 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c003520 con 0x7f6c9c004d00 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c004d00 0x7f6c9c005180 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009fc0 tx=0x7f6c9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b15ca889af11473 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac02cc20 con 0x7f6c9c004d00 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cac00f040 con 0x7f6c9c004d00 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.431+0000 7f6cb1ffb640 1 -- 172.21.15.78:0/3335716761 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac034440 con 0x7f6c9c004d00 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3335716761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c004d00 msgr2=0x7f6c9c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3335716761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c004d00 0x7f6c9c005180 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009fc0 tx=0x7f6c9c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb37fe640 1 -- 172.21.15.78:0/3335716761 reap_dead start 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3335716761 shutdown_connections 2024-02-19T14:55:41.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3335716761 >> 172.21.15.78:0/3335716761 conn(0x7f6cb406b700 msgr2=0x7f6cb40aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:41.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3335716761 shutdown_connections 2024-02-19T14:55:41.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.432+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3335716761 wait complete. 2024-02-19T14:55:41.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.433+0000 7f6cb97b5640 1 Processor -- start 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.433+0000 7f6cb97b5640 1 -- start start 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb97b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cb40acc10 0x7f6cb414e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb97b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 0x7f6cb414eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb97b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb4156930 0x7f6cb4156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb97b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4070500 con 0x7f6cb4071650 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb97b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40700b0 con 0x7f6cb40acc10 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb97b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4158d50 con 0x7f6cb4156930 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 0x7f6cb414eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 0x7f6cb414eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40426/0 (socket says 172.21.15.78:40426) 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 -- 172.21.15.78:0/3082856678 learned_addr learned my addr 172.21.15.78:0/3082856678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb2ffd640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cb40acc10 0x7f6cb414e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb4156930 0x7f6cb4156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 -- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb4156930 msgr2=0x7f6cb4156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb4156930 0x7f6cb4156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 -- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cb40acc10 msgr2=0x7f6cb414e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cb40acc10 0x7f6cb414e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb27fc640 1 -- 172.21.15.78:0/3082856678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb40ab780 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb2ffd640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cb40acc10 0x7f6cb414e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.434+0000 7f6cb37fe640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb4156930 0x7f6cb4156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.435+0000 7f6cb27fc640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 0x7f6cb414eca0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8001a50 tx=0x7f6ca8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.435+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80014c0 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.435+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca800fa20 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.435+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca800fdd0 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.435+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb40aba60 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.435+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb414f1e0 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.436+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4156650 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.436+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6ca801d460 con 0x7f6cb4071650 2024-02-19T14:55:41.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.437+0000 7f6c9b7fe640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ca006b840 0x7f6ca006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.437+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6ca808a580 con 0x7f6cb4071650 2024-02-19T14:55:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.437+0000 7f6cb2ffd640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ca006b840 0x7f6ca006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.437+0000 7f6cb2ffd640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ca006b840 0x7f6ca006dd00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009ab0 tx=0x7f6ca4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.441+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6cb4156650 con 0x7f6cb4071650 2024-02-19T14:55:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.619+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb4156650 con 0x7f6cb4071650 2024-02-19T14:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.620+0000 7f6c9b7fe640 1 -- 172.21.15.78:0/3082856678 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16940 (secure 0 0 0) 0x7f6cb4156650 con 0x7f6cb4071650 2024-02-19T14:55:41.623 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:41.623 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":111,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:39.953654+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:36.521408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:37.271805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ca006b840 msgr2=0x7f6ca006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ca006b840 0x7f6ca006dd00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009ab0 tx=0x7f6ca4008040 comp rx=0 tx=0).stop 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 msgr2=0x7f6cb414eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 0x7f6cb414eca0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8001a50 tx=0x7f6ca8007720 comp rx=0 tx=0).stop 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 shutdown_connections 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ca006b840 0x7f6ca006dd00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cb4071650 0x7f6cb414eca0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6cb4156930 0x7f6cb4156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 --2- 172.21.15.78:0/3082856678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cb40acc10 0x7f6cb414e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 >> 172.21.15.78:0/3082856678 conn(0x7f6cb406b700 msgr2=0x7f6cb4155240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 shutdown_connections 2024-02-19T14:55:41.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.623+0000 7f6cb97b5640 1 -- 172.21.15.78:0/3082856678 wait complete. 2024-02-19T14:55:41.640 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:55:41.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.775+0000 7f642835c640 1 Processor -- start 2024-02-19T14:55:41.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.775+0000 7f642835c640 1 -- start start 2024-02-19T14:55:41.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f642835c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64200c25e0 0x7f64201526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f642835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f64200c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f642835c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6420072250 con 0x7f64200c49c0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f642835c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6420152c90 con 0x7f64200c25e0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f642835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6420072480 con 0x7f64200c29c0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64200c25e0 0x7f64201526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64200c25e0 0x7f64201526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40434/0 (socket says 172.21.15.78:40434) 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 -- 172.21.15.78:0/3368945490 learned_addr learned my addr 172.21.15.78:0/3368945490 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64258f7640 1 --2- 172.21.15.78:0/3368945490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f64200c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 -- 172.21.15.78:0/3368945490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 msgr2=0x7f64200c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 --2- 172.21.15.78:0/3368945490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f64200c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 -- 172.21.15.78:0/3368945490 >> v1:172.21.15.150:6789/0 conn(0x7f64200c49c0 legacy=0x7f642006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.776+0000 7f64260f8640 1 -- 172.21.15.78:0/3368945490 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64200c3eb0 con 0x7f64200c25e0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.777+0000 7f64260f8640 1 --2- 172.21.15.78:0/3368945490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64200c25e0 0x7f64201526a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f641c009870 tx=0x7f641c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8cb7984ea088e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.777+0000 7f64250f6640 1 -- 172.21.15.78:0/3368945490 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c0317c0 con 0x7f64200c25e0 2024-02-19T14:55:41.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.777+0000 7f64250f6640 1 -- 172.21.15.78:0/3368945490 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c031920 con 0x7f64200c25e0 2024-02-19T14:55:41.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.777+0000 7f64250f6640 1 -- 172.21.15.78:0/3368945490 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c031cb0 con 0x7f64200c25e0 2024-02-19T14:55:41.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.777+0000 7f642835c640 1 -- 172.21.15.78:0/3368945490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64200c25e0 msgr2=0x7f64201526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.777+0000 7f642835c640 1 --2- 172.21.15.78:0/3368945490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64200c25e0 0x7f64201526a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f641c009870 tx=0x7f641c02b860 comp rx=0 tx=0).stop 2024-02-19T14:55:41.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.778+0000 7f642835c640 1 -- 172.21.15.78:0/3368945490 shutdown_connections 2024-02-19T14:55:41.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.778+0000 7f642835c640 1 --2- 172.21.15.78:0/3368945490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f64200c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.778+0000 7f642835c640 1 --2- 172.21.15.78:0/3368945490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64200c25e0 0x7f64201526a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.778+0000 7f642835c640 1 -- 172.21.15.78:0/3368945490 >> 172.21.15.78:0/3368945490 conn(0x7f642006b270 msgr2=0x7f642006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.778+0000 7f642835c640 1 -- 172.21.15.78:0/3368945490 shutdown_connections 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.778+0000 7f642835c640 1 -- 172.21.15.78:0/3368945490 wait complete. 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 Processor -- start 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 -- start start 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64200c25e0 0x7f64200ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f6420155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 0x7f642013e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420143d90 con 0x7f6420145ec0 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64200a9b40 con 0x7f64200c25e0 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f642835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6420143110 con 0x7f64200c29c0 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.779+0000 7f64268f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 0x7f642013e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 0x7f642013e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40436/0 (socket says 172.21.15.78:40436) 2024-02-19T14:55:41.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 -- 172.21.15.78:0/251890279 learned_addr learned my addr 172.21.15.78:0/251890279 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 -- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 msgr2=0x7f6420155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64258f7640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f6420155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64260f8640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64200c25e0 0x7f64200ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f6420155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 -- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64200c25e0 msgr2=0x7f64200ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64200c25e0 0x7f64200ac260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 -- 172.21.15.78:0/251890279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0094d0 con 0x7f6420145ec0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f64268f9640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 0x7f642013e740 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f641800c700 tx=0x7f641800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800aa50 con 0x7f6420145ec0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641800abb0 con 0x7f6420145ec0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64201433f0 con 0x7f6420145ec0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800ce10 con 0x7f6420145ec0 2024-02-19T14:55:41.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.780+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64201439b0 con 0x7f6420145ec0 2024-02-19T14:55:41.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.781+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6420074970 con 0x7f6420145ec0 2024-02-19T14:55:41.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.781+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6418012420 con 0x7f6420145ec0 2024-02-19T14:55:41.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.782+0000 7f6416ffd640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f640406b810 0x7f640406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:41.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.782+0000 7f64260f8640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f640406b810 0x7f640406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:41.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.782+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f641808a7f0 con 0x7f6420145ec0 2024-02-19T14:55:41.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.783+0000 7f64260f8640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f640406b810 0x7f640406dcd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f641c036000 tx=0x7f641c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:41.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.786+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6418055110 con 0x7f6420145ec0 2024-02-19T14:55:41.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.979+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6420072ce0 con 0x7f640406b810 2024-02-19T14:55:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.994+0000 7f6416ffd640 1 -- 172.21.15.78:0/251890279 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284417 (secure 0 0 0) 0x7f641c031b90 con 0x7f640406b810 2024-02-19T14:55:42.000 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f640406b810 msgr2=0x7f640406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f640406b810 0x7f640406dcd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f641c036000 tx=0x7f641c03d040 comp rx=0 tx=0).stop 2024-02-19T14:55:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 msgr2=0x7f642013e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 0x7f642013e740 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f641800c700 tx=0x7f641800cbc0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 shutdown_connections 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f640406b810 0x7f640406dcd0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6420145ec0 0x7f642013e740 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64200c29c0 0x7f6420155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 --2- 172.21.15.78:0/251890279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64200c25e0 0x7f64200ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 >> 172.21.15.78:0/251890279 conn(0x7f642006b270 msgr2=0x7f642006df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 shutdown_connections 2024-02-19T14:55:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:41.997+0000 7f642835c640 1 -- 172.21.15.78:0/251890279 wait complete. 2024-02-19T14:55:42.013 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-02-19T14:55:40.102378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257000,"kb_used_data":11712,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754717720,"statfs":{"total":773094113280,"available":772830945280,"internally_reserved":0,"allocated":11993088,"data_stored":6412102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000588"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955144+0000","last_change":"2024-02-19T14:55:37.955144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955144+0000","last_undegraded":"2024-02-19T14:55:37.955144+0000","last_fullsized":"2024-02-19T14:55:37.955144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962463+0000","last_change":"2024-02-19T14:55:37.962463+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962463+0000","last_undegraded":"2024-02-19T14:55:37.962463+0000","last_fullsized":"2024-02-19T14:55:37.962463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958706+0000","last_change":"2024-02-19T14:55:37.958706+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958706+0000","last_undegraded":"2024-02-19T14:55:37.958706+0000","last_fullsized":"2024-02-19T14:55:37.958706+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961496+0000","last_change":"2024-02-19T14:55:37.961496+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961496+0000","last_undegraded":"2024-02-19T14:55:37.961496+0000","last_fullsized":"2024-02-19T14:55:37.961496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956553+0000","last_change":"2024-02-19T14:55:37.956553+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956553+0000","last_undegraded":"2024-02-19T14:55:37.956553+0000","last_fullsized":"2024-02-19T14:55:37.956553+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959221+0000","last_change":"2024-02-19T14:55:37.959221+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959221+0000","last_undegraded":"2024-02-19T14:55:37.959221+0000","last_fullsized":"2024-02-19T14:55:37.959221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958511+0000","last_change":"2024-02-19T14:55:37.958511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958511+0000","last_undegraded":"2024-02-19T14:55:37.958511+0000","last_fullsized":"2024-02-19T14:55:37.958511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960041+0000","last_change":"2024-02-19T14:55:37.960041+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960041+0000","last_undegraded":"2024-02-19T14:55:37.960041+0000","last_fullsized":"2024-02-19T14:55:37.960041+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965608+0000","last_change":"2024-02-19T14:55:37.965608+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965608+0000","last_undegraded":"2024-02-19T14:55:37.965608+0000","last_fullsized":"2024-02-19T14:55:37.965608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956058+0000","last_change":"2024-02-19T14:55:37.956058+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956058+0000","last_undegraded":"2024-02-19T14:55:37.956058+0000","last_fullsized":"2024-02-19T14:55:37.956058+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961425+0000","last_change":"2024-02-19T14:55:37.961425+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961425+0000","last_undegraded":"2024-02-19T14:55:37.961425+0000","last_fullsized":"2024-02-19T14:55:37.961425+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967924+0000","last_change":"2024-02-19T14:55:37.967924+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967924+0000","last_undegraded":"2024-02-19T14:55:37.967924+0000","last_fullsized":"2024-02-19T14:55:37.967924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960544+0000","last_change":"2024-02-19T14:55:37.960544+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960544+0000","last_undegraded":"2024-02-19T14:55:37.960544+0000","last_fullsized":"2024-02-19T14:55:37.960544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956259+0000","last_change":"2024-02-19T14:55:37.956259+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956259+0000","last_undegraded":"2024-02-19T14:55:37.956259+0000","last_fullsized":"2024-02-19T14:55:37.956259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961548+0000","last_change":"2024-02-19T14:55:37.961548+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961548+0000","last_undegraded":"2024-02-19T14:55:37.961548+0000","last_fullsized":"2024-02-19T14:55:37.961548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965387+0000","last_change":"2024-02-19T14:55:37.965387+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965387+0000","last_undegraded":"2024-02-19T14:55:37.965387+0000","last_fullsized":"2024-02-19T14:55:37.965387+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962059+0000","last_change":"2024-02-19T14:55:37.962059+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962059+0000","last_undegraded":"2024-02-19T14:55:37.962059+0000","last_fullsized":"2024-02-19T14:55:37.962059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959638+0000","last_change":"2024-02-19T14:55:37.959638+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959638+0000","last_undegraded":"2024-02-19T14:55:37.959638+0000","last_fullsized":"2024-02-19T14:55:37.959638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958119+0000","last_change":"2024-02-19T14:55:37.958119+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958119+0000","last_undegraded":"2024-02-19T14:55:37.958119+0000","last_fullsized":"2024-02-19T14:55:37.958119+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962661+0000","last_change":"2024-02-19T14:55:37.962661+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962661+0000","last_undegraded":"2024-02-19T14:55:37.962661+0000","last_fullsized":"2024-02-19T14:55:37.962661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955835+0000","last_change":"2024-02-19T14:55:37.955835+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955835+0000","last_undegraded":"2024-02-19T14:55:37.955835+0000","last_fullsized":"2024-02-19T14:55:37.955835+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960561+0000","last_change":"2024-02-19T14:55:37.960561+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960561+0000","last_undegraded":"2024-02-19T14:55:37.960561+0000","last_fullsized":"2024-02-19T14:55:37.960561+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958144+0000","last_change":"2024-02-19T14:55:37.958144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958144+0000","last_undegraded":"2024-02-19T14:55:37.958144+0000","last_fullsized":"2024-02-19T14:55:37.958144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958449+0000","last_change":"2024-02-19T14:55:37.958449+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958449+0000","last_undegraded":"2024-02-19T14:55:37.958449+0000","last_fullsized":"2024-02-19T14:55:37.958449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957844+0000","last_change":"2024-02-19T14:55:37.957844+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957844+0000","last_undegraded":"2024-02-19T14:55:37.957844+0000","last_fullsized":"2024-02-19T14:55:37.957844+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959212+0000","last_change":"2024-02-19T14:55:37.959212+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959212+0000","last_undegraded":"2024-02-19T14:55:37.959212+0000","last_fullsized":"2024-02-19T14:55:37.959212+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961013+0000","last_change":"2024-02-19T14:55:37.961013+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961013+0000","last_undegraded":"2024-02-19T14:55:37.961013+0000","last_fullsized":"2024-02-19T14:55:37.961013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957840+0000","last_change":"2024-02-19T14:55:37.957840+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957840+0000","last_undegraded":"2024-02-19T14:55:37.957840+0000","last_fullsized":"2024-02-19T14:55:37.957840+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957510+0000","last_change":"2024-02-19T14:55:37.957510+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957510+0000","last_undegraded":"2024-02-19T14:55:37.957510+0000","last_fullsized":"2024-02-19T14:55:37.957510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956289+0000","last_change":"2024-02-19T14:55:37.956289+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956289+0000","last_undegraded":"2024-02-19T14:55:37.956289+0000","last_fullsized":"2024-02-19T14:55:37.956289+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.966383+0000","last_change":"2024-02-19T14:55:37.966383+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.966383+0000","last_undegraded":"2024-02-19T14:55:37.966383+0000","last_fullsized":"2024-02-19T14:55:37.966383+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969047+0000","last_change":"2024-02-19T14:55:37.967453+0000","last_active":"2024-02-19T14:55:37.969047+0000","last_peered":"2024-02-19T14:55:37.969047+0000","last_clean":"2024-02-19T14:55:37.969047+0000","last_became_active":"2024-02-19T14:55:37.967091+0000","last_became_peered":"2024-02-19T14:55:37.967091+0000","last_unstale":"2024-02-19T14:55:37.969047+0000","last_undegraded":"2024-02-19T14:55:37.969047+0000","last_fullsized":"2024-02-19T14:55:37.969047+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962010+0000","last_change":"2024-02-19T14:55:37.962010+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962010+0000","last_undegraded":"2024-02-19T14:55:37.962010+0000","last_fullsized":"2024-02-19T14:55:37.962010+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977066+0000","last_change":"2024-02-19T14:55:37.974585+0000","last_active":"2024-02-19T14:55:37.977066+0000","last_peered":"2024-02-19T14:55:37.977066+0000","last_clean":"2024-02-19T14:55:37.977066+0000","last_became_active":"2024-02-19T14:55:37.974280+0000","last_became_peered":"2024-02-19T14:55:37.974280+0000","last_unstale":"2024-02-19T14:55:37.977066+0000","last_undegraded":"2024-02-19T14:55:37.977066+0000","last_fullsized":"2024-02-19T14:55:37.977066+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:15.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957606+0000","last_change":"2024-02-19T14:55:37.957606+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957606+0000","last_undegraded":"2024-02-19T14:55:37.957606+0000","last_fullsized":"2024-02-19T14:55:37.957606+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973784+0000","last_change":"2024-02-19T14:55:37.971928+0000","last_active":"2024-02-19T14:55:37.973784+0000","last_peered":"2024-02-19T14:55:37.973784+0000","last_clean":"2024-02-19T14:55:37.973784+0000","last_became_active":"2024-02-19T14:55:37.971646+0000","last_became_peered":"2024-02-19T14:55:37.971646+0000","last_unstale":"2024-02-19T14:55:37.973784+0000","last_undegraded":"2024-02-19T14:55:37.973784+0000","last_fullsized":"2024-02-19T14:55:37.973784+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:38.067553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959582+0000","last_change":"2024-02-19T14:55:37.959582+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959582+0000","last_undegraded":"2024-02-19T14:55:37.959582+0000","last_fullsized":"2024-02-19T14:55:37.959582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977059+0000","last_change":"2024-02-19T14:55:37.973007+0000","last_active":"2024-02-19T14:55:37.977059+0000","last_peered":"2024-02-19T14:55:37.977059+0000","last_clean":"2024-02-19T14:55:37.977059+0000","last_became_active":"2024-02-19T14:55:37.972480+0000","last_became_peered":"2024-02-19T14:55:37.972480+0000","last_unstale":"2024-02-19T14:55:37.977059+0000","last_undegraded":"2024-02-19T14:55:37.977059+0000","last_fullsized":"2024-02-19T14:55:37.977059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.968277+0000","last_change":"2024-02-19T14:55:37.968277+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.968277+0000","last_undegraded":"2024-02-19T14:55:37.968277+0000","last_fullsized":"2024-02-19T14:55:37.968277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.982286+0000","last_change":"2024-02-19T14:55:37.975968+0000","last_active":"2024-02-19T14:55:37.982286+0000","last_peered":"2024-02-19T14:55:37.982286+0000","last_clean":"2024-02-19T14:55:37.982286+0000","last_became_active":"2024-02-19T14:55:37.975693+0000","last_became_peered":"2024-02-19T14:55:37.975693+0000","last_unstale":"2024-02-19T14:55:37.982286+0000","last_undegraded":"2024-02-19T14:55:37.982286+0000","last_fullsized":"2024-02-19T14:55:37.982286+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955143+0000","last_change":"2024-02-19T14:55:37.955143+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955143+0000","last_undegraded":"2024-02-19T14:55:37.955143+0000","last_fullsized":"2024-02-19T14:55:37.955143+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977070+0000","last_change":"2024-02-19T14:55:37.975370+0000","last_active":"2024-02-19T14:55:37.977070+0000","last_peered":"2024-02-19T14:55:37.977070+0000","last_clean":"2024-02-19T14:55:37.977070+0000","last_became_active":"2024-02-19T14:55:37.974133+0000","last_became_peered":"2024-02-19T14:55:37.974133+0000","last_unstale":"2024-02-19T14:55:37.977070+0000","last_undegraded":"2024-02-19T14:55:37.977070+0000","last_fullsized":"2024-02-19T14:55:37.977070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:31.145774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959131+0000","last_change":"2024-02-19T14:55:37.959131+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959131+0000","last_undegraded":"2024-02-19T14:55:37.959131+0000","last_fullsized":"2024-02-19T14:55:37.959131+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967695+0000","last_change":"2024-02-19T14:55:37.965578+0000","last_active":"2024-02-19T14:55:37.967695+0000","last_peered":"2024-02-19T14:55:37.967695+0000","last_clean":"2024-02-19T14:55:37.967695+0000","last_became_active":"2024-02-19T14:55:37.965063+0000","last_became_peered":"2024-02-19T14:55:37.965063+0000","last_unstale":"2024-02-19T14:55:37.967695+0000","last_undegraded":"2024-02-19T14:55:37.967695+0000","last_fullsized":"2024-02-19T14:55:37.967695+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967476+0000","last_change":"2024-02-19T14:55:37.967476+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967476+0000","last_undegraded":"2024-02-19T14:55:37.967476+0000","last_fullsized":"2024-02-19T14:55:37.967476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979469+0000","last_change":"2024-02-19T14:55:37.976412+0000","last_active":"2024-02-19T14:55:37.979469+0000","last_peered":"2024-02-19T14:55:37.979469+0000","last_clean":"2024-02-19T14:55:37.979469+0000","last_became_active":"2024-02-19T14:55:37.975831+0000","last_became_peered":"2024-02-19T14:55:37.975831+0000","last_unstale":"2024-02-19T14:55:37.979469+0000","last_undegraded":"2024-02-19T14:55:37.979469+0000","last_fullsized":"2024-02-19T14:55:37.979469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:55:53.499236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959395+0000","last_change":"2024-02-19T14:55:37.959395+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959395+0000","last_undegraded":"2024-02-19T14:55:37.959395+0000","last_fullsized":"2024-02-19T14:55:37.959395+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.971160+0000","last_change":"2024-02-19T14:55:37.969860+0000","last_active":"2024-02-19T14:55:37.971160+0000","last_peered":"2024-02-19T14:55:37.971160+0000","last_clean":"2024-02-19T14:55:37.971160+0000","last_became_active":"2024-02-19T14:55:37.969561+0000","last_became_peered":"2024-02-19T14:55:37.969561+0000","last_unstale":"2024-02-19T14:55:37.971160+0000","last_undegraded":"2024-02-19T14:55:37.971160+0000","last_fullsized":"2024-02-19T14:55:37.971160+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956550+0000","last_change":"2024-02-19T14:55:37.956550+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956550+0000","last_undegraded":"2024-02-19T14:55:37.956550+0000","last_fullsized":"2024-02-19T14:55:37.956550+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973119+0000","last_change":"2024-02-19T14:55:37.972249+0000","last_active":"2024-02-19T14:55:37.973119+0000","last_peered":"2024-02-19T14:55:37.973119+0000","last_clean":"2024-02-19T14:55:37.973119+0000","last_became_active":"2024-02-19T14:55:37.971802+0000","last_became_peered":"2024-02-19T14:55:37.971802+0000","last_unstale":"2024-02-19T14:55:37.973119+0000","last_undegraded":"2024-02-19T14:55:37.973119+0000","last_fullsized":"2024-02-19T14:55:37.973119+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961622+0000","last_change":"2024-02-19T14:55:37.961622+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961622+0000","last_undegraded":"2024-02-19T14:55:37.961622+0000","last_fullsized":"2024-02-19T14:55:37.961622+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970044+0000","last_change":"2024-02-19T14:55:37.968601+0000","last_active":"2024-02-19T14:55:37.970044+0000","last_peered":"2024-02-19T14:55:37.970044+0000","last_clean":"2024-02-19T14:55:37.970044+0000","last_became_active":"2024-02-19T14:55:37.968117+0000","last_became_peered":"2024-02-19T14:55:37.968117+0000","last_unstale":"2024-02-19T14:55:37.970044+0000","last_undegraded":"2024-02-19T14:55:37.970044+0000","last_fullsized":"2024-02-19T14:55:37.970044+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958050+0000","last_change":"2024-02-19T14:55:37.958050+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958050+0000","last_undegraded":"2024-02-19T14:55:37.958050+0000","last_fullsized":"2024-02-19T14:55:37.958050+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972821+0000","last_change":"2024-02-19T14:55:37.968678+0000","last_active":"2024-02-19T14:55:37.972821+0000","last_peered":"2024-02-19T14:55:37.972821+0000","last_clean":"2024-02-19T14:55:37.972821+0000","last_became_active":"2024-02-19T14:55:37.968384+0000","last_became_peered":"2024-02-19T14:55:37.968384+0000","last_unstale":"2024-02-19T14:55:37.972821+0000","last_undegraded":"2024-02-19T14:55:37.972821+0000","last_fullsized":"2024-02-19T14:55:37.972821+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:19.012872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960027+0000","last_change":"2024-02-19T14:55:37.960027+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960027+0000","last_undegraded":"2024-02-19T14:55:37.960027+0000","last_fullsized":"2024-02-19T14:55:37.960027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975368+0000","last_change":"2024-02-19T14:55:37.969264+0000","last_active":"2024-02-19T14:55:37.975368+0000","last_peered":"2024-02-19T14:55:37.975368+0000","last_clean":"2024-02-19T14:55:37.975368+0000","last_became_active":"2024-02-19T14:55:37.968958+0000","last_became_peered":"2024-02-19T14:55:37.968958+0000","last_unstale":"2024-02-19T14:55:37.975368+0000","last_undegraded":"2024-02-19T14:55:37.975368+0000","last_fullsized":"2024-02-19T14:55:37.975368+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967940+0000","last_change":"2024-02-19T14:55:37.967940+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967940+0000","last_undegraded":"2024-02-19T14:55:37.967940+0000","last_fullsized":"2024-02-19T14:55:37.967940+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975543+0000","last_change":"2024-02-19T14:55:37.970686+0000","last_active":"2024-02-19T14:55:37.975543+0000","last_peered":"2024-02-19T14:55:37.975543+0000","last_clean":"2024-02-19T14:55:37.975543+0000","last_became_active":"2024-02-19T14:55:37.970405+0000","last_became_peered":"2024-02-19T14:55:37.970405+0000","last_unstale":"2024-02-19T14:55:37.975543+0000","last_undegraded":"2024-02-19T14:55:37.975543+0000","last_fullsized":"2024-02-19T14:55:37.975543+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957316+0000","last_change":"2024-02-19T14:55:37.957316+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957316+0000","last_undegraded":"2024-02-19T14:55:37.957316+0000","last_fullsized":"2024-02-19T14:55:37.957316+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975423+0000","last_change":"2024-02-19T14:55:37.969681+0000","last_active":"2024-02-19T14:55:37.975423+0000","last_peered":"2024-02-19T14:55:37.975423+0000","last_clean":"2024-02-19T14:55:37.975423+0000","last_became_active":"2024-02-19T14:55:37.969413+0000","last_became_peered":"2024-02-19T14:55:37.969413+0000","last_unstale":"2024-02-19T14:55:37.975423+0000","last_undegraded":"2024-02-19T14:55:37.975423+0000","last_fullsized":"2024-02-19T14:55:37.975423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957511+0000","last_change":"2024-02-19T14:55:37.957511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957511+0000","last_undegraded":"2024-02-19T14:55:37.957511+0000","last_fullsized":"2024-02-19T14:55:37.957511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977083+0000","last_change":"2024-02-19T14:55:37.973068+0000","last_active":"2024-02-19T14:55:37.977083+0000","last_peered":"2024-02-19T14:55:37.977083+0000","last_clean":"2024-02-19T14:55:37.977083+0000","last_became_active":"2024-02-19T14:55:37.972754+0000","last_became_peered":"2024-02-19T14:55:37.972754+0000","last_unstale":"2024-02-19T14:55:37.977083+0000","last_undegraded":"2024-02-19T14:55:37.977083+0000","last_fullsized":"2024-02-19T14:55:37.977083+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:14.587666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955566+0000","last_change":"2024-02-19T14:55:37.955566+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955566+0000","last_undegraded":"2024-02-19T14:55:37.955566+0000","last_fullsized":"2024-02-19T14:55:37.955566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969867+0000","last_change":"2024-02-19T14:55:37.963382+0000","last_active":"2024-02-19T14:55:37.969867+0000","last_peered":"2024-02-19T14:55:37.969867+0000","last_clean":"2024-02-19T14:55:37.969867+0000","last_became_active":"2024-02-19T14:55:37.963016+0000","last_became_peered":"2024-02-19T14:55:37.963016+0000","last_unstale":"2024-02-19T14:55:37.969867+0000","last_undegraded":"2024-02-19T14:55:37.969867+0000","last_fullsized":"2024-02-19T14:55:37.969867+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:43.420537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959865+0000","last_change":"2024-02-19T14:55:37.959865+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959865+0000","last_undegraded":"2024-02-19T14:55:37.959865+0000","last_fullsized":"2024-02-19T14:55:37.959865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970634+0000","last_change":"2024-02-19T14:55:37.968907+0000","last_active":"2024-02-19T14:55:37.970634+0000","last_peered":"2024-02-19T14:55:37.970634+0000","last_clean":"2024-02-19T14:55:37.970634+0000","last_became_active":"2024-02-19T14:55:37.968628+0000","last_became_peered":"2024-02-19T14:55:37.968628+0000","last_unstale":"2024-02-19T14:55:37.970634+0000","last_undegraded":"2024-02-19T14:55:37.970634+0000","last_fullsized":"2024-02-19T14:55:37.970634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956669+0000","last_change":"2024-02-19T14:55:37.956669+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956669+0000","last_undegraded":"2024-02-19T14:55:37.956669+0000","last_fullsized":"2024-02-19T14:55:37.956669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972073+0000","last_change":"2024-02-19T14:55:37.970491+0000","last_active":"2024-02-19T14:55:37.972073+0000","last_peered":"2024-02-19T14:55:37.972073+0000","last_clean":"2024-02-19T14:55:37.972073+0000","last_became_active":"2024-02-19T14:55:37.970198+0000","last_became_peered":"2024-02-19T14:55:37.970198+0000","last_unstale":"2024-02-19T14:55:37.972073+0000","last_undegraded":"2024-02-19T14:55:37.972073+0000","last_fullsized":"2024-02-19T14:55:37.972073+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961151+0000","last_change":"2024-02-19T14:55:37.961151+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961151+0000","last_undegraded":"2024-02-19T14:55:37.961151+0000","last_fullsized":"2024-02-19T14:55:37.961151+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972058+0000","last_change":"2024-02-19T14:55:37.970614+0000","last_active":"2024-02-19T14:55:37.972058+0000","last_peered":"2024-02-19T14:55:37.972058+0000","last_clean":"2024-02-19T14:55:37.972058+0000","last_became_active":"2024-02-19T14:55:37.970319+0000","last_became_peered":"2024-02-19T14:55:37.970319+0000","last_unstale":"2024-02-19T14:55:37.972058+0000","last_undegraded":"2024-02-19T14:55:37.972058+0000","last_fullsized":"2024-02-19T14:55:37.972058+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:54.859424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960541+0000","last_change":"2024-02-19T14:55:37.960541+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960541+0000","last_undegraded":"2024-02-19T14:55:37.960541+0000","last_fullsized":"2024-02-19T14:55:37.960541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970350+0000","last_change":"2024-02-19T14:55:37.968398+0000","last_active":"2024-02-19T14:55:37.970350+0000","last_peered":"2024-02-19T14:55:37.970350+0000","last_clean":"2024-02-19T14:55:37.970350+0000","last_became_active":"2024-02-19T14:55:37.967723+0000","last_became_peered":"2024-02-19T14:55:37.967723+0000","last_unstale":"2024-02-19T14:55:37.970350+0000","last_undegraded":"2024-02-19T14:55:37.970350+0000","last_fullsized":"2024-02-19T14:55:37.970350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958633+0000","last_change":"2024-02-19T14:55:37.958633+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958633+0000","last_undegraded":"2024-02-19T14:55:37.958633+0000","last_fullsized":"2024-02-19T14:55:37.958633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973485+0000","last_change":"2024-02-19T14:55:37.968061+0000","last_active":"2024-02-19T14:55:37.973485+0000","last_peered":"2024-02-19T14:55:37.973485+0000","last_clean":"2024-02-19T14:55:37.973485+0000","last_became_active":"2024-02-19T14:55:37.967758+0000","last_became_peered":"2024-02-19T14:55:37.967758+0000","last_unstale":"2024-02-19T14:55:37.973485+0000","last_undegraded":"2024-02-19T14:55:37.973485+0000","last_fullsized":"2024-02-19T14:55:37.973485+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957115+0000","last_change":"2024-02-19T14:55:37.957115+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957115+0000","last_undegraded":"2024-02-19T14:55:37.957115+0000","last_fullsized":"2024-02-19T14:55:37.957115+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.976536+0000","last_change":"2024-02-19T14:55:37.972341+0000","last_active":"2024-02-19T14:55:37.976536+0000","last_peered":"2024-02-19T14:55:37.976536+0000","last_clean":"2024-02-19T14:55:37.976536+0000","last_became_active":"2024-02-19T14:55:37.971999+0000","last_became_peered":"2024-02-19T14:55:37.971999+0000","last_unstale":"2024-02-19T14:55:37.976536+0000","last_undegraded":"2024-02-19T14:55:37.976536+0000","last_fullsized":"2024-02-19T14:55:37.976536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:10.347679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960567+0000","last_change":"2024-02-19T14:55:37.960567+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960567+0000","last_undegraded":"2024-02-19T14:55:37.960567+0000","last_fullsized":"2024-02-19T14:55:37.960567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972094+0000","last_change":"2024-02-19T14:55:37.969752+0000","last_active":"2024-02-19T14:55:37.972094+0000","last_peered":"2024-02-19T14:55:37.972094+0000","last_clean":"2024-02-19T14:55:37.972094+0000","last_became_active":"2024-02-19T14:55:37.969493+0000","last_became_peered":"2024-02-19T14:55:37.969493+0000","last_unstale":"2024-02-19T14:55:37.972094+0000","last_undegraded":"2024-02-19T14:55:37.972094+0000","last_fullsized":"2024-02-19T14:55:37.972094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:04.496898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.964917+0000","last_change":"2024-02-19T14:55:37.964917+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.964917+0000","last_undegraded":"2024-02-19T14:55:37.964917+0000","last_fullsized":"2024-02-19T14:55:37.964917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969023+0000","last_change":"2024-02-19T14:55:37.967599+0000","last_active":"2024-02-19T14:55:37.969023+0000","last_peered":"2024-02-19T14:55:37.969023+0000","last_clean":"2024-02-19T14:55:37.969023+0000","last_became_active":"2024-02-19T14:55:37.967115+0000","last_became_peered":"2024-02-19T14:55:37.967115+0000","last_unstale":"2024-02-19T14:55:37.969023+0000","last_undegraded":"2024-02-19T14:55:37.969023+0000","last_fullsized":"2024-02-19T14:55:37.969023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961069+0000","last_change":"2024-02-19T14:55:37.961069+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961069+0000","last_undegraded":"2024-02-19T14:55:37.961069+0000","last_fullsized":"2024-02-19T14:55:37.961069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967739+0000","last_change":"2024-02-19T14:55:37.966403+0000","last_active":"2024-02-19T14:55:37.967739+0000","last_peered":"2024-02-19T14:55:37.967739+0000","last_clean":"2024-02-19T14:55:37.967739+0000","last_became_active":"2024-02-19T14:55:37.966038+0000","last_became_peered":"2024-02-19T14:55:37.966038+0000","last_unstale":"2024-02-19T14:55:37.967739+0000","last_undegraded":"2024-02-19T14:55:37.967739+0000","last_fullsized":"2024-02-19T14:55:37.967739+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956739+0000","last_change":"2024-02-19T14:55:37.956739+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956739+0000","last_undegraded":"2024-02-19T14:55:37.956739+0000","last_fullsized":"2024-02-19T14:55:37.956739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975620+0000","last_change":"2024-02-19T14:55:37.974443+0000","last_active":"2024-02-19T14:55:37.975620+0000","last_peered":"2024-02-19T14:55:37.975620+0000","last_clean":"2024-02-19T14:55:37.975620+0000","last_became_active":"2024-02-19T14:55:37.974158+0000","last_became_peered":"2024-02-19T14:55:37.974158+0000","last_unstale":"2024-02-19T14:55:37.975620+0000","last_undegraded":"2024-02-19T14:55:37.975620+0000","last_fullsized":"2024-02-19T14:55:37.975620+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:04.393457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960714+0000","last_change":"2024-02-19T14:55:37.960714+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960714+0000","last_undegraded":"2024-02-19T14:55:37.960714+0000","last_fullsized":"2024-02-19T14:55:37.960714+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.966097+0000","last_change":"2024-02-19T14:55:37.958882+0000","last_active":"2024-02-19T14:55:37.966097+0000","last_peered":"2024-02-19T14:55:37.966097+0000","last_clean":"2024-02-19T14:55:37.966097+0000","last_became_active":"2024-02-19T14:55:37.958615+0000","last_became_peered":"2024-02-19T14:55:37.958615+0000","last_unstale":"2024-02-19T14:55:37.966097+0000","last_undegraded":"2024-02-19T14:55:37.966097+0000","last_fullsized":"2024-02-19T14:55:37.966097+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:25.413669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959715+0000","last_change":"2024-02-19T14:55:37.959715+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959715+0000","last_undegraded":"2024-02-19T14:55:37.959715+0000","last_fullsized":"2024-02-19T14:55:37.959715+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967023+0000","last_change":"2024-02-19T14:55:37.962848+0000","last_active":"2024-02-19T14:55:37.967023+0000","last_peered":"2024-02-19T14:55:37.967023+0000","last_clean":"2024-02-19T14:55:37.967023+0000","last_became_active":"2024-02-19T14:55:37.962569+0000","last_became_peered":"2024-02-19T14:55:37.962569+0000","last_unstale":"2024-02-19T14:55:37.967023+0000","last_undegraded":"2024-02-19T14:55:37.967023+0000","last_fullsized":"2024-02-19T14:55:37.967023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:20.244221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959539+0000","last_change":"2024-02-19T14:55:37.959539+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959539+0000","last_undegraded":"2024-02-19T14:55:37.959539+0000","last_fullsized":"2024-02-19T14:55:37.959539+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979462+0000","last_change":"2024-02-19T14:55:37.976122+0000","last_active":"2024-02-19T14:55:37.979462+0000","last_peered":"2024-02-19T14:55:37.979462+0000","last_clean":"2024-02-19T14:55:37.979462+0000","last_became_active":"2024-02-19T14:55:37.975685+0000","last_became_peered":"2024-02-19T14:55:37.975685+0000","last_unstale":"2024-02-19T14:55:37.979462+0000","last_undegraded":"2024-02-19T14:55:37.979462+0000","last_fullsized":"2024-02-19T14:55:37.979462+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:07.357078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967011+0000","last_change":"2024-02-19T14:55:37.967011+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967011+0000","last_undegraded":"2024-02-19T14:55:37.967011+0000","last_fullsized":"2024-02-19T14:55:37.967011+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970364+0000","last_change":"2024-02-19T14:55:37.968205+0000","last_active":"2024-02-19T14:55:37.970364+0000","last_peered":"2024-02-19T14:55:37.970364+0000","last_clean":"2024-02-19T14:55:37.970364+0000","last_became_active":"2024-02-19T14:55:37.967904+0000","last_became_peered":"2024-02-19T14:55:37.967904+0000","last_unstale":"2024-02-19T14:55:37.970364+0000","last_undegraded":"2024-02-19T14:55:37.970364+0000","last_fullsized":"2024-02-19T14:55:37.970364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962297+0000","last_change":"2024-02-19T14:55:37.962297+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962297+0000","last_undegraded":"2024-02-19T14:55:37.962297+0000","last_fullsized":"2024-02-19T14:55:37.962297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975350+0000","last_change":"2024-02-19T14:55:37.969453+0000","last_active":"2024-02-19T14:55:37.975350+0000","last_peered":"2024-02-19T14:55:37.975350+0000","last_clean":"2024-02-19T14:55:37.975350+0000","last_became_active":"2024-02-19T14:55:37.968536+0000","last_became_peered":"2024-02-19T14:55:37.968536+0000","last_unstale":"2024-02-19T14:55:37.975350+0000","last_undegraded":"2024-02-19T14:55:37.975350+0000","last_fullsized":"2024-02-19T14:55:37.975350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":337,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.961256+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:37.961256+0000","last_peered":"2024-02-19T14:55:37.961256+0000","last_clean":"2024-02-19T14:55:37.961256+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:37.961256+0000","last_undegraded":"2024-02-19T14:55:37.961256+0000","last_fullsized":"2024-02-19T14:55:37.961256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.443}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.433},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.624}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.625},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.754},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.80700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.441},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.25900000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.505},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.54700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.314},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.217},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.498},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.315},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:42.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:42.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.151+0000 7f12b5683640 1 Processor -- start 2024-02-19T14:55:42.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.151+0000 7f12b5683640 1 -- start start 2024-02-19T14:55:42.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12b5683640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12b00c5130 0x7f12b00c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12b5683640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12b00c5510 0x7f12b01559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12b5683640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00c3cf0 con 0x7f12b0158f80 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12b5683640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b0158b60 con 0x7f12b00c5130 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12b5683640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b0155f00 con 0x7f12b00c5510 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12b00c5130 0x7f12b00c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12b00c5130 0x7f12b00c30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40444/0 (socket says 172.21.15.78:40444) 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 -- 172.21.15.78:0/3067239501 learned_addr learned my addr 172.21.15.78:0/3067239501 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12a67fc640 1 --1- 172.21.15.78:0/3067239501 >> v1:172.21.15.78:6789/0 conn(0x7f12b0158f80 0x7f12b00c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60624/0 (socket says 172.21.15.78:60624) 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12af7fe640 1 --2- 172.21.15.78:0/3067239501 >> v2:172.21.15.150:3300/0 conn(0x7f12b00c5510 0x7f12b01559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 -- 172.21.15.78:0/3067239501 >> v2:172.21.15.150:3300/0 conn(0x7f12b00c5510 msgr2=0x7f12b01559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 --2- 172.21.15.78:0/3067239501 >> v2:172.21.15.150:3300/0 conn(0x7f12b00c5510 0x7f12b01559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 -- 172.21.15.78:0/3067239501 >> v1:172.21.15.78:6789/0 conn(0x7f12b0158f80 legacy=0x7f12b00c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12aeffd640 1 -- 172.21.15.78:0/3067239501 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0159400 con 0x7f12b00c5130 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12ae7fc640 1 -- 172.21.15.78:0/3067239501 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597761829 0 0) 0x7f12b00c3cf0 con 0x7f12b0158f80 2024-02-19T14:55:42.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.152+0000 7f12af7fe640 1 --2- 172.21.15.78:0/3067239501 >> v2:172.21.15.150:3300/0 conn(0x7f12b00c5510 0x7f12b01559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:42.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.153+0000 7f12aeffd640 1 --2- 172.21.15.78:0/3067239501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5130 0x7f12b00c30a0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f12b00c8670 tx=0x7f12a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9d8445f4386f4a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.153+0000 7f12ae7fc640 1 -- 172.21.15.78:0/3067239501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0030860 con 0x7f12b00c5130 2024-02-19T14:55:42.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.153+0000 7f12ae7fc640 1 -- 172.21.15.78:0/3067239501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a00309c0 con 0x7f12b00c5130 2024-02-19T14:55:42.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.153+0000 7f12ae7fc640 1 -- 172.21.15.78:0/3067239501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0030d50 con 0x7f12b00c5130 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.153+0000 7f12b5683640 1 -- 172.21.15.78:0/3067239501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5130 msgr2=0x7f12b00c30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.153+0000 7f12b5683640 1 --2- 172.21.15.78:0/3067239501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5130 0x7f12b00c30a0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f12b00c8670 tx=0x7f12a002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.154+0000 7f12b5683640 1 -- 172.21.15.78:0/3067239501 shutdown_connections 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.154+0000 7f12b5683640 1 --2- 172.21.15.78:0/3067239501 >> v2:172.21.15.150:3300/0 conn(0x7f12b00c5510 0x7f12b01559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.154+0000 7f12b5683640 1 --2- 172.21.15.78:0/3067239501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5130 0x7f12b00c30a0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.154+0000 7f12b5683640 1 -- 172.21.15.78:0/3067239501 >> 172.21.15.78:0/3067239501 conn(0x7f12b006b4b0 msgr2=0x7f12b00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.154+0000 7f12b5683640 1 -- 172.21.15.78:0/3067239501 shutdown_connections 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.154+0000 7f12b5683640 1 -- 172.21.15.78:0/3067239501 wait complete. 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 Processor -- start 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 -- start start 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b00c5130 0x7f12b0157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 0x7f12b01e7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b0158f80 0x7f12b01ede70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00a8a40 con 0x7f12b00c5510 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b01ed9e0 con 0x7f12b00c5130 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.155+0000 7f12b5683640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b01ea950 con 0x7f12b0158f80 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 0x7f12b01e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 0x7f12b01e7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40448/0 (socket says 172.21.15.78:40448) 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 -- 172.21.15.78:0/2342430931 learned_addr learned my addr 172.21.15.78:0/2342430931 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 -- 172.21.15.78:0/2342430931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b0158f80 msgr2=0x7f12b01ede70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b0158f80 0x7f12b01ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 -- 172.21.15.78:0/2342430931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b00c5130 msgr2=0x7f12b0157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b00c5130 0x7f12b0157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 -- 172.21.15.78:0/2342430931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a00094d0 con 0x7f12b00c5510 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a67fc640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 0x7f12b01e7860 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f129800c430 tx=0x7f129800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12980074e0 con 0x7f12b00c5510 2024-02-19T14:55:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298007660 con 0x7f12b00c5510 2024-02-19T14:55:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b01eac30 con 0x7f12b00c5510 2024-02-19T14:55:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298004a20 con 0x7f12b00c5510 2024-02-19T14:55:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.156+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b01eb1f0 con 0x7f12b00c5510 2024-02-19T14:55:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.157+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b01587b0 con 0x7f12b00c5510 2024-02-19T14:55:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.158+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1298004b80 con 0x7f12b00c5510 2024-02-19T14:55:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.158+0000 7f12a7fff640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129006b810 0x7f129006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.158+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f129808a180 con 0x7f12b00c5510 2024-02-19T14:55:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.158+0000 7f12aeffd640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129006b810 0x7f129006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.159+0000 7f12aeffd640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129006b810 0x7f129006dcd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f12a0036000 tx=0x7f12a003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.162+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1298054a10 con 0x7f12b00c5510 2024-02-19T14:55:42.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.336+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12b0158530 con 0x7f129006b810 2024-02-19T14:55:42.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.342+0000 7f12a7fff640 1 -- 172.21.15.78:0/2342430931 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284417 (secure 0 0 0) 0x7f12a0030bb0 con 0x7f129006b810 2024-02-19T14:55:42.348 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:42.350 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129006b810 msgr2=0x7f129006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129006b810 0x7f129006dcd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f12a0036000 tx=0x7f12a003b040 comp rx=0 tx=0).stop 2024-02-19T14:55:42.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 msgr2=0x7f12b01e7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 0x7f12b01e7860 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f129800c430 tx=0x7f129800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 shutdown_connections 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f129006b810 0x7f129006dcd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b00c5510 0x7f12b01e7860 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.345+0000 7f12b5683640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12b0158f80 0x7f12b01ede70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.346+0000 7f12b5683640 1 --2- 172.21.15.78:0/2342430931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b00c5130 0x7f12b0157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.346+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 >> 172.21.15.78:0/2342430931 conn(0x7f12b006b4b0 msgr2=0x7f12b00c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.346+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 shutdown_connections 2024-02-19T14:55:42.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.346+0000 7f12b5683640 1 -- 172.21.15.78:0/2342430931 wait complete. 2024-02-19T14:55:42.359 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-02-19T14:55:42.102662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257000,"kb_used_data":11712,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754717720,"statfs":{"total":773094113280,"available":772830945280,"internally_reserved":0,"allocated":11993088,"data_stored":6412102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.152492"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955144+0000","last_change":"2024-02-19T14:55:37.955144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955144+0000","last_undegraded":"2024-02-19T14:55:37.955144+0000","last_fullsized":"2024-02-19T14:55:37.955144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962463+0000","last_change":"2024-02-19T14:55:37.962463+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962463+0000","last_undegraded":"2024-02-19T14:55:37.962463+0000","last_fullsized":"2024-02-19T14:55:37.962463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958706+0000","last_change":"2024-02-19T14:55:37.958706+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958706+0000","last_undegraded":"2024-02-19T14:55:37.958706+0000","last_fullsized":"2024-02-19T14:55:37.958706+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961496+0000","last_change":"2024-02-19T14:55:37.961496+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961496+0000","last_undegraded":"2024-02-19T14:55:37.961496+0000","last_fullsized":"2024-02-19T14:55:37.961496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956553+0000","last_change":"2024-02-19T14:55:37.956553+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956553+0000","last_undegraded":"2024-02-19T14:55:37.956553+0000","last_fullsized":"2024-02-19T14:55:37.956553+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959221+0000","last_change":"2024-02-19T14:55:37.959221+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959221+0000","last_undegraded":"2024-02-19T14:55:37.959221+0000","last_fullsized":"2024-02-19T14:55:37.959221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958511+0000","last_change":"2024-02-19T14:55:37.958511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958511+0000","last_undegraded":"2024-02-19T14:55:37.958511+0000","last_fullsized":"2024-02-19T14:55:37.958511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960041+0000","last_change":"2024-02-19T14:55:37.960041+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960041+0000","last_undegraded":"2024-02-19T14:55:37.960041+0000","last_fullsized":"2024-02-19T14:55:37.960041+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965608+0000","last_change":"2024-02-19T14:55:37.965608+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965608+0000","last_undegraded":"2024-02-19T14:55:37.965608+0000","last_fullsized":"2024-02-19T14:55:37.965608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956058+0000","last_change":"2024-02-19T14:55:37.956058+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956058+0000","last_undegraded":"2024-02-19T14:55:37.956058+0000","last_fullsized":"2024-02-19T14:55:37.956058+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961425+0000","last_change":"2024-02-19T14:55:37.961425+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961425+0000","last_undegraded":"2024-02-19T14:55:37.961425+0000","last_fullsized":"2024-02-19T14:55:37.961425+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967924+0000","last_change":"2024-02-19T14:55:37.967924+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967924+0000","last_undegraded":"2024-02-19T14:55:37.967924+0000","last_fullsized":"2024-02-19T14:55:37.967924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960544+0000","last_change":"2024-02-19T14:55:37.960544+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960544+0000","last_undegraded":"2024-02-19T14:55:37.960544+0000","last_fullsized":"2024-02-19T14:55:37.960544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956259+0000","last_change":"2024-02-19T14:55:37.956259+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956259+0000","last_undegraded":"2024-02-19T14:55:37.956259+0000","last_fullsized":"2024-02-19T14:55:37.956259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961548+0000","last_change":"2024-02-19T14:55:37.961548+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961548+0000","last_undegraded":"2024-02-19T14:55:37.961548+0000","last_fullsized":"2024-02-19T14:55:37.961548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965387+0000","last_change":"2024-02-19T14:55:37.965387+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965387+0000","last_undegraded":"2024-02-19T14:55:37.965387+0000","last_fullsized":"2024-02-19T14:55:37.965387+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962059+0000","last_change":"2024-02-19T14:55:37.962059+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962059+0000","last_undegraded":"2024-02-19T14:55:37.962059+0000","last_fullsized":"2024-02-19T14:55:37.962059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959638+0000","last_change":"2024-02-19T14:55:37.959638+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959638+0000","last_undegraded":"2024-02-19T14:55:37.959638+0000","last_fullsized":"2024-02-19T14:55:37.959638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958119+0000","last_change":"2024-02-19T14:55:37.958119+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958119+0000","last_undegraded":"2024-02-19T14:55:37.958119+0000","last_fullsized":"2024-02-19T14:55:37.958119+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962661+0000","last_change":"2024-02-19T14:55:37.962661+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962661+0000","last_undegraded":"2024-02-19T14:55:37.962661+0000","last_fullsized":"2024-02-19T14:55:37.962661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955835+0000","last_change":"2024-02-19T14:55:37.955835+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955835+0000","last_undegraded":"2024-02-19T14:55:37.955835+0000","last_fullsized":"2024-02-19T14:55:37.955835+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960561+0000","last_change":"2024-02-19T14:55:37.960561+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960561+0000","last_undegraded":"2024-02-19T14:55:37.960561+0000","last_fullsized":"2024-02-19T14:55:37.960561+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958144+0000","last_change":"2024-02-19T14:55:37.958144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958144+0000","last_undegraded":"2024-02-19T14:55:37.958144+0000","last_fullsized":"2024-02-19T14:55:37.958144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958449+0000","last_change":"2024-02-19T14:55:37.958449+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958449+0000","last_undegraded":"2024-02-19T14:55:37.958449+0000","last_fullsized":"2024-02-19T14:55:37.958449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957844+0000","last_change":"2024-02-19T14:55:37.957844+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957844+0000","last_undegraded":"2024-02-19T14:55:37.957844+0000","last_fullsized":"2024-02-19T14:55:37.957844+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959212+0000","last_change":"2024-02-19T14:55:37.959212+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959212+0000","last_undegraded":"2024-02-19T14:55:37.959212+0000","last_fullsized":"2024-02-19T14:55:37.959212+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961013+0000","last_change":"2024-02-19T14:55:37.961013+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961013+0000","last_undegraded":"2024-02-19T14:55:37.961013+0000","last_fullsized":"2024-02-19T14:55:37.961013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957840+0000","last_change":"2024-02-19T14:55:37.957840+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957840+0000","last_undegraded":"2024-02-19T14:55:37.957840+0000","last_fullsized":"2024-02-19T14:55:37.957840+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957510+0000","last_change":"2024-02-19T14:55:37.957510+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957510+0000","last_undegraded":"2024-02-19T14:55:37.957510+0000","last_fullsized":"2024-02-19T14:55:37.957510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956289+0000","last_change":"2024-02-19T14:55:37.956289+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956289+0000","last_undegraded":"2024-02-19T14:55:37.956289+0000","last_fullsized":"2024-02-19T14:55:37.956289+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.966383+0000","last_change":"2024-02-19T14:55:37.966383+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.966383+0000","last_undegraded":"2024-02-19T14:55:37.966383+0000","last_fullsized":"2024-02-19T14:55:37.966383+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969047+0000","last_change":"2024-02-19T14:55:37.967453+0000","last_active":"2024-02-19T14:55:37.969047+0000","last_peered":"2024-02-19T14:55:37.969047+0000","last_clean":"2024-02-19T14:55:37.969047+0000","last_became_active":"2024-02-19T14:55:37.967091+0000","last_became_peered":"2024-02-19T14:55:37.967091+0000","last_unstale":"2024-02-19T14:55:37.969047+0000","last_undegraded":"2024-02-19T14:55:37.969047+0000","last_fullsized":"2024-02-19T14:55:37.969047+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962010+0000","last_change":"2024-02-19T14:55:37.962010+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962010+0000","last_undegraded":"2024-02-19T14:55:37.962010+0000","last_fullsized":"2024-02-19T14:55:37.962010+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977066+0000","last_change":"2024-02-19T14:55:37.974585+0000","last_active":"2024-02-19T14:55:37.977066+0000","last_peered":"2024-02-19T14:55:37.977066+0000","last_clean":"2024-02-19T14:55:37.977066+0000","last_became_active":"2024-02-19T14:55:37.974280+0000","last_became_peered":"2024-02-19T14:55:37.974280+0000","last_unstale":"2024-02-19T14:55:37.977066+0000","last_undegraded":"2024-02-19T14:55:37.977066+0000","last_fullsized":"2024-02-19T14:55:37.977066+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:15.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957606+0000","last_change":"2024-02-19T14:55:37.957606+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957606+0000","last_undegraded":"2024-02-19T14:55:37.957606+0000","last_fullsized":"2024-02-19T14:55:37.957606+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973784+0000","last_change":"2024-02-19T14:55:37.971928+0000","last_active":"2024-02-19T14:55:37.973784+0000","last_peered":"2024-02-19T14:55:37.973784+0000","last_clean":"2024-02-19T14:55:37.973784+0000","last_became_active":"2024-02-19T14:55:37.971646+0000","last_became_peered":"2024-02-19T14:55:37.971646+0000","last_unstale":"2024-02-19T14:55:37.973784+0000","last_undegraded":"2024-02-19T14:55:37.973784+0000","last_fullsized":"2024-02-19T14:55:37.973784+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:38.067553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959582+0000","last_change":"2024-02-19T14:55:37.959582+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959582+0000","last_undegraded":"2024-02-19T14:55:37.959582+0000","last_fullsized":"2024-02-19T14:55:37.959582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977059+0000","last_change":"2024-02-19T14:55:37.973007+0000","last_active":"2024-02-19T14:55:37.977059+0000","last_peered":"2024-02-19T14:55:37.977059+0000","last_clean":"2024-02-19T14:55:37.977059+0000","last_became_active":"2024-02-19T14:55:37.972480+0000","last_became_peered":"2024-02-19T14:55:37.972480+0000","last_unstale":"2024-02-19T14:55:37.977059+0000","last_undegraded":"2024-02-19T14:55:37.977059+0000","last_fullsized":"2024-02-19T14:55:37.977059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.968277+0000","last_change":"2024-02-19T14:55:37.968277+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.968277+0000","last_undegraded":"2024-02-19T14:55:37.968277+0000","last_fullsized":"2024-02-19T14:55:37.968277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.982286+0000","last_change":"2024-02-19T14:55:37.975968+0000","last_active":"2024-02-19T14:55:37.982286+0000","last_peered":"2024-02-19T14:55:37.982286+0000","last_clean":"2024-02-19T14:55:37.982286+0000","last_became_active":"2024-02-19T14:55:37.975693+0000","last_became_peered":"2024-02-19T14:55:37.975693+0000","last_unstale":"2024-02-19T14:55:37.982286+0000","last_undegraded":"2024-02-19T14:55:37.982286+0000","last_fullsized":"2024-02-19T14:55:37.982286+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955143+0000","last_change":"2024-02-19T14:55:37.955143+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955143+0000","last_undegraded":"2024-02-19T14:55:37.955143+0000","last_fullsized":"2024-02-19T14:55:37.955143+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977070+0000","last_change":"2024-02-19T14:55:37.975370+0000","last_active":"2024-02-19T14:55:37.977070+0000","last_peered":"2024-02-19T14:55:37.977070+0000","last_clean":"2024-02-19T14:55:37.977070+0000","last_became_active":"2024-02-19T14:55:37.974133+0000","last_became_peered":"2024-02-19T14:55:37.974133+0000","last_unstale":"2024-02-19T14:55:37.977070+0000","last_undegraded":"2024-02-19T14:55:37.977070+0000","last_fullsized":"2024-02-19T14:55:37.977070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:31.145774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959131+0000","last_change":"2024-02-19T14:55:37.959131+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959131+0000","last_undegraded":"2024-02-19T14:55:37.959131+0000","last_fullsized":"2024-02-19T14:55:37.959131+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967695+0000","last_change":"2024-02-19T14:55:37.965578+0000","last_active":"2024-02-19T14:55:37.967695+0000","last_peered":"2024-02-19T14:55:37.967695+0000","last_clean":"2024-02-19T14:55:37.967695+0000","last_became_active":"2024-02-19T14:55:37.965063+0000","last_became_peered":"2024-02-19T14:55:37.965063+0000","last_unstale":"2024-02-19T14:55:37.967695+0000","last_undegraded":"2024-02-19T14:55:37.967695+0000","last_fullsized":"2024-02-19T14:55:37.967695+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967476+0000","last_change":"2024-02-19T14:55:37.967476+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967476+0000","last_undegraded":"2024-02-19T14:55:37.967476+0000","last_fullsized":"2024-02-19T14:55:37.967476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979469+0000","last_change":"2024-02-19T14:55:37.976412+0000","last_active":"2024-02-19T14:55:37.979469+0000","last_peered":"2024-02-19T14:55:37.979469+0000","last_clean":"2024-02-19T14:55:37.979469+0000","last_became_active":"2024-02-19T14:55:37.975831+0000","last_became_peered":"2024-02-19T14:55:37.975831+0000","last_unstale":"2024-02-19T14:55:37.979469+0000","last_undegraded":"2024-02-19T14:55:37.979469+0000","last_fullsized":"2024-02-19T14:55:37.979469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:55:53.499236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959395+0000","last_change":"2024-02-19T14:55:37.959395+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959395+0000","last_undegraded":"2024-02-19T14:55:37.959395+0000","last_fullsized":"2024-02-19T14:55:37.959395+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.971160+0000","last_change":"2024-02-19T14:55:37.969860+0000","last_active":"2024-02-19T14:55:37.971160+0000","last_peered":"2024-02-19T14:55:37.971160+0000","last_clean":"2024-02-19T14:55:37.971160+0000","last_became_active":"2024-02-19T14:55:37.969561+0000","last_became_peered":"2024-02-19T14:55:37.969561+0000","last_unstale":"2024-02-19T14:55:37.971160+0000","last_undegraded":"2024-02-19T14:55:37.971160+0000","last_fullsized":"2024-02-19T14:55:37.971160+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956550+0000","last_change":"2024-02-19T14:55:37.956550+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956550+0000","last_undegraded":"2024-02-19T14:55:37.956550+0000","last_fullsized":"2024-02-19T14:55:37.956550+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973119+0000","last_change":"2024-02-19T14:55:37.972249+0000","last_active":"2024-02-19T14:55:37.973119+0000","last_peered":"2024-02-19T14:55:37.973119+0000","last_clean":"2024-02-19T14:55:37.973119+0000","last_became_active":"2024-02-19T14:55:37.971802+0000","last_became_peered":"2024-02-19T14:55:37.971802+0000","last_unstale":"2024-02-19T14:55:37.973119+0000","last_undegraded":"2024-02-19T14:55:37.973119+0000","last_fullsized":"2024-02-19T14:55:37.973119+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961622+0000","last_change":"2024-02-19T14:55:37.961622+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961622+0000","last_undegraded":"2024-02-19T14:55:37.961622+0000","last_fullsized":"2024-02-19T14:55:37.961622+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970044+0000","last_change":"2024-02-19T14:55:37.968601+0000","last_active":"2024-02-19T14:55:37.970044+0000","last_peered":"2024-02-19T14:55:37.970044+0000","last_clean":"2024-02-19T14:55:37.970044+0000","last_became_active":"2024-02-19T14:55:37.968117+0000","last_became_peered":"2024-02-19T14:55:37.968117+0000","last_unstale":"2024-02-19T14:55:37.970044+0000","last_undegraded":"2024-02-19T14:55:37.970044+0000","last_fullsized":"2024-02-19T14:55:37.970044+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958050+0000","last_change":"2024-02-19T14:55:37.958050+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958050+0000","last_undegraded":"2024-02-19T14:55:37.958050+0000","last_fullsized":"2024-02-19T14:55:37.958050+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972821+0000","last_change":"2024-02-19T14:55:37.968678+0000","last_active":"2024-02-19T14:55:37.972821+0000","last_peered":"2024-02-19T14:55:37.972821+0000","last_clean":"2024-02-19T14:55:37.972821+0000","last_became_active":"2024-02-19T14:55:37.968384+0000","last_became_peered":"2024-02-19T14:55:37.968384+0000","last_unstale":"2024-02-19T14:55:37.972821+0000","last_undegraded":"2024-02-19T14:55:37.972821+0000","last_fullsized":"2024-02-19T14:55:37.972821+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:19.012872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960027+0000","last_change":"2024-02-19T14:55:37.960027+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960027+0000","last_undegraded":"2024-02-19T14:55:37.960027+0000","last_fullsized":"2024-02-19T14:55:37.960027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975368+0000","last_change":"2024-02-19T14:55:37.969264+0000","last_active":"2024-02-19T14:55:37.975368+0000","last_peered":"2024-02-19T14:55:37.975368+0000","last_clean":"2024-02-19T14:55:37.975368+0000","last_became_active":"2024-02-19T14:55:37.968958+0000","last_became_peered":"2024-02-19T14:55:37.968958+0000","last_unstale":"2024-02-19T14:55:37.975368+0000","last_undegraded":"2024-02-19T14:55:37.975368+0000","last_fullsized":"2024-02-19T14:55:37.975368+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967940+0000","last_change":"2024-02-19T14:55:37.967940+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967940+0000","last_undegraded":"2024-02-19T14:55:37.967940+0000","last_fullsized":"2024-02-19T14:55:37.967940+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975543+0000","last_change":"2024-02-19T14:55:37.970686+0000","last_active":"2024-02-19T14:55:37.975543+0000","last_peered":"2024-02-19T14:55:37.975543+0000","last_clean":"2024-02-19T14:55:37.975543+0000","last_became_active":"2024-02-19T14:55:37.970405+0000","last_became_peered":"2024-02-19T14:55:37.970405+0000","last_unstale":"2024-02-19T14:55:37.975543+0000","last_undegraded":"2024-02-19T14:55:37.975543+0000","last_fullsized":"2024-02-19T14:55:37.975543+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957316+0000","last_change":"2024-02-19T14:55:37.957316+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957316+0000","last_undegraded":"2024-02-19T14:55:37.957316+0000","last_fullsized":"2024-02-19T14:55:37.957316+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975423+0000","last_change":"2024-02-19T14:55:37.969681+0000","last_active":"2024-02-19T14:55:37.975423+0000","last_peered":"2024-02-19T14:55:37.975423+0000","last_clean":"2024-02-19T14:55:37.975423+0000","last_became_active":"2024-02-19T14:55:37.969413+0000","last_became_peered":"2024-02-19T14:55:37.969413+0000","last_unstale":"2024-02-19T14:55:37.975423+0000","last_undegraded":"2024-02-19T14:55:37.975423+0000","last_fullsized":"2024-02-19T14:55:37.975423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957511+0000","last_change":"2024-02-19T14:55:37.957511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957511+0000","last_undegraded":"2024-02-19T14:55:37.957511+0000","last_fullsized":"2024-02-19T14:55:37.957511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977083+0000","last_change":"2024-02-19T14:55:37.973068+0000","last_active":"2024-02-19T14:55:37.977083+0000","last_peered":"2024-02-19T14:55:37.977083+0000","last_clean":"2024-02-19T14:55:37.977083+0000","last_became_active":"2024-02-19T14:55:37.972754+0000","last_became_peered":"2024-02-19T14:55:37.972754+0000","last_unstale":"2024-02-19T14:55:37.977083+0000","last_undegraded":"2024-02-19T14:55:37.977083+0000","last_fullsized":"2024-02-19T14:55:37.977083+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:14.587666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955566+0000","last_change":"2024-02-19T14:55:37.955566+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955566+0000","last_undegraded":"2024-02-19T14:55:37.955566+0000","last_fullsized":"2024-02-19T14:55:37.955566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969867+0000","last_change":"2024-02-19T14:55:37.963382+0000","last_active":"2024-02-19T14:55:37.969867+0000","last_peered":"2024-02-19T14:55:37.969867+0000","last_clean":"2024-02-19T14:55:37.969867+0000","last_became_active":"2024-02-19T14:55:37.963016+0000","last_became_peered":"2024-02-19T14:55:37.963016+0000","last_unstale":"2024-02-19T14:55:37.969867+0000","last_undegraded":"2024-02-19T14:55:37.969867+0000","last_fullsized":"2024-02-19T14:55:37.969867+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:43.420537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959865+0000","last_change":"2024-02-19T14:55:37.959865+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959865+0000","last_undegraded":"2024-02-19T14:55:37.959865+0000","last_fullsized":"2024-02-19T14:55:37.959865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970634+0000","last_change":"2024-02-19T14:55:37.968907+0000","last_active":"2024-02-19T14:55:37.970634+0000","last_peered":"2024-02-19T14:55:37.970634+0000","last_clean":"2024-02-19T14:55:37.970634+0000","last_became_active":"2024-02-19T14:55:37.968628+0000","last_became_peered":"2024-02-19T14:55:37.968628+0000","last_unstale":"2024-02-19T14:55:37.970634+0000","last_undegraded":"2024-02-19T14:55:37.970634+0000","last_fullsized":"2024-02-19T14:55:37.970634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956669+0000","last_change":"2024-02-19T14:55:37.956669+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956669+0000","last_undegraded":"2024-02-19T14:55:37.956669+0000","last_fullsized":"2024-02-19T14:55:37.956669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972073+0000","last_change":"2024-02-19T14:55:37.970491+0000","last_active":"2024-02-19T14:55:37.972073+0000","last_peered":"2024-02-19T14:55:37.972073+0000","last_clean":"2024-02-19T14:55:37.972073+0000","last_became_active":"2024-02-19T14:55:37.970198+0000","last_became_peered":"2024-02-19T14:55:37.970198+0000","last_unstale":"2024-02-19T14:55:37.972073+0000","last_undegraded":"2024-02-19T14:55:37.972073+0000","last_fullsized":"2024-02-19T14:55:37.972073+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961151+0000","last_change":"2024-02-19T14:55:37.961151+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961151+0000","last_undegraded":"2024-02-19T14:55:37.961151+0000","last_fullsized":"2024-02-19T14:55:37.961151+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972058+0000","last_change":"2024-02-19T14:55:37.970614+0000","last_active":"2024-02-19T14:55:37.972058+0000","last_peered":"2024-02-19T14:55:37.972058+0000","last_clean":"2024-02-19T14:55:37.972058+0000","last_became_active":"2024-02-19T14:55:37.970319+0000","last_became_peered":"2024-02-19T14:55:37.970319+0000","last_unstale":"2024-02-19T14:55:37.972058+0000","last_undegraded":"2024-02-19T14:55:37.972058+0000","last_fullsized":"2024-02-19T14:55:37.972058+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:54.859424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960541+0000","last_change":"2024-02-19T14:55:37.960541+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960541+0000","last_undegraded":"2024-02-19T14:55:37.960541+0000","last_fullsized":"2024-02-19T14:55:37.960541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970350+0000","last_change":"2024-02-19T14:55:37.968398+0000","last_active":"2024-02-19T14:55:37.970350+0000","last_peered":"2024-02-19T14:55:37.970350+0000","last_clean":"2024-02-19T14:55:37.970350+0000","last_became_active":"2024-02-19T14:55:37.967723+0000","last_became_peered":"2024-02-19T14:55:37.967723+0000","last_unstale":"2024-02-19T14:55:37.970350+0000","last_undegraded":"2024-02-19T14:55:37.970350+0000","last_fullsized":"2024-02-19T14:55:37.970350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958633+0000","last_change":"2024-02-19T14:55:37.958633+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958633+0000","last_undegraded":"2024-02-19T14:55:37.958633+0000","last_fullsized":"2024-02-19T14:55:37.958633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973485+0000","last_change":"2024-02-19T14:55:37.968061+0000","last_active":"2024-02-19T14:55:37.973485+0000","last_peered":"2024-02-19T14:55:37.973485+0000","last_clean":"2024-02-19T14:55:37.973485+0000","last_became_active":"2024-02-19T14:55:37.967758+0000","last_became_peered":"2024-02-19T14:55:37.967758+0000","last_unstale":"2024-02-19T14:55:37.973485+0000","last_undegraded":"2024-02-19T14:55:37.973485+0000","last_fullsized":"2024-02-19T14:55:37.973485+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957115+0000","last_change":"2024-02-19T14:55:37.957115+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957115+0000","last_undegraded":"2024-02-19T14:55:37.957115+0000","last_fullsized":"2024-02-19T14:55:37.957115+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.976536+0000","last_change":"2024-02-19T14:55:37.972341+0000","last_active":"2024-02-19T14:55:37.976536+0000","last_peered":"2024-02-19T14:55:37.976536+0000","last_clean":"2024-02-19T14:55:37.976536+0000","last_became_active":"2024-02-19T14:55:37.971999+0000","last_became_peered":"2024-02-19T14:55:37.971999+0000","last_unstale":"2024-02-19T14:55:37.976536+0000","last_undegraded":"2024-02-19T14:55:37.976536+0000","last_fullsized":"2024-02-19T14:55:37.976536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:10.347679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960567+0000","last_change":"2024-02-19T14:55:37.960567+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960567+0000","last_undegraded":"2024-02-19T14:55:37.960567+0000","last_fullsized":"2024-02-19T14:55:37.960567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972094+0000","last_change":"2024-02-19T14:55:37.969752+0000","last_active":"2024-02-19T14:55:37.972094+0000","last_peered":"2024-02-19T14:55:37.972094+0000","last_clean":"2024-02-19T14:55:37.972094+0000","last_became_active":"2024-02-19T14:55:37.969493+0000","last_became_peered":"2024-02-19T14:55:37.969493+0000","last_unstale":"2024-02-19T14:55:37.972094+0000","last_undegraded":"2024-02-19T14:55:37.972094+0000","last_fullsized":"2024-02-19T14:55:37.972094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:04.496898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.964917+0000","last_change":"2024-02-19T14:55:37.964917+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.964917+0000","last_undegraded":"2024-02-19T14:55:37.964917+0000","last_fullsized":"2024-02-19T14:55:37.964917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969023+0000","last_change":"2024-02-19T14:55:37.967599+0000","last_active":"2024-02-19T14:55:37.969023+0000","last_peered":"2024-02-19T14:55:37.969023+0000","last_clean":"2024-02-19T14:55:37.969023+0000","last_became_active":"2024-02-19T14:55:37.967115+0000","last_became_peered":"2024-02-19T14:55:37.967115+0000","last_unstale":"2024-02-19T14:55:37.969023+0000","last_undegraded":"2024-02-19T14:55:37.969023+0000","last_fullsized":"2024-02-19T14:55:37.969023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961069+0000","last_change":"2024-02-19T14:55:37.961069+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961069+0000","last_undegraded":"2024-02-19T14:55:37.961069+0000","last_fullsized":"2024-02-19T14:55:37.961069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967739+0000","last_change":"2024-02-19T14:55:37.966403+0000","last_active":"2024-02-19T14:55:37.967739+0000","last_peered":"2024-02-19T14:55:37.967739+0000","last_clean":"2024-02-19T14:55:37.967739+0000","last_became_active":"2024-02-19T14:55:37.966038+0000","last_became_peered":"2024-02-19T14:55:37.966038+0000","last_unstale":"2024-02-19T14:55:37.967739+0000","last_undegraded":"2024-02-19T14:55:37.967739+0000","last_fullsized":"2024-02-19T14:55:37.967739+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956739+0000","last_change":"2024-02-19T14:55:37.956739+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956739+0000","last_undegraded":"2024-02-19T14:55:37.956739+0000","last_fullsized":"2024-02-19T14:55:37.956739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975620+0000","last_change":"2024-02-19T14:55:37.974443+0000","last_active":"2024-02-19T14:55:37.975620+0000","last_peered":"2024-02-19T14:55:37.975620+0000","last_clean":"2024-02-19T14:55:37.975620+0000","last_became_active":"2024-02-19T14:55:37.974158+0000","last_became_peered":"2024-02-19T14:55:37.974158+0000","last_unstale":"2024-02-19T14:55:37.975620+0000","last_undegraded":"2024-02-19T14:55:37.975620+0000","last_fullsized":"2024-02-19T14:55:37.975620+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:04.393457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960714+0000","last_change":"2024-02-19T14:55:37.960714+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960714+0000","last_undegraded":"2024-02-19T14:55:37.960714+0000","last_fullsized":"2024-02-19T14:55:37.960714+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.966097+0000","last_change":"2024-02-19T14:55:37.958882+0000","last_active":"2024-02-19T14:55:37.966097+0000","last_peered":"2024-02-19T14:55:37.966097+0000","last_clean":"2024-02-19T14:55:37.966097+0000","last_became_active":"2024-02-19T14:55:37.958615+0000","last_became_peered":"2024-02-19T14:55:37.958615+0000","last_unstale":"2024-02-19T14:55:37.966097+0000","last_undegraded":"2024-02-19T14:55:37.966097+0000","last_fullsized":"2024-02-19T14:55:37.966097+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:25.413669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959715+0000","last_change":"2024-02-19T14:55:37.959715+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959715+0000","last_undegraded":"2024-02-19T14:55:37.959715+0000","last_fullsized":"2024-02-19T14:55:37.959715+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967023+0000","last_change":"2024-02-19T14:55:37.962848+0000","last_active":"2024-02-19T14:55:37.967023+0000","last_peered":"2024-02-19T14:55:37.967023+0000","last_clean":"2024-02-19T14:55:37.967023+0000","last_became_active":"2024-02-19T14:55:37.962569+0000","last_became_peered":"2024-02-19T14:55:37.962569+0000","last_unstale":"2024-02-19T14:55:37.967023+0000","last_undegraded":"2024-02-19T14:55:37.967023+0000","last_fullsized":"2024-02-19T14:55:37.967023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:20.244221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959539+0000","last_change":"2024-02-19T14:55:37.959539+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959539+0000","last_undegraded":"2024-02-19T14:55:37.959539+0000","last_fullsized":"2024-02-19T14:55:37.959539+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979462+0000","last_change":"2024-02-19T14:55:37.976122+0000","last_active":"2024-02-19T14:55:37.979462+0000","last_peered":"2024-02-19T14:55:37.979462+0000","last_clean":"2024-02-19T14:55:37.979462+0000","last_became_active":"2024-02-19T14:55:37.975685+0000","last_became_peered":"2024-02-19T14:55:37.975685+0000","last_unstale":"2024-02-19T14:55:37.979462+0000","last_undegraded":"2024-02-19T14:55:37.979462+0000","last_fullsized":"2024-02-19T14:55:37.979462+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:07.357078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967011+0000","last_change":"2024-02-19T14:55:37.967011+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967011+0000","last_undegraded":"2024-02-19T14:55:37.967011+0000","last_fullsized":"2024-02-19T14:55:37.967011+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970364+0000","last_change":"2024-02-19T14:55:37.968205+0000","last_active":"2024-02-19T14:55:37.970364+0000","last_peered":"2024-02-19T14:55:37.970364+0000","last_clean":"2024-02-19T14:55:37.970364+0000","last_became_active":"2024-02-19T14:55:37.967904+0000","last_became_peered":"2024-02-19T14:55:37.967904+0000","last_unstale":"2024-02-19T14:55:37.970364+0000","last_undegraded":"2024-02-19T14:55:37.970364+0000","last_fullsized":"2024-02-19T14:55:37.970364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962297+0000","last_change":"2024-02-19T14:55:37.962297+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962297+0000","last_undegraded":"2024-02-19T14:55:37.962297+0000","last_fullsized":"2024-02-19T14:55:37.962297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975350+0000","last_change":"2024-02-19T14:55:37.969453+0000","last_active":"2024-02-19T14:55:37.975350+0000","last_peered":"2024-02-19T14:55:37.975350+0000","last_clean":"2024-02-19T14:55:37.975350+0000","last_became_active":"2024-02-19T14:55:37.968536+0000","last_became_peered":"2024-02-19T14:55:37.968536+0000","last_unstale":"2024-02-19T14:55:37.975350+0000","last_undegraded":"2024-02-19T14:55:37.975350+0000","last_fullsized":"2024-02-19T14:55:37.975350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":337,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.961256+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:37.961256+0000","last_peered":"2024-02-19T14:55:37.961256+0000","last_clean":"2024-02-19T14:55:37.961256+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:37.961256+0000","last_undegraded":"2024-02-19T14:55:37.961256+0000","last_fullsized":"2024-02-19T14:55:37.961256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.443}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.433},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.624}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.625},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.754},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.80700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.441},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.25900000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.505},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.54700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.314},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.217},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.498},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.315},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:42.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:42.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 Processor -- start 2024-02-19T14:55:42.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 -- start start 2024-02-19T14:55:42.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0cb40c19f0 0x7f0cb4156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb40c1dd0 0x7f0cb40c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb4155a50 con 0x7f0cb4071410 2024-02-19T14:55:42.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb4152af0 con 0x7f0cb40c19f0 2024-02-19T14:55:42.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.498+0000 7f0cbbcd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4155f80 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0cb40c19f0 0x7f0cb4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0cb40c19f0 0x7f0cb4156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:58454/0 (socket says 172.21.15.78:58454) 2024-02-19T14:55:42.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cba271640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb40c1dd0 0x7f0cb40c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 -- 172.21.15.78:0/3913439429 learned_addr learned my addr 172.21.15.78:0/3913439429 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 -- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb40c1dd0 msgr2=0x7f0cb40c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 --2- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb40c1dd0 0x7f0cb40c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 -- 172.21.15.78:0/3913439429 >> v1:172.21.15.150:6789/0 conn(0x7f0cb4071410 legacy=0x7f0cb40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb9a70640 1 -- 172.21.15.78:0/3913439429 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4070840 con 0x7f0cb40c19f0 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cb8a6e640 1 -- 172.21.15.78:0/3913439429 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306811979 0 0) 0x7f0cb4155a50 con 0x7f0cb4071410 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.499+0000 7f0cba271640 1 --2- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb40c1dd0 0x7f0cb40c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.500+0000 7f0cb9a70640 1 --2- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb4156440 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009870 tx=0x7f0cac02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b0eb920dafeca8a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.500+0000 7f0cb8a6e640 1 -- 172.21.15.78:0/3913439429 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0317c0 con 0x7f0cb40c19f0 2024-02-19T14:55:42.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.500+0000 7f0cb8a6e640 1 -- 172.21.15.78:0/3913439429 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac031920 con 0x7f0cb40c19f0 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.500+0000 7f0cb8a6e640 1 -- 172.21.15.78:0/3913439429 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac031cb0 con 0x7f0cb40c19f0 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 msgr2=0x7f0cb4156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb4156440 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009870 tx=0x7f0cac02b860 comp rx=0 tx=0).stop 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/3913439429 shutdown_connections 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb40c1dd0 0x7f0cb40c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/3913439429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb4156440 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/3913439429 >> 172.21.15.78:0/3913439429 conn(0x7f0cb40aa370 msgr2=0x7f0cb40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/3913439429 shutdown_connections 2024-02-19T14:55:42.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.501+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/3913439429 wait complete. 2024-02-19T14:55:42.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.502+0000 7f0cbbcd4640 1 Processor -- start 2024-02-19T14:55:42.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.502+0000 7f0cbbcd4640 1 -- start start 2024-02-19T14:55:42.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.502+0000 7f0cbbcd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb4071410 0x7f0cb414bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb406cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 0x7f0cb41eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb40adeb0 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb40c5840 con 0x7f0cb40c19f0 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb41eee50 con 0x7f0cb4071410 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 0x7f0cb41eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 0x7f0cb41eacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40462/0 (socket says 172.21.15.78:40462) 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 -- 172.21.15.78:0/1326866957 learned_addr learned my addr 172.21.15.78:0/1326866957 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cb926f640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb406cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cb9a70640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb4071410 0x7f0cb414bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 -- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb4071410 msgr2=0x7f0cb414bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb4071410 0x7f0cb414bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 -- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 msgr2=0x7f0cb406cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb406cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 -- 172.21.15.78:0/1326866957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0009a70 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cba271640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 0x7f0cb41eacf0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000efc0 tx=0x7f0cb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0009cd0 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb000f040 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cac0094d0 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0003990 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.503+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb414bc40 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.505+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb40c8320 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.505+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0cb0014480 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.505+0000 7f0caa7fc640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c9806b810 0x7f0c9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.505+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0cb008ad20 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.505+0000 7f0cb9a70640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c9806b810 0x7f0c9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.506+0000 7f0cb9a70640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c9806b810 0x7f0c9806dcd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0cac036000 tx=0x7f0cac03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.510+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0cb00556f0 con 0x7f0cb40c1dd0 2024-02-19T14:55:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.683+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cb4155ca0 con 0x7f0c9806b810 2024-02-19T14:55:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.690+0000 7f0caa7fc640 1 -- 172.21.15.78:0/1326866957 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284417 (secure 0 0 0) 0x7f0cac031b10 con 0x7f0c9806b810 2024-02-19T14:55:42.696 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c9806b810 msgr2=0x7f0c9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c9806b810 0x7f0c9806dcd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0cac036000 tx=0x7f0cac03d040 comp rx=0 tx=0).stop 2024-02-19T14:55:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 msgr2=0x7f0cb41eacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 0x7f0cb41eacf0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000efc0 tx=0x7f0cb000c490 comp rx=0 tx=0).stop 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 shutdown_connections 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c9806b810 0x7f0c9806dcd0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0cb40c1dd0 0x7f0cb41eacf0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0cb40c19f0 0x7f0cb406cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 --2- 172.21.15.78:0/1326866957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0cb4071410 0x7f0cb414bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 >> 172.21.15.78:0/1326866957 conn(0x7f0cb40aa370 msgr2=0x7f0cb4158330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 shutdown_connections 2024-02-19T14:55:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.693+0000 7f0cbbcd4640 1 -- 172.21.15.78:0/1326866957 wait complete. 2024-02-19T14:55:42.707 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-02-19T14:55:42.102662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257000,"kb_used_data":11712,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754717720,"statfs":{"total":773094113280,"available":772830945280,"internally_reserved":0,"allocated":11993088,"data_stored":6412102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.152492"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955144+0000","last_change":"2024-02-19T14:55:37.955144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955144+0000","last_undegraded":"2024-02-19T14:55:37.955144+0000","last_fullsized":"2024-02-19T14:55:37.955144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962463+0000","last_change":"2024-02-19T14:55:37.962463+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962463+0000","last_undegraded":"2024-02-19T14:55:37.962463+0000","last_fullsized":"2024-02-19T14:55:37.962463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958706+0000","last_change":"2024-02-19T14:55:37.958706+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958706+0000","last_undegraded":"2024-02-19T14:55:37.958706+0000","last_fullsized":"2024-02-19T14:55:37.958706+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961496+0000","last_change":"2024-02-19T14:55:37.961496+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961496+0000","last_undegraded":"2024-02-19T14:55:37.961496+0000","last_fullsized":"2024-02-19T14:55:37.961496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956553+0000","last_change":"2024-02-19T14:55:37.956553+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956553+0000","last_undegraded":"2024-02-19T14:55:37.956553+0000","last_fullsized":"2024-02-19T14:55:37.956553+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959221+0000","last_change":"2024-02-19T14:55:37.959221+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959221+0000","last_undegraded":"2024-02-19T14:55:37.959221+0000","last_fullsized":"2024-02-19T14:55:37.959221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958511+0000","last_change":"2024-02-19T14:55:37.958511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958511+0000","last_undegraded":"2024-02-19T14:55:37.958511+0000","last_fullsized":"2024-02-19T14:55:37.958511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960041+0000","last_change":"2024-02-19T14:55:37.960041+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960041+0000","last_undegraded":"2024-02-19T14:55:37.960041+0000","last_fullsized":"2024-02-19T14:55:37.960041+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965608+0000","last_change":"2024-02-19T14:55:37.965608+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965608+0000","last_undegraded":"2024-02-19T14:55:37.965608+0000","last_fullsized":"2024-02-19T14:55:37.965608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956058+0000","last_change":"2024-02-19T14:55:37.956058+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956058+0000","last_undegraded":"2024-02-19T14:55:37.956058+0000","last_fullsized":"2024-02-19T14:55:37.956058+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961425+0000","last_change":"2024-02-19T14:55:37.961425+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961425+0000","last_undegraded":"2024-02-19T14:55:37.961425+0000","last_fullsized":"2024-02-19T14:55:37.961425+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967924+0000","last_change":"2024-02-19T14:55:37.967924+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967924+0000","last_undegraded":"2024-02-19T14:55:37.967924+0000","last_fullsized":"2024-02-19T14:55:37.967924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960544+0000","last_change":"2024-02-19T14:55:37.960544+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960544+0000","last_undegraded":"2024-02-19T14:55:37.960544+0000","last_fullsized":"2024-02-19T14:55:37.960544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956259+0000","last_change":"2024-02-19T14:55:37.956259+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956259+0000","last_undegraded":"2024-02-19T14:55:37.956259+0000","last_fullsized":"2024-02-19T14:55:37.956259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961548+0000","last_change":"2024-02-19T14:55:37.961548+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961548+0000","last_undegraded":"2024-02-19T14:55:37.961548+0000","last_fullsized":"2024-02-19T14:55:37.961548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965387+0000","last_change":"2024-02-19T14:55:37.965387+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965387+0000","last_undegraded":"2024-02-19T14:55:37.965387+0000","last_fullsized":"2024-02-19T14:55:37.965387+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962059+0000","last_change":"2024-02-19T14:55:37.962059+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962059+0000","last_undegraded":"2024-02-19T14:55:37.962059+0000","last_fullsized":"2024-02-19T14:55:37.962059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959638+0000","last_change":"2024-02-19T14:55:37.959638+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959638+0000","last_undegraded":"2024-02-19T14:55:37.959638+0000","last_fullsized":"2024-02-19T14:55:37.959638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958119+0000","last_change":"2024-02-19T14:55:37.958119+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958119+0000","last_undegraded":"2024-02-19T14:55:37.958119+0000","last_fullsized":"2024-02-19T14:55:37.958119+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962661+0000","last_change":"2024-02-19T14:55:37.962661+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962661+0000","last_undegraded":"2024-02-19T14:55:37.962661+0000","last_fullsized":"2024-02-19T14:55:37.962661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955835+0000","last_change":"2024-02-19T14:55:37.955835+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955835+0000","last_undegraded":"2024-02-19T14:55:37.955835+0000","last_fullsized":"2024-02-19T14:55:37.955835+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960561+0000","last_change":"2024-02-19T14:55:37.960561+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960561+0000","last_undegraded":"2024-02-19T14:55:37.960561+0000","last_fullsized":"2024-02-19T14:55:37.960561+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958144+0000","last_change":"2024-02-19T14:55:37.958144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958144+0000","last_undegraded":"2024-02-19T14:55:37.958144+0000","last_fullsized":"2024-02-19T14:55:37.958144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958449+0000","last_change":"2024-02-19T14:55:37.958449+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958449+0000","last_undegraded":"2024-02-19T14:55:37.958449+0000","last_fullsized":"2024-02-19T14:55:37.958449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957844+0000","last_change":"2024-02-19T14:55:37.957844+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957844+0000","last_undegraded":"2024-02-19T14:55:37.957844+0000","last_fullsized":"2024-02-19T14:55:37.957844+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959212+0000","last_change":"2024-02-19T14:55:37.959212+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959212+0000","last_undegraded":"2024-02-19T14:55:37.959212+0000","last_fullsized":"2024-02-19T14:55:37.959212+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961013+0000","last_change":"2024-02-19T14:55:37.961013+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961013+0000","last_undegraded":"2024-02-19T14:55:37.961013+0000","last_fullsized":"2024-02-19T14:55:37.961013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957840+0000","last_change":"2024-02-19T14:55:37.957840+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957840+0000","last_undegraded":"2024-02-19T14:55:37.957840+0000","last_fullsized":"2024-02-19T14:55:37.957840+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957510+0000","last_change":"2024-02-19T14:55:37.957510+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957510+0000","last_undegraded":"2024-02-19T14:55:37.957510+0000","last_fullsized":"2024-02-19T14:55:37.957510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956289+0000","last_change":"2024-02-19T14:55:37.956289+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956289+0000","last_undegraded":"2024-02-19T14:55:37.956289+0000","last_fullsized":"2024-02-19T14:55:37.956289+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.966383+0000","last_change":"2024-02-19T14:55:37.966383+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.966383+0000","last_undegraded":"2024-02-19T14:55:37.966383+0000","last_fullsized":"2024-02-19T14:55:37.966383+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969047+0000","last_change":"2024-02-19T14:55:37.967453+0000","last_active":"2024-02-19T14:55:37.969047+0000","last_peered":"2024-02-19T14:55:37.969047+0000","last_clean":"2024-02-19T14:55:37.969047+0000","last_became_active":"2024-02-19T14:55:37.967091+0000","last_became_peered":"2024-02-19T14:55:37.967091+0000","last_unstale":"2024-02-19T14:55:37.969047+0000","last_undegraded":"2024-02-19T14:55:37.969047+0000","last_fullsized":"2024-02-19T14:55:37.969047+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962010+0000","last_change":"2024-02-19T14:55:37.962010+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962010+0000","last_undegraded":"2024-02-19T14:55:37.962010+0000","last_fullsized":"2024-02-19T14:55:37.962010+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977066+0000","last_change":"2024-02-19T14:55:37.974585+0000","last_active":"2024-02-19T14:55:37.977066+0000","last_peered":"2024-02-19T14:55:37.977066+0000","last_clean":"2024-02-19T14:55:37.977066+0000","last_became_active":"2024-02-19T14:55:37.974280+0000","last_became_peered":"2024-02-19T14:55:37.974280+0000","last_unstale":"2024-02-19T14:55:37.977066+0000","last_undegraded":"2024-02-19T14:55:37.977066+0000","last_fullsized":"2024-02-19T14:55:37.977066+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:15.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957606+0000","last_change":"2024-02-19T14:55:37.957606+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957606+0000","last_undegraded":"2024-02-19T14:55:37.957606+0000","last_fullsized":"2024-02-19T14:55:37.957606+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973784+0000","last_change":"2024-02-19T14:55:37.971928+0000","last_active":"2024-02-19T14:55:37.973784+0000","last_peered":"2024-02-19T14:55:37.973784+0000","last_clean":"2024-02-19T14:55:37.973784+0000","last_became_active":"2024-02-19T14:55:37.971646+0000","last_became_peered":"2024-02-19T14:55:37.971646+0000","last_unstale":"2024-02-19T14:55:37.973784+0000","last_undegraded":"2024-02-19T14:55:37.973784+0000","last_fullsized":"2024-02-19T14:55:37.973784+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:38.067553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959582+0000","last_change":"2024-02-19T14:55:37.959582+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959582+0000","last_undegraded":"2024-02-19T14:55:37.959582+0000","last_fullsized":"2024-02-19T14:55:37.959582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977059+0000","last_change":"2024-02-19T14:55:37.973007+0000","last_active":"2024-02-19T14:55:37.977059+0000","last_peered":"2024-02-19T14:55:37.977059+0000","last_clean":"2024-02-19T14:55:37.977059+0000","last_became_active":"2024-02-19T14:55:37.972480+0000","last_became_peered":"2024-02-19T14:55:37.972480+0000","last_unstale":"2024-02-19T14:55:37.977059+0000","last_undegraded":"2024-02-19T14:55:37.977059+0000","last_fullsized":"2024-02-19T14:55:37.977059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.968277+0000","last_change":"2024-02-19T14:55:37.968277+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.968277+0000","last_undegraded":"2024-02-19T14:55:37.968277+0000","last_fullsized":"2024-02-19T14:55:37.968277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.982286+0000","last_change":"2024-02-19T14:55:37.975968+0000","last_active":"2024-02-19T14:55:37.982286+0000","last_peered":"2024-02-19T14:55:37.982286+0000","last_clean":"2024-02-19T14:55:37.982286+0000","last_became_active":"2024-02-19T14:55:37.975693+0000","last_became_peered":"2024-02-19T14:55:37.975693+0000","last_unstale":"2024-02-19T14:55:37.982286+0000","last_undegraded":"2024-02-19T14:55:37.982286+0000","last_fullsized":"2024-02-19T14:55:37.982286+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955143+0000","last_change":"2024-02-19T14:55:37.955143+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955143+0000","last_undegraded":"2024-02-19T14:55:37.955143+0000","last_fullsized":"2024-02-19T14:55:37.955143+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977070+0000","last_change":"2024-02-19T14:55:37.975370+0000","last_active":"2024-02-19T14:55:37.977070+0000","last_peered":"2024-02-19T14:55:37.977070+0000","last_clean":"2024-02-19T14:55:37.977070+0000","last_became_active":"2024-02-19T14:55:37.974133+0000","last_became_peered":"2024-02-19T14:55:37.974133+0000","last_unstale":"2024-02-19T14:55:37.977070+0000","last_undegraded":"2024-02-19T14:55:37.977070+0000","last_fullsized":"2024-02-19T14:55:37.977070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:31.145774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959131+0000","last_change":"2024-02-19T14:55:37.959131+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959131+0000","last_undegraded":"2024-02-19T14:55:37.959131+0000","last_fullsized":"2024-02-19T14:55:37.959131+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967695+0000","last_change":"2024-02-19T14:55:37.965578+0000","last_active":"2024-02-19T14:55:37.967695+0000","last_peered":"2024-02-19T14:55:37.967695+0000","last_clean":"2024-02-19T14:55:37.967695+0000","last_became_active":"2024-02-19T14:55:37.965063+0000","last_became_peered":"2024-02-19T14:55:37.965063+0000","last_unstale":"2024-02-19T14:55:37.967695+0000","last_undegraded":"2024-02-19T14:55:37.967695+0000","last_fullsized":"2024-02-19T14:55:37.967695+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967476+0000","last_change":"2024-02-19T14:55:37.967476+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967476+0000","last_undegraded":"2024-02-19T14:55:37.967476+0000","last_fullsized":"2024-02-19T14:55:37.967476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979469+0000","last_change":"2024-02-19T14:55:37.976412+0000","last_active":"2024-02-19T14:55:37.979469+0000","last_peered":"2024-02-19T14:55:37.979469+0000","last_clean":"2024-02-19T14:55:37.979469+0000","last_became_active":"2024-02-19T14:55:37.975831+0000","last_became_peered":"2024-02-19T14:55:37.975831+0000","last_unstale":"2024-02-19T14:55:37.979469+0000","last_undegraded":"2024-02-19T14:55:37.979469+0000","last_fullsized":"2024-02-19T14:55:37.979469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:55:53.499236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959395+0000","last_change":"2024-02-19T14:55:37.959395+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959395+0000","last_undegraded":"2024-02-19T14:55:37.959395+0000","last_fullsized":"2024-02-19T14:55:37.959395+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.971160+0000","last_change":"2024-02-19T14:55:37.969860+0000","last_active":"2024-02-19T14:55:37.971160+0000","last_peered":"2024-02-19T14:55:37.971160+0000","last_clean":"2024-02-19T14:55:37.971160+0000","last_became_active":"2024-02-19T14:55:37.969561+0000","last_became_peered":"2024-02-19T14:55:37.969561+0000","last_unstale":"2024-02-19T14:55:37.971160+0000","last_undegraded":"2024-02-19T14:55:37.971160+0000","last_fullsized":"2024-02-19T14:55:37.971160+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956550+0000","last_change":"2024-02-19T14:55:37.956550+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956550+0000","last_undegraded":"2024-02-19T14:55:37.956550+0000","last_fullsized":"2024-02-19T14:55:37.956550+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973119+0000","last_change":"2024-02-19T14:55:37.972249+0000","last_active":"2024-02-19T14:55:37.973119+0000","last_peered":"2024-02-19T14:55:37.973119+0000","last_clean":"2024-02-19T14:55:37.973119+0000","last_became_active":"2024-02-19T14:55:37.971802+0000","last_became_peered":"2024-02-19T14:55:37.971802+0000","last_unstale":"2024-02-19T14:55:37.973119+0000","last_undegraded":"2024-02-19T14:55:37.973119+0000","last_fullsized":"2024-02-19T14:55:37.973119+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961622+0000","last_change":"2024-02-19T14:55:37.961622+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961622+0000","last_undegraded":"2024-02-19T14:55:37.961622+0000","last_fullsized":"2024-02-19T14:55:37.961622+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970044+0000","last_change":"2024-02-19T14:55:37.968601+0000","last_active":"2024-02-19T14:55:37.970044+0000","last_peered":"2024-02-19T14:55:37.970044+0000","last_clean":"2024-02-19T14:55:37.970044+0000","last_became_active":"2024-02-19T14:55:37.968117+0000","last_became_peered":"2024-02-19T14:55:37.968117+0000","last_unstale":"2024-02-19T14:55:37.970044+0000","last_undegraded":"2024-02-19T14:55:37.970044+0000","last_fullsized":"2024-02-19T14:55:37.970044+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958050+0000","last_change":"2024-02-19T14:55:37.958050+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958050+0000","last_undegraded":"2024-02-19T14:55:37.958050+0000","last_fullsized":"2024-02-19T14:55:37.958050+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972821+0000","last_change":"2024-02-19T14:55:37.968678+0000","last_active":"2024-02-19T14:55:37.972821+0000","last_peered":"2024-02-19T14:55:37.972821+0000","last_clean":"2024-02-19T14:55:37.972821+0000","last_became_active":"2024-02-19T14:55:37.968384+0000","last_became_peered":"2024-02-19T14:55:37.968384+0000","last_unstale":"2024-02-19T14:55:37.972821+0000","last_undegraded":"2024-02-19T14:55:37.972821+0000","last_fullsized":"2024-02-19T14:55:37.972821+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:19.012872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960027+0000","last_change":"2024-02-19T14:55:37.960027+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960027+0000","last_undegraded":"2024-02-19T14:55:37.960027+0000","last_fullsized":"2024-02-19T14:55:37.960027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975368+0000","last_change":"2024-02-19T14:55:37.969264+0000","last_active":"2024-02-19T14:55:37.975368+0000","last_peered":"2024-02-19T14:55:37.975368+0000","last_clean":"2024-02-19T14:55:37.975368+0000","last_became_active":"2024-02-19T14:55:37.968958+0000","last_became_peered":"2024-02-19T14:55:37.968958+0000","last_unstale":"2024-02-19T14:55:37.975368+0000","last_undegraded":"2024-02-19T14:55:37.975368+0000","last_fullsized":"2024-02-19T14:55:37.975368+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967940+0000","last_change":"2024-02-19T14:55:37.967940+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967940+0000","last_undegraded":"2024-02-19T14:55:37.967940+0000","last_fullsized":"2024-02-19T14:55:37.967940+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975543+0000","last_change":"2024-02-19T14:55:37.970686+0000","last_active":"2024-02-19T14:55:37.975543+0000","last_peered":"2024-02-19T14:55:37.975543+0000","last_clean":"2024-02-19T14:55:37.975543+0000","last_became_active":"2024-02-19T14:55:37.970405+0000","last_became_peered":"2024-02-19T14:55:37.970405+0000","last_unstale":"2024-02-19T14:55:37.975543+0000","last_undegraded":"2024-02-19T14:55:37.975543+0000","last_fullsized":"2024-02-19T14:55:37.975543+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957316+0000","last_change":"2024-02-19T14:55:37.957316+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957316+0000","last_undegraded":"2024-02-19T14:55:37.957316+0000","last_fullsized":"2024-02-19T14:55:37.957316+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975423+0000","last_change":"2024-02-19T14:55:37.969681+0000","last_active":"2024-02-19T14:55:37.975423+0000","last_peered":"2024-02-19T14:55:37.975423+0000","last_clean":"2024-02-19T14:55:37.975423+0000","last_became_active":"2024-02-19T14:55:37.969413+0000","last_became_peered":"2024-02-19T14:55:37.969413+0000","last_unstale":"2024-02-19T14:55:37.975423+0000","last_undegraded":"2024-02-19T14:55:37.975423+0000","last_fullsized":"2024-02-19T14:55:37.975423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957511+0000","last_change":"2024-02-19T14:55:37.957511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957511+0000","last_undegraded":"2024-02-19T14:55:37.957511+0000","last_fullsized":"2024-02-19T14:55:37.957511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977083+0000","last_change":"2024-02-19T14:55:37.973068+0000","last_active":"2024-02-19T14:55:37.977083+0000","last_peered":"2024-02-19T14:55:37.977083+0000","last_clean":"2024-02-19T14:55:37.977083+0000","last_became_active":"2024-02-19T14:55:37.972754+0000","last_became_peered":"2024-02-19T14:55:37.972754+0000","last_unstale":"2024-02-19T14:55:37.977083+0000","last_undegraded":"2024-02-19T14:55:37.977083+0000","last_fullsized":"2024-02-19T14:55:37.977083+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:14.587666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955566+0000","last_change":"2024-02-19T14:55:37.955566+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955566+0000","last_undegraded":"2024-02-19T14:55:37.955566+0000","last_fullsized":"2024-02-19T14:55:37.955566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969867+0000","last_change":"2024-02-19T14:55:37.963382+0000","last_active":"2024-02-19T14:55:37.969867+0000","last_peered":"2024-02-19T14:55:37.969867+0000","last_clean":"2024-02-19T14:55:37.969867+0000","last_became_active":"2024-02-19T14:55:37.963016+0000","last_became_peered":"2024-02-19T14:55:37.963016+0000","last_unstale":"2024-02-19T14:55:37.969867+0000","last_undegraded":"2024-02-19T14:55:37.969867+0000","last_fullsized":"2024-02-19T14:55:37.969867+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:43.420537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959865+0000","last_change":"2024-02-19T14:55:37.959865+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959865+0000","last_undegraded":"2024-02-19T14:55:37.959865+0000","last_fullsized":"2024-02-19T14:55:37.959865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970634+0000","last_change":"2024-02-19T14:55:37.968907+0000","last_active":"2024-02-19T14:55:37.970634+0000","last_peered":"2024-02-19T14:55:37.970634+0000","last_clean":"2024-02-19T14:55:37.970634+0000","last_became_active":"2024-02-19T14:55:37.968628+0000","last_became_peered":"2024-02-19T14:55:37.968628+0000","last_unstale":"2024-02-19T14:55:37.970634+0000","last_undegraded":"2024-02-19T14:55:37.970634+0000","last_fullsized":"2024-02-19T14:55:37.970634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956669+0000","last_change":"2024-02-19T14:55:37.956669+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956669+0000","last_undegraded":"2024-02-19T14:55:37.956669+0000","last_fullsized":"2024-02-19T14:55:37.956669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972073+0000","last_change":"2024-02-19T14:55:37.970491+0000","last_active":"2024-02-19T14:55:37.972073+0000","last_peered":"2024-02-19T14:55:37.972073+0000","last_clean":"2024-02-19T14:55:37.972073+0000","last_became_active":"2024-02-19T14:55:37.970198+0000","last_became_peered":"2024-02-19T14:55:37.970198+0000","last_unstale":"2024-02-19T14:55:37.972073+0000","last_undegraded":"2024-02-19T14:55:37.972073+0000","last_fullsized":"2024-02-19T14:55:37.972073+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961151+0000","last_change":"2024-02-19T14:55:37.961151+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961151+0000","last_undegraded":"2024-02-19T14:55:37.961151+0000","last_fullsized":"2024-02-19T14:55:37.961151+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972058+0000","last_change":"2024-02-19T14:55:37.970614+0000","last_active":"2024-02-19T14:55:37.972058+0000","last_peered":"2024-02-19T14:55:37.972058+0000","last_clean":"2024-02-19T14:55:37.972058+0000","last_became_active":"2024-02-19T14:55:37.970319+0000","last_became_peered":"2024-02-19T14:55:37.970319+0000","last_unstale":"2024-02-19T14:55:37.972058+0000","last_undegraded":"2024-02-19T14:55:37.972058+0000","last_fullsized":"2024-02-19T14:55:37.972058+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:54.859424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960541+0000","last_change":"2024-02-19T14:55:37.960541+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960541+0000","last_undegraded":"2024-02-19T14:55:37.960541+0000","last_fullsized":"2024-02-19T14:55:37.960541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970350+0000","last_change":"2024-02-19T14:55:37.968398+0000","last_active":"2024-02-19T14:55:37.970350+0000","last_peered":"2024-02-19T14:55:37.970350+0000","last_clean":"2024-02-19T14:55:37.970350+0000","last_became_active":"2024-02-19T14:55:37.967723+0000","last_became_peered":"2024-02-19T14:55:37.967723+0000","last_unstale":"2024-02-19T14:55:37.970350+0000","last_undegraded":"2024-02-19T14:55:37.970350+0000","last_fullsized":"2024-02-19T14:55:37.970350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958633+0000","last_change":"2024-02-19T14:55:37.958633+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958633+0000","last_undegraded":"2024-02-19T14:55:37.958633+0000","last_fullsized":"2024-02-19T14:55:37.958633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973485+0000","last_change":"2024-02-19T14:55:37.968061+0000","last_active":"2024-02-19T14:55:37.973485+0000","last_peered":"2024-02-19T14:55:37.973485+0000","last_clean":"2024-02-19T14:55:37.973485+0000","last_became_active":"2024-02-19T14:55:37.967758+0000","last_became_peered":"2024-02-19T14:55:37.967758+0000","last_unstale":"2024-02-19T14:55:37.973485+0000","last_undegraded":"2024-02-19T14:55:37.973485+0000","last_fullsized":"2024-02-19T14:55:37.973485+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957115+0000","last_change":"2024-02-19T14:55:37.957115+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957115+0000","last_undegraded":"2024-02-19T14:55:37.957115+0000","last_fullsized":"2024-02-19T14:55:37.957115+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.976536+0000","last_change":"2024-02-19T14:55:37.972341+0000","last_active":"2024-02-19T14:55:37.976536+0000","last_peered":"2024-02-19T14:55:37.976536+0000","last_clean":"2024-02-19T14:55:37.976536+0000","last_became_active":"2024-02-19T14:55:37.971999+0000","last_became_peered":"2024-02-19T14:55:37.971999+0000","last_unstale":"2024-02-19T14:55:37.976536+0000","last_undegraded":"2024-02-19T14:55:37.976536+0000","last_fullsized":"2024-02-19T14:55:37.976536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:10.347679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960567+0000","last_change":"2024-02-19T14:55:37.960567+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960567+0000","last_undegraded":"2024-02-19T14:55:37.960567+0000","last_fullsized":"2024-02-19T14:55:37.960567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972094+0000","last_change":"2024-02-19T14:55:37.969752+0000","last_active":"2024-02-19T14:55:37.972094+0000","last_peered":"2024-02-19T14:55:37.972094+0000","last_clean":"2024-02-19T14:55:37.972094+0000","last_became_active":"2024-02-19T14:55:37.969493+0000","last_became_peered":"2024-02-19T14:55:37.969493+0000","last_unstale":"2024-02-19T14:55:37.972094+0000","last_undegraded":"2024-02-19T14:55:37.972094+0000","last_fullsized":"2024-02-19T14:55:37.972094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:04.496898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.964917+0000","last_change":"2024-02-19T14:55:37.964917+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.964917+0000","last_undegraded":"2024-02-19T14:55:37.964917+0000","last_fullsized":"2024-02-19T14:55:37.964917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969023+0000","last_change":"2024-02-19T14:55:37.967599+0000","last_active":"2024-02-19T14:55:37.969023+0000","last_peered":"2024-02-19T14:55:37.969023+0000","last_clean":"2024-02-19T14:55:37.969023+0000","last_became_active":"2024-02-19T14:55:37.967115+0000","last_became_peered":"2024-02-19T14:55:37.967115+0000","last_unstale":"2024-02-19T14:55:37.969023+0000","last_undegraded":"2024-02-19T14:55:37.969023+0000","last_fullsized":"2024-02-19T14:55:37.969023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961069+0000","last_change":"2024-02-19T14:55:37.961069+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961069+0000","last_undegraded":"2024-02-19T14:55:37.961069+0000","last_fullsized":"2024-02-19T14:55:37.961069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967739+0000","last_change":"2024-02-19T14:55:37.966403+0000","last_active":"2024-02-19T14:55:37.967739+0000","last_peered":"2024-02-19T14:55:37.967739+0000","last_clean":"2024-02-19T14:55:37.967739+0000","last_became_active":"2024-02-19T14:55:37.966038+0000","last_became_peered":"2024-02-19T14:55:37.966038+0000","last_unstale":"2024-02-19T14:55:37.967739+0000","last_undegraded":"2024-02-19T14:55:37.967739+0000","last_fullsized":"2024-02-19T14:55:37.967739+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956739+0000","last_change":"2024-02-19T14:55:37.956739+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956739+0000","last_undegraded":"2024-02-19T14:55:37.956739+0000","last_fullsized":"2024-02-19T14:55:37.956739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975620+0000","last_change":"2024-02-19T14:55:37.974443+0000","last_active":"2024-02-19T14:55:37.975620+0000","last_peered":"2024-02-19T14:55:37.975620+0000","last_clean":"2024-02-19T14:55:37.975620+0000","last_became_active":"2024-02-19T14:55:37.974158+0000","last_became_peered":"2024-02-19T14:55:37.974158+0000","last_unstale":"2024-02-19T14:55:37.975620+0000","last_undegraded":"2024-02-19T14:55:37.975620+0000","last_fullsized":"2024-02-19T14:55:37.975620+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:04.393457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960714+0000","last_change":"2024-02-19T14:55:37.960714+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960714+0000","last_undegraded":"2024-02-19T14:55:37.960714+0000","last_fullsized":"2024-02-19T14:55:37.960714+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.966097+0000","last_change":"2024-02-19T14:55:37.958882+0000","last_active":"2024-02-19T14:55:37.966097+0000","last_peered":"2024-02-19T14:55:37.966097+0000","last_clean":"2024-02-19T14:55:37.966097+0000","last_became_active":"2024-02-19T14:55:37.958615+0000","last_became_peered":"2024-02-19T14:55:37.958615+0000","last_unstale":"2024-02-19T14:55:37.966097+0000","last_undegraded":"2024-02-19T14:55:37.966097+0000","last_fullsized":"2024-02-19T14:55:37.966097+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:25.413669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959715+0000","last_change":"2024-02-19T14:55:37.959715+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959715+0000","last_undegraded":"2024-02-19T14:55:37.959715+0000","last_fullsized":"2024-02-19T14:55:37.959715+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967023+0000","last_change":"2024-02-19T14:55:37.962848+0000","last_active":"2024-02-19T14:55:37.967023+0000","last_peered":"2024-02-19T14:55:37.967023+0000","last_clean":"2024-02-19T14:55:37.967023+0000","last_became_active":"2024-02-19T14:55:37.962569+0000","last_became_peered":"2024-02-19T14:55:37.962569+0000","last_unstale":"2024-02-19T14:55:37.967023+0000","last_undegraded":"2024-02-19T14:55:37.967023+0000","last_fullsized":"2024-02-19T14:55:37.967023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:20.244221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959539+0000","last_change":"2024-02-19T14:55:37.959539+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959539+0000","last_undegraded":"2024-02-19T14:55:37.959539+0000","last_fullsized":"2024-02-19T14:55:37.959539+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979462+0000","last_change":"2024-02-19T14:55:37.976122+0000","last_active":"2024-02-19T14:55:37.979462+0000","last_peered":"2024-02-19T14:55:37.979462+0000","last_clean":"2024-02-19T14:55:37.979462+0000","last_became_active":"2024-02-19T14:55:37.975685+0000","last_became_peered":"2024-02-19T14:55:37.975685+0000","last_unstale":"2024-02-19T14:55:37.979462+0000","last_undegraded":"2024-02-19T14:55:37.979462+0000","last_fullsized":"2024-02-19T14:55:37.979462+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:07.357078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967011+0000","last_change":"2024-02-19T14:55:37.967011+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967011+0000","last_undegraded":"2024-02-19T14:55:37.967011+0000","last_fullsized":"2024-02-19T14:55:37.967011+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970364+0000","last_change":"2024-02-19T14:55:37.968205+0000","last_active":"2024-02-19T14:55:37.970364+0000","last_peered":"2024-02-19T14:55:37.970364+0000","last_clean":"2024-02-19T14:55:37.970364+0000","last_became_active":"2024-02-19T14:55:37.967904+0000","last_became_peered":"2024-02-19T14:55:37.967904+0000","last_unstale":"2024-02-19T14:55:37.970364+0000","last_undegraded":"2024-02-19T14:55:37.970364+0000","last_fullsized":"2024-02-19T14:55:37.970364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962297+0000","last_change":"2024-02-19T14:55:37.962297+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962297+0000","last_undegraded":"2024-02-19T14:55:37.962297+0000","last_fullsized":"2024-02-19T14:55:37.962297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975350+0000","last_change":"2024-02-19T14:55:37.969453+0000","last_active":"2024-02-19T14:55:37.975350+0000","last_peered":"2024-02-19T14:55:37.975350+0000","last_clean":"2024-02-19T14:55:37.975350+0000","last_became_active":"2024-02-19T14:55:37.968536+0000","last_became_peered":"2024-02-19T14:55:37.968536+0000","last_unstale":"2024-02-19T14:55:37.975350+0000","last_undegraded":"2024-02-19T14:55:37.975350+0000","last_fullsized":"2024-02-19T14:55:37.975350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":337,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.961256+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:37.961256+0000","last_peered":"2024-02-19T14:55:37.961256+0000","last_clean":"2024-02-19T14:55:37.961256+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:37.961256+0000","last_undegraded":"2024-02-19T14:55:37.961256+0000","last_fullsized":"2024-02-19T14:55:37.961256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.443}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.433},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.624}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.625},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.754},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.80700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.441},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.25900000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.505},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.54700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.314},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.217},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.498},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.315},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:42.711 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3f is not active+clean 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.955144+0000', 'last_change': '2024-02-19T14:55:37.955144+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.955144+0000', 'last_undegraded': '2024-02-19T14:55:37.955144+0000', 'last_fullsized': '2024-02-19T14:55:37.955144+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3e is not active+clean 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.962463+0000', 'last_change': '2024-02-19T14:55:37.962463+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.962463+0000', 'last_undegraded': '2024-02-19T14:55:37.962463+0000', 'last_fullsized': '2024-02-19T14:55:37.962463+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3d is not active+clean 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958706+0000', 'last_change': '2024-02-19T14:55:37.958706+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958706+0000', 'last_undegraded': '2024-02-19T14:55:37.958706+0000', 'last_fullsized': '2024-02-19T14:55:37.958706+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3c is not active+clean 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961496+0000', 'last_change': '2024-02-19T14:55:37.961496+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961496+0000', 'last_undegraded': '2024-02-19T14:55:37.961496+0000', 'last_fullsized': '2024-02-19T14:55:37.961496+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.712 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3b is not active+clean 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956553+0000', 'last_change': '2024-02-19T14:55:37.956553+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956553+0000', 'last_undegraded': '2024-02-19T14:55:37.956553+0000', 'last_fullsized': '2024-02-19T14:55:37.956553+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3a is not active+clean 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959221+0000', 'last_change': '2024-02-19T14:55:37.959221+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959221+0000', 'last_undegraded': '2024-02-19T14:55:37.959221+0000', 'last_fullsized': '2024-02-19T14:55:37.959221+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.39 is not active+clean 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959543+0000', 'last_change': '2024-02-19T14:55:37.959543+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959543+0000', 'last_undegraded': '2024-02-19T14:55:37.959543+0000', 'last_fullsized': '2024-02-19T14:55:37.959543+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.38 is not active+clean 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958511+0000', 'last_change': '2024-02-19T14:55:37.958511+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958511+0000', 'last_undegraded': '2024-02-19T14:55:37.958511+0000', 'last_fullsized': '2024-02-19T14:55:37.958511+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.37 is not active+clean 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960041+0000', 'last_change': '2024-02-19T14:55:37.960041+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960041+0000', 'last_undegraded': '2024-02-19T14:55:37.960041+0000', 'last_fullsized': '2024-02-19T14:55:37.960041+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.713 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.36 is not active+clean 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.965608+0000', 'last_change': '2024-02-19T14:55:37.965608+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.965608+0000', 'last_undegraded': '2024-02-19T14:55:37.965608+0000', 'last_fullsized': '2024-02-19T14:55:37.965608+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.35 is not active+clean 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956058+0000', 'last_change': '2024-02-19T14:55:37.956058+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956058+0000', 'last_undegraded': '2024-02-19T14:55:37.956058+0000', 'last_fullsized': '2024-02-19T14:55:37.956058+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.34 is not active+clean 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961425+0000', 'last_change': '2024-02-19T14:55:37.961425+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961425+0000', 'last_undegraded': '2024-02-19T14:55:37.961425+0000', 'last_fullsized': '2024-02-19T14:55:37.961425+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.33 is not active+clean 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.967924+0000', 'last_change': '2024-02-19T14:55:37.967924+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.967924+0000', 'last_undegraded': '2024-02-19T14:55:37.967924+0000', 'last_fullsized': '2024-02-19T14:55:37.967924+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.32 is not active+clean 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960544+0000', 'last_change': '2024-02-19T14:55:37.960544+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960544+0000', 'last_undegraded': '2024-02-19T14:55:37.960544+0000', 'last_fullsized': '2024-02-19T14:55:37.960544+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.714 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.31 is not active+clean 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956259+0000', 'last_change': '2024-02-19T14:55:37.956259+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956259+0000', 'last_undegraded': '2024-02-19T14:55:37.956259+0000', 'last_fullsized': '2024-02-19T14:55:37.956259+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.30 is not active+clean 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961548+0000', 'last_change': '2024-02-19T14:55:37.961548+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961548+0000', 'last_undegraded': '2024-02-19T14:55:37.961548+0000', 'last_fullsized': '2024-02-19T14:55:37.961548+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2f is not active+clean 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.965387+0000', 'last_change': '2024-02-19T14:55:37.965387+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.965387+0000', 'last_undegraded': '2024-02-19T14:55:37.965387+0000', 'last_fullsized': '2024-02-19T14:55:37.965387+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2e is not active+clean 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.962059+0000', 'last_change': '2024-02-19T14:55:37.962059+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.962059+0000', 'last_undegraded': '2024-02-19T14:55:37.962059+0000', 'last_fullsized': '2024-02-19T14:55:37.962059+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2d is not active+clean 2024-02-19T14:55:42.715 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959638+0000', 'last_change': '2024-02-19T14:55:37.959638+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959638+0000', 'last_undegraded': '2024-02-19T14:55:37.959638+0000', 'last_fullsized': '2024-02-19T14:55:37.959638+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2c is not active+clean 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958119+0000', 'last_change': '2024-02-19T14:55:37.958119+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958119+0000', 'last_undegraded': '2024-02-19T14:55:37.958119+0000', 'last_fullsized': '2024-02-19T14:55:37.958119+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2b is not active+clean 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.962661+0000', 'last_change': '2024-02-19T14:55:37.962661+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.962661+0000', 'last_undegraded': '2024-02-19T14:55:37.962661+0000', 'last_fullsized': '2024-02-19T14:55:37.962661+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2a is not active+clean 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.955835+0000', 'last_change': '2024-02-19T14:55:37.955835+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.955835+0000', 'last_undegraded': '2024-02-19T14:55:37.955835+0000', 'last_fullsized': '2024-02-19T14:55:37.955835+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.29 is not active+clean 2024-02-19T14:55:42.716 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960561+0000', 'last_change': '2024-02-19T14:55:37.960561+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960561+0000', 'last_undegraded': '2024-02-19T14:55:37.960561+0000', 'last_fullsized': '2024-02-19T14:55:37.960561+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.28 is not active+clean 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958144+0000', 'last_change': '2024-02-19T14:55:37.958144+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958144+0000', 'last_undegraded': '2024-02-19T14:55:37.958144+0000', 'last_fullsized': '2024-02-19T14:55:37.958144+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.27 is not active+clean 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958449+0000', 'last_change': '2024-02-19T14:55:37.958449+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958449+0000', 'last_undegraded': '2024-02-19T14:55:37.958449+0000', 'last_fullsized': '2024-02-19T14:55:37.958449+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.26 is not active+clean 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957844+0000', 'last_change': '2024-02-19T14:55:37.957844+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957844+0000', 'last_undegraded': '2024-02-19T14:55:37.957844+0000', 'last_fullsized': '2024-02-19T14:55:37.957844+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.25 is not active+clean 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959212+0000', 'last_change': '2024-02-19T14:55:37.959212+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959212+0000', 'last_undegraded': '2024-02-19T14:55:37.959212+0000', 'last_fullsized': '2024-02-19T14:55:37.959212+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.24 is not active+clean 2024-02-19T14:55:42.717 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961013+0000', 'last_change': '2024-02-19T14:55:37.961013+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961013+0000', 'last_undegraded': '2024-02-19T14:55:37.961013+0000', 'last_fullsized': '2024-02-19T14:55:37.961013+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.23 is not active+clean 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957840+0000', 'last_change': '2024-02-19T14:55:37.957840+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957840+0000', 'last_undegraded': '2024-02-19T14:55:37.957840+0000', 'last_fullsized': '2024-02-19T14:55:37.957840+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.22 is not active+clean 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959543+0000', 'last_change': '2024-02-19T14:55:37.959543+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959543+0000', 'last_undegraded': '2024-02-19T14:55:37.959543+0000', 'last_fullsized': '2024-02-19T14:55:37.959543+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.21 is not active+clean 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957510+0000', 'last_change': '2024-02-19T14:55:37.957510+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957510+0000', 'last_undegraded': '2024-02-19T14:55:37.957510+0000', 'last_fullsized': '2024-02-19T14:55:37.957510+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.718 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.20 is not active+clean 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956289+0000', 'last_change': '2024-02-19T14:55:37.956289+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956289+0000', 'last_undegraded': '2024-02-19T14:55:37.956289+0000', 'last_fullsized': '2024-02-19T14:55:37.956289+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1c is not active+clean 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.966383+0000', 'last_change': '2024-02-19T14:55:37.966383+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.966383+0000', 'last_undegraded': '2024-02-19T14:55:37.966383+0000', 'last_fullsized': '2024-02-19T14:55:37.966383+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1d is not active+clean 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.962010+0000', 'last_change': '2024-02-19T14:55:37.962010+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.962010+0000', 'last_undegraded': '2024-02-19T14:55:37.962010+0000', 'last_fullsized': '2024-02-19T14:55:37.962010+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1e is not active+clean 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957606+0000', 'last_change': '2024-02-19T14:55:37.957606+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957606+0000', 'last_undegraded': '2024-02-19T14:55:37.957606+0000', 'last_fullsized': '2024-02-19T14:55:37.957606+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1f is not active+clean 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959582+0000', 'last_change': '2024-02-19T14:55:37.959582+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959582+0000', 'last_undegraded': '2024-02-19T14:55:37.959582+0000', 'last_fullsized': '2024-02-19T14:55:37.959582+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.18 is not active+clean 2024-02-19T14:55:42.719 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.968277+0000', 'last_change': '2024-02-19T14:55:37.968277+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.968277+0000', 'last_undegraded': '2024-02-19T14:55:37.968277+0000', 'last_fullsized': '2024-02-19T14:55:37.968277+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.19 is not active+clean 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.955143+0000', 'last_change': '2024-02-19T14:55:37.955143+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.955143+0000', 'last_undegraded': '2024-02-19T14:55:37.955143+0000', 'last_fullsized': '2024-02-19T14:55:37.955143+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1a is not active+clean 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959131+0000', 'last_change': '2024-02-19T14:55:37.959131+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959131+0000', 'last_undegraded': '2024-02-19T14:55:37.959131+0000', 'last_fullsized': '2024-02-19T14:55:37.959131+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1b is not active+clean 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.967476+0000', 'last_change': '2024-02-19T14:55:37.967476+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.967476+0000', 'last_undegraded': '2024-02-19T14:55:37.967476+0000', 'last_fullsized': '2024-02-19T14:55:37.967476+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.14 is not active+clean 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959395+0000', 'last_change': '2024-02-19T14:55:37.959395+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959395+0000', 'last_undegraded': '2024-02-19T14:55:37.959395+0000', 'last_fullsized': '2024-02-19T14:55:37.959395+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.15 is not active+clean 2024-02-19T14:55:42.720 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956550+0000', 'last_change': '2024-02-19T14:55:37.956550+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956550+0000', 'last_undegraded': '2024-02-19T14:55:37.956550+0000', 'last_fullsized': '2024-02-19T14:55:37.956550+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.16 is not active+clean 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961622+0000', 'last_change': '2024-02-19T14:55:37.961622+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961622+0000', 'last_undegraded': '2024-02-19T14:55:37.961622+0000', 'last_fullsized': '2024-02-19T14:55:37.961622+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.17 is not active+clean 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958050+0000', 'last_change': '2024-02-19T14:55:37.958050+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958050+0000', 'last_undegraded': '2024-02-19T14:55:37.958050+0000', 'last_fullsized': '2024-02-19T14:55:37.958050+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.10 is not active+clean 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960027+0000', 'last_change': '2024-02-19T14:55:37.960027+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960027+0000', 'last_undegraded': '2024-02-19T14:55:37.960027+0000', 'last_fullsized': '2024-02-19T14:55:37.960027+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.11 is not active+clean 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.967940+0000', 'last_change': '2024-02-19T14:55:37.967940+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.967940+0000', 'last_undegraded': '2024-02-19T14:55:37.967940+0000', 'last_fullsized': '2024-02-19T14:55:37.967940+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.12 is not active+clean 2024-02-19T14:55:42.721 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957316+0000', 'last_change': '2024-02-19T14:55:37.957316+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957316+0000', 'last_undegraded': '2024-02-19T14:55:37.957316+0000', 'last_fullsized': '2024-02-19T14:55:37.957316+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.c is not active+clean 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957511+0000', 'last_change': '2024-02-19T14:55:37.957511+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957511+0000', 'last_undegraded': '2024-02-19T14:55:37.957511+0000', 'last_fullsized': '2024-02-19T14:55:37.957511+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.d is not active+clean 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.955566+0000', 'last_change': '2024-02-19T14:55:37.955566+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.955566+0000', 'last_undegraded': '2024-02-19T14:55:37.955566+0000', 'last_fullsized': '2024-02-19T14:55:37.955566+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.e is not active+clean 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959865+0000', 'last_change': '2024-02-19T14:55:37.959865+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959865+0000', 'last_undegraded': '2024-02-19T14:55:37.959865+0000', 'last_fullsized': '2024-02-19T14:55:37.959865+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.f is not active+clean 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956669+0000', 'last_change': '2024-02-19T14:55:37.956669+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956669+0000', 'last_undegraded': '2024-02-19T14:55:37.956669+0000', 'last_fullsized': '2024-02-19T14:55:37.956669+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.8 is not active+clean 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961151+0000', 'last_change': '2024-02-19T14:55:37.961151+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961151+0000', 'last_undegraded': '2024-02-19T14:55:37.961151+0000', 'last_fullsized': '2024-02-19T14:55:37.961151+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.722 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.9 is not active+clean 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960541+0000', 'last_change': '2024-02-19T14:55:37.960541+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960541+0000', 'last_undegraded': '2024-02-19T14:55:37.960541+0000', 'last_fullsized': '2024-02-19T14:55:37.960541+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.a is not active+clean 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.958633+0000', 'last_change': '2024-02-19T14:55:37.958633+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.958633+0000', 'last_undegraded': '2024-02-19T14:55:37.958633+0000', 'last_fullsized': '2024-02-19T14:55:37.958633+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.b is not active+clean 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.957115+0000', 'last_change': '2024-02-19T14:55:37.957115+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.957115+0000', 'last_undegraded': '2024-02-19T14:55:37.957115+0000', 'last_fullsized': '2024-02-19T14:55:37.957115+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.4 is not active+clean 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960567+0000', 'last_change': '2024-02-19T14:55:37.960567+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960567+0000', 'last_undegraded': '2024-02-19T14:55:37.960567+0000', 'last_fullsized': '2024-02-19T14:55:37.960567+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.5 is not active+clean 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.964917+0000', 'last_change': '2024-02-19T14:55:37.964917+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.964917+0000', 'last_undegraded': '2024-02-19T14:55:37.964917+0000', 'last_fullsized': '2024-02-19T14:55:37.964917+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.6 is not active+clean 2024-02-19T14:55:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.961069+0000', 'last_change': '2024-02-19T14:55:37.961069+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.961069+0000', 'last_undegraded': '2024-02-19T14:55:37.961069+0000', 'last_fullsized': '2024-02-19T14:55:37.961069+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.7 is not active+clean 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.956739+0000', 'last_change': '2024-02-19T14:55:37.956739+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.956739+0000', 'last_undegraded': '2024-02-19T14:55:37.956739+0000', 'last_fullsized': '2024-02-19T14:55:37.956739+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.0 is not active+clean 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.960714+0000', 'last_change': '2024-02-19T14:55:37.960714+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.960714+0000', 'last_undegraded': '2024-02-19T14:55:37.960714+0000', 'last_fullsized': '2024-02-19T14:55:37.960714+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1 is not active+clean 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959715+0000', 'last_change': '2024-02-19T14:55:37.959715+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959715+0000', 'last_undegraded': '2024-02-19T14:55:37.959715+0000', 'last_fullsized': '2024-02-19T14:55:37.959715+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2 is not active+clean 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.959539+0000', 'last_change': '2024-02-19T14:55:37.959539+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.959539+0000', 'last_undegraded': '2024-02-19T14:55:37.959539+0000', 'last_fullsized': '2024-02-19T14:55:37.959539+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3 is not active+clean 2024-02-19T14:55:42.724 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.967011+0000', 'last_change': '2024-02-19T14:55:37.967011+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.967011+0000', 'last_undegraded': '2024-02-19T14:55:37.967011+0000', 'last_fullsized': '2024-02-19T14:55:37.967011+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:55:42.725 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.13 is not active+clean 2024-02-19T14:55:42.725 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 109, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:55:37.962297+0000', 'last_change': '2024-02-19T14:55:37.962297+0000', 'last_active': '2024-02-19T14:55:37.947129+0000', 'last_peered': '2024-02-19T14:55:37.947129+0000', 'last_clean': '2024-02-19T14:55:37.947129+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:55:37.962297+0000', 'last_undegraded': '2024-02-19T14:55:37.962297+0000', 'last_fullsized': '2024-02-19T14:55:37.962297+0000', 'mapping_epoch': 109, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 109, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'last_clean_scrub_stamp': '2024-02-19T14:55:37.947129+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:55:42.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T14:55:42.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.845+0000 7f3020823640 1 Processor -- start 2024-02-19T14:55:42.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.845+0000 7f3020823640 1 -- start start 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3020823640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c152830 0x7f301c0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3020823640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f301c0c5760 0x7f301c0c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3020823640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c1586a0 con 0x7f301c0c77c0 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3020823640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301c0c6550 con 0x7f301c0c5760 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3020823640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c159320 con 0x7f301c152830 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f301c0c5760 0x7f301c0c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f301c0c5760 0x7f301c0c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40466/0 (socket says 172.21.15.78:40466) 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 -- 172.21.15.78:0/3545400652 learned_addr learned my addr 172.21.15.78:0/3545400652 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f301a59c640 1 --2- 172.21.15.78:0/3545400652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c152830 0x7f301c0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 -- 172.21.15.78:0/3545400652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c152830 msgr2=0x7f301c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 --2- 172.21.15.78:0/3545400652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c152830 0x7f301c0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 -- 172.21.15.78:0/3545400652 >> v1:172.21.15.150:6789/0 conn(0x7f301c0c77c0 legacy=0x7f301c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.846+0000 7f3019d9b640 1 -- 172.21.15.78:0/3545400652 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c158980 con 0x7f301c0c5760 2024-02-19T14:55:42.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.847+0000 7f3019d9b640 1 --2- 172.21.15.78:0/3545400652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c7280 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f3010009870 tx=0x7f301002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3b86f589455529 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.847+0000 7f301959a640 1 -- 172.21.15.78:0/3545400652 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010031640 con 0x7f301c0c5760 2024-02-19T14:55:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.847+0000 7f301959a640 1 -- 172.21.15.78:0/3545400652 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30100317a0 con 0x7f301c0c5760 2024-02-19T14:55:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.847+0000 7f301959a640 1 -- 172.21.15.78:0/3545400652 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010031b30 con 0x7f301c0c5760 2024-02-19T14:55:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 -- 172.21.15.78:0/3545400652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 msgr2=0x7f301c0c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 --2- 172.21.15.78:0/3545400652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c7280 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f3010009870 tx=0x7f301002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 -- 172.21.15.78:0/3545400652 shutdown_connections 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 --2- 172.21.15.78:0/3545400652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c7280 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 --2- 172.21.15.78:0/3545400652 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c152830 0x7f301c0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 -- 172.21.15.78:0/3545400652 >> 172.21.15.78:0/3545400652 conn(0x7f301c0ca8a0 msgr2=0x7f301c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 -- 172.21.15.78:0/3545400652 shutdown_connections 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.848+0000 7f3020823640 1 -- 172.21.15.78:0/3545400652 wait complete. 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.850+0000 7f3020823640 1 Processor -- start 2024-02-19T14:55:42.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.850+0000 7f3020823640 1 -- start start 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3020823640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3020823640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c0c77c0 0x7f301c0c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3020823640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c152830 0x7f301c148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3020823640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0abc70 con 0x7f301c0c5760 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3020823640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c06d380 con 0x7f301c152830 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3020823640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0ab990 con 0x7f301c0c77c0 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40478/0 (socket says 172.21.15.78:40478) 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 -- 172.21.15.78:0/3598546135 learned_addr learned my addr 172.21.15.78:0/3598546135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3019d9b640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c0c77c0 0x7f301c0c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301ad9d640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c152830 0x7f301c148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 -- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c0c77c0 msgr2=0x7f301c0c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c0c77c0 0x7f301c0c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 -- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c152830 msgr2=0x7f301c148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c152830 0x7f301c148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 -- 172.21.15.78:0/3598546135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100094d0 con 0x7f301c0c5760 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f3019d9b640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c0c77c0 0x7f301c0c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301ad9d640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c152830 0x7f301c148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f301a59c640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c1300 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f300c00d5f0 tx=0x7f300c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.851+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c0125f0 con 0x7f301c0c5760 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.852+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c012750 con 0x7f301c0c5760 2024-02-19T14:55:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.852+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c0becd0 con 0x7f301c0c5760 2024-02-19T14:55:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.852+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c012b00 con 0x7f301c0c5760 2024-02-19T14:55:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.852+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c0bf290 con 0x7f301c0c5760 2024-02-19T14:55:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.853+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301c0beb30 con 0x7f301c0c5760 2024-02-19T14:55:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.853+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f300c012c60 con 0x7f301c0c5760 2024-02-19T14:55:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.853+0000 7f300affd640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ffc06b840 0x7f2ffc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.853+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f300c08a830 con 0x7f301c0c5760 2024-02-19T14:55:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.853+0000 7f3019d9b640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ffc06b840 0x7f2ffc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.854+0000 7f3019d9b640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ffc06b840 0x7f2ffc06dd00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3010036000 tx=0x7f301003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:42.862+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f301c0beb30 con 0x7f301c0c5760 2024-02-19T14:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.122+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f301c0beb30 con 0x7f301c0c5760 2024-02-19T14:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.123+0000 7f300affd640 1 -- 172.21.15.78:0/3598546135 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1455 (secure 0 0 0) 0x7f301c0beb30 con 0x7f301c0c5760 2024-02-19T14:55:43.126 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:43.126 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":247,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":111,"num_osds":8,"num_up_osds":8,"osd_up_since":1708354307,"num_in_osds":8,"osd_in_since":1708354298,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":263168000,"bytes_avail":772830945280,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":84,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":7305}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-02-19T14:55:28.100671+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T14:55:43.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ffc06b840 msgr2=0x7f2ffc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ffc06b840 0x7f2ffc06dd00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3010036000 tx=0x7f301003c040 comp rx=0 tx=0).stop 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 msgr2=0x7f301c0c1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c1300 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f300c00d5f0 tx=0x7f300c00dab0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 shutdown_connections 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ffc06b840 0x7f2ffc06dd00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c152830 0x7f301c148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f301c0c77c0 0x7f301c0c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 --2- 172.21.15.78:0/3598546135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f301c0c5760 0x7f301c0c1300 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.125+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 >> 172.21.15.78:0/3598546135 conn(0x7f301c0ca8a0 msgr2=0x7f301c150050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.126+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 shutdown_connections 2024-02-19T14:55:43.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.126+0000 7f3020823640 1 -- 172.21.15.78:0/3598546135 wait complete. 2024-02-19T14:55:43.139 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T14:55:43.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.238+0000 7f4fe81e9640 1 Processor -- start 2024-02-19T14:55:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.238+0000 7f4fe81e9640 1 -- start start 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe81e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe81e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4fe0070720 0x7f4fe0070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe81e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe00c65a0 con 0x7f4fe00c7910 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe81e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe00c7530 con 0x7f4fe0070720 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe81e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe01537b0 con 0x7f4fe00c5560 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4fe0070720 0x7f4fe0070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4fe0070720 0x7f4fe0070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40490/0 (socket says 172.21.15.78:40490) 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 -- 172.21.15.78:0/3508318473 learned_addr learned my addr 172.21.15.78:0/3508318473 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 -- 172.21.15.78:0/3508318473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 msgr2=0x7f4fe00c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:43.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 --2- 172.21.15.78:0/3508318473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 -- 172.21.15.78:0/3508318473 >> v1:172.21.15.150:6789/0 conn(0x7f4fe00c7910 legacy=0x7f4fe0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:43.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.239+0000 7f4fe6786640 1 -- 172.21.15.78:0/3508318473 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0153a90 con 0x7f4fe0070720 2024-02-19T14:55:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.240+0000 7f4fe6786640 1 --2- 172.21.15.78:0/3508318473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe0070720 0x7f4fe0070b70 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009e50 tx=0x7f4fd802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f62f4a0b5cb8893 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.240+0000 7f4fe4f83640 1 -- 172.21.15.78:0/3508318473 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd802c950 con 0x7f4fe0070720 2024-02-19T14:55:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.240+0000 7f4fe4f83640 1 -- 172.21.15.78:0/3508318473 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd802cab0 con 0x7f4fe0070720 2024-02-19T14:55:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.240+0000 7f4fe4f83640 1 -- 172.21.15.78:0/3508318473 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd802ce40 con 0x7f4fe0070720 2024-02-19T14:55:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 -- 172.21.15.78:0/3508318473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe0070720 msgr2=0x7f4fe0070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/3508318473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe0070720 0x7f4fe0070b70 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009e50 tx=0x7f4fd802bdb0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 -- 172.21.15.78:0/3508318473 shutdown_connections 2024-02-19T14:55:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/3508318473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe0070720 0x7f4fe0070b70 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/3508318473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 -- 172.21.15.78:0/3508318473 >> 172.21.15.78:0/3508318473 conn(0x7f4fe006b4b0 msgr2=0x7f4fe00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 -- 172.21.15.78:0/3508318473 shutdown_connections 2024-02-19T14:55:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.241+0000 7f4fe81e9640 1 -- 172.21.15.78:0/3508318473 wait complete. 2024-02-19T14:55:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.242+0000 7f4fe81e9640 1 Processor -- start 2024-02-19T14:55:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.242+0000 7f4fe81e9640 1 -- start start 2024-02-19T14:55:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe81e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe0070720 0x7f4fe006c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe81e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe81e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 0x7f4fe014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe81e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe014f840 con 0x7f4fe00c7910 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe81e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe014fb20 con 0x7f4fe0070720 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe81e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe014e020 con 0x7f4fe00c5560 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 0x7f4fe014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 0x7f4fe014e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40500/0 (socket says 172.21.15.78:40500) 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 -- 172.21.15.78:0/2065557968 learned_addr learned my addr 172.21.15.78:0/2065557968 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe5784640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe5f85640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe0070720 0x7f4fe006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 -- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 msgr2=0x7f4fe00c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 -- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe0070720 msgr2=0x7f4fe006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe0070720 0x7f4fe006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 -- 172.21.15.78:0/2065557968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd8009b30 con 0x7f4fe00c7910 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.243+0000 7f4fe6786640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 0x7f4fe014e820 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8036040 tx=0x7f4fd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.244+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd803e450 con 0x7f4fe00c7910 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.244+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd803e5d0 con 0x7f4fe00c7910 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.244+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe00c5ac0 con 0x7f4fe00c7910 2024-02-19T14:55:43.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.244+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd803e980 con 0x7f4fe00c7910 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.244+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe00c2280 con 0x7f4fe00c7910 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.245+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe014e540 con 0x7f4fe00c7910 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.245+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4fd803eae0 con 0x7f4fe00c7910 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.246+0000 7f4fde7fc640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fcc06b840 0x7f4fcc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.246+0000 7f4fe5f85640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fcc06b840 0x7f4fcc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.246+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4fd80ad000 con 0x7f4fe00c7910 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.246+0000 7f4fe5f85640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fcc06b840 0x7f4fcc06dd00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000b810 tx=0x7f4fd0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.251+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4fe014e540 con 0x7f4fe00c7910 2024-02-19T14:55:43.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.425+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4fe00c5940 con 0x7f4fcc06b840 2024-02-19T14:55:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.432+0000 7f4fde7fc640 1 -- 172.21.15.78:0/2065557968 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284417 (secure 0 0 0) 0x7f4fd000b400 con 0x7f4fcc06b840 2024-02-19T14:55:43.438 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:43.439 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fcc06b840 msgr2=0x7f4fcc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fcc06b840 0x7f4fcc06dd00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000b810 tx=0x7f4fd0008040 comp rx=0 tx=0).stop 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 msgr2=0x7f4fe014e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 0x7f4fe014e820 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8036040 tx=0x7f4fd8004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 shutdown_connections 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4fcc06b840 0x7f4fcc06dd00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fe00c7910 0x7f4fe014e820 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fe00c5560 0x7f4fe00c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 --2- 172.21.15.78:0/2065557968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe0070720 0x7f4fe006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 >> 172.21.15.78:0/2065557968 conn(0x7f4fe006b4b0 msgr2=0x7f4fe0154030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 shutdown_connections 2024-02-19T14:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:43.435+0000 7f4fe81e9640 1 -- 172.21.15.78:0/2065557968 wait complete. 2024-02-19T14:55:43.452 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-02-19T14:55:42.102662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257000,"kb_used_data":11712,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754717720,"statfs":{"total":773094113280,"available":772830945280,"internally_reserved":0,"allocated":11993088,"data_stored":6412102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.152492"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955144+0000","last_change":"2024-02-19T14:55:37.955144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955144+0000","last_undegraded":"2024-02-19T14:55:37.955144+0000","last_fullsized":"2024-02-19T14:55:37.955144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962463+0000","last_change":"2024-02-19T14:55:37.962463+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962463+0000","last_undegraded":"2024-02-19T14:55:37.962463+0000","last_fullsized":"2024-02-19T14:55:37.962463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958706+0000","last_change":"2024-02-19T14:55:37.958706+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958706+0000","last_undegraded":"2024-02-19T14:55:37.958706+0000","last_fullsized":"2024-02-19T14:55:37.958706+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961496+0000","last_change":"2024-02-19T14:55:37.961496+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961496+0000","last_undegraded":"2024-02-19T14:55:37.961496+0000","last_fullsized":"2024-02-19T14:55:37.961496+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956553+0000","last_change":"2024-02-19T14:55:37.956553+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956553+0000","last_undegraded":"2024-02-19T14:55:37.956553+0000","last_fullsized":"2024-02-19T14:55:37.956553+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959221+0000","last_change":"2024-02-19T14:55:37.959221+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959221+0000","last_undegraded":"2024-02-19T14:55:37.959221+0000","last_fullsized":"2024-02-19T14:55:37.959221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958511+0000","last_change":"2024-02-19T14:55:37.958511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958511+0000","last_undegraded":"2024-02-19T14:55:37.958511+0000","last_fullsized":"2024-02-19T14:55:37.958511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960041+0000","last_change":"2024-02-19T14:55:37.960041+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960041+0000","last_undegraded":"2024-02-19T14:55:37.960041+0000","last_fullsized":"2024-02-19T14:55:37.960041+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965608+0000","last_change":"2024-02-19T14:55:37.965608+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965608+0000","last_undegraded":"2024-02-19T14:55:37.965608+0000","last_fullsized":"2024-02-19T14:55:37.965608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956058+0000","last_change":"2024-02-19T14:55:37.956058+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956058+0000","last_undegraded":"2024-02-19T14:55:37.956058+0000","last_fullsized":"2024-02-19T14:55:37.956058+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961425+0000","last_change":"2024-02-19T14:55:37.961425+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961425+0000","last_undegraded":"2024-02-19T14:55:37.961425+0000","last_fullsized":"2024-02-19T14:55:37.961425+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967924+0000","last_change":"2024-02-19T14:55:37.967924+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967924+0000","last_undegraded":"2024-02-19T14:55:37.967924+0000","last_fullsized":"2024-02-19T14:55:37.967924+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960544+0000","last_change":"2024-02-19T14:55:37.960544+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960544+0000","last_undegraded":"2024-02-19T14:55:37.960544+0000","last_fullsized":"2024-02-19T14:55:37.960544+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956259+0000","last_change":"2024-02-19T14:55:37.956259+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956259+0000","last_undegraded":"2024-02-19T14:55:37.956259+0000","last_fullsized":"2024-02-19T14:55:37.956259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961548+0000","last_change":"2024-02-19T14:55:37.961548+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961548+0000","last_undegraded":"2024-02-19T14:55:37.961548+0000","last_fullsized":"2024-02-19T14:55:37.961548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.965387+0000","last_change":"2024-02-19T14:55:37.965387+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.965387+0000","last_undegraded":"2024-02-19T14:55:37.965387+0000","last_fullsized":"2024-02-19T14:55:37.965387+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962059+0000","last_change":"2024-02-19T14:55:37.962059+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962059+0000","last_undegraded":"2024-02-19T14:55:37.962059+0000","last_fullsized":"2024-02-19T14:55:37.962059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959638+0000","last_change":"2024-02-19T14:55:37.959638+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959638+0000","last_undegraded":"2024-02-19T14:55:37.959638+0000","last_fullsized":"2024-02-19T14:55:37.959638+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958119+0000","last_change":"2024-02-19T14:55:37.958119+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958119+0000","last_undegraded":"2024-02-19T14:55:37.958119+0000","last_fullsized":"2024-02-19T14:55:37.958119+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962661+0000","last_change":"2024-02-19T14:55:37.962661+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962661+0000","last_undegraded":"2024-02-19T14:55:37.962661+0000","last_fullsized":"2024-02-19T14:55:37.962661+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955835+0000","last_change":"2024-02-19T14:55:37.955835+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955835+0000","last_undegraded":"2024-02-19T14:55:37.955835+0000","last_fullsized":"2024-02-19T14:55:37.955835+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960561+0000","last_change":"2024-02-19T14:55:37.960561+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960561+0000","last_undegraded":"2024-02-19T14:55:37.960561+0000","last_fullsized":"2024-02-19T14:55:37.960561+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958144+0000","last_change":"2024-02-19T14:55:37.958144+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958144+0000","last_undegraded":"2024-02-19T14:55:37.958144+0000","last_fullsized":"2024-02-19T14:55:37.958144+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958449+0000","last_change":"2024-02-19T14:55:37.958449+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958449+0000","last_undegraded":"2024-02-19T14:55:37.958449+0000","last_fullsized":"2024-02-19T14:55:37.958449+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957844+0000","last_change":"2024-02-19T14:55:37.957844+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957844+0000","last_undegraded":"2024-02-19T14:55:37.957844+0000","last_fullsized":"2024-02-19T14:55:37.957844+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959212+0000","last_change":"2024-02-19T14:55:37.959212+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959212+0000","last_undegraded":"2024-02-19T14:55:37.959212+0000","last_fullsized":"2024-02-19T14:55:37.959212+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961013+0000","last_change":"2024-02-19T14:55:37.961013+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961013+0000","last_undegraded":"2024-02-19T14:55:37.961013+0000","last_fullsized":"2024-02-19T14:55:37.961013+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957840+0000","last_change":"2024-02-19T14:55:37.957840+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957840+0000","last_undegraded":"2024-02-19T14:55:37.957840+0000","last_fullsized":"2024-02-19T14:55:37.957840+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959543+0000","last_change":"2024-02-19T14:55:37.959543+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959543+0000","last_undegraded":"2024-02-19T14:55:37.959543+0000","last_fullsized":"2024-02-19T14:55:37.959543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957510+0000","last_change":"2024-02-19T14:55:37.957510+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957510+0000","last_undegraded":"2024-02-19T14:55:37.957510+0000","last_fullsized":"2024-02-19T14:55:37.957510+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956289+0000","last_change":"2024-02-19T14:55:37.956289+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956289+0000","last_undegraded":"2024-02-19T14:55:37.956289+0000","last_fullsized":"2024-02-19T14:55:37.956289+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.966383+0000","last_change":"2024-02-19T14:55:37.966383+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.966383+0000","last_undegraded":"2024-02-19T14:55:37.966383+0000","last_fullsized":"2024-02-19T14:55:37.966383+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969047+0000","last_change":"2024-02-19T14:55:37.967453+0000","last_active":"2024-02-19T14:55:37.969047+0000","last_peered":"2024-02-19T14:55:37.969047+0000","last_clean":"2024-02-19T14:55:37.969047+0000","last_became_active":"2024-02-19T14:55:37.967091+0000","last_became_peered":"2024-02-19T14:55:37.967091+0000","last_unstale":"2024-02-19T14:55:37.969047+0000","last_undegraded":"2024-02-19T14:55:37.969047+0000","last_fullsized":"2024-02-19T14:55:37.969047+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962010+0000","last_change":"2024-02-19T14:55:37.962010+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962010+0000","last_undegraded":"2024-02-19T14:55:37.962010+0000","last_fullsized":"2024-02-19T14:55:37.962010+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977066+0000","last_change":"2024-02-19T14:55:37.974585+0000","last_active":"2024-02-19T14:55:37.977066+0000","last_peered":"2024-02-19T14:55:37.977066+0000","last_clean":"2024-02-19T14:55:37.977066+0000","last_became_active":"2024-02-19T14:55:37.974280+0000","last_became_peered":"2024-02-19T14:55:37.974280+0000","last_unstale":"2024-02-19T14:55:37.977066+0000","last_undegraded":"2024-02-19T14:55:37.977066+0000","last_fullsized":"2024-02-19T14:55:37.977066+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:15.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957606+0000","last_change":"2024-02-19T14:55:37.957606+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957606+0000","last_undegraded":"2024-02-19T14:55:37.957606+0000","last_fullsized":"2024-02-19T14:55:37.957606+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973784+0000","last_change":"2024-02-19T14:55:37.971928+0000","last_active":"2024-02-19T14:55:37.973784+0000","last_peered":"2024-02-19T14:55:37.973784+0000","last_clean":"2024-02-19T14:55:37.973784+0000","last_became_active":"2024-02-19T14:55:37.971646+0000","last_became_peered":"2024-02-19T14:55:37.971646+0000","last_unstale":"2024-02-19T14:55:37.973784+0000","last_undegraded":"2024-02-19T14:55:37.973784+0000","last_fullsized":"2024-02-19T14:55:37.973784+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:38.067553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959582+0000","last_change":"2024-02-19T14:55:37.959582+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959582+0000","last_undegraded":"2024-02-19T14:55:37.959582+0000","last_fullsized":"2024-02-19T14:55:37.959582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977059+0000","last_change":"2024-02-19T14:55:37.973007+0000","last_active":"2024-02-19T14:55:37.977059+0000","last_peered":"2024-02-19T14:55:37.977059+0000","last_clean":"2024-02-19T14:55:37.977059+0000","last_became_active":"2024-02-19T14:55:37.972480+0000","last_became_peered":"2024-02-19T14:55:37.972480+0000","last_unstale":"2024-02-19T14:55:37.977059+0000","last_undegraded":"2024-02-19T14:55:37.977059+0000","last_fullsized":"2024-02-19T14:55:37.977059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.968277+0000","last_change":"2024-02-19T14:55:37.968277+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.968277+0000","last_undegraded":"2024-02-19T14:55:37.968277+0000","last_fullsized":"2024-02-19T14:55:37.968277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.982286+0000","last_change":"2024-02-19T14:55:37.975968+0000","last_active":"2024-02-19T14:55:37.982286+0000","last_peered":"2024-02-19T14:55:37.982286+0000","last_clean":"2024-02-19T14:55:37.982286+0000","last_became_active":"2024-02-19T14:55:37.975693+0000","last_became_peered":"2024-02-19T14:55:37.975693+0000","last_unstale":"2024-02-19T14:55:37.982286+0000","last_undegraded":"2024-02-19T14:55:37.982286+0000","last_fullsized":"2024-02-19T14:55:37.982286+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:29.578159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955143+0000","last_change":"2024-02-19T14:55:37.955143+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955143+0000","last_undegraded":"2024-02-19T14:55:37.955143+0000","last_fullsized":"2024-02-19T14:55:37.955143+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977070+0000","last_change":"2024-02-19T14:55:37.975370+0000","last_active":"2024-02-19T14:55:37.977070+0000","last_peered":"2024-02-19T14:55:37.977070+0000","last_clean":"2024-02-19T14:55:37.977070+0000","last_became_active":"2024-02-19T14:55:37.974133+0000","last_became_peered":"2024-02-19T14:55:37.974133+0000","last_unstale":"2024-02-19T14:55:37.977070+0000","last_undegraded":"2024-02-19T14:55:37.977070+0000","last_fullsized":"2024-02-19T14:55:37.977070+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:31.145774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959131+0000","last_change":"2024-02-19T14:55:37.959131+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959131+0000","last_undegraded":"2024-02-19T14:55:37.959131+0000","last_fullsized":"2024-02-19T14:55:37.959131+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967695+0000","last_change":"2024-02-19T14:55:37.965578+0000","last_active":"2024-02-19T14:55:37.967695+0000","last_peered":"2024-02-19T14:55:37.967695+0000","last_clean":"2024-02-19T14:55:37.967695+0000","last_became_active":"2024-02-19T14:55:37.965063+0000","last_became_peered":"2024-02-19T14:55:37.965063+0000","last_unstale":"2024-02-19T14:55:37.967695+0000","last_undegraded":"2024-02-19T14:55:37.967695+0000","last_fullsized":"2024-02-19T14:55:37.967695+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967476+0000","last_change":"2024-02-19T14:55:37.967476+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967476+0000","last_undegraded":"2024-02-19T14:55:37.967476+0000","last_fullsized":"2024-02-19T14:55:37.967476+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979469+0000","last_change":"2024-02-19T14:55:37.976412+0000","last_active":"2024-02-19T14:55:37.979469+0000","last_peered":"2024-02-19T14:55:37.979469+0000","last_clean":"2024-02-19T14:55:37.979469+0000","last_became_active":"2024-02-19T14:55:37.975831+0000","last_became_peered":"2024-02-19T14:55:37.975831+0000","last_unstale":"2024-02-19T14:55:37.979469+0000","last_undegraded":"2024-02-19T14:55:37.979469+0000","last_fullsized":"2024-02-19T14:55:37.979469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:55:53.499236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959395+0000","last_change":"2024-02-19T14:55:37.959395+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959395+0000","last_undegraded":"2024-02-19T14:55:37.959395+0000","last_fullsized":"2024-02-19T14:55:37.959395+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.971160+0000","last_change":"2024-02-19T14:55:37.969860+0000","last_active":"2024-02-19T14:55:37.971160+0000","last_peered":"2024-02-19T14:55:37.971160+0000","last_clean":"2024-02-19T14:55:37.971160+0000","last_became_active":"2024-02-19T14:55:37.969561+0000","last_became_peered":"2024-02-19T14:55:37.969561+0000","last_unstale":"2024-02-19T14:55:37.971160+0000","last_undegraded":"2024-02-19T14:55:37.971160+0000","last_fullsized":"2024-02-19T14:55:37.971160+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956550+0000","last_change":"2024-02-19T14:55:37.956550+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956550+0000","last_undegraded":"2024-02-19T14:55:37.956550+0000","last_fullsized":"2024-02-19T14:55:37.956550+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973119+0000","last_change":"2024-02-19T14:55:37.972249+0000","last_active":"2024-02-19T14:55:37.973119+0000","last_peered":"2024-02-19T14:55:37.973119+0000","last_clean":"2024-02-19T14:55:37.973119+0000","last_became_active":"2024-02-19T14:55:37.971802+0000","last_became_peered":"2024-02-19T14:55:37.971802+0000","last_unstale":"2024-02-19T14:55:37.973119+0000","last_undegraded":"2024-02-19T14:55:37.973119+0000","last_fullsized":"2024-02-19T14:55:37.973119+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961622+0000","last_change":"2024-02-19T14:55:37.961622+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961622+0000","last_undegraded":"2024-02-19T14:55:37.961622+0000","last_fullsized":"2024-02-19T14:55:37.961622+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970044+0000","last_change":"2024-02-19T14:55:37.968601+0000","last_active":"2024-02-19T14:55:37.970044+0000","last_peered":"2024-02-19T14:55:37.970044+0000","last_clean":"2024-02-19T14:55:37.970044+0000","last_became_active":"2024-02-19T14:55:37.968117+0000","last_became_peered":"2024-02-19T14:55:37.968117+0000","last_unstale":"2024-02-19T14:55:37.970044+0000","last_undegraded":"2024-02-19T14:55:37.970044+0000","last_fullsized":"2024-02-19T14:55:37.970044+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:24.815693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958050+0000","last_change":"2024-02-19T14:55:37.958050+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958050+0000","last_undegraded":"2024-02-19T14:55:37.958050+0000","last_fullsized":"2024-02-19T14:55:37.958050+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972821+0000","last_change":"2024-02-19T14:55:37.968678+0000","last_active":"2024-02-19T14:55:37.972821+0000","last_peered":"2024-02-19T14:55:37.972821+0000","last_clean":"2024-02-19T14:55:37.972821+0000","last_became_active":"2024-02-19T14:55:37.968384+0000","last_became_peered":"2024-02-19T14:55:37.968384+0000","last_unstale":"2024-02-19T14:55:37.972821+0000","last_undegraded":"2024-02-19T14:55:37.972821+0000","last_fullsized":"2024-02-19T14:55:37.972821+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:19.012872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960027+0000","last_change":"2024-02-19T14:55:37.960027+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960027+0000","last_undegraded":"2024-02-19T14:55:37.960027+0000","last_fullsized":"2024-02-19T14:55:37.960027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975368+0000","last_change":"2024-02-19T14:55:37.969264+0000","last_active":"2024-02-19T14:55:37.975368+0000","last_peered":"2024-02-19T14:55:37.975368+0000","last_clean":"2024-02-19T14:55:37.975368+0000","last_became_active":"2024-02-19T14:55:37.968958+0000","last_became_peered":"2024-02-19T14:55:37.968958+0000","last_unstale":"2024-02-19T14:55:37.975368+0000","last_undegraded":"2024-02-19T14:55:37.975368+0000","last_fullsized":"2024-02-19T14:55:37.975368+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967940+0000","last_change":"2024-02-19T14:55:37.967940+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967940+0000","last_undegraded":"2024-02-19T14:55:37.967940+0000","last_fullsized":"2024-02-19T14:55:37.967940+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975543+0000","last_change":"2024-02-19T14:55:37.970686+0000","last_active":"2024-02-19T14:55:37.975543+0000","last_peered":"2024-02-19T14:55:37.975543+0000","last_clean":"2024-02-19T14:55:37.975543+0000","last_became_active":"2024-02-19T14:55:37.970405+0000","last_became_peered":"2024-02-19T14:55:37.970405+0000","last_unstale":"2024-02-19T14:55:37.975543+0000","last_undegraded":"2024-02-19T14:55:37.975543+0000","last_fullsized":"2024-02-19T14:55:37.975543+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957316+0000","last_change":"2024-02-19T14:55:37.957316+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957316+0000","last_undegraded":"2024-02-19T14:55:37.957316+0000","last_fullsized":"2024-02-19T14:55:37.957316+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975423+0000","last_change":"2024-02-19T14:55:37.969681+0000","last_active":"2024-02-19T14:55:37.975423+0000","last_peered":"2024-02-19T14:55:37.975423+0000","last_clean":"2024-02-19T14:55:37.975423+0000","last_became_active":"2024-02-19T14:55:37.969413+0000","last_became_peered":"2024-02-19T14:55:37.969413+0000","last_unstale":"2024-02-19T14:55:37.975423+0000","last_undegraded":"2024-02-19T14:55:37.975423+0000","last_fullsized":"2024-02-19T14:55:37.975423+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957511+0000","last_change":"2024-02-19T14:55:37.957511+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957511+0000","last_undegraded":"2024-02-19T14:55:37.957511+0000","last_fullsized":"2024-02-19T14:55:37.957511+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.977083+0000","last_change":"2024-02-19T14:55:37.973068+0000","last_active":"2024-02-19T14:55:37.977083+0000","last_peered":"2024-02-19T14:55:37.977083+0000","last_clean":"2024-02-19T14:55:37.977083+0000","last_became_active":"2024-02-19T14:55:37.972754+0000","last_became_peered":"2024-02-19T14:55:37.972754+0000","last_unstale":"2024-02-19T14:55:37.977083+0000","last_undegraded":"2024-02-19T14:55:37.977083+0000","last_fullsized":"2024-02-19T14:55:37.977083+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:14.587666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.955566+0000","last_change":"2024-02-19T14:55:37.955566+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.955566+0000","last_undegraded":"2024-02-19T14:55:37.955566+0000","last_fullsized":"2024-02-19T14:55:37.955566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969867+0000","last_change":"2024-02-19T14:55:37.963382+0000","last_active":"2024-02-19T14:55:37.969867+0000","last_peered":"2024-02-19T14:55:37.969867+0000","last_clean":"2024-02-19T14:55:37.969867+0000","last_became_active":"2024-02-19T14:55:37.963016+0000","last_became_peered":"2024-02-19T14:55:37.963016+0000","last_unstale":"2024-02-19T14:55:37.969867+0000","last_undegraded":"2024-02-19T14:55:37.969867+0000","last_fullsized":"2024-02-19T14:55:37.969867+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:25:43.420537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959865+0000","last_change":"2024-02-19T14:55:37.959865+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959865+0000","last_undegraded":"2024-02-19T14:55:37.959865+0000","last_fullsized":"2024-02-19T14:55:37.959865+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970634+0000","last_change":"2024-02-19T14:55:37.968907+0000","last_active":"2024-02-19T14:55:37.970634+0000","last_peered":"2024-02-19T14:55:37.970634+0000","last_clean":"2024-02-19T14:55:37.970634+0000","last_became_active":"2024-02-19T14:55:37.968628+0000","last_became_peered":"2024-02-19T14:55:37.968628+0000","last_unstale":"2024-02-19T14:55:37.970634+0000","last_undegraded":"2024-02-19T14:55:37.970634+0000","last_fullsized":"2024-02-19T14:55:37.970634+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956669+0000","last_change":"2024-02-19T14:55:37.956669+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956669+0000","last_undegraded":"2024-02-19T14:55:37.956669+0000","last_fullsized":"2024-02-19T14:55:37.956669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972073+0000","last_change":"2024-02-19T14:55:37.970491+0000","last_active":"2024-02-19T14:55:37.972073+0000","last_peered":"2024-02-19T14:55:37.972073+0000","last_clean":"2024-02-19T14:55:37.972073+0000","last_became_active":"2024-02-19T14:55:37.970198+0000","last_became_peered":"2024-02-19T14:55:37.970198+0000","last_unstale":"2024-02-19T14:55:37.972073+0000","last_undegraded":"2024-02-19T14:55:37.972073+0000","last_fullsized":"2024-02-19T14:55:37.972073+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961151+0000","last_change":"2024-02-19T14:55:37.961151+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961151+0000","last_undegraded":"2024-02-19T14:55:37.961151+0000","last_fullsized":"2024-02-19T14:55:37.961151+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972058+0000","last_change":"2024-02-19T14:55:37.970614+0000","last_active":"2024-02-19T14:55:37.972058+0000","last_peered":"2024-02-19T14:55:37.972058+0000","last_clean":"2024-02-19T14:55:37.972058+0000","last_became_active":"2024-02-19T14:55:37.970319+0000","last_became_peered":"2024-02-19T14:55:37.970319+0000","last_unstale":"2024-02-19T14:55:37.972058+0000","last_undegraded":"2024-02-19T14:55:37.972058+0000","last_fullsized":"2024-02-19T14:55:37.972058+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:54.859424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960541+0000","last_change":"2024-02-19T14:55:37.960541+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960541+0000","last_undegraded":"2024-02-19T14:55:37.960541+0000","last_fullsized":"2024-02-19T14:55:37.960541+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970350+0000","last_change":"2024-02-19T14:55:37.968398+0000","last_active":"2024-02-19T14:55:37.970350+0000","last_peered":"2024-02-19T14:55:37.970350+0000","last_clean":"2024-02-19T14:55:37.970350+0000","last_became_active":"2024-02-19T14:55:37.967723+0000","last_became_peered":"2024-02-19T14:55:37.967723+0000","last_unstale":"2024-02-19T14:55:37.970350+0000","last_undegraded":"2024-02-19T14:55:37.970350+0000","last_fullsized":"2024-02-19T14:55:37.970350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.958633+0000","last_change":"2024-02-19T14:55:37.958633+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.958633+0000","last_undegraded":"2024-02-19T14:55:37.958633+0000","last_fullsized":"2024-02-19T14:55:37.958633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.973485+0000","last_change":"2024-02-19T14:55:37.968061+0000","last_active":"2024-02-19T14:55:37.973485+0000","last_peered":"2024-02-19T14:55:37.973485+0000","last_clean":"2024-02-19T14:55:37.973485+0000","last_became_active":"2024-02-19T14:55:37.967758+0000","last_became_peered":"2024-02-19T14:55:37.967758+0000","last_unstale":"2024-02-19T14:55:37.973485+0000","last_undegraded":"2024-02-19T14:55:37.973485+0000","last_fullsized":"2024-02-19T14:55:37.973485+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.957115+0000","last_change":"2024-02-19T14:55:37.957115+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.957115+0000","last_undegraded":"2024-02-19T14:55:37.957115+0000","last_fullsized":"2024-02-19T14:55:37.957115+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.976536+0000","last_change":"2024-02-19T14:55:37.972341+0000","last_active":"2024-02-19T14:55:37.976536+0000","last_peered":"2024-02-19T14:55:37.976536+0000","last_clean":"2024-02-19T14:55:37.976536+0000","last_became_active":"2024-02-19T14:55:37.971999+0000","last_became_peered":"2024-02-19T14:55:37.971999+0000","last_unstale":"2024-02-19T14:55:37.976536+0000","last_undegraded":"2024-02-19T14:55:37.976536+0000","last_fullsized":"2024-02-19T14:55:37.976536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:10.347679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960567+0000","last_change":"2024-02-19T14:55:37.960567+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960567+0000","last_undegraded":"2024-02-19T14:55:37.960567+0000","last_fullsized":"2024-02-19T14:55:37.960567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.972094+0000","last_change":"2024-02-19T14:55:37.969752+0000","last_active":"2024-02-19T14:55:37.972094+0000","last_peered":"2024-02-19T14:55:37.972094+0000","last_clean":"2024-02-19T14:55:37.972094+0000","last_became_active":"2024-02-19T14:55:37.969493+0000","last_became_peered":"2024-02-19T14:55:37.969493+0000","last_unstale":"2024-02-19T14:55:37.972094+0000","last_undegraded":"2024-02-19T14:55:37.972094+0000","last_fullsized":"2024-02-19T14:55:37.972094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:04.496898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.964917+0000","last_change":"2024-02-19T14:55:37.964917+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.964917+0000","last_undegraded":"2024-02-19T14:55:37.964917+0000","last_fullsized":"2024-02-19T14:55:37.964917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.969023+0000","last_change":"2024-02-19T14:55:37.967599+0000","last_active":"2024-02-19T14:55:37.969023+0000","last_peered":"2024-02-19T14:55:37.969023+0000","last_clean":"2024-02-19T14:55:37.969023+0000","last_became_active":"2024-02-19T14:55:37.967115+0000","last_became_peered":"2024-02-19T14:55:37.967115+0000","last_unstale":"2024-02-19T14:55:37.969023+0000","last_undegraded":"2024-02-19T14:55:37.969023+0000","last_fullsized":"2024-02-19T14:55:37.969023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:10:07.207053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.961069+0000","last_change":"2024-02-19T14:55:37.961069+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.961069+0000","last_undegraded":"2024-02-19T14:55:37.961069+0000","last_fullsized":"2024-02-19T14:55:37.961069+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967739+0000","last_change":"2024-02-19T14:55:37.966403+0000","last_active":"2024-02-19T14:55:37.967739+0000","last_peered":"2024-02-19T14:55:37.967739+0000","last_clean":"2024-02-19T14:55:37.967739+0000","last_became_active":"2024-02-19T14:55:37.966038+0000","last_became_peered":"2024-02-19T14:55:37.966038+0000","last_unstale":"2024-02-19T14:55:37.967739+0000","last_undegraded":"2024-02-19T14:55:37.967739+0000","last_fullsized":"2024-02-19T14:55:37.967739+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:40.702533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.956739+0000","last_change":"2024-02-19T14:55:37.956739+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.956739+0000","last_undegraded":"2024-02-19T14:55:37.956739+0000","last_fullsized":"2024-02-19T14:55:37.956739+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975620+0000","last_change":"2024-02-19T14:55:37.974443+0000","last_active":"2024-02-19T14:55:37.975620+0000","last_peered":"2024-02-19T14:55:37.975620+0000","last_clean":"2024-02-19T14:55:37.975620+0000","last_became_active":"2024-02-19T14:55:37.974158+0000","last_became_peered":"2024-02-19T14:55:37.974158+0000","last_unstale":"2024-02-19T14:55:37.975620+0000","last_undegraded":"2024-02-19T14:55:37.975620+0000","last_fullsized":"2024-02-19T14:55:37.975620+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:04.393457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.960714+0000","last_change":"2024-02-19T14:55:37.960714+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.960714+0000","last_undegraded":"2024-02-19T14:55:37.960714+0000","last_fullsized":"2024-02-19T14:55:37.960714+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.966097+0000","last_change":"2024-02-19T14:55:37.958882+0000","last_active":"2024-02-19T14:55:37.966097+0000","last_peered":"2024-02-19T14:55:37.966097+0000","last_clean":"2024-02-19T14:55:37.966097+0000","last_became_active":"2024-02-19T14:55:37.958615+0000","last_became_peered":"2024-02-19T14:55:37.958615+0000","last_unstale":"2024-02-19T14:55:37.966097+0000","last_undegraded":"2024-02-19T14:55:37.966097+0000","last_fullsized":"2024-02-19T14:55:37.966097+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:17:25.413669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959715+0000","last_change":"2024-02-19T14:55:37.959715+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959715+0000","last_undegraded":"2024-02-19T14:55:37.959715+0000","last_fullsized":"2024-02-19T14:55:37.959715+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.967023+0000","last_change":"2024-02-19T14:55:37.962848+0000","last_active":"2024-02-19T14:55:37.967023+0000","last_peered":"2024-02-19T14:55:37.967023+0000","last_clean":"2024-02-19T14:55:37.967023+0000","last_became_active":"2024-02-19T14:55:37.962569+0000","last_became_peered":"2024-02-19T14:55:37.962569+0000","last_unstale":"2024-02-19T14:55:37.967023+0000","last_undegraded":"2024-02-19T14:55:37.967023+0000","last_fullsized":"2024-02-19T14:55:37.967023+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:20.244221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.959539+0000","last_change":"2024-02-19T14:55:37.959539+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.959539+0000","last_undegraded":"2024-02-19T14:55:37.959539+0000","last_fullsized":"2024-02-19T14:55:37.959539+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.979462+0000","last_change":"2024-02-19T14:55:37.976122+0000","last_active":"2024-02-19T14:55:37.979462+0000","last_peered":"2024-02-19T14:55:37.979462+0000","last_clean":"2024-02-19T14:55:37.979462+0000","last_became_active":"2024-02-19T14:55:37.975685+0000","last_became_peered":"2024-02-19T14:55:37.975685+0000","last_unstale":"2024-02-19T14:55:37.979462+0000","last_undegraded":"2024-02-19T14:55:37.979462+0000","last_fullsized":"2024-02-19T14:55:37.979462+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:07.357078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.967011+0000","last_change":"2024-02-19T14:55:37.967011+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.967011+0000","last_undegraded":"2024-02-19T14:55:37.967011+0000","last_fullsized":"2024-02-19T14:55:37.967011+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.970364+0000","last_change":"2024-02-19T14:55:37.968205+0000","last_active":"2024-02-19T14:55:37.970364+0000","last_peered":"2024-02-19T14:55:37.970364+0000","last_clean":"2024-02-19T14:55:37.970364+0000","last_became_active":"2024-02-19T14:55:37.967904+0000","last_became_peered":"2024-02-19T14:55:37.967904+0000","last_unstale":"2024-02-19T14:55:37.970364+0000","last_undegraded":"2024-02-19T14:55:37.970364+0000","last_fullsized":"2024-02-19T14:55:37.970364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":109,"state":"creating+peering","last_fresh":"2024-02-19T14:55:37.962297+0000","last_change":"2024-02-19T14:55:37.962297+0000","last_active":"2024-02-19T14:55:37.947129+0000","last_peered":"2024-02-19T14:55:37.947129+0000","last_clean":"2024-02-19T14:55:37.947129+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:55:37.962297+0000","last_undegraded":"2024-02-19T14:55:37.962297+0000","last_fullsized":"2024-02-19T14:55:37.962297+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.975350+0000","last_change":"2024-02-19T14:55:37.969453+0000","last_active":"2024-02-19T14:55:37.975350+0000","last_peered":"2024-02-19T14:55:37.975350+0000","last_clean":"2024-02-19T14:55:37.975350+0000","last_became_active":"2024-02-19T14:55:37.968536+0000","last_became_peered":"2024-02-19T14:55:37.968536+0000","last_unstale":"2024-02-19T14:55:37.975350+0000","last_undegraded":"2024-02-19T14:55:37.975350+0000","last_fullsized":"2024-02-19T14:55:37.975350+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":337,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-19T14:55:37.961256+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:37.961256+0000","last_peered":"2024-02-19T14:55:37.961256+0000","last_clean":"2024-02-19T14:55:37.961256+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:37.961256+0000","last_undegraded":"2024-02-19T14:55:37.961256+0000","last_fullsized":"2024-02-19T14:55:37.961256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.443}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.433},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.624}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.625},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.754},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.80700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.70199999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.441},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53100000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.25900000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.505},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.54700000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28336,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343504,"statfs":{"total":96636764160,"available":96607748096,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.314},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1347584,"data_stored":652254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.217},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.498},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":1941504,"data_stored":1242622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.315},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:44.221 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:55:44.219+0000 7f7c5eaf1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:46.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:55:46.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.556+0000 7f6874c46640 1 Processor -- start 2024-02-19T14:55:46.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.556+0000 7f6874c46640 1 -- start start 2024-02-19T14:55:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f6874c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700bdf20 0x7f68700ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f6874c46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68700c55f0 0x7f687015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f6874c46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6870158c40 con 0x7f68700ac390 2024-02-19T14:55:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f6874c46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700c5a60 con 0x7f68700c55f0 2024-02-19T14:55:46.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f6874c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700c8de0 con 0x7f68700bdf20 2024-02-19T14:55:46.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f686ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68700c55f0 0x7f687015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.557+0000 7f6865d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700bdf20 0x7f68700ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f686e59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f68700ac390 0x7f6870158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42124/0 (socket says 172.21.15.78:42124) 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f686e59c640 1 -- 172.21.15.78:0/2338254330 learned_addr learned my addr 172.21.15.78:0/2338254330 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f686ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68700c55f0 0x7f687015f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:58488/0 (socket says 172.21.15.78:58488) 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f686dd9b640 1 -- 172.21.15.78:0/2338254330 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923136560 0 0) 0x7f6870158c40 con 0x7f68700ac390 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f686dd9b640 1 -- 172.21.15.78:0/2338254330 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6850003520 con 0x7f68700ac390 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f6865d9b640 1 -- 172.21.15.78:0/2338254330 >> v2:172.21.15.150:3300/0 conn(0x7f68700c55f0 msgr2=0x7f687015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f6865d9b640 1 --2- 172.21.15.78:0/2338254330 >> v2:172.21.15.150:3300/0 conn(0x7f68700c55f0 0x7f687015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f6865d9b640 1 -- 172.21.15.78:0/2338254330 >> v1:172.21.15.150:6789/0 conn(0x7f68700ac390 legacy=0x7f6870158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.558+0000 7f6865d9b640 1 -- 172.21.15.78:0/2338254330 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870155a60 con 0x7f68700bdf20 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f6865d9b640 1 --2- 172.21.15.78:0/2338254330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700bdf20 0x7f68700ac770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68600097a0 tx=0x7f686002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=970cb29f291837a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f686dd9b640 1 -- 172.21.15.78:0/2338254330 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686002f050 con 0x7f68700bdf20 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f686dd9b640 1 -- 172.21.15.78:0/2338254330 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6860031700 con 0x7f68700bdf20 2024-02-19T14:55:46.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f686dd9b640 1 -- 172.21.15.78:0/2338254330 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860031a70 con 0x7f68700bdf20 2024-02-19T14:55:46.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f6874c46640 1 -- 172.21.15.78:0/2338254330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700bdf20 msgr2=0x7f68700ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f6874c46640 1 --2- 172.21.15.78:0/2338254330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700bdf20 0x7f68700ac770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f68600097a0 tx=0x7f686002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f6874c46640 1 -- 172.21.15.78:0/2338254330 shutdown_connections 2024-02-19T14:55:46.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.559+0000 7f6874c46640 1 --2- 172.21.15.78:0/2338254330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700bdf20 0x7f68700ac770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.560+0000 7f6874c46640 1 --2- 172.21.15.78:0/2338254330 >> v2:172.21.15.150:3300/0 conn(0x7f68700c55f0 0x7f687015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.560+0000 7f6874c46640 1 -- 172.21.15.78:0/2338254330 >> 172.21.15.78:0/2338254330 conn(0x7f68700ca890 msgr2=0x7f68700caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:46.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.560+0000 7f6874c46640 1 -- 172.21.15.78:0/2338254330 shutdown_connections 2024-02-19T14:55:46.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.560+0000 7f6874c46640 1 -- 172.21.15.78:0/2338254330 wait complete. 2024-02-19T14:55:46.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.561+0000 7f6874c46640 1 Processor -- start 2024-02-19T14:55:46.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.561+0000 7f6874c46640 1 -- start start 2024-02-19T14:55:46.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.562+0000 7f6874c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700ac390 0x7f68701eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.562+0000 7f6874c46640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 0x7f68700c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.562+0000 7f6874c46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700c55f0 0x7f68701518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.562+0000 7f6874c46640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701eb800 con 0x7f68700bdf20 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.562+0000 7f6874c46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700be710 con 0x7f68700c55f0 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6874c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700bfbc0 con 0x7f68700ac390 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 0x7f68700c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 0x7f68700c3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40516/0 (socket says 172.21.15.78:40516) 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 -- 172.21.15.78:0/4042360744 learned_addr learned my addr 172.21.15.78:0/4042360744 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f686e59c640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700ac390 0x7f68701eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f686ed9d640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700c55f0 0x7f68701518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 -- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700ac390 msgr2=0x7f68701eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700ac390 0x7f68701eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 -- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700c55f0 msgr2=0x7f68701518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700c55f0 0x7f68701518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 -- 172.21.15.78:0/4042360744 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6860009400 con 0x7f68700bdf20 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f686ed9d640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700c55f0 0x7f68701518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:46.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6865d9b640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 0x7f68700c3ba0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6860036040 tx=0x7f68600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:46.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860031ef0 con 0x7f68700bdf20 2024-02-19T14:55:46.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700bfea0 con 0x7f68700bdf20 2024-02-19T14:55:46.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68600044c0 con 0x7f68700bdf20 2024-02-19T14:55:46.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860003820 con 0x7f68700bdf20 2024-02-19T14:55:46.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.563+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700be3a0 con 0x7f68700bdf20 2024-02-19T14:55:46.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.564+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68700be8c0 con 0x7f68700bdf20 2024-02-19T14:55:46.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.565+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6860003980 con 0x7f68700bdf20 2024-02-19T14:55:46.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.565+0000 7f68677fe640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f685406b840 0x7f685406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.566+0000 7f686e59c640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f685406b840 0x7f685406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.566+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f68600abed0 con 0x7f68700bdf20 2024-02-19T14:55:46.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.566+0000 7f686e59c640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f685406b840 0x7f685406dd00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6858005880 tx=0x7f6858009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:46.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.570+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68700be8c0 con 0x7f68700bdf20 2024-02-19T14:55:46.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.743+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68700cb240 con 0x7f685406b840 2024-02-19T14:55:46.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.750+0000 7f68677fe640 1 -- 172.21.15.78:0/4042360744 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289755 (secure 0 0 0) 0x7f68700cb240 con 0x7f685406b840 2024-02-19T14:55:46.756 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:46.757 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f685406b840 msgr2=0x7f685406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f685406b840 0x7f685406dd00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6858005880 tx=0x7f6858009040 comp rx=0 tx=0).stop 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 msgr2=0x7f68700c3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 0x7f68700c3ba0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f6860036040 tx=0x7f68600046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 shutdown_connections 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f685406b840 0x7f685406dd00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68700bdf20 0x7f68700c3ba0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700c55f0 0x7f68701518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 --2- 172.21.15.78:0/4042360744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68700ac390 0x7f68701eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 >> 172.21.15.78:0/4042360744 conn(0x7f68700ca890 msgr2=0x7f6870154060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:46.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 shutdown_connections 2024-02-19T14:55:46.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.753+0000 7f6874c46640 1 -- 172.21.15.78:0/4042360744 wait complete. 2024-02-19T14:55:46.767 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-02-19T14:55:46.103454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261484,"kb_used_data":12040,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713236,"statfs":{"total":773094113280,"available":772826353664,"internally_reserved":0,"allocated":12328960,"data_stored":6567796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001435"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958928+0000","last_change":"2024-02-19T14:55:38.971534+0000","last_active":"2024-02-19T14:55:39.958928+0000","last_peered":"2024-02-19T14:55:39.958928+0000","last_clean":"2024-02-19T14:55:39.958928+0000","last_became_active":"2024-02-19T14:55:38.971244+0000","last_became_peered":"2024-02-19T14:55:38.971244+0000","last_unstale":"2024-02-19T14:55:39.958928+0000","last_undegraded":"2024-02-19T14:55:39.958928+0000","last_fullsized":"2024-02-19T14:55:39.958928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:30.473100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957540+0000","last_change":"2024-02-19T14:55:38.978080+0000","last_active":"2024-02-19T14:55:39.957540+0000","last_peered":"2024-02-19T14:55:39.957540+0000","last_clean":"2024-02-19T14:55:39.957540+0000","last_became_active":"2024-02-19T14:55:38.977594+0000","last_became_peered":"2024-02-19T14:55:38.977594+0000","last_unstale":"2024-02-19T14:55:39.957540+0000","last_undegraded":"2024-02-19T14:55:39.957540+0000","last_fullsized":"2024-02-19T14:55:39.957540+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:05.501464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957601+0000","last_change":"2024-02-19T14:55:38.969399+0000","last_active":"2024-02-19T14:55:39.957601+0000","last_peered":"2024-02-19T14:55:39.957601+0000","last_clean":"2024-02-19T14:55:39.957601+0000","last_became_active":"2024-02-19T14:55:38.969073+0000","last_became_peered":"2024-02-19T14:55:38.969073+0000","last_unstale":"2024-02-19T14:55:39.957601+0000","last_undegraded":"2024-02-19T14:55:39.957601+0000","last_fullsized":"2024-02-19T14:55:39.957601+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:05.398022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356389+0000","last_change":"2024-02-19T14:55:38.983371+0000","last_active":"2024-02-19T14:55:40.356389+0000","last_peered":"2024-02-19T14:55:40.356389+0000","last_clean":"2024-02-19T14:55:40.356389+0000","last_became_active":"2024-02-19T14:55:38.983056+0000","last_became_peered":"2024-02-19T14:55:38.983056+0000","last_unstale":"2024-02-19T14:55:40.356389+0000","last_undegraded":"2024-02-19T14:55:40.356389+0000","last_fullsized":"2024-02-19T14:55:40.356389+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:54.003862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958970+0000","last_change":"2024-02-19T14:55:38.969344+0000","last_active":"2024-02-19T14:55:39.958970+0000","last_peered":"2024-02-19T14:55:39.958970+0000","last_clean":"2024-02-19T14:55:39.958970+0000","last_became_active":"2024-02-19T14:55:38.969054+0000","last_became_peered":"2024-02-19T14:55:38.969054+0000","last_unstale":"2024-02-19T14:55:39.958970+0000","last_undegraded":"2024-02-19T14:55:39.958970+0000","last_fullsized":"2024-02-19T14:55:39.958970+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:40.367707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972848+0000","last_change":"2024-02-19T14:55:38.982015+0000","last_active":"2024-02-19T14:55:39.972848+0000","last_peered":"2024-02-19T14:55:39.972848+0000","last_clean":"2024-02-19T14:55:39.972848+0000","last_became_active":"2024-02-19T14:55:38.981312+0000","last_became_peered":"2024-02-19T14:55:38.981312+0000","last_unstale":"2024-02-19T14:55:39.972848+0000","last_undegraded":"2024-02-19T14:55:39.972848+0000","last_fullsized":"2024-02-19T14:55:39.972848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:41.932243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972911+0000","last_change":"2024-02-19T14:55:38.981838+0000","last_active":"2024-02-19T14:55:39.972911+0000","last_peered":"2024-02-19T14:55:39.972911+0000","last_clean":"2024-02-19T14:55:39.972911+0000","last_became_active":"2024-02-19T14:55:38.981550+0000","last_became_peered":"2024-02-19T14:55:38.981550+0000","last_unstale":"2024-02-19T14:55:39.972911+0000","last_undegraded":"2024-02-19T14:55:39.972911+0000","last_fullsized":"2024-02-19T14:55:39.972911+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:53.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957038+0000","last_change":"2024-02-19T14:55:38.964142+0000","last_active":"2024-02-19T14:55:39.957038+0000","last_peered":"2024-02-19T14:55:39.957038+0000","last_clean":"2024-02-19T14:55:39.957038+0000","last_became_active":"2024-02-19T14:55:38.963902+0000","last_became_peered":"2024-02-19T14:55:38.963902+0000","last_unstale":"2024-02-19T14:55:39.957038+0000","last_undegraded":"2024-02-19T14:55:39.957038+0000","last_fullsized":"2024-02-19T14:55:39.957038+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:50.557866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356634+0000","last_change":"2024-02-19T14:55:38.979604+0000","last_active":"2024-02-19T14:55:40.356634+0000","last_peered":"2024-02-19T14:55:40.356634+0000","last_clean":"2024-02-19T14:55:40.356634+0000","last_became_active":"2024-02-19T14:55:38.978489+0000","last_became_peered":"2024-02-19T14:55:38.978489+0000","last_unstale":"2024-02-19T14:55:40.356634+0000","last_undegraded":"2024-02-19T14:55:40.356634+0000","last_fullsized":"2024-02-19T14:55:40.356634+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:48:08.828790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149479+0000","last_change":"2024-02-19T14:55:38.972564+0000","last_active":"2024-02-19T14:55:40.149479+0000","last_peered":"2024-02-19T14:55:40.149479+0000","last_clean":"2024-02-19T14:55:40.149479+0000","last_became_active":"2024-02-19T14:55:38.972290+0000","last_became_peered":"2024-02-19T14:55:38.972290+0000","last_unstale":"2024-02-19T14:55:40.149479+0000","last_undegraded":"2024-02-19T14:55:40.149479+0000","last_fullsized":"2024-02-19T14:55:40.149479+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:20:45.866375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958566+0000","last_change":"2024-02-19T14:55:38.969319+0000","last_active":"2024-02-19T14:55:39.958566+0000","last_peered":"2024-02-19T14:55:39.958566+0000","last_clean":"2024-02-19T14:55:39.958566+0000","last_became_active":"2024-02-19T14:55:38.968984+0000","last_became_peered":"2024-02-19T14:55:38.968984+0000","last_unstale":"2024-02-19T14:55:39.958566+0000","last_undegraded":"2024-02-19T14:55:39.958566+0000","last_fullsized":"2024-02-19T14:55:39.958566+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:27.052909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973181+0000","last_change":"2024-02-19T14:55:38.977770+0000","last_active":"2024-02-19T14:55:39.973181+0000","last_peered":"2024-02-19T14:55:39.973181+0000","last_clean":"2024-02-19T14:55:39.973181+0000","last_became_active":"2024-02-19T14:55:38.977457+0000","last_became_peered":"2024-02-19T14:55:38.977457+0000","last_unstale":"2024-02-19T14:55:39.973181+0000","last_undegraded":"2024-02-19T14:55:39.973181+0000","last_fullsized":"2024-02-19T14:55:39.973181+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:18.333619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149210+0000","last_change":"2024-02-19T14:55:38.968346+0000","last_active":"2024-02-19T14:55:40.149210+0000","last_peered":"2024-02-19T14:55:40.149210+0000","last_clean":"2024-02-19T14:55:40.149210+0000","last_became_active":"2024-02-19T14:55:38.968079+0000","last_became_peered":"2024-02-19T14:55:38.968079+0000","last_unstale":"2024-02-19T14:55:40.149210+0000","last_undegraded":"2024-02-19T14:55:40.149210+0000","last_fullsized":"2024-02-19T14:55:40.149210+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:30.473100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957176+0000","last_change":"2024-02-19T14:55:38.974765+0000","last_active":"2024-02-19T14:55:39.957176+0000","last_peered":"2024-02-19T14:55:39.957176+0000","last_clean":"2024-02-19T14:55:39.957176+0000","last_became_active":"2024-02-19T14:55:38.974502+0000","last_became_peered":"2024-02-19T14:55:38.974502+0000","last_unstale":"2024-02-19T14:55:39.957176+0000","last_undegraded":"2024-02-19T14:55:39.957176+0000","last_fullsized":"2024-02-19T14:55:39.957176+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:35.236954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958466+0000","last_change":"2024-02-19T14:55:38.968670+0000","last_active":"2024-02-19T14:55:39.958466+0000","last_peered":"2024-02-19T14:55:39.958466+0000","last_clean":"2024-02-19T14:55:39.958466+0000","last_became_active":"2024-02-19T14:55:38.968366+0000","last_became_peered":"2024-02-19T14:55:38.968366+0000","last_unstale":"2024-02-19T14:55:39.958466+0000","last_undegraded":"2024-02-19T14:55:39.958466+0000","last_fullsized":"2024-02-19T14:55:39.958466+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:50.444783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972713+0000","last_change":"2024-02-19T14:55:38.977677+0000","last_active":"2024-02-19T14:55:39.972713+0000","last_peered":"2024-02-19T14:55:39.972713+0000","last_clean":"2024-02-19T14:55:39.972713+0000","last_became_active":"2024-02-19T14:55:38.977394+0000","last_became_peered":"2024-02-19T14:55:38.977394+0000","last_unstale":"2024-02-19T14:55:39.972713+0000","last_undegraded":"2024-02-19T14:55:39.972713+0000","last_fullsized":"2024-02-19T14:55:39.972713+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:41.269597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149368+0000","last_change":"2024-02-19T14:55:38.972517+0000","last_active":"2024-02-19T14:55:40.149368+0000","last_peered":"2024-02-19T14:55:40.149368+0000","last_clean":"2024-02-19T14:55:40.149368+0000","last_became_active":"2024-02-19T14:55:38.972249+0000","last_became_peered":"2024-02-19T14:55:38.972249+0000","last_unstale":"2024-02-19T14:55:40.149368+0000","last_undegraded":"2024-02-19T14:55:40.149368+0000","last_fullsized":"2024-02-19T14:55:40.149368+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:32.375624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957567+0000","last_change":"2024-02-19T14:55:38.977844+0000","last_active":"2024-02-19T14:55:39.957567+0000","last_peered":"2024-02-19T14:55:39.957567+0000","last_clean":"2024-02-19T14:55:39.957567+0000","last_became_active":"2024-02-19T14:55:38.977261+0000","last_became_peered":"2024-02-19T14:55:38.977261+0000","last_unstale":"2024-02-19T14:55:39.957567+0000","last_undegraded":"2024-02-19T14:55:39.957567+0000","last_fullsized":"2024-02-19T14:55:39.957567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:55.863990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970744+0000","last_change":"2024-02-19T14:55:38.979022+0000","last_active":"2024-02-19T14:55:39.970744+0000","last_peered":"2024-02-19T14:55:39.970744+0000","last_clean":"2024-02-19T14:55:39.970744+0000","last_became_active":"2024-02-19T14:55:38.978532+0000","last_became_peered":"2024-02-19T14:55:38.978532+0000","last_unstale":"2024-02-19T14:55:39.970744+0000","last_undegraded":"2024-02-19T14:55:39.970744+0000","last_fullsized":"2024-02-19T14:55:39.970744+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:10.082698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.255987+0000","last_change":"2024-02-19T14:55:38.973466+0000","last_active":"2024-02-19T14:55:40.255987+0000","last_peered":"2024-02-19T14:55:40.255987+0000","last_clean":"2024-02-19T14:55:40.255987+0000","last_became_active":"2024-02-19T14:55:38.973126+0000","last_became_peered":"2024-02-19T14:55:38.973126+0000","last_unstale":"2024-02-19T14:55:40.255987+0000","last_undegraded":"2024-02-19T14:55:40.255987+0000","last_fullsized":"2024-02-19T14:55:40.255987+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:49:41.797576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356412+0000","last_change":"2024-02-19T14:55:38.982214+0000","last_active":"2024-02-19T14:55:40.356412+0000","last_peered":"2024-02-19T14:55:40.356412+0000","last_clean":"2024-02-19T14:55:40.356412+0000","last_became_active":"2024-02-19T14:55:38.981944+0000","last_became_peered":"2024-02-19T14:55:38.981944+0000","last_unstale":"2024-02-19T14:55:40.356412+0000","last_undegraded":"2024-02-19T14:55:40.356412+0000","last_fullsized":"2024-02-19T14:55:40.356412+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:50.557866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958729+0000","last_change":"2024-02-19T14:55:38.967976+0000","last_active":"2024-02-19T14:55:39.958729+0000","last_peered":"2024-02-19T14:55:39.958729+0000","last_clean":"2024-02-19T14:55:39.958729+0000","last_became_active":"2024-02-19T14:55:38.967132+0000","last_became_peered":"2024-02-19T14:55:38.967132+0000","last_unstale":"2024-02-19T14:55:39.958729+0000","last_undegraded":"2024-02-19T14:55:39.958729+0000","last_fullsized":"2024-02-19T14:55:39.958729+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:46:39.072118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356214+0000","last_change":"2024-02-19T14:55:38.979797+0000","last_active":"2024-02-19T14:55:40.356214+0000","last_peered":"2024-02-19T14:55:40.356214+0000","last_clean":"2024-02-19T14:55:40.356214+0000","last_became_active":"2024-02-19T14:55:38.978985+0000","last_became_peered":"2024-02-19T14:55:38.978985+0000","last_unstale":"2024-02-19T14:55:40.356214+0000","last_undegraded":"2024-02-19T14:55:40.356214+0000","last_fullsized":"2024-02-19T14:55:40.356214+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:38.730795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959489+0000","last_change":"2024-02-19T14:55:38.972126+0000","last_active":"2024-02-19T14:55:39.959489+0000","last_peered":"2024-02-19T14:55:39.959489+0000","last_clean":"2024-02-19T14:55:39.959489+0000","last_became_active":"2024-02-19T14:55:38.971843+0000","last_became_peered":"2024-02-19T14:55:38.971843+0000","last_unstale":"2024-02-19T14:55:39.959489+0000","last_undegraded":"2024-02-19T14:55:39.959489+0000","last_fullsized":"2024-02-19T14:55:39.959489+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:34.214253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959699+0000","last_change":"2024-02-19T14:55:38.976404+0000","last_active":"2024-02-19T14:55:39.959699+0000","last_peered":"2024-02-19T14:55:39.959699+0000","last_clean":"2024-02-19T14:55:39.959699+0000","last_became_active":"2024-02-19T14:55:38.976127+0000","last_became_peered":"2024-02-19T14:55:38.976127+0000","last_unstale":"2024-02-19T14:55:39.959699+0000","last_undegraded":"2024-02-19T14:55:39.959699+0000","last_fullsized":"2024-02-19T14:55:39.959699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:46.095180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.256116+0000","last_change":"2024-02-19T14:55:38.961819+0000","last_active":"2024-02-19T14:55:40.256116+0000","last_peered":"2024-02-19T14:55:40.256116+0000","last_clean":"2024-02-19T14:55:40.256116+0000","last_became_active":"2024-02-19T14:55:38.961524+0000","last_became_peered":"2024-02-19T14:55:38.961524+0000","last_unstale":"2024-02-19T14:55:40.256116+0000","last_undegraded":"2024-02-19T14:55:40.256116+0000","last_fullsized":"2024-02-19T14:55:40.256116+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:34:26.574465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970791+0000","last_change":"2024-02-19T14:55:38.978795+0000","last_active":"2024-02-19T14:55:39.970791+0000","last_peered":"2024-02-19T14:55:39.970791+0000","last_clean":"2024-02-19T14:55:39.970791+0000","last_became_active":"2024-02-19T14:55:38.978156+0000","last_became_peered":"2024-02-19T14:55:38.978156+0000","last_unstale":"2024-02-19T14:55:39.970791+0000","last_undegraded":"2024-02-19T14:55:39.970791+0000","last_fullsized":"2024-02-19T14:55:39.970791+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:13:25.023230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973217+0000","last_change":"2024-02-19T14:55:38.982376+0000","last_active":"2024-02-19T14:55:39.973217+0000","last_peered":"2024-02-19T14:55:39.973217+0000","last_clean":"2024-02-19T14:55:39.973217+0000","last_became_active":"2024-02-19T14:55:38.981596+0000","last_became_peered":"2024-02-19T14:55:38.981596+0000","last_unstale":"2024-02-19T14:55:39.973217+0000","last_undegraded":"2024-02-19T14:55:39.973217+0000","last_fullsized":"2024-02-19T14:55:39.973217+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:24.820303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.256177+0000","last_change":"2024-02-19T14:55:38.967021+0000","last_active":"2024-02-19T14:55:40.256177+0000","last_peered":"2024-02-19T14:55:40.256177+0000","last_clean":"2024-02-19T14:55:40.256177+0000","last_became_active":"2024-02-19T14:55:38.966767+0000","last_became_peered":"2024-02-19T14:55:38.966767+0000","last_unstale":"2024-02-19T14:55:40.256177+0000","last_undegraded":"2024-02-19T14:55:40.256177+0000","last_fullsized":"2024-02-19T14:55:40.256177+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:09:59.973777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957260+0000","last_change":"2024-02-19T14:55:38.968077+0000","last_active":"2024-02-19T14:55:39.957260+0000","last_peered":"2024-02-19T14:55:39.957260+0000","last_clean":"2024-02-19T14:55:39.957260+0000","last_became_active":"2024-02-19T14:55:38.967643+0000","last_became_peered":"2024-02-19T14:55:38.967643+0000","last_unstale":"2024-02-19T14:55:39.957260+0000","last_undegraded":"2024-02-19T14:55:39.957260+0000","last_fullsized":"2024-02-19T14:55:39.957260+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:19:23.327714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959803+0000","last_change":"2024-02-19T14:55:38.974569+0000","last_active":"2024-02-19T14:55:39.959803+0000","last_peered":"2024-02-19T14:55:39.959803+0000","last_clean":"2024-02-19T14:55:39.959803+0000","last_became_active":"2024-02-19T14:55:38.974303+0000","last_became_peered":"2024-02-19T14:55:38.974303+0000","last_unstale":"2024-02-19T14:55:39.959803+0000","last_undegraded":"2024-02-19T14:55:39.959803+0000","last_fullsized":"2024-02-19T14:55:39.959803+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:08:12.356752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959563+0000","last_change":"2024-02-19T14:55:38.972118+0000","last_active":"2024-02-19T14:55:39.959563+0000","last_peered":"2024-02-19T14:55:39.959563+0000","last_clean":"2024-02-19T14:55:39.959563+0000","last_became_active":"2024-02-19T14:55:38.971851+0000","last_became_peered":"2024-02-19T14:55:38.971851+0000","last_unstale":"2024-02-19T14:55:39.959563+0000","last_undegraded":"2024-02-19T14:55:39.959563+0000","last_fullsized":"2024-02-19T14:55:39.959563+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:02.225134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149455+0000","last_change":"2024-02-19T14:55:38.973632+0000","last_active":"2024-02-19T14:55:40.149455+0000","last_peered":"2024-02-19T14:55:40.149455+0000","last_clean":"2024-02-19T14:55:40.149455+0000","last_became_active":"2024-02-19T14:55:38.972837+0000","last_became_peered":"2024-02-19T14:55:38.972837+0000","last_unstale":"2024-02-19T14:55:40.149455+0000","last_undegraded":"2024-02-19T14:55:40.149455+0000","last_fullsized":"2024-02-19T14:55:40.149455+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:39:34.214253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"110'4","reported_seq":18,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970864+0000","last_change":"2024-02-19T14:55:37.967453+0000","last_active":"2024-02-19T14:55:39.970864+0000","last_peered":"2024-02-19T14:55:39.970864+0000","last_clean":"2024-02-19T14:55:39.970864+0000","last_became_active":"2024-02-19T14:55:37.967091+0000","last_became_peered":"2024-02-19T14:55:37.967091+0000","last_unstale":"2024-02-19T14:55:39.970864+0000","last_undegraded":"2024-02-19T14:55:39.970864+0000","last_fullsized":"2024-02-19T14:55:39.970864+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:00:26.048343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972955+0000","last_change":"2024-02-19T14:55:38.981039+0000","last_active":"2024-02-19T14:55:39.972955+0000","last_peered":"2024-02-19T14:55:39.972955+0000","last_clean":"2024-02-19T14:55:39.972955+0000","last_became_active":"2024-02-19T14:55:38.980747+0000","last_became_peered":"2024-02-19T14:55:38.980747+0000","last_unstale":"2024-02-19T14:55:39.972955+0000","last_undegraded":"2024-02-19T14:55:39.972955+0000","last_fullsized":"2024-02-19T14:55:39.972955+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:47.451979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149499+0000","last_change":"2024-02-19T14:55:37.974585+0000","last_active":"2024-02-19T14:55:40.149499+0000","last_peered":"2024-02-19T14:55:40.149499+0000","last_clean":"2024-02-19T14:55:40.149499+0000","last_became_active":"2024-02-19T14:55:37.974280+0000","last_became_peered":"2024-02-19T14:55:37.974280+0000","last_unstale":"2024-02-19T14:55:40.149499+0000","last_undegraded":"2024-02-19T14:55:40.149499+0000","last_fullsized":"2024-02-19T14:55:40.149499+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:18.393306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959662+0000","last_change":"2024-02-19T14:55:38.973883+0000","last_active":"2024-02-19T14:55:39.959662+0000","last_peered":"2024-02-19T14:55:39.959662+0000","last_clean":"2024-02-19T14:55:39.959662+0000","last_became_active":"2024-02-19T14:55:38.973623+0000","last_became_peered":"2024-02-19T14:55:38.973623+0000","last_unstale":"2024-02-19T14:55:39.959662+0000","last_undegraded":"2024-02-19T14:55:39.959662+0000","last_fullsized":"2024-02-19T14:55:39.959662+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:23:27.887675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356612+0000","last_change":"2024-02-19T14:55:37.971928+0000","last_active":"2024-02-19T14:55:40.356612+0000","last_peered":"2024-02-19T14:55:40.356612+0000","last_clean":"2024-02-19T14:55:40.356612+0000","last_became_active":"2024-02-19T14:55:37.971646+0000","last_became_peered":"2024-02-19T14:55:37.971646+0000","last_unstale":"2024-02-19T14:55:40.356612+0000","last_undegraded":"2024-02-19T14:55:40.356612+0000","last_fullsized":"2024-02-19T14:55:40.356612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:23:44.520278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356632+0000","last_change":"2024-02-19T14:55:38.982180+0000","last_active":"2024-02-19T14:55:40.356632+0000","last_peered":"2024-02-19T14:55:40.356632+0000","last_clean":"2024-02-19T14:55:40.356632+0000","last_became_active":"2024-02-19T14:55:38.981910+0000","last_became_peered":"2024-02-19T14:55:38.981910+0000","last_unstale":"2024-02-19T14:55:40.356632+0000","last_undegraded":"2024-02-19T14:55:40.356632+0000","last_fullsized":"2024-02-19T14:55:40.356632+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:19:23.327714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149535+0000","last_change":"2024-02-19T14:55:37.973007+0000","last_active":"2024-02-19T14:55:40.149535+0000","last_peered":"2024-02-19T14:55:40.149535+0000","last_clean":"2024-02-19T14:55:40.149535+0000","last_became_active":"2024-02-19T14:55:37.972480+0000","last_became_peered":"2024-02-19T14:55:37.972480+0000","last_unstale":"2024-02-19T14:55:40.149535+0000","last_undegraded":"2024-02-19T14:55:40.149535+0000","last_fullsized":"2024-02-19T14:55:40.149535+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:58:21.621651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149196+0000","last_change":"2024-02-19T14:55:38.967644+0000","last_active":"2024-02-19T14:55:40.149196+0000","last_peered":"2024-02-19T14:55:40.149196+0000","last_clean":"2024-02-19T14:55:40.149196+0000","last_became_active":"2024-02-19T14:55:38.967365+0000","last_became_peered":"2024-02-19T14:55:38.967365+0000","last_unstale":"2024-02-19T14:55:40.149196+0000","last_undegraded":"2024-02-19T14:55:40.149196+0000","last_fullsized":"2024-02-19T14:55:40.149196+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:11.352245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958949+0000","last_change":"2024-02-19T14:55:37.975968+0000","last_active":"2024-02-19T14:55:39.958949+0000","last_peered":"2024-02-19T14:55:39.958949+0000","last_clean":"2024-02-19T14:55:39.958949+0000","last_became_active":"2024-02-19T14:55:37.975693+0000","last_became_peered":"2024-02-19T14:55:37.975693+0000","last_unstale":"2024-02-19T14:55:39.958949+0000","last_undegraded":"2024-02-19T14:55:39.958949+0000","last_fullsized":"2024-02-19T14:55:39.958949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:14.587666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958693+0000","last_change":"2024-02-19T14:55:38.981961+0000","last_active":"2024-02-19T14:55:39.958693+0000","last_peered":"2024-02-19T14:55:39.958693+0000","last_clean":"2024-02-19T14:55:39.958693+0000","last_became_active":"2024-02-19T14:55:38.981361+0000","last_became_peered":"2024-02-19T14:55:38.981361+0000","last_unstale":"2024-02-19T14:55:39.958693+0000","last_undegraded":"2024-02-19T14:55:39.958693+0000","last_fullsized":"2024-02-19T14:55:39.958693+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:08.446452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972817+0000","last_change":"2024-02-19T14:55:37.975370+0000","last_active":"2024-02-19T14:55:39.972817+0000","last_peered":"2024-02-19T14:55:39.972817+0000","last_clean":"2024-02-19T14:55:39.972817+0000","last_became_active":"2024-02-19T14:55:37.974133+0000","last_became_peered":"2024-02-19T14:55:37.974133+0000","last_unstale":"2024-02-19T14:55:39.972817+0000","last_undegraded":"2024-02-19T14:55:39.972817+0000","last_fullsized":"2024-02-19T14:55:39.972817+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:26.904352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957261+0000","last_change":"2024-02-19T14:55:38.965771+0000","last_active":"2024-02-19T14:55:39.957261+0000","last_peered":"2024-02-19T14:55:39.957261+0000","last_clean":"2024-02-19T14:55:39.957261+0000","last_became_active":"2024-02-19T14:55:38.965299+0000","last_became_peered":"2024-02-19T14:55:38.965299+0000","last_unstale":"2024-02-19T14:55:39.957261+0000","last_undegraded":"2024-02-19T14:55:39.957261+0000","last_fullsized":"2024-02-19T14:55:39.957261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:48:08.828790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959577+0000","last_change":"2024-02-19T14:55:37.965578+0000","last_active":"2024-02-19T14:55:39.959577+0000","last_peered":"2024-02-19T14:55:39.959577+0000","last_clean":"2024-02-19T14:55:39.959577+0000","last_became_active":"2024-02-19T14:55:37.965063+0000","last_became_peered":"2024-02-19T14:55:37.965063+0000","last_unstale":"2024-02-19T14:55:39.959577+0000","last_undegraded":"2024-02-19T14:55:39.959577+0000","last_fullsized":"2024-02-19T14:55:39.959577+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:27:19.012872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149378+0000","last_change":"2024-02-19T14:55:38.966727+0000","last_active":"2024-02-19T14:55:40.149378+0000","last_peered":"2024-02-19T14:55:40.149378+0000","last_clean":"2024-02-19T14:55:40.149378+0000","last_became_active":"2024-02-19T14:55:38.966448+0000","last_became_peered":"2024-02-19T14:55:38.966448+0000","last_unstale":"2024-02-19T14:55:40.149378+0000","last_undegraded":"2024-02-19T14:55:40.149378+0000","last_fullsized":"2024-02-19T14:55:40.149378+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:13:25.023230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972820+0000","last_change":"2024-02-19T14:55:37.976412+0000","last_active":"2024-02-19T14:55:39.972820+0000","last_peered":"2024-02-19T14:55:39.972820+0000","last_clean":"2024-02-19T14:55:39.972820+0000","last_became_active":"2024-02-19T14:55:37.975831+0000","last_became_peered":"2024-02-19T14:55:37.975831+0000","last_unstale":"2024-02-19T14:55:39.972820+0000","last_undegraded":"2024-02-19T14:55:39.972820+0000","last_fullsized":"2024-02-19T14:55:39.972820+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:10.902604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970772+0000","last_change":"2024-02-19T14:55:38.977516+0000","last_active":"2024-02-19T14:55:39.970772+0000","last_peered":"2024-02-19T14:55:39.970772+0000","last_clean":"2024-02-19T14:55:39.970772+0000","last_became_active":"2024-02-19T14:55:38.977197+0000","last_became_peered":"2024-02-19T14:55:38.977197+0000","last_unstale":"2024-02-19T14:55:39.970772+0000","last_undegraded":"2024-02-19T14:55:39.970772+0000","last_fullsized":"2024-02-19T14:55:39.970772+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:32.375624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970879+0000","last_change":"2024-02-19T14:55:37.969860+0000","last_active":"2024-02-19T14:55:39.970879+0000","last_peered":"2024-02-19T14:55:39.970879+0000","last_clean":"2024-02-19T14:55:39.970879+0000","last_became_active":"2024-02-19T14:55:37.969561+0000","last_became_peered":"2024-02-19T14:55:37.969561+0000","last_unstale":"2024-02-19T14:55:39.970879+0000","last_undegraded":"2024-02-19T14:55:39.970879+0000","last_fullsized":"2024-02-19T14:55:39.970879+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:07.441886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959700+0000","last_change":"2024-02-19T14:55:38.978925+0000","last_active":"2024-02-19T14:55:39.959700+0000","last_peered":"2024-02-19T14:55:39.959700+0000","last_clean":"2024-02-19T14:55:39.959700+0000","last_became_active":"2024-02-19T14:55:38.978590+0000","last_became_peered":"2024-02-19T14:55:38.978590+0000","last_unstale":"2024-02-19T14:55:39.959700+0000","last_undegraded":"2024-02-19T14:55:39.959700+0000","last_fullsized":"2024-02-19T14:55:39.959700+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:29.055095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959752+0000","last_change":"2024-02-19T14:55:37.972249+0000","last_active":"2024-02-19T14:55:39.959752+0000","last_peered":"2024-02-19T14:55:39.959752+0000","last_clean":"2024-02-19T14:55:39.959752+0000","last_became_active":"2024-02-19T14:55:37.971802+0000","last_became_peered":"2024-02-19T14:55:37.971802+0000","last_unstale":"2024-02-19T14:55:39.959752+0000","last_undegraded":"2024-02-19T14:55:39.959752+0000","last_fullsized":"2024-02-19T14:55:39.959752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957672+0000","last_change":"2024-02-19T14:55:38.971831+0000","last_active":"2024-02-19T14:55:39.957672+0000","last_peered":"2024-02-19T14:55:39.957672+0000","last_clean":"2024-02-19T14:55:39.957672+0000","last_became_active":"2024-02-19T14:55:38.971561+0000","last_became_peered":"2024-02-19T14:55:38.971561+0000","last_unstale":"2024-02-19T14:55:39.957672+0000","last_undegraded":"2024-02-19T14:55:39.957672+0000","last_fullsized":"2024-02-19T14:55:39.957672+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:26:32.150340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"110'3","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959794+0000","last_change":"2024-02-19T14:55:37.968601+0000","last_active":"2024-02-19T14:55:39.959794+0000","last_peered":"2024-02-19T14:55:39.959794+0000","last_clean":"2024-02-19T14:55:39.959794+0000","last_became_active":"2024-02-19T14:55:37.968117+0000","last_became_peered":"2024-02-19T14:55:37.968117+0000","last_unstale":"2024-02-19T14:55:39.959794+0000","last_undegraded":"2024-02-19T14:55:39.959794+0000","last_fullsized":"2024-02-19T14:55:39.959794+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:40:23.977764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959933+0000","last_change":"2024-02-19T14:55:38.977857+0000","last_active":"2024-02-19T14:55:39.959933+0000","last_peered":"2024-02-19T14:55:39.959933+0000","last_clean":"2024-02-19T14:55:39.959933+0000","last_became_active":"2024-02-19T14:55:38.977500+0000","last_became_peered":"2024-02-19T14:55:38.977500+0000","last_unstale":"2024-02-19T14:55:39.959933+0000","last_undegraded":"2024-02-19T14:55:39.959933+0000","last_fullsized":"2024-02-19T14:55:39.959933+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:54.003862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"110'6","reported_seq":20,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957405+0000","last_change":"2024-02-19T14:55:37.968678+0000","last_active":"2024-02-19T14:55:39.957405+0000","last_peered":"2024-02-19T14:55:39.957405+0000","last_clean":"2024-02-19T14:55:39.957405+0000","last_became_active":"2024-02-19T14:55:37.968384+0000","last_became_peered":"2024-02-19T14:55:37.968384+0000","last_unstale":"2024-02-19T14:55:39.957405+0000","last_undegraded":"2024-02-19T14:55:39.957405+0000","last_fullsized":"2024-02-19T14:55:39.957405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:29.468534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970785+0000","last_change":"2024-02-19T14:55:38.968344+0000","last_active":"2024-02-19T14:55:39.970785+0000","last_peered":"2024-02-19T14:55:39.970785+0000","last_clean":"2024-02-19T14:55:39.970785+0000","last_became_active":"2024-02-19T14:55:38.968039+0000","last_became_peered":"2024-02-19T14:55:38.968039+0000","last_unstale":"2024-02-19T14:55:39.970785+0000","last_undegraded":"2024-02-19T14:55:39.970785+0000","last_fullsized":"2024-02-19T14:55:39.970785+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:23:45.524844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957392+0000","last_change":"2024-02-19T14:55:37.969264+0000","last_active":"2024-02-19T14:55:39.957392+0000","last_peered":"2024-02-19T14:55:39.957392+0000","last_clean":"2024-02-19T14:55:39.957392+0000","last_became_active":"2024-02-19T14:55:37.968958+0000","last_became_peered":"2024-02-19T14:55:37.968958+0000","last_unstale":"2024-02-19T14:55:39.957392+0000","last_undegraded":"2024-02-19T14:55:39.957392+0000","last_fullsized":"2024-02-19T14:55:39.957392+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:09.078132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149258+0000","last_change":"2024-02-19T14:55:38.977158+0000","last_active":"2024-02-19T14:55:40.149258+0000","last_peered":"2024-02-19T14:55:40.149258+0000","last_clean":"2024-02-19T14:55:40.149258+0000","last_became_active":"2024-02-19T14:55:38.976887+0000","last_became_peered":"2024-02-19T14:55:38.976887+0000","last_unstale":"2024-02-19T14:55:40.149258+0000","last_undegraded":"2024-02-19T14:55:40.149258+0000","last_fullsized":"2024-02-19T14:55:40.149258+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:40.367707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957384+0000","last_change":"2024-02-19T14:55:37.970686+0000","last_active":"2024-02-19T14:55:39.957384+0000","last_peered":"2024-02-19T14:55:39.957384+0000","last_clean":"2024-02-19T14:55:39.957384+0000","last_became_active":"2024-02-19T14:55:37.970405+0000","last_became_peered":"2024-02-19T14:55:37.970405+0000","last_unstale":"2024-02-19T14:55:39.957384+0000","last_undegraded":"2024-02-19T14:55:39.957384+0000","last_fullsized":"2024-02-19T14:55:39.957384+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:23:44.520278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.255963+0000","last_change":"2024-02-19T14:55:38.968129+0000","last_active":"2024-02-19T14:55:40.255963+0000","last_peered":"2024-02-19T14:55:40.255963+0000","last_clean":"2024-02-19T14:55:40.255963+0000","last_became_active":"2024-02-19T14:55:38.967853+0000","last_became_peered":"2024-02-19T14:55:38.967853+0000","last_unstale":"2024-02-19T14:55:40.255963+0000","last_undegraded":"2024-02-19T14:55:40.255963+0000","last_fullsized":"2024-02-19T14:55:40.255963+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:28:44.982581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957188+0000","last_change":"2024-02-19T14:55:37.969681+0000","last_active":"2024-02-19T14:55:39.957188+0000","last_peered":"2024-02-19T14:55:39.957188+0000","last_clean":"2024-02-19T14:55:39.957188+0000","last_became_active":"2024-02-19T14:55:37.969413+0000","last_became_peered":"2024-02-19T14:55:37.969413+0000","last_unstale":"2024-02-19T14:55:39.957188+0000","last_undegraded":"2024-02-19T14:55:39.957188+0000","last_fullsized":"2024-02-19T14:55:39.957188+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:39.363141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959522+0000","last_change":"2024-02-19T14:55:38.976548+0000","last_active":"2024-02-19T14:55:39.959522+0000","last_peered":"2024-02-19T14:55:39.959522+0000","last_clean":"2024-02-19T14:55:39.959522+0000","last_became_active":"2024-02-19T14:55:38.976228+0000","last_became_peered":"2024-02-19T14:55:38.976228+0000","last_unstale":"2024-02-19T14:55:39.959522+0000","last_undegraded":"2024-02-19T14:55:39.959522+0000","last_fullsized":"2024-02-19T14:55:39.959522+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:20:45.866375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149459+0000","last_change":"2024-02-19T14:55:37.973068+0000","last_active":"2024-02-19T14:55:40.149459+0000","last_peered":"2024-02-19T14:55:40.149459+0000","last_clean":"2024-02-19T14:55:40.149459+0000","last_became_active":"2024-02-19T14:55:37.972754+0000","last_became_peered":"2024-02-19T14:55:37.972754+0000","last_unstale":"2024-02-19T14:55:40.149459+0000","last_undegraded":"2024-02-19T14:55:40.149459+0000","last_fullsized":"2024-02-19T14:55:40.149459+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:55:03.280609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958777+0000","last_change":"2024-02-19T14:55:38.967934+0000","last_active":"2024-02-19T14:55:39.958777+0000","last_peered":"2024-02-19T14:55:39.958777+0000","last_clean":"2024-02-19T14:55:39.958777+0000","last_became_active":"2024-02-19T14:55:38.967240+0000","last_became_peered":"2024-02-19T14:55:38.967240+0000","last_unstale":"2024-02-19T14:55:39.958777+0000","last_undegraded":"2024-02-19T14:55:39.958777+0000","last_fullsized":"2024-02-19T14:55:39.958777+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:11.352245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.256133+0000","last_change":"2024-02-19T14:55:37.963382+0000","last_active":"2024-02-19T14:55:40.256133+0000","last_peered":"2024-02-19T14:55:40.256133+0000","last_clean":"2024-02-19T14:55:40.256133+0000","last_became_active":"2024-02-19T14:55:37.963016+0000","last_became_peered":"2024-02-19T14:55:37.963016+0000","last_unstale":"2024-02-19T14:55:40.256133+0000","last_undegraded":"2024-02-19T14:55:40.256133+0000","last_fullsized":"2024-02-19T14:55:40.256133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:14.364508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973150+0000","last_change":"2024-02-19T14:55:38.976343+0000","last_active":"2024-02-19T14:55:39.973150+0000","last_peered":"2024-02-19T14:55:39.973150+0000","last_clean":"2024-02-19T14:55:39.973150+0000","last_became_active":"2024-02-19T14:55:38.976053+0000","last_became_peered":"2024-02-19T14:55:38.976053+0000","last_unstale":"2024-02-19T14:55:39.973150+0000","last_undegraded":"2024-02-19T14:55:39.973150+0000","last_fullsized":"2024-02-19T14:55:39.973150+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:51.170159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"110'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356684+0000","last_change":"2024-02-19T14:55:37.968907+0000","last_active":"2024-02-19T14:55:40.356684+0000","last_peered":"2024-02-19T14:55:40.356684+0000","last_clean":"2024-02-19T14:55:40.356684+0000","last_became_active":"2024-02-19T14:55:37.968628+0000","last_became_peered":"2024-02-19T14:55:37.968628+0000","last_unstale":"2024-02-19T14:55:40.356684+0000","last_undegraded":"2024-02-19T14:55:40.356684+0000","last_fullsized":"2024-02-19T14:55:40.356684+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:13:24.018664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959938+0000","last_change":"2024-02-19T14:55:38.979769+0000","last_active":"2024-02-19T14:55:39.959938+0000","last_peered":"2024-02-19T14:55:39.959938+0000","last_clean":"2024-02-19T14:55:39.959938+0000","last_became_active":"2024-02-19T14:55:38.978068+0000","last_became_peered":"2024-02-19T14:55:38.978068+0000","last_unstale":"2024-02-19T14:55:39.959938+0000","last_undegraded":"2024-02-19T14:55:39.959938+0000","last_fullsized":"2024-02-19T14:55:39.959938+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:38.730795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356456+0000","last_change":"2024-02-19T14:55:37.970491+0000","last_active":"2024-02-19T14:55:40.356456+0000","last_peered":"2024-02-19T14:55:40.356456+0000","last_clean":"2024-02-19T14:55:40.356456+0000","last_became_active":"2024-02-19T14:55:37.970198+0000","last_became_peered":"2024-02-19T14:55:37.970198+0000","last_unstale":"2024-02-19T14:55:40.356456+0000","last_undegraded":"2024-02-19T14:55:40.356456+0000","last_fullsized":"2024-02-19T14:55:40.356456+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:39.363141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972728+0000","last_change":"2024-02-19T14:55:38.974338+0000","last_active":"2024-02-19T14:55:39.972728+0000","last_peered":"2024-02-19T14:55:39.972728+0000","last_clean":"2024-02-19T14:55:39.972728+0000","last_became_active":"2024-02-19T14:55:38.974076+0000","last_became_peered":"2024-02-19T14:55:38.974076+0000","last_unstale":"2024-02-19T14:55:39.972728+0000","last_undegraded":"2024-02-19T14:55:39.972728+0000","last_fullsized":"2024-02-19T14:55:39.972728+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:20.330288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972865+0000","last_change":"2024-02-19T14:55:37.970614+0000","last_active":"2024-02-19T14:55:39.972865+0000","last_peered":"2024-02-19T14:55:39.972865+0000","last_clean":"2024-02-19T14:55:39.972865+0000","last_became_active":"2024-02-19T14:55:37.970319+0000","last_became_peered":"2024-02-19T14:55:37.970319+0000","last_unstale":"2024-02-19T14:55:39.972865+0000","last_undegraded":"2024-02-19T14:55:39.972865+0000","last_fullsized":"2024-02-19T14:55:39.972865+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:01.645096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957093+0000","last_change":"2024-02-19T14:55:38.971165+0000","last_active":"2024-02-19T14:55:39.957093+0000","last_peered":"2024-02-19T14:55:39.957093+0000","last_clean":"2024-02-19T14:55:39.957093+0000","last_became_active":"2024-02-19T14:55:38.970883+0000","last_became_peered":"2024-02-19T14:55:38.970883+0000","last_unstale":"2024-02-19T14:55:39.957093+0000","last_undegraded":"2024-02-19T14:55:39.957093+0000","last_fullsized":"2024-02-19T14:55:39.957093+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:05.068820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356094+0000","last_change":"2024-02-19T14:55:37.968398+0000","last_active":"2024-02-19T14:55:40.356094+0000","last_peered":"2024-02-19T14:55:40.356094+0000","last_clean":"2024-02-19T14:55:40.356094+0000","last_became_active":"2024-02-19T14:55:37.967723+0000","last_became_peered":"2024-02-19T14:55:37.967723+0000","last_unstale":"2024-02-19T14:55:40.356094+0000","last_undegraded":"2024-02-19T14:55:40.356094+0000","last_fullsized":"2024-02-19T14:55:40.356094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:29.468534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957718+0000","last_change":"2024-02-19T14:55:38.964144+0000","last_active":"2024-02-19T14:55:39.957718+0000","last_peered":"2024-02-19T14:55:39.957718+0000","last_clean":"2024-02-19T14:55:39.957718+0000","last_became_active":"2024-02-19T14:55:38.963874+0000","last_became_peered":"2024-02-19T14:55:38.963874+0000","last_unstale":"2024-02-19T14:55:39.957718+0000","last_undegraded":"2024-02-19T14:55:39.957718+0000","last_fullsized":"2024-02-19T14:55:39.957718+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:08.361643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957198+0000","last_change":"2024-02-19T14:55:37.968061+0000","last_active":"2024-02-19T14:55:39.957198+0000","last_peered":"2024-02-19T14:55:39.957198+0000","last_clean":"2024-02-19T14:55:39.957198+0000","last_became_active":"2024-02-19T14:55:37.967758+0000","last_became_peered":"2024-02-19T14:55:37.967758+0000","last_unstale":"2024-02-19T14:55:39.957198+0000","last_undegraded":"2024-02-19T14:55:39.957198+0000","last_fullsized":"2024-02-19T14:55:39.957198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:31.371059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959720+0000","last_change":"2024-02-19T14:55:38.977835+0000","last_active":"2024-02-19T14:55:39.959720+0000","last_peered":"2024-02-19T14:55:39.959720+0000","last_clean":"2024-02-19T14:55:39.959720+0000","last_became_active":"2024-02-19T14:55:38.977537+0000","last_became_peered":"2024-02-19T14:55:38.977537+0000","last_unstale":"2024-02-19T14:55:39.959720+0000","last_undegraded":"2024-02-19T14:55:39.959720+0000","last_fullsized":"2024-02-19T14:55:39.959720+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:45:16.358780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.355962+0000","last_change":"2024-02-19T14:55:37.972341+0000","last_active":"2024-02-19T14:55:40.355962+0000","last_peered":"2024-02-19T14:55:40.355962+0000","last_clean":"2024-02-19T14:55:40.355962+0000","last_became_active":"2024-02-19T14:55:37.971999+0000","last_became_peered":"2024-02-19T14:55:37.971999+0000","last_unstale":"2024-02-19T14:55:40.355962+0000","last_undegraded":"2024-02-19T14:55:40.355962+0000","last_fullsized":"2024-02-19T14:55:40.355962+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:09.078132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973321+0000","last_change":"2024-02-19T14:55:38.979757+0000","last_active":"2024-02-19T14:55:39.973321+0000","last_peered":"2024-02-19T14:55:39.973321+0000","last_clean":"2024-02-19T14:55:39.973321+0000","last_became_active":"2024-02-19T14:55:38.979355+0000","last_became_peered":"2024-02-19T14:55:38.979355+0000","last_unstale":"2024-02-19T14:55:39.973321+0000","last_undegraded":"2024-02-19T14:55:39.973321+0000","last_fullsized":"2024-02-19T14:55:39.973321+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:18.724473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"110'2","reported_seq":16,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973055+0000","last_change":"2024-02-19T14:55:37.969752+0000","last_active":"2024-02-19T14:55:39.973055+0000","last_peered":"2024-02-19T14:55:39.973055+0000","last_clean":"2024-02-19T14:55:39.973055+0000","last_became_active":"2024-02-19T14:55:37.969493+0000","last_became_peered":"2024-02-19T14:55:37.969493+0000","last_unstale":"2024-02-19T14:55:39.973055+0000","last_undegraded":"2024-02-19T14:55:39.973055+0000","last_fullsized":"2024-02-19T14:55:39.973055+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:36:08.482046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149289+0000","last_change":"2024-02-19T14:55:38.968815+0000","last_active":"2024-02-19T14:55:40.149289+0000","last_peered":"2024-02-19T14:55:40.149289+0000","last_clean":"2024-02-19T14:55:40.149289+0000","last_became_active":"2024-02-19T14:55:38.968564+0000","last_became_peered":"2024-02-19T14:55:38.968564+0000","last_unstale":"2024-02-19T14:55:40.149289+0000","last_undegraded":"2024-02-19T14:55:40.149289+0000","last_fullsized":"2024-02-19T14:55:40.149289+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:10.082698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"111'1","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970898+0000","last_change":"2024-02-19T14:55:37.967599+0000","last_active":"2024-02-19T14:55:39.970898+0000","last_peered":"2024-02-19T14:55:39.970898+0000","last_clean":"2024-02-19T14:55:39.970898+0000","last_became_active":"2024-02-19T14:55:37.967115+0000","last_became_peered":"2024-02-19T14:55:37.967115+0000","last_unstale":"2024-02-19T14:55:39.970898+0000","last_undegraded":"2024-02-19T14:55:39.970898+0000","last_fullsized":"2024-02-19T14:55:39.970898+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:24:03.591460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356634+0000","last_change":"2024-02-19T14:55:38.979124+0000","last_active":"2024-02-19T14:55:40.356634+0000","last_peered":"2024-02-19T14:55:40.356634+0000","last_clean":"2024-02-19T14:55:40.356634+0000","last_became_active":"2024-02-19T14:55:38.978035+0000","last_became_peered":"2024-02-19T14:55:38.978035+0000","last_unstale":"2024-02-19T14:55:40.356634+0000","last_undegraded":"2024-02-19T14:55:40.356634+0000","last_fullsized":"2024-02-19T14:55:40.356634+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:43:35.236954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.970855+0000","last_change":"2024-02-19T14:55:37.966403+0000","last_active":"2024-02-19T14:55:39.970855+0000","last_peered":"2024-02-19T14:55:39.970855+0000","last_clean":"2024-02-19T14:55:39.970855+0000","last_became_active":"2024-02-19T14:55:37.966038+0000","last_became_peered":"2024-02-19T14:55:37.966038+0000","last_unstale":"2024-02-19T14:55:39.970855+0000","last_undegraded":"2024-02-19T14:55:39.970855+0000","last_fullsized":"2024-02-19T14:55:39.970855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:49.440217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.256164+0000","last_change":"2024-02-19T14:55:38.975118+0000","last_active":"2024-02-19T14:55:40.256164+0000","last_peered":"2024-02-19T14:55:40.256164+0000","last_clean":"2024-02-19T14:55:40.256164+0000","last_became_active":"2024-02-19T14:55:38.974838+0000","last_became_peered":"2024-02-19T14:55:38.974838+0000","last_unstale":"2024-02-19T14:55:40.256164+0000","last_undegraded":"2024-02-19T14:55:40.256164+0000","last_fullsized":"2024-02-19T14:55:40.256164+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:44:22.246466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973305+0000","last_change":"2024-02-19T14:55:37.974443+0000","last_active":"2024-02-19T14:55:39.973305+0000","last_peered":"2024-02-19T14:55:39.973305+0000","last_clean":"2024-02-19T14:55:39.973305+0000","last_became_active":"2024-02-19T14:55:37.974158+0000","last_became_peered":"2024-02-19T14:55:37.974158+0000","last_unstale":"2024-02-19T14:55:39.973305+0000","last_undegraded":"2024-02-19T14:55:39.973305+0000","last_fullsized":"2024-02-19T14:55:39.973305+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:55:59.039941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.973057+0000","last_change":"2024-02-19T14:55:38.977105+0000","last_active":"2024-02-19T14:55:39.973057+0000","last_peered":"2024-02-19T14:55:39.973057+0000","last_clean":"2024-02-19T14:55:39.973057+0000","last_became_active":"2024-02-19T14:55:38.974745+0000","last_became_peered":"2024-02-19T14:55:38.974745+0000","last_unstale":"2024-02-19T14:55:39.973057+0000","last_undegraded":"2024-02-19T14:55:39.973057+0000","last_fullsized":"2024-02-19T14:55:39.973057+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:20:39.439718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.959048+0000","last_change":"2024-02-19T14:55:37.958882+0000","last_active":"2024-02-19T14:55:39.959048+0000","last_peered":"2024-02-19T14:55:39.959048+0000","last_clean":"2024-02-19T14:55:39.959048+0000","last_became_active":"2024-02-19T14:55:37.958615+0000","last_became_peered":"2024-02-19T14:55:37.958615+0000","last_unstale":"2024-02-19T14:55:39.959048+0000","last_undegraded":"2024-02-19T14:55:39.959048+0000","last_fullsized":"2024-02-19T14:55:39.959048+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:28:15.143698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957135+0000","last_change":"2024-02-19T14:55:38.968676+0000","last_active":"2024-02-19T14:55:39.957135+0000","last_peered":"2024-02-19T14:55:39.957135+0000","last_clean":"2024-02-19T14:55:39.957135+0000","last_became_active":"2024-02-19T14:55:38.968402+0000","last_became_peered":"2024-02-19T14:55:38.968402+0000","last_unstale":"2024-02-19T14:55:39.957135+0000","last_undegraded":"2024-02-19T14:55:39.957135+0000","last_fullsized":"2024-02-19T14:55:39.957135+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:35.454605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.958804+0000","last_change":"2024-02-19T14:55:37.962848+0000","last_active":"2024-02-19T14:55:39.958804+0000","last_peered":"2024-02-19T14:55:39.958804+0000","last_clean":"2024-02-19T14:55:39.958804+0000","last_became_active":"2024-02-19T14:55:37.962569+0000","last_became_peered":"2024-02-19T14:55:37.962569+0000","last_unstale":"2024-02-19T14:55:39.958804+0000","last_undegraded":"2024-02-19T14:55:39.958804+0000","last_fullsized":"2024-02-19T14:55:39.958804+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:55:03.280609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356240+0000","last_change":"2024-02-19T14:55:38.977004+0000","last_active":"2024-02-19T14:55:40.356240+0000","last_peered":"2024-02-19T14:55:40.356240+0000","last_clean":"2024-02-19T14:55:40.356240+0000","last_became_active":"2024-02-19T14:55:38.976601+0000","last_became_peered":"2024-02-19T14:55:38.976601+0000","last_unstale":"2024-02-19T14:55:40.356240+0000","last_undegraded":"2024-02-19T14:55:40.356240+0000","last_fullsized":"2024-02-19T14:55:40.356240+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:32:05.068820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.972771+0000","last_change":"2024-02-19T14:55:37.976122+0000","last_active":"2024-02-19T14:55:39.972771+0000","last_peered":"2024-02-19T14:55:39.972771+0000","last_clean":"2024-02-19T14:55:39.972771+0000","last_became_active":"2024-02-19T14:55:37.975685+0000","last_became_peered":"2024-02-19T14:55:37.975685+0000","last_unstale":"2024-02-19T14:55:39.972771+0000","last_undegraded":"2024-02-19T14:55:39.972771+0000","last_fullsized":"2024-02-19T14:55:39.972771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:25:54.171047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.149379+0000","last_change":"2024-02-19T14:55:38.978865+0000","last_active":"2024-02-19T14:55:40.149379+0000","last_peered":"2024-02-19T14:55:40.149379+0000","last_clean":"2024-02-19T14:55:40.149379+0000","last_became_active":"2024-02-19T14:55:38.978580+0000","last_became_peered":"2024-02-19T14:55:38.978580+0000","last_unstale":"2024-02-19T14:55:40.149379+0000","last_undegraded":"2024-02-19T14:55:40.149379+0000","last_fullsized":"2024-02-19T14:55:40.149379+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:23:45.524844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.355974+0000","last_change":"2024-02-19T14:55:37.968205+0000","last_active":"2024-02-19T14:55:40.355974+0000","last_peered":"2024-02-19T14:55:40.355974+0000","last_clean":"2024-02-19T14:55:40.355974+0000","last_became_active":"2024-02-19T14:55:37.967904+0000","last_became_peered":"2024-02-19T14:55:37.967904+0000","last_unstale":"2024-02-19T14:55:40.355974+0000","last_undegraded":"2024-02-19T14:55:40.355974+0000","last_fullsized":"2024-02-19T14:55:40.355974+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:31.371059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356448+0000","last_change":"2024-02-19T14:55:38.982868+0000","last_active":"2024-02-19T14:55:40.356448+0000","last_peered":"2024-02-19T14:55:40.356448+0000","last_clean":"2024-02-19T14:55:40.356448+0000","last_became_active":"2024-02-19T14:55:38.982602+0000","last_became_peered":"2024-02-19T14:55:38.982602+0000","last_unstale":"2024-02-19T14:55:40.356448+0000","last_undegraded":"2024-02-19T14:55:40.356448+0000","last_fullsized":"2024-02-19T14:55:40.356448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:37.947129+0000","last_clean_scrub_stamp":"2024-02-19T14:55:37.947129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:35.454605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"110'8","reported_seq":22,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:39.957018+0000","last_change":"2024-02-19T14:55:37.969453+0000","last_active":"2024-02-19T14:55:39.957018+0000","last_peered":"2024-02-19T14:55:39.957018+0000","last_clean":"2024-02-19T14:55:39.957018+0000","last_became_active":"2024-02-19T14:55:37.968536+0000","last_became_peered":"2024-02-19T14:55:37.968536+0000","last_unstale":"2024-02-19T14:55:39.957018+0000","last_undegraded":"2024-02-19T14:55:39.957018+0000","last_fullsized":"2024-02-19T14:55:39.957018+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:55:36.942564+0000","last_clean_scrub_stamp":"2024-02-19T14:55:36.942564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:10.347679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":341,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-02-19T14:55:40.356345+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:55:40.356345+0000","last_peered":"2024-02-19T14:55:40.356345+0000","last_clean":"2024-02-19T14:55:40.356345+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:55:40.356345+0000","last_undegraded":"2024-02-19T14:55:40.356345+0000","last_fullsized":"2024-02-19T14:55:40.356345+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1994752,"data_stored":1263924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46700000000000003,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.626,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71099999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.59599999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76200000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.501,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72899999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.50800000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77900000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.34499999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.5,"15min":0.502},"min":{"1min":0.439,"5min":0.311,"15min":0.311},"max":{"1min":0.73699999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.44},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.75800000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41499999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59599999999999997,"15min":0.60499999999999998},"min":{"1min":0.43099999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82799999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.51700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1388544,"data_stored":673362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68100000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.499}]},{"osd":1,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69899999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56699999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68899999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.442}]},{"osd":3,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66500000000000004,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.62}]},{"osd":5,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.315,"15min":0.315},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:54:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.46000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.41799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78100000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.751}]}]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1392640,"data_stored":672940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.505,"15min":0.49399999999999999},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72599999999999998,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.71399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.71699999999999997,"5min":0.872,"15min":0.872},"last":0.432},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84499999999999997,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.64500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.55000000000000004,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.77300000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55200000000000005}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.748,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.503},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.629}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.39500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.69799999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1396736,"data_stored":674392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78000000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.434}]},{"osd":1,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.623,"15min":0.627},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76900000000000002,"5min":1.038,"15min":1.038},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.34200000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79000000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.39000000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79000000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.379}]},{"osd":6,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56399999999999995,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92500000000000004,"5min":1.038,"15min":1.038},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.39900000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86199999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.501},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.40300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60999999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1400832,"data_stored":674426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.33300000000000002,"15min":0.307},"min":{"1min":0.186,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.694,"5min":3.694,"15min":3.694},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.35799999999999998,"15min":0.32100000000000001},"min":{"1min":0.22,"5min":0.189,"15min":0.189},"max":{"1min":4.835,"5min":4.835,"15min":4.835},"last":0.315}]},{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.314,"15min":0.311},"min":{"1min":0.20200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":6.3810000000000002,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.40799999999999997,"15min":0.38400000000000001},"min":{"1min":0.29699999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6320000000000001,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.38300000000000001,"15min":0.34499999999999997},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.9889999999999999,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.497},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.61799999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":3.6669999999999998,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.629,"15min":0.61099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.65700000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.45800000000000002,"5min":0.439,"15min":0.439},"max":{"1min":3.7189999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0760000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.56499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1384448,"data_stored":672780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.32000000000000001,"15min":0.312},"min":{"1min":0.182,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47599999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":0.29599999999999999,"15min":0.29699999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.39200000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.312}]},{"osd":1,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.373,"15min":0.373},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.40400000000000003,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.45900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88500000000000001,"5min":1.236,"15min":1.236},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.90200000000000002,"5min":0.995,"15min":0.995},"last":0.52600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1388544,"data_stored":672802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28399999999999997,"5min":0.32400000000000001,"15min":0.33400000000000002},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.251},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32700000000000001,"15min":0.33700000000000002},"min":{"1min":0.21099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.42399999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.30399999999999999,"15min":0.308},"min":{"1min":0.20699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.53500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51400000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.37},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.41799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.313,"5min":0.309,"15min":0.309},"max":{"1min":0.68000000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73899999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53200000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56499999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70099999999999996,"5min":1.01,"15min":1.01},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67700000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75800000000000001,"5min":1.026,"15min":1.026},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73399999999999999,"5min":0.996,"15min":0.996},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.5},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.60499999999999998},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.54600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1982464,"data_stored":1263170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.193,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.42499999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.371,"5min":0.44500000000000001,"15min":0.44500000000000001},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.40799999999999997,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.436,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.248},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.499,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.317}]},{"osd":4,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.58199999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72699999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66300000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58999999999999997,"15min":0.61199999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88900000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.5}]},{"osd":6,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.59499999999999997,"15min":0.60499999999999998},"min":{"1min":0.45800000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82399999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:54:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59099999999999997,"15min":0.61099999999999999},"min":{"1min":0.41399999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.91600000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57999999999999996,"15min":0.59399999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73499999999999999,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:55:46.772 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:55:46.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:55:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.911+0000 7f2f7612c640 1 Processor -- start 2024-02-19T14:55:46.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.911+0000 7f2f7612c640 1 -- start start 2024-02-19T14:55:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f7612c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2f701496b0 0x7f2f70153900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f7612c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700a9480 con 0x7f2f700c7bf0 2024-02-19T14:55:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f7612c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700a9720 con 0x7f2f7006f380 2024-02-19T14:55:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f7612c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f700a9de0 con 0x7f2f701496b0 2024-02-19T14:55:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f6f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2f701496b0 0x7f2f70153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f67fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2f700c7bf0 0x7f2f700c4330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60626/0 (socket says 172.21.15.78:60626) 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f6f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2f701496b0 0x7f2f70153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40524/0 (socket says 172.21.15.78:40524) 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.912+0000 7f2f67fff640 1 -- 172.21.15.78:0/922056423 learned_addr learned my addr 172.21.15.78:0/922056423 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.913+0000 7f2f6f7fe640 1 -- 172.21.15.78:0/922056423 >> v1:172.21.15.150:6789/0 conn(0x7f2f7006f380 legacy=0x7f2f7006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.913+0000 7f2f6f7fe640 1 -- 172.21.15.78:0/922056423 >> v1:172.21.15.78:6789/0 conn(0x7f2f700c7bf0 legacy=0x7f2f700c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.913+0000 7f2f6f7fe640 1 -- 172.21.15.78:0/922056423 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700a9a00 con 0x7f2f701496b0 2024-02-19T14:55:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.913+0000 7f2f6effd640 1 -- 172.21.15.78:0/922056423 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709926802 0 0) 0x7f2f700a9480 con 0x7f2f700c7bf0 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.913+0000 7f2f6f7fe640 1 --2- 172.21.15.78:0/922056423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70153900 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f2f600097a0 tx=0x7f2f6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9759e568c09780d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f6effd640 1 -- 172.21.15.78:0/922056423 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60031830 con 0x7f2f701496b0 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f6effd640 1 -- 172.21.15.78:0/922056423 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60031990 con 0x7f2f701496b0 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f6effd640 1 -- 172.21.15.78:0/922056423 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60031d20 con 0x7f2f701496b0 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 -- 172.21.15.78:0/922056423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 msgr2=0x7f2f70153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 --2- 172.21.15.78:0/922056423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70153900 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f2f600097a0 tx=0x7f2f6002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 -- 172.21.15.78:0/922056423 shutdown_connections 2024-02-19T14:55:46.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 --2- 172.21.15.78:0/922056423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70153900 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 -- 172.21.15.78:0/922056423 >> 172.21.15.78:0/922056423 conn(0x7f2f7006ea10 msgr2=0x7f2f700a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 -- 172.21.15.78:0/922056423 shutdown_connections 2024-02-19T14:55:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.914+0000 7f2f7612c640 1 -- 172.21.15.78:0/922056423 wait complete. 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.915+0000 7f2f7612c640 1 Processor -- start 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.915+0000 7f2f7612c640 1 -- start start 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f7612c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f7006f380 0x7f2f701ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f7612c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f700c7bf0 0x7f2f7014ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f7612c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f7612c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7014bba0 con 0x7f2f701496b0 2024-02-19T14:55:46.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f7612c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7014b590 con 0x7f2f700c7bf0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f7612c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f701ec4e0 con 0x7f2f7006f380 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70145a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40540/0 (socket says 172.21.15.78:40540) 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 -- 172.21.15.78:0/57684907 learned_addr learned my addr 172.21.15.78:0/57684907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 -- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f7006f380 msgr2=0x7f2f701ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6f7fe640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f7006f380 0x7f2f701ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f67fff640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f700c7bf0 0x7f2f7014ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f7006f380 0x7f2f701ebba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 -- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f700c7bf0 msgr2=0x7f2f7014ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f700c7bf0 0x7f2f7014ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.916+0000 7f2f6ffff640 1 -- 172.21.15.78:0/57684907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60009400 con 0x7f2f701496b0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.917+0000 7f2f6ffff640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70145a50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c6b0 tx=0x7f2f6800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.917+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68003b70 con 0x7f2f701496b0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.917+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f68003cf0 con 0x7f2f701496b0 2024-02-19T14:55:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.917+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7014b830 con 0x7f2f701496b0 2024-02-19T14:55:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.917+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68003ea0 con 0x7f2f701496b0 2024-02-19T14:55:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.917+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f70068180 con 0x7f2f701496b0 2024-02-19T14:55:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.918+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70074990 con 0x7f2f701496b0 2024-02-19T14:55:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.918+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2f68005310 con 0x7f2f701496b0 2024-02-19T14:55:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.919+0000 7f2f6cff9640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f5006b840 0x7f2f5006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.919+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2f6808a060 con 0x7f2f701496b0 2024-02-19T14:55:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.919+0000 7f2f6f7fe640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f5006b840 0x7f2f5006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.919+0000 7f2f6f7fe640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f5006b840 0x7f2f5006dd00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2f60036000 tx=0x7f2f60008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:46.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:46.923+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2f68054980 con 0x7f2f701496b0 2024-02-19T14:55:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.173+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2f700a8660 con 0x7f2f701496b0 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.175+0000 7f2f6cff9640 1 -- 172.21.15.78:0/57684907 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2f68058040 con 0x7f2f701496b0 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:55:47.178 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:55:47.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f5006b840 msgr2=0x7f2f5006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f5006b840 0x7f2f5006dd00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2f60036000 tx=0x7f2f60008030 comp rx=0 tx=0).stop 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 msgr2=0x7f2f70145a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70145a50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c6b0 tx=0x7f2f6800cb70 comp rx=0 tx=0).stop 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 shutdown_connections 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2f5006b840 0x7f2f5006dd00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f701496b0 0x7f2f70145a50 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f700c7bf0 0x7f2f7014ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 --2- 172.21.15.78:0/57684907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2f7006f380 0x7f2f701ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 >> 172.21.15.78:0/57684907 conn(0x7f2f7006ea10 msgr2=0x7f2f7006f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 shutdown_connections 2024-02-19T14:55:47.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.178+0000 7f2f7612c640 1 -- 172.21.15.78:0/57684907 wait complete. 2024-02-19T14:55:47.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:47.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.290+0000 7fdc02f5e640 1 Processor -- start 2024-02-19T14:55:47.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.290+0000 7fdc02f5e640 1 -- start start 2024-02-19T14:55:47.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc02f5e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc150460 0x7fdbfc0c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:47.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc02f5e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdbfc0721a0 0x7fdbfc0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:47.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc02f5e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc072c60 con 0x7fdbfc0c7d90 2024-02-19T14:55:47.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc02f5e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbfc153e80 con 0x7fdbfc0721a0 2024-02-19T14:55:47.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc02f5e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1538c0 con 0x7fdbfc150460 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdbfbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdbfc0721a0 0x7fdbfc0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdbfbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdbfc0721a0 0x7fdbfc0c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40556/0 (socket says 172.21.15.78:40556) 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc014fb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdbfc0c7d90 0x7fdbfc0c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60628/0 (socket says 172.21.15.78:60628) 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdbfbfff640 1 -- 172.21.15.78:0/203732801 learned_addr learned my addr 172.21.15.78:0/203732801 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdc00cfa640 1 --2- 172.21.15.78:0/203732801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc150460 0x7fdbfc0c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdbfbfff640 1 -- 172.21.15.78:0/203732801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc150460 msgr2=0x7fdbfc0c1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.291+0000 7fdbfbfff640 1 --2- 172.21.15.78:0/203732801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc150460 0x7fdbfc0c1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfbfff640 1 -- 172.21.15.78:0/203732801 >> v1:172.21.15.78:6789/0 conn(0x7fdbfc0c7d90 legacy=0x7fdbfc0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfbfff640 1 -- 172.21.15.78:0/203732801 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc153ba0 con 0x7fdbfc0721a0 2024-02-19T14:55:47.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfb7fe640 1 -- 172.21.15.78:0/203732801 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208631898 0 0) 0x7fdbfc072c60 con 0x7fdbfc0c7d90 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfbfff640 1 --2- 172.21.15.78:0/203732801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c7850 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009870 tx=0x7fdbe402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcf92feaa3cd33f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfb7fe640 1 -- 172.21.15.78:0/203732801 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4030840 con 0x7fdbfc0721a0 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfb7fe640 1 -- 172.21.15.78:0/203732801 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe40309a0 con 0x7fdbfc0721a0 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.292+0000 7fdbfb7fe640 1 -- 172.21.15.78:0/203732801 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4030d30 con 0x7fdbfc0721a0 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 -- 172.21.15.78:0/203732801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc0721a0 msgr2=0x7fdbfc0c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/203732801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c7850 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009870 tx=0x7fdbe402b980 comp rx=0 tx=0).stop 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 -- 172.21.15.78:0/203732801 shutdown_connections 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/203732801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c7850 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/203732801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc150460 0x7fdbfc0c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 -- 172.21.15.78:0/203732801 >> 172.21.15.78:0/203732801 conn(0x7fdbfc06c170 msgr2=0x7fdbfc06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:47.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 -- 172.21.15.78:0/203732801 shutdown_connections 2024-02-19T14:55:47.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.293+0000 7fdc02f5e640 1 -- 172.21.15.78:0/203732801 wait complete. 2024-02-19T14:55:47.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.294+0000 7fdc02f5e640 1 Processor -- start 2024-02-19T14:55:47.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.294+0000 7fdc02f5e640 1 -- start start 2024-02-19T14:55:47.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc02f5e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:47.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc02f5e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc0c7d90 0x7fdbfc152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:47.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc02f5e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 0x7fdbfc1eaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:47.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc02f5e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0aad40 con 0x7fdbfc150460 2024-02-19T14:55:47.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc02f5e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc157310 con 0x7fdbfc0721a0 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc02f5e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc1ee9d0 con 0x7fdbfc0c7d90 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 0x7fdbfc1eaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 0x7fdbfc1eaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40568/0 (socket says 172.21.15.78:40568) 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 -- 172.21.15.78:0/4119802216 learned_addr learned my addr 172.21.15.78:0/4119802216 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc00cfa640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdbfbfff640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc0c7d90 0x7fdbfc152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 -- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc0c7d90 msgr2=0x7fdbfc152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc0c7d90 0x7fdbfc152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 -- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbfc0721a0 msgr2=0x7fdbfc0c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.295+0000 7fdc014fb640 1 -- 172.21.15.78:0/4119802216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40094d0 con 0x7fdbfc150460 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.296+0000 7fdc014fb640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 0x7fdbfc1eaa50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00c450 tx=0x7fdbec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.296+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0074e0 con 0x7fdbfc150460 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.296+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec007660 con 0x7fdbfc150460 2024-02-19T14:55:47.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.296+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc1eecb0 con 0x7fdbfc150460 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.296+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec004a20 con 0x7fdbfc150460 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.296+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc0ac0f0 con 0x7fdbfc150460 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.297+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc070150 con 0x7fdbfc150460 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.297+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdbec004b80 con 0x7fdbfc150460 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.298+0000 7fdbf97fa640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbdc06b810 0x7fdbdc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.298+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbec089760 con 0x7fdbfc150460 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.298+0000 7fdc00cfa640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbdc06b810 0x7fdbdc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:47.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.298+0000 7fdc00cfa640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbdc06b810 0x7fdbdc06dcd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4003b80 tx=0x7fdbf40098d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:47.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.302+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdbec0540a0 con 0x7fdbfc150460 2024-02-19T14:55:47.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.515+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbfc074990 con 0x7fdbfc150460 2024-02-19T14:55:47.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.516+0000 7fdbf97fa640 1 -- 172.21.15.78:0/4119802216 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fdbec057760 con 0x7fdbfc150460 2024-02-19T14:55:47.518 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:47.518 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5927,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:40.368516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2978708219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2978708219},{"type":"v1","addr":"172.21.15.78:6837","nonce":2978708219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:47.519 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 84 2024-02-19T14:55:47.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbdc06b810 msgr2=0x7fdbdc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbdc06b810 0x7fdbdc06dcd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4003b80 tx=0x7fdbf40098d0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 msgr2=0x7fdbfc1eaa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 0x7fdbfc1eaa50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00c450 tx=0x7fdbec00c910 comp rx=0 tx=0).stop 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 shutdown_connections 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbdc06b810 0x7fdbdc06dcd0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbfc150460 0x7fdbfc1eaa50 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbfc0c7d90 0x7fdbfc152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 --2- 172.21.15.78:0/4119802216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbfc0721a0 0x7fdbfc0c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 >> 172.21.15.78:0/4119802216 conn(0x7fdbfc06c170 msgr2=0x7fdbfc072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 shutdown_connections 2024-02-19T14:55:47.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:47.518+0000 7fdc02f5e640 1 -- 172.21.15.78:0/4119802216 wait complete. 2024-02-19T14:55:47.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:55:38.311785+0000', 'modified': '2024-02-19T14:55:40.368516+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7305': {'gid': 7305, 'name': 'c', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/2978708219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2978708219}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2978708219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:47.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:47.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:55:48.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.033+0000 7fc39867b640 1 Processor -- start 2024-02-19T14:55:48.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.033+0000 7fc39867b640 1 -- start start 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc39867b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3900719e0 0x7fc3900c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc39867b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900c3d90 0x7fc3900c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc39867b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39006ff90 con 0x7fc39014d210 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc39867b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc390070270 con 0x7fc3900719e0 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc39867b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900c3970 con 0x7fc3900c3d90 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3900719e0 0x7fc3900c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3900719e0 0x7fc3900c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33346/0 (socket says 172.21.15.78:33346) 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 -- 172.21.15.78:0/646265055 learned_addr learned my addr 172.21.15.78:0/646265055 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 -- 172.21.15.78:0/646265055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900c3d90 msgr2=0x7fc3900c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 --2- 172.21.15.78:0/646265055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900c3d90 0x7fc3900c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 -- 172.21.15.78:0/646265055 >> v1:172.21.15.150:6789/0 conn(0x7fc39014d210 legacy=0x7fc3900c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.034+0000 7fc395c16640 1 -- 172.21.15.78:0/646265055 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc390156fc0 con 0x7fc3900719e0 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.035+0000 7fc395c16640 1 --2- 172.21.15.78:0/646265055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3900719e0 0x7fc3900c64d0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fc384009870 tx=0x7fc38402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4ada716f6e9d67 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.035+0000 7fc395415640 1 -- 172.21.15.78:0/646265055 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840317e0 con 0x7fc3900719e0 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.035+0000 7fc395415640 1 -- 172.21.15.78:0/646265055 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384031940 con 0x7fc3900719e0 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.035+0000 7fc395415640 1 -- 172.21.15.78:0/646265055 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384031cd0 con 0x7fc3900719e0 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 -- 172.21.15.78:0/646265055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3900719e0 msgr2=0x7fc3900c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 --2- 172.21.15.78:0/646265055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3900719e0 0x7fc3900c64d0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fc384009870 tx=0x7fc38402b880 comp rx=0 tx=0).stop 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 -- 172.21.15.78:0/646265055 shutdown_connections 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 --2- 172.21.15.78:0/646265055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900c3d90 0x7fc3900c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 --2- 172.21.15.78:0/646265055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3900719e0 0x7fc3900c64d0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 -- 172.21.15.78:0/646265055 >> 172.21.15.78:0/646265055 conn(0x7fc3900c5050 msgr2=0x7fc3900c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 -- 172.21.15.78:0/646265055 shutdown_connections 2024-02-19T14:55:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.036+0000 7fc39867b640 1 -- 172.21.15.78:0/646265055 wait complete. 2024-02-19T14:55:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.038+0000 7fc39867b640 1 Processor -- start 2024-02-19T14:55:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.038+0000 7fc39867b640 1 -- start start 2024-02-19T14:55:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900719e0 0x7fc3900be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3900c3d90 0x7fc3901ec120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 0x7fc3901ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc390153e80 con 0x7fc39014d210 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900c0190 con 0x7fc3900c3d90 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3901eccd0 con 0x7fc3900719e0 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 0x7fc3901ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 0x7fc3901ec660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33358/0 (socket says 172.21.15.78:33358) 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 -- 172.21.15.78:0/837673012 learned_addr learned my addr 172.21.15.78:0/837673012 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396417640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900719e0 0x7fc3900be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc395c16640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3900c3d90 0x7fc3901ec120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 -- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900719e0 msgr2=0x7fc3900be9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900719e0 0x7fc3900be9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 -- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3900c3d90 msgr2=0x7fc3901ec120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3900c3d90 0x7fc3901ec120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 -- 172.21.15.78:0/837673012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3840094d0 con 0x7fc39014d210 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396417640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900719e0 0x7fc3900be9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc395c16640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3900c3d90 0x7fc3901ec120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc396c18640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 0x7fc3901ec660 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fc38000b3f0 tx=0x7fc38000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc38effd640 1 -- 172.21.15.78:0/837673012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380011830 con 0x7fc39014d210 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc38effd640 1 -- 172.21.15.78:0/837673012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3800119b0 con 0x7fc39014d210 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc38effd640 1 -- 172.21.15.78:0/837673012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380011d60 con 0x7fc39014d210 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.039+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3901ecf70 con 0x7fc39014d210 2024-02-19T14:55:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.040+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3901dfbf0 con 0x7fc39014d210 2024-02-19T14:55:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.041+0000 7fc38effd640 1 -- 172.21.15.78:0/837673012 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc38001a400 con 0x7fc39014d210 2024-02-19T14:55:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.041+0000 7fc38effd640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc37c06b840 0x7fc37c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.041+0000 7fc38effd640 1 -- 172.21.15.78:0/837673012 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc38008a4f0 con 0x7fc39014d210 2024-02-19T14:55:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.041+0000 7fc396417640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc37c06b840 0x7fc37c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.042+0000 7fc396417640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc37c06b840 0x7fc37c06dd00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc388007eb0 tx=0x7fc3880073c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 Processor -- start 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 -- start start 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593509308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593508f5ae0 0x55935092fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 0x5593508f4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593508afcd0 con 0x5593509328b0 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559350862f20 con 0x5593508f5ae0 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.043+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559350860450 con 0x55935092f740 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 0x5593508f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 0x5593508f4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33368/0 (socket says 172.21.15.78:33368) 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 -- 172.21.15.78:0/1424659364 learned_addr learned my addr 172.21.15.78:0/1424659364 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396417640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593509308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc395c16640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593508f5ae0 0x55935092fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 -- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 msgr2=0x5593509308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593509308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 -- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593508f5ae0 msgr2=0x55935092fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593508f5ae0 0x55935092fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 -- 172.21.15.78:0/1424659364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559350810f20 con 0x5593509328b0 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc395c16640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593508f5ae0 0x55935092fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396417640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593509308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc396c18640 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 0x5593508f4860 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fc380060c40 tx=0x7fc380058a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7538fa270a7c04c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc377fff640 1 -- 172.21.15.78:0/1424659364 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380056d60 con 0x5593509328b0 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc377fff640 1 -- 172.21.15.78:0/1424659364 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3800550d0 con 0x5593509328b0 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.044+0000 7fc377fff640 1 -- 172.21.15.78:0/1424659364 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38008a2d0 con 0x5593509328b0 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 -- 172.21.15.78:0/1424659364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 msgr2=0x5593508f4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 0x5593508f4860 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fc380060c40 tx=0x7fc380058a60 comp rx=0 tx=0).stop 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 -- 172.21.15.78:0/1424659364 shutdown_connections 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5593509328b0 0x5593508f4860 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593508f5ae0 0x55935092fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1424659364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593509308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 -- 172.21.15.78:0/1424659364 >> 172.21.15.78:0/1424659364 conn(0x5593508f1120 msgr2=0x559350915b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 -- 172.21.15.78:0/1424659364 shutdown_connections 2024-02-19T14:55:48.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.045+0000 7fc39b24c740 1 -- 172.21.15.78:0/1424659364 wait complete. 2024-02-19T14:55:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 -- start start 2024-02-19T14:55:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 Processor -- start 2024-02-19T14:55:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593508f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x5593509123d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350944640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55935085bc30 con 0x55935092d6d0 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559350800d30 con 0x5593509328b0 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.046+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55935052f310 con 0x55935092f740 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc39b24c740 10 client.0 _ll_register_callbacks cb 0x559350808d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350944640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350944640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33374/0 (socket says 172.21.15.78:33374) 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 -- 172.21.15.78:0/3144246500 learned_addr learned my addr 172.21.15.78:0/3144246500 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc395c16640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x5593509123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396417640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593508f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 -- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 msgr2=0x5593508f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593508f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 -- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 msgr2=0x5593509123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x5593509123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 -- 172.21.15.78:0/3144246500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3901dfbf0 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc396c18640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350944640 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fc3800589a0 tx=0x7fc3800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2e85265f98a0eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc3757fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc3757fa640 1 -- 172.21.15.78:0/3144246500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38008a2d0 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc3757fa640 1 -- 172.21.15.78:0/3144246500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc380083c70 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55935055b6c0 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc3757fa640 1 -- 172.21.15.78:0/3144246500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc380083e70 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc39b24c740 10 client.7479 fetch_fsmap learned FSMap version 84 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.047+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55935085a2b0 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc3757fa640 1 -- 172.21.15.78:0/3144246500 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55935055b6c0 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc3757fa640 1 -- 172.21.15.78:0/3144246500 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc380057eb0 con 0x55935092d6d0 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc39b24c740 10 client.7479 fetch_fsmap finished waiting for FSMap version 84 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc39b24c740 10 client.7479 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc39b24c740 20 client.7479 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x559350944e30 0x559350927a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc39b24c740 4 client.7479 mds_command: new command op to 7305 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 --> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593505ac270 con 0x559350944e30 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc396417640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x559350944e30 0x559350927a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc3757fa640 10 client.7479 ms_handle_connect on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.048+0000 7fc396417640 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x559350944e30 0x559350927a70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0c8949dbe0a9233 server_cookie=3566a35875423b28 in_seq=0 out_seq=0 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc3757fa640 1 -- 172.21.15.78:0/3144246500 <== mds.0 v2:172.21.15.78:6835/2978708219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5593505ac270 con 0x559350944e30 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc3757fa640 10 client.7479 handle_command_reply: tid=0 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 client.7479 shutdown 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 msgr2=0x559350944640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350944640 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fc3800589a0 tx=0x7fc3800046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 shutdown_connections 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x559350944e30 0x559350927a70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350944640 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x5593509123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3144246500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092f740 0x5593508f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc3757fa640 0 client.7479 ms_handle_reset on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 >> 172.21.15.78:0/3144246500 conn(0x5593508f5ae0 msgr2=0x559350915f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 shutdown_connections 2024-02-19T14:55:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 1 -- 172.21.15.78:0/3144246500 wait complete. 2024-02-19T14:55:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.049+0000 7fc39b24c740 20 client.7479 trim_cache size 0 max 16384 2024-02-19T14:55:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 Processor -- start 2024-02-19T14:55:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 -- start start 2024-02-19T14:55:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092d6d0 0x55935093e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 0x559350930ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x559350924910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593507e1490 con 0x55935092f740 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55935085add0 con 0x5593509328b0 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559350941f70 con 0x55935092d6d0 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc395c16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 0x559350930ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc395c16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 0x559350930ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33390/0 (socket says 172.21.15.78:33390) 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.066+0000 7fc395c16640 1 -- 172.21.15.78:0/1295835422 learned_addr learned my addr 172.21.15.78:0/1295835422 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc396417640 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092d6d0 0x55935093e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc396c18640 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x559350924910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc395c16640 1 -- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092d6d0 msgr2=0x55935093e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc395c16640 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092d6d0 0x55935093e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc395c16640 1 -- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 msgr2=0x559350924910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc395c16640 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x559350924910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc395c16640 1 -- 172.21.15.78:0/1295835422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593507f6a20 con 0x55935092f740 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc395c16640 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 0x559350930ae0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc384009990 tx=0x7fc384030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f424c7df65b05a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc3777fe640 1 -- 172.21.15.78:0/1295835422 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384031cb0 con 0x55935092f740 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc3777fe640 1 -- 172.21.15.78:0/1295835422 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384031e10 con 0x55935092f740 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc3777fe640 1 -- 172.21.15.78:0/1295835422 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840456a0 con 0x55935092f740 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 -- 172.21.15.78:0/1295835422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 msgr2=0x559350930ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 0x559350930ae0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc384009990 tx=0x7fc384030000 comp rx=0 tx=0).stop 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 -- 172.21.15.78:0/1295835422 shutdown_connections 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092f740 0x559350930ae0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593509328b0 0x559350924910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 --2- 172.21.15.78:0/1295835422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55935092d6d0 0x55935093e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.067+0000 7fc39b24c740 1 -- 172.21.15.78:0/1295835422 >> 172.21.15.78:0/1295835422 conn(0x5593508f5ae0 msgr2=0x5593508f49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.068+0000 7fc39b24c740 1 -- 172.21.15.78:0/1295835422 shutdown_connections 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.068+0000 7fc39b24c740 1 -- 172.21.15.78:0/1295835422 wait complete. 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.068+0000 7fc39b24c740 1 -- start start 2024-02-19T14:55:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.068+0000 7fc39b24c740 1 Processor -- start 2024-02-19T14:55:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350934180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935092f740 0x5593508eede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5593509328b0 0x5593509a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559350925190 con 0x55935092d6d0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559350931290 con 0x55935092f740 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559350925c50 con 0x5593509328b0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc39b24c740 10 client.0 _ll_register_callbacks cb 0x7fc38006f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350934180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350934180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33398/0 (socket says 172.21.15.78:33398) 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 -- 172.21.15.78:0/3761530101 learned_addr learned my addr 172.21.15.78:0/3761530101 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc395c16640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935092f740 0x5593508eede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396c18640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5593509328b0 0x5593509a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 -- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5593509328b0 msgr2=0x5593509a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5593509328b0 0x5593509a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 -- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935092f740 msgr2=0x5593508eede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935092f740 0x5593508eede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.069+0000 7fc396417640 1 -- 172.21.15.78:0/3761530101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc390156fc0 con 0x55935092d6d0 2024-02-19T14:55:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc396417640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350934180 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc38800cb20 tx=0x7fc3880049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c76aa4373706fc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc3757fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc3757fa640 1 -- 172.21.15.78:0/3761530101 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388027a10 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc3757fa640 1 -- 172.21.15.78:0/3761530101 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc388023070 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3800550d0 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc3757fa640 1 -- 172.21.15.78:0/3761530101 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc388027b70 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 10 client.7485 fetch_fsmap learned FSMap version 84 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x55935081c780 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc3757fa640 1 -- 172.21.15.78:0/3761530101 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3800550d0 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc3757fa640 1 -- 172.21.15.78:0/3761530101 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc38802d020 con 0x55935092d6d0 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 10 client.7485 fetch_fsmap finished waiting for FSMap version 84 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 10 client.7485 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 20 client.7485 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5593509a8240 0x5593509a1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 4 client.7485 mds_command: new command op to 7305 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 --> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559350598610 con 0x5593509a8240 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.070+0000 7fc395c16640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5593509a8240 0x5593509a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc3757fa640 10 client.7485 ms_handle_connect on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc395c16640 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5593509a8240 0x5593509a1370 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2180471c53b696b server_cookie=f714b433843640f7 in_seq=0 out_seq=0 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc3757fa640 1 -- 172.21.15.78:0/3761530101 <== mds.0 v2:172.21.15.78:6835/2978708219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559350598610 con 0x5593509a8240 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc3757fa640 10 client.7485 handle_command_reply: tid=0 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 client.7485 shutdown 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 msgr2=0x559350934180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350934180 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc38800cb20 tx=0x7fc3880049c0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 shutdown_connections 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5593509a8240 0x5593509a1370 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5593509328b0 0x5593509a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55935092f740 0x5593508eede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc3757fa640 0 client.7485 ms_handle_reset on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 --2- 172.21.15.78:0/3761530101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55935092d6d0 0x559350934180 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 >> 172.21.15.78:0/3761530101 conn(0x5593508f5ae0 msgr2=0x5593508f48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 shutdown_connections 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 1 -- 172.21.15.78:0/3761530101 wait complete. 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.071+0000 7fc39b24c740 20 client.7485 trim_cache size 0 max 16384 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7305, 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 9.1068032809999995, 2024-02-19T14:55:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 84, 2024-02-19T14:55:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 111, 2024-02-19T14:55:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 111, 2024-02-19T14:55:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 14.238141611 2024-02-19T14:55:48.081 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc37c06b840 msgr2=0x7fc37c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc37c06b840 0x7fc37c06dd00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc388007eb0 tx=0x7fc3880073c0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 msgr2=0x7fc3901ec660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 0x7fc3901ec660 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fc38000b3f0 tx=0x7fc38000b8b0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 shutdown_connections 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc37c06b840 0x7fc37c06dd00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc39014d210 0x7fc3901ec660 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3900c3d90 0x7fc3901ec120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 --2- 172.21.15.78:0/837673012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3900719e0 0x7fc3900be9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 >> 172.21.15.78:0/837673012 conn(0x7fc3900c5050 msgr2=0x7fc39006f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 shutdown_connections 2024-02-19T14:55:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.072+0000 7fc39867b640 1 -- 172.21.15.78:0/837673012 wait complete. 2024-02-19T14:55:48.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.188+0000 7f81bd64e640 1 Processor -- start 2024-02-19T14:55:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.188+0000 7f81bd64e640 1 -- start start 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.188+0000 7f81bd64e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 0x7f81b8159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.188+0000 7f81bd64e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b806ffa0 con 0x7f81b80c78a0 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.188+0000 7f81bd64e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b806fc70 con 0x7f81b80c4d80 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.188+0000 7f81bd64e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b806b720 con 0x7f81b8155010 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f81b80c78a0 0x7f81b80c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60412/0 (socket says 172.21.15.78:60412) 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b6ffd640 1 -- 172.21.15.78:0/280409831 learned_addr learned my addr 172.21.15.78:0/280409831 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b77fe640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 0x7f81b8159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898322175 0 0) 0x7f81b806ffa0 con 0x7f81b80c78a0 2024-02-19T14:55:48.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4003520 con 0x7f81b80c78a0 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549179194 0 0) 0x7f81a4003520 con 0x7f81b80c78a0 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 msgr2=0x7f81b8159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 0x7f81b8159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 >> v1:172.21.15.150:6789/0 conn(0x7f81b80c4d80 legacy=0x7f81b81523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b806b930 con 0x7f81b80c78a0 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f81a8001670 con 0x7f81b80c78a0 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 >> v1:172.21.15.78:6789/0 conn(0x7f81b80c78a0 legacy=0x7f81b80c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 0x7f81a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a4004d00 0x7f81a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b806b720 con 0x7f81a4003f20 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a400bb10 con 0x7f81a4004d00 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a400bcd0 con 0x7f81b8155010 2024-02-19T14:55:48.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81affff640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b6ffd640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 0x7f81a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.189+0000 7f81b77fe640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a4004d00 0x7f81a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81affff640 1 -- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 msgr2=0x7f81a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81affff640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8155010 0x7f81a40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81affff640 1 -- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a4004d00 msgr2=0x7f81a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81affff640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a4004d00 0x7f81a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81affff640 1 -- 172.21.15.78:0/280409831 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4003520 con 0x7f81a4003f20 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81b77fe640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a4004d00 0x7f81a4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81affff640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81a40047c0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f81a0001750 tx=0x7f81a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900f84581537a8c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a00052d0 con 0x7f81a4003f20 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a002e050 con 0x7f81a4003f20 2024-02-19T14:55:48.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.190+0000 7f81b67fc640 1 -- 172.21.15.78:0/280409831 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a00035f0 con 0x7f81a4003f20 2024-02-19T14:55:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81bd64e640 1 -- 172.21.15.78:0/280409831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 msgr2=0x7f81a40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81bd64e640 1 --2- 172.21.15.78:0/280409831 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81a40047c0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f81a0001750 tx=0x7f81a40057f0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81b77fe640 1 -- 172.21.15.78:0/280409831 reap_dead start 2024-02-19T14:55:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81bd64e640 1 -- 172.21.15.78:0/280409831 shutdown_connections 2024-02-19T14:55:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81bd64e640 1 -- 172.21.15.78:0/280409831 >> 172.21.15.78:0/280409831 conn(0x7f81b8066e70 msgr2=0x7f81b80a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81bd64e640 1 -- 172.21.15.78:0/280409831 shutdown_connections 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.191+0000 7f81bd64e640 1 -- 172.21.15.78:0/280409831 wait complete. 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.192+0000 7f81bd64e640 1 Processor -- start 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.192+0000 7f81bd64e640 1 -- start start 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81bd64e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81b80c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81bd64e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b814e920 0x7f81b814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81bd64e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b814b5f0 0x7f81b814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81bd64e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b81eaf00 con 0x7f81a4003f20 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81bd64e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b8159130 con 0x7f81b814e920 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81bd64e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b81ea560 con 0x7f81b814b5f0 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81b80c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81b80c3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33422/0 (socket says 172.21.15.78:33422) 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 -- 172.21.15.78:0/1930506018 learned_addr learned my addr 172.21.15.78:0/1930506018 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b77fe640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b814b5f0 0x7f81b814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81affff640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b814e920 0x7f81b814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 -- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b814b5f0 msgr2=0x7f81b814b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b814b5f0 0x7f81b814b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 -- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b814e920 msgr2=0x7f81b814aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b814e920 0x7f81b814aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 -- 172.21.15.78:0/1930506018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b81ea840 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b77fe640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b814b5f0 0x7f81b814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.193+0000 7f81b6ffd640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81b80c3590 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f81a8000fb0 tx=0x7f81a8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.194+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800f9c0 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.194+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800fb20 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.194+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b81eab20 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.194+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800feb0 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.194+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b814be00 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.195+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b814bff0 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.196+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81a801d460 con 0x7f81a4003f20 2024-02-19T14:55:48.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.196+0000 7f81af7fe640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f819806b840 0x7f819806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.196+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81a808a450 con 0x7f81a4003f20 2024-02-19T14:55:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.196+0000 7f81affff640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f819806b840 0x7f819806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.196+0000 7f81affff640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f819806b840 0x7f819806dd00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f81a0001750 tx=0x7f81a0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.200+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81a8054d70 con 0x7f81a4003f20 2024-02-19T14:55:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.454+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b81eb220 con 0x7f81a4003f20 2024-02-19T14:55:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.454+0000 7f81af7fe640 1 -- 172.21.15.78:0/1930506018 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f81a8058430 con 0x7f81a4003f20 2024-02-19T14:55:48.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:48.457 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5927,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:40.368516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2978708219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2978708219},{"type":"v1","addr":"172.21.15.78:6837","nonce":2978708219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 84 2024-02-19T14:55:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f819806b840 msgr2=0x7f819806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f819806b840 0x7f819806dd00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f81a0001750 tx=0x7f81a0009030 comp rx=0 tx=0).stop 2024-02-19T14:55:48.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 msgr2=0x7f81b80c3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81b80c3590 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f81a8000fb0 tx=0x7f81a8007720 comp rx=0 tx=0).stop 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 shutdown_connections 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f819806b840 0x7f819806dd00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b814b5f0 0x7f81b814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b814e920 0x7f81b814aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 --2- 172.21.15.78:0/1930506018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81a4003f20 0x7f81b80c3590 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 >> 172.21.15.78:0/1930506018 conn(0x7f81b8066e70 msgr2=0x7f81b8153010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 shutdown_connections 2024-02-19T14:55:48.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.457+0000 7f81bd64e640 1 -- 172.21.15.78:0/1930506018 wait complete. 2024-02-19T14:55:48.471 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:38.311785+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.78:6837/2978708219", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2978708219, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2978708219, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7305, "incarnation": 82, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-19T14:55:40.368516+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.150:6833/2663291681", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2663291681, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2663291681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5927, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/2437926852", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2437926852, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2437926852, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5930, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/3062592461", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3062592461, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3062592461, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7302, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:48.471 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:55:48.471 INFO:tasks.cephfs.test_failover:current = [{'gid': 7302, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/3062592461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3062592461}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3062592461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 5927, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 7305, 'name': 'c', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/2978708219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2978708219}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2978708219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5930, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2437926852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2437926852}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2437926852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}] 2024-02-19T14:55:48.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-02-19T14:55:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.593+0000 7f520f8b5640 1 Processor -- start 2024-02-19T14:55:48.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.593+0000 7f520f8b5640 1 -- start start 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.594+0000 7f520f8b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080c49c0 0x7f52080c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.594+0000 7f520f8b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f52080ab600 0x7f52080c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.594+0000 7f520f8b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5208152d30 con 0x7f52080ad890 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.594+0000 7f520f8b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52080aba90 con 0x7f52080ab600 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.594+0000 7f520f8b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52080c7860 con 0x7f52080c49c0 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520d651640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f52080ad890 0x7f52081534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60426/0 (socket says 172.21.15.78:60426) 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520d651640 1 -- 172.21.15.78:0/2252822987 learned_addr learned my addr 172.21.15.78:0/2252822987 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:48.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520ce50640 1 --2- 172.21.15.78:0/2252822987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080c49c0 0x7f52080c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520de52640 1 --2- 172.21.15.78:0/2252822987 >> v2:172.21.15.150:3300/0 conn(0x7f52080ab600 0x7f52080c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f51fffff640 1 -- 172.21.15.78:0/2252822987 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039297166 0 0) 0x7f5208152d30 con 0x7f52080ad890 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f51fffff640 1 -- 172.21.15.78:0/2252822987 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f4003520 con 0x7f52080ad890 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f51fffff640 1 -- 172.21.15.78:0/2252822987 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3046928492 0 0) 0x7f51f4003520 con 0x7f52080ad890 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520ce50640 1 -- 172.21.15.78:0/2252822987 >> v2:172.21.15.150:3300/0 conn(0x7f52080ab600 msgr2=0x7f52080c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520ce50640 1 --2- 172.21.15.78:0/2252822987 >> v2:172.21.15.150:3300/0 conn(0x7f52080ab600 0x7f52080c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520ce50640 1 -- 172.21.15.78:0/2252822987 >> v1:172.21.15.78:6789/0 conn(0x7f52080ad890 legacy=0x7f52081534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.595+0000 7f520ce50640 1 -- 172.21.15.78:0/2252822987 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52080c7ce0 con 0x7f52080c49c0 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.596+0000 7f520ce50640 1 --2- 172.21.15.78:0/2252822987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080c49c0 0x7f52080c64f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51f80097a0 tx=0x7f51f802b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6958f234f7a8d6a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.596+0000 7f51fffff640 1 -- 172.21.15.78:0/2252822987 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200001a10 con 0x7f52080c49c0 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.596+0000 7f51fffff640 1 -- 172.21.15.78:0/2252822987 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f802e050 con 0x7f52080c49c0 2024-02-19T14:55:48.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.596+0000 7f51fffff640 1 -- 172.21.15.78:0/2252822987 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8030730 con 0x7f52080c49c0 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 -- 172.21.15.78:0/2252822987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080c49c0 msgr2=0x7f52080c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 --2- 172.21.15.78:0/2252822987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080c49c0 0x7f52080c64f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51f80097a0 tx=0x7f51f802b660 comp rx=0 tx=0).stop 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 -- 172.21.15.78:0/2252822987 shutdown_connections 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 --2- 172.21.15.78:0/2252822987 >> v2:172.21.15.150:3300/0 conn(0x7f52080ab600 0x7f52080c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 --2- 172.21.15.78:0/2252822987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080c49c0 0x7f52080c64f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 -- 172.21.15.78:0/2252822987 >> 172.21.15.78:0/2252822987 conn(0x7f520806b700 msgr2=0x7f52080aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 -- 172.21.15.78:0/2252822987 shutdown_connections 2024-02-19T14:55:48.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.597+0000 7f520f8b5640 1 -- 172.21.15.78:0/2252822987 wait complete. 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.599+0000 7f520f8b5640 1 Processor -- start 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.599+0000 7f520f8b5640 1 -- start start 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520f8b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080ab600 0x7f52080c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520f8b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 0x7f52081ede00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520f8b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52080c49c0 0x7f52081ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520f8b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208071670 con 0x7f52080ad890 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520f8b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52080a8c40 con 0x7f52080c49c0 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520f8b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52080aa860 con 0x7f52080ab600 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 0x7f52081ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 0x7f52081ede00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33434/0 (socket says 172.21.15.78:33434) 2024-02-19T14:55:48.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 -- 172.21.15.78:0/873768629 learned_addr learned my addr 172.21.15.78:0/873768629 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520de52640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52080c49c0 0x7f52081ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520d651640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080ab600 0x7f52080c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 -- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080ab600 msgr2=0x7f52080c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080ab600 0x7f52080c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 -- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52080c49c0 msgr2=0x7f52081ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52080c49c0 0x7f52081ee340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 -- 172.21.15.78:0/873768629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f8009400 con 0x7f52080ad890 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.600+0000 7f520ce50640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 0x7f52081ede00 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098c0 tx=0x7f51f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.601+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f803b810 con 0x7f52080ad890 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.601+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f803b970 con 0x7f52080ad890 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.601+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52080a8240 con 0x7f52080ad890 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.601+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f803bd20 con 0x7f52080ad890 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.601+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5208071290 con 0x7f52080ad890 2024-02-19T14:55:48.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.602+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52080717b0 con 0x7f52080ad890 2024-02-19T14:55:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.603+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f51f8039040 con 0x7f52080ad890 2024-02-19T14:55:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.603+0000 7f51fdffb640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51ec06b810 0x7f51ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.603+0000 7f520d651640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51ec06b810 0x7f51ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.603+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f51f80ab870 con 0x7f52080ad890 2024-02-19T14:55:48.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.604+0000 7f520d651640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51ec06b810 0x7f51ec06dcd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5208152a90 tx=0x7f5200007f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.608+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f52080717b0 con 0x7f52080ad890 2024-02-19T14:55:48.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.786+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f52080717b0 con 0x7f52080ad890 2024-02-19T14:55:48.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.789+0000 7f51fdffb640 1 -- 172.21.15.78:0/873768629 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v36)=0 v36) v1 ==== 116+0+0 (secure 0 0 0) 0x7f52080717b0 con 0x7f52080ad890 2024-02-19T14:55:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51ec06b810 msgr2=0x7f51ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51ec06b810 0x7f51ec06dcd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5208152a90 tx=0x7f5200007f90 comp rx=0 tx=0).stop 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 msgr2=0x7f52081ede00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 0x7f52081ede00 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f51f80098c0 tx=0x7f51f80046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 shutdown_connections 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f51ec06b810 0x7f51ec06dcd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52080ad890 0x7f52081ede00 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52080c49c0 0x7f52081ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 --2- 172.21.15.78:0/873768629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52080ab600 0x7f52080c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 >> 172.21.15.78:0/873768629 conn(0x7f520806b700 msgr2=0x7f52080c5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 shutdown_connections 2024-02-19T14:55:48.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.792+0000 7f520f8b5640 1 -- 172.21.15.78:0/873768629 wait complete. 2024-02-19T14:55:48.808 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.906+0000 7f5491039640 1 Processor -- start 2024-02-19T14:55:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.906+0000 7f5491039640 1 -- start start 2024-02-19T14:55:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f5491039640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f548c0c5130 0x7f548c0c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f5491039640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f548c158f80 0x7f548c0c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f5491039640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c0c3cf0 con 0x7f548c1559c0 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f5491039640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548c158b60 con 0x7f548c0c5130 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f5491039640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548c159400 con 0x7f548c158f80 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f548c0c5130 0x7f548c0c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f548c0c5130 0x7f548c0c30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33450/0 (socket says 172.21.15.78:33450) 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 -- 172.21.15.78:0/472705316 learned_addr learned my addr 172.21.15.78:0/472705316 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548259c640 1 --2- 172.21.15.78:0/472705316 >> v2:172.21.15.150:3300/0 conn(0x7f548c158f80 0x7f548c0c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 -- 172.21.15.78:0/472705316 >> v2:172.21.15.150:3300/0 conn(0x7f548c158f80 msgr2=0x7f548c0c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 --2- 172.21.15.78:0/472705316 >> v2:172.21.15.150:3300/0 conn(0x7f548c158f80 0x7f548c0c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 -- 172.21.15.78:0/472705316 >> v1:172.21.15.150:6789/0 conn(0x7f548c1559c0 legacy=0x7f548c0c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.907+0000 7f548ad9d640 1 -- 172.21.15.78:0/472705316 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c158530 con 0x7f548c0c5130 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.908+0000 7f548ad9d640 1 --2- 172.21.15.78:0/472705316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c0c5130 0x7f548c0c30a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f548c0c5100 tx=0x7f547c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf94bf77bf372b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.908+0000 7f548a59c640 1 -- 172.21.15.78:0/472705316 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c031870 con 0x7f548c0c5130 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.908+0000 7f548a59c640 1 -- 172.21.15.78:0/472705316 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c0319d0 con 0x7f548c0c5130 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.908+0000 7f548a59c640 1 -- 172.21.15.78:0/472705316 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c031d60 con 0x7f548c0c5130 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 -- 172.21.15.78:0/472705316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c0c5130 msgr2=0x7f548c0c30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 --2- 172.21.15.78:0/472705316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c0c5130 0x7f548c0c30a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f548c0c5100 tx=0x7f547c02b910 comp rx=0 tx=0).stop 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 -- 172.21.15.78:0/472705316 shutdown_connections 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 --2- 172.21.15.78:0/472705316 >> v2:172.21.15.150:3300/0 conn(0x7f548c158f80 0x7f548c0c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 --2- 172.21.15.78:0/472705316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c0c5130 0x7f548c0c30a0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 -- 172.21.15.78:0/472705316 >> 172.21.15.78:0/472705316 conn(0x7f548c06b4b0 msgr2=0x7f548c0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 -- 172.21.15.78:0/472705316 shutdown_connections 2024-02-19T14:55:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.909+0000 7f5491039640 1 -- 172.21.15.78:0/472705316 wait complete. 2024-02-19T14:55:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.910+0000 7f5491039640 1 Processor -- start 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.910+0000 7f5491039640 1 -- start start 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f5491039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f548c0c5130 0x7f548c0acc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f5491039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f548c1559c0 0x7f548c148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f5491039640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 0x7f548c1eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f5491039640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1eae90 con 0x7f548c158f80 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f5491039640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1eebb0 con 0x7f548c0c5130 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f5491039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548c1ef0d0 con 0x7f548c1559c0 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 0x7f548c1eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 0x7f548c1eb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33462/0 (socket says 172.21.15.78:33462) 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 -- 172.21.15.78:0/1715919099 learned_addr learned my addr 172.21.15.78:0/1715919099 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548ad9d640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f548c0c5130 0x7f548c0acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548259c640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f548c1559c0 0x7f548c148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 -- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f548c1559c0 msgr2=0x7f548c148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f548c1559c0 0x7f548c148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 -- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f548c0c5130 msgr2=0x7f548c0acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f548c0c5130 0x7f548c0acc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548b59e640 1 -- 172.21.15.78:0/1715919099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0094d0 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.911+0000 7f548ad9d640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f548c0c5130 0x7f548c0acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.912+0000 7f548b59e640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 0x7f548c1eb2e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f548400c780 tx=0x7f548400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.912+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484004b60 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.912+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548400ce90 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.912+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c153b90 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.912+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548400a850 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.912+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c1eedf0 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.913+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c1eb000 con 0x7f548c158f80 2024-02-19T14:55:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.913+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f548400a9b0 con 0x7f548c158f80 2024-02-19T14:55:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.914+0000 7f5483fff640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f546c06b840 0x7f546c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.914+0000 7f548ad9d640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f546c06b840 0x7f546c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.914+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f548408a4c0 con 0x7f548c158f80 2024-02-19T14:55:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.914+0000 7f548ad9d640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f546c06b840 0x7f546c06dd00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f547c036000 tx=0x7f547c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:48.918+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f548c1eb000 con 0x7f548c158f80 2024-02-19T14:55:49.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.131+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f548c1eafd0 con 0x7f548c158f80 2024-02-19T14:55:49.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.132+0000 7f5483fff640 1 -- 172.21.15.78:0/1715919099 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f5484054d60 con 0x7f548c158f80 2024-02-19T14:55:49.135 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:49.135 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5927,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:40.368516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/2978708219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2978708219},{"type":"v1","addr":"172.21.15.78:6837","nonce":2978708219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:49.136 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 84 2024-02-19T14:55:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f546c06b840 msgr2=0x7f546c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f546c06b840 0x7f546c06dd00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f547c036000 tx=0x7f547c03c040 comp rx=0 tx=0).stop 2024-02-19T14:55:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 msgr2=0x7f548c1eb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 0x7f548c1eb2e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f548400c780 tx=0x7f548400cc40 comp rx=0 tx=0).stop 2024-02-19T14:55:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 shutdown_connections 2024-02-19T14:55:49.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f546c06b840 0x7f546c06dd00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f548c158f80 0x7f548c1eb2e0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f548c1559c0 0x7f548c148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 --2- 172.21.15.78:0/1715919099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f548c0c5130 0x7f548c0acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 >> 172.21.15.78:0/1715919099 conn(0x7f548c06b4b0 msgr2=0x7f548c0c7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 shutdown_connections 2024-02-19T14:55:49.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.134+0000 7f5491039640 1 -- 172.21.15.78:0/1715919099 wait complete. 2024-02-19T14:55:49.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:55:38.311785+0000', 'modified': '2024-02-19T14:55:40.368516+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7305': {'gid': 7305, 'name': 'c', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/2978708219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2978708219}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2978708219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:49.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:49.550 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:55:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.648+0000 7f3b5101b640 1 Processor -- start 2024-02-19T14:55:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.648+0000 7f3b5101b640 1 -- start start 2024-02-19T14:55:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b5101b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b5101b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3b4c06d750 0x7f3b4c15d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b5101b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c150820 con 0x7f3b4c150440 2024-02-19T14:55:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b5101b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c0c1b60 con 0x7f3b4c06d750 2024-02-19T14:55:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b5101b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c1537d0 con 0x7f3b4c06ed20 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3b4c06d750 0x7f3b4c15d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3b4c06d750 0x7f3b4c15d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33470/0 (socket says 172.21.15.78:33470) 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3554256728 learned_addr learned my addr 172.21.15.78:0/3554256728 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/3554256728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3554256728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 msgr2=0x7f3b4c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/3554256728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c06d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3554256728 >> v1:172.21.15.150:6789/0 conn(0x7f3b4c150440 legacy=0x7f3b4c0c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.649+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3554256728 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c153d10 con 0x7f3b4c06d750 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.650+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/3554256728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c15d720 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3b44009e50 tx=0x7f3b4402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22694fb7563c567e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.650+0000 7f3b49d9b640 1 -- 172.21.15.78:0/3554256728 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4402cb00 con 0x7f3b4c06d750 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.650+0000 7f3b49d9b640 1 -- 172.21.15.78:0/3554256728 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4402cc60 con 0x7f3b4c06d750 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.650+0000 7f3b49d9b640 1 -- 172.21.15.78:0/3554256728 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b440343f0 con 0x7f3b4c06d750 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 -- 172.21.15.78:0/3554256728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 msgr2=0x7f3b4c15d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 --2- 172.21.15.78:0/3554256728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c15d720 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f3b44009e50 tx=0x7f3b4402bf60 comp rx=0 tx=0).stop 2024-02-19T14:55:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 -- 172.21.15.78:0/3554256728 shutdown_connections 2024-02-19T14:55:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 --2- 172.21.15.78:0/3554256728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c15d720 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 --2- 172.21.15.78:0/3554256728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 -- 172.21.15.78:0/3554256728 >> 172.21.15.78:0/3554256728 conn(0x7f3b4c06b270 msgr2=0x7f3b4c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 -- 172.21.15.78:0/3554256728 shutdown_connections 2024-02-19T14:55:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.651+0000 7f3b5101b640 1 -- 172.21.15.78:0/3554256728 wait complete. 2024-02-19T14:55:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.652+0000 7f3b5101b640 1 Processor -- start 2024-02-19T14:55:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.652+0000 7f3b5101b640 1 -- start start 2024-02-19T14:55:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b5101b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c14baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b5101b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b5101b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b4c150440 0x7f3b4c151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b5101b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c154190 con 0x7f3b4c06d750 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b5101b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c1554c0 con 0x7f3b4c150440 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b5101b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c156f70 con 0x7f3b4c06ed20 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c14baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c14baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33480/0 (socket says 172.21.15.78:33480) 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 -- 172.21.15.78:0/1841442773 learned_addr learned my addr 172.21.15.78:0/1841442773 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b4c150440 0x7f3b4c151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 -- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 msgr2=0x7f3b4c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 -- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b4c150440 msgr2=0x7f3b4c151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b4c150440 0x7f3b4c151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.653+0000 7f3b4ad9d640 1 -- 172.21.15.78:0/1841442773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b44009b30 con 0x7f3b4c06d750 2024-02-19T14:55:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.654+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c14baa0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00b3f0 tx=0x7f3b3c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.654+0000 7f3b437fe640 1 -- 172.21.15.78:0/1841442773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c011830 con 0x7f3b4c06d750 2024-02-19T14:55:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.654+0000 7f3b437fe640 1 -- 172.21.15.78:0/1841442773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c011990 con 0x7f3b4c06d750 2024-02-19T14:55:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.654+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c155150 con 0x7f3b4c06d750 2024-02-19T14:55:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.654+0000 7f3b437fe640 1 -- 172.21.15.78:0/1841442773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c011d40 con 0x7f3b4c06d750 2024-02-19T14:55:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.654+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c145fb0 con 0x7f3b4c06d750 2024-02-19T14:55:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.656+0000 7f3b437fe640 1 -- 172.21.15.78:0/1841442773 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3b3c01a400 con 0x7f3b4c06d750 2024-02-19T14:55:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.656+0000 7f3b437fe640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b2c06b810 0x7f3b2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.657+0000 7f3b437fe640 1 -- 172.21.15.78:0/1841442773 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b3c089fe0 con 0x7f3b4c06d750 2024-02-19T14:55:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.657+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b2c06b810 0x7f3b2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.657+0000 7f3b53bf0740 1 Processor -- start 2024-02-19T14:55:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.657+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b2c06b810 0x7f3b2c06dcd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c1531d0 tx=0x7f3b34008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.657+0000 7f3b53bf0740 1 -- start start 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 0x5592cbcabb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcae8b0 0x5592cbc70860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbc2bcd0 con 0x5592cbc71ae0 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbbdef20 con 0x5592cbcae8b0 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592cbbdc450 con 0x5592cbcab740 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 0x5592cbcabb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 0x5592cbcabb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33492/0 (socket says 172.21.15.78:33492) 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 -- 172.21.15.78:0/2440258987 learned_addr learned my addr 172.21.15.78:0/2440258987 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcae8b0 0x5592cbc70860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 -- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 msgr2=0x5592cbcac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 -- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcae8b0 msgr2=0x5592cbc70860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcae8b0 0x5592cbc70860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 -- 172.21.15.78:0/2440258987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592cbb8cf20 con 0x5592cbc71ae0 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcae8b0 0x5592cbc70860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 0x5592cbcabb20 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3b340041b0 tx=0x7f3b34013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0df6ba6b4e243c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b40ff9640 1 -- 172.21.15.78:0/2440258987 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b340038e0 con 0x5592cbc71ae0 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.658+0000 7f3b40ff9640 1 -- 172.21.15.78:0/2440258987 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b34003a40 con 0x5592cbc71ae0 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b40ff9640 1 -- 172.21.15.78:0/2440258987 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34003db0 con 0x5592cbc71ae0 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2440258987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 msgr2=0x5592cbcabb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 0x5592cbcabb20 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3b340041b0 tx=0x7f3b34013000 comp rx=0 tx=0).stop 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2440258987 shutdown_connections 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbc71ae0 0x5592cbcabb20 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcae8b0 0x5592cbc70860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2440258987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2440258987 >> 172.21.15.78:0/2440258987 conn(0x5592cbc6d120 msgr2=0x5592cbc91b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2440258987 shutdown_connections 2024-02-19T14:55:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.659+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2440258987 wait complete. 2024-02-19T14:55:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.660+0000 7f3b53bf0740 1 -- start start 2024-02-19T14:55:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.660+0000 7f3b53bf0740 1 Processor -- start 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbca3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcae8b0 0x5592cbc6fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 0x5592cbc8e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbbd7c30 con 0x5592cbca8ab0 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbb7cd30 con 0x5592cbcab740 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592cb8ab310 con 0x5592cbcae8b0 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b53bf0740 10 client.0 _ll_register_callbacks cb 0x5592cbb84d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b4b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 0x5592cbc8e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b4b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 0x5592cbc8e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33496/0 (socket says 172.21.15.78:33496) 2024-02-19T14:55:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.661+0000 7f3b4b59e640 1 -- 172.21.15.78:0/740533769 learned_addr learned my addr 172.21.15.78:0/740533769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcae8b0 0x5592cbc6fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbca3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4b59e640 1 -- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcae8b0 msgr2=0x5592cbc6fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcae8b0 0x5592cbc6fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4b59e640 1 -- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 msgr2=0x5592cbca3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbca3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4b59e640 1 -- 172.21.15.78:0/740533769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3400e040 con 0x5592cbca8ab0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 0x5592cbc8e3d0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3b44036040 tx=0x7f3b44004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486765a1d6e8a4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b29ffb640 1 -- 172.21.15.78:0/740533769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44034bf0 con 0x5592cbca8ab0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b29ffb640 1 -- 172.21.15.78:0/740533769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b44045070 con 0x5592cbca8ab0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592cb8d76c0 con 0x5592cbca8ab0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b29ffb640 1 -- 172.21.15.78:0/740533769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b44034d50 con 0x5592cbca8ab0 2024-02-19T14:55:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b53bf0740 10 client.7515 fetch_fsmap learned FSMap version 84 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x5592cbbd62b0 con 0x5592cbca8ab0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.662+0000 7f3b29ffb640 1 -- 172.21.15.78:0/740533769 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5592cb8d76c0 con 0x5592cbca8ab0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b29ffb640 1 -- 172.21.15.78:0/740533769 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3b4403c040 con 0x5592cbca8ab0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b53bf0740 10 client.7515 fetch_fsmap finished waiting for FSMap version 84 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b53bf0740 10 client.7515 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b53bf0740 20 client.7515 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcc03f0 0x5592cbcc07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b53bf0740 4 client.7515 mds_command: new command op to 7305 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 --> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592cb928270 con 0x5592cbcc03f0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcc03f0 0x5592cbcc07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b29ffb640 10 client.7515 ms_handle_connect on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.663+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcc03f0 0x5592cbcc07d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5687554ec3c77531 server_cookie=891424b8ee9fa0a1 in_seq=0 out_seq=0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b29ffb640 1 -- 172.21.15.78:0/740533769 <== mds.0 v2:172.21.15.78:6835/2978708219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5592cb928270 con 0x5592cbcc03f0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b29ffb640 10 client.7515 handle_command_reply: tid=0 2024-02-19T14:55:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 client.7515 shutdown 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 msgr2=0x5592cbc8e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 0x5592cbc8e3d0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3b44036040 tx=0x7f3b44004ab0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 shutdown_connections 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcc03f0 0x5592cbcc07d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbca8ab0 0x5592cbc8e3d0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcae8b0 0x5592cbc6fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/740533769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbca3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 >> 172.21.15.78:0/740533769 conn(0x5592cbc71ae0 msgr2=0x5592cbc91960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b29ffb640 0 client.7515 ms_handle_reset on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 shutdown_connections 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 1 -- 172.21.15.78:0/740533769 wait complete. 2024-02-19T14:55:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.664+0000 7f3b53bf0740 20 client.7515 trim_cache size 0 max 16384 2024-02-19T14:55:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.680+0000 7f3b53bf0740 1 Processor -- start 2024-02-19T14:55:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.680+0000 7f3b53bf0740 1 -- start start 2024-02-19T14:55:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbca8ab0 0x5592cbc8e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbcad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbca2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbb5d490 con 0x5592cbcae8b0 2024-02-19T14:55:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbca7020 con 0x5592cbcab740 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592cbca46a0 con 0x5592cbca8ab0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbca2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbca2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33512/0 (socket says 172.21.15.78:33512) 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3313723607 learned_addr learned my addr 172.21.15.78:0/3313723607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3313723607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbca8ab0 msgr2=0x5592cbc8e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbca8ab0 0x5592cbc8e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3313723607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 msgr2=0x5592cbcad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbcad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.681+0000 7f3b4b59e640 1 -- 172.21.15.78:0/3313723607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592cbb72a20 con 0x5592cbcae8b0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbca2c90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3b440093e0 tx=0x7f3b44041070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f22f9bd0615ff1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b2bfff640 1 -- 172.21.15.78:0/3313723607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44003990 con 0x5592cbcae8b0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b2bfff640 1 -- 172.21.15.78:0/3313723607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b44043070 con 0x5592cbcae8b0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b2bfff640 1 -- 172.21.15.78:0/3313723607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44003cf0 con 0x5592cbcae8b0 2024-02-19T14:55:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 -- 172.21.15.78:0/3313723607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 msgr2=0x5592cbca2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbca2c90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3b440093e0 tx=0x7f3b44041070 comp rx=0 tx=0).stop 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 -- 172.21.15.78:0/3313723607 shutdown_connections 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbca2c90 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbcab740 0x5592cbcad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/3313723607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbca8ab0 0x5592cbc8e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 -- 172.21.15.78:0/3313723607 >> 172.21.15.78:0/3313723607 conn(0x5592cbc71ae0 msgr2=0x5592cbc66760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 -- 172.21.15.78:0/3313723607 shutdown_connections 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.682+0000 7f3b53bf0740 1 -- 172.21.15.78:0/3313723607 wait complete. 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.683+0000 7f3b53bf0740 1 -- start start 2024-02-19T14:55:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.683+0000 7f3b53bf0740 1 Processor -- start 2024-02-19T14:55:49.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.683+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbca8ab0 0x5592cbc91e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcaff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbd09800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbc92340 con 0x5592cbcae8b0 2024-02-19T14:55:49.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592cbc92530 con 0x5592cbca8ab0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592cbd24970 con 0x5592cbcab740 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 10 client.0 _ll_register_callbacks cb 0x7f3b44009510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbd09800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbd09800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33522/0 (socket says 172.21.15.78:33522) 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 -- 172.21.15.78:0/2153679865 learned_addr learned my addr 172.21.15.78:0/2153679865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbca8ab0 0x5592cbc91e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4a59c640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcaff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 -- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 msgr2=0x5592cbcaff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcaff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 -- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbca8ab0 msgr2=0x5592cbc91e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbca8ab0 0x5592cbc91e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 -- 172.21.15.78:0/2153679865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592cbbd62b0 con 0x5592cbcae8b0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b4b59e640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbd09800 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3b440070b0 tx=0x7f3b44009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f923b7694b387d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:49.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2153679865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44003ea0 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2153679865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b44045070 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b34003a40 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.684+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2153679865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b44004030 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 10 client.7521 fetch_fsmap learned FSMap version 84 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x5592cbb98780 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2153679865 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b34003a40 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2153679865 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 84) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3b4403c040 con 0x5592cbcae8b0 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 10 client.7521 fetch_fsmap finished waiting for FSMap version 84 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 10 client.7521 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 20 client.7521 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcb0530 0x5592cbd09d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 4 client.7521 mds_command: new command op to 7305 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:49.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 --> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5592cb914610 con 0x5592cbcb0530 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcb0530 0x5592cbd09d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b29ffb640 10 client.7521 ms_handle_connect on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b4ad9d640 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcb0530 0x5592cbd09d40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de794189db9a0495 server_cookie=d5151937e4045d3f in_seq=0 out_seq=0 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b29ffb640 1 -- 172.21.15.78:0/2153679865 <== mds.0 v2:172.21.15.78:6835/2978708219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5592cb914610 con 0x5592cbcb0530 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b29ffb640 10 client.7521 handle_command_reply: tid=0 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 1 client.7521 shutdown 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.685+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 msgr2=0x5592cbd09800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbd09800 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3b440070b0 tx=0x7f3b44009810 comp rx=0 tx=0).stop 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 shutdown_connections 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:6835/2978708219,v1:172.21.15.78:6837/2978708219] conn(0x5592cbcb0530 0x5592cbd09d40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592cbcae8b0 0x5592cbd09800 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b29ffb640 0 client.7521 ms_handle_reset on v2:172.21.15.78:6835/2978708219 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592cbcab740 0x5592cbcaff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 --2- 172.21.15.78:0/2153679865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592cbca8ab0 0x5592cbc91e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 >> 172.21.15.78:0/2153679865 conn(0x5592cbc71ae0 msgr2=0x5592cbcac420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 shutdown_connections 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 1 -- 172.21.15.78:0/2153679865 wait complete. 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b53bf0740 20 client.7521 trim_cache size 0 max 16384 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7305, 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:49.695 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 10.721286618000001, 2024-02-19T14:55:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 84, 2024-02-19T14:55:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 111, 2024-02-19T14:55:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 111, 2024-02-19T14:55:49.696 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 15.852631593 2024-02-19T14:55:49.696 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b2c06b810 msgr2=0x7f3b2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b2c06b810 0x7f3b2c06dcd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c1531d0 tx=0x7f3b34008040 comp rx=0 tx=0).stop 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 msgr2=0x7f3b4c14baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c14baa0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00b3f0 tx=0x7f3b3c00b8b0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 shutdown_connections 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b2c06b810 0x7f3b2c06dcd0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b4c150440 0x7f3b4c151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b4c06ed20 0x7f3b4c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 --2- 172.21.15.78:0/1841442773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c06d750 0x7f3b4c14baa0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 >> 172.21.15.78:0/1841442773 conn(0x7f3b4c06b270 msgr2=0x7f3b4c0c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 shutdown_connections 2024-02-19T14:55:49.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:49.686+0000 7f3b5101b640 1 -- 172.21.15.78:0/1841442773 wait complete. 2024-02-19T14:55:49.700 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:38.311785+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.78:6837/2978708219", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2978708219, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2978708219, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7305, "incarnation": 82, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-19T14:55:40.368516+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.150:6833/2663291681", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2663291681, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2663291681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5927, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/2437926852", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2437926852, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2437926852, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5930, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/3062592461", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3062592461, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3062592461, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7302, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:49.701 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:55:49.701 INFO:tasks.cephfs.test_failover:current = [{'gid': 7302, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/3062592461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3062592461}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3062592461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 5927, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 7305, 'name': 'c', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/2978708219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2978708219}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2978708219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5930, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2437926852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2437926852}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2437926852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}] 2024-02-19T14:55:49.701 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-02-19T14:55:49.701 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:55:50.190 INFO:tasks.ceph.mds.c.smithi078.stderr: -264> 2024-02-19T14:55:44.219+0000 7f7c5eaf1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:50.216 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:55:54.223 INFO:tasks.ceph.mds.b.smithi150.stderr:2024-02-19T14:55:54.221+0000 7fd0a4208640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:55:54.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:54.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 Processor -- start 2024-02-19T14:55:54.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 -- start start 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4f0159320 0x7fa4f013add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 0x7fa4f0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f0155fd0 con 0x7fa4f00c17e0 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f00c2030 con 0x7fa4f0159320 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.809+0000 7fa4f6491640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0155190 con 0x7fa4f01555e0 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4f4a2e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 0x7fa4f0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4f0159320 0x7fa4f013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4ef7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa4f00c17e0 0x7fa4f00718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53776/0 (socket says 172.21.15.78:53776) 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4ef7fe640 1 -- 172.21.15.78:0/2451141197 learned_addr learned my addr 172.21.15.78:0/2451141197 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4f4a2e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 0x7fa4f0155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33128/0 (socket says 172.21.15.78:33128) 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4f0159320 0x7fa4f013add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57650/0 (socket says 172.21.15.78:57650) 2024-02-19T14:55:54.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4effff640 1 -- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 msgr2=0x7fa4f0155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:54.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4effff640 1 --2- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 0x7fa4f0155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:54.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4effff640 1 -- 172.21.15.78:0/2451141197 >> v1:172.21.15.150:6789/0 conn(0x7fa4f00c17e0 legacy=0x7fa4f00718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:54.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4eeffd640 1 -- 172.21.15.78:0/2451141197 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675926138 0 0) 0x7fa4f0155fd0 con 0x7fa4f00c17e0 2024-02-19T14:55:54.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4effff640 1 -- 172.21.15.78:0/2451141197 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f00c4190 con 0x7fa4f0159320 2024-02-19T14:55:54.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.810+0000 7fa4f4a2e640 1 --2- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 0x7fa4f0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:54.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.811+0000 7fa4effff640 1 --2- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f0159320 0x7fa4f013add0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00097a0 tx=0x7fa4e002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42763d0b376b90f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:54.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.811+0000 7fa4eeffd640 1 -- 172.21.15.78:0/2451141197 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e002e070 con 0x7fa4f0159320 2024-02-19T14:55:54.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.811+0000 7fa4eeffd640 1 -- 172.21.15.78:0/2451141197 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e00315a0 con 0x7fa4f0159320 2024-02-19T14:55:54.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.811+0000 7fa4eeffd640 1 -- 172.21.15.78:0/2451141197 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0031930 con 0x7fa4f0159320 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 -- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f0159320 msgr2=0x7fa4f013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 --2- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f0159320 0x7fa4f013add0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00097a0 tx=0x7fa4e002b840 comp rx=0 tx=0).stop 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 -- 172.21.15.78:0/2451141197 shutdown_connections 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 --2- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f01555e0 0x7fa4f0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 --2- 172.21.15.78:0/2451141197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f0159320 0x7fa4f013add0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 -- 172.21.15.78:0/2451141197 >> 172.21.15.78:0/2451141197 conn(0x7fa4f00c7800 msgr2=0x7fa4f00c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 -- 172.21.15.78:0/2451141197 shutdown_connections 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.812+0000 7fa4f6491640 1 -- 172.21.15.78:0/2451141197 wait complete. 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.813+0000 7fa4f6491640 1 Processor -- start 2024-02-19T14:55:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.813+0000 7fa4f6491640 1 -- start start 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f00c17e0 0x7fa4f01ee170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f01555e0 0x7fa4f013d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 0x7fa4f0136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0137b60 con 0x7fa4f0159320 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f01e1dd0 con 0x7fa4f01555e0 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f01eeab0 con 0x7fa4f00c17e0 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 0x7fa4f0136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 0x7fa4f0136ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33552/0 (socket says 172.21.15.78:33552) 2024-02-19T14:55:54.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 -- 172.21.15.78:0/3517338655 learned_addr learned my addr 172.21.15.78:0/3517338655 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4effff640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f00c17e0 0x7fa4f01ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4ef7fe640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f01555e0 0x7fa4f013d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 -- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f00c17e0 msgr2=0x7fa4f01ee170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f00c17e0 0x7fa4f01ee170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 -- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f01555e0 msgr2=0x7fa4f013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f01555e0 0x7fa4f013d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 -- 172.21.15.78:0/3517338655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8009a70 con 0x7fa4f0159320 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f4a2e640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 0x7fa4f0136ad0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800efc0 tx=0x7fa4e8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8009d70 con 0x7fa4f0159320 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e800f040 con 0x7fa4f0159320 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e0009400 con 0x7fa4f0159320 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.814+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8003990 con 0x7fa4f0159320 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.815+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f01377a0 con 0x7fa4f0159320 2024-02-19T14:55:54.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.816+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f0074990 con 0x7fa4f0159320 2024-02-19T14:55:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.816+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa4e80144a0 con 0x7fa4f0159320 2024-02-19T14:55:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.816+0000 7fa4ecff9640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4dc06b840 0x7fa4dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.816+0000 7fa4effff640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4dc06b840 0x7fa4dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.816+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa4e808ae40 con 0x7fa4f0159320 2024-02-19T14:55:54.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.817+0000 7fa4effff640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4dc06b840 0x7fa4dc06dd00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0036000 tx=0x7fa4e003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:54.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:54.821+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa4e80557c0 con 0x7fa4f0159320 2024-02-19T14:55:55.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.030+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4f0067660 con 0x7fa4f0159320 2024-02-19T14:55:55.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.030+0000 7fa4ecff9640 1 -- 172.21.15.78:0/3517338655 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4947 (secure 0 0 0) 0x7fa4e8058e80 con 0x7fa4f0159320 2024-02-19T14:55:55.034 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:55.034 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7527,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2007073435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2007073435},{"type":"v1","addr":"172.21.15.78:6837","nonce":2007073435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:52.195539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:55.034 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 89 2024-02-19T14:55:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4dc06b840 msgr2=0x7fa4dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4dc06b840 0x7fa4dc06dd00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0036000 tx=0x7fa4e003d040 comp rx=0 tx=0).stop 2024-02-19T14:55:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 msgr2=0x7fa4f0136ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 0x7fa4f0136ad0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800efc0 tx=0x7fa4e8009f90 comp rx=0 tx=0).stop 2024-02-19T14:55:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 shutdown_connections 2024-02-19T14:55:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4dc06b840 0x7fa4dc06dd00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f0159320 0x7fa4f0136ad0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f01555e0 0x7fa4f013d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 --2- 172.21.15.78:0/3517338655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f00c17e0 0x7fa4f01ee170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 >> 172.21.15.78:0/3517338655 conn(0x7fa4f00c7800 msgr2=0x7fa4f00c02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 shutdown_connections 2024-02-19T14:55:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.032+0000 7fa4f6491640 1 -- 172.21.15.78:0/3517338655 wait complete. 2024-02-19T14:55:55.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:55:38.311785+0000', 'modified': '2024-02-19T14:55:52.195539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5927}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5927': {'gid': 5927, 'name': 'b', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:55.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:55.449 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-19T14:55:55.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.547+0000 7f0061bbc640 1 Processor -- start 2024-02-19T14:55:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.547+0000 7f0061bbc640 1 -- start start 2024-02-19T14:55:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.547+0000 7f0061bbc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f005c14f080 0x7f005c0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f0061bbc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f005c0c6db0 0x7f005c0c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f0061bbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c155040 con 0x7f005c0c4d20 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f0061bbc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c0c57d0 con 0x7f005c0c6db0 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f0061bbc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c155300 con 0x7f005c14f080 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f005c0c6db0 0x7f005c0c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f005c0c6db0 0x7f005c0c47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33554/0 (socket says 172.21.15.78:33554) 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 -- 172.21.15.78:0/1776623595 learned_addr learned my addr 172.21.15.78:0/1776623595 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005b7fe640 1 --2- 172.21.15.78:0/1776623595 >> v2:172.21.15.150:3300/0 conn(0x7f005c14f080 0x7f005c0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 -- 172.21.15.78:0/1776623595 >> v2:172.21.15.150:3300/0 conn(0x7f005c14f080 msgr2=0x7f005c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 --2- 172.21.15.78:0/1776623595 >> v2:172.21.15.150:3300/0 conn(0x7f005c14f080 0x7f005c0c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 -- 172.21.15.78:0/1776623595 >> v1:172.21.15.150:6789/0 conn(0x7f005c0c4d20 legacy=0x7f005c158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.548+0000 7f005affd640 1 -- 172.21.15.78:0/1776623595 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c1555e0 con 0x7f005c0c6db0 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f005affd640 1 --2- 172.21.15.78:0/1776623595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c0c47e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f004c009870 tx=0x7f004c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89140c1de20132d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f005a7fc640 1 -- 172.21.15.78:0/1776623595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c0317e0 con 0x7f005c0c6db0 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f005a7fc640 1 -- 172.21.15.78:0/1776623595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c031940 con 0x7f005c0c6db0 2024-02-19T14:55:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f005a7fc640 1 -- 172.21.15.78:0/1776623595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c031cd0 con 0x7f005c0c6db0 2024-02-19T14:55:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 -- 172.21.15.78:0/1776623595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 msgr2=0x7f005c0c47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 --2- 172.21.15.78:0/1776623595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c0c47e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f004c009870 tx=0x7f004c02b880 comp rx=0 tx=0).stop 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 -- 172.21.15.78:0/1776623595 shutdown_connections 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 --2- 172.21.15.78:0/1776623595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c0c47e0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 --2- 172.21.15.78:0/1776623595 >> v2:172.21.15.150:3300/0 conn(0x7f005c14f080 0x7f005c0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 -- 172.21.15.78:0/1776623595 >> 172.21.15.78:0/1776623595 conn(0x7f005c06d9e0 msgr2=0x7f005c0a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 -- 172.21.15.78:0/1776623595 shutdown_connections 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.549+0000 7f0061bbc640 1 -- 172.21.15.78:0/1776623595 wait complete. 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.550+0000 7f0061bbc640 1 Processor -- start 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 -- start start 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f005c0c4d20 0x7f005c14dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f005c14f080 0x7f005c159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c142a00 con 0x7f005c0c6db0 2024-02-19T14:55:55.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0aa840 con 0x7f005c14f080 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f0061bbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0ab3c0 con 0x7f005c0c4d20 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f005affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f005affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33562/0 (socket says 172.21.15.78:33562) 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.551+0000 7f005affd640 1 -- 172.21.15.78:0/3404176895 learned_addr learned my addr 172.21.15.78:0/3404176895 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f005c0c4d20 0x7f005c14dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005bfff640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f005c14f080 0x7f005c159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005affd640 1 -- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f005c0c4d20 msgr2=0x7f005c14dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005affd640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f005c0c4d20 0x7f005c14dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005affd640 1 -- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f005c14f080 msgr2=0x7f005c159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005affd640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f005c14f080 0x7f005c159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005affd640 1 -- 172.21.15.78:0/3404176895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c0094d0 con 0x7f005c0c6db0 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005bfff640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f005c14f080 0x7f005c159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:55.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f005c0c4d20 0x7f005c14dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f005affd640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c158cd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f004c000c00 tx=0x7f004c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f0043fff640 1 -- 172.21.15.78:0/3404176895 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c032880 con 0x7f005c0c6db0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f0043fff640 1 -- 172.21.15.78:0/3404176895 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c032a20 con 0x7f005c0c6db0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f0043fff640 1 -- 172.21.15.78:0/3404176895 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c032dd0 con 0x7f005c0c6db0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c0aaa90 con 0x7f005c0c6db0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.552+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c0ab050 con 0x7f005c0c6db0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.554+0000 7f0043fff640 1 -- 172.21.15.78:0/3404176895 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f004c04e450 con 0x7f005c0c6db0 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.554+0000 7f0043fff640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f004806b840 0x7f004806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.554+0000 7f0043fff640 1 -- 172.21.15.78:0/3404176895 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f004c0ac580 con 0x7f005c0c6db0 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.554+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f004806b840 0x7f004806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.554+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f004806b840 0x7f004806dd00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0050009b90 tx=0x7f0050008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.555+0000 7f0064799740 1 Processor -- start 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.555+0000 7f0064799740 1 -- start start 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f0064799740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597ab088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aacdae0 0x55597ab07b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597aacc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f0064799740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597aa87cd0 con 0x55597ab07740 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597aa3af20 con 0x55597ab0a8b0 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55597aa38450 con 0x55597aacdae0 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597ab088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597ab088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33576/0 (socket says 172.21.15.78:33576) 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 -- 172.21.15.78:0/3028573204 learned_addr learned my addr 172.21.15.78:0/3028573204 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005affd640 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aacdae0 0x55597ab07b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005bfff640 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597aacc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 -- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aacdae0 msgr2=0x55597ab07b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aacdae0 0x55597ab07b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 -- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 msgr2=0x55597aacc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597aacc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 -- 172.21.15.78:0/3028573204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55597a9e8f20 con 0x55597ab07740 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597ab088b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0050011e60 tx=0x7f0050014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9abef8da74cbb49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f00417fa640 1 -- 172.21.15.78:0/3028573204 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050015e50 con 0x55597ab07740 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f00417fa640 1 -- 172.21.15.78:0/3028573204 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050021440 con 0x55597ab07740 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.556+0000 7f00417fa640 1 -- 172.21.15.78:0/3028573204 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00500217b0 con 0x55597ab07740 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 -- 172.21.15.78:0/3028573204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 msgr2=0x55597ab088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597ab088b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0050011e60 tx=0x7f0050014000 comp rx=0 tx=0).stop 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 -- 172.21.15.78:0/3028573204 shutdown_connections 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597aacc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aacdae0 0x55597ab07b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 --2- 172.21.15.78:0/3028573204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597ab088b0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 -- 172.21.15.78:0/3028573204 >> 172.21.15.78:0/3028573204 conn(0x55597aac9120 msgr2=0x55597aaedb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 -- 172.21.15.78:0/3028573204 shutdown_connections 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.557+0000 7f0064799740 1 -- 172.21.15.78:0/3028573204 wait complete. 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.558+0000 7f0064799740 1 -- start start 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.558+0000 7f0064799740 1 Processor -- start 2024-02-19T14:55:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.558+0000 7f0064799740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597aaed390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab1c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aaeaf80 0x55597aadf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597aa33c30 con 0x55597ab07740 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597a9d8d30 con 0x55597ab0a8b0 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55597a707310 con 0x55597aaeaf80 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 10 client.0 _ll_register_callbacks cb 0x55597a9e0d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597aaed390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597aaed390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33582/0 (socket says 172.21.15.78:33582) 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 -- 172.21.15.78:0/3387103139 learned_addr learned my addr 172.21.15.78:0/3387103139 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005bfff640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aaeaf80 0x55597aadf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005affd640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab1c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 -- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aaeaf80 msgr2=0x55597aadf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aaeaf80 0x55597aadf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 -- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 msgr2=0x55597ab1c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab1c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 -- 172.21.15.78:0/3387103139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0050011b40 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f005b7fe640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597aaed390 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0050008010 tx=0x7f00500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd122c71940e5da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f003a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f003a7fc640 1 -- 172.21.15.78:0/3387103139 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050015e50 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f003a7fc640 1 -- 172.21.15.78:0/3387103139 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0050021d50 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55597a7336c0 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f003a7fc640 1 -- 172.21.15.78:0/3387103139 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0050035410 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 10 client.7542 fetch_fsmap learned FSMap version 89 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.559+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55597aa322b0 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f003a7fc640 1 -- 172.21.15.78:0/3387103139 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55597a7336c0 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f003a7fc640 1 -- 172.21.15.78:0/3387103139 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 89) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0050035570 con 0x55597ab07740 2024-02-19T14:55:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f0064799740 10 client.7542 fetch_fsmap finished waiting for FSMap version 89 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f0064799740 10 client.7542 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f0064799740 20 client.7542 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f0064799740 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aaffa70 0x55597aaffe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f0064799740 4 client.7542 mds_command: new command op to 5927 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 --> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55597a784270 con 0x55597aaffa70 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f005affd640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aaffa70 0x55597aaffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f003a7fc640 10 client.7542 ms_handle_connect on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.560+0000 7f005affd640 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aaffa70 0x55597aaffe50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f292d61b434f8a server_cookie=a2442f7d4c0564c4 in_seq=0 out_seq=0 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f003a7fc640 1 -- 172.21.15.78:0/3387103139 <== mds.0 v2:172.21.15.150:6832/2663291681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55597a784270 con 0x55597aaffa70 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f003a7fc640 10 client.7542 handle_command_reply: tid=0 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 client.7542 shutdown 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 msgr2=0x55597aaed390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597aaed390 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0050008010 tx=0x7f00500046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 shutdown_connections 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aaffa70 0x55597aaffe50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597aaeaf80 0x55597aadf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab1c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f003a7fc640 0 client.7542 ms_handle_reset on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 --2- 172.21.15.78:0/3387103139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab07740 0x55597aaed390 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 >> 172.21.15.78:0/3387103139 conn(0x55597aacdae0 msgr2=0x55597ab08e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 shutdown_connections 2024-02-19T14:55:55.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 1 -- 172.21.15.78:0/3387103139 wait complete. 2024-02-19T14:55:55.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.561+0000 7f0064799740 20 client.7542 trim_cache size 0 max 16384 2024-02-19T14:55:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.578+0000 7f0064799740 1 Processor -- start 2024-02-19T14:55:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.578+0000 7f0064799740 1 -- start start 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.578+0000 7f0064799740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 0x55597ab05820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab02f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab034a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0064799740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597a9b9490 con 0x55597aaeaf80 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597aadf780 con 0x55597ab0a8b0 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55597aac8d50 con 0x55597ab07740 2024-02-19T14:55:55.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 0x55597ab05820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 0x55597ab05820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33586/0 (socket says 172.21.15.78:33586) 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 -- 172.21.15.78:0/2003556513 learned_addr learned my addr 172.21.15.78:0/2003556513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005bfff640 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab034a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005affd640 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab02f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 -- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 msgr2=0x55597ab02f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab02f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 -- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 msgr2=0x55597ab034a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab034a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 -- 172.21.15.78:0/2003556513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55597a9cea20 con 0x55597aaeaf80 2024-02-19T14:55:55.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005affd640 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab02f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f005b7fe640 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 0x55597ab05820 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f00500095b0 tx=0x7f005002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e705eaf9cef757ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0040ff9640 1 -- 172.21.15.78:0/2003556513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050004640 con 0x55597aaeaf80 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0040ff9640 1 -- 172.21.15.78:0/2003556513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050006a80 con 0x55597aaeaf80 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.579+0000 7f0040ff9640 1 -- 172.21.15.78:0/2003556513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050011d60 con 0x55597aaeaf80 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 -- 172.21.15.78:0/2003556513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 msgr2=0x55597ab05820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 0x55597ab05820 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f00500095b0 tx=0x7f005002c000 comp rx=0 tx=0).stop 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 -- 172.21.15.78:0/2003556513 shutdown_connections 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597ab0a8b0 0x55597ab034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab02f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 --2- 172.21.15.78:0/2003556513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597aaeaf80 0x55597ab05820 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 -- 172.21.15.78:0/2003556513 >> 172.21.15.78:0/2003556513 conn(0x55597aacdae0 msgr2=0x55597aac2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 -- 172.21.15.78:0/2003556513 shutdown_connections 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.580+0000 7f0064799740 1 -- 172.21.15.78:0/2003556513 wait complete. 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 -- start start 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 Processor -- start 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597aaeaf80 0x55597ab007f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab05a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 0x55597aade210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597ab15450 con 0x55597ab0a8b0 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55597ab15650 con 0x55597aaeaf80 2024-02-19T14:55:55.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55597ab7be00 con 0x55597ab07740 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f0064799740 10 client.0 _ll_register_callbacks cb 0x7f00540016a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f005bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 0x55597aade210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.581+0000 7f005bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 0x55597aade210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33596/0 (socket says 172.21.15.78:33596) 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 -- 172.21.15.78:0/2212147900 learned_addr learned my addr 172.21.15.78:0/2212147900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005affd640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab05a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005b7fe640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597aaeaf80 0x55597ab007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 -- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 msgr2=0x55597ab05a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab05a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 -- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597aaeaf80 msgr2=0x55597ab007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597aaeaf80 0x55597ab007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 -- 172.21.15.78:0/2212147900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55597aa322b0 con 0x55597ab0a8b0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f005bfff640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 0x55597aade210 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f00540012e0 tx=0x7f0054017e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4508a58f616e296 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f003a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:55.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f003a7fc640 1 -- 172.21.15.78:0/2212147900 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054003c40 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f003a7fc640 1 -- 172.21.15.78:0/2212147900 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0054003da0 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55597ab01340 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f003a7fc640 1 -- 172.21.15.78:0/2212147900 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0054006b50 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 10 client.7548 fetch_fsmap learned FSMap version 89 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55597a9f4780 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f003a7fc640 1 -- 172.21.15.78:0/2212147900 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55597ab01340 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f003a7fc640 1 -- 172.21.15.78:0/2212147900 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 89) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0054006cb0 con 0x55597ab0a8b0 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 10 client.7548 fetch_fsmap finished waiting for FSMap version 89 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 10 client.7548 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 20 client.7548 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aade930 0x55597ab74ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 4 client.7548 mds_command: new command op to 5927 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.582+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 --> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0050021440 con 0x55597aade930 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f005b7fe640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aade930 0x55597ab74ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f003a7fc640 10 client.7548 ms_handle_connect on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f005b7fe640 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aade930 0x55597ab74ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a127c829f107a45 server_cookie=6b89ec9229eecec1 in_seq=0 out_seq=0 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f003a7fc640 1 -- 172.21.15.78:0/2212147900 <== mds.0 v2:172.21.15.150:6832/2663291681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7f0050021440 con 0x55597aade930 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f003a7fc640 10 client.7548 handle_command_reply: tid=0 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 client.7548 shutdown 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 msgr2=0x55597aade210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 0x55597aade210 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f00540012e0 tx=0x7f0054017e60 comp rx=0 tx=0).stop 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 shutdown_connections 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55597aade930 0x55597ab74ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55597ab0a8b0 0x55597aade210 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55597ab07740 0x55597ab05a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f003a7fc640 0 client.7548 ms_handle_reset on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 --2- 172.21.15.78:0/2212147900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55597aaeaf80 0x55597ab007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 >> 172.21.15.78:0/2212147900 conn(0x55597aacdae0 msgr2=0x55597aafe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 shutdown_connections 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 1 -- 172.21.15.78:0/2212147900 wait complete. 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.583+0000 7f0064799740 20 client.7548 trim_cache size 0 max 16384 2024-02-19T14:55:55.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5927, 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 5.3987825300000001, 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 89, 2024-02-19T14:55:55.597 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 112, 2024-02-19T14:55:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 112, 2024-02-19T14:55:55.598 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 21.735946253000002 2024-02-19T14:55:55.598 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f004806b840 msgr2=0x7f004806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f004806b840 0x7f004806dd00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0050009b90 tx=0x7f0050008040 comp rx=0 tx=0).stop 2024-02-19T14:55:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 msgr2=0x7f005c158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c158cd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f004c000c00 tx=0x7f004c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 shutdown_connections 2024-02-19T14:55:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f004806b840 0x7f004806dd00 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f005c0c6db0 0x7f005c158cd0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f005c14f080 0x7f005c159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 --2- 172.21.15.78:0/3404176895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f005c0c4d20 0x7f005c14dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 >> 172.21.15.78:0/3404176895 conn(0x7f005c06d9e0 msgr2=0x7f005c0714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 shutdown_connections 2024-02-19T14:55:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.584+0000 7f0061bbc640 1 -- 172.21.15.78:0/3404176895 wait complete. 2024-02-19T14:55:55.601 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:38.311785+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5927": { "addr": "172.21.15.150:6833/2663291681", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2663291681, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2663291681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5927, "incarnation": 86, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 112, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-19T14:55:52.195539+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5927 } } } ], "standbys": [ { "addr": "172.21.15.150:6835/2437926852", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2437926852, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2437926852, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5930, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/3062592461", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3062592461, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3062592461, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7302, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/2007073435", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2007073435, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2007073435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7527, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:55.602 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:55:55.602 INFO:tasks.cephfs.test_failover:current = [{'gid': 7302, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/3062592461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3062592461}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3062592461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 5927, 'name': 'b', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7527, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/2007073435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2007073435}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2007073435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 87}, {'gid': 5930, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2437926852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2437926852}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2437926852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}] 2024-02-19T14:55:55.602 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:55:55.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-02-19T14:55:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.701+0000 7fd3cb24e640 1 Processor -- start 2024-02-19T14:55:55.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.701+0000 7fd3cb24e640 1 -- start start 2024-02-19T14:55:55.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3cb24e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152a50 0x7fd3c4071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3cb24e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c4071bf0 con 0x7fd3c40702d0 2024-02-19T14:55:55.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3cb24e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c4158a20 con 0x7fd3c4152670 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3cb24e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40c6780 con 0x7fd3c4152a50 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3c97eb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd3c40702d0 0x7fd3c4158eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60446/0 (socket says 172.21.15.78:60446) 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3c97eb640 1 -- 172.21.15.78:0/1044860878 learned_addr learned my addr 172.21.15.78:0/1044860878 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.702+0000 7fd3c3fff640 1 --2- 172.21.15.78:0/1044860878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152a50 0x7fd3c4071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c37fe640 1 -- 172.21.15.78:0/1044860878 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478650065 0 0) 0x7fd3c4071bf0 con 0x7fd3c40702d0 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c37fe640 1 -- 172.21.15.78:0/1044860878 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac003520 con 0x7fd3c40702d0 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c37fe640 1 -- 172.21.15.78:0/1044860878 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481074478 0 0) 0x7fd3c4158a20 con 0x7fd3c4152670 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c3fff640 1 -- 172.21.15.78:0/1044860878 >> v1:172.21.15.150:6789/0 conn(0x7fd3c4152670 legacy=0x7fd3c4155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c3fff640 1 -- 172.21.15.78:0/1044860878 >> v1:172.21.15.78:6789/0 conn(0x7fd3c40702d0 legacy=0x7fd3c4158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c3fff640 1 -- 172.21.15.78:0/1044860878 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c4071630 con 0x7fd3c4152a50 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.703+0000 7fd3c3fff640 1 --2- 172.21.15.78:0/1044860878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152a50 0x7fd3c4071050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd3b80097d0 tx=0x7fd3b802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8e14706d87a39eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3c37fe640 1 -- 172.21.15.78:0/1044860878 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8030880 con 0x7fd3c4152a50 2024-02-19T14:55:55.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3c37fe640 1 -- 172.21.15.78:0/1044860878 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b80309e0 con 0x7fd3c4152a50 2024-02-19T14:55:55.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3c37fe640 1 -- 172.21.15.78:0/1044860878 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8030d50 con 0x7fd3c4152a50 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 -- 172.21.15.78:0/1044860878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152a50 msgr2=0x7fd3c4071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/1044860878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152a50 0x7fd3c4071050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd3b80097d0 tx=0x7fd3b802b9c0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 -- 172.21.15.78:0/1044860878 shutdown_connections 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/1044860878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152a50 0x7fd3c4071050 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 -- 172.21.15.78:0/1044860878 >> 172.21.15.78:0/1044860878 conn(0x7fd3c40c85e0 msgr2=0x7fd3c40c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 -- 172.21.15.78:0/1044860878 shutdown_connections 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.704+0000 7fd3cb24e640 1 -- 172.21.15.78:0/1044860878 wait complete. 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.705+0000 7fd3cb24e640 1 Processor -- start 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.705+0000 7fd3cb24e640 1 -- start start 2024-02-19T14:55:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3cb24e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3c40702d0 0x7fd3c40ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3cb24e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152670 0x7fd3c41ead00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3cb24e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 0x7fd3c40c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3cb24e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c40acd80 con 0x7fd3c4154800 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3cb24e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c41eb240 con 0x7fd3c40702d0 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3cb24e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c41de220 con 0x7fd3c4152670 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 0x7fd3c40c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 0x7fd3c40c34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33610/0 (socket says 172.21.15.78:33610) 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 -- 172.21.15.78:0/77514439 learned_addr learned my addr 172.21.15.78:0/77514439 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c3fff640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152670 0x7fd3c41ead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c8fea640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3c40702d0 0x7fd3c40ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 -- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152670 msgr2=0x7fd3c41ead00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152670 0x7fd3c41ead00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 -- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3c40702d0 msgr2=0x7fd3c40ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3c40702d0 0x7fd3c40ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.706+0000 7fd3c97eb640 1 -- 172.21.15.78:0/77514439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b8009400 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.707+0000 7fd3c97eb640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 0x7fd3c40c34e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00cb40 tx=0x7fd3bc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.707+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00aca0 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.707+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc00ae00 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.707+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c4154370 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.707+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc012620 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.707+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c41ddf40 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.708+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c40676a0 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.708+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd3bc007070 con 0x7fd3c4154800 2024-02-19T14:55:55.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.709+0000 7fd3c17fa640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3b006b810 0x7fd3b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.709+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd3bc08a330 con 0x7fd3c4154800 2024-02-19T14:55:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.709+0000 7fd3c8fea640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3b006b810 0x7fd3b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.709+0000 7fd3c8fea640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3b006b810 0x7fd3b006dcd0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000f60 tx=0x7fd3b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.713+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd3bc054cb0 con 0x7fd3c4154800 2024-02-19T14:55:55.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:55.921+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fd3c4154500 con 0x7fd3c4154800 2024-02-19T14:55:56.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:55:56.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:55:56.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.209+0000 7fd3c17fa640 1 -- 172.21.15.78:0/77514439 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v90) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd3bc058370 con 0x7fd3c4154800 2024-02-19T14:55:56.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3b006b810 msgr2=0x7fd3b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3b006b810 0x7fd3b006dcd0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4000f60 tx=0x7fd3b4009040 comp rx=0 tx=0).stop 2024-02-19T14:55:56.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 msgr2=0x7fd3c40c34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 0x7fd3c40c34e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00cb40 tx=0x7fd3bc00a060 comp rx=0 tx=0).stop 2024-02-19T14:55:56.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 shutdown_connections 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3b006b810 0x7fd3b006dcd0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3c4154800 0x7fd3c40c34e0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3c4152670 0x7fd3c41ead00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 --2- 172.21.15.78:0/77514439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3c40702d0 0x7fd3c40ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 >> 172.21.15.78:0/77514439 conn(0x7fd3c40c85e0 msgr2=0x7fd3c40c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 shutdown_connections 2024-02-19T14:55:56.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.211+0000 7fd3cb24e640 1 -- 172.21.15.78:0/77514439 wait complete. 2024-02-19T14:55:56.227 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb31368f640 1 Processor -- start 2024-02-19T14:55:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb31368f640 1 -- start start 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb31368f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb30c06ed20 0x7fb30c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb31368f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30c150820 con 0x7fb30c06d750 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb31368f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30c1537d0 con 0x7fb30c150440 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb31368f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb30c0c1b60 con 0x7fb30c06ed20 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb311c2c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb30c06d750 0x7fb30c15d720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60460/0 (socket says 172.21.15.78:60460) 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb311c2c640 1 -- 172.21.15.78:0/3129398462 learned_addr learned my addr 172.21.15.78:0/3129398462 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.323+0000 7fb310c2a640 1 --2- 172.21.15.78:0/3129398462 >> v2:172.21.15.150:3300/0 conn(0x7fb30c06ed20 0x7fb30c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.324+0000 7fb303fff640 1 -- 172.21.15.78:0/3129398462 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644136342 0 0) 0x7fb30c150820 con 0x7fb30c06d750 2024-02-19T14:55:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.324+0000 7fb303fff640 1 -- 172.21.15.78:0/3129398462 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f8003520 con 0x7fb30c06d750 2024-02-19T14:55:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.324+0000 7fb310c2a640 1 -- 172.21.15.78:0/3129398462 >> v1:172.21.15.150:6789/0 conn(0x7fb30c150440 legacy=0x7fb30c0c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.324+0000 7fb310c2a640 1 -- 172.21.15.78:0/3129398462 >> v1:172.21.15.78:6789/0 conn(0x7fb30c06d750 legacy=0x7fb30c15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.324+0000 7fb310c2a640 1 -- 172.21.15.78:0/3129398462 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c153d10 con 0x7fb30c06ed20 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb310c2a640 1 --2- 172.21.15.78:0/3129398462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06ed20 0x7fb30c06d210 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb30c153960 tx=0x7fb2fc02b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aefc603f3a36cfe4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb303fff640 1 -- 172.21.15.78:0/3129398462 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc030890 con 0x7fb30c06ed20 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb303fff640 1 -- 172.21.15.78:0/3129398462 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc0309f0 con 0x7fb30c06ed20 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb303fff640 1 -- 172.21.15.78:0/3129398462 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc030d60 con 0x7fb30c06ed20 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 -- 172.21.15.78:0/3129398462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06ed20 msgr2=0x7fb30c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 --2- 172.21.15.78:0/3129398462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06ed20 0x7fb30c06d210 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb30c153960 tx=0x7fb2fc02b9d0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 -- 172.21.15.78:0/3129398462 shutdown_connections 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 --2- 172.21.15.78:0/3129398462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06ed20 0x7fb30c06d210 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 -- 172.21.15.78:0/3129398462 >> 172.21.15.78:0/3129398462 conn(0x7fb30c06b270 msgr2=0x7fb30c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 -- 172.21.15.78:0/3129398462 shutdown_connections 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.325+0000 7fb31368f640 1 -- 172.21.15.78:0/3129398462 wait complete. 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.326+0000 7fb31368f640 1 Processor -- start 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.326+0000 7fb31368f640 1 -- start start 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31368f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06d750 0x7fb30c145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31368f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 0x7fb30c1460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31368f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb30c150440 0x7fb30c1ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31368f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c152b90 con 0x7fb30c06ed20 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31368f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c1465f0 con 0x7fb30c06d750 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31368f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c1debd0 con 0x7fb30c150440 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 0x7fb30c1460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 0x7fb30c1460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33618/0 (socket says 172.21.15.78:33618) 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 -- 172.21.15.78:0/2434666978 learned_addr learned my addr 172.21.15.78:0/2434666978 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb311c2c640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb30c150440 0x7fb30c1ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31142b640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06d750 0x7fb30c145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 -- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb30c150440 msgr2=0x7fb30c1ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb30c150440 0x7fb30c1ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 -- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06d750 msgr2=0x7fb30c145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06d750 0x7fb30c145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb310c2a640 1 -- 172.21.15.78:0/2434666978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc009400 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.327+0000 7fb31142b640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06d750 0x7fb30c145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.328+0000 7fb310c2a640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 0x7fb30c1460b0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc000c00 tx=0x7fb2fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.328+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc039c90 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.328+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc03f040 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.328+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30c1dee40 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.328+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc004030 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.328+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30c1527b0 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.329+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30c074990 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.329+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb2fc004190 con 0x7fb30c06ed20 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.330+0000 7fb301ffb640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2f006b840 0x7fb2f006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.330+0000 7fb31142b640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2f006b840 0x7fb2f006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.330+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb2fc0ac570 con 0x7fb30c06ed20 2024-02-19T14:55:56.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.330+0000 7fb31142b640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2f006b840 0x7fb2f006dd00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb308001660 tx=0x7fb308009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.334+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2fc077010 con 0x7fb30c06ed20 2024-02-19T14:55:56.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.543+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb30c14ebc0 con 0x7fb30c06ed20 2024-02-19T14:55:56.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.543+0000 7fb301ffb640 1 -- 172.21.15.78:0/2434666978 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4954 (secure 0 0 0) 0x7fb2fc07a6d0 con 0x7fb30c06ed20 2024-02-19T14:55:56.547 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:56.547 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":91,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:56.213067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7527":{"gid":7527,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.78:6837/2007073435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2007073435},{"type":"v1","addr":"172.21.15.78:6837","nonce":2007073435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:56.548 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 91 2024-02-19T14:55:56.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2f006b840 msgr2=0x7fb2f006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2f006b840 0x7fb2f006dd00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb308001660 tx=0x7fb308009040 comp rx=0 tx=0).stop 2024-02-19T14:55:56.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 msgr2=0x7fb30c1460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 0x7fb30c1460b0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc000c00 tx=0x7fb2fc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 shutdown_connections 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2f006b840 0x7fb2f006dd00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb30c06ed20 0x7fb30c1460b0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb30c150440 0x7fb30c1ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 --2- 172.21.15.78:0/2434666978 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb30c06d750 0x7fb30c145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 >> 172.21.15.78:0/2434666978 conn(0x7fb30c06b270 msgr2=0x7fb30c06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 shutdown_connections 2024-02-19T14:55:56.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.545+0000 7fb31368f640 1 -- 172.21.15.78:0/2434666978 wait complete. 2024-02-19T14:55:56.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-19T14:55:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.659+0000 7f81c384e640 1 Processor -- start 2024-02-19T14:55:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.659+0000 7f81c384e640 1 -- start start 2024-02-19T14:55:56.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c384e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc152610 0x7f81bc0c1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c384e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81bc06f270 0x7f81bc158550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c384e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81bc0c1cf0 con 0x7f81bc155cd0 2024-02-19T14:55:56.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c384e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc154fe0 con 0x7f81bc06f270 2024-02-19T14:55:56.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c384e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0723b0 con 0x7f81bc152610 2024-02-19T14:55:56.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c1deb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81bc06f270 0x7f81bc158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c1deb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81bc06f270 0x7f81bc158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33630/0 (socket says 172.21.15.78:33630) 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c1deb640 1 -- 172.21.15.78:0/2661285539 learned_addr learned my addr 172.21.15.78:0/2661285539 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.660+0000 7f81c15ea640 1 --2- 172.21.15.78:0/2661285539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc152610 0x7f81bc0c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81c1deb640 1 -- 172.21.15.78:0/2661285539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc152610 msgr2=0x7f81bc0c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81c1deb640 1 --2- 172.21.15.78:0/2661285539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc152610 0x7f81bc0c1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81c1deb640 1 -- 172.21.15.78:0/2661285539 >> v1:172.21.15.150:6789/0 conn(0x7f81bc155cd0 legacy=0x7f81bc072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81c1deb640 1 -- 172.21.15.78:0/2661285539 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc0c1a10 con 0x7f81bc06f270 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81c1deb640 1 --2- 172.21.15.78:0/2661285539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc06f270 0x7f81bc158550 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009870 tx=0x7f81b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0e20058c7a7cae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81b3fff640 1 -- 172.21.15.78:0/2661285539 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4031640 con 0x7f81bc06f270 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81b3fff640 1 -- 172.21.15.78:0/2661285539 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b40317a0 con 0x7f81bc06f270 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.661+0000 7f81b3fff640 1 -- 172.21.15.78:0/2661285539 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4031b30 con 0x7f81bc06f270 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 -- 172.21.15.78:0/2661285539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc06f270 msgr2=0x7f81bc158550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 --2- 172.21.15.78:0/2661285539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc06f270 0x7f81bc158550 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009870 tx=0x7f81b402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 -- 172.21.15.78:0/2661285539 shutdown_connections 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 --2- 172.21.15.78:0/2661285539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc06f270 0x7f81bc158550 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 --2- 172.21.15.78:0/2661285539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc152610 0x7f81bc0c1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 -- 172.21.15.78:0/2661285539 >> 172.21.15.78:0/2661285539 conn(0x7f81bc0c7590 msgr2=0x7f81bc0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 -- 172.21.15.78:0/2661285539 shutdown_connections 2024-02-19T14:55:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.662+0000 7f81c384e640 1 -- 172.21.15.78:0/2661285539 wait complete. 2024-02-19T14:55:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.663+0000 7f81c384e640 1 Processor -- start 2024-02-19T14:55:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.663+0000 7f81c384e640 1 -- start start 2024-02-19T14:55:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c384e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc06f270 0x7f81bc0ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c384e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 0x7f81bc14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c384e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81bc155cd0 0x7f81bc14cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c384e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc14d440 con 0x7f81bc152610 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c384e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc157e50 con 0x7f81bc155cd0 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c384e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0a7290 con 0x7f81bc06f270 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 0x7f81bc14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 0x7f81bc14c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33636/0 (socket says 172.21.15.78:33636) 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 -- 172.21.15.78:0/3897678180 learned_addr learned my addr 172.21.15.78:0/3897678180 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c1deb640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81bc155cd0 0x7f81bc14cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c15ea640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc06f270 0x7f81bc0ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 -- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc06f270 msgr2=0x7f81bc0ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc06f270 0x7f81bc0ab6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 -- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81bc155cd0 msgr2=0x7f81bc14cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81bc155cd0 0x7f81bc14cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 -- 172.21.15.78:0/3897678180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40094d0 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.664+0000 7f81c0de9640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 0x7f81bc14c9c0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00c400 tx=0x7f81ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.665+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0052d0 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.665+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac005450 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.665+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc0a7570 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.665+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac007260 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.665+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc1583d0 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.666+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc074990 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.666+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81ac0073c0 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.666+0000 7f81b1ffb640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81a006b840 0x7f81a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.667+0000 7f81c15ea640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81a006b840 0x7f81a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.667+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f81ac089570 con 0x7f81bc152610 2024-02-19T14:55:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.667+0000 7f81c15ea640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81a006b840 0x7f81a006dd00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f81b80113e0 tx=0x7f81b800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.671+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81ac053e60 con 0x7f81bc152610 2024-02-19T14:55:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.842+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f81bc0c1cf0 con 0x7f81bc152610 2024-02-19T14:55:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.845+0000 7f81b1ffb640 1 -- 172.21.15.78:0/3897678180 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v37)=0 v37) v1 ==== 116+0+0 (secure 0 0 0) 0x7f81bc0c1cf0 con 0x7f81bc152610 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81a006b840 msgr2=0x7f81a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81a006b840 0x7f81a006dd00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f81b80113e0 tx=0x7f81b800b040 comp rx=0 tx=0).stop 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 msgr2=0x7f81bc14c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 0x7f81bc14c9c0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00c400 tx=0x7f81ac00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 shutdown_connections 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81a006b840 0x7f81a006dd00 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81bc152610 0x7f81bc14c9c0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81bc155cd0 0x7f81bc14cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 --2- 172.21.15.78:0/3897678180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81bc06f270 0x7f81bc0ab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 >> 172.21.15.78:0/3897678180 conn(0x7f81bc0c7590 msgr2=0x7f81bc150460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:56.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 shutdown_connections 2024-02-19T14:55:56.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.847+0000 7f81c384e640 1 -- 172.21.15.78:0/3897678180 wait complete. 2024-02-19T14:55:56.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:55:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 Processor -- start 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 -- start start 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 0x7f335c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 0x7f335c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335c1551f0 con 0x7f335c0c8e00 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335c155490 con 0x7f335c0c5730 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.960+0000 7f33617bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c155770 con 0x7f335c152700 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 0x7f335c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335a7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 0x7f335c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335b7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f335c0c8e00 0x7f335c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53812/0 (socket says 172.21.15.78:53812) 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335b7fe640 1 -- 172.21.15.78:0/1362428557 learned_addr learned my addr 172.21.15.78:0/1362428557 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335a7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 0x7f335c06c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57750/0 (socket says 172.21.15.78:57750) 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 0x7f335c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33230/0 (socket says 172.21.15.78:33230) 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335affd640 1 -- 172.21.15.78:0/1362428557 >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 msgr2=0x7f335c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335affd640 1 --2- 172.21.15.78:0/1362428557 >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 0x7f335c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335affd640 1 -- 172.21.15.78:0/1362428557 >> v1:172.21.15.150:6789/0 conn(0x7f335c0c8e00 legacy=0x7f335c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335affd640 1 -- 172.21.15.78:0/1362428557 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c155a50 con 0x7f335c152700 2024-02-19T14:55:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f335a7fc640 1 --2- 172.21.15.78:0/1362428557 >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 0x7f335c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:55:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.961+0000 7f3359ffb640 1 -- 172.21.15.78:0/1362428557 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130174736 0 0) 0x7f335c1551f0 con 0x7f335c0c8e00 2024-02-19T14:55:56.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f335affd640 1 --2- 172.21.15.78:0/1362428557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 0x7f335c0c51f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f334c009d90 tx=0x7f334c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f54b53a673982f60 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f3359ffb640 1 -- 172.21.15.78:0/1362428557 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c02cc50 con 0x7f335c152700 2024-02-19T14:55:56.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f3359ffb640 1 -- 172.21.15.78:0/1362428557 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c02cdb0 con 0x7f335c152700 2024-02-19T14:55:56.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f3359ffb640 1 -- 172.21.15.78:0/1362428557 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c034660 con 0x7f335c152700 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 -- 172.21.15.78:0/1362428557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 msgr2=0x7f335c0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 --2- 172.21.15.78:0/1362428557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 0x7f335c0c51f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f334c009d90 tx=0x7f334c02bea0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 -- 172.21.15.78:0/1362428557 shutdown_connections 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 --2- 172.21.15.78:0/1362428557 >> v2:172.21.15.150:3300/0 conn(0x7f335c0c5730 0x7f335c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 --2- 172.21.15.78:0/1362428557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c152700 0x7f335c0c51f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 -- 172.21.15.78:0/1362428557 >> 172.21.15.78:0/1362428557 conn(0x7f335c0cac90 msgr2=0x7f335c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 -- 172.21.15.78:0/1362428557 shutdown_connections 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.962+0000 7f33617bc640 1 -- 172.21.15.78:0/1362428557 wait complete. 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 Processor -- start 2024-02-19T14:55:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 -- start start 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 0x7f335c1ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335c0c8e00 0x7f335c0aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f335c152700 0x7f335c0acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c149970 con 0x7f335c152700 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c0aa7f0 con 0x7f335c0c8e00 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f33617bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c0aaa70 con 0x7f335c0c5730 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f335affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 0x7f335c1ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f335a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335c0c8e00 0x7f335c0aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f335affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 0x7f335c1ea4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33246/0 (socket says 172.21.15.78:33246) 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f335affd640 1 -- 172.21.15.78:0/2724058419 learned_addr learned my addr 172.21.15.78:0/2724058419 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.964+0000 7f335a7fc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335c0c8e00 0x7f335c0aa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57752/0 (socket says 172.21.15.78:57752) 2024-02-19T14:55:56.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335b7fe640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f335c152700 0x7f335c0acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335affd640 1 -- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335c0c8e00 msgr2=0x7f335c0aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335affd640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335c0c8e00 0x7f335c0aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335affd640 1 -- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f335c152700 msgr2=0x7f335c0acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335affd640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f335c152700 0x7f335c0acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335affd640 1 -- 172.21.15.78:0/2724058419 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009400 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f335affd640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 0x7f335c1ea4a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f334c009d90 tx=0x7f334c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c040650 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c0407b0 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c009a70 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c040b60 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.965+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335c0ab030 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.966+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335c0c5be0 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.967+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f334c040cc0 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.967+0000 7f33537fe640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.967+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f334c0ad0d0 con 0x7f335c0c5730 2024-02-19T14:55:56.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.967+0000 7f335a7fc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:56.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.967+0000 7f335a7fc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f335c0be900 tx=0x7f3344009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:56.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:56.973+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f334c077ad0 con 0x7f335c0c5730 2024-02-19T14:55:57.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.178+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f335c0c4ec0 con 0x7f335c0c5730 2024-02-19T14:55:57.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.179+0000 7f33537fe640 1 -- 172.21.15.78:0/2724058419 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4954 (secure 0 0 0) 0x7f334c07b190 con 0x7f335c0c5730 2024-02-19T14:55:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:55:57.182 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5930,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":91,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:56.213067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7527":{"gid":7527,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.78:6837/2007073435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2007073435},{"type":"v1","addr":"172.21.15.78:6837","nonce":2007073435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:55:57.182 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 91 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 msgr2=0x7f333c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f335c0be900 tx=0x7f3344009190 comp rx=0 tx=0).stop 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 msgr2=0x7f335c1ea4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 0x7f335c1ea4a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f334c009d90 tx=0x7f334c004a40 comp rx=0 tx=0).stop 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 shutdown_connections 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f333c06b810 0x7f333c06dcd0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f335c152700 0x7f335c0acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335c0c8e00 0x7f335c0aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 --2- 172.21.15.78:0/2724058419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f335c0c5730 0x7f335c1ea4a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 >> 172.21.15.78:0/2724058419 conn(0x7f335c0cac90 msgr2=0x7f335c14e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 shutdown_connections 2024-02-19T14:55:57.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.181+0000 7f33617bc640 1 -- 172.21.15.78:0/2724058419 wait complete. 2024-02-19T14:55:57.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-02-19T14:55:38.311785+0000', 'modified': '2024-02-19T14:55:56.213067+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5927}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5927': {'gid': 5927, 'name': 'b', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7527': {'gid': 7527, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.78:6837/2007073435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2007073435}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2007073435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:55:57.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:55:57.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-19T14:55:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c950d640 1 Processor -- start 2024-02-19T14:55:57.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c950d640 1 -- start start 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c950d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27c4152700 0x7f27c40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c950d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c41551f0 con 0x7f27c40c8e00 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c950d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c4155490 con 0x7f27c40c5730 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c950d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c4155770 con 0x7f27c4152700 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27c4152700 0x7f27c40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f27c40c8e00 0x7f27c415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48768/0 (socket says 172.21.15.78:48768) 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.698+0000 7f27c37fe640 1 -- 172.21.15.78:0/1577791936 learned_addr learned my addr 172.21.15.78:0/1577791936 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.699+0000 7f27c2ffd640 1 -- 172.21.15.78:0/1577791936 >> v1:172.21.15.150:6789/0 conn(0x7f27c40c5730 legacy=0x7f27c406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.699+0000 7f27c2ffd640 1 -- 172.21.15.78:0/1577791936 >> v1:172.21.15.78:6789/0 conn(0x7f27c40c8e00 legacy=0x7f27c415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.699+0000 7f27c1ffb640 1 -- 172.21.15.78:0/1577791936 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536575929 0 0) 0x7f27c41551f0 con 0x7f27c40c8e00 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.699+0000 7f27c2ffd640 1 -- 172.21.15.78:0/1577791936 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4155a50 con 0x7f27c4152700 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.699+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/1577791936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c4152700 0x7f27c40c51f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097a0 tx=0x7f27bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3c87bf18f80283 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c1ffb640 1 -- 172.21.15.78:0/1577791936 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc031830 con 0x7f27c4152700 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c1ffb640 1 -- 172.21.15.78:0/1577791936 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc031990 con 0x7f27c4152700 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c1ffb640 1 -- 172.21.15.78:0/1577791936 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc031d20 con 0x7f27c4152700 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 -- 172.21.15.78:0/1577791936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c4152700 msgr2=0x7f27c40c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 --2- 172.21.15.78:0/1577791936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c4152700 0x7f27c40c51f0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097a0 tx=0x7f27bc02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 -- 172.21.15.78:0/1577791936 shutdown_connections 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 --2- 172.21.15.78:0/1577791936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c4152700 0x7f27c40c51f0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 -- 172.21.15.78:0/1577791936 >> 172.21.15.78:0/1577791936 conn(0x7f27c40cac90 msgr2=0x7f27c40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 -- 172.21.15.78:0/1577791936 shutdown_connections 2024-02-19T14:55:57.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.700+0000 7f27c950d640 1 -- 172.21.15.78:0/1577791936 wait complete. 2024-02-19T14:55:57.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.701+0000 7f27c950d640 1 Processor -- start 2024-02-19T14:55:57.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.701+0000 7f27c950d640 1 -- start start 2024-02-19T14:55:57.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c950d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 0x7f27c41ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c950d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c40c8e00 0x7f27c41ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c950d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27c4152700 0x7f27c40c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c950d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40c2780 con 0x7f27c40c5730 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c950d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40aaf90 con 0x7f27c40c8e00 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c950d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c41eee50 con 0x7f27c4152700 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 0x7f27c41ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 0x7f27c41ee3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53906/0 (socket says 172.21.15.78:53906) 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 -- 172.21.15.78:0/3536464478 learned_addr learned my addr 172.21.15.78:0/3536464478 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c37fe640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27c4152700 0x7f27c40c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c27fc640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c40c8e00 0x7f27c41ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 -- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27c4152700 msgr2=0x7f27c40c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27c4152700 0x7f27c40c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 -- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c40c8e00 msgr2=0x7f27c41ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c40c8e00 0x7f27c41ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c2ffd640 1 -- 172.21.15.78:0/3536464478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009400 con 0x7f27c40c5730 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.702+0000 7f27c37fe640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27c4152700 0x7f27c40c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.703+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 0x7f27c41ee3d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097a0 tx=0x7f27bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.703+0000 7f27ab7fe640 1 -- 172.21.15.78:0/3536464478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc004030 con 0x7f27c40c5730 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.703+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c40ab230 con 0x7f27c40c5730 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.703+0000 7f27ab7fe640 1 -- 172.21.15.78:0/3536464478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc0041b0 con 0x7f27c40c5730 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.703+0000 7f27ab7fe640 1 -- 172.21.15.78:0/3536464478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc03b7e0 con 0x7f27c40c5730 2024-02-19T14:55:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.703+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c40ab7f0 con 0x7f27c40c5730 2024-02-19T14:55:57.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.704+0000 7f27ab7fe640 1 -- 172.21.15.78:0/3536464478 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f27bc043070 con 0x7f27c40c5730 2024-02-19T14:55:57.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.705+0000 7f27ab7fe640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f27b006b810 0x7f27b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.705+0000 7f27c27fc640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f27b006b810 0x7f27b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.705+0000 7f27ab7fe640 1 -- 172.21.15.78:0/3536464478 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f27bc0ac130 con 0x7f27c40c5730 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.705+0000 7f27c27fc640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f27b006b810 0x7f27b006dcd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f27c40c1440 tx=0x7f27b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.706+0000 7f27cc1c1740 1 Processor -- start 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.706+0000 7f27cc1c1740 1 -- start start 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce258b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6cdeaae0 0x55bc6ce24b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6cde9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cda4cd0 con 0x55bc6ce278b0 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cd57f20 con 0x55bc6ce24740 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cd55450 con 0x55bc6cdeaae0 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6cde9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6cde9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53922/0 (socket says 172.21.15.78:53922) 2024-02-19T14:55:57.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 -- 172.21.15.78:0/3926687111 learned_addr learned my addr 172.21.15.78:0/3926687111 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c27fc640 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6cdeaae0 0x55bc6ce24b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 -- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6cdeaae0 msgr2=0x55bc6ce24b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6cdeaae0 0x55bc6ce24b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 -- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 msgr2=0x55bc6ce258b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce258b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 -- 172.21.15.78:0/3926687111 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc6cd05f20 con 0x55bc6ce278b0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27c37fe640 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6cde9860 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f27b4007470 tx=0x7f27b4005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad740735033cf36e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27a8ff9640 1 -- 172.21.15.78:0/3926687111 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4007680 con 0x55bc6ce278b0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27a8ff9640 1 -- 172.21.15.78:0/3926687111 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b4001890 con 0x55bc6ce278b0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.707+0000 7f27a8ff9640 1 -- 172.21.15.78:0/3926687111 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4001c20 con 0x55bc6ce278b0 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 -- 172.21.15.78:0/3926687111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 msgr2=0x55bc6cde9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6cde9860 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f27b4007470 tx=0x7f27b4005bc0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 -- 172.21.15.78:0/3926687111 shutdown_connections 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6cde9860 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6cdeaae0 0x55bc6ce24b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/3926687111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce258b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 -- 172.21.15.78:0/3926687111 >> 172.21.15.78:0/3926687111 conn(0x55bc6cde6120 msgr2=0x55bc6ce0ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 -- 172.21.15.78:0/3926687111 shutdown_connections 2024-02-19T14:55:57.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.708+0000 7f27cc1c1740 1 -- 172.21.15.78:0/3926687111 wait complete. 2024-02-19T14:55:57.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 -- start start 2024-02-19T14:55:57.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 Processor -- start 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce24740 0x55bc6cde83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce20560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce1fba0 0x55bc6ce220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cd50c30 con 0x55bc6ce278b0 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6ccf5d30 con 0x55bc6ce1fba0 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc6ca24310 con 0x55bc6ce24740 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.709+0000 7f27cc1c1740 10 client.0 _ll_register_callbacks cb 0x55bc6ccfdd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce20560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce20560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53932/0 (socket says 172.21.15.78:53932) 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 -- 172.21.15.78:0/2860790462 learned_addr learned my addr 172.21.15.78:0/2860790462 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c37fe640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce1fba0 0x55bc6ce220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce24740 0x55bc6cde83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 -- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce24740 msgr2=0x55bc6cde83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce24740 0x55bc6cde83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 -- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce1fba0 msgr2=0x55bc6ce220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce1fba0 0x55bc6ce220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 -- 172.21.15.78:0/2860790462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b400f040 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27c27fc640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce20560 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f27b800ebe0 tx=0x7f27b80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e7a7fea7f18d21 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2860790462 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8013420 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2860790462 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f27b8022c80 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc6ca506c0 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2860790462 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27b8022e40 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27cc1c1740 10 client.7584 fetch_fsmap learned FSMap version 91 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x55bc6cd4f2b0 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.710+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2860790462 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bc6ca506c0 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2860790462 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 91) v1 ==== 3282+0+0 (secure 0 0 0) 0x7f27b8020070 con 0x55bc6ce278b0 2024-02-19T14:55:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27cc1c1740 10 client.7584 fetch_fsmap finished waiting for FSMap version 91 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27cc1c1740 10 client.7584 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27cc1c1740 20 client.7584 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce391f0 0x55bc6ce395d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27cc1c1740 4 client.7584 mds_command: new command op to 5927 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 --> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc6caa1270 con 0x55bc6ce391f0 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce391f0 0x55bc6ce395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27a1ffb640 10 client.7584 ms_handle_connect on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.711+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce391f0 0x55bc6ce395d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e55e216179c42c server_cookie=724691b524f77ac8 in_seq=0 out_seq=0 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2860790462 <== mds.0 v2:172.21.15.150:6832/2663291681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bc6caa1270 con 0x55bc6ce391f0 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27a1ffb640 10 client.7584 handle_command_reply: tid=0 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 client.7584 shutdown 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 msgr2=0x55bc6ce20560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce20560 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f27b800ebe0 tx=0x7f27b80014a0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 shutdown_connections 2024-02-19T14:55:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce391f0 0x55bc6ce395d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce20560 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27a1ffb640 0 client.7584 ms_handle_reset on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce1fba0 0x55bc6ce220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2860790462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce24740 0x55bc6cde83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 >> 172.21.15.78:0/2860790462 conn(0x55bc6cdeaae0 msgr2=0x55bc6ce256e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 shutdown_connections 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2860790462 wait complete. 2024-02-19T14:55:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.712+0000 7f27cc1c1740 20 client.7584 trim_cache size 0 max 16384 2024-02-19T14:55:57.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 Processor -- start 2024-02-19T14:55:57.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 -- start start 2024-02-19T14:55:57.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce1fba0 0x55bc6ce1db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce1ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce0aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6ccd6490 con 0x55bc6ce278b0 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6ce25c40 con 0x55bc6ce24740 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc6ce270f0 con 0x55bc6ce1fba0 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27c37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce0aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27c37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce0aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53946/0 (socket says 172.21.15.78:53946) 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27c37fe640 1 -- 172.21.15.78:0/1409869317 learned_addr learned my addr 172.21.15.78:0/1409869317 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce1fba0 0x55bc6ce1db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.729+0000 7f27c27fc640 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce1ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27c37fe640 1 -- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce1fba0 msgr2=0x55bc6ce1db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27c37fe640 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce1fba0 0x55bc6ce1db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27c37fe640 1 -- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 msgr2=0x55bc6ce1ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27c37fe640 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce1ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27c37fe640 1 -- 172.21.15.78:0/1409869317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc6cceba20 con 0x55bc6ce278b0 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27c37fe640 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce0aab0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f27b4000f60 tx=0x7f27b400c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcd2a94fa19e479 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27a3fff640 1 -- 172.21.15.78:0/1409869317 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4007680 con 0x55bc6ce278b0 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27a3fff640 1 -- 172.21.15.78:0/1409869317 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b400b070 con 0x55bc6ce278b0 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27a3fff640 1 -- 172.21.15.78:0/1409869317 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4001c40 con 0x55bc6ce278b0 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 -- 172.21.15.78:0/1409869317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 msgr2=0x55bc6ce0aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce0aab0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f27b4000f60 tx=0x7f27b400c000 comp rx=0 tx=0).stop 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 -- 172.21.15.78:0/1409869317 shutdown_connections 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce278b0 0x55bc6ce0aab0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce1ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/1409869317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce1fba0 0x55bc6ce1db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 -- 172.21.15.78:0/1409869317 >> 172.21.15.78:0/1409869317 conn(0x55bc6cdeaae0 msgr2=0x55bc6ce20370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.730+0000 7f27cc1c1740 1 -- 172.21.15.78:0/1409869317 shutdown_connections 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.731+0000 7f27cc1c1740 1 -- 172.21.15.78:0/1409869317 wait complete. 2024-02-19T14:55:57.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.731+0000 7f27cc1c1740 1 -- start start 2024-02-19T14:55:57.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.731+0000 7f27cc1c1740 1 Processor -- start 2024-02-19T14:55:57.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 0x55bc6cde72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce26240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce278b0 0x55bc6cde9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cde93a0 con 0x55bc6ce1fba0 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cdfdd80 con 0x55bc6ce24740 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc6cdfd220 con 0x55bc6ce278b0 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 10 client.0 _ll_register_callbacks cb 0x7f27b8003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 0x55bc6cde72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 0x55bc6cde72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53958/0 (socket says 172.21.15.78:53958) 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 -- 172.21.15.78:0/2080102524 learned_addr learned my addr 172.21.15.78:0/2080102524 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c37fe640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce278b0 0x55bc6cde9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c27fc640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce26240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 -- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce278b0 msgr2=0x55bc6cde9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce278b0 0x55bc6cde9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 -- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 msgr2=0x55bc6ce26240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce26240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 -- 172.21.15.78:0/2080102524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc6cd05f20 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27c2ffd640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 0x55bc6cde72e0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f27bc07a4f0 tx=0x7f27bc047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d4da403d3721a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2080102524 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc082f00 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2080102524 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f27bc08e040 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27b4001890 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2080102524 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27bc076e60 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 10 client.7590 fetch_fsmap learned FSMap version 91 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.732+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x55bc6cd11780 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2080102524 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc096070 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2080102524 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 91) v1 ==== 3282+0+0 (secure 0 0 0) 0x7f27bc092190 con 0x55bc6ce1fba0 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 10 client.7590 fetch_fsmap finished waiting for FSMap version 91 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 10 client.7590 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 20 client.7590 populate_metadata read hostname 'smithi078' 2024-02-19T14:55:57.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce045b0 0x55bc6ce04990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 4 client.7590 mds_command: new command op to 5927 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 --> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bc6ca8d610 con 0x55bc6ce045b0 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27c27fc640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce045b0 0x55bc6ce04990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27a1ffb640 10 client.7590 ms_handle_connect on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27c27fc640 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce045b0 0x55bc6ce04990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=555dde863602848b server_cookie=d83e3703d8266175 in_seq=0 out_seq=0 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27a1ffb640 1 -- 172.21.15.78:0/2080102524 <== mds.0 v2:172.21.15.150:6832/2663291681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55bc6ca8d610 con 0x55bc6ce045b0 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27a1ffb640 10 client.7590 handle_command_reply: tid=0 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 1 client.7590 shutdown 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 msgr2=0x55bc6cde72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.733+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 0x55bc6cde72e0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f27bc07a4f0 tx=0x7f27bc047040 comp rx=0 tx=0).stop 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 shutdown_connections 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55bc6ce045b0 0x55bc6ce04990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bc6ce278b0 0x55bc6cde9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc6ce24740 0x55bc6ce26240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27a1ffb640 0 client.7590 ms_handle_reset on v2:172.21.15.150:6832/2663291681 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 --2- 172.21.15.78:0/2080102524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc6ce1fba0 0x55bc6cde72e0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 >> 172.21.15.78:0/2080102524 conn(0x55bc6cdeaae0 msgr2=0x55bc6ce21e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 shutdown_connections 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 1 -- 172.21.15.78:0/2080102524 wait complete. 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27cc1c1740 20 client.7590 trim_cache size 0 max 16384 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:55:57.743 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5927, 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 7.5480129250000001, 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 91, 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 112, 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 112, 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 23.885180218999999 2024-02-19T14:55:57.744 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f27b006b810 msgr2=0x7f27b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f27b006b810 0x7f27b006dcd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f27c40c1440 tx=0x7f27b8009040 comp rx=0 tx=0).stop 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 msgr2=0x7f27c41ee3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 0x7f27c41ee3d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097a0 tx=0x7f27bc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 shutdown_connections 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f27b006b810 0x7f27b006dcd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27c4152700 0x7f27c40c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c40c8e00 0x7f27c41ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 --2- 172.21.15.78:0/3536464478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27c40c5730 0x7f27c41ee3d0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:55:57.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 >> 172.21.15.78:0/3536464478 conn(0x7f27c40cac90 msgr2=0x7f27c406c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:55:57.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 shutdown_connections 2024-02-19T14:55:57.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:55:57.734+0000 7f27c950d640 1 -- 172.21.15.78:0/3536464478 wait complete. 2024-02-19T14:55:57.747 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:38.311785+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 91, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5927": { "addr": "172.21.15.150:6833/2663291681", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2663291681, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2663291681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5927, "incarnation": 86, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_7527": { "addr": "172.21.15.78:6837/2007073435", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2007073435, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2007073435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7527, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:standby-replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 112, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-19T14:55:56.213067+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5927 } } } ], "standbys": [ { "addr": "172.21.15.150:6835/2437926852", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2437926852, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2437926852, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5930, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/3062592461", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3062592461, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3062592461, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7302, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:55:57.748 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:standby-replay'}] 2024-02-19T14:55:57.749 INFO:tasks.cephfs.test_failover:current = [{'gid': 7302, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/3062592461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3062592461}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3062592461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 5927, 'name': 'b', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7527, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.78:6837/2007073435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2007073435}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2007073435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5930, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2437926852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2437926852}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2437926852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}] 2024-02-19T14:55:57.749 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-02-19T14:55:57.749 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:55:58.253 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:56:02.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:02.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.859+0000 7fab4cb27640 1 Processor -- start 2024-02-19T14:56:02.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.859+0000 7fab4cb27640 1 -- start start 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4cb27640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c8500 0x7fab480c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4cb27640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab480c2f50 0x7fab48158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4cb27640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480c34a0 con 0x7fab480c42e0 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4cb27640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4806c090 con 0x7fab480c2f50 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4cb27640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4806d6a0 con 0x7fab480c8500 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab45d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fab480c42e0 0x7fab48158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48784/0 (socket says 172.21.15.78:48784) 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab45d9b640 1 -- 172.21.15.78:0/1801740810 learned_addr learned my addr 172.21.15.78:0/1801740810 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:02.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4659c640 1 --2- 172.21.15.78:0/1801740810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c8500 0x7fab480c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab46d9d640 1 --2- 172.21.15.78:0/1801740810 >> v2:172.21.15.150:3300/0 conn(0x7fab480c2f50 0x7fab48158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4559a640 1 -- 172.21.15.78:0/1801740810 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342607696 0 0) 0x7fab480c34a0 con 0x7fab480c42e0 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4559a640 1 -- 172.21.15.78:0/1801740810 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34003520 con 0x7fab480c42e0 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4659c640 1 -- 172.21.15.78:0/1801740810 >> v2:172.21.15.150:3300/0 conn(0x7fab480c2f50 msgr2=0x7fab48158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4659c640 1 --2- 172.21.15.78:0/1801740810 >> v2:172.21.15.150:3300/0 conn(0x7fab480c2f50 0x7fab48158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4659c640 1 -- 172.21.15.78:0/1801740810 >> v1:172.21.15.78:6789/0 conn(0x7fab480c42e0 legacy=0x7fab48158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab4659c640 1 -- 172.21.15.78:0/1801740810 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4806c8e0 con 0x7fab480c8500 2024-02-19T14:56:02.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.860+0000 7fab46d9d640 1 --2- 172.21.15.78:0/1801740810 >> v2:172.21.15.150:3300/0 conn(0x7fab480c2f50 0x7fab48158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:02.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.861+0000 7fab4659c640 1 --2- 172.21.15.78:0/1801740810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c8500 0x7fab480c88e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fab380097a0 tx=0x7fab3802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18002ff38897d128 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:02.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.861+0000 7fab4559a640 1 -- 172.21.15.78:0/1801740810 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3802e070 con 0x7fab480c8500 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.861+0000 7fab4559a640 1 -- 172.21.15.78:0/1801740810 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab380315a0 con 0x7fab480c8500 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.861+0000 7fab4559a640 1 -- 172.21.15.78:0/1801740810 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38031910 con 0x7fab480c8500 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 -- 172.21.15.78:0/1801740810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c8500 msgr2=0x7fab480c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 --2- 172.21.15.78:0/1801740810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c8500 0x7fab480c88e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fab380097a0 tx=0x7fab3802b840 comp rx=0 tx=0).stop 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 -- 172.21.15.78:0/1801740810 shutdown_connections 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 --2- 172.21.15.78:0/1801740810 >> v2:172.21.15.150:3300/0 conn(0x7fab480c2f50 0x7fab48158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 --2- 172.21.15.78:0/1801740810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c8500 0x7fab480c88e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 -- 172.21.15.78:0/1801740810 >> 172.21.15.78:0/1801740810 conn(0x7fab480aa300 msgr2=0x7fab480aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 -- 172.21.15.78:0/1801740810 shutdown_connections 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.862+0000 7fab4cb27640 1 -- 172.21.15.78:0/1801740810 wait complete. 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.863+0000 7fab4cb27640 1 Processor -- start 2024-02-19T14:56:02.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.863+0000 7fab4cb27640 1 -- start start 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4cb27640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c2f50 0x7fab48149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4cb27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab480c8500 0x7fab481e1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4cb27640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 0x7fab480ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4cb27640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4814a4a0 con 0x7fab480acad0 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4cb27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481ebb10 con 0x7fab480c8500 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4cb27640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4814a150 con 0x7fab480c2f50 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 0x7fab480ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 0x7fab480ab3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53976/0 (socket says 172.21.15.78:53976) 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 -- 172.21.15.78:0/3738295524 learned_addr learned my addr 172.21.15.78:0/3738295524 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab45d9b640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab480c8500 0x7fab481e1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab4659c640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c2f50 0x7fab48149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:02.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 -- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c2f50 msgr2=0x7fab48149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c2f50 0x7fab48149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 -- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab480c8500 msgr2=0x7fab481e1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab480c8500 0x7fab481e1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 -- 172.21.15.78:0/3738295524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40009a70 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab46d9d640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 0x7fab480ab3f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fab4000efc0 tx=0x7fab40009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.864+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40009d70 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.865+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab38009400 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.865+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000f040 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.865+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40003990 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.865+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab481ebd50 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.865+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4814a7c0 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.866+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fab40014480 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.866+0000 7fab3effd640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab2806b840 0x7fab2806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.866+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fab4008aea0 con 0x7fab480acad0 2024-02-19T14:56:02.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.866+0000 7fab4659c640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab2806b840 0x7fab2806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:02.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.867+0000 7fab4659c640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab2806b840 0x7fab2806dd00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fab38036000 tx=0x7fab3803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:02.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:02.871+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fab40055710 con 0x7fab480acad0 2024-02-19T14:56:03.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.082+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab4814a5e0 con 0x7fab480acad0 2024-02-19T14:56:03.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.083+0000 7fab3effd640 1 -- 172.21.15.78:0/3738295524 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4951 (secure 0 0 0) 0x7fab4814a5e0 con 0x7fab480acad0 2024-02-19T14:56:03.085 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:03.085 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7599,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3069973851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3069973851},{"type":"v1","addr":"172.21.15.78:6837","nonce":3069973851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:58.223166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5930":{"gid":5930,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:56:03.086 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 94 2024-02-19T14:56:03.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab2806b840 msgr2=0x7fab2806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab2806b840 0x7fab2806dd00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fab38036000 tx=0x7fab3803d040 comp rx=0 tx=0).stop 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 msgr2=0x7fab480ab3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 0x7fab480ab3f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fab4000efc0 tx=0x7fab40009f90 comp rx=0 tx=0).stop 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 shutdown_connections 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab2806b840 0x7fab2806dd00 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab480acad0 0x7fab480ab3f0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab480c8500 0x7fab481e1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 --2- 172.21.15.78:0/3738295524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab480c2f50 0x7fab48149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 >> 172.21.15.78:0/3738295524 conn(0x7fab480aa300 msgr2=0x7fab481553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 shutdown_connections 2024-02-19T14:56:03.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.085+0000 7fab4cb27640 1 -- 172.21.15.78:0/3738295524 wait complete. 2024-02-19T14:56:03.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-02-19T14:55:38.311785+0000', 'modified': '2024-02-19T14:55:58.223166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5927}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5927': {'gid': 5927, 'name': 'b', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5930': {'gid': 5930, 'name': 'd', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.150:6835/2437926852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2437926852}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2437926852}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:03.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:56:03.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-19T14:56:03.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.600+0000 7f8454055640 1 Processor -- start 2024-02-19T14:56:03.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f8454055640 1 -- start start 2024-02-19T14:56:03.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f8454055640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f844c0c8e00 0x7f844c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f8454055640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c06d910 0x7f844c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f8454055640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c06cd20 con 0x7f844c152700 2024-02-19T14:56:03.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f8454055640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c1551f0 con 0x7f844c0c8e00 2024-02-19T14:56:03.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f8454055640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c155490 con 0x7f844c06d910 2024-02-19T14:56:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f84515f0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f844c0c8e00 0x7f844c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.601+0000 7f84525f2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c06d910 0x7f844c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f8451df1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f844c152700 0x7f844c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51416/0 (socket says 172.21.15.78:51416) 2024-02-19T14:56:03.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f8451df1640 1 -- 172.21.15.78:0/3198050540 learned_addr learned my addr 172.21.15.78:0/3198050540 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f84525f2640 1 -- 172.21.15.78:0/3198050540 >> v2:172.21.15.150:3300/0 conn(0x7f844c0c8e00 msgr2=0x7f844c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f84525f2640 1 --2- 172.21.15.78:0/3198050540 >> v2:172.21.15.150:3300/0 conn(0x7f844c0c8e00 0x7f844c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f84525f2640 1 -- 172.21.15.78:0/3198050540 >> v1:172.21.15.150:6789/0 conn(0x7f844c152700 legacy=0x7f844c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f84525f2640 1 -- 172.21.15.78:0/3198050540 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c155770 con 0x7f844c06d910 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f8450def640 1 -- 172.21.15.78:0/3198050540 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919422970 0 0) 0x7f844c06cd20 con 0x7f844c152700 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.602+0000 7f84515f0640 1 --2- 172.21.15.78:0/3198050540 >> v2:172.21.15.150:3300/0 conn(0x7f844c0c8e00 0x7f844c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f84525f2640 1 --2- 172.21.15.78:0/3198050540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c06d910 0x7f844c06c7e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edd8cf72b2d754b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8450def640 1 -- 172.21.15.78:0/3198050540 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844402cbc0 con 0x7f844c06d910 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8450def640 1 -- 172.21.15.78:0/3198050540 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844402cd20 con 0x7f844c06d910 2024-02-19T14:56:03.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8450def640 1 -- 172.21.15.78:0/3198050540 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444034660 con 0x7f844c06d910 2024-02-19T14:56:03.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 -- 172.21.15.78:0/3198050540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c06d910 msgr2=0x7f844c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 --2- 172.21.15.78:0/3198050540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c06d910 0x7f844c06c7e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844402be10 comp rx=0 tx=0).stop 2024-02-19T14:56:03.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 -- 172.21.15.78:0/3198050540 shutdown_connections 2024-02-19T14:56:03.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 --2- 172.21.15.78:0/3198050540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c06d910 0x7f844c06c7e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 --2- 172.21.15.78:0/3198050540 >> v2:172.21.15.150:3300/0 conn(0x7f844c0c8e00 0x7f844c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 -- 172.21.15.78:0/3198050540 >> 172.21.15.78:0/3198050540 conn(0x7f844c0cac90 msgr2=0x7f844c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 -- 172.21.15.78:0/3198050540 shutdown_connections 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.603+0000 7f8454055640 1 -- 172.21.15.78:0/3198050540 wait complete. 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.604+0000 7f8454055640 1 Processor -- start 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 -- start start 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f844c06d910 0x7f844c0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 0x7f844c0c1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c152700 0x7f844c1ea610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0ab690 con 0x7f844c0c8e00 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c156c90 con 0x7f844c06d910 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8454055640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c149740 con 0x7f844c152700 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f84515f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 0x7f844c0c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f84515f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 0x7f844c0c1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53984/0 (socket says 172.21.15.78:53984) 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f84515f0640 1 -- 172.21.15.78:0/2970963630 learned_addr learned my addr 172.21.15.78:0/2970963630 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f8451df1640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f844c06d910 0x7f844c0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.605+0000 7f84525f2640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c152700 0x7f844c1ea610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84515f0640 1 -- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c152700 msgr2=0x7f844c1ea610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84515f0640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c152700 0x7f844c1ea610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84515f0640 1 -- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f844c06d910 msgr2=0x7f844c0c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84515f0640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f844c06d910 0x7f844c0c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84515f0640 1 -- 172.21.15.78:0/2970963630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c009400 con 0x7f844c0c8e00 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84515f0640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 0x7f844c0c1240 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ebe0 tx=0x7f843c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84427fc640 1 -- 172.21.15.78:0/2970963630 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c009760 con 0x7f844c0c8e00 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84427fc640 1 -- 172.21.15.78:0/2970963630 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c00cbd0 con 0x7f844c0c8e00 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f84427fc640 1 -- 172.21.15.78:0/2970963630 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c00eee0 con 0x7f844c0c8e00 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8444009a70 con 0x7f844c0c8e00 2024-02-19T14:56:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.606+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c0aad60 con 0x7f844c0c8e00 2024-02-19T14:56:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.607+0000 7f84427fc640 1 -- 172.21.15.78:0/2970963630 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f843c013470 con 0x7f844c0c8e00 2024-02-19T14:56:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.608+0000 7f84427fc640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f843006b840 0x7f843006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.608+0000 7f84427fc640 1 -- 172.21.15.78:0/2970963630 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f843c08a4d0 con 0x7f844c0c8e00 2024-02-19T14:56:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.608+0000 7f8451df1640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f843006b840 0x7f843006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.608+0000 7f8451df1640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f843006b840 0x7f843006dd00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8448007fd0 tx=0x7f8448007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 Processor -- start 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 -- start start 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1618ae0 0x55fde1652b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde1617860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde15d2cd0 con 0x55fde1652740 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde1585f20 con 0x55fde16558b0 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.609+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fde1583450 con 0x55fde1618ae0 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53990/0 (socket says 172.21.15.78:53990) 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 -- 172.21.15.78:0/273628188 learned_addr learned my addr 172.21.15.78:0/273628188 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:03.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f84515f0640 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1618ae0 0x55fde1652b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f84525f2640 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde1617860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 -- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1618ae0 msgr2=0x55fde1652b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1618ae0 0x55fde1652b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 -- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 msgr2=0x55fde1617860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde1617860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 -- 172.21.15.78:0/273628188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fde1533f20 con 0x55fde1652740 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f8451df1640 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16538b0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8448004fd0 tx=0x7f8448013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de931f4d1f8f29b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f842f7fe640 1 -- 172.21.15.78:0/273628188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448001cf0 con 0x55fde1652740 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f842f7fe640 1 -- 172.21.15.78:0/273628188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8448013b80 con 0x55fde1652740 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.610+0000 7f842f7fe640 1 -- 172.21.15.78:0/273628188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448012350 con 0x55fde1652740 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 -- 172.21.15.78:0/273628188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 msgr2=0x55fde16538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16538b0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8448004fd0 tx=0x7f8448013000 comp rx=0 tx=0).stop 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 -- 172.21.15.78:0/273628188 shutdown_connections 2024-02-19T14:56:03.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde1617860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1618ae0 0x55fde1652b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 --2- 172.21.15.78:0/273628188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16538b0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 -- 172.21.15.78:0/273628188 >> 172.21.15.78:0/273628188 conn(0x55fde1614120 msgr2=0x55fde1638b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 -- 172.21.15.78:0/273628188 shutdown_connections 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 -- 172.21.15.78:0/273628188 wait complete. 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 -- start start 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.611+0000 7f8456c33740 1 Processor -- start 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde1652740 0x55fde1638210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1650e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde162a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde157ec30 con 0x55fde16558b0 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde1523d30 con 0x55fde1652740 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fde1252310 con 0x55fde16359c0 2024-02-19T14:56:03.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8456c33740 10 client.0 _ll_register_callbacks cb 0x55fde152bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f84515f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1650e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f84515f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1650e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54002/0 (socket says 172.21.15.78:54002) 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f84515f0640 1 -- 172.21.15.78:0/714068253 learned_addr learned my addr 172.21.15.78:0/714068253 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f8451df1640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde1652740 0x55fde1638210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.612+0000 7f84525f2640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde162a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f84515f0640 1 -- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 msgr2=0x55fde162a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f84515f0640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde162a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f84515f0640 1 -- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde1652740 msgr2=0x55fde1638210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f84515f0640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde1652740 0x55fde1638210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f84515f0640 1 -- 172.21.15.78:0/714068253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8448004cb0 con 0x55fde16558b0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f84515f0640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1650e00 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f843c055010 tx=0x7f843c058cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c504c66a67b764 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f842cff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f842cff9640 1 -- 172.21.15.78:0/714068253 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c056f70 con 0x55fde16558b0 2024-02-19T14:56:03.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f842cff9640 1 -- 172.21.15.78:0/714068253 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f843c083ce0 con 0x55fde16558b0 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fde127e6c0 con 0x55fde16558b0 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f842cff9640 1 -- 172.21.15.78:0/714068253 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f843c083ea0 con 0x55fde16558b0 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 10 client.7614 fetch_fsmap learned FSMap version 94 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55fde157d2b0 con 0x55fde16558b0 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f842cff9640 1 -- 172.21.15.78:0/714068253 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c074070 con 0x55fde16558b0 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f842cff9640 1 -- 172.21.15.78:0/714068253 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 94) v1 ==== 3282+0+0 (secure 0 0 0) 0x7f843c011070 con 0x55fde16558b0 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 10 client.7614 fetch_fsmap finished waiting for FSMap version 94 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 10 client.7614 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 20 client.7614 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde164aa70 0x55fde164ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 4 client.7614 mds_command: new command op to 5927 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 --> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fde12cf270 con 0x55fde164aa70 2024-02-19T14:56:03.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.613+0000 7f8451df1640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde164aa70 0x55fde164ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.614+0000 7f842cff9640 10 client.7614 ms_handle_connect on v2:172.21.15.150:6832/2663291681 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.614+0000 7f8451df1640 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde164aa70 0x55fde164ae50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e47d39f820f31e9 server_cookie=748aa8daa47e5e72 in_seq=0 out_seq=0 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f842cff9640 1 -- 172.21.15.78:0/714068253 <== mds.0 v2:172.21.15.150:6832/2663291681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fde12cf270 con 0x55fde164aa70 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f842cff9640 10 client.7614 handle_command_reply: tid=0 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 client.7614 shutdown 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 msgr2=0x55fde1650e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1650e00 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f843c055010 tx=0x7f843c058cf0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 shutdown_connections 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde164aa70 0x55fde164ae50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1650e00 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde162a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f842cff9640 0 client.7614 ms_handle_reset on v2:172.21.15.150:6832/2663291681 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 --2- 172.21.15.78:0/714068253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde1652740 0x55fde1638210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 >> 172.21.15.78:0/714068253 conn(0x55fde1618ae0 msgr2=0x55fde1653e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 shutdown_connections 2024-02-19T14:56:03.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 1 -- 172.21.15.78:0/714068253 wait complete. 2024-02-19T14:56:03.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.615+0000 7f8456c33740 20 client.7614 trim_cache size 0 max 16384 2024-02-19T14:56:03.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 Processor -- start 2024-02-19T14:56:03.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 -- start start 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16359c0 0x55fde164ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1652740 0x55fde1612f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1613490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde1504490 con 0x55fde16558b0 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde1653420 con 0x55fde16359c0 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fde164ff90 con 0x55fde1652740 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f84525f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1613490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f84525f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1613490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54010/0 (socket says 172.21.15.78:54010) 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f84525f2640 1 -- 172.21.15.78:0/1190651517 learned_addr learned my addr 172.21.15.78:0/1190651517 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f8451df1640 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16359c0 0x55fde164ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.632+0000 7f84515f0640 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1652740 0x55fde1612f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f84525f2640 1 -- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1652740 msgr2=0x55fde1612f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f84525f2640 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1652740 0x55fde1612f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f84525f2640 1 -- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16359c0 msgr2=0x55fde164ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f84525f2640 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16359c0 0x55fde164ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f84525f2640 1 -- 172.21.15.78:0/1190651517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fde1649a90 con 0x55fde16558b0 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f84525f2640 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1613490 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8444009eb0 tx=0x7f844402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e7a80455f3a58e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f842effd640 1 -- 172.21.15.78:0/1190651517 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444045070 con 0x55fde16558b0 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f842effd640 1 -- 172.21.15.78:0/1190651517 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844402ce90 con 0x55fde16558b0 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f842effd640 1 -- 172.21.15.78:0/1190651517 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444003cd0 con 0x55fde16558b0 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f8456c33740 1 -- 172.21.15.78:0/1190651517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 msgr2=0x55fde1613490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.633+0000 7f8456c33740 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1613490 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f8444009eb0 tx=0x7f844402e000 comp rx=0 tx=0).stop 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 -- 172.21.15.78:0/1190651517 shutdown_connections 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde16558b0 0x55fde1613490 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde1652740 0x55fde1612f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 --2- 172.21.15.78:0/1190651517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16359c0 0x55fde164ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 -- 172.21.15.78:0/1190651517 >> 172.21.15.78:0/1190651517 conn(0x55fde1618ae0 msgr2=0x55fde164ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 -- 172.21.15.78:0/1190651517 shutdown_connections 2024-02-19T14:56:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 -- 172.21.15.78:0/1190651517 wait complete. 2024-02-19T14:56:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 -- start start 2024-02-19T14:56:03.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.634+0000 7f8456c33740 1 Processor -- start 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde16279d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16575e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde166a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde1638860 con 0x55fde1652740 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fde16c7c60 con 0x55fde16558b0 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fde1627f70 con 0x55fde16359c0 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8456c33740 10 client.0 _ll_register_callbacks cb 0x7f843c058510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54022/0 (socket says 172.21.15.78:54022) 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 -- 172.21.15.78:0/3893670339 learned_addr learned my addr 172.21.15.78:0/3893670339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f8451df1640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde16279d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84525f2640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde166a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 -- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 msgr2=0x55fde16279d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde16279d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 -- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 msgr2=0x55fde166a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde166a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.635+0000 7f84515f0640 1 -- 172.21.15.78:0/3893670339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c155770 con 0x55fde1652740 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f84515f0640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16575e0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f843c054f10 tx=0x7f843c088780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abd2500c49dfb6f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f842cff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f842cff9640 1 -- 172.21.15.78:0/3893670339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c056f70 con 0x55fde1652740 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f842cff9640 1 -- 172.21.15.78:0/3893670339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f843c058000 con 0x55fde1652740 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8448013b80 con 0x55fde1652740 2024-02-19T14:56:03.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f842cff9640 1 -- 172.21.15.78:0/3893670339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f843c0581c0 con 0x55fde1652740 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 10 client.7620 fetch_fsmap learned FSMap version 94 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55fde1519a20 con 0x55fde1652740 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f842cff9640 1 -- 172.21.15.78:0/3893670339 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8448013b80 con 0x55fde1652740 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f842cff9640 1 -- 172.21.15.78:0/3893670339 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 94) v1 ==== 3282+0+0 (secure 0 0 0) 0x7f843c011040 con 0x55fde1652740 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 10 client.7620 fetch_fsmap finished waiting for FSMap version 94 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 10 client.7620 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 20 client.7620 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde16c13c0 0x55fde16c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 4 client.7620 mds_command: new command op to 5927 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 --> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fde12bb610 con 0x55fde16c13c0 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.636+0000 7f8451df1640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde16c13c0 0x55fde16c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f842cff9640 10 client.7620 ms_handle_connect on v2:172.21.15.150:6832/2663291681 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8451df1640 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde16c13c0 0x55fde16c17a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bffeeb1b7201da94 server_cookie=e107d991909e1185 in_seq=0 out_seq=0 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f842cff9640 1 -- 172.21.15.78:0/3893670339 <== mds.0 v2:172.21.15.150:6832/2663291681 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55fde12bb610 con 0x55fde16c13c0 2024-02-19T14:56:03.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f842cff9640 10 client.7620 handle_command_reply: tid=0 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 client.7620 shutdown 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 msgr2=0x55fde16575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16575e0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f843c054f10 tx=0x7f843c088780 comp rx=0 tx=0).stop 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 shutdown_connections 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:6832/2663291681,v1:172.21.15.150:6833/2663291681] conn(0x55fde16c13c0 0x55fde16c17a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fde1652740 0x55fde16575e0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f842cff9640 0 client.7620 ms_handle_reset on v2:172.21.15.150:6832/2663291681 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fde16558b0 0x55fde166a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 --2- 172.21.15.78:0/3893670339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55fde16359c0 0x55fde16279d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 >> 172.21.15.78:0/3893670339 conn(0x55fde1618ae0 msgr2=0x55fde164ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 shutdown_connections 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 1 -- 172.21.15.78:0/3893670339 wait complete. 2024-02-19T14:56:03.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.637+0000 7f8456c33740 20 client.7620 trim_cache size 0 max 16384 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5927, 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 13.451395916999999, 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 93, 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 112, 2024-02-19T14:56:03.647 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 112, 2024-02-19T14:56:03.648 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 29.788564329 2024-02-19T14:56:03.648 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:03.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f843006b840 msgr2=0x7f843006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f843006b840 0x7f843006dd00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8448007fd0 tx=0x7f8448007de0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 msgr2=0x7f844c0c1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 0x7f844c0c1240 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ebe0 tx=0x7f843c00c520 comp rx=0 tx=0).stop 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 shutdown_connections 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f843006b840 0x7f843006dd00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f844c0c8e00 0x7f844c0c1240 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f844c152700 0x7f844c1ea610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 --2- 172.21.15.78:0/2970963630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f844c06d910 0x7f844c0c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 >> 172.21.15.78:0/2970963630 conn(0x7f844c0cac90 msgr2=0x7f844c14e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 shutdown_connections 2024-02-19T14:56:03.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.638+0000 7f8454055640 1 -- 172.21.15.78:0/2970963630 wait complete. 2024-02-19T14:56:03.651 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:55:38.311785+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 93, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5927": { "addr": "172.21.15.150:6833/2663291681", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2663291681, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2663291681, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5927, "incarnation": 86, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_5930": { "addr": "172.21.15.150:6835/2437926852", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2437926852, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2437926852, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5930, "incarnation": 0, "join_fscid": 8, "name": "d", "rank": 0, "state": "up:standby-replay", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-19T14:55:58.223166+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5927 } } } ], "standbys": [ { "addr": "172.21.15.78:6836/3062592461", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 3062592461, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 3062592461, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 81, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7302, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3069973851", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3069973851, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3069973851, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 94, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7599, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:56:03.652 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:standby-replay'}] 2024-02-19T14:56:03.652 INFO:tasks.cephfs.test_failover:current = [{'gid': 7302, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/3062592461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3062592461}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3062592461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 81}, {'gid': 5927, 'name': 'b', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6833/2663291681', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2663291681}, {'type': 'v1', 'addr': '172.21.15.150:6833', 'nonce': 2663291681}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7599, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3069973851', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3069973851}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3069973851}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 94}, {'gid': 5930, 'name': 'd', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.150:6835/2437926852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2437926852}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2437926852}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:56:03.652 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:56:03.652 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:56:03.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:03.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:03.737 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:56:03.737 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:56:03.737 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:03.752 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:56:03.753 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:56:03.753 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:03.768 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:56:03.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:03.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.889+0000 7fc9d60ed640 1 Processor -- start 2024-02-19T14:56:03.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.889+0000 7fc9d60ed640 1 -- start start 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9d60ed640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9d00c87a0 0x7fc9d01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9d60ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d0155ef0 con 0x7fc9d0152480 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9d60ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d006dec0 con 0x7fc9d00c7070 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9d60ed640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d006d520 con 0x7fc9d00c87a0 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9d00c87a0 0x7fc9d01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cf7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc9d0152480 0x7fc9d006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48792/0 (socket says 172.21.15.78:48792) 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9d00c87a0 0x7fc9d01559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54038/0 (socket says 172.21.15.78:54038) 2024-02-19T14:56:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cf7fe640 1 -- 172.21.15.78:0/1868828752 learned_addr learned my addr 172.21.15.78:0/1868828752 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:03.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cffff640 1 -- 172.21.15.78:0/1868828752 >> v1:172.21.15.150:6789/0 conn(0x7fc9d00c7070 legacy=0x7fc9d00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cffff640 1 -- 172.21.15.78:0/1868828752 >> v1:172.21.15.78:6789/0 conn(0x7fc9d0152480 legacy=0x7fc9d006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9cffff640 1 -- 172.21.15.78:0/1868828752 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d00c75c0 con 0x7fc9d00c87a0 2024-02-19T14:56:03.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.890+0000 7fc9ce7fc640 1 -- 172.21.15.78:0/1868828752 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833470349 0 0) 0x7fc9c0001110 con 0x7fc9d0152480 2024-02-19T14:56:03.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.891+0000 7fc9cffff640 1 --2- 172.21.15.78:0/1868828752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d00c87a0 0x7fc9d01559b0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff29480ae1d04c0c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.891+0000 7fc9ce7fc640 1 -- 172.21.15.78:0/1868828752 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c802cd00 con 0x7fc9d00c87a0 2024-02-19T14:56:03.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.891+0000 7fc9ce7fc640 1 -- 172.21.15.78:0/1868828752 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c802ce60 con 0x7fc9d00c87a0 2024-02-19T14:56:03.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.891+0000 7fc9ce7fc640 1 -- 172.21.15.78:0/1868828752 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8034660 con 0x7fc9d00c87a0 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.891+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1868828752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d00c87a0 msgr2=0x7fc9d01559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.891+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1868828752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d00c87a0 0x7fc9d01559b0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c802bf50 comp rx=0 tx=0).stop 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.892+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1868828752 shutdown_connections 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.892+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1868828752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d00c87a0 0x7fc9d01559b0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.892+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1868828752 >> 172.21.15.78:0/1868828752 conn(0x7fc9d00ca8a0 msgr2=0x7fc9d00cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.892+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1868828752 shutdown_connections 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.892+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1868828752 wait complete. 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 Processor -- start 2024-02-19T14:56:03.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 -- start start 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d00c7070 0x7fc9d00aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9d00c87a0 0x7fc9d014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 0x7fc9d00bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00be7a0 con 0x7fc9d0152480 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00bfa60 con 0x7fc9d00c7070 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.893+0000 7fc9d60ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00c16f0 con 0x7fc9d00c87a0 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 0x7fc9d00bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 0x7fc9d00bfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54050/0 (socket says 172.21.15.78:54050) 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 -- 172.21.15.78:0/1607756409 learned_addr learned my addr 172.21.15.78:0/1607756409 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:03.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cf7fe640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d00c7070 0x7fc9d00aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9ceffd640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9d00c87a0 0x7fc9d014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 -- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9d00c87a0 msgr2=0x7fc9d014d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9d00c87a0 0x7fc9d014d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 -- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d00c7070 msgr2=0x7fc9d00aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d00c7070 0x7fc9d00aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 -- 172.21.15.78:0/1607756409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c8009a70 con 0x7fc9d0152480 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9cffff640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 0x7fc9d00bfe10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c803d600 con 0x7fc9d0152480 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c803d780 con 0x7fc9d0152480 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d00acbd0 con 0x7fc9d0152480 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c803db30 con 0x7fc9d0152480 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.894+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d00be500 con 0x7fc9d0152480 2024-02-19T14:56:03.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.895+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d00acd60 con 0x7fc9d0152480 2024-02-19T14:56:03.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.896+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc9c803b040 con 0x7fc9d0152480 2024-02-19T14:56:03.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.896+0000 7fc9b7fff640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9bc06b810 0x7fc9bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:03.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.896+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc9c80ac370 con 0x7fc9d0152480 2024-02-19T14:56:03.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.896+0000 7fc9cf7fe640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9bc06b810 0x7fc9bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:03.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.896+0000 7fc9cf7fe640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9bc06b810 0x7fc9bc06dcd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0000f60 tx=0x7fc9c0007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:03.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:03.900+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc9c8076c70 con 0x7fc9d0152480 2024-02-19T14:56:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.105+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9d0074990 con 0x7fc9d0152480 2024-02-19T14:56:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.106+0000 7fc9b7fff640 1 -- 172.21.15.78:0/1607756409 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4951 (secure 0 0 0) 0x7fc9c807a330 con 0x7fc9d0152480 2024-02-19T14:56:04.108 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:04.109 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7599,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3069973851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3069973851},{"type":"v1","addr":"172.21.15.78:6837","nonce":3069973851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:58.223166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5930":{"gid":5930,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:56:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 94 2024-02-19T14:56:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9bc06b810 msgr2=0x7fc9bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9bc06b810 0x7fc9bc06dcd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0000f60 tx=0x7fc9c0007e40 comp rx=0 tx=0).stop 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 msgr2=0x7fc9d00bfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 0x7fc9d00bfe10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c8004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 shutdown_connections 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9bc06b810 0x7fc9bc06dcd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.107+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d0152480 0x7fc9d00bfe10 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.108+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9d00c87a0 0x7fc9d014d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.108+0000 7fc9d60ed640 1 --2- 172.21.15.78:0/1607756409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d00c7070 0x7fc9d00aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.108+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 >> 172.21.15.78:0/1607756409 conn(0x7fc9d00ca8a0 msgr2=0x7fc9d0154540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.108+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 shutdown_connections 2024-02-19T14:56:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.108+0000 7fc9d60ed640 1 -- 172.21.15.78:0/1607756409 wait complete. 2024-02-19T14:56:04.121 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:04.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.218+0000 7f6d61846640 1 Processor -- start 2024-02-19T14:56:04.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d61846640 1 -- start start 2024-02-19T14:56:04.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d61846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 0x7f6d5c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d61846640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d5c0c8e00 0x7f6d5c15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d61846640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c1551f0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d61846640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5c155490 con 0x7f6d5c0c8e00 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d61846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c155770 con 0x7f6d5c152700 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d5affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 0x7f6d5c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d5b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d5c0c8e00 0x7f6d5c15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d5b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d5c0c8e00 0x7f6d5c15f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:55824/0 (socket says 172.21.15.78:55824) 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d5affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 0x7f6d5c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:57162/0 (socket says 172.21.15.78:57162) 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d5b7fe640 1 -- 172.21.15.78:0/2788658062 learned_addr learned my addr 172.21.15.78:0/2788658062 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.219+0000 7f6d5a7fc640 1 --1- 172.21.15.78:0/2788658062 >> v1:172.21.15.150:6789/0 conn(0x7f6d5c0c5730 0x7f6d5c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51448/0 (socket says 172.21.15.78:51448) 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.220+0000 7f6d5b7fe640 1 -- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 msgr2=0x7f6d5c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.220+0000 7f6d5b7fe640 1 --2- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 0x7f6d5c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.220+0000 7f6d5b7fe640 1 -- 172.21.15.78:0/2788658062 >> v1:172.21.15.150:6789/0 conn(0x7f6d5c0c5730 legacy=0x7f6d5c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.220+0000 7f6d5b7fe640 1 -- 172.21.15.78:0/2788658062 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c155a50 con 0x7f6d5c0c8e00 2024-02-19T14:56:04.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.220+0000 7f6d5affd640 1 --2- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 0x7f6d5c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:04.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d5b7fe640 1 --2- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c0c8e00 0x7f6d5c15f980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009870 tx=0x7f6d4c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1306c6ddf7b96b57 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d59ffb640 1 -- 172.21.15.78:0/2788658062 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c030840 con 0x7f6d5c0c8e00 2024-02-19T14:56:04.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d59ffb640 1 -- 172.21.15.78:0/2788658062 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c0309a0 con 0x7f6d5c0c8e00 2024-02-19T14:56:04.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d59ffb640 1 -- 172.21.15.78:0/2788658062 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c030d30 con 0x7f6d5c0c8e00 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 -- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c0c8e00 msgr2=0x7f6d5c15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 --2- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c0c8e00 0x7f6d5c15f980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009870 tx=0x7f6d4c02b980 comp rx=0 tx=0).stop 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 -- 172.21.15.78:0/2788658062 shutdown_connections 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 --2- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c0c8e00 0x7f6d5c15f980 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 --2- 172.21.15.78:0/2788658062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c152700 0x7f6d5c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 -- 172.21.15.78:0/2788658062 >> 172.21.15.78:0/2788658062 conn(0x7f6d5c0cac90 msgr2=0x7f6d5c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 -- 172.21.15.78:0/2788658062 shutdown_connections 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.221+0000 7f6d61846640 1 -- 172.21.15.78:0/2788658062 wait complete. 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.222+0000 7f6d61846640 1 Processor -- start 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 -- start start 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 0x7f6d5c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c0c8e00 0x7f6d5c0abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c152700 0x7f6d5c0ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0ad490 con 0x7f6d5c0c5730 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0adcf0 con 0x7f6d5c152700 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d61846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0ac870 con 0x7f6d5c0c8e00 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d5affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 0x7f6d5c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d5affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 0x7f6d5c0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54052/0 (socket says 172.21.15.78:54052) 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d5affd640 1 -- 172.21.15.78:0/204455719 learned_addr learned my addr 172.21.15.78:0/204455719 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d5b7fe640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c152700 0x7f6d5c0ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.223+0000 7f6d5a7fc640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c0c8e00 0x7f6d5c0abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d5affd640 1 -- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c0c8e00 msgr2=0x7f6d5c0abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d5affd640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c0c8e00 0x7f6d5c0abdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d5affd640 1 -- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c152700 msgr2=0x7f6d5c0ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d5affd640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c152700 0x7f6d5c0ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d5affd640 1 -- 172.21.15.78:0/204455719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d54009400 con 0x7f6d5c0c5730 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d5affd640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 0x7f6d5c0afc60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400e960 tx=0x7f6d5400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5400ca00 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5400cb60 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c0094d0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54013470 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.224+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c0ad6a0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.225+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c074990 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.225+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6d540135d0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.226+0000 7f6d43fff640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d4806b840 0x7f6d4806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.226+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6d5408a4b0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.226+0000 7f6d5a7fc640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d4806b840 0x7f6d4806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.226+0000 7f6d5a7fc640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d4806b840 0x7f6d4806dd00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c14dd00 tx=0x7f6d500016d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.230+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6d54054e30 con 0x7f6d5c0c5730 2024-02-19T14:56:04.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.407+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d5c0b04c0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.408+0000 7f6d43fff640 1 -- 172.21.15.78:0/204455719 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17069 (secure 0 0 0) 0x7f6d540584f0 con 0x7f6d5c0c5730 2024-02-19T14:56:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:04.411 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":112,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:50.184642+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:36.521408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:37.271805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2978708219":"2024-02-20T14:55:50.184610+0000","172.21.15.78:6835/2978708219":"2024-02-20T14:55:50.184610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:04.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d4806b840 msgr2=0x7f6d4806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d4806b840 0x7f6d4806dd00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c14dd00 tx=0x7f6d500016d0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 msgr2=0x7f6d5c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 0x7f6d5c0afc60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400e960 tx=0x7f6d5400ee20 comp rx=0 tx=0).stop 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 shutdown_connections 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6d4806b840 0x7f6d4806dd00 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d5c152700 0x7f6d5c0ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d5c0c8e00 0x7f6d5c0abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 --2- 172.21.15.78:0/204455719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d5c0c5730 0x7f6d5c0afc60 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.410+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 >> 172.21.15.78:0/204455719 conn(0x7f6d5c0cac90 msgr2=0x7f6d5c0cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.411+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 shutdown_connections 2024-02-19T14:56:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.411+0000 7f6d61846640 1 -- 172.21.15.78:0/204455719 wait complete. 2024-02-19T14:56:04.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:55:36.521408+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:55:37.271805+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:56:04.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:56:04.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.525+0000 7f78ed6df640 1 Processor -- start 2024-02-19T14:56:04.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.525+0000 7f78ed6df640 1 -- start start 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78ed6df640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78e80c51b0 0x7f78e80c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78ed6df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c5590 0x7f78e80c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78ed6df640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78e80c59f0 0x7f78e80ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78ed6df640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e80ca6d0 con 0x7f78e80c51b0 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78ed6df640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e80c9200 con 0x7f78e80c59f0 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78ed6df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80cb720 con 0x7f78e80c5590 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78e80c51b0 0x7f78e80c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78e80c51b0 0x7f78e80c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54066/0 (socket says 172.21.15.78:54066) 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 -- 172.21.15.78:0/463134854 learned_addr learned my addr 172.21.15.78:0/463134854 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e77fe640 1 --2- 172.21.15.78:0/463134854 >> v2:172.21.15.150:3300/0 conn(0x7f78e80c59f0 0x7f78e80ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e67fc640 1 --2- 172.21.15.78:0/463134854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c5590 0x7f78e80c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 -- 172.21.15.78:0/463134854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c5590 msgr2=0x7f78e80c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 --2- 172.21.15.78:0/463134854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c5590 0x7f78e80c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 -- 172.21.15.78:0/463134854 >> v2:172.21.15.150:3300/0 conn(0x7f78e80c59f0 msgr2=0x7f78e80ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 --2- 172.21.15.78:0/463134854 >> v2:172.21.15.150:3300/0 conn(0x7f78e80c59f0 0x7f78e80ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.526+0000 7f78e6ffd640 1 -- 172.21.15.78:0/463134854 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e8158ff0 con 0x7f78e80c51b0 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.527+0000 7f78e6ffd640 1 --2- 172.21.15.78:0/463134854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c51b0 0x7f78e80c8950 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f78e80c16d0 tx=0x7f78d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78338fcf21b5daf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.527+0000 7f78e5ffb640 1 -- 172.21.15.78:0/463134854 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8031870 con 0x7f78e80c51b0 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.527+0000 7f78e5ffb640 1 -- 172.21.15.78:0/463134854 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d80319d0 con 0x7f78e80c51b0 2024-02-19T14:56:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.527+0000 7f78e5ffb640 1 -- 172.21.15.78:0/463134854 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8031d60 con 0x7f78e80c51b0 2024-02-19T14:56:04.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 -- 172.21.15.78:0/463134854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c51b0 msgr2=0x7f78e80c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 --2- 172.21.15.78:0/463134854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c51b0 0x7f78e80c8950 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f78e80c16d0 tx=0x7f78d802b910 comp rx=0 tx=0).stop 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 -- 172.21.15.78:0/463134854 shutdown_connections 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 --2- 172.21.15.78:0/463134854 >> v2:172.21.15.150:3300/0 conn(0x7f78e80c59f0 0x7f78e80ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 --2- 172.21.15.78:0/463134854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c5590 0x7f78e80c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 --2- 172.21.15.78:0/463134854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c51b0 0x7f78e80c8950 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 -- 172.21.15.78:0/463134854 >> 172.21.15.78:0/463134854 conn(0x7f78e80ca830 msgr2=0x7f78e80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 -- 172.21.15.78:0/463134854 shutdown_connections 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.528+0000 7f78ed6df640 1 -- 172.21.15.78:0/463134854 wait complete. 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.529+0000 7f78ed6df640 1 Processor -- start 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.529+0000 7f78ed6df640 1 -- start start 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c51b0 0x7f78e80c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 0x7f78e80c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78e80c59f0 0x7f78e814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80c31b0 con 0x7f78e80c5590 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80c4160 con 0x7f78e80c59f0 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80c19a0 con 0x7f78e80c51b0 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 0x7f78e80c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 0x7f78e80c2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54074/0 (socket says 172.21.15.78:54074) 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 -- 172.21.15.78:0/140035329 learned_addr learned my addr 172.21.15.78:0/140035329 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e77fe640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78e80c59f0 0x7f78e814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e6ffd640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c51b0 0x7f78e80c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 -- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c51b0 msgr2=0x7f78e80c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c51b0 0x7f78e80c2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 -- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78e80c59f0 msgr2=0x7f78e814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78e80c59f0 0x7f78e814a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 -- 172.21.15.78:0/140035329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80094d0 con 0x7f78e80c5590 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78e67fc640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 0x7f78e80c2c70 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00e9e0 tx=0x7f78dc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc009670 con 0x7f78e80c5590 2024-02-19T14:56:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc00c9d0 con 0x7f78e80c5590 2024-02-19T14:56:04.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e80c1ca0 con 0x7f78e80c5590 2024-02-19T14:56:04.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc00cd80 con 0x7f78e80c5590 2024-02-19T14:56:04.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.530+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e8157da0 con 0x7f78e80c5590 2024-02-19T14:56:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.531+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e8074990 con 0x7f78e80c5590 2024-02-19T14:56:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.532+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f78dc013470 con 0x7f78e80c5590 2024-02-19T14:56:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.532+0000 7f78cf7fe640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78d406b840 0x7f78d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.532+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f78dc08a470 con 0x7f78e80c5590 2024-02-19T14:56:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.532+0000 7f78e6ffd640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78d406b840 0x7f78d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.533+0000 7f78e6ffd640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78d406b840 0x7f78d406dd00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f78d8036000 tx=0x7f78d803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.537+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78dc054df0 con 0x7f78e80c5590 2024-02-19T14:56:04.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.705+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78e814c480 con 0x7f78e80c5590 2024-02-19T14:56:04.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.705+0000 7f78cf7fe640 1 -- 172.21.15.78:0/140035329 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78dc0584b0 con 0x7f78e80c5590 2024-02-19T14:56:04.708 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78d406b840 msgr2=0x7f78d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78d406b840 0x7f78d406dd00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f78d8036000 tx=0x7f78d803c040 comp rx=0 tx=0).stop 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 msgr2=0x7f78e80c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 0x7f78e80c2c70 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00e9e0 tx=0x7f78dc00eea0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 shutdown_connections 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f78d406b840 0x7f78d406dd00 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78e80c5590 0x7f78e80c2c70 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78e80c59f0 0x7f78e814a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 --2- 172.21.15.78:0/140035329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78e80c51b0 0x7f78e80c2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 >> 172.21.15.78:0/140035329 conn(0x7f78e80ca830 msgr2=0x7f78e8151950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:04.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 shutdown_connections 2024-02-19T14:56:04.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.707+0000 7f78ed6df640 1 -- 172.21.15.78:0/140035329 wait complete. 2024-02-19T14:56:04.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:56:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.820+0000 7f0797a07640 1 Processor -- start 2024-02-19T14:56:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.820+0000 7f0797a07640 1 -- start start 2024-02-19T14:56:04.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0797a07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f0790158550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0797a07640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0790072ec0 con 0x7f0790072180 2024-02-19T14:56:04.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0797a07640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079006f270 con 0x7f0790152610 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0797a07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079006fa40 con 0x7f07900730a0 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0795fa4640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0790072180 0x7f079015f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48802/0 (socket says 172.21.15.78:48802) 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0795fa4640 1 -- 172.21.15.78:0/1233263591 learned_addr learned my addr 172.21.15.78:0/1233263591 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0794fa2640 1 --2- 172.21.15.78:0/1233263591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f0790158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810929835 0 0) 0x7f0790072ec0 con 0x7f0790072180 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f077c003520 con 0x7f0790072180 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596878494 0 0) 0x7f079006f270 con 0x7f0790152610 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.821+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790072ec0 con 0x7f0790152610 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.822+0000 7f0794fa2640 1 -- 172.21.15.78:0/1233263591 >> v1:172.21.15.150:6789/0 conn(0x7f0790152610 legacy=0x7f07900c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.822+0000 7f0794fa2640 1 -- 172.21.15.78:0/1233263591 >> v1:172.21.15.78:6789/0 conn(0x7f0790072180 legacy=0x7f079015f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.822+0000 7f0794fa2640 1 -- 172.21.15.78:0/1233263591 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079006f4c0 con 0x7f07900730a0 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.822+0000 7f0794fa2640 1 --2- 172.21.15.78:0/1233263591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f0790158550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07800097d0 tx=0x7f078002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf6f1b2c913f83f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.822+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780031800 con 0x7f07900730a0 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0780031960 con 0x7f07900730a0 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0787fff640 1 -- 172.21.15.78:0/1233263591 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780031cf0 con 0x7f07900730a0 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 -- 172.21.15.78:0/1233263591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 msgr2=0x7f0790158550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 --2- 172.21.15.78:0/1233263591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f0790158550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07800097d0 tx=0x7f078002b8a0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 -- 172.21.15.78:0/1233263591 shutdown_connections 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 --2- 172.21.15.78:0/1233263591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f0790158550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 -- 172.21.15.78:0/1233263591 >> 172.21.15.78:0/1233263591 conn(0x7f07900c7590 msgr2=0x7f07900c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 -- 172.21.15.78:0/1233263591 shutdown_connections 2024-02-19T14:56:04.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.823+0000 7f0797a07640 1 -- 172.21.15.78:0/1233263591 wait complete. 2024-02-19T14:56:04.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.824+0000 7f0797a07640 1 Processor -- start 2024-02-19T14:56:04.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.824+0000 7f0797a07640 1 -- start start 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0797a07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 0x7f07900c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0797a07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f07900bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0797a07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0790152610 0x7f079014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0797a07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900be970 con 0x7f0790072180 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0797a07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07901e2430 con 0x7f0790152610 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0797a07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07901dffa0 con 0x7f07900730a0 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 0x7f07900c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 0x7f07900c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54086/0 (socket says 172.21.15.78:54086) 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 -- 172.21.15.78:0/3784172675 learned_addr learned my addr 172.21.15.78:0/3784172675 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0794fa2640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f07900bc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f0795fa4640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0790152610 0x7f079014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 -- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 msgr2=0x7f07900bc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f07900bc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 -- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0790152610 msgr2=0x7f079014f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0790152610 0x7f079014f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 -- 172.21.15.78:0/3784172675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0780009400 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.825+0000 7f07957a3640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 0x7f07900c63f0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f078c00c7a0 tx=0x7f078c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.826+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c003b70 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.826+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c003cf0 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.826+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07900c0220 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.826+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c00aa30 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.826+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07901df950 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.827+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0790074990 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.827+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f078c00ab90 con 0x7f0790072180 2024-02-19T14:56:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.828+0000 7f0785ffb640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f077406b840 0x7f077406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.828+0000 7f0794fa2640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f077406b840 0x7f077406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.828+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f078c08a130 con 0x7f0790072180 2024-02-19T14:56:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.828+0000 7f0794fa2640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f077406b840 0x7f077406dd00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0780036000 tx=0x7f078003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:04.832+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f078c054a20 con 0x7f0790072180 2024-02-19T14:56:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.004+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0790155070 con 0x7f0790072180 2024-02-19T14:56:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.004+0000 7f0785ffb640 1 -- 172.21.15.78:0/3784172675 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f078c003e60 con 0x7f0790072180 2024-02-19T14:56:05.007 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f077406b840 msgr2=0x7f077406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f077406b840 0x7f077406dd00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0780036000 tx=0x7f078003c040 comp rx=0 tx=0).stop 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 msgr2=0x7f07900c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 0x7f07900c63f0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f078c00c7a0 tx=0x7f078c00cc60 comp rx=0 tx=0).stop 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 shutdown_connections 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f077406b840 0x7f077406dd00 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0790152610 0x7f079014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07900730a0 0x7f07900bc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 --2- 172.21.15.78:0/3784172675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790072180 0x7f07900c63f0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 >> 172.21.15.78:0/3784172675 conn(0x7f07900c7590 msgr2=0x7f07900c3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 shutdown_connections 2024-02-19T14:56:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.006+0000 7f0797a07640 1 -- 172.21.15.78:0/3784172675 wait complete. 2024-02-19T14:56:05.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:56:05.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.119+0000 7f86c4047640 1 Processor -- start 2024-02-19T14:56:05.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.119+0000 7f86c4047640 1 -- start start 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c4047640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86bc152480 0x7f86bc1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c4047640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c4047640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86bc155ef0 con 0x7f86bc0c7070 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c4047640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86bc06dec0 con 0x7f86bc152480 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c4047640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc06d520 con 0x7f86bc0c87a0 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c1de3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86bc152480 0x7f86bc1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c1de3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86bc152480 0x7f86bc1559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54096/0 (socket says 172.21.15.78:54096) 2024-02-19T14:56:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c1de3640 1 -- 172.21.15.78:0/111952054 learned_addr learned my addr 172.21.15.78:0/111952054 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.120+0000 7f86c25e4640 1 --2- 172.21.15.78:0/111952054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c1de3640 1 -- 172.21.15.78:0/111952054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 msgr2=0x7f86bc0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c1de3640 1 --2- 172.21.15.78:0/111952054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c1de3640 1 -- 172.21.15.78:0/111952054 >> v1:172.21.15.150:6789/0 conn(0x7f86bc0c7070 legacy=0x7f86bc06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c1de3640 1 -- 172.21.15.78:0/111952054 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc0c75c0 con 0x7f86bc152480 2024-02-19T14:56:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c1de3640 1 --2- 172.21.15.78:0/111952054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc152480 0x7f86bc1559b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f86b8009870 tx=0x7f86b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd8e3f3715a929 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c0de1640 1 -- 172.21.15.78:0/111952054 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8030840 con 0x7f86bc152480 2024-02-19T14:56:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c0de1640 1 -- 172.21.15.78:0/111952054 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b80309a0 con 0x7f86bc152480 2024-02-19T14:56:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.121+0000 7f86c0de1640 1 -- 172.21.15.78:0/111952054 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8030d30 con 0x7f86bc152480 2024-02-19T14:56:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 -- 172.21.15.78:0/111952054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc152480 msgr2=0x7f86bc1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 --2- 172.21.15.78:0/111952054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc152480 0x7f86bc1559b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f86b8009870 tx=0x7f86b802b980 comp rx=0 tx=0).stop 2024-02-19T14:56:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 -- 172.21.15.78:0/111952054 shutdown_connections 2024-02-19T14:56:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 --2- 172.21.15.78:0/111952054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 --2- 172.21.15.78:0/111952054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc152480 0x7f86bc1559b0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 -- 172.21.15.78:0/111952054 >> 172.21.15.78:0/111952054 conn(0x7f86bc0ca8a0 msgr2=0x7f86bc0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 -- 172.21.15.78:0/111952054 shutdown_connections 2024-02-19T14:56:05.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.122+0000 7f86c4047640 1 -- 172.21.15.78:0/111952054 wait complete. 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.123+0000 7f86c4047640 1 Processor -- start 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.123+0000 7f86c4047640 1 -- start start 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c4047640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 0x7f86bc1e0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c4047640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc1e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c4047640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86bc152480 0x7f86bc149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c4047640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc0aceb0 con 0x7f86bc0c7070 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c4047640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc1e13a0 con 0x7f86bc152480 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c4047640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86bc1df620 con 0x7f86bc0c87a0 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 0x7f86bc1e0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 0x7f86bc1e0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54100/0 (socket says 172.21.15.78:54100) 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 -- 172.21.15.78:0/2790512897 learned_addr learned my addr 172.21.15.78:0/2790512897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c15e2640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc1e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c25e4640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86bc152480 0x7f86bc149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 -- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 msgr2=0x7f86bc1e8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc1e8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 -- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86bc152480 msgr2=0x7f86bc149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86bc152480 0x7f86bc149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 -- 172.21.15.78:0/2790512897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80094d0 con 0x7f86bc0c7070 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.124+0000 7f86c1de3640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 0x7f86bc1e0e60 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f86b8000c00 tx=0x7f86b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.125+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8039a90 con 0x7f86bc0c7070 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.125+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8039bf0 con 0x7f86bc0c7070 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.125+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86bc1eed80 con 0x7f86bc0c7070 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.125+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8043410 con 0x7f86bc0c7070 2024-02-19T14:56:05.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.125+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86bc1defd0 con 0x7f86bc0c7070 2024-02-19T14:56:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.126+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86bc074990 con 0x7f86bc0c7070 2024-02-19T14:56:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.126+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f86b804c3b0 con 0x7f86bc0c7070 2024-02-19T14:56:05.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.126+0000 7f86b27fc640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86a006b810 0x7f86a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.127+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f86b80ac650 con 0x7f86bc0c7070 2024-02-19T14:56:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.127+0000 7f86c15e2640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86a006b810 0x7f86a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.127+0000 7f86c15e2640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86a006b810 0x7f86a006dcd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f86bc1e8ba0 tx=0x7f86ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.131+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f86b80770f0 con 0x7f86bc0c7070 2024-02-19T14:56:05.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.303+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f86bc068220 con 0x7f86bc0c7070 2024-02-19T14:56:05.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.304+0000 7f86b27fc640 1 -- 172.21.15.78:0/2790512897 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86b807a7b0 con 0x7f86bc0c7070 2024-02-19T14:56:05.307 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86a006b810 msgr2=0x7f86a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86a006b810 0x7f86a006dcd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f86bc1e8ba0 tx=0x7f86ac009040 comp rx=0 tx=0).stop 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 msgr2=0x7f86bc1e0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 0x7f86bc1e0e60 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f86b8000c00 tx=0x7f86b8004730 comp rx=0 tx=0).stop 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 shutdown_connections 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86a006b810 0x7f86a006dcd0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86bc152480 0x7f86bc149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86bc0c87a0 0x7f86bc1e8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 --2- 172.21.15.78:0/2790512897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86bc0c7070 0x7f86bc1e0e60 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 >> 172.21.15.78:0/2790512897 conn(0x7f86bc0ca8a0 msgr2=0x7f86bc1531b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 shutdown_connections 2024-02-19T14:56:05.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.306+0000 7f86c4047640 1 -- 172.21.15.78:0/2790512897 wait complete. 2024-02-19T14:56:05.319 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:05.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.415+0000 7f81e7e1c640 1 Processor -- start 2024-02-19T14:56:05.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e7e1c640 1 -- start start 2024-02-19T14:56:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e7e1c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81e00c4980 0x7f81e00c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e7e1c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00ae670 con 0x7f81e00add70 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e7e1c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00716b0 con 0x7f81e00705b0 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e7e1c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e0153780 con 0x7f81e00c4980 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e5bb8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f81e00add70 0x7f81e006eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48810/0 (socket says 172.21.15.78:48810) 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.416+0000 7f81e5bb8640 1 -- 172.21.15.78:0/3949101159 learned_addr learned my addr 172.21.15.78:0/3949101159 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.417+0000 7f81e53b7640 1 --2- 172.21.15.78:0/3949101159 >> v2:172.21.15.150:3300/0 conn(0x7f81e00c4980 0x7f81e00c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.417+0000 7f81e4bb6640 1 -- 172.21.15.78:0/3949101159 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338104567 0 0) 0x7f81e00ae670 con 0x7f81e00add70 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.417+0000 7f81e4bb6640 1 -- 172.21.15.78:0/3949101159 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc003520 con 0x7f81e00add70 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.417+0000 7f81e53b7640 1 -- 172.21.15.78:0/3949101159 >> v1:172.21.15.150:6789/0 conn(0x7f81e00705b0 legacy=0x7f81e00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.417+0000 7f81e53b7640 1 -- 172.21.15.78:0/3949101159 >> v1:172.21.15.78:6789/0 conn(0x7f81e00add70 legacy=0x7f81e006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.417+0000 7f81e53b7640 1 -- 172.21.15.78:0/3949101159 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0153a60 con 0x7f81e00c4980 2024-02-19T14:56:05.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e53b7640 1 --2- 172.21.15.78:0/3949101159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e00c2f60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0097a0 tx=0x7f81dc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a250e2f54f87ad81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e4bb6640 1 -- 172.21.15.78:0/3949101159 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc031830 con 0x7f81e00c4980 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e4bb6640 1 -- 172.21.15.78:0/3949101159 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc031990 con 0x7f81e00c4980 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e4bb6640 1 -- 172.21.15.78:0/3949101159 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc031d20 con 0x7f81e00c4980 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e7e1c640 1 -- 172.21.15.78:0/3949101159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 msgr2=0x7f81e00c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/3949101159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e00c2f60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0097a0 tx=0x7f81dc02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e7e1c640 1 -- 172.21.15.78:0/3949101159 shutdown_connections 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/3949101159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e00c2f60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.418+0000 7f81e7e1c640 1 -- 172.21.15.78:0/3949101159 >> 172.21.15.78:0/3949101159 conn(0x7f81e00c85e0 msgr2=0x7f81e00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.419+0000 7f81e7e1c640 1 -- 172.21.15.78:0/3949101159 shutdown_connections 2024-02-19T14:56:05.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.419+0000 7f81e7e1c640 1 -- 172.21.15.78:0/3949101159 wait complete. 2024-02-19T14:56:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 Processor -- start 2024-02-19T14:56:05.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 -- start start 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81e00705b0 0x7f81e01ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 0x7f81e00c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e01ea7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e01ee390 con 0x7f81e00add70 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e0155250 con 0x7f81e00c4980 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e7e1c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e01edda0 con 0x7f81e00705b0 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e53b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 0x7f81e00c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e53b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 0x7f81e00c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54106/0 (socket says 172.21.15.78:54106) 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e53b7640 1 -- 172.21.15.78:0/2327335035 learned_addr learned my addr 172.21.15.78:0/2327335035 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e63b9640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e01ea7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.420+0000 7f81e5bb8640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81e00705b0 0x7f81e01ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e53b7640 1 -- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81e00705b0 msgr2=0x7f81e01ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e53b7640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81e00705b0 0x7f81e01ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e53b7640 1 -- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 msgr2=0x7f81e01ea7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e53b7640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e01ea7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e53b7640 1 -- 172.21.15.78:0/2327335035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc009400 con 0x7f81e00add70 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e53b7640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 0x7f81e00c1b60 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f81dc034040 tx=0x7f81dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc03a8f0 con 0x7f81e00add70 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc03aa70 con 0x7f81e00add70 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc03ae20 con 0x7f81e00add70 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e01eecf0 con 0x7f81e00add70 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.421+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e0154f70 con 0x7f81e00add70 2024-02-19T14:56:05.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.422+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e01ee4d0 con 0x7f81e00add70 2024-02-19T14:56:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.422+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81dc04d440 con 0x7f81e00add70 2024-02-19T14:56:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.423+0000 7f81d67fc640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81c406b810 0x7f81c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.423+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f81dc0ac1d0 con 0x7f81e00add70 2024-02-19T14:56:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.423+0000 7f81e5bb8640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81c406b810 0x7f81c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.423+0000 7f81e5bb8640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81c406b810 0x7f81c406dcd0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f81d00011e0 tx=0x7f81d0007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.427+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81e01ee4d0 con 0x7f81e00add70 2024-02-19T14:56:05.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.634+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81e01eecf0 con 0x7f81e00add70 2024-02-19T14:56:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.634+0000 7f81d67fc640 1 -- 172.21.15.78:0/2327335035 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4951 (secure 0 0 0) 0x7f81e01eecf0 con 0x7f81e00add70 2024-02-19T14:56:05.637 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:05.637 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7599,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3069973851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3069973851},{"type":"v1","addr":"172.21.15.78:6837","nonce":3069973851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:58.223166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5930":{"gid":5930,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:56:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 94 2024-02-19T14:56:05.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81c406b810 msgr2=0x7f81c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81c406b810 0x7f81c406dcd0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f81d00011e0 tx=0x7f81d0007d30 comp rx=0 tx=0).stop 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 msgr2=0x7f81e00c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 0x7f81e00c1b60 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f81dc034040 tx=0x7f81dc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 shutdown_connections 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81c406b810 0x7f81c406dcd0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81e00add70 0x7f81e00c1b60 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81e00c4980 0x7f81e01ea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 --2- 172.21.15.78:0/2327335035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81e00705b0 0x7f81e01ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 >> 172.21.15.78:0/2327335035 conn(0x7f81e00c85e0 msgr2=0x7f81e00a8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 shutdown_connections 2024-02-19T14:56:05.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.636+0000 7f81e7e1c640 1 -- 172.21.15.78:0/2327335035 wait complete. 2024-02-19T14:56:05.650 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:05.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 Processor -- start 2024-02-19T14:56:05.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 -- start start 2024-02-19T14:56:05.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f812006ae80 0x7f812006b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81200c1550 0x7f81200c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200aa1d0 con 0x7f81200c0950 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8120159310 con 0x7f81200c1550 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.747+0000 7f8126da2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81200a9a60 con 0x7f812006ae80 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81200c1550 0x7f81200c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81200c1550 0x7f81200c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54116/0 (socket says 172.21.15.78:54116) 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 -- 172.21.15.78:0/694855778 learned_addr learned my addr 172.21.15.78:0/694855778 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f8125da0640 1 --2- 172.21.15.78:0/694855778 >> v2:172.21.15.150:3300/0 conn(0x7f812006ae80 0x7f812006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 -- 172.21.15.78:0/694855778 >> v2:172.21.15.150:3300/0 conn(0x7f812006ae80 msgr2=0x7f812006b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 --2- 172.21.15.78:0/694855778 >> v2:172.21.15.150:3300/0 conn(0x7f812006ae80 0x7f812006b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 -- 172.21.15.78:0/694855778 >> v1:172.21.15.150:6789/0 conn(0x7f81200c0950 legacy=0x7f812006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f812559f640 1 -- 172.21.15.78:0/694855778 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120072e00 con 0x7f81200c1550 2024-02-19T14:56:05.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.748+0000 7f8124d9e640 1 -- 172.21.15.78:0/694855778 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003747179 0 0) 0x7f8118001490 con 0x7f81200c0950 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f812559f640 1 --2- 172.21.15.78:0/694855778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c1550 0x7f81200c3570 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f81200709c0 tx=0x7f811c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35c7508c8a1cf2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8124d9e640 1 -- 172.21.15.78:0/694855778 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c031870 con 0x7f81200c1550 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8124d9e640 1 -- 172.21.15.78:0/694855778 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c0319d0 con 0x7f81200c1550 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8124d9e640 1 -- 172.21.15.78:0/694855778 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c031d60 con 0x7f81200c1550 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 -- 172.21.15.78:0/694855778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c1550 msgr2=0x7f81200c3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 --2- 172.21.15.78:0/694855778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c1550 0x7f81200c3570 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f81200709c0 tx=0x7f811c02b910 comp rx=0 tx=0).stop 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 -- 172.21.15.78:0/694855778 shutdown_connections 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 --2- 172.21.15.78:0/694855778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c1550 0x7f81200c3570 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 --2- 172.21.15.78:0/694855778 >> v2:172.21.15.150:3300/0 conn(0x7f812006ae80 0x7f812006b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 -- 172.21.15.78:0/694855778 >> 172.21.15.78:0/694855778 conn(0x7f81200c4020 msgr2=0x7f81200c4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.749+0000 7f8126da2640 1 -- 172.21.15.78:0/694855778 shutdown_connections 2024-02-19T14:56:05.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.750+0000 7f8126da2640 1 -- 172.21.15.78:0/694855778 wait complete. 2024-02-19T14:56:05.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 Processor -- start 2024-02-19T14:56:05.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 -- start start 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f812006ae80 0x7f81200c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81200c0950 0x7f8120148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 0x7f81201487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120151f70 con 0x7f81200c6940 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81200c72f0 con 0x7f812006ae80 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f8126da2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81201edf70 con 0x7f81200c0950 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f81265a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 0x7f81201487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f81265a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 0x7f81201487d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54126/0 (socket says 172.21.15.78:54126) 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.751+0000 7f81265a1640 1 -- 172.21.15.78:0/1081386527 learned_addr learned my addr 172.21.15.78:0/1081386527 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f8125da0640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f812006ae80 0x7f81200c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f812559f640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81200c0950 0x7f8120148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81265a1640 1 -- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81200c0950 msgr2=0x7f8120148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81265a1640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81200c0950 0x7f8120148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81265a1640 1 -- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f812006ae80 msgr2=0x7f81200c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81265a1640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f812006ae80 0x7f81200c6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81265a1640 1 -- 172.21.15.78:0/1081386527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c0094d0 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81265a1640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 0x7f81201487d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f811800c800 tx=0x7f811800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118003da0 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811800ce50 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118005290 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81201ee250 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.752+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81201ee810 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.753+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8120158870 con 0x7f81200c6940 2024-02-19T14:56:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.753+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81180053f0 con 0x7f81200c6940 2024-02-19T14:56:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.754+0000 7f81167fc640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f810406b840 0x7f810406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.754+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f81180898c0 con 0x7f81200c6940 2024-02-19T14:56:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.754+0000 7f8125da0640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f810406b840 0x7f810406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.754+0000 7f8125da0640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f810406b840 0x7f810406dd00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f812014c680 tx=0x7f8110009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:05.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.758+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8118054240 con 0x7f81200c6940 2024-02-19T14:56:05.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.930+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81200aa1d0 con 0x7f81200c6940 2024-02-19T14:56:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.931+0000 7f81167fc640 1 -- 172.21.15.78:0/1081386527 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17069 (secure 0 0 0) 0x7f81200aa1d0 con 0x7f81200c6940 2024-02-19T14:56:05.934 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:05.934 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":112,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:50.184642+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:36.521408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:37.271805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2978708219":"2024-02-20T14:55:50.184610+0000","172.21.15.78:6835/2978708219":"2024-02-20T14:55:50.184610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f810406b840 msgr2=0x7f810406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f810406b840 0x7f810406dd00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f812014c680 tx=0x7f8110009190 comp rx=0 tx=0).stop 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 msgr2=0x7f81201487d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 0x7f81201487d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f811800c800 tx=0x7f811800cc00 comp rx=0 tx=0).stop 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 shutdown_connections 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f810406b840 0x7f810406dd00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81200c6940 0x7f81201487d0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81200c0950 0x7f8120148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 --2- 172.21.15.78:0/1081386527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f812006ae80 0x7f81200c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 >> 172.21.15.78:0/1081386527 conn(0x7f81200c4020 msgr2=0x7f812006ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 shutdown_connections 2024-02-19T14:56:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:05.933+0000 7f8126da2640 1 -- 172.21.15.78:0/1081386527 wait complete. 2024-02-19T14:56:05.947 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:56:05.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:06.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.044+0000 7f95a5f6c640 1 Processor -- start 2024-02-19T14:56:06.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.044+0000 7f95a5f6c640 1 -- start start 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f95a5f6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f95a5f6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95a00c55f0 0x7f95a015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f95a5f6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a0158c40 con 0x7f95a00bdf20 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f95a5f6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00c5a60 con 0x7f95a00c55f0 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f95a5f6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00c8de0 con 0x7f95a00ac390 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959effd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f95a00bdf20 0x7f95a0158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51478/0 (socket says 172.21.15.78:51478) 2024-02-19T14:56:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959effd640 1 -- 172.21.15.78:0/188131917 learned_addr learned my addr 172.21.15.78:0/188131917 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959ffff640 1 --2- 172.21.15.78:0/188131917 >> v2:172.21.15.150:3300/0 conn(0x7f95a00c55f0 0x7f95a015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959f7fe640 1 -- 172.21.15.78:0/188131917 >> v2:172.21.15.150:3300/0 conn(0x7f95a00c55f0 msgr2=0x7f95a015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959f7fe640 1 --2- 172.21.15.78:0/188131917 >> v2:172.21.15.150:3300/0 conn(0x7f95a00c55f0 0x7f95a015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959f7fe640 1 -- 172.21.15.78:0/188131917 >> v1:172.21.15.150:6789/0 conn(0x7f95a00bdf20 legacy=0x7f95a0158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.045+0000 7f959f7fe640 1 -- 172.21.15.78:0/188131917 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a0155a60 con 0x7f95a00ac390 2024-02-19T14:56:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.046+0000 7f959f7fe640 1 --2- 172.21.15.78:0/188131917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00ac770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f95900097a0 tx=0x7f959002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ece4718f602b97a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.046+0000 7f959e7fc640 1 -- 172.21.15.78:0/188131917 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959002f050 con 0x7f95a00ac390 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.046+0000 7f959e7fc640 1 -- 172.21.15.78:0/188131917 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9590031700 con 0x7f95a00ac390 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.046+0000 7f959e7fc640 1 -- 172.21.15.78:0/188131917 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590031a90 con 0x7f95a00ac390 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 -- 172.21.15.78:0/188131917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 msgr2=0x7f95a00ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/188131917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00ac770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f95900097a0 tx=0x7f959002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 -- 172.21.15.78:0/188131917 shutdown_connections 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/188131917 >> v2:172.21.15.150:3300/0 conn(0x7f95a00c55f0 0x7f95a015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/188131917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00ac770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 -- 172.21.15.78:0/188131917 >> 172.21.15.78:0/188131917 conn(0x7f95a00ca890 msgr2=0x7f95a00caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 -- 172.21.15.78:0/188131917 shutdown_connections 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.047+0000 7f95a5f6c640 1 -- 172.21.15.78:0/188131917 wait complete. 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.048+0000 7f95a5f6c640 1 Processor -- start 2024-02-19T14:56:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.048+0000 7f95a5f6c640 1 -- start start 2024-02-19T14:56:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f95a5f6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f95a5f6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a00bdf20 0x7f95a00bcd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f95a5f6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a00c55f0 0x7f95a00bd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f95a5f6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00c0840 con 0x7f95a00bdf20 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f95a5f6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00be330 con 0x7f95a00c55f0 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f95a5f6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a01ea830 con 0x7f95a00ac390 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a00bdf20 0x7f95a00bcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:57256/0 (socket says 172.21.15.78:57256) 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959f7fe640 1 -- 172.21.15.78:0/4159858497 learned_addr learned my addr 172.21.15.78:0/4159858497 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959ffff640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a00c55f0 0x7f95a00bd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959effd640 1 -- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 msgr2=0x7f95a00c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959effd640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959effd640 1 -- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a00c55f0 msgr2=0x7f95a00bd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959effd640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a00c55f0 0x7f95a00bd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.049+0000 7f959effd640 1 -- 172.21.15.78:0/4159858497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590009400 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.050+0000 7f959effd640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a00bdf20 0x7f95a00bcd60 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f9594000f60 tx=0x7f959400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.050+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594005ce0 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.050+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9594005e40 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.050+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a00c1970 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.050+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594007310 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.050+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a00c1f30 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.051+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a0074990 con 0x7f95a00bdf20 2024-02-19T14:56:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.051+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9594007470 con 0x7f95a00bdf20 2024-02-19T14:56:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.051+0000 7f9587fff640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f958c06b810 0x7f958c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.052+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9594089840 con 0x7f95a00bdf20 2024-02-19T14:56:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.052+0000 7f959f7fe640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f958c06b810 0x7f958c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.052+0000 7f959f7fe640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f958c06b810 0x7f958c06dcd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f959002b7e0 tx=0x7f9590008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.056+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9594054140 con 0x7f95a00bdf20 2024-02-19T14:56:06.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.264+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95a006da00 con 0x7f95a00bdf20 2024-02-19T14:56:06.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.264+0000 7f9587fff640 1 -- 172.21.15.78:0/4159858497 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4951 (secure 0 0 0) 0x7f959405ce60 con 0x7f95a00bdf20 2024-02-19T14:56:06.267 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:06.267 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7599,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3069973851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3069973851},{"type":"v1","addr":"172.21.15.78:6837","nonce":3069973851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:58.223166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5930":{"gid":5930,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:56:06.267 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 94 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f958c06b810 msgr2=0x7f958c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f958c06b810 0x7f958c06dcd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f959002b7e0 tx=0x7f9590008030 comp rx=0 tx=0).stop 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a00bdf20 msgr2=0x7f95a00bcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a00bdf20 0x7f95a00bcd60 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f9594000f60 tx=0x7f959400c800 comp rx=0 tx=0).stop 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 shutdown_connections 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f958c06b810 0x7f958c06dcd0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a00bdf20 0x7f95a00bcd60 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a00c55f0 0x7f95a00bd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 --2- 172.21.15.78:0/4159858497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95a00ac390 0x7f95a00c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 >> 172.21.15.78:0/4159858497 conn(0x7f95a00ca890 msgr2=0x7f95a006c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:06.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 shutdown_connections 2024-02-19T14:56:06.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.266+0000 7f95a5f6c640 1 -- 172.21.15.78:0/4159858497 wait complete. 2024-02-19T14:56:06.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:06.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.376+0000 7fc9cbc76640 1 Processor -- start 2024-02-19T14:56:06.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.376+0000 7fc9cbc76640 1 -- start start 2024-02-19T14:56:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9cbc76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c25e0 0x7fc9c41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9cbc76640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9c40c29c0 0x7fc9c40c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9cbc76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9c40c49c0 0x7fc9c406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9cbc76640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c4152c90 con 0x7fc9c40c49c0 2024-02-19T14:56:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9cbc76640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c4072250 con 0x7fc9c40c29c0 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9cbc76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4072480 con 0x7fc9c40c25e0 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9c40c49c0 0x7fc9c406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc9c40c49c0 0x7fc9c406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54140/0 (socket says 172.21.15.78:54140) 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 -- 172.21.15.78:0/100955895 learned_addr learned my addr 172.21.15.78:0/100955895 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9c9a12640 1 --2- 172.21.15.78:0/100955895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c25e0 0x7fc9c41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9c9211640 1 --2- 172.21.15.78:0/100955895 >> v2:172.21.15.150:3300/0 conn(0x7fc9c40c29c0 0x7fc9c40c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 -- 172.21.15.78:0/100955895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c25e0 msgr2=0x7fc9c41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 --2- 172.21.15.78:0/100955895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c25e0 0x7fc9c41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 -- 172.21.15.78:0/100955895 >> v2:172.21.15.150:3300/0 conn(0x7fc9c40c29c0 msgr2=0x7fc9c40c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 --2- 172.21.15.78:0/100955895 >> v2:172.21.15.150:3300/0 conn(0x7fc9c40c29c0 0x7fc9c40c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.377+0000 7fc9ca213640 1 -- 172.21.15.78:0/100955895 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40c3eb0 con 0x7fc9c40c49c0 2024-02-19T14:56:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.378+0000 7fc9ca213640 1 --2- 172.21.15.78:0/100955895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40c49c0 0x7fc9c406e420 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4072650 tx=0x7fc9bc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b711ca5b05b45ba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.378+0000 7fc9c8a10640 1 -- 172.21.15.78:0/100955895 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc02c9c0 con 0x7fc9c40c49c0 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.378+0000 7fc9c8a10640 1 -- 172.21.15.78:0/100955895 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc02cb20 con 0x7fc9c40c49c0 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.378+0000 7fc9c8a10640 1 -- 172.21.15.78:0/100955895 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0343f0 con 0x7fc9c40c49c0 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 -- 172.21.15.78:0/100955895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40c49c0 msgr2=0x7fc9c406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/100955895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40c49c0 0x7fc9c406e420 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4072650 tx=0x7fc9bc02be20 comp rx=0 tx=0).stop 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 -- 172.21.15.78:0/100955895 shutdown_connections 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/100955895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40c49c0 0x7fc9c406e420 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/100955895 >> v2:172.21.15.150:3300/0 conn(0x7fc9c40c29c0 0x7fc9c40c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/100955895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c25e0 0x7fc9c41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 -- 172.21.15.78:0/100955895 >> 172.21.15.78:0/100955895 conn(0x7fc9c406b270 msgr2=0x7fc9c406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 -- 172.21.15.78:0/100955895 shutdown_connections 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.379+0000 7fc9cbc76640 1 -- 172.21.15.78:0/100955895 wait complete. 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.380+0000 7fc9cbc76640 1 Processor -- start 2024-02-19T14:56:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.380+0000 7fc9cbc76640 1 -- start start 2024-02-19T14:56:06.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c40c25e0 0x7fc9c415cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c29c0 0x7fc9c4146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 0x7fc9c4152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40bb950 con 0x7fc9c40bfd80 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c41eb960 con 0x7fc9c40c25e0 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c415d5c0 con 0x7fc9c40c29c0 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 0x7fc9c4152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 0x7fc9c4152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54154/0 (socket says 172.21.15.78:54154) 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 -- 172.21.15.78:0/3749707792 learned_addr learned my addr 172.21.15.78:0/3749707792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9c9211640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c29c0 0x7fc9c4146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9c9a12640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c40c25e0 0x7fc9c415cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 -- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c29c0 msgr2=0x7fc9c4146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c29c0 0x7fc9c4146910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 -- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c40c25e0 msgr2=0x7fc9c415cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c40c25e0 0x7fc9c415cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 -- 172.21.15.78:0/3749707792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc009b30 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ca213640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 0x7fc9c4152dd0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009e50 tx=0x7fc9bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc009510 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc03e450 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc03e800 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.381+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c415d8a0 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.382+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c415de60 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.382+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c41ebb20 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.383+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc9bc03e960 con 0x7fc9c40bfd80 2024-02-19T14:56:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.383+0000 7fc9ba7fc640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9a806b840 0x7fc9a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.383+0000 7fc9c9a12640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9a806b840 0x7fc9a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.383+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc9bc0acaf0 con 0x7fc9c40bfd80 2024-02-19T14:56:06.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.384+0000 7fc9c9a12640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9a806b840 0x7fc9a806dd00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc9c414f210 tx=0x7fc9c0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.388+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc9bc077470 con 0x7fc9c40bfd80 2024-02-19T14:56:06.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.596+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9c4074990 con 0x7fc9c40bfd80 2024-02-19T14:56:06.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.597+0000 7fc9ba7fc640 1 -- 172.21.15.78:0/3749707792 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4951 (secure 0 0 0) 0x7fc9bc07ab30 con 0x7fc9c40bfd80 2024-02-19T14:56:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:06.600 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7599,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3069973851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3069973851},{"type":"v1","addr":"172.21.15.78:6837","nonce":3069973851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T14:55:38.311785+0000","modified":"2024-02-19T14:55:58.223166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5927},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5927":{"gid":5927,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/2663291681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2663291681},{"type":"v1","addr":"172.21.15.150:6833","nonce":2663291681}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5930":{"gid":5930,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.150:6835/2437926852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2437926852},{"type":"v1","addr":"172.21.15.150:6835","nonce":2437926852}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-19T14:56:06.600 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 94 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9a806b840 msgr2=0x7fc9a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9a806b840 0x7fc9a806dd00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc9c414f210 tx=0x7fc9c0009190 comp rx=0 tx=0).stop 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 msgr2=0x7fc9c4152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 0x7fc9c4152dd0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009e50 tx=0x7fc9bc004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 shutdown_connections 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9a806b840 0x7fc9a806dd00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9c40bfd80 0x7fc9c4152dd0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9c40c29c0 0x7fc9c4146910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 --2- 172.21.15.78:0/3749707792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9c40c25e0 0x7fc9c415cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 >> 172.21.15.78:0/3749707792 conn(0x7fc9c406b270 msgr2=0x7fc9c40c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:06.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 shutdown_connections 2024-02-19T14:56:06.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.599+0000 7fc9cbc76640 1 -- 172.21.15.78:0/3749707792 wait complete. 2024-02-19T14:56:06.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.709+0000 7f849b0c7640 1 Processor -- start 2024-02-19T14:56:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.709+0000 7f849b0c7640 1 -- start start 2024-02-19T14:56:06.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.709+0000 7f849b0c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8494155e00 0x7f84940c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.709+0000 7f849b0c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f849415f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f849b0c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84940c7df0 con 0x7f84940c67e0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f849b0c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84940c4310 con 0x7f8494155e00 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f849b0c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940c77c0 con 0x7f84940c39e0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8493fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f84940c67e0 0x7f849406c390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48824/0 (socket says 172.21.15.78:48824) 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8493fff640 1 -- 172.21.15.78:0/3584737298 learned_addr learned my addr 172.21.15.78:0/3584737298 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8499664640 1 --2- 172.21.15.78:0/3584737298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f849415f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8498e63640 1 --2- 172.21.15.78:0/3584737298 >> v2:172.21.15.150:3300/0 conn(0x7f8494155e00 0x7f84940c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f84937fe640 1 -- 172.21.15.78:0/3584737298 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898377368 0 0) 0x7f84940c7df0 con 0x7f84940c67e0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f84937fe640 1 -- 172.21.15.78:0/3584737298 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484003520 con 0x7f84940c67e0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8499664640 1 -- 172.21.15.78:0/3584737298 >> v2:172.21.15.150:3300/0 conn(0x7f8494155e00 msgr2=0x7f84940c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8499664640 1 --2- 172.21.15.78:0/3584737298 >> v2:172.21.15.150:3300/0 conn(0x7f8494155e00 0x7f84940c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8499664640 1 -- 172.21.15.78:0/3584737298 >> v1:172.21.15.78:6789/0 conn(0x7f84940c67e0 legacy=0x7f849406c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.710+0000 7f8499664640 1 -- 172.21.15.78:0/3584737298 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940c70d0 con 0x7f84940c39e0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.711+0000 7f8499664640 1 --2- 172.21.15.78:0/3584737298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f849415f9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f848c009d90 tx=0x7f848c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e9cb728cb0c0c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.711+0000 7f84937fe640 1 -- 172.21.15.78:0/3584737298 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c02cbc0 con 0x7f84940c39e0 2024-02-19T14:56:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.711+0000 7f84937fe640 1 -- 172.21.15.78:0/3584737298 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c02cd20 con 0x7f84940c39e0 2024-02-19T14:56:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.711+0000 7f84937fe640 1 -- 172.21.15.78:0/3584737298 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c034640 con 0x7f84940c39e0 2024-02-19T14:56:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.711+0000 7f849b0c7640 1 -- 172.21.15.78:0/3584737298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 msgr2=0x7f849415f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.711+0000 7f849b0c7640 1 --2- 172.21.15.78:0/3584737298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f849415f9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f848c009d90 tx=0x7f848c02be10 comp rx=0 tx=0).stop 2024-02-19T14:56:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.712+0000 7f849b0c7640 1 -- 172.21.15.78:0/3584737298 shutdown_connections 2024-02-19T14:56:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.712+0000 7f849b0c7640 1 --2- 172.21.15.78:0/3584737298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f849415f9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.712+0000 7f849b0c7640 1 --2- 172.21.15.78:0/3584737298 >> v2:172.21.15.150:3300/0 conn(0x7f8494155e00 0x7f84940c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.712+0000 7f849b0c7640 1 -- 172.21.15.78:0/3584737298 >> 172.21.15.78:0/3584737298 conn(0x7f849406b700 msgr2=0x7f84940aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.712+0000 7f849b0c7640 1 -- 172.21.15.78:0/3584737298 shutdown_connections 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.712+0000 7f849b0c7640 1 -- 172.21.15.78:0/3584737298 wait complete. 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 Processor -- start 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 -- start start 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f8494099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 0x7f84941edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494155e00 0x7f8494150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849409a710 con 0x7f84940c67e0 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494158fd0 con 0x7f8494155e00 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.713+0000 7f849b0c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8494099fa0 con 0x7f84940c39e0 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 0x7f84941edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 0x7f84941edd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54160/0 (socket says 172.21.15.78:54160) 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 -- 172.21.15.78:0/4284230447 learned_addr learned my addr 172.21.15.78:0/4284230447 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8498e63640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f8494099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8499664640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494155e00 0x7f8494150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 -- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 msgr2=0x7f8494099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f8494099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 -- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494155e00 msgr2=0x7f8494150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494155e00 0x7f8494150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 -- 172.21.15.78:0/4284230447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c009a70 con 0x7f84940c67e0 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f8493fff640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 0x7f84941edd40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f848000c430 tx=0x7f848000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8480005ce0 con 0x7f84940c67e0 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8480005e40 con 0x7f84940c67e0 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84941ee610 con 0x7f84940c67e0 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84800072f0 con 0x7f84940c67e0 2024-02-19T14:56:06.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.714+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8494159220 con 0x7f84940c67e0 2024-02-19T14:56:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.715+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84940c6cb0 con 0x7f84940c67e0 2024-02-19T14:56:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.716+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8480007450 con 0x7f84940c67e0 2024-02-19T14:56:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.716+0000 7f84917fa640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f847806b840 0x7f847806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.716+0000 7f8498e63640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f847806b840 0x7f847806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.716+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f84800898c0 con 0x7f84940c67e0 2024-02-19T14:56:06.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.717+0000 7f8498e63640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f847806b840 0x7f847806dd00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8488009b40 tx=0x7f8488009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:06.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.721+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8480054240 con 0x7f84940c67e0 2024-02-19T14:56:06.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.894+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8494074990 con 0x7f84940c67e0 2024-02-19T14:56:06.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.895+0000 7f84917fa640 1 -- 172.21.15.78:0/4284230447 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17069 (secure 0 0 0) 0x7f8480017090 con 0x7f84940c67e0 2024-02-19T14:56:06.898 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:06.898 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":112,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:55:50.184642+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:55:36.521408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-19T14:55:37.271805+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2978708219":"2024-02-20T14:55:50.184610+0000","172.21.15.78:6835/2978708219":"2024-02-20T14:55:50.184610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f847806b840 msgr2=0x7f847806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f847806b840 0x7f847806dd00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8488009b40 tx=0x7f8488009190 comp rx=0 tx=0).stop 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 msgr2=0x7f84941edd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 0x7f84941edd40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f848000c430 tx=0x7f848000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 shutdown_connections 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f847806b840 0x7f847806dd00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84940c67e0 0x7f84941edd40 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494155e00 0x7f8494150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 --2- 172.21.15.78:0/4284230447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c39e0 0x7f8494099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:06.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 >> 172.21.15.78:0/4284230447 conn(0x7f849406b700 msgr2=0x7f84940ad400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:06.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 shutdown_connections 2024-02-19T14:56:06.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:06.897+0000 7f849b0c7640 1 -- 172.21.15.78:0/4284230447 wait complete. 2024-02-19T14:56:06.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:56:07.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.009+0000 7fd605d83640 1 Processor -- start 2024-02-19T14:56:07.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.009+0000 7fd605d83640 1 -- start start 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd605d83640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd6000c25e0 0x7fd6001526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd605d83640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6000c29c0 0x7fd6000c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd605d83640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd600072250 con 0x7fd6000c49c0 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd605d83640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd600152c90 con 0x7fd6000c25e0 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd605d83640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd600072480 con 0x7fd6000c29c0 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5ff7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd6000c25e0 0x7fd6001526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5fffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd6000c49c0 0x7fd60006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48826/0 (socket says 172.21.15.78:48826) 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5fffff640 1 -- 172.21.15.78:0/3408296076 learned_addr learned my addr 172.21.15.78:0/3408296076 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5feffd640 1 --2- 172.21.15.78:0/3408296076 >> v2:172.21.15.150:3300/0 conn(0x7fd6000c29c0 0x7fd6000c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5ff7fe640 1 -- 172.21.15.78:0/3408296076 >> v2:172.21.15.150:3300/0 conn(0x7fd6000c29c0 msgr2=0x7fd6000c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5ff7fe640 1 --2- 172.21.15.78:0/3408296076 >> v2:172.21.15.150:3300/0 conn(0x7fd6000c29c0 0x7fd6000c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5ff7fe640 1 -- 172.21.15.78:0/3408296076 >> v1:172.21.15.78:6789/0 conn(0x7fd6000c49c0 legacy=0x7fd60006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5ff7fe640 1 -- 172.21.15.78:0/3408296076 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6000c3eb0 con 0x7fd6000c25e0 2024-02-19T14:56:07.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.010+0000 7fd5fe7fc640 1 -- 172.21.15.78:0/3408296076 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967672754 0 0) 0x7fd600072250 con 0x7fd6000c49c0 2024-02-19T14:56:07.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.011+0000 7fd5ff7fe640 1 --2- 172.21.15.78:0/3408296076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c25e0 0x7fd6001526a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd6000c59a0 tx=0x7fd5f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0c41af77291ecf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:07.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.011+0000 7fd5fe7fc640 1 -- 172.21.15.78:0/3408296076 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f002e070 con 0x7fd6000c25e0 2024-02-19T14:56:07.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.011+0000 7fd5fe7fc640 1 -- 172.21.15.78:0/3408296076 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f00315a0 con 0x7fd6000c25e0 2024-02-19T14:56:07.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.011+0000 7fd5fe7fc640 1 -- 172.21.15.78:0/3408296076 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0031930 con 0x7fd6000c25e0 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 -- 172.21.15.78:0/3408296076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c25e0 msgr2=0x7fd6001526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 --2- 172.21.15.78:0/3408296076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c25e0 0x7fd6001526a0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd6000c59a0 tx=0x7fd5f002b840 comp rx=0 tx=0).stop 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 -- 172.21.15.78:0/3408296076 shutdown_connections 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 --2- 172.21.15.78:0/3408296076 >> v2:172.21.15.150:3300/0 conn(0x7fd6000c29c0 0x7fd6000c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 --2- 172.21.15.78:0/3408296076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c25e0 0x7fd6001526a0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 -- 172.21.15.78:0/3408296076 >> 172.21.15.78:0/3408296076 conn(0x7fd60006b270 msgr2=0x7fd60006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 -- 172.21.15.78:0/3408296076 shutdown_connections 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.012+0000 7fd605d83640 1 -- 172.21.15.78:0/3408296076 wait complete. 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.013+0000 7fd605d83640 1 Processor -- start 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.013+0000 7fd605d83640 1 -- start start 2024-02-19T14:56:07.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd605d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6000c25e0 0x7fd6001e86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd605d83640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 0x7fd600154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd605d83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6001e90c0 0x7fd6001540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd605d83640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6001e8c10 con 0x7fd6000c29c0 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd605d83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6000c0240 con 0x7fd6000c25e0 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd605d83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6000c0520 con 0x7fd6001e90c0 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 0x7fd600154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 0x7fd600154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54182/0 (socket says 172.21.15.78:54182) 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 -- 172.21.15.78:0/136521859 learned_addr learned my addr 172.21.15.78:0/136521859 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5ff7fe640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6000c25e0 0x7fd6001e86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5fffff640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6001e90c0 0x7fd6001540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 -- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6001e90c0 msgr2=0x7fd6001540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6001e90c0 0x7fd6001540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 -- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6000c25e0 msgr2=0x7fd6001e86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6000c25e0 0x7fd6001e86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 -- 172.21.15.78:0/136521859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f0009400 con 0x7fd6000c29c0 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5feffd640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 0x7fd600154e00 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400e9b0 tx=0x7fd5f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.014+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4009670 con 0x7fd6000c29c0 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.015+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f400c9a0 con 0x7fd6000c29c0 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.015+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6001535d0 con 0x7fd6000c29c0 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.015+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400cd50 con 0x7fd6000c29c0 2024-02-19T14:56:07.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.015+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd600153b90 con 0x7fd6000c29c0 2024-02-19T14:56:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.016+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd600074990 con 0x7fd6000c29c0 2024-02-19T14:56:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.016+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd5f4013470 con 0x7fd6000c29c0 2024-02-19T14:56:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.016+0000 7fd5e7fff640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5ec06b840 0x7fd5ec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.016+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd5f408a520 con 0x7fd6000c29c0 2024-02-19T14:56:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.017+0000 7fd5ff7fe640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5ec06b840 0x7fd5ec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.017+0000 7fd5ff7fe640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5ec06b840 0x7fd5ec06dd00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0036000 tx=0x7fd5f003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:07.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.021+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd5f4054e10 con 0x7fd6000c29c0 2024-02-19T14:56:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.226+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd60014a270 con 0x7fd6000c29c0 2024-02-19T14:56:07.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:56:07.227+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:56:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.228+0000 7fd5e7fff640 1 -- 172.21.15.78:0/136521859 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd5f40584d0 con 0x7fd6000c29c0 2024-02-19T14:56:07.231 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5ec06b840 msgr2=0x7fd5ec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5ec06b840 0x7fd5ec06dd00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0036000 tx=0x7fd5f003d040 comp rx=0 tx=0).stop 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 msgr2=0x7fd600154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 0x7fd600154e00 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400e9b0 tx=0x7fd5f400ee70 comp rx=0 tx=0).stop 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 shutdown_connections 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5ec06b840 0x7fd5ec06dd00 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6000c29c0 0x7fd600154e00 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6001e90c0 0x7fd6001540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 --2- 172.21.15.78:0/136521859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6000c25e0 0x7fd6001e86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 >> 172.21.15.78:0/136521859 conn(0x7fd60006b270 msgr2=0x7fd60006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 shutdown_connections 2024-02-19T14:56:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.230+0000 7fd605d83640 1 -- 172.21.15.78:0/136521859 wait complete. 2024-02-19T14:56:07.236 INFO:tasks.ceph.mds.b.smithi150.stderr: -566> 2024-02-19T14:55:54.221+0000 7fd0a4208640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:56:07.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:56:07.258 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:56:07.258 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:56:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 Processor -- start 2024-02-19T14:56:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 -- start start 2024-02-19T14:56:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb974152700 0x7fb9740c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb9740c5730 0x7fb97406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9741551f0 con 0x7fb9740c8e00 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb974155490 con 0x7fb974152700 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.344+0000 7fb979cdf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb974155770 con 0x7fb9740c5730 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb974152700 0x7fb9740c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb974152700 0x7fb9740c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54186/0 (socket says 172.21.15.78:54186) 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 -- 172.21.15.78:0/2034417657 learned_addr learned my addr 172.21.15.78:0/2034417657 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 -- 172.21.15.78:0/2034417657 >> v2:172.21.15.150:3300/0 conn(0x7fb9740c5730 msgr2=0x7fb97406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 --2- 172.21.15.78:0/2034417657 >> v2:172.21.15.150:3300/0 conn(0x7fb9740c5730 0x7fb97406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 -- 172.21.15.78:0/2034417657 >> v1:172.21.15.150:6789/0 conn(0x7fb9740c8e00 legacy=0x7fb974158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:07.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.345+0000 7fb9737fe640 1 -- 172.21.15.78:0/2034417657 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb974155a50 con 0x7fb974152700 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb9737fe640 1 --2- 172.21.15.78:0/2034417657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb974152700 0x7fb9740c51f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb9740c7620 tx=0x7fb96402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9b831a570e3d13 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb9727fc640 1 -- 172.21.15.78:0/2034417657 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640317e0 con 0x7fb974152700 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb9727fc640 1 -- 172.21.15.78:0/2034417657 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb964031940 con 0x7fb974152700 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb9727fc640 1 -- 172.21.15.78:0/2034417657 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964031cd0 con 0x7fb974152700 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 -- 172.21.15.78:0/2034417657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb974152700 msgr2=0x7fb9740c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 --2- 172.21.15.78:0/2034417657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb974152700 0x7fb9740c51f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb9740c7620 tx=0x7fb96402b880 comp rx=0 tx=0).stop 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 -- 172.21.15.78:0/2034417657 shutdown_connections 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 --2- 172.21.15.78:0/2034417657 >> v2:172.21.15.150:3300/0 conn(0x7fb9740c5730 0x7fb97406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 --2- 172.21.15.78:0/2034417657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb974152700 0x7fb9740c51f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 -- 172.21.15.78:0/2034417657 >> 172.21.15.78:0/2034417657 conn(0x7fb9740cac90 msgr2=0x7fb9740cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 -- 172.21.15.78:0/2034417657 shutdown_connections 2024-02-19T14:56:07.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.346+0000 7fb979cdf640 1 -- 172.21.15.78:0/2034417657 wait complete. 2024-02-19T14:56:07.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.347+0000 7fb979cdf640 1 Processor -- start 2024-02-19T14:56:07.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 -- start start 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9740c5730 0x7fb9740ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 0x7fb9741ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb974152700 0x7fb9741ec290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740abaf0 con 0x7fb9740c8e00 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740aac40 con 0x7fb974152700 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb979cdf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740aaee0 con 0x7fb9740c5730 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 0x7fb9741ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 0x7fb9741ebd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54190/0 (socket says 172.21.15.78:54190) 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb972ffd640 1 -- 172.21.15.78:0/1375884986 learned_addr learned my addr 172.21.15.78:0/1375884986 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb9737fe640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9740c5730 0x7fb9740ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.348+0000 7fb973fff640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb974152700 0x7fb9741ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb972ffd640 1 -- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9740c5730 msgr2=0x7fb9740ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb972ffd640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9740c5730 0x7fb9740ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb972ffd640 1 -- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb974152700 msgr2=0x7fb9741ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb972ffd640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb974152700 0x7fb9741ec290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb972ffd640 1 -- 172.21.15.78:0/1375884986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9640094d0 con 0x7fb9740c8e00 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb972ffd640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 0x7fb9741ebd50 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb96800b4a0 tx=0x7fb96800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968010610 con 0x7fb9740c8e00 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb968010770 con 0x7fb9740c8e00 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9740ab1c0 con 0x7fb9740c8e00 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968010b20 con 0x7fb9740c8e00 2024-02-19T14:56:07.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.349+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9740ab780 con 0x7fb9740c8e00 2024-02-19T14:56:07.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.350+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9740abc60 con 0x7fb9740c8e00 2024-02-19T14:56:07.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.350+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb968010c80 con 0x7fb9740c8e00 2024-02-19T14:56:07.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.351+0000 7fb95bfff640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb96006b810 0x7fb96006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:07.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.351+0000 7fb9737fe640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb96006b810 0x7fb96006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:07.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.351+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fb968089f20 con 0x7fb9740c8e00 2024-02-19T14:56:07.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.351+0000 7fb9737fe640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb96006b810 0x7fb96006dcd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb964036000 tx=0x7fb96403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:07.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.355+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb968054840 con 0x7fb9740c8e00 2024-02-19T14:56:07.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:07.567+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb974074990 con 0x7fb9740c8e00 2024-02-19T14:56:08.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.259+0000 7fb95bfff640 1 -- 172.21.15.78:0/1375884986 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb9680140a0 con 0x7fb9740c8e00 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb96006b810 msgr2=0x7fb96006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb96006b810 0x7fb96006dcd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb964036000 tx=0x7fb96403d040 comp rx=0 tx=0).stop 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 msgr2=0x7fb9741ebd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 0x7fb9741ebd50 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb96800b4a0 tx=0x7fb96800b960 comp rx=0 tx=0).stop 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 shutdown_connections 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb96006b810 0x7fb96006dcd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9740c8e00 0x7fb9741ebd50 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb974152700 0x7fb9741ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 --2- 172.21.15.78:0/1375884986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9740c5730 0x7fb9740ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 >> 172.21.15.78:0/1375884986 conn(0x7fb9740cac90 msgr2=0x7fb9740cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:08.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 shutdown_connections 2024-02-19T14:56:08.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.262+0000 7fb979cdf640 1 -- 172.21.15.78:0/1375884986 wait complete. 2024-02-19T14:56:08.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:56:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.379+0000 7fea5a18c640 1 Processor -- start 2024-02-19T14:56:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.379+0000 7fea5a18c640 1 -- start start 2024-02-19T14:56:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.379+0000 7fea5a18c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 0x7fea5406d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.379+0000 7fea5a18c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5406dc70 con 0x7fea540c8730 2024-02-19T14:56:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.379+0000 7fea5a18c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea54158ef0 con 0x7fea54154ea0 2024-02-19T14:56:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.379+0000 7fea5a18c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea54155a40 con 0x7fea54152670 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea52ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fea540c8730 0x7fea540c51b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42314/0 (socket says 172.21.15.78:42314) 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea52ffd640 1 -- 172.21.15.78:0/1442878130 learned_addr learned my addr 172.21.15.78:0/1442878130 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea537fe640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 0x7fea5406d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265947191 0 0) 0x7fea5406dc70 con 0x7fea540c8730 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c003520 con 0x7fea540c8730 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2374011668 0 0) 0x7fea3c003520 con 0x7fea540c8730 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 msgr2=0x7fea5406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 0x7fea5406d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 >> v1:172.21.15.150:6789/0 conn(0x7fea54154ea0 legacy=0x7fea54155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea540be190 con 0x7fea540c8730 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fea44001670 con 0x7fea540c8730 2024-02-19T14:56:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 >> v1:172.21.15.78:6789/0 conn(0x7fea540c8730 legacy=0x7fea540c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea3c0039a0 0x7fea3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 0x7fea3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c004d00 0x7fea3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c00bb10 con 0x7fea3c0039a0 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c00bc80 con 0x7fea3c004d00 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c00be40 con 0x7fea54152670 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea537fe640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea3c0039a0 0x7fea3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea52ffd640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 0x7fea3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea53fff640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c004d00 0x7fea3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea537fe640 1 -- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 msgr2=0x7fea3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea537fe640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea54152670 0x7fea3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea537fe640 1 -- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c004d00 msgr2=0x7fea3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.380+0000 7fea537fe640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c004d00 0x7fea3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.381+0000 7fea537fe640 1 -- 172.21.15.78:0/1442878130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c003520 con 0x7fea3c0039a0 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.381+0000 7fea537fe640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea3c0039a0 0x7fea3c003dc0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fea48009ac0 tx=0x7fea3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bf2892e7080b1f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.381+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4802c6b0 con 0x7fea3c0039a0 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.381+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4802c810 con 0x7fea3c0039a0 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.381+0000 7fea527fc640 1 -- 172.21.15.78:0/1442878130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4802cba0 con 0x7fea3c0039a0 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea5a18c640 1 -- 172.21.15.78:0/1442878130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea3c0039a0 msgr2=0x7fea3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea5a18c640 1 --2- 172.21.15.78:0/1442878130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea3c0039a0 0x7fea3c003dc0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fea48009ac0 tx=0x7fea3c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea53fff640 1 -- 172.21.15.78:0/1442878130 reap_dead start 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea5a18c640 1 -- 172.21.15.78:0/1442878130 shutdown_connections 2024-02-19T14:56:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea5a18c640 1 -- 172.21.15.78:0/1442878130 >> 172.21.15.78:0/1442878130 conn(0x7fea54072790 msgr2=0x7fea54072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea5a18c640 1 -- 172.21.15.78:0/1442878130 shutdown_connections 2024-02-19T14:56:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.382+0000 7fea5a18c640 1 -- 172.21.15.78:0/1442878130 wait complete. 2024-02-19T14:56:08.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.383+0000 7fea5a18c640 1 Processor -- start 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.383+0000 7fea5a18c640 1 -- start start 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea5a18c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c0039a0 0x7fea540c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea5a18c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea540ae9e0 0x7fea541e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea5a18c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 0x7fea540aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea5a18c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea540ac1a0 con 0x7fea541e7aa0 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea5a18c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5414f760 con 0x7fea3c0039a0 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea5a18c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea541ea2d0 con 0x7fea540ae9e0 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 0x7fea540aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 0x7fea540aa370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45080/0 (socket says 172.21.15.78:45080) 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 -- 172.21.15.78:0/37752306 learned_addr learned my addr 172.21.15.78:0/37752306 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea52ffd640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea540ae9e0 0x7fea541e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea537fe640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c0039a0 0x7fea540c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 -- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea540ae9e0 msgr2=0x7fea541e7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea540ae9e0 0x7fea541e7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 -- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c0039a0 msgr2=0x7fea540c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c0039a0 0x7fea540c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 -- 172.21.15.78:0/37752306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea541ea5b0 con 0x7fea541e7aa0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.384+0000 7fea53fff640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 0x7fea540aa370 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fea54155280 tx=0x7fea4c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.385+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c003970 con 0x7fea541e7aa0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.385+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c003ad0 con 0x7fea541e7aa0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.385+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea541ea890 con 0x7fea541e7aa0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.385+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c003e80 con 0x7fea541e7aa0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.385+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea540af7b0 con 0x7fea541e7aa0 2024-02-19T14:56:08.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.386+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea541eadb0 con 0x7fea541e7aa0 2024-02-19T14:56:08.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.386+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fea4c0050d0 con 0x7fea541e7aa0 2024-02-19T14:56:08.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.386+0000 7fea3bfff640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4006b810 0x7fea4006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:08.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.386+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fea4c08a0a0 con 0x7fea541e7aa0 2024-02-19T14:56:08.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.386+0000 7fea537fe640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4006b810 0x7fea4006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:08.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.387+0000 7fea537fe640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4006b810 0x7fea4006dcd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fea48036000 tx=0x7fea48030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:08.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.391+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fea541eadb0 con 0x7fea541e7aa0 2024-02-19T14:56:08.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:08.563+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fea541eadb0 con 0x7fea541e7aa0 2024-02-19T14:56:09.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.269+0000 7fea3bfff640 1 -- 172.21.15.78:0/37752306 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v114) v1 ==== 181+0+0 (secure 0 0 0) 0x7fea541eadb0 con 0x7fea541e7aa0 2024-02-19T14:56:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4006b810 msgr2=0x7fea4006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4006b810 0x7fea4006dcd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fea48036000 tx=0x7fea48030000 comp rx=0 tx=0).stop 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 msgr2=0x7fea540aa370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 0x7fea540aa370 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fea54155280 tx=0x7fea4c00ec10 comp rx=0 tx=0).stop 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 shutdown_connections 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4006b810 0x7fea4006dcd0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea541e7aa0 0x7fea540aa370 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea540ae9e0 0x7fea541e7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 --2- 172.21.15.78:0/37752306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea3c0039a0 0x7fea540c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 >> 172.21.15.78:0/37752306 conn(0x7fea54072790 msgr2=0x7fea54154400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 shutdown_connections 2024-02-19T14:56:09.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.271+0000 7fea5a18c640 1 -- 172.21.15.78:0/37752306 wait complete. 2024-02-19T14:56:09.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:56:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.389+0000 7fe80196c640 1 Processor -- start 2024-02-19T14:56:09.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.389+0000 7fe80196c640 1 -- start start 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe80196c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7fc06d000 0x7fe7fc06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe80196c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7fc155010 0x7fe7fc155460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe80196c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc159210 con 0x7fe7fc152620 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe80196c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc0c1f00 con 0x7fe7fc06d000 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe80196c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc06d980 con 0x7fe7fc155010 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7fc06d000 0x7fe7fc06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7fc06d000 0x7fe7fc06d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45082/0 (socket says 172.21.15.78:45082) 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 -- 172.21.15.78:0/352800883 learned_addr learned my addr 172.21.15.78:0/352800883 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7faffd640 1 --1- 172.21.15.78:0/352800883 >> v1:172.21.15.78:6789/0 conn(0x7fe7fc152620 0x7fe7fc0c6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42326/0 (socket says 172.21.15.78:42326) 2024-02-19T14:56:09.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fb7fe640 1 --2- 172.21.15.78:0/352800883 >> v2:172.21.15.150:3300/0 conn(0x7fe7fc155010 0x7fe7fc155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 -- 172.21.15.78:0/352800883 >> v2:172.21.15.150:3300/0 conn(0x7fe7fc155010 msgr2=0x7fe7fc155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 --2- 172.21.15.78:0/352800883 >> v2:172.21.15.150:3300/0 conn(0x7fe7fc155010 0x7fe7fc155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7f9ffb640 1 -- 172.21.15.78:0/352800883 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983252735 0 0) 0x7fe7fc159210 con 0x7fe7fc152620 2024-02-19T14:56:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 -- 172.21.15.78:0/352800883 >> v1:172.21.15.78:6789/0 conn(0x7fe7fc152620 legacy=0x7fe7fc0c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fa7fc640 1 -- 172.21.15.78:0/352800883 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0c6420 con 0x7fe7fc06d000 2024-02-19T14:56:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.390+0000 7fe7fb7fe640 1 --2- 172.21.15.78:0/352800883 >> v2:172.21.15.150:3300/0 conn(0x7fe7fc155010 0x7fe7fc155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:09.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe7fa7fc640 1 --2- 172.21.15.78:0/352800883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc06d440 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc06cd10 tx=0x7fe7f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed334dabd1b22a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:09.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe7f9ffb640 1 -- 172.21.15.78:0/352800883 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0031870 con 0x7fe7fc06d000 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe7f9ffb640 1 -- 172.21.15.78:0/352800883 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f00319d0 con 0x7fe7fc06d000 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe7f9ffb640 1 -- 172.21.15.78:0/352800883 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0031d60 con 0x7fe7fc06d000 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe80196c640 1 -- 172.21.15.78:0/352800883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 msgr2=0x7fe7fc06d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe80196c640 1 --2- 172.21.15.78:0/352800883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc06d440 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc06cd10 tx=0x7fe7f002b910 comp rx=0 tx=0).stop 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe80196c640 1 -- 172.21.15.78:0/352800883 shutdown_connections 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe80196c640 1 --2- 172.21.15.78:0/352800883 >> v2:172.21.15.150:3300/0 conn(0x7fe7fc155010 0x7fe7fc155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe80196c640 1 --2- 172.21.15.78:0/352800883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc06d440 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.391+0000 7fe80196c640 1 -- 172.21.15.78:0/352800883 >> 172.21.15.78:0/352800883 conn(0x7fe7fc0caae0 msgr2=0x7fe7fc0caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.392+0000 7fe80196c640 1 -- 172.21.15.78:0/352800883 shutdown_connections 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.392+0000 7fe80196c640 1 -- 172.21.15.78:0/352800883 wait complete. 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 Processor -- start 2024-02-19T14:56:09.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 -- start start 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc0bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7fc152620 0x7fe7fc14a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7fc155010 0x7fe7fc14aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1569d0 con 0x7fe7fc06d000 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1583c0 con 0x7fe7fc152620 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe80196c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0ca2a0 con 0x7fe7fc155010 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.393+0000 7fe7faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc0bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc0bffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45094/0 (socket says 172.21.15.78:45094) 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 -- 172.21.15.78:0/3346313559 learned_addr learned my addr 172.21.15.78:0/3346313559 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 -- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7fc155010 msgr2=0x7fe7fc14aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7fb7fe640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7fc155010 0x7fe7fc14aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:09.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7fa7fc640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7fc152620 0x7fe7fc14a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7fc155010 0x7fe7fc14aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 -- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7fc152620 msgr2=0x7fe7fc14a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7fc152620 0x7fe7fc14a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 -- 172.21.15.78:0/3346313559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f00094d0 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7fa7fc640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7fc152620 0x7fe7fc14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7faffd640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc0bffa0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00c4d0 tx=0x7fe7ec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec0074e0 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec007660 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc0c92b0 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec004a20 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.394+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc14c2c0 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.395+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc0c9110 con 0x7fe7fc06d000 2024-02-19T14:56:09.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.396+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe7ec004b80 con 0x7fe7fc06d000 2024-02-19T14:56:09.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.396+0000 7fe7e3fff640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7e806b810 0x7fe7e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:09.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.396+0000 7fe7fa7fc640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7e806b810 0x7fe7e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:09.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.396+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fe7ec089740 con 0x7fe7fc06d000 2024-02-19T14:56:09.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.397+0000 7fe7fa7fc640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7e806b810 0x7fe7e806dcd0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0036000 tx=0x7fe7f003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:09.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.401+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe7fc0c9110 con 0x7fe7fc06d000 2024-02-19T14:56:09.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:09.574+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe7fc0c9110 con 0x7fe7fc06d000 2024-02-19T14:56:10.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.273+0000 7fe7e3fff640 1 -- 172.21.15.78:0/3346313559 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v115) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe7fc0c9110 con 0x7fe7fc06d000 2024-02-19T14:56:10.276 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7e806b810 msgr2=0x7fe7e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7e806b810 0x7fe7e806dcd0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0036000 tx=0x7fe7f003c040 comp rx=0 tx=0).stop 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 msgr2=0x7fe7fc0bffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc0bffa0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00c4d0 tx=0x7fe7ec00c990 comp rx=0 tx=0).stop 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 shutdown_connections 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7e806b810 0x7fe7e806dcd0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7fc155010 0x7fe7fc14aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7fc152620 0x7fe7fc14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 --2- 172.21.15.78:0/3346313559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7fc06d000 0x7fe7fc0bffa0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 >> 172.21.15.78:0/3346313559 conn(0x7fe7fc0caae0 msgr2=0x7fe7fc0c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:10.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 shutdown_connections 2024-02-19T14:56:10.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.277+0000 7fe80196c640 1 -- 172.21.15.78:0/3346313559 wait complete. 2024-02-19T14:56:10.293 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-19T14:56:10.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.394+0000 7fc1d73d0640 1 Processor -- start 2024-02-19T14:56:10.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.394+0000 7fc1d73d0640 1 -- start start 2024-02-19T14:56:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d73d0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1d00ad890 0x7fc1d00c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d73d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d73d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d00aba60 con 0x7fc1d00c49c0 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d73d0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d0152d30 con 0x7fc1d00ad890 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d73d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d00c7860 con 0x7fc1d00ab600 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d596d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d516c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1d00ad890 0x7fc1d00c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d496b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc1d00c49c0 0x7fc1d01534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:36560/0 (socket says 172.21.15.78:36560) 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d596d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:44712/0 (socket says 172.21.15.78:44712) 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.395+0000 7fc1d496b640 1 -- 172.21.15.78:0/2818462028 learned_addr learned my addr 172.21.15.78:0/2818462028 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.396+0000 7fc1d516c640 1 -- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 msgr2=0x7fc1d00c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.396+0000 7fc1d516c640 1 --2- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.396+0000 7fc1d516c640 1 -- 172.21.15.78:0/2818462028 >> v1:172.21.15.150:6789/0 conn(0x7fc1d00c49c0 legacy=0x7fc1d01534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.396+0000 7fc1d516c640 1 -- 172.21.15.78:0/2818462028 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d00c7ce0 con 0x7fc1d00ad890 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d516c640 1 --2- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d00c64f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0097a0 tx=0x7fc1cc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f2f06b754089ba7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1c7fff640 1 -- 172.21.15.78:0/2818462028 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc02f050 con 0x7fc1d00ad890 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1c7fff640 1 -- 172.21.15.78:0/2818462028 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1cc031700 con 0x7fc1d00ad890 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1c7fff640 1 -- 172.21.15.78:0/2818462028 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc031a90 con 0x7fc1d00ad890 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 -- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 msgr2=0x7fc1d00c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d00c64f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0097a0 tx=0x7fc1cc02b7a0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 -- 172.21.15.78:0/2818462028 shutdown_connections 2024-02-19T14:56:10.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/2818462028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d00c64f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 -- 172.21.15.78:0/2818462028 >> 172.21.15.78:0/2818462028 conn(0x7fc1d006b700 msgr2=0x7fc1d00aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 -- 172.21.15.78:0/2818462028 shutdown_connections 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.397+0000 7fc1d73d0640 1 -- 172.21.15.78:0/2818462028 wait complete. 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.398+0000 7fc1d73d0640 1 Processor -- start 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.398+0000 7fc1d73d0640 1 -- start start 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d73d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d73d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d01ea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d73d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 0x7fc1d01ead30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d73d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0070640 con 0x7fc1d00c49c0 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d73d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d01ee550 con 0x7fc1d00ad890 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d73d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d006ff20 con 0x7fc1d00ab600 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d596d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 0x7fc1d01ead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d596d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 0x7fc1d01ead30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45098/0 (socket says 172.21.15.78:45098) 2024-02-19T14:56:10.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d596d640 1 -- 172.21.15.78:0/3113310592 learned_addr learned my addr 172.21.15.78:0/3113310592 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d516c640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.399+0000 7fc1d496b640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d01ea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d596d640 1 -- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 msgr2=0x7fc1d00c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d596d640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d596d640 1 -- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 msgr2=0x7fc1d01ea7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d596d640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d01ea7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d596d640 1 -- 172.21.15.78:0/3113310592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc009400 con 0x7fc1d00c49c0 2024-02-19T14:56:10.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d596d640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 0x7fc1d01ead30 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800efc0 tx=0x7fc1c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8009ba0 con 0x7fc1d00c49c0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c800f040 con 0x7fc1d00c49c0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0070200 con 0x7fc1d00c49c0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80146c0 con 0x7fc1d00c49c0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.400+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d01ee1e0 con 0x7fc1d00c49c0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.401+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d01edd60 con 0x7fc1d00c49c0 2024-02-19T14:56:10.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.401+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc1c8014820 con 0x7fc1d00c49c0 2024-02-19T14:56:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.402+0000 7fc1c5ffb640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b406b810 0x7fc1b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.402+0000 7fc1d516c640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b406b810 0x7fc1b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.402+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc1c808acb0 con 0x7fc1d00c49c0 2024-02-19T14:56:10.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.402+0000 7fc1d516c640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b406b810 0x7fc1b406dcd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc035000 tx=0x7fc1cc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:10.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.406+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1d01edd60 con 0x7fc1d00c49c0 2024-02-19T14:56:10.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.585+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7fc1d01edd60 con 0x7fc1d00c49c0 2024-02-19T14:56:10.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.588+0000 7fc1c5ffb640 1 -- 172.21.15.78:0/3113310592 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v38) v1 ==== 96+0+0 (secure 0 0 0) 0x7fc1d01edd60 con 0x7fc1d00c49c0 2024-02-19T14:56:10.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b406b810 msgr2=0x7fc1b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b406b810 0x7fc1b406dcd0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc035000 tx=0x7fc1cc008030 comp rx=0 tx=0).stop 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 msgr2=0x7fc1d01ead30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 0x7fc1d01ead30 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800efc0 tx=0x7fc1c800c490 comp rx=0 tx=0).stop 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 shutdown_connections 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b406b810 0x7fc1b406dcd0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00c49c0 0x7fc1d01ead30 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d00ad890 0x7fc1d01ea7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.590+0000 7fc1d73d0640 1 --2- 172.21.15.78:0/3113310592 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d00ab600 0x7fc1d00c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.591+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 >> 172.21.15.78:0/3113310592 conn(0x7fc1d006b700 msgr2=0x7fc1d00a9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.591+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 shutdown_connections 2024-02-19T14:56:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.591+0000 7fc1d73d0640 1 -- 172.21.15.78:0/3113310592 wait complete. 2024-02-19T14:56:10.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:56:10.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.703+0000 7f22e5474640 1 Processor -- start 2024-02-19T14:56:10.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.703+0000 7f22e5474640 1 -- start start 2024-02-19T14:56:10.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22e5474640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e0071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22e5474640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22e00702d0 0x7f22e0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22e5474640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e0071bf0 con 0x7f22e0152670 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22e5474640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e0158a20 con 0x7f22e00702d0 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22e5474640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e00c6780 con 0x7f22e0152a50 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22deffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f22e0152670 0x7f22e0155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42334/0 (socket says 172.21.15.78:42334) 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22deffd640 1 -- 172.21.15.78:0/1736901241 learned_addr learned my addr 172.21.15.78:0/1736901241 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.704+0000 7f22df7fe640 1 --2- 172.21.15.78:0/1736901241 >> v2:172.21.15.78:3300/0 conn(0x7f22e00702d0 0x7f22e0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22de7fc640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e0071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22df7fe640 1 -- 172.21.15.78:0/1736901241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 msgr2=0x7f22e0071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22df7fe640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e0071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22df7fe640 1 -- 172.21.15.78:0/1736901241 >> v1:172.21.15.78:6789/0 conn(0x7f22e0152670 legacy=0x7f22e0155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22df7fe640 1 -- 172.21.15.78:0/1736901241 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0071630 con 0x7f22e00702d0 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22ddffb640 1 -- 172.21.15.78:0/1736901241 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027555159 0 0) 0x7f22e0071bf0 con 0x7f22e0152670 2024-02-19T14:56:10.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22de7fc640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e0071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22df7fe640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e00702d0 0x7f22e0158eb0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f22d80097a0 tx=0x7f22d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603766b2a992b209 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.705+0000 7f22ddffb640 1 -- 172.21.15.78:0/1736901241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8030610 con 0x7f22e00702d0 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22ddffb640 1 -- 172.21.15.78:0/1736901241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d8030770 con 0x7f22e00702d0 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22ddffb640 1 -- 172.21.15.78:0/1736901241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8030b00 con 0x7f22e00702d0 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 -- 172.21.15.78:0/1736901241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e00702d0 msgr2=0x7f22e0158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e00702d0 0x7f22e0158eb0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f22d80097a0 tx=0x7f22d802b960 comp rx=0 tx=0).stop 2024-02-19T14:56:10.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 -- 172.21.15.78:0/1736901241 shutdown_connections 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e00702d0 0x7f22e0158eb0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 --2- 172.21.15.78:0/1736901241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e0071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 -- 172.21.15.78:0/1736901241 >> 172.21.15.78:0/1736901241 conn(0x7f22e00c85e0 msgr2=0x7f22e00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 -- 172.21.15.78:0/1736901241 shutdown_connections 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.706+0000 7f22e5474640 1 -- 172.21.15.78:0/1736901241 wait complete. 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.707+0000 7f22e5474640 1 Processor -- start 2024-02-19T14:56:10.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.707+0000 7f22e5474640 1 -- start start 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22e5474640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00702d0 0x7f22e0154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22e5474640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 0x7f22e01edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22e5474640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e01ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22e5474640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e01edb50 con 0x7f22e0152670 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22e5474640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e01551b0 con 0x7f22e00702d0 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22e5474640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0155490 con 0x7f22e0152a50 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 0x7f22e01edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 0x7f22e01edf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45118/0 (socket says 172.21.15.78:45118) 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 -- 172.21.15.78:0/1154317135 learned_addr learned my addr 172.21.15.78:0/1154317135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22df7fe640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e01ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22deffd640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00702d0 0x7f22e0154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 -- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 msgr2=0x7f22e01ee4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e01ee4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 -- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00702d0 msgr2=0x7f22e0154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00702d0 0x7f22e0154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.708+0000 7f22de7fc640 1 -- 172.21.15.78:0/1154317135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d8009400 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.709+0000 7f22de7fc640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 0x7f22e01edf70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c7f0 tx=0x7f22d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.709+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d400cea0 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.709+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4007a30 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.709+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e0150020 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.709+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4007dc0 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.709+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e0154680 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.710+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e00676a0 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.710+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f22d40113f0 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.710+0000 7f22c77fe640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22cc06b810 0x7f22cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.711+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f22d408a1b0 con 0x7f22e0152670 2024-02-19T14:56:10.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.711+0000 7f22deffd640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22cc06b810 0x7f22cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.711+0000 7f22deffd640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22cc06b810 0x7f22cc06dcd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f22d00012e0 tx=0x7f22d0007ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.715+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f22d4054d80 con 0x7f22e0152670 2024-02-19T14:56:10.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.887+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f22e01edc90 con 0x7f22e0152670 2024-02-19T14:56:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.890+0000 7f22c77fe640 1 -- 172.21.15.78:0/1154317135 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v39) v1 ==== 96+0+0 (secure 0 0 0) 0x7f22e01edc90 con 0x7f22e0152670 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22cc06b810 msgr2=0x7f22cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22cc06b810 0x7f22cc06dcd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f22d00012e0 tx=0x7f22d0007ed0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 msgr2=0x7f22e01edf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 0x7f22e01edf70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c7f0 tx=0x7f22d400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 shutdown_connections 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22cc06b810 0x7f22cc06dcd0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22e0152670 0x7f22e01edf70 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22e0152a50 0x7f22e01ee4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 --2- 172.21.15.78:0/1154317135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00702d0 0x7f22e0154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 >> 172.21.15.78:0/1154317135 conn(0x7f22e00c85e0 msgr2=0x7f22e00a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 shutdown_connections 2024-02-19T14:56:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:10.892+0000 7f22e5474640 1 -- 172.21.15.78:0/1154317135 wait complete. 2024-02-19T14:56:10.905 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:56:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.005+0000 7f17d6f65640 1 Processor -- start 2024-02-19T14:56:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.006+0000 7f17d6f65640 1 -- start start 2024-02-19T14:56:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.006+0000 7f17d6f65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17d0150440 0x7f17d00c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.006+0000 7f17d6f65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d006d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.006+0000 7f17d6f65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d0150820 con 0x7f17d006ed20 2024-02-19T14:56:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.006+0000 7f17d6f65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d00c1b60 con 0x7f17d0150440 2024-02-19T14:56:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.006+0000 7f17d6f65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d01537d0 con 0x7f17d00c3800 2024-02-19T14:56:11.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d4d01640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17d0150440 0x7f17d00c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d5502640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d5502640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d006d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:44736/0 (socket says 172.21.15.78:44736) 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d4d01640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17d0150440 0x7f17d00c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45432/0 (socket says 172.21.15.78:45432) 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d5502640 1 -- 172.21.15.78:0/3147360507 learned_addr learned my addr 172.21.15.78:0/3147360507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17cffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f17d006ed20 0x7f17d006d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:36574/0 (socket says 172.21.15.78:36574) 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d4d01640 1 -- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 msgr2=0x7f17d006d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d4d01640 1 --2- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d006d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d4d01640 1 -- 172.21.15.78:0/3147360507 >> v1:172.21.15.150:6789/0 conn(0x7f17d006ed20 legacy=0x7f17d006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d4d01640 1 -- 172.21.15.78:0/3147360507 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0153d10 con 0x7f17d0150440 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.007+0000 7f17d5502640 1 --2- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d006d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.008+0000 7f17d4d01640 1 --2- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d0150440 0x7f17d00c32c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f17d0153f00 tx=0x7f17c402b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bf068d4d9240d5a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.008+0000 7f17cf7fe640 1 -- 172.21.15.78:0/3147360507 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4031890 con 0x7f17d0150440 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.008+0000 7f17cf7fe640 1 -- 172.21.15.78:0/3147360507 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c40319f0 con 0x7f17d0150440 2024-02-19T14:56:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.008+0000 7f17cf7fe640 1 -- 172.21.15.78:0/3147360507 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4031d80 con 0x7f17d0150440 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.008+0000 7f17d6f65640 1 -- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d0150440 msgr2=0x7f17d00c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.008+0000 7f17d6f65640 1 --2- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d0150440 0x7f17d00c32c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f17d0153f00 tx=0x7f17c402b930 comp rx=0 tx=0).stop 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.009+0000 7f17d6f65640 1 -- 172.21.15.78:0/3147360507 shutdown_connections 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.009+0000 7f17d6f65640 1 --2- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d006d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.009+0000 7f17d6f65640 1 --2- 172.21.15.78:0/3147360507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d0150440 0x7f17d00c32c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.009+0000 7f17d6f65640 1 -- 172.21.15.78:0/3147360507 >> 172.21.15.78:0/3147360507 conn(0x7f17d006b270 msgr2=0x7f17d006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.009+0000 7f17d6f65640 1 -- 172.21.15.78:0/3147360507 shutdown_connections 2024-02-19T14:56:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.009+0000 7f17d6f65640 1 -- 172.21.15.78:0/3147360507 wait complete. 2024-02-19T14:56:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 Processor -- start 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 -- start start 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d006ed20 0x7f17d0156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d01432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 0x7f17d0152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d01dcae0 con 0x7f17d0150440 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d014b6e0 con 0x7f17d006ed20 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.010+0000 7f17d6f65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d01dc7b0 con 0x7f17d00c3800 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 0x7f17d0152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 0x7f17d0152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45126/0 (socket says 172.21.15.78:45126) 2024-02-19T14:56:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 -- 172.21.15.78:0/4055295809 learned_addr learned my addr 172.21.15.78:0/4055295809 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17cffff640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d01432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d4d01640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d006ed20 0x7f17d0156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 -- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 msgr2=0x7f17d01432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d01432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 -- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d006ed20 msgr2=0x7f17d0156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d006ed20 0x7f17d0156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 -- 172.21.15.78:0/4055295809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8009b30 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d5502640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 0x7f17d0152840 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f17c800efc0 tx=0x7f17c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8018070 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c800f040 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c800ce00 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c4009400 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.011+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d01e9a80 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.012+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d0074990 con 0x7f17d0150440 2024-02-19T14:56:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.013+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f17c8014480 con 0x7f17d0150440 2024-02-19T14:56:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.013+0000 7f17cd7fa640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f17b406b810 0x7f17b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.013+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f17c808a3c0 con 0x7f17d0150440 2024-02-19T14:56:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.013+0000 7f17d4d01640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f17b406b810 0x7f17b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.014+0000 7f17d4d01640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f17b406b810 0x7f17b406dcd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f17c4036000 tx=0x7f17c403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.017+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f17c8054f90 con 0x7f17d0150440 2024-02-19T14:56:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.192+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f17d01dcc20 con 0x7f17d0150440 2024-02-19T14:56:11.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.195+0000 7f17cd7fa640 1 -- 172.21.15.78:0/4055295809 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v40) v1 ==== 104+0+0 (secure 0 0 0) 0x7f17d01dcc20 con 0x7f17d0150440 2024-02-19T14:56:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f17b406b810 msgr2=0x7f17b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f17b406b810 0x7f17b406dcd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f17c4036000 tx=0x7f17c403c040 comp rx=0 tx=0).stop 2024-02-19T14:56:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 msgr2=0x7f17d0152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 0x7f17d0152840 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f17c800efc0 tx=0x7f17c800c490 comp rx=0 tx=0).stop 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 shutdown_connections 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f17b406b810 0x7f17b406dcd0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17d0150440 0x7f17d0152840 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17d00c3800 0x7f17d01432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 --2- 172.21.15.78:0/4055295809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17d006ed20 0x7f17d0156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 >> 172.21.15.78:0/4055295809 conn(0x7f17d006b270 msgr2=0x7f17d014f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 shutdown_connections 2024-02-19T14:56:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.197+0000 7f17d6f65640 1 -- 172.21.15.78:0/4055295809 wait complete. 2024-02-19T14:56:11.211 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-02-19T14:56:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.309+0000 7f1d003d7640 1 Processor -- start 2024-02-19T14:56:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.309+0000 7f1d003d7640 1 -- start start 2024-02-19T14:56:11.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1d003d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf80c5130 0x7f1cf80c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1d003d7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1cf8158f80 0x7f1cf80c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1d003d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1cf81559c0 0x7f1cf80c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1d003d7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf8158b60 con 0x7f1cf8158f80 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1d003d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf80c3cf0 con 0x7f1cf81559c0 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1d003d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8159400 con 0x7f1cf80c5130 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1cfd972640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1cf8158f80 0x7f1cf80c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1cfd972640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1cf8158f80 0x7f1cf80c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45136/0 (socket says 172.21.15.78:45136) 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1cfd972640 1 -- 172.21.15.78:0/2065063070 learned_addr learned my addr 172.21.15.78:0/2065063070 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1cfe173640 1 --2- 172.21.15.78:0/2065063070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf80c5130 0x7f1cf80c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.310+0000 7f1cfe974640 1 --2- 172.21.15.78:0/2065063070 >> v2:172.21.15.150:3300/0 conn(0x7f1cf81559c0 0x7f1cf80c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd972640 1 -- 172.21.15.78:0/2065063070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf80c5130 msgr2=0x7f1cf80c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd972640 1 --2- 172.21.15.78:0/2065063070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf80c5130 0x7f1cf80c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd972640 1 -- 172.21.15.78:0/2065063070 >> v2:172.21.15.150:3300/0 conn(0x7f1cf81559c0 msgr2=0x7f1cf80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:11.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd972640 1 --2- 172.21.15.78:0/2065063070 >> v2:172.21.15.150:3300/0 conn(0x7f1cf81559c0 0x7f1cf80c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd972640 1 -- 172.21.15.78:0/2065063070 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8158530 con 0x7f1cf8158f80 2024-02-19T14:56:11.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd972640 1 --2- 172.21.15.78:0/2065063070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf8158f80 0x7f1cf80c5510 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80732c0 tx=0x7f1ce802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304c60268b483378 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.311+0000 7f1cfd171640 1 -- 172.21.15.78:0/2065063070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8031870 con 0x7f1cf8158f80 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1cfd171640 1 -- 172.21.15.78:0/2065063070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce80319d0 con 0x7f1cf8158f80 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1cfd171640 1 -- 172.21.15.78:0/2065063070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8031d60 con 0x7f1cf8158f80 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 -- 172.21.15.78:0/2065063070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf8158f80 msgr2=0x7f1cf80c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 --2- 172.21.15.78:0/2065063070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf8158f80 0x7f1cf80c5510 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80732c0 tx=0x7f1ce802b910 comp rx=0 tx=0).stop 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 -- 172.21.15.78:0/2065063070 shutdown_connections 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 --2- 172.21.15.78:0/2065063070 >> v2:172.21.15.150:3300/0 conn(0x7f1cf81559c0 0x7f1cf80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 --2- 172.21.15.78:0/2065063070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf8158f80 0x7f1cf80c5510 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 --2- 172.21.15.78:0/2065063070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf80c5130 0x7f1cf80c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 -- 172.21.15.78:0/2065063070 >> 172.21.15.78:0/2065063070 conn(0x7f1cf806b4b0 msgr2=0x7f1cf80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 -- 172.21.15.78:0/2065063070 shutdown_connections 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.312+0000 7f1d003d7640 1 -- 172.21.15.78:0/2065063070 wait complete. 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.313+0000 7f1d003d7640 1 Processor -- start 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.313+0000 7f1d003d7640 1 -- start start 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1d003d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c5130 0x7f1cf8154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1d003d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 0x7f1cf81de980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1d003d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8158f80 0x7f1cf81deec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1d003d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf81e0490 con 0x7f1cf81559c0 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1d003d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8155010 con 0x7f1cf80c5130 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1d003d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf81dfc10 con 0x7f1cf8158f80 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 0x7f1cf81de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 0x7f1cf81de980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45142/0 (socket says 172.21.15.78:45142) 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 -- 172.21.15.78:0/3922985604 learned_addr learned my addr 172.21.15.78:0/3922985604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfe974640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8158f80 0x7f1cf81deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfe173640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c5130 0x7f1cf8154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 -- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8158f80 msgr2=0x7f1cf81deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8158f80 0x7f1cf81deec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 -- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c5130 msgr2=0x7f1cf8154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c5130 0x7f1cf8154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 -- 172.21.15.78:0/3922985604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80094d0 con 0x7f1cf81559c0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.314+0000 7f1cfd972640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 0x7f1cf81de980 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8000c00 tx=0x7f1ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.315+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce803ac60 con 0x7f1cf81559c0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.315+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce803adc0 con 0x7f1cf81559c0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.315+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf81ebd80 con 0x7f1cf81559c0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.315+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8044630 con 0x7f1cf81559c0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.315+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf81e01b0 con 0x7f1cf81559c0 2024-02-19T14:56:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.316+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf8074990 con 0x7f1cf81559c0 2024-02-19T14:56:11.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.316+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1ce804d450 con 0x7f1cf81559c0 2024-02-19T14:56:11.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.316+0000 7f1ceeffd640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cdc06b810 0x7f1cdc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:11.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.317+0000 7f1cfe173640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cdc06b810 0x7f1cdc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:11.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.317+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1ce802d040 con 0x7f1cf81559c0 2024-02-19T14:56:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.317+0000 7f1cfe173640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cdc06b810 0x7f1cdc06dcd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009a20 tx=0x7f1cf4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:11.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.321+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ce8077130 con 0x7f1cf81559c0 2024-02-19T14:56:11.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:11.556+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7f1cf80c9400 con 0x7f1cf81559c0 2024-02-19T14:56:12.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.280+0000 7f1ceeffd640 1 -- 172.21.15.78:0/3922985604 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v259) v1 ==== 153+0+0 (secure 0 0 0) 0x7f1ce807a7f0 con 0x7f1cf81559c0 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cdc06b810 msgr2=0x7f1cdc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cdc06b810 0x7f1cdc06dcd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009a20 tx=0x7f1cf4009190 comp rx=0 tx=0).stop 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 msgr2=0x7f1cf81de980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 0x7f1cf81de980 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8000c00 tx=0x7f1ce80046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 shutdown_connections 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1cdc06b810 0x7f1cdc06dcd0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf81559c0 0x7f1cf81de980 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8158f80 0x7f1cf81deec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 --2- 172.21.15.78:0/3922985604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c5130 0x7f1cf8154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 >> 172.21.15.78:0/3922985604 conn(0x7f1cf806b4b0 msgr2=0x7f1cf8157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:12.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 shutdown_connections 2024-02-19T14:56:12.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.283+0000 7f1d003d7640 1 -- 172.21.15.78:0/3922985604 wait complete. 2024-02-19T14:56:12.298 INFO:tasks.cephfs_test_runner:That a standby-replay daemon with weak affinity is replaced by a stronger one. ... ok 2024-02-19T14:56:12.298 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) That a sole MDS with mds_join_fs set will come back as active eventually even after failover. 2024-02-19T14:56:12.298 INFO:tasks.cephfs_test_runner:test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:56:12.298 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.397+0000 7feac242a640 1 Processor -- start 2024-02-19T14:56:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.397+0000 7feac242a640 1 -- start start 2024-02-19T14:56:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feac242a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 0x7feabc0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feac242a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc158c40 con 0x7feabc0c55f0 2024-02-19T14:56:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feac242a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0c5a60 con 0x7feabc0bdf20 2024-02-19T14:56:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feac242a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c8de0 con 0x7feabc0ac390 2024-02-19T14:56:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feac09c7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7feabc0c55f0 0x7feabc15f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42336/0 (socket says 172.21.15.78:42336) 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feac09c7640 1 -- 172.21.15.78:0/3431693510 learned_addr learned my addr 172.21.15.78:0/3431693510 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabbfff640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 0x7feabc0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603181457 0 0) 0x7feabc158c40 con 0x7feabc0c55f0 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa4003520 con 0x7feabc0c55f0 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 788356421 0 0) 0x7feaa4003520 con 0x7feabc0c55f0 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 msgr2=0x7feabc0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabaffd640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 0x7feabc0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.398+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 >> v1:172.21.15.150:6789/0 conn(0x7feabc0bdf20 legacy=0x7feabc158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc155a60 con 0x7feabc0c55f0 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7feab40019f0 con 0x7feabc0c55f0 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 >> v1:172.21.15.78:6789/0 conn(0x7feabc0c55f0 legacy=0x7feabc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 0x7feaa40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaa4003f20 0x7feaa40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaa4004d00 0x7feaa4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c8de0 con 0x7feaa4004d00 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa400bbc0 con 0x7feaa4003f20 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa400bd80 con 0x7feabc0ac390 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feac09c7640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaa4004d00 0x7feaa4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabbfff640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 0x7feaa40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feabb7fe640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaa4003f20 0x7feaa40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feac09c7640 1 -- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 msgr2=0x7feaa40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feac09c7640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feabc0ac390 0x7feaa40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feac09c7640 1 -- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaa4003f20 msgr2=0x7feaa40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feac09c7640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaa4003f20 0x7feaa40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.399+0000 7feac09c7640 1 -- 172.21.15.78:0/3431693510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa4003520 con 0x7feaa4004d00 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac09c7640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaa4004d00 0x7feaa4005180 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7feab400a830 tx=0x7feaa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763edef8da47a083 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab40072d0 con 0x7feaa4004d00 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab402f070 con 0x7feaa4004d00 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feabaffd640 1 -- 172.21.15.78:0/3431693510 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab40076f0 con 0x7feaa4004d00 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac242a640 1 -- 172.21.15.78:0/3431693510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaa4004d00 msgr2=0x7feaa4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac242a640 1 --2- 172.21.15.78:0/3431693510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feaa4004d00 0x7feaa4005180 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7feab400a830 tx=0x7feaa40057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac09c7640 1 -- 172.21.15.78:0/3431693510 reap_dead start 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac242a640 1 -- 172.21.15.78:0/3431693510 shutdown_connections 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac242a640 1 -- 172.21.15.78:0/3431693510 >> 172.21.15.78:0/3431693510 conn(0x7feabc0ca890 msgr2=0x7feabc0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac242a640 1 -- 172.21.15.78:0/3431693510 shutdown_connections 2024-02-19T14:56:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.400+0000 7feac242a640 1 -- 172.21.15.78:0/3431693510 wait complete. 2024-02-19T14:56:12.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 Processor -- start 2024-02-19T14:56:12.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 -- start start 2024-02-19T14:56:12.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaa4004d00 0x7feabc0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 0x7feabc0bce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feabc0c1090 0x7feabc0bd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c0560 con 0x7feabc0c33c0 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c4200 con 0x7feabc0c1090 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feac242a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c47c0 con 0x7feaa4004d00 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 0x7feabc0bce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 0x7feabc0bce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45174/0 (socket says 172.21.15.78:45174) 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.402+0000 7feabb7fe640 1 -- 172.21.15.78:0/2070334550 learned_addr learned my addr 172.21.15.78:0/2070334550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabb7fe640 1 -- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaa4004d00 msgr2=0x7feabc0aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabbfff640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaa4004d00 0x7feabc0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feac09c7640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feabc0c1090 0x7feabc0bd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabb7fe640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaa4004d00 0x7feabc0aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabb7fe640 1 -- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feabc0c1090 msgr2=0x7feabc0bd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabb7fe640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feabc0c1090 0x7feabc0bd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabb7fe640 1 -- 172.21.15.78:0/2070334550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc1ee520 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabbfff640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaa4004d00 0x7feabc0aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feac09c7640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feabc0c1090 0x7feabc0bd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feabb7fe640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 0x7feabc0bce00 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7feab000f820 tx=0x7feab000fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab00050d0 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0005230 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab000d6d0 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc1ee800 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.403+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0af190 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.404+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc0c18d0 con 0x7feabc0c33c0 2024-02-19T14:56:12.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.404+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feab000d830 con 0x7feabc0c33c0 2024-02-19T14:56:12.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.405+0000 7feab8ff9640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feaa806b810 0x7feaa806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.405+0000 7feabbfff640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feaa806b810 0x7feaa806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.405+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7feab0089ef0 con 0x7feabc0c33c0 2024-02-19T14:56:12.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.405+0000 7feabbfff640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feaa806b810 0x7feaa806dcd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7feaac009ab0 tx=0x7feaac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:12.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.409+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feab0054ac0 con 0x7feabc0c33c0 2024-02-19T14:56:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.581+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feabc074990 con 0x7feabc0c33c0 2024-02-19T14:56:12.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.582+0000 7feab8ff9640 1 -- 172.21.15.78:0/2070334550 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+13423 (secure 0 0 0) 0x7feab0086480 con 0x7feabc0c33c0 2024-02-19T14:56:12.585 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:12.585 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":115,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:10.270346+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6833/2663291681":"2024-02-20T14:56:07.227904+0000","172.21.15.150:6832/2663291681":"2024-02-20T14:56:07.227904+0000","172.21.15.78:6837/2978708219":"2024-02-20T14:55:50.184610+0000","172.21.15.78:6835/2978708219":"2024-02-20T14:55:50.184610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feaa806b810 msgr2=0x7feaa806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feaa806b810 0x7feaa806dcd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7feaac009ab0 tx=0x7feaac008040 comp rx=0 tx=0).stop 2024-02-19T14:56:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 msgr2=0x7feabc0bce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 0x7feabc0bce00 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7feab000f820 tx=0x7feab000fce0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 shutdown_connections 2024-02-19T14:56:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feaa806b810 0x7feaa806dcd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feabc0c33c0 0x7feabc0bce00 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feabc0c1090 0x7feabc0bd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 --2- 172.21.15.78:0/2070334550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaa4004d00 0x7feabc0aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 >> 172.21.15.78:0/2070334550 conn(0x7feabc0ca890 msgr2=0x7feabc0aa790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 shutdown_connections 2024-02-19T14:56:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.584+0000 7feac242a640 1 -- 172.21.15.78:0/2070334550 wait complete. 2024-02-19T14:56:12.598 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:56:12.598 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:56:12.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.699+0000 7f1db59a2640 1 Processor -- start 2024-02-19T14:56:12.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.699+0000 7f1db59a2640 1 -- start start 2024-02-19T14:56:12.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.699+0000 7f1db59a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1db01526b0 0x7f1db00c8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db59a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1db00c50a0 0x7f1db00c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db59a2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db0155200 con 0x7f1db0158e70 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db59a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db0154ef0 con 0x7f1db01526b0 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db59a2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db0155da0 con 0x7f1db00c50a0 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db49a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1db01526b0 0x7f1db00c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1daffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1db0158e70 0x7f1db0155690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42344/0 (socket says 172.21.15.78:42344) 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1daffff640 1 -- 172.21.15.78:0/2464098642 learned_addr learned my addr 172.21.15.78:0/2464098642 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db51a1640 1 --2- 172.21.15.78:0/2464098642 >> v2:172.21.15.150:3300/0 conn(0x7f1db00c50a0 0x7f1db00c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db49a0640 1 -- 172.21.15.78:0/2464098642 >> v2:172.21.15.150:3300/0 conn(0x7f1db00c50a0 msgr2=0x7f1db00c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db49a0640 1 --2- 172.21.15.78:0/2464098642 >> v2:172.21.15.150:3300/0 conn(0x7f1db00c50a0 0x7f1db00c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db49a0640 1 -- 172.21.15.78:0/2464098642 >> v1:172.21.15.78:6789/0 conn(0x7f1db0158e70 legacy=0x7f1db0155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1daf7fe640 1 -- 172.21.15.78:0/2464098642 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866352648 0 0) 0x7f1db0155200 con 0x7f1db0158e70 2024-02-19T14:56:12.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.700+0000 7f1db49a0640 1 -- 172.21.15.78:0/2464098642 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db00c7240 con 0x7f1db01526b0 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db49a0640 1 --2- 172.21.15.78:0/2464098642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db01526b0 0x7f1db00c8620 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f1db0156000 tx=0x7f1da402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbad3270e0e3355d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1daf7fe640 1 -- 172.21.15.78:0/2464098642 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4031830 con 0x7f1db01526b0 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1daf7fe640 1 -- 172.21.15.78:0/2464098642 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4031990 con 0x7f1db01526b0 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1daf7fe640 1 -- 172.21.15.78:0/2464098642 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4031d20 con 0x7f1db01526b0 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 -- 172.21.15.78:0/2464098642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db01526b0 msgr2=0x7f1db00c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2464098642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db01526b0 0x7f1db00c8620 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f1db0156000 tx=0x7f1da402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 -- 172.21.15.78:0/2464098642 shutdown_connections 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2464098642 >> v2:172.21.15.150:3300/0 conn(0x7f1db00c50a0 0x7f1db00c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2464098642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db01526b0 0x7f1db00c8620 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 -- 172.21.15.78:0/2464098642 >> 172.21.15.78:0/2464098642 conn(0x7f1db00ac290 msgr2=0x7f1db00ac6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:12.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 -- 172.21.15.78:0/2464098642 shutdown_connections 2024-02-19T14:56:12.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.701+0000 7f1db59a2640 1 -- 172.21.15.78:0/2464098642 wait complete. 2024-02-19T14:56:12.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.702+0000 7f1db59a2640 1 Processor -- start 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 -- start start 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 0x7f1db00ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1db01526b0 0x7f1db00c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1db0158e70 0x7f1db0157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0156260 con 0x7f1db00c50a0 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db01564b0 con 0x7f1db01526b0 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db59a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0156790 con 0x7f1db0158e70 2024-02-19T14:56:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db49a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 0x7f1db00ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db49a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 0x7f1db00ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45192/0 (socket says 172.21.15.78:45192) 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.703+0000 7f1db49a0640 1 -- 172.21.15.78:0/2666079407 learned_addr learned my addr 172.21.15.78:0/2666079407 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db49a0640 1 -- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1db0158e70 msgr2=0x7f1db0157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db49a0640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1db0158e70 0x7f1db0157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db49a0640 1 -- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1db01526b0 msgr2=0x7f1db00c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1daffff640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1db01526b0 0x7f1db00c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db49a0640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1db01526b0 0x7f1db00c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db49a0640 1 -- 172.21.15.78:0/2666079407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009400 con 0x7f1db00c50a0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db49a0640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 0x7f1db00ad310 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f1da4000c00 tx=0x7f1da40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da403a8f0 con 0x7f1db00c50a0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da403aa70 con 0x7f1db00c50a0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db0156a70 con 0x7f1db00c50a0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da403ae20 con 0x7f1db00c50a0 2024-02-19T14:56:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.704+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db00afbc0 con 0x7f1db00c50a0 2024-02-19T14:56:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.705+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db0156f90 con 0x7f1db00c50a0 2024-02-19T14:56:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.706+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1da404d450 con 0x7f1db00c50a0 2024-02-19T14:56:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.706+0000 7f1dad7fa640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d9406b840 0x7f1d9406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.706+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1da40abff0 con 0x7f1db00c50a0 2024-02-19T14:56:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.706+0000 7f1daffff640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d9406b840 0x7f1d9406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.706+0000 7f1daffff640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d9406b840 0x7f1d9406dd00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c007fd0 tx=0x7f1d9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:12.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.710+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1db0156f90 con 0x7f1db00c50a0 2024-02-19T14:56:12.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.883+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1db0156f90 con 0x7f1db00c50a0 2024-02-19T14:56:12.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.884+0000 7f1dad7fa640 1 -- 172.21.15.78:0/2666079407 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1db0156f90 con 0x7f1db00c50a0 2024-02-19T14:56:12.887 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:56:12.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d9406b840 msgr2=0x7f1d9406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d9406b840 0x7f1d9406dd00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c007fd0 tx=0x7f1d9c009040 comp rx=0 tx=0).stop 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 msgr2=0x7f1db00ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 0x7f1db00ad310 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f1da4000c00 tx=0x7f1da40046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 shutdown_connections 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d9406b840 0x7f1d9406dd00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1db0158e70 0x7f1db0157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1db01526b0 0x7f1db00c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 --2- 172.21.15.78:0/2666079407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1db00c50a0 0x7f1db00ad310 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 >> 172.21.15.78:0/2666079407 conn(0x7f1db00ac290 msgr2=0x7f1db00ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 shutdown_connections 2024-02-19T14:56:12.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.886+0000 7f1db59a2640 1 -- 172.21.15.78:0/2666079407 wait complete. 2024-02-19T14:56:12.899 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-02-19T14:56:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.996+0000 7f98e3ac3640 1 Processor -- start 2024-02-19T14:56:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.996+0000 7f98e3ac3640 1 -- start start 2024-02-19T14:56:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e3ac3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 0x7f98dc1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e3ac3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 0x7f98dc0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e3ac3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc155ef0 con 0x7f98dc152480 2024-02-19T14:56:13.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e3ac3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98dc06dec0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e3ac3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc06d520 con 0x7f98dc0c7070 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e2060640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 0x7f98dc0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e105e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 0x7f98dc1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e105e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 0x7f98dc1559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:44782/0 (socket says 172.21.15.78:44782) 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e185f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f98dc152480 0x7f98dc06ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:36602/0 (socket says 172.21.15.78:36602) 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e2060640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 0x7f98dc0c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:45490/0 (socket says 172.21.15.78:45490) 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.997+0000 7f98e105e640 1 -- 172.21.15.78:0/2641895143 learned_addr learned my addr 172.21.15.78:0/2641895143 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.998+0000 7f98e105e640 1 -- 172.21.15.78:0/2641895143 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 msgr2=0x7f98dc0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.998+0000 7f98e105e640 1 --2- 172.21.15.78:0/2641895143 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 0x7f98dc0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.998+0000 7f98e105e640 1 -- 172.21.15.78:0/2641895143 >> v1:172.21.15.150:6789/0 conn(0x7f98dc152480 legacy=0x7f98dc06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.998+0000 7f98e105e640 1 -- 172.21.15.78:0/2641895143 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0c75c0 con 0x7f98dc0c7070 2024-02-19T14:56:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.998+0000 7f98e2060640 1 --2- 172.21.15.78:0/2641895143 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 0x7f98dc0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.998+0000 7f98e105e640 1 --2- 172.21.15.78:0/2641895143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 0x7f98dc1559b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009870 tx=0x7f98cc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=640548241d72f53f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e085d640 1 -- 172.21.15.78:0/2641895143 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc030840 con 0x7f98dc0c7070 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e085d640 1 -- 172.21.15.78:0/2641895143 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc0309a0 con 0x7f98dc0c7070 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e085d640 1 -- 172.21.15.78:0/2641895143 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc030d30 con 0x7f98dc0c7070 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 -- 172.21.15.78:0/2641895143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 msgr2=0x7f98dc1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/2641895143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 0x7f98dc1559b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009870 tx=0x7f98cc02b980 comp rx=0 tx=0).stop 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 -- 172.21.15.78:0/2641895143 shutdown_connections 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/2641895143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c7070 0x7f98dc1559b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/2641895143 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c87a0 0x7f98dc0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 -- 172.21.15.78:0/2641895143 >> 172.21.15.78:0/2641895143 conn(0x7f98dc0ca8a0 msgr2=0x7f98dc0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 -- 172.21.15.78:0/2641895143 shutdown_connections 2024-02-19T14:56:13.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:12.999+0000 7f98e3ac3640 1 -- 172.21.15.78:0/2641895143 wait complete. 2024-02-19T14:56:13.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.000+0000 7f98e3ac3640 1 Processor -- start 2024-02-19T14:56:13.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.000+0000 7f98e3ac3640 1 -- start start 2024-02-19T14:56:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e3ac3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc0c7070 0x7f98dc1eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e3ac3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 0x7f98dc0ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e3ac3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152480 0x7f98dc0be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e3ac3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0c1540 con 0x7f98dc0c87a0 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e3ac3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0c0d50 con 0x7f98dc0c7070 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e3ac3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0c0a70 con 0x7f98dc152480 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e105e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 0x7f98dc0ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e105e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 0x7f98dc0ac620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45194/0 (socket says 172.21.15.78:45194) 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e105e640 1 -- 172.21.15.78:0/3050178064 learned_addr learned my addr 172.21.15.78:0/3050178064 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e185f640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc0c7070 0x7f98dc1eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.001+0000 7f98e2060640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152480 0x7f98dc0be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e105e640 1 -- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152480 msgr2=0x7f98dc0be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e105e640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152480 0x7f98dc0be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e105e640 1 -- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc0c7070 msgr2=0x7f98dc1eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e105e640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc0c7070 0x7f98dc1eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e105e640 1 -- 172.21.15.78:0/3050178064 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4009a70 con 0x7f98dc0c87a0 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e105e640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 0x7f98dc0ac620 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc039a90 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc039bf0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc0094d0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc043410 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.002+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc1eadf0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.003+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc155cd0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.003+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f98cc04c3b0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.004+0000 7f98d27fc640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98c006b810 0x7f98c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.004+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f98cc0ac350 con 0x7f98dc0c87a0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.004+0000 7f98e185f640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98c006b810 0x7f98c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.004+0000 7f98e185f640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98c006b810 0x7f98c006dcd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f98d8007fd0 tx=0x7f98d8007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.008+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98cc0770f0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.246+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f98dc074990 con 0x7f98dc0c87a0 2024-02-19T14:56:13.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.281+0000 7f98d27fc640 1 -- 172.21.15.78:0/3050178064 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v260) v1 ==== 148+0+0 (secure 0 0 0) 0x7f98cc07a7b0 con 0x7f98dc0c87a0 2024-02-19T14:56:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98c006b810 msgr2=0x7f98c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98c006b810 0x7f98c006dcd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f98d8007fd0 tx=0x7f98d8007e80 comp rx=0 tx=0).stop 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 msgr2=0x7f98dc0ac620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 0x7f98dc0ac620 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc004730 comp rx=0 tx=0).stop 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 shutdown_connections 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98c006b810 0x7f98c006dcd0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c87a0 0x7f98dc0ac620 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152480 0x7f98dc0be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 --2- 172.21.15.78:0/3050178064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc0c7070 0x7f98dc1eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 >> 172.21.15.78:0/3050178064 conn(0x7f98dc0ca8a0 msgr2=0x7f98dc074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 shutdown_connections 2024-02-19T14:56:13.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.283+0000 7f98e3ac3640 1 -- 172.21.15.78:0/3050178064 wait complete. 2024-02-19T14:56:13.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:56:13.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.396+0000 7f84d06ff640 1 Processor -- start 2024-02-19T14:56:13.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.396+0000 7f84d06ff640 1 -- start start 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84d06ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c80c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84d06ff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84c80c42e0 0x7f84c8158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84d06ff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c80c34a0 con 0x7f84c80c2f50 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84d06ff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c806c090 con 0x7f84c80c42e0 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84d06ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c806d6a0 con 0x7f84c80c8500 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cdc9a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84c80c42e0 0x7f84c8158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cec9c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f84c80c2f50 0x7f84c8158b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42356/0 (socket says 172.21.15.78:42356) 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cec9c640 1 -- 172.21.15.78:0/4061335134 learned_addr learned my addr 172.21.15.78:0/4061335134 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:13.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cdc9a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84c80c42e0 0x7f84c8158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45210/0 (socket says 172.21.15.78:45210) 2024-02-19T14:56:13.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84ce49b640 1 --2- 172.21.15.78:0/4061335134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c80c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cdc9a640 1 -- 172.21.15.78:0/4061335134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 msgr2=0x7f84c80c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:13.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cdc9a640 1 --2- 172.21.15.78:0/4061335134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c80c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cdc9a640 1 -- 172.21.15.78:0/4061335134 >> v1:172.21.15.78:6789/0 conn(0x7f84c80c2f50 legacy=0x7f84c8158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.397+0000 7f84cdc9a640 1 -- 172.21.15.78:0/4061335134 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c806c8e0 con 0x7f84c80c42e0 2024-02-19T14:56:13.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.398+0000 7f84cdc9a640 1 --2- 172.21.15.78:0/4061335134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c42e0 0x7f84c8158650 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f84b80097a0 tx=0x7f84b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a4a4180b907249 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.398+0000 7f84cd499640 1 -- 172.21.15.78:0/4061335134 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b802e070 con 0x7f84c80c42e0 2024-02-19T14:56:13.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.398+0000 7f84cd499640 1 -- 172.21.15.78:0/4061335134 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b80315a0 con 0x7f84c80c42e0 2024-02-19T14:56:13.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.398+0000 7f84cd499640 1 -- 172.21.15.78:0/4061335134 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8031930 con 0x7f84c80c42e0 2024-02-19T14:56:13.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 -- 172.21.15.78:0/4061335134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c42e0 msgr2=0x7f84c8158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 --2- 172.21.15.78:0/4061335134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c42e0 0x7f84c8158650 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f84b80097a0 tx=0x7f84b802b840 comp rx=0 tx=0).stop 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 -- 172.21.15.78:0/4061335134 shutdown_connections 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 --2- 172.21.15.78:0/4061335134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c42e0 0x7f84c8158650 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 --2- 172.21.15.78:0/4061335134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c80c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 -- 172.21.15.78:0/4061335134 >> 172.21.15.78:0/4061335134 conn(0x7f84c80aa300 msgr2=0x7f84c80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 -- 172.21.15.78:0/4061335134 shutdown_connections 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.399+0000 7f84d06ff640 1 -- 172.21.15.78:0/4061335134 wait complete. 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 Processor -- start 2024-02-19T14:56:13.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 -- start start 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 0x7f84c81518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c81e09a0 0x7f84c81eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c81eb0c0 con 0x7f84c80c2f50 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c81de660 con 0x7f84c81e09a0 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.400+0000 7f84d06ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c80acd40 con 0x7f84c80c8500 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 0x7f84c81518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 0x7f84c81518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45226/0 (socket says 172.21.15.78:45226) 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 -- 172.21.15.78:0/3056375763 learned_addr learned my addr 172.21.15.78:0/3056375763 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 -- 172.21.15.78:0/3056375763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 msgr2=0x7f84c814b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 -- 172.21.15.78:0/3056375763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c81e09a0 msgr2=0x7f84c81eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c81e09a0 0x7f84c81eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 -- 172.21.15.78:0/3056375763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b8009400 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84ce49b640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 0x7f84c81518e0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c7f0 tx=0x7f84c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4004220 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4004380 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c80ad020 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007840 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.401+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c80ab3c0 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.402+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c8074990 con 0x7f84c80c2f50 2024-02-19T14:56:13.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.403+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f84c4007bb0 con 0x7f84c80c2f50 2024-02-19T14:56:13.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.403+0000 7f84beffd640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84ac06b840 0x7f84ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.403+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f84c4089fc0 con 0x7f84c80c2f50 2024-02-19T14:56:13.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.403+0000 7f84cdc9a640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84ac06b840 0x7f84ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.404+0000 7f84cdc9a640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84ac06b840 0x7f84ac06dd00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f84b8036000 tx=0x7f84b8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.407+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f84c40bddc0 con 0x7f84c80c2f50 2024-02-19T14:56:13.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.578+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f84c80ae030 con 0x7f84c80c2f50 2024-02-19T14:56:13.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.581+0000 7f84beffd640 1 -- 172.21.15.78:0/3056375763 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v41)=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f84c401e020 con 0x7f84c80c2f50 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84ac06b840 msgr2=0x7f84ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84ac06b840 0x7f84ac06dd00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f84b8036000 tx=0x7f84b8030000 comp rx=0 tx=0).stop 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 msgr2=0x7f84c81518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 0x7f84c81518e0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f84c400c7f0 tx=0x7f84c400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 shutdown_connections 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84ac06b840 0x7f84ac06dd00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84c81e09a0 0x7f84c81eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84c80c8500 0x7f84c814b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 --2- 172.21.15.78:0/3056375763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84c80c2f50 0x7f84c81518e0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 >> 172.21.15.78:0/3056375763 conn(0x7f84c80aa300 msgr2=0x7f84c8153e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:13.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 shutdown_connections 2024-02-19T14:56:13.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.583+0000 7f84d06ff640 1 -- 172.21.15.78:0/3056375763 wait complete. 2024-02-19T14:56:13.596 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:56:13.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:13.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:13.679 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:56:13.679 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:56:13.679 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:13.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:56:13.695 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:56:13.695 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:13.712 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:56:13.712 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:56:13.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:13.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 Processor -- start 2024-02-19T14:56:13.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 -- start start 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06e4152700 0x7f06e40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06e40c5730 0x7f06e406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e41551f0 con 0x7f06e40c8e00 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e4155490 con 0x7f06e4152700 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.832+0000 7f06e9a0e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e4155770 con 0x7f06e40c5730 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06e4152700 0x7f06e40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f06e40c8e00 0x7f06e4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42366/0 (socket says 172.21.15.78:42366) 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e37fe640 1 -- 172.21.15.78:0/3487579373 learned_addr learned my addr 172.21.15.78:0/3487579373 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e27fc640 1 --2- 172.21.15.78:0/3487579373 >> v2:172.21.15.150:3300/0 conn(0x7f06e40c5730 0x7f06e406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e2ffd640 1 -- 172.21.15.78:0/3487579373 >> v2:172.21.15.150:3300/0 conn(0x7f06e40c5730 msgr2=0x7f06e406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e2ffd640 1 --2- 172.21.15.78:0/3487579373 >> v2:172.21.15.150:3300/0 conn(0x7f06e40c5730 0x7f06e406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e2ffd640 1 -- 172.21.15.78:0/3487579373 >> v1:172.21.15.78:6789/0 conn(0x7f06e40c8e00 legacy=0x7f06e4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e1ffb640 1 -- 172.21.15.78:0/3487579373 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82126838 0 0) 0x7f06e41551f0 con 0x7f06e40c8e00 2024-02-19T14:56:13.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.833+0000 7f06e2ffd640 1 -- 172.21.15.78:0/3487579373 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e4155a50 con 0x7f06e4152700 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e2ffd640 1 --2- 172.21.15.78:0/3487579373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e4152700 0x7f06e40c51f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f06e40c7620 tx=0x7f06d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6b5f479ccfcdb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e1ffb640 1 -- 172.21.15.78:0/3487579373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d402e070 con 0x7f06e4152700 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e1ffb640 1 -- 172.21.15.78:0/3487579373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d40315a0 con 0x7f06e4152700 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e1ffb640 1 -- 172.21.15.78:0/3487579373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d4031930 con 0x7f06e4152700 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3487579373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e4152700 msgr2=0x7f06e40c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3487579373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e4152700 0x7f06e40c51f0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f06e40c7620 tx=0x7f06d402b840 comp rx=0 tx=0).stop 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3487579373 shutdown_connections 2024-02-19T14:56:13.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3487579373 >> v2:172.21.15.150:3300/0 conn(0x7f06e40c5730 0x7f06e406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3487579373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e4152700 0x7f06e40c51f0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3487579373 >> 172.21.15.78:0/3487579373 conn(0x7f06e40cac90 msgr2=0x7f06e40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:13.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3487579373 shutdown_connections 2024-02-19T14:56:13.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.834+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3487579373 wait complete. 2024-02-19T14:56:13.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.835+0000 7f06e9a0e640 1 Processor -- start 2024-02-19T14:56:13.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 -- start start 2024-02-19T14:56:13.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e40c5730 0x7f06e41ea710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 0x7f06e40aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e4152700 0x7f06e40aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e40af8f0 con 0x7f06e40c8e00 2024-02-19T14:56:13.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e40acb80 con 0x7f06e40c5730 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e9a0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e40ac370 con 0x7f06e4152700 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 0x7f06e40aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 0x7f06e40aa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45256/0 (socket says 172.21.15.78:45256) 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.836+0000 7f06e27fc640 1 -- 172.21.15.78:0/3743795866 learned_addr learned my addr 172.21.15.78:0/3743795866 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e2ffd640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e40c5730 0x7f06e41ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e27fc640 1 -- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e4152700 msgr2=0x7f06e40aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e37fe640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e4152700 0x7f06e40aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e27fc640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e4152700 0x7f06e40aaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e27fc640 1 -- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e40c5730 msgr2=0x7f06e41ea710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e27fc640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e40c5730 0x7f06e41ea710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e27fc640 1 -- 172.21.15.78:0/3743795866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d4009400 con 0x7f06e40c8e00 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e2ffd640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e40c5730 0x7f06e41ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e37fe640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e4152700 0x7f06e40aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e27fc640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 0x7f06e40aa660 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f06d800e9e0 tx=0x7f06d800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8009670 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d800c970 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e41eb050 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d800cd20 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.837+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e40ac8a0 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.838+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e40be0c0 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.838+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f06d8013470 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.839+0000 7f06cbfff640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06d006b810 0x7f06d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.839+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f06d808a270 con 0x7f06e40c8e00 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.839+0000 7f06e2ffd640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06d006b810 0x7f06d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:13.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.839+0000 7f06e2ffd640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06d006b810 0x7f06d006dcd0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f06d4036000 tx=0x7f06d403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:13.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:13.843+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f06d8054e40 con 0x7f06e40c8e00 2024-02-19T14:56:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.050+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06e4074990 con 0x7f06e40c8e00 2024-02-19T14:56:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.051+0000 7f06cbfff640 1 -- 172.21.15.78:0/3743795866 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f06d8021e90 con 0x7f06e40c8e00 2024-02-19T14:56:14.054 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:14.054 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/570741186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":570741186},{"type":"v1","addr":"172.21.15.150:6835","nonce":570741186}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":6952,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3760210003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3760210003},{"type":"v1","addr":"172.21.15.150:6833","nonce":3760210003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":7302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3062592461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3062592461},{"type":"v1","addr":"172.21.15.78:6836","nonce":3062592461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":7599,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3069973851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3069973851},{"type":"v1","addr":"172.21.15.78:6837","nonce":3069973851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[]} 2024-02-19T14:56:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 96 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06d006b810 msgr2=0x7f06d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06d006b810 0x7f06d006dcd0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f06d4036000 tx=0x7f06d403d040 comp rx=0 tx=0).stop 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 msgr2=0x7f06e40aa660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 0x7f06e40aa660 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f06d800e9e0 tx=0x7f06d800eea0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 shutdown_connections 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f06d006b810 0x7f06d006dcd0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06e40c8e00 0x7f06e40aa660 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e4152700 0x7f06e40aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 --2- 172.21.15.78:0/3743795866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e40c5730 0x7f06e41ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 >> 172.21.15.78:0/3743795866 conn(0x7f06e40cac90 msgr2=0x7f06e4150950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 shutdown_connections 2024-02-19T14:56:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.053+0000 7f06e9a0e640 1 -- 172.21.15.78:0/3743795866 wait complete. 2024-02-19T14:56:14.067 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:56:14.067 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:56:14.068 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:56:14.068 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:14.068 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:56:14.068 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:56:14.069 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:56:14.069 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:14.069 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:56:14.069 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:56:14.070 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:56:14.070 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:14.070 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:56:14.070 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:56:14.071 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:56:14.071 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:14.079 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:56:14.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:56:14.080 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:56:14.080 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:56:14.084 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:56:14.084 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:56:14.085 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:56:14.085 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:56:14.120 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:56:14.121 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:14.121 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:56:14.121 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:14.126 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:56:14.126 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:14.126 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:56:14.127 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:14.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:56:14.196 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:56:14.197 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:56:14.210 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:56:14.211 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:56:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.229+0000 7fbc79b65640 1 Processor -- start 2024-02-19T14:56:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.229+0000 7fbc79b65640 1 -- start start 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc79b65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc740c8e00 0x7fbc7406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc79b65640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc7406d910 0x7fbc7406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc79b65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7406cd20 con 0x7fbc74152700 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc79b65640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc741551f0 con 0x7fbc7406d910 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc79b65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74155490 con 0x7fbc740c8e00 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc7406d910 0x7fbc7406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc7406d910 0x7fbc7406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45294/0 (socket says 172.21.15.78:45294) 2024-02-19T14:56:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 -- 172.21.15.78:0/255743639 learned_addr learned my addr 172.21.15.78:0/255743639 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc72ffd640 1 --2- 172.21.15.78:0/255743639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc740c8e00 0x7fbc7406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 -- 172.21.15.78:0/255743639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc740c8e00 msgr2=0x7fbc7406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 --2- 172.21.15.78:0/255743639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc740c8e00 0x7fbc7406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 -- 172.21.15.78:0/255743639 >> v1:172.21.15.150:6789/0 conn(0x7fbc74152700 legacy=0x7fbc740c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.230+0000 7fbc73fff640 1 -- 172.21.15.78:0/255743639 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74155770 con 0x7fbc7406d910 2024-02-19T14:56:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.231+0000 7fbc73fff640 1 --2- 172.21.15.78:0/255743639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc7406c7e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009e50 tx=0x7fbc6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b47f81fb6cb564 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.231+0000 7fbc727fc640 1 -- 172.21.15.78:0/255743639 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c02ca70 con 0x7fbc7406d910 2024-02-19T14:56:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.231+0000 7fbc727fc640 1 -- 172.21.15.78:0/255743639 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c02cbd0 con 0x7fbc7406d910 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.231+0000 7fbc727fc640 1 -- 172.21.15.78:0/255743639 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0343f0 con 0x7fbc7406d910 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 -- 172.21.15.78:0/255743639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 msgr2=0x7fbc7406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 --2- 172.21.15.78:0/255743639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc7406c7e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009e50 tx=0x7fbc6c02bed0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 -- 172.21.15.78:0/255743639 shutdown_connections 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 --2- 172.21.15.78:0/255743639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc7406c7e0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 --2- 172.21.15.78:0/255743639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc740c8e00 0x7fbc7406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 -- 172.21.15.78:0/255743639 >> 172.21.15.78:0/255743639 conn(0x7fbc740cac90 msgr2=0x7fbc740cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 -- 172.21.15.78:0/255743639 shutdown_connections 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.232+0000 7fbc79b65640 1 -- 172.21.15.78:0/255743639 wait complete. 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.233+0000 7fbc79b65640 1 Processor -- start 2024-02-19T14:56:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.233+0000 7fbc79b65640 1 -- start start 2024-02-19T14:56:14.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc741ee070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:14.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc740c8e00 0x7fbc74148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:14.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc74152700 0x7fbc74149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc741eeb80 con 0x7fbc7406d910 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74156f00 con 0x7fbc740c8e00 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740aae60 con 0x7fbc74152700 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc741ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc741ee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45302/0 (socket says 172.21.15.78:45302) 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 -- 172.21.15.78:0/4129731484 learned_addr learned my addr 172.21.15.78:0/4129731484 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc73fff640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc74152700 0x7fbc74149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 -- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc74152700 msgr2=0x7fbc74149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc74152700 0x7fbc74149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc72ffd640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc740c8e00 0x7fbc74148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 -- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc740c8e00 msgr2=0x7fbc74148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc740c8e00 0x7fbc74148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 -- 172.21.15.78:0/4129731484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c009b30 con 0x7fbc7406d910 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc73fff640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc74152700 0x7fbc74149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc737fe640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc741ee070 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c3f0 tx=0x7fbc6400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc640052d0 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64005450 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc740ab140 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64007260 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.234+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc740ab700 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.235+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc740c4c70 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.236+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbc640073c0 con 0x7fbc7406d910 2024-02-19T14:56:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.236+0000 7fbc6bfff640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc5406b810 0x7fbc5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:14.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.236+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fbc640893d0 con 0x7fbc7406d910 2024-02-19T14:56:14.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.236+0000 7fbc72ffd640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc5406b810 0x7fbc5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:14.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.237+0000 7fbc72ffd640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc5406b810 0x7fbc5406dcd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009a10 tx=0x7fbc5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:14.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.241+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbc64053fa0 con 0x7fbc7406d910 2024-02-19T14:56:14.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:14.429+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbc74074990 con 0x7fbc7406d910 2024-02-19T14:56:15.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.312+0000 7fbc6bfff640 1 -- 172.21.15.78:0/4129731484 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v120) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbc64057660 con 0x7fbc7406d910 2024-02-19T14:56:15.315 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc5406b810 msgr2=0x7fbc5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc5406b810 0x7fbc5406dcd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009a10 tx=0x7fbc5c008040 comp rx=0 tx=0).stop 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 msgr2=0x7fbc741ee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc741ee070 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c3f0 tx=0x7fbc6400c8b0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 shutdown_connections 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc5406b810 0x7fbc5406dcd0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc74152700 0x7fbc74149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc740c8e00 0x7fbc74148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 --2- 172.21.15.78:0/4129731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc7406d910 0x7fbc741ee070 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 >> 172.21.15.78:0/4129731484 conn(0x7fbc740cac90 msgr2=0x7fbc7406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 shutdown_connections 2024-02-19T14:56:15.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.315+0000 7fbc79b65640 1 -- 172.21.15.78:0/4129731484 wait complete. 2024-02-19T14:56:15.331 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:56:15.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.433+0000 7fd9877e2640 1 Processor -- start 2024-02-19T14:56:15.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.433+0000 7fd9877e2640 1 -- start start 2024-02-19T14:56:15.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd9877e2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9801528d0 0x7fd9800c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd9877e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 0x7fd980155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd9877e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd980072e80 con 0x7fd980072180 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd9877e2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9800734c0 con 0x7fd9801528d0 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd9877e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98006e990 con 0x7fd9800c19c0 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd985d7f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd980072180 0x7fd98006ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42390/0 (socket says 172.21.15.78:42390) 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd985d7f640 1 -- 172.21.15.78:0/4164979746 learned_addr learned my addr 172.21.15.78:0/4164979746 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd984d7d640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 0x7fd980155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.434+0000 7fd98557e640 1 --2- 172.21.15.78:0/4164979746 >> v2:172.21.15.150:3300/0 conn(0x7fd9801528d0 0x7fd9800c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039036733 0 0) 0x7fd978001110 con 0x7fd980072180 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96c003520 con 0x7fd980072180 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132691061 0 0) 0x7fd96c003520 con 0x7fd980072180 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 msgr2=0x7fd980155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:15.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 0x7fd980155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 >> v2:172.21.15.150:3300/0 conn(0x7fd9801528d0 msgr2=0x7fd9800c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 --2- 172.21.15.78:0/4164979746 >> v2:172.21.15.150:3300/0 conn(0x7fd9801528d0 0x7fd9800c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9800731e0 con 0x7fd980072180 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fd978001800 con 0x7fd980072180 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 >> v1:172.21.15.78:6789/0 conn(0x7fd980072180 legacy=0x7fd98006ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 0x7fd96c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96c003ef0 0x7fd96c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c004d00 0x7fd96c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800734c0 con 0x7fd96c004d00 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98006e990 con 0x7fd96c003ef0 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c00b9d0 con 0x7fd9800c19c0 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd985d7f640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c004d00 0x7fd96c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd98557e640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 0x7fd96c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd984d7d640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96c003ef0 0x7fd96c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd985d7f640 1 -- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 msgr2=0x7fd96c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd985d7f640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9800c19c0 0x7fd96c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd985d7f640 1 -- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96c003ef0 msgr2=0x7fd96c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd985d7f640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96c003ef0 0x7fd96c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.435+0000 7fd985d7f640 1 -- 172.21.15.78:0/4164979746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c003520 con 0x7fd96c004d00 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.436+0000 7fd985d7f640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c004d00 0x7fd96c005180 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd97800c240 tx=0x7fd96c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b372a6578a68c2f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.436+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978005ad0 con 0x7fd96c004d00 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.436+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978005c30 con 0x7fd96c004d00 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.436+0000 7fd977fff640 1 -- 172.21.15.78:0/4164979746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97802c8b0 con 0x7fd96c004d00 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd9877e2640 1 -- 172.21.15.78:0/4164979746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c004d00 msgr2=0x7fd96c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd9877e2640 1 --2- 172.21.15.78:0/4164979746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd96c004d00 0x7fd96c005180 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd97800c240 tx=0x7fd96c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd985d7f640 1 -- 172.21.15.78:0/4164979746 reap_dead start 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd9877e2640 1 -- 172.21.15.78:0/4164979746 shutdown_connections 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd9877e2640 1 -- 172.21.15.78:0/4164979746 >> 172.21.15.78:0/4164979746 conn(0x7fd9800c7590 msgr2=0x7fd9800c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd9877e2640 1 -- 172.21.15.78:0/4164979746 shutdown_connections 2024-02-19T14:56:15.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.437+0000 7fd9877e2640 1 -- 172.21.15.78:0/4164979746 wait complete. 2024-02-19T14:56:15.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.438+0000 7fd9877e2640 1 Processor -- start 2024-02-19T14:56:15.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.438+0000 7fd9877e2640 1 -- start start 2024-02-19T14:56:15.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 0x7fd9800a7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9801ee1f0 0x7fd9801ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd98014ad50 0x7fd98014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9801eebb0 con 0x7fd98014cd70 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800a8cf0 con 0x7fd9801ee1f0 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800a7a10 con 0x7fd98014ad50 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 0x7fd9800a7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 0x7fd9800a7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45312/0 (socket says 172.21.15.78:45312) 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 -- 172.21.15.78:0/630541252 learned_addr learned my addr 172.21.15.78:0/630541252 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 -- 172.21.15.78:0/630541252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd98014ad50 msgr2=0x7fd98014b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd98014ad50 0x7fd98014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 -- 172.21.15.78:0/630541252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9801ee1f0 msgr2=0x7fd9801ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9801ee1f0 0x7fd9801ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 -- 172.21.15.78:0/630541252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd980154250 con 0x7fd98014cd70 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd98557e640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 0x7fd9800a7c70 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011e30 tx=0x7fd97c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:15.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00e040 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c00f040 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9800a7450 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.439+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c009c00 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.440+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd980155e80 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.440+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd980152cb0 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.441+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd97c015850 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.441+0000 7fd975ffb640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd96406b810 0x7fd96406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.441+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd97c08aea0 con 0x7fd98014cd70 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.441+0000 7fd984d7d640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd96406b810 0x7fd96406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.442+0000 7fd984d7d640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd96406b810 0x7fd96406dcd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd98014e6e0 tx=0x7fd970009320 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:15.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.446+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd97c055b10 con 0x7fd98014cd70 2024-02-19T14:56:15.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.682+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd980074990 con 0x7fd98014cd70 2024-02-19T14:56:15.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.683+0000 7fd975ffb640 1 -- 172.21.15.78:0/630541252 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fd97c0591d0 con 0x7fd98014cd70 2024-02-19T14:56:15.685 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:15.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:56:15.694 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:15.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:15.696 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:56:15.697 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd96406b810 msgr2=0x7fd96406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd96406b810 0x7fd96406dcd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd98014e6e0 tx=0x7fd970009320 comp rx=0 tx=0).stop 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 msgr2=0x7fd9800a7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 0x7fd9800a7c70 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011e30 tx=0x7fd97c00f580 comp rx=0 tx=0).stop 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 shutdown_connections 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd96406b810 0x7fd96406dcd0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd98014ad50 0x7fd98014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9801ee1f0 0x7fd9801ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 --2- 172.21.15.78:0/630541252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd98014cd70 0x7fd9800a7c70 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 >> 172.21.15.78:0/630541252 conn(0x7fd9800c7590 msgr2=0x7fd9800c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:15.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 shutdown_connections 2024-02-19T14:56:15.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.685+0000 7fd9877e2640 1 -- 172.21.15.78:0/630541252 wait complete. 2024-02-19T14:56:15.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:15.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.797+0000 7f4db6c1c640 1 Processor -- start 2024-02-19T14:56:15.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.797+0000 7f4db6c1c640 1 -- start start 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db6c1c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4db006ed20 0x7f4db006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db6c1c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db006d750 0x7f4db015d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db6c1c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0150820 con 0x7f4db0150440 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db6c1c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db00c1b60 con 0x7f4db006ed20 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db6c1c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db01537d0 con 0x7f4db006d750 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4daffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4db006ed20 0x7f4db006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db49b8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4db0150440 0x7f4db00c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42402/0 (socket says 172.21.15.78:42402) 2024-02-19T14:56:15.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db49b8640 1 -- 172.21.15.78:0/1047082386 learned_addr learned my addr 172.21.15.78:0/1047082386 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4db51b9640 1 --2- 172.21.15.78:0/1047082386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db006d750 0x7f4db015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.798+0000 7f4daf7fe640 1 -- 172.21.15.78:0/1047082386 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946815830 0 0) 0x7f4db0150820 con 0x7f4db0150440 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daf7fe640 1 -- 172.21.15.78:0/1047082386 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d98003520 con 0x7f4db0150440 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daffff640 1 -- 172.21.15.78:0/1047082386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db006d750 msgr2=0x7f4db015d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daffff640 1 --2- 172.21.15.78:0/1047082386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db006d750 0x7f4db015d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daffff640 1 -- 172.21.15.78:0/1047082386 >> v1:172.21.15.78:6789/0 conn(0x7f4db0150440 legacy=0x7f4db00c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daffff640 1 -- 172.21.15.78:0/1047082386 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0153d10 con 0x7f4db006ed20 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daffff640 1 --2- 172.21.15.78:0/1047082386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006ed20 0x7f4db006d210 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f4db0153f00 tx=0x7f4da402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e4b341d3d6fb7f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daf7fe640 1 -- 172.21.15.78:0/1047082386 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4031890 con 0x7f4db006ed20 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.799+0000 7f4daf7fe640 1 -- 172.21.15.78:0/1047082386 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40319f0 con 0x7f4db006ed20 2024-02-19T14:56:15.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4daf7fe640 1 -- 172.21.15.78:0/1047082386 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4031d80 con 0x7f4db006ed20 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 -- 172.21.15.78:0/1047082386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006ed20 msgr2=0x7f4db006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/1047082386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006ed20 0x7f4db006d210 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f4db0153f00 tx=0x7f4da402b930 comp rx=0 tx=0).stop 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 -- 172.21.15.78:0/1047082386 shutdown_connections 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/1047082386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006ed20 0x7f4db006d210 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/1047082386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db006d750 0x7f4db015d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 -- 172.21.15.78:0/1047082386 >> 172.21.15.78:0/1047082386 conn(0x7f4db006b270 msgr2=0x7f4db006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 -- 172.21.15.78:0/1047082386 shutdown_connections 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.800+0000 7f4db6c1c640 1 -- 172.21.15.78:0/1047082386 wait complete. 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.801+0000 7f4db6c1c640 1 Processor -- start 2024-02-19T14:56:15.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.801+0000 7f4db6c1c640 1 -- start start 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db6c1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 0x7f4db01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db6c1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db006ed20 0x7f4db01513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db6c1c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db0150440 0x7f4db01568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db6c1c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0155060 con 0x7f4db006d750 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db6c1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0155760 con 0x7f4db006ed20 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db6c1c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0154bd0 con 0x7f4db0150440 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 0x7f4db01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 0x7f4db01521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45332/0 (socket says 172.21.15.78:45332) 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 -- 172.21.15.78:0/176568109 learned_addr learned my addr 172.21.15.78:0/176568109 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db51b9640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db0150440 0x7f4db01568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4daffff640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db006ed20 0x7f4db01513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 -- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db0150440 msgr2=0x7f4db01568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db0150440 0x7f4db01568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 -- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db006ed20 msgr2=0x7f4db01513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db006ed20 0x7f4db01513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db49b8640 1 -- 172.21.15.78:0/176568109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da4009400 con 0x7f4db006d750 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4daffff640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db006ed20 0x7f4db01513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.802+0000 7f4db51b9640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db0150440 0x7f4db01568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.803+0000 7f4db49b8640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 0x7f4db01521a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f4da000c3c0 tx=0x7f4da000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.803+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0005ce0 con 0x7f4db006d750 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.803+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da0005e40 con 0x7f4db006d750 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.803+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db01e7420 con 0x7f4db006d750 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.803+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da00072f0 con 0x7f4db006d750 2024-02-19T14:56:15.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.803+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db01e76e0 con 0x7f4db006d750 2024-02-19T14:56:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.804+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db01e7270 con 0x7f4db006d750 2024-02-19T14:56:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.804+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4da0007450 con 0x7f4db006d750 2024-02-19T14:56:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.804+0000 7f4dad7fa640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9c06b810 0x7f4d9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.804+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4da00895b0 con 0x7f4db006d750 2024-02-19T14:56:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.805+0000 7f4daffff640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9c06b810 0x7f4d9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:15.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.805+0000 7f4daffff640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9c06b810 0x7f4d9c06dcd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4da4036000 tx=0x7f4da403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:15.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.809+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4da00541a0 con 0x7f4db006d750 2024-02-19T14:56:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.982+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db00c0b70 con 0x7f4db006d750 2024-02-19T14:56:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.983+0000 7f4dad7fa640 1 -- 172.21.15.78:0/176568109 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f4da0057860 con 0x7f4db006d750 2024-02-19T14:56:15.986 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:15.986 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":120,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:15.311510+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9c06b810 msgr2=0x7f4d9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9c06b810 0x7f4d9c06dcd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4da4036000 tx=0x7f4da403c040 comp rx=0 tx=0).stop 2024-02-19T14:56:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 msgr2=0x7f4db01521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 0x7f4db01521a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f4da000c3c0 tx=0x7f4da000c880 comp rx=0 tx=0).stop 2024-02-19T14:56:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 shutdown_connections 2024-02-19T14:56:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9c06b810 0x7f4d9c06dcd0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db0150440 0x7f4db01568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db006ed20 0x7f4db01513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 --2- 172.21.15.78:0/176568109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db006d750 0x7f4db01521a0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 >> 172.21.15.78:0/176568109 conn(0x7f4db006b270 msgr2=0x7f4db006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 shutdown_connections 2024-02-19T14:56:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:15.985+0000 7f4db6c1c640 1 -- 172.21.15.78:0/176568109 wait complete. 2024-02-19T14:56:15.999 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:56:15.999 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:56:16.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.099+0000 7f586fbd7640 1 Processor -- start 2024-02-19T14:56:16.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 -- start start 2024-02-19T14:56:16.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f58680add70 0x7f58680c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680c2f60 0x7f586806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f586806f530 0x7f58680705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58680ae670 con 0x7f586806f530 2024-02-19T14:56:16.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58680716b0 con 0x7f58680add70 2024-02-19T14:56:16.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586fbd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58680c33a0 con 0x7f58680c2f60 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586e174640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f586806f530 0x7f58680705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586e174640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f586806f530 0x7f58680705b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45344/0 (socket says 172.21.15.78:45344) 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.100+0000 7f586e174640 1 -- 172.21.15.78:0/2138090299 learned_addr learned my addr 172.21.15.78:0/2138090299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586d172640 1 --2- 172.21.15.78:0/2138090299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680c2f60 0x7f586806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586d973640 1 --2- 172.21.15.78:0/2138090299 >> v2:172.21.15.150:3300/0 conn(0x7f58680add70 0x7f58680c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586e174640 1 -- 172.21.15.78:0/2138090299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680c2f60 msgr2=0x7f586806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586e174640 1 --2- 172.21.15.78:0/2138090299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680c2f60 0x7f586806eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586e174640 1 -- 172.21.15.78:0/2138090299 >> v2:172.21.15.150:3300/0 conn(0x7f58680add70 msgr2=0x7f58680c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586e174640 1 --2- 172.21.15.78:0/2138090299 >> v2:172.21.15.150:3300/0 conn(0x7f58680add70 0x7f58680c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586e174640 1 -- 172.21.15.78:0/2138090299 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5868153780 con 0x7f586806f530 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.101+0000 7f586e174640 1 --2- 172.21.15.78:0/2138090299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f58680705b0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f58680c4fc0 tx=0x7f586002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4d60c312e41ac6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586c971640 1 -- 172.21.15.78:0/2138090299 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586002c9e0 con 0x7f586806f530 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586c971640 1 -- 172.21.15.78:0/2138090299 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586002cb40 con 0x7f586806f530 2024-02-19T14:56:16.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586c971640 1 -- 172.21.15.78:0/2138090299 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600343f0 con 0x7f586806f530 2024-02-19T14:56:16.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 -- 172.21.15.78:0/2138090299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 msgr2=0x7f58680705b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 --2- 172.21.15.78:0/2138090299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f58680705b0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f58680c4fc0 tx=0x7f586002be40 comp rx=0 tx=0).stop 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 -- 172.21.15.78:0/2138090299 shutdown_connections 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 --2- 172.21.15.78:0/2138090299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f58680705b0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 --2- 172.21.15.78:0/2138090299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680c2f60 0x7f586806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 --2- 172.21.15.78:0/2138090299 >> v2:172.21.15.150:3300/0 conn(0x7f58680add70 0x7f58680c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 -- 172.21.15.78:0/2138090299 >> 172.21.15.78:0/2138090299 conn(0x7f58680c85e0 msgr2=0x7f58680c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 -- 172.21.15.78:0/2138090299 shutdown_connections 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.102+0000 7f586fbd7640 1 -- 172.21.15.78:0/2138090299 wait complete. 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.103+0000 7f586fbd7640 1 Processor -- start 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.103+0000 7f586fbd7640 1 -- start start 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586fbd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f5868150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586fbd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680add70 0x7f58681eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586fbd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58680c2f60 0x7f58681ebdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586fbd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868154db0 con 0x7f586806f530 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586fbd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868154640 con 0x7f58680c2f60 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586fbd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5868152a30 con 0x7f58680add70 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f5868150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f5868150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45352/0 (socket says 172.21.15.78:45352) 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 -- 172.21.15.78:0/1984038884 learned_addr learned my addr 172.21.15.78:0/1984038884 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d172640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680add70 0x7f58681eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586e174640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58680c2f60 0x7f58681ebdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 -- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680add70 msgr2=0x7f58681eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680add70 0x7f58681eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 -- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58680c2f60 msgr2=0x7f58681ebdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58680c2f60 0x7f58681ebdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d973640 1 -- 172.21.15.78:0/1984038884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5860009b30 con 0x7f586806f530 2024-02-19T14:56:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586d172640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680add70 0x7f58681eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.104+0000 7f586e174640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58680c2f60 0x7f58681ebdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.105+0000 7f586d973640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f5868150d40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f586400d620 tx=0x7f586400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.105+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586400b620 con 0x7f586806f530 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.105+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586400b780 con 0x7f586806f530 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.105+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586814e300 con 0x7f586806f530 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.105+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586400bb30 con 0x7f586806f530 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.105+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5868151830 con 0x7f586806f530 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.106+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868154c00 con 0x7f586806f530 2024-02-19T14:56:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.106+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f586400bc90 con 0x7f586806f530 2024-02-19T14:56:16.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.107+0000 7f585e7fc640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f584c06b810 0x7f584c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.107+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f586408a500 con 0x7f586806f530 2024-02-19T14:56:16.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.107+0000 7f586d172640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f584c06b810 0x7f584c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.107+0000 7f586d172640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f584c06b810 0x7f584c06dcd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5868152960 tx=0x7f5858008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.111+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f58640550e0 con 0x7f586806f530 2024-02-19T14:56:16.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.281+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58680676a0 con 0x7f586806f530 2024-02-19T14:56:16.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.281+0000 7f585e7fc640 1 -- 172.21.15.78:0/1984038884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58640587a0 con 0x7f586806f530 2024-02-19T14:56:16.284 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f584c06b810 msgr2=0x7f584c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f584c06b810 0x7f584c06dcd0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5868152960 tx=0x7f5858008040 comp rx=0 tx=0).stop 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 msgr2=0x7f5868150d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f5868150d40 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f586400d620 tx=0x7f586400dae0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 shutdown_connections 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f584c06b810 0x7f584c06dcd0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58680c2f60 0x7f58681ebdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58680add70 0x7f58681eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 --2- 172.21.15.78:0/1984038884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f586806f530 0x7f5868150d40 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 >> 172.21.15.78:0/1984038884 conn(0x7f58680c85e0 msgr2=0x7f58680a81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 shutdown_connections 2024-02-19T14:56:16.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.283+0000 7f586fbd7640 1 -- 172.21.15.78:0/1984038884 wait complete. 2024-02-19T14:56:16.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:56:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.394+0000 7f5c9b40e640 1 Processor -- start 2024-02-19T14:56:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.394+0000 7f5c9b40e640 1 -- start start 2024-02-19T14:56:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c9b40e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c940c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c9b40e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c940ab600 0x7f5c940c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c9b40e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c94152d30 con 0x7f5c940ad890 2024-02-19T14:56:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c9b40e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c940aba90 con 0x7f5c940ab600 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c9b40e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940c7860 con 0x7f5c940c49c0 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c991aa640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5c940ad890 0x7f5c941534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42412/0 (socket says 172.21.15.78:42412) 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c991aa640 1 -- 172.21.15.78:0/2982119636 learned_addr learned my addr 172.21.15.78:0/2982119636 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c999ab640 1 --2- 172.21.15.78:0/2982119636 >> v2:172.21.15.78:3300/0 conn(0x7f5c940ab600 0x7f5c940c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c989a9640 1 --2- 172.21.15.78:0/2982119636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c940c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c8bfff640 1 -- 172.21.15.78:0/2982119636 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427625264 0 0) 0x7f5c94152d30 con 0x7f5c940ad890 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c8bfff640 1 -- 172.21.15.78:0/2982119636 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80003520 con 0x7f5c940ad890 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c999ab640 1 -- 172.21.15.78:0/2982119636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 msgr2=0x7f5c940c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c999ab640 1 --2- 172.21.15.78:0/2982119636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c940c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c999ab640 1 -- 172.21.15.78:0/2982119636 >> v1:172.21.15.78:6789/0 conn(0x7f5c940ad890 legacy=0x7f5c941534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.395+0000 7f5c999ab640 1 -- 172.21.15.78:0/2982119636 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c940c7ce0 con 0x7f5c940ab600 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.396+0000 7f5c999ab640 1 --2- 172.21.15.78:0/2982119636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ab600 0x7f5c940c5000 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009d90 tx=0x7f5c8c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384992a6eb50964f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.396+0000 7f5c8bfff640 1 -- 172.21.15.78:0/2982119636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c02cb30 con 0x7f5c940ab600 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.396+0000 7f5c8bfff640 1 -- 172.21.15.78:0/2982119636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c02cc90 con 0x7f5c940ab600 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.396+0000 7f5c8bfff640 1 -- 172.21.15.78:0/2982119636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c034440 con 0x7f5c940ab600 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 -- 172.21.15.78:0/2982119636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ab600 msgr2=0x7f5c940c5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/2982119636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ab600 0x7f5c940c5000 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009d90 tx=0x7f5c8c02bd80 comp rx=0 tx=0).stop 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 -- 172.21.15.78:0/2982119636 shutdown_connections 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/2982119636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ab600 0x7f5c940c5000 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/2982119636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c940c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 -- 172.21.15.78:0/2982119636 >> 172.21.15.78:0/2982119636 conn(0x7f5c9406b700 msgr2=0x7f5c940aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 -- 172.21.15.78:0/2982119636 shutdown_connections 2024-02-19T14:56:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.397+0000 7f5c9b40e640 1 -- 172.21.15.78:0/2982119636 wait complete. 2024-02-19T14:56:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.398+0000 7f5c9b40e640 1 Processor -- start 2024-02-19T14:56:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.398+0000 7f5c9b40e640 1 -- start start 2024-02-19T14:56:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940ab600 0x7f5c94155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 0x7f5c940c1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c94154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941ea8a0 con 0x7f5c940ad890 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940c2400 con 0x7f5c940ab600 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941eaaf0 con 0x7f5c940c49c0 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 0x7f5c940c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 0x7f5c940c1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45366/0 (socket says 172.21.15.78:45366) 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 -- 172.21.15.78:0/3393083993 learned_addr learned my addr 172.21.15.78:0/3393083993 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c999ab640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c94154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 -- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 msgr2=0x7f5c94154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c94154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 -- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940ab600 msgr2=0x7f5c94155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940ab600 0x7f5c94155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 -- 172.21.15.78:0/3393083993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c009a70 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c989a9640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 0x7f5c940c1980 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400d800 tx=0x7f5c8400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400b6c0 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8400b820 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c941eadd0 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.399+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400bbb0 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.400+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c94155d90 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.401+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c94074990 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.401+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5c8400bd10 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.401+0000 7f5c89ffb640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c7806b810 0x7f5c7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.401+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5c8408a4e0 con 0x7f5c940ad890 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.401+0000 7f5c991aa640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c7806b810 0x7f5c7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.402+0000 7f5c991aa640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c7806b810 0x7f5c7806dcd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5c90000f60 tx=0x7f5c900078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.405+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5c84055150 con 0x7f5c940ad890 2024-02-19T14:56:16.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.578+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5c940c7ac0 con 0x7f5c940ad890 2024-02-19T14:56:16.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.578+0000 7f5c89ffb640 1 -- 172.21.15.78:0/3393083993 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v120) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5c84058810 con 0x7f5c940ad890 2024-02-19T14:56:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:16.581 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:56:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:56:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:56:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:16.582 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:56:16.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c7806b810 msgr2=0x7f5c7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c7806b810 0x7f5c7806dcd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5c90000f60 tx=0x7f5c900078f0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 msgr2=0x7f5c940c1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 0x7f5c940c1980 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400d800 tx=0x7f5c8400dcc0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 shutdown_connections 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c7806b810 0x7f5c7806dcd0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ad890 0x7f5c940c1980 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c940c49c0 0x7f5c94154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 --2- 172.21.15.78:0/3393083993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940ab600 0x7f5c94155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 >> 172.21.15.78:0/3393083993 conn(0x7f5c9406b700 msgr2=0x7f5c94151590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 shutdown_connections 2024-02-19T14:56:16.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.580+0000 7f5c9b40e640 1 -- 172.21.15.78:0/3393083993 wait complete. 2024-02-19T14:56:16.594 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:56:16.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:56:16.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.695+0000 7f864aeae640 1 Processor -- start 2024-02-19T14:56:16.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.695+0000 7f864aeae640 1 -- start start 2024-02-19T14:56:16.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864aeae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8644152720 0x7f86440bd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864aeae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440bdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864aeae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8644158520 0x7f8644158990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864aeae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864406fa90 con 0x7f8644158520 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864aeae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86440c8560 con 0x7f8644152720 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864aeae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8644155ea0 con 0x7f8644070ff0 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8644158520 0x7f8644158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8644158520 0x7f8644158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45370/0 (socket says 172.21.15.78:45370) 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 -- 172.21.15.78:0/2577367875 learned_addr learned my addr 172.21.15.78:0/2577367875 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f8643fff640 1 --2- 172.21.15.78:0/2577367875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440bdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f8648c4a640 1 --2- 172.21.15.78:0/2577367875 >> v2:172.21.15.150:3300/0 conn(0x7f8644152720 0x7f86440bd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 -- 172.21.15.78:0/2577367875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 msgr2=0x7f86440bdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 --2- 172.21.15.78:0/2577367875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440bdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 -- 172.21.15.78:0/2577367875 >> v2:172.21.15.150:3300/0 conn(0x7f8644152720 msgr2=0x7f86440bd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 --2- 172.21.15.78:0/2577367875 >> v2:172.21.15.150:3300/0 conn(0x7f8644152720 0x7f86440bd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.696+0000 7f864944b640 1 -- 172.21.15.78:0/2577367875 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86440c80d0 con 0x7f8644158520 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.697+0000 7f864944b640 1 --2- 172.21.15.78:0/2577367875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644158520 0x7f8644158990 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f86440c9800 tx=0x7f863c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ededf4148ab2d19e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.697+0000 7f86437fe640 1 -- 172.21.15.78:0/2577367875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c02c9e0 con 0x7f8644158520 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.697+0000 7f86437fe640 1 -- 172.21.15.78:0/2577367875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c02cb40 con 0x7f8644158520 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.697+0000 7f86437fe640 1 -- 172.21.15.78:0/2577367875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c0343f0 con 0x7f8644158520 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 -- 172.21.15.78:0/2577367875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644158520 msgr2=0x7f8644158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 --2- 172.21.15.78:0/2577367875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644158520 0x7f8644158990 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f86440c9800 tx=0x7f863c02be40 comp rx=0 tx=0).stop 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 -- 172.21.15.78:0/2577367875 shutdown_connections 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 --2- 172.21.15.78:0/2577367875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644158520 0x7f8644158990 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 --2- 172.21.15.78:0/2577367875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440bdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 --2- 172.21.15.78:0/2577367875 >> v2:172.21.15.150:3300/0 conn(0x7f8644152720 0x7f86440bd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 -- 172.21.15.78:0/2577367875 >> 172.21.15.78:0/2577367875 conn(0x7f86440aa370 msgr2=0x7f86440aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:16.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 -- 172.21.15.78:0/2577367875 shutdown_connections 2024-02-19T14:56:16.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.698+0000 7f864aeae640 1 -- 172.21.15.78:0/2577367875 wait complete. 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.699+0000 7f864aeae640 1 Processor -- start 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.699+0000 7f864aeae640 1 -- start start 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864aeae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864aeae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 0x7f864414acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864aeae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8644158520 0x7f864414b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864aeae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864406dc50 con 0x7f8644152720 2024-02-19T14:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864aeae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864414bc00 con 0x7f8644158520 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864aeae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86440c34e0 con 0x7f8644070ff0 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 0x7f864414acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 0x7f864414acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45372/0 (socket says 172.21.15.78:45372) 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 -- 172.21.15.78:0/2872661303 learned_addr learned my addr 172.21.15.78:0/2872661303 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8648c4a640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f864944b640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8644158520 0x7f864414b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 -- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 msgr2=0x7f86440a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 -- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8644158520 msgr2=0x7f864414b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8644158520 0x7f864414b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 -- 172.21.15.78:0/2872661303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c009b30 con 0x7f8644152720 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8648c4a640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.700+0000 7f8643fff640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 0x7f864414acf0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f863800d7d0 tx=0x7f863800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.701+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863800b6c0 con 0x7f8644152720 2024-02-19T14:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.701+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863800b840 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.701+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86440c3200 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.701+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863800bbf0 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.701+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8644154e90 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.702+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644074990 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.702+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f863800bd50 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.702+0000 7f86417fa640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863006b810 0x7f863006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.703+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f863808a4c0 con 0x7f8644152720 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.703+0000 7f8648c4a640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863006b810 0x7f863006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.703+0000 7f8648c4a640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863006b810 0x7f863006dcd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8634006fd0 tx=0x7f8634008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.707+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8638055130 con 0x7f8644152720 2024-02-19T14:56:16.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:16.878+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f864406dd90 con 0x7f8644152720 2024-02-19T14:56:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.321+0000 7f86417fa640 1 -- 172.21.15.78:0/2872661303 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v121) v1 ==== 138+0+0 (secure 0 0 0) 0x7f864406dd90 con 0x7f8644152720 2024-02-19T14:56:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863006b810 msgr2=0x7f863006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863006b810 0x7f863006dcd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8634006fd0 tx=0x7f8634008040 comp rx=0 tx=0).stop 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 msgr2=0x7f864414acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 0x7f864414acf0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f863800d7d0 tx=0x7f863800dc90 comp rx=0 tx=0).stop 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 shutdown_connections 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863006b810 0x7f863006dcd0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8644152720 0x7f864414acf0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8644158520 0x7f864414b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 --2- 172.21.15.78:0/2872661303 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8644070ff0 0x7f86440a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.323+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 >> 172.21.15.78:0/2872661303 conn(0x7f86440aa370 msgr2=0x7f86440c6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.324+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 shutdown_connections 2024-02-19T14:56:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.324+0000 7f864aeae640 1 -- 172.21.15.78:0/2872661303 wait complete. 2024-02-19T14:56:17.340 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:56:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a51747640 1 Processor -- start 2024-02-19T14:56:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a51747640 1 -- start start 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a51747640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a4c0c9b80 0x7f4a4c0ca030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a51747640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4c152a60 con 0x7f4a4c152680 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a51747640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4c0ca570 con 0x7f4a4c0ace90 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a51747640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4c0aa050 con 0x7f4a4c0c9b80 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a4affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4a4c152680 0x7f4a4c155000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42424/0 (socket says 172.21.15.78:42424) 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.444+0000 7f4a4affd640 1 -- 172.21.15.78:0/2940406517 learned_addr learned my addr 172.21.15.78:0/2940406517 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4b7fe640 1 --2- 172.21.15.78:0/2940406517 >> v2:172.21.15.150:3300/0 conn(0x7f4a4c0c9b80 0x7f4a4c0ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007165636 0 0) 0x7f4a4c152a60 con 0x7f4a4c152680 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a34003520 con 0x7f4a4c152680 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973516150 0 0) 0x7f4a34003520 con 0x7f4a4c152680 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 >> v2:172.21.15.150:3300/0 conn(0x7f4a4c0c9b80 msgr2=0x7f4a4c0ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 --2- 172.21.15.78:0/2940406517 >> v2:172.21.15.150:3300/0 conn(0x7f4a4c0c9b80 0x7f4a4c0ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 >> v1:172.21.15.150:6789/0 conn(0x7f4a4c0ace90 legacy=0x7f4a4c0c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c1557e0 con 0x7f4a4c152680 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4a3c0019f0 con 0x7f4a4c152680 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 >> v1:172.21.15.78:6789/0 conn(0x7f4a4c152680 legacy=0x7f4a4c155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a34003970 0x7f4a34003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a34004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a340050e0 0x7f4a3400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0aa050 con 0x7f4a34004370 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3400c020 con 0x7f4a340050e0 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3400c1e0 con 0x7f4a34003970 2024-02-19T14:56:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a34004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4b7fe640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a340050e0 0x7f4a3400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4affd640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a34003970 0x7f4a34003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4a7fc640 1 -- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a34003970 msgr2=0x7f4a34003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a34003970 0x7f4a34003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4a7fc640 1 -- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a340050e0 msgr2=0x7f4a3400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a340050e0 0x7f4a3400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4a7fc640 1 -- 172.21.15.78:0/2940406517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a34003520 con 0x7f4a34004370 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4b7fe640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a340050e0 0x7f4a3400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.445+0000 7f4a4b7fe640 1 -- 172.21.15.78:0/2940406517 reap_dead start 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.446+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a34004840 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f4a40001830 tx=0x7f4a340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de96fb879a70b59 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.446+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40032600 con 0x7f4a34004370 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.446+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4002e050 con 0x7f4a34004370 2024-02-19T14:56:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.446+0000 7f4a49ffb640 1 -- 172.21.15.78:0/2940406517 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a400329c0 con 0x7f4a34004370 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 -- 172.21.15.78:0/2940406517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 msgr2=0x7f4a34004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a34004840 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f4a40001830 tx=0x7f4a340057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 -- 172.21.15.78:0/2940406517 shutdown_connections 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 --2- 172.21.15.78:0/2940406517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a34004840 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 -- 172.21.15.78:0/2940406517 >> 172.21.15.78:0/2940406517 conn(0x7f4a4c0ac370 msgr2=0x7f4a4c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 -- 172.21.15.78:0/2940406517 shutdown_connections 2024-02-19T14:56:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.447+0000 7f4a51747640 1 -- 172.21.15.78:0/2940406517 wait complete. 2024-02-19T14:56:17.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.448+0000 7f4a51747640 1 Processor -- start 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.448+0000 7f4a51747640 1 -- start start 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a4c1ee090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a4c14eff0 0x7f4a4c0c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a4c0c4260 0x7f4a4c0c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c14c920 con 0x7f4a34004370 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c0c4c50 con 0x7f4a4c0c4260 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c14f7b0 con 0x7f4a4c14eff0 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a4c1ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a4c1ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45392/0 (socket says 172.21.15.78:45392) 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 -- 172.21.15.78:0/1229381328 learned_addr learned my addr 172.21.15.78:0/1229381328 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4b7fe640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a4c0c4260 0x7f4a4c0c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a4c14eff0 0x7f4a4c0c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 -- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a4c14eff0 msgr2=0x7f4a4c0c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a4c14eff0 0x7f4a4c0c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 -- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a4c0c4260 msgr2=0x7f4a4c0c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a4c0c4260 0x7f4a4c0c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 -- 172.21.15.78:0/1229381328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c14cb70 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a4affd640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a4c1ee090 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c007a10 tx=0x7f4a3c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00ee50 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c013420 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c14ce50 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.449+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0137d0 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.450+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c0c1290 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.451+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c14d370 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.451+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4a3c013930 con 0x7f4a34004370 2024-02-19T14:56:17.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.452+0000 7f4a337fe640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a3806b810 0x7f4a3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.452+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a3806b810 0x7f4a3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.452+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f4a3c08a8a0 con 0x7f4a34004370 2024-02-19T14:56:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.452+0000 7f4a4a7fc640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a3806b810 0x7f4a3806dcd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4a40035000 tx=0x7f4a40030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:17.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.456+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a4c14d370 con 0x7f4a34004370 2024-02-19T14:56:17.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:17.628+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4a4c14d370 con 0x7f4a34004370 2024-02-19T14:56:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.325+0000 7f4a337fe640 1 -- 172.21.15.78:0/1229381328 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v122) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4a4c14d370 con 0x7f4a34004370 2024-02-19T14:56:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a3806b810 msgr2=0x7f4a3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a3806b810 0x7f4a3806dcd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4a40035000 tx=0x7f4a40030000 comp rx=0 tx=0).stop 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 msgr2=0x7f4a4c1ee090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a4c1ee090 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c007a10 tx=0x7f4a3c00e530 comp rx=0 tx=0).stop 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 shutdown_connections 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a3806b810 0x7f4a3806dcd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a4c0c4260 0x7f4a4c0c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a4c14eff0 0x7f4a4c0c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 --2- 172.21.15.78:0/1229381328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a34004370 0x7f4a4c1ee090 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 >> 172.21.15.78:0/1229381328 conn(0x7f4a4c0ac370 msgr2=0x7f4a4c151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 shutdown_connections 2024-02-19T14:56:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.328+0000 7f4a51747640 1 -- 172.21.15.78:0/1229381328 wait complete. 2024-02-19T14:56:18.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:56:18.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.450+0000 7fec814f4640 1 Processor -- start 2024-02-19T14:56:18.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.450+0000 7fec814f4640 1 -- start start 2024-02-19T14:56:18.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec814f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec814f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec7c06d910 0x7fec7c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec814f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7c06cd20 con 0x7fec7c152700 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec814f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7c1551f0 con 0x7fec7c06d910 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec814f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c155490 con 0x7fec7c0c8e00 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec7a7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec7b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec7c06d910 0x7fec7c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:18.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec7a7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c06c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:33354/0 (socket says 172.21.15.78:33354) 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.451+0000 7fec7a7fc640 1 -- 172.21.15.78:0/3924898511 learned_addr learned my addr 172.21.15.78:0/3924898511 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.452+0000 7fec7b7fe640 1 -- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 msgr2=0x7fec7c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.452+0000 7fec7b7fe640 1 --2- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.452+0000 7fec7b7fe640 1 -- 172.21.15.78:0/3924898511 >> v1:172.21.15.150:6789/0 conn(0x7fec7c152700 legacy=0x7fec7c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.452+0000 7fec7b7fe640 1 -- 172.21.15.78:0/3924898511 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c155770 con 0x7fec7c06d910 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.452+0000 7fec7a7fc640 1 --2- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.452+0000 7fec7b7fe640 1 --2- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c06d910 0x7fec7c06c7e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fec740097a0 tx=0x7fec7402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d54d769f4c1b53e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec79ffb640 1 -- 172.21.15.78:0/3924898511 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7402e070 con 0x7fec7c06d910 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec79ffb640 1 -- 172.21.15.78:0/3924898511 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec740315a0 con 0x7fec7c06d910 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec79ffb640 1 -- 172.21.15.78:0/3924898511 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74031930 con 0x7fec7c06d910 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 -- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c06d910 msgr2=0x7fec7c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 --2- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c06d910 0x7fec7c06c7e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fec740097a0 tx=0x7fec7402b840 comp rx=0 tx=0).stop 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 -- 172.21.15.78:0/3924898511 shutdown_connections 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 --2- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 --2- 172.21.15.78:0/3924898511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c06d910 0x7fec7c06c7e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 -- 172.21.15.78:0/3924898511 >> 172.21.15.78:0/3924898511 conn(0x7fec7c0cac90 msgr2=0x7fec7c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 -- 172.21.15.78:0/3924898511 shutdown_connections 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.453+0000 7fec814f4640 1 -- 172.21.15.78:0/3924898511 wait complete. 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.454+0000 7fec814f4640 1 Processor -- start 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.454+0000 7fec814f4640 1 -- start start 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec814f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 0x7fec7c148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec814f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c1ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec814f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c152700 0x7fec7c1ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec814f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1eebe0 con 0x7fec7c06d910 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec814f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1ef0a0 con 0x7fec7c152700 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec814f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c156710 con 0x7fec7c0c8e00 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec7affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 0x7fec7c148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec7affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 0x7fec7c148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54452/0 (socket says 172.21.15.78:54452) 2024-02-19T14:56:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec7affd640 1 -- 172.21.15.78:0/3072851419 learned_addr learned my addr 172.21.15.78:0/3072851419 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.455+0000 7fec7a7fc640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c1ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7b7fe640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c152700 0x7fec7c1ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7affd640 1 -- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 msgr2=0x7fec7c1ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7affd640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c1ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7affd640 1 -- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c152700 msgr2=0x7fec7c1ee6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7affd640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c152700 0x7fec7c1ee6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7affd640 1 -- 172.21.15.78:0/3072851419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700094d0 con 0x7fec7c06d910 2024-02-19T14:56:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7a7fc640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c1ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7b7fe640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c152700 0x7fec7c1ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec7affd640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 0x7fec7c148610 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c790 tx=0x7fec6c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00cea0 con 0x7fec7c06d910 2024-02-19T14:56:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c006d50 con 0x7fec7c06d910 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c003bb0 con 0x7fec7c06d910 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74009400 con 0x7fec7c06d910 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.456+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c1eedc0 con 0x7fec7c06d910 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.457+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7c152ae0 con 0x7fec7c06d910 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.457+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fec6c003d10 con 0x7fec7c06d910 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.458+0000 7fec637fe640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec6806b810 0x7fec6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.458+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fec6c089e30 con 0x7fec7c06d910 2024-02-19T14:56:18.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.458+0000 7fec7a7fc640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec6806b810 0x7fec6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:18.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.458+0000 7fec7a7fc640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec6806b810 0x7fec6806dcd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0c5910 tx=0x7fec70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:18.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.462+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fec6c0582e0 con 0x7fec7c06d910 2024-02-19T14:56:18.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:18.669+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec7c0688f0 con 0x7fec7c06d910 2024-02-19T14:56:19.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.342+0000 7fec637fe640 1 -- 172.21.15.78:0/3072851419 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v98) v1 ==== 174+0+0 (secure 0 0 0) 0x7fec6c057e70 con 0x7fec7c06d910 2024-02-19T14:56:19.345 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 19 and data pool 20 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.349+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec6806b810 msgr2=0x7fec6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.349+0000 7fec814f4640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec6806b810 0x7fec6806dcd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0c5910 tx=0x7fec70008040 comp rx=0 tx=0).stop 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.349+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 msgr2=0x7fec7c148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.349+0000 7fec814f4640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 0x7fec7c148610 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c790 tx=0x7fec6c00cc50 comp rx=0 tx=0).stop 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.350+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 shutdown_connections 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.350+0000 7fec814f4640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec6806b810 0x7fec6806dcd0 secure :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0c5910 tx=0x7fec70008040 comp rx=0 tx=0).stop 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.350+0000 7fec814f4640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c152700 0x7fec7c1ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.350+0000 7fec814f4640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7c0c8e00 0x7fec7c1ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.350+0000 7fec814f4640 1 --2- 172.21.15.78:0/3072851419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec7c06d910 0x7fec7c148610 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.350+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 >> 172.21.15.78:0/3072851419 conn(0x7fec7c0cac90 msgr2=0x7fec7c14ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:19.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.351+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 shutdown_connections 2024-02-19T14:56:19.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.352+0000 7fec814f4640 1 -- 172.21.15.78:0/3072851419 wait complete. 2024-02-19T14:56:19.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:19.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.471+0000 7f053be74640 1 Processor -- start 2024-02-19T14:56:19.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.471+0000 7f053be74640 1 -- start start 2024-02-19T14:56:19.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053be74640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f05340c67e0 0x7f053406c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053be74640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053406d460 0x7f053406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053be74640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340c7df0 con 0x7f0534155e00 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053be74640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340c4310 con 0x7f053406d460 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053be74640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053406ce10 con 0x7f05340c67e0 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053406d460 0x7f053406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053406d460 0x7f053406c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54466/0 (socket says 172.21.15.78:54466) 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 -- 172.21.15.78:0/1776371249 learned_addr learned my addr 172.21.15.78:0/1776371249 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 -- 172.21.15.78:0/1776371249 >> v2:172.21.15.150:3300/0 conn(0x7f05340c67e0 msgr2=0x7f053406c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 --2- 172.21.15.78:0/1776371249 >> v2:172.21.15.150:3300/0 conn(0x7f05340c67e0 0x7f053406c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 -- 172.21.15.78:0/1776371249 >> v1:172.21.15.150:6789/0 conn(0x7f0534155e00 legacy=0x7f05340c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.472+0000 7f053a411640 1 -- 172.21.15.78:0/1776371249 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340c77c0 con 0x7f053406d460 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.473+0000 7f053a411640 1 --2- 172.21.15.78:0/1776371249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053406d460 0x7f053406c8d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0534071b10 tx=0x7f052c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592a995a92ccd15 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.473+0000 7f0538c0e640 1 -- 172.21.15.78:0/1776371249 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c02c9f0 con 0x7f053406d460 2024-02-19T14:56:19.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.473+0000 7f0538c0e640 1 -- 172.21.15.78:0/1776371249 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c02cb50 con 0x7f053406d460 2024-02-19T14:56:19.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.473+0000 7f0538c0e640 1 -- 172.21.15.78:0/1776371249 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0343f0 con 0x7f053406d460 2024-02-19T14:56:19.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 -- 172.21.15.78:0/1776371249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053406d460 msgr2=0x7f053406c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 --2- 172.21.15.78:0/1776371249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053406d460 0x7f053406c8d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f0534071b10 tx=0x7f052c02be50 comp rx=0 tx=0).stop 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 -- 172.21.15.78:0/1776371249 shutdown_connections 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 --2- 172.21.15.78:0/1776371249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053406d460 0x7f053406c8d0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 --2- 172.21.15.78:0/1776371249 >> v2:172.21.15.150:3300/0 conn(0x7f05340c67e0 0x7f053406c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 -- 172.21.15.78:0/1776371249 >> 172.21.15.78:0/1776371249 conn(0x7f053406b700 msgr2=0x7f05340aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 -- 172.21.15.78:0/1776371249 shutdown_connections 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.474+0000 7f053be74640 1 -- 172.21.15.78:0/1776371249 wait complete. 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.475+0000 7f053be74640 1 Processor -- start 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.475+0000 7f053be74640 1 -- start start 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.475+0000 7f053be74640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053406d460 0x7f0534158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.475+0000 7f053be74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 0x7f05341eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053be74640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0534155e00 0x7f05341eb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053be74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0534156490 con 0x7f05340c67e0 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053be74640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05340c93b0 con 0x7f053406d460 2024-02-19T14:56:19.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053be74640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05341ebc10 con 0x7f0534155e00 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 0x7f05341eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 0x7f05341eb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54476/0 (socket says 172.21.15.78:54476) 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 -- 172.21.15.78:0/3065378194 learned_addr learned my addr 172.21.15.78:0/3065378194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f0539c10640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053406d460 0x7f0534158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053a411640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0534155e00 0x7f05341eb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 -- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0534155e00 msgr2=0x7f05341eb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0534155e00 0x7f05341eb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 -- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053406d460 msgr2=0x7f0534158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053406d460 0x7f0534158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 -- 172.21.15.78:0/3065378194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009b30 con 0x7f05340c67e0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053940f640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 0x7f05341eb190 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f052400b410 tx=0x7f052400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011830 con 0x7f05340c67e0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05240119b0 con 0x7f05340c67e0 2024-02-19T14:56:19.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0534158dd0 con 0x7f05340c67e0 2024-02-19T14:56:19.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524011d60 con 0x7f05340c67e0 2024-02-19T14:56:19.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.476+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05340a9690 con 0x7f05340c67e0 2024-02-19T14:56:19.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.477+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05340c7390 con 0x7f05340c67e0 2024-02-19T14:56:19.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.478+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f052401a400 con 0x7f05340c67e0 2024-02-19T14:56:19.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.478+0000 7f052a7fc640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f051806b810 0x7f051806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.478+0000 7f0539c10640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f051806b810 0x7f051806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.478+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f052408a400 con 0x7f05340c67e0 2024-02-19T14:56:19.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.479+0000 7f0539c10640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f051806b810 0x7f051806dcd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f05300015d0 tx=0x7f0530009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:19.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.483+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05340c7390 con 0x7f05340c67e0 2024-02-19T14:56:19.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.655+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05340c7390 con 0x7f05340c67e0 2024-02-19T14:56:19.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.656+0000 7f052a7fc640 1 -- 172.21.15.78:0/3065378194 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+16901 (secure 0 0 0) 0x7f05340c7390 con 0x7f05340c67e0 2024-02-19T14:56:19.659 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:19.659 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":123,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:18.670749+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:16.897563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:17.646704+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:19.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f051806b810 msgr2=0x7f051806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f051806b810 0x7f051806dcd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f05300015d0 tx=0x7f0530009040 comp rx=0 tx=0).stop 2024-02-19T14:56:19.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 msgr2=0x7f05341eb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 0x7f05341eb190 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f052400b410 tx=0x7f052400b8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 shutdown_connections 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f051806b810 0x7f051806dcd0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05340c67e0 0x7f05341eb190 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0534155e00 0x7f05341eb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 --2- 172.21.15.78:0/3065378194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053406d460 0x7f0534158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 >> 172.21.15.78:0/3065378194 conn(0x7f053406b700 msgr2=0x7f05341545b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 shutdown_connections 2024-02-19T14:56:19.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.658+0000 7f053be74640 1 -- 172.21.15.78:0/3065378194 wait complete. 2024-02-19T14:56:19.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.770+0000 7f72fd855640 1 Processor -- start 2024-02-19T14:56:19.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.770+0000 7f72fd855640 1 -- start start 2024-02-19T14:56:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72fd855640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72f8152180 0x7f72f8152980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72fd855640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f806b710 con 0x7f72f80c1d40 2024-02-19T14:56:19.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72fd855640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f8152560 con 0x7f72f80c1880 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72fd855640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f8072430 con 0x7f72f8152180 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72f77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f72f80c1d40 0x7f72f806e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55008/0 (socket says 172.21.15.78:55008) 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72f77fe640 1 -- 172.21.15.78:0/3237623476 learned_addr learned my addr 172.21.15.78:0/3237623476 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.771+0000 7f72f6ffd640 1 --2- 172.21.15.78:0/3237623476 >> v2:172.21.15.150:3300/0 conn(0x7f72f8152180 0x7f72f8152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745789180 0 0) 0x7f72f806b710 con 0x7f72f80c1d40 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e0003520 con 0x7f72f80c1d40 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3668792294 0 0) 0x7f72e0003520 con 0x7f72f80c1d40 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 >> v2:172.21.15.150:3300/0 conn(0x7f72f8152180 msgr2=0x7f72f8152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 --2- 172.21.15.78:0/3237623476 >> v2:172.21.15.150:3300/0 conn(0x7f72f8152180 0x7f72f8152980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 >> v1:172.21.15.150:6789/0 conn(0x7f72f80c1880 legacy=0x7f72f8072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f806f0a0 con 0x7f72f80c1d40 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f72f0001670 con 0x7f72f80c1d40 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 >> v1:172.21.15.78:6789/0 conn(0x7f72f80c1d40 legacy=0x7f72f806e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e0003970 0x7f72e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72e00047c0 0x7f72e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72e00052a0 0x7f72e000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8072430 con 0x7f72e00052a0 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e000e380 con 0x7f72e00047c0 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e000e540 con 0x7f72e0003970 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f77fe640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72e00052a0 0x7f72e000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f6ffd640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e0003970 0x7f72e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f67fc640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72e00047c0 0x7f72e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f77fe640 1 -- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e0003970 msgr2=0x7f72e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f77fe640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e0003970 0x7f72e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f77fe640 1 -- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72e00047c0 msgr2=0x7f72e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f77fe640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72e00047c0 0x7f72e0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.772+0000 7f72f77fe640 1 -- 172.21.15.78:0/3237623476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e0003520 con 0x7f72e00052a0 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.773+0000 7f72f77fe640 1 -- 172.21.15.78:0/3237623476 reap_dead start 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.773+0000 7f72f77fe640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72e00052a0 0x7f72e000db90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f72f000c210 tx=0x7f72e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b958b49ff817f56 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.773+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0005ad0 con 0x7f72e00052a0 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.773+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0005c30 con 0x7f72e00052a0 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.773+0000 7f72f5ffb640 1 -- 172.21.15.78:0/3237623476 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f002c890 con 0x7f72e00052a0 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 -- 172.21.15.78:0/3237623476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72e00052a0 msgr2=0x7f72e000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72e00052a0 0x7f72e000db90 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f72f000c210 tx=0x7f72e00057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 -- 172.21.15.78:0/3237623476 shutdown_connections 2024-02-19T14:56:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 --2- 172.21.15.78:0/3237623476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72e00052a0 0x7f72e000db90 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 -- 172.21.15.78:0/3237623476 >> 172.21.15.78:0/3237623476 conn(0x7f72f80a8700 msgr2=0x7f72f80a8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 -- 172.21.15.78:0/3237623476 shutdown_connections 2024-02-19T14:56:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.774+0000 7f72fd855640 1 -- 172.21.15.78:0/3237623476 wait complete. 2024-02-19T14:56:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.775+0000 7f72fd855640 1 Processor -- start 2024-02-19T14:56:19.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.775+0000 7f72fd855640 1 -- start start 2024-02-19T14:56:19.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e00052a0 0x7f72f814f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f80c4750 0x7f72f8144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 0x7f72f81e6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c5660 con 0x7f72f8145020 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f814dc80 con 0x7f72f80c4750 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f81e9e50 con 0x7f72e00052a0 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 0x7f72f81e6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 0x7f72f81e6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54488/0 (socket says 172.21.15.78:54488) 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 -- 172.21.15.78:0/896627860 learned_addr learned my addr 172.21.15.78:0/896627860 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f6ffd640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e00052a0 0x7f72f814f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f67fc640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f80c4750 0x7f72f8144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 -- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e00052a0 msgr2=0x7f72f814f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e00052a0 0x7f72f814f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 -- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f80c4750 msgr2=0x7f72f8144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f80c4750 0x7f72f8144ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 -- 172.21.15.78:0/896627860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f81ea130 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72f77fe640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 0x7f72f81e6c30 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f72f000c210 tx=0x7f72f0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00049e0 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0004b40 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f81ea410 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0001640 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.776+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f81ea9d0 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.777+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f80c2a30 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.778+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f72f0032070 con 0x7f72f8145020 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.778+0000 7f72dffff640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72e406b840 0x7f72e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:19.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.778+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72f003c070 con 0x7f72f8145020 2024-02-19T14:56:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.778+0000 7f72f6ffd640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72e406b840 0x7f72e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.778+0000 7f72f6ffd640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72e406b840 0x7f72e406dd00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f72e8014450 tx=0x7f72e800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:19.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.782+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f72f0075a90 con 0x7f72f8145020 2024-02-19T14:56:19.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.956+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72f8074990 con 0x7f72f8145020 2024-02-19T14:56:19.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.957+0000 7f72dffff640 1 -- 172.21.15.78:0/896627860 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+16901 (secure 0 0 0) 0x7f72f0044090 con 0x7f72f8145020 2024-02-19T14:56:19.960 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:19.960 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":123,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:18.670749+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:16.897563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:17.646704+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72e406b840 msgr2=0x7f72e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72e406b840 0x7f72e406dd00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f72e8014450 tx=0x7f72e800b040 comp rx=0 tx=0).stop 2024-02-19T14:56:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 msgr2=0x7f72f81e6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 0x7f72f81e6c30 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f72f000c210 tx=0x7f72f0001a00 comp rx=0 tx=0).stop 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 shutdown_connections 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72e406b840 0x7f72e406dd00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f8145020 0x7f72f81e6c30 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f80c4750 0x7f72f8144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 --2- 172.21.15.78:0/896627860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72e00052a0 0x7f72f814f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 >> 172.21.15.78:0/896627860 conn(0x7f72f80a8700 msgr2=0x7f72f806e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 shutdown_connections 2024-02-19T14:56:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:19.959+0000 7f72fd855640 1 -- 172.21.15.78:0/896627860 wait complete. 2024-02-19T14:56:19.973 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:56:20.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.073+0000 7f4f336c0640 1 Processor -- start 2024-02-19T14:56:20.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.073+0000 7f4f336c0640 1 -- start start 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f336c0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f2c0c8e00 0x7f4f2c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f336c0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c06d910 0x7f4f2c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f336c0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c06cd20 con 0x7f4f2c152700 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f336c0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c1551f0 con 0x7f4f2c0c8e00 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f336c0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c155490 con 0x7f4f2c06d910 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f3145c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4f2c152700 0x7f4f2c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55024/0 (socket says 172.21.15.78:55024) 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f3145c640 1 -- 172.21.15.78:0/3047378662 learned_addr learned my addr 172.21.15.78:0/3047378662 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211722135 0 0) 0x7f4f2c06cd20 con 0x7f4f2c152700 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f18003520 con 0x7f4f2c152700 2024-02-19T14:56:20.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/3047378662 >> v2:172.21.15.150:3300/0 conn(0x7f4f2c0c8e00 0x7f4f2c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f31c5d640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c06d910 0x7f4f2c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340137843 0 0) 0x7f4f18003520 con 0x7f4f2c152700 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c06d910 msgr2=0x7f4f2c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c06d910 0x7f4f2c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 >> v2:172.21.15.150:3300/0 conn(0x7f4f2c0c8e00 msgr2=0x7f4f2c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 --2- 172.21.15.78:0/3047378662 >> v2:172.21.15.150:3300/0 conn(0x7f4f2c0c8e00 0x7f4f2c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c155770 con 0x7f4f2c152700 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4f28001670 con 0x7f4f2c152700 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 >> v1:172.21.15.78:6789/0 conn(0x7f4f2c152700 legacy=0x7f4f2c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f18003970 0x7f4f18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f180047c0 0x7f4f18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f18005180 0x7f4f1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1551f0 con 0x7f4f180047c0 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c155490 con 0x7f4f18005180 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1800bff0 con 0x7f4f18003970 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f31c5d640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c06d910 0x7f4f2c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.074+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f180047c0 0x7f4f18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f3145c640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f18003970 0x7f4f18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f31c5d640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f18005180 0x7f4f1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f30c5b640 1 -- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f18003970 msgr2=0x7f4f18003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f18003970 0x7f4f18003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f30c5b640 1 -- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f18005180 msgr2=0x7f4f1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f18005180 0x7f4f1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f30c5b640 1 -- 172.21.15.78:0/3047378662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f18003520 con 0x7f4f180047c0 2024-02-19T14:56:20.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f180047c0 0x7f4f18004c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009ac0 tx=0x7f4f180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ed2ca84a6a2ddc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c02c710 con 0x7f4f180047c0 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c02c870 con 0x7f4f180047c0 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.075+0000 7f4f23fff640 1 -- 172.21.15.78:0/3047378662 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c02cc00 con 0x7f4f180047c0 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f336c0640 1 -- 172.21.15.78:0/3047378662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f180047c0 msgr2=0x7f4f18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f336c0640 1 --2- 172.21.15.78:0/3047378662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f180047c0 0x7f4f18004c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009ac0 tx=0x7f4f180057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f31c5d640 1 -- 172.21.15.78:0/3047378662 reap_dead start 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f336c0640 1 -- 172.21.15.78:0/3047378662 shutdown_connections 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f336c0640 1 -- 172.21.15.78:0/3047378662 >> 172.21.15.78:0/3047378662 conn(0x7f4f2c0cac90 msgr2=0x7f4f2c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f336c0640 1 -- 172.21.15.78:0/3047378662 shutdown_connections 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.076+0000 7f4f336c0640 1 -- 172.21.15.78:0/3047378662 wait complete. 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.077+0000 7f4f336c0640 1 Processor -- start 2024-02-19T14:56:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.077+0000 7f4f336c0640 1 -- start start 2024-02-19T14:56:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f336c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f180047c0 0x7f4f2c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f336c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 0x7f4f2c148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f336c0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c149370 0x7f4f2c0add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f336c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0ac3c0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f336c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0aa8c0 con 0x7f4f180047c0 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f336c0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0aa5e0 con 0x7f4f2c149370 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 0x7f4f2c148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 0x7f4f2c148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54494/0 (socket says 172.21.15.78:54494) 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 -- 172.21.15.78:0/1382531210 learned_addr learned my addr 172.21.15.78:0/1382531210 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f31c5d640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c149370 0x7f4f2c0add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f3145c640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f180047c0 0x7f4f2c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 -- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c149370 msgr2=0x7f4f2c0add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c149370 0x7f4f2c0add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 -- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f180047c0 msgr2=0x7f4f2c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f180047c0 0x7f4f2c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 -- 172.21.15.78:0/1382531210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0abe00 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f3145c640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f180047c0 0x7f4f2c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.078+0000 7f4f30c5b640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 0x7f4f2c148e30 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c000c00 tx=0x7f4f1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.079+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c003810 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.079+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c02de90 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.079+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c0ac0e0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.079+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c003b90 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.079+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c0acdd0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.080+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c156210 con 0x7f4f2c1489d0 2024-02-19T14:56:20.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.080+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4f1c003cf0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.081+0000 7f4f21ffb640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f1006b810 0x7f4f1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.081+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f1c0ac3f0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.081+0000 7f4f3145c640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f1006b810 0x7f4f1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.081+0000 7f4f3145c640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f1006b810 0x7f4f1006dcd0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4f280058b0 tx=0x7f4f28001010 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.085+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4f1c076bf0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.290+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4f2c14dbd0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.350+0000 7f4f21ffb640 1 -- 172.21.15.78:0/1382531210 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v99) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4f1c07a2b0 con 0x7f4f2c1489d0 2024-02-19T14:56:20.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f1006b810 msgr2=0x7f4f1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f1006b810 0x7f4f1006dcd0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4f280058b0 tx=0x7f4f28001010 comp rx=0 tx=0).stop 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 msgr2=0x7f4f2c148e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 0x7f4f2c148e30 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c000c00 tx=0x7f4f1c004730 comp rx=0 tx=0).stop 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 shutdown_connections 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4f1006b810 0x7f4f1006dcd0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f2c1489d0 0x7f4f2c148e30 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f2c149370 0x7f4f2c0add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 --2- 172.21.15.78:0/1382531210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f180047c0 0x7f4f2c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 >> 172.21.15.78:0/1382531210 conn(0x7f4f2c0cac90 msgr2=0x7f4f2c1500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 shutdown_connections 2024-02-19T14:56:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.353+0000 7f4f336c0640 1 -- 172.21.15.78:0/1382531210 wait complete. 2024-02-19T14:56:20.367 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:56:20.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:56:20.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.468+0000 7f130712c640 1 Processor -- start 2024-02-19T14:56:20.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.468+0000 7f130712c640 1 -- start start 2024-02-19T14:56:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f130712c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c7910 0x7f13000c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f130712c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1300070720 0x7f1300070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f130712c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000c65a0 con 0x7f13000c5560 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f130712c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13000c7530 con 0x7f1300070720 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f130712c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13001537b0 con 0x7f13000c7910 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1300070720 0x7f1300070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1300070720 0x7f1300070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54498/0 (socket says 172.21.15.78:54498) 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 -- 172.21.15.78:0/2483912769 learned_addr learned my addr 172.21.15.78:0/2483912769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f12f7fff640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c7910 0x7f13000c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 -- 172.21.15.78:0/2483912769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c7910 msgr2=0x7f13000c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c7910 0x7f13000c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 -- 172.21.15.78:0/2483912769 >> v1:172.21.15.150:6789/0 conn(0x7f13000c5560 legacy=0x7f1300154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f13056c9640 1 -- 172.21.15.78:0/2483912769 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300153a90 con 0x7f1300070720 2024-02-19T14:56:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.469+0000 7f12f7fff640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c7910 0x7f13000c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.470+0000 7f13056c9640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1300070720 0x7f1300070b70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009e50 tx=0x7f12f802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c46c21e41cbd21a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f12f77fe640 1 -- 172.21.15.78:0/2483912769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f802c950 con 0x7f1300070720 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f12f77fe640 1 -- 172.21.15.78:0/2483912769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f802cab0 con 0x7f1300070720 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f12f77fe640 1 -- 172.21.15.78:0/2483912769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f802ce40 con 0x7f1300070720 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 -- 172.21.15.78:0/2483912769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1300070720 msgr2=0x7f1300070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1300070720 0x7f1300070b70 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009e50 tx=0x7f12f802bdb0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 -- 172.21.15.78:0/2483912769 shutdown_connections 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1300070720 0x7f1300070b70 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 --2- 172.21.15.78:0/2483912769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c7910 0x7f13000c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 -- 172.21.15.78:0/2483912769 >> 172.21.15.78:0/2483912769 conn(0x7f130006b4b0 msgr2=0x7f13000c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 -- 172.21.15.78:0/2483912769 shutdown_connections 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.471+0000 7f130712c640 1 -- 172.21.15.78:0/2483912769 wait complete. 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.472+0000 7f130712c640 1 Processor -- start 2024-02-19T14:56:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.472+0000 7f130712c640 1 -- start start 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f130712c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1300070720 0x7f13001df510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f130712c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c5560 0x7f13001dc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f130712c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 0x7f13001dcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f130712c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13001dfd50 con 0x7f13000c7910 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f130712c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13001e0030 con 0x7f1300070720 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f130712c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13001e9610 con 0x7f13000c5560 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 0x7f13001dcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 0x7f13001dcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54504/0 (socket says 172.21.15.78:54504) 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 -- 172.21.15.78:0/2823167755 learned_addr learned my addr 172.21.15.78:0/2823167755 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 -- 172.21.15.78:0/2823167755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c5560 msgr2=0x7f13001dc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c5560 0x7f13001dc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 -- 172.21.15.78:0/2823167755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1300070720 msgr2=0x7f13001df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1300070720 0x7f13001df510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.473+0000 7f13056c9640 1 -- 172.21.15.78:0/2823167755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8009b30 con 0x7f13000c7910 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.474+0000 7f13056c9640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 0x7f13001dcc10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f12f8036040 tx=0x7f12f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.474+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f803e450 con 0x7f13000c7910 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.474+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f803e5d0 con 0x7f13000c7910 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.474+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f803e980 con 0x7f13000c7910 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.474+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13000c5ac0 con 0x7f13000c7910 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.474+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13001e9250 con 0x7f13000c7910 2024-02-19T14:56:20.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.475+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1300074990 con 0x7f13000c7910 2024-02-19T14:56:20.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.475+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f12f803eae0 con 0x7f13000c7910 2024-02-19T14:56:20.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.475+0000 7f12f57fa640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12ec06b810 0x7f12ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:20.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.476+0000 7f1304ec8640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12ec06b810 0x7f12ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:20.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.476+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12f80ad1b0 con 0x7f13000c7910 2024-02-19T14:56:20.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.476+0000 7f1304ec8640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12ec06b810 0x7f12ec06dcd0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009650 tx=0x7f12fc006ba0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:20.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.480+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f12f8077ad0 con 0x7f13000c7910 2024-02-19T14:56:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:20.691+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f13001dc2f0 con 0x7f13000c7910 2024-02-19T14:56:21.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.351+0000 7f12f57fa640 1 -- 172.21.15.78:0/2823167755 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v100) v1 ==== 122+0+0 (secure 0 0 0) 0x7f12f807b190 con 0x7f13000c7910 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.353+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12ec06b810 msgr2=0x7f12ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.353+0000 7f130712c640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12ec06b810 0x7f12ec06dcd0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009650 tx=0x7f12fc006ba0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.353+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 msgr2=0x7f13001dcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.353+0000 7f130712c640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 0x7f13001dcc10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f12f8036040 tx=0x7f12f8004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 shutdown_connections 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12ec06b810 0x7f12ec06dcd0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13000c7910 0x7f13001dcc10 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13000c5560 0x7f13001dc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 --2- 172.21.15.78:0/2823167755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1300070720 0x7f13001df510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 >> 172.21.15.78:0/2823167755 conn(0x7f130006b4b0 msgr2=0x7f1300070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:21.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 shutdown_connections 2024-02-19T14:56:21.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.354+0000 7f130712c640 1 -- 172.21.15.78:0/2823167755 wait complete. 2024-02-19T14:56:21.367 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:56:21.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.464+0000 7f9890f79640 1 Processor -- start 2024-02-19T14:56:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f9890f79640 1 -- start start 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f9890f79640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f988c14d210 0x7f988c0c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f9890f79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 0x7f988c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f9890f79640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c06ff90 con 0x7f988c0c3d90 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f9890f79640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c070270 con 0x7f988c14d210 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f9890f79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0c3970 con 0x7f988c0719e0 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f988ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f988c0c3d90 0x7f988c152ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55028/0 (socket says 172.21.15.78:55028) 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.465+0000 7f988ad9d640 1 -- 172.21.15.78:0/1283688079 learned_addr learned my addr 172.21.15.78:0/1283688079 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 0x7f988c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780624940 0 0) 0x7f988c06ff90 con 0x7f988c0c3d90 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878003520 con 0x7f988c0c3d90 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f988a59c640 1 --2- 172.21.15.78:0/1283688079 >> v2:172.21.15.150:3300/0 conn(0x7f988c14d210 0x7f988c0c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3773878740 0 0) 0x7f9878003520 con 0x7f988c0c3d90 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 msgr2=0x7f988c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 0x7f988c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 >> v2:172.21.15.150:3300/0 conn(0x7f988c14d210 msgr2=0x7f988c0c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 --2- 172.21.15.78:0/1283688079 >> v2:172.21.15.150:3300/0 conn(0x7f988c14d210 0x7f988c0c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c156fc0 con 0x7f988c0c3d90 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f98840019f0 con 0x7f988c0c3d90 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 >> v1:172.21.15.78:6789/0 conn(0x7f988c0c3d90 legacy=0x7f988c152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 0x7f9878003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f98780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878004d00 0x7f9878005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c070270 con 0x7f9878003ef0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0c3970 con 0x7f9878004d00 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987800b9d0 con 0x7f988c0719e0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f98780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f988a59c640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 0x7f9878003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f988ad9d640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878004d00 0x7f9878005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 -- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 msgr2=0x7f9878003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c0719e0 0x7f9878003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 -- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878004d00 msgr2=0x7f9878005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878004d00 0x7f9878005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.466+0000 7f9883fff640 1 -- 172.21.15.78:0/1283688079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9878003520 con 0x7f9878003ef0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.467+0000 7f9883fff640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f98780047c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f9874009ac0 tx=0x7f98780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce1861dc0d27df3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.467+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987402c680 con 0x7f9878003ef0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.467+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987402c7e0 con 0x7f9878003ef0 2024-02-19T14:56:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.467+0000 7f9889d9b640 1 -- 172.21.15.78:0/1283688079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987402cb70 con 0x7f9878003ef0 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f9890f79640 1 -- 172.21.15.78:0/1283688079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 msgr2=0x7f98780047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f9890f79640 1 --2- 172.21.15.78:0/1283688079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f98780047c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f9874009ac0 tx=0x7f98780057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f988ad9d640 1 -- 172.21.15.78:0/1283688079 reap_dead start 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f9890f79640 1 -- 172.21.15.78:0/1283688079 shutdown_connections 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f9890f79640 1 -- 172.21.15.78:0/1283688079 >> 172.21.15.78:0/1283688079 conn(0x7f988c0c5050 msgr2=0x7f988c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f9890f79640 1 -- 172.21.15.78:0/1283688079 shutdown_connections 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.468+0000 7f9890f79640 1 -- 172.21.15.78:0/1283688079 wait complete. 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.469+0000 7f9890f79640 1 Processor -- start 2024-02-19T14:56:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.469+0000 7f9890f79640 1 -- start start 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9890f79640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f988c1e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9890f79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c1465a0 0x7f988c0ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9890f79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988c0ac9e0 0x7f988c1ec100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9890f79640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0abd90 con 0x7f9878003ef0 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9890f79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0adcf0 con 0x7f988c0ac9e0 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9890f79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0adfd0 con 0x7f988c1465a0 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f988c1e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f988c1e9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54534/0 (socket says 172.21.15.78:54534) 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 -- 172.21.15.78:0/2617910826 learned_addr learned my addr 172.21.15.78:0/2617910826 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f9883fff640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c1465a0 0x7f988c0ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988ad9d640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988c0ac9e0 0x7f988c1ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 -- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c1465a0 msgr2=0x7f988c0ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c1465a0 0x7f988c0ac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 -- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988c0ac9e0 msgr2=0x7f988c1ec100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988c0ac9e0 0x7f988c1ec100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988a59c640 1 -- 172.21.15.78:0/2617910826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0c3660 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.470+0000 7f988ad9d640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988c0ac9e0 0x7f988c1ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.471+0000 7f988a59c640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f988c1e9d90 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f987c011a70 tx=0x7f987c011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.471+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c009680 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.471+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c00fb50 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.471+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c0ae2b0 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.471+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00fee0 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.471+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c0acec0 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.472+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c074990 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.472+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f987c014440 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.473+0000 7f9882ffd640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f986c06b840 0x7f986c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.473+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f987c08abf0 con 0x7f9878003ef0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.473+0000 7f9883fff640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f986c06b840 0x7f986c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.473+0000 7f9883fff640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f986c06b840 0x7f986c06dd00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9874036000 tx=0x7f9874030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:21.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.477+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f987c055490 con 0x7f9878003ef0 2024-02-19T14:56:21.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.690+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f988c0c2c30 con 0x7f9878003ef0 2024-02-19T14:56:21.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.691+0000 7f9882ffd640 1 -- 172.21.15.78:0/2617910826 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f987c058b50 con 0x7f9878003ef0 2024-02-19T14:56:21.693 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:21.694 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7788,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:20.692270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7791":{"gid":7791,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/100250519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":100250519},{"type":"v1","addr":"172.21.15.78:6837","nonce":100250519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 100 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f986c06b840 msgr2=0x7f986c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f986c06b840 0x7f986c06dd00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9874036000 tx=0x7f9874030000 comp rx=0 tx=0).stop 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 msgr2=0x7f988c1e9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f988c1e9d90 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f987c011a70 tx=0x7f987c011f30 comp rx=0 tx=0).stop 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 shutdown_connections 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f986c06b840 0x7f986c06dd00 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988c0ac9e0 0x7f988c1ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f988c1465a0 0x7f988c0ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 --2- 172.21.15.78:0/2617910826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9878003ef0 0x7f988c1e9d90 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 >> 172.21.15.78:0/2617910826 conn(0x7f988c0c5050 msgr2=0x7f988c14ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:21.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 shutdown_connections 2024-02-19T14:56:21.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.693+0000 7f9890f79640 1 -- 172.21.15.78:0/2617910826 wait complete. 2024-02-19T14:56:21.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:21.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.809+0000 7f3f6b1f7640 1 Processor -- start 2024-02-19T14:56:21.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.809+0000 7f3f6b1f7640 1 -- start start 2024-02-19T14:56:21.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f6b1f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3f6406e350 0x7f3f640c0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f6b1f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f641529f0 con 0x7f3f640c06b0 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f6b1f7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640bf9b0 con 0x7f3f64152610 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f6b1f7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6406da70 con 0x7f3f6406e350 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f63fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3f6406e350 0x7f3f640c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f63fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3f6406e350 0x7f3f640c0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54546/0 (socket says 172.21.15.78:54546) 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f69794640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3f640c06b0 0x7f3f640c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55040/0 (socket says 172.21.15.78:55040) 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f63fff640 1 -- 172.21.15.78:0/1434020175 learned_addr learned my addr 172.21.15.78:0/1434020175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f63fff640 1 -- 172.21.15.78:0/1434020175 >> v1:172.21.15.150:6789/0 conn(0x7f3f64152610 legacy=0x7f3f64158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f63fff640 1 -- 172.21.15.78:0/1434020175 >> v1:172.21.15.78:6789/0 conn(0x7f3f640c06b0 legacy=0x7f3f640c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f63fff640 1 -- 172.21.15.78:0/1434020175 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6406e8c0 con 0x7f3f6406e350 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.810+0000 7f3f637fe640 1 -- 172.21.15.78:0/1434020175 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493784573 0 0) 0x7f3f641529f0 con 0x7f3f640c06b0 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.811+0000 7f3f63fff640 1 --2- 172.21.15.78:0/1434020175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640c0170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009870 tx=0x7f3f5002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5561a7e127240837 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.811+0000 7f3f637fe640 1 -- 172.21.15.78:0/1434020175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50030860 con 0x7f3f6406e350 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.811+0000 7f3f637fe640 1 -- 172.21.15.78:0/1434020175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f500309c0 con 0x7f3f6406e350 2024-02-19T14:56:21.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.811+0000 7f3f637fe640 1 -- 172.21.15.78:0/1434020175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50030d50 con 0x7f3f6406e350 2024-02-19T14:56:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.811+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/1434020175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 msgr2=0x7f3f640c0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.811+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/1434020175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640c0170 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009870 tx=0x7f3f5002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.812+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/1434020175 shutdown_connections 2024-02-19T14:56:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.812+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/1434020175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640c0170 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.812+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/1434020175 >> 172.21.15.78:0/1434020175 conn(0x7f3f6406b270 msgr2=0x7f3f6406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.812+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/1434020175 shutdown_connections 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.812+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/1434020175 wait complete. 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.813+0000 7f3f6b1f7640 1 Processor -- start 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.813+0000 7f3f6b1f7640 1 -- start start 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f640c06b0 0x7f3f6414c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f64152610 0x7f3f6414ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640ab530 con 0x7f3f6406e350 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641eeea0 con 0x7f3f640c06b0 2024-02-19T14:56:21.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64153ed0 con 0x7f3f64152610 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640a86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54558/0 (socket says 172.21.15.78:54558) 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 -- 172.21.15.78:0/162812941 learned_addr learned my addr 172.21.15.78:0/162812941 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f69794640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f64152610 0x7f3f6414ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f63fff640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f640c06b0 0x7f3f6414c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 -- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f64152610 msgr2=0x7f3f6414ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f64152610 0x7f3f6414ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 -- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f640c06b0 msgr2=0x7f3f6414c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f640c06b0 0x7f3f6414c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 -- 172.21.15.78:0/162812941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f500094d0 con 0x7f3f6406e350 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f63fff640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f640c06b0 0x7f3f6414c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f68f93640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640a86a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b3f0 tx=0x7f3f5800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58011830 con 0x7f3f6406e350 2024-02-19T14:56:21.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58011990 con 0x7f3f6406e350 2024-02-19T14:56:21.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f641541b0 con 0x7f3f6406e350 2024-02-19T14:56:21.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58011d40 con 0x7f3f6406e350 2024-02-19T14:56:21.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.814+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f640a7bd0 con 0x7f3f6406e350 2024-02-19T14:56:21.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.815+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f64074990 con 0x7f3f6406e350 2024-02-19T14:56:21.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.816+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3f5801a400 con 0x7f3f6406e350 2024-02-19T14:56:21.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.817+0000 7f3f617fa640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3f4806b810 0x7f3f4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:21.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.817+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3f5808a1f0 con 0x7f3f6406e350 2024-02-19T14:56:21.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.817+0000 7f3f63fff640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3f4806b810 0x7f3f4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:21.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.817+0000 7f3f63fff640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3f4806b810 0x7f3f4806dcd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3f50036000 tx=0x7f3f50008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:21.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.821+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f58054a90 con 0x7f3f6406e350 2024-02-19T14:56:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:21.999+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f64072ee0 con 0x7f3f6406e350 2024-02-19T14:56:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.000+0000 7f3f617fa640 1 -- 172.21.15.78:0/162812941 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16888 (secure 0 0 0) 0x7f3f58058150 con 0x7f3f6406e350 2024-02-19T14:56:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:22.004 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":124,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:20.328626+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:16.897563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:17.646704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3f4806b810 msgr2=0x7f3f4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3f4806b810 0x7f3f4806dcd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3f50036000 tx=0x7f3f50008030 comp rx=0 tx=0).stop 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 msgr2=0x7f3f640a86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640a86a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b3f0 tx=0x7f3f5800b8b0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 shutdown_connections 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3f4806b810 0x7f3f4806dcd0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f64152610 0x7f3f6414ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f640c06b0 0x7f3f6414c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 --2- 172.21.15.78:0/162812941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6406e350 0x7f3f640a86a0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 >> 172.21.15.78:0/162812941 conn(0x7f3f6406b270 msgr2=0x7f3f641503f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:22.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 shutdown_connections 2024-02-19T14:56:22.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.002+0000 7f3f6b1f7640 1 -- 172.21.15.78:0/162812941 wait complete. 2024-02-19T14:56:22.016 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:56:22.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:56:22.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.115+0000 7f61013c9640 1 Processor -- start 2024-02-19T14:56:22.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.115+0000 7f61013c9640 1 -- start start 2024-02-19T14:56:22.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f61013c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f60fc0c5560 0x7f60fc0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f61013c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60fc070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f61013c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60fc0c65a0 con 0x7f60fc0c7910 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f61013c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60fc0c7530 con 0x7f60fc0c5560 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f61013c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1537b0 con 0x7f60fc070720 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60fa7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f60fc0c7910 0x7f60fc154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55050/0 (socket says 172.21.15.78:55050) 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60fa7fc640 1 -- 172.21.15.78:0/2898800363 learned_addr learned my addr 172.21.15.78:0/2898800363 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362959447 0 0) 0x7f60fc0c65a0 con 0x7f60fc0c7910 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60fb7fe640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60fc070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e4003520 con 0x7f60fc0c7910 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60faffd640 1 --2- 172.21.15.78:0/2898800363 >> v2:172.21.15.150:3300/0 conn(0x7f60fc0c5560 0x7f60fc0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987005827 0 0) 0x7f60e4003520 con 0x7f60fc0c7910 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 msgr2=0x7f60fc070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60fc070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 >> v2:172.21.15.150:3300/0 conn(0x7f60fc0c5560 msgr2=0x7f60fc0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 --2- 172.21.15.78:0/2898800363 >> v2:172.21.15.150:3300/0 conn(0x7f60fc0c5560 0x7f60fc0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc153a90 con 0x7f60fc0c7910 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f60f0001670 con 0x7f60fc0c7910 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.116+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 >> v1:172.21.15.78:6789/0 conn(0x7f60fc0c7910 legacy=0x7f60fc154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60f9ffb640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e4003970 0x7f60e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60f9ffb640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60f9ffb640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0c7530 con 0x7f60e4004d00 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1537b0 con 0x7f60e4003970 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e400ba00 con 0x7f60fc070720 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fb7fe640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60faffd640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e4003970 0x7f60e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fa7fc640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fb7fe640 1 -- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 msgr2=0x7f60e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fb7fe640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60e40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fb7fe640 1 -- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e4003970 msgr2=0x7f60e4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fb7fe640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e4003970 0x7f60e4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fb7fe640 1 -- 172.21.15.78:0/2898800363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e4003520 con 0x7f60e4004d00 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.117+0000 7f60fa7fc640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc070720 0x7f60e40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f60fb7fe640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60e4005180 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009fc0 tx=0x7f60e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b02aa91ac0e6190 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f402cbf0 con 0x7f60e4004d00 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f400f040 con 0x7f60e4004d00 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f60f9ffb640 1 -- 172.21.15.78:0/2898800363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4034440 con 0x7f60e4004d00 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f61013c9640 1 -- 172.21.15.78:0/2898800363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 msgr2=0x7f60e4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f61013c9640 1 --2- 172.21.15.78:0/2898800363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60e4005180 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009fc0 tx=0x7f60e40057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f60fb7fe640 1 -- 172.21.15.78:0/2898800363 reap_dead start 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f61013c9640 1 -- 172.21.15.78:0/2898800363 shutdown_connections 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f61013c9640 1 -- 172.21.15.78:0/2898800363 >> 172.21.15.78:0/2898800363 conn(0x7f60fc06b4b0 msgr2=0x7f60fc0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f61013c9640 1 -- 172.21.15.78:0/2898800363 shutdown_connections 2024-02-19T14:56:22.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.118+0000 7f61013c9640 1 -- 172.21.15.78:0/2898800363 wait complete. 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 Processor -- start 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 -- start start 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60fc1e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc0bd6c0 0x7f60fc0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60fc1ebea0 0x7f60fc14cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1eba80 con 0x7f60e4004d00 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1eb730 con 0x7f60fc1ebea0 2024-02-19T14:56:22.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f61013c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1e9100 con 0x7f60fc0bd6c0 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f60faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60fc1e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f60faffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60fc1e8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54578/0 (socket says 172.21.15.78:54578) 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.120+0000 7f60faffd640 1 -- 172.21.15.78:0/2092630526 learned_addr learned my addr 172.21.15.78:0/2092630526 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60fb7fe640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60fc1ebea0 0x7f60fc14cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60fa7fc640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc0bd6c0 0x7f60fc0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60faffd640 1 -- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc0bd6c0 msgr2=0x7f60fc0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60faffd640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc0bd6c0 0x7f60fc0ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60faffd640 1 -- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60fc1ebea0 msgr2=0x7f60fc14cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60faffd640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60fc1ebea0 0x7f60fc14cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60faffd640 1 -- 172.21.15.78:0/2092630526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc0c5ac0 con 0x7f60e4004d00 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60faffd640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60fc1e8bc0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f60ec011990 tx=0x7f60ec011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec00fad0 con 0x7f60e4004d00 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ec00fc30 con 0x7f60e4004d00 2024-02-19T14:56:22.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc0aa160 con 0x7f60e4004d00 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec014440 con 0x7f60e4004d00 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.121+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc0aa720 con 0x7f60e4004d00 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.122+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f60ec01d400 con 0x7f60e4004d00 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.122+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc074990 con 0x7f60e4004d00 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.123+0000 7f60e37fe640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f60e806b840 0x7f60e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.123+0000 7f60fa7fc640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f60e806b840 0x7f60e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.123+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f60ec08ab70 con 0x7f60e4004d00 2024-02-19T14:56:22.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.123+0000 7f60fa7fc640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f60e806b840 0x7f60e806dd00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60fc06fea0 tx=0x7f60f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.127+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f60ec055490 con 0x7f60e4004d00 2024-02-19T14:56:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.356+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f60fc0c5940 con 0x7f60e806b840 2024-02-19T14:56:22.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.363+0000 7f60e37fe640 1 -- 172.21.15.78:0/2092630526 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277809 (secure 0 0 0) 0x7f60f0010980 con 0x7f60e806b840 2024-02-19T14:56:22.368 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:22.370 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:56:22.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f60e806b840 msgr2=0x7f60e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f60e806b840 0x7f60e806dd00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60fc06fea0 tx=0x7f60f0009040 comp rx=0 tx=0).stop 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 msgr2=0x7f60fc1e8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60fc1e8bc0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f60ec011990 tx=0x7f60ec011e50 comp rx=0 tx=0).stop 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 shutdown_connections 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f60e806b840 0x7f60e806dd00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60fc1ebea0 0x7f60fc14cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60fc0bd6c0 0x7f60fc0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 --2- 172.21.15.78:0/2092630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60e4004d00 0x7f60fc1e8bc0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 >> 172.21.15.78:0/2092630526 conn(0x7f60fc06b4b0 msgr2=0x7f60fc0c4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 shutdown_connections 2024-02-19T14:56:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.366+0000 7f61013c9640 1 -- 172.21.15.78:0/2092630526 wait complete. 2024-02-19T14:56:22.380 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-02-19T14:56:22.111024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262452,"kb_used_data":13000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712268,"statfs":{"total":773094113280,"available":772825362432,"internally_reserved":0,"allocated":13312000,"data_stored":7041680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785582"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328970+0000","last_change":"2024-02-19T14:56:17.328970+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328970+0000","last_undegraded":"2024-02-19T14:56:17.328970+0000","last_fullsized":"2024-02-19T14:56:17.328970+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326761+0000","last_change":"2024-02-19T14:56:17.326761+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326761+0000","last_undegraded":"2024-02-19T14:56:17.326761+0000","last_fullsized":"2024-02-19T14:56:17.326761+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328900+0000","last_change":"2024-02-19T14:56:17.328900+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328900+0000","last_undegraded":"2024-02-19T14:56:17.328900+0000","last_fullsized":"2024-02-19T14:56:17.328900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329386+0000","last_change":"2024-02-19T14:56:17.329386+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329386+0000","last_undegraded":"2024-02-19T14:56:17.329386+0000","last_fullsized":"2024-02-19T14:56:17.329386+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329529+0000","last_change":"2024-02-19T14:56:17.329529+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329529+0000","last_undegraded":"2024-02-19T14:56:17.329529+0000","last_fullsized":"2024-02-19T14:56:17.329529+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326738+0000","last_change":"2024-02-19T14:56:17.326738+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326738+0000","last_undegraded":"2024-02-19T14:56:17.326738+0000","last_fullsized":"2024-02-19T14:56:17.326738+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325558+0000","last_change":"2024-02-19T14:56:17.325558+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325558+0000","last_undegraded":"2024-02-19T14:56:17.325558+0000","last_fullsized":"2024-02-19T14:56:17.325558+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328599+0000","last_change":"2024-02-19T14:56:17.328599+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328599+0000","last_undegraded":"2024-02-19T14:56:17.328599+0000","last_fullsized":"2024-02-19T14:56:17.328599+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328526+0000","last_change":"2024-02-19T14:56:17.328526+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328526+0000","last_undegraded":"2024-02-19T14:56:17.328526+0000","last_fullsized":"2024-02-19T14:56:17.328526+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330943+0000","last_change":"2024-02-19T14:56:17.330943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330943+0000","last_undegraded":"2024-02-19T14:56:17.330943+0000","last_fullsized":"2024-02-19T14:56:17.330943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327472+0000","last_change":"2024-02-19T14:56:17.327472+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327472+0000","last_undegraded":"2024-02-19T14:56:17.327472+0000","last_fullsized":"2024-02-19T14:56:17.327472+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327318+0000","last_change":"2024-02-19T14:56:17.327318+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327318+0000","last_undegraded":"2024-02-19T14:56:17.327318+0000","last_fullsized":"2024-02-19T14:56:17.327318+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330450+0000","last_change":"2024-02-19T14:56:17.330450+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330450+0000","last_undegraded":"2024-02-19T14:56:17.330450+0000","last_fullsized":"2024-02-19T14:56:17.330450+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328398+0000","last_change":"2024-02-19T14:56:17.328398+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328398+0000","last_undegraded":"2024-02-19T14:56:17.328398+0000","last_fullsized":"2024-02-19T14:56:17.328398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325514+0000","last_change":"2024-02-19T14:56:17.325514+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325514+0000","last_undegraded":"2024-02-19T14:56:17.325514+0000","last_fullsized":"2024-02-19T14:56:17.325514+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327919+0000","last_change":"2024-02-19T14:56:17.327919+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327919+0000","last_undegraded":"2024-02-19T14:56:17.327919+0000","last_fullsized":"2024-02-19T14:56:17.327919+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328967+0000","last_change":"2024-02-19T14:56:17.328967+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328967+0000","last_undegraded":"2024-02-19T14:56:17.328967+0000","last_fullsized":"2024-02-19T14:56:17.328967+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325076+0000","last_change":"2024-02-19T14:56:17.325076+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325076+0000","last_undegraded":"2024-02-19T14:56:17.325076+0000","last_fullsized":"2024-02-19T14:56:17.325076+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327092+0000","last_change":"2024-02-19T14:56:17.327092+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327092+0000","last_undegraded":"2024-02-19T14:56:17.327092+0000","last_fullsized":"2024-02-19T14:56:17.327092+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326808+0000","last_change":"2024-02-19T14:56:17.326808+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326808+0000","last_undegraded":"2024-02-19T14:56:17.326808+0000","last_fullsized":"2024-02-19T14:56:17.326808+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.331308+0000","last_change":"2024-02-19T14:56:17.331308+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.331308+0000","last_undegraded":"2024-02-19T14:56:17.331308+0000","last_fullsized":"2024-02-19T14:56:17.331308+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327292+0000","last_change":"2024-02-19T14:56:17.327292+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327292+0000","last_undegraded":"2024-02-19T14:56:17.327292+0000","last_fullsized":"2024-02-19T14:56:17.327292+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325041+0000","last_change":"2024-02-19T14:56:17.325041+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325041+0000","last_undegraded":"2024-02-19T14:56:17.325041+0000","last_fullsized":"2024-02-19T14:56:17.325041+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330488+0000","last_change":"2024-02-19T14:56:17.330488+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330488+0000","last_undegraded":"2024-02-19T14:56:17.330488+0000","last_fullsized":"2024-02-19T14:56:17.330488+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327797+0000","last_change":"2024-02-19T14:56:17.327797+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327797+0000","last_undegraded":"2024-02-19T14:56:17.327797+0000","last_fullsized":"2024-02-19T14:56:17.327797+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326817+0000","last_change":"2024-02-19T14:56:17.326817+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326817+0000","last_undegraded":"2024-02-19T14:56:17.326817+0000","last_fullsized":"2024-02-19T14:56:17.326817+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325062+0000","last_change":"2024-02-19T14:56:17.325062+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325062+0000","last_undegraded":"2024-02-19T14:56:17.325062+0000","last_fullsized":"2024-02-19T14:56:17.325062+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327943+0000","last_change":"2024-02-19T14:56:17.327943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327943+0000","last_undegraded":"2024-02-19T14:56:17.327943+0000","last_fullsized":"2024-02-19T14:56:17.327943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327309+0000","last_change":"2024-02-19T14:56:17.327309+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327309+0000","last_undegraded":"2024-02-19T14:56:17.327309+0000","last_fullsized":"2024-02-19T14:56:17.327309+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.324976+0000","last_change":"2024-02-19T14:56:17.324976+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.324976+0000","last_undegraded":"2024-02-19T14:56:17.324976+0000","last_fullsized":"2024-02-19T14:56:17.324976+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326986+0000","last_change":"2024-02-19T14:56:17.326986+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326986+0000","last_undegraded":"2024-02-19T14:56:17.326986+0000","last_fullsized":"2024-02-19T14:56:17.326986+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328812+0000","last_change":"2024-02-19T14:56:17.328812+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328812+0000","last_undegraded":"2024-02-19T14:56:17.328812+0000","last_fullsized":"2024-02-19T14:56:17.328812+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":359,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-02-19T14:56:17.329716+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:17.329716+0000","last_peered":"2024-02-19T14:56:17.329716+0000","last_clean":"2024-02-19T14:56:17.329716+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:17.329716+0000","last_undegraded":"2024-02-19T14:56:17.329716+0000","last_fullsized":"2024-02-19T14:56:17.329716+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33240,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338600,"statfs":{"total":96636764160,"available":96602726400,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.46700000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.495},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.35199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69499999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.42699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67300000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66300000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.316}]},{"osd":5,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68799999999999994,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65800000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.59999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.59199999999999997},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.48499999999999999,"15min":0.46700000000000003},"min":{"1min":0.34999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.64000000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70899999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.313}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.40799999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.373,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69899999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.49199999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.75,"5min":0.872,"15min":0.872},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68200000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65400000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69799999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72099999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":0.877,"15min":0.877},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.57699999999999996},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.77900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.29599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.752},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.84999999999999998,"5min":1.03,"15min":1.03},"last":0.81000000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71399999999999997,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74199999999999999,"5min":0.746,"15min":0.746},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.877,"15min":0.877},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.376,"5min":0.37,"15min":0.37},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":1.038,"15min":1.038},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.378}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":1.038,"15min":1.038},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.629},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73299999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34100000000000003,"15min":0.31},"min":{"1min":0.22700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48999999999999999,"5min":3.694,"15min":3.694},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.189,"15min":0.189},"max":{"1min":0.47499999999999998,"5min":4.835,"15min":4.835},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.313,"15min":0.311},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.48199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42699999999999999,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58799999999999997,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38900000000000001,"15min":0.34699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54500000000000004,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77500000000000002,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76600000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.318}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.217,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.42199999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.498,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.307},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42199999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35399999999999998,"15min":0.36299999999999999},"min":{"1min":0.251,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47299999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57899999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57699999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.68500000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76400000000000001,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":1.01,"15min":1.01},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69899999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.026,"15min":1.026},"last":0.498},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33244,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338596,"statfs":{"total":96636764160,"available":96602722304,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.224,"15min":0.224},"max":{"1min":0.48399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58499999999999996,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79200000000000004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.58499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.443},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.86399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75600000000000001,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:56:22.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:56:22.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.519+0000 7fbcc7d93640 1 Processor -- start 2024-02-19T14:56:22.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.520+0000 7fbcc7d93640 1 -- start start 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc7d93640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbcc00c5560 0x7fbcc00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc7d93640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc00c65a0 con 0x7fbcc0070720 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc7d93640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc00c7530 con 0x7fbcc00c7910 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc7d93640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc01537b0 con 0x7fbcc00c5560 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc6330640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbcc0070720 0x7fbcc0070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55058/0 (socket says 172.21.15.78:55058) 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc6330640 1 -- 172.21.15.78:0/1649183416 learned_addr learned my addr 172.21.15.78:0/1649183416 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc5b2f640 1 --2- 172.21.15.78:0/1649183416 >> v2:172.21.15.150:3300/0 conn(0x7fbcc00c5560 0x7fbcc00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc4b2d640 1 -- 172.21.15.78:0/1649183416 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576651932 0 0) 0x7fbcc00c65a0 con 0x7fbcc0070720 2024-02-19T14:56:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc4b2d640 1 -- 172.21.15.78:0/1649183416 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcac003520 con 0x7fbcc0070720 2024-02-19T14:56:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc4b2d640 1 -- 172.21.15.78:0/1649183416 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 211092508 0 0) 0x7fbcac003520 con 0x7fbcc0070720 2024-02-19T14:56:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/1649183416 >> v1:172.21.15.150:6789/0 conn(0x7fbcc00c7910 legacy=0x7fbcc0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/1649183416 >> v1:172.21.15.78:6789/0 conn(0x7fbcc0070720 legacy=0x7fbcc0070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.521+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/1649183416 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc0153a90 con 0x7fbcc00c5560 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.522+0000 7fbcc5b2f640 1 --2- 172.21.15.78:0/1649183416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00c6ff0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8009d90 tx=0x7fbcb802bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd488e2b77d30d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.522+0000 7fbcc4b2d640 1 -- 172.21.15.78:0/1649183416 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc001670 con 0x7fbcc00c5560 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc4b2d640 1 -- 172.21.15.78:0/1649183416 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb802c820 con 0x7fbcc00c5560 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc4b2d640 1 -- 172.21.15.78:0/1649183416 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb802cb90 con 0x7fbcc00c5560 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 -- 172.21.15.78:0/1649183416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 msgr2=0x7fbcc00c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/1649183416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00c6ff0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8009d90 tx=0x7fbcb802bcf0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 -- 172.21.15.78:0/1649183416 shutdown_connections 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/1649183416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00c6ff0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 -- 172.21.15.78:0/1649183416 >> 172.21.15.78:0/1649183416 conn(0x7fbcc006b4b0 msgr2=0x7fbcc00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:22.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 -- 172.21.15.78:0/1649183416 shutdown_connections 2024-02-19T14:56:22.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.523+0000 7fbcc7d93640 1 -- 172.21.15.78:0/1649183416 wait complete. 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.524+0000 7fbcc7d93640 1 Processor -- start 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.524+0000 7fbcc7d93640 1 -- start start 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc7d93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 0x7fbcc01e8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc7d93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc7d93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc00c7910 0x7fbcc00aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc7d93640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00a8630 con 0x7fbcc0070720 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc7d93640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00a8910 con 0x7fbcc00c5560 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc7d93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc00a8bf0 con 0x7fbcc00c7910 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 0x7fbcc01e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 0x7fbcc01e8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54590/0 (socket says 172.21.15.78:54590) 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/4019560222 learned_addr learned my addr 172.21.15.78:0/4019560222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc532e640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc6330640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc00c7910 0x7fbcc00aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc00c7910 msgr2=0x7fbcc00aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc00c7910 0x7fbcc00aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 msgr2=0x7fbcc00a96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00a96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.525+0000 7fbcc5b2f640 1 -- 172.21.15.78:0/4019560222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb8009a70 con 0x7fbcc0070720 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.526+0000 7fbcc5b2f640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 0x7fbcc01e8600 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8000f80 tx=0x7fbcb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.526+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8009450 con 0x7fbcc0070720 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.526+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb802ce10 con 0x7fbcc0070720 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.526+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb803e670 con 0x7fbcc0070720 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.526+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc00c5ac0 con 0x7fbcc0070720 2024-02-19T14:56:22.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.526+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc00a91b0 con 0x7fbcc0070720 2024-02-19T14:56:22.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.527+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc0074990 con 0x7fbcc0070720 2024-02-19T14:56:22.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.527+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbcb803b050 con 0x7fbcc0070720 2024-02-19T14:56:22.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.528+0000 7fbcb67fc640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbca406b810 0x7fbca406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.528+0000 7fbcc532e640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbca406b810 0x7fbca406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.528+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbcb80ac9d0 con 0x7fbcc0070720 2024-02-19T14:56:22.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.528+0000 7fbcc532e640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbca406b810 0x7fbca406dcd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0152410 tx=0x7fbcb0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.532+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbcb80774c0 con 0x7fbcc0070720 2024-02-19T14:56:22.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.706+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbcc00c5940 con 0x7fbca406b810 2024-02-19T14:56:22.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.713+0000 7fbcb67fc640 1 -- 172.21.15.78:0/4019560222 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277809 (secure 0 0 0) 0x7fbcb00014a0 con 0x7fbca406b810 2024-02-19T14:56:22.718 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:22.719 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbca406b810 msgr2=0x7fbca406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbca406b810 0x7fbca406dcd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0152410 tx=0x7fbcb0009040 comp rx=0 tx=0).stop 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 msgr2=0x7fbcc01e8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 0x7fbcc01e8600 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8000f80 tx=0x7fbcb8004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 shutdown_connections 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbca406b810 0x7fbca406dcd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc00c7910 0x7fbcc00aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc00c5560 0x7fbcc00a96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 --2- 172.21.15.78:0/4019560222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc0070720 0x7fbcc01e8600 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 >> 172.21.15.78:0/4019560222 conn(0x7fbcc006b4b0 msgr2=0x7fbcc0151270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 shutdown_connections 2024-02-19T14:56:22.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.716+0000 7fbcc7d93640 1 -- 172.21.15.78:0/4019560222 wait complete. 2024-02-19T14:56:22.732 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-02-19T14:56:22.111024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262452,"kb_used_data":13000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712268,"statfs":{"total":773094113280,"available":772825362432,"internally_reserved":0,"allocated":13312000,"data_stored":7041680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785582"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328970+0000","last_change":"2024-02-19T14:56:17.328970+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328970+0000","last_undegraded":"2024-02-19T14:56:17.328970+0000","last_fullsized":"2024-02-19T14:56:17.328970+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326761+0000","last_change":"2024-02-19T14:56:17.326761+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326761+0000","last_undegraded":"2024-02-19T14:56:17.326761+0000","last_fullsized":"2024-02-19T14:56:17.326761+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328900+0000","last_change":"2024-02-19T14:56:17.328900+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328900+0000","last_undegraded":"2024-02-19T14:56:17.328900+0000","last_fullsized":"2024-02-19T14:56:17.328900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329386+0000","last_change":"2024-02-19T14:56:17.329386+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329386+0000","last_undegraded":"2024-02-19T14:56:17.329386+0000","last_fullsized":"2024-02-19T14:56:17.329386+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329529+0000","last_change":"2024-02-19T14:56:17.329529+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329529+0000","last_undegraded":"2024-02-19T14:56:17.329529+0000","last_fullsized":"2024-02-19T14:56:17.329529+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326738+0000","last_change":"2024-02-19T14:56:17.326738+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326738+0000","last_undegraded":"2024-02-19T14:56:17.326738+0000","last_fullsized":"2024-02-19T14:56:17.326738+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325558+0000","last_change":"2024-02-19T14:56:17.325558+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325558+0000","last_undegraded":"2024-02-19T14:56:17.325558+0000","last_fullsized":"2024-02-19T14:56:17.325558+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328599+0000","last_change":"2024-02-19T14:56:17.328599+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328599+0000","last_undegraded":"2024-02-19T14:56:17.328599+0000","last_fullsized":"2024-02-19T14:56:17.328599+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328526+0000","last_change":"2024-02-19T14:56:17.328526+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328526+0000","last_undegraded":"2024-02-19T14:56:17.328526+0000","last_fullsized":"2024-02-19T14:56:17.328526+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330943+0000","last_change":"2024-02-19T14:56:17.330943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330943+0000","last_undegraded":"2024-02-19T14:56:17.330943+0000","last_fullsized":"2024-02-19T14:56:17.330943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327472+0000","last_change":"2024-02-19T14:56:17.327472+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327472+0000","last_undegraded":"2024-02-19T14:56:17.327472+0000","last_fullsized":"2024-02-19T14:56:17.327472+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327318+0000","last_change":"2024-02-19T14:56:17.327318+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327318+0000","last_undegraded":"2024-02-19T14:56:17.327318+0000","last_fullsized":"2024-02-19T14:56:17.327318+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330450+0000","last_change":"2024-02-19T14:56:17.330450+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330450+0000","last_undegraded":"2024-02-19T14:56:17.330450+0000","last_fullsized":"2024-02-19T14:56:17.330450+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328398+0000","last_change":"2024-02-19T14:56:17.328398+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328398+0000","last_undegraded":"2024-02-19T14:56:17.328398+0000","last_fullsized":"2024-02-19T14:56:17.328398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325514+0000","last_change":"2024-02-19T14:56:17.325514+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325514+0000","last_undegraded":"2024-02-19T14:56:17.325514+0000","last_fullsized":"2024-02-19T14:56:17.325514+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327919+0000","last_change":"2024-02-19T14:56:17.327919+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327919+0000","last_undegraded":"2024-02-19T14:56:17.327919+0000","last_fullsized":"2024-02-19T14:56:17.327919+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328967+0000","last_change":"2024-02-19T14:56:17.328967+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328967+0000","last_undegraded":"2024-02-19T14:56:17.328967+0000","last_fullsized":"2024-02-19T14:56:17.328967+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325076+0000","last_change":"2024-02-19T14:56:17.325076+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325076+0000","last_undegraded":"2024-02-19T14:56:17.325076+0000","last_fullsized":"2024-02-19T14:56:17.325076+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327092+0000","last_change":"2024-02-19T14:56:17.327092+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327092+0000","last_undegraded":"2024-02-19T14:56:17.327092+0000","last_fullsized":"2024-02-19T14:56:17.327092+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326808+0000","last_change":"2024-02-19T14:56:17.326808+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326808+0000","last_undegraded":"2024-02-19T14:56:17.326808+0000","last_fullsized":"2024-02-19T14:56:17.326808+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.331308+0000","last_change":"2024-02-19T14:56:17.331308+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.331308+0000","last_undegraded":"2024-02-19T14:56:17.331308+0000","last_fullsized":"2024-02-19T14:56:17.331308+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327292+0000","last_change":"2024-02-19T14:56:17.327292+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327292+0000","last_undegraded":"2024-02-19T14:56:17.327292+0000","last_fullsized":"2024-02-19T14:56:17.327292+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325041+0000","last_change":"2024-02-19T14:56:17.325041+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325041+0000","last_undegraded":"2024-02-19T14:56:17.325041+0000","last_fullsized":"2024-02-19T14:56:17.325041+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330488+0000","last_change":"2024-02-19T14:56:17.330488+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330488+0000","last_undegraded":"2024-02-19T14:56:17.330488+0000","last_fullsized":"2024-02-19T14:56:17.330488+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327797+0000","last_change":"2024-02-19T14:56:17.327797+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327797+0000","last_undegraded":"2024-02-19T14:56:17.327797+0000","last_fullsized":"2024-02-19T14:56:17.327797+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326817+0000","last_change":"2024-02-19T14:56:17.326817+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326817+0000","last_undegraded":"2024-02-19T14:56:17.326817+0000","last_fullsized":"2024-02-19T14:56:17.326817+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325062+0000","last_change":"2024-02-19T14:56:17.325062+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325062+0000","last_undegraded":"2024-02-19T14:56:17.325062+0000","last_fullsized":"2024-02-19T14:56:17.325062+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327943+0000","last_change":"2024-02-19T14:56:17.327943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327943+0000","last_undegraded":"2024-02-19T14:56:17.327943+0000","last_fullsized":"2024-02-19T14:56:17.327943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327309+0000","last_change":"2024-02-19T14:56:17.327309+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327309+0000","last_undegraded":"2024-02-19T14:56:17.327309+0000","last_fullsized":"2024-02-19T14:56:17.327309+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.324976+0000","last_change":"2024-02-19T14:56:17.324976+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.324976+0000","last_undegraded":"2024-02-19T14:56:17.324976+0000","last_fullsized":"2024-02-19T14:56:17.324976+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326986+0000","last_change":"2024-02-19T14:56:17.326986+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326986+0000","last_undegraded":"2024-02-19T14:56:17.326986+0000","last_fullsized":"2024-02-19T14:56:17.326986+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328812+0000","last_change":"2024-02-19T14:56:17.328812+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328812+0000","last_undegraded":"2024-02-19T14:56:17.328812+0000","last_fullsized":"2024-02-19T14:56:17.328812+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":359,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-02-19T14:56:17.329716+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:17.329716+0000","last_peered":"2024-02-19T14:56:17.329716+0000","last_clean":"2024-02-19T14:56:17.329716+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:17.329716+0000","last_undegraded":"2024-02-19T14:56:17.329716+0000","last_fullsized":"2024-02-19T14:56:17.329716+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33240,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338600,"statfs":{"total":96636764160,"available":96602726400,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.46700000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.495},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.35199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69499999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.42699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67300000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66300000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.316}]},{"osd":5,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68799999999999994,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65800000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.59999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.59199999999999997},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.48499999999999999,"15min":0.46700000000000003},"min":{"1min":0.34999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.64000000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70899999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.313}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.40799999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.373,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69899999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.49199999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.75,"5min":0.872,"15min":0.872},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68200000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65400000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69799999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72099999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":0.877,"15min":0.877},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.57699999999999996},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.77900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.29599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.752},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.84999999999999998,"5min":1.03,"15min":1.03},"last":0.81000000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71399999999999997,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74199999999999999,"5min":0.746,"15min":0.746},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.877,"15min":0.877},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.376,"5min":0.37,"15min":0.37},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":1.038,"15min":1.038},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.378}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":1.038,"15min":1.038},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.629},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73299999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34100000000000003,"15min":0.31},"min":{"1min":0.22700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48999999999999999,"5min":3.694,"15min":3.694},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.189,"15min":0.189},"max":{"1min":0.47499999999999998,"5min":4.835,"15min":4.835},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.313,"15min":0.311},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.48199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42699999999999999,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58799999999999997,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38900000000000001,"15min":0.34699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54500000000000004,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77500000000000002,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76600000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.318}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.217,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.42199999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.498,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.307},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42199999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35399999999999998,"15min":0.36299999999999999},"min":{"1min":0.251,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47299999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57899999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57699999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.68500000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76400000000000001,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":1.01,"15min":1.01},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69899999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.026,"15min":1.026},"last":0.498},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33244,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338596,"statfs":{"total":96636764160,"available":96602722304,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.224,"15min":0.224},"max":{"1min":0.48399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58499999999999996,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79200000000000004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.58499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.443},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.86399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75600000000000001,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:56:22.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:56:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 Processor -- start 2024-02-19T14:56:22.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 -- start start 2024-02-19T14:56:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33e00c78a0 0x7f33e0159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33e00c4d80 0x7f33e00c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e006ffa0 con 0x7f33e00c48f0 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e006b720 con 0x7f33e00c78a0 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.871+0000 7f33e4eb5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e006fc70 con 0x7f33e00c4d80 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33e00c78a0 0x7f33e0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33e00c78a0 0x7f33e0159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54604/0 (socket says 172.21.15.78:54604) 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 -- 172.21.15.78:0/2716324683 learned_addr learned my addr 172.21.15.78:0/2716324683 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 -- 172.21.15.78:0/2716324683 >> v2:172.21.15.150:3300/0 conn(0x7f33e00c4d80 msgr2=0x7f33e00c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 --2- 172.21.15.78:0/2716324683 >> v2:172.21.15.150:3300/0 conn(0x7f33e00c4d80 0x7f33e00c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 -- 172.21.15.78:0/2716324683 >> v1:172.21.15.150:6789/0 conn(0x7f33e00c48f0 legacy=0x7f33e01523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:22.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.872+0000 7f33de59c640 1 -- 172.21.15.78:0/2716324683 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e006b930 con 0x7f33e00c78a0 2024-02-19T14:56:22.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33de59c640 1 --2- 172.21.15.78:0/2716324683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c78a0 0x7f33e0159010 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f33e00c1ef0 tx=0x7f33d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a494416ff059f062 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33ddd9b640 1 -- 172.21.15.78:0/2716324683 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d00317e0 con 0x7f33e00c78a0 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33ddd9b640 1 -- 172.21.15.78:0/2716324683 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0031940 con 0x7f33e00c78a0 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33ddd9b640 1 -- 172.21.15.78:0/2716324683 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0031cd0 con 0x7f33e00c78a0 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2716324683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c78a0 msgr2=0x7f33e0159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2716324683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c78a0 0x7f33e0159010 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f33e00c1ef0 tx=0x7f33d002b880 comp rx=0 tx=0).stop 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2716324683 shutdown_connections 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2716324683 >> v2:172.21.15.150:3300/0 conn(0x7f33e00c4d80 0x7f33e00c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2716324683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c78a0 0x7f33e0159010 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2716324683 >> 172.21.15.78:0/2716324683 conn(0x7f33e0066e70 msgr2=0x7f33e00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2716324683 shutdown_connections 2024-02-19T14:56:22.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.873+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2716324683 wait complete. 2024-02-19T14:56:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 Processor -- start 2024-02-19T14:56:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 -- start start 2024-02-19T14:56:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33e00c48f0 0x7f33e01491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 0x7f33e00c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e00c78a0 0x7f33e01ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e01eefd0 con 0x7f33e00c4d80 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e01496f0 con 0x7f33e00c78a0 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33e4eb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e00c2140 con 0x7f33e00c48f0 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 0x7f33e00c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33d7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 0x7f33e00c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54612/0 (socket says 172.21.15.78:54612) 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.875+0000 7f33d7fff640 1 -- 172.21.15.78:0/2041090153 learned_addr learned my addr 172.21.15.78:0/2041090153 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33de59c640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33e00c48f0 0x7f33e01491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33ded9d640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e00c78a0 0x7f33e01ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d7fff640 1 -- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33e00c48f0 msgr2=0x7f33e01491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d7fff640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33e00c48f0 0x7f33e01491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d7fff640 1 -- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e00c78a0 msgr2=0x7f33e01ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d7fff640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e00c78a0 0x7f33e01ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d7fff640 1 -- 172.21.15.78:0/2041090153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00094d0 con 0x7f33e00c4d80 2024-02-19T14:56:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33ded9d640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e00c78a0 0x7f33e01ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d7fff640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 0x7f33e00c2f00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f33c800b4d0 tx=0x7f33c800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8010870 con 0x7f33e00c4d80 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c80109d0 con 0x7f33e00c4d80 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e00c2420 con 0x7f33e00c4d80 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8010d80 con 0x7f33e00c4d80 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.876+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e00c29e0 con 0x7f33e00c4d80 2024-02-19T14:56:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.877+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e0074990 con 0x7f33e00c4d80 2024-02-19T14:56:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.878+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f33c800f3c0 con 0x7f33e00c4d80 2024-02-19T14:56:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.878+0000 7f33d6ffd640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f33c006b810 0x7f33c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.879+0000 7f33de59c640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f33c006b810 0x7f33c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.879+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f33c808a230 con 0x7f33e00c4d80 2024-02-19T14:56:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.879+0000 7f33de59c640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f33c006b810 0x7f33c006dcd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f33d0036000 tx=0x7f33d003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:22.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:22.883+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f33c8054b80 con 0x7f33e00c4d80 2024-02-19T14:56:23.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.056+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33e00aca00 con 0x7f33c006b810 2024-02-19T14:56:23.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.063+0000 7f33d6ffd640 1 -- 172.21.15.78:0/2041090153 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277809 (secure 0 0 0) 0x7f33e00aca00 con 0x7f33c006b810 2024-02-19T14:56:23.069 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:23.070 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:56:23.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.066+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f33c006b810 msgr2=0x7f33c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.066+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f33c006b810 0x7f33c006dcd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f33d0036000 tx=0x7f33d003d040 comp rx=0 tx=0).stop 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.066+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 msgr2=0x7f33e00c2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.066+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 0x7f33e00c2f00 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f33c800b4d0 tx=0x7f33c800b990 comp rx=0 tx=0).stop 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 shutdown_connections 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f33c006b810 0x7f33c006dcd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33e00c4d80 0x7f33e00c2f00 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e00c78a0 0x7f33e01ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 --2- 172.21.15.78:0/2041090153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33e00c48f0 0x7f33e01491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 >> 172.21.15.78:0/2041090153 conn(0x7f33e0066e70 msgr2=0x7f33e00c5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 shutdown_connections 2024-02-19T14:56:23.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.067+0000 7f33e4eb5640 1 -- 172.21.15.78:0/2041090153 wait complete. 2024-02-19T14:56:23.080 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-02-19T14:56:22.111024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262452,"kb_used_data":13000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712268,"statfs":{"total":773094113280,"available":772825362432,"internally_reserved":0,"allocated":13312000,"data_stored":7041680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785582"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328970+0000","last_change":"2024-02-19T14:56:17.328970+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328970+0000","last_undegraded":"2024-02-19T14:56:17.328970+0000","last_fullsized":"2024-02-19T14:56:17.328970+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326761+0000","last_change":"2024-02-19T14:56:17.326761+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326761+0000","last_undegraded":"2024-02-19T14:56:17.326761+0000","last_fullsized":"2024-02-19T14:56:17.326761+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328900+0000","last_change":"2024-02-19T14:56:17.328900+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328900+0000","last_undegraded":"2024-02-19T14:56:17.328900+0000","last_fullsized":"2024-02-19T14:56:17.328900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329386+0000","last_change":"2024-02-19T14:56:17.329386+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329386+0000","last_undegraded":"2024-02-19T14:56:17.329386+0000","last_fullsized":"2024-02-19T14:56:17.329386+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329529+0000","last_change":"2024-02-19T14:56:17.329529+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329529+0000","last_undegraded":"2024-02-19T14:56:17.329529+0000","last_fullsized":"2024-02-19T14:56:17.329529+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326738+0000","last_change":"2024-02-19T14:56:17.326738+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326738+0000","last_undegraded":"2024-02-19T14:56:17.326738+0000","last_fullsized":"2024-02-19T14:56:17.326738+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325558+0000","last_change":"2024-02-19T14:56:17.325558+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325558+0000","last_undegraded":"2024-02-19T14:56:17.325558+0000","last_fullsized":"2024-02-19T14:56:17.325558+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328599+0000","last_change":"2024-02-19T14:56:17.328599+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328599+0000","last_undegraded":"2024-02-19T14:56:17.328599+0000","last_fullsized":"2024-02-19T14:56:17.328599+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328526+0000","last_change":"2024-02-19T14:56:17.328526+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328526+0000","last_undegraded":"2024-02-19T14:56:17.328526+0000","last_fullsized":"2024-02-19T14:56:17.328526+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330943+0000","last_change":"2024-02-19T14:56:17.330943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330943+0000","last_undegraded":"2024-02-19T14:56:17.330943+0000","last_fullsized":"2024-02-19T14:56:17.330943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327472+0000","last_change":"2024-02-19T14:56:17.327472+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327472+0000","last_undegraded":"2024-02-19T14:56:17.327472+0000","last_fullsized":"2024-02-19T14:56:17.327472+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327318+0000","last_change":"2024-02-19T14:56:17.327318+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327318+0000","last_undegraded":"2024-02-19T14:56:17.327318+0000","last_fullsized":"2024-02-19T14:56:17.327318+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330450+0000","last_change":"2024-02-19T14:56:17.330450+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330450+0000","last_undegraded":"2024-02-19T14:56:17.330450+0000","last_fullsized":"2024-02-19T14:56:17.330450+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328398+0000","last_change":"2024-02-19T14:56:17.328398+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328398+0000","last_undegraded":"2024-02-19T14:56:17.328398+0000","last_fullsized":"2024-02-19T14:56:17.328398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325514+0000","last_change":"2024-02-19T14:56:17.325514+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325514+0000","last_undegraded":"2024-02-19T14:56:17.325514+0000","last_fullsized":"2024-02-19T14:56:17.325514+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327919+0000","last_change":"2024-02-19T14:56:17.327919+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327919+0000","last_undegraded":"2024-02-19T14:56:17.327919+0000","last_fullsized":"2024-02-19T14:56:17.327919+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328967+0000","last_change":"2024-02-19T14:56:17.328967+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328967+0000","last_undegraded":"2024-02-19T14:56:17.328967+0000","last_fullsized":"2024-02-19T14:56:17.328967+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325076+0000","last_change":"2024-02-19T14:56:17.325076+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325076+0000","last_undegraded":"2024-02-19T14:56:17.325076+0000","last_fullsized":"2024-02-19T14:56:17.325076+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327092+0000","last_change":"2024-02-19T14:56:17.327092+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327092+0000","last_undegraded":"2024-02-19T14:56:17.327092+0000","last_fullsized":"2024-02-19T14:56:17.327092+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326808+0000","last_change":"2024-02-19T14:56:17.326808+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326808+0000","last_undegraded":"2024-02-19T14:56:17.326808+0000","last_fullsized":"2024-02-19T14:56:17.326808+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.331308+0000","last_change":"2024-02-19T14:56:17.331308+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.331308+0000","last_undegraded":"2024-02-19T14:56:17.331308+0000","last_fullsized":"2024-02-19T14:56:17.331308+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327292+0000","last_change":"2024-02-19T14:56:17.327292+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327292+0000","last_undegraded":"2024-02-19T14:56:17.327292+0000","last_fullsized":"2024-02-19T14:56:17.327292+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325041+0000","last_change":"2024-02-19T14:56:17.325041+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325041+0000","last_undegraded":"2024-02-19T14:56:17.325041+0000","last_fullsized":"2024-02-19T14:56:17.325041+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330488+0000","last_change":"2024-02-19T14:56:17.330488+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330488+0000","last_undegraded":"2024-02-19T14:56:17.330488+0000","last_fullsized":"2024-02-19T14:56:17.330488+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327797+0000","last_change":"2024-02-19T14:56:17.327797+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327797+0000","last_undegraded":"2024-02-19T14:56:17.327797+0000","last_fullsized":"2024-02-19T14:56:17.327797+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326817+0000","last_change":"2024-02-19T14:56:17.326817+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326817+0000","last_undegraded":"2024-02-19T14:56:17.326817+0000","last_fullsized":"2024-02-19T14:56:17.326817+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325062+0000","last_change":"2024-02-19T14:56:17.325062+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325062+0000","last_undegraded":"2024-02-19T14:56:17.325062+0000","last_fullsized":"2024-02-19T14:56:17.325062+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327943+0000","last_change":"2024-02-19T14:56:17.327943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327943+0000","last_undegraded":"2024-02-19T14:56:17.327943+0000","last_fullsized":"2024-02-19T14:56:17.327943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327309+0000","last_change":"2024-02-19T14:56:17.327309+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327309+0000","last_undegraded":"2024-02-19T14:56:17.327309+0000","last_fullsized":"2024-02-19T14:56:17.327309+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.324976+0000","last_change":"2024-02-19T14:56:17.324976+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.324976+0000","last_undegraded":"2024-02-19T14:56:17.324976+0000","last_fullsized":"2024-02-19T14:56:17.324976+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326986+0000","last_change":"2024-02-19T14:56:17.326986+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326986+0000","last_undegraded":"2024-02-19T14:56:17.326986+0000","last_fullsized":"2024-02-19T14:56:17.326986+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328812+0000","last_change":"2024-02-19T14:56:17.328812+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328812+0000","last_undegraded":"2024-02-19T14:56:17.328812+0000","last_fullsized":"2024-02-19T14:56:17.328812+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":359,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-02-19T14:56:17.329716+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:17.329716+0000","last_peered":"2024-02-19T14:56:17.329716+0000","last_clean":"2024-02-19T14:56:17.329716+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:17.329716+0000","last_undegraded":"2024-02-19T14:56:17.329716+0000","last_fullsized":"2024-02-19T14:56:17.329716+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33240,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338600,"statfs":{"total":96636764160,"available":96602726400,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.46700000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.495},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.35199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69499999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.42699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67300000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66300000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.316}]},{"osd":5,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68799999999999994,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65800000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.59999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.59199999999999997},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.48499999999999999,"15min":0.46700000000000003},"min":{"1min":0.34999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.64000000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70899999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.313}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.40799999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.373,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69899999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.49199999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.75,"5min":0.872,"15min":0.872},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68200000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65400000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69799999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72099999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":0.877,"15min":0.877},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.57699999999999996},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.77900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.29599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.752},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.84999999999999998,"5min":1.03,"15min":1.03},"last":0.81000000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71399999999999997,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74199999999999999,"5min":0.746,"15min":0.746},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.877,"15min":0.877},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.376,"5min":0.37,"15min":0.37},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":1.038,"15min":1.038},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.378}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":1.038,"15min":1.038},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.629},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73299999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34100000000000003,"15min":0.31},"min":{"1min":0.22700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48999999999999999,"5min":3.694,"15min":3.694},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.189,"15min":0.189},"max":{"1min":0.47499999999999998,"5min":4.835,"15min":4.835},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.313,"15min":0.311},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.48199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42699999999999999,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58799999999999997,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38900000000000001,"15min":0.34699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54500000000000004,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77500000000000002,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76600000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.318}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.217,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.42199999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.498,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.307},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42199999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35399999999999998,"15min":0.36299999999999999},"min":{"1min":0.251,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47299999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57899999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57699999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.68500000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76400000000000001,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":1.01,"15min":1.01},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69899999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.026,"15min":1.026},"last":0.498},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33244,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338596,"statfs":{"total":96636764160,"available":96602722304,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.224,"15min":0.224},"max":{"1min":0.48399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58499999999999996,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79200000000000004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.58499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.443},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.86399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75600000000000001,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:56:23.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3f is not active+clean 2024-02-19T14:56:23.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3e is not active+clean 2024-02-19T14:56:23.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3d is not active+clean 2024-02-19T14:56:23.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3c is not active+clean 2024-02-19T14:56:23.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3b is not active+clean 2024-02-19T14:56:23.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3a is not active+clean 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.39 is not active+clean 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.38 is not active+clean 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.37 is not active+clean 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.36 is not active+clean 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.35 is not active+clean 2024-02-19T14:56:23.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.34 is not active+clean 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.33 is not active+clean 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.32 is not active+clean 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.31 is not active+clean 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.30 is not active+clean 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2f is not active+clean 2024-02-19T14:56:23.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2e is not active+clean 2024-02-19T14:56:23.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2d is not active+clean 2024-02-19T14:56:23.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2c is not active+clean 2024-02-19T14:56:23.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2b is not active+clean 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2a is not active+clean 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.29 is not active+clean 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.28 is not active+clean 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.27 is not active+clean 2024-02-19T14:56:23.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.26 is not active+clean 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.25 is not active+clean 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.24 is not active+clean 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.23 is not active+clean 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.22 is not active+clean 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.21 is not active+clean 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.20 is not active+clean 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.18 is not active+clean 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1f is not active+clean 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328970+0000', 'last_change': '2024-02-19T14:56:17.328970+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328970+0000', 'last_undegraded': '2024-02-19T14:56:17.328970+0000', 'last_fullsized': '2024-02-19T14:56:17.328970+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:56:23.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.19 is not active+clean 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1e is not active+clean 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.326761+0000', 'last_change': '2024-02-19T14:56:17.326761+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.326761+0000', 'last_undegraded': '2024-02-19T14:56:17.326761+0000', 'last_fullsized': '2024-02-19T14:56:17.326761+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1a is not active+clean 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1d is not active+clean 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328900+0000', 'last_change': '2024-02-19T14:56:17.328900+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328900+0000', 'last_undegraded': '2024-02-19T14:56:17.328900+0000', 'last_fullsized': '2024-02-19T14:56:17.328900+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1b is not active+clean 2024-02-19T14:56:23.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1c is not active+clean 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.329386+0000', 'last_change': '2024-02-19T14:56:17.329386+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.329386+0000', 'last_undegraded': '2024-02-19T14:56:17.329386+0000', 'last_fullsized': '2024-02-19T14:56:17.329386+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1c is not active+clean 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1b is not active+clean 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.329529+0000', 'last_change': '2024-02-19T14:56:17.329529+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.329529+0000', 'last_undegraded': '2024-02-19T14:56:17.329529+0000', 'last_fullsized': '2024-02-19T14:56:17.329529+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1d is not active+clean 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1a is not active+clean 2024-02-19T14:56:23.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.326738+0000', 'last_change': '2024-02-19T14:56:17.326738+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.326738+0000', 'last_undegraded': '2024-02-19T14:56:17.326738+0000', 'last_fullsized': '2024-02-19T14:56:17.326738+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1e is not active+clean 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.19 is not active+clean 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.325558+0000', 'last_change': '2024-02-19T14:56:17.325558+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.325558+0000', 'last_undegraded': '2024-02-19T14:56:17.325558+0000', 'last_fullsized': '2024-02-19T14:56:17.325558+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1f is not active+clean 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.18 is not active+clean 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328599+0000', 'last_change': '2024-02-19T14:56:17.328599+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328599+0000', 'last_undegraded': '2024-02-19T14:56:17.328599+0000', 'last_fullsized': '2024-02-19T14:56:17.328599+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:56:23.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.10 is not active+clean 2024-02-19T14:56:23.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.17 is not active+clean 2024-02-19T14:56:23.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328526+0000', 'last_change': '2024-02-19T14:56:17.328526+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328526+0000', 'last_undegraded': '2024-02-19T14:56:17.328526+0000', 'last_fullsized': '2024-02-19T14:56:17.328526+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:56:23.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.11 is not active+clean 2024-02-19T14:56:23.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.16 is not active+clean 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.330943+0000', 'last_change': '2024-02-19T14:56:17.330943+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.330943+0000', 'last_undegraded': '2024-02-19T14:56:17.330943+0000', 'last_fullsized': '2024-02-19T14:56:17.330943+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.12 is not active+clean 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.15 is not active+clean 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327472+0000', 'last_change': '2024-02-19T14:56:17.327472+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327472+0000', 'last_undegraded': '2024-02-19T14:56:17.327472+0000', 'last_fullsized': '2024-02-19T14:56:17.327472+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.13 is not active+clean 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.14 is not active+clean 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327318+0000', 'last_change': '2024-02-19T14:56:17.327318+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327318+0000', 'last_undegraded': '2024-02-19T14:56:17.327318+0000', 'last_fullsized': '2024-02-19T14:56:17.327318+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.14 is not active+clean 2024-02-19T14:56:23.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.13 is not active+clean 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.330450+0000', 'last_change': '2024-02-19T14:56:17.330450+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.330450+0000', 'last_undegraded': '2024-02-19T14:56:17.330450+0000', 'last_fullsized': '2024-02-19T14:56:17.330450+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.16 is not active+clean 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.11 is not active+clean 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328398+0000', 'last_change': '2024-02-19T14:56:17.328398+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328398+0000', 'last_undegraded': '2024-02-19T14:56:17.328398+0000', 'last_fullsized': '2024-02-19T14:56:17.328398+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.17 is not active+clean 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.10 is not active+clean 2024-02-19T14:56:23.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.325514+0000', 'last_change': '2024-02-19T14:56:17.325514+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.325514+0000', 'last_undegraded': '2024-02-19T14:56:17.325514+0000', 'last_fullsized': '2024-02-19T14:56:17.325514+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.8 is not active+clean 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.f is not active+clean 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327919+0000', 'last_change': '2024-02-19T14:56:17.327919+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327919+0000', 'last_undegraded': '2024-02-19T14:56:17.327919+0000', 'last_fullsized': '2024-02-19T14:56:17.327919+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.9 is not active+clean 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.e is not active+clean 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328967+0000', 'last_change': '2024-02-19T14:56:17.328967+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328967+0000', 'last_undegraded': '2024-02-19T14:56:17.328967+0000', 'last_fullsized': '2024-02-19T14:56:17.328967+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.a is not active+clean 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.d is not active+clean 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.325076+0000', 'last_change': '2024-02-19T14:56:17.325076+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.325076+0000', 'last_undegraded': '2024-02-19T14:56:17.325076+0000', 'last_fullsized': '2024-02-19T14:56:17.325076+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.b is not active+clean 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.c is not active+clean 2024-02-19T14:56:23.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327092+0000', 'last_change': '2024-02-19T14:56:17.327092+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327092+0000', 'last_undegraded': '2024-02-19T14:56:17.327092+0000', 'last_fullsized': '2024-02-19T14:56:17.327092+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.c is not active+clean 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.b is not active+clean 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.326808+0000', 'last_change': '2024-02-19T14:56:17.326808+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.326808+0000', 'last_undegraded': '2024-02-19T14:56:17.326808+0000', 'last_fullsized': '2024-02-19T14:56:17.326808+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.d is not active+clean 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.a is not active+clean 2024-02-19T14:56:23.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.331308+0000', 'last_change': '2024-02-19T14:56:17.331308+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.331308+0000', 'last_undegraded': '2024-02-19T14:56:17.331308+0000', 'last_fullsized': '2024-02-19T14:56:17.331308+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.e is not active+clean 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.9 is not active+clean 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327292+0000', 'last_change': '2024-02-19T14:56:17.327292+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327292+0000', 'last_undegraded': '2024-02-19T14:56:17.327292+0000', 'last_fullsized': '2024-02-19T14:56:17.327292+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.f is not active+clean 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.8 is not active+clean 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.325041+0000', 'last_change': '2024-02-19T14:56:17.325041+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.325041+0000', 'last_undegraded': '2024-02-19T14:56:17.325041+0000', 'last_fullsized': '2024-02-19T14:56:17.325041+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:56:23.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.0 is not active+clean 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.7 is not active+clean 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.330488+0000', 'last_change': '2024-02-19T14:56:17.330488+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.330488+0000', 'last_undegraded': '2024-02-19T14:56:17.330488+0000', 'last_fullsized': '2024-02-19T14:56:17.330488+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1 is not active+clean 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.6 is not active+clean 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327797+0000', 'last_change': '2024-02-19T14:56:17.327797+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327797+0000', 'last_undegraded': '2024-02-19T14:56:17.327797+0000', 'last_fullsized': '2024-02-19T14:56:17.327797+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2 is not active+clean 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.5 is not active+clean 2024-02-19T14:56:23.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.326817+0000', 'last_change': '2024-02-19T14:56:17.326817+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.326817+0000', 'last_undegraded': '2024-02-19T14:56:17.326817+0000', 'last_fullsized': '2024-02-19T14:56:17.326817+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3 is not active+clean 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.4 is not active+clean 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.325062+0000', 'last_change': '2024-02-19T14:56:17.325062+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.325062+0000', 'last_undegraded': '2024-02-19T14:56:17.325062+0000', 'last_fullsized': '2024-02-19T14:56:17.325062+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.4 is not active+clean 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3 is not active+clean 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327943+0000', 'last_change': '2024-02-19T14:56:17.327943+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327943+0000', 'last_undegraded': '2024-02-19T14:56:17.327943+0000', 'last_fullsized': '2024-02-19T14:56:17.327943+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.5 is not active+clean 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2 is not active+clean 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.327309+0000', 'last_change': '2024-02-19T14:56:17.327309+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.327309+0000', 'last_undegraded': '2024-02-19T14:56:17.327309+0000', 'last_fullsized': '2024-02-19T14:56:17.327309+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.6 is not active+clean 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1 is not active+clean 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.324976+0000', 'last_change': '2024-02-19T14:56:17.324976+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.324976+0000', 'last_undegraded': '2024-02-19T14:56:17.324976+0000', 'last_fullsized': '2024-02-19T14:56:17.324976+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.7 is not active+clean 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.0 is not active+clean 2024-02-19T14:56:23.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.326986+0000', 'last_change': '2024-02-19T14:56:17.326986+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.326986+0000', 'last_undegraded': '2024-02-19T14:56:17.326986+0000', 'last_fullsized': '2024-02-19T14:56:17.326986+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:56:23.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.15 is not active+clean 2024-02-19T14:56:23.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T14:56:18.322503+0000', 'last_change': '2024-02-19T14:56:18.322503+0000', 'last_active': '2024-02-19T14:56:18.322503+0000', 'last_peered': '2024-02-19T14:56:18.322503+0000', 'last_clean': '2024-02-19T14:56:18.322503+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:18.322503+0000', 'last_undegraded': '2024-02-19T14:56:18.322503+0000', 'last_fullsized': '2024-02-19T14:56:18.322503+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:18.322503+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T14:56:23.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.12 is not active+clean 2024-02-19T14:56:23.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 121, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:56:17.328812+0000', 'last_change': '2024-02-19T14:56:17.328812+0000', 'last_active': '2024-02-19T14:56:17.317845+0000', 'last_peered': '2024-02-19T14:56:17.317845+0000', 'last_clean': '2024-02-19T14:56:17.317845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:56:17.328812+0000', 'last_undegraded': '2024-02-19T14:56:17.328812+0000', 'last_fullsized': '2024-02-19T14:56:17.328812+0000', 'mapping_epoch': 121, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 121, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'last_clean_scrub_stamp': '2024-02-19T14:56:17.317845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:56:23.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T14:56:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.217+0000 7f39ac33a640 1 Processor -- start 2024-02-19T14:56:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.217+0000 7f39ac33a640 1 -- start start 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39ac33a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39ac33a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39a40c2f60 0x7f39a406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39ac33a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40ae670 con 0x7f39a406f530 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39ac33a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40716b0 con 0x7f39a40c2f60 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39ac33a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40c33a0 con 0x7f39a40add70 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39a98d5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39a40c2f60 0x7f39a406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39aa8d7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f39a406f530 0x7f39a40705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55066/0 (socket says 172.21.15.78:55066) 2024-02-19T14:56:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39aa8d7640 1 -- 172.21.15.78:0/4192358830 learned_addr learned my addr 172.21.15.78:0/4192358830 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.218+0000 7f39a98d5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39a40c2f60 0x7f39a406eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54622/0 (socket says 172.21.15.78:54622) 2024-02-19T14:56:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39aa0d6640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a40c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39a98d5640 1 -- 172.21.15.78:0/4192358830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 msgr2=0x7f39a40c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:23.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39a98d5640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a40c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39a98d5640 1 -- 172.21.15.78:0/4192358830 >> v1:172.21.15.78:6789/0 conn(0x7f39a406f530 legacy=0x7f39a40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39a90d4640 1 -- 172.21.15.78:0/4192358830 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721432482 0 0) 0x7f39a40ae670 con 0x7f39a406f530 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39a98d5640 1 -- 172.21.15.78:0/4192358830 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4153780 con 0x7f39a40c2f60 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39aa0d6640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a40c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.219+0000 7f39a98d5640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a40c2f60 0x7f39a406eff0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f39940097a0 tx=0x7f399402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325bb5787c5fd549 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39a90d4640 1 -- 172.21.15.78:0/4192358830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399402e070 con 0x7f39a40c2f60 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39a90d4640 1 -- 172.21.15.78:0/4192358830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39940315a0 con 0x7f39a40c2f60 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39a90d4640 1 -- 172.21.15.78:0/4192358830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3994031930 con 0x7f39a40c2f60 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 -- 172.21.15.78:0/4192358830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a40c2f60 msgr2=0x7f39a406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a40c2f60 0x7f39a406eff0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f39940097a0 tx=0x7f399402b840 comp rx=0 tx=0).stop 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 -- 172.21.15.78:0/4192358830 shutdown_connections 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a40c2f60 0x7f39a406eff0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 --2- 172.21.15.78:0/4192358830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a40c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 -- 172.21.15.78:0/4192358830 >> 172.21.15.78:0/4192358830 conn(0x7f39a40c85e0 msgr2=0x7f39a40c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 -- 172.21.15.78:0/4192358830 shutdown_connections 2024-02-19T14:56:23.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.220+0000 7f39ac33a640 1 -- 172.21.15.78:0/4192358830 wait complete. 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.221+0000 7f39ac33a640 1 Processor -- start 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.221+0000 7f39ac33a640 1 -- start start 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39ac33a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 0x7f39a40c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39ac33a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a414d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39ac33a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39a40c2f60 0x7f39a414dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39ac33a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4099d00 con 0x7f39a406f530 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39ac33a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a406d1b0 con 0x7f39a40c2f60 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39ac33a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a406d980 con 0x7f39a40add70 2024-02-19T14:56:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 0x7f39a40c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 0x7f39a40c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54624/0 (socket says 172.21.15.78:54624) 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 -- 172.21.15.78:0/354898946 learned_addr learned my addr 172.21.15.78:0/354898946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39a98d5640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a414d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa8d7640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39a40c2f60 0x7f39a414dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 -- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 msgr2=0x7f39a414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a414d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 -- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39a40c2f60 msgr2=0x7f39a414dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39a40c2f60 0x7f39a414dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 -- 172.21.15.78:0/354898946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994009400 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa8d7640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39a40c2f60 0x7f39a414dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.222+0000 7f39aa0d6640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 0x7f39a40c1b60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f39a000e730 tx=0x7f39a000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.223+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0004230 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.223+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a0004390 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.223+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000c8f0 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.223+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a41eb2f0 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.223+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a406d6a0 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.224+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a406e160 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.224+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f39a000ca50 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.224+0000 7f399affd640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f398806b810 0x7f398806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.225+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f39a0089f20 con 0x7f39a406f530 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.225+0000 7f39a98d5640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f398806b810 0x7f398806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.225+0000 7f39a98d5640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f398806b810 0x7f398806dcd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3994036000 tx=0x7f399403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.229+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f39a0054840 con 0x7f39a406f530 2024-02-19T14:56:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.476+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f39a40676a0 con 0x7f39a406f530 2024-02-19T14:56:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.476+0000 7f399affd640 1 -- 172.21.15.78:0/354898946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1402 (secure 0 0 0) 0x7f39a0020590 con 0x7f39a406f530 2024-02-19T14:56:23.479 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:23.479 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":287,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":124,"num_osds":8,"num_up_osds":8,"osd_up_since":1708354307,"num_in_osds":8,"osd_in_since":1708354298,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":268750848,"bytes_avail":772825362432,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":100,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":7791}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-02-19T14:56:08.108498+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T14:56:23.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f398806b810 msgr2=0x7f398806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f398806b810 0x7f398806dcd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3994036000 tx=0x7f399403d040 comp rx=0 tx=0).stop 2024-02-19T14:56:23.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 msgr2=0x7f39a40c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 0x7f39a40c1b60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f39a000e730 tx=0x7f39a000ebf0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 shutdown_connections 2024-02-19T14:56:23.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f398806b810 0x7f398806dcd0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39a40c2f60 0x7f39a414dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39a40add70 0x7f39a414d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 --2- 172.21.15.78:0/354898946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39a406f530 0x7f39a40c1b60 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 >> 172.21.15.78:0/354898946 conn(0x7f39a40c85e0 msgr2=0x7f39a40c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:23.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 shutdown_connections 2024-02-19T14:56:23.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.479+0000 7f39ac33a640 1 -- 172.21.15.78:0/354898946 wait complete. 2024-02-19T14:56:23.492 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T14:56:23.493 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:56:23.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.590+0000 7fb8e7637640 1 Processor -- start 2024-02-19T14:56:23.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.590+0000 7fb8e7637640 1 -- start start 2024-02-19T14:56:23.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e7637640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8e0150560 0x7fb8e00c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e7637640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e0156e80 con 0x7fb8e00721a0 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e7637640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e0072d70 con 0x7fb8e006bb00 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e7637640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e006ffc0 con 0x7fb8e0150560 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e5bd4640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb8e00721a0 0x7fb8e015d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55082/0 (socket says 172.21.15.78:55082) 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e5bd4640 1 -- 172.21.15.78:0/220157537 learned_addr learned my addr 172.21.15.78:0/220157537 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407007538 0 0) 0x7fb8e0156e80 con 0x7fb8e00721a0 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.591+0000 7fb8e53d3640 1 --2- 172.21.15.78:0/220157537 >> v2:172.21.15.150:3300/0 conn(0x7fb8e0150560 0x7fb8e00c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8cc003520 con 0x7fb8e00721a0 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1122445852 0 0) 0x7fb8cc003520 con 0x7fb8e00721a0 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 >> v2:172.21.15.150:3300/0 conn(0x7fb8e0150560 msgr2=0x7fb8e00c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 --2- 172.21.15.78:0/220157537 >> v2:172.21.15.150:3300/0 conn(0x7fb8e0150560 0x7fb8e00c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 >> v1:172.21.15.150:6789/0 conn(0x7fb8e006bb00 legacy=0x7fb8e00c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0073050 con 0x7fb8e00721a0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb8d80019f0 con 0x7fb8e00721a0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 >> v1:172.21.15.78:6789/0 conn(0x7fb8e00721a0 legacy=0x7fb8e015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc0039c0 0x7fb8cc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8cc0051d0 0x7fb8cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc00c0f0 con 0x7fb8cc0047c0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc00c260 con 0x7fb8cc0039c0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc00c420 con 0x7fb8cc0051d0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e53d3640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc0039c0 0x7fb8cc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e5bd4640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8cc0051d0 0x7fb8cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e4bd2640 1 -- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8cc0051d0 msgr2=0x7fb8cc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8cc0051d0 0x7fb8cc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e4bd2640 1 -- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc0039c0 msgr2=0x7fb8cc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc0039c0 0x7fb8cc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e4bd2640 1 -- 172.21.15.78:0/220157537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc003520 con 0x7fb8cc0047c0 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e5bd4640 1 -- 172.21.15.78:0/220157537 reap_dead start 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.592+0000 7fb8e53d3640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc0039c0 0x7fb8cc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.593+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8cc004c40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00c5070 tx=0x7fb8cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9626176ce8e092 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.593+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00052d0 con 0x7fb8cc0047c0 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.593+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d0005430 con 0x7fb8cc0047c0 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.593+0000 7fb8d7fff640 1 -- 172.21.15.78:0/220157537 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0034b00 con 0x7fb8cc0047c0 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 -- 172.21.15.78:0/220157537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 msgr2=0x7fb8cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8cc004c40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00c5070 tx=0x7fb8cc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 -- 172.21.15.78:0/220157537 shutdown_connections 2024-02-19T14:56:23.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 --2- 172.21.15.78:0/220157537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8cc004c40 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 -- 172.21.15.78:0/220157537 >> 172.21.15.78:0/220157537 conn(0x7fb8e006c170 msgr2=0x7fb8e006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:23.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 -- 172.21.15.78:0/220157537 shutdown_connections 2024-02-19T14:56:23.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.594+0000 7fb8e7637640 1 -- 172.21.15.78:0/220157537 wait complete. 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.595+0000 7fb8e7637640 1 Processor -- start 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.595+0000 7fb8e7637640 1 -- start start 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e7637640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8e014e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e7637640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8e01536b0 0x7fb8e00abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e7637640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8e0146a60 0x7fb8e0146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e7637640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01e9310 con 0x7fb8cc0047c0 2024-02-19T14:56:23.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e7637640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01e95b0 con 0x7fb8e0146a60 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e7637640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01e9890 con 0x7fb8e01536b0 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8e014e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8e014e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54646/0 (socket says 172.21.15.78:54646) 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 -- 172.21.15.78:0/4033193446 learned_addr learned my addr 172.21.15.78:0/4033193446 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8e01536b0 0x7fb8e00abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e5bd4640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8e0146a60 0x7fb8e0146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 -- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8e01536b0 msgr2=0x7fb8e00abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8e01536b0 0x7fb8e00abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 -- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8e0146a60 msgr2=0x7fb8e0146f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8e0146a60 0x7fb8e0146f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 -- 172.21.15.78:0/4033193446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e01e9b70 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.596+0000 7fb8e53d3640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8e014e6d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc006c00 tx=0x7fb8dc00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.597+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc011c90 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.597+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc011df0 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.597+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc014660 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.597+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e014ec10 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.597+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e01ea090 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.598+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e01e9d00 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.598+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb8dc0147c0 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.599+0000 7fb8d5ffb640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb8c406b840 0x7fb8c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.599+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb8c406b840 0x7fb8c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.599+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb8dc08a960 con 0x7fb8cc0047c0 2024-02-19T14:56:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.599+0000 7fb8e4bd2640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb8c406b840 0x7fb8c406dd00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb8d003c000 tx=0x7fb8d0038000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:23.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.603+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb8dc055330 con 0x7fb8cc0047c0 2024-02-19T14:56:23.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.777+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8e01e9720 con 0x7fb8c406b840 2024-02-19T14:56:23.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.784+0000 7fb8d5ffb640 1 -- 172.21.15.78:0/4033193446 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277809 (secure 0 0 0) 0x7fb8e01e9720 con 0x7fb8c406b840 2024-02-19T14:56:23.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:23.794 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:56:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb8c406b840 msgr2=0x7fb8c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb8c406b840 0x7fb8c406dd00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb8d003c000 tx=0x7fb8d0038000 comp rx=0 tx=0).stop 2024-02-19T14:56:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 msgr2=0x7fb8e014e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:23.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8e014e6d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc006c00 tx=0x7fb8dc00f610 comp rx=0 tx=0).stop 2024-02-19T14:56:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 shutdown_connections 2024-02-19T14:56:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb8c406b840 0x7fb8c406dd00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8e0146a60 0x7fb8e0146f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8e01536b0 0x7fb8e00abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 --2- 172.21.15.78:0/4033193446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8cc0047c0 0x7fb8e014e6d0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.787+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 >> 172.21.15.78:0/4033193446 conn(0x7fb8e006c170 msgr2=0x7fb8e006c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.788+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 shutdown_connections 2024-02-19T14:56:23.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:23.788+0000 7fb8e7637640 1 -- 172.21.15.78:0/4033193446 wait complete. 2024-02-19T14:56:23.801 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-02-19T14:56:22.111024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262452,"kb_used_data":13000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712268,"statfs":{"total":773094113280,"available":772825362432,"internally_reserved":0,"allocated":13312000,"data_stored":7041680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785582"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328970+0000","last_change":"2024-02-19T14:56:17.328970+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328970+0000","last_undegraded":"2024-02-19T14:56:17.328970+0000","last_fullsized":"2024-02-19T14:56:17.328970+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326761+0000","last_change":"2024-02-19T14:56:17.326761+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326761+0000","last_undegraded":"2024-02-19T14:56:17.326761+0000","last_fullsized":"2024-02-19T14:56:17.326761+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328900+0000","last_change":"2024-02-19T14:56:17.328900+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328900+0000","last_undegraded":"2024-02-19T14:56:17.328900+0000","last_fullsized":"2024-02-19T14:56:17.328900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329386+0000","last_change":"2024-02-19T14:56:17.329386+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329386+0000","last_undegraded":"2024-02-19T14:56:17.329386+0000","last_fullsized":"2024-02-19T14:56:17.329386+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.329529+0000","last_change":"2024-02-19T14:56:17.329529+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.329529+0000","last_undegraded":"2024-02-19T14:56:17.329529+0000","last_fullsized":"2024-02-19T14:56:17.329529+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326738+0000","last_change":"2024-02-19T14:56:17.326738+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326738+0000","last_undegraded":"2024-02-19T14:56:17.326738+0000","last_fullsized":"2024-02-19T14:56:17.326738+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325558+0000","last_change":"2024-02-19T14:56:17.325558+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325558+0000","last_undegraded":"2024-02-19T14:56:17.325558+0000","last_fullsized":"2024-02-19T14:56:17.325558+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328599+0000","last_change":"2024-02-19T14:56:17.328599+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328599+0000","last_undegraded":"2024-02-19T14:56:17.328599+0000","last_fullsized":"2024-02-19T14:56:17.328599+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328526+0000","last_change":"2024-02-19T14:56:17.328526+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328526+0000","last_undegraded":"2024-02-19T14:56:17.328526+0000","last_fullsized":"2024-02-19T14:56:17.328526+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330943+0000","last_change":"2024-02-19T14:56:17.330943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330943+0000","last_undegraded":"2024-02-19T14:56:17.330943+0000","last_fullsized":"2024-02-19T14:56:17.330943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327472+0000","last_change":"2024-02-19T14:56:17.327472+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327472+0000","last_undegraded":"2024-02-19T14:56:17.327472+0000","last_fullsized":"2024-02-19T14:56:17.327472+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327318+0000","last_change":"2024-02-19T14:56:17.327318+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327318+0000","last_undegraded":"2024-02-19T14:56:17.327318+0000","last_fullsized":"2024-02-19T14:56:17.327318+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330450+0000","last_change":"2024-02-19T14:56:17.330450+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330450+0000","last_undegraded":"2024-02-19T14:56:17.330450+0000","last_fullsized":"2024-02-19T14:56:17.330450+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328398+0000","last_change":"2024-02-19T14:56:17.328398+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328398+0000","last_undegraded":"2024-02-19T14:56:17.328398+0000","last_fullsized":"2024-02-19T14:56:17.328398+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325514+0000","last_change":"2024-02-19T14:56:17.325514+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325514+0000","last_undegraded":"2024-02-19T14:56:17.325514+0000","last_fullsized":"2024-02-19T14:56:17.325514+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327919+0000","last_change":"2024-02-19T14:56:17.327919+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327919+0000","last_undegraded":"2024-02-19T14:56:17.327919+0000","last_fullsized":"2024-02-19T14:56:17.327919+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328967+0000","last_change":"2024-02-19T14:56:17.328967+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328967+0000","last_undegraded":"2024-02-19T14:56:17.328967+0000","last_fullsized":"2024-02-19T14:56:17.328967+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325076+0000","last_change":"2024-02-19T14:56:17.325076+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325076+0000","last_undegraded":"2024-02-19T14:56:17.325076+0000","last_fullsized":"2024-02-19T14:56:17.325076+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327092+0000","last_change":"2024-02-19T14:56:17.327092+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327092+0000","last_undegraded":"2024-02-19T14:56:17.327092+0000","last_fullsized":"2024-02-19T14:56:17.327092+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326808+0000","last_change":"2024-02-19T14:56:17.326808+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326808+0000","last_undegraded":"2024-02-19T14:56:17.326808+0000","last_fullsized":"2024-02-19T14:56:17.326808+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.331308+0000","last_change":"2024-02-19T14:56:17.331308+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.331308+0000","last_undegraded":"2024-02-19T14:56:17.331308+0000","last_fullsized":"2024-02-19T14:56:17.331308+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327292+0000","last_change":"2024-02-19T14:56:17.327292+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327292+0000","last_undegraded":"2024-02-19T14:56:17.327292+0000","last_fullsized":"2024-02-19T14:56:17.327292+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325041+0000","last_change":"2024-02-19T14:56:17.325041+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325041+0000","last_undegraded":"2024-02-19T14:56:17.325041+0000","last_fullsized":"2024-02-19T14:56:17.325041+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.330488+0000","last_change":"2024-02-19T14:56:17.330488+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.330488+0000","last_undegraded":"2024-02-19T14:56:17.330488+0000","last_fullsized":"2024-02-19T14:56:17.330488+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327797+0000","last_change":"2024-02-19T14:56:17.327797+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327797+0000","last_undegraded":"2024-02-19T14:56:17.327797+0000","last_fullsized":"2024-02-19T14:56:17.327797+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326817+0000","last_change":"2024-02-19T14:56:17.326817+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326817+0000","last_undegraded":"2024-02-19T14:56:17.326817+0000","last_fullsized":"2024-02-19T14:56:17.326817+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.325062+0000","last_change":"2024-02-19T14:56:17.325062+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.325062+0000","last_undegraded":"2024-02-19T14:56:17.325062+0000","last_fullsized":"2024-02-19T14:56:17.325062+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327943+0000","last_change":"2024-02-19T14:56:17.327943+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327943+0000","last_undegraded":"2024-02-19T14:56:17.327943+0000","last_fullsized":"2024-02-19T14:56:17.327943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.327309+0000","last_change":"2024-02-19T14:56:17.327309+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.327309+0000","last_undegraded":"2024-02-19T14:56:17.327309+0000","last_fullsized":"2024-02-19T14:56:17.327309+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.324976+0000","last_change":"2024-02-19T14:56:17.324976+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.324976+0000","last_undegraded":"2024-02-19T14:56:17.324976+0000","last_fullsized":"2024-02-19T14:56:17.324976+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.326986+0000","last_change":"2024-02-19T14:56:17.326986+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.326986+0000","last_undegraded":"2024-02-19T14:56:17.326986+0000","last_fullsized":"2024-02-19T14:56:17.326986+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:18.322503+0000","last_change":"2024-02-19T14:56:18.322503+0000","last_active":"2024-02-19T14:56:18.322503+0000","last_peered":"2024-02-19T14:56:18.322503+0000","last_clean":"2024-02-19T14:56:18.322503+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:18.322503+0000","last_undegraded":"2024-02-19T14:56:18.322503+0000","last_fullsized":"2024-02-19T14:56:18.322503+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":121,"state":"creating+peering","last_fresh":"2024-02-19T14:56:17.328812+0000","last_change":"2024-02-19T14:56:17.328812+0000","last_active":"2024-02-19T14:56:17.317845+0000","last_peered":"2024-02-19T14:56:17.317845+0000","last_clean":"2024-02-19T14:56:17.317845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:17.328812+0000","last_undegraded":"2024-02-19T14:56:17.328812+0000","last_fullsized":"2024-02-19T14:56:17.328812+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":359,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-02-19T14:56:17.329716+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:17.329716+0000","last_peered":"2024-02-19T14:56:17.329716+0000","last_clean":"2024-02-19T14:56:17.329716+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:17.329716+0000","last_undegraded":"2024-02-19T14:56:17.329716+0000","last_fullsized":"2024-02-19T14:56:17.329716+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33240,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338600,"statfs":{"total":96636764160,"available":96602726400,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.46700000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.495},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.35199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69499999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.42699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67300000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66300000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.316}]},{"osd":5,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68799999999999994,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65800000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.59999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.59199999999999997},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46400000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.48499999999999999,"15min":0.46700000000000003},"min":{"1min":0.34999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.49399999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.64000000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70899999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.313}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.40799999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.373,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69899999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.49199999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.75,"5min":0.872,"15min":0.872},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68200000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65400000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69799999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72099999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":0.877,"15min":0.877},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.57699999999999996},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.77900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.29599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.752},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.84999999999999998,"5min":1.03,"15min":1.03},"last":0.81000000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71399999999999997,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74199999999999999,"5min":0.746,"15min":0.746},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.877,"15min":0.877},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.376,"5min":0.37,"15min":0.37},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.5},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":1.038,"15min":1.038},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.378}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":1.038,"15min":1.038},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.629},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73299999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34100000000000003,"15min":0.31},"min":{"1min":0.22700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48999999999999999,"5min":3.694,"15min":3.694},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.189,"15min":0.189},"max":{"1min":0.47499999999999998,"5min":4.835,"15min":4.835},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.313,"15min":0.311},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.48199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42699999999999999,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58799999999999997,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38900000000000001,"15min":0.34699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54500000000000004,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66200000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77500000000000002,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76600000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.318}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":732618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.217,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.42199999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.498,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.307},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42199999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35399999999999998,"15min":0.36299999999999999},"min":{"1min":0.251,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47299999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57899999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57699999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.68500000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76400000000000001,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":1.01,"15min":1.01},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69899999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.026,"15min":1.026},"last":0.498},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33244,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338596,"statfs":{"total":96636764160,"available":96602722304,"internally_reserved":0,"allocated":2109440,"data_stored":1322986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.224,"15min":0.224},"max":{"1min":0.48399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58499999999999996,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79200000000000004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.58499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41999999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.443},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.86399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75600000000000001,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:56:24.591 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:56:24.589+0000 7fbd25ed3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:56:26.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:56:26.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:56:26.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:56:26.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.907+0000 7feb1e05c640 1 Processor -- start 2024-02-19T14:56:26.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.907+0000 7feb1e05c640 1 -- start start 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb1e05c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feb18152700 0x7feb180c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb1e05c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb180c8e00 0x7feb1815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb1e05c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb181551f0 con 0x7feb180c5730 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb1e05c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb18155490 con 0x7feb180c8e00 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb1e05c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb18155770 con 0x7feb18152700 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb17fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb180c8e00 0x7feb1815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb17fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb180c8e00 0x7feb1815f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54648/0 (socket says 172.21.15.78:54648) 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb16ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7feb180c5730 0x7feb1806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55096/0 (socket says 172.21.15.78:55096) 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.908+0000 7feb17fff640 1 -- 172.21.15.78:0/4075523657 learned_addr learned my addr 172.21.15.78:0/4075523657 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb177fe640 1 --2- 172.21.15.78:0/4075523657 >> v2:172.21.15.150:3300/0 conn(0x7feb18152700 0x7feb180c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb17fff640 1 -- 172.21.15.78:0/4075523657 >> v2:172.21.15.150:3300/0 conn(0x7feb18152700 msgr2=0x7feb180c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb17fff640 1 --2- 172.21.15.78:0/4075523657 >> v2:172.21.15.150:3300/0 conn(0x7feb18152700 0x7feb180c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:26.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb17fff640 1 -- 172.21.15.78:0/4075523657 >> v1:172.21.15.78:6789/0 conn(0x7feb180c5730 legacy=0x7feb1806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:26.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb167fc640 1 -- 172.21.15.78:0/4075523657 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581332127 0 0) 0x7feb181551f0 con 0x7feb180c5730 2024-02-19T14:56:26.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb17fff640 1 -- 172.21.15.78:0/4075523657 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18155a50 con 0x7feb180c8e00 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.909+0000 7feb17fff640 1 --2- 172.21.15.78:0/4075523657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c8e00 0x7feb1815f980 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7feb1808bad0 tx=0x7feb1002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7330b71cb9ca83c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb167fc640 1 -- 172.21.15.78:0/4075523657 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1002ca80 con 0x7feb180c8e00 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb167fc640 1 -- 172.21.15.78:0/4075523657 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1002cbe0 con 0x7feb180c8e00 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb167fc640 1 -- 172.21.15.78:0/4075523657 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb100343f0 con 0x7feb180c8e00 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 -- 172.21.15.78:0/4075523657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c8e00 msgr2=0x7feb1815f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 --2- 172.21.15.78:0/4075523657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c8e00 0x7feb1815f980 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7feb1808bad0 tx=0x7feb1002bee0 comp rx=0 tx=0).stop 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 -- 172.21.15.78:0/4075523657 shutdown_connections 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 --2- 172.21.15.78:0/4075523657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c8e00 0x7feb1815f980 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 --2- 172.21.15.78:0/4075523657 >> v2:172.21.15.150:3300/0 conn(0x7feb18152700 0x7feb180c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:26.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 -- 172.21.15.78:0/4075523657 >> 172.21.15.78:0/4075523657 conn(0x7feb180cac90 msgr2=0x7feb180cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 -- 172.21.15.78:0/4075523657 shutdown_connections 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.910+0000 7feb1e05c640 1 -- 172.21.15.78:0/4075523657 wait complete. 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.911+0000 7feb1e05c640 1 Processor -- start 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 -- start start 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 0x7feb181ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb180c8e00 0x7feb180ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18152700 0x7feb18148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb181eaf90 con 0x7feb180c5730 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb181eb4c0 con 0x7feb18152700 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb1e05c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb18154ed0 con 0x7feb180c8e00 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 0x7feb181ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 0x7feb181ee6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54650/0 (socket says 172.21.15.78:54650) 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb177fe640 1 -- 172.21.15.78:0/480081396 learned_addr learned my addr 172.21.15.78:0/480081396 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:26.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.912+0000 7feb177fe640 1 -- 172.21.15.78:0/480081396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb180c8e00 msgr2=0x7feb180ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb177fe640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb180c8e00 0x7feb180ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb177fe640 1 -- 172.21.15.78:0/480081396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18152700 msgr2=0x7feb18148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb177fe640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18152700 0x7feb18148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb177fe640 1 -- 172.21.15.78:0/480081396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10009b30 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb177fe640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 0x7feb181ee6b0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7feb0800e870 tx=0x7feb0800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080094f0 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0800c940 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb181eb1e0 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0800ccf0 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.913+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb181edeb0 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.914+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb18074990 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.915+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feb08013470 con 0x7feb180c5730 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.915+0000 7feafffff640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb0406b810 0x7feb0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:26.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.915+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb0808a3e0 con 0x7feb180c5730 2024-02-19T14:56:26.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.915+0000 7feb16ffd640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb0406b810 0x7feb0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:26.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.915+0000 7feb16ffd640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb0406b810 0x7feb0406dcd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7feb0c007fd0 tx=0x7feb0c007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:26.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:26.919+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb08054d00 con 0x7feb180c5730 2024-02-19T14:56:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.088+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb18155f70 con 0x7feb0406b810 2024-02-19T14:56:27.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.095+0000 7feafffff640 1 -- 172.21.15.78:0/480081396 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290239 (secure 0 0 0) 0x7feb0c0070d0 con 0x7feb0406b810 2024-02-19T14:56:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:27.102 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb0406b810 msgr2=0x7feb0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb0406b810 0x7feb0406dcd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7feb0c007fd0 tx=0x7feb0c007920 comp rx=0 tx=0).stop 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 msgr2=0x7feb181ee6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 0x7feb181ee6b0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7feb0800e870 tx=0x7feb0800ed30 comp rx=0 tx=0).stop 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 shutdown_connections 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb0406b810 0x7feb0406dcd0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18152700 0x7feb18148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb180c8e00 0x7feb180ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 --2- 172.21.15.78:0/480081396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb180c5730 0x7feb181ee6b0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 >> 172.21.15.78:0/480081396 conn(0x7feb180cac90 msgr2=0x7feb1814a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 shutdown_connections 2024-02-19T14:56:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.098+0000 7feb1e05c640 1 -- 172.21.15.78:0/480081396 wait complete. 2024-02-19T14:56:27.111 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-02-19T14:56:26.111739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262820,"kb_used_data":13352,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711900,"statfs":{"total":773094113280,"available":772824985600,"internally_reserved":0,"allocated":13672448,"data_stored":7210644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.781808"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719765+0000","last_change":"2024-02-19T14:56:19.350554+0000","last_active":"2024-02-19T14:56:21.719765+0000","last_peered":"2024-02-19T14:56:21.719765+0000","last_clean":"2024-02-19T14:56:21.719765+0000","last_became_active":"2024-02-19T14:56:19.350270+0000","last_became_peered":"2024-02-19T14:56:19.350270+0000","last_unstale":"2024-02-19T14:56:21.719765+0000","last_undegraded":"2024-02-19T14:56:21.719765+0000","last_fullsized":"2024-02-19T14:56:21.719765+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:42.600508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340841+0000","last_change":"2024-02-19T14:56:19.343775+0000","last_active":"2024-02-19T14:56:20.340841+0000","last_peered":"2024-02-19T14:56:20.340841+0000","last_clean":"2024-02-19T14:56:20.340841+0000","last_became_active":"2024-02-19T14:56:19.343501+0000","last_became_peered":"2024-02-19T14:56:19.343501+0000","last_unstale":"2024-02-19T14:56:20.340841+0000","last_undegraded":"2024-02-19T14:56:20.340841+0000","last_fullsized":"2024-02-19T14:56:20.340841+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:34.255709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720301+0000","last_change":"2024-02-19T14:56:19.359415+0000","last_active":"2024-02-19T14:56:21.720301+0000","last_peered":"2024-02-19T14:56:21.720301+0000","last_clean":"2024-02-19T14:56:21.720301+0000","last_became_active":"2024-02-19T14:56:19.359125+0000","last_became_peered":"2024-02-19T14:56:19.359125+0000","last_unstale":"2024-02-19T14:56:21.720301+0000","last_undegraded":"2024-02-19T14:56:21.720301+0000","last_fullsized":"2024-02-19T14:56:21.720301+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:00.705662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340393+0000","last_change":"2024-02-19T14:56:19.348969+0000","last_active":"2024-02-19T14:56:20.340393+0000","last_peered":"2024-02-19T14:56:20.340393+0000","last_clean":"2024-02-19T14:56:20.340393+0000","last_became_active":"2024-02-19T14:56:19.348658+0000","last_became_peered":"2024-02-19T14:56:19.348658+0000","last_unstale":"2024-02-19T14:56:20.340393+0000","last_undegraded":"2024-02-19T14:56:20.340393+0000","last_fullsized":"2024-02-19T14:56:20.340393+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:14:47.235348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719683+0000","last_change":"2024-02-19T14:56:19.349527+0000","last_active":"2024-02-19T14:56:21.719683+0000","last_peered":"2024-02-19T14:56:21.719683+0000","last_clean":"2024-02-19T14:56:21.719683+0000","last_became_active":"2024-02-19T14:56:19.349188+0000","last_became_peered":"2024-02-19T14:56:19.349188+0000","last_unstale":"2024-02-19T14:56:21.719683+0000","last_undegraded":"2024-02-19T14:56:21.719683+0000","last_fullsized":"2024-02-19T14:56:21.719683+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:40:32.646550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341933+0000","last_change":"2024-02-19T14:56:19.358728+0000","last_active":"2024-02-19T14:56:20.341933+0000","last_peered":"2024-02-19T14:56:20.341933+0000","last_clean":"2024-02-19T14:56:20.341933+0000","last_became_active":"2024-02-19T14:56:19.358414+0000","last_became_peered":"2024-02-19T14:56:19.358414+0000","last_unstale":"2024-02-19T14:56:20.341933+0000","last_undegraded":"2024-02-19T14:56:20.341933+0000","last_fullsized":"2024-02-19T14:56:20.341933+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:34.255709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719675+0000","last_change":"2024-02-19T14:56:19.353677+0000","last_active":"2024-02-19T14:56:21.719675+0000","last_peered":"2024-02-19T14:56:21.719675+0000","last_clean":"2024-02-19T14:56:21.719675+0000","last_became_active":"2024-02-19T14:56:19.352866+0000","last_became_peered":"2024-02-19T14:56:19.352866+0000","last_unstale":"2024-02-19T14:56:21.719675+0000","last_undegraded":"2024-02-19T14:56:21.719675+0000","last_fullsized":"2024-02-19T14:56:21.719675+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:27:18.005571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340357+0000","last_change":"2024-02-19T14:56:19.352510+0000","last_active":"2024-02-19T14:56:20.340357+0000","last_peered":"2024-02-19T14:56:20.340357+0000","last_clean":"2024-02-19T14:56:20.340357+0000","last_became_active":"2024-02-19T14:56:19.352168+0000","last_became_peered":"2024-02-19T14:56:19.352168+0000","last_unstale":"2024-02-19T14:56:20.340357+0000","last_undegraded":"2024-02-19T14:56:20.340357+0000","last_fullsized":"2024-02-19T14:56:20.340357+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:08.284292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342112+0000","last_change":"2024-02-19T14:56:19.341643+0000","last_active":"2024-02-19T14:56:20.342112+0000","last_peered":"2024-02-19T14:56:20.342112+0000","last_clean":"2024-02-19T14:56:20.342112+0000","last_became_active":"2024-02-19T14:56:19.341347+0000","last_became_peered":"2024-02-19T14:56:19.341347+0000","last_unstale":"2024-02-19T14:56:20.342112+0000","last_undegraded":"2024-02-19T14:56:20.342112+0000","last_fullsized":"2024-02-19T14:56:20.342112+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:58.708993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720330+0000","last_change":"2024-02-19T14:56:19.352523+0000","last_active":"2024-02-19T14:56:21.720330+0000","last_peered":"2024-02-19T14:56:21.720330+0000","last_clean":"2024-02-19T14:56:21.720330+0000","last_became_active":"2024-02-19T14:56:19.352229+0000","last_became_peered":"2024-02-19T14:56:19.352229+0000","last_unstale":"2024-02-19T14:56:21.720330+0000","last_undegraded":"2024-02-19T14:56:21.720330+0000","last_fullsized":"2024-02-19T14:56:21.720330+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:40.795390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340472+0000","last_change":"2024-02-19T14:56:19.353650+0000","last_active":"2024-02-19T14:56:20.340472+0000","last_peered":"2024-02-19T14:56:20.340472+0000","last_clean":"2024-02-19T14:56:20.340472+0000","last_became_active":"2024-02-19T14:56:19.353379+0000","last_became_peered":"2024-02-19T14:56:19.353379+0000","last_unstale":"2024-02-19T14:56:20.340472+0000","last_undegraded":"2024-02-19T14:56:20.340472+0000","last_fullsized":"2024-02-19T14:56:20.340472+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:11:59.922017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719536+0000","last_change":"2024-02-19T14:56:19.346119+0000","last_active":"2024-02-19T14:56:21.719536+0000","last_peered":"2024-02-19T14:56:21.719536+0000","last_clean":"2024-02-19T14:56:21.719536+0000","last_became_active":"2024-02-19T14:56:19.345848+0000","last_became_peered":"2024-02-19T14:56:19.345848+0000","last_unstale":"2024-02-19T14:56:21.719536+0000","last_undegraded":"2024-02-19T14:56:21.719536+0000","last_fullsized":"2024-02-19T14:56:21.719536+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:21:26.241749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342018+0000","last_change":"2024-02-19T14:56:19.358440+0000","last_active":"2024-02-19T14:56:20.342018+0000","last_peered":"2024-02-19T14:56:20.342018+0000","last_clean":"2024-02-19T14:56:20.342018+0000","last_became_active":"2024-02-19T14:56:19.358147+0000","last_became_peered":"2024-02-19T14:56:19.358147+0000","last_unstale":"2024-02-19T14:56:20.342018+0000","last_undegraded":"2024-02-19T14:56:20.342018+0000","last_fullsized":"2024-02-19T14:56:20.342018+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:27.827353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719423+0000","last_change":"2024-02-19T14:56:19.348939+0000","last_active":"2024-02-19T14:56:21.719423+0000","last_peered":"2024-02-19T14:56:21.719423+0000","last_clean":"2024-02-19T14:56:21.719423+0000","last_became_active":"2024-02-19T14:56:19.348694+0000","last_became_peered":"2024-02-19T14:56:19.348694+0000","last_unstale":"2024-02-19T14:56:21.719423+0000","last_undegraded":"2024-02-19T14:56:21.719423+0000","last_fullsized":"2024-02-19T14:56:21.719423+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:24:25.900218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719129+0000","last_change":"2024-02-19T14:56:19.350412+0000","last_active":"2024-02-19T14:56:21.719129+0000","last_peered":"2024-02-19T14:56:21.719129+0000","last_clean":"2024-02-19T14:56:21.719129+0000","last_became_active":"2024-02-19T14:56:19.350040+0000","last_became_peered":"2024-02-19T14:56:19.350040+0000","last_unstale":"2024-02-19T14:56:21.719129+0000","last_undegraded":"2024-02-19T14:56:21.719129+0000","last_fullsized":"2024-02-19T14:56:21.719129+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:20.743081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341736+0000","last_change":"2024-02-19T14:56:19.348673+0000","last_active":"2024-02-19T14:56:20.341736+0000","last_peered":"2024-02-19T14:56:20.341736+0000","last_clean":"2024-02-19T14:56:20.341736+0000","last_became_active":"2024-02-19T14:56:19.346878+0000","last_became_peered":"2024-02-19T14:56:19.346878+0000","last_unstale":"2024-02-19T14:56:20.341736+0000","last_undegraded":"2024-02-19T14:56:20.341736+0000","last_fullsized":"2024-02-19T14:56:20.341736+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:21.644971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342231+0000","last_change":"2024-02-19T14:56:19.340789+0000","last_active":"2024-02-19T14:56:20.342231+0000","last_peered":"2024-02-19T14:56:20.342231+0000","last_clean":"2024-02-19T14:56:20.342231+0000","last_became_active":"2024-02-19T14:56:19.340500+0000","last_became_peered":"2024-02-19T14:56:19.340500+0000","last_unstale":"2024-02-19T14:56:20.342231+0000","last_undegraded":"2024-02-19T14:56:20.342231+0000","last_fullsized":"2024-02-19T14:56:20.342231+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:59.099848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719722+0000","last_change":"2024-02-19T14:56:19.345076+0000","last_active":"2024-02-19T14:56:21.719722+0000","last_peered":"2024-02-19T14:56:21.719722+0000","last_clean":"2024-02-19T14:56:21.719722+0000","last_became_active":"2024-02-19T14:56:19.344819+0000","last_became_peered":"2024-02-19T14:56:19.344819+0000","last_unstale":"2024-02-19T14:56:21.719722+0000","last_undegraded":"2024-02-19T14:56:21.719722+0000","last_fullsized":"2024-02-19T14:56:21.719722+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:53.303058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720387+0000","last_change":"2024-02-19T14:56:19.353769+0000","last_active":"2024-02-19T14:56:21.720387+0000","last_peered":"2024-02-19T14:56:21.720387+0000","last_clean":"2024-02-19T14:56:21.720387+0000","last_became_active":"2024-02-19T14:56:19.353349+0000","last_became_peered":"2024-02-19T14:56:19.353349+0000","last_unstale":"2024-02-19T14:56:21.720387+0000","last_undegraded":"2024-02-19T14:56:21.720387+0000","last_fullsized":"2024-02-19T14:56:21.720387+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:57:35.234054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719930+0000","last_change":"2024-02-19T14:56:19.350581+0000","last_active":"2024-02-19T14:56:21.719930+0000","last_peered":"2024-02-19T14:56:21.719930+0000","last_clean":"2024-02-19T14:56:21.719930+0000","last_became_active":"2024-02-19T14:56:19.348132+0000","last_became_peered":"2024-02-19T14:56:19.348132+0000","last_unstale":"2024-02-19T14:56:21.719930+0000","last_undegraded":"2024-02-19T14:56:21.719930+0000","last_fullsized":"2024-02-19T14:56:21.719930+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:39.165012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340773+0000","last_change":"2024-02-19T14:56:19.348835+0000","last_active":"2024-02-19T14:56:20.340773+0000","last_peered":"2024-02-19T14:56:20.340773+0000","last_clean":"2024-02-19T14:56:20.340773+0000","last_became_active":"2024-02-19T14:56:19.344591+0000","last_became_peered":"2024-02-19T14:56:19.344591+0000","last_unstale":"2024-02-19T14:56:20.340773+0000","last_undegraded":"2024-02-19T14:56:20.340773+0000","last_fullsized":"2024-02-19T14:56:20.340773+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:22.307617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719363+0000","last_change":"2024-02-19T14:56:19.341462+0000","last_active":"2024-02-19T14:56:21.719363+0000","last_peered":"2024-02-19T14:56:21.719363+0000","last_clean":"2024-02-19T14:56:21.719363+0000","last_became_active":"2024-02-19T14:56:19.341212+0000","last_became_peered":"2024-02-19T14:56:19.341212+0000","last_unstale":"2024-02-19T14:56:21.719363+0000","last_undegraded":"2024-02-19T14:56:21.719363+0000","last_fullsized":"2024-02-19T14:56:21.719363+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:12.750998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340521+0000","last_change":"2024-02-19T14:56:19.347601+0000","last_active":"2024-02-19T14:56:20.340521+0000","last_peered":"2024-02-19T14:56:20.340521+0000","last_clean":"2024-02-19T14:56:20.340521+0000","last_became_active":"2024-02-19T14:56:19.347338+0000","last_became_peered":"2024-02-19T14:56:19.347338+0000","last_unstale":"2024-02-19T14:56:20.340521+0000","last_undegraded":"2024-02-19T14:56:20.340521+0000","last_fullsized":"2024-02-19T14:56:20.340521+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:53:06.250904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341803+0000","last_change":"2024-02-19T14:56:19.339394+0000","last_active":"2024-02-19T14:56:20.341803+0000","last_peered":"2024-02-19T14:56:20.341803+0000","last_clean":"2024-02-19T14:56:20.341803+0000","last_became_active":"2024-02-19T14:56:19.339176+0000","last_became_peered":"2024-02-19T14:56:19.339176+0000","last_unstale":"2024-02-19T14:56:20.341803+0000","last_undegraded":"2024-02-19T14:56:20.341803+0000","last_fullsized":"2024-02-19T14:56:20.341803+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:00.705662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720701+0000","last_change":"2024-02-19T14:56:19.363151+0000","last_active":"2024-02-19T14:56:21.720701+0000","last_peered":"2024-02-19T14:56:21.720701+0000","last_clean":"2024-02-19T14:56:21.720701+0000","last_became_active":"2024-02-19T14:56:19.362858+0000","last_became_peered":"2024-02-19T14:56:19.362858+0000","last_unstale":"2024-02-19T14:56:21.720701+0000","last_undegraded":"2024-02-19T14:56:21.720701+0000","last_fullsized":"2024-02-19T14:56:21.720701+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:19.815092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720615+0000","last_change":"2024-02-19T14:56:19.358174+0000","last_active":"2024-02-19T14:56:21.720615+0000","last_peered":"2024-02-19T14:56:21.720615+0000","last_clean":"2024-02-19T14:56:21.720615+0000","last_became_active":"2024-02-19T14:56:19.357876+0000","last_became_peered":"2024-02-19T14:56:19.357876+0000","last_unstale":"2024-02-19T14:56:21.720615+0000","last_undegraded":"2024-02-19T14:56:21.720615+0000","last_fullsized":"2024-02-19T14:56:21.720615+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:27.827353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720539+0000","last_change":"2024-02-19T14:56:19.359570+0000","last_active":"2024-02-19T14:56:21.720539+0000","last_peered":"2024-02-19T14:56:21.720539+0000","last_clean":"2024-02-19T14:56:21.720539+0000","last_became_active":"2024-02-19T14:56:19.359283+0000","last_became_peered":"2024-02-19T14:56:19.359283+0000","last_unstale":"2024-02-19T14:56:21.720539+0000","last_undegraded":"2024-02-19T14:56:21.720539+0000","last_fullsized":"2024-02-19T14:56:21.720539+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:01.262369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720368+0000","last_change":"2024-02-19T14:56:19.353292+0000","last_active":"2024-02-19T14:56:21.720368+0000","last_peered":"2024-02-19T14:56:21.720368+0000","last_clean":"2024-02-19T14:56:21.720368+0000","last_became_active":"2024-02-19T14:56:19.353039+0000","last_became_peered":"2024-02-19T14:56:19.353039+0000","last_unstale":"2024-02-19T14:56:21.720368+0000","last_undegraded":"2024-02-19T14:56:21.720368+0000","last_fullsized":"2024-02-19T14:56:21.720368+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:25.486783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720278+0000","last_change":"2024-02-19T14:56:19.351731+0000","last_active":"2024-02-19T14:56:21.720278+0000","last_peered":"2024-02-19T14:56:21.720278+0000","last_clean":"2024-02-19T14:56:21.720278+0000","last_became_active":"2024-02-19T14:56:19.351472+0000","last_became_peered":"2024-02-19T14:56:19.351472+0000","last_unstale":"2024-02-19T14:56:21.720278+0000","last_undegraded":"2024-02-19T14:56:21.720278+0000","last_fullsized":"2024-02-19T14:56:21.720278+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:21.644971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720058+0000","last_change":"2024-02-19T14:56:19.363170+0000","last_active":"2024-02-19T14:56:21.720058+0000","last_peered":"2024-02-19T14:56:21.720058+0000","last_clean":"2024-02-19T14:56:21.720058+0000","last_became_active":"2024-02-19T14:56:19.362881+0000","last_became_peered":"2024-02-19T14:56:19.362881+0000","last_unstale":"2024-02-19T14:56:21.720058+0000","last_undegraded":"2024-02-19T14:56:21.720058+0000","last_fullsized":"2024-02-19T14:56:21.720058+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:11:59.922017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340527+0000","last_change":"2024-02-19T14:56:19.344445+0000","last_active":"2024-02-19T14:56:20.340527+0000","last_peered":"2024-02-19T14:56:20.340527+0000","last_clean":"2024-02-19T14:56:20.340527+0000","last_became_active":"2024-02-19T14:56:19.344158+0000","last_became_peered":"2024-02-19T14:56:19.344158+0000","last_unstale":"2024-02-19T14:56:20.340527+0000","last_undegraded":"2024-02-19T14:56:20.340527+0000","last_fullsized":"2024-02-19T14:56:20.340527+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:26:35.550986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720149+0000","last_change":"2024-02-19T14:56:19.351432+0000","last_active":"2024-02-19T14:56:21.720149+0000","last_peered":"2024-02-19T14:56:21.720149+0000","last_clean":"2024-02-19T14:56:21.720149+0000","last_became_active":"2024-02-19T14:56:19.351128+0000","last_became_peered":"2024-02-19T14:56:19.351128+0000","last_unstale":"2024-02-19T14:56:21.720149+0000","last_undegraded":"2024-02-19T14:56:21.720149+0000","last_fullsized":"2024-02-19T14:56:21.720149+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:15.983986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340537+0000","last_change":"2024-02-19T14:56:19.348355+0000","last_active":"2024-02-19T14:56:20.340537+0000","last_peered":"2024-02-19T14:56:20.340537+0000","last_clean":"2024-02-19T14:56:20.340537+0000","last_became_active":"2024-02-19T14:56:19.348081+0000","last_became_peered":"2024-02-19T14:56:19.348081+0000","last_unstale":"2024-02-19T14:56:20.340537+0000","last_undegraded":"2024-02-19T14:56:20.340537+0000","last_fullsized":"2024-02-19T14:56:20.340537+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:52.282544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"123'4","reported_seq":18,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720683+0000","last_change":"2024-02-19T14:56:18.349493+0000","last_active":"2024-02-19T14:56:21.720683+0000","last_peered":"2024-02-19T14:56:21.720683+0000","last_clean":"2024-02-19T14:56:21.720683+0000","last_became_active":"2024-02-19T14:56:18.349160+0000","last_became_peered":"2024-02-19T14:56:18.349160+0000","last_unstale":"2024-02-19T14:56:21.720683+0000","last_undegraded":"2024-02-19T14:56:21.720683+0000","last_fullsized":"2024-02-19T14:56:21.720683+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:47.732359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719493+0000","last_change":"2024-02-19T14:56:19.350770+0000","last_active":"2024-02-19T14:56:21.719493+0000","last_peered":"2024-02-19T14:56:21.719493+0000","last_clean":"2024-02-19T14:56:21.719493+0000","last_became_active":"2024-02-19T14:56:19.350499+0000","last_became_peered":"2024-02-19T14:56:19.350499+0000","last_unstale":"2024-02-19T14:56:21.719493+0000","last_undegraded":"2024-02-19T14:56:21.719493+0000","last_fullsized":"2024-02-19T14:56:21.719493+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:14:05.398604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"123'1","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719863+0000","last_change":"2024-02-19T14:56:18.342445+0000","last_active":"2024-02-19T14:56:21.719863+0000","last_peered":"2024-02-19T14:56:21.719863+0000","last_clean":"2024-02-19T14:56:21.719863+0000","last_became_active":"2024-02-19T14:56:18.342174+0000","last_became_peered":"2024-02-19T14:56:18.342174+0000","last_unstale":"2024-02-19T14:56:21.719863+0000","last_undegraded":"2024-02-19T14:56:21.719863+0000","last_fullsized":"2024-02-19T14:56:21.719863+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:21.302959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340895+0000","last_change":"2024-02-19T14:56:19.346840+0000","last_active":"2024-02-19T14:56:20.340895+0000","last_peered":"2024-02-19T14:56:20.340895+0000","last_clean":"2024-02-19T14:56:20.340895+0000","last_became_active":"2024-02-19T14:56:19.346557+0000","last_became_peered":"2024-02-19T14:56:19.346557+0000","last_unstale":"2024-02-19T14:56:20.340895+0000","last_undegraded":"2024-02-19T14:56:20.340895+0000","last_fullsized":"2024-02-19T14:56:20.340895+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:49:00.705662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"123'2","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720597+0000","last_change":"2024-02-19T14:56:18.344174+0000","last_active":"2024-02-19T14:56:21.720597+0000","last_peered":"2024-02-19T14:56:21.720597+0000","last_clean":"2024-02-19T14:56:21.720597+0000","last_became_active":"2024-02-19T14:56:18.343883+0000","last_became_peered":"2024-02-19T14:56:18.343883+0000","last_unstale":"2024-02-19T14:56:21.720597+0000","last_undegraded":"2024-02-19T14:56:21.720597+0000","last_fullsized":"2024-02-19T14:56:21.720597+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:33.251050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719287+0000","last_change":"2024-02-19T14:56:19.348384+0000","last_active":"2024-02-19T14:56:21.719287+0000","last_peered":"2024-02-19T14:56:21.719287+0000","last_clean":"2024-02-19T14:56:21.719287+0000","last_became_active":"2024-02-19T14:56:19.348153+0000","last_became_peered":"2024-02-19T14:56:19.348153+0000","last_unstale":"2024-02-19T14:56:21.719287+0000","last_undegraded":"2024-02-19T14:56:21.719287+0000","last_fullsized":"2024-02-19T14:56:21.719287+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:02:50.458072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"123'2","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720244+0000","last_change":"2024-02-19T14:56:18.345093+0000","last_active":"2024-02-19T14:56:21.720244+0000","last_peered":"2024-02-19T14:56:21.720244+0000","last_clean":"2024-02-19T14:56:21.720244+0000","last_became_active":"2024-02-19T14:56:18.344596+0000","last_became_peered":"2024-02-19T14:56:18.344596+0000","last_unstale":"2024-02-19T14:56:21.720244+0000","last_undegraded":"2024-02-19T14:56:21.720244+0000","last_fullsized":"2024-02-19T14:56:21.720244+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:59.701004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341077+0000","last_change":"2024-02-19T14:56:19.340210+0000","last_active":"2024-02-19T14:56:20.341077+0000","last_peered":"2024-02-19T14:56:20.341077+0000","last_clean":"2024-02-19T14:56:20.341077+0000","last_became_active":"2024-02-19T14:56:19.339907+0000","last_became_peered":"2024-02-19T14:56:19.339907+0000","last_unstale":"2024-02-19T14:56:20.341077+0000","last_undegraded":"2024-02-19T14:56:20.341077+0000","last_fullsized":"2024-02-19T14:56:20.341077+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:10:29.979482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"123'1","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340935+0000","last_change":"2024-02-19T14:56:18.346743+0000","last_active":"2024-02-19T14:56:20.340935+0000","last_peered":"2024-02-19T14:56:20.340935+0000","last_clean":"2024-02-19T14:56:20.340935+0000","last_became_active":"2024-02-19T14:56:18.346468+0000","last_became_peered":"2024-02-19T14:56:18.346468+0000","last_unstale":"2024-02-19T14:56:20.340935+0000","last_undegraded":"2024-02-19T14:56:20.340935+0000","last_fullsized":"2024-02-19T14:56:20.340935+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:35.234706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341069+0000","last_change":"2024-02-19T14:56:19.348993+0000","last_active":"2024-02-19T14:56:20.341069+0000","last_peered":"2024-02-19T14:56:20.341069+0000","last_clean":"2024-02-19T14:56:20.341069+0000","last_became_active":"2024-02-19T14:56:19.348730+0000","last_became_peered":"2024-02-19T14:56:19.348730+0000","last_unstale":"2024-02-19T14:56:20.341069+0000","last_undegraded":"2024-02-19T14:56:20.341069+0000","last_fullsized":"2024-02-19T14:56:20.341069+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:56:40.419881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"123'1","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719744+0000","last_change":"2024-02-19T14:56:18.344768+0000","last_active":"2024-02-19T14:56:21.719744+0000","last_peered":"2024-02-19T14:56:21.719744+0000","last_clean":"2024-02-19T14:56:21.719744+0000","last_became_active":"2024-02-19T14:56:18.344520+0000","last_became_peered":"2024-02-19T14:56:18.344520+0000","last_unstale":"2024-02-19T14:56:21.719744+0000","last_undegraded":"2024-02-19T14:56:21.719744+0000","last_fullsized":"2024-02-19T14:56:21.719744+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:53:05.246246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719814+0000","last_change":"2024-02-19T14:56:19.355657+0000","last_active":"2024-02-19T14:56:21.719814+0000","last_peered":"2024-02-19T14:56:21.719814+0000","last_clean":"2024-02-19T14:56:21.719814+0000","last_became_active":"2024-02-19T14:56:19.355410+0000","last_became_peered":"2024-02-19T14:56:19.355410+0000","last_unstale":"2024-02-19T14:56:21.719814+0000","last_undegraded":"2024-02-19T14:56:21.719814+0000","last_fullsized":"2024-02-19T14:56:21.719814+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:47.040368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341650+0000","last_change":"2024-02-19T14:56:18.339343+0000","last_active":"2024-02-19T14:56:20.341650+0000","last_peered":"2024-02-19T14:56:20.341650+0000","last_clean":"2024-02-19T14:56:20.341650+0000","last_became_active":"2024-02-19T14:56:18.339088+0000","last_became_peered":"2024-02-19T14:56:18.339088+0000","last_unstale":"2024-02-19T14:56:20.341650+0000","last_undegraded":"2024-02-19T14:56:20.341650+0000","last_fullsized":"2024-02-19T14:56:20.341650+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:42.020378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720919+0000","last_change":"2024-02-19T14:56:19.350838+0000","last_active":"2024-02-19T14:56:21.720919+0000","last_peered":"2024-02-19T14:56:21.720919+0000","last_clean":"2024-02-19T14:56:21.720919+0000","last_became_active":"2024-02-19T14:56:19.350551+0000","last_became_peered":"2024-02-19T14:56:19.350551+0000","last_unstale":"2024-02-19T14:56:21.720919+0000","last_undegraded":"2024-02-19T14:56:21.720919+0000","last_fullsized":"2024-02-19T14:56:21.720919+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:10.435827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719611+0000","last_change":"2024-02-19T14:56:18.346789+0000","last_active":"2024-02-19T14:56:21.719611+0000","last_peered":"2024-02-19T14:56:21.719611+0000","last_clean":"2024-02-19T14:56:21.719611+0000","last_became_active":"2024-02-19T14:56:18.346531+0000","last_became_peered":"2024-02-19T14:56:18.346531+0000","last_unstale":"2024-02-19T14:56:21.719611+0000","last_undegraded":"2024-02-19T14:56:21.719611+0000","last_fullsized":"2024-02-19T14:56:21.719611+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:42.020378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719130+0000","last_change":"2024-02-19T14:56:19.354318+0000","last_active":"2024-02-19T14:56:21.719130+0000","last_peered":"2024-02-19T14:56:21.719130+0000","last_clean":"2024-02-19T14:56:21.719130+0000","last_became_active":"2024-02-19T14:56:19.354061+0000","last_became_peered":"2024-02-19T14:56:19.354061+0000","last_unstale":"2024-02-19T14:56:21.719130+0000","last_undegraded":"2024-02-19T14:56:21.719130+0000","last_fullsized":"2024-02-19T14:56:21.719130+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:10.848474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340677+0000","last_change":"2024-02-19T14:56:18.342960+0000","last_active":"2024-02-19T14:56:20.340677+0000","last_peered":"2024-02-19T14:56:20.340677+0000","last_clean":"2024-02-19T14:56:20.340677+0000","last_became_active":"2024-02-19T14:56:18.342695+0000","last_became_peered":"2024-02-19T14:56:18.342695+0000","last_unstale":"2024-02-19T14:56:20.340677+0000","last_undegraded":"2024-02-19T14:56:20.340677+0000","last_fullsized":"2024-02-19T14:56:20.340677+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:51.277886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341820+0000","last_change":"2024-02-19T14:56:19.349621+0000","last_active":"2024-02-19T14:56:20.341820+0000","last_peered":"2024-02-19T14:56:20.341820+0000","last_clean":"2024-02-19T14:56:20.341820+0000","last_became_active":"2024-02-19T14:56:19.349344+0000","last_became_peered":"2024-02-19T14:56:19.349344+0000","last_unstale":"2024-02-19T14:56:20.341820+0000","last_undegraded":"2024-02-19T14:56:20.341820+0000","last_fullsized":"2024-02-19T14:56:20.341820+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:22.307617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719287+0000","last_change":"2024-02-19T14:56:18.343863+0000","last_active":"2024-02-19T14:56:21.719287+0000","last_peered":"2024-02-19T14:56:21.719287+0000","last_clean":"2024-02-19T14:56:21.719287+0000","last_became_active":"2024-02-19T14:56:18.343608+0000","last_became_peered":"2024-02-19T14:56:18.343608+0000","last_unstale":"2024-02-19T14:56:21.719287+0000","last_undegraded":"2024-02-19T14:56:21.719287+0000","last_fullsized":"2024-02-19T14:56:21.719287+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:05:58.768588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720088+0000","last_change":"2024-02-19T14:56:19.353795+0000","last_active":"2024-02-19T14:56:21.720088+0000","last_peered":"2024-02-19T14:56:21.720088+0000","last_clean":"2024-02-19T14:56:21.720088+0000","last_became_active":"2024-02-19T14:56:19.353378+0000","last_became_peered":"2024-02-19T14:56:19.353378+0000","last_unstale":"2024-02-19T14:56:21.720088+0000","last_undegraded":"2024-02-19T14:56:21.720088+0000","last_fullsized":"2024-02-19T14:56:21.720088+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:53:06.250904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"123'3","reported_seq":17,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720711+0000","last_change":"2024-02-19T14:56:18.346589+0000","last_active":"2024-02-19T14:56:21.720711+0000","last_peered":"2024-02-19T14:56:21.720711+0000","last_clean":"2024-02-19T14:56:21.720711+0000","last_became_active":"2024-02-19T14:56:18.346292+0000","last_became_peered":"2024-02-19T14:56:18.346292+0000","last_unstale":"2024-02-19T14:56:21.720711+0000","last_undegraded":"2024-02-19T14:56:21.720711+0000","last_fullsized":"2024-02-19T14:56:21.720711+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:25:26.822695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720512+0000","last_change":"2024-02-19T14:56:19.357606+0000","last_active":"2024-02-19T14:56:21.720512+0000","last_peered":"2024-02-19T14:56:21.720512+0000","last_clean":"2024-02-19T14:56:21.720512+0000","last_became_active":"2024-02-19T14:56:19.357319+0000","last_became_peered":"2024-02-19T14:56:19.357319+0000","last_unstale":"2024-02-19T14:56:21.720512+0000","last_undegraded":"2024-02-19T14:56:21.720512+0000","last_fullsized":"2024-02-19T14:56:21.720512+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:34.255709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"123'6","reported_seq":20,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720279+0000","last_change":"2024-02-19T14:56:18.349678+0000","last_active":"2024-02-19T14:56:21.720279+0000","last_peered":"2024-02-19T14:56:21.720279+0000","last_clean":"2024-02-19T14:56:21.720279+0000","last_became_active":"2024-02-19T14:56:18.349033+0000","last_became_peered":"2024-02-19T14:56:18.349033+0000","last_unstale":"2024-02-19T14:56:21.720279+0000","last_undegraded":"2024-02-19T14:56:21.720279+0000","last_fullsized":"2024-02-19T14:56:21.720279+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:21.302959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720504+0000","last_change":"2024-02-19T14:56:19.350814+0000","last_active":"2024-02-19T14:56:21.720504+0000","last_peered":"2024-02-19T14:56:21.720504+0000","last_clean":"2024-02-19T14:56:21.720504+0000","last_became_active":"2024-02-19T14:56:19.350494+0000","last_became_peered":"2024-02-19T14:56:19.350494+0000","last_unstale":"2024-02-19T14:56:21.720504+0000","last_undegraded":"2024-02-19T14:56:21.720504+0000","last_fullsized":"2024-02-19T14:56:21.720504+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:03.912386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"123'2","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719278+0000","last_change":"2024-02-19T14:56:18.346720+0000","last_active":"2024-02-19T14:56:21.719278+0000","last_peered":"2024-02-19T14:56:21.719278+0000","last_clean":"2024-02-19T14:56:21.719278+0000","last_became_active":"2024-02-19T14:56:18.346446+0000","last_became_peered":"2024-02-19T14:56:18.346446+0000","last_unstale":"2024-02-19T14:56:21.719278+0000","last_undegraded":"2024-02-19T14:56:21.719278+0000","last_fullsized":"2024-02-19T14:56:21.719278+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:21.077815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720525+0000","last_change":"2024-02-19T14:56:19.351336+0000","last_active":"2024-02-19T14:56:21.720525+0000","last_peered":"2024-02-19T14:56:21.720525+0000","last_clean":"2024-02-19T14:56:21.720525+0000","last_became_active":"2024-02-19T14:56:19.350862+0000","last_became_peered":"2024-02-19T14:56:19.350862+0000","last_unstale":"2024-02-19T14:56:21.720525+0000","last_undegraded":"2024-02-19T14:56:21.720525+0000","last_fullsized":"2024-02-19T14:56:21.720525+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:49:52.539911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719790+0000","last_change":"2024-02-19T14:56:18.342864+0000","last_active":"2024-02-19T14:56:21.719790+0000","last_peered":"2024-02-19T14:56:21.719790+0000","last_clean":"2024-02-19T14:56:21.719790+0000","last_became_active":"2024-02-19T14:56:18.342584+0000","last_became_peered":"2024-02-19T14:56:18.342584+0000","last_unstale":"2024-02-19T14:56:21.719790+0000","last_undegraded":"2024-02-19T14:56:21.719790+0000","last_fullsized":"2024-02-19T14:56:21.719790+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:27:11.521056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720850+0000","last_change":"2024-02-19T14:56:19.345116+0000","last_active":"2024-02-19T14:56:21.720850+0000","last_peered":"2024-02-19T14:56:21.720850+0000","last_clean":"2024-02-19T14:56:21.720850+0000","last_became_active":"2024-02-19T14:56:19.344851+0000","last_became_peered":"2024-02-19T14:56:19.344851+0000","last_unstale":"2024-02-19T14:56:21.720850+0000","last_undegraded":"2024-02-19T14:56:21.720850+0000","last_fullsized":"2024-02-19T14:56:21.720850+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:57:35.234054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"123'8","reported_seq":22,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341788+0000","last_change":"2024-02-19T14:56:18.343459+0000","last_active":"2024-02-19T14:56:20.341788+0000","last_peered":"2024-02-19T14:56:20.341788+0000","last_clean":"2024-02-19T14:56:20.341788+0000","last_became_active":"2024-02-19T14:56:18.343192+0000","last_became_peered":"2024-02-19T14:56:18.343192+0000","last_unstale":"2024-02-19T14:56:20.341788+0000","last_undegraded":"2024-02-19T14:56:20.341788+0000","last_fullsized":"2024-02-19T14:56:20.341788+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:36:48.857328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719701+0000","last_change":"2024-02-19T14:56:19.358072+0000","last_active":"2024-02-19T14:56:21.719701+0000","last_peered":"2024-02-19T14:56:21.719701+0000","last_clean":"2024-02-19T14:56:21.719701+0000","last_became_active":"2024-02-19T14:56:19.357786+0000","last_became_peered":"2024-02-19T14:56:19.357786+0000","last_unstale":"2024-02-19T14:56:21.719701+0000","last_undegraded":"2024-02-19T14:56:21.719701+0000","last_fullsized":"2024-02-19T14:56:21.719701+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:03:27.839253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"123'1","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340800+0000","last_change":"2024-02-19T14:56:18.346790+0000","last_active":"2024-02-19T14:56:20.340800+0000","last_peered":"2024-02-19T14:56:20.340800+0000","last_clean":"2024-02-19T14:56:20.340800+0000","last_became_active":"2024-02-19T14:56:18.346472+0000","last_became_peered":"2024-02-19T14:56:18.346472+0000","last_unstale":"2024-02-19T14:56:20.340800+0000","last_undegraded":"2024-02-19T14:56:20.340800+0000","last_fullsized":"2024-02-19T14:56:20.340800+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:56:39.415222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720215+0000","last_change":"2024-02-19T14:56:19.357076+0000","last_active":"2024-02-19T14:56:21.720215+0000","last_peered":"2024-02-19T14:56:21.720215+0000","last_clean":"2024-02-19T14:56:21.720215+0000","last_became_active":"2024-02-19T14:56:19.356138+0000","last_became_peered":"2024-02-19T14:56:19.356138+0000","last_unstale":"2024-02-19T14:56:21.720215+0000","last_undegraded":"2024-02-19T14:56:21.720215+0000","last_fullsized":"2024-02-19T14:56:21.720215+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:34:39.165012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"123'2","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720435+0000","last_change":"2024-02-19T14:56:18.349368+0000","last_active":"2024-02-19T14:56:21.720435+0000","last_peered":"2024-02-19T14:56:21.720435+0000","last_clean":"2024-02-19T14:56:21.720435+0000","last_became_active":"2024-02-19T14:56:18.349062+0000","last_became_peered":"2024-02-19T14:56:18.349062+0000","last_unstale":"2024-02-19T14:56:21.720435+0000","last_undegraded":"2024-02-19T14:56:21.720435+0000","last_fullsized":"2024-02-19T14:56:21.720435+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:56:33.874518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720055+0000","last_change":"2024-02-19T14:56:19.356870+0000","last_active":"2024-02-19T14:56:21.720055+0000","last_peered":"2024-02-19T14:56:21.720055+0000","last_clean":"2024-02-19T14:56:21.720055+0000","last_became_active":"2024-02-19T14:56:19.356590+0000","last_became_peered":"2024-02-19T14:56:19.356590+0000","last_unstale":"2024-02-19T14:56:21.720055+0000","last_undegraded":"2024-02-19T14:56:21.720055+0000","last_fullsized":"2024-02-19T14:56:21.720055+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:18.579871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"123'1","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342107+0000","last_change":"2024-02-19T14:56:18.338194+0000","last_active":"2024-02-19T14:56:20.342107+0000","last_peered":"2024-02-19T14:56:20.342107+0000","last_clean":"2024-02-19T14:56:20.342107+0000","last_became_active":"2024-02-19T14:56:18.337928+0000","last_became_peered":"2024-02-19T14:56:18.337928+0000","last_unstale":"2024-02-19T14:56:20.342107+0000","last_undegraded":"2024-02-19T14:56:20.342107+0000","last_fullsized":"2024-02-19T14:56:20.342107+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:51.277886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720675+0000","last_change":"2024-02-19T14:56:19.355832+0000","last_active":"2024-02-19T14:56:21.720675+0000","last_peered":"2024-02-19T14:56:21.720675+0000","last_clean":"2024-02-19T14:56:21.720675+0000","last_became_active":"2024-02-19T14:56:19.355535+0000","last_became_peered":"2024-02-19T14:56:19.355535+0000","last_unstale":"2024-02-19T14:56:21.720675+0000","last_undegraded":"2024-02-19T14:56:21.720675+0000","last_fullsized":"2024-02-19T14:56:21.720675+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:08:44.117858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719943+0000","last_change":"2024-02-19T14:56:18.346686+0000","last_active":"2024-02-19T14:56:21.719943+0000","last_peered":"2024-02-19T14:56:21.719943+0000","last_clean":"2024-02-19T14:56:21.719943+0000","last_became_active":"2024-02-19T14:56:18.346442+0000","last_became_peered":"2024-02-19T14:56:18.346442+0000","last_unstale":"2024-02-19T14:56:21.719943+0000","last_undegraded":"2024-02-19T14:56:21.719943+0000","last_fullsized":"2024-02-19T14:56:21.719943+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:48:59.701004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719890+0000","last_change":"2024-02-19T14:56:19.341026+0000","last_active":"2024-02-19T14:56:21.719890+0000","last_peered":"2024-02-19T14:56:21.719890+0000","last_clean":"2024-02-19T14:56:21.719890+0000","last_became_active":"2024-02-19T14:56:19.340726+0000","last_became_peered":"2024-02-19T14:56:19.340726+0000","last_unstale":"2024-02-19T14:56:21.719890+0000","last_undegraded":"2024-02-19T14:56:21.719890+0000","last_fullsized":"2024-02-19T14:56:21.719890+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:18.579871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"123'2","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341085+0000","last_change":"2024-02-19T14:56:18.344415+0000","last_active":"2024-02-19T14:56:20.341085+0000","last_peered":"2024-02-19T14:56:20.341085+0000","last_clean":"2024-02-19T14:56:20.341085+0000","last_became_active":"2024-02-19T14:56:18.344103+0000","last_became_peered":"2024-02-19T14:56:18.344103+0000","last_unstale":"2024-02-19T14:56:20.341085+0000","last_undegraded":"2024-02-19T14:56:20.341085+0000","last_fullsized":"2024-02-19T14:56:20.341085+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:51:44.768738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719594+0000","last_change":"2024-02-19T14:56:19.348313+0000","last_active":"2024-02-19T14:56:21.719594+0000","last_peered":"2024-02-19T14:56:21.719594+0000","last_clean":"2024-02-19T14:56:21.719594+0000","last_became_active":"2024-02-19T14:56:19.348063+0000","last_became_peered":"2024-02-19T14:56:19.348063+0000","last_unstale":"2024-02-19T14:56:21.719594+0000","last_undegraded":"2024-02-19T14:56:21.719594+0000","last_fullsized":"2024-02-19T14:56:21.719594+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:40:14.589628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719416+0000","last_change":"2024-02-19T14:56:18.341288+0000","last_active":"2024-02-19T14:56:21.719416+0000","last_peered":"2024-02-19T14:56:21.719416+0000","last_clean":"2024-02-19T14:56:21.719416+0000","last_became_active":"2024-02-19T14:56:18.341028+0000","last_became_peered":"2024-02-19T14:56:18.341028+0000","last_unstale":"2024-02-19T14:56:21.719416+0000","last_undegraded":"2024-02-19T14:56:21.719416+0000","last_fullsized":"2024-02-19T14:56:21.719416+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:59:01.996933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719889+0000","last_change":"2024-02-19T14:56:19.355848+0000","last_active":"2024-02-19T14:56:21.719889+0000","last_peered":"2024-02-19T14:56:21.719889+0000","last_clean":"2024-02-19T14:56:21.719889+0000","last_became_active":"2024-02-19T14:56:19.355553+0000","last_became_peered":"2024-02-19T14:56:19.355553+0000","last_unstale":"2024-02-19T14:56:21.719889+0000","last_undegraded":"2024-02-19T14:56:21.719889+0000","last_fullsized":"2024-02-19T14:56:21.719889+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:27:18.005571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720579+0000","last_change":"2024-02-19T14:56:18.349853+0000","last_active":"2024-02-19T14:56:21.720579+0000","last_peered":"2024-02-19T14:56:21.720579+0000","last_clean":"2024-02-19T14:56:21.720579+0000","last_became_active":"2024-02-19T14:56:18.349527+0000","last_became_peered":"2024-02-19T14:56:18.349527+0000","last_unstale":"2024-02-19T14:56:21.720579+0000","last_undegraded":"2024-02-19T14:56:21.720579+0000","last_fullsized":"2024-02-19T14:56:21.720579+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:11:58.917358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340613+0000","last_change":"2024-02-19T14:56:19.354839+0000","last_active":"2024-02-19T14:56:20.340613+0000","last_peered":"2024-02-19T14:56:20.340613+0000","last_clean":"2024-02-19T14:56:20.340613+0000","last_became_active":"2024-02-19T14:56:19.354550+0000","last_became_peered":"2024-02-19T14:56:19.354550+0000","last_unstale":"2024-02-19T14:56:20.340613+0000","last_undegraded":"2024-02-19T14:56:20.340613+0000","last_fullsized":"2024-02-19T14:56:20.340613+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:21.644971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341854+0000","last_change":"2024-02-19T14:56:18.344454+0000","last_active":"2024-02-19T14:56:20.341854+0000","last_peered":"2024-02-19T14:56:20.341854+0000","last_clean":"2024-02-19T14:56:20.341854+0000","last_became_active":"2024-02-19T14:56:18.343893+0000","last_became_peered":"2024-02-19T14:56:18.343893+0000","last_unstale":"2024-02-19T14:56:20.341854+0000","last_undegraded":"2024-02-19T14:56:20.341854+0000","last_fullsized":"2024-02-19T14:56:20.341854+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:26:34.546328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719868+0000","last_change":"2024-02-19T14:56:19.347599+0000","last_active":"2024-02-19T14:56:21.719868+0000","last_peered":"2024-02-19T14:56:21.719868+0000","last_clean":"2024-02-19T14:56:21.719868+0000","last_became_active":"2024-02-19T14:56:19.347311+0000","last_became_peered":"2024-02-19T14:56:19.347311+0000","last_unstale":"2024-02-19T14:56:21.719868+0000","last_undegraded":"2024-02-19T14:56:21.719868+0000","last_fullsized":"2024-02-19T14:56:21.719868+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:49.203223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"123'2","reported_seq":16,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719814+0000","last_change":"2024-02-19T14:56:18.346507+0000","last_active":"2024-02-19T14:56:21.719814+0000","last_peered":"2024-02-19T14:56:21.719814+0000","last_clean":"2024-02-19T14:56:21.719814+0000","last_became_active":"2024-02-19T14:56:18.346245+0000","last_became_peered":"2024-02-19T14:56:18.346245+0000","last_unstale":"2024-02-19T14:56:21.719814+0000","last_undegraded":"2024-02-19T14:56:21.719814+0000","last_fullsized":"2024-02-19T14:56:21.719814+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:55:43.655891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719892+0000","last_change":"2024-02-19T14:56:19.357136+0000","last_active":"2024-02-19T14:56:21.719892+0000","last_peered":"2024-02-19T14:56:21.719892+0000","last_clean":"2024-02-19T14:56:21.719892+0000","last_became_active":"2024-02-19T14:56:19.356823+0000","last_became_peered":"2024-02-19T14:56:19.356823+0000","last_unstale":"2024-02-19T14:56:21.719892+0000","last_undegraded":"2024-02-19T14:56:21.719892+0000","last_fullsized":"2024-02-19T14:56:21.719892+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:14:47.235348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"124'1","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341291+0000","last_change":"2024-02-19T14:56:18.338883+0000","last_active":"2024-02-19T14:56:20.341291+0000","last_peered":"2024-02-19T14:56:20.341291+0000","last_clean":"2024-02-19T14:56:20.341291+0000","last_became_active":"2024-02-19T14:56:18.338572+0000","last_became_peered":"2024-02-19T14:56:18.338572+0000","last_unstale":"2024-02-19T14:56:20.341291+0000","last_undegraded":"2024-02-19T14:56:20.341291+0000","last_fullsized":"2024-02-19T14:56:20.341291+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:44.872180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720728+0000","last_change":"2024-02-19T14:56:19.353423+0000","last_active":"2024-02-19T14:56:21.720728+0000","last_peered":"2024-02-19T14:56:21.720728+0000","last_clean":"2024-02-19T14:56:21.720728+0000","last_became_active":"2024-02-19T14:56:19.353133+0000","last_became_peered":"2024-02-19T14:56:19.353133+0000","last_unstale":"2024-02-19T14:56:21.720728+0000","last_undegraded":"2024-02-19T14:56:21.720728+0000","last_fullsized":"2024-02-19T14:56:21.720728+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:11:46.891032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341117+0000","last_change":"2024-02-19T14:56:18.340894+0000","last_active":"2024-02-19T14:56:20.341117+0000","last_peered":"2024-02-19T14:56:20.341117+0000","last_clean":"2024-02-19T14:56:20.341117+0000","last_became_active":"2024-02-19T14:56:18.340609+0000","last_became_peered":"2024-02-19T14:56:18.340609+0000","last_unstale":"2024-02-19T14:56:20.341117+0000","last_undegraded":"2024-02-19T14:56:20.341117+0000","last_fullsized":"2024-02-19T14:56:20.341117+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:20:02.698430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720798+0000","last_change":"2024-02-19T14:56:19.352654+0000","last_active":"2024-02-19T14:56:21.720798+0000","last_peered":"2024-02-19T14:56:21.720798+0000","last_clean":"2024-02-19T14:56:21.720798+0000","last_became_active":"2024-02-19T14:56:19.352348+0000","last_became_peered":"2024-02-19T14:56:19.352348+0000","last_unstale":"2024-02-19T14:56:21.720798+0000","last_undegraded":"2024-02-19T14:56:21.720798+0000","last_fullsized":"2024-02-19T14:56:21.720798+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:03:07.869942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341971+0000","last_change":"2024-02-19T14:56:18.343868+0000","last_active":"2024-02-19T14:56:20.341971+0000","last_peered":"2024-02-19T14:56:20.341971+0000","last_clean":"2024-02-19T14:56:20.341971+0000","last_became_active":"2024-02-19T14:56:18.343595+0000","last_became_peered":"2024-02-19T14:56:18.343595+0000","last_unstale":"2024-02-19T14:56:20.341971+0000","last_undegraded":"2024-02-19T14:56:20.341971+0000","last_fullsized":"2024-02-19T14:56:20.341971+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:56:39.415222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342050+0000","last_change":"2024-02-19T14:56:19.340888+0000","last_active":"2024-02-19T14:56:20.342050+0000","last_peered":"2024-02-19T14:56:20.342050+0000","last_clean":"2024-02-19T14:56:20.342050+0000","last_became_active":"2024-02-19T14:56:19.340610+0000","last_became_peered":"2024-02-19T14:56:19.340610+0000","last_unstale":"2024-02-19T14:56:20.342050+0000","last_undegraded":"2024-02-19T14:56:20.342050+0000","last_fullsized":"2024-02-19T14:56:20.342050+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:21:19.815092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.340742+0000","last_change":"2024-02-19T14:56:18.346460+0000","last_active":"2024-02-19T14:56:20.340742+0000","last_peered":"2024-02-19T14:56:20.340742+0000","last_clean":"2024-02-19T14:56:20.340742+0000","last_became_active":"2024-02-19T14:56:18.346176+0000","last_became_peered":"2024-02-19T14:56:18.346176+0000","last_unstale":"2024-02-19T14:56:20.340742+0000","last_undegraded":"2024-02-19T14:56:20.340742+0000","last_fullsized":"2024-02-19T14:56:20.340742+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:26:34.546328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342151+0000","last_change":"2024-02-19T14:56:19.339022+0000","last_active":"2024-02-19T14:56:20.342151+0000","last_peered":"2024-02-19T14:56:20.342151+0000","last_clean":"2024-02-19T14:56:20.342151+0000","last_became_active":"2024-02-19T14:56:19.338743+0000","last_became_peered":"2024-02-19T14:56:19.338743+0000","last_unstale":"2024-02-19T14:56:20.342151+0000","last_undegraded":"2024-02-19T14:56:20.342151+0000","last_fullsized":"2024-02-19T14:56:20.342151+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:05.195677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720112+0000","last_change":"2024-02-19T14:56:18.344871+0000","last_active":"2024-02-19T14:56:21.720112+0000","last_peered":"2024-02-19T14:56:21.720112+0000","last_clean":"2024-02-19T14:56:21.720112+0000","last_became_active":"2024-02-19T14:56:18.344632+0000","last_became_peered":"2024-02-19T14:56:18.344632+0000","last_unstale":"2024-02-19T14:56:21.720112+0000","last_undegraded":"2024-02-19T14:56:21.720112+0000","last_fullsized":"2024-02-19T14:56:21.720112+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:20.640313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.342076+0000","last_change":"2024-02-19T14:56:19.350121+0000","last_active":"2024-02-19T14:56:20.342076+0000","last_peered":"2024-02-19T14:56:20.342076+0000","last_clean":"2024-02-19T14:56:20.342076+0000","last_became_active":"2024-02-19T14:56:19.349780+0000","last_became_peered":"2024-02-19T14:56:19.349780+0000","last_unstale":"2024-02-19T14:56:20.342076+0000","last_undegraded":"2024-02-19T14:56:20.342076+0000","last_fullsized":"2024-02-19T14:56:20.342076+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:00:31.545533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:20.341749+0000","last_change":"2024-02-19T14:56:18.344032+0000","last_active":"2024-02-19T14:56:20.341749+0000","last_peered":"2024-02-19T14:56:20.341749+0000","last_clean":"2024-02-19T14:56:20.341749+0000","last_became_active":"2024-02-19T14:56:18.343759+0000","last_became_peered":"2024-02-19T14:56:18.343759+0000","last_unstale":"2024-02-19T14:56:20.341749+0000","last_undegraded":"2024-02-19T14:56:20.341749+0000","last_fullsized":"2024-02-19T14:56:20.341749+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:50:07.279634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720880+0000","last_change":"2024-02-19T14:56:19.354068+0000","last_active":"2024-02-19T14:56:21.720880+0000","last_peered":"2024-02-19T14:56:21.720880+0000","last_clean":"2024-02-19T14:56:21.720880+0000","last_became_active":"2024-02-19T14:56:19.353825+0000","last_became_peered":"2024-02-19T14:56:19.353825+0000","last_unstale":"2024-02-19T14:56:21.720880+0000","last_undegraded":"2024-02-19T14:56:21.720880+0000","last_fullsized":"2024-02-19T14:56:21.720880+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:42:34.077666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719655+0000","last_change":"2024-02-19T14:56:18.344103+0000","last_active":"2024-02-19T14:56:21.719655+0000","last_peered":"2024-02-19T14:56:21.719655+0000","last_clean":"2024-02-19T14:56:21.719655+0000","last_became_active":"2024-02-19T14:56:18.343847+0000","last_became_peered":"2024-02-19T14:56:18.343847+0000","last_unstale":"2024-02-19T14:56:21.719655+0000","last_undegraded":"2024-02-19T14:56:21.719655+0000","last_fullsized":"2024-02-19T14:56:21.719655+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:27:20.640313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.720486+0000","last_change":"2024-02-19T14:56:19.359391+0000","last_active":"2024-02-19T14:56:21.720486+0000","last_peered":"2024-02-19T14:56:21.720486+0000","last_clean":"2024-02-19T14:56:21.720486+0000","last_became_active":"2024-02-19T14:56:19.358802+0000","last_became_peered":"2024-02-19T14:56:19.358802+0000","last_unstale":"2024-02-19T14:56:21.720486+0000","last_undegraded":"2024-02-19T14:56:21.720486+0000","last_fullsized":"2024-02-19T14:56:21.720486+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:18.322503+0000","last_clean_scrub_stamp":"2024-02-19T14:56:18.322503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:22.307617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719854+0000","last_change":"2024-02-19T14:56:18.349718+0000","last_active":"2024-02-19T14:56:21.719854+0000","last_peered":"2024-02-19T14:56:21.719854+0000","last_clean":"2024-02-19T14:56:21.719854+0000","last_became_active":"2024-02-19T14:56:18.349077+0000","last_became_peered":"2024-02-19T14:56:18.349077+0000","last_unstale":"2024-02-19T14:56:21.719854+0000","last_undegraded":"2024-02-19T14:56:21.719854+0000","last_fullsized":"2024-02-19T14:56:21.719854+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:17.317845+0000","last_clean_scrub_stamp":"2024-02-19T14:56:17.317845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:53:05.246246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":365,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-02-19T14:56:21.719456+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:21.719456+0000","last_peered":"2024-02-19T14:56:21.719456+0000","last_clean":"2024-02-19T14:56:21.719456+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:21.719456+0000","last_undegraded":"2024-02-19T14:56:21.719456+0000","last_fullsized":"2024-02-19T14:56:21.719456+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33296,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338544,"statfs":{"total":96636764160,"available":96602669056,"internally_reserved":0,"allocated":2162688,"data_stored":1345206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.46700000000000003},"min":{"1min":0.33300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.495},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.35199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.69499999999999995,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.42699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.375,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72299999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67300000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.48699999999999999,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66300000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68799999999999994,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65800000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58199999999999996,"15min":0.59999999999999998},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70399999999999996,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.57499999999999996,"15min":0.59199999999999997},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.495}]}]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1560576,"data_stored":753808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.48499999999999999,"15min":0.46700000000000003},"min":{"1min":0.34999999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.49199999999999999,"15min":0.49299999999999999},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54500000000000004,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.504}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.64000000000000001,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68600000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48499999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70899999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52200000000000002,"15min":0.54200000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70899999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82699999999999996,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.58899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.625}]}]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":753158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.373,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69899999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52000000000000002,"15min":0.504},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.49199999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.75,"5min":0.872,"15min":0.872},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68200000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65400000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68799999999999994,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73499999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.64100000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.51300000000000001,"15min":0.52600000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72099999999999997,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.626},{"interface":"front","average":{"1min":0.503,"5min":0.54500000000000004,"15min":0.56499999999999995},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.76500000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":0.877,"15min":0.877},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.57699999999999996},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.29599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.84999999999999998,"5min":1.03,"15min":1.03},"last":0.79200000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1552384,"data_stored":753136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.33900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71399999999999997,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.74199999999999999,"5min":0.746,"15min":0.746},"last":0.28599999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.877,"15min":0.877},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.376,"5min":0.37,"15min":0.37},"max":{"1min":0.69199999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54200000000000004,"15min":0.54500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52200000000000002},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80900000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":1.038,"15min":1.038},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.495}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":1.038,"15min":1.038},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.623,"15min":0.629},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73299999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.58599999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32716,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339124,"statfs":{"total":96636764160,"available":96603262976,"internally_reserved":0,"allocated":1568768,"data_stored":754818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34100000000000003,"15min":0.31},"min":{"1min":0.22700000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.48999999999999999,"5min":3.694,"15min":3.694},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.36199999999999999,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.189,"15min":0.189},"max":{"1min":0.47499999999999998,"5min":4.835,"15min":4.835},"last":0.23400000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.313,"15min":0.311},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.48199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42699999999999999,"15min":0.379},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.55600000000000005,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.27500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58799999999999997,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.373},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.38900000000000001,"15min":0.34699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54500000000000004,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29399999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.37,"15min":0.37},"max":{"1min":0.81100000000000005,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.63,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55700000000000005,"15min":0.56999999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82799999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77500000000000002,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76600000000000001,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32708,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339132,"statfs":{"total":96636764160,"available":96603271168,"internally_reserved":0,"allocated":1560576,"data_stored":753752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27500000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.32100000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.25},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.5}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.56699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32704,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339136,"statfs":{"total":96636764160,"available":96603275264,"internally_reserved":0,"allocated":1556480,"data_stored":753206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.217,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.42199999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.498,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.307},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42199999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35399999999999998,"15min":0.36299999999999999},"min":{"1min":0.251,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47299999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39200000000000002,"15min":0.39700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57899999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57699999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.68500000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76400000000000001,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":1.01,"15min":1.01},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69899999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.626}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71099999999999997,"5min":1.026,"15min":1.026},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69599999999999995,"5min":0.996,"15min":0.996},"last":0.69299999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":1.036,"15min":1.036},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2154496,"data_stored":1343560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.29399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.251}]},{"osd":2,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.372},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34699999999999998,"15min":0.35099999999999998},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.224,"15min":0.224},"max":{"1min":0.48399999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.442},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.58599999999999997},"min":{"1min":0.41299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84799999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58499999999999996,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79200000000000004,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56799999999999995,"15min":0.58499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40300000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.432}]},{"osd":7,"last update":"Mon Feb 19 14:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.39300000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.86399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.59299999999999997},"min":{"1min":0.435,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75600000000000001,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:56:27.116 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:56:27.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:56:27.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.257+0000 7f83599c9640 1 Processor -- start 2024-02-19T14:56:27.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.257+0000 7f83599c9640 1 -- start start 2024-02-19T14:56:27.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83599c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 0x7f8354158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83599c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83540c34a0 con 0x7f83540c42e0 2024-02-19T14:56:27.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83599c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835406c8e0 con 0x7f83540c8500 2024-02-19T14:56:27.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83599c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835406c090 con 0x7f83540c2f50 2024-02-19T14:56:27.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83527fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f83540c42e0 0x7f835415d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55098/0 (socket says 172.21.15.78:55098) 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83527fc640 1 -- 172.21.15.78:0/3309294065 learned_addr learned my addr 172.21.15.78:0/3309294065 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83537fe640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 0x7f8354158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281008705 0 0) 0x7f83540c34a0 con 0x7f83540c42e0 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8340003520 con 0x7f83540c42e0 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985483641 0 0) 0x7f835406c8e0 con 0x7f83540c8500 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83540c34a0 con 0x7f83540c8500 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536577063 0 0) 0x7f8340003520 con 0x7f83540c42e0 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 msgr2=0x7f8354158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 0x7f8354158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 >> v1:172.21.15.150:6789/0 conn(0x7f83540c8500 legacy=0x7f8354158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835406cbc0 con 0x7f83540c42e0 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f833c001670 con 0x7f83540c42e0 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 >> v1:172.21.15.78:6789/0 conn(0x7f83540c42e0 legacy=0x7f835415d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8340003b30 0x7f8340003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400047c0 0x7f8340004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 0x7f83400051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834000bb10 con 0x7f83400047c0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834000bc50 con 0x7f8340003b30 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834000bde0 con 0x7f83540c2f50 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.258+0000 7f83527fc640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400047c0 0x7f8340004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f8352ffd640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8340003b30 0x7f8340003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f83537fe640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 0x7f83400051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f83527fc640 1 -- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 msgr2=0x7f83400051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f83527fc640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83540c2f50 0x7f83400051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f83527fc640 1 -- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8340003b30 msgr2=0x7f8340003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f83527fc640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8340003b30 0x7f8340003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.259+0000 7f83527fc640 1 -- 172.21.15.78:0/3309294065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340003520 con 0x7f83400047c0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.260+0000 7f83527fc640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400047c0 0x7f8340004c40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f833c007fd0 tx=0x7f83400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eff6856c83961a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.260+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c005ad0 con 0x7f83400047c0 2024-02-19T14:56:27.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.260+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c005c30 con 0x7f83400047c0 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.260+0000 7f8351ffb640 1 -- 172.21.15.78:0/3309294065 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c007070 con 0x7f83400047c0 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83599c9640 1 -- 172.21.15.78:0/3309294065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400047c0 msgr2=0x7f8340004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83599c9640 1 --2- 172.21.15.78:0/3309294065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400047c0 0x7f8340004c40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f833c007fd0 tx=0x7f83400057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83537fe640 1 -- 172.21.15.78:0/3309294065 reap_dead start 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83599c9640 1 -- 172.21.15.78:0/3309294065 shutdown_connections 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83599c9640 1 -- 172.21.15.78:0/3309294065 >> 172.21.15.78:0/3309294065 conn(0x7f83540aa300 msgr2=0x7f83540aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83599c9640 1 -- 172.21.15.78:0/3309294065 shutdown_connections 2024-02-19T14:56:27.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.261+0000 7f83599c9640 1 -- 172.21.15.78:0/3309294065 wait complete. 2024-02-19T14:56:27.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.262+0000 7f83599c9640 1 Processor -- start 2024-02-19T14:56:27.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.262+0000 7f83599c9640 1 -- start start 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83400047c0 0x7f83540c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 0x7f835414dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83541eacf0 0x7f83541eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83541ee260 con 0x7f83540ad260 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83541eeda0 con 0x7f83541eacf0 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83541ee470 con 0x7f83400047c0 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 0x7f835414dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 0x7f835414dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54666/0 (socket says 172.21.15.78:54666) 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 -- 172.21.15.78:0/2432564427 learned_addr learned my addr 172.21.15.78:0/2432564427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f8352ffd640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83400047c0 0x7f83540c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83537fe640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83541eacf0 0x7f83541eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 -- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83400047c0 msgr2=0x7f83540c52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83400047c0 0x7f83540c52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 -- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83541eacf0 msgr2=0x7f83541eb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83541eacf0 0x7f83541eb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 -- 172.21.15.78:0/2432564427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83541ee750 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83527fc640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 0x7f835414dbf0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f833c007f70 tx=0x7f833c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c03c040 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c007480 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c005ab0 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.263+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83541eea30 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.264+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835414e130 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.265+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8354071c20 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.265+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f833c032070 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.265+0000 7f834bfff640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f833406b840 0x7f833406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.265+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f833c0ab470 con 0x7f83540ad260 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.265+0000 7f8352ffd640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f833406b840 0x7f833406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.266+0000 7f8352ffd640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f833406b840 0x7f833406dd00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8344000c00 tx=0x7f8344009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:27.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.270+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f833c075d90 con 0x7f83540ad260 2024-02-19T14:56:27.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.525+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8354074990 con 0x7f83540ad260 2024-02-19T14:56:27.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.527+0000 7f834bfff640 1 -- 172.21.15.78:0/2432564427 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f833c079450 con 0x7f83540ad260 2024-02-19T14:56:27.530 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:56:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:56:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:56:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:56:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:56:27.530 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f833406b840 msgr2=0x7f833406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f833406b840 0x7f833406dd00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8344000c00 tx=0x7f8344009040 comp rx=0 tx=0).stop 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 msgr2=0x7f835414dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 0x7f835414dbf0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f833c007f70 tx=0x7f833c02e070 comp rx=0 tx=0).stop 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 shutdown_connections 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f833406b840 0x7f833406dd00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83540ad260 0x7f835414dbf0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83541eacf0 0x7f83541eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 --2- 172.21.15.78:0/2432564427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83400047c0 0x7f83540c52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 >> 172.21.15.78:0/2432564427 conn(0x7f83540aa300 msgr2=0x7f8354153b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 shutdown_connections 2024-02-19T14:56:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.529+0000 7f83599c9640 1 -- 172.21.15.78:0/2432564427 wait complete. 2024-02-19T14:56:27.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.639+0000 7f24660d4640 1 Processor -- start 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.639+0000 7f24660d4640 1 -- start start 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f24660d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f24600c8e00 0x7f246006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f24660d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f246006d910 0x7f246006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f24660d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246006cd20 con 0x7f2460152700 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f24660d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24601551f0 con 0x7f246006d910 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f24660d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2460155490 con 0x7f24600c8e00 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f246006d910 0x7f246006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f246006d910 0x7f246006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47370/0 (socket says 172.21.15.78:47370) 2024-02-19T14:56:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 -- 172.21.15.78:0/4019098028 learned_addr learned my addr 172.21.15.78:0/4019098028 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 -- 172.21.15.78:0/4019098028 >> v2:172.21.15.150:3300/0 conn(0x7f24600c8e00 msgr2=0x7f246006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 --2- 172.21.15.78:0/4019098028 >> v2:172.21.15.150:3300/0 conn(0x7f24600c8e00 0x7f246006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 -- 172.21.15.78:0/4019098028 >> v1:172.21.15.150:6789/0 conn(0x7f2460152700 legacy=0x7f24600c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.640+0000 7f245ffff640 1 -- 172.21.15.78:0/4019098028 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2460155770 con 0x7f246006d910 2024-02-19T14:56:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f245ffff640 1 --2- 172.21.15.78:0/4019098028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f246006c7e0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f24600c7620 tx=0x7f245802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a239f0628c9501 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f245effd640 1 -- 172.21.15.78:0/4019098028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245802ca80 con 0x7f246006d910 2024-02-19T14:56:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f245effd640 1 -- 172.21.15.78:0/4019098028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245802cbe0 con 0x7f246006d910 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f245effd640 1 -- 172.21.15.78:0/4019098028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24580343f0 con 0x7f246006d910 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 -- 172.21.15.78:0/4019098028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 msgr2=0x7f246006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 --2- 172.21.15.78:0/4019098028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f246006c7e0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f24600c7620 tx=0x7f245802bee0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 -- 172.21.15.78:0/4019098028 shutdown_connections 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 --2- 172.21.15.78:0/4019098028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f246006c7e0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 --2- 172.21.15.78:0/4019098028 >> v2:172.21.15.150:3300/0 conn(0x7f24600c8e00 0x7f246006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 -- 172.21.15.78:0/4019098028 >> 172.21.15.78:0/4019098028 conn(0x7f24600cac90 msgr2=0x7f24600cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 -- 172.21.15.78:0/4019098028 shutdown_connections 2024-02-19T14:56:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.642+0000 7f24660d4640 1 -- 172.21.15.78:0/4019098028 wait complete. 2024-02-19T14:56:27.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.643+0000 7f24660d4640 1 Processor -- start 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.643+0000 7f24660d4640 1 -- start start 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f24660d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f24601ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f24660d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24600c8e00 0x7f24600aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f24660d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2460152700 0x7f24600c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f24660d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24601497d0 con 0x7f246006d910 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f24660d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600c1940 con 0x7f2460152700 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f24660d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24600c1c00 con 0x7f24600c8e00 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f24601ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f24601ea5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47382/0 (socket says 172.21.15.78:47382) 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f245f7fe640 1 -- 172.21.15.78:0/2295024655 learned_addr learned my addr 172.21.15.78:0/2295024655 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f2457fff640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24600c8e00 0x7f24600aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.644+0000 7f245ffff640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2460152700 0x7f24600c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245f7fe640 1 -- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24600c8e00 msgr2=0x7f24600aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245f7fe640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24600c8e00 0x7f24600aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245f7fe640 1 -- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2460152700 msgr2=0x7f24600c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245f7fe640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2460152700 0x7f24600c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245f7fe640 1 -- 172.21.15.78:0/2295024655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458009b30 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245f7fe640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f24601ea5a0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f245000b3e0 tx=0x7f245000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450011830 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24500119b0 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450011d60 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2460159140 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.645+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24600acac0 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.646+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460074990 con 0x7f246006d910 2024-02-19T14:56:27.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.646+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f245001a400 con 0x7f246006d910 2024-02-19T14:56:27.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.647+0000 7f245cff9640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f244006b810 0x7f244006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:27.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.647+0000 7f2457fff640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f244006b810 0x7f244006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:27.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.647+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f245008a410 con 0x7f246006d910 2024-02-19T14:56:27.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.647+0000 7f2457fff640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f244006b810 0x7f244006dcd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f246014d4f0 tx=0x7f2448008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:27.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.651+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2450054d30 con 0x7f246006d910 2024-02-19T14:56:27.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.867+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24600c8770 con 0x7f246006d910 2024-02-19T14:56:27.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.868+0000 7f245cff9640 1 -- 172.21.15.78:0/2295024655 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f24500583f0 con 0x7f246006d910 2024-02-19T14:56:27.871 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:27.871 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7788,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:20.692270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7791":{"gid":7791,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/100250519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":100250519},{"type":"v1","addr":"172.21.15.78:6837","nonce":100250519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:27.871 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 100 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f244006b810 msgr2=0x7f244006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f244006b810 0x7f244006dcd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f246014d4f0 tx=0x7f2448008040 comp rx=0 tx=0).stop 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 msgr2=0x7f24601ea5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f24601ea5a0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f245000b3e0 tx=0x7f245000b8a0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 shutdown_connections 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f244006b810 0x7f244006dcd0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2460152700 0x7f24600c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24600c8e00 0x7f24600aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 --2- 172.21.15.78:0/2295024655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f246006d910 0x7f24601ea5a0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 >> 172.21.15.78:0/2295024655 conn(0x7f24600cac90 msgr2=0x7f246014ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 shutdown_connections 2024-02-19T14:56:27.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:27.870+0000 7f24660d4640 1 -- 172.21.15.78:0/2295024655 wait complete. 2024-02-19T14:56:28.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:20.692270+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7791': {'gid': 7791, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/100250519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 100250519}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 100250519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:28.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:56:28.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:56:28.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 Processor -- start 2024-02-19T14:56:28.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 -- start start 2024-02-19T14:56:28.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f87380c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8738070720 0x7f8738070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87380c65a0 con 0x7f87380c7910 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380c7530 con 0x7f8738070720 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f873c8f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87381537b0 con 0x7f87380c5560 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f872dd9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f87380c7910 0x7f8738154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40768/0 (socket says 172.21.15.78:40768) 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.393+0000 7f872dd9b640 1 -- 172.21.15.78:0/1316985563 learned_addr learned my addr 172.21.15.78:0/1316985563 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8736d9d640 1 --2- 172.21.15.78:0/1316985563 >> v2:172.21.15.150:3300/0 conn(0x7f8738070720 0x7f8738070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f87380c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650256435 0 0) 0x7f87380c65a0 con 0x7f87380c7910 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8724003520 con 0x7f87380c7910 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711962524 0 0) 0x7f8724003520 con 0x7f87380c7910 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 msgr2=0x7f87380c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f87380c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 >> v2:172.21.15.150:3300/0 conn(0x7f8738070720 msgr2=0x7f8738070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 --2- 172.21.15.78:0/1316985563 >> v2:172.21.15.150:3300/0 conn(0x7f8738070720 0x7f8738070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738153a90 con 0x7f87380c7910 2024-02-19T14:56:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f8720001670 con 0x7f87380c7910 2024-02-19T14:56:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 >> v1:172.21.15.78:6789/0 conn(0x7f87380c7910 legacy=0x7f8738154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8724003970 0x7f8724003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87240047c0 0x7f8724004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f8724005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380c7530 con 0x7f8724003970 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87381537b0 con 0x7f87240047c0 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872400ba00 con 0x7f87380c5560 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8724003970 0x7f8724003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f872dd9b640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87240047c0 0x7f8724004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8736d9d640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f8724005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 -- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 msgr2=0x7f8724005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f8724005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 -- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87240047c0 msgr2=0x7f8724004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87240047c0 0x7f8724004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f873659c640 1 -- 172.21.15.78:0/1316985563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724003520 con 0x7f8724003970 2024-02-19T14:56:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.394+0000 7f8736d9d640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87380c5560 0x7f8724005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.395+0000 7f873659c640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8724003970 0x7f8724003d90 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f8728009ac0 tx=0x7f87240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b594d20ea484eb23 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.395+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872802c6b0 con 0x7f8724003970 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.395+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872802c810 con 0x7f8724003970 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.395+0000 7f8735d9b640 1 -- 172.21.15.78:0/1316985563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872802cba0 con 0x7f8724003970 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f873c8f4640 1 -- 172.21.15.78:0/1316985563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8724003970 msgr2=0x7f8724003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f873c8f4640 1 --2- 172.21.15.78:0/1316985563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8724003970 0x7f8724003d90 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f8728009ac0 tx=0x7f87240057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f8736d9d640 1 -- 172.21.15.78:0/1316985563 reap_dead start 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f873c8f4640 1 -- 172.21.15.78:0/1316985563 shutdown_connections 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f873c8f4640 1 -- 172.21.15.78:0/1316985563 >> 172.21.15.78:0/1316985563 conn(0x7f873806b4b0 msgr2=0x7f87380c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f873c8f4640 1 -- 172.21.15.78:0/1316985563 shutdown_connections 2024-02-19T14:56:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.396+0000 7f873c8f4640 1 -- 172.21.15.78:0/1316985563 wait complete. 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.398+0000 7f873c8f4640 1 Processor -- start 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.398+0000 7f873c8f4640 1 -- start start 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f873c8f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8724003970 0x7f87380bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f873c8f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 0x7f87380a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f873c8f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f873814cbc0 0x7f873814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f873c8f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873814d5b0 con 0x7f87380c0190 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f873c8f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380a8760 con 0x7f873814cbc0 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f873c8f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380c0610 con 0x7f8724003970 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f872dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 0x7f87380a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f872dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 0x7f87380a8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47398/0 (socket says 172.21.15.78:47398) 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.399+0000 7f872dd9b640 1 -- 172.21.15.78:0/820015949 learned_addr learned my addr 172.21.15.78:0/820015949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f8736d9d640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f873814cbc0 0x7f873814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f873659c640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8724003970 0x7f87380bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872dd9b640 1 -- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8724003970 msgr2=0x7f87380bd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872dd9b640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8724003970 0x7f87380bd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872dd9b640 1 -- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f873814cbc0 msgr2=0x7f873814d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872dd9b640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f873814cbc0 0x7f873814d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872dd9b640 1 -- 172.21.15.78:0/820015949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380c5ac0 con 0x7f87380c0190 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f8736d9d640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f873814cbc0 0x7f873814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f873659c640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8724003970 0x7f87380bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872dd9b640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 0x7f87380a8f20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f8720001120 tx=0x7f8720007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872f7fe640 1 -- 172.21.15.78:0/820015949 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872000f780 con 0x7f87380c0190 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872f7fe640 1 -- 172.21.15.78:0/820015949 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872000f900 con 0x7f87380c0190 2024-02-19T14:56:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f872f7fe640 1 -- 172.21.15.78:0/820015949 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872000fcb0 con 0x7f87380c0190 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87380a9d00 con 0x7f87380c0190 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.400+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87380aa2c0 con 0x7f87380c0190 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.402+0000 7f872f7fe640 1 -- 172.21.15.78:0/820015949 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f872001d460 con 0x7f87380c0190 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.402+0000 7f872f7fe640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f871806b840 0x7f871806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.402+0000 7f872f7fe640 1 -- 172.21.15.78:0/820015949 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f872008a310 con 0x7f87380c0190 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.402+0000 7f873659c640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f871806b840 0x7f871806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.403+0000 7f873659c640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f871806b840 0x7f871806dd00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8728036000 tx=0x7f8728030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.403+0000 7f873f5a9740 1 Processor -- start 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.403+0000 7f873f5a9740 1 -- start start 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c499450 0x555c7c4d9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4d9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873f5a9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c458cd0 con 0x555c7c49dc20 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c40bf20 con 0x555c7c499450 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c7c409450 con 0x555c7c49eae0 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47400/0 (socket says 172.21.15.78:47400) 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 -- 172.21.15.78:0/2187332792 learned_addr learned my addr 172.21.15.78:0/2187332792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f873659c640 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c499450 0x555c7c4d9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f872dd9b640 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4d9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 -- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 msgr2=0x555c7c4d9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4d9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 -- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c499450 msgr2=0x555c7c4d9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c499450 0x555c7c4d9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 -- 172.21.15.78:0/2187332792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c7c3b9f20 con 0x555c7c49dc20 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.404+0000 7f8736d9d640 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fd0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f8738157340 tx=0x7f8730009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7b4709fb000a27 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f8717fff640 1 -- 172.21.15.78:0/2187332792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000e040 con 0x555c7c49dc20 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f8717fff640 1 -- 172.21.15.78:0/2187332792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873000f040 con 0x555c7c49dc20 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f8717fff640 1 -- 172.21.15.78:0/2187332792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730016a30 con 0x555c7c49dc20 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 -- 172.21.15.78:0/2187332792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 msgr2=0x555c7c497fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fd0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f8738157340 tx=0x7f8730009750 comp rx=0 tx=0).stop 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 -- 172.21.15.78:0/2187332792 shutdown_connections 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fd0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4d9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 --2- 172.21.15.78:0/2187332792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c499450 0x555c7c4d9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 -- 172.21.15.78:0/2187332792 >> 172.21.15.78:0/2187332792 conn(0x555c7c49a120 msgr2=0x555c7c4beb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 -- 172.21.15.78:0/2187332792 shutdown_connections 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.405+0000 7f873f5a9740 1 -- 172.21.15.78:0/2187332792 wait complete. 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.406+0000 7f873f5a9740 1 -- start start 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.406+0000 7f873f5a9740 1 Processor -- start 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c4c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c4edd50 0x555c7c4f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c404c30 con 0x555c7c49dc20 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c3a9d30 con 0x555c7c49eae0 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c7c0d8310 con 0x555c7c4edd50 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873f5a9740 10 client.0 _ll_register_callbacks cb 0x555c7c3b1d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c4c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c4c1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47402/0 (socket says 172.21.15.78:47402) 2024-02-19T14:56:28.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 -- 172.21.15.78:0/1627114737 learned_addr learned my addr 172.21.15.78:0/1627114737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f872dd9b640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f8736d9d640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c4edd50 0x555c7c4f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 -- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c4edd50 msgr2=0x555c7c4f1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c4edd50 0x555c7c4f1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 -- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 msgr2=0x555c7c4d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 -- 172.21.15.78:0/1627114737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8730009de0 con 0x555c7c49dc20 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f872dd9b640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f8736d9d640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c4edd50 0x555c7c4f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f873659c640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c4c1de0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f8728037040 tx=0x7f872802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715926bbe29922db server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.407+0000 7f87157fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f87157fa640 1 -- 172.21.15.78:0/1627114737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728003b60 con 0x555c7c49dc20 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f87157fa640 1 -- 172.21.15.78:0/1627114737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8728003cc0 con 0x555c7c49dc20 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c7c1046c0 con 0x555c7c49dc20 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f87157fa640 1 -- 172.21.15.78:0/1627114737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8728034440 con 0x555c7c49dc20 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 10 client.7959 fetch_fsmap learned FSMap version 100 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x555c7c4032b0 con 0x555c7c49dc20 2024-02-19T14:56:28.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f87157fa640 1 -- 172.21.15.78:0/1627114737 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555c7c1046c0 con 0x555c7c49dc20 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f87157fa640 1 -- 172.21.15.78:0/1627114737 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 100) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f87280345a0 con 0x555c7c49dc20 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 10 client.7959 fetch_fsmap finished waiting for FSMap version 100 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 10 client.7959 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 20 client.7959 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c4d40e0 0x555c7c4d44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 4 client.7959 mds_command: new command op to 7791 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 --> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c7c155270 con 0x555c7c4d40e0 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f872dd9b640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c4d40e0 0x555c7c4d44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f87157fa640 10 client.7959 ms_handle_connect on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.408+0000 7f872dd9b640 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c4d40e0 0x555c7c4d44c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60bebd6e1213b747 server_cookie=dd3419681a134519 in_seq=0 out_seq=0 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f87157fa640 1 -- 172.21.15.78:0/1627114737 <== mds.0 v2:172.21.15.78:6835/100250519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555c7c155270 con 0x555c7c4d40e0 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f87157fa640 10 client.7959 handle_command_reply: tid=0 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 client.7959 shutdown 2024-02-19T14:56:28.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 msgr2=0x555c7c4c1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c4c1de0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f8728037040 tx=0x7f872802e070 comp rx=0 tx=0).stop 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 shutdown_connections 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c4d40e0 0x555c7c4d44c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c4edd50 0x555c7c4f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f87157fa640 0 client.7959 ms_handle_reset on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 --2- 172.21.15.78:0/1627114737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c4c1de0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 >> 172.21.15.78:0/1627114737 conn(0x555c7c499450 msgr2=0x555c7c4dd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 shutdown_connections 2024-02-19T14:56:28.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.409+0000 7f873f5a9740 1 -- 172.21.15.78:0/1627114737 wait complete. 2024-02-19T14:56:28.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.410+0000 7f873f5a9740 20 client.7959 trim_cache size 0 max 16384 2024-02-19T14:56:28.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.426+0000 7f873f5a9740 1 Processor -- start 2024-02-19T14:56:28.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.426+0000 7f873f5a9740 1 -- start start 2024-02-19T14:56:28.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49dc20 0x555c7c497130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 0x555c7c4d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f873f5a9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c38a490 con 0x555c7c4edd50 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c4d57c0 con 0x555c7c49eae0 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c7c4dbb90 con 0x555c7c49dc20 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 0x555c7c4d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 0x555c7c4d4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47404/0 (socket says 172.21.15.78:47404) 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 -- 172.21.15.78:0/3934954101 learned_addr learned my addr 172.21.15.78:0/3934954101 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 -- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49dc20 msgr2=0x555c7c497130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49dc20 0x555c7c497130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 -- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 msgr2=0x555c7c4d44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f872dd9b640 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 -- 172.21.15.78:0/3934954101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c7c39fa20 con 0x555c7c4edd50 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f8736d9d640 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 0x555c7c4d4a00 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f873000b010 tx=0x7f8730010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb98a37f55cfa47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f87177fe640 1 -- 172.21.15.78:0/3934954101 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730024070 con 0x555c7c4edd50 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f87177fe640 1 -- 172.21.15.78:0/3934954101 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730009c70 con 0x555c7c4edd50 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.427+0000 7f87177fe640 1 -- 172.21.15.78:0/3934954101 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000e040 con 0x555c7c4edd50 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 -- 172.21.15.78:0/3934954101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 msgr2=0x555c7c4d4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 0x555c7c4d4a00 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f873000b010 tx=0x7f8730010000 comp rx=0 tx=0).stop 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 -- 172.21.15.78:0/3934954101 shutdown_connections 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c4edd50 0x555c7c4d4a00 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c49eae0 0x555c7c4d44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 --2- 172.21.15.78:0/3934954101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49dc20 0x555c7c497130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 -- 172.21.15.78:0/3934954101 >> 172.21.15.78:0/3934954101 conn(0x555c7c499450 msgr2=0x555c7c4d9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 -- 172.21.15.78:0/3934954101 shutdown_connections 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 -- 172.21.15.78:0/3934954101 wait complete. 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.428+0000 7f873f5a9740 1 -- start start 2024-02-19T14:56:28.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 Processor -- start 2024-02-19T14:56:28.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4f8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c4edd50 0x555c7c544490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c4c8300 con 0x555c7c49dc20 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c7c4bbf60 con 0x555c7c4edd50 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c7c4c7e40 con 0x555c7c49eae0 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.429+0000 7f873f5a9740 10 client.0 _ll_register_callbacks cb 0x7f8728037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47410/0 (socket says 172.21.15.78:47410) 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 -- 172.21.15.78:0/470486115 learned_addr learned my addr 172.21.15.78:0/470486115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f872dd9b640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f8736d9d640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c4edd50 0x555c7c544490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 -- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 msgr2=0x555c7c4f8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4f8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 -- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c4edd50 msgr2=0x555c7c544490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c4edd50 0x555c7c544490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 -- 172.21.15.78:0/470486115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c7c3b9f20 con 0x555c7c49dc20 2024-02-19T14:56:28.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f872dd9b640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f8736d9d640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c4edd50 0x555c7c544490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873659c640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fb0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f8728004130 tx=0x7f87280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed1037e9d3aa2d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f87157fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f87157fa640 1 -- 172.21.15.78:0/470486115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728034cb0 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f87157fa640 1 -- 172.21.15.78:0/470486115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8728034e10 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f873000f040 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f87157fa640 1 -- 172.21.15.78:0/470486115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87280039e0 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873f5a9740 10 client.7965 fetch_fsmap learned FSMap version 100 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x555c7c3c5780 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.430+0000 7f87157fa640 1 -- 172.21.15.78:0/470486115 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873000f040 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f87157fa640 1 -- 172.21.15.78:0/470486115 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 100) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8728003b40 con 0x555c7c49dc20 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 10 client.7965 fetch_fsmap finished waiting for FSMap version 100 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 10 client.7965 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 20 client.7965 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:28.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c5449d0 0x555c7c4d44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 4 client.7965 mds_command: new command op to 7791 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 --> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555c7c141610 con 0x555c7c5449d0 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f872dd9b640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c5449d0 0x555c7c4d44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f87157fa640 10 client.7965 ms_handle_connect on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f872dd9b640 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c5449d0 0x555c7c4d44c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25b06f5e2e34d5f server_cookie=26aa1e153a921c07 in_seq=0 out_seq=0 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f87157fa640 1 -- 172.21.15.78:0/470486115 <== mds.0 v2:172.21.15.78:6835/100250519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x555c7c141610 con 0x555c7c5449d0 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f87157fa640 10 client.7965 handle_command_reply: tid=0 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 client.7965 shutdown 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 msgr2=0x555c7c497fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fb0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f8728004130 tx=0x7f87280046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 shutdown_connections 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x555c7c5449d0 0x555c7c4d44c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555c7c4edd50 0x555c7c544490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f87157fa640 0 client.7965 ms_handle_reset on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x555c7c49eae0 0x555c7c4f8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 --2- 172.21.15.78:0/470486115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555c7c49dc20 0x555c7c497fb0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 >> 172.21.15.78:0/470486115 conn(0x555c7c499450 msgr2=0x555c7c4f1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 shutdown_connections 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.431+0000 7f873f5a9740 1 -- 172.21.15.78:0/470486115 wait complete. 2024-02-19T14:56:28.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873f5a9740 20 client.7965 trim_cache size 0 max 16384 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7791, 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 9.0885863049999998, 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 100, 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 124, 2024-02-19T14:56:28.442 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 124, 2024-02-19T14:56:28.443 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 14.235529392 2024-02-19T14:56:28.443 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:28.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f871806b840 msgr2=0x7f871806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f871806b840 0x7f871806dd00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8728036000 tx=0x7f8728030000 comp rx=0 tx=0).stop 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 msgr2=0x7f87380a8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 0x7f87380a8f20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f8720001120 tx=0x7f8720007430 comp rx=0 tx=0).stop 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 shutdown_connections 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f871806b840 0x7f871806dd00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87380c0190 0x7f87380a8f20 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f873814cbc0 0x7f873814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 --2- 172.21.15.78:0/820015949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8724003970 0x7f87380bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 >> 172.21.15.78:0/820015949 conn(0x7f873806b4b0 msgr2=0x7f8738152710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 shutdown_connections 2024-02-19T14:56:28.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.432+0000 7f873c8f4640 1 -- 172.21.15.78:0/820015949 wait complete. 2024-02-19T14:56:28.445 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.544+0000 7febd651e640 1 Processor -- start 2024-02-19T14:56:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.544+0000 7febd651e640 1 -- start start 2024-02-19T14:56:28.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd651e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febd0152700 0x7febd00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd651e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd651e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7febd00c8e00 0x7febd006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd651e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd006cd20 con 0x7febd00c8e00 2024-02-19T14:56:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd651e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd01551f0 con 0x7febd0152700 2024-02-19T14:56:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd651e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd0155490 con 0x7febd00c5730 2024-02-19T14:56:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7febd00c8e00 0x7febd006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7febd00c8e00 0x7febd006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47424/0 (socket says 172.21.15.78:47424) 2024-02-19T14:56:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 -- 172.21.15.78:0/457238601 learned_addr learned my addr 172.21.15.78:0/457238601 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febcffff640 1 --2- 172.21.15.78:0/457238601 >> v2:172.21.15.150:3300/0 conn(0x7febd0152700 0x7febd00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febcf7fe640 1 --2- 172.21.15.78:0/457238601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 -- 172.21.15.78:0/457238601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 msgr2=0x7febd006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 --2- 172.21.15.78:0/457238601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 -- 172.21.15.78:0/457238601 >> v2:172.21.15.150:3300/0 conn(0x7febd0152700 msgr2=0x7febd00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 --2- 172.21.15.78:0/457238601 >> v2:172.21.15.150:3300/0 conn(0x7febd0152700 0x7febd00c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.546+0000 7febd4abb640 1 -- 172.21.15.78:0/457238601 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd0155770 con 0x7febd00c8e00 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.547+0000 7febd4abb640 1 --2- 172.21.15.78:0/457238601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd006c7e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7febd008bad0 tx=0x7febc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abb8b54f283851d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.547+0000 7febceffd640 1 -- 172.21.15.78:0/457238601 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc802ca70 con 0x7febd00c8e00 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.547+0000 7febceffd640 1 -- 172.21.15.78:0/457238601 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc802cbd0 con 0x7febd00c8e00 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.547+0000 7febceffd640 1 -- 172.21.15.78:0/457238601 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc80343f0 con 0x7febd00c8e00 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 -- 172.21.15.78:0/457238601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 msgr2=0x7febd006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 --2- 172.21.15.78:0/457238601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd006c7e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7febd008bad0 tx=0x7febc802bed0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 -- 172.21.15.78:0/457238601 shutdown_connections 2024-02-19T14:56:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 --2- 172.21.15.78:0/457238601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd006c7e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 --2- 172.21.15.78:0/457238601 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 --2- 172.21.15.78:0/457238601 >> v2:172.21.15.150:3300/0 conn(0x7febd0152700 0x7febd00c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 -- 172.21.15.78:0/457238601 >> 172.21.15.78:0/457238601 conn(0x7febd00cac90 msgr2=0x7febd00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 -- 172.21.15.78:0/457238601 shutdown_connections 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.548+0000 7febd651e640 1 -- 172.21.15.78:0/457238601 wait complete. 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.549+0000 7febd651e640 1 Processor -- start 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.549+0000 7febd651e640 1 -- start start 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd00c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd01ed850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd0152700 0x7febd01edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd01ee870 con 0x7febd00c8e00 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd01496b0 con 0x7febd0152700 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd00abdd0 con 0x7febd00c5730 2024-02-19T14:56:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd01ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd01ed850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47426/0 (socket says 172.21.15.78:47426) 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 -- 172.21.15.78:0/1921953441 learned_addr learned my addr 172.21.15.78:0/1921953441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcffff640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd00c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd4abb640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd0152700 0x7febd01edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 -- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 msgr2=0x7febd00c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd00c0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 -- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd0152700 msgr2=0x7febd01edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd0152700 0x7febd01edd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 -- 172.21.15.78:0/1921953441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8009b30 con 0x7febd00c8e00 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd4abb640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd0152700 0x7febd01edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febcf7fe640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd01ed850 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7febc400e9b0 tx=0x7febc400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc4009670 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc400c9a0 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd0157040 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc400cd50 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.550+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd01ee2d0 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.551+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd0156e80 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.552+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7febc4013470 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.552+0000 7febccff9640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7febbc06b810 0x7febbc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.552+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7febc408a510 con 0x7febd00c8e00 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.552+0000 7febcffff640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7febbc06b810 0x7febbc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.553+0000 7febcffff640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7febbc06b810 0x7febbc06dcd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7febd01e8170 tx=0x7febc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.556+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7febc4054db0 con 0x7febd00c8e00 2024-02-19T14:56:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.842+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febd0074990 con 0x7febd00c8e00 2024-02-19T14:56:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.842+0000 7febccff9640 1 -- 172.21.15.78:0/1921953441 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4945 (secure 0 0 0) 0x7febc4058470 con 0x7febd00c8e00 2024-02-19T14:56:28.845 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:28.845 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7788,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:20.692270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7791":{"gid":7791,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/100250519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":100250519},{"type":"v1","addr":"172.21.15.78:6837","nonce":100250519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 100 2024-02-19T14:56:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7febbc06b810 msgr2=0x7febbc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7febbc06b810 0x7febbc06dcd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7febd01e8170 tx=0x7febc0008040 comp rx=0 tx=0).stop 2024-02-19T14:56:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 msgr2=0x7febd01ed850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd01ed850 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7febc400e9b0 tx=0x7febc400ee70 comp rx=0 tx=0).stop 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 shutdown_connections 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7febbc06b810 0x7febbc06dcd0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febd00c8e00 0x7febd01ed850 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd0152700 0x7febd01edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 --2- 172.21.15.78:0/1921953441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd00c5730 0x7febd00c0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 >> 172.21.15.78:0/1921953441 conn(0x7febd00cac90 msgr2=0x7febd014ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 shutdown_connections 2024-02-19T14:56:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.844+0000 7febd651e640 1 -- 172.21.15.78:0/1921953441 wait complete. 2024-02-19T14:56:28.858 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:18.671108+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7791": { "addr": "172.21.15.78:6837/100250519", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 100250519, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 100250519, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7791, "incarnation": 98, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-19T14:56:20.692270+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7791 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/4213660424", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 4213660424, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 4213660424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6239, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/1857207464", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1857207464, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1857207464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7036, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1870596299", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1870596299, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1870596299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7788, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:56:28.858 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:56:28.859 INFO:tasks.cephfs.test_failover:current = [{'gid': 7788, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/1870596299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1870596299}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1870596299}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 6239, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/4213660424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4213660424}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4213660424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7791, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/100250519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 100250519}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 100250519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7036, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1857207464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1857207464}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1857207464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}] 2024-02-19T14:56:28.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-02-19T14:56:28.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.963+0000 7fe0b7d5e640 1 Processor -- start 2024-02-19T14:56:28.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.964+0000 7fe0b7d5e640 1 -- start start 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.964+0000 7fe0b7d5e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe0b0152670 0x7fe0b0071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.964+0000 7fe0b7d5e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 0x7fe0b0155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.964+0000 7fe0b7d5e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b0071bf0 con 0x7fe0b00702d0 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.964+0000 7fe0b7d5e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b0158a20 con 0x7fe0b0152670 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.964+0000 7fe0b7d5e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b0155550 con 0x7fe0b0152a50 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 0x7fe0b0155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b5afa640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe0b0152670 0x7fe0b0071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 0x7fe0b0155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:56278/0 (socket says 172.21.15.78:56278) 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b62fb640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe0b00702d0 0x7fe0b0158eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:59256/0 (socket says 172.21.15.78:59256) 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 -- 172.21.15.78:0/1045873795 learned_addr learned my addr 172.21.15.78:0/1045873795 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b4af8640 1 -- 172.21.15.78:0/1045873795 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823597559 0 0) 0x7fe0b0071bf0 con 0x7fe0b00702d0 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b4af8640 1 -- 172.21.15.78:0/1045873795 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c003520 con 0x7fe0b00702d0 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 -- 172.21.15.78:0/1045873795 >> v2:172.21.15.150:3300/0 conn(0x7fe0b0152670 msgr2=0x7fe0b0071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 --2- 172.21.15.78:0/1045873795 >> v2:172.21.15.150:3300/0 conn(0x7fe0b0152670 0x7fe0b0071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 -- 172.21.15.78:0/1045873795 >> v1:172.21.15.150:6789/0 conn(0x7fe0b00702d0 legacy=0x7fe0b0158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b52f9640 1 -- 172.21.15.78:0/1045873795 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00c6780 con 0x7fe0b0152a50 2024-02-19T14:56:28.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.965+0000 7fe0b5afa640 1 --2- 172.21.15.78:0/1045873795 >> v2:172.21.15.150:3300/0 conn(0x7fe0b0152670 0x7fe0b0071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:28.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.966+0000 7fe0b52f9640 1 --2- 172.21.15.78:0/1045873795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 0x7fe0b0155010 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009870 tx=0x7fe0a002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58db48dbe5ad0556 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.966+0000 7fe0b4af8640 1 -- 172.21.15.78:0/1045873795 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a002f840 con 0x7fe0b0152a50 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.966+0000 7fe0b4af8640 1 -- 172.21.15.78:0/1045873795 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a002f9a0 con 0x7fe0b0152a50 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.966+0000 7fe0b4af8640 1 -- 172.21.15.78:0/1045873795 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a002fd30 con 0x7fe0b0152a50 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/1045873795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 msgr2=0x7fe0b0155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/1045873795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 0x7fe0b0155010 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009870 tx=0x7fe0a002ba20 comp rx=0 tx=0).stop 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/1045873795 shutdown_connections 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/1045873795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152a50 0x7fe0b0155010 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/1045873795 >> v2:172.21.15.150:3300/0 conn(0x7fe0b0152670 0x7fe0b0071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/1045873795 >> 172.21.15.78:0/1045873795 conn(0x7fe0b00c85e0 msgr2=0x7fe0b00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/1045873795 shutdown_connections 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.967+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/1045873795 wait complete. 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.968+0000 7fe0b7d5e640 1 Processor -- start 2024-02-19T14:56:28.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.968+0000 7fe0b7d5e640 1 -- start start 2024-02-19T14:56:28.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 0x7fe0b0145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152670 0x7fe0b01506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0152a50 0x7fe0b0157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b0146230 con 0x7fe0b00702d0 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b01eb570 con 0x7fe0b0152a50 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b01504d0 con 0x7fe0b0152670 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 0x7fe0b0145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 0x7fe0b0145cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47440/0 (socket says 172.21.15.78:47440) 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 -- 172.21.15.78:0/2770820649 learned_addr learned my addr 172.21.15.78:0/2770820649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b52f9640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152670 0x7fe0b01506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b62fb640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0152a50 0x7fe0b0157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 -- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152670 msgr2=0x7fe0b01506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152670 0x7fe0b01506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 -- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0152a50 msgr2=0x7fe0b0157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0152a50 0x7fe0b0157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 -- 172.21.15.78:0/2770820649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac009400 con 0x7fe0b00702d0 2024-02-19T14:56:28.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b5afa640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 0x7fe0b0145cf0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00ec10 tx=0x7fe0ac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac009760 con 0x7fe0b00702d0 2024-02-19T14:56:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a00094d0 con 0x7fe0b00702d0 2024-02-19T14:56:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac00cc30 con 0x7fe0b00702d0 2024-02-19T14:56:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.969+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac013470 con 0x7fe0b00702d0 2024-02-19T14:56:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.970+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b01eaf20 con 0x7fe0b00702d0 2024-02-19T14:56:28.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.971+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b00676a0 con 0x7fe0b00702d0 2024-02-19T14:56:28.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.971+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe0ac0135d0 con 0x7fe0b00702d0 2024-02-19T14:56:28.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.972+0000 7fe0a67fc640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe09406b810 0x7fe09406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:28.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.972+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0ac08a5b0 con 0x7fe0b00702d0 2024-02-19T14:56:28.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.972+0000 7fe0b52f9640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe09406b810 0x7fe09406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:28.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.972+0000 7fe0b52f9640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe09406b810 0x7fe09406dcd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0036000 tx=0x7fe0a0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:28.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:28.976+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe0ac054f00 con 0x7fe0b00702d0 2024-02-19T14:56:29.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.151+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fe0b0070120 con 0x7fe0b00702d0 2024-02-19T14:56:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.154+0000 7fe0a67fc640 1 -- 172.21.15.78:0/2770820649 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v42)=0 v42) v1 ==== 116+0+0 (secure 0 0 0) 0x7fe0ac0585c0 con 0x7fe0b00702d0 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe09406b810 msgr2=0x7fe09406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe09406b810 0x7fe09406dcd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0036000 tx=0x7fe0a0008030 comp rx=0 tx=0).stop 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 msgr2=0x7fe0b0145cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 0x7fe0b0145cf0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00ec10 tx=0x7fe0ac00c520 comp rx=0 tx=0).stop 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 shutdown_connections 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe09406b810 0x7fe09406dcd0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0152a50 0x7fe0b0157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0b0152670 0x7fe0b01506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 --2- 172.21.15.78:0/2770820649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0b00702d0 0x7fe0b0145cf0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 >> 172.21.15.78:0/2770820649 conn(0x7fe0b00c85e0 msgr2=0x7fe0b00a82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 shutdown_connections 2024-02-19T14:56:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.156+0000 7fe0b7d5e640 1 -- 172.21.15.78:0/2770820649 wait complete. 2024-02-19T14:56:29.173 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:29.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.273+0000 7fb15c44e640 1 Processor -- start 2024-02-19T14:56:29.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.273+0000 7fb15c44e640 1 -- start start 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15c44e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb1541537a0 0x7fb1540c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15c44e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 0x7fb154156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15c44e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1541563b0 con 0x7fb15414cd50 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15c44e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb154153cd0 con 0x7fb1541537a0 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15c44e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15406b6e0 con 0x7fb154156860 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15a1ea640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb15414cd50 0x7fb1540c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40772/0 (socket says 172.21.15.78:40772) 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.274+0000 7fb15a1ea640 1 -- 172.21.15.78:0/2389683888 learned_addr learned my addr 172.21.15.78:0/2389683888 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 0x7fb154156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1599e9640 1 --2- 172.21.15.78:0/2389683888 >> v2:172.21.15.150:3300/0 conn(0x7fb1541537a0 0x7fb1540c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936123612 0 0) 0x7fb1541563b0 con 0x7fb15414cd50 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb140003520 con 0x7fb15414cd50 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1499806058 0 0) 0x7fb140003520 con 0x7fb15414cd50 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 msgr2=0x7fb154156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 0x7fb154156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 >> v2:172.21.15.150:3300/0 conn(0x7fb1541537a0 msgr2=0x7fb1540c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 --2- 172.21.15.78:0/2389683888 >> v2:172.21.15.150:3300/0 conn(0x7fb1541537a0 0x7fb1540c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154157230 con 0x7fb15414cd50 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb150001670 con 0x7fb15414cd50 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1599e9640 1 --2- 172.21.15.78:0/2389683888 >> v2:172.21.15.150:3300/0 conn(0x7fb1541537a0 0x7fb1540c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 >> v1:172.21.15.78:6789/0 conn(0x7fb15414cd50 legacy=0x7fb1540c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb140003970 0x7fb140003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 0x7fb1400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb140005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154153cd0 con 0x7fb140004d00 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15406b6e0 con 0x7fb140003970 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14000ba00 con 0x7fb154156860 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb140005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb1599e9640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 0x7fb1400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a1ea640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb140003970 0x7fb140003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 -- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 msgr2=0x7fb1400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb154156860 0x7fb1400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 -- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb140003970 msgr2=0x7fb140003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb140003970 0x7fb140003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.275+0000 7fb15a9eb640 1 -- 172.21.15.78:0/2389683888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb140003520 con 0x7fb140004d00 2024-02-19T14:56:29.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15a9eb640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb140005180 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009fc0 tx=0x7fb1400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0daa841e350d99e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:29.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c02cc20 con 0x7fb140004d00 2024-02-19T14:56:29.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c00f040 con 0x7fb140004d00 2024-02-19T14:56:29.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb1591e8640 1 -- 172.21.15.78:0/2389683888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c034440 con 0x7fb140004d00 2024-02-19T14:56:29.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15c44e640 1 -- 172.21.15.78:0/2389683888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 msgr2=0x7fb140005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15c44e640 1 --2- 172.21.15.78:0/2389683888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb140005180 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009fc0 tx=0x7fb1400057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15a9eb640 1 -- 172.21.15.78:0/2389683888 reap_dead start 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15c44e640 1 -- 172.21.15.78:0/2389683888 shutdown_connections 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15c44e640 1 -- 172.21.15.78:0/2389683888 >> 172.21.15.78:0/2389683888 conn(0x7fb154066e70 msgr2=0x7fb1540a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15c44e640 1 -- 172.21.15.78:0/2389683888 shutdown_connections 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.277+0000 7fb15c44e640 1 -- 172.21.15.78:0/2389683888 wait complete. 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.278+0000 7fb15c44e640 1 Processor -- start 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 -- start start 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb1541eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb154145fc0 0x7fb154155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1541eaf00 0x7fb1541eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154146690 con 0x7fb140004d00 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541ebea0 con 0x7fb154145fc0 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15c44e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb154146bc0 con 0x7fb1541eaf00 2024-02-19T14:56:29.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15a1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb1541eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15a1ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb1541eb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47458/0 (socket says 172.21.15.78:47458) 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15a1ea640 1 -- 172.21.15.78:0/939936401 learned_addr learned my addr 172.21.15.78:0/939936401 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb15a9eb640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1541eaf00 0x7fb1541eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.279+0000 7fb1599e9640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb154145fc0 0x7fb154155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15a1ea640 1 -- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1541eaf00 msgr2=0x7fb1541eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15a1ea640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1541eaf00 0x7fb1541eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15a1ea640 1 -- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb154145fc0 msgr2=0x7fb154155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15a1ea640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb154145fc0 0x7fb154155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15a1ea640 1 -- 172.21.15.78:0/939936401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154152640 con 0x7fb140004d00 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15a1ea640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb1541eb960 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb150000f60 tx=0x7fb1500077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000f960 con 0x7fb140004d00 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15000fae0 con 0x7fb140004d00 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1540c6db0 con 0x7fb140004d00 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000fe90 con 0x7fb140004d00 2024-02-19T14:56:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.280+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1541ee580 con 0x7fb140004d00 2024-02-19T14:56:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.281+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb154074990 con 0x7fb140004d00 2024-02-19T14:56:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.281+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb15001d460 con 0x7fb140004d00 2024-02-19T14:56:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.282+0000 7fb14affd640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb13806b810 0x7fb13806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.282+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb15008a620 con 0x7fb140004d00 2024-02-19T14:56:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.282+0000 7fb1599e9640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb13806b810 0x7fb13806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.282+0000 7fb1599e9640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb13806b810 0x7fb13806dcd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb1540a99f0 tx=0x7fb144008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:29.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.286+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb150054eb0 con 0x7fb140004d00 2024-02-19T14:56:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.676+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb154152900 con 0x7fb140004d00 2024-02-19T14:56:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.677+0000 7fb14affd640 1 -- 172.21.15.78:0/939936401 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4945 (secure 0 0 0) 0x7fb150058570 con 0x7fb140004d00 2024-02-19T14:56:29.680 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:29.680 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7788,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:20.692270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7791":{"gid":7791,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/100250519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":100250519},{"type":"v1","addr":"172.21.15.78:6837","nonce":100250519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 100 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb13806b810 msgr2=0x7fb13806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb13806b810 0x7fb13806dcd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb1540a99f0 tx=0x7fb144008040 comp rx=0 tx=0).stop 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 msgr2=0x7fb1541eb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb1541eb960 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb150000f60 tx=0x7fb1500077a0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 shutdown_connections 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb13806b810 0x7fb13806dcd0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1541eaf00 0x7fb1541eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb154145fc0 0x7fb154155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 --2- 172.21.15.78:0/939936401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb140004d00 0x7fb1541eb960 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 >> 172.21.15.78:0/939936401 conn(0x7fb154066e70 msgr2=0x7fb15406f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:29.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 shutdown_connections 2024-02-19T14:56:29.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:29.679+0000 7fb15c44e640 1 -- 172.21.15.78:0/939936401 wait complete. 2024-02-19T14:56:30.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:20.692270+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7791': {'gid': 7791, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/100250519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 100250519}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 100250519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:30.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:56:30.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:56:30.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.195+0000 7fe147381640 1 Processor -- start 2024-02-19T14:56:30.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.195+0000 7fe147381640 1 -- start start 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe147381640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe140152a50 0x7fe140071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe147381640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe140071bf0 con 0x7fe140152670 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe147381640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe140158a20 con 0x7fe1400702d0 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe147381640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1400c6780 con 0x7fe140152a50 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe14511d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe140152670 0x7fe140155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40774/0 (socket says 172.21.15.78:40774) 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe14511d640 1 -- 172.21.15.78:0/2028744155 learned_addr learned my addr 172.21.15.78:0/2028744155 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe14491c640 1 --2- 172.21.15.78:0/2028744155 >> v2:172.21.15.150:3300/0 conn(0x7fe140152a50 0x7fe140071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324440735 0 0) 0x7fe140071bf0 con 0x7fe140152670 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe12c003520 con 0x7fe140152670 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567339068 0 0) 0x7fe12c003520 con 0x7fe140152670 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 >> v2:172.21.15.150:3300/0 conn(0x7fe140152a50 msgr2=0x7fe140071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 --2- 172.21.15.78:0/2028744155 >> v2:172.21.15.150:3300/0 conn(0x7fe140152a50 0x7fe140071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 >> v1:172.21.15.150:6789/0 conn(0x7fe1400702d0 legacy=0x7fe140158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140071630 con 0x7fe140152670 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe13c001670 con 0x7fe140152670 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 >> v1:172.21.15.78:6789/0 conn(0x7fe140152670 legacy=0x7fe140155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c003970 0x7fe12c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe12c0047c0 0x7fe12c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe12c005180 0x7fe12c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400c6780 con 0x7fe12c0047c0 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c00c030 con 0x7fe12c003970 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.196+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c00c1f0 con 0x7fe12c005180 2024-02-19T14:56:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14491c640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe12c0047c0 0x7fe12c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14511d640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c003970 0x7fe12c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14591e640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe12c005180 0x7fe12c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14491c640 1 -- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe12c005180 msgr2=0x7fe12c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14491c640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe12c005180 0x7fe12c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14491c640 1 -- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c003970 msgr2=0x7fe12c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14491c640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c003970 0x7fe12c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14491c640 1 -- 172.21.15.78:0/2028744155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c003520 con 0x7fe12c0047c0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.197+0000 7fe14591e640 1 -- 172.21.15.78:0/2028744155 reap_dead start 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe14491c640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe12c0047c0 0x7fe12c004c40 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe130009ac0 tx=0x7fe12c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ea22917b602251 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13002c6d0 con 0x7fe12c0047c0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13002c830 con 0x7fe12c0047c0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe137fff640 1 -- 172.21.15.78:0/2028744155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13002cbc0 con 0x7fe12c0047c0 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 -- 172.21.15.78:0/2028744155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe12c0047c0 msgr2=0x7fe12c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe12c0047c0 0x7fe12c004c40 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe130009ac0 tx=0x7fe12c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 -- 172.21.15.78:0/2028744155 shutdown_connections 2024-02-19T14:56:30.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 --2- 172.21.15.78:0/2028744155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe12c0047c0 0x7fe12c004c40 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 -- 172.21.15.78:0/2028744155 >> 172.21.15.78:0/2028744155 conn(0x7fe1400c85e0 msgr2=0x7fe1400c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:30.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 -- 172.21.15.78:0/2028744155 shutdown_connections 2024-02-19T14:56:30.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.198+0000 7fe147381640 1 -- 172.21.15.78:0/2028744155 wait complete. 2024-02-19T14:56:30.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.199+0000 7fe147381640 1 Processor -- start 2024-02-19T14:56:30.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 -- start start 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c0047c0 0x7fe14014a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe140149440 0x7fe14014b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 0x7fe1401e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140153440 con 0x7fe1401dfe10 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe140158300 con 0x7fe12c0047c0 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe147381640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe140152e80 con 0x7fe140149440 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe14591e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 0x7fe1401e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe14591e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 0x7fe1401e02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47480/0 (socket says 172.21.15.78:47480) 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.200+0000 7fe14591e640 1 -- 172.21.15.78:0/32203710 learned_addr learned my addr 172.21.15.78:0/32203710 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14511d640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c0047c0 0x7fe14014a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14591e640 1 -- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe140149440 msgr2=0x7fe14014b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14491c640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe140149440 0x7fe14014b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14591e640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe140149440 0x7fe14014b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14591e640 1 -- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c0047c0 msgr2=0x7fe14014a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14591e640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c0047c0 0x7fe14014a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14591e640 1 -- 172.21.15.78:0/32203710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140153160 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe14591e640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 0x7fe1401e02c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fe14006ee00 tx=0x7fe13800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe135ffb640 1 -- 172.21.15.78:0/32203710 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13800eed0 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe135ffb640 1 -- 172.21.15.78:0/32203710 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138003970 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe140156520 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe135ffb640 1 -- 172.21.15.78:0/32203710 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138003d00 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.201+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14014b810 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.203+0000 7fe135ffb640 1 -- 172.21.15.78:0/32203710 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe1380050d0 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.203+0000 7fe135ffb640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe12406b840 0x7fe12406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.203+0000 7fe14511d640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe12406b840 0x7fe12406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.203+0000 7fe135ffb640 1 -- 172.21.15.78:0/32203710 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe138089fa0 con 0x7fe1401dfe10 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe14511d640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe12406b840 0x7fe12406dd00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe13c007c60 tx=0x7fe13c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe149bfd740 1 Processor -- start 2024-02-19T14:56:30.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe149bfd740 1 -- start start 2024-02-19T14:56:30.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2e98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 0x56431a2e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.204+0000 7fe149bfd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a268cd0 con 0x56431a2aeae0 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a21bf20 con 0x56431a2e8740 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56431a219450 con 0x56431a2eb8b0 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 0x56431a2e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 0x56431a2e8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47496/0 (socket says 172.21.15.78:47496) 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 -- 172.21.15.78:0/380785955 learned_addr learned my addr 172.21.15.78:0/380785955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 -- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 msgr2=0x56431a2ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14511d640 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2e98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 -- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 msgr2=0x56431a2e98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2e98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 -- 172.21.15.78:0/380785955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431a1c9f20 con 0x56431a2aeae0 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe14491c640 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 0x56431a2e8b20 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe130037040 tx=0x7fe13002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedaaa6250695ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe12affd640 1 -- 172.21.15.78:0/380785955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe130004240 con 0x56431a2aeae0 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe12affd640 1 -- 172.21.15.78:0/380785955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1300043a0 con 0x56431a2aeae0 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.205+0000 7fe12affd640 1 -- 172.21.15.78:0/380785955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13002cbe0 con 0x56431a2aeae0 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 -- 172.21.15.78:0/380785955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 msgr2=0x56431a2e8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 0x56431a2e8b20 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe130037040 tx=0x7fe13002e000 comp rx=0 tx=0).stop 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 -- 172.21.15.78:0/380785955 shutdown_connections 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2aeae0 0x56431a2e8b20 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 --2- 172.21.15.78:0/380785955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2e98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 -- 172.21.15.78:0/380785955 >> 172.21.15.78:0/380785955 conn(0x56431a2aa120 msgr2=0x56431a2ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 -- 172.21.15.78:0/380785955 shutdown_connections 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 -- 172.21.15.78:0/380785955 wait complete. 2024-02-19T14:56:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.206+0000 7fe149bfd740 1 -- start start 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 Processor -- start 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2ed890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2e3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2c0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a214c30 con 0x56431a2cc040 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a1b9d30 con 0x56431a2e8740 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x564319ee8310 con 0x56431a2eb8b0 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe149bfd740 10 client.0 _ll_register_callbacks cb 0x56431a1c1d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.207+0000 7fe14591e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2c0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47498/0 (socket says 172.21.15.78:47498) 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 -- 172.21.15.78:0/3610964745 learned_addr learned my addr 172.21.15.78:0/3610964745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14491c640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14511d640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 -- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 msgr2=0x56431a2e3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2e3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 -- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 msgr2=0x56431a2ed890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2ed890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 -- 172.21.15.78:0/3610964745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c003520 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe14591e640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2c0870 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fe138060800 tx=0x7fe138058e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73536e520793983e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe11ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe11ffff640 1 -- 172.21.15.78:0/3610964745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138060d30 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe11ffff640 1 -- 172.21.15.78:0/3610964745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe138056920 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564319f146c0 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe11ffff640 1 -- 172.21.15.78:0/3610964745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe138054c90 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe149bfd740 10 client.7992 fetch_fsmap learned FSMap version 100 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x56431a2132b0 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe11ffff640 1 -- 172.21.15.78:0/3610964745 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564319f146c0 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe11ffff640 1 -- 172.21.15.78:0/3610964745 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 100) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe138057a70 con 0x56431a2cc040 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe149bfd740 10 client.7992 fetch_fsmap finished waiting for FSMap version 100 2024-02-19T14:56:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.208+0000 7fe149bfd740 10 client.7992 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe149bfd740 20 client.7992 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fcf80 0x56431a2fd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe149bfd740 4 client.7992 mds_command: new command op to 7791 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 --> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564319f65270 con 0x56431a2fcf80 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe14511d640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fcf80 0x56431a2fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe11ffff640 10 client.7992 ms_handle_connect on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.209+0000 7fe14511d640 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fcf80 0x56431a2fd360 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8e5aa32787edb8a server_cookie=acb465b0bc3174dc in_seq=0 out_seq=0 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe11ffff640 1 -- 172.21.15.78:0/3610964745 <== mds.0 v2:172.21.15.78:6835/100250519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564319f65270 con 0x56431a2fcf80 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe11ffff640 10 client.7992 handle_command_reply: tid=0 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 client.7992 shutdown 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 msgr2=0x56431a2c0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2c0870 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fe138060800 tx=0x7fe138058e60 comp rx=0 tx=0).stop 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 shutdown_connections 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fcf80 0x56431a2fd360 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2c0870 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2eb8b0 0x56431a2e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3610964745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2e8740 0x56431a2ed890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 >> 172.21.15.78:0/3610964745 conn(0x56431a2aeae0 msgr2=0x56431a2ecd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe11ffff640 0 client.7992 ms_handle_reset on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 shutdown_connections 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 1 -- 172.21.15.78:0/3610964745 wait complete. 2024-02-19T14:56:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.210+0000 7fe149bfd740 20 client.7992 trim_cache size 0 max 16384 2024-02-19T14:56:30.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.226+0000 7fe149bfd740 1 Processor -- start 2024-02-19T14:56:30.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.226+0000 7fe149bfd740 1 -- start start 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a2cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2eb8b0 0x56431a2ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe149bfd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a19a490 con 0x56431a2cc040 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a2edf20 con 0x56431a2eb8b0 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56431a303770 con 0x56431a2e8740 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47502/0 (socket says 172.21.15.78:47502) 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 -- 172.21.15.78:0/3346229883 learned_addr learned my addr 172.21.15.78:0/3346229883 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14491c640 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a2cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14591e640 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2eb8b0 0x56431a2ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 -- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 msgr2=0x56431a2cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a2cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 -- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2eb8b0 msgr2=0x56431a2ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.227+0000 7fe14511d640 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2eb8b0 0x56431a2ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe14511d640 1 -- 172.21.15.78:0/3346229883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431a1afa20 con 0x56431a2cc040 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe14591e640 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2eb8b0 0x56431a2ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe14511d640 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2cec60 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fe13c007430 tx=0x7fe13c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a303d2bc13f8f3c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe12a7fc640 1 -- 172.21.15.78:0/3346229883 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13c007720 con 0x56431a2cc040 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe12a7fc640 1 -- 172.21.15.78:0/3346229883 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13c027d30 con 0x56431a2cc040 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe12a7fc640 1 -- 172.21.15.78:0/3346229883 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13c027ea0 con 0x56431a2cc040 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 -- 172.21.15.78:0/3346229883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 msgr2=0x56431a2cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2cec60 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fe13c007430 tx=0x7fe13c016000 comp rx=0 tx=0).stop 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 -- 172.21.15.78:0/3346229883 shutdown_connections 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2eb8b0 0x56431a2ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a2cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3346229883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2cc040 0x56431a2cec60 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 -- 172.21.15.78:0/3346229883 >> 172.21.15.78:0/3346229883 conn(0x56431a2aeae0 msgr2=0x56431a2e2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:30.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 -- 172.21.15.78:0/3346229883 shutdown_connections 2024-02-19T14:56:30.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.228+0000 7fe149bfd740 1 -- 172.21.15.78:0/3346229883 wait complete. 2024-02-19T14:56:30.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.229+0000 7fe149bfd740 1 -- start start 2024-02-19T14:56:30.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.229+0000 7fe149bfd740 1 Processor -- start 2024-02-19T14:56:30.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2cc040 0x56431a2ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a35d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 0x56431a353120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a2e9910 con 0x56431a2eb8b0 2024-02-19T14:56:30.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431a2e0a70 con 0x56431a2cc040 2024-02-19T14:56:30.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56431a2e0ed0 con 0x56431a2e8740 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 10 client.0 _ll_register_callbacks cb 0x7fe1380548c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 0x56431a353120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 0x56431a353120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47516/0 (socket says 172.21.15.78:47516) 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 -- 172.21.15.78:0/3325987268 learned_addr learned my addr 172.21.15.78:0/3325987268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14511d640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2cc040 0x56431a2ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14491c640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a35d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 -- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 msgr2=0x56431a35d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a35d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 -- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2cc040 msgr2=0x56431a2ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2cc040 0x56431a2ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 -- 172.21.15.78:0/3325987268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13c01a040 con 0x56431a2eb8b0 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14511d640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2cc040 0x56431a2ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:30.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe14591e640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 0x56431a353120 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe138058320 tx=0x7fe1380014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125d942db1e07224 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe11ffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe11ffff640 1 -- 172.21.15.78:0/3325987268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138003e60 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe11ffff640 1 -- 172.21.15.78:0/3325987268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe138060d30 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1300043a0 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe11ffff640 1 -- 172.21.15.78:0/3325987268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe138057a70 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 10 client.7998 fetch_fsmap learned FSMap version 100 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.231+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x56431a1d5780 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe11ffff640 1 -- 172.21.15.78:0/3325987268 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1300043a0 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe11ffff640 1 -- 172.21.15.78:0/3325987268 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 100) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe138054370 con 0x56431a2eb8b0 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 10 client.7998 fetch_fsmap finished waiting for FSMap version 100 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 10 client.7998 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 20 client.7998 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:30.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fd840 0x56431a2fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 4 client.7998 mds_command: new command op to 7791 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 --> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564319f51610 con 0x56431a2fd840 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe14511d640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fd840 0x56431a2fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe11ffff640 10 client.7998 ms_handle_connect on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe14511d640 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fd840 0x56431a2fdc20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a9702edf626ffb9 server_cookie=7fc61b65cf4ae8cb in_seq=0 out_seq=0 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe11ffff640 1 -- 172.21.15.78:0/3325987268 <== mds.0 v2:172.21.15.78:6835/100250519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x564319f51610 con 0x56431a2fd840 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe11ffff640 10 client.7998 handle_command_reply: tid=0 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.232+0000 7fe149bfd740 1 client.7998 shutdown 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 msgr2=0x56431a353120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 0x56431a353120 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe138058320 tx=0x7fe1380014a0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 shutdown_connections 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56431a2fd840 0x56431a2fdc20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56431a2eb8b0 0x56431a353120 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56431a2e8740 0x56431a35d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe11ffff640 0 client.7998 ms_handle_reset on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 --2- 172.21.15.78:0/3325987268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56431a2cc040 0x56431a2ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 >> 172.21.15.78:0/3325987268 conn(0x56431a2aeae0 msgr2=0x56431a3039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 shutdown_connections 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 1 -- 172.21.15.78:0/3325987268 wait complete. 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe149bfd740 20 client.7998 trim_cache size 0 max 16384 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7791, 2024-02-19T14:56:30.242 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 10.889856201000001, 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 100, 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 124, 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 124, 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 16.036799219999999 2024-02-19T14:56:30.243 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe12406b840 msgr2=0x7fe12406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe12406b840 0x7fe12406dd00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe13c007c60 tx=0x7fe13c009040 comp rx=0 tx=0).stop 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 msgr2=0x7fe1401e02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 0x7fe1401e02c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fe14006ee00 tx=0x7fe13800eb90 comp rx=0 tx=0).stop 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 shutdown_connections 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe12406b840 0x7fe12406dd00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1401dfe10 0x7fe1401e02c0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe140149440 0x7fe14014b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 --2- 172.21.15.78:0/32203710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe12c0047c0 0x7fe14014a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:30.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.233+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 >> 172.21.15.78:0/32203710 conn(0x7fe1400c85e0 msgr2=0x7fe14014e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:30.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.234+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 shutdown_connections 2024-02-19T14:56:30.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:30.234+0000 7fe147381640 1 -- 172.21.15.78:0/32203710 wait complete. 2024-02-19T14:56:30.250 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:18.671108+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7791": { "addr": "172.21.15.78:6837/100250519", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 100250519, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 100250519, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7791, "incarnation": 98, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-19T14:56:20.692270+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7791 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/4213660424", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 4213660424, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 4213660424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6239, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/1857207464", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1857207464, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1857207464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7036, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1870596299", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1870596299, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1870596299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7788, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:56:30.251 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 9, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:56:30.251 INFO:tasks.cephfs.test_failover:current = [{'gid': 7788, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/1870596299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1870596299}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1870596299}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 6239, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/4213660424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4213660424}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4213660424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7791, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/100250519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 100250519}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 100250519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7036, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1857207464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1857207464}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1857207464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}] 2024-02-19T14:56:30.251 DEBUG:tasks.cephfs.test_failover:9 != -1 2024-02-19T14:56:30.252 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:56:35.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:35.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.353+0000 7fcef0f8a640 1 Processor -- start 2024-02-19T14:56:35.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.353+0000 7fcef0f8a640 1 -- start start 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcef0f8a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 0x7fceec0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcef0f8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec0c8e00 0x7fceec15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcef0f8a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceec1551f0 con 0x7fceec0c5730 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcef0f8a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceec155490 con 0x7fceec152700 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcef0f8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec155770 con 0x7fceec0c8e00 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fceead9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec0c8e00 0x7fceec15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fceea59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 0x7fceec0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcee9d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fceec0c5730 0x7fceec06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:59274/0 (socket says 172.21.15.78:59274) 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fceea59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 0x7fceec0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:50940/0 (socket says 172.21.15.78:50940) 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.354+0000 7fcee9d9b640 1 -- 172.21.15.78:0/538853521 learned_addr learned my addr 172.21.15.78:0/538853521 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.355+0000 7fceead9d640 1 -- 172.21.15.78:0/538853521 >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 msgr2=0x7fceec0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.355+0000 7fceead9d640 1 --2- 172.21.15.78:0/538853521 >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 0x7fceec0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.355+0000 7fceead9d640 1 -- 172.21.15.78:0/538853521 >> v1:172.21.15.150:6789/0 conn(0x7fceec0c5730 legacy=0x7fceec06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.355+0000 7fceead9d640 1 -- 172.21.15.78:0/538853521 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec155a50 con 0x7fceec0c8e00 2024-02-19T14:56:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.355+0000 7fceea59c640 1 --2- 172.21.15.78:0/538853521 >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 0x7fceec0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.356+0000 7fceead9d640 1 --2- 172.21.15.78:0/538853521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec0c8e00 0x7fceec15f980 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009d90 tx=0x7fcee402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67f15c8c8e5f615 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.356+0000 7fcee959a640 1 -- 172.21.15.78:0/538853521 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee402cc50 con 0x7fceec0c8e00 2024-02-19T14:56:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.356+0000 7fcee959a640 1 -- 172.21.15.78:0/538853521 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee402cdb0 con 0x7fceec0c8e00 2024-02-19T14:56:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.356+0000 7fcee959a640 1 -- 172.21.15.78:0/538853521 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4034660 con 0x7fceec0c8e00 2024-02-19T14:56:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 -- 172.21.15.78:0/538853521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec0c8e00 msgr2=0x7fceec15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/538853521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec0c8e00 0x7fceec15f980 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009d90 tx=0x7fcee402bea0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 -- 172.21.15.78:0/538853521 shutdown_connections 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/538853521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec0c8e00 0x7fceec15f980 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/538853521 >> v2:172.21.15.150:3300/0 conn(0x7fceec152700 0x7fceec0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 -- 172.21.15.78:0/538853521 >> 172.21.15.78:0/538853521 conn(0x7fceec0cac90 msgr2=0x7fceec0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 -- 172.21.15.78:0/538853521 shutdown_connections 2024-02-19T14:56:35.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.357+0000 7fcef0f8a640 1 -- 172.21.15.78:0/538853521 wait complete. 2024-02-19T14:56:35.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.359+0000 7fcef0f8a640 1 Processor -- start 2024-02-19T14:56:35.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.359+0000 7fcef0f8a640 1 -- start start 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcef0f8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0c5730 0x7fceec0c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcef0f8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 0x7fceec1eb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcef0f8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec152700 0x7fceec1ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcef0f8a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec1eac70 con 0x7fceec0c8e00 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcef0f8a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec1eb640 con 0x7fceec0c5730 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcef0f8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec1eeb80 con 0x7fceec152700 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 0x7fceec1eb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 0x7fceec1eb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47528/0 (socket says 172.21.15.78:47528) 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 -- 172.21.15.78:0/3623783833 learned_addr learned my addr 172.21.15.78:0/3623783833 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fceea59c640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0c5730 0x7fceec0c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fceead9d640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec152700 0x7fceec1ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:35.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 -- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec152700 msgr2=0x7fceec1ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec152700 0x7fceec1ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 -- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0c5730 msgr2=0x7fceec0c08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0c5730 0x7fceec0c08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 -- 172.21.15.78:0/3623783833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc009400 con 0x7fceec0c8e00 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcee9d9b640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 0x7fceec1eb100 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c400 tx=0x7fcee000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.360+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee0005ce0 con 0x7fceec0c8e00 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.361+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee0005e40 con 0x7fceec0c8e00 2024-02-19T14:56:35.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.361+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee00072d0 con 0x7fceec0c8e00 2024-02-19T14:56:35.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.361+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee4009a70 con 0x7fceec0c8e00 2024-02-19T14:56:35.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.361+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceec0c0ea0 con 0x7fceec0c8e00 2024-02-19T14:56:35.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.362+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceec074990 con 0x7fceec0c8e00 2024-02-19T14:56:35.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.362+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcee00074f0 con 0x7fceec0c8e00 2024-02-19T14:56:35.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.362+0000 7fcedaffd640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcecc06b810 0x7fcecc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:35.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.363+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcee0089840 con 0x7fceec0c8e00 2024-02-19T14:56:35.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.363+0000 7fceea59c640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcecc06b810 0x7fcecc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:35.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.363+0000 7fceea59c640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcecc06b810 0x7fcecc06dcd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009c30 tx=0x7fcedc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:35.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.367+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcee0054210 con 0x7fceec0c8e00 2024-02-19T14:56:35.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.595+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceec0c8770 con 0x7fceec0c8e00 2024-02-19T14:56:35.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.595+0000 7fcedaffd640 1 -- 172.21.15.78:0/3623783833 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4944 (secure 0 0 0) 0x7fcee00578d0 con 0x7fceec0c8e00 2024-02-19T14:56:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:35.599 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7788,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:31.368255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7791":{"gid":7791,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6837/100250519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":100250519},{"type":"v1","addr":"172.21.15.78:6837","nonce":100250519}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:35.601 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 101 2024-02-19T14:56:35.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcecc06b810 msgr2=0x7fcecc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:35.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcecc06b810 0x7fcecc06dcd0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009c30 tx=0x7fcedc009190 comp rx=0 tx=0).stop 2024-02-19T14:56:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 msgr2=0x7fceec1eb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 0x7fceec1eb100 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c400 tx=0x7fcee000c8c0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 shutdown_connections 2024-02-19T14:56:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcecc06b810 0x7fcecc06dcd0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fceec0c8e00 0x7fceec1eb100 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceec152700 0x7fceec1ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 --2- 172.21.15.78:0/3623783833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceec0c5730 0x7fceec0c08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.597+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 >> 172.21.15.78:0/3623783833 conn(0x7fceec0cac90 msgr2=0x7fceec0c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.598+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 shutdown_connections 2024-02-19T14:56:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:35.598+0000 7fcef0f8a640 1 -- 172.21.15.78:0/3623783833 wait complete. 2024-02-19T14:56:36.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:31.368255+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7791': {'gid': 7791, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.78:6837/100250519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 100250519}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 100250519}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:36.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:56:36.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5ad8ed640 1 Processor -- start 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5ad8ed640 1 -- start start 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5ad8ed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb5a80c30a0 0x7fb5a8158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5ad8ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a80c3cf0 con 0x7fb5a80c5130 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5ad8ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a80c59a0 con 0x7fb5a80c5510 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5ad8ed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a8158b60 con 0x7fb5a80c30a0 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5a6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb5a80c5130 0x7fb5a80c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40776/0 (socket says 172.21.15.78:40776) 2024-02-19T14:56:36.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.121+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/638749707 learned_addr learned my addr 172.21.15.78:0/638749707 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554727564 0 0) 0x7fb5a80c3cf0 con 0x7fb5a80c5130 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb594003520 con 0x7fb5a80c5130 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb59ffff640 1 --2- 172.21.15.78:0/638749707 >> v2:172.21.15.150:3300/0 conn(0x7fb5a80c30a0 0x7fb5a8158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3239961859 0 0) 0x7fb594003520 con 0x7fb5a80c5130 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 >> v2:172.21.15.150:3300/0 conn(0x7fb5a80c30a0 msgr2=0x7fb5a8158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 --2- 172.21.15.78:0/638749707 >> v2:172.21.15.150:3300/0 conn(0x7fb5a80c30a0 0x7fb5a8158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 >> v1:172.21.15.150:6789/0 conn(0x7fb5a80c5510 legacy=0x7fb5a815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a8158530 con 0x7fb5a80c5130 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb598001670 con 0x7fb5a80c5130 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb59ffff640 1 --2- 172.21.15.78:0/638749707 >> v2:172.21.15.150:3300/0 conn(0x7fb5a80c30a0 0x7fb5a8158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 >> v1:172.21.15.78:6789/0 conn(0x7fb5a80c5130 legacy=0x7fb5a80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb594003970 0x7fb594003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940047c0 0x7fb594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594005180 0x7fb59400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8158b60 con 0x7fb594003970 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59400c0d0 con 0x7fb5940047c0 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59400c290 con 0x7fb594005180 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb594003970 0x7fb594003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb59ffff640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940047c0 0x7fb594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594005180 0x7fb59400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594005180 msgr2=0x7fb59400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594005180 0x7fb59400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940047c0 msgr2=0x7fb594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940047c0 0x7fb594004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a77fe640 1 -- 172.21.15.78:0/638749707 reap_dead start 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.122+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/638749707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb594003520 con 0x7fb594003970 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.123+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb594003970 0x7fb594003d90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c210 tx=0x7fb5940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569d77a09dbd1556 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.123+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598005ad0 con 0x7fb594003970 2024-02-19T14:56:36.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.123+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb598005c30 con 0x7fb594003970 2024-02-19T14:56:36.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.123+0000 7fb5a67fc640 1 -- 172.21.15.78:0/638749707 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59802ca90 con 0x7fb594003970 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/638749707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb594003970 msgr2=0x7fb594003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb594003970 0x7fb594003d90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c210 tx=0x7fb5940057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/638749707 shutdown_connections 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/638749707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb594003970 0x7fb594003d90 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/638749707 >> 172.21.15.78:0/638749707 conn(0x7fb5a806b4b0 msgr2=0x7fb5a80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/638749707 shutdown_connections 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.124+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/638749707 wait complete. 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.125+0000 7fb5ad8ed640 1 Processor -- start 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.125+0000 7fb5ad8ed640 1 -- start start 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5ad8ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594003970 0x7fb5a8154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5ad8ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a814e640 0x7fb5a81eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5ad8ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 0x7fb5a814c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5ad8ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a81eb7f0 con 0x7fb5a81eb340 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5ad8ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8152a80 con 0x7fb5a814e640 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5ad8ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a81521e0 con 0x7fb594003970 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 0x7fb5a814c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 0x7fb5a814c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47550/0 (socket says 172.21.15.78:47550) 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 -- 172.21.15.78:0/3491993404 learned_addr learned my addr 172.21.15.78:0/3491993404 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594003970 0x7fb5a8154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb59ffff640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a814e640 0x7fb5a81eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 -- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594003970 msgr2=0x7fb5a8154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594003970 0x7fb5a8154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 -- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a814e640 msgr2=0x7fb5a81eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a814e640 0x7fb5a81eae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 -- 172.21.15.78:0/3491993404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a81524c0 con 0x7fb5a81eb340 2024-02-19T14:56:36.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594003970 0x7fb5a8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.126+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 0x7fb5a814c060 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80c58f0 tx=0x7fb5a000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.127+0000 7fb5a4ff9640 1 -- 172.21.15.78:0/3491993404 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0003b80 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.127+0000 7fb5a4ff9640 1 -- 172.21.15.78:0/3491993404 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a0003d00 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.127+0000 7fb5a4ff9640 1 -- 172.21.15.78:0/3491993404 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0005310 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.127+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a81527a0 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.127+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a8154ac0 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.128+0000 7fb5a4ff9640 1 -- 172.21.15.78:0/3491993404 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb5a0005470 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.129+0000 7fb5a4ff9640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb58806b810 0x7fb58806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.129+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb58806b810 0x7fb58806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.129+0000 7fb5a4ff9640 1 -- 172.21.15.78:0/3491993404 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb5a008a0b0 con 0x7fb5a81eb340 2024-02-19T14:56:36.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.129+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb58806b810 0x7fb58806dcd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb598034000 tx=0x7fb598007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.130+0000 7fb5b05a4740 1 Processor -- start 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.130+0000 7fb5b05a4740 1 -- start start 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x56483390a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648338cfae0 0x564833909b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x5648338ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564833889cd0 con 0x56483390c8b0 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56483383cf20 con 0x5648338cfae0 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56483383a450 con 0x564833909740 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x5648338ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x5648338ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47556/0 (socket says 172.21.15.78:47556) 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 -- 172.21.15.78:0/1576679034 learned_addr learned my addr 172.21.15.78:0/1576679034 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x56483390a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb59ffff640 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648338cfae0 0x564833909b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 -- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 msgr2=0x56483390a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x56483390a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 -- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648338cfae0 msgr2=0x564833909b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648338cfae0 0x564833909b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 -- 172.21.15.78:0/1576679034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648337eaf20 con 0x56483390c8b0 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x56483390a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x5648338ce860 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0060880 tx=0x7fb5a0058000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457f58ec4151c61a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb59d7fa640 1 -- 172.21.15.78:0/1576679034 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00586a0 con 0x56483390c8b0 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb59d7fa640 1 -- 172.21.15.78:0/1576679034 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a005c7f0 con 0x56483390c8b0 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.131+0000 7fb59d7fa640 1 -- 172.21.15.78:0/1576679034 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0060db0 con 0x56483390c8b0 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 -- 172.21.15.78:0/1576679034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 msgr2=0x5648338ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x5648338ce860 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0060880 tx=0x7fb5a0058000 comp rx=0 tx=0).stop 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 -- 172.21.15.78:0/1576679034 shutdown_connections 2024-02-19T14:56:36.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x5648338ce860 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5648338cfae0 0x564833909b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/1576679034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x56483390a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 -- 172.21.15.78:0/1576679034 >> 172.21.15.78:0/1576679034 conn(0x5648338cb120 msgr2=0x5648338efb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 -- 172.21.15.78:0/1576679034 shutdown_connections 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.132+0000 7fb5b05a4740 1 -- 172.21.15.78:0/1576679034 wait complete. 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 -- start start 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 Processor -- start 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x5648338ef310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x56483391b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833901a70 0x5648338eba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564833835c30 con 0x56483390c8b0 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648337dad30 con 0x564833909740 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x564833509310 con 0x564833901a70 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb5b05a4740 10 client.0 _ll_register_callbacks cb 0x5648337e2d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb59ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x56483391b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb59ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x56483391b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47570/0 (socket says 172.21.15.78:47570) 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.133+0000 7fb59ffff640 1 -- 172.21.15.78:0/4139640948 learned_addr learned my addr 172.21.15.78:0/4139640948 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833901a70 0x5648338eba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x5648338ef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb59ffff640 1 -- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833901a70 msgr2=0x5648338eba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb59ffff640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833901a70 0x5648338eba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb59ffff640 1 -- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 msgr2=0x5648338ef310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb59ffff640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x5648338ef310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb59ffff640 1 -- 172.21.15.78:0/4139640948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a8154ac0 con 0x56483390c8b0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x5648338ef310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb59ffff640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x56483391b350 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb590006fd0 tx=0x7fb590015730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a99ec3054666af server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5867fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5867fc640 1 -- 172.21.15.78:0/4139640948 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590004220 con 0x56483390c8b0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5867fc640 1 -- 172.21.15.78:0/4139640948 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb590015e80 con 0x56483390c8b0 2024-02-19T14:56:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648335356c0 con 0x56483390c8b0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5867fc640 1 -- 172.21.15.78:0/4139640948 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb590004380 con 0x56483390c8b0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5b05a4740 10 client.8013 fetch_fsmap learned FSMap version 101 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x5648338342b0 con 0x56483390c8b0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5867fc640 1 -- 172.21.15.78:0/4139640948 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5648335356c0 con 0x56483390c8b0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5867fc640 1 -- 172.21.15.78:0/4139640948 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 101) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb590003610 con 0x56483390c8b0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5b05a4740 10 client.8013 fetch_fsmap finished waiting for FSMap version 101 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.134+0000 7fb5b05a4740 10 client.8013 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5b05a4740 20 client.8013 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56483391e180 0x56483391e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5b05a4740 4 client.8013 mds_command: new command op to 7791 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 --> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564833586270 con 0x56483391e180 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56483391e180 0x56483391e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5867fc640 10 client.8013 ms_handle_connect on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56483391e180 0x56483391e560 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee777a8912414d58 server_cookie=d196dfe2ca314818 in_seq=0 out_seq=0 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5867fc640 1 -- 172.21.15.78:0/4139640948 <== mds.0 v2:172.21.15.78:6835/100250519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564833586270 con 0x56483391e180 2024-02-19T14:56:36.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.135+0000 7fb5867fc640 10 client.8013 handle_command_reply: tid=0 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 client.8013 shutdown 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 msgr2=0x56483391b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x56483391b350 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fb590006fd0 tx=0x7fb590015730 comp rx=0 tx=0).stop 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 shutdown_connections 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x56483391e180 0x56483391e560 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56483390c8b0 0x56483391b350 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833901a70 0x5648338eba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5867fc640 0 client.8013 ms_handle_reset on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/4139640948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x5648338ef310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 >> 172.21.15.78:0/4139640948 conn(0x5648338cfae0 msgr2=0x5648338c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 shutdown_connections 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 1 -- 172.21.15.78:0/4139640948 wait complete. 2024-02-19T14:56:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.136+0000 7fb5b05a4740 20 client.8013 trim_cache size 0 max 16384 2024-02-19T14:56:36.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.152+0000 7fb5b05a4740 1 Processor -- start 2024-02-19T14:56:36.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.152+0000 7fb5b05a4740 1 -- start start 2024-02-19T14:56:36.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x564833902280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56483390c8b0 0x5648339027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648337bb490 con 0x564833901a70 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56483391aed0 con 0x564833909740 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56483390c120 con 0x56483390c8b0 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47580/0 (socket says 172.21.15.78:47580) 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.153+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/420064209 learned_addr learned my addr 172.21.15.78:0/420064209 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56483390c8b0 msgr2=0x5648339027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56483390c8b0 0x5648339027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb59ffff640 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x564833902280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56483390c8b0 0x5648339027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 msgr2=0x564833902280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x564833902280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/420064209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648337d0a20 con 0x564833901a70 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390ae40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c330 tx=0x7fb59802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75430051ae89ce3b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb59cff9640 1 -- 172.21.15.78:0/420064209 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598049040 con 0x564833901a70 2024-02-19T14:56:36.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb59cff9640 1 -- 172.21.15.78:0/420064209 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb598007270 con 0x564833901a70 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb59cff9640 1 -- 172.21.15.78:0/420064209 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59804e4b0 con 0x564833901a70 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5b05a4740 1 -- 172.21.15.78:0/420064209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 msgr2=0x56483390ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.154+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390ae40 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c330 tx=0x7fb59802e000 comp rx=0 tx=0).stop 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 -- 172.21.15.78:0/420064209 shutdown_connections 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56483390c8b0 0x5648339027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564833909740 0x564833902280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/420064209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390ae40 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 -- 172.21.15.78:0/420064209 >> 172.21.15.78:0/420064209 conn(0x5648338cfae0 msgr2=0x564833905bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 -- 172.21.15.78:0/420064209 shutdown_connections 2024-02-19T14:56:36.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 -- 172.21.15.78:0/420064209 wait complete. 2024-02-19T14:56:36.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 -- start start 2024-02-19T14:56:36.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.155+0000 7fb5b05a4740 1 Processor -- start 2024-02-19T14:56:36.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x5648338ed390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56483390c8b0 0x564833974140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648338ed090 con 0x564833901a70 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564833901190 con 0x56483390c8b0 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648338ecea0 con 0x564833909740 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5b05a4740 10 client.0 _ll_register_callbacks cb 0x7fb590009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47590/0 (socket says 172.21.15.78:47590) 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2493609891 learned_addr learned my addr 172.21.15.78:0/2493609891 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb59ffff640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x5648338ed390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a77fe640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56483390c8b0 0x564833974140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 msgr2=0x5648338ed390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x5648338ed390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56483390c8b0 msgr2=0x564833974140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56483390c8b0 0x564833974140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.156+0000 7fb5a6ffd640 1 -- 172.21.15.78:0/2493609891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb594003520 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5a6ffd640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390e180 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c210 tx=0x7fb5a80c8f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44826a1a8a32902 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5867fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5867fc640 1 -- 172.21.15.78:0/2493609891 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598049070 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5867fc640 1 -- 172.21.15.78:0/2493609891 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb5980075f0 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564833906840 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5867fc640 1 -- 172.21.15.78:0/2493609891 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb59803c040 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 10 client.8019 fetch_fsmap learned FSMap version 101 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=2+}) v3 -- 0x5648337f6780 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5867fc640 1 -- 172.21.15.78:0/2493609891 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564833906840 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5867fc640 1 -- 172.21.15.78:0/2493609891 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 101) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb59804c070 con 0x564833901a70 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 10 client.8019 fetch_fsmap finished waiting for FSMap version 101 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 10 client.8019 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:36.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 20 client.8019 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x564833974680 0x564833916fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 4 client.8019 mds_command: new command op to 7791 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 --> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564833901900 con 0x564833974680 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.157+0000 7fb59ffff640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x564833974680 0x564833916fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5867fc640 10 client.8019 ms_handle_connect on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb59ffff640 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x564833974680 0x564833916fe0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d6ddcf4b9e5d2c server_cookie=3489945016aa8850 in_seq=0 out_seq=0 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5867fc640 1 -- 172.21.15.78:0/2493609891 <== mds.0 v2:172.21.15.78:6835/100250519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x564833901900 con 0x564833974680 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5867fc640 10 client.8019 handle_command_reply: tid=0 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 client.8019 shutdown 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 msgr2=0x56483390e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390e180 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c210 tx=0x7fb5a80c8f40 comp rx=0 tx=0).stop 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 shutdown_connections 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:6835/100250519,v1:172.21.15.78:6837/100250519] conn(0x564833974680 0x564833916fe0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56483390c8b0 0x564833974140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x564833909740 0x5648338ed390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 --2- 172.21.15.78:0/2493609891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564833901a70 0x56483390e180 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5867fc640 0 client.8019 ms_handle_reset on v2:172.21.15.78:6835/100250519 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 >> 172.21.15.78:0/2493609891 conn(0x5648338cfae0 msgr2=0x564833905bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 shutdown_connections 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 1 -- 172.21.15.78:0/2493609891 wait complete. 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.158+0000 7fb5b05a4740 20 client.8019 trim_cache size 0 max 16384 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7791, 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:56:36.167 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 16.815229061, 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 101, 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 124, 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 124, 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 21.962174739999998 2024-02-19T14:56:36.168 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:36.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb58806b810 msgr2=0x7fb58806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb58806b810 0x7fb58806dcd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb598034000 tx=0x7fb598007000 comp rx=0 tx=0).stop 2024-02-19T14:56:36.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 msgr2=0x7fb5a814c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 0x7fb5a814c060 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80c58f0 tx=0x7fb5a000fc90 comp rx=0 tx=0).stop 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 shutdown_connections 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb58806b810 0x7fb58806dcd0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5a81eb340 0x7fb5a814c060 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a814e640 0x7fb5a81eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 --2- 172.21.15.78:0/3491993404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb594003970 0x7fb5a8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 >> 172.21.15.78:0/3491993404 conn(0x7fb5a806b4b0 msgr2=0x7fb5a81557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 shutdown_connections 2024-02-19T14:56:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.159+0000 7fb5ad8ed640 1 -- 172.21.15.78:0/3491993404 wait complete. 2024-02-19T14:56:36.172 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:18.671108+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7791": { "addr": "172.21.15.78:6837/100250519", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 100250519, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 100250519, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7791, "incarnation": 98, "join_fscid": 9, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-19T14:56:31.368255+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7791 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/4213660424", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 4213660424, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 4213660424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6239, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/1857207464", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1857207464, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1857207464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7036, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1870596299", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1870596299, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1870596299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7788, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:56:36.173 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 9, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:56:36.173 INFO:tasks.cephfs.test_failover:current = [{'gid': 7788, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/1870596299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1870596299}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1870596299}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 6239, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/4213660424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4213660424}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4213660424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7791, 'name': 'c', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.78:6837/100250519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 100250519}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 100250519}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7036, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1857207464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1857207464}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1857207464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}] 2024-02-19T14:56:36.174 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:56:36.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 9:0 2024-02-19T14:56:36.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a3b246640 1 Processor -- start 2024-02-19T14:56:36.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a3b246640 1 -- start start 2024-02-19T14:56:36.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a3b246640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a340c8500 0x7f5a340c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a3b246640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a340c34a0 con 0x7f5a340c42e0 2024-02-19T14:56:36.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a3b246640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3406c8e0 con 0x7f5a340c2f50 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a3b246640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3406c090 con 0x7f5a340c8500 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a38fe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a340c8500 0x7f5a340c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.274+0000 7f5a33fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5a340c42e0 0x7f5a34158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40788/0 (socket says 172.21.15.78:40788) 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.275+0000 7f5a38fe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5a340c8500 0x7f5a340c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47592/0 (socket says 172.21.15.78:47592) 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.275+0000 7f5a33fff640 1 -- 172.21.15.78:0/3850822680 learned_addr learned my addr 172.21.15.78:0/3850822680 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.275+0000 7f5a38fe2640 1 -- 172.21.15.78:0/3850822680 >> v1:172.21.15.150:6789/0 conn(0x7f5a340c2f50 legacy=0x7f5a3415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.275+0000 7f5a38fe2640 1 -- 172.21.15.78:0/3850822680 >> v1:172.21.15.78:6789/0 conn(0x7f5a340c42e0 legacy=0x7f5a34158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.275+0000 7f5a337fe640 1 -- 172.21.15.78:0/3850822680 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360638415 0 0) 0x7f5a340c34a0 con 0x7f5a340c42e0 2024-02-19T14:56:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.275+0000 7f5a38fe2640 1 -- 172.21.15.78:0/3850822680 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3406cbc0 con 0x7f5a340c8500 2024-02-19T14:56:36.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a38fe2640 1 --2- 172.21.15.78:0/3850822680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a340c88e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009d90 tx=0x7f5a2c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faeda434a3bc69d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a337fe640 1 -- 172.21.15.78:0/3850822680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c02cc60 con 0x7f5a340c8500 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a337fe640 1 -- 172.21.15.78:0/3850822680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c02cdc0 con 0x7f5a340c8500 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a337fe640 1 -- 172.21.15.78:0/3850822680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c034660 con 0x7f5a340c8500 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 -- 172.21.15.78:0/3850822680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 msgr2=0x7f5a340c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 --2- 172.21.15.78:0/3850822680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a340c88e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009d90 tx=0x7f5a2c02beb0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 -- 172.21.15.78:0/3850822680 shutdown_connections 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 --2- 172.21.15.78:0/3850822680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a340c88e0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 -- 172.21.15.78:0/3850822680 >> 172.21.15.78:0/3850822680 conn(0x7f5a340aa300 msgr2=0x7f5a340aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 -- 172.21.15.78:0/3850822680 shutdown_connections 2024-02-19T14:56:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.276+0000 7f5a3b246640 1 -- 172.21.15.78:0/3850822680 wait complete. 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 Processor -- start 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 -- start start 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a340c2f50 0x7f5a34154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a3414e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a341426d0 0x7f5a34141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341553b0 con 0x7f5a340c8500 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341555a0 con 0x7f5a341426d0 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a3b246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341470c0 con 0x7f5a340c2f50 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a33fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a3414e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a33fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a3414e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47594/0 (socket says 172.21.15.78:47594) 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.278+0000 7f5a33fff640 1 -- 172.21.15.78:0/1701878135 learned_addr learned my addr 172.21.15.78:0/1701878135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a38fe2640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a340c2f50 0x7f5a34154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a33fff640 1 -- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a340c2f50 msgr2=0x7f5a34154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a33fff640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a340c2f50 0x7f5a34154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a33fff640 1 -- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a341426d0 msgr2=0x7f5a34141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a397e3640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a341426d0 0x7f5a34141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a33fff640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a341426d0 0x7f5a34141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a33fff640 1 -- 172.21.15.78:0/1701878135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c009a70 con 0x7f5a340c8500 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a33fff640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a3414e360 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800c4a0 tx=0x7f5a2800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a280074e0 con 0x7f5a340c8500 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28007640 con 0x7f5a340c8500 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a341473a0 con 0x7f5a340c8500 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28004a00 con 0x7f5a340c8500 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.279+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a34153970 con 0x7f5a340c8500 2024-02-19T14:56:36.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.280+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a34071c20 con 0x7f5a340c8500 2024-02-19T14:56:36.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.281+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5a28004b60 con 0x7f5a340c8500 2024-02-19T14:56:36.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.281+0000 7f5a317fa640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5a2006b810 0x7f5a2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.281+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a28089830 con 0x7f5a340c8500 2024-02-19T14:56:36.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.281+0000 7f5a38fe2640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5a2006b810 0x7f5a2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.281+0000 7f5a38fe2640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5a2006b810 0x7f5a2006dcd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c036000 tx=0x7f5a2c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.289+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5a28054150 con 0x7f5a340c8500 2024-02-19T14:56:36.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.536+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "9:0"} v 0) v1 -- 0x7f5a34155760 con 0x7f5a340c8500 2024-02-19T14:56:36.539 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:56:36.537+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:56:36.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.538+0000 7f5a317fa640 1 -- 172.21.15.78:0/1701878135 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "9:0"}]=0 failed mds gid 7791 v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5a28018090 con 0x7f5a340c8500 2024-02-19T14:56:36.541 INFO:teuthology.orchestra.run.smithi078.stderr:failed mds gid 7791 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5a2006b810 msgr2=0x7f5a2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5a2006b810 0x7f5a2006dcd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c036000 tx=0x7f5a2c030000 comp rx=0 tx=0).stop 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 msgr2=0x7f5a3414e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a3414e360 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800c4a0 tx=0x7f5a2800c960 comp rx=0 tx=0).stop 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 shutdown_connections 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5a2006b810 0x7f5a2006dcd0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a340c8500 0x7f5a3414e360 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a341426d0 0x7f5a34141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 --2- 172.21.15.78:0/1701878135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5a340c2f50 0x7f5a34154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 >> 172.21.15.78:0/1701878135 conn(0x7f5a340aa300 msgr2=0x7f5a340a9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 shutdown_connections 2024-02-19T14:56:36.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.541+0000 7f5a3b246640 1 -- 172.21.15.78:0/1701878135 wait complete. 2024-02-19T14:56:36.545 INFO:tasks.ceph.mds.c.smithi078.stderr: -502> 2024-02-19T14:56:24.589+0000 7fbd25ed3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:56:36.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:36.572 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:56:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 Processor -- start 2024-02-19T14:56:36.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 -- start start 2024-02-19T14:56:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14ec155f90 0x7f14ec06d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14ec0c6d30 0x7f14ec0c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec0c6af0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec0c7940 con 0x7f14ec0c6d30 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.657+0000 7f14f1fbc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec0c7c20 con 0x7f14ec155f90 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14ec0c6d30 0x7f14ec0c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f14ec0c6d30 0x7f14ec0c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47630/0 (socket says 172.21.15.78:47630) 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 -- 172.21.15.78:0/179113324 learned_addr learned my addr 172.21.15.78:0/179113324 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eb7fe640 1 --2- 172.21.15.78:0/179113324 >> v2:172.21.15.150:3300/0 conn(0x7f14ec155f90 0x7f14ec06d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 -- 172.21.15.78:0/179113324 >> v2:172.21.15.150:3300/0 conn(0x7f14ec155f90 msgr2=0x7f14ec06d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 --2- 172.21.15.78:0/179113324 >> v2:172.21.15.150:3300/0 conn(0x7f14ec155f90 0x7f14ec06d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 -- 172.21.15.78:0/179113324 >> v1:172.21.15.150:6789/0 conn(0x7f14ec0c3aa0 legacy=0x7f14ec1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.658+0000 7f14eaffd640 1 -- 172.21.15.78:0/179113324 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec0c7f00 con 0x7f14ec0c6d30 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14eaffd640 1 --2- 172.21.15.78:0/179113324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c6d30 0x7f14ec0c3560 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0c2a40 tx=0x7f14e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334f9f51938b79fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14ea7fc640 1 -- 172.21.15.78:0/179113324 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0031870 con 0x7f14ec0c6d30 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14ea7fc640 1 -- 172.21.15.78:0/179113324 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e00319d0 con 0x7f14ec0c6d30 2024-02-19T14:56:36.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14ea7fc640 1 -- 172.21.15.78:0/179113324 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0031d60 con 0x7f14ec0c6d30 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 -- 172.21.15.78:0/179113324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c6d30 msgr2=0x7f14ec0c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/179113324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c6d30 0x7f14ec0c3560 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0c2a40 tx=0x7f14e002b910 comp rx=0 tx=0).stop 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 -- 172.21.15.78:0/179113324 shutdown_connections 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/179113324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c6d30 0x7f14ec0c3560 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/179113324 >> v2:172.21.15.150:3300/0 conn(0x7f14ec155f90 0x7f14ec06d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 -- 172.21.15.78:0/179113324 >> 172.21.15.78:0/179113324 conn(0x7f14ec06b700 msgr2=0x7f14ec0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 -- 172.21.15.78:0/179113324 shutdown_connections 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.659+0000 7f14f1fbc640 1 -- 172.21.15.78:0/179113324 wait complete. 2024-02-19T14:56:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.660+0000 7f14f1fbc640 1 Processor -- start 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 -- start start 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 0x7f14ec0ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14ec0c6d30 0x7f14ec14f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14ec155f90 0x7f14ec14fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1defe0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec06f2e0 con 0x7f14ec155f90 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14f1fbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec154bc0 con 0x7f14ec0c6d30 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14eb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 0x7f14ec0ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14eb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 0x7f14ec0ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47636/0 (socket says 172.21.15.78:47636) 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.661+0000 7f14eb7fe640 1 -- 172.21.15.78:0/2817431698 learned_addr learned my addr 172.21.15.78:0/2817431698 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14ebfff640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14ec155f90 0x7f14ec14fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eaffd640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14ec0c6d30 0x7f14ec14f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eb7fe640 1 -- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14ec0c6d30 msgr2=0x7f14ec14f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eb7fe640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14ec0c6d30 0x7f14ec14f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eb7fe640 1 -- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14ec155f90 msgr2=0x7f14ec14fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eb7fe640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14ec155f90 0x7f14ec14fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eb7fe640 1 -- 172.21.15.78:0/2817431698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00094d0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eaffd640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14ec0c6d30 0x7f14ec14f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14eb7fe640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 0x7f14ec0ac6b0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00d5f0 tx=0x7f14dc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc0125f0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc012750 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec1503d0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc012b00 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.662+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec1578e0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.663+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec074990 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.664+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f14dc012c60 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.664+0000 7f14d3fff640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14d806b810 0x7f14d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.664+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f14dc08a760 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.664+0000 7f14eaffd640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14d806b810 0x7f14d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.664+0000 7f14eaffd640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14d806b810 0x7f14d806dcd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f14e0036000 tx=0x7f14e003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:36.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.668+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f14dc055030 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.876+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14ec0c75a0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.876+0000 7f14d3fff640 1 -- 172.21.15.78:0/2817431698 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4160 (secure 0 0 0) 0x7f14dc0586f0 con 0x7f14ec0c3aa0 2024-02-19T14:56:36.879 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:36.879 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:36.542299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7788},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7788":{"gid":7788,"name":"a","rank":0,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 103 2024-02-19T14:56:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14d806b810 msgr2=0x7f14d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14d806b810 0x7f14d806dcd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f14e0036000 tx=0x7f14e003c040 comp rx=0 tx=0).stop 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 msgr2=0x7f14ec0ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 0x7f14ec0ac6b0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00d5f0 tx=0x7f14dc00dab0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 shutdown_connections 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f14d806b810 0x7f14d806dcd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14ec155f90 0x7f14ec14fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14ec0c6d30 0x7f14ec14f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 --2- 172.21.15.78:0/2817431698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14ec0c3aa0 0x7f14ec0ac6b0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.878+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 >> 172.21.15.78:0/2817431698 conn(0x7f14ec06b700 msgr2=0x7f14ec0aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.879+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 shutdown_connections 2024-02-19T14:56:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:36.879+0000 7f14f1fbc640 1 -- 172.21.15.78:0/2817431698 wait complete. 2024-02-19T14:56:37.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:36.542299+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 125, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7788}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7788': {'gid': 7788, 'name': 'a', 'rank': 0, 'incarnation': 103, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.78:6836/1870596299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1870596299}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1870596299}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:37.295 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7788:up:replay 2024-02-19T14:56:38.296 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:38.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.403+0000 7f468345c640 1 Processor -- start 2024-02-19T14:56:38.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.403+0000 7f468345c640 1 -- start start 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f468345c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f467c0c32c0 0x7f467c0c3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f468345c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c15ca30 0x7f467c15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f468345c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c0c19d0 con 0x7f467c158df0 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f468345c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c0c4740 con 0x7f467c0c32c0 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f468345c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0c4a20 con 0x7f467c15ca30 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46809f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f467c0c32c0 0x7f467c0c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46811f8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f467c158df0 0x7f467c0c0f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56926/0 (socket says 172.21.15.78:56926) 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46811f8640 1 -- 172.21.15.78:0/3196423848 learned_addr learned my addr 172.21.15.78:0/3196423848 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46809f7640 1 -- 172.21.15.78:0/3196423848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c15ca30 msgr2=0x7f467c15cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46809f7640 1 --2- 172.21.15.78:0/3196423848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c15ca30 0x7f467c15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46809f7640 1 -- 172.21.15.78:0/3196423848 >> v1:172.21.15.78:6789/0 conn(0x7f467c158df0 legacy=0x7f467c0c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.404+0000 7f46809f7640 1 -- 172.21.15.78:0/3196423848 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c0c4d00 con 0x7f467c0c32c0 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.405+0000 7f46809f7640 1 --2- 172.21.15.78:0/3196423848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c0c3770 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f46780097a0 tx=0x7f467802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf9671535d931a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.405+0000 7f4673fff640 1 -- 172.21.15.78:0/3196423848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467802e070 con 0x7f467c0c32c0 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.405+0000 7f4673fff640 1 -- 172.21.15.78:0/3196423848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46780315a0 con 0x7f467c0c32c0 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.405+0000 7f4673fff640 1 -- 172.21.15.78:0/3196423848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678031930 con 0x7f467c0c32c0 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 -- 172.21.15.78:0/3196423848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 msgr2=0x7f467c0c3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 --2- 172.21.15.78:0/3196423848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c0c3770 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f46780097a0 tx=0x7f467802b840 comp rx=0 tx=0).stop 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 -- 172.21.15.78:0/3196423848 shutdown_connections 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 --2- 172.21.15.78:0/3196423848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c0c3770 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 --2- 172.21.15.78:0/3196423848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c15ca30 0x7f467c15cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 -- 172.21.15.78:0/3196423848 >> 172.21.15.78:0/3196423848 conn(0x7f467c06b270 msgr2=0x7f467c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 -- 172.21.15.78:0/3196423848 shutdown_connections 2024-02-19T14:56:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.406+0000 7f468345c640 1 -- 172.21.15.78:0/3196423848 wait complete. 2024-02-19T14:56:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.407+0000 7f468345c640 1 Processor -- start 2024-02-19T14:56:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.407+0000 7f468345c640 1 -- start start 2024-02-19T14:56:38.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c1f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c158df0 0x7f467c156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f467c15ca30 0x7f467c0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c1f2d20 con 0x7f467c0c32c0 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0a8250 con 0x7f467c15ca30 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c0a6310 con 0x7f467c158df0 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c1f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c1f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51312/0 (socket says 172.21.15.78:51312) 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 -- 172.21.15.78:0/4260338472 learned_addr learned my addr 172.21.15.78:0/4260338472 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 -- 172.21.15.78:0/4260338472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c158df0 msgr2=0x7f467c156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c158df0 0x7f467c156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 -- 172.21.15.78:0/4260338472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f467c15ca30 msgr2=0x7f467c0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f467c15ca30 0x7f467c0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 -- 172.21.15.78:0/4260338472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678009400 con 0x7f467c0c32c0 2024-02-19T14:56:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f46811f8640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c1f2730 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c430 tx=0x7f466c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c005ce0 con 0x7f467c0c32c0 2024-02-19T14:56:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c005e40 con 0x7f467c0c32c0 2024-02-19T14:56:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c0a47f0 con 0x7f467c0c32c0 2024-02-19T14:56:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.408+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c0072f0 con 0x7f467c0c32c0 2024-02-19T14:56:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.409+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c0a4510 con 0x7f467c0c32c0 2024-02-19T14:56:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.410+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c074990 con 0x7f467c0c32c0 2024-02-19T14:56:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.410+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f466c007450 con 0x7f467c0c32c0 2024-02-19T14:56:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.410+0000 7f4671ffb640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f466006b810 0x7f466006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.410+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f466c0898f0 con 0x7f467c0c32c0 2024-02-19T14:56:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.410+0000 7f46809f7640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f466006b810 0x7f466006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.411+0000 7f46809f7640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f466006b810 0x7f466006dcd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4678036000 tx=0x7f4678030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.415+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f466c0541c0 con 0x7f467c0c32c0 2024-02-19T14:56:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.628+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f467c0c4480 con 0x7f467c0c32c0 2024-02-19T14:56:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.628+0000 7f4671ffb640 1 -- 172.21.15.78:0/4260338472 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4951 (secure 0 0 0) 0x7f466c017090 con 0x7f467c0c32c0 2024-02-19T14:56:38.631 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:38.631 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8034,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":104}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:37.552665+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7788},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7788":{"gid":7788,"name":"a","rank":0,"incarnation":103,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.78:6836/1870596299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1870596299},{"type":"v1","addr":"172.21.15.78:6836","nonce":1870596299}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 105 2024-02-19T14:56:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.630+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f466006b810 msgr2=0x7f466006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.630+0000 7f468345c640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f466006b810 0x7f466006dcd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4678036000 tx=0x7f4678030000 comp rx=0 tx=0).stop 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 msgr2=0x7f467c1f2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c1f2730 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c430 tx=0x7f466c00c8f0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 shutdown_connections 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f466006b810 0x7f466006dcd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f467c15ca30 0x7f467c0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f467c158df0 0x7f467c156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 --2- 172.21.15.78:0/4260338472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f467c0c32c0 0x7f467c1f2730 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 >> 172.21.15.78:0/4260338472 conn(0x7f467c06b270 msgr2=0x7f467c06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 shutdown_connections 2024-02-19T14:56:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:38.631+0000 7f468345c640 1 -- 172.21.15.78:0/4260338472 wait complete. 2024-02-19T14:56:39.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:37.552665+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 125, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7788}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7788': {'gid': 7788, 'name': 'a', 'rank': 0, 'incarnation': 103, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.78:6836/1870596299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1870596299}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1870596299}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:39.046 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7788:up:rejoin 2024-02-19T14:56:39.593 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:56:40.048 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:40.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.159+0000 7fd5543d0640 1 Processor -- start 2024-02-19T14:56:40.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.159+0000 7fd5543d0640 1 -- start start 2024-02-19T14:56:40.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.160+0000 7fd5543d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd54c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.160+0000 7fd5543d0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0ae670 con 0x7fd54c0705b0 2024-02-19T14:56:40.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.160+0000 7fd5543d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0716b0 con 0x7fd54c0add70 2024-02-19T14:56:40.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.160+0000 7fd5543d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c153780 con 0x7fd54c0c4980 2024-02-19T14:56:40.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55296d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd54c0705b0 0x7fd54c0709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56956/0 (socket says 172.21.15.78:56956) 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55296d640 1 -- 172.21.15.78:0/2028330136 learned_addr learned my addr 172.21.15.78:0/2028330136 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55196b640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd54c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638957860 0 0) 0x7fd54c0ae670 con 0x7fd54c0705b0 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534003520 con 0x7fd54c0705b0 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271301729 0 0) 0x7fd54c0716b0 con 0x7fd54c0add70 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c0ae670 con 0x7fd54c0add70 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479960795 0 0) 0x7fd534003520 con 0x7fd54c0705b0 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 msgr2=0x7fd54c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd54c0c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 >> v1:172.21.15.150:6789/0 conn(0x7fd54c0add70 legacy=0x7fd54c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c153a60 con 0x7fd54c0705b0 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fd53c001670 con 0x7fd54c0705b0 2024-02-19T14:56:40.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 >> v1:172.21.15.78:6789/0 conn(0x7fd54c0705b0 legacy=0x7fd54c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd534003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5340047c0 0x7fd534004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd5340051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c153780 con 0x7fd534003b30 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53400bb10 con 0x7fd5340047c0 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53400bca0 con 0x7fd54c0c4980 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55216c640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd534003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55196b640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5340047c0 0x7fd534004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.161+0000 7fd55296d640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd5340051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55216c640 1 -- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 msgr2=0x7fd5340051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55216c640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd5340051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55216c640 1 -- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5340047c0 msgr2=0x7fd534004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55216c640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5340047c0 0x7fd534004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55216c640 1 -- 172.21.15.78:0/2028330136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534003520 con 0x7fd534003b30 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55296d640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c4980 0x7fd5340051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55216c640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd534003f30 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd544001900 tx=0x7fd5340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671f1a22b960717 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440052d0 con 0x7fd534003b30 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd544005430 con 0x7fd534003b30 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.162+0000 7fd55116a640 1 -- 172.21.15.78:0/2028330136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544003810 con 0x7fd534003b30 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd5543d0640 1 -- 172.21.15.78:0/2028330136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 msgr2=0x7fd534003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd5543d0640 1 --2- 172.21.15.78:0/2028330136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd534003f30 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd544001900 tx=0x7fd5340057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd55296d640 1 -- 172.21.15.78:0/2028330136 reap_dead start 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd5543d0640 1 -- 172.21.15.78:0/2028330136 shutdown_connections 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd5543d0640 1 -- 172.21.15.78:0/2028330136 >> 172.21.15.78:0/2028330136 conn(0x7fd54c0c85e0 msgr2=0x7fd54c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd5543d0640 1 -- 172.21.15.78:0/2028330136 shutdown_connections 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.163+0000 7fd5543d0640 1 -- 172.21.15.78:0/2028330136 wait complete. 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.164+0000 7fd5543d0640 1 Processor -- start 2024-02-19T14:56:40.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.164+0000 7fd5543d0640 1 -- start start 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd5543d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd54c0c2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd5543d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c1b00 0x7fd54c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd5543d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c14abb0 0x7fd54c14b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd5543d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c154050 con 0x7fd534003b30 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd5543d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c1ec430 con 0x7fd54c14abb0 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd5543d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c1ec710 con 0x7fd54c0c1b00 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd54c0c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd54c0c2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51350/0 (socket says 172.21.15.78:51350) 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 -- 172.21.15.78:0/3427589496 learned_addr learned my addr 172.21.15.78:0/3427589496 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:40.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55296d640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c14abb0 0x7fd54c14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55196b640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c1b00 0x7fd54c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 -- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c1b00 msgr2=0x7fd54c14d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c1b00 0x7fd54c14d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 -- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c14abb0 msgr2=0x7fd54c14b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c14abb0 0x7fd54c14b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 -- 172.21.15.78:0/3427589496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c1542a0 con 0x7fd534003b30 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55296d640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c14abb0 0x7fd54c14b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.165+0000 7fd55216c640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd54c0c2050 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd544035040 tx=0x7fd5440012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.166+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544007250 con 0x7fd534003b30 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.166+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5440073b0 con 0x7fd534003b30 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.166+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c154580 con 0x7fd534003b30 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.166+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440035f0 con 0x7fd534003b30 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.166+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c152090 con 0x7fd534003b30 2024-02-19T14:56:40.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.167+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c06f680 con 0x7fd534003b30 2024-02-19T14:56:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.167+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd544003750 con 0x7fd534003b30 2024-02-19T14:56:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.167+0000 7fd54affd640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd53806b840 0x7fd53806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.168+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fd5440ac3e0 con 0x7fd534003b30 2024-02-19T14:56:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.168+0000 7fd55196b640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd53806b840 0x7fd53806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.168+0000 7fd55196b640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd53806b840 0x7fd53806dd00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd54c152420 tx=0x7fd54000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:40.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.172+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd544076d90 con 0x7fd534003b30 2024-02-19T14:56:40.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.379+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54c074990 con 0x7fd534003b30 2024-02-19T14:56:40.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.379+0000 7fd54affd640 1 -- 172.21.15.78:0/3427589496 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4159 (secure 0 0 0) 0x7fd54407a450 con 0x7fd534003b30 2024-02-19T14:56:40.382 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:40.382 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:39.558518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:replay","state_seq":1,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:40.382 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 107 2024-02-19T14:56:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd53806b840 msgr2=0x7fd53806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd53806b840 0x7fd53806dd00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd54c152420 tx=0x7fd54000b040 comp rx=0 tx=0).stop 2024-02-19T14:56:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 msgr2=0x7fd54c0c2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd54c0c2050 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd544035040 tx=0x7fd5440012d0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 shutdown_connections 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd53806b840 0x7fd53806dd00 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c14abb0 0x7fd54c14b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd54c0c1b00 0x7fd54c14d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 --2- 172.21.15.78:0/3427589496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd534003b30 0x7fd54c0c2050 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 >> 172.21.15.78:0/3427589496 conn(0x7fd54c0c85e0 msgr2=0x7fd54c0c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 shutdown_connections 2024-02-19T14:56:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:40.381+0000 7fd5543d0640 1 -- 172.21.15.78:0/3427589496 wait complete. 2024-02-19T14:56:40.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:39.558518+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 126, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8034}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8034': {'gid': 8034, 'name': 'c', 'rank': 0, 'incarnation': 107, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.78:6837/2501419582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2501419582}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2501419582}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:40.798 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8034:up:replay 2024-02-19T14:56:41.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:41.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.907+0000 7f69a492e640 1 Processor -- start 2024-02-19T14:56:41.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.907+0000 7f69a492e640 1 -- start start 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f69a492e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f69a0152480 0x7f69a01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f69a492e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69a00c7070 0x7f69a006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f69a492e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a0155ef0 con 0x7f69a006d350 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f69a492e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a006dec0 con 0x7f69a00c7070 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f69a492e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a00c75c0 con 0x7f69a0152480 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69a00c7070 0x7f69a006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69a00c7070 0x7f69a006ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51356/0 (socket says 172.21.15.78:51356) 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 -- 172.21.15.78:0/1974921453 learned_addr learned my addr 172.21.15.78:0/1974921453 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f699e59c640 1 --2- 172.21.15.78:0/1974921453 >> v2:172.21.15.150:3300/0 conn(0x7f69a0152480 0x7f69a01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 -- 172.21.15.78:0/1974921453 >> v2:172.21.15.150:3300/0 conn(0x7f69a0152480 msgr2=0x7f69a01559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 --2- 172.21.15.78:0/1974921453 >> v2:172.21.15.150:3300/0 conn(0x7f69a0152480 0x7f69a01559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:41.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 -- 172.21.15.78:0/1974921453 >> v1:172.21.15.150:6789/0 conn(0x7f69a006d350 legacy=0x7f69a00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f6997fff640 1 -- 172.21.15.78:0/1974921453 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a00c78a0 con 0x7f69a00c7070 2024-02-19T14:56:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.908+0000 7f699e59c640 1 --2- 172.21.15.78:0/1974921453 >> v2:172.21.15.150:3300/0 conn(0x7f69a0152480 0x7f69a01559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.909+0000 7f6997fff640 1 --2- 172.21.15.78:0/1974921453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00c7070 0x7f69a006ce10 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f69a00aa1b0 tx=0x7f698802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bc633c6d400362 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.909+0000 7f699dd9b640 1 -- 172.21.15.78:0/1974921453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988031870 con 0x7f69a00c7070 2024-02-19T14:56:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.909+0000 7f699dd9b640 1 -- 172.21.15.78:0/1974921453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69880319d0 con 0x7f69a00c7070 2024-02-19T14:56:41.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.909+0000 7f699dd9b640 1 -- 172.21.15.78:0/1974921453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988031d60 con 0x7f69a00c7070 2024-02-19T14:56:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 -- 172.21.15.78:0/1974921453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00c7070 msgr2=0x7f69a006ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:41.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 --2- 172.21.15.78:0/1974921453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00c7070 0x7f69a006ce10 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f69a00aa1b0 tx=0x7f698802b910 comp rx=0 tx=0).stop 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 -- 172.21.15.78:0/1974921453 shutdown_connections 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 --2- 172.21.15.78:0/1974921453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00c7070 0x7f69a006ce10 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 --2- 172.21.15.78:0/1974921453 >> v2:172.21.15.150:3300/0 conn(0x7f69a0152480 0x7f69a01559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 -- 172.21.15.78:0/1974921453 >> 172.21.15.78:0/1974921453 conn(0x7f69a00ca8a0 msgr2=0x7f69a00cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 -- 172.21.15.78:0/1974921453 shutdown_connections 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.910+0000 7f69a492e640 1 -- 172.21.15.78:0/1974921453 wait complete. 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.911+0000 7f69a492e640 1 Processor -- start 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.911+0000 7f69a492e640 1 -- start start 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69a00c7070 0x7f69a014d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 0x7f69a00c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69a00aa590 0x7f69a00aaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a014ddf0 con 0x7f69a0152480 2024-02-19T14:56:41.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0158870 con 0x7f69a00aa590 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0158b50 con 0x7f69a00c7070 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 0x7f69a00c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 0x7f69a00c0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51370/0 (socket says 172.21.15.78:51370) 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 -- 172.21.15.78:0/2776132705 learned_addr learned my addr 172.21.15.78:0/2776132705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f699ed9d640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69a00aa590 0x7f69a00aaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f699e59c640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69a00c7070 0x7f69a014d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 -- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69a00c7070 msgr2=0x7f69a014d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69a00c7070 0x7f69a014d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 -- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69a00aa590 msgr2=0x7f69a00aaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69a00aa590 0x7f69a00aaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 -- 172.21.15.78:0/2776132705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69880094d0 con 0x7f69a0152480 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6997fff640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 0x7f69a00c0810 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f6988000c00 tx=0x7f69880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988031840 con 0x7f69a0152480 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69880443f0 con 0x7f69a0152480 2024-02-19T14:56:41.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a0158e30 con 0x7f69a0152480 2024-02-19T14:56:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69880447a0 con 0x7f69a0152480 2024-02-19T14:56:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.912+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a00c0d50 con 0x7f69a0152480 2024-02-19T14:56:41.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.913+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0074990 con 0x7f69a0152480 2024-02-19T14:56:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.914+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f698804d450 con 0x7f69a0152480 2024-02-19T14:56:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.914+0000 7f6996ffd640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f698006b840 0x7f698006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.915+0000 7f699e59c640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f698006b840 0x7f698006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.915+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f698802d040 con 0x7f69a0152480 2024-02-19T14:56:41.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.915+0000 7f699e59c640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f698006b840 0x7f698006dd00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f69a00c1000 tx=0x7f6990008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:41.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:41.919+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6988077230 con 0x7f69a0152480 2024-02-19T14:56:42.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.125+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a0155ef0 con 0x7f69a0152480 2024-02-19T14:56:42.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.126+0000 7f6996ffd640 1 -- 172.21.15.78:0/2776132705 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f69a0155ef0 con 0x7f69a0152480 2024-02-19T14:56:42.129 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:42.129 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":109,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:40.568938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:42.129 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 109 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f698006b840 msgr2=0x7f698006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f698006b840 0x7f698006dd00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f69a00c1000 tx=0x7f6990008040 comp rx=0 tx=0).stop 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 msgr2=0x7f69a00c0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 0x7f69a00c0810 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f6988000c00 tx=0x7f69880046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 shutdown_connections 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f698006b840 0x7f698006dd00 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a0152480 0x7f69a00c0810 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69a00aa590 0x7f69a00aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 --2- 172.21.15.78:0/2776132705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69a00c7070 0x7f69a014d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 >> 172.21.15.78:0/2776132705 conn(0x7f69a00ca8a0 msgr2=0x7f69a00af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 shutdown_connections 2024-02-19T14:56:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:42.128+0000 7f69a492e640 1 -- 172.21.15.78:0/2776132705 wait complete. 2024-02-19T14:56:42.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:40.568938+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 126, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8034}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8034': {'gid': 8034, 'name': 'c', 'rank': 0, 'incarnation': 107, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.78:6837/2501419582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2501419582}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2501419582}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:42.544 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8034:up:rejoin 2024-02-19T14:56:43.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.644+0000 7f96ace8f640 1 Processor -- start 2024-02-19T14:56:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.644+0000 7f96ace8f640 1 -- start start 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96ace8f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96a814f1a0 0x7f96a8072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96ace8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 0x7f96a8152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96ace8f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a80c5970 con 0x7f96a806fbc0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96ace8f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a80c5750 con 0x7f96a814f1a0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96ace8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a80c5bc0 con 0x7f96a806ea50 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96a6d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f96a806fbc0 0x7f96a80c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56972/0 (socket says 172.21.15.78:56972) 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96a6d9d640 1 -- 172.21.15.78:0/3938307296 learned_addr learned my addr 172.21.15.78:0/3938307296 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630531532 0 0) 0x7f96a80c5970 con 0x7f96a806fbc0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9690003520 con 0x7f96a806fbc0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96a659c640 1 --2- 172.21.15.78:0/3938307296 >> v2:172.21.15.150:3300/0 conn(0x7f96a814f1a0 0x7f96a8072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.645+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 0x7f96a8152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058577315 0 0) 0x7f9690003520 con 0x7f96a806fbc0 2024-02-19T14:56:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 msgr2=0x7f96a8152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 0x7f96a8152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 >> v2:172.21.15.150:3300/0 conn(0x7f96a814f1a0 msgr2=0x7f96a8072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 --2- 172.21.15.78:0/3938307296 >> v2:172.21.15.150:3300/0 conn(0x7f96a814f1a0 0x7f96a8072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a80c5ea0 con 0x7f96a806fbc0 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f96a00019f0 con 0x7f96a806fbc0 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 >> v1:172.21.15.78:6789/0 conn(0x7f96a806fbc0 legacy=0x7f96a80c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 0x7f9690003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9690003ef0 0x7f96900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9690004d00 0x7f9690005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a80c5750 con 0x7f9690003ef0 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a80c5bc0 con 0x7f9690004d00 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969000ba60 con 0x7f96a806ea50 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9690003ef0 0x7f96900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a659c640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 0x7f9690003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a6d9d640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9690004d00 0x7f9690005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a5d9b640 1 -- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 msgr2=0x7f9690003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96a806ea50 0x7f9690003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a5d9b640 1 -- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9690004d00 msgr2=0x7f9690005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9690004d00 0x7f9690005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.646+0000 7f96a5d9b640 1 -- 172.21.15.78:0/3938307296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9690003520 con 0x7f9690003ef0 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.647+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9690003ef0 0x7f96900047c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f9698009ac0 tx=0x7f96900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be1144a1c74166a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.647+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969802c6b0 con 0x7f9690003ef0 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.647+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969802c810 con 0x7f9690003ef0 2024-02-19T14:56:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.647+0000 7f96a559a640 1 -- 172.21.15.78:0/3938307296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969802cba0 con 0x7f9690003ef0 2024-02-19T14:56:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96ace8f640 1 -- 172.21.15.78:0/3938307296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9690003ef0 msgr2=0x7f96900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96ace8f640 1 --2- 172.21.15.78:0/3938307296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9690003ef0 0x7f96900047c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f9698009ac0 tx=0x7f96900057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96a6d9d640 1 -- 172.21.15.78:0/3938307296 reap_dead start 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96ace8f640 1 -- 172.21.15.78:0/3938307296 shutdown_connections 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96ace8f640 1 -- 172.21.15.78:0/3938307296 >> 172.21.15.78:0/3938307296 conn(0x7f96a806c170 msgr2=0x7f96a806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96ace8f640 1 -- 172.21.15.78:0/3938307296 shutdown_connections 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.648+0000 7f96ace8f640 1 -- 172.21.15.78:0/3938307296 wait complete. 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.649+0000 7f96ace8f640 1 Processor -- start 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.649+0000 7f96ace8f640 1 -- start start 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96ace8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9690003ef0 0x7f96a81ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96ace8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a80c2550 0x7f96a806b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96ace8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 0x7f96a80aa330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96ace8f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a81ea530 con 0x7f96a81eb1e0 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96ace8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a81512f0 con 0x7f96a80c2550 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96ace8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a81ee9f0 con 0x7f9690003ef0 2024-02-19T14:56:43.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 0x7f96a80aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 0x7f96a80aa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51374/0 (socket says 172.21.15.78:51374) 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 -- 172.21.15.78:0/541783325 learned_addr learned my addr 172.21.15.78:0/541783325 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a659c640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9690003ef0 0x7f96a81ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a80c2550 0x7f96a806b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 -- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9690003ef0 msgr2=0x7f96a81ee4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9690003ef0 0x7f96a81ee4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 -- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a80c2550 msgr2=0x7f96a806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a80c2550 0x7f96a806b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 -- 172.21.15.78:0/541783325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a81ead50 con 0x7f96a81eb1e0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a659c640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9690003ef0 0x7f96a81ee4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a5d9b640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a80c2550 0x7f96a806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f96a6d9d640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 0x7f96a80aa330 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f96a0002230 tx=0x7f96a000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.650+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a000ebb0 con 0x7f96a81eb1e0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.651+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a000ed30 con 0x7f96a81eb1e0 2024-02-19T14:56:43.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.651+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a81ea740 con 0x7f96a81eb1e0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.651+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0013660 con 0x7f96a81eb1e0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.651+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a81ea9e0 con 0x7f96a81eb1e0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.652+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a8074990 con 0x7f96a81eb1e0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.652+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f96a00137c0 con 0x7f96a81eb1e0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.652+0000 7f9696ffd640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f968806b810 0x7f968806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.652+0000 7f96a659c640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f968806b810 0x7f968806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.653+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f96a008a720 con 0x7f96a81eb1e0 2024-02-19T14:56:43.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.653+0000 7f96a659c640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f968806b810 0x7f968806dcd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f969c009a80 tx=0x7f969c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:43.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.657+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f96a0054f90 con 0x7f96a81eb1e0 2024-02-19T14:56:43.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.916+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96a806e830 con 0x7f96a81eb1e0 2024-02-19T14:56:43.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.917+0000 7f9696ffd640 1 -- 172.21.15.78:0/541783325 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f96a000eea0 con 0x7f96a81eb1e0 2024-02-19T14:56:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:43.920 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:41.574035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:43.920 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 110 2024-02-19T14:56:43.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f968806b810 msgr2=0x7f968806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:43.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f968806b810 0x7f968806dcd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f969c009a80 tx=0x7f969c008040 comp rx=0 tx=0).stop 2024-02-19T14:56:43.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 msgr2=0x7f96a80aa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:43.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 0x7f96a80aa330 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f96a0002230 tx=0x7f96a000e530 comp rx=0 tx=0).stop 2024-02-19T14:56:43.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 shutdown_connections 2024-02-19T14:56:43.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f968806b810 0x7f968806dcd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96a81eb1e0 0x7f96a80aa330 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96a80c2550 0x7f96a806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 --2- 172.21.15.78:0/541783325 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9690003ef0 0x7f96a81ee4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:43.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 >> 172.21.15.78:0/541783325 conn(0x7f96a806c170 msgr2=0x7f96a80a7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:43.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 shutdown_connections 2024-02-19T14:56:43.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:43.919+0000 7f96ace8f640 1 -- 172.21.15.78:0/541783325 wait complete. 2024-02-19T14:56:44.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:18.671108+0000', 'modified': '2024-02-19T14:56:41.574035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 126, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8034}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8034': {'gid': 8034, 'name': 'c', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.78:6837/2501419582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2501419582}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2501419582}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:56:44.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:56:44.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:56:44.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.436+0000 7f2850015640 1 Processor -- start 2024-02-19T14:56:44.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.436+0000 7f2850015640 1 -- start start 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f2850015640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f28480c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f2850015640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f284806f490 0x7f2848154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f2850015640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284806b470 con 0x7f2848158ef0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f2850015640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2848155510 con 0x7f284806f490 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f2850015640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2848152a90 con 0x7f28481526b0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f284e5b2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2848158ef0 0x7f284806f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56980/0 (socket says 172.21.15.78:56980) 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f284e5b2640 1 -- 172.21.15.78:0/4231177371 learned_addr learned my addr 172.21.15.78:0/4231177371 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f284ddb1640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f28480c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f284d5b0640 1 --2- 172.21.15.78:0/4231177371 >> v2:172.21.15.150:3300/0 conn(0x7f284806f490 0x7f2848154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114902124 0 0) 0x7f284806b470 con 0x7f2848158ef0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.437+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834003520 con 0x7f2848158ef0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360137994 0 0) 0x7f2834003520 con 0x7f2848158ef0 2024-02-19T14:56:44.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 msgr2=0x7f28480c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f28480c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 >> v2:172.21.15.150:3300/0 conn(0x7f284806f490 msgr2=0x7f2848154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 --2- 172.21.15.78:0/4231177371 >> v2:172.21.15.150:3300/0 conn(0x7f284806f490 0x7f2848154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284806b9a0 con 0x7f2848158ef0 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f28400019f0 con 0x7f2848158ef0 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284ddb1640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f28480c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 >> v1:172.21.15.78:6789/0 conn(0x7f2848158ef0 legacy=0x7f284806f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834003970 0x7f2834003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2834004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f2834005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2848155510 con 0x7f28340047c0 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2848152a90 con 0x7f2834003970 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283400b9d0 con 0x7f28481526b0 2024-02-19T14:56:44.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 --2- 172.21.15.78:0/4231177371 >> v2:172.21.15.150:3300/0 conn(0x7f284806f490 0x7f2848154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2834004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284ddb1640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834003970 0x7f2834003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284e5b2640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f2834005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 -- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 msgr2=0x7f2834005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28481526b0 0x7f2834005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 -- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834003970 msgr2=0x7f2834003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834003970 0x7f2834003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.438+0000 7f284d5b0640 1 -- 172.21.15.78:0/4231177371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838009400 con 0x7f28340047c0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f284d5b0640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2834004c40 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f2844009c70 tx=0x7f28340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd80b6da64ca1e5b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844004020 con 0x7f28340047c0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844004180 con 0x7f28340047c0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f284cdaf640 1 -- 172.21.15.78:0/4231177371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284402c960 con 0x7f28340047c0 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f2850015640 1 -- 172.21.15.78:0/4231177371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 msgr2=0x7f2834004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f2850015640 1 --2- 172.21.15.78:0/4231177371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2834004c40 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f2844009c70 tx=0x7f28340057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f284e5b2640 1 -- 172.21.15.78:0/4231177371 reap_dead start 2024-02-19T14:56:44.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f2850015640 1 -- 172.21.15.78:0/4231177371 shutdown_connections 2024-02-19T14:56:44.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.439+0000 7f2850015640 1 -- 172.21.15.78:0/4231177371 >> 172.21.15.78:0/4231177371 conn(0x7f284806ea10 msgr2=0x7f28480a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.440+0000 7f2850015640 1 -- 172.21.15.78:0/4231177371 shutdown_connections 2024-02-19T14:56:44.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.440+0000 7f2850015640 1 -- 172.21.15.78:0/4231177371 wait complete. 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 Processor -- start 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 -- start start 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2848157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28480c2320 0x7f284814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2848157cd0 0x7f284814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28480c3980 con 0x7f28340047c0 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2848150270 con 0x7f2848157cd0 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f2850015640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28480c2820 con 0x7f28480c2320 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f284ddb1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2848157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f284ddb1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2848157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51396/0 (socket says 172.21.15.78:51396) 2024-02-19T14:56:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.441+0000 7f284ddb1640 1 -- 172.21.15.78:0/3412573762 learned_addr learned my addr 172.21.15.78:0/3412573762 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28480c2320 0x7f284814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284e5b2640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2848157cd0 0x7f284814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284ddb1640 1 -- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28480c2320 msgr2=0x7f284814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28480c2320 0x7f284814b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284ddb1640 1 -- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2848157cd0 msgr2=0x7f284814b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2848157cd0 0x7f284814b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284ddb1640 1 -- 172.21.15.78:0/3412573762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28480c2b00 con 0x7f28340047c0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2848157790 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f2838011cd0 tx=0x7f283800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f283e7fc640 1 -- 172.21.15.78:0/3412573762 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283800fd20 con 0x7f28340047c0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f283e7fc640 1 -- 172.21.15.78:0/3412573762 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283800fe80 con 0x7f28340047c0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f283e7fc640 1 -- 172.21.15.78:0/3412573762 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2838004240 con 0x7f28340047c0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28480c2de0 con 0x7f28340047c0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.442+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28481ea6f0 con 0x7f28340047c0 2024-02-19T14:56:44.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.444+0000 7f283e7fc640 1 -- 172.21.15.78:0/3412573762 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f28380043a0 con 0x7f28340047c0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.444+0000 7f283e7fc640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f282c06b840 0x7f282c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.444+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f282c06b840 0x7f282c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.444+0000 7f283e7fc640 1 -- 172.21.15.78:0/3412573762 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f283808aa70 con 0x7f28340047c0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.445+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f282c06b840 0x7f282c06dd00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2844035000 tx=0x7f2844030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.445+0000 7f2852bf3740 1 Processor -- start 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.445+0000 7f2852bf3740 1 -- start start 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 0x5592a4c15b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bda860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f2852bf3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4b95cd0 con 0x5592a4bdbae0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4b48f20 con 0x5592a4c188b0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592a4b46450 con 0x5592a4c15740 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 0x5592a4c15b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 0x5592a4c15b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51402/0 (socket says 172.21.15.78:51402) 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 -- 172.21.15.78:0/3943731180 learned_addr learned my addr 172.21.15.78:0/3943731180 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284e5b2640 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bda860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 -- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 msgr2=0x5592a4c168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 -- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 msgr2=0x5592a4bda860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bda860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 -- 172.21.15.78:0/3943731180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592a4af6f20 con 0x5592a4bdbae0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 0x5592a4c15b20 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f284400f200 tx=0x7f284402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5406363c778dffc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.446+0000 7f282b7fe640 1 -- 172.21.15.78:0/3943731180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284402ca20 con 0x5592a4bdbae0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f282b7fe640 1 -- 172.21.15.78:0/3943731180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284402cb80 con 0x5592a4bdbae0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f282b7fe640 1 -- 172.21.15.78:0/3943731180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844006040 con 0x5592a4bdbae0 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 -- 172.21.15.78:0/3943731180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 msgr2=0x5592a4c15b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 0x5592a4c15b20 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f284400f200 tx=0x7f284402e000 comp rx=0 tx=0).stop 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 -- 172.21.15.78:0/3943731180 shutdown_connections 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4bdbae0 0x5592a4c15b20 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bda860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3943731180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 -- 172.21.15.78:0/3943731180 >> 172.21.15.78:0/3943731180 conn(0x5592a4bd7120 msgr2=0x5592a4bfbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 -- 172.21.15.78:0/3943731180 shutdown_connections 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.447+0000 7f2852bf3740 1 -- 172.21.15.78:0/3943731180 wait complete. 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.448+0000 7f2852bf3740 1 -- start start 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.448+0000 7f2852bf3740 1 Processor -- start 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c0edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bd93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bed700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4b41c30 con 0x5592a4c27430 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4ae6d30 con 0x5592a4c188b0 2024-02-19T14:56:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592a4815310 con 0x5592a4c15740 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2852bf3740 10 client.0 _ll_register_callbacks cb 0x5592a4aeed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bed700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bed700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51404/0 (socket says 172.21.15.78:51404) 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 -- 172.21.15.78:0/572374307 learned_addr learned my addr 172.21.15.78:0/572374307 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284d5b0640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bd93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284ddb1640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c0edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 -- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 msgr2=0x5592a4c0edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c0edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 -- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 msgr2=0x5592a4bd93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bd93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 -- 172.21.15.78:0/572374307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2838009400 con 0x5592a4c27430 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f284e5b2640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bed700 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f28481592d0 tx=0x7f2840007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd255633481bd47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2828ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.449+0000 7f2828ff9640 1 -- 172.21.15.78:0/572374307 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840013630 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2828ff9640 1 -- 172.21.15.78:0/572374307 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f2840013790 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592a48416c0 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2828ff9640 1 -- 172.21.15.78:0/572374307 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2840013950 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 10 client.8088 fetch_fsmap learned FSMap version 110 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x5592a4b402b0 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2828ff9640 1 -- 172.21.15.78:0/572374307 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5592a48416c0 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2828ff9640 1 -- 172.21.15.78:0/572374307 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 110) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f2840013ab0 con 0x5592a4c27430 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 10 client.8088 fetch_fsmap finished waiting for FSMap version 110 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 10 client.8088 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 20 client.8088 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4c29f70 0x5592a4c2a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 4 client.8088 mds_command: new command op to 8034 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 --> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592a4892270 con 0x5592a4c29f70 2024-02-19T14:56:44.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f284ddb1640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4c29f70 0x5592a4c2a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.450+0000 7f2828ff9640 10 client.8088 ms_handle_connect on v2:172.21.15.78:6835/2501419582 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f284ddb1640 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4c29f70 0x5592a4c2a350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82be742340aaad4 server_cookie=dd417742f4f735cf in_seq=0 out_seq=0 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2828ff9640 1 -- 172.21.15.78:0/572374307 <== mds.0 v2:172.21.15.78:6835/2501419582 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5592a4892270 con 0x5592a4c29f70 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2828ff9640 10 client.8088 handle_command_reply: tid=0 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 client.8088 shutdown 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 msgr2=0x5592a4bed700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bed700 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f28481592d0 tx=0x7f2840007500 comp rx=0 tx=0).stop 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 shutdown_connections 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4c29f70 0x5592a4c2a350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bed700 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c188b0 0x5592a4bd93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2828ff9640 0 client.8088 ms_handle_reset on v2:172.21.15.78:6835/2501419582 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 --2- 172.21.15.78:0/572374307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c15740 0x5592a4c0edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 >> 172.21.15.78:0/572374307 conn(0x5592a4bdbae0 msgr2=0x5592a4c166c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.451+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 shutdown_connections 2024-02-19T14:56:44.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.452+0000 7f2852bf3740 1 -- 172.21.15.78:0/572374307 wait complete. 2024-02-19T14:56:44.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.452+0000 7f2852bf3740 20 client.8088 trim_cache size 0 max 16384 2024-02-19T14:56:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.483+0000 7f2852bf3740 1 Processor -- start 2024-02-19T14:56:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.483+0000 7f2852bf3740 1 -- start start 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.484+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4c17030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c188b0 0x5592a4bd8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bd9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f2852bf3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4ac7490 con 0x5592a4c27430 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4c12260 con 0x5592a4c15740 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592a4b73380 con 0x5592a4c188b0 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bd9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bd9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51418/0 (socket says 172.21.15.78:51418) 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 -- 172.21.15.78:0/1887353056 learned_addr learned my addr 172.21.15.78:0/1887353056 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:44.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 -- 172.21.15.78:0/1887353056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c188b0 msgr2=0x5592a4bd8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c188b0 0x5592a4bd8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 -- 172.21.15.78:0/1887353056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 msgr2=0x5592a4c17030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4c17030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 -- 172.21.15.78:0/1887353056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592a4adca20 con 0x5592a4c27430 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f284e5b2640 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bd9400 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f2840000f80 tx=0x7f284000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe6d01b7538fc2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f282affd640 1 -- 172.21.15.78:0/1887353056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840013630 con 0x5592a4c27430 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f282affd640 1 -- 172.21.15.78:0/1887353056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28400137b0 con 0x5592a4c27430 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.485+0000 7f282affd640 1 -- 172.21.15.78:0/1887353056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840013960 con 0x5592a4c27430 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 -- 172.21.15.78:0/1887353056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 msgr2=0x5592a4bd9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bd9400 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f2840000f80 tx=0x7f284000e000 comp rx=0 tx=0).stop 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 -- 172.21.15.78:0/1887353056 shutdown_connections 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c27430 0x5592a4bd9400 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c188b0 0x5592a4bd8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 --2- 172.21.15.78:0/1887353056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4c17030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 -- 172.21.15.78:0/1887353056 >> 172.21.15.78:0/1887353056 conn(0x5592a4bdbae0 msgr2=0x5592a4c166c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 -- 172.21.15.78:0/1887353056 shutdown_connections 2024-02-19T14:56:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.486+0000 7f2852bf3740 1 -- 172.21.15.78:0/1887353056 wait complete. 2024-02-19T14:56:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.487+0000 7f2852bf3740 1 -- start start 2024-02-19T14:56:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.487+0000 7f2852bf3740 1 Processor -- start 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4bfb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 0x5592a4bd5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c27430 0x5592a4c82ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4c39f10 con 0x5592a4c188b0 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a4c1abc0 con 0x5592a4c15740 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592a4bec740 con 0x5592a4c27430 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f2852bf3740 10 client.0 _ll_register_callbacks cb 0x7f2840001390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 0x5592a4bd5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 0x5592a4bd5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51432/0 (socket says 172.21.15.78:51432) 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 -- 172.21.15.78:0/3094585480 learned_addr learned my addr 172.21.15.78:0/3094585480 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284e5b2640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c27430 0x5592a4c82ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4bfb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 -- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c27430 msgr2=0x5592a4c82ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c27430 0x5592a4c82ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 -- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 msgr2=0x5592a4bfb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4bfb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.488+0000 7f284d5b0640 1 -- 172.21.15.78:0/3094585480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592a4b402b0 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f284d5b0640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 0x5592a4bd5660 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f2844009c40 tx=0x7f2844004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56fcfb1c99a18b90 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2828ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2828ff9640 1 -- 172.21.15.78:0/3094585480 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844006250 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2828ff9640 1 -- 172.21.15.78:0/3094585480 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f28440063b0 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f284402cb80 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2828ff9640 1 -- 172.21.15.78:0/3094585480 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2844006570 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 10 client.8094 fetch_fsmap learned FSMap version 110 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x5592a4b02780 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2828ff9640 1 -- 172.21.15.78:0/3094585480 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284402cb80 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2828ff9640 1 -- 172.21.15.78:0/3094585480 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 110) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f28440066d0 con 0x5592a4c188b0 2024-02-19T14:56:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 10 client.8094 fetch_fsmap finished waiting for FSMap version 110 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 10 client.8094 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 20 client.8094 populate_metadata read hostname 'smithi078' 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4bec300 0x5592a4c83220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 4 client.8094 mds_command: new command op to 8034 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 --> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5592a487e610 con 0x5592a4bec300 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.489+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4bec300 0x5592a4c83220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2828ff9640 10 client.8094 ms_handle_connect on v2:172.21.15.78:6835/2501419582 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f284ddb1640 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4bec300 0x5592a4c83220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb291da90413d134 server_cookie=f44913d7824acf81 in_seq=0 out_seq=0 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2828ff9640 1 -- 172.21.15.78:0/3094585480 <== mds.0 v2:172.21.15.78:6835/2501419582 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5592a487e610 con 0x5592a4bec300 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2828ff9640 10 client.8094 handle_command_reply: tid=0 2024-02-19T14:56:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 client.8094 shutdown 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 msgr2=0x5592a4bd5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 0x5592a4bd5660 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f2844009c40 tx=0x7f2844004730 comp rx=0 tx=0).stop 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 shutdown_connections 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:6835/2501419582,v1:172.21.15.78:6837/2501419582] conn(0x5592a4bec300 0x5592a4c83220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a4c188b0 0x5592a4bd5660 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5592a4c27430 0x5592a4c82ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2828ff9640 0 client.8094 ms_handle_reset on v2:172.21.15.78:6835/2501419582 2024-02-19T14:56:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 --2- 172.21.15.78:0/3094585480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5592a4c15740 0x5592a4bfb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 >> 172.21.15.78:0/3094585480 conn(0x5592a4bdbae0 msgr2=0x5592a4bd4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 shutdown_connections 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 1 -- 172.21.15.78:0/3094585480 wait complete. 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.490+0000 7f2852bf3740 20 client.8094 trim_cache size 0 max 16384 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 8034, 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:56:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:56:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.9306718629999997, 2024-02-19T14:56:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 110, 2024-02-19T14:56:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 126, 2024-02-19T14:56:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 126, 2024-02-19T14:56:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 7.9190142830000001 2024-02-19T14:56:44.501 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f282c06b840 msgr2=0x7f282c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f282c06b840 0x7f282c06dd00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2844035000 tx=0x7f2844030000 comp rx=0 tx=0).stop 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 msgr2=0x7f2848157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2848157790 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f2838011cd0 tx=0x7f283800f610 comp rx=0 tx=0).stop 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 shutdown_connections 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f282c06b840 0x7f282c06dd00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2848157cd0 0x7f284814b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28480c2320 0x7f284814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 --2- 172.21.15.78:0/3412573762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28340047c0 0x7f2848157790 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 >> 172.21.15.78:0/3412573762 conn(0x7f284806ea10 msgr2=0x7f28480c7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 shutdown_connections 2024-02-19T14:56:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.491+0000 7f2850015640 1 -- 172.21.15.78:0/3412573762 wait complete. 2024-02-19T14:56:44.507 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 110, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:18.671108+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 110, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8034": { "addr": "172.21.15.78:6837/2501419582", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 2501419582, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 2501419582, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8034, "incarnation": 107, "join_fscid": 9, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 126, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-19T14:56:41.574035+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8034 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/4213660424", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 4213660424, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 4213660424, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6239, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/1857207464", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1857207464, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1857207464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7036, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/976486486", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 976486486, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 976486486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 108, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8058, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:56:44.508 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 9, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:56:44.508 INFO:tasks.cephfs.test_failover:current = [{'gid': 8058, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/976486486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 976486486}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 976486486}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 108}, {'gid': 6239, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/4213660424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4213660424}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4213660424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 8034, 'name': 'c', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.78:6837/2501419582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 2501419582}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 2501419582}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7036, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1857207464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1857207464}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1857207464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}] 2024-02-19T14:56:44.508 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-19T14:56:44.509 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:56:44.509 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:44.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:44.590 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:56:44.591 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:56:44.591 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:44.606 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:56:44.606 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:56:44.606 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:44.648 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:56:44.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:44.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.745+0000 7fa4c898c640 1 Processor -- start 2024-02-19T14:56:44.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.745+0000 7fa4c898c640 1 -- start start 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.746+0000 7fa4c898c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 0x7fa4c4156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.746+0000 7fa4c898c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c4155a50 con 0x7fa4c40c1dd0 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.746+0000 7fa4c898c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c4155f80 con 0x7fa4c4071410 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.746+0000 7fa4c898c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4152af0 con 0x7fa4c40c19f0 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.746+0000 7fa4c2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa4c40c1dd0 0x7fa4c40c2200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56982/0 (socket says 172.21.15.78:56982) 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.746+0000 7fa4c2d9d640 1 -- 172.21.15.78:0/183062519 learned_addr learned my addr 172.21.15.78:0/183062519 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166585892 0 0) 0x7fa4c4155a50 con 0x7fa4c40c1dd0 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4ac003520 con 0x7fa4c40c1dd0 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 0x7fa4c4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806296461 0 0) 0x7fa4ac003520 con 0x7fa4c40c1dd0 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 msgr2=0x7fa4c4156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 0x7fa4c4156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 >> v1:172.21.15.150:6789/0 conn(0x7fa4c4071410 legacy=0x7fa4c40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c4070840 con 0x7fa4c40c1dd0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fa4b4001670 con 0x7fa4c40c1dd0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 >> v1:172.21.15.78:6789/0 conn(0x7fa4c40c1dd0 legacy=0x7fa4c40c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4ac0039a0 0x7fa4ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4ac0047c0 0x7fa4ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 0x7fa4ac005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4152af0 con 0x7fa4ac0039a0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac00bb40 con 0x7fa4ac0047c0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac00bd00 con 0x7fa4c40c19f0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4ac0039a0 0x7fa4ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c2d9d640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 0x7fa4ac005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c1d9b640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4ac0047c0 0x7fa4ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 -- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 msgr2=0x7fa4ac005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4c40c19f0 0x7fa4ac005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 -- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4ac0047c0 msgr2=0x7fa4ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4ac0047c0 0x7fa4ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c259c640 1 -- 172.21.15.78:0/183062519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac003520 con 0x7fa4ac0039a0 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.747+0000 7fa4c1d9b640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4ac0047c0 0x7fa4ac004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.748+0000 7fa4c259c640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4ac0039a0 0x7fa4ac003dc0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009ac0 tx=0x7fa4ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7b51f116960c3e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.748+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc02c6b0 con 0x7fa4ac0039a0 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.748+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc02c810 con 0x7fa4ac0039a0 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.748+0000 7fa4c159a640 1 -- 172.21.15.78:0/183062519 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc02cba0 con 0x7fa4ac0039a0 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c898c640 1 -- 172.21.15.78:0/183062519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4ac0039a0 msgr2=0x7fa4ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c898c640 1 --2- 172.21.15.78:0/183062519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4ac0039a0 0x7fa4ac003dc0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009ac0 tx=0x7fa4ac0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c2d9d640 1 -- 172.21.15.78:0/183062519 reap_dead start 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c898c640 1 -- 172.21.15.78:0/183062519 shutdown_connections 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c898c640 1 -- 172.21.15.78:0/183062519 >> 172.21.15.78:0/183062519 conn(0x7fa4c40aa370 msgr2=0x7fa4c40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c898c640 1 -- 172.21.15.78:0/183062519 shutdown_connections 2024-02-19T14:56:44.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.749+0000 7fa4c898c640 1 -- 172.21.15.78:0/183062519 wait complete. 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.750+0000 7fa4c898c640 1 Processor -- start 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.750+0000 7fa4c898c640 1 -- start start 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c898c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4ac0039a0 0x7fa4c41e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c898c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c41520e0 0x7fa4c41eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c898c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 0x7fa4c40c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c898c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c41eae40 con 0x7fa4c41ebb20 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c898c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c41dfa20 con 0x7fa4c41520e0 2024-02-19T14:56:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c898c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c41dfcc0 con 0x7fa4ac0039a0 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 0x7fa4c40c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 0x7fa4c40c4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51458/0 (socket says 172.21.15.78:51458) 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 -- 172.21.15.78:0/155380930 learned_addr learned my addr 172.21.15.78:0/155380930 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c259c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4ac0039a0 0x7fa4c41e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c1d9b640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c41520e0 0x7fa4c41eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 -- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4ac0039a0 msgr2=0x7fa4c41e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4ac0039a0 0x7fa4c41e0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 -- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c41520e0 msgr2=0x7fa4c41eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c41520e0 0x7fa4c41eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.751+0000 7fa4c2d9d640 1 -- 172.21.15.78:0/155380930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c41dffa0 con 0x7fa4c41ebb20 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.752+0000 7fa4c2d9d640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 0x7fa4c40c4780 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4070bc0 tx=0x7fa4b4007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.752+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4007770 con 0x7fa4c41ebb20 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.752+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b400f700 con 0x7fa4c41ebb20 2024-02-19T14:56:44.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.752+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c41eb0c0 con 0x7fa4c41ebb20 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.752+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b400fa90 con 0x7fa4c41ebb20 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.752+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c41de550 con 0x7fa4c41ebb20 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.753+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c41e01c0 con 0x7fa4c41ebb20 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.753+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa4b400fbf0 con 0x7fa4c41ebb20 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.754+0000 7fa4b2ffd640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4a406b810 0x7fa4a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.754+0000 7fa4c259c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4a406b810 0x7fa4a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.754+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa4b408a0d0 con 0x7fa4c41ebb20 2024-02-19T14:56:44.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.754+0000 7fa4c259c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4a406b810 0x7fa4a406dcd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc036000 tx=0x7fa4bc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:44.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.758+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa4b4054960 con 0x7fa4c41ebb20 2024-02-19T14:56:44.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.973+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4c40c8320 con 0x7fa4c41ebb20 2024-02-19T14:56:44.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.974+0000 7fa4b2ffd640 1 -- 172.21.15.78:0/155380930 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4948 (secure 0 0 0) 0x7fa4b400fea0 con 0x7fa4c41ebb20 2024-02-19T14:56:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:44.977 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:41.574035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:44.977 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 110 2024-02-19T14:56:44.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4a406b810 msgr2=0x7fa4a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4a406b810 0x7fa4a406dcd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc036000 tx=0x7fa4bc030000 comp rx=0 tx=0).stop 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 msgr2=0x7fa4c40c4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 0x7fa4c40c4780 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4070bc0 tx=0x7fa4b4007430 comp rx=0 tx=0).stop 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 shutdown_connections 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4a406b810 0x7fa4a406dcd0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4c41ebb20 0x7fa4c40c4780 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c41520e0 0x7fa4c41eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.975+0000 7fa4c898c640 1 --2- 172.21.15.78:0/155380930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4ac0039a0 0x7fa4c41e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.976+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 >> 172.21.15.78:0/155380930 conn(0x7fa4c40aa370 msgr2=0x7fa4c40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.976+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 shutdown_connections 2024-02-19T14:56:44.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:44.976+0000 7fa4c898c640 1 -- 172.21.15.78:0/155380930 wait complete. 2024-02-19T14:56:44.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:45.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.099+0000 7f14bc8ff640 1 Processor -- start 2024-02-19T14:56:45.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.099+0000 7f14bc8ff640 1 -- start start 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14bc8ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14b80c2f50 0x7f14b8158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14bc8ff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b80c34a0 con 0x7f14b80c8500 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14bc8ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b806c8e0 con 0x7f14b80c42e0 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14bc8ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b806c090 con 0x7f14b80c2f50 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f14b80c8500 0x7f14b8158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56986/0 (socket says 172.21.15.78:56986) 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b659c640 1 -- 172.21.15.78:0/3998880206 learned_addr learned my addr 172.21.15.78:0/3998880206 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b6d9d640 1 --2- 172.21.15.78:0/3998880206 >> v2:172.21.15.150:3300/0 conn(0x7f14b80c2f50 0x7f14b8158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b559a640 1 -- 172.21.15.78:0/3998880206 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076008050 0 0) 0x7f14b80c34a0 con 0x7f14b80c8500 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b559a640 1 -- 172.21.15.78:0/3998880206 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a0003520 con 0x7f14b80c8500 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b559a640 1 -- 172.21.15.78:0/3998880206 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392093551 0 0) 0x7f14a0003520 con 0x7f14b80c8500 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b6d9d640 1 -- 172.21.15.78:0/3998880206 >> v1:172.21.15.150:6789/0 conn(0x7f14b80c42e0 legacy=0x7f14b815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b6d9d640 1 -- 172.21.15.78:0/3998880206 >> v1:172.21.15.78:6789/0 conn(0x7f14b80c8500 legacy=0x7f14b8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.100+0000 7f14b6d9d640 1 -- 172.21.15.78:0/3998880206 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b806cbc0 con 0x7f14b80c2f50 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.101+0000 7f14b6d9d640 1 --2- 172.21.15.78:0/3998880206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c2f50 0x7f14b8158d60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009d90 tx=0x7f14b002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5abcdc40df4e5176 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.101+0000 7f14b559a640 1 -- 172.21.15.78:0/3998880206 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8001670 con 0x7f14b80c2f50 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.101+0000 7f14b559a640 1 -- 172.21.15.78:0/3998880206 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b002c8b0 con 0x7f14b80c2f50 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.101+0000 7f14b559a640 1 -- 172.21.15.78:0/3998880206 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b002cc40 con 0x7f14b80c2f50 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 -- 172.21.15.78:0/3998880206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c2f50 msgr2=0x7f14b8158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/3998880206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c2f50 0x7f14b8158d60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009d90 tx=0x7f14b002bd80 comp rx=0 tx=0).stop 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 -- 172.21.15.78:0/3998880206 shutdown_connections 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/3998880206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c2f50 0x7f14b8158d60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 -- 172.21.15.78:0/3998880206 >> 172.21.15.78:0/3998880206 conn(0x7f14b80aa300 msgr2=0x7f14b80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 -- 172.21.15.78:0/3998880206 shutdown_connections 2024-02-19T14:56:45.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.102+0000 7f14bc8ff640 1 -- 172.21.15.78:0/3998880206 wait complete. 2024-02-19T14:56:45.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.103+0000 7f14bc8ff640 1 Processor -- start 2024-02-19T14:56:45.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.103+0000 7f14bc8ff640 1 -- start start 2024-02-19T14:56:45.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14b80c2f50 0x7f14b81e0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c8500 0x7f14b8156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 0x7f14b8070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b806f680 con 0x7f14b8158a50 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b8158330 con 0x7f14b80c8500 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b81eb6a0 con 0x7f14b80c2f50 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 0x7f14b8070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 0x7f14b8070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51464/0 (socket says 172.21.15.78:51464) 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 -- 172.21.15.78:0/2592345360 learned_addr learned my addr 172.21.15.78:0/2592345360 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b5d9b640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c8500 0x7f14b8156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b659c640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14b80c2f50 0x7f14b81e0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 -- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14b80c2f50 msgr2=0x7f14b81e0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14b80c2f50 0x7f14b81e0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 -- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c8500 msgr2=0x7f14b8156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c8500 0x7f14b8156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 -- 172.21.15.78:0/2592345360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b0009a70 con 0x7f14b8158a50 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b659c640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14b80c2f50 0x7f14b81e0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14b6d9d640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 0x7f14b8070020 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009d90 tx=0x7f14b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b002ce80 con 0x7f14b8158a50 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0009450 con 0x7f14b8158a50 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b81eb980 con 0x7f14b8158a50 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.104+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0003bb0 con 0x7f14b8158a50 2024-02-19T14:56:45.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.105+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b806fbb0 con 0x7f14b8158a50 2024-02-19T14:56:45.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.105+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b8071c20 con 0x7f14b8158a50 2024-02-19T14:56:45.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.106+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f14b003b050 con 0x7f14b8158a50 2024-02-19T14:56:45.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.106+0000 7f14a6ffd640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f149806b810 0x7f149806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.106+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f14b00ac700 con 0x7f14b8158a50 2024-02-19T14:56:45.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.106+0000 7f14b659c640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f149806b810 0x7f149806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.107+0000 7f14b659c640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f149806b810 0x7f149806dcd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f14a8005880 tx=0x7f14a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.115+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f14b0077140 con 0x7f14b8158a50 2024-02-19T14:56:45.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.292+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14b8074990 con 0x7f14b8158a50 2024-02-19T14:56:45.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.294+0000 7f14a6ffd640 1 -- 172.21.15.78:0/2592345360 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17145 (secure 0 0 0) 0x7f14b007fe60 con 0x7f14b8158a50 2024-02-19T14:56:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:45.297 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:39.558539+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:16.897563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:17.646704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6834/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6837/100250519":"2024-02-20T14:56:36.537604+0000","172.21.15.78:6835/100250519":"2024-02-20T14:56:36.537604+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:45.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.295+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f149806b810 msgr2=0x7f149806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.295+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f149806b810 0x7f149806dcd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f14a8005880 tx=0x7f14a8009040 comp rx=0 tx=0).stop 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.295+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 msgr2=0x7f14b8070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.295+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 0x7f14b8070020 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009d90 tx=0x7f14b0004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 shutdown_connections 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f149806b810 0x7f149806dcd0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14b8158a50 0x7f14b8070020 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80c8500 0x7f14b8156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 --2- 172.21.15.78:0/2592345360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14b80c2f50 0x7f14b81e0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 >> 172.21.15.78:0/2592345360 conn(0x7f14b80aa300 msgr2=0x7f14b80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 shutdown_connections 2024-02-19T14:56:45.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.296+0000 7f14bc8ff640 1 -- 172.21.15.78:0/2592345360 wait complete. 2024-02-19T14:56:45.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:56:16.897563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '123', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:56:17.646704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:56:45.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:56:45.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.410+0000 7f1159d8c640 1 Processor -- start 2024-02-19T14:56:45.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.410+0000 7f1159d8c640 1 -- start start 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1159d8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f11540719d0 0x7f1154071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1159d8c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11540c1120 con 0x7f11540c15b0 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1159d8c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11540c6080 con 0x7f115414ece0 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1159d8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11540c57e0 con 0x7f11540719d0 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1153fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f11540c15b0 0x7f11540c1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56988/0 (socket says 172.21.15.78:56988) 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1153fff640 1 -- 172.21.15.78:0/3831872677 learned_addr learned my addr 172.21.15.78:0/3831872677 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:45.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036817117 0 0) 0x7f11540c1120 con 0x7f11540c15b0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1152ffd640 1 --2- 172.21.15.78:0/3831872677 >> v2:172.21.15.150:3300/0 conn(0x7f11540719d0 0x7f1154071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c003520 con 0x7f11540c15b0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1546627076 0 0) 0x7f113c003520 con 0x7f11540c15b0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 >> v2:172.21.15.150:3300/0 conn(0x7f11540719d0 msgr2=0x7f1154071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 --2- 172.21.15.78:0/3831872677 >> v2:172.21.15.150:3300/0 conn(0x7f11540719d0 0x7f1154071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 >> v1:172.21.15.150:6789/0 conn(0x7f115414ece0 legacy=0x7f11540c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11540c5ac0 con 0x7f11540c15b0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f114c0019f0 con 0x7f11540c15b0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 >> v1:172.21.15.78:6789/0 conn(0x7f11540c15b0 legacy=0x7f11540c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f113c003970 0x7f113c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f113c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113c005180 0x7f113c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11540c57e0 con 0x7f113c0047c0 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c00c020 con 0x7f113c005180 2024-02-19T14:56:45.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c00c1e0 con 0x7f113c003970 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.411+0000 7f1152ffd640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f113c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f11537fe640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f113c003970 0x7f113c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1153fff640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113c005180 0x7f113c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1152ffd640 1 -- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f113c003970 msgr2=0x7f113c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1152ffd640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f113c003970 0x7f113c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1152ffd640 1 -- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113c005180 msgr2=0x7f113c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1152ffd640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113c005180 0x7f113c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1153fff640 1 -- 172.21.15.78:0/3831872677 reap_dead start 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.412+0000 7f1152ffd640 1 -- 172.21.15.78:0/3831872677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c003520 con 0x7f113c0047c0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1152ffd640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f113c004c40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1148009ac0 tx=0x7f113c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f67a708fe949f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114802c630 con 0x7f113c0047c0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114802c790 con 0x7f113c0047c0 2024-02-19T14:56:45.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f11527fc640 1 -- 172.21.15.78:0/3831872677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114802cb20 con 0x7f113c0047c0 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 -- 172.21.15.78:0/3831872677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 msgr2=0x7f113c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f113c004c40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1148009ac0 tx=0x7f113c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 -- 172.21.15.78:0/3831872677 shutdown_connections 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 --2- 172.21.15.78:0/3831872677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f113c004c40 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 -- 172.21.15.78:0/3831872677 >> 172.21.15.78:0/3831872677 conn(0x7f115406b270 msgr2=0x7f115406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 -- 172.21.15.78:0/3831872677 shutdown_connections 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.413+0000 7f1159d8c640 1 -- 172.21.15.78:0/3831872677 wait complete. 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.414+0000 7f1159d8c640 1 Processor -- start 2024-02-19T14:56:45.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 -- start start 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f11541528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11541556a0 0x7f115414adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f115414b300 0x7f11541517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115414ba30 con 0x7f113c0047c0 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11541540a0 con 0x7f11541556a0 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f1159d8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11541512d0 con 0x7f115414b300 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f11537fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f11541528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f11537fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f11541528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51482/0 (socket says 172.21.15.78:51482) 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.415+0000 7f11537fe640 1 -- 172.21.15.78:0/534626505 learned_addr learned my addr 172.21.15.78:0/534626505 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f11537fe640 1 -- 172.21.15.78:0/534626505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f115414b300 msgr2=0x7f11541517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f11537fe640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f115414b300 0x7f11541517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f11537fe640 1 -- 172.21.15.78:0/534626505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11541556a0 msgr2=0x7f115414adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f11537fe640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11541556a0 0x7f115414adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f11537fe640 1 -- 172.21.15.78:0/534626505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1154155100 con 0x7f113c0047c0 2024-02-19T14:56:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f11537fe640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f11541528c0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f1144011980 tx=0x7f1144011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440054e0 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114400f940 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114400fcd0 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115414bc40 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.416+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115414b7b0 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.417+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1154151440 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.417+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1144018020 con 0x7f113c0047c0 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.418+0000 7f113bfff640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f114006b840 0x7f114006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.418+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f114408cc10 con 0x7f113c0047c0 2024-02-19T14:56:45.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.418+0000 7f1152ffd640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f114006b840 0x7f114006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.418+0000 7f1152ffd640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f114006b840 0x7f114006dd00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1148036000 tx=0x7f114802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.421+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1154151440 con 0x7f113c0047c0 2024-02-19T14:56:45.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.597+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1154151440 con 0x7f113c0047c0 2024-02-19T14:56:45.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.597+0000 7f113bfff640 1 -- 172.21.15.78:0/534626505 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1154151440 con 0x7f113c0047c0 2024-02-19T14:56:45.601 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f114006b840 msgr2=0x7f114006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f114006b840 0x7f114006dd00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1148036000 tx=0x7f114802e000 comp rx=0 tx=0).stop 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 msgr2=0x7f11541528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f11541528c0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f1144011980 tx=0x7f1144011e40 comp rx=0 tx=0).stop 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 shutdown_connections 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f114006b840 0x7f114006dd00 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f115414b300 0x7f11541517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11541556a0 0x7f115414adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 --2- 172.21.15.78:0/534626505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f113c0047c0 0x7f11541528c0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 >> 172.21.15.78:0/534626505 conn(0x7f115406b270 msgr2=0x7f11540a6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:45.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.599+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 shutdown_connections 2024-02-19T14:56:45.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.600+0000 7f1159d8c640 1 -- 172.21.15.78:0/534626505 wait complete. 2024-02-19T14:56:45.614 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:56:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.712+0000 7f84e026b640 1 Processor -- start 2024-02-19T14:56:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.712+0000 7f84e026b640 1 -- start start 2024-02-19T14:56:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84e026b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d806da10 0x7f84d80723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84e026b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84d8072910 0x7f84d8158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84e026b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d80c1d80 con 0x7f84d80acb50 2024-02-19T14:56:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84e026b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d806caf0 con 0x7f84d8072910 2024-02-19T14:56:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84e026b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80c37e0 con 0x7f84d806da10 2024-02-19T14:56:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de808640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f84d8072910 0x7f84d8158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de007640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f84d80acb50 0x7f84d8155900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56994/0 (socket says 172.21.15.78:56994) 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de007640 1 -- 172.21.15.78:0/2644555317 learned_addr learned my addr 172.21.15.78:0/2644555317 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84dd806640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d806da10 0x7f84d80723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de808640 1 -- 172.21.15.78:0/2644555317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d806da10 msgr2=0x7f84d80723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de808640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d806da10 0x7f84d80723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de808640 1 -- 172.21.15.78:0/2644555317 >> v1:172.21.15.78:6789/0 conn(0x7f84d80acb50 legacy=0x7f84d8155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84de808640 1 -- 172.21.15.78:0/2644555317 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d806c5a0 con 0x7f84d8072910 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84dd005640 1 -- 172.21.15.78:0/2644555317 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682883374 0 0) 0x7f84d80c1d80 con 0x7f84d80acb50 2024-02-19T14:56:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.713+0000 7f84dd806640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d806da10 0x7f84d80723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.714+0000 7f84de808640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d8072910 0x7f84d8158670 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f84c80097a0 tx=0x7f84c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b9bbdc48d2dc9f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.714+0000 7f84dd005640 1 -- 172.21.15.78:0/2644555317 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c802e070 con 0x7f84d8072910 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.714+0000 7f84dd005640 1 -- 172.21.15.78:0/2644555317 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c80315a0 con 0x7f84d8072910 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.714+0000 7f84dd005640 1 -- 172.21.15.78:0/2644555317 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c8031930 con 0x7f84d8072910 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 -- 172.21.15.78:0/2644555317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d8072910 msgr2=0x7f84d8158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d8072910 0x7f84d8158670 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f84c80097a0 tx=0x7f84c802b840 comp rx=0 tx=0).stop 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 -- 172.21.15.78:0/2644555317 shutdown_connections 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d8072910 0x7f84d8158670 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 --2- 172.21.15.78:0/2644555317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d806da10 0x7f84d80723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 -- 172.21.15.78:0/2644555317 >> 172.21.15.78:0/2644555317 conn(0x7f84d80718e0 msgr2=0x7f84d8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 -- 172.21.15.78:0/2644555317 shutdown_connections 2024-02-19T14:56:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.715+0000 7f84e026b640 1 -- 172.21.15.78:0/2644555317 wait complete. 2024-02-19T14:56:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.716+0000 7f84e026b640 1 Processor -- start 2024-02-19T14:56:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.716+0000 7f84e026b640 1 -- start start 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 0x7f84d81494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d8072910 0x7f84d814beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84d80acb50 0x7f84d8156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8149a30 con 0x7f84d806da10 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8151de0 con 0x7f84d80acb50 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8157c80 con 0x7f84d8072910 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 0x7f84d81494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 0x7f84d81494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51498/0 (socket says 172.21.15.78:51498) 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 -- 172.21.15.78:0/1205412261 learned_addr learned my addr 172.21.15.78:0/1205412261 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 -- 172.21.15.78:0/1205412261 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d8072910 msgr2=0x7f84d814beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d8072910 0x7f84d814beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 -- 172.21.15.78:0/1205412261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84d80acb50 msgr2=0x7f84d8156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84d80acb50 0x7f84d8156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 -- 172.21.15.78:0/1205412261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c8009400 con 0x7f84d806da10 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84de007640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 0x7f84d81494f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f84d000c430 tx=0x7f84d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00074e0 con 0x7f84d806da10 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0007660 con 0x7f84d806da10 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0004a20 con 0x7f84d806da10 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.717+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d80abd70 con 0x7f84d806da10 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.718+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d80abfe0 con 0x7f84d806da10 2024-02-19T14:56:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.719+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d8074990 con 0x7f84d806da10 2024-02-19T14:56:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.719+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f84d0004b80 con 0x7f84d806da10 2024-02-19T14:56:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.719+0000 7f84d6ffd640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84c406b810 0x7f84c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.719+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f84d008a180 con 0x7f84d806da10 2024-02-19T14:56:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.719+0000 7f84dd806640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84c406b810 0x7f84c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.720+0000 7f84dd806640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84c406b810 0x7f84c406dcd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f84d814e870 tx=0x7f84cc0083d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:45.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.724+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f84d00549f0 con 0x7f84d806da10 2024-02-19T14:56:45.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.901+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84d806c2e0 con 0x7f84d806da10 2024-02-19T14:56:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.901+0000 7f84d6ffd640 1 -- 172.21.15.78:0/1205412261 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84d00580b0 con 0x7f84d806da10 2024-02-19T14:56:45.905 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84c406b810 msgr2=0x7f84c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84c406b810 0x7f84c406dcd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f84d814e870 tx=0x7f84cc0083d0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 msgr2=0x7f84d81494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 0x7f84d81494f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f84d000c430 tx=0x7f84d000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 shutdown_connections 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f84c406b810 0x7f84c406dcd0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84d80acb50 0x7f84d8156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84d8072910 0x7f84d814beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 --2- 172.21.15.78:0/1205412261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84d806da10 0x7f84d81494f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 >> 172.21.15.78:0/1205412261 conn(0x7f84d80718e0 msgr2=0x7f84d8153120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:45.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 shutdown_connections 2024-02-19T14:56:45.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:45.904+0000 7f84e026b640 1 -- 172.21.15.78:0/1205412261 wait complete. 2024-02-19T14:56:45.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:56:46.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.018+0000 7fe7dcbc9640 1 Processor -- start 2024-02-19T14:56:46.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.018+0000 7fe7dcbc9640 1 -- start start 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7dcbc9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7dcbc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806ff90 con 0x7fe7d80719e0 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7dcbc9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8156fc0 con 0x7fe7d814d210 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7dcbc9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d8070270 con 0x7fe7d80c3d90 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7d6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7d6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51510/0 (socket says 172.21.15.78:51510) 2024-02-19T14:56:46.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.019+0000 7fe7d6d9d640 1 -- 172.21.15.78:0/1390593687 learned_addr learned my addr 172.21.15.78:0/1390593687 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.020+0000 7fe7d6d9d640 1 -- 172.21.15.78:0/1390593687 >> v1:172.21.15.150:6789/0 conn(0x7fe7d814d210 legacy=0x7fe7d80c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.020+0000 7fe7d6d9d640 1 -- 172.21.15.78:0/1390593687 >> v1:172.21.15.78:6789/0 conn(0x7fe7d80719e0 legacy=0x7fe7d80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.020+0000 7fe7d6d9d640 1 -- 172.21.15.78:0/1390593687 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d81572a0 con 0x7fe7d80c3d90 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.020+0000 7fe7d559a640 1 -- 172.21.15.78:0/1390593687 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472336794 0 0) 0x7fe7d806ff90 con 0x7fe7d80719e0 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.020+0000 7fe7d6d9d640 1 --2- 172.21.15.78:0/1390593687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d815e800 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009e50 tx=0x7fe7d002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe5235d1955844f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.020+0000 7fe7d559a640 1 -- 172.21.15.78:0/1390593687 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d002cb20 con 0x7fe7d80c3d90 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7d559a640 1 -- 172.21.15.78:0/1390593687 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d002cc80 con 0x7fe7d80c3d90 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7d559a640 1 -- 172.21.15.78:0/1390593687 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00343f0 con 0x7fe7d80c3d90 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/1390593687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 msgr2=0x7fe7d815e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/1390593687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d815e800 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009e50 tx=0x7fe7d002bf80 comp rx=0 tx=0).stop 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/1390593687 shutdown_connections 2024-02-19T14:56:46.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/1390593687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80c3d90 0x7fe7d815e800 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/1390593687 >> 172.21.15.78:0/1390593687 conn(0x7fe7d80c5050 msgr2=0x7fe7d80c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/1390593687 shutdown_connections 2024-02-19T14:56:46.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.021+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/1390593687 wait complete. 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.022+0000 7fe7dcbc9640 1 Processor -- start 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.022+0000 7fe7dcbc9640 1 -- start start 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7dcbc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 0x7fe7d8144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7dcbc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d81452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7dcbc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7dcbc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80c3970 con 0x7fe7d80719e0 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7dcbc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d8142a30 con 0x7fe7d80c3d90 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7dcbc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d81e92d0 con 0x7fe7d814d210 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 0x7fe7d8144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 0x7fe7d8144db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51516/0 (socket says 172.21.15.78:51516) 2024-02-19T14:56:46.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 -- 172.21.15.78:0/3524446349 learned_addr learned my addr 172.21.15.78:0/3524446349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d5d9b640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d81452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 -- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 msgr2=0x7fe7d80ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d6d9d640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d80ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 -- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 msgr2=0x7fe7d81452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.023+0000 7fe7d659c640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d81452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7d659c640 1 -- 172.21.15.78:0/3524446349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d0009b30 con 0x7fe7d80719e0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7d5d9b640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d81452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7d659c640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 0x7fe7d8144db0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c400 tx=0x7fe7cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0052d0 con 0x7fe7d80719e0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc005430 con 0x7fe7d80719e0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d81e95b0 con 0x7fe7d80719e0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc003830 con 0x7fe7d80719e0 2024-02-19T14:56:46.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.024+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d8149c30 con 0x7fe7d80719e0 2024-02-19T14:56:46.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.025+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d8074990 con 0x7fe7d80719e0 2024-02-19T14:56:46.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.025+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe7cc003990 con 0x7fe7d80719e0 2024-02-19T14:56:46.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.026+0000 7fe7c6ffd640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7b806b810 0x7fe7b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.026+0000 7fe7d5d9b640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7b806b810 0x7fe7b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.026+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe7cc0899a0 con 0x7fe7d80719e0 2024-02-19T14:56:46.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.026+0000 7fe7d5d9b640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7b806b810 0x7fe7b806dcd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe7d814c780 tx=0x7fe7c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.030+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe7cc054180 con 0x7fe7d80719e0 2024-02-19T14:56:46.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.206+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7d80c3100 con 0x7fe7d80719e0 2024-02-19T14:56:46.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.207+0000 7fe7c6ffd640 1 -- 172.21.15.78:0/3524446349 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe7cc017020 con 0x7fe7d80719e0 2024-02-19T14:56:46.211 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7b806b810 msgr2=0x7fe7b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7b806b810 0x7fe7b806dcd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe7d814c780 tx=0x7fe7c8009040 comp rx=0 tx=0).stop 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 msgr2=0x7fe7d8144db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 0x7fe7d8144db0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c400 tx=0x7fe7cc00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 shutdown_connections 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe7b806b810 0x7fe7b806dcd0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d814d210 0x7fe7d80ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c3d90 0x7fe7d81452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 --2- 172.21.15.78:0/3524446349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7d80719e0 0x7fe7d8144db0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 >> 172.21.15.78:0/3524446349 conn(0x7fe7d80c5050 msgr2=0x7fe7d80c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 shutdown_connections 2024-02-19T14:56:46.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.209+0000 7fe7dcbc9640 1 -- 172.21.15.78:0/3524446349 wait complete. 2024-02-19T14:56:46.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:46.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.324+0000 7fda36b6c640 1 Processor -- start 2024-02-19T14:56:46.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.325+0000 7fda36b6c640 1 -- start start 2024-02-19T14:56:46.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda36b6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda30152700 0x7fda300c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda36b6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda300c5730 0x7fda3006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda36b6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda300c8e00 0x7fda3006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda36b6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3006cd20 con 0x7fda300c5730 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda36b6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda301551f0 con 0x7fda300c8e00 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda36b6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda30155490 con 0x7fda30152700 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda300c5730 0x7fda3006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fda300c5730 0x7fda3006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51530/0 (socket says 172.21.15.78:51530) 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 -- 172.21.15.78:0/1585262516 learned_addr learned my addr 172.21.15.78:0/1585262516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda34908640 1 --2- 172.21.15.78:0/1585262516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda30152700 0x7fda300c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 -- 172.21.15.78:0/1585262516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda30152700 msgr2=0x7fda300c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 --2- 172.21.15.78:0/1585262516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda30152700 0x7fda300c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 -- 172.21.15.78:0/1585262516 >> v2:172.21.15.150:3300/0 conn(0x7fda300c8e00 msgr2=0x7fda3006c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda35109640 1 --2- 172.21.15.78:0/1585262516 >> v2:172.21.15.150:3300/0 conn(0x7fda300c8e00 0x7fda3006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 --2- 172.21.15.78:0/1585262516 >> v2:172.21.15.150:3300/0 conn(0x7fda300c8e00 0x7fda3006c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda2ffff640 1 -- 172.21.15.78:0/1585262516 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30155770 con 0x7fda300c5730 2024-02-19T14:56:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.326+0000 7fda35109640 1 --2- 172.21.15.78:0/1585262516 >> v2:172.21.15.150:3300/0 conn(0x7fda300c8e00 0x7fda3006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.327+0000 7fda2ffff640 1 --2- 172.21.15.78:0/1585262516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c5730 0x7fda3006c2a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fda300c8bc0 tx=0x7fda1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6789dc6179a3586 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.327+0000 7fda2f7fe640 1 -- 172.21.15.78:0/1585262516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c031870 con 0x7fda300c5730 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.327+0000 7fda2f7fe640 1 -- 172.21.15.78:0/1585262516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c0319d0 con 0x7fda300c5730 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.327+0000 7fda2f7fe640 1 -- 172.21.15.78:0/1585262516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c031d60 con 0x7fda300c5730 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 -- 172.21.15.78:0/1585262516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c5730 msgr2=0x7fda3006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 --2- 172.21.15.78:0/1585262516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c5730 0x7fda3006c2a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fda300c8bc0 tx=0x7fda1c02b910 comp rx=0 tx=0).stop 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 -- 172.21.15.78:0/1585262516 shutdown_connections 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 --2- 172.21.15.78:0/1585262516 >> v2:172.21.15.150:3300/0 conn(0x7fda300c8e00 0x7fda3006c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 --2- 172.21.15.78:0/1585262516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c5730 0x7fda3006c2a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 --2- 172.21.15.78:0/1585262516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda30152700 0x7fda300c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 -- 172.21.15.78:0/1585262516 >> 172.21.15.78:0/1585262516 conn(0x7fda300cac90 msgr2=0x7fda300cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 -- 172.21.15.78:0/1585262516 shutdown_connections 2024-02-19T14:56:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.328+0000 7fda36b6c640 1 -- 172.21.15.78:0/1585262516 wait complete. 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.329+0000 7fda36b6c640 1 Processor -- start 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.329+0000 7fda36b6c640 1 -- start start 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda300c5730 0x7fda301548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 0x7fda301e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda30152700 0x7fda301e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300c2300 con 0x7fda300c8e00 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda301edd00 con 0x7fda30152700 2024-02-19T14:56:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda301edfe0 con 0x7fda300c5730 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 0x7fda301e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 0x7fda301e7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51540/0 (socket says 172.21.15.78:51540) 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 -- 172.21.15.78:0/3408946702 learned_addr learned my addr 172.21.15.78:0/3408946702 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda34908640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda300c5730 0x7fda301548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda35109640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda30152700 0x7fda301e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 -- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda300c5730 msgr2=0x7fda301548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda300c5730 0x7fda301548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 -- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda30152700 msgr2=0x7fda301e7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda30152700 0x7fda301e7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 -- 172.21.15.78:0/3408946702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c0094d0 con 0x7fda300c8e00 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2ffff640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 0x7fda301e7920 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000c00 tx=0x7fda1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c03ad10 con 0x7fda300c8e00 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c03ae70 con 0x7fda300c8e00 2024-02-19T14:56:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda301ee2c0 con 0x7fda300c8e00 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c044610 con 0x7fda300c8e00 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.330+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda301ee880 con 0x7fda300c8e00 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.331+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda301ea800 con 0x7fda300c8e00 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.332+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fda1c04d450 con 0x7fda300c8e00 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.332+0000 7fda2d7fa640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda1406b810 0x7fda1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.332+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fda1c02d040 con 0x7fda300c8e00 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.332+0000 7fda34908640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda1406b810 0x7fda1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.333+0000 7fda34908640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda1406b810 0x7fda1406dcd0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.337+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda301ea800 con 0x7fda300c8e00 2024-02-19T14:56:46.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.551+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda30074990 con 0x7fda300c8e00 2024-02-19T14:56:46.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.551+0000 7fda2d7fa640 1 -- 172.21.15.78:0/3408946702 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4948 (secure 0 0 0) 0x7fda1c077370 con 0x7fda300c8e00 2024-02-19T14:56:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:46.555 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:41.574035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:46.555 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 110 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda1406b810 msgr2=0x7fda1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda1406b810 0x7fda1406dcd0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24008040 comp rx=0 tx=0).stop 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 msgr2=0x7fda301e7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 0x7fda301e7920 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000c00 tx=0x7fda1c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 shutdown_connections 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fda1406b810 0x7fda1406dcd0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda300c8e00 0x7fda301e7920 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda30152700 0x7fda301e7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 --2- 172.21.15.78:0/3408946702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda300c5730 0x7fda301548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 >> 172.21.15.78:0/3408946702 conn(0x7fda300cac90 msgr2=0x7fda30150a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 shutdown_connections 2024-02-19T14:56:46.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.553+0000 7fda36b6c640 1 -- 172.21.15.78:0/3408946702 wait complete. 2024-02-19T14:56:46.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:46.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.669+0000 7fdb3fa09640 1 Processor -- start 2024-02-19T14:56:46.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.669+0000 7fdb3fa09640 1 -- start start 2024-02-19T14:56:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3fa09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb38150320 0x7fdb38152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3fa09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb38071850 0x7fdb381534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3fa09640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb38156e90 con 0x7fdb38153a20 2024-02-19T14:56:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3fa09640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb380c2fc0 con 0x7fdb38071850 2024-02-19T14:56:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3fa09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb38157150 con 0x7fdb38150320 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3cfa4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb38071850 0x7fdb381534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3cfa4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb38071850 0x7fdb381534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51554/0 (socket says 172.21.15.78:51554) 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.670+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/1945731702 learned_addr learned my addr 172.21.15.78:0/1945731702 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/1945731702 >> v2:172.21.15.150:3300/0 conn(0x7fdb38150320 msgr2=0x7fdb38152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb3cfa4640 1 --2- 172.21.15.78:0/1945731702 >> v2:172.21.15.150:3300/0 conn(0x7fdb38150320 0x7fdb38152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/1945731702 >> v1:172.21.15.150:6789/0 conn(0x7fdb38153a20 legacy=0x7fdb380c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/1945731702 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380c3280 con 0x7fdb38071850 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb3cfa4640 1 --2- 172.21.15.78:0/1945731702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38071850 0x7fdb381534e0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fdb380aaed0 tx=0x7fdb2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95bdbdbb60f22fd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb2ffff640 1 -- 172.21.15.78:0/1945731702 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28031870 con 0x7fdb38071850 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb2ffff640 1 -- 172.21.15.78:0/1945731702 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb280319d0 con 0x7fdb38071850 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.671+0000 7fdb2ffff640 1 -- 172.21.15.78:0/1945731702 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28031d60 con 0x7fdb38071850 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 -- 172.21.15.78:0/1945731702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38071850 msgr2=0x7fdb381534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/1945731702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38071850 0x7fdb381534e0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fdb380aaed0 tx=0x7fdb2802b910 comp rx=0 tx=0).stop 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 -- 172.21.15.78:0/1945731702 shutdown_connections 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/1945731702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38071850 0x7fdb381534e0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/1945731702 >> v2:172.21.15.150:3300/0 conn(0x7fdb38150320 0x7fdb38152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 -- 172.21.15.78:0/1945731702 >> 172.21.15.78:0/1945731702 conn(0x7fdb380aa370 msgr2=0x7fdb380aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 -- 172.21.15.78:0/1945731702 shutdown_connections 2024-02-19T14:56:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.672+0000 7fdb3fa09640 1 -- 172.21.15.78:0/1945731702 wait complete. 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.673+0000 7fdb3fa09640 1 Processor -- start 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.673+0000 7fdb3fa09640 1 -- start start 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3fa09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb38071850 0x7fdb381518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3fa09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 0x7fdb381551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3fa09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb38153a20 0x7fdb38155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3fa09640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380ade70 con 0x7fdb38150320 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3fa09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38150db0 con 0x7fdb38153a20 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3fa09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38147110 con 0x7fdb38071850 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 0x7fdb381551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 0x7fdb381551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51570/0 (socket says 172.21.15.78:51570) 2024-02-19T14:56:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/3778236529 learned_addr learned my addr 172.21.15.78:0/3778236529 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3dfa6640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb38153a20 0x7fdb38155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3d7a5640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb38071850 0x7fdb381518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb38071850 msgr2=0x7fdb381518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb38071850 0x7fdb381518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb38153a20 msgr2=0x7fdb38155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb38153a20 0x7fdb38155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 -- 172.21.15.78:0/3778236529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280094d0 con 0x7fdb38150320 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb3cfa4640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 0x7fdb381551f0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fdb28000c00 tx=0x7fdb280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.674+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2803ad10 con 0x7fdb38150320 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.675+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2803ae70 con 0x7fdb38150320 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.675+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28044610 con 0x7fdb38150320 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.675+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb380ae350 con 0x7fdb38150320 2024-02-19T14:56:46.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.675+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb381ec940 con 0x7fdb38150320 2024-02-19T14:56:46.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.676+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3814a7c0 con 0x7fdb38150320 2024-02-19T14:56:46.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.676+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdb2804d450 con 0x7fdb38150320 2024-02-19T14:56:46.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.677+0000 7fdb2dffb640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb1c06b810 0x7fdb1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.677+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fdb2802d040 con 0x7fdb38150320 2024-02-19T14:56:46.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.677+0000 7fdb3d7a5640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb1c06b810 0x7fdb1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.677+0000 7fdb3d7a5640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb1c06b810 0x7fdb1c06dcd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdb38068390 tx=0x7fdb34009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.681+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdb28077210 con 0x7fdb38150320 2024-02-19T14:56:46.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.862+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb38074990 con 0x7fdb38150320 2024-02-19T14:56:46.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.863+0000 7fdb2dffb640 1 -- 172.21.15.78:0/3778236529 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17145 (secure 0 0 0) 0x7fdb2807a8d0 con 0x7fdb38150320 2024-02-19T14:56:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:46.867 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:39.558539+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:16.897563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:17.646704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6834/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6837/100250519":"2024-02-20T14:56:36.537604+0000","172.21.15.78:6835/100250519":"2024-02-20T14:56:36.537604+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb1c06b810 msgr2=0x7fdb1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb1c06b810 0x7fdb1c06dcd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdb38068390 tx=0x7fdb34009110 comp rx=0 tx=0).stop 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 msgr2=0x7fdb381551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 0x7fdb381551f0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fdb28000c00 tx=0x7fdb280046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 shutdown_connections 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb1c06b810 0x7fdb1c06dcd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb38150320 0x7fdb381551f0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb38153a20 0x7fdb38155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 --2- 172.21.15.78:0/3778236529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb38071850 0x7fdb381518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 >> 172.21.15.78:0/3778236529 conn(0x7fdb380aa370 msgr2=0x7fdb38070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 shutdown_connections 2024-02-19T14:56:46.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.865+0000 7fdb3fa09640 1 -- 172.21.15.78:0/3778236529 wait complete. 2024-02-19T14:56:46.880 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:56:46.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:46.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.977+0000 7f26e6091640 1 Processor -- start 2024-02-19T14:56:46.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.977+0000 7f26e6091640 1 -- start start 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26e6091640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26e00c5560 0x7f26e00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26e6091640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c7910 0x7f26e0154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26e6091640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26e0070720 0x7f26e0070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26e6091640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00c65a0 con 0x7f26e0070720 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26e6091640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00c7530 con 0x7f26e00c5560 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26e6091640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01537b0 con 0x7f26e00c7910 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26e0070720 0x7f26e0070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26e0070720 0x7f26e0070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51578/0 (socket says 172.21.15.78:51578) 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 -- 172.21.15.78:0/1613829715 learned_addr learned my addr 172.21.15.78:0/1613829715 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26deffd640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c7910 0x7f26e0154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26df7fe640 1 --2- 172.21.15.78:0/1613829715 >> v2:172.21.15.150:3300/0 conn(0x7f26e00c5560 0x7f26e00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 -- 172.21.15.78:0/1613829715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c7910 msgr2=0x7f26e0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c7910 0x7f26e0154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 -- 172.21.15.78:0/1613829715 >> v2:172.21.15.150:3300/0 conn(0x7f26e00c5560 msgr2=0x7f26e00c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 --2- 172.21.15.78:0/1613829715 >> v2:172.21.15.150:3300/0 conn(0x7f26e00c5560 0x7f26e00c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26dffff640 1 -- 172.21.15.78:0/1613829715 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0153a90 con 0x7f26e0070720 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26deffd640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c7910 0x7f26e0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.978+0000 7f26df7fe640 1 --2- 172.21.15.78:0/1613829715 >> v2:172.21.15.150:3300/0 conn(0x7f26e00c5560 0x7f26e00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.979+0000 7f26dffff640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e0070720 0x7f26e0070b70 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f26e006bd60 tx=0x7f26d802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66dd26b7e6bf0af2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.979+0000 7f26de7fc640 1 -- 172.21.15.78:0/1613829715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d802c950 con 0x7f26e0070720 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.979+0000 7f26de7fc640 1 -- 172.21.15.78:0/1613829715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d802cab0 con 0x7f26e0070720 2024-02-19T14:56:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.979+0000 7f26de7fc640 1 -- 172.21.15.78:0/1613829715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d802ce40 con 0x7f26e0070720 2024-02-19T14:56:46.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 -- 172.21.15.78:0/1613829715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e0070720 msgr2=0x7f26e0070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e0070720 0x7f26e0070b70 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f26e006bd60 tx=0x7f26d802bdb0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 -- 172.21.15.78:0/1613829715 shutdown_connections 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e0070720 0x7f26e0070b70 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 --2- 172.21.15.78:0/1613829715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c7910 0x7f26e0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 --2- 172.21.15.78:0/1613829715 >> v2:172.21.15.150:3300/0 conn(0x7f26e00c5560 0x7f26e00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 -- 172.21.15.78:0/1613829715 >> 172.21.15.78:0/1613829715 conn(0x7f26e006b4b0 msgr2=0x7f26e00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 -- 172.21.15.78:0/1613829715 shutdown_connections 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.980+0000 7f26e6091640 1 -- 172.21.15.78:0/1613829715 wait complete. 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.981+0000 7f26e6091640 1 Processor -- start 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.981+0000 7f26e6091640 1 -- start start 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26e6091640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e0070720 0x7f26e01df6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26e6091640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c5560 0x7f26e01dfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26e6091640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 0x7f26e00a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26e6091640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e014d0c0 con 0x7f26e00c7910 2024-02-19T14:56:46.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26e6091640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01e0120 con 0x7f26e0070720 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26e6091640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00aa3b0 con 0x7f26e00c5560 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 0x7f26e00a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 0x7f26e00a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51592/0 (socket says 172.21.15.78:51592) 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 -- 172.21.15.78:0/937191540 learned_addr learned my addr 172.21.15.78:0/937191540 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26deffd640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c5560 0x7f26e01dfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26df7fe640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e0070720 0x7f26e01df6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 -- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c5560 msgr2=0x7f26e01dfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c5560 0x7f26e01dfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 -- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e0070720 msgr2=0x7f26e01df6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e0070720 0x7f26e01df6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 -- 172.21.15.78:0/937191540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d8009b30 con 0x7f26e00c7910 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.982+0000 7f26dffff640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 0x7f26e00a8b90 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f26d8036040 tx=0x7f26d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.983+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8009510 con 0x7f26e00c7910 2024-02-19T14:56:46.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.983+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d803e450 con 0x7f26e00c7910 2024-02-19T14:56:46.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.983+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e014cde0 con 0x7f26e00c7910 2024-02-19T14:56:46.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.983+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d803e800 con 0x7f26e00c7910 2024-02-19T14:56:46.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.983+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e00aa0d0 con 0x7f26e00c7910 2024-02-19T14:56:46.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.984+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e00a9c50 con 0x7f26e00c7910 2024-02-19T14:56:46.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.984+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f26d803e960 con 0x7f26e00c7910 2024-02-19T14:56:46.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.985+0000 7f26d7fff640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c006b840 0x7f26c006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:46.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.985+0000 7f26df7fe640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c006b840 0x7f26c006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:46.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.985+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f26d80ad1d0 con 0x7f26e00c7910 2024-02-19T14:56:46.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.985+0000 7f26df7fe640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c006b840 0x7f26c006dd00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f26e01ecaa0 tx=0x7f26d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:46.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:46.989+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f26e00a9c50 con 0x7f26e00c7910 2024-02-19T14:56:47.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.202+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e00a88b0 con 0x7f26e00c7910 2024-02-19T14:56:47.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.203+0000 7f26d7fff640 1 -- 172.21.15.78:0/937191540 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f26e00a88b0 con 0x7f26e00c7910 2024-02-19T14:56:47.206 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:47.207 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:41.574035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:47.207 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 110 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.204+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c006b840 msgr2=0x7f26c006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c006b840 0x7f26c006dd00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f26e01ecaa0 tx=0x7f26d0008040 comp rx=0 tx=0).stop 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 msgr2=0x7f26e00a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 0x7f26e00a8b90 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f26d8036040 tx=0x7f26d8004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 shutdown_connections 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26c006b840 0x7f26c006dd00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26e00c7910 0x7f26e00a8b90 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26e00c5560 0x7f26e01dfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 --2- 172.21.15.78:0/937191540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e0070720 0x7f26e01df6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 >> 172.21.15.78:0/937191540 conn(0x7f26e006b4b0 msgr2=0x7f26e01550b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 shutdown_connections 2024-02-19T14:56:47.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.205+0000 7f26e6091640 1 -- 172.21.15.78:0/937191540 wait complete. 2024-02-19T14:56:47.220 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:47.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.352+0000 7fc44adc2640 1 Processor -- start 2024-02-19T14:56:47.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.352+0000 7fc44adc2640 1 -- start start 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44adc2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc4440c5560 0x7fc4440c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44adc2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc444070720 0x7fc444070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44adc2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4440c65a0 con 0x7fc4440c7910 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44adc2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4440c7530 con 0x7fc444070720 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44adc2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4441537b0 con 0x7fc4440c5560 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44935f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc444070720 0x7fc444070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc443fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc4440c7910 0x7fc444154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57012/0 (socket says 172.21.15.78:57012) 2024-02-19T14:56:47.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc443fff640 1 -- 172.21.15.78:0/4292119745 learned_addr learned my addr 172.21.15.78:0/4292119745 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc44935f640 1 --2- 172.21.15.78:0/4292119745 >> v2:172.21.15.78:3300/0 conn(0x7fc444070720 0x7fc444070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51602/0 (socket says 172.21.15.78:51602) 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.353+0000 7fc448b5e640 1 --2- 172.21.15.78:0/4292119745 >> v2:172.21.15.150:3300/0 conn(0x7fc4440c5560 0x7fc4440c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc44935f640 1 -- 172.21.15.78:0/4292119745 >> v2:172.21.15.150:3300/0 conn(0x7fc4440c5560 msgr2=0x7fc4440c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc44935f640 1 --2- 172.21.15.78:0/4292119745 >> v2:172.21.15.150:3300/0 conn(0x7fc4440c5560 0x7fc4440c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc44935f640 1 -- 172.21.15.78:0/4292119745 >> v1:172.21.15.78:6789/0 conn(0x7fc4440c7910 legacy=0x7fc444154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc44935f640 1 -- 172.21.15.78:0/4292119745 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444153a90 con 0x7fc444070720 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc4437fe640 1 -- 172.21.15.78:0/4292119745 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041912538 0 0) 0x7fc4440c65a0 con 0x7fc4440c7910 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc448b5e640 1 --2- 172.21.15.78:0/4292119745 >> v2:172.21.15.150:3300/0 conn(0x7fc4440c5560 0x7fc4440c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc44935f640 1 --2- 172.21.15.78:0/4292119745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc444070b70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc44406bd60 tx=0x7fc43c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1f2f07ac16c8ea server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc4437fe640 1 -- 172.21.15.78:0/4292119745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c02cb60 con 0x7fc444070720 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc4437fe640 1 -- 172.21.15.78:0/4292119745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43c02ccc0 con 0x7fc444070720 2024-02-19T14:56:47.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.354+0000 7fc4437fe640 1 -- 172.21.15.78:0/4292119745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c034660 con 0x7fc444070720 2024-02-19T14:56:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 -- 172.21.15.78:0/4292119745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 msgr2=0x7fc444070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 --2- 172.21.15.78:0/4292119745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc444070b70 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc44406bd60 tx=0x7fc43c02bdb0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 -- 172.21.15.78:0/4292119745 shutdown_connections 2024-02-19T14:56:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 --2- 172.21.15.78:0/4292119745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc444070b70 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 --2- 172.21.15.78:0/4292119745 >> v2:172.21.15.150:3300/0 conn(0x7fc4440c5560 0x7fc4440c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 -- 172.21.15.78:0/4292119745 >> 172.21.15.78:0/4292119745 conn(0x7fc44406b4b0 msgr2=0x7fc4440c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 -- 172.21.15.78:0/4292119745 shutdown_connections 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.355+0000 7fc44adc2640 1 -- 172.21.15.78:0/4292119745 wait complete. 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.356+0000 7fc44adc2640 1 Processor -- start 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.356+0000 7fc44adc2640 1 -- start start 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44adc2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc4440be0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44adc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4440c5560 0x7fc4441e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44adc2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4440c7910 0x7fc4440acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44adc2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440be7b0 con 0x7fc444070720 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44adc2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440a81d0 con 0x7fc4440c5560 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44adc2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440a84b0 con 0x7fc4440c7910 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc4440be0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc4440be0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51616/0 (socket says 172.21.15.78:51616) 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 -- 172.21.15.78:0/1420771789 learned_addr learned my addr 172.21.15.78:0/1420771789 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc44935f640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4440c7910 0x7fc4440acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc443fff640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4440c5560 0x7fc4441e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 -- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4440c7910 msgr2=0x7fc4440acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4440c7910 0x7fc4440acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 -- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4440c5560 msgr2=0x7fc4441e8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4440c5560 0x7fc4441e8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc448b5e640 1 -- 172.21.15.78:0/1420771789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c009a70 con 0x7fc444070720 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.357+0000 7fc443fff640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4440c5560 0x7fc4441e8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.358+0000 7fc448b5e640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc4440be0b0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c7f0 tx=0x7fc43800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.358+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438007820 con 0x7fc444070720 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.358+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc438007980 con 0x7fc444070720 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.358+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4440c5ac0 con 0x7fc444070720 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.358+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438007cf0 con 0x7fc444070720 2024-02-19T14:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.358+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4440a8a70 con 0x7fc444070720 2024-02-19T14:56:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.359+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc444074990 con 0x7fc444070720 2024-02-19T14:56:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.359+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc4380113f0 con 0x7fc444070720 2024-02-19T14:56:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.360+0000 7fc4417fa640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42806b810 0x7fc42806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.360+0000 7fc443fff640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42806b810 0x7fc42806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.360+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc43808a650 con 0x7fc444070720 2024-02-19T14:56:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.360+0000 7fc443fff640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42806b810 0x7fc42806dcd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc444156160 tx=0x7fc430007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.364+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc438054f70 con 0x7fc444070720 2024-02-19T14:56:47.557 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:56:47.554+0000 7f020f17e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:56:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.642+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4440c76f0 con 0x7fc444070720 2024-02-19T14:56:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.642+0000 7fc4417fa640 1 -- 172.21.15.78:0/1420771789 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4948 (secure 0 0 0) 0x7fc438058630 con 0x7fc444070720 2024-02-19T14:56:47.646 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:47.646 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":110,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:18.671108+0000","modified":"2024-02-19T14:56:41.574035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8034":{"gid":8034,"name":"c","rank":0,"incarnation":107,"state":"up:active","state_seq":4,"addr":"172.21.15.78:6837/2501419582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2501419582},{"type":"v1","addr":"172.21.15.78:6837","nonce":2501419582}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-19T14:56:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 110 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42806b810 msgr2=0x7fc42806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42806b810 0x7fc42806dcd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc444156160 tx=0x7fc430007ca0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 msgr2=0x7fc4440be0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc4440be0b0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c7f0 tx=0x7fc43800ccb0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 shutdown_connections 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc42806b810 0x7fc42806dcd0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4440c7910 0x7fc4440acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4440c5560 0x7fc4441e8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 --2- 172.21.15.78:0/1420771789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc444070720 0x7fc4440be0b0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 >> 172.21.15.78:0/1420771789 conn(0x7fc44406b4b0 msgr2=0x7fc4441519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 shutdown_connections 2024-02-19T14:56:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.645+0000 7fc44adc2640 1 -- 172.21.15.78:0/1420771789 wait complete. 2024-02-19T14:56:47.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:47.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.761+0000 7f76683f6640 1 Processor -- start 2024-02-19T14:56:47.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.761+0000 7f76683f6640 1 -- start start 2024-02-19T14:56:47.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f76683f6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76600add70 0x7f76600c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f76683f6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600ae670 con 0x7f76600705b0 2024-02-19T14:56:47.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f76683f6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600716b0 con 0x7f76600c2f60 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f76683f6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76600c33d0 con 0x7f76600add70 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f7666192640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76600add70 0x7f76600c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f7666192640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f76600add70 0x7f76600c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34546/0 (socket says 172.21.15.78:34546) 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f7666993640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f76600705b0 0x7f76600709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47846/0 (socket says 172.21.15.78:47846) 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.762+0000 7f7666192640 1 -- 172.21.15.78:0/1998268559 learned_addr learned my addr 172.21.15.78:0/1998268559 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.763+0000 7f7666192640 1 -- 172.21.15.78:0/1998268559 >> v1:172.21.15.150:6789/0 conn(0x7f76600c2f60 legacy=0x7f766006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.763+0000 7f7665190640 1 -- 172.21.15.78:0/1998268559 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269893795 0 0) 0x7f76600ae670 con 0x7f76600705b0 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.763+0000 7f7666192640 1 -- 172.21.15.78:0/1998268559 >> v1:172.21.15.78:6789/0 conn(0x7f76600705b0 legacy=0x7f76600709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.763+0000 7f7666192640 1 -- 172.21.15.78:0/1998268559 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660153780 con 0x7f76600add70 2024-02-19T14:56:47.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.764+0000 7f7666192640 1 --2- 172.21.15.78:0/1998268559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600add70 0x7f76600c4980 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f765c009e50 tx=0x7f765c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1886e4b21e40c91 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.764+0000 7f7665190640 1 -- 172.21.15.78:0/1998268559 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c02ca80 con 0x7f76600add70 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.764+0000 7f7665190640 1 -- 172.21.15.78:0/1998268559 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c02cbe0 con 0x7f76600add70 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.764+0000 7f7665190640 1 -- 172.21.15.78:0/1998268559 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c0343f0 con 0x7f76600add70 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.764+0000 7f76683f6640 1 -- 172.21.15.78:0/1998268559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600add70 msgr2=0x7f76600c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.764+0000 7f76683f6640 1 --2- 172.21.15.78:0/1998268559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600add70 0x7f76600c4980 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f765c009e50 tx=0x7f765c02bee0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.765+0000 7f76683f6640 1 -- 172.21.15.78:0/1998268559 shutdown_connections 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.765+0000 7f76683f6640 1 --2- 172.21.15.78:0/1998268559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600add70 0x7f76600c4980 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.765+0000 7f76683f6640 1 -- 172.21.15.78:0/1998268559 >> 172.21.15.78:0/1998268559 conn(0x7f76600c85e0 msgr2=0x7f76600c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.765+0000 7f76683f6640 1 -- 172.21.15.78:0/1998268559 shutdown_connections 2024-02-19T14:56:47.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.765+0000 7f76683f6640 1 -- 172.21.15.78:0/1998268559 wait complete. 2024-02-19T14:56:47.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 Processor -- start 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 -- start start 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76600705b0 0x7f76601544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76600add70 0x7f7660146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 0x7f7660146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76601474c0 con 0x7f76600c2f60 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76601442b0 con 0x7f76600add70 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f76683f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7660144500 con 0x7f76600705b0 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f7666993640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 0x7f7660146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f7666993640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 0x7f7660146f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34562/0 (socket says 172.21.15.78:34562) 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.766+0000 7f7666993640 1 -- 172.21.15.78:0/142082915 learned_addr learned my addr 172.21.15.78:0/142082915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666192640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76600705b0 0x7f76601544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7665991640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76600add70 0x7f7660146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666993640 1 -- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76600705b0 msgr2=0x7f76601544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666993640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76600705b0 0x7f76601544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666993640 1 -- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76600add70 msgr2=0x7f7660146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666993640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76600add70 0x7f7660146a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666993640 1 -- 172.21.15.78:0/142082915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c009b30 con 0x7f76600c2f60 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7666993640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 0x7f7660146f80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f765800c410 tx=0x7f765800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76580074e0 con 0x7f76600c2f60 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658007640 con 0x7f76600c2f60 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76601eb940 con 0x7f76600c2f60 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658004a20 con 0x7f76600c2f60 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.767+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76601ebf00 con 0x7f76600c2f60 2024-02-19T14:56:47.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.768+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7660074990 con 0x7f76600c2f60 2024-02-19T14:56:47.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.769+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7658004b80 con 0x7f76600c2f60 2024-02-19T14:56:47.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.769+0000 7f7656ffd640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f764406b840 0x7f764406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:47.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.769+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f765808a190 con 0x7f76600c2f60 2024-02-19T14:56:47.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.769+0000 7f7666192640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f764406b840 0x7f764406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:47.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.770+0000 7f7666192640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f764406b840 0x7f764406dd00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f765c035000 tx=0x7f765c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:47.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.773+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f76580584a0 con 0x7f76600c2f60 2024-02-19T14:56:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.948+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7660071490 con 0x7f76600c2f60 2024-02-19T14:56:47.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.949+0000 7f7656ffd640 1 -- 172.21.15.78:0/142082915 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17145 (secure 0 0 0) 0x7f7658001060 con 0x7f76600c2f60 2024-02-19T14:56:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:47.953 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:39.558539+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:16.897563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:17.646704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6834/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6837/100250519":"2024-02-20T14:56:36.537604+0000","172.21.15.78:6835/100250519":"2024-02-20T14:56:36.537604+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f764406b840 msgr2=0x7f764406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f764406b840 0x7f764406dd00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f765c035000 tx=0x7f765c030000 comp rx=0 tx=0).stop 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 msgr2=0x7f7660146f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 0x7f7660146f80 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f765800c410 tx=0x7f765800c8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 shutdown_connections 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f764406b840 0x7f764406dd00 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76600c2f60 0x7f7660146f80 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76600add70 0x7f7660146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 --2- 172.21.15.78:0/142082915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76600705b0 0x7f76601544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 >> 172.21.15.78:0/142082915 conn(0x7f76600c85e0 msgr2=0x7f76600a9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 shutdown_connections 2024-02-19T14:56:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:47.951+0000 7f76683f6640 1 -- 172.21.15.78:0/142082915 wait complete. 2024-02-19T14:56:47.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:56:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.064+0000 7f1157d8a640 1 Processor -- start 2024-02-19T14:56:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.064+0000 7f1157d8a640 1 -- start start 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1157d8a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11500c19f0 0x7f1150156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1157d8a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150155a50 con 0x7f11500c1dd0 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1157d8a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150155f80 con 0x7f1150071410 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1157d8a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1150152af0 con 0x7f11500c19f0 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1155b26640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11500c19f0 0x7f1150156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1156327640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f11500c1dd0 0x7f11500c21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47854/0 (socket says 172.21.15.78:47854) 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1155b26640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11500c19f0 0x7f1150156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34574/0 (socket says 172.21.15.78:34574) 2024-02-19T14:56:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1156327640 1 -- 172.21.15.78:0/3506450185 learned_addr learned my addr 172.21.15.78:0/3506450185 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1155b26640 1 -- 172.21.15.78:0/3506450185 >> v1:172.21.15.150:6789/0 conn(0x7f1150071410 legacy=0x7f11500c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1155b26640 1 -- 172.21.15.78:0/3506450185 >> v1:172.21.15.78:6789/0 conn(0x7f11500c1dd0 legacy=0x7f11500c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1155b26640 1 -- 172.21.15.78:0/3506450185 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150070840 con 0x7f11500c19f0 2024-02-19T14:56:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.065+0000 7f1154b24640 1 -- 172.21.15.78:0/3506450185 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750113009 0 0) 0x7f1150155a50 con 0x7f11500c1dd0 2024-02-19T14:56:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.066+0000 7f1155b26640 1 --2- 172.21.15.78:0/3506450185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11500c19f0 0x7f1150156440 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f11400097a0 tx=0x7f114002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4294b713595192 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.066+0000 7f1154b24640 1 -- 172.21.15.78:0/3506450185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114002f050 con 0x7f11500c19f0 2024-02-19T14:56:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.066+0000 7f1154b24640 1 -- 172.21.15.78:0/3506450185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140031700 con 0x7f11500c19f0 2024-02-19T14:56:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.066+0000 7f1154b24640 1 -- 172.21.15.78:0/3506450185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140031a90 con 0x7f11500c19f0 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 -- 172.21.15.78:0/3506450185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11500c19f0 msgr2=0x7f1150156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 --2- 172.21.15.78:0/3506450185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11500c19f0 0x7f1150156440 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f11400097a0 tx=0x7f114002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 -- 172.21.15.78:0/3506450185 shutdown_connections 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 --2- 172.21.15.78:0/3506450185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11500c19f0 0x7f1150156440 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 -- 172.21.15.78:0/3506450185 >> 172.21.15.78:0/3506450185 conn(0x7f11500aa370 msgr2=0x7f11500aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 -- 172.21.15.78:0/3506450185 shutdown_connections 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.067+0000 7f1157d8a640 1 -- 172.21.15.78:0/3506450185 wait complete. 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.068+0000 7f1157d8a640 1 Processor -- start 2024-02-19T14:56:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.068+0000 7f1157d8a640 1 -- start start 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 0x7f11501e1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11500c19f0 0x7f11500c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11500c1dd0 0x7f11501ec290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11501ebe00 con 0x7f1150071410 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11501eee40 con 0x7f11500c19f0 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11501ee8c0 con 0x7f11500c1dd0 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 0x7f11501e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 0x7f11501e1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34578/0 (socket says 172.21.15.78:34578) 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 -- 172.21.15.78:0/2846274997 learned_addr learned my addr 172.21.15.78:0/2846274997 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 -- 172.21.15.78:0/2846274997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11500c1dd0 msgr2=0x7f11501ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11500c1dd0 0x7f11501ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 -- 172.21.15.78:0/2846274997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11500c19f0 msgr2=0x7f11500c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11500c19f0 0x7f11500c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 -- 172.21.15.78:0/2846274997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1140009400 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1155b26640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 0x7f11501e1960 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f11400097a0 tx=0x7f11400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140033c30 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140033d90 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11501eeb60 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114003d6b0 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.069+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11500c4430 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.070+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11500c8320 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.071+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f114003a040 con 0x7f1150071410 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.071+0000 7f11467fc640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f113406b810 0x7f113406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.071+0000 7f1155325640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f113406b810 0x7f113406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.071+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f11400ac1c0 con 0x7f1150071410 2024-02-19T14:56:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.072+0000 7f1155325640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f113406b810 0x7f113406dcd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f114c006fd0 tx=0x7f114c006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.076+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1140076a30 con 0x7f1150071410 2024-02-19T14:56:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.289+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f11500c45c0 con 0x7f1150071410 2024-02-19T14:56:48.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:56:48.290+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:56:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.291+0000 7f11467fc640 1 -- 172.21.15.78:0/2846274997 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) v1 ==== 161+0+0 (secure 0 0 0) 0x7f114007a0f0 con 0x7f1150071410 2024-02-19T14:56:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f113406b810 msgr2=0x7f113406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f113406b810 0x7f113406dcd0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f114c006fd0 tx=0x7f114c006bc0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 msgr2=0x7f11501e1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 0x7f11501e1960 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f11400097a0 tx=0x7f11400046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 shutdown_connections 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f113406b810 0x7f113406dcd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11500c1dd0 0x7f11501ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11500c19f0 0x7f11500c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 --2- 172.21.15.78:0/2846274997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1150071410 0x7f11501e1960 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 >> 172.21.15.78:0/2846274997 conn(0x7f11500aa370 msgr2=0x7f11500aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 shutdown_connections 2024-02-19T14:56:48.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.294+0000 7f1157d8a640 1 -- 172.21.15.78:0/2846274997 wait complete. 2024-02-19T14:56:48.299 INFO:tasks.ceph.mds.c.smithi078.stderr: -40> 2024-02-19T14:56:47.554+0000 7f020f17e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:56:48.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:56:48.323 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:56:48.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.408+0000 7fd5fe92c640 1 Processor -- start 2024-02-19T14:56:48.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.408+0000 7fd5fe92c640 1 -- start start 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5fe92c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f80c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5fe92c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5f8072ee0 0x7fd5f814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5fe92c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f814f8f0 con 0x7fd5f814ef50 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5fe92c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f80c35c0 con 0x7fd5f8072ee0 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5fe92c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80c38a0 con 0x7fd5f80c78a0 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5fcec9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5f8072ee0 0x7fd5f814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5f77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd5f814ef50 0x7fd5f80727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47874/0 (socket says 172.21.15.78:47874) 2024-02-19T14:56:48.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5f77fe640 1 -- 172.21.15.78:0/2987961033 learned_addr learned my addr 172.21.15.78:0/2987961033 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.409+0000 7fd5f7fff640 1 --2- 172.21.15.78:0/2987961033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f80c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5f6ffd640 1 -- 172.21.15.78:0/2987961033 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487836938 0 0) 0x7fd5f814f8f0 con 0x7fd5f814ef50 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5f6ffd640 1 -- 172.21.15.78:0/2987961033 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e0003520 con 0x7fd5f814ef50 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5fcec9640 1 -- 172.21.15.78:0/2987961033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 msgr2=0x7fd5f80c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5fcec9640 1 --2- 172.21.15.78:0/2987961033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f80c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5fcec9640 1 -- 172.21.15.78:0/2987961033 >> v1:172.21.15.78:6789/0 conn(0x7fd5f814ef50 legacy=0x7fd5f80727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5fcec9640 1 -- 172.21.15.78:0/2987961033 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f80c3b80 con 0x7fd5f8072ee0 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5fcec9640 1 --2- 172.21.15.78:0/2987961033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f814f3b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009d90 tx=0x7fd5f002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f2d1ade37e0615 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.410+0000 7fd5f6ffd640 1 -- 172.21.15.78:0/2987961033 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f002cc50 con 0x7fd5f8072ee0 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5f6ffd640 1 -- 172.21.15.78:0/2987961033 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f002cdb0 con 0x7fd5f8072ee0 2024-02-19T14:56:48.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5f6ffd640 1 -- 172.21.15.78:0/2987961033 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0034660 con 0x7fd5f8072ee0 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 -- 172.21.15.78:0/2987961033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 msgr2=0x7fd5f814f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/2987961033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f814f3b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009d90 tx=0x7fd5f002bea0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 -- 172.21.15.78:0/2987961033 shutdown_connections 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/2987961033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f814f3b0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/2987961033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f80c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 -- 172.21.15.78:0/2987961033 >> 172.21.15.78:0/2987961033 conn(0x7fd5f8066e70 msgr2=0x7fd5f80a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 -- 172.21.15.78:0/2987961033 shutdown_connections 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.411+0000 7fd5fe92c640 1 -- 172.21.15.78:0/2987961033 wait complete. 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.412+0000 7fd5fe92c640 1 Processor -- start 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.412+0000 7fd5fe92c640 1 -- start start 2024-02-19T14:56:48.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5fe92c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f80ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5fe92c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5fe92c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5f814ef50 0x7fd5f814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5fe92c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f814b920 con 0x7fd5f8072ee0 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5fe92c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80bc760 con 0x7fd5f814ef50 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5fe92c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80bdbb0 con 0x7fd5f80c78a0 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f80ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f80ab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34600/0 (socket says 172.21.15.78:34600) 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 -- 172.21.15.78:0/1673648565 learned_addr learned my addr 172.21.15.78:0/1673648565 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 -- 172.21.15.78:0/1673648565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 msgr2=0x7fd5f814aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 -- 172.21.15.78:0/1673648565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5f814ef50 msgr2=0x7fd5f814b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5f814ef50 0x7fd5f814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.413+0000 7fd5f7fff640 1 -- 172.21.15.78:0/1673648565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f0009a70 con 0x7fd5f8072ee0 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.414+0000 7fd5f7fff640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f80ab7a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800e9b0 tx=0x7fd5e800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.414+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8009670 con 0x7fd5f8072ee0 2024-02-19T14:56:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.414+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e800c9a0 con 0x7fd5f8072ee0 2024-02-19T14:56:48.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.414+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f81e6db0 con 0x7fd5f8072ee0 2024-02-19T14:56:48.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.414+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e800cd50 con 0x7fd5f8072ee0 2024-02-19T14:56:48.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.414+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f8150fc0 con 0x7fd5f8072ee0 2024-02-19T14:56:48.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.415+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f8074990 con 0x7fd5f8072ee0 2024-02-19T14:56:48.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.415+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd5e8013470 con 0x7fd5f8072ee0 2024-02-19T14:56:48.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.416+0000 7fd5f4ff9640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5e406b840 0x7fd5e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:48.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.416+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fd5e808a6a0 con 0x7fd5f8072ee0 2024-02-19T14:56:48.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.416+0000 7fd5f77fe640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5e406b840 0x7fd5e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:48.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.416+0000 7fd5f77fe640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5e406b840 0x7fd5e406dd00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000f60 tx=0x7fd5ec0078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:48.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.420+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd5e8054e30 con 0x7fd5f8072ee0 2024-02-19T14:56:48.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:48.627+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd5f814e610 con 0x7fd5f8072ee0 2024-02-19T14:56:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.293+0000 7fd5f4ff9640 1 -- 172.21.15.78:0/1673648565 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v112) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd5e80584f0 con 0x7fd5f8072ee0 2024-02-19T14:56:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5e406b840 msgr2=0x7fd5e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5e406b840 0x7fd5e406dd00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000f60 tx=0x7fd5ec0078f0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 msgr2=0x7fd5f80ab7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f80ab7a0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800e9b0 tx=0x7fd5e800ee70 comp rx=0 tx=0).stop 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 shutdown_connections 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd5e406b840 0x7fd5e406dd00 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5f814ef50 0x7fd5f814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5f80c78a0 0x7fd5f814aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 --2- 172.21.15.78:0/1673648565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5f8072ee0 0x7fd5f80ab7a0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.295+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 >> 172.21.15.78:0/1673648565 conn(0x7fd5f8066e70 msgr2=0x7fd5f814cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.296+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 shutdown_connections 2024-02-19T14:56:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.296+0000 7fd5fe92c640 1 -- 172.21.15.78:0/1673648565 wait complete. 2024-02-19T14:56:49.312 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:56:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.457+0000 7fc1dc5f7640 1 Processor -- start 2024-02-19T14:56:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.457+0000 7fc1dc5f7640 1 -- start start 2024-02-19T14:56:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.458+0000 7fc1dc5f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 0x7fc1d40ab650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.458+0000 7fc1dc5f7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1d40abb90 0x7fc1d406af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.458+0000 7fc1dc5f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40acaf0 con 0x7fc1d41504d0 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.458+0000 7fc1dc5f7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d40aa7d0 con 0x7fc1d40abb90 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.458+0000 7fc1dc5f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40ad0b0 con 0x7fc1d40ac160 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1da393640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc1d41504d0 0x7fc1d41537b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47878/0 (socket says 172.21.15.78:47878) 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1da393640 1 -- 172.21.15.78:0/2416063425 learned_addr learned my addr 172.21.15.78:0/2416063425 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684933561 0 0) 0x7fc1d40acaf0 con 0x7fc1d41504d0 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c0003520 con 0x7fc1d41504d0 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9b92640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 0x7fc1d40ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1dab94640 1 --2- 172.21.15.78:0/2416063425 >> v2:172.21.15.150:3300/0 conn(0x7fc1d40abb90 0x7fc1d406af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 82506698 0 0) 0x7fc1c0003520 con 0x7fc1d41504d0 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 msgr2=0x7fc1d40ab650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 0x7fc1d40ab650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 >> v2:172.21.15.150:3300/0 conn(0x7fc1d40abb90 msgr2=0x7fc1d406af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 --2- 172.21.15.78:0/2416063425 >> v2:172.21.15.150:3300/0 conn(0x7fc1d40abb90 0x7fc1d406af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40acdd0 con 0x7fc1d41504d0 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fc1c4001670 con 0x7fc1d41504d0 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1dab94640 1 --2- 172.21.15.78:0/2416063425 >> v2:172.21.15.150:3300/0 conn(0x7fc1d40abb90 0x7fc1d406af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 >> v1:172.21.15.78:6789/0 conn(0x7fc1d41504d0 legacy=0x7fc1d41537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c0003970 0x7fc1c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 0x7fc1c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0004d00 0x7fc1c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40ad0b0 con 0x7fc1c0003970 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c000b980 con 0x7fc1c0004d00 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c000bac0 con 0x7fc1d40ac160 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1da393640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c0003970 0x7fc1c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1d9b92640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 0x7fc1c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.459+0000 7fc1dab94640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0004d00 0x7fc1c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1da393640 1 -- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 msgr2=0x7fc1c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1da393640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1d40ac160 0x7fc1c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1da393640 1 -- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0004d00 msgr2=0x7fc1c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1da393640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0004d00 0x7fc1c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1da393640 1 -- 172.21.15.78:0/2416063425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0003520 con 0x7fc1c0003970 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1dab94640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0004d00 0x7fc1c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1da393640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c0003970 0x7fc1c0003d90 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c2c0 tx=0x7fc1c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a87b803fe669034 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4005ad0 con 0x7fc1c0003970 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4005c30 con 0x7fc1c0003970 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.460+0000 7fc1d9391640 1 -- 172.21.15.78:0/2416063425 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c402c8b0 con 0x7fc1c0003970 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2416063425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c0003970 msgr2=0x7fc1c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2416063425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c0003970 0x7fc1c0003d90 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c2c0 tx=0x7fc1c00057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dab94640 1 -- 172.21.15.78:0/2416063425 reap_dead start 2024-02-19T14:56:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2416063425 shutdown_connections 2024-02-19T14:56:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2416063425 >> 172.21.15.78:0/2416063425 conn(0x7fc1d40c3ac0 msgr2=0x7fc1d40c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2416063425 shutdown_connections 2024-02-19T14:56:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.461+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2416063425 wait complete. 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.462+0000 7fc1dc5f7640 1 Processor -- start 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.462+0000 7fc1dc5f7640 1 -- start start 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dc5f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c0003970 0x7fc1d40bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dc5f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 0x7fc1d40c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dc5f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d40c1920 0x7fc1d41eba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dc5f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40c0350 con 0x7fc1d4156b50 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dc5f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41571c0 con 0x7fc1d40c1920 2024-02-19T14:56:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dc5f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41eb600 con 0x7fc1c0003970 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 0x7fc1d40c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 0x7fc1d40c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34620/0 (socket says 172.21.15.78:34620) 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 -- 172.21.15.78:0/2913449554 learned_addr learned my addr 172.21.15.78:0/2913449554 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1da393640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c0003970 0x7fc1d40bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dab94640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d40c1920 0x7fc1d41eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 -- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c0003970 msgr2=0x7fc1d40bbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c0003970 0x7fc1d40bbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 -- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d40c1920 msgr2=0x7fc1d41eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d40c1920 0x7fc1d41eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 -- 172.21.15.78:0/2913449554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40c2900 con 0x7fc1d4156b50 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1dab94640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d40c1920 0x7fc1d41eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1d9b92640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 0x7fc1d40c5530 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0011910 tx=0x7fc1d0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.463+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0009590 con 0x7fc1d4156b50 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.464+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d0004240 con 0x7fc1d4156b50 2024-02-19T14:56:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.464+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d40c2060 con 0x7fc1d4156b50 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.464+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d000f880 con 0x7fc1d4156b50 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.464+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d40c2620 con 0x7fc1d4156b50 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.465+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d4074990 con 0x7fc1d4156b50 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.465+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc1d000f9e0 con 0x7fc1d4156b50 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.465+0000 7fc1caffd640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b806b810 0x7fc1b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.465+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fc1d008aa10 con 0x7fc1d4156b50 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.465+0000 7fc1da393640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b806b810 0x7fc1b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.466+0000 7fc1da393640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b806b810 0x7fc1b806dcd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4034000 tx=0x7fc1c4007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.470+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1d00552e0 con 0x7fc1d4156b50 2024-02-19T14:56:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:49.643+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc1d41eb740 con 0x7fc1d4156b50 2024-02-19T14:56:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.603+0000 7fc1caffd640 1 -- 172.21.15.78:0/2913449554 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v128) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc1d00190d0 con 0x7fc1d4156b50 2024-02-19T14:56:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b806b810 msgr2=0x7fc1b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b806b810 0x7fc1b806dcd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4034000 tx=0x7fc1c4007000 comp rx=0 tx=0).stop 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 msgr2=0x7fc1d40c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 0x7fc1d40c5530 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0011910 tx=0x7fc1d0011dd0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 shutdown_connections 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1b806b810 0x7fc1b806dcd0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d4156b50 0x7fc1d40c5530 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1d40c1920 0x7fc1d41eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 --2- 172.21.15.78:0/2913449554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c0003970 0x7fc1d40bbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 >> 172.21.15.78:0/2913449554 conn(0x7fc1d40c3ac0 msgr2=0x7fc1d40c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 shutdown_connections 2024-02-19T14:56:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.606+0000 7fc1dc5f7640 1 -- 172.21.15.78:0/2913449554 wait complete. 2024-02-19T14:56:50.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:56:50.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.730+0000 7fa01bc39640 1 Processor -- start 2024-02-19T14:56:50.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.730+0000 7fa01bc39640 1 -- start start 2024-02-19T14:56:50.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01bc39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0141527d0 0x7fa0140c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:50.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01bc39640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0140c5360 0x7fa0141550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:50.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01bc39640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01406fa90 con 0x7fa0140c7f10 2024-02-19T14:56:50.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01bc39640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0141592b0 con 0x7fa0140c5360 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01bc39640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa014071280 con 0x7fa0141527d0 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0191d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0140c5360 0x7fa0141550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01a1d6640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa0140c7f10 0x7fa014155610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47894/0 (socket says 172.21.15.78:47894) 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0191d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa0140c5360 0x7fa0141550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34632/0 (socket says 172.21.15.78:34632) 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa01a1d6640 1 -- 172.21.15.78:0/658803361 learned_addr learned my addr 172.21.15.78:0/658803361 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0191d4640 1 -- 172.21.15.78:0/658803361 >> v2:172.21.15.150:3300/0 conn(0x7fa0141527d0 msgr2=0x7fa0140c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0191d4640 1 --2- 172.21.15.78:0/658803361 >> v2:172.21.15.150:3300/0 conn(0x7fa0141527d0 0x7fa0140c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0191d4640 1 -- 172.21.15.78:0/658803361 >> v1:172.21.15.78:6789/0 conn(0x7fa0140c7f10 legacy=0x7fa014155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0189d3640 1 -- 172.21.15.78:0/658803361 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855139128 0 0) 0x7fa01406fa90 con 0x7fa0140c7f10 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.731+0000 7fa0191d4640 1 -- 172.21.15.78:0/658803361 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01406f4d0 con 0x7fa0140c5360 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.732+0000 7fa0191d4640 1 --2- 172.21.15.78:0/658803361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141550d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa0140ab160 tx=0x7fa00402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee6a06732c11ed5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.732+0000 7fa0189d3640 1 -- 172.21.15.78:0/658803361 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004031830 con 0x7fa0140c5360 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.732+0000 7fa0189d3640 1 -- 172.21.15.78:0/658803361 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004031990 con 0x7fa0140c5360 2024-02-19T14:56:50.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.732+0000 7fa0189d3640 1 -- 172.21.15.78:0/658803361 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004031d20 con 0x7fa0140c5360 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 -- 172.21.15.78:0/658803361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 msgr2=0x7fa0141550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 --2- 172.21.15.78:0/658803361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141550d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa0140ab160 tx=0x7fa00402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 -- 172.21.15.78:0/658803361 shutdown_connections 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 --2- 172.21.15.78:0/658803361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141550d0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 --2- 172.21.15.78:0/658803361 >> v2:172.21.15.150:3300/0 conn(0x7fa0141527d0 0x7fa0140c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 -- 172.21.15.78:0/658803361 >> 172.21.15.78:0/658803361 conn(0x7fa0140aa370 msgr2=0x7fa0140aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 -- 172.21.15.78:0/658803361 shutdown_connections 2024-02-19T14:56:50.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.733+0000 7fa01bc39640 1 -- 172.21.15.78:0/658803361 wait complete. 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 Processor -- start 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 -- start start 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0140c7f10 0x7fa0140c2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0141527d0 0x7fa0141eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0141eb140 con 0x7fa0140c5360 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa014158050 con 0x7fa0140c7f10 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa01bc39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140ab570 con 0x7fa0141527d0 2024-02-19T14:56:50.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.734+0000 7fa0199d5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34640/0 (socket says 172.21.15.78:34640) 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 -- 172.21.15.78:0/1207612808 learned_addr learned my addr 172.21.15.78:0/1207612808 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0191d4640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0140c7f10 0x7fa0140c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa01a1d6640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0141527d0 0x7fa0141eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 -- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0141527d0 msgr2=0x7fa0141eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0141527d0 0x7fa0141eb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 -- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0140c7f10 msgr2=0x7fa0140c2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0140c7f10 0x7fa0140c2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 -- 172.21.15.78:0/1207612808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004009400 con 0x7fa0140c5360 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0191d4640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0140c7f10 0x7fa0140c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa01a1d6640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0141527d0 0x7fa0141eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa0199d5640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141563c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa01000b410 tx=0x7fa01000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010011830 con 0x7fa0140c5360 2024-02-19T14:56:50.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0100119b0 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010011d60 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0140ab850 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.735+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0140abe10 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.736+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0141eb280 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.737+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa01001a400 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.737+0000 7fa00a7fc640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ff806b810 0x7f9ff806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.737+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5921+0+0 (secure 0 0 0) 0x7fa01008a3a0 con 0x7fa0140c5360 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.737+0000 7fa0191d4640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ff806b810 0x7f9ff806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.738+0000 7fa0191d4640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ff806b810 0x7f9ff806dcd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa004036000 tx=0x7fa004008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.741+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa010054e20 con 0x7fa0140c5360 2024-02-19T14:56:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:50.917+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa014155d20 con 0x7fa0140c5360 2024-02-19T14:56:51.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.605+0000 7fa00a7fc640 1 -- 172.21.15.78:0/1207612808 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v129) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa0100584e0 con 0x7fa0140c5360 2024-02-19T14:56:51.608 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:56:51.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ff806b810 msgr2=0x7f9ff806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ff806b810 0x7f9ff806dcd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa004036000 tx=0x7fa004008030 comp rx=0 tx=0).stop 2024-02-19T14:56:51.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 msgr2=0x7fa0141563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141563c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa01000b410 tx=0x7fa01000b8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 shutdown_connections 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ff806b810 0x7f9ff806dcd0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0141527d0 0x7fa0141eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0140c7f10 0x7fa0140c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 --2- 172.21.15.78:0/1207612808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0140c5360 0x7fa0141563c0 secure :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa01000b410 tx=0x7fa01000b8d0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.609+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 >> 172.21.15.78:0/1207612808 conn(0x7fa0140aa370 msgr2=0x7fa014154960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.610+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 shutdown_connections 2024-02-19T14:56:51.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.610+0000 7fa01bc39640 1 -- 172.21.15.78:0/1207612808 wait complete. 2024-02-19T14:56:51.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-19T14:56:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.729+0000 7fa88b4eb640 1 Processor -- start 2024-02-19T14:56:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.729+0000 7fa88b4eb640 1 -- start start 2024-02-19T14:56:51.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa88b4eb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8840713a0 0x7fa884071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:51.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa88b4eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa8840c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa88b4eb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8840c69f0 con 0x7fa8840c8b60 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa88b4eb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8840c32c0 con 0x7fa8840713a0 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa88b4eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa884070500 con 0x7fa8840c29e0 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa888a86640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa8840c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa889a88640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa8840c8b60 0x7fa8841536b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:55756/0 (socket says 172.21.15.78:55756) 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa889287640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8840713a0 0x7fa884071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa889a88640 1 -- 172.21.15.78:0/3343617015 learned_addr learned my addr 172.21.15.78:0/3343617015 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa889287640 1 -- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 msgr2=0x7fa8840c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.730+0000 7fa889287640 1 --2- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa8840c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.731+0000 7fa889287640 1 -- 172.21.15.78:0/3343617015 >> v1:172.21.15.150:6789/0 conn(0x7fa8840c8b60 legacy=0x7fa8841536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.731+0000 7fa889287640 1 -- 172.21.15.78:0/3343617015 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88406fe40 con 0x7fa8840713a0 2024-02-19T14:56:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.731+0000 7fa888a86640 1 --2- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa8840c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.731+0000 7fa889287640 1 --2- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa884071780 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097d0 tx=0x7fa88002b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b314134209ab1bf5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.731+0000 7fa87bfff640 1 -- 172.21.15.78:0/3343617015 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880031770 con 0x7fa8840713a0 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa87bfff640 1 -- 172.21.15.78:0/3343617015 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8800318d0 con 0x7fa8840713a0 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa87bfff640 1 -- 172.21.15.78:0/3343617015 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880031c60 con 0x7fa8840713a0 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 msgr2=0x7fa884071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa884071780 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097d0 tx=0x7fa88002b810 comp rx=0 tx=0).stop 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3343617015 shutdown_connections 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa8840c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3343617015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa884071780 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3343617015 >> 172.21.15.78:0/3343617015 conn(0x7fa8840aa300 msgr2=0x7fa8840aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3343617015 shutdown_connections 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.732+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3343617015 wait complete. 2024-02-19T14:56:51.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.733+0000 7fa88b4eb640 1 Processor -- start 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.733+0000 7fa88b4eb640 1 -- start start 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa88b4eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa88406cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa88b4eb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa884151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa88b4eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 0x7fa884154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa88b4eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8841eaea0 con 0x7fa8840c8b60 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa88b4eb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8841eb3d0 con 0x7fa8840713a0 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa88b4eb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88406c0c0 con 0x7fa8840c29e0 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 0x7fa884154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:51.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889287640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa88406cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 0x7fa884154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34644/0 (socket says 172.21.15.78:34644) 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 -- 172.21.15.78:0/3541317115 learned_addr learned my addr 172.21.15.78:0/3541317115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa888a86640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa884151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 -- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 msgr2=0x7fa884151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa884151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 -- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 msgr2=0x7fa88406cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa88406cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.734+0000 7fa889a88640 1 -- 172.21.15.78:0/3541317115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874009400 con 0x7fa8840c8b60 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.735+0000 7fa889a88640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 0x7fa884154070 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00ca80 tx=0x7fa87c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.735+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c00ac30 con 0x7fa8840c8b60 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.735+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87c00ad90 con 0x7fa8840c8b60 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.735+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa880009400 con 0x7fa8840c8b60 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.735+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c012600 con 0x7fa8840c8b60 2024-02-19T14:56:51.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.735+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88406c360 con 0x7fa8840c8b60 2024-02-19T14:56:51.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.736+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa884074990 con 0x7fa8840c8b60 2024-02-19T14:56:51.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.736+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa87c007070 con 0x7fa8840c8b60 2024-02-19T14:56:51.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.737+0000 7fa879ffb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa86806b840 0x7fa86806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:51.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.737+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fa87c089ea0 con 0x7fa8840c8b60 2024-02-19T14:56:51.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.737+0000 7fa889287640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa86806b840 0x7fa86806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:51.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.737+0000 7fa889287640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa86806b840 0x7fa86806dd00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa880036000 tx=0x7fa88003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:51.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.741+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa87c054ac0 con 0x7fa8840c8b60 2024-02-19T14:56:51.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.915+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fa8840c7f80 con 0x7fa8840c8b60 2024-02-19T14:56:51.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.918+0000 7fa879ffb640 1 -- 172.21.15.78:0/3541317115 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v43) v1 ==== 96+0+0 (secure 0 0 0) 0x7fa87c017020 con 0x7fa8840c8b60 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa86806b840 msgr2=0x7fa86806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa86806b840 0x7fa86806dd00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa880036000 tx=0x7fa88003d040 comp rx=0 tx=0).stop 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 msgr2=0x7fa884154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 0x7fa884154070 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00ca80 tx=0x7fa87c00cf40 comp rx=0 tx=0).stop 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 shutdown_connections 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa86806b840 0x7fa86806dd00 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8840c8b60 0x7fa884154070 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8840c29e0 0x7fa884151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 --2- 172.21.15.78:0/3541317115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8840713a0 0x7fa88406cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 >> 172.21.15.78:0/3541317115 conn(0x7fa8840aa300 msgr2=0x7fa8840aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 shutdown_connections 2024-02-19T14:56:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:51.920+0000 7fa88b4eb640 1 -- 172.21.15.78:0/3541317115 wait complete. 2024-02-19T14:56:51.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:56:52.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.033+0000 7fb0a491e640 1 Processor -- start 2024-02-19T14:56:52.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.033+0000 7fb0a491e640 1 -- start start 2024-02-19T14:56:52.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.033+0000 7fb0a491e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb0a00c29c0 0x7fb0a01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.033+0000 7fb0a491e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a0072250 con 0x7fb0a00c5be0 2024-02-19T14:56:52.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.033+0000 7fb0a491e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a0072480 con 0x7fb0a00c25e0 2024-02-19T14:56:52.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.033+0000 7fb0a491e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a0152c90 con 0x7fb0a00c29c0 2024-02-19T14:56:52.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb0a00c5be0 0x7fb0a006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47900/0 (socket says 172.21.15.78:47900) 2024-02-19T14:56:52.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09ed9d640 1 -- 172.21.15.78:0/3995671462 learned_addr learned my addr 172.21.15.78:0/3995671462 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09dd9b640 1 --2- 172.21.15.78:0/3995671462 >> v2:172.21.15.150:3300/0 conn(0x7fb0a00c29c0 0x7fb0a01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185310571 0 0) 0x7fb0a0072250 con 0x7fb0a00c5be0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb088003520 con 0x7fb0a00c5be0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010455588 0 0) 0x7fb0a0072480 con 0x7fb0a00c25e0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a0072250 con 0x7fb0a00c25e0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1518502153 0 0) 0x7fb088003520 con 0x7fb0a00c5be0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 >> v2:172.21.15.150:3300/0 conn(0x7fb0a00c29c0 msgr2=0x7fb0a01526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 --2- 172.21.15.78:0/3995671462 >> v2:172.21.15.150:3300/0 conn(0x7fb0a00c29c0 0x7fb0a01526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 >> v1:172.21.15.150:6789/0 conn(0x7fb0a00c25e0 legacy=0x7fb0a00c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a00c4b90 con 0x7fb0a00c5be0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb0980019f0 con 0x7fb0a00c5be0 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 >> v1:172.21.15.78:6789/0 conn(0x7fb0a00c5be0 legacy=0x7fb0a006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb088003b50 0x7fb088003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb08800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0152c90 con 0x7fb0880051d0 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08800e310 con 0x7fb088003b50 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08800e4d0 con 0x7fb0880047c0 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09ed9d640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb08800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09e59c640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb088003b50 0x7fb088003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.034+0000 7fb09dd9b640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 -- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0880047c0 msgr2=0x7fb088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 -- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb088003b50 msgr2=0x7fb088003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb088003b50 0x7fb088003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 -- 172.21.15.78:0/3995671462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088003520 con 0x7fb0880051d0 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09dd9b640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 -- 172.21.15.78:0/3995671462 reap_dead start 2024-02-19T14:56:52.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09ed9d640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb08800db90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb09800a710 tx=0x7fb0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a8c1cebbf5b05 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098034630 con 0x7fb0880051d0 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09802f070 con 0x7fb0880051d0 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.035+0000 7fb09d59a640 1 -- 172.21.15.78:0/3995671462 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098034a10 con 0x7fb0880051d0 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 -- 172.21.15.78:0/3995671462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 msgr2=0x7fb08800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb08800db90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb09800a710 tx=0x7fb0880057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 -- 172.21.15.78:0/3995671462 shutdown_connections 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 --2- 172.21.15.78:0/3995671462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb08800db90 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 -- 172.21.15.78:0/3995671462 >> 172.21.15.78:0/3995671462 conn(0x7fb0a006b270 msgr2=0x7fb0a006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 -- 172.21.15.78:0/3995671462 shutdown_connections 2024-02-19T14:56:52.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.036+0000 7fb0a491e640 1 -- 172.21.15.78:0/3995671462 wait complete. 2024-02-19T14:56:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.037+0000 7fb0a491e640 1 Processor -- start 2024-02-19T14:56:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.037+0000 7fb0a491e640 1 -- start start 2024-02-19T14:56:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb0a491e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb0a00a8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb0a491e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0a00c01d0 0x7fb0a0154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb0a491e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0a01554e0 0x7fb0a00bb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb0a491e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00bcf40 con 0x7fb0880051d0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb0a491e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00bc980 con 0x7fb0a01554e0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb0a491e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a01e68f0 con 0x7fb0a00c01d0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb0a00a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb0a00a8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34658/0 (socket says 172.21.15.78:34658) 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09e59c640 1 -- 172.21.15.78:0/2449789678 learned_addr learned my addr 172.21.15.78:0/2449789678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09dd9b640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0a00c01d0 0x7fb0a0154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09ed9d640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0a01554e0 0x7fb0a00bb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09e59c640 1 -- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0a00c01d0 msgr2=0x7fb0a0154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09e59c640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0a00c01d0 0x7fb0a0154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.038+0000 7fb09e59c640 1 -- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0a01554e0 msgr2=0x7fb0a00bb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb09e59c640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0a01554e0 0x7fb0a00bb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb09e59c640 1 -- 172.21.15.78:0/2449789678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a00a87d0 con 0x7fb0880051d0 2024-02-19T14:56:52.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb09e59c640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb0a00a8290 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb09000f7a0 tx=0x7fb09000fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000fe50 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090005320 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a01e6b90 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090007290 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.039+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a00c0820 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.040+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a0074990 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.040+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb0900073f0 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.041+0000 7fb08effd640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb08006b840 0x7fb08006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.041+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fb090089970 con 0x7fb0880051d0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.041+0000 7fb09dd9b640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb08006b840 0x7fb08006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.041+0000 7fb09dd9b640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb08006b840 0x7fb08006dd00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb0a006ea20 tx=0x7fb094008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.045+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb090054590 con 0x7fb0880051d0 2024-02-19T14:56:52.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.217+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb0a00bc760 con 0x7fb0880051d0 2024-02-19T14:56:52.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.220+0000 7fb08effd640 1 -- 172.21.15.78:0/2449789678 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb090005490 con 0x7fb0880051d0 2024-02-19T14:56:52.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb08006b840 msgr2=0x7fb08006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb08006b840 0x7fb08006dd00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb0a006ea20 tx=0x7fb094008040 comp rx=0 tx=0).stop 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 msgr2=0x7fb0a00a8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb0a00a8290 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb09000f7a0 tx=0x7fb09000fc60 comp rx=0 tx=0).stop 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 shutdown_connections 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb08006b840 0x7fb08006dd00 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0a01554e0 0x7fb0a00bb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0a00c01d0 0x7fb0a0154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 --2- 172.21.15.78:0/2449789678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0880051d0 0x7fb0a00a8290 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 >> 172.21.15.78:0/2449789678 conn(0x7fb0a006b270 msgr2=0x7fb0a014fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 shutdown_connections 2024-02-19T14:56:52.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.222+0000 7fb0a491e640 1 -- 172.21.15.78:0/2449789678 wait complete. 2024-02-19T14:56:52.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-02-19T14:56:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.332+0000 7fe585b3e640 1 Processor -- start 2024-02-19T14:56:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.332+0000 7fe585b3e640 1 -- start start 2024-02-19T14:56:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe585b3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe5800c25e0 0x7fe5801526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe585b3e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe5800c5be0 0x7fe58006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe585b3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe580072250 con 0x7fe5800c29c0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe585b3e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe580152c90 con 0x7fe5800c5be0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe585b3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe580072480 con 0x7fe5800c25e0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe5800c5be0 0x7fe58006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe5800c5be0 0x7fe58006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34662/0 (socket says 172.21.15.78:34662) 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 -- 172.21.15.78:0/1809289812 learned_addr learned my addr 172.21.15.78:0/1809289812 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57f7fe640 1 --2- 172.21.15.78:0/1809289812 >> v2:172.21.15.150:3300/0 conn(0x7fe5800c25e0 0x7fe5801526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 -- 172.21.15.78:0/1809289812 >> v2:172.21.15.150:3300/0 conn(0x7fe5800c25e0 msgr2=0x7fe5801526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 --2- 172.21.15.78:0/1809289812 >> v2:172.21.15.150:3300/0 conn(0x7fe5800c25e0 0x7fe5801526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 -- 172.21.15.78:0/1809289812 >> v1:172.21.15.150:6789/0 conn(0x7fe5800c29c0 legacy=0x7fe5800c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.333+0000 7fe57ffff640 1 -- 172.21.15.78:0/1809289812 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800c4b90 con 0x7fe5800c5be0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.334+0000 7fe57ffff640 1 --2- 172.21.15.78:0/1809289812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c5be0 0x7fe58006e420 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fe58006f120 tx=0x7fe57802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b524b3ae2493d52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.334+0000 7fe57e7fc640 1 -- 172.21.15.78:0/1809289812 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57802c9f0 con 0x7fe5800c5be0 2024-02-19T14:56:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.334+0000 7fe57e7fc640 1 -- 172.21.15.78:0/1809289812 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57802cb50 con 0x7fe5800c5be0 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.334+0000 7fe57e7fc640 1 -- 172.21.15.78:0/1809289812 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5780343f0 con 0x7fe5800c5be0 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 -- 172.21.15.78:0/1809289812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c5be0 msgr2=0x7fe58006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1809289812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c5be0 0x7fe58006e420 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fe58006f120 tx=0x7fe57802be50 comp rx=0 tx=0).stop 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 -- 172.21.15.78:0/1809289812 shutdown_connections 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1809289812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c5be0 0x7fe58006e420 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1809289812 >> v2:172.21.15.150:3300/0 conn(0x7fe5800c25e0 0x7fe5801526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 -- 172.21.15.78:0/1809289812 >> 172.21.15.78:0/1809289812 conn(0x7fe58006b270 msgr2=0x7fe58006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 -- 172.21.15.78:0/1809289812 shutdown_connections 2024-02-19T14:56:52.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.335+0000 7fe585b3e640 1 -- 172.21.15.78:0/1809289812 wait complete. 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.336+0000 7fe585b3e640 1 Processor -- start 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.336+0000 7fe585b3e640 1 -- start start 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5800c25e0 0x7fe5801eb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 0x7fe580149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5800c5be0 0x7fe580149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800a8db0 con 0x7fe5800c29c0 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801e9680 con 0x7fe5800c25e0 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe580149d90 con 0x7fe5800c5be0 2024-02-19T14:56:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 0x7fe580149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 0x7fe580149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34664/0 (socket says 172.21.15.78:34664) 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 -- 172.21.15.78:0/1636332260 learned_addr learned my addr 172.21.15.78:0/1636332260 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57ffff640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5800c5be0 0x7fe580149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57f7fe640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5800c25e0 0x7fe5801eb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 -- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5800c5be0 msgr2=0x7fe580149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5800c5be0 0x7fe580149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 -- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5800c25e0 msgr2=0x7fe5801eb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5800c25e0 0x7fe5801eb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 -- 172.21.15.78:0/1636332260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe578009b30 con 0x7fe5800c29c0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe57effd640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 0x7fe580149310 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fe57400c430 tx=0x7fe57400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574011790 con 0x7fe5800c29c0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5740118f0 con 0x7fe5800c29c0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5801ec470 con 0x7fe5800c29c0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574011ce0 con 0x7fe5800c29c0 2024-02-19T14:56:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.337+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58014a410 con 0x7fe5800c29c0 2024-02-19T14:56:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.338+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5801e9880 con 0x7fe5800c29c0 2024-02-19T14:56:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.339+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe57401c460 con 0x7fe5800c29c0 2024-02-19T14:56:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.339+0000 7fe567fff640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe56c06b840 0x7fe56c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.339+0000 7fe57f7fe640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe56c06b840 0x7fe56c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.339+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe57408a050 con 0x7fe5800c29c0 2024-02-19T14:56:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.340+0000 7fe57f7fe640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe56c06b840 0x7fe56c06dd00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe570006fd0 tx=0x7fe570008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.344+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe5801e9880 con 0x7fe5800c29c0 2024-02-19T14:56:52.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.585+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7fe5801e97f0 con 0x7fe5800c29c0 2024-02-19T14:56:52.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.616+0000 7fe567fff640 1 -- 172.21.15.78:0/1636332260 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v296) v1 ==== 145+0+0 (secure 0 0 0) 0x7fe574054bc0 con 0x7fe5800c29c0 2024-02-19T14:56:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe56c06b840 msgr2=0x7fe56c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe56c06b840 0x7fe56c06dd00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe570006fd0 tx=0x7fe570008040 comp rx=0 tx=0).stop 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 msgr2=0x7fe580149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 0x7fe580149310 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fe57400c430 tx=0x7fe57400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 shutdown_connections 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe56c06b840 0x7fe56c06dd00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5800c29c0 0x7fe580149310 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5800c5be0 0x7fe580149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 --2- 172.21.15.78:0/1636332260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5800c25e0 0x7fe5801eb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 >> 172.21.15.78:0/1636332260 conn(0x7fe58006b270 msgr2=0x7fe58006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 shutdown_connections 2024-02-19T14:56:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.618+0000 7fe585b3e640 1 -- 172.21.15.78:0/1636332260 wait complete. 2024-02-19T14:56:52.632 INFO:tasks.cephfs_test_runner:That a sole MDS with mds_join_fs set will come back as active eventually even after failover. ... ok 2024-02-19T14:56:52.632 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will cause failover if another high-affinity standby exists. 2024-02-19T14:56:52.633 INFO:tasks.cephfs_test_runner:test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:56:52.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.730+0000 7f4c5ec46640 1 Processor -- start 2024-02-19T14:56:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.730+0000 7f4c5ec46640 1 -- start start 2024-02-19T14:56:52.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5ec46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c5806d000 0x7f4c58159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5ec46640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4c580c6560 0x7f4c5806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5ec46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580c6a60 con 0x7f4c58156650 2024-02-19T14:56:52.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5ec46640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c581554b0 con 0x7f4c580c6560 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5ec46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58155790 con 0x7f4c5806d000 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4c580c6560 0x7f4c5806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4c580c6560 0x7f4c5806d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34666/0 (socket says 172.21.15.78:34666) 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 -- 172.21.15.78:0/4032056175 learned_addr learned my addr 172.21.15.78:0/4032056175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5c9e2640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c5806d000 0x7f4c58159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 -- 172.21.15.78:0/4032056175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c5806d000 msgr2=0x7f4c58159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c5806d000 0x7f4c58159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 -- 172.21.15.78:0/4032056175 >> v1:172.21.15.150:6789/0 conn(0x7f4c58156650 legacy=0x7f4c58156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c4ffff640 1 -- 172.21.15.78:0/4032056175 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58155a70 con 0x7f4c580c6560 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.731+0000 7f4c5c9e2640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c5806d000 0x7f4c58159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.732+0000 7f4c4ffff640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c580c6560 0x7f4c5806d3e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4c48009850 tx=0x7f4c4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678d0d8b1cb5bda4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.732+0000 7f4c4f7fe640 1 -- 172.21.15.78:0/4032056175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4802e070 con 0x7f4c580c6560 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.732+0000 7f4c4f7fe640 1 -- 172.21.15.78:0/4032056175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48004030 con 0x7f4c580c6560 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.732+0000 7f4c4f7fe640 1 -- 172.21.15.78:0/4032056175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48004420 con 0x7f4c580c6560 2024-02-19T14:56:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.732+0000 7f4c5ec46640 1 -- 172.21.15.78:0/4032056175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c580c6560 msgr2=0x7f4c5806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.732+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c580c6560 0x7f4c5806d3e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f4c48009850 tx=0x7f4c4802b840 comp rx=0 tx=0).stop 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.733+0000 7f4c5ec46640 1 -- 172.21.15.78:0/4032056175 shutdown_connections 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.733+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c580c6560 0x7f4c5806d3e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.733+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/4032056175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c5806d000 0x7f4c58159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.733+0000 7f4c5ec46640 1 -- 172.21.15.78:0/4032056175 >> 172.21.15.78:0/4032056175 conn(0x7f4c580ca870 msgr2=0x7f4c580cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.733+0000 7f4c5ec46640 1 -- 172.21.15.78:0/4032056175 shutdown_connections 2024-02-19T14:56:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.733+0000 7f4c5ec46640 1 -- 172.21.15.78:0/4032056175 wait complete. 2024-02-19T14:56:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 Processor -- start 2024-02-19T14:56:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 -- start start 2024-02-19T14:56:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c5806d000 0x7f4c5814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c580c6560 0x7f4c58151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 0x7f4c58151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c580ca530 con 0x7f4c58156650 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581426b0 con 0x7f4c5806d000 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.734+0000 7f4c5ec46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58142970 con 0x7f4c580c6560 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 0x7f4c58151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 0x7f4c58151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34678/0 (socket says 172.21.15.78:34678) 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 -- 172.21.15.78:0/2474082449 learned_addr learned my addr 172.21.15.78:0/2474082449 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c4ffff640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c580c6560 0x7f4c58151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5c9e2640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c5806d000 0x7f4c5814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 -- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c580c6560 msgr2=0x7f4c58151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c580c6560 0x7f4c58151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 -- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c5806d000 msgr2=0x7f4c5814e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c5806d000 0x7f4c5814e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 -- 172.21.15.78:0/2474082449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c48009460 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5c9e2640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c5806d000 0x7f4c5814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5d1e3640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 0x7f4c58151a80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000b970 tx=0x7f4c5000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000cb40 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5000cca0 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50014680 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c58156110 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.735+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c580ca1c0 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.736+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5814e630 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.737+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4c5001d3b0 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.737+0000 7f4c4d7fa640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4406b840 0x7f4c4406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.737+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f4c50089f00 con 0x7f4c58156650 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.737+0000 7f4c5c9e2640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4406b840 0x7f4c4406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.738+0000 7f4c5c9e2640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4406b840 0x7f4c4406dd00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c54006fd0 tx=0x7f4c54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.741+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4c500549e0 con 0x7f4c58156650 2024-02-19T14:56:52.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.913+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c58074990 con 0x7f4c58156650 2024-02-19T14:56:52.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.914+0000 7f4c4d7fa640 1 -- 172.21.15.78:0/2474082449 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+13549 (secure 0 0 0) 0x7f4c500580a0 con 0x7f4c58156650 2024-02-19T14:56:52.917 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:52.917 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":129,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:51.605161+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/2501419582":"2024-02-20T14:56:48.291972+0000","172.21.15.78:6835/2501419582":"2024-02-20T14:56:48.291972+0000","172.21.15.78:6836/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6834/1870596299":"2024-02-20T14:56:39.558499+0000","172.21.15.78:6837/100250519":"2024-02-20T14:56:36.537604+0000","172.21.15.78:6835/100250519":"2024-02-20T14:56:36.537604+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4406b840 msgr2=0x7f4c4406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4406b840 0x7f4c4406dd00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c54006fd0 tx=0x7f4c54008040 comp rx=0 tx=0).stop 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 msgr2=0x7f4c58151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 0x7f4c58151a80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000b970 tx=0x7f4c5000be30 comp rx=0 tx=0).stop 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 shutdown_connections 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4406b840 0x7f4c4406dd00 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c58156650 0x7f4c58151a80 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c580c6560 0x7f4c58151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 --2- 172.21.15.78:0/2474082449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c5806d000 0x7f4c5814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 >> 172.21.15.78:0/2474082449 conn(0x7f4c580ca870 msgr2=0x7f4c580c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:52.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 shutdown_connections 2024-02-19T14:56:52.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:52.916+0000 7f4c5ec46640 1 -- 172.21.15.78:0/2474082449 wait complete. 2024-02-19T14:56:52.929 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:56:52.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:56:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.029+0000 7fea37f20640 1 Processor -- start 2024-02-19T14:56:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea37f20640 1 -- start start 2024-02-19T14:56:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea37f20640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea3006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea37f20640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fea3006d750 0x7fea3015d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea37f20640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30150820 con 0x7fea30150440 2024-02-19T14:56:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea37f20640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300c1b60 con 0x7fea3006d750 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea37f20640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea301537d0 con 0x7fea3006ed20 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea35cbc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fea30150440 0x7fea300c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47908/0 (socket says 172.21.15.78:47908) 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.030+0000 7fea35cbc640 1 -- 172.21.15.78:0/3653239039 learned_addr learned my addr 172.21.15.78:0/3653239039 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea354bb640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea3006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214597274 0 0) 0x7fea30150820 con 0x7fea30150440 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 --2- 172.21.15.78:0/3653239039 >> v2:172.21.15.150:3300/0 conn(0x7fea3006d750 0x7fea3015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1c003520 con 0x7fea30150440 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297382366 0 0) 0x7fea1c003520 con 0x7fea30150440 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 msgr2=0x7fea3006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea3006d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 >> v2:172.21.15.150:3300/0 conn(0x7fea3006d750 msgr2=0x7fea3015d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 --2- 172.21.15.78:0/3653239039 >> v2:172.21.15.150:3300/0 conn(0x7fea3006d750 0x7fea3015d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30153d10 con 0x7fea30150440 2024-02-19T14:56:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fea2c001670 con 0x7fea30150440 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 >> v1:172.21.15.78:6789/0 conn(0x7fea30150440 legacy=0x7fea300c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea1c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea1c003ef0 0x7fea1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300c1b60 con 0x7fea1c004d00 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea301537d0 con 0x7fea1c003ef0 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c00b9b0 con 0x7fea3006ed20 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea35cbc640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea1c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea354bb640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea1c003ef0 0x7fea1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 -- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 msgr2=0x7fea1c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea1c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 -- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea1c003ef0 msgr2=0x7fea1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea1c003ef0 0x7fea1c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea364bd640 1 -- 172.21.15.78:0/3653239039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c003520 con 0x7fea1c004d00 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.031+0000 7fea35cbc640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea3006ed20 0x7fea1c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.032+0000 7fea364bd640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fea28009fc0 tx=0x7fea1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8b800c80dbd84f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.032+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2802cc60 con 0x7fea1c004d00 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.032+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2800f040 con 0x7fea1c004d00 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.032+0000 7fea34cba640 1 -- 172.21.15.78:0/3653239039 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28034440 con 0x7fea1c004d00 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea37f20640 1 -- 172.21.15.78:0/3653239039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea1c004d00 msgr2=0x7fea1c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea37f20640 1 --2- 172.21.15.78:0/3653239039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea1c004d00 0x7fea1c005180 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fea28009fc0 tx=0x7fea1c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea364bd640 1 -- 172.21.15.78:0/3653239039 reap_dead start 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea37f20640 1 -- 172.21.15.78:0/3653239039 shutdown_connections 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea37f20640 1 -- 172.21.15.78:0/3653239039 >> 172.21.15.78:0/3653239039 conn(0x7fea3006b270 msgr2=0x7fea3006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea37f20640 1 -- 172.21.15.78:0/3653239039 shutdown_connections 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.033+0000 7fea37f20640 1 -- 172.21.15.78:0/3653239039 wait complete. 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.034+0000 7fea37f20640 1 Processor -- start 2024-02-19T14:56:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.034+0000 7fea37f20640 1 -- start start 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea1c004d00 0x7fea300a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea30155890 0x7fea3014b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 0x7fea30145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea300a7d70 con 0x7fea301457b0 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea301546a0 con 0x7fea30155890 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea30146170 con 0x7fea1c004d00 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 0x7fea30145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 0x7fea30145c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34694/0 (socket says 172.21.15.78:34694) 2024-02-19T14:56:53.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 -- 172.21.15.78:0/3833234100 learned_addr learned my addr 172.21.15.78:0/3833234100 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea354bb640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea30155890 0x7fea3014b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea35cbc640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea1c004d00 0x7fea300a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 -- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea1c004d00 msgr2=0x7fea300a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea1c004d00 0x7fea300a7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 -- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea30155890 msgr2=0x7fea3014b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea30155890 0x7fea3014b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 -- 172.21.15.78:0/3833234100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea300a8b80 con 0x7fea301457b0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea35cbc640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea1c004d00 0x7fea300a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea364bd640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 0x7fea30145c30 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fea2802c6d0 tx=0x7fea28009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28043070 con 0x7fea301457b0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28034b50 con 0x7fea301457b0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28003990 con 0x7fea301457b0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea300a8e60 con 0x7fea301457b0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.035+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3014b750 con 0x7fea301457b0 2024-02-19T14:56:53.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.036+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30154190 con 0x7fea301457b0 2024-02-19T14:56:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.037+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fea2803e440 con 0x7fea301457b0 2024-02-19T14:56:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.037+0000 7fea267fc640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea1406b840 0x7fea1406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.037+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fea280abce0 con 0x7fea301457b0 2024-02-19T14:56:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.037+0000 7fea35cbc640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea1406b840 0x7fea1406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.038+0000 7fea35cbc640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea1406b840 0x7fea1406dd00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fea2c001eb0 tx=0x7fea2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.041+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fea30154190 con 0x7fea301457b0 2024-02-19T14:56:53.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.212+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea30154190 con 0x7fea301457b0 2024-02-19T14:56:53.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.212+0000 7fea267fc640 1 -- 172.21.15.78:0/3833234100 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea30154190 con 0x7fea301457b0 2024-02-19T14:56:53.215 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:56:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea1406b840 msgr2=0x7fea1406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea1406b840 0x7fea1406dd00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fea2c001eb0 tx=0x7fea2c009040 comp rx=0 tx=0).stop 2024-02-19T14:56:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 msgr2=0x7fea30145c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 0x7fea30145c30 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fea2802c6d0 tx=0x7fea28009750 comp rx=0 tx=0).stop 2024-02-19T14:56:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 shutdown_connections 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea1406b840 0x7fea1406dd00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea301457b0 0x7fea30145c30 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea30155890 0x7fea3014b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 --2- 172.21.15.78:0/3833234100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea1c004d00 0x7fea300a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 >> 172.21.15.78:0/3833234100 conn(0x7fea3006b270 msgr2=0x7fea3006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 shutdown_connections 2024-02-19T14:56:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.214+0000 7fea37f20640 1 -- 172.21.15.78:0/3833234100 wait complete. 2024-02-19T14:56:53.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-02-19T14:56:53.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.325+0000 7f34dee55640 1 Processor -- start 2024-02-19T14:56:53.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.325+0000 7f34dee55640 1 -- start start 2024-02-19T14:56:53.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dee55640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d8155f90 0x7f34d806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dee55640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34d81588a0 0x7f34d8158d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dee55640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80c6af0 con 0x7f34d80c6d30 2024-02-19T14:56:53.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dee55640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d80c7940 con 0x7f34d81588a0 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dee55640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80c7c20 con 0x7f34d8155f90 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dd3f2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34d81588a0 0x7f34d8158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dcbf1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d8155f90 0x7f34d806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dd3f2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34d81588a0 0x7f34d8158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:43878/0 (socket says 172.21.15.78:43878) 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34dd3f2640 1 -- 172.21.15.78:0/2585684244 learned_addr learned my addr 172.21.15.78:0/2585684244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.326+0000 7f34d7fff640 1 --1- 172.21.15.78:0/2585684244 >> v1:172.21.15.150:6789/0 conn(0x7f34d80c6d30 0x7f34d80c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:55782/0 (socket says 172.21.15.78:55782) 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.327+0000 7f34dd3f2640 1 -- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d8155f90 msgr2=0x7f34d806d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.327+0000 7f34dd3f2640 1 --2- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d8155f90 0x7f34d806d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.327+0000 7f34dd3f2640 1 -- 172.21.15.78:0/2585684244 >> v1:172.21.15.150:6789/0 conn(0x7f34d80c6d30 legacy=0x7f34d80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.327+0000 7f34dd3f2640 1 -- 172.21.15.78:0/2585684244 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80c7f00 con 0x7f34d81588a0 2024-02-19T14:56:53.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.327+0000 7f34dcbf1640 1 --2- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d8155f90 0x7f34d806d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:53.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dd3f2640 1 --2- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d81588a0 0x7f34d8158d10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009e50 tx=0x7f34d002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27cf8f193839d45c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34d77fe640 1 -- 172.21.15.78:0/2585684244 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d002ca70 con 0x7f34d81588a0 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34d77fe640 1 -- 172.21.15.78:0/2585684244 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d002cbd0 con 0x7f34d81588a0 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34d77fe640 1 -- 172.21.15.78:0/2585684244 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d00343f0 con 0x7f34d81588a0 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 -- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d81588a0 msgr2=0x7f34d8158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 --2- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d81588a0 0x7f34d8158d10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009e50 tx=0x7f34d002bed0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 -- 172.21.15.78:0/2585684244 shutdown_connections 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 --2- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d81588a0 0x7f34d8158d10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 --2- 172.21.15.78:0/2585684244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d8155f90 0x7f34d806d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 -- 172.21.15.78:0/2585684244 >> 172.21.15.78:0/2585684244 conn(0x7f34d806b700 msgr2=0x7f34d80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 -- 172.21.15.78:0/2585684244 shutdown_connections 2024-02-19T14:56:53.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.328+0000 7f34dee55640 1 -- 172.21.15.78:0/2585684244 wait complete. 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 Processor -- start 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 -- start start 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d80c6d30 0x7f34d80a8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d8155f90 0x7f34d806fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 0x7f34d814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80c92b0 con 0x7f34d81588a0 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81eb260 con 0x7f34d8155f90 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.330+0000 7f34dee55640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80a9140 con 0x7f34d80c6d30 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 0x7f34d814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 0x7f34d814e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34696/0 (socket says 172.21.15.78:34696) 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 -- 172.21.15.78:0/557958402 learned_addr learned my addr 172.21.15.78:0/557958402 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dcbf1640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d80c6d30 0x7f34d80a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34d7fff640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d8155f90 0x7f34d806fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 -- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d80c6d30 msgr2=0x7f34d80a8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d80c6d30 0x7f34d80a8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 -- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d8155f90 msgr2=0x7f34d806fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d8155f90 0x7f34d806fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 -- 172.21.15.78:0/557958402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009400 con 0x7f34d81588a0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dd3f2640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 0x7f34d814e960 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f34d0036040 tx=0x7f34d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d0009510 con 0x7f34d81588a0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d003e450 con 0x7f34d81588a0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d0009b30 con 0x7f34d81588a0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d003e820 con 0x7f34d81588a0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.331+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d80a8ea0 con 0x7f34d81588a0 2024-02-19T14:56:53.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.332+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d8074990 con 0x7f34d81588a0 2024-02-19T14:56:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.333+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f34d003e980 con 0x7f34d81588a0 2024-02-19T14:56:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.333+0000 7f34d57fa640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34c406b810 0x7f34c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.333+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f34d00ace90 con 0x7f34d81588a0 2024-02-19T14:56:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.333+0000 7f34dcbf1640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34c406b810 0x7f34c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.333+0000 7f34dcbf1640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34c406b810 0x7f34c406dcd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f34d8154460 tx=0x7f34c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.337+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f34d00778f0 con 0x7f34d81588a0 2024-02-19T14:56:53.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.573+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f34d81eb3a0 con 0x7f34d81588a0 2024-02-19T14:56:53.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.618+0000 7f34d57fa640 1 -- 172.21.15.78:0/557958402 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v297) v1 ==== 147+0+0 (secure 0 0 0) 0x7f34d007afb0 con 0x7f34d81588a0 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34c406b810 msgr2=0x7f34c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34c406b810 0x7f34c406dcd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f34d8154460 tx=0x7f34c8008040 comp rx=0 tx=0).stop 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 msgr2=0x7f34d814e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 0x7f34d814e960 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f34d0036040 tx=0x7f34d0004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 shutdown_connections 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34c406b810 0x7f34c406dcd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34d81588a0 0x7f34d814e960 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34d8155f90 0x7f34d806fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 --2- 172.21.15.78:0/557958402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34d80c6d30 0x7f34d80a8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 >> 172.21.15.78:0/557958402 conn(0x7f34d806b700 msgr2=0x7f34d80c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 shutdown_connections 2024-02-19T14:56:53.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.620+0000 7f34dee55640 1 -- 172.21.15.78:0/557958402 wait complete. 2024-02-19T14:56:53.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:56:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 Processor -- start 2024-02-19T14:56:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 -- start start 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f99d40c25e0 0x7f99d41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c5be0 0x7f99d406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4072250 con 0x7f99d40c29c0 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d4152c90 con 0x7f99d40c25e0 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.734+0000 7f99db36b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d4072480 con 0x7f99d40c5be0 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d9107640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f99d40c25e0 0x7f99d41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d8906640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f99d40c29c0 0x7f99d40c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47914/0 (socket says 172.21.15.78:47914) 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d8906640 1 -- 172.21.15.78:0/803521104 learned_addr learned my addr 172.21.15.78:0/803521104 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d9908640 1 --2- 172.21.15.78:0/803521104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c5be0 0x7f99d406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d9107640 1 -- 172.21.15.78:0/803521104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c5be0 msgr2=0x7f99d406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d9107640 1 --2- 172.21.15.78:0/803521104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c5be0 0x7f99d406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d9107640 1 -- 172.21.15.78:0/803521104 >> v1:172.21.15.78:6789/0 conn(0x7f99d40c29c0 legacy=0x7f99d40c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99cbfff640 1 -- 172.21.15.78:0/803521104 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483904673 0 0) 0x7f99d4072250 con 0x7f99d40c29c0 2024-02-19T14:56:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.735+0000 7f99d9107640 1 -- 172.21.15.78:0/803521104 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40c4b90 con 0x7f99d40c25e0 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99d9107640 1 --2- 172.21.15.78:0/803521104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c25e0 0x7f99d41526a0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f99d000ba40 tx=0x7f99d002dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d571f94285828027 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99cbfff640 1 -- 172.21.15.78:0/803521104 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d002eda0 con 0x7f99d40c25e0 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99cbfff640 1 -- 172.21.15.78:0/803521104 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d0036440 con 0x7f99d40c25e0 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99cbfff640 1 -- 172.21.15.78:0/803521104 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d00367d0 con 0x7f99d40c25e0 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 -- 172.21.15.78:0/803521104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c25e0 msgr2=0x7f99d41526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 --2- 172.21.15.78:0/803521104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c25e0 0x7f99d41526a0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f99d000ba40 tx=0x7f99d002dff0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 -- 172.21.15.78:0/803521104 shutdown_connections 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 --2- 172.21.15.78:0/803521104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c5be0 0x7f99d406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 --2- 172.21.15.78:0/803521104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c25e0 0x7f99d41526a0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 -- 172.21.15.78:0/803521104 >> 172.21.15.78:0/803521104 conn(0x7f99d406b270 msgr2=0x7f99d406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.736+0000 7f99db36b640 1 -- 172.21.15.78:0/803521104 shutdown_connections 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.737+0000 7f99db36b640 1 -- 172.21.15.78:0/803521104 wait complete. 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 Processor -- start 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 -- start start 2024-02-19T14:56:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c25e0 0x7f99d40c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 0x7f99d41e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d40c5be0 0x7f99d414c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d40a9180 con 0x7f99d40c29c0 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41553a0 con 0x7f99d40c5be0 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99db36b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d41e9520 con 0x7f99d40c25e0 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99d8906640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 0x7f99d41e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99d8906640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 0x7f99d41e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34724/0 (socket says 172.21.15.78:34724) 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.738+0000 7f99d8906640 1 -- 172.21.15.78:0/359098268 learned_addr learned my addr 172.21.15.78:0/359098268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99d8906640 1 -- 172.21.15.78:0/359098268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c25e0 msgr2=0x7f99d40c10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99d8906640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c25e0 0x7f99d40c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99d8906640 1 -- 172.21.15.78:0/359098268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d40c5be0 msgr2=0x7f99d414c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99d8906640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d40c5be0 0x7f99d414c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99d8906640 1 -- 172.21.15.78:0/359098268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d000b6a0 con 0x7f99d40c29c0 2024-02-19T14:56:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99d8906640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 0x7f99d41e7280 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f99c400c430 tx=0x7f99c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c40074e0 con 0x7f99d40c29c0 2024-02-19T14:56:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c4007660 con 0x7f99d40c29c0 2024-02-19T14:56:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d40a8ea0 con 0x7f99d40c29c0 2024-02-19T14:56:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c4004a20 con 0x7f99d40c29c0 2024-02-19T14:56:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.739+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d41e9aa0 con 0x7f99d40c29c0 2024-02-19T14:56:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.740+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d41e9380 con 0x7f99d40c29c0 2024-02-19T14:56:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.740+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f99c4004b80 con 0x7f99d40c29c0 2024-02-19T14:56:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.741+0000 7f99c9ffb640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f99b806b810 0x7f99b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.741+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f99c4089e80 con 0x7f99d40c29c0 2024-02-19T14:56:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.741+0000 7f99d9107640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f99b806b810 0x7f99b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.741+0000 7f99d9107640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f99b806b810 0x7f99b806dcd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f99d0038000 tx=0x7f99d0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:53.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.745+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f99d41e9380 con 0x7f99d40c29c0 2024-02-19T14:56:53.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.915+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f99d41e9380 con 0x7f99d40c29c0 2024-02-19T14:56:53.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.918+0000 7f99c9ffb640 1 -- 172.21.15.78:0/359098268 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v45)=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7f99d41e9380 con 0x7f99d40c29c0 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f99b806b810 msgr2=0x7f99b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f99b806b810 0x7f99b806dcd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f99d0038000 tx=0x7f99d0030000 comp rx=0 tx=0).stop 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 msgr2=0x7f99d41e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 0x7f99d41e7280 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f99c400c430 tx=0x7f99c400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 shutdown_connections 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f99b806b810 0x7f99b806dcd0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99d40c29c0 0x7f99d41e7280 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d40c5be0 0x7f99d414c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 --2- 172.21.15.78:0/359098268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99d40c25e0 0x7f99d40c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 >> 172.21.15.78:0/359098268 conn(0x7f99d406b270 msgr2=0x7f99d406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 shutdown_connections 2024-02-19T14:56:53.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:53.920+0000 7f99db36b640 1 -- 172.21.15.78:0/359098268 wait complete. 2024-02-19T14:56:53.933 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:56:53.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:53.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:56:54.048 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:56:54.048 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:56:54.048 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:54.063 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:56:54.064 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:56:54.064 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:56:54.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:56:54.081 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:56:54.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:56:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.200+0000 7fba45623640 1 Processor -- start 2024-02-19T14:56:54.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.200+0000 7fba45623640 1 -- start start 2024-02-19T14:56:54.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.201+0000 7fba45623640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.201+0000 7fba45623640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba400c7f50 0x7fba40158560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.201+0000 7fba45623640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba400c83a0 con 0x7fba400c2990 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.201+0000 7fba45623640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4006cad0 con 0x7fba400c7f50 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.201+0000 7fba45623640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba40155080 con 0x7fba400c7b70 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3e7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fba400c2990 0x7fba4006d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:55794/0 (socket says 172.21.15.78:55794) 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba400c7f50 0x7fba40158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:49476/0 (socket says 172.21.15.78:49476) 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3e7fc640 1 -- 172.21.15.78:0/1285775268 learned_addr learned my addr 172.21.15.78:0/1285775268 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3f7fe640 1 -- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 msgr2=0x7fba40155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3f7fe640 1 --2- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3f7fe640 1 -- 172.21.15.78:0/1285775268 >> v1:172.21.15.150:6789/0 conn(0x7fba400c2990 legacy=0x7fba4006d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3f7fe640 1 -- 172.21.15.78:0/1285775268 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40152960 con 0x7fba400c7f50 2024-02-19T14:56:54.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.202+0000 7fba3effd640 1 --2- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:56:54.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba3f7fe640 1 --2- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba40158560 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fba380097a0 tx=0x7fba3802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfed17ff8dd4662f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:54.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba3dffb640 1 -- 172.21.15.78:0/1285775268 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38030610 con 0x7fba400c7f50 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba3dffb640 1 -- 172.21.15.78:0/1285775268 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba38030770 con 0x7fba400c7f50 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba3dffb640 1 -- 172.21.15.78:0/1285775268 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38030b00 con 0x7fba400c7f50 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 -- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 msgr2=0x7fba40158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 --2- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba40158560 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fba380097a0 tx=0x7fba3802b960 comp rx=0 tx=0).stop 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 -- 172.21.15.78:0/1285775268 shutdown_connections 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 --2- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba40158560 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 --2- 172.21.15.78:0/1285775268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 -- 172.21.15.78:0/1285775268 >> 172.21.15.78:0/1285775268 conn(0x7fba400c5050 msgr2=0x7fba400c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 -- 172.21.15.78:0/1285775268 shutdown_connections 2024-02-19T14:56:54.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.203+0000 7fba45623640 1 -- 172.21.15.78:0/1285775268 wait complete. 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 Processor -- start 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 -- start start 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 0x7fba40099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba401de5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4014e250 con 0x7fba400c2990 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4014df70 con 0x7fba400c7f50 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba45623640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba401eb840 con 0x7fba400c7b70 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba3effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 0x7fba40099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba3effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 0x7fba40099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34730/0 (socket says 172.21.15.78:34730) 2024-02-19T14:56:54.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.205+0000 7fba3effd640 1 -- 172.21.15.78:0/244626875 learned_addr learned my addr 172.21.15.78:0/244626875 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3e7fc640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3f7fe640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba401de5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3effd640 1 -- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 msgr2=0x7fba40157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3effd640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3effd640 1 -- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 msgr2=0x7fba401de5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3effd640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba401de5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3effd640 1 -- 172.21.15.78:0/244626875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30009a70 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba3effd640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 0x7fba40099a60 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fba3000efc0 tx=0x7fba3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30014680 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3000f040 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba38009400 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba30014a00 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.206+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba40158530 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.207+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba40074990 con 0x7fba400c2990 2024-02-19T14:56:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.207+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fba30014b60 con 0x7fba400c2990 2024-02-19T14:56:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.208+0000 7fba277fe640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba2c06b810 0x7fba2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.208+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fba3008b1a0 con 0x7fba400c2990 2024-02-19T14:56:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.208+0000 7fba3e7fc640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba2c06b810 0x7fba2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.208+0000 7fba3e7fc640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba2c06b810 0x7fba2c06dcd0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fba40151560 tx=0x7fba34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.212+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba30055d10 con 0x7fba400c2990 2024-02-19T14:56:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.417+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba400c58f0 con 0x7fba400c2990 2024-02-19T14:56:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.418+0000 7fba277fe640 1 -- 172.21.15.78:0/244626875 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+3705 (secure 0 0 0) 0x7fba300593d0 con 0x7fba400c2990 2024-02-19T14:56:54.421 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:54.421 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6239,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4213660424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4213660424},{"type":"v1","addr":"172.21.15.150:6834","nonce":4213660424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1857207464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1857207464},{"type":"v1","addr":"172.21.15.150:6835","nonce":1857207464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":8058,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/976486486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":976486486},{"type":"v1","addr":"172.21.15.78:6836","nonce":976486486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":8181,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2050679427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2050679427},{"type":"v1","addr":"172.21.15.78:6837","nonce":2050679427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[]} 2024-02-19T14:56:54.421 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 112 2024-02-19T14:56:54.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.419+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba2c06b810 msgr2=0x7fba2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.419+0000 7fba45623640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba2c06b810 0x7fba2c06dcd0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fba40151560 tx=0x7fba34009040 comp rx=0 tx=0).stop 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.419+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 msgr2=0x7fba40099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.419+0000 7fba45623640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 0x7fba40099a60 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fba3000efc0 tx=0x7fba3000c490 comp rx=0 tx=0).stop 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.419+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 shutdown_connections 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.419+0000 7fba45623640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fba2c06b810 0x7fba2c06dcd0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.420+0000 7fba45623640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba400c7f50 0x7fba401de5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.420+0000 7fba45623640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba400c7b70 0x7fba40157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.420+0000 7fba45623640 1 --2- 172.21.15.78:0/244626875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba400c2990 0x7fba40099a60 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.420+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 >> 172.21.15.78:0/244626875 conn(0x7fba400c5050 msgr2=0x7fba401508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.420+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 shutdown_connections 2024-02-19T14:56:54.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.420+0000 7fba45623640 1 -- 172.21.15.78:0/244626875 wait complete. 2024-02-19T14:56:54.433 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:56:54.433 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:56:54.434 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:56:54.434 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:54.434 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:56:54.434 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:56:54.434 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:56:54.434 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:54.434 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:56:54.435 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:56:54.435 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:56:54.435 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:54.435 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:56:54.435 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:56:54.435 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:56:54.435 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:56:54.443 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:56:54.443 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:56:54.443 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:56:54.444 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:56:54.447 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:56:54.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:56:54.447 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:56:54.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:56:54.486 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:56:54.486 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:54.486 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:56:54.486 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:54.489 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:56:54.489 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:54.490 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:56:54.490 DEBUG:teuthology.parallel:result is None 2024-02-19T14:56:54.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:56:54.562 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:56:54.562 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:56:54.573 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:56:54.573 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:56:54.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.594+0000 7f3994d6c640 1 Processor -- start 2024-02-19T14:56:54.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.594+0000 7f3994d6c640 1 -- start start 2024-02-19T14:56:54.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3994d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3990152700 0x7f39900c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3994d6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39900c8e00 0x7f399015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3994d6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39901551f0 con 0x7f39900c5730 2024-02-19T14:56:54.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3994d6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3990155490 con 0x7f39900c8e00 2024-02-19T14:56:54.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3994d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3990155770 con 0x7f3990152700 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39900c8e00 0x7f399015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3987fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f39900c5730 0x7f399006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47934/0 (socket says 172.21.15.78:47934) 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f3987fff640 1 -- 172.21.15.78:0/2358545880 learned_addr learned my addr 172.21.15.78:0/2358545880 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398e59c640 1 --2- 172.21.15.78:0/2358545880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3990152700 0x7f39900c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398ed9d640 1 -- 172.21.15.78:0/2358545880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3990152700 msgr2=0x7f39900c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398ed9d640 1 --2- 172.21.15.78:0/2358545880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3990152700 0x7f39900c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398ed9d640 1 -- 172.21.15.78:0/2358545880 >> v1:172.21.15.78:6789/0 conn(0x7f39900c5730 legacy=0x7f399006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398ed9d640 1 -- 172.21.15.78:0/2358545880 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3990155a50 con 0x7f39900c8e00 2024-02-19T14:56:54.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.595+0000 7f398dd9b640 1 -- 172.21.15.78:0/2358545880 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176327209 0 0) 0x7f39901551f0 con 0x7f39900c5730 2024-02-19T14:56:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.596+0000 7f398ed9d640 1 --2- 172.21.15.78:0/2358545880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39900c8e00 0x7f399015f980 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3988009d90 tx=0x7f398802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7374bfac1e030ae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.596+0000 7f398dd9b640 1 -- 172.21.15.78:0/2358545880 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398802cc50 con 0x7f39900c8e00 2024-02-19T14:56:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.596+0000 7f398dd9b640 1 -- 172.21.15.78:0/2358545880 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398802cdb0 con 0x7f39900c8e00 2024-02-19T14:56:54.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.596+0000 7f398dd9b640 1 -- 172.21.15.78:0/2358545880 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988034660 con 0x7f39900c8e00 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 -- 172.21.15.78:0/2358545880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39900c8e00 msgr2=0x7f399015f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2358545880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39900c8e00 0x7f399015f980 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f3988009d90 tx=0x7f398802bea0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 -- 172.21.15.78:0/2358545880 shutdown_connections 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2358545880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39900c8e00 0x7f399015f980 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2358545880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3990152700 0x7f39900c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 -- 172.21.15.78:0/2358545880 >> 172.21.15.78:0/2358545880 conn(0x7f39900cac90 msgr2=0x7f39900cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 -- 172.21.15.78:0/2358545880 shutdown_connections 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.597+0000 7f3994d6c640 1 -- 172.21.15.78:0/2358545880 wait complete. 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.598+0000 7f3994d6c640 1 Processor -- start 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.598+0000 7f3994d6c640 1 -- start start 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.598+0000 7f3994d6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39900c5730 0x7f39900aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.598+0000 7f3994d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39900c8e00 0x7f39901e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3994d6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 0x7f39901e7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3994d6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900c1330 con 0x7f3990152700 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3994d6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39901e8170 con 0x7f39900c5730 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3994d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39900ab280 con 0x7f39900c8e00 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 0x7f39901e7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 0x7f39901e7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34770/0 (socket says 172.21.15.78:34770) 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 -- 172.21.15.78:0/2652332023 learned_addr learned my addr 172.21.15.78:0/2652332023 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3987fff640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39900c8e00 0x7f39901e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398e59c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39900c5730 0x7f39900aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 -- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39900c8e00 msgr2=0x7f39901e76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39900c8e00 0x7f39901e76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 -- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39900c5730 msgr2=0x7f39900aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39900c5730 0x7f39900aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 -- 172.21.15.78:0/2652332023 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3988009a70 con 0x7f3990152700 2024-02-19T14:56:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f398ed9d640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 0x7f39901e7c30 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3988009d90 tx=0x7f3988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:54.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988003b90 con 0x7f3990152700 2024-02-19T14:56:54.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988003d10 con 0x7f3990152700 2024-02-19T14:56:54.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988004270 con 0x7f3990152700 2024-02-19T14:56:54.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39900ab560 con 0x7f3990152700 2024-02-19T14:56:54.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.599+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39900abb20 con 0x7f3990152700 2024-02-19T14:56:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.600+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3990074990 con 0x7f3990152700 2024-02-19T14:56:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.601+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f39880043d0 con 0x7f3990152700 2024-02-19T14:56:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.601+0000 7f3986ffd640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f397006b810 0x7f397006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.601+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f39880aca30 con 0x7f3990152700 2024-02-19T14:56:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.601+0000 7f398e59c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f397006b810 0x7f397006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.602+0000 7f398e59c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f397006b810 0x7f397006dcd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3980006fd0 tx=0x7f3980008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.606+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3988077510 con 0x7f3990152700 2024-02-19T14:56:54.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:54.790+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f39900c8770 con 0x7f3990152700 2024-02-19T14:56:55.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.634+0000 7f3986ffd640 1 -- 172.21.15.78:0/2652332023 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v134) v1 ==== 97+0+0 (secure 0 0 0) 0x7f398807abd0 con 0x7f3990152700 2024-02-19T14:56:55.637 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f397006b810 msgr2=0x7f397006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f397006b810 0x7f397006dcd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3980006fd0 tx=0x7f3980008040 comp rx=0 tx=0).stop 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 msgr2=0x7f39901e7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 0x7f39901e7c30 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3988009d90 tx=0x7f3988004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 shutdown_connections 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f397006b810 0x7f397006dcd0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3990152700 0x7f39901e7c30 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39900c8e00 0x7f39901e76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 --2- 172.21.15.78:0/2652332023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39900c5730 0x7f39900aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 >> 172.21.15.78:0/2652332023 conn(0x7f39900cac90 msgr2=0x7f399014e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:55.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 shutdown_connections 2024-02-19T14:56:55.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.637+0000 7f3994d6c640 1 -- 172.21.15.78:0/2652332023 wait complete. 2024-02-19T14:56:55.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:56:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.757+0000 7fa58a674640 1 Processor -- start 2024-02-19T14:56:55.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.757+0000 7fa58a674640 1 -- start start 2024-02-19T14:56:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa58a674640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa58414d210 0x7fa5840c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa58a674640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5840719e0 0x7fa5840c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa58a674640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58406ff90 con 0x7fa5840c3d90 2024-02-19T14:56:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa58a674640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa584070270 con 0x7fa58414d210 2024-02-19T14:56:55.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa58a674640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5840c3970 con 0x7fa5840719e0 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa58414d210 0x7fa5840c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa58414d210 0x7fa5840c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34774/0 (socket says 172.21.15.78:34774) 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 -- 172.21.15.78:0/3888990631 learned_addr learned my addr 172.21.15.78:0/3888990631 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa5837fe640 1 --2- 172.21.15.78:0/3888990631 >> v2:172.21.15.150:3300/0 conn(0x7fa5840719e0 0x7fa5840c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 -- 172.21.15.78:0/3888990631 >> v2:172.21.15.150:3300/0 conn(0x7fa5840719e0 msgr2=0x7fa5840c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 --2- 172.21.15.78:0/3888990631 >> v2:172.21.15.150:3300/0 conn(0x7fa5840719e0 0x7fa5840c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 -- 172.21.15.78:0/3888990631 >> v1:172.21.15.150:6789/0 conn(0x7fa5840c3d90 legacy=0x7fa584152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa583fff640 1 -- 172.21.15.78:0/3888990631 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa584156fc0 con 0x7fa58414d210 2024-02-19T14:56:55.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.758+0000 7fa5837fe640 1 --2- 172.21.15.78:0/3888990631 >> v2:172.21.15.150:3300/0 conn(0x7fa5840719e0 0x7fa5840c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.759+0000 7fa583fff640 1 --2- 172.21.15.78:0/3888990631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa5840c4290 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fa5840ab3a0 tx=0x7fa57402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313b5a739a71f5af server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.759+0000 7fa582ffd640 1 -- 172.21.15.78:0/3888990631 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574031870 con 0x7fa58414d210 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.759+0000 7fa582ffd640 1 -- 172.21.15.78:0/3888990631 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5740319d0 con 0x7fa58414d210 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.759+0000 7fa582ffd640 1 -- 172.21.15.78:0/3888990631 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574031d60 con 0x7fa58414d210 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 -- 172.21.15.78:0/3888990631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 msgr2=0x7fa5840c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 --2- 172.21.15.78:0/3888990631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa5840c4290 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fa5840ab3a0 tx=0x7fa57402b910 comp rx=0 tx=0).stop 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 -- 172.21.15.78:0/3888990631 shutdown_connections 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 --2- 172.21.15.78:0/3888990631 >> v2:172.21.15.150:3300/0 conn(0x7fa5840719e0 0x7fa5840c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 --2- 172.21.15.78:0/3888990631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa5840c4290 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 -- 172.21.15.78:0/3888990631 >> 172.21.15.78:0/3888990631 conn(0x7fa5840c5050 msgr2=0x7fa5840c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 -- 172.21.15.78:0/3888990631 shutdown_connections 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.760+0000 7fa58a674640 1 -- 172.21.15.78:0/3888990631 wait complete. 2024-02-19T14:56:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.761+0000 7fa58a674640 1 Processor -- start 2024-02-19T14:56:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.761+0000 7fa58a674640 1 -- start start 2024-02-19T14:56:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa58a674640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5840719e0 0x7fa5840c1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa58a674640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5840c3d90 0x7fa58406c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:55.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa58a674640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa58406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa58a674640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584151110 con 0x7fa58414d210 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa58a674640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58406bf90 con 0x7fa5840c3d90 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa58a674640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58406ce60 con 0x7fa5840719e0 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa588c11640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa58406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa588c11640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa58406c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34776/0 (socket says 172.21.15.78:34776) 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa588c11640 1 -- 172.21.15.78:0/416971230 learned_addr learned my addr 172.21.15.78:0/416971230 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa583fff640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5840719e0 0x7fa5840c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.762+0000 7fa5837fe640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5840c3d90 0x7fa58406c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa588c11640 1 -- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5840719e0 msgr2=0x7fa5840c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa588c11640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5840719e0 0x7fa5840c1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa588c11640 1 -- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5840c3d90 msgr2=0x7fa58406c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa588c11640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5840c3d90 0x7fa58406c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa588c11640 1 -- 172.21.15.78:0/416971230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740094d0 con 0x7fa58414d210 2024-02-19T14:56:55.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa588c11640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa58406c920 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00c730 tx=0x7fa57c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c003b70 con 0x7fa58414d210 2024-02-19T14:56:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c003cd0 con 0x7fa58414d210 2024-02-19T14:56:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c00aa90 con 0x7fa58414d210 2024-02-19T14:56:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5840c3660 con 0x7fa58414d210 2024-02-19T14:56:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.763+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa584151ba0 con 0x7fa58414d210 2024-02-19T14:56:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.764+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58406c100 con 0x7fa58414d210 2024-02-19T14:56:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.765+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa57c00abf0 con 0x7fa58414d210 2024-02-19T14:56:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.765+0000 7fa580ff9640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa57006b810 0x7fa57006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.765+0000 7fa583fff640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa57006b810 0x7fa57006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.765+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa57c089c20 con 0x7fa58414d210 2024-02-19T14:56:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.766+0000 7fa583fff640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa57006b810 0x7fa57006dcd0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa574036000 tx=0x7fa57403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:55.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:55.770+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa58406c100 con 0x7fa58414d210 2024-02-19T14:56:56.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.005+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa58406c100 con 0x7fa58414d210 2024-02-19T14:56:56.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.005+0000 7fa580ff9640 1 -- 172.21.15.78:0/416971230 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fa58406c100 con 0x7fa58414d210 2024-02-19T14:56:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:56.008 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:56:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:56:56.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:56.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:56:56.010 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.012 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:56:56.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.016 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:56:56.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.018 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:56:56.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa57006b810 msgr2=0x7fa57006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa57006b810 0x7fa57006dcd0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa574036000 tx=0x7fa57403c040 comp rx=0 tx=0).stop 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 msgr2=0x7fa58406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa58406c920 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00c730 tx=0x7fa57c00cbf0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 shutdown_connections 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa57006b810 0x7fa57006dcd0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa58414d210 0x7fa58406c920 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5840c3d90 0x7fa58406c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 --2- 172.21.15.78:0/416971230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa5840719e0 0x7fa5840c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 >> 172.21.15.78:0/416971230 conn(0x7fa5840c5050 msgr2=0x7fa5840c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 shutdown_connections 2024-02-19T14:56:56.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.007+0000 7fa58a674640 1 -- 172.21.15.78:0/416971230 wait complete. 2024-02-19T14:56:56.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.120+0000 7f5674807640 1 Processor -- start 2024-02-19T14:56:56.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.120+0000 7f5674807640 1 -- start start 2024-02-19T14:56:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f5674807640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56700c8500 0x7f56700c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f5674807640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56700c34a0 con 0x7f56700c2f50 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f5674807640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567006c8e0 con 0x7f56700c42e0 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f5674807640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567006c090 con 0x7f56700c8500 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56700c8500 0x7f56700c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f56700c2f50 0x7f567015f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47942/0 (socket says 172.21.15.78:47942) 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56700c8500 0x7f56700c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34782/0 (socket says 172.21.15.78:34782) 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566ed9d640 1 -- 172.21.15.78:0/1908564314 learned_addr learned my addr 172.21.15.78:0/1908564314 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566e59c640 1 -- 172.21.15.78:0/1908564314 >> v1:172.21.15.150:6789/0 conn(0x7f56700c42e0 legacy=0x7f5670158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566e59c640 1 -- 172.21.15.78:0/1908564314 >> v1:172.21.15.78:6789/0 conn(0x7f56700c2f50 legacy=0x7f567015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566e59c640 1 -- 172.21.15.78:0/1908564314 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567006cbc0 con 0x7f56700c8500 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.121+0000 7f566dd9b640 1 -- 172.21.15.78:0/1908564314 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203593218 0 0) 0x7f56700c34a0 con 0x7f56700c2f50 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f566e59c640 1 --2- 172.21.15.78:0/1908564314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c8500 0x7f56700c88e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f566002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff81dd314fbea06 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f566dd9b640 1 -- 172.21.15.78:0/1908564314 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566002e070 con 0x7f56700c8500 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f566dd9b640 1 -- 172.21.15.78:0/1908564314 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56600315a0 con 0x7f56700c8500 2024-02-19T14:56:56.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f566dd9b640 1 -- 172.21.15.78:0/1908564314 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660031930 con 0x7f56700c8500 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 -- 172.21.15.78:0/1908564314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c8500 msgr2=0x7f56700c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 --2- 172.21.15.78:0/1908564314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c8500 0x7f56700c88e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f566002b840 comp rx=0 tx=0).stop 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 -- 172.21.15.78:0/1908564314 shutdown_connections 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 --2- 172.21.15.78:0/1908564314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c8500 0x7f56700c88e0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 -- 172.21.15.78:0/1908564314 >> 172.21.15.78:0/1908564314 conn(0x7f56700aa300 msgr2=0x7f56700aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 -- 172.21.15.78:0/1908564314 shutdown_connections 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.122+0000 7f5674807640 1 -- 172.21.15.78:0/1908564314 wait complete. 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 Processor -- start 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 -- start start 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 0x7f567014ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56700c8500 0x7f56700ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5670149fc0 0x7f567014ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56701426d0 con 0x7f56700c2f50 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567014bf60 con 0x7f56700c8500 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f5674807640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567014b2a0 con 0x7f5670149fc0 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.124+0000 7f566e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 0x7f567014ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 0x7f567014ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34796/0 (socket says 172.21.15.78:34796) 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 -- 172.21.15.78:0/2454781264 learned_addr learned my addr 172.21.15.78:0/2454781264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5667fff640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56700c8500 0x7f56700ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566ed9d640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5670149fc0 0x7f567014ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 -- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5670149fc0 msgr2=0x7f567014ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5670149fc0 0x7f567014ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 -- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56700c8500 msgr2=0x7f56700ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56700c8500 0x7f56700ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 -- 172.21.15.78:0/2454781264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5660009400 con 0x7f56700c2f50 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5667fff640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56700c8500 0x7f56700ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:56.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566ed9d640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5670149fc0 0x7f567014ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f566e59c640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 0x7f567014ad60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f56600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660032900 con 0x7f56700c2f50 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5660032a60 con 0x7f56700c2f50 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56701eba40 con 0x7f56700c2f50 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660032e10 con 0x7f56700c2f50 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.125+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56701ebf70 con 0x7f56700c2f50 2024-02-19T14:56:56.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.126+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5670071c20 con 0x7f56700c2f50 2024-02-19T14:56:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.127+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f566004d3f0 con 0x7f56700c2f50 2024-02-19T14:56:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.127+0000 7f5666ffd640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565006b810 0x7f565006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.127+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f56600abd50 con 0x7f56700c2f50 2024-02-19T14:56:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.127+0000 7f5667fff640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565006b810 0x7f565006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.127+0000 7f5667fff640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565006b810 0x7f565006dcd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f56700c8280 tx=0x7f5658009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.131+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56600af030 con 0x7f56700c2f50 2024-02-19T14:56:56.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.304+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5670074990 con 0x7f56700c2f50 2024-02-19T14:56:56.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.305+0000 7f5666ffd640 1 -- 172.21.15.78:0/2454781264 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f5660076940 con 0x7f56700c2f50 2024-02-19T14:56:56.307 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:56.308 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":134,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:55.633967+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:56.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.306+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565006b810 msgr2=0x7f565006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.306+0000 7f5674807640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565006b810 0x7f565006dcd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f56700c8280 tx=0x7f5658009040 comp rx=0 tx=0).stop 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.306+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 msgr2=0x7f567014ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.306+0000 7f5674807640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 0x7f567014ad60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f56600046c0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 shutdown_connections 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f565006b810 0x7f565006dcd0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5670149fc0 0x7f567014ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56700c8500 0x7f56700ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 --2- 172.21.15.78:0/2454781264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56700c2f50 0x7f567014ad60 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 >> 172.21.15.78:0/2454781264 conn(0x7f56700aa300 msgr2=0x7f5670154a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 shutdown_connections 2024-02-19T14:56:56.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.307+0000 7f5674807640 1 -- 172.21.15.78:0/2454781264 wait complete. 2024-02-19T14:56:56.320 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:56:56.320 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:56:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.420+0000 7fceb09b5640 1 Processor -- start 2024-02-19T14:56:56.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.420+0000 7fceb09b5640 1 -- start start 2024-02-19T14:56:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceb09b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fceac06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceb09b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fceac06d910 0x7fceac06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceb09b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceac06cd20 con 0x7fceac152700 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceb09b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceac1551f0 con 0x7fceac06d910 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceb09b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac155490 con 0x7fceac0c8e00 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceaa59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fceac152700 0x7fceac0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47958/0 (socket says 172.21.15.78:47958) 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceaa59c640 1 -- 172.21.15.78:0/3587558277 learned_addr learned my addr 172.21.15.78:0/3587558277 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822262351 0 0) 0x7fceac06cd20 con 0x7fceac152700 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce94003520 con 0x7fceac152700 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fceac06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.421+0000 7fceaad9d640 1 --2- 172.21.15.78:0/3587558277 >> v2:172.21.15.150:3300/0 conn(0x7fceac06d910 0x7fceac06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2282807759 0 0) 0x7fce94003520 con 0x7fceac152700 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 msgr2=0x7fceac06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fceac06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 >> v2:172.21.15.150:3300/0 conn(0x7fceac06d910 msgr2=0x7fceac06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 --2- 172.21.15.78:0/3587558277 >> v2:172.21.15.150:3300/0 conn(0x7fceac06d910 0x7fceac06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac155770 con 0x7fceac152700 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fce9c0019f0 con 0x7fceac152700 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 >> v1:172.21.15.78:6789/0 conn(0x7fceac152700 legacy=0x7fceac0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fce94003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fce940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce94004d00 0x7fce94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac1551f0 con 0x7fce94003ef0 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac155490 con 0x7fce94004d00 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9400b9d0 con 0x7fceac0c8e00 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fce940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fceaa59c640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fce94003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fceaad9d640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce94004d00 0x7fce94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea9d9b640 1 -- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 msgr2=0x7fce94003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fce94003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea9d9b640 1 -- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce94004d00 msgr2=0x7fce94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce94004d00 0x7fce94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fcea9d9b640 1 -- 172.21.15.78:0/3587558277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94003520 con 0x7fce94003ef0 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.422+0000 7fceaa59c640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac0c8e00 0x7fce94003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fce940047c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009af0 tx=0x7fce940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d88896327ddfb6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea002c6b0 con 0x7fce94003ef0 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea002c810 con 0x7fce94003ef0 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fcea959a640 1 -- 172.21.15.78:0/3587558277 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea002cba0 con 0x7fce94003ef0 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fceb09b5640 1 -- 172.21.15.78:0/3587558277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 msgr2=0x7fce940047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3587558277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fce940047c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009af0 tx=0x7fce940057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.423+0000 7fceaad9d640 1 -- 172.21.15.78:0/3587558277 reap_dead start 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.424+0000 7fceb09b5640 1 -- 172.21.15.78:0/3587558277 shutdown_connections 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.424+0000 7fceb09b5640 1 -- 172.21.15.78:0/3587558277 >> 172.21.15.78:0/3587558277 conn(0x7fceac0cac90 msgr2=0x7fceac0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.424+0000 7fceb09b5640 1 -- 172.21.15.78:0/3587558277 shutdown_connections 2024-02-19T14:56:56.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.424+0000 7fceb09b5640 1 -- 172.21.15.78:0/3587558277 wait complete. 2024-02-19T14:56:56.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.425+0000 7fceb09b5640 1 Processor -- start 2024-02-19T14:56:56.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.425+0000 7fceb09b5640 1 -- start start 2024-02-19T14:56:56.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fceac1ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac1ee9c0 0x7fceac0ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceac0acad0 0x7fceac0c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0c2510 con 0x7fce94003ef0 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac157d50 con 0x7fceac0acad0 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac156e60 con 0x7fceac1ee9c0 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fceac1ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fceac1ee480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34810/0 (socket says 172.21.15.78:34810) 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 -- 172.21.15.78:0/3787349351 learned_addr learned my addr 172.21.15.78:0/3787349351 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaad9d640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceac0acad0 0x7fceac0c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac1ee9c0 0x7fceac0ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 -- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac1ee9c0 msgr2=0x7fceac0ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac1ee9c0 0x7fceac0ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 -- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceac0acad0 msgr2=0x7fceac0c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceac0acad0 0x7fceac0c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 -- 172.21.15.78:0/3787349351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac157140 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceaa59c640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fceac1ee480 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fce9c007b20 tx=0x7fce9c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c013420 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c013580 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0138f0 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac157420 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.426+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac1579e0 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.427+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac157ba0 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.428+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fce9c013a50 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.428+0000 7fce9affd640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fce8c06b810 0x7fce8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.428+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fce9c08a910 con 0x7fce94003ef0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.428+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fce8c06b810 0x7fce8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.428+0000 7fcea9d9b640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fce8c06b810 0x7fce8c06dcd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fcea0036000 tx=0x7fcea0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.432+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fce9c0554f0 con 0x7fce94003ef0 2024-02-19T14:56:56.481 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:56:56.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:56:56.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.607+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fceac074990 con 0x7fce94003ef0 2024-02-19T14:56:56.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.607+0000 7fce9affd640 1 -- 172.21.15.78:0/3787349351 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce9c058bb0 con 0x7fce94003ef0 2024-02-19T14:56:56.610 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:56:56.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fce8c06b810 msgr2=0x7fce8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fce8c06b810 0x7fce8c06dcd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fcea0036000 tx=0x7fcea0030000 comp rx=0 tx=0).stop 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 msgr2=0x7fceac1ee480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fceac1ee480 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fce9c007b20 tx=0x7fce9c00e530 comp rx=0 tx=0).stop 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 shutdown_connections 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fce8c06b810 0x7fce8c06dcd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceac0acad0 0x7fceac0c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceac1ee9c0 0x7fceac0ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 --2- 172.21.15.78:0/3787349351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce94003ef0 0x7fceac1ee480 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.609+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 >> 172.21.15.78:0/3787349351 conn(0x7fceac0cac90 msgr2=0x7fceac0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.610+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 shutdown_connections 2024-02-19T14:56:56.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.610+0000 7fceb09b5640 1 -- 172.21.15.78:0/3787349351 wait complete. 2024-02-19T14:56:56.623 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:56:56.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.720+0000 7f237e960640 1 Processor -- start 2024-02-19T14:56:56.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.720+0000 7f237e960640 1 -- start start 2024-02-19T14:56:56.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237e960640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f23780718f0 0x7f2378071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237e960640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2378159130 con 0x7f2378155070 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237e960640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23780ad030 con 0x7f23780acc50 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237e960640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23780c2cf0 con 0x7f23780718f0 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237cefd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f23780718f0 0x7f2378071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f23777fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2378155070 0x7f2378155500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47962/0 (socket says 172.21.15.78:47962) 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f23777fe640 1 -- 172.21.15.78:0/3205921901 learned_addr learned my addr 172.21.15.78:0/3205921901 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f2376ffd640 1 -- 172.21.15.78:0/3205921901 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056957750 0 0) 0x7f2378159130 con 0x7f2378155070 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f2376ffd640 1 -- 172.21.15.78:0/3205921901 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2360003520 con 0x7f2378155070 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237cefd640 1 -- 172.21.15.78:0/3205921901 >> v1:172.21.15.150:6789/0 conn(0x7f23780acc50 legacy=0x7f23780c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237cefd640 1 -- 172.21.15.78:0/3205921901 >> v1:172.21.15.78:6789/0 conn(0x7f2378155070 legacy=0x7f2378155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.721+0000 7f237cefd640 1 -- 172.21.15.78:0/3205921901 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23781595c0 con 0x7f23780718f0 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.722+0000 7f237cefd640 1 --2- 172.21.15.78:0/3205921901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780718f0 0x7f2378071d60 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2370009d90 tx=0x7f237002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac2c93c8949d6be server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.722+0000 7f2376ffd640 1 -- 172.21.15.78:0/3205921901 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237002cbd0 con 0x7f23780718f0 2024-02-19T14:56:56.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.722+0000 7f2376ffd640 1 -- 172.21.15.78:0/3205921901 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237002cd30 con 0x7f23780718f0 2024-02-19T14:56:56.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.722+0000 7f2376ffd640 1 -- 172.21.15.78:0/3205921901 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237002cea0 con 0x7f23780718f0 2024-02-19T14:56:56.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 -- 172.21.15.78:0/3205921901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780718f0 msgr2=0x7f2378071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 --2- 172.21.15.78:0/3205921901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780718f0 0x7f2378071d60 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2370009d90 tx=0x7f237002be20 comp rx=0 tx=0).stop 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 -- 172.21.15.78:0/3205921901 shutdown_connections 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 --2- 172.21.15.78:0/3205921901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780718f0 0x7f2378071d60 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 -- 172.21.15.78:0/3205921901 >> 172.21.15.78:0/3205921901 conn(0x7f23780c7800 msgr2=0x7f23780c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 -- 172.21.15.78:0/3205921901 shutdown_connections 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.723+0000 7f237e960640 1 -- 172.21.15.78:0/3205921901 wait complete. 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.724+0000 7f237e960640 1 Processor -- start 2024-02-19T14:56:56.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.724+0000 7f237e960640 1 -- start start 2024-02-19T14:56:56.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23780718f0 0x7f2378133930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 0x7f23781ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2378155070 0x7f2378151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23780c2030 con 0x7f23780acc50 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2378136d00 con 0x7f2378155070 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23780c5480 con 0x7f23780718f0 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 0x7f23781ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 0x7f23781ebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34816/0 (socket says 172.21.15.78:34816) 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 -- 172.21.15.78:0/4063846180 learned_addr learned my addr 172.21.15.78:0/4063846180 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 -- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23780718f0 msgr2=0x7f2378133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f2377fff640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23780718f0 0x7f2378133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237cefd640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2378155070 0x7f2378151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23780718f0 0x7f2378133930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 -- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2378155070 msgr2=0x7f2378151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2378155070 0x7f2378151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 -- 172.21.15.78:0/4063846180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370009a70 con 0x7f23780acc50 2024-02-19T14:56:56.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237cefd640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2378155070 0x7f2378151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f23777fe640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 0x7f23781ebce0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c390 tx=0x7f236c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c005ce0 con 0x7f23780acc50 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c005e40 con 0x7f23780acc50 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23780c1b60 con 0x7f23780acc50 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c0072f0 con 0x7f23780acc50 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.725+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23780c5a00 con 0x7f23780acc50 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.726+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237806c820 con 0x7f23780acc50 2024-02-19T14:56:56.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.727+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f236c007450 con 0x7f23780acc50 2024-02-19T14:56:56.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.727+0000 7f2374ff9640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f236406b810 0x7f236406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:56.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.727+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f236c089560 con 0x7f23780acc50 2024-02-19T14:56:56.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.727+0000 7f2377fff640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f236406b810 0x7f236406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:56.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.728+0000 7f2377fff640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f236406b810 0x7f236406dcd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2368003d90 tx=0x7f2368009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:56.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.732+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f236c0541d0 con 0x7f23780acc50 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.904+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2378074990 con 0x7f23780acc50 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.904+0000 7f2374ff9640 1 -- 172.21.15.78:0/4063846180 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v134) v1 ==== 84+0+65 (secure 0 0 0) 0x7f236c017060 con 0x7f23780acc50 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:56:56.907 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:56:56.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f236406b810 msgr2=0x7f236406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f236406b810 0x7f236406dcd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2368003d90 tx=0x7f2368009040 comp rx=0 tx=0).stop 2024-02-19T14:56:56.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 msgr2=0x7f23781ebce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:56.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 0x7f23781ebce0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c390 tx=0x7f236c00c850 comp rx=0 tx=0).stop 2024-02-19T14:56:56.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 shutdown_connections 2024-02-19T14:56:56.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f236406b810 0x7f236406dcd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23780acc50 0x7f23781ebce0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2378155070 0x7f2378151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 --2- 172.21.15.78:0/4063846180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23780718f0 0x7f2378133930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:56.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 >> 172.21.15.78:0/4063846180 conn(0x7f23780c7800 msgr2=0x7f237806be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:56.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 shutdown_connections 2024-02-19T14:56:56.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:56.906+0000 7f237e960640 1 -- 172.21.15.78:0/4063846180 wait complete. 2024-02-19T14:56:56.920 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:56:56.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:56:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.021+0000 7fc183639640 1 Processor -- start 2024-02-19T14:56:57.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.021+0000 7fc183639640 1 -- start start 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc183639640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc17c0c8730 0x7fc17c06d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc183639640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc17c154ea0 0x7fc17c155330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc183639640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c06dc70 con 0x7fc17c152670 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc183639640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c158ef0 con 0x7fc17c0c8730 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc183639640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c0be190 con 0x7fc17c154ea0 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc17c0c8730 0x7fc17c06d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc17c0c8730 0x7fc17c06d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34830/0 (socket says 172.21.15.78:34830) 2024-02-19T14:56:57.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 -- 172.21.15.78:0/1798892096 learned_addr learned my addr 172.21.15.78:0/1798892096 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc181bd6640 1 --2- 172.21.15.78:0/1798892096 >> v2:172.21.15.150:3300/0 conn(0x7fc17c154ea0 0x7fc17c155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 -- 172.21.15.78:0/1798892096 >> v2:172.21.15.150:3300/0 conn(0x7fc17c154ea0 msgr2=0x7fc17c155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 --2- 172.21.15.78:0/1798892096 >> v2:172.21.15.150:3300/0 conn(0x7fc17c154ea0 0x7fc17c155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 -- 172.21.15.78:0/1798892096 >> v1:172.21.15.150:6789/0 conn(0x7fc17c152670 legacy=0x7fc17c0c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc180bd4640 1 -- 172.21.15.78:0/1798892096 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c06dea0 con 0x7fc17c0c8730 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.022+0000 7fc181bd6640 1 --2- 172.21.15.78:0/1798892096 >> v2:172.21.15.150:3300/0 conn(0x7fc17c154ea0 0x7fc17c155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.023+0000 7fc180bd4640 1 --2- 172.21.15.78:0/1798892096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c0c8730 0x7fc17c06d730 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0c1890 tx=0x7fc16c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed8325a6c443522 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.023+0000 7fc173fff640 1 -- 172.21.15.78:0/1798892096 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c0317e0 con 0x7fc17c0c8730 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.023+0000 7fc173fff640 1 -- 172.21.15.78:0/1798892096 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16c031940 con 0x7fc17c0c8730 2024-02-19T14:56:57.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.023+0000 7fc173fff640 1 -- 172.21.15.78:0/1798892096 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031cd0 con 0x7fc17c0c8730 2024-02-19T14:56:57.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 -- 172.21.15.78:0/1798892096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c0c8730 msgr2=0x7fc17c06d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 --2- 172.21.15.78:0/1798892096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c0c8730 0x7fc17c06d730 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0c1890 tx=0x7fc16c02b880 comp rx=0 tx=0).stop 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 -- 172.21.15.78:0/1798892096 shutdown_connections 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 --2- 172.21.15.78:0/1798892096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c0c8730 0x7fc17c06d730 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 --2- 172.21.15.78:0/1798892096 >> v2:172.21.15.150:3300/0 conn(0x7fc17c154ea0 0x7fc17c155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 -- 172.21.15.78:0/1798892096 >> 172.21.15.78:0/1798892096 conn(0x7fc17c072790 msgr2=0x7fc17c072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 -- 172.21.15.78:0/1798892096 shutdown_connections 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.024+0000 7fc183639640 1 -- 172.21.15.78:0/1798892096 wait complete. 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.025+0000 7fc183639640 1 Processor -- start 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.025+0000 7fc183639640 1 -- start start 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc17c0c8730 0x7fc17c1ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc17c152670 0x7fc17c153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 0x7fc17c14ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0ae570 con 0x7fc17c154ea0 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c1eead0 con 0x7fc17c152670 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0adf20 con 0x7fc17c0c8730 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 0x7fc17c14ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 0x7fc17c14ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34846/0 (socket says 172.21.15.78:34846) 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 -- 172.21.15.78:0/1080058408 learned_addr learned my addr 172.21.15.78:0/1080058408 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc180bd4640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc17c152670 0x7fc17c153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc1813d5640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc17c0c8730 0x7fc17c1ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 -- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc17c0c8730 msgr2=0x7fc17c1ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc17c0c8730 0x7fc17c1ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 -- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc17c152670 msgr2=0x7fc17c153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc17c152670 0x7fc17c153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 -- 172.21.15.78:0/1080058408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0094d0 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc181bd6640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 0x7fc17c14ab40 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fc17400c7c0 tx=0x7fc17400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174004230 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc174004390 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174007860 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.026+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c0ae200 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.027+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c1ee760 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.028+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c1572d0 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.028+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc1740079c0 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.028+0000 7fc171ffb640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc16006b810 0x7fc16006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.028+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc17408a230 con 0x7fc17c154ea0 2024-02-19T14:56:57.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.028+0000 7fc1813d5640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc16006b810 0x7fc16006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.029+0000 7fc1813d5640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc16006b810 0x7fc16006dcd0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc178001ca0 tx=0x7fc178009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:57.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.032+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc17c1572d0 con 0x7fc17c154ea0 2024-02-19T14:56:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.204+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc17c1572d0 con 0x7fc17c154ea0 2024-02-19T14:56:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.644+0000 7fc171ffb640 1 -- 172.21.15.78:0/1080058408 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v135) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc17c1572d0 con 0x7fc17c154ea0 2024-02-19T14:56:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc16006b810 msgr2=0x7fc16006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc16006b810 0x7fc16006dcd0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc178001ca0 tx=0x7fc178009040 comp rx=0 tx=0).stop 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 msgr2=0x7fc17c14ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 0x7fc17c14ab40 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fc17400c7c0 tx=0x7fc17400cc80 comp rx=0 tx=0).stop 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 shutdown_connections 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc16006b810 0x7fc16006dcd0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc17c154ea0 0x7fc17c14ab40 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc17c152670 0x7fc17c153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 --2- 172.21.15.78:0/1080058408 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc17c0c8730 0x7fc17c1ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 >> 172.21.15.78:0/1080058408 conn(0x7fc17c072790 msgr2=0x7fc17c155650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 shutdown_connections 2024-02-19T14:56:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.646+0000 7fc183639640 1 -- 172.21.15.78:0/1080058408 wait complete. 2024-02-19T14:56:57.662 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:56:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.767+0000 7f1a947d6640 1 Processor -- start 2024-02-19T14:56:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.767+0000 7f1a947d6640 1 -- start start 2024-02-19T14:56:57.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a947d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1a8c156de0 0x7f1a8c0ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a947d6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c157270 con 0x7f1a8c0ad0d0 2024-02-19T14:56:57.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a947d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0c5bf0 con 0x7f1a8c0ab600 2024-02-19T14:56:57.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a947d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c153420 con 0x7f1a8c156de0 2024-02-19T14:56:57.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a92572640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1a8c0ad0d0 0x7f1a8c152d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45888/0 (socket says 172.21.15.78:45888) 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a92572640 1 -- 172.21.15.78:0/1156537542 learned_addr learned my addr 172.21.15.78:0/1156537542 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a92d73640 1 --2- 172.21.15.78:0/1156537542 >> v2:172.21.15.150:3300/0 conn(0x7f1a8c156de0 0x7f1a8c0ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086418717 0 0) 0x7f1a8c157270 con 0x7f1a8c0ad0d0 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.768+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a78003520 con 0x7f1a8c0ad0d0 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72966137 0 0) 0x7f1a7c001110 con 0x7f1a8c0ab600 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8c157270 con 0x7f1a8c0ab600 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263927823 0 0) 0x7f1a78003520 con 0x7f1a8c0ad0d0 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 >> v2:172.21.15.150:3300/0 conn(0x7f1a8c156de0 msgr2=0x7f1a8c0ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 --2- 172.21.15.78:0/1156537542 >> v2:172.21.15.150:3300/0 conn(0x7f1a8c156de0 0x7f1a8c0ad4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 >> v1:172.21.15.150:6789/0 conn(0x7f1a8c0ab600 legacy=0x7f1a8c14c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0c7fe0 con 0x7f1a8c0ad0d0 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1a880019f0 con 0x7f1a8c0ad0d0 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 >> v1:172.21.15.78:6789/0 conn(0x7f1a8c0ad0d0 legacy=0x7f1a8c152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a78003b50 0x7f1a78003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78005240 0x7f1a7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c153420 con 0x7f1a78003b50 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7800c180 con 0x7f1a780047c0 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7800c340 con 0x7f1a78005240 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92572640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a78003b50 0x7f1a78003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a91d71640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92d73640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78005240 0x7f1a7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92572640 1 -- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78005240 msgr2=0x7f1a7800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92572640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78005240 0x7f1a7800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92572640 1 -- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a780047c0 msgr2=0x7f1a78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92572640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a780047c0 0x7f1a78004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92572640 1 -- 172.21.15.78:0/1156537542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a78003520 con 0x7f1a78003b50 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92d73640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78005240 0x7f1a7800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.769+0000 7f1a92d73640 1 -- 172.21.15.78:0/1156537542 reap_dead start 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.770+0000 7f1a92572640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a78003b50 0x7f1a78003f50 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800a700 tx=0x7f1a780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e17091cfc8bc7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.770+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88034630 con 0x7f1a78003b50 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.770+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8802f070 con 0x7f1a78003b50 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.770+0000 7f1a91570640 1 -- 172.21.15.78:0/1156537542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88034a30 con 0x7f1a78003b50 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 -- 172.21.15.78:0/1156537542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a78003b50 msgr2=0x7f1a78003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a78003b50 0x7f1a78003f50 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800a700 tx=0x7f1a780057f0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 -- 172.21.15.78:0/1156537542 shutdown_connections 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1156537542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a78003b50 0x7f1a78003f50 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 -- 172.21.15.78:0/1156537542 >> 172.21.15.78:0/1156537542 conn(0x7f1a8c06b6a0 msgr2=0x7f1a8c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 -- 172.21.15.78:0/1156537542 shutdown_connections 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.771+0000 7f1a947d6640 1 -- 172.21.15.78:0/1156537542 wait complete. 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 Processor -- start 2024-02-19T14:56:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 -- start start 2024-02-19T14:56:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a78003b50 0x7f1a8c0a7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a8c0bf310 0x7f1a8c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 0x7f1a8c14eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0c1a40 con 0x7f1a8c14e640 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0c2350 con 0x7f1a78003b50 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.772+0000 7f1a947d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c14f030 con 0x7f1a8c0bf310 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 0x7f1a8c14eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 0x7f1a8c14eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41484/0 (socket says 172.21.15.78:41484) 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 -- 172.21.15.78:0/1068391489 learned_addr learned my addr 172.21.15.78:0/1068391489 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92572640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a78003b50 0x7f1a8c0a7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a91d71640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a8c0bf310 0x7f1a8c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 -- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a8c0bf310 msgr2=0x7f1a8c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a8c0bf310 0x7f1a8c0710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 -- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a78003b50 msgr2=0x7f1a8c0a7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a78003b50 0x7f1a8c0a7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 -- 172.21.15.78:0/1068391489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0a7ad0 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a91d71640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a8c0bf310 0x7f1a8c0710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a92d73640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 0x7f1a8c14eaf0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0ac4a0 tx=0x7f1a84010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400e8d0 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8400ea30 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c0a7d00 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400edc0 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.773+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c0aa450 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.774+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c067ed0 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.775+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1a84020410 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.775+0000 7f1a82ffd640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a7006b840 0x7f1a7006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.775+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f1a840899f0 con 0x7f1a8c14e640 2024-02-19T14:56:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.775+0000 7f1a92572640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a7006b840 0x7f1a7006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.776+0000 7f1a92572640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a7006b840 0x7f1a7006dd00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1a88034000 tx=0x7f1a8802f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.780+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a84054460 con 0x7f1a8c14e640 2024-02-19T14:56:57.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:57.953+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1a8c074990 con 0x7f1a8c14e640 2024-02-19T14:56:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.649+0000 7f1a82ffd640 1 -- 172.21.15.78:0/1068391489 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v136) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1a84057b20 con 0x7f1a8c14e640 2024-02-19T14:56:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a7006b840 msgr2=0x7f1a7006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a7006b840 0x7f1a7006dd00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1a88034000 tx=0x7f1a8802f000 comp rx=0 tx=0).stop 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 msgr2=0x7f1a8c14eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 0x7f1a8c14eaf0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0ac4a0 tx=0x7f1a84010c20 comp rx=0 tx=0).stop 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 shutdown_connections 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a7006b840 0x7f1a7006dd00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a8c14e640 0x7f1a8c14eaf0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a8c0bf310 0x7f1a8c0710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 --2- 172.21.15.78:0/1068391489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a78003b50 0x7f1a8c0a7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 >> 172.21.15.78:0/1068391489 conn(0x7f1a8c06b6a0 msgr2=0x7f1a8c1536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:58.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 shutdown_connections 2024-02-19T14:56:58.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.652+0000 7f1a947d6640 1 -- 172.21.15.78:0/1068391489 wait complete. 2024-02-19T14:56:58.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:56:58.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.777+0000 7fbae6372640 1 Processor -- start 2024-02-19T14:56:58.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.777+0000 7fbae6372640 1 -- start start 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae6372640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbae00add70 0x7fbae00c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae6372640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00c2f60 0x7fbae006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae6372640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbae006f530 0x7fbae00705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae6372640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae00ae670 con 0x7fbae006f530 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae6372640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae00716b0 con 0x7fbae00add70 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae6372640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae00c33a0 con 0x7fbae00c2f60 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae490f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbae006f530 0x7fbae00705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae490f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbae006f530 0x7fbae00705b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41490/0 (socket says 172.21.15.78:41490) 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbae490f640 1 -- 172.21.15.78:0/4178051686 learned_addr learned my addr 172.21.15.78:0/4178051686 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.778+0000 7fbadffff640 1 --2- 172.21.15.78:0/4178051686 >> v2:172.21.15.150:3300/0 conn(0x7fbae00add70 0x7fbae00c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbadf7fe640 1 --2- 172.21.15.78:0/4178051686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00c2f60 0x7fbae006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbae490f640 1 -- 172.21.15.78:0/4178051686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00c2f60 msgr2=0x7fbae006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbae490f640 1 --2- 172.21.15.78:0/4178051686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00c2f60 0x7fbae006eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbae490f640 1 -- 172.21.15.78:0/4178051686 >> v2:172.21.15.150:3300/0 conn(0x7fbae00add70 msgr2=0x7fbae00c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbae490f640 1 --2- 172.21.15.78:0/4178051686 >> v2:172.21.15.150:3300/0 conn(0x7fbae00add70 0x7fbae00c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbae490f640 1 -- 172.21.15.78:0/4178051686 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0153780 con 0x7fbae006f530 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbadffff640 1 --2- 172.21.15.78:0/4178051686 >> v2:172.21.15.150:3300/0 conn(0x7fbae00add70 0x7fbae00c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbae490f640 1 --2- 172.21.15.78:0/4178051686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae006f530 0x7fbae00705b0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbae00c4fc0 tx=0x7fbad802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d52d39bdc414b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.779+0000 7fbadeffd640 1 -- 172.21.15.78:0/4178051686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad802c9e0 con 0x7fbae006f530 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbadeffd640 1 -- 172.21.15.78:0/4178051686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad802cb40 con 0x7fbae006f530 2024-02-19T14:56:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbadeffd640 1 -- 172.21.15.78:0/4178051686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad80343f0 con 0x7fbae006f530 2024-02-19T14:56:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 -- 172.21.15.78:0/4178051686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae006f530 msgr2=0x7fbae00705b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 --2- 172.21.15.78:0/4178051686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae006f530 0x7fbae00705b0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fbae00c4fc0 tx=0x7fbad802be40 comp rx=0 tx=0).stop 2024-02-19T14:56:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 -- 172.21.15.78:0/4178051686 shutdown_connections 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 --2- 172.21.15.78:0/4178051686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae006f530 0x7fbae00705b0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 --2- 172.21.15.78:0/4178051686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00c2f60 0x7fbae006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 --2- 172.21.15.78:0/4178051686 >> v2:172.21.15.150:3300/0 conn(0x7fbae00add70 0x7fbae00c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 -- 172.21.15.78:0/4178051686 >> 172.21.15.78:0/4178051686 conn(0x7fbae00c85e0 msgr2=0x7fbae00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 -- 172.21.15.78:0/4178051686 shutdown_connections 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.780+0000 7fbae6372640 1 -- 172.21.15.78:0/4178051686 wait complete. 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.781+0000 7fbae6372640 1 Processor -- start 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.781+0000 7fbae6372640 1 -- start start 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae6372640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae006f530 0x7fbae006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae6372640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00add70 0x7fbae00c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae6372640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 0x7fbae006ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae6372640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae01eb1c0 con 0x7fbae00c2f60 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae6372640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae006c0c0 con 0x7fbae006f530 2024-02-19T14:56:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae6372640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae006c3a0 con 0x7fbae00add70 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 0x7fbae006ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 0x7fbae006ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41492/0 (socket says 172.21.15.78:41492) 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 -- 172.21.15.78:0/173335925 learned_addr learned my addr 172.21.15.78:0/173335925 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbadf7fe640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00add70 0x7fbae00c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbadffff640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae006f530 0x7fbae006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 -- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00add70 msgr2=0x7fbae00c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00add70 0x7fbae00c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 -- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae006f530 msgr2=0x7fbae006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.782+0000 7fbae490f640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae006f530 0x7fbae006dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbae490f640 1 -- 172.21.15.78:0/173335925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009b30 con 0x7fbae00c2f60 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbadf7fe640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00add70 0x7fbae00c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbadffff640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae006f530 0x7fbae006dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbae490f640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 0x7fbae006ce80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbad8036040 tx=0x7fbad8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad8009510 con 0x7fbae00c2f60 2024-02-19T14:56:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad803e450 con 0x7fbae00c2f60 2024-02-19T14:56:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad803e800 con 0x7fbae00c2f60 2024-02-19T14:56:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae006c680 con 0x7fbae00c2f60 2024-02-19T14:56:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.783+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae00c20a0 con 0x7fbae00c2f60 2024-02-19T14:56:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.784+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae00676a0 con 0x7fbae00c2f60 2024-02-19T14:56:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.784+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbad803e960 con 0x7fbae00c2f60 2024-02-19T14:56:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.785+0000 7fbadcff9640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbacc06b840 0x7fbacc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.785+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fbad80ad0e0 con 0x7fbae00c2f60 2024-02-19T14:56:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.785+0000 7fbadffff640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbacc06b840 0x7fbacc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.785+0000 7fbadffff640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbacc06b840 0x7fbacc06dd00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbad0006fd0 tx=0x7fbad0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:58.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.789+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbad8077b10 con 0x7fbae00c2f60 2024-02-19T14:56:59.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:58.998+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbae00a9540 con 0x7fbae00c2f60 2024-02-19T14:56:59.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.651+0000 7fbadcff9640 1 -- 172.21.15.78:0/173335925 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v114) v1 ==== 174+0+0 (secure 0 0 0) 0x7fbad807b1d0 con 0x7fbae00c2f60 2024-02-19T14:56:59.654 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 21 and data pool 22 2024-02-19T14:56:59.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.655+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbacc06b840 msgr2=0x7fbacc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.655+0000 7fbae6372640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbacc06b840 0x7fbacc06dd00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbad0006fd0 tx=0x7fbad0008040 comp rx=0 tx=0).stop 2024-02-19T14:56:59.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.655+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 msgr2=0x7fbae006ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.655+0000 7fbae6372640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 0x7fbae006ce80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fbad8036040 tx=0x7fbad8004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:59.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.657+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 shutdown_connections 2024-02-19T14:56:59.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.657+0000 7fbae6372640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbacc06b840 0x7fbacc06dd00 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.657+0000 7fbae6372640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbae00c2f60 0x7fbae006ce80 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.657+0000 7fbae6372640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbae00add70 0x7fbae00c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.657+0000 7fbae6372640 1 --2- 172.21.15.78:0/173335925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae006f530 0x7fbae006dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.657+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 >> 172.21.15.78:0/173335925 conn(0x7fbae00c85e0 msgr2=0x7fbae00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:59.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.659+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 shutdown_connections 2024-02-19T14:56:59.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.660+0000 7fbae6372640 1 -- 172.21.15.78:0/173335925 wait complete. 2024-02-19T14:56:59.689 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:56:59.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.789+0000 7fa61e6ff640 1 Processor -- start 2024-02-19T14:56:59.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.789+0000 7fa61e6ff640 1 -- start start 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61e6ff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6180c7700 0x7fa61806cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61e6ff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6181586a0 con 0x7fa6180c7280 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61e6ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa618158980 con 0x7fa618152830 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61e6ff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180c6550 con 0x7fa6180c7700 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61cc9c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6180c7700 0x7fa61806cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa6177fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa6180c7280 0x7fa61806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45892/0 (socket says 172.21.15.78:45892) 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61cc9c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6180c7700 0x7fa61806cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41498/0 (socket says 172.21.15.78:41498) 2024-02-19T14:56:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa6177fe640 1 -- 172.21.15.78:0/1692981781 learned_addr learned my addr 172.21.15.78:0/1692981781 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:56:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.790+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1692981781 >> v1:172.21.15.150:6789/0 conn(0x7fa618152830 legacy=0x7fa6180c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:56:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.791+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1692981781 >> v1:172.21.15.78:6789/0 conn(0x7fa6180c7280 legacy=0x7fa61806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.791+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1692981781 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618158c60 con 0x7fa6180c7700 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.791+0000 7fa61cc9c640 1 --2- 172.21.15.78:0/1692981781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6180c7700 0x7fa61806cc80 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa610009d90 tx=0x7fa61002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1ed93faa185410 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.791+0000 7fa616ffd640 1 -- 172.21.15.78:0/1692981781 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61002cbd0 con 0x7fa6180c7700 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.791+0000 7fa616ffd640 1 -- 172.21.15.78:0/1692981781 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61002cd30 con 0x7fa6180c7700 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.791+0000 7fa616ffd640 1 -- 172.21.15.78:0/1692981781 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61002cea0 con 0x7fa6180c7700 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1692981781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6180c7700 msgr2=0x7fa61806cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1692981781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6180c7700 0x7fa61806cc80 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa610009d90 tx=0x7fa61002be20 comp rx=0 tx=0).stop 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1692981781 shutdown_connections 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1692981781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6180c7700 0x7fa61806cc80 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1692981781 >> 172.21.15.78:0/1692981781 conn(0x7fa6180ca8a0 msgr2=0x7fa6180cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1692981781 shutdown_connections 2024-02-19T14:56:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.792+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1692981781 wait complete. 2024-02-19T14:56:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.793+0000 7fa61e6ff640 1 Processor -- start 2024-02-19T14:56:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.793+0000 7fa61e6ff640 1 -- start start 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61e6ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c7280 0x7fa61814cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61e6ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6180c7700 0x7fa618157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61e6ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 0x7fa6181eb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61e6ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618159320 con 0x7fa618152830 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61e6ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180aafc0 con 0x7fa6180c7280 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61e6ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180abcc0 con 0x7fa6180c7700 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 0x7fa6181eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 0x7fa6181eb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41508/0 (socket says 172.21.15.78:41508) 2024-02-19T14:56:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1552917192 learned_addr learned my addr 172.21.15.78:0/1552917192 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa617fff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c7280 0x7fa61814cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa6177fe640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6180c7700 0x7fa618157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6180c7700 msgr2=0x7fa618157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6180c7700 0x7fa618157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c7280 msgr2=0x7fa61814cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c7280 0x7fa61814cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 -- 172.21.15.78:0/1552917192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610009a70 con 0x7fa618152830 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa6177fe640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6180c7700 0x7fa618157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa61cc9c640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 0x7fa6181eb770 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa610037040 tx=0x7fa610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.794+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61002cea0 con 0x7fa618152830 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.795+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610040650 con 0x7fa618152830 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.795+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6180aa720 con 0x7fa618152830 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.795+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610040a00 con 0x7fa618152830 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.795+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6180aace0 con 0x7fa618152830 2024-02-19T14:56:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.796+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6180c8b40 con 0x7fa618152830 2024-02-19T14:56:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.796+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa610040b60 con 0x7fa618152830 2024-02-19T14:56:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.796+0000 7fa614ff9640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa60406b810 0x7fa60406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:56:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.796+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6100ad110 con 0x7fa618152830 2024-02-19T14:56:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.796+0000 7fa617fff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa60406b810 0x7fa60406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:56:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.797+0000 7fa617fff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa60406b810 0x7fa60406dcd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa608006fd0 tx=0x7fa608008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:56:59.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.801+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa610077a30 con 0x7fa618152830 2024-02-19T14:56:59.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.974+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa618074990 con 0x7fa618152830 2024-02-19T14:56:59.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.975+0000 7fa614ff9640 1 -- 172.21.15.78:0/1552917192 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16893 (secure 0 0 0) 0x7fa61007b0f0 con 0x7fa618152830 2024-02-19T14:56:59.978 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:56:59.978 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":137,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:58.999836+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:57.223056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:57.972812+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa60406b810 msgr2=0x7fa60406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa60406b810 0x7fa60406dcd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa608006fd0 tx=0x7fa608008040 comp rx=0 tx=0).stop 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 msgr2=0x7fa6181eb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 0x7fa6181eb770 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fa610037040 tx=0x7fa610004a40 comp rx=0 tx=0).stop 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 shutdown_connections 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa60406b810 0x7fa60406dcd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa618152830 0x7fa6181eb770 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6180c7700 0x7fa618157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 --2- 172.21.15.78:0/1552917192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c7280 0x7fa61814cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:56:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 >> 172.21.15.78:0/1552917192 conn(0x7fa6180ca8a0 msgr2=0x7fa618150050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:56:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 shutdown_connections 2024-02-19T14:56:59.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:56:59.977+0000 7fa61e6ff640 1 -- 172.21.15.78:0/1552917192 wait complete. 2024-02-19T14:56:59.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:00.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.089+0000 7fd3493e2640 1 Processor -- start 2024-02-19T14:57:00.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.089+0000 7fd3493e2640 1 -- start start 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd3493e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd3493e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3441551f0 con 0x7fd3440c5730 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd3493e2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd344155490 con 0x7fd3440c8e00 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd3493e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd344155770 con 0x7fd344152700 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd3427fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd3440c5730 0x7fd34406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45900/0 (socket says 172.21.15.78:45900) 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd3427fc640 1 -- 172.21.15.78:0/2520510626 learned_addr learned my addr 172.21.15.78:0/2520510626 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd342ffd640 1 --2- 172.21.15.78:0/2520510626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875201944 0 0) 0x7fd3441551f0 con 0x7fd3440c5730 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c003520 con 0x7fd3440c5730 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681671567 0 0) 0x7fd344155490 con 0x7fd3440c8e00 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3441551f0 con 0x7fd3440c8e00 2024-02-19T14:57:00.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1595696208 0 0) 0x7fd32c003520 con 0x7fd3440c5730 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd342ffd640 1 -- 172.21.15.78:0/2520510626 >> v1:172.21.15.150:6789/0 conn(0x7fd3440c8e00 legacy=0x7fd34415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd342ffd640 1 -- 172.21.15.78:0/2520510626 >> v1:172.21.15.78:6789/0 conn(0x7fd3440c5730 legacy=0x7fd34406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.090+0000 7fd342ffd640 1 -- 172.21.15.78:0/2520510626 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344155a50 con 0x7fd344152700 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.091+0000 7fd342ffd640 1 --2- 172.21.15.78:0/2520510626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c51f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd3340097d0 tx=0x7fd33402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bd43aad65607f52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.091+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3380015f0 con 0x7fd344152700 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.091+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33402e050 con 0x7fd344152700 2024-02-19T14:57:00.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.091+0000 7fd341ffb640 1 -- 172.21.15.78:0/2520510626 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd334030710 con 0x7fd344152700 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 -- 172.21.15.78:0/2520510626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 msgr2=0x7fd3440c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2520510626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c51f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd3340097d0 tx=0x7fd33402b800 comp rx=0 tx=0).stop 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 -- 172.21.15.78:0/2520510626 shutdown_connections 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2520510626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c51f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 -- 172.21.15.78:0/2520510626 >> 172.21.15.78:0/2520510626 conn(0x7fd3440cac90 msgr2=0x7fd3440cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 -- 172.21.15.78:0/2520510626 shutdown_connections 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.092+0000 7fd3493e2640 1 -- 172.21.15.78:0/2520510626 wait complete. 2024-02-19T14:57:00.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.093+0000 7fd3493e2640 1 Processor -- start 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.093+0000 7fd3493e2640 1 -- start start 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3493e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 0x7fd3441edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3493e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3440c8e00 0x7fd3440c03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3493e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3493e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440c2480 con 0x7fd3440c5730 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3493e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3441497b0 con 0x7fd3440c8e00 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3493e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440c0e60 con 0x7fd344152700 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 0x7fd3441edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 0x7fd3441edd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41524/0 (socket says 172.21.15.78:41524) 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 -- 172.21.15.78:0/2932383855 learned_addr learned my addr 172.21.15.78:0/2932383855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3427fc640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3440c8e00 0x7fd3440c03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3437fe640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 -- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 msgr2=0x7fd3440c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 -- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3440c8e00 msgr2=0x7fd3440c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3440c8e00 0x7fd3440c03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd342ffd640 1 -- 172.21.15.78:0/2932383855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334009400 con 0x7fd3440c5730 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.094+0000 7fd3427fc640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3440c8e00 0x7fd3440c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.095+0000 7fd342ffd640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 0x7fd3441edd40 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd33402b840 tx=0x7fd3340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.095+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33403b810 con 0x7fd3440c5730 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.095+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33403b970 con 0x7fd3440c5730 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.095+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33403bce0 con 0x7fd3440c5730 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.095+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3440ace30 con 0x7fd3440c5730 2024-02-19T14:57:00.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.095+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3441ee940 con 0x7fd3440c5730 2024-02-19T14:57:00.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.096+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd344074990 con 0x7fd3440c5730 2024-02-19T14:57:00.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.096+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd334039040 con 0x7fd3440c5730 2024-02-19T14:57:00.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.097+0000 7fd32b7fe640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd33006b810 0x7fd33006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.097+0000 7fd3427fc640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd33006b810 0x7fd33006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.097+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd3340aba40 con 0x7fd3440c5730 2024-02-19T14:57:00.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.097+0000 7fd3427fc640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd33006b810 0x7fd33006dcd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd3440b0950 tx=0x7fd338009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:00.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.101+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd3340ade60 con 0x7fd3440c5730 2024-02-19T14:57:00.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.274+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3440c8770 con 0x7fd3440c5730 2024-02-19T14:57:00.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.275+0000 7fd32b7fe640 1 -- 172.21.15.78:0/2932383855 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16893 (secure 0 0 0) 0x7fd33404b020 con 0x7fd3440c5730 2024-02-19T14:57:00.278 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:00.278 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":137,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:56:58.999836+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:57.223056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:57.972812+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd33006b810 msgr2=0x7fd33006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd33006b810 0x7fd33006dcd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd3440b0950 tx=0x7fd338009040 comp rx=0 tx=0).stop 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 msgr2=0x7fd3441edd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 0x7fd3441edd40 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd33402b840 tx=0x7fd3340046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 shutdown_connections 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd33006b810 0x7fd33006dcd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd344152700 0x7fd3440c0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3440c8e00 0x7fd3440c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 --2- 172.21.15.78:0/2932383855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3440c5730 0x7fd3441edd40 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 >> 172.21.15.78:0/2932383855 conn(0x7fd3440cac90 msgr2=0x7fd34414d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:00.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 shutdown_connections 2024-02-19T14:57:00.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.277+0000 7fd3493e2640 1 -- 172.21.15.78:0/2932383855 wait complete. 2024-02-19T14:57:00.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:57:00.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.391+0000 7f9513702640 1 Processor -- start 2024-02-19T14:57:00.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.391+0000 7f9513702640 1 -- start start 2024-02-19T14:57:00.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9513702640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9513702640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f950c0718f0 0x7f950c071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9513702640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c159130 con 0x7f950c155070 2024-02-19T14:57:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9513702640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c0c2cf0 con 0x7f950c0718f0 2024-02-19T14:57:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9513702640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0c2030 con 0x7f950c0acc50 2024-02-19T14:57:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9511c9f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f950c0718f0 0x7f950c071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f951149e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f951149e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c0c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:40684/0 (socket says 172.21.15.78:40684) 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9511c9f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f950c0718f0 0x7f950c071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:58704/0 (socket says 172.21.15.78:58704) 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f9510c9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f950c155070 0x7f950c1554d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46112/0 (socket says 172.21.15.78:46112) 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.392+0000 7f951149e640 1 -- 172.21.15.78:0/2211801029 learned_addr learned my addr 172.21.15.78:0/2211801029 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.393+0000 7f951149e640 1 -- 172.21.15.78:0/2211801029 >> v2:172.21.15.150:3300/0 conn(0x7f950c0718f0 msgr2=0x7f950c071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.393+0000 7f951149e640 1 --2- 172.21.15.78:0/2211801029 >> v2:172.21.15.150:3300/0 conn(0x7f950c0718f0 0x7f950c071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.393+0000 7f951149e640 1 -- 172.21.15.78:0/2211801029 >> v1:172.21.15.150:6789/0 conn(0x7f950c155070 legacy=0x7f950c1554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.393+0000 7f951149e640 1 -- 172.21.15.78:0/2211801029 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c0ad030 con 0x7f950c0acc50 2024-02-19T14:57:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f951149e640 1 --2- 172.21.15.78:0/2211801029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c0c17e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9508009e50 tx=0x7f950802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b1fdaaf641dc52d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9503fff640 1 -- 172.21.15.78:0/2211801029 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950802ca70 con 0x7f950c0acc50 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9503fff640 1 -- 172.21.15.78:0/2211801029 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950802cbd0 con 0x7f950c0acc50 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9503fff640 1 -- 172.21.15.78:0/2211801029 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080343f0 con 0x7f950c0acc50 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 -- 172.21.15.78:0/2211801029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 msgr2=0x7f950c0c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 --2- 172.21.15.78:0/2211801029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c0c17e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9508009e50 tx=0x7f950802bed0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 -- 172.21.15.78:0/2211801029 shutdown_connections 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 --2- 172.21.15.78:0/2211801029 >> v2:172.21.15.150:3300/0 conn(0x7f950c0718f0 0x7f950c071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 --2- 172.21.15.78:0/2211801029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c0c17e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 -- 172.21.15.78:0/2211801029 >> 172.21.15.78:0/2211801029 conn(0x7f950c0c7800 msgr2=0x7f950c0c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 -- 172.21.15.78:0/2211801029 shutdown_connections 2024-02-19T14:57:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.394+0000 7f9513702640 1 -- 172.21.15.78:0/2211801029 wait complete. 2024-02-19T14:57:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.395+0000 7f9513702640 1 Processor -- start 2024-02-19T14:57:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.395+0000 7f9513702640 1 -- start start 2024-02-19T14:57:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f9513702640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 0x7f950c133a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f9513702640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f9513702640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f950c155070 0x7f950c135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f9513702640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c133220 con 0x7f950c0718f0 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f9513702640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0c6610 con 0x7f950c155070 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f9513702640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0c4a70 con 0x7f950c0acc50 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f951149e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 0x7f950c133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f951149e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 0x7f950c133a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41528/0 (socket says 172.21.15.78:41528) 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.396+0000 7f951149e640 1 -- 172.21.15.78:0/140195396 learned_addr learned my addr 172.21.15.78:0/140195396 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9511c9f640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f950c155070 0x7f950c135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9510c9d640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f951149e640 1 -- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 msgr2=0x7f950c152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f951149e640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f951149e640 1 -- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f950c155070 msgr2=0x7f950c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f951149e640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f950c155070 0x7f950c135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f951149e640 1 -- 172.21.15.78:0/140195396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508009b30 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f951149e640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 0x7f950c133a80 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f9508000f80 tx=0x7f9508004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508009510 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950803e450 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c1eb940 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950803e800 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.397+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c0c4ff0 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.398+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c06c820 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.398+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f950803e960 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.399+0000 7f9501ffb640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f94f006b810 0x7f94f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.399+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95080ad090 con 0x7f950c0718f0 2024-02-19T14:57:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.399+0000 7f9510c9d640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f94f006b810 0x7f94f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:00.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.399+0000 7f9510c9d640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f94f006b810 0x7f94f006dcd0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f950c1ef350 tx=0x7f94fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:00.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.403+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9508077930 con 0x7f950c0718f0 2024-02-19T14:57:00.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:00.610+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f950c074990 con 0x7f950c0718f0 2024-02-19T14:57:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.590+0000 7f9501ffb640 1 -- 172.21.15.78:0/140195396 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v116) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9508098e60 con 0x7f950c0718f0 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f94f006b810 msgr2=0x7f94f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f94f006b810 0x7f94f006dcd0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f950c1ef350 tx=0x7f94fc009040 comp rx=0 tx=0).stop 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 msgr2=0x7f950c133a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 0x7f950c133a80 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f9508000f80 tx=0x7f9508004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 shutdown_connections 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f94f006b810 0x7f94f006dcd0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f950c155070 0x7f950c135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f950c0acc50 0x7f950c152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 --2- 172.21.15.78:0/140195396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f950c0718f0 0x7f950c133a80 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 >> 172.21.15.78:0/140195396 conn(0x7f950c0c7800 msgr2=0x7f950c0c0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 shutdown_connections 2024-02-19T14:57:01.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.594+0000 7f9513702640 1 -- 172.21.15.78:0/140195396 wait complete. 2024-02-19T14:57:01.610 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:57:01.610 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:57:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.717+0000 7f43f1f93640 1 Processor -- start 2024-02-19T14:57:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.717+0000 7f43f1f93640 1 -- start start 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43f1f93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43ec0c5be0 0x7f43ec1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43f1f93640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec072250 con 0x7f43ec0c25e0 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43f1f93640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec072480 con 0x7f43ec0c29c0 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43f1f93640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec152c90 con 0x7f43ec0c5be0 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43ebfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f43ec0c5be0 0x7f43ec1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43eb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f43ec0c25e0 0x7f43ec0c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45912/0 (socket says 172.21.15.78:45912) 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43eb7fe640 1 -- 172.21.15.78:0/175471350 learned_addr learned my addr 172.21.15.78:0/175471350 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43ebfff640 1 -- 172.21.15.78:0/175471350 >> v1:172.21.15.150:6789/0 conn(0x7f43ec0c29c0 legacy=0x7f43ec06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43ebfff640 1 -- 172.21.15.78:0/175471350 >> v1:172.21.15.78:6789/0 conn(0x7f43ec0c25e0 legacy=0x7f43ec0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.718+0000 7f43ebfff640 1 -- 172.21.15.78:0/175471350 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec0c4b90 con 0x7f43ec0c5be0 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.719+0000 7f43ebfff640 1 --2- 172.21.15.78:0/175471350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec1526a0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009d90 tx=0x7f43e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813159bc9931b60d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.719+0000 7f43ea7fc640 1 -- 172.21.15.78:0/175471350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e402cc60 con 0x7f43ec0c5be0 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.719+0000 7f43ea7fc640 1 -- 172.21.15.78:0/175471350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e402cdc0 con 0x7f43ec0c5be0 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.719+0000 7f43ea7fc640 1 -- 172.21.15.78:0/175471350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4034660 con 0x7f43ec0c5be0 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 -- 172.21.15.78:0/175471350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 msgr2=0x7f43ec1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 --2- 172.21.15.78:0/175471350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec1526a0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009d90 tx=0x7f43e402beb0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 -- 172.21.15.78:0/175471350 shutdown_connections 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 --2- 172.21.15.78:0/175471350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec1526a0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 -- 172.21.15.78:0/175471350 >> 172.21.15.78:0/175471350 conn(0x7f43ec06b270 msgr2=0x7f43ec06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 -- 172.21.15.78:0/175471350 shutdown_connections 2024-02-19T14:57:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.720+0000 7f43f1f93640 1 -- 172.21.15.78:0/175471350 wait complete. 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.721+0000 7f43f1f93640 1 Processor -- start 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.721+0000 7f43f1f93640 1 -- start start 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.721+0000 7f43f1f93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43ec0c25e0 0x7f43ec0bd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.721+0000 7f43f1f93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43ec0c29c0 0x7f43ec1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43f1f93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec0bfa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43f1f93640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec153020 con 0x7f43ec0c5be0 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43f1f93640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec13d3e0 con 0x7f43ec0c29c0 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43f1f93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec1e86b0 con 0x7f43ec0c25e0 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec0bfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec0bfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41540/0 (socket says 172.21.15.78:41540) 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 -- 172.21.15.78:0/2999378016 learned_addr learned my addr 172.21.15.78:0/2999378016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43eaffd640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43ec0c29c0 0x7f43ec1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43eb7fe640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43ec0c25e0 0x7f43ec0bd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 -- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43ec0c25e0 msgr2=0x7f43ec0bd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43ec0c25e0 0x7f43ec0bd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 -- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43ec0c29c0 msgr2=0x7f43ec1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43ec0c29c0 0x7f43ec1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 -- 172.21.15.78:0/2999378016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e4009a70 con 0x7f43ec0c5be0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43ebfff640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec0bfa30 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009d90 tx=0x7f43e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4040650 con 0x7f43ec0c5be0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e40407d0 con 0x7f43ec0c5be0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec144120 con 0x7f43ec0c5be0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4040b80 con 0x7f43ec0c5be0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.722+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec0bec70 con 0x7f43ec0c5be0 2024-02-19T14:57:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.723+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec0ac580 con 0x7f43ec0c5be0 2024-02-19T14:57:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.724+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f43e4040ce0 con 0x7f43ec0c5be0 2024-02-19T14:57:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.724+0000 7f43d3fff640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43d806b810 0x7f43d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.724+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43e40acf70 con 0x7f43ec0c5be0 2024-02-19T14:57:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.724+0000 7f43eb7fe640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43d806b810 0x7f43d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.725+0000 7f43eb7fe640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43d806b810 0x7f43d806dcd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005880 tx=0x7f43dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:01.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.729+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f43e40e0dc0 con 0x7f43ec0c5be0 2024-02-19T14:57:01.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:01.935+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f43ec144120 con 0x7f43ec0c5be0 2024-02-19T14:57:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.674+0000 7f43d3fff640 1 -- 172.21.15.78:0/2999378016 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v117) v1 ==== 122+0+0 (secure 0 0 0) 0x7f43ec144120 con 0x7f43ec0c5be0 2024-02-19T14:57:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43d806b810 msgr2=0x7f43d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43d806b810 0x7f43d806dcd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005880 tx=0x7f43dc009040 comp rx=0 tx=0).stop 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 msgr2=0x7f43ec0bfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec0bfa30 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009d90 tx=0x7f43e4004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 shutdown_connections 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f43d806b810 0x7f43d806dcd0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43ec0c5be0 0x7f43ec0bfa30 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43ec0c29c0 0x7f43ec1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 --2- 172.21.15.78:0/2999378016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43ec0c25e0 0x7f43ec0bd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 >> 172.21.15.78:0/2999378016 conn(0x7f43ec06b270 msgr2=0x7f43ec074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 shutdown_connections 2024-02-19T14:57:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.676+0000 7f43f1f93640 1 -- 172.21.15.78:0/2999378016 wait complete. 2024-02-19T14:57:02.691 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:57:02.691 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 Processor -- start 2024-02-19T14:57:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 -- start start 2024-02-19T14:57:02.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72840c8e00 0x7f728406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f728406d910 0x7f728406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728406cd20 con 0x7f7284152700 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72841551f0 con 0x7f728406d910 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72897d0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7284155490 con 0x7f72840c8e00 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.790+0000 7f72837fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f728406d910 0x7f728406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72837fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f728406d910 0x7f728406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41548/0 (socket says 172.21.15.78:41548) 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f7282ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7284152700 0x7f72840c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45922/0 (socket says 172.21.15.78:45922) 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72837fe640 1 -- 172.21.15.78:0/3472685028 learned_addr learned my addr 172.21.15.78:0/3472685028 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72827fc640 1 --2- 172.21.15.78:0/3472685028 >> v2:172.21.15.150:3300/0 conn(0x7f72840c8e00 0x7f728406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72837fe640 1 -- 172.21.15.78:0/3472685028 >> v2:172.21.15.150:3300/0 conn(0x7f72840c8e00 msgr2=0x7f728406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72837fe640 1 --2- 172.21.15.78:0/3472685028 >> v2:172.21.15.150:3300/0 conn(0x7f72840c8e00 0x7f728406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72837fe640 1 -- 172.21.15.78:0/3472685028 >> v1:172.21.15.78:6789/0 conn(0x7f7284152700 legacy=0x7f72840c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f72837fe640 1 -- 172.21.15.78:0/3472685028 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7284155770 con 0x7f728406d910 2024-02-19T14:57:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.791+0000 7f7281ffb640 1 -- 172.21.15.78:0/3472685028 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819968799 0 0) 0x7f728406cd20 con 0x7f7284152700 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72837fe640 1 --2- 172.21.15.78:0/3472685028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f728406d910 0x7f728406c7e0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f72840c7620 tx=0x7f727c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27cee2a4600002f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f7281ffb640 1 -- 172.21.15.78:0/3472685028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c02cb20 con 0x7f728406d910 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f7281ffb640 1 -- 172.21.15.78:0/3472685028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c02cc80 con 0x7f728406d910 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f7281ffb640 1 -- 172.21.15.78:0/3472685028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c0343f0 con 0x7f728406d910 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 -- 172.21.15.78:0/3472685028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f728406d910 msgr2=0x7f728406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 --2- 172.21.15.78:0/3472685028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f728406d910 0x7f728406c7e0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f72840c7620 tx=0x7f727c02bf80 comp rx=0 tx=0).stop 2024-02-19T14:57:02.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 -- 172.21.15.78:0/3472685028 shutdown_connections 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 --2- 172.21.15.78:0/3472685028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f728406d910 0x7f728406c7e0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 --2- 172.21.15.78:0/3472685028 >> v2:172.21.15.150:3300/0 conn(0x7f72840c8e00 0x7f728406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 -- 172.21.15.78:0/3472685028 >> 172.21.15.78:0/3472685028 conn(0x7f72840cac90 msgr2=0x7f72840cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 -- 172.21.15.78:0/3472685028 shutdown_connections 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.792+0000 7f72897d0640 1 -- 172.21.15.78:0/3472685028 wait complete. 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 Processor -- start 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 -- start start 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f728406d910 0x7f72840bde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72840c8e00 0x7f72840c0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 0x7f72840c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72841eae90 con 0x7f7284152700 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72840be380 con 0x7f72840c8e00 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72897d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72841483d0 con 0x7f728406d910 2024-02-19T14:57:02.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 0x7f72840c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 0x7f72840c3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41552/0 (socket says 172.21.15.78:41552) 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72837fe640 1 -- 172.21.15.78:0/2816253422 learned_addr learned my addr 172.21.15.78:0/2816253422 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f72827fc640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72840c8e00 0x7f72840c0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.794+0000 7f7282ffd640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f728406d910 0x7f72840bde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72837fe640 1 -- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f728406d910 msgr2=0x7f72840bde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72837fe640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f728406d910 0x7f72840bde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72837fe640 1 -- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72840c8e00 msgr2=0x7f72840c0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:02.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72837fe640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72840c8e00 0x7f72840c0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72837fe640 1 -- 172.21.15.78:0/2816253422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c009b30 con 0x7f7284152700 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72837fe640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 0x7f72840c3b70 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f727c009f40 tx=0x7f727c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c009510 con 0x7f7284152700 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c02c920 con 0x7f7284152700 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72841ea560 con 0x7f7284152700 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c03e690 con 0x7f7284152700 2024-02-19T14:57:02.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.795+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72841eab20 con 0x7f7284152700 2024-02-19T14:57:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.796+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72841eb040 con 0x7f7284152700 2024-02-19T14:57:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.796+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f727c03e7f0 con 0x7f7284152700 2024-02-19T14:57:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.797+0000 7f727b7fe640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f726406b810 0x7f726406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.797+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f727c0accc0 con 0x7f7284152700 2024-02-19T14:57:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.797+0000 7f7282ffd640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f726406b810 0x7f726406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:02.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.797+0000 7f7282ffd640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f726406b810 0x7f726406dcd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f726c007fd0 tx=0x7f726c007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:02.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:02.801+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f72841eb040 con 0x7f7284152700 2024-02-19T14:57:03.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.012+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72841eb040 con 0x7f7284152700 2024-02-19T14:57:03.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.012+0000 7f727b7fe640 1 -- 172.21.15.78:0/2816253422 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f72841eb040 con 0x7f7284152700 2024-02-19T14:57:03.015 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:03.015 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7351,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:01.936470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8268":{"gid":8268,"name":"b","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6834/2425259283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2425259283},{"type":"v1","addr":"172.21.15.150:6834","nonce":2425259283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:03.016 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 117 2024-02-19T14:57:03.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.014+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f726406b810 msgr2=0x7f726406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.014+0000 7f72897d0640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f726406b810 0x7f726406dcd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f726c007fd0 tx=0x7f726c007de0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.014+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 msgr2=0x7f72840c3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.014+0000 7f72897d0640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 0x7f72840c3b70 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f727c009f40 tx=0x7f727c004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 shutdown_connections 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f726406b810 0x7f726406dcd0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7284152700 0x7f72840c3b70 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72840c8e00 0x7f72840c0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 --2- 172.21.15.78:0/2816253422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f728406d910 0x7f72840bde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 >> 172.21.15.78:0/2816253422 conn(0x7f72840cac90 msgr2=0x7f728406c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 shutdown_connections 2024-02-19T14:57:03.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.015+0000 7f72897d0640 1 -- 172.21.15.78:0/2816253422 wait complete. 2024-02-19T14:57:03.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:03.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.127+0000 7f4f0e89d640 1 Processor -- start 2024-02-19T14:57:03.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.127+0000 7f4f0e89d640 1 -- start start 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f0e89d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f080c57b0 0x7f4f08151c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f0e89d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0814c420 con 0x7f4f080c4020 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f0e89d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f080c5d00 con 0x7f4f0814c040 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f0e89d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f080c62c0 con 0x7f4f080c57b0 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f077fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4f080c57b0 0x7f4f08151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f0ce3a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4f080c4020 0x7f4f0815c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45932/0 (socket says 172.21.15.78:45932) 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f0ce3a640 1 -- 172.21.15.78:0/4247072957 learned_addr learned my addr 172.21.15.78:0/4247072957 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f077fe640 1 -- 172.21.15.78:0/4247072957 >> v1:172.21.15.150:6789/0 conn(0x7f4f0814c040 legacy=0x7f4f080c6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f077fe640 1 -- 172.21.15.78:0/4247072957 >> v1:172.21.15.78:6789/0 conn(0x7f4f080c4020 legacy=0x7f4f0815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f077fe640 1 -- 172.21.15.78:0/4247072957 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0806fdf0 con 0x7f4f080c57b0 2024-02-19T14:57:03.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.128+0000 7f4f06ffd640 1 -- 172.21.15.78:0/4247072957 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453300954 0 0) 0x7f4f00001490 con 0x7f4f080c4020 2024-02-19T14:57:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.129+0000 7f4f077fe640 1 --2- 172.21.15.78:0/4247072957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c57b0 0x7f4f08151c50 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0097a0 tx=0x7f4efc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd24593c4634762 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.129+0000 7f4f06ffd640 1 -- 172.21.15.78:0/4247072957 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc02e070 con 0x7f4f080c57b0 2024-02-19T14:57:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.129+0000 7f4f06ffd640 1 -- 172.21.15.78:0/4247072957 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc0315a0 con 0x7f4f080c57b0 2024-02-19T14:57:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.129+0000 7f4f06ffd640 1 -- 172.21.15.78:0/4247072957 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc031930 con 0x7f4f080c57b0 2024-02-19T14:57:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.129+0000 7f4f0e89d640 1 -- 172.21.15.78:0/4247072957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c57b0 msgr2=0x7f4f08151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.129+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/4247072957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c57b0 0x7f4f08151c50 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0097a0 tx=0x7f4efc02b840 comp rx=0 tx=0).stop 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.130+0000 7f4f0e89d640 1 -- 172.21.15.78:0/4247072957 shutdown_connections 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.130+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/4247072957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c57b0 0x7f4f08151c50 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.130+0000 7f4f0e89d640 1 -- 172.21.15.78:0/4247072957 >> 172.21.15.78:0/4247072957 conn(0x7f4f080c7590 msgr2=0x7f4f080c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.130+0000 7f4f0e89d640 1 -- 172.21.15.78:0/4247072957 shutdown_connections 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.130+0000 7f4f0e89d640 1 -- 172.21.15.78:0/4247072957 wait complete. 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.131+0000 7f4f0e89d640 1 Processor -- start 2024-02-19T14:57:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.131+0000 7f4f0e89d640 1 -- start start 2024-02-19T14:57:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.131+0000 7f4f0e89d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 0x7f4f081df8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f080c57b0 0x7f4f0813d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f0814c040 0x7f4f0813de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080aaa30 con 0x7f4f080c4020 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f081eaa40 con 0x7f4f0814c040 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f08147430 con 0x7f4f080c57b0 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 0x7f4f081df8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 0x7f4f081df8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41568/0 (socket says 172.21.15.78:41568) 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 -- 172.21.15.78:0/2701005317 learned_addr learned my addr 172.21.15.78:0/2701005317 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f077fe640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f080c57b0 0x7f4f0813d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0ce3a640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f0814c040 0x7f4f0813de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 -- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f080c57b0 msgr2=0x7f4f0813d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f080c57b0 0x7f4f0813d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 -- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f0814c040 msgr2=0x7f4f0813de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f0814c040 0x7f4f0813de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 -- 172.21.15.78:0/2701005317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc009400 con 0x7f4f080c4020 2024-02-19T14:57:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0ce3a640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f0814c040 0x7f4f0813de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f07fff640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 0x7f4f081df8f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c450 tx=0x7f4ef800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80052d0 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef8005430 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8012870 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f080ab450 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.132+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f080aae90 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.133+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f080aacf0 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.134+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4ef80129d0 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.134+0000 7f4f04ff9640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4ef406b810 0x7f4ef406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.134+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4ef8089d70 con 0x7f4f080c4020 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.134+0000 7f4f077fe640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4ef406b810 0x7f4ef406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.134+0000 7f4f077fe640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4ef406b810 0x7f4ef406dcd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4efc036000 tx=0x7f4efc03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.138+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4f080aacf0 con 0x7f4f080c4020 2024-02-19T14:57:03.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.309+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f080aacf0 con 0x7f4f080c4020 2024-02-19T14:57:03.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.310+0000 7f4f04ff9640 1 -- 172.21.15.78:0/2701005317 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16880 (secure 0 0 0) 0x7f4f080aacf0 con 0x7f4f080c4020 2024-02-19T14:57:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:03.313 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":138,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:00.661738+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:57.223056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:57.972812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:03.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4ef406b810 msgr2=0x7f4ef406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4ef406b810 0x7f4ef406dcd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4efc036000 tx=0x7f4efc03d040 comp rx=0 tx=0).stop 2024-02-19T14:57:03.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 msgr2=0x7f4f081df8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 0x7f4f081df8f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c450 tx=0x7f4ef800c910 comp rx=0 tx=0).stop 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 shutdown_connections 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4ef406b810 0x7f4ef406dcd0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f0814c040 0x7f4f0813de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f080c57b0 0x7f4f0813d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 --2- 172.21.15.78:0/2701005317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f080c4020 0x7f4f081df8f0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 >> 172.21.15.78:0/2701005317 conn(0x7f4f080c7590 msgr2=0x7f4f08149740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 shutdown_connections 2024-02-19T14:57:03.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.312+0000 7f4f0e89d640 1 -- 172.21.15.78:0/2701005317 wait complete. 2024-02-19T14:57:03.326 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:57:03.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:03.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 Processor -- start 2024-02-19T14:57:03.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 -- start start 2024-02-19T14:57:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcadc14ece0 0x7fcadc0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcadc0c5270 0x7fcadc0719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc0c1120 con 0x7fcadc071f10 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc0c6080 con 0x7fcadc14ece0 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae324e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc0c57e0 con 0x7fcadc0c5270 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.423+0000 7fcae0fea640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcadc14ece0 0x7fcadc0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae17eb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcadc071f10 0x7fcadc0c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45934/0 (socket says 172.21.15.78:45934) 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae0fea640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcadc14ece0 0x7fcadc0c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41570/0 (socket says 172.21.15.78:41570) 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae17eb640 1 -- 172.21.15.78:0/3166534924 learned_addr learned my addr 172.21.15.78:0/3166534924 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcadbfff640 1 --2- 172.21.15.78:0/3166534924 >> v2:172.21.15.150:3300/0 conn(0x7fcadc0c5270 0x7fcadc0719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae0fea640 1 -- 172.21.15.78:0/3166534924 >> v2:172.21.15.150:3300/0 conn(0x7fcadc0c5270 msgr2=0x7fcadc0719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae0fea640 1 --2- 172.21.15.78:0/3166534924 >> v2:172.21.15.150:3300/0 conn(0x7fcadc0c5270 0x7fcadc0719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae0fea640 1 -- 172.21.15.78:0/3166534924 >> v1:172.21.15.78:6789/0 conn(0x7fcadc071f10 legacy=0x7fcadc0c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcadb7fe640 1 -- 172.21.15.78:0/3166534924 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888813300 0 0) 0x7fcadc0c1120 con 0x7fcadc071f10 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.424+0000 7fcae0fea640 1 -- 172.21.15.78:0/3166534924 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc0c5ac0 con 0x7fcadc14ece0 2024-02-19T14:57:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae0fea640 1 --2- 172.21.15.78:0/3166534924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc0c4d30 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcadc155b80 tx=0x7fcad402cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d34dfe6dff7ad6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcadb7fe640 1 -- 172.21.15.78:0/3166534924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad402db70 con 0x7fcadc14ece0 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcadb7fe640 1 -- 172.21.15.78:0/3166534924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad402dcd0 con 0x7fcadc14ece0 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcadb7fe640 1 -- 172.21.15.78:0/3166534924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad4008660 con 0x7fcadc14ece0 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 -- 172.21.15.78:0/3166534924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 msgr2=0x7fcadc0c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 --2- 172.21.15.78:0/3166534924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc0c4d30 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcadc155b80 tx=0x7fcad402cfd0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 -- 172.21.15.78:0/3166534924 shutdown_connections 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 --2- 172.21.15.78:0/3166534924 >> v2:172.21.15.150:3300/0 conn(0x7fcadc0c5270 0x7fcadc0719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 --2- 172.21.15.78:0/3166534924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc0c4d30 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 -- 172.21.15.78:0/3166534924 >> 172.21.15.78:0/3166534924 conn(0x7fcadc06b270 msgr2=0x7fcadc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 -- 172.21.15.78:0/3166534924 shutdown_connections 2024-02-19T14:57:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.425+0000 7fcae324e640 1 -- 172.21.15.78:0/3166534924 wait complete. 2024-02-19T14:57:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.426+0000 7fcae324e640 1 Processor -- start 2024-02-19T14:57:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.426+0000 7fcae324e640 1 -- start start 2024-02-19T14:57:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae324e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcadc071f10 0x7fcadc1e7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae324e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcadc0c5270 0x7fcadc1497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae324e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc1e7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae324e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0bd3f0 con 0x7fcadc14ece0 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae324e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc1e6ab0 con 0x7fcadc0c5270 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae324e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc1e6d90 con 0x7fcadc071f10 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae17eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc1e7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae17eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc1e7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41572/0 (socket says 172.21.15.78:41572) 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae17eb640 1 -- 172.21.15.78:0/2580433098 learned_addr learned my addr 172.21.15.78:0/2580433098 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcadbfff640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcadc0c5270 0x7fcadc1497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.427+0000 7fcae0fea640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcadc071f10 0x7fcadc1e7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae17eb640 1 -- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcadc071f10 msgr2=0x7fcadc1e7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae17eb640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcadc071f10 0x7fcadc1e7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae17eb640 1 -- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcadc0c5270 msgr2=0x7fcadc1497b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae17eb640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcadc0c5270 0x7fcadc1497b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae17eb640 1 -- 172.21.15.78:0/2580433098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad400ab80 con 0x7fcadc14ece0 2024-02-19T14:57:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae17eb640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc1e7e90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00c450 tx=0x7fcacc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0074e0 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc007660 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcadc1e7070 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc004a20 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.428+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcadc14d4c0 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.429+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcadc1e72b0 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.429+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcacc004b80 con 0x7fcadc14ece0 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.430+0000 7fcad97fa640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcabc06b810 0x7fcabc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.430+0000 7fcae0fea640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcabc06b810 0x7fcabc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.430+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcacc0896c0 con 0x7fcadc14ece0 2024-02-19T14:57:03.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.430+0000 7fcae0fea640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcabc06b810 0x7fcabc06dcd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcad4036000 tx=0x7fcad4031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.434+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcadc1e72b0 con 0x7fcadc14ece0 2024-02-19T14:57:03.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.601+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcadc073210 con 0x7fcabc06b810 2024-02-19T14:57:03.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.608+0000 7fcad97fa640 1 -- 172.21.15.78:0/2580433098 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277494 (secure 0 0 0) 0x7fcad4046090 con 0x7fcabc06b810 2024-02-19T14:57:03.614 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:03.615 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcabc06b810 msgr2=0x7fcabc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcabc06b810 0x7fcabc06dcd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcad4036000 tx=0x7fcad4031000 comp rx=0 tx=0).stop 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 msgr2=0x7fcadc1e7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc1e7e90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00c450 tx=0x7fcacc00c910 comp rx=0 tx=0).stop 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 shutdown_connections 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcabc06b810 0x7fcabc06dcd0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcadc14ece0 0x7fcadc1e7e90 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcadc0c5270 0x7fcadc1497b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 --2- 172.21.15.78:0/2580433098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcadc071f10 0x7fcadc1e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 >> 172.21.15.78:0/2580433098 conn(0x7fcadc06b270 msgr2=0x7fcadc06e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 shutdown_connections 2024-02-19T14:57:03.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.611+0000 7fcae324e640 1 -- 172.21.15.78:0/2580433098 wait complete. 2024-02-19T14:57:03.625 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-02-19T14:57:02.118449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":263824,"kb_used_data":14336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710896,"statfs":{"total":773094113280,"available":772823957504,"internally_reserved":0,"allocated":14680064,"data_stored":7705122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.472867"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652900+0000","last_change":"2024-02-19T14:56:57.652900+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652900+0000","last_undegraded":"2024-02-19T14:56:57.652900+0000","last_fullsized":"2024-02-19T14:56:57.652900+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651707+0000","last_change":"2024-02-19T14:56:57.651707+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651707+0000","last_undegraded":"2024-02-19T14:56:57.651707+0000","last_fullsized":"2024-02-19T14:56:57.651707+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648723+0000","last_change":"2024-02-19T14:56:57.648723+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648723+0000","last_undegraded":"2024-02-19T14:56:57.648723+0000","last_fullsized":"2024-02-19T14:56:57.648723+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647430+0000","last_change":"2024-02-19T14:56:57.647430+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647430+0000","last_undegraded":"2024-02-19T14:56:57.647430+0000","last_fullsized":"2024-02-19T14:56:57.647430+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648001+0000","last_change":"2024-02-19T14:56:57.648001+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648001+0000","last_undegraded":"2024-02-19T14:56:57.648001+0000","last_fullsized":"2024-02-19T14:56:57.648001+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652213+0000","last_change":"2024-02-19T14:56:57.652213+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652213+0000","last_undegraded":"2024-02-19T14:56:57.652213+0000","last_fullsized":"2024-02-19T14:56:57.652213+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648684+0000","last_change":"2024-02-19T14:56:57.648684+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648684+0000","last_undegraded":"2024-02-19T14:56:57.648684+0000","last_fullsized":"2024-02-19T14:56:57.648684+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651885+0000","last_change":"2024-02-19T14:56:57.651885+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651885+0000","last_undegraded":"2024-02-19T14:56:57.651885+0000","last_fullsized":"2024-02-19T14:56:57.651885+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651784+0000","last_change":"2024-02-19T14:56:57.651784+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651784+0000","last_undegraded":"2024-02-19T14:56:57.651784+0000","last_fullsized":"2024-02-19T14:56:57.651784+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651683+0000","last_change":"2024-02-19T14:56:57.651683+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651683+0000","last_undegraded":"2024-02-19T14:56:57.651683+0000","last_fullsized":"2024-02-19T14:56:57.651683+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647968+0000","last_change":"2024-02-19T14:56:57.647968+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647968+0000","last_undegraded":"2024-02-19T14:56:57.647968+0000","last_fullsized":"2024-02-19T14:56:57.647968+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651961+0000","last_change":"2024-02-19T14:56:57.651961+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651961+0000","last_undegraded":"2024-02-19T14:56:57.651961+0000","last_fullsized":"2024-02-19T14:56:57.651961+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652488+0000","last_change":"2024-02-19T14:56:57.652488+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652488+0000","last_undegraded":"2024-02-19T14:56:57.652488+0000","last_fullsized":"2024-02-19T14:56:57.652488+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648673+0000","last_change":"2024-02-19T14:56:57.648673+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648673+0000","last_undegraded":"2024-02-19T14:56:57.648673+0000","last_fullsized":"2024-02-19T14:56:57.648673+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651750+0000","last_change":"2024-02-19T14:56:57.651750+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651750+0000","last_undegraded":"2024-02-19T14:56:57.651750+0000","last_fullsized":"2024-02-19T14:56:57.651750+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651885+0000","last_change":"2024-02-19T14:56:57.651885+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651885+0000","last_undegraded":"2024-02-19T14:56:57.651885+0000","last_fullsized":"2024-02-19T14:56:57.651885+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647593+0000","last_change":"2024-02-19T14:56:57.647593+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647593+0000","last_undegraded":"2024-02-19T14:56:57.647593+0000","last_fullsized":"2024-02-19T14:56:57.647593+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652025+0000","last_change":"2024-02-19T14:56:57.652025+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652025+0000","last_undegraded":"2024-02-19T14:56:57.652025+0000","last_fullsized":"2024-02-19T14:56:57.652025+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647463+0000","last_change":"2024-02-19T14:56:57.647463+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647463+0000","last_undegraded":"2024-02-19T14:56:57.647463+0000","last_fullsized":"2024-02-19T14:56:57.647463+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652096+0000","last_change":"2024-02-19T14:56:57.652096+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652096+0000","last_undegraded":"2024-02-19T14:56:57.652096+0000","last_fullsized":"2024-02-19T14:56:57.652096+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647654+0000","last_change":"2024-02-19T14:56:57.647654+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647654+0000","last_undegraded":"2024-02-19T14:56:57.647654+0000","last_fullsized":"2024-02-19T14:56:57.647654+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648157+0000","last_change":"2024-02-19T14:56:57.648157+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648157+0000","last_undegraded":"2024-02-19T14:56:57.648157+0000","last_fullsized":"2024-02-19T14:56:57.648157+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647893+0000","last_change":"2024-02-19T14:56:57.647893+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647893+0000","last_undegraded":"2024-02-19T14:56:57.647893+0000","last_fullsized":"2024-02-19T14:56:57.647893+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652426+0000","last_change":"2024-02-19T14:56:57.652426+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652426+0000","last_undegraded":"2024-02-19T14:56:57.652426+0000","last_fullsized":"2024-02-19T14:56:57.652426+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647466+0000","last_change":"2024-02-19T14:56:57.647466+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647466+0000","last_undegraded":"2024-02-19T14:56:57.647466+0000","last_fullsized":"2024-02-19T14:56:57.647466+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651678+0000","last_change":"2024-02-19T14:56:57.651678+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651678+0000","last_undegraded":"2024-02-19T14:56:57.651678+0000","last_fullsized":"2024-02-19T14:56:57.651678+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651923+0000","last_change":"2024-02-19T14:56:57.651923+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651923+0000","last_undegraded":"2024-02-19T14:56:57.651923+0000","last_fullsized":"2024-02-19T14:56:57.651923+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":386,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-02-19T14:56:57.653409+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:57.653409+0000","last_peered":"2024-02-19T14:56:57.653409+0000","last_clean":"2024-02-19T14:56:57.653409+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:57.653409+0000","last_undegraded":"2024-02-19T14:56:57.653409+0000","last_fullsized":"2024-02-19T14:56:57.653409+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2277376,"data_stored":1404364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.504}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.625},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.754},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.871}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.85399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":820205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.623},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":820205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.372},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.54800000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.443},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32828,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.318}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.84799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.307}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.629}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.312},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":1.0740000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33412,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338428,"statfs":{"total":96636764160,"available":96602550272,"internally_reserved":0,"allocated":2277376,"data_stored":1404364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.308}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:03.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 Processor -- start 2024-02-19T14:57:03.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 -- start start 2024-02-19T14:57:03.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee240c7910 0x7fee240c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee24070720 0x7fee24070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee240c65a0 con 0x7fee240c5560 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee240c7530 con 0x7fee24070720 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.763+0000 7fee29d6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee241537b0 con 0x7fee240c7910 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee23fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee24070720 0x7fee24070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee237fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee240c5560 0x7fee24154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45950/0 (socket says 172.21.15.78:45950) 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee237fe640 1 -- 172.21.15.78:0/1955485717 learned_addr learned my addr 172.21.15.78:0/1955485717 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee22ffd640 1 --2- 172.21.15.78:0/1955485717 >> v2:172.21.15.150:3300/0 conn(0x7fee240c7910 0x7fee240c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee227fc640 1 -- 172.21.15.78:0/1955485717 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634729011 0 0) 0x7fee240c65a0 con 0x7fee240c5560 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee227fc640 1 -- 172.21.15.78:0/1955485717 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c003520 con 0x7fee240c5560 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee23fff640 1 -- 172.21.15.78:0/1955485717 >> v2:172.21.15.150:3300/0 conn(0x7fee240c7910 msgr2=0x7fee240c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee23fff640 1 --2- 172.21.15.78:0/1955485717 >> v2:172.21.15.150:3300/0 conn(0x7fee240c7910 0x7fee240c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee23fff640 1 -- 172.21.15.78:0/1955485717 >> v1:172.21.15.78:6789/0 conn(0x7fee240c5560 legacy=0x7fee24154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.764+0000 7fee23fff640 1 -- 172.21.15.78:0/1955485717 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee24153a90 con 0x7fee24070720 2024-02-19T14:57:03.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee23fff640 1 --2- 172.21.15.78:0/1955485717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee24070720 0x7fee24070b70 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fee240c91b0 tx=0x7fee1c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b257805c54083b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee227fc640 1 -- 172.21.15.78:0/1955485717 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c02cb60 con 0x7fee24070720 2024-02-19T14:57:03.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee227fc640 1 -- 172.21.15.78:0/1955485717 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c02ccc0 con 0x7fee24070720 2024-02-19T14:57:03.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee227fc640 1 -- 172.21.15.78:0/1955485717 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c034660 con 0x7fee24070720 2024-02-19T14:57:03.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 -- 172.21.15.78:0/1955485717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee24070720 msgr2=0x7fee24070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 --2- 172.21.15.78:0/1955485717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee24070720 0x7fee24070b70 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fee240c91b0 tx=0x7fee1c02bdb0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 -- 172.21.15.78:0/1955485717 shutdown_connections 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 --2- 172.21.15.78:0/1955485717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee24070720 0x7fee24070b70 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 --2- 172.21.15.78:0/1955485717 >> v2:172.21.15.150:3300/0 conn(0x7fee240c7910 0x7fee240c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 -- 172.21.15.78:0/1955485717 >> 172.21.15.78:0/1955485717 conn(0x7fee2406b4b0 msgr2=0x7fee240c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 -- 172.21.15.78:0/1955485717 shutdown_connections 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.765+0000 7fee29d6c640 1 -- 172.21.15.78:0/1955485717 wait complete. 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.766+0000 7fee29d6c640 1 Processor -- start 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 -- start start 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee24070720 0x7fee241dfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 0x7fee241dec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240c7910 0x7fee2414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee240a7ef0 con 0x7fee240c5560 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee240aa4a0 con 0x7fee240c7910 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee29d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee240aaa40 con 0x7fee24070720 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee22ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 0x7fee241dec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee22ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 0x7fee241dec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41592/0 (socket says 172.21.15.78:41592) 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee22ffd640 1 -- 172.21.15.78:0/3667770292 learned_addr learned my addr 172.21.15.78:0/3667770292 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee237fe640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee24070720 0x7fee241dfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.767+0000 7fee23fff640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240c7910 0x7fee2414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee22ffd640 1 -- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee24070720 msgr2=0x7fee241dfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee22ffd640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee24070720 0x7fee241dfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee22ffd640 1 -- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240c7910 msgr2=0x7fee2414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee22ffd640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240c7910 0x7fee2414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee22ffd640 1 -- 172.21.15.78:0/3667770292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c009a70 con 0x7fee240c5560 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee22ffd640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 0x7fee241dec00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e840 tx=0x7fee1800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee180094f0 con 0x7fee240c5560 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1800c940 con 0x7fee240c5560 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee240c5ac0 con 0x7fee240c5560 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1800ccd0 con 0x7fee240c5560 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.768+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee240aa760 con 0x7fee240c5560 2024-02-19T14:57:03.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.769+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee24074990 con 0x7fee240c5560 2024-02-19T14:57:03.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.770+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fee18013470 con 0x7fee240c5560 2024-02-19T14:57:03.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.770+0000 7fee0bfff640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee1006b810 0x7fee1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:03.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.770+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fee1808a2d0 con 0x7fee240c5560 2024-02-19T14:57:03.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.770+0000 7fee237fe640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee1006b810 0x7fee1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:03.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.770+0000 7fee237fe640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee1006b810 0x7fee1006dcd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fee14005880 tx=0x7fee14009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:03.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.774+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee18054ca0 con 0x7fee240c5560 2024-02-19T14:57:03.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.944+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee240c6ad0 con 0x7fee1006b810 2024-02-19T14:57:03.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.951+0000 7fee0bfff640 1 -- 172.21.15.78:0/3667770292 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277494 (secure 0 0 0) 0x7fee1400f820 con 0x7fee1006b810 2024-02-19T14:57:03.956 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:03.958 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee1006b810 msgr2=0x7fee1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee1006b810 0x7fee1006dcd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fee14005880 tx=0x7fee14009040 comp rx=0 tx=0).stop 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 msgr2=0x7fee241dec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 0x7fee241dec00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e840 tx=0x7fee1800ed00 comp rx=0 tx=0).stop 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 shutdown_connections 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee1006b810 0x7fee1006dcd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c5560 0x7fee241dec00 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240c7910 0x7fee2414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 --2- 172.21.15.78:0/3667770292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee24070720 0x7fee241dfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.953+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 >> 172.21.15.78:0/3667770292 conn(0x7fee2406b4b0 msgr2=0x7fee241518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.954+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 shutdown_connections 2024-02-19T14:57:03.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:03.954+0000 7fee29d6c640 1 -- 172.21.15.78:0/3667770292 wait complete. 2024-02-19T14:57:03.967 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-02-19T14:57:02.118449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":263824,"kb_used_data":14336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710896,"statfs":{"total":773094113280,"available":772823957504,"internally_reserved":0,"allocated":14680064,"data_stored":7705122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.472867"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652900+0000","last_change":"2024-02-19T14:56:57.652900+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652900+0000","last_undegraded":"2024-02-19T14:56:57.652900+0000","last_fullsized":"2024-02-19T14:56:57.652900+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651707+0000","last_change":"2024-02-19T14:56:57.651707+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651707+0000","last_undegraded":"2024-02-19T14:56:57.651707+0000","last_fullsized":"2024-02-19T14:56:57.651707+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648723+0000","last_change":"2024-02-19T14:56:57.648723+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648723+0000","last_undegraded":"2024-02-19T14:56:57.648723+0000","last_fullsized":"2024-02-19T14:56:57.648723+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647430+0000","last_change":"2024-02-19T14:56:57.647430+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647430+0000","last_undegraded":"2024-02-19T14:56:57.647430+0000","last_fullsized":"2024-02-19T14:56:57.647430+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648001+0000","last_change":"2024-02-19T14:56:57.648001+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648001+0000","last_undegraded":"2024-02-19T14:56:57.648001+0000","last_fullsized":"2024-02-19T14:56:57.648001+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652213+0000","last_change":"2024-02-19T14:56:57.652213+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652213+0000","last_undegraded":"2024-02-19T14:56:57.652213+0000","last_fullsized":"2024-02-19T14:56:57.652213+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648684+0000","last_change":"2024-02-19T14:56:57.648684+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648684+0000","last_undegraded":"2024-02-19T14:56:57.648684+0000","last_fullsized":"2024-02-19T14:56:57.648684+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651885+0000","last_change":"2024-02-19T14:56:57.651885+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651885+0000","last_undegraded":"2024-02-19T14:56:57.651885+0000","last_fullsized":"2024-02-19T14:56:57.651885+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651784+0000","last_change":"2024-02-19T14:56:57.651784+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651784+0000","last_undegraded":"2024-02-19T14:56:57.651784+0000","last_fullsized":"2024-02-19T14:56:57.651784+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651683+0000","last_change":"2024-02-19T14:56:57.651683+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651683+0000","last_undegraded":"2024-02-19T14:56:57.651683+0000","last_fullsized":"2024-02-19T14:56:57.651683+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647968+0000","last_change":"2024-02-19T14:56:57.647968+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647968+0000","last_undegraded":"2024-02-19T14:56:57.647968+0000","last_fullsized":"2024-02-19T14:56:57.647968+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651961+0000","last_change":"2024-02-19T14:56:57.651961+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651961+0000","last_undegraded":"2024-02-19T14:56:57.651961+0000","last_fullsized":"2024-02-19T14:56:57.651961+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652488+0000","last_change":"2024-02-19T14:56:57.652488+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652488+0000","last_undegraded":"2024-02-19T14:56:57.652488+0000","last_fullsized":"2024-02-19T14:56:57.652488+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648673+0000","last_change":"2024-02-19T14:56:57.648673+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648673+0000","last_undegraded":"2024-02-19T14:56:57.648673+0000","last_fullsized":"2024-02-19T14:56:57.648673+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651750+0000","last_change":"2024-02-19T14:56:57.651750+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651750+0000","last_undegraded":"2024-02-19T14:56:57.651750+0000","last_fullsized":"2024-02-19T14:56:57.651750+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651885+0000","last_change":"2024-02-19T14:56:57.651885+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651885+0000","last_undegraded":"2024-02-19T14:56:57.651885+0000","last_fullsized":"2024-02-19T14:56:57.651885+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647593+0000","last_change":"2024-02-19T14:56:57.647593+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647593+0000","last_undegraded":"2024-02-19T14:56:57.647593+0000","last_fullsized":"2024-02-19T14:56:57.647593+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652025+0000","last_change":"2024-02-19T14:56:57.652025+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652025+0000","last_undegraded":"2024-02-19T14:56:57.652025+0000","last_fullsized":"2024-02-19T14:56:57.652025+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647463+0000","last_change":"2024-02-19T14:56:57.647463+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647463+0000","last_undegraded":"2024-02-19T14:56:57.647463+0000","last_fullsized":"2024-02-19T14:56:57.647463+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652096+0000","last_change":"2024-02-19T14:56:57.652096+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652096+0000","last_undegraded":"2024-02-19T14:56:57.652096+0000","last_fullsized":"2024-02-19T14:56:57.652096+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647654+0000","last_change":"2024-02-19T14:56:57.647654+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647654+0000","last_undegraded":"2024-02-19T14:56:57.647654+0000","last_fullsized":"2024-02-19T14:56:57.647654+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.648157+0000","last_change":"2024-02-19T14:56:57.648157+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.648157+0000","last_undegraded":"2024-02-19T14:56:57.648157+0000","last_fullsized":"2024-02-19T14:56:57.648157+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:57.640036+0000","last_change":"2024-02-19T14:56:57.640036+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.640036+0000","last_undegraded":"2024-02-19T14:56:57.640036+0000","last_fullsized":"2024-02-19T14:56:57.640036+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647893+0000","last_change":"2024-02-19T14:56:57.647893+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647893+0000","last_undegraded":"2024-02-19T14:56:57.647893+0000","last_fullsized":"2024-02-19T14:56:57.647893+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.652426+0000","last_change":"2024-02-19T14:56:57.652426+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.652426+0000","last_undegraded":"2024-02-19T14:56:57.652426+0000","last_fullsized":"2024-02-19T14:56:57.652426+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.647466+0000","last_change":"2024-02-19T14:56:57.647466+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.647466+0000","last_undegraded":"2024-02-19T14:56:57.647466+0000","last_fullsized":"2024-02-19T14:56:57.647466+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651678+0000","last_change":"2024-02-19T14:56:57.651678+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651678+0000","last_undegraded":"2024-02-19T14:56:57.651678+0000","last_fullsized":"2024-02-19T14:56:57.651678+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T14:56:58.645116+0000","last_change":"2024-02-19T14:56:58.645116+0000","last_active":"2024-02-19T14:56:58.645116+0000","last_peered":"2024-02-19T14:56:58.645116+0000","last_clean":"2024-02-19T14:56:58.645116+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:58.645116+0000","last_undegraded":"2024-02-19T14:56:58.645116+0000","last_fullsized":"2024-02-19T14:56:58.645116+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-02-19T14:56:57.651923+0000","last_change":"2024-02-19T14:56:57.651923+0000","last_active":"2024-02-19T14:56:57.640036+0000","last_peered":"2024-02-19T14:56:57.640036+0000","last_clean":"2024-02-19T14:56:57.640036+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:56:57.651923+0000","last_undegraded":"2024-02-19T14:56:57.651923+0000","last_fullsized":"2024-02-19T14:56:57.651923+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":386,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-02-19T14:56:57.653409+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:56:57.653409+0000","last_peered":"2024-02-19T14:56:57.653409+0000","last_clean":"2024-02-19T14:56:57.653409+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:56:57.653409+0000","last_undegraded":"2024-02-19T14:56:57.653409+0000","last_fullsized":"2024-02-19T14:56:57.653409+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2277376,"data_stored":1404364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.504}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.625},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.754},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.871}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.85399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":820205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.623},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":820205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.372},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.54800000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.443},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32828,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.318}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.79200000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.84799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.315},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.56399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.29999999999999999},"min":{"1min":0.247,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47799999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.307}]},{"osd":1,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60699999999999998,"5min":1.512,"15min":1.512},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55000000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38}]},{"osd":4,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.879,"5min":1.133,"15min":1.133},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85999999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.629}]},{"osd":5,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89900000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.58899999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":1.115,"15min":1.115},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":1.236,"15min":1.236},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:55:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86099999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.995,"15min":0.995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1683456,"data_stored":813996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.312},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":1.0740000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33412,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338428,"statfs":{"total":96636764160,"available":96602550272,"internally_reserved":0,"allocated":2277376,"data_stored":1404364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.308}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:03.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.105+0000 7fd09c0a4640 1 Processor -- start 2024-02-19T14:57:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.105+0000 7fd09c0a4640 1 -- start start 2024-02-19T14:57:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09c0a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd094152480 0x7fd0941559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09c0a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd0940c7070 0x7fd09406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09c0a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd094155ef0 con 0x7fd09406d350 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09c0a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09406dec0 con 0x7fd0940c7070 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09c0a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940c75c0 con 0x7fd094152480 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09a641640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd09406d350 0x7fd0940c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45958/0 (socket says 172.21.15.78:45958) 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09a641640 1 -- 172.21.15.78:0/2978640499 learned_addr learned my addr 172.21.15.78:0/2978640499 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd099e40640 1 --2- 172.21.15.78:0/2978640499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd094152480 0x7fd0941559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09963f640 1 --2- 172.21.15.78:0/2978640499 >> v2:172.21.15.150:3300/0 conn(0x7fd0940c7070 0x7fd09406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd098e3e640 1 -- 172.21.15.78:0/2978640499 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564175496 0 0) 0x7fd08c001490 con 0x7fd09406d350 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd098e3e640 1 -- 172.21.15.78:0/2978640499 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd080003520 con 0x7fd09406d350 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd099e40640 1 -- 172.21.15.78:0/2978640499 >> v2:172.21.15.150:3300/0 conn(0x7fd0940c7070 msgr2=0x7fd09406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd099e40640 1 --2- 172.21.15.78:0/2978640499 >> v2:172.21.15.150:3300/0 conn(0x7fd0940c7070 0x7fd09406ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd099e40640 1 -- 172.21.15.78:0/2978640499 >> v1:172.21.15.78:6789/0 conn(0x7fd09406d350 legacy=0x7fd0940c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd099e40640 1 -- 172.21.15.78:0/2978640499 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0940c78a0 con 0x7fd094152480 2024-02-19T14:57:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.106+0000 7fd09963f640 1 --2- 172.21.15.78:0/2978640499 >> v2:172.21.15.150:3300/0 conn(0x7fd0940c7070 0x7fd09406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.107+0000 7fd099e40640 1 --2- 172.21.15.78:0/2978640499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd094152480 0x7fd0941559b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd0900097d0 tx=0x7fd09002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d73dc4d0d3a193d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.107+0000 7fd098e3e640 1 -- 172.21.15.78:0/2978640499 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090031800 con 0x7fd094152480 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.107+0000 7fd098e3e640 1 -- 172.21.15.78:0/2978640499 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090031960 con 0x7fd094152480 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.107+0000 7fd098e3e640 1 -- 172.21.15.78:0/2978640499 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090031cf0 con 0x7fd094152480 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 -- 172.21.15.78:0/2978640499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd094152480 msgr2=0x7fd0941559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/2978640499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd094152480 0x7fd0941559b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd0900097d0 tx=0x7fd09002b8a0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 -- 172.21.15.78:0/2978640499 shutdown_connections 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/2978640499 >> v2:172.21.15.150:3300/0 conn(0x7fd0940c7070 0x7fd09406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/2978640499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd094152480 0x7fd0941559b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 -- 172.21.15.78:0/2978640499 >> 172.21.15.78:0/2978640499 conn(0x7fd0940ca8a0 msgr2=0x7fd0940cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 -- 172.21.15.78:0/2978640499 shutdown_connections 2024-02-19T14:57:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.108+0000 7fd09c0a4640 1 -- 172.21.15.78:0/2978640499 wait complete. 2024-02-19T14:57:04.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.109+0000 7fd09c0a4640 1 Processor -- start 2024-02-19T14:57:04.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.109+0000 7fd09c0a4640 1 -- start start 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0940c7070 0x7fd0940acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd094152480 0x7fd0941dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 0x7fd0941e0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940ab500 con 0x7fd0940c1090 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0941e14e0 con 0x7fd094152480 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0940ac4b0 con 0x7fd0940c7070 2024-02-19T14:57:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 0x7fd0941e0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 0x7fd0941e0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41596/0 (socket says 172.21.15.78:41596) 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 -- 172.21.15.78:0/3089925976 learned_addr learned my addr 172.21.15.78:0/3089925976 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09963f640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd094152480 0x7fd0941dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd099e40640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0940c7070 0x7fd0940acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 -- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0940c7070 msgr2=0x7fd0940acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0940c7070 0x7fd0940acc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 -- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd094152480 msgr2=0x7fd0941dfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd094152480 0x7fd0941dfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 -- 172.21.15.78:0/3089925976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd084009400 con 0x7fd0940c1090 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09a641640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 0x7fd0941e0080 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00cba0 tx=0x7fd08c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c00ad40 con 0x7fd0940c1090 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c012400 con 0x7fd0940c1090 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c00aea0 con 0x7fd0940c1090 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd090009400 con 0x7fd0940c1090 2024-02-19T14:57:04.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.110+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd094158c90 con 0x7fd0940c1090 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.111+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09414fa90 con 0x7fd0940c1090 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.112+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd08c007070 con 0x7fd0940c1090 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.112+0000 7fd08a7fc640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd07806b810 0x7fd07806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.112+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd08c08a080 con 0x7fd0940c1090 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.112+0000 7fd099e40640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd07806b810 0x7fd07806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.113+0000 7fd099e40640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd07806b810 0x7fd07806dcd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd090036000 tx=0x7fd09003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.117+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd08c054a50 con 0x7fd0940c1090 2024-02-19T14:57:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.285+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd09406c610 con 0x7fd07806b810 2024-02-19T14:57:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.292+0000 7fd08a7fc640 1 -- 172.21.15.78:0/3089925976 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290096 (secure 0 0 0) 0x7fd090044400 con 0x7fd07806b810 2024-02-19T14:57:04.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd07806b810 msgr2=0x7fd07806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd07806b810 0x7fd07806dcd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd090036000 tx=0x7fd09003c040 comp rx=0 tx=0).stop 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 msgr2=0x7fd0941e0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 0x7fd0941e0080 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00cba0 tx=0x7fd08c00a430 comp rx=0 tx=0).stop 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 shutdown_connections 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd07806b810 0x7fd07806dcd0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0940c1090 0x7fd0941e0080 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd094152480 0x7fd0941dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 --2- 172.21.15.78:0/3089925976 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0940c7070 0x7fd0940acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 >> 172.21.15.78:0/3089925976 conn(0x7fd0940ca8a0 msgr2=0x7fd0940c9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 shutdown_connections 2024-02-19T14:57:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.295+0000 7fd09c0a4640 1 -- 172.21.15.78:0/3089925976 wait complete. 2024-02-19T14:57:04.308 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-02-19T14:57:04.118909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264260,"kb_used_data":14760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710460,"statfs":{"total":773094113280,"available":772823511040,"internally_reserved":0,"allocated":15114240,"data_stored":7911404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001168"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669478+0000","last_change":"2024-02-19T14:56:59.677644+0000","last_active":"2024-02-19T14:57:00.669478+0000","last_peered":"2024-02-19T14:57:00.669478+0000","last_clean":"2024-02-19T14:57:00.669478+0000","last_became_active":"2024-02-19T14:56:59.677373+0000","last_became_peered":"2024-02-19T14:56:59.677373+0000","last_unstale":"2024-02-19T14:57:00.669478+0000","last_undegraded":"2024-02-19T14:57:00.669478+0000","last_fullsized":"2024-02-19T14:57:00.669478+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:09.037633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669486+0000","last_change":"2024-02-19T14:56:59.672957+0000","last_active":"2024-02-19T14:57:00.669486+0000","last_peered":"2024-02-19T14:57:00.669486+0000","last_clean":"2024-02-19T14:57:00.669486+0000","last_became_active":"2024-02-19T14:56:59.672677+0000","last_became_peered":"2024-02-19T14:56:59.672677+0000","last_unstale":"2024-02-19T14:57:00.669486+0000","last_undegraded":"2024-02-19T14:57:00.669486+0000","last_fullsized":"2024-02-19T14:57:00.669486+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:21.790642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669448+0000","last_change":"2024-02-19T14:56:59.670902+0000","last_active":"2024-02-19T14:57:00.669448+0000","last_peered":"2024-02-19T14:57:00.669448+0000","last_clean":"2024-02-19T14:57:00.669448+0000","last_became_active":"2024-02-19T14:56:59.670608+0000","last_became_peered":"2024-02-19T14:56:59.670608+0000","last_unstale":"2024-02-19T14:57:00.669448+0000","last_undegraded":"2024-02-19T14:57:00.669448+0000","last_fullsized":"2024-02-19T14:57:00.669448+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:05.105129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760676+0000","last_change":"2024-02-19T14:56:59.674985+0000","last_active":"2024-02-19T14:57:00.760676+0000","last_peered":"2024-02-19T14:57:00.760676+0000","last_clean":"2024-02-19T14:57:00.760676+0000","last_became_active":"2024-02-19T14:56:59.674722+0000","last_became_peered":"2024-02-19T14:56:59.674722+0000","last_unstale":"2024-02-19T14:57:00.760676+0000","last_undegraded":"2024-02-19T14:57:00.760676+0000","last_fullsized":"2024-02-19T14:57:00.760676+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.220112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761659+0000","last_change":"2024-02-19T14:56:59.678611+0000","last_active":"2024-02-19T14:57:00.761659+0000","last_peered":"2024-02-19T14:57:00.761659+0000","last_clean":"2024-02-19T14:57:00.761659+0000","last_became_active":"2024-02-19T14:56:59.678350+0000","last_became_peered":"2024-02-19T14:56:59.678350+0000","last_unstale":"2024-02-19T14:57:00.761659+0000","last_undegraded":"2024-02-19T14:57:00.761659+0000","last_fullsized":"2024-02-19T14:57:00.761659+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:13.843441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760880+0000","last_change":"2024-02-19T14:56:59.679879+0000","last_active":"2024-02-19T14:57:00.760880+0000","last_peered":"2024-02-19T14:57:00.760880+0000","last_clean":"2024-02-19T14:57:00.760880+0000","last_became_active":"2024-02-19T14:56:59.678532+0000","last_became_peered":"2024-02-19T14:56:59.678532+0000","last_unstale":"2024-02-19T14:57:00.760880+0000","last_undegraded":"2024-02-19T14:57:00.760880+0000","last_fullsized":"2024-02-19T14:57:00.760880+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:43.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667796+0000","last_change":"2024-02-19T14:56:59.671847+0000","last_active":"2024-02-19T14:57:00.667796+0000","last_peered":"2024-02-19T14:57:00.667796+0000","last_clean":"2024-02-19T14:57:00.667796+0000","last_became_active":"2024-02-19T14:56:59.671586+0000","last_became_peered":"2024-02-19T14:56:59.671586+0000","last_unstale":"2024-02-19T14:57:00.667796+0000","last_undegraded":"2024-02-19T14:57:00.667796+0000","last_fullsized":"2024-02-19T14:57:00.667796+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667989+0000","last_change":"2024-02-19T14:56:59.676911+0000","last_active":"2024-02-19T14:57:00.667989+0000","last_peered":"2024-02-19T14:57:00.667989+0000","last_clean":"2024-02-19T14:57:00.667989+0000","last_became_active":"2024-02-19T14:56:59.676642+0000","last_became_peered":"2024-02-19T14:56:59.676642+0000","last_unstale":"2024-02-19T14:57:00.667989+0000","last_undegraded":"2024-02-19T14:57:00.667989+0000","last_fullsized":"2024-02-19T14:57:00.667989+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:41.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761313+0000","last_change":"2024-02-19T14:56:59.669174+0000","last_active":"2024-02-19T14:57:00.761313+0000","last_peered":"2024-02-19T14:57:00.761313+0000","last_clean":"2024-02-19T14:57:00.761313+0000","last_became_active":"2024-02-19T14:56:59.668890+0000","last_became_peered":"2024-02-19T14:56:59.668890+0000","last_unstale":"2024-02-19T14:57:00.761313+0000","last_undegraded":"2024-02-19T14:57:00.761313+0000","last_fullsized":"2024-02-19T14:57:00.761313+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:39.962805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760970+0000","last_change":"2024-02-19T14:56:59.672943+0000","last_active":"2024-02-19T14:57:00.760970+0000","last_peered":"2024-02-19T14:57:00.760970+0000","last_clean":"2024-02-19T14:57:00.760970+0000","last_became_active":"2024-02-19T14:56:59.672668+0000","last_became_peered":"2024-02-19T14:56:59.672668+0000","last_unstale":"2024-02-19T14:57:00.760970+0000","last_undegraded":"2024-02-19T14:57:00.760970+0000","last_fullsized":"2024-02-19T14:57:00.760970+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:50.415366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761214+0000","last_change":"2024-02-19T14:56:59.669750+0000","last_active":"2024-02-19T14:57:00.761214+0000","last_peered":"2024-02-19T14:57:00.761214+0000","last_clean":"2024-02-19T14:57:00.761214+0000","last_became_active":"2024-02-19T14:56:59.669490+0000","last_became_peered":"2024-02-19T14:56:59.669490+0000","last_unstale":"2024-02-19T14:57:00.761214+0000","last_undegraded":"2024-02-19T14:57:00.761214+0000","last_fullsized":"2024-02-19T14:57:00.761214+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:27.557961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760802+0000","last_change":"2024-02-19T14:56:59.675636+0000","last_active":"2024-02-19T14:57:00.760802+0000","last_peered":"2024-02-19T14:57:00.760802+0000","last_clean":"2024-02-19T14:57:00.760802+0000","last_became_active":"2024-02-19T14:56:59.674106+0000","last_became_peered":"2024-02-19T14:56:59.674106+0000","last_unstale":"2024-02-19T14:57:00.760802+0000","last_undegraded":"2024-02-19T14:57:00.760802+0000","last_fullsized":"2024-02-19T14:57:00.760802+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:49.712701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761587+0000","last_change":"2024-02-19T14:56:59.671801+0000","last_active":"2024-02-19T14:57:00.761587+0000","last_peered":"2024-02-19T14:57:00.761587+0000","last_clean":"2024-02-19T14:57:00.761587+0000","last_became_active":"2024-02-19T14:56:59.671332+0000","last_became_peered":"2024-02-19T14:56:59.671332+0000","last_unstale":"2024-02-19T14:57:00.761587+0000","last_undegraded":"2024-02-19T14:57:00.761587+0000","last_fullsized":"2024-02-19T14:57:00.761587+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:39.015073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760548+0000","last_change":"2024-02-19T14:56:59.681973+0000","last_active":"2024-02-19T14:57:00.760548+0000","last_peered":"2024-02-19T14:57:00.760548+0000","last_clean":"2024-02-19T14:57:00.760548+0000","last_became_active":"2024-02-19T14:56:59.681702+0000","last_became_peered":"2024-02-19T14:56:59.681702+0000","last_unstale":"2024-02-19T14:57:00.760548+0000","last_undegraded":"2024-02-19T14:57:00.760548+0000","last_fullsized":"2024-02-19T14:57:00.760548+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:03:48.192555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669252+0000","last_change":"2024-02-19T14:56:59.677121+0000","last_active":"2024-02-19T14:57:00.669252+0000","last_peered":"2024-02-19T14:57:00.669252+0000","last_clean":"2024-02-19T14:57:00.669252+0000","last_became_active":"2024-02-19T14:56:59.676541+0000","last_became_peered":"2024-02-19T14:56:59.676541+0000","last_unstale":"2024-02-19T14:57:00.669252+0000","last_undegraded":"2024-02-19T14:57:00.669252+0000","last_fullsized":"2024-02-19T14:57:00.669252+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760928+0000","last_change":"2024-02-19T14:56:59.670144+0000","last_active":"2024-02-19T14:57:00.760928+0000","last_peered":"2024-02-19T14:57:00.760928+0000","last_clean":"2024-02-19T14:57:00.760928+0000","last_became_active":"2024-02-19T14:56:59.669884+0000","last_became_peered":"2024-02-19T14:56:59.669884+0000","last_unstale":"2024-02-19T14:57:00.760928+0000","last_undegraded":"2024-02-19T14:57:00.760928+0000","last_fullsized":"2024-02-19T14:57:00.760928+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:11.367956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761230+0000","last_change":"2024-02-19T14:56:59.682420+0000","last_active":"2024-02-19T14:57:00.761230+0000","last_peered":"2024-02-19T14:57:00.761230+0000","last_clean":"2024-02-19T14:57:00.761230+0000","last_became_active":"2024-02-19T14:56:59.682141+0000","last_became_peered":"2024-02-19T14:56:59.682141+0000","last_unstale":"2024-02-19T14:57:00.761230+0000","last_undegraded":"2024-02-19T14:57:00.761230+0000","last_fullsized":"2024-02-19T14:57:00.761230+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760755+0000","last_change":"2024-02-19T14:56:59.681178+0000","last_active":"2024-02-19T14:57:00.760755+0000","last_peered":"2024-02-19T14:57:00.760755+0000","last_clean":"2024-02-19T14:57:00.760755+0000","last_became_active":"2024-02-19T14:56:59.680802+0000","last_became_peered":"2024-02-19T14:56:59.680802+0000","last_unstale":"2024-02-19T14:57:00.760755+0000","last_undegraded":"2024-02-19T14:57:00.760755+0000","last_fullsized":"2024-02-19T14:57:00.760755+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:32.096559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761012+0000","last_change":"2024-02-19T14:56:59.683504+0000","last_active":"2024-02-19T14:57:00.761012+0000","last_peered":"2024-02-19T14:57:00.761012+0000","last_clean":"2024-02-19T14:57:00.761012+0000","last_became_active":"2024-02-19T14:56:59.683207+0000","last_became_peered":"2024-02-19T14:56:59.683207+0000","last_unstale":"2024-02-19T14:57:00.761012+0000","last_undegraded":"2024-02-19T14:57:00.761012+0000","last_fullsized":"2024-02-19T14:57:00.761012+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:50.415366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760700+0000","last_change":"2024-02-19T14:56:59.671258+0000","last_active":"2024-02-19T14:57:00.760700+0000","last_peered":"2024-02-19T14:57:00.760700+0000","last_clean":"2024-02-19T14:57:00.760700+0000","last_became_active":"2024-02-19T14:56:59.670922+0000","last_became_peered":"2024-02-19T14:56:59.670922+0000","last_unstale":"2024-02-19T14:57:00.760700+0000","last_undegraded":"2024-02-19T14:57:00.760700+0000","last_fullsized":"2024-02-19T14:57:00.760700+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:25.573016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761658+0000","last_change":"2024-02-19T14:56:59.669256+0000","last_active":"2024-02-19T14:57:00.761658+0000","last_peered":"2024-02-19T14:57:00.761658+0000","last_clean":"2024-02-19T14:57:00.761658+0000","last_became_active":"2024-02-19T14:56:59.668975+0000","last_became_peered":"2024-02-19T14:56:59.668975+0000","last_unstale":"2024-02-19T14:57:00.761658+0000","last_undegraded":"2024-02-19T14:57:00.761658+0000","last_fullsized":"2024-02-19T14:57:00.761658+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:06.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668252+0000","last_change":"2024-02-19T14:56:59.672561+0000","last_active":"2024-02-19T14:57:00.668252+0000","last_peered":"2024-02-19T14:57:00.668252+0000","last_clean":"2024-02-19T14:57:00.668252+0000","last_became_active":"2024-02-19T14:56:59.672171+0000","last_became_peered":"2024-02-19T14:56:59.672171+0000","last_unstale":"2024-02-19T14:57:00.668252+0000","last_undegraded":"2024-02-19T14:57:00.668252+0000","last_fullsized":"2024-02-19T14:57:00.668252+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:36.585855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668809+0000","last_change":"2024-02-19T14:56:59.677599+0000","last_active":"2024-02-19T14:57:00.668809+0000","last_peered":"2024-02-19T14:57:00.668809+0000","last_clean":"2024-02-19T14:57:00.668809+0000","last_became_active":"2024-02-19T14:56:59.677306+0000","last_became_peered":"2024-02-19T14:56:59.677306+0000","last_unstale":"2024-02-19T14:57:00.668809+0000","last_undegraded":"2024-02-19T14:57:00.668809+0000","last_fullsized":"2024-02-19T14:57:00.668809+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:56.306598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760380+0000","last_change":"2024-02-19T14:56:59.670448+0000","last_active":"2024-02-19T14:57:00.760380+0000","last_peered":"2024-02-19T14:57:00.760380+0000","last_clean":"2024-02-19T14:57:00.760380+0000","last_became_active":"2024-02-19T14:56:59.670155+0000","last_became_peered":"2024-02-19T14:56:59.670155+0000","last_unstale":"2024-02-19T14:57:00.760380+0000","last_undegraded":"2024-02-19T14:57:00.760380+0000","last_fullsized":"2024-02-19T14:57:00.760380+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:07.818574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669346+0000","last_change":"2024-02-19T14:56:59.678644+0000","last_active":"2024-02-19T14:57:00.669346+0000","last_peered":"2024-02-19T14:57:00.669346+0000","last_clean":"2024-02-19T14:57:00.669346+0000","last_became_active":"2024-02-19T14:56:59.678353+0000","last_became_peered":"2024-02-19T14:56:59.678353+0000","last_unstale":"2024-02-19T14:57:00.669346+0000","last_undegraded":"2024-02-19T14:57:00.669346+0000","last_fullsized":"2024-02-19T14:57:00.669346+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:05.105129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668644+0000","last_change":"2024-02-19T14:56:59.679167+0000","last_active":"2024-02-19T14:57:00.668644+0000","last_peered":"2024-02-19T14:57:00.668644+0000","last_clean":"2024-02-19T14:57:00.668644+0000","last_became_active":"2024-02-19T14:56:59.678635+0000","last_became_peered":"2024-02-19T14:56:59.678635+0000","last_unstale":"2024-02-19T14:57:00.668644+0000","last_undegraded":"2024-02-19T14:57:00.668644+0000","last_fullsized":"2024-02-19T14:57:00.668644+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:34.374030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761176+0000","last_change":"2024-02-19T14:56:59.669279+0000","last_active":"2024-02-19T14:57:00.761176+0000","last_peered":"2024-02-19T14:57:00.761176+0000","last_clean":"2024-02-19T14:57:00.761176+0000","last_became_active":"2024-02-19T14:56:59.668990+0000","last_became_peered":"2024-02-19T14:56:59.668990+0000","last_unstale":"2024-02-19T14:57:00.761176+0000","last_undegraded":"2024-02-19T14:57:00.761176+0000","last_fullsized":"2024-02-19T14:57:00.761176+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:37.163331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760625+0000","last_change":"2024-02-19T14:56:59.674868+0000","last_active":"2024-02-19T14:57:00.760625+0000","last_peered":"2024-02-19T14:57:00.760625+0000","last_clean":"2024-02-19T14:57:00.760625+0000","last_became_active":"2024-02-19T14:56:59.674597+0000","last_became_peered":"2024-02-19T14:56:59.674597+0000","last_unstale":"2024-02-19T14:57:00.760625+0000","last_undegraded":"2024-02-19T14:57:00.760625+0000","last_fullsized":"2024-02-19T14:57:00.760625+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:33:36.449995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668644+0000","last_change":"2024-02-19T14:56:59.675815+0000","last_active":"2024-02-19T14:57:00.668644+0000","last_peered":"2024-02-19T14:57:00.668644+0000","last_clean":"2024-02-19T14:57:00.668644+0000","last_became_active":"2024-02-19T14:56:59.671817+0000","last_became_peered":"2024-02-19T14:56:59.671817+0000","last_unstale":"2024-02-19T14:57:00.668644+0000","last_undegraded":"2024-02-19T14:57:00.668644+0000","last_fullsized":"2024-02-19T14:57:00.668644+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:24.209883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760503+0000","last_change":"2024-02-19T14:56:59.675553+0000","last_active":"2024-02-19T14:57:00.760503+0000","last_peered":"2024-02-19T14:57:00.760503+0000","last_clean":"2024-02-19T14:57:00.760503+0000","last_became_active":"2024-02-19T14:56:59.675281+0000","last_became_peered":"2024-02-19T14:56:59.675281+0000","last_unstale":"2024-02-19T14:57:00.760503+0000","last_undegraded":"2024-02-19T14:57:00.760503+0000","last_fullsized":"2024-02-19T14:57:00.760503+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:36:18.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760888+0000","last_change":"2024-02-19T14:56:59.669519+0000","last_active":"2024-02-19T14:57:00.760888+0000","last_peered":"2024-02-19T14:57:00.760888+0000","last_clean":"2024-02-19T14:57:00.760888+0000","last_became_active":"2024-02-19T14:56:59.667272+0000","last_became_peered":"2024-02-19T14:56:59.667272+0000","last_unstale":"2024-02-19T14:57:00.760888+0000","last_undegraded":"2024-02-19T14:57:00.760888+0000","last_fullsized":"2024-02-19T14:57:00.760888+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:40.323947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668136+0000","last_change":"2024-02-19T14:56:59.676162+0000","last_active":"2024-02-19T14:57:00.668136+0000","last_peered":"2024-02-19T14:57:00.668136+0000","last_clean":"2024-02-19T14:57:00.668136+0000","last_became_active":"2024-02-19T14:56:59.675867+0000","last_became_peered":"2024-02-19T14:56:59.675867+0000","last_unstale":"2024-02-19T14:57:00.668136+0000","last_undegraded":"2024-02-19T14:57:00.668136+0000","last_fullsized":"2024-02-19T14:57:00.668136+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:27.213645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669432+0000","last_change":"2024-02-19T14:56:59.665767+0000","last_active":"2024-02-19T14:57:00.669432+0000","last_peered":"2024-02-19T14:57:00.669432+0000","last_clean":"2024-02-19T14:57:00.669432+0000","last_became_active":"2024-02-19T14:56:59.665503+0000","last_became_peered":"2024-02-19T14:56:59.665503+0000","last_unstale":"2024-02-19T14:57:00.669432+0000","last_undegraded":"2024-02-19T14:57:00.669432+0000","last_fullsized":"2024-02-19T14:57:00.669432+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:50.909922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"137'4","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761521+0000","last_change":"2024-02-19T14:56:58.666557+0000","last_active":"2024-02-19T14:57:00.761521+0000","last_peered":"2024-02-19T14:57:00.761521+0000","last_clean":"2024-02-19T14:57:00.761521+0000","last_became_active":"2024-02-19T14:56:58.666286+0000","last_became_peered":"2024-02-19T14:56:58.666286+0000","last_unstale":"2024-02-19T14:57:00.761521+0000","last_undegraded":"2024-02-19T14:57:00.761521+0000","last_fullsized":"2024-02-19T14:57:00.761521+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:49.904842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669498+0000","last_change":"2024-02-19T14:56:59.674314+0000","last_active":"2024-02-19T14:57:00.669498+0000","last_peered":"2024-02-19T14:57:00.669498+0000","last_clean":"2024-02-19T14:57:00.669498+0000","last_became_active":"2024-02-19T14:56:59.673451+0000","last_became_peered":"2024-02-19T14:56:59.673451+0000","last_unstale":"2024-02-19T14:57:00.669498+0000","last_undegraded":"2024-02-19T14:57:00.669498+0000","last_fullsized":"2024-02-19T14:57:00.669498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:09.037633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760607+0000","last_change":"2024-02-19T14:56:58.672364+0000","last_active":"2024-02-19T14:57:00.760607+0000","last_peered":"2024-02-19T14:57:00.760607+0000","last_clean":"2024-02-19T14:57:00.760607+0000","last_became_active":"2024-02-19T14:56:58.672089+0000","last_became_peered":"2024-02-19T14:56:58.672089+0000","last_unstale":"2024-02-19T14:57:00.760607+0000","last_undegraded":"2024-02-19T14:57:00.760607+0000","last_fullsized":"2024-02-19T14:57:00.760607+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:04.100049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668467+0000","last_change":"2024-02-19T14:56:59.670090+0000","last_active":"2024-02-19T14:57:00.668467+0000","last_peered":"2024-02-19T14:57:00.668467+0000","last_clean":"2024-02-19T14:57:00.668467+0000","last_became_active":"2024-02-19T14:56:59.669839+0000","last_became_peered":"2024-02-19T14:56:59.669839+0000","last_unstale":"2024-02-19T14:57:00.668467+0000","last_undegraded":"2024-02-19T14:57:00.668467+0000","last_fullsized":"2024-02-19T14:57:00.668467+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:46.789869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761059+0000","last_change":"2024-02-19T14:56:58.675313+0000","last_active":"2024-02-19T14:57:00.761059+0000","last_peered":"2024-02-19T14:57:00.761059+0000","last_clean":"2024-02-19T14:57:00.761059+0000","last_became_active":"2024-02-19T14:56:58.675047+0000","last_became_peered":"2024-02-19T14:56:58.675047+0000","last_unstale":"2024-02-19T14:57:00.761059+0000","last_undegraded":"2024-02-19T14:57:00.761059+0000","last_fullsized":"2024-02-19T14:57:00.761059+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:52:41.628070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760957+0000","last_change":"2024-02-19T14:56:59.679353+0000","last_active":"2024-02-19T14:57:00.760957+0000","last_peered":"2024-02-19T14:57:00.760957+0000","last_clean":"2024-02-19T14:57:00.760957+0000","last_became_active":"2024-02-19T14:56:59.679070+0000","last_became_peered":"2024-02-19T14:56:59.679070+0000","last_unstale":"2024-02-19T14:57:00.760957+0000","last_undegraded":"2024-02-19T14:57:00.760957+0000","last_fullsized":"2024-02-19T14:57:00.760957+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:11.367956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669346+0000","last_change":"2024-02-19T14:56:58.668754+0000","last_active":"2024-02-19T14:57:00.669346+0000","last_peered":"2024-02-19T14:57:00.669346+0000","last_clean":"2024-02-19T14:57:00.669346+0000","last_became_active":"2024-02-19T14:56:58.668195+0000","last_became_peered":"2024-02-19T14:56:58.668195+0000","last_unstale":"2024-02-19T14:57:00.669346+0000","last_undegraded":"2024-02-19T14:57:00.669346+0000","last_fullsized":"2024-02-19T14:57:00.669346+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760423+0000","last_change":"2024-02-19T14:56:59.674433+0000","last_active":"2024-02-19T14:57:00.760423+0000","last_peered":"2024-02-19T14:57:00.760423+0000","last_clean":"2024-02-19T14:57:00.760423+0000","last_became_active":"2024-02-19T14:56:59.674111+0000","last_became_peered":"2024-02-19T14:56:59.674111+0000","last_unstale":"2024-02-19T14:57:00.760423+0000","last_undegraded":"2024-02-19T14:57:00.760423+0000","last_fullsized":"2024-02-19T14:57:00.760423+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:39.579884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668697+0000","last_change":"2024-02-19T14:56:58.672280+0000","last_active":"2024-02-19T14:57:00.668697+0000","last_peered":"2024-02-19T14:57:00.668697+0000","last_clean":"2024-02-19T14:57:00.668697+0000","last_became_active":"2024-02-19T14:56:58.671978+0000","last_became_peered":"2024-02-19T14:56:58.671978+0000","last_unstale":"2024-02-19T14:57:00.668697+0000","last_undegraded":"2024-02-19T14:57:00.668697+0000","last_fullsized":"2024-02-19T14:57:00.668697+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:38.026526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760682+0000","last_change":"2024-02-19T14:56:59.674348+0000","last_active":"2024-02-19T14:57:00.760682+0000","last_peered":"2024-02-19T14:57:00.760682+0000","last_clean":"2024-02-19T14:57:00.760682+0000","last_became_active":"2024-02-19T14:56:59.674060+0000","last_became_peered":"2024-02-19T14:56:59.674060+0000","last_unstale":"2024-02-19T14:57:00.760682+0000","last_undegraded":"2024-02-19T14:57:00.760682+0000","last_fullsized":"2024-02-19T14:57:00.760682+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:40.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761105+0000","last_change":"2024-02-19T14:56:58.673135+0000","last_active":"2024-02-19T14:57:00.761105+0000","last_peered":"2024-02-19T14:57:00.761105+0000","last_clean":"2024-02-19T14:57:00.761105+0000","last_became_active":"2024-02-19T14:56:58.672822+0000","last_became_peered":"2024-02-19T14:56:58.672822+0000","last_unstale":"2024-02-19T14:57:00.761105+0000","last_undegraded":"2024-02-19T14:57:00.761105+0000","last_fullsized":"2024-02-19T14:57:00.761105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:08.349505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669233+0000","last_change":"2024-02-19T14:56:59.673661+0000","last_active":"2024-02-19T14:57:00.669233+0000","last_peered":"2024-02-19T14:57:00.669233+0000","last_clean":"2024-02-19T14:57:00.669233+0000","last_became_active":"2024-02-19T14:56:59.672005+0000","last_became_peered":"2024-02-19T14:56:59.672005+0000","last_unstale":"2024-02-19T14:57:00.669233+0000","last_undegraded":"2024-02-19T14:57:00.669233+0000","last_fullsized":"2024-02-19T14:57:00.669233+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:53.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667726+0000","last_change":"2024-02-19T14:56:58.666464+0000","last_active":"2024-02-19T14:57:00.667726+0000","last_peered":"2024-02-19T14:57:00.667726+0000","last_clean":"2024-02-19T14:57:00.667726+0000","last_became_active":"2024-02-19T14:56:58.665994+0000","last_became_peered":"2024-02-19T14:56:58.665994+0000","last_unstale":"2024-02-19T14:57:00.667726+0000","last_undegraded":"2024-02-19T14:57:00.667726+0000","last_fullsized":"2024-02-19T14:57:00.667726+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:45.784789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668891+0000","last_change":"2024-02-19T14:56:59.676047+0000","last_active":"2024-02-19T14:57:00.668891+0000","last_peered":"2024-02-19T14:57:00.668891+0000","last_clean":"2024-02-19T14:57:00.668891+0000","last_became_active":"2024-02-19T14:56:59.675731+0000","last_became_peered":"2024-02-19T14:56:59.675731+0000","last_unstale":"2024-02-19T14:57:00.668891+0000","last_undegraded":"2024-02-19T14:57:00.668891+0000","last_fullsized":"2024-02-19T14:57:00.668891+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:50.909922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761024+0000","last_change":"2024-02-19T14:56:58.663619+0000","last_active":"2024-02-19T14:57:00.761024+0000","last_peered":"2024-02-19T14:57:00.761024+0000","last_clean":"2024-02-19T14:57:00.761024+0000","last_became_active":"2024-02-19T14:56:58.663358+0000","last_became_peered":"2024-02-19T14:56:58.663358+0000","last_unstale":"2024-02-19T14:57:00.761024+0000","last_undegraded":"2024-02-19T14:57:00.761024+0000","last_fullsized":"2024-02-19T14:57:00.761024+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:35.580775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669142+0000","last_change":"2024-02-19T14:56:59.669001+0000","last_active":"2024-02-19T14:57:00.669142+0000","last_peered":"2024-02-19T14:57:00.669142+0000","last_clean":"2024-02-19T14:57:00.669142+0000","last_became_active":"2024-02-19T14:56:59.667676+0000","last_became_peered":"2024-02-19T14:56:59.667676+0000","last_unstale":"2024-02-19T14:57:00.669142+0000","last_undegraded":"2024-02-19T14:57:00.669142+0000","last_fullsized":"2024-02-19T14:57:00.669142+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:38:21.118003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760966+0000","last_change":"2024-02-19T14:56:58.670376+0000","last_active":"2024-02-19T14:57:00.760966+0000","last_peered":"2024-02-19T14:57:00.760966+0000","last_clean":"2024-02-19T14:57:00.760966+0000","last_became_active":"2024-02-19T14:56:58.670065+0000","last_became_peered":"2024-02-19T14:56:58.670065+0000","last_unstale":"2024-02-19T14:57:00.760966+0000","last_undegraded":"2024-02-19T14:57:00.760966+0000","last_fullsized":"2024-02-19T14:57:00.760966+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:08.032552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761353+0000","last_change":"2024-02-19T14:56:59.669101+0000","last_active":"2024-02-19T14:57:00.761353+0000","last_peered":"2024-02-19T14:57:00.761353+0000","last_clean":"2024-02-19T14:57:00.761353+0000","last_became_active":"2024-02-19T14:56:59.668601+0000","last_became_peered":"2024-02-19T14:56:59.668601+0000","last_unstale":"2024-02-19T14:57:00.761353+0000","last_undegraded":"2024-02-19T14:57:00.761353+0000","last_fullsized":"2024-02-19T14:57:00.761353+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:12:40.244630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669055+0000","last_change":"2024-02-19T14:56:58.673499+0000","last_active":"2024-02-19T14:57:00.669055+0000","last_peered":"2024-02-19T14:57:00.669055+0000","last_clean":"2024-02-19T14:57:00.669055+0000","last_became_active":"2024-02-19T14:56:58.673014+0000","last_became_peered":"2024-02-19T14:56:58.673014+0000","last_unstale":"2024-02-19T14:57:00.669055+0000","last_undegraded":"2024-02-19T14:57:00.669055+0000","last_fullsized":"2024-02-19T14:57:00.669055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:55.301518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760881+0000","last_change":"2024-02-19T14:56:59.680952+0000","last_active":"2024-02-19T14:57:00.760881+0000","last_peered":"2024-02-19T14:57:00.760881+0000","last_clean":"2024-02-19T14:57:00.760881+0000","last_became_active":"2024-02-19T14:56:59.680634+0000","last_became_peered":"2024-02-19T14:56:59.680634+0000","last_unstale":"2024-02-19T14:57:00.760881+0000","last_undegraded":"2024-02-19T14:57:00.760881+0000","last_fullsized":"2024-02-19T14:57:00.760881+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:08.161866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"137'3","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761402+0000","last_change":"2024-02-19T14:56:58.664991+0000","last_active":"2024-02-19T14:57:00.761402+0000","last_peered":"2024-02-19T14:57:00.761402+0000","last_clean":"2024-02-19T14:57:00.761402+0000","last_became_active":"2024-02-19T14:56:58.664715+0000","last_became_peered":"2024-02-19T14:56:58.664715+0000","last_unstale":"2024-02-19T14:57:00.761402+0000","last_undegraded":"2024-02-19T14:57:00.761402+0000","last_fullsized":"2024-02-19T14:57:00.761402+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:14.196709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668295+0000","last_change":"2024-02-19T14:56:59.669119+0000","last_active":"2024-02-19T14:57:00.668295+0000","last_peered":"2024-02-19T14:57:00.668295+0000","last_clean":"2024-02-19T14:57:00.668295+0000","last_became_active":"2024-02-19T14:56:59.668836+0000","last_became_peered":"2024-02-19T14:56:59.668836+0000","last_unstale":"2024-02-19T14:57:00.668295+0000","last_undegraded":"2024-02-19T14:57:00.668295+0000","last_fullsized":"2024-02-19T14:57:00.668295+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:14.400279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761709+0000","last_change":"2024-02-19T14:56:58.666985+0000","last_active":"2024-02-19T14:57:00.761709+0000","last_peered":"2024-02-19T14:57:00.761709+0000","last_clean":"2024-02-19T14:57:00.761709+0000","last_became_active":"2024-02-19T14:56:58.666667+0000","last_became_peered":"2024-02-19T14:56:58.666667+0000","last_unstale":"2024-02-19T14:57:00.761709+0000","last_undegraded":"2024-02-19T14:57:00.761709+0000","last_fullsized":"2024-02-19T14:57:00.761709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:35.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669279+0000","last_change":"2024-02-19T14:56:59.677403+0000","last_active":"2024-02-19T14:57:00.669279+0000","last_peered":"2024-02-19T14:57:00.669279+0000","last_clean":"2024-02-19T14:57:00.669279+0000","last_became_active":"2024-02-19T14:56:59.676857+0000","last_became_peered":"2024-02-19T14:56:59.676857+0000","last_unstale":"2024-02-19T14:57:00.669279+0000","last_undegraded":"2024-02-19T14:57:00.669279+0000","last_fullsized":"2024-02-19T14:57:00.669279+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:36.387245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760486+0000","last_change":"2024-02-19T14:56:58.673001+0000","last_active":"2024-02-19T14:57:00.760486+0000","last_peered":"2024-02-19T14:57:00.760486+0000","last_clean":"2024-02-19T14:57:00.760486+0000","last_became_active":"2024-02-19T14:56:58.672751+0000","last_became_peered":"2024-02-19T14:56:58.672751+0000","last_unstale":"2024-02-19T14:57:00.760486+0000","last_undegraded":"2024-02-19T14:57:00.760486+0000","last_fullsized":"2024-02-19T14:57:00.760486+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:13.395199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761015+0000","last_change":"2024-02-19T14:56:59.667686+0000","last_active":"2024-02-19T14:57:00.761015+0000","last_peered":"2024-02-19T14:57:00.761015+0000","last_clean":"2024-02-19T14:57:00.761015+0000","last_became_active":"2024-02-19T14:56:59.667391+0000","last_became_peered":"2024-02-19T14:56:59.667391+0000","last_unstale":"2024-02-19T14:57:00.761015+0000","last_undegraded":"2024-02-19T14:57:00.761015+0000","last_fullsized":"2024-02-19T14:57:00.761015+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:25.913809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760722+0000","last_change":"2024-02-19T14:56:58.670054+0000","last_active":"2024-02-19T14:57:00.760722+0000","last_peered":"2024-02-19T14:57:00.760722+0000","last_clean":"2024-02-19T14:57:00.760722+0000","last_became_active":"2024-02-19T14:56:58.669599+0000","last_became_peered":"2024-02-19T14:56:58.669599+0000","last_unstale":"2024-02-19T14:57:00.760722+0000","last_undegraded":"2024-02-19T14:57:00.760722+0000","last_fullsized":"2024-02-19T14:57:00.760722+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:26.208565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669254+0000","last_change":"2024-02-19T14:56:59.671613+0000","last_active":"2024-02-19T14:57:00.669254+0000","last_peered":"2024-02-19T14:57:00.669254+0000","last_clean":"2024-02-19T14:57:00.669254+0000","last_became_active":"2024-02-19T14:56:59.671340+0000","last_became_peered":"2024-02-19T14:56:59.671340+0000","last_unstale":"2024-02-19T14:57:00.669254+0000","last_undegraded":"2024-02-19T14:57:00.669254+0000","last_fullsized":"2024-02-19T14:57:00.669254+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:34.374030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"137'8","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667544+0000","last_change":"2024-02-19T14:56:58.668292+0000","last_active":"2024-02-19T14:57:00.667544+0000","last_peered":"2024-02-19T14:57:00.667544+0000","last_clean":"2024-02-19T14:57:00.667544+0000","last_became_active":"2024-02-19T14:56:58.668000+0000","last_became_peered":"2024-02-19T14:56:58.668000+0000","last_unstale":"2024-02-19T14:57:00.667544+0000","last_undegraded":"2024-02-19T14:57:00.667544+0000","last_fullsized":"2024-02-19T14:57:00.667544+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760620+0000","last_change":"2024-02-19T14:56:59.674229+0000","last_active":"2024-02-19T14:57:00.760620+0000","last_peered":"2024-02-19T14:57:00.760620+0000","last_clean":"2024-02-19T14:57:00.760620+0000","last_became_active":"2024-02-19T14:56:59.673948+0000","last_became_peered":"2024-02-19T14:56:59.673948+0000","last_unstale":"2024-02-19T14:57:00.760620+0000","last_undegraded":"2024-02-19T14:57:00.760620+0000","last_fullsized":"2024-02-19T14:57:00.760620+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:43.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760686+0000","last_change":"2024-02-19T14:56:58.668198+0000","last_active":"2024-02-19T14:57:00.760686+0000","last_peered":"2024-02-19T14:57:00.760686+0000","last_clean":"2024-02-19T14:57:00.760686+0000","last_became_active":"2024-02-19T14:56:58.667965+0000","last_became_peered":"2024-02-19T14:56:58.667965+0000","last_unstale":"2024-02-19T14:57:00.760686+0000","last_undegraded":"2024-02-19T14:57:00.760686+0000","last_fullsized":"2024-02-19T14:57:00.760686+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:28.054550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761080+0000","last_change":"2024-02-19T14:56:59.684025+0000","last_active":"2024-02-19T14:57:00.761080+0000","last_peered":"2024-02-19T14:57:00.761080+0000","last_clean":"2024-02-19T14:57:00.761080+0000","last_became_active":"2024-02-19T14:56:59.683791+0000","last_became_peered":"2024-02-19T14:56:59.683791+0000","last_unstale":"2024-02-19T14:57:00.761080+0000","last_undegraded":"2024-02-19T14:57:00.761080+0000","last_fullsized":"2024-02-19T14:57:00.761080+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:25.913809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760700+0000","last_change":"2024-02-19T14:56:58.669910+0000","last_active":"2024-02-19T14:57:00.760700+0000","last_peered":"2024-02-19T14:57:00.760700+0000","last_clean":"2024-02-19T14:57:00.760700+0000","last_became_active":"2024-02-19T14:56:58.669594+0000","last_became_peered":"2024-02-19T14:56:58.669594+0000","last_unstale":"2024-02-19T14:57:00.760700+0000","last_undegraded":"2024-02-19T14:57:00.760700+0000","last_fullsized":"2024-02-19T14:57:00.760700+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:20.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668860+0000","last_change":"2024-02-19T14:56:59.679318+0000","last_active":"2024-02-19T14:57:00.668860+0000","last_peered":"2024-02-19T14:57:00.668860+0000","last_clean":"2024-02-19T14:57:00.668860+0000","last_became_active":"2024-02-19T14:56:59.678816+0000","last_became_peered":"2024-02-19T14:56:59.678816+0000","last_unstale":"2024-02-19T14:57:00.668860+0000","last_undegraded":"2024-02-19T14:57:00.668860+0000","last_fullsized":"2024-02-19T14:57:00.668860+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:36.387245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669503+0000","last_change":"2024-02-19T14:56:58.669660+0000","last_active":"2024-02-19T14:57:00.669503+0000","last_peered":"2024-02-19T14:57:00.669503+0000","last_clean":"2024-02-19T14:57:00.669503+0000","last_became_active":"2024-02-19T14:56:58.669395+0000","last_became_peered":"2024-02-19T14:56:58.669395+0000","last_unstale":"2024-02-19T14:57:00.669503+0000","last_undegraded":"2024-02-19T14:57:00.669503+0000","last_fullsized":"2024-02-19T14:57:00.669503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:45.784789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669458+0000","last_change":"2024-02-19T14:56:59.676902+0000","last_active":"2024-02-19T14:57:00.669458+0000","last_peered":"2024-02-19T14:57:00.669458+0000","last_clean":"2024-02-19T14:57:00.669458+0000","last_became_active":"2024-02-19T14:56:59.676621+0000","last_became_peered":"2024-02-19T14:56:59.676621+0000","last_unstale":"2024-02-19T14:57:00.669458+0000","last_undegraded":"2024-02-19T14:57:00.669458+0000","last_fullsized":"2024-02-19T14:57:00.669458+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:21.790642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760848+0000","last_change":"2024-02-19T14:56:58.673331+0000","last_active":"2024-02-19T14:57:00.760848+0000","last_peered":"2024-02-19T14:57:00.760848+0000","last_clean":"2024-02-19T14:57:00.760848+0000","last_became_active":"2024-02-19T14:56:58.673067+0000","last_became_peered":"2024-02-19T14:56:58.673067+0000","last_unstale":"2024-02-19T14:57:00.760848+0000","last_undegraded":"2024-02-19T14:57:00.760848+0000","last_fullsized":"2024-02-19T14:57:00.760848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760722+0000","last_change":"2024-02-19T14:56:59.684112+0000","last_active":"2024-02-19T14:57:00.760722+0000","last_peered":"2024-02-19T14:57:00.760722+0000","last_clean":"2024-02-19T14:57:00.760722+0000","last_became_active":"2024-02-19T14:56:59.683866+0000","last_became_peered":"2024-02-19T14:56:59.683866+0000","last_unstale":"2024-02-19T14:57:00.760722+0000","last_undegraded":"2024-02-19T14:57:00.760722+0000","last_fullsized":"2024-02-19T14:57:00.760722+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:40.323947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760568+0000","last_change":"2024-02-19T14:56:58.672326+0000","last_active":"2024-02-19T14:57:00.760568+0000","last_peered":"2024-02-19T14:57:00.760568+0000","last_clean":"2024-02-19T14:57:00.760568+0000","last_became_active":"2024-02-19T14:56:58.672056+0000","last_became_peered":"2024-02-19T14:56:58.672056+0000","last_unstale":"2024-02-19T14:57:00.760568+0000","last_undegraded":"2024-02-19T14:57:00.760568+0000","last_fullsized":"2024-02-19T14:57:00.760568+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:04.357396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669386+0000","last_change":"2024-02-19T14:56:59.679132+0000","last_active":"2024-02-19T14:57:00.669386+0000","last_peered":"2024-02-19T14:57:00.669386+0000","last_clean":"2024-02-19T14:57:00.669386+0000","last_became_active":"2024-02-19T14:56:59.678819+0000","last_became_peered":"2024-02-19T14:56:59.678819+0000","last_unstale":"2024-02-19T14:57:00.669386+0000","last_undegraded":"2024-02-19T14:57:00.669386+0000","last_fullsized":"2024-02-19T14:57:00.669386+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:09.354585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668134+0000","last_change":"2024-02-19T14:56:58.664244+0000","last_active":"2024-02-19T14:57:00.668134+0000","last_peered":"2024-02-19T14:57:00.668134+0000","last_clean":"2024-02-19T14:57:00.668134+0000","last_became_active":"2024-02-19T14:56:58.663902+0000","last_became_peered":"2024-02-19T14:56:58.663902+0000","last_unstale":"2024-02-19T14:57:00.668134+0000","last_undegraded":"2024-02-19T14:57:00.668134+0000","last_fullsized":"2024-02-19T14:57:00.668134+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:10.863066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760504+0000","last_change":"2024-02-19T14:56:59.677739+0000","last_active":"2024-02-19T14:57:00.760504+0000","last_peered":"2024-02-19T14:57:00.760504+0000","last_clean":"2024-02-19T14:57:00.760504+0000","last_became_active":"2024-02-19T14:56:59.677418+0000","last_became_peered":"2024-02-19T14:56:59.677418+0000","last_unstale":"2024-02-19T14:57:00.760504+0000","last_undegraded":"2024-02-19T14:57:00.760504+0000","last_fullsized":"2024-02-19T14:57:00.760504+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:13:48.895089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760318+0000","last_change":"2024-02-19T14:56:58.672372+0000","last_active":"2024-02-19T14:57:00.760318+0000","last_peered":"2024-02-19T14:57:00.760318+0000","last_clean":"2024-02-19T14:57:00.760318+0000","last_became_active":"2024-02-19T14:56:58.672074+0000","last_became_peered":"2024-02-19T14:56:58.672074+0000","last_unstale":"2024-02-19T14:57:00.760318+0000","last_undegraded":"2024-02-19T14:57:00.760318+0000","last_fullsized":"2024-02-19T14:57:00.760318+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:20.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669638+0000","last_change":"2024-02-19T14:56:59.676678+0000","last_active":"2024-02-19T14:57:00.669638+0000","last_peered":"2024-02-19T14:57:00.669638+0000","last_clean":"2024-02-19T14:57:00.669638+0000","last_became_active":"2024-02-19T14:56:59.676424+0000","last_became_peered":"2024-02-19T14:56:59.676424+0000","last_unstale":"2024-02-19T14:57:00.669638+0000","last_undegraded":"2024-02-19T14:57:00.669638+0000","last_fullsized":"2024-02-19T14:57:00.669638+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:05.362476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668478+0000","last_change":"2024-02-19T14:56:58.669077+0000","last_active":"2024-02-19T14:57:00.668478+0000","last_peered":"2024-02-19T14:57:00.668478+0000","last_clean":"2024-02-19T14:57:00.668478+0000","last_became_active":"2024-02-19T14:56:58.668349+0000","last_became_peered":"2024-02-19T14:56:58.668349+0000","last_unstale":"2024-02-19T14:57:00.668478+0000","last_undegraded":"2024-02-19T14:57:00.668478+0000","last_fullsized":"2024-02-19T14:57:00.668478+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:03:38.417380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761364+0000","last_change":"2024-02-19T14:56:59.679793+0000","last_active":"2024-02-19T14:57:00.761364+0000","last_peered":"2024-02-19T14:57:00.761364+0000","last_clean":"2024-02-19T14:57:00.761364+0000","last_became_active":"2024-02-19T14:56:59.677982+0000","last_became_peered":"2024-02-19T14:56:59.677982+0000","last_unstale":"2024-02-19T14:57:00.761364+0000","last_undegraded":"2024-02-19T14:57:00.761364+0000","last_fullsized":"2024-02-19T14:57:00.761364+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:15.488805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761179+0000","last_change":"2024-02-19T14:56:58.672366+0000","last_active":"2024-02-19T14:57:00.761179+0000","last_peered":"2024-02-19T14:57:00.761179+0000","last_clean":"2024-02-19T14:57:00.761179+0000","last_became_active":"2024-02-19T14:56:58.672085+0000","last_became_peered":"2024-02-19T14:56:58.672085+0000","last_unstale":"2024-02-19T14:57:00.761179+0000","last_undegraded":"2024-02-19T14:57:00.761179+0000","last_fullsized":"2024-02-19T14:57:00.761179+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760763+0000","last_change":"2024-02-19T14:56:59.681440+0000","last_active":"2024-02-19T14:57:00.760763+0000","last_peered":"2024-02-19T14:57:00.760763+0000","last_clean":"2024-02-19T14:57:00.760763+0000","last_became_active":"2024-02-19T14:56:59.681022+0000","last_became_peered":"2024-02-19T14:56:59.681022+0000","last_unstale":"2024-02-19T14:57:00.760763+0000","last_undegraded":"2024-02-19T14:57:00.760763+0000","last_fullsized":"2024-02-19T14:57:00.760763+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:57.697404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669083+0000","last_change":"2024-02-19T14:56:58.670839+0000","last_active":"2024-02-19T14:57:00.669083+0000","last_peered":"2024-02-19T14:57:00.669083+0000","last_clean":"2024-02-19T14:57:00.669083+0000","last_became_active":"2024-02-19T14:56:58.670507+0000","last_became_peered":"2024-02-19T14:56:58.670507+0000","last_unstale":"2024-02-19T14:57:00.669083+0000","last_undegraded":"2024-02-19T14:57:00.669083+0000","last_fullsized":"2024-02-19T14:57:00.669083+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:38:20.112923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669006+0000","last_change":"2024-02-19T14:56:59.678942+0000","last_active":"2024-02-19T14:57:00.669006+0000","last_peered":"2024-02-19T14:57:00.669006+0000","last_clean":"2024-02-19T14:57:00.669006+0000","last_became_active":"2024-02-19T14:56:59.676927+0000","last_became_peered":"2024-02-19T14:56:59.676927+0000","last_unstale":"2024-02-19T14:57:00.669006+0000","last_undegraded":"2024-02-19T14:57:00.669006+0000","last_fullsized":"2024-02-19T14:57:00.669006+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:09.354585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669547+0000","last_change":"2024-02-19T14:56:58.660184+0000","last_active":"2024-02-19T14:57:00.669547+0000","last_peered":"2024-02-19T14:57:00.669547+0000","last_clean":"2024-02-19T14:57:00.669547+0000","last_became_active":"2024-02-19T14:56:58.659918+0000","last_became_peered":"2024-02-19T14:56:58.659918+0000","last_unstale":"2024-02-19T14:57:00.669547+0000","last_undegraded":"2024-02-19T14:57:00.669547+0000","last_fullsized":"2024-02-19T14:57:00.669547+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761435+0000","last_change":"2024-02-19T14:56:59.678477+0000","last_active":"2024-02-19T14:57:00.761435+0000","last_peered":"2024-02-19T14:57:00.761435+0000","last_clean":"2024-02-19T14:57:00.761435+0000","last_became_active":"2024-02-19T14:56:59.678159+0000","last_became_peered":"2024-02-19T14:56:59.678159+0000","last_unstale":"2024-02-19T14:57:00.761435+0000","last_undegraded":"2024-02-19T14:57:00.761435+0000","last_fullsized":"2024-02-19T14:57:00.761435+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:18.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761441+0000","last_change":"2024-02-19T14:56:58.667075+0000","last_active":"2024-02-19T14:57:00.761441+0000","last_peered":"2024-02-19T14:57:00.761441+0000","last_clean":"2024-02-19T14:57:00.761441+0000","last_became_active":"2024-02-19T14:56:58.666748+0000","last_became_peered":"2024-02-19T14:56:58.666748+0000","last_unstale":"2024-02-19T14:57:00.761441+0000","last_undegraded":"2024-02-19T14:57:00.761441+0000","last_fullsized":"2024-02-19T14:57:00.761441+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:33.368950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668430+0000","last_change":"2024-02-19T14:56:59.680460+0000","last_active":"2024-02-19T14:57:00.668430+0000","last_peered":"2024-02-19T14:57:00.668430+0000","last_clean":"2024-02-19T14:57:00.668430+0000","last_became_active":"2024-02-19T14:56:59.680154+0000","last_became_peered":"2024-02-19T14:56:59.680154+0000","last_unstale":"2024-02-19T14:57:00.668430+0000","last_undegraded":"2024-02-19T14:57:00.668430+0000","last_fullsized":"2024-02-19T14:57:00.668430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:50.758440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668947+0000","last_change":"2024-02-19T14:56:58.672776+0000","last_active":"2024-02-19T14:57:00.668947+0000","last_peered":"2024-02-19T14:57:00.668947+0000","last_clean":"2024-02-19T14:57:00.668947+0000","last_became_active":"2024-02-19T14:56:58.672154+0000","last_became_peered":"2024-02-19T14:56:58.672154+0000","last_unstale":"2024-02-19T14:57:00.668947+0000","last_undegraded":"2024-02-19T14:57:00.668947+0000","last_fullsized":"2024-02-19T14:57:00.668947+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:23.204802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761142+0000","last_change":"2024-02-19T14:56:59.681056+0000","last_active":"2024-02-19T14:57:00.761142+0000","last_peered":"2024-02-19T14:57:00.761142+0000","last_clean":"2024-02-19T14:57:00.761142+0000","last_became_active":"2024-02-19T14:56:59.680740+0000","last_became_peered":"2024-02-19T14:56:59.680740+0000","last_unstale":"2024-02-19T14:57:00.761142+0000","last_undegraded":"2024-02-19T14:57:00.761142+0000","last_fullsized":"2024-02-19T14:57:00.761142+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:53.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760440+0000","last_change":"2024-02-19T14:56:58.667461+0000","last_active":"2024-02-19T14:57:00.760440+0000","last_peered":"2024-02-19T14:57:00.760440+0000","last_clean":"2024-02-19T14:57:00.760440+0000","last_became_active":"2024-02-19T14:56:58.667182+0000","last_became_peered":"2024-02-19T14:56:58.667182+0000","last_unstale":"2024-02-19T14:57:00.760440+0000","last_undegraded":"2024-02-19T14:57:00.760440+0000","last_fullsized":"2024-02-19T14:57:00.760440+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:27:51.843246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669251+0000","last_change":"2024-02-19T14:56:59.673100+0000","last_active":"2024-02-19T14:57:00.669251+0000","last_peered":"2024-02-19T14:57:00.669251+0000","last_clean":"2024-02-19T14:57:00.669251+0000","last_became_active":"2024-02-19T14:56:59.672757+0000","last_became_peered":"2024-02-19T14:56:59.672757+0000","last_unstale":"2024-02-19T14:57:00.669251+0000","last_undegraded":"2024-02-19T14:57:00.669251+0000","last_fullsized":"2024-02-19T14:57:00.669251+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:36.585855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667848+0000","last_change":"2024-02-19T14:56:58.664744+0000","last_active":"2024-02-19T14:57:00.667848+0000","last_peered":"2024-02-19T14:57:00.667848+0000","last_clean":"2024-02-19T14:57:00.667848+0000","last_became_active":"2024-02-19T14:56:58.663422+0000","last_became_peered":"2024-02-19T14:56:58.663422+0000","last_unstale":"2024-02-19T14:57:00.667848+0000","last_undegraded":"2024-02-19T14:57:00.667848+0000","last_fullsized":"2024-02-19T14:57:00.667848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668501+0000","last_change":"2024-02-19T14:56:59.675481+0000","last_active":"2024-02-19T14:57:00.668501+0000","last_peered":"2024-02-19T14:57:00.668501+0000","last_clean":"2024-02-19T14:57:00.668501+0000","last_became_active":"2024-02-19T14:56:59.671484+0000","last_became_peered":"2024-02-19T14:56:59.671484+0000","last_unstale":"2024-02-19T14:57:00.668501+0000","last_undegraded":"2024-02-19T14:57:00.668501+0000","last_fullsized":"2024-02-19T14:57:00.668501+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:52.205158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760386+0000","last_change":"2024-02-19T14:56:58.673320+0000","last_active":"2024-02-19T14:57:00.760386+0000","last_peered":"2024-02-19T14:57:00.760386+0000","last_clean":"2024-02-19T14:57:00.760386+0000","last_became_active":"2024-02-19T14:56:58.673046+0000","last_became_peered":"2024-02-19T14:56:58.673046+0000","last_unstale":"2024-02-19T14:57:00.760386+0000","last_undegraded":"2024-02-19T14:57:00.760386+0000","last_fullsized":"2024-02-19T14:57:00.760386+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:04.357396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669280+0000","last_change":"2024-02-19T14:56:59.669865+0000","last_active":"2024-02-19T14:57:00.669280+0000","last_peered":"2024-02-19T14:57:00.669280+0000","last_clean":"2024-02-19T14:57:00.669280+0000","last_became_active":"2024-02-19T14:56:59.669568+0000","last_became_peered":"2024-02-19T14:56:59.669568+0000","last_unstale":"2024-02-19T14:57:00.669280+0000","last_undegraded":"2024-02-19T14:57:00.669280+0000","last_fullsized":"2024-02-19T14:57:00.669280+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:05.809396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"137'6","reported_seq":20,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761112+0000","last_change":"2024-02-19T14:56:58.668744+0000","last_active":"2024-02-19T14:57:00.761112+0000","last_peered":"2024-02-19T14:57:00.761112+0000","last_clean":"2024-02-19T14:57:00.761112+0000","last_became_active":"2024-02-19T14:56:58.668464+0000","last_became_peered":"2024-02-19T14:56:58.668464+0000","last_unstale":"2024-02-19T14:57:00.761112+0000","last_undegraded":"2024-02-19T14:57:00.761112+0000","last_fullsized":"2024-02-19T14:57:00.761112+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:08.032552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":392,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760484+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:00.760484+0000","last_peered":"2024-02-19T14:57:00.760484+0000","last_clean":"2024-02-19T14:57:00.760484+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:00.760484+0000","last_undegraded":"2024-02-19T14:57:00.760484+0000","last_fullsized":"2024-02-19T14:57:00.760484+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42099999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.378}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.436},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.26800000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.307},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.36599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":840801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.443},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.433}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.32900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.27500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.311}]}]},{"osd":5,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1732608,"data_stored":840731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.498}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.438},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.28199999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.254}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.41499999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1744896,"data_stored":842343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.435}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.245},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.25900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.29299999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.308},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.33200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1740800,"data_stored":841347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.307}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.41399999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.373}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.38400000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1740800,"data_stored":840843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.42099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.505}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":840801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.47799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2351104,"data_stored":1433417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41799999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.442}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:04.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T14:57:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.445+0000 7fc541d0d640 1 Processor -- start 2024-02-19T14:57:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.445+0000 7fc541d0d640 1 -- start start 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc541d0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc541d0d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c155ef0 con 0x7fc53c0c7070 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc541d0d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c06dec0 con 0x7fc53c0c87a0 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc541d0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c06d520 con 0x7fc53c152480 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc53affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc53c0c7070 0x7fc53c06ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45968/0 (socket says 172.21.15.78:45968) 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc53affd640 1 -- 172.21.15.78:0/3744132792 learned_addr learned my addr 172.21.15.78:0/3744132792 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc53b7fe640 1 --2- 172.21.15.78:0/3744132792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677939430 0 0) 0x7fc530001110 con 0x7fc53c0c7070 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.446+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc524003520 con 0x7fc53c0c7070 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.447+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250514099 0 0) 0x7fc53c06dec0 con 0x7fc53c0c87a0 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.447+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc530001110 con 0x7fc53c0c87a0 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.447+0000 7fc53b7fe640 1 -- 172.21.15.78:0/3744132792 >> v1:172.21.15.150:6789/0 conn(0x7fc53c0c87a0 legacy=0x7fc53c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.447+0000 7fc53b7fe640 1 -- 172.21.15.78:0/3744132792 >> v1:172.21.15.78:6789/0 conn(0x7fc53c0c7070 legacy=0x7fc53c06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.447+0000 7fc53b7fe640 1 -- 172.21.15.78:0/3744132792 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0c75c0 con 0x7fc53c152480 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.447+0000 7fc53b7fe640 1 --2- 172.21.15.78:0/3744132792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c1559b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0097d0 tx=0x7fc52c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f561d4ebfe4580b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c031890 con 0x7fc53c152480 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c0319f0 con 0x7fc53c152480 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc53a7fc640 1 -- 172.21.15.78:0/3744132792 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c031d80 con 0x7fc53c152480 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 -- 172.21.15.78:0/3744132792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 msgr2=0x7fc53c1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 --2- 172.21.15.78:0/3744132792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c1559b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0097d0 tx=0x7fc52c02b930 comp rx=0 tx=0).stop 2024-02-19T14:57:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 -- 172.21.15.78:0/3744132792 shutdown_connections 2024-02-19T14:57:04.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 --2- 172.21.15.78:0/3744132792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c1559b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 -- 172.21.15.78:0/3744132792 >> 172.21.15.78:0/3744132792 conn(0x7fc53c0ca8a0 msgr2=0x7fc53c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:04.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 -- 172.21.15.78:0/3744132792 shutdown_connections 2024-02-19T14:57:04.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.448+0000 7fc541d0d640 1 -- 172.21.15.78:0/3744132792 wait complete. 2024-02-19T14:57:04.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.449+0000 7fc541d0d640 1 Processor -- start 2024-02-19T14:57:04.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.449+0000 7fc541d0d640 1 -- start start 2024-02-19T14:57:04.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc541d0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 0x7fc53c0ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc541d0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc53c0c87a0 0x7fc53c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc541d0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c14db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc541d0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1ea7e0 con 0x7fc53c0c7070 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc541d0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0bfb90 con 0x7fc53c0c87a0 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc541d0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0c1680 con 0x7fc53c152480 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 0x7fc53c0ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 0x7fc53c0ac620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41610/0 (socket says 172.21.15.78:41610) 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 -- 172.21.15.78:0/2901061581 learned_addr learned my addr 172.21.15.78:0/2901061581 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53bfff640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c14db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53affd640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc53c0c87a0 0x7fc53c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 -- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 msgr2=0x7fc53c14db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c14db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 -- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc53c0c87a0 msgr2=0x7fc53c157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc53c0c87a0 0x7fc53c157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.450+0000 7fc53b7fe640 1 -- 172.21.15.78:0/2901061581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c009400 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.451+0000 7fc53b7fe640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 0x7fc53c0ac620 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0097d0 tx=0x7fc52c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.451+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c03abe0 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.451+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c03ad60 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.451+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c044630 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.451+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c158a80 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.451+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c158d20 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.452+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c158f10 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.452+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc52c04d450 con 0x7fc53c0c7070 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.453+0000 7fc523fff640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc52806b810 0x7fc52806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.453+0000 7fc53affd640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc52806b810 0x7fc52806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.453+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc52c0ac6d0 con 0x7fc53c0c7070 2024-02-19T14:57:04.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.453+0000 7fc53affd640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc52806b810 0x7fc52806dcd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc530000f60 tx=0x7fc530007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.457+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc52c0770a0 con 0x7fc53c0c7070 2024-02-19T14:57:04.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.698+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc53c074990 con 0x7fc53c0c7070 2024-02-19T14:57:04.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.698+0000 7fc523fff640 1 -- 172.21.15.78:0/2901061581 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1437 (secure 0 0 0) 0x7fc52c07a760 con 0x7fc53c0c7070 2024-02-19T14:57:04.701 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:04.701 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":328,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":138,"num_osds":8,"num_up_osds":8,"osd_up_since":1708354307,"num_in_osds":8,"osd_in_since":1708354298,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":270602240,"bytes_avail":772823511040,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":117,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"b","status":"up:active","gid":8268}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-02-19T14:56:40.114496+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T14:57:04.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc52806b810 msgr2=0x7fc52806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc52806b810 0x7fc52806dcd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc530000f60 tx=0x7fc530007d30 comp rx=0 tx=0).stop 2024-02-19T14:57:04.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 msgr2=0x7fc53c0ac620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 0x7fc53c0ac620 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0097d0 tx=0x7fc52c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 shutdown_connections 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc52806b810 0x7fc52806dcd0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc53c152480 0x7fc53c14db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc53c0c87a0 0x7fc53c157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 --2- 172.21.15.78:0/2901061581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc53c0c7070 0x7fc53c0ac620 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.700+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 >> 172.21.15.78:0/2901061581 conn(0x7fc53c0ca8a0 msgr2=0x7fc53c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.701+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 shutdown_connections 2024-02-19T14:57:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.701+0000 7fc541d0d640 1 -- 172.21.15.78:0/2901061581 wait complete. 2024-02-19T14:57:04.714 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T14:57:04.714 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:04.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.811+0000 7f8ffc810640 1 Processor -- start 2024-02-19T14:57:04.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.811+0000 7f8ffc810640 1 -- start start 2024-02-19T14:57:04.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.811+0000 7f8ffc810640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ff80c29c0 0x7f8ff81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.811+0000 7f8ffc810640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff8072250 con 0x7f8ff80c25e0 2024-02-19T14:57:04.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.811+0000 7f8ffc810640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff8072480 con 0x7f8ff80c5be0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.811+0000 7f8ffc810640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff8152c90 con 0x7f8ff80c29c0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8ff80c25e0 0x7f8ff80c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45974/0 (socket says 172.21.15.78:45974) 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff659c640 1 -- 172.21.15.78:0/3878592765 learned_addr learned my addr 172.21.15.78:0/3878592765 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8fedd9b640 1 --2- 172.21.15.78:0/3878592765 >> v2:172.21.15.150:3300/0 conn(0x7f8ff80c29c0 0x7f8ff81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693923434 0 0) 0x7f8ff8072250 con 0x7f8ff80c25e0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe4003520 con 0x7f8ff80c25e0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079132363 0 0) 0x7f8ff8072480 con 0x7f8ff80c5be0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8072250 con 0x7f8ff80c5be0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8fedd9b640 1 -- 172.21.15.78:0/3878592765 >> v1:172.21.15.150:6789/0 conn(0x7f8ff80c5be0 legacy=0x7f8ff806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8fedd9b640 1 -- 172.21.15.78:0/3878592765 >> v1:172.21.15.78:6789/0 conn(0x7f8ff80c25e0 legacy=0x7f8ff80c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333753618 0 0) 0x7f8fe4003520 con 0x7f8ff80c25e0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.812+0000 7f8fedd9b640 1 -- 172.21.15.78:0/3878592765 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff80c4b90 con 0x7f8ff80c29c0 2024-02-19T14:57:04.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8fedd9b640 1 --2- 172.21.15.78:0/3878592765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c29c0 0x7f8ff81526a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00097a0 tx=0x7f8fe002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe6ecd0abef7bd4d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8001670 con 0x7f8ff80c29c0 2024-02-19T14:57:04.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe002e050 con 0x7f8ff80c29c0 2024-02-19T14:57:04.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ff5d9b640 1 -- 172.21.15.78:0/3878592765 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0030730 con 0x7f8ff80c29c0 2024-02-19T14:57:04.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ffc810640 1 -- 172.21.15.78:0/3878592765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c29c0 msgr2=0x7f8ff81526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ffc810640 1 --2- 172.21.15.78:0/3878592765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c29c0 0x7f8ff81526a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00097a0 tx=0x7f8fe002b790 comp rx=0 tx=0).stop 2024-02-19T14:57:04.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ffc810640 1 -- 172.21.15.78:0/3878592765 shutdown_connections 2024-02-19T14:57:04.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ffc810640 1 --2- 172.21.15.78:0/3878592765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c29c0 0x7f8ff81526a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.813+0000 7f8ffc810640 1 -- 172.21.15.78:0/3878592765 >> 172.21.15.78:0/3878592765 conn(0x7f8ff806b270 msgr2=0x7f8ff806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.814+0000 7f8ffc810640 1 -- 172.21.15.78:0/3878592765 shutdown_connections 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.814+0000 7f8ffc810640 1 -- 172.21.15.78:0/3878592765 wait complete. 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.815+0000 7f8ffc810640 1 Processor -- start 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.815+0000 7f8ffc810640 1 -- start start 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c25e0 0x7f8ff81e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 0x7f8ff806d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff80c5be0 0x7f8ff80a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff81e8f40 con 0x7f8ff80c29c0 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80bbdb0 con 0x7f8ff80c25e0 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80a7950 con 0x7f8ff80c5be0 2024-02-19T14:57:04.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 0x7f8ff806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 0x7f8ff806d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41620/0 (socket says 172.21.15.78:41620) 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 -- 172.21.15.78:0/1228172826 learned_addr learned my addr 172.21.15.78:0/1228172826 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ff659c640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c25e0 0x7f8ff81e8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ff6d9d640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff80c5be0 0x7f8ff80a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 -- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff80c5be0 msgr2=0x7f8ff80a6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff80c5be0 0x7f8ff80a6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 -- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c25e0 msgr2=0x7f8ff81e8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c25e0 0x7f8ff81e8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 -- 172.21.15.78:0/1228172826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe0009400 con 0x7f8ff80c29c0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fedd9b640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 0x7f8ff806d580 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00098c0 tx=0x7f8fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe003b810 con 0x7f8ff80c29c0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe003b970 con 0x7f8ff80c29c0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff80bd4f0 con 0x7f8ff80c29c0 2024-02-19T14:57:04.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe003bd20 con 0x7f8ff80c29c0 2024-02-19T14:57:04.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.816+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff80bc2e0 con 0x7f8ff80c29c0 2024-02-19T14:57:04.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.817+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff80c6340 con 0x7f8ff80c29c0 2024-02-19T14:57:04.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.818+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8fe0039040 con 0x7f8ff80c29c0 2024-02-19T14:57:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.818+0000 7f8fef7fe640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fd806b810 0x7f8fd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.818+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fe00ab990 con 0x7f8ff80c29c0 2024-02-19T14:57:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.818+0000 7f8ff659c640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fd806b810 0x7f8fd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:04.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.819+0000 7f8ff659c640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fd806b810 0x7f8fd806dcd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000fd0 tx=0x7f8fe8001690 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:04.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.822+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8fe00762e0 con 0x7f8ff80c29c0 2024-02-19T14:57:04.931 INFO:tasks.ceph.mds.b.smithi150.stderr:2024-02-19T14:57:04.928+0000 7f4587e4e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.990+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ff8072a00 con 0x7f8fd806b810 2024-02-19T14:57:05.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:04.997+0000 7f8fef7fe640 1 -- 172.21.15.78:0/1228172826 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290096 (secure 0 0 0) 0x7f8fe800f940 con 0x7f8fd806b810 2024-02-19T14:57:05.003 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:05.004 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fd806b810 msgr2=0x7f8fd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fd806b810 0x7f8fd806dcd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000fd0 tx=0x7f8fe8001690 comp rx=0 tx=0).stop 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 msgr2=0x7f8ff806d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 0x7f8ff806d580 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00098c0 tx=0x7f8fe00046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 shutdown_connections 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fd806b810 0x7f8fd806dcd0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff80c29c0 0x7f8ff806d580 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff80c5be0 0x7f8ff80a6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 --2- 172.21.15.78:0/1228172826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff80c25e0 0x7f8ff81e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 >> 172.21.15.78:0/1228172826 conn(0x7f8ff806b270 msgr2=0x7f8ff8155050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:05.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 shutdown_connections 2024-02-19T14:57:05.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:05.000+0000 7f8ffc810640 1 -- 172.21.15.78:0/1228172826 wait complete. 2024-02-19T14:57:05.014 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-02-19T14:57:04.118909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264260,"kb_used_data":14760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710460,"statfs":{"total":773094113280,"available":772823511040,"internally_reserved":0,"allocated":15114240,"data_stored":7911404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001168"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669478+0000","last_change":"2024-02-19T14:56:59.677644+0000","last_active":"2024-02-19T14:57:00.669478+0000","last_peered":"2024-02-19T14:57:00.669478+0000","last_clean":"2024-02-19T14:57:00.669478+0000","last_became_active":"2024-02-19T14:56:59.677373+0000","last_became_peered":"2024-02-19T14:56:59.677373+0000","last_unstale":"2024-02-19T14:57:00.669478+0000","last_undegraded":"2024-02-19T14:57:00.669478+0000","last_fullsized":"2024-02-19T14:57:00.669478+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:09.037633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669486+0000","last_change":"2024-02-19T14:56:59.672957+0000","last_active":"2024-02-19T14:57:00.669486+0000","last_peered":"2024-02-19T14:57:00.669486+0000","last_clean":"2024-02-19T14:57:00.669486+0000","last_became_active":"2024-02-19T14:56:59.672677+0000","last_became_peered":"2024-02-19T14:56:59.672677+0000","last_unstale":"2024-02-19T14:57:00.669486+0000","last_undegraded":"2024-02-19T14:57:00.669486+0000","last_fullsized":"2024-02-19T14:57:00.669486+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:21.790642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669448+0000","last_change":"2024-02-19T14:56:59.670902+0000","last_active":"2024-02-19T14:57:00.669448+0000","last_peered":"2024-02-19T14:57:00.669448+0000","last_clean":"2024-02-19T14:57:00.669448+0000","last_became_active":"2024-02-19T14:56:59.670608+0000","last_became_peered":"2024-02-19T14:56:59.670608+0000","last_unstale":"2024-02-19T14:57:00.669448+0000","last_undegraded":"2024-02-19T14:57:00.669448+0000","last_fullsized":"2024-02-19T14:57:00.669448+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:05.105129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760676+0000","last_change":"2024-02-19T14:56:59.674985+0000","last_active":"2024-02-19T14:57:00.760676+0000","last_peered":"2024-02-19T14:57:00.760676+0000","last_clean":"2024-02-19T14:57:00.760676+0000","last_became_active":"2024-02-19T14:56:59.674722+0000","last_became_peered":"2024-02-19T14:56:59.674722+0000","last_unstale":"2024-02-19T14:57:00.760676+0000","last_undegraded":"2024-02-19T14:57:00.760676+0000","last_fullsized":"2024-02-19T14:57:00.760676+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.220112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761659+0000","last_change":"2024-02-19T14:56:59.678611+0000","last_active":"2024-02-19T14:57:00.761659+0000","last_peered":"2024-02-19T14:57:00.761659+0000","last_clean":"2024-02-19T14:57:00.761659+0000","last_became_active":"2024-02-19T14:56:59.678350+0000","last_became_peered":"2024-02-19T14:56:59.678350+0000","last_unstale":"2024-02-19T14:57:00.761659+0000","last_undegraded":"2024-02-19T14:57:00.761659+0000","last_fullsized":"2024-02-19T14:57:00.761659+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:13.843441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760880+0000","last_change":"2024-02-19T14:56:59.679879+0000","last_active":"2024-02-19T14:57:00.760880+0000","last_peered":"2024-02-19T14:57:00.760880+0000","last_clean":"2024-02-19T14:57:00.760880+0000","last_became_active":"2024-02-19T14:56:59.678532+0000","last_became_peered":"2024-02-19T14:56:59.678532+0000","last_unstale":"2024-02-19T14:57:00.760880+0000","last_undegraded":"2024-02-19T14:57:00.760880+0000","last_fullsized":"2024-02-19T14:57:00.760880+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:43.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667796+0000","last_change":"2024-02-19T14:56:59.671847+0000","last_active":"2024-02-19T14:57:00.667796+0000","last_peered":"2024-02-19T14:57:00.667796+0000","last_clean":"2024-02-19T14:57:00.667796+0000","last_became_active":"2024-02-19T14:56:59.671586+0000","last_became_peered":"2024-02-19T14:56:59.671586+0000","last_unstale":"2024-02-19T14:57:00.667796+0000","last_undegraded":"2024-02-19T14:57:00.667796+0000","last_fullsized":"2024-02-19T14:57:00.667796+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667989+0000","last_change":"2024-02-19T14:56:59.676911+0000","last_active":"2024-02-19T14:57:00.667989+0000","last_peered":"2024-02-19T14:57:00.667989+0000","last_clean":"2024-02-19T14:57:00.667989+0000","last_became_active":"2024-02-19T14:56:59.676642+0000","last_became_peered":"2024-02-19T14:56:59.676642+0000","last_unstale":"2024-02-19T14:57:00.667989+0000","last_undegraded":"2024-02-19T14:57:00.667989+0000","last_fullsized":"2024-02-19T14:57:00.667989+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:41.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761313+0000","last_change":"2024-02-19T14:56:59.669174+0000","last_active":"2024-02-19T14:57:00.761313+0000","last_peered":"2024-02-19T14:57:00.761313+0000","last_clean":"2024-02-19T14:57:00.761313+0000","last_became_active":"2024-02-19T14:56:59.668890+0000","last_became_peered":"2024-02-19T14:56:59.668890+0000","last_unstale":"2024-02-19T14:57:00.761313+0000","last_undegraded":"2024-02-19T14:57:00.761313+0000","last_fullsized":"2024-02-19T14:57:00.761313+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:39.962805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760970+0000","last_change":"2024-02-19T14:56:59.672943+0000","last_active":"2024-02-19T14:57:00.760970+0000","last_peered":"2024-02-19T14:57:00.760970+0000","last_clean":"2024-02-19T14:57:00.760970+0000","last_became_active":"2024-02-19T14:56:59.672668+0000","last_became_peered":"2024-02-19T14:56:59.672668+0000","last_unstale":"2024-02-19T14:57:00.760970+0000","last_undegraded":"2024-02-19T14:57:00.760970+0000","last_fullsized":"2024-02-19T14:57:00.760970+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:50.415366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761214+0000","last_change":"2024-02-19T14:56:59.669750+0000","last_active":"2024-02-19T14:57:00.761214+0000","last_peered":"2024-02-19T14:57:00.761214+0000","last_clean":"2024-02-19T14:57:00.761214+0000","last_became_active":"2024-02-19T14:56:59.669490+0000","last_became_peered":"2024-02-19T14:56:59.669490+0000","last_unstale":"2024-02-19T14:57:00.761214+0000","last_undegraded":"2024-02-19T14:57:00.761214+0000","last_fullsized":"2024-02-19T14:57:00.761214+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:27.557961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760802+0000","last_change":"2024-02-19T14:56:59.675636+0000","last_active":"2024-02-19T14:57:00.760802+0000","last_peered":"2024-02-19T14:57:00.760802+0000","last_clean":"2024-02-19T14:57:00.760802+0000","last_became_active":"2024-02-19T14:56:59.674106+0000","last_became_peered":"2024-02-19T14:56:59.674106+0000","last_unstale":"2024-02-19T14:57:00.760802+0000","last_undegraded":"2024-02-19T14:57:00.760802+0000","last_fullsized":"2024-02-19T14:57:00.760802+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:49.712701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761587+0000","last_change":"2024-02-19T14:56:59.671801+0000","last_active":"2024-02-19T14:57:00.761587+0000","last_peered":"2024-02-19T14:57:00.761587+0000","last_clean":"2024-02-19T14:57:00.761587+0000","last_became_active":"2024-02-19T14:56:59.671332+0000","last_became_peered":"2024-02-19T14:56:59.671332+0000","last_unstale":"2024-02-19T14:57:00.761587+0000","last_undegraded":"2024-02-19T14:57:00.761587+0000","last_fullsized":"2024-02-19T14:57:00.761587+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:39.015073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760548+0000","last_change":"2024-02-19T14:56:59.681973+0000","last_active":"2024-02-19T14:57:00.760548+0000","last_peered":"2024-02-19T14:57:00.760548+0000","last_clean":"2024-02-19T14:57:00.760548+0000","last_became_active":"2024-02-19T14:56:59.681702+0000","last_became_peered":"2024-02-19T14:56:59.681702+0000","last_unstale":"2024-02-19T14:57:00.760548+0000","last_undegraded":"2024-02-19T14:57:00.760548+0000","last_fullsized":"2024-02-19T14:57:00.760548+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:03:48.192555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669252+0000","last_change":"2024-02-19T14:56:59.677121+0000","last_active":"2024-02-19T14:57:00.669252+0000","last_peered":"2024-02-19T14:57:00.669252+0000","last_clean":"2024-02-19T14:57:00.669252+0000","last_became_active":"2024-02-19T14:56:59.676541+0000","last_became_peered":"2024-02-19T14:56:59.676541+0000","last_unstale":"2024-02-19T14:57:00.669252+0000","last_undegraded":"2024-02-19T14:57:00.669252+0000","last_fullsized":"2024-02-19T14:57:00.669252+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760928+0000","last_change":"2024-02-19T14:56:59.670144+0000","last_active":"2024-02-19T14:57:00.760928+0000","last_peered":"2024-02-19T14:57:00.760928+0000","last_clean":"2024-02-19T14:57:00.760928+0000","last_became_active":"2024-02-19T14:56:59.669884+0000","last_became_peered":"2024-02-19T14:56:59.669884+0000","last_unstale":"2024-02-19T14:57:00.760928+0000","last_undegraded":"2024-02-19T14:57:00.760928+0000","last_fullsized":"2024-02-19T14:57:00.760928+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:11.367956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761230+0000","last_change":"2024-02-19T14:56:59.682420+0000","last_active":"2024-02-19T14:57:00.761230+0000","last_peered":"2024-02-19T14:57:00.761230+0000","last_clean":"2024-02-19T14:57:00.761230+0000","last_became_active":"2024-02-19T14:56:59.682141+0000","last_became_peered":"2024-02-19T14:56:59.682141+0000","last_unstale":"2024-02-19T14:57:00.761230+0000","last_undegraded":"2024-02-19T14:57:00.761230+0000","last_fullsized":"2024-02-19T14:57:00.761230+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760755+0000","last_change":"2024-02-19T14:56:59.681178+0000","last_active":"2024-02-19T14:57:00.760755+0000","last_peered":"2024-02-19T14:57:00.760755+0000","last_clean":"2024-02-19T14:57:00.760755+0000","last_became_active":"2024-02-19T14:56:59.680802+0000","last_became_peered":"2024-02-19T14:56:59.680802+0000","last_unstale":"2024-02-19T14:57:00.760755+0000","last_undegraded":"2024-02-19T14:57:00.760755+0000","last_fullsized":"2024-02-19T14:57:00.760755+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:32.096559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761012+0000","last_change":"2024-02-19T14:56:59.683504+0000","last_active":"2024-02-19T14:57:00.761012+0000","last_peered":"2024-02-19T14:57:00.761012+0000","last_clean":"2024-02-19T14:57:00.761012+0000","last_became_active":"2024-02-19T14:56:59.683207+0000","last_became_peered":"2024-02-19T14:56:59.683207+0000","last_unstale":"2024-02-19T14:57:00.761012+0000","last_undegraded":"2024-02-19T14:57:00.761012+0000","last_fullsized":"2024-02-19T14:57:00.761012+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:50.415366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760700+0000","last_change":"2024-02-19T14:56:59.671258+0000","last_active":"2024-02-19T14:57:00.760700+0000","last_peered":"2024-02-19T14:57:00.760700+0000","last_clean":"2024-02-19T14:57:00.760700+0000","last_became_active":"2024-02-19T14:56:59.670922+0000","last_became_peered":"2024-02-19T14:56:59.670922+0000","last_unstale":"2024-02-19T14:57:00.760700+0000","last_undegraded":"2024-02-19T14:57:00.760700+0000","last_fullsized":"2024-02-19T14:57:00.760700+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:25.573016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761658+0000","last_change":"2024-02-19T14:56:59.669256+0000","last_active":"2024-02-19T14:57:00.761658+0000","last_peered":"2024-02-19T14:57:00.761658+0000","last_clean":"2024-02-19T14:57:00.761658+0000","last_became_active":"2024-02-19T14:56:59.668975+0000","last_became_peered":"2024-02-19T14:56:59.668975+0000","last_unstale":"2024-02-19T14:57:00.761658+0000","last_undegraded":"2024-02-19T14:57:00.761658+0000","last_fullsized":"2024-02-19T14:57:00.761658+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:06.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668252+0000","last_change":"2024-02-19T14:56:59.672561+0000","last_active":"2024-02-19T14:57:00.668252+0000","last_peered":"2024-02-19T14:57:00.668252+0000","last_clean":"2024-02-19T14:57:00.668252+0000","last_became_active":"2024-02-19T14:56:59.672171+0000","last_became_peered":"2024-02-19T14:56:59.672171+0000","last_unstale":"2024-02-19T14:57:00.668252+0000","last_undegraded":"2024-02-19T14:57:00.668252+0000","last_fullsized":"2024-02-19T14:57:00.668252+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:36.585855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668809+0000","last_change":"2024-02-19T14:56:59.677599+0000","last_active":"2024-02-19T14:57:00.668809+0000","last_peered":"2024-02-19T14:57:00.668809+0000","last_clean":"2024-02-19T14:57:00.668809+0000","last_became_active":"2024-02-19T14:56:59.677306+0000","last_became_peered":"2024-02-19T14:56:59.677306+0000","last_unstale":"2024-02-19T14:57:00.668809+0000","last_undegraded":"2024-02-19T14:57:00.668809+0000","last_fullsized":"2024-02-19T14:57:00.668809+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:56.306598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760380+0000","last_change":"2024-02-19T14:56:59.670448+0000","last_active":"2024-02-19T14:57:00.760380+0000","last_peered":"2024-02-19T14:57:00.760380+0000","last_clean":"2024-02-19T14:57:00.760380+0000","last_became_active":"2024-02-19T14:56:59.670155+0000","last_became_peered":"2024-02-19T14:56:59.670155+0000","last_unstale":"2024-02-19T14:57:00.760380+0000","last_undegraded":"2024-02-19T14:57:00.760380+0000","last_fullsized":"2024-02-19T14:57:00.760380+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:07.818574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669346+0000","last_change":"2024-02-19T14:56:59.678644+0000","last_active":"2024-02-19T14:57:00.669346+0000","last_peered":"2024-02-19T14:57:00.669346+0000","last_clean":"2024-02-19T14:57:00.669346+0000","last_became_active":"2024-02-19T14:56:59.678353+0000","last_became_peered":"2024-02-19T14:56:59.678353+0000","last_unstale":"2024-02-19T14:57:00.669346+0000","last_undegraded":"2024-02-19T14:57:00.669346+0000","last_fullsized":"2024-02-19T14:57:00.669346+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:05.105129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668644+0000","last_change":"2024-02-19T14:56:59.679167+0000","last_active":"2024-02-19T14:57:00.668644+0000","last_peered":"2024-02-19T14:57:00.668644+0000","last_clean":"2024-02-19T14:57:00.668644+0000","last_became_active":"2024-02-19T14:56:59.678635+0000","last_became_peered":"2024-02-19T14:56:59.678635+0000","last_unstale":"2024-02-19T14:57:00.668644+0000","last_undegraded":"2024-02-19T14:57:00.668644+0000","last_fullsized":"2024-02-19T14:57:00.668644+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:34.374030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761176+0000","last_change":"2024-02-19T14:56:59.669279+0000","last_active":"2024-02-19T14:57:00.761176+0000","last_peered":"2024-02-19T14:57:00.761176+0000","last_clean":"2024-02-19T14:57:00.761176+0000","last_became_active":"2024-02-19T14:56:59.668990+0000","last_became_peered":"2024-02-19T14:56:59.668990+0000","last_unstale":"2024-02-19T14:57:00.761176+0000","last_undegraded":"2024-02-19T14:57:00.761176+0000","last_fullsized":"2024-02-19T14:57:00.761176+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:37.163331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760625+0000","last_change":"2024-02-19T14:56:59.674868+0000","last_active":"2024-02-19T14:57:00.760625+0000","last_peered":"2024-02-19T14:57:00.760625+0000","last_clean":"2024-02-19T14:57:00.760625+0000","last_became_active":"2024-02-19T14:56:59.674597+0000","last_became_peered":"2024-02-19T14:56:59.674597+0000","last_unstale":"2024-02-19T14:57:00.760625+0000","last_undegraded":"2024-02-19T14:57:00.760625+0000","last_fullsized":"2024-02-19T14:57:00.760625+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:33:36.449995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668644+0000","last_change":"2024-02-19T14:56:59.675815+0000","last_active":"2024-02-19T14:57:00.668644+0000","last_peered":"2024-02-19T14:57:00.668644+0000","last_clean":"2024-02-19T14:57:00.668644+0000","last_became_active":"2024-02-19T14:56:59.671817+0000","last_became_peered":"2024-02-19T14:56:59.671817+0000","last_unstale":"2024-02-19T14:57:00.668644+0000","last_undegraded":"2024-02-19T14:57:00.668644+0000","last_fullsized":"2024-02-19T14:57:00.668644+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:24.209883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760503+0000","last_change":"2024-02-19T14:56:59.675553+0000","last_active":"2024-02-19T14:57:00.760503+0000","last_peered":"2024-02-19T14:57:00.760503+0000","last_clean":"2024-02-19T14:57:00.760503+0000","last_became_active":"2024-02-19T14:56:59.675281+0000","last_became_peered":"2024-02-19T14:56:59.675281+0000","last_unstale":"2024-02-19T14:57:00.760503+0000","last_undegraded":"2024-02-19T14:57:00.760503+0000","last_fullsized":"2024-02-19T14:57:00.760503+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:36:18.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760888+0000","last_change":"2024-02-19T14:56:59.669519+0000","last_active":"2024-02-19T14:57:00.760888+0000","last_peered":"2024-02-19T14:57:00.760888+0000","last_clean":"2024-02-19T14:57:00.760888+0000","last_became_active":"2024-02-19T14:56:59.667272+0000","last_became_peered":"2024-02-19T14:56:59.667272+0000","last_unstale":"2024-02-19T14:57:00.760888+0000","last_undegraded":"2024-02-19T14:57:00.760888+0000","last_fullsized":"2024-02-19T14:57:00.760888+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:40.323947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668136+0000","last_change":"2024-02-19T14:56:59.676162+0000","last_active":"2024-02-19T14:57:00.668136+0000","last_peered":"2024-02-19T14:57:00.668136+0000","last_clean":"2024-02-19T14:57:00.668136+0000","last_became_active":"2024-02-19T14:56:59.675867+0000","last_became_peered":"2024-02-19T14:56:59.675867+0000","last_unstale":"2024-02-19T14:57:00.668136+0000","last_undegraded":"2024-02-19T14:57:00.668136+0000","last_fullsized":"2024-02-19T14:57:00.668136+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:27.213645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669432+0000","last_change":"2024-02-19T14:56:59.665767+0000","last_active":"2024-02-19T14:57:00.669432+0000","last_peered":"2024-02-19T14:57:00.669432+0000","last_clean":"2024-02-19T14:57:00.669432+0000","last_became_active":"2024-02-19T14:56:59.665503+0000","last_became_peered":"2024-02-19T14:56:59.665503+0000","last_unstale":"2024-02-19T14:57:00.669432+0000","last_undegraded":"2024-02-19T14:57:00.669432+0000","last_fullsized":"2024-02-19T14:57:00.669432+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:50.909922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"137'4","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761521+0000","last_change":"2024-02-19T14:56:58.666557+0000","last_active":"2024-02-19T14:57:00.761521+0000","last_peered":"2024-02-19T14:57:00.761521+0000","last_clean":"2024-02-19T14:57:00.761521+0000","last_became_active":"2024-02-19T14:56:58.666286+0000","last_became_peered":"2024-02-19T14:56:58.666286+0000","last_unstale":"2024-02-19T14:57:00.761521+0000","last_undegraded":"2024-02-19T14:57:00.761521+0000","last_fullsized":"2024-02-19T14:57:00.761521+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:49.904842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669498+0000","last_change":"2024-02-19T14:56:59.674314+0000","last_active":"2024-02-19T14:57:00.669498+0000","last_peered":"2024-02-19T14:57:00.669498+0000","last_clean":"2024-02-19T14:57:00.669498+0000","last_became_active":"2024-02-19T14:56:59.673451+0000","last_became_peered":"2024-02-19T14:56:59.673451+0000","last_unstale":"2024-02-19T14:57:00.669498+0000","last_undegraded":"2024-02-19T14:57:00.669498+0000","last_fullsized":"2024-02-19T14:57:00.669498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:09.037633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760607+0000","last_change":"2024-02-19T14:56:58.672364+0000","last_active":"2024-02-19T14:57:00.760607+0000","last_peered":"2024-02-19T14:57:00.760607+0000","last_clean":"2024-02-19T14:57:00.760607+0000","last_became_active":"2024-02-19T14:56:58.672089+0000","last_became_peered":"2024-02-19T14:56:58.672089+0000","last_unstale":"2024-02-19T14:57:00.760607+0000","last_undegraded":"2024-02-19T14:57:00.760607+0000","last_fullsized":"2024-02-19T14:57:00.760607+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:04.100049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668467+0000","last_change":"2024-02-19T14:56:59.670090+0000","last_active":"2024-02-19T14:57:00.668467+0000","last_peered":"2024-02-19T14:57:00.668467+0000","last_clean":"2024-02-19T14:57:00.668467+0000","last_became_active":"2024-02-19T14:56:59.669839+0000","last_became_peered":"2024-02-19T14:56:59.669839+0000","last_unstale":"2024-02-19T14:57:00.668467+0000","last_undegraded":"2024-02-19T14:57:00.668467+0000","last_fullsized":"2024-02-19T14:57:00.668467+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:46.789869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761059+0000","last_change":"2024-02-19T14:56:58.675313+0000","last_active":"2024-02-19T14:57:00.761059+0000","last_peered":"2024-02-19T14:57:00.761059+0000","last_clean":"2024-02-19T14:57:00.761059+0000","last_became_active":"2024-02-19T14:56:58.675047+0000","last_became_peered":"2024-02-19T14:56:58.675047+0000","last_unstale":"2024-02-19T14:57:00.761059+0000","last_undegraded":"2024-02-19T14:57:00.761059+0000","last_fullsized":"2024-02-19T14:57:00.761059+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:52:41.628070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760957+0000","last_change":"2024-02-19T14:56:59.679353+0000","last_active":"2024-02-19T14:57:00.760957+0000","last_peered":"2024-02-19T14:57:00.760957+0000","last_clean":"2024-02-19T14:57:00.760957+0000","last_became_active":"2024-02-19T14:56:59.679070+0000","last_became_peered":"2024-02-19T14:56:59.679070+0000","last_unstale":"2024-02-19T14:57:00.760957+0000","last_undegraded":"2024-02-19T14:57:00.760957+0000","last_fullsized":"2024-02-19T14:57:00.760957+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:11.367956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669346+0000","last_change":"2024-02-19T14:56:58.668754+0000","last_active":"2024-02-19T14:57:00.669346+0000","last_peered":"2024-02-19T14:57:00.669346+0000","last_clean":"2024-02-19T14:57:00.669346+0000","last_became_active":"2024-02-19T14:56:58.668195+0000","last_became_peered":"2024-02-19T14:56:58.668195+0000","last_unstale":"2024-02-19T14:57:00.669346+0000","last_undegraded":"2024-02-19T14:57:00.669346+0000","last_fullsized":"2024-02-19T14:57:00.669346+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760423+0000","last_change":"2024-02-19T14:56:59.674433+0000","last_active":"2024-02-19T14:57:00.760423+0000","last_peered":"2024-02-19T14:57:00.760423+0000","last_clean":"2024-02-19T14:57:00.760423+0000","last_became_active":"2024-02-19T14:56:59.674111+0000","last_became_peered":"2024-02-19T14:56:59.674111+0000","last_unstale":"2024-02-19T14:57:00.760423+0000","last_undegraded":"2024-02-19T14:57:00.760423+0000","last_fullsized":"2024-02-19T14:57:00.760423+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:39.579884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668697+0000","last_change":"2024-02-19T14:56:58.672280+0000","last_active":"2024-02-19T14:57:00.668697+0000","last_peered":"2024-02-19T14:57:00.668697+0000","last_clean":"2024-02-19T14:57:00.668697+0000","last_became_active":"2024-02-19T14:56:58.671978+0000","last_became_peered":"2024-02-19T14:56:58.671978+0000","last_unstale":"2024-02-19T14:57:00.668697+0000","last_undegraded":"2024-02-19T14:57:00.668697+0000","last_fullsized":"2024-02-19T14:57:00.668697+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:38.026526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760682+0000","last_change":"2024-02-19T14:56:59.674348+0000","last_active":"2024-02-19T14:57:00.760682+0000","last_peered":"2024-02-19T14:57:00.760682+0000","last_clean":"2024-02-19T14:57:00.760682+0000","last_became_active":"2024-02-19T14:56:59.674060+0000","last_became_peered":"2024-02-19T14:56:59.674060+0000","last_unstale":"2024-02-19T14:57:00.760682+0000","last_undegraded":"2024-02-19T14:57:00.760682+0000","last_fullsized":"2024-02-19T14:57:00.760682+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:40.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761105+0000","last_change":"2024-02-19T14:56:58.673135+0000","last_active":"2024-02-19T14:57:00.761105+0000","last_peered":"2024-02-19T14:57:00.761105+0000","last_clean":"2024-02-19T14:57:00.761105+0000","last_became_active":"2024-02-19T14:56:58.672822+0000","last_became_peered":"2024-02-19T14:56:58.672822+0000","last_unstale":"2024-02-19T14:57:00.761105+0000","last_undegraded":"2024-02-19T14:57:00.761105+0000","last_fullsized":"2024-02-19T14:57:00.761105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:08.349505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669233+0000","last_change":"2024-02-19T14:56:59.673661+0000","last_active":"2024-02-19T14:57:00.669233+0000","last_peered":"2024-02-19T14:57:00.669233+0000","last_clean":"2024-02-19T14:57:00.669233+0000","last_became_active":"2024-02-19T14:56:59.672005+0000","last_became_peered":"2024-02-19T14:56:59.672005+0000","last_unstale":"2024-02-19T14:57:00.669233+0000","last_undegraded":"2024-02-19T14:57:00.669233+0000","last_fullsized":"2024-02-19T14:57:00.669233+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:53.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667726+0000","last_change":"2024-02-19T14:56:58.666464+0000","last_active":"2024-02-19T14:57:00.667726+0000","last_peered":"2024-02-19T14:57:00.667726+0000","last_clean":"2024-02-19T14:57:00.667726+0000","last_became_active":"2024-02-19T14:56:58.665994+0000","last_became_peered":"2024-02-19T14:56:58.665994+0000","last_unstale":"2024-02-19T14:57:00.667726+0000","last_undegraded":"2024-02-19T14:57:00.667726+0000","last_fullsized":"2024-02-19T14:57:00.667726+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:45.784789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668891+0000","last_change":"2024-02-19T14:56:59.676047+0000","last_active":"2024-02-19T14:57:00.668891+0000","last_peered":"2024-02-19T14:57:00.668891+0000","last_clean":"2024-02-19T14:57:00.668891+0000","last_became_active":"2024-02-19T14:56:59.675731+0000","last_became_peered":"2024-02-19T14:56:59.675731+0000","last_unstale":"2024-02-19T14:57:00.668891+0000","last_undegraded":"2024-02-19T14:57:00.668891+0000","last_fullsized":"2024-02-19T14:57:00.668891+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:50.909922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761024+0000","last_change":"2024-02-19T14:56:58.663619+0000","last_active":"2024-02-19T14:57:00.761024+0000","last_peered":"2024-02-19T14:57:00.761024+0000","last_clean":"2024-02-19T14:57:00.761024+0000","last_became_active":"2024-02-19T14:56:58.663358+0000","last_became_peered":"2024-02-19T14:56:58.663358+0000","last_unstale":"2024-02-19T14:57:00.761024+0000","last_undegraded":"2024-02-19T14:57:00.761024+0000","last_fullsized":"2024-02-19T14:57:00.761024+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:35.580775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669142+0000","last_change":"2024-02-19T14:56:59.669001+0000","last_active":"2024-02-19T14:57:00.669142+0000","last_peered":"2024-02-19T14:57:00.669142+0000","last_clean":"2024-02-19T14:57:00.669142+0000","last_became_active":"2024-02-19T14:56:59.667676+0000","last_became_peered":"2024-02-19T14:56:59.667676+0000","last_unstale":"2024-02-19T14:57:00.669142+0000","last_undegraded":"2024-02-19T14:57:00.669142+0000","last_fullsized":"2024-02-19T14:57:00.669142+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:38:21.118003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760966+0000","last_change":"2024-02-19T14:56:58.670376+0000","last_active":"2024-02-19T14:57:00.760966+0000","last_peered":"2024-02-19T14:57:00.760966+0000","last_clean":"2024-02-19T14:57:00.760966+0000","last_became_active":"2024-02-19T14:56:58.670065+0000","last_became_peered":"2024-02-19T14:56:58.670065+0000","last_unstale":"2024-02-19T14:57:00.760966+0000","last_undegraded":"2024-02-19T14:57:00.760966+0000","last_fullsized":"2024-02-19T14:57:00.760966+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:08.032552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761353+0000","last_change":"2024-02-19T14:56:59.669101+0000","last_active":"2024-02-19T14:57:00.761353+0000","last_peered":"2024-02-19T14:57:00.761353+0000","last_clean":"2024-02-19T14:57:00.761353+0000","last_became_active":"2024-02-19T14:56:59.668601+0000","last_became_peered":"2024-02-19T14:56:59.668601+0000","last_unstale":"2024-02-19T14:57:00.761353+0000","last_undegraded":"2024-02-19T14:57:00.761353+0000","last_fullsized":"2024-02-19T14:57:00.761353+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:12:40.244630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669055+0000","last_change":"2024-02-19T14:56:58.673499+0000","last_active":"2024-02-19T14:57:00.669055+0000","last_peered":"2024-02-19T14:57:00.669055+0000","last_clean":"2024-02-19T14:57:00.669055+0000","last_became_active":"2024-02-19T14:56:58.673014+0000","last_became_peered":"2024-02-19T14:56:58.673014+0000","last_unstale":"2024-02-19T14:57:00.669055+0000","last_undegraded":"2024-02-19T14:57:00.669055+0000","last_fullsized":"2024-02-19T14:57:00.669055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:55.301518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760881+0000","last_change":"2024-02-19T14:56:59.680952+0000","last_active":"2024-02-19T14:57:00.760881+0000","last_peered":"2024-02-19T14:57:00.760881+0000","last_clean":"2024-02-19T14:57:00.760881+0000","last_became_active":"2024-02-19T14:56:59.680634+0000","last_became_peered":"2024-02-19T14:56:59.680634+0000","last_unstale":"2024-02-19T14:57:00.760881+0000","last_undegraded":"2024-02-19T14:57:00.760881+0000","last_fullsized":"2024-02-19T14:57:00.760881+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:08.161866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"137'3","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761402+0000","last_change":"2024-02-19T14:56:58.664991+0000","last_active":"2024-02-19T14:57:00.761402+0000","last_peered":"2024-02-19T14:57:00.761402+0000","last_clean":"2024-02-19T14:57:00.761402+0000","last_became_active":"2024-02-19T14:56:58.664715+0000","last_became_peered":"2024-02-19T14:56:58.664715+0000","last_unstale":"2024-02-19T14:57:00.761402+0000","last_undegraded":"2024-02-19T14:57:00.761402+0000","last_fullsized":"2024-02-19T14:57:00.761402+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:14.196709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668295+0000","last_change":"2024-02-19T14:56:59.669119+0000","last_active":"2024-02-19T14:57:00.668295+0000","last_peered":"2024-02-19T14:57:00.668295+0000","last_clean":"2024-02-19T14:57:00.668295+0000","last_became_active":"2024-02-19T14:56:59.668836+0000","last_became_peered":"2024-02-19T14:56:59.668836+0000","last_unstale":"2024-02-19T14:57:00.668295+0000","last_undegraded":"2024-02-19T14:57:00.668295+0000","last_fullsized":"2024-02-19T14:57:00.668295+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:14.400279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761709+0000","last_change":"2024-02-19T14:56:58.666985+0000","last_active":"2024-02-19T14:57:00.761709+0000","last_peered":"2024-02-19T14:57:00.761709+0000","last_clean":"2024-02-19T14:57:00.761709+0000","last_became_active":"2024-02-19T14:56:58.666667+0000","last_became_peered":"2024-02-19T14:56:58.666667+0000","last_unstale":"2024-02-19T14:57:00.761709+0000","last_undegraded":"2024-02-19T14:57:00.761709+0000","last_fullsized":"2024-02-19T14:57:00.761709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:35.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669279+0000","last_change":"2024-02-19T14:56:59.677403+0000","last_active":"2024-02-19T14:57:00.669279+0000","last_peered":"2024-02-19T14:57:00.669279+0000","last_clean":"2024-02-19T14:57:00.669279+0000","last_became_active":"2024-02-19T14:56:59.676857+0000","last_became_peered":"2024-02-19T14:56:59.676857+0000","last_unstale":"2024-02-19T14:57:00.669279+0000","last_undegraded":"2024-02-19T14:57:00.669279+0000","last_fullsized":"2024-02-19T14:57:00.669279+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:36.387245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760486+0000","last_change":"2024-02-19T14:56:58.673001+0000","last_active":"2024-02-19T14:57:00.760486+0000","last_peered":"2024-02-19T14:57:00.760486+0000","last_clean":"2024-02-19T14:57:00.760486+0000","last_became_active":"2024-02-19T14:56:58.672751+0000","last_became_peered":"2024-02-19T14:56:58.672751+0000","last_unstale":"2024-02-19T14:57:00.760486+0000","last_undegraded":"2024-02-19T14:57:00.760486+0000","last_fullsized":"2024-02-19T14:57:00.760486+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:13.395199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761015+0000","last_change":"2024-02-19T14:56:59.667686+0000","last_active":"2024-02-19T14:57:00.761015+0000","last_peered":"2024-02-19T14:57:00.761015+0000","last_clean":"2024-02-19T14:57:00.761015+0000","last_became_active":"2024-02-19T14:56:59.667391+0000","last_became_peered":"2024-02-19T14:56:59.667391+0000","last_unstale":"2024-02-19T14:57:00.761015+0000","last_undegraded":"2024-02-19T14:57:00.761015+0000","last_fullsized":"2024-02-19T14:57:00.761015+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:25.913809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760722+0000","last_change":"2024-02-19T14:56:58.670054+0000","last_active":"2024-02-19T14:57:00.760722+0000","last_peered":"2024-02-19T14:57:00.760722+0000","last_clean":"2024-02-19T14:57:00.760722+0000","last_became_active":"2024-02-19T14:56:58.669599+0000","last_became_peered":"2024-02-19T14:56:58.669599+0000","last_unstale":"2024-02-19T14:57:00.760722+0000","last_undegraded":"2024-02-19T14:57:00.760722+0000","last_fullsized":"2024-02-19T14:57:00.760722+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:26.208565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669254+0000","last_change":"2024-02-19T14:56:59.671613+0000","last_active":"2024-02-19T14:57:00.669254+0000","last_peered":"2024-02-19T14:57:00.669254+0000","last_clean":"2024-02-19T14:57:00.669254+0000","last_became_active":"2024-02-19T14:56:59.671340+0000","last_became_peered":"2024-02-19T14:56:59.671340+0000","last_unstale":"2024-02-19T14:57:00.669254+0000","last_undegraded":"2024-02-19T14:57:00.669254+0000","last_fullsized":"2024-02-19T14:57:00.669254+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:34.374030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"137'8","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667544+0000","last_change":"2024-02-19T14:56:58.668292+0000","last_active":"2024-02-19T14:57:00.667544+0000","last_peered":"2024-02-19T14:57:00.667544+0000","last_clean":"2024-02-19T14:57:00.667544+0000","last_became_active":"2024-02-19T14:56:58.668000+0000","last_became_peered":"2024-02-19T14:56:58.668000+0000","last_unstale":"2024-02-19T14:57:00.667544+0000","last_undegraded":"2024-02-19T14:57:00.667544+0000","last_fullsized":"2024-02-19T14:57:00.667544+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760620+0000","last_change":"2024-02-19T14:56:59.674229+0000","last_active":"2024-02-19T14:57:00.760620+0000","last_peered":"2024-02-19T14:57:00.760620+0000","last_clean":"2024-02-19T14:57:00.760620+0000","last_became_active":"2024-02-19T14:56:59.673948+0000","last_became_peered":"2024-02-19T14:56:59.673948+0000","last_unstale":"2024-02-19T14:57:00.760620+0000","last_undegraded":"2024-02-19T14:57:00.760620+0000","last_fullsized":"2024-02-19T14:57:00.760620+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:43.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760686+0000","last_change":"2024-02-19T14:56:58.668198+0000","last_active":"2024-02-19T14:57:00.760686+0000","last_peered":"2024-02-19T14:57:00.760686+0000","last_clean":"2024-02-19T14:57:00.760686+0000","last_became_active":"2024-02-19T14:56:58.667965+0000","last_became_peered":"2024-02-19T14:56:58.667965+0000","last_unstale":"2024-02-19T14:57:00.760686+0000","last_undegraded":"2024-02-19T14:57:00.760686+0000","last_fullsized":"2024-02-19T14:57:00.760686+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:28.054550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761080+0000","last_change":"2024-02-19T14:56:59.684025+0000","last_active":"2024-02-19T14:57:00.761080+0000","last_peered":"2024-02-19T14:57:00.761080+0000","last_clean":"2024-02-19T14:57:00.761080+0000","last_became_active":"2024-02-19T14:56:59.683791+0000","last_became_peered":"2024-02-19T14:56:59.683791+0000","last_unstale":"2024-02-19T14:57:00.761080+0000","last_undegraded":"2024-02-19T14:57:00.761080+0000","last_fullsized":"2024-02-19T14:57:00.761080+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:25.913809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760700+0000","last_change":"2024-02-19T14:56:58.669910+0000","last_active":"2024-02-19T14:57:00.760700+0000","last_peered":"2024-02-19T14:57:00.760700+0000","last_clean":"2024-02-19T14:57:00.760700+0000","last_became_active":"2024-02-19T14:56:58.669594+0000","last_became_peered":"2024-02-19T14:56:58.669594+0000","last_unstale":"2024-02-19T14:57:00.760700+0000","last_undegraded":"2024-02-19T14:57:00.760700+0000","last_fullsized":"2024-02-19T14:57:00.760700+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:20.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668860+0000","last_change":"2024-02-19T14:56:59.679318+0000","last_active":"2024-02-19T14:57:00.668860+0000","last_peered":"2024-02-19T14:57:00.668860+0000","last_clean":"2024-02-19T14:57:00.668860+0000","last_became_active":"2024-02-19T14:56:59.678816+0000","last_became_peered":"2024-02-19T14:56:59.678816+0000","last_unstale":"2024-02-19T14:57:00.668860+0000","last_undegraded":"2024-02-19T14:57:00.668860+0000","last_fullsized":"2024-02-19T14:57:00.668860+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:36.387245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669503+0000","last_change":"2024-02-19T14:56:58.669660+0000","last_active":"2024-02-19T14:57:00.669503+0000","last_peered":"2024-02-19T14:57:00.669503+0000","last_clean":"2024-02-19T14:57:00.669503+0000","last_became_active":"2024-02-19T14:56:58.669395+0000","last_became_peered":"2024-02-19T14:56:58.669395+0000","last_unstale":"2024-02-19T14:57:00.669503+0000","last_undegraded":"2024-02-19T14:57:00.669503+0000","last_fullsized":"2024-02-19T14:57:00.669503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:45.784789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669458+0000","last_change":"2024-02-19T14:56:59.676902+0000","last_active":"2024-02-19T14:57:00.669458+0000","last_peered":"2024-02-19T14:57:00.669458+0000","last_clean":"2024-02-19T14:57:00.669458+0000","last_became_active":"2024-02-19T14:56:59.676621+0000","last_became_peered":"2024-02-19T14:56:59.676621+0000","last_unstale":"2024-02-19T14:57:00.669458+0000","last_undegraded":"2024-02-19T14:57:00.669458+0000","last_fullsized":"2024-02-19T14:57:00.669458+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:21.790642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760848+0000","last_change":"2024-02-19T14:56:58.673331+0000","last_active":"2024-02-19T14:57:00.760848+0000","last_peered":"2024-02-19T14:57:00.760848+0000","last_clean":"2024-02-19T14:57:00.760848+0000","last_became_active":"2024-02-19T14:56:58.673067+0000","last_became_peered":"2024-02-19T14:56:58.673067+0000","last_unstale":"2024-02-19T14:57:00.760848+0000","last_undegraded":"2024-02-19T14:57:00.760848+0000","last_fullsized":"2024-02-19T14:57:00.760848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760722+0000","last_change":"2024-02-19T14:56:59.684112+0000","last_active":"2024-02-19T14:57:00.760722+0000","last_peered":"2024-02-19T14:57:00.760722+0000","last_clean":"2024-02-19T14:57:00.760722+0000","last_became_active":"2024-02-19T14:56:59.683866+0000","last_became_peered":"2024-02-19T14:56:59.683866+0000","last_unstale":"2024-02-19T14:57:00.760722+0000","last_undegraded":"2024-02-19T14:57:00.760722+0000","last_fullsized":"2024-02-19T14:57:00.760722+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:40.323947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760568+0000","last_change":"2024-02-19T14:56:58.672326+0000","last_active":"2024-02-19T14:57:00.760568+0000","last_peered":"2024-02-19T14:57:00.760568+0000","last_clean":"2024-02-19T14:57:00.760568+0000","last_became_active":"2024-02-19T14:56:58.672056+0000","last_became_peered":"2024-02-19T14:56:58.672056+0000","last_unstale":"2024-02-19T14:57:00.760568+0000","last_undegraded":"2024-02-19T14:57:00.760568+0000","last_fullsized":"2024-02-19T14:57:00.760568+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:04.357396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669386+0000","last_change":"2024-02-19T14:56:59.679132+0000","last_active":"2024-02-19T14:57:00.669386+0000","last_peered":"2024-02-19T14:57:00.669386+0000","last_clean":"2024-02-19T14:57:00.669386+0000","last_became_active":"2024-02-19T14:56:59.678819+0000","last_became_peered":"2024-02-19T14:56:59.678819+0000","last_unstale":"2024-02-19T14:57:00.669386+0000","last_undegraded":"2024-02-19T14:57:00.669386+0000","last_fullsized":"2024-02-19T14:57:00.669386+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:09.354585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668134+0000","last_change":"2024-02-19T14:56:58.664244+0000","last_active":"2024-02-19T14:57:00.668134+0000","last_peered":"2024-02-19T14:57:00.668134+0000","last_clean":"2024-02-19T14:57:00.668134+0000","last_became_active":"2024-02-19T14:56:58.663902+0000","last_became_peered":"2024-02-19T14:56:58.663902+0000","last_unstale":"2024-02-19T14:57:00.668134+0000","last_undegraded":"2024-02-19T14:57:00.668134+0000","last_fullsized":"2024-02-19T14:57:00.668134+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:10.863066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760504+0000","last_change":"2024-02-19T14:56:59.677739+0000","last_active":"2024-02-19T14:57:00.760504+0000","last_peered":"2024-02-19T14:57:00.760504+0000","last_clean":"2024-02-19T14:57:00.760504+0000","last_became_active":"2024-02-19T14:56:59.677418+0000","last_became_peered":"2024-02-19T14:56:59.677418+0000","last_unstale":"2024-02-19T14:57:00.760504+0000","last_undegraded":"2024-02-19T14:57:00.760504+0000","last_fullsized":"2024-02-19T14:57:00.760504+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:13:48.895089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760318+0000","last_change":"2024-02-19T14:56:58.672372+0000","last_active":"2024-02-19T14:57:00.760318+0000","last_peered":"2024-02-19T14:57:00.760318+0000","last_clean":"2024-02-19T14:57:00.760318+0000","last_became_active":"2024-02-19T14:56:58.672074+0000","last_became_peered":"2024-02-19T14:56:58.672074+0000","last_unstale":"2024-02-19T14:57:00.760318+0000","last_undegraded":"2024-02-19T14:57:00.760318+0000","last_fullsized":"2024-02-19T14:57:00.760318+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:20.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669638+0000","last_change":"2024-02-19T14:56:59.676678+0000","last_active":"2024-02-19T14:57:00.669638+0000","last_peered":"2024-02-19T14:57:00.669638+0000","last_clean":"2024-02-19T14:57:00.669638+0000","last_became_active":"2024-02-19T14:56:59.676424+0000","last_became_peered":"2024-02-19T14:56:59.676424+0000","last_unstale":"2024-02-19T14:57:00.669638+0000","last_undegraded":"2024-02-19T14:57:00.669638+0000","last_fullsized":"2024-02-19T14:57:00.669638+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:05.362476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668478+0000","last_change":"2024-02-19T14:56:58.669077+0000","last_active":"2024-02-19T14:57:00.668478+0000","last_peered":"2024-02-19T14:57:00.668478+0000","last_clean":"2024-02-19T14:57:00.668478+0000","last_became_active":"2024-02-19T14:56:58.668349+0000","last_became_peered":"2024-02-19T14:56:58.668349+0000","last_unstale":"2024-02-19T14:57:00.668478+0000","last_undegraded":"2024-02-19T14:57:00.668478+0000","last_fullsized":"2024-02-19T14:57:00.668478+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:03:38.417380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761364+0000","last_change":"2024-02-19T14:56:59.679793+0000","last_active":"2024-02-19T14:57:00.761364+0000","last_peered":"2024-02-19T14:57:00.761364+0000","last_clean":"2024-02-19T14:57:00.761364+0000","last_became_active":"2024-02-19T14:56:59.677982+0000","last_became_peered":"2024-02-19T14:56:59.677982+0000","last_unstale":"2024-02-19T14:57:00.761364+0000","last_undegraded":"2024-02-19T14:57:00.761364+0000","last_fullsized":"2024-02-19T14:57:00.761364+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:15.488805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761179+0000","last_change":"2024-02-19T14:56:58.672366+0000","last_active":"2024-02-19T14:57:00.761179+0000","last_peered":"2024-02-19T14:57:00.761179+0000","last_clean":"2024-02-19T14:57:00.761179+0000","last_became_active":"2024-02-19T14:56:58.672085+0000","last_became_peered":"2024-02-19T14:56:58.672085+0000","last_unstale":"2024-02-19T14:57:00.761179+0000","last_undegraded":"2024-02-19T14:57:00.761179+0000","last_fullsized":"2024-02-19T14:57:00.761179+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760763+0000","last_change":"2024-02-19T14:56:59.681440+0000","last_active":"2024-02-19T14:57:00.760763+0000","last_peered":"2024-02-19T14:57:00.760763+0000","last_clean":"2024-02-19T14:57:00.760763+0000","last_became_active":"2024-02-19T14:56:59.681022+0000","last_became_peered":"2024-02-19T14:56:59.681022+0000","last_unstale":"2024-02-19T14:57:00.760763+0000","last_undegraded":"2024-02-19T14:57:00.760763+0000","last_fullsized":"2024-02-19T14:57:00.760763+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:57.697404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669083+0000","last_change":"2024-02-19T14:56:58.670839+0000","last_active":"2024-02-19T14:57:00.669083+0000","last_peered":"2024-02-19T14:57:00.669083+0000","last_clean":"2024-02-19T14:57:00.669083+0000","last_became_active":"2024-02-19T14:56:58.670507+0000","last_became_peered":"2024-02-19T14:56:58.670507+0000","last_unstale":"2024-02-19T14:57:00.669083+0000","last_undegraded":"2024-02-19T14:57:00.669083+0000","last_fullsized":"2024-02-19T14:57:00.669083+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:38:20.112923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669006+0000","last_change":"2024-02-19T14:56:59.678942+0000","last_active":"2024-02-19T14:57:00.669006+0000","last_peered":"2024-02-19T14:57:00.669006+0000","last_clean":"2024-02-19T14:57:00.669006+0000","last_became_active":"2024-02-19T14:56:59.676927+0000","last_became_peered":"2024-02-19T14:56:59.676927+0000","last_unstale":"2024-02-19T14:57:00.669006+0000","last_undegraded":"2024-02-19T14:57:00.669006+0000","last_fullsized":"2024-02-19T14:57:00.669006+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:09.354585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669547+0000","last_change":"2024-02-19T14:56:58.660184+0000","last_active":"2024-02-19T14:57:00.669547+0000","last_peered":"2024-02-19T14:57:00.669547+0000","last_clean":"2024-02-19T14:57:00.669547+0000","last_became_active":"2024-02-19T14:56:58.659918+0000","last_became_peered":"2024-02-19T14:56:58.659918+0000","last_unstale":"2024-02-19T14:57:00.669547+0000","last_undegraded":"2024-02-19T14:57:00.669547+0000","last_fullsized":"2024-02-19T14:57:00.669547+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761435+0000","last_change":"2024-02-19T14:56:59.678477+0000","last_active":"2024-02-19T14:57:00.761435+0000","last_peered":"2024-02-19T14:57:00.761435+0000","last_clean":"2024-02-19T14:57:00.761435+0000","last_became_active":"2024-02-19T14:56:59.678159+0000","last_became_peered":"2024-02-19T14:56:59.678159+0000","last_unstale":"2024-02-19T14:57:00.761435+0000","last_undegraded":"2024-02-19T14:57:00.761435+0000","last_fullsized":"2024-02-19T14:57:00.761435+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:18.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761441+0000","last_change":"2024-02-19T14:56:58.667075+0000","last_active":"2024-02-19T14:57:00.761441+0000","last_peered":"2024-02-19T14:57:00.761441+0000","last_clean":"2024-02-19T14:57:00.761441+0000","last_became_active":"2024-02-19T14:56:58.666748+0000","last_became_peered":"2024-02-19T14:56:58.666748+0000","last_unstale":"2024-02-19T14:57:00.761441+0000","last_undegraded":"2024-02-19T14:57:00.761441+0000","last_fullsized":"2024-02-19T14:57:00.761441+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:33.368950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668430+0000","last_change":"2024-02-19T14:56:59.680460+0000","last_active":"2024-02-19T14:57:00.668430+0000","last_peered":"2024-02-19T14:57:00.668430+0000","last_clean":"2024-02-19T14:57:00.668430+0000","last_became_active":"2024-02-19T14:56:59.680154+0000","last_became_peered":"2024-02-19T14:56:59.680154+0000","last_unstale":"2024-02-19T14:57:00.668430+0000","last_undegraded":"2024-02-19T14:57:00.668430+0000","last_fullsized":"2024-02-19T14:57:00.668430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:50.758440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668947+0000","last_change":"2024-02-19T14:56:58.672776+0000","last_active":"2024-02-19T14:57:00.668947+0000","last_peered":"2024-02-19T14:57:00.668947+0000","last_clean":"2024-02-19T14:57:00.668947+0000","last_became_active":"2024-02-19T14:56:58.672154+0000","last_became_peered":"2024-02-19T14:56:58.672154+0000","last_unstale":"2024-02-19T14:57:00.668947+0000","last_undegraded":"2024-02-19T14:57:00.668947+0000","last_fullsized":"2024-02-19T14:57:00.668947+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:23.204802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761142+0000","last_change":"2024-02-19T14:56:59.681056+0000","last_active":"2024-02-19T14:57:00.761142+0000","last_peered":"2024-02-19T14:57:00.761142+0000","last_clean":"2024-02-19T14:57:00.761142+0000","last_became_active":"2024-02-19T14:56:59.680740+0000","last_became_peered":"2024-02-19T14:56:59.680740+0000","last_unstale":"2024-02-19T14:57:00.761142+0000","last_undegraded":"2024-02-19T14:57:00.761142+0000","last_fullsized":"2024-02-19T14:57:00.761142+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:53.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760440+0000","last_change":"2024-02-19T14:56:58.667461+0000","last_active":"2024-02-19T14:57:00.760440+0000","last_peered":"2024-02-19T14:57:00.760440+0000","last_clean":"2024-02-19T14:57:00.760440+0000","last_became_active":"2024-02-19T14:56:58.667182+0000","last_became_peered":"2024-02-19T14:56:58.667182+0000","last_unstale":"2024-02-19T14:57:00.760440+0000","last_undegraded":"2024-02-19T14:57:00.760440+0000","last_fullsized":"2024-02-19T14:57:00.760440+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:27:51.843246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669251+0000","last_change":"2024-02-19T14:56:59.673100+0000","last_active":"2024-02-19T14:57:00.669251+0000","last_peered":"2024-02-19T14:57:00.669251+0000","last_clean":"2024-02-19T14:57:00.669251+0000","last_became_active":"2024-02-19T14:56:59.672757+0000","last_became_peered":"2024-02-19T14:56:59.672757+0000","last_unstale":"2024-02-19T14:57:00.669251+0000","last_undegraded":"2024-02-19T14:57:00.669251+0000","last_fullsized":"2024-02-19T14:57:00.669251+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:36.585855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667848+0000","last_change":"2024-02-19T14:56:58.664744+0000","last_active":"2024-02-19T14:57:00.667848+0000","last_peered":"2024-02-19T14:57:00.667848+0000","last_clean":"2024-02-19T14:57:00.667848+0000","last_became_active":"2024-02-19T14:56:58.663422+0000","last_became_peered":"2024-02-19T14:56:58.663422+0000","last_unstale":"2024-02-19T14:57:00.667848+0000","last_undegraded":"2024-02-19T14:57:00.667848+0000","last_fullsized":"2024-02-19T14:57:00.667848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668501+0000","last_change":"2024-02-19T14:56:59.675481+0000","last_active":"2024-02-19T14:57:00.668501+0000","last_peered":"2024-02-19T14:57:00.668501+0000","last_clean":"2024-02-19T14:57:00.668501+0000","last_became_active":"2024-02-19T14:56:59.671484+0000","last_became_peered":"2024-02-19T14:56:59.671484+0000","last_unstale":"2024-02-19T14:57:00.668501+0000","last_undegraded":"2024-02-19T14:57:00.668501+0000","last_fullsized":"2024-02-19T14:57:00.668501+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:52.205158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760386+0000","last_change":"2024-02-19T14:56:58.673320+0000","last_active":"2024-02-19T14:57:00.760386+0000","last_peered":"2024-02-19T14:57:00.760386+0000","last_clean":"2024-02-19T14:57:00.760386+0000","last_became_active":"2024-02-19T14:56:58.673046+0000","last_became_peered":"2024-02-19T14:56:58.673046+0000","last_unstale":"2024-02-19T14:57:00.760386+0000","last_undegraded":"2024-02-19T14:57:00.760386+0000","last_fullsized":"2024-02-19T14:57:00.760386+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:04.357396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669280+0000","last_change":"2024-02-19T14:56:59.669865+0000","last_active":"2024-02-19T14:57:00.669280+0000","last_peered":"2024-02-19T14:57:00.669280+0000","last_clean":"2024-02-19T14:57:00.669280+0000","last_became_active":"2024-02-19T14:56:59.669568+0000","last_became_peered":"2024-02-19T14:56:59.669568+0000","last_unstale":"2024-02-19T14:57:00.669280+0000","last_undegraded":"2024-02-19T14:57:00.669280+0000","last_fullsized":"2024-02-19T14:57:00.669280+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:05.809396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"137'6","reported_seq":20,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761112+0000","last_change":"2024-02-19T14:56:58.668744+0000","last_active":"2024-02-19T14:57:00.761112+0000","last_peered":"2024-02-19T14:57:00.761112+0000","last_clean":"2024-02-19T14:57:00.761112+0000","last_became_active":"2024-02-19T14:56:58.668464+0000","last_became_peered":"2024-02-19T14:56:58.668464+0000","last_unstale":"2024-02-19T14:57:00.761112+0000","last_undegraded":"2024-02-19T14:57:00.761112+0000","last_fullsized":"2024-02-19T14:57:00.761112+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:08.032552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":392,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760484+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:00.760484+0000","last_peered":"2024-02-19T14:57:00.760484+0000","last_clean":"2024-02-19T14:57:00.760484+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:00.760484+0000","last_undegraded":"2024-02-19T14:57:00.760484+0000","last_fullsized":"2024-02-19T14:57:00.760484+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42099999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.378}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.436},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.26800000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.33300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.307},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.36599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":840801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.443},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.433}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.32900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.27500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.311}]}]},{"osd":5,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1732608,"data_stored":840731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.498}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.438},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.28199999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.254}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.41499999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1744896,"data_stored":842343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.435}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.245},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.25900000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.29299999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.308},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.33200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1740800,"data_stored":841347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.307}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.41399999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.373}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.38400000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1740800,"data_stored":840843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.42099999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.505}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":840801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.441}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.47799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2351104,"data_stored":1433417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41799999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.442}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:08.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.128+0000 7feb36177640 1 Processor -- start 2024-02-19T14:57:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.128+0000 7feb36177640 1 -- start start 2024-02-19T14:57:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb36177640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feb300c8e00 0x7feb3006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb36177640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 0x7feb3006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb36177640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3006cd20 con 0x7feb30152700 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb36177640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb301551f0 con 0x7feb300c8e00 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb36177640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb30155490 con 0x7feb3006d910 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb2ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 0x7feb3006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb2effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feb300c8e00 0x7feb3006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.129+0000 7feb2ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 0x7feb3006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:48712/0 (socket says 172.21.15.78:48712) 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.130+0000 7feb2ffff640 1 -- 172.21.15.78:0/1043525141 learned_addr learned my addr 172.21.15.78:0/1043525141 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.130+0000 7feb2ffff640 1 -- 172.21.15.78:0/1043525141 >> v2:172.21.15.150:3300/0 conn(0x7feb300c8e00 msgr2=0x7feb3006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.130+0000 7feb2ffff640 1 --2- 172.21.15.78:0/1043525141 >> v2:172.21.15.150:3300/0 conn(0x7feb300c8e00 0x7feb3006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.130+0000 7feb2ffff640 1 -- 172.21.15.78:0/1043525141 >> v1:172.21.15.150:6789/0 conn(0x7feb30152700 legacy=0x7feb300c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.130+0000 7feb2ffff640 1 -- 172.21.15.78:0/1043525141 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30155770 con 0x7feb3006d910 2024-02-19T14:57:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.130+0000 7feb2effd640 1 --2- 172.21.15.78:0/1043525141 >> v2:172.21.15.150:3300/0 conn(0x7feb300c8e00 0x7feb3006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb2ffff640 1 --2- 172.21.15.78:0/1043525141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 0x7feb3006c7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb28009e50 tx=0x7feb2802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=760215362d76c68c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb2e7fc640 1 -- 172.21.15.78:0/1043525141 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2802ca70 con 0x7feb3006d910 2024-02-19T14:57:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb2e7fc640 1 -- 172.21.15.78:0/1043525141 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2802cbd0 con 0x7feb3006d910 2024-02-19T14:57:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb2e7fc640 1 -- 172.21.15.78:0/1043525141 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb280343f0 con 0x7feb3006d910 2024-02-19T14:57:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 -- 172.21.15.78:0/1043525141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 msgr2=0x7feb3006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 --2- 172.21.15.78:0/1043525141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 0x7feb3006c7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb28009e50 tx=0x7feb2802bed0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 -- 172.21.15.78:0/1043525141 shutdown_connections 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 --2- 172.21.15.78:0/1043525141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb3006d910 0x7feb3006c7e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 --2- 172.21.15.78:0/1043525141 >> v2:172.21.15.150:3300/0 conn(0x7feb300c8e00 0x7feb3006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 -- 172.21.15.78:0/1043525141 >> 172.21.15.78:0/1043525141 conn(0x7feb300cac90 msgr2=0x7feb300cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 -- 172.21.15.78:0/1043525141 shutdown_connections 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.131+0000 7feb36177640 1 -- 172.21.15.78:0/1043525141 wait complete. 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.132+0000 7feb36177640 1 Processor -- start 2024-02-19T14:57:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 -- start start 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 0x7feb300c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb300c8e00 0x7feb301ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb30152700 0x7feb301ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300c2170 con 0x7feb3006d910 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300bf300 con 0x7feb300c8e00 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb36177640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb301ee990 con 0x7feb30152700 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.133+0000 7feb2f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 0x7feb300c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 0x7feb300c1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51006/0 (socket says 172.21.15.78:51006) 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 -- 172.21.15.78:0/2302669266 learned_addr learned my addr 172.21.15.78:0/2302669266 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2effd640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb300c8e00 0x7feb301ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2ffff640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb30152700 0x7feb301ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 -- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb30152700 msgr2=0x7feb301ebd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb30152700 0x7feb301ebd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 -- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb300c8e00 msgr2=0x7feb301ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb300c8e00 0x7feb301ee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 -- 172.21.15.78:0/2302669266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24009400 con 0x7feb3006d910 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2effd640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb300c8e00 0x7feb301ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb2f7fe640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 0x7feb300c1c30 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7feb2000c3c0 tx=0x7feb2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200052d0 con 0x7feb3006d910 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb20005450 con 0x7feb3006d910 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb28009b30 con 0x7feb3006d910 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20007260 con 0x7feb3006d910 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.134+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb300bef90 con 0x7feb3006d910 2024-02-19T14:57:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.135+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb300beb10 con 0x7feb3006d910 2024-02-19T14:57:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.136+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feb200073c0 con 0x7feb3006d910 2024-02-19T14:57:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.136+0000 7feb17fff640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb1c06b810 0x7feb1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.136+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb200895a0 con 0x7feb3006d910 2024-02-19T14:57:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.136+0000 7feb2effd640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb1c06b810 0x7feb1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.137+0000 7feb2effd640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb1c06b810 0x7feb1c06dcd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feb301eef70 tx=0x7feb24009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.141+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb300beb10 con 0x7feb3006d910 2024-02-19T14:57:08.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.308+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb300c1600 con 0x7feb1c06b810 2024-02-19T14:57:08.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.315+0000 7feb17fff640 1 -- 172.21.15.78:0/2302669266 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289940 (secure 0 0 0) 0x7feb300c1600 con 0x7feb1c06b810 2024-02-19T14:57:08.321 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:08.322 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:08.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb1c06b810 msgr2=0x7feb1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb1c06b810 0x7feb1c06dcd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feb301eef70 tx=0x7feb24009190 comp rx=0 tx=0).stop 2024-02-19T14:57:08.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 msgr2=0x7feb300c1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 0x7feb300c1c30 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7feb2000c3c0 tx=0x7feb2000c880 comp rx=0 tx=0).stop 2024-02-19T14:57:08.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 shutdown_connections 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb1c06b810 0x7feb1c06dcd0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb30152700 0x7feb301ebd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb300c8e00 0x7feb301ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 --2- 172.21.15.78:0/2302669266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb3006d910 0x7feb300c1c30 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 >> 172.21.15.78:0/2302669266 conn(0x7feb300cac90 msgr2=0x7feb3014e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 shutdown_connections 2024-02-19T14:57:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.318+0000 7feb36177640 1 -- 172.21.15.78:0/2302669266 wait complete. 2024-02-19T14:57:08.332 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-02-19T14:57:08.119713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264260,"kb_used_data":14760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710460,"statfs":{"total":773094113280,"available":772823511040,"internally_reserved":0,"allocated":15114240,"data_stored":7911404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.465193"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669478+0000","last_change":"2024-02-19T14:56:59.677644+0000","last_active":"2024-02-19T14:57:00.669478+0000","last_peered":"2024-02-19T14:57:00.669478+0000","last_clean":"2024-02-19T14:57:00.669478+0000","last_became_active":"2024-02-19T14:56:59.677373+0000","last_became_peered":"2024-02-19T14:56:59.677373+0000","last_unstale":"2024-02-19T14:57:00.669478+0000","last_undegraded":"2024-02-19T14:57:00.669478+0000","last_fullsized":"2024-02-19T14:57:00.669478+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:09.037633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669486+0000","last_change":"2024-02-19T14:56:59.672957+0000","last_active":"2024-02-19T14:57:00.669486+0000","last_peered":"2024-02-19T14:57:00.669486+0000","last_clean":"2024-02-19T14:57:00.669486+0000","last_became_active":"2024-02-19T14:56:59.672677+0000","last_became_peered":"2024-02-19T14:56:59.672677+0000","last_unstale":"2024-02-19T14:57:00.669486+0000","last_undegraded":"2024-02-19T14:57:00.669486+0000","last_fullsized":"2024-02-19T14:57:00.669486+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:21.790642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669448+0000","last_change":"2024-02-19T14:56:59.670902+0000","last_active":"2024-02-19T14:57:00.669448+0000","last_peered":"2024-02-19T14:57:00.669448+0000","last_clean":"2024-02-19T14:57:00.669448+0000","last_became_active":"2024-02-19T14:56:59.670608+0000","last_became_peered":"2024-02-19T14:56:59.670608+0000","last_unstale":"2024-02-19T14:57:00.669448+0000","last_undegraded":"2024-02-19T14:57:00.669448+0000","last_fullsized":"2024-02-19T14:57:00.669448+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:05.105129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760676+0000","last_change":"2024-02-19T14:56:59.674985+0000","last_active":"2024-02-19T14:57:00.760676+0000","last_peered":"2024-02-19T14:57:00.760676+0000","last_clean":"2024-02-19T14:57:00.760676+0000","last_became_active":"2024-02-19T14:56:59.674722+0000","last_became_peered":"2024-02-19T14:56:59.674722+0000","last_unstale":"2024-02-19T14:57:00.760676+0000","last_undegraded":"2024-02-19T14:57:00.760676+0000","last_fullsized":"2024-02-19T14:57:00.760676+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:31:48.220112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761659+0000","last_change":"2024-02-19T14:56:59.678611+0000","last_active":"2024-02-19T14:57:00.761659+0000","last_peered":"2024-02-19T14:57:00.761659+0000","last_clean":"2024-02-19T14:57:00.761659+0000","last_became_active":"2024-02-19T14:56:59.678350+0000","last_became_peered":"2024-02-19T14:56:59.678350+0000","last_unstale":"2024-02-19T14:57:00.761659+0000","last_undegraded":"2024-02-19T14:57:00.761659+0000","last_fullsized":"2024-02-19T14:57:00.761659+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:13.843441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760880+0000","last_change":"2024-02-19T14:56:59.679879+0000","last_active":"2024-02-19T14:57:00.760880+0000","last_peered":"2024-02-19T14:57:00.760880+0000","last_clean":"2024-02-19T14:57:00.760880+0000","last_became_active":"2024-02-19T14:56:59.678532+0000","last_became_peered":"2024-02-19T14:56:59.678532+0000","last_unstale":"2024-02-19T14:57:00.760880+0000","last_undegraded":"2024-02-19T14:57:00.760880+0000","last_fullsized":"2024-02-19T14:57:00.760880+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:43.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667796+0000","last_change":"2024-02-19T14:56:59.671847+0000","last_active":"2024-02-19T14:57:00.667796+0000","last_peered":"2024-02-19T14:57:00.667796+0000","last_clean":"2024-02-19T14:57:00.667796+0000","last_became_active":"2024-02-19T14:56:59.671586+0000","last_became_peered":"2024-02-19T14:56:59.671586+0000","last_unstale":"2024-02-19T14:57:00.667796+0000","last_undegraded":"2024-02-19T14:57:00.667796+0000","last_fullsized":"2024-02-19T14:57:00.667796+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667989+0000","last_change":"2024-02-19T14:56:59.676911+0000","last_active":"2024-02-19T14:57:00.667989+0000","last_peered":"2024-02-19T14:57:00.667989+0000","last_clean":"2024-02-19T14:57:00.667989+0000","last_became_active":"2024-02-19T14:56:59.676642+0000","last_became_peered":"2024-02-19T14:56:59.676642+0000","last_unstale":"2024-02-19T14:57:00.667989+0000","last_undegraded":"2024-02-19T14:57:00.667989+0000","last_fullsized":"2024-02-19T14:57:00.667989+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:41.584982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761313+0000","last_change":"2024-02-19T14:56:59.669174+0000","last_active":"2024-02-19T14:57:00.761313+0000","last_peered":"2024-02-19T14:57:00.761313+0000","last_clean":"2024-02-19T14:57:00.761313+0000","last_became_active":"2024-02-19T14:56:59.668890+0000","last_became_peered":"2024-02-19T14:56:59.668890+0000","last_unstale":"2024-02-19T14:57:00.761313+0000","last_undegraded":"2024-02-19T14:57:00.761313+0000","last_fullsized":"2024-02-19T14:57:00.761313+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:47:39.962805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760970+0000","last_change":"2024-02-19T14:56:59.672943+0000","last_active":"2024-02-19T14:57:00.760970+0000","last_peered":"2024-02-19T14:57:00.760970+0000","last_clean":"2024-02-19T14:57:00.760970+0000","last_became_active":"2024-02-19T14:56:59.672668+0000","last_became_peered":"2024-02-19T14:56:59.672668+0000","last_unstale":"2024-02-19T14:57:00.760970+0000","last_undegraded":"2024-02-19T14:57:00.760970+0000","last_fullsized":"2024-02-19T14:57:00.760970+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:50.415366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761214+0000","last_change":"2024-02-19T14:56:59.669750+0000","last_active":"2024-02-19T14:57:00.761214+0000","last_peered":"2024-02-19T14:57:00.761214+0000","last_clean":"2024-02-19T14:57:00.761214+0000","last_became_active":"2024-02-19T14:56:59.669490+0000","last_became_peered":"2024-02-19T14:56:59.669490+0000","last_unstale":"2024-02-19T14:57:00.761214+0000","last_undegraded":"2024-02-19T14:57:00.761214+0000","last_fullsized":"2024-02-19T14:57:00.761214+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:15:27.557961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760802+0000","last_change":"2024-02-19T14:56:59.675636+0000","last_active":"2024-02-19T14:57:00.760802+0000","last_peered":"2024-02-19T14:57:00.760802+0000","last_clean":"2024-02-19T14:57:00.760802+0000","last_became_active":"2024-02-19T14:56:59.674106+0000","last_became_peered":"2024-02-19T14:56:59.674106+0000","last_unstale":"2024-02-19T14:57:00.760802+0000","last_undegraded":"2024-02-19T14:57:00.760802+0000","last_fullsized":"2024-02-19T14:57:00.760802+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:56:49.712701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761587+0000","last_change":"2024-02-19T14:56:59.671801+0000","last_active":"2024-02-19T14:57:00.761587+0000","last_peered":"2024-02-19T14:57:00.761587+0000","last_clean":"2024-02-19T14:57:00.761587+0000","last_became_active":"2024-02-19T14:56:59.671332+0000","last_became_peered":"2024-02-19T14:56:59.671332+0000","last_unstale":"2024-02-19T14:57:00.761587+0000","last_undegraded":"2024-02-19T14:57:00.761587+0000","last_fullsized":"2024-02-19T14:57:00.761587+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:39.015073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760548+0000","last_change":"2024-02-19T14:56:59.681973+0000","last_active":"2024-02-19T14:57:00.760548+0000","last_peered":"2024-02-19T14:57:00.760548+0000","last_clean":"2024-02-19T14:57:00.760548+0000","last_became_active":"2024-02-19T14:56:59.681702+0000","last_became_peered":"2024-02-19T14:56:59.681702+0000","last_unstale":"2024-02-19T14:57:00.760548+0000","last_undegraded":"2024-02-19T14:57:00.760548+0000","last_fullsized":"2024-02-19T14:57:00.760548+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:03:48.192555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669252+0000","last_change":"2024-02-19T14:56:59.677121+0000","last_active":"2024-02-19T14:57:00.669252+0000","last_peered":"2024-02-19T14:57:00.669252+0000","last_clean":"2024-02-19T14:57:00.669252+0000","last_became_active":"2024-02-19T14:56:59.676541+0000","last_became_peered":"2024-02-19T14:56:59.676541+0000","last_unstale":"2024-02-19T14:57:00.669252+0000","last_undegraded":"2024-02-19T14:57:00.669252+0000","last_fullsized":"2024-02-19T14:57:00.669252+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760928+0000","last_change":"2024-02-19T14:56:59.670144+0000","last_active":"2024-02-19T14:57:00.760928+0000","last_peered":"2024-02-19T14:57:00.760928+0000","last_clean":"2024-02-19T14:57:00.760928+0000","last_became_active":"2024-02-19T14:56:59.669884+0000","last_became_peered":"2024-02-19T14:56:59.669884+0000","last_unstale":"2024-02-19T14:57:00.760928+0000","last_undegraded":"2024-02-19T14:57:00.760928+0000","last_fullsized":"2024-02-19T14:57:00.760928+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:11.367956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761230+0000","last_change":"2024-02-19T14:56:59.682420+0000","last_active":"2024-02-19T14:57:00.761230+0000","last_peered":"2024-02-19T14:57:00.761230+0000","last_clean":"2024-02-19T14:57:00.761230+0000","last_became_active":"2024-02-19T14:56:59.682141+0000","last_became_peered":"2024-02-19T14:56:59.682141+0000","last_unstale":"2024-02-19T14:57:00.761230+0000","last_undegraded":"2024-02-19T14:57:00.761230+0000","last_fullsized":"2024-02-19T14:57:00.761230+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:54:38.083894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760755+0000","last_change":"2024-02-19T14:56:59.681178+0000","last_active":"2024-02-19T14:57:00.760755+0000","last_peered":"2024-02-19T14:57:00.760755+0000","last_clean":"2024-02-19T14:57:00.760755+0000","last_became_active":"2024-02-19T14:56:59.680802+0000","last_became_peered":"2024-02-19T14:56:59.680802+0000","last_unstale":"2024-02-19T14:57:00.760755+0000","last_undegraded":"2024-02-19T14:57:00.760755+0000","last_fullsized":"2024-02-19T14:57:00.760755+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:58:32.096559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761012+0000","last_change":"2024-02-19T14:56:59.683504+0000","last_active":"2024-02-19T14:57:00.761012+0000","last_peered":"2024-02-19T14:57:00.761012+0000","last_clean":"2024-02-19T14:57:00.761012+0000","last_became_active":"2024-02-19T14:56:59.683207+0000","last_became_peered":"2024-02-19T14:56:59.683207+0000","last_unstale":"2024-02-19T14:57:00.761012+0000","last_undegraded":"2024-02-19T14:57:00.761012+0000","last_fullsized":"2024-02-19T14:57:00.761012+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:40:50.415366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760700+0000","last_change":"2024-02-19T14:56:59.671258+0000","last_active":"2024-02-19T14:57:00.760700+0000","last_peered":"2024-02-19T14:57:00.760700+0000","last_clean":"2024-02-19T14:57:00.760700+0000","last_became_active":"2024-02-19T14:56:59.670922+0000","last_became_peered":"2024-02-19T14:56:59.670922+0000","last_unstale":"2024-02-19T14:57:00.760700+0000","last_undegraded":"2024-02-19T14:57:00.760700+0000","last_fullsized":"2024-02-19T14:57:00.760700+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:25.573016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761658+0000","last_change":"2024-02-19T14:56:59.669256+0000","last_active":"2024-02-19T14:57:00.761658+0000","last_peered":"2024-02-19T14:57:00.761658+0000","last_clean":"2024-02-19T14:57:00.761658+0000","last_became_active":"2024-02-19T14:56:59.668975+0000","last_became_peered":"2024-02-19T14:56:59.668975+0000","last_unstale":"2024-02-19T14:57:00.761658+0000","last_undegraded":"2024-02-19T14:57:00.761658+0000","last_fullsized":"2024-02-19T14:57:00.761658+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:06.556391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668252+0000","last_change":"2024-02-19T14:56:59.672561+0000","last_active":"2024-02-19T14:57:00.668252+0000","last_peered":"2024-02-19T14:57:00.668252+0000","last_clean":"2024-02-19T14:57:00.668252+0000","last_became_active":"2024-02-19T14:56:59.672171+0000","last_became_peered":"2024-02-19T14:56:59.672171+0000","last_unstale":"2024-02-19T14:57:00.668252+0000","last_undegraded":"2024-02-19T14:57:00.668252+0000","last_fullsized":"2024-02-19T14:57:00.668252+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:36.585855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668809+0000","last_change":"2024-02-19T14:56:59.677599+0000","last_active":"2024-02-19T14:57:00.668809+0000","last_peered":"2024-02-19T14:57:00.668809+0000","last_clean":"2024-02-19T14:57:00.668809+0000","last_became_active":"2024-02-19T14:56:59.677306+0000","last_became_peered":"2024-02-19T14:56:59.677306+0000","last_unstale":"2024-02-19T14:57:00.668809+0000","last_undegraded":"2024-02-19T14:57:00.668809+0000","last_fullsized":"2024-02-19T14:57:00.668809+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:56.306598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760380+0000","last_change":"2024-02-19T14:56:59.670448+0000","last_active":"2024-02-19T14:57:00.760380+0000","last_peered":"2024-02-19T14:57:00.760380+0000","last_clean":"2024-02-19T14:57:00.760380+0000","last_became_active":"2024-02-19T14:56:59.670155+0000","last_became_peered":"2024-02-19T14:56:59.670155+0000","last_unstale":"2024-02-19T14:57:00.760380+0000","last_undegraded":"2024-02-19T14:57:00.760380+0000","last_fullsized":"2024-02-19T14:57:00.760380+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:07.818574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669346+0000","last_change":"2024-02-19T14:56:59.678644+0000","last_active":"2024-02-19T14:57:00.669346+0000","last_peered":"2024-02-19T14:57:00.669346+0000","last_clean":"2024-02-19T14:57:00.669346+0000","last_became_active":"2024-02-19T14:56:59.678353+0000","last_became_peered":"2024-02-19T14:56:59.678353+0000","last_unstale":"2024-02-19T14:57:00.669346+0000","last_undegraded":"2024-02-19T14:57:00.669346+0000","last_fullsized":"2024-02-19T14:57:00.669346+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:05.105129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668644+0000","last_change":"2024-02-19T14:56:59.679167+0000","last_active":"2024-02-19T14:57:00.668644+0000","last_peered":"2024-02-19T14:57:00.668644+0000","last_clean":"2024-02-19T14:57:00.668644+0000","last_became_active":"2024-02-19T14:56:59.678635+0000","last_became_peered":"2024-02-19T14:56:59.678635+0000","last_unstale":"2024-02-19T14:57:00.668644+0000","last_undegraded":"2024-02-19T14:57:00.668644+0000","last_fullsized":"2024-02-19T14:57:00.668644+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:34.374030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761176+0000","last_change":"2024-02-19T14:56:59.669279+0000","last_active":"2024-02-19T14:57:00.761176+0000","last_peered":"2024-02-19T14:57:00.761176+0000","last_clean":"2024-02-19T14:57:00.761176+0000","last_became_active":"2024-02-19T14:56:59.668990+0000","last_became_peered":"2024-02-19T14:56:59.668990+0000","last_unstale":"2024-02-19T14:57:00.761176+0000","last_undegraded":"2024-02-19T14:57:00.761176+0000","last_fullsized":"2024-02-19T14:57:00.761176+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:37.163331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760625+0000","last_change":"2024-02-19T14:56:59.674868+0000","last_active":"2024-02-19T14:57:00.760625+0000","last_peered":"2024-02-19T14:57:00.760625+0000","last_clean":"2024-02-19T14:57:00.760625+0000","last_became_active":"2024-02-19T14:56:59.674597+0000","last_became_peered":"2024-02-19T14:56:59.674597+0000","last_unstale":"2024-02-19T14:57:00.760625+0000","last_undegraded":"2024-02-19T14:57:00.760625+0000","last_fullsized":"2024-02-19T14:57:00.760625+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:33:36.449995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668644+0000","last_change":"2024-02-19T14:56:59.675815+0000","last_active":"2024-02-19T14:57:00.668644+0000","last_peered":"2024-02-19T14:57:00.668644+0000","last_clean":"2024-02-19T14:57:00.668644+0000","last_became_active":"2024-02-19T14:56:59.671817+0000","last_became_peered":"2024-02-19T14:56:59.671817+0000","last_unstale":"2024-02-19T14:57:00.668644+0000","last_undegraded":"2024-02-19T14:57:00.668644+0000","last_fullsized":"2024-02-19T14:57:00.668644+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:24.209883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760503+0000","last_change":"2024-02-19T14:56:59.675553+0000","last_active":"2024-02-19T14:57:00.760503+0000","last_peered":"2024-02-19T14:57:00.760503+0000","last_clean":"2024-02-19T14:57:00.760503+0000","last_became_active":"2024-02-19T14:56:59.675281+0000","last_became_peered":"2024-02-19T14:56:59.675281+0000","last_unstale":"2024-02-19T14:57:00.760503+0000","last_undegraded":"2024-02-19T14:57:00.760503+0000","last_fullsized":"2024-02-19T14:57:00.760503+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:36:18.221340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760888+0000","last_change":"2024-02-19T14:56:59.669519+0000","last_active":"2024-02-19T14:57:00.760888+0000","last_peered":"2024-02-19T14:57:00.760888+0000","last_clean":"2024-02-19T14:57:00.760888+0000","last_became_active":"2024-02-19T14:56:59.667272+0000","last_became_peered":"2024-02-19T14:56:59.667272+0000","last_unstale":"2024-02-19T14:57:00.760888+0000","last_undegraded":"2024-02-19T14:57:00.760888+0000","last_fullsized":"2024-02-19T14:57:00.760888+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:40.323947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668136+0000","last_change":"2024-02-19T14:56:59.676162+0000","last_active":"2024-02-19T14:57:00.668136+0000","last_peered":"2024-02-19T14:57:00.668136+0000","last_clean":"2024-02-19T14:57:00.668136+0000","last_became_active":"2024-02-19T14:56:59.675867+0000","last_became_peered":"2024-02-19T14:56:59.675867+0000","last_unstale":"2024-02-19T14:57:00.668136+0000","last_undegraded":"2024-02-19T14:57:00.668136+0000","last_fullsized":"2024-02-19T14:57:00.668136+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:27.213645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669432+0000","last_change":"2024-02-19T14:56:59.665767+0000","last_active":"2024-02-19T14:57:00.669432+0000","last_peered":"2024-02-19T14:57:00.669432+0000","last_clean":"2024-02-19T14:57:00.669432+0000","last_became_active":"2024-02-19T14:56:59.665503+0000","last_became_peered":"2024-02-19T14:56:59.665503+0000","last_unstale":"2024-02-19T14:57:00.669432+0000","last_undegraded":"2024-02-19T14:57:00.669432+0000","last_fullsized":"2024-02-19T14:57:00.669432+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:50.909922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"137'4","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761521+0000","last_change":"2024-02-19T14:56:58.666557+0000","last_active":"2024-02-19T14:57:00.761521+0000","last_peered":"2024-02-19T14:57:00.761521+0000","last_clean":"2024-02-19T14:57:00.761521+0000","last_became_active":"2024-02-19T14:56:58.666286+0000","last_became_peered":"2024-02-19T14:56:58.666286+0000","last_unstale":"2024-02-19T14:57:00.761521+0000","last_undegraded":"2024-02-19T14:57:00.761521+0000","last_fullsized":"2024-02-19T14:57:00.761521+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:49.904842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669498+0000","last_change":"2024-02-19T14:56:59.674314+0000","last_active":"2024-02-19T14:57:00.669498+0000","last_peered":"2024-02-19T14:57:00.669498+0000","last_clean":"2024-02-19T14:57:00.669498+0000","last_became_active":"2024-02-19T14:56:59.673451+0000","last_became_peered":"2024-02-19T14:56:59.673451+0000","last_unstale":"2024-02-19T14:57:00.669498+0000","last_undegraded":"2024-02-19T14:57:00.669498+0000","last_fullsized":"2024-02-19T14:57:00.669498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:09.037633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760607+0000","last_change":"2024-02-19T14:56:58.672364+0000","last_active":"2024-02-19T14:57:00.760607+0000","last_peered":"2024-02-19T14:57:00.760607+0000","last_clean":"2024-02-19T14:57:00.760607+0000","last_became_active":"2024-02-19T14:56:58.672089+0000","last_became_peered":"2024-02-19T14:56:58.672089+0000","last_unstale":"2024-02-19T14:57:00.760607+0000","last_undegraded":"2024-02-19T14:57:00.760607+0000","last_fullsized":"2024-02-19T14:57:00.760607+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:04.100049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668467+0000","last_change":"2024-02-19T14:56:59.670090+0000","last_active":"2024-02-19T14:57:00.668467+0000","last_peered":"2024-02-19T14:57:00.668467+0000","last_clean":"2024-02-19T14:57:00.668467+0000","last_became_active":"2024-02-19T14:56:59.669839+0000","last_became_peered":"2024-02-19T14:56:59.669839+0000","last_unstale":"2024-02-19T14:57:00.668467+0000","last_undegraded":"2024-02-19T14:57:00.668467+0000","last_fullsized":"2024-02-19T14:57:00.668467+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:46.789869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761059+0000","last_change":"2024-02-19T14:56:58.675313+0000","last_active":"2024-02-19T14:57:00.761059+0000","last_peered":"2024-02-19T14:57:00.761059+0000","last_clean":"2024-02-19T14:57:00.761059+0000","last_became_active":"2024-02-19T14:56:58.675047+0000","last_became_peered":"2024-02-19T14:56:58.675047+0000","last_unstale":"2024-02-19T14:57:00.761059+0000","last_undegraded":"2024-02-19T14:57:00.761059+0000","last_fullsized":"2024-02-19T14:57:00.761059+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:52:41.628070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760957+0000","last_change":"2024-02-19T14:56:59.679353+0000","last_active":"2024-02-19T14:57:00.760957+0000","last_peered":"2024-02-19T14:57:00.760957+0000","last_clean":"2024-02-19T14:57:00.760957+0000","last_became_active":"2024-02-19T14:56:59.679070+0000","last_became_peered":"2024-02-19T14:56:59.679070+0000","last_unstale":"2024-02-19T14:57:00.760957+0000","last_undegraded":"2024-02-19T14:57:00.760957+0000","last_fullsized":"2024-02-19T14:57:00.760957+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:11.367956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669346+0000","last_change":"2024-02-19T14:56:58.668754+0000","last_active":"2024-02-19T14:57:00.669346+0000","last_peered":"2024-02-19T14:57:00.669346+0000","last_clean":"2024-02-19T14:57:00.669346+0000","last_became_active":"2024-02-19T14:56:58.668195+0000","last_became_peered":"2024-02-19T14:56:58.668195+0000","last_unstale":"2024-02-19T14:57:00.669346+0000","last_undegraded":"2024-02-19T14:57:00.669346+0000","last_fullsized":"2024-02-19T14:57:00.669346+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760423+0000","last_change":"2024-02-19T14:56:59.674433+0000","last_active":"2024-02-19T14:57:00.760423+0000","last_peered":"2024-02-19T14:57:00.760423+0000","last_clean":"2024-02-19T14:57:00.760423+0000","last_became_active":"2024-02-19T14:56:59.674111+0000","last_became_peered":"2024-02-19T14:56:59.674111+0000","last_unstale":"2024-02-19T14:57:00.760423+0000","last_undegraded":"2024-02-19T14:57:00.760423+0000","last_fullsized":"2024-02-19T14:57:00.760423+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:39.579884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668697+0000","last_change":"2024-02-19T14:56:58.672280+0000","last_active":"2024-02-19T14:57:00.668697+0000","last_peered":"2024-02-19T14:57:00.668697+0000","last_clean":"2024-02-19T14:57:00.668697+0000","last_became_active":"2024-02-19T14:56:58.671978+0000","last_became_peered":"2024-02-19T14:56:58.671978+0000","last_unstale":"2024-02-19T14:57:00.668697+0000","last_undegraded":"2024-02-19T14:57:00.668697+0000","last_fullsized":"2024-02-19T14:57:00.668697+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:18:38.026526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760682+0000","last_change":"2024-02-19T14:56:59.674348+0000","last_active":"2024-02-19T14:57:00.760682+0000","last_peered":"2024-02-19T14:57:00.760682+0000","last_clean":"2024-02-19T14:57:00.760682+0000","last_became_active":"2024-02-19T14:56:59.674060+0000","last_became_peered":"2024-02-19T14:56:59.674060+0000","last_unstale":"2024-02-19T14:57:00.760682+0000","last_undegraded":"2024-02-19T14:57:00.760682+0000","last_fullsized":"2024-02-19T14:57:00.760682+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:25:40.373677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761105+0000","last_change":"2024-02-19T14:56:58.673135+0000","last_active":"2024-02-19T14:57:00.761105+0000","last_peered":"2024-02-19T14:57:00.761105+0000","last_clean":"2024-02-19T14:57:00.761105+0000","last_became_active":"2024-02-19T14:56:58.672822+0000","last_became_peered":"2024-02-19T14:56:58.672822+0000","last_unstale":"2024-02-19T14:57:00.761105+0000","last_undegraded":"2024-02-19T14:57:00.761105+0000","last_fullsized":"2024-02-19T14:57:00.761105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:08.349505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669233+0000","last_change":"2024-02-19T14:56:59.673661+0000","last_active":"2024-02-19T14:57:00.669233+0000","last_peered":"2024-02-19T14:57:00.669233+0000","last_clean":"2024-02-19T14:57:00.669233+0000","last_became_active":"2024-02-19T14:56:59.672005+0000","last_became_peered":"2024-02-19T14:56:59.672005+0000","last_unstale":"2024-02-19T14:57:00.669233+0000","last_undegraded":"2024-02-19T14:57:00.669233+0000","last_fullsized":"2024-02-19T14:57:00.669233+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:53.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667726+0000","last_change":"2024-02-19T14:56:58.666464+0000","last_active":"2024-02-19T14:57:00.667726+0000","last_peered":"2024-02-19T14:57:00.667726+0000","last_clean":"2024-02-19T14:57:00.667726+0000","last_became_active":"2024-02-19T14:56:58.665994+0000","last_became_peered":"2024-02-19T14:56:58.665994+0000","last_unstale":"2024-02-19T14:57:00.667726+0000","last_undegraded":"2024-02-19T14:57:00.667726+0000","last_fullsized":"2024-02-19T14:57:00.667726+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:45.784789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668891+0000","last_change":"2024-02-19T14:56:59.676047+0000","last_active":"2024-02-19T14:57:00.668891+0000","last_peered":"2024-02-19T14:57:00.668891+0000","last_clean":"2024-02-19T14:57:00.668891+0000","last_became_active":"2024-02-19T14:56:59.675731+0000","last_became_peered":"2024-02-19T14:56:59.675731+0000","last_unstale":"2024-02-19T14:57:00.668891+0000","last_undegraded":"2024-02-19T14:57:00.668891+0000","last_fullsized":"2024-02-19T14:57:00.668891+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:50.909922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761024+0000","last_change":"2024-02-19T14:56:58.663619+0000","last_active":"2024-02-19T14:57:00.761024+0000","last_peered":"2024-02-19T14:57:00.761024+0000","last_clean":"2024-02-19T14:57:00.761024+0000","last_became_active":"2024-02-19T14:56:58.663358+0000","last_became_peered":"2024-02-19T14:56:58.663358+0000","last_unstale":"2024-02-19T14:57:00.761024+0000","last_undegraded":"2024-02-19T14:57:00.761024+0000","last_fullsized":"2024-02-19T14:57:00.761024+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:35.580775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669142+0000","last_change":"2024-02-19T14:56:59.669001+0000","last_active":"2024-02-19T14:57:00.669142+0000","last_peered":"2024-02-19T14:57:00.669142+0000","last_clean":"2024-02-19T14:57:00.669142+0000","last_became_active":"2024-02-19T14:56:59.667676+0000","last_became_peered":"2024-02-19T14:56:59.667676+0000","last_unstale":"2024-02-19T14:57:00.669142+0000","last_undegraded":"2024-02-19T14:57:00.669142+0000","last_fullsized":"2024-02-19T14:57:00.669142+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:38:21.118003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760966+0000","last_change":"2024-02-19T14:56:58.670376+0000","last_active":"2024-02-19T14:57:00.760966+0000","last_peered":"2024-02-19T14:57:00.760966+0000","last_clean":"2024-02-19T14:57:00.760966+0000","last_became_active":"2024-02-19T14:56:58.670065+0000","last_became_peered":"2024-02-19T14:56:58.670065+0000","last_unstale":"2024-02-19T14:57:00.760966+0000","last_undegraded":"2024-02-19T14:57:00.760966+0000","last_fullsized":"2024-02-19T14:57:00.760966+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:08.032552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761353+0000","last_change":"2024-02-19T14:56:59.669101+0000","last_active":"2024-02-19T14:57:00.761353+0000","last_peered":"2024-02-19T14:57:00.761353+0000","last_clean":"2024-02-19T14:57:00.761353+0000","last_became_active":"2024-02-19T14:56:59.668601+0000","last_became_peered":"2024-02-19T14:56:59.668601+0000","last_unstale":"2024-02-19T14:57:00.761353+0000","last_undegraded":"2024-02-19T14:57:00.761353+0000","last_fullsized":"2024-02-19T14:57:00.761353+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:12:40.244630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669055+0000","last_change":"2024-02-19T14:56:58.673499+0000","last_active":"2024-02-19T14:57:00.669055+0000","last_peered":"2024-02-19T14:57:00.669055+0000","last_clean":"2024-02-19T14:57:00.669055+0000","last_became_active":"2024-02-19T14:56:58.673014+0000","last_became_peered":"2024-02-19T14:56:58.673014+0000","last_unstale":"2024-02-19T14:57:00.669055+0000","last_undegraded":"2024-02-19T14:57:00.669055+0000","last_fullsized":"2024-02-19T14:57:00.669055+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T14:58:55.301518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760881+0000","last_change":"2024-02-19T14:56:59.680952+0000","last_active":"2024-02-19T14:57:00.760881+0000","last_peered":"2024-02-19T14:57:00.760881+0000","last_clean":"2024-02-19T14:57:00.760881+0000","last_became_active":"2024-02-19T14:56:59.680634+0000","last_became_peered":"2024-02-19T14:56:59.680634+0000","last_unstale":"2024-02-19T14:57:00.760881+0000","last_undegraded":"2024-02-19T14:57:00.760881+0000","last_fullsized":"2024-02-19T14:57:00.760881+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:08.161866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"137'3","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761402+0000","last_change":"2024-02-19T14:56:58.664991+0000","last_active":"2024-02-19T14:57:00.761402+0000","last_peered":"2024-02-19T14:57:00.761402+0000","last_clean":"2024-02-19T14:57:00.761402+0000","last_became_active":"2024-02-19T14:56:58.664715+0000","last_became_peered":"2024-02-19T14:56:58.664715+0000","last_unstale":"2024-02-19T14:57:00.761402+0000","last_undegraded":"2024-02-19T14:57:00.761402+0000","last_fullsized":"2024-02-19T14:57:00.761402+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:57:14.196709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668295+0000","last_change":"2024-02-19T14:56:59.669119+0000","last_active":"2024-02-19T14:57:00.668295+0000","last_peered":"2024-02-19T14:57:00.668295+0000","last_clean":"2024-02-19T14:57:00.668295+0000","last_became_active":"2024-02-19T14:56:59.668836+0000","last_became_peered":"2024-02-19T14:56:59.668836+0000","last_unstale":"2024-02-19T14:57:00.668295+0000","last_undegraded":"2024-02-19T14:57:00.668295+0000","last_fullsized":"2024-02-19T14:57:00.668295+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:14.400279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761709+0000","last_change":"2024-02-19T14:56:58.666985+0000","last_active":"2024-02-19T14:57:00.761709+0000","last_peered":"2024-02-19T14:57:00.761709+0000","last_clean":"2024-02-19T14:57:00.761709+0000","last_became_active":"2024-02-19T14:56:58.666667+0000","last_became_peered":"2024-02-19T14:56:58.666667+0000","last_unstale":"2024-02-19T14:57:00.761709+0000","last_undegraded":"2024-02-19T14:57:00.761709+0000","last_fullsized":"2024-02-19T14:57:00.761709+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:35.382164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669279+0000","last_change":"2024-02-19T14:56:59.677403+0000","last_active":"2024-02-19T14:57:00.669279+0000","last_peered":"2024-02-19T14:57:00.669279+0000","last_clean":"2024-02-19T14:57:00.669279+0000","last_became_active":"2024-02-19T14:56:59.676857+0000","last_became_peered":"2024-02-19T14:56:59.676857+0000","last_unstale":"2024-02-19T14:57:00.669279+0000","last_undegraded":"2024-02-19T14:57:00.669279+0000","last_fullsized":"2024-02-19T14:57:00.669279+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:36.387245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760486+0000","last_change":"2024-02-19T14:56:58.673001+0000","last_active":"2024-02-19T14:57:00.760486+0000","last_peered":"2024-02-19T14:57:00.760486+0000","last_clean":"2024-02-19T14:57:00.760486+0000","last_became_active":"2024-02-19T14:56:58.672751+0000","last_became_peered":"2024-02-19T14:56:58.672751+0000","last_unstale":"2024-02-19T14:57:00.760486+0000","last_undegraded":"2024-02-19T14:57:00.760486+0000","last_fullsized":"2024-02-19T14:57:00.760486+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:13.395199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761015+0000","last_change":"2024-02-19T14:56:59.667686+0000","last_active":"2024-02-19T14:57:00.761015+0000","last_peered":"2024-02-19T14:57:00.761015+0000","last_clean":"2024-02-19T14:57:00.761015+0000","last_became_active":"2024-02-19T14:56:59.667391+0000","last_became_peered":"2024-02-19T14:56:59.667391+0000","last_unstale":"2024-02-19T14:57:00.761015+0000","last_undegraded":"2024-02-19T14:57:00.761015+0000","last_fullsized":"2024-02-19T14:57:00.761015+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:25.913809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760722+0000","last_change":"2024-02-19T14:56:58.670054+0000","last_active":"2024-02-19T14:57:00.760722+0000","last_peered":"2024-02-19T14:57:00.760722+0000","last_clean":"2024-02-19T14:57:00.760722+0000","last_became_active":"2024-02-19T14:56:58.669599+0000","last_became_peered":"2024-02-19T14:56:58.669599+0000","last_unstale":"2024-02-19T14:57:00.760722+0000","last_undegraded":"2024-02-19T14:57:00.760722+0000","last_fullsized":"2024-02-19T14:57:00.760722+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:26.208565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669254+0000","last_change":"2024-02-19T14:56:59.671613+0000","last_active":"2024-02-19T14:57:00.669254+0000","last_peered":"2024-02-19T14:57:00.669254+0000","last_clean":"2024-02-19T14:57:00.669254+0000","last_became_active":"2024-02-19T14:56:59.671340+0000","last_became_peered":"2024-02-19T14:56:59.671340+0000","last_unstale":"2024-02-19T14:57:00.669254+0000","last_undegraded":"2024-02-19T14:57:00.669254+0000","last_fullsized":"2024-02-19T14:57:00.669254+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:34.374030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"137'8","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667544+0000","last_change":"2024-02-19T14:56:58.668292+0000","last_active":"2024-02-19T14:57:00.667544+0000","last_peered":"2024-02-19T14:57:00.667544+0000","last_clean":"2024-02-19T14:57:00.667544+0000","last_became_active":"2024-02-19T14:56:58.668000+0000","last_became_peered":"2024-02-19T14:56:58.668000+0000","last_unstale":"2024-02-19T14:57:00.667544+0000","last_undegraded":"2024-02-19T14:57:00.667544+0000","last_fullsized":"2024-02-19T14:57:00.667544+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760620+0000","last_change":"2024-02-19T14:56:59.674229+0000","last_active":"2024-02-19T14:57:00.760620+0000","last_peered":"2024-02-19T14:57:00.760620+0000","last_clean":"2024-02-19T14:57:00.760620+0000","last_became_active":"2024-02-19T14:56:59.673948+0000","last_became_peered":"2024-02-19T14:56:59.673948+0000","last_unstale":"2024-02-19T14:57:00.760620+0000","last_undegraded":"2024-02-19T14:57:00.760620+0000","last_fullsized":"2024-02-19T14:57:00.760620+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:43.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760686+0000","last_change":"2024-02-19T14:56:58.668198+0000","last_active":"2024-02-19T14:57:00.760686+0000","last_peered":"2024-02-19T14:57:00.760686+0000","last_clean":"2024-02-19T14:57:00.760686+0000","last_became_active":"2024-02-19T14:56:58.667965+0000","last_became_peered":"2024-02-19T14:56:58.667965+0000","last_unstale":"2024-02-19T14:57:00.760686+0000","last_undegraded":"2024-02-19T14:57:00.760686+0000","last_fullsized":"2024-02-19T14:57:00.760686+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:28.054550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761080+0000","last_change":"2024-02-19T14:56:59.684025+0000","last_active":"2024-02-19T14:57:00.761080+0000","last_peered":"2024-02-19T14:57:00.761080+0000","last_clean":"2024-02-19T14:57:00.761080+0000","last_became_active":"2024-02-19T14:56:59.683791+0000","last_became_peered":"2024-02-19T14:56:59.683791+0000","last_unstale":"2024-02-19T14:57:00.761080+0000","last_undegraded":"2024-02-19T14:57:00.761080+0000","last_fullsized":"2024-02-19T14:57:00.761080+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:57:25.913809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760700+0000","last_change":"2024-02-19T14:56:58.669910+0000","last_active":"2024-02-19T14:57:00.760700+0000","last_peered":"2024-02-19T14:57:00.760700+0000","last_clean":"2024-02-19T14:57:00.760700+0000","last_became_active":"2024-02-19T14:56:58.669594+0000","last_became_peered":"2024-02-19T14:56:58.669594+0000","last_unstale":"2024-02-19T14:57:00.760700+0000","last_undegraded":"2024-02-19T14:57:00.760700+0000","last_fullsized":"2024-02-19T14:57:00.760700+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:20.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668860+0000","last_change":"2024-02-19T14:56:59.679318+0000","last_active":"2024-02-19T14:57:00.668860+0000","last_peered":"2024-02-19T14:57:00.668860+0000","last_clean":"2024-02-19T14:57:00.668860+0000","last_became_active":"2024-02-19T14:56:59.678816+0000","last_became_peered":"2024-02-19T14:56:59.678816+0000","last_unstale":"2024-02-19T14:57:00.668860+0000","last_undegraded":"2024-02-19T14:57:00.668860+0000","last_fullsized":"2024-02-19T14:57:00.668860+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:36.387245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669503+0000","last_change":"2024-02-19T14:56:58.669660+0000","last_active":"2024-02-19T14:57:00.669503+0000","last_peered":"2024-02-19T14:57:00.669503+0000","last_clean":"2024-02-19T14:57:00.669503+0000","last_became_active":"2024-02-19T14:56:58.669395+0000","last_became_peered":"2024-02-19T14:56:58.669395+0000","last_unstale":"2024-02-19T14:57:00.669503+0000","last_undegraded":"2024-02-19T14:57:00.669503+0000","last_fullsized":"2024-02-19T14:57:00.669503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:51:45.784789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669458+0000","last_change":"2024-02-19T14:56:59.676902+0000","last_active":"2024-02-19T14:57:00.669458+0000","last_peered":"2024-02-19T14:57:00.669458+0000","last_clean":"2024-02-19T14:57:00.669458+0000","last_became_active":"2024-02-19T14:56:59.676621+0000","last_became_peered":"2024-02-19T14:56:59.676621+0000","last_unstale":"2024-02-19T14:57:00.669458+0000","last_undegraded":"2024-02-19T14:57:00.669458+0000","last_fullsized":"2024-02-19T14:57:00.669458+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:21.790642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760848+0000","last_change":"2024-02-19T14:56:58.673331+0000","last_active":"2024-02-19T14:57:00.760848+0000","last_peered":"2024-02-19T14:57:00.760848+0000","last_clean":"2024-02-19T14:57:00.760848+0000","last_became_active":"2024-02-19T14:56:58.673067+0000","last_became_peered":"2024-02-19T14:56:58.673067+0000","last_unstale":"2024-02-19T14:57:00.760848+0000","last_undegraded":"2024-02-19T14:57:00.760848+0000","last_fullsized":"2024-02-19T14:57:00.760848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760722+0000","last_change":"2024-02-19T14:56:59.684112+0000","last_active":"2024-02-19T14:57:00.760722+0000","last_peered":"2024-02-19T14:57:00.760722+0000","last_clean":"2024-02-19T14:57:00.760722+0000","last_became_active":"2024-02-19T14:56:59.683866+0000","last_became_peered":"2024-02-19T14:56:59.683866+0000","last_unstale":"2024-02-19T14:57:00.760722+0000","last_undegraded":"2024-02-19T14:57:00.760722+0000","last_fullsized":"2024-02-19T14:57:00.760722+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:40.323947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760568+0000","last_change":"2024-02-19T14:56:58.672326+0000","last_active":"2024-02-19T14:57:00.760568+0000","last_peered":"2024-02-19T14:57:00.760568+0000","last_clean":"2024-02-19T14:57:00.760568+0000","last_became_active":"2024-02-19T14:56:58.672056+0000","last_became_peered":"2024-02-19T14:56:58.672056+0000","last_unstale":"2024-02-19T14:57:00.760568+0000","last_undegraded":"2024-02-19T14:57:00.760568+0000","last_fullsized":"2024-02-19T14:57:00.760568+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:04.357396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669386+0000","last_change":"2024-02-19T14:56:59.679132+0000","last_active":"2024-02-19T14:57:00.669386+0000","last_peered":"2024-02-19T14:57:00.669386+0000","last_clean":"2024-02-19T14:57:00.669386+0000","last_became_active":"2024-02-19T14:56:59.678819+0000","last_became_peered":"2024-02-19T14:56:59.678819+0000","last_unstale":"2024-02-19T14:57:00.669386+0000","last_undegraded":"2024-02-19T14:57:00.669386+0000","last_fullsized":"2024-02-19T14:57:00.669386+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:09.354585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668134+0000","last_change":"2024-02-19T14:56:58.664244+0000","last_active":"2024-02-19T14:57:00.668134+0000","last_peered":"2024-02-19T14:57:00.668134+0000","last_clean":"2024-02-19T14:57:00.668134+0000","last_became_active":"2024-02-19T14:56:58.663902+0000","last_became_peered":"2024-02-19T14:56:58.663902+0000","last_unstale":"2024-02-19T14:57:00.668134+0000","last_undegraded":"2024-02-19T14:57:00.668134+0000","last_fullsized":"2024-02-19T14:57:00.668134+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:10.863066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760504+0000","last_change":"2024-02-19T14:56:59.677739+0000","last_active":"2024-02-19T14:57:00.760504+0000","last_peered":"2024-02-19T14:57:00.760504+0000","last_clean":"2024-02-19T14:57:00.760504+0000","last_became_active":"2024-02-19T14:56:59.677418+0000","last_became_peered":"2024-02-19T14:56:59.677418+0000","last_unstale":"2024-02-19T14:57:00.760504+0000","last_undegraded":"2024-02-19T14:57:00.760504+0000","last_fullsized":"2024-02-19T14:57:00.760504+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:13:48.895089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760318+0000","last_change":"2024-02-19T14:56:58.672372+0000","last_active":"2024-02-19T14:57:00.760318+0000","last_peered":"2024-02-19T14:57:00.760318+0000","last_clean":"2024-02-19T14:57:00.760318+0000","last_became_active":"2024-02-19T14:56:58.672074+0000","last_became_peered":"2024-02-19T14:56:58.672074+0000","last_unstale":"2024-02-19T14:57:00.760318+0000","last_undegraded":"2024-02-19T14:57:00.760318+0000","last_fullsized":"2024-02-19T14:57:00.760318+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:14:20.785562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669638+0000","last_change":"2024-02-19T14:56:59.676678+0000","last_active":"2024-02-19T14:57:00.669638+0000","last_peered":"2024-02-19T14:57:00.669638+0000","last_clean":"2024-02-19T14:57:00.669638+0000","last_became_active":"2024-02-19T14:56:59.676424+0000","last_became_peered":"2024-02-19T14:56:59.676424+0000","last_unstale":"2024-02-19T14:57:00.669638+0000","last_undegraded":"2024-02-19T14:57:00.669638+0000","last_fullsized":"2024-02-19T14:57:00.669638+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:05.362476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668478+0000","last_change":"2024-02-19T14:56:58.669077+0000","last_active":"2024-02-19T14:57:00.668478+0000","last_peered":"2024-02-19T14:57:00.668478+0000","last_clean":"2024-02-19T14:57:00.668478+0000","last_became_active":"2024-02-19T14:56:58.668349+0000","last_became_peered":"2024-02-19T14:56:58.668349+0000","last_unstale":"2024-02-19T14:57:00.668478+0000","last_undegraded":"2024-02-19T14:57:00.668478+0000","last_fullsized":"2024-02-19T14:57:00.668478+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:03:38.417380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761364+0000","last_change":"2024-02-19T14:56:59.679793+0000","last_active":"2024-02-19T14:57:00.761364+0000","last_peered":"2024-02-19T14:57:00.761364+0000","last_clean":"2024-02-19T14:57:00.761364+0000","last_became_active":"2024-02-19T14:56:59.677982+0000","last_became_peered":"2024-02-19T14:56:59.677982+0000","last_unstale":"2024-02-19T14:57:00.761364+0000","last_undegraded":"2024-02-19T14:57:00.761364+0000","last_fullsized":"2024-02-19T14:57:00.761364+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:32:15.488805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761179+0000","last_change":"2024-02-19T14:56:58.672366+0000","last_active":"2024-02-19T14:57:00.761179+0000","last_peered":"2024-02-19T14:57:00.761179+0000","last_clean":"2024-02-19T14:57:00.761179+0000","last_became_active":"2024-02-19T14:56:58.672085+0000","last_became_peered":"2024-02-19T14:56:58.672085+0000","last_unstale":"2024-02-19T14:57:00.761179+0000","last_undegraded":"2024-02-19T14:57:00.761179+0000","last_fullsized":"2024-02-19T14:57:00.761179+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:49.753360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760763+0000","last_change":"2024-02-19T14:56:59.681440+0000","last_active":"2024-02-19T14:57:00.760763+0000","last_peered":"2024-02-19T14:57:00.760763+0000","last_clean":"2024-02-19T14:57:00.760763+0000","last_became_active":"2024-02-19T14:56:59.681022+0000","last_became_peered":"2024-02-19T14:56:59.681022+0000","last_unstale":"2024-02-19T14:57:00.760763+0000","last_undegraded":"2024-02-19T14:57:00.760763+0000","last_fullsized":"2024-02-19T14:57:00.760763+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:32:57.697404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669083+0000","last_change":"2024-02-19T14:56:58.670839+0000","last_active":"2024-02-19T14:57:00.669083+0000","last_peered":"2024-02-19T14:57:00.669083+0000","last_clean":"2024-02-19T14:57:00.669083+0000","last_became_active":"2024-02-19T14:56:58.670507+0000","last_became_peered":"2024-02-19T14:56:58.670507+0000","last_unstale":"2024-02-19T14:57:00.669083+0000","last_undegraded":"2024-02-19T14:57:00.669083+0000","last_fullsized":"2024-02-19T14:57:00.669083+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:38:20.112923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669006+0000","last_change":"2024-02-19T14:56:59.678942+0000","last_active":"2024-02-19T14:57:00.669006+0000","last_peered":"2024-02-19T14:57:00.669006+0000","last_clean":"2024-02-19T14:57:00.669006+0000","last_became_active":"2024-02-19T14:56:59.676927+0000","last_became_peered":"2024-02-19T14:56:59.676927+0000","last_unstale":"2024-02-19T14:57:00.669006+0000","last_undegraded":"2024-02-19T14:57:00.669006+0000","last_fullsized":"2024-02-19T14:57:00.669006+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:09.354585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669547+0000","last_change":"2024-02-19T14:56:58.660184+0000","last_active":"2024-02-19T14:57:00.669547+0000","last_peered":"2024-02-19T14:57:00.669547+0000","last_clean":"2024-02-19T14:57:00.669547+0000","last_became_active":"2024-02-19T14:56:58.659918+0000","last_became_peered":"2024-02-19T14:56:58.659918+0000","last_unstale":"2024-02-19T14:57:00.669547+0000","last_undegraded":"2024-02-19T14:57:00.669547+0000","last_fullsized":"2024-02-19T14:57:00.669547+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761435+0000","last_change":"2024-02-19T14:56:59.678477+0000","last_active":"2024-02-19T14:57:00.761435+0000","last_peered":"2024-02-19T14:57:00.761435+0000","last_clean":"2024-02-19T14:57:00.761435+0000","last_became_active":"2024-02-19T14:56:59.678159+0000","last_became_peered":"2024-02-19T14:56:59.678159+0000","last_unstale":"2024-02-19T14:57:00.761435+0000","last_undegraded":"2024-02-19T14:57:00.761435+0000","last_fullsized":"2024-02-19T14:57:00.761435+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:18.891871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761441+0000","last_change":"2024-02-19T14:56:58.667075+0000","last_active":"2024-02-19T14:57:00.761441+0000","last_peered":"2024-02-19T14:57:00.761441+0000","last_clean":"2024-02-19T14:57:00.761441+0000","last_became_active":"2024-02-19T14:56:58.666748+0000","last_became_peered":"2024-02-19T14:56:58.666748+0000","last_unstale":"2024-02-19T14:57:00.761441+0000","last_undegraded":"2024-02-19T14:57:00.761441+0000","last_fullsized":"2024-02-19T14:57:00.761441+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:36:33.368950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668430+0000","last_change":"2024-02-19T14:56:59.680460+0000","last_active":"2024-02-19T14:57:00.668430+0000","last_peered":"2024-02-19T14:57:00.668430+0000","last_clean":"2024-02-19T14:57:00.668430+0000","last_became_active":"2024-02-19T14:56:59.680154+0000","last_became_peered":"2024-02-19T14:56:59.680154+0000","last_unstale":"2024-02-19T14:57:00.668430+0000","last_undegraded":"2024-02-19T14:57:00.668430+0000","last_fullsized":"2024-02-19T14:57:00.668430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:53:50.758440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668947+0000","last_change":"2024-02-19T14:56:58.672776+0000","last_active":"2024-02-19T14:57:00.668947+0000","last_peered":"2024-02-19T14:57:00.668947+0000","last_clean":"2024-02-19T14:57:00.668947+0000","last_became_active":"2024-02-19T14:56:58.672154+0000","last_became_peered":"2024-02-19T14:56:58.672154+0000","last_unstale":"2024-02-19T14:57:00.668947+0000","last_undegraded":"2024-02-19T14:57:00.668947+0000","last_fullsized":"2024-02-19T14:57:00.668947+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:23.204802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761142+0000","last_change":"2024-02-19T14:56:59.681056+0000","last_active":"2024-02-19T14:57:00.761142+0000","last_peered":"2024-02-19T14:57:00.761142+0000","last_clean":"2024-02-19T14:57:00.761142+0000","last_became_active":"2024-02-19T14:56:59.680740+0000","last_became_peered":"2024-02-19T14:56:59.680740+0000","last_unstale":"2024-02-19T14:57:00.761142+0000","last_undegraded":"2024-02-19T14:57:00.761142+0000","last_fullsized":"2024-02-19T14:57:00.761142+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:29:53.596686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760440+0000","last_change":"2024-02-19T14:56:58.667461+0000","last_active":"2024-02-19T14:57:00.760440+0000","last_peered":"2024-02-19T14:57:00.760440+0000","last_clean":"2024-02-19T14:57:00.760440+0000","last_became_active":"2024-02-19T14:56:58.667182+0000","last_became_peered":"2024-02-19T14:56:58.667182+0000","last_unstale":"2024-02-19T14:57:00.760440+0000","last_undegraded":"2024-02-19T14:57:00.760440+0000","last_fullsized":"2024-02-19T14:57:00.760440+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:27:51.843246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669251+0000","last_change":"2024-02-19T14:56:59.673100+0000","last_active":"2024-02-19T14:57:00.669251+0000","last_peered":"2024-02-19T14:57:00.669251+0000","last_clean":"2024-02-19T14:57:00.669251+0000","last_became_active":"2024-02-19T14:56:59.672757+0000","last_became_peered":"2024-02-19T14:56:59.672757+0000","last_unstale":"2024-02-19T14:57:00.669251+0000","last_undegraded":"2024-02-19T14:57:00.669251+0000","last_fullsized":"2024-02-19T14:57:00.669251+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:45:36.585855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.667848+0000","last_change":"2024-02-19T14:56:58.664744+0000","last_active":"2024-02-19T14:57:00.667848+0000","last_peered":"2024-02-19T14:57:00.667848+0000","last_clean":"2024-02-19T14:57:00.667848+0000","last_became_active":"2024-02-19T14:56:58.663422+0000","last_became_peered":"2024-02-19T14:56:58.663422+0000","last_unstale":"2024-02-19T14:57:00.667848+0000","last_undegraded":"2024-02-19T14:57:00.667848+0000","last_fullsized":"2024-02-19T14:57:00.667848+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:51.200078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.668501+0000","last_change":"2024-02-19T14:56:59.675481+0000","last_active":"2024-02-19T14:57:00.668501+0000","last_peered":"2024-02-19T14:57:00.668501+0000","last_clean":"2024-02-19T14:57:00.668501+0000","last_became_active":"2024-02-19T14:56:59.671484+0000","last_became_peered":"2024-02-19T14:56:59.671484+0000","last_unstale":"2024-02-19T14:57:00.668501+0000","last_undegraded":"2024-02-19T14:57:00.668501+0000","last_fullsized":"2024-02-19T14:57:00.668501+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:19:52.205158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760386+0000","last_change":"2024-02-19T14:56:58.673320+0000","last_active":"2024-02-19T14:57:00.760386+0000","last_peered":"2024-02-19T14:57:00.760386+0000","last_clean":"2024-02-19T14:57:00.760386+0000","last_became_active":"2024-02-19T14:56:58.673046+0000","last_became_peered":"2024-02-19T14:56:58.673046+0000","last_unstale":"2024-02-19T14:57:00.760386+0000","last_undegraded":"2024-02-19T14:57:00.760386+0000","last_fullsized":"2024-02-19T14:57:00.760386+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:04.357396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.669280+0000","last_change":"2024-02-19T14:56:59.669865+0000","last_active":"2024-02-19T14:57:00.669280+0000","last_peered":"2024-02-19T14:57:00.669280+0000","last_clean":"2024-02-19T14:57:00.669280+0000","last_became_active":"2024-02-19T14:56:59.669568+0000","last_became_peered":"2024-02-19T14:56:59.669568+0000","last_unstale":"2024-02-19T14:57:00.669280+0000","last_undegraded":"2024-02-19T14:57:00.669280+0000","last_fullsized":"2024-02-19T14:57:00.669280+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:58.645116+0000","last_clean_scrub_stamp":"2024-02-19T14:56:58.645116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:05.809396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"137'6","reported_seq":20,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.761112+0000","last_change":"2024-02-19T14:56:58.668744+0000","last_active":"2024-02-19T14:57:00.761112+0000","last_peered":"2024-02-19T14:57:00.761112+0000","last_clean":"2024-02-19T14:57:00.761112+0000","last_became_active":"2024-02-19T14:56:58.668464+0000","last_became_peered":"2024-02-19T14:56:58.668464+0000","last_unstale":"2024-02-19T14:57:00.761112+0000","last_undegraded":"2024-02-19T14:57:00.761112+0000","last_fullsized":"2024-02-19T14:57:00.761112+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:56:57.640036+0000","last_clean_scrub_stamp":"2024-02-19T14:56:57.640036+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:08.032552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":392,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-02-19T14:57:00.760484+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:00.760484+0000","last_peered":"2024-02-19T14:57:00.760484+0000","last_clean":"2024-02-19T14:57:00.760484+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:00.760484+0000","last_undegraded":"2024-02-19T14:57:00.760484+0000","last_fullsized":"2024-02-19T14:57:00.760484+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2330624,"data_stored":1431121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.503},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.441},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.374}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.35599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":840801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.437}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.39400000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.36399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.379}]}]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1732608,"data_stored":840731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.372},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.436}]}]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1744896,"data_stored":842343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.374},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.39400000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.29799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.28299999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.255},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.315}]}]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1740800,"data_stored":841347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.315}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.45800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.36799999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.47099999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.50600000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.52500000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1740800,"data_stored":840843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.39700000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.33600000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.373},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.498},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.44},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.54400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":840801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.249},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.27800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.31}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.38200000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.435}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.44600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2351104,"data_stored":1433417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.375},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.439},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54000000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.503}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:08.337 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:57:08.338 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:57:08.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.477+0000 7fb260927640 1 Processor -- start 2024-02-19T14:57:08.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.477+0000 7fb260927640 1 -- start start 2024-02-19T14:57:08.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb260927640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb25c0c7280 0x7fb25c0c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb260927640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c1586a0 con 0x7fb25c06c570 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb260927640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c158980 con 0x7fb25c152830 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb260927640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25c0c6550 con 0x7fb25c0c7280 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb253fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb25c0c7280 0x7fb25c0c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb25ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb25c06c570 0x7fb25c06c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44204/0 (socket says 172.21.15.78:44204) 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb25ad9d640 1 -- 172.21.15.78:0/3697467926 learned_addr learned my addr 172.21.15.78:0/3697467926 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb253fff640 1 -- 172.21.15.78:0/3697467926 >> v1:172.21.15.150:6789/0 conn(0x7fb25c152830 legacy=0x7fb25c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb253fff640 1 -- 172.21.15.78:0/3697467926 >> v1:172.21.15.78:6789/0 conn(0x7fb25c06c570 legacy=0x7fb25c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb259d9b640 1 -- 172.21.15.78:0/3697467926 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063396761 0 0) 0x7fb25c1586a0 con 0x7fb25c06c570 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.478+0000 7fb253fff640 1 -- 172.21.15.78:0/3697467926 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c158c60 con 0x7fb25c0c7280 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb253fff640 1 --2- 172.21.15.78:0/3697467926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c0c7280 0x7fb25c0c7700 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fb2440097a0 tx=0x7fb24402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7372619de705d056 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb259d9b640 1 -- 172.21.15.78:0/3697467926 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244031830 con 0x7fb25c0c7280 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb259d9b640 1 -- 172.21.15.78:0/3697467926 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb244031990 con 0x7fb25c0c7280 2024-02-19T14:57:08.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb259d9b640 1 -- 172.21.15.78:0/3697467926 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244031d20 con 0x7fb25c0c7280 2024-02-19T14:57:08.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb260927640 1 -- 172.21.15.78:0/3697467926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c0c7280 msgr2=0x7fb25c0c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb260927640 1 --2- 172.21.15.78:0/3697467926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c0c7280 0x7fb25c0c7700 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fb2440097a0 tx=0x7fb24402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb260927640 1 -- 172.21.15.78:0/3697467926 shutdown_connections 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb260927640 1 --2- 172.21.15.78:0/3697467926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c0c7280 0x7fb25c0c7700 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.479+0000 7fb260927640 1 -- 172.21.15.78:0/3697467926 >> 172.21.15.78:0/3697467926 conn(0x7fb25c0ca8a0 msgr2=0x7fb25c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.480+0000 7fb260927640 1 -- 172.21.15.78:0/3697467926 shutdown_connections 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.480+0000 7fb260927640 1 -- 172.21.15.78:0/3697467926 wait complete. 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 Processor -- start 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 -- start start 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb25c06c570 0x7fb25c0c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb25c0c7280 0x7fb25c0ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 0x7fb25c0c0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c0c7c40 con 0x7fb25c152830 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c159320 con 0x7fb25c06c570 2024-02-19T14:57:08.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb260927640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c0bf770 con 0x7fb25c0c7280 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb25ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 0x7fb25c0c0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.481+0000 7fb25ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 0x7fb25c0c0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51028/0 (socket says 172.21.15.78:51028) 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 -- 172.21.15.78:0/1092231628 learned_addr learned my addr 172.21.15.78:0/1092231628 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb253fff640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb25c0c7280 0x7fb25c0ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25a59c640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb25c06c570 0x7fb25c0c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 -- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb25c0c7280 msgr2=0x7fb25c0ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb25c0c7280 0x7fb25c0ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 -- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb25c06c570 msgr2=0x7fb25c0c2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb25c06c570 0x7fb25c0c2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 -- 172.21.15.78:0/1092231628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244009400 con 0x7fb25c152830 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb25ad9d640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 0x7fb25c0c0bc0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb25400c7a0 tx=0x7fb25400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25400a640 con 0x7fb25c152830 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25400a7a0 con 0x7fb25c152830 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25400abd0 con 0x7fb25c152830 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c0c3340 con 0x7fb25c152830 2024-02-19T14:57:08.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.482+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c0aca90 con 0x7fb25c152830 2024-02-19T14:57:08.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.483+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c0abd70 con 0x7fb25c152830 2024-02-19T14:57:08.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.484+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb25400ad30 con 0x7fb25c152830 2024-02-19T14:57:08.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.484+0000 7fb252ffd640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb23c06b810 0x7fb23c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.484+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb254089aa0 con 0x7fb25c152830 2024-02-19T14:57:08.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.484+0000 7fb25a59c640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb23c06b810 0x7fb23c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.484+0000 7fb25a59c640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb23c06b810 0x7fb23c06dcd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fb24c000f60 tx=0x7fb24c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.489+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb254054470 con 0x7fb25c152830 2024-02-19T14:57:08.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.731+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb25c074990 con 0x7fb25c152830 2024-02-19T14:57:08.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.733+0000 7fb252ffd640 1 -- 172.21.15.78:0/1092231628 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb254018020 con 0x7fb25c152830 2024-02-19T14:57:08.736 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:57:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:57:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:57:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:57:08.736 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:57:08.736 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb23c06b810 msgr2=0x7fb23c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb23c06b810 0x7fb23c06dcd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fb24c000f60 tx=0x7fb24c009040 comp rx=0 tx=0).stop 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 msgr2=0x7fb25c0c0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 0x7fb25c0c0bc0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb25400c7a0 tx=0x7fb25400cc60 comp rx=0 tx=0).stop 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 shutdown_connections 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb23c06b810 0x7fb23c06dcd0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb25c152830 0x7fb25c0c0bc0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb25c0c7280 0x7fb25c0ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 --2- 172.21.15.78:0/1092231628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb25c06c570 0x7fb25c0c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 >> 172.21.15.78:0/1092231628 conn(0x7fb25c0ca8a0 msgr2=0x7fb25c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:08.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 shutdown_connections 2024-02-19T14:57:08.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.735+0000 7fb260927640 1 -- 172.21.15.78:0/1092231628 wait complete. 2024-02-19T14:57:08.749 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:08.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.845+0000 7f568418a640 1 Processor -- start 2024-02-19T14:57:08.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f568418a640 1 -- start start 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f568418a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c0719d0 0x7f567c071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f568418a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f567c0c15b0 0x7f567c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f568418a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0c1120 con 0x7f567c14ece0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f568418a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c0c6080 con 0x7f567c0c15b0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f568418a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0c57e0 con 0x7f567c0719d0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.846+0000 7f5681f26640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f567c14ece0 0x7f567c0c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44214/0 (socket says 172.21.15.78:44214) 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5681f26640 1 -- 172.21.15.78:0/1605549753 learned_addr learned my addr 172.21.15.78:0/1605549753 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5681725640 1 --2- 172.21.15.78:0/1605549753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c0719d0 0x7f567c071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5682727640 1 --2- 172.21.15.78:0/1605549753 >> v2:172.21.15.150:3300/0 conn(0x7f567c0c15b0 0x7f567c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5680f24640 1 -- 172.21.15.78:0/1605549753 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643127263 0 0) 0x7f567c0c1120 con 0x7f567c14ece0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5680f24640 1 -- 172.21.15.78:0/1605549753 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5664003520 con 0x7f567c14ece0 2024-02-19T14:57:08.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5681725640 1 -- 172.21.15.78:0/1605549753 >> v2:172.21.15.150:3300/0 conn(0x7f567c0c15b0 msgr2=0x7f567c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:08.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5681725640 1 --2- 172.21.15.78:0/1605549753 >> v2:172.21.15.150:3300/0 conn(0x7f567c0c15b0 0x7f567c0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5681725640 1 -- 172.21.15.78:0/1605549753 >> v1:172.21.15.78:6789/0 conn(0x7f567c14ece0 legacy=0x7f567c0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5681725640 1 -- 172.21.15.78:0/1605549753 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0c5ac0 con 0x7f567c0719d0 2024-02-19T14:57:08.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.847+0000 7f5682727640 1 --2- 172.21.15.78:0/1605549753 >> v2:172.21.15.150:3300/0 conn(0x7f567c0c15b0 0x7f567c0c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:08.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f5681725640 1 --2- 172.21.15.78:0/1605549753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c0719d0 0x7f567c071e50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f567002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfbe542b7219fca8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f5680f24640 1 -- 172.21.15.78:0/1605549753 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670031830 con 0x7f567c0719d0 2024-02-19T14:57:08.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f5680f24640 1 -- 172.21.15.78:0/1605549753 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670031990 con 0x7f567c0719d0 2024-02-19T14:57:08.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f5680f24640 1 -- 172.21.15.78:0/1605549753 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670031d20 con 0x7f567c0719d0 2024-02-19T14:57:08.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 -- 172.21.15.78:0/1605549753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c0719d0 msgr2=0x7f567c071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 --2- 172.21.15.78:0/1605549753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c0719d0 0x7f567c071e50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f56700097a0 tx=0x7f567002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 -- 172.21.15.78:0/1605549753 shutdown_connections 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 --2- 172.21.15.78:0/1605549753 >> v2:172.21.15.150:3300/0 conn(0x7f567c0c15b0 0x7f567c0c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 --2- 172.21.15.78:0/1605549753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c0719d0 0x7f567c071e50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 -- 172.21.15.78:0/1605549753 >> 172.21.15.78:0/1605549753 conn(0x7f567c06b270 msgr2=0x7f567c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 -- 172.21.15.78:0/1605549753 shutdown_connections 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.848+0000 7f568418a640 1 -- 172.21.15.78:0/1605549753 wait complete. 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.849+0000 7f568418a640 1 Processor -- start 2024-02-19T14:57:08.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 -- start start 2024-02-19T14:57:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f567c0719d0 0x7f567c1ead10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 0x7f567c1dc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c14ece0 0x7f567c1dc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c147af0 con 0x7f567c0c15b0 2024-02-19T14:57:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c1472e0 con 0x7f567c0719d0 2024-02-19T14:57:08.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f568418a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c1eb630 con 0x7f567c14ece0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f5681725640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 0x7f567c1dc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f5681725640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 0x7f567c1dc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51044/0 (socket says 172.21.15.78:51044) 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f5681725640 1 -- 172.21.15.78:0/4041755539 learned_addr learned my addr 172.21.15.78:0/4041755539 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f5681f26640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f567c0719d0 0x7f567c1ead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.850+0000 7f5682727640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c14ece0 0x7f567c1dc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f5681725640 1 -- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c14ece0 msgr2=0x7f567c1dc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f5681725640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c14ece0 0x7f567c1dc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f5681725640 1 -- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f567c0719d0 msgr2=0x7f567c1ead10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f5681725640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f567c0719d0 0x7f567c1ead10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f5681725640 1 -- 172.21.15.78:0/4041755539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c009400 con 0x7f567c0c15b0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f5681725640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 0x7f567c1dc040 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f56700098c0 tx=0x7f56700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670040070 con 0x7f567c0c15b0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567003b5c0 con 0x7f567c0c15b0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567003b970 con 0x7f567c0c15b0 2024-02-19T14:57:08.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5670009400 con 0x7f567c0c15b0 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.851+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c1e8290 con 0x7f567c0c15b0 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.852+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c1eb850 con 0x7f567c0c15b0 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.852+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5670044070 con 0x7f567c0c15b0 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.853+0000 7f567a7fc640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f566806b810 0x7f566806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.853+0000 7f5681f26640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f566806b810 0x7f566806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.853+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f56700ad9d0 con 0x7f567c0c15b0 2024-02-19T14:57:08.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.853+0000 7f5681f26640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f566806b810 0x7f566806dcd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5674005c00 tx=0x7f5674009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:08.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:08.857+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f567c1eb850 con 0x7f567c0c15b0 2024-02-19T14:57:09.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.066+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567c1eb850 con 0x7f567c0c15b0 2024-02-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.066+0000 7f567a7fc640 1 -- 172.21.15.78:0/4041755539 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f567c1eb850 con 0x7f567c0c15b0 2024-02-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:09.069 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7351,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:01.936470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8268":{"gid":8268,"name":"b","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6834/2425259283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2425259283},{"type":"v1","addr":"172.21.15.150:6834","nonce":2425259283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:09.070 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 117 2024-02-19T14:57:09.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f566806b810 msgr2=0x7f566806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f566806b810 0x7f566806dcd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5674005c00 tx=0x7f5674009040 comp rx=0 tx=0).stop 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 msgr2=0x7f567c1dc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 0x7f567c1dc040 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f56700098c0 tx=0x7f56700046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 shutdown_connections 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f566806b810 0x7f566806dcd0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f567c0c15b0 0x7f567c1dc040 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f567c14ece0 0x7f567c1dc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 --2- 172.21.15.78:0/4041755539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f567c0719d0 0x7f567c1ead10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 >> 172.21.15.78:0/4041755539 conn(0x7f567c06b270 msgr2=0x7f567c1544d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 shutdown_connections 2024-02-19T14:57:09.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.069+0000 7f568418a640 1 -- 172.21.15.78:0/4041755539 wait complete. 2024-02-19T14:57:09.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:01.936470+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8268': {'gid': 8268, 'name': 'b', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6834/2425259283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2425259283}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 2425259283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:09.485 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:57:09.485 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-19T14:57:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.584+0000 7fe948223640 1 Processor -- start 2024-02-19T14:57:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe948223640 1 -- start start 2024-02-19T14:57:09.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe948223640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe940152700 0x7fe9400c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe948223640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe94006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe948223640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9401551f0 con 0x7fe9400c8e00 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe948223640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe940155490 con 0x7fe940152700 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe948223640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940155770 con 0x7fe9400c5730 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.585+0000 7fe945fbf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe940152700 0x7fe9400c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe9467c0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe9400c8e00 0x7fe940158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44224/0 (socket says 172.21.15.78:44224) 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe9467c0640 1 -- 172.21.15.78:0/3579383543 learned_addr learned my addr 172.21.15.78:0/3579383543 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe9457be640 1 --2- 172.21.15.78:0/3579383543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe94006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe945fbf640 1 -- 172.21.15.78:0/3579383543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 msgr2=0x7fe94006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe945fbf640 1 --2- 172.21.15.78:0/3579383543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe94006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe945fbf640 1 -- 172.21.15.78:0/3579383543 >> v1:172.21.15.78:6789/0 conn(0x7fe9400c8e00 legacy=0x7fe940158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe945fbf640 1 -- 172.21.15.78:0/3579383543 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940155a50 con 0x7fe940152700 2024-02-19T14:57:09.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.586+0000 7fe944fbd640 1 -- 172.21.15.78:0/3579383543 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179648336 0 0) 0x7fe9401551f0 con 0x7fe9400c8e00 2024-02-19T14:57:09.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe945fbf640 1 --2- 172.21.15.78:0/3579383543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe940152700 0x7fe9400c51f0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fe9300097a0 tx=0x7fe93002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7669fa30773a6e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe944fbd640 1 -- 172.21.15.78:0/3579383543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930031830 con 0x7fe940152700 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe944fbd640 1 -- 172.21.15.78:0/3579383543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe930031990 con 0x7fe940152700 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe944fbd640 1 -- 172.21.15.78:0/3579383543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930031d20 con 0x7fe940152700 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 -- 172.21.15.78:0/3579383543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe940152700 msgr2=0x7fe9400c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 --2- 172.21.15.78:0/3579383543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe940152700 0x7fe9400c51f0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fe9300097a0 tx=0x7fe93002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 -- 172.21.15.78:0/3579383543 shutdown_connections 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 --2- 172.21.15.78:0/3579383543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe94006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 --2- 172.21.15.78:0/3579383543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe940152700 0x7fe9400c51f0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 -- 172.21.15.78:0/3579383543 >> 172.21.15.78:0/3579383543 conn(0x7fe9400cac90 msgr2=0x7fe9400cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 -- 172.21.15.78:0/3579383543 shutdown_connections 2024-02-19T14:57:09.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.587+0000 7fe948223640 1 -- 172.21.15.78:0/3579383543 wait complete. 2024-02-19T14:57:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.588+0000 7fe948223640 1 Processor -- start 2024-02-19T14:57:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.588+0000 7fe948223640 1 -- start start 2024-02-19T14:57:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe948223640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe9400bef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe948223640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 0x7fe9400ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe948223640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe940152700 0x7fe9401edd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe948223640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400be950 con 0x7fe9400c8e00 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe948223640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400ad100 con 0x7fe940152700 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe948223640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9400bf470 con 0x7fe9400c5730 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 0x7fe9400ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 0x7fe9400ab560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51060/0 (socket says 172.21.15.78:51060) 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 -- 172.21.15.78:0/2053973685 learned_addr learned my addr 172.21.15.78:0/2053973685 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9467c0640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe940152700 0x7fe9401edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe945fbf640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe9400bef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 -- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 msgr2=0x7fe9400bef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe9400bef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 -- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe940152700 msgr2=0x7fe9401edd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe940152700 0x7fe9401edd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.589+0000 7fe9457be640 1 -- 172.21.15.78:0/2053973685 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930009400 con 0x7fe9400c8e00 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.590+0000 7fe9457be640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 0x7fe9400ab560 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00e9b0 tx=0x7fe93c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.590+0000 7fe9367fc640 1 -- 172.21.15.78:0/2053973685 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c009670 con 0x7fe9400c8e00 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.590+0000 7fe9367fc640 1 -- 172.21.15.78:0/2053973685 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c00c9a0 con 0x7fe9400c8e00 2024-02-19T14:57:09.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.590+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9400abaa0 con 0x7fe9400c8e00 2024-02-19T14:57:09.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.590+0000 7fe9367fc640 1 -- 172.21.15.78:0/2053973685 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00cd50 con 0x7fe9400c8e00 2024-02-19T14:57:09.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.590+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9400aad60 con 0x7fe9400c8e00 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.591+0000 7fe9367fc640 1 -- 172.21.15.78:0/2053973685 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe93c013470 con 0x7fe9400c8e00 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.592+0000 7fe9367fc640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe92406b810 0x7fe92406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.592+0000 7fe9367fc640 1 -- 172.21.15.78:0/2053973685 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe93c08a4d0 con 0x7fe9400c8e00 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.592+0000 7fe945fbf640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe92406b810 0x7fe92406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.592+0000 7fe945fbf640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe92406b810 0x7fe92406dcd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe930036000 tx=0x7fe93003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 Processor -- start 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 -- start start 2024-02-19T14:57:09.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a33748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3339ae0 0x55c2a3373b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 0x55c2a3338860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a32f3cd0 con 0x55c2a33768b0 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a32a6f20 con 0x55c2a3339ae0 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2a32a4450 con 0x55c2a3373740 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.593+0000 7fe9467c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 0x55c2a3338860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 0x55c2a3338860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51070/0 (socket says 172.21.15.78:51070) 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 -- 172.21.15.78:0/3727425820 learned_addr learned my addr 172.21.15.78:0/3727425820 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9457be640 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3339ae0 0x55c2a3373b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe945fbf640 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a33748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 -- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 msgr2=0x55c2a33748b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a33748b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 -- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3339ae0 msgr2=0x55c2a3373b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3339ae0 0x55c2a3373b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 -- 172.21.15.78:0/3727425820 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2a3254f20 con 0x55c2a33768b0 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe945fbf640 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a33748b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9467c0640 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 0x55c2a3338860 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fe938001b20 tx=0x7fe938001fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f15d3f087fea15 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9237fe640 1 -- 172.21.15.78:0/3727425820 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe938007850 con 0x55c2a33768b0 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9237fe640 1 -- 172.21.15.78:0/3727425820 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9380079b0 con 0x55c2a33768b0 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.594+0000 7fe9237fe640 1 -- 172.21.15.78:0/3727425820 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9380135f0 con 0x55c2a33768b0 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 -- 172.21.15.78:0/3727425820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 msgr2=0x55c2a3338860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 0x55c2a3338860 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fe938001b20 tx=0x7fe938001fe0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 -- 172.21.15.78:0/3727425820 shutdown_connections 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a33768b0 0x55c2a3338860 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3339ae0 0x55c2a3373b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 --2- 172.21.15.78:0/3727425820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a33748b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 -- 172.21.15.78:0/3727425820 >> 172.21.15.78:0/3727425820 conn(0x55c2a3335120 msgr2=0x55c2a3359b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 -- 172.21.15.78:0/3727425820 shutdown_connections 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 -- 172.21.15.78:0/3727425820 wait complete. 2024-02-19T14:57:09.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 -- start start 2024-02-19T14:57:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.595+0000 7fe94ae04740 1 Processor -- start 2024-02-19T14:57:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a336de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3371450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a334b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a329fc30 con 0x55c2a3338a50 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a3244d30 con 0x55c2a33768b0 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2a2f73310 con 0x55c2a3373740 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe94ae04740 10 client.0 _ll_register_callbacks cb 0x55c2a324cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a334b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a334b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51074/0 (socket says 172.21.15.78:51074) 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 -- 172.21.15.78:0/109132700 learned_addr learned my addr 172.21.15.78:0/109132700 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe945fbf640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a336de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9457be640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3371450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 -- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 msgr2=0x55c2a336de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a336de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 -- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 msgr2=0x55c2a3371450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3371450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.596+0000 7fe9467c0640 1 -- 172.21.15.78:0/109132700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe938004df0 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe9467c0640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a334b7e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe938001560 tx=0x7fe938001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d90ab53738606a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe920ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe920ff9640 1 -- 172.21.15.78:0/109132700 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93801d400 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe920ff9640 1 -- 172.21.15.78:0/109132700 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe93801d560 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2a2f9f6c0 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe920ff9640 1 -- 172.21.15.78:0/109132700 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe93801d720 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 10 client.8439 fetch_fsmap learned FSMap version 117 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55c2a329e2b0 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe920ff9640 1 -- 172.21.15.78:0/109132700 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c2a2f9f6c0 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe920ff9640 1 -- 172.21.15.78:0/109132700 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 117) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe938012070 con 0x55c2a3338a50 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 10 client.8439 fetch_fsmap finished waiting for FSMap version 117 2024-02-19T14:57:09.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 10 client.8439 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 20 client.8439 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33882a0 0x55c2a3388680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 4 client.8439 mds_command: new command op to 8268 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 --> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2a2ff0270 con 0x55c2a33882a0 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.597+0000 7fe945fbf640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33882a0 0x55c2a3388680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.598+0000 7fe920ff9640 10 client.8439 ms_handle_connect on v2:172.21.15.150:6832/2425259283 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.598+0000 7fe945fbf640 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33882a0 0x55c2a3388680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f985d73d9e6173 server_cookie=95fbc93bb2c444bc in_seq=0 out_seq=0 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe920ff9640 1 -- 172.21.15.78:0/109132700 <== mds.0 v2:172.21.15.150:6832/2425259283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2a2ff0270 con 0x55c2a33882a0 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe920ff9640 10 client.8439 handle_command_reply: tid=0 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 client.8439 shutdown 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 msgr2=0x55c2a334b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a334b7e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe938001560 tx=0x7fe938001250 comp rx=0 tx=0).stop 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 shutdown_connections 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33882a0 0x55c2a3388680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a334b7e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3371450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 --2- 172.21.15.78:0/109132700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a336de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 >> 172.21.15.78:0/109132700 conn(0x55c2a3339ae0 msgr2=0x55c2a3331f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe920ff9640 0 client.8439 ms_handle_reset on v2:172.21.15.150:6832/2425259283 2024-02-19T14:57:09.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 shutdown_connections 2024-02-19T14:57:09.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 1 -- 172.21.15.78:0/109132700 wait complete. 2024-02-19T14:57:09.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.599+0000 7fe94ae04740 20 client.8439 trim_cache size 0 max 16384 2024-02-19T14:57:09.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 Processor -- start 2024-02-19T14:57:09.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 -- start start 2024-02-19T14:57:09.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3338a50 0x55c2a3336ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 0x55c2a3374e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a33768b0 0x55c2a33881a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a3225490 con 0x55c2a3373740 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a3337f90 con 0x55c2a3338a50 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2a332dbe0 con 0x55c2a33768b0 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe9457be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 0x55c2a3374e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe9457be640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 0x55c2a3374e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51090/0 (socket says 172.21.15.78:51090) 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe9457be640 1 -- 172.21.15.78:0/407825572 learned_addr learned my addr 172.21.15.78:0/407825572 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe945fbf640 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3338a50 0x55c2a3336ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.616+0000 7fe9467c0640 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a33768b0 0x55c2a33881a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe9457be640 1 -- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a33768b0 msgr2=0x55c2a33881a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe9457be640 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a33768b0 0x55c2a33881a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe9457be640 1 -- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3338a50 msgr2=0x55c2a3336ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe9457be640 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3338a50 0x55c2a3336ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe9457be640 1 -- 172.21.15.78:0/407825572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2a323aa20 con 0x55c2a3373740 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe9457be640 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 0x55c2a3374e40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe93c058870 tx=0x7fe93c08a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5b6c1758beadb4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe922ffd640 1 -- 172.21.15.78:0/407825572 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c056c10 con 0x55c2a3373740 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe922ffd640 1 -- 172.21.15.78:0/407825572 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c056e70 con 0x55c2a3373740 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe922ffd640 1 -- 172.21.15.78:0/407825572 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c060d40 con 0x55c2a3373740 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 -- 172.21.15.78:0/407825572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 msgr2=0x55c2a3374e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 0x55c2a3374e40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe93c058870 tx=0x7fe93c08a040 comp rx=0 tx=0).stop 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 -- 172.21.15.78:0/407825572 shutdown_connections 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3373740 0x55c2a3374e40 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a33768b0 0x55c2a33881a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 --2- 172.21.15.78:0/407825572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a3338a50 0x55c2a3336ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 -- 172.21.15.78:0/407825572 >> 172.21.15.78:0/407825572 conn(0x55c2a3339ae0 msgr2=0x55c2a3381dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 -- 172.21.15.78:0/407825572 shutdown_connections 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.617+0000 7fe94ae04740 1 -- 172.21.15.78:0/407825572 wait complete. 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.618+0000 7fe94ae04740 1 -- start start 2024-02-19T14:57:09.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.618+0000 7fe94ae04740 1 Processor -- start 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a3348de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a3371450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3388660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a3384d20 con 0x55c2a3338a50 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2a336f330 con 0x55c2a33768b0 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2a3336510 con 0x55c2a3373740 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 10 client.0 _ll_register_callbacks cb 0x7fe93c009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:09.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a3348de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a3348de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51092/0 (socket says 172.21.15.78:51092) 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 -- 172.21.15.78:0/2093428061 learned_addr learned my addr 172.21.15.78:0/2093428061 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe9457be640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a3371450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe9467c0640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3388660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 -- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 msgr2=0x55c2a3371450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a3371450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 -- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 msgr2=0x55c2a3388660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3388660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 -- 172.21.15.78:0/2093428061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9400aad60 con 0x55c2a3338a50 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe945fbf640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a3348de0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe9300098c0 tx=0x7fe930004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988e16084e5b1ffc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe920ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe920ff9640 1 -- 172.21.15.78:0/2093428061 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930040040 con 0x55c2a3338a50 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe920ff9640 1 -- 172.21.15.78:0/2093428061 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe930057980 con 0x55c2a3338a50 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.619+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9380079b0 con 0x55c2a3338a50 2024-02-19T14:57:09.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe920ff9640 1 -- 172.21.15.78:0/2093428061 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe930057b40 con 0x55c2a3338a50 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 10 client.8445 fetch_fsmap learned FSMap version 117 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55c2a3260780 con 0x55c2a3338a50 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe920ff9640 1 -- 172.21.15.78:0/2093428061 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9380079b0 con 0x55c2a3338a50 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe920ff9640 1 -- 172.21.15.78:0/2093428061 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 117) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe93005d020 con 0x55c2a3338a50 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 10 client.8445 fetch_fsmap finished waiting for FSMap version 117 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 10 client.8445 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 20 client.8445 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33e1970 0x55c2a3397580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 4 client.8445 mds_command: new command op to 8268 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 --> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c2a2fdc610 con 0x55c2a33e1970 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe9457be640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33e1970 0x55c2a3397580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe920ff9640 10 client.8445 ms_handle_connect on v2:172.21.15.150:6832/2425259283 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.620+0000 7fe9457be640 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33e1970 0x55c2a3397580 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=980db46af74b4f71 server_cookie=2b5cd045076832b9 in_seq=0 out_seq=0 2024-02-19T14:57:09.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe920ff9640 1 -- 172.21.15.78:0/2093428061 <== mds.0 v2:172.21.15.150:6832/2425259283 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55c2a2fdc610 con 0x55c2a33e1970 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe920ff9640 10 client.8445 handle_command_reply: tid=0 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 client.8445 shutdown 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 msgr2=0x55c2a3348de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a3348de0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe9300098c0 tx=0x7fe930004730 comp rx=0 tx=0).stop 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 shutdown_connections 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:6832/2425259283,v1:172.21.15.150:6834/2425259283] conn(0x55c2a33e1970 0x55c2a3397580 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c2a33768b0 0x55c2a3388660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe920ff9640 0 client.8445 ms_handle_reset on v2:172.21.15.150:6832/2425259283 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c2a3373740 0x55c2a3371450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 --2- 172.21.15.78:0/2093428061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2a3338a50 0x55c2a3348de0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 >> 172.21.15.78:0/2093428061 conn(0x55c2a3339ae0 msgr2=0x55c2a3374df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 shutdown_connections 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 1 -- 172.21.15.78:0/2093428061 wait complete. 2024-02-19T14:57:09.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe94ae04740 20 client.8445 trim_cache size 0 max 16384 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 8268, 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 9.9689738719999994, 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 117, 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 138, 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 138, 2024-02-19T14:57:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 15.048472253 2024-02-19T14:57:09.631 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe92406b810 msgr2=0x7fe92406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe92406b810 0x7fe92406dcd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe930036000 tx=0x7fe93003d040 comp rx=0 tx=0).stop 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 msgr2=0x7fe9400ab560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 0x7fe9400ab560 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00e9b0 tx=0x7fe93c00ee70 comp rx=0 tx=0).stop 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 shutdown_connections 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe92406b810 0x7fe92406dcd0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9400c8e00 0x7fe9400ab560 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe940152700 0x7fe9401edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 --2- 172.21.15.78:0/2053973685 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9400c5730 0x7fe9400bef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 >> 172.21.15.78:0/2053973685 conn(0x7fe9400cac90 msgr2=0x7fe9400cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.621+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 shutdown_connections 2024-02-19T14:57:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.622+0000 7fe948223640 1 -- 172.21.15.78:0/2053973685 wait complete. 2024-02-19T14:57:09.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.732+0000 7efd8a840640 1 Processor -- start 2024-02-19T14:57:09.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.732+0000 7efd8a840640 1 -- start start 2024-02-19T14:57:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd8a840640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efd8406d550 0x7efd840c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd8a840640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd840c6af0 con 0x7efd841588a0 2024-02-19T14:57:09.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd8a840640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd840c7940 con 0x7efd84155f90 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd8a840640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd840c7270 con 0x7efd8406d550 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd88ddd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efd841588a0 0x7efd84158ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44236/0 (socket says 172.21.15.78:44236) 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd88ddd640 1 -- 172.21.15.78:0/2492262967 learned_addr learned my addr 172.21.15.78:0/2492262967 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd837fe640 1 --2- 172.21.15.78:0/2492262967 >> v2:172.21.15.150:3300/0 conn(0x7efd8406d550 0x7efd840c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd82ffd640 1 -- 172.21.15.78:0/2492262967 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625900459 0 0) 0x7efd840c6af0 con 0x7efd841588a0 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd82ffd640 1 -- 172.21.15.78:0/2492262967 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd6c003520 con 0x7efd841588a0 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd837fe640 1 -- 172.21.15.78:0/2492262967 >> v1:172.21.15.150:6789/0 conn(0x7efd84155f90 legacy=0x7efd840c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd837fe640 1 -- 172.21.15.78:0/2492262967 >> v1:172.21.15.78:6789/0 conn(0x7efd841588a0 legacy=0x7efd84158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.733+0000 7efd837fe640 1 -- 172.21.15.78:0/2492262967 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd840c7c20 con 0x7efd8406d550 2024-02-19T14:57:09.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.734+0000 7efd837fe640 1 --2- 172.21.15.78:0/2492262967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd840c6d30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efd780097a0 tx=0x7efd7802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cc940caee565383 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.734+0000 7efd82ffd640 1 -- 172.21.15.78:0/2492262967 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78031830 con 0x7efd8406d550 2024-02-19T14:57:09.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.734+0000 7efd82ffd640 1 -- 172.21.15.78:0/2492262967 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78031990 con 0x7efd8406d550 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.734+0000 7efd82ffd640 1 -- 172.21.15.78:0/2492262967 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78031d20 con 0x7efd8406d550 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 -- 172.21.15.78:0/2492262967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 msgr2=0x7efd840c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 --2- 172.21.15.78:0/2492262967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd840c6d30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efd780097a0 tx=0x7efd7802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 -- 172.21.15.78:0/2492262967 shutdown_connections 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 --2- 172.21.15.78:0/2492262967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd840c6d30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 -- 172.21.15.78:0/2492262967 >> 172.21.15.78:0/2492262967 conn(0x7efd8406b700 msgr2=0x7efd840aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 -- 172.21.15.78:0/2492262967 shutdown_connections 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.735+0000 7efd8a840640 1 -- 172.21.15.78:0/2492262967 wait complete. 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.736+0000 7efd8a840640 1 Processor -- start 2024-02-19T14:57:09.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.736+0000 7efd8a840640 1 -- start start 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd8a840640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd841eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd8a840640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 0x7efd841edd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd8a840640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd841588a0 0x7efd840ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd8a840640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840ac7a0 con 0x7efd84155f90 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd8a840640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840ad770 con 0x7efd8406d550 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd8a840640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd841484d0 con 0x7efd841588a0 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 0x7efd841edd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 0x7efd841edd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51100/0 (socket says 172.21.15.78:51100) 2024-02-19T14:57:09.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 -- 172.21.15.78:0/3952883101 learned_addr learned my addr 172.21.15.78:0/3952883101 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd83fff640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd841eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd88ddd640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd841588a0 0x7efd840ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 -- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd841588a0 msgr2=0x7efd840ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd841588a0 0x7efd840ab660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 -- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 msgr2=0x7efd841eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd841eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 -- 172.21.15.78:0/3952883101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78009400 con 0x7efd84155f90 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd83fff640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd841eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.737+0000 7efd837fe640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 0x7efd841edd60 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7efd78000c00 tx=0x7efd780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.738+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78004230 con 0x7efd84155f90 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.738+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78004390 con 0x7efd84155f90 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.738+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7803a880 con 0x7efd84155f90 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.738+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8406f630 con 0x7efd84155f90 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.738+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd841eb720 con 0x7efd84155f90 2024-02-19T14:57:09.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.739+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd840c5fd0 con 0x7efd84155f90 2024-02-19T14:57:09.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.739+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efd7803a9e0 con 0x7efd84155f90 2024-02-19T14:57:09.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.739+0000 7efd80ff9640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd7006b810 0x7efd7006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:09.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.740+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efd780ac910 con 0x7efd84155f90 2024-02-19T14:57:09.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.740+0000 7efd83fff640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd7006b810 0x7efd7006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:09.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.740+0000 7efd83fff640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd7006b810 0x7efd7006dcd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efd74001b70 tx=0x7efd74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:09.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.744+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efd78077370 con 0x7efd84155f90 2024-02-19T14:57:09.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.950+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd84074990 con 0x7efd84155f90 2024-02-19T14:57:09.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.950+0000 7efd80ff9640 1 -- 172.21.15.78:0/3952883101 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+4954 (secure 0 0 0) 0x7efd78035020 con 0x7efd84155f90 2024-02-19T14:57:09.954 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:09.954 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":117,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7351,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:01.936470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8268":{"gid":8268,"name":"b","rank":0,"incarnation":114,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6834/2425259283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2425259283},{"type":"v1","addr":"172.21.15.150:6834","nonce":2425259283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:09.954 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 117 2024-02-19T14:57:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd7006b810 msgr2=0x7efd7006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd7006b810 0x7efd7006dcd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efd74001b70 tx=0x7efd74009040 comp rx=0 tx=0).stop 2024-02-19T14:57:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 msgr2=0x7efd841edd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 0x7efd841edd60 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7efd78000c00 tx=0x7efd780046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 shutdown_connections 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd7006b810 0x7efd7006dcd0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd84155f90 0x7efd841edd60 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd841588a0 0x7efd840ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 --2- 172.21.15.78:0/3952883101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd8406d550 0x7efd841eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.952+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 >> 172.21.15.78:0/3952883101 conn(0x7efd8406b700 msgr2=0x7efd841535f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.953+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 shutdown_connections 2024-02-19T14:57:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:09.953+0000 7efd8a840640 1 -- 172.21.15.78:0/3952883101 wait complete. 2024-02-19T14:57:09.967 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:59.000185+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8268": { "addr": "172.21.15.150:6834/2425259283", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2425259283, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 2425259283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8268, "incarnation": 114, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-19T14:57:01.936470+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8268 } } } ], "standbys": [ { "addr": "172.21.15.150:6835/1869543632", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1869543632, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1869543632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7351, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1973988605", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1973988605, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1973988605, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8259, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3197225278", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3197225278, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3197225278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:57:09.967 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:57:09.967 INFO:tasks.cephfs.test_failover:current = [{'gid': 8262, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3197225278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3197225278}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3197225278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}, {'gid': 8268, 'name': 'b', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6834/2425259283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 2425259283}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 2425259283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8259, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6836/1973988605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1973988605}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1973988605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}, {'gid': 7351, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}] 2024-02-19T14:57:09.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-19T14:57:10.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.068+0000 7ff51749a640 1 Processor -- start 2024-02-19T14:57:10.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.068+0000 7ff51749a640 1 -- start start 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff51749a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff510150320 0x7ff510152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff51749a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff510071850 0x7ff5101534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff51749a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff510156e90 con 0x7ff510153a20 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff51749a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100c2fc0 con 0x7ff510150320 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff51749a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff510157150 con 0x7ff510071850 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff510150320 0x7ff510152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff510150320 0x7ff510152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51102/0 (socket says 172.21.15.78:51102) 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 -- 172.21.15.78:0/1877995375 learned_addr learned my addr 172.21.15.78:0/1877995375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff514a35640 1 --2- 172.21.15.78:0/1877995375 >> v2:172.21.15.150:3300/0 conn(0x7ff510071850 0x7ff5101534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 -- 172.21.15.78:0/1877995375 >> v2:172.21.15.150:3300/0 conn(0x7ff510071850 msgr2=0x7ff5101534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 --2- 172.21.15.78:0/1877995375 >> v2:172.21.15.150:3300/0 conn(0x7ff510071850 0x7ff5101534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 -- 172.21.15.78:0/1877995375 >> v1:172.21.15.150:6789/0 conn(0x7ff510153a20 legacy=0x7ff5100c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.069+0000 7ff515236640 1 -- 172.21.15.78:0/1877995375 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100c3280 con 0x7ff510150320 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.070+0000 7ff515236640 1 --2- 172.21.15.78:0/1877995375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510150320 0x7ff510152fa0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff5100c6ab0 tx=0x7ff50c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f17690c17863512 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.070+0000 7ff507fff640 1 -- 172.21.15.78:0/1877995375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c02cb20 con 0x7ff510150320 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.070+0000 7ff507fff640 1 -- 172.21.15.78:0/1877995375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c02cc80 con 0x7ff510150320 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.070+0000 7ff507fff640 1 -- 172.21.15.78:0/1877995375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c0343f0 con 0x7ff510150320 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 -- 172.21.15.78:0/1877995375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510150320 msgr2=0x7ff510152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 --2- 172.21.15.78:0/1877995375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510150320 0x7ff510152fa0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff5100c6ab0 tx=0x7ff50c02bf80 comp rx=0 tx=0).stop 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 -- 172.21.15.78:0/1877995375 shutdown_connections 2024-02-19T14:57:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 --2- 172.21.15.78:0/1877995375 >> v2:172.21.15.150:3300/0 conn(0x7ff510071850 0x7ff5101534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 --2- 172.21.15.78:0/1877995375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510150320 0x7ff510152fa0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 -- 172.21.15.78:0/1877995375 >> 172.21.15.78:0/1877995375 conn(0x7ff5100aa370 msgr2=0x7ff5100aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 -- 172.21.15.78:0/1877995375 shutdown_connections 2024-02-19T14:57:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.071+0000 7ff51749a640 1 -- 172.21.15.78:0/1877995375 wait complete. 2024-02-19T14:57:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.072+0000 7ff51749a640 1 Processor -- start 2024-02-19T14:57:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.072+0000 7ff51749a640 1 -- start start 2024-02-19T14:57:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 0x7ff51014b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff510150320 0x7ff510151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff510153a20 0x7ff510146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510151c30 con 0x7ff510071850 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5101eca00 con 0x7ff510150320 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5101ebc40 con 0x7ff510153a20 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 0x7ff51014b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 0x7ff51014b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51116/0 (socket says 172.21.15.78:51116) 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 -- 172.21.15.78:0/2449938000 learned_addr learned my addr 172.21.15.78:0/2449938000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515a37640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff510153a20 0x7ff510146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff514a35640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff510150320 0x7ff510151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 -- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff510153a20 msgr2=0x7ff510146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff510153a20 0x7ff510146af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 -- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff510150320 msgr2=0x7ff510151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff510150320 0x7ff510151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 -- 172.21.15.78:0/2449938000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c009b30 con 0x7ff510071850 2024-02-19T14:57:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff515236640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 0x7ff51014b920 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff50c000f80 tx=0x7ff50c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c009510 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c007e60 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5101ebf20 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.073+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c03e690 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.074+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5101ec4e0 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.075+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5101ec700 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.075+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff50c03e7f0 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.075+0000 7ff505ffb640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4f406b810 0x7ff4f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.075+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff50c0acc70 con 0x7ff510071850 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.075+0000 7ff514a35640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4f406b810 0x7ff4f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.076+0000 7ff514a35640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4f406b810 0x7ff4f406dcd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff5100c66f0 tx=0x7ff500009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.080+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff50c077640 con 0x7ff510071850 2024-02-19T14:57:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.250+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7ff5100680b0 con 0x7ff510071850 2024-02-19T14:57:10.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.253+0000 7ff505ffb640 1 -- 172.21.15.78:0/2449938000 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v46)=0 v46) v1 ==== 116+0+0 (secure 0 0 0) 0x7ff50c07ad00 con 0x7ff510071850 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4f406b810 msgr2=0x7ff4f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4f406b810 0x7ff4f406dcd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff5100c66f0 tx=0x7ff500009190 comp rx=0 tx=0).stop 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 msgr2=0x7ff51014b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 0x7ff51014b920 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff50c000f80 tx=0x7ff50c004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 shutdown_connections 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff4f406b810 0x7ff4f406dcd0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff510153a20 0x7ff510146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff510150320 0x7ff510151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 --2- 172.21.15.78:0/2449938000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff510071850 0x7ff51014b920 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 >> 172.21.15.78:0/2449938000 conn(0x7ff5100aa370 msgr2=0x7ff5101565e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 shutdown_connections 2024-02-19T14:57:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.255+0000 7ff51749a640 1 -- 172.21.15.78:0/2449938000 wait complete. 2024-02-19T14:57:10.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-02-19T14:57:10.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.368+0000 7f410e993640 1 Processor -- start 2024-02-19T14:57:10.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.368+0000 7f410e993640 1 -- start start 2024-02-19T14:57:10.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f410e993640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4108155f90 0x7f410806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f410e993640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41081588a0 0x7f4108158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f410e993640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41080c6af0 con 0x7f41080c6d30 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f410e993640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41080c7940 con 0x7f4108155f90 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f410e993640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41080c7c20 con 0x7f41081588a0 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4108155f90 0x7f410806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4108155f90 0x7f410806d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51122/0 (socket says 172.21.15.78:51122) 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 -- 172.21.15.78:0/3377242649 learned_addr learned my addr 172.21.15.78:0/3377242649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f410cf30640 1 --2- 172.21.15.78:0/3377242649 >> v2:172.21.15.150:3300/0 conn(0x7f41081588a0 0x7f4108158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 -- 172.21.15.78:0/3377242649 >> v2:172.21.15.150:3300/0 conn(0x7f41081588a0 msgr2=0x7f4108158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 --2- 172.21.15.78:0/3377242649 >> v2:172.21.15.150:3300/0 conn(0x7f41081588a0 0x7f4108158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 -- 172.21.15.78:0/3377242649 >> v1:172.21.15.150:6789/0 conn(0x7f41080c6d30 legacy=0x7f41080c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.369+0000 7f4107fff640 1 -- 172.21.15.78:0/3377242649 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41080c7f00 con 0x7f4108155f90 2024-02-19T14:57:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.370+0000 7f4107fff640 1 --2- 172.21.15.78:0/3377242649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410806d550 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f41080c6560 tx=0x7f40f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de43f07f03bb0a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.370+0000 7f4106ffd640 1 -- 172.21.15.78:0/3377242649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8031870 con 0x7f4108155f90 2024-02-19T14:57:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.370+0000 7f4106ffd640 1 -- 172.21.15.78:0/3377242649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f80319d0 con 0x7f4108155f90 2024-02-19T14:57:10.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.370+0000 7f4106ffd640 1 -- 172.21.15.78:0/3377242649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8031d60 con 0x7f4108155f90 2024-02-19T14:57:10.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 -- 172.21.15.78:0/3377242649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 msgr2=0x7f410806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 --2- 172.21.15.78:0/3377242649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410806d550 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f41080c6560 tx=0x7f40f802b910 comp rx=0 tx=0).stop 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 -- 172.21.15.78:0/3377242649 shutdown_connections 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 --2- 172.21.15.78:0/3377242649 >> v2:172.21.15.150:3300/0 conn(0x7f41081588a0 0x7f4108158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 --2- 172.21.15.78:0/3377242649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410806d550 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 -- 172.21.15.78:0/3377242649 >> 172.21.15.78:0/3377242649 conn(0x7f410806b700 msgr2=0x7f41080aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 -- 172.21.15.78:0/3377242649 shutdown_connections 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.371+0000 7f410e993640 1 -- 172.21.15.78:0/3377242649 wait complete. 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.372+0000 7f410e993640 1 Processor -- start 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.372+0000 7f410e993640 1 -- start start 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41080c6d30 0x7f4108157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410814f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41081588a0 0x7f410814fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108150360 con 0x7f4108155f90 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108071a50 con 0x7f41080c6d30 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410806d310 con 0x7f41081588a0 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410814f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410814f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51138/0 (socket says 172.21.15.78:51138) 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 -- 172.21.15.78:0/1420853665 learned_addr learned my addr 172.21.15.78:0/1420853665 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f4107fff640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41080c6d30 0x7f4108157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410cf30640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41081588a0 0x7f410814fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 -- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41081588a0 msgr2=0x7f410814fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41081588a0 0x7f410814fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 -- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41080c6d30 msgr2=0x7f4108157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41080c6d30 0x7f4108157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 -- 172.21.15.78:0/1420853665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80094d0 con 0x7f4108155f90 2024-02-19T14:57:10.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f41077fe640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410814f8e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c400 tx=0x7f40fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0052d0 con 0x7f4108155f90 2024-02-19T14:57:10.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc005450 con 0x7f4108155f90 2024-02-19T14:57:10.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41081eb1a0 con 0x7f4108155f90 2024-02-19T14:57:10.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc007260 con 0x7f4108155f90 2024-02-19T14:57:10.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.373+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4108071390 con 0x7f4108155f90 2024-02-19T14:57:10.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.374+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4108074990 con 0x7f4108155f90 2024-02-19T14:57:10.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.375+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f40fc0073c0 con 0x7f4108155f90 2024-02-19T14:57:10.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.375+0000 7f4104ff9640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40f406b810 0x7f40f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.376+0000 7f4107fff640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40f406b810 0x7f40f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.376+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f40fc089670 con 0x7f4108155f90 2024-02-19T14:57:10.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.376+0000 7f4107fff640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40f406b810 0x7f40f406dcd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f40f8036000 tx=0x7f40f803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.380+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f40fc054040 con 0x7f4108155f90 2024-02-19T14:57:10.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.551+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f41080c75a0 con 0x7f4108155f90 2024-02-19T14:57:10.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.554+0000 7f4104ff9640 1 -- 172.21.15.78:0/1420853665 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v47)=0 v47) v1 ==== 116+0+0 (secure 0 0 0) 0x7f40fc057700 con 0x7f4108155f90 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40f406b810 msgr2=0x7f40f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40f406b810 0x7f40f406dcd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f40f8036000 tx=0x7f40f803c040 comp rx=0 tx=0).stop 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 msgr2=0x7f410814f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410814f8e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c400 tx=0x7f40fc00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 shutdown_connections 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40f406b810 0x7f40f406dcd0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4108155f90 0x7f410814f8e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41081588a0 0x7f410814fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 --2- 172.21.15.78:0/1420853665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41080c6d30 0x7f4108157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 >> 172.21.15.78:0/1420853665 conn(0x7f410806b700 msgr2=0x7f41080aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 shutdown_connections 2024-02-19T14:57:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.556+0000 7f410e993640 1 -- 172.21.15.78:0/1420853665 wait complete. 2024-02-19T14:57:10.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:10.598 INFO:tasks.ceph.mds.b.smithi150.stderr: -210> 2024-02-19T14:57:04.928+0000 7f4587e4e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:10.621 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:57:10.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.669+0000 7ff807e9a640 1 Processor -- start 2024-02-19T14:57:10.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.669+0000 7ff807e9a640 1 -- start start 2024-02-19T14:57:10.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff807e9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 0x7ff8000c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff807e9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8000c2f60 0x7ff80006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff807e9a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8000ae670 con 0x7ff80006f530 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff807e9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8000716b0 con 0x7ff8000c2f60 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff807e9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000c33a0 con 0x7ff8000add70 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff806437640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff80006f530 0x7ff8000705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44240/0 (socket says 172.21.15.78:44240) 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.670+0000 7ff806437640 1 -- 172.21.15.78:0/1838316769 learned_addr learned my addr 172.21.15.78:0/1838316769 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805c36640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 0x7ff8000c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585922508 0 0) 0x7ff8000ae670 con 0x7ff80006f530 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7ec003520 con 0x7ff80006f530 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 --2- 172.21.15.78:0/1838316769 >> v2:172.21.15.150:3300/0 conn(0x7ff8000c2f60 0x7ff80006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403495837 0 0) 0x7ff7ec003520 con 0x7ff80006f530 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 msgr2=0x7ff8000c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 0x7ff8000c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 >> v2:172.21.15.150:3300/0 conn(0x7ff8000c2f60 msgr2=0x7ff80006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 --2- 172.21.15.78:0/1838316769 >> v2:172.21.15.150:3300/0 conn(0x7ff8000c2f60 0x7ff80006eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800153780 con 0x7ff80006f530 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7ff7fc0019f0 con 0x7ff80006f530 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 >> v1:172.21.15.78:6789/0 conn(0x7ff80006f530 legacy=0x7ff8000705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ec003970 0x7ff7ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec0047c0 0x7ff7ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 0x7ff7ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000716b0 con 0x7ff7ec0047c0 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000c33a0 con 0x7ff7ec003970 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec00b9d0 con 0x7ff8000add70 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec0047c0 0x7ff7ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805c36640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ec003970 0x7ff7ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff806437640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 0x7ff7ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 -- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 msgr2=0x7ff7ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:10.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8000add70 0x7ff7ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 -- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ec003970 msgr2=0x7ff7ec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ec003970 0x7ff7ec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.671+0000 7ff805435640 1 -- 172.21.15.78:0/1838316769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec003520 con 0x7ff7ec0047c0 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.672+0000 7ff805435640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec0047c0 0x7ff7ec004c40 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009af0 tx=0x7ff7ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff966e97dc3fdb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.672+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f002c710 con 0x7ff7ec0047c0 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.672+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f002c870 con 0x7ff7ec0047c0 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.672+0000 7ff804c34640 1 -- 172.21.15.78:0/1838316769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f002cc00 con 0x7ff7ec0047c0 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff807e9a640 1 -- 172.21.15.78:0/1838316769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec0047c0 msgr2=0x7ff7ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff807e9a640 1 --2- 172.21.15.78:0/1838316769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7ec0047c0 0x7ff7ec004c40 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009af0 tx=0x7ff7ec0057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff806437640 1 -- 172.21.15.78:0/1838316769 reap_dead start 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff807e9a640 1 -- 172.21.15.78:0/1838316769 shutdown_connections 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff807e9a640 1 -- 172.21.15.78:0/1838316769 >> 172.21.15.78:0/1838316769 conn(0x7ff8000c85e0 msgr2=0x7ff8000c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff807e9a640 1 -- 172.21.15.78:0/1838316769 shutdown_connections 2024-02-19T14:57:10.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.673+0000 7ff807e9a640 1 -- 172.21.15.78:0/1838316769 wait complete. 2024-02-19T14:57:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.674+0000 7ff807e9a640 1 Processor -- start 2024-02-19T14:57:10.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.674+0000 7ff807e9a640 1 -- start start 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.674+0000 7ff807e9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7ec0047c0 0x7ff8001ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 0x7ff8000c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000c1e30 0x7ff8001ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8001ea2a0 con 0x7ff800156ed0 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8001ed890 con 0x7ff8000c1e30 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff800155020 con 0x7ff7ec0047c0 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 0x7ff8000c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 0x7ff8000c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51162/0 (socket says 172.21.15.78:51162) 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 -- 172.21.15.78:0/2630922874 learned_addr learned my addr 172.21.15.78:0/2630922874 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805c36640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7ec0047c0 0x7ff8001ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff806437640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000c1e30 0x7ff8001ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 -- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7ec0047c0 msgr2=0x7ff8001ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7ec0047c0 0x7ff8001ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 -- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000c1e30 msgr2=0x7ff8001ea6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000c1e30 0x7ff8001ea6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 -- 172.21.15.78:0/2630922874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80014c570 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff806437640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000c1e30 0x7ff8001ea6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805c36640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7ec0047c0 0x7ff8001ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff805435640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 0x7ff8000c18f0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009c10 tx=0x7ff7f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0003810 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f0003970 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff800156010 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0003d20 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.675+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80006c400 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.676+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8001ea410 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.677+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff7f0040450 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.677+0000 7ff7f67fc640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7e406b840 0x7ff7e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.677+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff7f00ac0b0 con 0x7ff800156ed0 2024-02-19T14:57:10.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.677+0000 7ff805c36640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7e406b840 0x7ff7e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:10.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.677+0000 7ff805c36640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7e406b840 0x7ff7e406dd00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8006d40 tx=0x7ff7f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:10.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.681+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff8001ea410 con 0x7ff800156ed0 2024-02-19T14:57:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.889+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff80014c550 con 0x7ff800156ed0 2024-02-19T14:57:10.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.890+0000 7ff7f67fc640 1 -- 172.21.15.78:0/2630922874 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+4161 (secure 0 0 0) 0x7ff7f0076980 con 0x7ff800156ed0 2024-02-19T14:57:10.892 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:10.893 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":118,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:10.590085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7351},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7351":{"gid":7351,"name":"d","rank":0,"incarnation":118,"state":"up:replay","state_seq":5,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:10.893 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 118 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7e406b840 msgr2=0x7ff7e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7e406b840 0x7ff7e406dd00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8006d40 tx=0x7ff7f8008040 comp rx=0 tx=0).stop 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 msgr2=0x7ff8000c18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 0x7ff8000c18f0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009c10 tx=0x7ff7f0004730 comp rx=0 tx=0).stop 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 shutdown_connections 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7e406b840 0x7ff7e406dd00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff800156ed0 0x7ff8000c18f0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000c1e30 0x7ff8001ea6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 --2- 172.21.15.78:0/2630922874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7ec0047c0 0x7ff8001ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 >> 172.21.15.78:0/2630922874 conn(0x7ff8000c85e0 msgr2=0x7ff8000c6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 shutdown_connections 2024-02-19T14:57:10.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:10.892+0000 7ff807e9a640 1 -- 172.21.15.78:0/2630922874 wait complete. 2024-02-19T14:57:11.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 118, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:10.590085+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7351}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7351': {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:replay', 'state_seq': 5, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:11.308 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7351:up:replay 2024-02-19T14:57:12.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.424+0000 7f7b06676640 1 Processor -- start 2024-02-19T14:57:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.424+0000 7f7b06676640 1 -- start start 2024-02-19T14:57:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7b06676640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7b000c19f0 0x7f7b00156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7b06676640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7b00071410 0x7f7b000c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7b06676640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b00155a50 con 0x7f7b000c7570 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7b06676640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b00152af0 con 0x7f7b00071410 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7b06676640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b00155f80 con 0x7f7b000c19f0 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7aff7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7b00071410 0x7f7b000c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7aff7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7b00071410 0x7f7b000c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51176/0 (socket says 172.21.15.78:51176) 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.425+0000 7f7aff7fe640 1 -- 172.21.15.78:0/719197776 learned_addr learned my addr 172.21.15.78:0/719197776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.426+0000 7f7afffff640 1 --2- 172.21.15.78:0/719197776 >> v2:172.21.15.150:3300/0 conn(0x7f7b000c19f0 0x7f7b00156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.426+0000 7f7aff7fe640 1 -- 172.21.15.78:0/719197776 >> v2:172.21.15.150:3300/0 conn(0x7f7b000c19f0 msgr2=0x7f7b00156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.426+0000 7f7aff7fe640 1 --2- 172.21.15.78:0/719197776 >> v2:172.21.15.150:3300/0 conn(0x7f7b000c19f0 0x7f7b00156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.426+0000 7f7aff7fe640 1 -- 172.21.15.78:0/719197776 >> v1:172.21.15.150:6789/0 conn(0x7f7b000c7570 legacy=0x7f7b000c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.426+0000 7f7aff7fe640 1 -- 172.21.15.78:0/719197776 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00070840 con 0x7f7b00071410 2024-02-19T14:57:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.426+0000 7f7aff7fe640 1 --2- 172.21.15.78:0/719197776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c7030 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7b000c2ca0 tx=0x7f7af402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84c48956f34d9a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7afeffd640 1 -- 172.21.15.78:0/719197776 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4031750 con 0x7f7b00071410 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7afeffd640 1 -- 172.21.15.78:0/719197776 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af40318b0 con 0x7f7b00071410 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7afeffd640 1 -- 172.21.15.78:0/719197776 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4031c40 con 0x7f7b00071410 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 -- 172.21.15.78:0/719197776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 msgr2=0x7f7b000c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 --2- 172.21.15.78:0/719197776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c7030 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7b000c2ca0 tx=0x7f7af402b7f0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 -- 172.21.15.78:0/719197776 shutdown_connections 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 --2- 172.21.15.78:0/719197776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c7030 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 --2- 172.21.15.78:0/719197776 >> v2:172.21.15.150:3300/0 conn(0x7f7b000c19f0 0x7f7b00156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 -- 172.21.15.78:0/719197776 >> 172.21.15.78:0/719197776 conn(0x7f7b000aa370 msgr2=0x7f7b000aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 -- 172.21.15.78:0/719197776 shutdown_connections 2024-02-19T14:57:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.427+0000 7f7b06676640 1 -- 172.21.15.78:0/719197776 wait complete. 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.428+0000 7f7b06676640 1 Processor -- start 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 -- start start 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b000c19f0 0x7f7b000c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b000c7570 0x7f7b000c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b000c4790 con 0x7f7b00071410 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00153710 con 0x7f7b000c19f0 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.429+0000 7f7b06676640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b001eb7c0 con 0x7f7b000c7570 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51186/0 (socket says 172.21.15.78:51186) 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 -- 172.21.15.78:0/3651534198 learned_addr learned my addr 172.21.15.78:0/3651534198 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7b04c13640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b000c7570 0x7f7b000c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7aff7fe640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b000c19f0 0x7f7b000c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 -- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b000c7570 msgr2=0x7f7b000c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b000c7570 0x7f7b000c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 -- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b000c19f0 msgr2=0x7f7b000c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b000c19f0 0x7f7b000c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 -- 172.21.15.78:0/3651534198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40094d0 con 0x7f7b00071410 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afffff640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c5870 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f7af000d800 tx=0x7f7af000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000b6c0 con 0x7f7b00071410 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af000b820 con 0x7f7b00071410 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b001ebaa0 con 0x7f7b00071410 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000bbb0 con 0x7f7b00071410 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.430+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b001533a0 con 0x7f7b00071410 2024-02-19T14:57:12.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.431+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b00153530 con 0x7f7b00071410 2024-02-19T14:57:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.432+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7af000bd10 con 0x7f7b00071410 2024-02-19T14:57:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.432+0000 7f7afcff9640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7aec06b810 0x7f7aec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.432+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7af008a750 con 0x7f7b00071410 2024-02-19T14:57:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.432+0000 7f7aff7fe640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7aec06b810 0x7f7aec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.433+0000 7f7aff7fe640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7aec06b810 0x7f7aec06dcd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7af4035000 tx=0x7f7af4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:12.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.437+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7af00550d0 con 0x7f7b00071410 2024-02-19T14:57:12.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.645+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b000c8320 con 0x7f7b00071410 2024-02-19T14:57:12.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.645+0000 7f7afcff9640 1 -- 172.21.15.78:0/3651534198 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f7af0058790 con 0x7f7b00071410 2024-02-19T14:57:12.648 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:12.648 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":120,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:11.596004+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7351},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7351":{"gid":7351,"name":"d","rank":0,"incarnation":118,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:12.648 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 120 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7aec06b810 msgr2=0x7f7aec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7aec06b810 0x7f7aec06dcd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7af4035000 tx=0x7f7af4008030 comp rx=0 tx=0).stop 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 msgr2=0x7f7b000c5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c5870 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f7af000d800 tx=0x7f7af000dcc0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 shutdown_connections 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7aec06b810 0x7f7aec06dcd0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b000c7570 0x7f7b000c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b000c19f0 0x7f7b000c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 --2- 172.21.15.78:0/3651534198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b00071410 0x7f7b000c5870 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:12.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 >> 172.21.15.78:0/3651534198 conn(0x7f7b000aa370 msgr2=0x7f7b0014dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:12.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 shutdown_connections 2024-02-19T14:57:12.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:12.647+0000 7f7b06676640 1 -- 172.21.15.78:0/3651534198 wait complete. 2024-02-19T14:57:13.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 120, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:11.596004+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7351}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7351': {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:13.064 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7351:up:rejoin 2024-02-19T14:57:14.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:14.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 Processor -- start 2024-02-19T14:57:14.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 -- start start 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c4152480 0x7fb5c41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb5c40c7070 0x7fb5c406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb5c406d350 0x7fb5c40c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c4155ef0 con 0x7fb5c40c7070 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c406dec0 con 0x7fb5c406d350 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.179+0000 7fb5cb0c3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40c75c0 con 0x7fb5c4152480 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb5c40c7070 0x7fb5c406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb5c40c7070 0x7fb5c406ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51190/0 (socket says 172.21.15.78:51190) 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 -- 172.21.15.78:0/2165841431 learned_addr learned my addr 172.21.15.78:0/2165841431 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c8e5f640 1 --2- 172.21.15.78:0/2165841431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c4152480 0x7fb5c41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c9660640 1 --2- 172.21.15.78:0/2165841431 >> v2:172.21.15.150:3300/0 conn(0x7fb5c406d350 0x7fb5c40c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 -- 172.21.15.78:0/2165841431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c4152480 msgr2=0x7fb5c41559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 --2- 172.21.15.78:0/2165841431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c4152480 0x7fb5c41559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 -- 172.21.15.78:0/2165841431 >> v2:172.21.15.150:3300/0 conn(0x7fb5c406d350 msgr2=0x7fb5c40c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 --2- 172.21.15.78:0/2165841431 >> v2:172.21.15.150:3300/0 conn(0x7fb5c406d350 0x7fb5c40c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.180+0000 7fb5c3fff640 1 -- 172.21.15.78:0/2165841431 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c40c78a0 con 0x7fb5c40c7070 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.181+0000 7fb5c3fff640 1 --2- 172.21.15.78:0/2165841431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c406ce10 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009870 tx=0x7fb5b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4617fb0a89c4cd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.181+0000 7fb5c37fe640 1 -- 172.21.15.78:0/2165841431 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0031640 con 0x7fb5c40c7070 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.181+0000 7fb5c37fe640 1 -- 172.21.15.78:0/2165841431 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b00317a0 con 0x7fb5c40c7070 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.181+0000 7fb5c37fe640 1 -- 172.21.15.78:0/2165841431 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0031b30 con 0x7fb5c40c7070 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/2165841431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 msgr2=0x7fb5c406ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/2165841431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c406ce10 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009870 tx=0x7fb5b002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/2165841431 shutdown_connections 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/2165841431 >> v2:172.21.15.150:3300/0 conn(0x7fb5c406d350 0x7fb5c40c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/2165841431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c406ce10 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/2165841431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c4152480 0x7fb5c41559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/2165841431 >> 172.21.15.78:0/2165841431 conn(0x7fb5c40ca8a0 msgr2=0x7fb5c40cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/2165841431 shutdown_connections 2024-02-19T14:57:14.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.182+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/2165841431 wait complete. 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.183+0000 7fb5cb0c3640 1 Processor -- start 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.183+0000 7fb5cb0c3640 1 -- start start 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5cb0c3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c4149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5cb0c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c4152480 0x7fb5c40ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5cb0c3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c41eaba0 0x7fb5c41eb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5cb0c3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40ae440 con 0x7fb5c40c7070 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5cb0c3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c41ef050 con 0x7fb5c4152480 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5cb0c3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c41eebc0 con 0x7fb5c41eaba0 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c4149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c4149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51192/0 (socket says 172.21.15.78:51192) 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 -- 172.21.15.78:0/876061664 learned_addr learned my addr 172.21.15.78:0/876061664 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c3fff640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c4152480 0x7fb5c40ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c9660640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c41eaba0 0x7fb5c41eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 -- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c41eaba0 msgr2=0x7fb5c41eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c41eaba0 0x7fb5c41eb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 -- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c4152480 msgr2=0x7fb5c40ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c4152480 0x7fb5c40ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 -- 172.21.15.78:0/876061664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b00094d0 con 0x7fb5c40c7070 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.184+0000 7fb5c8e5f640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c4149f40 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e9e0 tx=0x7fb5b800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.185+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8009670 con 0x7fb5c40c7070 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.185+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b800c970 con 0x7fb5c40c7070 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.185+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c41ee290 con 0x7fb5c40c7070 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.185+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b800cd20 con 0x7fb5c40c7070 2024-02-19T14:57:14.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.185+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c41ee850 con 0x7fb5c40c7070 2024-02-19T14:57:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.186+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c4074990 con 0x7fb5c40c7070 2024-02-19T14:57:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.186+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb5b8013470 con 0x7fb5c40c7070 2024-02-19T14:57:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.186+0000 7fb5c17fa640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5a806b810 0x7fb5a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.187+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb5b808a3e0 con 0x7fb5c40c7070 2024-02-19T14:57:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.187+0000 7fb5c3fff640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5a806b810 0x7fb5a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.187+0000 7fb5c3fff640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5a806b810 0x7fb5a806dcd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0036000 tx=0x7fb5b003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.191+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb5b8054d60 con 0x7fb5c40c7070 2024-02-19T14:57:14.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.400+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5c414fa80 con 0x7fb5c40c7070 2024-02-19T14:57:14.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.401+0000 7fb5c17fa640 1 -- 172.21.15.78:0/876061664 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4956 (secure 0 0 0) 0x7fb5b8058420 con 0x7fb5c40c7070 2024-02-19T14:57:14.404 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:14.404 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:12.597030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7351},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7351":{"gid":7351,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:14.404 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 121 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5a806b810 msgr2=0x7fb5a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5a806b810 0x7fb5a806dcd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0036000 tx=0x7fb5b003c040 comp rx=0 tx=0).stop 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 msgr2=0x7fb5c4149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c4149f40 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800e9e0 tx=0x7fb5b800eea0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 shutdown_connections 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb5a806b810 0x7fb5a806dcd0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c41eaba0 0x7fb5c41eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c4152480 0x7fb5c40ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 --2- 172.21.15.78:0/876061664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5c40c7070 0x7fb5c4149f40 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 >> 172.21.15.78:0/876061664 conn(0x7fb5c40ca8a0 msgr2=0x7fb5c414cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 shutdown_connections 2024-02-19T14:57:14.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.403+0000 7fb5cb0c3640 1 -- 172.21.15.78:0/876061664 wait complete. 2024-02-19T14:57:14.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:12.597030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7351}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7351': {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:14.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:57:14.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:57:14.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.919+0000 7f7776240640 1 Processor -- start 2024-02-19T14:57:14.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.919+0000 7f7776240640 1 -- start start 2024-02-19T14:57:14.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.919+0000 7f7776240640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f77700c50a0 0x7f77700c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.919+0000 7f7776240640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7770071c20 con 0x7f7770071790 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.919+0000 7f7776240640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77700c7fe0 con 0x7f77700c5480 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.919+0000 7f7776240640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77700c82c0 con 0x7f77700c50a0 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f776f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f77700c50a0 0x7f77700c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f7767fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7770071790 0x7f777006cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44250/0 (socket says 172.21.15.78:44250) 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f7767fff640 1 -- 172.21.15.78:0/1100962610 learned_addr learned my addr 172.21.15.78:0/1100962610 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f776f7fe640 1 -- 172.21.15.78:0/1100962610 >> v1:172.21.15.150:6789/0 conn(0x7f77700c5480 legacy=0x7f7770155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f776f7fe640 1 -- 172.21.15.78:0/1100962610 >> v1:172.21.15.78:6789/0 conn(0x7f7770071790 legacy=0x7f777006cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f776effd640 1 -- 172.21.15.78:0/1100962610 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703345731 0 0) 0x7f7770071c20 con 0x7f7770071790 2024-02-19T14:57:14.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.920+0000 7f776f7fe640 1 -- 172.21.15.78:0/1100962610 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777006c700 con 0x7f77700c50a0 2024-02-19T14:57:14.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f776f7fe640 1 --2- 172.21.15.78:0/1100962610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c50a0 0x7f77700c73e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f77600097a0 tx=0x7f776002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c675f7d9be66dd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f776effd640 1 -- 172.21.15.78:0/1100962610 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760031830 con 0x7f77700c50a0 2024-02-19T14:57:14.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f776effd640 1 -- 172.21.15.78:0/1100962610 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760031990 con 0x7f77700c50a0 2024-02-19T14:57:14.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f776effd640 1 -- 172.21.15.78:0/1100962610 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760031d20 con 0x7f77700c50a0 2024-02-19T14:57:14.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f7776240640 1 -- 172.21.15.78:0/1100962610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c50a0 msgr2=0x7f77700c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f7776240640 1 --2- 172.21.15.78:0/1100962610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c50a0 0x7f77700c73e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f77600097a0 tx=0x7f776002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f7776240640 1 -- 172.21.15.78:0/1100962610 shutdown_connections 2024-02-19T14:57:14.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f7776240640 1 --2- 172.21.15.78:0/1100962610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c50a0 0x7f77700c73e0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f7776240640 1 -- 172.21.15.78:0/1100962610 >> 172.21.15.78:0/1100962610 conn(0x7f777006b700 msgr2=0x7f77700aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.921+0000 7f7776240640 1 -- 172.21.15.78:0/1100962610 shutdown_connections 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.922+0000 7f7776240640 1 -- 172.21.15.78:0/1100962610 wait complete. 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 Processor -- start 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 -- start start 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77700c50a0 0x7f77701ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 0x7f7770070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7770158930 0x7f77701ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77701ea230 con 0x7f77700c5480 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77700ab800 con 0x7f7770158930 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7776240640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77700ab520 con 0x7f77700c50a0 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7767fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 0x7f7770070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7767fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 0x7f7770070770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51216/0 (socket says 172.21.15.78:51216) 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f7767fff640 1 -- 172.21.15.78:0/1832419478 learned_addr learned my addr 172.21.15.78:0/1832419478 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f776f7fe640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77700c50a0 0x7f77701ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.923+0000 7f776ffff640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7770158930 0x7f77701ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7767fff640 1 -- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77700c50a0 msgr2=0x7f77701ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7767fff640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77700c50a0 0x7f77701ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7767fff640 1 -- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7770158930 msgr2=0x7f77701ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7767fff640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7770158930 0x7f77701ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7767fff640 1 -- 172.21.15.78:0/1832419478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760009400 con 0x7f77700c5480 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f776ffff640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7770158930 0x7f77701ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7767fff640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 0x7f7770070770 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f775800c460 tx=0x7f775800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f776cff9640 1 -- 172.21.15.78:0/1832419478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77580074e0 con 0x7f77700c5480 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f776cff9640 1 -- 172.21.15.78:0/1832419478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758007680 con 0x7f77700c5480 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7770156530 con 0x7f77700c5480 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f776cff9640 1 -- 172.21.15.78:0/1832419478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758004a00 con 0x7f77700c5480 2024-02-19T14:57:14.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.924+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77701ed720 con 0x7f77700c5480 2024-02-19T14:57:14.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.925+0000 7f776cff9640 1 -- 172.21.15.78:0/1832419478 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7758004b60 con 0x7f77700c5480 2024-02-19T14:57:14.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.926+0000 7f776cff9640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f775006b840 0x7f775006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.926+0000 7f776cff9640 1 -- 172.21.15.78:0/1832419478 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7758089800 con 0x7f77700c5480 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.926+0000 7f776f7fe640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f775006b840 0x7f775006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.926+0000 7f776f7fe640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f775006b840 0x7f775006dd00 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7760036000 tx=0x7f776003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 Processor -- start 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 -- start start 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56d7740 0x5644b56d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 0x5644b56d7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56da8b0 0x5644b569c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b5657cd0 con 0x5644b569dae0 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.927+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b560af20 con 0x5644b56da8b0 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644b5608450 con 0x5644b56d7740 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 0x5644b56d7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 0x5644b56d7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51218/0 (socket says 172.21.15.78:51218) 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 -- 172.21.15.78:0/1785997733 learned_addr learned my addr 172.21.15.78:0/1785997733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f776f7fe640 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56d7740 0x5644b56d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f776ffff640 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56da8b0 0x5644b569c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 -- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56d7740 msgr2=0x5644b56d88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56d7740 0x5644b56d88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 -- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56da8b0 msgr2=0x5644b569c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56da8b0 0x5644b569c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 -- 172.21.15.78:0/1785997733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644b55b8f20 con 0x5644b569dae0 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f7767fff640 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 0x5644b56d7b20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f7758054340 tx=0x7f7758057fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f248baa1443eb2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f77657fa640 1 -- 172.21.15.78:0/1785997733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775805cea0 con 0x5644b569dae0 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f77657fa640 1 -- 172.21.15.78:0/1785997733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758004e10 con 0x5644b569dae0 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.928+0000 7f77657fa640 1 -- 172.21.15.78:0/1785997733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758054550 con 0x5644b569dae0 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 -- 172.21.15.78:0/1785997733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 msgr2=0x5644b56d7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 0x5644b56d7b20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f7758054340 tx=0x7f7758057fa0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 -- 172.21.15.78:0/1785997733 shutdown_connections 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b569dae0 0x5644b56d7b20 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56da8b0 0x5644b569c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 --2- 172.21.15.78:0/1785997733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56d7740 0x5644b56d88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 -- 172.21.15.78:0/1785997733 >> 172.21.15.78:0/1785997733 conn(0x5644b5699120 msgr2=0x5644b56bdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 -- 172.21.15.78:0/1785997733 shutdown_connections 2024-02-19T14:57:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 -- 172.21.15.78:0/1785997733 wait complete. 2024-02-19T14:57:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.929+0000 7f7778e18740 1 -- start start 2024-02-19T14:57:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 Processor -- start 2024-02-19T14:57:14.936 INFO:tasks.ceph.mds.d.smithi150.stderr:2024-02-19T14:57:14.928+0000 7f178f783640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:14.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56d5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b569b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b56af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b5603c30 con 0x5644b56d7740 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b55a8d30 con 0x5644b56baf80 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644b52d7310 con 0x5644b56da8b0 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f776f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7778e18740 10 client.0 _ll_register_callbacks cb 0x5644b55b0d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f776f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56d5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51222/0 (socket says 172.21.15.78:51222) 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f776f7fe640 1 -- 172.21.15.78:0/4138223994 learned_addr learned my addr 172.21.15.78:0/4138223994 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f7767fff640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b569b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.930+0000 7f776ffff640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b56af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f776f7fe640 1 -- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 msgr2=0x5644b569b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f776f7fe640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b569b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f776f7fe640 1 -- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 msgr2=0x5644b56af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f776f7fe640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b56af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f776f7fe640 1 -- 172.21.15.78:0/4138223994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77701ed720 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f776f7fe640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56d5050 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f776002bee0 tx=0x7f776002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929d2320ca6a381 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f774e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f774e7fc640 1 -- 172.21.15.78:0/4138223994 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776004c040 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f774e7fc640 1 -- 172.21.15.78:0/4138223994 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7760045db0 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5644b53036c0 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f774e7fc640 1 -- 172.21.15.78:0/4138223994 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f776003f040 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 10 client.8502 fetch_fsmap learned FSMap version 121 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x5644b56022b0 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f774e7fc640 1 -- 172.21.15.78:0/4138223994 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5644b53036c0 con 0x5644b56d7740 2024-02-19T14:57:14.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f774e7fc640 1 -- 172.21.15.78:0/4138223994 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7760031d20 con 0x5644b56d7740 2024-02-19T14:57:14.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 10 client.8502 fetch_fsmap finished waiting for FSMap version 121 2024-02-19T14:57:14.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 10 client.8502 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:57:14.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 20 client.8502 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:14.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56d2950 0x5644b56d2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 4 client.8502 mds_command: new command op to 7351 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:57:14.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.931+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 --> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644b5354270 con 0x5644b56d2950 2024-02-19T14:57:14.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.932+0000 7f7767fff640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56d2950 0x5644b56d2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.932+0000 7f774e7fc640 10 client.8502 ms_handle_connect on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:14.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.932+0000 7f7767fff640 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56d2950 0x5644b56d2d30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc44dba1484e418e server_cookie=5683a76c60bc4a34 in_seq=0 out_seq=0 2024-02-19T14:57:14.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f774e7fc640 1 -- 172.21.15.78:0/4138223994 <== mds.0 v2:172.21.15.150:6833/1869543632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5644b5354270 con 0x5644b56d2950 2024-02-19T14:57:14.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f774e7fc640 10 client.8502 handle_command_reply: tid=0 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 client.8502 shutdown 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 msgr2=0x5644b56d5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56d5050 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f776002bee0 tx=0x7f776002e000 comp rx=0 tx=0).stop 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 shutdown_connections 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b56af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b569b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56d2950 0x5644b56d2d30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 --2- 172.21.15.78:0/4138223994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56d5050 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f774e7fc640 0 client.8502 ms_handle_reset on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:14.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 >> 172.21.15.78:0/4138223994 conn(0x5644b569dae0 msgr2=0x5644b56bdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 shutdown_connections 2024-02-19T14:57:14.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 1 -- 172.21.15.78:0/4138223994 wait complete. 2024-02-19T14:57:14.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.933+0000 7f7778e18740 20 client.8502 trim_cache size 0 max 16384 2024-02-19T14:57:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.949+0000 7f7778e18740 1 Processor -- start 2024-02-19T14:57:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.949+0000 7f7778e18740 1 -- start start 2024-02-19T14:57:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b5697f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7778e18740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56ba180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b56ba6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7778e18740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b5589490 con 0x5644b56d7740 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b5692ff0 con 0x5644b56baf80 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644b56bb640 con 0x5644b56da8b0 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7767fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56ba180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7767fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56ba180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51234/0 (socket says 172.21.15.78:51234) 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f7767fff640 1 -- 172.21.15.78:0/1185104615 learned_addr learned my addr 172.21.15.78:0/1185104615 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f776f7fe640 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b5697f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.950+0000 7f776ffff640 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b56ba6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7767fff640 1 -- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 msgr2=0x5644b56ba6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7767fff640 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b56ba6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7767fff640 1 -- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 msgr2=0x5644b5697f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7767fff640 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b5697f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7767fff640 1 -- 172.21.15.78:0/1185104615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644b56d6af0 con 0x5644b56d7740 2024-02-19T14:57:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7767fff640 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56ba180 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7758087c30 tx=0x7f775801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22348b6ac68872ab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7764ff9640 1 -- 172.21.15.78:0/1185104615 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758057a20 con 0x5644b56d7740 2024-02-19T14:57:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7764ff9640 1 -- 172.21.15.78:0/1185104615 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758007e20 con 0x5644b56d7740 2024-02-19T14:57:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7764ff9640 1 -- 172.21.15.78:0/1185104615 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758054550 con 0x5644b56d7740 2024-02-19T14:57:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7778e18740 1 -- 172.21.15.78:0/1185104615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 msgr2=0x5644b56ba180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.951+0000 7f7778e18740 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56ba180 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7758087c30 tx=0x7f775801f000 comp rx=0 tx=0).stop 2024-02-19T14:57:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 -- 172.21.15.78:0/1185104615 shutdown_connections 2024-02-19T14:57:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56d7740 0x5644b56ba180 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56da8b0 0x5644b56ba6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 --2- 172.21.15.78:0/1185104615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56baf80 0x5644b5697f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 -- 172.21.15.78:0/1185104615 >> 172.21.15.78:0/1185104615 conn(0x5644b569dae0 msgr2=0x5644b56e87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 -- 172.21.15.78:0/1185104615 shutdown_connections 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 -- 172.21.15.78:0/1185104615 wait complete. 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 -- start start 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.952+0000 7f7778e18740 1 Processor -- start 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56baf80 0x5644b56d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56d7740 0x5644b56ba580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 0x5644b56ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b56f8980 con 0x5644b56da8b0 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b574c480 con 0x5644b56d7740 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644b56d7180 con 0x5644b56baf80 2024-02-19T14:57:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7778e18740 10 client.0 _ll_register_callbacks cb 0x7f776003a500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 0x5644b56ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56baf80 0x5644b56d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 0x5644b56ae1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51242/0 (socket says 172.21.15.78:51242) 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 -- 172.21.15.78:0/104349965 learned_addr learned my addr 172.21.15.78:0/104349965 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f7767fff640 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56d7740 0x5644b56ba580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 -- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56baf80 msgr2=0x5644b56d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56baf80 0x5644b56d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 -- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56d7740 msgr2=0x5644b56ba580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56d7740 0x5644b56ba580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 -- 172.21.15.78:0/104349965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644b55b8f20 con 0x5644b56da8b0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f776ffff640 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 0x5644b56ae1b0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f77680012e0 tx=0x7f7768017e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e01b208aad4843 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.953+0000 7f774e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f774e7fc640 1 -- 172.21.15.78:0/104349965 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768006e20 con 0x5644b56da8b0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f774e7fc640 1 -- 172.21.15.78:0/104349965 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f776801dae0 con 0x5644b56da8b0 2024-02-19T14:57:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7758004e10 con 0x5644b56da8b0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f774e7fc640 1 -- 172.21.15.78:0/104349965 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f776801dca0 con 0x5644b56da8b0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 10 client.8508 fetch_fsmap learned FSMap version 121 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x5644b559ea20 con 0x5644b56da8b0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f774e7fc640 1 -- 172.21.15.78:0/104349965 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758004e10 con 0x5644b56da8b0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f774e7fc640 1 -- 172.21.15.78:0/104349965 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f776801f030 con 0x5644b56da8b0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 10 client.8508 fetch_fsmap finished waiting for FSMap version 121 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 10 client.8508 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 20 client.8508 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56ae9c0 0x5644b57454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 4 client.8508 mds_command: new command op to 7351 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 --> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5644b5340610 con 0x5644b56ae9c0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f776f7fe640 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56ae9c0 0x5644b57454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f774e7fc640 10 client.8508 ms_handle_connect on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.954+0000 7f776f7fe640 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56ae9c0 0x5644b57454e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c09c4c60284689cf server_cookie=eb738d43b0b423a9 in_seq=0 out_seq=0 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f774e7fc640 1 -- 172.21.15.78:0/104349965 <== mds.0 v2:172.21.15.150:6833/1869543632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5644b5340610 con 0x5644b56ae9c0 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f774e7fc640 10 client.8508 handle_command_reply: tid=0 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 client.8508 shutdown 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 msgr2=0x5644b56ae1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 0x5644b56ae1b0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f77680012e0 tx=0x7f7768017e60 comp rx=0 tx=0).stop 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 shutdown_connections 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5644b56ae9c0 0x5644b57454e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5644b56da8b0 0x5644b56ae1b0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5644b56d7740 0x5644b56ba580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f774e7fc640 0 client.8508 ms_handle_reset on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 --2- 172.21.15.78:0/104349965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5644b56baf80 0x5644b56d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 >> 172.21.15.78:0/104349965 conn(0x5644b569dae0 msgr2=0x5644b56dac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 shutdown_connections 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 1 -- 172.21.15.78:0/104349965 wait complete. 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7778e18740 20 client.8508 trim_cache size 0 max 16384 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7351, 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:57:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.3638160030000002, 2024-02-19T14:57:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 121, 2024-02-19T14:57:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 139, 2024-02-19T14:57:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 139, 2024-02-19T14:57:14.966 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 20.382456023 2024-02-19T14:57:14.966 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f775006b840 msgr2=0x7f775006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f775006b840 0x7f775006dd00 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7760036000 tx=0x7f776003c040 comp rx=0 tx=0).stop 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 msgr2=0x7f7770070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 0x7f7770070770 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f775800c460 tx=0x7f775800c920 comp rx=0 tx=0).stop 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 shutdown_connections 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f775006b840 0x7f775006dd00 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77700c5480 0x7f7770070770 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7770158930 0x7f77701ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 --2- 172.21.15.78:0/1832419478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77700c50a0 0x7f77701ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.955+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 >> 172.21.15.78:0/1832419478 conn(0x7f777006b700 msgr2=0x7f77701541f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.956+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 shutdown_connections 2024-02-19T14:57:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:14.956+0000 7f7776240640 1 -- 172.21.15.78:0/1832419478 wait complete. 2024-02-19T14:57:14.969 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:59.000185+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7351": { "addr": "172.21.15.150:6835/1869543632", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1869543632, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1869543632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7351, "incarnation": 118, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 139, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-19T14:57:12.597030+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7351 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/3153272641", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3153272641, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3153272641, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 119, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7522, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1973988605", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1973988605, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1973988605, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8259, "incarnation": 0, "join_fscid": 10, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 5 }, { "addr": "172.21.15.78:6837/3197225278", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3197225278, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3197225278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:57:14.969 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 10, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-02-19T14:57:14.969 INFO:tasks.cephfs.test_failover:current = [{'gid': 8262, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3197225278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3197225278}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3197225278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}, {'gid': 7522, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3153272641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3153272641}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3153272641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 119}, {'gid': 8259, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 5, 'addr': '172.21.15.78:6836/1973988605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1973988605}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1973988605}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:57:14.970 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-19T14:57:19.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.080+0000 7fc72e98c640 1 Processor -- start 2024-02-19T14:57:20.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.080+0000 7fc72e98c640 1 -- start start 2024-02-19T14:57:20.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc72e98c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc728152700 0x7fc7280c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc72e98c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc7280c8e00 0x7fc72815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc72e98c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7281551f0 con 0x7fc7280c5730 2024-02-19T14:57:20.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc72e98c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc728155490 con 0x7fc728152700 2024-02-19T14:57:20.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc72e98c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc728155770 con 0x7fc7280c8e00 2024-02-19T14:57:20.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc727fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc728152700 0x7fc7280c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc7277fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc7280c5730 0x7fc72806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51732/0 (socket says 172.21.15.78:51732) 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc7277fe640 1 -- 172.21.15.78:0/480164286 learned_addr learned my addr 172.21.15.78:0/480164286 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.081+0000 7fc72cf29640 1 --2- 172.21.15.78:0/480164286 >> v2:172.21.15.150:3300/0 conn(0x7fc7280c8e00 0x7fc72815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.082+0000 7fc727fff640 1 -- 172.21.15.78:0/480164286 >> v2:172.21.15.150:3300/0 conn(0x7fc7280c8e00 msgr2=0x7fc72815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.082+0000 7fc727fff640 1 --2- 172.21.15.78:0/480164286 >> v2:172.21.15.150:3300/0 conn(0x7fc7280c8e00 0x7fc72815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.082+0000 7fc727fff640 1 -- 172.21.15.78:0/480164286 >> v1:172.21.15.78:6789/0 conn(0x7fc7280c5730 legacy=0x7fc72806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.082+0000 7fc727fff640 1 -- 172.21.15.78:0/480164286 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc728155a50 con 0x7fc728152700 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.082+0000 7fc726ffd640 1 -- 172.21.15.78:0/480164286 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655677312 0 0) 0x7fc7281551f0 con 0x7fc7280c5730 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.082+0000 7fc727fff640 1 --2- 172.21.15.78:0/480164286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc728152700 0x7fc7280c51f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fc7281596f0 tx=0x7fc71802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af1b4a7b951f6eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc726ffd640 1 -- 172.21.15.78:0/480164286 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71802e070 con 0x7fc728152700 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc726ffd640 1 -- 172.21.15.78:0/480164286 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7180315a0 con 0x7fc728152700 2024-02-19T14:57:20.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc726ffd640 1 -- 172.21.15.78:0/480164286 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718031930 con 0x7fc728152700 2024-02-19T14:57:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 -- 172.21.15.78:0/480164286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc728152700 msgr2=0x7fc7280c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 --2- 172.21.15.78:0/480164286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc728152700 0x7fc7280c51f0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fc7281596f0 tx=0x7fc71802b840 comp rx=0 tx=0).stop 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 -- 172.21.15.78:0/480164286 shutdown_connections 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 --2- 172.21.15.78:0/480164286 >> v2:172.21.15.150:3300/0 conn(0x7fc7280c8e00 0x7fc72815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 --2- 172.21.15.78:0/480164286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc728152700 0x7fc7280c51f0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 -- 172.21.15.78:0/480164286 >> 172.21.15.78:0/480164286 conn(0x7fc7280cac90 msgr2=0x7fc7280cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 -- 172.21.15.78:0/480164286 shutdown_connections 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.083+0000 7fc72e98c640 1 -- 172.21.15.78:0/480164286 wait complete. 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.084+0000 7fc72e98c640 1 Processor -- start 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 -- start start 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 0x7fc7281ee650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7280c8e00 0x7fc7280c0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc728152700 0x7fc7280ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280c0f50 con 0x7fc7280c5730 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281edb70 con 0x7fc7280c8e00 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72e98c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281ede50 con 0x7fc728152700 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc727fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 0x7fc7281ee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc727fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 0x7fc7281ee650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57120/0 (socket says 172.21.15.78:57120) 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc727fff640 1 -- 172.21.15.78:0/540202673 learned_addr learned my addr 172.21.15.78:0/540202673 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc7277fe640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7280c8e00 0x7fc7280c0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.085+0000 7fc72cf29640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc728152700 0x7fc7280ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc727fff640 1 -- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc728152700 msgr2=0x7fc7280ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc727fff640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc728152700 0x7fc7280ac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc727fff640 1 -- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7280c8e00 msgr2=0x7fc7280c0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc727fff640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7280c8e00 0x7fc7280c0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc727fff640 1 -- 172.21.15.78:0/540202673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc718009400 con 0x7fc7280c5730 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc727fff640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 0x7fc7281ee650 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fc718000c00 tx=0x7fc7180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718032840 con 0x7fc7280c5730 2024-02-19T14:57:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7180329c0 con 0x7fc7280c5730 2024-02-19T14:57:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7281ee130 con 0x7fc7280c5730 2024-02-19T14:57:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718032d70 con 0x7fc7280c5730 2024-02-19T14:57:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.086+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7281513c0 con 0x7fc7280c5730 2024-02-19T14:57:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.087+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7281ee340 con 0x7fc7280c5730 2024-02-19T14:57:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.087+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc71804d3f0 con 0x7fc7280c5730 2024-02-19T14:57:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.088+0000 7fc724ff9640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71406b810 0x7fc71406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.088+0000 7fc7277fe640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71406b810 0x7fc71406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.088+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc7180abe00 con 0x7fc7280c5730 2024-02-19T14:57:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.088+0000 7fc7277fe640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71406b810 0x7fc71406dcd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc728155330 tx=0x7fc71c007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.092+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc7180766f0 con 0x7fc7280c5730 2024-02-19T14:57:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.300+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7280688f0 con 0x7fc7280c5730 2024-02-19T14:57:20.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.301+0000 7fc724ff9640 1 -- 172.21.15.78:0/540202673 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4956 (secure 0 0 0) 0x7fc718079db0 con 0x7fc7280c5730 2024-02-19T14:57:20.304 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:20.304 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:12.597030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7351},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7351":{"gid":7351,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:20.304 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 121 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71406b810 msgr2=0x7fc71406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71406b810 0x7fc71406dcd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc728155330 tx=0x7fc71c007d50 comp rx=0 tx=0).stop 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 msgr2=0x7fc7281ee650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 0x7fc7281ee650 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fc718000c00 tx=0x7fc7180046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 shutdown_connections 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71406b810 0x7fc71406dcd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc728152700 0x7fc7280ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7280c8e00 0x7fc7280c0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 --2- 172.21.15.78:0/540202673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7280c5730 0x7fc7281ee650 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 >> 172.21.15.78:0/540202673 conn(0x7fc7280cac90 msgr2=0x7fc7280cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 shutdown_connections 2024-02-19T14:57:20.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.303+0000 7fc72e98c640 1 -- 172.21.15.78:0/540202673 wait complete. 2024-02-19T14:57:20.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:12.597030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7351}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7351': {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:20.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:57:20.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.821+0000 7f69bc8d4640 1 Processor -- start 2024-02-19T14:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.821+0000 7f69bc8d4640 1 -- start start 2024-02-19T14:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69bc8d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69b8150440 0x7f69b80c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69bc8d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b80c3800 0x7f69b806d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69bc8d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b8150820 con 0x7f69b806ed20 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69bc8d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80c1b60 con 0x7f69b8150440 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69bc8d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b81537d0 con 0x7f69b80c3800 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69b659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69b8150440 0x7f69b80c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69b659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69b8150440 0x7f69b80c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57132/0 (socket says 172.21.15.78:57132) 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.822+0000 7f69b659c640 1 -- 172.21.15.78:0/275295838 learned_addr learned my addr 172.21.15.78:0/275295838 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.823+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/275295838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b80c3800 0x7f69b806d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.823+0000 7f69b659c640 1 -- 172.21.15.78:0/275295838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b80c3800 msgr2=0x7f69b806d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.823+0000 7f69b659c640 1 --2- 172.21.15.78:0/275295838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b80c3800 0x7f69b806d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.823+0000 7f69b659c640 1 -- 172.21.15.78:0/275295838 >> v1:172.21.15.150:6789/0 conn(0x7f69b806ed20 legacy=0x7f69b806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.823+0000 7f69b659c640 1 -- 172.21.15.78:0/275295838 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8153d10 con 0x7f69b8150440 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.823+0000 7f69b659c640 1 --2- 172.21.15.78:0/275295838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b8150440 0x7f69b80c32c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f69b8153f00 tx=0x7f69a802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443bc92151355320 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69b559a640 1 -- 172.21.15.78:0/275295838 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a802cb70 con 0x7f69b8150440 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69b559a640 1 -- 172.21.15.78:0/275295838 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a802ccd0 con 0x7f69b8150440 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69b559a640 1 -- 172.21.15.78:0/275295838 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8034610 con 0x7f69b8150440 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 -- 172.21.15.78:0/275295838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b8150440 msgr2=0x7f69b80c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/275295838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b8150440 0x7f69b80c32c0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f69b8153f00 tx=0x7f69a802bfd0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 -- 172.21.15.78:0/275295838 shutdown_connections 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/275295838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b80c3800 0x7f69b806d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/275295838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b8150440 0x7f69b80c32c0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 -- 172.21.15.78:0/275295838 >> 172.21.15.78:0/275295838 conn(0x7f69b806b270 msgr2=0x7f69b806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 -- 172.21.15.78:0/275295838 shutdown_connections 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.824+0000 7f69bc8d4640 1 -- 172.21.15.78:0/275295838 wait complete. 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.825+0000 7f69bc8d4640 1 Processor -- start 2024-02-19T14:57:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.825+0000 7f69bc8d4640 1 -- start start 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69bc8d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b806ed20 0x7f69b8146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69bc8d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 0x7f69b80bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69bc8d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b8150440 0x7f69b8146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69bc8d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8156ea0 con 0x7f69b80c3800 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69bc8d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b81eeb20 con 0x7f69b806ed20 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69bc8d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8146a90 con 0x7f69b8150440 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 0x7f69b80bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 0x7f69b80bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57146/0 (socket says 172.21.15.78:57146) 2024-02-19T14:57:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 -- 172.21.15.78:0/127309561 learned_addr learned my addr 172.21.15.78:0/127309561 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b659c640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b806ed20 0x7f69b8146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b8150440 0x7f69b8146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 -- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b8150440 msgr2=0x7f69b8146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b8150440 0x7f69b8146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 -- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b806ed20 msgr2=0x7f69b8146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b806ed20 0x7f69b8146550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b5d9b640 1 -- 172.21.15.78:0/127309561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009b30 con 0x7f69b80c3800 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.826+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b8150440 0x7f69b8146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.827+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 0x7f69b80bf9b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c460 tx=0x7f69ac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.827+0000 7f69a6ffd640 1 -- 172.21.15.78:0/127309561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0052d0 con 0x7f69b80c3800 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.827+0000 7f69a6ffd640 1 -- 172.21.15.78:0/127309561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac005450 con 0x7f69b80c3800 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.827+0000 7f69a6ffd640 1 -- 172.21.15.78:0/127309561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac007260 con 0x7f69b80c3800 2024-02-19T14:57:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.827+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b81edf10 con 0x7f69b80c3800 2024-02-19T14:57:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.827+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b81ee4d0 con 0x7f69b80c3800 2024-02-19T14:57:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.829+0000 7f69a6ffd640 1 -- 172.21.15.78:0/127309561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f69ac0073c0 con 0x7f69b80c3800 2024-02-19T14:57:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.829+0000 7f69a6ffd640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f699806b810 0x7f699806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.829+0000 7f69a6ffd640 1 -- 172.21.15.78:0/127309561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f69ac089b50 con 0x7f69b80c3800 2024-02-19T14:57:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.829+0000 7f69b659c640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f699806b810 0x7f699806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.829+0000 7f69b659c640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f699806b810 0x7f699806dcd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f69a8035000 tx=0x7f69a8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.830+0000 7f69bf592740 1 Processor -- start 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.830+0000 7f69bf592740 1 -- start start 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f2800ae0 0x5624f283ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69bf592740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f27ff860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69bf592740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f27bacd0 con 0x5624f283d8b0 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f276df20 con 0x5624f283a740 2024-02-19T14:57:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624f276b450 con 0x5624f2800ae0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f27ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f27ff860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57152/0 (socket says 172.21.15.78:57152) 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 -- 172.21.15.78:0/2487767969 learned_addr learned my addr 172.21.15.78:0/2487767969 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b659c640 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f2800ae0 0x5624f283ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 -- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f2800ae0 msgr2=0x5624f283ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f2800ae0 0x5624f283ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 -- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 msgr2=0x5624f283b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 -- 172.21.15.78:0/2487767969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624f271bf20 con 0x5624f283d8b0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f27ff860 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009c10 tx=0x7f69b0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5edc35001afd10d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f6997fff640 1 -- 172.21.15.78:0/2487767969 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b00096b0 con 0x5624f283d8b0 2024-02-19T14:57:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f6997fff640 1 -- 172.21.15.78:0/2487767969 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b0011ed0 con 0x5624f283d8b0 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.831+0000 7f6997fff640 1 -- 172.21.15.78:0/2487767969 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0015680 con 0x5624f283d8b0 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 -- 172.21.15.78:0/2487767969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 msgr2=0x5624f27ff860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f27ff860 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009c10 tx=0x7f69b0009190 comp rx=0 tx=0).stop 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 -- 172.21.15.78:0/2487767969 shutdown_connections 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f27ff860 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f2800ae0 0x5624f283ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 --2- 172.21.15.78:0/2487767969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 -- 172.21.15.78:0/2487767969 >> 172.21.15.78:0/2487767969 conn(0x5624f27fc120 msgr2=0x5624f2820b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 -- 172.21.15.78:0/2487767969 shutdown_connections 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.832+0000 7f69bf592740 1 -- 172.21.15.78:0/2487767969 wait complete. 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.833+0000 7f69bf592740 1 -- start start 2024-02-19T14:57:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.833+0000 7f69bf592740 1 Processor -- start 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f284bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f284c4d0 0x5624f281d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f2766c30 con 0x5624f283d8b0 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f270bd30 con 0x5624f284c4d0 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624f243a310 con 0x5624f283a740 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 10 client.0 _ll_register_callbacks cb 0x5624f2713d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f284bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f284bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57166/0 (socket says 172.21.15.78:57166) 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2543165936 learned_addr learned my addr 172.21.15.78:0/2543165936 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b659c640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f284c4d0 0x5624f281d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 msgr2=0x5624f283f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f284c4d0 msgr2=0x5624f281d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f284c4d0 0x5624f281d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2543165936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b0009810 con 0x5624f283d8b0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f284bf90 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0580c0 tx=0x7f69ac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e5f3b176301326 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69957fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69957fa640 1 -- 172.21.15.78:0/2543165936 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac085070 con 0x5624f283d8b0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69957fa640 1 -- 172.21.15.78:0/2543165936 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f69ac060890 con 0x5624f283d8b0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624f24666c0 con 0x5624f283d8b0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69957fa640 1 -- 172.21.15.78:0/2543165936 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69ac056480 con 0x5624f283d8b0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 10 client.8529 fetch_fsmap learned FSMap version 121 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.834+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x5624f27652b0 con 0x5624f283d8b0 2024-02-19T14:57:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69957fa640 1 -- 172.21.15.78:0/2543165936 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5624f24666c0 con 0x5624f283d8b0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69957fa640 1 -- 172.21.15.78:0/2543165936 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69ac019030 con 0x5624f283d8b0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69bf592740 10 client.8529 fetch_fsmap finished waiting for FSMap version 121 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69bf592740 10 client.8529 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69bf592740 20 client.8529 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69bf592740 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f284f1f0 0x5624f284f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69bf592740 4 client.8529 mds_command: new command op to 7351 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 --> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624f24b7270 con 0x5624f284f1f0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69b659c640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f284f1f0 0x5624f284f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69b659c640 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f284f1f0 0x5624f284f5f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41630a6714f50071 server_cookie=88f755fb2880a397 in_seq=0 out_seq=0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.835+0000 7f69957fa640 10 client.8529 ms_handle_connect on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69957fa640 1 -- 172.21.15.78:0/2543165936 <== mds.0 v2:172.21.15.150:6833/1869543632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5624f24b7270 con 0x5624f284f1f0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69957fa640 10 client.8529 handle_command_reply: tid=0 2024-02-19T14:57:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 client.8529 shutdown 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 msgr2=0x5624f284bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f284bf90 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0580c0 tx=0x7f69ac000bc0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 shutdown_connections 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f284f1f0 0x5624f284f5f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f284bf90 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69957fa640 0 client.8529 ms_handle_reset on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f284c4d0 0x5624f281d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 --2- 172.21.15.78:0/2543165936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.836+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 >> 172.21.15.78:0/2543165936 conn(0x5624f2800ae0 msgr2=0x5624f283ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.837+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 shutdown_connections 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.837+0000 7f69bf592740 1 -- 172.21.15.78:0/2543165936 wait complete. 2024-02-19T14:57:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.837+0000 7f69bf592740 20 client.8529 trim_cache size 0 max 16384 2024-02-19T14:57:20.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.853+0000 7f69bf592740 1 Processor -- start 2024-02-19T14:57:20.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.853+0000 7f69bf592740 1 -- start start 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69bf592740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f283be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283c380 0x5624f2855620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69bf592740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f26ec490 con 0x5624f283d8b0 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f27f5660 con 0x5624f283a740 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624f2831f20 con 0x5624f283c380 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f283be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f283be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57174/0 (socket says 172.21.15.78:57174) 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2576759812 learned_addr learned my addr 172.21.15.78:0/2576759812 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283c380 0x5624f2855620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283c380 msgr2=0x5624f2855620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283c380 0x5624f2855620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 msgr2=0x5624f283f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 -- 172.21.15.78:0/2576759812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624f2701a20 con 0x5624f283d8b0 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f283be40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f69ac060360 tx=0x7f69ac01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1519a7a53a347da3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.854+0000 7f69977fe640 1 -- 172.21.15.78:0/2576759812 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac085040 con 0x5624f283d8b0 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69977fe640 1 -- 172.21.15.78:0/2576759812 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac06d070 con 0x5624f283d8b0 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69977fe640 1 -- 172.21.15.78:0/2576759812 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac056480 con 0x5624f283d8b0 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 -- 172.21.15.78:0/2576759812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 msgr2=0x5624f283be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f283be40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f69ac060360 tx=0x7f69ac01c000 comp rx=0 tx=0).stop 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 -- 172.21.15.78:0/2576759812 shutdown_connections 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f283be40 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283c380 0x5624f2855620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 --2- 172.21.15.78:0/2576759812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283a740 0x5624f283f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 -- 172.21.15.78:0/2576759812 >> 172.21.15.78:0/2576759812 conn(0x5624f2800ae0 msgr2=0x5624f284bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 -- 172.21.15.78:0/2576759812 shutdown_connections 2024-02-19T14:57:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.855+0000 7f69bf592740 1 -- 172.21.15.78:0/2576759812 wait complete. 2024-02-19T14:57:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 -- start start 2024-02-19T14:57:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 Processor -- start 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f27fb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283c380 0x5624f28aba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f28a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f283f310 con 0x5624f283d8b0 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624f28325d0 con 0x5624f283c380 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.856+0000 7f69bf592740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624f2837f60 con 0x5624f283a740 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69bf592740 10 client.0 _ll_register_callbacks cb 0x7f69ac057b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f28a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f28a85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57188/0 (socket says 172.21.15.78:57188) 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 -- 172.21.15.78:0/831073457 learned_addr learned my addr 172.21.15.78:0/831073457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b659c640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f27fb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283c380 0x5624f28aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 -- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 msgr2=0x5624f27fb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f27fb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 -- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283c380 msgr2=0x5624f28aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283c380 0x5624f28aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 -- 172.21.15.78:0/831073457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624f27652b0 con 0x5624f283d8b0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b5d9b640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283c380 0x5624f28aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69b6d9d640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f28a85a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f69b0018000 tx=0x7f69b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8ce14f36084547 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69957fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69957fa640 1 -- 172.21.15.78:0/831073457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0023040 con 0x5624f283d8b0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69957fa640 1 -- 172.21.15.78:0/831073457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f69b002c870 con 0x5624f283d8b0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69b0011ed0 con 0x5624f283d8b0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69957fa640 1 -- 172.21.15.78:0/831073457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69b002ca30 con 0x5624f283d8b0 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69bf592740 10 client.8535 fetch_fsmap learned FSMap version 121 2024-02-19T14:57:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x5624f2727780 con 0x5624f283d8b0 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.857+0000 7f69957fa640 1 -- 172.21.15.78:0/831073457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b0011ed0 con 0x5624f283d8b0 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69957fa640 1 -- 172.21.15.78:0/831073457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69b002cb90 con 0x5624f283d8b0 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 10 client.8535 fetch_fsmap finished waiting for FSMap version 121 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 10 client.8535 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 20 client.8535 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f28a8b50 0x5624f28ae990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 4 client.8535 mds_command: new command op to 7351 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 --> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5624f24a3610 con 0x5624f28a8b50 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69b659c640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f28a8b50 0x5624f28ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69957fa640 10 client.8535 ms_handle_connect on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69b659c640 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f28a8b50 0x5624f28ae990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9feac59f5d002e2f server_cookie=57da6dc8ab52f7d1 in_seq=0 out_seq=0 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69957fa640 1 -- 172.21.15.78:0/831073457 <== mds.0 v2:172.21.15.150:6833/1869543632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5624f24a3610 con 0x5624f28a8b50 2024-02-19T14:57:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69957fa640 10 client.8535 handle_command_reply: tid=0 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 1 client.8535 shutdown 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 msgr2=0x5624f28a85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.858+0000 7f69bf592740 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f28a85a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f69b0018000 tx=0x7f69b0004730 comp rx=0 tx=0).stop 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 shutdown_connections 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5624f28a8b50 0x5624f28ae990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5624f283d8b0 0x5624f28a85a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5624f283c380 0x5624f28aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 --2- 172.21.15.78:0/831073457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5624f283a740 0x5624f27fb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69957fa640 0 client.8535 ms_handle_reset on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 >> 172.21.15.78:0/831073457 conn(0x5624f2800ae0 msgr2=0x5624f283ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 shutdown_connections 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 1 -- 172.21.15.78:0/831073457 wait complete. 2024-02-19T14:57:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bf592740 20 client.8535 trim_cache size 0 max 16384 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7351, 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 10.267490128, 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 121, 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 139, 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 139, 2024-02-19T14:57:20.869 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 26.286130872000001 2024-02-19T14:57:20.870 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f699806b810 msgr2=0x7f699806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f699806b810 0x7f699806dcd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f69a8035000 tx=0x7f69a8030000 comp rx=0 tx=0).stop 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 msgr2=0x7f69b80bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 0x7f69b80bf9b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c460 tx=0x7f69ac00c920 comp rx=0 tx=0).stop 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 shutdown_connections 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f699806b810 0x7f699806dcd0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69b80c3800 0x7f69b80bf9b0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69b8150440 0x7f69b8146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 --2- 172.21.15.78:0/127309561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b806ed20 0x7f69b8146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 >> 172.21.15.78:0/127309561 conn(0x7f69b806b270 msgr2=0x7f69b814dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 shutdown_connections 2024-02-19T14:57:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.859+0000 7f69bc8d4640 1 -- 172.21.15.78:0/127309561 wait complete. 2024-02-19T14:57:20.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:57:20.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.975+0000 7f9462dc9640 1 Processor -- start 2024-02-19T14:57:20.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9462dc9640 1 -- start start 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9462dc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c0c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9462dc9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f945c0c5510 0x7f945c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9462dc9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c0c3cf0 con 0x7f945c158f80 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9462dc9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c158b60 con 0x7f945c0c5510 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9462dc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c155f00 con 0x7f945c0c5130 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9461366640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f945c0c5510 0x7f945c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f9460b65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c0c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f945bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f945c158f80 0x7f945c0c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:43466/0 (socket says 172.21.15.78:43466) 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.976+0000 7f945bfff640 1 -- 172.21.15.78:0/1131680167 learned_addr learned my addr 172.21.15.78:0/1131680167 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.977+0000 7f9461366640 1 -- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 msgr2=0x7f945c0c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.977+0000 7f9461366640 1 --2- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c0c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.977+0000 7f9461366640 1 -- 172.21.15.78:0/1131680167 >> v1:172.21.15.150:6789/0 conn(0x7f945c158f80 legacy=0x7f945c0c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.977+0000 7f9461366640 1 -- 172.21.15.78:0/1131680167 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c159400 con 0x7f945c0c5510 2024-02-19T14:57:20.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.977+0000 7f945b7fe640 1 -- 172.21.15.78:0/1131680167 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825305973 0 0) 0x7f945c0c3cf0 con 0x7f945c158f80 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.977+0000 7f9461366640 1 --2- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c0c5510 0x7f945c1559c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9454009d90 tx=0x7f945402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d0fff50d67b52b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f945b7fe640 1 -- 172.21.15.78:0/1131680167 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945402cc50 con 0x7f945c0c5510 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f945b7fe640 1 -- 172.21.15.78:0/1131680167 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945402cdb0 con 0x7f945c0c5510 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f945b7fe640 1 -- 172.21.15.78:0/1131680167 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9454034660 con 0x7f945c0c5510 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 -- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c0c5510 msgr2=0x7f945c1559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 --2- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c0c5510 0x7f945c1559c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9454009d90 tx=0x7f945402bea0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 -- 172.21.15.78:0/1131680167 shutdown_connections 2024-02-19T14:57:20.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 --2- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c0c5510 0x7f945c1559c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 --2- 172.21.15.78:0/1131680167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c0c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 -- 172.21.15.78:0/1131680167 >> 172.21.15.78:0/1131680167 conn(0x7f945c06b4b0 msgr2=0x7f945c0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 -- 172.21.15.78:0/1131680167 shutdown_connections 2024-02-19T14:57:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.978+0000 7f9462dc9640 1 -- 172.21.15.78:0/1131680167 wait complete. 2024-02-19T14:57:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.979+0000 7f9462dc9640 1 Processor -- start 2024-02-19T14:57:20.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.979+0000 7f9462dc9640 1 -- start start 2024-02-19T14:57:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9462dc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9462dc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 0x7f945c148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9462dc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c158f80 0x7f945c148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9462dc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1ebb90 con 0x7f945c0c5510 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9462dc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1ec150 con 0x7f945c158f80 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9462dc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c1548f0 con 0x7f945c0c5130 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f945bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 0x7f945c148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f945bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 0x7f945c148400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57196/0 (socket says 172.21.15.78:57196) 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9460b65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f945bfff640 1 -- 172.21.15.78:0/4035250490 learned_addr learned my addr 172.21.15.78:0/4035250490 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.980+0000 7f9461366640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c158f80 0x7f945c148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f945bfff640 1 -- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 msgr2=0x7f945c154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f945bfff640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f945bfff640 1 -- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c158f80 msgr2=0x7f945c148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f945bfff640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c158f80 0x7f945c148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f945bfff640 1 -- 172.21.15.78:0/4035250490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454009a70 con 0x7f945c0c5510 2024-02-19T14:57:20.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f945bfff640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 0x7f945c148400 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f945000c3f0 tx=0x7f945000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94500074e0 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9450007640 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c148e80 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450004a00 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.981+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c1dfb20 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.982+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c1dfd40 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.982+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9450004b60 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.983+0000 7f94597fa640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f944806b810 0x7f944806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.983+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9450089fa0 con 0x7f945c0c5510 2024-02-19T14:57:20.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.983+0000 7f9460b65640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f944806b810 0x7f944806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:20.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.983+0000 7f9460b65640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f944806b810 0x7f944806dcd0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f944c009b40 tx=0x7f944c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:20.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:20.987+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f94500bddc0 con 0x7f945c0c5510 2024-02-19T14:57:21.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.158+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f945c074990 con 0x7f945c0c5510 2024-02-19T14:57:21.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.161+0000 7f94597fa640 1 -- 172.21.15.78:0/4035250490 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v48) v1 ==== 96+0+0 (secure 0 0 0) 0x7f9450054920 con 0x7f945c0c5510 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.163+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f944806b810 msgr2=0x7f944806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.163+0000 7f9462dc9640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f944806b810 0x7f944806dcd0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f944c009b40 tx=0x7f944c009190 comp rx=0 tx=0).stop 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 msgr2=0x7f945c148400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 0x7f945c148400 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f945000c3f0 tx=0x7f945000c8b0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 shutdown_connections 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f944806b810 0x7f944806dcd0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f945c0c5510 0x7f945c148400 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f945c158f80 0x7f945c148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 --2- 172.21.15.78:0/4035250490 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f945c0c5130 0x7f945c154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 >> 172.21.15.78:0/4035250490 conn(0x7f945c06b4b0 msgr2=0x7f945c157a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 shutdown_connections 2024-02-19T14:57:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.164+0000 7f9462dc9640 1 -- 172.21.15.78:0/4035250490 wait complete. 2024-02-19T14:57:21.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:21.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 Processor -- start 2024-02-19T14:57:21.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 -- start start 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd5940c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd594158f80 0x7fd5940c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5941559c0 0x7fd5940c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd594158b60 con 0x7fd594158f80 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940c3cf0 con 0x7fd5941559c0 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd599148640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd594159400 con 0x7fd5940c5130 2024-02-19T14:57:21.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd58bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd594158f80 0x7fd5940c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.275+0000 7fd58bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd594158f80 0x7fd5940c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57208/0 (socket says 172.21.15.78:57208) 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 -- 172.21.15.78:0/3777062211 learned_addr learned my addr 172.21.15.78:0/3777062211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd59359e640 1 --2- 172.21.15.78:0/3777062211 >> v2:172.21.15.150:3300/0 conn(0x7fd5941559c0 0x7fd5940c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd592d9d640 1 --2- 172.21.15.78:0/3777062211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd5940c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 -- 172.21.15.78:0/3777062211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 msgr2=0x7fd5940c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 --2- 172.21.15.78:0/3777062211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd5940c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 -- 172.21.15.78:0/3777062211 >> v2:172.21.15.150:3300/0 conn(0x7fd5941559c0 msgr2=0x7fd5940c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 --2- 172.21.15.78:0/3777062211 >> v2:172.21.15.150:3300/0 conn(0x7fd5941559c0 0x7fd5940c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 -- 172.21.15.78:0/3777062211 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594158530 con 0x7fd594158f80 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.276+0000 7fd58bfff640 1 --2- 172.21.15.78:0/3777062211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd594158f80 0x7fd5940c5510 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd5940732c0 tx=0x7fd57c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92862293fe7a1c9f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd59259c640 1 -- 172.21.15.78:0/3777062211 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031870 con 0x7fd594158f80 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd59259c640 1 -- 172.21.15.78:0/3777062211 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c0319d0 con 0x7fd594158f80 2024-02-19T14:57:21.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd59259c640 1 -- 172.21.15.78:0/3777062211 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031d60 con 0x7fd594158f80 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 -- 172.21.15.78:0/3777062211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd594158f80 msgr2=0x7fd5940c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 --2- 172.21.15.78:0/3777062211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd594158f80 0x7fd5940c5510 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd5940732c0 tx=0x7fd57c02b910 comp rx=0 tx=0).stop 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 -- 172.21.15.78:0/3777062211 shutdown_connections 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 --2- 172.21.15.78:0/3777062211 >> v2:172.21.15.150:3300/0 conn(0x7fd5941559c0 0x7fd5940c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 --2- 172.21.15.78:0/3777062211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd594158f80 0x7fd5940c5510 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 --2- 172.21.15.78:0/3777062211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd5940c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 -- 172.21.15.78:0/3777062211 >> 172.21.15.78:0/3777062211 conn(0x7fd59406b4b0 msgr2=0x7fd5940c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 -- 172.21.15.78:0/3777062211 shutdown_connections 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.277+0000 7fd599148640 1 -- 172.21.15.78:0/3777062211 wait complete. 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.278+0000 7fd599148640 1 Processor -- start 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 -- start start 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd594154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 0x7fd5941de980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594158f80 0x7fd5941deec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941e0490 con 0x7fd5941559c0 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594155010 con 0x7fd594158f80 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd599148640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941dfc10 con 0x7fd5940c5130 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd58bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 0x7fd5941de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd58bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 0x7fd5941de980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57212/0 (socket says 172.21.15.78:57212) 2024-02-19T14:57:21.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd58bfff640 1 -- 172.21.15.78:0/3396634150 learned_addr learned my addr 172.21.15.78:0/3396634150 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd592d9d640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd594154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.279+0000 7fd59359e640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594158f80 0x7fd5941deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58bfff640 1 -- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 msgr2=0x7fd594154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58bfff640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd594154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58bfff640 1 -- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594158f80 msgr2=0x7fd5941deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58bfff640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594158f80 0x7fd5941deec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58bfff640 1 -- 172.21.15.78:0/3396634150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0094d0 con 0x7fd5941559c0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58bfff640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 0x7fd5941de980 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c03ac60 con 0x7fd5941559c0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c03adc0 con 0x7fd5941559c0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c044630 con 0x7fd5941559c0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5941ebd80 con 0x7fd5941559c0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.280+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5941e01b0 con 0x7fd5941559c0 2024-02-19T14:57:21.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.281+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd594074990 con 0x7fd5941559c0 2024-02-19T14:57:21.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.281+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd57c04d450 con 0x7fd5941559c0 2024-02-19T14:57:21.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.282+0000 7fd58b7fe640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57406b810 0x7fd57406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:21.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.282+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd57c02d040 con 0x7fd5941559c0 2024-02-19T14:57:21.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.282+0000 7fd592d9d640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57406b810 0x7fd57406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:21.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.282+0000 7fd592d9d640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57406b810 0x7fd57406dcd0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd584009a20 tx=0x7fd584009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:21.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.286+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd57c077130 con 0x7fd5941559c0 2024-02-19T14:57:21.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.495+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5940c9400 con 0x7fd5941559c0 2024-02-19T14:57:21.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.495+0000 7fd58b7fe640 1 -- 172.21.15.78:0/3396634150 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4956 (secure 0 0 0) 0x7fd57c07a7f0 con 0x7fd5941559c0 2024-02-19T14:57:21.498 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:21.498 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":121,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":8259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":5,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:12.597030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7351},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7351":{"gid":7351,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/1869543632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1869543632},{"type":"v1","addr":"172.21.15.150:6835","nonce":1869543632}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:21.499 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 121 2024-02-19T14:57:21.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57406b810 msgr2=0x7fd57406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57406b810 0x7fd57406dcd0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd584009a20 tx=0x7fd584009190 comp rx=0 tx=0).stop 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 msgr2=0x7fd5941de980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 0x7fd5941de980 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 shutdown_connections 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57406b810 0x7fd57406dcd0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5941559c0 0x7fd5941de980 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594158f80 0x7fd5941deec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 --2- 172.21.15.78:0/3396634150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5940c5130 0x7fd594154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 >> 172.21.15.78:0/3396634150 conn(0x7fd59406b4b0 msgr2=0x7fd594157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.497+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 shutdown_connections 2024-02-19T14:57:21.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:21.498+0000 7fd599148640 1 -- 172.21.15.78:0/3396634150 wait complete. 2024-02-19T14:57:21.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:12.597030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7351}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7351': {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:21.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:57:21.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.013+0000 7f03f7519640 1 Processor -- start 2024-02-19T14:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.013+0000 7f03f7519640 1 -- start start 2024-02-19T14:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f7519640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03f006d920 0x7f03f015e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f7519640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f0150820 con 0x7f03f0150440 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f7519640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f01537d0 con 0x7f03f006ed20 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f7519640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00c1b60 con 0x7f03f006d920 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f52b5640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f03f0150440 0x7f03f00c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51734/0 (socket says 172.21.15.78:51734) 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f52b5640 1 -- 172.21.15.78:0/2572436201 learned_addr learned my addr 172.21.15.78:0/2572436201 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/2572436201 >> v2:172.21.15.150:3300/0 conn(0x7f03f006d920 0x7f03f015e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113617069 0 0) 0x7f03f0150820 con 0x7f03f0150440 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.014+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc003520 con 0x7f03f0150440 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238925525 0 0) 0x7f03f01537d0 con 0x7f03f006ed20 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f0150820 con 0x7f03f006ed20 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03f5ab6640 1 -- 172.21.15.78:0/2572436201 >> v1:172.21.15.150:6789/0 conn(0x7f03f006ed20 legacy=0x7f03f006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03f5ab6640 1 -- 172.21.15.78:0/2572436201 >> v1:172.21.15.78:6789/0 conn(0x7f03f0150440 legacy=0x7f03f00c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2399240240 0 0) 0x7f03dc003520 con 0x7f03f0150440 2024-02-19T14:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03f5ab6640 1 -- 172.21.15.78:0/2572436201 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f0153d10 con 0x7f03f006d920 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.015+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/2572436201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f006d920 0x7f03f015e730 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f03f0153960 tx=0x7f03e802be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d3b15708f0f206e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec001670 con 0x7f03f006d920 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e802c9c0 con 0x7f03f006d920 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03e7fff640 1 -- 172.21.15.78:0/2572436201 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e802cd30 con 0x7f03f006d920 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 -- 172.21.15.78:0/2572436201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f006d920 msgr2=0x7f03f015e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 --2- 172.21.15.78:0/2572436201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f006d920 0x7f03f015e730 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f03f0153960 tx=0x7f03e802be90 comp rx=0 tx=0).stop 2024-02-19T14:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 -- 172.21.15.78:0/2572436201 shutdown_connections 2024-02-19T14:57:22.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 --2- 172.21.15.78:0/2572436201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f006d920 0x7f03f015e730 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 -- 172.21.15.78:0/2572436201 >> 172.21.15.78:0/2572436201 conn(0x7f03f006b270 msgr2=0x7f03f006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:22.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 -- 172.21.15.78:0/2572436201 shutdown_connections 2024-02-19T14:57:22.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.016+0000 7f03f7519640 1 -- 172.21.15.78:0/2572436201 wait complete. 2024-02-19T14:57:22.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.017+0000 7f03f7519640 1 Processor -- start 2024-02-19T14:57:22.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.017+0000 7f03f7519640 1 -- start start 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f7519640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03f006d920 0x7f03f014bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f7519640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 0x7f03f01522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f7519640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f0150440 0x7f03f01e8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f7519640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f014c420 con 0x7f03f006ed20 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f7519640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f0154a10 con 0x7f03f0150440 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f7519640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f0154270 con 0x7f03f006d920 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 0x7f03f01522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 0x7f03f01522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57220/0 (socket says 172.21.15.78:57220) 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 -- 172.21.15.78:0/3794784728 learned_addr learned my addr 172.21.15.78:0/3794784728 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:22.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f0150440 0x7f03f01e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03f006d920 0x7f03f014bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 -- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03f006d920 msgr2=0x7f03f014bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03f006d920 0x7f03f014bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 -- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f0150440 msgr2=0x7f03f01e8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f0150440 0x7f03f01e8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.018+0000 7f03f4ab4640 1 -- 172.21.15.78:0/3794784728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e8009a70 con 0x7f03f006ed20 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.019+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 0x7f03f01522e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f03e000c490 tx=0x7f03e000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.019+0000 7f03e5ffb640 1 -- 172.21.15.78:0/3794784728 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00052d0 con 0x7f03f006ed20 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.019+0000 7f03e5ffb640 1 -- 172.21.15.78:0/3794784728 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e0005430 con 0x7f03f006ed20 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.019+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f0151b90 con 0x7f03f006ed20 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.019+0000 7f03e5ffb640 1 -- 172.21.15.78:0/3794784728 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e0012870 con 0x7f03f006ed20 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.019+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f0154c60 con 0x7f03f006ed20 2024-02-19T14:57:22.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.020+0000 7f03e5ffb640 1 -- 172.21.15.78:0/3794784728 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f03e00129d0 con 0x7f03f006ed20 2024-02-19T14:57:22.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.021+0000 7f03e5ffb640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03d406b810 0x7f03d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.021+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03d406b810 0x7f03d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.021+0000 7f03e5ffb640 1 -- 172.21.15.78:0/3794784728 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f03e0089dc0 con 0x7f03f006ed20 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.021+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03d406b810 0x7f03d406dcd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0011e0 tx=0x7f03ec001690 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.022+0000 7f03fa1cc740 1 Processor -- start 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.022+0000 7f03fa1cc740 1 -- start start 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.022+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb00f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.022+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cafd4ae0 0x5651cb00eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 0x5651cafd3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651caf8ecd0 con 0x5651cb0118b0 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651caf41f20 con 0x5651cb00e740 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651caf3f450 con 0x5651cafd4ae0 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 0x5651cafd3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 0x5651cafd3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57222/0 (socket says 172.21.15.78:57222) 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 -- 172.21.15.78:0/1480295501 learned_addr learned my addr 172.21.15.78:0/1480295501 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f52b5640 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb00f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cafd4ae0 0x5651cb00eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 -- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cafd4ae0 msgr2=0x5651cb00eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cafd4ae0 0x5651cb00eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 -- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 msgr2=0x5651cb00f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb00f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 -- 172.21.15.78:0/1480295501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651caeeff20 con 0x5651cb0118b0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 0x5651cafd3860 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f03e802c4a0 tx=0x7f03e802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f32998de950e5b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03daffd640 1 -- 172.21.15.78:0/1480295501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8034440 con 0x5651cb0118b0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03daffd640 1 -- 172.21.15.78:0/1480295501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e80345a0 con 0x5651cb0118b0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.023+0000 7f03daffd640 1 -- 172.21.15.78:0/1480295501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8034910 con 0x5651cb0118b0 2024-02-19T14:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 -- 172.21.15.78:0/1480295501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 msgr2=0x5651cafd3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 0x5651cafd3860 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f03e802c4a0 tx=0x7f03e802e000 comp rx=0 tx=0).stop 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 -- 172.21.15.78:0/1480295501 shutdown_connections 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb0118b0 0x5651cafd3860 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cafd4ae0 0x5651cb00eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/1480295501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb00f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 -- 172.21.15.78:0/1480295501 >> 172.21.15.78:0/1480295501 conn(0x5651cafd0120 msgr2=0x5651caff4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 -- 172.21.15.78:0/1480295501 shutdown_connections 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 -- 172.21.15.78:0/1480295501 wait complete. 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 -- start start 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.024+0000 7f03fa1cc740 1 Processor -- start 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb0154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb016540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 0x5651caff19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651caf3ac30 con 0x5651cb016a80 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651caedfd30 con 0x5651cb00e740 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651cac0e310 con 0x5651cb0118b0 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03fa1cc740 10 client.0 _ll_register_callbacks cb 0x5651caee7d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03f5ab6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 0x5651caff19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03f5ab6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 0x5651caff19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57226/0 (socket says 172.21.15.78:57226) 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03f5ab6640 1 -- 172.21.15.78:0/3341266608 learned_addr learned my addr 172.21.15.78:0/3341266608 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03f52b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb0154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.025+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb016540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03f5ab6640 1 -- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 msgr2=0x5651cb016540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb016540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03f5ab6640 1 -- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 msgr2=0x5651cb0154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb0154d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03f5ab6640 1 -- 172.21.15.78:0/3341266608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f0153d10 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 0x5651caff19e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009e50 tx=0x7f03e8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebff37b8d14d8ab0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03cffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03cffff640 1 -- 172.21.15.78:0/3341266608 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8034440 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03cffff640 1 -- 172.21.15.78:0/3341266608 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f03e8034910 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5651cac3a6c0 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03cffff640 1 -- 172.21.15.78:0/3341266608 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03e8034ad0 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 10 client.8556 fetch_fsmap learned FSMap version 121 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x5651caf392b0 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03cffff640 1 -- 172.21.15.78:0/3341266608 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5651cac3a6c0 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03cffff640 1 -- 172.21.15.78:0/3341266608 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f03e8034c30 con 0x5651cb016a80 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 10 client.8556 fetch_fsmap finished waiting for FSMap version 121 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 10 client.8556 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 20 client.8556 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb023220 0x5651cb023600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 4 client.8556 mds_command: new command op to 7351 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.026+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 --> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651cac8b270 con 0x5651cb023220 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.027+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb023220 0x5651cb023600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.027+0000 7f03cffff640 10 client.8556 ms_handle_connect on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.027+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb023220 0x5651cb023600 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3683f09d4645a64 server_cookie=e01107f4fc022225 in_seq=0 out_seq=0 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03cffff640 1 -- 172.21.15.78:0/3341266608 <== mds.0 v2:172.21.15.150:6833/1869543632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5651cac8b270 con 0x5651cb023220 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03cffff640 10 client.8556 handle_command_reply: tid=0 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 client.8556 shutdown 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 msgr2=0x5651caff19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 0x5651caff19e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009e50 tx=0x7f03e8004ab0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 shutdown_connections 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb023220 0x5651cb023600 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb016a80 0x5651caff19e0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03cffff640 0 client.8556 ms_handle_reset on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb016540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3341266608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb00e740 0x5651cb0154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 >> 172.21.15.78:0/3341266608 conn(0x5651cafd4ae0 msgr2=0x5651cafccf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 shutdown_connections 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3341266608 wait complete. 2024-02-19T14:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.028+0000 7f03fa1cc740 20 client.8556 trim_cache size 0 max 16384 2024-02-19T14:57:22.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 Processor -- start 2024-02-19T14:57:22.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 -- start start 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651caff1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb006730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb006c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651caec0490 con 0x5651cb00e740 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651cb015940 con 0x5651cb016a80 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651cafc9ff0 con 0x5651cb0118b0 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03f52b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651caff1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03f52b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651caff1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57228/0 (socket says 172.21.15.78:57228) 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.045+0000 7f03f52b5640 1 -- 172.21.15.78:0/3887121327 learned_addr learned my addr 172.21.15.78:0/3887121327 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:22.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb006730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb006c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f52b5640 1 -- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 msgr2=0x5651cb006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb006730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f52b5640 1 -- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 msgr2=0x5651cb006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb006c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f52b5640 1 -- 172.21.15.78:0/3887121327 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651caed5a20 con 0x5651cb00e740 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03f52b5640 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651caff1250 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001340 tx=0x7f03ec00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde8090541b38e24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03da7fc640 1 -- 172.21.15.78:0/3887121327 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec029340 con 0x5651cb00e740 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03da7fc640 1 -- 172.21.15.78:0/3887121327 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec005ca0 con 0x5651cb00e740 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.046+0000 7f03da7fc640 1 -- 172.21.15.78:0/3887121327 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec006680 con 0x5651cb00e740 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3887121327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 msgr2=0x5651caff1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651caff1250 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001340 tx=0x7f03ec00a000 comp rx=0 tx=0).stop 2024-02-19T14:57:22.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3887121327 shutdown_connections 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb006c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb006730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/3887121327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651caff1250 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3887121327 >> 172.21.15.78:0/3887121327 conn(0x5651cafd4ae0 msgr2=0x5651cb007b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3887121327 shutdown_connections 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 -- 172.21.15.78:0/3887121327 wait complete. 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 -- start start 2024-02-19T14:57:22.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.047+0000 7f03fa1cc740 1 Processor -- start 2024-02-19T14:57:22.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651cb028260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb0800d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb007660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651cb0287a0 con 0x5651cb00e740 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651cb0832a0 con 0x5651cb016a80 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651cb0766c0 con 0x5651cb0118b0 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03fa1cc740 10 client.0 _ll_register_callbacks cb 0x7f03e8043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651cb028260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651cb028260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57234/0 (socket says 172.21.15.78:57234) 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 -- 172.21.15.78:0/2502652057 learned_addr learned my addr 172.21.15.78:0/2502652057 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb0800d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f5ab6640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb007660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 -- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 msgr2=0x5651cb0800d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb0800d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 -- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 msgr2=0x5651cb007660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb007660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.048+0000 7f03f52b5640 1 -- 172.21.15.78:0/2502652057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651caefb780 con 0x5651cb00e740 2024-02-19T14:57:22.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03f52b5640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651cb028260 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001eb0 tx=0x7f03ec001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b244a92daa6df3b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03cffff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03cffff640 1 -- 172.21.15.78:0/2502652057 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec029930 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03cffff640 1 -- 172.21.15.78:0/2502652057 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f03ec01b040 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03e80345a0 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03cffff640 1 -- 172.21.15.78:0/2502652057 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03ec029340 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 10 client.8562 fetch_fsmap learned FSMap version 121 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x5651caee1f10 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03cffff640 1 -- 172.21.15.78:0/2502652057 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e80345a0 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03cffff640 1 -- 172.21.15.78:0/2502652057 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f03ec030070 con 0x5651cb00e740 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 10 client.8562 fetch_fsmap finished waiting for FSMap version 121 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 10 client.8562 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 20 client.8562 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:22.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb007ba0 0x5651cb006500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 4 client.8562 mds_command: new command op to 7351 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 --> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5651cac77610 con 0x5651cb007ba0 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.049+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb007ba0 0x5651cb006500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03f4ab4640 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb007ba0 0x5651cb006500 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1df6f0dc06e0e49 server_cookie=f37ee0684f872a4 in_seq=0 out_seq=0 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03cffff640 10 client.8562 ms_handle_connect on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03cffff640 1 -- 172.21.15.78:0/2502652057 <== mds.0 v2:172.21.15.150:6833/1869543632 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5651cac77610 con 0x5651cb007ba0 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03cffff640 10 client.8562 handle_command_reply: tid=0 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 client.8562 shutdown 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 msgr2=0x5651cb028260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651cb028260 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f03ec001eb0 tx=0x7f03ec001150 comp rx=0 tx=0).stop 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 shutdown_connections 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5651cb016a80 0x5651cb007660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5651cb0118b0 0x5651cb0800d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.150:6833/1869543632,v1:172.21.15.150:6835/1869543632] conn(0x5651cb007ba0 0x5651cb006500 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 --2- 172.21.15.78:0/2502652057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651cb00e740 0x5651cb028260 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03cffff640 0 client.8562 ms_handle_reset on v2:172.21.15.150:6833/1869543632 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 >> 172.21.15.78:0/2502652057 conn(0x5651cafd4ae0 msgr2=0x5651caff15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 shutdown_connections 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 1 -- 172.21.15.78:0/2502652057 wait complete. 2024-02-19T14:57:22.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.050+0000 7f03fa1cc740 20 client.8562 trim_cache size 0 max 16384 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7351, 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 11.459005588, 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 121, 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 139, 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 139, 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 27.477643886999999 2024-02-19T14:57:22.061 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:22.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03d406b810 msgr2=0x7f03d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03d406b810 0x7f03d406dcd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0011e0 tx=0x7f03ec001690 comp rx=0 tx=0).stop 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 msgr2=0x7f03f01522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 0x7f03f01522e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f03e000c490 tx=0x7f03e000c950 comp rx=0 tx=0).stop 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 shutdown_connections 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03d406b810 0x7f03d406dcd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03f006ed20 0x7f03f01522e0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f0150440 0x7f03f01e8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 --2- 172.21.15.78:0/3794784728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03f006d920 0x7f03f014bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 >> 172.21.15.78:0/3794784728 conn(0x7f03f006b270 msgr2=0x7f03f006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 shutdown_connections 2024-02-19T14:57:22.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:22.051+0000 7f03f7519640 1 -- 172.21.15.78:0/3794784728 wait complete. 2024-02-19T14:57:22.064 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:59.000185+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7351": { "addr": "172.21.15.150:6835/1869543632", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1869543632, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1869543632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7351, "incarnation": 118, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 139, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-19T14:57:12.597030+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7351 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/3153272641", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3153272641, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3153272641, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 119, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7522, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6836/1973988605", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1973988605, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1973988605, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 118, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8259, "incarnation": 0, "join_fscid": 10, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 5 }, { "addr": "172.21.15.78:6837/3197225278", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3197225278, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3197225278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:57:22.065 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 10, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:57:22.065 INFO:tasks.cephfs.test_failover:current = [{'gid': 8262, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3197225278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3197225278}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3197225278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}, {'gid': 7522, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3153272641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3153272641}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3153272641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 119}, {'gid': 8259, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 5, 'addr': '172.21.15.78:6836/1973988605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1973988605}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1973988605}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 118}, {'gid': 7351, 'name': 'd', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.150:6835/1869543632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1869543632}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1869543632}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:57:22.066 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-02-19T14:57:22.066 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T14:57:24.636 INFO:tasks.ceph.mds.d.smithi150.stderr: -451> 2024-02-19T14:57:14.928+0000 7f178f783640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:24.661 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:57:26.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:57:26.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:57:27.066 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:27.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.178+0000 7f8e9bfbd640 1 Processor -- start 2024-02-19T14:57:27.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.178+0000 7f8e9bfbd640 1 -- start start 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e9bfbd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e94153b60 0x7f8e940ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e9bfbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 0x7f8e9406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e9bfbd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e940c35e0 con 0x7f8e940c3200 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e9bfbd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9414d3c0 con 0x7f8e94153b60 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e9bfbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940acad0 con 0x7f8e940c4350 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e99d59640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8e940c3200 0x7f8e940718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51742/0 (socket says 172.21.15.78:51742) 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e99d59640 1 -- 172.21.15.78:0/2240277401 learned_addr learned my addr 172.21.15.78:0/2240277401 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e99558640 1 --2- 172.21.15.78:0/2240277401 >> v2:172.21.15.150:3300/0 conn(0x7f8e94153b60 0x7f8e940ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e9a55a640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 0x7f8e9406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.179+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178851558 0 0) 0x7f8e940c35e0 con 0x7f8e940c3200 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e80003520 con 0x7f8e940c3200 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635295295 0 0) 0x7f8e80003520 con 0x7f8e940c3200 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 msgr2=0x7f8e9406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 0x7f8e9406ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 >> v2:172.21.15.150:3300/0 conn(0x7f8e94153b60 msgr2=0x7f8e940ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 --2- 172.21.15.78:0/2240277401 >> v2:172.21.15.150:3300/0 conn(0x7f8e94153b60 0x7f8e940ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940ad000 con 0x7f8e940c3200 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f8e90001670 con 0x7f8e940c3200 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e99558640 1 --2- 172.21.15.78:0/2240277401 >> v2:172.21.15.150:3300/0 conn(0x7f8e94153b60 0x7f8e940ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 >> v1:172.21.15.78:6789/0 conn(0x7f8e940c3200 legacy=0x7f8e940718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e80003970 0x7f8e80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 0x7f8e800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e80004d00 0x7f8e80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940acad0 con 0x7f8e80004d00 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8000b980 con 0x7f8e80003970 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8000bac0 con 0x7f8e940c4350 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e9a55a640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e80004d00 0x7f8e80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e99d59640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e80003970 0x7f8e80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e99558640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 0x7f8e800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e9a55a640 1 -- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 msgr2=0x7f8e800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e9a55a640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e940c4350 0x7f8e800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e9a55a640 1 -- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e80003970 msgr2=0x7f8e80003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e9a55a640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e80003970 0x7f8e80003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.180+0000 7f8e9a55a640 1 -- 172.21.15.78:0/2240277401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e80003520 con 0x7f8e80004d00 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.181+0000 7f8e9a55a640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e80004d00 0x7f8e80005180 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009fc0 tx=0x7f8e800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819b3be3e6870abf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.181+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c02cc20 con 0x7f8e80004d00 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.181+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c00f040 con 0x7f8e80004d00 2024-02-19T14:57:27.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.181+0000 7f8e98d57640 1 -- 172.21.15.78:0/2240277401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c034440 con 0x7f8e80004d00 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/2240277401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e80004d00 msgr2=0x7f8e80005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/2240277401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e80004d00 0x7f8e80005180 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009fc0 tx=0x7f8e800057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9a55a640 1 -- 172.21.15.78:0/2240277401 reap_dead start 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/2240277401 shutdown_connections 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/2240277401 >> 172.21.15.78:0/2240277401 conn(0x7f8e940c7800 msgr2=0x7f8e940c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/2240277401 shutdown_connections 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.182+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/2240277401 wait complete. 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.183+0000 7f8e9bfbd640 1 Processor -- start 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.183+0000 7f8e9bfbd640 1 -- start start 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9bfbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 0x7f8e94156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9bfbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e94148f10 0x7f8e94155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9bfbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e941ec620 0x7f8e94147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9bfbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e94148da0 con 0x7f8e941ec240 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9bfbd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940bc880 con 0x7f8e94148f10 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9bfbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940bd0e0 con 0x7f8e941ec620 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 0x7f8e94156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 0x7f8e94156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57248/0 (socket says 172.21.15.78:57248) 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 -- 172.21.15.78:0/865215991 learned_addr learned my addr 172.21.15.78:0/865215991 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99558640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e94148f10 0x7f8e94155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e9a55a640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e941ec620 0x7f8e94147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 -- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e941ec620 msgr2=0x7f8e94147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e941ec620 0x7f8e94147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 -- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e94148f10 msgr2=0x7f8e94155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e94148f10 0x7f8e94155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 -- 172.21.15.78:0/865215991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e94157370 con 0x7f8e941ec240 2024-02-19T14:57:27.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e99d59640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 0x7f8e94156e30 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f8e90007110 tx=0x7f8e900075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.184+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000f960 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.185+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9000fae0 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.185+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e940bc4e0 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.185+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000fe90 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.185+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e94155fc0 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.186+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e94074990 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.186+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8e9001d460 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.186+0000 7f8e8a7fc640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e7806b840 0x7f8e7806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.186+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8e9008a3f0 con 0x7f8e941ec240 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.186+0000 7f8e99558640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e7806b840 0x7f8e7806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:27.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.187+0000 7f8e99558640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e7806b840 0x7f8e7806dd00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8e94149350 tx=0x7f8e84009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:27.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.191+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8e90054d10 con 0x7f8e941ec240 2024-02-19T14:57:27.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.398+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e940c4130 con 0x7f8e941ec240 2024-02-19T14:57:27.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.398+0000 7f8e8a7fc640 1 -- 172.21.15.78:0/865215991 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+4957 (secure 0 0 0) 0x7f8e900583d0 con 0x7f8e941ec240 2024-02-19T14:57:27.401 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:27.401 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":125,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:25.636416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"c","rank":0,"incarnation":123,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:27.402 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 125 2024-02-19T14:57:27.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e7806b840 msgr2=0x7f8e7806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:27.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e7806b840 0x7f8e7806dd00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8e94149350 tx=0x7f8e84009190 comp rx=0 tx=0).stop 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 msgr2=0x7f8e94156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 0x7f8e94156e30 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f8e90007110 tx=0x7f8e900075d0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 shutdown_connections 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e7806b840 0x7f8e7806dd00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e941ec620 0x7f8e94147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e94148f10 0x7f8e94155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 --2- 172.21.15.78:0/865215991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e941ec240 0x7f8e94156e30 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 >> 172.21.15.78:0/865215991 conn(0x7f8e940c7800 msgr2=0x7f8e94074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.400+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 shutdown_connections 2024-02-19T14:57:27.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:27.401+0000 7f8e9bfbd640 1 -- 172.21.15.78:0/865215991 wait complete. 2024-02-19T14:57:27.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 125, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:25.636416+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 140, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8259': {'gid': 8259, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:rejoin', 'state_seq': 10, 'addr': '172.21.15.78:6836/1973988605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1973988605}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1973988605}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:27.817 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8259:up:rejoin 2024-02-19T14:57:28.818 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:28.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.917+0000 7effb31ac640 1 Processor -- start 2024-02-19T14:57:28.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.917+0000 7effb31ac640 1 -- start start 2024-02-19T14:57:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.918+0000 7effb31ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 0x7effac0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.918+0000 7effb31ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac1551f0 con 0x7effac0c8e00 2024-02-19T14:57:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.918+0000 7effb31ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac155490 con 0x7effac0c5730 2024-02-19T14:57:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.918+0000 7effb31ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac155770 con 0x7effac152700 2024-02-19T14:57:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effb1749640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7effac0c8e00 0x7effac15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41356/0 (socket says 172.21.15.78:41356) 2024-02-19T14:57:28.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effb1749640 1 -- 172.21.15.78:0/156561020 learned_addr learned my addr 172.21.15.78:0/156561020 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effb0f48640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 0x7effac0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368736825 0 0) 0x7effac1551f0 con 0x7effac0c8e00 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff9c003520 con 0x7effac0c8e00 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3685273046 0 0) 0x7eff9c003520 con 0x7effac0c8e00 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 msgr2=0x7effac0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 0x7effac0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 >> v1:172.21.15.150:6789/0 conn(0x7effac0c5730 legacy=0x7effac06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac155a50 con 0x7effac0c8e00 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7effa40019f0 con 0x7effac0c8e00 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 >> v1:172.21.15.78:6789/0 conn(0x7effac0c8e00 legacy=0x7effac15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 0x7eff9c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff9c003ee0 0x7eff9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff9c004d00 0x7eff9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac155770 con 0x7eff9c004d00 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c00b9e0 con 0x7eff9c003ee0 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c00bba0 con 0x7effac152700 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effb1749640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff9c004d00 0x7eff9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effb0f48640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 0x7eff9c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.919+0000 7effabfff640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff9c003ee0 0x7eff9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effb1749640 1 -- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 msgr2=0x7eff9c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effb1749640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac152700 0x7eff9c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effb1749640 1 -- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff9c003ee0 msgr2=0x7eff9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effb1749640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff9c003ee0 0x7eff9c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effb1749640 1 -- 172.21.15.78:0/156561020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c003520 con 0x7eff9c004d00 2024-02-19T14:57:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effb1749640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff9c004d00 0x7eff9c005180 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7effa400a840 tx=0x7eff9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8ec71cdb2d3b9b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa402cea0 con 0x7eff9c004d00 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.920+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa402f070 con 0x7eff9c004d00 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effab7fe640 1 -- 172.21.15.78:0/156561020 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa40348b0 con 0x7eff9c004d00 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb31ac640 1 -- 172.21.15.78:0/156561020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff9c004d00 msgr2=0x7eff9c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb31ac640 1 --2- 172.21.15.78:0/156561020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff9c004d00 0x7eff9c005180 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7effa400a840 tx=0x7eff9c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb1749640 1 -- 172.21.15.78:0/156561020 reap_dead start 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb31ac640 1 -- 172.21.15.78:0/156561020 shutdown_connections 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb31ac640 1 -- 172.21.15.78:0/156561020 >> 172.21.15.78:0/156561020 conn(0x7effac0cac90 msgr2=0x7effac0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb31ac640 1 -- 172.21.15.78:0/156561020 shutdown_connections 2024-02-19T14:57:28.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.921+0000 7effb31ac640 1 -- 172.21.15.78:0/156561020 wait complete. 2024-02-19T14:57:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.922+0000 7effb31ac640 1 Processor -- start 2024-02-19T14:57:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.922+0000 7effb31ac640 1 -- start start 2024-02-19T14:57:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb31ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effac1ed830 0x7effac1edc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb31ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 0x7effac0c27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb31ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac149360 0x7effac0c2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb31ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac0ac400 con 0x7effac0c4550 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb31ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac0c2350 con 0x7effac1ed830 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb31ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac0c3260 con 0x7effac149360 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effabfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 0x7effac0c27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effabfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 0x7effac0c27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50578/0 (socket says 172.21.15.78:50578) 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effabfff640 1 -- 172.21.15.78:0/2587025711 learned_addr learned my addr 172.21.15.78:0/2587025711 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.923+0000 7effb0f48640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effac1ed830 0x7effac1edc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effb1749640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac149360 0x7effac0c2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effabfff640 1 -- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac149360 msgr2=0x7effac0c2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effabfff640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac149360 0x7effac0c2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effabfff640 1 -- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effac1ed830 msgr2=0x7effac1edc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effabfff640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effac1ed830 0x7effac1edc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effabfff640 1 -- 172.21.15.78:0/2587025711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0aa350 con 0x7effac0c4550 2024-02-19T14:57:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effabfff640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 0x7effac0c27e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7eff9800f7c0 tx=0x7eff9800fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800fed0 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff980052e0 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac0c3780 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800d970 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.924+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac1ee2e0 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.925+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac0c24e0 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.925+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7eff9800dad0 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.926+0000 7effa97fa640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff9006b810 0x7eff9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.926+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7eff9808a510 con 0x7effac0c4550 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.926+0000 7effb0f48640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff9006b810 0x7eff9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.926+0000 7effb0f48640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff9006b810 0x7eff9006dcd0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7effa0009b00 tx=0x7effa0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:28.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:28.930+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7eff98054cf0 con 0x7effac0c4550 2024-02-19T14:57:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.135+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effac074990 con 0x7effac0c4550 2024-02-19T14:57:29.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.136+0000 7effa97fa640 1 -- 172.21.15.78:0/2587025711 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4957 (secure 0 0 0) 0x7eff980583b0 con 0x7effac0c4550 2024-02-19T14:57:29.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:29.139 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:26.646012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":11,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:29.139 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 126 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff9006b810 msgr2=0x7eff9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff9006b810 0x7eff9006dcd0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7effa0009b00 tx=0x7effa0008040 comp rx=0 tx=0).stop 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 msgr2=0x7effac0c27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 0x7effac0c27e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7eff9800f7c0 tx=0x7eff9800fc80 comp rx=0 tx=0).stop 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 shutdown_connections 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff9006b810 0x7eff9006dcd0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effac0c4550 0x7effac0c27e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effac149360 0x7effac0c2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 --2- 172.21.15.78:0/2587025711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effac1ed830 0x7effac1edc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 >> 172.21.15.78:0/2587025711 conn(0x7effac0cac90 msgr2=0x7effac14ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 shutdown_connections 2024-02-19T14:57:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.139+0000 7effb31ac640 1 -- 172.21.15.78:0/2587025711 wait complete. 2024-02-19T14:57:29.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 126, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:56:59.000185+0000', 'modified': '2024-02-19T14:57:26.646012+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 140, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8259}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8259': {'gid': 8259, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.78:6836/1973988605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1973988605}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1973988605}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:29.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:57:29.555 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:57:29.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.654+0000 7f3eeeeeb640 1 Processor -- start 2024-02-19T14:57:29.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.654+0000 7f3eeeeeb640 1 -- start start 2024-02-19T14:57:29.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eeeeeb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3ee80add70 0x7f3ee80c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eeeeeb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee80c2f60 0x7f3ee806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eeeeeb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80ae670 con 0x7f3ee806f530 2024-02-19T14:57:29.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eeeeeb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee80716b0 con 0x7f3ee80add70 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eeeeeb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80c33a0 con 0x7f3ee80c2f60 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eecc87640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3ee80add70 0x7f3ee80c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eed488640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3ee806f530 0x7f3ee80705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41366/0 (socket says 172.21.15.78:41366) 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eed488640 1 -- 172.21.15.78:0/4000027614 learned_addr learned my addr 172.21.15.78:0/4000027614 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/4000027614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee80c2f60 0x7f3ee806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eecc87640 1 -- 172.21.15.78:0/4000027614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee80c2f60 msgr2=0x7f3ee806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eecc87640 1 --2- 172.21.15.78:0/4000027614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee80c2f60 0x7f3ee806eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eecc87640 1 -- 172.21.15.78:0/4000027614 >> v1:172.21.15.78:6789/0 conn(0x7f3ee806f530 legacy=0x7f3ee80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3eecc87640 1 -- 172.21.15.78:0/4000027614 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee8153780 con 0x7f3ee80add70 2024-02-19T14:57:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.655+0000 7f3ee77fe640 1 -- 172.21.15.78:0/4000027614 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930901351 0 0) 0x7f3ee80ae670 con 0x7f3ee806f530 2024-02-19T14:57:29.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.657+0000 7f3eecc87640 1 --2- 172.21.15.78:0/4000027614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c4980 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0097a0 tx=0x7f3edc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33314e0f873336ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.660+0000 7f3ee77fe640 1 -- 172.21.15.78:0/4000027614 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc02e070 con 0x7f3ee80add70 2024-02-19T14:57:29.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.660+0000 7f3ee77fe640 1 -- 172.21.15.78:0/4000027614 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc0315a0 con 0x7f3ee80add70 2024-02-19T14:57:29.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.660+0000 7f3ee77fe640 1 -- 172.21.15.78:0/4000027614 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc031930 con 0x7f3ee80add70 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/4000027614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 msgr2=0x7f3ee80c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/4000027614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c4980 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0097a0 tx=0x7f3edc02b840 comp rx=0 tx=0).stop 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/4000027614 shutdown_connections 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/4000027614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee80c2f60 0x7f3ee806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/4000027614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c4980 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/4000027614 >> 172.21.15.78:0/4000027614 conn(0x7f3ee80c85e0 msgr2=0x7f3ee80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/4000027614 shutdown_connections 2024-02-19T14:57:29.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.661+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/4000027614 wait complete. 2024-02-19T14:57:29.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.662+0000 7f3eeeeeb640 1 Processor -- start 2024-02-19T14:57:29.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.662+0000 7f3eeeeeb640 1 -- start start 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eeeeeb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee806f530 0x7f3ee81546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eeeeeb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eeeeeb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ee80c2f60 0x7f3ee80ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eeeeeb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8155ef0 con 0x7f3ee80add70 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eeeeeb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8154df0 con 0x7f3ee80c2f60 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eeeeeb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee806c130 con 0x7f3ee806f530 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50608/0 (socket says 172.21.15.78:50608) 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2143695785 learned_addr learned my addr 172.21.15.78:0/2143695785 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:29.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee806f530 msgr2=0x7f3ee81546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eecc87640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee806f530 0x7f3ee81546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3eed488640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ee80c2f60 0x7f3ee80ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee806f530 0x7f3ee81546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ee80c2f60 msgr2=0x7f3ee80ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ee80c2f60 0x7f3ee80ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2143695785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc009400 con 0x7f3ee80add70 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.663+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c1b60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e870 tx=0x7f3ed400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.664+0000 7f3ee57fa640 1 -- 172.21.15.78:0/2143695785 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40094f0 con 0x7f3ee80add70 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.664+0000 7f3ee57fa640 1 -- 172.21.15.78:0/2143695785 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed400c940 con 0x7f3ee80add70 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.664+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee806c410 con 0x7f3ee80add70 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.664+0000 7f3ee57fa640 1 -- 172.21.15.78:0/2143695785 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed400ccf0 con 0x7f3ee80add70 2024-02-19T14:57:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.664+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee80c20a0 con 0x7f3ee80add70 2024-02-19T14:57:29.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.665+0000 7f3ee57fa640 1 -- 172.21.15.78:0/2143695785 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3ed4013470 con 0x7f3ee80add70 2024-02-19T14:57:29.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.666+0000 7f3ee57fa640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ecc06b810 0x7f3ecc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.666+0000 7f3ee57fa640 1 -- 172.21.15.78:0/2143695785 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f3ed408a520 con 0x7f3ee80add70 2024-02-19T14:57:29.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.666+0000 7f3eecc87640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ecc06b810 0x7f3ecc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.666+0000 7f3eecc87640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ecc06b810 0x7f3ecc06dcd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3edc036000 tx=0x7f3edc03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 Processor -- start 2024-02-19T14:57:29.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 -- start start 2024-02-19T14:57:29.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 0x5571712238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 0x557171222b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712258b0 0x5571711e7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571711a2cd0 con 0x5571711e8ae0 2024-02-19T14:57:29.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557171155f20 con 0x5571712258b0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.667+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x557171153450 con 0x557171222740 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 0x557171222b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 0x557171222b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50620/0 (socket says 172.21.15.78:50620) 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2382968569 learned_addr learned my addr 172.21.15.78:0/2382968569 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2382968569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 msgr2=0x5571712238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 0x5571712238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2382968569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712258b0 msgr2=0x5571711e7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712258b0 0x5571711e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 -- 172.21.15.78:0/2382968569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557171103f20 con 0x5571711e8ae0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 0x557171222b20 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4088780 tx=0x7f3ed40584f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dd7e4b69aaf2a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ed27fc640 1 -- 172.21.15.78:0/2382968569 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4056d60 con 0x5571711e8ae0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ed27fc640 1 -- 172.21.15.78:0/2382968569 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed40550d0 con 0x5571711e8ae0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.668+0000 7f3ed27fc640 1 -- 172.21.15.78:0/2382968569 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4060f00 con 0x5571711e8ae0 2024-02-19T14:57:29.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/2382968569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 msgr2=0x557171222b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 0x557171222b20 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4088780 tx=0x7f3ed40584f0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/2382968569 shutdown_connections 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571711e8ae0 0x557171222b20 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712258b0 0x5571711e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/2382968569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 0x5571712238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/2382968569 >> 172.21.15.78:0/2382968569 conn(0x5571711e4120 msgr2=0x557171208b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/2382968569 shutdown_connections 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/2382968569 wait complete. 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.669+0000 7f3ef1ba0740 1 -- start start 2024-02-19T14:57:29.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 Processor -- start 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x55717121be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x557171234140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171205600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55717114ec30 con 0x557171222740 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571710f3d30 con 0x5571712208f0 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x557170e22310 con 0x5571712258b0 2024-02-19T14:57:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3ef1ba0740 10 client.0 _ll_register_callbacks cb 0x5571710fbd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3eecc87640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x55717121be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3eecc87640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x55717121be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50626/0 (socket says 172.21.15.78:50626) 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.670+0000 7f3eecc87640 1 -- 172.21.15.78:0/1822508447 learned_addr learned my addr 172.21.15.78:0/1822508447 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eed488640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171205600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x557171234140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eecc87640 1 -- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 msgr2=0x557171234140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eecc87640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x557171234140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eecc87640 1 -- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 msgr2=0x557171205600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eecc87640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171205600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eecc87640 1 -- 172.21.15.78:0/1822508447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee80c20a0 con 0x557171222740 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3eecc87640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x55717121be90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f3edc034040 tx=0x7f3edc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89790c04ae5d837 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ec77fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1822508447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc046610 con 0x557171222740 2024-02-19T14:57:29.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1822508447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f3edc051d30 con 0x557171222740 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557170e4e6c0 con 0x557171222740 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1822508447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3edc046770 con 0x557171222740 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ef1ba0740 10 client.8595 fetch_fsmap learned FSMap version 126 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55717114d2b0 con 0x557171222740 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1822508447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557170e4e6c0 con 0x557171222740 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1822508447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 126) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3edc0468d0 con 0x557171222740 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.671+0000 7f3ef1ba0740 10 client.8595 fetch_fsmap finished waiting for FSMap version 126 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ef1ba0740 10 client.8595 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ef1ba0740 20 client.8595 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:29.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x5571712373d0 0x5571712377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ef1ba0740 4 client.8595 mds_command: new command op to 8259 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 --> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557170e9f270 con 0x5571712373d0 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x5571712373d0 0x5571712377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ec77fe640 10 client.8595 ms_handle_connect on v2:172.21.15.78:6834/1973988605 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.672+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x5571712373d0 0x5571712377b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df4c6c0d276599f1 server_cookie=911391a573322ab2 in_seq=0 out_seq=0 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1822508447 <== mds.0 v2:172.21.15.78:6834/1973988605 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557170e9f270 con 0x5571712373d0 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ec77fe640 10 client.8595 handle_command_reply: tid=0 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 client.8595 shutdown 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 msgr2=0x55717121be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x55717121be90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f3edc034040 tx=0x7f3edc0046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 shutdown_connections 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x5571712373d0 0x5571712377b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171205600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ec77fe640 0 client.8595 ms_handle_reset on v2:172.21.15.78:6834/1973988605 2024-02-19T14:57:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x557171234140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1822508447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x55717121be90 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 >> 172.21.15.78:0/1822508447 conn(0x5571711e8ae0 msgr2=0x557171223e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 shutdown_connections 2024-02-19T14:57:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1822508447 wait complete. 2024-02-19T14:57:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.673+0000 7f3ef1ba0740 20 client.8595 trim_cache size 0 max 16384 2024-02-19T14:57:29.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.689+0000 7f3ef1ba0740 1 Processor -- start 2024-02-19T14:57:29.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 -- start start 2024-02-19T14:57:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171220d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 0x557171208ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 0x557171219f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571710d4490 con 0x5571712258b0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571711e3bd0 con 0x5571712208f0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5571712218c0 con 0x557171222740 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3eed488640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 0x557171219f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3eed488640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 0x557171219f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50634/0 (socket says 172.21.15.78:50634) 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.690+0000 7f3eed488640 1 -- 172.21.15.78:0/3169117753 learned_addr learned my addr 172.21.15.78:0/3169117753 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eed488640 1 -- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 msgr2=0x557171208ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eed488640 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 0x557171208ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eed488640 1 -- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 msgr2=0x557171220d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eecc87640 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171220d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eed488640 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171220d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eed488640 1 -- 172.21.15.78:0/3169117753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571710e9a20 con 0x5571712258b0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3eed488640 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 0x557171219f30 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000f80 tx=0x7f3ee0007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34002d6a1f4b1747 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ed1ffb640 1 -- 172.21.15.78:0/3169117753 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0001b90 con 0x5571712258b0 2024-02-19T14:57:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ed1ffb640 1 -- 172.21.15.78:0/3169117753 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee000d070 con 0x5571712258b0 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ed1ffb640 1 -- 172.21.15.78:0/3169117753 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0001ef0 con 0x5571712258b0 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/3169117753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 msgr2=0x557171219f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 0x557171219f30 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000f80 tx=0x7f3ee0007800 comp rx=0 tx=0).stop 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/3169117753 shutdown_connections 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571712258b0 0x557171219f30 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557171222740 0x557171208ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/3169117753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x557171220d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.691+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/3169117753 >> 172.21.15.78:0/3169117753 conn(0x5571711e8ae0 msgr2=0x55717121bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.692+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/3169117753 shutdown_connections 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.692+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/3169117753 wait complete. 2024-02-19T14:57:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.692+0000 7f3ef1ba0740 1 -- start start 2024-02-19T14:57:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.692+0000 7f3ef1ba0740 1 Processor -- start 2024-02-19T14:57:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x5571712304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x5571712381c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x5571712901a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571712091c0 con 0x557171222740 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571712093d0 con 0x5571712208f0 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5571712379c0 con 0x5571712258b0 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ef1ba0740 10 client.0 _ll_register_callbacks cb 0x7f3ed4009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x5571712381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x5571712381c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50646/0 (socket says 172.21.15.78:50646) 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 -- 172.21.15.78:0/1685177993 learned_addr learned my addr 172.21.15.78:0/1685177993 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3eecc87640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x5571712304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 -- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 msgr2=0x5571712901a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3eed488640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x5571712901a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x5571712901a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 -- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 msgr2=0x5571712304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x5571712304d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 -- 172.21.15.78:0/1685177993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee0007050 con 0x557171222740 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3eed488640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x5571712901a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:29.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ee7fff640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x5571712381c0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4088860 tx=0x7f3ed40886e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470fdc5628570259 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.693+0000 7f3ec77fe640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1685177993 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4060ce0 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1685177993 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f3ed4060e40 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ed40550d0 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1685177993 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ed408a0b0 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 10 client.8601 fetch_fsmap learned FSMap version 126 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55717110f780 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1685177993 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed407b080 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1685177993 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 126) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3ed4011070 con 0x557171222740 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 10 client.8601 fetch_fsmap finished waiting for FSMap version 126 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 10 client.8601 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 20 client.8601 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:29.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x557171290750 0x557171246620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 4 client.8601 mds_command: new command op to 8259 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 --> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557170e8b610 con 0x557171290750 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3eecc87640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x557171290750 0x557171246620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3ec77fe640 10 client.8601 ms_handle_connect on v2:172.21.15.78:6834/1973988605 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.694+0000 7f3eecc87640 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x557171290750 0x557171246620 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aac58c4273ef5c9 server_cookie=cce1df5c4e9a48e3 in_seq=0 out_seq=0 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ec77fe640 1 -- 172.21.15.78:0/1685177993 <== mds.0 v2:172.21.15.78:6834/1973988605 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x557170e8b610 con 0x557171290750 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ec77fe640 10 client.8601 handle_command_reply: tid=0 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 client.8601 shutdown 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 msgr2=0x5571712381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x5571712381c0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4088860 tx=0x7f3ed40886e0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 shutdown_connections 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:6834/1973988605,v1:172.21.15.78:6836/1973988605] conn(0x557171290750 0x557171246620 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557171222740 0x5571712381c0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ec77fe640 0 client.8601 ms_handle_reset on v2:172.21.15.78:6834/1973988605 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5571712258b0 0x5571712901a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 --2- 172.21.15.78:0/1685177993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571712208f0 0x5571712304d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 >> 172.21.15.78:0/1685177993 conn(0x5571711e8ae0 msgr2=0x557171208ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 shutdown_connections 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 1 -- 172.21.15.78:0/1685177993 wait complete. 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3ef1ba0740 20 client.8601 trim_cache size 0 max 16384 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:57:29.704 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 8259, 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 5.0652233679999998, 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 126, 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 140, 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 140, 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 35.133902843000001 2024-02-19T14:57:29.705 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:29.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ecc06b810 msgr2=0x7f3ecc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ecc06b810 0x7f3ecc06dcd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3edc036000 tx=0x7f3edc03d040 comp rx=0 tx=0).stop 2024-02-19T14:57:29.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 msgr2=0x7f3ee80c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.695+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c1b60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e870 tx=0x7f3ed400ed30 comp rx=0 tx=0).stop 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 shutdown_connections 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ecc06b810 0x7f3ecc06dcd0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ee80add70 0x7f3ee80c1b60 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ee80c2f60 0x7f3ee80ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 --2- 172.21.15.78:0/2143695785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ee806f530 0x7f3ee81546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 >> 172.21.15.78:0/2143695785 conn(0x7f3ee80c85e0 msgr2=0x7f3ee80a9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 shutdown_connections 2024-02-19T14:57:29.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.696+0000 7f3eeeeeb640 1 -- 172.21.15.78:0/2143695785 wait complete. 2024-02-19T14:57:29.709 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 126, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:56:59.000185+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8259": { "addr": "172.21.15.78:6836/1973988605", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 1973988605, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 1973988605, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8259, "incarnation": 123, "join_fscid": 10, "name": "c", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 140, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-19T14:57:26.646012+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8259 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/3153272641", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3153272641, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 3153272641, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 119, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7522, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/1637684624", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1637684624, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1637684624, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 124, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7621, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/3197225278", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 3197225278, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 3197225278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 113, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:57:29.710 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 10, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:57:29.710 INFO:tasks.cephfs.test_failover:current = [{'gid': 8262, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/3197225278', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3197225278}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3197225278}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 113}, {'gid': 7522, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/3153272641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 3153272641}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 3153272641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 119}, {'gid': 8259, 'name': 'c', 'rank': 0, 'incarnation': 123, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.78:6836/1973988605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 1973988605}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 1973988605}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7621, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/1637684624', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1637684624}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1637684624}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 124}] 2024-02-19T14:57:29.711 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-19T14:57:29.711 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:57:29.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:57:29.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:57:29.814 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:57:29.814 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:57:29.814 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:57:29.830 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:57:29.830 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:57:29.830 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:57:29.848 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:57:29.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:29.930 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:57:29.928+0000 7fdf0eb37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:29.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.968+0000 7fc67a88f640 1 Processor -- start 2024-02-19T14:57:29.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.969+0000 7fc67a88f640 1 -- start start 2024-02-19T14:57:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc67a88f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 0x7fc6740c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc67a88f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67406ff90 con 0x7fc67414d210 2024-02-19T14:57:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc67a88f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc674156fc0 con 0x7fc6740c3d90 2024-02-19T14:57:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc67a88f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc674070270 con 0x7fc6740719e0 2024-02-19T14:57:29.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc673fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc67414d210 0x7fc6740c4290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41374/0 (socket says 172.21.15.78:41374) 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc673fff640 1 -- 172.21.15.78:0/2118774750 learned_addr learned my addr 172.21.15.78:0/2118774750 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149685231 0 0) 0x7fc67406ff90 con 0x7fc67414d210 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65c003520 con 0x7fc67414d210 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc6737fe640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 0x7fc6740c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306835968 0 0) 0x7fc65c003520 con 0x7fc67414d210 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 msgr2=0x7fc6740c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 0x7fc6740c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 >> v1:172.21.15.150:6789/0 conn(0x7fc6740c3d90 legacy=0x7fc67415d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6741572a0 con 0x7fc67414d210 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fc664001670 con 0x7fc67414d210 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 >> v1:172.21.15.78:6789/0 conn(0x7fc67414d210 legacy=0x7fc6740c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc65c0039a0 0x7fc65c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 0x7fc65c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc65c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc674070270 con 0x7fc65c004d00 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c00bb60 con 0x7fc65c0039a0 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.970+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c00bd20 con 0x7fc6740719e0 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc673fff640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc65c0039a0 0x7fc65c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc678e2c640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc65c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc6737fe640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 0x7fc65c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc678e2c640 1 -- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 msgr2=0x7fc65c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc678e2c640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740719e0 0x7fc65c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc678e2c640 1 -- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc65c0039a0 msgr2=0x7fc65c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc678e2c640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc65c0039a0 0x7fc65c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.971+0000 7fc678e2c640 1 -- 172.21.15.78:0/2118774750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c003520 con 0x7fc65c004d00 2024-02-19T14:57:29.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.972+0000 7fc678e2c640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc65c005180 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc6740c41e0 tx=0x7fc65c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe16e2b2a8def94 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.972+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c003b70 con 0x7fc65c004d00 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.972+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c003cd0 con 0x7fc65c004d00 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.972+0000 7fc672ffd640 1 -- 172.21.15.78:0/2118774750 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c005310 con 0x7fc65c004d00 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc67a88f640 1 -- 172.21.15.78:0/2118774750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 msgr2=0x7fc65c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc67a88f640 1 --2- 172.21.15.78:0/2118774750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc65c005180 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc6740c41e0 tx=0x7fc65c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc678e2c640 1 -- 172.21.15.78:0/2118774750 reap_dead start 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc67a88f640 1 -- 172.21.15.78:0/2118774750 shutdown_connections 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc67a88f640 1 -- 172.21.15.78:0/2118774750 >> 172.21.15.78:0/2118774750 conn(0x7fc6740c5050 msgr2=0x7fc6740c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc67a88f640 1 -- 172.21.15.78:0/2118774750 shutdown_connections 2024-02-19T14:57:29.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.973+0000 7fc67a88f640 1 -- 172.21.15.78:0/2118774750 wait complete. 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.974+0000 7fc67a88f640 1 Processor -- start 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.974+0000 7fc67a88f640 1 -- start start 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc67a88f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc6740c1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc67a88f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740c1ec0 0x7fc6741ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc67a88f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6741eeb40 0x7fc6741493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc67a88f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740c3970 con 0x7fc65c004d00 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc67a88f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6741eb090 con 0x7fc6741eeb40 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc67a88f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc674152b10 con 0x7fc6740c1ec0 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc6740c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc6740c1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50662/0 (socket says 172.21.15.78:50662) 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 -- 172.21.15.78:0/568333549 learned_addr learned my addr 172.21.15.78:0/568333549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc678e2c640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6741eeb40 0x7fc6741493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc6737fe640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740c1ec0 0x7fc6741ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 -- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740c1ec0 msgr2=0x7fc6741ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740c1ec0 0x7fc6741ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 -- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6741eeb40 msgr2=0x7fc6741493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6741eeb40 0x7fc6741493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 -- 172.21.15.78:0/568333549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6740c3660 con 0x7fc65c004d00 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc678e2c640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6741eeb40 0x7fc6741493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc6737fe640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740c1ec0 0x7fc6741ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.975+0000 7fc673fff640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc6740c1980 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc664007290 tx=0x7fc664007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.976+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664001500 con 0x7fc65c004d00 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.976+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66400f940 con 0x7fc65c004d00 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.976+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6741eb350 con 0x7fc65c004d00 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.976+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66400fcd0 con 0x7fc65c004d00 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.976+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6741eb910 con 0x7fc65c004d00 2024-02-19T14:57:29.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.977+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67414c760 con 0x7fc65c004d00 2024-02-19T14:57:29.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.977+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc66401d460 con 0x7fc65c004d00 2024-02-19T14:57:29.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.978+0000 7fc670ff9640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66006b840 0x7fc66006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:29.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.978+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc66408a4e0 con 0x7fc65c004d00 2024-02-19T14:57:29.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.978+0000 7fc6737fe640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66006b840 0x7fc66006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:29.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.978+0000 7fc6737fe640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66006b840 0x7fc66006dd00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc67414c1c0 tx=0x7fc668008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:29.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:29.982+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc664054d50 con 0x7fc65c004d00 2024-02-19T14:57:30.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.197+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6740c2c30 con 0x7fc65c004d00 2024-02-19T14:57:30.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.198+0000 7fc670ff9640 1 -- 172.21.15.78:0/568333549 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4957 (secure 0 0 0) 0x7fc664058410 con 0x7fc65c004d00 2024-02-19T14:57:30.201 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:30.201 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:26.646012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":11,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:30.201 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 126 2024-02-19T14:57:30.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66006b840 msgr2=0x7fc66006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66006b840 0x7fc66006dd00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc67414c1c0 tx=0x7fc668008040 comp rx=0 tx=0).stop 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 msgr2=0x7fc6740c1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc6740c1980 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fc664007290 tx=0x7fc664007750 comp rx=0 tx=0).stop 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 shutdown_connections 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc66006b840 0x7fc66006dd00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6741eeb40 0x7fc6741493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6740c1ec0 0x7fc6741ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 --2- 172.21.15.78:0/568333549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc65c004d00 0x7fc6740c1980 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 >> 172.21.15.78:0/568333549 conn(0x7fc6740c5050 msgr2=0x7fc674155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 shutdown_connections 2024-02-19T14:57:30.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.200+0000 7fc67a88f640 1 -- 172.21.15.78:0/568333549 wait complete. 2024-02-19T14:57:30.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:30.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.324+0000 7f8dc97ac640 1 Processor -- start 2024-02-19T14:57:30.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.324+0000 7f8dc97ac640 1 -- start start 2024-02-19T14:57:30.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc97ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc4152700 0x7f8dc40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc97ac640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8dc40c5730 0x7f8dc406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc97ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc41551f0 con 0x7f8dc40c8e00 2024-02-19T14:57:30.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc97ac640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc4155490 con 0x7f8dc40c5730 2024-02-19T14:57:30.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc97ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4155770 con 0x7f8dc4152700 2024-02-19T14:57:30.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dbbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8dc40c5730 0x7f8dc406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8dc40c8e00 0x7f8dc4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41380/0 (socket says 172.21.15.78:41380) 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dbbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8dc40c5730 0x7f8dc406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50670/0 (socket says 172.21.15.78:50670) 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc37fe640 1 -- 172.21.15.78:0/2377906238 learned_addr learned my addr 172.21.15.78:0/2377906238 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc2ffd640 1 --2- 172.21.15.78:0/2377906238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc4152700 0x7f8dc40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dbbfff640 1 -- 172.21.15.78:0/2377906238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc4152700 msgr2=0x7f8dc40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dbbfff640 1 --2- 172.21.15.78:0/2377906238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc4152700 0x7f8dc40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dbbfff640 1 -- 172.21.15.78:0/2377906238 >> v1:172.21.15.78:6789/0 conn(0x7f8dc40c8e00 legacy=0x7f8dc4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dbbfff640 1 -- 172.21.15.78:0/2377906238 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc4155a50 con 0x7f8dc40c5730 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.325+0000 7f8dc27fc640 1 -- 172.21.15.78:0/2377906238 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630960417 0 0) 0x7f8dc41551f0 con 0x7f8dc40c8e00 2024-02-19T14:57:30.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.326+0000 7f8dbbfff640 1 --2- 172.21.15.78:0/2377906238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c5730 0x7f8dc406c2a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0097a0 tx=0x7f8dac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732d238ee44032b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.326+0000 7f8dc27fc640 1 -- 172.21.15.78:0/2377906238 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac031830 con 0x7f8dc40c5730 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.326+0000 7f8dc27fc640 1 -- 172.21.15.78:0/2377906238 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac031990 con 0x7f8dc40c5730 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.326+0000 7f8dc27fc640 1 -- 172.21.15.78:0/2377906238 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac031d20 con 0x7f8dc40c5730 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 -- 172.21.15.78:0/2377906238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c5730 msgr2=0x7f8dc406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/2377906238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c5730 0x7f8dc406c2a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0097a0 tx=0x7f8dac02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 -- 172.21.15.78:0/2377906238 shutdown_connections 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/2377906238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c5730 0x7f8dc406c2a0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/2377906238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc4152700 0x7f8dc40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 -- 172.21.15.78:0/2377906238 >> 172.21.15.78:0/2377906238 conn(0x7f8dc40cac90 msgr2=0x7f8dc40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 -- 172.21.15.78:0/2377906238 shutdown_connections 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.327+0000 7f8dc97ac640 1 -- 172.21.15.78:0/2377906238 wait complete. 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 Processor -- start 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 -- start start 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc40c5730 0x7f8dc4158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 0x7f8dc40c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc4152700 0x7f8dc40aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40acde0 con 0x7f8dc40c8e00 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4148c40 con 0x7f8dc4152700 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dc97ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc40ad230 con 0x7f8dc40c5730 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.328+0000 7f8dbbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 0x7f8dc40c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 0x7f8dc40c39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50686/0 (socket says 172.21.15.78:50686) 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 -- 172.21.15.78:0/3869574022 learned_addr learned my addr 172.21.15.78:0/3869574022 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dc37fe640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc4152700 0x7f8dc40aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dc2ffd640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc40c5730 0x7f8dc4158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 -- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc40c5730 msgr2=0x7f8dc4158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc40c5730 0x7f8dc4158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 -- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc4152700 msgr2=0x7f8dc40aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc4152700 0x7f8dc40aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 -- 172.21.15.78:0/3869574022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac009400 con 0x7f8dc40c8e00 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbbfff640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 0x7f8dc40c39d0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009890 tx=0x7f8dac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac03b980 con 0x7f8dc40c8e00 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac03bb00 con 0x7f8dc40c8e00 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc4148f00 con 0x7f8dc40c8e00 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac045400 con 0x7f8dc40c8e00 2024-02-19T14:57:30.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.329+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc41ebb40 con 0x7f8dc40c8e00 2024-02-19T14:57:30.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.330+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc40acf50 con 0x7f8dc40c8e00 2024-02-19T14:57:30.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.331+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8dac04e430 con 0x7f8dc40c8e00 2024-02-19T14:57:30.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.331+0000 7f8dbb7fe640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8da406b810 0x7f8da406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.331+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8dac0ac5c0 con 0x7f8dc40c8e00 2024-02-19T14:57:30.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.331+0000 7f8dc2ffd640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8da406b810 0x7f8da406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.331+0000 7f8dc2ffd640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8da406b810 0x7f8da406dcd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4151e70 tx=0x7f8db4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.335+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8dc40acf50 con 0x7f8dc40c8e00 2024-02-19T14:57:30.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.514+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dc40acf20 con 0x7f8dc40c8e00 2024-02-19T14:57:30.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.515+0000 7f8dbb7fe640 1 -- 172.21.15.78:0/3869574022 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17143 (secure 0 0 0) 0x7f8dac076e30 con 0x7f8dc40c8e00 2024-02-19T14:57:30.518 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:30.518 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":140,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:24.626444+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:57.223056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:57.972812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6833/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6834/2425259283":"2024-02-20T14:57:10.590067+0000","172.21.15.150:6832/2425259283":"2024-02-20T14:57:10.590067+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:30.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8da406b810 msgr2=0x7f8da406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8da406b810 0x7f8da406dcd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4151e70 tx=0x7f8db4008040 comp rx=0 tx=0).stop 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 msgr2=0x7f8dc40c39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 0x7f8dc40c39d0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009890 tx=0x7f8dac0046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 shutdown_connections 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8da406b810 0x7f8da406dcd0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8dc40c8e00 0x7f8dc40c39d0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc4152700 0x7f8dc40aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 --2- 172.21.15.78:0/3869574022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc40c5730 0x7f8dc4158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 >> 172.21.15.78:0/3869574022 conn(0x7f8dc40cac90 msgr2=0x7f8dc414d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 shutdown_connections 2024-02-19T14:57:30.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.517+0000 7f8dc97ac640 1 -- 172.21.15.78:0/3869574022 wait complete. 2024-02-19T14:57:30.532 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:56:57.223056+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:56:57.972812+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:57:30.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:57:30.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.633+0000 7f426ed2c640 1 Processor -- start 2024-02-19T14:57:30.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.633+0000 7f426ed2c640 1 -- start start 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426ed2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4268150560 0x7f42680c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426ed2c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f42680721a0 0x7f42680c6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426ed2c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4268156e80 con 0x7f426806bb00 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426ed2c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426806ffc0 con 0x7f4268150560 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426ed2c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42680c6cd0 con 0x7f42680721a0 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426cac8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4268150560 0x7f42680c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426cac8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4268150560 0x7f42680c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50694/0 (socket says 172.21.15.78:50694) 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.634+0000 7f426cac8640 1 -- 172.21.15.78:0/3361104662 learned_addr learned my addr 172.21.15.78:0/3361104662 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.635+0000 7f426d2c9640 1 --2- 172.21.15.78:0/3361104662 >> v2:172.21.15.150:3300/0 conn(0x7f42680721a0 0x7f42680c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.635+0000 7f426cac8640 1 -- 172.21.15.78:0/3361104662 >> v2:172.21.15.150:3300/0 conn(0x7f42680721a0 msgr2=0x7f42680c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.635+0000 7f426cac8640 1 --2- 172.21.15.78:0/3361104662 >> v2:172.21.15.150:3300/0 conn(0x7f42680721a0 0x7f42680c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.635+0000 7f426cac8640 1 -- 172.21.15.78:0/3361104662 >> v1:172.21.15.150:6789/0 conn(0x7f426806bb00 legacy=0x7f42680c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:30.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.635+0000 7f426cac8640 1 -- 172.21.15.78:0/3361104662 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268072d70 con 0x7f4268150560 2024-02-19T14:57:30.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.635+0000 7f426d2c9640 1 --2- 172.21.15.78:0/3361104662 >> v2:172.21.15.150:3300/0 conn(0x7f42680721a0 0x7f42680c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.636+0000 7f426cac8640 1 --2- 172.21.15.78:0/3361104662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4268150560 0x7f42680c77d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f426806b810 tx=0x7f425c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82f9bd01a83932 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.636+0000 7f42677fe640 1 -- 172.21.15.78:0/3361104662 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0317e0 con 0x7f4268150560 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.636+0000 7f42677fe640 1 -- 172.21.15.78:0/3361104662 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c031940 con 0x7f4268150560 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.636+0000 7f42677fe640 1 -- 172.21.15.78:0/3361104662 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c031cd0 con 0x7f4268150560 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 -- 172.21.15.78:0/3361104662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4268150560 msgr2=0x7f42680c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3361104662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4268150560 0x7f42680c77d0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f426806b810 tx=0x7f425c02b880 comp rx=0 tx=0).stop 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 -- 172.21.15.78:0/3361104662 shutdown_connections 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3361104662 >> v2:172.21.15.150:3300/0 conn(0x7f42680721a0 0x7f42680c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3361104662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4268150560 0x7f42680c77d0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 -- 172.21.15.78:0/3361104662 >> 172.21.15.78:0/3361104662 conn(0x7f426806c170 msgr2=0x7f426806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:30.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 -- 172.21.15.78:0/3361104662 shutdown_connections 2024-02-19T14:57:30.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.637+0000 7f426ed2c640 1 -- 172.21.15.78:0/3361104662 wait complete. 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.638+0000 7f426ed2c640 1 Processor -- start 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.638+0000 7f426ed2c640 1 -- start start 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.638+0000 7f426ed2c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426806bb00 0x7f42680c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 0x7f42680c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4268150560 0x7f42680a8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680c3090 con 0x7f42680721a0 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680c24e0 con 0x7f426806bb00 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42681e9450 con 0x7f4268150560 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 0x7f42680c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 0x7f42680c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50700/0 (socket says 172.21.15.78:50700) 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 -- 172.21.15.78:0/3806416910 learned_addr learned my addr 172.21.15.78:0/3806416910 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:30.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426cac8640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426806bb00 0x7f42680c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426d2c9640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4268150560 0x7f42680a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 -- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4268150560 msgr2=0x7f42680a8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4268150560 0x7f42680a8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 -- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426806bb00 msgr2=0x7f42680c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426806bb00 0x7f42680c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 -- 172.21.15.78:0/3806416910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c0094d0 con 0x7f42680721a0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f4267fff640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 0x7f42680c49b0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f425400c3f0 tx=0x7f425400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254016070 con 0x7f42680721a0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4254007020 con 0x7f42680721a0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42681e9980 con 0x7f42680721a0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42540073b0 con 0x7f42680721a0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.639+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42680c2050 con 0x7f42680721a0 2024-02-19T14:57:30.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.640+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42680c21e0 con 0x7f42680721a0 2024-02-19T14:57:30.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.641+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4254007510 con 0x7f42680721a0 2024-02-19T14:57:30.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.641+0000 7f42657fa640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f424c06b810 0x7f424c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.642+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f4254089ab0 con 0x7f42680721a0 2024-02-19T14:57:30.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.642+0000 7f426cac8640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f424c06b810 0x7f424c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.642+0000 7f426cac8640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f424c06b810 0x7f424c06dcd0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f425c036000 tx=0x7f425c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.646+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f42540543d0 con 0x7f42680721a0 2024-02-19T14:57:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.824+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4268156150 con 0x7f42680721a0 2024-02-19T14:57:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.824+0000 7f42657fa640 1 -- 172.21.15.78:0/3806416910 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4254057a90 con 0x7f42680721a0 2024-02-19T14:57:30.827 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:57:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f424c06b810 msgr2=0x7f424c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f424c06b810 0x7f424c06dcd0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f425c036000 tx=0x7f425c03d040 comp rx=0 tx=0).stop 2024-02-19T14:57:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 msgr2=0x7f42680c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 0x7f42680c49b0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f425400c3f0 tx=0x7f425400c8b0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 shutdown_connections 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f424c06b810 0x7f424c06dcd0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42680721a0 0x7f42680c49b0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4268150560 0x7f42680a8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 --2- 172.21.15.78:0/3806416910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426806bb00 0x7f42680c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 >> 172.21.15.78:0/3806416910 conn(0x7f426806c170 msgr2=0x7f426814f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 shutdown_connections 2024-02-19T14:57:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.826+0000 7f426ed2c640 1 -- 172.21.15.78:0/3806416910 wait complete. 2024-02-19T14:57:30.840 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:57:30.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.941+0000 7fb24bbf4640 1 Processor -- start 2024-02-19T14:57:30.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.941+0000 7fb24bbf4640 1 -- start start 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24bbf4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2440c5560 0x7fb2440c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24bbf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb244070720 0x7fb244070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24bbf4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2440c65a0 con 0x7fb2440c7910 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24bbf4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2440c7530 con 0x7fb2440c5560 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24bbf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441537b0 con 0x7fb244070720 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2440c5560 0x7fb2440c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2440c5560 0x7fb2440c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50712/0 (socket says 172.21.15.78:50712) 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 -- 172.21.15.78:0/3206690449 learned_addr learned my addr 172.21.15.78:0/3206690449 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24a191640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb244070720 0x7fb244070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 -- 172.21.15.78:0/3206690449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb244070720 msgr2=0x7fb244070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb244070720 0x7fb244070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 -- 172.21.15.78:0/3206690449 >> v1:172.21.15.150:6789/0 conn(0x7fb2440c7910 legacy=0x7fb244154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb249990640 1 -- 172.21.15.78:0/3206690449 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244153a90 con 0x7fb2440c5560 2024-02-19T14:57:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.942+0000 7fb24a191640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb244070720 0x7fb244070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.943+0000 7fb249990640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2440c5560 0x7fb2440c6ff0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb240009870 tx=0x7fb24002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9b2571fd060040 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.943+0000 7fb24898e640 1 -- 172.21.15.78:0/3206690449 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240031740 con 0x7fb2440c5560 2024-02-19T14:57:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.943+0000 7fb24898e640 1 -- 172.21.15.78:0/3206690449 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2400318a0 con 0x7fb2440c5560 2024-02-19T14:57:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.943+0000 7fb24898e640 1 -- 172.21.15.78:0/3206690449 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240031c30 con 0x7fb2440c5560 2024-02-19T14:57:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 -- 172.21.15.78:0/3206690449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2440c5560 msgr2=0x7fb2440c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2440c5560 0x7fb2440c6ff0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb240009870 tx=0x7fb24002b7e0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 -- 172.21.15.78:0/3206690449 shutdown_connections 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb244070720 0x7fb244070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/3206690449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2440c5560 0x7fb2440c6ff0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 -- 172.21.15.78:0/3206690449 >> 172.21.15.78:0/3206690449 conn(0x7fb24406b4b0 msgr2=0x7fb2440c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 -- 172.21.15.78:0/3206690449 shutdown_connections 2024-02-19T14:57:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.944+0000 7fb24bbf4640 1 -- 172.21.15.78:0/3206690449 wait complete. 2024-02-19T14:57:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.945+0000 7fb24bbf4640 1 Processor -- start 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.945+0000 7fb24bbf4640 1 -- start start 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24bbf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 0x7fb2440a8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24bbf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2440c5560 0x7fb2441ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24bbf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2440c7910 0x7fb2441eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24bbf4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24406c3a0 con 0x7fb244070720 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24bbf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441ee510 con 0x7fb2440c7910 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24bbf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441edf00 con 0x7fb2440c5560 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 0x7fb2440a8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 0x7fb2440a8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50718/0 (socket says 172.21.15.78:50718) 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 -- 172.21.15.78:0/4024297346 learned_addr learned my addr 172.21.15.78:0/4024297346 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24a191640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2440c7910 0x7fb2441eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb24918f640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2440c5560 0x7fb2441ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 -- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2440c5560 msgr2=0x7fb2441ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2440c5560 0x7fb2441ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 -- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2440c7910 msgr2=0x7fb2441eacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2440c7910 0x7fb2441eacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.946+0000 7fb249990640 1 -- 172.21.15.78:0/4024297346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2400094d0 con 0x7fb244070720 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.947+0000 7fb249990640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 0x7fb2440a8250 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb240009870 tx=0x7fb2400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.947+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240033c70 con 0x7fb244070720 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.947+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb240033dd0 con 0x7fb244070720 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.947+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2440c5ac0 con 0x7fb244070720 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.947+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24003c640 con 0x7fb244070720 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.947+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24406c0c0 con 0x7fb244070720 2024-02-19T14:57:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.948+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2441edd60 con 0x7fb244070720 2024-02-19T14:57:30.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.949+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb24003a040 con 0x7fb244070720 2024-02-19T14:57:30.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.949+0000 7fb23a7fc640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22806b810 0x7fb22806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:30.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.949+0000 7fb24918f640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22806b810 0x7fb22806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:30.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.949+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fb2400abe60 con 0x7fb244070720 2024-02-19T14:57:30.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.950+0000 7fb24918f640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22806b810 0x7fb22806dcd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb2440c5020 tx=0x7fb234009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:30.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:30.954+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2441edd60 con 0x7fb244070720 2024-02-19T14:57:31.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.130+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2441edd60 con 0x7fb244070720 2024-02-19T14:57:31.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.130+0000 7fb23a7fc640 1 -- 172.21.15.78:0/4024297346 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb240038020 con 0x7fb244070720 2024-02-19T14:57:31.133 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22806b810 msgr2=0x7fb22806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22806b810 0x7fb22806dcd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb2440c5020 tx=0x7fb234009040 comp rx=0 tx=0).stop 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 msgr2=0x7fb2440a8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 0x7fb2440a8250 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb240009870 tx=0x7fb2400046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 shutdown_connections 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb22806b810 0x7fb22806dcd0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2440c7910 0x7fb2441eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2440c5560 0x7fb2441ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 --2- 172.21.15.78:0/4024297346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb244070720 0x7fb2440a8250 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 >> 172.21.15.78:0/4024297346 conn(0x7fb24406b4b0 msgr2=0x7fb244150900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:31.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 shutdown_connections 2024-02-19T14:57:31.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.132+0000 7fb24bbf4640 1 -- 172.21.15.78:0/4024297346 wait complete. 2024-02-19T14:57:31.145 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:57:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.245+0000 7f350f89a640 1 Processor -- start 2024-02-19T14:57:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350f89a640 1 -- start start 2024-02-19T14:57:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350f89a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3508152700 0x7f35080c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350f89a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35080c8e00 0x7f350815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350f89a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35081551f0 con 0x7f35080c5730 2024-02-19T14:57:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350f89a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3508155490 con 0x7f35080c8e00 2024-02-19T14:57:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350f89a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3508155770 con 0x7f3508152700 2024-02-19T14:57:31.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350de37640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35080c8e00 0x7f350815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350de37640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35080c8e00 0x7f350815f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50730/0 (socket says 172.21.15.78:50730) 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350de37640 1 -- 172.21.15.78:0/4169727488 learned_addr learned my addr 172.21.15.78:0/4169727488 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.246+0000 7f350ce35640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f35080c5730 0x7f350806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41394/0 (socket says 172.21.15.78:41394) 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.247+0000 7f350d636640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3508152700 0x7f35080c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.247+0000 7f350de37640 1 -- 172.21.15.78:0/4169727488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3508152700 msgr2=0x7f35080c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.247+0000 7f350de37640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3508152700 0x7f35080c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.247+0000 7f350de37640 1 -- 172.21.15.78:0/4169727488 >> v1:172.21.15.78:6789/0 conn(0x7f35080c5730 legacy=0x7f350806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.247+0000 7f350de37640 1 -- 172.21.15.78:0/4169727488 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508155a50 con 0x7f35080c8e00 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.247+0000 7f350d636640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3508152700 0x7f35080c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350de37640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35080c8e00 0x7f350815f980 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f3500009e50 tx=0x7f350002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cba4add028bc5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f34fffff640 1 -- 172.21.15.78:0/4169727488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350002c8d0 con 0x7f35080c8e00 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f34fffff640 1 -- 172.21.15.78:0/4169727488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350002ca30 con 0x7f35080c8e00 2024-02-19T14:57:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f34fffff640 1 -- 172.21.15.78:0/4169727488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350002cdc0 con 0x7f35080c8e00 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 -- 172.21.15.78:0/4169727488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35080c8e00 msgr2=0x7f350815f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35080c8e00 0x7f350815f980 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f3500009e50 tx=0x7f350002bdb0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 -- 172.21.15.78:0/4169727488 shutdown_connections 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35080c8e00 0x7f350815f980 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 --2- 172.21.15.78:0/4169727488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3508152700 0x7f35080c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 -- 172.21.15.78:0/4169727488 >> 172.21.15.78:0/4169727488 conn(0x7f35080cac90 msgr2=0x7f35080cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 -- 172.21.15.78:0/4169727488 shutdown_connections 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.248+0000 7f350f89a640 1 -- 172.21.15.78:0/4169727488 wait complete. 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.249+0000 7f350f89a640 1 Processor -- start 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.249+0000 7f350f89a640 1 -- start start 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350f89a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35080c5730 0x7f35080c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350f89a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35080c8e00 0x7f35081e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350f89a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 0x7f35081e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350f89a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35080ac920 con 0x7f3508152700 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350f89a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35080c1940 con 0x7f35080c5730 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350f89a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35080c21e0 con 0x7f35080c8e00 2024-02-19T14:57:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 0x7f35081e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 0x7f35081e7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50744/0 (socket says 172.21.15.78:50744) 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 -- 172.21.15.78:0/3702407848 learned_addr learned my addr 172.21.15.78:0/3702407848 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350d636640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35080c5730 0x7f35080c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350ce35640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35080c8e00 0x7f35081e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 -- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35080c8e00 msgr2=0x7f35081e7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35080c8e00 0x7f35081e7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 -- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35080c5730 msgr2=0x7f35080c3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35080c5730 0x7f35080c3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350de37640 1 -- 172.21.15.78:0/3702407848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3500009b30 con 0x7f3508152700 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.250+0000 7f350d636640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35080c5730 0x7f35080c3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.251+0000 7f350de37640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 0x7f35081e7f60 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f3500009e50 tx=0x7f3500004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.251+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500009510 con 0x7f3508152700 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.251+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35000079b0 con 0x7f3508152700 2024-02-19T14:57:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.251+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35081eb4e0 con 0x7f3508152700 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.251+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500007d60 con 0x7f3508152700 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.251+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35080c1e70 con 0x7f3508152700 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.252+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35080c2350 con 0x7f3508152700 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.252+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f350003e450 con 0x7f3508152700 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.253+0000 7f34fdffb640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34ec06b810 0x7f34ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.253+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f35000acca0 con 0x7f3508152700 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.253+0000 7f350d636640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34ec06b810 0x7f34ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.253+0000 7f350d636640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34ec06b810 0x7f34ec06dcd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f34f8006fd0 tx=0x7f34f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.257+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3500077510 con 0x7f3508152700 2024-02-19T14:57:31.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.434+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3508074990 con 0x7f3508152700 2024-02-19T14:57:31.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.434+0000 7f34fdffb640 1 -- 172.21.15.78:0/3702407848 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+11 (secure 0 0 0) 0x7f350007abd0 con 0x7f3508152700 2024-02-19T14:57:31.437 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34ec06b810 msgr2=0x7f34ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34ec06b810 0x7f34ec06dcd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f34f8006fd0 tx=0x7f34f8008040 comp rx=0 tx=0).stop 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 msgr2=0x7f35081e7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 0x7f35081e7f60 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f3500009e50 tx=0x7f3500004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 shutdown_connections 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f34ec06b810 0x7f34ec06dcd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3508152700 0x7f35081e7f60 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35080c8e00 0x7f35081e7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 --2- 172.21.15.78:0/3702407848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35080c5730 0x7f35080c3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 >> 172.21.15.78:0/3702407848 conn(0x7f35080cac90 msgr2=0x7f350806c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 shutdown_connections 2024-02-19T14:57:31.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.436+0000 7f350f89a640 1 -- 172.21.15.78:0/3702407848 wait complete. 2024-02-19T14:57:31.449 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:31.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.547+0000 7f1014023640 1 Processor -- start 2024-02-19T14:57:31.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.547+0000 7f1014023640 1 -- start start 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.547+0000 7f1014023640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f100c06c9b0 0x7f100c160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1014023640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c1551f0 con 0x7f100c0c8e00 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1014023640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c155490 con 0x7f100c152700 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1014023640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c155770 con 0x7f100c06c9b0 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f10115be640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f100c0c8e00 0x7f100c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41398/0 (socket says 172.21.15.78:41398) 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f10115be640 1 -- 172.21.15.78:0/1424544783 learned_addr learned my addr 172.21.15.78:0/1424544783 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f10125c0640 1 --2- 172.21.15.78:0/1424544783 >> v2:172.21.15.150:3300/0 conn(0x7f100c06c9b0 0x7f100c160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28573117 0 0) 0x7f100c1551f0 con 0x7f100c0c8e00 2024-02-19T14:57:31.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff8003520 con 0x7f100c0c8e00 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3010412983 0 0) 0x7f0ff8003520 con 0x7f100c0c8e00 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 >> v2:172.21.15.150:3300/0 conn(0x7f100c06c9b0 msgr2=0x7f100c160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 --2- 172.21.15.78:0/1424544783 >> v2:172.21.15.150:3300/0 conn(0x7f100c06c9b0 0x7f100c160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 >> v1:172.21.15.150:6789/0 conn(0x7f100c152700 legacy=0x7f100c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c155a50 con 0x7f100c0c8e00 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0ffc001670 con 0x7f100c0c8e00 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 >> v1:172.21.15.78:6789/0 conn(0x7f100c0c8e00 legacy=0x7f100c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ff8003970 0x7f0ff8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ff80047c0 0x7f0ff8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8005180 0x7f0ff800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c155770 con 0x7f0ff8003970 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff800c020 con 0x7f0ff8005180 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff800c1e0 con 0x7f0ff80047c0 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f1011dbf640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ff8003970 0x7f0ff8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f10125c0640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8005180 0x7f0ff800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.548+0000 7f10115be640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ff80047c0 0x7f0ff8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1011dbf640 1 -- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ff80047c0 msgr2=0x7f0ff8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1011dbf640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ff80047c0 0x7f0ff8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1011dbf640 1 -- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8005180 msgr2=0x7f0ff800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1011dbf640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8005180 0x7f0ff800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f10125c0640 1 -- 172.21.15.78:0/1424544783 reap_dead start 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1011dbf640 1 -- 172.21.15.78:0/1424544783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8003520 con 0x7f0ff8003970 2024-02-19T14:57:31.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f10115be640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ff80047c0 0x7f0ff8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1011dbf640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ff8003970 0x7f0ff8003d90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f1008001750 tx=0x7f0ff80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4782682e673954c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080052d0 con 0x7f0ff8003970 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100802e050 con 0x7f0ff8003970 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.549+0000 7f1010dbd640 1 -- 172.21.15.78:0/1424544783 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008003830 con 0x7f0ff8003970 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 -- 172.21.15.78:0/1424544783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ff8003970 msgr2=0x7f0ff8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ff8003970 0x7f0ff8003d90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f1008001750 tx=0x7f0ff80057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 -- 172.21.15.78:0/1424544783 shutdown_connections 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 --2- 172.21.15.78:0/1424544783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ff8003970 0x7f0ff8003d90 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 -- 172.21.15.78:0/1424544783 >> 172.21.15.78:0/1424544783 conn(0x7f100c0cac90 msgr2=0x7f100c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 -- 172.21.15.78:0/1424544783 shutdown_connections 2024-02-19T14:57:31.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.550+0000 7f1014023640 1 -- 172.21.15.78:0/1424544783 wait complete. 2024-02-19T14:57:31.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.551+0000 7f1014023640 1 Processor -- start 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.551+0000 7f1014023640 1 -- start start 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1014023640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8003970 0x7f100c1ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1014023640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f100c0b0160 0x7f100c158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1014023640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 0x7f100c157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1014023640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1564f0 con 0x7f100c0adee0 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1014023640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c1567d0 con 0x7f0ff8003970 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1014023640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c156ab0 con 0x7f100c0b0160 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 0x7f100c157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 0x7f100c157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50768/0 (socket says 172.21.15.78:50768) 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 -- 172.21.15.78:0/2590588478 learned_addr learned my addr 172.21.15.78:0/2590588478 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10115be640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f100c0b0160 0x7f100c158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1011dbf640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8003970 0x7f100c1ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 -- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f100c0b0160 msgr2=0x7f100c158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f100c0b0160 0x7f100c158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 -- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8003970 msgr2=0x7f100c1ea900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8003970 0x7f100c1ea900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 -- 172.21.15.78:0/2590588478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c156d90 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f1011dbf640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8003970 0x7f100c1ea900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10115be640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f100c0b0160 0x7f100c158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.552+0000 7f10125c0640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 0x7f100c157590 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f100c155120 tx=0x7f100400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.553+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100400e040 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.553+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100400f040 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.553+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c157070 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.553+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004015430 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.553+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c159160 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.554+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c1572b0 con 0x7f100c0adee0 2024-02-19T14:57:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.554+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1004015590 con 0x7f100c0adee0 2024-02-19T14:57:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.554+0000 7f10027fc640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ff006b810 0x7f0ff006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.554+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f100408aee0 con 0x7f100c0adee0 2024-02-19T14:57:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.555+0000 7f1011dbf640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ff006b810 0x7f0ff006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.555+0000 7f1011dbf640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ff006b810 0x7f0ff006dcd0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1008033000 tx=0x7f1008009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.559+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f10040556c0 con 0x7f100c0adee0 2024-02-19T14:57:31.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.778+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f100c074990 con 0x7f100c0adee0 2024-02-19T14:57:31.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.779+0000 7f10027fc640 1 -- 172.21.15.78:0/2590588478 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4957 (secure 0 0 0) 0x7f1004058d80 con 0x7f100c0adee0 2024-02-19T14:57:31.782 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:31.782 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:26.646012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":11,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:31.782 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 126 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ff006b810 msgr2=0x7f0ff006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ff006b810 0x7f0ff006dcd0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1008033000 tx=0x7f1008009030 comp rx=0 tx=0).stop 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 msgr2=0x7f100c157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 0x7f100c157590 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f100c155120 tx=0x7f100400f1b0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 shutdown_connections 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ff006b810 0x7f0ff006dcd0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f100c0adee0 0x7f100c157590 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f100c0b0160 0x7f100c158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 --2- 172.21.15.78:0/2590588478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ff8003970 0x7f100c1ea900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 >> 172.21.15.78:0/2590588478 conn(0x7f100c0cac90 msgr2=0x7f100c0c7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:31.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 shutdown_connections 2024-02-19T14:57:31.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.781+0000 7f1014023640 1 -- 172.21.15.78:0/2590588478 wait complete. 2024-02-19T14:57:31.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:31.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.911+0000 7fca20c4e640 1 Processor -- start 2024-02-19T14:57:31.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.911+0000 7fca20c4e640 1 -- start start 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca20c4e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca1c0c78a0 0x7fca1c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca20c4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c14ef50 0x7fca1c14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca20c4e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fca1c0727d0 0x7fca1c072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca20c4e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c14f8f0 con 0x7fca1c0727d0 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca20c4e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c0c35c0 con 0x7fca1c0c78a0 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca20c4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0c38a0 con 0x7fca1c14ef50 2024-02-19T14:57:31.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fca1c0727d0 0x7fca1c072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fca1c0727d0 0x7fca1c072bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50770/0 (socket says 172.21.15.78:50770) 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 -- 172.21.15.78:0/27149649 learned_addr learned my addr 172.21.15.78:0/27149649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1a59c640 1 --2- 172.21.15.78:0/27149649 >> v2:172.21.15.150:3300/0 conn(0x7fca1c0c78a0 0x7fca1c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 -- 172.21.15.78:0/27149649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c14ef50 msgr2=0x7fca1c14f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 --2- 172.21.15.78:0/27149649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c14ef50 0x7fca1c14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 -- 172.21.15.78:0/27149649 >> v2:172.21.15.150:3300/0 conn(0x7fca1c0c78a0 msgr2=0x7fca1c0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.912+0000 7fca1ad9d640 1 --2- 172.21.15.78:0/27149649 >> v2:172.21.15.150:3300/0 conn(0x7fca1c0c78a0 0x7fca1c0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.913+0000 7fca1ad9d640 1 -- 172.21.15.78:0/27149649 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c0c3b80 con 0x7fca1c0727d0 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.913+0000 7fca1ad9d640 1 --2- 172.21.15.78:0/27149649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c0727d0 0x7fca1c072bf0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0c80d0 tx=0x7fca1402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72ee7fbd4488078 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.913+0000 7fca1959a640 1 -- 172.21.15.78:0/27149649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1402ea80 con 0x7fca1c0727d0 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.913+0000 7fca1959a640 1 -- 172.21.15.78:0/27149649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1402ebe0 con 0x7fca1c0727d0 2024-02-19T14:57:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.913+0000 7fca1959a640 1 -- 172.21.15.78:0/27149649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14009600 con 0x7fca1c0727d0 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 -- 172.21.15.78:0/27149649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c0727d0 msgr2=0x7fca1c072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 --2- 172.21.15.78:0/27149649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c0727d0 0x7fca1c072bf0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0c80d0 tx=0x7fca1402e0e0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 -- 172.21.15.78:0/27149649 shutdown_connections 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 --2- 172.21.15.78:0/27149649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c0727d0 0x7fca1c072bf0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 --2- 172.21.15.78:0/27149649 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c14ef50 0x7fca1c14f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 --2- 172.21.15.78:0/27149649 >> v2:172.21.15.150:3300/0 conn(0x7fca1c0c78a0 0x7fca1c0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 -- 172.21.15.78:0/27149649 >> 172.21.15.78:0/27149649 conn(0x7fca1c066e70 msgr2=0x7fca1c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 -- 172.21.15.78:0/27149649 shutdown_connections 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.914+0000 7fca20c4e640 1 -- 172.21.15.78:0/27149649 wait complete. 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.915+0000 7fca20c4e640 1 Processor -- start 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.915+0000 7fca20c4e640 1 -- start start 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca20c4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1c0727d0 0x7fca1c0a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca20c4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c0c78a0 0x7fca1c147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca20c4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 0x7fca1c147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca20c4e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1499d0 con 0x7fca1c14ef50 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca20c4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1ea610 con 0x7fca1c0727d0 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca20c4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0bcdb0 con 0x7fca1c0c78a0 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 0x7fca1c147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 0x7fca1c147c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50776/0 (socket says 172.21.15.78:50776) 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 -- 172.21.15.78:0/3798719095 learned_addr learned my addr 172.21.15.78:0/3798719095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1a59c640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1c0727d0 0x7fca1c0a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca19d9b640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c0c78a0 0x7fca1c147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 -- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c0c78a0 msgr2=0x7fca1c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c0c78a0 0x7fca1c147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 -- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1c0727d0 msgr2=0x7fca1c0a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1c0727d0 0x7fca1c0a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.916+0000 7fca1ad9d640 1 -- 172.21.15.78:0/3798719095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1400bd40 con 0x7fca1c14ef50 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.917+0000 7fca1ad9d640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 0x7fca1c147c90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fca14038040 tx=0x7fca14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.917+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1402ea80 con 0x7fca1c14ef50 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.917+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1400b720 con 0x7fca1c14ef50 2024-02-19T14:57:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.917+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c0bc7f0 con 0x7fca1c14ef50 2024-02-19T14:57:31.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.917+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14040880 con 0x7fca1c14ef50 2024-02-19T14:57:31.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.917+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c0aa110 con 0x7fca1c14ef50 2024-02-19T14:57:31.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.918+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c1497a0 con 0x7fca1c14ef50 2024-02-19T14:57:31.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.918+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fca1402ed50 con 0x7fca1c14ef50 2024-02-19T14:57:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.919+0000 7fca0affd640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9fc06b810 0x7fc9fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.919+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fca140af110 con 0x7fca1c14ef50 2024-02-19T14:57:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.919+0000 7fca1a59c640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9fc06b810 0x7fc9fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.919+0000 7fca1a59c640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9fc06b810 0x7fc9fc06dcd0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fca1000aac0 tx=0x7fca10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:31.923+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fca1c1497a0 con 0x7fca1c14ef50 2024-02-19T14:57:32.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.134+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca1c1497a0 con 0x7fca1c14ef50 2024-02-19T14:57:32.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.135+0000 7fca0affd640 1 -- 172.21.15.78:0/3798719095 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17143 (secure 0 0 0) 0x7fca1c1497a0 con 0x7fca1c14ef50 2024-02-19T14:57:32.138 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:32.139 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":140,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:24.626444+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:57.223056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:57.972812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6833/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6834/2425259283":"2024-02-20T14:57:10.590067+0000","172.21.15.150:6832/2425259283":"2024-02-20T14:57:10.590067+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.137+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9fc06b810 msgr2=0x7fc9fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.137+0000 7fca20c4e640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9fc06b810 0x7fc9fc06dcd0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fca1000aac0 tx=0x7fca10008040 comp rx=0 tx=0).stop 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.137+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 msgr2=0x7fca1c147c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.137+0000 7fca20c4e640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 0x7fca1c147c90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fca14038040 tx=0x7fca14004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 shutdown_connections 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9fc06b810 0x7fc9fc06dcd0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca1c14ef50 0x7fca1c147c90 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca1c0c78a0 0x7fca1c147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 --2- 172.21.15.78:0/3798719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1c0727d0 0x7fca1c0a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 >> 172.21.15.78:0/3798719095 conn(0x7fca1c066e70 msgr2=0x7fca1c151630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 shutdown_connections 2024-02-19T14:57:32.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.138+0000 7fca20c4e640 1 -- 172.21.15.78:0/3798719095 wait complete. 2024-02-19T14:57:32.151 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:57:32.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:32.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.254+0000 7fa511807640 1 Processor -- start 2024-02-19T14:57:32.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.254+0000 7fa511807640 1 -- start start 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa511807640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa50c152700 0x7fa50c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa511807640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c1551f0 con 0x7fa50c0c5730 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa511807640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50c155490 con 0x7fa50c0c8e00 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa511807640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50c155770 con 0x7fa50c152700 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa503fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa50c0c5730 0x7fa50c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41410/0 (socket says 172.21.15.78:41410) 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa503fff640 1 -- 172.21.15.78:0/2053937169 learned_addr learned my addr 172.21.15.78:0/2053937169 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50affd640 1 --2- 172.21.15.78:0/2053937169 >> v2:172.21.15.150:3300/0 conn(0x7fa50c152700 0x7fa50c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569593890 0 0) 0x7fa50c1551f0 con 0x7fa50c0c5730 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f8003520 con 0x7fa50c0c5730 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955441463 0 0) 0x7fa4f8003520 con 0x7fa50c0c5730 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 >> v2:172.21.15.150:3300/0 conn(0x7fa50c152700 msgr2=0x7fa50c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 --2- 172.21.15.78:0/2053937169 >> v2:172.21.15.150:3300/0 conn(0x7fa50c152700 0x7fa50c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 >> v1:172.21.15.150:6789/0 conn(0x7fa50c0c8e00 legacy=0x7fa50c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c155a50 con 0x7fa50c0c5730 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fa4f4001670 con 0x7fa50c0c5730 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 >> v1:172.21.15.78:6789/0 conn(0x7fa50c0c5730 legacy=0x7fa50c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f8003970 0x7fa4f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f80047c0 0x7fa4f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f8005180 0x7fa4f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c155770 con 0x7fa4f8005180 2024-02-19T14:57:32.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f800c120 con 0x7fa4f8003970 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.255+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f800c2e0 con 0x7fa4f80047c0 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f8005180 0x7fa4f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50affd640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f8003970 0x7fa4f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa503fff640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f80047c0 0x7fa4f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f80047c0 msgr2=0x7fa4f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f80047c0 0x7fa4f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f8003970 msgr2=0x7fa4f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f8003970 0x7fa4f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.256+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2053937169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8003520 con 0x7fa4f8005180 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2053937169 reap_dead start 2024-02-19T14:57:32.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f8005180 0x7fa4f800b980 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa504001e50 tx=0x7fa4f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57c21ae5b207d3b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504003b70 con 0x7fa4f8005180 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa504003d10 con 0x7fa4f8005180 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa50a7fc640 1 -- 172.21.15.78:0/2053937169 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504005310 con 0x7fa4f8005180 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 -- 172.21.15.78:0/2053937169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f8005180 msgr2=0x7fa4f800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f8005180 0x7fa4f800b980 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa504001e50 tx=0x7fa4f80057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 -- 172.21.15.78:0/2053937169 shutdown_connections 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 --2- 172.21.15.78:0/2053937169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4f8005180 0x7fa4f800b980 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 -- 172.21.15.78:0/2053937169 >> 172.21.15.78:0/2053937169 conn(0x7fa50c0cac90 msgr2=0x7fa50c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 -- 172.21.15.78:0/2053937169 shutdown_connections 2024-02-19T14:57:32.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.257+0000 7fa511807640 1 -- 172.21.15.78:0/2053937169 wait complete. 2024-02-19T14:57:32.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.258+0000 7fa511807640 1 Processor -- start 2024-02-19T14:57:32.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.258+0000 7fa511807640 1 -- start start 2024-02-19T14:57:32.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa511807640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f8005180 0x7fa50c0c0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa511807640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0aa280 0x7fa50c0bf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa511807640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 0x7fa50c0abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa511807640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c1ed9e0 con 0x7fa50c0bf710 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa511807640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c0bfba0 con 0x7fa50c0aa280 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa511807640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c0be410 con 0x7fa4f8005180 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 0x7fa50c0abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 0x7fa50c0abc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50792/0 (socket says 172.21.15.78:50792) 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2279257886 learned_addr learned my addr 172.21.15.78:0/2279257886 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa503fff640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0aa280 0x7fa50c0bf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50affd640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f8005180 0x7fa50c0c0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f8005180 msgr2=0x7fa50c0c0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f8005180 0x7fa50c0c0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0aa280 msgr2=0x7fa50c0bf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.259+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0aa280 0x7fa50c0bf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa50b7fe640 1 -- 172.21.15.78:0/2279257886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c0be6f0 con 0x7fa50c0bf710 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa503fff640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0aa280 0x7fa50c0bf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa50b7fe640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 0x7fa50c0abc00 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa504001f70 tx=0x7fa5040014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504006d50 con 0x7fa50c0bf710 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa504039480 con 0x7fa50c0bf710 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c0be9d0 con 0x7fa50c0bf710 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504039850 con 0x7fa50c0bf710 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.260+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c0c0ec0 con 0x7fa50c0bf710 2024-02-19T14:57:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.261+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50c0beef0 con 0x7fa50c0bf710 2024-02-19T14:57:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.262+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa5040399b0 con 0x7fa50c0bf710 2024-02-19T14:57:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.262+0000 7fa508ff9640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ec06b810 0x7fa4ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.262+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa5040ac070 con 0x7fa50c0bf710 2024-02-19T14:57:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.262+0000 7fa50affd640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ec06b810 0x7fa4ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.263+0000 7fa50affd640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ec06b810 0x7fa4ec06dcd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009a80 tx=0x7fa4fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.267+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa50c0beef0 con 0x7fa50c0bf710 2024-02-19T14:57:32.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.472+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa50c0beef0 con 0x7fa50c0bf710 2024-02-19T14:57:32.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.473+0000 7fa508ff9640 1 -- 172.21.15.78:0/2279257886 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4957 (secure 0 0 0) 0x7fa50c0beef0 con 0x7fa50c0bf710 2024-02-19T14:57:32.475 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:32.476 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:26.646012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":11,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:32.476 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 126 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.474+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ec06b810 msgr2=0x7fa4ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.474+0000 7fa511807640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ec06b810 0x7fa4ec06dcd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009a80 tx=0x7fa4fc008040 comp rx=0 tx=0).stop 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.474+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 msgr2=0x7fa50c0abc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.474+0000 7fa511807640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 0x7fa50c0abc00 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa504001f70 tx=0x7fa5040014a0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 shutdown_connections 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa4ec06b810 0x7fa4ec06dcd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa50c0bf710 0x7fa50c0abc00 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0aa280 0x7fa50c0bf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 --2- 172.21.15.78:0/2279257886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4f8005180 0x7fa50c0c0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 >> 172.21.15.78:0/2279257886 conn(0x7fa50c0cac90 msgr2=0x7fa50c14e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 shutdown_connections 2024-02-19T14:57:32.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.475+0000 7fa511807640 1 -- 172.21.15.78:0/2279257886 wait complete. 2024-02-19T14:57:32.488 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.585+0000 7fd7a921c640 1 Processor -- start 2024-02-19T14:57:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd7a921c640 1 -- start start 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd7a921c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd7a414ef50 0x7fd7a414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd7a921c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7a40727d0 0x7fd7a4072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd7a921c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40c35c0 con 0x7fd7a40c78a0 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd7a921c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a40c4bc0 con 0x7fd7a414ef50 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd7a921c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a414f920 con 0x7fd7a40727d0 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd79a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd7a414ef50 0x7fd7a414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd79a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd7a414ef50 0x7fd7a414f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50798/0 (socket says 172.21.15.78:50798) 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.586+0000 7fd79a59c640 1 -- 172.21.15.78:0/744203095 learned_addr learned my addr 172.21.15.78:0/744203095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.587+0000 7fd7a359e640 1 --2- 172.21.15.78:0/744203095 >> v2:172.21.15.150:3300/0 conn(0x7fd7a40727d0 0x7fd7a4072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.587+0000 7fd79a59c640 1 -- 172.21.15.78:0/744203095 >> v2:172.21.15.150:3300/0 conn(0x7fd7a40727d0 msgr2=0x7fd7a4072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.587+0000 7fd79a59c640 1 --2- 172.21.15.78:0/744203095 >> v2:172.21.15.150:3300/0 conn(0x7fd7a40727d0 0x7fd7a4072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.587+0000 7fd79a59c640 1 -- 172.21.15.78:0/744203095 >> v1:172.21.15.150:6789/0 conn(0x7fd7a40c78a0 legacy=0x7fd7a40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.587+0000 7fd79a59c640 1 -- 172.21.15.78:0/744203095 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40c38a0 con 0x7fd7a414ef50 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.587+0000 7fd7a359e640 1 --2- 172.21.15.78:0/744203095 >> v2:172.21.15.150:3300/0 conn(0x7fd7a40727d0 0x7fd7a4072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd79a59c640 1 --2- 172.21.15.78:0/744203095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a414ef50 0x7fd7a414f3e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40a8ea0 tx=0x7fd78c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd2fc1549fc9312 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a259c640 1 -- 172.21.15.78:0/744203095 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c02ee90 con 0x7fd7a414ef50 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a259c640 1 -- 172.21.15.78:0/744203095 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c036440 con 0x7fd7a414ef50 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a259c640 1 -- 172.21.15.78:0/744203095 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0367d0 con 0x7fd7a414ef50 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 -- 172.21.15.78:0/744203095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a414ef50 msgr2=0x7fd7a414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 --2- 172.21.15.78:0/744203095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a414ef50 0x7fd7a414f3e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40a8ea0 tx=0x7fd78c02e0e0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 -- 172.21.15.78:0/744203095 shutdown_connections 2024-02-19T14:57:32.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 --2- 172.21.15.78:0/744203095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a414ef50 0x7fd7a414f3e0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 --2- 172.21.15.78:0/744203095 >> v2:172.21.15.150:3300/0 conn(0x7fd7a40727d0 0x7fd7a4072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 -- 172.21.15.78:0/744203095 >> 172.21.15.78:0/744203095 conn(0x7fd7a4066e70 msgr2=0x7fd7a40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:32.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 -- 172.21.15.78:0/744203095 shutdown_connections 2024-02-19T14:57:32.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.588+0000 7fd7a921c640 1 -- 172.21.15.78:0/744203095 wait complete. 2024-02-19T14:57:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.589+0000 7fd7a921c640 1 Processor -- start 2024-02-19T14:57:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 -- start start 2024-02-19T14:57:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 0x7fd7a4148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7a40c78a0 0x7fd7a40aba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7a414ef50 0x7fd7a414fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40ab170 con 0x7fd7a40727d0 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40ac4d0 con 0x7fd7a40c78a0 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a921c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40aa880 con 0x7fd7a414ef50 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 0x7fd7a4148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 0x7fd7a4148cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50804/0 (socket says 172.21.15.78:50804) 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.590+0000 7fd7a2d9d640 1 -- 172.21.15.78:0/3956489489 learned_addr learned my addr 172.21.15.78:0/3956489489 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd79a59c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7a40c78a0 0x7fd7a40aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a359e640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7a414ef50 0x7fd7a414fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a2d9d640 1 -- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7a414ef50 msgr2=0x7fd7a414fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a2d9d640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7a414ef50 0x7fd7a414fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a2d9d640 1 -- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7a40c78a0 msgr2=0x7fd7a40aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a2d9d640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7a40c78a0 0x7fd7a40aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a2d9d640 1 -- 172.21.15.78:0/3956489489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c00b770 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a2d9d640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 0x7fd7a4148cc0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c430 tx=0x7fd79400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794011790 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7940118f0 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794011ce0 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a40aab20 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.591+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a4149200 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.592+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a4074990 con 0x7fd7a40727d0 2024-02-19T14:57:32.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.593+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd79401c460 con 0x7fd7a40727d0 2024-02-19T14:57:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.593+0000 7fd79bfff640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.593+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fd79408a2c0 con 0x7fd7a40727d0 2024-02-19T14:57:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.593+0000 7fd79a59c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:32.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.594+0000 7fd79a59c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd78c038000 tx=0x7fd78c032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.597+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd794054b30 con 0x7fd7a40727d0 2024-02-19T14:57:32.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.926+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7a406d680 con 0x7fd7a40727d0 2024-02-19T14:57:32.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.926+0000 7fd79bfff640 1 -- 172.21.15.78:0/3956489489 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+4957 (secure 0 0 0) 0x7fd7940581f0 con 0x7fd7a40727d0 2024-02-19T14:57:32.929 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:32.929 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":126,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:56:59.000185+0000","modified":"2024-02-19T14:57:26.646012+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":11,"addr":"172.21.15.78:6836/1973988605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1973988605},{"type":"v1","addr":"172.21.15.78:6836","nonce":1973988605}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-19T14:57:32.929 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 126 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 msgr2=0x7fd78406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd78c038000 tx=0x7fd78c032000 comp rx=0 tx=0).stop 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 msgr2=0x7fd7a4148cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 0x7fd7a4148cc0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c430 tx=0x7fd79400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 shutdown_connections 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7a414ef50 0x7fd7a414fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7a40c78a0 0x7fd7a40aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 --2- 172.21.15.78:0/3956489489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a40727d0 0x7fd7a4148cc0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 >> 172.21.15.78:0/3956489489 conn(0x7fd7a4066e70 msgr2=0x7fd7a40c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 shutdown_connections 2024-02-19T14:57:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:32.929+0000 7fd7a921c640 1 -- 172.21.15.78:0/3956489489 wait complete. 2024-02-19T14:57:32.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.042+0000 7f377ceb5640 1 Processor -- start 2024-02-19T14:57:33.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f377ceb5640 1 -- start start 2024-02-19T14:57:33.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f377ceb5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f37781537a0 0x7f37780c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f377ceb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f3778156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f377ceb5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37781563b0 con 0x7f377814cd50 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f377ceb5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3778153cd0 con 0x7f37781537a0 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f377ceb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377806b6e0 con 0x7f3778156860 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f376ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f37781537a0 0x7f37780c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f376ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f37781537a0 0x7f37780c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50810/0 (socket says 172.21.15.78:50810) 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.043+0000 7f376ffff640 1 -- 172.21.15.78:0/641820810 learned_addr learned my addr 172.21.15.78:0/641820810 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.044+0000 7f376ffff640 1 -- 172.21.15.78:0/641820810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 msgr2=0x7f3778156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.044+0000 7f376ffff640 1 --2- 172.21.15.78:0/641820810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f3778156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.044+0000 7f376ffff640 1 -- 172.21.15.78:0/641820810 >> v1:172.21.15.150:6789/0 conn(0x7f377814cd50 legacy=0x7f37780c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:33.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.044+0000 7f376ffff640 1 -- 172.21.15.78:0/641820810 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3778157230 con 0x7f37781537a0 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.044+0000 7f376ffff640 1 --2- 172.21.15.78:0/641820810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c4cd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3760009870 tx=0x7f376002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb34e742cfabaae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f3775d9b640 1 -- 172.21.15.78:0/641820810 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760031740 con 0x7f37781537a0 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f3775d9b640 1 -- 172.21.15.78:0/641820810 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37600318a0 con 0x7f37781537a0 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f3775d9b640 1 -- 172.21.15.78:0/641820810 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760031c30 con 0x7f37781537a0 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 -- 172.21.15.78:0/641820810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 msgr2=0x7f37780c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 --2- 172.21.15.78:0/641820810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c4cd0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3760009870 tx=0x7f376002b7e0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 -- 172.21.15.78:0/641820810 shutdown_connections 2024-02-19T14:57:33.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 --2- 172.21.15.78:0/641820810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f3778156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 --2- 172.21.15.78:0/641820810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c4cd0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 -- 172.21.15.78:0/641820810 >> 172.21.15.78:0/641820810 conn(0x7f3778066e70 msgr2=0x7f37780a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:33.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 -- 172.21.15.78:0/641820810 shutdown_connections 2024-02-19T14:57:33.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.045+0000 7f377ceb5640 1 -- 172.21.15.78:0/641820810 wait complete. 2024-02-19T14:57:33.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.046+0000 7f377ceb5640 1 Processor -- start 2024-02-19T14:57:33.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.046+0000 7f377ceb5640 1 -- start start 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377ceb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377814cd50 0x7f377814c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377ceb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377ceb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f37781ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377ceb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37781de4e0 con 0x7f37781537a0 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377ceb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37781dec50 con 0x7f377814cd50 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377ceb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37780c1ce0 con 0x7f3778156860 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f376ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f376ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50814/0 (socket says 172.21.15.78:50814) 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f376ffff640 1 -- 172.21.15.78:0/2709091595 learned_addr learned my addr 172.21.15.78:0/2709091595 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f377659c640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377814cd50 0x7f377814c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f376ffff640 1 -- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 msgr2=0x7f37781ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.047+0000 7f3776d9d640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f37781ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376ffff640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f37781ebd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376ffff640 1 -- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377814cd50 msgr2=0x7f377814c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376ffff640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377814cd50 0x7f377814c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376ffff640 1 -- 172.21.15.78:0/2709091595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37600094d0 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f377659c640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377814cd50 0x7f377814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376ffff640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c0e70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f3760009870 tx=0x7f37600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760033c70 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3760033dd0 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37781ec830 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376003d6b0 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.048+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37781de730 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.049+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3778074990 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.049+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f376003a040 con 0x7f37781537a0 2024-02-19T14:57:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.050+0000 7f376effd640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375806b810 0x7f375806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.050+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f37600ac280 con 0x7f37781537a0 2024-02-19T14:57:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.050+0000 7f377659c640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375806b810 0x7f375806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.050+0000 7f377659c640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375806b810 0x7f375806dcd0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f37780c8080 tx=0x7f3768008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.054+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3760076af0 con 0x7f37781537a0 2024-02-19T14:57:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.326+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37780c4230 con 0x7f37781537a0 2024-02-19T14:57:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.327+0000 7f376effd640 1 -- 172.21.15.78:0/2709091595 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17143 (secure 0 0 0) 0x7f376007a1b0 con 0x7f37781537a0 2024-02-19T14:57:33.330 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:33.330 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":140,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:24.626444+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:56:57.223056+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-19T14:56:57.972812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6833/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6834/2425259283":"2024-02-20T14:57:10.590067+0000","172.21.15.150:6832/2425259283":"2024-02-20T14:57:10.590067+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375806b810 msgr2=0x7f375806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375806b810 0x7f375806dcd0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f37780c8080 tx=0x7f3768008040 comp rx=0 tx=0).stop 2024-02-19T14:57:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 msgr2=0x7f37780c0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c0e70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f3760009870 tx=0x7f37600046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 shutdown_connections 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f375806b810 0x7f375806dcd0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781537a0 0x7f37780c0e70 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3778156860 0x7f37781ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 --2- 172.21.15.78:0/2709091595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377814cd50 0x7f377814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 >> 172.21.15.78:0/2709091595 conn(0x7f3778066e70 msgr2=0x7f37781525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.329+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 shutdown_connections 2024-02-19T14:57:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.330+0000 7f377ceb5640 1 -- 172.21.15.78:0/2709091595 wait complete. 2024-02-19T14:57:33.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:57:33.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.443+0000 7f503f20e640 1 Processor -- start 2024-02-19T14:57:33.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.443+0000 7f503f20e640 1 -- start start 2024-02-19T14:57:33.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.444+0000 7f503f20e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380713a0 0x7f5038071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.444+0000 7f503f20e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50380c8b60 0x7f50381536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.444+0000 7f503f20e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503806fe40 con 0x7f50380c29e0 2024-02-19T14:57:33.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.444+0000 7f503f20e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50380c32c0 con 0x7f50380c8b60 2024-02-19T14:57:33.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.444+0000 7f503f20e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5038070500 con 0x7f50380713a0 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50380c8b60 0x7f50381536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50380c8b60 0x7f50381536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:38464/0 (socket says 172.21.15.78:38464) 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 -- 172.21.15.78:0/2595485961 learned_addr learned my addr 172.21.15.78:0/2595485961 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f5037fff640 1 --1- 172.21.15.78:0/2595485961 >> v1:172.21.15.150:6789/0 conn(0x7f50380c29e0 0x7f50380c64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:40484/0 (socket says 172.21.15.78:40484) 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503cfaa640 1 --2- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380713a0 0x7f5038071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 -- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380713a0 msgr2=0x7f5038071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 --2- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380713a0 0x7f5038071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 -- 172.21.15.78:0/2595485961 >> v1:172.21.15.150:6789/0 conn(0x7f50380c29e0 legacy=0x7f50380c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.445+0000 7f503d7ab640 1 -- 172.21.15.78:0/2595485961 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5038153bf0 con 0x7f50380c8b60 2024-02-19T14:57:33.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.446+0000 7f503d7ab640 1 --2- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381536b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5030009e50 tx=0x7f503002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=438ef7532fe442e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.446+0000 7f50377fe640 1 -- 172.21.15.78:0/2595485961 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503002ca70 con 0x7f50380c8b60 2024-02-19T14:57:33.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.446+0000 7f50377fe640 1 -- 172.21.15.78:0/2595485961 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503002cbd0 con 0x7f50380c8b60 2024-02-19T14:57:33.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.446+0000 7f50377fe640 1 -- 172.21.15.78:0/2595485961 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50300343f0 con 0x7f50380c8b60 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 -- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 msgr2=0x7f50381536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 --2- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381536b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5030009e50 tx=0x7f503002bed0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 -- 172.21.15.78:0/2595485961 shutdown_connections 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 --2- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381536b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 --2- 172.21.15.78:0/2595485961 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380713a0 0x7f5038071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 -- 172.21.15.78:0/2595485961 >> 172.21.15.78:0/2595485961 conn(0x7f50380aa300 msgr2=0x7f50380aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 -- 172.21.15.78:0/2595485961 shutdown_connections 2024-02-19T14:57:33.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.447+0000 7f503f20e640 1 -- 172.21.15.78:0/2595485961 wait complete. 2024-02-19T14:57:33.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.448+0000 7f503f20e640 1 Processor -- start 2024-02-19T14:57:33.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.448+0000 7f503f20e640 1 -- start start 2024-02-19T14:57:33.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503f20e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 0x7f50380c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503f20e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380c29e0 0x7f5038154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503f20e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503f20e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50381eafb0 con 0x7f50380713a0 2024-02-19T14:57:33.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503f20e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5038156a80 con 0x7f50380c8b60 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503f20e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50381eacd0 con 0x7f50380c29e0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 0x7f50380c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 0x7f50380c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50822/0 (socket says 172.21.15.78:50822) 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 -- 172.21.15.78:0/602280073 learned_addr learned my addr 172.21.15.78:0/602280073 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503d7ab640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f5037fff640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380c29e0 0x7f5038154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 -- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380c29e0 msgr2=0x7f5038154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380c29e0 0x7f5038154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 -- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 msgr2=0x7f50381ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 -- 172.21.15.78:0/602280073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5030009b30 con 0x7f50380713a0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.449+0000 7f503cfaa640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 0x7f50380c1bf0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f502800e9b0 tx=0x7f502800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.450+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028009670 con 0x7f50380713a0 2024-02-19T14:57:33.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.450+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502800c9a0 con 0x7f50380713a0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.450+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503806c130 con 0x7f50380713a0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.450+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502800cd50 con 0x7f50380713a0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.450+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50380c2130 con 0x7f50380713a0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.451+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50380addb0 con 0x7f50380713a0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.451+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5028013470 con 0x7f50380713a0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.452+0000 7f50357fa640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f502406b810 0x7f502406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.452+0000 7f5037fff640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f502406b810 0x7f502406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.452+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f502808a500 con 0x7f50380713a0 2024-02-19T14:57:33.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.452+0000 7f5037fff640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f502406b810 0x7f502406dcd0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f50380c29b0 tx=0x7f502c007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.456+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5028054e20 con 0x7f50380713a0 2024-02-19T14:57:33.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.669+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5038074990 con 0x7f50380713a0 2024-02-19T14:57:33.672 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:57:33.669+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:57:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.671+0000 7f50357fa640 1 -- 172.21.15.78:0/602280073 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v127) v1 ==== 161+0+0 (secure 0 0 0) 0x7f50280584e0 con 0x7f50380713a0 2024-02-19T14:57:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:57:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.673+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f502406b810 msgr2=0x7f502406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.673+0000 7f503f20e640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f502406b810 0x7f502406dcd0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f50380c29b0 tx=0x7f502c007e80 comp rx=0 tx=0).stop 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.673+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 msgr2=0x7f50380c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.673+0000 7f503f20e640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 0x7f50380c1bf0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f502800e9b0 tx=0x7f502800ee70 comp rx=0 tx=0).stop 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 shutdown_connections 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f502406b810 0x7f502406dcd0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50380c8b60 0x7f50381ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50380c29e0 0x7f5038154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 --2- 172.21.15.78:0/602280073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380713a0 0x7f50380c1bf0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 >> 172.21.15.78:0/602280073 conn(0x7f50380aa300 msgr2=0x7f50380a8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 shutdown_connections 2024-02-19T14:57:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.674+0000 7f503f20e640 1 -- 172.21.15.78:0/602280073 wait complete. 2024-02-19T14:57:33.678 INFO:tasks.ceph.mds.c.smithi078.stderr: -122> 2024-02-19T14:57:29.928+0000 7fdf0eb37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:33.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:57:33.705 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:57:33.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.791+0000 7f3c4c3bd640 1 Processor -- start 2024-02-19T14:57:33.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.791+0000 7f3c4c3bd640 1 -- start start 2024-02-19T14:57:33.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4c3bd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c44152480 0x7f3c441559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4c3bd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c440c87a0 0x7f3c440c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4c3bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c44155ef0 con 0x7f3c440c7070 2024-02-19T14:57:33.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4c3bd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4406dec0 con 0x7f3c44152480 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4c3bd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4406d520 con 0x7f3c440c87a0 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a159640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c44152480 0x7f3c441559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c49958640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3c440c7070 0x7f3c4406ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41424/0 (socket says 172.21.15.78:41424) 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c49958640 1 -- 172.21.15.78:0/99939445 learned_addr learned my addr 172.21.15.78:0/99939445 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a95a640 1 --2- 172.21.15.78:0/99939445 >> v2:172.21.15.150:3300/0 conn(0x7f3c440c87a0 0x7f3c440c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a159640 1 -- 172.21.15.78:0/99939445 >> v2:172.21.15.150:3300/0 conn(0x7f3c440c87a0 msgr2=0x7f3c440c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a159640 1 --2- 172.21.15.78:0/99939445 >> v2:172.21.15.150:3300/0 conn(0x7f3c440c87a0 0x7f3c440c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a159640 1 -- 172.21.15.78:0/99939445 >> v1:172.21.15.78:6789/0 conn(0x7f3c440c7070 legacy=0x7f3c4406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a159640 1 -- 172.21.15.78:0/99939445 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c440c75c0 con 0x7f3c44152480 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c49157640 1 -- 172.21.15.78:0/99939445 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361727770 0 0) 0x7f3c34001110 con 0x7f3c440c7070 2024-02-19T14:57:33.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.792+0000 7f3c4a95a640 1 --2- 172.21.15.78:0/99939445 >> v2:172.21.15.150:3300/0 conn(0x7f3c440c87a0 0x7f3c440c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.793+0000 7f3c4a159640 1 --2- 172.21.15.78:0/99939445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c44152480 0x7f3c441559b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f3c440ca780 tx=0x7f3c4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918b6572e7378d14 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.793+0000 7f3c49157640 1 -- 172.21.15.78:0/99939445 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40031830 con 0x7f3c44152480 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.793+0000 7f3c49157640 1 -- 172.21.15.78:0/99939445 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40031990 con 0x7f3c44152480 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.793+0000 7f3c49157640 1 -- 172.21.15.78:0/99939445 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40031d20 con 0x7f3c44152480 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/99939445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c44152480 msgr2=0x7f3c441559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/99939445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c44152480 0x7f3c441559b0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f3c440ca780 tx=0x7f3c4002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/99939445 shutdown_connections 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/99939445 >> v2:172.21.15.150:3300/0 conn(0x7f3c440c87a0 0x7f3c440c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/99939445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c44152480 0x7f3c441559b0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/99939445 >> 172.21.15.78:0/99939445 conn(0x7f3c440ca8a0 msgr2=0x7f3c440cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/99939445 shutdown_connections 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.794+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/99939445 wait complete. 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.795+0000 7f3c4c3bd640 1 Processor -- start 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.795+0000 7f3c4c3bd640 1 -- start start 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c440c7070 0x7f3c440af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 0x7f3c441567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c44152480 0x7f3c44156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440adeb0 con 0x7f3c440c87a0 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440ae750 con 0x7f3c440c7070 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440afde0 con 0x7f3c44152480 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 0x7f3c441567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 0x7f3c441567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50854/0 (socket says 172.21.15.78:50854) 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 -- 172.21.15.78:0/2063180522 learned_addr learned my addr 172.21.15.78:0/2063180522 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4a95a640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c44152480 0x7f3c44156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4a159640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c440c7070 0x7f3c440af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 -- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c44152480 msgr2=0x7f3c44156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c44152480 0x7f3c44156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 -- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c440c7070 msgr2=0x7f3c440af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c440c7070 0x7f3c440af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 -- 172.21.15.78:0/2063180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40009400 con 0x7f3c440c87a0 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4a159640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c440c7070 0x7f3c440af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c49958640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 0x7f3c441567a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c7b0 tx=0x7f3c3400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c340072d0 con 0x7f3c440c87a0 2024-02-19T14:57:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c34007450 con 0x7f3c440c87a0 2024-02-19T14:57:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c440ae100 con 0x7f3c440c87a0 2024-02-19T14:57:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34001d10 con 0x7f3c440c87a0 2024-02-19T14:57:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.796+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c441544f0 con 0x7f3c440c87a0 2024-02-19T14:57:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.797+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44074990 con 0x7f3c440c87a0 2024-02-19T14:57:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.798+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3c340047e0 con 0x7f3c440c87a0 2024-02-19T14:57:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.798+0000 7f3c3affd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c2806b810 0x7f3c2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.798+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f3c340896f0 con 0x7f3c440c87a0 2024-02-19T14:57:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.798+0000 7f3c4a159640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c2806b810 0x7f3c2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.799+0000 7f3c4a159640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c2806b810 0x7f3c2806dcd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3c40036000 tx=0x7f3c40008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:33.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:33.802+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3c34053fc0 con 0x7f3c440c87a0 2024-02-19T14:57:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.011+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3c44155ef0 con 0x7f3c440c87a0 2024-02-19T14:57:34.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.683+0000 7f3c3affd640 1 -- 172.21.15.78:0/2063180522 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v128) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3c44155ef0 con 0x7f3c440c87a0 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c2806b810 msgr2=0x7f3c2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c2806b810 0x7f3c2806dcd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3c40036000 tx=0x7f3c40008030 comp rx=0 tx=0).stop 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 msgr2=0x7f3c441567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 0x7f3c441567a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c7b0 tx=0x7f3c3400cc70 comp rx=0 tx=0).stop 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 shutdown_connections 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c2806b810 0x7f3c2806dcd0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c440c87a0 0x7f3c441567a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c44152480 0x7f3c44156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 --2- 172.21.15.78:0/2063180522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c440c7070 0x7f3c440af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 >> 172.21.15.78:0/2063180522 conn(0x7f3c440ca8a0 msgr2=0x7f3c440c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:34.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 shutdown_connections 2024-02-19T14:57:34.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.685+0000 7f3c4c3bd640 1 -- 172.21.15.78:0/2063180522 wait complete. 2024-02-19T14:57:34.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:57:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.846+0000 7fee2c4df640 1 Processor -- start 2024-02-19T14:57:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.846+0000 7fee2c4df640 1 -- start start 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2c4df640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee240ab8c0 0x7fee240c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2c4df640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee240a9630 0x7fee2406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2c4df640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee241530c0 con 0x7fee240c4ec0 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2c4df640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee240aa2f0 con 0x7fee240a9630 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2c4df640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee240c5c40 con 0x7fee240ab8c0 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2aa7c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee240a9630 0x7fee2406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2a27b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee240c4ec0 0x7fee24153550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41430/0 (socket says 172.21.15.78:41430) 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2a27b640 1 -- 172.21.15.78:0/319630565 learned_addr learned my addr 172.21.15.78:0/319630565 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2aa7c640 1 -- 172.21.15.78:0/319630565 >> v2:172.21.15.150:3300/0 conn(0x7fee240ab8c0 msgr2=0x7fee240c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2aa7c640 1 --2- 172.21.15.78:0/319630565 >> v2:172.21.15.150:3300/0 conn(0x7fee240ab8c0 0x7fee240c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2aa7c640 1 -- 172.21.15.78:0/319630565 >> v1:172.21.15.78:6789/0 conn(0x7fee240c4ec0 legacy=0x7fee24153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee29279640 1 -- 172.21.15.78:0/319630565 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38968259 0 0) 0x7fee241530c0 con 0x7fee240c4ec0 2024-02-19T14:57:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.847+0000 7fee2aa7c640 1 -- 172.21.15.78:0/319630565 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240c5ee0 con 0x7fee240a9630 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee2aa7c640 1 --2- 172.21.15.78:0/319630565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240a9630 0x7fee2406ae80 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fee240a6270 tx=0x7fee1c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc565b36a1a65147 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee29279640 1 -- 172.21.15.78:0/319630565 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c02cbf0 con 0x7fee240a9630 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee29279640 1 -- 172.21.15.78:0/319630565 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c02cd50 con 0x7fee240a9630 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee29279640 1 -- 172.21.15.78:0/319630565 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c034660 con 0x7fee240a9630 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee2c4df640 1 -- 172.21.15.78:0/319630565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240a9630 msgr2=0x7fee2406ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee2c4df640 1 --2- 172.21.15.78:0/319630565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240a9630 0x7fee2406ae80 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fee240a6270 tx=0x7fee1c02be40 comp rx=0 tx=0).stop 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.848+0000 7fee2c4df640 1 -- 172.21.15.78:0/319630565 shutdown_connections 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.849+0000 7fee2c4df640 1 --2- 172.21.15.78:0/319630565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240a9630 0x7fee2406ae80 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.849+0000 7fee2c4df640 1 --2- 172.21.15.78:0/319630565 >> v2:172.21.15.150:3300/0 conn(0x7fee240ab8c0 0x7fee240c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.849+0000 7fee2c4df640 1 -- 172.21.15.78:0/319630565 >> 172.21.15.78:0/319630565 conn(0x7fee240c6fe0 msgr2=0x7fee240a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.849+0000 7fee2c4df640 1 -- 172.21.15.78:0/319630565 shutdown_connections 2024-02-19T14:57:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.849+0000 7fee2c4df640 1 -- 172.21.15.78:0/319630565 wait complete. 2024-02-19T14:57:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 Processor -- start 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 -- start start 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240a9630 0x7fee240c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee240ab8c0 0x7fee2414fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 0x7fee240bc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee240befb0 con 0x7fee240c4ec0 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241ec520 con 0x7fee240a9630 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2c4df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee241ec800 con 0x7fee240ab8c0 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2aa7c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 0x7fee240bc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2aa7c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 0x7fee240bc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50882/0 (socket says 172.21.15.78:50882) 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.850+0000 7fee2aa7c640 1 -- 172.21.15.78:0/1634166899 learned_addr learned my addr 172.21.15.78:0/1634166899 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2aa7c640 1 -- 172.21.15.78:0/1634166899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee240ab8c0 msgr2=0x7fee2414fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2aa7c640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee240ab8c0 0x7fee2414fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2aa7c640 1 -- 172.21.15.78:0/1634166899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240a9630 msgr2=0x7fee240c01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2aa7c640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240a9630 0x7fee240c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2aa7c640 1 -- 172.21.15.78:0/1634166899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c009a70 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2aa7c640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 0x7fee240bc520 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009e80 tx=0x7fee1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c040450 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c0405b0 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee240bf9b0 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c040940 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.851+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee240bf6b0 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.852+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee240678b0 con 0x7fee240c4ec0 2024-02-19T14:57:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.852+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fee1c040aa0 con 0x7fee240c4ec0 2024-02-19T14:57:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.853+0000 7fee1affd640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee0806b840 0x7fee0806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.853+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fee1c0acfd0 con 0x7fee240c4ec0 2024-02-19T14:57:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.853+0000 7fee2a27b640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee0806b840 0x7fee0806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.853+0000 7fee2a27b640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee0806b840 0x7fee0806dd00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fee20007fd0 tx=0x7fee20007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:34.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:34.857+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee1c0777f0 con 0x7fee240c4ec0 2024-02-19T14:57:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.027+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fee240bfed0 con 0x7fee240c4ec0 2024-02-19T14:57:35.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.694+0000 7fee1affd640 1 -- 172.21.15.78:0/1634166899 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) v1 ==== 181+0+0 (secure 0 0 0) 0x7fee240bfed0 con 0x7fee240c4ec0 2024-02-19T14:57:35.697 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee0806b840 msgr2=0x7fee0806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee0806b840 0x7fee0806dd00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fee20007fd0 tx=0x7fee20007920 comp rx=0 tx=0).stop 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 msgr2=0x7fee240bc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 0x7fee240bc520 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009e80 tx=0x7fee1c004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 shutdown_connections 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee0806b840 0x7fee0806dd00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee240c4ec0 0x7fee240bc520 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee240ab8c0 0x7fee2414fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 --2- 172.21.15.78:0/1634166899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee240a9630 0x7fee240c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 >> 172.21.15.78:0/1634166899 conn(0x7fee240c6fe0 msgr2=0x7fee240c25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:35.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 shutdown_connections 2024-02-19T14:57:35.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.696+0000 7fee2c4df640 1 -- 172.21.15.78:0/1634166899 wait complete. 2024-02-19T14:57:35.713 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:57:35.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.814+0000 7f2b41054640 1 Processor -- start 2024-02-19T14:57:35.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.814+0000 7f2b41054640 1 -- start start 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b41054640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c155da0 0x7f2b3c0c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b41054640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2b3c0adde0 0x7f2b3c0c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b41054640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0c27e0 con 0x7f2b3c0c6960 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b41054640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c0c3a90 con 0x7f2b3c0adde0 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b41054640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0c3d70 con 0x7f2b3c155da0 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b3b59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2b3c0adde0 0x7f2b3c0c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b3b59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2b3c0adde0 0x7f2b3c0c2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50896/0 (socket says 172.21.15.78:50896) 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b3b59e640 1 -- 172.21.15.78:0/39068048 learned_addr learned my addr 172.21.15.78:0/39068048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.815+0000 7f2b3ad9d640 1 --2- 172.21.15.78:0/39068048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c155da0 0x7f2b3c0c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:35.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b3b59e640 1 -- 172.21.15.78:0/39068048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c155da0 msgr2=0x7f2b3c0c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b3b59e640 1 --2- 172.21.15.78:0/39068048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c155da0 0x7f2b3c0c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b3b59e640 1 -- 172.21.15.78:0/39068048 >> v1:172.21.15.78:6789/0 conn(0x7f2b3c0c6960 legacy=0x7f2b3c0c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b3b59e640 1 -- 172.21.15.78:0/39068048 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c06d070 con 0x7f2b3c0adde0 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b3b59e640 1 --2- 172.21.15.78:0/39068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c2170 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009870 tx=0x7f2b3402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa0c94f19805a0c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b39d9b640 1 -- 172.21.15.78:0/39068048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34031640 con 0x7f2b3c0adde0 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b39d9b640 1 -- 172.21.15.78:0/39068048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b340317a0 con 0x7f2b3c0adde0 2024-02-19T14:57:35.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.816+0000 7f2b39d9b640 1 -- 172.21.15.78:0/39068048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34031b30 con 0x7f2b3c0adde0 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 -- 172.21.15.78:0/39068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c0adde0 msgr2=0x7f2b3c0c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 --2- 172.21.15.78:0/39068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c2170 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009870 tx=0x7f2b3402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 -- 172.21.15.78:0/39068048 shutdown_connections 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 --2- 172.21.15.78:0/39068048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c2170 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 --2- 172.21.15.78:0/39068048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c155da0 0x7f2b3c0c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 -- 172.21.15.78:0/39068048 >> 172.21.15.78:0/39068048 conn(0x7f2b3c0aa3e0 msgr2=0x7f2b3c0aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 -- 172.21.15.78:0/39068048 shutdown_connections 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.817+0000 7f2b41054640 1 -- 172.21.15.78:0/39068048 wait complete. 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.818+0000 7f2b41054640 1 Processor -- start 2024-02-19T14:57:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.818+0000 7f2b41054640 1 -- start start 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b41054640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b41054640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c0c6960 0x7f2b3c1eddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b41054640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 0x7f2b3c14db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b41054640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c1ea7c0 con 0x7f2b3c155da0 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b41054640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c1eb000 con 0x7f2b3c0adde0 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b41054640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0abce0 con 0x7f2b3c0c6960 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 0x7f2b3c14db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 0x7f2b3c14db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50908/0 (socket says 172.21.15.78:50908) 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 -- 172.21.15.78:0/3093488425 learned_addr learned my addr 172.21.15.78:0/3093488425 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3ad9d640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3a59c640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c0c6960 0x7f2b3c1eddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 -- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c0c6960 msgr2=0x7f2b3c1eddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:35.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c0c6960 0x7f2b3c1eddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 -- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b3c0adde0 msgr2=0x7f2b3c0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.819+0000 7f2b3b59e640 1 -- 172.21.15.78:0/3093488425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b340094d0 con 0x7f2b3c155da0 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.820+0000 7f2b3b59e640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 0x7f2b3c14db20 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009870 tx=0x7f2b340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.820+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3403ac60 con 0x7f2b3c155da0 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.820+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3403ade0 con 0x7f2b3c155da0 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.820+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c06d710 con 0x7f2b3c155da0 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.820+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34044630 con 0x7f2b3c155da0 2024-02-19T14:57:35.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.820+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c1eac90 con 0x7f2b3c155da0 2024-02-19T14:57:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.821+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c06e120 con 0x7f2b3c155da0 2024-02-19T14:57:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.821+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2b3402f050 con 0x7f2b3c155da0 2024-02-19T14:57:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.821+0000 7f2b237fe640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b2806b810 0x7f2b2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.822+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5921+0+0 (secure 0 0 0) 0x7f2b340ac090 con 0x7f2b3c155da0 2024-02-19T14:57:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.822+0000 7f2b3ad9d640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b2806b810 0x7f2b2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.822+0000 7f2b3ad9d640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b2806b810 0x7f2b2806dcd0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c003aa0 tx=0x7f2b2c009830 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:35.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:35.826+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2b34076a00 con 0x7f2b3c155da0 2024-02-19T14:57:36.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.004+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2b3c1eb170 con 0x7f2b3c155da0 2024-02-19T14:57:36.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.702+0000 7f2b237fe640 1 -- 172.21.15.78:0/3093488425 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2b3407a0c0 con 0x7f2b3c155da0 2024-02-19T14:57:36.705 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:57:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.705+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b2806b810 msgr2=0x7f2b2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.705+0000 7f2b41054640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b2806b810 0x7f2b2806dcd0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c003aa0 tx=0x7f2b2c009830 comp rx=0 tx=0).stop 2024-02-19T14:57:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.705+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 msgr2=0x7f2b3c14db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.705+0000 7f2b41054640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 0x7f2b3c14db20 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009870 tx=0x7f2b340046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 shutdown_connections 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b2806b810 0x7f2b2806dcd0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b3c155da0 0x7f2b3c14db20 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b3c0c6960 0x7f2b3c1eddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 --2- 172.21.15.78:0/3093488425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b3c0adde0 0x7f2b3c0c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 >> 172.21.15.78:0/3093488425 conn(0x7f2b3c0aa3e0 msgr2=0x7f2b3c0aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 shutdown_connections 2024-02-19T14:57:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.706+0000 7f2b41054640 1 -- 172.21.15.78:0/3093488425 wait complete. 2024-02-19T14:57:36.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-19T14:57:36.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.827+0000 7f31c6ec3640 1 Processor -- start 2024-02-19T14:57:36.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.827+0000 7f31c6ec3640 1 -- start start 2024-02-19T14:57:36.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c6ec3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31c00c6d30 0x7f31c0158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c6ec3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c00c6af0 con 0x7f31c0155f90 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c6ec3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c00c7940 con 0x7f31c006d550 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c6ec3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c00c7c20 con 0x7f31c00c6d30 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c4c5f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f31c0155f90 0x7f31c00c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41448/0 (socket says 172.21.15.78:41448) 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c4c5f640 1 -- 172.21.15.78:0/157543156 learned_addr learned my addr 172.21.15.78:0/157543156 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963709212 0 0) 0x7f31c00c6af0 con 0x7f31c0155f90 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.828+0000 7f31c5460640 1 --2- 172.21.15.78:0/157543156 >> v2:172.21.15.150:3300/0 conn(0x7f31c00c6d30 0x7f31c0158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a8003520 con 0x7f31c0155f90 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750255506 0 0) 0x7f31a8003520 con 0x7f31c0155f90 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 >> v2:172.21.15.150:3300/0 conn(0x7f31c00c6d30 msgr2=0x7f31c0158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 --2- 172.21.15.78:0/157543156 >> v2:172.21.15.150:3300/0 conn(0x7f31c00c6d30 0x7f31c0158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 >> v1:172.21.15.150:6789/0 conn(0x7f31c006d550 legacy=0x7f31c01588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c00c7f00 con 0x7f31c0155f90 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f31b0001670 con 0x7f31c0155f90 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 >> v1:172.21.15.78:6789/0 conn(0x7f31c0155f90 legacy=0x7f31c00c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31a8003970 0x7f31a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31a80047c0 0x7f31a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8005180 0x7f31a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00c7c20 con 0x7f31a8003970 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a800c120 con 0x7f31a8005180 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a800c2e0 con 0x7f31a80047c0 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31a8003970 0x7f31a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c5460640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8005180 0x7f31a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31bffff640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31a80047c0 0x7f31a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c4c5f640 1 -- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31a80047c0 msgr2=0x7f31a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:36.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31a80047c0 0x7f31a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c4c5f640 1 -- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8005180 msgr2=0x7f31a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8005180 0x7f31a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c4c5f640 1 -- 172.21.15.78:0/157543156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8003520 con 0x7f31a8003970 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c5460640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8005180 0x7f31a800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.829+0000 7f31c5460640 1 -- 172.21.15.78:0/157543156 reap_dead start 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.830+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31a8003970 0x7f31a8003d90 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c240 tx=0x7f31a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a142574d0fdc55dc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.830+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0005ad0 con 0x7f31a8003970 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.830+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b0005c30 con 0x7f31a8003970 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.830+0000 7f31bf7fe640 1 -- 172.21.15.78:0/157543156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b002cab0 con 0x7f31a8003970 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.830+0000 7f31c6ec3640 1 -- 172.21.15.78:0/157543156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31a8003970 msgr2=0x7f31a8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.830+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31a8003970 0x7f31a8003d90 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c240 tx=0x7f31a80057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.831+0000 7f31c6ec3640 1 -- 172.21.15.78:0/157543156 shutdown_connections 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.831+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/157543156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31a8003970 0x7f31a8003d90 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.831+0000 7f31c6ec3640 1 -- 172.21.15.78:0/157543156 >> 172.21.15.78:0/157543156 conn(0x7f31c006b700 msgr2=0x7f31c00aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.831+0000 7f31c6ec3640 1 -- 172.21.15.78:0/157543156 shutdown_connections 2024-02-19T14:57:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.831+0000 7f31c6ec3640 1 -- 172.21.15.78:0/157543156 wait complete. 2024-02-19T14:57:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.832+0000 7f31c6ec3640 1 Processor -- start 2024-02-19T14:57:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.832+0000 7f31c6ec3640 1 -- start start 2024-02-19T14:57:36.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.832+0000 7f31c6ec3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003970 0x7f31c01eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31c014eeb0 0x7f31c014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 0x7f31c00ad340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0071790 con 0x7f31c00acec0 2024-02-19T14:57:36.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00a8a70 con 0x7f31a8003970 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00a8520 con 0x7f31c014eeb0 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 0x7f31c00ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 0x7f31c00ad340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50918/0 (socket says 172.21.15.78:50918) 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 -- 172.21.15.78:0/1152281946 learned_addr learned my addr 172.21.15.78:0/1152281946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003970 0x7f31c01eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31bffff640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31c014eeb0 0x7f31c014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 -- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31c014eeb0 msgr2=0x7f31c014e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31c014eeb0 0x7f31c014e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 -- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003970 msgr2=0x7f31c01eb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003970 0x7f31c01eb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 -- 172.21.15.78:0/1152281946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c01eb5e0 con 0x7f31c00acec0 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31bffff640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31c014eeb0 0x7f31c014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:36.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003970 0x7f31c01eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c5460640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 0x7f31c00ad340 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f31c00c29f0 tx=0x7f31b8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800e050 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b800f040 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c00a8790 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8003990 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.833+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c00ab800 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.834+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c0074990 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.835+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f31b8015440 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.835+0000 7f31bd7fa640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31ac06b840 0x7f31ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.835+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f31b808a5b0 con 0x7f31c00acec0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.835+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31ac06b840 0x7f31ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:36.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.836+0000 7f31c4c5f640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31ac06b840 0x7f31ac06dd00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f31b0034000 tx=0x7f31b0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:36.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:36.840+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31b8055120 con 0x7f31c00acec0 2024-02-19T14:57:37.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.017+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f31c00c75a0 con 0x7f31c00acec0 2024-02-19T14:57:37.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.020+0000 7f31bd7fa640 1 -- 172.21.15.78:0/1152281946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v49) v1 ==== 96+0+0 (secure 0 0 0) 0x7f31b80587e0 con 0x7f31c00acec0 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31ac06b840 msgr2=0x7f31ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31ac06b840 0x7f31ac06dd00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f31b0034000 tx=0x7f31b0009030 comp rx=0 tx=0).stop 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 msgr2=0x7f31c00ad340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 0x7f31c00ad340 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f31c00c29f0 tx=0x7f31b8010590 comp rx=0 tx=0).stop 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 shutdown_connections 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31ac06b840 0x7f31ac06dd00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31c00acec0 0x7f31c00ad340 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31c014eeb0 0x7f31c014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 --2- 172.21.15.78:0/1152281946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003970 0x7f31c01eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 >> 172.21.15.78:0/1152281946 conn(0x7f31c006b700 msgr2=0x7f31c00aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 shutdown_connections 2024-02-19T14:57:37.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.022+0000 7f31c6ec3640 1 -- 172.21.15.78:0/1152281946 wait complete. 2024-02-19T14:57:37.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:57:37.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.135+0000 7fa94cb2c640 1 Processor -- start 2024-02-19T14:57:37.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.135+0000 7fa94cb2c640 1 -- start start 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.136+0000 7fa94cb2c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa9480c5560 0x7fa9480c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.136+0000 7fa94cb2c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9480c65a0 con 0x7fa9480c7910 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.136+0000 7fa94cb2c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9480c7530 con 0x7fa948070720 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.136+0000 7fa94cb2c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9481537b0 con 0x7fa9480c5560 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.136+0000 7fa945d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa9480c7910 0x7fa948154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41452/0 (socket says 172.21.15.78:41452) 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.136+0000 7fa945d9b640 1 -- 172.21.15.78:0/1350627087 learned_addr learned my addr 172.21.15.78:0/1350627087 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214053042 0 0) 0x7fa9480c65a0 con 0x7fa9480c7910 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 --2- 172.21.15.78:0/1350627087 >> v2:172.21.15.150:3300/0 conn(0x7fa9480c5560 0x7fa9480c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa930003520 con 0x7fa9480c7910 2024-02-19T14:57:37.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4285281667 0 0) 0x7fa930003520 con 0x7fa9480c7910 2024-02-19T14:57:37.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 >> v2:172.21.15.150:3300/0 conn(0x7fa9480c5560 msgr2=0x7fa9480c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:37.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 --2- 172.21.15.78:0/1350627087 >> v2:172.21.15.150:3300/0 conn(0x7fa9480c5560 0x7fa9480c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 >> v1:172.21.15.150:6789/0 conn(0x7fa948070720 legacy=0x7fa948070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa948153a90 con 0x7fa9480c7910 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fa93c001670 con 0x7fa9480c7910 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 >> v1:172.21.15.78:6789/0 conn(0x7fa9480c7910 legacy=0x7fa948154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa930003970 0x7fa930003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9300052a0 0x7fa93000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9481537b0 con 0x7fa930003970 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93000c170 con 0x7fa9300052a0 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93000c330 con 0x7fa9300047c0 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa930003970 0x7fa930003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa946d9d640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9300052a0 0x7fa93000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa945d9b640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 -- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9300047c0 msgr2=0x7fa930004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 -- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9300052a0 msgr2=0x7fa93000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:37.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9300052a0 0x7fa93000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa946d9d640 1 -- 172.21.15.78:0/1350627087 reap_dead start 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.137+0000 7fa94659c640 1 -- 172.21.15.78:0/1350627087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930003520 con 0x7fa930003970 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.138+0000 7fa94659c640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa930003970 0x7fa930003d90 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fa938009f90 tx=0x7fa9300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3a1e47fa4bf978 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.138+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93802cbd0 con 0x7fa930003970 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.138+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93800f040 con 0x7fa930003970 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.138+0000 7fa94559a640 1 -- 172.21.15.78:0/1350627087 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938034440 con 0x7fa930003970 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 -- 172.21.15.78:0/1350627087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa930003970 msgr2=0x7fa930003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa930003970 0x7fa930003d90 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fa938009f90 tx=0x7fa9300057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 -- 172.21.15.78:0/1350627087 shutdown_connections 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/1350627087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa930003970 0x7fa930003d90 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 -- 172.21.15.78:0/1350627087 >> 172.21.15.78:0/1350627087 conn(0x7fa94806b4b0 msgr2=0x7fa9480c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 -- 172.21.15.78:0/1350627087 shutdown_connections 2024-02-19T14:57:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.139+0000 7fa94cb2c640 1 -- 172.21.15.78:0/1350627087 wait complete. 2024-02-19T14:57:37.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.140+0000 7fa94cb2c640 1 Processor -- start 2024-02-19T14:57:37.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.140+0000 7fa94cb2c640 1 -- start start 2024-02-19T14:57:37.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94cb2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa930003970 0x7fa94806dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94cb2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 0x7fa9480c1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94cb2c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9481ea7c0 0x7fa9481eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94cb2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9480aadc0 con 0x7fa9480a8380 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94cb2c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9480ab2d0 con 0x7fa9481ea7c0 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94cb2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9481eb180 con 0x7fa930003970 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 0x7fa9480c1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 0x7fa9480c1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50946/0 (socket says 172.21.15.78:50946) 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 -- 172.21.15.78:0/2037397028 learned_addr learned my addr 172.21.15.78:0/2037397028 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa946d9d640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9481ea7c0 0x7fa9481eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa94659c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa930003970 0x7fa94806dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 -- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa930003970 msgr2=0x7fa94806dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa930003970 0x7fa94806dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 -- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9481ea7c0 msgr2=0x7fa9481eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9481ea7c0 0x7fa9481eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 -- 172.21.15.78:0/2037397028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9480c5ac0 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.141+0000 7fa945d9b640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 0x7fa9480c1d70 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000f60 tx=0x7fa93c007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.142+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c001490 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.142+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c00f9c0 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.142+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9480aaff0 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.142+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c00fd50 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.142+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94806d7b0 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.142+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa948074990 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.143+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa93c01d460 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.143+0000 7fa936ffd640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa92806b840 0x7fa92806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.144+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fa93c08a030 con 0x7fa9480a8380 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.144+0000 7fa94659c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa92806b840 0x7fa92806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.144+0000 7fa94659c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa92806b840 0x7fa92806dd00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa938036000 tx=0x7fa93802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.148+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa93c054c50 con 0x7fa9480a8380 2024-02-19T14:57:37.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.319+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fa9480c76f0 con 0x7fa9480a8380 2024-02-19T14:57:37.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.319+0000 7fa936ffd640 1 -- 172.21.15.78:0/2037397028 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v49) v1 ==== 96+0+0 (secure 0 0 0) 0x7fa93c058310 con 0x7fa9480a8380 2024-02-19T14:57:37.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa92806b840 msgr2=0x7fa92806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa92806b840 0x7fa92806dd00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa938036000 tx=0x7fa93802e000 comp rx=0 tx=0).stop 2024-02-19T14:57:37.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 msgr2=0x7fa9480c1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 0x7fa9480c1d70 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000f60 tx=0x7fa93c007660 comp rx=0 tx=0).stop 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 shutdown_connections 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa92806b840 0x7fa92806dd00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9480a8380 0x7fa9480c1d70 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9481ea7c0 0x7fa9481eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 --2- 172.21.15.78:0/2037397028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa930003970 0x7fa94806dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 >> 172.21.15.78:0/2037397028 conn(0x7fa94806b4b0 msgr2=0x7fa9480c3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 shutdown_connections 2024-02-19T14:57:37.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.321+0000 7fa94cb2c640 1 -- 172.21.15.78:0/2037397028 wait complete. 2024-02-19T14:57:37.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:57:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.434+0000 7f20bd888640 1 Processor -- start 2024-02-19T14:57:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.434+0000 7f20bd888640 1 -- start start 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20bd888640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b8152700 0x7f20b80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20bd888640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20b80c8e00 0x7f20b815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20bd888640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b81551f0 con 0x7f20b80c5730 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20bd888640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b8155490 con 0x7f20b80c8e00 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20bd888640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b8155770 con 0x7f20b8152700 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b8152700 0x7f20b80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20b80c8e00 0x7f20b815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20affff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f20b80c5730 0x7f20b806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:40514/0 (socket says 172.21.15.78:40514) 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20b80c8e00 0x7f20b815f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:38618/0 (socket says 172.21.15.78:38618) 2024-02-19T14:57:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b8152700 0x7f20b80c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:54118/0 (socket says 172.21.15.78:54118) 2024-02-19T14:57:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 -- 172.21.15.78:0/491144170 learned_addr learned my addr 172.21.15.78:0/491144170 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 -- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b8152700 msgr2=0x7f20b80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 --2- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b8152700 0x7f20b80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 -- 172.21.15.78:0/491144170 >> v1:172.21.15.150:6789/0 conn(0x7f20b80c5730 legacy=0x7f20b806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b77fe640 1 -- 172.21.15.78:0/491144170 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b8155a50 con 0x7f20b80c8e00 2024-02-19T14:57:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.435+0000 7f20b67fc640 1 -- 172.21.15.78:0/491144170 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283168008 0 0) 0x7f20b81551f0 con 0x7f20b80c5730 2024-02-19T14:57:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.436+0000 7f20b77fe640 1 --2- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b80c8e00 0x7f20b815f980 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009e50 tx=0x7f20b002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f488ae0ca0968214 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.436+0000 7f20b67fc640 1 -- 172.21.15.78:0/491144170 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b002cb00 con 0x7f20b80c8e00 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.436+0000 7f20b67fc640 1 -- 172.21.15.78:0/491144170 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b002cc60 con 0x7f20b80c8e00 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.436+0000 7f20b67fc640 1 -- 172.21.15.78:0/491144170 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b00343f0 con 0x7f20b80c8e00 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 -- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b80c8e00 msgr2=0x7f20b815f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 --2- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b80c8e00 0x7f20b815f980 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009e50 tx=0x7f20b002bf60 comp rx=0 tx=0).stop 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 -- 172.21.15.78:0/491144170 shutdown_connections 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 --2- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b80c8e00 0x7f20b815f980 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 --2- 172.21.15.78:0/491144170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b8152700 0x7f20b80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 -- 172.21.15.78:0/491144170 >> 172.21.15.78:0/491144170 conn(0x7f20b80cac90 msgr2=0x7f20b80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 -- 172.21.15.78:0/491144170 shutdown_connections 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.437+0000 7f20bd888640 1 -- 172.21.15.78:0/491144170 wait complete. 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.438+0000 7f20bd888640 1 Processor -- start 2024-02-19T14:57:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.438+0000 7f20bd888640 1 -- start start 2024-02-19T14:57:37.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20bd888640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b80c5730 0x7f20b80aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20bd888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 0x7f20b81e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20bd888640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b8152700 0x7f20b81e7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20bd888640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b80c1330 con 0x7f20b80c8e00 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20bd888640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b81e8170 con 0x7f20b8152700 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20bd888640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b80ab280 con 0x7f20b80c5730 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 0x7f20b81e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20b6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b80c5730 0x7f20b80aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 0x7f20b81e76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50960/0 (socket says 172.21.15.78:50960) 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 -- 172.21.15.78:0/2198359877 learned_addr learned my addr 172.21.15.78:0/2198359877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20b77fe640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b8152700 0x7f20b81e7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 -- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b80c5730 msgr2=0x7f20b80aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b80c5730 0x7f20b80aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 -- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b8152700 msgr2=0x7f20b81e7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b8152700 0x7f20b81e7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 -- 172.21.15.78:0/2198359877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b0009b30 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20b6ffd640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b80c5730 0x7f20b80aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20b77fe640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b8152700 0x7f20b81e7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20affff640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 0x7f20b81e76f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c4d0 tx=0x7f20a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.439+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a00074e0 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.440+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0007660 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.440+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b80ab560 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.440+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0004a20 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.440+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b80abb20 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.441+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b8074990 con 0x7f20b80c8e00 2024-02-19T14:57:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.441+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f20a0004b80 con 0x7f20b80c8e00 2024-02-19T14:57:37.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.441+0000 7f20b4ff9640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f209806b810 0x7f209806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.442+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f20a0089550 con 0x7f20b80c8e00 2024-02-19T14:57:37.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.442+0000 7f20b6ffd640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f209806b810 0x7f209806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.442+0000 7f20b6ffd640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f209806b810 0x7f209806dcd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009e90 tx=0x7f20a8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.446+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f20a00540e0 con 0x7f20b80c8e00 2024-02-19T14:57:37.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.618+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f20b80c8770 con 0x7f20b80c8e00 2024-02-19T14:57:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.621+0000 7f20b4ff9640 1 -- 172.21.15.78:0/2198359877 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f20a00577a0 con 0x7f20b80c8e00 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f209806b810 msgr2=0x7f209806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f209806b810 0x7f209806dcd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009e90 tx=0x7f20a8009190 comp rx=0 tx=0).stop 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 msgr2=0x7f20b81e76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 0x7f20b81e76f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c4d0 tx=0x7f20a000c990 comp rx=0 tx=0).stop 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 shutdown_connections 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f209806b810 0x7f209806dcd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20b80c8e00 0x7f20b81e76f0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b8152700 0x7f20b81e7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 --2- 172.21.15.78:0/2198359877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20b80c5730 0x7f20b80aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 >> 172.21.15.78:0/2198359877 conn(0x7f20b80cac90 msgr2=0x7f20b814e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 shutdown_connections 2024-02-19T14:57:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.623+0000 7f20bd888640 1 -- 172.21.15.78:0/2198359877 wait complete. 2024-02-19T14:57:37.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-02-19T14:57:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.735+0000 7fe07e927640 1 Processor -- start 2024-02-19T14:57:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.735+0000 7fe07e927640 1 -- start start 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07e927640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe078155780 0x7fe078160920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07e927640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe078072e80 con 0x7fe0781528d0 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07e927640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07806e990 con 0x7fe078072180 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07e927640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0780734c0 con 0x7fe078155780 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07cec4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe078155780 0x7fe078160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07cec4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe078155780 0x7fe078160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52958/0 (socket says 172.21.15.78:52958) 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07cec4640 1 -- 172.21.15.78:0/1326667920 learned_addr learned my addr 172.21.15.78:0/1326667920 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe077fff640 1 --1- 172.21.15.78:0/1326667920 >> v1:172.21.15.78:6789/0 conn(0x7fe0781528d0 0x7fe0780c1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40828/0 (socket says 172.21.15.78:40828) 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07cec4640 1 -- 172.21.15.78:0/1326667920 >> v1:172.21.15.150:6789/0 conn(0x7fe078072180 legacy=0x7fe078155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07cec4640 1 -- 172.21.15.78:0/1326667920 >> v1:172.21.15.78:6789/0 conn(0x7fe0781528d0 legacy=0x7fe0780c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe07cec4640 1 -- 172.21.15.78:0/1326667920 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0780731e0 con 0x7fe078155780 2024-02-19T14:57:37.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.736+0000 7fe076ffd640 1 -- 172.21.15.78:0/1326667920 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141636300 0 0) 0x7fe068001110 con 0x7fe0781528d0 2024-02-19T14:57:37.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.737+0000 7fe07cec4640 1 --2- 172.21.15.78:0/1326667920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078155780 0x7fe078160920 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe070009e50 tx=0x7fe07002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c079447fb11e6146 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.737+0000 7fe076ffd640 1 -- 172.21.15.78:0/1326667920 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07002cb20 con 0x7fe078155780 2024-02-19T14:57:37.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.737+0000 7fe076ffd640 1 -- 172.21.15.78:0/1326667920 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07002cc80 con 0x7fe078155780 2024-02-19T14:57:37.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.737+0000 7fe076ffd640 1 -- 172.21.15.78:0/1326667920 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0700343f0 con 0x7fe078155780 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 -- 172.21.15.78:0/1326667920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078155780 msgr2=0x7fe078160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 --2- 172.21.15.78:0/1326667920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078155780 0x7fe078160920 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe070009e50 tx=0x7fe07002bf80 comp rx=0 tx=0).stop 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 -- 172.21.15.78:0/1326667920 shutdown_connections 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 --2- 172.21.15.78:0/1326667920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078155780 0x7fe078160920 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 -- 172.21.15.78:0/1326667920 >> 172.21.15.78:0/1326667920 conn(0x7fe0780c7590 msgr2=0x7fe0780c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 -- 172.21.15.78:0/1326667920 shutdown_connections 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.738+0000 7fe07e927640 1 -- 172.21.15.78:0/1326667920 wait complete. 2024-02-19T14:57:37.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.739+0000 7fe07e927640 1 Processor -- start 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.739+0000 7fe07e927640 1 -- start start 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 0x7fe0780aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0781528d0 0x7fe078158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe078155780 0x7fe0781eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781ee680 con 0x7fe078072180 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780a8590 con 0x7fe0781528d0 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781eebb0 con 0x7fe078155780 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 0x7fe0780aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 0x7fe0780aa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52966/0 (socket says 172.21.15.78:52966) 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 -- 172.21.15.78:0/3492246344 learned_addr learned my addr 172.21.15.78:0/3492246344 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07cec4640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe078155780 0x7fe0781eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe0777fe640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0781528d0 0x7fe078158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 -- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe078155780 msgr2=0x7fe0781eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe078155780 0x7fe0781eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 -- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0781528d0 msgr2=0x7fe078158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0781528d0 0x7fe078158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 -- 172.21.15.78:0/3492246344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070009b30 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe077fff640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 0x7fe0780aa420 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe06800c7b0 tx=0x7fe06800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0680072d0 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe068007450 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068001d10 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0781ee8d0 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.740+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0781eab00 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.741+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0781545e0 con 0x7fe078072180 2024-02-19T14:57:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.742+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe0680047e0 con 0x7fe078072180 2024-02-19T14:57:37.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.742+0000 7fe074ff9640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe06406b810 0x7fe06406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:37.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.742+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe068089400 con 0x7fe078072180 2024-02-19T14:57:37.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.742+0000 7fe0777fe640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe06406b810 0x7fe06406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:37.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.742+0000 7fe0777fe640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe06406b810 0x7fe06406dcd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe0781e84c0 tx=0x7fe06c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.746+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe068053f90 con 0x7fe078072180 2024-02-19T14:57:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:37.985+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7fe078072e80 con 0x7fe078072180 2024-02-19T14:57:38.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.712+0000 7fe074ff9640 1 -- 172.21.15.78:0/3492246344 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v336) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe078072e80 con 0x7fe078072180 2024-02-19T14:57:38.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe06406b810 msgr2=0x7fe06406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:38.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe06406b810 0x7fe06406dcd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe0781e84c0 tx=0x7fe06c009040 comp rx=0 tx=0).stop 2024-02-19T14:57:38.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 msgr2=0x7fe0780aa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:38.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 0x7fe0780aa420 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe06800c7b0 tx=0x7fe06800cc70 comp rx=0 tx=0).stop 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 shutdown_connections 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe06406b810 0x7fe06406dcd0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe078155780 0x7fe0781eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0781528d0 0x7fe078158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 --2- 172.21.15.78:0/3492246344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe078072180 0x7fe0780aa420 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.714+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 >> 172.21.15.78:0/3492246344 conn(0x7fe0780c7590 msgr2=0x7fe0780c3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.715+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 shutdown_connections 2024-02-19T14:57:38.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.715+0000 7fe07e927640 1 -- 172.21.15.78:0/3492246344 wait complete. 2024-02-19T14:57:38.729 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will cause failover if another high-affinity standby exists. ... ok 2024-02-19T14:57:38.730 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-02-19T14:57:38.730 INFO:tasks.cephfs_test_runner:test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-19T14:57:38.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.831+0000 7fe17e53e640 1 Processor -- start 2024-02-19T14:57:38.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.831+0000 7fe17e53e640 1 -- start start 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17e53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17e53e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe1780c55f0 0x7fe17815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17e53e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe178158c40 con 0x7fe1780ac390 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17e53e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1780c5a60 con 0x7fe1780c55f0 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17e53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780c8de0 con 0x7fe1780bdf20 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe1780c55f0 0x7fe17815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe1780c55f0 0x7fe17815f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52976/0 (socket says 172.21.15.78:52976) 2024-02-19T14:57:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 -- 172.21.15.78:0/872781371 learned_addr learned my addr 172.21.15.78:0/872781371 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 -- 172.21.15.78:0/872781371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 msgr2=0x7fe1780ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 --2- 172.21.15.78:0/872781371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 -- 172.21.15.78:0/872781371 >> v1:172.21.15.150:6789/0 conn(0x7fe1780ac390 legacy=0x7fe178158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.832+0000 7fe17cadb640 1 -- 172.21.15.78:0/872781371 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178155a60 con 0x7fe1780c55f0 2024-02-19T14:57:38.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.833+0000 7fe17cadb640 1 --2- 172.21.15.78:0/872781371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe17815f8c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe170009e50 tx=0x7fe17002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f185afbab9950549 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:38.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.833+0000 7fe176ffd640 1 -- 172.21.15.78:0/872781371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17002c950 con 0x7fe1780c55f0 2024-02-19T14:57:38.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.833+0000 7fe176ffd640 1 -- 172.21.15.78:0/872781371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17002cab0 con 0x7fe1780c55f0 2024-02-19T14:57:38.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.833+0000 7fe176ffd640 1 -- 172.21.15.78:0/872781371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17002ce40 con 0x7fe1780c55f0 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 -- 172.21.15.78:0/872781371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 msgr2=0x7fe17815f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 --2- 172.21.15.78:0/872781371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe17815f8c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe170009e50 tx=0x7fe17002bdb0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 -- 172.21.15.78:0/872781371 shutdown_connections 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 --2- 172.21.15.78:0/872781371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe17815f8c0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 --2- 172.21.15.78:0/872781371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 -- 172.21.15.78:0/872781371 >> 172.21.15.78:0/872781371 conn(0x7fe1780ca890 msgr2=0x7fe1780caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 -- 172.21.15.78:0/872781371 shutdown_connections 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.834+0000 7fe17e53e640 1 -- 172.21.15.78:0/872781371 wait complete. 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.835+0000 7fe17e53e640 1 Processor -- start 2024-02-19T14:57:38.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.835+0000 7fe17e53e640 1 -- start start 2024-02-19T14:57:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1780ac390 0x7fe1781e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe178151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780c0b10 con 0x7fe1780c55f0 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780be540 con 0x7fe1780ac390 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1781e23d0 con 0x7fe1780bdf20 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe178151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe178151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52984/0 (socket says 172.21.15.78:52984) 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 -- 172.21.15.78:0/3641857660 learned_addr learned my addr 172.21.15.78:0/3641857660 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe177fff640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1780ac390 0x7fe1781e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe1777fe640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 -- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 msgr2=0x7fe1780c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 -- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1780ac390 msgr2=0x7fe1781e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1780ac390 0x7fe1781e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 -- 172.21.15.78:0/3641857660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe170009b30 con 0x7fe1780c55f0 2024-02-19T14:57:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe1777fe640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17cadb640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe178151bd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe170036040 tx=0x7fe170004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17003e450 con 0x7fe1780c55f0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17003e5d0 con 0x7fe1780c55f0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1780c1970 con 0x7fe1780c55f0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17003e980 con 0x7fe1780c55f0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.836+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1781eb7f0 con 0x7fe1780c55f0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.837+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1781eb640 con 0x7fe1780c55f0 2024-02-19T14:57:38.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.838+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe17003eae0 con 0x7fe1780c55f0 2024-02-19T14:57:38.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.838+0000 7fe174ff9640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16406b810 0x7fe16406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:38.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.839+0000 7fe177fff640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16406b810 0x7fe16406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:38.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.839+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe1700acf00 con 0x7fe1780c55f0 2024-02-19T14:57:38.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.839+0000 7fe177fff640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16406b810 0x7fe16406dcd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe178154b10 tx=0x7fe168009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:38.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:38.843+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe170077a70 con 0x7fe1780c55f0 2024-02-19T14:57:39.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.013+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe178074990 con 0x7fe1780c55f0 2024-02-19T14:57:39.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.014+0000 7fe174ff9640 1 -- 172.21.15.78:0/3641857660 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+13555 (secure 0 0 0) 0x7fe17007b130 con 0x7fe1780c55f0 2024-02-19T14:57:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:39.017 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":143,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:36.695669+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/1973988605":"2024-02-20T14:57:33.670487+0000","172.21.15.78:6834/1973988605":"2024-02-20T14:57:33.670487+0000","172.21.15.150:6835/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6833/1869543632":"2024-02-20T14:57:24.626396+0000","172.21.15.150:6834/2425259283":"2024-02-20T14:57:10.590067+0000","172.21.15.150:6832/2425259283":"2024-02-20T14:57:10.590067+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16406b810 msgr2=0x7fe16406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16406b810 0x7fe16406dcd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe178154b10 tx=0x7fe168009040 comp rx=0 tx=0).stop 2024-02-19T14:57:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 msgr2=0x7fe178151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe178151bd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe170036040 tx=0x7fe170004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 shutdown_connections 2024-02-19T14:57:39.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe16406b810 0x7fe16406dcd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1780c55f0 0x7fe178151bd0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1780bdf20 0x7fe1780c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 --2- 172.21.15.78:0/3641857660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1780ac390 0x7fe1781e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 >> 172.21.15.78:0/3641857660 conn(0x7fe1780ca890 msgr2=0x7fe1780cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 shutdown_connections 2024-02-19T14:57:39.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.016+0000 7fe17e53e640 1 -- 172.21.15.78:0/3641857660 wait complete. 2024-02-19T14:57:39.030 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:57:39.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:57:39.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.129+0000 7f4b2f4c7640 1 Processor -- start 2024-02-19T14:57:39.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.129+0000 7f4b2f4c7640 1 -- start start 2024-02-19T14:57:39.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2f4c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2814ece0 0x7f4b280c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2f4c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b280c5270 0x7f4b280719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2f4c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b280c1120 con 0x7f4b28071f10 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2f4c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b280c6080 con 0x7f4b280c5270 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2f4c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280c57e0 con 0x7f4b2814ece0 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2ca62640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b280c5270 0x7f4b280719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2ca62640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b280c5270 0x7f4b280719d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52994/0 (socket says 172.21.15.78:52994) 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2ca62640 1 -- 172.21.15.78:0/2000147340 learned_addr learned my addr 172.21.15.78:0/2000147340 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.130+0000 7f4b2d263640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2814ece0 0x7f4b280c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b2ca62640 1 -- 172.21.15.78:0/2000147340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2814ece0 msgr2=0x7f4b280c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b2ca62640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2814ece0 0x7f4b280c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b2ca62640 1 -- 172.21.15.78:0/2000147340 >> v1:172.21.15.150:6789/0 conn(0x7f4b28071f10 legacy=0x7f4b280c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b2ca62640 1 -- 172.21.15.78:0/2000147340 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280c5ac0 con 0x7f4b280c5270 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b2d263640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2814ece0 0x7f4b280c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:39.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b2ca62640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c5270 0x7f4b280719d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800bb00 tx=0x7f4b1802df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f7033fa213339e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b1ffff640 1 -- 172.21.15.78:0/2000147340 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1802ece0 con 0x7f4b280c5270 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b1ffff640 1 -- 172.21.15.78:0/2000147340 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18032070 con 0x7f4b280c5270 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.131+0000 7f4b1ffff640 1 -- 172.21.15.78:0/2000147340 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18038620 con 0x7f4b280c5270 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/2000147340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c5270 msgr2=0x7f4b280719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c5270 0x7f4b280719d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800bb00 tx=0x7f4b1802df30 comp rx=0 tx=0).stop 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/2000147340 shutdown_connections 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c5270 0x7f4b280719d0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/2000147340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b2814ece0 0x7f4b280c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/2000147340 >> 172.21.15.78:0/2000147340 conn(0x7f4b2806b270 msgr2=0x7f4b2806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/2000147340 shutdown_connections 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.132+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/2000147340 wait complete. 2024-02-19T14:57:39.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.133+0000 7f4b2f4c7640 1 Processor -- start 2024-02-19T14:57:39.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.133+0000 7f4b2f4c7640 1 -- start start 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2f4c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b28071f10 0x7f4b280bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2f4c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b280c5270 0x7f4b2814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2f4c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 0x7f4b2814b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2f4c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b28150cb0 con 0x7f4b2814ece0 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2f4c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b281ee930 con 0x7f4b280c5270 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2f4c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b28150f70 con 0x7f4b28071f10 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 0x7f4b2814b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 0x7f4b2814b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53006/0 (socket says 172.21.15.78:53006) 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 -- 172.21.15.78:0/3698365100 learned_addr learned my addr 172.21.15.78:0/3698365100 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:39.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2d263640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b28071f10 0x7f4b280bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2ca62640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b280c5270 0x7f4b2814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 -- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b28071f10 msgr2=0x7f4b280bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b28071f10 0x7f4b280bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 -- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b280c5270 msgr2=0x7f4b2814b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b280c5270 0x7f4b2814b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 -- 172.21.15.78:0/3698365100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1800b760 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b2da64640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 0x7f4b2814b850 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f4b20001eb0 tx=0x7f4b2000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.134+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b200052d0 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.135+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b20005430 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.135+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b28151250 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.135+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b20003810 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.135+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b280c0160 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.136+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b28074990 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.136+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b20003970 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.136+0000 7f4b1dffb640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b0c06b810 0x7f4b0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.136+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f4b200895d0 con 0x7f4b2814ece0 2024-02-19T14:57:39.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.136+0000 7f4b2d263640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b0c06b810 0x7f4b0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.137+0000 7f4b2d263640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b0c06b810 0x7f4b0c06dcd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009fd0 tx=0x7f4b24009870 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.141+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b200541f0 con 0x7f4b2814ece0 2024-02-19T14:57:39.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.313+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b2806e930 con 0x7f4b2814ece0 2024-02-19T14:57:39.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.313+0000 7f4b1dffb640 1 -- 172.21.15.78:0/3698365100 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b200578b0 con 0x7f4b2814ece0 2024-02-19T14:57:39.316 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:57:39.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b0c06b810 msgr2=0x7f4b0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b0c06b810 0x7f4b0c06dcd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009fd0 tx=0x7f4b24009870 comp rx=0 tx=0).stop 2024-02-19T14:57:39.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 msgr2=0x7f4b2814b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 0x7f4b2814b850 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f4b20001eb0 tx=0x7f4b2000c890 comp rx=0 tx=0).stop 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 shutdown_connections 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b0c06b810 0x7f4b0c06dcd0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b2814ece0 0x7f4b2814b850 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b280c5270 0x7f4b2814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.315+0000 7f4b2f4c7640 1 --2- 172.21.15.78:0/3698365100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b28071f10 0x7f4b280bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.316+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 >> 172.21.15.78:0/3698365100 conn(0x7f4b2806b270 msgr2=0x7f4b2814d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.316+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 shutdown_connections 2024-02-19T14:57:39.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.316+0000 7f4b2f4c7640 1 -- 172.21.15.78:0/3698365100 wait complete. 2024-02-19T14:57:39.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-02-19T14:57:39.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.426+0000 7f0473a15640 1 Processor -- start 2024-02-19T14:57:39.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.426+0000 7f0473a15640 1 -- start start 2024-02-19T14:57:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0473a15640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f046c0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0473a15640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c158c40 con 0x7f046c0bdf20 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0473a15640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c0c5a60 con 0x7f046c0c55f0 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0473a15640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0c8de0 con 0x7f046c0ac390 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0470fb0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f046c0bdf20 0x7f046c158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40840/0 (socket says 172.21.15.78:40840) 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0470fb0640 1 -- 172.21.15.78:0/253019553 learned_addr learned my addr 172.21.15.78:0/253019553 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074448038 0 0) 0x7f046c158c40 con 0x7f046c0bdf20 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0458003520 con 0x7f046c0bdf20 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.427+0000 7f04717b1640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f046c0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2858579455 0 0) 0x7f0458003520 con 0x7f046c0bdf20 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 msgr2=0x7f046c0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f046c0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 >> v1:172.21.15.150:6789/0 conn(0x7f046c0c55f0 legacy=0x7f046c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c155a60 con 0x7f046c0bdf20 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f045c001670 con 0x7f046c0bdf20 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f04717b1640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f046c0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:39.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 >> v1:172.21.15.78:6789/0 conn(0x7f046c0bdf20 legacy=0x7f046c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f04580039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f04580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0458004d00 0x7f0458005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0c8de0 con 0x7f0458003f90 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045800bc10 con 0x7f0458004d00 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045800bdd0 con 0x7f046c0ac390 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0470fb0640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f04580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f04717b1640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f04580039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0471fb2640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0458004d00 0x7f0458005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0470fb0640 1 -- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 msgr2=0x7f04580039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0470fb0640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0ac390 0x7f04580039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0470fb0640 1 -- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0458004d00 msgr2=0x7f0458005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0470fb0640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0458004d00 0x7f0458005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.428+0000 7f0470fb0640 1 -- 172.21.15.78:0/253019553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0458003520 con 0x7f0458003f90 2024-02-19T14:57:39.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0470fb0640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f04580047c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f046c0be300 tx=0x7f04580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951ab32b43697190 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c005ad0 con 0x7f0458003f90 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c005c50 con 0x7f0458003f90 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0463fff640 1 -- 172.21.15.78:0/253019553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c02c890 con 0x7f0458003f90 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0473a15640 1 -- 172.21.15.78:0/253019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 msgr2=0x7f04580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0473a15640 1 --2- 172.21.15.78:0/253019553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f04580047c0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f046c0be300 tx=0x7f04580057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.429+0000 7f0471fb2640 1 -- 172.21.15.78:0/253019553 reap_dead start 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.430+0000 7f0473a15640 1 -- 172.21.15.78:0/253019553 shutdown_connections 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.430+0000 7f0473a15640 1 -- 172.21.15.78:0/253019553 >> 172.21.15.78:0/253019553 conn(0x7f046c0ca890 msgr2=0x7f046c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.430+0000 7f0473a15640 1 -- 172.21.15.78:0/253019553 shutdown_connections 2024-02-19T14:57:39.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.430+0000 7f0473a15640 1 -- 172.21.15.78:0/253019553 wait complete. 2024-02-19T14:57:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 Processor -- start 2024-02-19T14:57:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 -- start start 2024-02-19T14:57:39.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f046c0c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1ee210 0x7f046c157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0bcbd0 0x7f046c0bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0bd590 con 0x7f0458003f90 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0c3760 con 0x7f046c1ee210 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f0473a15640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0af2a0 con 0x7f046c0bcbd0 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f04717b1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f046c0c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f04717b1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f046c0c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53030/0 (socket says 172.21.15.78:53030) 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.431+0000 7f04717b1640 1 -- 172.21.15.78:0/2336852354 learned_addr learned my addr 172.21.15.78:0/2336852354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f04717b1640 1 -- 172.21.15.78:0/2336852354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0bcbd0 msgr2=0x7f046c0bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f04717b1640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0bcbd0 0x7f046c0bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f04717b1640 1 -- 172.21.15.78:0/2336852354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1ee210 msgr2=0x7f046c157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f04717b1640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1ee210 0x7f046c157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f04717b1640 1 -- 172.21.15.78:0/2336852354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c0c41e0 con 0x7f0458003f90 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f04717b1640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f046c0c11b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f0468011600 tx=0x7f0468011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468004440 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0468003610 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c1ee690 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04680039c0 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.432+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c0c33f0 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.433+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c074990 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.434+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0468003b20 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.434+0000 7f0461ffb640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f045006b840 0x7f045006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.434+0000 7f0470fb0640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f045006b840 0x7f045006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.434+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f046808a690 con 0x7f0458003f90 2024-02-19T14:57:39.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.435+0000 7f0470fb0640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f045006b840 0x7f045006dd00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f045c034000 tx=0x7f045c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.439+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0468055170 con 0x7f0458003f90 2024-02-19T14:57:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.675+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7f046c06da00 con 0x7f0458003f90 2024-02-19T14:57:39.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.712+0000 7f0461ffb640 1 -- 172.21.15.78:0/2336852354 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v337) v1 ==== 149+0+0 (secure 0 0 0) 0x7f0468058830 con 0x7f0458003f90 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f045006b840 msgr2=0x7f045006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f045006b840 0x7f045006dd00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f045c034000 tx=0x7f045c007000 comp rx=0 tx=0).stop 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 msgr2=0x7f046c0c11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f046c0c11b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f0468011600 tx=0x7f0468011ac0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 shutdown_connections 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f045006b840 0x7f045006dd00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f046c0bcbd0 0x7f046c0bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1ee210 0x7f046c157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 --2- 172.21.15.78:0/2336852354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0458003f90 0x7f046c0c11b0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 >> 172.21.15.78:0/2336852354 conn(0x7f046c0ca890 msgr2=0x7f046c06c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 shutdown_connections 2024-02-19T14:57:39.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.714+0000 7f0473a15640 1 -- 172.21.15.78:0/2336852354 wait complete. 2024-02-19T14:57:39.728 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:57:39.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.827+0000 7f8ae349a640 1 Processor -- start 2024-02-19T14:57:39.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 -- start start 2024-02-19T14:57:39.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8adc149790 0x7f8adc06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8adc0723d0 0x7f8adc072800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc152c30 0x7f8adc1530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc149b70 con 0x7f8adc0723d0 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc06f420 con 0x7f8adc149790 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae349a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0ab760 con 0x7f8adc152c30 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae0a35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8adc0723d0 0x7f8adc072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae0a35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8adc0723d0 0x7f8adc072800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53044/0 (socket says 172.21.15.78:53044) 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.828+0000 7f8ae0a35640 1 -- 172.21.15.78:0/1141988163 learned_addr learned my addr 172.21.15.78:0/1141988163 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae1236640 1 --2- 172.21.15.78:0/1141988163 >> v2:172.21.15.150:3300/0 conn(0x7f8adc149790 0x7f8adc06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae0a35640 1 -- 172.21.15.78:0/1141988163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc152c30 msgr2=0x7f8adc1530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae0a35640 1 --2- 172.21.15.78:0/1141988163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc152c30 0x7f8adc1530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae0a35640 1 -- 172.21.15.78:0/1141988163 >> v2:172.21.15.150:3300/0 conn(0x7f8adc149790 msgr2=0x7f8adc06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae0a35640 1 --2- 172.21.15.78:0/1141988163 >> v2:172.21.15.150:3300/0 conn(0x7f8adc149790 0x7f8adc06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae0a35640 1 -- 172.21.15.78:0/1141988163 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc0aba40 con 0x7f8adc0723d0 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.829+0000 7f8ae0a35640 1 --2- 172.21.15.78:0/1141988163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc0723d0 0x7f8adc072800 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f8adc06ccd0 tx=0x7f8acc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5619d9ce1e1ada0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ad3fff640 1 -- 172.21.15.78:0/1141988163 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc031750 con 0x7f8adc0723d0 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ad3fff640 1 -- 172.21.15.78:0/1141988163 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc0318b0 con 0x7f8adc0723d0 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ad3fff640 1 -- 172.21.15.78:0/1141988163 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc031c40 con 0x7f8adc0723d0 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 -- 172.21.15.78:0/1141988163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc0723d0 msgr2=0x7f8adc072800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 --2- 172.21.15.78:0/1141988163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc0723d0 0x7f8adc072800 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f8adc06ccd0 tx=0x7f8acc02b7f0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 -- 172.21.15.78:0/1141988163 shutdown_connections 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 --2- 172.21.15.78:0/1141988163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc152c30 0x7f8adc1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 --2- 172.21.15.78:0/1141988163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc0723d0 0x7f8adc072800 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 --2- 172.21.15.78:0/1141988163 >> v2:172.21.15.150:3300/0 conn(0x7f8adc149790 0x7f8adc06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 -- 172.21.15.78:0/1141988163 >> 172.21.15.78:0/1141988163 conn(0x7f8adc0718e0 msgr2=0x7f8adc071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 -- 172.21.15.78:0/1141988163 shutdown_connections 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.830+0000 7f8ae349a640 1 -- 172.21.15.78:0/1141988163 wait complete. 2024-02-19T14:57:39.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.831+0000 7f8ae349a640 1 Processor -- start 2024-02-19T14:57:39.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.831+0000 7f8ae349a640 1 -- start start 2024-02-19T14:57:39.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae349a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc0723d0 0x7f8adc0c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae349a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8adc149790 0x7f8adc0c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae349a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 0x7f8adc143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae349a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc1dccc0 con 0x7f8adc152c30 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae349a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc1ddc90 con 0x7f8adc149790 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae349a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc1ee3e0 con 0x7f8adc0723d0 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 0x7f8adc143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 0x7f8adc143070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53050/0 (socket says 172.21.15.78:53050) 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 -- 172.21.15.78:0/2198292979 learned_addr learned my addr 172.21.15.78:0/2198292979 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1236640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc0723d0 0x7f8adc0c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae0a35640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8adc149790 0x7f8adc0c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 -- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc0723d0 msgr2=0x7f8adc0c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc0723d0 0x7f8adc0c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 -- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8adc149790 msgr2=0x7f8adc0c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8adc149790 0x7f8adc0c0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:39.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae1a37640 1 -- 172.21.15.78:0/2198292979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc0094d0 con 0x7f8adc152c30 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.832+0000 7f8ae0a35640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8adc149790 0x7f8adc0c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.833+0000 7f8ae1a37640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 0x7f8adc143070 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400b970 tx=0x7f8ad400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.833+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400cb40 con 0x7f8adc152c30 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.833+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad400ccc0 con 0x7f8adc152c30 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.833+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc1ee6c0 con 0x7f8adc152c30 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.833+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4014680 con 0x7f8adc152c30 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.833+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc1eec80 con 0x7f8adc152c30 2024-02-19T14:57:39.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.834+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc074990 con 0x7f8adc152c30 2024-02-19T14:57:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.834+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8ad401d3b0 con 0x7f8adc152c30 2024-02-19T14:57:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.835+0000 7f8ad1ffb640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ac006b810 0x7f8ac006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.835+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f8ad408a060 con 0x7f8adc152c30 2024-02-19T14:57:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.835+0000 7f8ae1236640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ac006b810 0x7f8ac006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:39.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.835+0000 7f8ae1236640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ac006b810 0x7f8ac006dcd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8adc14ff20 tx=0x7f8ad8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:39.839+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8ad4054b40 con 0x7f8adc152c30 2024-02-19T14:57:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.010+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8adc0acff0 con 0x7f8adc152c30 2024-02-19T14:57:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.013+0000 7f8ad1ffb640 1 -- 172.21.15.78:0/2198292979 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8ad40190c0 con 0x7f8adc152c30 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ac006b810 msgr2=0x7f8ac006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ac006b810 0x7f8ac006dcd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8adc14ff20 tx=0x7f8ad8008040 comp rx=0 tx=0).stop 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 msgr2=0x7f8adc143070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 0x7f8adc143070 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400b970 tx=0x7f8ad400be30 comp rx=0 tx=0).stop 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 shutdown_connections 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ac006b810 0x7f8ac006dcd0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8adc152c30 0x7f8adc143070 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8adc149790 0x7f8adc0c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 --2- 172.21.15.78:0/2198292979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8adc0723d0 0x7f8adc0c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 >> 172.21.15.78:0/2198292979 conn(0x7f8adc0718e0 msgr2=0x7f8adc071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:40.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 shutdown_connections 2024-02-19T14:57:40.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.015+0000 7f8ae349a640 1 -- 172.21.15.78:0/2198292979 wait complete. 2024-02-19T14:57:40.029 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:57:40.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:57:40.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:57:40.127 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:57:40.127 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:57:40.127 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:57:40.142 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:57:40.142 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:57:40.143 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:57:40.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:57:40.160 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:57:40.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.279+0000 7f5b8d288640 1 Processor -- start 2024-02-19T14:57:40.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.279+0000 7f5b8d288640 1 -- start start 2024-02-19T14:57:40.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b8d288640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5b880c8e00 0x7f5b8806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b8d288640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b881551f0 con 0x7f5b8806d910 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b8d288640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b88155490 con 0x7f5b88152700 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b8d288640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b88155770 con 0x7f5b880c8e00 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b7ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5b880c8e00 0x7f5b8806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b877fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5b8806d910 0x7f5b8815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40844/0 (socket says 172.21.15.78:40844) 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b7ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5b880c8e00 0x7f5b8806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53066/0 (socket says 172.21.15.78:53066) 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b877fe640 1 -- 172.21.15.78:0/3156395176 learned_addr learned my addr 172.21.15.78:0/3156395176 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b7ffff640 1 -- 172.21.15.78:0/3156395176 >> v1:172.21.15.150:6789/0 conn(0x7f5b88152700 legacy=0x7f5b880c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b7ffff640 1 -- 172.21.15.78:0/3156395176 >> v1:172.21.15.78:6789/0 conn(0x7f5b8806d910 legacy=0x7f5b8815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b7ffff640 1 -- 172.21.15.78:0/3156395176 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88155a50 con 0x7f5b880c8e00 2024-02-19T14:57:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.280+0000 7f5b867fc640 1 -- 172.21.15.78:0/3156395176 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267534188 0 0) 0x7f5b881551f0 con 0x7f5b8806d910 2024-02-19T14:57:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.281+0000 7f5b7ffff640 1 --2- 172.21.15.78:0/3156395176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b880c8e00 0x7f5b8806c2a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f5b700097a0 tx=0x7f5b7002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50f71941d994de0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.281+0000 7f5b867fc640 1 -- 172.21.15.78:0/3156395176 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70030610 con 0x7f5b880c8e00 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.281+0000 7f5b867fc640 1 -- 172.21.15.78:0/3156395176 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b70030770 con 0x7f5b880c8e00 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.281+0000 7f5b867fc640 1 -- 172.21.15.78:0/3156395176 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70030b00 con 0x7f5b880c8e00 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 -- 172.21.15.78:0/3156395176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b880c8e00 msgr2=0x7f5b8806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 --2- 172.21.15.78:0/3156395176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b880c8e00 0x7f5b8806c2a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f5b700097a0 tx=0x7f5b7002b960 comp rx=0 tx=0).stop 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 -- 172.21.15.78:0/3156395176 shutdown_connections 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 --2- 172.21.15.78:0/3156395176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b880c8e00 0x7f5b8806c2a0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 -- 172.21.15.78:0/3156395176 >> 172.21.15.78:0/3156395176 conn(0x7f5b880cac90 msgr2=0x7f5b880cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 -- 172.21.15.78:0/3156395176 shutdown_connections 2024-02-19T14:57:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.282+0000 7f5b8d288640 1 -- 172.21.15.78:0/3156395176 wait complete. 2024-02-19T14:57:40.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 Processor -- start 2024-02-19T14:57:40.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 -- start start 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b8806d910 0x7f5b880c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880c8e00 0x7f5b880ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 0x7f5b88147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880aa280 con 0x7f5b88152700 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880acdf0 con 0x7f5b880c8e00 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b8d288640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880ad0d0 con 0x7f5b8806d910 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 0x7f5b88147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.283+0000 7f5b877fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 0x7f5b88147cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53078/0 (socket says 172.21.15.78:53078) 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 -- 172.21.15.78:0/1341468756 learned_addr learned my addr 172.21.15.78:0/1341468756 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b7ffff640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880c8e00 0x7f5b880ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b86ffd640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b8806d910 0x7f5b880c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 -- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b8806d910 msgr2=0x7f5b880c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b8806d910 0x7f5b880c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 -- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880c8e00 msgr2=0x7f5b880ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880c8e00 0x7f5b880ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 -- 172.21.15.78:0/1341468756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b70009400 con 0x7f5b88152700 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b877fe640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 0x7f5b88147cf0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f5b8000c7a0 tx=0x7f5b8000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000a640 con 0x7f5b88152700 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8000a7a0 con 0x7f5b88152700 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b880aa490 con 0x7f5b88152700 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000abd0 con 0x7f5b88152700 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.284+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b880ab560 con 0x7f5b88152700 2024-02-19T14:57:40.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.285+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b880aa9b0 con 0x7f5b88152700 2024-02-19T14:57:40.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.286+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5b8000ad30 con 0x7f5b88152700 2024-02-19T14:57:40.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.286+0000 7f5b7f7fe640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b6806b810 0x7f5b6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.286+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f5b80089900 con 0x7f5b88152700 2024-02-19T14:57:40.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.286+0000 7f5b86ffd640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b6806b810 0x7f5b6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.286+0000 7f5b86ffd640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b6806b810 0x7f5b6806dcd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000f60 tx=0x7f5b78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:40.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.290+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5b880aa9b0 con 0x7f5b88152700 2024-02-19T14:57:40.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.497+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b880aa9b0 con 0x7f5b88152700 2024-02-19T14:57:40.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.497+0000 7f5b7f7fe640 1 -- 172.21.15.78:0/1341468756 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+3707 (secure 0 0 0) 0x7f5b880aa9b0 con 0x7f5b88152700 2024-02-19T14:57:40.501 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:40.501 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":128,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7522,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3153272641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3153272641},{"type":"v1","addr":"172.21.15.150:6834","nonce":3153272641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119},{"gid":7621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1637684624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1637684624},{"type":"v1","addr":"172.21.15.150:6835","nonce":1637684624}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":124},{"gid":8262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3197225278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3197225278},{"type":"v1","addr":"172.21.15.78:6837","nonce":3197225278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113},{"gid":8676,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/417597144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":417597144},{"type":"v1","addr":"172.21.15.78:6836","nonce":417597144}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[]} 2024-02-19T14:57:40.501 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 128 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b6806b810 msgr2=0x7f5b6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b6806b810 0x7f5b6806dcd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000f60 tx=0x7f5b78009040 comp rx=0 tx=0).stop 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 msgr2=0x7f5b88147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 0x7f5b88147cf0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f5b8000c7a0 tx=0x7f5b8000cc60 comp rx=0 tx=0).stop 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 shutdown_connections 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5b6806b810 0x7f5b6806dcd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b88152700 0x7f5b88147cf0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880c8e00 0x7f5b880ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 --2- 172.21.15.78:0/1341468756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b8806d910 0x7f5b880c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 >> 172.21.15.78:0/1341468756 conn(0x7f5b880cac90 msgr2=0x7f5b8806d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:40.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 shutdown_connections 2024-02-19T14:57:40.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.499+0000 7f5b8d288640 1 -- 172.21.15.78:0/1341468756 wait complete. 2024-02-19T14:57:40.514 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:57:40.514 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:57:40.514 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:57:40.514 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:57:40.515 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:57:40.515 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:57:40.515 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:57:40.515 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:57:40.515 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:57:40.515 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:57:40.515 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:57:40.515 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:57:40.516 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:57:40.516 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:57:40.516 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:57:40.516 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:57:40.523 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:57:40.523 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:57:40.524 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:57:40.524 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:57:40.527 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:57:40.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:57:40.528 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:57:40.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:57:40.565 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:57:40.566 DEBUG:teuthology.parallel:result is None 2024-02-19T14:57:40.566 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:57:40.566 DEBUG:teuthology.parallel:result is None 2024-02-19T14:57:40.570 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:57:40.570 DEBUG:teuthology.parallel:result is None 2024-02-19T14:57:40.570 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:57:40.571 DEBUG:teuthology.parallel:result is None 2024-02-19T14:57:40.571 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:57:40.641 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:57:40.642 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:57:40.652 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:57:40.652 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:57:40.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.673+0000 7f6462b02640 1 Processor -- start 2024-02-19T14:57:40.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.673+0000 7f6462b02640 1 -- start start 2024-02-19T14:57:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f6462b02640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f6462b02640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f645c06d910 0x7f645c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f6462b02640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c06cd20 con 0x7f645c152700 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f6462b02640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645c1551f0 con 0x7f645c06d910 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f6462b02640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645c155490 con 0x7f645c0c8e00 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f645c06d910 0x7f645c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f645c06d910 0x7f645c06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53122/0 (socket says 172.21.15.78:53122) 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 -- 172.21.15.78:0/1344981025 learned_addr learned my addr 172.21.15.78:0/1344981025 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f645bfff640 1 --2- 172.21.15.78:0/1344981025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 -- 172.21.15.78:0/1344981025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 msgr2=0x7f645c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 --2- 172.21.15.78:0/1344981025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 -- 172.21.15.78:0/1344981025 >> v1:172.21.15.150:6789/0 conn(0x7f645c152700 legacy=0x7f645c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.674+0000 7f646109f640 1 -- 172.21.15.78:0/1344981025 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c155770 con 0x7f645c06d910 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.675+0000 7f646109f640 1 --2- 172.21.15.78:0/1344981025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c06c7e0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f6454009e50 tx=0x7f645402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c26a6626a84ed3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.675+0000 7f645b7fe640 1 -- 172.21.15.78:0/1344981025 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645402ca70 con 0x7f645c06d910 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.675+0000 7f645b7fe640 1 -- 172.21.15.78:0/1344981025 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645402cbd0 con 0x7f645c06d910 2024-02-19T14:57:40.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.675+0000 7f645b7fe640 1 -- 172.21.15.78:0/1344981025 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64540343f0 con 0x7f645c06d910 2024-02-19T14:57:40.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 -- 172.21.15.78:0/1344981025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 msgr2=0x7f645c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 --2- 172.21.15.78:0/1344981025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c06c7e0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f6454009e50 tx=0x7f645402bed0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 -- 172.21.15.78:0/1344981025 shutdown_connections 2024-02-19T14:57:40.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 --2- 172.21.15.78:0/1344981025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c06c7e0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 --2- 172.21.15.78:0/1344981025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 -- 172.21.15.78:0/1344981025 >> 172.21.15.78:0/1344981025 conn(0x7f645c0cac90 msgr2=0x7f645c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 -- 172.21.15.78:0/1344981025 shutdown_connections 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.676+0000 7f6462b02640 1 -- 172.21.15.78:0/1344981025 wait complete. 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 Processor -- start 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 -- start start 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c0c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c0ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f645c152700 0x7f645c1e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c0c1220 con 0x7f645c06d910 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c1eab50 con 0x7f645c152700 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.677+0000 7f6462b02640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645c1eae30 con 0x7f645c0c8e00 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c0c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c0c0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53124/0 (socket says 172.21.15.78:53124) 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 -- 172.21.15.78:0/397881095 learned_addr learned my addr 172.21.15.78:0/397881095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:40.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f645bfff640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c0ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 -- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 msgr2=0x7f645c0ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646109f640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f645c152700 0x7f645c1e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c0ac110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 -- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f645c152700 msgr2=0x7f645c1e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f645c152700 0x7f645c1e79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 -- 172.21.15.78:0/397881095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454009b30 con 0x7f645c06d910 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f646089e640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c0c0ce0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f645000c400 tx=0x7f645000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64500052d0 con 0x7f645c06d910 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6450005430 con 0x7f645c06d910 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c1eb110 con 0x7f645c06d910 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450003830 con 0x7f645c06d910 2024-02-19T14:57:40.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.678+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c1eb6d0 con 0x7f645c06d910 2024-02-19T14:57:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.679+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c074990 con 0x7f645c06d910 2024-02-19T14:57:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.680+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6450003990 con 0x7f645c06d910 2024-02-19T14:57:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.680+0000 7f64597fa640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f644006b810 0x7f644006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:40.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.680+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f6450089600 con 0x7f645c06d910 2024-02-19T14:57:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.680+0000 7f645bfff640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f644006b810 0x7f644006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.681+0000 7f645bfff640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f644006b810 0x7f644006dcd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f645c0af160 tx=0x7f6448008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:40.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.684+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6450054170 con 0x7f645c06d910 2024-02-19T14:57:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:40.871+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f645c0c8770 con 0x7f645c06d910 2024-02-19T14:57:41.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.725+0000 7f64597fa640 1 -- 172.21.15.78:0/397881095 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v148) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6450057830 con 0x7f645c06d910 2024-02-19T14:57:41.729 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f644006b810 msgr2=0x7f644006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f644006b810 0x7f644006dcd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f645c0af160 tx=0x7f6448008040 comp rx=0 tx=0).stop 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 msgr2=0x7f645c0c0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c0c0ce0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f645000c400 tx=0x7f645000c8c0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 shutdown_connections 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f644006b810 0x7f644006dcd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f645c152700 0x7f645c1e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f645c0c8e00 0x7f645c0ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 --2- 172.21.15.78:0/397881095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f645c06d910 0x7f645c0c0ce0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 >> 172.21.15.78:0/397881095 conn(0x7f645c0cac90 msgr2=0x7f645c151680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 shutdown_connections 2024-02-19T14:57:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.728+0000 7f6462b02640 1 -- 172.21.15.78:0/397881095 wait complete. 2024-02-19T14:57:41.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:57:41.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.841+0000 7f2a1fef4640 1 Processor -- start 2024-02-19T14:57:41.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.841+0000 7f2a1fef4640 1 -- start start 2024-02-19T14:57:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1fef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 0x7f2a180c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1fef4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a180c6af0 con 0x7f2a18155f90 2024-02-19T14:57:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1fef4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a180c7940 con 0x7f2a181588a0 2024-02-19T14:57:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1fef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180c7270 con 0x7f2a1806d550 2024-02-19T14:57:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1dc90640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2a18155f90 0x7f2a180c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40846/0 (socket says 172.21.15.78:40846) 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1dc90640 1 -- 172.21.15.78:0/3067148789 learned_addr learned my addr 172.21.15.78:0/3067148789 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109588205 0 0) 0x7f2a180c6af0 con 0x7f2a18155f90 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a04003520 con 0x7f2a18155f90 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.842+0000 7f2a1d48f640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 0x7f2a180c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269794638 0 0) 0x7f2a04003520 con 0x7f2a18155f90 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 msgr2=0x7f2a180c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 0x7f2a180c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 >> v1:172.21.15.150:6789/0 conn(0x7f2a181588a0 legacy=0x7f2a18158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180c7c20 con 0x7f2a18155f90 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2a140019f0 con 0x7f2a18155f90 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 >> v1:172.21.15.78:6789/0 conn(0x7f2a18155f90 legacy=0x7f2a180c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a040039a0 0x7f2a04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040047c0 0x7f2a04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 0x7f2a0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180c7270 con 0x7f2a040039a0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040054e0 con 0x7f2a040047c0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0400bec0 con 0x7f2a1806d550 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a040039a0 0x7f2a04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1d48f640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040047c0 0x7f2a04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1e491640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 0x7f2a0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1dc90640 1 -- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 msgr2=0x7f2a0400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a1806d550 0x7f2a0400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1dc90640 1 -- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040047c0 msgr2=0x7f2a04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040047c0 0x7f2a04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1dc90640 1 -- 172.21.15.78:0/3067148789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04003520 con 0x7f2a040039a0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.843+0000 7f2a1d48f640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040047c0 0x7f2a04004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a040039a0 0x7f2a04003dc0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400a820 tx=0x7f2a040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18698b99112cb97 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140072d0 con 0x7f2a040039a0 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1402f070 con 0x7f2a040039a0 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1cc8e640 1 -- 172.21.15.78:0/3067148789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140076b0 con 0x7f2a040039a0 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3067148789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a040039a0 msgr2=0x7f2a04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3067148789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a040039a0 0x7f2a04003dc0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400a820 tx=0x7f2a040057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.844+0000 7f2a1e491640 1 -- 172.21.15.78:0/3067148789 reap_dead start 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.845+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3067148789 shutdown_connections 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.845+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3067148789 >> 172.21.15.78:0/3067148789 conn(0x7f2a1806b700 msgr2=0x7f2a180aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.845+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3067148789 shutdown_connections 2024-02-19T14:57:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.845+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3067148789 wait complete. 2024-02-19T14:57:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 Processor -- start 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 -- start start 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040039a0 0x7f2a181eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a181e2350 0x7f2a1814e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 0x7f2a1814c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180abe90 con 0x7f2a180acff0 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a181eef40 con 0x7f2a040039a0 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1fef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180ab3f0 con 0x7f2a181e2350 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1e491640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 0x7f2a1814c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1e491640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 0x7f2a1814c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53140/0 (socket says 172.21.15.78:53140) 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.846+0000 7f2a1e491640 1 -- 172.21.15.78:0/3528774619 learned_addr learned my addr 172.21.15.78:0/3528774619 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1d48f640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a181e2350 0x7f2a1814e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040039a0 0x7f2a181eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1e491640 1 -- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a181e2350 msgr2=0x7f2a1814e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1e491640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a181e2350 0x7f2a1814e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1e491640 1 -- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040039a0 msgr2=0x7f2a181eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1e491640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040039a0 0x7f2a181eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1e491640 1 -- 172.21.15.78:0/3528774619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180ab6d0 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1d48f640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a181e2350 0x7f2a1814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040039a0 0x7f2a181eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1e491640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 0x7f2a1814c140 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2a10000c00 tx=0x7f2a1000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100052e0 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a10005440 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a18156410 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10003830 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.847+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1814eab0 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.848+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a18074990 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.849+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2a10003990 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.849+0000 7f2a0e7fc640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29fc06b810 0x7f29fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.849+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29fc06b810 0x7f29fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.849+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2a10089570 con 0x7f2a180acff0 2024-02-19T14:57:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.849+0000 7f2a1dc90640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29fc06b810 0x7f29fc06dcd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2a14034000 tx=0x7f2a14007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:41.853+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2a100541e0 con 0x7f2a180acff0 2024-02-19T14:57:42.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.086+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2a180c75a0 con 0x7f2a180acff0 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.087+0000 7f2a0e7fc640 1 -- 172.21.15.78:0/3528774619 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f2a1000eea0 con 0x7f2a180acff0 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:57:42.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:57:42.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.093 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.095 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.097 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.097 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:57:42.097 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.100 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:57:42.101 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:57:42.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:57:42.103 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:57:42.105 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29fc06b810 msgr2=0x7f29fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29fc06b810 0x7f29fc06dcd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2a14034000 tx=0x7f2a14007000 comp rx=0 tx=0).stop 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 msgr2=0x7f2a1814c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 0x7f2a1814c140 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2a10000c00 tx=0x7f2a1000e820 comp rx=0 tx=0).stop 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 shutdown_connections 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29fc06b810 0x7f29fc06dcd0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a180acff0 0x7f2a1814c140 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a181e2350 0x7f2a1814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 --2- 172.21.15.78:0/3528774619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a040039a0 0x7f2a181eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 >> 172.21.15.78:0/3528774619 conn(0x7f2a1806b700 msgr2=0x7f2a180c3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 shutdown_connections 2024-02-19T14:57:42.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.089+0000 7f2a1fef4640 1 -- 172.21.15.78:0/3528774619 wait complete. 2024-02-19T14:57:42.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:42.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.204+0000 7fe2fa2c7640 1 Processor -- start 2024-02-19T14:57:42.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.204+0000 7fe2fa2c7640 1 -- start start 2024-02-19T14:57:42.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.205+0000 7fe2fa2c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2f406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.205+0000 7fe2fa2c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40c6af0 con 0x7fe2f40c6d30 2024-02-19T14:57:42.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.205+0000 7fe2fa2c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40c7940 con 0x7fe2f41588a0 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.205+0000 7fe2fa2c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40c7c20 con 0x7fe2f4155f90 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.205+0000 7fe2f37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe2f40c6d30 0x7fe2f40c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40858/0 (socket says 172.21.15.78:40858) 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.205+0000 7fe2f37fe640 1 -- 172.21.15.78:0/1489723333 learned_addr learned my addr 172.21.15.78:0/1489723333 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763014171 0 0) 0x7fe2f40c6af0 con 0x7fe2f40c6d30 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f3fff640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2f406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2dc003520 con 0x7fe2f40c6d30 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14546993 0 0) 0x7fe2dc003520 con 0x7fe2f40c6d30 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 msgr2=0x7fe2f406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2f406d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 >> v1:172.21.15.150:6789/0 conn(0x7fe2f41588a0 legacy=0x7fe2f4158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40c7f00 con 0x7fe2f40c6d30 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe2e8001670 con 0x7fe2f40c6d30 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 >> v1:172.21.15.78:6789/0 conn(0x7fe2f40c6d30 legacy=0x7fe2f40c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2dc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2dc003f20 0x7fe2dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc004d00 0x7fe2dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40c7c20 con 0x7fe2dc003f20 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00bb10 con 0x7fe2dc004d00 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00bcd0 con 0x7fe2f4155f90 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f37fe640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2dc003f20 0x7fe2dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f3fff640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2dc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f8864640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc004d00 0x7fe2dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f37fe640 1 -- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 msgr2=0x7fe2dc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f37fe640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2dc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f37fe640 1 -- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc004d00 msgr2=0x7fe2dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f37fe640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc004d00 0x7fe2dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f37fe640 1 -- 172.21.15.78:0/1489723333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc003520 con 0x7fe2dc003f20 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f3fff640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2f4155f90 0x7fe2dc0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.206+0000 7fe2f8864640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2dc004d00 0x7fe2dc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.207+0000 7fe2f37fe640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2dc003f20 0x7fe2dc0047c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800c160 tx=0x7fe2dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ee79323cf541eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.207+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8005ad0 con 0x7fe2dc003f20 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.207+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e8005c50 con 0x7fe2dc003f20 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.207+0000 7fe2f2ffd640 1 -- 172.21.15.78:0/1489723333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e802c870 con 0x7fe2dc003f20 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1489723333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2dc003f20 msgr2=0x7fe2dc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1489723333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2dc003f20 0x7fe2dc0047c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fe2e800c160 tx=0x7fe2dc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2f8864640 1 -- 172.21.15.78:0/1489723333 reap_dead start 2024-02-19T14:57:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1489723333 shutdown_connections 2024-02-19T14:57:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1489723333 >> 172.21.15.78:0/1489723333 conn(0x7fe2f406b700 msgr2=0x7fe2f40aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1489723333 shutdown_connections 2024-02-19T14:57:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.208+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1489723333 wait complete. 2024-02-19T14:57:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.209+0000 7fe2fa2c7640 1 Processor -- start 2024-02-19T14:57:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.209+0000 7fe2fa2c7640 1 -- start start 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2dc003f20 0x7fe2f41483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2f4156370 0x7fe2f40ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 0x7fe2f4157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f414c760 con 0x7fe2f4149230 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4148ce0 con 0x7fe2f4156370 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f4148f30 con 0x7fe2dc003f20 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 0x7fe2f4157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 0x7fe2f4157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53158/0 (socket says 172.21.15.78:53158) 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 -- 172.21.15.78:0/1654463103 learned_addr learned my addr 172.21.15.78:0/1654463103 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f3fff640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2dc003f20 0x7fe2f41483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f37fe640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2f4156370 0x7fe2f40ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 -- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2dc003f20 msgr2=0x7fe2f41483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2dc003f20 0x7fe2f41483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 -- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2f4156370 msgr2=0x7fe2f40ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2f4156370 0x7fe2f40ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 -- 172.21.15.78:0/1654463103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f41eb8c0 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f8864640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 0x7fe2f4157da0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40c6560 tx=0x7fe2ec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec006d50 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec003970 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f41ebba0 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec003d60 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.210+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f414fd10 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.212+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f41ebd90 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.212+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe2ec005070 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.212+0000 7fe2f0ff9640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2e006b840 0x7fe2e006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.212+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe2ec089c90 con 0x7fe2f4149230 2024-02-19T14:57:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.212+0000 7fe2f3fff640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2e006b840 0x7fe2e006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.213+0000 7fe2f3fff640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2e006b840 0x7fe2e006dd00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009aa0 tx=0x7fe2e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.217+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2ec054900 con 0x7fe2f4149230 2024-02-19T14:57:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.390+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2f4074990 con 0x7fe2f4149230 2024-02-19T14:57:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.391+0000 7fe2f0ff9640 1 -- 172.21.15.78:0/1654463103 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+13162 (secure 0 0 0) 0x7fe2ec057fc0 con 0x7fe2f4149230 2024-02-19T14:57:42.394 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:42.394 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":148,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:41.725868+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:42.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2e006b840 msgr2=0x7fe2e006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2e006b840 0x7fe2e006dd00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009aa0 tx=0x7fe2e4008040 comp rx=0 tx=0).stop 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 msgr2=0x7fe2f4157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 0x7fe2f4157da0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40c6560 tx=0x7fe2ec00dc00 comp rx=0 tx=0).stop 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 shutdown_connections 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2e006b840 0x7fe2e006dd00 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f4149230 0x7fe2f4157da0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2f4156370 0x7fe2f40ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.392+0000 7fe2fa2c7640 1 --2- 172.21.15.78:0/1654463103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2dc003f20 0x7fe2f41483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.393+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 >> 172.21.15.78:0/1654463103 conn(0x7fe2f406b700 msgr2=0x7fe2f40c1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.393+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 shutdown_connections 2024-02-19T14:57:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.393+0000 7fe2fa2c7640 1 -- 172.21.15.78:0/1654463103 wait complete. 2024-02-19T14:57:42.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:57:42.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:57:42.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.507+0000 7f1cfd8c7640 1 Processor -- start 2024-02-19T14:57:42.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.507+0000 7f1cfd8c7640 1 -- start start 2024-02-19T14:57:42.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cfd8c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1cf80c78a0 0x7f1cf8159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cfd8c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1cf8155010 0x7f1cf81523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cfd8c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf806ffa0 con 0x7f1cf80c4d80 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cfd8c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf806b720 con 0x7f1cf80c78a0 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cfd8c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf806fc70 con 0x7f1cf8155010 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cf6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1cf80c78a0 0x7f1cf8159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cf67fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1cf80c4d80 0x7f1cf80c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40874/0 (socket says 172.21.15.78:40874) 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cf67fc640 1 -- 172.21.15.78:0/959562538 learned_addr learned my addr 172.21.15.78:0/959562538 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.508+0000 7f1cf77fe640 1 --2- 172.21.15.78:0/959562538 >> v2:172.21.15.150:3300/0 conn(0x7f1cf8155010 0x7f1cf81523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf6ffd640 1 -- 172.21.15.78:0/959562538 >> v2:172.21.15.150:3300/0 conn(0x7f1cf8155010 msgr2=0x7f1cf81523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf6ffd640 1 --2- 172.21.15.78:0/959562538 >> v2:172.21.15.150:3300/0 conn(0x7f1cf8155010 0x7f1cf81523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf6ffd640 1 -- 172.21.15.78:0/959562538 >> v1:172.21.15.78:6789/0 conn(0x7f1cf80c4d80 legacy=0x7f1cf80c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf5ffb640 1 -- 172.21.15.78:0/959562538 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518477165 0 0) 0x7f1cf806ffa0 con 0x7f1cf80c4d80 2024-02-19T14:57:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf6ffd640 1 -- 172.21.15.78:0/959562538 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf806b930 con 0x7f1cf80c78a0 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf6ffd640 1 --2- 172.21.15.78:0/959562538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf8159010 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80c5210 tx=0x7f1ce802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955022d52affb6eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf5ffb640 1 -- 172.21.15.78:0/959562538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce802cbf0 con 0x7f1cf80c78a0 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf5ffb640 1 -- 172.21.15.78:0/959562538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce802cd50 con 0x7f1cf80c78a0 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.509+0000 7f1cf5ffb640 1 -- 172.21.15.78:0/959562538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8034660 con 0x7f1cf80c78a0 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/959562538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 msgr2=0x7f1cf8159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/959562538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf8159010 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80c5210 tx=0x7f1ce802be40 comp rx=0 tx=0).stop 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/959562538 shutdown_connections 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/959562538 >> v2:172.21.15.150:3300/0 conn(0x7f1cf8155010 0x7f1cf81523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/959562538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf8159010 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/959562538 >> 172.21.15.78:0/959562538 conn(0x7f1cf8066e70 msgr2=0x7f1cf80a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/959562538 shutdown_connections 2024-02-19T14:57:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.510+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/959562538 wait complete. 2024-02-19T14:57:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.511+0000 7f1cfd8c7640 1 Processor -- start 2024-02-19T14:57:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.511+0000 7f1cfd8c7640 1 -- start start 2024-02-19T14:57:42.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cfd8c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c4d80 0x7f1cf8151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cfd8c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf814b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cfd8c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8155010 0x7f1cf814b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cfd8c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf814ba90 con 0x7f1cf80c78a0 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cfd8c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8157450 con 0x7f1cf80c4d80 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cfd8c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf80ab740 con 0x7f1cf8155010 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf814b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf814b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53184/0 (socket says 172.21.15.78:53184) 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 -- 172.21.15.78:0/1865958482 learned_addr learned my addr 172.21.15.78:0/1865958482 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf6ffd640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c4d80 0x7f1cf8151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf77fe640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8155010 0x7f1cf814b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 -- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8155010 msgr2=0x7f1cf814b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8155010 0x7f1cf814b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 -- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c4d80 msgr2=0x7f1cf8151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c4d80 0x7f1cf8151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf67fc640 1 -- 172.21.15.78:0/1865958482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce8009a70 con 0x7f1cf80c78a0 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.512+0000 7f1cf6ffd640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c4d80 0x7f1cf8151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.513+0000 7f1cf67fc640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf814b010 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c4d0 tx=0x7f1cec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.513+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0074e0 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.513+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec007640 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.513+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf81ee7d0 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.513+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec004a20 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.513+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf8157170 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.514+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf8074990 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.514+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1cec004b80 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.514+0000 7f1cdffff640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ce406b840 0x7f1ce406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.514+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1cec089450 con 0x7f1cf80c78a0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.514+0000 7f1cf6ffd640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ce406b840 0x7f1ce406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.515+0000 7f1cf6ffd640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ce406b840 0x7f1ce406dd00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8036000 tx=0x7f1ce8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.519+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1cec0540c0 con 0x7f1cf80c78a0 2024-02-19T14:57:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.688+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1cf814f930 con 0x7f1cf80c78a0 2024-02-19T14:57:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.688+0000 7f1cdffff640 1 -- 172.21.15.78:0/1865958482 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1cec057780 con 0x7f1cf80c78a0 2024-02-19T14:57:42.692 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:57:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ce406b840 msgr2=0x7f1ce406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ce406b840 0x7f1ce406dd00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8036000 tx=0x7f1ce8030000 comp rx=0 tx=0).stop 2024-02-19T14:57:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 msgr2=0x7f1cf814b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf814b010 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c4d0 tx=0x7f1cec00c990 comp rx=0 tx=0).stop 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 shutdown_connections 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1ce406b840 0x7f1ce406dd00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf80c78a0 0x7f1cf814b010 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1cf8155010 0x7f1cf814b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 --2- 172.21.15.78:0/1865958482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1cf80c4d80 0x7f1cf8151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 >> 172.21.15.78:0/1865958482 conn(0x7f1cf8066e70 msgr2=0x7f1cf81521a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 shutdown_connections 2024-02-19T14:57:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.690+0000 7f1cfd8c7640 1 -- 172.21.15.78:0/1865958482 wait complete. 2024-02-19T14:57:42.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:57:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.802+0000 7fe15d447640 1 Processor -- start 2024-02-19T14:57:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.802+0000 7fe15d447640 1 -- start start 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe15d447640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 0x7fe158160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe15d447640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1581551f0 con 0x7fe158152700 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe15d447640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe158155490 con 0x7fe1580c8e00 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe15d447640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe158155770 con 0x7fe15806c9b0 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe156ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe158152700 0x7fe1580c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40880/0 (socket says 172.21.15.78:40880) 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe156ffd640 1 -- 172.21.15.78:0/892580755 learned_addr learned my addr 172.21.15.78:0/892580755 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802447886 0 0) 0x7fe1581551f0 con 0x7fe158152700 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe144003520 con 0x7fe158152700 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1577fe640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 0x7fe158160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679054082 0 0) 0x7fe144003520 con 0x7fe158152700 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 msgr2=0x7fe158160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 0x7fe158160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 >> v1:172.21.15.150:6789/0 conn(0x7fe1580c8e00 legacy=0x7fe15806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe158155a50 con 0x7fe158152700 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fe148001670 con 0x7fe158152700 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 >> v1:172.21.15.78:6789/0 conn(0x7fe158152700 legacy=0x7fe1580c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440039a0 0x7fe144003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1440047c0 0x7fe144004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 0x7fe1440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158155770 con 0x7fe1440047c0 2024-02-19T14:57:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14400bab0 con 0x7fe1440039a0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14400bc70 con 0x7fe15806c9b0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe14ffff640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1440047c0 0x7fe144004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe156ffd640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440039a0 0x7fe144003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.803+0000 7fe1577fe640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 0x7fe1440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe14ffff640 1 -- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 msgr2=0x7fe1440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe14ffff640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe15806c9b0 0x7fe1440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe14ffff640 1 -- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440039a0 msgr2=0x7fe144003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe14ffff640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440039a0 0x7fe144003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe14ffff640 1 -- 172.21.15.78:0/892580755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144003520 con 0x7fe1440047c0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe14ffff640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1440047c0 0x7fe144004c40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fe140001750 tx=0x7fe1440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6ff013de3810de server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.804+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1400052d0 con 0x7fe1440047c0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14002e050 con 0x7fe1440047c0 2024-02-19T14:57:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe1567fc640 1 -- 172.21.15.78:0/892580755 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1400035f0 con 0x7fe1440047c0 2024-02-19T14:57:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe15d447640 1 -- 172.21.15.78:0/892580755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1440047c0 msgr2=0x7fe144004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe15d447640 1 --2- 172.21.15.78:0/892580755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1440047c0 0x7fe144004c40 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fe140001750 tx=0x7fe1440057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe1577fe640 1 -- 172.21.15.78:0/892580755 reap_dead start 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe15d447640 1 -- 172.21.15.78:0/892580755 shutdown_connections 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe15d447640 1 -- 172.21.15.78:0/892580755 >> 172.21.15.78:0/892580755 conn(0x7fe1580cac90 msgr2=0x7fe1580cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe15d447640 1 -- 172.21.15.78:0/892580755 shutdown_connections 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.805+0000 7fe15d447640 1 -- 172.21.15.78:0/892580755 wait complete. 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.806+0000 7fe15d447640 1 Processor -- start 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.806+0000 7fe15d447640 1 -- start start 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe15d447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440047c0 0x7fe1581ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe15d447640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1581587b0 0x7fe1580bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe15d447640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 0x7fe158157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe15d447640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158156c40 con 0x7fe1580c0000 2024-02-19T14:57:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe15d447640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580c4820 con 0x7fe1440047c0 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe15d447640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581ee830 con 0x7fe1581587b0 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 0x7fe158157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 0x7fe158157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53206/0 (socket says 172.21.15.78:53206) 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 -- 172.21.15.78:0/3125985374 learned_addr learned my addr 172.21.15.78:0/3125985374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe156ffd640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440047c0 0x7fe1581ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 -- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1581587b0 msgr2=0x7fe1580bfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe14ffff640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1581587b0 0x7fe1580bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1581587b0 0x7fe1580bfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 -- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440047c0 msgr2=0x7fe1581ee2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440047c0 0x7fe1581ee2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 -- 172.21.15.78:0/3125985374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580c4540 con 0x7fe1580c0000 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe156ffd640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440047c0 0x7fe1581ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.807+0000 7fe1577fe640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 0x7fe158157060 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fe15808c490 tx=0x7fe150007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.808+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1500046a0 con 0x7fe1580c0000 2024-02-19T14:57:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.808+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe150004800 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.808+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe158156580 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.808+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150015640 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.808+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1580c0480 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.809+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1580be080 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.809+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe1500157a0 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.810+0000 7fe14f7fe640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe13806b810 0x7fe13806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.810+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe15008a640 con 0x7fe1580c0000 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.810+0000 7fe156ffd640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe13806b810 0x7fe13806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.810+0000 7fe156ffd640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe13806b810 0x7fe13806dcd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe148001050 tx=0x7fe148009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.814+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe1500552b0 con 0x7fe1580c0000 2024-02-19T14:57:42.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.986+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe158074990 con 0x7fe1580c0000 2024-02-19T14:57:42.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.987+0000 7fe14f7fe640 1 -- 172.21.15.78:0/3125985374 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v148) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe150058970 con 0x7fe1580c0000 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:42.991 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:57:42.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe13806b810 msgr2=0x7fe13806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe13806b810 0x7fe13806dcd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe148001050 tx=0x7fe148009040 comp rx=0 tx=0).stop 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 msgr2=0x7fe158157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 0x7fe158157060 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fe15808c490 tx=0x7fe150007700 comp rx=0 tx=0).stop 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 shutdown_connections 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe13806b810 0x7fe13806dcd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1580c0000 0x7fe158157060 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe1581587b0 0x7fe1580bfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 --2- 172.21.15.78:0/3125985374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1440047c0 0x7fe1581ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 >> 172.21.15.78:0/3125985374 conn(0x7fe1580cac90 msgr2=0x7fe15806c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 shutdown_connections 2024-02-19T14:57:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:42.989+0000 7fe15d447640 1 -- 172.21.15.78:0/3125985374 wait complete. 2024-02-19T14:57:43.003 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:57:43.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:57:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.104+0000 7fb2399f6640 1 Processor -- start 2024-02-19T14:57:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.104+0000 7fb2399f6640 1 -- start start 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2399f6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2340c14e0 0x7fb2341555c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2399f6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb234073520 con 0x7fb23414efa0 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2399f6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340c0ad0 con 0x7fb2340c7f40 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2399f6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23414f380 con 0x7fb2340c14e0 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb232ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb23414efa0 0x7fb23406b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40890/0 (socket says 172.21.15.78:40890) 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb232ffd640 1 -- 172.21.15.78:0/2665757330 learned_addr learned my addr 172.21.15.78:0/2665757330 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2337fe640 1 --2- 172.21.15.78:0/2665757330 >> v2:172.21.15.150:3300/0 conn(0x7fb2340c14e0 0x7fb2341555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832364792 0 0) 0x7fb234073520 con 0x7fb23414efa0 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.105+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb220003520 con 0x7fb23414efa0 2024-02-19T14:57:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419874406 0 0) 0x7fb2340c0ad0 con 0x7fb2340c7f40 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234073520 con 0x7fb2340c7f40 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77775568 0 0) 0x7fb220003520 con 0x7fb23414efa0 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 >> v2:172.21.15.150:3300/0 conn(0x7fb2340c14e0 msgr2=0x7fb2341555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 --2- 172.21.15.78:0/2665757330 >> v2:172.21.15.150:3300/0 conn(0x7fb2340c14e0 0x7fb2341555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 >> v1:172.21.15.150:6789/0 conn(0x7fb2340c7f40 legacy=0x7fb234154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340c1240 con 0x7fb23414efa0 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb224001670 con 0x7fb23414efa0 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 >> v1:172.21.15.78:6789/0 conn(0x7fb23414efa0 legacy=0x7fb23406b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb220003b00 0x7fb220003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2200047c0 0x7fb220004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb220005180 0x7fb22000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23414f380 con 0x7fb2200047c0 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22000c0b0 con 0x7fb220005180 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22000c270 con 0x7fb220003b00 2024-02-19T14:57:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb22bfff640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2200047c0 0x7fb220004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2337fe640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb220005180 0x7fb22000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb232ffd640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb220003b00 0x7fb220003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb22bfff640 1 -- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb220003b00 msgr2=0x7fb220003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb22bfff640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb220003b00 0x7fb220003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb22bfff640 1 -- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb220005180 msgr2=0x7fb22000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb22bfff640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb220005180 0x7fb22000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb22bfff640 1 -- 172.21.15.78:0/2665757330 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb220003520 con 0x7fb2200047c0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2337fe640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb220005180 0x7fb22000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.106+0000 7fb2337fe640 1 -- 172.21.15.78:0/2665757330 reap_dead start 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.107+0000 7fb22bfff640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2200047c0 0x7fb220004c40 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0018a0 tx=0x7fb2200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f87cc435ac85a7a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.107+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0050d0 con 0x7fb2200047c0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.107+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21c005230 con 0x7fb2200047c0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.107+0000 7fb2327fc640 1 -- 172.21.15.78:0/2665757330 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0035f0 con 0x7fb2200047c0 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 -- 172.21.15.78:0/2665757330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2200047c0 msgr2=0x7fb220004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2200047c0 0x7fb220004c40 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0018a0 tx=0x7fb2200057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 -- 172.21.15.78:0/2665757330 shutdown_connections 2024-02-19T14:57:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2665757330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2200047c0 0x7fb220004c40 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 -- 172.21.15.78:0/2665757330 >> 172.21.15.78:0/2665757330 conn(0x7fb23406bf90 msgr2=0x7fb23406c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 -- 172.21.15.78:0/2665757330 shutdown_connections 2024-02-19T14:57:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.108+0000 7fb2399f6640 1 -- 172.21.15.78:0/2665757330 wait complete. 2024-02-19T14:57:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.109+0000 7fb2399f6640 1 Processor -- start 2024-02-19T14:57:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.109+0000 7fb2399f6640 1 -- start start 2024-02-19T14:57:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2200047c0 0x7fb234149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb234158f70 0x7fb234147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 0x7fb2340abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234157cd0 con 0x7fb234147960 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234157ee0 con 0x7fb234158f70 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2341581c0 con 0x7fb2200047c0 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 0x7fb2340abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 0x7fb2340abab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53224/0 (socket says 172.21.15.78:53224) 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 -- 172.21.15.78:0/2638260004 learned_addr learned my addr 172.21.15.78:0/2638260004 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb232ffd640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2200047c0 0x7fb234149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb22bfff640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb234158f70 0x7fb234147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 -- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2200047c0 msgr2=0x7fb234149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2200047c0 0x7fb234149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 -- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb234158f70 msgr2=0x7fb234147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb234158f70 0x7fb234147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 -- 172.21.15.78:0/2638260004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340aafd0 con 0x7fb234147960 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2337fe640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 0x7fb2340abab0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fb2340736f0 tx=0x7fb22c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c01b070 con 0x7fb234147960 2024-02-19T14:57:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c00f040 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2340ab2b0 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00e050 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.110+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb234149ad0 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.112+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2340ac7b0 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.112+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb22c015440 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.112+0000 7fb230ff9640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb21406b840 0x7fb21406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.112+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb22c08a000 con 0x7fb234147960 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.112+0000 7fb232ffd640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb21406b840 0x7fb21406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.113+0000 7fb232ffd640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb21406b840 0x7fb21406dd00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb224005ab0 tx=0x7fb224007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:43.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.117+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb22c054c90 con 0x7fb234147960 2024-02-19T14:57:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.290+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb234074990 con 0x7fb234147960 2024-02-19T14:57:43.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.731+0000 7fb230ff9640 1 -- 172.21.15.78:0/2638260004 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v149) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb22c058350 con 0x7fb234147960 2024-02-19T14:57:43.735 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:57:43.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb21406b840 msgr2=0x7fb21406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb21406b840 0x7fb21406dd00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb224005ab0 tx=0x7fb224007040 comp rx=0 tx=0).stop 2024-02-19T14:57:43.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 msgr2=0x7fb2340abab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 0x7fb2340abab0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fb2340736f0 tx=0x7fb22c010590 comp rx=0 tx=0).stop 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 shutdown_connections 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb21406b840 0x7fb21406dd00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb234147960 0x7fb2340abab0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb234158f70 0x7fb234147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 --2- 172.21.15.78:0/2638260004 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2200047c0 0x7fb234149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.733+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 >> 172.21.15.78:0/2638260004 conn(0x7fb23406bf90 msgr2=0x7fb2340c5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.734+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 shutdown_connections 2024-02-19T14:57:43.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.734+0000 7fb2399f6640 1 -- 172.21.15.78:0/2638260004 wait complete. 2024-02-19T14:57:43.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:57:43.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.859+0000 7f8f02c0e640 1 Processor -- start 2024-02-19T14:57:43.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.859+0000 7f8f02c0e640 1 -- start start 2024-02-19T14:57:43.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8f02c0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc0c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8f02c0e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8efc0c8ee0 0x7f8efc0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8f02c0e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc152a60 con 0x7f8efc152680 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8f02c0e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0aa050 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8f02c0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1557e0 con 0x7f8efc0ace90 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8efbfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc0c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8efbfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc0c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:34602/0 (socket says 172.21.15.78:34602) 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8efbfff640 1 -- 172.21.15.78:0/4063129400 learned_addr learned my addr 172.21.15.78:0/4063129400 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.860+0000 7f8f011ab640 1 --2- 172.21.15.78:0/4063129400 >> v2:172.21.15.150:3300/0 conn(0x7f8efc0c8ee0 0x7f8efc0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.861+0000 7f8efb7fe640 1 -- 172.21.15.78:0/4063129400 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841796455 0 0) 0x7f8efc152a60 con 0x7f8efc152680 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.861+0000 7f8efbfff640 1 -- 172.21.15.78:0/4063129400 >> v2:172.21.15.150:3300/0 conn(0x7f8efc0c8ee0 msgr2=0x7f8efc0c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.861+0000 7f8efbfff640 1 --2- 172.21.15.78:0/4063129400 >> v2:172.21.15.150:3300/0 conn(0x7f8efc0c8ee0 0x7f8efc0c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.861+0000 7f8efbfff640 1 -- 172.21.15.78:0/4063129400 >> v1:172.21.15.150:6789/0 conn(0x7f8efc152680 legacy=0x7f8efc155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.861+0000 7f8efbfff640 1 -- 172.21.15.78:0/4063129400 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc155ac0 con 0x7f8efc0ace90 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.861+0000 7f8efbfff640 1 --2- 172.21.15.78:0/4063129400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc0c89a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009870 tx=0x7f8ef002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77cc083d89339d67 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8efb7fe640 1 -- 172.21.15.78:0/4063129400 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0031640 con 0x7f8efc0ace90 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8efb7fe640 1 -- 172.21.15.78:0/4063129400 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef00317a0 con 0x7f8efc0ace90 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8efb7fe640 1 -- 172.21.15.78:0/4063129400 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0031b30 con 0x7f8efc0ace90 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 -- 172.21.15.78:0/4063129400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 msgr2=0x7f8efc0c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/4063129400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc0c89a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009870 tx=0x7f8ef002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 -- 172.21.15.78:0/4063129400 shutdown_connections 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/4063129400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc0c89a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/4063129400 >> v2:172.21.15.150:3300/0 conn(0x7f8efc0c8ee0 0x7f8efc0c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 -- 172.21.15.78:0/4063129400 >> 172.21.15.78:0/4063129400 conn(0x7f8efc0ac370 msgr2=0x7f8efc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 -- 172.21.15.78:0/4063129400 shutdown_connections 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.862+0000 7f8f02c0e640 1 -- 172.21.15.78:0/4063129400 wait complete. 2024-02-19T14:57:43.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.863+0000 7f8f02c0e640 1 Processor -- start 2024-02-19T14:57:43.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.863+0000 7f8f02c0e640 1 -- start start 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f02c0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc1edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f02c0e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 0x7f8efc0bd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f02c0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8efc152680 0x7f8efc0bdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f02c0e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0bfb50 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f02c0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc1eec90 con 0x7f8efc152680 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f02c0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0c42c0 con 0x7f8efc0ace90 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8efbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 0x7f8efc0bd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8efbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 0x7f8efc0bd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53240/0 (socket says 172.21.15.78:53240) 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8efbfff640 1 -- 172.21.15.78:0/3926969200 learned_addr learned my addr 172.21.15.78:0/3926969200 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f009aa640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc1edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.864+0000 7f8f011ab640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8efc152680 0x7f8efc0bdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8efbfff640 1 -- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 msgr2=0x7f8efc1edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8efbfff640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc1edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8efbfff640 1 -- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8efc152680 msgr2=0x7f8efc0bdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8efbfff640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8efc152680 0x7f8efc0bdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8efbfff640 1 -- 172.21.15.78:0/3926969200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00094d0 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8efbfff640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 0x7f8efc0bd980 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0000c00 tx=0x7f8ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef003a930 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef003aa90 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef003ae40 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc1ee9b0 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.865+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc0c3f50 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.866+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc074990 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.866+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8ef002f050 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.867+0000 7f8ef97fa640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ee806b810 0x7f8ee806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.867+0000 7f8f009aa640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ee806b810 0x7f8ee806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.867+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f8ef00abde0 con 0x7f8efc0c8ee0 2024-02-19T14:57:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.867+0000 7f8f009aa640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ee806b810 0x7f8ee806dcd0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8eec000f60 tx=0x7f8eec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:43.872+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8ef0076990 con 0x7f8efc0c8ee0 2024-02-19T14:57:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.041+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8efc0c4400 con 0x7f8efc0c8ee0 2024-02-19T14:57:44.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.734+0000 7f8ef97fa640 1 -- 172.21.15.78:0/3926969200 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v150) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8ef007a050 con 0x7f8efc0c8ee0 2024-02-19T14:57:44.737 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:57:44.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.737+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ee806b810 msgr2=0x7f8ee806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:44.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.737+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ee806b810 0x7f8ee806dcd0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8eec000f60 tx=0x7f8eec009040 comp rx=0 tx=0).stop 2024-02-19T14:57:44.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.737+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 msgr2=0x7f8efc0bd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:44.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.737+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 0x7f8efc0bd980 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0000c00 tx=0x7f8ef00046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.738+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 shutdown_connections 2024-02-19T14:57:44.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.738+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ee806b810 0x7f8ee806dcd0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.738+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8efc0c8ee0 0x7f8efc0bd980 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.738+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8efc152680 0x7f8efc0bdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.738+0000 7f8f02c0e640 1 --2- 172.21.15.78:0/3926969200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8efc0ace90 0x7f8efc1edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.738+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 >> 172.21.15.78:0/3926969200 conn(0x7f8efc0ac370 msgr2=0x7f8efc06d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:44.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.739+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 shutdown_connections 2024-02-19T14:57:44.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.740+0000 7f8f02c0e640 1 -- 172.21.15.78:0/3926969200 wait complete. 2024-02-19T14:57:44.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:57:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 Processor -- start 2024-02-19T14:57:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 -- start start 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4b80c7070 0x7fd4b81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd4b80c87a0 0x7fd4b80c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b8155ef0 con 0x7fd4b8152480 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b806dec0 con 0x7fd4b80c87a0 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4bdfbd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b806d520 con 0x7fd4b80c7070 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd4b80c87a0 0x7fd4b80c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd4b80c87a0 0x7fd4b80c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53252/0 (socket says 172.21.15.78:53252) 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3247888335 learned_addr learned my addr 172.21.15.78:0/3247888335 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b6ffd640 1 --2- 172.21.15.78:0/3247888335 >> v2:172.21.15.150:3300/0 conn(0x7fd4b80c7070 0x7fd4b81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3247888335 >> v2:172.21.15.150:3300/0 conn(0x7fd4b80c7070 msgr2=0x7fd4b81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 --2- 172.21.15.78:0/3247888335 >> v2:172.21.15.150:3300/0 conn(0x7fd4b80c7070 0x7fd4b81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3247888335 >> v1:172.21.15.150:6789/0 conn(0x7fd4b8152480 legacy=0x7fd4b806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.872+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3247888335 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80c75c0 con 0x7fd4b80c87a0 2024-02-19T14:57:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.873+0000 7fd4b7fff640 1 --2- 172.21.15.78:0/3247888335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b80c87a0 0x7fd4b80c5220 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80c6a90 tx=0x7fd4b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1c47c1a2e6cca8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.873+0000 7fd4b67fc640 1 -- 172.21.15.78:0/3247888335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b002ca80 con 0x7fd4b80c87a0 2024-02-19T14:57:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.873+0000 7fd4b67fc640 1 -- 172.21.15.78:0/3247888335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b002cbe0 con 0x7fd4b80c87a0 2024-02-19T14:57:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.873+0000 7fd4b67fc640 1 -- 172.21.15.78:0/3247888335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00343f0 con 0x7fd4b80c87a0 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3247888335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b80c87a0 msgr2=0x7fd4b80c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3247888335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b80c87a0 0x7fd4b80c5220 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80c6a90 tx=0x7fd4b002bee0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3247888335 shutdown_connections 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3247888335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b80c87a0 0x7fd4b80c5220 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3247888335 >> v2:172.21.15.150:3300/0 conn(0x7fd4b80c7070 0x7fd4b81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3247888335 >> 172.21.15.78:0/3247888335 conn(0x7fd4b80ca8a0 msgr2=0x7fd4b80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3247888335 shutdown_connections 2024-02-19T14:57:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.874+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3247888335 wait complete. 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.875+0000 7fd4bdfbd640 1 Processor -- start 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.875+0000 7fd4bdfbd640 1 -- start start 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.875+0000 7fd4bdfbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4b80c7070 0x7fd4b80ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4b80c87a0 0x7fd4b814d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 0x7fd4b814dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80afd70 con 0x7fd4b8152480 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80aefb0 con 0x7fd4b80c87a0 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80af7b0 con 0x7fd4b80c7070 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 0x7fd4b814dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 0x7fd4b814dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53258/0 (socket says 172.21.15.78:53258) 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3585549169 learned_addr learned my addr 172.21.15.78:0/3585549169 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b6ffd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4b80c87a0 0x7fd4b814d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b77fe640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4b80c7070 0x7fd4b80ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4b80c7070 msgr2=0x7fd4b80ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4b80c7070 0x7fd4b80ac3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4b80c87a0 msgr2=0x7fd4b814d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4b80c87a0 0x7fd4b814d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 -- 172.21.15.78:0/3585549169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b0009b30 con 0x7fd4b8152480 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b6ffd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4b80c87a0 0x7fd4b814d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4b7fff640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 0x7fd4b814dc20 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0036040 tx=0x7fd4b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0009510 con 0x7fd4b8152480 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b003e450 con 0x7fd4b8152480 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b80afa90 con 0x7fd4b8152480 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b003e7e0 con 0x7fd4b8152480 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.876+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b814e160 con 0x7fd4b8152480 2024-02-19T14:57:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.877+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b80c4d60 con 0x7fd4b8152480 2024-02-19T14:57:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.878+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd4b002cd50 con 0x7fd4b8152480 2024-02-19T14:57:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.878+0000 7fd49ffff640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4a406b810 0x7fd4a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.878+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fd4b00acc30 con 0x7fd4b8152480 2024-02-19T14:57:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.878+0000 7fd4b77fe640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4a406b810 0x7fd4a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.879+0000 7fd4b77fe640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4a406b810 0x7fd4a406dcd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8006fd0 tx=0x7fd4a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:44.883+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4b00776d0 con 0x7fd4b8152480 2024-02-19T14:57:45.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.093+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd4b80aea30 con 0x7fd4b8152480 2024-02-19T14:57:45.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.756+0000 7fd49ffff640 1 -- 172.21.15.78:0/3585549169 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v130) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd4b80aea30 con 0x7fd4b8152480 2024-02-19T14:57:45.759 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 23 and data pool 24 2024-02-19T14:57:45.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.759+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4a406b810 msgr2=0x7fd4a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:45.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.759+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4a406b810 0x7fd4a406dcd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8006fd0 tx=0x7fd4a8008040 comp rx=0 tx=0).stop 2024-02-19T14:57:45.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.759+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 msgr2=0x7fd4b814dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:45.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.759+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 0x7fd4b814dc20 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0036040 tx=0x7fd4b0004a40 comp rx=0 tx=0).stop 2024-02-19T14:57:45.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.761+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 shutdown_connections 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.761+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd4a406b810 0x7fd4a406dcd0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.761+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4b8152480 0x7fd4b814dc20 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.761+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4b80c87a0 0x7fd4b814d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.761+0000 7fd4bdfbd640 1 --2- 172.21.15.78:0/3585549169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4b80c7070 0x7fd4b80ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.761+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 >> 172.21.15.78:0/3585549169 conn(0x7fd4b80ca8a0 msgr2=0x7fd4b80c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.762+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 shutdown_connections 2024-02-19T14:57:45.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.762+0000 7fd4bdfbd640 1 -- 172.21.15.78:0/3585549169 wait complete. 2024-02-19T14:57:45.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:45.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.892+0000 7f1a7cd0d640 1 Processor -- start 2024-02-19T14:57:45.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.892+0000 7f1a7cd0d640 1 -- start start 2024-02-19T14:57:45.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7cd0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a780c42e0 0x7f1a78158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7cd0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1a780c2f50 0x7f1a780c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7cd0d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a780c34a0 con 0x7f1a780c8500 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7cd0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7806c090 con 0x7f1a780c2f50 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7cd0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7806d6a0 con 0x7f1a780c42e0 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1a780c8500 0x7f1a78158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40902/0 (socket says 172.21.15.78:40902) 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7659c640 1 -- 172.21.15.78:0/3682515740 learned_addr learned my addr 172.21.15.78:0/3682515740 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291516762 0 0) 0x7f1a780c34a0 con 0x7f1a780c8500 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a64003520 con 0x7f1a780c8500 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a76d9d640 1 --2- 172.21.15.78:0/3682515740 >> v2:172.21.15.150:3300/0 conn(0x7f1a780c2f50 0x7f1a780c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a6ffff640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a780c42e0 0x7f1a78158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625555560 0 0) 0x7f1a64003520 con 0x7f1a780c8500 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a780c42e0 msgr2=0x7f1a78158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:45.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a780c42e0 0x7f1a78158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 >> v2:172.21.15.150:3300/0 conn(0x7f1a780c2f50 msgr2=0x7f1a780c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 --2- 172.21.15.78:0/3682515740 >> v2:172.21.15.150:3300/0 conn(0x7f1a780c2f50 0x7f1a780c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7806c8e0 con 0x7f1a780c8500 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1a68001670 con 0x7f1a780c8500 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 >> v1:172.21.15.78:6789/0 conn(0x7f1a780c8500 legacy=0x7f1a78158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a64003970 0x7f1a64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a640047c0 0x7f1a64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64005180 0x7f1a6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7806d6a0 con 0x7f1a64003970 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6400bf70 con 0x7f1a64005180 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6400c130 con 0x7f1a640047c0 2024-02-19T14:57:45.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a6ffff640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a780c42e0 0x7f1a78158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.893+0000 7f1a7659c640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a64003970 0x7f1a64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a76d9d640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64005180 0x7f1a6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a6ffff640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a640047c0 0x7f1a64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a7659c640 1 -- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a640047c0 msgr2=0x7f1a64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a7659c640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a640047c0 0x7f1a64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a7659c640 1 -- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64005180 msgr2=0x7f1a6400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a7659c640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64005180 0x7f1a6400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a7659c640 1 -- 172.21.15.78:0/3682515740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a64003520 con 0x7f1a64003970 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.894+0000 7f1a7659c640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a64003970 0x7f1a64003d90 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800c190 tx=0x7f1a640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9059f49fc3c55e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68005ad0 con 0x7f1a64003970 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a68005c50 con 0x7f1a64003970 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a75d9b640 1 -- 172.21.15.78:0/3682515740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6802c810 con 0x7f1a64003970 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/3682515740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a64003970 msgr2=0x7f1a64003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:45.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/3682515740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a64003970 0x7f1a64003d90 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800c190 tx=0x7f1a640057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a76d9d640 1 -- 172.21.15.78:0/3682515740 reap_dead start 2024-02-19T14:57:45.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/3682515740 shutdown_connections 2024-02-19T14:57:45.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/3682515740 >> 172.21.15.78:0/3682515740 conn(0x7f1a780aa300 msgr2=0x7f1a780aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:45.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/3682515740 shutdown_connections 2024-02-19T14:57:45.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.895+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/3682515740 wait complete. 2024-02-19T14:57:45.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.896+0000 7f1a7cd0d640 1 Processor -- start 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.896+0000 7f1a7cd0d640 1 -- start start 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7cd0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64003970 0x7f1a78152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7cd0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78071060 0x7f1a780ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7cd0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 0x7f1a78070440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7cd0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a781ea9a0 con 0x7f1a7806ffc0 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7cd0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7806f290 con 0x7f1a64003970 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7cd0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a781eabb0 con 0x7f1a78071060 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a76d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 0x7f1a78070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a76d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 0x7f1a78070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53274/0 (socket says 172.21.15.78:53274) 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a76d9d640 1 -- 172.21.15.78:0/1502745855 learned_addr learned my addr 172.21.15.78:0/1502745855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a7659c640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64003970 0x7f1a78152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a76d9d640 1 -- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78071060 msgr2=0x7f1a780ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a6ffff640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78071060 0x7f1a780ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.897+0000 7f1a76d9d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78071060 0x7f1a780ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a76d9d640 1 -- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64003970 msgr2=0x7f1a78152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:45.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a76d9d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64003970 0x7f1a78152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a76d9d640 1 -- 172.21.15.78:0/1502745855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7806f4e0 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a7659c640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64003970 0x7f1a78152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a76d9d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 0x7f1a78070440 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f1a780c33a0 tx=0x7f1a7000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7000e040 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7000f040 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7806f7c0 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a70015430 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.898+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a780acb70 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.899+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a78071c20 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.899+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1a70015590 con 0x7f1a7806ffc0 2024-02-19T14:57:45.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.900+0000 7f1a6effd640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a5806b810 0x7f1a5806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:45.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.900+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a7008ad70 con 0x7f1a7806ffc0 2024-02-19T14:57:45.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.900+0000 7f1a7659c640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a5806b810 0x7f1a5806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:45.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.900+0000 7f1a7659c640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a5806b810 0x7f1a5806dcd0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1a68034000 tx=0x7f1a68007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:45.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:45.904+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a70055690 con 0x7f1a7806ffc0 2024-02-19T14:57:46.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.076+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a7806fce0 con 0x7f1a7806ffc0 2024-02-19T14:57:46.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.077+0000 7f1a6effd640 1 -- 172.21.15.78:0/1502745855 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16885 (secure 0 0 0) 0x7f1a7806fce0 con 0x7f1a7806ffc0 2024-02-19T14:57:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":151,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:45.095434+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a5806b810 msgr2=0x7f1a5806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a5806b810 0x7f1a5806dcd0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1a68034000 tx=0x7f1a68007000 comp rx=0 tx=0).stop 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 msgr2=0x7f1a78070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 0x7f1a78070440 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f1a780c33a0 tx=0x7f1a7000f580 comp rx=0 tx=0).stop 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 shutdown_connections 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1a5806b810 0x7f1a5806dcd0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a7806ffc0 0x7f1a78070440 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a78071060 0x7f1a780ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 --2- 172.21.15.78:0/1502745855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a64003970 0x7f1a78152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 >> 172.21.15.78:0/1502745855 conn(0x7f1a780aa300 msgr2=0x7f1a780c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:46.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 shutdown_connections 2024-02-19T14:57:46.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.079+0000 7f1a7cd0d640 1 -- 172.21.15.78:0/1502745855 wait complete. 2024-02-19T14:57:46.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:46.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.190+0000 7fca7a777640 1 Processor -- start 2024-02-19T14:57:46.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.190+0000 7fca7a777640 1 -- start start 2024-02-19T14:57:46.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca7a777640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fca740c25e0 0x7fca741526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca7a777640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca740c29c0 0x7fca740c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca7a777640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca740c49c0 0x7fca7406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca7a777640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca74152c90 con 0x7fca740c25e0 2024-02-19T14:57:46.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca7a777640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca74072250 con 0x7fca740c29c0 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca7a777640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca74072480 con 0x7fca740c49c0 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fca740c25e0 0x7fca741526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fca740c25e0 0x7fca741526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53284/0 (socket says 172.21.15.78:53284) 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 -- 172.21.15.78:0/4203472422 learned_addr learned my addr 172.21.15.78:0/4203472422 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 -- 172.21.15.78:0/4203472422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca740c49c0 msgr2=0x7fca7406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 --2- 172.21.15.78:0/4203472422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca740c49c0 0x7fca7406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 -- 172.21.15.78:0/4203472422 >> v2:172.21.15.150:3300/0 conn(0x7fca740c29c0 msgr2=0x7fca740c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 --2- 172.21.15.78:0/4203472422 >> v2:172.21.15.150:3300/0 conn(0x7fca740c29c0 0x7fca740c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.191+0000 7fca73fff640 1 -- 172.21.15.78:0/4203472422 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740c3eb0 con 0x7fca740c25e0 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.192+0000 7fca73fff640 1 --2- 172.21.15.78:0/4203472422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c25e0 0x7fca741526a0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fca740c59a0 tx=0x7fca6402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96b48737f19dddd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.192+0000 7fca72ffd640 1 -- 172.21.15.78:0/4203472422 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca640317c0 con 0x7fca740c25e0 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.192+0000 7fca72ffd640 1 -- 172.21.15.78:0/4203472422 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca64031920 con 0x7fca740c25e0 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.192+0000 7fca72ffd640 1 -- 172.21.15.78:0/4203472422 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64031cb0 con 0x7fca740c25e0 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 -- 172.21.15.78:0/4203472422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c25e0 msgr2=0x7fca741526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 --2- 172.21.15.78:0/4203472422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c25e0 0x7fca741526a0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fca740c59a0 tx=0x7fca6402b860 comp rx=0 tx=0).stop 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 -- 172.21.15.78:0/4203472422 shutdown_connections 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 --2- 172.21.15.78:0/4203472422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca740c49c0 0x7fca7406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 --2- 172.21.15.78:0/4203472422 >> v2:172.21.15.150:3300/0 conn(0x7fca740c29c0 0x7fca740c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 --2- 172.21.15.78:0/4203472422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c25e0 0x7fca741526a0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 -- 172.21.15.78:0/4203472422 >> 172.21.15.78:0/4203472422 conn(0x7fca7406b270 msgr2=0x7fca7406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 -- 172.21.15.78:0/4203472422 shutdown_connections 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.193+0000 7fca7a777640 1 -- 172.21.15.78:0/4203472422 wait complete. 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.194+0000 7fca7a777640 1 Processor -- start 2024-02-19T14:57:46.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.194+0000 7fca7a777640 1 -- start start 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca7a777640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca740c25e0 0x7fca741e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca7a777640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 0x7fca74147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca7a777640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca74143590 0x7fca741461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca7a777640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca74155e60 con 0x7fca740c29c0 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca7a777640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca74149c40 con 0x7fca740c25e0 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca7a777640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca741e86b0 con 0x7fca74143590 2024-02-19T14:57:46.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 0x7fca74147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 0x7fca74147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53294/0 (socket says 172.21.15.78:53294) 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 -- 172.21.15.78:0/3213560480 learned_addr learned my addr 172.21.15.78:0/3213560480 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca73fff640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca740c25e0 0x7fca741e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca78d14640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca74143590 0x7fca741461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 -- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca74143590 msgr2=0x7fca741461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca74143590 0x7fca741461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 -- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca740c25e0 msgr2=0x7fca741e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca740c25e0 0x7fca741e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 -- 172.21.15.78:0/3213560480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640094d0 con 0x7fca740c29c0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca737fe640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 0x7fca74147550 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fca6800b4d0 tx=0x7fca6800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68010870 con 0x7fca740c29c0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.195+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680109d0 con 0x7fca740c29c0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.196+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca741e8c30 con 0x7fca740c29c0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.196+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68010d80 con 0x7fca740c29c0 2024-02-19T14:57:46.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.196+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca740a5ec0 con 0x7fca740c29c0 2024-02-19T14:57:46.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.196+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca741e8dc0 con 0x7fca740c29c0 2024-02-19T14:57:46.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.197+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fca6800f3c0 con 0x7fca740c29c0 2024-02-19T14:57:46.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.197+0000 7fca70ff9640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca6006b840 0x7fca6006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.197+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fca6808a2b0 con 0x7fca740c29c0 2024-02-19T14:57:46.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.197+0000 7fca73fff640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca6006b840 0x7fca6006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.198+0000 7fca73fff640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca6006b840 0x7fca6006dd00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fca64036000 tx=0x7fca6403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.202+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fca68054bd0 con 0x7fca740c29c0 2024-02-19T14:57:46.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.375+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca74074990 con 0x7fca740c29c0 2024-02-19T14:57:46.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.377+0000 7fca70ff9640 1 -- 172.21.15.78:0/3213560480 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16885 (secure 0 0 0) 0x7fca68058290 con 0x7fca740c29c0 2024-02-19T14:57:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:46.380 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":151,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:45.095434+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca6006b840 msgr2=0x7fca6006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca6006b840 0x7fca6006dd00 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fca64036000 tx=0x7fca6403d040 comp rx=0 tx=0).stop 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 msgr2=0x7fca74147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 0x7fca74147550 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fca6800b4d0 tx=0x7fca6800b990 comp rx=0 tx=0).stop 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 shutdown_connections 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fca6006b840 0x7fca6006dd00 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca740c29c0 0x7fca74147550 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca74143590 0x7fca741461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 --2- 172.21.15.78:0/3213560480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca740c25e0 0x7fca741e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 >> 172.21.15.78:0/3213560480 conn(0x7fca7406b270 msgr2=0x7fca740c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:46.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 shutdown_connections 2024-02-19T14:57:46.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.379+0000 7fca7a777640 1 -- 172.21.15.78:0/3213560480 wait complete. 2024-02-19T14:57:46.393 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:57:46.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 Processor -- start 2024-02-19T14:57:46.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 -- start start 2024-02-19T14:57:46.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1044073220 0x7f104406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f104406bae0 0x7f104406f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104406b620 con 0x7f104414d110 2024-02-19T14:57:46.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104406ea00 con 0x7f104406bae0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.493+0000 7f104accd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1044157230 con 0x7f1044073220 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f104406bae0 0x7f104406f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f104406bae0 0x7f104406f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53306/0 (socket says 172.21.15.78:53306) 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 -- 172.21.15.78:0/1948385743 learned_addr learned my addr 172.21.15.78:0/1948385743 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f1043fff640 1 --2- 172.21.15.78:0/1948385743 >> v2:172.21.15.150:3300/0 conn(0x7f1044073220 0x7f104406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 -- 172.21.15.78:0/1948385743 >> v2:172.21.15.150:3300/0 conn(0x7f1044073220 msgr2=0x7f104406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 --2- 172.21.15.78:0/1948385743 >> v2:172.21.15.150:3300/0 conn(0x7f1044073220 0x7f104406ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 -- 172.21.15.78:0/1948385743 >> v1:172.21.15.150:6789/0 conn(0x7f104414d110 legacy=0x7f10440ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f104926a640 1 -- 172.21.15.78:0/1948385743 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044072180 con 0x7f104406bae0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.494+0000 7f1043fff640 1 --2- 172.21.15.78:0/1948385743 >> v2:172.21.15.150:3300/0 conn(0x7f1044073220 0x7f104406ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104926a640 1 --2- 172.21.15.78:0/1948385743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f104406bae0 0x7f104406f360 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f10440c5070 tx=0x7f103c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98faa6fcfbea2498 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f10437fe640 1 -- 172.21.15.78:0/1948385743 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c02c9f0 con 0x7f104406bae0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f10437fe640 1 -- 172.21.15.78:0/1948385743 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c02cb50 con 0x7f104406bae0 2024-02-19T14:57:46.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f10437fe640 1 -- 172.21.15.78:0/1948385743 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c0343f0 con 0x7f104406bae0 2024-02-19T14:57:46.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 -- 172.21.15.78:0/1948385743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f104406bae0 msgr2=0x7f104406f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 --2- 172.21.15.78:0/1948385743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f104406bae0 0x7f104406f360 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f10440c5070 tx=0x7f103c02be50 comp rx=0 tx=0).stop 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 -- 172.21.15.78:0/1948385743 shutdown_connections 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 --2- 172.21.15.78:0/1948385743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f104406bae0 0x7f104406f360 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 --2- 172.21.15.78:0/1948385743 >> v2:172.21.15.150:3300/0 conn(0x7f1044073220 0x7f104406ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 -- 172.21.15.78:0/1948385743 >> 172.21.15.78:0/1948385743 conn(0x7f10440c7590 msgr2=0x7f10440c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 -- 172.21.15.78:0/1948385743 shutdown_connections 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.495+0000 7f104accd640 1 -- 172.21.15.78:0/1948385743 wait complete. 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.496+0000 7f104accd640 1 Processor -- start 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.496+0000 7f104accd640 1 -- start start 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f104accd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f104406bae0 0x7f10441ebd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f104accd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 0x7f10441ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f104accd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f104414d110 0x7f10441eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f104accd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10441ec8b0 con 0x7f1044073220 2024-02-19T14:57:46.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f104accd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10441ea210 con 0x7f104406bae0 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f104accd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10441ec5d0 con 0x7f104414d110 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f1043fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 0x7f10441ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f1043fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 0x7f10441ea6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53308/0 (socket says 172.21.15.78:53308) 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.497+0000 7f1043fff640 1 -- 172.21.15.78:0/3411306907 learned_addr learned my addr 172.21.15.78:0/3411306907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1043fff640 1 -- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f104414d110 msgr2=0x7f10441eabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1043fff640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f104414d110 0x7f10441eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1048a69640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f104406bae0 0x7f10441ebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1043fff640 1 -- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f104406bae0 msgr2=0x7f10441ebd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1043fff640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f104406bae0 0x7f10441ebd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1043fff640 1 -- 172.21.15.78:0/3411306907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c009b30 con 0x7f1044073220 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f1043fff640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 0x7f10441ea6a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f103000c7f0 tx=0x7f103000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030007a30 con 0x7f1044073220 2024-02-19T14:57:46.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1030007b90 con 0x7f1044073220 2024-02-19T14:57:46.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103000ce40 con 0x7f1044073220 2024-02-19T14:57:46.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10441e9bc0 con 0x7f1044073220 2024-02-19T14:57:46.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.498+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10441ec2d0 con 0x7f1044073220 2024-02-19T14:57:46.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.499+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10441e9a20 con 0x7f1044073220 2024-02-19T14:57:46.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.499+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f10300113f0 con 0x7f1044073220 2024-02-19T14:57:46.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.500+0000 7f10417fa640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f102806b840 0x7f102806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.500+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f103008a5d0 con 0x7f1044073220 2024-02-19T14:57:46.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.500+0000 7f1048a69640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f102806b840 0x7f102806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.500+0000 7f1048a69640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f102806b840 0x7f102806dd00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1038003d90 tx=0x7f1038009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.504+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f10441e9a20 con 0x7f1044073220 2024-02-19T14:57:46.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.712+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f10441e9a20 con 0x7f1044073220 2024-02-19T14:57:46.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.767+0000 7f10417fa640 1 -- 172.21.15.78:0/3411306907 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v131) v1 ==== 118+0+0 (secure 0 0 0) 0x7f10441e9a20 con 0x7f1044073220 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f102806b840 msgr2=0x7f102806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f102806b840 0x7f102806dd00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1038003d90 tx=0x7f1038009040 comp rx=0 tx=0).stop 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 msgr2=0x7f10441ea6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 0x7f10441ea6a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f103000c7f0 tx=0x7f103000cbf0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 shutdown_connections 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f102806b840 0x7f102806dd00 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1044073220 0x7f10441ea6a0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f104414d110 0x7f10441eabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 --2- 172.21.15.78:0/3411306907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f104406bae0 0x7f10441ebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 >> 172.21.15.78:0/3411306907 conn(0x7f10440c7590 msgr2=0x7f10440c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:46.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 shutdown_connections 2024-02-19T14:57:46.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.769+0000 7f104accd640 1 -- 172.21.15.78:0/3411306907 wait complete. 2024-02-19T14:57:46.784 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:57:46.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:57:46.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.885+0000 7f7a9b5c9640 1 Processor -- start 2024-02-19T14:57:46.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.885+0000 7f7a9b5c9640 1 -- start start 2024-02-19T14:57:46.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a9b5c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 0x7f7a94159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a9b5c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a940c6a60 con 0x7f7a9406d000 2024-02-19T14:57:46.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a9b5c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a941554b0 con 0x7f7a940c6560 2024-02-19T14:57:46.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a9b5c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94155790 con 0x7f7a94156d60 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a99365640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7a9406d000 0x7f7a94156650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40910/0 (socket says 172.21.15.78:40910) 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a99365640 1 -- 172.21.15.78:0/2247797404 learned_addr learned my addr 172.21.15.78:0/2247797404 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a99b66640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 0x7f7a94159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185635604 0 0) 0x7f7a940c6a60 con 0x7f7a9406d000 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a80003520 con 0x7f7a9406d000 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631287392 0 0) 0x7f7a941554b0 con 0x7f7a940c6560 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a940c6a60 con 0x7f7a940c6560 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179088553 0 0) 0x7f7a80003520 con 0x7f7a9406d000 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 msgr2=0x7f7a94159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 0x7f7a94159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 >> v1:172.21.15.150:6789/0 conn(0x7f7a940c6560 legacy=0x7f7a9415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94155a70 con 0x7f7a9406d000 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.886+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f7a90001670 con 0x7f7a9406d000 2024-02-19T14:57:46.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 >> v1:172.21.15.78:6789/0 conn(0x7f7a9406d000 legacy=0x7f7a94156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a80003b30 0x7f7a80003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 0x7f7a800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a80004d00 0x7f7a80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94155790 con 0x7f7a80003b30 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8000ba80 con 0x7f7a80004d00 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8000bc40 con 0x7f7a94156d60 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99365640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a80003b30 0x7f7a80003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99b66640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a80004d00 0x7f7a80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a98b64640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 0x7f7a800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99365640 1 -- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 msgr2=0x7f7a800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99365640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a94156d60 0x7f7a800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99365640 1 -- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a80004d00 msgr2=0x7f7a80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99365640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a80004d00 0x7f7a80005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.887+0000 7f7a99365640 1 -- 172.21.15.78:0/2247797404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80003520 con 0x7f7a80003b30 2024-02-19T14:57:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a99365640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a80003b30 0x7f7a80003f30 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f7a90007fd0 tx=0x7f7a800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da3def1ec75ae9a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a90005ad0 con 0x7f7a80003b30 2024-02-19T14:57:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a90005c50 con 0x7f7a80003b30 2024-02-19T14:57:46.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a8bfff640 1 -- 172.21.15.78:0/2247797404 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9002c850 con 0x7f7a80003b30 2024-02-19T14:57:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/2247797404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a80003b30 msgr2=0x7f7a80003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/2247797404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a80003b30 0x7f7a80003f30 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f7a90007fd0 tx=0x7f7a800057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a99b66640 1 -- 172.21.15.78:0/2247797404 reap_dead start 2024-02-19T14:57:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/2247797404 shutdown_connections 2024-02-19T14:57:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/2247797404 >> 172.21.15.78:0/2247797404 conn(0x7f7a940ca870 msgr2=0x7f7a940cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/2247797404 shutdown_connections 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.888+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/2247797404 wait complete. 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.889+0000 7f7a9b5c9640 1 Processor -- start 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 -- start start 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a80003b30 0x7f7a940c0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 0x7f7a940c0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a9414bcb0 0x7f7a941eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941ec360 con 0x7f7a9414c200 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9414d7a0 con 0x7f7a9414bcb0 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a9b5c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941ef3a0 con 0x7f7a80003b30 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a98b64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 0x7f7a940c0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a98b64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 0x7f7a940c0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53324/0 (socket says 172.21.15.78:53324) 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a98b64640 1 -- 172.21.15.78:0/1573212868 learned_addr learned my addr 172.21.15.78:0/1573212868 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.890+0000 7f7a99365640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a80003b30 0x7f7a940c0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a99b66640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a9414bcb0 0x7f7a941eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a98b64640 1 -- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a80003b30 msgr2=0x7f7a940c0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a98b64640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a80003b30 0x7f7a940c0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a98b64640 1 -- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a9414bcb0 msgr2=0x7f7a941eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a98b64640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a9414bcb0 0x7f7a941eaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a98b64640 1 -- 172.21.15.78:0/1573212868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94156110 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a99365640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a80003b30 0x7f7a940c0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a98b64640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 0x7f7a940c0990 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400e5d0 tx=0x7f7a8400ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840053a0 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84005500 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a941dfa80 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84003880 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.891+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a941df430 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.892+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a94074990 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.893+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7a840039e0 con 0x7f7a9414c200 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.893+0000 7f7a89ffb640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a7806b840 0x7f7a7806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.893+0000 7f7a99365640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a7806b840 0x7f7a7806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:46.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.893+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7a84089ef0 con 0x7f7a9414c200 2024-02-19T14:57:46.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.894+0000 7f7a99365640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a7806b840 0x7f7a7806dd00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7a90034000 tx=0x7f7a90007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:46.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:46.897+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7a84054810 con 0x7f7a9414c200 2024-02-19T14:57:47.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.112+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7a940b0960 con 0x7f7a9414c200 2024-02-19T14:57:47.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.776+0000 7f7a89ffb640 1 -- 172.21.15.78:0/1573212868 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v132) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7a84057ed0 con 0x7f7a9414c200 2024-02-19T14:57:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.778+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a7806b840 msgr2=0x7f7a7806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:47.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.778+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a7806b840 0x7f7a7806dd00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7a90034000 tx=0x7f7a90007000 comp rx=0 tx=0).stop 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 msgr2=0x7f7a940c0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 0x7f7a940c0990 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400e5d0 tx=0x7f7a8400ea90 comp rx=0 tx=0).stop 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 shutdown_connections 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a7806b840 0x7f7a7806dd00 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a9414c200 0x7f7a940c0990 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a9414bcb0 0x7f7a941eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 --2- 172.21.15.78:0/1573212868 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a80003b30 0x7f7a940c0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 >> 172.21.15.78:0/1573212868 conn(0x7f7a940ca870 msgr2=0x7f7a94156600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 shutdown_connections 2024-02-19T14:57:47.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.779+0000 7f7a9b5c9640 1 -- 172.21.15.78:0/1573212868 wait complete. 2024-02-19T14:57:47.792 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:57:47.792 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.891+0000 7fb296076640 1 Processor -- start 2024-02-19T14:57:47.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.891+0000 7fb296076640 1 -- start start 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb296076640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2900c78a0 0x7fb2900c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb296076640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb29014ef50 0x7fb29014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb296076640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2900c35c0 con 0x7fb2900727d0 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb296076640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29014f920 con 0x7fb29014ef50 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb296076640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2900c38a0 con 0x7fb2900c78a0 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb29014ef50 0x7fb29014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb29014ef50 0x7fb29014f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46868/0 (socket says 172.21.15.78:46868) 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 -- 172.21.15.78:0/3881741003 learned_addr learned my addr 172.21.15.78:0/3881741003 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 -- 172.21.15.78:0/3881741003 >> v2:172.21.15.150:3300/0 conn(0x7fb2900c78a0 msgr2=0x7fb2900c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 --2- 172.21.15.78:0/3881741003 >> v2:172.21.15.150:3300/0 conn(0x7fb2900c78a0 0x7fb2900c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 -- 172.21.15.78:0/3881741003 >> v1:172.21.15.150:6789/0 conn(0x7fb2900727d0 legacy=0x7fb290072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.892+0000 7fb28effd640 1 -- 172.21.15.78:0/3881741003 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2900c3b80 con 0x7fb29014ef50 2024-02-19T14:57:47.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.893+0000 7fb28effd640 1 --2- 172.21.15.78:0/3881741003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb29014ef50 0x7fb29014f3e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fb2900c80d0 tx=0x7fb28402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afcdf9db387e19a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.893+0000 7fb28e7fc640 1 -- 172.21.15.78:0/3881741003 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284031870 con 0x7fb29014ef50 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.893+0000 7fb28e7fc640 1 -- 172.21.15.78:0/3881741003 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2840319d0 con 0x7fb29014ef50 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.893+0000 7fb28e7fc640 1 -- 172.21.15.78:0/3881741003 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284031d60 con 0x7fb29014ef50 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 -- 172.21.15.78:0/3881741003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb29014ef50 msgr2=0x7fb29014f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 --2- 172.21.15.78:0/3881741003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb29014ef50 0x7fb29014f3e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fb2900c80d0 tx=0x7fb28402b910 comp rx=0 tx=0).stop 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 -- 172.21.15.78:0/3881741003 shutdown_connections 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 --2- 172.21.15.78:0/3881741003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb29014ef50 0x7fb29014f3e0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 --2- 172.21.15.78:0/3881741003 >> v2:172.21.15.150:3300/0 conn(0x7fb2900c78a0 0x7fb2900c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 -- 172.21.15.78:0/3881741003 >> 172.21.15.78:0/3881741003 conn(0x7fb290066e70 msgr2=0x7fb2900a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 -- 172.21.15.78:0/3881741003 shutdown_connections 2024-02-19T14:57:47.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.894+0000 7fb296076640 1 -- 172.21.15.78:0/3881741003 wait complete. 2024-02-19T14:57:47.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.895+0000 7fb296076640 1 Processor -- start 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.895+0000 7fb296076640 1 -- start start 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb296076640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2900727d0 0x7fb290147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb296076640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 0x7fb29015dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb296076640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb29014ef50 0x7fb2901edcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb296076640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb290148d90 con 0x7fb2900c78a0 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb296076640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb290147240 con 0x7fb29014ef50 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb296076640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb290148fe0 con 0x7fb2900727d0 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 0x7fb29015dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 0x7fb29015dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46878/0 (socket says 172.21.15.78:46878) 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 -- 172.21.15.78:0/4161453730 learned_addr learned my addr 172.21.15.78:0/4161453730 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28ffff640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb29014ef50 0x7fb2901edcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28f7fe640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2900727d0 0x7fb290147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:47.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 -- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2900727d0 msgr2=0x7fb290147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2900727d0 0x7fb290147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 -- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb29014ef50 msgr2=0x7fb2901edcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb29014ef50 0x7fb2901edcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 -- 172.21.15.78:0/4161453730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840094d0 con 0x7fb2900c78a0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.896+0000 7fb28effd640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 0x7fb29015dc40 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fb284000c00 tx=0x7fb2840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.897+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28403ad30 con 0x7fb2900c78a0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.897+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28403ae90 con 0x7fb2900c78a0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.897+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284044610 con 0x7fb2900c78a0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.897+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb290147620 con 0x7fb2900c78a0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.897+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29015e180 con 0x7fb2900c78a0 2024-02-19T14:57:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.898+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb290147b40 con 0x7fb2900c78a0 2024-02-19T14:57:47.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.899+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb28404d450 con 0x7fb2900c78a0 2024-02-19T14:57:47.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.899+0000 7fb277fff640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb27c06b840 0x7fb27c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:47.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.899+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb28402d040 con 0x7fb2900c78a0 2024-02-19T14:57:47.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.899+0000 7fb28f7fe640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb27c06b840 0x7fb27c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:47.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.899+0000 7fb28f7fe640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb27c06b840 0x7fb27c06dd00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb280009890 tx=0x7fb280008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:47.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:47.903+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb290147b40 con 0x7fb2900c78a0 2024-02-19T14:57:48.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.112+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb290147b40 con 0x7fb2900c78a0 2024-02-19T14:57:48.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.113+0000 7fb277fff640 1 -- 172.21.15.78:0/4161453730 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+4951 (secure 0 0 0) 0x7fb290147b40 con 0x7fb2900c78a0 2024-02-19T14:57:48.115 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:48.116 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":132,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7768,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":7771,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":8769,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2997633530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2997633530},{"type":"v1","addr":"172.21.15.78:6836","nonce":2997633530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:57:47.113438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/903597448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":903597448},{"type":"v1","addr":"172.21.15.78:6837","nonce":903597448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-02-19T14:57:48.116 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 132 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb27c06b840 msgr2=0x7fb27c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb27c06b840 0x7fb27c06dd00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb280009890 tx=0x7fb280008040 comp rx=0 tx=0).stop 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 msgr2=0x7fb29015dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 0x7fb29015dc40 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fb284000c00 tx=0x7fb2840046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 shutdown_connections 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb27c06b840 0x7fb27c06dd00 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2900c78a0 0x7fb29015dc40 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb29014ef50 0x7fb2901edcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 --2- 172.21.15.78:0/4161453730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2900727d0 0x7fb290147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 >> 172.21.15.78:0/4161453730 conn(0x7fb290066e70 msgr2=0x7fb290150fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 shutdown_connections 2024-02-19T14:57:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.115+0000 7fb296076640 1 -- 172.21.15.78:0/4161453730 wait complete. 2024-02-19T14:57:48.132 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.230+0000 7f3d702ae640 1 Processor -- start 2024-02-19T14:57:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.230+0000 7f3d702ae640 1 -- start start 2024-02-19T14:57:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d702ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 0x7f3d68159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d702ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d68155010 0x7f3d681523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d702ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6806ffa0 con 0x7f3d680c78a0 2024-02-19T14:57:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d702ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6806b720 con 0x7f3d68155010 2024-02-19T14:57:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d702ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6806fc70 con 0x7f3d680c4d80 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6e84b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d68155010 0x7f3d681523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6d849640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 0x7f3d68159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6e04a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3d680c78a0 0x7f3d680c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:54178/0 (socket says 172.21.15.78:54178) 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6d849640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 0x7f3d68159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51398/0 (socket says 172.21.15.78:51398) 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6e04a640 1 -- 172.21.15.78:0/3273510791 learned_addr learned my addr 172.21.15.78:0/3273510791 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6e84b640 1 --2- 172.21.15.78:0/3273510791 >> v2:172.21.15.150:3300/0 conn(0x7f3d68155010 0x7f3d681523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:49578/0 (socket says 172.21.15.78:49578) 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6d849640 1 -- 172.21.15.78:0/3273510791 >> v2:172.21.15.150:3300/0 conn(0x7f3d68155010 msgr2=0x7f3d681523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6d849640 1 --2- 172.21.15.78:0/3273510791 >> v2:172.21.15.150:3300/0 conn(0x7f3d68155010 0x7f3d681523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6d849640 1 -- 172.21.15.78:0/3273510791 >> v1:172.21.15.150:6789/0 conn(0x7f3d680c78a0 legacy=0x7f3d680c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.231+0000 7f3d6d849640 1 -- 172.21.15.78:0/3273510791 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6806b930 con 0x7f3d680c4d80 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.232+0000 7f3d6d849640 1 --2- 172.21.15.78:0/3273510791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 0x7f3d68159010 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3d580097a0 tx=0x7f3d5802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d52845b339526b86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.232+0000 7f3d6d048640 1 -- 172.21.15.78:0/3273510791 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5802e070 con 0x7f3d680c4d80 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.232+0000 7f3d6d048640 1 -- 172.21.15.78:0/3273510791 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d580315a0 con 0x7f3d680c4d80 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.232+0000 7f3d6d048640 1 -- 172.21.15.78:0/3273510791 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58031930 con 0x7f3d680c4d80 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 -- 172.21.15.78:0/3273510791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 msgr2=0x7f3d68159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 --2- 172.21.15.78:0/3273510791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 0x7f3d68159010 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3d580097a0 tx=0x7f3d5802b840 comp rx=0 tx=0).stop 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 -- 172.21.15.78:0/3273510791 shutdown_connections 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 --2- 172.21.15.78:0/3273510791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d680c4d80 0x7f3d68159010 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 --2- 172.21.15.78:0/3273510791 >> v2:172.21.15.150:3300/0 conn(0x7f3d68155010 0x7f3d681523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 -- 172.21.15.78:0/3273510791 >> 172.21.15.78:0/3273510791 conn(0x7f3d68066e70 msgr2=0x7f3d680a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 -- 172.21.15.78:0/3273510791 shutdown_connections 2024-02-19T14:57:48.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.233+0000 7f3d702ae640 1 -- 172.21.15.78:0/3273510791 wait complete. 2024-02-19T14:57:48.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.234+0000 7f3d702ae640 1 Processor -- start 2024-02-19T14:57:48.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.234+0000 7f3d702ae640 1 -- start start 2024-02-19T14:57:48.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d680c4d80 0x7f3d681edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 0x7f3d68148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d68155010 0x7f3d680c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d680c2d20 con 0x7f3d680c78a0 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68157e10 con 0x7f3d680c4d80 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d681e2250 con 0x7f3d68155010 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 0x7f3d68148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 0x7f3d68148aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46886/0 (socket says 172.21.15.78:46886) 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 -- 172.21.15.78:0/1931780189 learned_addr learned my addr 172.21.15.78:0/1931780189 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6e84b640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d68155010 0x7f3d680c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6e04a640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d680c4d80 0x7f3d681edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 -- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d68155010 msgr2=0x7f3d680c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d68155010 0x7f3d680c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 -- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d680c4d80 msgr2=0x7f3d681edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d680c4d80 0x7f3d681edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 -- 172.21.15.78:0/1931780189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d58009400 con 0x7f3d680c78a0 2024-02-19T14:57:48.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6e04a640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d680c4d80 0x7f3d681edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d6d849640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 0x7f3d68148aa0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802be50 tx=0x7f3d580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58032630 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58032790 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d680c3e10 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.235+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58032b40 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.236+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d681e1a40 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.237+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d680c74a0 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.237+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3d58032ca0 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.237+0000 7f3d5effd640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d4c06b810 0x7f3d4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.237+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d580ac250 con 0x7f3d680c78a0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.237+0000 7f3d6e04a640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d4c06b810 0x7f3d4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.238+0000 7f3d6e04a640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d4c06b810 0x7f3d4c06dcd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000f60 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.245+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d58076ae0 con 0x7f3d680c78a0 2024-02-19T14:57:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.423+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d68159310 con 0x7f3d680c78a0 2024-02-19T14:57:48.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.424+0000 7f3d5effd640 1 -- 172.21.15.78:0/1931780189 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16872 (secure 0 0 0) 0x7f3d5807a1a0 con 0x7f3d680c78a0 2024-02-19T14:57:48.427 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":152,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:46.756992+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:48.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d4c06b810 msgr2=0x7f3d4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d4c06b810 0x7f3d4c06dcd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000f60 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 msgr2=0x7f3d68148aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 0x7f3d68148aa0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802be50 tx=0x7f3d580046c0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 shutdown_connections 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d4c06b810 0x7f3d4c06dcd0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d680c78a0 0x7f3d68148aa0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d68155010 0x7f3d680c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 --2- 172.21.15.78:0/1931780189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d680c4d80 0x7f3d681edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 >> 172.21.15.78:0/1931780189 conn(0x7f3d68066e70 msgr2=0x7f3d68152e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 shutdown_connections 2024-02-19T14:57:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.426+0000 7f3d702ae640 1 -- 172.21.15.78:0/1931780189 wait complete. 2024-02-19T14:57:48.439 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:57:48.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:48.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.538+0000 7f410ad6c640 1 Processor -- start 2024-02-19T14:57:48.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.538+0000 7f410ad6c640 1 -- start start 2024-02-19T14:57:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.538+0000 7f410ad6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f41040c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.538+0000 7f410ad6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41041551f0 con 0x7f41040c8e00 2024-02-19T14:57:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.538+0000 7f410ad6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4104155490 con 0x7f41040c5730 2024-02-19T14:57:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f410ad6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104155770 con 0x7f4104152700 2024-02-19T14:57:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f4109309640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f41040c8e00 0x7f410415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58306/0 (socket says 172.21.15.78:58306) 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f4109309640 1 -- 172.21.15.78:0/2714643201 learned_addr learned my addr 172.21.15.78:0/2714643201 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f4108b08640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f41040c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994780639 0 0) 0x7f41041551f0 con 0x7f41040c8e00 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f0003520 con 0x7f41040c8e00 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223949870 0 0) 0x7f40f0003520 con 0x7f41040c8e00 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 msgr2=0x7f41040c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f41040c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 >> v1:172.21.15.150:6789/0 conn(0x7f41040c5730 legacy=0x7f410406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104155a50 con 0x7f41040c8e00 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f4108b08640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f41040c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f40fc0019f0 con 0x7f41040c8e00 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 >> v1:172.21.15.78:6789/0 conn(0x7f41040c8e00 legacy=0x7f410415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40f00039a0 0x7f40f0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f0004350 0x7f40f0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f40f0005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4104155770 con 0x7f40f0004350 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f000b980 con 0x7f40f00039a0 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f000baf0 con 0x7f4104152700 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f4103fff640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f0004350 0x7f40f0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.539+0000 7f4108b08640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40f00039a0 0x7f40f0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4109309640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f40f0005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4103fff640 1 -- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 msgr2=0x7f40f0005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4103fff640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4104152700 0x7f40f0005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4103fff640 1 -- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40f00039a0 msgr2=0x7f40f0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4103fff640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40f00039a0 0x7f40f0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4103fff640 1 -- 172.21.15.78:0/2714643201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4009400 con 0x7f40f0004350 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.540+0000 7f4103fff640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f0004350 0x7f40f0004820 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f40ec001860 tx=0x7f40f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710825675780829b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec033410 con 0x7f40f0004350 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec033590 con 0x7f40f0004350 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f41037fe640 1 -- 172.21.15.78:0/2714643201 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec033940 con 0x7f40f0004350 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f410ad6c640 1 -- 172.21.15.78:0/2714643201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f0004350 msgr2=0x7f40f0004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f410ad6c640 1 --2- 172.21.15.78:0/2714643201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f0004350 0x7f40f0004820 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f40ec001860 tx=0x7f40f00057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f4109309640 1 -- 172.21.15.78:0/2714643201 reap_dead start 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f410ad6c640 1 -- 172.21.15.78:0/2714643201 shutdown_connections 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f410ad6c640 1 -- 172.21.15.78:0/2714643201 >> 172.21.15.78:0/2714643201 conn(0x7f41040cac90 msgr2=0x7f41040cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:48.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f410ad6c640 1 -- 172.21.15.78:0/2714643201 shutdown_connections 2024-02-19T14:57:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.541+0000 7f410ad6c640 1 -- 172.21.15.78:0/2714643201 wait complete. 2024-02-19T14:57:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.542+0000 7f410ad6c640 1 Processor -- start 2024-02-19T14:57:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.542+0000 7f410ad6c640 1 -- start start 2024-02-19T14:57:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f410ad6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41041ee300 0x7f41041ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f410ad6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4104154cf0 0x7f41040ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f410ad6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 0x7f41041ec200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f410ad6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040abcf0 con 0x7f41040be830 2024-02-19T14:57:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f410ad6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040acc80 con 0x7f4104154cf0 2024-02-19T14:57:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f410ad6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41041eedb0 con 0x7f41041ee300 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f4109309640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 0x7f41041ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f4109309640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 0x7f41041ec200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46902/0 (socket says 172.21.15.78:46902) 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f4109309640 1 -- 172.21.15.78:0/3209113910 learned_addr learned my addr 172.21.15.78:0/3209113910 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f4103fff640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4104154cf0 0x7f41040ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f4108b08640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41041ee300 0x7f41041ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.543+0000 7f4109309640 1 -- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41041ee300 msgr2=0x7f41041ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f4109309640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41041ee300 0x7f41041ee6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f4109309640 1 -- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4104154cf0 msgr2=0x7f41040ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f4109309640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4104154cf0 0x7f41040ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f4109309640 1 -- 172.21.15.78:0/3209113910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41041ef050 con 0x7f41040be830 2024-02-19T14:57:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f4103fff640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4104154cf0 0x7f41040ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f4109309640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 0x7f41041ec200 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f40fc004bc0 tx=0x7f40fc00a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc013600 con 0x7f41040be830 2024-02-19T14:57:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc0137a0 con 0x7f41040be830 2024-02-19T14:57:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc013b50 con 0x7f41040be830 2024-02-19T14:57:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41040abf40 con 0x7f41040be830 2024-02-19T14:57:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.544+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41041ec740 con 0x7f41040be830 2024-02-19T14:57:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.545+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41040ac460 con 0x7f41040be830 2024-02-19T14:57:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.545+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f40fc013cb0 con 0x7f41040be830 2024-02-19T14:57:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.546+0000 7f41017fa640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40e406b840 0x7f40e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.546+0000 7f4108b08640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40e406b840 0x7f40e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.546+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f40fc08ae40 con 0x7f41040be830 2024-02-19T14:57:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.546+0000 7f4108b08640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40e406b840 0x7f40e406dd00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009af0 tx=0x7f40f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.554+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f41040ac460 con 0x7f41040be830 2024-02-19T14:57:48.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.729+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4104155f70 con 0x7f40e406b840 2024-02-19T14:57:48.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.743+0000 7f41017fa640 1 -- 172.21.15.78:0/3209113910 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290154 (secure 0 0 0) 0x7f40f40139d0 con 0x7f40e406b840 2024-02-19T14:57:48.750 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:48.751 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40e406b840 msgr2=0x7f40e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40e406b840 0x7f40e406dd00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009af0 tx=0x7f40f4008040 comp rx=0 tx=0).stop 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 msgr2=0x7f41041ec200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 0x7f41041ec200 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f40fc004bc0 tx=0x7f40fc00a6f0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 shutdown_connections 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f40e406b840 0x7f40e406dd00 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41040be830 0x7f41041ec200 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4104154cf0 0x7f41040ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 --2- 172.21.15.78:0/3209113910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41041ee300 0x7f41041ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 >> 172.21.15.78:0/3209113910 conn(0x7f41040cac90 msgr2=0x7f41040c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.746+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 shutdown_connections 2024-02-19T14:57:48.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.747+0000 7f410ad6c640 1 -- 172.21.15.78:0/3209113910 wait complete. 2024-02-19T14:57:48.760 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-02-19T14:57:48.127541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265604,"kb_used_data":16144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709116,"statfs":{"total":773094113280,"available":772822134784,"internally_reserved":0,"allocated":16531456,"data_stored":8598716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.393488"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761978+0000","last_change":"2024-02-19T14:57:45.778368+0000","last_active":"2024-02-19T14:57:46.761978+0000","last_peered":"2024-02-19T14:57:46.761978+0000","last_clean":"2024-02-19T14:57:46.761978+0000","last_became_active":"2024-02-19T14:57:45.777197+0000","last_became_peered":"2024-02-19T14:57:45.777197+0000","last_unstale":"2024-02-19T14:57:46.761978+0000","last_undegraded":"2024-02-19T14:57:46.761978+0000","last_fullsized":"2024-02-19T14:57:46.761978+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:12.001914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565330+0000","last_change":"2024-02-19T14:57:45.774781+0000","last_active":"2024-02-19T14:57:47.565330+0000","last_peered":"2024-02-19T14:57:47.565330+0000","last_clean":"2024-02-19T14:57:47.565330+0000","last_became_active":"2024-02-19T14:57:45.774498+0000","last_became_peered":"2024-02-19T14:57:45.774498+0000","last_unstale":"2024-02-19T14:57:47.565330+0000","last_undegraded":"2024-02-19T14:57:47.565330+0000","last_fullsized":"2024-02-19T14:57:47.565330+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:33.646592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761696+0000","last_change":"2024-02-19T14:57:45.782283+0000","last_active":"2024-02-19T14:57:46.761696+0000","last_peered":"2024-02-19T14:57:46.761696+0000","last_clean":"2024-02-19T14:57:46.761696+0000","last_became_active":"2024-02-19T14:57:45.779080+0000","last_became_peered":"2024-02-19T14:57:45.779080+0000","last_unstale":"2024-02-19T14:57:46.761696+0000","last_undegraded":"2024-02-19T14:57:46.761696+0000","last_fullsized":"2024-02-19T14:57:46.761696+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:26.412051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761778+0000","last_change":"2024-02-19T14:57:45.778032+0000","last_active":"2024-02-19T14:57:46.761778+0000","last_peered":"2024-02-19T14:57:46.761778+0000","last_clean":"2024-02-19T14:57:46.761778+0000","last_became_active":"2024-02-19T14:57:45.777184+0000","last_became_peered":"2024-02-19T14:57:45.777184+0000","last_unstale":"2024-02-19T14:57:46.761778+0000","last_undegraded":"2024-02-19T14:57:46.761778+0000","last_fullsized":"2024-02-19T14:57:46.761778+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:36.503471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762161+0000","last_change":"2024-02-19T14:57:45.777557+0000","last_active":"2024-02-19T14:57:46.762161+0000","last_peered":"2024-02-19T14:57:46.762161+0000","last_clean":"2024-02-19T14:57:46.762161+0000","last_became_active":"2024-02-19T14:57:45.777296+0000","last_became_peered":"2024-02-19T14:57:45.777296+0000","last_unstale":"2024-02-19T14:57:46.762161+0000","last_undegraded":"2024-02-19T14:57:46.762161+0000","last_fullsized":"2024-02-19T14:57:46.762161+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565194+0000","last_change":"2024-02-19T14:57:45.781001+0000","last_active":"2024-02-19T14:57:47.565194+0000","last_peered":"2024-02-19T14:57:47.565194+0000","last_clean":"2024-02-19T14:57:47.565194+0000","last_became_active":"2024-02-19T14:57:45.780476+0000","last_became_peered":"2024-02-19T14:57:45.780476+0000","last_unstale":"2024-02-19T14:57:47.565194+0000","last_undegraded":"2024-02-19T14:57:47.565194+0000","last_fullsized":"2024-02-19T14:57:47.565194+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761452+0000","last_change":"2024-02-19T14:57:45.783602+0000","last_active":"2024-02-19T14:57:46.761452+0000","last_peered":"2024-02-19T14:57:46.761452+0000","last_clean":"2024-02-19T14:57:46.761452+0000","last_became_active":"2024-02-19T14:57:45.783269+0000","last_became_peered":"2024-02-19T14:57:45.783269+0000","last_unstale":"2024-02-19T14:57:46.761452+0000","last_undegraded":"2024-02-19T14:57:46.761452+0000","last_fullsized":"2024-02-19T14:57:46.761452+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761289+0000","last_change":"2024-02-19T14:57:45.763954+0000","last_active":"2024-02-19T14:57:46.761289+0000","last_peered":"2024-02-19T14:57:46.761289+0000","last_clean":"2024-02-19T14:57:46.761289+0000","last_became_active":"2024-02-19T14:57:45.763656+0000","last_became_peered":"2024-02-19T14:57:45.763656+0000","last_unstale":"2024-02-19T14:57:46.761289+0000","last_undegraded":"2024-02-19T14:57:46.761289+0000","last_fullsized":"2024-02-19T14:57:46.761289+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761791+0000","last_change":"2024-02-19T14:57:45.772149+0000","last_active":"2024-02-19T14:57:46.761791+0000","last_peered":"2024-02-19T14:57:46.761791+0000","last_clean":"2024-02-19T14:57:46.761791+0000","last_became_active":"2024-02-19T14:57:45.771811+0000","last_became_peered":"2024-02-19T14:57:45.771811+0000","last_unstale":"2024-02-19T14:57:46.761791+0000","last_undegraded":"2024-02-19T14:57:46.761791+0000","last_fullsized":"2024-02-19T14:57:46.761791+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761674+0000","last_change":"2024-02-19T14:57:45.769091+0000","last_active":"2024-02-19T14:57:46.761674+0000","last_peered":"2024-02-19T14:57:46.761674+0000","last_clean":"2024-02-19T14:57:46.761674+0000","last_became_active":"2024-02-19T14:57:45.768819+0000","last_became_peered":"2024-02-19T14:57:45.768819+0000","last_unstale":"2024-02-19T14:57:46.761674+0000","last_undegraded":"2024-02-19T14:57:46.761674+0000","last_fullsized":"2024-02-19T14:57:46.761674+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565427+0000","last_change":"2024-02-19T14:57:45.771342+0000","last_active":"2024-02-19T14:57:47.565427+0000","last_peered":"2024-02-19T14:57:47.565427+0000","last_clean":"2024-02-19T14:57:47.565427+0000","last_became_active":"2024-02-19T14:57:45.771047+0000","last_became_peered":"2024-02-19T14:57:45.771047+0000","last_unstale":"2024-02-19T14:57:47.565427+0000","last_undegraded":"2024-02-19T14:57:47.565427+0000","last_fullsized":"2024-02-19T14:57:47.565427+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565535+0000","last_change":"2024-02-19T14:57:45.783578+0000","last_active":"2024-02-19T14:57:47.565535+0000","last_peered":"2024-02-19T14:57:47.565535+0000","last_clean":"2024-02-19T14:57:47.565535+0000","last_became_active":"2024-02-19T14:57:45.783301+0000","last_became_peered":"2024-02-19T14:57:45.783301+0000","last_unstale":"2024-02-19T14:57:47.565535+0000","last_undegraded":"2024-02-19T14:57:47.565535+0000","last_fullsized":"2024-02-19T14:57:47.565535+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:14.016549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564782+0000","last_change":"2024-02-19T14:57:45.782352+0000","last_active":"2024-02-19T14:57:47.564782+0000","last_peered":"2024-02-19T14:57:47.564782+0000","last_clean":"2024-02-19T14:57:47.564782+0000","last_became_active":"2024-02-19T14:57:45.781489+0000","last_became_peered":"2024-02-19T14:57:45.781489+0000","last_unstale":"2024-02-19T14:57:47.564782+0000","last_undegraded":"2024-02-19T14:57:47.564782+0000","last_fullsized":"2024-02-19T14:57:47.564782+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:33.948498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761963+0000","last_change":"2024-02-19T14:57:45.776389+0000","last_active":"2024-02-19T14:57:46.761963+0000","last_peered":"2024-02-19T14:57:46.761963+0000","last_clean":"2024-02-19T14:57:46.761963+0000","last_became_active":"2024-02-19T14:57:45.776133+0000","last_became_peered":"2024-02-19T14:57:45.776133+0000","last_unstale":"2024-02-19T14:57:46.761963+0000","last_undegraded":"2024-02-19T14:57:46.761963+0000","last_fullsized":"2024-02-19T14:57:46.761963+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565013+0000","last_change":"2024-02-19T14:57:45.774486+0000","last_active":"2024-02-19T14:57:47.565013+0000","last_peered":"2024-02-19T14:57:47.565013+0000","last_clean":"2024-02-19T14:57:47.565013+0000","last_became_active":"2024-02-19T14:57:45.773909+0000","last_became_peered":"2024-02-19T14:57:45.773909+0000","last_unstale":"2024-02-19T14:57:47.565013+0000","last_undegraded":"2024-02-19T14:57:47.565013+0000","last_fullsized":"2024-02-19T14:57:47.565013+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:24.066471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761271+0000","last_change":"2024-02-19T14:57:45.776654+0000","last_active":"2024-02-19T14:57:46.761271+0000","last_peered":"2024-02-19T14:57:46.761271+0000","last_clean":"2024-02-19T14:57:46.761271+0000","last_became_active":"2024-02-19T14:57:45.776192+0000","last_became_peered":"2024-02-19T14:57:45.776192+0000","last_unstale":"2024-02-19T14:57:46.761271+0000","last_undegraded":"2024-02-19T14:57:46.761271+0000","last_fullsized":"2024-02-19T14:57:46.761271+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:14:34.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761933+0000","last_change":"2024-02-19T14:57:45.784872+0000","last_active":"2024-02-19T14:57:46.761933+0000","last_peered":"2024-02-19T14:57:46.761933+0000","last_clean":"2024-02-19T14:57:46.761933+0000","last_became_active":"2024-02-19T14:57:45.784602+0000","last_became_peered":"2024-02-19T14:57:45.784602+0000","last_unstale":"2024-02-19T14:57:46.761933+0000","last_undegraded":"2024-02-19T14:57:46.761933+0000","last_fullsized":"2024-02-19T14:57:46.761933+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566330+0000","last_change":"2024-02-19T14:57:45.771733+0000","last_active":"2024-02-19T14:57:47.566330+0000","last_peered":"2024-02-19T14:57:47.566330+0000","last_clean":"2024-02-19T14:57:47.566330+0000","last_became_active":"2024-02-19T14:57:45.771454+0000","last_became_peered":"2024-02-19T14:57:45.771454+0000","last_unstale":"2024-02-19T14:57:47.566330+0000","last_undegraded":"2024-02-19T14:57:47.566330+0000","last_fullsized":"2024-02-19T14:57:47.566330+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:52.644495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565582+0000","last_change":"2024-02-19T14:57:45.772926+0000","last_active":"2024-02-19T14:57:47.565582+0000","last_peered":"2024-02-19T14:57:47.565582+0000","last_clean":"2024-02-19T14:57:47.565582+0000","last_became_active":"2024-02-19T14:57:45.772628+0000","last_became_peered":"2024-02-19T14:57:45.772628+0000","last_unstale":"2024-02-19T14:57:47.565582+0000","last_undegraded":"2024-02-19T14:57:47.565582+0000","last_fullsized":"2024-02-19T14:57:47.565582+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564207+0000","last_change":"2024-02-19T14:57:45.774821+0000","last_active":"2024-02-19T14:57:47.564207+0000","last_peered":"2024-02-19T14:57:47.564207+0000","last_clean":"2024-02-19T14:57:47.564207+0000","last_became_active":"2024-02-19T14:57:45.774541+0000","last_became_peered":"2024-02-19T14:57:45.774541+0000","last_unstale":"2024-02-19T14:57:47.564207+0000","last_undegraded":"2024-02-19T14:57:47.564207+0000","last_fullsized":"2024-02-19T14:57:47.564207+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:34.280659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564907+0000","last_change":"2024-02-19T14:57:45.784067+0000","last_active":"2024-02-19T14:57:47.564907+0000","last_peered":"2024-02-19T14:57:47.564907+0000","last_clean":"2024-02-19T14:57:47.564907+0000","last_became_active":"2024-02-19T14:57:45.783769+0000","last_became_peered":"2024-02-19T14:57:45.783769+0000","last_unstale":"2024-02-19T14:57:47.564907+0000","last_undegraded":"2024-02-19T14:57:47.564907+0000","last_fullsized":"2024-02-19T14:57:47.564907+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:37.778199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565392+0000","last_change":"2024-02-19T14:57:45.776863+0000","last_active":"2024-02-19T14:57:47.565392+0000","last_peered":"2024-02-19T14:57:47.565392+0000","last_clean":"2024-02-19T14:57:47.565392+0000","last_became_active":"2024-02-19T14:57:45.776525+0000","last_became_peered":"2024-02-19T14:57:45.776525+0000","last_unstale":"2024-02-19T14:57:47.565392+0000","last_undegraded":"2024-02-19T14:57:47.565392+0000","last_fullsized":"2024-02-19T14:57:47.565392+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:29.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761800+0000","last_change":"2024-02-19T14:57:45.773937+0000","last_active":"2024-02-19T14:57:46.761800+0000","last_peered":"2024-02-19T14:57:46.761800+0000","last_clean":"2024-02-19T14:57:46.761800+0000","last_became_active":"2024-02-19T14:57:45.773657+0000","last_became_peered":"2024-02-19T14:57:45.773657+0000","last_unstale":"2024-02-19T14:57:46.761800+0000","last_undegraded":"2024-02-19T14:57:46.761800+0000","last_fullsized":"2024-02-19T14:57:46.761800+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761357+0000","last_change":"2024-02-19T14:57:45.776452+0000","last_active":"2024-02-19T14:57:46.761357+0000","last_peered":"2024-02-19T14:57:46.761357+0000","last_clean":"2024-02-19T14:57:46.761357+0000","last_became_active":"2024-02-19T14:57:45.775815+0000","last_became_peered":"2024-02-19T14:57:45.775815+0000","last_unstale":"2024-02-19T14:57:46.761357+0000","last_undegraded":"2024-02-19T14:57:46.761357+0000","last_fullsized":"2024-02-19T14:57:46.761357+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.339211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565261+0000","last_change":"2024-02-19T14:57:45.773952+0000","last_active":"2024-02-19T14:57:47.565261+0000","last_peered":"2024-02-19T14:57:47.565261+0000","last_clean":"2024-02-19T14:57:47.565261+0000","last_became_active":"2024-02-19T14:57:45.773492+0000","last_became_peered":"2024-02-19T14:57:45.773492+0000","last_unstale":"2024-02-19T14:57:47.565261+0000","last_undegraded":"2024-02-19T14:57:47.565261+0000","last_fullsized":"2024-02-19T14:57:47.565261+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:31.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564769+0000","last_change":"2024-02-19T14:57:45.781678+0000","last_active":"2024-02-19T14:57:47.564769+0000","last_peered":"2024-02-19T14:57:47.564769+0000","last_clean":"2024-02-19T14:57:47.564769+0000","last_became_active":"2024-02-19T14:57:45.781402+0000","last_became_peered":"2024-02-19T14:57:45.781402+0000","last_unstale":"2024-02-19T14:57:47.564769+0000","last_undegraded":"2024-02-19T14:57:47.564769+0000","last_fullsized":"2024-02-19T14:57:47.564769+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:39.684791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565226+0000","last_change":"2024-02-19T14:57:45.779802+0000","last_active":"2024-02-19T14:57:47.565226+0000","last_peered":"2024-02-19T14:57:47.565226+0000","last_clean":"2024-02-19T14:57:47.565226+0000","last_became_active":"2024-02-19T14:57:45.779534+0000","last_became_peered":"2024-02-19T14:57:45.779534+0000","last_unstale":"2024-02-19T14:57:47.565226+0000","last_undegraded":"2024-02-19T14:57:47.565226+0000","last_fullsized":"2024-02-19T14:57:47.565226+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564977+0000","last_change":"2024-02-19T14:57:45.785669+0000","last_active":"2024-02-19T14:57:47.564977+0000","last_peered":"2024-02-19T14:57:47.564977+0000","last_clean":"2024-02-19T14:57:47.564977+0000","last_became_active":"2024-02-19T14:57:45.785083+0000","last_became_peered":"2024-02-19T14:57:45.785083+0000","last_unstale":"2024-02-19T14:57:47.564977+0000","last_undegraded":"2024-02-19T14:57:47.564977+0000","last_fullsized":"2024-02-19T14:57:47.564977+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:00.318619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565670+0000","last_change":"2024-02-19T14:57:45.776205+0000","last_active":"2024-02-19T14:57:47.565670+0000","last_peered":"2024-02-19T14:57:47.565670+0000","last_clean":"2024-02-19T14:57:47.565670+0000","last_became_active":"2024-02-19T14:57:45.775407+0000","last_became_peered":"2024-02-19T14:57:45.775407+0000","last_unstale":"2024-02-19T14:57:47.565670+0000","last_undegraded":"2024-02-19T14:57:47.565670+0000","last_fullsized":"2024-02-19T14:57:47.565670+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:37.658842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566097+0000","last_change":"2024-02-19T14:57:45.774589+0000","last_active":"2024-02-19T14:57:47.566097+0000","last_peered":"2024-02-19T14:57:47.566097+0000","last_clean":"2024-02-19T14:57:47.566097+0000","last_became_active":"2024-02-19T14:57:45.774344+0000","last_became_peered":"2024-02-19T14:57:45.774344+0000","last_unstale":"2024-02-19T14:57:47.566097+0000","last_undegraded":"2024-02-19T14:57:47.566097+0000","last_fullsized":"2024-02-19T14:57:47.566097+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:04.979975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565186+0000","last_change":"2024-02-19T14:57:45.774703+0000","last_active":"2024-02-19T14:57:47.565186+0000","last_peered":"2024-02-19T14:57:47.565186+0000","last_clean":"2024-02-19T14:57:47.565186+0000","last_became_active":"2024-02-19T14:57:45.773430+0000","last_became_peered":"2024-02-19T14:57:45.773430+0000","last_unstale":"2024-02-19T14:57:47.565186+0000","last_undegraded":"2024-02-19T14:57:47.565186+0000","last_fullsized":"2024-02-19T14:57:47.565186+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:56.911480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565010+0000","last_change":"2024-02-19T14:57:45.780128+0000","last_active":"2024-02-19T14:57:47.565010+0000","last_peered":"2024-02-19T14:57:47.565010+0000","last_clean":"2024-02-19T14:57:47.565010+0000","last_became_active":"2024-02-19T14:57:45.779825+0000","last_became_peered":"2024-02-19T14:57:45.779825+0000","last_unstale":"2024-02-19T14:57:47.565010+0000","last_undegraded":"2024-02-19T14:57:47.565010+0000","last_fullsized":"2024-02-19T14:57:47.565010+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:26.050909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565737+0000","last_change":"2024-02-19T14:57:45.781182+0000","last_active":"2024-02-19T14:57:47.565737+0000","last_peered":"2024-02-19T14:57:47.565737+0000","last_clean":"2024-02-19T14:57:47.565737+0000","last_became_active":"2024-02-19T14:57:45.780764+0000","last_became_peered":"2024-02-19T14:57:45.780764+0000","last_unstale":"2024-02-19T14:57:47.565737+0000","last_undegraded":"2024-02-19T14:57:47.565737+0000","last_fullsized":"2024-02-19T14:57:47.565737+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:23.510855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762101+0000","last_change":"2024-02-19T14:57:44.748896+0000","last_active":"2024-02-19T14:57:46.762101+0000","last_peered":"2024-02-19T14:57:46.762101+0000","last_clean":"2024-02-19T14:57:46.762101+0000","last_became_active":"2024-02-19T14:57:44.748624+0000","last_became_peered":"2024-02-19T14:57:44.748624+0000","last_unstale":"2024-02-19T14:57:46.762101+0000","last_undegraded":"2024-02-19T14:57:46.762101+0000","last_fullsized":"2024-02-19T14:57:46.762101+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:53:27.719470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564328+0000","last_change":"2024-02-19T14:57:45.778729+0000","last_active":"2024-02-19T14:57:47.564328+0000","last_peered":"2024-02-19T14:57:47.564328+0000","last_clean":"2024-02-19T14:57:47.564328+0000","last_became_active":"2024-02-19T14:57:45.778460+0000","last_became_peered":"2024-02-19T14:57:45.778460+0000","last_unstale":"2024-02-19T14:57:47.564328+0000","last_undegraded":"2024-02-19T14:57:47.564328+0000","last_fullsized":"2024-02-19T14:57:47.564328+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:46.115236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762297+0000","last_change":"2024-02-19T14:57:44.759512+0000","last_active":"2024-02-19T14:57:46.762297+0000","last_peered":"2024-02-19T14:57:46.762297+0000","last_clean":"2024-02-19T14:57:46.762297+0000","last_became_active":"2024-02-19T14:57:44.759251+0000","last_became_peered":"2024-02-19T14:57:44.759251+0000","last_unstale":"2024-02-19T14:57:46.762297+0000","last_undegraded":"2024-02-19T14:57:46.762297+0000","last_fullsized":"2024-02-19T14:57:46.762297+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:23.064686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762006+0000","last_change":"2024-02-19T14:57:45.775749+0000","last_active":"2024-02-19T14:57:46.762006+0000","last_peered":"2024-02-19T14:57:46.762006+0000","last_clean":"2024-02-19T14:57:46.762006+0000","last_became_active":"2024-02-19T14:57:45.775481+0000","last_became_peered":"2024-02-19T14:57:45.775481+0000","last_unstale":"2024-02-19T14:57:46.762006+0000","last_undegraded":"2024-02-19T14:57:46.762006+0000","last_fullsized":"2024-02-19T14:57:46.762006+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565702+0000","last_change":"2024-02-19T14:57:44.746085+0000","last_active":"2024-02-19T14:57:47.565702+0000","last_peered":"2024-02-19T14:57:47.565702+0000","last_clean":"2024-02-19T14:57:47.565702+0000","last_became_active":"2024-02-19T14:57:44.745804+0000","last_became_peered":"2024-02-19T14:57:44.745804+0000","last_unstale":"2024-02-19T14:57:47.565702+0000","last_undegraded":"2024-02-19T14:57:47.565702+0000","last_fullsized":"2024-02-19T14:57:47.565702+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565730+0000","last_change":"2024-02-19T14:57:45.784232+0000","last_active":"2024-02-19T14:57:47.565730+0000","last_peered":"2024-02-19T14:57:47.565730+0000","last_clean":"2024-02-19T14:57:47.565730+0000","last_became_active":"2024-02-19T14:57:45.784007+0000","last_became_peered":"2024-02-19T14:57:45.784007+0000","last_unstale":"2024-02-19T14:57:47.565730+0000","last_undegraded":"2024-02-19T14:57:47.565730+0000","last_fullsized":"2024-02-19T14:57:47.565730+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:50.022179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564425+0000","last_change":"2024-02-19T14:57:44.754196+0000","last_active":"2024-02-19T14:57:47.564425+0000","last_peered":"2024-02-19T14:57:47.564425+0000","last_clean":"2024-02-19T14:57:47.564425+0000","last_became_active":"2024-02-19T14:57:44.753938+0000","last_became_peered":"2024-02-19T14:57:44.753938+0000","last_unstale":"2024-02-19T14:57:47.564425+0000","last_undegraded":"2024-02-19T14:57:47.564425+0000","last_fullsized":"2024-02-19T14:57:47.564425+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:17.948844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761382+0000","last_change":"2024-02-19T14:57:45.776534+0000","last_active":"2024-02-19T14:57:46.761382+0000","last_peered":"2024-02-19T14:57:46.761382+0000","last_clean":"2024-02-19T14:57:46.761382+0000","last_became_active":"2024-02-19T14:57:45.776277+0000","last_became_peered":"2024-02-19T14:57:45.776277+0000","last_unstale":"2024-02-19T14:57:46.761382+0000","last_undegraded":"2024-02-19T14:57:46.761382+0000","last_fullsized":"2024-02-19T14:57:46.761382+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761508+0000","last_change":"2024-02-19T14:57:44.759675+0000","last_active":"2024-02-19T14:57:46.761508+0000","last_peered":"2024-02-19T14:57:46.761508+0000","last_clean":"2024-02-19T14:57:46.761508+0000","last_became_active":"2024-02-19T14:57:44.759404+0000","last_became_peered":"2024-02-19T14:57:44.759404+0000","last_unstale":"2024-02-19T14:57:46.761508+0000","last_undegraded":"2024-02-19T14:57:46.761508+0000","last_fullsized":"2024-02-19T14:57:46.761508+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:34.658675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564611+0000","last_change":"2024-02-19T14:57:45.777221+0000","last_active":"2024-02-19T14:57:47.564611+0000","last_peered":"2024-02-19T14:57:47.564611+0000","last_clean":"2024-02-19T14:57:47.564611+0000","last_became_active":"2024-02-19T14:57:45.776887+0000","last_became_peered":"2024-02-19T14:57:45.776887+0000","last_unstale":"2024-02-19T14:57:47.564611+0000","last_undegraded":"2024-02-19T14:57:47.564611+0000","last_fullsized":"2024-02-19T14:57:47.564611+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:43.785509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762088+0000","last_change":"2024-02-19T14:57:44.759410+0000","last_active":"2024-02-19T14:57:46.762088+0000","last_peered":"2024-02-19T14:57:46.762088+0000","last_clean":"2024-02-19T14:57:46.762088+0000","last_became_active":"2024-02-19T14:57:44.759125+0000","last_became_peered":"2024-02-19T14:57:44.759125+0000","last_unstale":"2024-02-19T14:57:46.762088+0000","last_undegraded":"2024-02-19T14:57:46.762088+0000","last_fullsized":"2024-02-19T14:57:46.762088+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:47.742311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564757+0000","last_change":"2024-02-19T14:57:45.778260+0000","last_active":"2024-02-19T14:57:47.564757+0000","last_peered":"2024-02-19T14:57:47.564757+0000","last_clean":"2024-02-19T14:57:47.564757+0000","last_became_active":"2024-02-19T14:57:45.778006+0000","last_became_peered":"2024-02-19T14:57:45.778006+0000","last_unstale":"2024-02-19T14:57:47.564757+0000","last_undegraded":"2024-02-19T14:57:47.564757+0000","last_fullsized":"2024-02-19T14:57:47.564757+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:35.800806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564154+0000","last_change":"2024-02-19T14:57:44.757229+0000","last_active":"2024-02-19T14:57:47.564154+0000","last_peered":"2024-02-19T14:57:47.564154+0000","last_clean":"2024-02-19T14:57:47.564154+0000","last_became_active":"2024-02-19T14:57:44.756966+0000","last_became_peered":"2024-02-19T14:57:44.756966+0000","last_unstale":"2024-02-19T14:57:47.564154+0000","last_undegraded":"2024-02-19T14:57:47.564154+0000","last_fullsized":"2024-02-19T14:57:47.564154+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:12.299965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761984+0000","last_change":"2024-02-19T14:57:45.781313+0000","last_active":"2024-02-19T14:57:46.761984+0000","last_peered":"2024-02-19T14:57:46.761984+0000","last_clean":"2024-02-19T14:57:46.761984+0000","last_became_active":"2024-02-19T14:57:45.778982+0000","last_became_peered":"2024-02-19T14:57:45.778982+0000","last_unstale":"2024-02-19T14:57:46.761984+0000","last_undegraded":"2024-02-19T14:57:46.761984+0000","last_fullsized":"2024-02-19T14:57:46.761984+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:23.251435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761411+0000","last_change":"2024-02-19T14:57:44.757934+0000","last_active":"2024-02-19T14:57:46.761411+0000","last_peered":"2024-02-19T14:57:46.761411+0000","last_clean":"2024-02-19T14:57:46.761411+0000","last_became_active":"2024-02-19T14:57:44.757660+0000","last_became_peered":"2024-02-19T14:57:44.757660+0000","last_unstale":"2024-02-19T14:57:46.761411+0000","last_undegraded":"2024-02-19T14:57:46.761411+0000","last_fullsized":"2024-02-19T14:57:46.761411+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565736+0000","last_change":"2024-02-19T14:57:45.776416+0000","last_active":"2024-02-19T14:57:47.565736+0000","last_peered":"2024-02-19T14:57:47.565736+0000","last_clean":"2024-02-19T14:57:47.565736+0000","last_became_active":"2024-02-19T14:57:45.775916+0000","last_became_peered":"2024-02-19T14:57:45.775916+0000","last_unstale":"2024-02-19T14:57:47.565736+0000","last_undegraded":"2024-02-19T14:57:47.565736+0000","last_fullsized":"2024-02-19T14:57:47.565736+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:33.825283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565133+0000","last_change":"2024-02-19T14:57:44.755422+0000","last_active":"2024-02-19T14:57:47.565133+0000","last_peered":"2024-02-19T14:57:47.565133+0000","last_clean":"2024-02-19T14:57:47.565133+0000","last_became_active":"2024-02-19T14:57:44.755152+0000","last_became_peered":"2024-02-19T14:57:44.755152+0000","last_unstale":"2024-02-19T14:57:47.565133+0000","last_undegraded":"2024-02-19T14:57:47.565133+0000","last_fullsized":"2024-02-19T14:57:47.565133+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:13.575356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564403+0000","last_change":"2024-02-19T14:57:45.781850+0000","last_active":"2024-02-19T14:57:47.564403+0000","last_peered":"2024-02-19T14:57:47.564403+0000","last_clean":"2024-02-19T14:57:47.564403+0000","last_became_active":"2024-02-19T14:57:45.781588+0000","last_became_peered":"2024-02-19T14:57:45.781588+0000","last_unstale":"2024-02-19T14:57:47.564403+0000","last_undegraded":"2024-02-19T14:57:47.564403+0000","last_fullsized":"2024-02-19T14:57:47.564403+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:48.140379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761644+0000","last_change":"2024-02-19T14:57:44.751600+0000","last_active":"2024-02-19T14:57:46.761644+0000","last_peered":"2024-02-19T14:57:46.761644+0000","last_clean":"2024-02-19T14:57:46.761644+0000","last_became_active":"2024-02-19T14:57:44.751311+0000","last_became_peered":"2024-02-19T14:57:44.751311+0000","last_unstale":"2024-02-19T14:57:46.761644+0000","last_undegraded":"2024-02-19T14:57:46.761644+0000","last_fullsized":"2024-02-19T14:57:46.761644+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:50.448796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564381+0000","last_change":"2024-02-19T14:57:45.782664+0000","last_active":"2024-02-19T14:57:47.564381+0000","last_peered":"2024-02-19T14:57:47.564381+0000","last_clean":"2024-02-19T14:57:47.564381+0000","last_became_active":"2024-02-19T14:57:45.782411+0000","last_became_peered":"2024-02-19T14:57:45.782411+0000","last_unstale":"2024-02-19T14:57:47.564381+0000","last_undegraded":"2024-02-19T14:57:47.564381+0000","last_fullsized":"2024-02-19T14:57:47.564381+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:26.461781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565078+0000","last_change":"2024-02-19T14:57:44.756497+0000","last_active":"2024-02-19T14:57:47.565078+0000","last_peered":"2024-02-19T14:57:47.565078+0000","last_clean":"2024-02-19T14:57:47.565078+0000","last_became_active":"2024-02-19T14:57:44.756213+0000","last_became_peered":"2024-02-19T14:57:44.756213+0000","last_unstale":"2024-02-19T14:57:47.565078+0000","last_undegraded":"2024-02-19T14:57:47.565078+0000","last_fullsized":"2024-02-19T14:57:47.565078+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:28.745803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566184+0000","last_change":"2024-02-19T14:57:45.780739+0000","last_active":"2024-02-19T14:57:47.566184+0000","last_peered":"2024-02-19T14:57:47.566184+0000","last_clean":"2024-02-19T14:57:47.566184+0000","last_became_active":"2024-02-19T14:57:45.780414+0000","last_became_peered":"2024-02-19T14:57:45.780414+0000","last_unstale":"2024-02-19T14:57:47.566184+0000","last_undegraded":"2024-02-19T14:57:47.566184+0000","last_fullsized":"2024-02-19T14:57:47.566184+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:25.103177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762347+0000","last_change":"2024-02-19T14:57:44.751458+0000","last_active":"2024-02-19T14:57:46.762347+0000","last_peered":"2024-02-19T14:57:46.762347+0000","last_clean":"2024-02-19T14:57:46.762347+0000","last_became_active":"2024-02-19T14:57:44.751194+0000","last_became_peered":"2024-02-19T14:57:44.751194+0000","last_unstale":"2024-02-19T14:57:46.762347+0000","last_undegraded":"2024-02-19T14:57:46.762347+0000","last_fullsized":"2024-02-19T14:57:46.762347+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:11:44.993761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761640+0000","last_change":"2024-02-19T14:57:45.775378+0000","last_active":"2024-02-19T14:57:46.761640+0000","last_peered":"2024-02-19T14:57:46.761640+0000","last_clean":"2024-02-19T14:57:46.761640+0000","last_became_active":"2024-02-19T14:57:45.774903+0000","last_became_peered":"2024-02-19T14:57:45.774903+0000","last_unstale":"2024-02-19T14:57:46.761640+0000","last_undegraded":"2024-02-19T14:57:46.761640+0000","last_fullsized":"2024-02-19T14:57:46.761640+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564493+0000","last_change":"2024-02-19T14:57:44.748468+0000","last_active":"2024-02-19T14:57:47.564493+0000","last_peered":"2024-02-19T14:57:47.564493+0000","last_clean":"2024-02-19T14:57:47.564493+0000","last_became_active":"2024-02-19T14:57:44.748055+0000","last_became_peered":"2024-02-19T14:57:44.748055+0000","last_unstale":"2024-02-19T14:57:47.564493+0000","last_undegraded":"2024-02-19T14:57:47.564493+0000","last_fullsized":"2024-02-19T14:57:47.564493+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:32.823498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566162+0000","last_change":"2024-02-19T14:57:45.780058+0000","last_active":"2024-02-19T14:57:47.566162+0000","last_peered":"2024-02-19T14:57:47.566162+0000","last_clean":"2024-02-19T14:57:47.566162+0000","last_became_active":"2024-02-19T14:57:45.779771+0000","last_became_peered":"2024-02-19T14:57:45.779771+0000","last_unstale":"2024-02-19T14:57:47.566162+0000","last_undegraded":"2024-02-19T14:57:47.566162+0000","last_fullsized":"2024-02-19T14:57:47.566162+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565311+0000","last_change":"2024-02-19T14:57:44.751398+0000","last_active":"2024-02-19T14:57:47.565311+0000","last_peered":"2024-02-19T14:57:47.565311+0000","last_clean":"2024-02-19T14:57:47.565311+0000","last_became_active":"2024-02-19T14:57:44.751099+0000","last_became_peered":"2024-02-19T14:57:44.751099+0000","last_unstale":"2024-02-19T14:57:47.565311+0000","last_undegraded":"2024-02-19T14:57:47.565311+0000","last_fullsized":"2024-02-19T14:57:47.565311+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:51.642710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564770+0000","last_change":"2024-02-19T14:57:45.783960+0000","last_active":"2024-02-19T14:57:47.564770+0000","last_peered":"2024-02-19T14:57:47.564770+0000","last_clean":"2024-02-19T14:57:47.564770+0000","last_became_active":"2024-02-19T14:57:45.783704+0000","last_became_peered":"2024-02-19T14:57:45.783704+0000","last_unstale":"2024-02-19T14:57:47.564770+0000","last_undegraded":"2024-02-19T14:57:47.564770+0000","last_fullsized":"2024-02-19T14:57:47.564770+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:24.171999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761510+0000","last_change":"2024-02-19T14:57:44.758621+0000","last_active":"2024-02-19T14:57:46.761510+0000","last_peered":"2024-02-19T14:57:46.761510+0000","last_clean":"2024-02-19T14:57:46.761510+0000","last_became_active":"2024-02-19T14:57:44.758296+0000","last_became_peered":"2024-02-19T14:57:44.758296+0000","last_unstale":"2024-02-19T14:57:46.761510+0000","last_undegraded":"2024-02-19T14:57:46.761510+0000","last_fullsized":"2024-02-19T14:57:46.761510+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761872+0000","last_change":"2024-02-19T14:57:45.773816+0000","last_active":"2024-02-19T14:57:46.761872+0000","last_peered":"2024-02-19T14:57:46.761872+0000","last_clean":"2024-02-19T14:57:46.761872+0000","last_became_active":"2024-02-19T14:57:45.773272+0000","last_became_peered":"2024-02-19T14:57:45.773272+0000","last_unstale":"2024-02-19T14:57:46.761872+0000","last_undegraded":"2024-02-19T14:57:46.761872+0000","last_fullsized":"2024-02-19T14:57:46.761872+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565198+0000","last_change":"2024-02-19T14:57:44.751265+0000","last_active":"2024-02-19T14:57:47.565198+0000","last_peered":"2024-02-19T14:57:47.565198+0000","last_clean":"2024-02-19T14:57:47.565198+0000","last_became_active":"2024-02-19T14:57:44.750991+0000","last_became_peered":"2024-02-19T14:57:44.750991+0000","last_unstale":"2024-02-19T14:57:47.565198+0000","last_undegraded":"2024-02-19T14:57:47.565198+0000","last_fullsized":"2024-02-19T14:57:47.565198+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:32.644808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566093+0000","last_change":"2024-02-19T14:57:45.773622+0000","last_active":"2024-02-19T14:57:47.566093+0000","last_peered":"2024-02-19T14:57:47.566093+0000","last_clean":"2024-02-19T14:57:47.566093+0000","last_became_active":"2024-02-19T14:57:45.773349+0000","last_became_peered":"2024-02-19T14:57:45.773349+0000","last_unstale":"2024-02-19T14:57:47.566093+0000","last_undegraded":"2024-02-19T14:57:47.566093+0000","last_fullsized":"2024-02-19T14:57:47.566093+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:01.576910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761815+0000","last_change":"2024-02-19T14:57:44.757425+0000","last_active":"2024-02-19T14:57:46.761815+0000","last_peered":"2024-02-19T14:57:46.761815+0000","last_clean":"2024-02-19T14:57:46.761815+0000","last_became_active":"2024-02-19T14:57:44.757006+0000","last_became_peered":"2024-02-19T14:57:44.757006+0000","last_unstale":"2024-02-19T14:57:46.761815+0000","last_undegraded":"2024-02-19T14:57:46.761815+0000","last_fullsized":"2024-02-19T14:57:46.761815+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:41.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565018+0000","last_change":"2024-02-19T14:57:45.781720+0000","last_active":"2024-02-19T14:57:47.565018+0000","last_peered":"2024-02-19T14:57:47.565018+0000","last_clean":"2024-02-19T14:57:47.565018+0000","last_became_active":"2024-02-19T14:57:45.781425+0000","last_became_peered":"2024-02-19T14:57:45.781425+0000","last_unstale":"2024-02-19T14:57:47.565018+0000","last_undegraded":"2024-02-19T14:57:47.565018+0000","last_fullsized":"2024-02-19T14:57:47.565018+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:36.396204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762308+0000","last_change":"2024-02-19T14:57:44.758828+0000","last_active":"2024-02-19T14:57:46.762308+0000","last_peered":"2024-02-19T14:57:46.762308+0000","last_clean":"2024-02-19T14:57:46.762308+0000","last_became_active":"2024-02-19T14:57:44.758387+0000","last_became_peered":"2024-02-19T14:57:44.758387+0000","last_unstale":"2024-02-19T14:57:46.762308+0000","last_undegraded":"2024-02-19T14:57:46.762308+0000","last_fullsized":"2024-02-19T14:57:46.762308+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565308+0000","last_change":"2024-02-19T14:57:45.780279+0000","last_active":"2024-02-19T14:57:47.565308+0000","last_peered":"2024-02-19T14:57:47.565308+0000","last_clean":"2024-02-19T14:57:47.565308+0000","last_became_active":"2024-02-19T14:57:45.779989+0000","last_became_peered":"2024-02-19T14:57:45.779989+0000","last_unstale":"2024-02-19T14:57:47.565308+0000","last_undegraded":"2024-02-19T14:57:47.565308+0000","last_fullsized":"2024-02-19T14:57:47.565308+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761545+0000","last_change":"2024-02-19T14:57:44.760014+0000","last_active":"2024-02-19T14:57:46.761545+0000","last_peered":"2024-02-19T14:57:46.761545+0000","last_clean":"2024-02-19T14:57:46.761545+0000","last_became_active":"2024-02-19T14:57:44.759749+0000","last_became_peered":"2024-02-19T14:57:44.759749+0000","last_unstale":"2024-02-19T14:57:46.761545+0000","last_undegraded":"2024-02-19T14:57:46.761545+0000","last_fullsized":"2024-02-19T14:57:46.761545+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565021+0000","last_change":"2024-02-19T14:57:45.785469+0000","last_active":"2024-02-19T14:57:47.565021+0000","last_peered":"2024-02-19T14:57:47.565021+0000","last_clean":"2024-02-19T14:57:47.565021+0000","last_became_active":"2024-02-19T14:57:45.784759+0000","last_became_peered":"2024-02-19T14:57:45.784759+0000","last_unstale":"2024-02-19T14:57:47.565021+0000","last_undegraded":"2024-02-19T14:57:47.565021+0000","last_fullsized":"2024-02-19T14:57:47.565021+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:26.131093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762272+0000","last_change":"2024-02-19T14:57:44.752558+0000","last_active":"2024-02-19T14:57:46.762272+0000","last_peered":"2024-02-19T14:57:46.762272+0000","last_clean":"2024-02-19T14:57:46.762272+0000","last_became_active":"2024-02-19T14:57:44.752290+0000","last_became_peered":"2024-02-19T14:57:44.752290+0000","last_unstale":"2024-02-19T14:57:46.762272+0000","last_undegraded":"2024-02-19T14:57:46.762272+0000","last_fullsized":"2024-02-19T14:57:46.762272+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:55.760619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762282+0000","last_change":"2024-02-19T14:57:45.776484+0000","last_active":"2024-02-19T14:57:46.762282+0000","last_peered":"2024-02-19T14:57:46.762282+0000","last_clean":"2024-02-19T14:57:46.762282+0000","last_became_active":"2024-02-19T14:57:45.776095+0000","last_became_peered":"2024-02-19T14:57:45.776095+0000","last_unstale":"2024-02-19T14:57:46.762282+0000","last_undegraded":"2024-02-19T14:57:46.762282+0000","last_fullsized":"2024-02-19T14:57:46.762282+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565130+0000","last_change":"2024-02-19T14:57:44.746247+0000","last_active":"2024-02-19T14:57:47.565130+0000","last_peered":"2024-02-19T14:57:47.565130+0000","last_clean":"2024-02-19T14:57:47.565130+0000","last_became_active":"2024-02-19T14:57:44.745941+0000","last_became_peered":"2024-02-19T14:57:44.745941+0000","last_unstale":"2024-02-19T14:57:47.565130+0000","last_undegraded":"2024-02-19T14:57:47.565130+0000","last_fullsized":"2024-02-19T14:57:47.565130+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762153+0000","last_change":"2024-02-19T14:57:45.766067+0000","last_active":"2024-02-19T14:57:46.762153+0000","last_peered":"2024-02-19T14:57:46.762153+0000","last_clean":"2024-02-19T14:57:46.762153+0000","last_became_active":"2024-02-19T14:57:45.765654+0000","last_became_peered":"2024-02-19T14:57:45.765654+0000","last_unstale":"2024-02-19T14:57:46.762153+0000","last_undegraded":"2024-02-19T14:57:46.762153+0000","last_fullsized":"2024-02-19T14:57:46.762153+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564248+0000","last_change":"2024-02-19T14:57:44.751718+0000","last_active":"2024-02-19T14:57:47.564248+0000","last_peered":"2024-02-19T14:57:47.564248+0000","last_clean":"2024-02-19T14:57:47.564248+0000","last_became_active":"2024-02-19T14:57:44.751403+0000","last_became_peered":"2024-02-19T14:57:44.751403+0000","last_unstale":"2024-02-19T14:57:47.564248+0000","last_undegraded":"2024-02-19T14:57:47.564248+0000","last_fullsized":"2024-02-19T14:57:47.564248+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:59.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566321+0000","last_change":"2024-02-19T14:57:45.775690+0000","last_active":"2024-02-19T14:57:47.566321+0000","last_peered":"2024-02-19T14:57:47.566321+0000","last_clean":"2024-02-19T14:57:47.566321+0000","last_became_active":"2024-02-19T14:57:45.775076+0000","last_became_peered":"2024-02-19T14:57:45.775076+0000","last_unstale":"2024-02-19T14:57:47.566321+0000","last_undegraded":"2024-02-19T14:57:47.566321+0000","last_fullsized":"2024-02-19T14:57:47.566321+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:40.949417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565869+0000","last_change":"2024-02-19T14:57:44.751791+0000","last_active":"2024-02-19T14:57:47.565869+0000","last_peered":"2024-02-19T14:57:47.565869+0000","last_clean":"2024-02-19T14:57:47.565869+0000","last_became_active":"2024-02-19T14:57:44.751480+0000","last_became_peered":"2024-02-19T14:57:44.751480+0000","last_unstale":"2024-02-19T14:57:47.565869+0000","last_undegraded":"2024-02-19T14:57:47.565869+0000","last_fullsized":"2024-02-19T14:57:47.565869+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564982+0000","last_change":"2024-02-19T14:57:45.772415+0000","last_active":"2024-02-19T14:57:47.564982+0000","last_peered":"2024-02-19T14:57:47.564982+0000","last_clean":"2024-02-19T14:57:47.564982+0000","last_became_active":"2024-02-19T14:57:45.771919+0000","last_became_peered":"2024-02-19T14:57:45.771919+0000","last_unstale":"2024-02-19T14:57:47.564982+0000","last_undegraded":"2024-02-19T14:57:47.564982+0000","last_fullsized":"2024-02-19T14:57:47.564982+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:29.983600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762317+0000","last_change":"2024-02-19T14:57:44.758252+0000","last_active":"2024-02-19T14:57:46.762317+0000","last_peered":"2024-02-19T14:57:46.762317+0000","last_clean":"2024-02-19T14:57:46.762317+0000","last_became_active":"2024-02-19T14:57:44.757979+0000","last_became_peered":"2024-02-19T14:57:44.757979+0000","last_unstale":"2024-02-19T14:57:46.762317+0000","last_undegraded":"2024-02-19T14:57:46.762317+0000","last_fullsized":"2024-02-19T14:57:46.762317+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:28.981816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761391+0000","last_change":"2024-02-19T14:57:45.772751+0000","last_active":"2024-02-19T14:57:46.761391+0000","last_peered":"2024-02-19T14:57:46.761391+0000","last_clean":"2024-02-19T14:57:46.761391+0000","last_became_active":"2024-02-19T14:57:45.772506+0000","last_became_peered":"2024-02-19T14:57:45.772506+0000","last_unstale":"2024-02-19T14:57:46.761391+0000","last_undegraded":"2024-02-19T14:57:46.761391+0000","last_fullsized":"2024-02-19T14:57:46.761391+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"152'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.761902+0000","last_change":"2024-02-19T14:57:44.756795+0000","last_active":"2024-02-19T14:57:47.761902+0000","last_peered":"2024-02-19T14:57:47.761902+0000","last_clean":"2024-02-19T14:57:47.761902+0000","last_became_active":"2024-02-19T14:57:44.756166+0000","last_became_peered":"2024-02-19T14:57:44.756166+0000","last_unstale":"2024-02-19T14:57:47.761902+0000","last_undegraded":"2024-02-19T14:57:47.761902+0000","last_fullsized":"2024-02-19T14:57:47.761902+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:13.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564315+0000","last_change":"2024-02-19T14:57:45.774855+0000","last_active":"2024-02-19T14:57:47.564315+0000","last_peered":"2024-02-19T14:57:47.564315+0000","last_clean":"2024-02-19T14:57:47.564315+0000","last_became_active":"2024-02-19T14:57:45.774572+0000","last_became_peered":"2024-02-19T14:57:45.774572+0000","last_unstale":"2024-02-19T14:57:47.564315+0000","last_undegraded":"2024-02-19T14:57:47.564315+0000","last_fullsized":"2024-02-19T14:57:47.564315+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:34.308217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564929+0000","last_change":"2024-02-19T14:57:44.749436+0000","last_active":"2024-02-19T14:57:47.564929+0000","last_peered":"2024-02-19T14:57:47.564929+0000","last_clean":"2024-02-19T14:57:47.564929+0000","last_became_active":"2024-02-19T14:57:44.749153+0000","last_became_peered":"2024-02-19T14:57:44.749153+0000","last_unstale":"2024-02-19T14:57:47.564929+0000","last_undegraded":"2024-02-19T14:57:47.564929+0000","last_fullsized":"2024-02-19T14:57:47.564929+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:36.657057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761396+0000","last_change":"2024-02-19T14:57:45.772279+0000","last_active":"2024-02-19T14:57:46.761396+0000","last_peered":"2024-02-19T14:57:46.761396+0000","last_clean":"2024-02-19T14:57:46.761396+0000","last_became_active":"2024-02-19T14:57:45.771983+0000","last_became_peered":"2024-02-19T14:57:45.771983+0000","last_unstale":"2024-02-19T14:57:46.761396+0000","last_undegraded":"2024-02-19T14:57:46.761396+0000","last_fullsized":"2024-02-19T14:57:46.761396+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564508+0000","last_change":"2024-02-19T14:57:44.758176+0000","last_active":"2024-02-19T14:57:47.564508+0000","last_peered":"2024-02-19T14:57:47.564508+0000","last_clean":"2024-02-19T14:57:47.564508+0000","last_became_active":"2024-02-19T14:57:44.757912+0000","last_became_peered":"2024-02-19T14:57:44.757912+0000","last_unstale":"2024-02-19T14:57:47.564508+0000","last_undegraded":"2024-02-19T14:57:47.564508+0000","last_fullsized":"2024-02-19T14:57:47.564508+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:26.671302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761703+0000","last_change":"2024-02-19T14:57:45.771150+0000","last_active":"2024-02-19T14:57:46.761703+0000","last_peered":"2024-02-19T14:57:46.761703+0000","last_clean":"2024-02-19T14:57:46.761703+0000","last_became_active":"2024-02-19T14:57:45.770740+0000","last_became_peered":"2024-02-19T14:57:45.770740+0000","last_unstale":"2024-02-19T14:57:46.761703+0000","last_undegraded":"2024-02-19T14:57:46.761703+0000","last_fullsized":"2024-02-19T14:57:46.761703+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:57.456060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761372+0000","last_change":"2024-02-19T14:57:44.752321+0000","last_active":"2024-02-19T14:57:46.761372+0000","last_peered":"2024-02-19T14:57:46.761372+0000","last_clean":"2024-02-19T14:57:46.761372+0000","last_became_active":"2024-02-19T14:57:44.752031+0000","last_became_peered":"2024-02-19T14:57:44.752031+0000","last_unstale":"2024-02-19T14:57:46.761372+0000","last_undegraded":"2024-02-19T14:57:46.761372+0000","last_fullsized":"2024-02-19T14:57:46.761372+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:54.123952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565409+0000","last_change":"2024-02-19T14:57:45.784058+0000","last_active":"2024-02-19T14:57:47.565409+0000","last_peered":"2024-02-19T14:57:47.565409+0000","last_clean":"2024-02-19T14:57:47.565409+0000","last_became_active":"2024-02-19T14:57:45.783772+0000","last_became_peered":"2024-02-19T14:57:45.783772+0000","last_unstale":"2024-02-19T14:57:47.565409+0000","last_undegraded":"2024-02-19T14:57:47.565409+0000","last_fullsized":"2024-02-19T14:57:47.565409+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:34.823875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564505+0000","last_change":"2024-02-19T14:57:44.743222+0000","last_active":"2024-02-19T14:57:47.564505+0000","last_peered":"2024-02-19T14:57:47.564505+0000","last_clean":"2024-02-19T14:57:47.564505+0000","last_became_active":"2024-02-19T14:57:44.742946+0000","last_became_peered":"2024-02-19T14:57:44.742946+0000","last_unstale":"2024-02-19T14:57:47.564505+0000","last_undegraded":"2024-02-19T14:57:47.564505+0000","last_fullsized":"2024-02-19T14:57:47.564505+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:22.509070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565019+0000","last_change":"2024-02-19T14:57:45.783349+0000","last_active":"2024-02-19T14:57:47.565019+0000","last_peered":"2024-02-19T14:57:47.565019+0000","last_clean":"2024-02-19T14:57:47.565019+0000","last_became_active":"2024-02-19T14:57:45.783101+0000","last_became_peered":"2024-02-19T14:57:45.783101+0000","last_unstale":"2024-02-19T14:57:47.565019+0000","last_undegraded":"2024-02-19T14:57:47.565019+0000","last_fullsized":"2024-02-19T14:57:47.565019+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:13:09.154323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565433+0000","last_change":"2024-02-19T14:57:44.747418+0000","last_active":"2024-02-19T14:57:47.565433+0000","last_peered":"2024-02-19T14:57:47.565433+0000","last_clean":"2024-02-19T14:57:47.565433+0000","last_became_active":"2024-02-19T14:57:44.746915+0000","last_became_peered":"2024-02-19T14:57:44.746915+0000","last_unstale":"2024-02-19T14:57:47.565433+0000","last_undegraded":"2024-02-19T14:57:47.565433+0000","last_fullsized":"2024-02-19T14:57:47.565433+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564793+0000","last_change":"2024-02-19T14:57:45.774106+0000","last_active":"2024-02-19T14:57:47.564793+0000","last_peered":"2024-02-19T14:57:47.564793+0000","last_clean":"2024-02-19T14:57:47.564793+0000","last_became_active":"2024-02-19T14:57:45.773847+0000","last_became_peered":"2024-02-19T14:57:45.773847+0000","last_unstale":"2024-02-19T14:57:47.564793+0000","last_undegraded":"2024-02-19T14:57:47.564793+0000","last_fullsized":"2024-02-19T14:57:47.564793+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:22.673960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761676+0000","last_change":"2024-02-19T14:57:44.752893+0000","last_active":"2024-02-19T14:57:46.761676+0000","last_peered":"2024-02-19T14:57:46.761676+0000","last_clean":"2024-02-19T14:57:46.761676+0000","last_became_active":"2024-02-19T14:57:44.752617+0000","last_became_peered":"2024-02-19T14:57:44.752617+0000","last_unstale":"2024-02-19T14:57:46.761676+0000","last_undegraded":"2024-02-19T14:57:46.761676+0000","last_fullsized":"2024-02-19T14:57:46.761676+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:45.368123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565305+0000","last_change":"2024-02-19T14:57:45.777456+0000","last_active":"2024-02-19T14:57:47.565305+0000","last_peered":"2024-02-19T14:57:47.565305+0000","last_clean":"2024-02-19T14:57:47.565305+0000","last_became_active":"2024-02-19T14:57:45.776998+0000","last_became_peered":"2024-02-19T14:57:45.776998+0000","last_unstale":"2024-02-19T14:57:47.565305+0000","last_undegraded":"2024-02-19T14:57:47.565305+0000","last_fullsized":"2024-02-19T14:57:47.565305+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:14.577140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564961+0000","last_change":"2024-02-19T14:57:44.757019+0000","last_active":"2024-02-19T14:57:47.564961+0000","last_peered":"2024-02-19T14:57:47.564961+0000","last_clean":"2024-02-19T14:57:47.564961+0000","last_became_active":"2024-02-19T14:57:44.756538+0000","last_became_peered":"2024-02-19T14:57:44.756538+0000","last_unstale":"2024-02-19T14:57:47.564961+0000","last_undegraded":"2024-02-19T14:57:47.564961+0000","last_fullsized":"2024-02-19T14:57:47.564961+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:33.822090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":417,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565826+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:47.565826+0000","last_peered":"2024-02-19T14:57:47.565826+0000","last_clean":"2024-02-19T14:57:47.565826+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:47.565826+0000","last_undegraded":"2024-02-19T14:57:47.565826+0000","last_fullsized":"2024-02-19T14:57:47.565826+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2506752,"data_stored":1518686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.435},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.60199999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.5}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.625}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574923,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1900544,"data_stored":921626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.32200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1921024,"data_stored":928442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.628}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.78900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1929216,"data_stored":929082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.495}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.69599999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.749}]}]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":930534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.255},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.29399999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.372}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.47799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1900544,"data_stored":921626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.379},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.246}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":929930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.375},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.73199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33640,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338200,"statfs":{"total":96636764160,"available":96602316800,"internally_reserved":0,"allocated":2514944,"data_stored":1518790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.224}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.51200000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47499999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.41199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:48.766 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:48.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.899+0000 7fde5bec0640 1 Processor -- start 2024-02-19T14:57:48.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5bec0640 1 -- start start 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5bec0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde54152700 0x7fde540c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5bec0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fde540c5730 0x7fde5406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5bec0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde541551f0 con 0x7fde540c8e00 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5bec0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde54155490 con 0x7fde54152700 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5bec0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde54155770 con 0x7fde540c5730 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde59c5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde54152700 0x7fde540c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde59c5c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fde54152700 0x7fde540c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46918/0 (socket says 172.21.15.78:46918) 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde5a45d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fde540c8e00 0x7fde54158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58318/0 (socket says 172.21.15.78:58318) 2024-02-19T14:57:48.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.900+0000 7fde59c5c640 1 -- 172.21.15.78:0/1091790931 learned_addr learned my addr 172.21.15.78:0/1091790931 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde59c5c640 1 -- 172.21.15.78:0/1091790931 >> v2:172.21.15.150:3300/0 conn(0x7fde540c5730 msgr2=0x7fde5406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde5945b640 1 --2- 172.21.15.78:0/1091790931 >> v2:172.21.15.150:3300/0 conn(0x7fde540c5730 0x7fde5406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde58c5a640 1 -- 172.21.15.78:0/1091790931 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834013678 0 0) 0x7fde541551f0 con 0x7fde540c8e00 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde59c5c640 1 --2- 172.21.15.78:0/1091790931 >> v2:172.21.15.150:3300/0 conn(0x7fde540c5730 0x7fde5406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde59c5c640 1 -- 172.21.15.78:0/1091790931 >> v1:172.21.15.78:6789/0 conn(0x7fde540c8e00 legacy=0x7fde54158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde59c5c640 1 -- 172.21.15.78:0/1091790931 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde54155a50 con 0x7fde54152700 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.901+0000 7fde59c5c640 1 --2- 172.21.15.78:0/1091790931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde54152700 0x7fde540c51f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fde540c7620 tx=0x7fde5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defe066e72bbdcaa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde58c5a640 1 -- 172.21.15.78:0/1091790931 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50031870 con 0x7fde54152700 2024-02-19T14:57:48.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde58c5a640 1 -- 172.21.15.78:0/1091790931 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde500319d0 con 0x7fde54152700 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde58c5a640 1 -- 172.21.15.78:0/1091790931 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde50031d60 con 0x7fde54152700 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 -- 172.21.15.78:0/1091790931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde54152700 msgr2=0x7fde540c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 --2- 172.21.15.78:0/1091790931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde54152700 0x7fde540c51f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fde540c7620 tx=0x7fde5002b910 comp rx=0 tx=0).stop 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 -- 172.21.15.78:0/1091790931 shutdown_connections 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 --2- 172.21.15.78:0/1091790931 >> v2:172.21.15.150:3300/0 conn(0x7fde540c5730 0x7fde5406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 --2- 172.21.15.78:0/1091790931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde54152700 0x7fde540c51f0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 -- 172.21.15.78:0/1091790931 >> 172.21.15.78:0/1091790931 conn(0x7fde540cac90 msgr2=0x7fde540cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 -- 172.21.15.78:0/1091790931 shutdown_connections 2024-02-19T14:57:48.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.902+0000 7fde5bec0640 1 -- 172.21.15.78:0/1091790931 wait complete. 2024-02-19T14:57:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.903+0000 7fde5bec0640 1 Processor -- start 2024-02-19T14:57:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.903+0000 7fde5bec0640 1 -- start start 2024-02-19T14:57:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5bec0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde540c5730 0x7fde54150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5bec0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 0x7fde540afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5bec0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde54152700 0x7fde540b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5bec0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5414b7d0 con 0x7fde540c8e00 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5bec0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde541eaea0 con 0x7fde54152700 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5bec0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde541ea570 con 0x7fde540c5730 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 0x7fde540afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 0x7fde540afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46932/0 (socket says 172.21.15.78:46932) 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 -- 172.21.15.78:0/783833913 learned_addr learned my addr 172.21.15.78:0/783833913 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5a45d640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde54152700 0x7fde540b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde59c5c640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde540c5730 0x7fde54150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 -- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde540c5730 msgr2=0x7fde54150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde540c5730 0x7fde54150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 -- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde54152700 msgr2=0x7fde540b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde54152700 0x7fde540b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.904+0000 7fde5945b640 1 -- 172.21.15.78:0/783833913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde500094d0 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.905+0000 7fde5945b640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 0x7fde540afb40 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c7f0 tx=0x7fde4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.905+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44004220 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.905+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde440043a0 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.905+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde541ea850 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.905+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44007860 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.905+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde541513f0 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.906+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde54074990 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.906+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fde440079c0 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.907+0000 7fde4a7fc640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde3806b840 0x7fde3806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.907+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde4408a170 con 0x7fde540c8e00 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.907+0000 7fde59c5c640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde3806b840 0x7fde3806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.907+0000 7fde59c5c640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde3806b840 0x7fde3806dd00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fde50036000 tx=0x7fde5003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:48.911+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fde44054a10 con 0x7fde540c8e00 2024-02-19T14:57:49.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.079+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fde54155f70 con 0x7fde3806b840 2024-02-19T14:57:49.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.086+0000 7fde4a7fc640 1 -- 172.21.15.78:0/783833913 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290154 (secure 0 0 0) 0x7fde50031bc0 con 0x7fde3806b840 2024-02-19T14:57:49.092 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:49.093 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:49.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde3806b840 msgr2=0x7fde3806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde3806b840 0x7fde3806dd00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fde50036000 tx=0x7fde5003c040 comp rx=0 tx=0).stop 2024-02-19T14:57:49.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 msgr2=0x7fde540afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 0x7fde540afb40 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c7f0 tx=0x7fde4400ccb0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 shutdown_connections 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fde3806b840 0x7fde3806dd00 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde540c8e00 0x7fde540afb40 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde54152700 0x7fde540b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 --2- 172.21.15.78:0/783833913 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde540c5730 0x7fde54150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.089+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 >> 172.21.15.78:0/783833913 conn(0x7fde540cac90 msgr2=0x7fde5414fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.090+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 shutdown_connections 2024-02-19T14:57:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.090+0000 7fde5bec0640 1 -- 172.21.15.78:0/783833913 wait complete. 2024-02-19T14:57:49.103 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-02-19T14:57:48.127541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265604,"kb_used_data":16144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709116,"statfs":{"total":773094113280,"available":772822134784,"internally_reserved":0,"allocated":16531456,"data_stored":8598716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.393488"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761978+0000","last_change":"2024-02-19T14:57:45.778368+0000","last_active":"2024-02-19T14:57:46.761978+0000","last_peered":"2024-02-19T14:57:46.761978+0000","last_clean":"2024-02-19T14:57:46.761978+0000","last_became_active":"2024-02-19T14:57:45.777197+0000","last_became_peered":"2024-02-19T14:57:45.777197+0000","last_unstale":"2024-02-19T14:57:46.761978+0000","last_undegraded":"2024-02-19T14:57:46.761978+0000","last_fullsized":"2024-02-19T14:57:46.761978+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:12.001914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565330+0000","last_change":"2024-02-19T14:57:45.774781+0000","last_active":"2024-02-19T14:57:47.565330+0000","last_peered":"2024-02-19T14:57:47.565330+0000","last_clean":"2024-02-19T14:57:47.565330+0000","last_became_active":"2024-02-19T14:57:45.774498+0000","last_became_peered":"2024-02-19T14:57:45.774498+0000","last_unstale":"2024-02-19T14:57:47.565330+0000","last_undegraded":"2024-02-19T14:57:47.565330+0000","last_fullsized":"2024-02-19T14:57:47.565330+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:33.646592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761696+0000","last_change":"2024-02-19T14:57:45.782283+0000","last_active":"2024-02-19T14:57:46.761696+0000","last_peered":"2024-02-19T14:57:46.761696+0000","last_clean":"2024-02-19T14:57:46.761696+0000","last_became_active":"2024-02-19T14:57:45.779080+0000","last_became_peered":"2024-02-19T14:57:45.779080+0000","last_unstale":"2024-02-19T14:57:46.761696+0000","last_undegraded":"2024-02-19T14:57:46.761696+0000","last_fullsized":"2024-02-19T14:57:46.761696+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:26.412051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761778+0000","last_change":"2024-02-19T14:57:45.778032+0000","last_active":"2024-02-19T14:57:46.761778+0000","last_peered":"2024-02-19T14:57:46.761778+0000","last_clean":"2024-02-19T14:57:46.761778+0000","last_became_active":"2024-02-19T14:57:45.777184+0000","last_became_peered":"2024-02-19T14:57:45.777184+0000","last_unstale":"2024-02-19T14:57:46.761778+0000","last_undegraded":"2024-02-19T14:57:46.761778+0000","last_fullsized":"2024-02-19T14:57:46.761778+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:36.503471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762161+0000","last_change":"2024-02-19T14:57:45.777557+0000","last_active":"2024-02-19T14:57:46.762161+0000","last_peered":"2024-02-19T14:57:46.762161+0000","last_clean":"2024-02-19T14:57:46.762161+0000","last_became_active":"2024-02-19T14:57:45.777296+0000","last_became_peered":"2024-02-19T14:57:45.777296+0000","last_unstale":"2024-02-19T14:57:46.762161+0000","last_undegraded":"2024-02-19T14:57:46.762161+0000","last_fullsized":"2024-02-19T14:57:46.762161+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565194+0000","last_change":"2024-02-19T14:57:45.781001+0000","last_active":"2024-02-19T14:57:47.565194+0000","last_peered":"2024-02-19T14:57:47.565194+0000","last_clean":"2024-02-19T14:57:47.565194+0000","last_became_active":"2024-02-19T14:57:45.780476+0000","last_became_peered":"2024-02-19T14:57:45.780476+0000","last_unstale":"2024-02-19T14:57:47.565194+0000","last_undegraded":"2024-02-19T14:57:47.565194+0000","last_fullsized":"2024-02-19T14:57:47.565194+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761452+0000","last_change":"2024-02-19T14:57:45.783602+0000","last_active":"2024-02-19T14:57:46.761452+0000","last_peered":"2024-02-19T14:57:46.761452+0000","last_clean":"2024-02-19T14:57:46.761452+0000","last_became_active":"2024-02-19T14:57:45.783269+0000","last_became_peered":"2024-02-19T14:57:45.783269+0000","last_unstale":"2024-02-19T14:57:46.761452+0000","last_undegraded":"2024-02-19T14:57:46.761452+0000","last_fullsized":"2024-02-19T14:57:46.761452+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761289+0000","last_change":"2024-02-19T14:57:45.763954+0000","last_active":"2024-02-19T14:57:46.761289+0000","last_peered":"2024-02-19T14:57:46.761289+0000","last_clean":"2024-02-19T14:57:46.761289+0000","last_became_active":"2024-02-19T14:57:45.763656+0000","last_became_peered":"2024-02-19T14:57:45.763656+0000","last_unstale":"2024-02-19T14:57:46.761289+0000","last_undegraded":"2024-02-19T14:57:46.761289+0000","last_fullsized":"2024-02-19T14:57:46.761289+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761791+0000","last_change":"2024-02-19T14:57:45.772149+0000","last_active":"2024-02-19T14:57:46.761791+0000","last_peered":"2024-02-19T14:57:46.761791+0000","last_clean":"2024-02-19T14:57:46.761791+0000","last_became_active":"2024-02-19T14:57:45.771811+0000","last_became_peered":"2024-02-19T14:57:45.771811+0000","last_unstale":"2024-02-19T14:57:46.761791+0000","last_undegraded":"2024-02-19T14:57:46.761791+0000","last_fullsized":"2024-02-19T14:57:46.761791+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761674+0000","last_change":"2024-02-19T14:57:45.769091+0000","last_active":"2024-02-19T14:57:46.761674+0000","last_peered":"2024-02-19T14:57:46.761674+0000","last_clean":"2024-02-19T14:57:46.761674+0000","last_became_active":"2024-02-19T14:57:45.768819+0000","last_became_peered":"2024-02-19T14:57:45.768819+0000","last_unstale":"2024-02-19T14:57:46.761674+0000","last_undegraded":"2024-02-19T14:57:46.761674+0000","last_fullsized":"2024-02-19T14:57:46.761674+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565427+0000","last_change":"2024-02-19T14:57:45.771342+0000","last_active":"2024-02-19T14:57:47.565427+0000","last_peered":"2024-02-19T14:57:47.565427+0000","last_clean":"2024-02-19T14:57:47.565427+0000","last_became_active":"2024-02-19T14:57:45.771047+0000","last_became_peered":"2024-02-19T14:57:45.771047+0000","last_unstale":"2024-02-19T14:57:47.565427+0000","last_undegraded":"2024-02-19T14:57:47.565427+0000","last_fullsized":"2024-02-19T14:57:47.565427+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565535+0000","last_change":"2024-02-19T14:57:45.783578+0000","last_active":"2024-02-19T14:57:47.565535+0000","last_peered":"2024-02-19T14:57:47.565535+0000","last_clean":"2024-02-19T14:57:47.565535+0000","last_became_active":"2024-02-19T14:57:45.783301+0000","last_became_peered":"2024-02-19T14:57:45.783301+0000","last_unstale":"2024-02-19T14:57:47.565535+0000","last_undegraded":"2024-02-19T14:57:47.565535+0000","last_fullsized":"2024-02-19T14:57:47.565535+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:14.016549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564782+0000","last_change":"2024-02-19T14:57:45.782352+0000","last_active":"2024-02-19T14:57:47.564782+0000","last_peered":"2024-02-19T14:57:47.564782+0000","last_clean":"2024-02-19T14:57:47.564782+0000","last_became_active":"2024-02-19T14:57:45.781489+0000","last_became_peered":"2024-02-19T14:57:45.781489+0000","last_unstale":"2024-02-19T14:57:47.564782+0000","last_undegraded":"2024-02-19T14:57:47.564782+0000","last_fullsized":"2024-02-19T14:57:47.564782+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:33.948498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761963+0000","last_change":"2024-02-19T14:57:45.776389+0000","last_active":"2024-02-19T14:57:46.761963+0000","last_peered":"2024-02-19T14:57:46.761963+0000","last_clean":"2024-02-19T14:57:46.761963+0000","last_became_active":"2024-02-19T14:57:45.776133+0000","last_became_peered":"2024-02-19T14:57:45.776133+0000","last_unstale":"2024-02-19T14:57:46.761963+0000","last_undegraded":"2024-02-19T14:57:46.761963+0000","last_fullsized":"2024-02-19T14:57:46.761963+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565013+0000","last_change":"2024-02-19T14:57:45.774486+0000","last_active":"2024-02-19T14:57:47.565013+0000","last_peered":"2024-02-19T14:57:47.565013+0000","last_clean":"2024-02-19T14:57:47.565013+0000","last_became_active":"2024-02-19T14:57:45.773909+0000","last_became_peered":"2024-02-19T14:57:45.773909+0000","last_unstale":"2024-02-19T14:57:47.565013+0000","last_undegraded":"2024-02-19T14:57:47.565013+0000","last_fullsized":"2024-02-19T14:57:47.565013+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:24.066471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761271+0000","last_change":"2024-02-19T14:57:45.776654+0000","last_active":"2024-02-19T14:57:46.761271+0000","last_peered":"2024-02-19T14:57:46.761271+0000","last_clean":"2024-02-19T14:57:46.761271+0000","last_became_active":"2024-02-19T14:57:45.776192+0000","last_became_peered":"2024-02-19T14:57:45.776192+0000","last_unstale":"2024-02-19T14:57:46.761271+0000","last_undegraded":"2024-02-19T14:57:46.761271+0000","last_fullsized":"2024-02-19T14:57:46.761271+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:14:34.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761933+0000","last_change":"2024-02-19T14:57:45.784872+0000","last_active":"2024-02-19T14:57:46.761933+0000","last_peered":"2024-02-19T14:57:46.761933+0000","last_clean":"2024-02-19T14:57:46.761933+0000","last_became_active":"2024-02-19T14:57:45.784602+0000","last_became_peered":"2024-02-19T14:57:45.784602+0000","last_unstale":"2024-02-19T14:57:46.761933+0000","last_undegraded":"2024-02-19T14:57:46.761933+0000","last_fullsized":"2024-02-19T14:57:46.761933+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566330+0000","last_change":"2024-02-19T14:57:45.771733+0000","last_active":"2024-02-19T14:57:47.566330+0000","last_peered":"2024-02-19T14:57:47.566330+0000","last_clean":"2024-02-19T14:57:47.566330+0000","last_became_active":"2024-02-19T14:57:45.771454+0000","last_became_peered":"2024-02-19T14:57:45.771454+0000","last_unstale":"2024-02-19T14:57:47.566330+0000","last_undegraded":"2024-02-19T14:57:47.566330+0000","last_fullsized":"2024-02-19T14:57:47.566330+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:52.644495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565582+0000","last_change":"2024-02-19T14:57:45.772926+0000","last_active":"2024-02-19T14:57:47.565582+0000","last_peered":"2024-02-19T14:57:47.565582+0000","last_clean":"2024-02-19T14:57:47.565582+0000","last_became_active":"2024-02-19T14:57:45.772628+0000","last_became_peered":"2024-02-19T14:57:45.772628+0000","last_unstale":"2024-02-19T14:57:47.565582+0000","last_undegraded":"2024-02-19T14:57:47.565582+0000","last_fullsized":"2024-02-19T14:57:47.565582+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564207+0000","last_change":"2024-02-19T14:57:45.774821+0000","last_active":"2024-02-19T14:57:47.564207+0000","last_peered":"2024-02-19T14:57:47.564207+0000","last_clean":"2024-02-19T14:57:47.564207+0000","last_became_active":"2024-02-19T14:57:45.774541+0000","last_became_peered":"2024-02-19T14:57:45.774541+0000","last_unstale":"2024-02-19T14:57:47.564207+0000","last_undegraded":"2024-02-19T14:57:47.564207+0000","last_fullsized":"2024-02-19T14:57:47.564207+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:34.280659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564907+0000","last_change":"2024-02-19T14:57:45.784067+0000","last_active":"2024-02-19T14:57:47.564907+0000","last_peered":"2024-02-19T14:57:47.564907+0000","last_clean":"2024-02-19T14:57:47.564907+0000","last_became_active":"2024-02-19T14:57:45.783769+0000","last_became_peered":"2024-02-19T14:57:45.783769+0000","last_unstale":"2024-02-19T14:57:47.564907+0000","last_undegraded":"2024-02-19T14:57:47.564907+0000","last_fullsized":"2024-02-19T14:57:47.564907+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:37.778199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565392+0000","last_change":"2024-02-19T14:57:45.776863+0000","last_active":"2024-02-19T14:57:47.565392+0000","last_peered":"2024-02-19T14:57:47.565392+0000","last_clean":"2024-02-19T14:57:47.565392+0000","last_became_active":"2024-02-19T14:57:45.776525+0000","last_became_peered":"2024-02-19T14:57:45.776525+0000","last_unstale":"2024-02-19T14:57:47.565392+0000","last_undegraded":"2024-02-19T14:57:47.565392+0000","last_fullsized":"2024-02-19T14:57:47.565392+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:29.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761800+0000","last_change":"2024-02-19T14:57:45.773937+0000","last_active":"2024-02-19T14:57:46.761800+0000","last_peered":"2024-02-19T14:57:46.761800+0000","last_clean":"2024-02-19T14:57:46.761800+0000","last_became_active":"2024-02-19T14:57:45.773657+0000","last_became_peered":"2024-02-19T14:57:45.773657+0000","last_unstale":"2024-02-19T14:57:46.761800+0000","last_undegraded":"2024-02-19T14:57:46.761800+0000","last_fullsized":"2024-02-19T14:57:46.761800+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761357+0000","last_change":"2024-02-19T14:57:45.776452+0000","last_active":"2024-02-19T14:57:46.761357+0000","last_peered":"2024-02-19T14:57:46.761357+0000","last_clean":"2024-02-19T14:57:46.761357+0000","last_became_active":"2024-02-19T14:57:45.775815+0000","last_became_peered":"2024-02-19T14:57:45.775815+0000","last_unstale":"2024-02-19T14:57:46.761357+0000","last_undegraded":"2024-02-19T14:57:46.761357+0000","last_fullsized":"2024-02-19T14:57:46.761357+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.339211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565261+0000","last_change":"2024-02-19T14:57:45.773952+0000","last_active":"2024-02-19T14:57:47.565261+0000","last_peered":"2024-02-19T14:57:47.565261+0000","last_clean":"2024-02-19T14:57:47.565261+0000","last_became_active":"2024-02-19T14:57:45.773492+0000","last_became_peered":"2024-02-19T14:57:45.773492+0000","last_unstale":"2024-02-19T14:57:47.565261+0000","last_undegraded":"2024-02-19T14:57:47.565261+0000","last_fullsized":"2024-02-19T14:57:47.565261+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:31.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564769+0000","last_change":"2024-02-19T14:57:45.781678+0000","last_active":"2024-02-19T14:57:47.564769+0000","last_peered":"2024-02-19T14:57:47.564769+0000","last_clean":"2024-02-19T14:57:47.564769+0000","last_became_active":"2024-02-19T14:57:45.781402+0000","last_became_peered":"2024-02-19T14:57:45.781402+0000","last_unstale":"2024-02-19T14:57:47.564769+0000","last_undegraded":"2024-02-19T14:57:47.564769+0000","last_fullsized":"2024-02-19T14:57:47.564769+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:39.684791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565226+0000","last_change":"2024-02-19T14:57:45.779802+0000","last_active":"2024-02-19T14:57:47.565226+0000","last_peered":"2024-02-19T14:57:47.565226+0000","last_clean":"2024-02-19T14:57:47.565226+0000","last_became_active":"2024-02-19T14:57:45.779534+0000","last_became_peered":"2024-02-19T14:57:45.779534+0000","last_unstale":"2024-02-19T14:57:47.565226+0000","last_undegraded":"2024-02-19T14:57:47.565226+0000","last_fullsized":"2024-02-19T14:57:47.565226+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564977+0000","last_change":"2024-02-19T14:57:45.785669+0000","last_active":"2024-02-19T14:57:47.564977+0000","last_peered":"2024-02-19T14:57:47.564977+0000","last_clean":"2024-02-19T14:57:47.564977+0000","last_became_active":"2024-02-19T14:57:45.785083+0000","last_became_peered":"2024-02-19T14:57:45.785083+0000","last_unstale":"2024-02-19T14:57:47.564977+0000","last_undegraded":"2024-02-19T14:57:47.564977+0000","last_fullsized":"2024-02-19T14:57:47.564977+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:00.318619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565670+0000","last_change":"2024-02-19T14:57:45.776205+0000","last_active":"2024-02-19T14:57:47.565670+0000","last_peered":"2024-02-19T14:57:47.565670+0000","last_clean":"2024-02-19T14:57:47.565670+0000","last_became_active":"2024-02-19T14:57:45.775407+0000","last_became_peered":"2024-02-19T14:57:45.775407+0000","last_unstale":"2024-02-19T14:57:47.565670+0000","last_undegraded":"2024-02-19T14:57:47.565670+0000","last_fullsized":"2024-02-19T14:57:47.565670+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:37.658842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566097+0000","last_change":"2024-02-19T14:57:45.774589+0000","last_active":"2024-02-19T14:57:47.566097+0000","last_peered":"2024-02-19T14:57:47.566097+0000","last_clean":"2024-02-19T14:57:47.566097+0000","last_became_active":"2024-02-19T14:57:45.774344+0000","last_became_peered":"2024-02-19T14:57:45.774344+0000","last_unstale":"2024-02-19T14:57:47.566097+0000","last_undegraded":"2024-02-19T14:57:47.566097+0000","last_fullsized":"2024-02-19T14:57:47.566097+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:04.979975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565186+0000","last_change":"2024-02-19T14:57:45.774703+0000","last_active":"2024-02-19T14:57:47.565186+0000","last_peered":"2024-02-19T14:57:47.565186+0000","last_clean":"2024-02-19T14:57:47.565186+0000","last_became_active":"2024-02-19T14:57:45.773430+0000","last_became_peered":"2024-02-19T14:57:45.773430+0000","last_unstale":"2024-02-19T14:57:47.565186+0000","last_undegraded":"2024-02-19T14:57:47.565186+0000","last_fullsized":"2024-02-19T14:57:47.565186+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:56.911480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565010+0000","last_change":"2024-02-19T14:57:45.780128+0000","last_active":"2024-02-19T14:57:47.565010+0000","last_peered":"2024-02-19T14:57:47.565010+0000","last_clean":"2024-02-19T14:57:47.565010+0000","last_became_active":"2024-02-19T14:57:45.779825+0000","last_became_peered":"2024-02-19T14:57:45.779825+0000","last_unstale":"2024-02-19T14:57:47.565010+0000","last_undegraded":"2024-02-19T14:57:47.565010+0000","last_fullsized":"2024-02-19T14:57:47.565010+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:26.050909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565737+0000","last_change":"2024-02-19T14:57:45.781182+0000","last_active":"2024-02-19T14:57:47.565737+0000","last_peered":"2024-02-19T14:57:47.565737+0000","last_clean":"2024-02-19T14:57:47.565737+0000","last_became_active":"2024-02-19T14:57:45.780764+0000","last_became_peered":"2024-02-19T14:57:45.780764+0000","last_unstale":"2024-02-19T14:57:47.565737+0000","last_undegraded":"2024-02-19T14:57:47.565737+0000","last_fullsized":"2024-02-19T14:57:47.565737+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:23.510855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762101+0000","last_change":"2024-02-19T14:57:44.748896+0000","last_active":"2024-02-19T14:57:46.762101+0000","last_peered":"2024-02-19T14:57:46.762101+0000","last_clean":"2024-02-19T14:57:46.762101+0000","last_became_active":"2024-02-19T14:57:44.748624+0000","last_became_peered":"2024-02-19T14:57:44.748624+0000","last_unstale":"2024-02-19T14:57:46.762101+0000","last_undegraded":"2024-02-19T14:57:46.762101+0000","last_fullsized":"2024-02-19T14:57:46.762101+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:53:27.719470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564328+0000","last_change":"2024-02-19T14:57:45.778729+0000","last_active":"2024-02-19T14:57:47.564328+0000","last_peered":"2024-02-19T14:57:47.564328+0000","last_clean":"2024-02-19T14:57:47.564328+0000","last_became_active":"2024-02-19T14:57:45.778460+0000","last_became_peered":"2024-02-19T14:57:45.778460+0000","last_unstale":"2024-02-19T14:57:47.564328+0000","last_undegraded":"2024-02-19T14:57:47.564328+0000","last_fullsized":"2024-02-19T14:57:47.564328+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:46.115236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762297+0000","last_change":"2024-02-19T14:57:44.759512+0000","last_active":"2024-02-19T14:57:46.762297+0000","last_peered":"2024-02-19T14:57:46.762297+0000","last_clean":"2024-02-19T14:57:46.762297+0000","last_became_active":"2024-02-19T14:57:44.759251+0000","last_became_peered":"2024-02-19T14:57:44.759251+0000","last_unstale":"2024-02-19T14:57:46.762297+0000","last_undegraded":"2024-02-19T14:57:46.762297+0000","last_fullsized":"2024-02-19T14:57:46.762297+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:23.064686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762006+0000","last_change":"2024-02-19T14:57:45.775749+0000","last_active":"2024-02-19T14:57:46.762006+0000","last_peered":"2024-02-19T14:57:46.762006+0000","last_clean":"2024-02-19T14:57:46.762006+0000","last_became_active":"2024-02-19T14:57:45.775481+0000","last_became_peered":"2024-02-19T14:57:45.775481+0000","last_unstale":"2024-02-19T14:57:46.762006+0000","last_undegraded":"2024-02-19T14:57:46.762006+0000","last_fullsized":"2024-02-19T14:57:46.762006+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565702+0000","last_change":"2024-02-19T14:57:44.746085+0000","last_active":"2024-02-19T14:57:47.565702+0000","last_peered":"2024-02-19T14:57:47.565702+0000","last_clean":"2024-02-19T14:57:47.565702+0000","last_became_active":"2024-02-19T14:57:44.745804+0000","last_became_peered":"2024-02-19T14:57:44.745804+0000","last_unstale":"2024-02-19T14:57:47.565702+0000","last_undegraded":"2024-02-19T14:57:47.565702+0000","last_fullsized":"2024-02-19T14:57:47.565702+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565730+0000","last_change":"2024-02-19T14:57:45.784232+0000","last_active":"2024-02-19T14:57:47.565730+0000","last_peered":"2024-02-19T14:57:47.565730+0000","last_clean":"2024-02-19T14:57:47.565730+0000","last_became_active":"2024-02-19T14:57:45.784007+0000","last_became_peered":"2024-02-19T14:57:45.784007+0000","last_unstale":"2024-02-19T14:57:47.565730+0000","last_undegraded":"2024-02-19T14:57:47.565730+0000","last_fullsized":"2024-02-19T14:57:47.565730+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:50.022179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564425+0000","last_change":"2024-02-19T14:57:44.754196+0000","last_active":"2024-02-19T14:57:47.564425+0000","last_peered":"2024-02-19T14:57:47.564425+0000","last_clean":"2024-02-19T14:57:47.564425+0000","last_became_active":"2024-02-19T14:57:44.753938+0000","last_became_peered":"2024-02-19T14:57:44.753938+0000","last_unstale":"2024-02-19T14:57:47.564425+0000","last_undegraded":"2024-02-19T14:57:47.564425+0000","last_fullsized":"2024-02-19T14:57:47.564425+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:17.948844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761382+0000","last_change":"2024-02-19T14:57:45.776534+0000","last_active":"2024-02-19T14:57:46.761382+0000","last_peered":"2024-02-19T14:57:46.761382+0000","last_clean":"2024-02-19T14:57:46.761382+0000","last_became_active":"2024-02-19T14:57:45.776277+0000","last_became_peered":"2024-02-19T14:57:45.776277+0000","last_unstale":"2024-02-19T14:57:46.761382+0000","last_undegraded":"2024-02-19T14:57:46.761382+0000","last_fullsized":"2024-02-19T14:57:46.761382+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761508+0000","last_change":"2024-02-19T14:57:44.759675+0000","last_active":"2024-02-19T14:57:46.761508+0000","last_peered":"2024-02-19T14:57:46.761508+0000","last_clean":"2024-02-19T14:57:46.761508+0000","last_became_active":"2024-02-19T14:57:44.759404+0000","last_became_peered":"2024-02-19T14:57:44.759404+0000","last_unstale":"2024-02-19T14:57:46.761508+0000","last_undegraded":"2024-02-19T14:57:46.761508+0000","last_fullsized":"2024-02-19T14:57:46.761508+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:34.658675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564611+0000","last_change":"2024-02-19T14:57:45.777221+0000","last_active":"2024-02-19T14:57:47.564611+0000","last_peered":"2024-02-19T14:57:47.564611+0000","last_clean":"2024-02-19T14:57:47.564611+0000","last_became_active":"2024-02-19T14:57:45.776887+0000","last_became_peered":"2024-02-19T14:57:45.776887+0000","last_unstale":"2024-02-19T14:57:47.564611+0000","last_undegraded":"2024-02-19T14:57:47.564611+0000","last_fullsized":"2024-02-19T14:57:47.564611+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:43.785509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762088+0000","last_change":"2024-02-19T14:57:44.759410+0000","last_active":"2024-02-19T14:57:46.762088+0000","last_peered":"2024-02-19T14:57:46.762088+0000","last_clean":"2024-02-19T14:57:46.762088+0000","last_became_active":"2024-02-19T14:57:44.759125+0000","last_became_peered":"2024-02-19T14:57:44.759125+0000","last_unstale":"2024-02-19T14:57:46.762088+0000","last_undegraded":"2024-02-19T14:57:46.762088+0000","last_fullsized":"2024-02-19T14:57:46.762088+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:47.742311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564757+0000","last_change":"2024-02-19T14:57:45.778260+0000","last_active":"2024-02-19T14:57:47.564757+0000","last_peered":"2024-02-19T14:57:47.564757+0000","last_clean":"2024-02-19T14:57:47.564757+0000","last_became_active":"2024-02-19T14:57:45.778006+0000","last_became_peered":"2024-02-19T14:57:45.778006+0000","last_unstale":"2024-02-19T14:57:47.564757+0000","last_undegraded":"2024-02-19T14:57:47.564757+0000","last_fullsized":"2024-02-19T14:57:47.564757+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:35.800806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564154+0000","last_change":"2024-02-19T14:57:44.757229+0000","last_active":"2024-02-19T14:57:47.564154+0000","last_peered":"2024-02-19T14:57:47.564154+0000","last_clean":"2024-02-19T14:57:47.564154+0000","last_became_active":"2024-02-19T14:57:44.756966+0000","last_became_peered":"2024-02-19T14:57:44.756966+0000","last_unstale":"2024-02-19T14:57:47.564154+0000","last_undegraded":"2024-02-19T14:57:47.564154+0000","last_fullsized":"2024-02-19T14:57:47.564154+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:12.299965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761984+0000","last_change":"2024-02-19T14:57:45.781313+0000","last_active":"2024-02-19T14:57:46.761984+0000","last_peered":"2024-02-19T14:57:46.761984+0000","last_clean":"2024-02-19T14:57:46.761984+0000","last_became_active":"2024-02-19T14:57:45.778982+0000","last_became_peered":"2024-02-19T14:57:45.778982+0000","last_unstale":"2024-02-19T14:57:46.761984+0000","last_undegraded":"2024-02-19T14:57:46.761984+0000","last_fullsized":"2024-02-19T14:57:46.761984+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:23.251435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761411+0000","last_change":"2024-02-19T14:57:44.757934+0000","last_active":"2024-02-19T14:57:46.761411+0000","last_peered":"2024-02-19T14:57:46.761411+0000","last_clean":"2024-02-19T14:57:46.761411+0000","last_became_active":"2024-02-19T14:57:44.757660+0000","last_became_peered":"2024-02-19T14:57:44.757660+0000","last_unstale":"2024-02-19T14:57:46.761411+0000","last_undegraded":"2024-02-19T14:57:46.761411+0000","last_fullsized":"2024-02-19T14:57:46.761411+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565736+0000","last_change":"2024-02-19T14:57:45.776416+0000","last_active":"2024-02-19T14:57:47.565736+0000","last_peered":"2024-02-19T14:57:47.565736+0000","last_clean":"2024-02-19T14:57:47.565736+0000","last_became_active":"2024-02-19T14:57:45.775916+0000","last_became_peered":"2024-02-19T14:57:45.775916+0000","last_unstale":"2024-02-19T14:57:47.565736+0000","last_undegraded":"2024-02-19T14:57:47.565736+0000","last_fullsized":"2024-02-19T14:57:47.565736+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:33.825283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565133+0000","last_change":"2024-02-19T14:57:44.755422+0000","last_active":"2024-02-19T14:57:47.565133+0000","last_peered":"2024-02-19T14:57:47.565133+0000","last_clean":"2024-02-19T14:57:47.565133+0000","last_became_active":"2024-02-19T14:57:44.755152+0000","last_became_peered":"2024-02-19T14:57:44.755152+0000","last_unstale":"2024-02-19T14:57:47.565133+0000","last_undegraded":"2024-02-19T14:57:47.565133+0000","last_fullsized":"2024-02-19T14:57:47.565133+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:13.575356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564403+0000","last_change":"2024-02-19T14:57:45.781850+0000","last_active":"2024-02-19T14:57:47.564403+0000","last_peered":"2024-02-19T14:57:47.564403+0000","last_clean":"2024-02-19T14:57:47.564403+0000","last_became_active":"2024-02-19T14:57:45.781588+0000","last_became_peered":"2024-02-19T14:57:45.781588+0000","last_unstale":"2024-02-19T14:57:47.564403+0000","last_undegraded":"2024-02-19T14:57:47.564403+0000","last_fullsized":"2024-02-19T14:57:47.564403+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:48.140379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761644+0000","last_change":"2024-02-19T14:57:44.751600+0000","last_active":"2024-02-19T14:57:46.761644+0000","last_peered":"2024-02-19T14:57:46.761644+0000","last_clean":"2024-02-19T14:57:46.761644+0000","last_became_active":"2024-02-19T14:57:44.751311+0000","last_became_peered":"2024-02-19T14:57:44.751311+0000","last_unstale":"2024-02-19T14:57:46.761644+0000","last_undegraded":"2024-02-19T14:57:46.761644+0000","last_fullsized":"2024-02-19T14:57:46.761644+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:50.448796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564381+0000","last_change":"2024-02-19T14:57:45.782664+0000","last_active":"2024-02-19T14:57:47.564381+0000","last_peered":"2024-02-19T14:57:47.564381+0000","last_clean":"2024-02-19T14:57:47.564381+0000","last_became_active":"2024-02-19T14:57:45.782411+0000","last_became_peered":"2024-02-19T14:57:45.782411+0000","last_unstale":"2024-02-19T14:57:47.564381+0000","last_undegraded":"2024-02-19T14:57:47.564381+0000","last_fullsized":"2024-02-19T14:57:47.564381+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:26.461781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565078+0000","last_change":"2024-02-19T14:57:44.756497+0000","last_active":"2024-02-19T14:57:47.565078+0000","last_peered":"2024-02-19T14:57:47.565078+0000","last_clean":"2024-02-19T14:57:47.565078+0000","last_became_active":"2024-02-19T14:57:44.756213+0000","last_became_peered":"2024-02-19T14:57:44.756213+0000","last_unstale":"2024-02-19T14:57:47.565078+0000","last_undegraded":"2024-02-19T14:57:47.565078+0000","last_fullsized":"2024-02-19T14:57:47.565078+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:28.745803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566184+0000","last_change":"2024-02-19T14:57:45.780739+0000","last_active":"2024-02-19T14:57:47.566184+0000","last_peered":"2024-02-19T14:57:47.566184+0000","last_clean":"2024-02-19T14:57:47.566184+0000","last_became_active":"2024-02-19T14:57:45.780414+0000","last_became_peered":"2024-02-19T14:57:45.780414+0000","last_unstale":"2024-02-19T14:57:47.566184+0000","last_undegraded":"2024-02-19T14:57:47.566184+0000","last_fullsized":"2024-02-19T14:57:47.566184+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:25.103177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762347+0000","last_change":"2024-02-19T14:57:44.751458+0000","last_active":"2024-02-19T14:57:46.762347+0000","last_peered":"2024-02-19T14:57:46.762347+0000","last_clean":"2024-02-19T14:57:46.762347+0000","last_became_active":"2024-02-19T14:57:44.751194+0000","last_became_peered":"2024-02-19T14:57:44.751194+0000","last_unstale":"2024-02-19T14:57:46.762347+0000","last_undegraded":"2024-02-19T14:57:46.762347+0000","last_fullsized":"2024-02-19T14:57:46.762347+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:11:44.993761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761640+0000","last_change":"2024-02-19T14:57:45.775378+0000","last_active":"2024-02-19T14:57:46.761640+0000","last_peered":"2024-02-19T14:57:46.761640+0000","last_clean":"2024-02-19T14:57:46.761640+0000","last_became_active":"2024-02-19T14:57:45.774903+0000","last_became_peered":"2024-02-19T14:57:45.774903+0000","last_unstale":"2024-02-19T14:57:46.761640+0000","last_undegraded":"2024-02-19T14:57:46.761640+0000","last_fullsized":"2024-02-19T14:57:46.761640+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564493+0000","last_change":"2024-02-19T14:57:44.748468+0000","last_active":"2024-02-19T14:57:47.564493+0000","last_peered":"2024-02-19T14:57:47.564493+0000","last_clean":"2024-02-19T14:57:47.564493+0000","last_became_active":"2024-02-19T14:57:44.748055+0000","last_became_peered":"2024-02-19T14:57:44.748055+0000","last_unstale":"2024-02-19T14:57:47.564493+0000","last_undegraded":"2024-02-19T14:57:47.564493+0000","last_fullsized":"2024-02-19T14:57:47.564493+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:32.823498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566162+0000","last_change":"2024-02-19T14:57:45.780058+0000","last_active":"2024-02-19T14:57:47.566162+0000","last_peered":"2024-02-19T14:57:47.566162+0000","last_clean":"2024-02-19T14:57:47.566162+0000","last_became_active":"2024-02-19T14:57:45.779771+0000","last_became_peered":"2024-02-19T14:57:45.779771+0000","last_unstale":"2024-02-19T14:57:47.566162+0000","last_undegraded":"2024-02-19T14:57:47.566162+0000","last_fullsized":"2024-02-19T14:57:47.566162+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565311+0000","last_change":"2024-02-19T14:57:44.751398+0000","last_active":"2024-02-19T14:57:47.565311+0000","last_peered":"2024-02-19T14:57:47.565311+0000","last_clean":"2024-02-19T14:57:47.565311+0000","last_became_active":"2024-02-19T14:57:44.751099+0000","last_became_peered":"2024-02-19T14:57:44.751099+0000","last_unstale":"2024-02-19T14:57:47.565311+0000","last_undegraded":"2024-02-19T14:57:47.565311+0000","last_fullsized":"2024-02-19T14:57:47.565311+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:51.642710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564770+0000","last_change":"2024-02-19T14:57:45.783960+0000","last_active":"2024-02-19T14:57:47.564770+0000","last_peered":"2024-02-19T14:57:47.564770+0000","last_clean":"2024-02-19T14:57:47.564770+0000","last_became_active":"2024-02-19T14:57:45.783704+0000","last_became_peered":"2024-02-19T14:57:45.783704+0000","last_unstale":"2024-02-19T14:57:47.564770+0000","last_undegraded":"2024-02-19T14:57:47.564770+0000","last_fullsized":"2024-02-19T14:57:47.564770+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:24.171999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761510+0000","last_change":"2024-02-19T14:57:44.758621+0000","last_active":"2024-02-19T14:57:46.761510+0000","last_peered":"2024-02-19T14:57:46.761510+0000","last_clean":"2024-02-19T14:57:46.761510+0000","last_became_active":"2024-02-19T14:57:44.758296+0000","last_became_peered":"2024-02-19T14:57:44.758296+0000","last_unstale":"2024-02-19T14:57:46.761510+0000","last_undegraded":"2024-02-19T14:57:46.761510+0000","last_fullsized":"2024-02-19T14:57:46.761510+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761872+0000","last_change":"2024-02-19T14:57:45.773816+0000","last_active":"2024-02-19T14:57:46.761872+0000","last_peered":"2024-02-19T14:57:46.761872+0000","last_clean":"2024-02-19T14:57:46.761872+0000","last_became_active":"2024-02-19T14:57:45.773272+0000","last_became_peered":"2024-02-19T14:57:45.773272+0000","last_unstale":"2024-02-19T14:57:46.761872+0000","last_undegraded":"2024-02-19T14:57:46.761872+0000","last_fullsized":"2024-02-19T14:57:46.761872+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565198+0000","last_change":"2024-02-19T14:57:44.751265+0000","last_active":"2024-02-19T14:57:47.565198+0000","last_peered":"2024-02-19T14:57:47.565198+0000","last_clean":"2024-02-19T14:57:47.565198+0000","last_became_active":"2024-02-19T14:57:44.750991+0000","last_became_peered":"2024-02-19T14:57:44.750991+0000","last_unstale":"2024-02-19T14:57:47.565198+0000","last_undegraded":"2024-02-19T14:57:47.565198+0000","last_fullsized":"2024-02-19T14:57:47.565198+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:32.644808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566093+0000","last_change":"2024-02-19T14:57:45.773622+0000","last_active":"2024-02-19T14:57:47.566093+0000","last_peered":"2024-02-19T14:57:47.566093+0000","last_clean":"2024-02-19T14:57:47.566093+0000","last_became_active":"2024-02-19T14:57:45.773349+0000","last_became_peered":"2024-02-19T14:57:45.773349+0000","last_unstale":"2024-02-19T14:57:47.566093+0000","last_undegraded":"2024-02-19T14:57:47.566093+0000","last_fullsized":"2024-02-19T14:57:47.566093+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:01.576910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761815+0000","last_change":"2024-02-19T14:57:44.757425+0000","last_active":"2024-02-19T14:57:46.761815+0000","last_peered":"2024-02-19T14:57:46.761815+0000","last_clean":"2024-02-19T14:57:46.761815+0000","last_became_active":"2024-02-19T14:57:44.757006+0000","last_became_peered":"2024-02-19T14:57:44.757006+0000","last_unstale":"2024-02-19T14:57:46.761815+0000","last_undegraded":"2024-02-19T14:57:46.761815+0000","last_fullsized":"2024-02-19T14:57:46.761815+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:41.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565018+0000","last_change":"2024-02-19T14:57:45.781720+0000","last_active":"2024-02-19T14:57:47.565018+0000","last_peered":"2024-02-19T14:57:47.565018+0000","last_clean":"2024-02-19T14:57:47.565018+0000","last_became_active":"2024-02-19T14:57:45.781425+0000","last_became_peered":"2024-02-19T14:57:45.781425+0000","last_unstale":"2024-02-19T14:57:47.565018+0000","last_undegraded":"2024-02-19T14:57:47.565018+0000","last_fullsized":"2024-02-19T14:57:47.565018+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:36.396204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762308+0000","last_change":"2024-02-19T14:57:44.758828+0000","last_active":"2024-02-19T14:57:46.762308+0000","last_peered":"2024-02-19T14:57:46.762308+0000","last_clean":"2024-02-19T14:57:46.762308+0000","last_became_active":"2024-02-19T14:57:44.758387+0000","last_became_peered":"2024-02-19T14:57:44.758387+0000","last_unstale":"2024-02-19T14:57:46.762308+0000","last_undegraded":"2024-02-19T14:57:46.762308+0000","last_fullsized":"2024-02-19T14:57:46.762308+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565308+0000","last_change":"2024-02-19T14:57:45.780279+0000","last_active":"2024-02-19T14:57:47.565308+0000","last_peered":"2024-02-19T14:57:47.565308+0000","last_clean":"2024-02-19T14:57:47.565308+0000","last_became_active":"2024-02-19T14:57:45.779989+0000","last_became_peered":"2024-02-19T14:57:45.779989+0000","last_unstale":"2024-02-19T14:57:47.565308+0000","last_undegraded":"2024-02-19T14:57:47.565308+0000","last_fullsized":"2024-02-19T14:57:47.565308+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761545+0000","last_change":"2024-02-19T14:57:44.760014+0000","last_active":"2024-02-19T14:57:46.761545+0000","last_peered":"2024-02-19T14:57:46.761545+0000","last_clean":"2024-02-19T14:57:46.761545+0000","last_became_active":"2024-02-19T14:57:44.759749+0000","last_became_peered":"2024-02-19T14:57:44.759749+0000","last_unstale":"2024-02-19T14:57:46.761545+0000","last_undegraded":"2024-02-19T14:57:46.761545+0000","last_fullsized":"2024-02-19T14:57:46.761545+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565021+0000","last_change":"2024-02-19T14:57:45.785469+0000","last_active":"2024-02-19T14:57:47.565021+0000","last_peered":"2024-02-19T14:57:47.565021+0000","last_clean":"2024-02-19T14:57:47.565021+0000","last_became_active":"2024-02-19T14:57:45.784759+0000","last_became_peered":"2024-02-19T14:57:45.784759+0000","last_unstale":"2024-02-19T14:57:47.565021+0000","last_undegraded":"2024-02-19T14:57:47.565021+0000","last_fullsized":"2024-02-19T14:57:47.565021+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:26.131093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762272+0000","last_change":"2024-02-19T14:57:44.752558+0000","last_active":"2024-02-19T14:57:46.762272+0000","last_peered":"2024-02-19T14:57:46.762272+0000","last_clean":"2024-02-19T14:57:46.762272+0000","last_became_active":"2024-02-19T14:57:44.752290+0000","last_became_peered":"2024-02-19T14:57:44.752290+0000","last_unstale":"2024-02-19T14:57:46.762272+0000","last_undegraded":"2024-02-19T14:57:46.762272+0000","last_fullsized":"2024-02-19T14:57:46.762272+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:55.760619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762282+0000","last_change":"2024-02-19T14:57:45.776484+0000","last_active":"2024-02-19T14:57:46.762282+0000","last_peered":"2024-02-19T14:57:46.762282+0000","last_clean":"2024-02-19T14:57:46.762282+0000","last_became_active":"2024-02-19T14:57:45.776095+0000","last_became_peered":"2024-02-19T14:57:45.776095+0000","last_unstale":"2024-02-19T14:57:46.762282+0000","last_undegraded":"2024-02-19T14:57:46.762282+0000","last_fullsized":"2024-02-19T14:57:46.762282+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565130+0000","last_change":"2024-02-19T14:57:44.746247+0000","last_active":"2024-02-19T14:57:47.565130+0000","last_peered":"2024-02-19T14:57:47.565130+0000","last_clean":"2024-02-19T14:57:47.565130+0000","last_became_active":"2024-02-19T14:57:44.745941+0000","last_became_peered":"2024-02-19T14:57:44.745941+0000","last_unstale":"2024-02-19T14:57:47.565130+0000","last_undegraded":"2024-02-19T14:57:47.565130+0000","last_fullsized":"2024-02-19T14:57:47.565130+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762153+0000","last_change":"2024-02-19T14:57:45.766067+0000","last_active":"2024-02-19T14:57:46.762153+0000","last_peered":"2024-02-19T14:57:46.762153+0000","last_clean":"2024-02-19T14:57:46.762153+0000","last_became_active":"2024-02-19T14:57:45.765654+0000","last_became_peered":"2024-02-19T14:57:45.765654+0000","last_unstale":"2024-02-19T14:57:46.762153+0000","last_undegraded":"2024-02-19T14:57:46.762153+0000","last_fullsized":"2024-02-19T14:57:46.762153+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564248+0000","last_change":"2024-02-19T14:57:44.751718+0000","last_active":"2024-02-19T14:57:47.564248+0000","last_peered":"2024-02-19T14:57:47.564248+0000","last_clean":"2024-02-19T14:57:47.564248+0000","last_became_active":"2024-02-19T14:57:44.751403+0000","last_became_peered":"2024-02-19T14:57:44.751403+0000","last_unstale":"2024-02-19T14:57:47.564248+0000","last_undegraded":"2024-02-19T14:57:47.564248+0000","last_fullsized":"2024-02-19T14:57:47.564248+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:59.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.566321+0000","last_change":"2024-02-19T14:57:45.775690+0000","last_active":"2024-02-19T14:57:47.566321+0000","last_peered":"2024-02-19T14:57:47.566321+0000","last_clean":"2024-02-19T14:57:47.566321+0000","last_became_active":"2024-02-19T14:57:45.775076+0000","last_became_peered":"2024-02-19T14:57:45.775076+0000","last_unstale":"2024-02-19T14:57:47.566321+0000","last_undegraded":"2024-02-19T14:57:47.566321+0000","last_fullsized":"2024-02-19T14:57:47.566321+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:40.949417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565869+0000","last_change":"2024-02-19T14:57:44.751791+0000","last_active":"2024-02-19T14:57:47.565869+0000","last_peered":"2024-02-19T14:57:47.565869+0000","last_clean":"2024-02-19T14:57:47.565869+0000","last_became_active":"2024-02-19T14:57:44.751480+0000","last_became_peered":"2024-02-19T14:57:44.751480+0000","last_unstale":"2024-02-19T14:57:47.565869+0000","last_undegraded":"2024-02-19T14:57:47.565869+0000","last_fullsized":"2024-02-19T14:57:47.565869+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564982+0000","last_change":"2024-02-19T14:57:45.772415+0000","last_active":"2024-02-19T14:57:47.564982+0000","last_peered":"2024-02-19T14:57:47.564982+0000","last_clean":"2024-02-19T14:57:47.564982+0000","last_became_active":"2024-02-19T14:57:45.771919+0000","last_became_peered":"2024-02-19T14:57:45.771919+0000","last_unstale":"2024-02-19T14:57:47.564982+0000","last_undegraded":"2024-02-19T14:57:47.564982+0000","last_fullsized":"2024-02-19T14:57:47.564982+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:29.983600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.762317+0000","last_change":"2024-02-19T14:57:44.758252+0000","last_active":"2024-02-19T14:57:46.762317+0000","last_peered":"2024-02-19T14:57:46.762317+0000","last_clean":"2024-02-19T14:57:46.762317+0000","last_became_active":"2024-02-19T14:57:44.757979+0000","last_became_peered":"2024-02-19T14:57:44.757979+0000","last_unstale":"2024-02-19T14:57:46.762317+0000","last_undegraded":"2024-02-19T14:57:46.762317+0000","last_fullsized":"2024-02-19T14:57:46.762317+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:28.981816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761391+0000","last_change":"2024-02-19T14:57:45.772751+0000","last_active":"2024-02-19T14:57:46.761391+0000","last_peered":"2024-02-19T14:57:46.761391+0000","last_clean":"2024-02-19T14:57:46.761391+0000","last_became_active":"2024-02-19T14:57:45.772506+0000","last_became_peered":"2024-02-19T14:57:45.772506+0000","last_unstale":"2024-02-19T14:57:46.761391+0000","last_undegraded":"2024-02-19T14:57:46.761391+0000","last_fullsized":"2024-02-19T14:57:46.761391+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"152'1","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.761902+0000","last_change":"2024-02-19T14:57:44.756795+0000","last_active":"2024-02-19T14:57:47.761902+0000","last_peered":"2024-02-19T14:57:47.761902+0000","last_clean":"2024-02-19T14:57:47.761902+0000","last_became_active":"2024-02-19T14:57:44.756166+0000","last_became_peered":"2024-02-19T14:57:44.756166+0000","last_unstale":"2024-02-19T14:57:47.761902+0000","last_undegraded":"2024-02-19T14:57:47.761902+0000","last_fullsized":"2024-02-19T14:57:47.761902+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:13.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564315+0000","last_change":"2024-02-19T14:57:45.774855+0000","last_active":"2024-02-19T14:57:47.564315+0000","last_peered":"2024-02-19T14:57:47.564315+0000","last_clean":"2024-02-19T14:57:47.564315+0000","last_became_active":"2024-02-19T14:57:45.774572+0000","last_became_peered":"2024-02-19T14:57:45.774572+0000","last_unstale":"2024-02-19T14:57:47.564315+0000","last_undegraded":"2024-02-19T14:57:47.564315+0000","last_fullsized":"2024-02-19T14:57:47.564315+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:34.308217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564929+0000","last_change":"2024-02-19T14:57:44.749436+0000","last_active":"2024-02-19T14:57:47.564929+0000","last_peered":"2024-02-19T14:57:47.564929+0000","last_clean":"2024-02-19T14:57:47.564929+0000","last_became_active":"2024-02-19T14:57:44.749153+0000","last_became_peered":"2024-02-19T14:57:44.749153+0000","last_unstale":"2024-02-19T14:57:47.564929+0000","last_undegraded":"2024-02-19T14:57:47.564929+0000","last_fullsized":"2024-02-19T14:57:47.564929+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:36.657057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761396+0000","last_change":"2024-02-19T14:57:45.772279+0000","last_active":"2024-02-19T14:57:46.761396+0000","last_peered":"2024-02-19T14:57:46.761396+0000","last_clean":"2024-02-19T14:57:46.761396+0000","last_became_active":"2024-02-19T14:57:45.771983+0000","last_became_peered":"2024-02-19T14:57:45.771983+0000","last_unstale":"2024-02-19T14:57:46.761396+0000","last_undegraded":"2024-02-19T14:57:46.761396+0000","last_fullsized":"2024-02-19T14:57:46.761396+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564508+0000","last_change":"2024-02-19T14:57:44.758176+0000","last_active":"2024-02-19T14:57:47.564508+0000","last_peered":"2024-02-19T14:57:47.564508+0000","last_clean":"2024-02-19T14:57:47.564508+0000","last_became_active":"2024-02-19T14:57:44.757912+0000","last_became_peered":"2024-02-19T14:57:44.757912+0000","last_unstale":"2024-02-19T14:57:47.564508+0000","last_undegraded":"2024-02-19T14:57:47.564508+0000","last_fullsized":"2024-02-19T14:57:47.564508+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:26.671302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761703+0000","last_change":"2024-02-19T14:57:45.771150+0000","last_active":"2024-02-19T14:57:46.761703+0000","last_peered":"2024-02-19T14:57:46.761703+0000","last_clean":"2024-02-19T14:57:46.761703+0000","last_became_active":"2024-02-19T14:57:45.770740+0000","last_became_peered":"2024-02-19T14:57:45.770740+0000","last_unstale":"2024-02-19T14:57:46.761703+0000","last_undegraded":"2024-02-19T14:57:46.761703+0000","last_fullsized":"2024-02-19T14:57:46.761703+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:57.456060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761372+0000","last_change":"2024-02-19T14:57:44.752321+0000","last_active":"2024-02-19T14:57:46.761372+0000","last_peered":"2024-02-19T14:57:46.761372+0000","last_clean":"2024-02-19T14:57:46.761372+0000","last_became_active":"2024-02-19T14:57:44.752031+0000","last_became_peered":"2024-02-19T14:57:44.752031+0000","last_unstale":"2024-02-19T14:57:46.761372+0000","last_undegraded":"2024-02-19T14:57:46.761372+0000","last_fullsized":"2024-02-19T14:57:46.761372+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:54.123952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565409+0000","last_change":"2024-02-19T14:57:45.784058+0000","last_active":"2024-02-19T14:57:47.565409+0000","last_peered":"2024-02-19T14:57:47.565409+0000","last_clean":"2024-02-19T14:57:47.565409+0000","last_became_active":"2024-02-19T14:57:45.783772+0000","last_became_peered":"2024-02-19T14:57:45.783772+0000","last_unstale":"2024-02-19T14:57:47.565409+0000","last_undegraded":"2024-02-19T14:57:47.565409+0000","last_fullsized":"2024-02-19T14:57:47.565409+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:34.823875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564505+0000","last_change":"2024-02-19T14:57:44.743222+0000","last_active":"2024-02-19T14:57:47.564505+0000","last_peered":"2024-02-19T14:57:47.564505+0000","last_clean":"2024-02-19T14:57:47.564505+0000","last_became_active":"2024-02-19T14:57:44.742946+0000","last_became_peered":"2024-02-19T14:57:44.742946+0000","last_unstale":"2024-02-19T14:57:47.564505+0000","last_undegraded":"2024-02-19T14:57:47.564505+0000","last_fullsized":"2024-02-19T14:57:47.564505+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:22.509070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565019+0000","last_change":"2024-02-19T14:57:45.783349+0000","last_active":"2024-02-19T14:57:47.565019+0000","last_peered":"2024-02-19T14:57:47.565019+0000","last_clean":"2024-02-19T14:57:47.565019+0000","last_became_active":"2024-02-19T14:57:45.783101+0000","last_became_peered":"2024-02-19T14:57:45.783101+0000","last_unstale":"2024-02-19T14:57:47.565019+0000","last_undegraded":"2024-02-19T14:57:47.565019+0000","last_fullsized":"2024-02-19T14:57:47.565019+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:13:09.154323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565433+0000","last_change":"2024-02-19T14:57:44.747418+0000","last_active":"2024-02-19T14:57:47.565433+0000","last_peered":"2024-02-19T14:57:47.565433+0000","last_clean":"2024-02-19T14:57:47.565433+0000","last_became_active":"2024-02-19T14:57:44.746915+0000","last_became_peered":"2024-02-19T14:57:44.746915+0000","last_unstale":"2024-02-19T14:57:47.565433+0000","last_undegraded":"2024-02-19T14:57:47.565433+0000","last_fullsized":"2024-02-19T14:57:47.565433+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564793+0000","last_change":"2024-02-19T14:57:45.774106+0000","last_active":"2024-02-19T14:57:47.564793+0000","last_peered":"2024-02-19T14:57:47.564793+0000","last_clean":"2024-02-19T14:57:47.564793+0000","last_became_active":"2024-02-19T14:57:45.773847+0000","last_became_peered":"2024-02-19T14:57:45.773847+0000","last_unstale":"2024-02-19T14:57:47.564793+0000","last_undegraded":"2024-02-19T14:57:47.564793+0000","last_fullsized":"2024-02-19T14:57:47.564793+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:22.673960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:46.761676+0000","last_change":"2024-02-19T14:57:44.752893+0000","last_active":"2024-02-19T14:57:46.761676+0000","last_peered":"2024-02-19T14:57:46.761676+0000","last_clean":"2024-02-19T14:57:46.761676+0000","last_became_active":"2024-02-19T14:57:44.752617+0000","last_became_peered":"2024-02-19T14:57:44.752617+0000","last_unstale":"2024-02-19T14:57:46.761676+0000","last_undegraded":"2024-02-19T14:57:46.761676+0000","last_fullsized":"2024-02-19T14:57:46.761676+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:45.368123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565305+0000","last_change":"2024-02-19T14:57:45.777456+0000","last_active":"2024-02-19T14:57:47.565305+0000","last_peered":"2024-02-19T14:57:47.565305+0000","last_clean":"2024-02-19T14:57:47.565305+0000","last_became_active":"2024-02-19T14:57:45.776998+0000","last_became_peered":"2024-02-19T14:57:45.776998+0000","last_unstale":"2024-02-19T14:57:47.565305+0000","last_undegraded":"2024-02-19T14:57:47.565305+0000","last_fullsized":"2024-02-19T14:57:47.565305+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:14.577140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.564961+0000","last_change":"2024-02-19T14:57:44.757019+0000","last_active":"2024-02-19T14:57:47.564961+0000","last_peered":"2024-02-19T14:57:47.564961+0000","last_clean":"2024-02-19T14:57:47.564961+0000","last_became_active":"2024-02-19T14:57:44.756538+0000","last_became_peered":"2024-02-19T14:57:44.756538+0000","last_unstale":"2024-02-19T14:57:47.564961+0000","last_undegraded":"2024-02-19T14:57:47.564961+0000","last_fullsized":"2024-02-19T14:57:47.564961+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:33.822090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":417,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-02-19T14:57:47.565826+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:47.565826+0000","last_peered":"2024-02-19T14:57:47.565826+0000","last_clean":"2024-02-19T14:57:47.565826+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:47.565826+0000","last_undegraded":"2024-02-19T14:57:47.565826+0000","last_fullsized":"2024-02-19T14:57:47.565826+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2506752,"data_stored":1518686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.372,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68300000000000005,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.435},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.496},"min":{"1min":0.40500000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68200000000000005,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.46800000000000003},"min":{"1min":0.34300000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60099999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.38900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.63300000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.50800000000000001},"min":{"1min":0.33600000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.60199999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77600000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75800000000000001,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.5}]},{"osd":4,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45000000000000001},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.53900000000000003},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.78900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.54700000000000004},"min":{"1min":0.35899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.77500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.625}]},{"osd":6,"last update":"Mon Feb 19 14:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.59899999999999998},"min":{"1min":0.34000000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73399999999999999,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57099999999999995,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79700000000000004,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574923,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1900544,"data_stored":921626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.48399999999999999,"15min":0.46700000000000003},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71599999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67800000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65300000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67100000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51600000000000001,"15min":0.52000000000000002},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67500000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51400000000000001,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.61399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.504,"15min":0.53600000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70199999999999996,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.503,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73699999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.58199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80000000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70399999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.32200000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1921024,"data_stored":928442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52600000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.628}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.78900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1929216,"data_stored":929082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.495}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.69599999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.749}]}]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33068,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338772,"statfs":{"total":96636764160,"available":96602902528,"internally_reserved":0,"allocated":1933312,"data_stored":930534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.159,"5min":0.159,"15min":0.159},"max":{"1min":0.61699999999999999,"5min":3.694,"15min":3.694},"last":0.255},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.60399999999999998,"5min":4.835,"15min":4.835},"last":0.29399999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.313},"min":{"1min":0.223,"5min":0.187,"15min":0.187},"max":{"1min":0.629,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69199999999999995,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.41599999999999998,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72699999999999998,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39500000000000002,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57999999999999996,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.372}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92300000000000004,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.42299999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.91000000000000003,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.61199999999999999},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98099999999999998,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54900000000000004,"15min":0.56699999999999995},"min":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69999999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.629,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.877,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60699999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.74399999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61299999999999999,"15min":0.64600000000000002},"min":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79900000000000004,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.47799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1900544,"data_stored":921626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.379},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.246}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":929930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.311,"15min":0.33000000000000002},"min":{"1min":0.18099999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54100000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.51900000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.308},"min":{"1min":0.20499999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.52900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35099999999999998,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60899999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.375},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.377,"15min":0.38900000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56499999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78200000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40600000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77000000000000002,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81699999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77700000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":1.026,"15min":1.026},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56899999999999995},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.996,"15min":0.996},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:56:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":1.036,"15min":1.036},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.73199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33640,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338200,"statfs":{"total":96636764160,"available":96602316800,"internally_reserved":0,"allocated":2514944,"data_stored":1518790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.309,"15min":0.30099999999999999},"min":{"1min":0.23599999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"last":0.224}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32700000000000001},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34499999999999997,"15min":0.35099999999999998},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.55200000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.622,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55400000000000005,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68000000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56399999999999995,"15min":0.60399999999999998},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.41999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60199999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.51200000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57699999999999996,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.59499999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47499999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57199999999999995,"15min":0.60399999999999998},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68899999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56100000000000005,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64200000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.41199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:49.107 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:57:49.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:57:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.248+0000 7fcb20c4e640 1 Processor -- start 2024-02-19T14:57:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.248+0000 7fcb20c4e640 1 -- start start 2024-02-19T14:57:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.249+0000 7fcb20c4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb1c152c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb20c4e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c06f190 con 0x7fcb1c153170 2024-02-19T14:57:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb20c4e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1c06f420 con 0x7fcb1c149790 2024-02-19T14:57:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb20c4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c149b70 con 0x7fcb1c06ed40 2024-02-19T14:57:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcb1c153170 0x7fcb1c15d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58322/0 (socket says 172.21.15.78:58322) 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1ad9d640 1 -- 172.21.15.78:0/3387650779 learned_addr learned my addr 172.21.15.78:0/3387650779 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb19d9b640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb1c152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048902953 0 0) 0x7fcb1c06f190 con 0x7fcb1c153170 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb04003520 con 0x7fcb1c153170 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042282362 0 0) 0x7fcb1c06f420 con 0x7fcb1c149790 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb1c06f190 con 0x7fcb1c149790 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3492604169 0 0) 0x7fcb04003520 con 0x7fcb1c153170 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 msgr2=0x7fcb1c152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb1c152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 >> v1:172.21.15.150:6789/0 conn(0x7fcb1c149790 legacy=0x7fcb1c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c0ab760 con 0x7fcb1c153170 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fcb140019f0 con 0x7fcb1c153170 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 >> v1:172.21.15.78:6789/0 conn(0x7fcb1c153170 legacy=0x7fcb1c15d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb04003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb040047c0 0x7fcb04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb04005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0400bb10 con 0x7fcb04003b30 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0400bc80 con 0x7fcb040047c0 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0400be40 con 0x7fcb1c06ed40 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.250+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb04003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb19d9b640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb040047c0 0x7fcb04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1ad9d640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb04005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 msgr2=0x7fcb04005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb04005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb040047c0 msgr2=0x7fcb04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb040047c0 0x7fcb04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3387650779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04003520 con 0x7fcb04003b30 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1ad9d640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c06ed40 0x7fcb04005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.251+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb04003f30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fcb100018d0 tx=0x7fcb040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad034a22b217d5a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb100052d0 con 0x7fcb04003b30 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10005430 con 0x7fcb04003b30 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb1959a640 1 -- 172.21.15.78:0/3387650779 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10003810 con 0x7fcb04003b30 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3387650779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 msgr2=0x7fcb04003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3387650779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb04003f30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fcb100018d0 tx=0x7fcb040057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb1ad9d640 1 -- 172.21.15.78:0/3387650779 reap_dead start 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3387650779 shutdown_connections 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3387650779 >> 172.21.15.78:0/3387650779 conn(0x7fcb1c0718e0 msgr2=0x7fcb1c071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3387650779 shutdown_connections 2024-02-19T14:57:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.252+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3387650779 wait complete. 2024-02-19T14:57:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.253+0000 7fcb20c4e640 1 Processor -- start 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 -- start start 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb1c1dc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c1528c0 0x7fcb1c1e9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb1c1e9dc0 0x7fcb1c145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c1dee80 con 0x7fcb04003b30 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c155a80 con 0x7fcb1c1e9dc0 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb20c4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c156ba0 con 0x7fcb1c1528c0 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb1c1dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb1c1dc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46958/0 (socket says 172.21.15.78:46958) 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3187922856 learned_addr learned my addr 172.21.15.78:0/3187922856 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1ad9d640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb1c1e9dc0 0x7fcb1c145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb19d9b640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c1528c0 0x7fcb1c1e9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c1528c0 msgr2=0x7fcb1c1e9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c1528c0 0x7fcb1c1e9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb1c1e9dc0 msgr2=0x7fcb1c145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb1c1e9dc0 0x7fcb1c145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.254+0000 7fcb1a59c640 1 -- 172.21.15.78:0/3187922856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1c155d20 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.255+0000 7fcb1a59c640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb1c1dc290 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fcb10005040 tx=0x7fcb100012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.255+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10006ea0 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.255+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10007450 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.255+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c156e80 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.255+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb100035f0 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.255+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c152d40 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.256+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c152f30 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.256+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcb10003750 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.257+0000 7fcb0affd640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcafc06b810 0x7fcafc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.257+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb100abff0 con 0x7fcb04003b30 2024-02-19T14:57:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.257+0000 7fcb19d9b640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcafc06b810 0x7fcafc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.257+0000 7fcb19d9b640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcafc06b810 0x7fcafc06dcd0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c1465c0 tx=0x7fcb0c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.261+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcb100769c0 con 0x7fcb04003b30 2024-02-19T14:57:49.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.525+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcb1c074990 con 0x7fcb04003b30 2024-02-19T14:57:49.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.526+0000 7fcb0affd640 1 -- 172.21.15.78:0/3187922856 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcb1004b0e0 con 0x7fcb04003b30 2024-02-19T14:57:49.529 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:57:49.529 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:57:49.529 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:57:49.529 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:57:49.529 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:57:49.530 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcafc06b810 msgr2=0x7fcafc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcafc06b810 0x7fcafc06dcd0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c1465c0 tx=0x7fcb0c008040 comp rx=0 tx=0).stop 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 msgr2=0x7fcb1c1dc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb1c1dc290 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fcb10005040 tx=0x7fcb100012d0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 shutdown_connections 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcafc06b810 0x7fcafc06dcd0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb1c1e9dc0 0x7fcb1c145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb1c1528c0 0x7fcb1c1e9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 --2- 172.21.15.78:0/3187922856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb04003b30 0x7fcb1c1dc290 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 >> 172.21.15.78:0/3187922856 conn(0x7fcb1c0718e0 msgr2=0x7fcb1c0c5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 shutdown_connections 2024-02-19T14:57:49.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.529+0000 7fcb20c4e640 1 -- 172.21.15.78:0/3187922856 wait complete. 2024-02-19T14:57:49.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.641+0000 7ff9be68f640 1 Processor -- start 2024-02-19T14:57:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.641+0000 7ff9be68f640 1 -- start start 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9be68f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b80c8e00 0x7ff9b806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9be68f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9b806d910 0x7ff9b806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9be68f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b806cd20 con 0x7ff9b8152700 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9be68f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b81551f0 con 0x7ff9b806d910 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9be68f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8155490 con 0x7ff9b80c8e00 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9b806d910 0x7ff9b806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9b806d910 0x7ff9b806c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46970/0 (socket says 172.21.15.78:46970) 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 -- 172.21.15.78:0/2198399945 learned_addr learned my addr 172.21.15.78:0/2198399945 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9b77fe640 1 --2- 172.21.15.78:0/2198399945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b80c8e00 0x7ff9b806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 -- 172.21.15.78:0/2198399945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b80c8e00 msgr2=0x7ff9b806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 --2- 172.21.15.78:0/2198399945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b80c8e00 0x7ff9b806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 -- 172.21.15.78:0/2198399945 >> v1:172.21.15.150:6789/0 conn(0x7ff9b8152700 legacy=0x7ff9b80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.642+0000 7ff9bcc2c640 1 -- 172.21.15.78:0/2198399945 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8155770 con 0x7ff9b806d910 2024-02-19T14:57:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.643+0000 7ff9bcc2c640 1 --2- 172.21.15.78:0/2198399945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b806c7e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009e50 tx=0x7ff9b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ccdfed43d2bf5c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.643+0000 7ff9b6ffd640 1 -- 172.21.15.78:0/2198399945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b002ca70 con 0x7ff9b806d910 2024-02-19T14:57:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.643+0000 7ff9b6ffd640 1 -- 172.21.15.78:0/2198399945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b002cbd0 con 0x7ff9b806d910 2024-02-19T14:57:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.643+0000 7ff9b6ffd640 1 -- 172.21.15.78:0/2198399945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b00343f0 con 0x7ff9b806d910 2024-02-19T14:57:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 -- 172.21.15.78:0/2198399945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 msgr2=0x7ff9b806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 --2- 172.21.15.78:0/2198399945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b806c7e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009e50 tx=0x7ff9b002bed0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 -- 172.21.15.78:0/2198399945 shutdown_connections 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 --2- 172.21.15.78:0/2198399945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b806c7e0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 --2- 172.21.15.78:0/2198399945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b80c8e00 0x7ff9b806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 -- 172.21.15.78:0/2198399945 >> 172.21.15.78:0/2198399945 conn(0x7ff9b80cac90 msgr2=0x7ff9b80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 -- 172.21.15.78:0/2198399945 shutdown_connections 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.644+0000 7ff9be68f640 1 -- 172.21.15.78:0/2198399945 wait complete. 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.645+0000 7ff9be68f640 1 Processor -- start 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.645+0000 7ff9be68f640 1 -- start start 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b81ee010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9b80c8e00 0x7ff9b81ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b8152700 0x7ff9b81e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80c1120 con 0x7ff9b806d910 2024-02-19T14:57:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80c4910 con 0x7ff9b80c8e00 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81eea90 con 0x7ff9b8152700 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b81ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b81ee010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46972/0 (socket says 172.21.15.78:46972) 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 -- 172.21.15.78:0/379661841 learned_addr learned my addr 172.21.15.78:0/379661841 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b77fe640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9b80c8e00 0x7ff9b81ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9bcc2c640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b8152700 0x7ff9b81e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 -- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b8152700 msgr2=0x7ff9b81e7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b8152700 0x7ff9b81e7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 -- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9b80c8e00 msgr2=0x7ff9b81ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9b80c8e00 0x7ff9b81ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 -- 172.21.15.78:0/379661841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b0009b30 con 0x7ff9b806d910 2024-02-19T14:57:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b77fe640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9b80c8e00 0x7ff9b81ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b7fff640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b81ee010 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c3c0 tx=0x7ff9a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80052d0 con 0x7ff9b806d910 2024-02-19T14:57:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a8005450 con 0x7ff9b806d910 2024-02-19T14:57:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b80c4390 con 0x7ff9b806d910 2024-02-19T14:57:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8007260 con 0x7ff9b806d910 2024-02-19T14:57:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.646+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b80c4630 con 0x7ff9b806d910 2024-02-19T14:57:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.647+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b8074990 con 0x7ff9b806d910 2024-02-19T14:57:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.648+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff9a80073c0 con 0x7ff9b806d910 2024-02-19T14:57:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.648+0000 7ff9b4ff9640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9a406b810 0x7ff9a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.649+0000 7ff9b77fe640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9a406b810 0x7ff9a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.649+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9a8089670 con 0x7ff9b806d910 2024-02-19T14:57:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.649+0000 7ff9b77fe640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9a406b810 0x7ff9a406dcd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80cb270 tx=0x7ff9ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:49.653+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff9a8053f90 con 0x7ff9b806d910 2024-02-19T14:57:50.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.027+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b80c8770 con 0x7ff9b806d910 2024-02-19T14:57:50.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.027+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/379661841 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+4951 (secure 0 0 0) 0x7ff9a8057650 con 0x7ff9b806d910 2024-02-19T14:57:50.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:50.031 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":132,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7768,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":7771,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":8769,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/2997633530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2997633530},{"type":"v1","addr":"172.21.15.78:6836","nonce":2997633530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:57:47.113438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/903597448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":903597448},{"type":"v1","addr":"172.21.15.78:6837","nonce":903597448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-02-19T14:57:50.031 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 132 2024-02-19T14:57:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9a406b810 msgr2=0x7ff9a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9a406b810 0x7ff9a406dcd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80cb270 tx=0x7ff9ac008040 comp rx=0 tx=0).stop 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 msgr2=0x7ff9b81ee010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b81ee010 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c3c0 tx=0x7ff9a800c880 comp rx=0 tx=0).stop 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 shutdown_connections 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9a406b810 0x7ff9a406dcd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9b8152700 0x7ff9b81e7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9b80c8e00 0x7ff9b81ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 --2- 172.21.15.78:0/379661841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9b806d910 0x7ff9b81ee010 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 >> 172.21.15.78:0/379661841 conn(0x7ff9b80cac90 msgr2=0x7ff9b81515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 shutdown_connections 2024-02-19T14:57:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.030+0000 7ff9be68f640 1 -- 172.21.15.78:0/379661841 wait complete. 2024-02-19T14:57:50.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:57:45.095751+0000', 'modified': '2024-02-19T14:57:47.113438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8772}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8772': {'gid': 8772, 'name': 'a', 'rank': 0, 'incarnation': 130, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/903597448', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 903597448}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 903597448}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:57:50.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:57:50.449 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T14:57:50.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 Processor -- start 2024-02-19T14:57:50.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 -- start start 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd75c0c5560 0x7fd75c0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c0c7910 0x7fd75c154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75c0c65a0 con 0x7fd75c070720 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75c0c7530 con 0x7fd75c0c5560 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.549+0000 7fd7616d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c1537b0 con 0x7fd75c0c7910 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75a7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c0c7910 0x7fd75c154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd75c0c5560 0x7fd75c0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75b7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd75c070720 0x7fd75c070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:54214/0 (socket says 172.21.15.78:54214) 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75b7fe640 1 -- 172.21.15.78:0/3110501250 learned_addr learned my addr 172.21.15.78:0/3110501250 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd759ffb640 1 -- 172.21.15.78:0/3110501250 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980033920 0 0) 0x7fd75c0c65a0 con 0x7fd75c070720 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3110501250 >> v2:172.21.15.150:3300/0 conn(0x7fd75c0c5560 msgr2=0x7fd75c0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3110501250 >> v2:172.21.15.150:3300/0 conn(0x7fd75c0c5560 0x7fd75c0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3110501250 >> v1:172.21.15.150:6789/0 conn(0x7fd75c070720 legacy=0x7fd75c070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.550+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3110501250 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c153a90 con 0x7fd75c0c7910 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.551+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3110501250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c0c7910 0x7fd75c154180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7440097a0 tx=0x7fd74402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=844f937377770d26 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.551+0000 7fd759ffb640 1 -- 172.21.15.78:0/3110501250 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74402f050 con 0x7fd75c0c7910 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.551+0000 7fd759ffb640 1 -- 172.21.15.78:0/3110501250 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd744031700 con 0x7fd75c0c7910 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.551+0000 7fd759ffb640 1 -- 172.21.15.78:0/3110501250 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744031a90 con 0x7fd75c0c7910 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 -- 172.21.15.78:0/3110501250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c0c7910 msgr2=0x7fd75c154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 --2- 172.21.15.78:0/3110501250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c0c7910 0x7fd75c154180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7440097a0 tx=0x7fd74402b7a0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 -- 172.21.15.78:0/3110501250 shutdown_connections 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 --2- 172.21.15.78:0/3110501250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c0c7910 0x7fd75c154180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 --2- 172.21.15.78:0/3110501250 >> v2:172.21.15.150:3300/0 conn(0x7fd75c0c5560 0x7fd75c0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 -- 172.21.15.78:0/3110501250 >> 172.21.15.78:0/3110501250 conn(0x7fd75c06b4b0 msgr2=0x7fd75c0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 -- 172.21.15.78:0/3110501250 shutdown_connections 2024-02-19T14:57:50.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.552+0000 7fd7616d4640 1 -- 172.21.15.78:0/3110501250 wait complete. 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.553+0000 7fd7616d4640 1 Processor -- start 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.553+0000 7fd7616d4640 1 -- start start 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7616d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c070720 0x7fd75c0aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7616d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd75c0c5560 0x7fd75c1e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7616d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 0x7fd75c0becf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7616d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c0aacf0 con 0x7fd75c0c7910 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7616d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c1e9910 con 0x7fd75c0c5560 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7616d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c0aa730 con 0x7fd75c070720 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 0x7fd75c0becf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 0x7fd75c0becf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46982/0 (socket says 172.21.15.78:46982) 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 -- 172.21.15.78:0/4091098819 learned_addr learned my addr 172.21.15.78:0/4091098819 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd75c0c5560 0x7fd75c1e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75affd640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c070720 0x7fd75c0aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 -- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c070720 msgr2=0x7fd75c0aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c070720 0x7fd75c0aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 -- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd75c0c5560 msgr2=0x7fd75c1e93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd75c0c5560 0x7fd75c1e93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 -- 172.21.15.78:0/4091098819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744009400 con 0x7fd75c0c7910 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 0x7fd75c0becf0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fd75400c730 tx=0x7fd75400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.554+0000 7fd7537fe640 1 -- 172.21.15.78:0/4091098819 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75400ce40 con 0x7fd75c0c7910 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.555+0000 7fd7537fe640 1 -- 172.21.15.78:0/4091098819 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75400a640 con 0x7fd75c0c7910 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.555+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75c0aaa10 con 0x7fd75c0c7910 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.555+0000 7fd7537fe640 1 -- 172.21.15.78:0/4091098819 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75400aa30 con 0x7fd75c0c7910 2024-02-19T14:57:50.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.555+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75c0bdb80 con 0x7fd75c0c7910 2024-02-19T14:57:50.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.556+0000 7fd7537fe640 1 -- 172.21.15.78:0/4091098819 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd75400ab90 con 0x7fd75c0c7910 2024-02-19T14:57:50.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.556+0000 7fd7537fe640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd73c06b840 0x7fd73c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.557+0000 7fd7537fe640 1 -- 172.21.15.78:0/4091098819 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd754089900 con 0x7fd75c0c7910 2024-02-19T14:57:50.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.557+0000 7fd75affd640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd73c06b840 0x7fd73c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.557+0000 7fd75affd640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd73c06b840 0x7fd73c06dd00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009aa0 tx=0x7fd74c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.557+0000 7fd764391740 1 Processor -- start 2024-02-19T14:57:50.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 -- start start 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c57ae0 0x55be98c91b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c948b0 0x55be98c56860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98c11cd0 con 0x55be98c91740 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98bc4f20 con 0x55be98c57ae0 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be98bc2450 con 0x55be98c948b0 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd75affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd75affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c928b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46986/0 (socket says 172.21.15.78:46986) 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd75affd640 1 -- 172.21.15.78:0/104578552 learned_addr learned my addr 172.21.15.78:0/104578552 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:50.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c948b0 0x55be98c56860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.558+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c57ae0 0x55be98c91b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75affd640 1 -- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c948b0 msgr2=0x55be98c56860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75affd640 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c948b0 0x55be98c56860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75affd640 1 -- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c57ae0 msgr2=0x55be98c91b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75affd640 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c57ae0 0x55be98c91b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75affd640 1 -- 172.21.15.78:0/104578552 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be98b72f20 con 0x55be98c91740 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c948b0 0x55be98c56860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd75affd640 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c928b0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd74c004470 tx=0x7fd74c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62355cb231dd8e6f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd750ff9640 1 -- 172.21.15.78:0/104578552 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c01c040 con 0x55be98c91740 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd750ff9640 1 -- 172.21.15.78:0/104578552 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c003820 con 0x55be98c91740 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.559+0000 7fd750ff9640 1 -- 172.21.15.78:0/104578552 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c003b90 con 0x55be98c91740 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 -- 172.21.15.78:0/104578552 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 msgr2=0x55be98c928b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c928b0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd74c004470 tx=0x7fd74c014000 comp rx=0 tx=0).stop 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 -- 172.21.15.78:0/104578552 shutdown_connections 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c948b0 0x55be98c56860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c57ae0 0x55be98c91b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 --2- 172.21.15.78:0/104578552 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c928b0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 -- 172.21.15.78:0/104578552 >> 172.21.15.78:0/104578552 conn(0x55be98c53120 msgr2=0x55be98c77b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 -- 172.21.15.78:0/104578552 shutdown_connections 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 -- 172.21.15.78:0/104578552 wait complete. 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 -- start start 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.560+0000 7fd764391740 1 Processor -- start 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c74910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98bbdc30 con 0x55be98c91740 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98b62d30 con 0x55be98c948b0 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be98891310 con 0x55be98c87fd0 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd764391740 10 client.0 _ll_register_callbacks cb 0x55be98b6ad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46994/0 (socket says 172.21.15.78:46994) 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 -- 172.21.15.78:0/3555762301 learned_addr learned my addr 172.21.15.78:0/3555762301 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c74910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 -- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 msgr2=0x55be98c74910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c74910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 -- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 msgr2=0x55be98c743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c743d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 -- 172.21.15.78:0/3555762301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c004120 con 0x55be98c91740 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c74910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.561+0000 7fd75affd640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c563d0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x55be98c923f0 tx=0x7fd74c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5e10a0e1b6f414 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd739ffb640 1 -- 172.21.15.78:0/3555762301 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c004560 con 0x55be98c91740 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd739ffb640 1 -- 172.21.15.78:0/3555762301 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd74c003e20 con 0x55be98c91740 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be988bd6c0 con 0x55be98c91740 2024-02-19T14:57:50.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd739ffb640 1 -- 172.21.15.78:0/3555762301 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd74c022cb0 con 0x55be98c91740 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 10 client.8904 fetch_fsmap learned FSMap version 132 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) v3 -- 0x55be98bbc2b0 con 0x55be98c91740 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd739ffb640 1 -- 172.21.15.78:0/3555762301 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55be988bd6c0 con 0x55be98c91740 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd739ffb640 1 -- 172.21.15.78:0/3555762301 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 132) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd74c035470 con 0x55be98c91740 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 10 client.8904 fetch_fsmap finished waiting for FSMap version 132 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 10 client.8904 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 20 client.8904 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98ca6290 0x55be98ca6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 4 client.8904 mds_command: new command op to 8772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 --> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be9890e270 con 0x55be98ca6290 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.562+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98ca6290 0x55be98ca6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd739ffb640 10 client.8904 ms_handle_connect on v2:172.21.15.78:6835/903597448 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98ca6290 0x55be98ca6670 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4abc482373585805 server_cookie=38c21ba93c447904 in_seq=0 out_seq=0 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd739ffb640 1 -- 172.21.15.78:0/3555762301 <== mds.0 v2:172.21.15.78:6835/903597448 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55be9890e270 con 0x55be98ca6290 2024-02-19T14:57:50.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd739ffb640 10 client.8904 handle_command_reply: tid=0 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 client.8904 shutdown 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 msgr2=0x55be98c563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c563d0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x55be98c923f0 tx=0x7fd74c004730 comp rx=0 tx=0).stop 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 shutdown_connections 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98ca6290 0x55be98ca6670 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c74910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd739ffb640 0 client.8904 ms_handle_reset on v2:172.21.15.78:6835/903597448 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 --2- 172.21.15.78:0/3555762301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c563d0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 >> 172.21.15.78:0/3555762301 conn(0x55be98c57ae0 msgr2=0x55be98c77a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 shutdown_connections 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.563+0000 7fd764391740 1 -- 172.21.15.78:0/3555762301 wait complete. 2024-02-19T14:57:50.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.564+0000 7fd764391740 20 client.8904 trim_cache size 0 max 16384 2024-02-19T14:57:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.580+0000 7fd764391740 1 Processor -- start 2024-02-19T14:57:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.580+0000 7fd764391740 1 -- start start 2024-02-19T14:57:50.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c4ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd764391740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c8d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd764391740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98b43490 con 0x55be98c91740 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98bbcde0 con 0x55be98c948b0 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be98c87d50 con 0x55be98c87fd0 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd75a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c8d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd75a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c8d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47002/0 (socket says 172.21.15.78:47002) 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3880117401 learned_addr learned my addr 172.21.15.78:0/3880117401 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd75affd640 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c4ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.581+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 msgr2=0x55be98c4ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c4ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 msgr2=0x55be98c8d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75a7fc640 1 -- 172.21.15.78:0/3880117401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be98c90af0 con 0x55be98c91740 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75affd640 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c4ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c8d0b0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fd7440097a0 tx=0x7fd744008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82318136f7a5e54 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd73bfff640 1 -- 172.21.15.78:0/3880117401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74402f040 con 0x55be98c91740 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd73bfff640 1 -- 172.21.15.78:0/3880117401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7440044a0 con 0x55be98c91740 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd73bfff640 1 -- 172.21.15.78:0/3880117401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744044040 con 0x55be98c91740 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 -- 172.21.15.78:0/3880117401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 msgr2=0x55be98c8d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c8d0b0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fd7440097a0 tx=0x7fd744008030 comp rx=0 tx=0).stop 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 -- 172.21.15.78:0/3880117401 shutdown_connections 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c91740 0x55be98c8d0b0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 --2- 172.21.15.78:0/3880117401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c87fd0 0x55be98c4ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 -- 172.21.15.78:0/3880117401 >> 172.21.15.78:0/3880117401 conn(0x55be98c57ae0 msgr2=0x55be98c91b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 -- 172.21.15.78:0/3880117401 shutdown_connections 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.582+0000 7fd764391740 1 -- 172.21.15.78:0/3880117401 wait complete. 2024-02-19T14:57:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.583+0000 7fd764391740 1 -- start start 2024-02-19T14:57:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.583+0000 7fd764391740 1 Processor -- start 2024-02-19T14:57:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 0x55be98cb60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c91740 0x55be98c54290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98d05ed0 con 0x55be98c87fd0 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be98c50890 con 0x55be98c948b0 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be98ca3ab0 con 0x55be98c91740 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd764391740 10 client.0 _ll_register_callbacks cb 0x7fd74c004340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 0x55be98cb60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 0x55be98cb60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47010/0 (socket says 172.21.15.78:47010) 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75affd640 1 -- 172.21.15.78:0/3587036380 learned_addr learned my addr 172.21.15.78:0/3587036380 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75affd640 1 -- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c91740 msgr2=0x55be98c54290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c91740 0x55be98c54290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.584+0000 7fd75affd640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c91740 0x55be98c54290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75affd640 1 -- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 msgr2=0x55be98c8c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75affd640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75affd640 1 -- 172.21.15.78:0/3587036380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75c153a90 con 0x55be98c87fd0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75b7fe640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c91740 0x55be98c54290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75affd640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 0x55be98cb60e0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fd74c035720 tx=0x7fd74c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e2ebb1e0374c26 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd739ffb640 1 -- 172.21.15.78:0/3587036380 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c022670 con 0x55be98c87fd0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd739ffb640 1 -- 172.21.15.78:0/3587036380 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd74c022ab0 con 0x55be98c87fd0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd74c003820 con 0x55be98c87fd0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd739ffb640 1 -- 172.21.15.78:0/3587036380 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd74c022c70 con 0x55be98c87fd0 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 10 client.8910 fetch_fsmap learned FSMap version 132 2024-02-19T14:57:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=2+}) v3 -- 0x55be98c8e580 con 0x55be98c87fd0 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd739ffb640 1 -- 172.21.15.78:0/3587036380 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c003820 con 0x55be98c87fd0 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd739ffb640 1 -- 172.21.15.78:0/3587036380 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 132) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd74c014440 con 0x55be98c87fd0 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 10 client.8910 fetch_fsmap finished waiting for FSMap version 132 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 10 client.8910 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 20 client.8910 populate_metadata read hostname 'smithi078' 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98d02a20 0x55be98d02e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 4 client.8910 mds_command: new command op to 8772 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 --> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55be988fa610 con 0x55be98d02a20 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.585+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98d02a20 0x55be98d02e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd75a7fc640 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98d02a20 0x55be98d02e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69e2d808b02080d0 server_cookie=6e474b285c132c92 in_seq=0 out_seq=0 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd739ffb640 10 client.8910 ms_handle_connect on v2:172.21.15.78:6835/903597448 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd739ffb640 1 -- 172.21.15.78:0/3587036380 <== mds.0 v2:172.21.15.78:6835/903597448 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55be988fa610 con 0x55be98d02a20 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd739ffb640 10 client.8910 handle_command_reply: tid=0 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 client.8910 shutdown 2024-02-19T14:57:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 msgr2=0x55be98cb60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 0x55be98cb60e0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fd74c035720 tx=0x7fd74c018000 comp rx=0 tx=0).stop 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 shutdown_connections 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:6835/903597448,v1:172.21.15.78:6837/903597448] conn(0x55be98d02a20 0x55be98d02e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55be98c948b0 0x55be98c8c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd739ffb640 0 client.8910 ms_handle_reset on v2:172.21.15.78:6835/903597448 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55be98c91740 0x55be98c54290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 --2- 172.21.15.78:0/3587036380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55be98c87fd0 0x55be98cb60e0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 >> 172.21.15.78:0/3587036380 conn(0x55be98c57ae0 msgr2=0x55be98c8faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 shutdown_connections 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.586+0000 7fd764391740 1 -- 172.21.15.78:0/3587036380 wait complete. 2024-02-19T14:57:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd764391740 20 client.8910 trim_cache size 0 max 16384 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 8772, 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.8297924649999997, 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 132, 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 152, 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 152, 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 9.9450763329999994 2024-02-19T14:57:50.596 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:57:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd73c06b840 msgr2=0x7fd73c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd73c06b840 0x7fd73c06dd00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009aa0 tx=0x7fd74c009190 comp rx=0 tx=0).stop 2024-02-19T14:57:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 msgr2=0x7fd75c0becf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 0x7fd75c0becf0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fd75400c730 tx=0x7fd75400cbf0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 shutdown_connections 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd73c06b840 0x7fd73c06dd00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd75c0c7910 0x7fd75c0becf0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd75c0c5560 0x7fd75c1e93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 --2- 172.21.15.78:0/4091098819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd75c070720 0x7fd75c0aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 >> 172.21.15.78:0/4091098819 conn(0x7fd75c06b4b0 msgr2=0x7fd75c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 shutdown_connections 2024-02-19T14:57:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.587+0000 7fd7616d4640 1 -- 172.21.15.78:0/4091098819 wait complete. 2024-02-19T14:57:50.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.699+0000 7fa47e39a640 1 Processor -- start 2024-02-19T14:57:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.699+0000 7fa47e39a640 1 -- start start 2024-02-19T14:57:50.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa47e39a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa478152700 0x7fa4780c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa47e39a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4781551f0 con 0x7fa4780c5730 2024-02-19T14:57:50.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa47e39a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa478155490 con 0x7fa4780c8e00 2024-02-19T14:57:50.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa47e39a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa478155770 con 0x7fa478152700 2024-02-19T14:57:50.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa4777fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa4780c5730 0x7fa47806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58332/0 (socket says 172.21.15.78:58332) 2024-02-19T14:57:50.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa4777fe640 1 -- 172.21.15.78:0/2008479917 learned_addr learned my addr 172.21.15.78:0/2008479917 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa477fff640 1 --2- 172.21.15.78:0/2008479917 >> v2:172.21.15.150:3300/0 conn(0x7fa478152700 0x7fa4780c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768026886 0 0) 0x7fa4781551f0 con 0x7fa4780c5730 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.700+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa460003520 con 0x7fa4780c5730 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 655687321 0 0) 0x7fa460003520 con 0x7fa4780c5730 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 >> v2:172.21.15.150:3300/0 conn(0x7fa478152700 msgr2=0x7fa4780c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 --2- 172.21.15.78:0/2008479917 >> v2:172.21.15.150:3300/0 conn(0x7fa478152700 0x7fa4780c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 >> v1:172.21.15.150:6789/0 conn(0x7fa4780c8e00 legacy=0x7fa47815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa478155a50 con 0x7fa4780c5730 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fa468001670 con 0x7fa4780c5730 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 >> v1:172.21.15.78:6789/0 conn(0x7fa4780c5730 legacy=0x7fa47806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa460003970 0x7fa460003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4600047c0 0x7fa460004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa460005180 0x7fa46000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa478155770 con 0x7fa460003970 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46000c120 con 0x7fa4600047c0 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46000c2e0 con 0x7fa460005180 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa477fff640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa460003970 0x7fa460003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa4777fe640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4600047c0 0x7fa460004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa47c937640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa460005180 0x7fa46000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa477fff640 1 -- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa460005180 msgr2=0x7fa46000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa477fff640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa460005180 0x7fa46000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa477fff640 1 -- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4600047c0 msgr2=0x7fa460004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa477fff640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4600047c0 0x7fa460004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa477fff640 1 -- 172.21.15.78:0/2008479917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa460003520 con 0x7fa460003970 2024-02-19T14:57:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa47c937640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa460005180 0x7fa46000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.701+0000 7fa47c937640 1 -- 172.21.15.78:0/2008479917 reap_dead start 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.702+0000 7fa477fff640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa460003970 0x7fa460003d90 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009ac0 tx=0x7fa4600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82e7390629312f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c02c6b0 con 0x7fa460003970 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c02c810 con 0x7fa460003970 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa476ffd640 1 -- 172.21.15.78:0/2008479917 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c02cba0 con 0x7fa460003970 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 -- 172.21.15.78:0/2008479917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa460003970 msgr2=0x7fa460003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa460003970 0x7fa460003d90 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009ac0 tx=0x7fa4600057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 -- 172.21.15.78:0/2008479917 shutdown_connections 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 --2- 172.21.15.78:0/2008479917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa460003970 0x7fa460003d90 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 -- 172.21.15.78:0/2008479917 >> 172.21.15.78:0/2008479917 conn(0x7fa4780cac90 msgr2=0x7fa4780cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 -- 172.21.15.78:0/2008479917 shutdown_connections 2024-02-19T14:57:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.703+0000 7fa47e39a640 1 -- 172.21.15.78:0/2008479917 wait complete. 2024-02-19T14:57:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.704+0000 7fa47e39a640 1 Processor -- start 2024-02-19T14:57:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.704+0000 7fa47e39a640 1 -- start start 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47e39a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa460003970 0x7fa4781ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47e39a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4780aa280 0x7fa478154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47e39a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 0x7fa4780c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47e39a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4780ac1d0 con 0x7fa4780c2140 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47e39a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4781eea30 con 0x7fa460003970 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47e39a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4780c1cb0 con 0x7fa4780aa280 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47c937640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 0x7fa4780c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47c937640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 0x7fa4780c25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47018/0 (socket says 172.21.15.78:47018) 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47c937640 1 -- 172.21.15.78:0/3259453561 learned_addr learned my addr 172.21.15.78:0/3259453561 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa477fff640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa460003970 0x7fa4781ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47c937640 1 -- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4780aa280 msgr2=0x7fa478154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa4777fe640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4780aa280 0x7fa478154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.705+0000 7fa47c937640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4780aa280 0x7fa478154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa47c937640 1 -- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa460003970 msgr2=0x7fa4781ee4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa47c937640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa460003970 0x7fa4781ee4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa47c937640 1 -- 172.21.15.78:0/3259453561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780c35a0 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa477fff640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa460003970 0x7fa4781ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa47c937640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 0x7fa4780c25c0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fa47808c490 tx=0x7fa47000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47000eed0 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470003970 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4780c3840 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470003d20 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.706+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4780c1940 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.707+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4780c1e40 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.707+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa4700050d0 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.708+0000 7fa474ff9640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa46406b810 0x7fa46406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.708+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa47008a090 con 0x7fa4780c2140 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.708+0000 7fa477fff640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa46406b810 0x7fa46406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.708+0000 7fa477fff640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa46406b810 0x7fa46406dcd0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa46c036000 tx=0x7fa46c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.712+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa470054920 con 0x7fa4780c2140 2024-02-19T14:57:50.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.891+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa478074990 con 0x7fa4780c2140 2024-02-19T14:57:50.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.892+0000 7fa474ff9640 1 -- 172.21.15.78:0/3259453561 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16872 (secure 0 0 0) 0x7fa470057fe0 con 0x7fa4780c2140 2024-02-19T14:57:50.895 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:50.895 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":152,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:46.756992+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa46406b810 msgr2=0x7fa46406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa46406b810 0x7fa46406dcd0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa46c036000 tx=0x7fa46c030000 comp rx=0 tx=0).stop 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 msgr2=0x7fa4780c25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 0x7fa4780c25c0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fa47808c490 tx=0x7fa47000eb90 comp rx=0 tx=0).stop 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 shutdown_connections 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa46406b810 0x7fa46406dcd0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780c2140 0x7fa4780c25c0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4780aa280 0x7fa478154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 --2- 172.21.15.78:0/3259453561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa460003970 0x7fa4781ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 >> 172.21.15.78:0/3259453561 conn(0x7fa4780cac90 msgr2=0x7fa4780cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:50.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 shutdown_connections 2024-02-19T14:57:50.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:50.894+0000 7fa47e39a640 1 -- 172.21.15.78:0/3259453561 wait complete. 2024-02-19T14:57:50.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:57:43.310860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:57:44.061255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:57:50.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:57:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 Processor -- start 2024-02-19T14:57:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 -- start start 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82800c8e00 0x7f828006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f828006d910 0x7f828006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828006cd20 con 0x7f8280152700 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82801551f0 con 0x7f828006d910 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.010+0000 7f828721c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8280155490 con 0x7f82800c8e00 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f82857b9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f828006d910 0x7f828006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f8284fb8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8280152700 0x7f82800c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58336/0 (socket says 172.21.15.78:58336) 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f8284fb8640 1 -- 172.21.15.78:0/45218064 learned_addr learned my addr 172.21.15.78:0/45218064 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f8277fff640 1 --2- 172.21.15.78:0/45218064 >> v2:172.21.15.150:3300/0 conn(0x7f82800c8e00 0x7f828006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f82857b9640 1 -- 172.21.15.78:0/45218064 >> v2:172.21.15.150:3300/0 conn(0x7f82800c8e00 msgr2=0x7f828006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f82857b9640 1 --2- 172.21.15.78:0/45218064 >> v2:172.21.15.150:3300/0 conn(0x7f82800c8e00 0x7f828006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f82857b9640 1 -- 172.21.15.78:0/45218064 >> v1:172.21.15.78:6789/0 conn(0x7f8280152700 legacy=0x7f82800c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f82857b9640 1 -- 172.21.15.78:0/45218064 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280155770 con 0x7f828006d910 2024-02-19T14:57:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.011+0000 7f82777fe640 1 -- 172.21.15.78:0/45218064 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164605372 0 0) 0x7f828006cd20 con 0x7f8280152700 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f82857b9640 1 --2- 172.21.15.78:0/45218064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f828006d910 0x7f828006c7e0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f82800c7620 tx=0x7f827802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdb1328f024466d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f82777fe640 1 -- 172.21.15.78:0/45218064 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827802cc80 con 0x7f828006d910 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f82777fe640 1 -- 172.21.15.78:0/45218064 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827802cde0 con 0x7f828006d910 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f82777fe640 1 -- 172.21.15.78:0/45218064 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278034660 con 0x7f828006d910 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 -- 172.21.15.78:0/45218064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f828006d910 msgr2=0x7f828006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 --2- 172.21.15.78:0/45218064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f828006d910 0x7f828006c7e0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f82800c7620 tx=0x7f827802bed0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 -- 172.21.15.78:0/45218064 shutdown_connections 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 --2- 172.21.15.78:0/45218064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f828006d910 0x7f828006c7e0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 --2- 172.21.15.78:0/45218064 >> v2:172.21.15.150:3300/0 conn(0x7f82800c8e00 0x7f828006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 -- 172.21.15.78:0/45218064 >> 172.21.15.78:0/45218064 conn(0x7f82800cac90 msgr2=0x7f82800cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 -- 172.21.15.78:0/45218064 shutdown_connections 2024-02-19T14:57:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.012+0000 7f828721c640 1 -- 172.21.15.78:0/45218064 wait complete. 2024-02-19T14:57:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.013+0000 7f828721c640 1 Processor -- start 2024-02-19T14:57:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.013+0000 7f828721c640 1 -- start start 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f828721c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f828006d910 0x7f82801eaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f828721c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 0x7f8280149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f828721c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8280152700 0x7f82801eddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f828721c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82801477b0 con 0x7f82800c8e00 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f828721c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800c4270 con 0x7f8280152700 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f828721c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82801ed930 con 0x7f828006d910 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f8277fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 0x7f8280149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f8277fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 0x7f8280149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47036/0 (socket says 172.21.15.78:47036) 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.014+0000 7f8277fff640 1 -- 172.21.15.78:0/2757634656 learned_addr learned my addr 172.21.15.78:0/2757634656 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f82857b9640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8280152700 0x7f82801eddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8284fb8640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f828006d910 0x7f82801eaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8277fff640 1 -- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f828006d910 msgr2=0x7f82801eaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8277fff640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f828006d910 0x7f82801eaf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8277fff640 1 -- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8280152700 msgr2=0x7f82801eddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8277fff640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8280152700 0x7f82801eddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8277fff640 1 -- 172.21.15.78:0/2757634656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8278009a70 con 0x7f82800c8e00 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8284fb8640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f828006d910 0x7f82801eaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f8277fff640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 0x7f8280149890 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f827000d800 tx=0x7f827000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827000b6c0 con 0x7f82800c8e00 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827000b840 con 0x7f82800c8e00 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827000bbf0 con 0x7f82800c8e00 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82800acee0 con 0x7f82800c8e00 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.015+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82800c47a0 con 0x7f82800c8e00 2024-02-19T14:57:51.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.016+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82801edaa0 con 0x7f82800c8e00 2024-02-19T14:57:51.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.017+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f827000bd50 con 0x7f82800c8e00 2024-02-19T14:57:51.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.017+0000 7f82757fa640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f826c06b810 0x7f826c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.017+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f827008a8a0 con 0x7f82800c8e00 2024-02-19T14:57:51.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.017+0000 7f8284fb8640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f826c06b810 0x7f826c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.017+0000 7f8284fb8640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f826c06b810 0x7f826c06dcd0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f827c000f60 tx=0x7f827c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.022+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f82700551c0 con 0x7f82800c8e00 2024-02-19T14:57:51.024 INFO:tasks.ceph.mds.a.smithi078.stderr:2024-02-19T14:57:51.022+0000 7fedcee23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:51.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.198+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8280074990 con 0x7f82800c8e00 2024-02-19T14:57:51.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.199+0000 7f82757fa640 1 -- 172.21.15.78:0/2757634656 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v152) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8270011340 con 0x7f82800c8e00 2024-02-19T14:57:51.202 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:57:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f826c06b810 msgr2=0x7f826c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f826c06b810 0x7f826c06dcd0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f827c000f60 tx=0x7f827c009040 comp rx=0 tx=0).stop 2024-02-19T14:57:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 msgr2=0x7f8280149890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 0x7f8280149890 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f827000d800 tx=0x7f827000dcc0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 shutdown_connections 2024-02-19T14:57:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f826c06b810 0x7f826c06dcd0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82800c8e00 0x7f8280149890 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8280152700 0x7f82801eddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 --2- 172.21.15.78:0/2757634656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f828006d910 0x7f82801eaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 >> 172.21.15.78:0/2757634656 conn(0x7f82800cac90 msgr2=0x7f828014b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 shutdown_connections 2024-02-19T14:57:51.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.201+0000 7f828721c640 1 -- 172.21.15.78:0/2757634656 wait complete. 2024-02-19T14:57:51.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:57:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.319+0000 7f3caf40e640 1 Processor -- start 2024-02-19T14:57:51.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.319+0000 7f3caf40e640 1 -- start start 2024-02-19T14:57:51.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3caf40e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3ca806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3caf40e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ca806d910 0x7f3ca806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3caf40e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca806cd20 con 0x7f3ca8152700 2024-02-19T14:57:51.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3caf40e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca81551f0 con 0x7f3ca806d910 2024-02-19T14:57:51.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3caf40e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8155490 con 0x7f3ca80c8e00 2024-02-19T14:57:51.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3cad1aa640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3ca8152700 0x7f3ca80c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58342/0 (socket says 172.21.15.78:58342) 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3cad1aa640 1 -- 172.21.15.78:0/3947070393 learned_addr learned my addr 172.21.15.78:0/3947070393 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968515621 0 0) 0x7f3ca806cd20 con 0x7f3ca8152700 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/3947070393 >> v2:172.21.15.150:3300/0 conn(0x7f3ca806d910 0x7f3ca806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c94003520 con 0x7f3ca8152700 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3cac9a9640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3ca806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2957848779 0 0) 0x7f3c94003520 con 0x7f3ca8152700 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 msgr2=0x7f3ca806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3ca806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 >> v2:172.21.15.150:3300/0 conn(0x7f3ca806d910 msgr2=0x7f3ca806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 --2- 172.21.15.78:0/3947070393 >> v2:172.21.15.150:3300/0 conn(0x7f3ca806d910 0x7f3ca806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8155770 con 0x7f3ca8152700 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3ca4001670 con 0x7f3ca8152700 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 >> v1:172.21.15.78:6789/0 conn(0x7f3ca8152700 legacy=0x7f3ca80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.320+0000 7f3c9ffff640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c94003970 0x7f3c94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3c9ffff640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3c940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3c9ffff640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c94004d00 0x7f3c94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca81551f0 con 0x7f3c94004d00 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8155490 con 0x7f3c94003970 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9400b980 con 0x7f3ca80c8e00 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c94004d00 0x7f3c94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad1aa640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c94003970 0x7f3c94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cac9a9640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3c940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad9ab640 1 -- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 msgr2=0x7f3c940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3c940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad9ab640 1 -- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c94003970 msgr2=0x7f3c94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c94003970 0x7f3c94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cad9ab640 1 -- 172.21.15.78:0/3947070393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c94003520 con 0x7f3c94004d00 2024-02-19T14:57:51.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.321+0000 7f3cac9a9640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c8e00 0x7f3c940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:51.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.322+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c94004d00 0x7f3c94005180 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009fc0 tx=0x7f3c940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a74de94b3b57cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.322+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca002cc60 con 0x7f3c94004d00 2024-02-19T14:57:51.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.322+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca000f040 con 0x7f3c94004d00 2024-02-19T14:57:51.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.322+0000 7f3c9ffff640 1 -- 172.21.15.78:0/3947070393 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0034440 con 0x7f3c94004d00 2024-02-19T14:57:51.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3caf40e640 1 -- 172.21.15.78:0/3947070393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c94004d00 msgr2=0x7f3c94005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3caf40e640 1 --2- 172.21.15.78:0/3947070393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c94004d00 0x7f3c94005180 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009fc0 tx=0x7f3c940057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3cad9ab640 1 -- 172.21.15.78:0/3947070393 reap_dead start 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3caf40e640 1 -- 172.21.15.78:0/3947070393 shutdown_connections 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3caf40e640 1 -- 172.21.15.78:0/3947070393 >> 172.21.15.78:0/3947070393 conn(0x7f3ca80cac90 msgr2=0x7f3ca80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3caf40e640 1 -- 172.21.15.78:0/3947070393 shutdown_connections 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.323+0000 7f3caf40e640 1 -- 172.21.15.78:0/3947070393 wait complete. 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.324+0000 7f3caf40e640 1 Processor -- start 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.325+0000 7f3caf40e640 1 -- start start 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c5360 0x7f3ca81ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca80af4b0 0x7f3ca80af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 0x7f3ca81ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca80c3550 con 0x7f3ca80afe70 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8158b80 con 0x7f3ca80af4b0 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca81eeb20 con 0x7f3ca80c5360 2024-02-19T14:57:51.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 0x7f3ca81ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 0x7f3ca81ea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47042/0 (socket says 172.21.15.78:47042) 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 -- 172.21.15.78:0/1637296200 learned_addr learned my addr 172.21.15.78:0/1637296200 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cac9a9640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca80af4b0 0x7f3ca80af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad1aa640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c5360 0x7f3ca81ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 -- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c5360 msgr2=0x7f3ca81ee5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c5360 0x7f3ca81ee5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 -- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca80af4b0 msgr2=0x7f3ca80af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca80af4b0 0x7f3ca80af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 -- 172.21.15.78:0/1637296200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8158e20 con 0x7f3ca80afe70 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad1aa640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c5360 0x7f3ca81ee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3cad9ab640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 0x7f3ca81ea7c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0037040 tx=0x7f3ca0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0034cd0 con 0x7f3ca80afe70 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca0034e30 con 0x7f3ca80afe70 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca8157c60 con 0x7f3ca80afe70 2024-02-19T14:57:51.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.326+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0003bd0 con 0x7f3ca80afe70 2024-02-19T14:57:51.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.327+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca8159100 con 0x7f3ca80afe70 2024-02-19T14:57:51.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.328+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca81ea630 con 0x7f3ca80afe70 2024-02-19T14:57:51.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.328+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3ca0003d30 con 0x7f3ca80afe70 2024-02-19T14:57:51.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.328+0000 7f3c9dffb640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c8c06b840 0x7f3c8c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.329+0000 7f3cad1aa640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c8c06b840 0x7f3c8c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.329+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ca00abaa0 con 0x7f3ca80afe70 2024-02-19T14:57:51.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.329+0000 7f3cad1aa640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c8c06b840 0x7f3c8c06dd00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4004840 tx=0x7f3ca4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.337+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ca00763e0 con 0x7f3ca80afe70 2024-02-19T14:57:51.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.603+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ca8074990 con 0x7f3ca80afe70 2024-02-19T14:57:51.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.604+0000 7f3c9dffb640 1 -- 172.21.15.78:0/1637296200 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v152) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ca0079aa0 con 0x7f3ca80afe70 2024-02-19T14:57:51.607 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c8c06b840 msgr2=0x7f3c8c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c8c06b840 0x7f3c8c06dd00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4004840 tx=0x7f3ca4009040 comp rx=0 tx=0).stop 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 msgr2=0x7f3ca81ea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 0x7f3ca81ea7c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0037040 tx=0x7f3ca0009750 comp rx=0 tx=0).stop 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 shutdown_connections 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3c8c06b840 0x7f3c8c06dd00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ca80afe70 0x7f3ca81ea7c0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca80af4b0 0x7f3ca80af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 --2- 172.21.15.78:0/1637296200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ca80c5360 0x7f3ca81ee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 >> 172.21.15.78:0/1637296200 conn(0x7f3ca80cac90 msgr2=0x7f3ca814e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 shutdown_connections 2024-02-19T14:57:51.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.606+0000 7f3caf40e640 1 -- 172.21.15.78:0/1637296200 wait complete. 2024-02-19T14:57:51.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:57:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.720+0000 7fdadfd0c640 1 Processor -- start 2024-02-19T14:57:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.720+0000 7fdadfd0c640 1 -- start start 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadfd0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdad80c5130 0x7fdad80c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadfd0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdad8158f80 0x7fdad80c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadfd0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadfd0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad8158b60 con 0x7fdad8158f80 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadfd0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad80c3cf0 con 0x7fdad80c5130 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadfd0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad8159400 con 0x7fdad81559c0 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdad8158f80 0x7fdad80c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdad8158f80 0x7fdad80c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47052/0 (socket says 172.21.15.78:47052) 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 -- 172.21.15.78:0/1883182059 learned_addr learned my addr 172.21.15.78:0/1883182059 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdaddaa8640 1 --2- 172.21.15.78:0/1883182059 >> v2:172.21.15.150:3300/0 conn(0x7fdad80c5130 0x7fdad80c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdade2a9640 1 --2- 172.21.15.78:0/1883182059 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 -- 172.21.15.78:0/1883182059 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 msgr2=0x7fdad80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 --2- 172.21.15.78:0/1883182059 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 -- 172.21.15.78:0/1883182059 >> v2:172.21.15.150:3300/0 conn(0x7fdad80c5130 msgr2=0x7fdad80c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 --2- 172.21.15.78:0/1883182059 >> v2:172.21.15.150:3300/0 conn(0x7fdad80c5130 0x7fdad80c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdadd2a7640 1 -- 172.21.15.78:0/1883182059 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad8158530 con 0x7fdad8158f80 2024-02-19T14:57:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.721+0000 7fdaddaa8640 1 --2- 172.21.15.78:0/1883182059 >> v2:172.21.15.150:3300/0 conn(0x7fdad80c5130 0x7fdad80c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadd2a7640 1 --2- 172.21.15.78:0/1883182059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad80c5510 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fdad80c9130 tx=0x7fdac802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e9fb7e33548179 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadcaa6640 1 -- 172.21.15.78:0/1883182059 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8031870 con 0x7fdad8158f80 2024-02-19T14:57:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadcaa6640 1 -- 172.21.15.78:0/1883182059 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac80319d0 con 0x7fdad8158f80 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadcaa6640 1 -- 172.21.15.78:0/1883182059 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8031d60 con 0x7fdad8158f80 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1883182059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 msgr2=0x7fdad80c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1883182059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad80c5510 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fdad80c9130 tx=0x7fdac802b910 comp rx=0 tx=0).stop 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1883182059 shutdown_connections 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1883182059 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1883182059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad80c5510 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1883182059 >> v2:172.21.15.150:3300/0 conn(0x7fdad80c5130 0x7fdad80c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.722+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1883182059 >> 172.21.15.78:0/1883182059 conn(0x7fdad806b4b0 msgr2=0x7fdad80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.723+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1883182059 shutdown_connections 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.723+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1883182059 wait complete. 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 Processor -- start 2024-02-19T14:57:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 -- start start 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad80c5130 0x7fdad8146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80aa240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad8070520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad81ee830 con 0x7fdad8158f80 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad8146590 con 0x7fdad80c5130 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdadfd0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad81edc40 con 0x7fdad81559c0 2024-02-19T14:57:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdade2a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdade2a9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad8070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47068/0 (socket says 172.21.15.78:47068) 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.724+0000 7fdade2a9640 1 -- 172.21.15.78:0/1855697663 learned_addr learned my addr 172.21.15.78:0/1855697663 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdaddaa8640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad80c5130 0x7fdad8146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdadd2a7640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdade2a9640 1 -- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 msgr2=0x7fdad80aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdade2a9640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdade2a9640 1 -- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad80c5130 msgr2=0x7fdad8146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdade2a9640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad80c5130 0x7fdad8146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdade2a9640 1 -- 172.21.15.78:0/1855697663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac80094d0 con 0x7fdad8158f80 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdaddaa8640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad80c5130 0x7fdad8146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdade2a9640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad8070520 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fdad000cd70 tx=0x7fdad000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0003b90 con 0x7fdad8158f80 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad0003cf0 con 0x7fdad8158f80 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad80aa780 con 0x7fdad8158f80 2024-02-19T14:57:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0004240 con 0x7fdad8158f80 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.725+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad81ee170 con 0x7fdad8158f80 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.726+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad81ee970 con 0x7fdad8158f80 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.726+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdad00043a0 con 0x7fdad8158f80 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.727+0000 7fdace7fc640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdabc06b810 0x7fdabc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.727+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdad0012070 con 0x7fdad8158f80 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.727+0000 7fdaddaa8640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdabc06b810 0x7fdabc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:51.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.727+0000 7fdaddaa8640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdabc06b810 0x7fdabc06dcd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdad80c8480 tx=0x7fdad4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:51.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.731+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdad81ee970 con 0x7fdad8158f80 2024-02-19T14:57:51.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.907+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdad81ee970 con 0x7fdad8158f80 2024-02-19T14:57:51.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.907+0000 7fdace7fc640 1 -- 172.21.15.78:0/1855697663 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v152) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdad81ee970 con 0x7fdad8158f80 2024-02-19T14:57:51.910 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:57:51.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdabc06b810 msgr2=0x7fdabc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdabc06b810 0x7fdabc06dcd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdad80c8480 tx=0x7fdad4008040 comp rx=0 tx=0).stop 2024-02-19T14:57:51.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 msgr2=0x7fdad8070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:51.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad8070520 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fdad000cd70 tx=0x7fdad000a660 comp rx=0 tx=0).stop 2024-02-19T14:57:51.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 shutdown_connections 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdabc06b810 0x7fdabc06dcd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdad8158f80 0x7fdad8070520 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdad81559c0 0x7fdad80aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 --2- 172.21.15.78:0/1855697663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad80c5130 0x7fdad8146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 >> 172.21.15.78:0/1855697663 conn(0x7fdad806b4b0 msgr2=0x7fdad814abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 shutdown_connections 2024-02-19T14:57:51.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:51.909+0000 7fdadfd0c640 1 -- 172.21.15.78:0/1855697663 wait complete. 2024-02-19T14:57:51.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:57:52.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.021+0000 7f378cca4640 1 Processor -- start 2024-02-19T14:57:52.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.021+0000 7f378cca4640 1 -- start start 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378cca4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 0x7f37880c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378cca4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f378814ef50 0x7f378814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378cca4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378814f8f0 con 0x7f37880727d0 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378cca4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37880c35c0 con 0x7f378814ef50 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378cca4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37880c38a0 con 0x7f37880c78a0 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f377dd9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f378814ef50 0x7f378814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 0x7f37880c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f3786d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f37880727d0 0x7f3788072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:54238/0 (socket says 172.21.15.78:54238) 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f378659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 0x7f37880c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51604/0 (socket says 172.21.15.78:51604) 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.022+0000 7f3786d9d640 1 -- 172.21.15.78:0/4136201756 learned_addr learned my addr 172.21.15.78:0/4136201756 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.023+0000 7f377dd9b640 1 -- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 msgr2=0x7f37880c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.023+0000 7f377dd9b640 1 --2- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 0x7f37880c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.023+0000 7f377dd9b640 1 -- 172.21.15.78:0/4136201756 >> v1:172.21.15.150:6789/0 conn(0x7f37880727d0 legacy=0x7f3788072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.023+0000 7f377dd9b640 1 -- 172.21.15.78:0/4136201756 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37880c3b80 con 0x7f378814ef50 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.023+0000 7f3785d9b640 1 -- 172.21.15.78:0/4136201756 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258772840 0 0) 0x7f378814f8f0 con 0x7f37880727d0 2024-02-19T14:57:52.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.023+0000 7f378659c640 1 --2- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 0x7f37880c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:52.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f377dd9b640 1 --2- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f378814ef50 0x7f378814f3b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f37700097a0 tx=0x7f377002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddcf2a1a20712f2c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f3785d9b640 1 -- 172.21.15.78:0/4136201756 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770031830 con 0x7f378814ef50 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f3785d9b640 1 -- 172.21.15.78:0/4136201756 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770031990 con 0x7f378814ef50 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f3785d9b640 1 -- 172.21.15.78:0/4136201756 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770031d20 con 0x7f378814ef50 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 -- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f378814ef50 msgr2=0x7f378814f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 --2- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f378814ef50 0x7f378814f3b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f37700097a0 tx=0x7f377002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 -- 172.21.15.78:0/4136201756 shutdown_connections 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 --2- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f378814ef50 0x7f378814f3b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 --2- 172.21.15.78:0/4136201756 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37880c78a0 0x7f37880c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 -- 172.21.15.78:0/4136201756 >> 172.21.15.78:0/4136201756 conn(0x7f3788066e70 msgr2=0x7f37880a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 -- 172.21.15.78:0/4136201756 shutdown_connections 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.024+0000 7f378cca4640 1 -- 172.21.15.78:0/4136201756 wait complete. 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 Processor -- start 2024-02-19T14:57:52.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 -- start start 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 0x7f3788148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37880c78a0 0x7f37881ee330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f378814ef50 0x7f37881507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37881eab30 con 0x7f37880727d0 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3788149650 con 0x7f37880c78a0 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.026+0000 7f378cca4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3788149370 con 0x7f378814ef50 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 0x7f3788148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 0x7f3788148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47074/0 (socket says 172.21.15.78:47074) 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 -- 172.21.15.78:0/2661218097 learned_addr learned my addr 172.21.15.78:0/2661218097 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:52.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f377dd9b640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37880c78a0 0x7f37881ee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f3786d9d640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f378814ef50 0x7f37881507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 -- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f378814ef50 msgr2=0x7f37881507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f378814ef50 0x7f37881507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 -- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37880c78a0 msgr2=0x7f37881ee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37880c78a0 0x7f37881ee330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 -- 172.21.15.78:0/2661218097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3778009400 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378659c640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 0x7f3788148c90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f377800ebe0 tx=0x7f377800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778009760 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377800cc30 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3770009400 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377800eee0 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.027+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37881ea7c0 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.028+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37881ea340 con 0x7f37880727d0 2024-02-19T14:57:52.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.029+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3778013470 con 0x7f37880727d0 2024-02-19T14:57:52.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.029+0000 7f377f7fe640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f376806b840 0x7f376806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.029+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f377808a570 con 0x7f37880727d0 2024-02-19T14:57:52.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.029+0000 7f377dd9b640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f376806b840 0x7f376806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.030+0000 7f377dd9b640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f376806b840 0x7f376806dd00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3770036000 tx=0x7f377003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.034+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f37881ea340 con 0x7f37880727d0 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.212+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f37881ea340 con 0x7f37880727d0 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.212+0000 7f377f7fe640 1 -- 172.21.15.78:0/2661218097 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v152) v1 ==== 84+0+209 (secure 0 0 0) 0x7f37881ea340 con 0x7f37880727d0 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:52.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 23, 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": "cephfs_metadata" 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 24, 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": "cephfs_data" 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:57:52.216 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f376806b840 msgr2=0x7f376806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f376806b840 0x7f376806dd00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3770036000 tx=0x7f377003c040 comp rx=0 tx=0).stop 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 msgr2=0x7f3788148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 0x7f3788148c90 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f377800ebe0 tx=0x7f377800c520 comp rx=0 tx=0).stop 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 shutdown_connections 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f376806b840 0x7f376806dd00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f378814ef50 0x7f37881507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37880c78a0 0x7f37881ee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 --2- 172.21.15.78:0/2661218097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37880727d0 0x7f3788148c90 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 >> 172.21.15.78:0/2661218097 conn(0x7f3788066e70 msgr2=0x7f3788151980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:52.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 shutdown_connections 2024-02-19T14:57:52.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.214+0000 7f378cca4640 1 -- 172.21.15.78:0/2661218097 wait complete. 2024-02-19T14:57:52.228 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-02-19T14:57:52.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-02-19T14:57:52.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 Processor -- start 2024-02-19T14:57:52.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 -- start start 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbce814efa0 0x7fbce806b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce806b720 0x7fbce81555c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce8073520 con 0x7fbce80c7f40 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce80c0ad0 con 0x7fbce814efa0 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.331+0000 7fbcedbd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce814f380 con 0x7fbce806b720 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbce80c7f40 0x7fbce8154eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58358/0 (socket says 172.21.15.78:58358) 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce6ffd640 1 -- 172.21.15.78:0/2996761424 learned_addr learned my addr 172.21.15.78:0/2996761424 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce77fe640 1 --2- 172.21.15.78:0/2996761424 >> v2:172.21.15.150:3300/0 conn(0x7fbce814efa0 0x7fbce806b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce7fff640 1 --2- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce806b720 0x7fbce81555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce67fc640 1 -- 172.21.15.78:0/2996761424 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244858269 0 0) 0x7fbce8073520 con 0x7fbce80c7f40 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce67fc640 1 -- 172.21.15.78:0/2996761424 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd0003520 con 0x7fbce80c7f40 2024-02-19T14:57:52.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce77fe640 1 -- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce806b720 msgr2=0x7fbce81555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:52.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce77fe640 1 --2- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce806b720 0x7fbce81555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce77fe640 1 -- 172.21.15.78:0/2996761424 >> v1:172.21.15.78:6789/0 conn(0x7fbce80c7f40 legacy=0x7fbce8154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.332+0000 7fbce77fe640 1 -- 172.21.15.78:0/2996761424 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce80c1240 con 0x7fbce814efa0 2024-02-19T14:57:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.333+0000 7fbce77fe640 1 --2- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce814efa0 0x7fbce806b1e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80097a0 tx=0x7fbcd802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9613073d1e72f78 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.333+0000 7fbce67fc640 1 -- 172.21.15.78:0/2996761424 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd802e070 con 0x7fbce814efa0 2024-02-19T14:57:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.333+0000 7fbce67fc640 1 -- 172.21.15.78:0/2996761424 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd80315a0 con 0x7fbce814efa0 2024-02-19T14:57:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.333+0000 7fbce67fc640 1 -- 172.21.15.78:0/2996761424 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8031930 con 0x7fbce814efa0 2024-02-19T14:57:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 -- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce814efa0 msgr2=0x7fbce806b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce814efa0 0x7fbce806b1e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80097a0 tx=0x7fbcd802b840 comp rx=0 tx=0).stop 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 -- 172.21.15.78:0/2996761424 shutdown_connections 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce806b720 0x7fbce81555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/2996761424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce814efa0 0x7fbce806b1e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 -- 172.21.15.78:0/2996761424 >> 172.21.15.78:0/2996761424 conn(0x7fbce806bf90 msgr2=0x7fbce806c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 -- 172.21.15.78:0/2996761424 shutdown_connections 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.334+0000 7fbcedbd0640 1 -- 172.21.15.78:0/2996761424 wait complete. 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 Processor -- start 2024-02-19T14:57:52.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 -- start start 2024-02-19T14:57:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce806b720 0x7fbce80c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce80c7f40 0x7fbce80abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 0x7fbce80ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce80ad3a0 con 0x7fbce814efa0 2024-02-19T14:57:52.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81590c0 con 0x7fbce806b720 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.335+0000 7fbcedbd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8158790 con 0x7fbce80c7f40 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 0x7fbce80ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 0x7fbce80ac280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47082/0 (socket says 172.21.15.78:47082) 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 -- 172.21.15.78:0/3794847053 learned_addr learned my addr 172.21.15.78:0/3794847053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce77fe640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce806b720 0x7fbce80c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce6ffd640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce80c7f40 0x7fbce80abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 -- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce80c7f40 msgr2=0x7fbce80abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce80c7f40 0x7fbce80abd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 -- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce806b720 msgr2=0x7fbce80c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce806b720 0x7fbce80c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 -- 172.21.15.78:0/3794847053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8009400 con 0x7fbce814efa0 2024-02-19T14:57:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbce7fff640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 0x7fbce80ac280 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fbce000efc0 tx=0x7fbce000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0014480 con 0x7fbce814efa0 2024-02-19T14:57:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce000f040 con 0x7fbce814efa0 2024-02-19T14:57:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce8158a70 con 0x7fbce814efa0 2024-02-19T14:57:52.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0014820 con 0x7fbce814efa0 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.336+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce80c2750 con 0x7fbce814efa0 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.337+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce8159270 con 0x7fbce814efa0 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.338+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbce0014980 con 0x7fbce814efa0 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.338+0000 7fbccffff640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcd406b810 0x7fbcd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.338+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbce008b090 con 0x7fbce814efa0 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.338+0000 7fbce77fe640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcd406b810 0x7fbcd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.339+0000 7fbce77fe640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcd406b810 0x7fbcd406dcd0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8036000 tx=0x7fbcd803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.346+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbce8159270 con 0x7fbce814efa0 2024-02-19T14:57:52.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.521+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbce8159230 con 0x7fbce814efa0 2024-02-19T14:57:52.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.796+0000 7fbccffff640 1 -- 172.21.15.78:0/3794847053 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v153) v1 ==== 140+0+0 (secure 0 0 0) 0x7fbce00180b0 con 0x7fbce814efa0 2024-02-19T14:57:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_metadata' created 2024-02-19T14:57:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.798+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcd406b810 msgr2=0x7fbcd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.798+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcd406b810 0x7fbcd406dcd0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8036000 tx=0x7fbcd803d040 comp rx=0 tx=0).stop 2024-02-19T14:57:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.798+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 msgr2=0x7fbce80ac280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.798+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 0x7fbce80ac280 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fbce000efc0 tx=0x7fbce000c490 comp rx=0 tx=0).stop 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 shutdown_connections 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcd406b810 0x7fbcd406dcd0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbce814efa0 0x7fbce80ac280 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbce80c7f40 0x7fbce80abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 --2- 172.21.15.78:0/3794847053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce806b720 0x7fbce80c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 >> 172.21.15.78:0/3794847053 conn(0x7fbce806bf90 msgr2=0x7fbce814e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 shutdown_connections 2024-02-19T14:57:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.799+0000 7fbcedbd0640 1 -- 172.21.15.78:0/3794847053 wait complete. 2024-02-19T14:57:52.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:57:52.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 Processor -- start 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 -- start start 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f68150720 0x7f7f680c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f680c4c50 0x7f7f680c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6806fc00 con 0x7f7f68070270 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f68070bf0 con 0x7f7f680c4c50 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.922+0000 7f7f6e7b6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f680c4700 con 0x7f7f68150720 2024-02-19T14:57:52.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f680c4c50 0x7f7f680c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f680c4c50 0x7f7f680c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47084/0 (socket says 172.21.15.78:47084) 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 -- 172.21.15.78:0/807528356 learned_addr learned my addr 172.21.15.78:0/807528356 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 -- 172.21.15.78:0/807528356 >> v2:172.21.15.150:3300/0 conn(0x7f7f68150720 msgr2=0x7f7f680c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f67fff640 1 --2- 172.21.15.78:0/807528356 >> v2:172.21.15.150:3300/0 conn(0x7f7f68150720 0x7f7f680c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 --2- 172.21.15.78:0/807528356 >> v2:172.21.15.150:3300/0 conn(0x7f7f68150720 0x7f7f680c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 -- 172.21.15.78:0/807528356 >> v1:172.21.15.150:6789/0 conn(0x7f7f68070270 legacy=0x7f7f680c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f677fe640 1 -- 172.21.15.78:0/807528356 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f68153a60 con 0x7f7f680c4c50 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.923+0000 7f7f67fff640 1 --2- 172.21.15.78:0/807528356 >> v2:172.21.15.150:3300/0 conn(0x7f7f68150720 0x7f7f680c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f677fe640 1 --2- 172.21.15.78:0/807528356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f680c4c50 0x7f7f680c6a10 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f7f6806ea90 tx=0x7f7f5c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce241e6d92063817 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f66ffd640 1 -- 172.21.15.78:0/807528356 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c0317e0 con 0x7f7f680c4c50 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f66ffd640 1 -- 172.21.15.78:0/807528356 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c031940 con 0x7f7f680c4c50 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f66ffd640 1 -- 172.21.15.78:0/807528356 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c031cd0 con 0x7f7f680c4c50 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/807528356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f680c4c50 msgr2=0x7f7f680c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/807528356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f680c4c50 0x7f7f680c6a10 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f7f6806ea90 tx=0x7f7f5c02b880 comp rx=0 tx=0).stop 2024-02-19T14:57:52.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/807528356 shutdown_connections 2024-02-19T14:57:52.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/807528356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f680c4c50 0x7f7f680c6a10 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/807528356 >> v2:172.21.15.150:3300/0 conn(0x7f7f68150720 0x7f7f680c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/807528356 >> 172.21.15.78:0/807528356 conn(0x7f7f680c5050 msgr2=0x7f7f680c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:52.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/807528356 shutdown_connections 2024-02-19T14:57:52.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.924+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/807528356 wait complete. 2024-02-19T14:57:52.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.925+0000 7f7f6e7b6640 1 Processor -- start 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 -- start start 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f68070270 0x7f7f6814d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f680c4c50 0x7f7f680c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 0x7f7f681457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6814de80 con 0x7f7f68150720 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6814b900 con 0x7f7f68070270 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6e7b6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f681eada0 con 0x7f7f680c4c50 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.926+0000 7f7f6cd53640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 0x7f7f681457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 0x7f7f681457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47100/0 (socket says 172.21.15.78:47100) 2024-02-19T14:57:52.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 -- 172.21.15.78:0/227192715 learned_addr learned my addr 172.21.15.78:0/227192715 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f67fff640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f68070270 0x7f7f6814d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f677fe640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f680c4c50 0x7f7f680c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 -- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f680c4c50 msgr2=0x7f7f680c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f680c4c50 0x7f7f680c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 -- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f68070270 msgr2=0x7f7f6814d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f68070270 0x7f7f6814d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 -- 172.21.15.78:0/227192715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0094d0 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f677fe640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f680c4c50 0x7f7f680c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6cd53640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 0x7f7f681457b0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000b9e0 tx=0x7f7f6000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60003b70 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f60003cd0 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f681eb230 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6000cb60 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.927+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f681ea750 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.928+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6806e160 con 0x7f7f68150720 2024-02-19T14:57:52.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.929+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7f6000ccc0 con 0x7f7f68150720 2024-02-19T14:57:52.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.929+0000 7f7f64ff9640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f5406b810 0x7f7f5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:52.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.929+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f7f6008a0c0 con 0x7f7f68150720 2024-02-19T14:57:52.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.929+0000 7f7f67fff640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f5406b810 0x7f7f5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:52.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.930+0000 7f7f67fff640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f5406b810 0x7f7f5406dcd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7f681e8110 tx=0x7f7f58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:52.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:52.934+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7f60054920 con 0x7f7f68150720 2024-02-19T14:57:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.107+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7f68074990 con 0x7f7f68150720 2024-02-19T14:57:53.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.801+0000 7f7f64ff9640 1 -- 172.21.15.78:0/227192715 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v154) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7f60057fe0 con 0x7f7f68150720 2024-02-19T14:57:53.804 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_data' created 2024-02-19T14:57:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f5406b810 msgr2=0x7f7f5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f5406b810 0x7f7f5406dcd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7f681e8110 tx=0x7f7f58008040 comp rx=0 tx=0).stop 2024-02-19T14:57:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 msgr2=0x7f7f681457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:53.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 0x7f7f681457b0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000b9e0 tx=0x7f7f6000bea0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 shutdown_connections 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f5406b810 0x7f7f5406dcd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f68150720 0x7f7f681457b0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f680c4c50 0x7f7f680c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 --2- 172.21.15.78:0/227192715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f68070270 0x7f7f6814d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.805+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 >> 172.21.15.78:0/227192715 conn(0x7f7f680c5050 msgr2=0x7f7f680c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:53.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.806+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 shutdown_connections 2024-02-19T14:57:53.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.807+0000 7f7f6e7b6640 1 -- 172.21.15.78:0/227192715 wait complete. 2024-02-19T14:57:53.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-02-19T14:57:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.935+0000 7fab32c91640 1 Processor -- start 2024-02-19T14:57:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.935+0000 7fab32c91640 1 -- start start 2024-02-19T14:57:53.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.936+0000 7fab32c91640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab2c14f1a0 0x7fab2c072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:53.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.936+0000 7fab32c91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06ea50 0x7fab2c152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:53.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.936+0000 7fab32c91640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c0c5970 con 0x7fab2c06fbc0 2024-02-19T14:57:53.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.936+0000 7fab32c91640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c0c5750 con 0x7fab2c14f1a0 2024-02-19T14:57:53.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.936+0000 7fab32c91640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0c5bc0 con 0x7fab2c06ea50 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab2c14f1a0 0x7fab2c072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06ea50 0x7fab2c152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab2c14f1a0 0x7fab2c072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:49882/0 (socket says 172.21.15.78:49882) 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 -- 172.21.15.78:0/3773709101 learned_addr learned my addr 172.21.15.78:0/3773709101 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 -- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06ea50 msgr2=0x7fab2c152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 --2- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06ea50 0x7fab2c152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 -- 172.21.15.78:0/3773709101 >> v1:172.21.15.150:6789/0 conn(0x7fab2c06fbc0 legacy=0x7fab2c0c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:53.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab30a2d640 1 -- 172.21.15.78:0/3773709101 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0c5ea0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.937+0000 7fab2bfff640 1 --2- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06ea50 0x7fab2c152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:53.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.938+0000 7fab30a2d640 1 --2- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c14f1a0 0x7fab2c072d70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009870 tx=0x7fab1c02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=450ab8ac719af4c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:53.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.938+0000 7fab2b7fe640 1 -- 172.21.15.78:0/3773709101 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1c0317c0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.938+0000 7fab2b7fe640 1 -- 172.21.15.78:0/3773709101 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1c031920 con 0x7fab2c14f1a0 2024-02-19T14:57:53.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.938+0000 7fab2b7fe640 1 -- 172.21.15.78:0/3773709101 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1c031cb0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.938+0000 7fab32c91640 1 -- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c14f1a0 msgr2=0x7fab2c072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.938+0000 7fab32c91640 1 --2- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c14f1a0 0x7fab2c072d70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009870 tx=0x7fab1c02b860 comp rx=0 tx=0).stop 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.939+0000 7fab32c91640 1 -- 172.21.15.78:0/3773709101 shutdown_connections 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.939+0000 7fab32c91640 1 --2- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06ea50 0x7fab2c152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.939+0000 7fab32c91640 1 --2- 172.21.15.78:0/3773709101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c14f1a0 0x7fab2c072d70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.939+0000 7fab32c91640 1 -- 172.21.15.78:0/3773709101 >> 172.21.15.78:0/3773709101 conn(0x7fab2c06c170 msgr2=0x7fab2c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.939+0000 7fab32c91640 1 -- 172.21.15.78:0/3773709101 shutdown_connections 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.939+0000 7fab32c91640 1 -- 172.21.15.78:0/3773709101 wait complete. 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 Processor -- start 2024-02-19T14:57:53.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 -- start start 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c06ea50 0x7fab2c06b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06fbc0 0x7fab2c1eaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 0x7fab2c1ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c1484b0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0c3290 con 0x7fab2c06ea50 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.940+0000 7fab32c91640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0c2d60 con 0x7fab2c06fbc0 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 0x7fab2c1ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 0x7fab2c1ee220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47114/0 (socket says 172.21.15.78:47114) 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 -- 172.21.15.78:0/3600237160 learned_addr learned my addr 172.21.15.78:0/3600237160 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab30a2d640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c06ea50 0x7fab2c06b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab2bfff640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06fbc0 0x7fab2c1eaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 -- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06fbc0 msgr2=0x7fab2c1eaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06fbc0 0x7fab2c1eaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 -- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c06ea50 msgr2=0x7fab2c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c06ea50 0x7fab2c06b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 -- 172.21.15.78:0/3600237160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab20009400 con 0x7fab2c14f1a0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab3122e640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 0x7fab2c1ee220 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c710 tx=0x7fab2400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24003b70 con 0x7fab2c14f1a0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab24003cf0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c0094d0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2400aa30 con 0x7fab2c14f1a0 2024-02-19T14:57:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.941+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c1edd90 con 0x7fab2c14f1a0 2024-02-19T14:57:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.942+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c1edf20 con 0x7fab2c14f1a0 2024-02-19T14:57:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.943+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fab2400ab90 con 0x7fab2c14f1a0 2024-02-19T14:57:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.943+0000 7fab297fa640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab1806b840 0x7fab1806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.943+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fab2408a1e0 con 0x7fab2c14f1a0 2024-02-19T14:57:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.944+0000 7fab30a2d640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab1806b840 0x7fab1806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:53.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.944+0000 7fab30a2d640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab1806b840 0x7fab1806dd00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fab1c036000 tx=0x7fab1c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:53.948+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fab240548c0 con 0x7fab2c14f1a0 2024-02-19T14:57:54.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.163+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7fab2c074990 con 0x7fab2c14f1a0 2024-02-19T14:57:54.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.815+0000 7fab297fa640 1 -- 172.21.15.78:0/3600237160 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 25 and data pool 26 v133) v1 ==== 177+0+0 (secure 0 0 0) 0x7fab24003e60 con 0x7fab2c14f1a0 2024-02-19T14:57:54.818 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 25 and data pool 26 2024-02-19T14:57:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.822+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab1806b840 msgr2=0x7fab1806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.822+0000 7fab32c91640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab1806b840 0x7fab1806dd00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fab1c036000 tx=0x7fab1c03d040 comp rx=0 tx=0).stop 2024-02-19T14:57:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.822+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 msgr2=0x7fab2c1ee220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:54.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.822+0000 7fab32c91640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 0x7fab2c1ee220 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c710 tx=0x7fab2400cbd0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.823+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 shutdown_connections 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.823+0000 7fab32c91640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab1806b840 0x7fab1806dd00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.823+0000 7fab32c91640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c14f1a0 0x7fab2c1ee220 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.823+0000 7fab32c91640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c06fbc0 0x7fab2c1eaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.823+0000 7fab32c91640 1 --2- 172.21.15.78:0/3600237160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c06ea50 0x7fab2c06b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.823+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 >> 172.21.15.78:0/3600237160 conn(0x7fab2c06c170 msgr2=0x7fab2c14e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:54.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.824+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 shutdown_connections 2024-02-19T14:57:54.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.824+0000 7fab32c91640 1 -- 172.21.15.78:0/3600237160 wait complete. 2024-02-19T14:57:54.844 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 Processor -- start 2024-02-19T14:57:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 -- start start 2024-02-19T14:57:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ab814f1a0 0x7f3ab8072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80c5970 con 0x7f3ab806fbc0 2024-02-19T14:57:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab80c5750 con 0x7f3ab814f1a0 2024-02-19T14:57:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.942+0000 7f3abe68f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80c5bc0 con 0x7f3ab806ea50 2024-02-19T14:57:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:54.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ab814f1a0 0x7f3ab8072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:54.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3abcc2c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3ab806fbc0 0x7f3ab80c78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:54260/0 (socket says 172.21.15.78:54260) 2024-02-19T14:57:54.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3abcc2c640 1 -- 172.21.15.78:0/3595943568 learned_addr learned my addr 172.21.15.78:0/3595943568 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab7fff640 1 -- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 msgr2=0x7f3ab8152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab7fff640 1 --2- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab7fff640 1 -- 172.21.15.78:0/3595943568 >> v1:172.21.15.150:6789/0 conn(0x7f3ab806fbc0 legacy=0x7f3ab80c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab7fff640 1 -- 172.21.15.78:0/3595943568 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab80c5ea0 con 0x7f3ab814f1a0 2024-02-19T14:57:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab6ffd640 1 -- 172.21.15.78:0/3595943568 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494017935 0 0) 0x7f3ab80c5970 con 0x7f3ab806fbc0 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.943+0000 7f3ab77fe640 1 --2- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.944+0000 7f3ab7fff640 1 --2- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8072d70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80097a0 tx=0x7f3aa802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bf89209c897c06c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.944+0000 7f3ab6ffd640 1 -- 172.21.15.78:0/3595943568 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80316f0 con 0x7f3ab814f1a0 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.944+0000 7f3ab6ffd640 1 -- 172.21.15.78:0/3595943568 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8031850 con 0x7f3ab814f1a0 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.944+0000 7f3ab6ffd640 1 -- 172.21.15.78:0/3595943568 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8031be0 con 0x7f3ab814f1a0 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.944+0000 7f3abe68f640 1 -- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab814f1a0 msgr2=0x7f3ab8072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.944+0000 7f3abe68f640 1 --2- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8072d70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80097a0 tx=0x7f3aa802b790 comp rx=0 tx=0).stop 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.945+0000 7f3abe68f640 1 -- 172.21.15.78:0/3595943568 shutdown_connections 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.945+0000 7f3abe68f640 1 --2- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.945+0000 7f3abe68f640 1 --2- 172.21.15.78:0/3595943568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8072d70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.945+0000 7f3abe68f640 1 -- 172.21.15.78:0/3595943568 >> 172.21.15.78:0/3595943568 conn(0x7f3ab806c170 msgr2=0x7f3ab806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:54.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.945+0000 7f3abe68f640 1 -- 172.21.15.78:0/3595943568 shutdown_connections 2024-02-19T14:57:54.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.945+0000 7f3abe68f640 1 -- 172.21.15.78:0/3595943568 wait complete. 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.946+0000 7f3abe68f640 1 Processor -- start 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.946+0000 7f3abe68f640 1 -- start start 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.946+0000 7f3abe68f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab806fbc0 0x7f3ab81eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8153640 con 0x7f3ab814f1a0 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8147c00 con 0x7f3ab806fbc0 2024-02-19T14:57:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81531b0 con 0x7f3ab806ea50 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8146eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47118/0 (socket says 172.21.15.78:47118) 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 -- 172.21.15.78:0/2109127089 learned_addr learned my addr 172.21.15.78:0/2109127089 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3ab7fff640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3ab77fe640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab806fbc0 0x7f3ab81eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 -- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 msgr2=0x7f3ab8155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 -- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab806fbc0 msgr2=0x7f3ab81eaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab806fbc0 0x7f3ab81eaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 -- 172.21.15.78:0/2109127089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac009400 con 0x7f3ab814f1a0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abcc2c640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8146eb0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c700 tx=0x7f3ab000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000ce10 con 0x7f3ab814f1a0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab000a640 con 0x7f3ab814f1a0 2024-02-19T14:57:54.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa8009400 con 0x7f3ab814f1a0 2024-02-19T14:57:54.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000aa70 con 0x7f3ab814f1a0 2024-02-19T14:57:54.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.947+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab81eba10 con 0x7f3ab814f1a0 2024-02-19T14:57:54.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.948+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab8153340 con 0x7f3ab814f1a0 2024-02-19T14:57:54.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.949+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3ab000abd0 con 0x7f3ab814f1a0 2024-02-19T14:57:54.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.949+0000 7f3ab4ff9640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3aa406b810 0x7f3aa406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:54.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.949+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3ab0089db0 con 0x7f3ab814f1a0 2024-02-19T14:57:54.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.949+0000 7f3ab7fff640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3aa406b810 0x7f3aa406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:54.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.950+0000 7f3ab7fff640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3aa406b810 0x7f3aa406dcd0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8036000 tx=0x7f3aa8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:54.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:54.954+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ab0054380 con 0x7f3ab814f1a0 2024-02-19T14:57:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.133+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ab81ebc00 con 0x7f3ab814f1a0 2024-02-19T14:57:55.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.134+0000 7f3ab4ff9640 1 -- 172.21.15.78:0/2109127089 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+20603 (secure 0 0 0) 0x7f3ab0018020 con 0x7f3ab814f1a0 2024-02-19T14:57:55.137 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:55.137 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":155,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:54.164387+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:55.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3aa406b810 msgr2=0x7f3aa406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3aa406b810 0x7f3aa406dcd0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8036000 tx=0x7f3aa8008030 comp rx=0 tx=0).stop 2024-02-19T14:57:55.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 msgr2=0x7f3ab8146eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8146eb0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c700 tx=0x7f3ab000cbc0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 shutdown_connections 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3aa406b810 0x7f3aa406dcd0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ab814f1a0 0x7f3ab8146eb0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ab806fbc0 0x7f3ab81eaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 --2- 172.21.15.78:0/2109127089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ab806ea50 0x7f3ab8155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 >> 172.21.15.78:0/2109127089 conn(0x7f3ab806c170 msgr2=0x7f3ab814ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 shutdown_connections 2024-02-19T14:57:55.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.136+0000 7f3abe68f640 1 -- 172.21.15.78:0/2109127089 wait complete. 2024-02-19T14:57:55.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.249+0000 7f5f3a809640 1 Processor -- start 2024-02-19T14:57:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.249+0000 7f5f3a809640 1 -- start start 2024-02-19T14:57:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f3a809640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f340719e0 0x7f5f340c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f3a809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f340c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f3a809640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3406ff90 con 0x7f5f3414d210 2024-02-19T14:57:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f3a809640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f34070270 con 0x7f5f340719e0 2024-02-19T14:57:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f3a809640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340c3970 con 0x7f5f340c3d90 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f38da6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f340c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f337fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f340719e0 0x7f5f340c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f38da6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f340c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51696/0 (socket says 172.21.15.78:51696) 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.250+0000 7f5f38da6640 1 -- 172.21.15.78:0/1094733009 learned_addr learned my addr 172.21.15.78:0/1094733009 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.251+0000 7f5f337fe640 1 -- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 msgr2=0x7f5f340c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.251+0000 7f5f337fe640 1 --2- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f340c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.251+0000 7f5f337fe640 1 -- 172.21.15.78:0/1094733009 >> v1:172.21.15.150:6789/0 conn(0x7f5f3414d210 legacy=0x7f5f340c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.251+0000 7f5f337fe640 1 -- 172.21.15.78:0/1094733009 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34156fc0 con 0x7f5f340719e0 2024-02-19T14:57:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.251+0000 7f5f38da6640 1 --2- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f340c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:57:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f337fe640 1 --2- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f340719e0 0x7f5f340c64d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5f240097a0 tx=0x7f5f2402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e30de8253fd3c4bb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f32ffd640 1 -- 172.21.15.78:0/1094733009 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2402e070 con 0x7f5f340719e0 2024-02-19T14:57:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f32ffd640 1 -- 172.21.15.78:0/1094733009 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f240315a0 con 0x7f5f340719e0 2024-02-19T14:57:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f32ffd640 1 -- 172.21.15.78:0/1094733009 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24031930 con 0x7f5f340719e0 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 -- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f340719e0 msgr2=0x7f5f340c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f340719e0 0x7f5f340c64d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5f240097a0 tx=0x7f5f2402b840 comp rx=0 tx=0).stop 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 -- 172.21.15.78:0/1094733009 shutdown_connections 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f340c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1094733009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f340719e0 0x7f5f340c64d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 -- 172.21.15.78:0/1094733009 >> 172.21.15.78:0/1094733009 conn(0x7f5f340c5050 msgr2=0x7f5f340c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 -- 172.21.15.78:0/1094733009 shutdown_connections 2024-02-19T14:57:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.252+0000 7f5f3a809640 1 -- 172.21.15.78:0/1094733009 wait complete. 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.253+0000 7f5f3a809640 1 Processor -- start 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 -- start start 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 0x7f5f341ec150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f341e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3414d210 0x7f5f341e9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3414e5c0 con 0x7f5f340719e0 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f341ec730 con 0x7f5f3414d210 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f3a809640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f341eca10 con 0x7f5f340c3d90 2024-02-19T14:57:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 0x7f5f341ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 0x7f5f341ec150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47126/0 (socket says 172.21.15.78:47126) 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f33fff640 1 -- 172.21.15.78:0/1393509401 learned_addr learned my addr 172.21.15.78:0/1393509401 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f337fe640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f341e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.254+0000 7f5f38da6640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3414d210 0x7f5f341e9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f33fff640 1 -- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 msgr2=0x7f5f341e8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f33fff640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f341e8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f33fff640 1 -- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3414d210 msgr2=0x7f5f341e9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f33fff640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3414d210 0x7f5f341e9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f33fff640 1 -- 172.21.15.78:0/1393509401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c009b30 con 0x7f5f340719e0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f33fff640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 0x7f5f341ec150 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c3c0 tx=0x7f5f2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f280052d0 con 0x7f5f340719e0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f28005450 con 0x7f5f340719e0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f24009400 con 0x7f5f340719e0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28007260 con 0x7f5f340719e0 2024-02-19T14:57:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.255+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f341ecfd0 con 0x7f5f340719e0 2024-02-19T14:57:55.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.256+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f34074990 con 0x7f5f340719e0 2024-02-19T14:57:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.257+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5f280073c0 con 0x7f5f340719e0 2024-02-19T14:57:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.257+0000 7f5f30ff9640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.257+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f5f28089920 con 0x7f5f340719e0 2024-02-19T14:57:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.257+0000 7f5f337fe640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.258+0000 7f5f337fe640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5f24036000 tx=0x7f5f2403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:55.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.264+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5f28053fa0 con 0x7f5f340719e0 2024-02-19T14:57:55.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.442+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f34143120 con 0x7f5f340719e0 2024-02-19T14:57:55.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.443+0000 7f5f30ff9640 1 -- 172.21.15.78:0/1393509401 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+20603 (secure 0 0 0) 0x7f5f28057660 con 0x7f5f340719e0 2024-02-19T14:57:55.446 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:55.446 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":155,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:54.164387+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:55.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 msgr2=0x7f5f2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5f24036000 tx=0x7f5f2403d040 comp rx=0 tx=0).stop 2024-02-19T14:57:55.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 msgr2=0x7f5f341ec150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 0x7f5f341ec150 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800c3c0 tx=0x7f5f2800c880 comp rx=0 tx=0).stop 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 shutdown_connections 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f2006b810 0x7f5f2006dcd0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3414d210 0x7f5f341e9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f340c3d90 0x7f5f341e8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 --2- 172.21.15.78:0/1393509401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f340719e0 0x7f5f341ec150 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 >> 172.21.15.78:0/1393509401 conn(0x7f5f340c5050 msgr2=0x7f5f340aaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 shutdown_connections 2024-02-19T14:57:55.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.445+0000 7f5f3a809640 1 -- 172.21.15.78:0/1393509401 wait complete. 2024-02-19T14:57:55.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-02-19T14:57:55.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.559+0000 7f5cbe05c640 1 Processor -- start 2024-02-19T14:57:55.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.559+0000 7f5cbe05c640 1 -- start start 2024-02-19T14:57:55.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cbe05c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cb80c6870 0x7f5cb80c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cbe05c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5cb8155070 0x7f5cb81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cbe05c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80c6460 con 0x7f5cb81525c0 2024-02-19T14:57:55.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cbe05c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb80c1f50 con 0x7f5cb8155070 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cbe05c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb81593a0 con 0x7f5cb80c6870 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cb7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5cb8155070 0x7f5cb81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cb7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5cb8155070 0x7f5cb81554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47134/0 (socket says 172.21.15.78:47134) 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cb7fff640 1 -- 172.21.15.78:0/463759055 learned_addr learned my addr 172.21.15.78:0/463759055 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.560+0000 7f5cb6ffd640 1 --2- 172.21.15.78:0/463759055 >> v2:172.21.15.150:3300/0 conn(0x7f5cb80c6870 0x7f5cb80c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.561+0000 7f5cb7fff640 1 -- 172.21.15.78:0/463759055 >> v2:172.21.15.150:3300/0 conn(0x7f5cb80c6870 msgr2=0x7f5cb80c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.561+0000 7f5cb7fff640 1 --2- 172.21.15.78:0/463759055 >> v2:172.21.15.150:3300/0 conn(0x7f5cb80c6870 0x7f5cb80c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.561+0000 7f5cb7fff640 1 -- 172.21.15.78:0/463759055 >> v1:172.21.15.150:6789/0 conn(0x7f5cb81525c0 legacy=0x7f5cb806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.561+0000 7f5cb7fff640 1 -- 172.21.15.78:0/463759055 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8158720 con 0x7f5cb8155070 2024-02-19T14:57:55.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.561+0000 7f5cb7fff640 1 --2- 172.21.15.78:0/463759055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb8155070 0x7f5cb81554e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80b0bb0 tx=0x7f5cb002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2a34f6674ec881 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:55.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.561+0000 7f5cb67fc640 1 -- 172.21.15.78:0/463759055 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb002ca80 con 0x7f5cb8155070 2024-02-19T14:57:55.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cb67fc640 1 -- 172.21.15.78:0/463759055 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb002cbe0 con 0x7f5cb8155070 2024-02-19T14:57:55.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cb67fc640 1 -- 172.21.15.78:0/463759055 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb00343f0 con 0x7f5cb8155070 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 -- 172.21.15.78:0/463759055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb8155070 msgr2=0x7f5cb81554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/463759055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb8155070 0x7f5cb81554e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80b0bb0 tx=0x7f5cb002bee0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 -- 172.21.15.78:0/463759055 shutdown_connections 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/463759055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb8155070 0x7f5cb81554e0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/463759055 >> v2:172.21.15.150:3300/0 conn(0x7f5cb80c6870 0x7f5cb80c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 -- 172.21.15.78:0/463759055 >> 172.21.15.78:0/463759055 conn(0x7f5cb80cab20 msgr2=0x7f5cb80caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 -- 172.21.15.78:0/463759055 shutdown_connections 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.562+0000 7f5cbe05c640 1 -- 172.21.15.78:0/463759055 wait complete. 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.564+0000 7f5cbe05c640 1 Processor -- start 2024-02-19T14:57:55.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.564+0000 7f5cbe05c640 1 -- start start 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cbe05c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cb80c6870 0x7f5cb8156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cbe05c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 0x7f5cb8143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cbe05c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5cb8155070 0x7f5cb80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cbe05c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb8157e70 con 0x7f5cb81525c0 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cbe05c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80ca2b0 con 0x7f5cb80c6870 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cbe05c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb81eb1a0 con 0x7f5cb8155070 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cb6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 0x7f5cb8143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cb6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 0x7f5cb8143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47146/0 (socket says 172.21.15.78:47146) 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cb6ffd640 1 -- 172.21.15.78:0/3522962400 learned_addr learned my addr 172.21.15.78:0/3522962400 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cb77fe640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cb80c6870 0x7f5cb8156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.565+0000 7f5cb7fff640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5cb8155070 0x7f5cb80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb6ffd640 1 -- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5cb8155070 msgr2=0x7f5cb80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb6ffd640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5cb8155070 0x7f5cb80c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb6ffd640 1 -- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cb80c6870 msgr2=0x7f5cb8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb6ffd640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cb80c6870 0x7f5cb8156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb6ffd640 1 -- 172.21.15.78:0/3522962400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0009b30 con 0x7f5cb81525c0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb77fe640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cb80c6870 0x7f5cb8156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cb6ffd640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 0x7f5cb8143480 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00d800 tx=0x7f5cac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac00b6c0 con 0x7f5cb81525c0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cac00b820 con 0x7f5cb81525c0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb81eb480 con 0x7f5cb81525c0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac00bbb0 con 0x7f5cb81525c0 2024-02-19T14:57:55.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.566+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb80c9e40 con 0x7f5cb81525c0 2024-02-19T14:57:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.567+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb81eafd0 con 0x7f5cb81525c0 2024-02-19T14:57:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.568+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5cac00bd10 con 0x7f5cb81525c0 2024-02-19T14:57:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.568+0000 7f5c9ffff640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ca406b810 0x7f5ca406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.568+0000 7f5cb77fe640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ca406b810 0x7f5ca406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:55.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.568+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f5cac08aba0 con 0x7f5cb81525c0 2024-02-19T14:57:55.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.569+0000 7f5cb77fe640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ca406b810 0x7f5ca406dcd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8001700 tx=0x7f5ca8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:55.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.573+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5cac055170 con 0x7f5cb81525c0 2024-02-19T14:57:55.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:55.913+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5cb8074990 con 0x7f5cb81525c0 2024-02-19T14:57:56.024 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:57:56.021+0000 7f1d0bb68640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:57:56.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:57:56.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:57:56.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.834+0000 7f5c9ffff640 1 -- 172.21.15.78:0/3522962400 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v135) v1 ==== 119+0+0 (secure 0 0 0) 0x7f5cac058830 con 0x7f5cb81525c0 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ca406b810 msgr2=0x7f5ca406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ca406b810 0x7f5ca406dcd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8001700 tx=0x7f5ca8009040 comp rx=0 tx=0).stop 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 msgr2=0x7f5cb8143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 0x7f5cb8143480 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00d800 tx=0x7f5cac00dcc0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 shutdown_connections 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ca406b810 0x7f5ca406dcd0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cb81525c0 0x7f5cb8143480 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5cb8155070 0x7f5cb80c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 --2- 172.21.15.78:0/3522962400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cb80c6870 0x7f5cb8156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 >> 172.21.15.78:0/3522962400 conn(0x7f5cb80cab20 msgr2=0x7f5cb80c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 shutdown_connections 2024-02-19T14:57:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.837+0000 7f5cbe05c640 1 -- 172.21.15.78:0/3522962400 wait complete. 2024-02-19T14:57:56.851 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:57:56.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-02-19T14:57:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.952+0000 7ff3f8839640 1 Processor -- start 2024-02-19T14:57:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.953+0000 7ff3f8839640 1 -- start start 2024-02-19T14:57:56.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f8839640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3f406d550 0x7ff3f40c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f8839640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40c6af0 con 0x7ff3f41588a0 2024-02-19T14:57:56.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f8839640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40c7940 con 0x7ff3f4155f90 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f8839640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f40c7270 con 0x7ff3f406d550 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff3f41588a0 0x7ff3f4158ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58366/0 (socket says 172.21.15.78:58366) 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/3417955629 learned_addr learned my addr 172.21.15.78:0/3417955629 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f1d9b640 1 --2- 172.21.15.78:0/3417955629 >> v2:172.21.15.150:3300/0 conn(0x7ff3f406d550 0x7ff3f40c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143162867 0 0) 0x7ff3f40c6af0 con 0x7ff3f41588a0 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.954+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f41588a0 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904334118 0 0) 0x7ff3f40c7940 con 0x7ff3f4155f90 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f40c6af0 con 0x7ff3f4155f90 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052641518 0 0) 0x7ff3dc003520 con 0x7ff3f41588a0 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 >> v2:172.21.15.150:3300/0 conn(0x7ff3f406d550 msgr2=0x7ff3f40c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 --2- 172.21.15.78:0/3417955629 >> v2:172.21.15.150:3300/0 conn(0x7ff3f406d550 0x7ff3f40c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 >> v1:172.21.15.150:6789/0 conn(0x7ff3f4155f90 legacy=0x7ff3f40c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40c7c20 con 0x7ff3f41588a0 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7ff3ec001970 con 0x7ff3f41588a0 2024-02-19T14:57:56.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 >> v1:172.21.15.78:6789/0 conn(0x7ff3f41588a0 legacy=0x7ff3f4158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3dc003b50 0x7ff3dc003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0047c0 0x7ff3dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3dc0051d0 0x7ff3dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f40c7270 con 0x7ff3dc0051d0 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc00c100 con 0x7ff3dc003b50 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc00c2c0 con 0x7ff3dc0047c0 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3dc0051d0 0x7ff3dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f259c640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3dc003b50 0x7ff3dc003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f1d9b640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0047c0 0x7ff3dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0047c0 msgr2=0x7ff3dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0047c0 0x7ff3dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3dc003b50 msgr2=0x7ff3dc003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3dc003b50 0x7ff3dc003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.955+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/3417955629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc003520 con 0x7ff3dc0051d0 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.956+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/3417955629 reap_dead start 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.956+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3dc0051d0 0x7ff3dc00b980 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00a680 tx=0x7ff3dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bad9ba556d612fa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.956+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0072d0 con 0x7ff3dc0051d0 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec02f070 con 0x7ff3dc0051d0 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f159a640 1 -- 172.21.15.78:0/3417955629 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0076b0 con 0x7ff3dc0051d0 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 -- 172.21.15.78:0/3417955629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3dc0051d0 msgr2=0x7ff3dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3dc0051d0 0x7ff3dc00b980 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00a680 tx=0x7ff3dc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 -- 172.21.15.78:0/3417955629 shutdown_connections 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 --2- 172.21.15.78:0/3417955629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3dc0051d0 0x7ff3dc00b980 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 -- 172.21.15.78:0/3417955629 >> 172.21.15.78:0/3417955629 conn(0x7ff3f406b700 msgr2=0x7ff3f40aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 -- 172.21.15.78:0/3417955629 shutdown_connections 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.957+0000 7ff3f8839640 1 -- 172.21.15.78:0/3417955629 wait complete. 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.958+0000 7ff3f8839640 1 Processor -- start 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.958+0000 7ff3f8839640 1 -- start start 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f8839640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0051d0 0x7ff3f4158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f8839640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4156bb0 0x7ff3f40c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f8839640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 0x7ff3f41eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f8839640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4070b90 con 0x7ff3f41eb460 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f8839640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f41eafd0 con 0x7ff3f4156bb0 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f8839640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4157370 con 0x7ff3dc0051d0 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 0x7ff3f41eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 0x7ff3f41eddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47152/0 (socket says 172.21.15.78:47152) 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/2693008657 learned_addr learned my addr 172.21.15.78:0/2693008657 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f1d9b640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4156bb0 0x7ff3f40c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.959+0000 7ff3f259c640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0051d0 0x7ff3f4158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0051d0 msgr2=0x7ff3f4158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0051d0 0x7ff3f4158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4156bb0 msgr2=0x7ff3f40c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4156bb0 0x7ff3f40c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f2d9d640 1 -- 172.21.15.78:0/2693008657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f406dae0 con 0x7ff3f41eb460 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f2d9d640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 0x7ff3f41eddd0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00a7a0 tx=0x7ff3ec001c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec001970 con 0x7ff3f41eb460 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec004b60 con 0x7ff3f41eb460 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec007400 con 0x7ff3f41eb460 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f4157090 con 0x7ff3f41eb460 2024-02-19T14:57:56.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.960+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f41eac60 con 0x7ff3f41eb460 2024-02-19T14:57:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.961+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f41ea7e0 con 0x7ff3f41eb460 2024-02-19T14:57:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.962+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff3ec031030 con 0x7ff3f41eb460 2024-02-19T14:57:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.962+0000 7ff3e2ffd640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff3d406b840 0x7ff3d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.962+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff3ec0ab8c0 con 0x7ff3f41eb460 2024-02-19T14:57:56.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.962+0000 7ff3f259c640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff3d406b840 0x7ff3d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:56.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.962+0000 7ff3f259c640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff3d406b840 0x7ff3d406dd00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4001920 tx=0x7ff3e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:56.967+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff3f41ea7e0 con 0x7ff3f41eb460 2024-02-19T14:57:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.185+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff3f41ea7e0 con 0x7ff3f41eb460 2024-02-19T14:57:57.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.835+0000 7ff3e2ffd640 1 -- 172.21.15.78:0/2693008657 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v136) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff3f41ea7e0 con 0x7ff3f41eb460 2024-02-19T14:57:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff3d406b840 msgr2=0x7ff3d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff3d406b840 0x7ff3d406dd00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4001920 tx=0x7ff3e4009040 comp rx=0 tx=0).stop 2024-02-19T14:57:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 msgr2=0x7ff3f41eddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 0x7ff3f41eddd0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00a7a0 tx=0x7ff3ec001c30 comp rx=0 tx=0).stop 2024-02-19T14:57:57.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 shutdown_connections 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff3d406b840 0x7ff3d406dd00 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3f41eb460 0x7ff3f41eddd0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4156bb0 0x7ff3f40c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 --2- 172.21.15.78:0/2693008657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3dc0051d0 0x7ff3f4158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 >> 172.21.15.78:0/2693008657 conn(0x7ff3f406b700 msgr2=0x7ff3f406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.837+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 shutdown_connections 2024-02-19T14:57:57.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.838+0000 7ff3f8839640 1 -- 172.21.15.78:0/2693008657 wait complete. 2024-02-19T14:57:57.852 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-02-19T14:57:57.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:57.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.989+0000 7fc3fea84640 1 Processor -- start 2024-02-19T14:57:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.989+0000 7fc3fea84640 1 -- start start 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.989+0000 7fc3fea84640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3f80c30a0 0x7fc3f8158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3fea84640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80c3cf0 con 0x7fc3f80c5130 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3fea84640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80c59a0 con 0x7fc3f80c5510 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3fea84640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f8158b60 con 0x7fc3f80c30a0 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3f80c30a0 0x7fc3f8158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3f80c30a0 0x7fc3f8158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41446/0 (socket says 172.21.15.78:41446) 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f7fff640 1 -- 172.21.15.78:0/1167949536 learned_addr learned my addr 172.21.15.78:0/1167949536 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3fc820640 1 --1- 172.21.15.78:0/1167949536 >> v1:172.21.15.78:6789/0 conn(0x7fc3f80c5130 0x7fc3f80c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37582/0 (socket says 172.21.15.78:37582) 2024-02-19T14:57:57.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f7fff640 1 -- 172.21.15.78:0/1167949536 >> v1:172.21.15.150:6789/0 conn(0x7fc3f80c5510 legacy=0x7fc3f815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:57.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f7fff640 1 -- 172.21.15.78:0/1167949536 >> v1:172.21.15.78:6789/0 conn(0x7fc3f80c5130 legacy=0x7fc3f80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:57.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f7fff640 1 -- 172.21.15.78:0/1167949536 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f8158530 con 0x7fc3f80c30a0 2024-02-19T14:57:57.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.990+0000 7fc3f77fe640 1 -- 172.21.15.78:0/1167949536 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210770841 0 0) 0x7fc3f80c3cf0 con 0x7fc3f80c5130 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.991+0000 7fc3f7fff640 1 --2- 172.21.15.78:0/1167949536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c30a0 0x7fc3f8158f80 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009870 tx=0x7fc3e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b5afc79d7882d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.991+0000 7fc3f77fe640 1 -- 172.21.15.78:0/1167949536 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4030860 con 0x7fc3f80c30a0 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.991+0000 7fc3f77fe640 1 -- 172.21.15.78:0/1167949536 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e40309c0 con 0x7fc3f80c30a0 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.991+0000 7fc3f77fe640 1 -- 172.21.15.78:0/1167949536 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4030d50 con 0x7fc3f80c30a0 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 -- 172.21.15.78:0/1167949536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c30a0 msgr2=0x7fc3f8158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 --2- 172.21.15.78:0/1167949536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c30a0 0x7fc3f8158f80 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009870 tx=0x7fc3e402b9a0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 -- 172.21.15.78:0/1167949536 shutdown_connections 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 --2- 172.21.15.78:0/1167949536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c30a0 0x7fc3f8158f80 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 -- 172.21.15.78:0/1167949536 >> 172.21.15.78:0/1167949536 conn(0x7fc3f806b4b0 msgr2=0x7fc3f80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:57.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 -- 172.21.15.78:0/1167949536 shutdown_connections 2024-02-19T14:57:57.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.992+0000 7fc3fea84640 1 -- 172.21.15.78:0/1167949536 wait complete. 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.993+0000 7fc3fea84640 1 Processor -- start 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.993+0000 7fc3fea84640 1 -- start start 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3f80c30a0 0x7fc3f8144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f80c5130 0x7fc3f8145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 0x7fc3f8146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f814c840 con 0x7fc3f80c5510 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f814d340 con 0x7fc3f80c5130 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f806f5e0 con 0x7fc3f80c30a0 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 0x7fc3f8146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 0x7fc3f8146290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41450/0 (socket says 172.21.15.78:41450) 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 -- 172.21.15.78:0/867309612 learned_addr learned my addr 172.21.15.78:0/867309612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3f7fff640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f80c5130 0x7fc3f8145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fc820640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3f80c30a0 0x7fc3f8144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 -- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3f80c30a0 msgr2=0x7fc3f8144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3f80c30a0 0x7fc3f8144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 -- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f80c5130 msgr2=0x7fc3f8145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f80c5130 0x7fc3f8145d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 -- 172.21.15.78:0/867309612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40094d0 con 0x7fc3f80c5510 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fd021640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 0x7fc3f8146290 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000b970 tx=0x7fc3f000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000cb40 con 0x7fc3f80c5510 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f000cca0 con 0x7fc3f80c5510 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f8145280 con 0x7fc3f80c5510 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.994+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0014680 con 0x7fc3f80c5510 2024-02-19T14:57:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.995+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f81ee910 con 0x7fc3f80c5510 2024-02-19T14:57:58.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.996+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f8074990 con 0x7fc3f80c5510 2024-02-19T14:57:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.996+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc3f001d3b0 con 0x7fc3f80c5510 2024-02-19T14:57:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.996+0000 7fc3f57fa640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3dc06b810 0x7fc3dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.996+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc3f008a530 con 0x7fc3f80c5510 2024-02-19T14:57:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.996+0000 7fc3fc820640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3dc06b810 0x7fc3dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:57.997+0000 7fc3fc820640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3dc06b810 0x7fc3dc06dcd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec000f60 tx=0x7fc3ec007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.001+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc3f0054b00 con 0x7fc3f80c5510 2024-02-19T14:57:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.355+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3f80c9400 con 0x7fc3f80c5510 2024-02-19T14:57:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.356+0000 7fc3f57fa640 1 -- 172.21.15.78:0/867309612 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+6197 (secure 0 0 0) 0x7fc3f00581c0 con 0x7fc3f80c5510 2024-02-19T14:57:58.358 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:58.359 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":136,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7768,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":7771,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:57:47.113438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/903597448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":903597448},{"type":"v1","addr":"172.21.15.78:6837","nonce":903597448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:57:57.186859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":5,"addr":"172.21.15.78:6836/2997633530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2997633530},{"type":"v1","addr":"172.21.15.78:6836","nonce":2997633530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:57:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 136 2024-02-19T14:57:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.358+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3dc06b810 msgr2=0x7fc3dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.358+0000 7fc3fea84640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3dc06b810 0x7fc3dc06dcd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec000f60 tx=0x7fc3ec007b50 comp rx=0 tx=0).stop 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.358+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 msgr2=0x7fc3f8146290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.358+0000 7fc3fea84640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 0x7fc3f8146290 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000b970 tx=0x7fc3f000be30 comp rx=0 tx=0).stop 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 shutdown_connections 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc3dc06b810 0x7fc3dc06dcd0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f80c5510 0x7fc3f8146290 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3f80c5130 0x7fc3f8145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 --2- 172.21.15.78:0/867309612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3f80c30a0 0x7fc3f8144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 >> 172.21.15.78:0/867309612 conn(0x7fc3f806b4b0 msgr2=0x7fc3f8149ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 shutdown_connections 2024-02-19T14:57:58.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.359+0000 7fc3fea84640 1 -- 172.21.15.78:0/867309612 wait complete. 2024-02-19T14:57:58.372 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:57:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.472+0000 7f15f4ba2640 1 Processor -- start 2024-02-19T14:57:58.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.472+0000 7f15f4ba2640 1 -- start start 2024-02-19T14:57:58.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.473+0000 7f15f4ba2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 0x7f15f015d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.473+0000 7f15f4ba2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f006faf0 con 0x7f15f00c2270 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.473+0000 7f15f4ba2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00c26d0 con 0x7f15f00c1e90 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.473+0000 7f15f4ba2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f00c1b60 con 0x7f15f0155780 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.473+0000 7f15eeffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f15f00c2270 0x7f15f0155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37588/0 (socket says 172.21.15.78:37588) 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.473+0000 7f15eeffd640 1 -- 172.21.15.78:0/1665273468 learned_addr learned my addr 172.21.15.78:0/1665273468 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15effff640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 0x7f15f015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230134212 0 0) 0x7f15f006faf0 con 0x7f15f00c2270 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d8003520 con 0x7f15f00c2270 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578764820 0 0) 0x7f15d8003520 con 0x7f15f00c2270 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 msgr2=0x7f15f015d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 0x7f15f015d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 >> v1:172.21.15.150:6789/0 conn(0x7f15f00c1e90 legacy=0x7f15f006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f006eee0 con 0x7f15f00c2270 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f15e4001670 con 0x7f15f00c2270 2024-02-19T14:57:58.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 >> v1:172.21.15.78:6789/0 conn(0x7f15f00c2270 legacy=0x7f15f0155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d80039a0 0x7f15d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 0x7f15d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d8004d00 0x7f15d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f00c1b60 con 0x7f15d8004d00 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d800bae0 con 0x7f15d80039a0 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d800bca0 con 0x7f15f0155780 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15effff640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d8004d00 0x7f15d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15ef7fe640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d80039a0 0x7f15d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.474+0000 7f15eeffd640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 0x7f15d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.475+0000 7f15effff640 1 -- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 msgr2=0x7f15d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.475+0000 7f15effff640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f0155780 0x7f15d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.475+0000 7f15effff640 1 -- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d80039a0 msgr2=0x7f15d8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.475+0000 7f15effff640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d80039a0 0x7f15d8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.475+0000 7f15effff640 1 -- 172.21.15.78:0/1665273468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d8003520 con 0x7f15d8004d00 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15effff640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d8004d00 0x7f15d8005180 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009fc0 tx=0x7f15d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932b36672756d146 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e802cc20 con 0x7f15d8004d00 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e800f040 con 0x7f15d8004d00 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15ee7fc640 1 -- 172.21.15.78:0/1665273468 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8034440 con 0x7f15d8004d00 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15f4ba2640 1 -- 172.21.15.78:0/1665273468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d8004d00 msgr2=0x7f15d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/1665273468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d8004d00 0x7f15d8005180 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009fc0 tx=0x7f15d80057f0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15effff640 1 -- 172.21.15.78:0/1665273468 reap_dead start 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15f4ba2640 1 -- 172.21.15.78:0/1665273468 shutdown_connections 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15f4ba2640 1 -- 172.21.15.78:0/1665273468 >> 172.21.15.78:0/1665273468 conn(0x7f15f00c7490 msgr2=0x7f15f00c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15f4ba2640 1 -- 172.21.15.78:0/1665273468 shutdown_connections 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.476+0000 7f15f4ba2640 1 -- 172.21.15.78:0/1665273468 wait complete. 2024-02-19T14:57:58.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.478+0000 7f15f4ba2640 1 Processor -- start 2024-02-19T14:57:58.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.478+0000 7f15f4ba2640 1 -- start start 2024-02-19T14:57:58.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.479+0000 7f15f4ba2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f006b0f0 0x7f15f00ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.479+0000 7f15f4ba2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 0x7f15f0144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.479+0000 7f15f4ba2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f01451e0 0x7f15f01dde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.479+0000 7f15f4ba2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f0154620 con 0x7f15f006b4d0 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.479+0000 7f15f4ba2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f01dd3c0 con 0x7f15f006b0f0 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.479+0000 7f15f4ba2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f0154870 con 0x7f15f01451e0 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 0x7f15f0144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 0x7f15f0144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41468/0 (socket says 172.21.15.78:41468) 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 -- 172.21.15.78:0/2237126284 learned_addr learned my addr 172.21.15.78:0/2237126284 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 -- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f01451e0 msgr2=0x7f15f01dde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f01451e0 0x7f15f01dde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 -- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f006b0f0 msgr2=0x7f15f00ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15ef7fe640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f006b0f0 0x7f15f00ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f006b0f0 0x7f15f00ad140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 -- 172.21.15.78:0/2237126284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f01dd630 con 0x7f15f006b4d0 2024-02-19T14:57:58.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15eeffd640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 0x7f15f0144ca0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f15e40011e0 tx=0x7f15e4007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e400f940 con 0x7f15f006b4d0 2024-02-19T14:57:58.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e400faa0 con 0x7f15f006b4d0 2024-02-19T14:57:58.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e400fe50 con 0x7f15f006b4d0 2024-02-19T14:57:58.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f01dd910 con 0x7f15f006b4d0 2024-02-19T14:57:58.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.480+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f0154b50 con 0x7f15f006b4d0 2024-02-19T14:57:58.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.482+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f00a9520 con 0x7f15f006b4d0 2024-02-19T14:57:58.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.482+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f15e401d460 con 0x7f15f006b4d0 2024-02-19T14:57:58.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.483+0000 7f15d7fff640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15dc06b810 0x7f15dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.483+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f15e408a7b0 con 0x7f15f006b4d0 2024-02-19T14:57:58.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.483+0000 7f15ef7fe640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15dc06b810 0x7f15dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.483+0000 7f15ef7fe640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15dc06b810 0x7f15dc06dcd0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f15e0005590 tx=0x7f15e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.487+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f15e4054d80 con 0x7f15f006b4d0 2024-02-19T14:57:58.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.666+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15f0074990 con 0x7f15f006b4d0 2024-02-19T14:57:58.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.668+0000 7f15d7fff640 1 -- 172.21.15.78:0/2237126284 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) v1 ==== 74+0+20590 (secure 0 0 0) 0x7f15e4058440 con 0x7f15f006b4d0 2024-02-19T14:57:58.671 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:58.671 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":156,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:57:55.815614+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15dc06b810 msgr2=0x7f15dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15dc06b810 0x7f15dc06dcd0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f15e0005590 tx=0x7f15e0009040 comp rx=0 tx=0).stop 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 msgr2=0x7f15f0144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 0x7f15f0144ca0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f15e40011e0 tx=0x7f15e4007720 comp rx=0 tx=0).stop 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 shutdown_connections 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15dc06b810 0x7f15dc06dcd0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f006b4d0 0x7f15f0144ca0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15f01451e0 0x7f15f01dde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 --2- 172.21.15.78:0/2237126284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f006b0f0 0x7f15f00ad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 >> 172.21.15.78:0/2237126284 conn(0x7f15f00c7490 msgr2=0x7f15f006c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:58.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 shutdown_connections 2024-02-19T14:57:58.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.670+0000 7f15f4ba2640 1 -- 172.21.15.78:0/2237126284 wait complete. 2024-02-19T14:57:58.684 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:57:58.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.783+0000 7fa2f932c640 1 Processor -- start 2024-02-19T14:57:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.783+0000 7fa2f932c640 1 -- start start 2024-02-19T14:57:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f932c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2f4152700 0x7fa2f40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f932c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2f40c5730 0x7fa2f406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f932c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f41551f0 con 0x7fa2f40c8e00 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f932c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f4155490 con 0x7fa2f40c5730 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f932c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f4155770 con 0x7fa2f4152700 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f27fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa2f40c5730 0x7fa2f406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa2f40c8e00 0x7fa2f4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37590/0 (socket says 172.21.15.78:37590) 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f37fe640 1 -- 172.21.15.78:0/3959778538 learned_addr learned my addr 172.21.15.78:0/3959778538 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f2ffd640 1 --2- 172.21.15.78:0/3959778538 >> v2:172.21.15.150:3300/0 conn(0x7fa2f4152700 0x7fa2f40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f27fc640 1 -- 172.21.15.78:0/3959778538 >> v2:172.21.15.150:3300/0 conn(0x7fa2f4152700 msgr2=0x7fa2f40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f27fc640 1 --2- 172.21.15.78:0/3959778538 >> v2:172.21.15.150:3300/0 conn(0x7fa2f4152700 0x7fa2f40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f27fc640 1 -- 172.21.15.78:0/3959778538 >> v1:172.21.15.78:6789/0 conn(0x7fa2f40c8e00 legacy=0x7fa2f4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f1ffb640 1 -- 172.21.15.78:0/3959778538 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464307927 0 0) 0x7fa2f41551f0 con 0x7fa2f40c8e00 2024-02-19T14:57:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.784+0000 7fa2f27fc640 1 -- 172.21.15.78:0/3959778538 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4155a50 con 0x7fa2f40c5730 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.785+0000 7fa2f27fc640 1 --2- 172.21.15.78:0/3959778538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f40c5730 0x7fa2f406c2a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa2f408bad0 tx=0x7fa2e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e940caa9947312e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f1ffb640 1 -- 172.21.15.78:0/3959778538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e802e070 con 0x7fa2f40c5730 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f1ffb640 1 -- 172.21.15.78:0/3959778538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e80315a0 con 0x7fa2f40c5730 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f1ffb640 1 -- 172.21.15.78:0/3959778538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8031930 con 0x7fa2f40c5730 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 -- 172.21.15.78:0/3959778538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f40c5730 msgr2=0x7fa2f406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3959778538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f40c5730 0x7fa2f406c2a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa2f408bad0 tx=0x7fa2e802b840 comp rx=0 tx=0).stop 2024-02-19T14:57:58.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 -- 172.21.15.78:0/3959778538 shutdown_connections 2024-02-19T14:57:58.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3959778538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f40c5730 0x7fa2f406c2a0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3959778538 >> v2:172.21.15.150:3300/0 conn(0x7fa2f4152700 0x7fa2f40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 -- 172.21.15.78:0/3959778538 >> 172.21.15.78:0/3959778538 conn(0x7fa2f40cac90 msgr2=0x7fa2f40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:58.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 -- 172.21.15.78:0/3959778538 shutdown_connections 2024-02-19T14:57:58.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.786+0000 7fa2f932c640 1 -- 172.21.15.78:0/3959778538 wait complete. 2024-02-19T14:57:58.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.788+0000 7fa2f932c640 1 Processor -- start 2024-02-19T14:57:58.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.788+0000 7fa2f932c640 1 -- start start 2024-02-19T14:57:58.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f932c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f40c5730 0x7fa2f40ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f932c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2f40c8e00 0x7fa2f40ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f932c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 0x7fa2f4146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f932c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f40aafd0 con 0x7fa2f4152700 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f932c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4156c90 con 0x7fa2f40c5730 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f932c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f40acb60 con 0x7fa2f40c8e00 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 0x7fa2f4146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 0x7fa2f4146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41488/0 (socket says 172.21.15.78:41488) 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 -- 172.21.15.78:0/3704168097 learned_addr learned my addr 172.21.15.78:0/3704168097 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f2ffd640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f40c5730 0x7fa2f40ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f27fc640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2f40c8e00 0x7fa2f40ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 -- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2f40c8e00 msgr2=0x7fa2f40ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2f40c8e00 0x7fa2f40ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 -- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f40c5730 msgr2=0x7fa2f40ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f40c5730 0x7fa2f40ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.789+0000 7fa2f37fe640 1 -- 172.21.15.78:0/3704168097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8009400 con 0x7fa2f4152700 2024-02-19T14:57:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.790+0000 7fa2f37fe640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 0x7fa2f4146be0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c7a0 tx=0x7fa2ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.790+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00a640 con 0x7fa2f4152700 2024-02-19T14:57:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.790+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec00a7a0 con 0x7fa2f4152700 2024-02-19T14:57:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.790+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00abd0 con 0x7fa2f4152700 2024-02-19T14:57:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.790+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f41594f0 con 0x7fa2f4152700 2024-02-19T14:57:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.790+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f41eb760 con 0x7fa2f4152700 2024-02-19T14:57:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.791+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f4074990 con 0x7fa2f4152700 2024-02-19T14:57:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.791+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa2ec00ad30 con 0x7fa2f4152700 2024-02-19T14:57:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.792+0000 7fa2db7fe640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa2e006b840 0x7fa2e006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.792+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa2ec089bd0 con 0x7fa2f4152700 2024-02-19T14:57:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.792+0000 7fa2f2ffd640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa2e006b840 0x7fa2e006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.792+0000 7fa2f2ffd640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa2e006b840 0x7fa2e006dd00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40079a0 tx=0x7fa2e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.796+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa2ec054250 con 0x7fa2f4152700 2024-02-19T14:57:58.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.978+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2f406ded0 con 0x7fa2e006b840 2024-02-19T14:57:58.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.990+0000 7fa2db7fe640 1 -- 172.21.15.78:0/3704168097 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528975 (secure 0 0 0) 0x7fa2e40108d0 con 0x7fa2e006b840 2024-02-19T14:57:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:59.003 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:59.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa2e006b840 msgr2=0x7fa2e006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa2e006b840 0x7fa2e006dd00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40079a0 tx=0x7fa2e4008040 comp rx=0 tx=0).stop 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 msgr2=0x7fa2f4146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 0x7fa2f4146be0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c7a0 tx=0x7fa2ec00cc60 comp rx=0 tx=0).stop 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 shutdown_connections 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa2e006b840 0x7fa2e006dd00 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f4152700 0x7fa2f4146be0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2f40c8e00 0x7fa2f40ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 --2- 172.21.15.78:0/3704168097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f40c5730 0x7fa2f40ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.995+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 >> 172.21.15.78:0/3704168097 conn(0x7fa2f40cac90 msgr2=0x7fa2f414bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.996+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 shutdown_connections 2024-02-19T14:57:59.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:58.996+0000 7fa2f932c640 1 -- 172.21.15.78:0/3704168097 wait complete. 2024-02-19T14:57:59.013 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-02-19T14:57:58.129155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":359,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266060,"kb_used_data":16616,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708660,"statfs":{"total":773094113280,"available":772821667840,"internally_reserved":0,"allocated":17014784,"data_stored":8858192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331425"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995097+0000","last_change":"2024-02-19T14:57:54.822787+0000","last_active":"2024-02-19T14:57:56.995097+0000","last_peered":"2024-02-19T14:57:56.995097+0000","last_clean":"2024-02-19T14:57:56.995097+0000","last_became_active":"2024-02-19T14:57:54.822487+0000","last_became_peered":"2024-02-19T14:57:54.822487+0000","last_unstale":"2024-02-19T14:57:56.995097+0000","last_undegraded":"2024-02-19T14:57:56.995097+0000","last_fullsized":"2024-02-19T14:57:56.995097+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:05.975465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826551+0000","last_change":"2024-02-19T14:57:45.778368+0000","last_active":"2024-02-19T14:57:55.826551+0000","last_peered":"2024-02-19T14:57:55.826551+0000","last_clean":"2024-02-19T14:57:55.826551+0000","last_became_active":"2024-02-19T14:57:45.777197+0000","last_became_peered":"2024-02-19T14:57:45.777197+0000","last_unstale":"2024-02-19T14:57:55.826551+0000","last_undegraded":"2024-02-19T14:57:55.826551+0000","last_fullsized":"2024-02-19T14:57:55.826551+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:12.001914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994756+0000","last_change":"2024-02-19T14:57:54.821802+0000","last_active":"2024-02-19T14:57:56.994756+0000","last_peered":"2024-02-19T14:57:56.994756+0000","last_clean":"2024-02-19T14:57:56.994756+0000","last_became_active":"2024-02-19T14:57:54.821533+0000","last_became_peered":"2024-02-19T14:57:54.821533+0000","last_unstale":"2024-02-19T14:57:56.994756+0000","last_undegraded":"2024-02-19T14:57:56.994756+0000","last_fullsized":"2024-02-19T14:57:56.994756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:37.441462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826317+0000","last_change":"2024-02-19T14:57:45.774781+0000","last_active":"2024-02-19T14:57:55.826317+0000","last_peered":"2024-02-19T14:57:55.826317+0000","last_clean":"2024-02-19T14:57:55.826317+0000","last_became_active":"2024-02-19T14:57:45.774498+0000","last_became_peered":"2024-02-19T14:57:45.774498+0000","last_unstale":"2024-02-19T14:57:55.826317+0000","last_undegraded":"2024-02-19T14:57:55.826317+0000","last_fullsized":"2024-02-19T14:57:55.826317+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:33.646592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825686+0000","last_change":"2024-02-19T14:57:54.829774+0000","last_active":"2024-02-19T14:57:55.825686+0000","last_peered":"2024-02-19T14:57:55.825686+0000","last_clean":"2024-02-19T14:57:55.825686+0000","last_became_active":"2024-02-19T14:57:54.829521+0000","last_became_peered":"2024-02-19T14:57:54.829521+0000","last_unstale":"2024-02-19T14:57:55.825686+0000","last_undegraded":"2024-02-19T14:57:55.825686+0000","last_fullsized":"2024-02-19T14:57:55.825686+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:24:13.797923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825662+0000","last_change":"2024-02-19T14:57:45.782283+0000","last_active":"2024-02-19T14:57:55.825662+0000","last_peered":"2024-02-19T14:57:55.825662+0000","last_clean":"2024-02-19T14:57:55.825662+0000","last_became_active":"2024-02-19T14:57:45.779080+0000","last_became_peered":"2024-02-19T14:57:45.779080+0000","last_unstale":"2024-02-19T14:57:55.825662+0000","last_undegraded":"2024-02-19T14:57:55.825662+0000","last_fullsized":"2024-02-19T14:57:55.825662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:26.412051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825706+0000","last_change":"2024-02-19T14:57:54.818112+0000","last_active":"2024-02-19T14:57:55.825706+0000","last_peered":"2024-02-19T14:57:55.825706+0000","last_clean":"2024-02-19T14:57:55.825706+0000","last_became_active":"2024-02-19T14:57:54.817836+0000","last_became_peered":"2024-02-19T14:57:54.817836+0000","last_unstale":"2024-02-19T14:57:55.825706+0000","last_undegraded":"2024-02-19T14:57:55.825706+0000","last_fullsized":"2024-02-19T14:57:55.825706+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:30:56.427520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826049+0000","last_change":"2024-02-19T14:57:45.778032+0000","last_active":"2024-02-19T14:57:55.826049+0000","last_peered":"2024-02-19T14:57:55.826049+0000","last_clean":"2024-02-19T14:57:55.826049+0000","last_became_active":"2024-02-19T14:57:45.777184+0000","last_became_peered":"2024-02-19T14:57:45.777184+0000","last_unstale":"2024-02-19T14:57:55.826049+0000","last_undegraded":"2024-02-19T14:57:55.826049+0000","last_fullsized":"2024-02-19T14:57:55.826049+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:36.503471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823128+0000","last_change":"2024-02-19T14:57:54.822500+0000","last_active":"2024-02-19T14:57:55.823128+0000","last_peered":"2024-02-19T14:57:55.823128+0000","last_clean":"2024-02-19T14:57:55.823128+0000","last_became_active":"2024-02-19T14:57:54.822233+0000","last_became_peered":"2024-02-19T14:57:54.822233+0000","last_unstale":"2024-02-19T14:57:55.823128+0000","last_undegraded":"2024-02-19T14:57:55.823128+0000","last_fullsized":"2024-02-19T14:57:55.823128+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:45.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994579+0000","last_change":"2024-02-19T14:57:45.777557+0000","last_active":"2024-02-19T14:57:56.994579+0000","last_peered":"2024-02-19T14:57:56.994579+0000","last_clean":"2024-02-19T14:57:56.994579+0000","last_became_active":"2024-02-19T14:57:45.777296+0000","last_became_peered":"2024-02-19T14:57:45.777296+0000","last_unstale":"2024-02-19T14:57:56.994579+0000","last_undegraded":"2024-02-19T14:57:56.994579+0000","last_fullsized":"2024-02-19T14:57:56.994579+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826224+0000","last_change":"2024-02-19T14:57:54.829408+0000","last_active":"2024-02-19T14:57:55.826224+0000","last_peered":"2024-02-19T14:57:55.826224+0000","last_clean":"2024-02-19T14:57:55.826224+0000","last_became_active":"2024-02-19T14:57:54.829129+0000","last_became_peered":"2024-02-19T14:57:54.829129+0000","last_unstale":"2024-02-19T14:57:55.826224+0000","last_undegraded":"2024-02-19T14:57:55.826224+0000","last_fullsized":"2024-02-19T14:57:55.826224+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:24:13.797923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825055+0000","last_change":"2024-02-19T14:57:45.781001+0000","last_active":"2024-02-19T14:57:55.825055+0000","last_peered":"2024-02-19T14:57:55.825055+0000","last_clean":"2024-02-19T14:57:55.825055+0000","last_became_active":"2024-02-19T14:57:45.780476+0000","last_became_peered":"2024-02-19T14:57:45.780476+0000","last_unstale":"2024-02-19T14:57:55.825055+0000","last_undegraded":"2024-02-19T14:57:55.825055+0000","last_fullsized":"2024-02-19T14:57:55.825055+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819210+0000","last_change":"2024-02-19T14:57:54.828759+0000","last_active":"2024-02-19T14:57:55.819210+0000","last_peered":"2024-02-19T14:57:55.819210+0000","last_clean":"2024-02-19T14:57:55.819210+0000","last_became_active":"2024-02-19T14:57:54.828501+0000","last_became_peered":"2024-02-19T14:57:54.828501+0000","last_unstale":"2024-02-19T14:57:55.819210+0000","last_undegraded":"2024-02-19T14:57:55.819210+0000","last_fullsized":"2024-02-19T14:57:55.819210+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:04.080418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825438+0000","last_change":"2024-02-19T14:57:45.783602+0000","last_active":"2024-02-19T14:57:55.825438+0000","last_peered":"2024-02-19T14:57:55.825438+0000","last_clean":"2024-02-19T14:57:55.825438+0000","last_became_active":"2024-02-19T14:57:45.783269+0000","last_became_peered":"2024-02-19T14:57:45.783269+0000","last_unstale":"2024-02-19T14:57:55.825438+0000","last_undegraded":"2024-02-19T14:57:55.825438+0000","last_fullsized":"2024-02-19T14:57:55.825438+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819026+0000","last_change":"2024-02-19T14:57:54.825478+0000","last_active":"2024-02-19T14:57:55.819026+0000","last_peered":"2024-02-19T14:57:55.819026+0000","last_clean":"2024-02-19T14:57:55.819026+0000","last_became_active":"2024-02-19T14:57:54.825228+0000","last_became_peered":"2024-02-19T14:57:54.825228+0000","last_unstale":"2024-02-19T14:57:55.819026+0000","last_undegraded":"2024-02-19T14:57:55.819026+0000","last_fullsized":"2024-02-19T14:57:55.819026+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:47.934567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825175+0000","last_change":"2024-02-19T14:57:45.763954+0000","last_active":"2024-02-19T14:57:55.825175+0000","last_peered":"2024-02-19T14:57:55.825175+0000","last_clean":"2024-02-19T14:57:55.825175+0000","last_became_active":"2024-02-19T14:57:45.763656+0000","last_became_peered":"2024-02-19T14:57:45.763656+0000","last_unstale":"2024-02-19T14:57:55.825175+0000","last_undegraded":"2024-02-19T14:57:55.825175+0000","last_fullsized":"2024-02-19T14:57:55.825175+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995200+0000","last_change":"2024-02-19T14:57:54.819808+0000","last_active":"2024-02-19T14:57:56.995200+0000","last_peered":"2024-02-19T14:57:56.995200+0000","last_clean":"2024-02-19T14:57:56.995200+0000","last_became_active":"2024-02-19T14:57:54.819516+0000","last_became_peered":"2024-02-19T14:57:54.819516+0000","last_unstale":"2024-02-19T14:57:56.995200+0000","last_undegraded":"2024-02-19T14:57:56.995200+0000","last_fullsized":"2024-02-19T14:57:56.995200+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:10.691659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826108+0000","last_change":"2024-02-19T14:57:45.772149+0000","last_active":"2024-02-19T14:57:55.826108+0000","last_peered":"2024-02-19T14:57:55.826108+0000","last_clean":"2024-02-19T14:57:55.826108+0000","last_became_active":"2024-02-19T14:57:45.771811+0000","last_became_peered":"2024-02-19T14:57:45.771811+0000","last_unstale":"2024-02-19T14:57:55.826108+0000","last_undegraded":"2024-02-19T14:57:55.826108+0000","last_fullsized":"2024-02-19T14:57:55.826108+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995027+0000","last_change":"2024-02-19T14:57:54.823091+0000","last_active":"2024-02-19T14:57:56.995027+0000","last_peered":"2024-02-19T14:57:56.995027+0000","last_clean":"2024-02-19T14:57:56.995027+0000","last_became_active":"2024-02-19T14:57:54.822833+0000","last_became_peered":"2024-02-19T14:57:54.822833+0000","last_unstale":"2024-02-19T14:57:56.995027+0000","last_undegraded":"2024-02-19T14:57:56.995027+0000","last_fullsized":"2024-02-19T14:57:56.995027+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:24:13.797923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825893+0000","last_change":"2024-02-19T14:57:45.769091+0000","last_active":"2024-02-19T14:57:55.825893+0000","last_peered":"2024-02-19T14:57:55.825893+0000","last_clean":"2024-02-19T14:57:55.825893+0000","last_became_active":"2024-02-19T14:57:45.768819+0000","last_became_peered":"2024-02-19T14:57:45.768819+0000","last_unstale":"2024-02-19T14:57:55.825893+0000","last_undegraded":"2024-02-19T14:57:55.825893+0000","last_fullsized":"2024-02-19T14:57:55.825893+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825839+0000","last_change":"2024-02-19T14:57:54.819549+0000","last_active":"2024-02-19T14:57:55.825839+0000","last_peered":"2024-02-19T14:57:55.825839+0000","last_clean":"2024-02-19T14:57:55.825839+0000","last_became_active":"2024-02-19T14:57:54.819297+0000","last_became_peered":"2024-02-19T14:57:54.819297+0000","last_unstale":"2024-02-19T14:57:55.825839+0000","last_undegraded":"2024-02-19T14:57:55.825839+0000","last_fullsized":"2024-02-19T14:57:55.825839+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:04.023224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995989+0000","last_change":"2024-02-19T14:57:45.771342+0000","last_active":"2024-02-19T14:57:56.995989+0000","last_peered":"2024-02-19T14:57:56.995989+0000","last_clean":"2024-02-19T14:57:56.995989+0000","last_became_active":"2024-02-19T14:57:45.771047+0000","last_became_peered":"2024-02-19T14:57:45.771047+0000","last_unstale":"2024-02-19T14:57:56.995989+0000","last_undegraded":"2024-02-19T14:57:56.995989+0000","last_fullsized":"2024-02-19T14:57:56.995989+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819576+0000","last_change":"2024-02-19T14:57:54.827482+0000","last_active":"2024-02-19T14:57:55.819576+0000","last_peered":"2024-02-19T14:57:55.819576+0000","last_clean":"2024-02-19T14:57:55.819576+0000","last_became_active":"2024-02-19T14:57:54.826130+0000","last_became_peered":"2024-02-19T14:57:54.826130+0000","last_unstale":"2024-02-19T14:57:55.819576+0000","last_undegraded":"2024-02-19T14:57:55.819576+0000","last_fullsized":"2024-02-19T14:57:55.819576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:16.125045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825873+0000","last_change":"2024-02-19T14:57:45.783578+0000","last_active":"2024-02-19T14:57:55.825873+0000","last_peered":"2024-02-19T14:57:55.825873+0000","last_clean":"2024-02-19T14:57:55.825873+0000","last_became_active":"2024-02-19T14:57:45.783301+0000","last_became_peered":"2024-02-19T14:57:45.783301+0000","last_unstale":"2024-02-19T14:57:55.825873+0000","last_undegraded":"2024-02-19T14:57:55.825873+0000","last_fullsized":"2024-02-19T14:57:55.825873+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:14.016549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825788+0000","last_change":"2024-02-19T14:57:54.822005+0000","last_active":"2024-02-19T14:57:55.825788+0000","last_peered":"2024-02-19T14:57:55.825788+0000","last_clean":"2024-02-19T14:57:55.825788+0000","last_became_active":"2024-02-19T14:57:54.821743+0000","last_became_peered":"2024-02-19T14:57:54.821743+0000","last_unstale":"2024-02-19T14:57:55.825788+0000","last_undegraded":"2024-02-19T14:57:55.825788+0000","last_fullsized":"2024-02-19T14:57:55.825788+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:05.123628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819637+0000","last_change":"2024-02-19T14:57:45.782352+0000","last_active":"2024-02-19T14:57:55.819637+0000","last_peered":"2024-02-19T14:57:55.819637+0000","last_clean":"2024-02-19T14:57:55.819637+0000","last_became_active":"2024-02-19T14:57:45.781489+0000","last_became_peered":"2024-02-19T14:57:45.781489+0000","last_unstale":"2024-02-19T14:57:55.819637+0000","last_undegraded":"2024-02-19T14:57:55.819637+0000","last_fullsized":"2024-02-19T14:57:55.819637+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:33.948498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818824+0000","last_change":"2024-02-19T14:57:54.828531+0000","last_active":"2024-02-19T14:57:55.818824+0000","last_peered":"2024-02-19T14:57:55.818824+0000","last_clean":"2024-02-19T14:57:55.818824+0000","last_became_active":"2024-02-19T14:57:54.828273+0000","last_became_peered":"2024-02-19T14:57:54.828273+0000","last_unstale":"2024-02-19T14:57:55.818824+0000","last_undegraded":"2024-02-19T14:57:55.818824+0000","last_fullsized":"2024-02-19T14:57:55.818824+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:23.513464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994648+0000","last_change":"2024-02-19T14:57:45.776389+0000","last_active":"2024-02-19T14:57:56.994648+0000","last_peered":"2024-02-19T14:57:56.994648+0000","last_clean":"2024-02-19T14:57:56.994648+0000","last_became_active":"2024-02-19T14:57:45.776133+0000","last_became_peered":"2024-02-19T14:57:45.776133+0000","last_unstale":"2024-02-19T14:57:56.994648+0000","last_undegraded":"2024-02-19T14:57:56.994648+0000","last_fullsized":"2024-02-19T14:57:56.994648+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826215+0000","last_change":"2024-02-19T14:57:54.824613+0000","last_active":"2024-02-19T14:57:55.826215+0000","last_peered":"2024-02-19T14:57:55.826215+0000","last_clean":"2024-02-19T14:57:55.826215+0000","last_became_active":"2024-02-19T14:57:54.824319+0000","last_became_peered":"2024-02-19T14:57:54.824319+0000","last_unstale":"2024-02-19T14:57:55.826215+0000","last_undegraded":"2024-02-19T14:57:55.826215+0000","last_fullsized":"2024-02-19T14:57:55.826215+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:54.699190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994675+0000","last_change":"2024-02-19T14:57:45.774486+0000","last_active":"2024-02-19T14:57:56.994675+0000","last_peered":"2024-02-19T14:57:56.994675+0000","last_clean":"2024-02-19T14:57:56.994675+0000","last_became_active":"2024-02-19T14:57:45.773909+0000","last_became_peered":"2024-02-19T14:57:45.773909+0000","last_unstale":"2024-02-19T14:57:56.994675+0000","last_undegraded":"2024-02-19T14:57:56.994675+0000","last_fullsized":"2024-02-19T14:57:56.994675+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:24.066471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822963+0000","last_change":"2024-02-19T14:57:54.824675+0000","last_active":"2024-02-19T14:57:55.822963+0000","last_peered":"2024-02-19T14:57:55.822963+0000","last_clean":"2024-02-19T14:57:55.822963+0000","last_became_active":"2024-02-19T14:57:54.824428+0000","last_became_peered":"2024-02-19T14:57:54.824428+0000","last_unstale":"2024-02-19T14:57:55.822963+0000","last_undegraded":"2024-02-19T14:57:55.822963+0000","last_fullsized":"2024-02-19T14:57:55.822963+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:35.476036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824841+0000","last_change":"2024-02-19T14:57:45.776654+0000","last_active":"2024-02-19T14:57:55.824841+0000","last_peered":"2024-02-19T14:57:55.824841+0000","last_clean":"2024-02-19T14:57:55.824841+0000","last_became_active":"2024-02-19T14:57:45.776192+0000","last_became_peered":"2024-02-19T14:57:45.776192+0000","last_unstale":"2024-02-19T14:57:55.824841+0000","last_undegraded":"2024-02-19T14:57:55.824841+0000","last_fullsized":"2024-02-19T14:57:55.824841+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:14:34.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826326+0000","last_change":"2024-02-19T14:57:54.821884+0000","last_active":"2024-02-19T14:57:55.826326+0000","last_peered":"2024-02-19T14:57:55.826326+0000","last_clean":"2024-02-19T14:57:55.826326+0000","last_became_active":"2024-02-19T14:57:54.821331+0000","last_became_peered":"2024-02-19T14:57:54.821331+0000","last_unstale":"2024-02-19T14:57:55.826326+0000","last_undegraded":"2024-02-19T14:57:55.826326+0000","last_fullsized":"2024-02-19T14:57:55.826326+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:45.122910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826209+0000","last_change":"2024-02-19T14:57:45.784872+0000","last_active":"2024-02-19T14:57:55.826209+0000","last_peered":"2024-02-19T14:57:55.826209+0000","last_clean":"2024-02-19T14:57:55.826209+0000","last_became_active":"2024-02-19T14:57:45.784602+0000","last_became_peered":"2024-02-19T14:57:45.784602+0000","last_unstale":"2024-02-19T14:57:55.826209+0000","last_undegraded":"2024-02-19T14:57:55.826209+0000","last_fullsized":"2024-02-19T14:57:55.826209+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819401+0000","last_change":"2024-02-19T14:57:54.821393+0000","last_active":"2024-02-19T14:57:55.819401+0000","last_peered":"2024-02-19T14:57:55.819401+0000","last_clean":"2024-02-19T14:57:55.819401+0000","last_became_active":"2024-02-19T14:57:54.820641+0000","last_became_peered":"2024-02-19T14:57:54.820641+0000","last_unstale":"2024-02-19T14:57:55.819401+0000","last_undegraded":"2024-02-19T14:57:55.819401+0000","last_fullsized":"2024-02-19T14:57:55.819401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:31.578824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995251+0000","last_change":"2024-02-19T14:57:45.771733+0000","last_active":"2024-02-19T14:57:56.995251+0000","last_peered":"2024-02-19T14:57:56.995251+0000","last_clean":"2024-02-19T14:57:56.995251+0000","last_became_active":"2024-02-19T14:57:45.771454+0000","last_became_peered":"2024-02-19T14:57:45.771454+0000","last_unstale":"2024-02-19T14:57:56.995251+0000","last_undegraded":"2024-02-19T14:57:56.995251+0000","last_fullsized":"2024-02-19T14:57:56.995251+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:52.644495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819447+0000","last_change":"2024-02-19T14:57:54.827044+0000","last_active":"2024-02-19T14:57:55.819447+0000","last_peered":"2024-02-19T14:57:55.819447+0000","last_clean":"2024-02-19T14:57:55.819447+0000","last_became_active":"2024-02-19T14:57:54.826224+0000","last_became_peered":"2024-02-19T14:57:54.826224+0000","last_unstale":"2024-02-19T14:57:55.819447+0000","last_undegraded":"2024-02-19T14:57:55.819447+0000","last_fullsized":"2024-02-19T14:57:55.819447+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:44.434234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.996004+0000","last_change":"2024-02-19T14:57:45.772926+0000","last_active":"2024-02-19T14:57:56.996004+0000","last_peered":"2024-02-19T14:57:56.996004+0000","last_clean":"2024-02-19T14:57:56.996004+0000","last_became_active":"2024-02-19T14:57:45.772628+0000","last_became_peered":"2024-02-19T14:57:45.772628+0000","last_unstale":"2024-02-19T14:57:56.996004+0000","last_undegraded":"2024-02-19T14:57:56.996004+0000","last_fullsized":"2024-02-19T14:57:56.996004+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825587+0000","last_change":"2024-02-19T14:57:54.828022+0000","last_active":"2024-02-19T14:57:55.825587+0000","last_peered":"2024-02-19T14:57:55.825587+0000","last_clean":"2024-02-19T14:57:55.825587+0000","last_became_active":"2024-02-19T14:57:54.827764+0000","last_became_peered":"2024-02-19T14:57:54.827764+0000","last_unstale":"2024-02-19T14:57:55.825587+0000","last_undegraded":"2024-02-19T14:57:55.825587+0000","last_fullsized":"2024-02-19T14:57:55.825587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:25.016688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823385+0000","last_change":"2024-02-19T14:57:45.774821+0000","last_active":"2024-02-19T14:57:55.823385+0000","last_peered":"2024-02-19T14:57:55.823385+0000","last_clean":"2024-02-19T14:57:55.823385+0000","last_became_active":"2024-02-19T14:57:45.774541+0000","last_became_peered":"2024-02-19T14:57:45.774541+0000","last_unstale":"2024-02-19T14:57:55.823385+0000","last_undegraded":"2024-02-19T14:57:55.823385+0000","last_fullsized":"2024-02-19T14:57:55.823385+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:34.280659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824647+0000","last_change":"2024-02-19T14:57:54.829474+0000","last_active":"2024-02-19T14:57:55.824647+0000","last_peered":"2024-02-19T14:57:55.824647+0000","last_clean":"2024-02-19T14:57:55.824647+0000","last_became_active":"2024-02-19T14:57:54.829146+0000","last_became_peered":"2024-02-19T14:57:54.829146+0000","last_unstale":"2024-02-19T14:57:55.824647+0000","last_undegraded":"2024-02-19T14:57:55.824647+0000","last_fullsized":"2024-02-19T14:57:55.824647+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.602087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819718+0000","last_change":"2024-02-19T14:57:45.784067+0000","last_active":"2024-02-19T14:57:55.819718+0000","last_peered":"2024-02-19T14:57:55.819718+0000","last_clean":"2024-02-19T14:57:55.819718+0000","last_became_active":"2024-02-19T14:57:45.783769+0000","last_became_peered":"2024-02-19T14:57:45.783769+0000","last_unstale":"2024-02-19T14:57:55.819718+0000","last_undegraded":"2024-02-19T14:57:55.819718+0000","last_fullsized":"2024-02-19T14:57:55.819718+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:37.778199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825493+0000","last_change":"2024-02-19T14:57:54.832333+0000","last_active":"2024-02-19T14:57:55.825493+0000","last_peered":"2024-02-19T14:57:55.825493+0000","last_clean":"2024-02-19T14:57:55.825493+0000","last_became_active":"2024-02-19T14:57:54.832059+0000","last_became_peered":"2024-02-19T14:57:54.832059+0000","last_unstale":"2024-02-19T14:57:55.825493+0000","last_undegraded":"2024-02-19T14:57:55.825493+0000","last_fullsized":"2024-02-19T14:57:55.825493+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.602087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824787+0000","last_change":"2024-02-19T14:57:45.776863+0000","last_active":"2024-02-19T14:57:55.824787+0000","last_peered":"2024-02-19T14:57:55.824787+0000","last_clean":"2024-02-19T14:57:55.824787+0000","last_became_active":"2024-02-19T14:57:45.776525+0000","last_became_peered":"2024-02-19T14:57:45.776525+0000","last_unstale":"2024-02-19T14:57:55.824787+0000","last_undegraded":"2024-02-19T14:57:55.824787+0000","last_fullsized":"2024-02-19T14:57:55.824787+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:29.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825716+0000","last_change":"2024-02-19T14:57:54.820132+0000","last_active":"2024-02-19T14:57:55.825716+0000","last_peered":"2024-02-19T14:57:55.825716+0000","last_clean":"2024-02-19T14:57:55.825716+0000","last_became_active":"2024-02-19T14:57:54.819765+0000","last_became_peered":"2024-02-19T14:57:54.819765+0000","last_unstale":"2024-02-19T14:57:55.825716+0000","last_undegraded":"2024-02-19T14:57:55.825716+0000","last_fullsized":"2024-02-19T14:57:55.825716+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:38.343447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994212+0000","last_change":"2024-02-19T14:57:45.773937+0000","last_active":"2024-02-19T14:57:56.994212+0000","last_peered":"2024-02-19T14:57:56.994212+0000","last_clean":"2024-02-19T14:57:56.994212+0000","last_became_active":"2024-02-19T14:57:45.773657+0000","last_became_peered":"2024-02-19T14:57:45.773657+0000","last_unstale":"2024-02-19T14:57:56.994212+0000","last_undegraded":"2024-02-19T14:57:56.994212+0000","last_fullsized":"2024-02-19T14:57:56.994212+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825575+0000","last_change":"2024-02-19T14:57:54.820021+0000","last_active":"2024-02-19T14:57:55.825575+0000","last_peered":"2024-02-19T14:57:55.825575+0000","last_clean":"2024-02-19T14:57:55.825575+0000","last_became_active":"2024-02-19T14:57:54.819576+0000","last_became_peered":"2024-02-19T14:57:54.819576+0000","last_unstale":"2024-02-19T14:57:55.825575+0000","last_undegraded":"2024-02-19T14:57:55.825575+0000","last_fullsized":"2024-02-19T14:57:55.825575+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:37.441462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824941+0000","last_change":"2024-02-19T14:57:45.776452+0000","last_active":"2024-02-19T14:57:55.824941+0000","last_peered":"2024-02-19T14:57:55.824941+0000","last_clean":"2024-02-19T14:57:55.824941+0000","last_became_active":"2024-02-19T14:57:45.775815+0000","last_became_peered":"2024-02-19T14:57:45.775815+0000","last_unstale":"2024-02-19T14:57:55.824941+0000","last_undegraded":"2024-02-19T14:57:55.824941+0000","last_fullsized":"2024-02-19T14:57:55.824941+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.339211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825699+0000","last_change":"2024-02-19T14:57:54.833743+0000","last_active":"2024-02-19T14:57:55.825699+0000","last_peered":"2024-02-19T14:57:55.825699+0000","last_clean":"2024-02-19T14:57:55.825699+0000","last_became_active":"2024-02-19T14:57:54.833488+0000","last_became_peered":"2024-02-19T14:57:54.833488+0000","last_unstale":"2024-02-19T14:57:55.825699+0000","last_undegraded":"2024-02-19T14:57:55.825699+0000","last_fullsized":"2024-02-19T14:57:55.825699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:05.123628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819695+0000","last_change":"2024-02-19T14:57:45.773952+0000","last_active":"2024-02-19T14:57:55.819695+0000","last_peered":"2024-02-19T14:57:55.819695+0000","last_clean":"2024-02-19T14:57:55.819695+0000","last_became_active":"2024-02-19T14:57:45.773492+0000","last_became_peered":"2024-02-19T14:57:45.773492+0000","last_unstale":"2024-02-19T14:57:55.819695+0000","last_undegraded":"2024-02-19T14:57:55.819695+0000","last_fullsized":"2024-02-19T14:57:55.819695+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:31.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819110+0000","last_change":"2024-02-19T14:57:54.831464+0000","last_active":"2024-02-19T14:57:55.819110+0000","last_peered":"2024-02-19T14:57:55.819110+0000","last_clean":"2024-02-19T14:57:55.819110+0000","last_became_active":"2024-02-19T14:57:54.831192+0000","last_became_peered":"2024-02-19T14:57:54.831192+0000","last_unstale":"2024-02-19T14:57:55.819110+0000","last_undegraded":"2024-02-19T14:57:55.819110+0000","last_fullsized":"2024-02-19T14:57:55.819110+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:24.433516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823446+0000","last_change":"2024-02-19T14:57:45.781678+0000","last_active":"2024-02-19T14:57:55.823446+0000","last_peered":"2024-02-19T14:57:55.823446+0000","last_clean":"2024-02-19T14:57:55.823446+0000","last_became_active":"2024-02-19T14:57:45.781402+0000","last_became_peered":"2024-02-19T14:57:45.781402+0000","last_unstale":"2024-02-19T14:57:55.823446+0000","last_undegraded":"2024-02-19T14:57:55.823446+0000","last_fullsized":"2024-02-19T14:57:55.823446+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:39.684791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823340+0000","last_change":"2024-02-19T14:57:54.821465+0000","last_active":"2024-02-19T14:57:55.823340+0000","last_peered":"2024-02-19T14:57:55.823340+0000","last_clean":"2024-02-19T14:57:55.823340+0000","last_became_active":"2024-02-19T14:57:54.821199+0000","last_became_peered":"2024-02-19T14:57:54.821199+0000","last_unstale":"2024-02-19T14:57:55.823340+0000","last_undegraded":"2024-02-19T14:57:55.823340+0000","last_fullsized":"2024-02-19T14:57:55.823340+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:10.640894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825748+0000","last_change":"2024-02-19T14:57:45.779802+0000","last_active":"2024-02-19T14:57:55.825748+0000","last_peered":"2024-02-19T14:57:55.825748+0000","last_clean":"2024-02-19T14:57:55.825748+0000","last_became_active":"2024-02-19T14:57:45.779534+0000","last_became_peered":"2024-02-19T14:57:45.779534+0000","last_unstale":"2024-02-19T14:57:55.825748+0000","last_undegraded":"2024-02-19T14:57:55.825748+0000","last_fullsized":"2024-02-19T14:57:55.825748+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995294+0000","last_change":"2024-02-19T14:57:54.833379+0000","last_active":"2024-02-19T14:57:56.995294+0000","last_peered":"2024-02-19T14:57:56.995294+0000","last_clean":"2024-02-19T14:57:56.995294+0000","last_became_active":"2024-02-19T14:57:54.833074+0000","last_became_peered":"2024-02-19T14:57:54.833074+0000","last_unstale":"2024-02-19T14:57:56.995294+0000","last_undegraded":"2024-02-19T14:57:56.995294+0000","last_fullsized":"2024-02-19T14:57:56.995294+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:35:44.451005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819212+0000","last_change":"2024-02-19T14:57:45.785669+0000","last_active":"2024-02-19T14:57:55.819212+0000","last_peered":"2024-02-19T14:57:55.819212+0000","last_clean":"2024-02-19T14:57:55.819212+0000","last_became_active":"2024-02-19T14:57:45.785083+0000","last_became_peered":"2024-02-19T14:57:45.785083+0000","last_unstale":"2024-02-19T14:57:55.819212+0000","last_undegraded":"2024-02-19T14:57:55.819212+0000","last_fullsized":"2024-02-19T14:57:55.819212+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:00.318619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823002+0000","last_change":"2024-02-19T14:57:54.823040+0000","last_active":"2024-02-19T14:57:55.823002+0000","last_peered":"2024-02-19T14:57:55.823002+0000","last_clean":"2024-02-19T14:57:55.823002+0000","last_became_active":"2024-02-19T14:57:54.822774+0000","last_became_peered":"2024-02-19T14:57:54.822774+0000","last_unstale":"2024-02-19T14:57:55.823002+0000","last_undegraded":"2024-02-19T14:57:55.823002+0000","last_fullsized":"2024-02-19T14:57:55.823002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:56:06.520044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824952+0000","last_change":"2024-02-19T14:57:45.776205+0000","last_active":"2024-02-19T14:57:55.824952+0000","last_peered":"2024-02-19T14:57:55.824952+0000","last_clean":"2024-02-19T14:57:55.824952+0000","last_became_active":"2024-02-19T14:57:45.775407+0000","last_became_peered":"2024-02-19T14:57:45.775407+0000","last_unstale":"2024-02-19T14:57:55.824952+0000","last_undegraded":"2024-02-19T14:57:55.824952+0000","last_fullsized":"2024-02-19T14:57:55.824952+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:37.658842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818843+0000","last_change":"2024-02-19T14:57:54.829145+0000","last_active":"2024-02-19T14:57:55.818843+0000","last_peered":"2024-02-19T14:57:55.818843+0000","last_clean":"2024-02-19T14:57:55.818843+0000","last_became_active":"2024-02-19T14:57:54.827708+0000","last_became_peered":"2024-02-19T14:57:54.827708+0000","last_unstale":"2024-02-19T14:57:55.818843+0000","last_undegraded":"2024-02-19T14:57:55.818843+0000","last_fullsized":"2024-02-19T14:57:55.818843+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:17.275536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994911+0000","last_change":"2024-02-19T14:57:45.774589+0000","last_active":"2024-02-19T14:57:56.994911+0000","last_peered":"2024-02-19T14:57:56.994911+0000","last_clean":"2024-02-19T14:57:56.994911+0000","last_became_active":"2024-02-19T14:57:45.774344+0000","last_became_peered":"2024-02-19T14:57:45.774344+0000","last_unstale":"2024-02-19T14:57:56.994911+0000","last_undegraded":"2024-02-19T14:57:56.994911+0000","last_fullsized":"2024-02-19T14:57:56.994911+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:04.979975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995451+0000","last_change":"2024-02-19T14:57:54.816982+0000","last_active":"2024-02-19T14:57:56.995451+0000","last_peered":"2024-02-19T14:57:56.995451+0000","last_clean":"2024-02-19T14:57:56.995451+0000","last_became_active":"2024-02-19T14:57:54.816702+0000","last_became_peered":"2024-02-19T14:57:54.816702+0000","last_unstale":"2024-02-19T14:57:56.995451+0000","last_undegraded":"2024-02-19T14:57:56.995451+0000","last_fullsized":"2024-02-19T14:57:56.995451+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:11.954004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994855+0000","last_change":"2024-02-19T14:57:45.774703+0000","last_active":"2024-02-19T14:57:56.994855+0000","last_peered":"2024-02-19T14:57:56.994855+0000","last_clean":"2024-02-19T14:57:56.994855+0000","last_became_active":"2024-02-19T14:57:45.773430+0000","last_became_peered":"2024-02-19T14:57:45.773430+0000","last_unstale":"2024-02-19T14:57:56.994855+0000","last_undegraded":"2024-02-19T14:57:56.994855+0000","last_fullsized":"2024-02-19T14:57:56.994855+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:56.911480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995277+0000","last_change":"2024-02-19T14:57:54.816335+0000","last_active":"2024-02-19T14:57:56.995277+0000","last_peered":"2024-02-19T14:57:56.995277+0000","last_clean":"2024-02-19T14:57:56.995277+0000","last_became_active":"2024-02-19T14:57:54.815840+0000","last_became_peered":"2024-02-19T14:57:54.815840+0000","last_unstale":"2024-02-19T14:57:56.995277+0000","last_undegraded":"2024-02-19T14:57:56.995277+0000","last_fullsized":"2024-02-19T14:57:56.995277+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:37:51.650069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995326+0000","last_change":"2024-02-19T14:57:45.780128+0000","last_active":"2024-02-19T14:57:56.995326+0000","last_peered":"2024-02-19T14:57:56.995326+0000","last_clean":"2024-02-19T14:57:56.995326+0000","last_became_active":"2024-02-19T14:57:45.779825+0000","last_became_peered":"2024-02-19T14:57:45.779825+0000","last_unstale":"2024-02-19T14:57:56.995326+0000","last_undegraded":"2024-02-19T14:57:56.995326+0000","last_fullsized":"2024-02-19T14:57:56.995326+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:26.050909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825694+0000","last_change":"2024-02-19T14:57:54.828181+0000","last_active":"2024-02-19T14:57:55.825694+0000","last_peered":"2024-02-19T14:57:55.825694+0000","last_clean":"2024-02-19T14:57:55.825694+0000","last_became_active":"2024-02-19T14:57:54.827867+0000","last_became_peered":"2024-02-19T14:57:54.827867+0000","last_unstale":"2024-02-19T14:57:55.825694+0000","last_undegraded":"2024-02-19T14:57:55.825694+0000","last_fullsized":"2024-02-19T14:57:55.825694+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:13.020379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994212+0000","last_change":"2024-02-19T14:57:53.820089+0000","last_active":"2024-02-19T14:57:56.994212+0000","last_peered":"2024-02-19T14:57:56.994212+0000","last_clean":"2024-02-19T14:57:56.994212+0000","last_became_active":"2024-02-19T14:57:53.819799+0000","last_became_peered":"2024-02-19T14:57:53.819799+0000","last_unstale":"2024-02-19T14:57:56.994212+0000","last_undegraded":"2024-02-19T14:57:56.994212+0000","last_fullsized":"2024-02-19T14:57:56.994212+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:09.687163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825974+0000","last_change":"2024-02-19T14:57:45.781182+0000","last_active":"2024-02-19T14:57:55.825974+0000","last_peered":"2024-02-19T14:57:55.825974+0000","last_clean":"2024-02-19T14:57:55.825974+0000","last_became_active":"2024-02-19T14:57:45.780764+0000","last_became_peered":"2024-02-19T14:57:45.780764+0000","last_unstale":"2024-02-19T14:57:55.825974+0000","last_undegraded":"2024-02-19T14:57:55.825974+0000","last_fullsized":"2024-02-19T14:57:55.825974+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:23.510855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":26,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826625+0000","last_change":"2024-02-19T14:57:44.748896+0000","last_active":"2024-02-19T14:57:55.826625+0000","last_peered":"2024-02-19T14:57:55.826625+0000","last_clean":"2024-02-19T14:57:55.826625+0000","last_became_active":"2024-02-19T14:57:44.748624+0000","last_became_peered":"2024-02-19T14:57:44.748624+0000","last_unstale":"2024-02-19T14:57:55.826625+0000","last_undegraded":"2024-02-19T14:57:55.826625+0000","last_fullsized":"2024-02-19T14:57:55.826625+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:53:27.719470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826462+0000","last_change":"2024-02-19T14:57:54.829689+0000","last_active":"2024-02-19T14:57:55.826462+0000","last_peered":"2024-02-19T14:57:55.826462+0000","last_clean":"2024-02-19T14:57:55.826462+0000","last_became_active":"2024-02-19T14:57:54.829434+0000","last_became_peered":"2024-02-19T14:57:54.829434+0000","last_unstale":"2024-02-19T14:57:55.826462+0000","last_undegraded":"2024-02-19T14:57:55.826462+0000","last_fullsized":"2024-02-19T14:57:55.826462+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:01.392908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995289+0000","last_change":"2024-02-19T14:57:53.816514+0000","last_active":"2024-02-19T14:57:56.995289+0000","last_peered":"2024-02-19T14:57:56.995289+0000","last_clean":"2024-02-19T14:57:56.995289+0000","last_became_active":"2024-02-19T14:57:53.816138+0000","last_became_peered":"2024-02-19T14:57:53.816138+0000","last_unstale":"2024-02-19T14:57:56.995289+0000","last_undegraded":"2024-02-19T14:57:56.995289+0000","last_fullsized":"2024-02-19T14:57:56.995289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:22.636629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818746+0000","last_change":"2024-02-19T14:57:45.778729+0000","last_active":"2024-02-19T14:57:55.818746+0000","last_peered":"2024-02-19T14:57:55.818746+0000","last_clean":"2024-02-19T14:57:55.818746+0000","last_became_active":"2024-02-19T14:57:45.778460+0000","last_became_peered":"2024-02-19T14:57:45.778460+0000","last_unstale":"2024-02-19T14:57:55.818746+0000","last_undegraded":"2024-02-19T14:57:55.818746+0000","last_fullsized":"2024-02-19T14:57:55.818746+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:46.115236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995061+0000","last_change":"2024-02-19T14:57:44.759512+0000","last_active":"2024-02-19T14:57:56.995061+0000","last_peered":"2024-02-19T14:57:56.995061+0000","last_clean":"2024-02-19T14:57:56.995061+0000","last_became_active":"2024-02-19T14:57:44.759251+0000","last_became_peered":"2024-02-19T14:57:44.759251+0000","last_unstale":"2024-02-19T14:57:56.995061+0000","last_undegraded":"2024-02-19T14:57:56.995061+0000","last_fullsized":"2024-02-19T14:57:56.995061+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:23.064686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822886+0000","last_change":"2024-02-19T14:57:54.830445+0000","last_active":"2024-02-19T14:57:55.822886+0000","last_peered":"2024-02-19T14:57:55.822886+0000","last_clean":"2024-02-19T14:57:55.822886+0000","last_became_active":"2024-02-19T14:57:54.829989+0000","last_became_peered":"2024-02-19T14:57:54.829989+0000","last_unstale":"2024-02-19T14:57:55.822886+0000","last_undegraded":"2024-02-19T14:57:55.822886+0000","last_fullsized":"2024-02-19T14:57:55.822886+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:38.403195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826036+0000","last_change":"2024-02-19T14:57:53.824642+0000","last_active":"2024-02-19T14:57:55.826036+0000","last_peered":"2024-02-19T14:57:55.826036+0000","last_clean":"2024-02-19T14:57:55.826036+0000","last_became_active":"2024-02-19T14:57:53.824274+0000","last_became_peered":"2024-02-19T14:57:53.824274+0000","last_unstale":"2024-02-19T14:57:55.826036+0000","last_undegraded":"2024-02-19T14:57:55.826036+0000","last_fullsized":"2024-02-19T14:57:55.826036+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:50.343572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994687+0000","last_change":"2024-02-19T14:57:45.775749+0000","last_active":"2024-02-19T14:57:56.994687+0000","last_peered":"2024-02-19T14:57:56.994687+0000","last_clean":"2024-02-19T14:57:56.994687+0000","last_became_active":"2024-02-19T14:57:45.775481+0000","last_became_peered":"2024-02-19T14:57:45.775481+0000","last_unstale":"2024-02-19T14:57:56.994687+0000","last_undegraded":"2024-02-19T14:57:56.994687+0000","last_fullsized":"2024-02-19T14:57:56.994687+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.996066+0000","last_change":"2024-02-19T14:57:44.746085+0000","last_active":"2024-02-19T14:57:56.996066+0000","last_peered":"2024-02-19T14:57:56.996066+0000","last_clean":"2024-02-19T14:57:56.996066+0000","last_became_active":"2024-02-19T14:57:44.745804+0000","last_became_peered":"2024-02-19T14:57:44.745804+0000","last_unstale":"2024-02-19T14:57:56.996066+0000","last_undegraded":"2024-02-19T14:57:56.996066+0000","last_fullsized":"2024-02-19T14:57:56.996066+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818833+0000","last_change":"2024-02-19T14:57:54.824114+0000","last_active":"2024-02-19T14:57:55.818833+0000","last_peered":"2024-02-19T14:57:55.818833+0000","last_clean":"2024-02-19T14:57:55.818833+0000","last_became_active":"2024-02-19T14:57:54.823842+0000","last_became_peered":"2024-02-19T14:57:54.823842+0000","last_unstale":"2024-02-19T14:57:55.818833+0000","last_undegraded":"2024-02-19T14:57:55.818833+0000","last_fullsized":"2024-02-19T14:57:55.818833+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:43.035644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826093+0000","last_change":"2024-02-19T14:57:53.819661+0000","last_active":"2024-02-19T14:57:55.826093+0000","last_peered":"2024-02-19T14:57:55.826093+0000","last_clean":"2024-02-19T14:57:55.826093+0000","last_became_active":"2024-02-19T14:57:53.819408+0000","last_became_peered":"2024-02-19T14:57:53.819408+0000","last_unstale":"2024-02-19T14:57:55.826093+0000","last_undegraded":"2024-02-19T14:57:55.826093+0000","last_fullsized":"2024-02-19T14:57:55.826093+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:22.076038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824979+0000","last_change":"2024-02-19T14:57:45.784232+0000","last_active":"2024-02-19T14:57:55.824979+0000","last_peered":"2024-02-19T14:57:55.824979+0000","last_clean":"2024-02-19T14:57:55.824979+0000","last_became_active":"2024-02-19T14:57:45.784007+0000","last_became_peered":"2024-02-19T14:57:45.784007+0000","last_unstale":"2024-02-19T14:57:55.824979+0000","last_undegraded":"2024-02-19T14:57:55.824979+0000","last_fullsized":"2024-02-19T14:57:55.824979+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:50.022179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823309+0000","last_change":"2024-02-19T14:57:44.754196+0000","last_active":"2024-02-19T14:57:55.823309+0000","last_peered":"2024-02-19T14:57:55.823309+0000","last_clean":"2024-02-19T14:57:55.823309+0000","last_became_active":"2024-02-19T14:57:44.753938+0000","last_became_peered":"2024-02-19T14:57:44.753938+0000","last_unstale":"2024-02-19T14:57:55.823309+0000","last_undegraded":"2024-02-19T14:57:55.823309+0000","last_fullsized":"2024-02-19T14:57:55.823309+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:17.948844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825843+0000","last_change":"2024-02-19T14:57:54.822487+0000","last_active":"2024-02-19T14:57:55.825843+0000","last_peered":"2024-02-19T14:57:55.825843+0000","last_clean":"2024-02-19T14:57:55.825843+0000","last_became_active":"2024-02-19T14:57:54.822187+0000","last_became_peered":"2024-02-19T14:57:54.822187+0000","last_unstale":"2024-02-19T14:57:55.825843+0000","last_undegraded":"2024-02-19T14:57:55.825843+0000","last_fullsized":"2024-02-19T14:57:55.825843+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:32.400071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994644+0000","last_change":"2024-02-19T14:57:53.823653+0000","last_active":"2024-02-19T14:57:56.994644+0000","last_peered":"2024-02-19T14:57:56.994644+0000","last_clean":"2024-02-19T14:57:56.994644+0000","last_became_active":"2024-02-19T14:57:53.822920+0000","last_became_peered":"2024-02-19T14:57:53.822920+0000","last_unstale":"2024-02-19T14:57:56.994644+0000","last_undegraded":"2024-02-19T14:57:56.994644+0000","last_fullsized":"2024-02-19T14:57:56.994644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:02.674757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825631+0000","last_change":"2024-02-19T14:57:45.776534+0000","last_active":"2024-02-19T14:57:55.825631+0000","last_peered":"2024-02-19T14:57:55.825631+0000","last_clean":"2024-02-19T14:57:55.825631+0000","last_became_active":"2024-02-19T14:57:45.776277+0000","last_became_peered":"2024-02-19T14:57:45.776277+0000","last_unstale":"2024-02-19T14:57:55.825631+0000","last_undegraded":"2024-02-19T14:57:55.825631+0000","last_fullsized":"2024-02-19T14:57:55.825631+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826026+0000","last_change":"2024-02-19T14:57:44.759675+0000","last_active":"2024-02-19T14:57:55.826026+0000","last_peered":"2024-02-19T14:57:55.826026+0000","last_clean":"2024-02-19T14:57:55.826026+0000","last_became_active":"2024-02-19T14:57:44.759404+0000","last_became_peered":"2024-02-19T14:57:44.759404+0000","last_unstale":"2024-02-19T14:57:55.826026+0000","last_undegraded":"2024-02-19T14:57:55.826026+0000","last_fullsized":"2024-02-19T14:57:55.826026+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:34.658675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826373+0000","last_change":"2024-02-19T14:57:54.817366+0000","last_active":"2024-02-19T14:57:55.826373+0000","last_peered":"2024-02-19T14:57:55.826373+0000","last_clean":"2024-02-19T14:57:55.826373+0000","last_became_active":"2024-02-19T14:57:54.817126+0000","last_became_peered":"2024-02-19T14:57:54.817126+0000","last_unstale":"2024-02-19T14:57:55.826373+0000","last_undegraded":"2024-02-19T14:57:55.826373+0000","last_fullsized":"2024-02-19T14:57:55.826373+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:25.058182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825191+0000","last_change":"2024-02-19T14:57:53.814585+0000","last_active":"2024-02-19T14:57:55.825191+0000","last_peered":"2024-02-19T14:57:55.825191+0000","last_clean":"2024-02-19T14:57:55.825191+0000","last_became_active":"2024-02-19T14:57:53.814309+0000","last_became_peered":"2024-02-19T14:57:53.814309+0000","last_unstale":"2024-02-19T14:57:55.825191+0000","last_undegraded":"2024-02-19T14:57:55.825191+0000","last_fullsized":"2024-02-19T14:57:55.825191+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:54.429397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823403+0000","last_change":"2024-02-19T14:57:45.777221+0000","last_active":"2024-02-19T14:57:55.823403+0000","last_peered":"2024-02-19T14:57:55.823403+0000","last_clean":"2024-02-19T14:57:55.823403+0000","last_became_active":"2024-02-19T14:57:45.776887+0000","last_became_peered":"2024-02-19T14:57:45.776887+0000","last_unstale":"2024-02-19T14:57:55.823403+0000","last_undegraded":"2024-02-19T14:57:55.823403+0000","last_fullsized":"2024-02-19T14:57:55.823403+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:43.785509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994740+0000","last_change":"2024-02-19T14:57:44.759410+0000","last_active":"2024-02-19T14:57:56.994740+0000","last_peered":"2024-02-19T14:57:56.994740+0000","last_clean":"2024-02-19T14:57:56.994740+0000","last_became_active":"2024-02-19T14:57:44.759125+0000","last_became_peered":"2024-02-19T14:57:44.759125+0000","last_unstale":"2024-02-19T14:57:56.994740+0000","last_undegraded":"2024-02-19T14:57:56.994740+0000","last_fullsized":"2024-02-19T14:57:56.994740+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:47.742311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825656+0000","last_change":"2024-02-19T14:57:54.831577+0000","last_active":"2024-02-19T14:57:55.825656+0000","last_peered":"2024-02-19T14:57:55.825656+0000","last_clean":"2024-02-19T14:57:55.825656+0000","last_became_active":"2024-02-19T14:57:54.830980+0000","last_became_peered":"2024-02-19T14:57:54.830980+0000","last_unstale":"2024-02-19T14:57:55.825656+0000","last_undegraded":"2024-02-19T14:57:55.825656+0000","last_fullsized":"2024-02-19T14:57:55.825656+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:38.343447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819427+0000","last_change":"2024-02-19T14:57:53.817680+0000","last_active":"2024-02-19T14:57:55.819427+0000","last_peered":"2024-02-19T14:57:55.819427+0000","last_clean":"2024-02-19T14:57:55.819427+0000","last_became_active":"2024-02-19T14:57:53.817435+0000","last_became_peered":"2024-02-19T14:57:53.817435+0000","last_unstale":"2024-02-19T14:57:55.819427+0000","last_undegraded":"2024-02-19T14:57:55.819427+0000","last_fullsized":"2024-02-19T14:57:55.819427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:24.053686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823526+0000","last_change":"2024-02-19T14:57:45.778260+0000","last_active":"2024-02-19T14:57:55.823526+0000","last_peered":"2024-02-19T14:57:55.823526+0000","last_clean":"2024-02-19T14:57:55.823526+0000","last_became_active":"2024-02-19T14:57:45.778006+0000","last_became_peered":"2024-02-19T14:57:45.778006+0000","last_unstale":"2024-02-19T14:57:55.823526+0000","last_undegraded":"2024-02-19T14:57:55.823526+0000","last_fullsized":"2024-02-19T14:57:55.823526+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:35.800806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823279+0000","last_change":"2024-02-19T14:57:44.757229+0000","last_active":"2024-02-19T14:57:55.823279+0000","last_peered":"2024-02-19T14:57:55.823279+0000","last_clean":"2024-02-19T14:57:55.823279+0000","last_became_active":"2024-02-19T14:57:44.756966+0000","last_became_peered":"2024-02-19T14:57:44.756966+0000","last_unstale":"2024-02-19T14:57:55.823279+0000","last_undegraded":"2024-02-19T14:57:55.823279+0000","last_fullsized":"2024-02-19T14:57:55.823279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:12.299965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819929+0000","last_change":"2024-02-19T14:57:54.823936+0000","last_active":"2024-02-19T14:57:55.819929+0000","last_peered":"2024-02-19T14:57:55.819929+0000","last_clean":"2024-02-19T14:57:55.819929+0000","last_became_active":"2024-02-19T14:57:54.823668+0000","last_became_peered":"2024-02-19T14:57:54.823668+0000","last_unstale":"2024-02-19T14:57:55.819929+0000","last_undegraded":"2024-02-19T14:57:55.819929+0000","last_fullsized":"2024-02-19T14:57:55.819929+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:16.936732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825752+0000","last_change":"2024-02-19T14:57:53.819203+0000","last_active":"2024-02-19T14:57:55.825752+0000","last_peered":"2024-02-19T14:57:55.825752+0000","last_clean":"2024-02-19T14:57:55.825752+0000","last_became_active":"2024-02-19T14:57:53.818961+0000","last_became_peered":"2024-02-19T14:57:53.818961+0000","last_unstale":"2024-02-19T14:57:55.825752+0000","last_undegraded":"2024-02-19T14:57:55.825752+0000","last_fullsized":"2024-02-19T14:57:55.825752+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:45.718331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826421+0000","last_change":"2024-02-19T14:57:45.781313+0000","last_active":"2024-02-19T14:57:55.826421+0000","last_peered":"2024-02-19T14:57:55.826421+0000","last_clean":"2024-02-19T14:57:55.826421+0000","last_became_active":"2024-02-19T14:57:45.778982+0000","last_became_peered":"2024-02-19T14:57:45.778982+0000","last_unstale":"2024-02-19T14:57:55.826421+0000","last_undegraded":"2024-02-19T14:57:55.826421+0000","last_fullsized":"2024-02-19T14:57:55.826421+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:23.251435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825378+0000","last_change":"2024-02-19T14:57:44.757934+0000","last_active":"2024-02-19T14:57:55.825378+0000","last_peered":"2024-02-19T14:57:55.825378+0000","last_clean":"2024-02-19T14:57:55.825378+0000","last_became_active":"2024-02-19T14:57:44.757660+0000","last_became_peered":"2024-02-19T14:57:44.757660+0000","last_unstale":"2024-02-19T14:57:55.825378+0000","last_undegraded":"2024-02-19T14:57:55.825378+0000","last_fullsized":"2024-02-19T14:57:55.825378+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825909+0000","last_change":"2024-02-19T14:57:54.817951+0000","last_active":"2024-02-19T14:57:55.825909+0000","last_peered":"2024-02-19T14:57:55.825909+0000","last_clean":"2024-02-19T14:57:55.825909+0000","last_became_active":"2024-02-19T14:57:54.817522+0000","last_became_peered":"2024-02-19T14:57:54.817522+0000","last_unstale":"2024-02-19T14:57:55.825909+0000","last_undegraded":"2024-02-19T14:57:55.825909+0000","last_fullsized":"2024-02-19T14:57:55.825909+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:06.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826173+0000","last_change":"2024-02-19T14:57:53.817990+0000","last_active":"2024-02-19T14:57:55.826173+0000","last_peered":"2024-02-19T14:57:55.826173+0000","last_clean":"2024-02-19T14:57:55.826173+0000","last_became_active":"2024-02-19T14:57:53.817661+0000","last_became_peered":"2024-02-19T14:57:53.817661+0000","last_unstale":"2024-02-19T14:57:55.826173+0000","last_undegraded":"2024-02-19T14:57:55.826173+0000","last_fullsized":"2024-02-19T14:57:55.826173+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.719948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825329+0000","last_change":"2024-02-19T14:57:45.776416+0000","last_active":"2024-02-19T14:57:55.825329+0000","last_peered":"2024-02-19T14:57:55.825329+0000","last_clean":"2024-02-19T14:57:55.825329+0000","last_became_active":"2024-02-19T14:57:45.775916+0000","last_became_peered":"2024-02-19T14:57:45.775916+0000","last_unstale":"2024-02-19T14:57:55.825329+0000","last_undegraded":"2024-02-19T14:57:55.825329+0000","last_fullsized":"2024-02-19T14:57:55.825329+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:33.825283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819592+0000","last_change":"2024-02-19T14:57:44.755422+0000","last_active":"2024-02-19T14:57:55.819592+0000","last_peered":"2024-02-19T14:57:55.819592+0000","last_clean":"2024-02-19T14:57:55.819592+0000","last_became_active":"2024-02-19T14:57:44.755152+0000","last_became_peered":"2024-02-19T14:57:44.755152+0000","last_unstale":"2024-02-19T14:57:55.819592+0000","last_undegraded":"2024-02-19T14:57:55.819592+0000","last_fullsized":"2024-02-19T14:57:55.819592+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:13.575356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824629+0000","last_change":"2024-02-19T14:57:54.826189+0000","last_active":"2024-02-19T14:57:55.824629+0000","last_peered":"2024-02-19T14:57:55.824629+0000","last_clean":"2024-02-19T14:57:55.824629+0000","last_became_active":"2024-02-19T14:57:54.825420+0000","last_became_peered":"2024-02-19T14:57:54.825420+0000","last_unstale":"2024-02-19T14:57:55.824629+0000","last_undegraded":"2024-02-19T14:57:55.824629+0000","last_fullsized":"2024-02-19T14:57:55.824629+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:42.150099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819261+0000","last_change":"2024-02-19T14:57:53.823263+0000","last_active":"2024-02-19T14:57:55.819261+0000","last_peered":"2024-02-19T14:57:55.819261+0000","last_clean":"2024-02-19T14:57:55.819261+0000","last_became_active":"2024-02-19T14:57:53.822954+0000","last_became_peered":"2024-02-19T14:57:53.822954+0000","last_unstale":"2024-02-19T14:57:55.819261+0000","last_undegraded":"2024-02-19T14:57:55.819261+0000","last_fullsized":"2024-02-19T14:57:55.819261+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:51.932100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819020+0000","last_change":"2024-02-19T14:57:45.781850+0000","last_active":"2024-02-19T14:57:55.819020+0000","last_peered":"2024-02-19T14:57:55.819020+0000","last_clean":"2024-02-19T14:57:55.819020+0000","last_became_active":"2024-02-19T14:57:45.781588+0000","last_became_peered":"2024-02-19T14:57:45.781588+0000","last_unstale":"2024-02-19T14:57:55.819020+0000","last_undegraded":"2024-02-19T14:57:55.819020+0000","last_fullsized":"2024-02-19T14:57:55.819020+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:48.140379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":25,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825805+0000","last_change":"2024-02-19T14:57:44.751600+0000","last_active":"2024-02-19T14:57:55.825805+0000","last_peered":"2024-02-19T14:57:55.825805+0000","last_clean":"2024-02-19T14:57:55.825805+0000","last_became_active":"2024-02-19T14:57:44.751311+0000","last_became_peered":"2024-02-19T14:57:44.751311+0000","last_unstale":"2024-02-19T14:57:55.825805+0000","last_undegraded":"2024-02-19T14:57:55.825805+0000","last_fullsized":"2024-02-19T14:57:55.825805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:50.448796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825401+0000","last_change":"2024-02-19T14:57:54.814493+0000","last_active":"2024-02-19T14:57:55.825401+0000","last_peered":"2024-02-19T14:57:55.825401+0000","last_clean":"2024-02-19T14:57:55.825401+0000","last_became_active":"2024-02-19T14:57:54.814248+0000","last_became_peered":"2024-02-19T14:57:54.814248+0000","last_unstale":"2024-02-19T14:57:55.825401+0000","last_undegraded":"2024-02-19T14:57:55.825401+0000","last_fullsized":"2024-02-19T14:57:55.825401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:35.195077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994552+0000","last_change":"2024-02-19T14:57:53.823746+0000","last_active":"2024-02-19T14:57:56.994552+0000","last_peered":"2024-02-19T14:57:56.994552+0000","last_clean":"2024-02-19T14:57:56.994552+0000","last_became_active":"2024-02-19T14:57:53.823475+0000","last_became_peered":"2024-02-19T14:57:53.823475+0000","last_unstale":"2024-02-19T14:57:56.994552+0000","last_undegraded":"2024-02-19T14:57:56.994552+0000","last_fullsized":"2024-02-19T14:57:56.994552+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:35:43.446510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823285+0000","last_change":"2024-02-19T14:57:45.782664+0000","last_active":"2024-02-19T14:57:55.823285+0000","last_peered":"2024-02-19T14:57:55.823285+0000","last_clean":"2024-02-19T14:57:55.823285+0000","last_became_active":"2024-02-19T14:57:45.782411+0000","last_became_peered":"2024-02-19T14:57:45.782411+0000","last_unstale":"2024-02-19T14:57:55.823285+0000","last_undegraded":"2024-02-19T14:57:55.823285+0000","last_fullsized":"2024-02-19T14:57:55.823285+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:26.461781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":28,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819709+0000","last_change":"2024-02-19T14:57:44.756497+0000","last_active":"2024-02-19T14:57:55.819709+0000","last_peered":"2024-02-19T14:57:55.819709+0000","last_clean":"2024-02-19T14:57:55.819709+0000","last_became_active":"2024-02-19T14:57:44.756213+0000","last_became_peered":"2024-02-19T14:57:44.756213+0000","last_unstale":"2024-02-19T14:57:55.819709+0000","last_undegraded":"2024-02-19T14:57:55.819709+0000","last_fullsized":"2024-02-19T14:57:55.819709+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:28.745803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995601+0000","last_change":"2024-02-19T14:57:54.831104+0000","last_active":"2024-02-19T14:57:56.995601+0000","last_peered":"2024-02-19T14:57:56.995601+0000","last_clean":"2024-02-19T14:57:56.995601+0000","last_became_active":"2024-02-19T14:57:54.830831+0000","last_became_peered":"2024-02-19T14:57:54.830831+0000","last_unstale":"2024-02-19T14:57:56.995601+0000","last_undegraded":"2024-02-19T14:57:56.995601+0000","last_fullsized":"2024-02-19T14:57:56.995601+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:03.679252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995835+0000","last_change":"2024-02-19T14:57:53.811681+0000","last_active":"2024-02-19T14:57:56.995835+0000","last_peered":"2024-02-19T14:57:56.995835+0000","last_clean":"2024-02-19T14:57:56.995835+0000","last_became_active":"2024-02-19T14:57:53.811323+0000","last_became_peered":"2024-02-19T14:57:53.811323+0000","last_unstale":"2024-02-19T14:57:56.995835+0000","last_undegraded":"2024-02-19T14:57:56.995835+0000","last_fullsized":"2024-02-19T14:57:56.995835+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:37.807076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994805+0000","last_change":"2024-02-19T14:57:45.780739+0000","last_active":"2024-02-19T14:57:56.994805+0000","last_peered":"2024-02-19T14:57:56.994805+0000","last_clean":"2024-02-19T14:57:56.994805+0000","last_became_active":"2024-02-19T14:57:45.780414+0000","last_became_peered":"2024-02-19T14:57:45.780414+0000","last_unstale":"2024-02-19T14:57:56.994805+0000","last_undegraded":"2024-02-19T14:57:56.994805+0000","last_fullsized":"2024-02-19T14:57:56.994805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:25.103177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994783+0000","last_change":"2024-02-19T14:57:44.751458+0000","last_active":"2024-02-19T14:57:56.994783+0000","last_peered":"2024-02-19T14:57:56.994783+0000","last_clean":"2024-02-19T14:57:56.994783+0000","last_became_active":"2024-02-19T14:57:44.751194+0000","last_became_peered":"2024-02-19T14:57:44.751194+0000","last_unstale":"2024-02-19T14:57:56.994783+0000","last_undegraded":"2024-02-19T14:57:56.994783+0000","last_fullsized":"2024-02-19T14:57:56.994783+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:11:44.993761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995269+0000","last_change":"2024-02-19T14:57:54.826834+0000","last_active":"2024-02-19T14:57:56.995269+0000","last_peered":"2024-02-19T14:57:56.995269+0000","last_clean":"2024-02-19T14:57:56.995269+0000","last_became_active":"2024-02-19T14:57:54.826547+0000","last_became_peered":"2024-02-19T14:57:54.826547+0000","last_unstale":"2024-02-19T14:57:56.995269+0000","last_undegraded":"2024-02-19T14:57:56.995269+0000","last_fullsized":"2024-02-19T14:57:56.995269+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:29.003031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825662+0000","last_change":"2024-02-19T14:57:53.821078+0000","last_active":"2024-02-19T14:57:55.825662+0000","last_peered":"2024-02-19T14:57:55.825662+0000","last_clean":"2024-02-19T14:57:55.825662+0000","last_became_active":"2024-02-19T14:57:53.820812+0000","last_became_peered":"2024-02-19T14:57:53.820812+0000","last_unstale":"2024-02-19T14:57:55.825662+0000","last_undegraded":"2024-02-19T14:57:55.825662+0000","last_fullsized":"2024-02-19T14:57:55.825662+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:56.803585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826124+0000","last_change":"2024-02-19T14:57:45.775378+0000","last_active":"2024-02-19T14:57:55.826124+0000","last_peered":"2024-02-19T14:57:55.826124+0000","last_clean":"2024-02-19T14:57:55.826124+0000","last_became_active":"2024-02-19T14:57:45.774903+0000","last_became_peered":"2024-02-19T14:57:45.774903+0000","last_unstale":"2024-02-19T14:57:55.826124+0000","last_undegraded":"2024-02-19T14:57:55.826124+0000","last_fullsized":"2024-02-19T14:57:55.826124+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819156+0000","last_change":"2024-02-19T14:57:44.748468+0000","last_active":"2024-02-19T14:57:55.819156+0000","last_peered":"2024-02-19T14:57:55.819156+0000","last_clean":"2024-02-19T14:57:55.819156+0000","last_became_active":"2024-02-19T14:57:44.748055+0000","last_became_peered":"2024-02-19T14:57:44.748055+0000","last_unstale":"2024-02-19T14:57:55.819156+0000","last_undegraded":"2024-02-19T14:57:55.819156+0000","last_fullsized":"2024-02-19T14:57:55.819156+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:32.823498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825516+0000","last_change":"2024-02-19T14:57:54.829063+0000","last_active":"2024-02-19T14:57:55.825516+0000","last_peered":"2024-02-19T14:57:55.825516+0000","last_clean":"2024-02-19T14:57:55.825516+0000","last_became_active":"2024-02-19T14:57:54.828658+0000","last_became_peered":"2024-02-19T14:57:54.828658+0000","last_unstale":"2024-02-19T14:57:55.825516+0000","last_undegraded":"2024-02-19T14:57:55.825516+0000","last_fullsized":"2024-02-19T14:57:55.825516+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:30:56.427520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995158+0000","last_change":"2024-02-19T14:57:53.827543+0000","last_active":"2024-02-19T14:57:56.995158+0000","last_peered":"2024-02-19T14:57:56.995158+0000","last_clean":"2024-02-19T14:57:56.995158+0000","last_became_active":"2024-02-19T14:57:53.827256+0000","last_became_peered":"2024-02-19T14:57:53.827256+0000","last_unstale":"2024-02-19T14:57:56.995158+0000","last_undegraded":"2024-02-19T14:57:56.995158+0000","last_fullsized":"2024-02-19T14:57:56.995158+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:54.174725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995993+0000","last_change":"2024-02-19T14:57:45.780058+0000","last_active":"2024-02-19T14:57:56.995993+0000","last_peered":"2024-02-19T14:57:56.995993+0000","last_clean":"2024-02-19T14:57:56.995993+0000","last_became_active":"2024-02-19T14:57:45.779771+0000","last_became_peered":"2024-02-19T14:57:45.779771+0000","last_unstale":"2024-02-19T14:57:56.995993+0000","last_undegraded":"2024-02-19T14:57:56.995993+0000","last_fullsized":"2024-02-19T14:57:56.995993+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994963+0000","last_change":"2024-02-19T14:57:44.751398+0000","last_active":"2024-02-19T14:57:56.994963+0000","last_peered":"2024-02-19T14:57:56.994963+0000","last_clean":"2024-02-19T14:57:56.994963+0000","last_became_active":"2024-02-19T14:57:44.751099+0000","last_became_peered":"2024-02-19T14:57:44.751099+0000","last_unstale":"2024-02-19T14:57:56.994963+0000","last_undegraded":"2024-02-19T14:57:56.994963+0000","last_fullsized":"2024-02-19T14:57:56.994963+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:51.642710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823209+0000","last_change":"2024-02-19T14:57:54.822447+0000","last_active":"2024-02-19T14:57:55.823209+0000","last_peered":"2024-02-19T14:57:55.823209+0000","last_clean":"2024-02-19T14:57:55.823209+0000","last_became_active":"2024-02-19T14:57:54.822177+0000","last_became_peered":"2024-02-19T14:57:54.822177+0000","last_unstale":"2024-02-19T14:57:55.823209+0000","last_undegraded":"2024-02-19T14:57:55.823209+0000","last_fullsized":"2024-02-19T14:57:55.823209+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:21.065898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819635+0000","last_change":"2024-02-19T14:57:53.816563+0000","last_active":"2024-02-19T14:57:55.819635+0000","last_peered":"2024-02-19T14:57:55.819635+0000","last_clean":"2024-02-19T14:57:55.819635+0000","last_became_active":"2024-02-19T14:57:53.816303+0000","last_became_peered":"2024-02-19T14:57:53.816303+0000","last_unstale":"2024-02-19T14:57:55.819635+0000","last_undegraded":"2024-02-19T14:57:55.819635+0000","last_fullsized":"2024-02-19T14:57:55.819635+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:33.963032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823479+0000","last_change":"2024-02-19T14:57:45.783960+0000","last_active":"2024-02-19T14:57:55.823479+0000","last_peered":"2024-02-19T14:57:55.823479+0000","last_clean":"2024-02-19T14:57:55.823479+0000","last_became_active":"2024-02-19T14:57:45.783704+0000","last_became_peered":"2024-02-19T14:57:45.783704+0000","last_unstale":"2024-02-19T14:57:55.823479+0000","last_undegraded":"2024-02-19T14:57:55.823479+0000","last_fullsized":"2024-02-19T14:57:55.823479+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:24.171999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":30,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825476+0000","last_change":"2024-02-19T14:57:44.758621+0000","last_active":"2024-02-19T14:57:55.825476+0000","last_peered":"2024-02-19T14:57:55.825476+0000","last_clean":"2024-02-19T14:57:55.825476+0000","last_became_active":"2024-02-19T14:57:44.758296+0000","last_became_peered":"2024-02-19T14:57:44.758296+0000","last_unstale":"2024-02-19T14:57:55.825476+0000","last_undegraded":"2024-02-19T14:57:55.825476+0000","last_fullsized":"2024-02-19T14:57:55.825476+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826140+0000","last_change":"2024-02-19T14:57:54.819816+0000","last_active":"2024-02-19T14:57:55.826140+0000","last_peered":"2024-02-19T14:57:55.826140+0000","last_clean":"2024-02-19T14:57:55.826140+0000","last_became_active":"2024-02-19T14:57:54.819239+0000","last_became_peered":"2024-02-19T14:57:54.819239+0000","last_unstale":"2024-02-19T14:57:55.826140+0000","last_undegraded":"2024-02-19T14:57:55.826140+0000","last_fullsized":"2024-02-19T14:57:55.826140+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:52.936596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822856+0000","last_change":"2024-02-19T14:57:53.822905+0000","last_active":"2024-02-19T14:57:55.822856+0000","last_peered":"2024-02-19T14:57:55.822856+0000","last_clean":"2024-02-19T14:57:55.822856+0000","last_became_active":"2024-02-19T14:57:53.822660+0000","last_became_peered":"2024-02-19T14:57:53.822660+0000","last_unstale":"2024-02-19T14:57:55.822856+0000","last_undegraded":"2024-02-19T14:57:55.822856+0000","last_fullsized":"2024-02-19T14:57:55.822856+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:03.185226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825281+0000","last_change":"2024-02-19T14:57:45.773816+0000","last_active":"2024-02-19T14:57:55.825281+0000","last_peered":"2024-02-19T14:57:55.825281+0000","last_clean":"2024-02-19T14:57:55.825281+0000","last_became_active":"2024-02-19T14:57:45.773272+0000","last_became_peered":"2024-02-19T14:57:45.773272+0000","last_unstale":"2024-02-19T14:57:55.825281+0000","last_undegraded":"2024-02-19T14:57:55.825281+0000","last_fullsized":"2024-02-19T14:57:55.825281+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819782+0000","last_change":"2024-02-19T14:57:44.751265+0000","last_active":"2024-02-19T14:57:55.819782+0000","last_peered":"2024-02-19T14:57:55.819782+0000","last_clean":"2024-02-19T14:57:55.819782+0000","last_became_active":"2024-02-19T14:57:44.750991+0000","last_became_peered":"2024-02-19T14:57:44.750991+0000","last_unstale":"2024-02-19T14:57:55.819782+0000","last_undegraded":"2024-02-19T14:57:55.819782+0000","last_fullsized":"2024-02-19T14:57:55.819782+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:32.644808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824782+0000","last_change":"2024-02-19T14:57:54.829897+0000","last_active":"2024-02-19T14:57:55.824782+0000","last_peered":"2024-02-19T14:57:55.824782+0000","last_clean":"2024-02-19T14:57:55.824782+0000","last_became_active":"2024-02-19T14:57:54.829631+0000","last_became_peered":"2024-02-19T14:57:54.829631+0000","last_unstale":"2024-02-19T14:57:55.824782+0000","last_undegraded":"2024-02-19T14:57:55.824782+0000","last_fullsized":"2024-02-19T14:57:55.824782+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:37.441462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994528+0000","last_change":"2024-02-19T14:57:53.827779+0000","last_active":"2024-02-19T14:57:56.994528+0000","last_peered":"2024-02-19T14:57:56.994528+0000","last_clean":"2024-02-19T14:57:56.994528+0000","last_became_active":"2024-02-19T14:57:53.827475+0000","last_became_peered":"2024-02-19T14:57:53.827475+0000","last_unstale":"2024-02-19T14:57:56.994528+0000","last_undegraded":"2024-02-19T14:57:56.994528+0000","last_fullsized":"2024-02-19T14:57:56.994528+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:54.429397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994968+0000","last_change":"2024-02-19T14:57:45.773622+0000","last_active":"2024-02-19T14:57:56.994968+0000","last_peered":"2024-02-19T14:57:56.994968+0000","last_clean":"2024-02-19T14:57:56.994968+0000","last_became_active":"2024-02-19T14:57:45.773349+0000","last_became_peered":"2024-02-19T14:57:45.773349+0000","last_unstale":"2024-02-19T14:57:56.994968+0000","last_undegraded":"2024-02-19T14:57:56.994968+0000","last_fullsized":"2024-02-19T14:57:56.994968+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:01.576910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826096+0000","last_change":"2024-02-19T14:57:44.757425+0000","last_active":"2024-02-19T14:57:55.826096+0000","last_peered":"2024-02-19T14:57:55.826096+0000","last_clean":"2024-02-19T14:57:55.826096+0000","last_became_active":"2024-02-19T14:57:44.757006+0000","last_became_peered":"2024-02-19T14:57:44.757006+0000","last_unstale":"2024-02-19T14:57:55.826096+0000","last_undegraded":"2024-02-19T14:57:55.826096+0000","last_fullsized":"2024-02-19T14:57:55.826096+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:41.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995458+0000","last_change":"2024-02-19T14:57:54.821943+0000","last_active":"2024-02-19T14:57:56.995458+0000","last_peered":"2024-02-19T14:57:56.995458+0000","last_clean":"2024-02-19T14:57:56.995458+0000","last_became_active":"2024-02-19T14:57:54.821645+0000","last_became_peered":"2024-02-19T14:57:54.821645+0000","last_unstale":"2024-02-19T14:57:56.995458+0000","last_undegraded":"2024-02-19T14:57:56.995458+0000","last_fullsized":"2024-02-19T14:57:56.995458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:42.150099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824836+0000","last_change":"2024-02-19T14:57:53.825554+0000","last_active":"2024-02-19T14:57:55.824836+0000","last_peered":"2024-02-19T14:57:55.824836+0000","last_clean":"2024-02-19T14:57:55.824836+0000","last_became_active":"2024-02-19T14:57:53.825279+0000","last_became_peered":"2024-02-19T14:57:53.825279+0000","last_unstale":"2024-02-19T14:57:55.824836+0000","last_undegraded":"2024-02-19T14:57:55.824836+0000","last_fullsized":"2024-02-19T14:57:55.824836+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:02.674757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995344+0000","last_change":"2024-02-19T14:57:45.781720+0000","last_active":"2024-02-19T14:57:56.995344+0000","last_peered":"2024-02-19T14:57:56.995344+0000","last_clean":"2024-02-19T14:57:56.995344+0000","last_became_active":"2024-02-19T14:57:45.781425+0000","last_became_peered":"2024-02-19T14:57:45.781425+0000","last_unstale":"2024-02-19T14:57:56.995344+0000","last_undegraded":"2024-02-19T14:57:56.995344+0000","last_fullsized":"2024-02-19T14:57:56.995344+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:36.396204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994945+0000","last_change":"2024-02-19T14:57:44.758828+0000","last_active":"2024-02-19T14:57:56.994945+0000","last_peered":"2024-02-19T14:57:56.994945+0000","last_clean":"2024-02-19T14:57:56.994945+0000","last_became_active":"2024-02-19T14:57:44.758387+0000","last_became_peered":"2024-02-19T14:57:44.758387+0000","last_unstale":"2024-02-19T14:57:56.994945+0000","last_undegraded":"2024-02-19T14:57:56.994945+0000","last_fullsized":"2024-02-19T14:57:56.994945+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826104+0000","last_change":"2024-02-19T14:57:54.821235+0000","last_active":"2024-02-19T14:57:55.826104+0000","last_peered":"2024-02-19T14:57:55.826104+0000","last_clean":"2024-02-19T14:57:55.826104+0000","last_became_active":"2024-02-19T14:57:54.820937+0000","last_became_peered":"2024-02-19T14:57:54.820937+0000","last_unstale":"2024-02-19T14:57:55.826104+0000","last_undegraded":"2024-02-19T14:57:55.826104+0000","last_fullsized":"2024-02-19T14:57:55.826104+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:09.622659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824459+0000","last_change":"2024-02-19T14:57:53.822411+0000","last_active":"2024-02-19T14:57:55.824459+0000","last_peered":"2024-02-19T14:57:55.824459+0000","last_clean":"2024-02-19T14:57:55.824459+0000","last_became_active":"2024-02-19T14:57:53.822167+0000","last_became_peered":"2024-02-19T14:57:53.822167+0000","last_unstale":"2024-02-19T14:57:55.824459+0000","last_undegraded":"2024-02-19T14:57:55.824459+0000","last_fullsized":"2024-02-19T14:57:55.824459+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:10.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995676+0000","last_change":"2024-02-19T14:57:45.780279+0000","last_active":"2024-02-19T14:57:56.995676+0000","last_peered":"2024-02-19T14:57:56.995676+0000","last_clean":"2024-02-19T14:57:56.995676+0000","last_became_active":"2024-02-19T14:57:45.779989+0000","last_became_peered":"2024-02-19T14:57:45.779989+0000","last_unstale":"2024-02-19T14:57:56.995676+0000","last_undegraded":"2024-02-19T14:57:56.995676+0000","last_fullsized":"2024-02-19T14:57:56.995676+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825920+0000","last_change":"2024-02-19T14:57:44.760014+0000","last_active":"2024-02-19T14:57:55.825920+0000","last_peered":"2024-02-19T14:57:55.825920+0000","last_clean":"2024-02-19T14:57:55.825920+0000","last_became_active":"2024-02-19T14:57:44.759749+0000","last_became_peered":"2024-02-19T14:57:44.759749+0000","last_unstale":"2024-02-19T14:57:55.825920+0000","last_undegraded":"2024-02-19T14:57:55.825920+0000","last_fullsized":"2024-02-19T14:57:55.825920+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826203+0000","last_change":"2024-02-19T14:57:54.830962+0000","last_active":"2024-02-19T14:57:55.826203+0000","last_peered":"2024-02-19T14:57:55.826203+0000","last_clean":"2024-02-19T14:57:55.826203+0000","last_became_active":"2024-02-19T14:57:54.830624+0000","last_became_peered":"2024-02-19T14:57:54.830624+0000","last_unstale":"2024-02-19T14:57:55.826203+0000","last_undegraded":"2024-02-19T14:57:55.826203+0000","last_fullsized":"2024-02-19T14:57:55.826203+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:04.023224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819644+0000","last_change":"2024-02-19T14:57:53.822010+0000","last_active":"2024-02-19T14:57:55.819644+0000","last_peered":"2024-02-19T14:57:55.819644+0000","last_clean":"2024-02-19T14:57:55.819644+0000","last_became_active":"2024-02-19T14:57:53.821590+0000","last_became_peered":"2024-02-19T14:57:53.821590+0000","last_unstale":"2024-02-19T14:57:55.819644+0000","last_undegraded":"2024-02-19T14:57:55.819644+0000","last_fullsized":"2024-02-19T14:57:55.819644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:50.523976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819446+0000","last_change":"2024-02-19T14:57:45.785469+0000","last_active":"2024-02-19T14:57:55.819446+0000","last_peered":"2024-02-19T14:57:55.819446+0000","last_clean":"2024-02-19T14:57:55.819446+0000","last_became_active":"2024-02-19T14:57:45.784759+0000","last_became_peered":"2024-02-19T14:57:45.784759+0000","last_unstale":"2024-02-19T14:57:55.819446+0000","last_undegraded":"2024-02-19T14:57:55.819446+0000","last_fullsized":"2024-02-19T14:57:55.819446+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:26.131093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994854+0000","last_change":"2024-02-19T14:57:44.752558+0000","last_active":"2024-02-19T14:57:56.994854+0000","last_peered":"2024-02-19T14:57:56.994854+0000","last_clean":"2024-02-19T14:57:56.994854+0000","last_became_active":"2024-02-19T14:57:44.752290+0000","last_became_peered":"2024-02-19T14:57:44.752290+0000","last_unstale":"2024-02-19T14:57:56.994854+0000","last_undegraded":"2024-02-19T14:57:56.994854+0000","last_fullsized":"2024-02-19T14:57:56.994854+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:55.760619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823459+0000","last_change":"2024-02-19T14:57:54.816689+0000","last_active":"2024-02-19T14:57:55.823459+0000","last_peered":"2024-02-19T14:57:55.823459+0000","last_clean":"2024-02-19T14:57:55.823459+0000","last_became_active":"2024-02-19T14:57:54.816409+0000","last_became_peered":"2024-02-19T14:57:54.816409+0000","last_unstale":"2024-02-19T14:57:55.823459+0000","last_undegraded":"2024-02-19T14:57:55.823459+0000","last_fullsized":"2024-02-19T14:57:55.823459+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:14.043959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819612+0000","last_change":"2024-02-19T14:57:53.820777+0000","last_active":"2024-02-19T14:57:55.819612+0000","last_peered":"2024-02-19T14:57:55.819612+0000","last_clean":"2024-02-19T14:57:55.819612+0000","last_became_active":"2024-02-19T14:57:53.820522+0000","last_became_peered":"2024-02-19T14:57:53.820522+0000","last_unstale":"2024-02-19T14:57:55.819612+0000","last_undegraded":"2024-02-19T14:57:55.819612+0000","last_fullsized":"2024-02-19T14:57:55.819612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:06:21.215428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995035+0000","last_change":"2024-02-19T14:57:45.776484+0000","last_active":"2024-02-19T14:57:56.995035+0000","last_peered":"2024-02-19T14:57:56.995035+0000","last_clean":"2024-02-19T14:57:56.995035+0000","last_became_active":"2024-02-19T14:57:45.776095+0000","last_became_peered":"2024-02-19T14:57:45.776095+0000","last_unstale":"2024-02-19T14:57:56.995035+0000","last_undegraded":"2024-02-19T14:57:56.995035+0000","last_fullsized":"2024-02-19T14:57:56.995035+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995639+0000","last_change":"2024-02-19T14:57:44.746247+0000","last_active":"2024-02-19T14:57:56.995639+0000","last_peered":"2024-02-19T14:57:56.995639+0000","last_clean":"2024-02-19T14:57:56.995639+0000","last_became_active":"2024-02-19T14:57:44.745941+0000","last_became_peered":"2024-02-19T14:57:44.745941+0000","last_unstale":"2024-02-19T14:57:56.995639+0000","last_undegraded":"2024-02-19T14:57:56.995639+0000","last_fullsized":"2024-02-19T14:57:56.995639+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823243+0000","last_change":"2024-02-19T14:57:54.818513+0000","last_active":"2024-02-19T14:57:55.823243+0000","last_peered":"2024-02-19T14:57:55.823243+0000","last_clean":"2024-02-19T14:57:55.823243+0000","last_became_active":"2024-02-19T14:57:54.818259+0000","last_became_peered":"2024-02-19T14:57:54.818259+0000","last_unstale":"2024-02-19T14:57:55.823243+0000","last_undegraded":"2024-02-19T14:57:55.823243+0000","last_fullsized":"2024-02-19T14:57:55.823243+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:35.114893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995162+0000","last_change":"2024-02-19T14:57:53.817199+0000","last_active":"2024-02-19T14:57:56.995162+0000","last_peered":"2024-02-19T14:57:56.995162+0000","last_clean":"2024-02-19T14:57:56.995162+0000","last_became_active":"2024-02-19T14:57:53.816930+0000","last_became_peered":"2024-02-19T14:57:53.816930+0000","last_unstale":"2024-02-19T14:57:56.995162+0000","last_undegraded":"2024-02-19T14:57:56.995162+0000","last_fullsized":"2024-02-19T14:57:56.995162+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:56.199868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994948+0000","last_change":"2024-02-19T14:57:45.766067+0000","last_active":"2024-02-19T14:57:56.994948+0000","last_peered":"2024-02-19T14:57:56.994948+0000","last_clean":"2024-02-19T14:57:56.994948+0000","last_became_active":"2024-02-19T14:57:45.765654+0000","last_became_peered":"2024-02-19T14:57:45.765654+0000","last_unstale":"2024-02-19T14:57:56.994948+0000","last_undegraded":"2024-02-19T14:57:56.994948+0000","last_fullsized":"2024-02-19T14:57:56.994948+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823239+0000","last_change":"2024-02-19T14:57:44.751718+0000","last_active":"2024-02-19T14:57:55.823239+0000","last_peered":"2024-02-19T14:57:55.823239+0000","last_clean":"2024-02-19T14:57:55.823239+0000","last_became_active":"2024-02-19T14:57:44.751403+0000","last_became_peered":"2024-02-19T14:57:44.751403+0000","last_unstale":"2024-02-19T14:57:55.823239+0000","last_undegraded":"2024-02-19T14:57:55.823239+0000","last_fullsized":"2024-02-19T14:57:55.823239+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:59.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825997+0000","last_change":"2024-02-19T14:57:54.814429+0000","last_active":"2024-02-19T14:57:55.825997+0000","last_peered":"2024-02-19T14:57:55.825997+0000","last_clean":"2024-02-19T14:57:55.825997+0000","last_became_active":"2024-02-19T14:57:54.814168+0000","last_became_peered":"2024-02-19T14:57:54.814168+0000","last_unstale":"2024-02-19T14:57:55.825997+0000","last_undegraded":"2024-02-19T14:57:55.825997+0000","last_fullsized":"2024-02-19T14:57:55.825997+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:34.967528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"156'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823515+0000","last_change":"2024-02-19T14:57:53.823400+0000","last_active":"2024-02-19T14:57:55.823515+0000","last_peered":"2024-02-19T14:57:55.823515+0000","last_clean":"2024-02-19T14:57:55.823515+0000","last_became_active":"2024-02-19T14:57:53.823152+0000","last_became_peered":"2024-02-19T14:57:53.823152+0000","last_unstale":"2024-02-19T14:57:55.823515+0000","last_undegraded":"2024-02-19T14:57:55.823515+0000","last_fullsized":"2024-02-19T14:57:55.823515+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:59.510070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826684+0000","last_change":"2024-02-19T14:57:45.775690+0000","last_active":"2024-02-19T14:57:55.826684+0000","last_peered":"2024-02-19T14:57:55.826684+0000","last_clean":"2024-02-19T14:57:55.826684+0000","last_became_active":"2024-02-19T14:57:45.775076+0000","last_became_peered":"2024-02-19T14:57:45.775076+0000","last_unstale":"2024-02-19T14:57:55.826684+0000","last_undegraded":"2024-02-19T14:57:55.826684+0000","last_fullsized":"2024-02-19T14:57:55.826684+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:40.949417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826056+0000","last_change":"2024-02-19T14:57:44.751791+0000","last_active":"2024-02-19T14:57:55.826056+0000","last_peered":"2024-02-19T14:57:55.826056+0000","last_clean":"2024-02-19T14:57:55.826056+0000","last_became_active":"2024-02-19T14:57:44.751480+0000","last_became_peered":"2024-02-19T14:57:44.751480+0000","last_unstale":"2024-02-19T14:57:55.826056+0000","last_undegraded":"2024-02-19T14:57:55.826056+0000","last_fullsized":"2024-02-19T14:57:55.826056+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994642+0000","last_change":"2024-02-19T14:57:54.820463+0000","last_active":"2024-02-19T14:57:56.994642+0000","last_peered":"2024-02-19T14:57:56.994642+0000","last_clean":"2024-02-19T14:57:56.994642+0000","last_became_active":"2024-02-19T14:57:54.820202+0000","last_became_peered":"2024-02-19T14:57:54.820202+0000","last_unstale":"2024-02-19T14:57:56.994642+0000","last_undegraded":"2024-02-19T14:57:56.994642+0000","last_fullsized":"2024-02-19T14:57:56.994642+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.602087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822816+0000","last_change":"2024-02-19T14:57:53.822348+0000","last_active":"2024-02-19T14:57:55.822816+0000","last_peered":"2024-02-19T14:57:55.822816+0000","last_clean":"2024-02-19T14:57:55.822816+0000","last_became_active":"2024-02-19T14:57:53.822101+0000","last_became_peered":"2024-02-19T14:57:53.822101+0000","last_unstale":"2024-02-19T14:57:55.822816+0000","last_undegraded":"2024-02-19T14:57:55.822816+0000","last_fullsized":"2024-02-19T14:57:55.822816+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:15:15.938236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995545+0000","last_change":"2024-02-19T14:57:45.772415+0000","last_active":"2024-02-19T14:57:56.995545+0000","last_peered":"2024-02-19T14:57:56.995545+0000","last_clean":"2024-02-19T14:57:56.995545+0000","last_became_active":"2024-02-19T14:57:45.771919+0000","last_became_peered":"2024-02-19T14:57:45.771919+0000","last_unstale":"2024-02-19T14:57:56.995545+0000","last_undegraded":"2024-02-19T14:57:56.995545+0000","last_fullsized":"2024-02-19T14:57:56.995545+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:29.983600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995264+0000","last_change":"2024-02-19T14:57:44.758252+0000","last_active":"2024-02-19T14:57:56.995264+0000","last_peered":"2024-02-19T14:57:56.995264+0000","last_clean":"2024-02-19T14:57:56.995264+0000","last_became_active":"2024-02-19T14:57:44.757979+0000","last_became_peered":"2024-02-19T14:57:44.757979+0000","last_unstale":"2024-02-19T14:57:56.995264+0000","last_undegraded":"2024-02-19T14:57:56.995264+0000","last_fullsized":"2024-02-19T14:57:56.995264+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:28.981816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819601+0000","last_change":"2024-02-19T14:57:54.821187+0000","last_active":"2024-02-19T14:57:55.819601+0000","last_peered":"2024-02-19T14:57:55.819601+0000","last_clean":"2024-02-19T14:57:55.819601+0000","last_became_active":"2024-02-19T14:57:54.820886+0000","last_became_peered":"2024-02-19T14:57:54.820886+0000","last_unstale":"2024-02-19T14:57:55.819601+0000","last_undegraded":"2024-02-19T14:57:55.819601+0000","last_fullsized":"2024-02-19T14:57:55.819601+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:07:55.760657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994793+0000","last_change":"2024-02-19T14:57:53.825683+0000","last_active":"2024-02-19T14:57:56.994793+0000","last_peered":"2024-02-19T14:57:56.994793+0000","last_clean":"2024-02-19T14:57:56.994793+0000","last_became_active":"2024-02-19T14:57:53.825366+0000","last_became_peered":"2024-02-19T14:57:53.825366+0000","last_unstale":"2024-02-19T14:57:56.994793+0000","last_undegraded":"2024-02-19T14:57:56.994793+0000","last_fullsized":"2024-02-19T14:57:56.994793+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:56.803585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825534+0000","last_change":"2024-02-19T14:57:45.772751+0000","last_active":"2024-02-19T14:57:55.825534+0000","last_peered":"2024-02-19T14:57:55.825534+0000","last_clean":"2024-02-19T14:57:55.825534+0000","last_became_active":"2024-02-19T14:57:45.772506+0000","last_became_peered":"2024-02-19T14:57:45.772506+0000","last_unstale":"2024-02-19T14:57:55.825534+0000","last_undegraded":"2024-02-19T14:57:55.825534+0000","last_fullsized":"2024-02-19T14:57:55.825534+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"152'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819925+0000","last_change":"2024-02-19T14:57:44.756795+0000","last_active":"2024-02-19T14:57:55.819925+0000","last_peered":"2024-02-19T14:57:55.819925+0000","last_clean":"2024-02-19T14:57:55.819925+0000","last_became_active":"2024-02-19T14:57:44.756166+0000","last_became_peered":"2024-02-19T14:57:44.756166+0000","last_unstale":"2024-02-19T14:57:55.819925+0000","last_undegraded":"2024-02-19T14:57:55.819925+0000","last_fullsized":"2024-02-19T14:57:55.819925+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:13.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825221+0000","last_change":"2024-02-19T14:57:54.826892+0000","last_active":"2024-02-19T14:57:55.825221+0000","last_peered":"2024-02-19T14:57:55.825221+0000","last_clean":"2024-02-19T14:57:55.825221+0000","last_became_active":"2024-02-19T14:57:54.826378+0000","last_became_peered":"2024-02-19T14:57:54.826378+0000","last_unstale":"2024-02-19T14:57:55.825221+0000","last_undegraded":"2024-02-19T14:57:55.825221+0000","last_fullsized":"2024-02-19T14:57:55.825221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:13:18.218308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826291+0000","last_change":"2024-02-19T14:57:53.819028+0000","last_active":"2024-02-19T14:57:55.826291+0000","last_peered":"2024-02-19T14:57:55.826291+0000","last_clean":"2024-02-19T14:57:55.826291+0000","last_became_active":"2024-02-19T14:57:53.818598+0000","last_became_peered":"2024-02-19T14:57:53.818598+0000","last_unstale":"2024-02-19T14:57:55.826291+0000","last_undegraded":"2024-02-19T14:57:55.826291+0000","last_fullsized":"2024-02-19T14:57:55.826291+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:54:30.909529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823038+0000","last_change":"2024-02-19T14:57:45.774855+0000","last_active":"2024-02-19T14:57:55.823038+0000","last_peered":"2024-02-19T14:57:55.823038+0000","last_clean":"2024-02-19T14:57:55.823038+0000","last_became_active":"2024-02-19T14:57:45.774572+0000","last_became_peered":"2024-02-19T14:57:45.774572+0000","last_unstale":"2024-02-19T14:57:55.823038+0000","last_undegraded":"2024-02-19T14:57:55.823038+0000","last_fullsized":"2024-02-19T14:57:55.823038+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:34.308217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819830+0000","last_change":"2024-02-19T14:57:44.749436+0000","last_active":"2024-02-19T14:57:55.819830+0000","last_peered":"2024-02-19T14:57:55.819830+0000","last_clean":"2024-02-19T14:57:55.819830+0000","last_became_active":"2024-02-19T14:57:44.749153+0000","last_became_peered":"2024-02-19T14:57:44.749153+0000","last_unstale":"2024-02-19T14:57:55.819830+0000","last_undegraded":"2024-02-19T14:57:55.819830+0000","last_fullsized":"2024-02-19T14:57:55.819830+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:36.657057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994434+0000","last_change":"2024-02-19T14:57:54.820331+0000","last_active":"2024-02-19T14:57:56.994434+0000","last_peered":"2024-02-19T14:57:56.994434+0000","last_clean":"2024-02-19T14:57:56.994434+0000","last_became_active":"2024-02-19T14:57:54.820073+0000","last_became_peered":"2024-02-19T14:57:54.820073+0000","last_unstale":"2024-02-19T14:57:56.994434+0000","last_undegraded":"2024-02-19T14:57:56.994434+0000","last_fullsized":"2024-02-19T14:57:56.994434+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:35.195077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994454+0000","last_change":"2024-02-19T14:57:53.827547+0000","last_active":"2024-02-19T14:57:56.994454+0000","last_peered":"2024-02-19T14:57:56.994454+0000","last_clean":"2024-02-19T14:57:56.994454+0000","last_became_active":"2024-02-19T14:57:53.827249+0000","last_became_peered":"2024-02-19T14:57:53.827249+0000","last_unstale":"2024-02-19T14:57:56.994454+0000","last_undegraded":"2024-02-19T14:57:56.994454+0000","last_fullsized":"2024-02-19T14:57:56.994454+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:37:50.645573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825814+0000","last_change":"2024-02-19T14:57:45.772279+0000","last_active":"2024-02-19T14:57:55.825814+0000","last_peered":"2024-02-19T14:57:55.825814+0000","last_clean":"2024-02-19T14:57:55.825814+0000","last_became_active":"2024-02-19T14:57:45.771983+0000","last_became_peered":"2024-02-19T14:57:45.771983+0000","last_unstale":"2024-02-19T14:57:55.825814+0000","last_undegraded":"2024-02-19T14:57:55.825814+0000","last_fullsized":"2024-02-19T14:57:55.825814+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823438+0000","last_change":"2024-02-19T14:57:44.758176+0000","last_active":"2024-02-19T14:57:55.823438+0000","last_peered":"2024-02-19T14:57:55.823438+0000","last_clean":"2024-02-19T14:57:55.823438+0000","last_became_active":"2024-02-19T14:57:44.757912+0000","last_became_peered":"2024-02-19T14:57:44.757912+0000","last_unstale":"2024-02-19T14:57:55.823438+0000","last_undegraded":"2024-02-19T14:57:55.823438+0000","last_fullsized":"2024-02-19T14:57:55.823438+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:26.671302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995931+0000","last_change":"2024-02-19T14:57:54.832738+0000","last_active":"2024-02-19T14:57:56.995931+0000","last_peered":"2024-02-19T14:57:56.995931+0000","last_clean":"2024-02-19T14:57:56.995931+0000","last_became_active":"2024-02-19T14:57:54.832010+0000","last_became_peered":"2024-02-19T14:57:54.832010+0000","last_unstale":"2024-02-19T14:57:56.995931+0000","last_undegraded":"2024-02-19T14:57:56.995931+0000","last_fullsized":"2024-02-19T14:57:56.995931+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:55.179220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995159+0000","last_change":"2024-02-19T14:57:53.821749+0000","last_active":"2024-02-19T14:57:56.995159+0000","last_peered":"2024-02-19T14:57:56.995159+0000","last_clean":"2024-02-19T14:57:56.995159+0000","last_became_active":"2024-02-19T14:57:53.821351+0000","last_became_peered":"2024-02-19T14:57:53.821351+0000","last_unstale":"2024-02-19T14:57:56.995159+0000","last_undegraded":"2024-02-19T14:57:56.995159+0000","last_fullsized":"2024-02-19T14:57:56.995159+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:11:54.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826152+0000","last_change":"2024-02-19T14:57:45.771150+0000","last_active":"2024-02-19T14:57:55.826152+0000","last_peered":"2024-02-19T14:57:55.826152+0000","last_clean":"2024-02-19T14:57:55.826152+0000","last_became_active":"2024-02-19T14:57:45.770740+0000","last_became_peered":"2024-02-19T14:57:45.770740+0000","last_unstale":"2024-02-19T14:57:55.826152+0000","last_undegraded":"2024-02-19T14:57:55.826152+0000","last_fullsized":"2024-02-19T14:57:55.826152+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:57.456060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826504+0000","last_change":"2024-02-19T14:57:44.752321+0000","last_active":"2024-02-19T14:57:55.826504+0000","last_peered":"2024-02-19T14:57:55.826504+0000","last_clean":"2024-02-19T14:57:55.826504+0000","last_became_active":"2024-02-19T14:57:44.752031+0000","last_became_peered":"2024-02-19T14:57:44.752031+0000","last_unstale":"2024-02-19T14:57:55.826504+0000","last_undegraded":"2024-02-19T14:57:55.826504+0000","last_fullsized":"2024-02-19T14:57:55.826504+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:54.123952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826232+0000","last_change":"2024-02-19T14:57:54.826939+0000","last_active":"2024-02-19T14:57:55.826232+0000","last_peered":"2024-02-19T14:57:55.826232+0000","last_clean":"2024-02-19T14:57:55.826232+0000","last_became_active":"2024-02-19T14:57:54.826612+0000","last_became_peered":"2024-02-19T14:57:54.826612+0000","last_unstale":"2024-02-19T14:57:55.826232+0000","last_undegraded":"2024-02-19T14:57:55.826232+0000","last_fullsized":"2024-02-19T14:57:55.826232+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:51.528471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995700+0000","last_change":"2024-02-19T14:57:53.816870+0000","last_active":"2024-02-19T14:57:56.995700+0000","last_peered":"2024-02-19T14:57:56.995700+0000","last_clean":"2024-02-19T14:57:56.995700+0000","last_became_active":"2024-02-19T14:57:53.816516+0000","last_became_peered":"2024-02-19T14:57:53.816516+0000","last_unstale":"2024-02-19T14:57:56.995700+0000","last_undegraded":"2024-02-19T14:57:56.995700+0000","last_fullsized":"2024-02-19T14:57:56.995700+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:41.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825830+0000","last_change":"2024-02-19T14:57:45.784058+0000","last_active":"2024-02-19T14:57:55.825830+0000","last_peered":"2024-02-19T14:57:55.825830+0000","last_clean":"2024-02-19T14:57:55.825830+0000","last_became_active":"2024-02-19T14:57:45.783772+0000","last_became_peered":"2024-02-19T14:57:45.783772+0000","last_unstale":"2024-02-19T14:57:55.825830+0000","last_undegraded":"2024-02-19T14:57:55.825830+0000","last_fullsized":"2024-02-19T14:57:55.825830+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:34.823875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819175+0000","last_change":"2024-02-19T14:57:44.743222+0000","last_active":"2024-02-19T14:57:55.819175+0000","last_peered":"2024-02-19T14:57:55.819175+0000","last_clean":"2024-02-19T14:57:55.819175+0000","last_became_active":"2024-02-19T14:57:44.742946+0000","last_became_peered":"2024-02-19T14:57:44.742946+0000","last_unstale":"2024-02-19T14:57:55.819175+0000","last_undegraded":"2024-02-19T14:57:55.819175+0000","last_fullsized":"2024-02-19T14:57:55.819175+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:22.509070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823232+0000","last_change":"2024-02-19T14:57:54.827418+0000","last_active":"2024-02-19T14:57:55.823232+0000","last_peered":"2024-02-19T14:57:55.823232+0000","last_clean":"2024-02-19T14:57:55.823232+0000","last_became_active":"2024-02-19T14:57:54.827167+0000","last_became_peered":"2024-02-19T14:57:54.827167+0000","last_unstale":"2024-02-19T14:57:55.823232+0000","last_undegraded":"2024-02-19T14:57:55.823232+0000","last_fullsized":"2024-02-19T14:57:55.823232+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:32.315419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995603+0000","last_change":"2024-02-19T14:57:53.817116+0000","last_active":"2024-02-19T14:57:56.995603+0000","last_peered":"2024-02-19T14:57:56.995603+0000","last_clean":"2024-02-19T14:57:56.995603+0000","last_became_active":"2024-02-19T14:57:53.816835+0000","last_became_peered":"2024-02-19T14:57:53.816835+0000","last_unstale":"2024-02-19T14:57:56.995603+0000","last_undegraded":"2024-02-19T14:57:56.995603+0000","last_fullsized":"2024-02-19T14:57:56.995603+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:42.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819737+0000","last_change":"2024-02-19T14:57:45.783349+0000","last_active":"2024-02-19T14:57:55.819737+0000","last_peered":"2024-02-19T14:57:55.819737+0000","last_clean":"2024-02-19T14:57:55.819737+0000","last_became_active":"2024-02-19T14:57:45.783101+0000","last_became_peered":"2024-02-19T14:57:45.783101+0000","last_unstale":"2024-02-19T14:57:55.819737+0000","last_undegraded":"2024-02-19T14:57:55.819737+0000","last_fullsized":"2024-02-19T14:57:55.819737+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:13:09.154323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995074+0000","last_change":"2024-02-19T14:57:44.747418+0000","last_active":"2024-02-19T14:57:56.995074+0000","last_peered":"2024-02-19T14:57:56.995074+0000","last_clean":"2024-02-19T14:57:56.995074+0000","last_became_active":"2024-02-19T14:57:44.746915+0000","last_became_peered":"2024-02-19T14:57:44.746915+0000","last_unstale":"2024-02-19T14:57:56.995074+0000","last_undegraded":"2024-02-19T14:57:56.995074+0000","last_fullsized":"2024-02-19T14:57:56.995074+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819537+0000","last_change":"2024-02-19T14:57:54.829645+0000","last_active":"2024-02-19T14:57:55.819537+0000","last_peered":"2024-02-19T14:57:55.819537+0000","last_clean":"2024-02-19T14:57:55.819537+0000","last_became_active":"2024-02-19T14:57:54.828869+0000","last_became_peered":"2024-02-19T14:57:54.828869+0000","last_unstale":"2024-02-19T14:57:55.819537+0000","last_undegraded":"2024-02-19T14:57:55.819537+0000","last_fullsized":"2024-02-19T14:57:55.819537+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:06.710733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995024+0000","last_change":"2024-02-19T14:57:53.820424+0000","last_active":"2024-02-19T14:57:56.995024+0000","last_peered":"2024-02-19T14:57:56.995024+0000","last_clean":"2024-02-19T14:57:56.995024+0000","last_became_active":"2024-02-19T14:57:53.820147+0000","last_became_peered":"2024-02-19T14:57:53.820147+0000","last_unstale":"2024-02-19T14:57:56.995024+0000","last_undegraded":"2024-02-19T14:57:56.995024+0000","last_fullsized":"2024-02-19T14:57:56.995024+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:10.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823505+0000","last_change":"2024-02-19T14:57:45.774106+0000","last_active":"2024-02-19T14:57:55.823505+0000","last_peered":"2024-02-19T14:57:55.823505+0000","last_clean":"2024-02-19T14:57:55.823505+0000","last_became_active":"2024-02-19T14:57:45.773847+0000","last_became_peered":"2024-02-19T14:57:45.773847+0000","last_unstale":"2024-02-19T14:57:55.823505+0000","last_undegraded":"2024-02-19T14:57:55.823505+0000","last_fullsized":"2024-02-19T14:57:55.823505+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:22.673960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994267+0000","last_change":"2024-02-19T14:57:44.752893+0000","last_active":"2024-02-19T14:57:56.994267+0000","last_peered":"2024-02-19T14:57:56.994267+0000","last_clean":"2024-02-19T14:57:56.994267+0000","last_became_active":"2024-02-19T14:57:44.752617+0000","last_became_peered":"2024-02-19T14:57:44.752617+0000","last_unstale":"2024-02-19T14:57:56.994267+0000","last_undegraded":"2024-02-19T14:57:56.994267+0000","last_fullsized":"2024-02-19T14:57:56.994267+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:45.368123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995022+0000","last_change":"2024-02-19T14:57:54.828418+0000","last_active":"2024-02-19T14:57:56.995022+0000","last_peered":"2024-02-19T14:57:56.995022+0000","last_clean":"2024-02-19T14:57:56.995022+0000","last_became_active":"2024-02-19T14:57:54.828147+0000","last_became_peered":"2024-02-19T14:57:54.828147+0000","last_unstale":"2024-02-19T14:57:56.995022+0000","last_undegraded":"2024-02-19T14:57:56.995022+0000","last_fullsized":"2024-02-19T14:57:56.995022+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:45.460188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994274+0000","last_change":"2024-02-19T14:57:53.823527+0000","last_active":"2024-02-19T14:57:56.994274+0000","last_peered":"2024-02-19T14:57:56.994274+0000","last_clean":"2024-02-19T14:57:56.994274+0000","last_became_active":"2024-02-19T14:57:53.823257+0000","last_became_peered":"2024-02-19T14:57:53.823257+0000","last_unstale":"2024-02-19T14:57:56.994274+0000","last_undegraded":"2024-02-19T14:57:56.994274+0000","last_fullsized":"2024-02-19T14:57:56.994274+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:39.435634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825144+0000","last_change":"2024-02-19T14:57:45.777456+0000","last_active":"2024-02-19T14:57:55.825144+0000","last_peered":"2024-02-19T14:57:55.825144+0000","last_clean":"2024-02-19T14:57:55.825144+0000","last_became_active":"2024-02-19T14:57:45.776998+0000","last_became_peered":"2024-02-19T14:57:45.776998+0000","last_unstale":"2024-02-19T14:57:55.825144+0000","last_undegraded":"2024-02-19T14:57:55.825144+0000","last_fullsized":"2024-02-19T14:57:55.825144+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:14.577140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819828+0000","last_change":"2024-02-19T14:57:44.757019+0000","last_active":"2024-02-19T14:57:55.819828+0000","last_peered":"2024-02-19T14:57:55.819828+0000","last_clean":"2024-02-19T14:57:55.819828+0000","last_became_active":"2024-02-19T14:57:44.756538+0000","last_became_peered":"2024-02-19T14:57:44.756538+0000","last_unstale":"2024-02-19T14:57:55.819828+0000","last_undegraded":"2024-02-19T14:57:55.819828+0000","last_fullsized":"2024-02-19T14:57:55.819828+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:33.822090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":426,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826138+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:55.826138+0000","last_peered":"2024-02-19T14:57:55.826138+0000","last_clean":"2024-02-19T14:57:55.826138+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:55.826138+0000","last_undegraded":"2024-02-19T14:57:55.826138+0000","last_fullsized":"2024-02-19T14:57:55.826138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33676,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338164,"statfs":{"total":96636764160,"available":96602279936,"internally_reserved":0,"allocated":2555904,"data_stored":1548845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":0.61599999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.504,"15min":0.496},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60699999999999998,"5min":0.75900000000000001,"15min":0.79500000000000004},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32500000000000001,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"min":{"1min":0.35299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64000000000000001,"5min":0.73999999999999999,"15min":0.74099999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.501},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70799999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.627}]},{"osd":3,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67600000000000005,"5min":0.81399999999999995,"15min":0.81499999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73499999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.81100000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48199999999999998,"15min":0.496},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.497,"15min":0.53300000000000003},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.75},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51900000000000002,"15min":0.54100000000000004},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55900000000000005,"15min":0.59299999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.88500000000000001},"last":0.83899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":958547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.48699999999999999,"15min":0.46800000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60999999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.49199999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":0.72899999999999998,"15min":0.81999999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69899999999999995,"5min":0.81000000000000005,"15min":0.88100000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.40999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70099999999999996,"5min":0.78100000000000003,"15min":0.90500000000000003},"last":0.502}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.84799999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.504,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.39200000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.52800000000000002},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.76600000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.60199999999999998},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71199999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.58199999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47699999999999998,"15min":0.53200000000000003},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":0.753,"15min":0.77400000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.53200000000000003,"15min":0.55200000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73499999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.52500000000000002,"15min":0.57299999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.69799999999999995,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53000000000000003,"15min":0.56200000000000006},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71499999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1986560,"data_stored":960235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.501}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.432}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.375},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.53200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1994752,"data_stored":960887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.38}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.434},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.505}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":960853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.40999999999999998,"5min":3.694,"15min":3.694},"last":0.372},{"interface":"front","average":{"1min":0.312,"5min":0.36899999999999999,"15min":0.32500000000000001},"min":{"1min":0.21299999999999999,"5min":0.192,"15min":0.189},"max":{"1min":0.41499999999999998,"5min":4.835,"15min":4.835},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32300000000000001,"15min":0.314},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.23100000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52600000000000002,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.434},{"interface":"front","average":{"1min":0.378,"5min":0.40500000000000003,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.33800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70799999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79900000000000004,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81799999999999995,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.45400000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60399999999999998,"15min":0.64600000000000002},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72399999999999998,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58999999999999997,"15min":0.63800000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84799999999999998,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.56000000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":959059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.20699999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.33100000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.312},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.375}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.69899999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.621}]}]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1974272,"data_stored":960089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.39700000000000002,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.33300000000000002},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.40100000000000002,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30599999999999999,"15min":0.309},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42999999999999999,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.36099999999999999},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.247}]},{"osd":3,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.316},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.38900000000000001},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.23100000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54000000000000004,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":1.0009999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54800000000000004,"15min":0.55800000000000005},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85299999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":0.80000000000000004,"5min":0.95499999999999996,"15min":1.026},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55100000000000005,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82399999999999995,"5min":0.996,"15min":0.996},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79500000000000004,"5min":1.036,"15min":1.036},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.56499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33704,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338136,"statfs":{"total":96636764160,"available":96602251264,"internally_reserved":0,"allocated":2580480,"data_stored":1549677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.31,"15min":0.30199999999999999},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.315,"15min":0.30499999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.38100000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.33800000000000002},"min":{"1min":0.24299999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.37,"15min":0.371},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54000000000000004,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.54000000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.59699999999999998},"min":{"1min":0.38500000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83699999999999997,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79800000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79000000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.81100000000000005,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:59.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:57:59.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.160+0000 7f1d2c1e9640 1 Processor -- start 2024-02-19T14:57:59.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.160+0000 7f1d2c1e9640 1 -- start start 2024-02-19T14:57:59.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2c1e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1d240add70 0x7f1d240c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2c1e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d24070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2c1e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d240ae670 con 0x7f1d240c2f60 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2c1e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d240716b0 con 0x7f1d240add70 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2c1e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d240c33d0 con 0x7f1d240705b0 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1d240add70 0x7f1d240c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1d240add70 0x7f1d240c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41494/0 (socket says 172.21.15.78:41494) 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 -- 172.21.15.78:0/74397109 learned_addr learned my addr 172.21.15.78:0/74397109 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2a786640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d24070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 -- 172.21.15.78:0/74397109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 msgr2=0x7f1d24070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d24070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 -- 172.21.15.78:0/74397109 >> v1:172.21.15.150:6789/0 conn(0x7f1d240c2f60 legacy=0x7f1d2406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d29f85640 1 -- 172.21.15.78:0/74397109 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d24153780 con 0x7f1d240add70 2024-02-19T14:57:59.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.161+0000 7f1d2a786640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d24070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:59.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.162+0000 7f1d29f85640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240add70 0x7f1d240c4980 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009870 tx=0x7f1d2002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c22c2492b8cc90 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d28f83640 1 -- 172.21.15.78:0/74397109 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200317c0 con 0x7f1d240add70 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d28f83640 1 -- 172.21.15.78:0/74397109 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d20031920 con 0x7f1d240add70 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d28f83640 1 -- 172.21.15.78:0/74397109 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d20031cb0 con 0x7f1d240add70 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/74397109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240add70 msgr2=0x7f1d240c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240add70 0x7f1d240c4980 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009870 tx=0x7f1d2002b860 comp rx=0 tx=0).stop 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/74397109 shutdown_connections 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d24070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/74397109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240add70 0x7f1d240c4980 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/74397109 >> 172.21.15.78:0/74397109 conn(0x7f1d240c85e0 msgr2=0x7f1d240c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/74397109 shutdown_connections 2024-02-19T14:57:59.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.163+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/74397109 wait complete. 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.164+0000 7f1d2c1e9640 1 Processor -- start 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 -- start start 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d241dc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d240add70 0x7f1d241ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 0x7f1d241ec040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241dfae0 con 0x7f1d240c2f60 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241ec880 con 0x7f1d240add70 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2c1e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241dfd40 con 0x7f1d240705b0 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2a786640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 0x7f1d241ec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2a786640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 0x7f1d241ec040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41500/0 (socket says 172.21.15.78:41500) 2024-02-19T14:57:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.165+0000 7f1d2a786640 1 -- 172.21.15.78:0/2507933537 learned_addr learned my addr 172.21.15.78:0/2507933537 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d29784640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d240add70 0x7f1d241ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2a786640 1 -- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 msgr2=0x7f1d241dc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d29f85640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d241dc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2a786640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d241dc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2a786640 1 -- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d240add70 msgr2=0x7f1d241ebb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2a786640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d240add70 0x7f1d241ebb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2a786640 1 -- 172.21.15.78:0/2507933537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d200094d0 con 0x7f1d240c2f60 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d29784640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d240add70 0x7f1d241ebb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d29f85640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d241dc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2a786640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 0x7f1d241ec040 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c00cd50 tx=0x7f1d1c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c00f680 con 0x7f1d240c2f60 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1c00f800 con 0x7f1d240c2f60 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d241e0020 con 0x7f1d240c2f60 2024-02-19T14:57:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c00fbb0 con 0x7f1d240c2f60 2024-02-19T14:57:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.166+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d241ec580 con 0x7f1d240c2f60 2024-02-19T14:57:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.167+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d240676a0 con 0x7f1d240c2f60 2024-02-19T14:57:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.168+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1d1c00b040 con 0x7f1d240c2f60 2024-02-19T14:57:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.168+0000 7f1d1a7fc640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d0806b810 0x7f1d0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.168+0000 7f1d29f85640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d0806b810 0x7f1d0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.168+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f1d1c08b060 con 0x7f1d240c2f60 2024-02-19T14:57:59.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.169+0000 7f1d29f85640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d0806b810 0x7f1d0806dcd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1d20036000 tx=0x7f1d2003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.173+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1d1c055630 con 0x7f1d240c2f60 2024-02-19T14:57:59.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.423+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d240ad7e0 con 0x7f1d0806b810 2024-02-19T14:57:59.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.435+0000 7f1d1a7fc640 1 -- 172.21.15.78:0/2507933537 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528975 (secure 0 0 0) 0x7f1d240ad7e0 con 0x7f1d0806b810 2024-02-19T14:57:59.444 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:59.450 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:57:59.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d0806b810 msgr2=0x7f1d0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d0806b810 0x7f1d0806dcd0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1d20036000 tx=0x7f1d2003d040 comp rx=0 tx=0).stop 2024-02-19T14:57:59.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 msgr2=0x7f1d241ec040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 0x7f1d241ec040 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c00cd50 tx=0x7f1d1c00a650 comp rx=0 tx=0).stop 2024-02-19T14:57:59.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 shutdown_connections 2024-02-19T14:57:59.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1d0806b810 0x7f1d0806dcd0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d240c2f60 0x7f1d241ec040 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d240add70 0x7f1d241ebb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 --2- 172.21.15.78:0/2507933537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d240705b0 0x7f1d241dc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 >> 172.21.15.78:0/2507933537 conn(0x7f1d240c85e0 msgr2=0x7f1d240ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 shutdown_connections 2024-02-19T14:57:59.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.441+0000 7f1d2c1e9640 1 -- 172.21.15.78:0/2507933537 wait complete. 2024-02-19T14:57:59.454 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-02-19T14:57:58.129155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":359,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266060,"kb_used_data":16616,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708660,"statfs":{"total":773094113280,"available":772821667840,"internally_reserved":0,"allocated":17014784,"data_stored":8858192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.331425"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995097+0000","last_change":"2024-02-19T14:57:54.822787+0000","last_active":"2024-02-19T14:57:56.995097+0000","last_peered":"2024-02-19T14:57:56.995097+0000","last_clean":"2024-02-19T14:57:56.995097+0000","last_became_active":"2024-02-19T14:57:54.822487+0000","last_became_peered":"2024-02-19T14:57:54.822487+0000","last_unstale":"2024-02-19T14:57:56.995097+0000","last_undegraded":"2024-02-19T14:57:56.995097+0000","last_fullsized":"2024-02-19T14:57:56.995097+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:05.975465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826551+0000","last_change":"2024-02-19T14:57:45.778368+0000","last_active":"2024-02-19T14:57:55.826551+0000","last_peered":"2024-02-19T14:57:55.826551+0000","last_clean":"2024-02-19T14:57:55.826551+0000","last_became_active":"2024-02-19T14:57:45.777197+0000","last_became_peered":"2024-02-19T14:57:45.777197+0000","last_unstale":"2024-02-19T14:57:55.826551+0000","last_undegraded":"2024-02-19T14:57:55.826551+0000","last_fullsized":"2024-02-19T14:57:55.826551+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:12.001914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994756+0000","last_change":"2024-02-19T14:57:54.821802+0000","last_active":"2024-02-19T14:57:56.994756+0000","last_peered":"2024-02-19T14:57:56.994756+0000","last_clean":"2024-02-19T14:57:56.994756+0000","last_became_active":"2024-02-19T14:57:54.821533+0000","last_became_peered":"2024-02-19T14:57:54.821533+0000","last_unstale":"2024-02-19T14:57:56.994756+0000","last_undegraded":"2024-02-19T14:57:56.994756+0000","last_fullsized":"2024-02-19T14:57:56.994756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:37.441462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826317+0000","last_change":"2024-02-19T14:57:45.774781+0000","last_active":"2024-02-19T14:57:55.826317+0000","last_peered":"2024-02-19T14:57:55.826317+0000","last_clean":"2024-02-19T14:57:55.826317+0000","last_became_active":"2024-02-19T14:57:45.774498+0000","last_became_peered":"2024-02-19T14:57:45.774498+0000","last_unstale":"2024-02-19T14:57:55.826317+0000","last_undegraded":"2024-02-19T14:57:55.826317+0000","last_fullsized":"2024-02-19T14:57:55.826317+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:33.646592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825686+0000","last_change":"2024-02-19T14:57:54.829774+0000","last_active":"2024-02-19T14:57:55.825686+0000","last_peered":"2024-02-19T14:57:55.825686+0000","last_clean":"2024-02-19T14:57:55.825686+0000","last_became_active":"2024-02-19T14:57:54.829521+0000","last_became_peered":"2024-02-19T14:57:54.829521+0000","last_unstale":"2024-02-19T14:57:55.825686+0000","last_undegraded":"2024-02-19T14:57:55.825686+0000","last_fullsized":"2024-02-19T14:57:55.825686+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:24:13.797923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825662+0000","last_change":"2024-02-19T14:57:45.782283+0000","last_active":"2024-02-19T14:57:55.825662+0000","last_peered":"2024-02-19T14:57:55.825662+0000","last_clean":"2024-02-19T14:57:55.825662+0000","last_became_active":"2024-02-19T14:57:45.779080+0000","last_became_peered":"2024-02-19T14:57:45.779080+0000","last_unstale":"2024-02-19T14:57:55.825662+0000","last_undegraded":"2024-02-19T14:57:55.825662+0000","last_fullsized":"2024-02-19T14:57:55.825662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:26.412051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825706+0000","last_change":"2024-02-19T14:57:54.818112+0000","last_active":"2024-02-19T14:57:55.825706+0000","last_peered":"2024-02-19T14:57:55.825706+0000","last_clean":"2024-02-19T14:57:55.825706+0000","last_became_active":"2024-02-19T14:57:54.817836+0000","last_became_peered":"2024-02-19T14:57:54.817836+0000","last_unstale":"2024-02-19T14:57:55.825706+0000","last_undegraded":"2024-02-19T14:57:55.825706+0000","last_fullsized":"2024-02-19T14:57:55.825706+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:30:56.427520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826049+0000","last_change":"2024-02-19T14:57:45.778032+0000","last_active":"2024-02-19T14:57:55.826049+0000","last_peered":"2024-02-19T14:57:55.826049+0000","last_clean":"2024-02-19T14:57:55.826049+0000","last_became_active":"2024-02-19T14:57:45.777184+0000","last_became_peered":"2024-02-19T14:57:45.777184+0000","last_unstale":"2024-02-19T14:57:55.826049+0000","last_undegraded":"2024-02-19T14:57:55.826049+0000","last_fullsized":"2024-02-19T14:57:55.826049+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:36.503471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823128+0000","last_change":"2024-02-19T14:57:54.822500+0000","last_active":"2024-02-19T14:57:55.823128+0000","last_peered":"2024-02-19T14:57:55.823128+0000","last_clean":"2024-02-19T14:57:55.823128+0000","last_became_active":"2024-02-19T14:57:54.822233+0000","last_became_peered":"2024-02-19T14:57:54.822233+0000","last_unstale":"2024-02-19T14:57:55.823128+0000","last_undegraded":"2024-02-19T14:57:55.823128+0000","last_fullsized":"2024-02-19T14:57:55.823128+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:41:45.567455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994579+0000","last_change":"2024-02-19T14:57:45.777557+0000","last_active":"2024-02-19T14:57:56.994579+0000","last_peered":"2024-02-19T14:57:56.994579+0000","last_clean":"2024-02-19T14:57:56.994579+0000","last_became_active":"2024-02-19T14:57:45.777296+0000","last_became_peered":"2024-02-19T14:57:45.777296+0000","last_unstale":"2024-02-19T14:57:56.994579+0000","last_undegraded":"2024-02-19T14:57:56.994579+0000","last_fullsized":"2024-02-19T14:57:56.994579+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826224+0000","last_change":"2024-02-19T14:57:54.829408+0000","last_active":"2024-02-19T14:57:55.826224+0000","last_peered":"2024-02-19T14:57:55.826224+0000","last_clean":"2024-02-19T14:57:55.826224+0000","last_became_active":"2024-02-19T14:57:54.829129+0000","last_became_peered":"2024-02-19T14:57:54.829129+0000","last_unstale":"2024-02-19T14:57:55.826224+0000","last_undegraded":"2024-02-19T14:57:55.826224+0000","last_fullsized":"2024-02-19T14:57:55.826224+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:24:13.797923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825055+0000","last_change":"2024-02-19T14:57:45.781001+0000","last_active":"2024-02-19T14:57:55.825055+0000","last_peered":"2024-02-19T14:57:55.825055+0000","last_clean":"2024-02-19T14:57:55.825055+0000","last_became_active":"2024-02-19T14:57:45.780476+0000","last_became_peered":"2024-02-19T14:57:45.780476+0000","last_unstale":"2024-02-19T14:57:55.825055+0000","last_undegraded":"2024-02-19T14:57:55.825055+0000","last_fullsized":"2024-02-19T14:57:55.825055+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819210+0000","last_change":"2024-02-19T14:57:54.828759+0000","last_active":"2024-02-19T14:57:55.819210+0000","last_peered":"2024-02-19T14:57:55.819210+0000","last_clean":"2024-02-19T14:57:55.819210+0000","last_became_active":"2024-02-19T14:57:54.828501+0000","last_became_peered":"2024-02-19T14:57:54.828501+0000","last_unstale":"2024-02-19T14:57:55.819210+0000","last_undegraded":"2024-02-19T14:57:55.819210+0000","last_fullsized":"2024-02-19T14:57:55.819210+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:04.080418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825438+0000","last_change":"2024-02-19T14:57:45.783602+0000","last_active":"2024-02-19T14:57:55.825438+0000","last_peered":"2024-02-19T14:57:55.825438+0000","last_clean":"2024-02-19T14:57:55.825438+0000","last_became_active":"2024-02-19T14:57:45.783269+0000","last_became_peered":"2024-02-19T14:57:45.783269+0000","last_unstale":"2024-02-19T14:57:55.825438+0000","last_undegraded":"2024-02-19T14:57:55.825438+0000","last_fullsized":"2024-02-19T14:57:55.825438+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819026+0000","last_change":"2024-02-19T14:57:54.825478+0000","last_active":"2024-02-19T14:57:55.819026+0000","last_peered":"2024-02-19T14:57:55.819026+0000","last_clean":"2024-02-19T14:57:55.819026+0000","last_became_active":"2024-02-19T14:57:54.825228+0000","last_became_peered":"2024-02-19T14:57:54.825228+0000","last_unstale":"2024-02-19T14:57:55.819026+0000","last_undegraded":"2024-02-19T14:57:55.819026+0000","last_fullsized":"2024-02-19T14:57:55.819026+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:47.934567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825175+0000","last_change":"2024-02-19T14:57:45.763954+0000","last_active":"2024-02-19T14:57:55.825175+0000","last_peered":"2024-02-19T14:57:55.825175+0000","last_clean":"2024-02-19T14:57:55.825175+0000","last_became_active":"2024-02-19T14:57:45.763656+0000","last_became_peered":"2024-02-19T14:57:45.763656+0000","last_unstale":"2024-02-19T14:57:55.825175+0000","last_undegraded":"2024-02-19T14:57:55.825175+0000","last_fullsized":"2024-02-19T14:57:55.825175+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995200+0000","last_change":"2024-02-19T14:57:54.819808+0000","last_active":"2024-02-19T14:57:56.995200+0000","last_peered":"2024-02-19T14:57:56.995200+0000","last_clean":"2024-02-19T14:57:56.995200+0000","last_became_active":"2024-02-19T14:57:54.819516+0000","last_became_peered":"2024-02-19T14:57:54.819516+0000","last_unstale":"2024-02-19T14:57:56.995200+0000","last_undegraded":"2024-02-19T14:57:56.995200+0000","last_fullsized":"2024-02-19T14:57:56.995200+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:10.691659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826108+0000","last_change":"2024-02-19T14:57:45.772149+0000","last_active":"2024-02-19T14:57:55.826108+0000","last_peered":"2024-02-19T14:57:55.826108+0000","last_clean":"2024-02-19T14:57:55.826108+0000","last_became_active":"2024-02-19T14:57:45.771811+0000","last_became_peered":"2024-02-19T14:57:45.771811+0000","last_unstale":"2024-02-19T14:57:55.826108+0000","last_undegraded":"2024-02-19T14:57:55.826108+0000","last_fullsized":"2024-02-19T14:57:55.826108+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995027+0000","last_change":"2024-02-19T14:57:54.823091+0000","last_active":"2024-02-19T14:57:56.995027+0000","last_peered":"2024-02-19T14:57:56.995027+0000","last_clean":"2024-02-19T14:57:56.995027+0000","last_became_active":"2024-02-19T14:57:54.822833+0000","last_became_peered":"2024-02-19T14:57:54.822833+0000","last_unstale":"2024-02-19T14:57:56.995027+0000","last_undegraded":"2024-02-19T14:57:56.995027+0000","last_fullsized":"2024-02-19T14:57:56.995027+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:24:13.797923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825893+0000","last_change":"2024-02-19T14:57:45.769091+0000","last_active":"2024-02-19T14:57:55.825893+0000","last_peered":"2024-02-19T14:57:55.825893+0000","last_clean":"2024-02-19T14:57:55.825893+0000","last_became_active":"2024-02-19T14:57:45.768819+0000","last_became_peered":"2024-02-19T14:57:45.768819+0000","last_unstale":"2024-02-19T14:57:55.825893+0000","last_undegraded":"2024-02-19T14:57:55.825893+0000","last_fullsized":"2024-02-19T14:57:55.825893+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825839+0000","last_change":"2024-02-19T14:57:54.819549+0000","last_active":"2024-02-19T14:57:55.825839+0000","last_peered":"2024-02-19T14:57:55.825839+0000","last_clean":"2024-02-19T14:57:55.825839+0000","last_became_active":"2024-02-19T14:57:54.819297+0000","last_became_peered":"2024-02-19T14:57:54.819297+0000","last_unstale":"2024-02-19T14:57:55.825839+0000","last_undegraded":"2024-02-19T14:57:55.825839+0000","last_fullsized":"2024-02-19T14:57:55.825839+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:04.023224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995989+0000","last_change":"2024-02-19T14:57:45.771342+0000","last_active":"2024-02-19T14:57:56.995989+0000","last_peered":"2024-02-19T14:57:56.995989+0000","last_clean":"2024-02-19T14:57:56.995989+0000","last_became_active":"2024-02-19T14:57:45.771047+0000","last_became_peered":"2024-02-19T14:57:45.771047+0000","last_unstale":"2024-02-19T14:57:56.995989+0000","last_undegraded":"2024-02-19T14:57:56.995989+0000","last_fullsized":"2024-02-19T14:57:56.995989+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819576+0000","last_change":"2024-02-19T14:57:54.827482+0000","last_active":"2024-02-19T14:57:55.819576+0000","last_peered":"2024-02-19T14:57:55.819576+0000","last_clean":"2024-02-19T14:57:55.819576+0000","last_became_active":"2024-02-19T14:57:54.826130+0000","last_became_peered":"2024-02-19T14:57:54.826130+0000","last_unstale":"2024-02-19T14:57:55.819576+0000","last_undegraded":"2024-02-19T14:57:55.819576+0000","last_fullsized":"2024-02-19T14:57:55.819576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:16.125045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825873+0000","last_change":"2024-02-19T14:57:45.783578+0000","last_active":"2024-02-19T14:57:55.825873+0000","last_peered":"2024-02-19T14:57:55.825873+0000","last_clean":"2024-02-19T14:57:55.825873+0000","last_became_active":"2024-02-19T14:57:45.783301+0000","last_became_peered":"2024-02-19T14:57:45.783301+0000","last_unstale":"2024-02-19T14:57:55.825873+0000","last_undegraded":"2024-02-19T14:57:55.825873+0000","last_fullsized":"2024-02-19T14:57:55.825873+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:14.016549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825788+0000","last_change":"2024-02-19T14:57:54.822005+0000","last_active":"2024-02-19T14:57:55.825788+0000","last_peered":"2024-02-19T14:57:55.825788+0000","last_clean":"2024-02-19T14:57:55.825788+0000","last_became_active":"2024-02-19T14:57:54.821743+0000","last_became_peered":"2024-02-19T14:57:54.821743+0000","last_unstale":"2024-02-19T14:57:55.825788+0000","last_undegraded":"2024-02-19T14:57:55.825788+0000","last_fullsized":"2024-02-19T14:57:55.825788+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:05.123628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819637+0000","last_change":"2024-02-19T14:57:45.782352+0000","last_active":"2024-02-19T14:57:55.819637+0000","last_peered":"2024-02-19T14:57:55.819637+0000","last_clean":"2024-02-19T14:57:55.819637+0000","last_became_active":"2024-02-19T14:57:45.781489+0000","last_became_peered":"2024-02-19T14:57:45.781489+0000","last_unstale":"2024-02-19T14:57:55.819637+0000","last_undegraded":"2024-02-19T14:57:55.819637+0000","last_fullsized":"2024-02-19T14:57:55.819637+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:33.948498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818824+0000","last_change":"2024-02-19T14:57:54.828531+0000","last_active":"2024-02-19T14:57:55.818824+0000","last_peered":"2024-02-19T14:57:55.818824+0000","last_clean":"2024-02-19T14:57:55.818824+0000","last_became_active":"2024-02-19T14:57:54.828273+0000","last_became_peered":"2024-02-19T14:57:54.828273+0000","last_unstale":"2024-02-19T14:57:55.818824+0000","last_undegraded":"2024-02-19T14:57:55.818824+0000","last_fullsized":"2024-02-19T14:57:55.818824+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:23.513464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994648+0000","last_change":"2024-02-19T14:57:45.776389+0000","last_active":"2024-02-19T14:57:56.994648+0000","last_peered":"2024-02-19T14:57:56.994648+0000","last_clean":"2024-02-19T14:57:56.994648+0000","last_became_active":"2024-02-19T14:57:45.776133+0000","last_became_peered":"2024-02-19T14:57:45.776133+0000","last_unstale":"2024-02-19T14:57:56.994648+0000","last_undegraded":"2024-02-19T14:57:56.994648+0000","last_fullsized":"2024-02-19T14:57:56.994648+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826215+0000","last_change":"2024-02-19T14:57:54.824613+0000","last_active":"2024-02-19T14:57:55.826215+0000","last_peered":"2024-02-19T14:57:55.826215+0000","last_clean":"2024-02-19T14:57:55.826215+0000","last_became_active":"2024-02-19T14:57:54.824319+0000","last_became_peered":"2024-02-19T14:57:54.824319+0000","last_unstale":"2024-02-19T14:57:55.826215+0000","last_undegraded":"2024-02-19T14:57:55.826215+0000","last_fullsized":"2024-02-19T14:57:55.826215+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:54.699190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994675+0000","last_change":"2024-02-19T14:57:45.774486+0000","last_active":"2024-02-19T14:57:56.994675+0000","last_peered":"2024-02-19T14:57:56.994675+0000","last_clean":"2024-02-19T14:57:56.994675+0000","last_became_active":"2024-02-19T14:57:45.773909+0000","last_became_peered":"2024-02-19T14:57:45.773909+0000","last_unstale":"2024-02-19T14:57:56.994675+0000","last_undegraded":"2024-02-19T14:57:56.994675+0000","last_fullsized":"2024-02-19T14:57:56.994675+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:24.066471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822963+0000","last_change":"2024-02-19T14:57:54.824675+0000","last_active":"2024-02-19T14:57:55.822963+0000","last_peered":"2024-02-19T14:57:55.822963+0000","last_clean":"2024-02-19T14:57:55.822963+0000","last_became_active":"2024-02-19T14:57:54.824428+0000","last_became_peered":"2024-02-19T14:57:54.824428+0000","last_unstale":"2024-02-19T14:57:55.822963+0000","last_undegraded":"2024-02-19T14:57:55.822963+0000","last_fullsized":"2024-02-19T14:57:55.822963+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:59:35.476036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824841+0000","last_change":"2024-02-19T14:57:45.776654+0000","last_active":"2024-02-19T14:57:55.824841+0000","last_peered":"2024-02-19T14:57:55.824841+0000","last_clean":"2024-02-19T14:57:55.824841+0000","last_became_active":"2024-02-19T14:57:45.776192+0000","last_became_peered":"2024-02-19T14:57:45.776192+0000","last_unstale":"2024-02-19T14:57:55.824841+0000","last_undegraded":"2024-02-19T14:57:55.824841+0000","last_fullsized":"2024-02-19T14:57:55.824841+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:14:34.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826326+0000","last_change":"2024-02-19T14:57:54.821884+0000","last_active":"2024-02-19T14:57:55.826326+0000","last_peered":"2024-02-19T14:57:55.826326+0000","last_clean":"2024-02-19T14:57:55.826326+0000","last_became_active":"2024-02-19T14:57:54.821331+0000","last_became_peered":"2024-02-19T14:57:54.821331+0000","last_unstale":"2024-02-19T14:57:55.826326+0000","last_undegraded":"2024-02-19T14:57:55.826326+0000","last_fullsized":"2024-02-19T14:57:55.826326+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:45.122910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826209+0000","last_change":"2024-02-19T14:57:45.784872+0000","last_active":"2024-02-19T14:57:55.826209+0000","last_peered":"2024-02-19T14:57:55.826209+0000","last_clean":"2024-02-19T14:57:55.826209+0000","last_became_active":"2024-02-19T14:57:45.784602+0000","last_became_peered":"2024-02-19T14:57:45.784602+0000","last_unstale":"2024-02-19T14:57:55.826209+0000","last_undegraded":"2024-02-19T14:57:55.826209+0000","last_fullsized":"2024-02-19T14:57:55.826209+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819401+0000","last_change":"2024-02-19T14:57:54.821393+0000","last_active":"2024-02-19T14:57:55.819401+0000","last_peered":"2024-02-19T14:57:55.819401+0000","last_clean":"2024-02-19T14:57:55.819401+0000","last_became_active":"2024-02-19T14:57:54.820641+0000","last_became_peered":"2024-02-19T14:57:54.820641+0000","last_unstale":"2024-02-19T14:57:55.819401+0000","last_undegraded":"2024-02-19T14:57:55.819401+0000","last_fullsized":"2024-02-19T14:57:55.819401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:28:31.578824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995251+0000","last_change":"2024-02-19T14:57:45.771733+0000","last_active":"2024-02-19T14:57:56.995251+0000","last_peered":"2024-02-19T14:57:56.995251+0000","last_clean":"2024-02-19T14:57:56.995251+0000","last_became_active":"2024-02-19T14:57:45.771454+0000","last_became_peered":"2024-02-19T14:57:45.771454+0000","last_unstale":"2024-02-19T14:57:56.995251+0000","last_undegraded":"2024-02-19T14:57:56.995251+0000","last_fullsized":"2024-02-19T14:57:56.995251+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:52.644495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819447+0000","last_change":"2024-02-19T14:57:54.827044+0000","last_active":"2024-02-19T14:57:55.819447+0000","last_peered":"2024-02-19T14:57:55.819447+0000","last_clean":"2024-02-19T14:57:55.819447+0000","last_became_active":"2024-02-19T14:57:54.826224+0000","last_became_peered":"2024-02-19T14:57:54.826224+0000","last_unstale":"2024-02-19T14:57:55.819447+0000","last_undegraded":"2024-02-19T14:57:55.819447+0000","last_fullsized":"2024-02-19T14:57:55.819447+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:04:44.434234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.996004+0000","last_change":"2024-02-19T14:57:45.772926+0000","last_active":"2024-02-19T14:57:56.996004+0000","last_peered":"2024-02-19T14:57:56.996004+0000","last_clean":"2024-02-19T14:57:56.996004+0000","last_became_active":"2024-02-19T14:57:45.772628+0000","last_became_peered":"2024-02-19T14:57:45.772628+0000","last_unstale":"2024-02-19T14:57:56.996004+0000","last_undegraded":"2024-02-19T14:57:56.996004+0000","last_fullsized":"2024-02-19T14:57:56.996004+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825587+0000","last_change":"2024-02-19T14:57:54.828022+0000","last_active":"2024-02-19T14:57:55.825587+0000","last_peered":"2024-02-19T14:57:55.825587+0000","last_clean":"2024-02-19T14:57:55.825587+0000","last_became_active":"2024-02-19T14:57:54.827764+0000","last_became_peered":"2024-02-19T14:57:54.827764+0000","last_unstale":"2024-02-19T14:57:55.825587+0000","last_undegraded":"2024-02-19T14:57:55.825587+0000","last_fullsized":"2024-02-19T14:57:55.825587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:25.016688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823385+0000","last_change":"2024-02-19T14:57:45.774821+0000","last_active":"2024-02-19T14:57:55.823385+0000","last_peered":"2024-02-19T14:57:55.823385+0000","last_clean":"2024-02-19T14:57:55.823385+0000","last_became_active":"2024-02-19T14:57:45.774541+0000","last_became_peered":"2024-02-19T14:57:45.774541+0000","last_unstale":"2024-02-19T14:57:55.823385+0000","last_undegraded":"2024-02-19T14:57:55.823385+0000","last_fullsized":"2024-02-19T14:57:55.823385+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:04:34.280659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824647+0000","last_change":"2024-02-19T14:57:54.829474+0000","last_active":"2024-02-19T14:57:55.824647+0000","last_peered":"2024-02-19T14:57:55.824647+0000","last_clean":"2024-02-19T14:57:55.824647+0000","last_became_active":"2024-02-19T14:57:54.829146+0000","last_became_peered":"2024-02-19T14:57:54.829146+0000","last_unstale":"2024-02-19T14:57:55.824647+0000","last_undegraded":"2024-02-19T14:57:55.824647+0000","last_fullsized":"2024-02-19T14:57:55.824647+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.602087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819718+0000","last_change":"2024-02-19T14:57:45.784067+0000","last_active":"2024-02-19T14:57:55.819718+0000","last_peered":"2024-02-19T14:57:55.819718+0000","last_clean":"2024-02-19T14:57:55.819718+0000","last_became_active":"2024-02-19T14:57:45.783769+0000","last_became_peered":"2024-02-19T14:57:45.783769+0000","last_unstale":"2024-02-19T14:57:55.819718+0000","last_undegraded":"2024-02-19T14:57:55.819718+0000","last_fullsized":"2024-02-19T14:57:55.819718+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:23:37.778199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825493+0000","last_change":"2024-02-19T14:57:54.832333+0000","last_active":"2024-02-19T14:57:55.825493+0000","last_peered":"2024-02-19T14:57:55.825493+0000","last_clean":"2024-02-19T14:57:55.825493+0000","last_became_active":"2024-02-19T14:57:54.832059+0000","last_became_peered":"2024-02-19T14:57:54.832059+0000","last_unstale":"2024-02-19T14:57:55.825493+0000","last_undegraded":"2024-02-19T14:57:55.825493+0000","last_fullsized":"2024-02-19T14:57:55.825493+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.602087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824787+0000","last_change":"2024-02-19T14:57:45.776863+0000","last_active":"2024-02-19T14:57:55.824787+0000","last_peered":"2024-02-19T14:57:55.824787+0000","last_clean":"2024-02-19T14:57:55.824787+0000","last_became_active":"2024-02-19T14:57:45.776525+0000","last_became_peered":"2024-02-19T14:57:45.776525+0000","last_unstale":"2024-02-19T14:57:55.824787+0000","last_undegraded":"2024-02-19T14:57:55.824787+0000","last_fullsized":"2024-02-19T14:57:55.824787+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:29.747588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825716+0000","last_change":"2024-02-19T14:57:54.820132+0000","last_active":"2024-02-19T14:57:55.825716+0000","last_peered":"2024-02-19T14:57:55.825716+0000","last_clean":"2024-02-19T14:57:55.825716+0000","last_became_active":"2024-02-19T14:57:54.819765+0000","last_became_peered":"2024-02-19T14:57:54.819765+0000","last_unstale":"2024-02-19T14:57:55.825716+0000","last_undegraded":"2024-02-19T14:57:55.825716+0000","last_fullsized":"2024-02-19T14:57:55.825716+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:38.343447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994212+0000","last_change":"2024-02-19T14:57:45.773937+0000","last_active":"2024-02-19T14:57:56.994212+0000","last_peered":"2024-02-19T14:57:56.994212+0000","last_clean":"2024-02-19T14:57:56.994212+0000","last_became_active":"2024-02-19T14:57:45.773657+0000","last_became_peered":"2024-02-19T14:57:45.773657+0000","last_unstale":"2024-02-19T14:57:56.994212+0000","last_undegraded":"2024-02-19T14:57:56.994212+0000","last_fullsized":"2024-02-19T14:57:56.994212+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:23.336087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825575+0000","last_change":"2024-02-19T14:57:54.820021+0000","last_active":"2024-02-19T14:57:55.825575+0000","last_peered":"2024-02-19T14:57:55.825575+0000","last_clean":"2024-02-19T14:57:55.825575+0000","last_became_active":"2024-02-19T14:57:54.819576+0000","last_became_peered":"2024-02-19T14:57:54.819576+0000","last_unstale":"2024-02-19T14:57:55.825575+0000","last_undegraded":"2024-02-19T14:57:55.825575+0000","last_fullsized":"2024-02-19T14:57:55.825575+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:37.441462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824941+0000","last_change":"2024-02-19T14:57:45.776452+0000","last_active":"2024-02-19T14:57:55.824941+0000","last_peered":"2024-02-19T14:57:55.824941+0000","last_clean":"2024-02-19T14:57:55.824941+0000","last_became_active":"2024-02-19T14:57:45.775815+0000","last_became_peered":"2024-02-19T14:57:45.775815+0000","last_unstale":"2024-02-19T14:57:55.824941+0000","last_undegraded":"2024-02-19T14:57:55.824941+0000","last_fullsized":"2024-02-19T14:57:55.824941+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:29.339211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825699+0000","last_change":"2024-02-19T14:57:54.833743+0000","last_active":"2024-02-19T14:57:55.825699+0000","last_peered":"2024-02-19T14:57:55.825699+0000","last_clean":"2024-02-19T14:57:55.825699+0000","last_became_active":"2024-02-19T14:57:54.833488+0000","last_became_peered":"2024-02-19T14:57:54.833488+0000","last_unstale":"2024-02-19T14:57:55.825699+0000","last_undegraded":"2024-02-19T14:57:55.825699+0000","last_fullsized":"2024-02-19T14:57:55.825699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:01:05.123628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819695+0000","last_change":"2024-02-19T14:57:45.773952+0000","last_active":"2024-02-19T14:57:55.819695+0000","last_peered":"2024-02-19T14:57:55.819695+0000","last_clean":"2024-02-19T14:57:55.819695+0000","last_became_active":"2024-02-19T14:57:45.773492+0000","last_became_peered":"2024-02-19T14:57:45.773492+0000","last_unstale":"2024-02-19T14:57:55.819695+0000","last_undegraded":"2024-02-19T14:57:55.819695+0000","last_fullsized":"2024-02-19T14:57:55.819695+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:31.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819110+0000","last_change":"2024-02-19T14:57:54.831464+0000","last_active":"2024-02-19T14:57:55.819110+0000","last_peered":"2024-02-19T14:57:55.819110+0000","last_clean":"2024-02-19T14:57:55.819110+0000","last_became_active":"2024-02-19T14:57:54.831192+0000","last_became_peered":"2024-02-19T14:57:54.831192+0000","last_unstale":"2024-02-19T14:57:55.819110+0000","last_undegraded":"2024-02-19T14:57:55.819110+0000","last_fullsized":"2024-02-19T14:57:55.819110+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:24.433516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823446+0000","last_change":"2024-02-19T14:57:45.781678+0000","last_active":"2024-02-19T14:57:55.823446+0000","last_peered":"2024-02-19T14:57:55.823446+0000","last_clean":"2024-02-19T14:57:55.823446+0000","last_became_active":"2024-02-19T14:57:45.781402+0000","last_became_peered":"2024-02-19T14:57:45.781402+0000","last_unstale":"2024-02-19T14:57:55.823446+0000","last_undegraded":"2024-02-19T14:57:55.823446+0000","last_fullsized":"2024-02-19T14:57:55.823446+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:39.684791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823340+0000","last_change":"2024-02-19T14:57:54.821465+0000","last_active":"2024-02-19T14:57:55.823340+0000","last_peered":"2024-02-19T14:57:55.823340+0000","last_clean":"2024-02-19T14:57:55.823340+0000","last_became_active":"2024-02-19T14:57:54.821199+0000","last_became_peered":"2024-02-19T14:57:54.821199+0000","last_unstale":"2024-02-19T14:57:55.823340+0000","last_undegraded":"2024-02-19T14:57:55.823340+0000","last_fullsized":"2024-02-19T14:57:55.823340+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:10.640894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825748+0000","last_change":"2024-02-19T14:57:45.779802+0000","last_active":"2024-02-19T14:57:55.825748+0000","last_peered":"2024-02-19T14:57:55.825748+0000","last_clean":"2024-02-19T14:57:55.825748+0000","last_became_active":"2024-02-19T14:57:45.779534+0000","last_became_peered":"2024-02-19T14:57:45.779534+0000","last_unstale":"2024-02-19T14:57:55.825748+0000","last_undegraded":"2024-02-19T14:57:55.825748+0000","last_fullsized":"2024-02-19T14:57:55.825748+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995294+0000","last_change":"2024-02-19T14:57:54.833379+0000","last_active":"2024-02-19T14:57:56.995294+0000","last_peered":"2024-02-19T14:57:56.995294+0000","last_clean":"2024-02-19T14:57:56.995294+0000","last_became_active":"2024-02-19T14:57:54.833074+0000","last_became_peered":"2024-02-19T14:57:54.833074+0000","last_unstale":"2024-02-19T14:57:56.995294+0000","last_undegraded":"2024-02-19T14:57:56.995294+0000","last_fullsized":"2024-02-19T14:57:56.995294+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:35:44.451005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819212+0000","last_change":"2024-02-19T14:57:45.785669+0000","last_active":"2024-02-19T14:57:55.819212+0000","last_peered":"2024-02-19T14:57:55.819212+0000","last_clean":"2024-02-19T14:57:55.819212+0000","last_became_active":"2024-02-19T14:57:45.785083+0000","last_became_peered":"2024-02-19T14:57:45.785083+0000","last_unstale":"2024-02-19T14:57:55.819212+0000","last_undegraded":"2024-02-19T14:57:55.819212+0000","last_fullsized":"2024-02-19T14:57:55.819212+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:00.318619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823002+0000","last_change":"2024-02-19T14:57:54.823040+0000","last_active":"2024-02-19T14:57:55.823002+0000","last_peered":"2024-02-19T14:57:55.823002+0000","last_clean":"2024-02-19T14:57:55.823002+0000","last_became_active":"2024-02-19T14:57:54.822774+0000","last_became_peered":"2024-02-19T14:57:54.822774+0000","last_unstale":"2024-02-19T14:57:55.823002+0000","last_undegraded":"2024-02-19T14:57:55.823002+0000","last_fullsized":"2024-02-19T14:57:55.823002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:56:06.520044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824952+0000","last_change":"2024-02-19T14:57:45.776205+0000","last_active":"2024-02-19T14:57:55.824952+0000","last_peered":"2024-02-19T14:57:55.824952+0000","last_clean":"2024-02-19T14:57:55.824952+0000","last_became_active":"2024-02-19T14:57:45.775407+0000","last_became_peered":"2024-02-19T14:57:45.775407+0000","last_unstale":"2024-02-19T14:57:55.824952+0000","last_undegraded":"2024-02-19T14:57:55.824952+0000","last_fullsized":"2024-02-19T14:57:55.824952+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:37.658842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818843+0000","last_change":"2024-02-19T14:57:54.829145+0000","last_active":"2024-02-19T14:57:55.818843+0000","last_peered":"2024-02-19T14:57:55.818843+0000","last_clean":"2024-02-19T14:57:55.818843+0000","last_became_active":"2024-02-19T14:57:54.827708+0000","last_became_peered":"2024-02-19T14:57:54.827708+0000","last_unstale":"2024-02-19T14:57:55.818843+0000","last_undegraded":"2024-02-19T14:57:55.818843+0000","last_fullsized":"2024-02-19T14:57:55.818843+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:17.275536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994911+0000","last_change":"2024-02-19T14:57:45.774589+0000","last_active":"2024-02-19T14:57:56.994911+0000","last_peered":"2024-02-19T14:57:56.994911+0000","last_clean":"2024-02-19T14:57:56.994911+0000","last_became_active":"2024-02-19T14:57:45.774344+0000","last_became_peered":"2024-02-19T14:57:45.774344+0000","last_unstale":"2024-02-19T14:57:56.994911+0000","last_undegraded":"2024-02-19T14:57:56.994911+0000","last_fullsized":"2024-02-19T14:57:56.994911+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:04.979975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995451+0000","last_change":"2024-02-19T14:57:54.816982+0000","last_active":"2024-02-19T14:57:56.995451+0000","last_peered":"2024-02-19T14:57:56.995451+0000","last_clean":"2024-02-19T14:57:56.995451+0000","last_became_active":"2024-02-19T14:57:54.816702+0000","last_became_peered":"2024-02-19T14:57:54.816702+0000","last_unstale":"2024-02-19T14:57:56.995451+0000","last_undegraded":"2024-02-19T14:57:56.995451+0000","last_fullsized":"2024-02-19T14:57:56.995451+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:11.954004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994855+0000","last_change":"2024-02-19T14:57:45.774703+0000","last_active":"2024-02-19T14:57:56.994855+0000","last_peered":"2024-02-19T14:57:56.994855+0000","last_clean":"2024-02-19T14:57:56.994855+0000","last_became_active":"2024-02-19T14:57:45.773430+0000","last_became_peered":"2024-02-19T14:57:45.773430+0000","last_unstale":"2024-02-19T14:57:56.994855+0000","last_undegraded":"2024-02-19T14:57:56.994855+0000","last_fullsized":"2024-02-19T14:57:56.994855+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:56.911480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995277+0000","last_change":"2024-02-19T14:57:54.816335+0000","last_active":"2024-02-19T14:57:56.995277+0000","last_peered":"2024-02-19T14:57:56.995277+0000","last_clean":"2024-02-19T14:57:56.995277+0000","last_became_active":"2024-02-19T14:57:54.815840+0000","last_became_peered":"2024-02-19T14:57:54.815840+0000","last_unstale":"2024-02-19T14:57:56.995277+0000","last_undegraded":"2024-02-19T14:57:56.995277+0000","last_fullsized":"2024-02-19T14:57:56.995277+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:37:51.650069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995326+0000","last_change":"2024-02-19T14:57:45.780128+0000","last_active":"2024-02-19T14:57:56.995326+0000","last_peered":"2024-02-19T14:57:56.995326+0000","last_clean":"2024-02-19T14:57:56.995326+0000","last_became_active":"2024-02-19T14:57:45.779825+0000","last_became_peered":"2024-02-19T14:57:45.779825+0000","last_unstale":"2024-02-19T14:57:56.995326+0000","last_undegraded":"2024-02-19T14:57:56.995326+0000","last_fullsized":"2024-02-19T14:57:56.995326+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:26.050909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825694+0000","last_change":"2024-02-19T14:57:54.828181+0000","last_active":"2024-02-19T14:57:55.825694+0000","last_peered":"2024-02-19T14:57:55.825694+0000","last_clean":"2024-02-19T14:57:55.825694+0000","last_became_active":"2024-02-19T14:57:54.827867+0000","last_became_peered":"2024-02-19T14:57:54.827867+0000","last_unstale":"2024-02-19T14:57:55.825694+0000","last_undegraded":"2024-02-19T14:57:55.825694+0000","last_fullsized":"2024-02-19T14:57:55.825694+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:13.020379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994212+0000","last_change":"2024-02-19T14:57:53.820089+0000","last_active":"2024-02-19T14:57:56.994212+0000","last_peered":"2024-02-19T14:57:56.994212+0000","last_clean":"2024-02-19T14:57:56.994212+0000","last_became_active":"2024-02-19T14:57:53.819799+0000","last_became_peered":"2024-02-19T14:57:53.819799+0000","last_unstale":"2024-02-19T14:57:56.994212+0000","last_undegraded":"2024-02-19T14:57:56.994212+0000","last_fullsized":"2024-02-19T14:57:56.994212+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:09.687163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825974+0000","last_change":"2024-02-19T14:57:45.781182+0000","last_active":"2024-02-19T14:57:55.825974+0000","last_peered":"2024-02-19T14:57:55.825974+0000","last_clean":"2024-02-19T14:57:55.825974+0000","last_became_active":"2024-02-19T14:57:45.780764+0000","last_became_peered":"2024-02-19T14:57:45.780764+0000","last_unstale":"2024-02-19T14:57:55.825974+0000","last_undegraded":"2024-02-19T14:57:55.825974+0000","last_fullsized":"2024-02-19T14:57:55.825974+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:23.510855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"151'4","reported_seq":26,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826625+0000","last_change":"2024-02-19T14:57:44.748896+0000","last_active":"2024-02-19T14:57:55.826625+0000","last_peered":"2024-02-19T14:57:55.826625+0000","last_clean":"2024-02-19T14:57:55.826625+0000","last_became_active":"2024-02-19T14:57:44.748624+0000","last_became_peered":"2024-02-19T14:57:44.748624+0000","last_unstale":"2024-02-19T14:57:55.826625+0000","last_undegraded":"2024-02-19T14:57:55.826625+0000","last_fullsized":"2024-02-19T14:57:55.826625+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:53:27.719470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826462+0000","last_change":"2024-02-19T14:57:54.829689+0000","last_active":"2024-02-19T14:57:55.826462+0000","last_peered":"2024-02-19T14:57:55.826462+0000","last_clean":"2024-02-19T14:57:55.826462+0000","last_became_active":"2024-02-19T14:57:54.829434+0000","last_became_peered":"2024-02-19T14:57:54.829434+0000","last_unstale":"2024-02-19T14:57:55.826462+0000","last_undegraded":"2024-02-19T14:57:55.826462+0000","last_fullsized":"2024-02-19T14:57:55.826462+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:01.392908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"155'8","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995289+0000","last_change":"2024-02-19T14:57:53.816514+0000","last_active":"2024-02-19T14:57:56.995289+0000","last_peered":"2024-02-19T14:57:56.995289+0000","last_clean":"2024-02-19T14:57:56.995289+0000","last_became_active":"2024-02-19T14:57:53.816138+0000","last_became_peered":"2024-02-19T14:57:53.816138+0000","last_unstale":"2024-02-19T14:57:56.995289+0000","last_undegraded":"2024-02-19T14:57:56.995289+0000","last_fullsized":"2024-02-19T14:57:56.995289+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:22.636629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818746+0000","last_change":"2024-02-19T14:57:45.778729+0000","last_active":"2024-02-19T14:57:55.818746+0000","last_peered":"2024-02-19T14:57:55.818746+0000","last_clean":"2024-02-19T14:57:55.818746+0000","last_became_active":"2024-02-19T14:57:45.778460+0000","last_became_peered":"2024-02-19T14:57:45.778460+0000","last_unstale":"2024-02-19T14:57:55.818746+0000","last_undegraded":"2024-02-19T14:57:55.818746+0000","last_fullsized":"2024-02-19T14:57:55.818746+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:46.115236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995061+0000","last_change":"2024-02-19T14:57:44.759512+0000","last_active":"2024-02-19T14:57:56.995061+0000","last_peered":"2024-02-19T14:57:56.995061+0000","last_clean":"2024-02-19T14:57:56.995061+0000","last_became_active":"2024-02-19T14:57:44.759251+0000","last_became_peered":"2024-02-19T14:57:44.759251+0000","last_unstale":"2024-02-19T14:57:56.995061+0000","last_undegraded":"2024-02-19T14:57:56.995061+0000","last_fullsized":"2024-02-19T14:57:56.995061+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:44:23.064686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822886+0000","last_change":"2024-02-19T14:57:54.830445+0000","last_active":"2024-02-19T14:57:55.822886+0000","last_peered":"2024-02-19T14:57:55.822886+0000","last_clean":"2024-02-19T14:57:55.822886+0000","last_became_active":"2024-02-19T14:57:54.829989+0000","last_became_peered":"2024-02-19T14:57:54.829989+0000","last_unstale":"2024-02-19T14:57:55.822886+0000","last_undegraded":"2024-02-19T14:57:55.822886+0000","last_fullsized":"2024-02-19T14:57:55.822886+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:38.403195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826036+0000","last_change":"2024-02-19T14:57:53.824642+0000","last_active":"2024-02-19T14:57:55.826036+0000","last_peered":"2024-02-19T14:57:55.826036+0000","last_clean":"2024-02-19T14:57:55.826036+0000","last_became_active":"2024-02-19T14:57:53.824274+0000","last_became_peered":"2024-02-19T14:57:53.824274+0000","last_unstale":"2024-02-19T14:57:55.826036+0000","last_undegraded":"2024-02-19T14:57:55.826036+0000","last_fullsized":"2024-02-19T14:57:55.826036+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:50.343572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994687+0000","last_change":"2024-02-19T14:57:45.775749+0000","last_active":"2024-02-19T14:57:56.994687+0000","last_peered":"2024-02-19T14:57:56.994687+0000","last_clean":"2024-02-19T14:57:56.994687+0000","last_became_active":"2024-02-19T14:57:45.775481+0000","last_became_peered":"2024-02-19T14:57:45.775481+0000","last_unstale":"2024-02-19T14:57:56.994687+0000","last_undegraded":"2024-02-19T14:57:56.994687+0000","last_fullsized":"2024-02-19T14:57:56.994687+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:45.635205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.996066+0000","last_change":"2024-02-19T14:57:44.746085+0000","last_active":"2024-02-19T14:57:56.996066+0000","last_peered":"2024-02-19T14:57:56.996066+0000","last_clean":"2024-02-19T14:57:56.996066+0000","last_became_active":"2024-02-19T14:57:44.745804+0000","last_became_peered":"2024-02-19T14:57:44.745804+0000","last_unstale":"2024-02-19T14:57:56.996066+0000","last_undegraded":"2024-02-19T14:57:56.996066+0000","last_fullsized":"2024-02-19T14:57:56.996066+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.818833+0000","last_change":"2024-02-19T14:57:54.824114+0000","last_active":"2024-02-19T14:57:55.818833+0000","last_peered":"2024-02-19T14:57:55.818833+0000","last_clean":"2024-02-19T14:57:55.818833+0000","last_became_active":"2024-02-19T14:57:54.823842+0000","last_became_peered":"2024-02-19T14:57:54.823842+0000","last_unstale":"2024-02-19T14:57:55.818833+0000","last_undegraded":"2024-02-19T14:57:55.818833+0000","last_fullsized":"2024-02-19T14:57:55.818833+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:50:43.035644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826093+0000","last_change":"2024-02-19T14:57:53.819661+0000","last_active":"2024-02-19T14:57:55.826093+0000","last_peered":"2024-02-19T14:57:55.826093+0000","last_clean":"2024-02-19T14:57:55.826093+0000","last_became_active":"2024-02-19T14:57:53.819408+0000","last_became_peered":"2024-02-19T14:57:53.819408+0000","last_unstale":"2024-02-19T14:57:55.826093+0000","last_undegraded":"2024-02-19T14:57:55.826093+0000","last_fullsized":"2024-02-19T14:57:55.826093+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:22.076038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824979+0000","last_change":"2024-02-19T14:57:45.784232+0000","last_active":"2024-02-19T14:57:55.824979+0000","last_peered":"2024-02-19T14:57:55.824979+0000","last_clean":"2024-02-19T14:57:55.824979+0000","last_became_active":"2024-02-19T14:57:45.784007+0000","last_became_peered":"2024-02-19T14:57:45.784007+0000","last_unstale":"2024-02-19T14:57:55.824979+0000","last_undegraded":"2024-02-19T14:57:55.824979+0000","last_fullsized":"2024-02-19T14:57:55.824979+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:36:50.022179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823309+0000","last_change":"2024-02-19T14:57:44.754196+0000","last_active":"2024-02-19T14:57:55.823309+0000","last_peered":"2024-02-19T14:57:55.823309+0000","last_clean":"2024-02-19T14:57:55.823309+0000","last_became_active":"2024-02-19T14:57:44.753938+0000","last_became_peered":"2024-02-19T14:57:44.753938+0000","last_unstale":"2024-02-19T14:57:55.823309+0000","last_undegraded":"2024-02-19T14:57:55.823309+0000","last_fullsized":"2024-02-19T14:57:55.823309+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:51:17.948844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825843+0000","last_change":"2024-02-19T14:57:54.822487+0000","last_active":"2024-02-19T14:57:55.825843+0000","last_peered":"2024-02-19T14:57:55.825843+0000","last_clean":"2024-02-19T14:57:55.825843+0000","last_became_active":"2024-02-19T14:57:54.822187+0000","last_became_peered":"2024-02-19T14:57:54.822187+0000","last_unstale":"2024-02-19T14:57:55.825843+0000","last_undegraded":"2024-02-19T14:57:55.825843+0000","last_fullsized":"2024-02-19T14:57:55.825843+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:32.400071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"155'3","reported_seq":17,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994644+0000","last_change":"2024-02-19T14:57:53.823653+0000","last_active":"2024-02-19T14:57:56.994644+0000","last_peered":"2024-02-19T14:57:56.994644+0000","last_clean":"2024-02-19T14:57:56.994644+0000","last_became_active":"2024-02-19T14:57:53.822920+0000","last_became_peered":"2024-02-19T14:57:53.822920+0000","last_unstale":"2024-02-19T14:57:56.994644+0000","last_undegraded":"2024-02-19T14:57:56.994644+0000","last_fullsized":"2024-02-19T14:57:56.994644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:02.674757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825631+0000","last_change":"2024-02-19T14:57:45.776534+0000","last_active":"2024-02-19T14:57:55.825631+0000","last_peered":"2024-02-19T14:57:55.825631+0000","last_clean":"2024-02-19T14:57:55.825631+0000","last_became_active":"2024-02-19T14:57:45.776277+0000","last_became_peered":"2024-02-19T14:57:45.776277+0000","last_unstale":"2024-02-19T14:57:55.825631+0000","last_undegraded":"2024-02-19T14:57:55.825631+0000","last_fullsized":"2024-02-19T14:57:55.825631+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:33.086114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826026+0000","last_change":"2024-02-19T14:57:44.759675+0000","last_active":"2024-02-19T14:57:55.826026+0000","last_peered":"2024-02-19T14:57:55.826026+0000","last_clean":"2024-02-19T14:57:55.826026+0000","last_became_active":"2024-02-19T14:57:44.759404+0000","last_became_peered":"2024-02-19T14:57:44.759404+0000","last_unstale":"2024-02-19T14:57:55.826026+0000","last_undegraded":"2024-02-19T14:57:55.826026+0000","last_fullsized":"2024-02-19T14:57:55.826026+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:34.658675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826373+0000","last_change":"2024-02-19T14:57:54.817366+0000","last_active":"2024-02-19T14:57:55.826373+0000","last_peered":"2024-02-19T14:57:55.826373+0000","last_clean":"2024-02-19T14:57:55.826373+0000","last_became_active":"2024-02-19T14:57:54.817126+0000","last_became_peered":"2024-02-19T14:57:54.817126+0000","last_unstale":"2024-02-19T14:57:55.826373+0000","last_undegraded":"2024-02-19T14:57:55.826373+0000","last_fullsized":"2024-02-19T14:57:55.826373+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:25.058182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"155'6","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825191+0000","last_change":"2024-02-19T14:57:53.814585+0000","last_active":"2024-02-19T14:57:55.825191+0000","last_peered":"2024-02-19T14:57:55.825191+0000","last_clean":"2024-02-19T14:57:55.825191+0000","last_became_active":"2024-02-19T14:57:53.814309+0000","last_became_peered":"2024-02-19T14:57:53.814309+0000","last_unstale":"2024-02-19T14:57:55.825191+0000","last_undegraded":"2024-02-19T14:57:55.825191+0000","last_fullsized":"2024-02-19T14:57:55.825191+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:54.429397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823403+0000","last_change":"2024-02-19T14:57:45.777221+0000","last_active":"2024-02-19T14:57:55.823403+0000","last_peered":"2024-02-19T14:57:55.823403+0000","last_clean":"2024-02-19T14:57:55.823403+0000","last_became_active":"2024-02-19T14:57:45.776887+0000","last_became_peered":"2024-02-19T14:57:45.776887+0000","last_unstale":"2024-02-19T14:57:55.823403+0000","last_undegraded":"2024-02-19T14:57:55.823403+0000","last_fullsized":"2024-02-19T14:57:55.823403+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:33:43.785509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994740+0000","last_change":"2024-02-19T14:57:44.759410+0000","last_active":"2024-02-19T14:57:56.994740+0000","last_peered":"2024-02-19T14:57:56.994740+0000","last_clean":"2024-02-19T14:57:56.994740+0000","last_became_active":"2024-02-19T14:57:44.759125+0000","last_became_peered":"2024-02-19T14:57:44.759125+0000","last_unstale":"2024-02-19T14:57:56.994740+0000","last_undegraded":"2024-02-19T14:57:56.994740+0000","last_fullsized":"2024-02-19T14:57:56.994740+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:47.742311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825656+0000","last_change":"2024-02-19T14:57:54.831577+0000","last_active":"2024-02-19T14:57:55.825656+0000","last_peered":"2024-02-19T14:57:55.825656+0000","last_clean":"2024-02-19T14:57:55.825656+0000","last_became_active":"2024-02-19T14:57:54.830980+0000","last_became_peered":"2024-02-19T14:57:54.830980+0000","last_unstale":"2024-02-19T14:57:55.825656+0000","last_undegraded":"2024-02-19T14:57:55.825656+0000","last_fullsized":"2024-02-19T14:57:55.825656+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:57:38.343447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819427+0000","last_change":"2024-02-19T14:57:53.817680+0000","last_active":"2024-02-19T14:57:55.819427+0000","last_peered":"2024-02-19T14:57:55.819427+0000","last_clean":"2024-02-19T14:57:55.819427+0000","last_became_active":"2024-02-19T14:57:53.817435+0000","last_became_peered":"2024-02-19T14:57:53.817435+0000","last_unstale":"2024-02-19T14:57:55.819427+0000","last_undegraded":"2024-02-19T14:57:55.819427+0000","last_fullsized":"2024-02-19T14:57:55.819427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:24.053686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823526+0000","last_change":"2024-02-19T14:57:45.778260+0000","last_active":"2024-02-19T14:57:55.823526+0000","last_peered":"2024-02-19T14:57:55.823526+0000","last_clean":"2024-02-19T14:57:55.823526+0000","last_became_active":"2024-02-19T14:57:45.778006+0000","last_became_peered":"2024-02-19T14:57:45.778006+0000","last_unstale":"2024-02-19T14:57:55.823526+0000","last_undegraded":"2024-02-19T14:57:55.823526+0000","last_fullsized":"2024-02-19T14:57:55.823526+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:57:35.800806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823279+0000","last_change":"2024-02-19T14:57:44.757229+0000","last_active":"2024-02-19T14:57:55.823279+0000","last_peered":"2024-02-19T14:57:55.823279+0000","last_clean":"2024-02-19T14:57:55.823279+0000","last_became_active":"2024-02-19T14:57:44.756966+0000","last_became_peered":"2024-02-19T14:57:44.756966+0000","last_unstale":"2024-02-19T14:57:55.823279+0000","last_undegraded":"2024-02-19T14:57:55.823279+0000","last_fullsized":"2024-02-19T14:57:55.823279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:12.299965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819929+0000","last_change":"2024-02-19T14:57:54.823936+0000","last_active":"2024-02-19T14:57:55.819929+0000","last_peered":"2024-02-19T14:57:55.819929+0000","last_clean":"2024-02-19T14:57:55.819929+0000","last_became_active":"2024-02-19T14:57:54.823668+0000","last_became_peered":"2024-02-19T14:57:54.823668+0000","last_unstale":"2024-02-19T14:57:55.819929+0000","last_undegraded":"2024-02-19T14:57:55.819929+0000","last_fullsized":"2024-02-19T14:57:55.819929+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:16.936732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825752+0000","last_change":"2024-02-19T14:57:53.819203+0000","last_active":"2024-02-19T14:57:55.825752+0000","last_peered":"2024-02-19T14:57:55.825752+0000","last_clean":"2024-02-19T14:57:55.825752+0000","last_became_active":"2024-02-19T14:57:53.818961+0000","last_became_peered":"2024-02-19T14:57:53.818961+0000","last_unstale":"2024-02-19T14:57:55.825752+0000","last_undegraded":"2024-02-19T14:57:55.825752+0000","last_fullsized":"2024-02-19T14:57:55.825752+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:45.718331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826421+0000","last_change":"2024-02-19T14:57:45.781313+0000","last_active":"2024-02-19T14:57:55.826421+0000","last_peered":"2024-02-19T14:57:55.826421+0000","last_clean":"2024-02-19T14:57:55.826421+0000","last_became_active":"2024-02-19T14:57:45.778982+0000","last_became_peered":"2024-02-19T14:57:45.778982+0000","last_unstale":"2024-02-19T14:57:55.826421+0000","last_undegraded":"2024-02-19T14:57:55.826421+0000","last_fullsized":"2024-02-19T14:57:55.826421+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:23.251435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825378+0000","last_change":"2024-02-19T14:57:44.757934+0000","last_active":"2024-02-19T14:57:55.825378+0000","last_peered":"2024-02-19T14:57:55.825378+0000","last_clean":"2024-02-19T14:57:55.825378+0000","last_became_active":"2024-02-19T14:57:44.757660+0000","last_became_peered":"2024-02-19T14:57:44.757660+0000","last_unstale":"2024-02-19T14:57:55.825378+0000","last_undegraded":"2024-02-19T14:57:55.825378+0000","last_fullsized":"2024-02-19T14:57:55.825378+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825909+0000","last_change":"2024-02-19T14:57:54.817951+0000","last_active":"2024-02-19T14:57:55.825909+0000","last_peered":"2024-02-19T14:57:55.825909+0000","last_clean":"2024-02-19T14:57:55.825909+0000","last_became_active":"2024-02-19T14:57:54.817522+0000","last_became_peered":"2024-02-19T14:57:54.817522+0000","last_unstale":"2024-02-19T14:57:55.825909+0000","last_undegraded":"2024-02-19T14:57:55.825909+0000","last_fullsized":"2024-02-19T14:57:55.825909+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:51:06.514104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826173+0000","last_change":"2024-02-19T14:57:53.817990+0000","last_active":"2024-02-19T14:57:55.826173+0000","last_peered":"2024-02-19T14:57:55.826173+0000","last_clean":"2024-02-19T14:57:55.826173+0000","last_became_active":"2024-02-19T14:57:53.817661+0000","last_became_peered":"2024-02-19T14:57:53.817661+0000","last_unstale":"2024-02-19T14:57:55.826173+0000","last_undegraded":"2024-02-19T14:57:55.826173+0000","last_fullsized":"2024-02-19T14:57:55.826173+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:49:43.719948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825329+0000","last_change":"2024-02-19T14:57:45.776416+0000","last_active":"2024-02-19T14:57:55.825329+0000","last_peered":"2024-02-19T14:57:55.825329+0000","last_clean":"2024-02-19T14:57:55.825329+0000","last_became_active":"2024-02-19T14:57:45.775916+0000","last_became_peered":"2024-02-19T14:57:45.775916+0000","last_unstale":"2024-02-19T14:57:55.825329+0000","last_undegraded":"2024-02-19T14:57:55.825329+0000","last_fullsized":"2024-02-19T14:57:55.825329+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:33.825283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819592+0000","last_change":"2024-02-19T14:57:44.755422+0000","last_active":"2024-02-19T14:57:55.819592+0000","last_peered":"2024-02-19T14:57:55.819592+0000","last_clean":"2024-02-19T14:57:55.819592+0000","last_became_active":"2024-02-19T14:57:44.755152+0000","last_became_peered":"2024-02-19T14:57:44.755152+0000","last_unstale":"2024-02-19T14:57:55.819592+0000","last_undegraded":"2024-02-19T14:57:55.819592+0000","last_fullsized":"2024-02-19T14:57:55.819592+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:13.575356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824629+0000","last_change":"2024-02-19T14:57:54.826189+0000","last_active":"2024-02-19T14:57:55.824629+0000","last_peered":"2024-02-19T14:57:55.824629+0000","last_clean":"2024-02-19T14:57:55.824629+0000","last_became_active":"2024-02-19T14:57:54.825420+0000","last_became_peered":"2024-02-19T14:57:54.825420+0000","last_unstale":"2024-02-19T14:57:55.824629+0000","last_undegraded":"2024-02-19T14:57:55.824629+0000","last_fullsized":"2024-02-19T14:57:55.824629+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:42.150099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819261+0000","last_change":"2024-02-19T14:57:53.823263+0000","last_active":"2024-02-19T14:57:55.819261+0000","last_peered":"2024-02-19T14:57:55.819261+0000","last_clean":"2024-02-19T14:57:55.819261+0000","last_became_active":"2024-02-19T14:57:53.822954+0000","last_became_peered":"2024-02-19T14:57:53.822954+0000","last_unstale":"2024-02-19T14:57:55.819261+0000","last_undegraded":"2024-02-19T14:57:55.819261+0000","last_fullsized":"2024-02-19T14:57:55.819261+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:51.932100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819020+0000","last_change":"2024-02-19T14:57:45.781850+0000","last_active":"2024-02-19T14:57:55.819020+0000","last_peered":"2024-02-19T14:57:55.819020+0000","last_clean":"2024-02-19T14:57:55.819020+0000","last_became_active":"2024-02-19T14:57:45.781588+0000","last_became_peered":"2024-02-19T14:57:45.781588+0000","last_unstale":"2024-02-19T14:57:55.819020+0000","last_undegraded":"2024-02-19T14:57:55.819020+0000","last_fullsized":"2024-02-19T14:57:55.819020+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:48.140379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"151'3","reported_seq":25,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825805+0000","last_change":"2024-02-19T14:57:44.751600+0000","last_active":"2024-02-19T14:57:55.825805+0000","last_peered":"2024-02-19T14:57:55.825805+0000","last_clean":"2024-02-19T14:57:55.825805+0000","last_became_active":"2024-02-19T14:57:44.751311+0000","last_became_peered":"2024-02-19T14:57:44.751311+0000","last_unstale":"2024-02-19T14:57:55.825805+0000","last_undegraded":"2024-02-19T14:57:55.825805+0000","last_fullsized":"2024-02-19T14:57:55.825805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:50.448796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825401+0000","last_change":"2024-02-19T14:57:54.814493+0000","last_active":"2024-02-19T14:57:55.825401+0000","last_peered":"2024-02-19T14:57:55.825401+0000","last_clean":"2024-02-19T14:57:55.825401+0000","last_became_active":"2024-02-19T14:57:54.814248+0000","last_became_peered":"2024-02-19T14:57:54.814248+0000","last_unstale":"2024-02-19T14:57:55.825401+0000","last_undegraded":"2024-02-19T14:57:55.825401+0000","last_fullsized":"2024-02-19T14:57:55.825401+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:35.195077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994552+0000","last_change":"2024-02-19T14:57:53.823746+0000","last_active":"2024-02-19T14:57:56.994552+0000","last_peered":"2024-02-19T14:57:56.994552+0000","last_clean":"2024-02-19T14:57:56.994552+0000","last_became_active":"2024-02-19T14:57:53.823475+0000","last_became_peered":"2024-02-19T14:57:53.823475+0000","last_unstale":"2024-02-19T14:57:56.994552+0000","last_undegraded":"2024-02-19T14:57:56.994552+0000","last_fullsized":"2024-02-19T14:57:56.994552+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:35:43.446510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823285+0000","last_change":"2024-02-19T14:57:45.782664+0000","last_active":"2024-02-19T14:57:55.823285+0000","last_peered":"2024-02-19T14:57:55.823285+0000","last_clean":"2024-02-19T14:57:55.823285+0000","last_became_active":"2024-02-19T14:57:45.782411+0000","last_became_peered":"2024-02-19T14:57:45.782411+0000","last_unstale":"2024-02-19T14:57:55.823285+0000","last_undegraded":"2024-02-19T14:57:55.823285+0000","last_fullsized":"2024-02-19T14:57:55.823285+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:26:26.461781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"151'6","reported_seq":28,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819709+0000","last_change":"2024-02-19T14:57:44.756497+0000","last_active":"2024-02-19T14:57:55.819709+0000","last_peered":"2024-02-19T14:57:55.819709+0000","last_clean":"2024-02-19T14:57:55.819709+0000","last_became_active":"2024-02-19T14:57:44.756213+0000","last_became_peered":"2024-02-19T14:57:44.756213+0000","last_unstale":"2024-02-19T14:57:55.819709+0000","last_undegraded":"2024-02-19T14:57:55.819709+0000","last_fullsized":"2024-02-19T14:57:55.819709+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:28.745803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995601+0000","last_change":"2024-02-19T14:57:54.831104+0000","last_active":"2024-02-19T14:57:56.995601+0000","last_peered":"2024-02-19T14:57:56.995601+0000","last_clean":"2024-02-19T14:57:56.995601+0000","last_became_active":"2024-02-19T14:57:54.830831+0000","last_became_peered":"2024-02-19T14:57:54.830831+0000","last_unstale":"2024-02-19T14:57:56.995601+0000","last_undegraded":"2024-02-19T14:57:56.995601+0000","last_fullsized":"2024-02-19T14:57:56.995601+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:03.679252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995835+0000","last_change":"2024-02-19T14:57:53.811681+0000","last_active":"2024-02-19T14:57:56.995835+0000","last_peered":"2024-02-19T14:57:56.995835+0000","last_clean":"2024-02-19T14:57:56.995835+0000","last_became_active":"2024-02-19T14:57:53.811323+0000","last_became_peered":"2024-02-19T14:57:53.811323+0000","last_unstale":"2024-02-19T14:57:56.995835+0000","last_undegraded":"2024-02-19T14:57:56.995835+0000","last_fullsized":"2024-02-19T14:57:56.995835+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:26:37.807076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994805+0000","last_change":"2024-02-19T14:57:45.780739+0000","last_active":"2024-02-19T14:57:56.994805+0000","last_peered":"2024-02-19T14:57:56.994805+0000","last_clean":"2024-02-19T14:57:56.994805+0000","last_became_active":"2024-02-19T14:57:45.780414+0000","last_became_peered":"2024-02-19T14:57:45.780414+0000","last_unstale":"2024-02-19T14:57:56.994805+0000","last_undegraded":"2024-02-19T14:57:56.994805+0000","last_fullsized":"2024-02-19T14:57:56.994805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:25.103177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994783+0000","last_change":"2024-02-19T14:57:44.751458+0000","last_active":"2024-02-19T14:57:56.994783+0000","last_peered":"2024-02-19T14:57:56.994783+0000","last_clean":"2024-02-19T14:57:56.994783+0000","last_became_active":"2024-02-19T14:57:44.751194+0000","last_became_peered":"2024-02-19T14:57:44.751194+0000","last_unstale":"2024-02-19T14:57:56.994783+0000","last_undegraded":"2024-02-19T14:57:56.994783+0000","last_fullsized":"2024-02-19T14:57:56.994783+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:11:44.993761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995269+0000","last_change":"2024-02-19T14:57:54.826834+0000","last_active":"2024-02-19T14:57:56.995269+0000","last_peered":"2024-02-19T14:57:56.995269+0000","last_clean":"2024-02-19T14:57:56.995269+0000","last_became_active":"2024-02-19T14:57:54.826547+0000","last_became_peered":"2024-02-19T14:57:54.826547+0000","last_unstale":"2024-02-19T14:57:56.995269+0000","last_undegraded":"2024-02-19T14:57:56.995269+0000","last_fullsized":"2024-02-19T14:57:56.995269+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:29.003031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825662+0000","last_change":"2024-02-19T14:57:53.821078+0000","last_active":"2024-02-19T14:57:55.825662+0000","last_peered":"2024-02-19T14:57:55.825662+0000","last_clean":"2024-02-19T14:57:55.825662+0000","last_became_active":"2024-02-19T14:57:53.820812+0000","last_became_peered":"2024-02-19T14:57:53.820812+0000","last_unstale":"2024-02-19T14:57:55.825662+0000","last_undegraded":"2024-02-19T14:57:55.825662+0000","last_fullsized":"2024-02-19T14:57:55.825662+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:56.803585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826124+0000","last_change":"2024-02-19T14:57:45.775378+0000","last_active":"2024-02-19T14:57:55.826124+0000","last_peered":"2024-02-19T14:57:55.826124+0000","last_clean":"2024-02-19T14:57:55.826124+0000","last_became_active":"2024-02-19T14:57:45.774903+0000","last_became_peered":"2024-02-19T14:57:45.774903+0000","last_unstale":"2024-02-19T14:57:55.826124+0000","last_undegraded":"2024-02-19T14:57:55.826124+0000","last_fullsized":"2024-02-19T14:57:55.826124+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:00.558674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819156+0000","last_change":"2024-02-19T14:57:44.748468+0000","last_active":"2024-02-19T14:57:55.819156+0000","last_peered":"2024-02-19T14:57:55.819156+0000","last_clean":"2024-02-19T14:57:55.819156+0000","last_became_active":"2024-02-19T14:57:44.748055+0000","last_became_peered":"2024-02-19T14:57:44.748055+0000","last_unstale":"2024-02-19T14:57:55.819156+0000","last_undegraded":"2024-02-19T14:57:55.819156+0000","last_fullsized":"2024-02-19T14:57:55.819156+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:32.823498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825516+0000","last_change":"2024-02-19T14:57:54.829063+0000","last_active":"2024-02-19T14:57:55.825516+0000","last_peered":"2024-02-19T14:57:55.825516+0000","last_clean":"2024-02-19T14:57:55.825516+0000","last_became_active":"2024-02-19T14:57:54.828658+0000","last_became_peered":"2024-02-19T14:57:54.828658+0000","last_unstale":"2024-02-19T14:57:55.825516+0000","last_undegraded":"2024-02-19T14:57:55.825516+0000","last_fullsized":"2024-02-19T14:57:55.825516+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:30:56.427520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"155'4","reported_seq":18,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995158+0000","last_change":"2024-02-19T14:57:53.827543+0000","last_active":"2024-02-19T14:57:56.995158+0000","last_peered":"2024-02-19T14:57:56.995158+0000","last_clean":"2024-02-19T14:57:56.995158+0000","last_became_active":"2024-02-19T14:57:53.827256+0000","last_became_peered":"2024-02-19T14:57:53.827256+0000","last_unstale":"2024-02-19T14:57:56.995158+0000","last_undegraded":"2024-02-19T14:57:56.995158+0000","last_fullsized":"2024-02-19T14:57:56.995158+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:54.174725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995993+0000","last_change":"2024-02-19T14:57:45.780058+0000","last_active":"2024-02-19T14:57:56.995993+0000","last_peered":"2024-02-19T14:57:56.995993+0000","last_clean":"2024-02-19T14:57:56.995993+0000","last_became_active":"2024-02-19T14:57:45.779771+0000","last_became_peered":"2024-02-19T14:57:45.779771+0000","last_unstale":"2024-02-19T14:57:56.995993+0000","last_undegraded":"2024-02-19T14:57:56.995993+0000","last_fullsized":"2024-02-19T14:57:56.995993+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994963+0000","last_change":"2024-02-19T14:57:44.751398+0000","last_active":"2024-02-19T14:57:56.994963+0000","last_peered":"2024-02-19T14:57:56.994963+0000","last_clean":"2024-02-19T14:57:56.994963+0000","last_became_active":"2024-02-19T14:57:44.751099+0000","last_became_peered":"2024-02-19T14:57:44.751099+0000","last_unstale":"2024-02-19T14:57:56.994963+0000","last_undegraded":"2024-02-19T14:57:56.994963+0000","last_fullsized":"2024-02-19T14:57:56.994963+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:32:51.642710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823209+0000","last_change":"2024-02-19T14:57:54.822447+0000","last_active":"2024-02-19T14:57:55.823209+0000","last_peered":"2024-02-19T14:57:55.823209+0000","last_clean":"2024-02-19T14:57:55.823209+0000","last_became_active":"2024-02-19T14:57:54.822177+0000","last_became_peered":"2024-02-19T14:57:54.822177+0000","last_unstale":"2024-02-19T14:57:55.823209+0000","last_undegraded":"2024-02-19T14:57:55.823209+0000","last_fullsized":"2024-02-19T14:57:55.823209+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:58:21.065898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819635+0000","last_change":"2024-02-19T14:57:53.816563+0000","last_active":"2024-02-19T14:57:55.819635+0000","last_peered":"2024-02-19T14:57:55.819635+0000","last_clean":"2024-02-19T14:57:55.819635+0000","last_became_active":"2024-02-19T14:57:53.816303+0000","last_became_peered":"2024-02-19T14:57:53.816303+0000","last_unstale":"2024-02-19T14:57:55.819635+0000","last_undegraded":"2024-02-19T14:57:55.819635+0000","last_fullsized":"2024-02-19T14:57:55.819635+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:33.963032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823479+0000","last_change":"2024-02-19T14:57:45.783960+0000","last_active":"2024-02-19T14:57:55.823479+0000","last_peered":"2024-02-19T14:57:55.823479+0000","last_clean":"2024-02-19T14:57:55.823479+0000","last_became_active":"2024-02-19T14:57:45.783704+0000","last_became_peered":"2024-02-19T14:57:45.783704+0000","last_unstale":"2024-02-19T14:57:55.823479+0000","last_undegraded":"2024-02-19T14:57:55.823479+0000","last_fullsized":"2024-02-19T14:57:55.823479+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:55:24.171999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"151'8","reported_seq":30,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825476+0000","last_change":"2024-02-19T14:57:44.758621+0000","last_active":"2024-02-19T14:57:55.825476+0000","last_peered":"2024-02-19T14:57:55.825476+0000","last_clean":"2024-02-19T14:57:55.825476+0000","last_became_active":"2024-02-19T14:57:44.758296+0000","last_became_peered":"2024-02-19T14:57:44.758296+0000","last_unstale":"2024-02-19T14:57:55.825476+0000","last_undegraded":"2024-02-19T14:57:55.825476+0000","last_fullsized":"2024-02-19T14:57:55.825476+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826140+0000","last_change":"2024-02-19T14:57:54.819816+0000","last_active":"2024-02-19T14:57:55.826140+0000","last_peered":"2024-02-19T14:57:55.826140+0000","last_clean":"2024-02-19T14:57:55.826140+0000","last_became_active":"2024-02-19T14:57:54.819239+0000","last_became_peered":"2024-02-19T14:57:54.819239+0000","last_unstale":"2024-02-19T14:57:55.826140+0000","last_undegraded":"2024-02-19T14:57:55.826140+0000","last_fullsized":"2024-02-19T14:57:55.826140+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:52.936596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822856+0000","last_change":"2024-02-19T14:57:53.822905+0000","last_active":"2024-02-19T14:57:55.822856+0000","last_peered":"2024-02-19T14:57:55.822856+0000","last_clean":"2024-02-19T14:57:55.822856+0000","last_became_active":"2024-02-19T14:57:53.822660+0000","last_became_peered":"2024-02-19T14:57:53.822660+0000","last_unstale":"2024-02-19T14:57:55.822856+0000","last_undegraded":"2024-02-19T14:57:55.822856+0000","last_fullsized":"2024-02-19T14:57:55.822856+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:03.185226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825281+0000","last_change":"2024-02-19T14:57:45.773816+0000","last_active":"2024-02-19T14:57:55.825281+0000","last_peered":"2024-02-19T14:57:55.825281+0000","last_clean":"2024-02-19T14:57:55.825281+0000","last_became_active":"2024-02-19T14:57:45.773272+0000","last_became_peered":"2024-02-19T14:57:45.773272+0000","last_unstale":"2024-02-19T14:57:55.825281+0000","last_undegraded":"2024-02-19T14:57:55.825281+0000","last_fullsized":"2024-02-19T14:57:55.825281+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:26:52.328924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819782+0000","last_change":"2024-02-19T14:57:44.751265+0000","last_active":"2024-02-19T14:57:55.819782+0000","last_peered":"2024-02-19T14:57:55.819782+0000","last_clean":"2024-02-19T14:57:55.819782+0000","last_became_active":"2024-02-19T14:57:44.750991+0000","last_became_peered":"2024-02-19T14:57:44.750991+0000","last_unstale":"2024-02-19T14:57:55.819782+0000","last_undegraded":"2024-02-19T14:57:55.819782+0000","last_fullsized":"2024-02-19T14:57:55.819782+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:32.644808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824782+0000","last_change":"2024-02-19T14:57:54.829897+0000","last_active":"2024-02-19T14:57:55.824782+0000","last_peered":"2024-02-19T14:57:55.824782+0000","last_clean":"2024-02-19T14:57:55.824782+0000","last_became_active":"2024-02-19T14:57:54.829631+0000","last_became_peered":"2024-02-19T14:57:54.829631+0000","last_unstale":"2024-02-19T14:57:55.824782+0000","last_undegraded":"2024-02-19T14:57:55.824782+0000","last_fullsized":"2024-02-19T14:57:55.824782+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:04:37.441462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994528+0000","last_change":"2024-02-19T14:57:53.827779+0000","last_active":"2024-02-19T14:57:56.994528+0000","last_peered":"2024-02-19T14:57:56.994528+0000","last_clean":"2024-02-19T14:57:56.994528+0000","last_became_active":"2024-02-19T14:57:53.827475+0000","last_became_peered":"2024-02-19T14:57:53.827475+0000","last_unstale":"2024-02-19T14:57:56.994528+0000","last_undegraded":"2024-02-19T14:57:56.994528+0000","last_fullsized":"2024-02-19T14:57:56.994528+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:54.429397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994968+0000","last_change":"2024-02-19T14:57:45.773622+0000","last_active":"2024-02-19T14:57:56.994968+0000","last_peered":"2024-02-19T14:57:56.994968+0000","last_clean":"2024-02-19T14:57:56.994968+0000","last_became_active":"2024-02-19T14:57:45.773349+0000","last_became_peered":"2024-02-19T14:57:45.773349+0000","last_unstale":"2024-02-19T14:57:56.994968+0000","last_undegraded":"2024-02-19T14:57:56.994968+0000","last_fullsized":"2024-02-19T14:57:56.994968+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:33:01.576910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826096+0000","last_change":"2024-02-19T14:57:44.757425+0000","last_active":"2024-02-19T14:57:55.826096+0000","last_peered":"2024-02-19T14:57:55.826096+0000","last_clean":"2024-02-19T14:57:55.826096+0000","last_became_active":"2024-02-19T14:57:44.757006+0000","last_became_peered":"2024-02-19T14:57:44.757006+0000","last_unstale":"2024-02-19T14:57:55.826096+0000","last_undegraded":"2024-02-19T14:57:55.826096+0000","last_fullsized":"2024-02-19T14:57:55.826096+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:18:41.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995458+0000","last_change":"2024-02-19T14:57:54.821943+0000","last_active":"2024-02-19T14:57:56.995458+0000","last_peered":"2024-02-19T14:57:56.995458+0000","last_clean":"2024-02-19T14:57:56.995458+0000","last_became_active":"2024-02-19T14:57:54.821645+0000","last_became_peered":"2024-02-19T14:57:54.821645+0000","last_unstale":"2024-02-19T14:57:56.995458+0000","last_undegraded":"2024-02-19T14:57:56.995458+0000","last_fullsized":"2024-02-19T14:57:56.995458+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:19:42.150099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824836+0000","last_change":"2024-02-19T14:57:53.825554+0000","last_active":"2024-02-19T14:57:55.824836+0000","last_peered":"2024-02-19T14:57:55.824836+0000","last_clean":"2024-02-19T14:57:55.824836+0000","last_became_active":"2024-02-19T14:57:53.825279+0000","last_became_peered":"2024-02-19T14:57:53.825279+0000","last_unstale":"2024-02-19T14:57:55.824836+0000","last_undegraded":"2024-02-19T14:57:55.824836+0000","last_fullsized":"2024-02-19T14:57:55.824836+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:02.674757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995344+0000","last_change":"2024-02-19T14:57:45.781720+0000","last_active":"2024-02-19T14:57:56.995344+0000","last_peered":"2024-02-19T14:57:56.995344+0000","last_clean":"2024-02-19T14:57:56.995344+0000","last_became_active":"2024-02-19T14:57:45.781425+0000","last_became_peered":"2024-02-19T14:57:45.781425+0000","last_unstale":"2024-02-19T14:57:56.995344+0000","last_undegraded":"2024-02-19T14:57:56.995344+0000","last_fullsized":"2024-02-19T14:57:56.995344+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:36.396204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"151'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994945+0000","last_change":"2024-02-19T14:57:44.758828+0000","last_active":"2024-02-19T14:57:56.994945+0000","last_peered":"2024-02-19T14:57:56.994945+0000","last_clean":"2024-02-19T14:57:56.994945+0000","last_became_active":"2024-02-19T14:57:44.758387+0000","last_became_peered":"2024-02-19T14:57:44.758387+0000","last_unstale":"2024-02-19T14:57:56.994945+0000","last_undegraded":"2024-02-19T14:57:56.994945+0000","last_fullsized":"2024-02-19T14:57:56.994945+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826104+0000","last_change":"2024-02-19T14:57:54.821235+0000","last_active":"2024-02-19T14:57:55.826104+0000","last_peered":"2024-02-19T14:57:55.826104+0000","last_clean":"2024-02-19T14:57:55.826104+0000","last_became_active":"2024-02-19T14:57:54.820937+0000","last_became_peered":"2024-02-19T14:57:54.820937+0000","last_unstale":"2024-02-19T14:57:55.826104+0000","last_undegraded":"2024-02-19T14:57:55.826104+0000","last_fullsized":"2024-02-19T14:57:55.826104+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:09.622659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.824459+0000","last_change":"2024-02-19T14:57:53.822411+0000","last_active":"2024-02-19T14:57:55.824459+0000","last_peered":"2024-02-19T14:57:55.824459+0000","last_clean":"2024-02-19T14:57:55.824459+0000","last_became_active":"2024-02-19T14:57:53.822167+0000","last_became_peered":"2024-02-19T14:57:53.822167+0000","last_unstale":"2024-02-19T14:57:55.824459+0000","last_undegraded":"2024-02-19T14:57:55.824459+0000","last_fullsized":"2024-02-19T14:57:55.824459+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:10.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995676+0000","last_change":"2024-02-19T14:57:45.780279+0000","last_active":"2024-02-19T14:57:56.995676+0000","last_peered":"2024-02-19T14:57:56.995676+0000","last_clean":"2024-02-19T14:57:56.995676+0000","last_became_active":"2024-02-19T14:57:45.779989+0000","last_became_peered":"2024-02-19T14:57:45.779989+0000","last_unstale":"2024-02-19T14:57:56.995676+0000","last_undegraded":"2024-02-19T14:57:56.995676+0000","last_fullsized":"2024-02-19T14:57:56.995676+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:49:19.939046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825920+0000","last_change":"2024-02-19T14:57:44.760014+0000","last_active":"2024-02-19T14:57:55.825920+0000","last_peered":"2024-02-19T14:57:55.825920+0000","last_clean":"2024-02-19T14:57:55.825920+0000","last_became_active":"2024-02-19T14:57:44.759749+0000","last_became_peered":"2024-02-19T14:57:44.759749+0000","last_unstale":"2024-02-19T14:57:55.825920+0000","last_undegraded":"2024-02-19T14:57:55.825920+0000","last_fullsized":"2024-02-19T14:57:55.825920+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:21:29.583449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826203+0000","last_change":"2024-02-19T14:57:54.830962+0000","last_active":"2024-02-19T14:57:55.826203+0000","last_peered":"2024-02-19T14:57:55.826203+0000","last_clean":"2024-02-19T14:57:55.826203+0000","last_became_active":"2024-02-19T14:57:54.830624+0000","last_became_peered":"2024-02-19T14:57:54.830624+0000","last_unstale":"2024-02-19T14:57:55.826203+0000","last_undegraded":"2024-02-19T14:57:55.826203+0000","last_fullsized":"2024-02-19T14:57:55.826203+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:04.023224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819644+0000","last_change":"2024-02-19T14:57:53.822010+0000","last_active":"2024-02-19T14:57:55.819644+0000","last_peered":"2024-02-19T14:57:55.819644+0000","last_clean":"2024-02-19T14:57:55.819644+0000","last_became_active":"2024-02-19T14:57:53.821590+0000","last_became_peered":"2024-02-19T14:57:53.821590+0000","last_unstale":"2024-02-19T14:57:55.819644+0000","last_undegraded":"2024-02-19T14:57:55.819644+0000","last_fullsized":"2024-02-19T14:57:55.819644+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:50.523976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819446+0000","last_change":"2024-02-19T14:57:45.785469+0000","last_active":"2024-02-19T14:57:55.819446+0000","last_peered":"2024-02-19T14:57:55.819446+0000","last_clean":"2024-02-19T14:57:55.819446+0000","last_became_active":"2024-02-19T14:57:45.784759+0000","last_became_peered":"2024-02-19T14:57:45.784759+0000","last_unstale":"2024-02-19T14:57:55.819446+0000","last_undegraded":"2024-02-19T14:57:55.819446+0000","last_fullsized":"2024-02-19T14:57:55.819446+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:26.131093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994854+0000","last_change":"2024-02-19T14:57:44.752558+0000","last_active":"2024-02-19T14:57:56.994854+0000","last_peered":"2024-02-19T14:57:56.994854+0000","last_clean":"2024-02-19T14:57:56.994854+0000","last_became_active":"2024-02-19T14:57:44.752290+0000","last_became_peered":"2024-02-19T14:57:44.752290+0000","last_unstale":"2024-02-19T14:57:56.994854+0000","last_undegraded":"2024-02-19T14:57:56.994854+0000","last_fullsized":"2024-02-19T14:57:56.994854+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:54:55.760619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823459+0000","last_change":"2024-02-19T14:57:54.816689+0000","last_active":"2024-02-19T14:57:55.823459+0000","last_peered":"2024-02-19T14:57:55.823459+0000","last_clean":"2024-02-19T14:57:55.823459+0000","last_became_active":"2024-02-19T14:57:54.816409+0000","last_became_peered":"2024-02-19T14:57:54.816409+0000","last_unstale":"2024-02-19T14:57:55.823459+0000","last_undegraded":"2024-02-19T14:57:55.823459+0000","last_fullsized":"2024-02-19T14:57:55.823459+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:14.043959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819612+0000","last_change":"2024-02-19T14:57:53.820777+0000","last_active":"2024-02-19T14:57:55.819612+0000","last_peered":"2024-02-19T14:57:55.819612+0000","last_clean":"2024-02-19T14:57:55.819612+0000","last_became_active":"2024-02-19T14:57:53.820522+0000","last_became_peered":"2024-02-19T14:57:53.820522+0000","last_unstale":"2024-02-19T14:57:55.819612+0000","last_undegraded":"2024-02-19T14:57:55.819612+0000","last_fullsized":"2024-02-19T14:57:55.819612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:06:21.215428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995035+0000","last_change":"2024-02-19T14:57:45.776484+0000","last_active":"2024-02-19T14:57:56.995035+0000","last_peered":"2024-02-19T14:57:56.995035+0000","last_clean":"2024-02-19T14:57:56.995035+0000","last_became_active":"2024-02-19T14:57:45.776095+0000","last_became_peered":"2024-02-19T14:57:45.776095+0000","last_unstale":"2024-02-19T14:57:56.995035+0000","last_undegraded":"2024-02-19T14:57:56.995035+0000","last_fullsized":"2024-02-19T14:57:56.995035+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:12.903898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995639+0000","last_change":"2024-02-19T14:57:44.746247+0000","last_active":"2024-02-19T14:57:56.995639+0000","last_peered":"2024-02-19T14:57:56.995639+0000","last_clean":"2024-02-19T14:57:56.995639+0000","last_became_active":"2024-02-19T14:57:44.745941+0000","last_became_peered":"2024-02-19T14:57:44.745941+0000","last_unstale":"2024-02-19T14:57:56.995639+0000","last_undegraded":"2024-02-19T14:57:56.995639+0000","last_fullsized":"2024-02-19T14:57:56.995639+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:03.540170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823243+0000","last_change":"2024-02-19T14:57:54.818513+0000","last_active":"2024-02-19T14:57:55.823243+0000","last_peered":"2024-02-19T14:57:55.823243+0000","last_clean":"2024-02-19T14:57:55.823243+0000","last_became_active":"2024-02-19T14:57:54.818259+0000","last_became_peered":"2024-02-19T14:57:54.818259+0000","last_unstale":"2024-02-19T14:57:55.823243+0000","last_undegraded":"2024-02-19T14:57:55.823243+0000","last_fullsized":"2024-02-19T14:57:55.823243+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:35.114893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995162+0000","last_change":"2024-02-19T14:57:53.817199+0000","last_active":"2024-02-19T14:57:56.995162+0000","last_peered":"2024-02-19T14:57:56.995162+0000","last_clean":"2024-02-19T14:57:56.995162+0000","last_became_active":"2024-02-19T14:57:53.816930+0000","last_became_peered":"2024-02-19T14:57:53.816930+0000","last_unstale":"2024-02-19T14:57:56.995162+0000","last_undegraded":"2024-02-19T14:57:56.995162+0000","last_fullsized":"2024-02-19T14:57:56.995162+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:56.199868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994948+0000","last_change":"2024-02-19T14:57:45.766067+0000","last_active":"2024-02-19T14:57:56.994948+0000","last_peered":"2024-02-19T14:57:56.994948+0000","last_clean":"2024-02-19T14:57:56.994948+0000","last_became_active":"2024-02-19T14:57:45.765654+0000","last_became_peered":"2024-02-19T14:57:45.765654+0000","last_unstale":"2024-02-19T14:57:56.994948+0000","last_undegraded":"2024-02-19T14:57:56.994948+0000","last_fullsized":"2024-02-19T14:57:56.994948+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:36.058926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823239+0000","last_change":"2024-02-19T14:57:44.751718+0000","last_active":"2024-02-19T14:57:55.823239+0000","last_peered":"2024-02-19T14:57:55.823239+0000","last_clean":"2024-02-19T14:57:55.823239+0000","last_became_active":"2024-02-19T14:57:44.751403+0000","last_became_peered":"2024-02-19T14:57:44.751403+0000","last_unstale":"2024-02-19T14:57:55.823239+0000","last_undegraded":"2024-02-19T14:57:55.823239+0000","last_fullsized":"2024-02-19T14:57:55.823239+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:59.486599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825997+0000","last_change":"2024-02-19T14:57:54.814429+0000","last_active":"2024-02-19T14:57:55.825997+0000","last_peered":"2024-02-19T14:57:55.825997+0000","last_clean":"2024-02-19T14:57:55.825997+0000","last_became_active":"2024-02-19T14:57:54.814168+0000","last_became_peered":"2024-02-19T14:57:54.814168+0000","last_unstale":"2024-02-19T14:57:55.825997+0000","last_undegraded":"2024-02-19T14:57:55.825997+0000","last_fullsized":"2024-02-19T14:57:55.825997+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:48:34.967528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"156'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823515+0000","last_change":"2024-02-19T14:57:53.823400+0000","last_active":"2024-02-19T14:57:55.823515+0000","last_peered":"2024-02-19T14:57:55.823515+0000","last_clean":"2024-02-19T14:57:55.823515+0000","last_became_active":"2024-02-19T14:57:53.823152+0000","last_became_peered":"2024-02-19T14:57:53.823152+0000","last_unstale":"2024-02-19T14:57:55.823515+0000","last_undegraded":"2024-02-19T14:57:55.823515+0000","last_fullsized":"2024-02-19T14:57:55.823515+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:11:59.510070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826684+0000","last_change":"2024-02-19T14:57:45.775690+0000","last_active":"2024-02-19T14:57:55.826684+0000","last_peered":"2024-02-19T14:57:55.826684+0000","last_clean":"2024-02-19T14:57:55.826684+0000","last_became_active":"2024-02-19T14:57:45.775076+0000","last_became_peered":"2024-02-19T14:57:45.775076+0000","last_unstale":"2024-02-19T14:57:55.826684+0000","last_undegraded":"2024-02-19T14:57:55.826684+0000","last_fullsized":"2024-02-19T14:57:55.826684+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:28:40.949417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826056+0000","last_change":"2024-02-19T14:57:44.751791+0000","last_active":"2024-02-19T14:57:55.826056+0000","last_peered":"2024-02-19T14:57:55.826056+0000","last_clean":"2024-02-19T14:57:55.826056+0000","last_became_active":"2024-02-19T14:57:44.751480+0000","last_became_peered":"2024-02-19T14:57:44.751480+0000","last_unstale":"2024-02-19T14:57:55.826056+0000","last_undegraded":"2024-02-19T14:57:55.826056+0000","last_fullsized":"2024-02-19T14:57:55.826056+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:17:54.459651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994642+0000","last_change":"2024-02-19T14:57:54.820463+0000","last_active":"2024-02-19T14:57:56.994642+0000","last_peered":"2024-02-19T14:57:56.994642+0000","last_clean":"2024-02-19T14:57:56.994642+0000","last_became_active":"2024-02-19T14:57:54.820202+0000","last_became_peered":"2024-02-19T14:57:54.820202+0000","last_unstale":"2024-02-19T14:57:56.994642+0000","last_undegraded":"2024-02-19T14:57:56.994642+0000","last_fullsized":"2024-02-19T14:57:56.994642+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:40.602087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.822816+0000","last_change":"2024-02-19T14:57:53.822348+0000","last_active":"2024-02-19T14:57:55.822816+0000","last_peered":"2024-02-19T14:57:55.822816+0000","last_clean":"2024-02-19T14:57:55.822816+0000","last_became_active":"2024-02-19T14:57:53.822101+0000","last_became_peered":"2024-02-19T14:57:53.822101+0000","last_unstale":"2024-02-19T14:57:55.822816+0000","last_undegraded":"2024-02-19T14:57:55.822816+0000","last_fullsized":"2024-02-19T14:57:55.822816+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:15:15.938236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995545+0000","last_change":"2024-02-19T14:57:45.772415+0000","last_active":"2024-02-19T14:57:56.995545+0000","last_peered":"2024-02-19T14:57:56.995545+0000","last_clean":"2024-02-19T14:57:56.995545+0000","last_became_active":"2024-02-19T14:57:45.771919+0000","last_became_peered":"2024-02-19T14:57:45.771919+0000","last_unstale":"2024-02-19T14:57:56.995545+0000","last_undegraded":"2024-02-19T14:57:56.995545+0000","last_fullsized":"2024-02-19T14:57:56.995545+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:29.983600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"151'2","reported_seq":24,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995264+0000","last_change":"2024-02-19T14:57:44.758252+0000","last_active":"2024-02-19T14:57:56.995264+0000","last_peered":"2024-02-19T14:57:56.995264+0000","last_clean":"2024-02-19T14:57:56.995264+0000","last_became_active":"2024-02-19T14:57:44.757979+0000","last_became_peered":"2024-02-19T14:57:44.757979+0000","last_unstale":"2024-02-19T14:57:56.995264+0000","last_undegraded":"2024-02-19T14:57:56.995264+0000","last_fullsized":"2024-02-19T14:57:56.995264+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:07:28.981816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819601+0000","last_change":"2024-02-19T14:57:54.821187+0000","last_active":"2024-02-19T14:57:55.819601+0000","last_peered":"2024-02-19T14:57:55.819601+0000","last_clean":"2024-02-19T14:57:55.819601+0000","last_became_active":"2024-02-19T14:57:54.820886+0000","last_became_peered":"2024-02-19T14:57:54.820886+0000","last_unstale":"2024-02-19T14:57:55.819601+0000","last_undegraded":"2024-02-19T14:57:55.819601+0000","last_fullsized":"2024-02-19T14:57:55.819601+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:07:55.760657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994793+0000","last_change":"2024-02-19T14:57:53.825683+0000","last_active":"2024-02-19T14:57:56.994793+0000","last_peered":"2024-02-19T14:57:56.994793+0000","last_clean":"2024-02-19T14:57:56.994793+0000","last_became_active":"2024-02-19T14:57:53.825366+0000","last_became_peered":"2024-02-19T14:57:53.825366+0000","last_unstale":"2024-02-19T14:57:56.994793+0000","last_undegraded":"2024-02-19T14:57:56.994793+0000","last_fullsized":"2024-02-19T14:57:56.994793+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:03:56.803585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825534+0000","last_change":"2024-02-19T14:57:45.772751+0000","last_active":"2024-02-19T14:57:55.825534+0000","last_peered":"2024-02-19T14:57:55.825534+0000","last_clean":"2024-02-19T14:57:55.825534+0000","last_became_active":"2024-02-19T14:57:45.772506+0000","last_became_peered":"2024-02-19T14:57:45.772506+0000","last_unstale":"2024-02-19T14:57:55.825534+0000","last_undegraded":"2024-02-19T14:57:55.825534+0000","last_fullsized":"2024-02-19T14:57:55.825534+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:15.952704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"152'1","reported_seq":23,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819925+0000","last_change":"2024-02-19T14:57:44.756795+0000","last_active":"2024-02-19T14:57:55.819925+0000","last_peered":"2024-02-19T14:57:55.819925+0000","last_clean":"2024-02-19T14:57:55.819925+0000","last_became_active":"2024-02-19T14:57:44.756166+0000","last_became_peered":"2024-02-19T14:57:44.756166+0000","last_unstale":"2024-02-19T14:57:55.819925+0000","last_undegraded":"2024-02-19T14:57:55.819925+0000","last_fullsized":"2024-02-19T14:57:55.819925+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:13.014764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825221+0000","last_change":"2024-02-19T14:57:54.826892+0000","last_active":"2024-02-19T14:57:55.825221+0000","last_peered":"2024-02-19T14:57:55.825221+0000","last_clean":"2024-02-19T14:57:55.825221+0000","last_became_active":"2024-02-19T14:57:54.826378+0000","last_became_peered":"2024-02-19T14:57:54.826378+0000","last_unstale":"2024-02-19T14:57:55.825221+0000","last_undegraded":"2024-02-19T14:57:55.825221+0000","last_fullsized":"2024-02-19T14:57:55.825221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:13:18.218308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826291+0000","last_change":"2024-02-19T14:57:53.819028+0000","last_active":"2024-02-19T14:57:55.826291+0000","last_peered":"2024-02-19T14:57:55.826291+0000","last_clean":"2024-02-19T14:57:55.826291+0000","last_became_active":"2024-02-19T14:57:53.818598+0000","last_became_peered":"2024-02-19T14:57:53.818598+0000","last_unstale":"2024-02-19T14:57:55.826291+0000","last_undegraded":"2024-02-19T14:57:55.826291+0000","last_fullsized":"2024-02-19T14:57:55.826291+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:54:30.909529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823038+0000","last_change":"2024-02-19T14:57:45.774855+0000","last_active":"2024-02-19T14:57:55.823038+0000","last_peered":"2024-02-19T14:57:55.823038+0000","last_clean":"2024-02-19T14:57:55.823038+0000","last_became_active":"2024-02-19T14:57:45.774572+0000","last_became_peered":"2024-02-19T14:57:45.774572+0000","last_unstale":"2024-02-19T14:57:55.823038+0000","last_undegraded":"2024-02-19T14:57:55.823038+0000","last_fullsized":"2024-02-19T14:57:55.823038+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:32:34.308217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819830+0000","last_change":"2024-02-19T14:57:44.749436+0000","last_active":"2024-02-19T14:57:55.819830+0000","last_peered":"2024-02-19T14:57:55.819830+0000","last_clean":"2024-02-19T14:57:55.819830+0000","last_became_active":"2024-02-19T14:57:44.749153+0000","last_became_peered":"2024-02-19T14:57:44.749153+0000","last_unstale":"2024-02-19T14:57:55.819830+0000","last_undegraded":"2024-02-19T14:57:55.819830+0000","last_fullsized":"2024-02-19T14:57:55.819830+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:01:36.657057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994434+0000","last_change":"2024-02-19T14:57:54.820331+0000","last_active":"2024-02-19T14:57:56.994434+0000","last_peered":"2024-02-19T14:57:56.994434+0000","last_clean":"2024-02-19T14:57:56.994434+0000","last_became_active":"2024-02-19T14:57:54.820073+0000","last_became_peered":"2024-02-19T14:57:54.820073+0000","last_unstale":"2024-02-19T14:57:56.994434+0000","last_undegraded":"2024-02-19T14:57:56.994434+0000","last_fullsized":"2024-02-19T14:57:56.994434+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:45:35.195077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994454+0000","last_change":"2024-02-19T14:57:53.827547+0000","last_active":"2024-02-19T14:57:56.994454+0000","last_peered":"2024-02-19T14:57:56.994454+0000","last_clean":"2024-02-19T14:57:56.994454+0000","last_became_active":"2024-02-19T14:57:53.827249+0000","last_became_peered":"2024-02-19T14:57:53.827249+0000","last_unstale":"2024-02-19T14:57:56.994454+0000","last_undegraded":"2024-02-19T14:57:56.994454+0000","last_fullsized":"2024-02-19T14:57:56.994454+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:37:50.645573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825814+0000","last_change":"2024-02-19T14:57:45.772279+0000","last_active":"2024-02-19T14:57:55.825814+0000","last_peered":"2024-02-19T14:57:55.825814+0000","last_clean":"2024-02-19T14:57:55.825814+0000","last_became_active":"2024-02-19T14:57:45.771983+0000","last_became_peered":"2024-02-19T14:57:45.771983+0000","last_unstale":"2024-02-19T14:57:55.825814+0000","last_undegraded":"2024-02-19T14:57:55.825814+0000","last_fullsized":"2024-02-19T14:57:55.825814+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:03.956395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823438+0000","last_change":"2024-02-19T14:57:44.758176+0000","last_active":"2024-02-19T14:57:55.823438+0000","last_peered":"2024-02-19T14:57:55.823438+0000","last_clean":"2024-02-19T14:57:55.823438+0000","last_became_active":"2024-02-19T14:57:44.757912+0000","last_became_peered":"2024-02-19T14:57:44.757912+0000","last_unstale":"2024-02-19T14:57:55.823438+0000","last_undegraded":"2024-02-19T14:57:55.823438+0000","last_fullsized":"2024-02-19T14:57:55.823438+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:26.671302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995931+0000","last_change":"2024-02-19T14:57:54.832738+0000","last_active":"2024-02-19T14:57:56.995931+0000","last_peered":"2024-02-19T14:57:56.995931+0000","last_clean":"2024-02-19T14:57:56.995931+0000","last_became_active":"2024-02-19T14:57:54.832010+0000","last_became_peered":"2024-02-19T14:57:54.832010+0000","last_unstale":"2024-02-19T14:57:56.995931+0000","last_undegraded":"2024-02-19T14:57:56.995931+0000","last_fullsized":"2024-02-19T14:57:56.995931+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:55:55.179220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995159+0000","last_change":"2024-02-19T14:57:53.821749+0000","last_active":"2024-02-19T14:57:56.995159+0000","last_peered":"2024-02-19T14:57:56.995159+0000","last_clean":"2024-02-19T14:57:56.995159+0000","last_became_active":"2024-02-19T14:57:53.821351+0000","last_became_peered":"2024-02-19T14:57:53.821351+0000","last_unstale":"2024-02-19T14:57:56.995159+0000","last_undegraded":"2024-02-19T14:57:56.995159+0000","last_fullsized":"2024-02-19T14:57:56.995159+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:11:54.055035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826152+0000","last_change":"2024-02-19T14:57:45.771150+0000","last_active":"2024-02-19T14:57:55.826152+0000","last_peered":"2024-02-19T14:57:55.826152+0000","last_clean":"2024-02-19T14:57:55.826152+0000","last_became_active":"2024-02-19T14:57:45.770740+0000","last_became_peered":"2024-02-19T14:57:45.770740+0000","last_unstale":"2024-02-19T14:57:55.826152+0000","last_undegraded":"2024-02-19T14:57:55.826152+0000","last_fullsized":"2024-02-19T14:57:55.826152+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:57.456060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826504+0000","last_change":"2024-02-19T14:57:44.752321+0000","last_active":"2024-02-19T14:57:55.826504+0000","last_peered":"2024-02-19T14:57:55.826504+0000","last_clean":"2024-02-19T14:57:55.826504+0000","last_became_active":"2024-02-19T14:57:44.752031+0000","last_became_peered":"2024-02-19T14:57:44.752031+0000","last_unstale":"2024-02-19T14:57:55.826504+0000","last_undegraded":"2024-02-19T14:57:55.826504+0000","last_fullsized":"2024-02-19T14:57:55.826504+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:45:54.123952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826232+0000","last_change":"2024-02-19T14:57:54.826939+0000","last_active":"2024-02-19T14:57:55.826232+0000","last_peered":"2024-02-19T14:57:55.826232+0000","last_clean":"2024-02-19T14:57:55.826232+0000","last_became_active":"2024-02-19T14:57:54.826612+0000","last_became_peered":"2024-02-19T14:57:54.826612+0000","last_unstale":"2024-02-19T14:57:55.826232+0000","last_undegraded":"2024-02-19T14:57:55.826232+0000","last_fullsized":"2024-02-19T14:57:55.826232+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:51.528471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995700+0000","last_change":"2024-02-19T14:57:53.816870+0000","last_active":"2024-02-19T14:57:56.995700+0000","last_peered":"2024-02-19T14:57:56.995700+0000","last_clean":"2024-02-19T14:57:56.995700+0000","last_became_active":"2024-02-19T14:57:53.816516+0000","last_became_peered":"2024-02-19T14:57:53.816516+0000","last_unstale":"2024-02-19T14:57:56.995700+0000","last_undegraded":"2024-02-19T14:57:56.995700+0000","last_fullsized":"2024-02-19T14:57:56.995700+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:41.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825830+0000","last_change":"2024-02-19T14:57:45.784058+0000","last_active":"2024-02-19T14:57:55.825830+0000","last_peered":"2024-02-19T14:57:55.825830+0000","last_clean":"2024-02-19T14:57:55.825830+0000","last_became_active":"2024-02-19T14:57:45.783772+0000","last_became_peered":"2024-02-19T14:57:45.783772+0000","last_unstale":"2024-02-19T14:57:55.825830+0000","last_undegraded":"2024-02-19T14:57:55.825830+0000","last_fullsized":"2024-02-19T14:57:55.825830+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:34.823875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819175+0000","last_change":"2024-02-19T14:57:44.743222+0000","last_active":"2024-02-19T14:57:55.819175+0000","last_peered":"2024-02-19T14:57:55.819175+0000","last_clean":"2024-02-19T14:57:55.819175+0000","last_became_active":"2024-02-19T14:57:44.742946+0000","last_became_peered":"2024-02-19T14:57:44.742946+0000","last_unstale":"2024-02-19T14:57:55.819175+0000","last_undegraded":"2024-02-19T14:57:55.819175+0000","last_fullsized":"2024-02-19T14:57:55.819175+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:22.509070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823232+0000","last_change":"2024-02-19T14:57:54.827418+0000","last_active":"2024-02-19T14:57:55.823232+0000","last_peered":"2024-02-19T14:57:55.823232+0000","last_clean":"2024-02-19T14:57:55.823232+0000","last_became_active":"2024-02-19T14:57:54.827167+0000","last_became_peered":"2024-02-19T14:57:54.827167+0000","last_unstale":"2024-02-19T14:57:55.823232+0000","last_undegraded":"2024-02-19T14:57:55.823232+0000","last_fullsized":"2024-02-19T14:57:55.823232+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:32.315419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"155'1","reported_seq":15,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995603+0000","last_change":"2024-02-19T14:57:53.817116+0000","last_active":"2024-02-19T14:57:56.995603+0000","last_peered":"2024-02-19T14:57:56.995603+0000","last_clean":"2024-02-19T14:57:56.995603+0000","last_became_active":"2024-02-19T14:57:53.816835+0000","last_became_peered":"2024-02-19T14:57:53.816835+0000","last_unstale":"2024-02-19T14:57:56.995603+0000","last_undegraded":"2024-02-19T14:57:56.995603+0000","last_fullsized":"2024-02-19T14:57:56.995603+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:42.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819737+0000","last_change":"2024-02-19T14:57:45.783349+0000","last_active":"2024-02-19T14:57:55.819737+0000","last_peered":"2024-02-19T14:57:55.819737+0000","last_clean":"2024-02-19T14:57:55.819737+0000","last_became_active":"2024-02-19T14:57:45.783101+0000","last_became_peered":"2024-02-19T14:57:45.783101+0000","last_unstale":"2024-02-19T14:57:55.819737+0000","last_undegraded":"2024-02-19T14:57:55.819737+0000","last_fullsized":"2024-02-19T14:57:55.819737+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:13:09.154323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995074+0000","last_change":"2024-02-19T14:57:44.747418+0000","last_active":"2024-02-19T14:57:56.995074+0000","last_peered":"2024-02-19T14:57:56.995074+0000","last_clean":"2024-02-19T14:57:56.995074+0000","last_became_active":"2024-02-19T14:57:44.746915+0000","last_became_peered":"2024-02-19T14:57:44.746915+0000","last_unstale":"2024-02-19T14:57:56.995074+0000","last_undegraded":"2024-02-19T14:57:56.995074+0000","last_fullsized":"2024-02-19T14:57:56.995074+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:09:58.929760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819537+0000","last_change":"2024-02-19T14:57:54.829645+0000","last_active":"2024-02-19T14:57:55.819537+0000","last_peered":"2024-02-19T14:57:55.819537+0000","last_clean":"2024-02-19T14:57:55.819537+0000","last_became_active":"2024-02-19T14:57:54.828869+0000","last_became_peered":"2024-02-19T14:57:54.828869+0000","last_unstale":"2024-02-19T14:57:55.819537+0000","last_undegraded":"2024-02-19T14:57:55.819537+0000","last_fullsized":"2024-02-19T14:57:55.819537+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:06.710733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"155'2","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995024+0000","last_change":"2024-02-19T14:57:53.820424+0000","last_active":"2024-02-19T14:57:56.995024+0000","last_peered":"2024-02-19T14:57:56.995024+0000","last_clean":"2024-02-19T14:57:56.995024+0000","last_became_active":"2024-02-19T14:57:53.820147+0000","last_became_peered":"2024-02-19T14:57:53.820147+0000","last_unstale":"2024-02-19T14:57:56.995024+0000","last_undegraded":"2024-02-19T14:57:56.995024+0000","last_fullsized":"2024-02-19T14:57:56.995024+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:10.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.823505+0000","last_change":"2024-02-19T14:57:45.774106+0000","last_active":"2024-02-19T14:57:55.823505+0000","last_peered":"2024-02-19T14:57:55.823505+0000","last_clean":"2024-02-19T14:57:55.823505+0000","last_became_active":"2024-02-19T14:57:45.773847+0000","last_became_peered":"2024-02-19T14:57:45.773847+0000","last_unstale":"2024-02-19T14:57:55.823505+0000","last_undegraded":"2024-02-19T14:57:55.823505+0000","last_fullsized":"2024-02-19T14:57:55.823505+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:46:22.673960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994267+0000","last_change":"2024-02-19T14:57:44.752893+0000","last_active":"2024-02-19T14:57:56.994267+0000","last_peered":"2024-02-19T14:57:56.994267+0000","last_clean":"2024-02-19T14:57:56.994267+0000","last_became_active":"2024-02-19T14:57:44.752617+0000","last_became_peered":"2024-02-19T14:57:44.752617+0000","last_unstale":"2024-02-19T14:57:56.994267+0000","last_undegraded":"2024-02-19T14:57:56.994267+0000","last_fullsized":"2024-02-19T14:57:56.994267+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:45.368123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.995022+0000","last_change":"2024-02-19T14:57:54.828418+0000","last_active":"2024-02-19T14:57:56.995022+0000","last_peered":"2024-02-19T14:57:56.995022+0000","last_clean":"2024-02-19T14:57:56.995022+0000","last_became_active":"2024-02-19T14:57:54.828147+0000","last_became_peered":"2024-02-19T14:57:54.828147+0000","last_unstale":"2024-02-19T14:57:56.995022+0000","last_undegraded":"2024-02-19T14:57:56.995022+0000","last_fullsized":"2024-02-19T14:57:56.995022+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:53.797205+0000","last_clean_scrub_stamp":"2024-02-19T14:57:53.797205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:25:45.460188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:56.994274+0000","last_change":"2024-02-19T14:57:53.823527+0000","last_active":"2024-02-19T14:57:56.994274+0000","last_peered":"2024-02-19T14:57:56.994274+0000","last_clean":"2024-02-19T14:57:56.994274+0000","last_became_active":"2024-02-19T14:57:53.823257+0000","last_became_peered":"2024-02-19T14:57:53.823257+0000","last_unstale":"2024-02-19T14:57:56.994274+0000","last_undegraded":"2024-02-19T14:57:56.994274+0000","last_fullsized":"2024-02-19T14:57:56.994274+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:52.792710+0000","last_clean_scrub_stamp":"2024-02-19T14:57:52.792710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:39.435634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.825144+0000","last_change":"2024-02-19T14:57:45.777456+0000","last_active":"2024-02-19T14:57:55.825144+0000","last_peered":"2024-02-19T14:57:55.825144+0000","last_clean":"2024-02-19T14:57:55.825144+0000","last_became_active":"2024-02-19T14:57:45.776998+0000","last_became_peered":"2024-02-19T14:57:45.776998+0000","last_unstale":"2024-02-19T14:57:55.825144+0000","last_undegraded":"2024-02-19T14:57:55.825144+0000","last_fullsized":"2024-02-19T14:57:55.825144+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:44.733221+0000","last_clean_scrub_stamp":"2024-02-19T14:57:44.733221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:14.577140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.819828+0000","last_change":"2024-02-19T14:57:44.757019+0000","last_active":"2024-02-19T14:57:55.819828+0000","last_peered":"2024-02-19T14:57:55.819828+0000","last_clean":"2024-02-19T14:57:55.819828+0000","last_became_active":"2024-02-19T14:57:44.756538+0000","last_became_peered":"2024-02-19T14:57:44.756538+0000","last_unstale":"2024-02-19T14:57:55.819828+0000","last_undegraded":"2024-02-19T14:57:55.819828+0000","last_fullsized":"2024-02-19T14:57:55.819828+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:57:43.731436+0000","last_clean_scrub_stamp":"2024-02-19T14:57:43.731436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:24:33.822090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":426,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-19T14:57:55.826138+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:57:55.826138+0000","last_peered":"2024-02-19T14:57:55.826138+0000","last_clean":"2024-02-19T14:57:55.826138+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:57:55.826138+0000","last_undegraded":"2024-02-19T14:57:55.826138+0000","last_fullsized":"2024-02-19T14:57:55.826138+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33676,"kb_used_data":2496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338164,"statfs":{"total":96636764160,"available":96602279936,"internally_reserved":0,"allocated":2555904,"data_stored":1548845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":0.61599999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.504,"15min":0.496},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60699999999999998,"5min":0.75900000000000001,"15min":0.79500000000000004},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32500000000000001,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"min":{"1min":0.35299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64000000000000001,"5min":0.73999999999999999,"15min":0.74099999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.501},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70799999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.627}]},{"osd":3,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67600000000000005,"5min":0.81399999999999995,"15min":0.81499999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73499999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.81100000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48199999999999998,"15min":0.496},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.68500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.497,"15min":0.53300000000000003},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.75},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51900000000000002,"15min":0.54100000000000004},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55900000000000005,"15min":0.59299999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.88500000000000001},"last":0.83899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":958547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.48699999999999999,"15min":0.46800000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60999999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.49199999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":0.72899999999999998,"15min":0.81999999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69899999999999995,"5min":0.81000000000000005,"15min":0.88100000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.40999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70099999999999996,"5min":0.78100000000000003,"15min":0.90500000000000003},"last":0.502}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.84799999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.504,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.39200000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.52800000000000002},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.76600000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.60199999999999998},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71199999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.58199999999999996}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47699999999999998,"15min":0.53200000000000003},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":0.753,"15min":0.77400000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.53200000000000003,"15min":0.55200000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73499999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.52500000000000002,"15min":0.57299999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.69799999999999995,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53000000000000003,"15min":0.56200000000000006},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71499999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1986560,"data_stored":960235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.497},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.52100000000000002,"15min":0.505},"min":{"1min":0.35199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.41699999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51300000000000001,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.77600000000000002,"5min":0.872,"15min":0.872},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.55800000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.755,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.501}]},{"osd":3,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.37,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.33000000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84399999999999997,"5min":0.877,"15min":0.877},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.28100000000000003,"5min":0.22,"15min":0.22},"max":{"1min":0.77900000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.432}]},{"osd":7,"last update":"Mon Feb 19 14:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58099999999999996,"15min":0.60299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.752,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.375},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.63},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81000000000000005,"5min":1.03,"15min":1.03},"last":0.53200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574926,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1994752,"data_stored":960887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.52900000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73999999999999999,"5min":0.746,"15min":0.746},"last":0.38}]},{"osd":1,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68400000000000005,"5min":0.877,"15min":0.877},"last":0.434},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51500000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75600000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65100000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.505}]},{"osd":3,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.64100000000000001,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70399999999999996,"5min":1.038,"15min":1.038},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71299999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71599999999999997,"5min":1.038,"15min":1.038},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78700000000000003,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":960853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.40999999999999998,"5min":3.694,"15min":3.694},"last":0.372},{"interface":"front","average":{"1min":0.312,"5min":0.36899999999999999,"15min":0.32500000000000001},"min":{"1min":0.21299999999999999,"5min":0.192,"15min":0.189},"max":{"1min":0.41499999999999998,"5min":4.835,"15min":4.835},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32300000000000001,"15min":0.314},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.23100000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52600000000000002,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.434},{"interface":"front","average":{"1min":0.378,"5min":0.40500000000000003,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.33800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70799999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79900000000000004,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81799999999999995,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.45400000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60399999999999998,"15min":0.64600000000000002},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72399999999999998,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58999999999999997,"15min":0.63800000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84799999999999998,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.56000000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":959059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.32500000000000001,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45300000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.311,"15min":0.30199999999999999},"min":{"1min":0.20300000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.45200000000000001,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.20699999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48299999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.35999999999999999,"15min":0.35899999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.60399999999999998,"5min":1.512,"15min":1.512},"last":0.33100000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.55900000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.312},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.375},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.375}]},{"osd":4,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":1.133,"15min":1.133},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76300000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71899999999999997,"5min":1.115,"15min":1.115},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.41799999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73999999999999999,"5min":1.236,"15min":1.236},"last":0.69899999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.45400000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76000000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.995,"15min":0.995},"last":0.621}]}]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1974272,"data_stored":960089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.39700000000000002,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.33300000000000002},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.40100000000000002,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30599999999999999,"15min":0.309},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42999999999999999,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.36099999999999999},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.247}]},{"osd":3,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.316},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.38900000000000001},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.23100000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54000000000000004,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":1.0009999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54800000000000004,"15min":0.55800000000000005},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85299999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":0.80000000000000004,"5min":0.95499999999999996,"15min":1.026},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55100000000000005,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82399999999999995,"5min":0.996,"15min":0.996},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79500000000000004,"5min":1.036,"15min":1.036},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.56499999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33704,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338136,"statfs":{"total":96636764160,"available":96602251264,"internally_reserved":0,"allocated":2580480,"data_stored":1549677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.31,"15min":0.30199999999999999},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.315,"15min":0.30499999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.38100000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.33800000000000002},"min":{"1min":0.24299999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.37,"15min":0.371},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54000000000000004,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.54000000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.59699999999999998},"min":{"1min":0.38500000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83699999999999997,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79800000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79000000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.81100000000000005,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:57:59.462 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:57:59.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.593+0000 7f03c7830640 1 Processor -- start 2024-02-19T14:57:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.593+0000 7f03c7830640 1 -- start start 2024-02-19T14:57:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c7830640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03c0150440 0x7f03c00c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c7830640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f03c006ed20 0x7f03c006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c7830640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c7830640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00c1b60 con 0x7f03c006ed20 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c7830640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c0150820 con 0x7f03c0150440 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c7830640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01537d0 con 0x7f03c00c3800 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f03c006ed20 0x7f03c006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f03c006ed20 0x7f03c006d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41502/0 (socket says 172.21.15.78:41502) 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 -- 172.21.15.78:0/1127261163 learned_addr learned my addr 172.21.15.78:0/1127261163 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c55cc640 1 --2- 172.21.15.78:0/1127261163 >> v2:172.21.15.150:3300/0 conn(0x7f03c0150440 0x7f03c00c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c5dcd640 1 --2- 172.21.15.78:0/1127261163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 -- 172.21.15.78:0/1127261163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 msgr2=0x7f03c006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 --2- 172.21.15.78:0/1127261163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c006d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 -- 172.21.15.78:0/1127261163 >> v2:172.21.15.150:3300/0 conn(0x7f03c0150440 msgr2=0x7f03c00c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:59.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 --2- 172.21.15.78:0/1127261163 >> v2:172.21.15.150:3300/0 conn(0x7f03c0150440 0x7f03c00c32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.594+0000 7f03c4dcb640 1 -- 172.21.15.78:0/1127261163 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0153d10 con 0x7f03c006ed20 2024-02-19T14:57:59.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.595+0000 7f03c4dcb640 1 --2- 172.21.15.78:0/1127261163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c006d210 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f03c00735c0 tx=0x7f03b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ecaa4a7990b8a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.595+0000 7f03b3fff640 1 -- 172.21.15.78:0/1127261163 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4030860 con 0x7f03c006ed20 2024-02-19T14:57:59.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.595+0000 7f03b3fff640 1 -- 172.21.15.78:0/1127261163 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b40309c0 con 0x7f03c006ed20 2024-02-19T14:57:59.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.595+0000 7f03b3fff640 1 -- 172.21.15.78:0/1127261163 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4030d50 con 0x7f03c006ed20 2024-02-19T14:57:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.595+0000 7f03c7830640 1 -- 172.21.15.78:0/1127261163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 msgr2=0x7f03c006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.595+0000 7f03c7830640 1 --2- 172.21.15.78:0/1127261163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c006d210 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f03c00735c0 tx=0x7f03b402b9a0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 -- 172.21.15.78:0/1127261163 shutdown_connections 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 --2- 172.21.15.78:0/1127261163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c006d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 --2- 172.21.15.78:0/1127261163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c006d210 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 --2- 172.21.15.78:0/1127261163 >> v2:172.21.15.150:3300/0 conn(0x7f03c0150440 0x7f03c00c32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 -- 172.21.15.78:0/1127261163 >> 172.21.15.78:0/1127261163 conn(0x7f03c006b270 msgr2=0x7f03c006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 -- 172.21.15.78:0/1127261163 shutdown_connections 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.596+0000 7f03c7830640 1 -- 172.21.15.78:0/1127261163 wait complete. 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 Processor -- start 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 -- start start 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c0147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c01453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03c0150440 0x7f03c0145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0152bb0 con 0x7f03c006ed20 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0151fa0 con 0x7f03c0150440 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.597+0000 7f03c7830640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0152280 con 0x7f03c00c3800 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c0147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c0147e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41514/0 (socket says 172.21.15.78:41514) 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 -- 172.21.15.78:0/1622106563 learned_addr learned my addr 172.21.15.78:0/1622106563 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 -- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 msgr2=0x7f03c01453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c01453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 -- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03c0150440 msgr2=0x7f03c0145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c5dcd640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03c0150440 0x7f03c0145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03c0150440 0x7f03c0145920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 -- 172.21.15.78:0/1622106563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40094d0 con 0x7f03c006ed20 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c5dcd640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03c0150440 0x7f03c0145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:57:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c55cc640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c0147e60 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00d5f0 tx=0x7f03bc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0125f0 con 0x7f03c006ed20 2024-02-19T14:57:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc012750 con 0x7f03c006ed20 2024-02-19T14:57:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc012b00 con 0x7f03c006ed20 2024-02-19T14:57:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.598+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c0152560 con 0x7f03c006ed20 2024-02-19T14:57:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.599+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c014b0f0 con 0x7f03c006ed20 2024-02-19T14:57:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.600+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c0074990 con 0x7f03c006ed20 2024-02-19T14:57:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.600+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f03bc012c60 con 0x7f03c006ed20 2024-02-19T14:57:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.600+0000 7f03b1ffb640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03a406b810 0x7f03a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.600+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f03bc08ab90 con 0x7f03c006ed20 2024-02-19T14:57:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.600+0000 7f03c4dcb640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03a406b810 0x7f03a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.601+0000 7f03c4dcb640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03a406b810 0x7f03a406dcd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f03b4036000 tx=0x7f03b403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.605+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f03bc0550d0 con 0x7f03c006ed20 2024-02-19T14:57:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.833+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c00c4a90 con 0x7f03c006ed20 2024-02-19T14:57:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.833+0000 7f03b1ffb640 1 -- 172.21.15.78:0/1622106563 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+6197 (secure 0 0 0) 0x7f03bc058790 con 0x7f03c006ed20 2024-02-19T14:57:59.836 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:57:59.836 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":136,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7768,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":7771,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:57:47.113438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/903597448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":903597448},{"type":"v1","addr":"172.21.15.78:6837","nonce":903597448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:57:57.186859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":5,"addr":"172.21.15.78:6836/2997633530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2997633530},{"type":"v1","addr":"172.21.15.78:6836","nonce":2997633530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:57:59.836 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 136 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03a406b810 msgr2=0x7f03a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03a406b810 0x7f03a406dcd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f03b4036000 tx=0x7f03b403b040 comp rx=0 tx=0).stop 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 msgr2=0x7f03c0147e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c0147e60 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00d5f0 tx=0x7f03bc00dab0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 shutdown_connections 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f03a406b810 0x7f03a406dcd0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03c0150440 0x7f03c0145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03c00c3800 0x7f03c01453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 --2- 172.21.15.78:0/1622106563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03c006ed20 0x7f03c0147e60 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 >> 172.21.15.78:0/1622106563 conn(0x7f03c006b270 msgr2=0x7f03c014efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 shutdown_connections 2024-02-19T14:57:59.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.836+0000 7f03c7830640 1 -- 172.21.15.78:0/1622106563 wait complete. 2024-02-19T14:57:59.852 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 136, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:57:45.095751+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 132, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8772": { "addr": "172.21.15.78:6837/903597448", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 903597448, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 903597448, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8772, "incarnation": 130, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-02-19T14:57:47.113438+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8772 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:57:54.164699+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8769": { "addr": "172.21.15.78:6836/2997633530", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2997633530, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2997633530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8769, "incarnation": 133, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-02-19T14:57:57.186859+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8769 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/1911236290", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1911236290, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 1911236290, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 129, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7768, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/2395461731", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 2395461731, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2395461731, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 129, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7771, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:57:59.853 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:57:59.855 INFO:tasks.cephfs.test_failover:current = [{'gid': 8772, 'name': 'a', 'rank': 0, 'incarnation': 130, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/903597448', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 903597448}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 903597448}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7771, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2395461731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2395461731}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2395461731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 129}, {'gid': 8769, 'name': 'c', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.78:6836/2997633530', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2997633530}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2997633530}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7768, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/1911236290', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 1911236290}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 1911236290}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 129}] 2024-02-19T14:57:59.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:57:59.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.957+0000 7efcf19e2640 1 Processor -- start 2024-02-19T14:57:59.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.957+0000 7efcf19e2640 1 -- start start 2024-02-19T14:57:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efcf19e2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efcec0713a0 0x7efcec071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efcf19e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efcec0c29e0 0x7efcec0c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efcf19e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0c8b60 0x7efcec1536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efcf19e2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec0c32c0 con 0x7efcec0c29e0 2024-02-19T14:57:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efcf19e2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec070500 con 0x7efcec0713a0 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efcf19e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec152ca0 con 0x7efcec0c8b60 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efcec0c29e0 0x7efcec0c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efcec0c29e0 0x7efcec0c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41530/0 (socket says 172.21.15.78:41530) 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 -- 172.21.15.78:0/1906592905 learned_addr learned my addr 172.21.15.78:0/1906592905 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efceaffd640 1 --2- 172.21.15.78:0/1906592905 >> v2:172.21.15.150:3300/0 conn(0x7efcec0713a0 0x7efcec071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efceb7fe640 1 --2- 172.21.15.78:0/1906592905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0c8b60 0x7efcec1536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 -- 172.21.15.78:0/1906592905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0c8b60 msgr2=0x7efcec1536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 --2- 172.21.15.78:0/1906592905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0c8b60 0x7efcec1536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 -- 172.21.15.78:0/1906592905 >> v2:172.21.15.150:3300/0 conn(0x7efcec0713a0 msgr2=0x7efcec071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 --2- 172.21.15.78:0/1906592905 >> v2:172.21.15.150:3300/0 conn(0x7efcec0713a0 0x7efcec071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efce27fc640 1 -- 172.21.15.78:0/1906592905 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0c69f0 con 0x7efcec0c29e0 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.958+0000 7efceaffd640 1 --2- 172.21.15.78:0/1906592905 >> v2:172.21.15.150:3300/0 conn(0x7efcec0713a0 0x7efcec071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:57:59.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.959+0000 7efce27fc640 1 --2- 172.21.15.78:0/1906592905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c29e0 0x7efcec0c64b0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7efcec0c7850 tx=0x7efcd402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6aff38ab87efdb4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.959+0000 7efcea7fc640 1 -- 172.21.15.78:0/1906592905 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd40317e0 con 0x7efcec0c29e0 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.959+0000 7efcea7fc640 1 -- 172.21.15.78:0/1906592905 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd4031940 con 0x7efcec0c29e0 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.959+0000 7efcea7fc640 1 -- 172.21.15.78:0/1906592905 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4031cd0 con 0x7efcec0c29e0 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 -- 172.21.15.78:0/1906592905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c29e0 msgr2=0x7efcec0c64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 --2- 172.21.15.78:0/1906592905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c29e0 0x7efcec0c64b0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7efcec0c7850 tx=0x7efcd402b880 comp rx=0 tx=0).stop 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 -- 172.21.15.78:0/1906592905 shutdown_connections 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 --2- 172.21.15.78:0/1906592905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0c8b60 0x7efcec1536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 --2- 172.21.15.78:0/1906592905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c29e0 0x7efcec0c64b0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 --2- 172.21.15.78:0/1906592905 >> v2:172.21.15.150:3300/0 conn(0x7efcec0713a0 0x7efcec071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 -- 172.21.15.78:0/1906592905 >> 172.21.15.78:0/1906592905 conn(0x7efcec0aa300 msgr2=0x7efcec0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 -- 172.21.15.78:0/1906592905 shutdown_connections 2024-02-19T14:57:59.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.960+0000 7efcf19e2640 1 -- 172.21.15.78:0/1906592905 wait complete. 2024-02-19T14:57:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.961+0000 7efcf19e2640 1 Processor -- start 2024-02-19T14:57:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.962+0000 7efcf19e2640 1 -- start start 2024-02-19T14:57:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efcf19e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0713a0 0x7efcec1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efcf19e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0c29e0 0x7efcec14ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efcf19e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 0x7efcec1e8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efcf19e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1e83b0 con 0x7efcec0c8b60 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efcf19e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1e8060 con 0x7efcec0c29e0 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efcf19e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1541b0 con 0x7efcec0713a0 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 0x7efcec1e8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 0x7efcec1e8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41534/0 (socket says 172.21.15.78:41534) 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 -- 172.21.15.78:0/3171492937 learned_addr learned my addr 172.21.15.78:0/3171492937 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceaffd640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0713a0 0x7efcec1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efce27fc640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0c29e0 0x7efcec14ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 -- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0713a0 msgr2=0x7efcec1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0713a0 0x7efcec1552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 -- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0c29e0 msgr2=0x7efcec14ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0c29e0 0x7efcec14ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:57:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 -- 172.21.15.78:0/3171492937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40094d0 con 0x7efcec0c8b60 2024-02-19T14:57:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efceb7fe640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 0x7efcec1e8840 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7efce4007e60 tx=0x7efce400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.963+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4014420 con 0x7efcec0c8b60 2024-02-19T14:57:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.964+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce40145a0 con 0x7efcec0c8b60 2024-02-19T14:57:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.964+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4014950 con 0x7efcec0c8b60 2024-02-19T14:57:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.964+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec154490 con 0x7efcec0c8b60 2024-02-19T14:57:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.964+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec1536b0 con 0x7efcec0c8b60 2024-02-19T14:57:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.965+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec1e8520 con 0x7efcec0c8b60 2024-02-19T14:57:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.965+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efce4014ab0 con 0x7efcec0c8b60 2024-02-19T14:57:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.966+0000 7efce8ff9640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efccc06b810 0x7efccc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:57:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.966+0000 7efceaffd640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efccc06b810 0x7efccc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:57:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.966+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7efce408adf0 con 0x7efcec0c8b60 2024-02-19T14:57:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.966+0000 7efceaffd640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efccc06b810 0x7efccc06dcd0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7efcec0adc10 tx=0x7efcdc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:57:59.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:57:59.970+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efce4055470 con 0x7efcec0c8b60 2024-02-19T14:58:00.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.185+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcec074990 con 0x7efcec0c8b60 2024-02-19T14:58:00.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.186+0000 7efce8ff9640 1 -- 172.21.15.78:0/3171492937 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+6197 (secure 0 0 0) 0x7efce4058b30 con 0x7efcec0c8b60 2024-02-19T14:58:00.188 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:00.189 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":136,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7768,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129},{"gid":7771,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":129}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:57:47.113438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":130,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/903597448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":903597448},{"type":"v1","addr":"172.21.15.78:6837","nonce":903597448}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:57:57.186859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":5,"addr":"172.21.15.78:6836/2997633530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2997633530},{"type":"v1","addr":"172.21.15.78:6836","nonce":2997633530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:00.189 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 136 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.187+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efccc06b810 msgr2=0x7efccc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.187+0000 7efcf19e2640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efccc06b810 0x7efccc06dcd0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7efcec0adc10 tx=0x7efcdc008040 comp rx=0 tx=0).stop 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.187+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 msgr2=0x7efcec1e8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.187+0000 7efcf19e2640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 0x7efcec1e8840 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7efce4007e60 tx=0x7efce400e4b0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 shutdown_connections 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efccc06b810 0x7efccc06dcd0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcec0c8b60 0x7efcec1e8840 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0c29e0 0x7efcec14ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 --2- 172.21.15.78:0/3171492937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec0713a0 0x7efcec1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 >> 172.21.15.78:0/3171492937 conn(0x7efcec0aa300 msgr2=0x7efcec0c42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 shutdown_connections 2024-02-19T14:58:00.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.188+0000 7efcf19e2640 1 -- 172.21.15.78:0/3171492937 wait complete. 2024-02-19T14:58:00.205 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 136, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:57:45.095751+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 132, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8772": { "addr": "172.21.15.78:6837/903597448", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6835", "nonce": 903597448, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 903597448, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8772, "incarnation": 130, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-02-19T14:57:47.113438+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8772 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:57:54.164699+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8769": { "addr": "172.21.15.78:6836/2997633530", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2997633530, "type": "v2" }, { "addr": "172.21.15.78:6836", "nonce": 2997633530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8769, "incarnation": 133, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-02-19T14:57:57.186859+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8769 } } } ], "standbys": [ { "addr": "172.21.15.150:6834/1911236290", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1911236290, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 1911236290, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 129, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7768, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6835/2395461731", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 2395461731, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2395461731, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 129, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7771, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:58:00.205 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-19T14:58:00.206 INFO:tasks.cephfs.test_failover:current = [{'gid': 8772, 'name': 'a', 'rank': 0, 'incarnation': 130, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/903597448', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 903597448}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 903597448}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7771, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6835/2395461731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2395461731}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2395461731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 129}, {'gid': 8769, 'name': 'c', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.78:6836/2997633530', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2997633530}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2997633530}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7768, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.150:6834/1911236290', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 1911236290}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 1911236290}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 129}] 2024-02-19T14:58:00.206 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-02-19T14:58:00.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.307+0000 7f2aff4ed640 1 Processor -- start 2024-02-19T14:58:00.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.307+0000 7f2aff4ed640 1 -- start start 2024-02-19T14:58:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2aff4ed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2af80c4980 0x7f2af8070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2aff4ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af8159e20 con 0x7f2af80c2f60 2024-02-19T14:58:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2aff4ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af815a380 con 0x7f2af8156920 2024-02-19T14:58:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2aff4ed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af80c3b00 con 0x7f2af80c4980 2024-02-19T14:58:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2afca88640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2af80c2f60 0x7f2af8070480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37592/0 (socket says 172.21.15.78:37592) 2024-02-19T14:58:00.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2afca88640 1 -- 172.21.15.78:0/1661710238 learned_addr learned my addr 172.21.15.78:0/1661710238 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2afda8a640 1 --2- 172.21.15.78:0/1661710238 >> v2:172.21.15.150:3300/0 conn(0x7f2af80c4980 0x7f2af8070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2aeffff640 1 -- 172.21.15.78:0/1661710238 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418611978 0 0) 0x7f2af8159e20 con 0x7f2af80c2f60 2024-02-19T14:58:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2aeffff640 1 -- 172.21.15.78:0/1661710238 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae4003520 con 0x7f2af80c2f60 2024-02-19T14:58:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2afda8a640 1 -- 172.21.15.78:0/1661710238 >> v1:172.21.15.150:6789/0 conn(0x7f2af8156920 legacy=0x7f2af806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2afda8a640 1 -- 172.21.15.78:0/1661710238 >> v1:172.21.15.78:6789/0 conn(0x7f2af80c2f60 legacy=0x7f2af8070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.309+0000 7f2afda8a640 1 -- 172.21.15.78:0/1661710238 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80717e0 con 0x7f2af80c4980 2024-02-19T14:58:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.310+0000 7f2afda8a640 1 --2- 172.21.15.78:0/1661710238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c4980 0x7f2af8070b90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009d90 tx=0x7f2af002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c92887c7e33eaeb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aeffff640 1 -- 172.21.15.78:0/1661710238 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af002cd00 con 0x7f2af80c4980 2024-02-19T14:58:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aeffff640 1 -- 172.21.15.78:0/1661710238 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af002ce60 con 0x7f2af80c4980 2024-02-19T14:58:00.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aeffff640 1 -- 172.21.15.78:0/1661710238 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0034660 con 0x7f2af80c4980 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 -- 172.21.15.78:0/1661710238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c4980 msgr2=0x7f2af8070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/1661710238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c4980 0x7f2af8070b90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009d90 tx=0x7f2af002bf50 comp rx=0 tx=0).stop 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 -- 172.21.15.78:0/1661710238 shutdown_connections 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/1661710238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c4980 0x7f2af8070b90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 -- 172.21.15.78:0/1661710238 >> 172.21.15.78:0/1661710238 conn(0x7f2af80c85e0 msgr2=0x7f2af80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 -- 172.21.15.78:0/1661710238 shutdown_connections 2024-02-19T14:58:00.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.311+0000 7f2aff4ed640 1 -- 172.21.15.78:0/1661710238 wait complete. 2024-02-19T14:58:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.313+0000 7f2aff4ed640 1 Processor -- start 2024-02-19T14:58:00.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.313+0000 7f2aff4ed640 1 -- start start 2024-02-19T14:58:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2aff4ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c2f60 0x7f2af81530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2aff4ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2af80c4980 0x7f2af80becf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2aff4ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 0x7f2af806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2aff4ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af80bd6f0 con 0x7f2af8156920 2024-02-19T14:58:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2aff4ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af81ef180 con 0x7f2af80c2f60 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2aff4ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af80bd8d0 con 0x7f2af80c4980 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2afda8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 0x7f2af806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2afda8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 0x7f2af806eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41546/0 (socket says 172.21.15.78:41546) 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.314+0000 7f2afda8a640 1 -- 172.21.15.78:0/526441473 learned_addr learned my addr 172.21.15.78:0/526441473 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afca88640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2af80c4980 0x7f2af80becf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afd289640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c2f60 0x7f2af81530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afda8a640 1 -- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2af80c4980 msgr2=0x7f2af80becf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afda8a640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2af80c4980 0x7f2af80becf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afda8a640 1 -- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c2f60 msgr2=0x7f2af81530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afda8a640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c2f60 0x7f2af81530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afda8a640 1 -- 172.21.15.78:0/526441473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af0009a70 con 0x7f2af8156920 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2afda8a640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 0x7f2af806eff0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009eb0 tx=0x7f2af0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af003d600 con 0x7f2af8156920 2024-02-19T14:58:00.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af003d780 con 0x7f2af8156920 2024-02-19T14:58:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af81eeb30 con 0x7f2af8156920 2024-02-19T14:58:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af003db30 con 0x7f2af8156920 2024-02-19T14:58:00.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.315+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af81ee7e0 con 0x7f2af8156920 2024-02-19T14:58:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.316+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af81ef330 con 0x7f2af8156920 2024-02-19T14:58:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.317+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2af003b040 con 0x7f2af8156920 2024-02-19T14:58:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.317+0000 7f2aedffb640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2adc06b810 0x7f2adc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.317+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f2af00acb00 con 0x7f2af8156920 2024-02-19T14:58:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.318+0000 7f2afd289640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2adc06b810 0x7f2adc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.318+0000 7f2afd289640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2adc06b810 0x7f2adc06dcd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007fd0 tx=0x7f2af4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.326+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2af81ef330 con 0x7f2af8156920 2024-02-19T14:58:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.503+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f2af81ef2f0 con 0x7f2af8156920 2024-02-19T14:58:00.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.506+0000 7f2aedffb640 1 -- 172.21.15.78:0/526441473 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v52)=0 v52) v1 ==== 117+0+0 (secure 0 0 0) 0x7f2af00770d0 con 0x7f2af8156920 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2adc06b810 msgr2=0x7f2adc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2adc06b810 0x7f2adc06dcd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2af4007fd0 tx=0x7f2af4009040 comp rx=0 tx=0).stop 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 msgr2=0x7f2af806eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 0x7f2af806eff0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009eb0 tx=0x7f2af0004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 shutdown_connections 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2adc06b810 0x7f2adc06dcd0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2af8156920 0x7f2af806eff0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2af80c4980 0x7f2af80becf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 --2- 172.21.15.78:0/526441473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c2f60 0x7f2af81530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 >> 172.21.15.78:0/526441473 conn(0x7f2af80c85e0 msgr2=0x7f2af80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 shutdown_connections 2024-02-19T14:58:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.509+0000 7f2aff4ed640 1 -- 172.21.15.78:0/526441473 wait complete. 2024-02-19T14:58:00.522 INFO:tasks.cephfs.test_failover:assigning affinity to cephfs2 for active mds (mds.c) 2024-02-19T14:58:00.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-02-19T14:58:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.625+0000 7fe2cd84e640 1 Processor -- start 2024-02-19T14:58:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.625+0000 7fe2cd84e640 1 -- start start 2024-02-19T14:58:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2cd84e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe2c80c6d60 0x7fe2c8154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2cd84e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2cd84e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c814f730 con 0x7fe2c806ff80 2024-02-19T14:58:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2cd84e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c80c1940 con 0x7fe2c80c6d60 2024-02-19T14:58:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2cd84e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c8072f70 con 0x7fe2c806e830 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe2c80c6d60 0x7fe2c8154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe2c80c6d60 0x7fe2c8154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41562/0 (socket says 172.21.15.78:41562) 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 -- 172.21.15.78:0/3862727263 learned_addr learned my addr 172.21.15.78:0/3862727263 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c67fc640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 -- 172.21.15.78:0/3862727263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 msgr2=0x7fe2c80c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 -- 172.21.15.78:0/3862727263 >> v1:172.21.15.150:6789/0 conn(0x7fe2c806ff80 legacy=0x7fe2c80c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c6ffd640 1 -- 172.21.15.78:0/3862727263 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80c1b50 con 0x7fe2c80c6d60 2024-02-19T14:58:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.626+0000 7fe2c67fc640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2c6ffd640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c8154f70 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009870 tx=0x7fe2bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa7e0170e364333 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2c5ffb640 1 -- 172.21.15.78:0/3862727263 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc031640 con 0x7fe2c80c6d60 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2c5ffb640 1 -- 172.21.15.78:0/3862727263 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc0317a0 con 0x7fe2c80c6d60 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2c5ffb640 1 -- 172.21.15.78:0/3862727263 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc031b30 con 0x7fe2c80c6d60 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3862727263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 msgr2=0x7fe2c8154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c8154f70 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009870 tx=0x7fe2bc02b8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3862727263 shutdown_connections 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3862727263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c8154f70 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3862727263 >> 172.21.15.78:0/3862727263 conn(0x7fe2c806c170 msgr2=0x7fe2c806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3862727263 shutdown_connections 2024-02-19T14:58:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.628+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3862727263 wait complete. 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.629+0000 7fe2cd84e640 1 Processor -- start 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 -- start start 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2c806ff80 0x7fe2c8149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c81ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c8158d50 con 0x7fe2c80c6d60 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c8159030 con 0x7fe2c806ff80 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2cd84e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c80a9880 con 0x7fe2c806e830 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c81ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2c77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c81ebbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41574/0 (socket says 172.21.15.78:41574) 2024-02-19T14:58:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.630+0000 7fe2c77fe640 1 -- 172.21.15.78:0/3815163653 learned_addr learned my addr 172.21.15.78:0/3815163653 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c67fc640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2c806ff80 0x7fe2c8149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c6ffd640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c77fe640 1 -- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 msgr2=0x7fe2c80a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c77fe640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80a9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c77fe640 1 -- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2c806ff80 msgr2=0x7fe2c8149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c77fe640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2c806ff80 0x7fe2c8149dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c77fe640 1 -- 172.21.15.78:0/3815163653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0094d0 con 0x7fe2c80c6d60 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2c77fe640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c81ebbc0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c6b0 tx=0x7fe2c000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0003b70 con 0x7fe2c80c6d60 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0003cf0 con 0x7fe2c80c6d60 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c814a310 con 0x7fe2c80c6d60 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0003ea0 con 0x7fe2c80c6d60 2024-02-19T14:58:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.631+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80aa620 con 0x7fe2c80c6d60 2024-02-19T14:58:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.632+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c8074990 con 0x7fe2c80c6d60 2024-02-19T14:58:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.633+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe2c0005310 con 0x7fe2c80c6d60 2024-02-19T14:58:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.633+0000 7fe2affff640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2b406b810 0x7fe2b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.633+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fe2c008a470 con 0x7fe2c80c6d60 2024-02-19T14:58:00.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.633+0000 7fe2c6ffd640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2b406b810 0x7fe2b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.634+0000 7fe2c6ffd640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2b406b810 0x7fe2b406dcd0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc036000 tx=0x7fe2bc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.642+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2c00549c0 con 0x7fe2c80c6d60 2024-02-19T14:58:00.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.853+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fe2c8158e90 con 0x7fe2c80c6d60 2024-02-19T14:58:00.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.855+0000 7fe2affff640 1 -- 172.21.15.78:0/3815163653 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v53)=0 v53) v1 ==== 117+0+0 (secure 0 0 0) 0x7fe2c8158e90 con 0x7fe2c80c6d60 2024-02-19T14:58:00.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2b406b810 msgr2=0x7fe2b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2b406b810 0x7fe2b406dcd0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc036000 tx=0x7fe2bc03c040 comp rx=0 tx=0).stop 2024-02-19T14:58:00.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 msgr2=0x7fe2c81ebbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c81ebbc0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c6b0 tx=0x7fe2c000cb70 comp rx=0 tx=0).stop 2024-02-19T14:58:00.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 shutdown_connections 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe2b406b810 0x7fe2b406dcd0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2c80c6d60 0x7fe2c81ebbc0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2c806ff80 0x7fe2c8149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 --2- 172.21.15.78:0/3815163653 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2c806e830 0x7fe2c80a9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 >> 172.21.15.78:0/3815163653 conn(0x7fe2c806c170 msgr2=0x7fe2c8155910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 shutdown_connections 2024-02-19T14:58:00.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.858+0000 7fe2cd84e640 1 -- 172.21.15.78:0/3815163653 wait complete. 2024-02-19T14:58:00.874 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 11:0 2024-02-19T14:58:00.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.992+0000 7faf0d965640 1 Processor -- start 2024-02-19T14:58:00.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.992+0000 7faf0d965640 1 -- start start 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf0d965640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faf080c5560 0x7faf080c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf0d965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c7910 0x7faf08154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf0d965640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf080c65a0 con 0x7faf08070720 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf0d965640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf080c7530 con 0x7faf080c5560 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf0d965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf081537b0 con 0x7faf080c7910 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faf080c5560 0x7faf080c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faf080c5560 0x7faf080c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41578/0 (socket says 172.21.15.78:41578) 2024-02-19T14:58:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 -- 172.21.15.78:0/250293436 learned_addr learned my addr 172.21.15.78:0/250293436 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf067fc640 1 --2- 172.21.15.78:0/250293436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c7910 0x7faf08154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 -- 172.21.15.78:0/250293436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c7910 msgr2=0x7faf08154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 --2- 172.21.15.78:0/250293436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c7910 0x7faf08154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 -- 172.21.15.78:0/250293436 >> v1:172.21.15.150:6789/0 conn(0x7faf08070720 legacy=0x7faf08070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.993+0000 7faf06ffd640 1 -- 172.21.15.78:0/250293436 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08153a90 con 0x7faf080c5560 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.994+0000 7faf06ffd640 1 --2- 172.21.15.78:0/250293436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf080c5560 0x7faf080c6ff0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7faf00009870 tx=0x7faf0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457dd72bfb7d0499 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.994+0000 7faf05ffb640 1 -- 172.21.15.78:0/250293436 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf000317c0 con 0x7faf080c5560 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.994+0000 7faf05ffb640 1 -- 172.21.15.78:0/250293436 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf00031920 con 0x7faf080c5560 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.994+0000 7faf05ffb640 1 -- 172.21.15.78:0/250293436 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00031cb0 con 0x7faf080c5560 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 -- 172.21.15.78:0/250293436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf080c5560 msgr2=0x7faf080c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 --2- 172.21.15.78:0/250293436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf080c5560 0x7faf080c6ff0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7faf00009870 tx=0x7faf0002b860 comp rx=0 tx=0).stop 2024-02-19T14:58:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 -- 172.21.15.78:0/250293436 shutdown_connections 2024-02-19T14:58:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 --2- 172.21.15.78:0/250293436 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c7910 0x7faf08154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 --2- 172.21.15.78:0/250293436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf080c5560 0x7faf080c6ff0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 -- 172.21.15.78:0/250293436 >> 172.21.15.78:0/250293436 conn(0x7faf0806b4b0 msgr2=0x7faf080c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 -- 172.21.15.78:0/250293436 shutdown_connections 2024-02-19T14:58:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.995+0000 7faf0d965640 1 -- 172.21.15.78:0/250293436 wait complete. 2024-02-19T14:58:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.996+0000 7faf0d965640 1 Processor -- start 2024-02-19T14:58:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.996+0000 7faf0d965640 1 -- start start 2024-02-19T14:58:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 0x7faf081ec940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c5560 0x7faf08148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080c7910 0x7faf080aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf080ad350 con 0x7faf08070720 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08149230 con 0x7faf080c7910 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf080a9c80 con 0x7faf080c5560 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 0x7faf081ec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 0x7faf081ec940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41582/0 (socket says 172.21.15.78:41582) 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 -- 172.21.15.78:0/1661309293 learned_addr learned my addr 172.21.15.78:0/1661309293 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf077fe640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080c7910 0x7faf080aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf067fc640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c5560 0x7faf08148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 -- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c5560 msgr2=0x7faf08148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c5560 0x7faf08148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 -- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080c7910 msgr2=0x7faf080aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080c7910 0x7faf080aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 -- 172.21.15.78:0/1661309293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000094d0 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf077fe640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080c7910 0x7faf080aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf067fc640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c5560 0x7faf08148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf06ffd640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 0x7faf081ec940 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7faf0002b8a0 tx=0x7faf000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00032860 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf000329e0 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf080a9f60 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00032d90 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.997+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf081ec140 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.998+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf08074990 con 0x7faf08070720 2024-02-19T14:58:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.999+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7faf0004e450 con 0x7faf08070720 2024-02-19T14:58:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:00.999+0000 7faeeffff640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faef406b810 0x7faef406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.000+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 6957+0+0 (secure 0 0 0) 0x7faf000ac830 con 0x7faf08070720 2024-02-19T14:58:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.000+0000 7faf067fc640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faef406b810 0x7faef406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.000+0000 7faf067fc640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faef406b810 0x7faef406dcd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7faf080c7d70 tx=0x7faefc0039d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:01.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.004+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faf00076e00 con 0x7faf08070720 2024-02-19T14:58:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.306+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "11:0"} v 0) v1 -- 0x7faf080c76f0 con 0x7faf08070720 2024-02-19T14:58:01.309 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:58:01.306+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:58:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.308+0000 7faeeffff640 1 -- 172.21.15.78:0/1661309293 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "11:0"}]=0 failed mds gid 8772 v137) v1 ==== 98+0+0 (secure 0 0 0) 0x7faf0007a4c0 con 0x7faf08070720 2024-02-19T14:58:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:failed mds gid 8772 2024-02-19T14:58:01.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faef406b810 msgr2=0x7faef406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faef406b810 0x7faef406dcd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7faf080c7d70 tx=0x7faefc0039d0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 msgr2=0x7faf081ec940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 0x7faf081ec940 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7faf0002b8a0 tx=0x7faf000046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 shutdown_connections 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faef406b810 0x7faef406dcd0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080c7910 0x7faf080aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf080c5560 0x7faf08148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 --2- 172.21.15.78:0/1661309293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf08070720 0x7faf081ec940 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.311+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 >> 172.21.15.78:0/1661309293 conn(0x7faf0806b4b0 msgr2=0x7faf08150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.312+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 shutdown_connections 2024-02-19T14:58:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.312+0000 7faf0d965640 1 -- 172.21.15.78:0/1661309293 wait complete. 2024-02-19T14:58:01.315 INFO:tasks.ceph.mds.a.smithi078.stderr: -335> 2024-02-19T14:57:51.022+0000 7fedcee23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:01.320 INFO:tasks.ceph.mds.c.smithi078.stderr: -219> 2024-02-19T14:57:56.021+0000 7f1d0bb68640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:01.332 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:01.346 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:58:01.349 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:58:01.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.430+0000 7fee68d6c640 1 Processor -- start 2024-02-19T14:58:01.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 -- start start 2024-02-19T14:58:01.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee640c29c0 0x7fee640c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee640c49c0 0x7fee6406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee64152c90 con 0x7fee640c29c0 2024-02-19T14:58:01.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee64072250 con 0x7fee640c49c0 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.431+0000 7fee68d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee64072480 con 0x7fee640c25e0 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee640c29c0 0x7fee640c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee640c29c0 0x7fee640c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41630/0 (socket says 172.21.15.78:41630) 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 -- 172.21.15.78:0/2914228555 learned_addr learned my addr 172.21.15.78:0/2914228555 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee62d9d640 1 --2- 172.21.15.78:0/2914228555 >> v2:172.21.15.150:3300/0 conn(0x7fee640c49c0 0x7fee6406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee6259c640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 -- 172.21.15.78:0/2914228555 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 msgr2=0x7fee641526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 -- 172.21.15.78:0/2914228555 >> v2:172.21.15.150:3300/0 conn(0x7fee640c49c0 msgr2=0x7fee6406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 --2- 172.21.15.78:0/2914228555 >> v2:172.21.15.150:3300/0 conn(0x7fee640c49c0 0x7fee6406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee61d9b640 1 -- 172.21.15.78:0/2914228555 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640c3eb0 con 0x7fee640c29c0 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.432+0000 7fee6259c640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee61d9b640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee640c4480 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fee64072650 tx=0x7fee4c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452d3e41a5f0d0fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee6159a640 1 -- 172.21.15.78:0/2914228555 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0317c0 con 0x7fee640c29c0 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee6159a640 1 -- 172.21.15.78:0/2914228555 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c031920 con 0x7fee640c29c0 2024-02-19T14:58:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee6159a640 1 -- 172.21.15.78:0/2914228555 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c031cb0 con 0x7fee640c29c0 2024-02-19T14:58:01.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 -- 172.21.15.78:0/2914228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 msgr2=0x7fee640c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee640c4480 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fee64072650 tx=0x7fee4c02b860 comp rx=0 tx=0).stop 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 -- 172.21.15.78:0/2914228555 shutdown_connections 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2914228555 >> v2:172.21.15.150:3300/0 conn(0x7fee640c49c0 0x7fee6406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee640c4480 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2914228555 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 -- 172.21.15.78:0/2914228555 >> 172.21.15.78:0/2914228555 conn(0x7fee6406b270 msgr2=0x7fee6406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 -- 172.21.15.78:0/2914228555 shutdown_connections 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.433+0000 7fee68d6c640 1 -- 172.21.15.78:0/2914228555 wait complete. 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 Processor -- start 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 -- start start 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee64155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee64153a30 0x7fee640a8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee641556e0 con 0x7fee640c29c0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee64153ec0 con 0x7fee64153a30 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee68d6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee641560b0 con 0x7fee640c25e0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee61d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee64155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee61d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee64155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41638/0 (socket says 172.21.15.78:41638) 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.435+0000 7fee61d9b640 1 -- 172.21.15.78:0/2541498033 learned_addr learned my addr 172.21.15.78:0/2541498033 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee6259c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee61d9b640 1 -- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 msgr2=0x7fee641de5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee62d9d640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee64153a30 0x7fee640a8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee61d9b640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641de5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee61d9b640 1 -- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee64153a30 msgr2=0x7fee640a8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee61d9b640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee64153a30 0x7fee640a8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee61d9b640 1 -- 172.21.15.78:0/2541498033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0094d0 con 0x7fee640c29c0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee61d9b640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee64155b70 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fee4c02b8a0 tx=0x7fee4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c032860 con 0x7fee640c29c0 2024-02-19T14:58:01.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4c032a00 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4c032db0 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee641533d0 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.436+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee64155370 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.437+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee64074990 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.437+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fee4c04e450 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.438+0000 7fee5affd640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4406b840 0x7fee4406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.438+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fee4c0ac810 con 0x7fee640c29c0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.438+0000 7fee6259c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4406b840 0x7fee4406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:01.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.438+0000 7fee6259c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4406b840 0x7fee4406dd00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fee64150120 tx=0x7fee5c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:01.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.442+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee4c076ca0 con 0x7fee640c29c0 2024-02-19T14:58:01.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.648+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee6414ef10 con 0x7fee640c29c0 2024-02-19T14:58:01.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.649+0000 7fee5affd640 1 -- 172.21.15.78:0/2541498033 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4621 (secure 0 0 0) 0x7fee4c07a360 con 0x7fee640c29c0 2024-02-19T14:58:01.652 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:01.652 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:01.312261+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:01.312233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:replay","state_seq":6,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:01.652 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 138 2024-02-19T14:58:01.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4406b840 msgr2=0x7fee4406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4406b840 0x7fee4406dd00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fee64150120 tx=0x7fee5c00b040 comp rx=0 tx=0).stop 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 msgr2=0x7fee64155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee64155b70 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fee4c02b8a0 tx=0x7fee4c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 shutdown_connections 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4406b840 0x7fee4406dd00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c29c0 0x7fee64155b70 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee64153a30 0x7fee640a8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 --2- 172.21.15.78:0/2541498033 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c25e0 0x7fee641de5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 >> 172.21.15.78:0/2541498033 conn(0x7fee6406b270 msgr2=0x7fee64150430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 shutdown_connections 2024-02-19T14:58:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:01.651+0000 7fee68d6c640 1 -- 172.21.15.78:0/2541498033 wait complete. 2024-02-19T14:58:02.066 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:57:45.095751+0000', 'modified': '2024-02-19T14:58:01.312261+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 157, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7771}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7771': {'gid': 7771, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.150:6835/2395461731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2395461731}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2395461731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:02.066 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7771:up:replay 2024-02-19T14:58:03.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.172+0000 7efe22872640 1 Processor -- start 2024-02-19T14:58:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.172+0000 7efe22872640 1 -- start start 2024-02-19T14:58:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.172+0000 7efe22872640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe1c152610 0x7efe1c158f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.172+0000 7efe22872640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 0x7efe1c0c0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe22872640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c1529f0 con 0x7efe1c0c06b0 2024-02-19T14:58:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe22872640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c06da70 con 0x7efe1c152610 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe22872640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0bf9b0 con 0x7efe1c06e350 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe20e0f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efe1c0c06b0 0x7efe1c0c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37608/0 (socket says 172.21.15.78:37608) 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe20e0f640 1 -- 172.21.15.78:0/1284673833 learned_addr learned my addr 172.21.15.78:0/1284673833 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302379034 0 0) 0x7efe1c1529f0 con 0x7efe1c0c06b0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1bfff640 1 --2- 172.21.15.78:0/1284673833 >> v2:172.21.15.150:3300/0 conn(0x7efe1c152610 0x7efe1c158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe04003520 con 0x7efe1c0c06b0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1b7fe640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 0x7efe1c0c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376006435 0 0) 0x7efe04003520 con 0x7efe1c0c06b0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 msgr2=0x7efe1c0c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 0x7efe1c0c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 >> v2:172.21.15.150:3300/0 conn(0x7efe1c152610 msgr2=0x7efe1c158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 --2- 172.21.15.78:0/1284673833 >> v2:172.21.15.150:3300/0 conn(0x7efe1c152610 0x7efe1c158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c159470 con 0x7efe1c0c06b0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7efe140019f0 con 0x7efe1c0c06b0 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 >> v1:172.21.15.78:6789/0 conn(0x7efe1c0c06b0 legacy=0x7efe1c0c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe04003970 0x7efe04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 0x7efe040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe04004d00 0x7efe04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c06da70 con 0x7efe04004d00 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0bf9b0 con 0x7efe04003970 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0400b980 con 0x7efe1c06e350 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.173+0000 7efe20e0f640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe04004d00 0x7efe04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe1b7fe640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 0x7efe040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe1bfff640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe04003970 0x7efe04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe20e0f640 1 -- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 msgr2=0x7efe040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe20e0f640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c06e350 0x7efe040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe20e0f640 1 -- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe04003970 msgr2=0x7efe04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe20e0f640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe04003970 0x7efe04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe20e0f640 1 -- 172.21.15.78:0/1284673833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe04003520 con 0x7efe04004d00 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe1bfff640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe04003970 0x7efe04003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.174+0000 7efe20e0f640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe04004d00 0x7efe04005180 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7efe1400a840 tx=0x7efe040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1429697484ef6f82 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14034630 con 0x7efe04004d00 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1402f070 con 0x7efe04004d00 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe1affd640 1 -- 172.21.15.78:0/1284673833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe140349f0 con 0x7efe04004d00 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe22872640 1 -- 172.21.15.78:0/1284673833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe04004d00 msgr2=0x7efe04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe22872640 1 --2- 172.21.15.78:0/1284673833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe04004d00 0x7efe04005180 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7efe1400a840 tx=0x7efe040057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe20e0f640 1 -- 172.21.15.78:0/1284673833 reap_dead start 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe22872640 1 -- 172.21.15.78:0/1284673833 shutdown_connections 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe22872640 1 -- 172.21.15.78:0/1284673833 >> 172.21.15.78:0/1284673833 conn(0x7efe1c06b270 msgr2=0x7efe1c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe22872640 1 -- 172.21.15.78:0/1284673833 shutdown_connections 2024-02-19T14:58:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.175+0000 7efe22872640 1 -- 172.21.15.78:0/1284673833 wait complete. 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.176+0000 7efe22872640 1 Processor -- start 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.176+0000 7efe22872640 1 -- start start 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe22872640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 0x7efe1c153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe22872640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c1eade0 0x7efe1c0aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe22872640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe1c14dc80 0x7efe1c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe22872640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0a9940 con 0x7efe1c0abab0 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe22872640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c14e640 con 0x7efe1c14dc80 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe22872640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c155e00 con 0x7efe1c1eade0 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe1bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 0x7efe1c153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe1bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 0x7efe1c153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41652/0 (socket says 172.21.15.78:41652) 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe1bfff640 1 -- 172.21.15.78:0/1977872815 learned_addr learned my addr 172.21.15.78:0/1977872815 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe20e0f640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe1c14dc80 0x7efe1c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.177+0000 7efe1b7fe640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c1eade0 0x7efe1c0aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe1bfff640 1 -- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c1eade0 msgr2=0x7efe1c0aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe1bfff640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c1eade0 0x7efe1c0aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe1bfff640 1 -- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe1c14dc80 msgr2=0x7efe1c14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe1bfff640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe1c14dc80 0x7efe1c14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe1bfff640 1 -- 172.21.15.78:0/1977872815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c1eb260 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe1bfff640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 0x7efe1c153080 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7efe0c011630 tx=0x7efe0c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c00f920 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c00fa80 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c0a9380 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c00fe10 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.178+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c0aa3d0 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.179+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c0aa230 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.179+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efe0c015460 con 0x7efe1c0abab0 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.180+0000 7efe18ff9640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe0806b810 0x7efe0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:03.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.180+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7efe0c08af70 con 0x7efe1c0abab0 2024-02-19T14:58:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.180+0000 7efe1b7fe640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe0806b810 0x7efe0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.180+0000 7efe1b7fe640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe0806b810 0x7efe0806dcd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efe1c155c10 tx=0x7efe10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:03.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.184+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efe0c0bedc0 con 0x7efe1c0abab0 2024-02-19T14:58:03.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.393+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe1c074990 con 0x7efe1c0abab0 2024-02-19T14:58:03.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.393+0000 7efe18ff9640 1 -- 172.21.15.78:0/1977872815 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6204 (secure 0 0 0) 0x7efe1c0aa230 con 0x7efe1c0abab0 2024-02-19T14:58:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:03.396 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:02.319888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:02.319402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 140 2024-02-19T14:58:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe0806b810 msgr2=0x7efe0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe0806b810 0x7efe0806dcd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efe1c155c10 tx=0x7efe10008040 comp rx=0 tx=0).stop 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 msgr2=0x7efe1c153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 0x7efe1c153080 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7efe0c011630 tx=0x7efe0c011af0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 shutdown_connections 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe0806b810 0x7efe0806dcd0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe1c14dc80 0x7efe1c14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe1c1eade0 0x7efe1c0aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 --2- 172.21.15.78:0/1977872815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe1c0abab0 0x7efe1c153080 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 >> 172.21.15.78:0/1977872815 conn(0x7efe1c06b270 msgr2=0x7efe1c151b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.395+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 shutdown_connections 2024-02-19T14:58:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:03.396+0000 7efe22872640 1 -- 172.21.15.78:0/1977872815 wait complete. 2024-02-19T14:58:03.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 140, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:57:45.095751+0000', 'modified': '2024-02-19T14:58:02.319888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 157, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7771}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7771': {'gid': 7771, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.150:6835/2395461731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2395461731}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2395461731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:03.811 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7771:up:rejoin 2024-02-19T14:58:04.812 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:04.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.925+0000 7fb73f2df640 1 Processor -- start 2024-02-19T14:58:04.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.925+0000 7fb73f2df640 1 -- start start 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73f2df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb7380c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73f2df640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb7380c19c0 0x7fb738155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73f2df640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb738072e80 con 0x7fb738072180 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73f2df640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380734c0 con 0x7fb7380c19c0 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73f2df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73806e990 con 0x7fb7381528d0 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73c87a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb7380c19c0 0x7fb738155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73d87c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb738072180 0x7fb73806ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37616/0 (socket says 172.21.15.78:37616) 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73d87c640 1 -- 172.21.15.78:0/3496984123 learned_addr learned my addr 172.21.15.78:0/3496984123 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73c87a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb7380c19c0 0x7fb738155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41666/0 (socket says 172.21.15.78:41666) 2024-02-19T14:58:04.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73d07b640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb7380c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:04.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73c87a640 1 -- 172.21.15.78:0/3496984123 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 msgr2=0x7fb7380c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:04.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73c87a640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb7380c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:04.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73c87a640 1 -- 172.21.15.78:0/3496984123 >> v1:172.21.15.78:6789/0 conn(0x7fb738072180 legacy=0x7fb73806ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:04.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73c87a640 1 -- 172.21.15.78:0/3496984123 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380731e0 con 0x7fb7380c19c0 2024-02-19T14:58:04.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb72ffff640 1 -- 172.21.15.78:0/3496984123 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754769512 0 0) 0x7fb730001490 con 0x7fb738072180 2024-02-19T14:58:04.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.926+0000 7fb73d07b640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb7380c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.927+0000 7fb73c87a640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb738155070 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fb7280097a0 tx=0x7fb72802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1062b6b343b295 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.927+0000 7fb72ffff640 1 -- 172.21.15.78:0/3496984123 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728031780 con 0x7fb7380c19c0 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.927+0000 7fb72ffff640 1 -- 172.21.15.78:0/3496984123 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280318e0 con 0x7fb7380c19c0 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.927+0000 7fb72ffff640 1 -- 172.21.15.78:0/3496984123 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728031c70 con 0x7fb7380c19c0 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 -- 172.21.15.78:0/3496984123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 msgr2=0x7fb738155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb738155070 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fb7280097a0 tx=0x7fb72802b820 comp rx=0 tx=0).stop 2024-02-19T14:58:04.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 -- 172.21.15.78:0/3496984123 shutdown_connections 2024-02-19T14:58:04.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb738155070 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:04.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3496984123 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb7380c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:04.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 -- 172.21.15.78:0/3496984123 >> 172.21.15.78:0/3496984123 conn(0x7fb7380c7590 msgr2=0x7fb7380c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:04.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 -- 172.21.15.78:0/3496984123 shutdown_connections 2024-02-19T14:58:04.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.928+0000 7fb73f2df640 1 -- 172.21.15.78:0/3496984123 wait complete. 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.929+0000 7fb73f2df640 1 Processor -- start 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.929+0000 7fb73f2df640 1 -- start start 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73f2df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738072180 0x7fb73814d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73f2df640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb7381477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73f2df640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb738147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73f2df640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7381e20e0 con 0x7fb7380c19c0 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73f2df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7381edfd0 con 0x7fb738072180 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73f2df640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7381ee2b0 con 0x7fb7381528d0 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb7381477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:04.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb7381477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41682/0 (socket says 172.21.15.78:41682) 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 -- 172.21.15.78:0/3129303666 learned_addr learned my addr 172.21.15.78:0/3129303666 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73d07b640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738072180 0x7fb73814d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73d87c640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb738147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 -- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 msgr2=0x7fb738147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb738147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 -- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738072180 msgr2=0x7fb73814d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738072180 0x7fb73814d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 -- 172.21.15.78:0/3129303666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734009400 con 0x7fb7380c19c0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73d07b640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738072180 0x7fb73814d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.930+0000 7fb73c87a640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb7381477b0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fb728034040 tx=0x7fb7280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.931+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728032820 con 0x7fb7380c19c0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.931+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280329a0 con 0x7fb7380c19c0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.931+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb728009400 con 0x7fb7380c19c0 2024-02-19T14:58:04.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.931+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728032d50 con 0x7fb7380c19c0 2024-02-19T14:58:04.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.931+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7381ee870 con 0x7fb7380c19c0 2024-02-19T14:58:04.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.932+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7381eea60 con 0x7fb7380c19c0 2024-02-19T14:58:04.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.933+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb72804e440 con 0x7fb7380c19c0 2024-02-19T14:58:04.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.933+0000 7fb72dffb640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb71c06b810 0x7fb71c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:04.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.933+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fb7280ac5e0 con 0x7fb7380c19c0 2024-02-19T14:58:04.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.933+0000 7fb73d07b640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb71c06b810 0x7fb71c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:04.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.934+0000 7fb73d07b640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb71c06b810 0x7fb71c06dcd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb73814a990 tx=0x7fb734009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:04.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:04.937+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb728076bb0 con 0x7fb7380c19c0 2024-02-19T14:58:05.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.148+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb738074990 con 0x7fb7380c19c0 2024-02-19T14:58:05.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.148+0000 7fb72dffb640 1 -- 172.21.15.78:0/3129303666 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fb72807a270 con 0x7fb7380c19c0 2024-02-19T14:58:05.151 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:05.151 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":141,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:03.320716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:03.320830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:05.152 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 141 2024-02-19T14:58:05.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb71c06b810 msgr2=0x7fb71c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb71c06b810 0x7fb71c06dcd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb73814a990 tx=0x7fb734009190 comp rx=0 tx=0).stop 2024-02-19T14:58:05.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 msgr2=0x7fb7381477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb7381477b0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fb728034040 tx=0x7fb7280046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 shutdown_connections 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb71c06b810 0x7fb71c06dcd0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7380c19c0 0x7fb7381477b0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.150+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7381528d0 0x7fb738147cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.151+0000 7fb73f2df640 1 --2- 172.21.15.78:0/3129303666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738072180 0x7fb73814d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.151+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 >> 172.21.15.78:0/3129303666 conn(0x7fb7380c7590 msgr2=0x7fb738156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.151+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 shutdown_connections 2024-02-19T14:58:05.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.151+0000 7fb73f2df640 1 -- 172.21.15.78:0/3129303666 wait complete. 2024-02-19T14:58:05.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 141, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:57:45.095751+0000', 'modified': '2024-02-19T14:58:03.320716+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 157, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7771}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7771': {'gid': 7771, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.150:6835/2395461731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2395461731}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2395461731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:05.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:58:05.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-19T14:58:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.667+0000 7f47114b1640 1 Processor -- start 2024-02-19T14:58:05.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.667+0000 7f47114b1640 1 -- start start 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f47114b1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c06bb00 0x7f470c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f47114b1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c156e80 con 0x7f470c0721a0 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f47114b1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c072d70 con 0x7f470c150560 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f47114b1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c06ffc0 con 0x7f470c06bb00 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f470b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f470c0721a0 0x7f470c0c65c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37630/0 (socket says 172.21.15.78:37630) 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f470b7fe640 1 -- 172.21.15.78:0/4277191105 learned_addr learned my addr 172.21.15.78:0/4277191105 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.668+0000 7f470a7fc640 1 --2- 172.21.15.78:0/4277191105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c06bb00 0x7f470c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871890491 0 0) 0x7f470c156e80 con 0x7f470c0721a0 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f4003520 con 0x7f470c0721a0 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811751379 0 0) 0x7f470c072d70 con 0x7f470c150560 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c156e80 con 0x7f470c150560 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f470a7fc640 1 -- 172.21.15.78:0/4277191105 >> v1:172.21.15.150:6789/0 conn(0x7f470c150560 legacy=0x7f470c0c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f470a7fc640 1 -- 172.21.15.78:0/4277191105 >> v1:172.21.15.78:6789/0 conn(0x7f470c0721a0 legacy=0x7f470c0c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.669+0000 7f470a7fc640 1 -- 172.21.15.78:0/4277191105 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c073050 con 0x7f470c06bb00 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f470a7fc640 1 --2- 172.21.15.78:0/4277191105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c06bb00 0x7f470c0c6080 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f47000097d0 tx=0x7f470002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f274418093237b7a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700031770 con 0x7f470c06bb00 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000318d0 con 0x7f470c06bb00 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f4709ffb640 1 -- 172.21.15.78:0/4277191105 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700031c40 con 0x7f470c06bb00 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 -- 172.21.15.78:0/4277191105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c06bb00 msgr2=0x7f470c0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 --2- 172.21.15.78:0/4277191105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c06bb00 0x7f470c0c6080 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f47000097d0 tx=0x7f470002b810 comp rx=0 tx=0).stop 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 -- 172.21.15.78:0/4277191105 shutdown_connections 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 --2- 172.21.15.78:0/4277191105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c06bb00 0x7f470c0c6080 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 -- 172.21.15.78:0/4277191105 >> 172.21.15.78:0/4277191105 conn(0x7f470c06c170 msgr2=0x7f470c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 -- 172.21.15.78:0/4277191105 shutdown_connections 2024-02-19T14:58:05.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.670+0000 7f47114b1640 1 -- 172.21.15.78:0/4277191105 wait complete. 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 Processor -- start 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 -- start start 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 0x7f470c1e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c0721a0 0x7f470c0c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c150560 0x7f470c147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0c4010 con 0x7f470c06bb00 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0c3d00 con 0x7f470c0721a0 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f47114b1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c0a75c0 con 0x7f470c150560 2024-02-19T14:58:05.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 0x7f470c1e8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 0x7f470c1e8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41686/0 (socket says 172.21.15.78:41686) 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.672+0000 7f470affd640 1 -- 172.21.15.78:0/1918927062 learned_addr learned my addr 172.21.15.78:0/1918927062 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470b7fe640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c150560 0x7f470c147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c0721a0 0x7f470c0c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470affd640 1 -- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c150560 msgr2=0x7f470c147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470affd640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c150560 0x7f470c147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470affd640 1 -- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c0721a0 msgr2=0x7f470c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470affd640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c0721a0 0x7f470c0c44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470affd640 1 -- 172.21.15.78:0/1918927062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700009400 con 0x7f470c06bb00 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f470affd640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 0x7f470c1e8a00 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00c870 tx=0x7f46fc00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f46f37fe640 1 -- 172.21.15.78:0/1918927062 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc003b70 con 0x7f470c06bb00 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f46f37fe640 1 -- 172.21.15.78:0/1918927062 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc003cf0 con 0x7f470c06bb00 2024-02-19T14:58:05.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c0a8170 con 0x7f470c06bb00 2024-02-19T14:58:05.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f46f37fe640 1 -- 172.21.15.78:0/1918927062 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc00aa90 con 0x7f470c06bb00 2024-02-19T14:58:05.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.673+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c0a8730 con 0x7f470c06bb00 2024-02-19T14:58:05.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.675+0000 7f46f37fe640 1 -- 172.21.15.78:0/1918927062 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f46fc00abf0 con 0x7f470c06bb00 2024-02-19T14:58:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.675+0000 7f46f37fe640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f806b810 0x7f46f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.675+0000 7f46f37fe640 1 -- 172.21.15.78:0/1918927062 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f46fc08a490 con 0x7f470c06bb00 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.675+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f806b810 0x7f46f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.675+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f806b810 0x7f46f806dcd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4700036000 tx=0x7f470003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.676+0000 7f4714085740 1 Processor -- start 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.676+0000 7f4714085740 1 -- start start 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150f9740 0x563b150fa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f4714085740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 0x563b150f9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150fc8b0 0x563b150be860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f4714085740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b15079cd0 con 0x563b150bfae0 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b1502cf20 con 0x563b150f9740 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b1502a450 con 0x563b150fc8b0 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 0x563b150f9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 0x563b150f9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41702/0 (socket says 172.21.15.78:41702) 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 -- 172.21.15.78:0/1005327039 learned_addr learned my addr 172.21.15.78:0/1005327039 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470affd640 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150f9740 0x563b150fa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470b7fe640 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150fc8b0 0x563b150be860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 -- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150fc8b0 msgr2=0x563b150be860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150fc8b0 0x563b150be860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 -- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150f9740 msgr2=0x563b150fa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150f9740 0x563b150fa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 -- 172.21.15.78:0/1005327039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b14fdaf20 con 0x563b150bfae0 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 0x563b150f9b20 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f47000097d0 tx=0x7f4700030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7597e4c2b832569 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f46f0ff9640 1 -- 172.21.15.78:0/1005327039 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470004b070 con 0x563b150bfae0 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.677+0000 7f46f0ff9640 1 -- 172.21.15.78:0/1005327039 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000469a0 con 0x563b150bfae0 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f46f0ff9640 1 -- 172.21.15.78:0/1005327039 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700046d10 con 0x563b150bfae0 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 -- 172.21.15.78:0/1005327039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 msgr2=0x563b150f9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 0x563b150f9b20 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f47000097d0 tx=0x7f4700030000 comp rx=0 tx=0).stop 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 -- 172.21.15.78:0/1005327039 shutdown_connections 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150bfae0 0x563b150f9b20 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150fc8b0 0x563b150be860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 --2- 172.21.15.78:0/1005327039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150f9740 0x563b150fa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 -- 172.21.15.78:0/1005327039 >> 172.21.15.78:0/1005327039 conn(0x563b150bb120 msgr2=0x563b150dfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 -- 172.21.15.78:0/1005327039 shutdown_connections 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.678+0000 7f4714085740 1 -- 172.21.15.78:0/1005327039 wait complete. 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.679+0000 7f4714085740 1 -- start start 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.679+0000 7f4714085740 1 Processor -- start 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 0x563b150bde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f7480 0x563b150dc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b15025c30 con 0x563b150f9740 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b14fcad30 con 0x563b150fc8b0 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b14cf9310 con 0x563b150f7480 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 10 client.0 _ll_register_callbacks cb 0x563b14fd2d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 0x563b150bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 0x563b150bde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41712/0 (socket says 172.21.15.78:41712) 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 -- 172.21.15.78:0/1415021027 learned_addr learned my addr 172.21.15.78:0/1415021027 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470b7fe640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f7480 0x563b150dc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 -- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f7480 msgr2=0x563b150dc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f7480 0x563b150dc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 -- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 msgr2=0x563b1510e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 -- 172.21.15.78:0/1415021027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c073050 con 0x563b150f9740 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f470affd640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 0x563b150bde80 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0584c0 tx=0x7f46fc058900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3836c033442ea659 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f46e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1415021027 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc056b80 con 0x563b150f9740 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1415021027 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f46fc060990 con 0x563b150f9740 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b14d256c0 con 0x563b150f9740 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1415021027 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46fc060b70 con 0x563b150f9740 2024-02-19T14:58:05.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 10 client.9078 fetch_fsmap learned FSMap version 141 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=2+}) v3 -- 0x563b150242b0 con 0x563b150f9740 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.680+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1415021027 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc074080 con 0x563b150f9740 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1415021027 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 141) v1 ==== 3889+0+0 (secure 0 0 0) 0x7f46fc019020 con 0x563b150f9740 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f4714085740 10 client.9078 fetch_fsmap finished waiting for FSMap version 141 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f4714085740 10 client.9078 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f4714085740 20 client.9078 populate_metadata read hostname 'smithi078' 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f4714085740 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150dc850 0x563b1510aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f4714085740 4 client.9078 mds_command: new command op to 7771 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 --> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b14d76270 con 0x563b150dc850 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150dc850 0x563b1510aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f46e9ffb640 10 client.9078 ms_handle_connect on v2:172.21.15.150:6833/2395461731 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.681+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150dc850 0x563b1510aaa0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60d0ded6de6d703 server_cookie=1bb9c4c41363cc65 in_seq=0 out_seq=0 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1415021027 <== mds.0 v2:172.21.15.150:6833/2395461731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563b14d76270 con 0x563b150dc850 2024-02-19T14:58:05.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f46e9ffb640 10 client.9078 handle_command_reply: tid=0 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 client.9078 shutdown 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 msgr2=0x563b150bde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 0x563b150bde80 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0584c0 tx=0x7f46fc058900 comp rx=0 tx=0).stop 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 shutdown_connections 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150dc850 0x563b1510aaa0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f7480 0x563b150dc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f46e9ffb640 0 client.9078 ms_handle_reset on v2:172.21.15.150:6833/2395461731 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 --2- 172.21.15.78:0/1415021027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f9740 0x563b150bde80 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 >> 172.21.15.78:0/1415021027 conn(0x563b150bfae0 msgr2=0x563b150dfab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 shutdown_connections 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 1 -- 172.21.15.78:0/1415021027 wait complete. 2024-02-19T14:58:05.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.682+0000 7f4714085740 20 client.9078 trim_cache size 0 max 16384 2024-02-19T14:58:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 Processor -- start 2024-02-19T14:58:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 -- start start 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b150ba140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b150dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b14fab490 con 0x563b150f7480 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b150bdf20 con 0x563b150fc8b0 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b1511e9a0 con 0x563b150f9740 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b150ba140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b150ba140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41720/0 (socket says 172.21.15.78:41720) 2024-02-19T14:58:05.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.699+0000 7f470affd640 1 -- 172.21.15.78:0/565811458 learned_addr learned my addr 172.21.15.78:0/565811458 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470a7fc640 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b150dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470b7fe640 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470affd640 1 -- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 msgr2=0x563b150dbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470affd640 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b150dbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470affd640 1 -- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 msgr2=0x563b1510aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470affd640 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470affd640 1 -- 172.21.15.78:0/565811458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b14fc0a20 con 0x563b150f7480 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f470affd640 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b150ba140 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f46fc088510 tx=0x7f46fc05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fcdab4bf022f87 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f46ebfff640 1 -- 172.21.15.78:0/565811458 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc00aea0 con 0x563b150f7480 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f46ebfff640 1 -- 172.21.15.78:0/565811458 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563b14d256c0 con 0x563b150f7480 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f46ebfff640 1 -- 172.21.15.78:0/565811458 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc0880d0 con 0x563b150f7480 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 -- 172.21.15.78:0/565811458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 msgr2=0x563b150ba140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b150ba140 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f46fc088510 tx=0x7f46fc05c000 comp rx=0 tx=0).stop 2024-02-19T14:58:05.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 -- 172.21.15.78:0/565811458 shutdown_connections 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1510aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b150dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 --2- 172.21.15.78:0/565811458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b150ba140 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.700+0000 7f4714085740 1 -- 172.21.15.78:0/565811458 >> 172.21.15.78:0/565811458 conn(0x563b150bfae0 msgr2=0x563b150dd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.701+0000 7f4714085740 1 -- 172.21.15.78:0/565811458 shutdown_connections 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.701+0000 7f4714085740 1 -- 172.21.15.78:0/565811458 wait complete. 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.701+0000 7f4714085740 1 -- start start 2024-02-19T14:58:05.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.701+0000 7f4714085740 1 Processor -- start 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b151675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b15106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1516d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b15114860 con 0x563b150f7480 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b1516a9c0 con 0x563b150fc8b0 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x563b15167b20 con 0x563b150f9740 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 10 client.0 _ll_register_callbacks cb 0x7f46fc058600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b151675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b151675e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41722/0 (socket says 172.21.15.78:41722) 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 -- 172.21.15.78:0/1052480839 learned_addr learned my addr 172.21.15.78:0/1052480839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b15106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470b7fe640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1516d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 -- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 msgr2=0x563b15106fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b15106fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 -- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 msgr2=0x563b1516d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1516d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 -- 172.21.15.78:0/1052480839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b150242b0 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f470affd640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b151675e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f46fc058430 tx=0x7f46fc088390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edaea11310f6155 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f46e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1052480839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc00aea0 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1052480839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f46fc056b80 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47000469a0 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.702+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1052480839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=141) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46fc058120 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 10 client.9084 fetch_fsmap learned FSMap version 141 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=141,monmap=2+}) v3 -- 0x563b14fe6780 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1052480839 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47000469a0 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1052480839 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 141) v1 ==== 3889+0+0 (secure 0 0 0) 0x7f46fc019030 con 0x563b150f7480 2024-02-19T14:58:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 10 client.9084 fetch_fsmap finished waiting for FSMap version 141 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 10 client.9084 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 20 client.9084 populate_metadata read hostname 'smithi078' 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150f2ad0 0x563b150d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 4 client.9084 mds_command: new command op to 7771 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 --> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563b14d62610 con 0x563b150f2ad0 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150f2ad0 0x563b150d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f46e9ffb640 10 client.9084 ms_handle_connect on v2:172.21.15.150:6833/2395461731 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f470a7fc640 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150f2ad0 0x563b150d8860 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f6bbed117e59ba server_cookie=2e724397659ec33c in_seq=0 out_seq=0 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f46e9ffb640 1 -- 172.21.15.78:0/1052480839 <== mds.0 v2:172.21.15.150:6833/2395461731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x563b14d62610 con 0x563b150f2ad0 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.703+0000 7f46e9ffb640 10 client.9084 handle_command_reply: tid=0 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 client.9084 shutdown 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 msgr2=0x563b151675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b151675e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f46fc058430 tx=0x7f46fc088390 comp rx=0 tx=0).stop 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 shutdown_connections 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:6833/2395461731,v1:172.21.15.150:6835/2395461731] conn(0x563b150f2ad0 0x563b150d8860 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b150fc8b0 0x563b1516d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x563b150f9740 0x563b15106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f46e9ffb640 0 client.9084 ms_handle_reset on v2:172.21.15.150:6833/2395461731 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 --2- 172.21.15.78:0/1052480839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563b150f7480 0x563b151675e0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 >> 172.21.15.78:0/1052480839 conn(0x563b150bfae0 msgr2=0x563b150f7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 shutdown_connections 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 1 -- 172.21.15.78:0/1052480839 wait complete. 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f4714085740 20 client.9084 trim_cache size 0 max 16384 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:58:05.713 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 7771, 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.3892758780000003, 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 141, 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 158, 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 158, 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 25.052370794000002 2024-02-19T14:58:05.714 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f806b810 msgr2=0x7f46f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f806b810 0x7f46f806dcd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4700036000 tx=0x7f470003d040 comp rx=0 tx=0).stop 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 msgr2=0x7f470c1e8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 0x7f470c1e8a00 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00c870 tx=0x7f46fc00cd30 comp rx=0 tx=0).stop 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 shutdown_connections 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f46f806b810 0x7f46f806dcd0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f470c150560 0x7f470c147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f470c0721a0 0x7f470c0c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 --2- 172.21.15.78:0/1918927062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f470c06bb00 0x7f470c1e8a00 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 >> 172.21.15.78:0/1918927062 conn(0x7f470c06c170 msgr2=0x7f470c0c77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 shutdown_connections 2024-02-19T14:58:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.704+0000 7f47114b1640 1 -- 172.21.15.78:0/1918927062 wait complete. 2024-02-19T14:58:05.717 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 141, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:57:45.095751+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7771": { "addr": "172.21.15.150:6835/2395461731", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 2395461731, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2395461731, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7771, "incarnation": 138, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-02-19T14:58:03.320716+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7771 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T14:57:54.164699+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_7768": { "addr": "172.21.15.150:6834/1911236290", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1911236290, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 1911236290, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7768, "incarnation": 138, "join_fscid": 12, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 158, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-02-19T14:58:03.320830+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7768 } } } ], "standbys": [ { "addr": "172.21.15.78:6835/2349030930", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6834", "nonce": 2349030930, "type": "v2" }, { "addr": "172.21.15.78:6835", "nonce": 2349030930, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 139, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9042, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.78:6837/1801432695", "addrs": { "addrvec": [ { "addr": "172.21.15.78:6836", "nonce": 1801432695, "type": "v2" }, { "addr": "172.21.15.78:6837", "nonce": 1801432695, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 139, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9045, "incarnation": 0, "join_fscid": 12, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-19T14:58:05.718 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b', 'state': 'up:active'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': 12, 'name': 'd'}] 2024-02-19T14:58:05.719 INFO:tasks.cephfs.test_failover:current = [{'gid': 9042, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6835/2349030930', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2349030930}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 2349030930}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 139}, {'gid': 7771, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.150:6835/2395461731', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2395461731}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2395461731}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 9045, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.78:6837/1801432695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 1801432695}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 1801432695}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 139}, {'gid': 7768, 'name': 'd', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.150:6834/1911236290', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 1911236290}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 1911236290}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-19T14:58:05.719 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-19T14:58:05.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:05.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.816+0000 7fe6d8bd7640 1 Processor -- start 2024-02-19T14:58:05.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.816+0000 7fe6d8bd7640 1 -- start start 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d8bd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c19f0 0x7fe6d4156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d8bd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d4155a50 con 0x7fe6d40c1dd0 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d8bd7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d4155f80 con 0x7fe6d4071410 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d8bd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d4152af0 con 0x7fe6d40c19f0 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe6d40c1dd0 0x7fe6d40c2200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37636/0 (socket says 172.21.15.78:37636) 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d259c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c19f0 0x7fe6d4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d2d9d640 1 -- 172.21.15.78:0/1097798082 learned_addr learned my addr 172.21.15.78:0/1097798082 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:05.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d159a640 1 -- 172.21.15.78:0/1097798082 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189427460 0 0) 0x7fe6d4155a50 con 0x7fe6d40c1dd0 2024-02-19T14:58:05.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d159a640 1 -- 172.21.15.78:0/1097798082 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc003520 con 0x7fe6d40c1dd0 2024-02-19T14:58:05.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d159a640 1 -- 172.21.15.78:0/1097798082 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180742142 0 0) 0x7fe6d4155f80 con 0x7fe6d4071410 2024-02-19T14:58:05.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d259c640 1 -- 172.21.15.78:0/1097798082 >> v1:172.21.15.150:6789/0 conn(0x7fe6d4071410 legacy=0x7fe6d40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d259c640 1 -- 172.21.15.78:0/1097798082 >> v1:172.21.15.78:6789/0 conn(0x7fe6d40c1dd0 legacy=0x7fe6d40c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.817+0000 7fe6d259c640 1 -- 172.21.15.78:0/1097798082 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d4070840 con 0x7fe6d40c19f0 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.818+0000 7fe6d259c640 1 --2- 172.21.15.78:0/1097798082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c19f0 0x7fe6d4156440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40097d0 tx=0x7fe6c402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3add618a4ce8d137 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.818+0000 7fe6d159a640 1 -- 172.21.15.78:0/1097798082 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4031770 con 0x7fe6d40c19f0 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.818+0000 7fe6d159a640 1 -- 172.21.15.78:0/1097798082 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c40318d0 con 0x7fe6d40c19f0 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.818+0000 7fe6d159a640 1 -- 172.21.15.78:0/1097798082 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4031c40 con 0x7fe6d40c19f0 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/1097798082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c19f0 msgr2=0x7fe6d4156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/1097798082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c19f0 0x7fe6d4156440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40097d0 tx=0x7fe6c402b810 comp rx=0 tx=0).stop 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/1097798082 shutdown_connections 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/1097798082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c19f0 0x7fe6d4156440 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/1097798082 >> 172.21.15.78:0/1097798082 conn(0x7fe6d40aa370 msgr2=0x7fe6d40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:05.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/1097798082 shutdown_connections 2024-02-19T14:58:05.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.819+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/1097798082 wait complete. 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.820+0000 7fe6d8bd7640 1 Processor -- start 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.820+0000 7fe6d8bd7640 1 -- start start 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d4071410 0x7fe6d4152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 0x7fe6d40c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c1dd0 0x7fe6d414d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40ab400 con 0x7fe6d40c19f0 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40abc10 con 0x7fe6d4071410 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40adef0 con 0x7fe6d40c1dd0 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 0x7fe6d40c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 0x7fe6d40c5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41724/0 (socket says 172.21.15.78:41724) 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 -- 172.21.15.78:0/4077311856 learned_addr learned my addr 172.21.15.78:0/4077311856 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d2d9d640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c1dd0 0x7fe6d414d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d259c640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d4071410 0x7fe6d4152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 -- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c1dd0 msgr2=0x7fe6d414d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c1dd0 0x7fe6d414d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 -- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d4071410 msgr2=0x7fe6d4152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d4071410 0x7fe6d4152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 -- 172.21.15.78:0/4077311856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c4009400 con 0x7fe6d40c19f0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d1d9b640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 0x7fe6d40c5750 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800c410 tx=0x7fe6c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80035f0 con 0x7fe6d40c19f0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c8003770 con 0x7fe6d40c19f0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d40ab650 con 0x7fe6d40c19f0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8003b20 con 0x7fe6d40c19f0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.821+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d40ac9e0 con 0x7fe6d40c19f0 2024-02-19T14:58:05.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.822+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d40c8320 con 0x7fe6d40c19f0 2024-02-19T14:58:05.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.823+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe6c8003c80 con 0x7fe6d40c19f0 2024-02-19T14:58:05.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.823+0000 7fe6c2ffd640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe6b406b810 0x7fe6b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:05.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.823+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fe6c8089ca0 con 0x7fe6d40c19f0 2024-02-19T14:58:05.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.823+0000 7fe6d259c640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe6b406b810 0x7fe6b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:05.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.824+0000 7fe6d259c640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe6b406b810 0x7fe6b406dcd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4036000 tx=0x7fe6c403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:05.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:05.828+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe6c8054270 con 0x7fe6d40c19f0 2024-02-19T14:58:06.024 INFO:tasks.ceph.mds.d.smithi150.stderr:2024-02-19T14:58:06.022+0000 7f3178a82640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:06.025 INFO:tasks.ceph.mds.b.smithi150.stderr:2024-02-19T14:58:06.023+0000 7fd74e277640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:06.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.034+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6d40c6620 con 0x7fe6d40c19f0 2024-02-19T14:58:06.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.034+0000 7fe6c2ffd640 1 -- 172.21.15.78:0/4077311856 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fe6c8017090 con 0x7fe6d40c19f0 2024-02-19T14:58:06.037 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:06.037 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":141,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:03.320716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:03.320830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:06.037 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 141 2024-02-19T14:58:06.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.036+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe6b406b810 msgr2=0x7fe6b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.036+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe6b406b810 0x7fe6b406dcd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4036000 tx=0x7fe6c403d040 comp rx=0 tx=0).stop 2024-02-19T14:58:06.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 msgr2=0x7fe6d40c5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 0x7fe6d40c5750 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800c410 tx=0x7fe6c800c8d0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 shutdown_connections 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe6b406b810 0x7fe6b406dcd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d40c19f0 0x7fe6d40c5750 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6d40c1dd0 0x7fe6d414d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 --2- 172.21.15.78:0/4077311856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d4071410 0x7fe6d4152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 >> 172.21.15.78:0/4077311856 conn(0x7fe6d40aa370 msgr2=0x7fe6d406cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 shutdown_connections 2024-02-19T14:58:06.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.037+0000 7fe6d8bd7640 1 -- 172.21.15.78:0/4077311856 wait complete. 2024-02-19T14:58:06.050 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:58:06.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:58:06.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:58:06.140 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:58:06.141 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:58:06.141 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:58:06.156 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:58:06.156 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:58:06.157 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:58:06.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:58:06.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.294+0000 7f023a16c640 1 Processor -- start 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.294+0000 7f023a16c640 1 -- start start 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.294+0000 7f023a16c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02340c2f70 0x7f023406c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f023a16c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0234156050 0x7f023406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f023a16c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02340717b0 con 0x7f02340c6a50 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f023a16c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340c7560 con 0x7f0234156050 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f023a16c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0234071a00 con 0x7f02340c2f70 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f02337fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f02340c6a50 0x7f0234155940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37644/0 (socket says 172.21.15.78:37644) 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f02337fe640 1 -- 172.21.15.78:0/4199933244 learned_addr learned my addr 172.21.15.78:0/4199933244 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f0233fff640 1 --2- 172.21.15.78:0/4199933244 >> v2:172.21.15.150:3300/0 conn(0x7f0234156050 0x7f023406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f0232ffd640 1 --2- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02340c2f70 0x7f023406c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f02327fc640 1 -- 172.21.15.78:0/4199933244 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405662874 0 0) 0x7f02340717b0 con 0x7f02340c6a50 2024-02-19T14:58:06.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f02327fc640 1 -- 172.21.15.78:0/4199933244 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f021c003520 con 0x7f02340c6a50 2024-02-19T14:58:06.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f0233fff640 1 -- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02340c2f70 msgr2=0x7f023406c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:06.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f0233fff640 1 --2- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02340c2f70 0x7f023406c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f0233fff640 1 -- 172.21.15.78:0/4199933244 >> v1:172.21.15.78:6789/0 conn(0x7f02340c6a50 legacy=0x7f0234155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.295+0000 7f0233fff640 1 -- 172.21.15.78:0/4199933244 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02340c8030 con 0x7f0234156050 2024-02-19T14:58:06.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.296+0000 7f0233fff640 1 --2- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0234156050 0x7f023406cf20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f022c009d90 tx=0x7f022c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445516474b001236 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.296+0000 7f02327fc640 1 -- 172.21.15.78:0/4199933244 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c02cbc0 con 0x7f0234156050 2024-02-19T14:58:06.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.296+0000 7f02327fc640 1 -- 172.21.15.78:0/4199933244 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c02cd20 con 0x7f0234156050 2024-02-19T14:58:06.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.296+0000 7f02327fc640 1 -- 172.21.15.78:0/4199933244 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c034640 con 0x7f0234156050 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 -- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0234156050 msgr2=0x7f023406cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 --2- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0234156050 0x7f023406cf20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f022c009d90 tx=0x7f022c02be10 comp rx=0 tx=0).stop 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 -- 172.21.15.78:0/4199933244 shutdown_connections 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 --2- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0234156050 0x7f023406cf20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 --2- 172.21.15.78:0/4199933244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02340c2f70 0x7f023406c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 -- 172.21.15.78:0/4199933244 >> 172.21.15.78:0/4199933244 conn(0x7f023406b700 msgr2=0x7f02340aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 -- 172.21.15.78:0/4199933244 shutdown_connections 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.297+0000 7f023a16c640 1 -- 172.21.15.78:0/4199933244 wait complete. 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.298+0000 7f023a16c640 1 Processor -- start 2024-02-19T14:58:06.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.298+0000 7f023a16c640 1 -- start start 2024-02-19T14:58:06.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02340c2f70 0x7f02341edcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 0x7f02341ecec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0234156050 0x7f02341ed400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340adb70 con 0x7f02340c6a50 2024-02-19T14:58:06.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0234159550 con 0x7f02340c2f70 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02340ad890 con 0x7f0234156050 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 0x7f02341ecec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 0x7f02341ecec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41736/0 (socket says 172.21.15.78:41736) 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 -- 172.21.15.78:0/1591315129 learned_addr learned my addr 172.21.15.78:0/1591315129 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f02337fe640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02340c2f70 0x7f02341edcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0233fff640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0234156050 0x7f02341ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 -- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0234156050 msgr2=0x7f02341ed400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0234156050 0x7f02341ed400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 -- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02340c2f70 msgr2=0x7f02341edcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02340c2f70 0x7f02341edcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 -- 172.21.15.78:0/1591315129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c009a70 con 0x7f02340c6a50 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f0232ffd640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 0x7f02341ecec0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f022800e870 tx=0x7f022800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228009490 con 0x7f02340c6a50 2024-02-19T14:58:06.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022800ee60 con 0x7f02340c6a50 2024-02-19T14:58:06.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02341ee590 con 0x7f02340c6a50 2024-02-19T14:58:06.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022800cb60 con 0x7f02340c6a50 2024-02-19T14:58:06.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.299+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02340abca0 con 0x7f02340c6a50 2024-02-19T14:58:06.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.300+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0234074990 con 0x7f02340c6a50 2024-02-19T14:58:06.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.301+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f022800ccc0 con 0x7f02340c6a50 2024-02-19T14:58:06.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.301+0000 7f021bfff640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f022006b810 0x7f022006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.301+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f022808a680 con 0x7f02340c6a50 2024-02-19T14:58:06.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.301+0000 7f02337fe640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f022006b810 0x7f022006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.302+0000 7f02337fe640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f022006b810 0x7f022006dcd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0224005880 tx=0x7f0224009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.306+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0228054c50 con 0x7f02340c6a50 2024-02-19T14:58:06.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.514+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02340c65c0 con 0x7f02340c6a50 2024-02-19T14:58:06.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.515+0000 7f021bfff640 1 -- 172.21.15.78:0/1591315129 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6204 (secure 0 0 0) 0x7f0228058310 con 0x7f02340c6a50 2024-02-19T14:58:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:06.518 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":141,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:03.320716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:03.320830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:06.518 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 141 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f022006b810 msgr2=0x7f022006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f022006b810 0x7f022006dcd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0224005880 tx=0x7f0224009040 comp rx=0 tx=0).stop 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 msgr2=0x7f02341ecec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 0x7f02341ecec0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f022800e870 tx=0x7f022800ed30 comp rx=0 tx=0).stop 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 shutdown_connections 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f022006b810 0x7f022006dcd0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02340c6a50 0x7f02341ecec0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0234156050 0x7f02341ed400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 --2- 172.21.15.78:0/1591315129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02340c2f70 0x7f02341edcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 >> 172.21.15.78:0/1591315129 conn(0x7f023406b700 msgr2=0x7f0234154ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 shutdown_connections 2024-02-19T14:58:06.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.517+0000 7f023a16c640 1 -- 172.21.15.78:0/1591315129 wait complete. 2024-02-19T14:58:06.531 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:06.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.628+0000 7f3203f47640 1 Processor -- start 2024-02-19T14:58:06.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f3203f47640 1 -- start start 2024-02-19T14:58:06.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f3203f47640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31fc155070 0x7f31fc155500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f3203f47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0718f0 0x7f31fc071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f3203f47640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc159130 con 0x7f31fc0acc50 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f3203f47640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31fc0c2cf0 con 0x7f31fc155070 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f3203f47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc0c2030 con 0x7f31fc0718f0 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f32014e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31fc155070 0x7f31fc155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f32014e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f31fc155070 0x7f31fc155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41738/0 (socket says 172.21.15.78:41738) 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f32014e2640 1 -- 172.21.15.78:0/622644232 learned_addr learned my addr 172.21.15.78:0/622644232 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.629+0000 7f32024e4640 1 --2- 172.21.15.78:0/622644232 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0718f0 0x7f31fc071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f3201ce3640 1 --1- 172.21.15.78:0/622644232 >> v1:172.21.15.150:6789/0 conn(0x7f31fc0acc50 0x7f31fc0c17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53948/0 (socket says 172.21.15.78:53948) 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f32014e2640 1 -- 172.21.15.78:0/622644232 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0718f0 msgr2=0x7f31fc071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f32014e2640 1 --2- 172.21.15.78:0/622644232 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0718f0 0x7f31fc071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f32014e2640 1 -- 172.21.15.78:0/622644232 >> v1:172.21.15.150:6789/0 conn(0x7f31fc0acc50 legacy=0x7f31fc0c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f3200ce1640 1 -- 172.21.15.78:0/622644232 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267260771 0 0) 0x7f31fc159130 con 0x7f31fc0acc50 2024-02-19T14:58:06.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f32014e2640 1 -- 172.21.15.78:0/622644232 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc0ad030 con 0x7f31fc155070 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.630+0000 7f32014e2640 1 --2- 172.21.15.78:0/622644232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc155070 0x7f31fc155500 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009870 tx=0x7f31ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f21d66a9094fd5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3200ce1640 1 -- 172.21.15.78:0/622644232 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec031640 con 0x7f31fc155070 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3200ce1640 1 -- 172.21.15.78:0/622644232 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec0317a0 con 0x7f31fc155070 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3200ce1640 1 -- 172.21.15.78:0/622644232 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec031b30 con 0x7f31fc155070 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 -- 172.21.15.78:0/622644232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc155070 msgr2=0x7f31fc155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 --2- 172.21.15.78:0/622644232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc155070 0x7f31fc155500 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009870 tx=0x7f31ec02b8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 -- 172.21.15.78:0/622644232 shutdown_connections 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 --2- 172.21.15.78:0/622644232 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0718f0 0x7f31fc071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 --2- 172.21.15.78:0/622644232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc155070 0x7f31fc155500 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 -- 172.21.15.78:0/622644232 >> 172.21.15.78:0/622644232 conn(0x7f31fc0c7800 msgr2=0x7f31fc0c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:06.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 -- 172.21.15.78:0/622644232 shutdown_connections 2024-02-19T14:58:06.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.631+0000 7f3203f47640 1 -- 172.21.15.78:0/622644232 wait complete. 2024-02-19T14:58:06.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.632+0000 7f3203f47640 1 Processor -- start 2024-02-19T14:58:06.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.632+0000 7f3203f47640 1 -- start start 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3203f47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 0x7f31fc1eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3203f47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0acc50 0x7f31fc1ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3203f47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31fc155070 0x7f31fc150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3203f47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc1eec20 con 0x7f31fc0718f0 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3203f47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc13a570 con 0x7f31fc155070 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3203f47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc13a850 con 0x7f31fc0acc50 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 0x7f31fc1eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 0x7f31fc1eb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41746/0 (socket says 172.21.15.78:41746) 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 -- 172.21.15.78:0/2098490520 learned_addr learned my addr 172.21.15.78:0/2098490520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f32014e2640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0acc50 0x7f31fc1ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f32024e4640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31fc155070 0x7f31fc150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 -- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0acc50 msgr2=0x7f31fc1ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0acc50 0x7f31fc1ee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 -- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31fc155070 msgr2=0x7f31fc150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.633+0000 7f3201ce3640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31fc155070 0x7f31fc150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f3201ce3640 1 -- 172.21.15.78:0/2098490520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec0094d0 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f3201ce3640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 0x7f31fc1eb2b0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c4d0 tx=0x7f31f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f40074e0 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f4007660 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31fc1eaab0 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4004a20 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.634+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31fc1513a0 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.635+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31fc1595a0 con 0x7f31fc0718f0 2024-02-19T14:58:06.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.635+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f31f4004b80 con 0x7f31fc0718f0 2024-02-19T14:58:06.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.636+0000 7f31f27fc640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31e006b810 0x7f31e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.636+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f31f4089b30 con 0x7f31fc0718f0 2024-02-19T14:58:06.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.636+0000 7f32014e2640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31e006b810 0x7f31e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.636+0000 7f32014e2640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31e006b810 0x7f31e006dcd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f31ec036000 tx=0x7f31ec03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.640+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31fc1595a0 con 0x7f31fc0718f0 2024-02-19T14:58:06.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.811+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31fc1595a0 con 0x7f31fc0718f0 2024-02-19T14:58:06.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.813+0000 7f31f27fc640 1 -- 172.21.15.78:0/2098490520 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+20847 (secure 0 0 0) 0x7f31fc1595a0 con 0x7f31fc0718f0 2024-02-19T14:58:06.816 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:06.816 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":158,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:01.312276+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31e006b810 msgr2=0x7f31e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31e006b810 0x7f31e006dcd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f31ec036000 tx=0x7f31ec03c040 comp rx=0 tx=0).stop 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 msgr2=0x7f31fc1eb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 0x7f31fc1eb2b0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c4d0 tx=0x7f31f400c990 comp rx=0 tx=0).stop 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 shutdown_connections 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31e006b810 0x7f31e006dcd0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31fc155070 0x7f31fc150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31fc0acc50 0x7f31fc1ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 --2- 172.21.15.78:0/2098490520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31fc0718f0 0x7f31fc1eb2b0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 >> 172.21.15.78:0/2098490520 conn(0x7f31fc0c7800 msgr2=0x7f31fc13da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:06.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 shutdown_connections 2024-02-19T14:58:06.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.815+0000 7f3203f47640 1 -- 172.21.15.78:0/2098490520 wait complete. 2024-02-19T14:58:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:57:43.310860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:57:44.061255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-19T14:57:52.540793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-19T14:57:53.126421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:58:06.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:58:06.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.929+0000 7fc212c50640 1 Processor -- start 2024-02-19T14:58:06.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.929+0000 7fc212c50640 1 -- start start 2024-02-19T14:58:06.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.929+0000 7fc212c50640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc20c14cd50 0x7fc20c0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc212c50640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c1563b0 con 0x7fc20c156860 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc212c50640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c153cd0 con 0x7fc20c1537a0 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc212c50640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20c06b6e0 con 0x7fc20c14cd50 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc2111ed640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc20c156860 0x7fc20c156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37652/0 (socket says 172.21.15.78:37652) 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc2111ed640 1 -- 172.21.15.78:0/744927277 learned_addr learned my addr 172.21.15.78:0/744927277 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc2109ec640 1 --2- 172.21.15.78:0/744927277 >> v2:172.21.15.150:3300/0 conn(0x7fc20c14cd50 0x7fc20c0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502402315 0 0) 0x7fc20c1563b0 con 0x7fc20c156860 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc003520 con 0x7fc20c156860 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033540974 0 0) 0x7fc20c153cd0 con 0x7fc20c1537a0 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc20c1563b0 con 0x7fc20c1537a0 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc2109ec640 1 -- 172.21.15.78:0/744927277 >> v1:172.21.15.150:6789/0 conn(0x7fc20c1537a0 legacy=0x7fc20c0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc2109ec640 1 -- 172.21.15.78:0/744927277 >> v1:172.21.15.78:6789/0 conn(0x7fc20c156860 legacy=0x7fc20c156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.930+0000 7fc2109ec640 1 -- 172.21.15.78:0/744927277 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c06b980 con 0x7fc20c14cd50 2024-02-19T14:58:06.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.931+0000 7fc2109ec640 1 --2- 172.21.15.78:0/744927277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c4cd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc2000097a0 tx=0x7fc20002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cf69d10f1aa6f75 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.931+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20002e070 con 0x7fc20c14cd50 2024-02-19T14:58:06.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.931+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2000315a0 con 0x7fc20c14cd50 2024-02-19T14:58:06.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.931+0000 7fc20b7fe640 1 -- 172.21.15.78:0/744927277 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc200031910 con 0x7fc20c14cd50 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 -- 172.21.15.78:0/744927277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 msgr2=0x7fc20c0c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 --2- 172.21.15.78:0/744927277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c4cd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc2000097a0 tx=0x7fc20002b840 comp rx=0 tx=0).stop 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 -- 172.21.15.78:0/744927277 shutdown_connections 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 --2- 172.21.15.78:0/744927277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c4cd0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 -- 172.21.15.78:0/744927277 >> 172.21.15.78:0/744927277 conn(0x7fc20c066e70 msgr2=0x7fc20c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 -- 172.21.15.78:0/744927277 shutdown_connections 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.932+0000 7fc212c50640 1 -- 172.21.15.78:0/744927277 wait complete. 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.933+0000 7fc212c50640 1 Processor -- start 2024-02-19T14:58:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.933+0000 7fc212c50640 1 -- start start 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.933+0000 7fc212c50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 0x7fc20c14d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc20c156860 0x7fc20c14acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c1494e0 con 0x7fc20c1537a0 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c14a370 con 0x7fc20c14cd50 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c0c7010 con 0x7fc20c156860 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 0x7fc20c14d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc2109ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 0x7fc20c14d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41750/0 (socket says 172.21.15.78:41750) 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 -- 172.21.15.78:0/2896968038 learned_addr learned my addr 172.21.15.78:0/2896968038 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc2111ed640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc20c156860 0x7fc20c14acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 -- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc20c156860 msgr2=0x7fc20c14acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc20c156860 0x7fc20c14acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 -- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 msgr2=0x7fc20c0c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 -- 172.21.15.78:0/2896968038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc200009400 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc2109ec640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc20bfff640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 0x7fc20c14d3e0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8001eb0 tx=0x7fc1f800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80052d0 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f8005430 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8003810 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c1ec8d0 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.934+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c14a8a0 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.935+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c074990 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.936+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc1f8003970 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.936+0000 7fc2097fa640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1f006b810 0x7fc1f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.936+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fc1f8089c90 con 0x7fc20c1537a0 2024-02-19T14:58:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.936+0000 7fc2109ec640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1f006b810 0x7fc1f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:06.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.937+0000 7fc2109ec640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1f006b810 0x7fc1f006dcd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc200036000 tx=0x7fc20003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:06.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:06.941+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1f80541d0 con 0x7fc20c1537a0 2024-02-19T14:58:07.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.112+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc20c0c4230 con 0x7fc20c1537a0 2024-02-19T14:58:07.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.112+0000 7fc2097fa640 1 -- 172.21.15.78:0/2896968038 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1f8017020 con 0x7fc20c1537a0 2024-02-19T14:58:07.115 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1f006b810 msgr2=0x7fc1f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1f006b810 0x7fc1f006dcd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc200036000 tx=0x7fc20003d040 comp rx=0 tx=0).stop 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 msgr2=0x7fc20c14d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 0x7fc20c14d3e0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8001eb0 tx=0x7fc1f800c820 comp rx=0 tx=0).stop 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 shutdown_connections 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1f006b810 0x7fc1f006dcd0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc20c1537a0 0x7fc20c14d3e0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc20c156860 0x7fc20c14acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 --2- 172.21.15.78:0/2896968038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c14cd50 0x7fc20c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 >> 172.21.15.78:0/2896968038 conn(0x7fc20c066e70 msgr2=0x7fc20c151470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:07.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 shutdown_connections 2024-02-19T14:58:07.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.114+0000 7fc212c50640 1 -- 172.21.15.78:0/2896968038 wait complete. 2024-02-19T14:58:07.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:58:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 Processor -- start 2024-02-19T14:58:07.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 -- start start 2024-02-19T14:58:07.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb04c0c8e00 0x7fb04c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb04c06d910 0x7fb04c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c06cd20 con 0x7fb04c152700 2024-02-19T14:58:07.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04c1551f0 con 0x7fb04c0c8e00 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.228+0000 7fb0515f7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04c155490 con 0x7fb04c06d910 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb04c0c8e00 0x7fb04c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb04c0c8e00 0x7fb04c06c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41754/0 (socket says 172.21.15.78:41754) 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2984076381 learned_addr learned my addr 172.21.15.78:0/2984076381 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb04c152700 0x7fb04c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37668/0 (socket says 172.21.15.78:37668) 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04b7fe640 1 --2- 172.21.15.78:0/2984076381 >> v2:172.21.15.150:3300/0 conn(0x7fb04c06d910 0x7fb04c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2984076381 >> v2:172.21.15.150:3300/0 conn(0x7fb04c06d910 msgr2=0x7fb04c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 --2- 172.21.15.78:0/2984076381 >> v2:172.21.15.150:3300/0 conn(0x7fb04c06d910 0x7fb04c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2984076381 >> v1:172.21.15.78:6789/0 conn(0x7fb04c152700 legacy=0x7fb04c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2984076381 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c155770 con 0x7fb04c0c8e00 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb049ffb640 1 -- 172.21.15.78:0/2984076381 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868020190 0 0) 0x7fb04c06cd20 con 0x7fb04c152700 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.229+0000 7fb04b7fe640 1 --2- 172.21.15.78:0/2984076381 >> v2:172.21.15.150:3300/0 conn(0x7fb04c06d910 0x7fb04c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb04a7fc640 1 --2- 172.21.15.78:0/2984076381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c06c2a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb04c1596f0 tx=0x7fb04002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdadd737496572e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb049ffb640 1 -- 172.21.15.78:0/2984076381 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040030860 con 0x7fb04c0c8e00 2024-02-19T14:58:07.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb049ffb640 1 -- 172.21.15.78:0/2984076381 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400309c0 con 0x7fb04c0c8e00 2024-02-19T14:58:07.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb049ffb640 1 -- 172.21.15.78:0/2984076381 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040030d50 con 0x7fb04c0c8e00 2024-02-19T14:58:07.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 -- 172.21.15.78:0/2984076381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 msgr2=0x7fb04c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2984076381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c06c2a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb04c1596f0 tx=0x7fb04002b9a0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 -- 172.21.15.78:0/2984076381 shutdown_connections 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2984076381 >> v2:172.21.15.150:3300/0 conn(0x7fb04c06d910 0x7fb04c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2984076381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c06c2a0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 -- 172.21.15.78:0/2984076381 >> 172.21.15.78:0/2984076381 conn(0x7fb04c0cac90 msgr2=0x7fb04c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 -- 172.21.15.78:0/2984076381 shutdown_connections 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.230+0000 7fb0515f7640 1 -- 172.21.15.78:0/2984076381 wait complete. 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 Processor -- start 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 -- start start 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c06d910 0x7fb04c0acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb04c152700 0x7fb04c0c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0aa280 con 0x7fb04c0c8e00 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0aa840 con 0x7fb04c06d910 2024-02-19T14:58:07.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb0515f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c1ee390 con 0x7fb04c152700 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb04a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb04a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c0c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41764/0 (socket says 172.21.15.78:41764) 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2738580997 learned_addr learned my addr 172.21.15.78:0/2738580997 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb04affd640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c06d910 0x7fb04c0acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.232+0000 7fb04b7fe640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb04c152700 0x7fb04c0c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb04c152700 msgr2=0x7fb04c0c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb04a7fc640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb04c152700 0x7fb04c0c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c06d910 msgr2=0x7fb04c0acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb04a7fc640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c06d910 0x7fb04c0acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb04a7fc640 1 -- 172.21.15.78:0/2738580997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044009a70 con 0x7fb04c0c8e00 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb04a7fc640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c0c0ab0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fb04002b9e0 tx=0x7fb040004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040039c90 con 0x7fb04c0c8e00 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb040039e30 con 0x7fb04c0c8e00 2024-02-19T14:58:07.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0400094d0 con 0x7fb04c0c8e00 2024-02-19T14:58:07.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040043640 con 0x7fb04c0c8e00 2024-02-19T14:58:07.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.233+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c0ad100 con 0x7fb04c0c8e00 2024-02-19T14:58:07.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.234+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c1ee780 con 0x7fb04c0c8e00 2024-02-19T14:58:07.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.234+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb0400437a0 con 0x7fb04c0c8e00 2024-02-19T14:58:07.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.235+0000 7fb0337fe640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03806b840 0x7fb03806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.235+0000 7fb04affd640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03806b840 0x7fb03806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.235+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fb04003e070 con 0x7fb04c0c8e00 2024-02-19T14:58:07.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.235+0000 7fb04affd640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03806b840 0x7fb03806dd00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb03c007fd0 tx=0x7fb03c007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.239+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb040077c00 con 0x7fb04c0c8e00 2024-02-19T14:58:07.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.411+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb04c152ae0 con 0x7fb04c0c8e00 2024-02-19T14:58:07.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.411+0000 7fb0337fe640 1 -- 172.21.15.78:0/2738580997 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v158) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb04007b2c0 con 0x7fb04c0c8e00 2024-02-19T14:58:07.414 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03806b840 msgr2=0x7fb03806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03806b840 0x7fb03806dd00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb03c007fd0 tx=0x7fb03c007de0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 msgr2=0x7fb04c0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c0c0ab0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fb04002b9e0 tx=0x7fb040004730 comp rx=0 tx=0).stop 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 shutdown_connections 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb03806b840 0x7fb03806dd00 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb04c0c8e00 0x7fb04c0c0ab0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb04c152700 0x7fb04c0c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 --2- 172.21.15.78:0/2738580997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c06d910 0x7fb04c0acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 >> 172.21.15.78:0/2738580997 conn(0x7fb04c0cac90 msgr2=0x7fb04c14ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:07.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 shutdown_connections 2024-02-19T14:58:07.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.413+0000 7fb0515f7640 1 -- 172.21.15.78:0/2738580997 wait complete. 2024-02-19T14:58:07.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:58:07.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.527+0000 7f5e666f4640 1 Processor -- start 2024-02-19T14:58:07.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.527+0000 7f5e666f4640 1 -- start start 2024-02-19T14:58:07.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e666f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e600ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e666f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5e600bdf20 0x7f5e600c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e666f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600c8de0 con 0x7f5e60158530 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e666f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e60155a60 con 0x7f5e600bdf20 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e666f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60155d40 con 0x7f5e600ac390 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e5f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5e600bdf20 0x7f5e600c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e5ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e600ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e64c91640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5e60158530 0x7f5e6006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50352/0 (socket says 172.21.15.78:50352) 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e64c91640 1 -- 172.21.15.78:0/1707555551 learned_addr learned my addr 172.21.15.78:0/1707555551 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e5ffff640 1 -- 172.21.15.78:0/1707555551 >> v2:172.21.15.150:3300/0 conn(0x7f5e600bdf20 msgr2=0x7f5e600c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.528+0000 7f5e5ffff640 1 --2- 172.21.15.78:0/1707555551 >> v2:172.21.15.150:3300/0 conn(0x7f5e600bdf20 0x7f5e600c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5ffff640 1 -- 172.21.15.78:0/1707555551 >> v1:172.21.15.150:6789/0 conn(0x7f5e60158530 legacy=0x7f5e6006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5effd640 1 -- 172.21.15.78:0/1707555551 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883033854 0 0) 0x7f5e600c8de0 con 0x7f5e60158530 2024-02-19T14:58:07.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5ffff640 1 -- 172.21.15.78:0/1707555551 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600c90a0 con 0x7f5e600ac390 2024-02-19T14:58:07.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5f7fe640 1 --2- 172.21.15.78:0/1707555551 >> v2:172.21.15.150:3300/0 conn(0x7f5e600bdf20 0x7f5e600c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5ffff640 1 --2- 172.21.15.78:0/1707555551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e600ac770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097a0 tx=0x7f5e5002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=932e2f5d2d98aa65 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5effd640 1 -- 172.21.15.78:0/1707555551 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5002f050 con 0x7f5e600ac390 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5effd640 1 -- 172.21.15.78:0/1707555551 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50031700 con 0x7f5e600ac390 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.529+0000 7f5e5effd640 1 -- 172.21.15.78:0/1707555551 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50031a90 con 0x7f5e600ac390 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 -- 172.21.15.78:0/1707555551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 msgr2=0x7f5e600ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 --2- 172.21.15.78:0/1707555551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e600ac770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5e500097a0 tx=0x7f5e5002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 -- 172.21.15.78:0/1707555551 shutdown_connections 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 --2- 172.21.15.78:0/1707555551 >> v2:172.21.15.150:3300/0 conn(0x7f5e600bdf20 0x7f5e600c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 --2- 172.21.15.78:0/1707555551 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e600ac770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 -- 172.21.15.78:0/1707555551 >> 172.21.15.78:0/1707555551 conn(0x7f5e600ca890 msgr2=0x7f5e600caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 -- 172.21.15.78:0/1707555551 shutdown_connections 2024-02-19T14:58:07.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.530+0000 7f5e666f4640 1 -- 172.21.15.78:0/1707555551 wait complete. 2024-02-19T14:58:07.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.531+0000 7f5e666f4640 1 Processor -- start 2024-02-19T14:58:07.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.531+0000 7f5e666f4640 1 -- start start 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e666f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e601547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e666f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e600bdf20 0x7f5e600ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e666f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 0x7f5e6014dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e666f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c2860 con 0x7f5e60158530 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e666f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c4430 con 0x7f5e600bdf20 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e666f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60154fb0 con 0x7f5e600ac390 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 0x7f5e6014dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 0x7f5e6014dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53456/0 (socket says 172.21.15.78:53456) 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e5ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e601547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 -- 172.21.15.78:0/3700585214 learned_addr learned my addr 172.21.15.78:0/3700585214 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e5f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e600bdf20 0x7f5e600ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 -- 172.21.15.78:0/3700585214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 msgr2=0x7f5e601547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e601547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 -- 172.21.15.78:0/3700585214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e600bdf20 msgr2=0x7f5e600ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e600bdf20 0x7f5e600ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.532+0000 7f5e64c91640 1 -- 172.21.15.78:0/3700585214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e54009400 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.533+0000 7f5e64c91640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 0x7f5e6014dad0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800c700 tx=0x7f5e5800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.533+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800ce10 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.533+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5800a640 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.533+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e50009400 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.533+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5800aa70 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.533+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e600c3b00 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.534+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e60074990 con 0x7f5e60158530 2024-02-19T14:58:07.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.534+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5e5800abd0 con 0x7f5e60158530 2024-02-19T14:58:07.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.534+0000 7f5e5cff9640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e4c06b810 0x7f5e4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.535+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f5e58089c70 con 0x7f5e60158530 2024-02-19T14:58:07.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.535+0000 7f5e5ffff640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e4c06b810 0x7f5e4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.535+0000 7f5e5ffff640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e4c06b810 0x7f5e4c06dcd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5e50035000 tx=0x7f5e50008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.539+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5e58054130 con 0x7f5e60158530 2024-02-19T14:58:07.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.708+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e600c4570 con 0x7f5e60158530 2024-02-19T14:58:07.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.709+0000 7f5e5cff9640 1 -- 172.21.15.78:0/3700585214 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v158) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e58018020 con 0x7f5e60158530 2024-02-19T14:58:07.711 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e4c06b810 msgr2=0x7f5e4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e4c06b810 0x7f5e4c06dcd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5e50035000 tx=0x7f5e50008030 comp rx=0 tx=0).stop 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 msgr2=0x7f5e6014dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 0x7f5e6014dad0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800c700 tx=0x7f5e5800cbc0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 shutdown_connections 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e4c06b810 0x7f5e4c06dcd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e60158530 0x7f5e6014dad0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e600bdf20 0x7f5e600ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 --2- 172.21.15.78:0/3700585214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e600ac390 0x7f5e601547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 >> 172.21.15.78:0/3700585214 conn(0x7f5e600ca890 msgr2=0x7f5e600cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 shutdown_connections 2024-02-19T14:58:07.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.711+0000 7f5e666f4640 1 -- 172.21.15.78:0/3700585214 wait complete. 2024-02-19T14:58:07.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-19T14:58:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.824+0000 7ff03b676640 1 Processor -- start 2024-02-19T14:58:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.824+0000 7ff03b676640 1 -- start start 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff03b676640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0340c19f0 0x7ff034156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff03b676640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff03b676640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff034155a50 con 0x7ff034071410 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff03b676640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff034152af0 con 0x7ff0340c19f0 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff03b676640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff034155f80 con 0x7ff0340c1dd0 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff038c11640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff034071410 0x7ff0340c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36710/0 (socket says 172.21.15.78:36710) 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff038c11640 1 -- 172.21.15.78:0/2129139699 learned_addr learned my addr 172.21.15.78:0/2129139699 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039412640 1 --2- 172.21.15.78:0/2129139699 >> v2:172.21.15.150:3300/0 conn(0x7ff0340c19f0 0x7ff034156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039c13640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff02bfff640 1 -- 172.21.15.78:0/2129139699 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744076177 0 0) 0x7ff034155a50 con 0x7ff034071410 2024-02-19T14:58:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff02bfff640 1 -- 172.21.15.78:0/2129139699 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff020003520 con 0x7ff034071410 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039412640 1 -- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 msgr2=0x7ff0340c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039412640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039412640 1 -- 172.21.15.78:0/2129139699 >> v1:172.21.15.78:6789/0 conn(0x7ff034071410 legacy=0x7ff0340c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039412640 1 -- 172.21.15.78:0/2129139699 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff034070840 con 0x7ff0340c19f0 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.825+0000 7ff039c13640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.826+0000 7ff039412640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0340c19f0 0x7ff034156440 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff0240097a0 tx=0x7ff02402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b204e147f63d9f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.826+0000 7ff02bfff640 1 -- 172.21.15.78:0/2129139699 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02402f050 con 0x7ff0340c19f0 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.826+0000 7ff02bfff640 1 -- 172.21.15.78:0/2129139699 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff024031700 con 0x7ff0340c19f0 2024-02-19T14:58:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.826+0000 7ff02bfff640 1 -- 172.21.15.78:0/2129139699 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024031a90 con 0x7ff0340c19f0 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 -- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0340c19f0 msgr2=0x7ff034156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0340c19f0 0x7ff034156440 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff0240097a0 tx=0x7ff02402b7a0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 -- 172.21.15.78:0/2129139699 shutdown_connections 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 --2- 172.21.15.78:0/2129139699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0340c19f0 0x7ff034156440 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 -- 172.21.15.78:0/2129139699 >> 172.21.15.78:0/2129139699 conn(0x7ff0340aa370 msgr2=0x7ff0340aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 -- 172.21.15.78:0/2129139699 shutdown_connections 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.827+0000 7ff03b676640 1 -- 172.21.15.78:0/2129139699 wait complete. 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.828+0000 7ff03b676640 1 Processor -- start 2024-02-19T14:58:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.828+0000 7ff03b676640 1 -- start start 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.828+0000 7ff03b676640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff034071410 0x7ff0340c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.828+0000 7ff03b676640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 0x7ff0340ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.828+0000 7ff03b676640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340abe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff03b676640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff034155070 con 0x7ff0340c19f0 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff03b676640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0341ee070 con 0x7ff034071410 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff03b676640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0341ee350 con 0x7ff0340c1dd0 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 0x7ff0340ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 0x7ff0340ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53468/0 (socket says 172.21.15.78:53468) 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 -- 172.21.15.78:0/4085600107 learned_addr learned my addr 172.21.15.78:0/4085600107 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff039c13640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff039412640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff034071410 0x7ff0340c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 -- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 msgr2=0x7ff0340abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340abe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 -- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff034071410 msgr2=0x7ff0340c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff034071410 0x7ff0340c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 -- 172.21.15.78:0/4085600107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c009a70 con 0x7ff0340c19f0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff038c11640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 0x7ff0340ab950 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff03000c390 tx=0x7ff03000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030005ce0 con 0x7ff0340c19f0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030005e40 con 0x7ff0340c19f0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff024009400 con 0x7ff0340c19f0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0300072f0 con 0x7ff0340c19f0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.829+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0341ee910 con 0x7ff0340c19f0 2024-02-19T14:58:07.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.830+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0340c8320 con 0x7ff0340c19f0 2024-02-19T14:58:07.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.831+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff030007450 con 0x7ff0340c19f0 2024-02-19T14:58:07.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.831+0000 7ff029ffb640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01806b810 0x7ff01806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:07.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.831+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7ff030089c70 con 0x7ff0340c19f0 2024-02-19T14:58:07.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.831+0000 7ff039412640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01806b810 0x7ff01806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:07.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.832+0000 7ff039412640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01806b810 0x7ff01806dcd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff024035000 tx=0x7ff024008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:07.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:07.836+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff030054120 con 0x7ff0340c19f0 2024-02-19T14:58:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.004+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff034074990 con 0x7ff0340c19f0 2024-02-19T14:58:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.005+0000 7ff029ffb640 1 -- 172.21.15.78:0/4085600107 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v158) v1 ==== 105+0+11 (secure 0 0 0) 0x7ff030017070 con 0x7ff0340c19f0 2024-02-19T14:58:08.008 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01806b810 msgr2=0x7ff01806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01806b810 0x7ff01806dcd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff024035000 tx=0x7ff024008030 comp rx=0 tx=0).stop 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 msgr2=0x7ff0340ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 0x7ff0340ab950 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ff03000c390 tx=0x7ff03000c790 comp rx=0 tx=0).stop 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 shutdown_connections 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01806b810 0x7ff01806dcd0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0340c19f0 0x7ff0340ab950 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0340c1dd0 0x7ff0340abe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 --2- 172.21.15.78:0/4085600107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff034071410 0x7ff0340c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 >> 172.21.15.78:0/4085600107 conn(0x7ff0340aa370 msgr2=0x7ff0340aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 shutdown_connections 2024-02-19T14:58:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.007+0000 7ff03b676640 1 -- 172.21.15.78:0/4085600107 wait complete. 2024-02-19T14:58:08.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-19T14:58:08.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.124+0000 7f4db869a640 1 Processor -- start 2024-02-19T14:58:08.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.124+0000 7f4db869a640 1 -- start start 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db869a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4db00c42e0 0x7f4db0158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db869a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4db00c2f50 0x7f4db00c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db869a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db00c34a0 con 0x7f4db00c8500 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db869a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db006c090 con 0x7f4db00c42e0 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db869a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db006d6a0 con 0x7f4db00c2f50 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5c35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4db00c42e0 0x7f4db0158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db6436640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4db00c8500 0x7f4db0158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36722/0 (socket says 172.21.15.78:36722) 2024-02-19T14:58:08.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5c35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4db00c42e0 0x7f4db0158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53474/0 (socket says 172.21.15.78:53474) 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db6436640 1 -- 172.21.15.78:0/4087929190 learned_addr learned my addr 172.21.15.78:0/4087929190 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db6c37640 1 --2- 172.21.15.78:0/4087929190 >> v2:172.21.15.150:3300/0 conn(0x7f4db00c2f50 0x7f4db00c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5c35640 1 -- 172.21.15.78:0/4087929190 >> v2:172.21.15.150:3300/0 conn(0x7f4db00c2f50 msgr2=0x7f4db00c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5c35640 1 --2- 172.21.15.78:0/4087929190 >> v2:172.21.15.150:3300/0 conn(0x7f4db00c2f50 0x7f4db00c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5c35640 1 -- 172.21.15.78:0/4087929190 >> v1:172.21.15.78:6789/0 conn(0x7f4db00c8500 legacy=0x7f4db0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5c35640 1 -- 172.21.15.78:0/4087929190 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db006c8e0 con 0x7f4db00c42e0 2024-02-19T14:58:08.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.125+0000 7f4db5434640 1 -- 172.21.15.78:0/4087929190 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350205778 0 0) 0x7f4db00c34a0 con 0x7f4db00c8500 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.126+0000 7f4db5c35640 1 --2- 172.21.15.78:0/4087929190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db00c42e0 0x7f4db0158d60 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4db00adea0 tx=0x7f4da002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6897561cacd12d44 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.126+0000 7f4db5434640 1 -- 172.21.15.78:0/4087929190 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da002e070 con 0x7f4db00c42e0 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.126+0000 7f4db5434640 1 -- 172.21.15.78:0/4087929190 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da00315a0 con 0x7f4db00c42e0 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.126+0000 7f4db5434640 1 -- 172.21.15.78:0/4087929190 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0031930 con 0x7f4db00c42e0 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 -- 172.21.15.78:0/4087929190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db00c42e0 msgr2=0x7f4db0158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 --2- 172.21.15.78:0/4087929190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db00c42e0 0x7f4db0158d60 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4db00adea0 tx=0x7f4da002b840 comp rx=0 tx=0).stop 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 -- 172.21.15.78:0/4087929190 shutdown_connections 2024-02-19T14:58:08.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 --2- 172.21.15.78:0/4087929190 >> v2:172.21.15.150:3300/0 conn(0x7f4db00c2f50 0x7f4db00c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 --2- 172.21.15.78:0/4087929190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db00c42e0 0x7f4db0158d60 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 -- 172.21.15.78:0/4087929190 >> 172.21.15.78:0/4087929190 conn(0x7f4db00aa300 msgr2=0x7f4db00aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 -- 172.21.15.78:0/4087929190 shutdown_connections 2024-02-19T14:58:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.127+0000 7f4db869a640 1 -- 172.21.15.78:0/4087929190 wait complete. 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.128+0000 7f4db869a640 1 Processor -- start 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.128+0000 7f4db869a640 1 -- start start 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db00c2f50 0x7f4db01548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db00c8500 0x7f4db01503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 0x7f4db0158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0154e00 con 0x7f4db0155150 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db00705f0 con 0x7f4db00c8500 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db01594d0 con 0x7f4db00c2f50 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 0x7f4db0158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 0x7f4db0158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53482/0 (socket says 172.21.15.78:53482) 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 -- 172.21.15.78:0/1210098362 learned_addr learned my addr 172.21.15.78:0/1210098362 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db5c35640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db00c8500 0x7f4db01503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6436640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db00c2f50 0x7f4db01548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 -- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db00c2f50 msgr2=0x7f4db01548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db00c2f50 0x7f4db01548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 -- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db00c8500 msgr2=0x7f4db01503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db00c8500 0x7f4db01503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 -- 172.21.15.78:0/1210098362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0009400 con 0x7f4db0155150 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db6c37640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 0x7f4db0158810 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f4da800efc0 tx=0x7f4da800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8014480 con 0x7f4db0155150 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da800f040 con 0x7f4db0155150 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db006fa80 con 0x7f4db0155150 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8014820 con 0x7f4db0155150 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.129+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db006fcd0 con 0x7f4db0155150 2024-02-19T14:58:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.130+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db0071c20 con 0x7f4db0155150 2024-02-19T14:58:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.131+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4da8014980 con 0x7f4db0155150 2024-02-19T14:58:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.131+0000 7f4da6ffd640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9406b840 0x7f4d9406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.131+0000 7f4db6436640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9406b840 0x7f4d9406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.131+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f4da808b440 con 0x7f4db0155150 2024-02-19T14:58:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.132+0000 7f4db6436640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9406b840 0x7f4d9406dd00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4dac007fd0 tx=0x7f4dac007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.135+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4da8055a10 con 0x7f4db0155150 2024-02-19T14:58:08.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.306+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f4db0074990 con 0x7f4db0155150 2024-02-19T14:58:08.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.307+0000 7f4da6ffd640 1 -- 172.21.15.78:0/1210098362 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v158) v1 ==== 101+0+11 (secure 0 0 0) 0x7f4da8018020 con 0x7f4db0155150 2024-02-19T14:58:08.310 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:58:08.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9406b840 msgr2=0x7f4d9406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9406b840 0x7f4d9406dd00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4dac007fd0 tx=0x7f4dac007de0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 msgr2=0x7f4db0158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 0x7f4db0158810 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f4da800efc0 tx=0x7f4da800c490 comp rx=0 tx=0).stop 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 shutdown_connections 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d9406b840 0x7f4d9406dd00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4db0155150 0x7f4db0158810 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4db00c8500 0x7f4db01503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 --2- 172.21.15.78:0/1210098362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4db00c2f50 0x7f4db01548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 >> 172.21.15.78:0/1210098362 conn(0x7f4db00aa300 msgr2=0x7f4db00c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 shutdown_connections 2024-02-19T14:58:08.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.309+0000 7f4db869a640 1 -- 172.21.15.78:0/1210098362 wait complete. 2024-02-19T14:58:08.322 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.420+0000 7fee6b6f4640 1 Processor -- start 2024-02-19T14:58:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.421+0000 7fee6b6f4640 1 -- start start 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.421+0000 7fee6b6f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c42e0 0x7fee64158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.421+0000 7fee6b6f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee640c2f50 0x7fee640c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.421+0000 7fee6b6f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee640c34a0 con 0x7fee640c8500 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.421+0000 7fee6b6f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6406c090 con 0x7fee640c2f50 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.421+0000 7fee6b6f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6406d6a0 con 0x7fee640c42e0 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee69490640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee640c8500 0x7fee64158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36724/0 (socket says 172.21.15.78:36724) 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee69490640 1 -- 172.21.15.78:0/639096617 learned_addr learned my addr 172.21.15.78:0/639096617 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee69c91640 1 --2- 172.21.15.78:0/639096617 >> v2:172.21.15.150:3300/0 conn(0x7fee640c2f50 0x7fee640c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee68c8f640 1 --2- 172.21.15.78:0/639096617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c42e0 0x7fee64158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee57fff640 1 -- 172.21.15.78:0/639096617 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414914751 0 0) 0x7fee640c34a0 con 0x7fee640c8500 2024-02-19T14:58:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee57fff640 1 -- 172.21.15.78:0/639096617 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee50003520 con 0x7fee640c8500 2024-02-19T14:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee68c8f640 1 -- 172.21.15.78:0/639096617 >> v2:172.21.15.150:3300/0 conn(0x7fee640c2f50 msgr2=0x7fee640c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee68c8f640 1 --2- 172.21.15.78:0/639096617 >> v2:172.21.15.150:3300/0 conn(0x7fee640c2f50 0x7fee640c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee68c8f640 1 -- 172.21.15.78:0/639096617 >> v1:172.21.15.78:6789/0 conn(0x7fee640c8500 legacy=0x7fee64158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee57fff640 1 -- 172.21.15.78:0/639096617 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289977357 0 0) 0x7fee50003520 con 0x7fee640c8500 2024-02-19T14:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee68c8f640 1 -- 172.21.15.78:0/639096617 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6406c8e0 con 0x7fee640c42e0 2024-02-19T14:58:08.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.422+0000 7fee69c91640 1 --2- 172.21.15.78:0/639096617 >> v2:172.21.15.150:3300/0 conn(0x7fee640c2f50 0x7fee640c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.423+0000 7fee68c8f640 1 --2- 172.21.15.78:0/639096617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c42e0 0x7fee64158d60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee600097a0 tx=0x7fee6002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1c3296fc0a10de8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.423+0000 7fee57fff640 1 -- 172.21.15.78:0/639096617 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58001670 con 0x7fee640c42e0 2024-02-19T14:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.423+0000 7fee57fff640 1 -- 172.21.15.78:0/639096617 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6002e050 con 0x7fee640c42e0 2024-02-19T14:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.423+0000 7fee57fff640 1 -- 172.21.15.78:0/639096617 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60030730 con 0x7fee640c42e0 2024-02-19T14:58:08.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.423+0000 7fee6b6f4640 1 -- 172.21.15.78:0/639096617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c42e0 msgr2=0x7fee64158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.423+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/639096617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c42e0 0x7fee64158d60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee600097a0 tx=0x7fee6002b710 comp rx=0 tx=0).stop 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.424+0000 7fee6b6f4640 1 -- 172.21.15.78:0/639096617 shutdown_connections 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.424+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/639096617 >> v2:172.21.15.150:3300/0 conn(0x7fee640c2f50 0x7fee640c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.424+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/639096617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee640c42e0 0x7fee64158d60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.424+0000 7fee6b6f4640 1 -- 172.21.15.78:0/639096617 >> 172.21.15.78:0/639096617 conn(0x7fee640aa300 msgr2=0x7fee640aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.424+0000 7fee6b6f4640 1 -- 172.21.15.78:0/639096617 shutdown_connections 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.424+0000 7fee6b6f4640 1 -- 172.21.15.78:0/639096617 wait complete. 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 Processor -- start 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 -- start start 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee640c2f50 0x7fee640c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 0x7fee641eb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee641519c0 0x7fee641eb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee641eea30 con 0x7fee640c8500 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee64152560 con 0x7fee640c2f50 2024-02-19T14:58:08.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee6b6f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee641506e0 con 0x7fee641519c0 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee68c8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 0x7fee641eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee68c8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 0x7fee641eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53492/0 (socket says 172.21.15.78:53492) 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.425+0000 7fee68c8f640 1 -- 172.21.15.78:0/3631030012 learned_addr learned my addr 172.21.15.78:0/3631030012 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee69490640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee640c2f50 0x7fee640c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee69c91640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee641519c0 0x7fee641eb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee68c8f640 1 -- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee641519c0 msgr2=0x7fee641eb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee68c8f640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee641519c0 0x7fee641eb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee68c8f640 1 -- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee640c2f50 msgr2=0x7fee640c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee68c8f640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee640c2f50 0x7fee640c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee68c8f640 1 -- 172.21.15.78:0/3631030012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c009a70 con 0x7fee640c8500 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee69c91640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee641519c0 0x7fee641eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee68c8f640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 0x7fee641eb440 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fee60000c00 tx=0x7fee600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6003b810 con 0x7fee640c8500 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6003b970 con 0x7fee640c8500 2024-02-19T14:58:08.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee60009400 con 0x7fee640c8500 2024-02-19T14:58:08.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6003bd20 con 0x7fee640c8500 2024-02-19T14:58:08.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.426+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee641eeef0 con 0x7fee640c8500 2024-02-19T14:58:08.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.427+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee64071c20 con 0x7fee640c8500 2024-02-19T14:58:08.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.428+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fee60039040 con 0x7fee640c8500 2024-02-19T14:58:08.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.428+0000 7fee55ffb640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4806b810 0x7fee4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.428+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fee600abd90 con 0x7fee640c8500 2024-02-19T14:58:08.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.428+0000 7fee69490640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4806b810 0x7fee4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.429+0000 7fee69490640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4806b810 0x7fee4806dcd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fee58006ad0 tx=0x7fee58001d20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.433+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee60076360 con 0x7fee640c8500 2024-02-19T14:58:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.636+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee64074990 con 0x7fee640c8500 2024-02-19T14:58:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.637+0000 7fee55ffb640 1 -- 172.21.15.78:0/3631030012 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fee60037090 con 0x7fee640c8500 2024-02-19T14:58:08.640 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:08.640 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":141,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:03.320716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:03.320830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 141 2024-02-19T14:58:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4806b810 msgr2=0x7fee4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4806b810 0x7fee4806dcd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fee58006ad0 tx=0x7fee58001d20 comp rx=0 tx=0).stop 2024-02-19T14:58:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 msgr2=0x7fee641eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 0x7fee641eb440 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fee60000c00 tx=0x7fee600046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 shutdown_connections 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee4806b810 0x7fee4806dcd0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee640c8500 0x7fee641eb440 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee641519c0 0x7fee641eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 --2- 172.21.15.78:0/3631030012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee640c2f50 0x7fee640c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 >> 172.21.15.78:0/3631030012 conn(0x7fee640aa300 msgr2=0x7fee64153f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 shutdown_connections 2024-02-19T14:58:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.639+0000 7fee6b6f4640 1 -- 172.21.15.78:0/3631030012 wait complete. 2024-02-19T14:58:08.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.750+0000 7f28fe94b640 1 Processor -- start 2024-02-19T14:58:08.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.750+0000 7f28fe94b640 1 -- start start 2024-02-19T14:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fe94b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28f8152830 0x7f28f80c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fe94b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f28f80c5760 0x7f28f80c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fe94b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f81586a0 con 0x7f28f80c77c0 2024-02-19T14:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fe94b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f80c6550 con 0x7f28f80c5760 2024-02-19T14:58:08.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fe94b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f8159320 con 0x7f28f8152830 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28f77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f28f80c5760 0x7f28f80c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fcee8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f28f80c77c0 0x7f28f806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36732/0 (socket says 172.21.15.78:36732) 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28fcee8640 1 -- 172.21.15.78:0/1340621105 learned_addr learned my addr 172.21.15.78:0/1340621105 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.751+0000 7f28f7fff640 1 --2- 172.21.15.78:0/1340621105 >> v2:172.21.15.150:3300/0 conn(0x7f28f8152830 0x7f28f80c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f7fff640 1 -- 172.21.15.78:0/1340621105 >> v2:172.21.15.78:3300/0 conn(0x7f28f80c5760 msgr2=0x7f28f80c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f7fff640 1 --2- 172.21.15.78:0/1340621105 >> v2:172.21.15.78:3300/0 conn(0x7f28f80c5760 0x7f28f80c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f7fff640 1 -- 172.21.15.78:0/1340621105 >> v1:172.21.15.78:6789/0 conn(0x7f28f80c77c0 legacy=0x7f28f806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f7fff640 1 -- 172.21.15.78:0/1340621105 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8158980 con 0x7f28f8152830 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f6ffd640 1 -- 172.21.15.78:0/1340621105 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318159473 0 0) 0x7f28f81586a0 con 0x7f28f80c77c0 2024-02-19T14:58:08.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f77fe640 1 --2- 172.21.15.78:0/1340621105 >> v2:172.21.15.78:3300/0 conn(0x7f28f80c5760 0x7f28f80c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.752+0000 7f28f7fff640 1 --2- 172.21.15.78:0/1340621105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f8152830 0x7f28f80c5220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f28f80c7250 tx=0x7f28e802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40d11b073d47ae1b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28f6ffd640 1 -- 172.21.15.78:0/1340621105 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e802e070 con 0x7f28f8152830 2024-02-19T14:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28f6ffd640 1 -- 172.21.15.78:0/1340621105 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e80315a0 con 0x7f28f8152830 2024-02-19T14:58:08.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28f6ffd640 1 -- 172.21.15.78:0/1340621105 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8031930 con 0x7f28f8152830 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 -- 172.21.15.78:0/1340621105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f8152830 msgr2=0x7f28f80c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 --2- 172.21.15.78:0/1340621105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f8152830 0x7f28f80c5220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f28f80c7250 tx=0x7f28e802b840 comp rx=0 tx=0).stop 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 -- 172.21.15.78:0/1340621105 shutdown_connections 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 --2- 172.21.15.78:0/1340621105 >> v2:172.21.15.78:3300/0 conn(0x7f28f80c5760 0x7f28f80c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 --2- 172.21.15.78:0/1340621105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f8152830 0x7f28f80c5220 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 -- 172.21.15.78:0/1340621105 >> 172.21.15.78:0/1340621105 conn(0x7f28f80ca8a0 msgr2=0x7f28f80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 -- 172.21.15.78:0/1340621105 shutdown_connections 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.753+0000 7f28fe94b640 1 -- 172.21.15.78:0/1340621105 wait complete. 2024-02-19T14:58:08.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.754+0000 7f28fe94b640 1 Processor -- start 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.754+0000 7f28fe94b640 1 -- start start 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fe94b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 0x7f28f81ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fe94b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f80c77c0 0x7f28f80c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fe94b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28f8152830 0x7f28f80c31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fe94b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f80c27f0 con 0x7f28f80c5760 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fe94b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f81ee890 con 0x7f28f80c77c0 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fe94b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f80aa360 con 0x7f28f8152830 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28f7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 0x7f28f81ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28f7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 0x7f28f81ede30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53516/0 (socket says 172.21.15.78:53516) 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28f7fff640 1 -- 172.21.15.78:0/2591926563 learned_addr learned my addr 172.21.15.78:0/2591926563 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28f77fe640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f80c77c0 0x7f28f80c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.755+0000 7f28fcee8640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28f8152830 0x7f28f80c31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f7fff640 1 -- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28f8152830 msgr2=0x7f28f80c31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f7fff640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28f8152830 0x7f28f80c31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f7fff640 1 -- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f80c77c0 msgr2=0x7f28f80c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f7fff640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f80c77c0 0x7f28f80c2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f7fff640 1 -- 172.21.15.78:0/2591926563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec009400 con 0x7f28f80c5760 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f77fe640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f80c77c0 0x7f28f80c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f7fff640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 0x7f28f81ede30 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009890 tx=0x7f28e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8032840 con 0x7f28f80c5760 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e8031f00 con 0x7f28f80c5760 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e8009400 con 0x7f28f80c5760 2024-02-19T14:58:08.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8032be0 con 0x7f28f80c5760 2024-02-19T14:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.756+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f80c3b50 con 0x7f28f80c5760 2024-02-19T14:58:08.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.757+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f80aa880 con 0x7f28f80c5760 2024-02-19T14:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.758+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f28e8032d40 con 0x7f28f80c5760 2024-02-19T14:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.758+0000 7f28f4ff9640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28e406b810 0x7f28e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.758+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f28e80ac4b0 con 0x7f28f80c5760 2024-02-19T14:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.758+0000 7f28f77fe640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28e406b810 0x7f28e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:08.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.759+0000 7f28f77fe640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28e406b810 0x7f28e406dcd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f28f80c16e0 tx=0x7f28ec009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:08.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.763+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f28e80769f0 con 0x7f28f80c5760 2024-02-19T14:58:08.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.935+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28f8074990 con 0x7f28f80c5760 2024-02-19T14:58:08.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.936+0000 7f28f4ff9640 1 -- 172.21.15.78:0/2591926563 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+20847 (secure 0 0 0) 0x7f28e807a0b0 con 0x7f28f80c5760 2024-02-19T14:58:08.939 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:08.939 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":158,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:01.312276+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:08.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28e406b810 msgr2=0x7f28e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28e406b810 0x7f28e406dcd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f28f80c16e0 tx=0x7f28ec009190 comp rx=0 tx=0).stop 2024-02-19T14:58:08.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 msgr2=0x7f28f81ede30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 0x7f28f81ede30 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009890 tx=0x7f28e80046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 shutdown_connections 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28e406b810 0x7f28e406dcd0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28f8152830 0x7f28f80c31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f80c77c0 0x7f28f80c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 --2- 172.21.15.78:0/2591926563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28f80c5760 0x7f28f81ede30 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 >> 172.21.15.78:0/2591926563 conn(0x7f28f80ca8a0 msgr2=0x7f28f8150680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 shutdown_connections 2024-02-19T14:58:08.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:08.938+0000 7f28fe94b640 1 -- 172.21.15.78:0/2591926563 wait complete. 2024-02-19T14:58:08.952 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:58:08.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:09.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.049+0000 7fb1ab5b5640 1 Processor -- start 2024-02-19T14:58:09.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1ab5b5640 1 -- start start 2024-02-19T14:58:09.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1ab5b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb1a40c8500 0x7fb1a40c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1ab5b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c42e0 0x7fb1a4158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1ab5b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40c34a0 con 0x7fb1a40c2f50 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1ab5b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a406c090 con 0x7fb1a40c8500 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1ab5b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a406d6a0 con 0x7fb1a40c42e0 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1a9351640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb1a40c8500 0x7fb1a40c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1a9351640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb1a40c8500 0x7fb1a40c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53532/0 (socket says 172.21.15.78:53532) 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1a9351640 1 -- 172.21.15.78:0/4038546734 learned_addr learned my addr 172.21.15.78:0/4038546734 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.050+0000 7fb1a8b50640 1 --2- 172.21.15.78:0/4038546734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c42e0 0x7fb1a4158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.051+0000 7fb1a9351640 1 -- 172.21.15.78:0/4038546734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c42e0 msgr2=0x7fb1a4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.051+0000 7fb1a9351640 1 --2- 172.21.15.78:0/4038546734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c42e0 0x7fb1a4158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.051+0000 7fb1a9351640 1 -- 172.21.15.78:0/4038546734 >> v1:172.21.15.78:6789/0 conn(0x7fb1a40c2f50 legacy=0x7fb1a4158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.051+0000 7fb1a9351640 1 -- 172.21.15.78:0/4038546734 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a406c8e0 con 0x7fb1a40c8500 2024-02-19T14:58:09.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.051+0000 7fb19bfff640 1 -- 172.21.15.78:0/4038546734 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472453738 0 0) 0x7fb1a40c34a0 con 0x7fb1a40c2f50 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.051+0000 7fb1a9351640 1 --2- 172.21.15.78:0/4038546734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a40c8500 0x7fb1a40c88e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009e50 tx=0x7fb1a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfa045810815a1e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb19bfff640 1 -- 172.21.15.78:0/4038546734 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a002ca70 con 0x7fb1a40c8500 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb19bfff640 1 -- 172.21.15.78:0/4038546734 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a002cbd0 con 0x7fb1a40c8500 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb19bfff640 1 -- 172.21.15.78:0/4038546734 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00343f0 con 0x7fb1a40c8500 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/4038546734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a40c8500 msgr2=0x7fb1a40c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/4038546734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a40c8500 0x7fb1a40c88e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009e50 tx=0x7fb1a002bed0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/4038546734 shutdown_connections 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/4038546734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c42e0 0x7fb1a4158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/4038546734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a40c8500 0x7fb1a40c88e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/4038546734 >> 172.21.15.78:0/4038546734 conn(0x7fb1a40aa300 msgr2=0x7fb1a40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/4038546734 shutdown_connections 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.052+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/4038546734 wait complete. 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.053+0000 7fb1ab5b5640 1 Processor -- start 2024-02-19T14:58:09.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.053+0000 7fb1ab5b5640 1 -- start start 2024-02-19T14:58:09.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1ab5b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1a40c2f50 0x7fb1a406f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1ab5b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c8500 0x7fb1a406f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1ab5b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 0x7fb1a4145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1ab5b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a4070850 con 0x7fb1a414dc60 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1ab5b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a414e720 con 0x7fb1a40c2f50 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1ab5b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a41eea50 con 0x7fb1a40c8500 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 0x7fb1a4145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a8b50640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c8500 0x7fb1a406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9351640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1a40c2f50 0x7fb1a406f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 0x7fb1a4145700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53540/0 (socket says 172.21.15.78:53540) 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 -- 172.21.15.78:0/2359926848 learned_addr learned my addr 172.21.15.78:0/2359926848 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:09.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 -- 172.21.15.78:0/2359926848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c8500 msgr2=0x7fb1a406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c8500 0x7fb1a406f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 -- 172.21.15.78:0/2359926848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1a40c2f50 msgr2=0x7fb1a406f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1a40c2f50 0x7fb1a406f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.054+0000 7fb1a9b52640 1 -- 172.21.15.78:0/2359926848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a0009b30 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.055+0000 7fb1a9b52640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 0x7fb1a4145700 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00c480 tx=0x7fb19c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.055+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c005480 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.055+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c0035f0 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.055+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a41eed30 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.055+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0039a0 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.055+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a40704e0 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.056+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a4071c20 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.056+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb19c003b00 con 0x7fb1a414dc60 2024-02-19T14:58:09.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.056+0000 7fb199ffb640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb18806b810 0x7fb18806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.057+0000 7fb1a9351640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb18806b810 0x7fb18806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.057+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fb19c08a110 con 0x7fb1a414dc60 2024-02-19T14:58:09.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.057+0000 7fb1a9351640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb18806b810 0x7fb18806dcd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0035000 tx=0x7fb1a0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.061+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb19c058210 con 0x7fb1a414dc60 2024-02-19T14:58:09.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.266+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1a4070a00 con 0x7fb1a414dc60 2024-02-19T14:58:09.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.267+0000 7fb199ffb640 1 -- 172.21.15.78:0/2359926848 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6204 (secure 0 0 0) 0x7fb19c057da0 con 0x7fb1a414dc60 2024-02-19T14:58:09.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:09.270 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":141,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:03.320716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:03.320830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:09.270 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 141 2024-02-19T14:58:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb18806b810 msgr2=0x7fb18806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb18806b810 0x7fb18806dcd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0035000 tx=0x7fb1a0030000 comp rx=0 tx=0).stop 2024-02-19T14:58:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 msgr2=0x7fb1a4145700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 0x7fb1a4145700 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00c480 tx=0x7fb19c00c940 comp rx=0 tx=0).stop 2024-02-19T14:58:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 shutdown_connections 2024-02-19T14:58:09.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb18806b810 0x7fb18806dcd0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1a414dc60 0x7fb1a4145700 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1a40c8500 0x7fb1a406f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 --2- 172.21.15.78:0/2359926848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1a40c2f50 0x7fb1a406f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 >> 172.21.15.78:0/2359926848 conn(0x7fb1a40aa300 msgr2=0x7fb1a40c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:09.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 shutdown_connections 2024-02-19T14:58:09.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.269+0000 7fb1ab5b5640 1 -- 172.21.15.78:0/2359926848 wait complete. 2024-02-19T14:58:09.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:09.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.380+0000 7f4a262f4640 1 Processor -- start 2024-02-19T14:58:09.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.380+0000 7f4a262f4640 1 -- start start 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a262f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a2006e350 0x7f4a200c0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a262f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a201529f0 con 0x7f4a20152610 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a262f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200bf9b0 con 0x7f4a200c06b0 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a262f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2006da70 con 0x7f4a2006e350 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a1ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4a20152610 0x7f4a20158f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36750/0 (socket says 172.21.15.78:36750) 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a1f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a2006e350 0x7f4a200c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a1ffff640 1 -- 172.21.15.78:0/2641196114 learned_addr learned my addr 172.21.15.78:0/2641196114 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a1effd640 1 -- 172.21.15.78:0/2641196114 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480192930 0 0) 0x7f4a201529f0 con 0x7f4a20152610 2024-02-19T14:58:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a1effd640 1 -- 172.21.15.78:0/2641196114 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a08003520 con 0x7f4a20152610 2024-02-19T14:58:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.381+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/2641196114 >> v1:172.21.15.150:6789/0 conn(0x7f4a200c06b0 legacy=0x7f4a200c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.382+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/2641196114 >> v1:172.21.15.78:6789/0 conn(0x7f4a20152610 legacy=0x7f4a20158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.382+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/2641196114 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2006e8c0 con 0x7f4a2006e350 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.382+0000 7f4a1f7fe640 1 --2- 172.21.15.78:0/2641196114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2006e350 0x7f4a200c0170 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f4a140097a0 tx=0x7f4a1402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecfdf5567610920 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.382+0000 7f4a1effd640 1 -- 172.21.15.78:0/2641196114 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14030610 con 0x7f4a2006e350 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.382+0000 7f4a1effd640 1 -- 172.21.15.78:0/2641196114 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14030770 con 0x7f4a2006e350 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.382+0000 7f4a1effd640 1 -- 172.21.15.78:0/2641196114 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14030b00 con 0x7f4a2006e350 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 -- 172.21.15.78:0/2641196114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2006e350 msgr2=0x7f4a200c0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 --2- 172.21.15.78:0/2641196114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2006e350 0x7f4a200c0170 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f4a140097a0 tx=0x7f4a1402b960 comp rx=0 tx=0).stop 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 -- 172.21.15.78:0/2641196114 shutdown_connections 2024-02-19T14:58:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 --2- 172.21.15.78:0/2641196114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a2006e350 0x7f4a200c0170 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 -- 172.21.15.78:0/2641196114 >> 172.21.15.78:0/2641196114 conn(0x7f4a2006b270 msgr2=0x7f4a2006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 -- 172.21.15.78:0/2641196114 shutdown_connections 2024-02-19T14:58:09.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.383+0000 7f4a262f4640 1 -- 172.21.15.78:0/2641196114 wait complete. 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.384+0000 7f4a262f4640 1 Processor -- start 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.384+0000 7f4a262f4640 1 -- start start 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a262f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a2006e350 0x7f4a2014a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a262f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 0x7f4a200ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a262f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a20152610 0x7f4a200aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a262f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a201edcf0 con 0x7f4a200c06b0 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a262f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a201edfd0 con 0x7f4a20152610 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a262f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a201ee2b0 con 0x7f4a2006e350 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 0x7f4a200ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 0x7f4a200ab890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53564/0 (socket says 172.21.15.78:53564) 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/880473580 learned_addr learned my addr 172.21.15.78:0/880473580 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a2006e350 0x7f4a2014a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a24891640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a20152610 0x7f4a200aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a2006e350 msgr2=0x7f4a2014a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a2006e350 0x7f4a2014a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a20152610 msgr2=0x7f4a200aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a20152610 0x7f4a200aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 -- 172.21.15.78:0/880473580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14009400 con 0x7f4a200c06b0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.385+0000 7f4a1f7fe640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 0x7f4a200ab890 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f4a140098c0 tx=0x7f4a14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.386+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1403e040 con 0x7f4a200c06b0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.386+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14039cb0 con 0x7f4a200c06b0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.386+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a201ee590 con 0x7f4a200c06b0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.386+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14044630 con 0x7f4a200c06b0 2024-02-19T14:58:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.386+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a200a8d50 con 0x7f4a200c06b0 2024-02-19T14:58:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.387+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a200a9000 con 0x7f4a200c06b0 2024-02-19T14:58:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.387+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4a1404d3b0 con 0x7f4a200c06b0 2024-02-19T14:58:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.388+0000 7f4a1cff9640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a0c06b810 0x7f4a0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.388+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f4a140aca10 con 0x7f4a200c06b0 2024-02-19T14:58:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.388+0000 7f4a1ffff640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a0c06b810 0x7f4a0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.388+0000 7f4a1ffff640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a0c06b810 0x7f4a0c06dcd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4a100011e0 tx=0x7f4a10007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.392+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a14076fe0 con 0x7f4a200c06b0 2024-02-19T14:58:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.598+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a20074990 con 0x7f4a200c06b0 2024-02-19T14:58:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.599+0000 7f4a1cff9640 1 -- 172.21.15.78:0/880473580 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6204 (secure 0 0 0) 0x7f4a1407a6a0 con 0x7f4a200c06b0 2024-02-19T14:58:09.602 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:09.602 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":141,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9045,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:45.095751+0000","modified":"2024-02-19T14:58:03.320716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7771":{"gid":7771,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2395461731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2395461731},{"type":"v1","addr":"172.21.15.150:6835","nonce":2395461731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:03.320830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":158,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7768},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7768":{"gid":7768,"name":"d","rank":0,"incarnation":138,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6834/1911236290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1911236290},{"type":"v1","addr":"172.21.15.150:6834","nonce":1911236290}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:09.602 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 141 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a0c06b810 msgr2=0x7f4a0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a0c06b810 0x7f4a0c06dcd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4a100011e0 tx=0x7f4a10007d30 comp rx=0 tx=0).stop 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 msgr2=0x7f4a200ab890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 0x7f4a200ab890 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f4a140098c0 tx=0x7f4a14004730 comp rx=0 tx=0).stop 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 shutdown_connections 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a0c06b810 0x7f4a0c06dcd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a200c06b0 0x7f4a200ab890 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a20152610 0x7f4a200aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 --2- 172.21.15.78:0/880473580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a2006e350 0x7f4a2014a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 >> 172.21.15.78:0/880473580 conn(0x7f4a2006b270 msgr2=0x7f4a200c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 shutdown_connections 2024-02-19T14:58:09.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.601+0000 7f4a262f4640 1 -- 172.21.15.78:0/880473580 wait complete. 2024-02-19T14:58:09.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:09.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.712+0000 7f4ca89ac640 1 Processor -- start 2024-02-19T14:58:09.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.712+0000 7f4ca89ac640 1 -- start start 2024-02-19T14:58:09.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca89ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ca40c8500 0x7f4ca40c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca89ac640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ca40c2f50 0x7f4ca4158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca89ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca40c34a0 con 0x7f4ca40c42e0 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca89ac640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca406c090 con 0x7f4ca40c2f50 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca89ac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca406d6a0 con 0x7f4ca40c8500 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca2d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ca40c2f50 0x7f4ca4158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4c9bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4ca40c42e0 0x7f4ca4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36752/0 (socket says 172.21.15.78:36752) 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca2d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ca40c2f50 0x7f4ca4158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53576/0 (socket says 172.21.15.78:53576) 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4c9bfff640 1 -- 172.21.15.78:0/722977707 learned_addr learned my addr 172.21.15.78:0/722977707 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca259c640 1 --2- 172.21.15.78:0/722977707 >> v2:172.21.15.150:3300/0 conn(0x7f4ca40c8500 0x7f4ca40c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/722977707 >> v2:172.21.15.150:3300/0 conn(0x7f4ca40c8500 msgr2=0x7f4ca40c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca2d9d640 1 --2- 172.21.15.78:0/722977707 >> v2:172.21.15.150:3300/0 conn(0x7f4ca40c8500 0x7f4ca40c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/722977707 >> v1:172.21.15.78:6789/0 conn(0x7f4ca40c42e0 legacy=0x7f4ca4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/722977707 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca406c8e0 con 0x7f4ca40c2f50 2024-02-19T14:58:09.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.713+0000 7f4ca1d9b640 1 -- 172.21.15.78:0/722977707 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434665412 0 0) 0x7f4ca40c34a0 con 0x7f4ca40c42e0 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca2d9d640 1 --2- 172.21.15.78:0/722977707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40c2f50 0x7f4ca4158d60 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40c9350 tx=0x7f4c9c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8e48a98e53855b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca1d9b640 1 -- 172.21.15.78:0/722977707 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c02cbd0 con 0x7f4ca40c2f50 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca1d9b640 1 -- 172.21.15.78:0/722977707 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c02cd30 con 0x7f4ca40c2f50 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca1d9b640 1 -- 172.21.15.78:0/722977707 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c02cea0 con 0x7f4ca40c2f50 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 -- 172.21.15.78:0/722977707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40c2f50 msgr2=0x7f4ca4158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/722977707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40c2f50 0x7f4ca4158d60 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40c9350 tx=0x7f4c9c02be20 comp rx=0 tx=0).stop 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 -- 172.21.15.78:0/722977707 shutdown_connections 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/722977707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40c2f50 0x7f4ca4158d60 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/722977707 >> v2:172.21.15.150:3300/0 conn(0x7f4ca40c8500 0x7f4ca40c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 -- 172.21.15.78:0/722977707 >> 172.21.15.78:0/722977707 conn(0x7f4ca40aa300 msgr2=0x7f4ca40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.714+0000 7f4ca89ac640 1 -- 172.21.15.78:0/722977707 shutdown_connections 2024-02-19T14:58:09.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.715+0000 7f4ca89ac640 1 -- 172.21.15.78:0/722977707 wait complete. 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 Processor -- start 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 -- start start 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ca40c2f50 0x7f4ca40acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ca40c8500 0x7f4ca414af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 0x7f4ca4156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40abe00 con 0x7f4ca4159030 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40c5800 con 0x7f4ca40c8500 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca89ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca41493f0 con 0x7f4ca40c2f50 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.716+0000 7f4ca2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 0x7f4ca4156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 0x7f4ca4156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53588/0 (socket says 172.21.15.78:53588) 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/292503337 learned_addr learned my addr 172.21.15.78:0/292503337 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:09.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca259c640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ca40c2f50 0x7f4ca40acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4c9bfff640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ca40c8500 0x7f4ca414af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ca40c2f50 msgr2=0x7f4ca40acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ca40c2f50 0x7f4ca40acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ca40c8500 msgr2=0x7f4ca414af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ca40c8500 0x7f4ca414af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 -- 172.21.15.78:0/292503337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c009a70 con 0x7f4ca4159030 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4c9bfff640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ca40c8500 0x7f4ca414af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca2d9d640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 0x7f4ca4156a80 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009d90 tx=0x7f4c9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c02cea0 con 0x7f4ca4159030 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c003bb0 con 0x7f4ca4159030 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c009450 con 0x7f4ca4159030 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca40ac050 con 0x7f4ca4159030 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.717+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca40ac610 con 0x7f4ca4159030 2024-02-19T14:58:09.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.718+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca4071c20 con 0x7f4ca4159030 2024-02-19T14:58:09.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.719+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4c9c004030 con 0x7f4ca4159030 2024-02-19T14:58:09.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.719+0000 7f4c9affd640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c8406b840 0x7f4c8406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:09.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.719+0000 7f4ca259c640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c8406b840 0x7f4c8406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:09.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.719+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f4c9c0ace00 con 0x7f4ca4159030 2024-02-19T14:58:09.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.720+0000 7f4ca259c640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c8406b840 0x7f4c8406dd00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009aa0 tx=0x7f4c94009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:09.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.724+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4c9c077320 con 0x7f4ca4159030 2024-02-19T14:58:09.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.900+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ca4074990 con 0x7f4ca4159030 2024-02-19T14:58:09.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.901+0000 7f4c9affd640 1 -- 172.21.15.78:0/292503337 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+20847 (secure 0 0 0) 0x7f4c9c07a9e0 con 0x7f4ca4159030 2024-02-19T14:58:09.904 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:09.904 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":158,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:01.312276+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:57:43.310860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-19T14:57:44.061255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:09.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c8406b840 msgr2=0x7f4c8406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c8406b840 0x7f4c8406dd00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009aa0 tx=0x7f4c94009190 comp rx=0 tx=0).stop 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 msgr2=0x7f4ca4156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 0x7f4ca4156a80 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009d90 tx=0x7f4c9c004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 shutdown_connections 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c8406b840 0x7f4c8406dd00 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca4159030 0x7f4ca4156a80 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ca40c8500 0x7f4ca414af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 --2- 172.21.15.78:0/292503337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ca40c2f50 0x7f4ca40acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 >> 172.21.15.78:0/292503337 conn(0x7f4ca40aa300 msgr2=0x7f4ca4155410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.903+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 shutdown_connections 2024-02-19T14:58:09.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:09.904+0000 7f4ca89ac640 1 -- 172.21.15.78:0/292503337 wait complete. 2024-02-19T14:58:09.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:58:10.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.015+0000 7fd430d2c640 1 Processor -- start 2024-02-19T14:58:10.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.015+0000 7fd430d2c640 1 -- start start 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.015+0000 7fd430d2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0ac390 0x7fd42c0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.015+0000 7fd430d2c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd42c0bdf20 0x7fd42c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.015+0000 7fd430d2c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c0c8de0 con 0x7fd42c158530 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd430d2c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd42c155a60 con 0x7fd42c0bdf20 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd430d2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c155d40 con 0x7fd42c0ac390 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd429d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd42c0bdf20 0x7fd42c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd42ad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd42c158530 0x7fd42c06c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50366/0 (socket says 172.21.15.78:50366) 2024-02-19T14:58:10.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd42ad9d640 1 -- 172.21.15.78:0/4063907470 learned_addr learned my addr 172.21.15.78:0/4063907470 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:10.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd42a59c640 1 --2- 172.21.15.78:0/4063907470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0ac390 0x7fd42c0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd429d9b640 1 -- 172.21.15.78:0/4063907470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0ac390 msgr2=0x7fd42c0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:10.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd429d9b640 1 --2- 172.21.15.78:0/4063907470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0ac390 0x7fd42c0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd42959a640 1 -- 172.21.15.78:0/4063907470 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355011922 0 0) 0x7fd42c0c8de0 con 0x7fd42c158530 2024-02-19T14:58:10.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd429d9b640 1 -- 172.21.15.78:0/4063907470 >> v1:172.21.15.150:6789/0 conn(0x7fd42c158530 legacy=0x7fd42c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.016+0000 7fd429d9b640 1 -- 172.21.15.78:0/4063907470 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c0c90a0 con 0x7fd42c0bdf20 2024-02-19T14:58:10.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd429d9b640 1 --2- 172.21.15.78:0/4063907470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0bdf20 0x7fd42c0c55f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fd4200097a0 tx=0x7fd42002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ac6f14acb15852 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd42959a640 1 -- 172.21.15.78:0/4063907470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42002f050 con 0x7fd42c0bdf20 2024-02-19T14:58:10.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd42959a640 1 -- 172.21.15.78:0/4063907470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd420031700 con 0x7fd42c0bdf20 2024-02-19T14:58:10.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd42959a640 1 -- 172.21.15.78:0/4063907470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd420031a90 con 0x7fd42c0bdf20 2024-02-19T14:58:10.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd430d2c640 1 -- 172.21.15.78:0/4063907470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0bdf20 msgr2=0x7fd42c0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4063907470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0bdf20 0x7fd42c0c55f0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fd4200097a0 tx=0x7fd42002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd430d2c640 1 -- 172.21.15.78:0/4063907470 shutdown_connections 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4063907470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c0bdf20 0x7fd42c0c55f0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4063907470 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0ac390 0x7fd42c0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.017+0000 7fd430d2c640 1 -- 172.21.15.78:0/4063907470 >> 172.21.15.78:0/4063907470 conn(0x7fd42c0ca890 msgr2=0x7fd42c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.018+0000 7fd430d2c640 1 -- 172.21.15.78:0/4063907470 shutdown_connections 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.018+0000 7fd430d2c640 1 -- 172.21.15.78:0/4063907470 wait complete. 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 Processor -- start 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 -- start start 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0ac390 0x7fd42c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0bdf20 0x7fd42c0bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 0x7fd42c0bd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c0c0450 con 0x7fd42c158530 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c0c48b0 con 0x7fd42c0ac390 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd430d2c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c157440 con 0x7fd42c0bdf20 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd42ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 0x7fd42c0bd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.019+0000 7fd42ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 0x7fd42c0bd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53602/0 (socket says 172.21.15.78:53602) 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 -- 172.21.15.78:0/4237707575 learned_addr learned my addr 172.21.15.78:0/4237707575 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd429d9b640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0bdf20 0x7fd42c0bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42a59c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0ac390 0x7fd42c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 -- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0bdf20 msgr2=0x7fd42c0bccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0bdf20 0x7fd42c0bccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 -- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0ac390 msgr2=0x7fd42c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0ac390 0x7fd42c157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 -- 172.21.15.78:0/4237707575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420009400 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd42ad9d640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 0x7fd42c0bd230 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fd42400c700 tx=0x7fd42400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42400ce10 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42400a640 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42c0bd770 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42400aa30 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.020+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd42c1eb1d0 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.021+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42c0751f0 con 0x7fd42c158530 2024-02-19T14:58:10.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.022+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd42400ab90 con 0x7fd42c158530 2024-02-19T14:58:10.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.022+0000 7fd41affd640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.022+0000 7fd42a59c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.022+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fd424089dd0 con 0x7fd42c158530 2024-02-19T14:58:10.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.023+0000 7fd42a59c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009b40 tx=0x7fd41c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.027+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd424054310 con 0x7fd42c158530 2024-02-19T14:58:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.233+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd42c074990 con 0x7fd42c158530 2024-02-19T14:58:10.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:58:10.234+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:58:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.235+0000 7fd41affd640 1 -- 172.21.15.78:0/4237707575 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v142) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd4240579d0 con 0x7fd42c158530 2024-02-19T14:58:10.238 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:58:10.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.238+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 msgr2=0x7fd40c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.238+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009b40 tx=0x7fd41c009190 comp rx=0 tx=0).stop 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.238+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 msgr2=0x7fd42c0bd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.238+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 0x7fd42c0bd230 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fd42400c700 tx=0x7fd42400cbc0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.238+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 shutdown_connections 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd40c06b810 0x7fd40c06dcd0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd42c158530 0x7fd42c0bd230 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd42c0bdf20 0x7fd42c0bccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 --2- 172.21.15.78:0/4237707575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd42c0ac390 0x7fd42c157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 >> 172.21.15.78:0/4237707575 conn(0x7fd42c0ca890 msgr2=0x7fd42c0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 shutdown_connections 2024-02-19T14:58:10.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.239+0000 7fd430d2c640 1 -- 172.21.15.78:0/4237707575 wait complete. 2024-02-19T14:58:10.243 INFO:tasks.ceph.mds.b.smithi150.stderr: -122> 2024-02-19T14:58:06.023+0000 7fd74e277640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:10.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:58:10.266 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:58:10.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.353+0000 7f107149a640 1 Processor -- start 2024-02-19T14:58:10.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.353+0000 7f107149a640 1 -- start start 2024-02-19T14:58:10.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f107149a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f106c152700 0x7f106c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f107149a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c8e00 0x7f106c15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f107149a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c1551f0 con 0x7f106c0c5730 2024-02-19T14:58:10.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f107149a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c155490 con 0x7f106c152700 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f107149a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c155770 con 0x7f106c0c8e00 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f106c0c5730 0x7f106c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36754/0 (socket says 172.21.15.78:36754) 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106a7fc640 1 -- 172.21.15.78:0/231976018 learned_addr learned my addr 172.21.15.78:0/231976018 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106b7fe640 1 --2- 172.21.15.78:0/231976018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c8e00 0x7f106c15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106affd640 1 --2- 172.21.15.78:0/231976018 >> v2:172.21.15.150:3300/0 conn(0x7f106c152700 0x7f106c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f1069ffb640 1 -- 172.21.15.78:0/231976018 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660863505 0 0) 0x7f106c1551f0 con 0x7f106c0c5730 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f1069ffb640 1 -- 172.21.15.78:0/231976018 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1054003520 con 0x7f106c0c5730 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106b7fe640 1 -- 172.21.15.78:0/231976018 >> v2:172.21.15.150:3300/0 conn(0x7f106c152700 msgr2=0x7f106c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106b7fe640 1 --2- 172.21.15.78:0/231976018 >> v2:172.21.15.150:3300/0 conn(0x7f106c152700 0x7f106c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106b7fe640 1 -- 172.21.15.78:0/231976018 >> v1:172.21.15.78:6789/0 conn(0x7f106c0c5730 legacy=0x7f106c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106b7fe640 1 -- 172.21.15.78:0/231976018 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c155a50 con 0x7f106c0c8e00 2024-02-19T14:58:10.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.354+0000 7f106affd640 1 --2- 172.21.15.78:0/231976018 >> v2:172.21.15.150:3300/0 conn(0x7f106c152700 0x7f106c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.355+0000 7f106b7fe640 1 --2- 172.21.15.78:0/231976018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c8e00 0x7f106c15f980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1064009d90 tx=0x7f106402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f63e9a2d3701a860 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.355+0000 7f1069ffb640 1 -- 172.21.15.78:0/231976018 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106402cc50 con 0x7f106c0c8e00 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.355+0000 7f1069ffb640 1 -- 172.21.15.78:0/231976018 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106402cdb0 con 0x7f106c0c8e00 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.355+0000 7f1069ffb640 1 -- 172.21.15.78:0/231976018 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064034660 con 0x7f106c0c8e00 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 -- 172.21.15.78:0/231976018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c8e00 msgr2=0x7f106c15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 --2- 172.21.15.78:0/231976018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c8e00 0x7f106c15f980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1064009d90 tx=0x7f106402bea0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 -- 172.21.15.78:0/231976018 shutdown_connections 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 --2- 172.21.15.78:0/231976018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c8e00 0x7f106c15f980 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 --2- 172.21.15.78:0/231976018 >> v2:172.21.15.150:3300/0 conn(0x7f106c152700 0x7f106c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 -- 172.21.15.78:0/231976018 >> 172.21.15.78:0/231976018 conn(0x7f106c0cac90 msgr2=0x7f106c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:10.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 -- 172.21.15.78:0/231976018 shutdown_connections 2024-02-19T14:58:10.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.356+0000 7f107149a640 1 -- 172.21.15.78:0/231976018 wait complete. 2024-02-19T14:58:10.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.357+0000 7f107149a640 1 Processor -- start 2024-02-19T14:58:10.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.357+0000 7f107149a640 1 -- start start 2024-02-19T14:58:10.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c5730 0x7f106c0ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 0x7f106c1eb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106c152700 0x7f106c148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0abcc0 con 0x7f106c0c8e00 2024-02-19T14:58:10.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1eb580 con 0x7f106c152700 2024-02-19T14:58:10.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0c1d90 con 0x7f106c0c5730 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 0x7f106c1eb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 0x7f106c1eb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53606/0 (socket says 172.21.15.78:53606) 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 -- 172.21.15.78:0/523849253 learned_addr learned my addr 172.21.15.78:0/523849253 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106affd640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c5730 0x7f106c0ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106b7fe640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106c152700 0x7f106c148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 -- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c5730 msgr2=0x7f106c0ab780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c5730 0x7f106c0ab780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 -- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106c152700 msgr2=0x7f106c148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106c152700 0x7f106c148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 -- 172.21.15.78:0/523849253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c009400 con 0x7f106c0c8e00 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f106a7fc640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 0x7f106c1eb040 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f106000c430 tx=0x7f106000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060005ce0 con 0x7f106c0c8e00 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1060005e40 con 0x7f106c0c8e00 2024-02-19T14:58:10.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064009a70 con 0x7f106c0c8e00 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600072f0 con 0x7f106c0c8e00 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.358+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c1edc50 con 0x7f106c0c8e00 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.359+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c074990 con 0x7f106c0c8e00 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.360+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1060007450 con 0x7f106c0c8e00 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.360+0000 7f10537fe640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f105806b840 0x7f105806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.360+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f1060089d50 con 0x7f106c0c8e00 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.360+0000 7f106affd640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f105806b840 0x7f105806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.361+0000 7f106affd640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f105806b840 0x7f105806dd00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f106c159680 tx=0x7f105c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.365+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f106008d030 con 0x7f106c0c8e00 2024-02-19T14:58:10.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.572+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f106c0c8770 con 0x7f106c0c8e00 2024-02-19T14:58:10.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.602+0000 7f10537fe640 1 -- 172.21.15.78:0/523849253 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v143) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1060054240 con 0x7f106c0c8e00 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f105806b840 msgr2=0x7f105806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f105806b840 0x7f105806dd00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f106c159680 tx=0x7f105c009190 comp rx=0 tx=0).stop 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 msgr2=0x7f106c1eb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 0x7f106c1eb040 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f106000c430 tx=0x7f106000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 shutdown_connections 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f105806b840 0x7f105806dd00 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f106c0c8e00 0x7f106c1eb040 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106c152700 0x7f106c148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 --2- 172.21.15.78:0/523849253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f106c0c5730 0x7f106c0ab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 >> 172.21.15.78:0/523849253 conn(0x7f106c0cac90 msgr2=0x7f106c0c9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:10.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 shutdown_connections 2024-02-19T14:58:10.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.604+0000 7f107149a640 1 -- 172.21.15.78:0/523849253 wait complete. 2024-02-19T14:58:10.610 INFO:tasks.ceph.mds.d.smithi150.stderr: -154> 2024-02-19T14:58:06.022+0000 7f3178a82640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:10.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:58:10.635 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:58:10.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.723+0000 7f7129674640 1 Processor -- start 2024-02-19T14:58:10.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.723+0000 7f7129674640 1 -- start start 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7129674640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712406d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7129674640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7124159210 con 0x7f7124152620 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7129674640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240c6420 con 0x7f7124155010 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7129674640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71240c1f00 con 0x7f712406d000 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7122ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7124152620 0x7f71240c6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36756/0 (socket says 172.21.15.78:36756) 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7122ffd640 1 -- 172.21.15.78:0/159961860 learned_addr learned my addr 172.21.15.78:0/159961860 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f71227fc640 1 --2- 172.21.15.78:0/159961860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894392391 0 0) 0x7f7124159210 con 0x7f7124152620 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f710c003520 con 0x7f7124152620 2024-02-19T14:58:10.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833538096 0 0) 0x7f71240c6420 con 0x7f7124155010 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7124159210 con 0x7f7124155010 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f71227fc640 1 -- 172.21.15.78:0/159961860 >> v1:172.21.15.150:6789/0 conn(0x7f7124155010 legacy=0x7f71241554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f71227fc640 1 -- 172.21.15.78:0/159961860 >> v1:172.21.15.78:6789/0 conn(0x7f7124152620 legacy=0x7f71240c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.724+0000 7f71227fc640 1 -- 172.21.15.78:0/159961860 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7124158600 con 0x7f712406d000 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.725+0000 7f71227fc640 1 --2- 172.21.15.78:0/159961860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712406d440 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f71180097d0 tx=0x7f711802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4cad4a46d745b47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.725+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118031800 con 0x7f712406d000 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.725+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7118031960 con 0x7f712406d000 2024-02-19T14:58:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.725+0000 7f7121ffb640 1 -- 172.21.15.78:0/159961860 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118031cd0 con 0x7f712406d000 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 -- 172.21.15.78:0/159961860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 msgr2=0x7f712406d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 --2- 172.21.15.78:0/159961860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712406d440 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f71180097d0 tx=0x7f711802b8a0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 -- 172.21.15.78:0/159961860 shutdown_connections 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 --2- 172.21.15.78:0/159961860 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712406d440 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 -- 172.21.15.78:0/159961860 >> 172.21.15.78:0/159961860 conn(0x7f71240caae0 msgr2=0x7f71240caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 -- 172.21.15.78:0/159961860 shutdown_connections 2024-02-19T14:58:10.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.726+0000 7f7129674640 1 -- 172.21.15.78:0/159961860 wait complete. 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.727+0000 7f7129674640 1 Processor -- start 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.727+0000 7f7129674640 1 -- start start 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 0x7f712414e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7124155010 0x7f71241ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712414c810 con 0x7f7124152620 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7124157360 con 0x7f7124155010 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712414b510 con 0x7f712406d000 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 0x7f712414e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 0x7f712414e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53616/0 (socket says 172.21.15.78:53616) 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 -- 172.21.15.78:0/2546362178 learned_addr learned my addr 172.21.15.78:0/2546362178 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7122ffd640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71237fe640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7124155010 0x7f71241ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 -- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 msgr2=0x7f712414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712414c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 -- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7124155010 msgr2=0x7f71241ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7124155010 0x7f71241ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 -- 172.21.15.78:0/2546362178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7118009400 con 0x7f7124152620 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71237fe640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7124155010 0x7f71241ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f71227fc640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 0x7f712414e0b0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f7118000c00 tx=0x7f71180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711803ac60 con 0x7f7124152620 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711803adc0 con 0x7f7124152620 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.728+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712414b7f0 con 0x7f7124152620 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.729+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118044630 con 0x7f7124152620 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.729+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712414bdb0 con 0x7f7124152620 2024-02-19T14:58:10.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.730+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712414b650 con 0x7f7124152620 2024-02-19T14:58:10.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.730+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f711804d450 con 0x7f7124152620 2024-02-19T14:58:10.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.730+0000 7f710b7fe640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711006b810 0x7f711006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:10.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.730+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 7301+0+0 (secure 0 0 0) 0x7f711802e070 con 0x7f7124152620 2024-02-19T14:58:10.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.730+0000 7f7122ffd640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711006b810 0x7f711006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:10.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.731+0000 7f7122ffd640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711006b810 0x7f711006dcd0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f71140012e0 tx=0x7f7114001ad0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:10.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.735+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f712414b650 con 0x7f7124152620 2024-02-19T14:58:10.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:10.908+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f712414b650 con 0x7f7124152620 2024-02-19T14:58:11.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.605+0000 7f710b7fe640 1 -- 172.21.15.78:0/2546362178 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v161) v1 ==== 181+0+0 (secure 0 0 0) 0x7f712414b650 con 0x7f7124152620 2024-02-19T14:58:11.608 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:58:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.607+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711006b810 msgr2=0x7f711006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.607+0000 7f7129674640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711006b810 0x7f711006dcd0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f71140012e0 tx=0x7f7114001ad0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.607+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 msgr2=0x7f712414e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.607+0000 7f7129674640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 0x7f712414e0b0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f7118000c00 tx=0x7f71180046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 shutdown_connections 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711006b810 0x7f711006dcd0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7124152620 0x7f712414e0b0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7124155010 0x7f71241ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 --2- 172.21.15.78:0/2546362178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f712406d000 0x7f712414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 >> 172.21.15.78:0/2546362178 conn(0x7f71240caae0 msgr2=0x7f71240c13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 shutdown_connections 2024-02-19T14:58:11.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.608+0000 7f7129674640 1 -- 172.21.15.78:0/2546362178 wait complete. 2024-02-19T14:58:11.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:58:11.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.728+0000 7f2727c72640 1 Processor -- start 2024-02-19T14:58:11.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2727c72640 1 -- start start 2024-02-19T14:58:11.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2727c72640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27200c1b90 0x7f272014e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:11.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2727c72640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f272014edf0 0x7f2720151810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2727c72640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27200c0ce0 con 0x7f272006d420 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2727c72640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27200c2160 con 0x7f27200c1b90 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2727c72640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27200c2400 con 0x7f272014edf0 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f272520d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f27200c1b90 0x7f272014e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2725a0e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f272006d420 0x7f272006d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36764/0 (socket says 172.21.15.78:36764) 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.729+0000 7f2725a0e640 1 -- 172.21.15.78:0/3225972272 learned_addr learned my addr 172.21.15.78:0/3225972272 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.730+0000 7f272620f640 1 --2- 172.21.15.78:0/3225972272 >> v2:172.21.15.150:3300/0 conn(0x7f272014edf0 0x7f2720151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.730+0000 7f272520d640 1 -- 172.21.15.78:0/3225972272 >> v2:172.21.15.150:3300/0 conn(0x7f272014edf0 msgr2=0x7f2720151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.730+0000 7f272520d640 1 --2- 172.21.15.78:0/3225972272 >> v2:172.21.15.150:3300/0 conn(0x7f272014edf0 0x7f2720151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.730+0000 7f272520d640 1 -- 172.21.15.78:0/3225972272 >> v1:172.21.15.78:6789/0 conn(0x7f272006d420 legacy=0x7f272006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.730+0000 7f2724a0c640 1 -- 172.21.15.78:0/3225972272 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553046479 0 0) 0x7f271c001110 con 0x7f272006d420 2024-02-19T14:58:11.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.730+0000 7f272520d640 1 -- 172.21.15.78:0/3225972272 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27200c26e0 con 0x7f27200c1b90 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f272520d640 1 --2- 172.21.15.78:0/3225972272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27200c1b90 0x7f272014e8b0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f2720155bd0 tx=0x7f271002c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162f6219a8e601ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2724a0c640 1 -- 172.21.15.78:0/3225972272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710030800 con 0x7f27200c1b90 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2724a0c640 1 -- 172.21.15.78:0/3225972272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710030960 con 0x7f27200c1b90 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2724a0c640 1 -- 172.21.15.78:0/3225972272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710030cf0 con 0x7f27200c1b90 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 -- 172.21.15.78:0/3225972272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27200c1b90 msgr2=0x7f272014e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 --2- 172.21.15.78:0/3225972272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27200c1b90 0x7f272014e8b0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f2720155bd0 tx=0x7f271002c9e0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 -- 172.21.15.78:0/3225972272 shutdown_connections 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 --2- 172.21.15.78:0/3225972272 >> v2:172.21.15.150:3300/0 conn(0x7f272014edf0 0x7f2720151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 --2- 172.21.15.78:0/3225972272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27200c1b90 0x7f272014e8b0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 -- 172.21.15.78:0/3225972272 >> 172.21.15.78:0/3225972272 conn(0x7f27200c6fe0 msgr2=0x7f27200a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:11.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 -- 172.21.15.78:0/3225972272 shutdown_connections 2024-02-19T14:58:11.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.731+0000 7f2727c72640 1 -- 172.21.15.78:0/3225972272 wait complete. 2024-02-19T14:58:11.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.732+0000 7f2727c72640 1 Processor -- start 2024-02-19T14:58:11.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.732+0000 7f2727c72640 1 -- start start 2024-02-19T14:58:11.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2727c72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f272006d420 0x7f27200a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:11.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2727c72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27200c1b90 0x7f27201eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:11.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2727c72640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 0x7f2720151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:11.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2727c72640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27201ddc90 con 0x7f272014edf0 2024-02-19T14:58:11.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2727c72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27200a9e80 con 0x7f272006d420 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2727c72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27201ea350 con 0x7f27200c1b90 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 0x7f2720151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 0x7f2720151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53636/0 (socket says 172.21.15.78:53636) 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 -- 172.21.15.78:0/1380238281 learned_addr learned my addr 172.21.15.78:0/1380238281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272520d640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27200c1b90 0x7f27201eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2725a0e640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f272006d420 0x7f27200a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 -- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27200c1b90 msgr2=0x7f27201eae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27200c1b90 0x7f27201eae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 -- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f272006d420 msgr2=0x7f27200a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f272006d420 0x7f27200a9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f272620f640 1 -- 172.21.15.78:0/1380238281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710009400 con 0x7f272014edf0 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.733+0000 7f2725a0e640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f272006d420 0x7f27200a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.734+0000 7f272620f640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 0x7f2720151710 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f271800efc0 tx=0x7f271800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.734+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718009ba0 con 0x7f272014edf0 2024-02-19T14:58:11.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.734+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271800f040 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.734+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27201ea630 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.734+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27180146c0 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.734+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27200a9b70 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.735+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27200678b0 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.735+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2718014820 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.736+0000 7f27167fc640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b840 0x7f270406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.736+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6863+0+0 (secure 0 0 0) 0x7f271808b060 con 0x7f272014edf0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.736+0000 7f2725a0e640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b840 0x7f270406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:11.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.736+0000 7f2725a0e640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b840 0x7f270406dd00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f271c000f60 tx=0x7f271c007a50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:11.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.740+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2718055690 con 0x7f272014edf0 2024-02-19T14:58:11.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:11.911+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f27201eab50 con 0x7f272014edf0 2024-02-19T14:58:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.621+0000 7f27167fc640 1 -- 172.21.15.78:0/1380238281 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v162) v1 ==== 169+0+0 (secure 0 0 0) 0x7f27201eab50 con 0x7f272014edf0 2024-02-19T14:58:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.625+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b840 msgr2=0x7f270406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.625+0000 7f2727c72640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b840 0x7f270406dd00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f271c000f60 tx=0x7f271c007a50 comp rx=0 tx=0).stop 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.625+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 msgr2=0x7f2720151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.625+0000 7f2727c72640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 0x7f2720151710 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f271800efc0 tx=0x7f271800c490 comp rx=0 tx=0).stop 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 shutdown_connections 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b840 0x7f270406dd00 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f272014edf0 0x7f2720151710 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27200c1b90 0x7f27201eae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 --2- 172.21.15.78:0/1380238281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f272006d420 0x7f27200a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 >> 172.21.15.78:0/1380238281 conn(0x7f27200c6fe0 msgr2=0x7f272006d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 shutdown_connections 2024-02-19T14:58:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.626+0000 7f2727c72640 1 -- 172.21.15.78:0/1380238281 wait complete. 2024-02-19T14:58:12.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:12.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.742+0000 7fb66181b640 1 Processor -- start 2024-02-19T14:58:12.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.742+0000 7fb66181b640 1 -- start start 2024-02-19T14:58:12.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.743+0000 7fb66181b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb65c156f70 0x7fb65c0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.743+0000 7fb66181b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c1563b0 con 0x7fb65c1537a0 2024-02-19T14:58:12.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.743+0000 7fb66181b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c153cd0 con 0x7fb65c14cd50 2024-02-19T14:58:12.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.743+0000 7fb66181b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65c06b6e0 con 0x7fb65c156f70 2024-02-19T14:58:12.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.743+0000 7fb65a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb65c1537a0 0x7fb65c156860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36776/0 (socket says 172.21.15.78:36776) 2024-02-19T14:58:12.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.743+0000 7fb65a7fc640 1 -- 172.21.15.78:0/4199753336 learned_addr learned my addr 172.21.15.78:0/4199753336 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:12.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/4199753336 >> v2:172.21.15.150:3300/0 conn(0x7fb65c156f70 0x7fb65c0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458464391 0 0) 0x7fb65c1563b0 con 0x7fb65c1537a0 2024-02-19T14:58:12.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb644003520 con 0x7fb65c1537a0 2024-02-19T14:58:12.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412006670 0 0) 0x7fb644003520 con 0x7fb65c1537a0 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 >> v2:172.21.15.150:3300/0 conn(0x7fb65c156f70 msgr2=0x7fb65c0c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 --2- 172.21.15.78:0/4199753336 >> v2:172.21.15.150:3300/0 conn(0x7fb65c156f70 0x7fb65c0c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 >> v1:172.21.15.150:6789/0 conn(0x7fb65c14cd50 legacy=0x7fb65c0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c06b980 con 0x7fb65c1537a0 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb650001670 con 0x7fb65c1537a0 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 >> v1:172.21.15.78:6789/0 conn(0x7fb65c1537a0 legacy=0x7fb65c156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644003970 0x7fb644003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440047c0 0x7fb644004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6440052a0 0x7fb64400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c06b6e0 con 0x7fb6440052a0 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64400c170 con 0x7fb644003970 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64400c330 con 0x7fb6440047c0 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6440052a0 0x7fb64400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65affd640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644003970 0x7fb644003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65a7fc640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440047c0 0x7fb644004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 -- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440047c0 msgr2=0x7fb644004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440047c0 0x7fb644004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 -- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644003970 msgr2=0x7fb644003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644003970 0x7fb644003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.744+0000 7fb65b7fe640 1 -- 172.21.15.78:0/4199753336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb644003520 con 0x7fb6440052a0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.745+0000 7fb65b7fe640 1 -- 172.21.15.78:0/4199753336 reap_dead start 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.745+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6440052a0 0x7fb64400b980 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fb654009a90 tx=0x7fb6440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ca5ddb9086caed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.745+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654004020 con 0x7fb6440052a0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.745+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb654004180 con 0x7fb6440052a0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.745+0000 7fb659ffb640 1 -- 172.21.15.78:0/4199753336 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6540044f0 con 0x7fb6440052a0 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 -- 172.21.15.78:0/4199753336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6440052a0 msgr2=0x7fb64400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6440052a0 0x7fb64400b980 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fb654009a90 tx=0x7fb6440057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 -- 172.21.15.78:0/4199753336 shutdown_connections 2024-02-19T14:58:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 --2- 172.21.15.78:0/4199753336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6440052a0 0x7fb64400b980 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 -- 172.21.15.78:0/4199753336 >> 172.21.15.78:0/4199753336 conn(0x7fb65c066e70 msgr2=0x7fb65c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 -- 172.21.15.78:0/4199753336 shutdown_connections 2024-02-19T14:58:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.746+0000 7fb66181b640 1 -- 172.21.15.78:0/4199753336 wait complete. 2024-02-19T14:58:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.747+0000 7fb66181b640 1 Processor -- start 2024-02-19T14:58:12.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.747+0000 7fb66181b640 1 -- start start 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440052a0 0x7fb65c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb65c14b850 0x7fb65c14d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 0x7fb65c14dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1e8ad0 con 0x7fb65c1e8f60 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c14ecf0 con 0x7fb65c14b850 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c14efd0 con 0x7fb6440052a0 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 0x7fb65c14dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 0x7fb65c14dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53658/0 (socket says 172.21.15.78:53658) 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 -- 172.21.15.78:0/3265639245 learned_addr learned my addr 172.21.15.78:0/3265639245 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65affd640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440052a0 0x7fb65c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65a7fc640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb65c14b850 0x7fb65c14d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 -- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440052a0 msgr2=0x7fb65c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440052a0 0x7fb65c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 -- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb65c14b850 msgr2=0x7fb65c14d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb65c14b850 0x7fb65c14d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 -- 172.21.15.78:0/3265639245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c1e8760 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65affd640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440052a0 0x7fb65c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb65b7fe640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 0x7fb65c14dc60 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb654009bb0 tx=0x7fb654004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654003810 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb654003970 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65c156030 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654003ce0 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.748+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c14e1a0 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.749+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65c1e88f0 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.750+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb654003e40 con 0x7fb65c1e8f60 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.750+0000 7fb643fff640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb64806b810 0x7fb64806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:12.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.750+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fb65403b070 con 0x7fb65c1e8f60 2024-02-19T14:58:12.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.750+0000 7fb65affd640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb64806b810 0x7fb64806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:12.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.751+0000 7fb65affd640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb64806b810 0x7fb64806dcd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000f80 tx=0x7fb64c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:12.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.755+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb654075c40 con 0x7fb65c1e8f60 2024-02-19T14:58:12.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.926+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb65c074990 con 0x7fb65c1e8f60 2024-02-19T14:58:12.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.928+0000 7fb643fff640 1 -- 172.21.15.78:0/3265639245 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+17401 (secure 0 0 0) 0x7fb654079300 con 0x7fb65c1e8f60 2024-02-19T14:58:12.930 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:12.931 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":162,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:12.618654+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6834/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6835/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.150:6832/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6833/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:12.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.929+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb64806b810 msgr2=0x7fb64806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.929+0000 7fb66181b640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb64806b810 0x7fb64806dcd0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000f80 tx=0x7fb64c009040 comp rx=0 tx=0).stop 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.929+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 msgr2=0x7fb65c14dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.929+0000 7fb66181b640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 0x7fb65c14dc60 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb654009bb0 tx=0x7fb654004730 comp rx=0 tx=0).stop 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 shutdown_connections 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb64806b810 0x7fb64806dcd0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb65c1e8f60 0x7fb65c14dc60 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb65c14b850 0x7fb65c14d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 --2- 172.21.15.78:0/3265639245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440052a0 0x7fb65c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 >> 172.21.15.78:0/3265639245 conn(0x7fb65c066e70 msgr2=0x7fb65c151f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 shutdown_connections 2024-02-19T14:58:12.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:12.930+0000 7fb66181b640 1 -- 172.21.15.78:0/3265639245 wait complete. 2024-02-19T14:58:12.943 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-19T14:57:52.540793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-19T14:57:53.126421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:58:12.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:58:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.043+0000 7f7b821a2640 1 Processor -- start 2024-02-19T14:58:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.044+0000 7f7b821a2640 1 -- start start 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.044+0000 7f7b821a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7b7c14d070 0x7f7b7c0ab580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.044+0000 7f7b821a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7b7c0a94c0 0x7f7b7c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.044+0000 7f7b821a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0abac0 con 0x7f7b7c1535e0 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.044+0000 7f7b821a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7c14d450 con 0x7f7b7c14d070 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.044+0000 7f7b821a2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7c0a9a80 con 0x7f7b7c0a94c0 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7b7c14d070 0x7f7b7c0ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7b7c14d070 0x7f7b7c0ab580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53672/0 (socket says 172.21.15.78:53672) 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 -- 172.21.15.78:0/3455678067 learned_addr learned my addr 172.21.15.78:0/3455678067 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b8099f640 1 --2- 172.21.15.78:0/3455678067 >> v2:172.21.15.150:3300/0 conn(0x7f7b7c0a94c0 0x7f7b7c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 -- 172.21.15.78:0/3455678067 >> v2:172.21.15.150:3300/0 conn(0x7f7b7c0a94c0 msgr2=0x7f7b7c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 --2- 172.21.15.78:0/3455678067 >> v2:172.21.15.150:3300/0 conn(0x7f7b7c0a94c0 0x7f7b7c156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 -- 172.21.15.78:0/3455678067 >> v1:172.21.15.150:6789/0 conn(0x7f7b7c1535e0 legacy=0x7f7b7c06dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b811a0640 1 -- 172.21.15.78:0/3455678067 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0a9d60 con 0x7f7b7c14d070 2024-02-19T14:58:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.045+0000 7f7b8099f640 1 --2- 172.21.15.78:0/3455678067 >> v2:172.21.15.150:3300/0 conn(0x7f7b7c0a94c0 0x7f7b7c156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b811a0640 1 --2- 172.21.15.78:0/3455678067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0ab580 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c150a80 tx=0x7f7b7802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6847c288df0c8d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b73fff640 1 -- 172.21.15.78:0/3455678067 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78031750 con 0x7f7b7c14d070 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b73fff640 1 -- 172.21.15.78:0/3455678067 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b780318b0 con 0x7f7b7c14d070 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b73fff640 1 -- 172.21.15.78:0/3455678067 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78031c40 con 0x7f7b7c14d070 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 -- 172.21.15.78:0/3455678067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 msgr2=0x7f7b7c0ab580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 --2- 172.21.15.78:0/3455678067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0ab580 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c150a80 tx=0x7f7b7802b7f0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 -- 172.21.15.78:0/3455678067 shutdown_connections 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 --2- 172.21.15.78:0/3455678067 >> v2:172.21.15.150:3300/0 conn(0x7f7b7c0a94c0 0x7f7b7c156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 --2- 172.21.15.78:0/3455678067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0ab580 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 -- 172.21.15.78:0/3455678067 >> 172.21.15.78:0/3455678067 conn(0x7f7b7c0c6410 msgr2=0x7f7b7c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 -- 172.21.15.78:0/3455678067 shutdown_connections 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.046+0000 7f7b821a2640 1 -- 172.21.15.78:0/3455678067 wait complete. 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.047+0000 7f7b821a2640 1 Processor -- start 2024-02-19T14:58:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.047+0000 7f7b821a2640 1 -- start start 2024-02-19T14:58:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b821a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7c0a94c0 0x7f7b7c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b821a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b821a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b7c1535e0 0x7f7b7c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b821a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c152c10 con 0x7f7b7c14d070 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b821a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c072bf0 con 0x7f7b7c0a94c0 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b821a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c14db10 con 0x7f7b7c1535e0 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b8099f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b8099f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0bfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53678/0 (socket says 172.21.15.78:53678) 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b8099f640 1 -- 172.21.15.78:0/22218019 learned_addr learned my addr 172.21.15.78:0/22218019 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b819a1640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b7c1535e0 0x7f7b7c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.048+0000 7f7b811a0640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7c0a94c0 0x7f7b7c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b8099f640 1 -- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b7c1535e0 msgr2=0x7f7b7c070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b8099f640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b7c1535e0 0x7f7b7c070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b8099f640 1 -- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7c0a94c0 msgr2=0x7f7b7c1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b8099f640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7c0a94c0 0x7f7b7c1526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b8099f640 1 -- 172.21.15.78:0/22218019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780094d0 con 0x7f7b7c14d070 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b811a0640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7c0a94c0 0x7f7b7c1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b8099f640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0bfb20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00c820 tx=0x7f7b6c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c007a30 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c007b90 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c1eb260 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00ce10 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.049+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c072880 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.050+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c1eb780 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.050+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7b6c0113f0 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.051+0000 7f7b71ffb640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7b6006b810 0x7f7b6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.051+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f7b6c08a510 con 0x7f7b7c14d070 2024-02-19T14:58:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.051+0000 7f7b811a0640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7b6006b810 0x7f7b6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.051+0000 7f7b811a0640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7b6006b810 0x7f7b6006dcd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7b78035000 tx=0x7f7b78008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.055+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7b6c054cf0 con 0x7f7b7c14d070 2024-02-19T14:58:13.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.228+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b7c074990 con 0x7f7b7c14d070 2024-02-19T14:58:13.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.228+0000 7f7b71ffb640 1 -- 172.21.15.78:0/22218019 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v162) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b6c0583b0 con 0x7f7b7c14d070 2024-02-19T14:58:13.231 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:58:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7b6006b810 msgr2=0x7f7b6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7b6006b810 0x7f7b6006dcd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7b78035000 tx=0x7f7b78008030 comp rx=0 tx=0).stop 2024-02-19T14:58:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 msgr2=0x7f7b7c0bfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0bfb20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00c820 tx=0x7f7b6c00cce0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 shutdown_connections 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7b6006b810 0x7f7b6006dcd0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b7c14d070 0x7f7b7c0bfb20 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7b7c1535e0 0x7f7b7c070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 --2- 172.21.15.78:0/22218019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7c0a94c0 0x7f7b7c1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 >> 172.21.15.78:0/22218019 conn(0x7f7b7c0c6410 msgr2=0x7f7b7c153ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.230+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 shutdown_connections 2024-02-19T14:58:13.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.231+0000 7f7b821a2640 1 -- 172.21.15.78:0/22218019 wait complete. 2024-02-19T14:58:13.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-19T14:58:13.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 Processor -- start 2024-02-19T14:58:13.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 -- start start 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 0x7f2bc81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2bc80c7070 0x7f2bc806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc8155ef0 con 0x7f2bc806d350 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc806dec0 con 0x7f2bc80c7070 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.345+0000 7f2bcd5bf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c75c0 con 0x7f2bc8152480 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 0x7f2bc81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc67fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2bc80c7070 0x7f2bc806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 0x7f2bc81559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:35150/0 (socket says 172.21.15.78:35150) 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2bc806d350 0x7f2bc80c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50406/0 (socket says 172.21.15.78:50406) 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc6ffd640 1 -- 172.21.15.78:0/615855332 learned_addr learned my addr 172.21.15.78:0/615855332 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc67fc640 1 -- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 msgr2=0x7f2bc81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc67fc640 1 --2- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 0x7f2bc81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc67fc640 1 -- 172.21.15.78:0/615855332 >> v1:172.21.15.150:6789/0 conn(0x7f2bc806d350 legacy=0x7f2bc80c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc67fc640 1 -- 172.21.15.78:0/615855332 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc80c78a0 con 0x7f2bc80c7070 2024-02-19T14:58:13.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.346+0000 7f2bc6ffd640 1 --2- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 0x7f2bc81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.347+0000 7f2bc67fc640 1 --2- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80c7070 0x7f2bc806ce10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0097d0 tx=0x7f2bbc02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=533db848a525ba70 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.347+0000 7f2bc5ffb640 1 -- 172.21.15.78:0/615855332 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc031800 con 0x7f2bc80c7070 2024-02-19T14:58:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.347+0000 7f2bc5ffb640 1 -- 172.21.15.78:0/615855332 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc031960 con 0x7f2bc80c7070 2024-02-19T14:58:13.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.347+0000 7f2bc5ffb640 1 -- 172.21.15.78:0/615855332 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc031cf0 con 0x7f2bc80c7070 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80c7070 msgr2=0x7f2bc806ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80c7070 0x7f2bc806ce10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0097d0 tx=0x7f2bbc02b8a0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/615855332 shutdown_connections 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80c7070 0x7f2bc806ce10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/615855332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc8152480 0x7f2bc81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/615855332 >> 172.21.15.78:0/615855332 conn(0x7f2bc80ca8a0 msgr2=0x7f2bc80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/615855332 shutdown_connections 2024-02-19T14:58:13.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.348+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/615855332 wait complete. 2024-02-19T14:58:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.349+0000 7f2bcd5bf640 1 Processor -- start 2024-02-19T14:58:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.349+0000 7f2bcd5bf640 1 -- start start 2024-02-19T14:58:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bcd5bf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 0x7f2bc80aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bcd5bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc8152480 0x7f2bc8149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bcd5bf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80abd50 0x7f2bc81edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bcd5bf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80aa740 con 0x7f2bc80c7070 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bcd5bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc8158770 con 0x7f2bc8152480 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bcd5bf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc8158a50 con 0x7f2bc80abd50 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 0x7f2bc80aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 0x7f2bc80aab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53686/0 (socket says 172.21.15.78:53686) 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 -- 172.21.15.78:0/3604746458 learned_addr learned my addr 172.21.15.78:0/3604746458 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc77fe640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80abd50 0x7f2bc81edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc67fc640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc8152480 0x7f2bc8149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 -- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80abd50 msgr2=0x7f2bc81edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80abd50 0x7f2bc81edd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 -- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc8152480 msgr2=0x7f2bc8149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc8152480 0x7f2bc8149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 -- 172.21.15.78:0/3604746458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80094d0 con 0x7f2bc80c7070 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2bc6ffd640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 0x7f2bc80aab60 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800e9b0 tx=0x7f2bb800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.350+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb800c9a0 con 0x7f2bc80c7070 2024-02-19T14:58:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.351+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc009400 con 0x7f2bc80c7070 2024-02-19T14:58:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.351+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb800cb00 con 0x7f2bc80c7070 2024-02-19T14:58:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.351+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb800ce70 con 0x7f2bc80c7070 2024-02-19T14:58:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.351+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc8149d60 con 0x7f2bc80c7070 2024-02-19T14:58:13.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.351+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc80ab0a0 con 0x7f2bc80c7070 2024-02-19T14:58:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.352+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2bb8013470 con 0x7f2bc80c7070 2024-02-19T14:58:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.352+0000 7f2baf7fe640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2bb406b810 0x7f2bb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.352+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f2bb808aa90 con 0x7f2bc80c7070 2024-02-19T14:58:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.352+0000 7f2bc67fc640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2bb406b810 0x7f2bb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.353+0000 7f2bc67fc640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2bb406b810 0x7f2bb406dcd0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc036000 tx=0x7f2bbc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.357+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2bb80551d0 con 0x7f2bc80c7070 2024-02-19T14:58:13.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.526+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2bc80aa9b0 con 0x7f2bc80c7070 2024-02-19T14:58:13.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.526+0000 7f2baf7fe640 1 -- 172.21.15.78:0/3604746458 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v162) v1 ==== 105+0+11 (secure 0 0 0) 0x7f2bb8058890 con 0x7f2bc80c7070 2024-02-19T14:58:13.529 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:58:13.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2bb406b810 msgr2=0x7f2bb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2bb406b810 0x7f2bb406dcd0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc036000 tx=0x7f2bbc008030 comp rx=0 tx=0).stop 2024-02-19T14:58:13.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 msgr2=0x7f2bc80aab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 0x7f2bc80aab60 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800e9b0 tx=0x7f2bb800ee70 comp rx=0 tx=0).stop 2024-02-19T14:58:13.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 shutdown_connections 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2bb406b810 0x7f2bb406dcd0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80abd50 0x7f2bc81edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc8152480 0x7f2bc8149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 --2- 172.21.15.78:0/3604746458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bc80c7070 0x7f2bc80aab60 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 >> 172.21.15.78:0/3604746458 conn(0x7f2bc80ca8a0 msgr2=0x7f2bc80c9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 shutdown_connections 2024-02-19T14:58:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.528+0000 7f2bcd5bf640 1 -- 172.21.15.78:0/3604746458 wait complete. 2024-02-19T14:58:13.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-19T14:58:13.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.646+0000 7fd524f6b640 1 Processor -- start 2024-02-19T14:58:13.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.646+0000 7fd524f6b640 1 -- start start 2024-02-19T14:58:13.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd524f6b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd520152480 0x7fd5201559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd524f6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c7070 0x7fd52006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd524f6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd520155ef0 con 0x7fd52006d350 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd524f6b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52006dec0 con 0x7fd520152480 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd524f6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200c75c0 con 0x7fd5200c7070 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd520152480 0x7fd5201559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd520152480 0x7fd5201559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53692/0 (socket says 172.21.15.78:53692) 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 -- 172.21.15.78:0/1215895094 learned_addr learned my addr 172.21.15.78:0/1215895094 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51dd9b640 1 --2- 172.21.15.78:0/1215895094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c7070 0x7fd52006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 -- 172.21.15.78:0/1215895094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c7070 msgr2=0x7fd52006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 --2- 172.21.15.78:0/1215895094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c7070 0x7fd52006ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 -- 172.21.15.78:0/1215895094 >> v1:172.21.15.150:6789/0 conn(0x7fd52006d350 legacy=0x7fd5200c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.647+0000 7fd51e59c640 1 -- 172.21.15.78:0/1215895094 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5200c78a0 con 0x7fd520152480 2024-02-19T14:58:13.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.648+0000 7fd51e59c640 1 --2- 172.21.15.78:0/1215895094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5201559b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd518009e80 tx=0x7fd51802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6b8fc8b5ac807e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.648+0000 7fd51d59a640 1 -- 172.21.15.78:0/1215895094 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51802caf0 con 0x7fd520152480 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.648+0000 7fd51d59a640 1 -- 172.21.15.78:0/1215895094 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51802cc50 con 0x7fd520152480 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.648+0000 7fd51d59a640 1 -- 172.21.15.78:0/1215895094 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180343f0 con 0x7fd520152480 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 -- 172.21.15.78:0/1215895094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 msgr2=0x7fd5201559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 --2- 172.21.15.78:0/1215895094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5201559b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd518009e80 tx=0x7fd51802bf50 comp rx=0 tx=0).stop 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 -- 172.21.15.78:0/1215895094 shutdown_connections 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 --2- 172.21.15.78:0/1215895094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c7070 0x7fd52006ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 --2- 172.21.15.78:0/1215895094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5201559b0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 -- 172.21.15.78:0/1215895094 >> 172.21.15.78:0/1215895094 conn(0x7fd5200ca8a0 msgr2=0x7fd5200cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 -- 172.21.15.78:0/1215895094 shutdown_connections 2024-02-19T14:58:13.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.649+0000 7fd524f6b640 1 -- 172.21.15.78:0/1215895094 wait complete. 2024-02-19T14:58:13.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.650+0000 7fd524f6b640 1 Processor -- start 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.650+0000 7fd524f6b640 1 -- start start 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd524f6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5200c7070 0x7fd5200c0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd524f6b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5200c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd524f6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c05b0 0x7fd5200be4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd524f6b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd520147c10 con 0x7fd520152480 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd524f6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200abfc0 con 0x7fd5200c7070 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd524f6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd520150cf0 con 0x7fd5200c05b0 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5200c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5200c0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53700/0 (socket says 172.21.15.78:53700) 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 -- 172.21.15.78:0/2183532149 learned_addr learned my addr 172.21.15.78:0/2183532149 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51e59c640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5200c7070 0x7fd5200c0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51ed9d640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c05b0 0x7fd5200be4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 -- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c05b0 msgr2=0x7fd5200be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c05b0 0x7fd5200be4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 -- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5200c7070 msgr2=0x7fd5200c0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5200c7070 0x7fd5200c0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51dd9b640 1 -- 172.21.15.78:0/2183532149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd518009b30 con 0x7fd520152480 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.651+0000 7fd51ed9d640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c05b0 0x7fd5200be4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.652+0000 7fd51dd9b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5200c0070 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e980 tx=0x7fd51400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.652+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd514009670 con 0x7fd520152480 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.652+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51400ca00 con 0x7fd520152480 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.652+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5200ac210 con 0x7fd520152480 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.652+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51400cd90 con 0x7fd520152480 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.652+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd520151220 con 0x7fd520152480 2024-02-19T14:58:13.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.653+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd520074990 con 0x7fd520152480 2024-02-19T14:58:13.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.654+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd514013470 con 0x7fd520152480 2024-02-19T14:58:13.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.654+0000 7fd50effd640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd50006b810 0x7fd50006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.654+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fd51408a5a0 con 0x7fd520152480 2024-02-19T14:58:13.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.654+0000 7fd51e59c640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd50006b810 0x7fd50006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.655+0000 7fd51e59c640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd50006b810 0x7fd50006dcd0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd518035000 tx=0x7fd518030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.658+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd514054e10 con 0x7fd520152480 2024-02-19T14:58:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.839+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fd52006dae0 con 0x7fd520152480 2024-02-19T14:58:13.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.839+0000 7fd50effd640 1 -- 172.21.15.78:0/2183532149 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v162) v1 ==== 101+0+11 (secure 0 0 0) 0x7fd5140584d0 con 0x7fd520152480 2024-02-19T14:58:13.842 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd50006b810 msgr2=0x7fd50006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd50006b810 0x7fd50006dcd0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd518035000 tx=0x7fd518030000 comp rx=0 tx=0).stop 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 msgr2=0x7fd5200c0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5200c0070 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fd51400e980 tx=0x7fd51400ee40 comp rx=0 tx=0).stop 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 shutdown_connections 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd50006b810 0x7fd50006dcd0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd520152480 0x7fd5200c0070 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5200c05b0 0x7fd5200be4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 --2- 172.21.15.78:0/2183532149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5200c7070 0x7fd5200c0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 >> 172.21.15.78:0/2183532149 conn(0x7fd5200ca8a0 msgr2=0x7fd5200cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 shutdown_connections 2024-02-19T14:58:13.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.841+0000 7fd524f6b640 1 -- 172.21.15.78:0/2183532149 wait complete. 2024-02-19T14:58:13.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.953+0000 7f0c6d71e640 1 Processor -- start 2024-02-19T14:58:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.953+0000 7f0c6d71e640 1 -- start start 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c6d71e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0c68152700 0x7f0c680c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c6d71e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c6815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c6d71e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c681551f0 con 0x7f0c680c5730 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c6d71e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c68155490 con 0x7f0c68152700 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c6d71e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c68155770 con 0x7f0c680c8e00 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c677fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c6815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c66ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0c68152700 0x7f0c680c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c667fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0c680c5730 0x7f0c6806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50436/0 (socket says 172.21.15.78:50436) 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c667fc640 1 -- 172.21.15.78:0/2047499201 learned_addr learned my addr 172.21.15.78:0/2047499201 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.954+0000 7f0c677fe640 1 --2- 172.21.15.78:0/2047499201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c6815f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:35196/0 (socket says 172.21.15.78:35196) 2024-02-19T14:58:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c65ffb640 1 -- 172.21.15.78:0/2047499201 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281289972 0 0) 0x7f0c681551f0 con 0x7f0c680c5730 2024-02-19T14:58:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c65ffb640 1 -- 172.21.15.78:0/2047499201 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c50003520 con 0x7f0c680c5730 2024-02-19T14:58:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c677fe640 1 -- 172.21.15.78:0/2047499201 >> v2:172.21.15.150:3300/0 conn(0x7f0c68152700 msgr2=0x7f0c680c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c677fe640 1 --2- 172.21.15.78:0/2047499201 >> v2:172.21.15.150:3300/0 conn(0x7f0c68152700 0x7f0c680c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c677fe640 1 -- 172.21.15.78:0/2047499201 >> v1:172.21.15.150:6789/0 conn(0x7f0c680c5730 legacy=0x7f0c6806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c677fe640 1 -- 172.21.15.78:0/2047499201 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68155a50 con 0x7f0c680c8e00 2024-02-19T14:58:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.955+0000 7f0c66ffd640 1 --2- 172.21.15.78:0/2047499201 >> v2:172.21.15.150:3300/0 conn(0x7f0c68152700 0x7f0c680c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c677fe640 1 --2- 172.21.15.78:0/2047499201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c6815f980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009d90 tx=0x7f0c6002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f04e404381a84b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c65ffb640 1 -- 172.21.15.78:0/2047499201 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6002cc50 con 0x7f0c680c8e00 2024-02-19T14:58:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c65ffb640 1 -- 172.21.15.78:0/2047499201 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6002cdb0 con 0x7f0c680c8e00 2024-02-19T14:58:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c65ffb640 1 -- 172.21.15.78:0/2047499201 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60034660 con 0x7f0c680c8e00 2024-02-19T14:58:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 -- 172.21.15.78:0/2047499201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 msgr2=0x7f0c6815f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/2047499201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c6815f980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009d90 tx=0x7f0c6002bea0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 -- 172.21.15.78:0/2047499201 shutdown_connections 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/2047499201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c6815f980 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/2047499201 >> v2:172.21.15.150:3300/0 conn(0x7f0c68152700 0x7f0c680c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 -- 172.21.15.78:0/2047499201 >> 172.21.15.78:0/2047499201 conn(0x7f0c680cac90 msgr2=0x7f0c680cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 -- 172.21.15.78:0/2047499201 shutdown_connections 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.956+0000 7f0c6d71e640 1 -- 172.21.15.78:0/2047499201 wait complete. 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 Processor -- start 2024-02-19T14:58:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 -- start start 2024-02-19T14:58:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c680c5730 0x7f0c680ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c68149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 0x7f0c681ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680aaa10 con 0x7f0c68152700 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c681edff0 con 0x7f0c680c5730 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c6d71e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c680ad1d0 con 0x7f0c680c8e00 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.958+0000 7f0c677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 0x7f0c681ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 0x7f0c681ee480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53714/0 (socket says 172.21.15.78:53714) 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 -- 172.21.15.78:0/4197186904 learned_addr learned my addr 172.21.15.78:0/4197186904 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c667fc640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c68149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c66ffd640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c680c5730 0x7f0c680ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 -- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 msgr2=0x7f0c68149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c68149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 -- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c680c5730 msgr2=0x7f0c680ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c680c5730 0x7f0c680ac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 -- 172.21.15.78:0/4197186904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c58009400 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c66ffd640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c680c5730 0x7f0c680ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c677fe640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 0x7f0c681ee480 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009d90 tx=0x7f0c60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60040650 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c600407b0 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60040b60 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c60009a70 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.959+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6814a040 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.960+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c681ee160 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.961+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0c60040cc0 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.961+0000 7f0c4f7fe640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c5406b810 0x7f0c5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.961+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f0c600ad0f0 con 0x7f0c68152700 2024-02-19T14:58:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.961+0000 7f0c66ffd640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c5406b810 0x7f0c5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.961+0000 7f0c66ffd640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c5406b810 0x7f0c5406dcd0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0c58009c80 tx=0x7f0c58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:13.965+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0c600778b0 con 0x7f0c68152700 2024-02-19T14:58:14.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.179+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c68074990 con 0x7f0c68152700 2024-02-19T14:58:14.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.179+0000 7f0c4f7fe640 1 -- 172.21.15.78:0/4197186904 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4957 (secure 0 0 0) 0x7f0c60009450 con 0x7f0c68152700 2024-02-19T14:58:14.182 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:14.182 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":146,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7058,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1421532744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1421532744},{"type":"v1","addr":"172.21.15.150:6835","nonce":1421532744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":8134,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1056326965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1056326965},{"type":"v1","addr":"172.21.15.150:6834","nonce":1056326965}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144},{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:12.625978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":160,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9045":{"gid":9045,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:14.183 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 146 2024-02-19T14:58:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c5406b810 msgr2=0x7f0c5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c5406b810 0x7f0c5406dcd0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0c58009c80 tx=0x7f0c58008040 comp rx=0 tx=0).stop 2024-02-19T14:58:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 msgr2=0x7f0c681ee480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 0x7f0c681ee480 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009d90 tx=0x7f0c60004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:14.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 shutdown_connections 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c5406b810 0x7f0c5406dcd0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c68152700 0x7f0c681ee480 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c680c8e00 0x7f0c68149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 --2- 172.21.15.78:0/4197186904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c680c5730 0x7f0c680ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 >> 172.21.15.78:0/4197186904 conn(0x7f0c680cac90 msgr2=0x7f0c681512f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 shutdown_connections 2024-02-19T14:58:14.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.181+0000 7f0c6d71e640 1 -- 172.21.15.78:0/4197186904 wait complete. 2024-02-19T14:58:14.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.293+0000 7faa0290d640 1 Processor -- start 2024-02-19T14:58:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa0290d640 1 -- start start 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa0290d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa9fc06f170 0x7fa9fc156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa0290d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc071050 con 0x7fa9fc156e70 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa0290d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0c6ea0 con 0x7fa9fc156550 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa0290d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9fc1572c0 con 0x7fa9fc06f170 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa00eaa640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa9fc156e70 0x7fa9fc0c7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36792/0 (socket says 172.21.15.78:36792) 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.294+0000 7faa00eaa640 1 -- 172.21.15.78:0/330517172 learned_addr learned my addr 172.21.15.78:0/330517172 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9fb7fe640 1 --2- 172.21.15.78:0/330517172 >> v2:172.21.15.150:3300/0 conn(0x7fa9fc06f170 0x7fa9fc156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639972757 0 0) 0x7fa9fc071050 con 0x7fa9fc156e70 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e4003520 con 0x7fa9fc156e70 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457450833 0 0) 0x7fa9fc0c6ea0 con 0x7fa9fc156550 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc071050 con 0x7fa9fc156550 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3864041081 0 0) 0x7fa9e4003520 con 0x7fa9fc156e70 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9fb7fe640 1 -- 172.21.15.78:0/330517172 >> v1:172.21.15.150:6789/0 conn(0x7fa9fc156550 legacy=0x7fa9fc0ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9fb7fe640 1 -- 172.21.15.78:0/330517172 >> v1:172.21.15.78:6789/0 conn(0x7fa9fc156e70 legacy=0x7fa9fc0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.295+0000 7fa9fb7fe640 1 -- 172.21.15.78:0/330517172 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0c7120 con 0x7fa9fc06f170 2024-02-19T14:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.296+0000 7fa9fb7fe640 1 --2- 172.21.15.78:0/330517172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc156930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00097a0 tx=0x7fa9f002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cff7447da5a3d46 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.296+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40019f0 con 0x7fa9fc06f170 2024-02-19T14:58:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.296+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f002e050 con 0x7fa9fc06f170 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.296+0000 7fa9faffd640 1 -- 172.21.15.78:0/330517172 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0030730 con 0x7fa9fc06f170 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.296+0000 7faa0290d640 1 -- 172.21.15.78:0/330517172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 msgr2=0x7fa9fc156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.297+0000 7faa0290d640 1 --2- 172.21.15.78:0/330517172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc156930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00097a0 tx=0x7fa9f002b790 comp rx=0 tx=0).stop 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.297+0000 7faa0290d640 1 -- 172.21.15.78:0/330517172 shutdown_connections 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.297+0000 7faa0290d640 1 --2- 172.21.15.78:0/330517172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc156930 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.297+0000 7faa0290d640 1 -- 172.21.15.78:0/330517172 >> 172.21.15.78:0/330517172 conn(0x7fa9fc0aa3e0 msgr2=0x7fa9fc0aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.297+0000 7faa0290d640 1 -- 172.21.15.78:0/330517172 shutdown_connections 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.297+0000 7faa0290d640 1 -- 172.21.15.78:0/330517172 wait complete. 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.298+0000 7faa0290d640 1 Processor -- start 2024-02-19T14:58:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.298+0000 7faa0290d640 1 -- start start 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.298+0000 7faa0290d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc1ec820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.298+0000 7faa0290d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9fc156550 0x7fa9fc0c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.298+0000 7faa0290d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 0x7fa9fc1e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa0290d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1dfa80 con 0x7fa9fc156e70 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa0290d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc151b30 con 0x7fa9fc06f170 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa0290d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc151e10 con 0x7fa9fc156550 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 0x7fa9fc1e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 0x7fa9fc1e88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53716/0 (socket says 172.21.15.78:53716) 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 -- 172.21.15.78:0/3688837220 learned_addr learned my addr 172.21.15.78:0/3688837220 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:14.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7fa9fb7fe640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9fc156550 0x7fa9fc0c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7fa9fbfff640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc1ec820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 -- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9fc156550 msgr2=0x7fa9fc0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9fc156550 0x7fa9fc0c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 -- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 msgr2=0x7fa9fc1ec820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc1ec820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 -- 172.21.15.78:0/3688837220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0009400 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa00eaa640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 0x7fa9fc1e88a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400cc60 tx=0x7fa9f400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400ce70 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f4012400 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc1520f0 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40127b0 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.299+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc0c5010 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.300+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc152330 con 0x7fa9fc156e70 2024-02-19T14:58:14.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.301+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa9f4012910 con 0x7fa9fc156e70 2024-02-19T14:58:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.301+0000 7fa9f8ff9640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9e806b810 0x7fa9e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.301+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fa9f408ac40 con 0x7fa9fc156e70 2024-02-19T14:58:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.301+0000 7fa9fbfff640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9e806b810 0x7fa9e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.302+0000 7fa9fbfff640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9e806b810 0x7fa9e806dcd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec001650 tx=0x7fa9ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.306+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9fc152330 con 0x7fa9fc156e70 2024-02-19T14:58:14.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.480+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9fc1dc940 con 0x7fa9fc156e70 2024-02-19T14:58:14.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.481+0000 7fa9f8ff9640 1 -- 172.21.15.78:0/3688837220 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+17401 (secure 0 0 0) 0x7fa9fc1dc940 con 0x7fa9fc156e70 2024-02-19T14:58:14.484 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:14.484 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":162,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:12.618654+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6834/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6835/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.150:6832/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6833/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9e806b810 msgr2=0x7fa9e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9e806b810 0x7fa9e806dcd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec001650 tx=0x7fa9ec009040 comp rx=0 tx=0).stop 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 msgr2=0x7fa9fc1e88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 0x7fa9fc1e88a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400cc60 tx=0x7fa9f400a430 comp rx=0 tx=0).stop 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 shutdown_connections 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9e806b810 0x7fa9e806dcd0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9fc156e70 0x7fa9fc1e88a0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9fc156550 0x7fa9fc0c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 --2- 172.21.15.78:0/3688837220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9fc06f170 0x7fa9fc1ec820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 >> 172.21.15.78:0/3688837220 conn(0x7fa9fc0aa3e0 msgr2=0x7fa9fc0a7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:14.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 shutdown_connections 2024-02-19T14:58:14.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.483+0000 7faa0290d640 1 -- 172.21.15.78:0/3688837220 wait complete. 2024-02-19T14:58:14.497 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-02-19T14:58:14.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.594+0000 7f478f6c9640 1 Processor -- start 2024-02-19T14:58:14.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.594+0000 7f478f6c9640 1 -- start start 2024-02-19T14:58:14.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478f6c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47880add70 0x7f47880c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478f6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880705b0 0x7f4788070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478f6c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880ae670 con 0x7f47880c2f60 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478f6c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880716b0 con 0x7f47880add70 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478f6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880c33d0 con 0x7f47880705b0 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478cc64640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f47880c2f60 0x7f478806eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36804/0 (socket says 172.21.15.78:36804) 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478cc64640 1 -- 172.21.15.78:0/993706788 learned_addr learned my addr 172.21.15.78:0/993706788 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478dc66640 1 --2- 172.21.15.78:0/993706788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880705b0 0x7f4788070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.595+0000 7f478d465640 1 --2- 172.21.15.78:0/993706788 >> v2:172.21.15.150:3300/0 conn(0x7f47880add70 0x7f47880c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f477ffff640 1 -- 172.21.15.78:0/993706788 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177714112 0 0) 0x7f47880ae670 con 0x7f47880c2f60 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f477ffff640 1 -- 172.21.15.78:0/993706788 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774003520 con 0x7f47880c2f60 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f478dc66640 1 -- 172.21.15.78:0/993706788 >> v2:172.21.15.150:3300/0 conn(0x7f47880add70 msgr2=0x7f47880c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f477ffff640 1 -- 172.21.15.78:0/993706788 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871555800 0 0) 0x7f4774003520 con 0x7f47880c2f60 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f478dc66640 1 --2- 172.21.15.78:0/993706788 >> v2:172.21.15.150:3300/0 conn(0x7f47880add70 0x7f47880c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f478dc66640 1 -- 172.21.15.78:0/993706788 >> v1:172.21.15.78:6789/0 conn(0x7f47880c2f60 legacy=0x7f478806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.596+0000 7f478dc66640 1 -- 172.21.15.78:0/993706788 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4788153780 con 0x7f47880705b0 2024-02-19T14:58:14.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478dc66640 1 --2- 172.21.15.78:0/993706788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880705b0 0x7f4788070a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4780009d90 tx=0x7f478002bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1380a827793d2793 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f477ffff640 1 -- 172.21.15.78:0/993706788 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778001670 con 0x7f47880705b0 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f477ffff640 1 -- 172.21.15.78:0/993706788 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478002c810 con 0x7f47880705b0 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f477ffff640 1 -- 172.21.15.78:0/993706788 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478002cba0 con 0x7f47880705b0 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 -- 172.21.15.78:0/993706788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880705b0 msgr2=0x7f4788070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 --2- 172.21.15.78:0/993706788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880705b0 0x7f4788070a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4780009d90 tx=0x7f478002bce0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 -- 172.21.15.78:0/993706788 shutdown_connections 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 --2- 172.21.15.78:0/993706788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880705b0 0x7f4788070a00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 --2- 172.21.15.78:0/993706788 >> v2:172.21.15.150:3300/0 conn(0x7f47880add70 0x7f47880c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 -- 172.21.15.78:0/993706788 >> 172.21.15.78:0/993706788 conn(0x7f47880c85e0 msgr2=0x7f47880c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.597+0000 7f478f6c9640 1 -- 172.21.15.78:0/993706788 shutdown_connections 2024-02-19T14:58:14.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.598+0000 7f478f6c9640 1 -- 172.21.15.78:0/993706788 wait complete. 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 Processor -- start 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 -- start start 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47880705b0 0x7f4788152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880add70 0x7f47881dcb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 0x7f47881ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4788150d80 con 0x7f47880c2f60 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47881ea510 con 0x7f47880705b0 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478f6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788152e30 con 0x7f47880add70 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478dc66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 0x7f47881ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478dc66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 0x7f47881ea9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53718/0 (socket says 172.21.15.78:53718) 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.599+0000 7f478dc66640 1 -- 172.21.15.78:0/435769391 learned_addr learned my addr 172.21.15.78:0/435769391 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:14.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478cc64640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880add70 0x7f47881dcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478d465640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47880705b0 0x7f4788152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478dc66640 1 -- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880add70 msgr2=0x7f47881dcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478dc66640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880add70 0x7f47881dcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478dc66640 1 -- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47880705b0 msgr2=0x7f4788152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478dc66640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47880705b0 0x7f4788152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478dc66640 1 -- 172.21.15.78:0/435769391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780009a70 con 0x7f47880c2f60 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478dc66640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 0x7f47881ea9a0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f4780009d90 tx=0x7f4780004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780009450 con 0x7f47880c2f60 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4780034c50 con 0x7f47880c2f60 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47881de950 con 0x7f47880c2f60 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478003e410 con 0x7f47880c2f60 2024-02-19T14:58:14.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.600+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47881deed0 con 0x7f47880c2f60 2024-02-19T14:58:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.601+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47880676a0 con 0x7f47880c2f60 2024-02-19T14:58:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.602+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f478003b050 con 0x7f47880c2f60 2024-02-19T14:58:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.602+0000 7f477dffb640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476c06b810 0x7f476c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.602+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f47800ac960 con 0x7f47880c2f60 2024-02-19T14:58:14.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.602+0000 7f478d465640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476c06b810 0x7f476c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.602+0000 7f478d465640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476c06b810 0x7f476c06dcd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4784009a10 tx=0x7f4784008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.609+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4780077120 con 0x7f47880c2f60 2024-02-19T14:58:14.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.826+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47881ea6a0 con 0x7f47880c2f60 2024-02-19T14:58:14.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.827+0000 7f477dffb640 1 -- 172.21.15.78:0/435769391 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4957 (secure 0 0 0) 0x7f478007a7e0 con 0x7f47880c2f60 2024-02-19T14:58:14.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:14.830 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":146,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7058,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1421532744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1421532744},{"type":"v1","addr":"172.21.15.150:6835","nonce":1421532744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":8134,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1056326965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1056326965},{"type":"v1","addr":"172.21.15.150:6834","nonce":1056326965}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144},{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:12.625978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":160,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9045":{"gid":9045,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:14.830 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 146 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.828+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476c06b810 msgr2=0x7f476c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.828+0000 7f478f6c9640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476c06b810 0x7f476c06dcd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4784009a10 tx=0x7f4784008040 comp rx=0 tx=0).stop 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.828+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 msgr2=0x7f47881ea9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.828+0000 7f478f6c9640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 0x7f47881ea9a0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f4780009d90 tx=0x7f4780004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 shutdown_connections 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f476c06b810 0x7f476c06dcd0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47880c2f60 0x7f47881ea9a0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47880add70 0x7f47881dcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 --2- 172.21.15.78:0/435769391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47880705b0 0x7f4788152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 >> 172.21.15.78:0/435769391 conn(0x7f47880c85e0 msgr2=0x7f47880a93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 shutdown_connections 2024-02-19T14:58:14.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.829+0000 7f478f6c9640 1 -- 172.21.15.78:0/435769391 wait complete. 2024-02-19T14:58:14.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:14.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.942+0000 7f0c2f688640 1 Processor -- start 2024-02-19T14:58:14.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.942+0000 7f0c2f688640 1 -- start start 2024-02-19T14:58:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.943+0000 7f0c2f688640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0c2814ece0 0x7f0c280c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.943+0000 7f0c2f688640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c280c1120 con 0x7f0c280c15b0 2024-02-19T14:58:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.943+0000 7f0c2f688640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c280c6080 con 0x7f0c280c5270 2024-02-19T14:58:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.943+0000 7f0c2f688640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280c57e0 con 0x7f0c2814ece0 2024-02-19T14:58:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.943+0000 7f0c2d424640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0c2814ece0 0x7f0c280c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.944+0000 7f0c2dc25640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0c280c15b0 0x7f0c280c1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36812/0 (socket says 172.21.15.78:36812) 2024-02-19T14:58:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.944+0000 7f0c2dc25640 1 -- 172.21.15.78:0/1715990437 learned_addr learned my addr 172.21.15.78:0/1715990437 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.944+0000 7f0c2d424640 1 -- 172.21.15.78:0/1715990437 >> v1:172.21.15.150:6789/0 conn(0x7f0c280c5270 legacy=0x7f0c280719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.944+0000 7f0c2d424640 1 -- 172.21.15.78:0/1715990437 >> v1:172.21.15.78:6789/0 conn(0x7f0c280c15b0 legacy=0x7f0c280c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.944+0000 7f0c1ffff640 1 -- 172.21.15.78:0/1715990437 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794853095 0 0) 0x7f0c280c1120 con 0x7f0c280c15b0 2024-02-19T14:58:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.944+0000 7f0c2d424640 1 -- 172.21.15.78:0/1715990437 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280c5ac0 con 0x7f0c2814ece0 2024-02-19T14:58:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.945+0000 7f0c2d424640 1 --2- 172.21.15.78:0/1715990437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c280c4d30 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000aea0 tx=0x7f0c2002cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412f8fe8ae64dc2b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.945+0000 7f0c1ffff640 1 -- 172.21.15.78:0/1715990437 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2002db60 con 0x7f0c2814ece0 2024-02-19T14:58:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.945+0000 7f0c1ffff640 1 -- 172.21.15.78:0/1715990437 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2002dcc0 con 0x7f0c2814ece0 2024-02-19T14:58:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.945+0000 7f0c1ffff640 1 -- 172.21.15.78:0/1715990437 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20008660 con 0x7f0c2814ece0 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 -- 172.21.15.78:0/1715990437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 msgr2=0x7f0c280c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 --2- 172.21.15.78:0/1715990437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c280c4d30 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000aea0 tx=0x7f0c2002cfc0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 -- 172.21.15.78:0/1715990437 shutdown_connections 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 --2- 172.21.15.78:0/1715990437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c280c4d30 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 -- 172.21.15.78:0/1715990437 >> 172.21.15.78:0/1715990437 conn(0x7f0c2806b270 msgr2=0x7f0c2806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 -- 172.21.15.78:0/1715990437 shutdown_connections 2024-02-19T14:58:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.946+0000 7f0c2f688640 1 -- 172.21.15.78:0/1715990437 wait complete. 2024-02-19T14:58:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.948+0000 7f0c2f688640 1 Processor -- start 2024-02-19T14:58:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.948+0000 7f0c2f688640 1 -- start start 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2f688640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c280c15b0 0x7f0c2814d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2f688640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c280c5270 0x7f0c281466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2f688640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c28146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2f688640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2814f770 con 0x7f0c2814ece0 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2f688640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2814fa50 con 0x7f0c280c5270 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2f688640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c281e7170 con 0x7f0c280c15b0 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c28146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c28146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53732/0 (socket says 172.21.15.78:53732) 2024-02-19T14:58:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 -- 172.21.15.78:0/3792810256 learned_addr learned my addr 172.21.15.78:0/3792810256 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2d424640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c280c15b0 0x7f0c2814d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 -- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c280c15b0 msgr2=0x7f0c2814d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c280c15b0 0x7f0c2814d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2cc23640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c280c5270 0x7f0c281466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 -- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c280c5270 msgr2=0x7f0c281466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c280c5270 0x7f0c281466a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2dc25640 1 -- 172.21.15.78:0/3792810256 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2000ab80 con 0x7f0c2814ece0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.949+0000 7f0c2d424640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c280c15b0 0x7f0c2814d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.950+0000 7f0c2dc25640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c28146be0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400c450 tx=0x7f0c2400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.950+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c240074e0 con 0x7f0c2814ece0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.950+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c24007660 con 0x7f0c2814ece0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.950+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2814fd30 con 0x7f0c2814ece0 2024-02-19T14:58:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.950+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24004a20 con 0x7f0c2814ece0 2024-02-19T14:58:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.950+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c28147950 con 0x7f0c2814ece0 2024-02-19T14:58:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.951+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0c24004b80 con 0x7f0c2814ece0 2024-02-19T14:58:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.951+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c28074990 con 0x7f0c2814ece0 2024-02-19T14:58:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.952+0000 7f0c1dffb640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c0c06b810 0x7f0c0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.952+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f0c240898e0 con 0x7f0c2814ece0 2024-02-19T14:58:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.952+0000 7f0c2d424640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c0c06b810 0x7f0c0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.952+0000 7f0c2d424640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c0c06b810 0x7f0c0c06dcd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0c20036000 tx=0x7f0c20031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:14.956+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0c240540a0 con 0x7f0c2814ece0 2024-02-19T14:58:15.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.222+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c280c2720 con 0x7f0c2814ece0 2024-02-19T14:58:15.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.223+0000 7f0c1dffb640 1 -- 172.21.15.78:0/3792810256 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4957 (secure 0 0 0) 0x7f0c280c2720 con 0x7f0c2814ece0 2024-02-19T14:58:15.226 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:15.226 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":146,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7058,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1421532744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1421532744},{"type":"v1","addr":"172.21.15.150:6835","nonce":1421532744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":8134,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1056326965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1056326965},{"type":"v1","addr":"172.21.15.150:6834","nonce":1056326965}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":144},{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:57:54.164699+0000","modified":"2024-02-19T14:58:12.625978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":160,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9045},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9045":{"gid":9045,"name":"c","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.78:6837/1801432695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1801432695},{"type":"v1","addr":"172.21.15.78:6837","nonce":1801432695}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-19T14:58:15.226 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 146 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c0c06b810 msgr2=0x7f0c0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c0c06b810 0x7f0c0c06dcd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0c20036000 tx=0x7f0c20031000 comp rx=0 tx=0).stop 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 msgr2=0x7f0c28146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c28146be0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400c450 tx=0x7f0c2400c910 comp rx=0 tx=0).stop 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 shutdown_connections 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0c0c06b810 0x7f0c0c06dcd0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2814ece0 0x7f0c28146be0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c280c5270 0x7f0c281466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 --2- 172.21.15.78:0/3792810256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c280c15b0 0x7f0c2814d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 >> 172.21.15.78:0/3792810256 conn(0x7f0c2806b270 msgr2=0x7f0c28155350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 shutdown_connections 2024-02-19T14:58:15.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.225+0000 7f0c2f688640 1 -- 172.21.15.78:0/3792810256 wait complete. 2024-02-19T14:58:15.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:15.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.358+0000 7f3292cc7640 1 Processor -- start 2024-02-19T14:58:15.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.358+0000 7f3292cc7640 1 -- start start 2024-02-19T14:58:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f3292cc7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f328c0c8e00 0x7f328c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f3292cc7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c1551f0 con 0x7f328c152700 2024-02-19T14:58:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f3292cc7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c155490 con 0x7f328c06d910 2024-02-19T14:58:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f3292cc7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c155770 con 0x7f328c0c8e00 2024-02-19T14:58:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f3290a63640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f328c152700 0x7f328c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36824/0 (socket says 172.21.15.78:36824) 2024-02-19T14:58:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f3290a63640 1 -- 172.21.15.78:0/1698068738 learned_addr learned my addr 172.21.15.78:0/1698068738 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258060036 0 0) 0x7f328c1551f0 con 0x7f328c152700 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327c003520 con 0x7f328c152700 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328bfff640 1 --2- 172.21.15.78:0/1698068738 >> v2:172.21.15.150:3300/0 conn(0x7f328c0c8e00 0x7f328c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747601914 0 0) 0x7f327c003520 con 0x7f328c152700 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 >> v2:172.21.15.150:3300/0 conn(0x7f328c0c8e00 msgr2=0x7f328c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 --2- 172.21.15.78:0/1698068738 >> v2:172.21.15.150:3300/0 conn(0x7f328c0c8e00 0x7f328c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 >> v1:172.21.15.150:6789/0 conn(0x7f328c06d910 legacy=0x7f328c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c155a50 con 0x7f328c152700 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3280001670 con 0x7f328c152700 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 >> v1:172.21.15.78:6789/0 conn(0x7f328c152700 legacy=0x7f328c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0039c0 0x7f327c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0047c0 0x7f327c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f327c005180 0x7f327c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c155770 con 0x7f327c0047c0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c00c080 con 0x7f327c005180 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.359+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c00c240 con 0x7f327c0039c0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0047c0 0x7f327c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f3290a63640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0039c0 0x7f327c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f3291264640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f327c005180 0x7f327c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 -- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0039c0 msgr2=0x7f327c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0039c0 0x7f327c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 -- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f327c005180 msgr2=0x7f327c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f327c005180 0x7f327c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 -- 172.21.15.78:0/1698068738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c003520 con 0x7f327c0047c0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f3291264640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f327c005180 0x7f327c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f3291264640 1 -- 172.21.15.78:0/1698068738 reap_dead start 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.360+0000 7f328bfff640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0047c0 0x7f327c004c40 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f3278009ac0 tx=0x7f327c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95c6322db725c71 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327802c6b0 con 0x7f327c0047c0 2024-02-19T14:58:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327802c810 con 0x7f327c0047c0 2024-02-19T14:58:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f328b7fe640 1 -- 172.21.15.78:0/1698068738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327802cba0 con 0x7f327c0047c0 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 -- 172.21.15.78:0/1698068738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0047c0 msgr2=0x7f327c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0047c0 0x7f327c004c40 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f3278009ac0 tx=0x7f327c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 -- 172.21.15.78:0/1698068738 shutdown_connections 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 --2- 172.21.15.78:0/1698068738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0047c0 0x7f327c004c40 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 -- 172.21.15.78:0/1698068738 >> 172.21.15.78:0/1698068738 conn(0x7f328c0cac90 msgr2=0x7f328c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 -- 172.21.15.78:0/1698068738 shutdown_connections 2024-02-19T14:58:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.361+0000 7f3292cc7640 1 -- 172.21.15.78:0/1698068738 wait complete. 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.362+0000 7f3292cc7640 1 Processor -- start 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 -- start start 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0047c0 0x7f328c0c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 0x7f328c0c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328c0c2030 0x7f328c1edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c148cf0 con 0x7f328c149220 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c1eb880 con 0x7f328c0c2030 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f3292cc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c1eb550 con 0x7f327c0047c0 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f328bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 0x7f328c0c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f328bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 0x7f328c0c1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53754/0 (socket says 172.21.15.78:53754) 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.363+0000 7f328bfff640 1 -- 172.21.15.78:0/609080377 learned_addr learned my addr 172.21.15.78:0/609080377 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f3290a63640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0047c0 0x7f328c0c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f3291264640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328c0c2030 0x7f328c1edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f328bfff640 1 -- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0047c0 msgr2=0x7f328c0c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f328bfff640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0047c0 0x7f328c0c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f328bfff640 1 -- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328c0c2030 msgr2=0x7f328c1edfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f328bfff640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328c0c2030 0x7f328c1edfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f328bfff640 1 -- 172.21.15.78:0/609080377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c148f40 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f328bfff640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 0x7f328c0c1af0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f3278037040 tx=0x7f3278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278003810 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278003970 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278003d20 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c0ace60 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.364+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c149790 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.365+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c1eb9c0 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.366+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3278034440 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.366+0000 7f32897fa640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327006b840 0x7f327006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.366+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f32780ac3d0 con 0x7f328c149220 2024-02-19T14:58:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.366+0000 7f3290a63640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327006b840 0x7f327006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.366+0000 7f3290a63640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327006b840 0x7f327006dd00 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.370+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f328c1eb9c0 con 0x7f328c149220 2024-02-19T14:58:15.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.548+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f328c1eb9c0 con 0x7f328c149220 2024-02-19T14:58:15.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.549+0000 7f32897fa640 1 -- 172.21.15.78:0/609080377 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+17401 (secure 0 0 0) 0x7f328c1eb9c0 con 0x7f328c149220 2024-02-19T14:58:15.552 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:15.552 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":162,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:12.618654+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-19T14:57:52.540793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-19T14:57:53.126421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6834/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6835/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.150:6832/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6833/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327006b840 msgr2=0x7f327006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327006b840 0x7f327006dd00 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3280000c00 tx=0x7f3280009040 comp rx=0 tx=0).stop 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 msgr2=0x7f328c0c1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 0x7f328c0c1af0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f3278037040 tx=0x7f3278004730 comp rx=0 tx=0).stop 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 shutdown_connections 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f327006b840 0x7f327006dd00 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f328c149220 0x7f328c0c1af0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328c0c2030 0x7f328c1edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 --2- 172.21.15.78:0/609080377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f327c0047c0 0x7f328c0c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 >> 172.21.15.78:0/609080377 conn(0x7f328c0cac90 msgr2=0x7f328c1514f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:15.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 shutdown_connections 2024-02-19T14:58:15.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.551+0000 7f3292cc7640 1 -- 172.21.15.78:0/609080377 wait complete. 2024-02-19T14:58:15.565 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-02-19T14:58:15.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.665+0000 7fd79e4c0640 1 Processor -- start 2024-02-19T14:58:15.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.665+0000 7fd79e4c0640 1 -- start start 2024-02-19T14:58:15.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd79e4c0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7980add70 0x7fd7980c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd79e4c0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980705b0 0x7fd798070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd79e4c0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7980ae670 con 0x7fd7980c2f60 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd79e4c0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980716b0 con 0x7fd7980add70 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd79e4c0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980c33d0 con 0x7fd7980705b0 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd797fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7980add70 0x7fd7980c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd79ca5d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980705b0 0x7fd798070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd7977fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd7980c2f60 0x7fd79806eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50472/0 (socket says 172.21.15.78:50472) 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.666+0000 7fd7977fe640 1 -- 172.21.15.78:0/237008205 learned_addr learned my addr 172.21.15.78:0/237008205 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.667+0000 7fd79ca5d640 1 -- 172.21.15.78:0/237008205 >> v2:172.21.15.150:3300/0 conn(0x7fd7980add70 msgr2=0x7fd7980c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.667+0000 7fd79ca5d640 1 --2- 172.21.15.78:0/237008205 >> v2:172.21.15.150:3300/0 conn(0x7fd7980add70 0x7fd7980c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.667+0000 7fd79ca5d640 1 -- 172.21.15.78:0/237008205 >> v1:172.21.15.150:6789/0 conn(0x7fd7980c2f60 legacy=0x7fd79806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.667+0000 7fd79ca5d640 1 -- 172.21.15.78:0/237008205 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798153780 con 0x7fd7980705b0 2024-02-19T14:58:15.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.667+0000 7fd797fff640 1 --2- 172.21.15.78:0/237008205 >> v2:172.21.15.150:3300/0 conn(0x7fd7980add70 0x7fd7980c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.667+0000 7fd79ca5d640 1 --2- 172.21.15.78:0/237008205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980705b0 0x7fd798070a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd79002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc76ce4785e9a9cc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd796ffd640 1 -- 172.21.15.78:0/237008205 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79002cbc0 con 0x7fd7980705b0 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd796ffd640 1 -- 172.21.15.78:0/237008205 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79002cd20 con 0x7fd7980705b0 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd796ffd640 1 -- 172.21.15.78:0/237008205 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790034660 con 0x7fd7980705b0 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 -- 172.21.15.78:0/237008205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980705b0 msgr2=0x7fd798070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/237008205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980705b0 0x7fd798070a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd790009d90 tx=0x7fd79002be10 comp rx=0 tx=0).stop 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 -- 172.21.15.78:0/237008205 shutdown_connections 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/237008205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980705b0 0x7fd798070a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/237008205 >> v2:172.21.15.150:3300/0 conn(0x7fd7980add70 0x7fd7980c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 -- 172.21.15.78:0/237008205 >> 172.21.15.78:0/237008205 conn(0x7fd7980c85e0 msgr2=0x7fd7980c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:15.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 -- 172.21.15.78:0/237008205 shutdown_connections 2024-02-19T14:58:15.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.668+0000 7fd79e4c0640 1 -- 172.21.15.78:0/237008205 wait complete. 2024-02-19T14:58:15.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.669+0000 7fd79e4c0640 1 Processor -- start 2024-02-19T14:58:15.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.669+0000 7fd79e4c0640 1 -- start start 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd79e4c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 0x7fd7980c1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd79e4c0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980add70 0x7fd798099c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd79e4c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7980c2f60 0x7fd79814e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd79e4c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980ac270 con 0x7fd7980705b0 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd79e4c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79809a4c0 con 0x7fd7980c2f60 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd79e4c0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7980ab8d0 con 0x7fd7980add70 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd797fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 0x7fd7980c1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd797fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 0x7fd7980c1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53764/0 (socket says 172.21.15.78:53764) 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd797fff640 1 -- 172.21.15.78:0/1124029402 learned_addr learned my addr 172.21.15.78:0/1124029402 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.670+0000 7fd7977fe640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980add70 0x7fd798099c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd79ca5d640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7980c2f60 0x7fd79814e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd797fff640 1 -- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980add70 msgr2=0x7fd798099c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd797fff640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980add70 0x7fd798099c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd797fff640 1 -- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7980c2f60 msgr2=0x7fd79814e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd797fff640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7980c2f60 0x7fd79814e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd797fff640 1 -- 172.21.15.78:0/1124029402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788009400 con 0x7fd7980705b0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd79ca5d640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7980c2f60 0x7fd79814e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd7977fe640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980add70 0x7fd798099c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd797fff640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 0x7fd7980c1bd0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fd78800ebe0 tx=0x7fd78800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788009760 con 0x7fd7980705b0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78800cc30 con 0x7fd7980705b0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78800eee0 con 0x7fd7980705b0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd790009a70 con 0x7fd7980705b0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.671+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7980abe90 con 0x7fd7980705b0 2024-02-19T14:58:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.672+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7980676a0 con 0x7fd7980705b0 2024-02-19T14:58:15.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.673+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd788013470 con 0x7fd7980705b0 2024-02-19T14:58:15.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.673+0000 7fd794ff9640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:15.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.673+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fd78808a6c0 con 0x7fd7980705b0 2024-02-19T14:58:15.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.673+0000 7fd7977fe640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:15.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.674+0000 7fd7977fe640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd798156ca0 tx=0x7fd78c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:15.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.678+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd788054e80 con 0x7fd7980705b0 2024-02-19T14:58:15.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.890+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7fd798074990 con 0x7fd7980705b0 2024-02-19T14:58:15.893 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:58:15.890+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:58:15.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.891+0000 7fd794ff9640 1 -- 172.21.15.78:0/1124029402 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v147) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd788058540 con 0x7fd7980705b0 2024-02-19T14:58:15.894 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 msgr2=0x7fd78406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd798156ca0 tx=0x7fd78c009040 comp rx=0 tx=0).stop 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 msgr2=0x7fd7980c1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 0x7fd7980c1bd0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fd78800ebe0 tx=0x7fd78800c520 comp rx=0 tx=0).stop 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 shutdown_connections 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd78406b810 0x7fd78406dcd0 secure :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd798156ca0 tx=0x7fd78c009040 comp rx=0 tx=0).stop 2024-02-19T14:58:15.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7980c2f60 0x7fd79814e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd7980add70 0x7fd798099c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 --2- 172.21.15.78:0/1124029402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7980705b0 0x7fd7980c1bd0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:15.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 >> 172.21.15.78:0/1124029402 conn(0x7fd7980c85e0 msgr2=0x7fd7981529a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:15.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 shutdown_connections 2024-02-19T14:58:15.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:15.894+0000 7fd79e4c0640 1 -- 172.21.15.78:0/1124029402 wait complete. 2024-02-19T14:58:15.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-02-19T14:58:15.922 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:58:16.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 Processor -- start 2024-02-19T14:58:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 -- start start 2024-02-19T14:58:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11a40c8e00 0x7f11a415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a41551f0 con 0x7f11a40c5730 2024-02-19T14:58:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a4155490 con 0x7f11a40c8e00 2024-02-19T14:58:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.008+0000 7f11aadb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4155770 con 0x7f11a4152700 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a9352640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11a40c8e00 0x7f11a415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f11a40c5730 0x7f11a406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36850/0 (socket says 172.21.15.78:36850) 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a9352640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11a40c8e00 0x7f11a415f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53794/0 (socket says 172.21.15.78:53794) 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a3fff640 1 -- 172.21.15.78:0/976271207 learned_addr learned my addr 172.21.15.78:0/976271207 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a8b51640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a9352640 1 -- 172.21.15.78:0/976271207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 msgr2=0x7f11a40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a9352640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a9352640 1 -- 172.21.15.78:0/976271207 >> v1:172.21.15.78:6789/0 conn(0x7f11a40c5730 legacy=0x7f11a406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a37fe640 1 -- 172.21.15.78:0/976271207 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038607286 0 0) 0x7f11a41551f0 con 0x7f11a40c5730 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a9352640 1 -- 172.21.15.78:0/976271207 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a4155a50 con 0x7f11a40c8e00 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.009+0000 7f11a8b51640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11a9352640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a415f980 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f11940097a0 tx=0x7f119402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fe047c99bbb5de server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11a37fe640 1 -- 172.21.15.78:0/976271207 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194031830 con 0x7f11a40c8e00 2024-02-19T14:58:16.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11a37fe640 1 -- 172.21.15.78:0/976271207 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1194031990 con 0x7f11a40c8e00 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11a37fe640 1 -- 172.21.15.78:0/976271207 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194031d20 con 0x7f11a40c8e00 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 -- 172.21.15.78:0/976271207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 msgr2=0x7f11a415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a415f980 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f11940097a0 tx=0x7f119402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 -- 172.21.15.78:0/976271207 shutdown_connections 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a415f980 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 --2- 172.21.15.78:0/976271207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 -- 172.21.15.78:0/976271207 >> 172.21.15.78:0/976271207 conn(0x7f11a40cac90 msgr2=0x7f11a40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 -- 172.21.15.78:0/976271207 shutdown_connections 2024-02-19T14:58:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.010+0000 7f11aadb5640 1 -- 172.21.15.78:0/976271207 wait complete. 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.011+0000 7f11aadb5640 1 Processor -- start 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.011+0000 7f11aadb5640 1 -- start start 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11aadb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11a40c5730 0x7f11a41edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11aadb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a40ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11aadb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11aadb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4149970 con 0x7f11a40c8e00 2024-02-19T14:58:16.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11aadb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41ee8c0 con 0x7f11a40c5730 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11aadb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40afe60 con 0x7f11a4152700 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a40ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11a3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a40ae4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53806/0 (socket says 172.21.15.78:53806) 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11a3fff640 1 -- 172.21.15.78:0/1683889141 learned_addr learned my addr 172.21.15.78:0/1683889141 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11a9352640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.012+0000 7f11a8b51640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11a40c5730 0x7f11a41edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a3fff640 1 -- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 msgr2=0x7f11a40ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a3fff640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a3fff640 1 -- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11a40c5730 msgr2=0x7f11a41edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a3fff640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11a40c5730 0x7f11a41edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a3fff640 1 -- 172.21.15.78:0/1683889141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1194009400 con 0x7f11a40c8e00 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a8b51640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11a40c5730 0x7f11a41edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a3fff640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a40ae4f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f118c00c4a0 tx=0x7f118c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:16.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c0074e0 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c007640 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c004a00 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a41ee5e0 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.013+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a40af0a0 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.014+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a40ae210 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.014+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f118c004b60 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.015+0000 7f11a17fa640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118406b810 0x7f118406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.015+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f118c089900 con 0x7f11a40c8e00 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.015+0000 7f11a8b51640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118406b810 0x7f118406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.015+0000 7f11a8b51640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118406b810 0x7f118406dcd0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f119c003fd0 tx=0x7f119c003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:16.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.019+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f11a40ae210 con 0x7f11a40c8e00 2024-02-19T14:58:16.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.227+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f11a40ae210 con 0x7f11a40c8e00 2024-02-19T14:58:16.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.896+0000 7f11a17fa640 1 -- 172.21.15.78:0/1683889141 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v148) v1 ==== 105+0+0 (secure 0 0 0) 0x7f11a40ae210 con 0x7f11a40c8e00 2024-02-19T14:58:16.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.898+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118406b810 msgr2=0x7f118406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:16.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.898+0000 7f11aadb5640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118406b810 0x7f118406dcd0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f119c003fd0 tx=0x7f119c003990 comp rx=0 tx=0).stop 2024-02-19T14:58:16.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.898+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 msgr2=0x7f11a40ae4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.898+0000 7f11aadb5640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a40ae4f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f118c00c4a0 tx=0x7f118c00c960 comp rx=0 tx=0).stop 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 shutdown_connections 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118406b810 0x7f118406dcd0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11a40c8e00 0x7f11a40ae4f0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11a4152700 0x7f11a40ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 --2- 172.21.15.78:0/1683889141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11a40c5730 0x7f11a41edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 >> 172.21.15.78:0/1683889141 conn(0x7f11a40cac90 msgr2=0x7f11a40cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 shutdown_connections 2024-02-19T14:58:16.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:16.899+0000 7f11aadb5640 1 -- 172.21.15.78:0/1683889141 wait complete. 2024-02-19T14:58:16.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-02-19T14:58:17.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.060+0000 7f383c284640 1 Processor -- start 2024-02-19T14:58:17.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.060+0000 7f383c284640 1 -- start start 2024-02-19T14:58:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383c284640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f383406d420 0x7f383406d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383c284640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f383406dd40 0x7f383414e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383c284640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38340c0ce0 con 0x7f38340c1b90 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383c284640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38340c2160 con 0x7f383406dd40 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383c284640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38340c2400 con 0x7f383406d420 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f383406dd40 0x7f383414e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f383406dd40 0x7f383414e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53812/0 (socket says 172.21.15.78:53812) 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 -- 172.21.15.78:0/3371126296 learned_addr learned my addr 172.21.15.78:0/3371126296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383a020640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f383406d420 0x7f383406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 -- 172.21.15.78:0/3371126296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f383406d420 msgr2=0x7f383406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f383406d420 0x7f383406d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 -- 172.21.15.78:0/3371126296 >> v1:172.21.15.150:6789/0 conn(0x7f38340c1b90 legacy=0x7f383414edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383981f640 1 -- 172.21.15.78:0/3371126296 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340c26e0 con 0x7f383406dd40 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.061+0000 7f383a020640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f383406d420 0x7f383406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383981f640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f383414e8b0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f382402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b311efb4f0dbae31 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383901e640 1 -- 172.21.15.78:0/3371126296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824031640 con 0x7f383406dd40 2024-02-19T14:58:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383901e640 1 -- 172.21.15.78:0/3371126296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38240317a0 con 0x7f383406dd40 2024-02-19T14:58:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383901e640 1 -- 172.21.15.78:0/3371126296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824031b30 con 0x7f383406dd40 2024-02-19T14:58:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383c284640 1 -- 172.21.15.78:0/3371126296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 msgr2=0x7f383414e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383c284640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f383414e8b0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f382402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383c284640 1 -- 172.21.15.78:0/3371126296 shutdown_connections 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383c284640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f383414e8b0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383c284640 1 --2- 172.21.15.78:0/3371126296 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f383406d420 0x7f383406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.062+0000 7f383c284640 1 -- 172.21.15.78:0/3371126296 >> 172.21.15.78:0/3371126296 conn(0x7f38340c6fe0 msgr2=0x7f38340a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.063+0000 7f383c284640 1 -- 172.21.15.78:0/3371126296 shutdown_connections 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.063+0000 7f383c284640 1 -- 172.21.15.78:0/3371126296 wait complete. 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 Processor -- start 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 -- start start 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f383406d420 0x7f3834154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f38340723b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38340c1b90 0x7f38340bdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38340bc6c0 con 0x7f383406dd40 2024-02-19T14:58:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38340bc100 con 0x7f383406d420 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383c284640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38340bc3e0 con 0x7f38340c1b90 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383981f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f38340723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383981f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f38340723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53828/0 (socket says 172.21.15.78:53828) 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.064+0000 7f383981f640 1 -- 172.21.15.78:0/2779341639 learned_addr learned my addr 172.21.15.78:0/2779341639 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383a821640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38340c1b90 0x7f38340bdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383a020640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f383406d420 0x7f3834154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383981f640 1 -- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38340c1b90 msgr2=0x7f38340bdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383981f640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38340c1b90 0x7f38340bdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383981f640 1 -- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f383406d420 msgr2=0x7f3834154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383981f640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f383406d420 0x7f3834154d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383981f640 1 -- 172.21.15.78:0/2779341639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38240094d0 con 0x7f383406dd40 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383a821640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38340c1b90 0x7f38340bdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383981f640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f38340723b0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f38240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382403ac60 con 0x7f383406dd40 2024-02-19T14:58:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382403ade0 con 0x7f383406dd40 2024-02-19T14:58:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824044630 con 0x7f383406dd40 2024-02-19T14:58:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3834154830 con 0x7f383406dd40 2024-02-19T14:58:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.065+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3834072e30 con 0x7f383406dd40 2024-02-19T14:58:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.066+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38340678b0 con 0x7f383406dd40 2024-02-19T14:58:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.066+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f382402f050 con 0x7f383406dd40 2024-02-19T14:58:17.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.067+0000 7f382affd640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f381806b840 0x7f381806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:17.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.067+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f38240ac410 con 0x7f383406dd40 2024-02-19T14:58:17.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.067+0000 7f383a020640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f381806b840 0x7f381806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:17.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.067+0000 7f383a020640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f381806b840 0x7f381806dd00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f38340c0200 tx=0x7f3830008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:17.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.071+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3824076b80 con 0x7f383406dd40 2024-02-19T14:58:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.243+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f38340abba0 con 0x7f383406dd40 2024-02-19T14:58:17.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.897+0000 7f382affd640 1 -- 172.21.15.78:0/2779341639 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v164) v1 ==== 184+0+0 (secure 0 0 0) 0x7f382407a240 con 0x7f383406dd40 2024-02-19T14:58:17.900 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_metadata' removed 2024-02-19T14:58:17.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f381806b840 msgr2=0x7f381806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f381806b840 0x7f381806dd00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f38340c0200 tx=0x7f3830008040 comp rx=0 tx=0).stop 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 msgr2=0x7f38340723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f38340723b0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f3824009870 tx=0x7f38240046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 shutdown_connections 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f381806b840 0x7f381806dd00 secure :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f38340c0200 tx=0x7f3830008040 comp rx=0 tx=0).stop 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f383406dd40 0x7f38340723b0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38340c1b90 0x7f38340bdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 --2- 172.21.15.78:0/2779341639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f383406d420 0x7f3834154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 >> 172.21.15.78:0/2779341639 conn(0x7f38340c6fe0 msgr2=0x7f3834147360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 shutdown_connections 2024-02-19T14:58:17.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:17.901+0000 7f383c284640 1 -- 172.21.15.78:0/2779341639 wait complete. 2024-02-19T14:58:17.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-02-19T14:58:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.018+0000 7f780835c640 1 Processor -- start 2024-02-19T14:58:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.018+0000 7f780835c640 1 -- start start 2024-02-19T14:58:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f780835c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78000c7cd0 0x7f78000a9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f780835c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78000a8530 0x7f7800152240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f780835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78000727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f780835c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7800072390 con 0x7f78000c7cd0 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f780835c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7800155df0 con 0x7f78000a8530 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f780835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000c0b90 con 0x7f7800152780 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78000c7cd0 0x7f78000a9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f78000c7cd0 0x7f78000a9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39272/0 (socket says 172.21.15.78:39272) 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 -- 172.21.15.78:0/3405229882 learned_addr learned my addr 172.21.15.78:0/3405229882 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78068f9640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78000727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78058f7640 1 --2- 172.21.15.78:0/3405229882 >> v2:172.21.15.150:3300/0 conn(0x7f78000a8530 0x7f7800152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 -- 172.21.15.78:0/3405229882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 msgr2=0x7f78000727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78000727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 -- 172.21.15.78:0/3405229882 >> v2:172.21.15.150:3300/0 conn(0x7f78000a8530 msgr2=0x7f7800152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 --2- 172.21.15.78:0/3405229882 >> v2:172.21.15.150:3300/0 conn(0x7f78000a8530 0x7f7800152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78060f8640 1 -- 172.21.15.78:0/3405229882 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78000c0e70 con 0x7f78000c7cd0 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78068f9640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78000727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.019+0000 7f78058f7640 1 --2- 172.21.15.78:0/3405229882 >> v2:172.21.15.150:3300/0 conn(0x7f78000a8530 0x7f7800152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:18.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.020+0000 7f78060f8640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000c7cd0 0x7f78000a9e00 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f780006b100 tx=0x7f77fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0d699e24f075f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:18.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.020+0000 7f78050f6640 1 -- 172.21.15.78:0/3405229882 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc02ca70 con 0x7f78000c7cd0 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.020+0000 7f78050f6640 1 -- 172.21.15.78:0/3405229882 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc02cbd0 con 0x7f78000c7cd0 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.020+0000 7f78050f6640 1 -- 172.21.15.78:0/3405229882 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0343f0 con 0x7f78000c7cd0 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 -- 172.21.15.78:0/3405229882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000c7cd0 msgr2=0x7f78000a9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000c7cd0 0x7f78000a9e00 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f780006b100 tx=0x7f77fc02bed0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 -- 172.21.15.78:0/3405229882 shutdown_connections 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78000727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 --2- 172.21.15.78:0/3405229882 >> v2:172.21.15.150:3300/0 conn(0x7f78000a8530 0x7f7800152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 --2- 172.21.15.78:0/3405229882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000c7cd0 0x7f78000a9e00 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 -- 172.21.15.78:0/3405229882 >> 172.21.15.78:0/3405229882 conn(0x7f780006f5e0 msgr2=0x7f780006f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 -- 172.21.15.78:0/3405229882 shutdown_connections 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.021+0000 7f780835c640 1 -- 172.21.15.78:0/3405229882 wait complete. 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.022+0000 7f780835c640 1 Processor -- start 2024-02-19T14:58:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.022+0000 7f780835c640 1 -- start start 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 0x7f78000aaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78000c7cd0 0x7f7800152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78001edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78001eec40 con 0x7f78000a8530 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78001ee960 con 0x7f78000c7cd0 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78001eb2a0 con 0x7f7800152780 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 0x7f78000aaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 0x7f78000aaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39274/0 (socket says 172.21.15.78:39274) 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 -- 172.21.15.78:0/2699679692 learned_addr learned my addr 172.21.15.78:0/2699679692 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78068f9640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78001edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78058f7640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78000c7cd0 0x7f7800152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 -- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 msgr2=0x7f78001edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78001edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 -- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78000c7cd0 msgr2=0x7f7800152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:18.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78000c7cd0 0x7f7800152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 -- 172.21.15.78:0/2699679692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc009b30 con 0x7f78000a8530 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f78060f8640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 0x7f78000aaaa0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000f80 tx=0x7f77fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc03e450 con 0x7f78000a8530 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc03e5f0 con 0x7f78000a8530 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78001eb580 con 0x7f78000a8530 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.023+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc03e9a0 con 0x7f78000a8530 2024-02-19T14:58:18.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.024+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78000c1f70 con 0x7f78000a8530 2024-02-19T14:58:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.024+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78001eb100 con 0x7f78000a8530 2024-02-19T14:58:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.025+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f77fc03eb00 con 0x7f78000a8530 2024-02-19T14:58:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.025+0000 7f77f6ffd640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77e406b810 0x7f77e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.025+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6095+0+0 (secure 0 0 0) 0x7f77fc0ad260 con 0x7f78000a8530 2024-02-19T14:58:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.025+0000 7f78058f7640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77e406b810 0x7f77e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.026+0000 7f78058f7640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77e406b810 0x7f77e406dcd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7800159710 tx=0x7f77f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.029+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78001eb100 con 0x7f78000a8530 2024-02-19T14:58:18.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.203+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f78000aa7c0 con 0x7f78000a8530 2024-02-19T14:58:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.904+0000 7f77f6ffd640 1 -- 172.21.15.78:0/2699679692 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v165) v1 ==== 172+0+0 (secure 0 0 0) 0x7f78000aa7c0 con 0x7f78000a8530 2024-02-19T14:58:18.907 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs2_data' removed 2024-02-19T14:58:18.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77e406b810 msgr2=0x7f77e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77e406b810 0x7f77e406dcd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7800159710 tx=0x7f77f0008040 comp rx=0 tx=0).stop 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 msgr2=0x7f78000aaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 0x7f78000aaaa0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000f80 tx=0x7f77fc004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 shutdown_connections 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f77e406b810 0x7f77e406dcd0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7800152780 0x7f78001edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78000c7cd0 0x7f7800152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 --2- 172.21.15.78:0/2699679692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78000a8530 0x7f78000aaaa0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.906+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 >> 172.21.15.78:0/2699679692 conn(0x7f780006f5e0 msgr2=0x7f7800074200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.907+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 shutdown_connections 2024-02-19T14:58:18.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:18.907+0000 7f780835c640 1 -- 172.21.15.78:0/2699679692 wait complete. 2024-02-19T14:58:18.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-19T14:58:19.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.025+0000 7f1ff149d640 1 Processor -- start 2024-02-19T14:58:19.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.025+0000 7f1ff149d640 1 -- start start 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1ff149d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1fec152480 0x7f1fec1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1ff149d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fec0c7070 0x7f1fec06ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1ff149d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec06d350 0x7f1fec0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1ff149d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec155ef0 con 0x7f1fec152480 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1ff149d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec06dec0 con 0x7f1fec0c7070 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1ff149d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0c75c0 con 0x7f1fec06d350 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1fec152480 0x7f1fec1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1fec152480 0x7f1fec1559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39284/0 (socket says 172.21.15.78:39284) 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 -- 172.21.15.78:0/2252208770 learned_addr learned my addr 172.21.15.78:0/2252208770 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feb7fe640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec06d350 0x7f1fec0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 -- 172.21.15.78:0/2252208770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec06d350 msgr2=0x7f1fec0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1fea7fc640 1 --2- 172.21.15.78:0/2252208770 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0c7070 0x7f1fec06ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec06d350 0x7f1fec0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 -- 172.21.15.78:0/2252208770 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0c7070 msgr2=0x7f1fec06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 --2- 172.21.15.78:0/2252208770 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0c7070 0x7f1fec06ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feaffd640 1 -- 172.21.15.78:0/2252208770 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0c78a0 con 0x7f1fec152480 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1feb7fe640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec06d350 0x7f1fec0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.026+0000 7f1fea7fc640 1 --2- 172.21.15.78:0/2252208770 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0c7070 0x7f1fec06ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.027+0000 7f1feaffd640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec1559b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0c6a90 tx=0x7f1fdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11be6f95944e16c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.027+0000 7f1fe9ffb640 1 -- 172.21.15.78:0/2252208770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc031870 con 0x7f1fec152480 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.027+0000 7f1fe9ffb640 1 -- 172.21.15.78:0/2252208770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc0319d0 con 0x7f1fec152480 2024-02-19T14:58:19.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.027+0000 7f1fe9ffb640 1 -- 172.21.15.78:0/2252208770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc031d60 con 0x7f1fec152480 2024-02-19T14:58:19.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 -- 172.21.15.78:0/2252208770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 msgr2=0x7f1fec1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec1559b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0c6a90 tx=0x7f1fdc02b910 comp rx=0 tx=0).stop 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 -- 172.21.15.78:0/2252208770 shutdown_connections 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec06d350 0x7f1fec0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 --2- 172.21.15.78:0/2252208770 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0c7070 0x7f1fec06ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 --2- 172.21.15.78:0/2252208770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec1559b0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 -- 172.21.15.78:0/2252208770 >> 172.21.15.78:0/2252208770 conn(0x7f1fec0ca8a0 msgr2=0x7f1fec0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 -- 172.21.15.78:0/2252208770 shutdown_connections 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.028+0000 7f1ff149d640 1 -- 172.21.15.78:0/2252208770 wait complete. 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.029+0000 7f1ff149d640 1 Processor -- start 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.029+0000 7f1ff149d640 1 -- start start 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1ff149d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec0c7070 0x7f1fec0ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1ff149d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec0be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1ff149d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0ac8f0 0x7f1fec0be910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1ff149d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0ad060 con 0x7f1fec152480 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1ff149d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0bfcb0 con 0x7f1fec0ac8f0 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1ff149d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec158fd0 con 0x7f1fec0c7070 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec0be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec0be3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39298/0 (socket says 172.21.15.78:39298) 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 -- 172.21.15.78:0/3758689659 learned_addr learned my addr 172.21.15.78:0/3758689659 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1feaffd640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec0c7070 0x7f1fec0ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1feb7fe640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0ac8f0 0x7f1fec0be910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 -- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec0c7070 msgr2=0x7f1fec0ac3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec0c7070 0x7f1fec0ac3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 -- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0ac8f0 msgr2=0x7f1fec0be910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0ac8f0 0x7f1fec0be910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 -- 172.21.15.78:0/3758689659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0094d0 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fea7fc640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec0be3d0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000c7f0 tx=0x7f1fe000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.030+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe000cea0 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.031+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe0004230 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.031+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec0bff50 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.031+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0007830 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.031+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec158c80 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.032+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec068990 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.032+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1fe0007990 con 0x7f1fec152480 2024-02-19T14:58:19.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.032+0000 7f1fd37fe640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fd806b810 0x7f1fd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.032+0000 7f1feaffd640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fd806b810 0x7f1fd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.032+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f1fe008a160 con 0x7f1fec152480 2024-02-19T14:58:19.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.033+0000 7f1feaffd640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fd806b810 0x7f1fd806dcd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc036000 tx=0x7f1fdc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.036+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1fe0054ba0 con 0x7f1fec152480 2024-02-19T14:58:19.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.208+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f1fec074990 con 0x7f1fec152480 2024-02-19T14:58:19.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.211+0000 7f1fd37fe640 1 -- 172.21.15.78:0/3758689659 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v54) v1 ==== 96+0+0 (secure 0 0 0) 0x7f1fe0058260 con 0x7f1fec152480 2024-02-19T14:58:19.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.213+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fd806b810 msgr2=0x7f1fd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.213+0000 7f1ff149d640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fd806b810 0x7f1fd806dcd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc036000 tx=0x7f1fdc03c040 comp rx=0 tx=0).stop 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.213+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 msgr2=0x7f1fec0be3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.213+0000 7f1ff149d640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec0be3d0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000c7f0 tx=0x7f1fe000ccb0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 shutdown_connections 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fd806b810 0x7f1fd806dcd0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fec152480 0x7f1fec0be3d0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0ac8f0 0x7f1fec0be910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 --2- 172.21.15.78:0/3758689659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fec0c7070 0x7f1fec0ac3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 >> 172.21.15.78:0/3758689659 conn(0x7f1fec0ca8a0 msgr2=0x7f1fec14e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 shutdown_connections 2024-02-19T14:58:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.214+0000 7f1ff149d640 1 -- 172.21.15.78:0/3758689659 wait complete. 2024-02-19T14:58:19.227 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-19T14:58:19.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.326+0000 7f90d6491640 1 Processor -- start 2024-02-19T14:58:19.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.326+0000 7f90d6491640 1 -- start start 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90d6491640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90d00718f0 0x7f90d0071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90d6491640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0159130 con 0x7f90d00acc50 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90d6491640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d00ad030 con 0x7f90d0155070 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90d6491640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d00c2cf0 con 0x7f90d00718f0 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90cffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f90d00acc50 0x7f90d00c17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58146/0 (socket says 172.21.15.78:58146) 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90cffff640 1 -- 172.21.15.78:0/1804652929 learned_addr learned my addr 172.21.15.78:0/1804652929 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760123619 0 0) 0x7f90d0159130 con 0x7f90d00acc50 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b8003520 con 0x7f90d00acc50 2024-02-19T14:58:19.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90d4a2e640 1 --2- 172.21.15.78:0/1804652929 >> v2:172.21.15.150:3300/0 conn(0x7f90d00718f0 0x7f90d0071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650831725 0 0) 0x7f90b8003520 con 0x7f90d00acc50 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 >> v2:172.21.15.150:3300/0 conn(0x7f90d00718f0 msgr2=0x7f90d0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 --2- 172.21.15.78:0/1804652929 >> v2:172.21.15.150:3300/0 conn(0x7f90d00718f0 0x7f90d0071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 >> v1:172.21.15.150:6789/0 conn(0x7f90d0155070 legacy=0x7f90d0155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d01595c0 con 0x7f90d00acc50 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f90c0001670 con 0x7f90d00acc50 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 >> v1:172.21.15.78:6789/0 conn(0x7f90d00acc50 legacy=0x7f90d00c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90b80047c0 0x7f90b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b80052a0 0x7f90b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00c2cf0 con 0x7f90b8003970 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b800c170 con 0x7f90b80052a0 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b800c330 con 0x7f90b80047c0 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.327+0000 7f90cffff640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90d4a2e640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b80052a0 0x7f90b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cf7fe640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90b80047c0 0x7f90b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cffff640 1 -- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90b80047c0 msgr2=0x7f90b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cffff640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90b80047c0 0x7f90b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cffff640 1 -- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b80052a0 msgr2=0x7f90b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cffff640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b80052a0 0x7f90b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90d4a2e640 1 -- 172.21.15.78:0/1804652929 reap_dead start 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cffff640 1 -- 172.21.15.78:0/1804652929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b8003520 con 0x7f90b8003970 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.328+0000 7f90cffff640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90b8003d90 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c210 tx=0x7f90b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a0d5f199fa1abf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0005ad0 con 0x7f90b8003970 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0005c30 con 0x7f90b8003970 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90ceffd640 1 -- 172.21.15.78:0/1804652929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c002ca90 con 0x7f90b8003970 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 -- 172.21.15.78:0/1804652929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 msgr2=0x7f90b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90b8003d90 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c210 tx=0x7f90b80057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 -- 172.21.15.78:0/1804652929 shutdown_connections 2024-02-19T14:58:19.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 --2- 172.21.15.78:0/1804652929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90b8003d90 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 -- 172.21.15.78:0/1804652929 >> 172.21.15.78:0/1804652929 conn(0x7f90d00c7800 msgr2=0x7f90d00c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 -- 172.21.15.78:0/1804652929 shutdown_connections 2024-02-19T14:58:19.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.329+0000 7f90d6491640 1 -- 172.21.15.78:0/1804652929 wait complete. 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.330+0000 7f90d6491640 1 Processor -- start 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 -- start start 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90d0133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90d0133f80 0x7f90d01396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90d0139be0 0x7f90d00c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00c2030 con 0x7f90b8003970 2024-02-19T14:58:19.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d01379a0 con 0x7f90d0139be0 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90d6491640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00c4a70 con 0x7f90d0133f80 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90d0133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90d0133a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39312/0 (socket says 172.21.15.78:39312) 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 -- 172.21.15.78:0/1874340604 learned_addr learned my addr 172.21.15.78:0/1874340604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 -- 172.21.15.78:0/1874340604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90d0133f80 msgr2=0x7f90d01396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90d0133f80 0x7f90d01396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 -- 172.21.15.78:0/1874340604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90d0139be0 msgr2=0x7f90d00c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90d0139be0 0x7f90d00c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.331+0000 7f90cffff640 1 -- 172.21.15.78:0/1874340604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d01eb5c0 con 0x7f90b8003970 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.332+0000 7f90cffff640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90d0133a40 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f90c0000f60 tx=0x7f90c002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.332+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c003c040 con 0x7f90b8003970 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.332+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c002cd20 con 0x7f90b8003970 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.332+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00c4d10 con 0x7f90b8003970 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.332+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0007290 con 0x7f90b8003970 2024-02-19T14:58:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.332+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d00c6580 con 0x7f90b8003970 2024-02-19T14:58:19.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.333+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d00c4ea0 con 0x7f90b8003970 2024-02-19T14:58:19.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.333+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f90c0032070 con 0x7f90b8003970 2024-02-19T14:58:19.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.333+0000 7f90ccff9640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90bc06b840 0x7f90bc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.334+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f90c00ab440 con 0x7f90b8003970 2024-02-19T14:58:19.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.334+0000 7f90cf7fe640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90bc06b840 0x7f90bc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.334+0000 7f90cf7fe640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90bc06b840 0x7f90bc06dd00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f90c4000c00 tx=0x7f90c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.338+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f90c0075f10 con 0x7f90b8003970 2024-02-19T14:58:19.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.508+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f90d006c820 con 0x7f90b8003970 2024-02-19T14:58:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.511+0000 7f90ccff9640 1 -- 172.21.15.78:0/1874340604 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v55) v1 ==== 96+0+0 (secure 0 0 0) 0x7f90c00795d0 con 0x7f90b8003970 2024-02-19T14:58:19.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90bc06b840 msgr2=0x7f90bc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90bc06b840 0x7f90bc06dd00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f90c4000c00 tx=0x7f90c4009040 comp rx=0 tx=0).stop 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 msgr2=0x7f90d0133a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90d0133a40 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f90c0000f60 tx=0x7f90c002e070 comp rx=0 tx=0).stop 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 shutdown_connections 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f90bc06b840 0x7f90bc06dd00 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90d0139be0 0x7f90d00c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90d0133f80 0x7f90d01396a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 --2- 172.21.15.78:0/1874340604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90b8003970 0x7f90d0133a40 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.513+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 >> 172.21.15.78:0/1874340604 conn(0x7f90d00c7800 msgr2=0x7f90d0071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.514+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 shutdown_connections 2024-02-19T14:58:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.514+0000 7f90d6491640 1 -- 172.21.15.78:0/1874340604 wait complete. 2024-02-19T14:58:19.527 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:58:19.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.629+0000 7f9eb11c7640 1 Processor -- start 2024-02-19T14:58:19.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.629+0000 7f9eb11c7640 1 -- start start 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.630+0000 7f9eb11c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9eac0c3200 0x7f9eac0ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.630+0000 7f9eb11c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac0718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.630+0000 7f9eb11c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eac0c35e0 con 0x7f9eac071e30 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.630+0000 7f9eb11c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac14d3c0 con 0x7f9eac0c3200 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.630+0000 7f9eb11c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0acad0 con 0x7f9eac153b60 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9eac0c3200 0x7f9eac0ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaa59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac0718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eab59e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9eac071e30 0x7f9eac06ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:33898/0 (socket says 172.21.15.78:33898) 2024-02-19T14:58:19.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eab59e640 1 -- 172.21.15.78:0/2825160886 learned_addr learned my addr 172.21.15.78:0/2825160886 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:19.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 msgr2=0x7f9eac0718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaad9d640 1 --2- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac0718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2825160886 >> v1:172.21.15.150:6789/0 conn(0x7f9eac071e30 legacy=0x7f9eac06ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2825160886 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac0ad000 con 0x7f9eac0c3200 2024-02-19T14:58:19.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.631+0000 7f9eaa59c640 1 --2- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac0718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eaad9d640 1 --2- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac0ac2e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0097a0 tx=0x7f9e9c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce12c2a52b14588 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9ea9d9b640 1 -- 172.21.15.78:0/2825160886 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c02d040 con 0x7f9eac0c3200 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9ea9d9b640 1 -- 172.21.15.78:0/2825160886 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c031810 con 0x7f9eac0c3200 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9ea9d9b640 1 -- 172.21.15.78:0/2825160886 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c031ba0 con 0x7f9eac0c3200 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 msgr2=0x7f9eac0ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac0ac2e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0097a0 tx=0x7f9e9c02b8b0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2825160886 shutdown_connections 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac0718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2825160886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac0ac2e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2825160886 >> 172.21.15.78:0/2825160886 conn(0x7f9eac0c7800 msgr2=0x7f9eac0c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.632+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2825160886 shutdown_connections 2024-02-19T14:58:19.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.633+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2825160886 wait complete. 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 Processor -- start 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 -- start start 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 0x7f9eac14d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac1484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0bc9b0 con 0x7f9eac071e30 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac1e8d90 con 0x7f9eac0c3200 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.634+0000 7f9eb11c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac1e92c0 con 0x7f9eac153b60 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 0x7f9eac14d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 0x7f9eac14d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39320/0 (socket says 172.21.15.78:39320) 2024-02-19T14:58:19.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2544243507 learned_addr learned my addr 172.21.15.78:0/2544243507 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eab59e640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaa59c640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac1484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 msgr2=0x7f9eac1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac1489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 msgr2=0x7f9eac1484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac1484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 -- 172.21.15.78:0/2544243507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea0009400 con 0x7f9eac071e30 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaa59c640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eaad9d640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 0x7f9eac14d780 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0098c0 tx=0x7f9e9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c03ad60 con 0x7f9eac071e30 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9c0423f0 con 0x7f9eac071e30 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c009400 con 0x7f9eac071e30 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c0427a0 con 0x7f9eac071e30 2024-02-19T14:58:19.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.635+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac1553d0 con 0x7f9eac071e30 2024-02-19T14:58:19.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.636+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac074990 con 0x7f9eac071e30 2024-02-19T14:58:19.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.637+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9e9c04b3b0 con 0x7f9eac071e30 2024-02-19T14:58:19.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.637+0000 7f9e937fe640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e9806b840 0x7f9e9806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.637+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f9e9c0ac180 con 0x7f9eac071e30 2024-02-19T14:58:19.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.637+0000 7f9eaa59c640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e9806b840 0x7f9e9806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.638+0000 7f9eaa59c640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e9806b840 0x7f9e9806dd00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9eac152450 tx=0x7f9ea0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.641+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9e9c076d00 con 0x7f9eac071e30 2024-02-19T14:58:19.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.813+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9eac0c4130 con 0x7f9eac071e30 2024-02-19T14:58:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.817+0000 7f9e937fe640 1 -- 172.21.15.78:0/2544243507 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9e9c07a3c0 con 0x7f9eac071e30 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e9806b840 msgr2=0x7f9e9806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e9806b840 0x7f9e9806dd00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9eac152450 tx=0x7f9ea0008040 comp rx=0 tx=0).stop 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 msgr2=0x7f9eac14d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 0x7f9eac14d780 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0098c0 tx=0x7f9e9c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 shutdown_connections 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e9806b840 0x7f9e9806dd00 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9eac153b60 0x7f9eac1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eac0c3200 0x7f9eac1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 --2- 172.21.15.78:0/2544243507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9eac071e30 0x7f9eac14d780 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 >> 172.21.15.78:0/2544243507 conn(0x7f9eac0c7800 msgr2=0x7f9eac074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 shutdown_connections 2024-02-19T14:58:19.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.819+0000 7f9eb11c7640 1 -- 172.21.15.78:0/2544243507 wait complete. 2024-02-19T14:58:19.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-02-19T14:58:19.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.930+0000 7f33892c7640 1 Processor -- start 2024-02-19T14:58:19.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.930+0000 7f33892c7640 1 -- start start 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33892c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3384152700 0x7f33840c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33892c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33841551f0 con 0x7f33840c5730 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33892c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3384155490 con 0x7f33840c8e00 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33892c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3384155770 con 0x7f3384152700 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f337bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f33840c5730 0x7f338406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58162/0 (socket says 172.21.15.78:58162) 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f337bfff640 1 -- 172.21.15.78:0/2135086701 learned_addr learned my addr 172.21.15.78:0/2135086701 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:19.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360271723 0 0) 0x7f33841551f0 con 0x7f33840c5730 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3370003520 con 0x7f33840c5730 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f3382ffd640 1 --2- 172.21.15.78:0/2135086701 >> v2:172.21.15.150:3300/0 conn(0x7f3384152700 0x7f33840c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3989686208 0 0) 0x7f3370003520 con 0x7f33840c5730 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 >> v2:172.21.15.150:3300/0 conn(0x7f3384152700 msgr2=0x7f33840c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 --2- 172.21.15.78:0/2135086701 >> v2:172.21.15.150:3300/0 conn(0x7f3384152700 0x7f33840c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 >> v1:172.21.15.150:6789/0 conn(0x7f33840c8e00 legacy=0x7f338415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3384155a50 con 0x7f33840c5730 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f336c001670 con 0x7f33840c5730 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f3382ffd640 1 --2- 172.21.15.78:0/2135086701 >> v2:172.21.15.150:3300/0 conn(0x7f3384152700 0x7f33840c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 >> v1:172.21.15.78:6789/0 conn(0x7f33840c5730 legacy=0x7f338406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3370003970 0x7f3370003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700047c0 0x7f3370004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33700052a0 0x7f337000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384155770 con 0x7f33700052a0 2024-02-19T14:58:19.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337000c170 con 0x7f3370003970 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337000c330 con 0x7f33700047c0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.931+0000 7f33837fe640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33700052a0 0x7f337000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f3382ffd640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3370003970 0x7f3370003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f337bfff640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700047c0 0x7f3370004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f33837fe640 1 -- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700047c0 msgr2=0x7f3370004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f33837fe640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700047c0 0x7f3370004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f33837fe640 1 -- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3370003970 msgr2=0x7f3370003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f33837fe640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3370003970 0x7f3370003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f33837fe640 1 -- 172.21.15.78:0/2135086701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370003520 con 0x7f33700052a0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.932+0000 7f33837fe640 1 -- 172.21.15.78:0/2135086701 reap_dead start 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33837fe640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33700052a0 0x7f337000b980 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f337c001e20 tx=0x7f33700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7ca24f59174ffb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c003b70 con 0x7f33700052a0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c003cd0 con 0x7f33700052a0 2024-02-19T14:58:19.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33827fc640 1 -- 172.21.15.78:0/2135086701 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c005310 con 0x7f33700052a0 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 -- 172.21.15.78:0/2135086701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33700052a0 msgr2=0x7f337000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33700052a0 0x7f337000b980 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f337c001e20 tx=0x7f33700057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 -- 172.21.15.78:0/2135086701 shutdown_connections 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 --2- 172.21.15.78:0/2135086701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33700052a0 0x7f337000b980 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 -- 172.21.15.78:0/2135086701 >> 172.21.15.78:0/2135086701 conn(0x7f33840cac90 msgr2=0x7f33840cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 -- 172.21.15.78:0/2135086701 shutdown_connections 2024-02-19T14:58:19.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.933+0000 7f33892c7640 1 -- 172.21.15.78:0/2135086701 wait complete. 2024-02-19T14:58:19.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.934+0000 7f33892c7640 1 Processor -- start 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.934+0000 7f33892c7640 1 -- start start 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f33892c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700052a0 0x7f33840be370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f33892c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 0x7f33840c0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f33892c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33840ae2f0 0x7f33840ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f33892c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840aae30 con 0x7f33840b0040 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f33892c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840aece0 con 0x7f33840ae2f0 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f33892c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33841eb1c0 con 0x7f33700052a0 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 0x7f33840c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 0x7f33840c0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39348/0 (socket says 172.21.15.78:39348) 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 -- 172.21.15.78:0/2709475151 learned_addr learned my addr 172.21.15.78:0/2709475151 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f3382ffd640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700052a0 0x7f33840be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 -- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700052a0 msgr2=0x7f33840be370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700052a0 0x7f33840be370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 -- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33840ae2f0 msgr2=0x7f33840ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33840ae2f0 0x7f33840ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.935+0000 7f337bfff640 1 -- 172.21.15.78:0/2709475151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840ad530 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.936+0000 7f337bfff640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 0x7f33840c0880 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f336c007290 tx=0x7f336c007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.936+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c00f980 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.936+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c00fb20 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.936+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c00fed0 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.936+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33840ad810 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.936+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33840addd0 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.937+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3384074990 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.937+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f336c01d460 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.937+0000 7f337b7fe640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f336406b840 0x7f336406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.938+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f336c08a3a0 con 0x7f33840b0040 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.938+0000 7f3382ffd640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f336406b840 0x7f336406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:19.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.938+0000 7f3382ffd640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f336406b840 0x7f336406dd00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3384159320 tx=0x7f3374008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:19.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:19.942+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f336c054e70 con 0x7f33840b0040 2024-02-19T14:58:20.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.181+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7f33840b04c0 con 0x7f33840b0040 2024-02-19T14:58:20.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.913+0000 7f337b7fe640 1 -- 172.21.15.78:0/2709475151 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v377) v1 ==== 146+0+0 (secure 0 0 0) 0x7f336c058530 con 0x7f33840b0040 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f336406b840 msgr2=0x7f336406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f336406b840 0x7f336406dd00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3384159320 tx=0x7f3374008040 comp rx=0 tx=0).stop 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 msgr2=0x7f33840c0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 0x7f33840c0880 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f336c007290 tx=0x7f336c007690 comp rx=0 tx=0).stop 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 shutdown_connections 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f336406b840 0x7f336406dd00 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33840b0040 0x7f33840c0880 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33840ae2f0 0x7f33840ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 --2- 172.21.15.78:0/2709475151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33700052a0 0x7f33840be370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:20.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 >> 172.21.15.78:0/2709475151 conn(0x7f33840cac90 msgr2=0x7f33840c9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 shutdown_connections 2024-02-19T14:58:20.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:20.915+0000 7f33892c7640 1 -- 172.21.15.78:0/2709475151 wait complete. 2024-02-19T14:58:20.930 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. ... ok 2024-02-19T14:58:20.930 INFO:tasks.cephfs_test_runner:Starting test: test_down (tasks.cephfs.test_failover.TestClusterResize) That down setting toggles and sets max_mds appropriately. 2024-02-19T14:58:20.930 INFO:tasks.cephfs_test_runner:test_down (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T14:58:20.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.030+0000 7fb80f384640 1 Processor -- start 2024-02-19T14:58:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.030+0000 7fb80f384640 1 -- start start 2024-02-19T14:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80f384640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb808159320 0x7fb80813add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80f384640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080c4190 con 0x7fb8080c17e0 2024-02-19T14:58:21.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80f384640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080c4470 con 0x7fb8081555e0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80f384640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8080c2030 con 0x7fb808159320 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80c91f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb8080c17e0 0x7fb8080718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58178/0 (socket says 172.21.15.78:58178) 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80c91f640 1 -- 172.21.15.78:0/4226400115 learned_addr learned my addr 172.21.15.78:0/4226400115 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb80d120640 1 --2- 172.21.15.78:0/4226400115 >> v2:172.21.15.150:3300/0 conn(0x7fb808159320 0x7fb80813add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434689487 0 0) 0x7fb8080c4190 con 0x7fb8080c17e0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4003520 con 0x7fb8080c17e0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634465237 0 0) 0x7fb7f4003520 con 0x7fb8080c17e0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 >> v2:172.21.15.150:3300/0 conn(0x7fb808159320 msgr2=0x7fb80813add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 --2- 172.21.15.78:0/4226400115 >> v2:172.21.15.150:3300/0 conn(0x7fb808159320 0x7fb80813add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 >> v1:172.21.15.150:6789/0 conn(0x7fb8081555e0 legacy=0x7fb80815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080c4750 con 0x7fb8080c17e0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb7f8001670 con 0x7fb8080c17e0 2024-02-19T14:58:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 >> v1:172.21.15.78:6789/0 conn(0x7fb8080c17e0 legacy=0x7fb8080718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7f4003970 0x7fb7f4003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f4004370 0x7fb7f4004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7f40050e0 0x7fb7f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080c2030 con 0x7fb7f40050e0 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400c020 con 0x7fb7f4003970 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.031+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400c1e0 con 0x7fb7f4004370 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7f40050e0 0x7fb7f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d120640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7f4003970 0x7fb7f4003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80c91f640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f4004370 0x7fb7f4004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 -- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f4004370 msgr2=0x7fb7f4004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f4004370 0x7fb7f4004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 -- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7f4003970 msgr2=0x7fb7f4003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7f4003970 0x7fb7f4003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 -- 172.21.15.78:0/4226400115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4003520 con 0x7fb7f40050e0 2024-02-19T14:58:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80c91f640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f4004370 0x7fb7f4004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.032+0000 7fb80d921640 1 -- 172.21.15.78:0/4226400115 reap_dead start 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80d921640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7f40050e0 0x7fb7f400b980 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fb800001f60 tx=0x7fb7f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299f1b4d72f73ae5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800003b70 con 0x7fb7f40050e0 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800003cf0 con 0x7fb7f40050e0 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb7fffff640 1 -- 172.21.15.78:0/4226400115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000052b0 con 0x7fb7f40050e0 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 -- 172.21.15.78:0/4226400115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7f40050e0 msgr2=0x7fb7f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7f40050e0 0x7fb7f400b980 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fb800001f60 tx=0x7fb7f40057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 -- 172.21.15.78:0/4226400115 shutdown_connections 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 --2- 172.21.15.78:0/4226400115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7f40050e0 0x7fb7f400b980 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 -- 172.21.15.78:0/4226400115 >> 172.21.15.78:0/4226400115 conn(0x7fb8080c7800 msgr2=0x7fb8080c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 -- 172.21.15.78:0/4226400115 shutdown_connections 2024-02-19T14:58:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.033+0000 7fb80f384640 1 -- 172.21.15.78:0/4226400115 wait complete. 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.034+0000 7fb80f384640 1 Processor -- start 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.034+0000 7fb80f384640 1 -- start start 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80f384640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f40050e0 0x7fb8081e7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80f384640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8081514f0 0x7fb8081e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80f384640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 0x7fb80813d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80f384640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081e8390 con 0x7fb80813d100 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80f384640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081eb620 con 0x7fb8081514f0 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80f384640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80813e5c0 con 0x7fb7f40050e0 2024-02-19T14:58:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 0x7fb80813d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 0x7fb80813d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39352/0 (socket says 172.21.15.78:39352) 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 -- 172.21.15.78:0/3731582767 learned_addr learned my addr 172.21.15.78:0/3731582767 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d120640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f40050e0 0x7fb8081e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80c91f640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8081514f0 0x7fb8081e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 -- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f40050e0 msgr2=0x7fb8081e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f40050e0 0x7fb8081e7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 -- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8081514f0 msgr2=0x7fb8081e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8081514f0 0x7fb8081e7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80d921640 1 -- 172.21.15.78:0/3731582767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8081eb0a0 con 0x7fb80813d100 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.035+0000 7fb80c91f640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8081514f0 0x7fb8081e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.036+0000 7fb80d921640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 0x7fb80813d580 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fb80002c150 tx=0x7fb8000015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.036+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800006d50 con 0x7fb80813d100 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.036+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800003b80 con 0x7fb80813d100 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.036+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80813b4d0 con 0x7fb80813d100 2024-02-19T14:58:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.036+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80002d6f0 con 0x7fb80813d100 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.036+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80813e0a0 con 0x7fb80813d100 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.037+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80813e2c0 con 0x7fb80813d100 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.038+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb80002d850 con 0x7fb80813d100 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.038+0000 7fb7fdffb640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7ec06b840 0x7fb7ec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.038+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fb800031030 con 0x7fb80813d100 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.038+0000 7fb80d120640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7ec06b840 0x7fb7ec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.038+0000 7fb80d120640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7ec06b840 0x7fb7ec06dd00 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb804009b40 tx=0x7fb804009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.042+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb800076a10 con 0x7fb80813d100 2024-02-19T14:58:21.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.214+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb808074990 con 0x7fb80813d100 2024-02-19T14:58:21.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.215+0000 7fb7fdffb640 1 -- 172.21.15.78:0/3731582767 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v165) v1 ==== 74+0+13813 (secure 0 0 0) 0x7fb80007a0d0 con 0x7fb80813d100 2024-02-19T14:58:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:21.218 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":165,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:18.898735+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6837/1801432695":"2024-02-20T14:58:15.891197+0000","172.21.15.150:6834/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6835/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.150:6832/1911236290":"2024-02-20T14:58:10.599219+0000","172.21.15.150:6833/2395461731":"2024-02-20T14:58:10.235050+0000","172.21.15.78:6836/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6834/2997633530":"2024-02-20T14:58:01.312219+0000","172.21.15.78:6836/1801432695":"2024-02-20T14:58:15.891197+0000","172.21.15.78:6837/903597448":"2024-02-20T14:58:01.307371+0000","172.21.15.78:6835/903597448":"2024-02-20T14:58:01.307371+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7ec06b840 msgr2=0x7fb7ec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7ec06b840 0x7fb7ec06dd00 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb804009b40 tx=0x7fb804009190 comp rx=0 tx=0).stop 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 msgr2=0x7fb80813d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 0x7fb80813d580 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fb80002c150 tx=0x7fb8000015b0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 shutdown_connections 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb7ec06b840 0x7fb7ec06dd00 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb80813d100 0x7fb80813d580 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8081514f0 0x7fb8081e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 --2- 172.21.15.78:0/3731582767 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7f40050e0 0x7fb8081e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 >> 172.21.15.78:0/3731582767 conn(0x7fb8080c7800 msgr2=0x7fb808153080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:21.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 shutdown_connections 2024-02-19T14:58:21.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.217+0000 7fb80f384640 1 -- 172.21.15.78:0/3731582767 wait complete. 2024-02-19T14:58:21.231 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:58:21.231 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:58:21.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.331+0000 7ff7c544e640 1 Processor -- start 2024-02-19T14:58:21.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.331+0000 7ff7c544e640 1 -- start start 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7c544e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7c00c8500 0x7ff7c00c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7c544e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7c00c2f50 0x7ff7c0158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7c544e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c00c34a0 con 0x7ff7c00c42e0 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7c544e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c006c090 con 0x7ff7c00c2f50 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7c544e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c006d6a0 con 0x7ff7c00c8500 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7c00c2f50 0x7ff7c0158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff7c00c2f50 0x7ff7c0158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39364/0 (socket says 172.21.15.78:39364) 2024-02-19T14:58:21.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 -- 172.21.15.78:0/466728169 learned_addr learned my addr 172.21.15.78:0/466728169 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7beffd640 1 --2- 172.21.15.78:0/466728169 >> v2:172.21.15.150:3300/0 conn(0x7ff7c00c8500 0x7ff7c00c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 -- 172.21.15.78:0/466728169 >> v2:172.21.15.150:3300/0 conn(0x7ff7c00c8500 msgr2=0x7ff7c00c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 --2- 172.21.15.78:0/466728169 >> v2:172.21.15.150:3300/0 conn(0x7ff7c00c8500 0x7ff7c00c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 -- 172.21.15.78:0/466728169 >> v1:172.21.15.150:6789/0 conn(0x7ff7c00c42e0 legacy=0x7ff7c0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7bf7fe640 1 -- 172.21.15.78:0/466728169 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c006c8e0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.332+0000 7ff7beffd640 1 --2- 172.21.15.78:0/466728169 >> v2:172.21.15.150:3300/0 conn(0x7ff7c00c8500 0x7ff7c00c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.333+0000 7ff7bf7fe640 1 --2- 172.21.15.78:0/466728169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c0158d60 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00c9350 tx=0x7ff7b802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1c32ecba37c502 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.333+0000 7ff7be7fc640 1 -- 172.21.15.78:0/466728169 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b802c9f0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.333+0000 7ff7be7fc640 1 -- 172.21.15.78:0/466728169 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b802cb50 con 0x7ff7c00c2f50 2024-02-19T14:58:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.333+0000 7ff7be7fc640 1 -- 172.21.15.78:0/466728169 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b80343f0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 -- 172.21.15.78:0/466728169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 msgr2=0x7ff7c0158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 --2- 172.21.15.78:0/466728169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c0158d60 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00c9350 tx=0x7ff7b802be50 comp rx=0 tx=0).stop 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 -- 172.21.15.78:0/466728169 shutdown_connections 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 --2- 172.21.15.78:0/466728169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c0158d60 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 --2- 172.21.15.78:0/466728169 >> v2:172.21.15.150:3300/0 conn(0x7ff7c00c8500 0x7ff7c00c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 -- 172.21.15.78:0/466728169 >> 172.21.15.78:0/466728169 conn(0x7ff7c00aa300 msgr2=0x7ff7c00aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 -- 172.21.15.78:0/466728169 shutdown_connections 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.334+0000 7ff7c544e640 1 -- 172.21.15.78:0/466728169 wait complete. 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 Processor -- start 2024-02-19T14:58:21.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 -- start start 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c00ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c00c8500 0x7ff7c00ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c00ab930 0x7ff7c01518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00ac400 con 0x7ff7c00c2f50 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00abe40 con 0x7ff7c00ab930 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.335+0000 7ff7c544e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00ac120 con 0x7ff7c00c8500 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c00ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c00ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39380/0 (socket says 172.21.15.78:39380) 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 -- 172.21.15.78:0/1917483136 learned_addr learned my addr 172.21.15.78:0/1917483136 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7b7fff640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c00c8500 0x7ff7c00ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7bf7fe640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c00ab930 0x7ff7c01518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 -- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c00c8500 msgr2=0x7ff7c00ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c00c8500 0x7ff7c00ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 -- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c00ab930 msgr2=0x7ff7c01518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c00ab930 0x7ff7c01518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 -- 172.21.15.78:0/1917483136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8009b30 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7beffd640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c00ad1b0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c7f0 tx=0x7ff7b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0007820 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b0007980 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c01510e0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0007cf0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.336+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c00ad6f0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.337+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c0071c20 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.338+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff7b00113f0 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.338+0000 7ff7b77fe640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7a006b810 0x7ff7a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.338+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7ff7b008a280 con 0x7ff7c00c2f50 2024-02-19T14:58:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.338+0000 7ff7b7fff640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7a006b810 0x7ff7a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.338+0000 7ff7b7fff640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7a006b810 0x7ff7a006dcd0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00aad60 tx=0x7ff7a8007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.342+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff7b0054d70 con 0x7ff7c00c2f50 2024-02-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.511+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7c0074990 con 0x7ff7c00c2f50 2024-02-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.511+0000 7ff7b77fe640 1 -- 172.21.15.78:0/1917483136 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v165) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff7b0058430 con 0x7ff7c00c2f50 2024-02-19T14:58:21.514 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7a006b810 msgr2=0x7ff7a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7a006b810 0x7ff7a006dcd0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00aad60 tx=0x7ff7a8007e30 comp rx=0 tx=0).stop 2024-02-19T14:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 msgr2=0x7ff7c00ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c00ad1b0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c7f0 tx=0x7ff7b000ccb0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 shutdown_connections 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff7a006b810 0x7ff7a006dcd0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c00ab930 0x7ff7c01518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c00c8500 0x7ff7c00ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 --2- 172.21.15.78:0/1917483136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff7c00c2f50 0x7ff7c00ad1b0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 >> 172.21.15.78:0/1917483136 conn(0x7ff7c00aa300 msgr2=0x7ff7c00c7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 shutdown_connections 2024-02-19T14:58:21.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.513+0000 7ff7c544e640 1 -- 172.21.15.78:0/1917483136 wait complete. 2024-02-19T14:58:21.526 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-02-19T14:58:21.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.624+0000 7fbd873c7640 1 Processor -- start 2024-02-19T14:58:21.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.624+0000 7fbd873c7640 1 -- start start 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd873c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbd80153b60 0x7fbd800ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd873c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800c35e0 con 0x7fbd800c3200 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd873c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800acad0 con 0x7fbd800c4350 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd873c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd8014d3c0 con 0x7fbd80153b60 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd84962640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbd80153b60 0x7fbd800ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd85163640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbd800c3200 0x7fbd800718f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58190/0 (socket says 172.21.15.78:58190) 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd85163640 1 -- 172.21.15.78:0/3608940303 learned_addr learned my addr 172.21.15.78:0/3608940303 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:21.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd84962640 1 -- 172.21.15.78:0/3608940303 >> v1:172.21.15.150:6789/0 conn(0x7fbd800c4350 legacy=0x7fbd8006ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:21.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd84962640 1 -- 172.21.15.78:0/3608940303 >> v1:172.21.15.78:6789/0 conn(0x7fbd800c3200 legacy=0x7fbd800718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd84962640 1 -- 172.21.15.78:0/3608940303 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd800ad000 con 0x7fbd80153b60 2024-02-19T14:58:21.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.625+0000 7fbd77fff640 1 -- 172.21.15.78:0/3608940303 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956053932 0 0) 0x7fbd800c35e0 con 0x7fbd800c3200 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.626+0000 7fbd84962640 1 --2- 172.21.15.78:0/3608940303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd800ac2e0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fbd700097a0 tx=0x7fbd7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fa0641ed7af0ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.626+0000 7fbd77fff640 1 -- 172.21.15.78:0/3608940303 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70031830 con 0x7fbd80153b60 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.626+0000 7fbd77fff640 1 -- 172.21.15.78:0/3608940303 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd70031990 con 0x7fbd80153b60 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.626+0000 7fbd77fff640 1 -- 172.21.15.78:0/3608940303 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70031d20 con 0x7fbd80153b60 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 -- 172.21.15.78:0/3608940303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 msgr2=0x7fbd800ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 --2- 172.21.15.78:0/3608940303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd800ac2e0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fbd700097a0 tx=0x7fbd7002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 -- 172.21.15.78:0/3608940303 shutdown_connections 2024-02-19T14:58:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 --2- 172.21.15.78:0/3608940303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd800ac2e0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 -- 172.21.15.78:0/3608940303 >> 172.21.15.78:0/3608940303 conn(0x7fbd800c7800 msgr2=0x7fbd800c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 -- 172.21.15.78:0/3608940303 shutdown_connections 2024-02-19T14:58:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.627+0000 7fbd873c7640 1 -- 172.21.15.78:0/3608940303 wait complete. 2024-02-19T14:58:21.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.628+0000 7fbd873c7640 1 Processor -- start 2024-02-19T14:58:21.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.628+0000 7fbd873c7640 1 -- start start 2024-02-19T14:58:21.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd800c3200 0x7fbd8014e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd800c4350 0x7fbd801e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd801e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8014e290 con 0x7fbd80153b60 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8014f930 con 0x7fbd800c4350 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8014fc10 con 0x7fbd800c3200 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd801e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd801e5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39406/0 (socket says 172.21.15.78:39406) 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 -- 172.21.15.78:0/732873228 learned_addr learned my addr 172.21.15.78:0/732873228 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85163640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd800c3200 0x7fbd8014e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd84962640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd800c4350 0x7fbd801e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 -- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd800c3200 msgr2=0x7fbd8014e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd800c3200 0x7fbd8014e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 -- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd800c4350 msgr2=0x7fbd801e5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd800c4350 0x7fbd801e5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 -- 172.21.15.78:0/732873228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70009400 con 0x7fbd80153b60 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85163640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd800c3200 0x7fbd8014e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:21.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd84962640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd800c4350 0x7fbd801e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd85964640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd801e5fd0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800c780 tx=0x7fbd7800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78004b60 con 0x7fbd80153b60 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7800ce90 con 0x7fbd80153b60 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8014fef0 con 0x7fbd80153b60 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.629+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7800a850 con 0x7fbd80153b60 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.630+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd80152bd0 con 0x7fbd80153b60 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.630+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80074990 con 0x7fbd80153b60 2024-02-19T14:58:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.631+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbd7800a9b0 con 0x7fbd80153b60 2024-02-19T14:58:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.631+0000 7fbd75ffb640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd6406b810 0x7fbd6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.631+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fbd7808a2c0 con 0x7fbd80153b60 2024-02-19T14:58:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.631+0000 7fbd85163640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd6406b810 0x7fbd6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.632+0000 7fbd85163640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd6406b810 0x7fbd6406dcd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000f60 tx=0x7fbd7c007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.636+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbd78054d90 con 0x7fbd80153b60 2024-02-19T14:58:21.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.870+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7fbd8014e3d0 con 0x7fbd80153b60 2024-02-19T14:58:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.914+0000 7fbd75ffb640 1 -- 172.21.15.78:0/732873228 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v378) v1 ==== 136+0+0 (secure 0 0 0) 0x7fbd78058450 con 0x7fbd80153b60 2024-02-19T14:58:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd6406b810 msgr2=0x7fbd6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd6406b810 0x7fbd6406dcd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c000f60 tx=0x7fbd7c007c10 comp rx=0 tx=0).stop 2024-02-19T14:58:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 msgr2=0x7fbd801e5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd801e5fd0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800c780 tx=0x7fbd7800cc40 comp rx=0 tx=0).stop 2024-02-19T14:58:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 shutdown_connections 2024-02-19T14:58:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd6406b810 0x7fbd6406dcd0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd80153b60 0x7fbd801e5fd0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd800c4350 0x7fbd801e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 --2- 172.21.15.78:0/732873228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd800c3200 0x7fbd8014e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:21.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 >> 172.21.15.78:0/732873228 conn(0x7fbd800c7800 msgr2=0x7fbd800c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:21.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 shutdown_connections 2024-02-19T14:58:21.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:21.916+0000 7fbd873c7640 1 -- 172.21.15.78:0/732873228 wait complete. 2024-02-19T14:58:21.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:58:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.029+0000 7f457725c640 1 Processor -- start 2024-02-19T14:58:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.029+0000 7f457725c640 1 -- start start 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.030+0000 7f457725c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 0x7f4570155780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.030+0000 7f457725c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700c6460 con 0x7f45700c6870 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.030+0000 7f457725c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700c7160 con 0x7f45701525c0 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.030+0000 7f457725c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45700c1f50 con 0x7f45700c6cd0 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.030+0000 7f456ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f45700c6870 0x7f4570155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58196/0 (socket says 172.21.15.78:58196) 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.030+0000 7f456ffff640 1 -- 172.21.15.78:0/105102627 learned_addr learned my addr 172.21.15.78:0/105102627 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f45757f9640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 0x7f4570155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392510827 0 0) 0x7f45700c6460 con 0x7f45700c6870 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4558003520 con 0x7f45700c6870 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935006001 0 0) 0x7f4558003520 con 0x7f45700c6870 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 msgr2=0x7f4570155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 0x7f4570155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 >> v1:172.21.15.150:6789/0 conn(0x7f45701525c0 legacy=0x7f457006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570158720 con 0x7f45700c6870 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4564001670 con 0x7f45700c6870 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 >> v1:172.21.15.78:6789/0 conn(0x7f45700c6870 legacy=0x7f4570155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45580039a0 0x7f4558003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 0x7f45580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4558004d00 0x7f4558005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45700c1f50 con 0x7f45580039a0 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455800ba30 con 0x7f4558004d00 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455800bbf0 con 0x7f45700c6cd0 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f4574ff8640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45580039a0 0x7f4558003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f456ffff640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 0x7f45580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f45757f9640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4558004d00 0x7f4558005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f4574ff8640 1 -- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 msgr2=0x7f45580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f4574ff8640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45700c6cd0 0x7f45580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f4574ff8640 1 -- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4558004d00 msgr2=0x7f4558005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f4574ff8640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4558004d00 0x7f4558005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f4574ff8640 1 -- 172.21.15.78:0/105102627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558003520 con 0x7f45580039a0 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.031+0000 7f45757f9640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4558004d00 0x7f4558005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.032+0000 7f4574ff8640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45580039a0 0x7f4558003d80 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f4560001860 tx=0x7f45580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1bd1e3a8746f78 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.032+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560033630 con 0x7f45580039a0 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.032+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45600337b0 con 0x7f45580039a0 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.032+0000 7f456f7fe640 1 -- 172.21.15.78:0/105102627 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560033b60 con 0x7f45580039a0 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f457725c640 1 -- 172.21.15.78:0/105102627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45580039a0 msgr2=0x7f4558003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f457725c640 1 --2- 172.21.15.78:0/105102627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45580039a0 0x7f4558003d80 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f4560001860 tx=0x7f45580057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f45757f9640 1 -- 172.21.15.78:0/105102627 reap_dead start 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f457725c640 1 -- 172.21.15.78:0/105102627 shutdown_connections 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f457725c640 1 -- 172.21.15.78:0/105102627 >> 172.21.15.78:0/105102627 conn(0x7f45700cab20 msgr2=0x7f45700caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f457725c640 1 -- 172.21.15.78:0/105102627 shutdown_connections 2024-02-19T14:58:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.033+0000 7f457725c640 1 -- 172.21.15.78:0/105102627 wait complete. 2024-02-19T14:58:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.034+0000 7f457725c640 1 Processor -- start 2024-02-19T14:58:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.034+0000 7f457725c640 1 -- start start 2024-02-19T14:58:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45580039a0 0x7f45701537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 0x7f45701411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45701416f0 0x7f4570155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701593a0 con 0x7f457014b720 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701eb750 con 0x7f45580039a0 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45701ebc80 con 0x7f45701416f0 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 0x7f45701411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 0x7f45701411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39426/0 (socket says 172.21.15.78:39426) 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 -- 172.21.15.78:0/356519207 learned_addr learned my addr 172.21.15.78:0/356519207 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f4574ff8640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45580039a0 0x7f45701537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f45757f9640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45701416f0 0x7f4570155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 -- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45701416f0 msgr2=0x7f4570155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45701416f0 0x7f4570155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 -- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45580039a0 msgr2=0x7f45701537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45580039a0 0x7f45701537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 -- 172.21.15.78:0/356519207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570141ba0 con 0x7f457014b720 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f45757f9640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45701416f0 0x7f4570155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f4574ff8640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45580039a0 0x7f45701537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456ffff640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 0x7f45701411b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f45640072f0 tx=0x7f45640077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45640010b0 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45640015e0 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45701eb9a0 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456400fba0 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.035+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4570154e00 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.036+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4570074990 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.037+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f456400fd00 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.037+0000 7f456d7fa640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f455c06b810 0x7f455c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.037+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f456408a200 con 0x7f457014b720 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.037+0000 7f4574ff8640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f455c06b810 0x7f455c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.038+0000 7f4574ff8640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f455c06b810 0x7f455c06dcd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4560036000 tx=0x7f4560031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.042+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4564054cd0 con 0x7f457014b720 2024-02-19T14:58:22.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.213+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4570150320 con 0x7f457014b720 2024-02-19T14:58:22.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.216+0000 7f456d7fa640 1 -- 172.21.15.78:0/356519207 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v57)=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4564058390 con 0x7f457014b720 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.218+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f455c06b810 msgr2=0x7f455c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.218+0000 7f457725c640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f455c06b810 0x7f455c06dcd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4560036000 tx=0x7f4560031000 comp rx=0 tx=0).stop 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.218+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 msgr2=0x7f45701411b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.218+0000 7f457725c640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 0x7f45701411b0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f45640072f0 tx=0x7f45640077b0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 shutdown_connections 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f455c06b810 0x7f455c06dcd0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f457014b720 0x7f45701411b0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45701416f0 0x7f4570155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 --2- 172.21.15.78:0/356519207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45580039a0 0x7f45701537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 >> 172.21.15.78:0/356519207 conn(0x7f45700cab20 msgr2=0x7f457014f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 shutdown_connections 2024-02-19T14:58:22.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.219+0000 7f457725c640 1 -- 172.21.15.78:0/356519207 wait complete. 2024-02-19T14:58:22.232 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:58:22.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:58:22.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:58:22.329 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:58:22.329 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:58:22.329 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:58:22.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:58:22.344 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:58:22.344 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:58:22.361 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:58:22.362 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:58:22.362 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:22.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.482+0000 7f465b5d0640 1 Processor -- start 2024-02-19T14:58:22.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.483+0000 7f465b5d0640 1 -- start start 2024-02-19T14:58:22.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.483+0000 7f465b5d0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4654152480 0x7f46541559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.483+0000 7f465b5d0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4654155ef0 con 0x7f46540c7070 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.483+0000 7f465b5d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465406dec0 con 0x7f46540c87a0 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.483+0000 7f465b5d0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f465406d520 con 0x7f4654152480 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.483+0000 7f465936c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4654152480 0x7f46541559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f465936c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4654152480 0x7f46541559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39440/0 (socket says 172.21.15.78:39440) 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f465936c640 1 -- 172.21.15.78:0/577097280 learned_addr learned my addr 172.21.15.78:0/577097280 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f4658b6b640 1 --1- 172.21.15.78:0/577097280 >> v1:172.21.15.78:6789/0 conn(0x7f46540c7070 0x7f465406ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58208/0 (socket says 172.21.15.78:58208) 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f465936c640 1 -- 172.21.15.78:0/577097280 >> v1:172.21.15.150:6789/0 conn(0x7f46540c87a0 legacy=0x7f46540c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f465936c640 1 -- 172.21.15.78:0/577097280 >> v1:172.21.15.78:6789/0 conn(0x7f46540c7070 legacy=0x7f465406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f4653fff640 1 -- 172.21.15.78:0/577097280 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515431498 0 0) 0x7f4648001110 con 0x7f46540c7070 2024-02-19T14:58:22.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.484+0000 7f465936c640 1 -- 172.21.15.78:0/577097280 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46540c75c0 con 0x7f4654152480 2024-02-19T14:58:22.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465936c640 1 --2- 172.21.15.78:0/577097280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4654152480 0x7f46541559b0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f464c009e50 tx=0x7f464c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1ce5b249e7dd4f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f4653fff640 1 -- 172.21.15.78:0/577097280 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c02cbb0 con 0x7f4654152480 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f4653fff640 1 -- 172.21.15.78:0/577097280 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c02cd10 con 0x7f4654152480 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f4653fff640 1 -- 172.21.15.78:0/577097280 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c034610 con 0x7f4654152480 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 -- 172.21.15.78:0/577097280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4654152480 msgr2=0x7f46541559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 --2- 172.21.15.78:0/577097280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4654152480 0x7f46541559b0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f464c009e50 tx=0x7f464c02c010 comp rx=0 tx=0).stop 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 -- 172.21.15.78:0/577097280 shutdown_connections 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 --2- 172.21.15.78:0/577097280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4654152480 0x7f46541559b0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 -- 172.21.15.78:0/577097280 >> 172.21.15.78:0/577097280 conn(0x7f46540ca8a0 msgr2=0x7f46540cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 -- 172.21.15.78:0/577097280 shutdown_connections 2024-02-19T14:58:22.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.485+0000 7f465b5d0640 1 -- 172.21.15.78:0/577097280 wait complete. 2024-02-19T14:58:22.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.486+0000 7f465b5d0640 1 Processor -- start 2024-02-19T14:58:22.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.486+0000 7f465b5d0640 1 -- start start 2024-02-19T14:58:22.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465b5d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 0x7f46540ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465b5d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46540c87a0 0x7f46540bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465b5d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4654152480 0x7f465414d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465b5d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46540c0d70 con 0x7f46540c7070 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465b5d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46540c1230 con 0x7f46540c87a0 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465b5d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4654159100 con 0x7f4654152480 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465936c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 0x7f46540ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465936c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 0x7f46540ac3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39446/0 (socket says 172.21.15.78:39446) 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f465936c640 1 -- 172.21.15.78:0/1462579519 learned_addr learned my addr 172.21.15.78:0/1462579519 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f4659b6d640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4654152480 0x7f465414d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.487+0000 7f4658b6b640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46540c87a0 0x7f46540bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465936c640 1 -- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4654152480 msgr2=0x7f465414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465936c640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4654152480 0x7f465414d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465936c640 1 -- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46540c87a0 msgr2=0x7f46540bfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465936c640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46540c87a0 0x7f46540bfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465936c640 1 -- 172.21.15.78:0/1462579519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c009b30 con 0x7f46540c7070 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465936c640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 0x7f46540ac3d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f464c009e50 tx=0x7f464c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c03e450 con 0x7f46540c7070 2024-02-19T14:58:22.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c03e5f0 con 0x7f46540c7070 2024-02-19T14:58:22.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46540c0f50 con 0x7f46540c7070 2024-02-19T14:58:22.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c03e9a0 con 0x7f46540c7070 2024-02-19T14:58:22.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.488+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46540ad100 con 0x7f46540c7070 2024-02-19T14:58:22.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.489+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46540c17c0 con 0x7f46540c7070 2024-02-19T14:58:22.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.490+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f464c03eb00 con 0x7f46540c7070 2024-02-19T14:58:22.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.490+0000 7f4651ffb640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f464006b810 0x7f464006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.490+0000 7f4658b6b640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f464006b810 0x7f464006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.490+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f464c0ad0a0 con 0x7f46540c7070 2024-02-19T14:58:22.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.491+0000 7f4658b6b640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f464006b810 0x7f464006dcd0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f46540ad3a0 tx=0x7f4648007a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.495+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f464c077b90 con 0x7f46540c7070 2024-02-19T14:58:22.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.700+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4654158bd0 con 0x7f46540c7070 2024-02-19T14:58:22.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.701+0000 7f4651ffb640 1 -- 172.21.15.78:0/1462579519 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+3710 (secure 0 0 0) 0x7f464c03edb0 con 0x7f46540c7070 2024-02-19T14:58:22.704 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:22.704 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":148,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7058,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1421532744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1421532744},{"type":"v1","addr":"172.21.15.150:6835","nonce":1421532744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":8134,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/1056326965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1056326965},{"type":"v1","addr":"172.21.15.150:6834","nonce":1056326965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":9042,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2349030930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2349030930},{"type":"v1","addr":"172.21.15.78:6835","nonce":2349030930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139},{"gid":9282,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1021713756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":1021713756},{"type":"v1","addr":"172.21.15.78:6837","nonce":1021713756}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148}],"filesystems":[]} 2024-02-19T14:58:22.704 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 148 2024-02-19T14:58:22.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f464006b810 msgr2=0x7f464006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f464006b810 0x7f464006dcd0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f46540ad3a0 tx=0x7f4648007a10 comp rx=0 tx=0).stop 2024-02-19T14:58:22.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 msgr2=0x7f46540ac3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 0x7f46540ac3d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f464c009e50 tx=0x7f464c004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:22.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 shutdown_connections 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f464006b810 0x7f464006dcd0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4654152480 0x7f465414d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46540c87a0 0x7f46540bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 --2- 172.21.15.78:0/1462579519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46540c7070 0x7f46540ac3d0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 >> 172.21.15.78:0/1462579519 conn(0x7f46540ca8a0 msgr2=0x7f46540af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 shutdown_connections 2024-02-19T14:58:22.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.703+0000 7f465b5d0640 1 -- 172.21.15.78:0/1462579519 wait complete. 2024-02-19T14:58:22.717 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:58:22.717 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:58:22.717 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:58:22.717 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:58:22.718 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:58:22.718 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:58:22.718 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:58:22.718 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:58:22.718 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:58:22.718 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:58:22.719 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:58:22.719 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:58:22.719 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:58:22.719 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:58:22.719 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:58:22.719 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:58:22.726 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:58:22.726 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:58:22.727 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:58:22.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:58:22.728 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:58:22.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:58:22.729 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:58:22.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:58:22.769 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:58:22.769 DEBUG:teuthology.parallel:result is None 2024-02-19T14:58:22.770 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:58:22.771 DEBUG:teuthology.parallel:result is None 2024-02-19T14:58:22.771 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:58:22.772 DEBUG:teuthology.parallel:result is None 2024-02-19T14:58:22.772 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:58:22.773 DEBUG:teuthology.parallel:result is None 2024-02-19T14:58:22.773 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:58:22.844 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:58:22.845 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:58:22.851 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:58:22.851 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:58:22.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.875+0000 7f1992de2640 1 Processor -- start 2024-02-19T14:58:22.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.875+0000 7f1992de2640 1 -- start start 2024-02-19T14:58:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1992de2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f198c0c8e00 0x7f198c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1992de2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 0x7f198c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1992de2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f198c06cd20 con 0x7f198c152700 2024-02-19T14:58:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1992de2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198c1551f0 con 0x7f198c0c8e00 2024-02-19T14:58:22.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1992de2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198c155490 con 0x7f198c06d910 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1990b7e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f198c152700 0x7f198c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58248/0 (socket says 172.21.15.78:58248) 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f1990b7e640 1 -- 172.21.15.78:0/1396229719 learned_addr learned my addr 172.21.15.78:0/1396229719 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198bfff640 1 --2- 172.21.15.78:0/1396229719 >> v2:172.21.15.150:3300/0 conn(0x7f198c0c8e00 0x7f198c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f199137f640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 0x7f198c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751259681 0 0) 0x7f198c06cd20 con 0x7f198c152700 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197c003520 con 0x7f198c152700 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409393268 0 0) 0x7f197c003520 con 0x7f198c152700 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 msgr2=0x7f198c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 0x7f198c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 >> v2:172.21.15.150:3300/0 conn(0x7f198c0c8e00 msgr2=0x7f198c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 --2- 172.21.15.78:0/1396229719 >> v2:172.21.15.150:3300/0 conn(0x7f198c0c8e00 0x7f198c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c155770 con 0x7f198c152700 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1980001670 con 0x7f198c152700 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.876+0000 7f198bfff640 1 --2- 172.21.15.78:0/1396229719 >> v2:172.21.15.150:3300/0 conn(0x7f198c0c8e00 0x7f198c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:22.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 >> v1:172.21.15.78:6789/0 conn(0x7f198c152700 legacy=0x7f198c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f197c003970 0x7f197c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 0x7f197c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c004d00 0x7f197c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1551f0 con 0x7f197c003970 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c155490 con 0x7f197c004d00 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c00b980 con 0x7f198c06d910 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f1990b7e640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f197c003970 0x7f197c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f199137f640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c004d00 0x7f197c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f198bfff640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 0x7f197c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f1990b7e640 1 -- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 msgr2=0x7f197c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f1990b7e640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c06d910 0x7f197c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f1990b7e640 1 -- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c004d00 msgr2=0x7f197c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f1990b7e640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c004d00 0x7f197c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.877+0000 7f1990b7e640 1 -- 172.21.15.78:0/1396229719 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c003520 con 0x7f197c003970 2024-02-19T14:58:22.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1990b7e640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f197c003970 0x7f197c003d90 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f198000c210 tx=0x7f197c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e634591754ce790 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198002cae0 con 0x7f197c003970 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198002cc40 con 0x7f197c003970 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f198b7fe640 1 -- 172.21.15.78:0/1396229719 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19800058d0 con 0x7f197c003970 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1992de2640 1 -- 172.21.15.78:0/1396229719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f197c003970 msgr2=0x7f197c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1992de2640 1 --2- 172.21.15.78:0/1396229719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f197c003970 0x7f197c003d90 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f198000c210 tx=0x7f197c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f199137f640 1 -- 172.21.15.78:0/1396229719 reap_dead start 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1992de2640 1 -- 172.21.15.78:0/1396229719 shutdown_connections 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1992de2640 1 -- 172.21.15.78:0/1396229719 >> 172.21.15.78:0/1396229719 conn(0x7f198c0cac90 msgr2=0x7f198c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1992de2640 1 -- 172.21.15.78:0/1396229719 shutdown_connections 2024-02-19T14:58:22.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.878+0000 7f1992de2640 1 -- 172.21.15.78:0/1396229719 wait complete. 2024-02-19T14:58:22.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.880+0000 7f1992de2640 1 Processor -- start 2024-02-19T14:58:22.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.880+0000 7f1992de2640 1 -- start start 2024-02-19T14:58:22.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c003970 0x7f198c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 0x7f198c0af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c1edf00 0x7f198c1ee390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1499e0 con 0x7f198c0ad490 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1580f0 con 0x7f197c003970 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1eea90 con 0x7f198c1edf00 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 0x7f198c0af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 0x7f198c0af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39512/0 (socket says 172.21.15.78:39512) 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 -- 172.21.15.78:0/2372542502 learned_addr learned my addr 172.21.15.78:0/2372542502 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1990b7e640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c003970 0x7f198c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f199137f640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c1edf00 0x7f198c1ee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 -- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c1edf00 msgr2=0x7f198c1ee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c1edf00 0x7f198c1ee390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 -- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c003970 msgr2=0x7f198c158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c003970 0x7f198c158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 -- 172.21.15.78:0/2372542502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c1eed30 con 0x7f198c0ad490 2024-02-19T14:58:22.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f199137f640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c1edf00 0x7f198c1ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f198bfff640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 0x7f198c0af760 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f1978011900 tx=0x7f1978011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197800fbd0 con 0x7f198c0ad490 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197800fd30 con 0x7f198c0ad490 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198c0ab0e0 con 0x7f198c0ad490 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.881+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1978015660 con 0x7f198c0ad490 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.882+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198c0ab6a0 con 0x7f198c0ad490 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.882+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198c14ff20 con 0x7f198c0ad490 2024-02-19T14:58:22.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.883+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f19780157c0 con 0x7f198c0ad490 2024-02-19T14:58:22.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.883+0000 7f19897fa640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197006b840 0x7f197006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:22.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.883+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f197808a970 con 0x7f198c0ad490 2024-02-19T14:58:22.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.883+0000 7f1990b7e640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197006b840 0x7f197006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:22.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.884+0000 7f1990b7e640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197006b840 0x7f197006dd00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1980034000 tx=0x7f1980007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:22.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:22.888+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1978055440 con 0x7f198c0ad490 2024-02-19T14:58:23.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.074+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f198c0ab890 con 0x7f198c0ad490 2024-02-19T14:58:23.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.940+0000 7f19897fa640 1 -- 172.21.15.78:0/2372542502 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v170) v1 ==== 97+0+0 (secure 0 0 0) 0x7f197800fea0 con 0x7f198c0ad490 2024-02-19T14:58:23.943 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197006b840 msgr2=0x7f197006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197006b840 0x7f197006dd00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1980034000 tx=0x7f1980007000 comp rx=0 tx=0).stop 2024-02-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 msgr2=0x7f198c0af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 0x7f198c0af760 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f1978011900 tx=0x7f1978011dc0 comp rx=0 tx=0).stop 2024-02-19T14:58:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 shutdown_connections 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197006b840 0x7f197006dd00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f198c0ad490 0x7f198c0af760 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f198c1edf00 0x7f198c1ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 --2- 172.21.15.78:0/2372542502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f197c003970 0x7f198c158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 >> 172.21.15.78:0/2372542502 conn(0x7f198c0cac90 msgr2=0x7f198c0c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 shutdown_connections 2024-02-19T14:58:23.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:23.942+0000 7f1992de2640 1 -- 172.21.15.78:0/2372542502 wait complete. 2024-02-19T14:58:23.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:58:24.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.057+0000 7ff0358b5640 1 Processor -- start 2024-02-19T14:58:24.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.057+0000 7ff0358b5640 1 -- start start 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff0358b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff030152700 0x7ff0300c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff0358b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 0x7ff03015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff0358b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0301551f0 con 0x7ff0300c5730 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff0358b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff030155490 con 0x7ff030152700 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff0358b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff030155770 con 0x7ff0300c8e00 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 0x7ff03015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff030152700 0x7ff0300c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 0x7ff03015f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50330/0 (socket says 172.21.15.78:50330) 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 -- 172.21.15.78:0/2210062251 learned_addr learned my addr 172.21.15.78:0/2210062251 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff027fff640 1 --1- 172.21.15.78:0/2210062251 >> v1:172.21.15.150:6789/0 conn(0x7ff0300c5730 0x7ff03006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:33976/0 (socket says 172.21.15.78:33976) 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 -- 172.21.15.78:0/2210062251 >> v2:172.21.15.150:3300/0 conn(0x7ff030152700 msgr2=0x7ff0300c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 --2- 172.21.15.78:0/2210062251 >> v2:172.21.15.150:3300/0 conn(0x7ff030152700 0x7ff0300c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 -- 172.21.15.78:0/2210062251 >> v1:172.21.15.150:6789/0 conn(0x7ff0300c5730 legacy=0x7ff03006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02f7fe640 1 -- 172.21.15.78:0/2210062251 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff030155a50 con 0x7ff0300c8e00 2024-02-19T14:58:24.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.058+0000 7ff02effd640 1 --2- 172.21.15.78:0/2210062251 >> v2:172.21.15.150:3300/0 conn(0x7ff030152700 0x7ff0300c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.059+0000 7ff02f7fe640 1 --2- 172.21.15.78:0/2210062251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 0x7ff03015f980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff028009e50 tx=0x7ff02802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfe1179dc62a1365 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.059+0000 7ff02e7fc640 1 -- 172.21.15.78:0/2210062251 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02802cb00 con 0x7ff0300c8e00 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.059+0000 7ff02e7fc640 1 -- 172.21.15.78:0/2210062251 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02802cc60 con 0x7ff0300c8e00 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.059+0000 7ff02e7fc640 1 -- 172.21.15.78:0/2210062251 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0280343f0 con 0x7ff0300c8e00 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 -- 172.21.15.78:0/2210062251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 msgr2=0x7ff03015f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 --2- 172.21.15.78:0/2210062251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 0x7ff03015f980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff028009e50 tx=0x7ff02802bf60 comp rx=0 tx=0).stop 2024-02-19T14:58:24.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 -- 172.21.15.78:0/2210062251 shutdown_connections 2024-02-19T14:58:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 --2- 172.21.15.78:0/2210062251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c8e00 0x7ff03015f980 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 --2- 172.21.15.78:0/2210062251 >> v2:172.21.15.150:3300/0 conn(0x7ff030152700 0x7ff0300c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 -- 172.21.15.78:0/2210062251 >> 172.21.15.78:0/2210062251 conn(0x7ff0300cac90 msgr2=0x7ff0300cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 -- 172.21.15.78:0/2210062251 shutdown_connections 2024-02-19T14:58:24.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.060+0000 7ff0358b5640 1 -- 172.21.15.78:0/2210062251 wait complete. 2024-02-19T14:58:24.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.061+0000 7ff0358b5640 1 Processor -- start 2024-02-19T14:58:24.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.061+0000 7ff0358b5640 1 -- start start 2024-02-19T14:58:24.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff0358b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c5730 0x7ff0301e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff0358b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 0x7ff0301e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff0358b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030152700 0x7ff0300ab150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff0358b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0301518f0 con 0x7ff0300c8e00 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff0358b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300aac70 con 0x7ff030152700 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff0358b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300c19e0 con 0x7ff0300c5730 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 0x7ff0301e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 0x7ff0301e7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39518/0 (socket says 172.21.15.78:39518) 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 -- 172.21.15.78:0/3611814465 learned_addr learned my addr 172.21.15.78:0/3611814465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff02effd640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c5730 0x7ff0301e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff02f7fe640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030152700 0x7ff0300ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 -- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c5730 msgr2=0x7ff0301e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c5730 0x7ff0301e79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 -- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030152700 msgr2=0x7ff0300ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030152700 0x7ff0300ab150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.062+0000 7ff027fff640 1 -- 172.21.15.78:0/3611814465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff020009400 con 0x7ff0300c8e00 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.063+0000 7ff027fff640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 0x7ff0301e7ef0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c400 tx=0x7ff01800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.063+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0180074e0 con 0x7ff0300c8e00 2024-02-19T14:58:24.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.063+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018007640 con 0x7ff0300c8e00 2024-02-19T14:58:24.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.063+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff028009b30 con 0x7ff0300c8e00 2024-02-19T14:58:24.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.063+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018004a00 con 0x7ff0300c8e00 2024-02-19T14:58:24.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.063+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0300c1fa0 con 0x7ff0300c8e00 2024-02-19T14:58:24.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.064+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff030074990 con 0x7ff0300c8e00 2024-02-19T14:58:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.064+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff018004b60 con 0x7ff0300c8e00 2024-02-19T14:58:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.064+0000 7ff02cff9640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01006b810 0x7ff01006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.065+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff0180898f0 con 0x7ff0300c8e00 2024-02-19T14:58:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.065+0000 7ff02effd640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01006b810 0x7ff01006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.065+0000 7ff02effd640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01006b810 0x7ff01006dcd0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff020009c30 tx=0x7ff020009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.069+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff018054590 con 0x7ff0300c8e00 2024-02-19T14:58:24.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.307+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff0300ca560 con 0x7ff0300c8e00 2024-02-19T14:58:24.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.308+0000 7ff02cff9640 1 -- 172.21.15.78:0/3611814465 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4724 (secure 0 0 0) 0x7ff018057c50 con 0x7ff0300c8e00 2024-02-19T14:58:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:58:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:58:24.317 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:58:24.319 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:58:24.320 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:58:24.321 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:58:24.322 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:58:24.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.309+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01006b810 msgr2=0x7ff01006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.309+0000 7ff0358b5640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01006b810 0x7ff01006dcd0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff020009c30 tx=0x7ff020009190 comp rx=0 tx=0).stop 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 msgr2=0x7ff0301e7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 0x7ff0301e7ef0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c400 tx=0x7ff01800c8c0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 shutdown_connections 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff01006b810 0x7ff01006dcd0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0300c8e00 0x7ff0301e7ef0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030152700 0x7ff0300ab150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 --2- 172.21.15.78:0/3611814465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0300c5730 0x7ff0301e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 >> 172.21.15.78:0/3611814465 conn(0x7ff0300cac90 msgr2=0x7ff030150ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 shutdown_connections 2024-02-19T14:58:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.310+0000 7ff0358b5640 1 -- 172.21.15.78:0/3611814465 wait complete. 2024-02-19T14:58:24.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:24.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.421+0000 7f3fd2f0d640 1 Processor -- start 2024-02-19T14:58:24.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd2f0d640 1 -- start start 2024-02-19T14:58:24.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd2f0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3fcc152700 0x7f3fcc0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd2f0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3fcc0c8e00 0x7f3fcc15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd2f0d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc1551f0 con 0x7f3fcc0c5730 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd2f0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fcc155490 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd2f0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fcc155770 con 0x7f3fcc152700 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd14aa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3fcc0c8e00 0x7f3fcc15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd14aa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3fcc0c8e00 0x7f3fcc15f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39532/0 (socket says 172.21.15.78:39532) 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.422+0000 7f3fd14aa640 1 -- 172.21.15.78:0/1776582914 learned_addr learned my addr 172.21.15.78:0/1776582914 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fd0ca9640 1 --2- 172.21.15.78:0/1776582914 >> v2:172.21.15.150:3300/0 conn(0x7f3fcc152700 0x7f3fcc0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fd14aa640 1 -- 172.21.15.78:0/1776582914 >> v2:172.21.15.150:3300/0 conn(0x7f3fcc152700 msgr2=0x7f3fcc0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fd14aa640 1 --2- 172.21.15.78:0/1776582914 >> v2:172.21.15.150:3300/0 conn(0x7f3fcc152700 0x7f3fcc0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fd14aa640 1 -- 172.21.15.78:0/1776582914 >> v1:172.21.15.150:6789/0 conn(0x7f3fcc0c5730 legacy=0x7f3fcc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fd14aa640 1 -- 172.21.15.78:0/1776582914 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc155a50 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fd14aa640 1 --2- 172.21.15.78:0/1776582914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc15f980 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc08bad0 tx=0x7f3fc402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc53741ea48f42ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.423+0000 7f3fcb7fe640 1 -- 172.21.15.78:0/1776582914 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc402c9f0 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fcb7fe640 1 -- 172.21.15.78:0/1776582914 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc402cb50 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fcb7fe640 1 -- 172.21.15.78:0/1776582914 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc40343f0 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1776582914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 msgr2=0x7f3fcc15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1776582914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc15f980 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc08bad0 tx=0x7f3fc402be50 comp rx=0 tx=0).stop 2024-02-19T14:58:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1776582914 shutdown_connections 2024-02-19T14:58:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1776582914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc15f980 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1776582914 >> v2:172.21.15.150:3300/0 conn(0x7f3fcc152700 0x7f3fcc0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1776582914 >> 172.21.15.78:0/1776582914 conn(0x7f3fcc0cac90 msgr2=0x7f3fcc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1776582914 shutdown_connections 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.424+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1776582914 wait complete. 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.425+0000 7f3fd2f0d640 1 Processor -- start 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.425+0000 7f3fd2f0d640 1 -- start start 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fd2f0d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fcc0c5730 0x7f3fcc1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fd2f0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc0c2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fd2f0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fcc152700 0x7f3fcc0c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fd2f0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c4270 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fd2f0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc158260 con 0x7f3fcc152700 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fd2f0d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c3410 con 0x7f3fcc0c5730 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc0c2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc0c2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39536/0 (socket says 172.21.15.78:39536) 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 -- 172.21.15.78:0/1873346722 learned_addr learned my addr 172.21.15.78:0/1873346722 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:24.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 -- 172.21.15.78:0/1873346722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fcc0c5730 msgr2=0x7f3fcc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fcc0c5730 0x7f3fcc1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 -- 172.21.15.78:0/1873346722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fcc152700 msgr2=0x7f3fcc0c2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fcc152700 0x7f3fcc0c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.426+0000 7f3fcbfff640 1 -- 172.21.15.78:0/1873346722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc4009b30 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.427+0000 7f3fcbfff640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc0c2990 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c430 tx=0x7f3fb800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.427+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011790 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.427+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb80118f0 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.427+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011ce0 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.427+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc1eb770 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.427+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc158470 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.428+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc074990 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.428+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3fb801c460 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.428+0000 7f3fc97fa640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b840 0x7f3fb006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.429+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3fb8089fd0 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.429+0000 7f3fd0ca9640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b840 0x7f3fb006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.429+0000 7f3fd0ca9640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b840 0x7f3fb006dd00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0006fd0 tx=0x7f3fc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.433+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3fb8054bb0 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.605+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fcc0c8770 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.605+0000 7f3fc97fa640 1 -- 172.21.15.78:0/1873346722 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f3fb8058270 con 0x7f3fcc0c8e00 2024-02-19T14:58:24.608 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:24.608 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":170,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:23.937572+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b840 msgr2=0x7f3fb006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b840 0x7f3fb006dd00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0006fd0 tx=0x7f3fc0008040 comp rx=0 tx=0).stop 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 msgr2=0x7f3fcc0c2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc0c2990 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c430 tx=0x7f3fb800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 shutdown_connections 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b840 0x7f3fb006dd00 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fcc0c8e00 0x7f3fcc0c2990 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fcc152700 0x7f3fcc0c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.607+0000 7f3fd2f0d640 1 --2- 172.21.15.78:0/1873346722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fcc0c5730 0x7f3fcc1566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.608+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 >> 172.21.15.78:0/1873346722 conn(0x7f3fcc0cac90 msgr2=0x7f3fcc14c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.608+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 shutdown_connections 2024-02-19T14:58:24.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.608+0000 7f3fd2f0d640 1 -- 172.21.15.78:0/1873346722 wait complete. 2024-02-19T14:58:24.621 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:58:24.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:58:24.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 Processor -- start 2024-02-19T14:58:24.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 -- start start 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e3c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e3c0c87a0 0x7f2e3c0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c155ef0 con 0x7f2e3c152480 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c06dec0 con 0x7f2e3c0c87a0 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.722+0000 7f2e40864640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c06d520 con 0x7f2e3c0c7070 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2e3c152480 0x7f2e3c06ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58256/0 (socket says 172.21.15.78:58256) 2024-02-19T14:58:24.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3a59c640 1 -- 172.21.15.78:0/95396258 learned_addr learned my addr 172.21.15.78:0/95396258 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759781612 0 0) 0x7f2e2c001110 con 0x7f2e3c152480 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24003520 con 0x7f2e3c152480 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3ad9d640 1 --2- 172.21.15.78:0/95396258 >> v2:172.21.15.150:3300/0 conn(0x7f2e3c0c87a0 0x7f2e3c0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e3c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462154385 0 0) 0x7f2e24003520 con 0x7f2e3c152480 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 msgr2=0x7f2e3c1559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e3c1559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 >> v2:172.21.15.150:3300/0 conn(0x7f2e3c0c87a0 msgr2=0x7f2e3c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 --2- 172.21.15.78:0/95396258 >> v2:172.21.15.150:3300/0 conn(0x7f2e3c0c87a0 0x7f2e3c0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c0c75c0 con 0x7f2e3c152480 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2e2c001800 con 0x7f2e3c152480 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 >> v1:172.21.15.78:6789/0 conn(0x7f2e3c152480 legacy=0x7f2e3c06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e24003970 0x7f2e24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e240047c0 0x7f2e24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c06dec0 con 0x7f2e240047c0 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c06d520 con 0x7f2e24003970 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2400b980 con 0x7f2e3c0c7070 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e240047c0 0x7f2e24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3a59c640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e24003970 0x7f2e24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.723+0000 7f2e3ad9d640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e39d9b640 1 -- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 msgr2=0x7f2e24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e39d9b640 1 -- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e24003970 msgr2=0x7f2e24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e24003970 0x7f2e24003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e39d9b640 1 -- 172.21.15.78:0/95396258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24003520 con 0x7f2e240047c0 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e3ad9d640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c0c7070 0x7f2e24005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:24.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e240047c0 0x7f2e24004c40 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009a90 tx=0x7f2e240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373fa17d8808a2bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3002c6b0 con 0x7f2e240047c0 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3002c810 con 0x7f2e240047c0 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.724+0000 7f2e3959a640 1 -- 172.21.15.78:0/95396258 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3002cbc0 con 0x7f2e240047c0 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e40864640 1 -- 172.21.15.78:0/95396258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e240047c0 msgr2=0x7f2e24004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e40864640 1 --2- 172.21.15.78:0/95396258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e240047c0 0x7f2e24004c40 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009a90 tx=0x7f2e240057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e3ad9d640 1 -- 172.21.15.78:0/95396258 reap_dead start 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e40864640 1 -- 172.21.15.78:0/95396258 shutdown_connections 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e40864640 1 -- 172.21.15.78:0/95396258 >> 172.21.15.78:0/95396258 conn(0x7f2e3c0ca8a0 msgr2=0x7f2e3c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e40864640 1 -- 172.21.15.78:0/95396258 shutdown_connections 2024-02-19T14:58:24.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.725+0000 7f2e40864640 1 -- 172.21.15.78:0/95396258 wait complete. 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.726+0000 7f2e40864640 1 Processor -- start 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.726+0000 7f2e40864640 1 -- start start 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e40864640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e240047c0 0x7f2e3c1edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e40864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 0x7f2e3c1463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e40864640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c158780 0x7f2e3c158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e40864640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c14a660 con 0x7f2e3c145f20 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e40864640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c1562d0 con 0x7f2e240047c0 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e40864640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c157630 con 0x7f2e3c158780 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 0x7f2e3c1463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 0x7f2e3c1463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39544/0 (socket says 172.21.15.78:39544) 2024-02-19T14:58:24.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 -- 172.21.15.78:0/84210164 learned_addr learned my addr 172.21.15.78:0/84210164 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e3a59c640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e240047c0 0x7f2e3c1edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e3ad9d640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c158780 0x7f2e3c158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 -- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c158780 msgr2=0x7f2e3c158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c158780 0x7f2e3c158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 -- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e240047c0 msgr2=0x7f2e3c1edeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e240047c0 0x7f2e3c1edeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.727+0000 7f2e39d9b640 1 -- 172.21.15.78:0/84210164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c156760 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.728+0000 7f2e39d9b640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 0x7f2e3c1463a0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009bb0 tx=0x7f2e30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.728+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e30034a80 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.728+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e30034c00 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.728+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3003e440 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.728+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c1468e0 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.728+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c14a380 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.729+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c156480 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.729+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2e3003e5a0 con 0x7f2e3c145f20 2024-02-19T14:58:24.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.730+0000 7f2e2affd640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e1c06b840 0x7f2e1c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:24.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.730+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2e300abbb0 con 0x7f2e3c145f20 2024-02-19T14:58:24.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.730+0000 7f2e3a59c640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e1c06b840 0x7f2e1c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:24.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.730+0000 7f2e3a59c640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e1c06b840 0x7f2e1c06dd00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c146fe0 tx=0x7f2e2c0079a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:24.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.734+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2e3c156480 con 0x7f2e3c145f20 2024-02-19T14:58:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.906+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e3c156440 con 0x7f2e3c145f20 2024-02-19T14:58:24.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.906+0000 7f2e2affd640 1 -- 172.21.15.78:0/84210164 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v170) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e300420a0 con 0x7f2e3c145f20 2024-02-19T14:58:24.909 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:58:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e1c06b840 msgr2=0x7f2e1c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e1c06b840 0x7f2e1c06dd00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c146fe0 tx=0x7f2e2c0079a0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 msgr2=0x7f2e3c1463a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:24.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 0x7f2e3c1463a0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f2e30009bb0 tx=0x7f2e30004730 comp rx=0 tx=0).stop 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 shutdown_connections 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e1c06b840 0x7f2e1c06dd00 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e3c145f20 0x7f2e3c1463a0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e3c158780 0x7f2e3c158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 --2- 172.21.15.78:0/84210164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e240047c0 0x7f2e3c1edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 >> 172.21.15.78:0/84210164 conn(0x7f2e3c0ca8a0 msgr2=0x7f2e3c0c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 shutdown_connections 2024-02-19T14:58:24.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:24.908+0000 7f2e40864640 1 -- 172.21.15.78:0/84210164 wait complete. 2024-02-19T14:58:24.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:58:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.019+0000 7f978eceb640 1 Processor -- start 2024-02-19T14:58:25.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.019+0000 7f978eceb640 1 -- start start 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978eceb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97880c25e0 0x7f97881526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978eceb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9788072250 con 0x7f97880c5be0 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978eceb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9788072480 con 0x7f97880c29c0 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978eceb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9788152c90 con 0x7f97880c25e0 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978d288640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f97880c5be0 0x7f978806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58258/0 (socket says 172.21.15.78:58258) 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978d288640 1 -- 172.21.15.78:0/1899776823 learned_addr learned my addr 172.21.15.78:0/1899776823 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978ca87640 1 --2- 172.21.15.78:0/1899776823 >> v2:172.21.15.150:3300/0 conn(0x7f97880c25e0 0x7f97881526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977884928 0 0) 0x7f9788072250 con 0x7f97880c5be0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9778003520 con 0x7f97880c5be0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698770232 0 0) 0x7f9788072480 con 0x7f97880c29c0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9788072250 con 0x7f97880c29c0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1843495310 0 0) 0x7f9778003520 con 0x7f97880c5be0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 >> v2:172.21.15.150:3300/0 conn(0x7f97880c25e0 msgr2=0x7f97881526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 --2- 172.21.15.78:0/1899776823 >> v2:172.21.15.150:3300/0 conn(0x7f97880c25e0 0x7f97881526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 >> v1:172.21.15.150:6789/0 conn(0x7f97880c29c0 legacy=0x7f97880c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97880c4b90 con 0x7f97880c5be0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f97800019f0 con 0x7f97880c5be0 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 >> v1:172.21.15.78:6789/0 conn(0x7f97880c5be0 legacy=0x7f978806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778003b50 0x7f9778003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97780047c0 0x7f9778004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97780051d0 0x7f977800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9788152c90 con 0x7f97780051d0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977800c100 con 0x7f9778003b50 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977800c2c0 con 0x7f97780047c0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978d288640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97780051d0 0x7f977800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.020+0000 7f978ca87640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778003b50 0x7f9778003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f9787fff640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97780047c0 0x7f9778004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 -- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97780047c0 msgr2=0x7f9778004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97780047c0 0x7f9778004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 -- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778003b50 msgr2=0x7f9778003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778003b50 0x7f9778003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 -- 172.21.15.78:0/1899776823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778003520 con 0x7f97780051d0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 -- 172.21.15.78:0/1899776823 reap_dead start 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.021+0000 7f978d288640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97780051d0 0x7f977800b980 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f978000a710 tx=0x7f97780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac4353de3f6dd80 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:25.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780034630 con 0x7f97780051d0 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978002f070 con 0x7f97780051d0 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f97877fe640 1 -- 172.21.15.78:0/1899776823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780034a10 con 0x7f97780051d0 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 -- 172.21.15.78:0/1899776823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97780051d0 msgr2=0x7f977800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97780051d0 0x7f977800b980 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f978000a710 tx=0x7f97780057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 -- 172.21.15.78:0/1899776823 shutdown_connections 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 --2- 172.21.15.78:0/1899776823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97780051d0 0x7f977800b980 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 -- 172.21.15.78:0/1899776823 >> 172.21.15.78:0/1899776823 conn(0x7f978806b270 msgr2=0x7f978806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 -- 172.21.15.78:0/1899776823 shutdown_connections 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.022+0000 7f978eceb640 1 -- 172.21.15.78:0/1899776823 wait complete. 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.023+0000 7f978eceb640 1 Processor -- start 2024-02-19T14:58:25.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.023+0000 7f978eceb640 1 -- start start 2024-02-19T14:58:25.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f978eceb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780051d0 0x7f9788155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f978eceb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 0x7f97880a7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f978eceb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97880aba30 0x7f97880a8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f978eceb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97881eb620 con 0x7f978806d310 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f978eceb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97881ec1a0 con 0x7f97780051d0 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f978eceb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97880a8880 con 0x7f97880aba30 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 0x7f97880a7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 0x7f97880a7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39558/0 (socket says 172.21.15.78:39558) 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 -- 172.21.15.78:0/2522768542 learned_addr learned my addr 172.21.15.78:0/2522768542 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 -- 172.21.15.78:0/2522768542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97880aba30 msgr2=0x7f97880a8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97880aba30 0x7f97880a8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 -- 172.21.15.78:0/2522768542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780051d0 msgr2=0x7f9788155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780051d0 0x7f9788155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.024+0000 7f9787fff640 1 -- 172.21.15.78:0/2522768542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97881eb870 con 0x7f978806d310 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.025+0000 7f9787fff640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 0x7f97880a7e00 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f977400f800 tx=0x7f977400fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:25.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.025+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977400feb0 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.025+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9774005320 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.025+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97881ebb50 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.025+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740072b0 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.025+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9788155c80 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.026+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9788074990 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.026+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9774007410 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.027+0000 7f97857fa640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.027+0000 7f978ca87640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.027+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9774089970 con 0x7f978806d310 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.027+0000 7f978ca87640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f977c009a40 tx=0x7f977c009320 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:25.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.031+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f97740545e0 con 0x7f978806d310 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.202+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f97881ec2e0 con 0x7f978806d310 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.202+0000 7f97857fa640 1 -- 172.21.15.78:0/2522768542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v170) v1 ==== 84+0+65 (secure 0 0 0) 0x7f977401a090 con 0x7f978806d310 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:58:25.205 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:58:25.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:58:25.206 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 msgr2=0x7f976c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f977c009a40 tx=0x7f977c009320 comp rx=0 tx=0).stop 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 msgr2=0x7f97880a7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 0x7f97880a7e00 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f977400f800 tx=0x7f977400fcc0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 shutdown_connections 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978806d310 0x7f97880a7e00 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97880aba30 0x7f97880a8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 --2- 172.21.15.78:0/2522768542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780051d0 0x7f9788155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 >> 172.21.15.78:0/2522768542 conn(0x7f978806b270 msgr2=0x7f97880c5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:25.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 shutdown_connections 2024-02-19T14:58:25.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.204+0000 7f978eceb640 1 -- 172.21.15.78:0/2522768542 wait complete. 2024-02-19T14:58:25.218 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:58:25.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:58:25.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 Processor -- start 2024-02-19T14:58:25.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 -- start start 2024-02-19T14:58:25.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8f00c7f10 0x7fc8f01550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc8f00bf0c0 0x7fc8f0155610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f006fa90 con 0x7fc8f01527d0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f01592b0 con 0x7fc8f00bf0c0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.319+0000 7fc8f80b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f0071280 con 0x7fc8f00c7f10 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f6652640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc8f00bf0c0 0x7fc8f0155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f5e51640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc8f01527d0 0x7fc8f00c4e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58266/0 (socket says 172.21.15.78:58266) 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f5e51640 1 -- 172.21.15.78:0/3401954242 learned_addr learned my addr 172.21.15.78:0/3401954242 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f5650640 1 --2- 172.21.15.78:0/3401954242 >> v2:172.21.15.150:3300/0 conn(0x7fc8f00c7f10 0x7fc8f01550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f6652640 1 -- 172.21.15.78:0/3401954242 >> v2:172.21.15.150:3300/0 conn(0x7fc8f00c7f10 msgr2=0x7fc8f01550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f6652640 1 --2- 172.21.15.78:0/3401954242 >> v2:172.21.15.150:3300/0 conn(0x7fc8f00c7f10 0x7fc8f01550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f6652640 1 -- 172.21.15.78:0/3401954242 >> v1:172.21.15.78:6789/0 conn(0x7fc8f01527d0 legacy=0x7fc8f00c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f4e4f640 1 -- 172.21.15.78:0/3401954242 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652790938 0 0) 0x7fc8f006fa90 con 0x7fc8f01527d0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.320+0000 7fc8f6652640 1 -- 172.21.15.78:0/3401954242 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f006f4d0 con 0x7fc8f00bf0c0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f6652640 1 --2- 172.21.15.78:0/3401954242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00bf0c0 0x7fc8f0155610 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0072650 tx=0x7fc8e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ad7e66a3d8c72a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:25.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f4e4f640 1 -- 172.21.15.78:0/3401954242 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e802cc80 con 0x7fc8f00bf0c0 2024-02-19T14:58:25.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f4e4f640 1 -- 172.21.15.78:0/3401954242 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e802cde0 con 0x7fc8f00bf0c0 2024-02-19T14:58:25.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f4e4f640 1 -- 172.21.15.78:0/3401954242 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8034660 con 0x7fc8f00bf0c0 2024-02-19T14:58:25.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 -- 172.21.15.78:0/3401954242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00bf0c0 msgr2=0x7fc8f0155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/3401954242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00bf0c0 0x7fc8f0155610 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0072650 tx=0x7fc8e802bed0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 -- 172.21.15.78:0/3401954242 shutdown_connections 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/3401954242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00bf0c0 0x7fc8f0155610 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/3401954242 >> v2:172.21.15.150:3300/0 conn(0x7fc8f00c7f10 0x7fc8f01550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 -- 172.21.15.78:0/3401954242 >> 172.21.15.78:0/3401954242 conn(0x7fc8f00aa370 msgr2=0x7fc8f00aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 -- 172.21.15.78:0/3401954242 shutdown_connections 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.321+0000 7fc8f80b5640 1 -- 172.21.15.78:0/3401954242 wait complete. 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.322+0000 7fc8f80b5640 1 Processor -- start 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 -- start start 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f00bf0c0 0x7fc8f00bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 0x7fc8f01ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f01527d0 0x7fc8f0150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01eec20 con 0x7fc8f00c7f10 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01ea760 con 0x7fc8f01527d0 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f80b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00bff40 con 0x7fc8f00bf0c0 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f5650640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 0x7fc8f01ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f5650640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 0x7fc8f01ee6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39570/0 (socket says 172.21.15.78:39570) 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.323+0000 7fc8f5650640 1 -- 172.21.15.78:0/159617021 learned_addr learned my addr 172.21.15.78:0/159617021 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f5650640 1 -- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f00bf0c0 msgr2=0x7fc8f00bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f5650640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f00bf0c0 0x7fc8f00bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f5650640 1 -- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f01527d0 msgr2=0x7fc8f0150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f6652640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f01527d0 0x7fc8f0150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f5650640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f01527d0 0x7fc8f0150d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f5650640 1 -- 172.21.15.78:0/159617021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e8009a70 con 0x7fc8f00c7f10 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f5650640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 0x7fc8f01ee6e0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00e930 tx=0x7fc8ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec009690 con 0x7fc8f00c7f10 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec00ca00 con 0x7fc8f00c7f10 2024-02-19T14:58:25.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f01eaa20 con 0x7fc8f00c7f10 2024-02-19T14:58:25.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec00cdb0 con 0x7fc8f00c7f10 2024-02-19T14:58:25.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.324+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f01eafe0 con 0x7fc8f00c7f10 2024-02-19T14:58:25.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.325+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f01eb1d0 con 0x7fc8f00c7f10 2024-02-19T14:58:25.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.325+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc8ec013470 con 0x7fc8f00c7f10 2024-02-19T14:58:25.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.326+0000 7fc8e67fc640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8d406b810 0x7fc8d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:25.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.326+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc8ec08a2c0 con 0x7fc8f00c7f10 2024-02-19T14:58:25.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.326+0000 7fc8f5e51640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8d406b810 0x7fc8d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:25.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.326+0000 7fc8f5e51640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8d406b810 0x7fc8d406dcd0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0007fd0 tx=0x7fc8e0007960 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:25.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.330+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc8ec054ff0 con 0x7fc8f00c7f10 2024-02-19T14:58:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.502+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc8f0074990 con 0x7fc8f00c7f10 2024-02-19T14:58:25.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.952+0000 7fc8e67fc640 1 -- 172.21.15.78:0/159617021 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v171) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc8ec0586b0 con 0x7fc8f00c7f10 2024-02-19T14:58:25.955 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:58:25.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8d406b810 msgr2=0x7fc8d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8d406b810 0x7fc8d406dcd0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0007fd0 tx=0x7fc8e0007960 comp rx=0 tx=0).stop 2024-02-19T14:58:25.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 msgr2=0x7fc8f01ee6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 0x7fc8f01ee6e0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00e930 tx=0x7fc8ec00edf0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 shutdown_connections 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8d406b810 0x7fc8d406dcd0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8f00c7f10 0x7fc8f01ee6e0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f01527d0 0x7fc8f0150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.954+0000 7fc8f80b5640 1 --2- 172.21.15.78:0/159617021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f00bf0c0 0x7fc8f00bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.955+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 >> 172.21.15.78:0/159617021 conn(0x7fc8f00aa370 msgr2=0x7fc8f00aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.955+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 shutdown_connections 2024-02-19T14:58:25.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:25.955+0000 7fc8f80b5640 1 -- 172.21.15.78:0/159617021 wait complete. 2024-02-19T14:58:25.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:58:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.077+0000 7fd40ddd0640 1 Processor -- start 2024-02-19T14:58:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.077+0000 7fd40ddd0640 1 -- start start 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd40ddd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd4080c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd40ddd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4081588e0 0x7fd408158d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd40ddd0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080c7270 con 0x7fd408155f70 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd40ddd0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd408156350 con 0x7fd4081588e0 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd40ddd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080c8300 con 0x7fd4080c67e0 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd407fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4081588e0 0x7fd408158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd4080c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd4077fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd408155f70 0x7fd4080c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:33988/0 (socket says 172.21.15.78:33988) 2024-02-19T14:58:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.078+0000 7fd4077fe640 1 -- 172.21.15.78:0/4230984534 learned_addr learned my addr 172.21.15.78:0/4230984534 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd4067fc640 1 -- 172.21.15.78:0/4230984534 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168632110 0 0) 0x7fd4080c7270 con 0x7fd408155f70 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd4067fc640 1 -- 172.21.15.78:0/4230984534 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0003520 con 0x7fd408155f70 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd407fff640 1 -- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 msgr2=0x7fd4080c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd407fff640 1 --2- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd4080c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd407fff640 1 -- 172.21.15.78:0/4230984534 >> v1:172.21.15.150:6789/0 conn(0x7fd408155f70 legacy=0x7fd4080c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd407fff640 1 -- 172.21.15.78:0/4230984534 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4080c4d70 con 0x7fd4081588e0 2024-02-19T14:58:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.079+0000 7fd406ffd640 1 --2- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd4080c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd407fff640 1 --2- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd408158d50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd400009d90 tx=0x7fd40002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d62518f1e280781a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd4067fc640 1 -- 172.21.15.78:0/4230984534 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40002cb30 con 0x7fd4081588e0 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd4067fc640 1 -- 172.21.15.78:0/4230984534 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40002cc90 con 0x7fd4081588e0 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd4067fc640 1 -- 172.21.15.78:0/4230984534 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400034440 con 0x7fd4081588e0 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 -- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 msgr2=0x7fd408158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd408158d50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd400009d90 tx=0x7fd40002bd80 comp rx=0 tx=0).stop 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 -- 172.21.15.78:0/4230984534 shutdown_connections 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd4080c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/4230984534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd408158d50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 -- 172.21.15.78:0/4230984534 >> 172.21.15.78:0/4230984534 conn(0x7fd40806b6a0 msgr2=0x7fd4080aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 -- 172.21.15.78:0/4230984534 shutdown_connections 2024-02-19T14:58:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.080+0000 7fd40ddd0640 1 -- 172.21.15.78:0/4230984534 wait complete. 2024-02-19T14:58:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.081+0000 7fd40ddd0640 1 Processor -- start 2024-02-19T14:58:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 -- start start 2024-02-19T14:58:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd408151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 0x7fd40806f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd40806fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081eba50 con 0x7fd408155f70 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081eae40 con 0x7fd4081588e0 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd40ddd0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081eb120 con 0x7fd4080c67e0 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 0x7fd40806f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 0x7fd40806f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39582/0 (socket says 172.21.15.78:39582) 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.082+0000 7fd406ffd640 1 -- 172.21.15.78:0/2753383524 learned_addr learned my addr 172.21.15.78:0/2753383524 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd407fff640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd40806fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd4077fe640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd408151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd406ffd640 1 -- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 msgr2=0x7fd408151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd406ffd640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd408151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd406ffd640 1 -- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 msgr2=0x7fd40806fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd406ffd640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd40806fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd406ffd640 1 -- 172.21.15.78:0/2753383524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc009400 con 0x7fd408155f70 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd407fff640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd40806fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd406ffd640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 0x7fd40806f5c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00e960 tx=0x7fd3fc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc00ca00 con 0x7fd408155f70 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc00cb60 con 0x7fd408155f70 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd400009a70 con 0x7fd408155f70 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc013470 con 0x7fd408155f70 2024-02-19T14:58:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.083+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4081eb6e0 con 0x7fd408155f70 2024-02-19T14:58:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.084+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40806f2e0 con 0x7fd408155f70 2024-02-19T14:58:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.085+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd3fc0135d0 con 0x7fd408155f70 2024-02-19T14:58:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.085+0000 7fd3effff640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b810 0x7fd3f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.085+0000 7fd4077fe640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b810 0x7fd3f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.085+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fd3fc08a410 con 0x7fd408155f70 2024-02-19T14:58:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.086+0000 7fd4077fe640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b810 0x7fd3f406dcd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8005880 tx=0x7fd3f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:26.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.090+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd40806f2e0 con 0x7fd408155f70 2024-02-19T14:58:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.263+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd40806f2e0 con 0x7fd408155f70 2024-02-19T14:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.957+0000 7fd3effff640 1 -- 172.21.15.78:0/2753383524 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v172) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd40806f2e0 con 0x7fd408155f70 2024-02-19T14:58:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b810 msgr2=0x7fd3f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b810 0x7fd3f406dcd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8005880 tx=0x7fd3f8009040 comp rx=0 tx=0).stop 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 msgr2=0x7fd40806f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 0x7fd40806f5c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00e960 tx=0x7fd3fc00ee20 comp rx=0 tx=0).stop 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 shutdown_connections 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd3f406b810 0x7fd3f406dcd0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd408155f70 0x7fd40806f5c0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081588e0 0x7fd40806fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 --2- 172.21.15.78:0/2753383524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c67e0 0x7fd408151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.960+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 >> 172.21.15.78:0/2753383524 conn(0x7fd40806b6a0 msgr2=0x7fd4080c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.961+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 shutdown_connections 2024-02-19T14:58:26.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:26.961+0000 7fd40ddd0640 1 -- 172.21.15.78:0/2753383524 wait complete. 2024-02-19T14:58:26.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:58:27.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:58:27.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:58:27.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.083+0000 7f1685c07640 1 Processor -- start 2024-02-19T14:58:27.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.083+0000 7f1685c07640 1 -- start start 2024-02-19T14:58:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f1685c07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f16800c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f1685c07640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16800719e0 0x7f16800c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f1685c07640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168006ff90 con 0x7f16800c3d90 2024-02-19T14:58:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f1685c07640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1680070270 con 0x7f16800719e0 2024-02-19T14:58:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f1685c07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16800c3970 con 0x7f168014d210 2024-02-19T14:58:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16800719e0 0x7f16800c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f16800c3d90 0x7f1680152ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58280/0 (socket says 172.21.15.78:58280) 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167ffff640 1 -- 172.21.15.78:0/1576328252 learned_addr learned my addr 172.21.15.78:0/1576328252 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167f7fe640 1 --2- 172.21.15.78:0/1576328252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f16800c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167effd640 1 -- 172.21.15.78:0/1576328252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 msgr2=0x7f16800c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167effd640 1 --2- 172.21.15.78:0/1576328252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f16800c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167effd640 1 -- 172.21.15.78:0/1576328252 >> v1:172.21.15.78:6789/0 conn(0x7f16800c3d90 legacy=0x7f1680152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167effd640 1 -- 172.21.15.78:0/1576328252 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680156fc0 con 0x7f16800719e0 2024-02-19T14:58:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.084+0000 7f167e7fc640 1 -- 172.21.15.78:0/1576328252 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953170558 0 0) 0x7f168006ff90 con 0x7f16800c3d90 2024-02-19T14:58:27.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.085+0000 7f167effd640 1 --2- 172.21.15.78:0/1576328252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800c64d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f16680097a0 tx=0x7f166802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a9c1e1c14a2035 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.085+0000 7f167e7fc640 1 -- 172.21.15.78:0/1576328252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166802e070 con 0x7f16800719e0 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.085+0000 7f167e7fc640 1 -- 172.21.15.78:0/1576328252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16680315a0 con 0x7f16800719e0 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.085+0000 7f167e7fc640 1 -- 172.21.15.78:0/1576328252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668031930 con 0x7f16800719e0 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 -- 172.21.15.78:0/1576328252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 msgr2=0x7f16800c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 --2- 172.21.15.78:0/1576328252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800c64d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f16680097a0 tx=0x7f166802b840 comp rx=0 tx=0).stop 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 -- 172.21.15.78:0/1576328252 shutdown_connections 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 --2- 172.21.15.78:0/1576328252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800c64d0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 --2- 172.21.15.78:0/1576328252 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f16800c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 -- 172.21.15.78:0/1576328252 >> 172.21.15.78:0/1576328252 conn(0x7f16800c5050 msgr2=0x7f16800c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 -- 172.21.15.78:0/1576328252 shutdown_connections 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.086+0000 7f1685c07640 1 -- 172.21.15.78:0/1576328252 wait complete. 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.087+0000 7f1685c07640 1 Processor -- start 2024-02-19T14:58:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.087+0000 7f1685c07640 1 -- start start 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f1685c07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f1685c07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16800c3d90 0x7f16800add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f1685c07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f1680142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f1685c07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801e8fb0 con 0x7f16800719e0 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f1685c07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168014d6a0 con 0x7f16800c3d90 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f1685c07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16801e94e0 con 0x7f168014d210 2024-02-19T14:58:27.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39602/0 (socket says 172.21.15.78:39602) 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 -- 172.21.15.78:0/1010987124 learned_addr learned my addr 172.21.15.78:0/1010987124 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167ffff640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f1680142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167effd640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16800c3d90 0x7f16800add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 -- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 msgr2=0x7f1680142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f1680142780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 -- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16800c3d90 msgr2=0x7f16800add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16800c3d90 0x7f16800add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 -- 172.21.15.78:0/1010987124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1668009400 con 0x7f16800719e0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.088+0000 7f167f7fe640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800ad7c0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f167000efc0 tx=0x7f167000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.089+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670009b80 con 0x7f16800719e0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.089+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167000f040 con 0x7f16800719e0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.089+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16800c3660 con 0x7f16800719e0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.089+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16700146a0 con 0x7f16800719e0 2024-02-19T14:58:27.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.089+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16800ab4a0 con 0x7f16800719e0 2024-02-19T14:58:27.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.090+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680074990 con 0x7f16800719e0 2024-02-19T14:58:27.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.090+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1670014800 con 0x7f16800719e0 2024-02-19T14:58:27.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.090+0000 7f1677fff640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f166006b840 0x7f166006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:27.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.091+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f167008aca0 con 0x7f16800719e0 2024-02-19T14:58:27.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.091+0000 7f167effd640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f166006b840 0x7f166006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:27.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.091+0000 7f167effd640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f166006b840 0x7f166006dd00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1668036000 tx=0x7f166803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:27.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.095+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f16700556d0 con 0x7f16800719e0 2024-02-19T14:58:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.305+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1680149d90 con 0x7f16800719e0 2024-02-19T14:58:27.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.959+0000 7f1677fff640 1 -- 172.21.15.78:0/1010987124 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v150) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1670058d90 con 0x7f16800719e0 2024-02-19T14:58:27.962 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 27 and data pool 28 2024-02-19T14:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.963+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f166006b840 msgr2=0x7f166006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.963+0000 7f1685c07640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f166006b840 0x7f166006dd00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1668036000 tx=0x7f166803d040 comp rx=0 tx=0).stop 2024-02-19T14:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.963+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 msgr2=0x7f16800ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.963+0000 7f1685c07640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800ad7c0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f167000efc0 tx=0x7f167000c490 comp rx=0 tx=0).stop 2024-02-19T14:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.964+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 shutdown_connections 2024-02-19T14:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.964+0000 7f1685c07640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f166006b840 0x7f166006dd00 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.964+0000 7f1685c07640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f168014d210 0x7f1680142780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.964+0000 7f1685c07640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16800c3d90 0x7f16800add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.964+0000 7f1685c07640 1 --2- 172.21.15.78:0/1010987124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16800719e0 0x7f16800ad7c0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.964+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 >> 172.21.15.78:0/1010987124 conn(0x7f16800c5050 msgr2=0x7f168014b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.965+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 shutdown_connections 2024-02-19T14:58:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:27.967+0000 7f1685c07640 1 -- 172.21.15.78:0/1010987124 wait complete. 2024-02-19T14:58:27.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:28.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.097+0000 7fadb0bc9640 1 Processor -- start 2024-02-19T14:58:28.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.097+0000 7fadb0bc9640 1 -- start start 2024-02-19T14:58:28.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadb0bc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadb0bc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac0c34a0 con 0x7fadac0c8500 2024-02-19T14:58:28.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadb0bc9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac06c8e0 con 0x7fadac0c42e0 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadb0bc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac06c090 con 0x7fadac0c2f50 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadaa59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fadac0c8500 0x7fadac158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47196/0 (socket says 172.21.15.78:47196) 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadaa59c640 1 -- 172.21.15.78:0/1734039133 learned_addr learned my addr 172.21.15.78:0/1734039133 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadaad9d640 1 --2- 172.21.15.78:0/1734039133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300552620 0 0) 0x7fadac0c34a0 con 0x7fadac0c8500 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad98003520 con 0x7fadac0c8500 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854383527 0 0) 0x7fadac06c8e0 con 0x7fadac0c42e0 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac0c34a0 con 0x7fadac0c42e0 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237842282 0 0) 0x7fad98003520 con 0x7fadac0c8500 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadaad9d640 1 -- 172.21.15.78:0/1734039133 >> v1:172.21.15.150:6789/0 conn(0x7fadac0c42e0 legacy=0x7fadac15d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadaad9d640 1 -- 172.21.15.78:0/1734039133 >> v1:172.21.15.78:6789/0 conn(0x7fadac0c8500 legacy=0x7fadac158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.098+0000 7fadaad9d640 1 -- 172.21.15.78:0/1734039133 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac06cbc0 con 0x7fadac0c2f50 2024-02-19T14:58:28.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.099+0000 7fadaad9d640 1 --2- 172.21.15.78:0/1734039133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac158d60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fada4009dc0 tx=0x7fada402bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a364c7dfcd7c86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.099+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c001670 con 0x7fadac0c2f50 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.099+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada402c880 con 0x7fadac0c2f50 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.099+0000 7fada9d9b640 1 -- 172.21.15.78:0/1734039133 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada402cc10 con 0x7fadac0c2f50 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 -- 172.21.15.78:0/1734039133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 msgr2=0x7fadac158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/1734039133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac158d60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fada4009dc0 tx=0x7fada402bd50 comp rx=0 tx=0).stop 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 -- 172.21.15.78:0/1734039133 shutdown_connections 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/1734039133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac158d60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 -- 172.21.15.78:0/1734039133 >> 172.21.15.78:0/1734039133 conn(0x7fadac0aa300 msgr2=0x7fadac0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 -- 172.21.15.78:0/1734039133 shutdown_connections 2024-02-19T14:58:28.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.100+0000 7fadb0bc9640 1 -- 172.21.15.78:0/1734039133 wait complete. 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.101+0000 7fadb0bc9640 1 Processor -- start 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 -- start start 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadac0c8500 0x7fadac158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 0x7fadac0c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac152b60 con 0x7fadac0c5020 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac0c5c70 con 0x7fadac0c8500 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.102+0000 7fadb0bc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1ef020 con 0x7fadac0c2f50 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 0x7fadac0c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 0x7fadac0c54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58534/0 (socket says 172.21.15.78:58534) 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 -- 172.21.15.78:0/4270943776 learned_addr learned my addr 172.21.15.78:0/4270943776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:28.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaa59c640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fada3fff640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadac0c8500 0x7fadac158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 -- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 msgr2=0x7fadac14fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac14fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 -- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadac0c8500 msgr2=0x7fadac158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadac0c8500 0x7fadac158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 -- 172.21.15.78:0/4270943776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada4009a70 con 0x7fadac0c5020 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaa59c640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadaad9d640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 0x7fadac0c54d0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fada402c360 tx=0x7fada4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4034e50 con 0x7fadac0c5020 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada402ce70 con 0x7fadac0c5020 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac14ff50 con 0x7fadac0c5020 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4040670 con 0x7fadac0c5020 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.103+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac1ee6f0 con 0x7fadac0c5020 2024-02-19T14:58:28.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.104+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac071c20 con 0x7fadac0c5020 2024-02-19T14:58:28.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.105+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fada40407d0 con 0x7fadac0c5020 2024-02-19T14:58:28.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.105+0000 7fada2ffd640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad8c06b840 0x7fad8c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.105+0000 7fadaa59c640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad8c06b840 0x7fad8c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.105+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fada40acb30 con 0x7fadac0c5020 2024-02-19T14:58:28.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.106+0000 7fadaa59c640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad8c06b840 0x7fad8c06dd00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fad9c001eb0 tx=0x7fad9c001690 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.110+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fada4077500 con 0x7fadac0c5020 2024-02-19T14:58:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.283+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadac074990 con 0x7fadac0c5020 2024-02-19T14:58:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.284+0000 7fada2ffd640 1 -- 172.21.15.78:0/4270943776 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+16953 (secure 0 0 0) 0x7fada4009450 con 0x7fadac0c5020 2024-02-19T14:58:28.287 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:28.287 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":173,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:27.306744+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:58:25.520675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-19T14:58:26.281810+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad8c06b840 msgr2=0x7fad8c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad8c06b840 0x7fad8c06dd00 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fad9c001eb0 tx=0x7fad9c001690 comp rx=0 tx=0).stop 2024-02-19T14:58:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 msgr2=0x7fadac0c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 0x7fadac0c54d0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fada402c360 tx=0x7fada4004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 shutdown_connections 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad8c06b840 0x7fad8c06dd00 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fadac0c5020 0x7fadac0c54d0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadac0c8500 0x7fadac158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 --2- 172.21.15.78:0/4270943776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadac0c2f50 0x7fadac14fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 >> 172.21.15.78:0/4270943776 conn(0x7fadac0aa300 msgr2=0x7fadac070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.286+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 shutdown_connections 2024-02-19T14:58:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.287+0000 7fadb0bc9640 1 -- 172.21.15.78:0/4270943776 wait complete. 2024-02-19T14:58:28.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.398+0000 7f58ee9e9640 1 Processor -- start 2024-02-19T14:58:28.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.398+0000 7f58ee9e9640 1 -- start start 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.398+0000 7f58ee9e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152670 0x7f58e8071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.398+0000 7f58ee9e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e8071bf0 con 0x7f58e80702d0 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.398+0000 7f58ee9e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e8158a20 con 0x7f58e8152a50 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.398+0000 7f58ee9e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e80c6780 con 0x7f58e8152670 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58ecf86640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f58e80702d0 0x7f58e8158eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47202/0 (socket says 172.21.15.78:47202) 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58ecf86640 1 -- 172.21.15.78:0/209209790 learned_addr learned my addr 172.21.15.78:0/209209790 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e7fff640 1 --2- 172.21.15.78:0/209209790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152670 0x7f58e8071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396410979 0 0) 0x7f58e8071bf0 con 0x7f58e80702d0 2024-02-19T14:58:28.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0003520 con 0x7f58e80702d0 2024-02-19T14:58:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085163152 0 0) 0x7f58e8158a20 con 0x7f58e8152a50 2024-02-19T14:58:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e8071bf0 con 0x7f58e8152a50 2024-02-19T14:58:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e7fff640 1 -- 172.21.15.78:0/209209790 >> v1:172.21.15.150:6789/0 conn(0x7f58e8152a50 legacy=0x7f58e8155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e7fff640 1 -- 172.21.15.78:0/209209790 >> v1:172.21.15.78:6789/0 conn(0x7f58e80702d0 legacy=0x7f58e8158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.399+0000 7f58e7fff640 1 -- 172.21.15.78:0/209209790 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e8071630 con 0x7f58e8152670 2024-02-19T14:58:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58e7fff640 1 --2- 172.21.15.78:0/209209790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152670 0x7f58e8071050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f58d80097d0 tx=0x7f58d802b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eff2353acd445c51 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8030890 con 0x7f58e8152670 2024-02-19T14:58:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d80309f0 con 0x7f58e8152670 2024-02-19T14:58:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58e6ffd640 1 -- 172.21.15.78:0/209209790 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8030d80 con 0x7f58e8152670 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58ee9e9640 1 -- 172.21.15.78:0/209209790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152670 msgr2=0x7f58e8071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/209209790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152670 0x7f58e8071050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f58d80097d0 tx=0x7f58d802b9d0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58ee9e9640 1 -- 172.21.15.78:0/209209790 shutdown_connections 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/209209790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152670 0x7f58e8071050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.400+0000 7f58ee9e9640 1 -- 172.21.15.78:0/209209790 >> 172.21.15.78:0/209209790 conn(0x7f58e80c85e0 msgr2=0x7f58e80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.401+0000 7f58ee9e9640 1 -- 172.21.15.78:0/209209790 shutdown_connections 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.401+0000 7f58ee9e9640 1 -- 172.21.15.78:0/209209790 wait complete. 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 Processor -- start 2024-02-19T14:58:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 -- start start 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 0x7f58e806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e8152670 0x7f58e8153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152a50 0x7f58e8153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e81545c0 con 0x7f58e80702d0 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e81eb230 con 0x7f58e8152670 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58ee9e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e81eb4d0 con 0x7f58e8152a50 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.402+0000 7f58e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 0x7f58e806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 0x7f58e806d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58542/0 (socket says 172.21.15.78:58542) 2024-02-19T14:58:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 -- 172.21.15.78:0/479430072 learned_addr learned my addr 172.21.15.78:0/479430072 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58ecf86640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152a50 0x7f58e8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e77fe640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e8152670 0x7f58e8153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 -- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152a50 msgr2=0x7f58e8153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152a50 0x7f58e8153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 -- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e8152670 msgr2=0x7f58e8153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e8152670 0x7f58e8153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 -- 172.21.15.78:0/479430072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d8009400 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e7fff640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 0x7f58e806d250 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f58d80098f0 tx=0x7f58d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d803e040 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d8039c50 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d80443f0 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e8154810 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.403+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e806d790 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.404+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e806e160 con 0x7f58e80702d0 2024-02-19T14:58:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.405+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f58d8044550 con 0x7f58e80702d0 2024-02-19T14:58:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.405+0000 7f58e4ff9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d406b810 0x7f58d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.405+0000 7f58e77fe640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d406b810 0x7f58d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.405+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f58d80acaa0 con 0x7f58e80702d0 2024-02-19T14:58:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.406+0000 7f58e77fe640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d406b810 0x7f58d406dcd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007fd0 tx=0x7f58dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.410+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f58d8077470 con 0x7f58e80702d0 2024-02-19T14:58:28.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.581+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e80676a0 con 0x7f58e80702d0 2024-02-19T14:58:28.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.583+0000 7f58e4ff9640 1 -- 172.21.15.78:0/479430072 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+16953 (secure 0 0 0) 0x7f58d807ab30 con 0x7f58e80702d0 2024-02-19T14:58:28.585 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:28.586 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":173,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:27.306744+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:58:25.520675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-19T14:58:26.281810+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:28.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d406b810 msgr2=0x7f58d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d406b810 0x7f58d406dcd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007fd0 tx=0x7f58dc009040 comp rx=0 tx=0).stop 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 msgr2=0x7f58e806d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 0x7f58e806d250 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f58d80098f0 tx=0x7f58d8004730 comp rx=0 tx=0).stop 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 shutdown_connections 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d406b810 0x7f58d406dcd0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58e8152a50 0x7f58e8153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e8152670 0x7f58e8153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 --2- 172.21.15.78:0/479430072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58e80702d0 0x7f58e806d250 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.584+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 >> 172.21.15.78:0/479430072 conn(0x7f58e80c85e0 msgr2=0x7f58e8158eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.585+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 shutdown_connections 2024-02-19T14:58:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.585+0000 7f58ee9e9640 1 -- 172.21.15.78:0/479430072 wait complete. 2024-02-19T14:58:28.598 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:58:28.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.698+0000 7fa928c09640 1 Processor -- start 2024-02-19T14:58:28.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.698+0000 7fa928c09640 1 -- start start 2024-02-19T14:58:28.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa928c09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa924152700 0x7fa9240c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa928c09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9240c8e00 0x7fa92415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa928c09640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9241551f0 con 0x7fa9240c5730 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa928c09640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa924155490 con 0x7fa9240c8e00 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa928c09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa924155770 con 0x7fa924152700 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa922d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa9240c8e00 0x7fa92415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa921d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa9240c5730 0x7fa92406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47204/0 (socket says 172.21.15.78:47204) 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.699+0000 7fa921d9b640 1 -- 172.21.15.78:0/1920497668 learned_addr learned my addr 172.21.15.78:0/1920497668 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa92259c640 1 --2- 172.21.15.78:0/1920497668 >> v2:172.21.15.150:3300/0 conn(0x7fa924152700 0x7fa9240c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa92159a640 1 -- 172.21.15.78:0/1920497668 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241136170 0 0) 0x7fa9241551f0 con 0x7fa9240c5730 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa92159a640 1 -- 172.21.15.78:0/1920497668 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c003520 con 0x7fa9240c5730 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa922d9d640 1 -- 172.21.15.78:0/1920497668 >> v2:172.21.15.150:3300/0 conn(0x7fa924152700 msgr2=0x7fa9240c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa922d9d640 1 --2- 172.21.15.78:0/1920497668 >> v2:172.21.15.150:3300/0 conn(0x7fa924152700 0x7fa9240c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa922d9d640 1 -- 172.21.15.78:0/1920497668 >> v1:172.21.15.78:6789/0 conn(0x7fa9240c5730 legacy=0x7fa92406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa922d9d640 1 -- 172.21.15.78:0/1920497668 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924155a50 con 0x7fa9240c8e00 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa922d9d640 1 --2- 172.21.15.78:0/1920497668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9240c8e00 0x7fa92415f980 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fa92408bad0 tx=0x7fa91c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4395d52f359390c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.700+0000 7fa92159a640 1 -- 172.21.15.78:0/1920497668 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c02e070 con 0x7fa9240c8e00 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa92159a640 1 -- 172.21.15.78:0/1920497668 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c0315a0 con 0x7fa9240c8e00 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa92159a640 1 -- 172.21.15.78:0/1920497668 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c031930 con 0x7fa9240c8e00 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 -- 172.21.15.78:0/1920497668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9240c8e00 msgr2=0x7fa92415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 --2- 172.21.15.78:0/1920497668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9240c8e00 0x7fa92415f980 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fa92408bad0 tx=0x7fa91c02b840 comp rx=0 tx=0).stop 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 -- 172.21.15.78:0/1920497668 shutdown_connections 2024-02-19T14:58:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 --2- 172.21.15.78:0/1920497668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9240c8e00 0x7fa92415f980 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 --2- 172.21.15.78:0/1920497668 >> v2:172.21.15.150:3300/0 conn(0x7fa924152700 0x7fa9240c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 -- 172.21.15.78:0/1920497668 >> 172.21.15.78:0/1920497668 conn(0x7fa9240cac90 msgr2=0x7fa9240cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 -- 172.21.15.78:0/1920497668 shutdown_connections 2024-02-19T14:58:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.701+0000 7fa928c09640 1 -- 172.21.15.78:0/1920497668 wait complete. 2024-02-19T14:58:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.702+0000 7fa928c09640 1 Processor -- start 2024-02-19T14:58:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.702+0000 7fa928c09640 1 -- start start 2024-02-19T14:58:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa928c09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9240c5730 0x7fa9241ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa928c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9240c8e00 0x7fa9240acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa928c09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 0x7fa9240c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa928c09640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9241edbb0 con 0x7fa924152700 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa928c09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240ab410 con 0x7fa9240c5730 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa928c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240aab70 con 0x7fa9240c8e00 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 0x7fa9240c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 0x7fa9240c2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58556/0 (socket says 172.21.15.78:58556) 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 -- 172.21.15.78:0/1487646340 learned_addr learned my addr 172.21.15.78:0/1487646340 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa92259c640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9240c5730 0x7fa9241ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa921d9b640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9240c8e00 0x7fa9240acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 -- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9240c8e00 msgr2=0x7fa9240acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9240c8e00 0x7fa9240acb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 -- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9240c5730 msgr2=0x7fa9241ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9240c5730 0x7fa9241ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa922d9d640 1 -- 172.21.15.78:0/1487646340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c009400 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.703+0000 7fa92259c640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9240c5730 0x7fa9241ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.704+0000 7fa922d9d640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 0x7fa9240c2a00 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fa91c034040 tx=0x7fa91c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.704+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c032840 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.704+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c0329c0 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.704+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c032d70 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.704+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9240aae50 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.704+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9241ede00 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.705+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa924074990 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.705+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa91c0453f0 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.705+0000 7fa912ffd640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa90406b810 0x7fa90406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.706+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa91c0ac2b0 con 0x7fa924152700 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.706+0000 7fa92259c640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa90406b810 0x7fa90406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:28.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.706+0000 7fa92259c640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa90406b810 0x7fa90406dcd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa9240bdfb0 tx=0x7fa9140039f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.710+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa91c076c80 con 0x7fa924152700 2024-02-19T14:58:28.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.917+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa9240c8770 con 0x7fa924152700 2024-02-19T14:58:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.984+0000 7fa912ffd640 1 -- 172.21.15.78:0/1487646340 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v151) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa91c07a340 con 0x7fa924152700 2024-02-19T14:58:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.986+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa90406b810 msgr2=0x7fa90406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.986+0000 7fa928c09640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa90406b810 0x7fa90406dcd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa9240bdfb0 tx=0x7fa9140039f0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.986+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 msgr2=0x7fa9240c2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.986+0000 7fa928c09640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 0x7fa9240c2a00 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fa91c034040 tx=0x7fa91c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 shutdown_connections 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa90406b810 0x7fa90406dcd0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa924152700 0x7fa9240c2a00 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9240c8e00 0x7fa9240acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 --2- 172.21.15.78:0/1487646340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9240c5730 0x7fa9241ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 >> 172.21.15.78:0/1487646340 conn(0x7fa9240cac90 msgr2=0x7fa9240ca120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 shutdown_connections 2024-02-19T14:58:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:28.987+0000 7fa928c09640 1 -- 172.21.15.78:0/1487646340 wait complete. 2024-02-19T14:58:29.000 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:58:29.001 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:58:29.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.101+0000 7fcd7e8a4640 1 Processor -- start 2024-02-19T14:58:29.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7e8a4640 1 -- start start 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7e8a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c8500 0x7fcd780c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7e8a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd780c2f50 0x7fcd78158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7e8a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd780c34a0 con 0x7fcd780c42e0 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7e8a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7806c090 con 0x7fcd780c2f50 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7e8a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7806d6a0 con 0x7fcd780c8500 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd7ce41640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd780c2f50 0x7fcd78158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd777fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcd780c42e0 0x7fcd78158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47206/0 (socket says 172.21.15.78:47206) 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.102+0000 7fcd777fe640 1 -- 172.21.15.78:0/3332235401 learned_addr learned my addr 172.21.15.78:0/3332235401 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd77fff640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c8500 0x7fcd780c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd7ce41640 1 -- 172.21.15.78:0/3332235401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c8500 msgr2=0x7fcd780c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd7ce41640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c8500 0x7fcd780c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd7ce41640 1 -- 172.21.15.78:0/3332235401 >> v1:172.21.15.78:6789/0 conn(0x7fcd780c42e0 legacy=0x7fcd78158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:29.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd76ffd640 1 -- 172.21.15.78:0/3332235401 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336103082 0 0) 0x7fcd780c34a0 con 0x7fcd780c42e0 2024-02-19T14:58:29.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd7ce41640 1 -- 172.21.15.78:0/3332235401 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7806c8e0 con 0x7fcd780c2f50 2024-02-19T14:58:29.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.103+0000 7fcd77fff640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c8500 0x7fcd780c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:29.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.104+0000 7fcd7ce41640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780c2f50 0x7fcd78158d60 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009d90 tx=0x7fcd7002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2dce8289eddd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:29.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.104+0000 7fcd76ffd640 1 -- 172.21.15.78:0/3332235401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7002cbc0 con 0x7fcd780c2f50 2024-02-19T14:58:29.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.104+0000 7fcd76ffd640 1 -- 172.21.15.78:0/3332235401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7002cd20 con 0x7fcd780c2f50 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.104+0000 7fcd76ffd640 1 -- 172.21.15.78:0/3332235401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70034660 con 0x7fcd780c2f50 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/3332235401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780c2f50 msgr2=0x7fcd78158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780c2f50 0x7fcd78158d60 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009d90 tx=0x7fcd7002be10 comp rx=0 tx=0).stop 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/3332235401 shutdown_connections 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd780c2f50 0x7fcd78158d60 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/3332235401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c8500 0x7fcd780c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/3332235401 >> 172.21.15.78:0/3332235401 conn(0x7fcd780aa300 msgr2=0x7fcd780aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/3332235401 shutdown_connections 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.105+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/3332235401 wait complete. 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.106+0000 7fcd7e8a4640 1 Processor -- start 2024-02-19T14:58:29.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.106+0000 7fcd7e8a4640 1 -- start start 2024-02-19T14:58:29.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c2f50 0x7fcd781ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:29.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd780c8500 0x7fcd781583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:29.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 0x7fcd78070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78157fc0 con 0x7fcd78159280 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd781ec270 con 0x7fcd780c8500 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78157620 con 0x7fcd780c2f50 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 0x7fcd78070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 0x7fcd78070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58570/0 (socket says 172.21.15.78:58570) 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 -- 172.21.15.78:0/576961884 learned_addr learned my addr 172.21.15.78:0/576961884 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd77fff640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c2f50 0x7fcd781ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd777fe640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd780c8500 0x7fcd781583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 -- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c2f50 msgr2=0x7fcd781ebd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c2f50 0x7fcd781ebd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 -- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd780c8500 msgr2=0x7fcd781583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd780c8500 0x7fcd781583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 -- 172.21.15.78:0/576961884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd70009a70 con 0x7fcd78159280 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7ce41640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 0x7fcd78070e40 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009d90 tx=0x7fcd70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:29.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70040650 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7002ce90 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd78157900 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70040a10 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.107+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd78158920 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.108+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd78071c20 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.109+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcd70040b70 con 0x7fcd78159280 2024-02-19T14:58:29.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.109+0000 7fcd74ff9640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd6406b810 0x7fcd6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:29.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.109+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcd700ad070 con 0x7fcd78159280 2024-02-19T14:58:29.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.109+0000 7fcd77fff640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd6406b810 0x7fcd6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:29.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.110+0000 7fcd77fff640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd6406b810 0x7fcd6406dcd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00a950 tx=0x7fcd6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:29.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.114+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd70077ab0 con 0x7fcd78159280 2024-02-19T14:58:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.324+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcd78158100 con 0x7fcd78159280 2024-02-19T14:58:29.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.990+0000 7fcd74ff9640 1 -- 172.21.15.78:0/576961884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v152) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd78158100 con 0x7fcd78159280 2024-02-19T14:58:29.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd6406b810 msgr2=0x7fcd6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:29.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd6406b810 0x7fcd6406dcd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00a950 tx=0x7fcd6c008040 comp rx=0 tx=0).stop 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 msgr2=0x7fcd78070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 0x7fcd78070e40 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009d90 tx=0x7fcd70004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 shutdown_connections 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd6406b810 0x7fcd6406dcd0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd78159280 0x7fcd78070e40 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd780c8500 0x7fcd781583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 --2- 172.21.15.78:0/576961884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd780c2f50 0x7fcd781ebd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 >> 172.21.15.78:0/576961884 conn(0x7fcd780aa300 msgr2=0x7fcd780c4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 shutdown_connections 2024-02-19T14:58:29.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:29.992+0000 7fcd7e8a4640 1 -- 172.21.15.78:0/576961884 wait complete. 2024-02-19T14:58:30.009 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:58:30.009 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.108+0000 7f37ccbc9640 1 Processor -- start 2024-02-19T14:58:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.108+0000 7f37ccbc9640 1 -- start start 2024-02-19T14:58:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37ccbc9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f37c80add70 0x7f37c80c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37ccbc9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f37c80705b0 0x7f37c80709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37ccbc9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c80ae670 con 0x7f37c80c2f60 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37ccbc9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80716b0 con 0x7f37c80705b0 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37ccbc9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80c33d0 con 0x7f37c80add70 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37c6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f37c80705b0 0x7f37c80709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37c6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f37c80705b0 0x7f37c80709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58574/0 (socket says 172.21.15.78:58574) 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.109+0000 7f37c6d9d640 1 -- 172.21.15.78:0/62703065 learned_addr learned my addr 172.21.15.78:0/62703065 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c6d9d640 1 -- 172.21.15.78:0/62703065 >> v2:172.21.15.150:3300/0 conn(0x7f37c80add70 msgr2=0x7f37c80c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c6d9d640 1 --2- 172.21.15.78:0/62703065 >> v2:172.21.15.150:3300/0 conn(0x7f37c80add70 0x7f37c80c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c6d9d640 1 -- 172.21.15.78:0/62703065 >> v1:172.21.15.150:6789/0 conn(0x7f37c80c2f60 legacy=0x7f37c806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c6d9d640 1 -- 172.21.15.78:0/62703065 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c8153780 con 0x7f37c80705b0 2024-02-19T14:58:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c6d9d640 1 --2- 172.21.15.78:0/62703065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80709c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f37c80c4fc0 tx=0x7f37c002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ecf7c7a54d6b81 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c559a640 1 -- 172.21.15.78:0/62703065 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c002c9f0 con 0x7f37c80705b0 2024-02-19T14:58:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c559a640 1 -- 172.21.15.78:0/62703065 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c002cb50 con 0x7f37c80705b0 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.110+0000 7f37c559a640 1 -- 172.21.15.78:0/62703065 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c00343f0 con 0x7f37c80705b0 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 -- 172.21.15.78:0/62703065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 msgr2=0x7f37c80709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/62703065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80709c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f37c80c4fc0 tx=0x7f37c002be50 comp rx=0 tx=0).stop 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 -- 172.21.15.78:0/62703065 shutdown_connections 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/62703065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80709c0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/62703065 >> v2:172.21.15.150:3300/0 conn(0x7f37c80add70 0x7f37c80c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 -- 172.21.15.78:0/62703065 >> 172.21.15.78:0/62703065 conn(0x7f37c80c85e0 msgr2=0x7f37c80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 -- 172.21.15.78:0/62703065 shutdown_connections 2024-02-19T14:58:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.111+0000 7f37ccbc9640 1 -- 172.21.15.78:0/62703065 wait complete. 2024-02-19T14:58:30.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.112+0000 7f37ccbc9640 1 Processor -- start 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.112+0000 7f37ccbc9640 1 -- start start 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37ccbc9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80c15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37ccbc9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37c80add70 0x7f37c814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37ccbc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c80c2f60 0x7f37c814b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37ccbc9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80c0200 con 0x7f37c80705b0 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37ccbc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c81ec050 con 0x7f37c80c2f60 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37ccbc9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8156c80 con 0x7f37c80add70 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80c15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80c15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58586/0 (socket says 172.21.15.78:58586) 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 -- 172.21.15.78:0/259266717 learned_addr learned my addr 172.21.15.78:0/259266717 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 -- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37c80add70 msgr2=0x7f37c814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37c80add70 0x7f37c814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c6d9d640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c80c2f60 0x7f37c814b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 -- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c80c2f60 msgr2=0x7f37c814b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c80c2f60 0x7f37c814b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.113+0000 7f37c659c640 1 -- 172.21.15.78:0/259266717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c0009b30 con 0x7f37c80705b0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.114+0000 7f37c659c640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80c15f0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f37b800b440 tx=0x7f37b800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.114+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8011890 con 0x7f37c80705b0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.114+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b80119f0 con 0x7f37c80705b0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.114+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c8156f60 con 0x7f37c80705b0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.114+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8011da0 con 0x7f37c80705b0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.114+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c80c2110 con 0x7f37c80705b0 2024-02-19T14:58:30.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.115+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c8074990 con 0x7f37c80705b0 2024-02-19T14:58:30.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.116+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f37b801a400 con 0x7f37c80705b0 2024-02-19T14:58:30.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.116+0000 7f37b6ffd640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f37a806b840 0x7f37a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.116+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f37b808a5b0 con 0x7f37c80705b0 2024-02-19T14:58:30.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.116+0000 7f37c5d9b640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f37a806b840 0x7f37a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.116+0000 7f37c5d9b640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f37a806b840 0x7f37a806dd00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f37c8152e90 tx=0x7f37bc007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.120+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f37b8054ed0 con 0x7f37c80705b0 2024-02-19T14:58:30.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.335+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37c8071490 con 0x7f37c80705b0 2024-02-19T14:58:30.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.335+0000 7f37b6ffd640 1 -- 172.21.15.78:0/259266717 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f37b8058590 con 0x7f37c80705b0 2024-02-19T14:58:30.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:30.338 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":152,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:29.325646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:30.339 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 152 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f37a806b840 msgr2=0x7f37a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f37a806b840 0x7f37a806dd00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f37c8152e90 tx=0x7f37bc007990 comp rx=0 tx=0).stop 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 msgr2=0x7f37c80c15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80c15f0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f37b800b440 tx=0x7f37b800b900 comp rx=0 tx=0).stop 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 shutdown_connections 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f37a806b840 0x7f37a806dd00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37c80c2f60 0x7f37c814b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37c80add70 0x7f37c814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 --2- 172.21.15.78:0/259266717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37c80705b0 0x7f37c80c15f0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 >> 172.21.15.78:0/259266717 conn(0x7f37c80c85e0 msgr2=0x7f37c8152440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:30.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 shutdown_connections 2024-02-19T14:58:30.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.338+0000 7f37ccbc9640 1 -- 172.21.15.78:0/259266717 wait complete. 2024-02-19T14:58:30.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 Processor -- start 2024-02-19T14:58:30.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 -- start start 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 0x7f8ba00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ba0070720 0x7f8ba0070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00c65a0 con 0x7f8ba00c7910 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba00c7530 con 0x7f8ba0070720 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.487+0000 7f8ba6b8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba01537b0 con 0x7f8ba00c5560 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 0x7f8ba00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba5129640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ba0070720 0x7f8ba0070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 0x7f8ba00c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:41190/0 (socket says 172.21.15.78:41190) 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8b9ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8ba00c7910 0x7f8ba0154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:55698/0 (socket says 172.21.15.78:55698) 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 -- 172.21.15.78:0/1137954495 learned_addr learned my addr 172.21.15.78:0/1137954495 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 -- 172.21.15.78:0/1137954495 >> v2:172.21.15.150:3300/0 conn(0x7f8ba0070720 msgr2=0x7f8ba0070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 --2- 172.21.15.78:0/1137954495 >> v2:172.21.15.150:3300/0 conn(0x7f8ba0070720 0x7f8ba0070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 -- 172.21.15.78:0/1137954495 >> v1:172.21.15.150:6789/0 conn(0x7f8ba00c7910 legacy=0x7f8ba0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba4928640 1 -- 172.21.15.78:0/1137954495 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba0153a90 con 0x7f8ba00c5560 2024-02-19T14:58:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.488+0000 7f8ba5129640 1 --2- 172.21.15.78:0/1137954495 >> v2:172.21.15.150:3300/0 conn(0x7f8ba0070720 0x7f8ba0070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.489+0000 7f8ba4928640 1 --2- 172.21.15.78:0/1137954495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 0x7f8ba00c6ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009870 tx=0x7f8b9402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e18c3465f15acf54 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.489+0000 7f8b9f7fe640 1 -- 172.21.15.78:0/1137954495 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b940317c0 con 0x7f8ba00c5560 2024-02-19T14:58:30.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8b9f7fe640 1 -- 172.21.15.78:0/1137954495 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b94031920 con 0x7f8ba00c5560 2024-02-19T14:58:30.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8b9f7fe640 1 -- 172.21.15.78:0/1137954495 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94031cb0 con 0x7f8ba00c5560 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/1137954495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 msgr2=0x7f8ba00c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/1137954495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 0x7f8ba00c6ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009870 tx=0x7f8b9402b860 comp rx=0 tx=0).stop 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/1137954495 shutdown_connections 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/1137954495 >> v2:172.21.15.150:3300/0 conn(0x7f8ba0070720 0x7f8ba0070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/1137954495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c5560 0x7f8ba00c6ff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/1137954495 >> 172.21.15.78:0/1137954495 conn(0x7f8ba006b4b0 msgr2=0x7f8ba00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/1137954495 shutdown_connections 2024-02-19T14:58:30.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.490+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/1137954495 wait complete. 2024-02-19T14:58:30.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.492+0000 7f8ba6b8c640 1 Processor -- start 2024-02-19T14:58:30.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.492+0000 7f8ba6b8c640 1 -- start start 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba6b8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 0x7f8ba0152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba6b8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ba00c5560 0x7f8ba00a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba6b8c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c7910 0x7f8ba00a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba6b8c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba01ec940 con 0x7f8ba0070720 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba6b8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba01ebb80 con 0x7f8ba00c5560 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba6b8c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba01ebe60 con 0x7f8ba00c7910 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.493+0000 7f8ba4928640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 0x7f8ba0152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 0x7f8ba0152b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58602/0 (socket says 172.21.15.78:58602) 2024-02-19T14:58:30.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 -- 172.21.15.78:0/2752447207 learned_addr learned my addr 172.21.15.78:0/2752447207 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8b9ffff640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ba00c5560 0x7f8ba00a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba5129640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c7910 0x7f8ba00a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 -- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c7910 msgr2=0x7f8ba00a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c7910 0x7f8ba00a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 -- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ba00c5560 msgr2=0x7f8ba00a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ba00c5560 0x7f8ba00a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 -- 172.21.15.78:0/2752447207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98009a70 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba5129640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c7910 0x7f8ba00a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba4928640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 0x7f8ba0152b20 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f8b9402b8a0 tx=0x7f8b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94032860 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b94032a00 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b940094d0 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94032db0 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.494+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba00c5ac0 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.496+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba0074990 con 0x7f8ba0070720 2024-02-19T14:58:30.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.496+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8b9404e450 con 0x7f8ba0070720 2024-02-19T14:58:30.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.496+0000 7f8b9d7fa640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8b8406b810 0x7f8b8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.496+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8b940ac320 con 0x7f8ba0070720 2024-02-19T14:58:30.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.496+0000 7f8b9ffff640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8b8406b810 0x7f8b8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.497+0000 7f8b9ffff640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8b8406b810 0x7f8b8406dcd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8ba01dff00 tx=0x7f8b8c007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.501+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b94076c40 con 0x7f8ba0070720 2024-02-19T14:58:30.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.702+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ba00c76f0 con 0x7f8ba0070720 2024-02-19T14:58:30.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.703+0000 7f8b9d7fa640 1 -- 172.21.15.78:0/2752447207 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16940 (secure 0 0 0) 0x7f8b9407a300 con 0x7f8ba0070720 2024-02-19T14:58:30.706 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:30.706 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:28.959861+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:58:25.520675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-19T14:58:26.281810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:30.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8b8406b810 msgr2=0x7f8b8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8b8406b810 0x7f8b8406dcd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8ba01dff00 tx=0x7f8b8c007e80 comp rx=0 tx=0).stop 2024-02-19T14:58:30.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 msgr2=0x7f8ba0152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 0x7f8ba0152b20 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f8b9402b8a0 tx=0x7f8b940046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 shutdown_connections 2024-02-19T14:58:30.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8b8406b810 0x7f8b8406dcd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ba00c7910 0x7f8ba00a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ba00c5560 0x7f8ba00a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 --2- 172.21.15.78:0/2752447207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ba0070720 0x7f8ba0152b20 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 >> 172.21.15.78:0/2752447207 conn(0x7f8ba006b4b0 msgr2=0x7f8ba0154080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:30.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 shutdown_connections 2024-02-19T14:58:30.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.705+0000 7f8ba6b8c640 1 -- 172.21.15.78:0/2752447207 wait complete. 2024-02-19T14:58:30.723 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:58:30.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:58:30.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.821+0000 7fdbe60bb640 1 Processor -- start 2024-02-19T14:58:30.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.821+0000 7fdbe60bb640 1 -- start start 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbe60bb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdbe014cd50 0x7fdbe00c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbe60bb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdbe01537a0 0x7fdbe00c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbe60bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe00c7b70 0x7fdbe0156860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbe60bb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe01563b0 con 0x7fdbe01537a0 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbe60bb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe0153cd0 con 0x7fdbe014cd50 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbe60bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe006b6e0 con 0x7fdbe00c7b70 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdbe01537a0 0x7fdbe00c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdbe01537a0 0x7fdbe00c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58616/0 (socket says 172.21.15.78:58616) 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 -- 172.21.15.78:0/924208836 learned_addr learned my addr 172.21.15.78:0/924208836 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbdf7fe640 1 --2- 172.21.15.78:0/924208836 >> v2:172.21.15.150:3300/0 conn(0x7fdbe014cd50 0x7fdbe00c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbdffff640 1 --2- 172.21.15.78:0/924208836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe00c7b70 0x7fdbe0156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 -- 172.21.15.78:0/924208836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe00c7b70 msgr2=0x7fdbe0156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/924208836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe00c7b70 0x7fdbe0156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 -- 172.21.15.78:0/924208836 >> v2:172.21.15.150:3300/0 conn(0x7fdbe014cd50 msgr2=0x7fdbe00c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/924208836 >> v2:172.21.15.150:3300/0 conn(0x7fdbe014cd50 0x7fdbe00c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.822+0000 7fdbd7fff640 1 -- 172.21.15.78:0/924208836 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe006b980 con 0x7fdbe01537a0 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.823+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/924208836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe01537a0 0x7fdbe00c7630 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00ace20 tx=0x7fdbc802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757d432f0f983958 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.823+0000 7fdbdeffd640 1 -- 172.21.15.78:0/924208836 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8031750 con 0x7fdbe01537a0 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.823+0000 7fdbdeffd640 1 -- 172.21.15.78:0/924208836 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc80318b0 con 0x7fdbe01537a0 2024-02-19T14:58:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.823+0000 7fdbdeffd640 1 -- 172.21.15.78:0/924208836 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8031c40 con 0x7fdbe01537a0 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 -- 172.21.15.78:0/924208836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe01537a0 msgr2=0x7fdbe00c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/924208836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe01537a0 0x7fdbe00c7630 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00ace20 tx=0x7fdbc802b7f0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 -- 172.21.15.78:0/924208836 shutdown_connections 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/924208836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe00c7b70 0x7fdbe0156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/924208836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe01537a0 0x7fdbe00c7630 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/924208836 >> v2:172.21.15.150:3300/0 conn(0x7fdbe014cd50 0x7fdbe00c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 -- 172.21.15.78:0/924208836 >> 172.21.15.78:0/924208836 conn(0x7fdbe0066e70 msgr2=0x7fdbe00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 -- 172.21.15.78:0/924208836 shutdown_connections 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.824+0000 7fdbe60bb640 1 -- 172.21.15.78:0/924208836 wait complete. 2024-02-19T14:58:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.825+0000 7fdbe60bb640 1 Processor -- start 2024-02-19T14:58:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.825+0000 7fdbe60bb640 1 -- start start 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 0x7fdbe01dc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe014cd50 0x7fdbe01ec370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbe01537a0 0x7fdbe01ddc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00c65b0 con 0x7fdbe00c7b70 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe01ecd20 con 0x7fdbe01537a0 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00c5f60 con 0x7fdbe014cd50 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 0x7fdbe01dc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 0x7fdbe01dc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58624/0 (socket says 172.21.15.78:58624) 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 -- 172.21.15.78:0/3174855801 learned_addr learned my addr 172.21.15.78:0/3174855801 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdffff640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbe01537a0 0x7fdbe01ddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe014cd50 0x7fdbe01ec370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 -- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe014cd50 msgr2=0x7fdbe01ec370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe014cd50 0x7fdbe01ec370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 -- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbe01537a0 msgr2=0x7fdbe01ddc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbe01537a0 0x7fdbe01ddc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 -- 172.21.15.78:0/3174855801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80094d0 con 0x7fdbe00c7b70 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe014cd50 0x7fdbe01ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdffff640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbe01537a0 0x7fdbe01ddc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdf7fe640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 0x7fdbe01dc380 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000e870 tx=0x7fdbd000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00094f0 con 0x7fdbe00c7b70 2024-02-19T14:58:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd000c940 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe00c6240 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.826+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd000ccf0 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.827+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe00c72a0 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.827+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe0074990 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.828+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdbd0013470 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.828+0000 7fdbdcff9640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbc006b810 0x7fdbc006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.828+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbc006b810 0x7fdbc006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.828+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbd008a430 con 0x7fdbe00c7b70 2024-02-19T14:58:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.829+0000 7fdbd7fff640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbc006b810 0x7fdbc006dcd0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8035000 tx=0x7fdbc8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:30.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:30.833+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdbd0054cc0 con 0x7fdbe00c7b70 2024-02-19T14:58:31.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.007+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbe006bc60 con 0x7fdbc006b810 2024-02-19T14:58:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.015+0000 7fdbdcff9640 1 -- 172.21.15.78:0/3174855801 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290001 (secure 0 0 0) 0x7fdbc8031b20 con 0x7fdbc006b810 2024-02-19T14:58:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:58:31.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbc006b810 msgr2=0x7fdbc006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbc006b810 0x7fdbc006dcd0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8035000 tx=0x7fdbc8008030 comp rx=0 tx=0).stop 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 msgr2=0x7fdbe01dc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 0x7fdbe01dc380 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000e870 tx=0x7fdbd000ed30 comp rx=0 tx=0).stop 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 shutdown_connections 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdbc006b810 0x7fdbc006dcd0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbe01537a0 0x7fdbe01ddc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbe014cd50 0x7fdbe01ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 --2- 172.21.15.78:0/3174855801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdbe00c7b70 0x7fdbe01dc380 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 >> 172.21.15.78:0/3174855801 conn(0x7fdbe0066e70 msgr2=0x7fdbe0154e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 shutdown_connections 2024-02-19T14:58:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.018+0000 7fdbe60bb640 1 -- 172.21.15.78:0/3174855801 wait complete. 2024-02-19T14:58:31.035 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-02-19T14:58:30.136051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267672,"kb_used_data":18188,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754707048,"statfs":{"total":773094113280,"available":772820017152,"internally_reserved":0,"allocated":18624512,"data_stored":9714772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.182050"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986814+0000","last_change":"2024-02-19T14:58:27.984713+0000","last_active":"2024-02-19T14:58:27.986814+0000","last_peered":"2024-02-19T14:58:27.986814+0000","last_clean":"2024-02-19T14:58:27.986814+0000","last_became_active":"2024-02-19T14:58:27.984466+0000","last_became_peered":"2024-02-19T14:58:27.984466+0000","last_unstale":"2024-02-19T14:58:27.986814+0000","last_undegraded":"2024-02-19T14:58:27.986814+0000","last_fullsized":"2024-02-19T14:58:27.986814+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:23.147062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.983305+0000","last_change":"2024-02-19T14:58:27.980688+0000","last_active":"2024-02-19T14:58:27.983305+0000","last_peered":"2024-02-19T14:58:27.983305+0000","last_clean":"2024-02-19T14:58:27.983305+0000","last_became_active":"2024-02-19T14:58:27.980399+0000","last_became_peered":"2024-02-19T14:58:27.980399+0000","last_unstale":"2024-02-19T14:58:27.983305+0000","last_undegraded":"2024-02-19T14:58:27.983305+0000","last_fullsized":"2024-02-19T14:58:27.983305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:10.635784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.979029+0000","last_change":"2024-02-19T14:58:27.974855+0000","last_active":"2024-02-19T14:58:27.979029+0000","last_peered":"2024-02-19T14:58:27.979029+0000","last_clean":"2024-02-19T14:58:27.979029+0000","last_became_active":"2024-02-19T14:58:27.974583+0000","last_became_peered":"2024-02-19T14:58:27.974583+0000","last_unstale":"2024-02-19T14:58:27.979029+0000","last_undegraded":"2024-02-19T14:58:27.979029+0000","last_fullsized":"2024-02-19T14:58:27.979029+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:37.236794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991371+0000","last_change":"2024-02-19T14:58:27.983684+0000","last_active":"2024-02-19T14:58:27.991371+0000","last_peered":"2024-02-19T14:58:27.991371+0000","last_clean":"2024-02-19T14:58:27.991371+0000","last_became_active":"2024-02-19T14:58:27.983447+0000","last_became_peered":"2024-02-19T14:58:27.983447+0000","last_unstale":"2024-02-19T14:58:27.991371+0000","last_undegraded":"2024-02-19T14:58:27.991371+0000","last_fullsized":"2024-02-19T14:58:27.991371+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:21.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986019+0000","last_change":"2024-02-19T14:58:27.982626+0000","last_active":"2024-02-19T14:58:27.986019+0000","last_peered":"2024-02-19T14:58:27.986019+0000","last_clean":"2024-02-19T14:58:27.986019+0000","last_became_active":"2024-02-19T14:58:27.979746+0000","last_became_peered":"2024-02-19T14:58:27.979746+0000","last_unstale":"2024-02-19T14:58:27.986019+0000","last_undegraded":"2024-02-19T14:58:27.986019+0000","last_fullsized":"2024-02-19T14:58:27.986019+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:18.279287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.980184+0000","last_change":"2024-02-19T14:58:27.977180+0000","last_active":"2024-02-19T14:58:27.980184+0000","last_peered":"2024-02-19T14:58:27.980184+0000","last_clean":"2024-02-19T14:58:27.980184+0000","last_became_active":"2024-02-19T14:58:27.976920+0000","last_became_peered":"2024-02-19T14:58:27.976920+0000","last_unstale":"2024-02-19T14:58:27.980184+0000","last_undegraded":"2024-02-19T14:58:27.980184+0000","last_fullsized":"2024-02-19T14:58:27.980184+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:56.669840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.984762+0000","last_change":"2024-02-19T14:58:27.978679+0000","last_active":"2024-02-19T14:58:27.984762+0000","last_peered":"2024-02-19T14:58:27.984762+0000","last_clean":"2024-02-19T14:58:27.984762+0000","last_became_active":"2024-02-19T14:58:27.978379+0000","last_became_peered":"2024-02-19T14:58:27.978379+0000","last_unstale":"2024-02-19T14:58:27.984762+0000","last_undegraded":"2024-02-19T14:58:27.984762+0000","last_fullsized":"2024-02-19T14:58:27.984762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:42.779035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.983836+0000","last_change":"2024-02-19T14:58:27.972055+0000","last_active":"2024-02-19T14:58:27.983836+0000","last_peered":"2024-02-19T14:58:27.983836+0000","last_clean":"2024-02-19T14:58:27.983836+0000","last_became_active":"2024-02-19T14:58:27.971799+0000","last_became_peered":"2024-02-19T14:58:27.971799+0000","last_unstale":"2024-02-19T14:58:27.983836+0000","last_undegraded":"2024-02-19T14:58:27.983836+0000","last_fullsized":"2024-02-19T14:58:27.983836+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:37:32.242539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993301+0000","last_change":"2024-02-19T14:58:27.980450+0000","last_active":"2024-02-19T14:58:27.993301+0000","last_peered":"2024-02-19T14:58:27.993301+0000","last_clean":"2024-02-19T14:58:27.993301+0000","last_became_active":"2024-02-19T14:58:27.980184+0000","last_became_peered":"2024-02-19T14:58:27.980184+0000","last_unstale":"2024-02-19T14:58:27.993301+0000","last_undegraded":"2024-02-19T14:58:27.993301+0000","last_fullsized":"2024-02-19T14:58:27.993301+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:39.867110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987515+0000","last_change":"2024-02-19T14:58:27.981112+0000","last_active":"2024-02-19T14:58:27.987515+0000","last_peered":"2024-02-19T14:58:27.987515+0000","last_clean":"2024-02-19T14:58:27.987515+0000","last_became_active":"2024-02-19T14:58:27.980837+0000","last_became_peered":"2024-02-19T14:58:27.980837+0000","last_unstale":"2024-02-19T14:58:27.987515+0000","last_undegraded":"2024-02-19T14:58:27.987515+0000","last_fullsized":"2024-02-19T14:58:27.987515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:24.806445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.976000+0000","last_change":"2024-02-19T14:58:27.972580+0000","last_active":"2024-02-19T14:58:27.976000+0000","last_peered":"2024-02-19T14:58:27.976000+0000","last_clean":"2024-02-19T14:58:27.976000+0000","last_became_active":"2024-02-19T14:58:27.971810+0000","last_became_peered":"2024-02-19T14:58:27.971810+0000","last_unstale":"2024-02-19T14:58:27.976000+0000","last_undegraded":"2024-02-19T14:58:27.976000+0000","last_fullsized":"2024-02-19T14:58:27.976000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:57.589892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987456+0000","last_change":"2024-02-19T14:58:27.983884+0000","last_active":"2024-02-19T14:58:27.987456+0000","last_peered":"2024-02-19T14:58:27.987456+0000","last_clean":"2024-02-19T14:58:27.987456+0000","last_became_active":"2024-02-19T14:58:27.983599+0000","last_became_peered":"2024-02-19T14:58:27.983599+0000","last_unstale":"2024-02-19T14:58:27.987456+0000","last_undegraded":"2024-02-19T14:58:27.987456+0000","last_fullsized":"2024-02-19T14:58:27.987456+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:34.549285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991403+0000","last_change":"2024-02-19T14:58:27.987638+0000","last_active":"2024-02-19T14:58:27.991403+0000","last_peered":"2024-02-19T14:58:27.991403+0000","last_clean":"2024-02-19T14:58:27.991403+0000","last_became_active":"2024-02-19T14:58:27.987382+0000","last_became_peered":"2024-02-19T14:58:27.987382+0000","last_unstale":"2024-02-19T14:58:27.991403+0000","last_undegraded":"2024-02-19T14:58:27.991403+0000","last_fullsized":"2024-02-19T14:58:27.991403+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:39.867110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993376+0000","last_change":"2024-02-19T14:58:27.987349+0000","last_active":"2024-02-19T14:58:27.993376+0000","last_peered":"2024-02-19T14:58:27.993376+0000","last_clean":"2024-02-19T14:58:27.993376+0000","last_became_active":"2024-02-19T14:58:27.987071+0000","last_became_peered":"2024-02-19T14:58:27.987071+0000","last_unstale":"2024-02-19T14:58:27.993376+0000","last_undegraded":"2024-02-19T14:58:27.993376+0000","last_fullsized":"2024-02-19T14:58:27.993376+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:16.168858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.977332+0000","last_change":"2024-02-19T14:58:27.974824+0000","last_active":"2024-02-19T14:58:27.977332+0000","last_peered":"2024-02-19T14:58:27.977332+0000","last_clean":"2024-02-19T14:58:27.977332+0000","last_became_active":"2024-02-19T14:58:27.973793+0000","last_became_peered":"2024-02-19T14:58:27.973793+0000","last_unstale":"2024-02-19T14:58:27.977332+0000","last_undegraded":"2024-02-19T14:58:27.977332+0000","last_fullsized":"2024-02-19T14:58:27.977332+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:17.607381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.981803+0000","last_change":"2024-02-19T14:58:27.976545+0000","last_active":"2024-02-19T14:58:27.981803+0000","last_peered":"2024-02-19T14:58:27.981803+0000","last_clean":"2024-02-19T14:58:27.981803+0000","last_became_active":"2024-02-19T14:58:27.976280+0000","last_became_peered":"2024-02-19T14:58:27.976280+0000","last_unstale":"2024-02-19T14:58:27.981803+0000","last_undegraded":"2024-02-19T14:58:27.981803+0000","last_fullsized":"2024-02-19T14:58:27.981803+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:04.735200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993283+0000","last_change":"2024-02-19T14:58:27.978262+0000","last_active":"2024-02-19T14:58:27.993283+0000","last_peered":"2024-02-19T14:58:27.993283+0000","last_clean":"2024-02-19T14:58:27.993283+0000","last_became_active":"2024-02-19T14:58:27.977737+0000","last_became_peered":"2024-02-19T14:58:27.977737+0000","last_unstale":"2024-02-19T14:58:27.993283+0000","last_undegraded":"2024-02-19T14:58:27.993283+0000","last_fullsized":"2024-02-19T14:58:27.993283+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:04.735200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.979741+0000","last_change":"2024-02-19T14:58:27.975054+0000","last_active":"2024-02-19T14:58:27.979741+0000","last_peered":"2024-02-19T14:58:27.979741+0000","last_clean":"2024-02-19T14:58:27.979741+0000","last_became_active":"2024-02-19T14:58:27.974789+0000","last_became_peered":"2024-02-19T14:58:27.974789+0000","last_unstale":"2024-02-19T14:58:27.979741+0000","last_undegraded":"2024-02-19T14:58:27.979741+0000","last_fullsized":"2024-02-19T14:58:27.979741+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:57.589892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990311+0000","last_change":"2024-02-19T14:58:27.984722+0000","last_active":"2024-02-19T14:58:27.990311+0000","last_peered":"2024-02-19T14:58:27.990311+0000","last_clean":"2024-02-19T14:58:27.990311+0000","last_became_active":"2024-02-19T14:58:27.984462+0000","last_became_peered":"2024-02-19T14:58:27.984462+0000","last_unstale":"2024-02-19T14:58:27.990311+0000","last_undegraded":"2024-02-19T14:58:27.990311+0000","last_fullsized":"2024-02-19T14:58:27.990311+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:30.360740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986173+0000","last_change":"2024-02-19T14:58:27.977158+0000","last_active":"2024-02-19T14:58:27.986173+0000","last_peered":"2024-02-19T14:58:27.986173+0000","last_clean":"2024-02-19T14:58:27.986173+0000","last_became_active":"2024-02-19T14:58:27.976915+0000","last_became_peered":"2024-02-19T14:58:27.976915+0000","last_unstale":"2024-02-19T14:58:27.986173+0000","last_undegraded":"2024-02-19T14:58:27.986173+0000","last_fullsized":"2024-02-19T14:58:27.986173+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:16.045644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986979+0000","last_change":"2024-02-19T14:58:27.980486+0000","last_active":"2024-02-19T14:58:27.986979+0000","last_peered":"2024-02-19T14:58:27.986979+0000","last_clean":"2024-02-19T14:58:27.986979+0000","last_became_active":"2024-02-19T14:58:27.980243+0000","last_became_peered":"2024-02-19T14:58:27.980243+0000","last_unstale":"2024-02-19T14:58:27.986979+0000","last_undegraded":"2024-02-19T14:58:27.986979+0000","last_fullsized":"2024-02-19T14:58:27.986979+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:43.848035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987240+0000","last_change":"2024-02-19T14:58:27.978357+0000","last_active":"2024-02-19T14:58:27.987240+0000","last_peered":"2024-02-19T14:58:27.987240+0000","last_clean":"2024-02-19T14:58:27.987240+0000","last_became_active":"2024-02-19T14:58:27.978112+0000","last_became_peered":"2024-02-19T14:58:27.978112+0000","last_unstale":"2024-02-19T14:58:27.987240+0000","last_undegraded":"2024-02-19T14:58:27.987240+0000","last_fullsized":"2024-02-19T14:58:27.987240+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:13.596506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993427+0000","last_change":"2024-02-19T14:58:27.990692+0000","last_active":"2024-02-19T14:58:27.993427+0000","last_peered":"2024-02-19T14:58:27.993427+0000","last_clean":"2024-02-19T14:58:27.993427+0000","last_became_active":"2024-02-19T14:58:27.990348+0000","last_became_peered":"2024-02-19T14:58:27.990348+0000","last_unstale":"2024-02-19T14:58:27.993427+0000","last_undegraded":"2024-02-19T14:58:27.993427+0000","last_fullsized":"2024-02-19T14:58:27.993427+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:41:31.500536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.984579+0000","last_change":"2024-02-19T14:58:27.977731+0000","last_active":"2024-02-19T14:58:27.984579+0000","last_peered":"2024-02-19T14:58:27.984579+0000","last_clean":"2024-02-19T14:58:27.984579+0000","last_became_active":"2024-02-19T14:58:27.977460+0000","last_became_peered":"2024-02-19T14:58:27.977460+0000","last_unstale":"2024-02-19T14:58:27.984579+0000","last_undegraded":"2024-02-19T14:58:27.984579+0000","last_fullsized":"2024-02-19T14:58:27.984579+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:05.556447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.977917+0000","last_change":"2024-02-19T14:58:27.972536+0000","last_active":"2024-02-19T14:58:27.977917+0000","last_peered":"2024-02-19T14:58:27.977917+0000","last_clean":"2024-02-19T14:58:27.977917+0000","last_became_active":"2024-02-19T14:58:27.972278+0000","last_became_peered":"2024-02-19T14:58:27.972278+0000","last_unstale":"2024-02-19T14:58:27.977917+0000","last_undegraded":"2024-02-19T14:58:27.977917+0000","last_fullsized":"2024-02-19T14:58:27.977917+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:21.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986031+0000","last_change":"2024-02-19T14:58:27.983244+0000","last_active":"2024-02-19T14:58:27.986031+0000","last_peered":"2024-02-19T14:58:27.986031+0000","last_clean":"2024-02-19T14:58:27.986031+0000","last_became_active":"2024-02-19T14:58:27.982987+0000","last_became_peered":"2024-02-19T14:58:27.982987+0000","last_unstale":"2024-02-19T14:58:27.986031+0000","last_undegraded":"2024-02-19T14:58:27.986031+0000","last_fullsized":"2024-02-19T14:58:27.986031+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:54:31.439375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.975957+0000","last_change":"2024-02-19T14:58:27.971575+0000","last_active":"2024-02-19T14:58:27.975957+0000","last_peered":"2024-02-19T14:58:27.975957+0000","last_clean":"2024-02-19T14:58:27.975957+0000","last_became_active":"2024-02-19T14:58:27.971053+0000","last_became_peered":"2024-02-19T14:58:27.971053+0000","last_unstale":"2024-02-19T14:58:27.975957+0000","last_undegraded":"2024-02-19T14:58:27.975957+0000","last_fullsized":"2024-02-19T14:58:27.975957+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:50.431913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987479+0000","last_change":"2024-02-19T14:58:27.984450+0000","last_active":"2024-02-19T14:58:27.987479+0000","last_peered":"2024-02-19T14:58:27.987479+0000","last_clean":"2024-02-19T14:58:27.987479+0000","last_became_active":"2024-02-19T14:58:27.983567+0000","last_became_peered":"2024-02-19T14:58:27.983567+0000","last_unstale":"2024-02-19T14:58:27.987479+0000","last_undegraded":"2024-02-19T14:58:27.987479+0000","last_fullsized":"2024-02-19T14:58:27.987479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:55.124259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.977453+0000","last_change":"2024-02-19T14:58:27.970961+0000","last_active":"2024-02-19T14:58:27.977453+0000","last_peered":"2024-02-19T14:58:27.977453+0000","last_clean":"2024-02-19T14:58:27.977453+0000","last_became_active":"2024-02-19T14:58:27.970689+0000","last_became_peered":"2024-02-19T14:58:27.970689+0000","last_unstale":"2024-02-19T14:58:27.977453+0000","last_undegraded":"2024-02-19T14:58:27.977453+0000","last_fullsized":"2024-02-19T14:58:27.977453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:50.431913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986619+0000","last_change":"2024-02-19T14:58:27.979671+0000","last_active":"2024-02-19T14:58:27.986619+0000","last_peered":"2024-02-19T14:58:27.986619+0000","last_clean":"2024-02-19T14:58:27.986619+0000","last_became_active":"2024-02-19T14:58:27.979417+0000","last_became_peered":"2024-02-19T14:58:27.979417+0000","last_unstale":"2024-02-19T14:58:27.986619+0000","last_undegraded":"2024-02-19T14:58:27.986619+0000","last_fullsized":"2024-02-19T14:58:27.986619+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:36.835628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982305+0000","last_change":"2024-02-19T14:58:27.979641+0000","last_active":"2024-02-19T14:58:27.982305+0000","last_peered":"2024-02-19T14:58:27.982305+0000","last_clean":"2024-02-19T14:58:27.982305+0000","last_became_active":"2024-02-19T14:58:27.979387+0000","last_became_peered":"2024-02-19T14:58:27.979387+0000","last_unstale":"2024-02-19T14:58:27.982305+0000","last_undegraded":"2024-02-19T14:58:27.982305+0000","last_fullsized":"2024-02-19T14:58:27.982305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:16.192020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985394+0000","last_change":"2024-02-19T14:58:27.981496+0000","last_active":"2024-02-19T14:58:27.985394+0000","last_peered":"2024-02-19T14:58:27.985394+0000","last_clean":"2024-02-19T14:58:27.985394+0000","last_became_active":"2024-02-19T14:58:27.981167+0000","last_became_peered":"2024-02-19T14:58:27.981167+0000","last_unstale":"2024-02-19T14:58:27.985394+0000","last_undegraded":"2024-02-19T14:58:27.985394+0000","last_fullsized":"2024-02-19T14:58:27.985394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:50.431913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993388+0000","last_change":"2024-02-19T14:58:27.989844+0000","last_active":"2024-02-19T14:58:27.993388+0000","last_peered":"2024-02-19T14:58:27.993388+0000","last_clean":"2024-02-19T14:58:27.993388+0000","last_became_active":"2024-02-19T14:58:27.989554+0000","last_became_peered":"2024-02-19T14:58:27.989554+0000","last_unstale":"2024-02-19T14:58:27.993388+0000","last_undegraded":"2024-02-19T14:58:27.993388+0000","last_fullsized":"2024-02-19T14:58:27.993388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:30.332650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"173'4","reported_seq":16,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993808+0000","last_change":"2024-02-19T14:58:26.972658+0000","last_active":"2024-02-19T14:58:27.993808+0000","last_peered":"2024-02-19T14:58:27.993808+0000","last_clean":"2024-02-19T14:58:27.993808+0000","last_became_active":"2024-02-19T14:58:26.972378+0000","last_became_peered":"2024-02-19T14:58:26.972378+0000","last_unstale":"2024-02-19T14:58:27.993808+0000","last_undegraded":"2024-02-19T14:58:27.993808+0000","last_fullsized":"2024-02-19T14:58:27.993808+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:12.589654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982543+0000","last_change":"2024-02-19T14:58:27.978079+0000","last_active":"2024-02-19T14:58:27.982543+0000","last_peered":"2024-02-19T14:58:27.982543+0000","last_clean":"2024-02-19T14:58:27.982543+0000","last_became_active":"2024-02-19T14:58:27.977825+0000","last_became_peered":"2024-02-19T14:58:27.977825+0000","last_unstale":"2024-02-19T14:58:27.982543+0000","last_undegraded":"2024-02-19T14:58:27.982543+0000","last_fullsized":"2024-02-19T14:58:27.982543+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:57.572536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994141+0000","last_change":"2024-02-19T14:58:26.975494+0000","last_active":"2024-02-19T14:58:27.994141+0000","last_peered":"2024-02-19T14:58:27.994141+0000","last_clean":"2024-02-19T14:58:27.994141+0000","last_became_active":"2024-02-19T14:58:26.975180+0000","last_became_peered":"2024-02-19T14:58:26.975180+0000","last_unstale":"2024-02-19T14:58:27.994141+0000","last_undegraded":"2024-02-19T14:58:27.994141+0000","last_fullsized":"2024-02-19T14:58:27.994141+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:25:58.712820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990229+0000","last_change":"2024-02-19T14:58:27.981826+0000","last_active":"2024-02-19T14:58:27.990229+0000","last_peered":"2024-02-19T14:58:27.990229+0000","last_clean":"2024-02-19T14:58:27.990229+0000","last_became_active":"2024-02-19T14:58:27.981551+0000","last_became_peered":"2024-02-19T14:58:27.981551+0000","last_unstale":"2024-02-19T14:58:27.990229+0000","last_undegraded":"2024-02-19T14:58:27.990229+0000","last_fullsized":"2024-02-19T14:58:27.990229+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:17.607381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991533+0000","last_change":"2024-02-19T14:58:26.971321+0000","last_active":"2024-02-19T14:58:27.991533+0000","last_peered":"2024-02-19T14:58:27.991533+0000","last_clean":"2024-02-19T14:58:27.991533+0000","last_became_active":"2024-02-19T14:58:26.971009+0000","last_became_peered":"2024-02-19T14:58:26.971009+0000","last_unstale":"2024-02-19T14:58:27.991533+0000","last_undegraded":"2024-02-19T14:58:27.991533+0000","last_fullsized":"2024-02-19T14:58:27.991533+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:48:20.276462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985995+0000","last_change":"2024-02-19T14:58:27.982177+0000","last_active":"2024-02-19T14:58:27.985995+0000","last_peered":"2024-02-19T14:58:27.985995+0000","last_clean":"2024-02-19T14:58:27.985995+0000","last_became_active":"2024-02-19T14:58:27.979467+0000","last_became_peered":"2024-02-19T14:58:27.979467+0000","last_unstale":"2024-02-19T14:58:27.985995+0000","last_undegraded":"2024-02-19T14:58:27.985995+0000","last_fullsized":"2024-02-19T14:58:27.985995+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:27.855566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993817+0000","last_change":"2024-02-19T14:58:26.977908+0000","last_active":"2024-02-19T14:58:27.993817+0000","last_peered":"2024-02-19T14:58:27.993817+0000","last_clean":"2024-02-19T14:58:27.993817+0000","last_became_active":"2024-02-19T14:58:26.977679+0000","last_became_peered":"2024-02-19T14:58:26.977679+0000","last_unstale":"2024-02-19T14:58:27.993817+0000","last_undegraded":"2024-02-19T14:58:27.993817+0000","last_fullsized":"2024-02-19T14:58:27.993817+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:06.447711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990262+0000","last_change":"2024-02-19T14:58:27.983534+0000","last_active":"2024-02-19T14:58:27.990262+0000","last_peered":"2024-02-19T14:58:27.990262+0000","last_clean":"2024-02-19T14:58:27.990262+0000","last_became_active":"2024-02-19T14:58:27.983253+0000","last_became_peered":"2024-02-19T14:58:27.983253+0000","last_unstale":"2024-02-19T14:58:27.990262+0000","last_undegraded":"2024-02-19T14:58:27.990262+0000","last_fullsized":"2024-02-19T14:58:27.990262+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:30.360740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.992225+0000","last_change":"2024-02-19T14:58:26.980151+0000","last_active":"2024-02-19T14:58:27.992225+0000","last_peered":"2024-02-19T14:58:27.992225+0000","last_clean":"2024-02-19T14:58:27.992225+0000","last_became_active":"2024-02-19T14:58:26.979862+0000","last_became_peered":"2024-02-19T14:58:26.979862+0000","last_unstale":"2024-02-19T14:58:27.992225+0000","last_undegraded":"2024-02-19T14:58:27.992225+0000","last_fullsized":"2024-02-19T14:58:27.992225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:54:30.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991283+0000","last_change":"2024-02-19T14:58:27.981918+0000","last_active":"2024-02-19T14:58:27.991283+0000","last_peered":"2024-02-19T14:58:27.991283+0000","last_clean":"2024-02-19T14:58:27.991283+0000","last_became_active":"2024-02-19T14:58:27.981648+0000","last_became_peered":"2024-02-19T14:58:27.981648+0000","last_unstale":"2024-02-19T14:58:27.991283+0000","last_undegraded":"2024-02-19T14:58:27.991283+0000","last_fullsized":"2024-02-19T14:58:27.991283+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:56:28.335597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970924+0000","last_change":"2024-02-19T14:58:26.980109+0000","last_active":"2024-02-19T14:58:27.970924+0000","last_peered":"2024-02-19T14:58:27.970924+0000","last_clean":"2024-02-19T14:58:27.970924+0000","last_became_active":"2024-02-19T14:58:26.979636+0000","last_became_peered":"2024-02-19T14:58:26.979636+0000","last_unstale":"2024-02-19T14:58:27.970924+0000","last_undegraded":"2024-02-19T14:58:27.970924+0000","last_fullsized":"2024-02-19T14:58:27.970924+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:33.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993260+0000","last_change":"2024-02-19T14:58:27.978037+0000","last_active":"2024-02-19T14:58:27.993260+0000","last_peered":"2024-02-19T14:58:27.993260+0000","last_clean":"2024-02-19T14:58:27.993260+0000","last_became_active":"2024-02-19T14:58:27.972931+0000","last_became_peered":"2024-02-19T14:58:27.972931+0000","last_unstale":"2024-02-19T14:58:27.993260+0000","last_undegraded":"2024-02-19T14:58:27.993260+0000","last_fullsized":"2024-02-19T14:58:27.993260+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:37.236794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963324+0000","last_change":"2024-02-19T14:58:26.974223+0000","last_active":"2024-02-19T14:58:27.963324+0000","last_peered":"2024-02-19T14:58:27.963324+0000","last_clean":"2024-02-19T14:58:27.963324+0000","last_became_active":"2024-02-19T14:58:26.973934+0000","last_became_peered":"2024-02-19T14:58:26.973934+0000","last_unstale":"2024-02-19T14:58:27.963324+0000","last_undegraded":"2024-02-19T14:58:27.963324+0000","last_fullsized":"2024-02-19T14:58:27.963324+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:33.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.980381+0000","last_change":"2024-02-19T14:58:27.975944+0000","last_active":"2024-02-19T14:58:27.980381+0000","last_peered":"2024-02-19T14:58:27.980381+0000","last_clean":"2024-02-19T14:58:27.980381+0000","last_became_active":"2024-02-19T14:58:27.975686+0000","last_became_peered":"2024-02-19T14:58:27.975686+0000","last_unstale":"2024-02-19T14:58:27.980381+0000","last_undegraded":"2024-02-19T14:58:27.980381+0000","last_fullsized":"2024-02-19T14:58:27.980381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:45.110380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.973636+0000","last_change":"2024-02-19T14:58:26.973275+0000","last_active":"2024-02-19T14:58:27.973636+0000","last_peered":"2024-02-19T14:58:27.973636+0000","last_clean":"2024-02-19T14:58:27.973636+0000","last_became_active":"2024-02-19T14:58:26.972810+0000","last_became_peered":"2024-02-19T14:58:26.972810+0000","last_unstale":"2024-02-19T14:58:27.973636+0000","last_undegraded":"2024-02-19T14:58:27.973636+0000","last_fullsized":"2024-02-19T14:58:27.973636+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:45.169903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987523+0000","last_change":"2024-02-19T14:58:27.984347+0000","last_active":"2024-02-19T14:58:27.987523+0000","last_peered":"2024-02-19T14:58:27.987523+0000","last_clean":"2024-02-19T14:58:27.987523+0000","last_became_active":"2024-02-19T14:58:27.984065+0000","last_became_peered":"2024-02-19T14:58:27.984065+0000","last_unstale":"2024-02-19T14:58:27.987523+0000","last_undegraded":"2024-02-19T14:58:27.987523+0000","last_fullsized":"2024-02-19T14:58:27.987523+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:17.590610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970859+0000","last_change":"2024-02-19T14:58:26.980157+0000","last_active":"2024-02-19T14:58:27.970859+0000","last_peered":"2024-02-19T14:58:27.970859+0000","last_clean":"2024-02-19T14:58:27.970859+0000","last_became_active":"2024-02-19T14:58:26.979578+0000","last_became_peered":"2024-02-19T14:58:26.979578+0000","last_unstale":"2024-02-19T14:58:27.970859+0000","last_undegraded":"2024-02-19T14:58:27.970859+0000","last_fullsized":"2024-02-19T14:58:27.970859+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:22.140210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985762+0000","last_change":"2024-02-19T14:58:27.981480+0000","last_active":"2024-02-19T14:58:27.985762+0000","last_peered":"2024-02-19T14:58:27.985762+0000","last_clean":"2024-02-19T14:58:27.985762+0000","last_became_active":"2024-02-19T14:58:27.980916+0000","last_became_peered":"2024-02-19T14:58:27.980916+0000","last_unstale":"2024-02-19T14:58:27.985762+0000","last_undegraded":"2024-02-19T14:58:27.985762+0000","last_fullsized":"2024-02-19T14:58:27.985762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:49.281422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"173'3","reported_seq":15,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994243+0000","last_change":"2024-02-19T14:58:26.975674+0000","last_active":"2024-02-19T14:58:27.994243+0000","last_peered":"2024-02-19T14:58:27.994243+0000","last_clean":"2024-02-19T14:58:27.994243+0000","last_became_active":"2024-02-19T14:58:26.975246+0000","last_became_peered":"2024-02-19T14:58:26.975246+0000","last_unstale":"2024-02-19T14:58:27.994243+0000","last_undegraded":"2024-02-19T14:58:27.994243+0000","last_fullsized":"2024-02-19T14:58:27.994243+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:22.140210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.989316+0000","last_change":"2024-02-19T14:58:27.986141+0000","last_active":"2024-02-19T14:58:27.989316+0000","last_peered":"2024-02-19T14:58:27.989316+0000","last_clean":"2024-02-19T14:58:27.989316+0000","last_became_active":"2024-02-19T14:58:27.985792+0000","last_became_peered":"2024-02-19T14:58:27.985792+0000","last_unstale":"2024-02-19T14:58:27.989316+0000","last_undegraded":"2024-02-19T14:58:27.989316+0000","last_fullsized":"2024-02-19T14:58:27.989316+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:56.797501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"173'6","reported_seq":18,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.996662+0000","last_change":"2024-02-19T14:58:26.977763+0000","last_active":"2024-02-19T14:58:27.996662+0000","last_peered":"2024-02-19T14:58:27.996662+0000","last_clean":"2024-02-19T14:58:27.996662+0000","last_became_active":"2024-02-19T14:58:26.977441+0000","last_became_peered":"2024-02-19T14:58:26.977441+0000","last_unstale":"2024-02-19T14:58:27.996662+0000","last_undegraded":"2024-02-19T14:58:27.996662+0000","last_fullsized":"2024-02-19T14:58:27.996662+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:06.447711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993412+0000","last_change":"2024-02-19T14:58:27.989892+0000","last_active":"2024-02-19T14:58:27.993412+0000","last_peered":"2024-02-19T14:58:27.993412+0000","last_clean":"2024-02-19T14:58:27.993412+0000","last_became_active":"2024-02-19T14:58:27.989618+0000","last_became_peered":"2024-02-19T14:58:27.989618+0000","last_unstale":"2024-02-19T14:58:27.993412+0000","last_undegraded":"2024-02-19T14:58:27.993412+0000","last_fullsized":"2024-02-19T14:58:27.993412+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:18.528717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991690+0000","last_change":"2024-02-19T14:58:26.967948+0000","last_active":"2024-02-19T14:58:27.991690+0000","last_peered":"2024-02-19T14:58:27.991690+0000","last_clean":"2024-02-19T14:58:27.991690+0000","last_became_active":"2024-02-19T14:58:26.967699+0000","last_became_peered":"2024-02-19T14:58:26.967699+0000","last_unstale":"2024-02-19T14:58:27.991690+0000","last_undegraded":"2024-02-19T14:58:27.991690+0000","last_fullsized":"2024-02-19T14:58:27.991690+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:38.320852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.975981+0000","last_change":"2024-02-19T14:58:27.972224+0000","last_active":"2024-02-19T14:58:27.975981+0000","last_peered":"2024-02-19T14:58:27.975981+0000","last_clean":"2024-02-19T14:58:27.975981+0000","last_became_active":"2024-02-19T14:58:27.971966+0000","last_became_peered":"2024-02-19T14:58:27.971966+0000","last_unstale":"2024-02-19T14:58:27.975981+0000","last_undegraded":"2024-02-19T14:58:27.975981+0000","last_fullsized":"2024-02-19T14:58:27.975981+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:21.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963319+0000","last_change":"2024-02-19T14:58:26.970089+0000","last_active":"2024-02-19T14:58:27.963319+0000","last_peered":"2024-02-19T14:58:27.963319+0000","last_clean":"2024-02-19T14:58:27.963319+0000","last_became_active":"2024-02-19T14:58:26.969826+0000","last_became_peered":"2024-02-19T14:58:26.969826+0000","last_unstale":"2024-02-19T14:58:27.963319+0000","last_undegraded":"2024-02-19T14:58:27.963319+0000","last_fullsized":"2024-02-19T14:58:27.963319+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:22.140210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993904+0000","last_change":"2024-02-19T14:58:27.992823+0000","last_active":"2024-02-19T14:58:27.993904+0000","last_peered":"2024-02-19T14:58:27.993904+0000","last_clean":"2024-02-19T14:58:27.993904+0000","last_became_active":"2024-02-19T14:58:27.986687+0000","last_became_peered":"2024-02-19T14:58:27.986687+0000","last_unstale":"2024-02-19T14:58:27.993904+0000","last_undegraded":"2024-02-19T14:58:27.993904+0000","last_fullsized":"2024-02-19T14:58:27.993904+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:50.093108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.966993+0000","last_change":"2024-02-19T14:58:26.978319+0000","last_active":"2024-02-19T14:58:27.966993+0000","last_peered":"2024-02-19T14:58:27.966993+0000","last_clean":"2024-02-19T14:58:27.966993+0000","last_became_active":"2024-02-19T14:58:26.977982+0000","last_became_peered":"2024-02-19T14:58:26.977982+0000","last_unstale":"2024-02-19T14:58:27.966993+0000","last_undegraded":"2024-02-19T14:58:27.966993+0000","last_fullsized":"2024-02-19T14:58:27.966993+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:54:30.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991153+0000","last_change":"2024-02-19T14:58:27.986576+0000","last_active":"2024-02-19T14:58:27.991153+0000","last_peered":"2024-02-19T14:58:27.991153+0000","last_clean":"2024-02-19T14:58:27.991153+0000","last_became_active":"2024-02-19T14:58:27.986297+0000","last_became_peered":"2024-02-19T14:58:27.986297+0000","last_unstale":"2024-02-19T14:58:27.991153+0000","last_undegraded":"2024-02-19T14:58:27.991153+0000","last_fullsized":"2024-02-19T14:58:27.991153+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:37.236794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"173'8","reported_seq":20,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994709+0000","last_change":"2024-02-19T14:58:26.973198+0000","last_active":"2024-02-19T14:58:27.994709+0000","last_peered":"2024-02-19T14:58:27.994709+0000","last_clean":"2024-02-19T14:58:27.994709+0000","last_became_active":"2024-02-19T14:58:26.972933+0000","last_became_peered":"2024-02-19T14:58:26.972933+0000","last_unstale":"2024-02-19T14:58:27.994709+0000","last_undegraded":"2024-02-19T14:58:27.994709+0000","last_fullsized":"2024-02-19T14:58:27.994709+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:38.124989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991213+0000","last_change":"2024-02-19T14:58:27.983712+0000","last_active":"2024-02-19T14:58:27.991213+0000","last_peered":"2024-02-19T14:58:27.991213+0000","last_clean":"2024-02-19T14:58:27.991213+0000","last_became_active":"2024-02-19T14:58:27.983102+0000","last_became_peered":"2024-02-19T14:58:27.983102+0000","last_unstale":"2024-02-19T14:58:27.991213+0000","last_undegraded":"2024-02-19T14:58:27.991213+0000","last_fullsized":"2024-02-19T14:58:27.991213+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:36.835628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993935+0000","last_change":"2024-02-19T14:58:26.969777+0000","last_active":"2024-02-19T14:58:27.993935+0000","last_peered":"2024-02-19T14:58:27.993935+0000","last_clean":"2024-02-19T14:58:27.993935+0000","last_became_active":"2024-02-19T14:58:26.969462+0000","last_became_peered":"2024-02-19T14:58:26.969462+0000","last_unstale":"2024-02-19T14:58:27.993935+0000","last_undegraded":"2024-02-19T14:58:27.993935+0000","last_fullsized":"2024-02-19T14:58:27.993935+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:02.496644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986952+0000","last_change":"2024-02-19T14:58:27.985905+0000","last_active":"2024-02-19T14:58:27.986952+0000","last_peered":"2024-02-19T14:58:27.986952+0000","last_clean":"2024-02-19T14:58:27.986952+0000","last_became_active":"2024-02-19T14:58:27.985665+0000","last_became_peered":"2024-02-19T14:58:27.985665+0000","last_unstale":"2024-02-19T14:58:27.986952+0000","last_undegraded":"2024-02-19T14:58:27.986952+0000","last_fullsized":"2024-02-19T14:58:27.986952+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:34.846534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.992665+0000","last_change":"2024-02-19T14:58:26.973761+0000","last_active":"2024-02-19T14:58:27.992665+0000","last_peered":"2024-02-19T14:58:27.992665+0000","last_clean":"2024-02-19T14:58:27.992665+0000","last_became_active":"2024-02-19T14:58:26.973498+0000","last_became_peered":"2024-02-19T14:58:26.973498+0000","last_unstale":"2024-02-19T14:58:27.992665+0000","last_undegraded":"2024-02-19T14:58:27.992665+0000","last_fullsized":"2024-02-19T14:58:27.992665+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:30.104435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982489+0000","last_change":"2024-02-19T14:58:27.979778+0000","last_active":"2024-02-19T14:58:27.982489+0000","last_peered":"2024-02-19T14:58:27.982489+0000","last_clean":"2024-02-19T14:58:27.982489+0000","last_became_active":"2024-02-19T14:58:27.979509+0000","last_became_peered":"2024-02-19T14:58:27.979509+0000","last_unstale":"2024-02-19T14:58:27.982489+0000","last_undegraded":"2024-02-19T14:58:27.982489+0000","last_fullsized":"2024-02-19T14:58:27.982489+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:27:21.765067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991980+0000","last_change":"2024-02-19T14:58:26.974164+0000","last_active":"2024-02-19T14:58:27.991980+0000","last_peered":"2024-02-19T14:58:27.991980+0000","last_clean":"2024-02-19T14:58:27.991980+0000","last_became_active":"2024-02-19T14:58:26.973870+0000","last_became_peered":"2024-02-19T14:58:26.973870+0000","last_unstale":"2024-02-19T14:58:27.991980+0000","last_undegraded":"2024-02-19T14:58:27.991980+0000","last_fullsized":"2024-02-19T14:58:27.991980+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:47:33.132337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986214+0000","last_change":"2024-02-19T14:58:27.977947+0000","last_active":"2024-02-19T14:58:27.986214+0000","last_peered":"2024-02-19T14:58:27.986214+0000","last_clean":"2024-02-19T14:58:27.986214+0000","last_became_active":"2024-02-19T14:58:27.977700+0000","last_became_peered":"2024-02-19T14:58:27.977700+0000","last_unstale":"2024-02-19T14:58:27.986214+0000","last_undegraded":"2024-02-19T14:58:27.986214+0000","last_fullsized":"2024-02-19T14:58:27.986214+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:23.169778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.973399+0000","last_change":"2024-02-19T14:58:26.969658+0000","last_active":"2024-02-19T14:58:27.973399+0000","last_peered":"2024-02-19T14:58:27.973399+0000","last_clean":"2024-02-19T14:58:27.973399+0000","last_became_active":"2024-02-19T14:58:26.969374+0000","last_became_peered":"2024-02-19T14:58:26.969374+0000","last_unstale":"2024-02-19T14:58:27.973399+0000","last_undegraded":"2024-02-19T14:58:27.973399+0000","last_fullsized":"2024-02-19T14:58:27.973399+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:14.299623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.989245+0000","last_change":"2024-02-19T14:58:27.982543+0000","last_active":"2024-02-19T14:58:27.989245+0000","last_peered":"2024-02-19T14:58:27.989245+0000","last_clean":"2024-02-19T14:58:27.989245+0000","last_became_active":"2024-02-19T14:58:27.982187+0000","last_became_peered":"2024-02-19T14:58:27.982187+0000","last_unstale":"2024-02-19T14:58:27.989245+0000","last_undegraded":"2024-02-19T14:58:27.989245+0000","last_fullsized":"2024-02-19T14:58:27.989245+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:01.784277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991208+0000","last_change":"2024-02-19T14:58:26.976093+0000","last_active":"2024-02-19T14:58:27.991208+0000","last_peered":"2024-02-19T14:58:27.991208+0000","last_clean":"2024-02-19T14:58:27.991208+0000","last_became_active":"2024-02-19T14:58:26.975811+0000","last_became_peered":"2024-02-19T14:58:26.975811+0000","last_unstale":"2024-02-19T14:58:27.991208+0000","last_undegraded":"2024-02-19T14:58:27.991208+0000","last_fullsized":"2024-02-19T14:58:27.991208+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:57.166212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.989929+0000","last_change":"2024-02-19T14:58:27.984817+0000","last_active":"2024-02-19T14:58:27.989929+0000","last_peered":"2024-02-19T14:58:27.989929+0000","last_clean":"2024-02-19T14:58:27.989929+0000","last_became_active":"2024-02-19T14:58:27.984568+0000","last_became_peered":"2024-02-19T14:58:27.984568+0000","last_unstale":"2024-02-19T14:58:27.989929+0000","last_undegraded":"2024-02-19T14:58:27.989929+0000","last_fullsized":"2024-02-19T14:58:27.989929+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:57.589892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.971672+0000","last_change":"2024-02-19T14:58:26.973029+0000","last_active":"2024-02-19T14:58:27.971672+0000","last_peered":"2024-02-19T14:58:27.971672+0000","last_clean":"2024-02-19T14:58:27.971672+0000","last_became_active":"2024-02-19T14:58:26.972763+0000","last_became_peered":"2024-02-19T14:58:26.972763+0000","last_unstale":"2024-02-19T14:58:27.971672+0000","last_undegraded":"2024-02-19T14:58:27.971672+0000","last_fullsized":"2024-02-19T14:58:27.971672+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:05.143158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985694+0000","last_change":"2024-02-19T14:58:27.979602+0000","last_active":"2024-02-19T14:58:27.985694+0000","last_peered":"2024-02-19T14:58:27.985694+0000","last_clean":"2024-02-19T14:58:27.985694+0000","last_became_active":"2024-02-19T14:58:27.979311+0000","last_became_peered":"2024-02-19T14:58:27.979311+0000","last_unstale":"2024-02-19T14:58:27.985694+0000","last_undegraded":"2024-02-19T14:58:27.985694+0000","last_fullsized":"2024-02-19T14:58:27.985694+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:17.590610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963794+0000","last_change":"2024-02-19T14:58:26.971962+0000","last_active":"2024-02-19T14:58:27.963794+0000","last_peered":"2024-02-19T14:58:27.963794+0000","last_clean":"2024-02-19T14:58:27.963794+0000","last_became_active":"2024-02-19T14:58:26.971694+0000","last_became_peered":"2024-02-19T14:58:26.971694+0000","last_unstale":"2024-02-19T14:58:27.963794+0000","last_undegraded":"2024-02-19T14:58:27.963794+0000","last_fullsized":"2024-02-19T14:58:27.963794+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:23.799593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990477+0000","last_change":"2024-02-19T14:58:27.984642+0000","last_active":"2024-02-19T14:58:27.990477+0000","last_peered":"2024-02-19T14:58:27.990477+0000","last_clean":"2024-02-19T14:58:27.990477+0000","last_became_active":"2024-02-19T14:58:27.984236+0000","last_became_peered":"2024-02-19T14:58:27.984236+0000","last_unstale":"2024-02-19T14:58:27.990477+0000","last_undegraded":"2024-02-19T14:58:27.990477+0000","last_fullsized":"2024-02-19T14:58:27.990477+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:56:28.335597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968288+0000","last_change":"2024-02-19T14:58:26.971509+0000","last_active":"2024-02-19T14:58:27.968288+0000","last_peered":"2024-02-19T14:58:27.968288+0000","last_clean":"2024-02-19T14:58:27.968288+0000","last_became_active":"2024-02-19T14:58:26.971237+0000","last_became_peered":"2024-02-19T14:58:26.971237+0000","last_unstale":"2024-02-19T14:58:27.968288+0000","last_undegraded":"2024-02-19T14:58:27.968288+0000","last_fullsized":"2024-02-19T14:58:27.968288+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:24:16.603636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991185+0000","last_change":"2024-02-19T14:58:27.983506+0000","last_active":"2024-02-19T14:58:27.991185+0000","last_peered":"2024-02-19T14:58:27.991185+0000","last_clean":"2024-02-19T14:58:27.991185+0000","last_became_active":"2024-02-19T14:58:27.981917+0000","last_became_peered":"2024-02-19T14:58:27.981917+0000","last_unstale":"2024-02-19T14:58:27.991185+0000","last_undegraded":"2024-02-19T14:58:27.991185+0000","last_fullsized":"2024-02-19T14:58:27.991185+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:45.110380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994490+0000","last_change":"2024-02-19T14:58:26.980620+0000","last_active":"2024-02-19T14:58:27.994490+0000","last_peered":"2024-02-19T14:58:27.994490+0000","last_clean":"2024-02-19T14:58:27.994490+0000","last_became_active":"2024-02-19T14:58:26.980332+0000","last_became_peered":"2024-02-19T14:58:26.980332+0000","last_unstale":"2024-02-19T14:58:27.994490+0000","last_undegraded":"2024-02-19T14:58:27.994490+0000","last_fullsized":"2024-02-19T14:58:27.994490+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:33.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.973969+0000","last_change":"2024-02-19T14:58:27.970234+0000","last_active":"2024-02-19T14:58:27.973969+0000","last_peered":"2024-02-19T14:58:27.973969+0000","last_clean":"2024-02-19T14:58:27.973969+0000","last_became_active":"2024-02-19T14:58:27.969889+0000","last_became_peered":"2024-02-19T14:58:27.969889+0000","last_unstale":"2024-02-19T14:58:27.973969+0000","last_undegraded":"2024-02-19T14:58:27.973969+0000","last_fullsized":"2024-02-19T14:58:27.973969+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:56.669840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970236+0000","last_change":"2024-02-19T14:58:26.975057+0000","last_active":"2024-02-19T14:58:27.970236+0000","last_peered":"2024-02-19T14:58:27.970236+0000","last_clean":"2024-02-19T14:58:27.970236+0000","last_became_active":"2024-02-19T14:58:26.974750+0000","last_became_peered":"2024-02-19T14:58:26.974750+0000","last_unstale":"2024-02-19T14:58:27.970236+0000","last_undegraded":"2024-02-19T14:58:27.970236+0000","last_fullsized":"2024-02-19T14:58:27.970236+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:02:18.872350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985787+0000","last_change":"2024-02-19T14:58:27.981675+0000","last_active":"2024-02-19T14:58:27.985787+0000","last_peered":"2024-02-19T14:58:27.985787+0000","last_clean":"2024-02-19T14:58:27.985787+0000","last_became_active":"2024-02-19T14:58:27.981244+0000","last_became_peered":"2024-02-19T14:58:27.981244+0000","last_unstale":"2024-02-19T14:58:27.985787+0000","last_undegraded":"2024-02-19T14:58:27.985787+0000","last_fullsized":"2024-02-19T14:58:27.985787+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:08:28.917033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963615+0000","last_change":"2024-02-19T14:58:26.973613+0000","last_active":"2024-02-19T14:58:27.963615+0000","last_peered":"2024-02-19T14:58:27.963615+0000","last_clean":"2024-02-19T14:58:27.963615+0000","last_became_active":"2024-02-19T14:58:26.973270+0000","last_became_peered":"2024-02-19T14:58:26.973270+0000","last_unstale":"2024-02-19T14:58:27.963615+0000","last_undegraded":"2024-02-19T14:58:27.963615+0000","last_fullsized":"2024-02-19T14:58:27.963615+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:42.841183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982515+0000","last_change":"2024-02-19T14:58:27.977846+0000","last_active":"2024-02-19T14:58:27.982515+0000","last_peered":"2024-02-19T14:58:27.982515+0000","last_clean":"2024-02-19T14:58:27.982515+0000","last_became_active":"2024-02-19T14:58:27.977567+0000","last_became_peered":"2024-02-19T14:58:27.977567+0000","last_unstale":"2024-02-19T14:58:27.982515+0000","last_undegraded":"2024-02-19T14:58:27.982515+0000","last_fullsized":"2024-02-19T14:58:27.982515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:13.282576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968637+0000","last_change":"2024-02-19T14:58:26.980994+0000","last_active":"2024-02-19T14:58:27.968637+0000","last_peered":"2024-02-19T14:58:27.968637+0000","last_clean":"2024-02-19T14:58:27.968637+0000","last_became_active":"2024-02-19T14:58:26.980700+0000","last_became_peered":"2024-02-19T14:58:26.980700+0000","last_unstale":"2024-02-19T14:58:27.968637+0000","last_undegraded":"2024-02-19T14:58:27.968637+0000","last_fullsized":"2024-02-19T14:58:27.968637+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:43.461510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986196+0000","last_change":"2024-02-19T14:58:27.977747+0000","last_active":"2024-02-19T14:58:27.986196+0000","last_peered":"2024-02-19T14:58:27.986196+0000","last_clean":"2024-02-19T14:58:27.986196+0000","last_became_active":"2024-02-19T14:58:27.977491+0000","last_became_peered":"2024-02-19T14:58:27.977491+0000","last_unstale":"2024-02-19T14:58:27.986196+0000","last_undegraded":"2024-02-19T14:58:27.986196+0000","last_fullsized":"2024-02-19T14:58:27.986196+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:26:05.731216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.967500+0000","last_change":"2024-02-19T14:58:26.974432+0000","last_active":"2024-02-19T14:58:27.967500+0000","last_peered":"2024-02-19T14:58:27.967500+0000","last_clean":"2024-02-19T14:58:27.967500+0000","last_became_active":"2024-02-19T14:58:26.974034+0000","last_became_peered":"2024-02-19T14:58:26.974034+0000","last_unstale":"2024-02-19T14:58:27.967500+0000","last_undegraded":"2024-02-19T14:58:27.967500+0000","last_fullsized":"2024-02-19T14:58:27.967500+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:55.230057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991254+0000","last_change":"2024-02-19T14:58:27.987350+0000","last_active":"2024-02-19T14:58:27.991254+0000","last_peered":"2024-02-19T14:58:27.991254+0000","last_clean":"2024-02-19T14:58:27.991254+0000","last_became_active":"2024-02-19T14:58:27.987083+0000","last_became_peered":"2024-02-19T14:58:27.987083+0000","last_unstale":"2024-02-19T14:58:27.991254+0000","last_undegraded":"2024-02-19T14:58:27.991254+0000","last_fullsized":"2024-02-19T14:58:27.991254+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:04.735200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968945+0000","last_change":"2024-02-19T14:58:26.980818+0000","last_active":"2024-02-19T14:58:27.968945+0000","last_peered":"2024-02-19T14:58:27.968945+0000","last_clean":"2024-02-19T14:58:27.968945+0000","last_became_active":"2024-02-19T14:58:26.980539+0000","last_became_peered":"2024-02-19T14:58:26.980539+0000","last_unstale":"2024-02-19T14:58:27.968945+0000","last_undegraded":"2024-02-19T14:58:27.968945+0000","last_fullsized":"2024-02-19T14:58:27.968945+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:15.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991377+0000","last_change":"2024-02-19T14:58:27.987592+0000","last_active":"2024-02-19T14:58:27.991377+0000","last_peered":"2024-02-19T14:58:27.991377+0000","last_clean":"2024-02-19T14:58:27.991377+0000","last_became_active":"2024-02-19T14:58:27.987321+0000","last_became_peered":"2024-02-19T14:58:27.987321+0000","last_unstale":"2024-02-19T14:58:27.991377+0000","last_undegraded":"2024-02-19T14:58:27.991377+0000","last_fullsized":"2024-02-19T14:58:27.991377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:17.590610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968998+0000","last_change":"2024-02-19T14:58:26.970929+0000","last_active":"2024-02-19T14:58:27.968998+0000","last_peered":"2024-02-19T14:58:27.968998+0000","last_clean":"2024-02-19T14:58:27.968998+0000","last_became_active":"2024-02-19T14:58:26.970565+0000","last_became_peered":"2024-02-19T14:58:26.970565+0000","last_unstale":"2024-02-19T14:58:27.968998+0000","last_undegraded":"2024-02-19T14:58:27.968998+0000","last_fullsized":"2024-02-19T14:58:27.968998+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:55.381270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.974008+0000","last_change":"2024-02-19T14:58:27.970712+0000","last_active":"2024-02-19T14:58:27.974008+0000","last_peered":"2024-02-19T14:58:27.974008+0000","last_clean":"2024-02-19T14:58:27.974008+0000","last_became_active":"2024-02-19T14:58:27.970436+0000","last_became_peered":"2024-02-19T14:58:27.970436+0000","last_unstale":"2024-02-19T14:58:27.974008+0000","last_undegraded":"2024-02-19T14:58:27.974008+0000","last_fullsized":"2024-02-19T14:58:27.974008+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:16.192020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970208+0000","last_change":"2024-02-19T14:58:26.974836+0000","last_active":"2024-02-19T14:58:27.970208+0000","last_peered":"2024-02-19T14:58:27.970208+0000","last_clean":"2024-02-19T14:58:27.970208+0000","last_became_active":"2024-02-19T14:58:26.974579+0000","last_became_peered":"2024-02-19T14:58:26.974579+0000","last_unstale":"2024-02-19T14:58:27.970208+0000","last_undegraded":"2024-02-19T14:58:27.970208+0000","last_fullsized":"2024-02-19T14:58:27.970208+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:01.152555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986318+0000","last_change":"2024-02-19T14:58:27.983348+0000","last_active":"2024-02-19T14:58:27.986318+0000","last_peered":"2024-02-19T14:58:27.986318+0000","last_clean":"2024-02-19T14:58:27.986318+0000","last_became_active":"2024-02-19T14:58:27.983071+0000","last_became_peered":"2024-02-19T14:58:27.983071+0000","last_unstale":"2024-02-19T14:58:27.986318+0000","last_undegraded":"2024-02-19T14:58:27.986318+0000","last_fullsized":"2024-02-19T14:58:27.986318+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:07.454563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994234+0000","last_change":"2024-02-19T14:58:26.981340+0000","last_active":"2024-02-19T14:58:27.994234+0000","last_peered":"2024-02-19T14:58:27.994234+0000","last_clean":"2024-02-19T14:58:27.994234+0000","last_became_active":"2024-02-19T14:58:26.981053+0000","last_became_peered":"2024-02-19T14:58:26.981053+0000","last_unstale":"2024-02-19T14:58:27.994234+0000","last_undegraded":"2024-02-19T14:58:27.994234+0000","last_fullsized":"2024-02-19T14:58:27.994234+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:55.662988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":457,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.971321+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:58:27.971321+0000","last_peered":"2024-02-19T14:58:27.971321+0000","last_clean":"2024-02-19T14:58:27.971321+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:58:27.971321+0000","last_undegraded":"2024-02-19T14:58:27.971321+0000","last_fullsized":"2024-02-19T14:58:27.971321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33888,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337952,"statfs":{"total":96636764160,"available":96602062848,"internally_reserved":0,"allocated":2768896,"data_stored":1655315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":0.61599999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.504,"15min":0.496},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60699999999999998,"5min":0.75900000000000001,"15min":0.79500000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32500000000000001,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"min":{"1min":0.35299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64000000000000001,"5min":0.73999999999999999,"15min":0.74099999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.501},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70799999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.623}]},{"osd":3,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67600000000000005,"5min":0.81399999999999995,"15min":0.81499999999999995},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73499999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48199999999999998,"15min":0.496},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.86899999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.497,"15min":0.53300000000000003},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51900000000000002,"15min":0.54100000000000004},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68100000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55900000000000005,"15min":0.59299999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.751},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.88500000000000001},"last":0.90500000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.48699999999999999,"15min":0.46800000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60999999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.49199999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.44}]},{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":0.72899999999999998,"15min":0.81999999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69899999999999995,"5min":0.81000000000000005,"15min":0.88100000000000001},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.40999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70099999999999996,"5min":0.78100000000000003,"15min":0.90500000000000003},"last":0.57299999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.84799999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.504,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.52800000000000002},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.76600000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.60199999999999998},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71199999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47699999999999998,"15min":0.53200000000000003},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":0.753,"15min":0.77400000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.53200000000000003,"15min":0.55200000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73499999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.37}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.52500000000000002,"15min":0.57299999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.69799999999999995,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53000000000000003,"15min":0.56200000000000006},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71499999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.49099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.372,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52200000000000002,"15min":0.505},"min":{"1min":0.36799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59599999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52100000000000002,"15min":0.496},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.872,"15min":0.872},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.29399999999999998},"max":{"1min":0.624,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54100000000000004,"15min":0.51500000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53700000000000003,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.71299999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.32400000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68400000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60099999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.48099999999999998,"15min":0.51500000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.74099999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.51800000000000002,"15min":0.55600000000000005},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.75,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.26400000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76900000000000002,"5min":0.877,"15min":0.877},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.51500000000000001,"15min":0.56299999999999994},"min":{"1min":0.24199999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.71699999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.55000000000000004,"15min":0.59199999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80300000000000005,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58899999999999997,"15min":0.621},"min":{"1min":0.35499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79200000000000004,"5min":1.03,"15min":1.03},"last":0.40300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.55600000000000005,"15min":0.54200000000000004},"min":{"1min":0.34399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.6879999999999997,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.60999999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.51900000000000002,"15min":0.52800000000000002},"min":{"1min":0.34300000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.33500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.7290000000000001,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.318,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.0429999999999999,"5min":0.63100000000000001,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.622},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.7010000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.73399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.55100000000000005,"15min":0.50700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.6989999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.60499999999999998,"15min":0.57499999999999996},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.6239999999999997,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54600000000000004,"15min":0.56499999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.315,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.621,"5min":2.621,"15min":2.621},"last":0.85399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2199552,"data_stored":1072240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.40999999999999998,"5min":3.694,"15min":3.694},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.312,"5min":0.36899999999999999,"15min":0.32500000000000001},"min":{"1min":0.21299999999999999,"5min":0.192,"15min":0.189},"max":{"1min":0.41499999999999998,"5min":4.835,"15min":4.835},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32300000000000001,"15min":0.314},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.23100000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52600000000000002,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.40500000000000003,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.24199999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.33800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70799999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.439}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79900000000000004,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81799999999999995,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.41499999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60399999999999998,"15min":0.64600000000000002},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72399999999999998,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58999999999999997,"15min":0.63800000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84799999999999998,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.40400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33400000000000002,"15min":0.317},"min":{"1min":0.20799999999999999,"5min":0.17999999999999999,"15min":0.17399999999999999},"max":{"1min":0.63600000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.31900000000000001,"15min":0.30499999999999999},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35499999999999998,"15min":0.34499999999999997},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.28199999999999997,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.66400000000000003,"5min":1.512,"15min":1.512},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.34100000000000003},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":0.55200000000000005,"5min":0.621,"15min":0.65700000000000003},"last":0.31},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67600000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.755,"5min":1.133,"15min":1.133},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.41499999999999998,"5min":0.36599999999999999,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63700000000000001,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57799999999999996,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.115,"15min":1.115},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":0.94199999999999995,"15min":1.236},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73699999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.78500000000000003,"5min":0.90200000000000002,"15min":0.995},"last":0.73499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2187264,"data_stored":1072114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.39700000000000002,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.33300000000000002},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.40100000000000002,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30599999999999999,"15min":0.309},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42999999999999999,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.251},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.36099999999999999},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.26800000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.38900000000000001},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54000000000000004,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":1.0009999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54800000000000004,"15min":0.55800000000000005},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85299999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.46600000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":0.80000000000000004,"5min":0.95499999999999996,"15min":1.026},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55100000000000005,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82399999999999995,"5min":0.996,"15min":0.996},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79500000000000004,"5min":1.036,"15min":1.036},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33892,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337948,"statfs":{"total":96636764160,"available":96602058752,"internally_reserved":0,"allocated":2768896,"data_stored":1655315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.31,"15min":0.30199999999999999},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.315,"15min":0.30499999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.33800000000000002},"min":{"1min":0.24299999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.37,"15min":0.371},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.496},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54000000000000004,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.59699999999999998},"min":{"1min":0.38500000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83699999999999997,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79800000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79000000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.81100000000000005,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:58:31.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:58:31.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.174+0000 7fb400842640 1 Processor -- start 2024-02-19T14:58:31.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.174+0000 7fb400842640 1 -- start start 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb400842640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb3fc152700 0x7fb3fc0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb400842640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc1551f0 con 0x7fb3fc0c5730 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb400842640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc155490 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb400842640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc155770 con 0x7fb3fc152700 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb3fa59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb3fc152700 0x7fb3fc0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb3f9d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb3fc0c5730 0x7fb3fc06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47218/0 (socket says 172.21.15.78:47218) 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb3f9d9b640 1 -- 172.21.15.78:0/1665723176 learned_addr learned my addr 172.21.15.78:0/1665723176 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb3fa59c640 1 -- 172.21.15.78:0/1665723176 >> v1:172.21.15.150:6789/0 conn(0x7fb3fc0c8e00 legacy=0x7fb3fc15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb3fa59c640 1 -- 172.21.15.78:0/1665723176 >> v1:172.21.15.78:6789/0 conn(0x7fb3fc0c5730 legacy=0x7fb3fc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.175+0000 7fb3fa59c640 1 -- 172.21.15.78:0/1665723176 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc155a50 con 0x7fb3fc152700 2024-02-19T14:58:31.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.176+0000 7fb3fa59c640 1 --2- 172.21.15.78:0/1665723176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc152700 0x7fb3fc0c51f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009d90 tx=0x7fb3f402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bb50d11a80f550 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:31.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.176+0000 7fb3f959a640 1 -- 172.21.15.78:0/1665723176 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f402cbd0 con 0x7fb3fc152700 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.176+0000 7fb3f959a640 1 -- 172.21.15.78:0/1665723176 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f402cd30 con 0x7fb3fc152700 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.176+0000 7fb3f959a640 1 -- 172.21.15.78:0/1665723176 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f402cea0 con 0x7fb3fc152700 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 -- 172.21.15.78:0/1665723176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc152700 msgr2=0x7fb3fc0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 --2- 172.21.15.78:0/1665723176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc152700 0x7fb3fc0c51f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009d90 tx=0x7fb3f402be20 comp rx=0 tx=0).stop 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 -- 172.21.15.78:0/1665723176 shutdown_connections 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 --2- 172.21.15.78:0/1665723176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc152700 0x7fb3fc0c51f0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 -- 172.21.15.78:0/1665723176 >> 172.21.15.78:0/1665723176 conn(0x7fb3fc0cac90 msgr2=0x7fb3fc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 -- 172.21.15.78:0/1665723176 shutdown_connections 2024-02-19T14:58:31.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.177+0000 7fb400842640 1 -- 172.21.15.78:0/1665723176 wait complete. 2024-02-19T14:58:31.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.178+0000 7fb400842640 1 Processor -- start 2024-02-19T14:58:31.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.178+0000 7fb400842640 1 -- start start 2024-02-19T14:58:31.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3fc0c5730 0x7fb3fc0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 0x7fb3fc1ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc152700 0x7fb3fc1ec740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1eee50 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1ec2b0 con 0x7fb3fc152700 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0c3620 con 0x7fb3fc0c5730 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 0x7fb3fc1ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 0x7fb3fc1ee0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58628/0 (socket says 172.21.15.78:58628) 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 -- 172.21.15.78:0/3388736496 learned_addr learned my addr 172.21.15.78:0/3388736496 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3fa59c640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3fc0c5730 0x7fb3fc0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3fad9d640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc152700 0x7fb3fc1ec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 -- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3fc0c5730 msgr2=0x7fb3fc0c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3fc0c5730 0x7fb3fc0c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 -- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc152700 msgr2=0x7fb3fc1ec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc152700 0x7fb3fc1ec740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 -- 172.21.15.78:0/3388736496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4009a70 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f9d9b640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 0x7fb3fc1ee0e0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00c400 tx=0x7fb3ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec005ce0 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec005e40 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc1eeb70 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec0072f0 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.179+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc1ee620 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.180+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc1ebda0 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.181+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb3ec007450 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.181+0000 7fb3f2ffd640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3dc06b840 0x7fb3dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.182+0000 7fb3fa59c640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3dc06b840 0x7fb3dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.182+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb3ec0898b0 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.182+0000 7fb3fa59c640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3dc06b840 0x7fb3dc06dd00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4036000 tx=0x7fb3f4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:31.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.186+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb3fc1ebda0 con 0x7fb3fc0c8e00 2024-02-19T14:58:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.418+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3fc155f70 con 0x7fb3dc06b840 2024-02-19T14:58:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.426+0000 7fb3f2ffd640 1 -- 172.21.15.78:0/3388736496 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290001 (secure 0 0 0) 0x7fb3f402caf0 con 0x7fb3dc06b840 2024-02-19T14:58:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:58:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3dc06b840 msgr2=0x7fb3dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3dc06b840 0x7fb3dc06dd00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4036000 tx=0x7fb3f4030000 comp rx=0 tx=0).stop 2024-02-19T14:58:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 msgr2=0x7fb3fc1ee0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 0x7fb3fc1ee0e0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00c400 tx=0x7fb3ec00c8c0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 shutdown_connections 2024-02-19T14:58:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3dc06b840 0x7fb3dc06dd00 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3fc0c8e00 0x7fb3fc1ee0e0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc152700 0x7fb3fc1ec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 --2- 172.21.15.78:0/3388736496 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3fc0c5730 0x7fb3fc0c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 >> 172.21.15.78:0/3388736496 conn(0x7fb3fc0cac90 msgr2=0x7fb3fc0c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 shutdown_connections 2024-02-19T14:58:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.429+0000 7fb400842640 1 -- 172.21.15.78:0/3388736496 wait complete. 2024-02-19T14:58:31.442 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-02-19T14:58:30.136051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267672,"kb_used_data":18188,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754707048,"statfs":{"total":773094113280,"available":772820017152,"internally_reserved":0,"allocated":18624512,"data_stored":9714772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.182050"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986814+0000","last_change":"2024-02-19T14:58:27.984713+0000","last_active":"2024-02-19T14:58:27.986814+0000","last_peered":"2024-02-19T14:58:27.986814+0000","last_clean":"2024-02-19T14:58:27.986814+0000","last_became_active":"2024-02-19T14:58:27.984466+0000","last_became_peered":"2024-02-19T14:58:27.984466+0000","last_unstale":"2024-02-19T14:58:27.986814+0000","last_undegraded":"2024-02-19T14:58:27.986814+0000","last_fullsized":"2024-02-19T14:58:27.986814+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:23.147062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.983305+0000","last_change":"2024-02-19T14:58:27.980688+0000","last_active":"2024-02-19T14:58:27.983305+0000","last_peered":"2024-02-19T14:58:27.983305+0000","last_clean":"2024-02-19T14:58:27.983305+0000","last_became_active":"2024-02-19T14:58:27.980399+0000","last_became_peered":"2024-02-19T14:58:27.980399+0000","last_unstale":"2024-02-19T14:58:27.983305+0000","last_undegraded":"2024-02-19T14:58:27.983305+0000","last_fullsized":"2024-02-19T14:58:27.983305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:10.635784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.979029+0000","last_change":"2024-02-19T14:58:27.974855+0000","last_active":"2024-02-19T14:58:27.979029+0000","last_peered":"2024-02-19T14:58:27.979029+0000","last_clean":"2024-02-19T14:58:27.979029+0000","last_became_active":"2024-02-19T14:58:27.974583+0000","last_became_peered":"2024-02-19T14:58:27.974583+0000","last_unstale":"2024-02-19T14:58:27.979029+0000","last_undegraded":"2024-02-19T14:58:27.979029+0000","last_fullsized":"2024-02-19T14:58:27.979029+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:37.236794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991371+0000","last_change":"2024-02-19T14:58:27.983684+0000","last_active":"2024-02-19T14:58:27.991371+0000","last_peered":"2024-02-19T14:58:27.991371+0000","last_clean":"2024-02-19T14:58:27.991371+0000","last_became_active":"2024-02-19T14:58:27.983447+0000","last_became_peered":"2024-02-19T14:58:27.983447+0000","last_unstale":"2024-02-19T14:58:27.991371+0000","last_undegraded":"2024-02-19T14:58:27.991371+0000","last_fullsized":"2024-02-19T14:58:27.991371+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:21.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986019+0000","last_change":"2024-02-19T14:58:27.982626+0000","last_active":"2024-02-19T14:58:27.986019+0000","last_peered":"2024-02-19T14:58:27.986019+0000","last_clean":"2024-02-19T14:58:27.986019+0000","last_became_active":"2024-02-19T14:58:27.979746+0000","last_became_peered":"2024-02-19T14:58:27.979746+0000","last_unstale":"2024-02-19T14:58:27.986019+0000","last_undegraded":"2024-02-19T14:58:27.986019+0000","last_fullsized":"2024-02-19T14:58:27.986019+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:35:18.279287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.980184+0000","last_change":"2024-02-19T14:58:27.977180+0000","last_active":"2024-02-19T14:58:27.980184+0000","last_peered":"2024-02-19T14:58:27.980184+0000","last_clean":"2024-02-19T14:58:27.980184+0000","last_became_active":"2024-02-19T14:58:27.976920+0000","last_became_peered":"2024-02-19T14:58:27.976920+0000","last_unstale":"2024-02-19T14:58:27.980184+0000","last_undegraded":"2024-02-19T14:58:27.980184+0000","last_fullsized":"2024-02-19T14:58:27.980184+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:56.669840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.984762+0000","last_change":"2024-02-19T14:58:27.978679+0000","last_active":"2024-02-19T14:58:27.984762+0000","last_peered":"2024-02-19T14:58:27.984762+0000","last_clean":"2024-02-19T14:58:27.984762+0000","last_became_active":"2024-02-19T14:58:27.978379+0000","last_became_peered":"2024-02-19T14:58:27.978379+0000","last_unstale":"2024-02-19T14:58:27.984762+0000","last_undegraded":"2024-02-19T14:58:27.984762+0000","last_fullsized":"2024-02-19T14:58:27.984762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:42.779035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.983836+0000","last_change":"2024-02-19T14:58:27.972055+0000","last_active":"2024-02-19T14:58:27.983836+0000","last_peered":"2024-02-19T14:58:27.983836+0000","last_clean":"2024-02-19T14:58:27.983836+0000","last_became_active":"2024-02-19T14:58:27.971799+0000","last_became_peered":"2024-02-19T14:58:27.971799+0000","last_unstale":"2024-02-19T14:58:27.983836+0000","last_undegraded":"2024-02-19T14:58:27.983836+0000","last_fullsized":"2024-02-19T14:58:27.983836+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:37:32.242539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993301+0000","last_change":"2024-02-19T14:58:27.980450+0000","last_active":"2024-02-19T14:58:27.993301+0000","last_peered":"2024-02-19T14:58:27.993301+0000","last_clean":"2024-02-19T14:58:27.993301+0000","last_became_active":"2024-02-19T14:58:27.980184+0000","last_became_peered":"2024-02-19T14:58:27.980184+0000","last_unstale":"2024-02-19T14:58:27.993301+0000","last_undegraded":"2024-02-19T14:58:27.993301+0000","last_fullsized":"2024-02-19T14:58:27.993301+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:39.867110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987515+0000","last_change":"2024-02-19T14:58:27.981112+0000","last_active":"2024-02-19T14:58:27.987515+0000","last_peered":"2024-02-19T14:58:27.987515+0000","last_clean":"2024-02-19T14:58:27.987515+0000","last_became_active":"2024-02-19T14:58:27.980837+0000","last_became_peered":"2024-02-19T14:58:27.980837+0000","last_unstale":"2024-02-19T14:58:27.987515+0000","last_undegraded":"2024-02-19T14:58:27.987515+0000","last_fullsized":"2024-02-19T14:58:27.987515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:24.806445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.976000+0000","last_change":"2024-02-19T14:58:27.972580+0000","last_active":"2024-02-19T14:58:27.976000+0000","last_peered":"2024-02-19T14:58:27.976000+0000","last_clean":"2024-02-19T14:58:27.976000+0000","last_became_active":"2024-02-19T14:58:27.971810+0000","last_became_peered":"2024-02-19T14:58:27.971810+0000","last_unstale":"2024-02-19T14:58:27.976000+0000","last_undegraded":"2024-02-19T14:58:27.976000+0000","last_fullsized":"2024-02-19T14:58:27.976000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:57.589892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987456+0000","last_change":"2024-02-19T14:58:27.983884+0000","last_active":"2024-02-19T14:58:27.987456+0000","last_peered":"2024-02-19T14:58:27.987456+0000","last_clean":"2024-02-19T14:58:27.987456+0000","last_became_active":"2024-02-19T14:58:27.983599+0000","last_became_peered":"2024-02-19T14:58:27.983599+0000","last_unstale":"2024-02-19T14:58:27.987456+0000","last_undegraded":"2024-02-19T14:58:27.987456+0000","last_fullsized":"2024-02-19T14:58:27.987456+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:27:34.549285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991403+0000","last_change":"2024-02-19T14:58:27.987638+0000","last_active":"2024-02-19T14:58:27.991403+0000","last_peered":"2024-02-19T14:58:27.991403+0000","last_clean":"2024-02-19T14:58:27.991403+0000","last_became_active":"2024-02-19T14:58:27.987382+0000","last_became_peered":"2024-02-19T14:58:27.987382+0000","last_unstale":"2024-02-19T14:58:27.991403+0000","last_undegraded":"2024-02-19T14:58:27.991403+0000","last_fullsized":"2024-02-19T14:58:27.991403+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:35:39.867110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993376+0000","last_change":"2024-02-19T14:58:27.987349+0000","last_active":"2024-02-19T14:58:27.993376+0000","last_peered":"2024-02-19T14:58:27.993376+0000","last_clean":"2024-02-19T14:58:27.993376+0000","last_became_active":"2024-02-19T14:58:27.987071+0000","last_became_peered":"2024-02-19T14:58:27.987071+0000","last_unstale":"2024-02-19T14:58:27.993376+0000","last_undegraded":"2024-02-19T14:58:27.993376+0000","last_fullsized":"2024-02-19T14:58:27.993376+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:16.168858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.977332+0000","last_change":"2024-02-19T14:58:27.974824+0000","last_active":"2024-02-19T14:58:27.977332+0000","last_peered":"2024-02-19T14:58:27.977332+0000","last_clean":"2024-02-19T14:58:27.977332+0000","last_became_active":"2024-02-19T14:58:27.973793+0000","last_became_peered":"2024-02-19T14:58:27.973793+0000","last_unstale":"2024-02-19T14:58:27.977332+0000","last_undegraded":"2024-02-19T14:58:27.977332+0000","last_fullsized":"2024-02-19T14:58:27.977332+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:17.607381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.981803+0000","last_change":"2024-02-19T14:58:27.976545+0000","last_active":"2024-02-19T14:58:27.981803+0000","last_peered":"2024-02-19T14:58:27.981803+0000","last_clean":"2024-02-19T14:58:27.981803+0000","last_became_active":"2024-02-19T14:58:27.976280+0000","last_became_peered":"2024-02-19T14:58:27.976280+0000","last_unstale":"2024-02-19T14:58:27.981803+0000","last_undegraded":"2024-02-19T14:58:27.981803+0000","last_fullsized":"2024-02-19T14:58:27.981803+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:04.735200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993283+0000","last_change":"2024-02-19T14:58:27.978262+0000","last_active":"2024-02-19T14:58:27.993283+0000","last_peered":"2024-02-19T14:58:27.993283+0000","last_clean":"2024-02-19T14:58:27.993283+0000","last_became_active":"2024-02-19T14:58:27.977737+0000","last_became_peered":"2024-02-19T14:58:27.977737+0000","last_unstale":"2024-02-19T14:58:27.993283+0000","last_undegraded":"2024-02-19T14:58:27.993283+0000","last_fullsized":"2024-02-19T14:58:27.993283+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:04.735200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.979741+0000","last_change":"2024-02-19T14:58:27.975054+0000","last_active":"2024-02-19T14:58:27.979741+0000","last_peered":"2024-02-19T14:58:27.979741+0000","last_clean":"2024-02-19T14:58:27.979741+0000","last_became_active":"2024-02-19T14:58:27.974789+0000","last_became_peered":"2024-02-19T14:58:27.974789+0000","last_unstale":"2024-02-19T14:58:27.979741+0000","last_undegraded":"2024-02-19T14:58:27.979741+0000","last_fullsized":"2024-02-19T14:58:27.979741+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:57.589892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990311+0000","last_change":"2024-02-19T14:58:27.984722+0000","last_active":"2024-02-19T14:58:27.990311+0000","last_peered":"2024-02-19T14:58:27.990311+0000","last_clean":"2024-02-19T14:58:27.990311+0000","last_became_active":"2024-02-19T14:58:27.984462+0000","last_became_peered":"2024-02-19T14:58:27.984462+0000","last_unstale":"2024-02-19T14:58:27.990311+0000","last_undegraded":"2024-02-19T14:58:27.990311+0000","last_fullsized":"2024-02-19T14:58:27.990311+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:30.360740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986173+0000","last_change":"2024-02-19T14:58:27.977158+0000","last_active":"2024-02-19T14:58:27.986173+0000","last_peered":"2024-02-19T14:58:27.986173+0000","last_clean":"2024-02-19T14:58:27.986173+0000","last_became_active":"2024-02-19T14:58:27.976915+0000","last_became_peered":"2024-02-19T14:58:27.976915+0000","last_unstale":"2024-02-19T14:58:27.986173+0000","last_undegraded":"2024-02-19T14:58:27.986173+0000","last_fullsized":"2024-02-19T14:58:27.986173+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:16.045644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986979+0000","last_change":"2024-02-19T14:58:27.980486+0000","last_active":"2024-02-19T14:58:27.986979+0000","last_peered":"2024-02-19T14:58:27.986979+0000","last_clean":"2024-02-19T14:58:27.986979+0000","last_became_active":"2024-02-19T14:58:27.980243+0000","last_became_peered":"2024-02-19T14:58:27.980243+0000","last_unstale":"2024-02-19T14:58:27.986979+0000","last_undegraded":"2024-02-19T14:58:27.986979+0000","last_fullsized":"2024-02-19T14:58:27.986979+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:43.848035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987240+0000","last_change":"2024-02-19T14:58:27.978357+0000","last_active":"2024-02-19T14:58:27.987240+0000","last_peered":"2024-02-19T14:58:27.987240+0000","last_clean":"2024-02-19T14:58:27.987240+0000","last_became_active":"2024-02-19T14:58:27.978112+0000","last_became_peered":"2024-02-19T14:58:27.978112+0000","last_unstale":"2024-02-19T14:58:27.987240+0000","last_undegraded":"2024-02-19T14:58:27.987240+0000","last_fullsized":"2024-02-19T14:58:27.987240+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:13.596506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993427+0000","last_change":"2024-02-19T14:58:27.990692+0000","last_active":"2024-02-19T14:58:27.993427+0000","last_peered":"2024-02-19T14:58:27.993427+0000","last_clean":"2024-02-19T14:58:27.993427+0000","last_became_active":"2024-02-19T14:58:27.990348+0000","last_became_peered":"2024-02-19T14:58:27.990348+0000","last_unstale":"2024-02-19T14:58:27.993427+0000","last_undegraded":"2024-02-19T14:58:27.993427+0000","last_fullsized":"2024-02-19T14:58:27.993427+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:41:31.500536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.984579+0000","last_change":"2024-02-19T14:58:27.977731+0000","last_active":"2024-02-19T14:58:27.984579+0000","last_peered":"2024-02-19T14:58:27.984579+0000","last_clean":"2024-02-19T14:58:27.984579+0000","last_became_active":"2024-02-19T14:58:27.977460+0000","last_became_peered":"2024-02-19T14:58:27.977460+0000","last_unstale":"2024-02-19T14:58:27.984579+0000","last_undegraded":"2024-02-19T14:58:27.984579+0000","last_fullsized":"2024-02-19T14:58:27.984579+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:37:05.556447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.977917+0000","last_change":"2024-02-19T14:58:27.972536+0000","last_active":"2024-02-19T14:58:27.977917+0000","last_peered":"2024-02-19T14:58:27.977917+0000","last_clean":"2024-02-19T14:58:27.977917+0000","last_became_active":"2024-02-19T14:58:27.972278+0000","last_became_peered":"2024-02-19T14:58:27.972278+0000","last_unstale":"2024-02-19T14:58:27.977917+0000","last_undegraded":"2024-02-19T14:58:27.977917+0000","last_fullsized":"2024-02-19T14:58:27.977917+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:21.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986031+0000","last_change":"2024-02-19T14:58:27.983244+0000","last_active":"2024-02-19T14:58:27.986031+0000","last_peered":"2024-02-19T14:58:27.986031+0000","last_clean":"2024-02-19T14:58:27.986031+0000","last_became_active":"2024-02-19T14:58:27.982987+0000","last_became_peered":"2024-02-19T14:58:27.982987+0000","last_unstale":"2024-02-19T14:58:27.986031+0000","last_undegraded":"2024-02-19T14:58:27.986031+0000","last_fullsized":"2024-02-19T14:58:27.986031+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:54:31.439375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.975957+0000","last_change":"2024-02-19T14:58:27.971575+0000","last_active":"2024-02-19T14:58:27.975957+0000","last_peered":"2024-02-19T14:58:27.975957+0000","last_clean":"2024-02-19T14:58:27.975957+0000","last_became_active":"2024-02-19T14:58:27.971053+0000","last_became_peered":"2024-02-19T14:58:27.971053+0000","last_unstale":"2024-02-19T14:58:27.975957+0000","last_undegraded":"2024-02-19T14:58:27.975957+0000","last_fullsized":"2024-02-19T14:58:27.975957+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:50.431913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987479+0000","last_change":"2024-02-19T14:58:27.984450+0000","last_active":"2024-02-19T14:58:27.987479+0000","last_peered":"2024-02-19T14:58:27.987479+0000","last_clean":"2024-02-19T14:58:27.987479+0000","last_became_active":"2024-02-19T14:58:27.983567+0000","last_became_peered":"2024-02-19T14:58:27.983567+0000","last_unstale":"2024-02-19T14:58:27.987479+0000","last_undegraded":"2024-02-19T14:58:27.987479+0000","last_fullsized":"2024-02-19T14:58:27.987479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:55.124259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.977453+0000","last_change":"2024-02-19T14:58:27.970961+0000","last_active":"2024-02-19T14:58:27.977453+0000","last_peered":"2024-02-19T14:58:27.977453+0000","last_clean":"2024-02-19T14:58:27.977453+0000","last_became_active":"2024-02-19T14:58:27.970689+0000","last_became_peered":"2024-02-19T14:58:27.970689+0000","last_unstale":"2024-02-19T14:58:27.977453+0000","last_undegraded":"2024-02-19T14:58:27.977453+0000","last_fullsized":"2024-02-19T14:58:27.977453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:50.431913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986619+0000","last_change":"2024-02-19T14:58:27.979671+0000","last_active":"2024-02-19T14:58:27.986619+0000","last_peered":"2024-02-19T14:58:27.986619+0000","last_clean":"2024-02-19T14:58:27.986619+0000","last_became_active":"2024-02-19T14:58:27.979417+0000","last_became_peered":"2024-02-19T14:58:27.979417+0000","last_unstale":"2024-02-19T14:58:27.986619+0000","last_undegraded":"2024-02-19T14:58:27.986619+0000","last_fullsized":"2024-02-19T14:58:27.986619+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:36.835628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982305+0000","last_change":"2024-02-19T14:58:27.979641+0000","last_active":"2024-02-19T14:58:27.982305+0000","last_peered":"2024-02-19T14:58:27.982305+0000","last_clean":"2024-02-19T14:58:27.982305+0000","last_became_active":"2024-02-19T14:58:27.979387+0000","last_became_peered":"2024-02-19T14:58:27.979387+0000","last_unstale":"2024-02-19T14:58:27.982305+0000","last_undegraded":"2024-02-19T14:58:27.982305+0000","last_fullsized":"2024-02-19T14:58:27.982305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:16.192020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985394+0000","last_change":"2024-02-19T14:58:27.981496+0000","last_active":"2024-02-19T14:58:27.985394+0000","last_peered":"2024-02-19T14:58:27.985394+0000","last_clean":"2024-02-19T14:58:27.985394+0000","last_became_active":"2024-02-19T14:58:27.981167+0000","last_became_peered":"2024-02-19T14:58:27.981167+0000","last_unstale":"2024-02-19T14:58:27.985394+0000","last_undegraded":"2024-02-19T14:58:27.985394+0000","last_fullsized":"2024-02-19T14:58:27.985394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:14:50.431913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993388+0000","last_change":"2024-02-19T14:58:27.989844+0000","last_active":"2024-02-19T14:58:27.993388+0000","last_peered":"2024-02-19T14:58:27.993388+0000","last_clean":"2024-02-19T14:58:27.993388+0000","last_became_active":"2024-02-19T14:58:27.989554+0000","last_became_peered":"2024-02-19T14:58:27.989554+0000","last_unstale":"2024-02-19T14:58:27.993388+0000","last_undegraded":"2024-02-19T14:58:27.993388+0000","last_fullsized":"2024-02-19T14:58:27.993388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:30.332650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"173'4","reported_seq":16,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993808+0000","last_change":"2024-02-19T14:58:26.972658+0000","last_active":"2024-02-19T14:58:27.993808+0000","last_peered":"2024-02-19T14:58:27.993808+0000","last_clean":"2024-02-19T14:58:27.993808+0000","last_became_active":"2024-02-19T14:58:26.972378+0000","last_became_peered":"2024-02-19T14:58:26.972378+0000","last_unstale":"2024-02-19T14:58:27.993808+0000","last_undegraded":"2024-02-19T14:58:27.993808+0000","last_fullsized":"2024-02-19T14:58:27.993808+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:30:12.589654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982543+0000","last_change":"2024-02-19T14:58:27.978079+0000","last_active":"2024-02-19T14:58:27.982543+0000","last_peered":"2024-02-19T14:58:27.982543+0000","last_clean":"2024-02-19T14:58:27.982543+0000","last_became_active":"2024-02-19T14:58:27.977825+0000","last_became_peered":"2024-02-19T14:58:27.977825+0000","last_unstale":"2024-02-19T14:58:27.982543+0000","last_undegraded":"2024-02-19T14:58:27.982543+0000","last_fullsized":"2024-02-19T14:58:27.982543+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:05:57.572536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994141+0000","last_change":"2024-02-19T14:58:26.975494+0000","last_active":"2024-02-19T14:58:27.994141+0000","last_peered":"2024-02-19T14:58:27.994141+0000","last_clean":"2024-02-19T14:58:27.994141+0000","last_became_active":"2024-02-19T14:58:26.975180+0000","last_became_peered":"2024-02-19T14:58:26.975180+0000","last_unstale":"2024-02-19T14:58:27.994141+0000","last_undegraded":"2024-02-19T14:58:27.994141+0000","last_fullsized":"2024-02-19T14:58:27.994141+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:25:58.712820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990229+0000","last_change":"2024-02-19T14:58:27.981826+0000","last_active":"2024-02-19T14:58:27.990229+0000","last_peered":"2024-02-19T14:58:27.990229+0000","last_clean":"2024-02-19T14:58:27.990229+0000","last_became_active":"2024-02-19T14:58:27.981551+0000","last_became_peered":"2024-02-19T14:58:27.981551+0000","last_unstale":"2024-02-19T14:58:27.990229+0000","last_undegraded":"2024-02-19T14:58:27.990229+0000","last_fullsized":"2024-02-19T14:58:27.990229+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:36:17.607381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991533+0000","last_change":"2024-02-19T14:58:26.971321+0000","last_active":"2024-02-19T14:58:27.991533+0000","last_peered":"2024-02-19T14:58:27.991533+0000","last_clean":"2024-02-19T14:58:27.991533+0000","last_became_active":"2024-02-19T14:58:26.971009+0000","last_became_peered":"2024-02-19T14:58:26.971009+0000","last_unstale":"2024-02-19T14:58:27.991533+0000","last_undegraded":"2024-02-19T14:58:27.991533+0000","last_fullsized":"2024-02-19T14:58:27.991533+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:48:20.276462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985995+0000","last_change":"2024-02-19T14:58:27.982177+0000","last_active":"2024-02-19T14:58:27.985995+0000","last_peered":"2024-02-19T14:58:27.985995+0000","last_clean":"2024-02-19T14:58:27.985995+0000","last_became_active":"2024-02-19T14:58:27.979467+0000","last_became_peered":"2024-02-19T14:58:27.979467+0000","last_unstale":"2024-02-19T14:58:27.985995+0000","last_undegraded":"2024-02-19T14:58:27.985995+0000","last_fullsized":"2024-02-19T14:58:27.985995+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:27.855566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993817+0000","last_change":"2024-02-19T14:58:26.977908+0000","last_active":"2024-02-19T14:58:27.993817+0000","last_peered":"2024-02-19T14:58:27.993817+0000","last_clean":"2024-02-19T14:58:27.993817+0000","last_became_active":"2024-02-19T14:58:26.977679+0000","last_became_peered":"2024-02-19T14:58:26.977679+0000","last_unstale":"2024-02-19T14:58:27.993817+0000","last_undegraded":"2024-02-19T14:58:27.993817+0000","last_fullsized":"2024-02-19T14:58:27.993817+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:06.447711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990262+0000","last_change":"2024-02-19T14:58:27.983534+0000","last_active":"2024-02-19T14:58:27.990262+0000","last_peered":"2024-02-19T14:58:27.990262+0000","last_clean":"2024-02-19T14:58:27.990262+0000","last_became_active":"2024-02-19T14:58:27.983253+0000","last_became_peered":"2024-02-19T14:58:27.983253+0000","last_unstale":"2024-02-19T14:58:27.990262+0000","last_undegraded":"2024-02-19T14:58:27.990262+0000","last_fullsized":"2024-02-19T14:58:27.990262+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:46:30.360740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.992225+0000","last_change":"2024-02-19T14:58:26.980151+0000","last_active":"2024-02-19T14:58:27.992225+0000","last_peered":"2024-02-19T14:58:27.992225+0000","last_clean":"2024-02-19T14:58:27.992225+0000","last_became_active":"2024-02-19T14:58:26.979862+0000","last_became_peered":"2024-02-19T14:58:26.979862+0000","last_unstale":"2024-02-19T14:58:27.992225+0000","last_undegraded":"2024-02-19T14:58:27.992225+0000","last_fullsized":"2024-02-19T14:58:27.992225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:54:30.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991283+0000","last_change":"2024-02-19T14:58:27.981918+0000","last_active":"2024-02-19T14:58:27.991283+0000","last_peered":"2024-02-19T14:58:27.991283+0000","last_clean":"2024-02-19T14:58:27.991283+0000","last_became_active":"2024-02-19T14:58:27.981648+0000","last_became_peered":"2024-02-19T14:58:27.981648+0000","last_unstale":"2024-02-19T14:58:27.991283+0000","last_undegraded":"2024-02-19T14:58:27.991283+0000","last_fullsized":"2024-02-19T14:58:27.991283+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:56:28.335597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970924+0000","last_change":"2024-02-19T14:58:26.980109+0000","last_active":"2024-02-19T14:58:27.970924+0000","last_peered":"2024-02-19T14:58:27.970924+0000","last_clean":"2024-02-19T14:58:27.970924+0000","last_became_active":"2024-02-19T14:58:26.979636+0000","last_became_peered":"2024-02-19T14:58:26.979636+0000","last_unstale":"2024-02-19T14:58:27.970924+0000","last_undegraded":"2024-02-19T14:58:27.970924+0000","last_fullsized":"2024-02-19T14:58:27.970924+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:33.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993260+0000","last_change":"2024-02-19T14:58:27.978037+0000","last_active":"2024-02-19T14:58:27.993260+0000","last_peered":"2024-02-19T14:58:27.993260+0000","last_clean":"2024-02-19T14:58:27.993260+0000","last_became_active":"2024-02-19T14:58:27.972931+0000","last_became_peered":"2024-02-19T14:58:27.972931+0000","last_unstale":"2024-02-19T14:58:27.993260+0000","last_undegraded":"2024-02-19T14:58:27.993260+0000","last_fullsized":"2024-02-19T14:58:27.993260+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:37.236794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963324+0000","last_change":"2024-02-19T14:58:26.974223+0000","last_active":"2024-02-19T14:58:27.963324+0000","last_peered":"2024-02-19T14:58:27.963324+0000","last_clean":"2024-02-19T14:58:27.963324+0000","last_became_active":"2024-02-19T14:58:26.973934+0000","last_became_peered":"2024-02-19T14:58:26.973934+0000","last_unstale":"2024-02-19T14:58:27.963324+0000","last_undegraded":"2024-02-19T14:58:27.963324+0000","last_fullsized":"2024-02-19T14:58:27.963324+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:33.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.980381+0000","last_change":"2024-02-19T14:58:27.975944+0000","last_active":"2024-02-19T14:58:27.980381+0000","last_peered":"2024-02-19T14:58:27.980381+0000","last_clean":"2024-02-19T14:58:27.980381+0000","last_became_active":"2024-02-19T14:58:27.975686+0000","last_became_peered":"2024-02-19T14:58:27.975686+0000","last_unstale":"2024-02-19T14:58:27.980381+0000","last_undegraded":"2024-02-19T14:58:27.980381+0000","last_fullsized":"2024-02-19T14:58:27.980381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:45.110380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.973636+0000","last_change":"2024-02-19T14:58:26.973275+0000","last_active":"2024-02-19T14:58:27.973636+0000","last_peered":"2024-02-19T14:58:27.973636+0000","last_clean":"2024-02-19T14:58:27.973636+0000","last_became_active":"2024-02-19T14:58:26.972810+0000","last_became_peered":"2024-02-19T14:58:26.972810+0000","last_unstale":"2024-02-19T14:58:27.973636+0000","last_undegraded":"2024-02-19T14:58:27.973636+0000","last_fullsized":"2024-02-19T14:58:27.973636+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:45.169903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.987523+0000","last_change":"2024-02-19T14:58:27.984347+0000","last_active":"2024-02-19T14:58:27.987523+0000","last_peered":"2024-02-19T14:58:27.987523+0000","last_clean":"2024-02-19T14:58:27.987523+0000","last_became_active":"2024-02-19T14:58:27.984065+0000","last_became_peered":"2024-02-19T14:58:27.984065+0000","last_unstale":"2024-02-19T14:58:27.987523+0000","last_undegraded":"2024-02-19T14:58:27.987523+0000","last_fullsized":"2024-02-19T14:58:27.987523+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:17.590610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970859+0000","last_change":"2024-02-19T14:58:26.980157+0000","last_active":"2024-02-19T14:58:27.970859+0000","last_peered":"2024-02-19T14:58:27.970859+0000","last_clean":"2024-02-19T14:58:27.970859+0000","last_became_active":"2024-02-19T14:58:26.979578+0000","last_became_peered":"2024-02-19T14:58:26.979578+0000","last_unstale":"2024-02-19T14:58:27.970859+0000","last_undegraded":"2024-02-19T14:58:27.970859+0000","last_fullsized":"2024-02-19T14:58:27.970859+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:22.140210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985762+0000","last_change":"2024-02-19T14:58:27.981480+0000","last_active":"2024-02-19T14:58:27.985762+0000","last_peered":"2024-02-19T14:58:27.985762+0000","last_clean":"2024-02-19T14:58:27.985762+0000","last_became_active":"2024-02-19T14:58:27.980916+0000","last_became_peered":"2024-02-19T14:58:27.980916+0000","last_unstale":"2024-02-19T14:58:27.985762+0000","last_undegraded":"2024-02-19T14:58:27.985762+0000","last_fullsized":"2024-02-19T14:58:27.985762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:28:49.281422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"173'3","reported_seq":15,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994243+0000","last_change":"2024-02-19T14:58:26.975674+0000","last_active":"2024-02-19T14:58:27.994243+0000","last_peered":"2024-02-19T14:58:27.994243+0000","last_clean":"2024-02-19T14:58:27.994243+0000","last_became_active":"2024-02-19T14:58:26.975246+0000","last_became_peered":"2024-02-19T14:58:26.975246+0000","last_unstale":"2024-02-19T14:58:27.994243+0000","last_undegraded":"2024-02-19T14:58:27.994243+0000","last_fullsized":"2024-02-19T14:58:27.994243+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:22.140210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.989316+0000","last_change":"2024-02-19T14:58:27.986141+0000","last_active":"2024-02-19T14:58:27.989316+0000","last_peered":"2024-02-19T14:58:27.989316+0000","last_clean":"2024-02-19T14:58:27.989316+0000","last_became_active":"2024-02-19T14:58:27.985792+0000","last_became_peered":"2024-02-19T14:58:27.985792+0000","last_unstale":"2024-02-19T14:58:27.989316+0000","last_undegraded":"2024-02-19T14:58:27.989316+0000","last_fullsized":"2024-02-19T14:58:27.989316+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:56.797501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"173'6","reported_seq":18,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.996662+0000","last_change":"2024-02-19T14:58:26.977763+0000","last_active":"2024-02-19T14:58:27.996662+0000","last_peered":"2024-02-19T14:58:27.996662+0000","last_clean":"2024-02-19T14:58:27.996662+0000","last_became_active":"2024-02-19T14:58:26.977441+0000","last_became_peered":"2024-02-19T14:58:26.977441+0000","last_unstale":"2024-02-19T14:58:27.996662+0000","last_undegraded":"2024-02-19T14:58:27.996662+0000","last_fullsized":"2024-02-19T14:58:27.996662+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:06.447711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993412+0000","last_change":"2024-02-19T14:58:27.989892+0000","last_active":"2024-02-19T14:58:27.993412+0000","last_peered":"2024-02-19T14:58:27.993412+0000","last_clean":"2024-02-19T14:58:27.993412+0000","last_became_active":"2024-02-19T14:58:27.989618+0000","last_became_peered":"2024-02-19T14:58:27.989618+0000","last_unstale":"2024-02-19T14:58:27.993412+0000","last_undegraded":"2024-02-19T14:58:27.993412+0000","last_fullsized":"2024-02-19T14:58:27.993412+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:18.528717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991690+0000","last_change":"2024-02-19T14:58:26.967948+0000","last_active":"2024-02-19T14:58:27.991690+0000","last_peered":"2024-02-19T14:58:27.991690+0000","last_clean":"2024-02-19T14:58:27.991690+0000","last_became_active":"2024-02-19T14:58:26.967699+0000","last_became_peered":"2024-02-19T14:58:26.967699+0000","last_unstale":"2024-02-19T14:58:27.991690+0000","last_undegraded":"2024-02-19T14:58:27.991690+0000","last_fullsized":"2024-02-19T14:58:27.991690+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:18:38.320852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.975981+0000","last_change":"2024-02-19T14:58:27.972224+0000","last_active":"2024-02-19T14:58:27.975981+0000","last_peered":"2024-02-19T14:58:27.975981+0000","last_clean":"2024-02-19T14:58:27.975981+0000","last_became_active":"2024-02-19T14:58:27.971966+0000","last_became_peered":"2024-02-19T14:58:27.971966+0000","last_unstale":"2024-02-19T14:58:27.975981+0000","last_undegraded":"2024-02-19T14:58:27.975981+0000","last_fullsized":"2024-02-19T14:58:27.975981+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:22:21.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963319+0000","last_change":"2024-02-19T14:58:26.970089+0000","last_active":"2024-02-19T14:58:27.963319+0000","last_peered":"2024-02-19T14:58:27.963319+0000","last_clean":"2024-02-19T14:58:27.963319+0000","last_became_active":"2024-02-19T14:58:26.969826+0000","last_became_peered":"2024-02-19T14:58:26.969826+0000","last_unstale":"2024-02-19T14:58:27.963319+0000","last_undegraded":"2024-02-19T14:58:27.963319+0000","last_fullsized":"2024-02-19T14:58:27.963319+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:25:22.140210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993904+0000","last_change":"2024-02-19T14:58:27.992823+0000","last_active":"2024-02-19T14:58:27.993904+0000","last_peered":"2024-02-19T14:58:27.993904+0000","last_clean":"2024-02-19T14:58:27.993904+0000","last_became_active":"2024-02-19T14:58:27.986687+0000","last_became_peered":"2024-02-19T14:58:27.986687+0000","last_unstale":"2024-02-19T14:58:27.993904+0000","last_undegraded":"2024-02-19T14:58:27.993904+0000","last_fullsized":"2024-02-19T14:58:27.993904+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:37:50.093108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.966993+0000","last_change":"2024-02-19T14:58:26.978319+0000","last_active":"2024-02-19T14:58:27.966993+0000","last_peered":"2024-02-19T14:58:27.966993+0000","last_clean":"2024-02-19T14:58:27.966993+0000","last_became_active":"2024-02-19T14:58:26.977982+0000","last_became_peered":"2024-02-19T14:58:26.977982+0000","last_unstale":"2024-02-19T14:58:27.966993+0000","last_undegraded":"2024-02-19T14:58:27.966993+0000","last_fullsized":"2024-02-19T14:58:27.966993+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:54:30.432523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991153+0000","last_change":"2024-02-19T14:58:27.986576+0000","last_active":"2024-02-19T14:58:27.991153+0000","last_peered":"2024-02-19T14:58:27.991153+0000","last_clean":"2024-02-19T14:58:27.991153+0000","last_became_active":"2024-02-19T14:58:27.986297+0000","last_became_peered":"2024-02-19T14:58:27.986297+0000","last_unstale":"2024-02-19T14:58:27.991153+0000","last_undegraded":"2024-02-19T14:58:27.991153+0000","last_fullsized":"2024-02-19T14:58:27.991153+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:02:37.236794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"173'8","reported_seq":20,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994709+0000","last_change":"2024-02-19T14:58:26.973198+0000","last_active":"2024-02-19T14:58:27.994709+0000","last_peered":"2024-02-19T14:58:27.994709+0000","last_clean":"2024-02-19T14:58:27.994709+0000","last_became_active":"2024-02-19T14:58:26.972933+0000","last_became_peered":"2024-02-19T14:58:26.972933+0000","last_unstale":"2024-02-19T14:58:27.994709+0000","last_undegraded":"2024-02-19T14:58:27.994709+0000","last_fullsized":"2024-02-19T14:58:27.994709+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:42:38.124989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991213+0000","last_change":"2024-02-19T14:58:27.983712+0000","last_active":"2024-02-19T14:58:27.991213+0000","last_peered":"2024-02-19T14:58:27.991213+0000","last_clean":"2024-02-19T14:58:27.991213+0000","last_became_active":"2024-02-19T14:58:27.983102+0000","last_became_peered":"2024-02-19T14:58:27.983102+0000","last_unstale":"2024-02-19T14:58:27.991213+0000","last_undegraded":"2024-02-19T14:58:27.991213+0000","last_fullsized":"2024-02-19T14:58:27.991213+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:35:36.835628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.993935+0000","last_change":"2024-02-19T14:58:26.969777+0000","last_active":"2024-02-19T14:58:27.993935+0000","last_peered":"2024-02-19T14:58:27.993935+0000","last_clean":"2024-02-19T14:58:27.993935+0000","last_became_active":"2024-02-19T14:58:26.969462+0000","last_became_peered":"2024-02-19T14:58:26.969462+0000","last_unstale":"2024-02-19T14:58:27.993935+0000","last_undegraded":"2024-02-19T14:58:27.993935+0000","last_fullsized":"2024-02-19T14:58:27.993935+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:55:02.496644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986952+0000","last_change":"2024-02-19T14:58:27.985905+0000","last_active":"2024-02-19T14:58:27.986952+0000","last_peered":"2024-02-19T14:58:27.986952+0000","last_clean":"2024-02-19T14:58:27.986952+0000","last_became_active":"2024-02-19T14:58:27.985665+0000","last_became_peered":"2024-02-19T14:58:27.985665+0000","last_unstale":"2024-02-19T14:58:27.986952+0000","last_undegraded":"2024-02-19T14:58:27.986952+0000","last_fullsized":"2024-02-19T14:58:27.986952+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:34.846534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.992665+0000","last_change":"2024-02-19T14:58:26.973761+0000","last_active":"2024-02-19T14:58:27.992665+0000","last_peered":"2024-02-19T14:58:27.992665+0000","last_clean":"2024-02-19T14:58:27.992665+0000","last_became_active":"2024-02-19T14:58:26.973498+0000","last_became_peered":"2024-02-19T14:58:26.973498+0000","last_unstale":"2024-02-19T14:58:27.992665+0000","last_undegraded":"2024-02-19T14:58:27.992665+0000","last_fullsized":"2024-02-19T14:58:27.992665+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:30.104435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982489+0000","last_change":"2024-02-19T14:58:27.979778+0000","last_active":"2024-02-19T14:58:27.982489+0000","last_peered":"2024-02-19T14:58:27.982489+0000","last_clean":"2024-02-19T14:58:27.982489+0000","last_became_active":"2024-02-19T14:58:27.979509+0000","last_became_peered":"2024-02-19T14:58:27.979509+0000","last_unstale":"2024-02-19T14:58:27.982489+0000","last_undegraded":"2024-02-19T14:58:27.982489+0000","last_fullsized":"2024-02-19T14:58:27.982489+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:27:21.765067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991980+0000","last_change":"2024-02-19T14:58:26.974164+0000","last_active":"2024-02-19T14:58:27.991980+0000","last_peered":"2024-02-19T14:58:27.991980+0000","last_clean":"2024-02-19T14:58:27.991980+0000","last_became_active":"2024-02-19T14:58:26.973870+0000","last_became_peered":"2024-02-19T14:58:26.973870+0000","last_unstale":"2024-02-19T14:58:27.991980+0000","last_undegraded":"2024-02-19T14:58:27.991980+0000","last_fullsized":"2024-02-19T14:58:27.991980+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:47:33.132337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986214+0000","last_change":"2024-02-19T14:58:27.977947+0000","last_active":"2024-02-19T14:58:27.986214+0000","last_peered":"2024-02-19T14:58:27.986214+0000","last_clean":"2024-02-19T14:58:27.986214+0000","last_became_active":"2024-02-19T14:58:27.977700+0000","last_became_peered":"2024-02-19T14:58:27.977700+0000","last_unstale":"2024-02-19T14:58:27.986214+0000","last_undegraded":"2024-02-19T14:58:27.986214+0000","last_fullsized":"2024-02-19T14:58:27.986214+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:29:23.169778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.973399+0000","last_change":"2024-02-19T14:58:26.969658+0000","last_active":"2024-02-19T14:58:27.973399+0000","last_peered":"2024-02-19T14:58:27.973399+0000","last_clean":"2024-02-19T14:58:27.973399+0000","last_became_active":"2024-02-19T14:58:26.969374+0000","last_became_peered":"2024-02-19T14:58:26.969374+0000","last_unstale":"2024-02-19T14:58:27.973399+0000","last_undegraded":"2024-02-19T14:58:27.973399+0000","last_fullsized":"2024-02-19T14:58:27.973399+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:20:14.299623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.989245+0000","last_change":"2024-02-19T14:58:27.982543+0000","last_active":"2024-02-19T14:58:27.989245+0000","last_peered":"2024-02-19T14:58:27.989245+0000","last_clean":"2024-02-19T14:58:27.989245+0000","last_became_active":"2024-02-19T14:58:27.982187+0000","last_became_peered":"2024-02-19T14:58:27.982187+0000","last_unstale":"2024-02-19T14:58:27.989245+0000","last_undegraded":"2024-02-19T14:58:27.989245+0000","last_fullsized":"2024-02-19T14:58:27.989245+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:01.784277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991208+0000","last_change":"2024-02-19T14:58:26.976093+0000","last_active":"2024-02-19T14:58:27.991208+0000","last_peered":"2024-02-19T14:58:27.991208+0000","last_clean":"2024-02-19T14:58:27.991208+0000","last_became_active":"2024-02-19T14:58:26.975811+0000","last_became_peered":"2024-02-19T14:58:26.975811+0000","last_unstale":"2024-02-19T14:58:27.991208+0000","last_undegraded":"2024-02-19T14:58:27.991208+0000","last_fullsized":"2024-02-19T14:58:27.991208+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:57.166212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.989929+0000","last_change":"2024-02-19T14:58:27.984817+0000","last_active":"2024-02-19T14:58:27.989929+0000","last_peered":"2024-02-19T14:58:27.989929+0000","last_clean":"2024-02-19T14:58:27.989929+0000","last_became_active":"2024-02-19T14:58:27.984568+0000","last_became_peered":"2024-02-19T14:58:27.984568+0000","last_unstale":"2024-02-19T14:58:27.989929+0000","last_undegraded":"2024-02-19T14:58:27.989929+0000","last_fullsized":"2024-02-19T14:58:27.989929+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:57.589892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.971672+0000","last_change":"2024-02-19T14:58:26.973029+0000","last_active":"2024-02-19T14:58:27.971672+0000","last_peered":"2024-02-19T14:58:27.971672+0000","last_clean":"2024-02-19T14:58:27.971672+0000","last_became_active":"2024-02-19T14:58:26.972763+0000","last_became_peered":"2024-02-19T14:58:26.972763+0000","last_unstale":"2024-02-19T14:58:27.971672+0000","last_undegraded":"2024-02-19T14:58:27.971672+0000","last_fullsized":"2024-02-19T14:58:27.971672+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:05.143158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985694+0000","last_change":"2024-02-19T14:58:27.979602+0000","last_active":"2024-02-19T14:58:27.985694+0000","last_peered":"2024-02-19T14:58:27.985694+0000","last_clean":"2024-02-19T14:58:27.985694+0000","last_became_active":"2024-02-19T14:58:27.979311+0000","last_became_peered":"2024-02-19T14:58:27.979311+0000","last_unstale":"2024-02-19T14:58:27.985694+0000","last_undegraded":"2024-02-19T14:58:27.985694+0000","last_fullsized":"2024-02-19T14:58:27.985694+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:17.590610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963794+0000","last_change":"2024-02-19T14:58:26.971962+0000","last_active":"2024-02-19T14:58:27.963794+0000","last_peered":"2024-02-19T14:58:27.963794+0000","last_clean":"2024-02-19T14:58:27.963794+0000","last_became_active":"2024-02-19T14:58:26.971694+0000","last_became_peered":"2024-02-19T14:58:26.971694+0000","last_unstale":"2024-02-19T14:58:27.963794+0000","last_undegraded":"2024-02-19T14:58:27.963794+0000","last_fullsized":"2024-02-19T14:58:27.963794+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:38:23.799593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.990477+0000","last_change":"2024-02-19T14:58:27.984642+0000","last_active":"2024-02-19T14:58:27.990477+0000","last_peered":"2024-02-19T14:58:27.990477+0000","last_clean":"2024-02-19T14:58:27.990477+0000","last_became_active":"2024-02-19T14:58:27.984236+0000","last_became_peered":"2024-02-19T14:58:27.984236+0000","last_unstale":"2024-02-19T14:58:27.990477+0000","last_undegraded":"2024-02-19T14:58:27.990477+0000","last_fullsized":"2024-02-19T14:58:27.990477+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:56:28.335597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968288+0000","last_change":"2024-02-19T14:58:26.971509+0000","last_active":"2024-02-19T14:58:27.968288+0000","last_peered":"2024-02-19T14:58:27.968288+0000","last_clean":"2024-02-19T14:58:27.968288+0000","last_became_active":"2024-02-19T14:58:26.971237+0000","last_became_peered":"2024-02-19T14:58:26.971237+0000","last_unstale":"2024-02-19T14:58:27.968288+0000","last_undegraded":"2024-02-19T14:58:27.968288+0000","last_fullsized":"2024-02-19T14:58:27.968288+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:24:16.603636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991185+0000","last_change":"2024-02-19T14:58:27.983506+0000","last_active":"2024-02-19T14:58:27.991185+0000","last_peered":"2024-02-19T14:58:27.991185+0000","last_clean":"2024-02-19T14:58:27.991185+0000","last_became_active":"2024-02-19T14:58:27.981917+0000","last_became_peered":"2024-02-19T14:58:27.981917+0000","last_unstale":"2024-02-19T14:58:27.991185+0000","last_undegraded":"2024-02-19T14:58:27.991185+0000","last_fullsized":"2024-02-19T14:58:27.991185+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:45.110380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994490+0000","last_change":"2024-02-19T14:58:26.980620+0000","last_active":"2024-02-19T14:58:27.994490+0000","last_peered":"2024-02-19T14:58:27.994490+0000","last_clean":"2024-02-19T14:58:27.994490+0000","last_became_active":"2024-02-19T14:58:26.980332+0000","last_became_peered":"2024-02-19T14:58:26.980332+0000","last_unstale":"2024-02-19T14:58:27.994490+0000","last_undegraded":"2024-02-19T14:58:27.994490+0000","last_fullsized":"2024-02-19T14:58:27.994490+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:33.839682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.973969+0000","last_change":"2024-02-19T14:58:27.970234+0000","last_active":"2024-02-19T14:58:27.973969+0000","last_peered":"2024-02-19T14:58:27.973969+0000","last_clean":"2024-02-19T14:58:27.973969+0000","last_became_active":"2024-02-19T14:58:27.969889+0000","last_became_peered":"2024-02-19T14:58:27.969889+0000","last_unstale":"2024-02-19T14:58:27.973969+0000","last_undegraded":"2024-02-19T14:58:27.973969+0000","last_fullsized":"2024-02-19T14:58:27.973969+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:56.669840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970236+0000","last_change":"2024-02-19T14:58:26.975057+0000","last_active":"2024-02-19T14:58:27.970236+0000","last_peered":"2024-02-19T14:58:27.970236+0000","last_clean":"2024-02-19T14:58:27.970236+0000","last_became_active":"2024-02-19T14:58:26.974750+0000","last_became_peered":"2024-02-19T14:58:26.974750+0000","last_unstale":"2024-02-19T14:58:27.970236+0000","last_undegraded":"2024-02-19T14:58:27.970236+0000","last_fullsized":"2024-02-19T14:58:27.970236+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:02:18.872350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.985787+0000","last_change":"2024-02-19T14:58:27.981675+0000","last_active":"2024-02-19T14:58:27.985787+0000","last_peered":"2024-02-19T14:58:27.985787+0000","last_clean":"2024-02-19T14:58:27.985787+0000","last_became_active":"2024-02-19T14:58:27.981244+0000","last_became_peered":"2024-02-19T14:58:27.981244+0000","last_unstale":"2024-02-19T14:58:27.985787+0000","last_undegraded":"2024-02-19T14:58:27.985787+0000","last_fullsized":"2024-02-19T14:58:27.985787+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:08:28.917033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.963615+0000","last_change":"2024-02-19T14:58:26.973613+0000","last_active":"2024-02-19T14:58:27.963615+0000","last_peered":"2024-02-19T14:58:27.963615+0000","last_clean":"2024-02-19T14:58:27.963615+0000","last_became_active":"2024-02-19T14:58:26.973270+0000","last_became_peered":"2024-02-19T14:58:26.973270+0000","last_unstale":"2024-02-19T14:58:27.963615+0000","last_undegraded":"2024-02-19T14:58:27.963615+0000","last_fullsized":"2024-02-19T14:58:27.963615+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:22:42.841183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.982515+0000","last_change":"2024-02-19T14:58:27.977846+0000","last_active":"2024-02-19T14:58:27.982515+0000","last_peered":"2024-02-19T14:58:27.982515+0000","last_clean":"2024-02-19T14:58:27.982515+0000","last_became_active":"2024-02-19T14:58:27.977567+0000","last_became_peered":"2024-02-19T14:58:27.977567+0000","last_unstale":"2024-02-19T14:58:27.982515+0000","last_undegraded":"2024-02-19T14:58:27.982515+0000","last_fullsized":"2024-02-19T14:58:27.982515+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:13.282576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968637+0000","last_change":"2024-02-19T14:58:26.980994+0000","last_active":"2024-02-19T14:58:27.968637+0000","last_peered":"2024-02-19T14:58:27.968637+0000","last_clean":"2024-02-19T14:58:27.968637+0000","last_became_active":"2024-02-19T14:58:26.980700+0000","last_became_peered":"2024-02-19T14:58:26.980700+0000","last_unstale":"2024-02-19T14:58:27.968637+0000","last_undegraded":"2024-02-19T14:58:27.968637+0000","last_fullsized":"2024-02-19T14:58:27.968637+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:43.461510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986196+0000","last_change":"2024-02-19T14:58:27.977747+0000","last_active":"2024-02-19T14:58:27.986196+0000","last_peered":"2024-02-19T14:58:27.986196+0000","last_clean":"2024-02-19T14:58:27.986196+0000","last_became_active":"2024-02-19T14:58:27.977491+0000","last_became_peered":"2024-02-19T14:58:27.977491+0000","last_unstale":"2024-02-19T14:58:27.986196+0000","last_undegraded":"2024-02-19T14:58:27.986196+0000","last_fullsized":"2024-02-19T14:58:27.986196+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:26:05.731216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.967500+0000","last_change":"2024-02-19T14:58:26.974432+0000","last_active":"2024-02-19T14:58:27.967500+0000","last_peered":"2024-02-19T14:58:27.967500+0000","last_clean":"2024-02-19T14:58:27.967500+0000","last_became_active":"2024-02-19T14:58:26.974034+0000","last_became_peered":"2024-02-19T14:58:26.974034+0000","last_unstale":"2024-02-19T14:58:27.967500+0000","last_undegraded":"2024-02-19T14:58:27.967500+0000","last_fullsized":"2024-02-19T14:58:27.967500+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:44:55.230057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991254+0000","last_change":"2024-02-19T14:58:27.987350+0000","last_active":"2024-02-19T14:58:27.991254+0000","last_peered":"2024-02-19T14:58:27.991254+0000","last_clean":"2024-02-19T14:58:27.991254+0000","last_became_active":"2024-02-19T14:58:27.987083+0000","last_became_peered":"2024-02-19T14:58:27.987083+0000","last_unstale":"2024-02-19T14:58:27.991254+0000","last_undegraded":"2024-02-19T14:58:27.991254+0000","last_fullsized":"2024-02-19T14:58:27.991254+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:04.735200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968945+0000","last_change":"2024-02-19T14:58:26.980818+0000","last_active":"2024-02-19T14:58:27.968945+0000","last_peered":"2024-02-19T14:58:27.968945+0000","last_clean":"2024-02-19T14:58:27.968945+0000","last_became_active":"2024-02-19T14:58:26.980539+0000","last_became_peered":"2024-02-19T14:58:26.980539+0000","last_unstale":"2024-02-19T14:58:27.968945+0000","last_undegraded":"2024-02-19T14:58:27.968945+0000","last_fullsized":"2024-02-19T14:58:27.968945+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:15.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.991377+0000","last_change":"2024-02-19T14:58:27.987592+0000","last_active":"2024-02-19T14:58:27.991377+0000","last_peered":"2024-02-19T14:58:27.991377+0000","last_clean":"2024-02-19T14:58:27.991377+0000","last_became_active":"2024-02-19T14:58:27.987321+0000","last_became_peered":"2024-02-19T14:58:27.987321+0000","last_unstale":"2024-02-19T14:58:27.991377+0000","last_undegraded":"2024-02-19T14:58:27.991377+0000","last_fullsized":"2024-02-19T14:58:27.991377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:17.590610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.968998+0000","last_change":"2024-02-19T14:58:26.970929+0000","last_active":"2024-02-19T14:58:27.968998+0000","last_peered":"2024-02-19T14:58:27.968998+0000","last_clean":"2024-02-19T14:58:27.968998+0000","last_became_active":"2024-02-19T14:58:26.970565+0000","last_became_peered":"2024-02-19T14:58:26.970565+0000","last_unstale":"2024-02-19T14:58:27.968998+0000","last_undegraded":"2024-02-19T14:58:27.968998+0000","last_fullsized":"2024-02-19T14:58:27.968998+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:51:55.381270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.974008+0000","last_change":"2024-02-19T14:58:27.970712+0000","last_active":"2024-02-19T14:58:27.974008+0000","last_peered":"2024-02-19T14:58:27.974008+0000","last_clean":"2024-02-19T14:58:27.974008+0000","last_became_active":"2024-02-19T14:58:27.970436+0000","last_became_peered":"2024-02-19T14:58:27.970436+0000","last_unstale":"2024-02-19T14:58:27.974008+0000","last_undegraded":"2024-02-19T14:58:27.974008+0000","last_fullsized":"2024-02-19T14:58:27.974008+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:16.192020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.970208+0000","last_change":"2024-02-19T14:58:26.974836+0000","last_active":"2024-02-19T14:58:27.970208+0000","last_peered":"2024-02-19T14:58:27.970208+0000","last_clean":"2024-02-19T14:58:27.970208+0000","last_became_active":"2024-02-19T14:58:26.974579+0000","last_became_peered":"2024-02-19T14:58:26.974579+0000","last_unstale":"2024-02-19T14:58:27.970208+0000","last_undegraded":"2024-02-19T14:58:27.970208+0000","last_fullsized":"2024-02-19T14:58:27.970208+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:50:01.152555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.986318+0000","last_change":"2024-02-19T14:58:27.983348+0000","last_active":"2024-02-19T14:58:27.986318+0000","last_peered":"2024-02-19T14:58:27.986318+0000","last_clean":"2024-02-19T14:58:27.986318+0000","last_became_active":"2024-02-19T14:58:27.983071+0000","last_became_peered":"2024-02-19T14:58:27.983071+0000","last_unstale":"2024-02-19T14:58:27.986318+0000","last_undegraded":"2024-02-19T14:58:27.986318+0000","last_fullsized":"2024-02-19T14:58:27.986318+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:26.953581+0000","last_clean_scrub_stamp":"2024-02-19T14:58:26.953581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:07.454563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.994234+0000","last_change":"2024-02-19T14:58:26.981340+0000","last_active":"2024-02-19T14:58:27.994234+0000","last_peered":"2024-02-19T14:58:27.994234+0000","last_clean":"2024-02-19T14:58:27.994234+0000","last_became_active":"2024-02-19T14:58:26.981053+0000","last_became_peered":"2024-02-19T14:58:26.981053+0000","last_unstale":"2024-02-19T14:58:27.994234+0000","last_undegraded":"2024-02-19T14:58:27.994234+0000","last_fullsized":"2024-02-19T14:58:27.994234+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:58:25.946729+0000","last_clean_scrub_stamp":"2024-02-19T14:58:25.946729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:06:55.662988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":457,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-19T14:58:27.971321+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:58:27.971321+0000","last_peered":"2024-02-19T14:58:27.971321+0000","last_clean":"2024-02-19T14:58:27.971321+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:58:27.971321+0000","last_undegraded":"2024-02-19T14:58:27.971321+0000","last_fullsized":"2024-02-19T14:58:27.971321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33888,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337952,"statfs":{"total":96636764160,"available":96602062848,"internally_reserved":0,"allocated":2768896,"data_stored":1655315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47599999999999998,"15min":0.46800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29499999999999998,"15min":0.27800000000000002},"max":{"1min":0.61599999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.504,"15min":0.496},"min":{"1min":0.28199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60699999999999998,"5min":0.75900000000000001,"15min":0.79500000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32500000000000001,"15min":0.27200000000000002},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.72999999999999998,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47899999999999998,"15min":0.50600000000000001},"min":{"1min":0.35299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64000000000000001,"5min":0.73999999999999999,"15min":0.74099999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52400000000000002,"15min":0.501},"min":{"1min":0.34699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70799999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.623}]},{"osd":3,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67600000000000005,"5min":0.81399999999999995,"15min":0.81499999999999995},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73499999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.48199999999999998,"15min":0.496},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.86899999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.497,"15min":0.53300000000000003},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76500000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51900000000000002,"15min":0.54100000000000004},"min":{"1min":0.27800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68100000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55900000000000005,"15min":0.59299999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.751},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.88500000000000001},"last":0.90500000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.48699999999999999,"15min":0.46800000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60999999999999999,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.49199999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.44}]},{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71399999999999997,"5min":0.72899999999999998,"15min":0.81999999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69899999999999995,"5min":0.81000000000000005,"15min":0.88100000000000001},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.40999999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70099999999999996,"5min":0.78100000000000003,"15min":0.90500000000000003},"last":0.57299999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.753,"5min":0.79400000000000004,"15min":0.84799999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.504,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.47899999999999998,"15min":0.52800000000000002},"min":{"1min":0.23599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.76600000000000001,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.60199999999999998},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71199999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47699999999999998,"15min":0.53200000000000003},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":0.753,"15min":0.77400000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.53200000000000003,"15min":0.55200000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.73499999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.37}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.52500000000000002,"15min":0.57299999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.69799999999999995,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53000000000000003,"15min":0.56200000000000006},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71499999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.49099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.372,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.52200000000000002,"15min":0.505},"min":{"1min":0.36799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59599999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52100000000000002,"15min":0.496},"min":{"1min":0.40100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.67100000000000004,"5min":0.872,"15min":0.872},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.29399999999999998},"max":{"1min":0.624,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54100000000000004,"15min":0.51500000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53700000000000003,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69399999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57499999999999996}]},{"osd":3,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.71299999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.32400000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68400000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60099999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.48099999999999998,"15min":0.51500000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.74099999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.51800000000000002,"15min":0.55600000000000005},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.75,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.26400000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76900000000000002,"5min":0.877,"15min":0.877},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.51500000000000001,"15min":0.56299999999999994},"min":{"1min":0.24199999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.71699999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.55000000000000004,"15min":0.59199999999999997},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80300000000000005,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58899999999999997,"15min":0.621},"min":{"1min":0.35499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79200000000000004,"5min":1.03,"15min":1.03},"last":0.40300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.55600000000000005,"15min":0.54200000000000004},"min":{"1min":0.34399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.6879999999999997,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.60999999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.51900000000000002,"15min":0.52800000000000002},"min":{"1min":0.34300000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.33500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.7290000000000001,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.318,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.0429999999999999,"5min":0.63100000000000001,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":10.629,"5min":10.629,"15min":10.629},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.60799999999999998,"15min":0.622},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":2.7010000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.73399999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.55100000000000005,"15min":0.50700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.6989999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.60499999999999998,"15min":0.57499999999999996},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.6239999999999997,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:57:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54600000000000004,"15min":0.56499999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.315,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.621,"5min":2.621,"15min":2.621},"last":0.85399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2199552,"data_stored":1072240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.34799999999999998,"15min":0.312},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.40999999999999998,"5min":3.694,"15min":3.694},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.312,"5min":0.36899999999999999,"15min":0.32500000000000001},"min":{"1min":0.21299999999999999,"5min":0.192,"15min":0.189},"max":{"1min":0.41499999999999998,"5min":4.835,"15min":4.835},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32300000000000001,"15min":0.314},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.52700000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42499999999999999,"15min":0.379},"min":{"1min":0.23100000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.47199999999999998,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41399999999999998,"15min":0.38600000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52600000000000002,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.40500000000000003,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.24199999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.55700000000000005},"min":{"1min":0.33800000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70799999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.57899999999999996,"15min":0.59399999999999997},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.439}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.60299999999999998,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79900000000000004,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81799999999999995,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.41499999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82099999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60399999999999998,"15min":0.64600000000000002},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72399999999999998,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58999999999999997,"15min":0.63800000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84799999999999998,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.40400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33308,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338532,"statfs":{"total":96636764160,"available":96602656768,"internally_reserved":0,"allocated":2174976,"data_stored":1064947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33400000000000002,"15min":0.317},"min":{"1min":0.20799999999999999,"5min":0.17999999999999999,"15min":0.17399999999999999},"max":{"1min":0.63600000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.31900000000000001,"15min":0.30499999999999999},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.35499999999999998,"15min":0.34499999999999997},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.28199999999999997,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.66400000000000003,"5min":1.512,"15min":1.512},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.34100000000000003},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":0.55200000000000005,"5min":0.621,"15min":0.65700000000000003},"last":0.31},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.377},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67600000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56799999999999995,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.755,"5min":1.133,"15min":1.133},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.41499999999999998,"5min":0.36599999999999999,"15min":0.33800000000000002},"max":{"1min":0.752,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63700000000000001,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57799999999999996,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.115,"15min":1.115},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.57299999999999995},"min":{"1min":0.39300000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76100000000000001,"5min":0.94199999999999995,"15min":1.236},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Feb 19 14:58:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73699999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41299999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.78500000000000003,"5min":0.90200000000000002,"15min":0.995},"last":0.73499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2187264,"data_stored":1072114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.39700000000000002,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.33300000000000002},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.40100000000000002,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30599999999999999,"15min":0.309},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.42999999999999999,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.251},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.36099999999999999},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.26800000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39800000000000002,"15min":0.39900000000000002},"min":{"1min":0.33600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54900000000000004,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.375,"15min":0.38900000000000001},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.54000000000000004,"15min":0.55700000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72999999999999998,"5min":0.77000000000000002,"15min":1.0009999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54800000000000004,"15min":0.55800000000000005},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85299999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78700000000000003,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.46600000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":0.80000000000000004,"5min":0.95499999999999996,"15min":1.026},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55100000000000005,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82399999999999995,"5min":0.996,"15min":0.996},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79500000000000004,"5min":1.036,"15min":1.036},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57499999999999996,"15min":0.59699999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73199999999999998,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33892,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337948,"statfs":{"total":96636764160,"available":96602058752,"internally_reserved":0,"allocated":2768896,"data_stored":1655315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.31,"15min":0.30199999999999999},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.315,"15min":0.30499999999999999},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.33800000000000002},"min":{"1min":0.24299999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.37,"15min":0.371},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76200000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.496},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54000000000000004,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54400000000000004,"15min":0.59699999999999998},"min":{"1min":0.38500000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83699999999999997,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54100000000000004,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79800000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.59499999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79000000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.47099999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.81100000000000005,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.58299999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:58:31.447 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:58:31.447 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:58:31.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.616+0000 7f35c0423640 1 Processor -- start 2024-02-19T14:58:31.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.616+0000 7f35c0423640 1 -- start start 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35c0423640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35b80acc50 0x7f35b80c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35c0423640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f35b8155070 0x7f35b8155500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35c0423640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b8071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35c0423640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80c2cf0 con 0x7f35b80acc50 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35c0423640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80c2030 con 0x7f35b8155070 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35c0423640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b8159130 con 0x7f35b80718f0 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35b80acc50 0x7f35b80c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35b80acc50 0x7f35b80c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58638/0 (socket says 172.21.15.78:58638) 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 -- 172.21.15.78:0/4144149782 learned_addr learned my addr 172.21.15.78:0/4144149782 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35bd9be640 1 --2- 172.21.15.78:0/4144149782 >> v2:172.21.15.150:3300/0 conn(0x7f35b8155070 0x7f35b8155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be9c0640 1 --2- 172.21.15.78:0/4144149782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b8071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 -- 172.21.15.78:0/4144149782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 msgr2=0x7f35b8071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 --2- 172.21.15.78:0/4144149782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b8071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 -- 172.21.15.78:0/4144149782 >> v2:172.21.15.150:3300/0 conn(0x7f35b8155070 msgr2=0x7f35b8155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 --2- 172.21.15.78:0/4144149782 >> v2:172.21.15.150:3300/0 conn(0x7f35b8155070 0x7f35b8155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.617+0000 7f35be1bf640 1 -- 172.21.15.78:0/4144149782 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80ad030 con 0x7f35b80acc50 2024-02-19T14:58:31.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.618+0000 7f35be1bf640 1 --2- 172.21.15.78:0/4144149782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b80acc50 0x7f35b80c17e0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f35b8072b00 tx=0x7f35b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c256076732ab309a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.618+0000 7f35bd1bd640 1 -- 172.21.15.78:0/4144149782 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40317e0 con 0x7f35b80acc50 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.618+0000 7f35bd1bd640 1 -- 172.21.15.78:0/4144149782 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b4031940 con 0x7f35b80acc50 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.618+0000 7f35bd1bd640 1 -- 172.21.15.78:0/4144149782 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4031cd0 con 0x7f35b80acc50 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 -- 172.21.15.78:0/4144149782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b80acc50 msgr2=0x7f35b80c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 --2- 172.21.15.78:0/4144149782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b80acc50 0x7f35b80c17e0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f35b8072b00 tx=0x7f35b402b880 comp rx=0 tx=0).stop 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 -- 172.21.15.78:0/4144149782 shutdown_connections 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 --2- 172.21.15.78:0/4144149782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b8071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 --2- 172.21.15.78:0/4144149782 >> v2:172.21.15.150:3300/0 conn(0x7f35b8155070 0x7f35b8155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 --2- 172.21.15.78:0/4144149782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b80acc50 0x7f35b80c17e0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 -- 172.21.15.78:0/4144149782 >> 172.21.15.78:0/4144149782 conn(0x7f35b80c7800 msgr2=0x7f35b80c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 -- 172.21.15.78:0/4144149782 shutdown_connections 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.619+0000 7f35c0423640 1 -- 172.21.15.78:0/4144149782 wait complete. 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 Processor -- start 2024-02-19T14:58:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 -- start start 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b81ee020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35b80acc50 0x7f35b80c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 0x7f35b80aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81ee960 con 0x7f35b8155070 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80c6570 con 0x7f35b80acc50 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.620+0000 7f35c0423640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b81eec00 con 0x7f35b80718f0 2024-02-19T14:58:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 0x7f35b80aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 0x7f35b80aba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58642/0 (socket says 172.21.15.78:58642) 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 -- 172.21.15.78:0/397977778 learned_addr learned my addr 172.21.15.78:0/397977778 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be1bf640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b81ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35bd9be640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35b80acc50 0x7f35b80c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 -- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 msgr2=0x7f35b81ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b81ee020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 -- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35b80acc50 msgr2=0x7f35b80c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35b80acc50 0x7f35b80c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 -- 172.21.15.78:0/397977778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40094d0 con 0x7f35b8155070 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35bd9be640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35b80acc50 0x7f35b80c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be1bf640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b81ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35be9c0640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 0x7f35b80aba60 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f35b0007e60 tx=0x7f35b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0014620 con 0x7f35b8155070 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000ee90 con 0x7f35b8155070 2024-02-19T14:58:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b80c6830 con 0x7f35b8155070 2024-02-19T14:58:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b00149e0 con 0x7f35b8155070 2024-02-19T14:58:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.621+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b8138bc0 con 0x7f35b8155070 2024-02-19T14:58:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.622+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b81595a0 con 0x7f35b8155070 2024-02-19T14:58:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.623+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f35b0014b40 con 0x7f35b8155070 2024-02-19T14:58:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.624+0000 7f35aeffd640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f359c06b840 0x7f359c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.624+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f35b008abd0 con 0x7f35b8155070 2024-02-19T14:58:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.624+0000 7f35be1bf640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f359c06b840 0x7f359c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.624+0000 7f35be1bf640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f359c06b840 0x7f359c06dd00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f35b4036000 tx=0x7f35b403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.628+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f35b81595a0 con 0x7f35b8155070 2024-02-19T14:58:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.948+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f35b81595a0 con 0x7f35b8155070 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.950+0000 7f35aeffd640 1 -- 172.21.15.78:0/397977778 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f35b81595a0 con 0x7f35b8155070 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:58:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:58:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f359c06b840 msgr2=0x7f359c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f359c06b840 0x7f359c06dd00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f35b4036000 tx=0x7f35b403d040 comp rx=0 tx=0).stop 2024-02-19T14:58:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 msgr2=0x7f35b80aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 0x7f35b80aba60 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f35b0007e60 tx=0x7f35b000e4b0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 shutdown_connections 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f359c06b840 0x7f359c06dd00 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35b8155070 0x7f35b80aba60 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35b80acc50 0x7f35b80c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 --2- 172.21.15.78:0/397977778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f35b80718f0 0x7f35b81ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 >> 172.21.15.78:0/397977778 conn(0x7f35b80c7800 msgr2=0x7f35b8154540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 shutdown_connections 2024-02-19T14:58:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:31.952+0000 7f35c0423640 1 -- 172.21.15.78:0/397977778 wait complete. 2024-02-19T14:58:31.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:32.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.086+0000 7f4badb0c640 1 Processor -- start 2024-02-19T14:58:32.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.086+0000 7f4badb0c640 1 -- start start 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4badb0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ba814d210 0x7f4ba80c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4badb0c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba806ff90 con 0x7f4ba80719e0 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4badb0c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba8156fc0 con 0x7f4ba80c3d90 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4badb0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba8070270 con 0x7f4ba814d210 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba77fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ba814d210 0x7f4ba80c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4ba80719e0 0x7f4ba80c64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47234/0 (socket says 172.21.15.78:47234) 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba6ffd640 1 -- 172.21.15.78:0/2122134376 learned_addr learned my addr 172.21.15.78:0/2122134376 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:32.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba77fe640 1 -- 172.21.15.78:0/2122134376 >> v1:172.21.15.150:6789/0 conn(0x7f4ba80c3d90 legacy=0x7f4ba815d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba77fe640 1 -- 172.21.15.78:0/2122134376 >> v1:172.21.15.78:6789/0 conn(0x7f4ba80719e0 legacy=0x7f4ba80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba67fc640 1 -- 172.21.15.78:0/2122134376 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665113575 0 0) 0x7f4ba806ff90 con 0x7f4ba80719e0 2024-02-19T14:58:32.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.087+0000 7f4ba77fe640 1 -- 172.21.15.78:0/2122134376 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba81572a0 con 0x7f4ba814d210 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.088+0000 7f4ba77fe640 1 --2- 172.21.15.78:0/2122134376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba814d210 0x7f4ba80c4290 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f4b980097a0 tx=0x7f4b9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f18fbb94622f83 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.088+0000 7f4ba67fc640 1 -- 172.21.15.78:0/2122134376 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9802e070 con 0x7f4ba814d210 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.088+0000 7f4ba67fc640 1 -- 172.21.15.78:0/2122134376 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b980315a0 con 0x7f4ba814d210 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.088+0000 7f4ba67fc640 1 -- 172.21.15.78:0/2122134376 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98031930 con 0x7f4ba814d210 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 -- 172.21.15.78:0/2122134376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba814d210 msgr2=0x7f4ba80c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 --2- 172.21.15.78:0/2122134376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba814d210 0x7f4ba80c4290 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f4b980097a0 tx=0x7f4b9802b840 comp rx=0 tx=0).stop 2024-02-19T14:58:32.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 -- 172.21.15.78:0/2122134376 shutdown_connections 2024-02-19T14:58:32.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 --2- 172.21.15.78:0/2122134376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba814d210 0x7f4ba80c4290 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 -- 172.21.15.78:0/2122134376 >> 172.21.15.78:0/2122134376 conn(0x7f4ba80c5050 msgr2=0x7f4ba80c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 -- 172.21.15.78:0/2122134376 shutdown_connections 2024-02-19T14:58:32.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.089+0000 7f4badb0c640 1 -- 172.21.15.78:0/2122134376 wait complete. 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.090+0000 7f4badb0c640 1 Processor -- start 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.090+0000 7f4badb0c640 1 -- start start 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4badb0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 0x7f4ba8146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4badb0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ba80c3d90 0x7f4ba81e9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4badb0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ba814d210 0x7f4ba80c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4badb0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80c3970 con 0x7f4ba80719e0 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4badb0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81ecfb0 con 0x7f4ba80c3d90 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4badb0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81e8ea0 con 0x7f4ba814d210 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 0x7f4ba8146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 0x7f4ba8146d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58672/0 (socket says 172.21.15.78:58672) 2024-02-19T14:58:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 -- 172.21.15.78:0/3547256058 learned_addr learned my addr 172.21.15.78:0/3547256058 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 -- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ba814d210 msgr2=0x7f4ba80c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba6ffd640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ba80c3d90 0x7f4ba81e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ba814d210 0x7f4ba80c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ba814d210 0x7f4ba80c7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 -- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ba80c3d90 msgr2=0x7f4ba81e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ba80c3d90 0x7f4ba81e9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 -- 172.21.15.78:0/3547256058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98009400 con 0x7f4ba80719e0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba6ffd640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ba80c3d90 0x7f4ba81e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba7fff640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ba814d210 0x7f4ba80c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.091+0000 7f4ba77fe640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 0x7f4ba8146d30 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f4b98000c00 tx=0x7f4b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.092+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98032900 con 0x7f4ba80719e0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.092+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98032a80 con 0x7f4ba80719e0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.092+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba80c3660 con 0x7f4ba80719e0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.092+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98032e30 con 0x7f4ba80719e0 2024-02-19T14:58:32.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.092+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba8153d20 con 0x7f4ba80719e0 2024-02-19T14:58:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.093+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8074990 con 0x7f4ba80719e0 2024-02-19T14:58:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.093+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b9803b3f0 con 0x7f4ba80719e0 2024-02-19T14:58:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.093+0000 7f4b8ffff640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b9406b840 0x7f4b9406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.093+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4b980ac3b0 con 0x7f4ba80719e0 2024-02-19T14:58:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.093+0000 7f4ba6ffd640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b9406b840 0x7f4b9406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.094+0000 7f4ba6ffd640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b9406b840 0x7f4b9406dd00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000f60 tx=0x7f4b9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.098+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b98076cd0 con 0x7f4ba80719e0 2024-02-19T14:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.324+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba80c2c30 con 0x7f4ba80719e0 2024-02-19T14:58:32.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.325+0000 7f4b8ffff640 1 -- 172.21.15.78:0/3547256058 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f4b9807a390 con 0x7f4ba80719e0 2024-02-19T14:58:32.328 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:32.328 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":152,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:29.325646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:32.328 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 152 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.326+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b9406b840 msgr2=0x7f4b9406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.326+0000 7f4badb0c640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b9406b840 0x7f4b9406dd00 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000f60 tx=0x7f4b9c009040 comp rx=0 tx=0).stop 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.326+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 msgr2=0x7f4ba8146d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.326+0000 7f4badb0c640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 0x7f4ba8146d30 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f4b98000c00 tx=0x7f4b980046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 shutdown_connections 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b9406b840 0x7f4b9406dd00 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ba814d210 0x7f4ba80c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ba80c3d90 0x7f4ba81e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 --2- 172.21.15.78:0/3547256058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ba80719e0 0x7f4ba8146d30 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 >> 172.21.15.78:0/3547256058 conn(0x7f4ba80c5050 msgr2=0x7f4ba80c4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 shutdown_connections 2024-02-19T14:58:32.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.327+0000 7f4badb0c640 1 -- 172.21.15.78:0/3547256058 wait complete. 2024-02-19T14:58:32.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 152, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:29.325646+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:32.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:58:32.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:58:32.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.843+0000 7fd9151a2640 1 Processor -- start 2024-02-19T14:58:32.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9151a2640 1 -- start start 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9151a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd91014f260 0x7fd91006be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9151a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd910066c90 0x7fd91015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9151a2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd910151a50 con 0x7fd9101527b0 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9151a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd910067190 con 0x7fd910066c90 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9151a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd910151cd0 con 0x7fd91014f260 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd9149a1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd910066c90 0x7fd91015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd90f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd9101527b0 0x7fd91006d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47236/0 (socket says 172.21.15.78:47236) 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.844+0000 7fd90f7fe640 1 -- 172.21.15.78:0/2823617403 learned_addr learned my addr 172.21.15.78:0/2823617403 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.845+0000 7fd90ffff640 1 --2- 172.21.15.78:0/2823617403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd91014f260 0x7fd91006be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.845+0000 7fd9149a1640 1 -- 172.21.15.78:0/2823617403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd91014f260 msgr2=0x7fd91006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.845+0000 7fd9149a1640 1 --2- 172.21.15.78:0/2823617403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd91014f260 0x7fd91006be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.845+0000 7fd9149a1640 1 -- 172.21.15.78:0/2823617403 >> v1:172.21.15.78:6789/0 conn(0x7fd9101527b0 legacy=0x7fd91006d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.845+0000 7fd9149a1640 1 -- 172.21.15.78:0/2823617403 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910151fb0 con 0x7fd910066c90 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.845+0000 7fd9149a1640 1 --2- 172.21.15.78:0/2823617403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd91015c3d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fd908009d90 tx=0x7fd90802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd138afba7c8f693 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd90effd640 1 -- 172.21.15.78:0/2823617403 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90802cc50 con 0x7fd910066c90 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd90effd640 1 -- 172.21.15.78:0/2823617403 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90802cdb0 con 0x7fd910066c90 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd90effd640 1 -- 172.21.15.78:0/2823617403 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908034660 con 0x7fd910066c90 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 -- 172.21.15.78:0/2823617403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 msgr2=0x7fd91015c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 --2- 172.21.15.78:0/2823617403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd91015c3d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fd908009d90 tx=0x7fd90802bea0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 -- 172.21.15.78:0/2823617403 shutdown_connections 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 --2- 172.21.15.78:0/2823617403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd91015c3d0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 --2- 172.21.15.78:0/2823617403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd91014f260 0x7fd91006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 -- 172.21.15.78:0/2823617403 >> 172.21.15.78:0/2823617403 conn(0x7fd9100c6ac0 msgr2=0x7fd9100c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 -- 172.21.15.78:0/2823617403 shutdown_connections 2024-02-19T14:58:32.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.846+0000 7fd9151a2640 1 -- 172.21.15.78:0/2823617403 wait complete. 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.847+0000 7fd9151a2640 1 Processor -- start 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.847+0000 7fd9151a2640 1 -- start start 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd9151a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd9101ec480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd9151a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd91014f260 0x7fd9101de0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd9151a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9101527b0 0x7fd9101de620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd9151a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100c10d0 con 0x7fd910066c90 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd9151a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101ed150 con 0x7fd91014f260 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd9151a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101deb60 con 0x7fd9101527b0 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd9101ec480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd9101ec480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58690/0 (socket says 172.21.15.78:58690) 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 -- 172.21.15.78:0/606668465 learned_addr learned my addr 172.21.15.78:0/606668465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 -- 172.21.15.78:0/606668465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9101527b0 msgr2=0x7fd9101de620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9101527b0 0x7fd9101de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 -- 172.21.15.78:0/606668465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd91014f260 msgr2=0x7fd9101de0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd91014f260 0x7fd9101de0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.848+0000 7fd90ffff640 1 -- 172.21.15.78:0/606668465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908009a70 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.849+0000 7fd90ffff640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd9101ec480 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fd90000d800 tx=0x7fd90000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.849+0000 7fd90cff9640 1 -- 172.21.15.78:0/606668465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000b6c0 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.849+0000 7fd90cff9640 1 -- 172.21.15.78:0/606668465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90000b820 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.849+0000 7fd90cff9640 1 -- 172.21.15.78:0/606668465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000bbb0 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.849+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9101ece70 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.849+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd910151490 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.850+0000 7fd90cff9640 1 -- 172.21.15.78:0/606668465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd90000bd10 con 0x7fd910066c90 2024-02-19T14:58:32.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.851+0000 7fd90cff9640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8fc06b840 0x7fd8fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.851+0000 7fd90cff9640 1 -- 172.21.15.78:0/606668465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd90008a810 con 0x7fd910066c90 2024-02-19T14:58:32.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.851+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8fc06b840 0x7fd8fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.851+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8fc06b840 0x7fd8fc06dd00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd9100c6790 tx=0x7fd904007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.853+0000 7fd918fda740 1 Processor -- start 2024-02-19T14:58:32.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.853+0000 7fd918fda740 1 -- start start 2024-02-19T14:58:32.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd918fda740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490baeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490b73ce0 0x55c490badd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490b72a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd918fda740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490b2dcd0 con 0x55c490bad940 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490ae0f20 con 0x55c490bb0ab0 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c490ade450 con 0x55c490b73ce0 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490baeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490baeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58706/0 (socket says 172.21.15.78:58706) 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 -- 172.21.15.78:0/3936411378 learned_addr learned my addr 172.21.15.78:0/3936411378 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490b73ce0 0x55c490badd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd9149a1640 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490b72a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 -- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490b73ce0 msgr2=0x55c490badd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490b73ce0 0x55c490badd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 -- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 msgr2=0x55c490b72a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490b72a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 -- 172.21.15.78:0/3936411378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c490a8ef20 con 0x55c490bad940 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.854+0000 7fd90ffff640 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490baeab0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fd900088fd0 tx=0x7fd900058920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d74c865c680c28 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd8f5ffb640 1 -- 172.21.15.78:0/3936411378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900070070 con 0x55c490bad940 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd8f5ffb640 1 -- 172.21.15.78:0/3936411378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9000615a0 con 0x55c490bad940 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd8f5ffb640 1 -- 172.21.15.78:0/3936411378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900055500 con 0x55c490bad940 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 -- 172.21.15.78:0/3936411378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 msgr2=0x55c490baeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490baeab0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fd900088fd0 tx=0x7fd900058920 comp rx=0 tx=0).stop 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 -- 172.21.15.78:0/3936411378 shutdown_connections 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490b72a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490b73ce0 0x55c490badd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 --2- 172.21.15.78:0/3936411378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490baeab0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 -- 172.21.15.78:0/3936411378 >> 172.21.15.78:0/3936411378 conn(0x55c490b6f320 msgr2=0x55c490b93d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 -- 172.21.15.78:0/3936411378 shutdown_connections 2024-02-19T14:58:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.855+0000 7fd918fda740 1 -- 172.21.15.78:0/3936411378 wait complete. 2024-02-19T14:58:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.856+0000 7fd918fda740 1 -- start start 2024-02-19T14:58:32.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.856+0000 7fd918fda740 1 Processor -- start 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490ba79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bbfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b90be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490ad9c30 con 0x55c490bad940 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490a7ed30 con 0x55c490bb0ab0 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4907ad310 con 0x55c490ba5ab0 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd918fda740 10 client.0 _ll_register_callbacks cb 0x55c490a86d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490ba79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490ba79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58710/0 (socket says 172.21.15.78:58710) 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 -- 172.21.15.78:0/3544667432 learned_addr learned my addr 172.21.15.78:0/3544667432 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bbfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd9149a1640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b90be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 -- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 msgr2=0x55c490b90be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b90be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 -- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 msgr2=0x55c490bbfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bbfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 -- 172.21.15.78:0/3544667432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd910151490 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bbfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd90ffff640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490ba79c0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fd90000b4b0 tx=0x7fd900020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f5db91806f4dd2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.857+0000 7fd8eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd8eeffd640 1 -- 172.21.15.78:0/3544667432 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900070040 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd8eeffd640 1 -- 172.21.15.78:0/3544667432 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd90005f070 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4907d96c0 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd8eeffd640 1 -- 172.21.15.78:0/3544667432 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd90005f230 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 10 client.9522 fetch_fsmap learned FSMap version 152 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=2+}) v3 -- 0x55c490ad82b0 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd8eeffd640 1 -- 172.21.15.78:0/3544667432 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c4907d96c0 con 0x55c490bad940 2024-02-19T14:58:32.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd8eeffd640 1 -- 172.21.15.78:0/3544667432 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 152) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd900061110 con 0x55c490bad940 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 10 client.9522 fetch_fsmap finished waiting for FSMap version 152 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 10 client.9522 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 20 client.9522 populate_metadata read hostname 'smithi078' 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490bc2590 0x55c490bc2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 4 client.9522 mds_command: new command op to 9375 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 --> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c49082a270 con 0x55c490bc2590 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.858+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490bc2590 0x55c490bc2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd8eeffd640 10 client.9522 ms_handle_connect on v2:172.21.15.78:6834/2602543074 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490bc2590 0x55c490bc2970 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eabb1f859b55519 server_cookie=170b904b942eb19d in_seq=0 out_seq=0 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd8eeffd640 1 -- 172.21.15.78:0/3544667432 <== mds.0 v2:172.21.15.78:6834/2602543074 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c49082a270 con 0x55c490bc2590 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd8eeffd640 10 client.9522 handle_command_reply: tid=0 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 client.9522 shutdown 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 msgr2=0x55c490ba79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490ba79c0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fd90000b4b0 tx=0x7fd900020070 comp rx=0 tx=0).stop 2024-02-19T14:58:32.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 shutdown_connections 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490bc2590 0x55c490bc2970 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b90be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bbfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd8eeffd640 0 client.9522 ms_handle_reset on v2:172.21.15.78:6834/2602543074 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 --2- 172.21.15.78:0/3544667432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490ba79c0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 >> 172.21.15.78:0/3544667432 conn(0x55c490b73ce0 msgr2=0x55c490baf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 shutdown_connections 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.859+0000 7fd918fda740 1 -- 172.21.15.78:0/3544667432 wait complete. 2024-02-19T14:58:32.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.860+0000 7fd918fda740 20 client.9522 trim_cache size 0 max 16384 2024-02-19T14:58:32.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.876+0000 7fd918fda740 1 Processor -- start 2024-02-19T14:58:32.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.876+0000 7fd918fda740 1 -- start start 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b93600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd918fda740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490bc89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bc8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd918fda740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490a5f490 con 0x55c490bad940 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490a8dd60 con 0x55c490bb0ab0 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c490ba6de0 con 0x55c490ba5ab0 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490bc89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490bc89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58724/0 (socket says 172.21.15.78:58724) 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1325246061 learned_addr learned my addr 172.21.15.78:0/1325246061 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1325246061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 msgr2=0x55c490b93600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b93600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1325246061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 msgr2=0x55c490bc8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bc8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1325246061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c490a74a20 con 0x55c490bad940 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490bc89c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fd904005880 tx=0x7fd904010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d04635ee2567c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd8f57fa640 1 -- 172.21.15.78:0/1325246061 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90401d040 con 0x55c490bad940 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd8f57fa640 1 -- 172.21.15.78:0/1325246061 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90401e040 con 0x55c490bad940 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.877+0000 7fd8f57fa640 1 -- 172.21.15.78:0/1325246061 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd904025900 con 0x55c490bad940 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 -- 172.21.15.78:0/1325246061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 msgr2=0x55c490bc89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490bc89c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fd904005880 tx=0x7fd904010000 comp rx=0 tx=0).stop 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 -- 172.21.15.78:0/1325246061 shutdown_connections 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490bc89c0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490bb0ab0 0x55c490bc8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 --2- 172.21.15.78:0/1325246061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490ba5ab0 0x55c490b93600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 -- 172.21.15.78:0/1325246061 >> 172.21.15.78:0/1325246061 conn(0x55c490b73ce0 msgr2=0x55c490ba7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 -- 172.21.15.78:0/1325246061 shutdown_connections 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 -- 172.21.15.78:0/1325246061 wait complete. 2024-02-19T14:58:32.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.878+0000 7fd918fda740 1 -- start start 2024-02-19T14:58:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 Processor -- start 2024-02-19T14:58:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490ba5ab0 0x55c490bae120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490c21da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490bb0ab0 0x55c490c06890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490c229b0 con 0x55c490bad940 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490bac420 con 0x55c490ba5ab0 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c490bac200 con 0x55c490bb0ab0 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd918fda740 10 client.0 _ll_register_callbacks cb 0x7fd908037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd90f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490c21da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd90f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490c21da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58736/0 (socket says 172.21.15.78:58736) 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1332226716 learned_addr learned my addr 172.21.15.78:0/1332226716 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.879+0000 7fd9149a1640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490bb0ab0 0x55c490c06890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90ffff640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490ba5ab0 0x55c490bae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490bb0ab0 msgr2=0x55c490c06890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490bb0ab0 0x55c490c06890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490ba5ab0 msgr2=0x55c490bae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490ba5ab0 0x55c490bae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90f7fe640 1 -- 172.21.15.78:0/1332226716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd910151a50 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd9149a1640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490bb0ab0 0x55c490c06890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90ffff640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490ba5ab0 0x55c490bae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd90f7fe640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490c21da0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fd904005e20 tx=0x7fd904001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c8de940df9cc84 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd8eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd8eeffd640 1 -- 172.21.15.78:0/1332226716 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90401d070 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd8eeffd640 1 -- 172.21.15.78:0/1332226716 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd904025c50 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9000615a0 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd8eeffd640 1 -- 172.21.15.78:0/1332226716 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd904025e10 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 10 client.9528 fetch_fsmap learned FSMap version 152 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=152,monmap=2+}) v3 -- 0x55c490a9a780 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd8eeffd640 1 -- 172.21.15.78:0/1332226716 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9000615a0 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd8eeffd640 1 -- 172.21.15.78:0/1332226716 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 152) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd904014430 con 0x55c490bad940 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 10 client.9528 fetch_fsmap finished waiting for FSMap version 152 2024-02-19T14:58:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 10 client.9528 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 20 client.9528 populate_metadata read hostname 'smithi078' 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490c06fe0 0x55c490bc5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 4 client.9528 mds_command: new command op to 9375 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.880+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 --> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c490816610 con 0x55c490c06fe0 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd90ffff640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490c06fe0 0x55c490bc5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd8eeffd640 10 client.9528 ms_handle_connect on v2:172.21.15.78:6834/2602543074 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd90ffff640 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490c06fe0 0x55c490bc5c50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ebc149440e7689c server_cookie=c57e1e459ff5aba8 in_seq=0 out_seq=0 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd8eeffd640 1 -- 172.21.15.78:0/1332226716 <== mds.0 v2:172.21.15.78:6834/2602543074 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55c490816610 con 0x55c490c06fe0 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd8eeffd640 10 client.9528 handle_command_reply: tid=0 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 client.9528 shutdown 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 msgr2=0x55c490c21da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490c21da0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fd904005e20 tx=0x7fd904001080 comp rx=0 tx=0).stop 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 shutdown_connections 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:6834/2602543074,v1:172.21.15.78:6836/2602543074] conn(0x55c490c06fe0 0x55c490bc5c50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c490bad940 0x55c490c21da0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c490bb0ab0 0x55c490c06890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd8eeffd640 0 client.9528 ms_handle_reset on v2:172.21.15.78:6834/2602543074 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 --2- 172.21.15.78:0/1332226716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c490ba5ab0 0x55c490bae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 >> 172.21.15.78:0/1332226716 conn(0x55c490b73ce0 msgr2=0x55c490b93600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 shutdown_connections 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.881+0000 7fd918fda740 1 -- 172.21.15.78:0/1332226716 wait complete. 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd918fda740 20 client.9528 trim_cache size 0 max 16384 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 9375, 2024-02-19T14:58:32.890 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.9218756030000002, 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 152, 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 174, 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 174, 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 10.037829035 2024-02-19T14:58:32.891 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8fc06b840 msgr2=0x7fd8fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8fc06b840 0x7fd8fc06dd00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd9100c6790 tx=0x7fd904007a40 comp rx=0 tx=0).stop 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 msgr2=0x7fd9101ec480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd9101ec480 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fd90000d800 tx=0x7fd90000dcc0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 shutdown_connections 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8fc06b840 0x7fd8fc06dd00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9101527b0 0x7fd9101de620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd91014f260 0x7fd9101de0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 --2- 172.21.15.78:0/606668465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd910066c90 0x7fd9101ec480 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 >> 172.21.15.78:0/606668465 conn(0x7fd9100c6ac0 msgr2=0x7fd9100c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 shutdown_connections 2024-02-19T14:58:32.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:32.882+0000 7fd9151a2640 1 -- 172.21.15.78:0/606668465 wait complete. 2024-02-19T14:58:32.896 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-19T14:58:33.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.006+0000 7f1906c07640 1 Processor -- start 2024-02-19T14:58:33.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.006+0000 7f1906c07640 1 -- start start 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.006+0000 7f1906c07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19000c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f1906c07640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19000c6960 0x7f19000c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f1906c07640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19000c27e0 con 0x7f19000adde0 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f1906c07640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000c3a90 con 0x7f19000c6960 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f1906c07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000c3d70 con 0x7f1900155da0 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19000c6960 0x7f19000c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19000c6960 0x7f19000c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58750/0 (socket says 172.21.15.78:58750) 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 -- 172.21.15.78:0/182846372 learned_addr learned my addr 172.21.15.78:0/182846372 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f19049a3640 1 --2- 172.21.15.78:0/182846372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19000c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 -- 172.21.15.78:0/182846372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 msgr2=0x7f19000c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 --2- 172.21.15.78:0/182846372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19000c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 -- 172.21.15.78:0/182846372 >> v1:172.21.15.150:6789/0 conn(0x7f19000adde0 legacy=0x7f19000c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.007+0000 7f18fffff640 1 -- 172.21.15.78:0/182846372 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190006d070 con 0x7f19000c6960 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f18fffff640 1 --2- 172.21.15.78:0/182846372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000c6960 0x7f19000c7820 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009870 tx=0x7f18ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9124d363ccfae39 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f18ff7fe640 1 -- 172.21.15.78:0/182846372 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec030840 con 0x7f19000c6960 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f18ff7fe640 1 -- 172.21.15.78:0/182846372 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec0309a0 con 0x7f19000c6960 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f18ff7fe640 1 -- 172.21.15.78:0/182846372 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec030d30 con 0x7f19000c6960 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f1906c07640 1 -- 172.21.15.78:0/182846372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000c6960 msgr2=0x7f19000c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f1906c07640 1 --2- 172.21.15.78:0/182846372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000c6960 0x7f19000c7820 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009870 tx=0x7f18ec02b980 comp rx=0 tx=0).stop 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f1906c07640 1 -- 172.21.15.78:0/182846372 shutdown_connections 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f1906c07640 1 --2- 172.21.15.78:0/182846372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000c6960 0x7f19000c7820 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f1906c07640 1 --2- 172.21.15.78:0/182846372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19000c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.008+0000 7f1906c07640 1 -- 172.21.15.78:0/182846372 >> 172.21.15.78:0/182846372 conn(0x7f19000aa3e0 msgr2=0x7f19000aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.009+0000 7f1906c07640 1 -- 172.21.15.78:0/182846372 shutdown_connections 2024-02-19T14:58:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.009+0000 7f1906c07640 1 -- 172.21.15.78:0/182846372 wait complete. 2024-02-19T14:58:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.010+0000 7f1906c07640 1 Processor -- start 2024-02-19T14:58:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.010+0000 7f1906c07640 1 -- start start 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.011+0000 7f1906c07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 0x7f19001588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19000c6960 0x7f1900153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19001eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190009bca0 con 0x7f19000adde0 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900157b80 con 0x7f19000c6960 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1900158e00 con 0x7f1900155da0 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 0x7f19001588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 0x7f19001588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58766/0 (socket says 172.21.15.78:58766) 2024-02-19T14:58:33.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 -- 172.21.15.78:0/3902926173 learned_addr learned my addr 172.21.15.78:0/3902926173 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f18fffff640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19000c6960 0x7f1900153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 -- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 msgr2=0x7f19001eda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19051a4640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19001eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19001eda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 -- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19000c6960 msgr2=0x7f1900153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19000c6960 0x7f1900153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 -- 172.21.15.78:0/3902926173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0094d0 con 0x7f19000adde0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f18fffff640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19000c6960 0x7f1900153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19051a4640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19001eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f19049a3640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 0x7f19001588c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d5f0 tx=0x7f18f400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40125f0 con 0x7f19000adde0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f4012750 con 0x7f19000adde0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4012b00 con 0x7f19000adde0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190006d710 con 0x7f19000adde0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.012+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19001580b0 con 0x7f19000adde0 2024-02-19T14:58:33.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.013+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190006e120 con 0x7f19000adde0 2024-02-19T14:58:33.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.014+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f18f4012c60 con 0x7f19000adde0 2024-02-19T14:58:33.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.014+0000 7f18fd7fa640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e406b810 0x7f18e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:33.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.014+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f18f408a7b0 con 0x7f19000adde0 2024-02-19T14:58:33.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.014+0000 7f18fffff640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e406b810 0x7f18e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:33.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.015+0000 7f18fffff640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e406b810 0x7f18e406dcd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f18ec036000 tx=0x7f18ec03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:33.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.023+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f18f40550d0 con 0x7f19000adde0 2024-02-19T14:58:33.226 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:58:33.223+0000 7fce1e183640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:33.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:33.241+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f19000c3150 con 0x7f19000adde0 2024-02-19T14:58:34.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.007+0000 7f18fd7fa640 1 -- 172.21.15.78:0/3902926173 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v153) v1 ==== 125+0+0 (secure 0 0 0) 0x7f18f4058790 con 0x7f19000adde0 2024-02-19T14:58:34.011 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked down. 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e406b810 msgr2=0x7f18e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e406b810 0x7f18e406dcd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f18ec036000 tx=0x7f18ec03b040 comp rx=0 tx=0).stop 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 msgr2=0x7f19001588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 0x7f19001588c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d5f0 tx=0x7f18f400dab0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 shutdown_connections 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f18e406b810 0x7f18e406dcd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1900155da0 0x7f19001eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19000c6960 0x7f1900153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 --2- 172.21.15.78:0/3902926173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19000adde0 0x7f19001588c0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 >> 172.21.15.78:0/3902926173 conn(0x7f19000aa3e0 msgr2=0x7f19000aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:34.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 shutdown_connections 2024-02-19T14:58:34.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.009+0000 7f1906c07640 1 -- 172.21.15.78:0/3902926173 wait complete. 2024-02-19T14:58:34.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:34.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.120+0000 7f6528a88640 1 Processor -- start 2024-02-19T14:58:34.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.120+0000 7f6528a88640 1 -- start start 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6528a88640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65240c29c0 0x7f65241526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6528a88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c5be0 0x7f652406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6528a88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6524072250 con 0x7f65240c25e0 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6528a88640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6524152c90 con 0x7f65240c29c0 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6528a88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6524072480 con 0x7f65240c5be0 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6521d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65240c29c0 0x7f65241526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f652259c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f65240c25e0 0x7f65240c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47248/0 (socket says 172.21.15.78:47248) 2024-02-19T14:58:34.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f652259c640 1 -- 172.21.15.78:0/1742957830 learned_addr learned my addr 172.21.15.78:0/1742957830 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:34.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6521d9b640 1 -- 172.21.15.78:0/1742957830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c5be0 msgr2=0x7f652406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:34.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6521d9b640 1 --2- 172.21.15.78:0/1742957830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c5be0 0x7f652406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6521d9b640 1 -- 172.21.15.78:0/1742957830 >> v1:172.21.15.78:6789/0 conn(0x7f65240c25e0 legacy=0x7f65240c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f652159a640 1 -- 172.21.15.78:0/1742957830 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773460499 0 0) 0x7f6524072250 con 0x7f65240c25e0 2024-02-19T14:58:34.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.121+0000 7f6521d9b640 1 -- 172.21.15.78:0/1742957830 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65240c4b90 con 0x7f65240c29c0 2024-02-19T14:58:34.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.122+0000 7f6521d9b640 1 --2- 172.21.15.78:0/1742957830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c29c0 0x7f65241526a0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f65180097a0 tx=0x7f651802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd1e30899daa47d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:34.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.122+0000 7f652159a640 1 -- 172.21.15.78:0/1742957830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651802e070 con 0x7f65240c29c0 2024-02-19T14:58:34.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.122+0000 7f652159a640 1 -- 172.21.15.78:0/1742957830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65180315a0 con 0x7f65240c29c0 2024-02-19T14:58:34.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.122+0000 7f652159a640 1 -- 172.21.15.78:0/1742957830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518031930 con 0x7f65240c29c0 2024-02-19T14:58:34.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 -- 172.21.15.78:0/1742957830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c29c0 msgr2=0x7f65241526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 --2- 172.21.15.78:0/1742957830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c29c0 0x7f65241526a0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f65180097a0 tx=0x7f651802b840 comp rx=0 tx=0).stop 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 -- 172.21.15.78:0/1742957830 shutdown_connections 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 --2- 172.21.15.78:0/1742957830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c29c0 0x7f65241526a0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 --2- 172.21.15.78:0/1742957830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c5be0 0x7f652406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 -- 172.21.15.78:0/1742957830 >> 172.21.15.78:0/1742957830 conn(0x7f652406b270 msgr2=0x7f652406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 -- 172.21.15.78:0/1742957830 shutdown_connections 2024-02-19T14:58:34.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.123+0000 7f6528a88640 1 -- 172.21.15.78:0/1742957830 wait complete. 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.124+0000 7f6528a88640 1 Processor -- start 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.124+0000 7f6528a88640 1 -- start start 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6528a88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 0x7f65241df700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6528a88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c29c0 0x7f65241e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6528a88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65240c5be0 0x7f65241ea220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6528a88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65240a9150 con 0x7f65240c25e0 2024-02-19T14:58:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6528a88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65240a8c40 con 0x7f65240c5be0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6528a88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65240a9630 con 0x7f65240c29c0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 0x7f65241df700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 0x7f65241df700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58796/0 (socket says 172.21.15.78:58796) 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 -- 172.21.15.78:0/281387883 learned_addr learned my addr 172.21.15.78:0/281387883 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6521d9b640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c29c0 0x7f65241e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f6522d9d640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65240c5be0 0x7f65241ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 -- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c29c0 msgr2=0x7f65241e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c29c0 0x7f65241e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 -- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65240c5be0 msgr2=0x7f65241ea220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65240c5be0 0x7f65241ea220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.125+0000 7f652259c640 1 -- 172.21.15.78:0/281387883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6518009400 con 0x7f65240c25e0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.126+0000 7f652259c640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 0x7f65241df700 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f651400c430 tx=0x7f651400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.126+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65140074e0 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.126+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6514007660 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.126+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65240c0a00 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.126+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6514004a20 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.126+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65240a8e70 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.127+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6524074990 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.127+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6514004b80 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.127+0000 7f6512ffd640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f650406b810 0x7f650406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.128+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f651408a0c0 con 0x7f65240c25e0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.128+0000 7f6521d9b640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f650406b810 0x7f650406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.128+0000 7f6521d9b640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f650406b810 0x7f650406dcd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f6518036000 tx=0x7f651803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.132+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f65140549e0 con 0x7f65240c25e0 2024-02-19T14:58:34.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.337+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65240c0b90 con 0x7f65240c25e0 2024-02-19T14:58:34.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.337+0000 7f6512ffd640 1 -- 172.21.15.78:0/281387883 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f65140580a0 con 0x7f65240c25e0 2024-02-19T14:58:34.341 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:34.341 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:34.341 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f650406b810 msgr2=0x7f650406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f650406b810 0x7f650406dcd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f6518036000 tx=0x7f651803d040 comp rx=0 tx=0).stop 2024-02-19T14:58:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 msgr2=0x7f65241df700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 0x7f65241df700 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f651400c430 tx=0x7f651400c8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 shutdown_connections 2024-02-19T14:58:34.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f650406b810 0x7f650406dcd0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65240c5be0 0x7f65241ea220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65240c29c0 0x7f65241e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 --2- 172.21.15.78:0/281387883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65240c25e0 0x7f65241df700 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 >> 172.21.15.78:0/281387883 conn(0x7f652406b270 msgr2=0x7f6524074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 shutdown_connections 2024-02-19T14:58:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:34.339+0000 7f6528a88640 1 -- 172.21.15.78:0/281387883 wait complete. 2024-02-19T14:58:34.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:34.757 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:35.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.865+0000 7f3e74d48640 1 Processor -- start 2024-02-19T14:58:35.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.865+0000 7f3e74d48640 1 -- start start 2024-02-19T14:58:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e74d48640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e70072180 0x7f3e70155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e74d48640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e70072e80 con 0x7f3e7006f270 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e74d48640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7006e990 con 0x7f3e701528d0 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e74d48640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e700734c0 con 0x7f3e70072180 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e6ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3e7006f270 0x7f3e7015f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47256/0 (socket says 172.21.15.78:47256) 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e6ed9d640 1 -- 172.21.15.78:0/2792464902 learned_addr learned my addr 172.21.15.78:0/2792464902 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e6dd9b640 1 --2- 172.21.15.78:0/2792464902 >> v2:172.21.15.150:3300/0 conn(0x7f3e70072180 0x7f3e70155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949741355 0 0) 0x7f3e68001490 con 0x7f3e7006f270 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.866+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e58003520 con 0x7f3e7006f270 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2908979488 0 0) 0x7f3e58003520 con 0x7f3e7006f270 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 >> v2:172.21.15.150:3300/0 conn(0x7f3e70072180 msgr2=0x7f3e70155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 --2- 172.21.15.78:0/2792464902 >> v2:172.21.15.150:3300/0 conn(0x7f3e70072180 0x7f3e70155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 >> v1:172.21.15.150:6789/0 conn(0x7f3e701528d0 legacy=0x7f3e700c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700731e0 con 0x7f3e7006f270 2024-02-19T14:58:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3e68001b80 con 0x7f3e7006f270 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 >> v1:172.21.15.78:6789/0 conn(0x7f3e7006f270 legacy=0x7f3e7015f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e580047c0 0x7f3e58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e58005180 0x7f3e5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e700734c0 con 0x7f3e58003970 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5800c120 con 0x7f3e580047c0 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5800c2e0 con 0x7f3e58005180 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6dd9b640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e580047c0 0x7f3e58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6ed9d640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e58005180 0x7f3e5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6e59c640 1 -- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e58005180 msgr2=0x7f3e5800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e58005180 0x7f3e5800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6e59c640 1 -- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e580047c0 msgr2=0x7f3e58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e580047c0 0x7f3e58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6e59c640 1 -- 172.21.15.78:0/2792464902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e58003520 con 0x7f3e58003970 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6ed9d640 1 -- 172.21.15.78:0/2792464902 reap_dead start 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.867+0000 7f3e6dd9b640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e580047c0 0x7f3e58004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.868+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e58003d90 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f3e60001750 tx=0x7f3e580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59909acae32c86a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.868+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e600052d0 con 0x7f3e58003970 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.868+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6002e050 con 0x7f3e58003970 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.868+0000 7f3e6d59a640 1 -- 172.21.15.78:0/2792464902 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60003830 con 0x7f3e58003970 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 -- 172.21.15.78:0/2792464902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 msgr2=0x7f3e58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e58003d90 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f3e60001750 tx=0x7f3e580057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 -- 172.21.15.78:0/2792464902 shutdown_connections 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 --2- 172.21.15.78:0/2792464902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e58003d90 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 -- 172.21.15.78:0/2792464902 >> 172.21.15.78:0/2792464902 conn(0x7f3e700c7590 msgr2=0x7f3e700c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 -- 172.21.15.78:0/2792464902 shutdown_connections 2024-02-19T14:58:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.869+0000 7f3e74d48640 1 -- 172.21.15.78:0/2792464902 wait complete. 2024-02-19T14:58:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.870+0000 7f3e74d48640 1 Processor -- start 2024-02-19T14:58:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.870+0000 7f3e74d48640 1 -- start start 2024-02-19T14:58:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e74d48640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e701df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e74d48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e701e17f0 0x7f3e7014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e74d48640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e701e1f30 0x7f3e701545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e74d48640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e701dfc70 con 0x7f3e58003970 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e74d48640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70155b80 con 0x7f3e701e17f0 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e74d48640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e701558a0 con 0x7f3e701e1f30 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e701df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e701df2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58812/0 (socket says 172.21.15.78:58812) 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 -- 172.21.15.78:0/3612869254 learned_addr learned my addr 172.21.15.78:0/3612869254 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6dd9b640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e701e17f0 0x7f3e7014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6ed9d640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e701e1f30 0x7f3e701545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 -- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e701e1f30 msgr2=0x7f3e701545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e701e1f30 0x7f3e701545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 -- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e701e17f0 msgr2=0x7f3e7014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e701e17f0 0x7f3e7014ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 -- 172.21.15.78:0/3612869254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e701de790 con 0x7f3e58003970 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e6e59c640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e701df2c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3e60000c00 tx=0x7f3e60001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e600076c0 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.871+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e60007a30 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.872+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60006be0 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.872+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e701dea70 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.872+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e701df800 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.873+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e701de5f0 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.873+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3e60006d40 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.873+0000 7f3e5effd640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5006b810 0x7f3e5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.874+0000 7f3e6dd9b640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5006b810 0x7f3e5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.874+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3e600abf10 con 0x7f3e58003970 2024-02-19T14:58:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.874+0000 7f3e6dd9b640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5006b810 0x7f3e5006dcd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3e70156360 tx=0x7f3e64008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:35.878+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3e701de5f0 con 0x7f3e58003970 2024-02-19T14:58:36.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.090+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e701de5f0 con 0x7f3e58003970 2024-02-19T14:58:36.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.091+0000 7f3e5effd640 1 -- 172.21.15.78:0/3612869254 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f3e701de5f0 con 0x7f3e58003970 2024-02-19T14:58:36.095 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:36.095 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:36.095 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5006b810 msgr2=0x7f3e5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5006b810 0x7f3e5006dcd0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3e70156360 tx=0x7f3e64008040 comp rx=0 tx=0).stop 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 msgr2=0x7f3e701df2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e701df2c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3e60000c00 tx=0x7f3e60001120 comp rx=0 tx=0).stop 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 shutdown_connections 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5006b810 0x7f3e5006dcd0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e701e1f30 0x7f3e701545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e701e17f0 0x7f3e7014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:36.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 --2- 172.21.15.78:0/3612869254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e58003970 0x7f3e701df2c0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:36.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 >> 172.21.15.78:0/3612869254 conn(0x7f3e700c7590 msgr2=0x7f3e700c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:36.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 shutdown_connections 2024-02-19T14:58:36.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:36.093+0000 7f3e74d48640 1 -- 172.21.15.78:0/3612869254 wait complete. 2024-02-19T14:58:36.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:36.510 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:37.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:37.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 Processor -- start 2024-02-19T14:58:37.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 -- start start 2024-02-19T14:58:37.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f769c152670 0x7f769c071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:37.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152a50 0x7f769c155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:37.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c071bf0 con 0x7f769c0702d0 2024-02-19T14:58:37.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c158a20 con 0x7f769c152670 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.617+0000 7f76a2007640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c155550 con 0x7f769c152a50 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f769c152670 0x7f769c071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f769c0702d0 0x7f769c158eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35816/0 (socket says 172.21.15.78:35816) 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769bfff640 1 -- 172.21.15.78:0/3505842780 learned_addr learned my addr 172.21.15.78:0/3505842780 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769b7fe640 1 -- 172.21.15.78:0/3505842780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152a50 msgr2=0x7f769c155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769b7fe640 1 --2- 172.21.15.78:0/3505842780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152a50 0x7f769c155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769b7fe640 1 -- 172.21.15.78:0/3505842780 >> v1:172.21.15.78:6789/0 conn(0x7f769c0702d0 legacy=0x7f769c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769b7fe640 1 -- 172.21.15.78:0/3505842780 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0c6780 con 0x7f769c152670 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.618+0000 7f769affd640 1 -- 172.21.15.78:0/3505842780 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650421343 0 0) 0x7f769c071bf0 con 0x7f769c0702d0 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f769b7fe640 1 --2- 172.21.15.78:0/3505842780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152670 0x7f769c071050 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f768c0097a0 tx=0x7f768c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fc9b52d481bcc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f769affd640 1 -- 172.21.15.78:0/3505842780 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c030610 con 0x7f769c152670 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f769affd640 1 -- 172.21.15.78:0/3505842780 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c030770 con 0x7f769c152670 2024-02-19T14:58:37.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f769affd640 1 -- 172.21.15.78:0/3505842780 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c030b00 con 0x7f769c152670 2024-02-19T14:58:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 -- 172.21.15.78:0/3505842780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152670 msgr2=0x7f769c071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 --2- 172.21.15.78:0/3505842780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152670 0x7f769c071050 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f768c0097a0 tx=0x7f768c02b960 comp rx=0 tx=0).stop 2024-02-19T14:58:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 -- 172.21.15.78:0/3505842780 shutdown_connections 2024-02-19T14:58:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 --2- 172.21.15.78:0/3505842780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152a50 0x7f769c155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 --2- 172.21.15.78:0/3505842780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152670 0x7f769c071050 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 -- 172.21.15.78:0/3505842780 >> 172.21.15.78:0/3505842780 conn(0x7f769c0c85e0 msgr2=0x7f769c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 -- 172.21.15.78:0/3505842780 shutdown_connections 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.619+0000 7f76a2007640 1 -- 172.21.15.78:0/3505842780 wait complete. 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.620+0000 7f76a2007640 1 Processor -- start 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 -- start start 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c0702d0 0x7f769c1ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152670 0x7f769c151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 0x7f769c1ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1eb750 con 0x7f769c152a50 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1eac90 con 0x7f769c0702d0 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f76a2007640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1ee0d0 con 0x7f769c152670 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f769bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 0x7f769c1ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f769bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 0x7f769c1ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36056/0 (socket says 172.21.15.78:36056) 2024-02-19T14:58:37.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.621+0000 7f769bfff640 1 -- 172.21.15.78:0/2844577686 learned_addr learned my addr 172.21.15.78:0/2844577686 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769bfff640 1 -- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152670 msgr2=0x7f769c151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f7692ffd640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152670 0x7f769c151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769b7fe640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c0702d0 0x7f769c1ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769bfff640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152670 0x7f769c151840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769bfff640 1 -- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c0702d0 msgr2=0x7f769c1ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769bfff640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c0702d0 0x7f769c1ea750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769bfff640 1 -- 172.21.15.78:0/2844577686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c009400 con 0x7f769c152a50 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f769bfff640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 0x7f769c1ee560 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f769400c700 tx=0x7f769400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769400ce10 con 0x7f769c152a50 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769400a640 con 0x7f769c152a50 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769400aa70 con 0x7f769c152a50 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c0ae650 con 0x7f769c152a50 2024-02-19T14:58:37.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.622+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c1ddc60 con 0x7f769c152a50 2024-02-19T14:58:37.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.623+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c06e160 con 0x7f769c152a50 2024-02-19T14:58:37.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.624+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f769400abd0 con 0x7f769c152a50 2024-02-19T14:58:37.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.624+0000 7f7698ff9640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f767c06b840 0x7f767c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:37.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.624+0000 7f769b7fe640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f767c06b840 0x7f767c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:37.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.624+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7694089a30 con 0x7f769c152a50 2024-02-19T14:58:37.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.625+0000 7f769b7fe640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f767c06b840 0x7f767c06dd00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f768c036000 tx=0x7f768c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:37.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.629+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7694054350 con 0x7f769c152a50 2024-02-19T14:58:37.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.838+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f769c0676a0 con 0x7f769c152a50 2024-02-19T14:58:37.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.839+0000 7f7698ff9640 1 -- 172.21.15.78:0/2844577686 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f7694018020 con 0x7f769c152a50 2024-02-19T14:58:37.843 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:37.843 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:37.843 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f767c06b840 msgr2=0x7f767c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f767c06b840 0x7f767c06dd00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f768c036000 tx=0x7f768c03b040 comp rx=0 tx=0).stop 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 msgr2=0x7f769c1ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 0x7f769c1ee560 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f769400c700 tx=0x7f769400cbc0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 shutdown_connections 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f767c06b840 0x7f767c06dd00 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f769c152a50 0x7f769c1ee560 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f769c152670 0x7f769c151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 --2- 172.21.15.78:0/2844577686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c0702d0 0x7f769c1ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 >> 172.21.15.78:0/2844577686 conn(0x7f769c0c85e0 msgr2=0x7f769c157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 shutdown_connections 2024-02-19T14:58:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:37.841+0000 7f76a2007640 1 -- 172.21.15.78:0/2844577686 wait complete. 2024-02-19T14:58:38.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:38.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:39.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:39.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.365+0000 7f9e65dbc640 1 Processor -- start 2024-02-19T14:58:39.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.365+0000 7f9e65dbc640 1 -- start start 2024-02-19T14:58:39.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e65dbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 0x7f9e600c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e65dbc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e60159130 con 0x7f9e60155070 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e65dbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e600ad030 con 0x7f9e600718f0 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e65dbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600c2cf0 con 0x7f9e600acc50 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9e60155070 0x7f9e601554d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35832/0 (socket says 172.21.15.78:35832) 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5effd640 1 -- 172.21.15.78:0/3631333090 learned_addr learned my addr 172.21.15.78:0/3631333090 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995077223 0 0) 0x7f9e60159130 con 0x7f9e60155070 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e48003520 con 0x7f9e60155070 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5f7fe640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 0x7f9e600c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288010862 0 0) 0x7f9e48003520 con 0x7f9e60155070 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 msgr2=0x7f9e600c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 0x7f9e600c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 >> v1:172.21.15.150:6789/0 conn(0x7f9e600718f0 legacy=0x7f9e60071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e601595c0 con 0x7f9e60155070 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9e54001670 con 0x7f9e60155070 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 >> v1:172.21.15.78:6789/0 conn(0x7f9e60155070 legacy=0x7f9e601554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 0x7f9e480039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e48003f20 0x7f9e480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e48004d00 0x7f9e48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600c2cf0 con 0x7f9e48004d00 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800bb10 con 0x7f9e48003f20 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4800bcd0 con 0x7f9e600acc50 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e48004d00 0x7f9e48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5effd640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e48003f20 0x7f9e480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5f7fe640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 0x7f9e480039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 msgr2=0x7f9e480039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e600acc50 0x7f9e480039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e48003f20 msgr2=0x7f9e480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e48003f20 0x7f9e480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.367+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3631333090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48003520 con 0x7f9e48004d00 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.368+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e48004d00 0x7f9e48005180 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f9e58001e50 tx=0x7f9e480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7401093613ac47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.368+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58003b70 con 0x7f9e48004d00 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.368+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e58003cd0 con 0x7f9e48004d00 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.368+0000 7f9e5e7fc640 1 -- 172.21.15.78:0/3631333090 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58005310 con 0x7f9e48004d00 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3631333090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e48004d00 msgr2=0x7f9e48005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3631333090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e48004d00 0x7f9e48005180 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f9e58001e50 tx=0x7f9e480057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3631333090 reap_dead start 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3631333090 shutdown_connections 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3631333090 >> 172.21.15.78:0/3631333090 conn(0x7f9e600c7800 msgr2=0x7f9e600c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3631333090 shutdown_connections 2024-02-19T14:58:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.369+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3631333090 wait complete. 2024-02-19T14:58:39.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.370+0000 7f9e65dbc640 1 Processor -- start 2024-02-19T14:58:39.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.370+0000 7f9e65dbc640 1 -- start start 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e601ede20 0x7f9e601ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e60137e50 0x7f9e601382d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 0x7f9e600c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600c2030 con 0x7f9e60138810 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60133c70 con 0x7f9e60137e50 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e601336d0 con 0x7f9e601ede20 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 0x7f9e600c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 0x7f9e600c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36074/0 (socket says 172.21.15.78:36074) 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3053921498 learned_addr learned my addr 172.21.15.78:0/3053921498 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5effd640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e60137e50 0x7f9e601382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5f7fe640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e601ede20 0x7f9e601ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e601ede20 msgr2=0x7f9e601ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e601ede20 0x7f9e601ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e60137e50 msgr2=0x7f9e601382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e60137e50 0x7f9e601382d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 -- 172.21.15.78:0/3053921498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e601ee8a0 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5f7fe640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e601ede20 0x7f9e601ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e5ffff640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 0x7f9e600c49d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f9e58001f40 tx=0x7f9e580014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5803e070 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e58007db0 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e60133990 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58006d90 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.371+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e600c6980 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.372+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6006c820 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.373+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9e58041070 con 0x7f9e60138810 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.373+0000 7f9e47fff640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e4c06b840 0x7f9e4c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.373+0000 7f9e5f7fe640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e4c06b840 0x7f9e4c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.374+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9e580ab4c0 con 0x7f9e60138810 2024-02-19T14:58:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.374+0000 7f9e5f7fe640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e4c06b840 0x7f9e4c06dd00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9e6013d8b0 tx=0x7f9e50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.378+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9e58075e00 con 0x7f9e60138810 2024-02-19T14:58:39.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.586+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e60074990 con 0x7f9e60138810 2024-02-19T14:58:39.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.587+0000 7f9e47fff640 1 -- 172.21.15.78:0/3053921498 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f9e580794c0 con 0x7f9e60138810 2024-02-19T14:58:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:39.590 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:39.590 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e4c06b840 msgr2=0x7f9e4c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e4c06b840 0x7f9e4c06dd00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9e6013d8b0 tx=0x7f9e50008040 comp rx=0 tx=0).stop 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 msgr2=0x7f9e600c49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 0x7f9e600c49d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f9e58001f40 tx=0x7f9e580014a0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 shutdown_connections 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e4c06b840 0x7f9e4c06dd00 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e60138810 0x7f9e600c49d0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e60137e50 0x7f9e601382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 --2- 172.21.15.78:0/3053921498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e601ede20 0x7f9e601ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:39.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 >> 172.21.15.78:0/3053921498 conn(0x7f9e600c7800 msgr2=0x7f9e60155490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:39.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 shutdown_connections 2024-02-19T14:58:39.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:39.589+0000 7f9e65dbc640 1 -- 172.21.15.78:0/3053921498 wait complete. 2024-02-19T14:58:40.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:40.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:41.007 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:41.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.116+0000 7fc479d0c640 1 Processor -- start 2024-02-19T14:58:41.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.116+0000 7fc479d0c640 1 -- start start 2024-02-19T14:58:41.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc479d0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c8500 0x7fc4740c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc479d0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc4740c42e0 0x7fc474158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc479d0c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740c34a0 con 0x7fc4740c2f50 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc479d0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47406c090 con 0x7fc4740c42e0 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc479d0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47406d6a0 con 0x7fc4740c8500 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc4740c42e0 0x7fc474158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc4740c42e0 0x7fc474158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36082/0 (socket says 172.21.15.78:36082) 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 -- 172.21.15.78:0/2085505103 learned_addr learned my addr 172.21.15.78:0/2085505103 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc4737fe640 1 --2- 172.21.15.78:0/2085505103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c8500 0x7fc4740c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 -- 172.21.15.78:0/2085505103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c8500 msgr2=0x7fc4740c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 --2- 172.21.15.78:0/2085505103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c8500 0x7fc4740c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 -- 172.21.15.78:0/2085505103 >> v1:172.21.15.150:6789/0 conn(0x7fc4740c2f50 legacy=0x7fc474158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:41.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.117+0000 7fc472ffd640 1 -- 172.21.15.78:0/2085505103 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47406c8e0 con 0x7fc4740c42e0 2024-02-19T14:58:41.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.118+0000 7fc472ffd640 1 --2- 172.21.15.78:0/2085505103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4740c42e0 0x7fc474158650 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fc468009870 tx=0x7fc46802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af0a7f6cf850438 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:41.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.118+0000 7fc4727fc640 1 -- 172.21.15.78:0/2085505103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4680317c0 con 0x7fc4740c42e0 2024-02-19T14:58:41.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.118+0000 7fc4727fc640 1 -- 172.21.15.78:0/2085505103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc468031920 con 0x7fc4740c42e0 2024-02-19T14:58:41.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.118+0000 7fc4727fc640 1 -- 172.21.15.78:0/2085505103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc468031cb0 con 0x7fc4740c42e0 2024-02-19T14:58:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 -- 172.21.15.78:0/2085505103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4740c42e0 msgr2=0x7fc474158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:41.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2085505103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4740c42e0 0x7fc474158650 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fc468009870 tx=0x7fc46802b860 comp rx=0 tx=0).stop 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 -- 172.21.15.78:0/2085505103 shutdown_connections 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2085505103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4740c42e0 0x7fc474158650 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2085505103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c8500 0x7fc4740c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 -- 172.21.15.78:0/2085505103 >> 172.21.15.78:0/2085505103 conn(0x7fc4740aa300 msgr2=0x7fc4740aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 -- 172.21.15.78:0/2085505103 shutdown_connections 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.119+0000 7fc479d0c640 1 -- 172.21.15.78:0/2085505103 wait complete. 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.120+0000 7fc479d0c640 1 Processor -- start 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.120+0000 7fc479d0c640 1 -- start start 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c2f50 0x7fc47414de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4740c8500 0x7fc4741eb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 0x7fc4741eb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:41.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740acf30 con 0x7fc4741503f0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740ad210 con 0x7fc4740c8500 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4741eac00 con 0x7fc4740c2f50 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 0x7fc4741eb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 0x7fc4741eb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36086/0 (socket says 172.21.15.78:36086) 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 -- 172.21.15.78:0/2104625068 learned_addr learned my addr 172.21.15.78:0/2104625068 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc4737fe640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c2f50 0x7fc47414de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc472ffd640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4740c8500 0x7fc4741eb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 -- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c2f50 msgr2=0x7fc47414de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c2f50 0x7fc47414de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 -- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4740c8500 msgr2=0x7fc4741eb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4740c8500 0x7fc4741eb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 -- 172.21.15.78:0/2104625068 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4680094d0 con 0x7fc4741503f0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc473fff640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 0x7fc4741eb660 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00c780 tx=0x7fc46c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c004b60 con 0x7fc4741503f0 2024-02-19T14:58:41.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c00ce90 con 0x7fc4741503f0 2024-02-19T14:58:41.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47414e3a0 con 0x7fc4741503f0 2024-02-19T14:58:41.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00a850 con 0x7fc4741503f0 2024-02-19T14:58:41.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.121+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc474152340 con 0x7fc4741503f0 2024-02-19T14:58:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.123+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc474071c20 con 0x7fc4741503f0 2024-02-19T14:58:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.123+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc46c00a9b0 con 0x7fc4741503f0 2024-02-19T14:58:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.123+0000 7fc45bfff640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc46006b810 0x7fc46006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.124+0000 7fc4737fe640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc46006b810 0x7fc46006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.124+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc46c08a3d0 con 0x7fc4741503f0 2024-02-19T14:58:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.124+0000 7fc4737fe640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc46006b810 0x7fc46006dcd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc47414e760 tx=0x7fc464008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:41.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.128+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc46c054d10 con 0x7fc4741503f0 2024-02-19T14:58:41.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.338+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc474074990 con 0x7fc4741503f0 2024-02-19T14:58:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.339+0000 7fc45bfff640 1 -- 172.21.15.78:0/2104625068 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7fc46c0583d0 con 0x7fc4741503f0 2024-02-19T14:58:41.342 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:41.342 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc46006b810 msgr2=0x7fc46006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc46006b810 0x7fc46006dcd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc47414e760 tx=0x7fc464008040 comp rx=0 tx=0).stop 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 msgr2=0x7fc4741eb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 0x7fc4741eb660 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00c780 tx=0x7fc46c00cc40 comp rx=0 tx=0).stop 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 shutdown_connections 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc46006b810 0x7fc46006dcd0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4741503f0 0x7fc4741eb660 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4740c8500 0x7fc4741eb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 --2- 172.21.15.78:0/2104625068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4740c2f50 0x7fc47414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 >> 172.21.15.78:0/2104625068 conn(0x7fc4740aa300 msgr2=0x7fc4740c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 shutdown_connections 2024-02-19T14:58:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:41.341+0000 7fc479d0c640 1 -- 172.21.15.78:0/2104625068 wait complete. 2024-02-19T14:58:41.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:41.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:42.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:42.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.873+0000 7f5ac3348640 1 Processor -- start 2024-02-19T14:58:42.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.873+0000 7f5ac3348640 1 -- start start 2024-02-19T14:58:42.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac3348640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5abc158c60 0x7f5abc1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac3348640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc072ec0 con 0x7f5abc0730a0 2024-02-19T14:58:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac3348640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc06f270 con 0x7f5abc152610 2024-02-19T14:58:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac3348640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5abc06fa40 con 0x7f5abc158c60 2024-02-19T14:58:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac08e3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5abc0730a0 0x7f5abc158550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35834/0 (socket says 172.21.15.78:35834) 2024-02-19T14:58:42.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac08e3640 1 -- 172.21.15.78:0/3351281490 learned_addr learned my addr 172.21.15.78:0/3351281490 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.874+0000 7f5ac18e5640 1 --2- 172.21.15.78:0/3351281490 >> v2:172.21.15.150:3300/0 conn(0x7f5abc158c60 0x7f5abc1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.875+0000 7f5ab3fff640 1 -- 172.21.15.78:0/3351281490 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032839166 0 0) 0x7f5abc072ec0 con 0x7f5abc0730a0 2024-02-19T14:58:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.875+0000 7f5ab3fff640 1 -- 172.21.15.78:0/3351281490 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8003520 con 0x7f5abc0730a0 2024-02-19T14:58:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.875+0000 7f5ac18e5640 1 -- 172.21.15.78:0/3351281490 >> v1:172.21.15.150:6789/0 conn(0x7f5abc152610 legacy=0x7f5abc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.875+0000 7f5ac18e5640 1 -- 172.21.15.78:0/3351281490 >> v1:172.21.15.78:6789/0 conn(0x7f5abc0730a0 legacy=0x7f5abc158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:42.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.875+0000 7f5ac18e5640 1 -- 172.21.15.78:0/3351281490 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc06f4c0 con 0x7f5abc158c60 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ac18e5640 1 --2- 172.21.15.78:0/3351281490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc158c60 0x7f5abc1608b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009dc0 tx=0x7f5ab402be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f93aedc4b2b85bbb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ab3fff640 1 -- 172.21.15.78:0/3351281490 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab402cc40 con 0x7f5abc158c60 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ab3fff640 1 -- 172.21.15.78:0/3351281490 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab402cda0 con 0x7f5abc158c60 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ab3fff640 1 -- 172.21.15.78:0/3351281490 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4034640 con 0x7f5abc158c60 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ac3348640 1 -- 172.21.15.78:0/3351281490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc158c60 msgr2=0x7f5abc1608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ac3348640 1 --2- 172.21.15.78:0/3351281490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc158c60 0x7f5abc1608b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009dc0 tx=0x7f5ab402be90 comp rx=0 tx=0).stop 2024-02-19T14:58:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ac3348640 1 -- 172.21.15.78:0/3351281490 shutdown_connections 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ac3348640 1 --2- 172.21.15.78:0/3351281490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc158c60 0x7f5abc1608b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.876+0000 7f5ac3348640 1 -- 172.21.15.78:0/3351281490 >> 172.21.15.78:0/3351281490 conn(0x7f5abc0c7590 msgr2=0x7f5abc0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.877+0000 7f5ac3348640 1 -- 172.21.15.78:0/3351281490 shutdown_connections 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.877+0000 7f5ac3348640 1 -- 172.21.15.78:0/3351281490 wait complete. 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 Processor -- start 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 -- start start 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 0x7f5abc0ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc152610 0x7f5abc1ea630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5abc158c60 0x7f5abc14e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc0ac0c0 con 0x7f5abc0730a0 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc0ad6f0 con 0x7f5abc152610 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac3348640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc0a9370 con 0x7f5abc158c60 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.878+0000 7f5ac10e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 0x7f5abc0ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 0x7f5abc0ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36090/0 (socket says 172.21.15.78:36090) 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 -- 172.21.15.78:0/2633522881 learned_addr learned my addr 172.21.15.78:0/2633522881 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac08e3640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc152610 0x7f5abc1ea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac18e5640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5abc158c60 0x7f5abc14e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 -- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5abc158c60 msgr2=0x7f5abc14e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5abc158c60 0x7f5abc14e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 -- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc152610 msgr2=0x7f5abc1ea630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc152610 0x7f5abc1ea630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 -- 172.21.15.78:0/2633522881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4009a70 con 0x7f5abc0730a0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac10e4640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 0x7f5abc0ad1b0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c400 tx=0x7f5ab800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80052d0 con 0x7f5abc0730a0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab8005450 con 0x7f5abc0730a0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc1eb2c0 con 0x7f5abc0730a0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.879+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8007260 con 0x7f5abc0730a0 2024-02-19T14:58:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.880+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc1eafe0 con 0x7f5abc0730a0 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.881+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc074990 con 0x7f5abc0730a0 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.881+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5ab80073c0 con 0x7f5abc0730a0 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.881+0000 7f5ab1ffb640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5aa006b810 0x7f5aa006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.882+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f5ab8089820 con 0x7f5abc0730a0 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.882+0000 7f5ac08e3640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5aa006b810 0x7f5aa006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.882+0000 7f5ac08e3640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5aa006b810 0x7f5aa006dcd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5abc0c2d40 tx=0x7f5aac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:42.886+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5ab8054060 con 0x7f5abc0730a0 2024-02-19T14:58:43.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.098+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5abc1553c0 con 0x7f5abc0730a0 2024-02-19T14:58:43.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.098+0000 7f5ab1ffb640 1 -- 172.21.15.78:0/2633522881 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f5ab8057720 con 0x7f5abc0730a0 2024-02-19T14:58:43.101 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:43.101 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:43.102 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:43.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5aa006b810 msgr2=0x7f5aa006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5aa006b810 0x7f5aa006dcd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5abc0c2d40 tx=0x7f5aac009040 comp rx=0 tx=0).stop 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 msgr2=0x7f5abc0ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 0x7f5abc0ad1b0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c400 tx=0x7f5ab800c8c0 comp rx=0 tx=0).stop 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 shutdown_connections 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5aa006b810 0x7f5aa006dcd0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5abc158c60 0x7f5abc14e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5abc152610 0x7f5abc1ea630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 --2- 172.21.15.78:0/2633522881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5abc0730a0 0x7f5abc0ad1b0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 >> 172.21.15.78:0/2633522881 conn(0x7f5abc0c7590 msgr2=0x7f5abc0c3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 shutdown_connections 2024-02-19T14:58:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:43.100+0000 7f5ac3348640 1 -- 172.21.15.78:0/2633522881 wait complete. 2024-02-19T14:58:43.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:43.516 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:44.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:44.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.626+0000 7f694c8ed640 1 Processor -- start 2024-02-19T14:58:44.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.626+0000 7f694c8ed640 1 -- start start 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.627+0000 7f694c8ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 0x7f6948070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.627+0000 7f694c8ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69480ae670 con 0x7f69480c4980 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.627+0000 7f694c8ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69480716b0 con 0x7f69480add70 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.627+0000 7f694c8ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69480c33b0 con 0x7f69480c2f60 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.627+0000 7f6945d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f69480c4980 0x7f69480705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35838/0 (socket says 172.21.15.78:35838) 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.627+0000 7f6945d9b640 1 -- 172.21.15.78:0/271255375 learned_addr learned my addr 172.21.15.78:0/271255375 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f6946d9d640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 0x7f6948070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181601595 0 0) 0x7f69480ae670 con 0x7f69480c4980 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6930003520 con 0x7f69480c4980 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236041428 0 0) 0x7f6930003520 con 0x7f69480c4980 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 msgr2=0x7f6948070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 0x7f6948070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 >> v1:172.21.15.150:6789/0 conn(0x7f69480add70 legacy=0x7f694806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948153780 con 0x7f69480c4980 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f693c001670 con 0x7f69480c4980 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 >> v1:172.21.15.78:6789/0 conn(0x7f69480c4980 legacy=0x7f69480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 0x7f69300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6930004d00 0x7f6930005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69480c33b0 con 0x7f6930004d00 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693000bab0 con 0x7f69300039a0 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693000bc70 con 0x7f69480c2f60 2024-02-19T14:58:44.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f6946d9d640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6930004d00 0x7f6930005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f694659c640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f6945d9b640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 0x7f69300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f6946d9d640 1 -- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 msgr2=0x7f69300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.628+0000 7f6946d9d640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69480c2f60 0x7f69300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f6946d9d640 1 -- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69300039a0 msgr2=0x7f6930003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f6946d9d640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69300039a0 0x7f6930003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f6946d9d640 1 -- 172.21.15.78:0/271255375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930003520 con 0x7f6930004d00 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f6946d9d640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6930004d00 0x7f6930005180 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6940009ac0 tx=0x7f69300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521bd661fac17926 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694002c6b0 con 0x7f6930004d00 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694002c810 con 0x7f6930004d00 2024-02-19T14:58:44.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.629+0000 7f694559a640 1 -- 172.21.15.78:0/271255375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694002cba0 con 0x7f6930004d00 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f694c8ed640 1 -- 172.21.15.78:0/271255375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6930004d00 msgr2=0x7f6930005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f694c8ed640 1 --2- 172.21.15.78:0/271255375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6930004d00 0x7f6930005180 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6940009ac0 tx=0x7f69300057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f6946d9d640 1 -- 172.21.15.78:0/271255375 reap_dead start 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f694c8ed640 1 -- 172.21.15.78:0/271255375 shutdown_connections 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f694c8ed640 1 -- 172.21.15.78:0/271255375 >> 172.21.15.78:0/271255375 conn(0x7f69480c85e0 msgr2=0x7f69480c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f694c8ed640 1 -- 172.21.15.78:0/271255375 shutdown_connections 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.630+0000 7f694c8ed640 1 -- 172.21.15.78:0/271255375 wait complete. 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.631+0000 7f694c8ed640 1 Processor -- start 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.631+0000 7f694c8ed640 1 -- start start 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694c8ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6930004d00 0x7f69481524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694c8ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480bfbd0 0x7f694814a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694c8ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 0x7f69481544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694c8ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6948154050 con 0x7f694814ae50 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694c8ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69480bd4e0 con 0x7f69480bfbd0 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694c8ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6948155c40 con 0x7f6930004d00 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 0x7f69481544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 0x7f69481544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36116/0 (socket says 172.21.15.78:36116) 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 -- 172.21.15.78:0/2675728020 learned_addr learned my addr 172.21.15.78:0/2675728020 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6945d9b640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480bfbd0 0x7f694814a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f694659c640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6930004d00 0x7f69481524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 -- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6930004d00 msgr2=0x7f69481524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6930004d00 0x7f69481524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 -- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480bfbd0 msgr2=0x7f694814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480bfbd0 0x7f694814a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.632+0000 7f6946d9d640 1 -- 172.21.15.78:0/2675728020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948155f20 con 0x7f694814ae50 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.633+0000 7f6946d9d640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 0x7f69481544a0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6940009be0 tx=0x7f6940004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:44.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.633+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940003810 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.633+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940003990 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.633+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940003d40 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.633+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6948156200 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.633+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69481e6080 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.634+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69481541c0 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.634+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f694003e440 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.635+0000 7f6936ffd640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f692806b840 0x7f692806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.635+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f6940000e40 con 0x7f694814ae50 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.635+0000 7f694659c640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f692806b840 0x7f692806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:44.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.635+0000 7f694659c640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f692806b840 0x7f692806dd00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6938003fb0 tx=0x7f6938009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:44.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.639+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f69481541c0 con 0x7f694814ae50 2024-02-19T14:58:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.844+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6948074990 con 0x7f694814ae50 2024-02-19T14:58:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.845+0000 7f6936ffd640 1 -- 172.21.15.78:0/2675728020 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f69400761d0 con 0x7f694814ae50 2024-02-19T14:58:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:44.848 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:44.848 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f692806b840 msgr2=0x7f692806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f692806b840 0x7f692806dd00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6938003fb0 tx=0x7f6938009040 comp rx=0 tx=0).stop 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 msgr2=0x7f69481544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 0x7f69481544a0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6940009be0 tx=0x7f6940004730 comp rx=0 tx=0).stop 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 shutdown_connections 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f692806b840 0x7f692806dd00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f694814ae50 0x7f69481544a0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480bfbd0 0x7f694814a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 --2- 172.21.15.78:0/2675728020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6930004d00 0x7f69481524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 >> 172.21.15.78:0/2675728020 conn(0x7f69480c85e0 msgr2=0x7f694814e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 shutdown_connections 2024-02-19T14:58:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:44.847+0000 7f694c8ed640 1 -- 172.21.15.78:0/2675728020 wait complete. 2024-02-19T14:58:45.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:45.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:46.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:46.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.370+0000 7f05e4bd7640 1 Processor -- start 2024-02-19T14:58:46.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.370+0000 7f05e4bd7640 1 -- start start 2024-02-19T14:58:46.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05e4bd7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05e00c4980 0x7f05e00c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:46.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05e4bd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e00ae670 con 0x7f05e00705b0 2024-02-19T14:58:46.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05e4bd7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05e00716b0 con 0x7f05e00add70 2024-02-19T14:58:46.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05e4bd7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05e0153780 con 0x7f05e00c4980 2024-02-19T14:58:46.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05ddd9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05e00c4980 0x7f05e00c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:46.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05ded9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f05e00705b0 0x7f05e00709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35844/0 (socket says 172.21.15.78:35844) 2024-02-19T14:58:46.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.371+0000 7f05ded9d640 1 -- 172.21.15.78:0/2379336539 learned_addr learned my addr 172.21.15.78:0/2379336539 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:46.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.372+0000 7f05ddd9b640 1 -- 172.21.15.78:0/2379336539 >> v1:172.21.15.150:6789/0 conn(0x7f05e00add70 legacy=0x7f05e006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:46.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.372+0000 7f05ddd9b640 1 -- 172.21.15.78:0/2379336539 >> v1:172.21.15.78:6789/0 conn(0x7f05e00705b0 legacy=0x7f05e00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:46.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.372+0000 7f05ddd9b640 1 -- 172.21.15.78:0/2379336539 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e0153a60 con 0x7f05e00c4980 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.372+0000 7f05ddd9b640 1 --2- 172.21.15.78:0/2379336539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00c4980 0x7f05e00c2f60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f05d40097a0 tx=0x7f05d402b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6361a5cfb2491b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05dd59a640 1 -- 172.21.15.78:0/2379336539 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4030710 con 0x7f05e00c4980 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05dd59a640 1 -- 172.21.15.78:0/2379336539 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d402e050 con 0x7f05e00c4980 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05dd59a640 1 -- 172.21.15.78:0/2379336539 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4030a90 con 0x7f05e00c4980 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 -- 172.21.15.78:0/2379336539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00c4980 msgr2=0x7f05e00c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/2379336539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00c4980 0x7f05e00c2f60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f05d40097a0 tx=0x7f05d402b680 comp rx=0 tx=0).stop 2024-02-19T14:58:46.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 -- 172.21.15.78:0/2379336539 shutdown_connections 2024-02-19T14:58:46.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/2379336539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00c4980 0x7f05e00c2f60 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 -- 172.21.15.78:0/2379336539 >> 172.21.15.78:0/2379336539 conn(0x7f05e00c85e0 msgr2=0x7f05e00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:46.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 -- 172.21.15.78:0/2379336539 shutdown_connections 2024-02-19T14:58:46.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.373+0000 7f05e4bd7640 1 -- 172.21.15.78:0/2379336539 wait complete. 2024-02-19T14:58:46.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 Processor -- start 2024-02-19T14:58:46.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 -- start start 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 0x7f05e01df250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05e00add70 0x7f05e00c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05e00c4980 0x7f05e01eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e01ea800 con 0x7f05e00705b0 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e006e030 con 0x7f05e00c4980 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05e4bd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e00c78f0 con 0x7f05e00add70 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05de59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 0x7f05e01df250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05de59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 0x7f05e01df250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36136/0 (socket says 172.21.15.78:36136) 2024-02-19T14:58:46.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.375+0000 7f05de59c640 1 -- 172.21.15.78:0/4068824988 learned_addr learned my addr 172.21.15.78:0/4068824988 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05ded9d640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05e00c4980 0x7f05e01eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05de59c640 1 -- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05e00add70 msgr2=0x7f05e00c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05ddd9b640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05e00add70 0x7f05e00c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05de59c640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05e00add70 0x7f05e00c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05de59c640 1 -- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05e00c4980 msgr2=0x7f05e01eaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05de59c640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05e00c4980 0x7f05e01eaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05de59c640 1 -- 172.21.15.78:0/4068824988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d4009400 con 0x7f05e00705b0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05ded9d640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05e00c4980 0x7f05e01eaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05de59c640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 0x7f05e01df250 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f05d000c7c0 tx=0x7f05d000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d000aa50 con 0x7f05e00705b0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d000abb0 con 0x7f05e00705b0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e00c7610 con 0x7f05e00705b0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0012420 con 0x7f05e00705b0 2024-02-19T14:58:46.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.376+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e01eaa10 con 0x7f05e00705b0 2024-02-19T14:58:46.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.377+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e01eac00 con 0x7f05e00705b0 2024-02-19T14:58:46.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.378+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f05d0012580 con 0x7f05e00705b0 2024-02-19T14:58:46.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.378+0000 7f05ceffd640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05c006b810 0x7f05c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:46.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.378+0000 7f05ddd9b640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05c006b810 0x7f05c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:46.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.378+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f05d008a290 con 0x7f05e00705b0 2024-02-19T14:58:46.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.378+0000 7f05ddd9b640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05c006b810 0x7f05c006dcd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f05d4034000 tx=0x7f05d4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:46.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.382+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05d0054b60 con 0x7f05e00705b0 2024-02-19T14:58:46.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.587+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05e0074990 con 0x7f05e00705b0 2024-02-19T14:58:46.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.588+0000 7f05ceffd640 1 -- 172.21.15.78:0/4068824988 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f05d0058220 con 0x7f05e00705b0 2024-02-19T14:58:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:46.591 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":154,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:34.010658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9375":{"gid":9375,"name":"c","rank":0,"incarnation":150,"state":"up:stopping","state_seq":3,"addr":"172.21.15.78:6836/2602543074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2602543074},{"type":"v1","addr":"172.21.15.78:6836","nonce":2602543074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:46.591 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 154 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05c006b810 msgr2=0x7f05c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05c006b810 0x7f05c006dcd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f05d4034000 tx=0x7f05d4008030 comp rx=0 tx=0).stop 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 msgr2=0x7f05e01df250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 0x7f05e01df250 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f05d000c7c0 tx=0x7f05d000cc80 comp rx=0 tx=0).stop 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 shutdown_connections 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05c006b810 0x7f05c006dcd0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05e00c4980 0x7f05e01eaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05e00add70 0x7f05e00c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 --2- 172.21.15.78:0/4068824988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05e00705b0 0x7f05e01df250 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 >> 172.21.15.78:0/4068824988 conn(0x7f05e00c85e0 msgr2=0x7f05e00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:46.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 shutdown_connections 2024-02-19T14:58:46.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:46.590+0000 7f05e4bd7640 1 -- 172.21.15.78:0/4068824988 wait complete. 2024-02-19T14:58:47.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:34.010658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9375}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9375': {'gid': 9375, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.78:6836/2602543074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 2602543074}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 2602543074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:47.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9375:up:stopping 2024-02-19T14:58:48.007 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.115+0000 7f681fb56640 1 Processor -- start 2024-02-19T14:58:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.115+0000 7f681fb56640 1 -- start start 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681fb56640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68181537a0 0x7f68180c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681fb56640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6818156860 0x7f6818156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681fb56640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68181563b0 con 0x7f681814cd50 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681fb56640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6818153cd0 con 0x7f6818156860 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681fb56640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681806b6e0 con 0x7f68181537a0 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681d8f2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f681814cd50 0x7f68180c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46770/0 (socket says 172.21.15.78:46770) 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681d8f2640 1 -- 172.21.15.78:0/105552122 learned_addr learned my addr 172.21.15.78:0/105552122 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681e0f3640 1 --2- 172.21.15.78:0/105552122 >> v2:172.21.15.78:3300/0 conn(0x7f6818156860 0x7f6818156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681c8f0640 1 -- 172.21.15.78:0/105552122 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333716870 0 0) 0x7f68181563b0 con 0x7f681814cd50 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681d0f1640 1 --2- 172.21.15.78:0/105552122 >> v2:172.21.15.150:3300/0 conn(0x7f68181537a0 0x7f68180c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681c8f0640 1 -- 172.21.15.78:0/105552122 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804003520 con 0x7f681814cd50 2024-02-19T14:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681e0f3640 1 -- 172.21.15.78:0/105552122 >> v2:172.21.15.150:3300/0 conn(0x7f68181537a0 msgr2=0x7f68180c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681e0f3640 1 --2- 172.21.15.78:0/105552122 >> v2:172.21.15.150:3300/0 conn(0x7f68181537a0 0x7f68180c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681e0f3640 1 -- 172.21.15.78:0/105552122 >> v1:172.21.15.78:6789/0 conn(0x7f681814cd50 legacy=0x7f68180c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.116+0000 7f681e0f3640 1 -- 172.21.15.78:0/105552122 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681806b980 con 0x7f6818156860 2024-02-19T14:58:48.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.117+0000 7f681e0f3640 1 --2- 172.21.15.78:0/105552122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6818156860 0x7f6818156cd0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f68180acad0 tx=0x7f681002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b257286fa6e582da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:48.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.117+0000 7f681c8f0640 1 -- 172.21.15.78:0/105552122 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681002cb60 con 0x7f6818156860 2024-02-19T14:58:48.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.117+0000 7f681c8f0640 1 -- 172.21.15.78:0/105552122 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681002ccc0 con 0x7f6818156860 2024-02-19T14:58:48.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.117+0000 7f681c8f0640 1 -- 172.21.15.78:0/105552122 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810034640 con 0x7f6818156860 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 -- 172.21.15.78:0/105552122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6818156860 msgr2=0x7f6818156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 --2- 172.21.15.78:0/105552122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6818156860 0x7f6818156cd0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f68180acad0 tx=0x7f681002bdb0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 -- 172.21.15.78:0/105552122 shutdown_connections 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 --2- 172.21.15.78:0/105552122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6818156860 0x7f6818156cd0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 --2- 172.21.15.78:0/105552122 >> v2:172.21.15.150:3300/0 conn(0x7f68181537a0 0x7f68180c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 -- 172.21.15.78:0/105552122 >> 172.21.15.78:0/105552122 conn(0x7f6818066e70 msgr2=0x7f68180a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 -- 172.21.15.78:0/105552122 shutdown_connections 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.118+0000 7f681fb56640 1 -- 172.21.15.78:0/105552122 wait complete. 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.119+0000 7f681fb56640 1 Processor -- start 2024-02-19T14:58:48.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.119+0000 7f681fb56640 1 -- start start 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 0x7f68180be680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68181537a0 0x7f68180c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6818156860 0x7f68181491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180bf5d0 con 0x7f681814cd50 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68181ee0d0 con 0x7f6818156860 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180ac7e0 con 0x7f68181537a0 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 0x7f68180be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 0x7f68180be680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37694/0 (socket says 172.21.15.78:37694) 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 -- 172.21.15.78:0/2854672074 learned_addr learned my addr 172.21.15.78:0/2854672074 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d0f1640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68181537a0 0x7f68180c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681e0f3640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6818156860 0x7f68181491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 -- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68181537a0 msgr2=0x7f68180c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68181537a0 0x7f68180c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 -- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6818156860 msgr2=0x7f68181491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6818156860 0x7f68181491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 -- 172.21.15.78:0/2854672074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810009a70 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681d8f2640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 0x7f68180be680 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f681400c390 tx=0x7f681400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814005ce0 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6814005e40 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68180abf40 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.120+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68140072f0 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.121+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68180ac500 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.121+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6818074990 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.122+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6814007450 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.122+0000 7f680e7fc640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f67fc06b810 0x7f67fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.122+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f68140898e0 con 0x7f681814cd50 2024-02-19T14:58:48.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.122+0000 7f681d0f1640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f67fc06b810 0x7f67fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.123+0000 7f681d0f1640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f67fc06b810 0x7f67fc06dcd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6818152cb0 tx=0x7f6808008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:48.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.127+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6814054260 con 0x7f681814cd50 2024-02-19T14:58:48.239 INFO:tasks.ceph.mds.c.smithi078.stderr: -619> 2024-02-19T14:58:33.223+0000 7fce1e183640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:48.264 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:58:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.340+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68180c4230 con 0x7f681814cd50 2024-02-19T14:58:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.341+0000 7f680e7fc640 1 -- 172.21.15.78:0/2854672074 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4152 (secure 0 0 0) 0x7f6814017090 con 0x7f681814cd50 2024-02-19T14:58:48.344 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:48.344 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":155,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:47.846937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 155 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f67fc06b810 msgr2=0x7f67fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f67fc06b810 0x7f67fc06dcd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6818152cb0 tx=0x7f6808008040 comp rx=0 tx=0).stop 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 msgr2=0x7f68180be680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 0x7f68180be680 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f681400c390 tx=0x7f681400c850 comp rx=0 tx=0).stop 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 shutdown_connections 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f67fc06b810 0x7f67fc06dcd0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6818156860 0x7f68181491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68181537a0 0x7f68180c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 --2- 172.21.15.78:0/2854672074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f681814cd50 0x7f68180be680 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 >> 172.21.15.78:0/2854672074 conn(0x7f6818066e70 msgr2=0x7f681806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 shutdown_connections 2024-02-19T14:58:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.343+0000 7f681fb56640 1 -- 172.21.15.78:0/2854672074 wait complete. 2024-02-19T14:58:48.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 155, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:47.846937+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:48.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-02-19T14:58:48.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.859+0000 7f7136fe4640 1 Processor -- start 2024-02-19T14:58:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.859+0000 7f7136fe4640 1 -- start start 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7136fe4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7130152480 0x7f71301559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7136fe4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71300c87a0 0x7f71300c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7136fe4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7130155ef0 con 0x7f71300c7070 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7136fe4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713006dec0 con 0x7f7130152480 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7136fe4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713006d520 con 0x7f71300c87a0 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7130152480 0x7f71301559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7130152480 0x7f71301559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37726/0 (socket says 172.21.15.78:37726) 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 -- 172.21.15.78:0/2254383145 learned_addr learned my addr 172.21.15.78:0/2254383145 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7135581640 1 --2- 172.21.15.78:0/2254383145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71300c87a0 0x7f71300c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 -- 172.21.15.78:0/2254383145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71300c87a0 msgr2=0x7f71300c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 --2- 172.21.15.78:0/2254383145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71300c87a0 0x7f71300c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 -- 172.21.15.78:0/2254383145 >> v1:172.21.15.150:6789/0 conn(0x7f71300c7070 legacy=0x7f713006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.860+0000 7f7134d80640 1 -- 172.21.15.78:0/2254383145 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71300c75c0 con 0x7f7130152480 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.861+0000 7f7134d80640 1 --2- 172.21.15.78:0/2254383145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7130152480 0x7f71301559b0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f7124009870 tx=0x7f712402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8622580958fc62bb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.861+0000 7f712f7fe640 1 -- 172.21.15.78:0/2254383145 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7124030840 con 0x7f7130152480 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.861+0000 7f712f7fe640 1 -- 172.21.15.78:0/2254383145 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71240309a0 con 0x7f7130152480 2024-02-19T14:58:48.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.861+0000 7f712f7fe640 1 -- 172.21.15.78:0/2254383145 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7124030d30 con 0x7f7130152480 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 -- 172.21.15.78:0/2254383145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7130152480 msgr2=0x7f71301559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 --2- 172.21.15.78:0/2254383145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7130152480 0x7f71301559b0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f7124009870 tx=0x7f712402b980 comp rx=0 tx=0).stop 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 -- 172.21.15.78:0/2254383145 shutdown_connections 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 --2- 172.21.15.78:0/2254383145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71300c87a0 0x7f71300c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 --2- 172.21.15.78:0/2254383145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7130152480 0x7f71301559b0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 -- 172.21.15.78:0/2254383145 >> 172.21.15.78:0/2254383145 conn(0x7f71300ca8a0 msgr2=0x7f71300cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 -- 172.21.15.78:0/2254383145 shutdown_connections 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.862+0000 7f7136fe4640 1 -- 172.21.15.78:0/2254383145 wait complete. 2024-02-19T14:58:48.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.863+0000 7f7136fe4640 1 Processor -- start 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.863+0000 7f7136fe4640 1 -- start start 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7136fe4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71300c7070 0x7f71300aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7136fe4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 0x7f7130149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7136fe4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7130152480 0x7f71300c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7136fe4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713014a320 con 0x7f71300c87a0 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7136fe4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7130149b40 con 0x7f71300c7070 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7136fe4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71301eea40 con 0x7f7130152480 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 0x7f7130149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 0x7f7130149de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37734/0 (socket says 172.21.15.78:37734) 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 -- 172.21.15.78:0/729156725 learned_addr learned my addr 172.21.15.78:0/729156725 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7134d80640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71300c7070 0x7f71300aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f7135581640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7130152480 0x7f71300c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 -- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7130152480 msgr2=0x7f71300c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7130152480 0x7f71300c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 -- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71300c7070 msgr2=0x7f71300aa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71300c7070 0x7f71300aa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 -- 172.21.15.78:0/729156725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240094d0 con 0x7f71300c87a0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.864+0000 7f712ffff640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 0x7f7130149de0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f711c00c3c0 tx=0x7f711c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.865+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c0052d0 con 0x7f71300c87a0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.865+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c005450 con 0x7f71300c87a0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.865+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71301eece0 con 0x7f71300c87a0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.865+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c003850 con 0x7f71300c87a0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.865+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71300c9d60 con 0x7f71300c87a0 2024-02-19T14:58:48.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.865+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71300aa580 con 0x7f71300c87a0 2024-02-19T14:58:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.866+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f711c0039b0 con 0x7f71300c87a0 2024-02-19T14:58:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.866+0000 7f712d7fa640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711406b810 0x7f711406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.866+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f711c089600 con 0x7f71300c87a0 2024-02-19T14:58:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.866+0000 7f7134d80640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711406b810 0x7f711406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:48.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.867+0000 7f7134d80640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711406b810 0x7f711406dcd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7124036000 tx=0x7f7124008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:48.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:48.871+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f71300aa580 con 0x7f71300c87a0 2024-02-19T14:58:49.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.077+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71300aa580 con 0x7f71300c87a0 2024-02-19T14:58:49.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.078+0000 7f712d7fa640 1 -- 172.21.15.78:0/729156725 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4152 (secure 0 0 0) 0x7f71300aa580 con 0x7f71300c87a0 2024-02-19T14:58:49.081 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:49.081 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":155,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:47.846937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:49.081 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 155 2024-02-19T14:58:49.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711406b810 msgr2=0x7f711406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711406b810 0x7f711406dcd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7124036000 tx=0x7f7124008030 comp rx=0 tx=0).stop 2024-02-19T14:58:49.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 msgr2=0x7f7130149de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 0x7f7130149de0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f711c00c3c0 tx=0x7f711c00c880 comp rx=0 tx=0).stop 2024-02-19T14:58:49.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 shutdown_connections 2024-02-19T14:58:49.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f711406b810 0x7f711406dcd0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71300c87a0 0x7f7130149de0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7130152480 0x7f71300c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 --2- 172.21.15.78:0/729156725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71300c7070 0x7f71300aa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 >> 172.21.15.78:0/729156725 conn(0x7f71300ca8a0 msgr2=0x7f71300cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:49.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 shutdown_connections 2024-02-19T14:58:49.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.080+0000 7f7136fe4640 1 -- 172.21.15.78:0/729156725 wait complete. 2024-02-19T14:58:49.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down false 2024-02-19T14:58:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.193+0000 7fb9c1fbc640 1 Processor -- start 2024-02-19T14:58:49.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.193+0000 7fb9c1fbc640 1 -- start start 2024-02-19T14:58:49.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9c1fbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 0x7fb9bc153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9c1fbc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9bc1535a0 con 0x7fb9bc0ab840 2024-02-19T14:58:49.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9c1fbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9bc0ac590 con 0x7fb9bc0aca90 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9c1fbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc152f90 con 0x7fb9bc071b60 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9bbfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb9bc0ab840 0x7fb9bc14ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46786/0 (socket says 172.21.15.78:46786) 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9bbfff640 1 -- 172.21.15.78:0/1284277268 learned_addr learned my addr 172.21.15.78:0/1284277268 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749291415 0 0) 0x7fb9bc1535a0 con 0x7fb9bc0ab840 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 0x7fb9bc153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.194+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4003520 con 0x7fb9bc0ab840 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781283754 0 0) 0x7fb9a4003520 con 0x7fb9bc0ab840 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 msgr2=0x7fb9bc153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 0x7fb9bc153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 >> v1:172.21.15.150:6789/0 conn(0x7fb9bc0aca90 legacy=0x7fb9bc06ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc06b590 con 0x7fb9bc0ab840 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb9b40019f0 con 0x7fb9bc0ab840 2024-02-19T14:58:49.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 >> v1:172.21.15.78:6789/0 conn(0x7fb9bc0ab840 legacy=0x7fb9bc14ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a40039a0 0x7fb9a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 0x7fb9a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc152f90 con 0x7fb9a40039a0 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a400ba30 con 0x7fb9a40047c0 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a400bbf0 con 0x7fb9bc071b60 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a40039a0 0x7fb9a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9baffd640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bbfff640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 0x7fb9a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bb7fe640 1 -- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 msgr2=0x7fb9a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc071b60 0x7fb9a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bb7fe640 1 -- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a40047c0 msgr2=0x7fb9a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9bb7fe640 1 -- 172.21.15.78:0/1284277268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a4003520 con 0x7fb9a40039a0 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.195+0000 7fb9baffd640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a40039a0 0x7fb9a4003dc0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009a90 tx=0x7fb9a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec131ee386591c8d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac02c680 con 0x7fb9a40039a0 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac02c7e0 con 0x7fb9a40039a0 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9ba7fc640 1 -- 172.21.15.78:0/1284277268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac02cb70 con 0x7fb9a40039a0 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/1284277268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a40039a0 msgr2=0x7fb9a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/1284277268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a40039a0 0x7fb9a4003dc0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009a90 tx=0x7fb9a40057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9bbfff640 1 -- 172.21.15.78:0/1284277268 reap_dead start 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/1284277268 shutdown_connections 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/1284277268 >> 172.21.15.78:0/1284277268 conn(0x7fb9bc0c7800 msgr2=0x7fb9bc0c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/1284277268 shutdown_connections 2024-02-19T14:58:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.196+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/1284277268 wait complete. 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.197+0000 7fb9c1fbc640 1 Processor -- start 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 -- start start 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc14ed70 0x7fb9bc150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 0x7fb9bc150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc1ebf10 0x7fb9bc1ec3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc152b50 con 0x7fb9bc152650 2024-02-19T14:58:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1e9470 con 0x7fb9bc14ed70 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9c1fbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1e9900 con 0x7fb9bc1ebf10 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 0x7fb9bc150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 0x7fb9bc150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37756/0 (socket says 172.21.15.78:37756) 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.198+0000 7fb9baffd640 1 -- 172.21.15.78:0/3405178452 learned_addr learned my addr 172.21.15.78:0/3405178452 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9baffd640 1 -- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc1ebf10 msgr2=0x7fb9bc1ec3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9baffd640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc1ebf10 0x7fb9bc1ec3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9baffd640 1 -- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc14ed70 msgr2=0x7fb9bc150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc14ed70 0x7fb9bc150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9baffd640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc14ed70 0x7fb9bc150a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9baffd640 1 -- 172.21.15.78:0/3405178452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc1514e0 con 0x7fb9bc152650 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9baffd640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 0x7fb9bc150fa0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000e5b0 tx=0x7fb9b000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00052e0 con 0x7fb9bc152650 2024-02-19T14:58:49.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0005440 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc1e8b40 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0003810 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.199+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc1e9100 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.200+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc1e9290 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.201+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb9b0003970 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.201+0000 7fb9a3fff640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb9a806b840 0x7fb9a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.201+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7fb9b008a0a0 con 0x7fb9bc152650 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.201+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb9a806b840 0x7fb9a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:49.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.201+0000 7fb9bb7fe640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb9a806b840 0x7fb9a806dd00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac036000 tx=0x7fb9ac030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.205+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb9b0054970 con 0x7fb9bc152650 2024-02-19T14:58:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:49.416+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"} v 0) v1 -- 0x7fb9bc074990 con 0x7fb9bc152650 2024-02-19T14:58:50.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:58:50.238+0000 7f16560b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:58:50.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.248+0000 7fb9a3fff640 1 -- 172.21.15.78:0/3405178452 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"}]=0 cephfs marked up, max_mds = 1 v157) v1 ==== 135+0+0 (secure 0 0 0) 0x7fb9b0058030 con 0x7fb9bc152650 2024-02-19T14:58:50.250 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked up, max_mds = 1 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb9a806b840 msgr2=0x7fb9a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb9a806b840 0x7fb9a806dd00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac036000 tx=0x7fb9ac030000 comp rx=0 tx=0).stop 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 msgr2=0x7fb9bc150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 0x7fb9bc150fa0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000e5b0 tx=0x7fb9b000ea70 comp rx=0 tx=0).stop 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 shutdown_connections 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb9a806b840 0x7fb9a806dd00 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9bc152650 0x7fb9bc150fa0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9bc1ebf10 0x7fb9bc1ec3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 --2- 172.21.15.78:0/3405178452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc14ed70 0x7fb9bc150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 >> 172.21.15.78:0/3405178452 conn(0x7fb9bc0c7800 msgr2=0x7fb9bc072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 shutdown_connections 2024-02-19T14:58:50.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.251+0000 7fb9c1fbc640 1 -- 172.21.15.78:0/3405178452 wait complete. 2024-02-19T14:58:50.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.370+0000 7f91ea9e9640 1 Processor -- start 2024-02-19T14:58:50.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.370+0000 7f91ea9e9640 1 -- start start 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91ea9e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c8500 0x7f91e40c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91ea9e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f91e40c42e0 0x7f91e4158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91ea9e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f91e40c2f50 0x7f91e4158b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91ea9e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e406c090 con 0x7f91e40c2f50 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91ea9e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e406d6a0 con 0x7f91e40c42e0 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91ea9e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40c34a0 con 0x7f91e40c8500 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f91e40c2f50 0x7f91e4158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f91e40c2f50 0x7f91e4158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37768/0 (socket says 172.21.15.78:37768) 2024-02-19T14:58:50.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 -- 172.21.15.78:0/317835387 learned_addr learned my addr 172.21.15.78:0/317835387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e3fff640 1 --2- 172.21.15.78:0/317835387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c8500 0x7f91e40c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e37fe640 1 --2- 172.21.15.78:0/317835387 >> v2:172.21.15.150:3300/0 conn(0x7f91e40c42e0 0x7f91e4158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 -- 172.21.15.78:0/317835387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c8500 msgr2=0x7f91e40c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 --2- 172.21.15.78:0/317835387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c8500 0x7f91e40c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 -- 172.21.15.78:0/317835387 >> v2:172.21.15.150:3300/0 conn(0x7f91e40c42e0 msgr2=0x7f91e4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 --2- 172.21.15.78:0/317835387 >> v2:172.21.15.150:3300/0 conn(0x7f91e40c42e0 0x7f91e4158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.371+0000 7f91e8f86640 1 -- 172.21.15.78:0/317835387 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e406c8e0 con 0x7f91e40c2f50 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.372+0000 7f91e8f86640 1 --2- 172.21.15.78:0/317835387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c2f50 0x7f91e4158b90 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f91e40c2a00 tx=0x7f91dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a574de8ba1ac6dbb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.372+0000 7f91e2ffd640 1 -- 172.21.15.78:0/317835387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc02c9e0 con 0x7f91e40c2f50 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.372+0000 7f91e2ffd640 1 -- 172.21.15.78:0/317835387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc02cb40 con 0x7f91e40c2f50 2024-02-19T14:58:50.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.372+0000 7f91e2ffd640 1 -- 172.21.15.78:0/317835387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0343f0 con 0x7f91e40c2f50 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 -- 172.21.15.78:0/317835387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c2f50 msgr2=0x7f91e4158b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/317835387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c2f50 0x7f91e4158b90 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f91e40c2a00 tx=0x7f91dc02be40 comp rx=0 tx=0).stop 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 -- 172.21.15.78:0/317835387 shutdown_connections 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/317835387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c2f50 0x7f91e4158b90 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/317835387 >> v2:172.21.15.150:3300/0 conn(0x7f91e40c42e0 0x7f91e4158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/317835387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c8500 0x7f91e40c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 -- 172.21.15.78:0/317835387 >> 172.21.15.78:0/317835387 conn(0x7f91e40aa300 msgr2=0x7f91e40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 -- 172.21.15.78:0/317835387 shutdown_connections 2024-02-19T14:58:50.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.373+0000 7f91ea9e9640 1 -- 172.21.15.78:0/317835387 wait complete. 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.374+0000 7f91ea9e9640 1 Processor -- start 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.374+0000 7f91ea9e9640 1 -- start start 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91ea9e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c2f50 0x7f91e40ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91ea9e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 0x7f91e40a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91ea9e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e40abbb0 0x7f91e41ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91ea9e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40ab930 con 0x7f91e40c8500 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91ea9e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e4157ba0 con 0x7f91e40abbb0 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91ea9e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e4157e60 con 0x7f91e40c2f50 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 0x7f91e40a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 0x7f91e40a9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37776/0 (socket says 172.21.15.78:37776) 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 -- 172.21.15.78:0/2186746544 learned_addr learned my addr 172.21.15.78:0/2186746544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e3fff640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c2f50 0x7f91e40ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e8f86640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e40abbb0 0x7f91e41ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 -- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c2f50 msgr2=0x7f91e40ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c2f50 0x7f91e40ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 -- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e40abbb0 msgr2=0x7f91e41ee120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e40abbb0 0x7f91e41ee120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 -- 172.21.15.78:0/2186746544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc009b30 con 0x7f91e40c8500 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e8f86640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e40abbb0 0x7f91e41ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.375+0000 7f91e37fe640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 0x7f91e40a9630 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f91d800d800 tx=0x7f91d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.376+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d800b6c0 con 0x7f91e40c8500 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.376+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d800b820 con 0x7f91e40c8500 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.376+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e4158140 con 0x7f91e40c8500 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.376+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d800bbb0 con 0x7f91e40c8500 2024-02-19T14:58:50.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.376+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e40a9b70 con 0x7f91e40c8500 2024-02-19T14:58:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.377+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e4071c20 con 0x7f91e40c8500 2024-02-19T14:58:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.377+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f91d800bd10 con 0x7f91e40c8500 2024-02-19T14:58:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.377+0000 7f91e0ff9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f91d006b810 0x7f91d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.377+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f91d808a8c0 con 0x7f91e40c8500 2024-02-19T14:58:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.377+0000 7f91e3fff640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f91d006b810 0x7f91d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.378+0000 7f91e3fff640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f91d006b810 0x7f91d006dcd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f91d4006fd0 tx=0x7f91d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:50.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.382+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f91d8055190 con 0x7f91e40c8500 2024-02-19T14:58:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.588+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91e4074990 con 0x7f91e40c8500 2024-02-19T14:58:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.588+0000 7f91e0ff9640 1 -- 172.21.15.78:0/2186746544 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f91d8058850 con 0x7f91e40c8500 2024-02-19T14:58:50.591 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:50.592 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":158,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.251236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:starting","state_seq":1,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 158 2024-02-19T14:58:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f91d006b810 msgr2=0x7f91d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f91d006b810 0x7f91d006dcd0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f91d4006fd0 tx=0x7f91d4008040 comp rx=0 tx=0).stop 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 msgr2=0x7f91e40a9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 0x7f91e40a9630 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f91d800d800 tx=0x7f91d800dcc0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 shutdown_connections 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f91d006b810 0x7f91d006dcd0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e40c8500 0x7f91e40a9630 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e40abbb0 0x7f91e41ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 --2- 172.21.15.78:0/2186746544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91e40c2f50 0x7f91e40ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.590+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 >> 172.21.15.78:0/2186746544 conn(0x7f91e40aa300 msgr2=0x7f91e40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.591+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 shutdown_connections 2024-02-19T14:58:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.591+0000 7f91ea9e9640 1 -- 172.21.15.78:0/2186746544 wait complete. 2024-02-19T14:58:50.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:50.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.701+0000 7f07afb81640 1 Processor -- start 2024-02-19T14:58:50.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.701+0000 7f07afb81640 1 -- start start 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07afb81640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07a80c5be0 0x7f07a81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07afb81640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a8072250 con 0x7f07a80c25e0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07afb81640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a8072480 con 0x7f07a80c29c0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07afb81640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a8152c90 con 0x7f07a80c5be0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ad91d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f07a80c25e0 0x7f07a80c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46798/0 (socket says 172.21.15.78:46798) 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ad91d640 1 -- 172.21.15.78:0/40752708 learned_addr learned my addr 172.21.15.78:0/40752708 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154145751 0 0) 0x7f07a8072250 con 0x7f07a80c25e0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0794003520 con 0x7f07a80c25e0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ae11e640 1 --2- 172.21.15.78:0/40752708 >> v2:172.21.15.150:3300/0 conn(0x7f07a80c5be0 0x7f07a81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 761919307 0 0) 0x7f0794003520 con 0x7f07a80c25e0 2024-02-19T14:58:50.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 >> v2:172.21.15.150:3300/0 conn(0x7f07a80c5be0 msgr2=0x7f07a81526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 --2- 172.21.15.78:0/40752708 >> v2:172.21.15.150:3300/0 conn(0x7f07a80c5be0 0x7f07a81526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 >> v1:172.21.15.150:6789/0 conn(0x7f07a80c29c0 legacy=0x7f07a806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80c4b90 con 0x7f07a80c25e0 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f07a4001670 con 0x7f07a80c25e0 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 >> v1:172.21.15.78:6789/0 conn(0x7f07a80c25e0 legacy=0x7f07a80c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0794003970 0x7f0794003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07940047c0 0x7f0794004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f079400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a8152c90 con 0x7f0794005180 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079400c120 con 0x7f07940047c0 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079400c2e0 con 0x7f0794003970 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ae11e640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f079400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ad11c640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07940047c0 0x7f0794004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.702+0000 7f07ad91d640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0794003970 0x7f0794003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 -- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0794003970 msgr2=0x7f0794003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0794003970 0x7f0794003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 -- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07940047c0 msgr2=0x7f0794004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07940047c0 0x7f0794004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 -- 172.21.15.78:0/40752708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794003520 con 0x7f0794005180 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ad11c640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07940047c0 0x7f0794004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 -- 172.21.15.78:0/40752708 reap_dead start 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ae11e640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f079400b980 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009fc0 tx=0x7f07940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261e28e773a6f3d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a002cc60 con 0x7f0794005180 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a000f040 con 0x7f0794005180 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.703+0000 7f07ac91b640 1 -- 172.21.15.78:0/40752708 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0003990 con 0x7f0794005180 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 -- 172.21.15.78:0/40752708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 msgr2=0x7f079400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f079400b980 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009fc0 tx=0x7f07940057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 -- 172.21.15.78:0/40752708 shutdown_connections 2024-02-19T14:58:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 --2- 172.21.15.78:0/40752708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f079400b980 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 -- 172.21.15.78:0/40752708 >> 172.21.15.78:0/40752708 conn(0x7f07a806b270 msgr2=0x7f07a806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 -- 172.21.15.78:0/40752708 shutdown_connections 2024-02-19T14:58:50.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.704+0000 7f07afb81640 1 -- 172.21.15.78:0/40752708 wait complete. 2024-02-19T14:58:50.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.705+0000 7f07afb81640 1 Processor -- start 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.705+0000 7f07afb81640 1 -- start start 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07afb81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f07a80a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07afb81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07a8146e10 0x7f07a81eba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07afb81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07a81ebf50 0x7f07a813c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07afb81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80a9100 con 0x7f0794005180 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07afb81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80aae70 con 0x7f07a81ebf50 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07afb81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81ec3d0 con 0x7f07a8146e10 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f07a80a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f07a80a8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37782/0 (socket says 172.21.15.78:37782) 2024-02-19T14:58:50.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 -- 172.21.15.78:0/3136435096 learned_addr learned my addr 172.21.15.78:0/3136435096 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad11c640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07a8146e10 0x7f07a81eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ae11e640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07a81ebf50 0x7f07a813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 -- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07a8146e10 msgr2=0x7f07a81eba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07a8146e10 0x7f07a81eba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 -- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07a81ebf50 msgr2=0x7f07a813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07a81ebf50 0x7f07a813c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 -- 172.21.15.78:0/3136435096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a81ec650 con 0x7f0794005180 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ae11e640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07a81ebf50 0x7f07a813c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f07ad91d640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f07a80a8bc0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f07a40072c0 tx=0x7f07a4007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:50.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.706+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a4004ea0 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.707+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a400f8e0 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.707+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a80ab080 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.707+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a400fc90 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.707+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a813c780 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.707+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a80ab5a0 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.708+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f07a401d460 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.708+0000 7f079e7fc640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f078c06b840 0x7f078c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.708+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f07a408a400 con 0x7f0794005180 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.708+0000 7f07ad11c640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f078c06b840 0x7f078c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:50.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.709+0000 7f07ad11c640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f078c06b840 0x7f078c06dd00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0798000c00 tx=0x7f0798009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:50.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.713+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f07a4054cd0 con 0x7f0794005180 2024-02-19T14:58:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.918+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07a806f1a0 con 0x7f0794005180 2024-02-19T14:58:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.918+0000 7f079e7fc640 1 -- 172.21.15.78:0/3136435096 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f07a4058390 con 0x7f0794005180 2024-02-19T14:58:50.921 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:50.922 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":158,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.251236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:starting","state_seq":1,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 158 2024-02-19T14:58:50.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f078c06b840 msgr2=0x7f078c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f078c06b840 0x7f078c06dd00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0798000c00 tx=0x7f0798009040 comp rx=0 tx=0).stop 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 msgr2=0x7f07a80a8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f07a80a8bc0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f07a40072c0 tx=0x7f07a4007780 comp rx=0 tx=0).stop 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 shutdown_connections 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f078c06b840 0x7f078c06dd00 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07a81ebf50 0x7f07a813c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07a8146e10 0x7f07a81eba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 --2- 172.21.15.78:0/3136435096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0794005180 0x7f07a80a8bc0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.920+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 >> 172.21.15.78:0/3136435096 conn(0x7f07a806b270 msgr2=0x7f07a80ab980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.921+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 shutdown_connections 2024-02-19T14:58:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:50.921+0000 7f07afb81640 1 -- 172.21.15.78:0/3136435096 wait complete. 2024-02-19T14:58:51.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 158, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:50.251236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9609}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9609': {'gid': 9609, 'name': 'c', 'rank': 0, 'incarnation': 158, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.78:6836/3613671589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3613671589}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3613671589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:51.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9609:up:starting 2024-02-19T14:58:52.338 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.436+0000 7f7afb5ef640 1 Processor -- start 2024-02-19T14:58:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.436+0000 7f7afb5ef640 1 -- start start 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7afb5ef640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7af414f1a0 0x7f7af4072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7afb5ef640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7af406ea50 0x7f7af4152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7afb5ef640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af40c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7afb5ef640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af40c5970 con 0x7f7af414f1a0 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7afb5ef640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af40c5750 con 0x7f7af406ea50 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7afb5ef640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40c5bc0 con 0x7f7af406fbc0 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af938b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7af414f1a0 0x7f7af4072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af938b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7af414f1a0 0x7f7af4072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37796/0 (socket says 172.21.15.78:37796) 2024-02-19T14:58:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af938b640 1 -- 172.21.15.78:0/2317339357 learned_addr learned my addr 172.21.15.78:0/2317339357 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af8b8a640 1 --2- 172.21.15.78:0/2317339357 >> v2:172.21.15.150:3300/0 conn(0x7f7af406ea50 0x7f7af4152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af9b8c640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af40c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af938b640 1 -- 172.21.15.78:0/2317339357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 msgr2=0x7f7af40c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.437+0000 7f7af938b640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af40c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7af938b640 1 -- 172.21.15.78:0/2317339357 >> v2:172.21.15.150:3300/0 conn(0x7f7af406ea50 msgr2=0x7f7af4152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7af938b640 1 --2- 172.21.15.78:0/2317339357 >> v2:172.21.15.150:3300/0 conn(0x7f7af406ea50 0x7f7af4152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7af938b640 1 -- 172.21.15.78:0/2317339357 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40c5ea0 con 0x7f7af414f1a0 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7af8b8a640 1 --2- 172.21.15.78:0/2317339357 >> v2:172.21.15.150:3300/0 conn(0x7f7af406ea50 0x7f7af4152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7af9b8c640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af40c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7af938b640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af4072d70 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f7af406f090 tx=0x7f7af002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0385de228587add server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7aebfff640 1 -- 172.21.15.78:0/2317339357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0031750 con 0x7f7af414f1a0 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7aebfff640 1 -- 172.21.15.78:0/2317339357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af00318b0 con 0x7f7af414f1a0 2024-02-19T14:58:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.438+0000 7f7aebfff640 1 -- 172.21.15.78:0/2317339357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0031c40 con 0x7f7af414f1a0 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 -- 172.21.15.78:0/2317339357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 msgr2=0x7f7af4072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af4072d70 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f7af406f090 tx=0x7f7af002b7f0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 -- 172.21.15.78:0/2317339357 shutdown_connections 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af40c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/2317339357 >> v2:172.21.15.150:3300/0 conn(0x7f7af406ea50 0x7f7af4152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/2317339357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af4072d70 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 -- 172.21.15.78:0/2317339357 >> 172.21.15.78:0/2317339357 conn(0x7f7af406c170 msgr2=0x7f7af406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 -- 172.21.15.78:0/2317339357 shutdown_connections 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.439+0000 7f7afb5ef640 1 -- 172.21.15.78:0/2317339357 wait complete. 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.440+0000 7f7afb5ef640 1 Processor -- start 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.440+0000 7f7afb5ef640 1 -- start start 2024-02-19T14:58:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7afb5ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af406ea50 0x7f7af41e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7afb5ef640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af41ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7afb5ef640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af41eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7afb5ef640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40c2c70 con 0x7f7af414f1a0 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7afb5ef640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40c4d60 con 0x7f7af406ea50 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7afb5ef640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40c2e60 con 0x7f7af406fbc0 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af41eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af41eacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37806/0 (socket says 172.21.15.78:37806) 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 -- 172.21.15.78:0/4086106513 learned_addr learned my addr 172.21.15.78:0/4086106513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af938b640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af406ea50 0x7f7af41e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 -- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 msgr2=0x7f7af41ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af41ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 -- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af406ea50 msgr2=0x7f7af41e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af406ea50 0x7f7af41e83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.441+0000 7f7af9b8c640 1 -- 172.21.15.78:0/4086106513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00094d0 con 0x7f7af414f1a0 2024-02-19T14:58:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.442+0000 7f7af9b8c640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af41eacd0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00cef0 tx=0x7f7aec00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.442+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00d840 con 0x7f7af414f1a0 2024-02-19T14:58:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.442+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec00d9c0 con 0x7f7af414f1a0 2024-02-19T14:58:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.442+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00dd70 con 0x7f7af414f1a0 2024-02-19T14:58:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.442+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af40c3140 con 0x7f7af414f1a0 2024-02-19T14:58:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.442+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af40c3700 con 0x7f7af414f1a0 2024-02-19T14:58:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.443+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af40c38f0 con 0x7f7af414f1a0 2024-02-19T14:58:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.443+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7aec01e3b0 con 0x7f7af414f1a0 2024-02-19T14:58:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.444+0000 7f7ae9ffb640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ad806b810 0x7f7ad806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.444+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f7aec08adf0 con 0x7f7af414f1a0 2024-02-19T14:58:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.444+0000 7f7af938b640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ad806b810 0x7f7ad806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.444+0000 7f7af938b640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ad806b810 0x7f7ad806dcd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7af0035000 tx=0x7f7af0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.448+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7aec0556f0 con 0x7f7af414f1a0 2024-02-19T14:58:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.658+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af4074990 con 0x7f7af414f1a0 2024-02-19T14:58:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.659+0000 7f7ae9ffb640 1 -- 172.21.15.78:0/4086106513 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f7aec01a020 con 0x7f7af414f1a0 2024-02-19T14:58:52.661 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:52.661 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.262364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:active","state_seq":2,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 159 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.660+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ad806b810 msgr2=0x7f7ad806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.660+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ad806b810 0x7f7ad806dcd0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7af0035000 tx=0x7f7af0008030 comp rx=0 tx=0).stop 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.660+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 msgr2=0x7f7af41eacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.660+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af41eacd0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00cef0 tx=0x7f7aec00a710 comp rx=0 tx=0).stop 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 shutdown_connections 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7ad806b810 0x7f7ad806dcd0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7af414f1a0 0x7f7af41eacd0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7af406fbc0 0x7f7af41ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 --2- 172.21.15.78:0/4086106513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af406ea50 0x7f7af41e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 >> 172.21.15.78:0/4086106513 conn(0x7f7af406c170 msgr2=0x7f7af4149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 shutdown_connections 2024-02-19T14:58:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:52.661+0000 7f7afb5ef640 1 -- 172.21.15.78:0/4086106513 wait complete. 2024-02-19T14:58:53.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:58:27.307125+0000', 'modified': '2024-02-19T14:58:50.262364+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9609}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9609': {'gid': 9609, 'name': 'c', 'rank': 0, 'incarnation': 158, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.78:6836/3613671589', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3613671589}, {'type': 'v1', 'addr': '172.21.15.78:6836', 'nonce': 3613671589}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:58:53.076 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:58:53.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:58:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.177+0000 7f5f5c28f640 1 Processor -- start 2024-02-19T14:58:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.177+0000 7f5f5c28f640 1 -- start start 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5c28f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f540c4d80 0x7f5f54159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5c28f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 0x7f5f541523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5c28f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5406ffa0 con 0x7f5f540c78a0 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5c28f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5406b720 con 0x7f5f540c4d80 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5c28f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5406fc70 con 0x7f5f54155010 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5a02b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5f540c78a0 0x7f5f540c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46800/0 (socket says 172.21.15.78:46800) 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5a02b640 1 -- 172.21.15.78:0/902455543 learned_addr learned my addr 172.21.15.78:0/902455543 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.178+0000 7f5f5982a640 1 --2- 172.21.15.78:0/902455543 >> v2:172.21.15.150:3300/0 conn(0x7f5f540c4d80 0x7f5f54159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 0x7f5f541523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233695623 0 0) 0x7f5f5406ffa0 con 0x7f5f540c78a0 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f40003520 con 0x7f5f540c78a0 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289016498 0 0) 0x7f5f40003520 con 0x7f5f540c78a0 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 msgr2=0x7f5f541523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 0x7f5f541523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 >> v2:172.21.15.150:3300/0 conn(0x7f5f540c4d80 msgr2=0x7f5f54159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 --2- 172.21.15.78:0/902455543 >> v2:172.21.15.150:3300/0 conn(0x7f5f540c4d80 0x7f5f54159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5406b930 con 0x7f5f540c78a0 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f5f50001670 con 0x7f5f540c78a0 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 >> v1:172.21.15.78:6789/0 conn(0x7f5f540c78a0 legacy=0x7f5f540c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 0x7f5f40003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40003ef0 0x7f5f400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f40004d00 0x7f5f40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5406b720 con 0x7f5f40004d00 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5406fc70 con 0x7f5f40003ef0 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4000b9d0 con 0x7f5f54155010 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f40004d00 0x7f5f40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5982a640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40003ef0 0x7f5f400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 0x7f5f40003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 -- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 msgr2=0x7f5f40003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f54155010 0x7f5f40003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 -- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40003ef0 msgr2=0x7f5f400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40003ef0 0x7f5f400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.179+0000 7f5f5a82c640 1 -- 172.21.15.78:0/902455543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f40003520 con 0x7f5f40004d00 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.180+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f40004d00 0x7f5f40005180 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00e170 tx=0x7f5f400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc613b3f356f6d47 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.180+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c02cbf0 con 0x7f5f40004d00 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.180+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c00f040 con 0x7f5f40004d00 2024-02-19T14:58:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.180+0000 7f5f59029640 1 -- 172.21.15.78:0/902455543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c034440 con 0x7f5f40004d00 2024-02-19T14:58:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5c28f640 1 -- 172.21.15.78:0/902455543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f40004d00 msgr2=0x7f5f40005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/902455543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f40004d00 0x7f5f40005180 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00e170 tx=0x7f5f400057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5a82c640 1 -- 172.21.15.78:0/902455543 reap_dead start 2024-02-19T14:58:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5c28f640 1 -- 172.21.15.78:0/902455543 shutdown_connections 2024-02-19T14:58:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5c28f640 1 -- 172.21.15.78:0/902455543 >> 172.21.15.78:0/902455543 conn(0x7f5f54066e70 msgr2=0x7f5f540a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5c28f640 1 -- 172.21.15.78:0/902455543 shutdown_connections 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.181+0000 7f5f5c28f640 1 -- 172.21.15.78:0/902455543 wait complete. 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.183+0000 7f5f5c28f640 1 Processor -- start 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.183+0000 7f5f5c28f640 1 -- start start 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5c28f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40004d00 0x7f5f5414e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5c28f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f5414c890 0x7f5f540c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5c28f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 0x7f5f541edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5c28f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f54145c00 con 0x7f5f540c7010 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5c28f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540abb30 con 0x7f5f40004d00 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5c28f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540ac140 con 0x7f5f5414c890 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 0x7f5f541edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 0x7f5f541edb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37834/0 (socket says 172.21.15.78:37834) 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1148389995 learned_addr learned my addr 172.21.15.78:0/1148389995 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5982a640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f5414c890 0x7f5f540c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40004d00 0x7f5f5414e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f5414c890 msgr2=0x7f5f540c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f5414c890 0x7f5f540c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40004d00 msgr2=0x7f5f5414e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40004d00 0x7f5f5414e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1148389995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540ab5e0 con 0x7f5f540c7010 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5982a640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f5414c890 0x7f5f540c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40004d00 0x7f5f5414e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 0x7f5f541edb10 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00e170 tx=0x7f5f4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f4affd640 1 -- 172.21.15.78:0/1148389995 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c034d50 con 0x7f5f540c7010 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.184+0000 7f5f4affd640 1 -- 172.21.15.78:0/1148389995 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c03e440 con 0x7f5f540c7010 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.185+0000 7f5f4affd640 1 -- 172.21.15.78:0/1148389995 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c03e7f0 con 0x7f5f540c7010 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.185+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f540ab300 con 0x7f5f540c7010 2024-02-19T14:58:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.185+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5414e610 con 0x7f5f540c7010 2024-02-19T14:58:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.186+0000 7f5f4affd640 1 -- 172.21.15.78:0/1148389995 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5f4c03e950 con 0x7f5f540c7010 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.187+0000 7f5f4affd640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.187+0000 7f5f4affd640 1 -- 172.21.15.78:0/1148389995 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f5f4c0ac560 con 0x7f5f540c7010 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.187+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.187+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5f50001050 tx=0x7f5f50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.188+0000 7f5f5ee68740 1 Processor -- start 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.188+0000 7f5f5ee68740 1 -- start start 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.188+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.188+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 0x56081c2d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.188+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d58b0 0x56081c297860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c252cd0 con 0x56081c298ae0 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c205f20 con 0x56081c2d58b0 2024-02-19T14:58:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081c203450 con 0x56081c2d2740 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 0x56081c2d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 0x56081c2d2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37842/0 (socket says 172.21.15.78:37842) 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 -- 172.21.15.78:0/1121679201 learned_addr learned my addr 172.21.15.78:0/1121679201 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d58b0 0x56081c297860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 -- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 msgr2=0x56081c2d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 -- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d58b0 msgr2=0x56081c297860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d58b0 0x56081c297860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 -- 172.21.15.78:0/1121679201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56081c1b3f20 con 0x56081c298ae0 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d58b0 0x56081c297860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f5982a640 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 0x56081c2d2b20 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f5f44009b60 tx=0x7f5f44009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f934afcdff350d4e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f37fff640 1 -- 172.21.15.78:0/1121679201 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44009e60 con 0x56081c298ae0 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f37fff640 1 -- 172.21.15.78:0/1121679201 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44013e70 con 0x56081c298ae0 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.189+0000 7f5f37fff640 1 -- 172.21.15.78:0/1121679201 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44004250 con 0x56081c298ae0 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1121679201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 msgr2=0x56081c2d2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 0x56081c2d2b20 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f5f44009b60 tx=0x7f5f44009190 comp rx=0 tx=0).stop 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1121679201 shutdown_connections 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c298ae0 0x56081c2d2b20 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d58b0 0x56081c297860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1121679201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1121679201 >> 172.21.15.78:0/1121679201 conn(0x56081c294120 msgr2=0x56081c2b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1121679201 shutdown_connections 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.190+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1121679201 wait complete. 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.191+0000 7f5f5ee68740 1 -- start start 2024-02-19T14:58:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.191+0000 7f5f5ee68740 1 Processor -- start 2024-02-19T14:58:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2e7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2c9250 0x56081c2965f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c1fec30 con 0x56081c2d58b0 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c1a3d30 con 0x56081c2c9250 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081bed2310 con 0x56081c2d2740 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5982a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5ee68740 10 client.0 _ll_register_callbacks cb 0x56081c1abd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5982a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2e7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37856/0 (socket says 172.21.15.78:37856) 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.192+0000 7f5f5982a640 1 -- 172.21.15.78:0/3301628342 learned_addr learned my addr 172.21.15.78:0/3301628342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5982a640 1 -- 172.21.15.78:0/3301628342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 msgr2=0x56081c2ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5982a640 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5982a640 1 -- 172.21.15.78:0/3301628342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2c9250 msgr2=0x56081c2965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5982a640 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2c9250 0x56081c2965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5982a640 1 -- 172.21.15.78:0/3301628342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f44009760 con 0x56081c2d58b0 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5982a640 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2e7940 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x56081c2d33f0 tx=0x7f5f440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4669571bc2078f45 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f357fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:58:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f357fa640 1 -- 172.21.15.78:0/3301628342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44003810 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f357fa640 1 -- 172.21.15.78:0/3301628342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f5f44003970 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56081befe6c0 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f357fa640 1 -- 172.21.15.78:0/3301628342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=159) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f44003b30 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5ee68740 10 client.9651 fetch_fsmap learned FSMap version 159 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=159,monmap=2+}) v3 -- 0x56081c1fd2b0 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f357fa640 1 -- 172.21.15.78:0/3301628342 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56081befe6c0 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.193+0000 7f5f357fa640 1 -- 172.21.15.78:0/3301628342 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 159) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5f44003c90 con 0x56081c2d58b0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5ee68740 10 client.9651 fetch_fsmap finished waiting for FSMap version 159 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5ee68740 10 client.9651 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5ee68740 20 client.9651 populate_metadata read hostname 'smithi078' 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c2e4000 0x56081c2e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5ee68740 4 client.9651 mds_command: new command op to 9609 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 --> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56081bf4f270 con 0x56081c2e4000 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c2e4000 0x56081c2e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f357fa640 10 client.9651 ms_handle_connect on v2:172.21.15.78:6834/3613671589 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.194+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c2e4000 0x56081c2e43e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6983ca0af07dbca8 server_cookie=98a87982d6e39100 in_seq=0 out_seq=0 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f357fa640 1 -- 172.21.15.78:0/3301628342 <== mds.0 v2:172.21.15.78:6834/3613671589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56081bf4f270 con 0x56081c2e4000 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f357fa640 10 client.9651 handle_command_reply: tid=0 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 client.9651 shutdown 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 msgr2=0x56081c2e7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2e7940 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x56081c2d33f0 tx=0x7f5f440046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 shutdown_connections 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c2e4000 0x56081c2e43e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2e7940 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2c9250 0x56081c2965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3301628342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2d2740 0x56081c2ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f357fa640 0 client.9651 ms_handle_reset on v2:172.21.15.78:6834/3613671589 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 >> 172.21.15.78:0/3301628342 conn(0x56081c298ae0 msgr2=0x56081c2d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 shutdown_connections 2024-02-19T14:58:53.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3301628342 wait complete. 2024-02-19T14:58:53.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.195+0000 7f5f5ee68740 20 client.9651 trim_cache size 0 max 16384 2024-02-19T14:58:53.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 Processor -- start 2024-02-19T14:58:53.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 -- start start 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c2c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c184490 con 0x56081c2d58b0 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c2969c0 con 0x56081c2d2740 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081c2cf950 con 0x56081c2c9250 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5a82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5a82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37872/0 (socket says 172.21.15.78:37872) 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.212+0000 7f5f5a82c640 1 -- 172.21.15.78:0/3990770634 learned_addr learned my addr 172.21.15.78:0/3990770634 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5a82c640 1 -- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 msgr2=0x56081c2d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5982a640 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c2c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5a82c640 1 -- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 msgr2=0x56081c2c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c2c9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5a82c640 1 -- 172.21.15.78:0/3990770634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56081c199a20 con 0x56081c2d58b0 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2ca450 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c07ab20 tx=0x7f5f4c07aa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726626ec57740374 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f377fe640 1 -- 172.21.15.78:0/3990770634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c07b290 con 0x56081c2d58b0 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f377fe640 1 -- 172.21.15.78:0/3990770634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c07ab90 con 0x56081c2d58b0 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f377fe640 1 -- 172.21.15.78:0/3990770634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c07ece0 con 0x56081c2d58b0 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3990770634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 msgr2=0x56081c2ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2ca450 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c07ab20 tx=0x7f5f4c07aa20 comp rx=0 tx=0).stop 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3990770634 shutdown_connections 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c2ca450 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c2c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/3990770634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.213+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3990770634 >> 172.21.15.78:0/3990770634 conn(0x56081c298ae0 msgr2=0x56081c2d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.214+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3990770634 shutdown_connections 2024-02-19T14:58:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.214+0000 7f5f5ee68740 1 -- 172.21.15.78:0/3990770634 wait complete. 2024-02-19T14:58:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.214+0000 7f5f5ee68740 1 -- start start 2024-02-19T14:58:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.214+0000 7f5f5ee68740 1 Processor -- start 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2ca100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c294eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c340780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c2969c0 con 0x56081c2d58b0 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56081c2953f0 con 0x56081c2d2740 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56081c2cba60 con 0x56081c2c9250 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5ee68740 10 client.0 _ll_register_callbacks cb 0x7f5f44009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c340780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c340780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37888/0 (socket says 172.21.15.78:37888) 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1428515684 learned_addr learned my addr 172.21.15.78:0/1428515684 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5982a640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c294eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 msgr2=0x56081c2ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 msgr2=0x56081c294eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c294eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 -- 172.21.15.78:0/1428515684 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5414e610 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.215+0000 7f5f5a82c640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c340780 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c02c900 tx=0x7f5f4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c698e953ab40c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f357fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f357fa640 1 -- 172.21.15.78:0/1428515684 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c07b290 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f357fa640 1 -- 172.21.15.78:0/1428515684 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f5f4c0832a0 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56081c2d6b80 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f357fa640 1 -- 172.21.15.78:0/1428515684 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=159) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f4c07ece0 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 10 client.9657 fetch_fsmap learned FSMap version 159 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=159,monmap=2+}) v3 -- 0x56081c1bf780 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f357fa640 1 -- 172.21.15.78:0/1428515684 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56081c2d6b80 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f357fa640 1 -- 172.21.15.78:0/1428515684 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 159) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5f4c07adf0 con 0x56081c2d58b0 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 10 client.9657 fetch_fsmap finished waiting for FSMap version 159 2024-02-19T14:58:53.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 10 client.9657 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 20 client.9657 populate_metadata read hostname 'smithi078' 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c340cc0 0x56081c2f6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 4 client.9657 mds_command: new command op to 9609 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 --> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f5f44013e70 con 0x56081c340cc0 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.216+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c340cc0 0x56081c2f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5a02b640 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c340cc0 0x56081c2f6760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e191cca3cfe08b46 server_cookie=828027b34d5e6425 in_seq=0 out_seq=0 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f357fa640 10 client.9657 ms_handle_connect on v2:172.21.15.78:6834/3613671589 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f357fa640 1 -- 172.21.15.78:0/1428515684 <== mds.0 v2:172.21.15.78:6834/3613671589 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x7f5f44013e70 con 0x56081c340cc0 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f357fa640 10 client.9657 handle_command_reply: tid=0 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 client.9657 shutdown 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 msgr2=0x56081c340780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c340780 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c02c900 tx=0x7f5f4c00b040 comp rx=0 tx=0).stop 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 shutdown_connections 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:6834/3613671589,v1:172.21.15.78:6836/3613671589] conn(0x56081c340cc0 0x56081c2f6760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56081c2d58b0 0x56081c340780 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56081c2d2740 0x56081c294eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f357fa640 0 client.9657 ms_handle_reset on v2:172.21.15.78:6834/3613671589 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 --2- 172.21.15.78:0/1428515684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56081c2c9250 0x56081c2ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 >> 172.21.15.78:0/1428515684 conn(0x56081c298ae0 msgr2=0x56081c2cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 shutdown_connections 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 1 -- 172.21.15.78:0/1428515684 wait complete. 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.217+0000 7f5f5ee68740 20 client.9657 trim_cache size 0 max 16384 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:58:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 9609, 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 2.9633061999999999, 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 159, 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 176, 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 176, 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 4.9542586709999998 2024-02-19T14:58:53.227 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 msgr2=0x7f5f3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5f50001050 tx=0x7f5f50009040 comp rx=0 tx=0).stop 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 msgr2=0x7f5f541edb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 0x7f5f541edb10 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00e170 tx=0x7f5f4c009750 comp rx=0 tx=0).stop 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 shutdown_connections 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5f3806b810 0x7f5f3806dcd0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f540c7010 0x7f5f541edb10 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5f5414c890 0x7f5f540c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 --2- 172.21.15.78:0/1148389995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40004d00 0x7f5f5414e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 >> 172.21.15.78:0/1148389995 conn(0x7f5f54066e70 msgr2=0x7f5f54155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 shutdown_connections 2024-02-19T14:58:53.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.218+0000 7f5f5c28f640 1 -- 172.21.15.78:0/1148389995 wait complete. 2024-02-19T14:58:53.231 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:53.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.330+0000 7f53c5a74640 1 Processor -- start 2024-02-19T14:58:53.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.330+0000 7f53c5a74640 1 -- start start 2024-02-19T14:58:53.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53c5a74640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53c00c6d60 0x7f53c0154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53c5a74640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53c006ff80 0x7f53c00c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53c5a74640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c014f730 con 0x7f53c006e830 2024-02-19T14:58:53.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53c5a74640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00c1940 con 0x7f53c006ff80 2024-02-19T14:58:53.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53c5a74640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00c16d0 con 0x7f53c00c6d60 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53c006ff80 0x7f53c00c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f53c006ff80 0x7f53c00c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37890/0 (socket says 172.21.15.78:37890) 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 -- 172.21.15.78:0/149188962 learned_addr learned my addr 172.21.15.78:0/149188962 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bf7fe640 1 --2- 172.21.15.78:0/149188962 >> v2:172.21.15.150:3300/0 conn(0x7f53c00c6d60 0x7f53c0154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 -- 172.21.15.78:0/149188962 >> v2:172.21.15.150:3300/0 conn(0x7f53c00c6d60 msgr2=0x7f53c0154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 --2- 172.21.15.78:0/149188962 >> v2:172.21.15.150:3300/0 conn(0x7f53c00c6d60 0x7f53c0154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 -- 172.21.15.78:0/149188962 >> v1:172.21.15.150:6789/0 conn(0x7f53c006e830 legacy=0x7f53c00c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.331+0000 7f53bffff640 1 -- 172.21.15.78:0/149188962 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c0072f70 con 0x7f53c006ff80 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.332+0000 7f53bffff640 1 --2- 172.21.15.78:0/149188962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c006ff80 0x7f53c00c7850 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f53c00704a0 tx=0x7f53b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a263f23015e7bc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.332+0000 7f53beffd640 1 -- 172.21.15.78:0/149188962 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b802ca80 con 0x7f53c006ff80 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.332+0000 7f53beffd640 1 -- 172.21.15.78:0/149188962 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b802cbe0 con 0x7f53c006ff80 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.332+0000 7f53beffd640 1 -- 172.21.15.78:0/149188962 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b80343f0 con 0x7f53c006ff80 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 -- 172.21.15.78:0/149188962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c006ff80 msgr2=0x7f53c00c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 --2- 172.21.15.78:0/149188962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c006ff80 0x7f53c00c7850 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f53c00704a0 tx=0x7f53b802bee0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 -- 172.21.15.78:0/149188962 shutdown_connections 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 --2- 172.21.15.78:0/149188962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c006ff80 0x7f53c00c7850 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 --2- 172.21.15.78:0/149188962 >> v2:172.21.15.150:3300/0 conn(0x7f53c00c6d60 0x7f53c0154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 -- 172.21.15.78:0/149188962 >> 172.21.15.78:0/149188962 conn(0x7f53c006c170 msgr2=0x7f53c006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 -- 172.21.15.78:0/149188962 shutdown_connections 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.333+0000 7f53c5a74640 1 -- 172.21.15.78:0/149188962 wait complete. 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.334+0000 7f53c5a74640 1 Processor -- start 2024-02-19T14:58:53.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.334+0000 7f53c5a74640 1 -- start start 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53c006e830 0x7f53c00c2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53c006ff80 0x7f53c0157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 0x7f53c0157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00c4f10 con 0x7f53c00c6d60 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0157280 con 0x7f53c006ff80 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0158eb0 con 0x7f53c006e830 2024-02-19T14:58:53.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 0x7f53c0157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 0x7f53c0157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37894/0 (socket says 172.21.15.78:37894) 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 -- 172.21.15.78:0/260872145 learned_addr learned my addr 172.21.15.78:0/260872145 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bf7fe640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53c006e830 0x7f53c00c2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53b6ffd640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53c006ff80 0x7f53c0157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 -- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53c006e830 msgr2=0x7f53c00c2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53c006e830 0x7f53c00c2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 -- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53c006ff80 msgr2=0x7f53c0157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53c006ff80 0x7f53c0157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 -- 172.21.15.78:0/260872145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b8009b30 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bffff640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 0x7f53c0157c50 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f53b8036040 tx=0x7f53b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8009510 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b803e450 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c0159190 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b803e7e0 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.335+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c00c4b30 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.336+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c0157430 con 0x7f53c00c6d60 2024-02-19T14:58:53.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.337+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f53b802cd50 con 0x7f53c00c6d60 2024-02-19T14:58:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.337+0000 7f53bcff9640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53a006b810 0x7f53a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.337+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f53b80ad030 con 0x7f53c00c6d60 2024-02-19T14:58:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.337+0000 7f53bf7fe640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53a006b810 0x7f53a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.338+0000 7f53bf7fe640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53a006b810 0x7f53a006dcd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f53b0006fd0 tx=0x7f53b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.341+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f53c0157430 con 0x7f53c00c6d60 2024-02-19T14:58:53.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.550+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53c01573f0 con 0x7f53c00c6d60 2024-02-19T14:58:53.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.550+0000 7f53bcff9640 1 -- 172.21.15.78:0/260872145 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f53b8077ad0 con 0x7f53c00c6d60 2024-02-19T14:58:53.553 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:53.553 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.262364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:active","state_seq":2,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:53.553 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 159 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53a006b810 msgr2=0x7f53a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53a006b810 0x7f53a006dcd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f53b0006fd0 tx=0x7f53b0008040 comp rx=0 tx=0).stop 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 msgr2=0x7f53c0157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 0x7f53c0157c50 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f53b8036040 tx=0x7f53b8004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 shutdown_connections 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f53a006b810 0x7f53a006dcd0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53c00c6d60 0x7f53c0157c50 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53c006ff80 0x7f53c0157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 --2- 172.21.15.78:0/260872145 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53c006e830 0x7f53c00c2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 >> 172.21.15.78:0/260872145 conn(0x7f53c006c170 msgr2=0x7f53c00721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 shutdown_connections 2024-02-19T14:58:53.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.552+0000 7f53c5a74640 1 -- 172.21.15.78:0/260872145 wait complete. 2024-02-19T14:58:53.566 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:58:53.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:58:53.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:58:53.659 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:58:53.659 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:58:53.659 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:58:53.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:58:53.674 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:58:53.674 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:58:53.691 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:58:53.691 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:53.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.812+0000 7f2e4bef4640 1 Processor -- start 2024-02-19T14:58:53.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.812+0000 7f2e4bef4640 1 -- start start 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4bef4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e4414cd50 0x7f2e440c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4bef4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2e44156860 0x7f2e44156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4bef4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e441563b0 con 0x7f2e441537a0 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4bef4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e44153cd0 con 0x7f2e44156860 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4bef4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4406b6e0 con 0x7f2e4414cd50 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4a491640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2e44156860 0x7f2e44156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4948f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2e441537a0 0x7f2e440c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46806/0 (socket says 172.21.15.78:46806) 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4948f640 1 -- 172.21.15.78:0/4203648692 learned_addr learned my addr 172.21.15.78:0/4203648692 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4a491640 1 -- 172.21.15.78:0/4203648692 >> v2:172.21.15.150:3300/0 conn(0x7f2e4414cd50 msgr2=0x7f2e440c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4a491640 1 --2- 172.21.15.78:0/4203648692 >> v2:172.21.15.150:3300/0 conn(0x7f2e4414cd50 0x7f2e440c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4a491640 1 -- 172.21.15.78:0/4203648692 >> v1:172.21.15.78:6789/0 conn(0x7f2e441537a0 legacy=0x7f2e440c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e48c8e640 1 -- 172.21.15.78:0/4203648692 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193656699 0 0) 0x7f2e441563b0 con 0x7f2e441537a0 2024-02-19T14:58:53.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.814+0000 7f2e4a491640 1 -- 172.21.15.78:0/4203648692 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4406b980 con 0x7f2e44156860 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.815+0000 7f2e4a491640 1 --2- 172.21.15.78:0/4203648692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e44156860 0x7f2e44156cd0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f2e440ace20 tx=0x7f2e3c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1443b70f82402f6b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.815+0000 7f2e48c8e640 1 -- 172.21.15.78:0/4203648692 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c02cb60 con 0x7f2e44156860 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.815+0000 7f2e48c8e640 1 -- 172.21.15.78:0/4203648692 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c02ccc0 con 0x7f2e44156860 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.815+0000 7f2e48c8e640 1 -- 172.21.15.78:0/4203648692 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c034660 con 0x7f2e44156860 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.815+0000 7f2e4bef4640 1 -- 172.21.15.78:0/4203648692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e44156860 msgr2=0x7f2e44156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.815+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/4203648692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e44156860 0x7f2e44156cd0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f2e440ace20 tx=0x7f2e3c02bdb0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.816+0000 7f2e4bef4640 1 -- 172.21.15.78:0/4203648692 shutdown_connections 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.816+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/4203648692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e44156860 0x7f2e44156cd0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.816+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/4203648692 >> v2:172.21.15.150:3300/0 conn(0x7f2e4414cd50 0x7f2e440c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.816+0000 7f2e4bef4640 1 -- 172.21.15.78:0/4203648692 >> 172.21.15.78:0/4203648692 conn(0x7f2e44066e70 msgr2=0x7f2e440a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.816+0000 7f2e4bef4640 1 -- 172.21.15.78:0/4203648692 shutdown_connections 2024-02-19T14:58:53.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.816+0000 7f2e4bef4640 1 -- 172.21.15.78:0/4203648692 wait complete. 2024-02-19T14:58:53.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 Processor -- start 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 -- start start 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e4414cd50 0x7f2e441498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 0x7f2e441dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e44156860 0x7f2e441edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e44149760 con 0x7f2e441537a0 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440aadd0 con 0x7f2e44156860 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4bef4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4414beb0 con 0x7f2e4414cd50 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4948f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 0x7f2e441dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.817+0000 7f2e4948f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 0x7f2e441dc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37906/0 (socket says 172.21.15.78:37906) 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 -- 172.21.15.78:0/1578218990 learned_addr learned my addr 172.21.15.78:0/1578218990 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4a491640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e44156860 0x7f2e441edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 -- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e4414cd50 msgr2=0x7f2e441498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e4414cd50 0x7f2e441498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 -- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e44156860 msgr2=0x7f2e441edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e44156860 0x7f2e441edf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 -- 172.21.15.78:0/1578218990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c009a70 con 0x7f2e441537a0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4948f640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 0x7f2e441dc5f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c410 tx=0x7f2e3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e340074e0 con 0x7f2e441537a0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e34007660 con 0x7f2e441537a0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e440aa9f0 con 0x7f2e441537a0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34004a20 con 0x7f2e441537a0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.818+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e441493a0 con 0x7f2e441537a0 2024-02-19T14:58:53.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.819+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e440aa850 con 0x7f2e441537a0 2024-02-19T14:58:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.820+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2e34004b80 con 0x7f2e441537a0 2024-02-19T14:58:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.820+0000 7f2e3a7fc640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e2806b810 0x7f2e2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.820+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f2e3408a160 con 0x7f2e441537a0 2024-02-19T14:58:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.820+0000 7f2e49c90640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e2806b810 0x7f2e2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.821+0000 7f2e49c90640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e2806b810 0x7f2e2806dcd0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009670 tx=0x7f2e40006bf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:53.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:53.824+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2e440aa850 con 0x7f2e441537a0 2024-02-19T14:58:54.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.040+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e440aa850 con 0x7f2e441537a0 2024-02-19T14:58:54.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.041+0000 7f2e3a7fc640 1 -- 172.21.15.78:0/1578218990 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f2e440aa850 con 0x7f2e441537a0 2024-02-19T14:58:54.044 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:54.044 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.262364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:active","state_seq":2,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:54.044 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 159 2024-02-19T14:58:54.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e2806b810 msgr2=0x7f2e2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e2806b810 0x7f2e2806dcd0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009670 tx=0x7f2e40006bf0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 msgr2=0x7f2e441dc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 0x7f2e441dc5f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c410 tx=0x7f2e3400c8d0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 shutdown_connections 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e2806b810 0x7f2e2806dcd0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e441537a0 0x7f2e441dc5f0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e44156860 0x7f2e441edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 --2- 172.21.15.78:0/1578218990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e4414cd50 0x7f2e441498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 >> 172.21.15.78:0/1578218990 conn(0x7f2e44066e70 msgr2=0x7f2e440c7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 shutdown_connections 2024-02-19T14:58:54.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.043+0000 7f2e4bef4640 1 -- 172.21.15.78:0/1578218990 wait complete. 2024-02-19T14:58:54.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.156+0000 7f093a82a640 1 Processor -- start 2024-02-19T14:58:54.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.156+0000 7f093a82a640 1 -- start start 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f093a82a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09340acb50 0x7f0934155900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f093a82a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0934158670 0x7f0934158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f093a82a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340c1d80 con 0x7f093406da10 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f093a82a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093406caf0 con 0x7f09340acb50 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f093a82a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09340c37e0 con 0x7f0934158670 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f0938dc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0934158670 0x7f0934158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f09337fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f093406da10 0x7f09340723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:48582/0 (socket says 172.21.15.78:48582) 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f09337fe640 1 -- 172.21.15.78:0/3528320930 learned_addr learned my addr 172.21.15.78:0/3528320930 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:54.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.157+0000 7f0933fff640 1 --2- 172.21.15.78:0/3528320930 >> v2:172.21.15.150:3300/0 conn(0x7f09340acb50 0x7f0934155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.158+0000 7f0938dc7640 1 -- 172.21.15.78:0/3528320930 >> v2:172.21.15.150:3300/0 conn(0x7f09340acb50 msgr2=0x7f0934155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:54.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.158+0000 7f0938dc7640 1 --2- 172.21.15.78:0/3528320930 >> v2:172.21.15.150:3300/0 conn(0x7f09340acb50 0x7f0934155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.158+0000 7f0938dc7640 1 -- 172.21.15.78:0/3528320930 >> v1:172.21.15.150:6789/0 conn(0x7f093406da10 legacy=0x7f09340723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.158+0000 7f0938dc7640 1 -- 172.21.15.78:0/3528320930 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093406c5a0 con 0x7f0934158670 2024-02-19T14:58:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.158+0000 7f0933fff640 1 --2- 172.21.15.78:0/3528320930 >> v2:172.21.15.150:3300/0 conn(0x7f09340acb50 0x7f0934155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.158+0000 7f0938dc7640 1 --2- 172.21.15.78:0/3528320930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0934158670 0x7f0934158ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f092c009d90 tx=0x7f092c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a7c258e6f4d043 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f0932ffd640 1 -- 172.21.15.78:0/3528320930 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c02cbc0 con 0x7f0934158670 2024-02-19T14:58:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f0932ffd640 1 -- 172.21.15.78:0/3528320930 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c02cd20 con 0x7f0934158670 2024-02-19T14:58:54.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f0932ffd640 1 -- 172.21.15.78:0/3528320930 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c034660 con 0x7f0934158670 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 -- 172.21.15.78:0/3528320930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0934158670 msgr2=0x7f0934158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 --2- 172.21.15.78:0/3528320930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0934158670 0x7f0934158ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f092c009d90 tx=0x7f092c02be10 comp rx=0 tx=0).stop 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 -- 172.21.15.78:0/3528320930 shutdown_connections 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 --2- 172.21.15.78:0/3528320930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0934158670 0x7f0934158ae0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 --2- 172.21.15.78:0/3528320930 >> v2:172.21.15.150:3300/0 conn(0x7f09340acb50 0x7f0934155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 -- 172.21.15.78:0/3528320930 >> 172.21.15.78:0/3528320930 conn(0x7f09340718e0 msgr2=0x7f0934071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 -- 172.21.15.78:0/3528320930 shutdown_connections 2024-02-19T14:58:54.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.159+0000 7f093a82a640 1 -- 172.21.15.78:0/3528320930 wait complete. 2024-02-19T14:58:54.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.160+0000 7f093a82a640 1 Processor -- start 2024-02-19T14:58:54.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 -- start start 2024-02-19T14:58:54.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f093406da10 0x7f09340a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09340acb50 0x7f09340c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 0x7f09341467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09340aaad0 con 0x7f0934158670 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09341485d0 con 0x7f09340acb50 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f093a82a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09341eafd0 con 0x7f093406da10 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f0938dc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 0x7f09341467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f0938dc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 0x7f09341467a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37916/0 (socket says 172.21.15.78:37916) 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.161+0000 7f0938dc7640 1 -- 172.21.15.78:0/1163194768 learned_addr learned my addr 172.21.15.78:0/1163194768 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f09337fe640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09340acb50 0x7f09340c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0933fff640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f093406da10 0x7f09340a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0938dc7640 1 -- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f093406da10 msgr2=0x7f09340a9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0938dc7640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f093406da10 0x7f09340a9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0938dc7640 1 -- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09340acb50 msgr2=0x7f09340c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0938dc7640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09340acb50 0x7f09340c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0938dc7640 1 -- 172.21.15.78:0/1163194768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924009400 con 0x7f0934158670 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0938dc7640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 0x7f09341467a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f092c009eb0 tx=0x7f092c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c040650 con 0x7f0934158670 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c02ce90 con 0x7f0934158670 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c009a70 con 0x7f0934158670 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c0409f0 con 0x7f0934158670 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.162+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09341eb500 con 0x7f0934158670 2024-02-19T14:58:54.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.163+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0934074990 con 0x7f0934158670 2024-02-19T14:58:54.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.163+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f092c040b50 con 0x7f0934158670 2024-02-19T14:58:54.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.164+0000 7f0930ff9640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f092006b810 0x7f092006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.164+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f092c0acf20 con 0x7f0934158670 2024-02-19T14:58:54.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.164+0000 7f0933fff640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f092006b810 0x7f092006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.164+0000 7f0933fff640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f092006b810 0x7f092006dcd0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0924009bc0 tx=0x7f0924008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.168+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f092c0777f0 con 0x7f0934158670 2024-02-19T14:58:54.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.422+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09341565a0 con 0x7f0934158670 2024-02-19T14:58:54.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.423+0000 7f0930ff9640 1 -- 172.21.15.78:0/1163194768 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+17085 (secure 0 0 0) 0x7f092c009450 con 0x7f0934158670 2024-02-19T14:58:54.426 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:54.426 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":176,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:42.205276+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:58:25.520675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-19T14:58:26.281810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"28.7","mappings":[{"from":6,"to":4}]},{"pgid":"28.2b","mappings":[{"from":6,"to":4}]},{"pgid":"28.38","mappings":[{"from":6,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:54.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f092006b810 msgr2=0x7f092006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f092006b810 0x7f092006dcd0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0924009bc0 tx=0x7f0924008040 comp rx=0 tx=0).stop 2024-02-19T14:58:54.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 msgr2=0x7f09341467a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 0x7f09341467a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f092c009eb0 tx=0x7f092c004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:54.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 shutdown_connections 2024-02-19T14:58:54.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f092006b810 0x7f092006dcd0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0934158670 0x7f09341467a0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09340acb50 0x7f09340c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 --2- 172.21.15.78:0/1163194768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f093406da10 0x7f09340a9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 >> 172.21.15.78:0/1163194768 conn(0x7f09340718e0 msgr2=0x7f0934157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 shutdown_connections 2024-02-19T14:58:54.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.425+0000 7f093a82a640 1 -- 172.21.15.78:0/1163194768 wait complete. 2024-02-19T14:58:54.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:58:25.520675+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '173', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:58:26.281810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:58:54.439 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:58:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.541+0000 7fc8d25c9640 1 Processor -- start 2024-02-19T14:58:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.541+0000 7fc8d25c9640 1 -- start start 2024-02-19T14:58:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.541+0000 7fc8d25c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8cc0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.541+0000 7fc8d25c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc0c65a0 con 0x7fc8cc070720 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.541+0000 7fc8d25c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc0c7530 con 0x7fc8cc0c7910 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.541+0000 7fc8d25c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc1537b0 con 0x7fc8cc0c5560 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8d0b66640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc8cc070720 0x7fc8cc070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46810/0 (socket says 172.21.15.78:46810) 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8d0b66640 1 -- 172.21.15.78:0/736990765 learned_addr learned my addr 172.21.15.78:0/736990765 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142671683 0 0) 0x7fc8cc0c65a0 con 0x7fc8cc070720 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4003520 con 0x7fc8cc070720 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8cc0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744655524 0 0) 0x7fc8b4003520 con 0x7fc8cc070720 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 msgr2=0x7fc8cc0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8cc0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 >> v1:172.21.15.150:6789/0 conn(0x7fc8cc0c7910 legacy=0x7fc8cc154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc153a90 con 0x7fc8cc070720 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fc8c40019f0 con 0x7fc8cc070720 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8cc0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 >> v1:172.21.15.78:6789/0 conn(0x7fc8cc070720 legacy=0x7fc8cc070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8b40039a0 0x7fc8b4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8b4004e20 0x7fc8b400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc1537b0 con 0x7fc8b4004e20 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40055e0 con 0x7fc8b40039a0 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b400bec0 con 0x7fc8cc0c5560 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8b4004e20 0x7fc8b400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8b40039a0 0x7fc8b4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.542+0000 7fc8cb7fe640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8d0b66640 1 -- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 msgr2=0x7fc8b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8d0b66640 1 -- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8b40039a0 msgr2=0x7fc8b4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8b40039a0 0x7fc8b4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8d0b66640 1 -- 172.21.15.78:0/736990765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4003520 con 0x7fc8b4004e20 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8b40039a0 0x7fc8b4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8cb7fe640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0c5560 0x7fc8b40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8b4004e20 0x7fc8b400b980 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400a830 tx=0x7fc8b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7423bb775b44166b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4034630 con 0x7fc8b4004e20 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c402f070 con 0x7fc8b4004e20 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.543+0000 7fc8caffd640 1 -- 172.21.15.78:0/736990765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40349f0 con 0x7fc8b4004e20 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d25c9640 1 -- 172.21.15.78:0/736990765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8b4004e20 msgr2=0x7fc8b400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/736990765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8b4004e20 0x7fc8b400b980 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400a830 tx=0x7fc8b40057f0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d0b66640 1 -- 172.21.15.78:0/736990765 reap_dead start 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d25c9640 1 -- 172.21.15.78:0/736990765 shutdown_connections 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d25c9640 1 -- 172.21.15.78:0/736990765 >> 172.21.15.78:0/736990765 conn(0x7fc8cc06b4b0 msgr2=0x7fc8cc0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d25c9640 1 -- 172.21.15.78:0/736990765 shutdown_connections 2024-02-19T14:58:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.544+0000 7fc8d25c9640 1 -- 172.21.15.78:0/736990765 wait complete. 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.546+0000 7fc8d25c9640 1 Processor -- start 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.546+0000 7fc8d25c9640 1 -- start start 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d25c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0a83e0 0x7fc8cc14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d25c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8cc0a9090 0x7fc8cc14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d25c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 0x7fc8cc0aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d25c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc14d850 con 0x7fc8cc0aa170 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d25c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0a9720 con 0x7fc8cc0a9090 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d25c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc1eee20 con 0x7fc8cc0a83e0 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 0x7fc8cc0aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 0x7fc8cc0aa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37936/0 (socket says 172.21.15.78:37936) 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 -- 172.21.15.78:0/1855671787 learned_addr learned my addr 172.21.15.78:0/1855671787 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8cb7fe640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8cc0a9090 0x7fc8cc14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0a83e0 0x7fc8cc14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 -- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0a83e0 msgr2=0x7fc8cc14c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0a83e0 0x7fc8cc14c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 -- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8cc0a9090 msgr2=0x7fc8cc14cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8cc0a9090 0x7fc8cc14cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8d0b66640 1 -- 172.21.15.78:0/1855671787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0c5ac0 con 0x7fc8cc0aa170 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.547+0000 7fc8cb7fe640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8cc0a9090 0x7fc8cc14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.548+0000 7fc8d0b66640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 0x7fc8cc0aa5f0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400a950 tx=0x7fc8c4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.548+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4034840 con 0x7fc8cc0aa170 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.548+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c402ce90 con 0x7fc8cc0aa170 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.548+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4034ba0 con 0x7fc8cc0aa170 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.548+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc0a9970 con 0x7fc8cc0aa170 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.548+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc0adad0 con 0x7fc8cc0aa170 2024-02-19T14:58:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.549+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc0c4fd0 con 0x7fc8cc0aa170 2024-02-19T14:58:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.550+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc8c4031030 con 0x7fc8cc0aa170 2024-02-19T14:58:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.550+0000 7fc8c8ff9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8b806b840 0x7fc8b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.550+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7fc8c40abd60 con 0x7fc8cc0aa170 2024-02-19T14:58:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.550+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8b806b840 0x7fc8b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.551+0000 7fc8cbfff640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8b806b840 0x7fc8b806dd00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000a830 tx=0x7fc8c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.554+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc8c4076650 con 0x7fc8cc0aa170 2024-02-19T14:58:54.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.732+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8cc154350 con 0x7fc8cc0aa170 2024-02-19T14:58:54.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.732+0000 7fc8c8ff9640 1 -- 172.21.15.78:0/1855671787 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v176) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8c4045070 con 0x7fc8cc0aa170 2024-02-19T14:58:54.735 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8b806b840 msgr2=0x7fc8b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8b806b840 0x7fc8b806dd00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000a830 tx=0x7fc8c0008040 comp rx=0 tx=0).stop 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 msgr2=0x7fc8cc0aa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 0x7fc8cc0aa5f0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400a950 tx=0x7fc8c4001d80 comp rx=0 tx=0).stop 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 shutdown_connections 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc8b806b840 0x7fc8b806dd00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8cc0aa170 0x7fc8cc0aa5f0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8cc0a9090 0x7fc8cc14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 --2- 172.21.15.78:0/1855671787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8cc0a83e0 0x7fc8cc14c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 >> 172.21.15.78:0/1855671787 conn(0x7fc8cc06b4b0 msgr2=0x7fc8cc0c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 shutdown_connections 2024-02-19T14:58:54.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.735+0000 7fc8d25c9640 1 -- 172.21.15.78:0/1855671787 wait complete. 2024-02-19T14:58:54.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:58:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.852+0000 7feda0cd4640 1 Processor -- start 2024-02-19T14:58:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.852+0000 7feda0cd4640 1 -- start start 2024-02-19T14:58:54.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7feda0cd4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fed9c0acb50 0x7fed9c155900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7feda0cd4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed9c158670 0x7fed9c158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7feda0cd4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c0c1d80 con 0x7fed9c06da10 2024-02-19T14:58:54.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7feda0cd4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9c06caf0 con 0x7fed9c0acb50 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7feda0cd4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9c0c37e0 con 0x7fed9c158670 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9a59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fed9c0acb50 0x7fed9c155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed99d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fed9c06da10 0x7fed9c0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46822/0 (socket says 172.21.15.78:46822) 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed99d9b640 1 -- 172.21.15.78:0/4056799667 learned_addr learned my addr 172.21.15.78:0/4056799667 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9ad9d640 1 --2- 172.21.15.78:0/4056799667 >> v2:172.21.15.150:3300/0 conn(0x7fed9c158670 0x7fed9c158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9a59c640 1 -- 172.21.15.78:0/4056799667 >> v2:172.21.15.150:3300/0 conn(0x7fed9c158670 msgr2=0x7fed9c158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9a59c640 1 --2- 172.21.15.78:0/4056799667 >> v2:172.21.15.150:3300/0 conn(0x7fed9c158670 0x7fed9c158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9a59c640 1 -- 172.21.15.78:0/4056799667 >> v1:172.21.15.78:6789/0 conn(0x7fed9c06da10 legacy=0x7fed9c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9959a640 1 -- 172.21.15.78:0/4056799667 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578167247 0 0) 0x7fed9c0c1d80 con 0x7fed9c06da10 2024-02-19T14:58:54.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.853+0000 7fed9a59c640 1 -- 172.21.15.78:0/4056799667 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c06c5a0 con 0x7fed9c0acb50 2024-02-19T14:58:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.854+0000 7fed9a59c640 1 --2- 172.21.15.78:0/4056799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c155900 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0c1720 tx=0x7fed9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52366bae1cd68539 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.854+0000 7fed9959a640 1 -- 172.21.15.78:0/4056799667 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9402e070 con 0x7fed9c0acb50 2024-02-19T14:58:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.854+0000 7fed9959a640 1 -- 172.21.15.78:0/4056799667 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed940315a0 con 0x7fed9c0acb50 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.854+0000 7fed9959a640 1 -- 172.21.15.78:0/4056799667 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed94031930 con 0x7fed9c0acb50 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 -- 172.21.15.78:0/4056799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 msgr2=0x7fed9c155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 --2- 172.21.15.78:0/4056799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c155900 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0c1720 tx=0x7fed9402b840 comp rx=0 tx=0).stop 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 -- 172.21.15.78:0/4056799667 shutdown_connections 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 --2- 172.21.15.78:0/4056799667 >> v2:172.21.15.150:3300/0 conn(0x7fed9c158670 0x7fed9c158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 --2- 172.21.15.78:0/4056799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c155900 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 -- 172.21.15.78:0/4056799667 >> 172.21.15.78:0/4056799667 conn(0x7fed9c0718e0 msgr2=0x7fed9c071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 -- 172.21.15.78:0/4056799667 shutdown_connections 2024-02-19T14:58:54.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.855+0000 7feda0cd4640 1 -- 172.21.15.78:0/4056799667 wait complete. 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.856+0000 7feda0cd4640 1 Processor -- start 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.856+0000 7feda0cd4640 1 -- start start 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7feda0cd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed9c06da10 0x7fed9c154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7feda0cd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c1472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7feda0cd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed9c158670 0x7fed9c1ed5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7feda0cd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c0aa580 con 0x7fed9c0acb50 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7feda0cd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c14a120 con 0x7fed9c06da10 2024-02-19T14:58:54.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7feda0cd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9c06bc40 con 0x7fed9c158670 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c1472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c1472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37954/0 (socket says 172.21.15.78:37954) 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 -- 172.21.15.78:0/804347196 learned_addr learned my addr 172.21.15.78:0/804347196 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed9ad9d640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed9c158670 0x7fed9c1ed5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed9a59c640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed9c06da10 0x7fed9c154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 -- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed9c158670 msgr2=0x7fed9c1ed5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed9c158670 0x7fed9c1ed5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 -- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed9c06da10 msgr2=0x7fed9c154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed9c06da10 0x7fed9c154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.857+0000 7fed99d9b640 1 -- 172.21.15.78:0/804347196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94009400 con 0x7fed9c0acb50 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.858+0000 7fed99d9b640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c1472f0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fed9000c430 tx=0x7fed9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.858+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed900074e0 con 0x7fed9c0acb50 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.858+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed90007660 con 0x7fed9c0acb50 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.858+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed90004a20 con 0x7fed9c0acb50 2024-02-19T14:58:54.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.858+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c149e40 con 0x7fed9c0acb50 2024-02-19T14:58:54.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.858+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c1ea7b0 con 0x7fed9c0acb50 2024-02-19T14:58:54.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.859+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c1ea9d0 con 0x7fed9c0acb50 2024-02-19T14:58:54.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.860+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fed90004b80 con 0x7fed9c0acb50 2024-02-19T14:58:54.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.860+0000 7fed8affd640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed7c06b810 0x7fed7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:54.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.860+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7fed9008a140 con 0x7fed9c0acb50 2024-02-19T14:58:54.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.860+0000 7fed9a59c640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed7c06b810 0x7fed7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:54.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.861+0000 7fed9a59c640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed7c06b810 0x7fed7c06dcd0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fed94036000 tx=0x7fed9403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:54.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:54.869+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fed90054a10 con 0x7fed9c0acb50 2024-02-19T14:58:55.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.044+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed9c1ea300 con 0x7fed9c0acb50 2024-02-19T14:58:55.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.045+0000 7fed8affd640 1 -- 172.21.15.78:0/804347196 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v176) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed900580d0 con 0x7fed9c0acb50 2024-02-19T14:58:55.048 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.046+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed7c06b810 msgr2=0x7fed7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.046+0000 7feda0cd4640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed7c06b810 0x7fed7c06dcd0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fed94036000 tx=0x7fed9403d040 comp rx=0 tx=0).stop 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 msgr2=0x7fed9c1472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c1472f0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fed9000c430 tx=0x7fed9000c8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 shutdown_connections 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed7c06b810 0x7fed7c06dcd0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed9c0acb50 0x7fed9c1472f0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed9c158670 0x7fed9c1ed5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 --2- 172.21.15.78:0/804347196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed9c06da10 0x7fed9c154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 >> 172.21.15.78:0/804347196 conn(0x7fed9c0718e0 msgr2=0x7fed9c157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 shutdown_connections 2024-02-19T14:58:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.047+0000 7feda0cd4640 1 -- 172.21.15.78:0/804347196 wait complete. 2024-02-19T14:58:55.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:58:55.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 Processor -- start 2024-02-19T14:58:55.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 -- start start 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c152700 0x7f5e1c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5e1c0c8e00 0x7f5e1c15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c1551f0 con 0x7f5e1c0c5730 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c155490 con 0x7f5e1c0c8e00 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e22fe9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c155770 con 0x7f5e1c152700 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e21586640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5e1c0c8e00 0x7f5e1c15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e1bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5e1c0c5730 0x7f5e1c06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46838/0 (socket says 172.21.15.78:46838) 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.168+0000 7f5e1bfff640 1 -- 172.21.15.78:0/2211242092 learned_addr learned my addr 172.21.15.78:0/2211242092 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.169+0000 7f5e20d85640 1 --2- 172.21.15.78:0/2211242092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c152700 0x7f5e1c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.169+0000 7f5e21586640 1 -- 172.21.15.78:0/2211242092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c152700 msgr2=0x7f5e1c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:55.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.169+0000 7f5e21586640 1 --2- 172.21.15.78:0/2211242092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c152700 0x7f5e1c0c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.169+0000 7f5e21586640 1 -- 172.21.15.78:0/2211242092 >> v1:172.21.15.78:6789/0 conn(0x7f5e1c0c5730 legacy=0x7f5e1c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.169+0000 7f5e21586640 1 -- 172.21.15.78:0/2211242092 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c155a50 con 0x7f5e1c0c8e00 2024-02-19T14:58:55.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e21586640 1 --2- 172.21.15.78:0/2211242092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c0c8e00 0x7f5e1c15f980 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009d90 tx=0x7f5e1402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea13cef44ca470c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e1b7fe640 1 -- 172.21.15.78:0/2211242092 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1402cc50 con 0x7f5e1c0c8e00 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e1b7fe640 1 -- 172.21.15.78:0/2211242092 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1402cdb0 con 0x7f5e1c0c8e00 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e1b7fe640 1 -- 172.21.15.78:0/2211242092 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14034660 con 0x7f5e1c0c8e00 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 -- 172.21.15.78:0/2211242092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c0c8e00 msgr2=0x7f5e1c15f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/2211242092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c0c8e00 0x7f5e1c15f980 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009d90 tx=0x7f5e1402bea0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 -- 172.21.15.78:0/2211242092 shutdown_connections 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/2211242092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c0c8e00 0x7f5e1c15f980 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/2211242092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c152700 0x7f5e1c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 -- 172.21.15.78:0/2211242092 >> 172.21.15.78:0/2211242092 conn(0x7f5e1c0cac90 msgr2=0x7f5e1c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 -- 172.21.15.78:0/2211242092 shutdown_connections 2024-02-19T14:58:55.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.170+0000 7f5e22fe9640 1 -- 172.21.15.78:0/2211242092 wait complete. 2024-02-19T14:58:55.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.172+0000 7f5e22fe9640 1 Processor -- start 2024-02-19T14:58:55.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.172+0000 7f5e22fe9640 1 -- start start 2024-02-19T14:58:55.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.173+0000 7f5e22fe9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c0c5730 0x7f5e1c0c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.173+0000 7f5e22fe9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e1c0c8e00 0x7f5e1c1e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.173+0000 7f5e22fe9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 0x7f5e1c0bee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.173+0000 7f5e22fe9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0acc30 con 0x7f5e1c152700 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.173+0000 7f5e22fe9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0ac0a0 con 0x7f5e1c0c8e00 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.173+0000 7f5e22fe9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0beb70 con 0x7f5e1c0c5730 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 0x7f5e1c0bee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 0x7f5e1c0bee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37970/0 (socket says 172.21.15.78:37970) 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 -- 172.21.15.78:0/3862044902 learned_addr learned my addr 172.21.15.78:0/3862044902 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 -- 172.21.15.78:0/3862044902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c0c5730 msgr2=0x7f5e1c0c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c0c5730 0x7f5e1c0c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 -- 172.21.15.78:0/3862044902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e1c0c8e00 msgr2=0x7f5e1c1e7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e1c0c8e00 0x7f5e1c1e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 -- 172.21.15.78:0/3862044902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e14009a70 con 0x7f5e1c152700 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e21586640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 0x7f5e1c0bee70 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009d90 tx=0x7f5e14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14040650 con 0x7f5e1c152700 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e140407d0 con 0x7f5e1c152700 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c0be890 con 0x7f5e1c152700 2024-02-19T14:58:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14040b80 con 0x7f5e1c152700 2024-02-19T14:58:55.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.174+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c1ee930 con 0x7f5e1c152700 2024-02-19T14:58:55.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.175+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c0688f0 con 0x7f5e1c152700 2024-02-19T14:58:55.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.176+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5e14040ce0 con 0x7f5e1c152700 2024-02-19T14:58:55.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.176+0000 7f5e197fa640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e0006b810 0x7f5e0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.176+0000 7f5e20d85640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e0006b810 0x7f5e0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.176+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f5e140ad170 con 0x7f5e1c152700 2024-02-19T14:58:55.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.177+0000 7f5e20d85640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e0006b810 0x7f5e0006dcd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000a810 tx=0x7f5e10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.180+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5e14077a40 con 0x7f5e1c152700 2024-02-19T14:58:55.242 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:58:55.239+0000 7fe05f43b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:55.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.359+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e1c074990 con 0x7f5e1c152700 2024-02-19T14:58:55.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.359+0000 7f5e197fa640 1 -- 172.21.15.78:0/3862044902 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v176) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e1407b100 con 0x7f5e1c152700 2024-02-19T14:58:55.362 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e0006b810 msgr2=0x7f5e0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e0006b810 0x7f5e0006dcd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000a810 tx=0x7f5e10008040 comp rx=0 tx=0).stop 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 msgr2=0x7f5e1c0bee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 0x7f5e1c0bee70 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009d90 tx=0x7f5e14004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 shutdown_connections 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5e0006b810 0x7f5e0006dcd0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e1c152700 0x7f5e1c0bee70 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e1c0c8e00 0x7f5e1c1e7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 --2- 172.21.15.78:0/3862044902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e1c0c5730 0x7f5e1c0c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 >> 172.21.15.78:0/3862044902 conn(0x7f5e1c0cac90 msgr2=0x7f5e1c14ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 shutdown_connections 2024-02-19T14:58:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.361+0000 7f5e22fe9640 1 -- 172.21.15.78:0/3862044902 wait complete. 2024-02-19T14:58:55.375 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:55.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.472+0000 7f6a79e0e640 1 Processor -- start 2024-02-19T14:58:55.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.472+0000 7f6a79e0e640 1 -- start start 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a79e0e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a740add70 0x7f6a740c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a79e0e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a740705b0 0x7f6a740709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a79e0e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740ae670 con 0x7f6a740c2f60 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a79e0e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740716b0 con 0x7f6a740705b0 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a79e0e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740c33d0 con 0x7f6a740add70 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a740705b0 0x7f6a740709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a740705b0 0x7f6a740709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37986/0 (socket says 172.21.15.78:37986) 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 -- 172.21.15.78:0/567447841 learned_addr learned my addr 172.21.15.78:0/567447841 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a737fe640 1 --2- 172.21.15.78:0/567447841 >> v2:172.21.15.150:3300/0 conn(0x7f6a740add70 0x7f6a740c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 -- 172.21.15.78:0/567447841 >> v2:172.21.15.150:3300/0 conn(0x7f6a740add70 msgr2=0x7f6a740c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 --2- 172.21.15.78:0/567447841 >> v2:172.21.15.150:3300/0 conn(0x7f6a740add70 0x7f6a740c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 -- 172.21.15.78:0/567447841 >> v1:172.21.15.150:6789/0 conn(0x7f6a740c2f60 legacy=0x7f6a7406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a73fff640 1 -- 172.21.15.78:0/567447841 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74153780 con 0x7f6a740705b0 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.473+0000 7f6a737fe640 1 --2- 172.21.15.78:0/567447841 >> v2:172.21.15.150:3300/0 conn(0x7f6a740add70 0x7f6a740c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.474+0000 7f6a73fff640 1 --2- 172.21.15.78:0/567447841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740705b0 0x7f6a740709c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f6a740c4fc0 tx=0x7f6a6c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f9c54bb22c09fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.474+0000 7f6a727fc640 1 -- 172.21.15.78:0/567447841 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c02c9f0 con 0x7f6a740705b0 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.474+0000 7f6a727fc640 1 -- 172.21.15.78:0/567447841 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c02cb50 con 0x7f6a740705b0 2024-02-19T14:58:55.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.474+0000 7f6a727fc640 1 -- 172.21.15.78:0/567447841 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0343f0 con 0x7f6a740705b0 2024-02-19T14:58:55.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 -- 172.21.15.78:0/567447841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740705b0 msgr2=0x7f6a740709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/567447841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740705b0 0x7f6a740709c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f6a740c4fc0 tx=0x7f6a6c02be50 comp rx=0 tx=0).stop 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 -- 172.21.15.78:0/567447841 shutdown_connections 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/567447841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740705b0 0x7f6a740709c0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/567447841 >> v2:172.21.15.150:3300/0 conn(0x7f6a740add70 0x7f6a740c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 -- 172.21.15.78:0/567447841 >> 172.21.15.78:0/567447841 conn(0x7f6a740c85e0 msgr2=0x7f6a740c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 -- 172.21.15.78:0/567447841 shutdown_connections 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.475+0000 7f6a79e0e640 1 -- 172.21.15.78:0/567447841 wait complete. 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.476+0000 7f6a79e0e640 1 Processor -- start 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.476+0000 7f6a79e0e640 1 -- start start 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a79e0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a740705b0 0x7f6a740c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a79e0e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 0x7f6a7414b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a79e0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a740c2f60 0x7f6a7414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a79e0e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a741eb010 con 0x7f6a740add70 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a79e0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7406bf90 con 0x7f6a740c2f60 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a79e0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7406df00 con 0x7f6a740705b0 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 0x7f6a7414b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 0x7f6a7414b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37994/0 (socket says 172.21.15.78:37994) 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 -- 172.21.15.78:0/2039367247 learned_addr learned my addr 172.21.15.78:0/2039367247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 -- 172.21.15.78:0/2039367247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a740705b0 msgr2=0x7f6a740c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a740705b0 0x7f6a740c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 -- 172.21.15.78:0/2039367247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a740c2f60 msgr2=0x7f6a7414bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a740c2f60 0x7f6a7414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 -- 172.21.15.78:0/2039367247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c009b30 con 0x7f6a740add70 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a72ffd640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 0x7f6a7414b7b0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c430 tx=0x7f6a6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68011790 con 0x7f6a740add70 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.477+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a680118f0 con 0x7f6a740add70 2024-02-19T14:58:55.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.478+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740a9c20 con 0x7f6a740add70 2024-02-19T14:58:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.478+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68011ce0 con 0x7f6a740add70 2024-02-19T14:58:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.478+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7406c250 con 0x7f6a740add70 2024-02-19T14:58:55.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.478+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a740676a0 con 0x7f6a740add70 2024-02-19T14:58:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.479+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6a6801c460 con 0x7f6a740add70 2024-02-19T14:58:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.479+0000 7f6a5bfff640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a6006b810 0x7f6a6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.479+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f6a6808a2c0 con 0x7f6a740add70 2024-02-19T14:58:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.479+0000 7f6a737fe640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a6006b810 0x7f6a6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.480+0000 7f6a737fe640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a6006b810 0x7f6a6006dcd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6a64006fd0 tx=0x7f6a64008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.484+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6a68054b90 con 0x7f6a740add70 2024-02-19T14:58:55.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.711+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a7406c440 con 0x7f6a740add70 2024-02-19T14:58:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.711+0000 7f6a5bfff640 1 -- 172.21.15.78:0/2039367247 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f6a68058250 con 0x7f6a740add70 2024-02-19T14:58:55.714 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:55.714 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.262364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:active","state_seq":2,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:55.714 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 159 2024-02-19T14:58:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a6006b810 msgr2=0x7f6a6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a6006b810 0x7f6a6006dcd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6a64006fd0 tx=0x7f6a64008040 comp rx=0 tx=0).stop 2024-02-19T14:58:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 msgr2=0x7f6a7414b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 0x7f6a7414b7b0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c430 tx=0x7f6a6800c8f0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 shutdown_connections 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a6006b810 0x7f6a6006dcd0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a740add70 0x7f6a7414b7b0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a740c2f60 0x7f6a7414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 --2- 172.21.15.78:0/2039367247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a740705b0 0x7f6a740c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 >> 172.21.15.78:0/2039367247 conn(0x7f6a740c85e0 msgr2=0x7f6a741524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 shutdown_connections 2024-02-19T14:58:55.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.713+0000 7f6a79e0e640 1 -- 172.21.15.78:0/2039367247 wait complete. 2024-02-19T14:58:55.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.823+0000 7fd81358c640 1 Processor -- start 2024-02-19T14:58:55.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.824+0000 7fd81358c640 1 -- start start 2024-02-19T14:58:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.824+0000 7fd81358c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.824+0000 7fd81358c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd80c072ee0 0x7fd80c14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.824+0000 7fd81358c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c14f8f0 con 0x7fd80c14ef50 2024-02-19T14:58:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.824+0000 7fd81358c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd80c0c35c0 con 0x7fd80c072ee0 2024-02-19T14:58:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.824+0000 7fd81358c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c0c38a0 con 0x7fd80c0c78a0 2024-02-19T14:58:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811b29640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd80c072ee0 0x7fd80c14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811328640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd810b27640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd80c14ef50 0x7fd80c0727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:48618/0 (socket says 172.21.15.78:48618) 2024-02-19T14:58:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd810b27640 1 -- 172.21.15.78:0/3223137584 learned_addr learned my addr 172.21.15.78:0/3223137584 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd803fff640 1 -- 172.21.15.78:0/3223137584 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578672902 0 0) 0x7fd80c14f8f0 con 0x7fd80c14ef50 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd803fff640 1 -- 172.21.15.78:0/3223137584 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f8003520 con 0x7fd80c14ef50 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811328640 1 -- 172.21.15.78:0/3223137584 >> v2:172.21.15.150:3300/0 conn(0x7fd80c072ee0 msgr2=0x7fd80c14f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811328640 1 --2- 172.21.15.78:0/3223137584 >> v2:172.21.15.150:3300/0 conn(0x7fd80c072ee0 0x7fd80c14f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811328640 1 -- 172.21.15.78:0/3223137584 >> v1:172.21.15.150:6789/0 conn(0x7fd80c14ef50 legacy=0x7fd80c0727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811328640 1 -- 172.21.15.78:0/3223137584 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c0c3b80 con 0x7fd80c0c78a0 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.825+0000 7fd811b29640 1 --2- 172.21.15.78:0/3223137584 >> v2:172.21.15.150:3300/0 conn(0x7fd80c072ee0 0x7fd80c14f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd811328640 1 --2- 172.21.15.78:0/3223137584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c0c44b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd8080097a0 tx=0x7fd80802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b56d95480513c61a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd803fff640 1 -- 172.21.15.78:0/3223137584 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808031830 con 0x7fd80c0c78a0 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd803fff640 1 -- 172.21.15.78:0/3223137584 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd808031990 con 0x7fd80c0c78a0 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd803fff640 1 -- 172.21.15.78:0/3223137584 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808031d00 con 0x7fd80c0c78a0 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd81358c640 1 -- 172.21.15.78:0/3223137584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 msgr2=0x7fd80c0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd81358c640 1 --2- 172.21.15.78:0/3223137584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c0c44b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd8080097a0 tx=0x7fd80802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd81358c640 1 -- 172.21.15.78:0/3223137584 shutdown_connections 2024-02-19T14:58:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd81358c640 1 --2- 172.21.15.78:0/3223137584 >> v2:172.21.15.150:3300/0 conn(0x7fd80c072ee0 0x7fd80c14f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd81358c640 1 --2- 172.21.15.78:0/3223137584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c0c44b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.826+0000 7fd81358c640 1 -- 172.21.15.78:0/3223137584 >> 172.21.15.78:0/3223137584 conn(0x7fd80c066e70 msgr2=0x7fd80c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.827+0000 7fd81358c640 1 -- 172.21.15.78:0/3223137584 shutdown_connections 2024-02-19T14:58:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.827+0000 7fd81358c640 1 -- 172.21.15.78:0/3223137584 wait complete. 2024-02-19T14:58:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 Processor -- start 2024-02-19T14:58:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 -- start start 2024-02-19T14:58:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 0x7fd80c0c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd80c14ef50 0x7fd80c144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c1446a0 con 0x7fd80c072ee0 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c1ee640 con 0x7fd80c14ef50 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd81358c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c124300 con 0x7fd80c0c78a0 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd811328640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 0x7fd80c0c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd811328640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 0x7fd80c0c0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38010/0 (socket says 172.21.15.78:38010) 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.828+0000 7fd811328640 1 -- 172.21.15.78:0/2321833020 learned_addr learned my addr 172.21.15.78:0/2321833020 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd810b27640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811b29640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd80c14ef50 0x7fd80c144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811328640 1 -- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 msgr2=0x7fd80c143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811328640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c143c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811328640 1 -- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd80c14ef50 msgr2=0x7fd80c144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811328640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd80c14ef50 0x7fd80c144160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811328640 1 -- 172.21.15.78:0/2321833020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804009a70 con 0x7fd80c072ee0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd810b27640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811b29640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd80c14ef50 0x7fd80c144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd811328640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 0x7fd80c0c0d40 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fd80802bee0 tx=0x7fd8080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80803a8f0 con 0x7fd80c072ee0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80803aa50 con 0x7fd80c072ee0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd808009400 con 0x7fd80c072ee0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80803ae00 con 0x7fd80c072ee0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.829+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80c1eb570 con 0x7fd80c072ee0 2024-02-19T14:58:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.830+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80c0c67e0 con 0x7fd80c072ee0 2024-02-19T14:58:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.831+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd80804d440 con 0x7fd80c072ee0 2024-02-19T14:58:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.832+0000 7fd801ffb640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd7f006b810 0x7fd7f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.832+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7fd8080ac560 con 0x7fd80c072ee0 2024-02-19T14:58:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.832+0000 7fd810b27640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd7f006b810 0x7fd7f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.832+0000 7fd810b27640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd7f006b810 0x7fd7f006dcd0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc007fd0 tx=0x7fd7fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:55.836+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd80c0c67e0 con 0x7fd80c072ee0 2024-02-19T14:58:56.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.054+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd80c0c67e0 con 0x7fd80c072ee0 2024-02-19T14:58:56.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.055+0000 7fd801ffb640 1 -- 172.21.15.78:0/2321833020 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+17085 (secure 0 0 0) 0x7fd80c0c67e0 con 0x7fd80c072ee0 2024-02-19T14:58:56.058 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:56.058 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":176,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:42.205276+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:58:25.520675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-19T14:58:26.281810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"28.7","mappings":[{"from":6,"to":4}]},{"pgid":"28.2b","mappings":[{"from":6,"to":4}]},{"pgid":"28.38","mappings":[{"from":6,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.057+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd7f006b810 msgr2=0x7fd7f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.057+0000 7fd81358c640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd7f006b810 0x7fd7f006dcd0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc007fd0 tx=0x7fd7fc009040 comp rx=0 tx=0).stop 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.057+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 msgr2=0x7fd80c0c0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.057+0000 7fd81358c640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 0x7fd80c0c0d40 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7fd80802bee0 tx=0x7fd8080046c0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 shutdown_connections 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd7f006b810 0x7fd7f006dcd0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd80c14ef50 0x7fd80c144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd80c0c78a0 0x7fd80c143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 --2- 172.21.15.78:0/2321833020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd80c072ee0 0x7fd80c0c0d40 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 >> 172.21.15.78:0/2321833020 conn(0x7fd80c066e70 msgr2=0x7fd80c1506d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:56.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 shutdown_connections 2024-02-19T14:58:56.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.058+0000 7fd81358c640 1 -- 172.21.15.78:0/2321833020 wait complete. 2024-02-19T14:58:56.071 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:58:56.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.170+0000 7ffae4d25640 1 Processor -- start 2024-02-19T14:58:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.170+0000 7ffae4d25640 1 -- start start 2024-02-19T14:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffae4d25640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffae00c78a0 0x7ffae0159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffae4d25640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffae00c4d80 0x7ffae00c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffae4d25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae01523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffae4d25640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae006b720 con 0x7ffae00c4d80 2024-02-19T14:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffae4d25640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae006ffa0 con 0x7ffae00c78a0 2024-02-19T14:58:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffae4d25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae006fc70 con 0x7ffae00c48f0 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffad7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffae00c4d80 0x7ffae00c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffad7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffae00c4d80 0x7ffae00c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38024/0 (socket says 172.21.15.78:38024) 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.171+0000 7ffad7fff640 1 -- 172.21.15.78:0/3400046455 learned_addr learned my addr 172.21.15.78:0/3400046455 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffade59c640 1 --2- 172.21.15.78:0/3400046455 >> v2:172.21.15.150:3300/0 conn(0x7ffae00c78a0 0x7ffae0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffaded9d640 1 --2- 172.21.15.78:0/3400046455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae01523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffad7fff640 1 -- 172.21.15.78:0/3400046455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 msgr2=0x7ffae01523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffad7fff640 1 --2- 172.21.15.78:0/3400046455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae01523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffad7fff640 1 -- 172.21.15.78:0/3400046455 >> v2:172.21.15.150:3300/0 conn(0x7ffae00c78a0 msgr2=0x7ffae0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffad7fff640 1 --2- 172.21.15.78:0/3400046455 >> v2:172.21.15.150:3300/0 conn(0x7ffae00c78a0 0x7ffae0159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffad7fff640 1 -- 172.21.15.78:0/3400046455 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae006b930 con 0x7ffae00c4d80 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffade59c640 1 --2- 172.21.15.78:0/3400046455 >> v2:172.21.15.150:3300/0 conn(0x7ffae00c78a0 0x7ffae0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.172+0000 7ffad7fff640 1 --2- 172.21.15.78:0/3400046455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c4d80 0x7ffae00c43b0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7ffae00c8420 tx=0x7ffac802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06e63e75d0bce1e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffaddd9b640 1 -- 172.21.15.78:0/3400046455 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80317e0 con 0x7ffae00c4d80 2024-02-19T14:58:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffaddd9b640 1 -- 172.21.15.78:0/3400046455 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac8031940 con 0x7ffae00c4d80 2024-02-19T14:58:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffaddd9b640 1 -- 172.21.15.78:0/3400046455 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8031cd0 con 0x7ffae00c4d80 2024-02-19T14:58:56.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 -- 172.21.15.78:0/3400046455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c4d80 msgr2=0x7ffae00c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 --2- 172.21.15.78:0/3400046455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c4d80 0x7ffae00c43b0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7ffae00c8420 tx=0x7ffac802b880 comp rx=0 tx=0).stop 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 -- 172.21.15.78:0/3400046455 shutdown_connections 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 --2- 172.21.15.78:0/3400046455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae01523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 --2- 172.21.15.78:0/3400046455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c4d80 0x7ffae00c43b0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 --2- 172.21.15.78:0/3400046455 >> v2:172.21.15.150:3300/0 conn(0x7ffae00c78a0 0x7ffae0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 -- 172.21.15.78:0/3400046455 >> 172.21.15.78:0/3400046455 conn(0x7ffae0066e70 msgr2=0x7ffae00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 -- 172.21.15.78:0/3400046455 shutdown_connections 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.173+0000 7ffae4d25640 1 -- 172.21.15.78:0/3400046455 wait complete. 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.174+0000 7ffae4d25640 1 Processor -- start 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.174+0000 7ffae4d25640 1 -- start start 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffae4d25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae00c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffae4d25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae00c4d80 0x7ffae01ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffae4d25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 0x7ffae01edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffae4d25640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae0149920 con 0x7ffae00c78a0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffae4d25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae01492f0 con 0x7ffae00c4d80 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffae4d25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae0155c80 con 0x7ffae00c48f0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 0x7ffae01edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 0x7ffae01edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38040/0 (socket says 172.21.15.78:38040) 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 -- 172.21.15.78:0/1517811181 learned_addr learned my addr 172.21.15.78:0/1517811181 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffad7fff640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae00c4d80 0x7ffae01ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffade59c640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae00c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 -- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 msgr2=0x7ffae00c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae00c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 -- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae00c4d80 msgr2=0x7ffae01ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae00c4d80 0x7ffae01ed9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.175+0000 7ffaded9d640 1 -- 172.21.15.78:0/1517811181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80094d0 con 0x7ffae00c78a0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.176+0000 7ffaded9d640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 0x7ffae01edef0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ffad800cbb0 tx=0x7ffad800a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.176+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad800cea0 con 0x7ffae00c78a0 2024-02-19T14:58:56.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.176+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad800abd0 con 0x7ffae00c78a0 2024-02-19T14:58:56.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.176+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae014aac0 con 0x7ffae00c78a0 2024-02-19T14:58:56.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.176+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8003990 con 0x7ffae00c78a0 2024-02-19T14:58:56.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.176+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae0149540 con 0x7ffae00c78a0 2024-02-19T14:58:56.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.177+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae0149a60 con 0x7ffae00c78a0 2024-02-19T14:58:56.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.177+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffad8011410 con 0x7ffae00c78a0 2024-02-19T14:58:56.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.178+0000 7ffad6ffd640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffac006b840 0x7ffac006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.178+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7ffad808a0f0 con 0x7ffae00c78a0 2024-02-19T14:58:56.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.178+0000 7ffade59c640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffac006b840 0x7ffac006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.178+0000 7ffade59c640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffac006b840 0x7ffac006dd00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ffad0008850 tx=0x7ffad0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.182+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffad80549c0 con 0x7ffae00c78a0 2024-02-19T14:58:56.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.463+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffae014f970 con 0x7ffae00c78a0 2024-02-19T14:58:56.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.463+0000 7ffad6ffd640 1 -- 172.21.15.78:0/1517811181 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4954 (secure 0 0 0) 0x7ffad8058080 con 0x7ffae00c78a0 2024-02-19T14:58:56.466 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:56.467 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.262364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:active","state_seq":2,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:56.467 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 159 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffac006b840 msgr2=0x7ffac006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffac006b840 0x7ffac006dd00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ffad0008850 tx=0x7ffad0008040 comp rx=0 tx=0).stop 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 msgr2=0x7ffae01edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 0x7ffae01edef0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ffad800cbb0 tx=0x7ffad800a4c0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 shutdown_connections 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffac006b840 0x7ffac006dd00 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffae00c78a0 0x7ffae01edef0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae00c4d80 0x7ffae01ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 --2- 172.21.15.78:0/1517811181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffae00c48f0 0x7ffae00c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 >> 172.21.15.78:0/1517811181 conn(0x7ffae0066e70 msgr2=0x7ffae01534d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:56.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 shutdown_connections 2024-02-19T14:58:56.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.466+0000 7ffae4d25640 1 -- 172.21.15.78:0/1517811181 wait complete. 2024-02-19T14:58:56.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:58:56.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f75a490d640 1 Processor -- start 2024-02-19T14:58:56.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f75a490d640 1 -- start start 2024-02-19T14:58:56.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f75a490d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75a006ff80 0x7f75a0154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f75a490d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a014f730 con 0x7f75a00c6d60 2024-02-19T14:58:56.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f75a490d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a00c16d0 con 0x7f75a006e830 2024-02-19T14:58:56.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f75a490d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a00c1940 con 0x7f75a006ff80 2024-02-19T14:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f759ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75a006ff80 0x7f75a0154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f759e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f75a00c6d60 0x7f75a00c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46842/0 (socket says 172.21.15.78:46842) 2024-02-19T14:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.578+0000 7f759e59c640 1 -- 172.21.15.78:0/2447032357 learned_addr learned my addr 172.21.15.78:0/2447032357 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.579+0000 7f759ed9d640 1 -- 172.21.15.78:0/2447032357 >> v1:172.21.15.150:6789/0 conn(0x7f75a006e830 legacy=0x7f75a00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.579+0000 7f759ed9d640 1 -- 172.21.15.78:0/2447032357 >> v1:172.21.15.78:6789/0 conn(0x7f75a00c6d60 legacy=0x7f75a00c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.579+0000 7f759ed9d640 1 -- 172.21.15.78:0/2447032357 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0072f70 con 0x7f75a006ff80 2024-02-19T14:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.579+0000 7f759ed9d640 1 --2- 172.21.15.78:0/2447032357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a006ff80 0x7f75a0154f70 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f7598009d90 tx=0x7f759802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e358cd93a8e1a279 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f759dd9b640 1 -- 172.21.15.78:0/2447032357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759802cd00 con 0x7f75a006ff80 2024-02-19T14:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f759dd9b640 1 -- 172.21.15.78:0/2447032357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759802ce60 con 0x7f75a006ff80 2024-02-19T14:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f759dd9b640 1 -- 172.21.15.78:0/2447032357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598034660 con 0x7f75a006ff80 2024-02-19T14:58:56.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 -- 172.21.15.78:0/2447032357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a006ff80 msgr2=0x7f75a0154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 --2- 172.21.15.78:0/2447032357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a006ff80 0x7f75a0154f70 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f7598009d90 tx=0x7f759802bf50 comp rx=0 tx=0).stop 2024-02-19T14:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 -- 172.21.15.78:0/2447032357 shutdown_connections 2024-02-19T14:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 --2- 172.21.15.78:0/2447032357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a006ff80 0x7f75a0154f70 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 -- 172.21.15.78:0/2447032357 >> 172.21.15.78:0/2447032357 conn(0x7f75a006c170 msgr2=0x7f75a006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 -- 172.21.15.78:0/2447032357 shutdown_connections 2024-02-19T14:58:56.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.580+0000 7f75a490d640 1 -- 172.21.15.78:0/2447032357 wait complete. 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.581+0000 7f75a490d640 1 Processor -- start 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 -- start start 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75a006e830 0x7f75a0150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75a006ff80 0x7f75a014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 0x7f75a01ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a01eef10 con 0x7f75a00c6d60 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0147eb0 con 0x7f75a006e830 2024-02-19T14:58:56.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f75a490d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0148190 con 0x7f75a006ff80 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f759ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 0x7f75a01ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f759ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 0x7f75a01ee1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38060/0 (socket says 172.21.15.78:38060) 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.582+0000 7f759ed9d640 1 -- 172.21.15.78:0/426440312 learned_addr learned my addr 172.21.15.78:0/426440312 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759e59c640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75a006e830 0x7f75a0150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f7595d9b640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75a006ff80 0x7f75a014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759ed9d640 1 -- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75a006ff80 msgr2=0x7f75a014a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759ed9d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75a006ff80 0x7f75a014a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759ed9d640 1 -- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75a006e830 msgr2=0x7f75a0150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759ed9d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75a006e830 0x7f75a0150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759ed9d640 1 -- 172.21.15.78:0/426440312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598009a70 con 0x7f75a00c6d60 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759e59c640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75a006e830 0x7f75a0150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f759ed9d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 0x7f75a01ee1f0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f7598009d90 tx=0x7f7598004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759803d600 con 0x7f75a00c6d60 2024-02-19T14:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759803d780 con 0x7f75a00c6d60 2024-02-19T14:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759803db30 con 0x7f75a00c6d60 2024-02-19T14:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a01eec30 con 0x7f75a00c6d60 2024-02-19T14:58:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.583+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a014a820 con 0x7f75a00c6d60 2024-02-19T14:58:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.584+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0148350 con 0x7f75a00c6d60 2024-02-19T14:58:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.584+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f759803b040 con 0x7f75a00c6d60 2024-02-19T14:58:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.585+0000 7f75977fe640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f758006b810 0x7f758006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.585+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f75980acb70 con 0x7f75a00c6d60 2024-02-19T14:58:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.585+0000 7f759e59c640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f758006b810 0x7f758006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.585+0000 7f759e59c640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f758006b810 0x7f758006dcd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f7590007fd0 tx=0x7f7590009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.589+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7598077560 con 0x7f75a00c6d60 2024-02-19T14:58:56.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.827+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a0074990 con 0x7f75a00c6d60 2024-02-19T14:58:56.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.827+0000 7f75977fe640 1 -- 172.21.15.78:0/426440312 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4954 (secure 0 0 0) 0x7f7598039090 con 0x7f75a00c6d60 2024-02-19T14:58:56.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:56.830 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:58:27.307125+0000","modified":"2024-02-19T14:58:50.262364+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9609},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9609":{"gid":9609,"name":"c","rank":0,"incarnation":158,"state":"up:active","state_seq":2,"addr":"172.21.15.78:6836/3613671589","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3613671589},{"type":"v1","addr":"172.21.15.78:6836","nonce":3613671589}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-19T14:58:56.831 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 159 2024-02-19T14:58:56.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f758006b810 msgr2=0x7f758006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f758006b810 0x7f758006dcd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f7590007fd0 tx=0x7f7590009040 comp rx=0 tx=0).stop 2024-02-19T14:58:56.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 msgr2=0x7f75a01ee1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 0x7f75a01ee1f0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f7598009d90 tx=0x7f7598004a40 comp rx=0 tx=0).stop 2024-02-19T14:58:56.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 shutdown_connections 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f758006b810 0x7f758006dcd0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75a00c6d60 0x7f75a01ee1f0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75a006ff80 0x7f75a014a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 --2- 172.21.15.78:0/426440312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75a006e830 0x7f75a0150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 >> 172.21.15.78:0/426440312 conn(0x7f75a006c170 msgr2=0x7f75a014d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 shutdown_connections 2024-02-19T14:58:56.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.829+0000 7f75a490d640 1 -- 172.21.15.78:0/426440312 wait complete. 2024-02-19T14:58:56.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:58:56.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.942+0000 7f9d8eb3e640 1 Processor -- start 2024-02-19T14:58:56.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.942+0000 7f9d8eb3e640 1 -- start start 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8eb3e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9d88149790 0x7f9d8806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8eb3e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8806f420 con 0x7f9d88072800 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8eb3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d880ab760 con 0x7f9d880723d0 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8eb3e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d88149b70 con 0x7f9d88149790 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8c8da640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9d88149790 0x7f9d8806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8d0db640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9d88072800 0x7f9d88153340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46844/0 (socket says 172.21.15.78:46844) 2024-02-19T14:58:56.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d8d0db640 1 -- 172.21.15.78:0/733574396 learned_addr learned my addr 172.21.15.78:0/733574396 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.943+0000 7f9d877fe640 1 -- 172.21.15.78:0/733574396 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044174487 0 0) 0x7f9d8806f420 con 0x7f9d88072800 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.944+0000 7f9d877fe640 1 -- 172.21.15.78:0/733574396 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d78003520 con 0x7f9d88072800 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.944+0000 7f9d8c8da640 1 -- 172.21.15.78:0/733574396 >> v1:172.21.15.150:6789/0 conn(0x7f9d880723d0 legacy=0x7f9d88152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.944+0000 7f9d8c8da640 1 -- 172.21.15.78:0/733574396 >> v1:172.21.15.78:6789/0 conn(0x7f9d88072800 legacy=0x7f9d88153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.944+0000 7f9d8c8da640 1 -- 172.21.15.78:0/733574396 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d880aba40 con 0x7f9d88149790 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.944+0000 7f9d8c8da640 1 --2- 172.21.15.78:0/733574396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88149790 0x7f9d8806ed40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0097a0 tx=0x7f9d7c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa90b54c7ab805a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.944+0000 7f9d877fe640 1 -- 172.21.15.78:0/733574396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c02f050 con 0x7f9d88149790 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d877fe640 1 -- 172.21.15.78:0/733574396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c031700 con 0x7f9d88149790 2024-02-19T14:58:56.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d877fe640 1 -- 172.21.15.78:0/733574396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c031a70 con 0x7f9d88149790 2024-02-19T14:58:56.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/733574396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88149790 msgr2=0x7f9d8806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/733574396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88149790 0x7f9d8806ed40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c0097a0 tx=0x7f9d7c02b7a0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/733574396 shutdown_connections 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/733574396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88149790 0x7f9d8806ed40 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/733574396 >> 172.21.15.78:0/733574396 conn(0x7f9d880718e0 msgr2=0x7f9d88071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/733574396 shutdown_connections 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.945+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/733574396 wait complete. 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.947+0000 7f9d8eb3e640 1 Processor -- start 2024-02-19T14:58:56.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.947+0000 7f9d8eb3e640 1 -- start start 2024-02-19T14:58:56.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8eb3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880723d0 0x7f9d8814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8eb3e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 0x7f9d8814d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8eb3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88149790 0x7f9d8814bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8eb3e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8814ba50 con 0x7f9d88072800 2024-02-19T14:58:56.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8eb3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d880c5ef0 con 0x7f9d880723d0 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8eb3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8814ae40 con 0x7f9d88149790 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 0x7f9d8814d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 0x7f9d8814d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38076/0 (socket says 172.21.15.78:38076) 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d87fff640 1 -- 172.21.15.78:0/44533884 learned_addr learned my addr 172.21.15.78:0/44533884 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d87fff640 1 -- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88149790 msgr2=0x7f9d8814bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d87fff640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88149790 0x7f9d8814bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d8c8da640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880723d0 0x7f9d8814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.948+0000 7f9d87fff640 1 -- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880723d0 msgr2=0x7f9d8814ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d87fff640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880723d0 0x7f9d8814ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d87fff640 1 -- 172.21.15.78:0/44533884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c009400 con 0x7f9d88072800 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d8c8da640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880723d0 0x7f9d8814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d87fff640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 0x7f9d8814d8c0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400b3f0 tx=0x7f9d7400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74011830 con 0x7f9d88072800 2024-02-19T14:58:56.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74011990 con 0x7f9d88072800 2024-02-19T14:58:56.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74011d40 con 0x7f9d88072800 2024-02-19T14:58:56.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8814b120 con 0x7f9d88072800 2024-02-19T14:58:56.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.949+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8814b6e0 con 0x7f9d88072800 2024-02-19T14:58:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.950+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8814bbc0 con 0x7f9d88072800 2024-02-19T14:58:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.950+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9d7401a400 con 0x7f9d88072800 2024-02-19T14:58:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.951+0000 7f9d857fa640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d6c06b810 0x7f9d6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.951+0000 7f9d8c8da640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d6c06b810 0x7f9d6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.951+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f9d7408a190 con 0x7f9d88072800 2024-02-19T14:58:56.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.951+0000 7f9d8c8da640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d6c06b810 0x7f9d6c06dcd0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c035000 tx=0x7f9d7c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:56.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:56.955+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9d74054a60 con 0x7f9d88072800 2024-02-19T14:58:57.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:58:57.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:58:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.188+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d88074990 con 0x7f9d88072800 2024-02-19T14:58:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.189+0000 7f9d857fa640 1 -- 172.21.15.78:0/44533884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+17085 (secure 0 0 0) 0x7f9d74058120 con 0x7f9d88072800 2024-02-19T14:58:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:58:57.192 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":176,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:58:42.205276+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:58:25.520675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-19T14:58:26.281810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"28.7","mappings":[{"from":6,"to":4}]},{"pgid":"28.2b","mappings":[{"from":6,"to":4}]},{"pgid":"28.38","mappings":[{"from":6,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:58:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d6c06b810 msgr2=0x7f9d6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d6c06b810 0x7f9d6c06dcd0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c035000 tx=0x7f9d7c008030 comp rx=0 tx=0).stop 2024-02-19T14:58:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 msgr2=0x7f9d8814d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 0x7f9d8814d8c0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400b3f0 tx=0x7f9d7400b8b0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 shutdown_connections 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9d6c06b810 0x7f9d6c06dcd0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d88072800 0x7f9d8814d8c0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88149790 0x7f9d8814bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 --2- 172.21.15.78:0/44533884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880723d0 0x7f9d8814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 >> 172.21.15.78:0/44533884 conn(0x7f9d880718e0 msgr2=0x7f9d88151210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 shutdown_connections 2024-02-19T14:58:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.191+0000 7f9d8eb3e640 1 -- 172.21.15.78:0/44533884 wait complete. 2024-02-19T14:58:57.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:58:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 Processor -- start 2024-02-19T14:58:57.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 -- start start 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f74140c4980 0x7f74140c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f74140705b0 0x7f74140709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74140ae670 con 0x7f74140add70 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74140716b0 con 0x7f74140705b0 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.319+0000 7f741a319640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7414153780 con 0x7f74140c4980 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f74140705b0 0x7f74140709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f74140705b0 0x7f74140709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38086/0 (socket says 172.21.15.78:38086) 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 -- 172.21.15.78:0/1243446344 learned_addr learned my addr 172.21.15.78:0/1243446344 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74137fe640 1 --2- 172.21.15.78:0/1243446344 >> v2:172.21.15.150:3300/0 conn(0x7f74140c4980 0x7f74140c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 -- 172.21.15.78:0/1243446344 >> v2:172.21.15.150:3300/0 conn(0x7f74140c4980 msgr2=0x7f74140c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 --2- 172.21.15.78:0/1243446344 >> v2:172.21.15.150:3300/0 conn(0x7f74140c4980 0x7f74140c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 -- 172.21.15.78:0/1243446344 >> v1:172.21.15.150:6789/0 conn(0x7f74140add70 legacy=0x7f741406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.320+0000 7f74188b6640 1 -- 172.21.15.78:0/1243446344 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414153a60 con 0x7f74140705b0 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.321+0000 7f74188b6640 1 --2- 172.21.15.78:0/1243446344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140705b0 0x7f74140709e0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7414073050 tx=0x7f740c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3b3ebffa3dc8b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.321+0000 7f7412ffd640 1 -- 172.21.15.78:0/1243446344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c02ca80 con 0x7f74140705b0 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.321+0000 7f7412ffd640 1 -- 172.21.15.78:0/1243446344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740c02cbe0 con 0x7f74140705b0 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.321+0000 7f7412ffd640 1 -- 172.21.15.78:0/1243446344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c0343f0 con 0x7f74140705b0 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.321+0000 7f741a319640 1 -- 172.21.15.78:0/1243446344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140705b0 msgr2=0x7f74140709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.321+0000 7f741a319640 1 --2- 172.21.15.78:0/1243446344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140705b0 0x7f74140709e0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7414073050 tx=0x7f740c02bee0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.322+0000 7f741a319640 1 -- 172.21.15.78:0/1243446344 shutdown_connections 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.322+0000 7f741a319640 1 --2- 172.21.15.78:0/1243446344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140705b0 0x7f74140709e0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.322+0000 7f741a319640 1 --2- 172.21.15.78:0/1243446344 >> v2:172.21.15.150:3300/0 conn(0x7f74140c4980 0x7f74140c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.322+0000 7f741a319640 1 -- 172.21.15.78:0/1243446344 >> 172.21.15.78:0/1243446344 conn(0x7f74140c85e0 msgr2=0x7f74140c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.322+0000 7f741a319640 1 -- 172.21.15.78:0/1243446344 shutdown_connections 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.322+0000 7f741a319640 1 -- 172.21.15.78:0/1243446344 wait complete. 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 Processor -- start 2024-02-19T14:58:57.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 -- start start 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74140705b0 0x7f74140c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 0x7f741414d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74140c4980 0x7f741414db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74141e9360 con 0x7f74140add70 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74141e8e00 con 0x7f74140705b0 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.323+0000 7f741a319640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74140c2220 con 0x7f74140c4980 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 0x7f741414d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 0x7f741414d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38092/0 (socket says 172.21.15.78:38092) 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 -- 172.21.15.78:0/1887950538 learned_addr learned my addr 172.21.15.78:0/1887950538 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74188b6640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74140c4980 0x7f741414db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f7413fff640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74140705b0 0x7f74140c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 -- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74140c4980 msgr2=0x7f741414db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74140c4980 0x7f741414db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 -- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74140705b0 msgr2=0x7f74140c1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74140705b0 0x7f74140c1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 -- 172.21.15.78:0/1887950538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c009b30 con 0x7f74140add70 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74188b6640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74140c4980 0x7f741414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f74137fe640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 0x7f741414d620 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f740800d800 tx=0x7f740800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740800b6c0 con 0x7f74140add70 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740800b820 con 0x7f74140add70 2024-02-19T14:58:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74141e9080 con 0x7f74140add70 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740800bbb0 con 0x7f74140add70 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.324+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7414156b90 con 0x7f74140add70 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.325+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74140676a0 con 0x7f74140add70 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.326+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f740800bd10 con 0x7f74140add70 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.326+0000 7f7410ff9640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740006b810 0x7f740006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.326+0000 7f7413fff640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740006b810 0x7f740006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.326+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f740808a880 con 0x7f74140add70 2024-02-19T14:58:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.327+0000 7f7413fff640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740006b810 0x7f740006dcd0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f7404000f60 tx=0x7f7404009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:57.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.330+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7408055150 con 0x7f74140add70 2024-02-19T14:58:57.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.570+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7414074990 con 0x7f74140add70 2024-02-19T14:58:57.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:58:57.570+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:58:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.572+0000 7f7410ff9640 1 -- 172.21.15.78:0/1887950538 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v160) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7408058810 con 0x7f74140add70 2024-02-19T14:58:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740006b810 msgr2=0x7f740006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740006b810 0x7f740006dcd0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f7404000f60 tx=0x7f7404009040 comp rx=0 tx=0).stop 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 msgr2=0x7f741414d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 0x7f741414d620 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f740800d800 tx=0x7f740800dcc0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 shutdown_connections 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740006b810 0x7f740006dcd0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74140add70 0x7f741414d620 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74140c4980 0x7f741414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 --2- 172.21.15.78:0/1887950538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74140705b0 0x7f74140c1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 >> 172.21.15.78:0/1887950538 conn(0x7f74140c85e0 msgr2=0x7f74140a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:57.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 shutdown_connections 2024-02-19T14:58:57.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.575+0000 7f741a319640 1 -- 172.21.15.78:0/1887950538 wait complete. 2024-02-19T14:58:57.579 INFO:tasks.ceph.mds.c.smithi078.stderr: -93> 2024-02-19T14:58:55.239+0000 7fe05f43b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:58:57.589 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:58:57.602 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:58:57.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 Processor -- start 2024-02-19T14:58:57.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 -- start start 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55780c6a50 0x7f5578155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55780c5920 0x7f5578155e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780717b0 con 0x7f55780c2f70 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780c7560 con 0x7f55780c5920 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.689+0000 7f558028f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5578071a00 con 0x7f55780c6a50 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557e82c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55780c5920 0x7f5578155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557d82a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f55780c2f70 0x7f557806c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52312/0 (socket says 172.21.15.78:52312) 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557d82a640 1 -- 172.21.15.78:0/3011225563 learned_addr learned my addr 172.21.15.78:0/3011225563 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557e82c640 1 -- 172.21.15.78:0/3011225563 >> v2:172.21.15.150:3300/0 conn(0x7f55780c6a50 msgr2=0x7f5578155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557e82c640 1 --2- 172.21.15.78:0/3011225563 >> v2:172.21.15.150:3300/0 conn(0x7f55780c6a50 0x7f5578155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557e82c640 1 -- 172.21.15.78:0/3011225563 >> v1:172.21.15.78:6789/0 conn(0x7f55780c2f70 legacy=0x7f557806c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557e82c640 1 -- 172.21.15.78:0/3011225563 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780c8030 con 0x7f55780c5920 2024-02-19T14:58:57.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.690+0000 7f557d029640 1 -- 172.21.15.78:0/3011225563 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117774543 0 0) 0x7f55780717b0 con 0x7f55780c2f70 2024-02-19T14:58:57.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f557e82c640 1 --2- 172.21.15.78:0/3011225563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f5578155e80 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f55780c2a20 tx=0x7f557002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13576e55d4bc3c5a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:57.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f557d029640 1 -- 172.21.15.78:0/3011225563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557002cbf0 con 0x7f55780c5920 2024-02-19T14:58:57.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f557d029640 1 -- 172.21.15.78:0/3011225563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557002cd50 con 0x7f55780c5920 2024-02-19T14:58:57.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f557d029640 1 -- 172.21.15.78:0/3011225563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570034660 con 0x7f55780c5920 2024-02-19T14:58:57.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 -- 172.21.15.78:0/3011225563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 msgr2=0x7f5578155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 --2- 172.21.15.78:0/3011225563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f5578155e80 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f55780c2a20 tx=0x7f557002be40 comp rx=0 tx=0).stop 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 -- 172.21.15.78:0/3011225563 shutdown_connections 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 --2- 172.21.15.78:0/3011225563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f5578155e80 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 --2- 172.21.15.78:0/3011225563 >> v2:172.21.15.150:3300/0 conn(0x7f55780c6a50 0x7f5578155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 -- 172.21.15.78:0/3011225563 >> 172.21.15.78:0/3011225563 conn(0x7f557806b700 msgr2=0x7f55780aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 -- 172.21.15.78:0/3011225563 shutdown_connections 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.691+0000 7f558028f640 1 -- 172.21.15.78:0/3011225563 wait complete. 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.692+0000 7f558028f640 1 Processor -- start 2024-02-19T14:58:57.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.692+0000 7f558028f640 1 -- start start 2024-02-19T14:58:57.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f558028f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780c2f70 0x7f55780a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f558028f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f557809a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f558028f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55780c6a50 0x7f55781ea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f558028f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557809a620 con 0x7f55780c5920 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f558028f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55780a8c50 con 0x7f55780c2f70 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f558028f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5578071180 con 0x7f55780c6a50 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f557809a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f557809a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57192/0 (socket says 172.21.15.78:57192) 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 -- 172.21.15.78:0/2918171403 learned_addr learned my addr 172.21.15.78:0/2918171403 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557e02b640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780c2f70 0x7f55780a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557e82c640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55780c6a50 0x7f55781ea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 -- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55780c6a50 msgr2=0x7f55781ea5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55780c6a50 0x7f55781ea5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 -- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780c2f70 msgr2=0x7f55780a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780c2f70 0x7f55780a96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.693+0000 7f557d82a640 1 -- 172.21.15.78:0/2918171403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5570009a70 con 0x7f55780c5920 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.694+0000 7f557d82a640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f557809a0e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f557400c470 tx=0x7f557400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.694+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740074e0 con 0x7f55780c5920 2024-02-19T14:58:57.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.694+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5574007640 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.694+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5574004a20 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.694+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55781eae20 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.694+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55780a8ea0 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.695+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5578074990 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.695+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5574004b80 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.695+0000 7f556effd640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f555c06b810 0x7f555c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.696+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f55740898c0 con 0x7f55780c5920 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.696+0000 7f557e02b640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f555c06b810 0x7f555c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:57.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.696+0000 7f557e02b640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f555c06b810 0x7f555c06dcd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f556800b810 tx=0x7f5568008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:57.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.700+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5574054130 con 0x7f55780c5920 2024-02-19T14:58:57.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:57.909+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f55780c65c0 con 0x7f55780c5920 2024-02-19T14:58:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.577+0000 7f556effd640 1 -- 172.21.15.78:0/2918171403 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v161) v1 ==== 104+0+0 (secure 0 0 0) 0x7f55740577f0 con 0x7f55780c5920 2024-02-19T14:58:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f555c06b810 msgr2=0x7f555c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:58.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f555c06b810 0x7f555c06dcd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f556800b810 tx=0x7f5568008040 comp rx=0 tx=0).stop 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 msgr2=0x7f557809a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f557809a0e0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f557400c470 tx=0x7f557400c930 comp rx=0 tx=0).stop 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 shutdown_connections 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f555c06b810 0x7f555c06dcd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55780c5920 0x7f557809a0e0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55780c6a50 0x7f55781ea5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 --2- 172.21.15.78:0/2918171403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780c2f70 0x7f55780a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 >> 172.21.15.78:0/2918171403 conn(0x7f557806b700 msgr2=0x7f5578153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 shutdown_connections 2024-02-19T14:58:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.580+0000 7f558028f640 1 -- 172.21.15.78:0/2918171403 wait complete. 2024-02-19T14:58:58.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:58:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.737+0000 7f64503b6640 1 Processor -- start 2024-02-19T14:58:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.737+0000 7f64503b6640 1 -- start start 2024-02-19T14:58:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f64503b6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f64480c5590 0x7f64480c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f64503b6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64480c2940 0x7f644815c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f64503b6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644806e210 con 0x7f644806e6a0 2024-02-19T14:58:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f64503b6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64480abb60 con 0x7f64480c2940 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f64503b6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644806f0a0 con 0x7f64480c5590 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f644e953640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64480c2940 0x7f644815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f644e953640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64480c2940 0x7f644815c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57202/0 (socket says 172.21.15.78:57202) 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.738+0000 7f644e953640 1 -- 172.21.15.78:0/1311043635 learned_addr learned my addr 172.21.15.78:0/1311043635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.739+0000 7f644e953640 1 -- 172.21.15.78:0/1311043635 >> v2:172.21.15.150:3300/0 conn(0x7f64480c5590 msgr2=0x7f64480c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.739+0000 7f644e953640 1 --2- 172.21.15.78:0/1311043635 >> v2:172.21.15.150:3300/0 conn(0x7f64480c5590 0x7f64480c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.739+0000 7f644e953640 1 -- 172.21.15.78:0/1311043635 >> v1:172.21.15.150:6789/0 conn(0x7f644806e6a0 legacy=0x7f6448151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.739+0000 7f644e953640 1 -- 172.21.15.78:0/1311043635 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64480c26f0 con 0x7f64480c2940 2024-02-19T14:58:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.739+0000 7f644e953640 1 --2- 172.21.15.78:0/1311043635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480c2940 0x7f644815c650 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f64480736f0 tx=0x7f644002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9642b383914e121 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f644d150640 1 -- 172.21.15.78:0/1311043635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644002ea00 con 0x7f64480c2940 2024-02-19T14:58:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f644d150640 1 -- 172.21.15.78:0/1311043635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644002eb60 con 0x7f64480c2940 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f644d150640 1 -- 172.21.15.78:0/1311043635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440009600 con 0x7f64480c2940 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 -- 172.21.15.78:0/1311043635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480c2940 msgr2=0x7f644815c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 --2- 172.21.15.78:0/1311043635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480c2940 0x7f644815c650 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f64480736f0 tx=0x7f644002e060 comp rx=0 tx=0).stop 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 -- 172.21.15.78:0/1311043635 shutdown_connections 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 --2- 172.21.15.78:0/1311043635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64480c2940 0x7f644815c650 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 --2- 172.21.15.78:0/1311043635 >> v2:172.21.15.150:3300/0 conn(0x7f64480c5590 0x7f64480c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 -- 172.21.15.78:0/1311043635 >> 172.21.15.78:0/1311043635 conn(0x7f64480c6fe0 msgr2=0x7f64480a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 -- 172.21.15.78:0/1311043635 shutdown_connections 2024-02-19T14:58:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.740+0000 7f64503b6640 1 -- 172.21.15.78:0/1311043635 wait complete. 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.741+0000 7f64503b6640 1 Processor -- start 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.741+0000 7f64503b6640 1 -- start start 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f64503b6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 0x7f644814e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f64503b6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64480c2940 0x7f64481478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f64503b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64480c5590 0x7f6448147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f64503b6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64480712e0 con 0x7f644806e6a0 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f64503b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448071af0 con 0x7f64480c5590 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f64503b6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6448148360 con 0x7f64480c2940 2024-02-19T14:58:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 0x7f644814e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 0x7f644814e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57210/0 (socket says 172.21.15.78:57210) 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 -- 172.21.15.78:0/935174672 learned_addr learned my addr 172.21.15.78:0/935174672 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644d951640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64480c2940 0x7f64481478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e953640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64480c5590 0x7f6448147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 -- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64480c2940 msgr2=0x7f64481478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64480c2940 0x7f64481478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 -- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64480c5590 msgr2=0x7f6448147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64480c5590 0x7f6448147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.742+0000 7f644e152640 1 -- 172.21.15.78:0/935174672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644000bd40 con 0x7f644806e6a0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.743+0000 7f644e152640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 0x7f644814e360 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f644400b440 tx=0x7f644400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.743+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444011890 con 0x7f644806e6a0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.743+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64440119f0 con 0x7f644806e6a0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.743+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64480bfc30 con 0x7f644806e6a0 2024-02-19T14:58:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.743+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444011da0 con 0x7f644806e6a0 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.743+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644814ff80 con 0x7f644806e6a0 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.744+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64480678b0 con 0x7f644806e6a0 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.744+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f644401a400 con 0x7f644806e6a0 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.744+0000 7f643effd640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f642c06b810 0x7f642c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.744+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f644408a5a0 con 0x7f644806e6a0 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.744+0000 7f644d951640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f642c06b810 0x7f642c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.745+0000 7f644d951640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f642c06b810 0x7f642c06dcd0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f64480a7ca0 tx=0x7f6438009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:58.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.749+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6444054ec0 con 0x7f644806e6a0 2024-02-19T14:58:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:58.920+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f64480a6520 con 0x7f644806e6a0 2024-02-19T14:58:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.578+0000 7f643effd640 1 -- 172.21.15.78:0/935174672 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v178) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6444058580 con 0x7f644806e6a0 2024-02-19T14:58:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:58:59.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f642c06b810 msgr2=0x7f642c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f642c06b810 0x7f642c06dcd0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f64480a7ca0 tx=0x7f6438009040 comp rx=0 tx=0).stop 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 msgr2=0x7f644814e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 0x7f644814e360 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f644400b440 tx=0x7f644400b900 comp rx=0 tx=0).stop 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 shutdown_connections 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f642c06b810 0x7f642c06dcd0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64480c5590 0x7f6448147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64480c2940 0x7f64481478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 --2- 172.21.15.78:0/935174672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f644806e6a0 0x7f644814e360 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 >> 172.21.15.78:0/935174672 conn(0x7f64480c6fe0 msgr2=0x7f64480c0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 shutdown_connections 2024-02-19T14:58:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.581+0000 7f64503b6640 1 -- 172.21.15.78:0/935174672 wait complete. 2024-02-19T14:58:59.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:58:59.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.699+0000 7fee9152c640 1 Processor -- start 2024-02-19T14:58:59.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.699+0000 7fee9152c640 1 -- start start 2024-02-19T14:58:59.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee9152c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee8c0c20a0 0x7fee8c0c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:59.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee9152c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:59.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee9152c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c0c8270 con 0x7fee8c148860 2024-02-19T14:58:59.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee9152c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c0c79d0 con 0x7fee8c0c20a0 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee9152c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c0c7cb0 con 0x7fee8c155870 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee8c0c20a0 0x7fee8c0c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee8c148860 0x7fee8c152680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52314/0 (socket says 172.21.15.78:52314) 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8affd640 1 -- 172.21.15.78:0/3656555407 learned_addr learned my addr 172.21.15.78:0/3656555407 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8a7fc640 1 -- 172.21.15.78:0/3656555407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 msgr2=0x7fee8c155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8b7fe640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee89ffb640 1 -- 172.21.15.78:0/3656555407 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596621944 0 0) 0x7fee8c0c8270 con 0x7fee8c148860 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8a7fc640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8a7fc640 1 -- 172.21.15.78:0/3656555407 >> v1:172.21.15.78:6789/0 conn(0x7fee8c148860 legacy=0x7fee8c152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8a7fc640 1 -- 172.21.15.78:0/3656555407 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c0c7f90 con 0x7fee8c0c20a0 2024-02-19T14:58:59.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.700+0000 7fee8b7fe640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.701+0000 7fee8a7fc640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c0c2500 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fee800097a0 tx=0x7fee8002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f856b65c2865dd1c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.701+0000 7fee89ffb640 1 -- 172.21.15.78:0/3656555407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8002f050 con 0x7fee8c0c20a0 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.701+0000 7fee89ffb640 1 -- 172.21.15.78:0/3656555407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80031700 con 0x7fee8c0c20a0 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.701+0000 7fee89ffb640 1 -- 172.21.15.78:0/3656555407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80031a90 con 0x7fee8c0c20a0 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 -- 172.21.15.78:0/3656555407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 msgr2=0x7fee8c0c2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c0c2500 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fee800097a0 tx=0x7fee8002b7a0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 -- 172.21.15.78:0/3656555407 shutdown_connections 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c0c2500 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 --2- 172.21.15.78:0/3656555407 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 -- 172.21.15.78:0/3656555407 >> 172.21.15.78:0/3656555407 conn(0x7fee8c06bf90 msgr2=0x7fee8c06c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 -- 172.21.15.78:0/3656555407 shutdown_connections 2024-02-19T14:58:59.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.702+0000 7fee9152c640 1 -- 172.21.15.78:0/3656555407 wait complete. 2024-02-19T14:58:59.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.703+0000 7fee9152c640 1 Processor -- start 2024-02-19T14:58:59.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.703+0000 7fee9152c640 1 -- start start 2024-02-19T14:58:59.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c14dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:59.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee8c148860 0x7fee8c13d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:59.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c13f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:59.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c144b50 con 0x7fee8c0c20a0 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c1e90d0 con 0x7fee8c148860 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c1405a0 con 0x7fee8c155870 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c14dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c14dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57226/0 (socket says 172.21.15.78:57226) 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 -- 172.21.15.78:0/2498403713 learned_addr learned my addr 172.21.15.78:0/2498403713 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 -- 172.21.15.78:0/2498403713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 msgr2=0x7fee8c13f9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c13f9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 -- 172.21.15.78:0/2498403713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee8c148860 msgr2=0x7fee8c13d9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee8c148860 0x7fee8c13d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 -- 172.21.15.78:0/2498403713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee80009400 con 0x7fee8c0c20a0 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee8affd640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c14dca0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00c7c0 tx=0x7fee7c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00a640 con 0x7fee8c0c20a0 2024-02-19T14:58:59.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c00a7c0 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c14e4c0 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.704+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00ab70 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.705+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c1e8d60 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.706+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c06eed0 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.706+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fee7c00acd0 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.706+0000 7fee737fe640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee7806b810 0x7fee7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.707+0000 7fee8a7fc640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee7806b810 0x7fee7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.707+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5836+0+0 (secure 0 0 0) 0x7fee7c089a50 con 0x7fee8c0c20a0 2024-02-19T14:58:59.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.707+0000 7fee8a7fc640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee7806b810 0x7fee7806dcd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fee80035000 tx=0x7fee80008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:58:59.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.711+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee7c054480 con 0x7fee8c0c20a0 2024-02-19T14:58:59.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:58:59.893+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fee8c074990 con 0x7fee8c0c20a0 2024-02-19T14:59:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.585+0000 7fee737fe640 1 -- 172.21.15.78:0/2498403713 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v179) v1 ==== 169+0+0 (secure 0 0 0) 0x7fee7c018020 con 0x7fee8c0c20a0 2024-02-19T14:59:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee7806b810 msgr2=0x7fee7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee7806b810 0x7fee7806dcd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fee80035000 tx=0x7fee80008030 comp rx=0 tx=0).stop 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 msgr2=0x7fee8c14dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c14dca0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00c7c0 tx=0x7fee7c00cc80 comp rx=0 tx=0).stop 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 shutdown_connections 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fee7806b810 0x7fee7806dcd0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee8c155870 0x7fee8c13f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee8c148860 0x7fee8c13d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 --2- 172.21.15.78:0/2498403713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8c0c20a0 0x7fee8c14dca0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 >> 172.21.15.78:0/2498403713 conn(0x7fee8c06bf90 msgr2=0x7fee8c145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 shutdown_connections 2024-02-19T14:59:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.588+0000 7fee9152c640 1 -- 172.21.15.78:0/2498403713 wait complete. 2024-02-19T14:59:00.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:59:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.710+0000 7f7016327640 1 Processor -- start 2024-02-19T14:59:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.710+0000 7f7016327640 1 -- start start 2024-02-19T14:59:00.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f7016327640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70100c87a0 0x7f70101559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f7016327640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7010155ef0 con 0x7f70100c7070 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f7016327640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701006dec0 con 0x7f7010152480 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f7016327640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701006d520 con 0x7f70100c87a0 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f70100c7070 0x7f70100c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52326/0 (socket says 172.21.15.78:52326) 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700f7fe640 1 -- 172.21.15.78:0/1292072570 learned_addr learned my addr 172.21.15.78:0/1292072570 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f70148c4640 1 --2- 172.21.15.78:0/1292072570 >> v2:172.21.15.150:3300/0 conn(0x7f70100c87a0 0x7f70101559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274003797 0 0) 0x7f7004001110 con 0x7f70100c7070 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff8003520 con 0x7f70100c7070 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2221553658 0 0) 0x7f6ff8003520 con 0x7f70100c7070 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 >> v2:172.21.15.150:3300/0 conn(0x7f70100c87a0 msgr2=0x7f70101559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 --2- 172.21.15.78:0/1292072570 >> v2:172.21.15.150:3300/0 conn(0x7f70100c87a0 0x7f70101559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 >> v1:172.21.15.150:6789/0 conn(0x7f7010152480 legacy=0x7f701006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100c75c0 con 0x7f70100c7070 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f7004001800 con 0x7f70100c7070 2024-02-19T14:59:00.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 >> v1:172.21.15.78:6789/0 conn(0x7f70100c7070 legacy=0x7f70100c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff8003970 0x7f6ff8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff8004370 0x7f6ff8004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f6ff800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701006d520 con 0x7f6ff8005040 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff800bfa0 con 0x7f6ff8003970 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.711+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff800c160 con 0x7f6ff8004370 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f6ff800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f700ffff640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff8003970 0x7f6ff8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f700f7fe640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff8004370 0x7f6ff8004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 -- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff8004370 msgr2=0x7f6ff8004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ff8004370 0x7f6ff8004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 -- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff8003970 msgr2=0x7f6ff8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff8003970 0x7f6ff8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 -- 172.21.15.78:0/1292072570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff8003520 con 0x7f6ff8005040 2024-02-19T14:59:00.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f700ffff640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff8003970 0x7f6ff8003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.712+0000 7f70148c4640 1 -- 172.21.15.78:0/1292072570 reap_dead start 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f70148c4640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f6ff800b980 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f7008009fc0 tx=0x7f6ff80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b67d6a9eee79a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700802cc60 con 0x7f6ff8005040 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700800f040 con 0x7f6ff8005040 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f700effd640 1 -- 172.21.15.78:0/1292072570 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008003990 con 0x7f6ff8005040 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 -- 172.21.15.78:0/1292072570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 msgr2=0x7f6ff800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f6ff800b980 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f7008009fc0 tx=0x7f6ff80057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 -- 172.21.15.78:0/1292072570 shutdown_connections 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 --2- 172.21.15.78:0/1292072570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f6ff800b980 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 -- 172.21.15.78:0/1292072570 >> 172.21.15.78:0/1292072570 conn(0x7f70100ca8a0 msgr2=0x7f70100cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 -- 172.21.15.78:0/1292072570 shutdown_connections 2024-02-19T14:59:00.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.713+0000 7f7016327640 1 -- 172.21.15.78:0/1292072570 wait complete. 2024-02-19T14:59:00.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.714+0000 7f7016327640 1 Processor -- start 2024-02-19T14:59:00.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 -- start start 2024-02-19T14:59:00.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f7010158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70100ab950 0x7f701014ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70100abdd0 0x7f701014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101ebbe0 con 0x7f6ff8005040 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100acfd0 con 0x7f70100ab950 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f7016327640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701014dd10 con 0x7f70100abdd0 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f700ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f7010158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f700ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f7010158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57244/0 (socket says 172.21.15.78:57244) 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f700ffff640 1 -- 172.21.15.78:0/2191684647 learned_addr learned my addr 172.21.15.78:0/2191684647 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f700f7fe640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70100ab950 0x7f701014ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.715+0000 7f70148c4640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70100abdd0 0x7f701014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700ffff640 1 -- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70100abdd0 msgr2=0x7f701014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700ffff640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70100abdd0 0x7f701014d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700ffff640 1 -- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70100ab950 msgr2=0x7f701014ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700ffff640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70100ab950 0x7f701014ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700ffff640 1 -- 172.21.15.78:0/2191684647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100aac00 con 0x7f6ff8005040 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f70148c4640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70100abdd0 0x7f701014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700ffff640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f7010158120 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f700000e5d0 tx=0x7f700000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000052e0 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000005440 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70100aaee0 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000003810 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.716+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70100c94f0 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.717+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100aaa50 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.717+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7000003970 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.718+0000 7f700cff9640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ffc06b810 0x7f6ffc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.718+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f7000089e00 con 0x7f6ff8005040 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.718+0000 7f700f7fe640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ffc06b810 0x7f6ffc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:00.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.718+0000 7f700f7fe640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ffc06b810 0x7f6ffc06dcd0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7004001050 tx=0x7f7004009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:00.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.722+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7000054a20 con 0x7f6ff8005040 2024-02-19T14:59:00.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.896+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7010074990 con 0x7f6ff8005040 2024-02-19T14:59:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.901+0000 7f700cff9640 1 -- 172.21.15.78:0/2191684647 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f70000580e0 con 0x7f6ff8005040 2024-02-19T14:59:00.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.903+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ffc06b810 msgr2=0x7f6ffc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.903+0000 7f7016327640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ffc06b810 0x7f6ffc06dcd0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7004001050 tx=0x7f7004009040 comp rx=0 tx=0).stop 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.903+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 msgr2=0x7f7010158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.903+0000 7f7016327640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f7010158120 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f700000e5d0 tx=0x7f700000ea90 comp rx=0 tx=0).stop 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 shutdown_connections 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6ffc06b810 0x7f6ffc06dcd0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70100abdd0 0x7f701014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70100ab950 0x7f701014ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 --2- 172.21.15.78:0/2191684647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ff8005040 0x7f7010158120 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 >> 172.21.15.78:0/2191684647 conn(0x7f70100ca8a0 msgr2=0x7f70100c03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 shutdown_connections 2024-02-19T14:59:00.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:00.904+0000 7f7016327640 1 -- 172.21.15.78:0/2191684647 wait complete. 2024-02-19T14:59:00.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.015+0000 7f2302e07640 1 Processor -- start 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.015+0000 7f2302e07640 1 -- start start 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2302e07640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f22fc06d550 0x7f22fc0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2302e07640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0c6af0 con 0x7f22fc155f90 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2302e07640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0c7940 con 0x7f22fc1588a0 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2302e07640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc0c7270 con 0x7f22fc06d550 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2300ba3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f22fc155f90 0x7f22fc0c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52332/0 (socket says 172.21.15.78:52332) 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2300ba3640 1 -- 172.21.15.78:0/2099878930 learned_addr learned my addr 172.21.15.78:0/2099878930 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440310525 0 0) 0x7f22fc0c6af0 con 0x7f22fc155f90 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec003520 con 0x7f22fc155f90 2024-02-19T14:59:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fbfff640 1 --2- 172.21.15.78:0/2099878930 >> v2:172.21.15.150:3300/0 conn(0x7f22fc06d550 0x7f22fc0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2881729157 0 0) 0x7f22ec003520 con 0x7f22fc155f90 2024-02-19T14:59:01.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 >> v2:172.21.15.150:3300/0 conn(0x7f22fc06d550 msgr2=0x7f22fc0c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:01.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 --2- 172.21.15.78:0/2099878930 >> v2:172.21.15.150:3300/0 conn(0x7f22fc06d550 0x7f22fc0c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 >> v1:172.21.15.150:6789/0 conn(0x7f22fc1588a0 legacy=0x7f22fc158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:01.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0c7c20 con 0x7f22fc155f90 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f22f0001670 con 0x7f22fc155f90 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 >> v1:172.21.15.78:6789/0 conn(0x7f22fc155f90 legacy=0x7f22fc0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22ec003970 0x7f22ec003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22ec0047c0 0x7f22ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec0052a0 0x7f22ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c7270 con 0x7f22ec003970 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec00c150 con 0x7f22ec0052a0 2024-02-19T14:59:01.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec00c310 con 0x7f22ec0047c0 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f22fbfff640 1 --2- 172.21.15.78:0/2099878930 >> v2:172.21.15.150:3300/0 conn(0x7f22fc06d550 0x7f22fc0c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22ec003970 0x7f22ec003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.016+0000 7f23013a4640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec0052a0 0x7f22ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f2300ba3640 1 -- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22ec0047c0 msgr2=0x7f22ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22ec0047c0 0x7f22ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f2300ba3640 1 -- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec0052a0 msgr2=0x7f22ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec0052a0 0x7f22ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f2300ba3640 1 -- 172.21.15.78:0/2099878930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec003520 con 0x7f22ec003970 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f23013a4640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec0052a0 0x7f22ec00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f23013a4640 1 -- 172.21.15.78:0/2099878930 reap_dead start 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22ec003970 0x7f22ec003d70 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f22f0001eb0 tx=0x7f22ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe401ba64cb51c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0005ad0 con 0x7f22ec003970 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f0005c50 con 0x7f22ec003970 2024-02-19T14:59:01.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.017+0000 7f22fb7fe640 1 -- 172.21.15.78:0/2099878930 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f002c8b0 con 0x7f22ec003970 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 -- 172.21.15.78:0/2099878930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22ec003970 msgr2=0x7f22ec003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22ec003970 0x7f22ec003d70 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f22f0001eb0 tx=0x7f22ec0057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 -- 172.21.15.78:0/2099878930 shutdown_connections 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 --2- 172.21.15.78:0/2099878930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22ec003970 0x7f22ec003d70 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 -- 172.21.15.78:0/2099878930 >> 172.21.15.78:0/2099878930 conn(0x7f22fc06b700 msgr2=0x7f22fc0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 -- 172.21.15.78:0/2099878930 shutdown_connections 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.018+0000 7f2302e07640 1 -- 172.21.15.78:0/2099878930 wait complete. 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.019+0000 7f2302e07640 1 Processor -- start 2024-02-19T14:59:01.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.019+0000 7f2302e07640 1 -- start start 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2302e07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec003970 0x7f22fc14f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2302e07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22fc14f030 0x7f22fc1eb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2302e07640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 0x7f22fc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2302e07640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc1ebc90 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2302e07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc06f6f0 con 0x7f22ec003970 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2302e07640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc071160 con 0x7f22fc14f030 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 0x7f22fc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 0x7f22fc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57264/0 (socket says 172.21.15.78:57264) 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 -- 172.21.15.78:0/2017732029 learned_addr learned my addr 172.21.15.78:0/2017732029 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec003970 0x7f22fc14f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f22fbfff640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22fc14f030 0x7f22fc1eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 -- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22fc14f030 msgr2=0x7f22fc1eb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22fc14f030 0x7f22fc1eb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 -- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec003970 msgr2=0x7f22fc14f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec003970 0x7f22fc14f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 -- 172.21.15.78:0/2017732029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0708c0 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec003970 0x7f22fc14f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f22fbfff640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22fc14f030 0x7f22fc1eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.020+0000 7f23013a4640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 0x7f22fc06fb80 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0c6590 tx=0x7f22f400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.021+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4006b50 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.021+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f4006cb0 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.021+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc070ba0 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.021+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004ba0 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.021+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc06f380 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.022+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc074990 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.022+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f22f4004d00 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.022+0000 7f22f97fa640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22e006b810 0x7f22e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.023+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f22f4089660 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.023+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22e006b810 0x7f22e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.023+0000 7f2300ba3640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22e006b810 0x7f22e006dcd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.027+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f22f4057d20 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.263+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f22fc0750b0 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.587+0000 7f22f97fa640 1 -- 172.21.15.78:0/2017732029 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v415) v1 ==== 133+0+0 (secure 0 0 0) 0x7f22f40578b0 con 0x7f22fc0ac6b0 2024-02-19T14:59:01.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22e006b810 msgr2=0x7f22e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22e006b810 0x7f22e006dcd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f0009030 comp rx=0 tx=0).stop 2024-02-19T14:59:01.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 msgr2=0x7f22fc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 0x7f22fc06fb80 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0c6590 tx=0x7f22f400e960 comp rx=0 tx=0).stop 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 shutdown_connections 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22e006b810 0x7f22e006dcd0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc0ac6b0 0x7f22fc06fb80 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22fc14f030 0x7f22fc1eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 --2- 172.21.15.78:0/2017732029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ec003970 0x7f22fc14f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 >> 172.21.15.78:0/2017732029 conn(0x7f22fc06b700 msgr2=0x7f22fc0aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 shutdown_connections 2024-02-19T14:59:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.589+0000 7f2302e07640 1 -- 172.21.15.78:0/2017732029 wait complete. 2024-02-19T14:59:01.603 INFO:tasks.cephfs_test_runner:That down setting toggles and sets max_mds appropriately. ... ok 2024-02-19T14:59:01.604 INFO:tasks.cephfs_test_runner:Starting test: test_down_grow (tasks.cephfs.test_failover.TestClusterResize) That setting max_mds undoes down. 2024-02-19T14:59:01.604 INFO:tasks.cephfs_test_runner:test_down_grow (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T14:59:01.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:01.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.702+0000 7fec762da640 1 Processor -- start 2024-02-19T14:59:01.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.702+0000 7fec762da640 1 -- start start 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.702+0000 7fec762da640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fec70152480 0x7fec701559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.702+0000 7fec762da640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec700c7070 0x7fec7006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec762da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7006d350 0x7fec700c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec762da640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec70155ef0 con 0x7fec70152480 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec762da640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7006dec0 con 0x7fec700c7070 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec762da640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec700c75c0 con 0x7fec7006d350 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fec70152480 0x7fec701559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fec70152480 0x7fec701559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57266/0 (socket says 172.21.15.78:57266) 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 -- 172.21.15.78:0/2379819990 learned_addr learned my addr 172.21.15.78:0/2379819990 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:01.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 -- 172.21.15.78:0/2379819990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7006d350 msgr2=0x7fec700c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:01.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 --2- 172.21.15.78:0/2379819990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7006d350 0x7fec700c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 -- 172.21.15.78:0/2379819990 >> v2:172.21.15.150:3300/0 conn(0x7fec700c7070 msgr2=0x7fec7006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:01.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 --2- 172.21.15.78:0/2379819990 >> v2:172.21.15.150:3300/0 conn(0x7fec700c7070 0x7fec7006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.703+0000 7fec6ffff640 1 -- 172.21.15.78:0/2379819990 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700c78a0 con 0x7fec70152480 2024-02-19T14:59:01.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec6ffff640 1 --2- 172.21.15.78:0/2379819990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec701559b0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fec700c6a90 tx=0x7fec6002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2133719eeea0a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:01.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec6effd640 1 -- 172.21.15.78:0/2379819990 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60031860 con 0x7fec70152480 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec6effd640 1 -- 172.21.15.78:0/2379819990 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec600319c0 con 0x7fec70152480 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec6effd640 1 -- 172.21.15.78:0/2379819990 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60031d50 con 0x7fec70152480 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 -- 172.21.15.78:0/2379819990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 msgr2=0x7fec701559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 --2- 172.21.15.78:0/2379819990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec701559b0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fec700c6a90 tx=0x7fec6002b900 comp rx=0 tx=0).stop 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 -- 172.21.15.78:0/2379819990 shutdown_connections 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 --2- 172.21.15.78:0/2379819990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec7006d350 0x7fec700c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 --2- 172.21.15.78:0/2379819990 >> v2:172.21.15.150:3300/0 conn(0x7fec700c7070 0x7fec7006ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 --2- 172.21.15.78:0/2379819990 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec701559b0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 -- 172.21.15.78:0/2379819990 >> 172.21.15.78:0/2379819990 conn(0x7fec700ca8a0 msgr2=0x7fec700cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 -- 172.21.15.78:0/2379819990 shutdown_connections 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.704+0000 7fec762da640 1 -- 172.21.15.78:0/2379819990 wait complete. 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.705+0000 7fec762da640 1 Processor -- start 2024-02-19T14:59:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 -- start start 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec700c7070 0x7fec700acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec7014d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec700c1410 0x7fec7014de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec701eafb0 con 0x7fec70152480 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec700be3d0 con 0x7fec700c7070 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec762da640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec70157fc0 con 0x7fec700c1410 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec7014d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec6f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec7014d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57278/0 (socket says 172.21.15.78:57278) 2024-02-19T14:59:01.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.706+0000 7fec6f7fe640 1 -- 172.21.15.78:0/3766852821 learned_addr learned my addr 172.21.15.78:0/3766852821 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6f7fe640 1 -- 172.21.15.78:0/3766852821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec700c1410 msgr2=0x7fec7014de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6f7fe640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec700c1410 0x7fec7014de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6f7fe640 1 -- 172.21.15.78:0/3766852821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec700c7070 msgr2=0x7fec700acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6f7fe640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec700c7070 0x7fec700acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6f7fe640 1 -- 172.21.15.78:0/3766852821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600094d0 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6f7fe640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec7014d8c0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fec6400b450 tx=0x7fec6400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011890 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec640119f0 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec701ef2f0 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64011da0 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.707+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec70158540 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.708+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec70158700 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.708+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fec6401a400 con 0x7fec70152480 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.709+0000 7fec6cff9640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec5c06b840 0x7fec5c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:01.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.709+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fec6408a330 con 0x7fec70152480 2024-02-19T14:59:01.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.709+0000 7fec6ffff640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec5c06b840 0x7fec5c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:01.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.709+0000 7fec6ffff640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec5c06b840 0x7fec5c06dd00 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fec60036000 tx=0x7fec60030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:01.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.713+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fec64054f50 con 0x7fec70152480 2024-02-19T14:59:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.884+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec701eb100 con 0x7fec70152480 2024-02-19T14:59:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.885+0000 7fec6cff9640 1 -- 172.21.15.78:0/3766852821 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+13291 (secure 0 0 0) 0x7fec701eb100 con 0x7fec70152480 2024-02-19T14:59:01.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:01.888 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":179,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:00.579604+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6836/3613671589":"2024-02-20T14:58:57.571607+0000","172.21.15.78:6834/3613671589":"2024-02-20T14:58:57.571607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec5c06b840 msgr2=0x7fec5c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec5c06b840 0x7fec5c06dd00 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fec60036000 tx=0x7fec60030000 comp rx=0 tx=0).stop 2024-02-19T14:59:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 msgr2=0x7fec7014d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec7014d8c0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fec6400b450 tx=0x7fec6400b910 comp rx=0 tx=0).stop 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 shutdown_connections 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec5c06b840 0x7fec5c06dd00 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec70152480 0x7fec7014d8c0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec700c1410 0x7fec7014de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 --2- 172.21.15.78:0/3766852821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec700c7070 0x7fec700acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 >> 172.21.15.78:0/3766852821 conn(0x7fec700ca8a0 msgr2=0x7fec700c9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 shutdown_connections 2024-02-19T14:59:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:01.887+0000 7fec762da640 1 -- 172.21.15.78:0/3766852821 wait complete. 2024-02-19T14:59:01.901 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:59:01.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:59:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.004+0000 7efcc3ac0640 1 Processor -- start 2024-02-19T14:59:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc3ac0640 1 -- start start 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc3ac0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efcbc0c25e0 0x7efcbc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc3ac0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc072250 con 0x7efcbc0c5be0 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc3ac0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc072480 con 0x7efcbc0c29c0 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc3ac0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc152c90 con 0x7efcbc0c25e0 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc205d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efcbc0c5be0 0x7efcbc06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52344/0 (socket says 172.21.15.78:52344) 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.005+0000 7efcc205d640 1 -- 172.21.15.78:0/3678492031 learned_addr learned my addr 172.21.15.78:0/3678492031 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc185c640 1 --2- 172.21.15.78:0/3678492031 >> v2:172.21.15.150:3300/0 conn(0x7efcbc0c25e0 0x7efcbc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639857986 0 0) 0x7efcbc072250 con 0x7efcbc0c5be0 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8003520 con 0x7efcbc0c5be0 2024-02-19T14:59:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262840302 0 0) 0x7efca8003520 con 0x7efcbc0c5be0 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 >> v2:172.21.15.150:3300/0 conn(0x7efcbc0c25e0 msgr2=0x7efcbc1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 --2- 172.21.15.78:0/3678492031 >> v2:172.21.15.150:3300/0 conn(0x7efcbc0c25e0 0x7efcbc1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 >> v1:172.21.15.150:6789/0 conn(0x7efcbc0c29c0 legacy=0x7efcbc0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc0c4b90 con 0x7efcbc0c5be0 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7efcb40019f0 con 0x7efcbc0c5be0 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 >> v1:172.21.15.78:6789/0 conn(0x7efcbc0c5be0 legacy=0x7efcbc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efca8003970 0x7efca8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efca8005180 0x7efca800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc152c90 con 0x7efca80047c0 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca800c120 con 0x7efca8005180 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca800c2e0 con 0x7efca8003970 2024-02-19T14:59:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc105b640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc185c640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efca8003970 0x7efca8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc205d640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efca8005180 0x7efca800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc105b640 1 -- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efca8003970 msgr2=0x7efca8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc105b640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efca8003970 0x7efca8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc105b640 1 -- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efca8005180 msgr2=0x7efca800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc105b640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efca8005180 0x7efca800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc105b640 1 -- 172.21.15.78:0/3678492031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8003520 con 0x7efca80047c0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc205d640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efca8005180 0x7efca800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.006+0000 7efcc205d640 1 -- 172.21.15.78:0/3678492031 reap_dead start 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.007+0000 7efcc105b640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efca8004c40 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7efcac001750 tx=0x7efca80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72638a600096dd54 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.007+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0052d0 con 0x7efca80047c0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.007+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac02e050 con 0x7efca80047c0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.007+0000 7efcc085a640 1 -- 172.21.15.78:0/3678492031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac003830 con 0x7efca80047c0 2024-02-19T14:59:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 -- 172.21.15.78:0/3678492031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 msgr2=0x7efca8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efca8004c40 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7efcac001750 tx=0x7efca80057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 -- 172.21.15.78:0/3678492031 shutdown_connections 2024-02-19T14:59:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/3678492031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efca8004c40 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 -- 172.21.15.78:0/3678492031 >> 172.21.15.78:0/3678492031 conn(0x7efcbc06b270 msgr2=0x7efcbc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 -- 172.21.15.78:0/3678492031 shutdown_connections 2024-02-19T14:59:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.008+0000 7efcc3ac0640 1 -- 172.21.15.78:0/3678492031 wait complete. 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.009+0000 7efcc3ac0640 1 Processor -- start 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.009+0000 7efcc3ac0640 1 -- start start 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efcbc155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc1db620 0x7efcbc14ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc14b3a0 0x7efcbc154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1ec3c0 con 0x7efca80047c0 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0c1210 con 0x7efcbc14b3a0 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1560b0 con 0x7efcbc1db620 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efcbc155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efcbc155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57296/0 (socket says 172.21.15.78:57296) 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 -- 172.21.15.78:0/2010928699 learned_addr learned my addr 172.21.15.78:0/2010928699 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc205d640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc14b3a0 0x7efcbc154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc105b640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc1db620 0x7efcbc14ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 -- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc1db620 msgr2=0x7efcbc14ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc1db620 0x7efcbc14ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 -- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc14b3a0 msgr2=0x7efcbc154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc14b3a0 0x7efcbc154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 -- 172.21.15.78:0/2010928699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc1eb7b0 con 0x7efca80047c0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc205d640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc14b3a0 0x7efcbc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc105b640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc1db620 0x7efcbc14ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc185c640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efcbc155b70 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7efcb8011700 tx=0x7efcb8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb800f9c0 con 0x7efca80047c0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb800fb20 con 0x7efca80047c0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb800fe90 con 0x7efca80047c0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc1eba90 con 0x7efca80047c0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.010+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc1ec050 con 0x7efca80047c0 2024-02-19T14:59:02.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.011+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc074990 con 0x7efca80047c0 2024-02-19T14:59:02.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.012+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efcb8015460 con 0x7efca80047c0 2024-02-19T14:59:02.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.012+0000 7efcb27fc640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca006b810 0x7efca006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.012+0000 7efcc105b640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca006b810 0x7efca006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.012+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7efcb808a8e0 con 0x7efca80047c0 2024-02-19T14:59:02.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.012+0000 7efcc105b640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca006b810 0x7efca006dcd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7efcac033000 tx=0x7efcac009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.016+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efcb8055500 con 0x7efca80047c0 2024-02-19T14:59:02.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.186+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efcbc14ff60 con 0x7efca80047c0 2024-02-19T14:59:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.186+0000 7efcb27fc640 1 -- 172.21.15.78:0/2010928699 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcb8058bc0 con 0x7efca80047c0 2024-02-19T14:59:02.189 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca006b810 msgr2=0x7efca006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca006b810 0x7efca006dcd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7efcac033000 tx=0x7efcac009030 comp rx=0 tx=0).stop 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 msgr2=0x7efcbc155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efcbc155b70 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7efcb8011700 tx=0x7efcb8011bc0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 shutdown_connections 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca006b810 0x7efca006dcd0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc14b3a0 0x7efcbc154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc1db620 0x7efcbc14ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 --2- 172.21.15.78:0/2010928699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efca80047c0 0x7efcbc155b70 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 >> 172.21.15.78:0/2010928699 conn(0x7efcbc06b270 msgr2=0x7efcbc0ac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:02.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 shutdown_connections 2024-02-19T14:59:02.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.188+0000 7efcc3ac0640 1 -- 172.21.15.78:0/2010928699 wait complete. 2024-02-19T14:59:02.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-02-19T14:59:02.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.298+0000 7f6015684640 1 Processor -- start 2024-02-19T14:59:02.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.298+0000 7f6015684640 1 -- start start 2024-02-19T14:59:02.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f6015684640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60100c8e00 0x7f601006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f6015684640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f601006d910 0x7f601006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f6015684640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601006cd20 con 0x7f6010152700 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f6015684640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60101551f0 con 0x7f601006d910 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f6015684640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6010155490 con 0x7f60100c8e00 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f601006d910 0x7f601006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6010152700 0x7f60100c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52346/0 (socket says 172.21.15.78:52346) 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600effd640 1 -- 172.21.15.78:0/151770395 learned_addr learned my addr 172.21.15.78:0/151770395 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600f7fe640 1 -- 172.21.15.78:0/151770395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60100c8e00 msgr2=0x7f601006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600e7fc640 1 --2- 172.21.15.78:0/151770395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60100c8e00 0x7f601006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600dffb640 1 -- 172.21.15.78:0/151770395 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537050806 0 0) 0x7f601006cd20 con 0x7f6010152700 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600f7fe640 1 --2- 172.21.15.78:0/151770395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60100c8e00 0x7f601006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600f7fe640 1 -- 172.21.15.78:0/151770395 >> v1:172.21.15.78:6789/0 conn(0x7f6010152700 legacy=0x7f60100c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.299+0000 7f600f7fe640 1 -- 172.21.15.78:0/151770395 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010155770 con 0x7f601006d910 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.300+0000 7f600f7fe640 1 --2- 172.21.15.78:0/151770395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f601006c7e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f60080097a0 tx=0x7f600802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7802c65f9c51f973 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.300+0000 7f600dffb640 1 -- 172.21.15.78:0/151770395 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600802e070 con 0x7f601006d910 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.300+0000 7f600dffb640 1 -- 172.21.15.78:0/151770395 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60080315a0 con 0x7f601006d910 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.300+0000 7f600dffb640 1 -- 172.21.15.78:0/151770395 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008031930 con 0x7f601006d910 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 -- 172.21.15.78:0/151770395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 msgr2=0x7f601006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 --2- 172.21.15.78:0/151770395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f601006c7e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f60080097a0 tx=0x7f600802b840 comp rx=0 tx=0).stop 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 -- 172.21.15.78:0/151770395 shutdown_connections 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 --2- 172.21.15.78:0/151770395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f601006c7e0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 --2- 172.21.15.78:0/151770395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60100c8e00 0x7f601006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 -- 172.21.15.78:0/151770395 >> 172.21.15.78:0/151770395 conn(0x7f60100cac90 msgr2=0x7f60100cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 -- 172.21.15.78:0/151770395 shutdown_connections 2024-02-19T14:59:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.301+0000 7f6015684640 1 -- 172.21.15.78:0/151770395 wait complete. 2024-02-19T14:59:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.302+0000 7f6015684640 1 Processor -- start 2024-02-19T14:59:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.302+0000 7f6015684640 1 -- start start 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f6015684640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f6010153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f6015684640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60100c8e00 0x7f60101eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f6015684640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6010152700 0x7f60101eb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f6015684640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60101eaaa0 con 0x7f601006d910 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f6015684640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6010156450 con 0x7f60100c8e00 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f6015684640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60100aa2b0 con 0x7f6010152700 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f6010153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f6010153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57328/0 (socket says 172.21.15.78:57328) 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 -- 172.21.15.78:0/3131110922 learned_addr learned my addr 172.21.15.78:0/3131110922 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600e7fc640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60100c8e00 0x7f60101eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 -- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6010152700 msgr2=0x7f60101eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6010152700 0x7f60101eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 -- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60100c8e00 msgr2=0x7f60101eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60100c8e00 0x7f60101eaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 -- 172.21.15.78:0/3131110922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6008009400 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600e7fc640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60100c8e00 0x7f60101eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.303+0000 7f600effd640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f6010153f20 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f600000c7c0 tx=0x7f600000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.304+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600000a640 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.304+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600000a7c0 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.304+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60100aab30 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.304+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600000ab70 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.304+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6010154460 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.305+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010074990 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.305+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f600000acd0 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.305+0000 7f5ff77fe640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ffc06b810 0x7f5ffc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.305+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6000089860 con 0x7f601006d910 2024-02-19T14:59:02.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.305+0000 7f600e7fc640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ffc06b810 0x7f5ffc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.306+0000 7f600e7fc640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ffc06b810 0x7f5ffc06dcd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f60101ef0d0 tx=0x7f6004008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.310+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6000054480 con 0x7f601006d910 2024-02-19T14:59:02.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.547+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f60100c1600 con 0x7f601006d910 2024-02-19T14:59:02.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.592+0000 7f5ff77fe640 1 -- 172.21.15.78:0/3131110922 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v416) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6000018020 con 0x7f601006d910 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ffc06b810 msgr2=0x7f5ffc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ffc06b810 0x7f5ffc06dcd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f60101ef0d0 tx=0x7f6004008040 comp rx=0 tx=0).stop 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 msgr2=0x7f6010153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f6010153f20 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f600000c7c0 tx=0x7f600000cc80 comp rx=0 tx=0).stop 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 shutdown_connections 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ffc06b810 0x7f5ffc06dcd0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6010152700 0x7f60101eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60100c8e00 0x7f60101eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 --2- 172.21.15.78:0/3131110922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f601006d910 0x7f6010153f20 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 >> 172.21.15.78:0/3131110922 conn(0x7f60100cac90 msgr2=0x7f60101516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:02.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 shutdown_connections 2024-02-19T14:59:02.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.594+0000 7f6015684640 1 -- 172.21.15.78:0/3131110922 wait complete. 2024-02-19T14:59:02.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:59:02.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.708+0000 7f64e96d4640 1 Processor -- start 2024-02-19T14:59:02.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.708+0000 7f64e96d4640 1 -- start start 2024-02-19T14:59:02.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e96d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f64e4152700 0x7f64e40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e96d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e96d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e41551f0 con 0x7f64e40c8e00 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e96d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e4155490 con 0x7f64e4152700 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e96d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e4155770 con 0x7f64e40c5730 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e2ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f64e4152700 0x7f64e40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f64e40c8e00 0x7f64e4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:60464/0 (socket says 172.21.15.78:60464) 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.709+0000 7f64e37fe640 1 -- 172.21.15.78:0/1630095992 learned_addr learned my addr 172.21.15.78:0/1630095992 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e27fc640 1 -- 172.21.15.78:0/1630095992 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465560976 0 0) 0x7f64e41551f0 con 0x7f64e40c8e00 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e27fc640 1 -- 172.21.15.78:0/1630095992 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d0003520 con 0x7f64e40c8e00 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e2ffd640 1 -- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 msgr2=0x7f64e406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e2ffd640 1 --2- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e2ffd640 1 -- 172.21.15.78:0/1630095992 >> v1:172.21.15.150:6789/0 conn(0x7f64e40c8e00 legacy=0x7f64e4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e2ffd640 1 -- 172.21.15.78:0/1630095992 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e4155a50 con 0x7f64e4152700 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64dbfff640 1 --2- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:02.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.710+0000 7f64e2ffd640 1 --2- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e40c51f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097a0 tx=0x7f64d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a80bfe4cb69b612d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e27fc640 1 -- 172.21.15.78:0/1630095992 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4031830 con 0x7f64e4152700 2024-02-19T14:59:02.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e27fc640 1 -- 172.21.15.78:0/1630095992 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d4031990 con 0x7f64e4152700 2024-02-19T14:59:02.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e27fc640 1 -- 172.21.15.78:0/1630095992 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4031d20 con 0x7f64e4152700 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 -- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 msgr2=0x7f64e40c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e40c51f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097a0 tx=0x7f64d402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 -- 172.21.15.78:0/1630095992 shutdown_connections 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1630095992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e40c51f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 -- 172.21.15.78:0/1630095992 >> 172.21.15.78:0/1630095992 conn(0x7f64e40cac90 msgr2=0x7f64e40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 -- 172.21.15.78:0/1630095992 shutdown_connections 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.711+0000 7f64e96d4640 1 -- 172.21.15.78:0/1630095992 wait complete. 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.712+0000 7f64e96d4640 1 Processor -- start 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.712+0000 7f64e96d4640 1 -- start start 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e96d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e40aa750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e96d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 0x7f64e40ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e96d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e41e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e96d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40ac6f0 con 0x7f64e40c8e00 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e96d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40afe70 con 0x7f64e4152700 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e96d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e4154d80 con 0x7f64e40c5730 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 0x7f64e40ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 0x7f64e40ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57338/0 (socket says 172.21.15.78:57338) 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 -- 172.21.15.78:0/1954022158 learned_addr learned my addr 172.21.15.78:0/1954022158 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e2ffd640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e40aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64e37fe640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e41e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 -- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 msgr2=0x7f64e40aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e40aa750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 -- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 msgr2=0x7f64e41e7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e41e7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.713+0000 7f64dbfff640 1 -- 172.21.15.78:0/1954022158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc009400 con 0x7f64e40c8e00 2024-02-19T14:59:02.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.714+0000 7f64dbfff640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 0x7f64e40ae740 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00ebe0 tx=0x7f64cc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.714+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc009760 con 0x7f64e40c8e00 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.714+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc00cc30 con 0x7f64e40c8e00 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.714+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d4009400 con 0x7f64e40c8e00 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.714+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00eee0 con 0x7f64e40c8e00 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.714+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e40afb90 con 0x7f64e40c8e00 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.715+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e4074990 con 0x7f64e40c8e00 2024-02-19T14:59:02.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.715+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f64cc013470 con 0x7f64e40c8e00 2024-02-19T14:59:02.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.716+0000 7f64db7fe640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f64c406b840 0x7f64c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:02.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.716+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f64cc08a250 con 0x7f64e40c8e00 2024-02-19T14:59:02.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.716+0000 7f64e2ffd640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f64c406b840 0x7f64c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:02.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.716+0000 7f64e2ffd640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f64c406b840 0x7f64c406dd00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f64d4036000 tx=0x7f64d403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:02.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.720+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64cc054f20 con 0x7f64e40c8e00 2024-02-19T14:59:02.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.890+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f64e40c6750 con 0x7f64e40c8e00 2024-02-19T14:59:02.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.893+0000 7f64db7fe640 1 -- 172.21.15.78:0/1954022158 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v59)=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f64cc0585e0 con 0x7f64e40c8e00 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f64c406b840 msgr2=0x7f64c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f64c406b840 0x7f64c406dd00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f64d4036000 tx=0x7f64d403c040 comp rx=0 tx=0).stop 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 msgr2=0x7f64e40ae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 0x7f64e40ae740 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00ebe0 tx=0x7f64cc00c520 comp rx=0 tx=0).stop 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 shutdown_connections 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f64c406b840 0x7f64c406dd00 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64e40c8e00 0x7f64e40ae740 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64e4152700 0x7f64e41e7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 --2- 172.21.15.78:0/1954022158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64e40c5730 0x7f64e40aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 >> 172.21.15.78:0/1954022158 conn(0x7f64e40cac90 msgr2=0x7f64e40cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:02.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 shutdown_connections 2024-02-19T14:59:02.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:02.896+0000 7f64e96d4640 1 -- 172.21.15.78:0/1954022158 wait complete. 2024-02-19T14:59:02.909 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:59:02.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:59:02.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:59:02.990 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:59:02.991 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:59:02.991 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:59:03.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:59:03.006 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:59:03.006 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:59:03.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:59:03.024 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:59:03.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.144+0000 7f004c1d7640 1 Processor -- start 2024-02-19T14:59:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.144+0000 7f004c1d7640 1 -- start start 2024-02-19T14:59:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.144+0000 7f004c1d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00440c7070 0x7f00441559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.144+0000 7f004c1d7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f00440c87a0 0x7f00440c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004c1d7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0044155ef0 con 0x7f0044152480 2024-02-19T14:59:03.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004c1d7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f004406dec0 con 0x7f00440c87a0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004c1d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f004406d520 con 0x7f00440c7070 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f00440c87a0 0x7f00440c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f00440c87a0 0x7f00440c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57350/0 (socket says 172.21.15.78:57350) 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 -- 172.21.15.78:0/2014990848 learned_addr learned my addr 172.21.15.78:0/2014990848 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f0049772640 1 --2- 172.21.15.78:0/2014990848 >> v2:172.21.15.150:3300/0 conn(0x7f00440c7070 0x7f00441559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 -- 172.21.15.78:0/2014990848 >> v2:172.21.15.150:3300/0 conn(0x7f00440c7070 msgr2=0x7f00441559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 --2- 172.21.15.78:0/2014990848 >> v2:172.21.15.150:3300/0 conn(0x7f00440c7070 0x7f00441559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 -- 172.21.15.78:0/2014990848 >> v1:172.21.15.150:6789/0 conn(0x7f0044152480 legacy=0x7f004406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.145+0000 7f004a774640 1 -- 172.21.15.78:0/2014990848 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00440c75c0 con 0x7f00440c87a0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004a774640 1 --2- 172.21.15.78:0/2014990848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f00440c5220 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f00440c6a90 tx=0x7f003c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15c435ddb280ad3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f0048f71640 1 -- 172.21.15.78:0/2014990848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c02cb20 con 0x7f00440c87a0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f0048f71640 1 -- 172.21.15.78:0/2014990848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c02cc80 con 0x7f00440c87a0 2024-02-19T14:59:03.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f0048f71640 1 -- 172.21.15.78:0/2014990848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c0343f0 con 0x7f00440c87a0 2024-02-19T14:59:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 -- 172.21.15.78:0/2014990848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 msgr2=0x7f00440c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 --2- 172.21.15.78:0/2014990848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f00440c5220 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f00440c6a90 tx=0x7f003c02bf80 comp rx=0 tx=0).stop 2024-02-19T14:59:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 -- 172.21.15.78:0/2014990848 shutdown_connections 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 --2- 172.21.15.78:0/2014990848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f00440c5220 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 --2- 172.21.15.78:0/2014990848 >> v2:172.21.15.150:3300/0 conn(0x7f00440c7070 0x7f00441559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 -- 172.21.15.78:0/2014990848 >> 172.21.15.78:0/2014990848 conn(0x7f00440ca8a0 msgr2=0x7f00440cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 -- 172.21.15.78:0/2014990848 shutdown_connections 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.146+0000 7f004c1d7640 1 -- 172.21.15.78:0/2014990848 wait complete. 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 Processor -- start 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 -- start start 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00440c7070 0x7f00441eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f0044158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0044152480 0x7f00441ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00441586f0 con 0x7f00440c87a0 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00441eb490 con 0x7f00440c7070 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f004c1d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00441eded0 con 0x7f0044152480 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f0049772640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f0044158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f0049772640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f0044158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57352/0 (socket says 172.21.15.78:57352) 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.148+0000 7f0049772640 1 -- 172.21.15.78:0/1935875929 learned_addr learned my addr 172.21.15.78:0/1935875929 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049772640 1 -- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0044152480 msgr2=0x7f00441ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049772640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0044152480 0x7f00441ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049f73640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00440c7070 0x7f00441eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049772640 1 -- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00440c7070 msgr2=0x7f00441eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049772640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00440c7070 0x7f00441eaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049772640 1 -- 172.21.15.78:0/1935875929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c009b30 con 0x7f00440c87a0 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f0049772640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f0044158d80 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f004000d7d0 tx=0x7f004000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040018070 con 0x7f00440c87a0 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0040004020 con 0x7f00440c87a0 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00440af670 con 0x7f00440c87a0 2024-02-19T14:59:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00400041b0 con 0x7f00440c87a0 2024-02-19T14:59:03.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.149+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00440afc10 con 0x7f00440c87a0 2024-02-19T14:59:03.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.150+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0044074990 con 0x7f00440c87a0 2024-02-19T14:59:03.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.150+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0040004310 con 0x7f00440c87a0 2024-02-19T14:59:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.151+0000 7f00367fc640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002806b840 0x7f002806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.151+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0040089e60 con 0x7f00440c87a0 2024-02-19T14:59:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.151+0000 7f0049f73640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002806b840 0x7f002806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.151+0000 7f0049f73640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002806b840 0x7f002806dd00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0044154400 tx=0x7f0038009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:03.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.155+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0040054a80 con 0x7f00440c87a0 2024-02-19T14:59:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.362+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00441ee010 con 0x7f00440c87a0 2024-02-19T14:59:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.362+0000 7f00367fc640 1 -- 172.21.15.78:0/1935875929 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+3704 (secure 0 0 0) 0x7f0040058140 con 0x7f00440c87a0 2024-02-19T14:59:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:03.366 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":161,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3287472275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3287472275},{"type":"v1","addr":"172.21.15.150:6834","nonce":3287472275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":7172,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3929715858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3929715858},{"type":"v1","addr":"172.21.15.150:6835","nonce":3929715858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":8269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/1939412590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":1939412590},{"type":"v1","addr":"172.21.15.78:6837","nonce":1939412590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":149},{"gid":9744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/64464814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":64464814},{"type":"v1","addr":"172.21.15.78:6836","nonce":64464814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[]} 2024-02-19T14:59:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 161 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002806b840 msgr2=0x7f002806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002806b840 0x7f002806dd00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0044154400 tx=0x7f0038009040 comp rx=0 tx=0).stop 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 msgr2=0x7f0044158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f0044158d80 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f004000d7d0 tx=0x7f004000dc90 comp rx=0 tx=0).stop 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 shutdown_connections 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f002806b840 0x7f002806dd00 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00440c87a0 0x7f0044158d80 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0044152480 0x7f00441ee360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.364+0000 7f004c1d7640 1 --2- 172.21.15.78:0/1935875929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00440c7070 0x7f00441eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.365+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 >> 172.21.15.78:0/1935875929 conn(0x7f00440ca8a0 msgr2=0x7f0044153810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.365+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 shutdown_connections 2024-02-19T14:59:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.365+0000 7f004c1d7640 1 -- 172.21.15.78:0/1935875929 wait complete. 2024-02-19T14:59:03.378 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:59:03.378 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:59:03.378 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:59:03.379 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:03.379 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:59:03.379 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:59:03.379 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:59:03.379 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:03.379 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:59:03.379 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:59:03.380 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:59:03.380 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:03.380 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:59:03.380 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:59:03.380 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:59:03.380 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:03.387 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:59:03.387 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:59:03.388 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:59:03.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:59:03.391 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:59:03.391 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:59:03.392 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:59:03.392 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:59:03.430 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:59:03.430 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:03.431 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:59:03.431 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:03.434 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:59:03.434 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:03.435 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:59:03.435 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:03.435 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:59:03.505 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:59:03.506 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:59:03.517 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:59:03.518 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:59:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.537+0000 7f6ca8a5c640 1 Processor -- start 2024-02-19T14:59:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.537+0000 7f6ca8a5c640 1 -- start start 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca8a5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40add70 0x7f6ca40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca8a5c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ca40705b0 0x7f6ca40709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca8a5c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40ae670 con 0x7f6ca40c2f60 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca8a5c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca40716b0 con 0x7f6ca40705b0 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca8a5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca40c33a0 con 0x7f6ca40add70 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca1d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6ca40c2f60 0x7f6ca406eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52360/0 (socket says 172.21.15.78:52360) 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca1d9b640 1 -- 172.21.15.78:0/3088211064 learned_addr learned my addr 172.21.15.78:0/3088211064 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca259c640 1 --2- 172.21.15.78:0/3088211064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40add70 0x7f6ca40c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca2d9d640 1 --2- 172.21.15.78:0/3088211064 >> v2:172.21.15.150:3300/0 conn(0x7f6ca40705b0 0x7f6ca40709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca159a640 1 -- 172.21.15.78:0/3088211064 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240070349 0 0) 0x7f6ca40ae670 con 0x7f6ca40c2f60 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca159a640 1 -- 172.21.15.78:0/3088211064 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c8c003520 con 0x7f6ca40c2f60 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca259c640 1 -- 172.21.15.78:0/3088211064 >> v2:172.21.15.150:3300/0 conn(0x7f6ca40705b0 msgr2=0x7f6ca40709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca259c640 1 --2- 172.21.15.78:0/3088211064 >> v2:172.21.15.150:3300/0 conn(0x7f6ca40705b0 0x7f6ca40709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca259c640 1 -- 172.21.15.78:0/3088211064 >> v1:172.21.15.78:6789/0 conn(0x7f6ca40c2f60 legacy=0x7f6ca406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca159a640 1 -- 172.21.15.78:0/3088211064 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2890267387 0 0) 0x7f6c8c003520 con 0x7f6ca40c2f60 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca259c640 1 -- 172.21.15.78:0/3088211064 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca4153780 con 0x7f6ca40add70 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.538+0000 7f6ca2d9d640 1 --2- 172.21.15.78:0/3088211064 >> v2:172.21.15.150:3300/0 conn(0x7f6ca40705b0 0x7f6ca40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.539+0000 7f6ca259c640 1 --2- 172.21.15.78:0/3088211064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40add70 0x7f6ca40c4980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c940097a0 tx=0x7f6c9402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a404391c12748b86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.539+0000 7f6ca159a640 1 -- 172.21.15.78:0/3088211064 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98001670 con 0x7f6ca40add70 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.539+0000 7f6ca159a640 1 -- 172.21.15.78:0/3088211064 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9402e050 con 0x7f6ca40add70 2024-02-19T14:59:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.539+0000 7f6ca159a640 1 -- 172.21.15.78:0/3088211064 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94030710 con 0x7f6ca40add70 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/3088211064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40add70 msgr2=0x7f6ca40c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/3088211064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40add70 0x7f6ca40c4980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6c940097a0 tx=0x7f6c9402b710 comp rx=0 tx=0).stop 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/3088211064 shutdown_connections 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/3088211064 >> v2:172.21.15.150:3300/0 conn(0x7f6ca40705b0 0x7f6ca40709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/3088211064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40add70 0x7f6ca40c4980 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/3088211064 >> 172.21.15.78:0/3088211064 conn(0x7f6ca40c85e0 msgr2=0x7f6ca40c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/3088211064 shutdown_connections 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.540+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/3088211064 wait complete. 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.541+0000 7f6ca8a5c640 1 Processor -- start 2024-02-19T14:59:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.541+0000 7f6ca8a5c640 1 -- start start 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca8a5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40705b0 0x7f6ca4156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca8a5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 0x7f6ca4147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca8a5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca40c2f60 0x7f6ca4151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca8a5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41528d0 con 0x7f6ca40add70 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca8a5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41571b0 con 0x7f6ca40c2f60 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca8a5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41e8380 con 0x7f6ca40705b0 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 0x7f6ca4147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 0x7f6ca4147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57386/0 (socket says 172.21.15.78:57386) 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 -- 172.21.15.78:0/1044438227 learned_addr learned my addr 172.21.15.78:0/1044438227 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca259c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40705b0 0x7f6ca4156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca2d9d640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca40c2f60 0x7f6ca4151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 -- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40705b0 msgr2=0x7f6ca4156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40705b0 0x7f6ca4156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 -- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca40c2f60 msgr2=0x7f6ca4151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca40c2f60 0x7f6ca4151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 -- 172.21.15.78:0/1044438227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009a70 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.542+0000 7f6ca1d9b640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 0x7f6ca4147600 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c750 tx=0x7f6c9800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.543+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98005ad0 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.543+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9800ce90 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.543+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c94009400 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.543+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98007070 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.543+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca41e8620 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.544+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca40676a0 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.544+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c980071d0 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.544+0000 7f6c92ffd640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8406b810 0x7f6c8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.544+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6c98089b80 con 0x7f6ca40add70 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.544+0000 7f6ca259c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8406b810 0x7f6c8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.545+0000 7f6ca259c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8406b810 0x7f6c8406dcd0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f6c94034000 tx=0x7f6c94008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:03.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.549+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c980547a0 con 0x7f6ca40add70 2024-02-19T14:59:03.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:03.738+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f6ca4074990 con 0x7f6ca40add70 2024-02-19T14:59:04.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.613+0000 7f6c92ffd640 1 -- 172.21.15.78:0/1044438227 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v184) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6c98057e60 con 0x7f6ca40add70 2024-02-19T14:59:04.616 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8406b810 msgr2=0x7f6c8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8406b810 0x7f6c8406dcd0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f6c94034000 tx=0x7f6c94008030 comp rx=0 tx=0).stop 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 msgr2=0x7f6ca4147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 0x7f6ca4147600 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c750 tx=0x7f6c9800cb50 comp rx=0 tx=0).stop 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 shutdown_connections 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c8406b810 0x7f6c8406dcd0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca40add70 0x7f6ca4147600 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ca40c2f60 0x7f6ca4151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 --2- 172.21.15.78:0/1044438227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ca40705b0 0x7f6ca4156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 >> 172.21.15.78:0/1044438227 conn(0x7f6ca40c85e0 msgr2=0x7f6ca4070990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:04.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 shutdown_connections 2024-02-19T14:59:04.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.616+0000 7f6ca8a5c640 1 -- 172.21.15.78:0/1044438227 wait complete. 2024-02-19T14:59:04.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:59:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.727+0000 7f0619b6c640 1 Processor -- start 2024-02-19T14:59:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.727+0000 7f0619b6c640 1 -- start start 2024-02-19T14:59:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f0619b6c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0614155f90 0x7f061406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f0619b6c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06140c6d30 0x7f06140c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f0619b6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c3aa0 0x7f06141588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f0619b6c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06140c6af0 con 0x7f06140c6d30 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f0619b6c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06140c7940 con 0x7f0614155f90 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f0619b6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06140c7c20 con 0x7f06140c3aa0 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f060affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06140c6d30 0x7f06140c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f060affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06140c6d30 0x7f06140c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57396/0 (socket says 172.21.15.78:57396) 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.728+0000 7f060affd640 1 -- 172.21.15.78:0/3920541432 learned_addr learned my addr 172.21.15.78:0/3920541432 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f0613fff640 1 --2- 172.21.15.78:0/3920541432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c3aa0 0x7f06141588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f06137fe640 1 --2- 172.21.15.78:0/3920541432 >> v2:172.21.15.150:3300/0 conn(0x7f0614155f90 0x7f061406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f060affd640 1 -- 172.21.15.78:0/3920541432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c3aa0 msgr2=0x7f06141588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f060affd640 1 --2- 172.21.15.78:0/3920541432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c3aa0 0x7f06141588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f060affd640 1 -- 172.21.15.78:0/3920541432 >> v2:172.21.15.150:3300/0 conn(0x7f0614155f90 msgr2=0x7f061406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f060affd640 1 --2- 172.21.15.78:0/3920541432 >> v2:172.21.15.150:3300/0 conn(0x7f0614155f90 0x7f061406d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f060affd640 1 -- 172.21.15.78:0/3920541432 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140c7f00 con 0x7f06140c6d30 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f060affd640 1 --2- 172.21.15.78:0/3920541432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c6d30 0x7f06140c3560 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f06140c2a40 tx=0x7f05fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea72d8f80a3d624f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.729+0000 7f0612ffd640 1 -- 172.21.15.78:0/3920541432 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0317e0 con 0x7f06140c6d30 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0612ffd640 1 -- 172.21.15.78:0/3920541432 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05fc031940 con 0x7f06140c6d30 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0612ffd640 1 -- 172.21.15.78:0/3920541432 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc031cd0 con 0x7f06140c6d30 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 -- 172.21.15.78:0/3920541432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c6d30 msgr2=0x7f06140c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3920541432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c6d30 0x7f06140c3560 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f06140c2a40 tx=0x7f05fc02b880 comp rx=0 tx=0).stop 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 -- 172.21.15.78:0/3920541432 shutdown_connections 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3920541432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c3aa0 0x7f06141588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3920541432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c6d30 0x7f06140c3560 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3920541432 >> v2:172.21.15.150:3300/0 conn(0x7f0614155f90 0x7f061406d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 -- 172.21.15.78:0/3920541432 >> 172.21.15.78:0/3920541432 conn(0x7f061406b700 msgr2=0x7f06140aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 -- 172.21.15.78:0/3920541432 shutdown_connections 2024-02-19T14:59:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.730+0000 7f0619b6c640 1 -- 172.21.15.78:0/3920541432 wait complete. 2024-02-19T14:59:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.731+0000 7f0619b6c640 1 Processor -- start 2024-02-19T14:59:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.731+0000 7f0619b6c640 1 -- start start 2024-02-19T14:59:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0619b6c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 0x7f06141eb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0619b6c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c6d30 0x7f0614099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0619b6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0614155f90 0x7f06140705a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0619b6c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06140acae0 con 0x7f06140c3aa0 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0619b6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06141ead70 con 0x7f0614155f90 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0619b6c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06141ea780 con 0x7f06140c6d30 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 0x7f06141eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 0x7f06141eb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57404/0 (socket says 172.21.15.78:57404) 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 -- 172.21.15.78:0/3518697028 learned_addr learned my addr 172.21.15.78:0/3518697028 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f060affd640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c6d30 0x7f0614099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f0613fff640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0614155f90 0x7f06140705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 -- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c6d30 msgr2=0x7f0614099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c6d30 0x7f0614099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 -- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0614155f90 msgr2=0x7f06140705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0614155f90 0x7f06140705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 -- 172.21.15.78:0/3518697028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc0094d0 con 0x7f06140c3aa0 2024-02-19T14:59:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.732+0000 7f06137fe640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 0x7f06141eb200 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f060400d5f0 tx=0x7f060400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.733+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06040125f0 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.733+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0604012750 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.733+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06141edf10 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.733+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604012b00 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.733+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06141eaa00 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.734+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0614074990 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.734+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0604012c60 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.734+0000 7f0610ff9640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05f806b810 0x7f05f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.734+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f060408a350 con 0x7f06140c3aa0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.735+0000 7f060affd640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05f806b810 0x7f05f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.735+0000 7f060affd640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05f806b810 0x7f05f806dcd0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f05fc036000 tx=0x7f05fc03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:04.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.739+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0604055070 con 0x7f06140c3aa0 2024-02-19T14:59:04.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.975+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f06140c75a0 con 0x7f06140c3aa0 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.976+0000 7f0610ff9640 1 -- 172.21.15.78:0/3518697028 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f0604088e60 con 0x7f06140c3aa0 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:59:04.981 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:59:04.983 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.984 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:59:04.985 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:59:04.986 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:59:04.988 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:04.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:04.990 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:59:04.991 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05f806b810 msgr2=0x7f05f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05f806b810 0x7f05f806dcd0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f05fc036000 tx=0x7f05fc03d040 comp rx=0 tx=0).stop 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 msgr2=0x7f06141eb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 0x7f06141eb200 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f060400d5f0 tx=0x7f060400dab0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 shutdown_connections 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05f806b810 0x7f05f806dcd0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0614155f90 0x7f06140705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06140c6d30 0x7f0614099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 --2- 172.21.15.78:0/3518697028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06140c3aa0 0x7f06141eb200 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:04.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 >> 172.21.15.78:0/3518697028 conn(0x7f061406b700 msgr2=0x7f06140aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:04.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 shutdown_connections 2024-02-19T14:59:04.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:04.978+0000 7f0619b6c640 1 -- 172.21.15.78:0/3518697028 wait complete. 2024-02-19T14:59:04.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:05.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.091+0000 7fcb4c072640 1 Processor -- start 2024-02-19T14:59:05.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.091+0000 7fcb4c072640 1 -- start start 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb4c072640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcb4414d110 0x7fcb440ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb4c072640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4406b620 con 0x7fcb44073220 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb4c072640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb44157230 con 0x7fcb4406bae0 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb4c072640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4406ea00 con 0x7fcb4414d110 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb4960d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcb44073220 0x7fcb4406ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52372/0 (socket says 172.21.15.78:52372) 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb4960d640 1 -- 172.21.15.78:0/3565161236 learned_addr learned my addr 172.21.15.78:0/3565161236 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817371153 0 0) 0x7fcb4406b620 con 0x7fcb44073220 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb30003520 con 0x7fcb44073220 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.092+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/3565161236 >> v2:172.21.15.150:3300/0 conn(0x7fcb4414d110 0x7fcb440ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3703207129 0 0) 0x7fcb30003520 con 0x7fcb44073220 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 >> v2:172.21.15.150:3300/0 conn(0x7fcb4414d110 msgr2=0x7fcb440ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 --2- 172.21.15.78:0/3565161236 >> v2:172.21.15.150:3300/0 conn(0x7fcb4414d110 0x7fcb440ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 >> v1:172.21.15.150:6789/0 conn(0x7fcb4406bae0 legacy=0x7fcb4406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44072180 con 0x7fcb44073220 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fcb34001670 con 0x7fcb44073220 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 >> v1:172.21.15.78:6789/0 conn(0x7fcb44073220 legacy=0x7fcb4406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb300039c0 0x7fcb30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300047c0 0x7fcb30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb300051d0 0x7fcb3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4406ea00 con 0x7fcb300039c0 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3000c120 con 0x7fcb300047c0 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3000c2e0 con 0x7fcb300051d0 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb300039c0 0x7fcb30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb4960d640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300047c0 0x7fcb30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb4a60f640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb300051d0 0x7fcb3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb49e0e640 1 -- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb300051d0 msgr2=0x7fcb3000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb300051d0 0x7fcb3000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb49e0e640 1 -- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300047c0 msgr2=0x7fcb30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300047c0 0x7fcb30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb49e0e640 1 -- 172.21.15.78:0/3565161236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30003520 con 0x7fcb300039c0 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb4960d640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300047c0 0x7fcb30004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.093+0000 7fcb4a60f640 1 -- 172.21.15.78:0/3565161236 reap_dead start 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb300039c0 0x7fcb30003de0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009f90 tx=0x7fcb300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b858bca4b231c0fb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4002cbf0 con 0x7fcb300039c0 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4000f040 con 0x7fcb300039c0 2024-02-19T14:59:05.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb48e0c640 1 -- 172.21.15.78:0/3565161236 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40034440 con 0x7fcb300039c0 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb4c072640 1 -- 172.21.15.78:0/3565161236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb300039c0 msgr2=0x7fcb30003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb4c072640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb300039c0 0x7fcb30003de0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009f90 tx=0x7fcb300057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.094+0000 7fcb4c072640 1 -- 172.21.15.78:0/3565161236 shutdown_connections 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.095+0000 7fcb4c072640 1 --2- 172.21.15.78:0/3565161236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb300039c0 0x7fcb30003de0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.095+0000 7fcb4c072640 1 -- 172.21.15.78:0/3565161236 >> 172.21.15.78:0/3565161236 conn(0x7fcb440c7590 msgr2=0x7fcb440c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.095+0000 7fcb4c072640 1 -- 172.21.15.78:0/3565161236 shutdown_connections 2024-02-19T14:59:05.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.095+0000 7fcb4c072640 1 -- 172.21.15.78:0/3565161236 wait complete. 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 Processor -- start 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 -- start start 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300039c0 0x7fcb44151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 0x7fcb44153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb441e9f20 0x7fcb44153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44150d20 con 0x7fcb441e8ab0 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4414b4e0 con 0x7fcb300039c0 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4c072640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44150f70 con 0x7fcb441e9f20 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4960d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 0x7fcb44153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4960d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 0x7fcb44153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57414/0 (socket says 172.21.15.78:57414) 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.096+0000 7fcb4960d640 1 -- 172.21.15.78:0/296797121 learned_addr learned my addr 172.21.15.78:0/296797121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:05.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4a60f640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb441e9f20 0x7fcb44153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300039c0 0x7fcb44151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4960d640 1 -- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb441e9f20 msgr2=0x7fcb44153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4960d640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb441e9f20 0x7fcb44153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4960d640 1 -- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300039c0 msgr2=0x7fcb44151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4960d640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300039c0 0x7fcb44151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4960d640 1 -- 172.21.15.78:0/296797121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44151250 con 0x7fcb441e8ab0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4960d640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 0x7fcb44153310 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcb34000fd0 tx=0x7fcb3400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400c630 con 0x7fcb441e8ab0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3400c790 con 0x7fcb441e8ab0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400cb00 con 0x7fcb441e8ab0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb44151530 con 0x7fcb441e8ab0 2024-02-19T14:59:05.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.097+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb440681a0 con 0x7fcb441e8ab0 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.098+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb44151a50 con 0x7fcb441e8ab0 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.098+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcb3400cc60 con 0x7fcb441e8ab0 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.099+0000 7fcb3a7fc640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb2806b810 0x7fcb2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.099+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcb34089de0 con 0x7fcb441e8ab0 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.099+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb2806b810 0x7fcb2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.099+0000 7fcb49e0e640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb2806b810 0x7fcb2806dcd0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcb40036000 tx=0x7fcb4002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.103+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcb34054a50 con 0x7fcb441e8ab0 2024-02-19T14:59:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.275+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb44074990 con 0x7fcb441e8ab0 2024-02-19T14:59:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.276+0000 7fcb3a7fc640 1 -- 172.21.15.78:0/296797121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+13162 (secure 0 0 0) 0x7fcb34058110 con 0x7fcb441e8ab0 2024-02-19T14:59:05.279 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:05.279 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":184,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:04.612245+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb2806b810 msgr2=0x7fcb2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb2806b810 0x7fcb2806dcd0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcb40036000 tx=0x7fcb4002e000 comp rx=0 tx=0).stop 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 msgr2=0x7fcb44153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 0x7fcb44153310 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fcb34000fd0 tx=0x7fcb3400eb50 comp rx=0 tx=0).stop 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 shutdown_connections 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb2806b810 0x7fcb2806dcd0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb441e8ab0 0x7fcb44153310 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb441e9f20 0x7fcb44153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 --2- 172.21.15.78:0/296797121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb300039c0 0x7fcb44151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 >> 172.21.15.78:0/296797121 conn(0x7fcb440c7590 msgr2=0x7fcb440a7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 shutdown_connections 2024-02-19T14:59:05.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.278+0000 7fcb4c072640 1 -- 172.21.15.78:0/296797121 wait complete. 2024-02-19T14:59:05.292 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:59:05.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:59:05.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 Processor -- start 2024-02-19T14:59:05.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 -- start start 2024-02-19T14:59:05.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f905c0c8e00 0x7f905c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c06d910 0x7f905c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f905c06cd20 con 0x7f905c152700 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905c1551f0 con 0x7f905c0c8e00 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f90631e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905c155490 con 0x7f905c06d910 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f905bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f905c0c8e00 0x7f905c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.392+0000 7f9060f7e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f905c152700 0x7f905c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52388/0 (socket says 172.21.15.78:52388) 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f9060f7e640 1 -- 172.21.15.78:0/2911907275 learned_addr learned my addr 172.21.15.78:0/2911907275 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f906177f640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c06d910 0x7f905c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f905bfff640 1 -- 172.21.15.78:0/2911907275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c06d910 msgr2=0x7f905c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f905bfff640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c06d910 0x7f905c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f905bfff640 1 -- 172.21.15.78:0/2911907275 >> v1:172.21.15.78:6789/0 conn(0x7f905c152700 legacy=0x7f905c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f905bfff640 1 -- 172.21.15.78:0/2911907275 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c155770 con 0x7f905c0c8e00 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f905b7fe640 1 -- 172.21.15.78:0/2911907275 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922623985 0 0) 0x7f905c06cd20 con 0x7f905c152700 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.393+0000 7f906177f640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c06d910 0x7f905c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f905bfff640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c0c8e00 0x7f905c06c2a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f90480097a0 tx=0x7f904802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be5eab11790c535 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f905b7fe640 1 -- 172.21.15.78:0/2911907275 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904802e070 con 0x7f905c0c8e00 2024-02-19T14:59:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f905b7fe640 1 -- 172.21.15.78:0/2911907275 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90480315a0 con 0x7f905c0c8e00 2024-02-19T14:59:05.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f905b7fe640 1 -- 172.21.15.78:0/2911907275 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9048031930 con 0x7f905c0c8e00 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 -- 172.21.15.78:0/2911907275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c0c8e00 msgr2=0x7f905c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c0c8e00 0x7f905c06c2a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f90480097a0 tx=0x7f904802b840 comp rx=0 tx=0).stop 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 -- 172.21.15.78:0/2911907275 shutdown_connections 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c0c8e00 0x7f905c06c2a0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 --2- 172.21.15.78:0/2911907275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c06d910 0x7f905c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 -- 172.21.15.78:0/2911907275 >> 172.21.15.78:0/2911907275 conn(0x7f905c0cac90 msgr2=0x7f905c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 -- 172.21.15.78:0/2911907275 shutdown_connections 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.394+0000 7f90631e2640 1 -- 172.21.15.78:0/2911907275 wait complete. 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.395+0000 7f90631e2640 1 Processor -- start 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.395+0000 7f90631e2640 1 -- start start 2024-02-19T14:59:05.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f90631e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f905c06d910 0x7f905c1eda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f90631e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c0c8e00 0x7f905c0c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f90631e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 0x7f905c0c1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f90631e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c157170 con 0x7f905c152700 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f90631e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905c1ee770 con 0x7f905c06d910 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f90631e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905c1593d0 con 0x7f905c0c8e00 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f906177f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 0x7f905c0c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f906177f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 0x7f905c0c1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57426/0 (socket says 172.21.15.78:57426) 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f906177f640 1 -- 172.21.15.78:0/2944369255 learned_addr learned my addr 172.21.15.78:0/2944369255 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.396+0000 7f905bfff640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c0c8e00 0x7f905c0c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f9060f7e640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f905c06d910 0x7f905c1eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f906177f640 1 -- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c0c8e00 msgr2=0x7f905c0c0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f906177f640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c0c8e00 0x7f905c0c0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f906177f640 1 -- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f905c06d910 msgr2=0x7f905c1eda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f906177f640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f905c06d910 0x7f905c1eda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f906177f640 1 -- 172.21.15.78:0/2944369255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048009400 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f906177f640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 0x7f905c0c1010 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f905400cef0 tx=0x7f905400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905400d840 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905400d9c0 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905c1ee490 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905400dd70 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.397+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905c159010 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.398+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905c1572b0 con 0x7f905c152700 2024-02-19T14:59:05.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.398+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f905401e3b0 con 0x7f905c152700 2024-02-19T14:59:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.399+0000 7f90597fa640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f904006b810 0x7f904006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.399+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f905408a9b0 con 0x7f905c152700 2024-02-19T14:59:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.399+0000 7f9060f7e640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f904006b810 0x7f904006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.399+0000 7f9060f7e640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f904006b810 0x7f904006dcd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9050000f60 tx=0x7f9050007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.403+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f905c1572b0 con 0x7f905c152700 2024-02-19T14:59:05.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.573+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f905c1572b0 con 0x7f905c152700 2024-02-19T14:59:05.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.573+0000 7f90597fa640 1 -- 172.21.15.78:0/2944369255 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7f905c1572b0 con 0x7f905c152700 2024-02-19T14:59:05.576 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:59:05.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f904006b810 msgr2=0x7f904006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f904006b810 0x7f904006dcd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9050000f60 tx=0x7f9050007c00 comp rx=0 tx=0).stop 2024-02-19T14:59:05.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 msgr2=0x7f905c0c1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 0x7f905c0c1010 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f905400cef0 tx=0x7f905400a710 comp rx=0 tx=0).stop 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 shutdown_connections 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f904006b810 0x7f904006dcd0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f905c152700 0x7f905c0c1010 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f905c0c8e00 0x7f905c0c0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 --2- 172.21.15.78:0/2944369255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f905c06d910 0x7f905c1eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 >> 172.21.15.78:0/2944369255 conn(0x7f905c0cac90 msgr2=0x7f905c0cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.575+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 shutdown_connections 2024-02-19T14:59:05.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.576+0000 7f90631e2640 1 -- 172.21.15.78:0/2944369255 wait complete. 2024-02-19T14:59:05.589 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.687+0000 7f9eedaa4640 1 Processor -- start 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.687+0000 7f9eedaa4640 1 -- start start 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9eedaa4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ee80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9eedaa4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee81551f0 con 0x7f9ee80c5730 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9eedaa4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee8155490 con 0x7f9ee80c8e00 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9eedaa4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8155770 con 0x7f9ee8152700 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9ee6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9ee80c5730 0x7f9ee806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52404/0 (socket says 172.21.15.78:52404) 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/1991916440 learned_addr learned my addr 172.21.15.78:0/1991916440 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:05.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.688+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ee80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536025356 0 0) 0x7f9ee81551f0 con 0x7f9ee80c5730 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed0003520 con 0x7f9ee80c5730 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804753038 0 0) 0x7f9ee8155490 con 0x7f9ee80c8e00 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee81551f0 con 0x7f9ee80c8e00 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3067811386 0 0) 0x7f9ed0003520 con 0x7f9ee80c5730 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 msgr2=0x7f9ee80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ee80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 >> v1:172.21.15.150:6789/0 conn(0x7f9ee80c8e00 legacy=0x7f9ee815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee8155a50 con 0x7f9ee80c5730 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9edc001670 con 0x7f9ee80c5730 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ee80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 >> v1:172.21.15.78:6789/0 conn(0x7f9ee80c5730 legacy=0x7f9ee806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ed0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ed0004070 0x7f9ed00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed0004d00 0x7f9ed0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8155770 con 0x7f9ed0004070 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed000ba30 con 0x7f9ed0004d00 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed000bbf0 con 0x7f9ee8152700 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ed0004070 0x7f9ed00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ed0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee7fff640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed0004d00 0x7f9ed0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 msgr2=0x7f9ed0003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ee8152700 0x7f9ed0003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed0004d00 msgr2=0x7f9ed0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed0004d00 0x7f9ed0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/1991916440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8009400 con 0x7f9ed0004070 2024-02-19T14:59:05.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.689+0000 7f9ee7fff640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed0004d00 0x7f9ed0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.690+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ed0004070 0x7f9ed00047c0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c190 tx=0x7f9ed00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b622f6ca8a8e770 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.690+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc005ad0 con 0x7f9ed0004070 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.690+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc005c30 con 0x7f9ed0004070 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.690+0000 7f9ee67fc640 1 -- 172.21.15.78:0/1991916440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc02c810 con 0x7f9ed0004070 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9eedaa4640 1 -- 172.21.15.78:0/1991916440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ed0004070 msgr2=0x7f9ed00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/1991916440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ed0004070 0x7f9ed00047c0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c190 tx=0x7f9ed00057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9ee7fff640 1 -- 172.21.15.78:0/1991916440 reap_dead start 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9eedaa4640 1 -- 172.21.15.78:0/1991916440 shutdown_connections 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9eedaa4640 1 -- 172.21.15.78:0/1991916440 >> 172.21.15.78:0/1991916440 conn(0x7f9ee80cac90 msgr2=0x7f9ee80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9eedaa4640 1 -- 172.21.15.78:0/1991916440 shutdown_connections 2024-02-19T14:59:05.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.691+0000 7f9eedaa4640 1 -- 172.21.15.78:0/1991916440 wait complete. 2024-02-19T14:59:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.692+0000 7f9eedaa4640 1 Processor -- start 2024-02-19T14:59:05.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.692+0000 7f9eedaa4640 1 -- start start 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ed0004070 0x7f9ee81ee020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 0x7f9ee81ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee81e7920 0x7f9ee81e7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80c47e0 con 0x7f9ee80acac0 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee81eeaa0 con 0x7f9ee81e7920 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee80acfc0 con 0x7f9ed0004070 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 0x7f9ee81ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 0x7f9ee81ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57434/0 (socket says 172.21.15.78:57434) 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/2453599541 learned_addr learned my addr 172.21.15.78:0/2453599541 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ed0004070 0x7f9ee81ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee7fff640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee81e7920 0x7f9ee81e7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ed0004070 msgr2=0x7f9ee81ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ed0004070 0x7f9ee81ee020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee81e7920 msgr2=0x7f9ee81e7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee81e7920 0x7f9ee81e7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 -- 172.21.15.78:0/2453599541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee81ebe30 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ed0004070 0x7f9ee81ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ee6ffd640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 0x7f9ee81ee560 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f9edc000fb0 tx=0x7f9ee806d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc02cc20 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc02cd80 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee81ec110 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc007270 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.693+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee80abcd0 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.694+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee8074990 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.695+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9edc032070 con 0x7f9ee80acac0 2024-02-19T14:59:05.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.695+0000 7f9ecffff640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ed406b810 0x7f9ed406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.695+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9edc03c040 con 0x7f9ee80acac0 2024-02-19T14:59:05.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.695+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ed406b810 0x7f9ed406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.696+0000 7f9ee77fe640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ed406b810 0x7f9ed406dcd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009b70 tx=0x7f9ed8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.700+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9edc076040 con 0x7f9ee80acac0 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.873+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9ee80c8770 con 0x7f9ee80acac0 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.873+0000 7f9ecffff640 1 -- 172.21.15.78:0/2453599541 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v184) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9edc079700 con 0x7f9ee80acac0 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:59:05.876 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:59:05.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:05.877 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:59:05.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ed406b810 msgr2=0x7f9ed406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ed406b810 0x7f9ed406dcd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009b70 tx=0x7f9ed8008040 comp rx=0 tx=0).stop 2024-02-19T14:59:05.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 msgr2=0x7f9ee81ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 0x7f9ee81ee560 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f9edc000fb0 tx=0x7f9ee806d6b0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 shutdown_connections 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ed406b810 0x7f9ed406dcd0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ee80acac0 0x7f9ee81ee560 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee81e7920 0x7f9ee81e7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 --2- 172.21.15.78:0/2453599541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ed0004070 0x7f9ee81ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 >> 172.21.15.78:0/2453599541 conn(0x7f9ee80cac90 msgr2=0x7f9ee814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 shutdown_connections 2024-02-19T14:59:05.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.875+0000 7f9eedaa4640 1 -- 172.21.15.78:0/2453599541 wait complete. 2024-02-19T14:59:05.889 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:59:05.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:59:05.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.989+0000 7f50269e2640 1 Processor -- start 2024-02-19T14:59:05.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.989+0000 7f50269e2640 1 -- start start 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f50269e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f502006d000 0x7f502006d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f50269e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f50269e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5020159210 con 0x7f5020152620 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f50269e2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50200c1f00 con 0x7f502006d000 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f50269e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502006d980 con 0x7f5020155010 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f501f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f502006d000 0x7f502006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f501ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5020152620 0x7f50200c6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52406/0 (socket says 172.21.15.78:52406) 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.990+0000 7f501ffff640 1 -- 172.21.15.78:0/3594956525 learned_addr learned my addr 172.21.15.78:0/3594956525 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501f7fe640 1 -- 172.21.15.78:0/3594956525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 msgr2=0x7f50201554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f5024f7f640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501f7fe640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501effd640 1 -- 172.21.15.78:0/3594956525 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040245966 0 0) 0x7f5020159210 con 0x7f5020152620 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501f7fe640 1 -- 172.21.15.78:0/3594956525 >> v1:172.21.15.78:6789/0 conn(0x7f5020152620 legacy=0x7f50200c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501f7fe640 1 -- 172.21.15.78:0/3594956525 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50200c6420 con 0x7f502006d000 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f5024f7f640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501f7fe640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f502006d000 0x7f502006d440 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f50140097a0 tx=0x7f501402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e67b8025d26e640 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501effd640 1 -- 172.21.15.78:0/3594956525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501402e070 con 0x7f502006d000 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501effd640 1 -- 172.21.15.78:0/3594956525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50140315a0 con 0x7f502006d000 2024-02-19T14:59:05.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.991+0000 7f501effd640 1 -- 172.21.15.78:0/3594956525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014031930 con 0x7f502006d000 2024-02-19T14:59:05.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 -- 172.21.15.78:0/3594956525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f502006d000 msgr2=0x7f502006d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:05.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f502006d000 0x7f502006d440 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f50140097a0 tx=0x7f501402b840 comp rx=0 tx=0).stop 2024-02-19T14:59:05.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 -- 172.21.15.78:0/3594956525 shutdown_connections 2024-02-19T14:59:05.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f502006d000 0x7f502006d440 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 --2- 172.21.15.78:0/3594956525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 -- 172.21.15.78:0/3594956525 >> 172.21.15.78:0/3594956525 conn(0x7f50200caae0 msgr2=0x7f50200caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 -- 172.21.15.78:0/3594956525 shutdown_connections 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.992+0000 7f50269e2640 1 -- 172.21.15.78:0/3594956525 wait complete. 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.993+0000 7f50269e2640 1 Processor -- start 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.993+0000 7f50269e2640 1 -- start start 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f50269e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f502006d000 0x7f5020153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f50269e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 0x7f50201eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f50269e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f50269e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5020158470 con 0x7f5020152620 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f50269e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50201eaae0 con 0x7f502006d000 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f50269e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5020157f40 con 0x7f5020155010 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 0x7f50201eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 0x7f50201eaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57454/0 (socket says 172.21.15.78:57454) 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 -- 172.21.15.78:0/81826602 learned_addr learned my addr 172.21.15.78:0/81826602 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:05.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501ffff640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f502006d000 0x7f5020153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f5024f7f640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 -- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 msgr2=0x7f50201eb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201eb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 -- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f502006d000 msgr2=0x7f5020153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f502006d000 0x7f5020153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f501f7fe640 1 -- 172.21.15.78:0/81826602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5014009400 con 0x7f5020152620 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.994+0000 7f5024f7f640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.995+0000 7f501f7fe640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 0x7f50201eaf30 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f5014034040 tx=0x7f50140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.995+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014032840 con 0x7f5020152620 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.995+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50140329c0 con 0x7f5020152620 2024-02-19T14:59:06.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.995+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014032d70 con 0x7f5020152620 2024-02-19T14:59:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.995+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5020158190 con 0x7f5020152620 2024-02-19T14:59:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.995+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50201eed40 con 0x7f5020152620 2024-02-19T14:59:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.996+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50201ee8c0 con 0x7f5020152620 2024-02-19T14:59:06.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.996+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f501404e430 con 0x7f5020152620 2024-02-19T14:59:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.997+0000 7f501cff9640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f500c06b840 0x7f500c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.997+0000 7f501ffff640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f500c06b840 0x7f500c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.997+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f50140ac090 con 0x7f5020152620 2024-02-19T14:59:06.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:05.997+0000 7f501ffff640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f500c06b840 0x7f500c06dd00 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5010000f60 tx=0x7f5010007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:06.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.001+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f50201ee8c0 con 0x7f5020152620 2024-02-19T14:59:06.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.173+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f50201eac50 con 0x7f5020152620 2024-02-19T14:59:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.619+0000 7f501cff9640 1 -- 172.21.15.78:0/81826602 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v185) v1 ==== 138+0+0 (secure 0 0 0) 0x7f50201eac50 con 0x7f5020152620 2024-02-19T14:59:06.622 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f500c06b840 msgr2=0x7f500c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f500c06b840 0x7f500c06dd00 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5010000f60 tx=0x7f5010007b70 comp rx=0 tx=0).stop 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 msgr2=0x7f50201eaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 0x7f50201eaf30 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f5014034040 tx=0x7f50140046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 shutdown_connections 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f500c06b840 0x7f500c06dd00 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5020152620 0x7f50201eaf30 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5020155010 0x7f50201eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 --2- 172.21.15.78:0/81826602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f502006d000 0x7f5020153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 >> 172.21.15.78:0/81826602 conn(0x7f50200caae0 msgr2=0x7f50200caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:06.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 shutdown_connections 2024-02-19T14:59:06.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.622+0000 7f50269e2640 1 -- 172.21.15.78:0/81826602 wait complete. 2024-02-19T14:59:06.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:59:06.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 Processor -- start 2024-02-19T14:59:06.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 -- start start 2024-02-19T14:59:06.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8f900c5220 0x7f8f900c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90155a80 0x7f8f90156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f90158aa0 con 0x7f8f90158f30 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f90152aa0 con 0x7f8f900c5220 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.744+0000 7f8f9518c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901569d0 con 0x7f8f90155a80 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90155a80 0x7f8f90156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8f900c5220 0x7f8f900c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8f59e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8f90158f30 0x7f8f900c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:60490/0 (socket says 172.21.15.78:60490) 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8f59e640 1 -- 172.21.15.78:0/2407666200 learned_addr learned my addr 172.21.15.78:0/2407666200 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8dd9b640 1 -- 172.21.15.78:0/2407666200 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406168601 0 0) 0x7f8f90158aa0 con 0x7f8f90158f30 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8dd9b640 1 -- 172.21.15.78:0/2407666200 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f78003520 con 0x7f8f90158f30 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90155a80 msgr2=0x7f8f90156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8ed9d640 1 --2- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90155a80 0x7f8f90156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/2407666200 >> v1:172.21.15.150:6789/0 conn(0x7f8f90158f30 legacy=0x7f8f900c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/2407666200 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900cafd0 con 0x7f8f900c5220 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.745+0000 7f8f8e59c640 1 --2- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90155a80 0x7f8f90156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.746+0000 7f8f8ed9d640 1 --2- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900c5220 0x7f8f900c5600 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8f840097a0 tx=0x7f8f8402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=479cb5f3e469c496 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.746+0000 7f8f8dd9b640 1 -- 172.21.15.78:0/2407666200 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84031780 con 0x7f8f900c5220 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.746+0000 7f8f8dd9b640 1 -- 172.21.15.78:0/2407666200 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f840318e0 con 0x7f8f900c5220 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.746+0000 7f8f8dd9b640 1 -- 172.21.15.78:0/2407666200 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84031c50 con 0x7f8f900c5220 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.746+0000 7f8f9518c640 1 -- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900c5220 msgr2=0x7f8f900c5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.746+0000 7f8f9518c640 1 --2- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900c5220 0x7f8f900c5600 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8f840097a0 tx=0x7f8f8402b820 comp rx=0 tx=0).stop 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.747+0000 7f8f9518c640 1 -- 172.21.15.78:0/2407666200 shutdown_connections 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.747+0000 7f8f9518c640 1 --2- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90155a80 0x7f8f90156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.747+0000 7f8f9518c640 1 --2- 172.21.15.78:0/2407666200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900c5220 0x7f8f900c5600 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.747+0000 7f8f9518c640 1 -- 172.21.15.78:0/2407666200 >> 172.21.15.78:0/2407666200 conn(0x7f8f90072790 msgr2=0x7f8f90072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.747+0000 7f8f9518c640 1 -- 172.21.15.78:0/2407666200 shutdown_connections 2024-02-19T14:59:06.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.747+0000 7f8f9518c640 1 -- 172.21.15.78:0/2407666200 wait complete. 2024-02-19T14:59:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.748+0000 7f8f9518c640 1 Processor -- start 2024-02-19T14:59:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.748+0000 7f8f9518c640 1 -- start start 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 0x7f8f901eec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f90155a80 0x7f8f900c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90158f30 0x7f8f9014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f90155520 con 0x7f8f900c5220 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901ef190 con 0x7f8f90155a80 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f900ac700 con 0x7f8f90158f30 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 0x7f8f901eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 0x7f8f901eec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57466/0 (socket says 172.21.15.78:57466) 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/3269061411 learned_addr learned my addr 172.21.15.78:0/3269061411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8f59e640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90158f30 0x7f8f9014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8e59c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f90155a80 0x7f8f900c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90158f30 msgr2=0x7f8f9014ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90158f30 0x7f8f9014ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f90155a80 msgr2=0x7f8f900c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f90155a80 0x7f8f900c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 -- 172.21.15.78:0/3269061411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80009400 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f8ed9d640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 0x7f8f901eec50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f8f84034040 tx=0x7f8f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84032820 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f84032980 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f84009400 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84032cf0 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.749+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f900accc0 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.750+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f900c6290 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.751+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8f8404e440 con 0x7f8f900c5220 2024-02-19T14:59:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.751+0000 7f8f777fe640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f7c06b810 0x7f8f7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.751+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f8f840ac110 con 0x7f8f900c5220 2024-02-19T14:59:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.751+0000 7f8f8e59c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f7c06b810 0x7f8f7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.752+0000 7f8f8e59c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f7c06b810 0x7f8f7c06dcd0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8f9014f8b0 tx=0x7f8f80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.756+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8f84076c10 con 0x7f8f900c5220 2024-02-19T14:59:06.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:06.928+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8f90074990 con 0x7f8f900c5220 2024-02-19T14:59:07.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.621+0000 7f8f777fe640 1 -- 172.21.15.78:0/3269061411 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v186) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8f8407a2d0 con 0x7f8f900c5220 2024-02-19T14:59:07.624 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f7c06b810 msgr2=0x7f8f7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f7c06b810 0x7f8f7c06dcd0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8f9014f8b0 tx=0x7f8f80008040 comp rx=0 tx=0).stop 2024-02-19T14:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 msgr2=0x7f8f901eec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 0x7f8f901eec50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f8f84034040 tx=0x7f8f840046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 shutdown_connections 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f7c06b810 0x7f8f7c06dcd0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f90158f30 0x7f8f9014ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f90155a80 0x7f8f900c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 --2- 172.21.15.78:0/3269061411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f900c5220 0x7f8f901eec50 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 >> 172.21.15.78:0/3269061411 conn(0x7f8f90072790 msgr2=0x7f8f90156200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 shutdown_connections 2024-02-19T14:59:07.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.624+0000 7f8f9518c640 1 -- 172.21.15.78:0/3269061411 wait complete. 2024-02-19T14:59:07.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:59:07.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.752+0000 7f16ad1e4640 1 Processor -- start 2024-02-19T14:59:07.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.752+0000 7f16ad1e4640 1 -- start start 2024-02-19T14:59:07.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.752+0000 7f16ad1e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 0x7f16a8155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16ad1e4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a80717b0 con 0x7f16a80c2f70 2024-02-19T14:59:07.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16ad1e4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a8071a00 con 0x7f16a80c5920 2024-02-19T14:59:07.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16ad1e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c7560 con 0x7f16a80c6a50 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f169e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f16a80c2f70 0x7f16a806c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39914/0 (socket says 172.21.15.78:39914) 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f169e59c640 1 -- 172.21.15.78:0/1563964988 learned_addr learned my addr 172.21.15.78:0/1563964988 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a6d9d640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 0x7f16a8155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974918728 0 0) 0x7f16a80717b0 con 0x7f16a80c2f70 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1694003520 con 0x7f16a80c2f70 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058859673 0 0) 0x7f1694003520 con 0x7f16a80c2f70 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 msgr2=0x7f16a8155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 0x7f16a8155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 >> v1:172.21.15.150:6789/0 conn(0x7f16a80c5920 legacy=0x7f16a806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a80c8030 con 0x7f16a80c2f70 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1690001670 con 0x7f16a80c2f70 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 >> v1:172.21.15.78:6789/0 conn(0x7f16a80c2f70 legacy=0x7f16a806c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16940039a0 0x7f1694003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 0x7f16940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f1694005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c7560 con 0x7f1694004d00 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169400bb40 con 0x7f16940039a0 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169400bd00 con 0x7f16a80c6a50 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a759e640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f1694005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.753+0000 7f16a6d9d640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16940039a0 0x7f1694003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f169e59c640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 0x7f16940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f16a759e640 1 -- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 msgr2=0x7f16940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f16a759e640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a80c6a50 0x7f16940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f16a759e640 1 -- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16940039a0 msgr2=0x7f1694003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f16a759e640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16940039a0 0x7f1694003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f16a759e640 1 -- 172.21.15.78:0/1563964988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1694003520 con 0x7f1694004d00 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.754+0000 7f16a759e640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f1694005180 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f16a0001e50 tx=0x7f16940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a1b520787174d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:07.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0003b70 con 0x7f1694004d00 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0003cd0 con 0x7f1694004d00 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16a659c640 1 -- 172.21.15.78:0/1563964988 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0005310 con 0x7f1694004d00 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16ad1e4640 1 -- 172.21.15.78:0/1563964988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 msgr2=0x7f1694005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/1563964988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f1694005180 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f16a0001e50 tx=0x7f16940057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16a759e640 1 -- 172.21.15.78:0/1563964988 reap_dead start 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16ad1e4640 1 -- 172.21.15.78:0/1563964988 shutdown_connections 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16ad1e4640 1 -- 172.21.15.78:0/1563964988 >> 172.21.15.78:0/1563964988 conn(0x7f16a806b700 msgr2=0x7f16a80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16ad1e4640 1 -- 172.21.15.78:0/1563964988 shutdown_connections 2024-02-19T14:59:07.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.755+0000 7f16ad1e4640 1 -- 172.21.15.78:0/1563964988 wait complete. 2024-02-19T14:59:07.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.756+0000 7f16ad1e4640 1 Processor -- start 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 -- start start 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f16a809a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a8158fd0 0x7f16a8158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16a809aa30 0x7f16a8156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a809bd70 con 0x7f1694004d00 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a8099c70 con 0x7f16a809aa30 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16ad1e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a8158c80 con 0x7f16a8158fd0 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16a6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f16a809a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16a6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f16a809a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32884/0 (socket says 172.21.15.78:32884) 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16a6d9d640 1 -- 172.21.15.78:0/795114462 learned_addr learned my addr 172.21.15.78:0/795114462 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f16a759e640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16a809aa30 0x7f16a8156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.757+0000 7f169e59c640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a8158fd0 0x7f16a8158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a6d9d640 1 -- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a8158fd0 msgr2=0x7f16a8158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a6d9d640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a8158fd0 0x7f16a8158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a6d9d640 1 -- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16a809aa30 msgr2=0x7f16a8156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a6d9d640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16a809aa30 0x7f16a8156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a6d9d640 1 -- 172.21.15.78:0/795114462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a80a9460 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a759e640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16a809aa30 0x7f16a8156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16a6d9d640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f16a809a100 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f16980119a0 tx=0x7f1698011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169800fa90 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169800fbf0 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a80a9740 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698014440 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.758+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a809aec0 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.759+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a8099a60 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.759+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f16980145a0 con 0x7f1694004d00 2024-02-19T14:59:07.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.760+0000 7f169ffff640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f168806b840 0x7f168806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:07.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.760+0000 7f169e59c640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f168806b840 0x7f168806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:07.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.760+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f169808aaa0 con 0x7f1694004d00 2024-02-19T14:59:07.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.760+0000 7f169e59c640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f168806b840 0x7f168806dd00 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f16a81593d0 tx=0x7f1690009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:07.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.764+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1698055400 con 0x7f1694004d00 2024-02-19T14:59:07.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:07.974+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f16a8074990 con 0x7f1694004d00 2024-02-19T14:59:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.648+0000 7f169ffff640 1 -- 172.21.15.78:0/795114462 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 29 and data pool 30 v163) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1698058ac0 con 0x7f1694004d00 2024-02-19T14:59:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 29 and data pool 30 2024-02-19T14:59:08.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.658+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f168806b840 msgr2=0x7f168806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.658+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f168806b840 0x7f168806dd00 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f16a81593d0 tx=0x7f1690009040 comp rx=0 tx=0).stop 2024-02-19T14:59:08.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.658+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 msgr2=0x7f16a809a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.658+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f16a809a100 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f16980119a0 tx=0x7f1698011e60 comp rx=0 tx=0).stop 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.660+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 shutdown_connections 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.660+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f168806b840 0x7f168806dd00 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.660+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16a809aa30 0x7f16a8156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.660+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16a8158fd0 0x7f16a8158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.660+0000 7f16ad1e4640 1 --2- 172.21.15.78:0/795114462 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1694004d00 0x7f16a809a100 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.660+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 >> 172.21.15.78:0/795114462 conn(0x7f16a806b700 msgr2=0x7f16a806f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.661+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 shutdown_connections 2024-02-19T14:59:08.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.661+0000 7f16ad1e4640 1 -- 172.21.15.78:0/795114462 wait complete. 2024-02-19T14:59:08.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:08.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.778+0000 7f18715f6640 1 Processor -- start 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.778+0000 7f18715f6640 1 -- start start 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f18715f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 0x7f186c0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f18715f6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c158c40 con 0x7f186c0bdf20 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f18715f6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c0c5a60 con 0x7f186c0c55f0 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f18715f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0c8de0 con 0x7f186c0ac390 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f186a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f186c0bdf20 0x7f186c158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39928/0 (socket says 172.21.15.78:39928) 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f186a7fc640 1 -- 172.21.15.78:0/250530170 learned_addr learned my addr 172.21.15.78:0/250530170 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f186affd640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 0x7f186c0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179911898 0 0) 0x7f186c158c40 con 0x7f186c0bdf20 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1854003520 con 0x7f186c0bdf20 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714832753 0 0) 0x7f1854003520 con 0x7f186c0bdf20 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 msgr2=0x7f186c0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 0x7f186c0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 >> v1:172.21.15.150:6789/0 conn(0x7f186c0c55f0 legacy=0x7f186c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c155a60 con 0x7f186c0bdf20 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f1860001670 con 0x7f186c0bdf20 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 >> v1:172.21.15.78:6789/0 conn(0x7f186c0bdf20 legacy=0x7f186c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 0x7f18540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.779+0000 7f1869ffb640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854003f20 0x7f18540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f1869ffb640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1854004d00 0x7f1854005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0c8de0 con 0x7f1854004d00 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185400bbc0 con 0x7f1854003f20 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185400bd80 con 0x7f186c0ac390 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186b7fe640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1854004d00 0x7f1854005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186affd640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 0x7f18540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186a7fc640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854003f20 0x7f18540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186b7fe640 1 -- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 msgr2=0x7f18540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186b7fe640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c0ac390 0x7f18540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186b7fe640 1 -- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854003f20 msgr2=0x7f18540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186b7fe640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854003f20 0x7f18540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.780+0000 7f186b7fe640 1 -- 172.21.15.78:0/250530170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854003520 con 0x7f1854004d00 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f186b7fe640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1854004d00 0x7f1854005180 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f1864001e50 tx=0x7f18540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c182d103e37cae server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864003b70 con 0x7f1854004d00 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864003cd0 con 0x7f1854004d00 2024-02-19T14:59:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f1869ffb640 1 -- 172.21.15.78:0/250530170 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864005310 con 0x7f1854004d00 2024-02-19T14:59:08.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f18715f6640 1 -- 172.21.15.78:0/250530170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1854004d00 msgr2=0x7f1854005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f18715f6640 1 --2- 172.21.15.78:0/250530170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1854004d00 0x7f1854005180 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f1864001e50 tx=0x7f18540057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f186b7fe640 1 -- 172.21.15.78:0/250530170 reap_dead start 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f18715f6640 1 -- 172.21.15.78:0/250530170 shutdown_connections 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f18715f6640 1 -- 172.21.15.78:0/250530170 >> 172.21.15.78:0/250530170 conn(0x7f186c0ca890 msgr2=0x7f186c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f18715f6640 1 -- 172.21.15.78:0/250530170 shutdown_connections 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.781+0000 7f18715f6640 1 -- 172.21.15.78:0/250530170 wait complete. 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 Processor -- start 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 -- start start 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 0x7f186c1eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c151b80 0x7f186c151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f186c0c2320 0x7f186c0c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0c3690 con 0x7f186c0af390 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0c18d0 con 0x7f186c0c2320 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f18715f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f186c0c1e90 con 0x7f186c151b80 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f186affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 0x7f186c1eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f186affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 0x7f186c1eeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32904/0 (socket says 172.21.15.78:32904) 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.783+0000 7f186affd640 1 -- 172.21.15.78:0/668330857 learned_addr learned my addr 172.21.15.78:0/668330857 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186a7fc640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c151b80 0x7f186c151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186b7fe640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f186c0c2320 0x7f186c0c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186affd640 1 -- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c151b80 msgr2=0x7f186c151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186affd640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c151b80 0x7f186c151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186affd640 1 -- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f186c0c2320 msgr2=0x7f186c0c27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186affd640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f186c0c2320 0x7f186c0c27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186affd640 1 -- 172.21.15.78:0/668330857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c1eeff0 con 0x7f186c0af390 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186a7fc640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c151b80 0x7f186c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f186affd640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 0x7f186c1eeab0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f185c0119a0 tx=0x7f185c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c00fa90 con 0x7f186c0af390 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c00fbf0 con 0x7f186c0af390 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c014440 con 0x7f186c0af390 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186c152520 con 0x7f186c0af390 2024-02-19T14:59:08.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.784+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186c0c0690 con 0x7f186c0af390 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.785+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c0c4bd0 con 0x7f186c0af390 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.785+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f185c0145a0 con 0x7f186c0af390 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.786+0000 7f18537fe640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f185806b840 0x7f185806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.786+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f185c08ab40 con 0x7f186c0af390 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.786+0000 7f186a7fc640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f185806b840 0x7f185806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.786+0000 7f186a7fc640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f185806b840 0x7f185806dd00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f186c0c49f0 tx=0x7f1860009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:08.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.790+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f185c0553d0 con 0x7f186c0af390 2024-02-19T14:59:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.961+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f186c074990 con 0x7f186c0af390 2024-02-19T14:59:08.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.963+0000 7f18537fe640 1 -- 172.21.15.78:0/668330857 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+16885 (secure 0 0 0) 0x7f185c058a90 con 0x7f186c0af390 2024-02-19T14:59:08.966 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:08.966 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":187,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:07.976102+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:06.192227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:06.947291+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.964+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f185806b840 msgr2=0x7f185806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.964+0000 7f18715f6640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f185806b840 0x7f185806dd00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f186c0c49f0 tx=0x7f1860009040 comp rx=0 tx=0).stop 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.964+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 msgr2=0x7f186c1eeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.964+0000 7f18715f6640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 0x7f186c1eeab0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f185c0119a0 tx=0x7f185c011e60 comp rx=0 tx=0).stop 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 shutdown_connections 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f185806b840 0x7f185806dd00 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f186c0c2320 0x7f186c0c27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f186c151b80 0x7f186c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 --2- 172.21.15.78:0/668330857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f186c0af390 0x7f186c1eeab0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 >> 172.21.15.78:0/668330857 conn(0x7f186c0ca890 msgr2=0x7f186c156110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 shutdown_connections 2024-02-19T14:59:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:08.965+0000 7f18715f6640 1 -- 172.21.15.78:0/668330857 wait complete. 2024-02-19T14:59:08.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:09.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.075+0000 7f199f90d640 1 Processor -- start 2024-02-19T14:59:09.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.075+0000 7f199f90d640 1 -- start start 2024-02-19T14:59:09.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199f90d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19980ac390 0x7f19980ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199f90d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19980c55f0 0x7f199815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199f90d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1998158c40 con 0x7f19980bdf20 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199f90d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980c8de0 con 0x7f19980c55f0 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199f90d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980c5a30 con 0x7f19980ac390 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199deaa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f19980c55f0 0x7f199815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199cea8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f19980bdf20 0x7f1998158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39934/0 (socket says 172.21.15.78:39934) 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.076+0000 7f199cea8640 1 -- 172.21.15.78:0/145309154 learned_addr learned my addr 172.21.15.78:0/145309154 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f199deaa640 1 -- 172.21.15.78:0/145309154 >> v2:172.21.15.150:3300/0 conn(0x7f19980ac390 msgr2=0x7f19980ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f199d6a9640 1 --2- 172.21.15.78:0/145309154 >> v2:172.21.15.150:3300/0 conn(0x7f19980ac390 0x7f19980ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f198ffff640 1 -- 172.21.15.78:0/145309154 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363183669 0 0) 0x7f1998158c40 con 0x7f19980bdf20 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f199deaa640 1 --2- 172.21.15.78:0/145309154 >> v2:172.21.15.150:3300/0 conn(0x7f19980ac390 0x7f19980ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f199deaa640 1 -- 172.21.15.78:0/145309154 >> v1:172.21.15.78:6789/0 conn(0x7f19980bdf20 legacy=0x7f1998158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f199deaa640 1 -- 172.21.15.78:0/145309154 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1998155a60 con 0x7f19980c55f0 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f199deaa640 1 --2- 172.21.15.78:0/145309154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980c55f0 0x7f199815f8c0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f19980ad2a0 tx=0x7f199002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b984c3951078938e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f198ffff640 1 -- 172.21.15.78:0/145309154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990031710 con 0x7f19980c55f0 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f198ffff640 1 -- 172.21.15.78:0/145309154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199002f050 con 0x7f19980c55f0 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.077+0000 7f198ffff640 1 -- 172.21.15.78:0/145309154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990031a90 con 0x7f19980c55f0 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 -- 172.21.15.78:0/145309154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980c55f0 msgr2=0x7f199815f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 --2- 172.21.15.78:0/145309154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980c55f0 0x7f199815f8c0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f19980ad2a0 tx=0x7f199002b7b0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 -- 172.21.15.78:0/145309154 shutdown_connections 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 --2- 172.21.15.78:0/145309154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980c55f0 0x7f199815f8c0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 --2- 172.21.15.78:0/145309154 >> v2:172.21.15.150:3300/0 conn(0x7f19980ac390 0x7f19980ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 -- 172.21.15.78:0/145309154 >> 172.21.15.78:0/145309154 conn(0x7f19980ca890 msgr2=0x7f19980caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:09.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 -- 172.21.15.78:0/145309154 shutdown_connections 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.078+0000 7f199f90d640 1 -- 172.21.15.78:0/145309154 wait complete. 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.079+0000 7f199f90d640 1 Processor -- start 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.079+0000 7f199f90d640 1 -- start start 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199f90d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980ac390 0x7f19980c3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199f90d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 0x7f19980bf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199f90d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19980c55f0 0x7f19980bfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199f90d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19980c0050 con 0x7f19980bdf20 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199f90d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19981ea780 con 0x7f19980ac390 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199f90d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19981eaa60 con 0x7f19980c55f0 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 0x7f19980bf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 0x7f19980bf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32922/0 (socket says 172.21.15.78:32922) 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 -- 172.21.15.78:0/2846725382 learned_addr learned my addr 172.21.15.78:0/2846725382 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:09.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199d6a9640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980ac390 0x7f19980c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 -- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19980c55f0 msgr2=0x7f19980bfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199deaa640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19980c55f0 0x7f19980bfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19980c55f0 0x7f19980bfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 -- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980ac390 msgr2=0x7f19980c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980ac390 0x7f19980c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 -- 172.21.15.78:0/2846725382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990009400 con 0x7f19980bdf20 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199d6a9640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980ac390 0x7f19980c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f199cea8640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 0x7f19980bf5d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f198800c4b0 tx=0x7f198800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.080+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19880074e0 con 0x7f19980bdf20 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.081+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1988007660 con 0x7f19980bdf20 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.081+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988004a20 con 0x7f19980bdf20 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.081+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19981ead40 con 0x7f19980bdf20 2024-02-19T14:59:09.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.081+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19980c3e40 con 0x7f19980bdf20 2024-02-19T14:59:09.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.082+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19981eda20 con 0x7f19980bdf20 2024-02-19T14:59:09.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.082+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1988004b80 con 0x7f19980bdf20 2024-02-19T14:59:09.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.082+0000 7f198dffb640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197c06b810 0x7f197c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.083+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f19880898e0 con 0x7f19980bdf20 2024-02-19T14:59:09.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.083+0000 7f199d6a9640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197c06b810 0x7f197c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.083+0000 7f199d6a9640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197c06b810 0x7f197c06dcd0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1994007f50 tx=0x7f19940074a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.087+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f19981eda20 con 0x7f19980bdf20 2024-02-19T14:59:09.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.260+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19981eda20 con 0x7f19980bdf20 2024-02-19T14:59:09.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.261+0000 7f198dffb640 1 -- 172.21.15.78:0/2846725382 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+16885 (secure 0 0 0) 0x7f19981eda20 con 0x7f19980bdf20 2024-02-19T14:59:09.264 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:09.264 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":187,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:07.976102+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:06.192227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:06.947291+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:09.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197c06b810 msgr2=0x7f197c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197c06b810 0x7f197c06dcd0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1994007f50 tx=0x7f19940074a0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 msgr2=0x7f19980bf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 0x7f19980bf5d0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f198800c4b0 tx=0x7f198800c970 comp rx=0 tx=0).stop 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 shutdown_connections 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f197c06b810 0x7f197c06dcd0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19980bdf20 0x7f19980bf5d0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19980c55f0 0x7f19980bfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 --2- 172.21.15.78:0/2846725382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980ac390 0x7f19980c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 >> 172.21.15.78:0/2846725382 conn(0x7f19980ca890 msgr2=0x7f199806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.263+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 shutdown_connections 2024-02-19T14:59:09.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.264+0000 7f199f90d640 1 -- 172.21.15.78:0/2846725382 wait complete. 2024-02-19T14:59:09.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:59:09.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.377+0000 7eff6a074640 1 Processor -- start 2024-02-19T14:59:09.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.377+0000 7eff6a074640 1 -- start start 2024-02-19T14:59:09.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff6a074640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff6406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff6a074640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6406b620 con 0x7eff6406bae0 2024-02-19T14:59:09.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff6a074640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff64157230 con 0x7eff6414d110 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff6a074640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6406ea00 con 0x7eff64073220 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff63fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7eff6406bae0 0x7eff6406f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39942/0 (socket says 172.21.15.78:39942) 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff63fff640 1 -- 172.21.15.78:0/4008047504 learned_addr learned my addr 172.21.15.78:0/4008047504 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff62ffd640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff6406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932262049 0 0) 0x7eff6406b620 con 0x7eff6406bae0 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.378+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff4c003520 con 0x7eff6406bae0 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283058966 0 0) 0x7eff4c003520 con 0x7eff6406bae0 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 msgr2=0x7eff6406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff6406ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 >> v1:172.21.15.150:6789/0 conn(0x7eff6414d110 legacy=0x7eff640ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff64072180 con 0x7eff6406bae0 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7eff5c0019f0 con 0x7eff6406bae0 2024-02-19T14:59:09.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 >> v1:172.21.15.78:6789/0 conn(0x7eff6406bae0 legacy=0x7eff6406f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff4c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff4c0047c0 0x7eff4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6406ea00 con 0x7eff4c0039a0 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c00bab0 con 0x7eff4c0047c0 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c00bc70 con 0x7eff64073220 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff637fe640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff4c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff62ffd640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff4c0047c0 0x7eff4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff63fff640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff637fe640 1 -- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 msgr2=0x7eff4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff637fe640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff637fe640 1 -- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff4c0047c0 msgr2=0x7eff4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff637fe640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff4c0047c0 0x7eff4c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff637fe640 1 -- 172.21.15.78:0/4008047504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c003520 con 0x7eff4c0039a0 2024-02-19T14:59:09.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.379+0000 7eff63fff640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff64073220 0x7eff4c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff637fe640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff4c003dc0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7eff54001750 tx=0x7eff4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee6a44f34c1f7e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff540052d0 con 0x7eff4c0039a0 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5402e050 con 0x7eff4c0039a0 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff627fc640 1 -- 172.21.15.78:0/4008047504 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff540035f0 con 0x7eff4c0039a0 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff6a074640 1 -- 172.21.15.78:0/4008047504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 msgr2=0x7eff4c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff6a074640 1 --2- 172.21.15.78:0/4008047504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff4c003dc0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7eff54001750 tx=0x7eff4c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff63fff640 1 -- 172.21.15.78:0/4008047504 reap_dead start 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff6a074640 1 -- 172.21.15.78:0/4008047504 shutdown_connections 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff6a074640 1 -- 172.21.15.78:0/4008047504 >> 172.21.15.78:0/4008047504 conn(0x7eff640c7590 msgr2=0x7eff640c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff6a074640 1 -- 172.21.15.78:0/4008047504 shutdown_connections 2024-02-19T14:59:09.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.380+0000 7eff6a074640 1 -- 172.21.15.78:0/4008047504 wait complete. 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 Processor -- start 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 -- start start 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff641eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff64145a30 0x7eff64145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff6414d760 0x7eff6414dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6414e820 con 0x7eff4c0039a0 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff641ebbd0 con 0x7eff64145a30 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff6a074640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff641463f0 con 0x7eff6414d760 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff637fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff641eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff637fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff641eb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32952/0 (socket says 172.21.15.78:32952) 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.382+0000 7eff637fe640 1 -- 172.21.15.78:0/3410367570 learned_addr learned my addr 172.21.15.78:0/3410367570 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff637fe640 1 -- 172.21.15.78:0/3410367570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff6414d760 msgr2=0x7eff6414dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff637fe640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff6414d760 0x7eff6414dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff637fe640 1 -- 172.21.15.78:0/3410367570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff64145a30 msgr2=0x7eff64145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff637fe640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff64145a30 0x7eff64145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff637fe640 1 -- 172.21.15.78:0/3410367570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff64150f00 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff637fe640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff641eb690 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7eff54008fd0 tx=0x7eff54001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54007a30 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff540067d0 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6414e120 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54006b80 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.383+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff64151e30 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.384+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff64156e60 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.384+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7eff54006ce0 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.385+0000 7eff4bfff640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff5006b840 0x7eff5006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.385+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6101+0+0 (secure 0 0 0) 0x7eff540abf90 con 0x7eff4c0039a0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.385+0000 7eff62ffd640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff5006b840 0x7eff5006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.385+0000 7eff62ffd640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff5006b840 0x7eff5006dd00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7eff64151370 tx=0x7eff58006c30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.389+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7eff540dfdc0 con 0x7eff4c0039a0 2024-02-19T14:59:09.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.599+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7eff6406f9f0 con 0x7eff4c0039a0 2024-02-19T14:59:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.648+0000 7eff4bfff640 1 -- 172.21.15.78:0/3410367570 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v164) v1 ==== 118+0+0 (secure 0 0 0) 0x7eff5404b0b0 con 0x7eff4c0039a0 2024-02-19T14:59:09.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.650+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff5006b840 msgr2=0x7eff5006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.650+0000 7eff6a074640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff5006b840 0x7eff5006dd00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7eff64151370 tx=0x7eff58006c30 comp rx=0 tx=0).stop 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.650+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 msgr2=0x7eff641eb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.650+0000 7eff6a074640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff641eb690 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7eff54008fd0 tx=0x7eff54001120 comp rx=0 tx=0).stop 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 shutdown_connections 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff5006b840 0x7eff5006dd00 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff6414d760 0x7eff6414dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff64145a30 0x7eff64145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 --2- 172.21.15.78:0/3410367570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff4c0039a0 0x7eff641eb690 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 >> 172.21.15.78:0/3410367570 conn(0x7eff640c7590 msgr2=0x7eff640c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 shutdown_connections 2024-02-19T14:59:09.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.651+0000 7eff6a074640 1 -- 172.21.15.78:0/3410367570 wait complete. 2024-02-19T14:59:09.666 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:59:09.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:59:09.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.766+0000 7f26fe8ae640 1 Processor -- start 2024-02-19T14:59:09.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.766+0000 7f26fe8ae640 1 -- start start 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fe8ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26f806d550 0x7f26f80c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fe8ae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26f81588a0 0x7f26f8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fe8ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f80c6af0 con 0x7f26f8155f90 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fe8ae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f80c7270 con 0x7f26f81588a0 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fe8ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f80c7940 con 0x7f26f806d550 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26f81588a0 0x7f26f8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f26f81588a0 0x7f26f8158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32958/0 (socket says 172.21.15.78:32958) 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 -- 172.21.15.78:0/3084473829 learned_addr learned my addr 172.21.15.78:0/3084473829 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26f77fe640 1 --2- 172.21.15.78:0/3084473829 >> v2:172.21.15.150:3300/0 conn(0x7f26f806d550 0x7f26f80c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 -- 172.21.15.78:0/3084473829 >> v2:172.21.15.150:3300/0 conn(0x7f26f806d550 msgr2=0x7f26f80c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:09.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 --2- 172.21.15.78:0/3084473829 >> v2:172.21.15.150:3300/0 conn(0x7f26f806d550 0x7f26f80c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 -- 172.21.15.78:0/3084473829 >> v1:172.21.15.150:6789/0 conn(0x7f26f8155f90 legacy=0x7f26f80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.767+0000 7f26fce4b640 1 -- 172.21.15.78:0/3084473829 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f80c7c20 con 0x7f26f81588a0 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.768+0000 7f26fce4b640 1 --2- 172.21.15.78:0/3084473829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f81588a0 0x7f26f8158ca0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f26f80c2b60 tx=0x7f26f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781bd65776bc7f3d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.768+0000 7f26f6ffd640 1 -- 172.21.15.78:0/3084473829 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f002ca80 con 0x7f26f81588a0 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.768+0000 7f26f6ffd640 1 -- 172.21.15.78:0/3084473829 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f002cbe0 con 0x7f26f81588a0 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.768+0000 7f26f6ffd640 1 -- 172.21.15.78:0/3084473829 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00343f0 con 0x7f26f81588a0 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 -- 172.21.15.78:0/3084473829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f81588a0 msgr2=0x7f26f8158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/3084473829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f81588a0 0x7f26f8158ca0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f26f80c2b60 tx=0x7f26f002bee0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 -- 172.21.15.78:0/3084473829 shutdown_connections 2024-02-19T14:59:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/3084473829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f81588a0 0x7f26f8158ca0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/3084473829 >> v2:172.21.15.150:3300/0 conn(0x7f26f806d550 0x7f26f80c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 -- 172.21.15.78:0/3084473829 >> 172.21.15.78:0/3084473829 conn(0x7f26f806b700 msgr2=0x7f26f80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 -- 172.21.15.78:0/3084473829 shutdown_connections 2024-02-19T14:59:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.769+0000 7f26fe8ae640 1 -- 172.21.15.78:0/3084473829 wait complete. 2024-02-19T14:59:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.770+0000 7f26fe8ae640 1 Processor -- start 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.770+0000 7f26fe8ae640 1 -- start start 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 0x7f26f80acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26f8155f90 0x7f26f814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26f81588a0 0x7f26f81eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f80abaa0 con 0x7f26f806d550 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8156650 con 0x7f26f81588a0 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8156370 con 0x7f26f8155f90 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 0x7f26f80acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 0x7f26f80acb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32968/0 (socket says 172.21.15.78:32968) 2024-02-19T14:59:09.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 -- 172.21.15.78:0/2832209345 learned_addr learned my addr 172.21.15.78:0/2832209345 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f77fe640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26f8155f90 0x7f26f814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fce4b640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26f81588a0 0x7f26f81eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 -- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26f8155f90 msgr2=0x7f26f814dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26f8155f90 0x7f26f814dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 -- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26f81588a0 msgr2=0x7f26f81eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26f81588a0 0x7f26f81eafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 -- 172.21.15.78:0/2832209345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f0009b30 con 0x7f26f806d550 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f7fff640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 0x7f26f80acb90 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f26e800c400 tx=0x7f26e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e80052d0 con 0x7f26f806d550 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e8005450 con 0x7f26f806d550 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f81ee760 con 0x7f26f806d550 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8007260 con 0x7f26f806d550 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.771+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f81eaa90 con 0x7f26f806d550 2024-02-19T14:59:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.772+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f8074990 con 0x7f26f806d550 2024-02-19T14:59:09.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.773+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f26e80073c0 con 0x7f26f806d550 2024-02-19T14:59:09.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.773+0000 7f26f4ff9640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26e406b810 0x7f26e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:09.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.773+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26e80896d0 con 0x7f26f806d550 2024-02-19T14:59:09.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.773+0000 7f26f77fe640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26e406b810 0x7f26e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:09.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.773+0000 7f26f77fe640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26e406b810 0x7f26e406dcd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009b40 tx=0x7f26ec009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:09.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.777+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f26e8054010 con 0x7f26f806d550 2024-02-19T14:59:09.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:09.984+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f26f80c75a0 con 0x7f26f806d550 2024-02-19T14:59:10.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.661+0000 7f26f4ff9640 1 -- 172.21.15.78:0/2832209345 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v165) v1 ==== 122+0+0 (secure 0 0 0) 0x7f26e80576d0 con 0x7f26f806d550 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26e406b810 msgr2=0x7f26e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26e406b810 0x7f26e406dcd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009b40 tx=0x7f26ec009190 comp rx=0 tx=0).stop 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 msgr2=0x7f26f80acb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 0x7f26f80acb90 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f26e800c400 tx=0x7f26e800c8c0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 shutdown_connections 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f26e406b810 0x7f26e406dcd0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26f81588a0 0x7f26f81eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26f8155f90 0x7f26f814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.663+0000 7f26fe8ae640 1 --2- 172.21.15.78:0/2832209345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26f806d550 0x7f26f80acb90 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.664+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 >> 172.21.15.78:0/2832209345 conn(0x7f26f806b700 msgr2=0x7f26f8154590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.664+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 shutdown_connections 2024-02-19T14:59:10.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.664+0000 7f26fe8ae640 1 -- 172.21.15.78:0/2832209345 wait complete. 2024-02-19T14:59:10.679 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:59:10.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:10.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.779+0000 7ff15b143640 1 Processor -- start 2024-02-19T14:59:10.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.779+0000 7ff15b143640 1 -- start start 2024-02-19T14:59:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff15b143640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1540c78a0 0x7ff1540c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff15b143640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff15414ef50 0x7ff15414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff15b143640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540c35c0 con 0x7ff1540727d0 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff15b143640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15414f920 con 0x7ff1540c78a0 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff15b143640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540c38a0 con 0x7ff15414ef50 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff158edf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1540c78a0 0x7ff1540c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff158edf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1540c78a0 0x7ff1540c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32972/0 (socket says 172.21.15.78:32972) 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff158edf640 1 -- 172.21.15.78:0/2689482852 learned_addr learned my addr 172.21.15.78:0/2689482852 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.780+0000 7ff153fff640 1 --2- 172.21.15.78:0/2689482852 >> v2:172.21.15.150:3300/0 conn(0x7ff15414ef50 0x7ff15414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff158edf640 1 -- 172.21.15.78:0/2689482852 >> v2:172.21.15.150:3300/0 conn(0x7ff15414ef50 msgr2=0x7ff15414f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff158edf640 1 --2- 172.21.15.78:0/2689482852 >> v2:172.21.15.150:3300/0 conn(0x7ff15414ef50 0x7ff15414f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff158edf640 1 -- 172.21.15.78:0/2689482852 >> v1:172.21.15.150:6789/0 conn(0x7ff1540727d0 legacy=0x7ff154072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff158edf640 1 -- 172.21.15.78:0/2689482852 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540c3b80 con 0x7ff1540c78a0 2024-02-19T14:59:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff158edf640 1 --2- 172.21.15.78:0/2689482852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff1540c44b0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7ff1540c1eb0 tx=0x7ff14402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6223222a5bf30a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff1537fe640 1 -- 172.21.15.78:0/2689482852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144036440 con 0x7ff1540c78a0 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff1537fe640 1 -- 172.21.15.78:0/2689482852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1440365a0 con 0x7ff1540c78a0 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.781+0000 7ff1537fe640 1 -- 172.21.15.78:0/2689482852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144036930 con 0x7ff1540c78a0 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 -- 172.21.15.78:0/2689482852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 msgr2=0x7ff1540c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 --2- 172.21.15.78:0/2689482852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff1540c44b0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7ff1540c1eb0 tx=0x7ff14402e170 comp rx=0 tx=0).stop 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 -- 172.21.15.78:0/2689482852 shutdown_connections 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 --2- 172.21.15.78:0/2689482852 >> v2:172.21.15.150:3300/0 conn(0x7ff15414ef50 0x7ff15414f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 --2- 172.21.15.78:0/2689482852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff1540c44b0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 -- 172.21.15.78:0/2689482852 >> 172.21.15.78:0/2689482852 conn(0x7ff154066e70 msgr2=0x7ff1540a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 -- 172.21.15.78:0/2689482852 shutdown_connections 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.782+0000 7ff15b143640 1 -- 172.21.15.78:0/2689482852 wait complete. 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.783+0000 7ff15b143640 1 Processor -- start 2024-02-19T14:59:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.783+0000 7ff15b143640 1 -- start start 2024-02-19T14:59:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff15b143640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540727d0 0x7ff1540aba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff15b143640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff154144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff15b143640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff15414ef50 0x7ff154144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff15b143640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541459b0 con 0x7ff1540c78a0 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff15b143640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15414a080 con 0x7ff15414ef50 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff15b143640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15414a360 con 0x7ff1540727d0 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff154144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff158edf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540727d0 0x7ff1540aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff154144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32974/0 (socket says 172.21.15.78:32974) 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 -- 172.21.15.78:0/454031001 learned_addr learned my addr 172.21.15.78:0/454031001 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff1596e0640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff15414ef50 0x7ff154144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 -- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540727d0 msgr2=0x7ff1540aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540727d0 0x7ff1540aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 -- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff15414ef50 msgr2=0x7ff154144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff15414ef50 0x7ff154144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 -- 172.21.15.78:0/454031001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14400b770 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff153fff640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff154144940 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7ff14800e870 tx=0x7ff14800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.784+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1480094f0 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.785+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14800c940 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.785+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff154148890 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.785+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800ccf0 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.785+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff154148e50 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.786+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154074990 con 0x7ff1540c78a0 2024-02-19T14:59:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.786+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff148013470 con 0x7ff1540c78a0 2024-02-19T14:59:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.786+0000 7ff1517fa640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff14006b840 0x7ff14006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.787+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff14808a330 con 0x7ff1540c78a0 2024-02-19T14:59:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.787+0000 7ff158edf640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff14006b840 0x7ff14006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.787+0000 7ff158edf640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff14006b840 0x7ff14006dd00 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff144038000 tx=0x7ff144032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:10.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.791+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff148054d00 con 0x7ff1540c78a0 2024-02-19T14:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.997+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff154073000 con 0x7ff1540c78a0 2024-02-19T14:59:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:10.998+0000 7ff1517fa640 1 -- 172.21.15.78:0/454031001 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4951 (secure 0 0 0) 0x7ff1480583c0 con 0x7ff1540c78a0 2024-02-19T14:59:11.001 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:11.001 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":165,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:09.985985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:11.001 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 165 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff14006b840 msgr2=0x7ff14006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff14006b840 0x7ff14006dd00 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff144038000 tx=0x7ff144032000 comp rx=0 tx=0).stop 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 msgr2=0x7ff154144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff154144940 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7ff14800e870 tx=0x7ff14800ed30 comp rx=0 tx=0).stop 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 shutdown_connections 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff14006b840 0x7ff14006dd00 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1540c78a0 0x7ff154144940 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff15414ef50 0x7ff154144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 --2- 172.21.15.78:0/454031001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540727d0 0x7ff1540aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 >> 172.21.15.78:0/454031001 conn(0x7ff154066e70 msgr2=0x7ff1540c59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 shutdown_connections 2024-02-19T14:59:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.000+0000 7ff15b143640 1 -- 172.21.15.78:0/454031001 wait complete. 2024-02-19T14:59:11.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:11.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.110+0000 7f0fcd674640 1 Processor -- start 2024-02-19T14:59:11.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fcd674640 1 -- start start 2024-02-19T14:59:11.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fcd674640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0fc80c8e00 0x7f0fc806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fcd674640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc806d910 0x7f0fc806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fcd674640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc806cd20 con 0x7f0fc8152700 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fcd674640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc81551f0 con 0x7f0fc80c8e00 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fcd674640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8155490 con 0x7f0fc806d910 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fc6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0fc8152700 0x7f0fc80c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39954/0 (socket says 172.21.15.78:39954) 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.111+0000 7f0fc6ffd640 1 -- 172.21.15.78:0/2907476741 learned_addr learned my addr 172.21.15.78:0/2907476741 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc77fe640 1 --2- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc806d910 0x7f0fc806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc67fc640 1 --2- 172.21.15.78:0/2907476741 >> v2:172.21.15.150:3300/0 conn(0x7f0fc80c8e00 0x7f0fc806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc5ffb640 1 -- 172.21.15.78:0/2907476741 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048142508 0 0) 0x7f0fc806cd20 con 0x7f0fc8152700 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc5ffb640 1 -- 172.21.15.78:0/2907476741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb0003520 con 0x7f0fc8152700 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc67fc640 1 -- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc806d910 msgr2=0x7f0fc806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc67fc640 1 --2- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc806d910 0x7f0fc806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc67fc640 1 -- 172.21.15.78:0/2907476741 >> v1:172.21.15.78:6789/0 conn(0x7f0fc8152700 legacy=0x7f0fc80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.112+0000 7f0fc67fc640 1 -- 172.21.15.78:0/2907476741 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8155770 con 0x7f0fc80c8e00 2024-02-19T14:59:11.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fc67fc640 1 --2- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc806c2a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0fb80097a0 tx=0x7f0fb802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2106cc8c06dc57a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fc5ffb640 1 -- 172.21.15.78:0/2907476741 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb802e070 con 0x7f0fc80c8e00 2024-02-19T14:59:11.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fc5ffb640 1 -- 172.21.15.78:0/2907476741 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb80315a0 con 0x7f0fc80c8e00 2024-02-19T14:59:11.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fc5ffb640 1 -- 172.21.15.78:0/2907476741 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8031910 con 0x7f0fc80c8e00 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 -- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 msgr2=0x7f0fc806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 --2- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc806c2a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0fb80097a0 tx=0x7f0fb802b840 comp rx=0 tx=0).stop 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 -- 172.21.15.78:0/2907476741 shutdown_connections 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 --2- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc806c2a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 --2- 172.21.15.78:0/2907476741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc806d910 0x7f0fc806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 -- 172.21.15.78:0/2907476741 >> 172.21.15.78:0/2907476741 conn(0x7f0fc80cac90 msgr2=0x7f0fc80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 -- 172.21.15.78:0/2907476741 shutdown_connections 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.113+0000 7f0fcd674640 1 -- 172.21.15.78:0/2907476741 wait complete. 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 Processor -- start 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 -- start start 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 0x7f0fc80c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc8157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc8152700 0x7f0fc81edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc80c27e0 con 0x7f0fc806d910 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc80aa3a0 con 0x7f0fc80c8e00 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fcd674640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc80aa680 con 0x7f0fc8152700 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fc6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 0x7f0fc80c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fc6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 0x7f0fc80c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:32978/0 (socket says 172.21.15.78:32978) 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.115+0000 7f0fc6ffd640 1 -- 172.21.15.78:0/3512099738 learned_addr learned my addr 172.21.15.78:0/3512099738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc77fe640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc8152700 0x7f0fc81edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc67fc640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc8157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc6ffd640 1 -- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc8152700 msgr2=0x7f0fc81edcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc6ffd640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc8152700 0x7f0fc81edcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc6ffd640 1 -- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 msgr2=0x7f0fc8157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc6ffd640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc8157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc6ffd640 1 -- 172.21.15.78:0/3512099738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8009400 con 0x7f0fc806d910 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fc6ffd640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 0x7f0fc80c3230 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c390 tx=0x7f0fbc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc005ce0 con 0x7f0fc806d910 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc005e40 con 0x7f0fc806d910 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc80aa960 con 0x7f0fc806d910 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0072f0 con 0x7f0fc806d910 2024-02-19T14:59:11.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.116+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc80c2d10 con 0x7f0fc806d910 2024-02-19T14:59:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.117+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc8074990 con 0x7f0fc806d910 2024-02-19T14:59:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.118+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0fbc007450 con 0x7f0fc806d910 2024-02-19T14:59:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.118+0000 7f0faf7fe640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0fb406b810 0x7f0fb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.118+0000 7f0fc67fc640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0fb406b810 0x7f0fb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.118+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0fbc089870 con 0x7f0fc806d910 2024-02-19T14:59:11.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.118+0000 7f0fc67fc640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0fb406b810 0x7f0fb406dcd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8036000 tx=0x7f0fb803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.122+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0fbc054240 con 0x7f0fc806d910 2024-02-19T14:59:11.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.294+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fc80c8770 con 0x7f0fc806d910 2024-02-19T14:59:11.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.295+0000 7f0faf7fe640 1 -- 172.21.15.78:0/3512099738 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+16872 (secure 0 0 0) 0x7f0fbc017090 con 0x7f0fc806d910 2024-02-19T14:59:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:11.298 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":188,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:09.648574+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:06.192227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:06.947291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0fb406b810 msgr2=0x7f0fb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0fb406b810 0x7f0fb406dcd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8036000 tx=0x7f0fb803d040 comp rx=0 tx=0).stop 2024-02-19T14:59:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 msgr2=0x7f0fc80c3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 0x7f0fc80c3230 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c390 tx=0x7f0fbc00c850 comp rx=0 tx=0).stop 2024-02-19T14:59:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 shutdown_connections 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0fb406b810 0x7f0fb406dcd0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fc8152700 0x7f0fc81edcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fc80c8e00 0x7f0fc8157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 --2- 172.21.15.78:0/3512099738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fc806d910 0x7f0fc80c3230 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 >> 172.21.15.78:0/3512099738 conn(0x7f0fc80cac90 msgr2=0x7f0fc8151650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 shutdown_connections 2024-02-19T14:59:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.297+0000 7f0fcd674640 1 -- 172.21.15.78:0/3512099738 wait complete. 2024-02-19T14:59:11.311 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:59:11.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:11.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1dcda0640 1 Processor -- start 2024-02-19T14:59:11.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1dcda0640 1 -- start start 2024-02-19T14:59:11.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1dcda0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1d80acc50 0x7ff1d80c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1dcda0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d8159130 con 0x7ff1d80718f0 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1dcda0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80ad030 con 0x7ff1d8155070 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1dcda0640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80c2cf0 con 0x7ff1d80acc50 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1d659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff1d80acc50 0x7ff1d80c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1d6d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff1d80718f0 0x7ff1d8071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39964/0 (socket says 172.21.15.78:39964) 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.409+0000 7ff1d6d9d640 1 -- 172.21.15.78:0/1981640537 learned_addr learned my addr 172.21.15.78:0/1981640537 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.410+0000 7ff1d659c640 1 -- 172.21.15.78:0/1981640537 >> v1:172.21.15.150:6789/0 conn(0x7ff1d8155070 legacy=0x7ff1d8155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.410+0000 7ff1d659c640 1 -- 172.21.15.78:0/1981640537 >> v1:172.21.15.78:6789/0 conn(0x7ff1d80718f0 legacy=0x7ff1d8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.410+0000 7ff1d659c640 1 -- 172.21.15.78:0/1981640537 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d81595c0 con 0x7ff1d80acc50 2024-02-19T14:59:11.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.410+0000 7ff1d659c640 1 --2- 172.21.15.78:0/1981640537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d80acc50 0x7ff1d80c17e0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097a0 tx=0x7ff1c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711cf022814257b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1d5d9b640 1 -- 172.21.15.78:0/1981640537 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8031830 con 0x7ff1d80acc50 2024-02-19T14:59:11.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1d5d9b640 1 -- 172.21.15.78:0/1981640537 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c8031990 con 0x7ff1d80acc50 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1d5d9b640 1 -- 172.21.15.78:0/1981640537 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8031d20 con 0x7ff1d80acc50 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 -- 172.21.15.78:0/1981640537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d80acc50 msgr2=0x7ff1d80c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/1981640537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d80acc50 0x7ff1d80c17e0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80097a0 tx=0x7ff1c802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 -- 172.21.15.78:0/1981640537 shutdown_connections 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/1981640537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d80acc50 0x7ff1d80c17e0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 -- 172.21.15.78:0/1981640537 >> 172.21.15.78:0/1981640537 conn(0x7ff1d80c7800 msgr2=0x7ff1d80c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 -- 172.21.15.78:0/1981640537 shutdown_connections 2024-02-19T14:59:11.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.411+0000 7ff1dcda0640 1 -- 172.21.15.78:0/1981640537 wait complete. 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.412+0000 7ff1dcda0640 1 Processor -- start 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.412+0000 7ff1dcda0640 1 -- start start 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1dcda0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1d80718f0 0x7ff1d81ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1dcda0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1d80acc50 0x7ff1d81387d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1dcda0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 0x7ff1d8138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1dcda0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80c2030 con 0x7ff1d8155070 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1dcda0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81e2280 con 0x7ff1d80718f0 2024-02-19T14:59:11.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1dcda0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8139f10 con 0x7ff1d80acc50 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1d6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 0x7ff1d8138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1d6d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 0x7ff1d8138d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33002/0 (socket says 172.21.15.78:33002) 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1d6d9d640 1 -- 172.21.15.78:0/244351721 learned_addr learned my addr 172.21.15.78:0/244351721 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1d659c640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1d80718f0 0x7ff1d81ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.413+0000 7ff1cffff640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1d80acc50 0x7ff1d81387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d6d9d640 1 -- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1d80acc50 msgr2=0x7ff1d81387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d6d9d640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1d80acc50 0x7ff1d81387d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d6d9d640 1 -- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1d80718f0 msgr2=0x7ff1d81ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d6d9d640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1d80718f0 0x7ff1d81ee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d6d9d640 1 -- 172.21.15.78:0/244351721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8009400 con 0x7ff1d8155070 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d659c640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1d80718f0 0x7ff1d81ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1d6d9d640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 0x7ff1d8138d10 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000cbc0 tx=0x7ff1d000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000ad40 con 0x7ff1d8155070 2024-02-19T14:59:11.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d0012400 con 0x7ff1d8155070 2024-02-19T14:59:11.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d81eebc0 con 0x7ff1d8155070 2024-02-19T14:59:11.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000aea0 con 0x7ff1d8155070 2024-02-19T14:59:11.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.414+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d81df440 con 0x7ff1d8155070 2024-02-19T14:59:11.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.415+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d81560a0 con 0x7ff1d8155070 2024-02-19T14:59:11.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.415+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff1d0007070 con 0x7ff1d8155070 2024-02-19T14:59:11.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.416+0000 7ff1ceffd640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1b806b840 0x7ff1b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.416+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff1d008a190 con 0x7ff1d8155070 2024-02-19T14:59:11.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.416+0000 7ff1d659c640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1b806b840 0x7ff1b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.416+0000 7ff1d659c640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1b806b840 0x7ff1b806dd00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8036000 tx=0x7ff1c803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.420+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff1d0054ad0 con 0x7ff1d8155070 2024-02-19T14:59:11.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.588+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1d80c1d20 con 0x7ff1b806b840 2024-02-19T14:59:11.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.595+0000 7ff1ceffd640 1 -- 172.21.15.78:0/244351721 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284544 (secure 0 0 0) 0x7ff1c8045410 con 0x7ff1b806b840 2024-02-19T14:59:11.601 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:11.602 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:11.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1b806b840 msgr2=0x7ff1b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1b806b840 0x7ff1b806dd00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8036000 tx=0x7ff1c803c040 comp rx=0 tx=0).stop 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 msgr2=0x7ff1d8138d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 0x7ff1d8138d10 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000cbc0 tx=0x7ff1d000a430 comp rx=0 tx=0).stop 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 shutdown_connections 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff1b806b840 0x7ff1b806dd00 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8155070 0x7ff1d8138d10 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1d80acc50 0x7ff1d81387d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 --2- 172.21.15.78:0/244351721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1d80718f0 0x7ff1d81ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 >> 172.21.15.78:0/244351721 conn(0x7ff1d80c7800 msgr2=0x7ff1d806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 shutdown_connections 2024-02-19T14:59:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.598+0000 7ff1dcda0640 1 -- 172.21.15.78:0/244351721 wait complete. 2024-02-19T14:59:11.611 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-02-19T14:59:10.144116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268968,"kb_used_data":19512,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754705752,"statfs":{"total":773094113280,"available":772818690048,"internally_reserved":0,"allocated":19980288,"data_stored":10364228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000656"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627095+0000","last_change":"2024-02-19T14:59:07.627095+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627095+0000","last_undegraded":"2024-02-19T14:59:07.627095+0000","last_fullsized":"2024-02-19T14:59:07.627095+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629907+0000","last_change":"2024-02-19T14:59:07.629907+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629907+0000","last_undegraded":"2024-02-19T14:59:07.629907+0000","last_fullsized":"2024-02-19T14:59:07.629907+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640087+0000","last_change":"2024-02-19T14:59:07.640087+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640087+0000","last_undegraded":"2024-02-19T14:59:07.640087+0000","last_fullsized":"2024-02-19T14:59:07.640087+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626117+0000","last_change":"2024-02-19T14:59:07.626117+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626117+0000","last_undegraded":"2024-02-19T14:59:07.626117+0000","last_fullsized":"2024-02-19T14:59:07.626117+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626189+0000","last_change":"2024-02-19T14:59:07.626189+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626189+0000","last_undegraded":"2024-02-19T14:59:07.626189+0000","last_fullsized":"2024-02-19T14:59:07.626189+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628611+0000","last_change":"2024-02-19T14:59:07.628611+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628611+0000","last_undegraded":"2024-02-19T14:59:07.628611+0000","last_fullsized":"2024-02-19T14:59:07.628611+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640069+0000","last_change":"2024-02-19T14:59:07.640069+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640069+0000","last_undegraded":"2024-02-19T14:59:07.640069+0000","last_fullsized":"2024-02-19T14:59:07.640069+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629382+0000","last_change":"2024-02-19T14:59:07.629382+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629382+0000","last_undegraded":"2024-02-19T14:59:07.629382+0000","last_fullsized":"2024-02-19T14:59:07.629382+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634618+0000","last_change":"2024-02-19T14:59:07.634618+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634618+0000","last_undegraded":"2024-02-19T14:59:07.634618+0000","last_fullsized":"2024-02-19T14:59:07.634618+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637179+0000","last_change":"2024-02-19T14:59:07.637179+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637179+0000","last_undegraded":"2024-02-19T14:59:07.637179+0000","last_fullsized":"2024-02-19T14:59:07.637179+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626760+0000","last_change":"2024-02-19T14:59:07.626760+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626760+0000","last_undegraded":"2024-02-19T14:59:07.626760+0000","last_fullsized":"2024-02-19T14:59:07.626760+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639587+0000","last_change":"2024-02-19T14:59:07.639587+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639587+0000","last_undegraded":"2024-02-19T14:59:07.639587+0000","last_fullsized":"2024-02-19T14:59:07.639587+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628891+0000","last_change":"2024-02-19T14:59:07.628891+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628891+0000","last_undegraded":"2024-02-19T14:59:07.628891+0000","last_fullsized":"2024-02-19T14:59:07.628891+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637655+0000","last_change":"2024-02-19T14:59:07.637655+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637655+0000","last_undegraded":"2024-02-19T14:59:07.637655+0000","last_fullsized":"2024-02-19T14:59:07.637655+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635774+0000","last_change":"2024-02-19T14:59:07.635774+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635774+0000","last_undegraded":"2024-02-19T14:59:07.635774+0000","last_fullsized":"2024-02-19T14:59:07.635774+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628909+0000","last_change":"2024-02-19T14:59:07.628909+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628909+0000","last_undegraded":"2024-02-19T14:59:07.628909+0000","last_fullsized":"2024-02-19T14:59:07.628909+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639159+0000","last_change":"2024-02-19T14:59:07.639159+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639159+0000","last_undegraded":"2024-02-19T14:59:07.639159+0000","last_fullsized":"2024-02-19T14:59:07.639159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636673+0000","last_change":"2024-02-19T14:59:07.636673+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636673+0000","last_undegraded":"2024-02-19T14:59:07.636673+0000","last_fullsized":"2024-02-19T14:59:07.636673+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627251+0000","last_change":"2024-02-19T14:59:07.627251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627251+0000","last_undegraded":"2024-02-19T14:59:07.627251+0000","last_fullsized":"2024-02-19T14:59:07.627251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629094+0000","last_change":"2024-02-19T14:59:07.629094+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629094+0000","last_undegraded":"2024-02-19T14:59:07.629094+0000","last_fullsized":"2024-02-19T14:59:07.629094+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635444+0000","last_change":"2024-02-19T14:59:07.635444+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635444+0000","last_undegraded":"2024-02-19T14:59:07.635444+0000","last_fullsized":"2024-02-19T14:59:07.635444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635646+0000","last_change":"2024-02-19T14:59:07.635646+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635646+0000","last_undegraded":"2024-02-19T14:59:07.635646+0000","last_fullsized":"2024-02-19T14:59:07.635646+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633781+0000","last_change":"2024-02-19T14:59:07.633781+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633781+0000","last_undegraded":"2024-02-19T14:59:07.633781+0000","last_fullsized":"2024-02-19T14:59:07.633781+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626539+0000","last_change":"2024-02-19T14:59:07.626539+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626539+0000","last_undegraded":"2024-02-19T14:59:07.626539+0000","last_fullsized":"2024-02-19T14:59:07.626539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626634+0000","last_change":"2024-02-19T14:59:07.626634+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626634+0000","last_undegraded":"2024-02-19T14:59:07.626634+0000","last_fullsized":"2024-02-19T14:59:07.626634+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634974+0000","last_change":"2024-02-19T14:59:07.634974+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634974+0000","last_undegraded":"2024-02-19T14:59:07.634974+0000","last_fullsized":"2024-02-19T14:59:07.634974+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628304+0000","last_change":"2024-02-19T14:59:07.628304+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628304+0000","last_undegraded":"2024-02-19T14:59:07.628304+0000","last_fullsized":"2024-02-19T14:59:07.628304+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638306+0000","last_change":"2024-02-19T14:59:07.638306+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638306+0000","last_undegraded":"2024-02-19T14:59:07.638306+0000","last_fullsized":"2024-02-19T14:59:07.638306+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629327+0000","last_change":"2024-02-19T14:59:07.629327+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629327+0000","last_undegraded":"2024-02-19T14:59:07.629327+0000","last_fullsized":"2024-02-19T14:59:07.629327+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629097+0000","last_change":"2024-02-19T14:59:07.629097+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629097+0000","last_undegraded":"2024-02-19T14:59:07.629097+0000","last_fullsized":"2024-02-19T14:59:07.629097+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633321+0000","last_change":"2024-02-19T14:59:07.633321+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633321+0000","last_undegraded":"2024-02-19T14:59:07.633321+0000","last_fullsized":"2024-02-19T14:59:07.633321+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628422+0000","last_change":"2024-02-19T14:59:07.628422+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628422+0000","last_undegraded":"2024-02-19T14:59:07.628422+0000","last_fullsized":"2024-02-19T14:59:07.628422+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634580+0000","last_change":"2024-02-19T14:59:07.634580+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634580+0000","last_undegraded":"2024-02-19T14:59:07.634580+0000","last_fullsized":"2024-02-19T14:59:07.634580+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640931+0000","last_change":"2024-02-19T14:59:07.638344+0000","last_active":"2024-02-19T14:59:07.640931+0000","last_peered":"2024-02-19T14:59:07.640931+0000","last_clean":"2024-02-19T14:59:07.640931+0000","last_became_active":"2024-02-19T14:59:07.638065+0000","last_became_peered":"2024-02-19T14:59:07.638065+0000","last_unstale":"2024-02-19T14:59:07.640931+0000","last_undegraded":"2024-02-19T14:59:07.640931+0000","last_fullsized":"2024-02-19T14:59:07.640931+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:55.830792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635080+0000","last_change":"2024-02-19T14:59:07.635080+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635080+0000","last_undegraded":"2024-02-19T14:59:07.635080+0000","last_fullsized":"2024-02-19T14:59:07.635080+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644762+0000","last_change":"2024-02-19T14:59:07.639820+0000","last_active":"2024-02-19T14:59:07.644762+0000","last_peered":"2024-02-19T14:59:07.644762+0000","last_clean":"2024-02-19T14:59:07.644762+0000","last_became_active":"2024-02-19T14:59:07.639571+0000","last_became_peered":"2024-02-19T14:59:07.639571+0000","last_unstale":"2024-02-19T14:59:07.644762+0000","last_undegraded":"2024-02-19T14:59:07.644762+0000","last_fullsized":"2024-02-19T14:59:07.644762+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:26:02.808996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637145+0000","last_change":"2024-02-19T14:59:07.637145+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637145+0000","last_undegraded":"2024-02-19T14:59:07.637145+0000","last_fullsized":"2024-02-19T14:59:07.637145+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642076+0000","last_change":"2024-02-19T14:59:07.641532+0000","last_active":"2024-02-19T14:59:07.642076+0000","last_peered":"2024-02-19T14:59:07.642076+0000","last_clean":"2024-02-19T14:59:07.642076+0000","last_became_active":"2024-02-19T14:59:07.641278+0000","last_became_peered":"2024-02-19T14:59:07.641278+0000","last_unstale":"2024-02-19T14:59:07.642076+0000","last_undegraded":"2024-02-19T14:59:07.642076+0000","last_fullsized":"2024-02-19T14:59:07.642076+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:57.560202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633978+0000","last_change":"2024-02-19T14:59:07.633978+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633978+0000","last_undegraded":"2024-02-19T14:59:07.633978+0000","last_fullsized":"2024-02-19T14:59:07.633978+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646713+0000","last_change":"2024-02-19T14:59:07.646204+0000","last_active":"2024-02-19T14:59:07.646713+0000","last_peered":"2024-02-19T14:59:07.646713+0000","last_clean":"2024-02-19T14:59:07.646713+0000","last_became_active":"2024-02-19T14:59:07.645880+0000","last_became_peered":"2024-02-19T14:59:07.645880+0000","last_unstale":"2024-02-19T14:59:07.646713+0000","last_undegraded":"2024-02-19T14:59:07.646713+0000","last_fullsized":"2024-02-19T14:59:07.646713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637251+0000","last_change":"2024-02-19T14:59:07.637251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637251+0000","last_undegraded":"2024-02-19T14:59:07.637251+0000","last_fullsized":"2024-02-19T14:59:07.637251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644114+0000","last_change":"2024-02-19T14:59:07.642010+0000","last_active":"2024-02-19T14:59:07.644114+0000","last_peered":"2024-02-19T14:59:07.644114+0000","last_clean":"2024-02-19T14:59:07.644114+0000","last_became_active":"2024-02-19T14:59:07.641735+0000","last_became_peered":"2024-02-19T14:59:07.641735+0000","last_unstale":"2024-02-19T14:59:07.644114+0000","last_undegraded":"2024-02-19T14:59:07.644114+0000","last_fullsized":"2024-02-19T14:59:07.644114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:05.299812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635252+0000","last_change":"2024-02-19T14:59:07.635252+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635252+0000","last_undegraded":"2024-02-19T14:59:07.635252+0000","last_fullsized":"2024-02-19T14:59:07.635252+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644650+0000","last_change":"2024-02-19T14:59:07.643447+0000","last_active":"2024-02-19T14:59:07.644650+0000","last_peered":"2024-02-19T14:59:07.644650+0000","last_clean":"2024-02-19T14:59:07.644650+0000","last_became_active":"2024-02-19T14:59:07.643150+0000","last_became_peered":"2024-02-19T14:59:07.643150+0000","last_unstale":"2024-02-19T14:59:07.644650+0000","last_undegraded":"2024-02-19T14:59:07.644650+0000","last_fullsized":"2024-02-19T14:59:07.644650+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:37.234470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634275+0000","last_change":"2024-02-19T14:59:07.634275+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634275+0000","last_undegraded":"2024-02-19T14:59:07.634275+0000","last_fullsized":"2024-02-19T14:59:07.634275+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644014+0000","last_change":"2024-02-19T14:59:07.641207+0000","last_active":"2024-02-19T14:59:07.644014+0000","last_peered":"2024-02-19T14:59:07.644014+0000","last_clean":"2024-02-19T14:59:07.644014+0000","last_became_active":"2024-02-19T14:59:07.640874+0000","last_became_peered":"2024-02-19T14:59:07.640874+0000","last_unstale":"2024-02-19T14:59:07.644014+0000","last_undegraded":"2024-02-19T14:59:07.644014+0000","last_fullsized":"2024-02-19T14:59:07.644014+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:16:13.230015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634281+0000","last_change":"2024-02-19T14:59:07.634281+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634281+0000","last_undegraded":"2024-02-19T14:59:07.634281+0000","last_fullsized":"2024-02-19T14:59:07.634281+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641906+0000","last_change":"2024-02-19T14:59:07.638597+0000","last_active":"2024-02-19T14:59:07.641906+0000","last_peered":"2024-02-19T14:59:07.641906+0000","last_clean":"2024-02-19T14:59:07.641906+0000","last_became_active":"2024-02-19T14:59:07.638358+0000","last_became_peered":"2024-02-19T14:59:07.638358+0000","last_unstale":"2024-02-19T14:59:07.641906+0000","last_undegraded":"2024-02-19T14:59:07.641906+0000","last_fullsized":"2024-02-19T14:59:07.641906+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:21.669002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634738+0000","last_change":"2024-02-19T14:59:07.634738+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634738+0000","last_undegraded":"2024-02-19T14:59:07.634738+0000","last_fullsized":"2024-02-19T14:59:07.634738+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644861+0000","last_change":"2024-02-19T14:59:07.643687+0000","last_active":"2024-02-19T14:59:07.644861+0000","last_peered":"2024-02-19T14:59:07.644861+0000","last_clean":"2024-02-19T14:59:07.644861+0000","last_became_active":"2024-02-19T14:59:07.643136+0000","last_became_peered":"2024-02-19T14:59:07.643136+0000","last_unstale":"2024-02-19T14:59:07.644861+0000","last_undegraded":"2024-02-19T14:59:07.644861+0000","last_fullsized":"2024-02-19T14:59:07.644861+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:28:01.427001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633392+0000","last_change":"2024-02-19T14:59:07.633392+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633392+0000","last_undegraded":"2024-02-19T14:59:07.633392+0000","last_fullsized":"2024-02-19T14:59:07.633392+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641877+0000","last_change":"2024-02-19T14:59:07.638366+0000","last_active":"2024-02-19T14:59:07.641877+0000","last_peered":"2024-02-19T14:59:07.641877+0000","last_clean":"2024-02-19T14:59:07.641877+0000","last_became_active":"2024-02-19T14:59:07.638109+0000","last_became_peered":"2024-02-19T14:59:07.638109+0000","last_unstale":"2024-02-19T14:59:07.641877+0000","last_undegraded":"2024-02-19T14:59:07.641877+0000","last_fullsized":"2024-02-19T14:59:07.641877+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627740+0000","last_change":"2024-02-19T14:59:07.627740+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627740+0000","last_undegraded":"2024-02-19T14:59:07.627740+0000","last_fullsized":"2024-02-19T14:59:07.627740+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647367+0000","last_change":"2024-02-19T14:59:07.646216+0000","last_active":"2024-02-19T14:59:07.647367+0000","last_peered":"2024-02-19T14:59:07.647367+0000","last_clean":"2024-02-19T14:59:07.647367+0000","last_became_active":"2024-02-19T14:59:07.645915+0000","last_became_peered":"2024-02-19T14:59:07.645915+0000","last_unstale":"2024-02-19T14:59:07.647367+0000","last_undegraded":"2024-02-19T14:59:07.647367+0000","last_fullsized":"2024-02-19T14:59:07.647367+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:28:14.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630383+0000","last_change":"2024-02-19T14:59:07.630383+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630383+0000","last_undegraded":"2024-02-19T14:59:07.630383+0000","last_fullsized":"2024-02-19T14:59:07.630383+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647108+0000","last_change":"2024-02-19T14:59:07.643409+0000","last_active":"2024-02-19T14:59:07.647108+0000","last_peered":"2024-02-19T14:59:07.647108+0000","last_clean":"2024-02-19T14:59:07.647108+0000","last_became_active":"2024-02-19T14:59:07.643105+0000","last_became_peered":"2024-02-19T14:59:07.643105+0000","last_unstale":"2024-02-19T14:59:07.647108+0000","last_undegraded":"2024-02-19T14:59:07.647108+0000","last_fullsized":"2024-02-19T14:59:07.647108+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:55.853954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633352+0000","last_change":"2024-02-19T14:59:07.633352+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633352+0000","last_undegraded":"2024-02-19T14:59:07.633352+0000","last_fullsized":"2024-02-19T14:59:07.633352+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640331+0000","last_change":"2024-02-19T14:59:07.639087+0000","last_active":"2024-02-19T14:59:07.640331+0000","last_peered":"2024-02-19T14:59:07.640331+0000","last_clean":"2024-02-19T14:59:07.640331+0000","last_became_active":"2024-02-19T14:59:07.638819+0000","last_became_peered":"2024-02-19T14:59:07.638819+0000","last_unstale":"2024-02-19T14:59:07.640331+0000","last_undegraded":"2024-02-19T14:59:07.640331+0000","last_fullsized":"2024-02-19T14:59:07.640331+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632905+0000","last_change":"2024-02-19T14:59:07.632905+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632905+0000","last_undegraded":"2024-02-19T14:59:07.632905+0000","last_fullsized":"2024-02-19T14:59:07.632905+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637435+0000","last_change":"2024-02-19T14:59:07.636744+0000","last_active":"2024-02-19T14:59:07.637435+0000","last_peered":"2024-02-19T14:59:07.637435+0000","last_clean":"2024-02-19T14:59:07.637435+0000","last_became_active":"2024-02-19T14:59:07.636464+0000","last_became_peered":"2024-02-19T14:59:07.636464+0000","last_unstale":"2024-02-19T14:59:07.637435+0000","last_undegraded":"2024-02-19T14:59:07.637435+0000","last_fullsized":"2024-02-19T14:59:07.637435+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:30.048428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633393+0000","last_change":"2024-02-19T14:59:07.633393+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633393+0000","last_undegraded":"2024-02-19T14:59:07.633393+0000","last_fullsized":"2024-02-19T14:59:07.633393+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646942+0000","last_change":"2024-02-19T14:59:07.643659+0000","last_active":"2024-02-19T14:59:07.646942+0000","last_peered":"2024-02-19T14:59:07.646942+0000","last_clean":"2024-02-19T14:59:07.646942+0000","last_became_active":"2024-02-19T14:59:07.643126+0000","last_became_peered":"2024-02-19T14:59:07.643126+0000","last_unstale":"2024-02-19T14:59:07.646942+0000","last_undegraded":"2024-02-19T14:59:07.646942+0000","last_fullsized":"2024-02-19T14:59:07.646942+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:30.093846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630599+0000","last_change":"2024-02-19T14:59:07.630599+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630599+0000","last_undegraded":"2024-02-19T14:59:07.630599+0000","last_fullsized":"2024-02-19T14:59:07.630599+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644152+0000","last_change":"2024-02-19T14:59:07.642484+0000","last_active":"2024-02-19T14:59:07.644152+0000","last_peered":"2024-02-19T14:59:07.644152+0000","last_clean":"2024-02-19T14:59:07.644152+0000","last_became_active":"2024-02-19T14:59:07.642254+0000","last_became_peered":"2024-02-19T14:59:07.642254+0000","last_unstale":"2024-02-19T14:59:07.644152+0000","last_undegraded":"2024-02-19T14:59:07.644152+0000","last_fullsized":"2024-02-19T14:59:07.644152+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:19.998342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626772+0000","last_change":"2024-02-19T14:59:07.626772+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626772+0000","last_undegraded":"2024-02-19T14:59:07.626772+0000","last_fullsized":"2024-02-19T14:59:07.626772+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.645052+0000","last_change":"2024-02-19T14:59:07.644552+0000","last_active":"2024-02-19T14:59:07.645052+0000","last_peered":"2024-02-19T14:59:07.645052+0000","last_clean":"2024-02-19T14:59:07.645052+0000","last_became_active":"2024-02-19T14:59:07.644274+0000","last_became_peered":"2024-02-19T14:59:07.644274+0000","last_unstale":"2024-02-19T14:59:07.645052+0000","last_undegraded":"2024-02-19T14:59:07.645052+0000","last_fullsized":"2024-02-19T14:59:07.645052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:31:22.200914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632440+0000","last_change":"2024-02-19T14:59:07.632440+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632440+0000","last_undegraded":"2024-02-19T14:59:07.632440+0000","last_fullsized":"2024-02-19T14:59:07.632440+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642826+0000","last_change":"2024-02-19T14:59:07.638149+0000","last_active":"2024-02-19T14:59:07.642826+0000","last_peered":"2024-02-19T14:59:07.642826+0000","last_clean":"2024-02-19T14:59:07.642826+0000","last_became_active":"2024-02-19T14:59:07.637845+0000","last_became_peered":"2024-02-19T14:59:07.637845+0000","last_unstale":"2024-02-19T14:59:07.642826+0000","last_undegraded":"2024-02-19T14:59:07.642826+0000","last_fullsized":"2024-02-19T14:59:07.642826+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:47.116497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634012+0000","last_change":"2024-02-19T14:59:07.634012+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634012+0000","last_undegraded":"2024-02-19T14:59:07.634012+0000","last_fullsized":"2024-02-19T14:59:07.634012+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647966+0000","last_change":"2024-02-19T14:59:07.645385+0000","last_active":"2024-02-19T14:59:07.647966+0000","last_peered":"2024-02-19T14:59:07.647966+0000","last_clean":"2024-02-19T14:59:07.647966+0000","last_became_active":"2024-02-19T14:59:07.644999+0000","last_became_peered":"2024-02-19T14:59:07.644999+0000","last_unstale":"2024-02-19T14:59:07.647966+0000","last_undegraded":"2024-02-19T14:59:07.647966+0000","last_fullsized":"2024-02-19T14:59:07.647966+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:03:16.898728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636317+0000","last_change":"2024-02-19T14:59:07.636317+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636317+0000","last_undegraded":"2024-02-19T14:59:07.636317+0000","last_fullsized":"2024-02-19T14:59:07.636317+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.648114+0000","last_change":"2024-02-19T14:59:07.646936+0000","last_active":"2024-02-19T14:59:07.648114+0000","last_peered":"2024-02-19T14:59:07.648114+0000","last_clean":"2024-02-19T14:59:07.648114+0000","last_became_active":"2024-02-19T14:59:07.645710+0000","last_became_peered":"2024-02-19T14:59:07.645710+0000","last_unstale":"2024-02-19T14:59:07.648114+0000","last_undegraded":"2024-02-19T14:59:07.648114+0000","last_fullsized":"2024-02-19T14:59:07.648114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:44.397134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626291+0000","last_change":"2024-02-19T14:59:07.626291+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626291+0000","last_undegraded":"2024-02-19T14:59:07.626291+0000","last_fullsized":"2024-02-19T14:59:07.626291+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.638399+0000","last_change":"2024-02-19T14:59:07.636425+0000","last_active":"2024-02-19T14:59:07.638399+0000","last_peered":"2024-02-19T14:59:07.638399+0000","last_clean":"2024-02-19T14:59:07.638399+0000","last_became_active":"2024-02-19T14:59:07.636154+0000","last_became_peered":"2024-02-19T14:59:07.636154+0000","last_unstale":"2024-02-19T14:59:07.638399+0000","last_undegraded":"2024-02-19T14:59:07.638399+0000","last_fullsized":"2024-02-19T14:59:07.638399+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:41.446210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630391+0000","last_change":"2024-02-19T14:59:07.630391+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630391+0000","last_undegraded":"2024-02-19T14:59:07.630391+0000","last_fullsized":"2024-02-19T14:59:07.630391+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640425+0000","last_change":"2024-02-19T14:59:07.638740+0000","last_active":"2024-02-19T14:59:07.640425+0000","last_peered":"2024-02-19T14:59:07.640425+0000","last_clean":"2024-02-19T14:59:07.640425+0000","last_became_active":"2024-02-19T14:59:07.638275+0000","last_became_peered":"2024-02-19T14:59:07.638275+0000","last_unstale":"2024-02-19T14:59:07.640425+0000","last_undegraded":"2024-02-19T14:59:07.640425+0000","last_fullsized":"2024-02-19T14:59:07.640425+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631957+0000","last_change":"2024-02-19T14:59:07.631957+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631957+0000","last_undegraded":"2024-02-19T14:59:07.631957+0000","last_fullsized":"2024-02-19T14:59:07.631957+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641644+0000","last_change":"2024-02-19T14:59:07.641182+0000","last_active":"2024-02-19T14:59:07.641644+0000","last_peered":"2024-02-19T14:59:07.641644+0000","last_clean":"2024-02-19T14:59:07.641644+0000","last_became_active":"2024-02-19T14:59:07.640867+0000","last_became_peered":"2024-02-19T14:59:07.640867+0000","last_unstale":"2024-02-19T14:59:07.641644+0000","last_undegraded":"2024-02-19T14:59:07.641644+0000","last_fullsized":"2024-02-19T14:59:07.641644+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:59.660494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629585+0000","last_change":"2024-02-19T14:59:07.629585+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629585+0000","last_undegraded":"2024-02-19T14:59:07.629585+0000","last_fullsized":"2024-02-19T14:59:07.629585+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643474+0000","last_change":"2024-02-19T14:59:07.640656+0000","last_active":"2024-02-19T14:59:07.643474+0000","last_peered":"2024-02-19T14:59:07.643474+0000","last_clean":"2024-02-19T14:59:07.643474+0000","last_became_active":"2024-02-19T14:59:07.640382+0000","last_became_peered":"2024-02-19T14:59:07.640382+0000","last_unstale":"2024-02-19T14:59:07.643474+0000","last_undegraded":"2024-02-19T14:59:07.643474+0000","last_fullsized":"2024-02-19T14:59:07.643474+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:43:14.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626274+0000","last_change":"2024-02-19T14:59:07.626274+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626274+0000","last_undegraded":"2024-02-19T14:59:07.626274+0000","last_fullsized":"2024-02-19T14:59:07.626274+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640387+0000","last_change":"2024-02-19T14:59:07.638470+0000","last_active":"2024-02-19T14:59:07.640387+0000","last_peered":"2024-02-19T14:59:07.640387+0000","last_clean":"2024-02-19T14:59:07.640387+0000","last_became_active":"2024-02-19T14:59:07.638197+0000","last_became_peered":"2024-02-19T14:59:07.638197+0000","last_unstale":"2024-02-19T14:59:07.640387+0000","last_undegraded":"2024-02-19T14:59:07.640387+0000","last_fullsized":"2024-02-19T14:59:07.640387+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:09.994584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630266+0000","last_change":"2024-02-19T14:59:07.630266+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630266+0000","last_undegraded":"2024-02-19T14:59:07.630266+0000","last_fullsized":"2024-02-19T14:59:07.630266+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643869+0000","last_change":"2024-02-19T14:59:07.640562+0000","last_active":"2024-02-19T14:59:07.643869+0000","last_peered":"2024-02-19T14:59:07.643869+0000","last_clean":"2024-02-19T14:59:07.643869+0000","last_became_active":"2024-02-19T14:59:07.640288+0000","last_became_peered":"2024-02-19T14:59:07.640288+0000","last_unstale":"2024-02-19T14:59:07.643869+0000","last_undegraded":"2024-02-19T14:59:07.643869+0000","last_fullsized":"2024-02-19T14:59:07.643869+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634982+0000","last_change":"2024-02-19T14:59:07.634982+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634982+0000","last_undegraded":"2024-02-19T14:59:07.634982+0000","last_fullsized":"2024-02-19T14:59:07.634982+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647753+0000","last_change":"2024-02-19T14:59:07.645290+0000","last_active":"2024-02-19T14:59:07.647753+0000","last_peered":"2024-02-19T14:59:07.647753+0000","last_clean":"2024-02-19T14:59:07.647753+0000","last_became_active":"2024-02-19T14:59:07.644933+0000","last_became_peered":"2024-02-19T14:59:07.644933+0000","last_unstale":"2024-02-19T14:59:07.647753+0000","last_undegraded":"2024-02-19T14:59:07.647753+0000","last_fullsized":"2024-02-19T14:59:07.647753+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:37.251826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630878+0000","last_change":"2024-02-19T14:59:07.630878+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630878+0000","last_undegraded":"2024-02-19T14:59:07.630878+0000","last_fullsized":"2024-02-19T14:59:07.630878+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647675+0000","last_change":"2024-02-19T14:59:07.643990+0000","last_active":"2024-02-19T14:59:07.647675+0000","last_peered":"2024-02-19T14:59:07.647675+0000","last_clean":"2024-02-19T14:59:07.647675+0000","last_became_active":"2024-02-19T14:59:07.643708+0000","last_became_peered":"2024-02-19T14:59:07.643708+0000","last_unstale":"2024-02-19T14:59:07.647675+0000","last_undegraded":"2024-02-19T14:59:07.647675+0000","last_fullsized":"2024-02-19T14:59:07.647675+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:00.752877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631620+0000","last_change":"2024-02-19T14:59:07.631620+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631620+0000","last_undegraded":"2024-02-19T14:59:07.631620+0000","last_fullsized":"2024-02-19T14:59:07.631620+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637822+0000","last_change":"2024-02-19T14:59:07.634059+0000","last_active":"2024-02-19T14:59:07.637822+0000","last_peered":"2024-02-19T14:59:07.637822+0000","last_clean":"2024-02-19T14:59:07.637822+0000","last_became_active":"2024-02-19T14:59:07.633768+0000","last_became_peered":"2024-02-19T14:59:07.633768+0000","last_unstale":"2024-02-19T14:59:07.637822+0000","last_undegraded":"2024-02-19T14:59:07.637822+0000","last_fullsized":"2024-02-19T14:59:07.637822+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:52.944510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633788+0000","last_change":"2024-02-19T14:59:07.633788+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633788+0000","last_undegraded":"2024-02-19T14:59:07.633788+0000","last_fullsized":"2024-02-19T14:59:07.633788+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.636397+0000","last_change":"2024-02-19T14:59:07.632792+0000","last_active":"2024-02-19T14:59:07.636397+0000","last_peered":"2024-02-19T14:59:07.636397+0000","last_clean":"2024-02-19T14:59:07.636397+0000","last_became_active":"2024-02-19T14:59:07.632529+0000","last_became_peered":"2024-02-19T14:59:07.632529+0000","last_unstale":"2024-02-19T14:59:07.636397+0000","last_undegraded":"2024-02-19T14:59:07.636397+0000","last_fullsized":"2024-02-19T14:59:07.636397+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638240+0000","last_change":"2024-02-19T14:59:07.638240+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638240+0000","last_undegraded":"2024-02-19T14:59:07.638240+0000","last_fullsized":"2024-02-19T14:59:07.638240+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646917+0000","last_change":"2024-02-19T14:59:07.643336+0000","last_active":"2024-02-19T14:59:07.646917+0000","last_peered":"2024-02-19T14:59:07.646917+0000","last_clean":"2024-02-19T14:59:07.646917+0000","last_became_active":"2024-02-19T14:59:07.642778+0000","last_became_peered":"2024-02-19T14:59:07.642778+0000","last_unstale":"2024-02-19T14:59:07.646917+0000","last_undegraded":"2024-02-19T14:59:07.646917+0000","last_fullsized":"2024-02-19T14:59:07.646917+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:36.331774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634502+0000","last_change":"2024-02-19T14:59:07.634502+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634502+0000","last_undegraded":"2024-02-19T14:59:07.634502+0000","last_fullsized":"2024-02-19T14:59:07.634502+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641427+0000","last_change":"2024-02-19T14:59:07.636409+0000","last_active":"2024-02-19T14:59:07.641427+0000","last_peered":"2024-02-19T14:59:07.641427+0000","last_clean":"2024-02-19T14:59:07.641427+0000","last_became_active":"2024-02-19T14:59:07.636136+0000","last_became_peered":"2024-02-19T14:59:07.636136+0000","last_unstale":"2024-02-19T14:59:07.641427+0000","last_undegraded":"2024-02-19T14:59:07.641427+0000","last_fullsized":"2024-02-19T14:59:07.641427+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":480,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.627573+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:07.627573+0000","last_peered":"2024-02-19T14:59:07.627573+0000","last_clean":"2024-02-19T14:59:07.627573+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:07.627573+0000","last_undegraded":"2024-02-19T14:59:07.627573+0000","last_fullsized":"2024-02-19T14:59:07.627573+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34060,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337780,"statfs":{"total":96636764160,"available":96601886720,"internally_reserved":0,"allocated":2949120,"data_stored":1741638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.505,"15min":0.495},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51600000000000001,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.748,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69299999999999995,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.627,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.629}]},{"osd":4,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46200000000000002,"15min":0.44900000000000001},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.753,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.499,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.75,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.79000000000000004,"5min":0.871,"15min":0.871},"last":0.498}]},{"osd":6,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.84399999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574939,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69299999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.92400000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.876},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.94899999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.68000000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.313},"min":{"1min":0.217,"5min":0.159,"15min":0.159},"max":{"1min":0.45100000000000001,"5min":3.694,"15min":3.694},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36799999999999999,"15min":0.32600000000000001},"min":{"1min":0.248,"5min":0.192,"15min":0.189},"max":{"1min":0.55300000000000005,"5min":4.835,"15min":4.835},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.41699999999999998,"15min":0.378},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.498,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.314}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41399999999999998,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76200000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.621},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78700000000000003,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.41999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72099999999999997,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.374},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.35299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.312},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.748}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.623},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.85799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.32600000000000001},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32300000000000001,"15min":0.33200000000000002},"min":{"1min":0.24299999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.45600000000000002,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36899999999999999,"15min":0.38500000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42199999999999999,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.54900000000000004},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.309},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.96799999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.44800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.56499999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.82399999999999995,"15min":0.996},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.036},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.71799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34048,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337792,"statfs":{"total":96636764160,"available":96601899008,"internally_reserved":0,"allocated":2936832,"data_stored":1734971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:11.616 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:11.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.748+0000 7f214ff1e640 1 Processor -- start 2024-02-19T14:59:11.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214ff1e640 1 -- start start 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214ff1e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f214806c9b0 0x7f2148160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214ff1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21481551f0 con 0x7f2148152700 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214ff1e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2148155490 con 0x7f21480c8e00 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214ff1e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2148155770 con 0x7f214806c9b0 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214dcba640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2148152700 0x7f21480c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39978/0 (socket says 172.21.15.78:39978) 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214dcba640 1 -- 172.21.15.78:0/2635649526 learned_addr learned my addr 172.21.15.78:0/2635649526 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.749+0000 7f214e4bb640 1 --2- 172.21.15.78:0/2635649526 >> v2:172.21.15.150:3300/0 conn(0x7f214806c9b0 0x7f2148160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.750+0000 7f214ccb8640 1 -- 172.21.15.78:0/2635649526 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934850172 0 0) 0x7f21481551f0 con 0x7f2148152700 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.750+0000 7f214ccb8640 1 -- 172.21.15.78:0/2635649526 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2134003520 con 0x7f2148152700 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.750+0000 7f214e4bb640 1 -- 172.21.15.78:0/2635649526 >> v1:172.21.15.150:6789/0 conn(0x7f21480c8e00 legacy=0x7f214806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.750+0000 7f214e4bb640 1 -- 172.21.15.78:0/2635649526 >> v1:172.21.15.78:6789/0 conn(0x7f2148152700 legacy=0x7f21480c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.750+0000 7f214e4bb640 1 -- 172.21.15.78:0/2635649526 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2148155a50 con 0x7f214806c9b0 2024-02-19T14:59:11.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214e4bb640 1 --2- 172.21.15.78:0/2635649526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214806c9b0 0x7f2148160990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2140009d90 tx=0x7f214002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4592cf923ef7db server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ccb8640 1 -- 172.21.15.78:0/2635649526 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214002cd00 con 0x7f214806c9b0 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ccb8640 1 -- 172.21.15.78:0/2635649526 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214002ce60 con 0x7f214806c9b0 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ccb8640 1 -- 172.21.15.78:0/2635649526 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140034660 con 0x7f214806c9b0 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 -- 172.21.15.78:0/2635649526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214806c9b0 msgr2=0x7f2148160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2635649526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214806c9b0 0x7f2148160990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2140009d90 tx=0x7f214002bf50 comp rx=0 tx=0).stop 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 -- 172.21.15.78:0/2635649526 shutdown_connections 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2635649526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214806c9b0 0x7f2148160990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 -- 172.21.15.78:0/2635649526 >> 172.21.15.78:0/2635649526 conn(0x7f21480cac90 msgr2=0x7f21480cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 -- 172.21.15.78:0/2635649526 shutdown_connections 2024-02-19T14:59:11.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.751+0000 7f214ff1e640 1 -- 172.21.15.78:0/2635649526 wait complete. 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.752+0000 7f214ff1e640 1 Processor -- start 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 -- start start 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 0x7f214814de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21480c8e00 0x7f2148146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2148152700 0x7f21481edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480aaf30 con 0x7f214806c9b0 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480abb90 con 0x7f2148152700 2024-02-19T14:59:11.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214ff1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480abe70 con 0x7f21480c8e00 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214dcba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 0x7f214814de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214dcba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 0x7f214814de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33016/0 (socket says 172.21.15.78:33016) 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214dcba640 1 -- 172.21.15.78:0/2093277010 learned_addr learned my addr 172.21.15.78:0/2093277010 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214e4bb640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2148152700 0x7f21481edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.753+0000 7f214d4b9640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21480c8e00 0x7f2148146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214dcba640 1 -- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21480c8e00 msgr2=0x7f2148146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214dcba640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21480c8e00 0x7f2148146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214dcba640 1 -- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2148152700 msgr2=0x7f21481edf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214dcba640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2148152700 0x7f21481edf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214dcba640 1 -- 172.21.15.78:0/2093277010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140009a70 con 0x7f214806c9b0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214dcba640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 0x7f214814de90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f213800c410 tx=0x7f213800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138005ce0 con 0x7f214806c9b0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138005e40 con 0x7f214806c9b0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21480ac150 con 0x7f214806c9b0 2024-02-19T14:59:11.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21380072f0 con 0x7f214806c9b0 2024-02-19T14:59:11.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.754+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214814e3d0 con 0x7f214806c9b0 2024-02-19T14:59:11.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.755+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148074990 con 0x7f214806c9b0 2024-02-19T14:59:11.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.755+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2138007450 con 0x7f214806c9b0 2024-02-19T14:59:11.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.756+0000 7f213e7fc640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f212c06b810 0x7f212c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:11.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.756+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2138089890 con 0x7f214806c9b0 2024-02-19T14:59:11.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.756+0000 7f214d4b9640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f212c06b810 0x7f212c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:11.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.756+0000 7f214d4b9640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f212c06b810 0x7f212c06dcd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2148159330 tx=0x7f2144009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:11.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.760+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2138054320 con 0x7f214806c9b0 2024-02-19T14:59:11.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.928+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2148155f70 con 0x7f212c06b810 2024-02-19T14:59:11.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.935+0000 7f213e7fc640 1 -- 172.21.15.78:0/2093277010 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284544 (secure 0 0 0) 0x7f2144001420 con 0x7f212c06b810 2024-02-19T14:59:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:11.941 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:11.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f212c06b810 msgr2=0x7f212c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f212c06b810 0x7f212c06dcd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f2148159330 tx=0x7f2144009040 comp rx=0 tx=0).stop 2024-02-19T14:59:11.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 msgr2=0x7f214814de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:11.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 0x7f214814de90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f213800c410 tx=0x7f213800c8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 shutdown_connections 2024-02-19T14:59:11.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f212c06b810 0x7f212c06dcd0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2148152700 0x7f21481edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21480c8e00 0x7f2148146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 --2- 172.21.15.78:0/2093277010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f214806c9b0 0x7f214814de90 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:11.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 >> 172.21.15.78:0/2093277010 conn(0x7f21480cac90 msgr2=0x7f21480cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:11.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 shutdown_connections 2024-02-19T14:59:11.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:11.938+0000 7f214ff1e640 1 -- 172.21.15.78:0/2093277010 wait complete. 2024-02-19T14:59:11.951 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-02-19T14:59:10.144116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268968,"kb_used_data":19512,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754705752,"statfs":{"total":773094113280,"available":772818690048,"internally_reserved":0,"allocated":19980288,"data_stored":10364228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000656"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627095+0000","last_change":"2024-02-19T14:59:07.627095+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627095+0000","last_undegraded":"2024-02-19T14:59:07.627095+0000","last_fullsized":"2024-02-19T14:59:07.627095+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629907+0000","last_change":"2024-02-19T14:59:07.629907+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629907+0000","last_undegraded":"2024-02-19T14:59:07.629907+0000","last_fullsized":"2024-02-19T14:59:07.629907+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640087+0000","last_change":"2024-02-19T14:59:07.640087+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640087+0000","last_undegraded":"2024-02-19T14:59:07.640087+0000","last_fullsized":"2024-02-19T14:59:07.640087+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626117+0000","last_change":"2024-02-19T14:59:07.626117+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626117+0000","last_undegraded":"2024-02-19T14:59:07.626117+0000","last_fullsized":"2024-02-19T14:59:07.626117+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626189+0000","last_change":"2024-02-19T14:59:07.626189+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626189+0000","last_undegraded":"2024-02-19T14:59:07.626189+0000","last_fullsized":"2024-02-19T14:59:07.626189+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628611+0000","last_change":"2024-02-19T14:59:07.628611+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628611+0000","last_undegraded":"2024-02-19T14:59:07.628611+0000","last_fullsized":"2024-02-19T14:59:07.628611+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640069+0000","last_change":"2024-02-19T14:59:07.640069+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640069+0000","last_undegraded":"2024-02-19T14:59:07.640069+0000","last_fullsized":"2024-02-19T14:59:07.640069+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629382+0000","last_change":"2024-02-19T14:59:07.629382+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629382+0000","last_undegraded":"2024-02-19T14:59:07.629382+0000","last_fullsized":"2024-02-19T14:59:07.629382+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634618+0000","last_change":"2024-02-19T14:59:07.634618+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634618+0000","last_undegraded":"2024-02-19T14:59:07.634618+0000","last_fullsized":"2024-02-19T14:59:07.634618+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637179+0000","last_change":"2024-02-19T14:59:07.637179+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637179+0000","last_undegraded":"2024-02-19T14:59:07.637179+0000","last_fullsized":"2024-02-19T14:59:07.637179+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626760+0000","last_change":"2024-02-19T14:59:07.626760+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626760+0000","last_undegraded":"2024-02-19T14:59:07.626760+0000","last_fullsized":"2024-02-19T14:59:07.626760+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639587+0000","last_change":"2024-02-19T14:59:07.639587+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639587+0000","last_undegraded":"2024-02-19T14:59:07.639587+0000","last_fullsized":"2024-02-19T14:59:07.639587+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628891+0000","last_change":"2024-02-19T14:59:07.628891+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628891+0000","last_undegraded":"2024-02-19T14:59:07.628891+0000","last_fullsized":"2024-02-19T14:59:07.628891+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637655+0000","last_change":"2024-02-19T14:59:07.637655+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637655+0000","last_undegraded":"2024-02-19T14:59:07.637655+0000","last_fullsized":"2024-02-19T14:59:07.637655+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635774+0000","last_change":"2024-02-19T14:59:07.635774+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635774+0000","last_undegraded":"2024-02-19T14:59:07.635774+0000","last_fullsized":"2024-02-19T14:59:07.635774+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628909+0000","last_change":"2024-02-19T14:59:07.628909+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628909+0000","last_undegraded":"2024-02-19T14:59:07.628909+0000","last_fullsized":"2024-02-19T14:59:07.628909+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639159+0000","last_change":"2024-02-19T14:59:07.639159+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639159+0000","last_undegraded":"2024-02-19T14:59:07.639159+0000","last_fullsized":"2024-02-19T14:59:07.639159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636673+0000","last_change":"2024-02-19T14:59:07.636673+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636673+0000","last_undegraded":"2024-02-19T14:59:07.636673+0000","last_fullsized":"2024-02-19T14:59:07.636673+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627251+0000","last_change":"2024-02-19T14:59:07.627251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627251+0000","last_undegraded":"2024-02-19T14:59:07.627251+0000","last_fullsized":"2024-02-19T14:59:07.627251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629094+0000","last_change":"2024-02-19T14:59:07.629094+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629094+0000","last_undegraded":"2024-02-19T14:59:07.629094+0000","last_fullsized":"2024-02-19T14:59:07.629094+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635444+0000","last_change":"2024-02-19T14:59:07.635444+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635444+0000","last_undegraded":"2024-02-19T14:59:07.635444+0000","last_fullsized":"2024-02-19T14:59:07.635444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635646+0000","last_change":"2024-02-19T14:59:07.635646+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635646+0000","last_undegraded":"2024-02-19T14:59:07.635646+0000","last_fullsized":"2024-02-19T14:59:07.635646+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633781+0000","last_change":"2024-02-19T14:59:07.633781+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633781+0000","last_undegraded":"2024-02-19T14:59:07.633781+0000","last_fullsized":"2024-02-19T14:59:07.633781+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626539+0000","last_change":"2024-02-19T14:59:07.626539+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626539+0000","last_undegraded":"2024-02-19T14:59:07.626539+0000","last_fullsized":"2024-02-19T14:59:07.626539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626634+0000","last_change":"2024-02-19T14:59:07.626634+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626634+0000","last_undegraded":"2024-02-19T14:59:07.626634+0000","last_fullsized":"2024-02-19T14:59:07.626634+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634974+0000","last_change":"2024-02-19T14:59:07.634974+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634974+0000","last_undegraded":"2024-02-19T14:59:07.634974+0000","last_fullsized":"2024-02-19T14:59:07.634974+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628304+0000","last_change":"2024-02-19T14:59:07.628304+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628304+0000","last_undegraded":"2024-02-19T14:59:07.628304+0000","last_fullsized":"2024-02-19T14:59:07.628304+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638306+0000","last_change":"2024-02-19T14:59:07.638306+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638306+0000","last_undegraded":"2024-02-19T14:59:07.638306+0000","last_fullsized":"2024-02-19T14:59:07.638306+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629327+0000","last_change":"2024-02-19T14:59:07.629327+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629327+0000","last_undegraded":"2024-02-19T14:59:07.629327+0000","last_fullsized":"2024-02-19T14:59:07.629327+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629097+0000","last_change":"2024-02-19T14:59:07.629097+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629097+0000","last_undegraded":"2024-02-19T14:59:07.629097+0000","last_fullsized":"2024-02-19T14:59:07.629097+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633321+0000","last_change":"2024-02-19T14:59:07.633321+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633321+0000","last_undegraded":"2024-02-19T14:59:07.633321+0000","last_fullsized":"2024-02-19T14:59:07.633321+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628422+0000","last_change":"2024-02-19T14:59:07.628422+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628422+0000","last_undegraded":"2024-02-19T14:59:07.628422+0000","last_fullsized":"2024-02-19T14:59:07.628422+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634580+0000","last_change":"2024-02-19T14:59:07.634580+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634580+0000","last_undegraded":"2024-02-19T14:59:07.634580+0000","last_fullsized":"2024-02-19T14:59:07.634580+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640931+0000","last_change":"2024-02-19T14:59:07.638344+0000","last_active":"2024-02-19T14:59:07.640931+0000","last_peered":"2024-02-19T14:59:07.640931+0000","last_clean":"2024-02-19T14:59:07.640931+0000","last_became_active":"2024-02-19T14:59:07.638065+0000","last_became_peered":"2024-02-19T14:59:07.638065+0000","last_unstale":"2024-02-19T14:59:07.640931+0000","last_undegraded":"2024-02-19T14:59:07.640931+0000","last_fullsized":"2024-02-19T14:59:07.640931+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:55.830792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635080+0000","last_change":"2024-02-19T14:59:07.635080+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635080+0000","last_undegraded":"2024-02-19T14:59:07.635080+0000","last_fullsized":"2024-02-19T14:59:07.635080+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644762+0000","last_change":"2024-02-19T14:59:07.639820+0000","last_active":"2024-02-19T14:59:07.644762+0000","last_peered":"2024-02-19T14:59:07.644762+0000","last_clean":"2024-02-19T14:59:07.644762+0000","last_became_active":"2024-02-19T14:59:07.639571+0000","last_became_peered":"2024-02-19T14:59:07.639571+0000","last_unstale":"2024-02-19T14:59:07.644762+0000","last_undegraded":"2024-02-19T14:59:07.644762+0000","last_fullsized":"2024-02-19T14:59:07.644762+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:26:02.808996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637145+0000","last_change":"2024-02-19T14:59:07.637145+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637145+0000","last_undegraded":"2024-02-19T14:59:07.637145+0000","last_fullsized":"2024-02-19T14:59:07.637145+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642076+0000","last_change":"2024-02-19T14:59:07.641532+0000","last_active":"2024-02-19T14:59:07.642076+0000","last_peered":"2024-02-19T14:59:07.642076+0000","last_clean":"2024-02-19T14:59:07.642076+0000","last_became_active":"2024-02-19T14:59:07.641278+0000","last_became_peered":"2024-02-19T14:59:07.641278+0000","last_unstale":"2024-02-19T14:59:07.642076+0000","last_undegraded":"2024-02-19T14:59:07.642076+0000","last_fullsized":"2024-02-19T14:59:07.642076+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:57.560202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633978+0000","last_change":"2024-02-19T14:59:07.633978+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633978+0000","last_undegraded":"2024-02-19T14:59:07.633978+0000","last_fullsized":"2024-02-19T14:59:07.633978+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646713+0000","last_change":"2024-02-19T14:59:07.646204+0000","last_active":"2024-02-19T14:59:07.646713+0000","last_peered":"2024-02-19T14:59:07.646713+0000","last_clean":"2024-02-19T14:59:07.646713+0000","last_became_active":"2024-02-19T14:59:07.645880+0000","last_became_peered":"2024-02-19T14:59:07.645880+0000","last_unstale":"2024-02-19T14:59:07.646713+0000","last_undegraded":"2024-02-19T14:59:07.646713+0000","last_fullsized":"2024-02-19T14:59:07.646713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637251+0000","last_change":"2024-02-19T14:59:07.637251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637251+0000","last_undegraded":"2024-02-19T14:59:07.637251+0000","last_fullsized":"2024-02-19T14:59:07.637251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644114+0000","last_change":"2024-02-19T14:59:07.642010+0000","last_active":"2024-02-19T14:59:07.644114+0000","last_peered":"2024-02-19T14:59:07.644114+0000","last_clean":"2024-02-19T14:59:07.644114+0000","last_became_active":"2024-02-19T14:59:07.641735+0000","last_became_peered":"2024-02-19T14:59:07.641735+0000","last_unstale":"2024-02-19T14:59:07.644114+0000","last_undegraded":"2024-02-19T14:59:07.644114+0000","last_fullsized":"2024-02-19T14:59:07.644114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:05.299812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635252+0000","last_change":"2024-02-19T14:59:07.635252+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635252+0000","last_undegraded":"2024-02-19T14:59:07.635252+0000","last_fullsized":"2024-02-19T14:59:07.635252+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644650+0000","last_change":"2024-02-19T14:59:07.643447+0000","last_active":"2024-02-19T14:59:07.644650+0000","last_peered":"2024-02-19T14:59:07.644650+0000","last_clean":"2024-02-19T14:59:07.644650+0000","last_became_active":"2024-02-19T14:59:07.643150+0000","last_became_peered":"2024-02-19T14:59:07.643150+0000","last_unstale":"2024-02-19T14:59:07.644650+0000","last_undegraded":"2024-02-19T14:59:07.644650+0000","last_fullsized":"2024-02-19T14:59:07.644650+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:37.234470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634275+0000","last_change":"2024-02-19T14:59:07.634275+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634275+0000","last_undegraded":"2024-02-19T14:59:07.634275+0000","last_fullsized":"2024-02-19T14:59:07.634275+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644014+0000","last_change":"2024-02-19T14:59:07.641207+0000","last_active":"2024-02-19T14:59:07.644014+0000","last_peered":"2024-02-19T14:59:07.644014+0000","last_clean":"2024-02-19T14:59:07.644014+0000","last_became_active":"2024-02-19T14:59:07.640874+0000","last_became_peered":"2024-02-19T14:59:07.640874+0000","last_unstale":"2024-02-19T14:59:07.644014+0000","last_undegraded":"2024-02-19T14:59:07.644014+0000","last_fullsized":"2024-02-19T14:59:07.644014+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:16:13.230015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634281+0000","last_change":"2024-02-19T14:59:07.634281+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634281+0000","last_undegraded":"2024-02-19T14:59:07.634281+0000","last_fullsized":"2024-02-19T14:59:07.634281+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641906+0000","last_change":"2024-02-19T14:59:07.638597+0000","last_active":"2024-02-19T14:59:07.641906+0000","last_peered":"2024-02-19T14:59:07.641906+0000","last_clean":"2024-02-19T14:59:07.641906+0000","last_became_active":"2024-02-19T14:59:07.638358+0000","last_became_peered":"2024-02-19T14:59:07.638358+0000","last_unstale":"2024-02-19T14:59:07.641906+0000","last_undegraded":"2024-02-19T14:59:07.641906+0000","last_fullsized":"2024-02-19T14:59:07.641906+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:21.669002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634738+0000","last_change":"2024-02-19T14:59:07.634738+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634738+0000","last_undegraded":"2024-02-19T14:59:07.634738+0000","last_fullsized":"2024-02-19T14:59:07.634738+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644861+0000","last_change":"2024-02-19T14:59:07.643687+0000","last_active":"2024-02-19T14:59:07.644861+0000","last_peered":"2024-02-19T14:59:07.644861+0000","last_clean":"2024-02-19T14:59:07.644861+0000","last_became_active":"2024-02-19T14:59:07.643136+0000","last_became_peered":"2024-02-19T14:59:07.643136+0000","last_unstale":"2024-02-19T14:59:07.644861+0000","last_undegraded":"2024-02-19T14:59:07.644861+0000","last_fullsized":"2024-02-19T14:59:07.644861+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:28:01.427001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633392+0000","last_change":"2024-02-19T14:59:07.633392+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633392+0000","last_undegraded":"2024-02-19T14:59:07.633392+0000","last_fullsized":"2024-02-19T14:59:07.633392+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641877+0000","last_change":"2024-02-19T14:59:07.638366+0000","last_active":"2024-02-19T14:59:07.641877+0000","last_peered":"2024-02-19T14:59:07.641877+0000","last_clean":"2024-02-19T14:59:07.641877+0000","last_became_active":"2024-02-19T14:59:07.638109+0000","last_became_peered":"2024-02-19T14:59:07.638109+0000","last_unstale":"2024-02-19T14:59:07.641877+0000","last_undegraded":"2024-02-19T14:59:07.641877+0000","last_fullsized":"2024-02-19T14:59:07.641877+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627740+0000","last_change":"2024-02-19T14:59:07.627740+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627740+0000","last_undegraded":"2024-02-19T14:59:07.627740+0000","last_fullsized":"2024-02-19T14:59:07.627740+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647367+0000","last_change":"2024-02-19T14:59:07.646216+0000","last_active":"2024-02-19T14:59:07.647367+0000","last_peered":"2024-02-19T14:59:07.647367+0000","last_clean":"2024-02-19T14:59:07.647367+0000","last_became_active":"2024-02-19T14:59:07.645915+0000","last_became_peered":"2024-02-19T14:59:07.645915+0000","last_unstale":"2024-02-19T14:59:07.647367+0000","last_undegraded":"2024-02-19T14:59:07.647367+0000","last_fullsized":"2024-02-19T14:59:07.647367+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:28:14.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630383+0000","last_change":"2024-02-19T14:59:07.630383+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630383+0000","last_undegraded":"2024-02-19T14:59:07.630383+0000","last_fullsized":"2024-02-19T14:59:07.630383+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647108+0000","last_change":"2024-02-19T14:59:07.643409+0000","last_active":"2024-02-19T14:59:07.647108+0000","last_peered":"2024-02-19T14:59:07.647108+0000","last_clean":"2024-02-19T14:59:07.647108+0000","last_became_active":"2024-02-19T14:59:07.643105+0000","last_became_peered":"2024-02-19T14:59:07.643105+0000","last_unstale":"2024-02-19T14:59:07.647108+0000","last_undegraded":"2024-02-19T14:59:07.647108+0000","last_fullsized":"2024-02-19T14:59:07.647108+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:55.853954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633352+0000","last_change":"2024-02-19T14:59:07.633352+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633352+0000","last_undegraded":"2024-02-19T14:59:07.633352+0000","last_fullsized":"2024-02-19T14:59:07.633352+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640331+0000","last_change":"2024-02-19T14:59:07.639087+0000","last_active":"2024-02-19T14:59:07.640331+0000","last_peered":"2024-02-19T14:59:07.640331+0000","last_clean":"2024-02-19T14:59:07.640331+0000","last_became_active":"2024-02-19T14:59:07.638819+0000","last_became_peered":"2024-02-19T14:59:07.638819+0000","last_unstale":"2024-02-19T14:59:07.640331+0000","last_undegraded":"2024-02-19T14:59:07.640331+0000","last_fullsized":"2024-02-19T14:59:07.640331+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632905+0000","last_change":"2024-02-19T14:59:07.632905+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632905+0000","last_undegraded":"2024-02-19T14:59:07.632905+0000","last_fullsized":"2024-02-19T14:59:07.632905+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637435+0000","last_change":"2024-02-19T14:59:07.636744+0000","last_active":"2024-02-19T14:59:07.637435+0000","last_peered":"2024-02-19T14:59:07.637435+0000","last_clean":"2024-02-19T14:59:07.637435+0000","last_became_active":"2024-02-19T14:59:07.636464+0000","last_became_peered":"2024-02-19T14:59:07.636464+0000","last_unstale":"2024-02-19T14:59:07.637435+0000","last_undegraded":"2024-02-19T14:59:07.637435+0000","last_fullsized":"2024-02-19T14:59:07.637435+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:30.048428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633393+0000","last_change":"2024-02-19T14:59:07.633393+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633393+0000","last_undegraded":"2024-02-19T14:59:07.633393+0000","last_fullsized":"2024-02-19T14:59:07.633393+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646942+0000","last_change":"2024-02-19T14:59:07.643659+0000","last_active":"2024-02-19T14:59:07.646942+0000","last_peered":"2024-02-19T14:59:07.646942+0000","last_clean":"2024-02-19T14:59:07.646942+0000","last_became_active":"2024-02-19T14:59:07.643126+0000","last_became_peered":"2024-02-19T14:59:07.643126+0000","last_unstale":"2024-02-19T14:59:07.646942+0000","last_undegraded":"2024-02-19T14:59:07.646942+0000","last_fullsized":"2024-02-19T14:59:07.646942+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:30.093846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630599+0000","last_change":"2024-02-19T14:59:07.630599+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630599+0000","last_undegraded":"2024-02-19T14:59:07.630599+0000","last_fullsized":"2024-02-19T14:59:07.630599+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644152+0000","last_change":"2024-02-19T14:59:07.642484+0000","last_active":"2024-02-19T14:59:07.644152+0000","last_peered":"2024-02-19T14:59:07.644152+0000","last_clean":"2024-02-19T14:59:07.644152+0000","last_became_active":"2024-02-19T14:59:07.642254+0000","last_became_peered":"2024-02-19T14:59:07.642254+0000","last_unstale":"2024-02-19T14:59:07.644152+0000","last_undegraded":"2024-02-19T14:59:07.644152+0000","last_fullsized":"2024-02-19T14:59:07.644152+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:19.998342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626772+0000","last_change":"2024-02-19T14:59:07.626772+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626772+0000","last_undegraded":"2024-02-19T14:59:07.626772+0000","last_fullsized":"2024-02-19T14:59:07.626772+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.645052+0000","last_change":"2024-02-19T14:59:07.644552+0000","last_active":"2024-02-19T14:59:07.645052+0000","last_peered":"2024-02-19T14:59:07.645052+0000","last_clean":"2024-02-19T14:59:07.645052+0000","last_became_active":"2024-02-19T14:59:07.644274+0000","last_became_peered":"2024-02-19T14:59:07.644274+0000","last_unstale":"2024-02-19T14:59:07.645052+0000","last_undegraded":"2024-02-19T14:59:07.645052+0000","last_fullsized":"2024-02-19T14:59:07.645052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:31:22.200914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632440+0000","last_change":"2024-02-19T14:59:07.632440+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632440+0000","last_undegraded":"2024-02-19T14:59:07.632440+0000","last_fullsized":"2024-02-19T14:59:07.632440+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642826+0000","last_change":"2024-02-19T14:59:07.638149+0000","last_active":"2024-02-19T14:59:07.642826+0000","last_peered":"2024-02-19T14:59:07.642826+0000","last_clean":"2024-02-19T14:59:07.642826+0000","last_became_active":"2024-02-19T14:59:07.637845+0000","last_became_peered":"2024-02-19T14:59:07.637845+0000","last_unstale":"2024-02-19T14:59:07.642826+0000","last_undegraded":"2024-02-19T14:59:07.642826+0000","last_fullsized":"2024-02-19T14:59:07.642826+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:47.116497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634012+0000","last_change":"2024-02-19T14:59:07.634012+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634012+0000","last_undegraded":"2024-02-19T14:59:07.634012+0000","last_fullsized":"2024-02-19T14:59:07.634012+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647966+0000","last_change":"2024-02-19T14:59:07.645385+0000","last_active":"2024-02-19T14:59:07.647966+0000","last_peered":"2024-02-19T14:59:07.647966+0000","last_clean":"2024-02-19T14:59:07.647966+0000","last_became_active":"2024-02-19T14:59:07.644999+0000","last_became_peered":"2024-02-19T14:59:07.644999+0000","last_unstale":"2024-02-19T14:59:07.647966+0000","last_undegraded":"2024-02-19T14:59:07.647966+0000","last_fullsized":"2024-02-19T14:59:07.647966+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:03:16.898728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636317+0000","last_change":"2024-02-19T14:59:07.636317+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636317+0000","last_undegraded":"2024-02-19T14:59:07.636317+0000","last_fullsized":"2024-02-19T14:59:07.636317+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.648114+0000","last_change":"2024-02-19T14:59:07.646936+0000","last_active":"2024-02-19T14:59:07.648114+0000","last_peered":"2024-02-19T14:59:07.648114+0000","last_clean":"2024-02-19T14:59:07.648114+0000","last_became_active":"2024-02-19T14:59:07.645710+0000","last_became_peered":"2024-02-19T14:59:07.645710+0000","last_unstale":"2024-02-19T14:59:07.648114+0000","last_undegraded":"2024-02-19T14:59:07.648114+0000","last_fullsized":"2024-02-19T14:59:07.648114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:44.397134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626291+0000","last_change":"2024-02-19T14:59:07.626291+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626291+0000","last_undegraded":"2024-02-19T14:59:07.626291+0000","last_fullsized":"2024-02-19T14:59:07.626291+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.638399+0000","last_change":"2024-02-19T14:59:07.636425+0000","last_active":"2024-02-19T14:59:07.638399+0000","last_peered":"2024-02-19T14:59:07.638399+0000","last_clean":"2024-02-19T14:59:07.638399+0000","last_became_active":"2024-02-19T14:59:07.636154+0000","last_became_peered":"2024-02-19T14:59:07.636154+0000","last_unstale":"2024-02-19T14:59:07.638399+0000","last_undegraded":"2024-02-19T14:59:07.638399+0000","last_fullsized":"2024-02-19T14:59:07.638399+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:41.446210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630391+0000","last_change":"2024-02-19T14:59:07.630391+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630391+0000","last_undegraded":"2024-02-19T14:59:07.630391+0000","last_fullsized":"2024-02-19T14:59:07.630391+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640425+0000","last_change":"2024-02-19T14:59:07.638740+0000","last_active":"2024-02-19T14:59:07.640425+0000","last_peered":"2024-02-19T14:59:07.640425+0000","last_clean":"2024-02-19T14:59:07.640425+0000","last_became_active":"2024-02-19T14:59:07.638275+0000","last_became_peered":"2024-02-19T14:59:07.638275+0000","last_unstale":"2024-02-19T14:59:07.640425+0000","last_undegraded":"2024-02-19T14:59:07.640425+0000","last_fullsized":"2024-02-19T14:59:07.640425+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631957+0000","last_change":"2024-02-19T14:59:07.631957+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631957+0000","last_undegraded":"2024-02-19T14:59:07.631957+0000","last_fullsized":"2024-02-19T14:59:07.631957+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641644+0000","last_change":"2024-02-19T14:59:07.641182+0000","last_active":"2024-02-19T14:59:07.641644+0000","last_peered":"2024-02-19T14:59:07.641644+0000","last_clean":"2024-02-19T14:59:07.641644+0000","last_became_active":"2024-02-19T14:59:07.640867+0000","last_became_peered":"2024-02-19T14:59:07.640867+0000","last_unstale":"2024-02-19T14:59:07.641644+0000","last_undegraded":"2024-02-19T14:59:07.641644+0000","last_fullsized":"2024-02-19T14:59:07.641644+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:59.660494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629585+0000","last_change":"2024-02-19T14:59:07.629585+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629585+0000","last_undegraded":"2024-02-19T14:59:07.629585+0000","last_fullsized":"2024-02-19T14:59:07.629585+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643474+0000","last_change":"2024-02-19T14:59:07.640656+0000","last_active":"2024-02-19T14:59:07.643474+0000","last_peered":"2024-02-19T14:59:07.643474+0000","last_clean":"2024-02-19T14:59:07.643474+0000","last_became_active":"2024-02-19T14:59:07.640382+0000","last_became_peered":"2024-02-19T14:59:07.640382+0000","last_unstale":"2024-02-19T14:59:07.643474+0000","last_undegraded":"2024-02-19T14:59:07.643474+0000","last_fullsized":"2024-02-19T14:59:07.643474+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:43:14.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626274+0000","last_change":"2024-02-19T14:59:07.626274+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626274+0000","last_undegraded":"2024-02-19T14:59:07.626274+0000","last_fullsized":"2024-02-19T14:59:07.626274+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640387+0000","last_change":"2024-02-19T14:59:07.638470+0000","last_active":"2024-02-19T14:59:07.640387+0000","last_peered":"2024-02-19T14:59:07.640387+0000","last_clean":"2024-02-19T14:59:07.640387+0000","last_became_active":"2024-02-19T14:59:07.638197+0000","last_became_peered":"2024-02-19T14:59:07.638197+0000","last_unstale":"2024-02-19T14:59:07.640387+0000","last_undegraded":"2024-02-19T14:59:07.640387+0000","last_fullsized":"2024-02-19T14:59:07.640387+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:09.994584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630266+0000","last_change":"2024-02-19T14:59:07.630266+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630266+0000","last_undegraded":"2024-02-19T14:59:07.630266+0000","last_fullsized":"2024-02-19T14:59:07.630266+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643869+0000","last_change":"2024-02-19T14:59:07.640562+0000","last_active":"2024-02-19T14:59:07.643869+0000","last_peered":"2024-02-19T14:59:07.643869+0000","last_clean":"2024-02-19T14:59:07.643869+0000","last_became_active":"2024-02-19T14:59:07.640288+0000","last_became_peered":"2024-02-19T14:59:07.640288+0000","last_unstale":"2024-02-19T14:59:07.643869+0000","last_undegraded":"2024-02-19T14:59:07.643869+0000","last_fullsized":"2024-02-19T14:59:07.643869+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634982+0000","last_change":"2024-02-19T14:59:07.634982+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634982+0000","last_undegraded":"2024-02-19T14:59:07.634982+0000","last_fullsized":"2024-02-19T14:59:07.634982+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647753+0000","last_change":"2024-02-19T14:59:07.645290+0000","last_active":"2024-02-19T14:59:07.647753+0000","last_peered":"2024-02-19T14:59:07.647753+0000","last_clean":"2024-02-19T14:59:07.647753+0000","last_became_active":"2024-02-19T14:59:07.644933+0000","last_became_peered":"2024-02-19T14:59:07.644933+0000","last_unstale":"2024-02-19T14:59:07.647753+0000","last_undegraded":"2024-02-19T14:59:07.647753+0000","last_fullsized":"2024-02-19T14:59:07.647753+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:37.251826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630878+0000","last_change":"2024-02-19T14:59:07.630878+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630878+0000","last_undegraded":"2024-02-19T14:59:07.630878+0000","last_fullsized":"2024-02-19T14:59:07.630878+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647675+0000","last_change":"2024-02-19T14:59:07.643990+0000","last_active":"2024-02-19T14:59:07.647675+0000","last_peered":"2024-02-19T14:59:07.647675+0000","last_clean":"2024-02-19T14:59:07.647675+0000","last_became_active":"2024-02-19T14:59:07.643708+0000","last_became_peered":"2024-02-19T14:59:07.643708+0000","last_unstale":"2024-02-19T14:59:07.647675+0000","last_undegraded":"2024-02-19T14:59:07.647675+0000","last_fullsized":"2024-02-19T14:59:07.647675+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:00.752877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631620+0000","last_change":"2024-02-19T14:59:07.631620+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631620+0000","last_undegraded":"2024-02-19T14:59:07.631620+0000","last_fullsized":"2024-02-19T14:59:07.631620+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637822+0000","last_change":"2024-02-19T14:59:07.634059+0000","last_active":"2024-02-19T14:59:07.637822+0000","last_peered":"2024-02-19T14:59:07.637822+0000","last_clean":"2024-02-19T14:59:07.637822+0000","last_became_active":"2024-02-19T14:59:07.633768+0000","last_became_peered":"2024-02-19T14:59:07.633768+0000","last_unstale":"2024-02-19T14:59:07.637822+0000","last_undegraded":"2024-02-19T14:59:07.637822+0000","last_fullsized":"2024-02-19T14:59:07.637822+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:52.944510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633788+0000","last_change":"2024-02-19T14:59:07.633788+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633788+0000","last_undegraded":"2024-02-19T14:59:07.633788+0000","last_fullsized":"2024-02-19T14:59:07.633788+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.636397+0000","last_change":"2024-02-19T14:59:07.632792+0000","last_active":"2024-02-19T14:59:07.636397+0000","last_peered":"2024-02-19T14:59:07.636397+0000","last_clean":"2024-02-19T14:59:07.636397+0000","last_became_active":"2024-02-19T14:59:07.632529+0000","last_became_peered":"2024-02-19T14:59:07.632529+0000","last_unstale":"2024-02-19T14:59:07.636397+0000","last_undegraded":"2024-02-19T14:59:07.636397+0000","last_fullsized":"2024-02-19T14:59:07.636397+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638240+0000","last_change":"2024-02-19T14:59:07.638240+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638240+0000","last_undegraded":"2024-02-19T14:59:07.638240+0000","last_fullsized":"2024-02-19T14:59:07.638240+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646917+0000","last_change":"2024-02-19T14:59:07.643336+0000","last_active":"2024-02-19T14:59:07.646917+0000","last_peered":"2024-02-19T14:59:07.646917+0000","last_clean":"2024-02-19T14:59:07.646917+0000","last_became_active":"2024-02-19T14:59:07.642778+0000","last_became_peered":"2024-02-19T14:59:07.642778+0000","last_unstale":"2024-02-19T14:59:07.646917+0000","last_undegraded":"2024-02-19T14:59:07.646917+0000","last_fullsized":"2024-02-19T14:59:07.646917+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:36.331774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634502+0000","last_change":"2024-02-19T14:59:07.634502+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634502+0000","last_undegraded":"2024-02-19T14:59:07.634502+0000","last_fullsized":"2024-02-19T14:59:07.634502+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641427+0000","last_change":"2024-02-19T14:59:07.636409+0000","last_active":"2024-02-19T14:59:07.641427+0000","last_peered":"2024-02-19T14:59:07.641427+0000","last_clean":"2024-02-19T14:59:07.641427+0000","last_became_active":"2024-02-19T14:59:07.636136+0000","last_became_peered":"2024-02-19T14:59:07.636136+0000","last_unstale":"2024-02-19T14:59:07.641427+0000","last_undegraded":"2024-02-19T14:59:07.641427+0000","last_fullsized":"2024-02-19T14:59:07.641427+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":480,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.627573+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:07.627573+0000","last_peered":"2024-02-19T14:59:07.627573+0000","last_clean":"2024-02-19T14:59:07.627573+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:07.627573+0000","last_undegraded":"2024-02-19T14:59:07.627573+0000","last_fullsized":"2024-02-19T14:59:07.627573+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34060,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337780,"statfs":{"total":96636764160,"available":96601886720,"internally_reserved":0,"allocated":2949120,"data_stored":1741638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.505,"15min":0.495},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51600000000000001,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.748,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69299999999999995,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.627,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.629}]},{"osd":4,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46200000000000002,"15min":0.44900000000000001},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.753,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.499,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.75,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.79000000000000004,"5min":0.871,"15min":0.871},"last":0.498}]},{"osd":6,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.84399999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574939,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69299999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.92400000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.876},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.94899999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.68000000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.313},"min":{"1min":0.217,"5min":0.159,"15min":0.159},"max":{"1min":0.45100000000000001,"5min":3.694,"15min":3.694},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36799999999999999,"15min":0.32600000000000001},"min":{"1min":0.248,"5min":0.192,"15min":0.189},"max":{"1min":0.55300000000000005,"5min":4.835,"15min":4.835},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.41699999999999998,"15min":0.378},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.498,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.314}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41399999999999998,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76200000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.621},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78700000000000003,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.41999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72099999999999997,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.374},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.35299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.312},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.748}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.623},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.85799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.32600000000000001},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32300000000000001,"15min":0.33200000000000002},"min":{"1min":0.24299999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.45600000000000002,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36899999999999999,"15min":0.38500000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42199999999999999,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.54900000000000004},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.309},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.96799999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.44800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.56499999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.82399999999999995,"15min":0.996},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.036},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.71799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34048,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337792,"statfs":{"total":96636764160,"available":96601899008,"internally_reserved":0,"allocated":2936832,"data_stored":1734971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:11.956 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.089+0000 7f3272b8a640 1 Processor -- start 2024-02-19T14:59:12.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.089+0000 7f3272b8a640 1 -- start start 2024-02-19T14:59:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3272b8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3272b8a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0c1120 con 0x7f326c0c15b0 2024-02-19T14:59:12.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3272b8a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0c6080 con 0x7f326c0c5270 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3272b8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0c57e0 con 0x7f326c14ece0 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3271127640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f326c0c15b0 0x7f326c0c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39992/0 (socket says 172.21.15.78:39992) 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3271127640 1 -- 172.21.15.78:0/500484121 learned_addr learned my addr 172.21.15.78:0/500484121 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f3270926640 1 --2- 172.21.15.78:0/500484121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f326b7fe640 1 -- 172.21.15.78:0/500484121 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256558618 0 0) 0x7f326c0c1120 con 0x7f326c0c15b0 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.090+0000 7f326b7fe640 1 -- 172.21.15.78:0/500484121 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3254003520 con 0x7f326c0c15b0 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.091+0000 7f326b7fe640 1 -- 172.21.15.78:0/500484121 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904567836 0 0) 0x7f326c0c6080 con 0x7f326c0c5270 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.091+0000 7f3270926640 1 -- 172.21.15.78:0/500484121 >> v1:172.21.15.150:6789/0 conn(0x7f326c0c5270 legacy=0x7f326c0719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.091+0000 7f3270926640 1 -- 172.21.15.78:0/500484121 >> v1:172.21.15.78:6789/0 conn(0x7f326c0c15b0 legacy=0x7f326c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.091+0000 7f3270926640 1 -- 172.21.15.78:0/500484121 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0c5ac0 con 0x7f326c14ece0 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.091+0000 7f3270926640 1 --2- 172.21.15.78:0/500484121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c0c4d30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f325c0097d0 tx=0x7f325c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f9fcb9c8963ed3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f326b7fe640 1 -- 172.21.15.78:0/500484121 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c031890 con 0x7f326c14ece0 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f326b7fe640 1 -- 172.21.15.78:0/500484121 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c0319f0 con 0x7f326c14ece0 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f326b7fe640 1 -- 172.21.15.78:0/500484121 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c031d80 con 0x7f326c14ece0 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 -- 172.21.15.78:0/500484121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 msgr2=0x7f326c0c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 --2- 172.21.15.78:0/500484121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c0c4d30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f325c0097d0 tx=0x7f325c02b930 comp rx=0 tx=0).stop 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 -- 172.21.15.78:0/500484121 shutdown_connections 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 --2- 172.21.15.78:0/500484121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c0c4d30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 -- 172.21.15.78:0/500484121 >> 172.21.15.78:0/500484121 conn(0x7f326c06b270 msgr2=0x7f326c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 -- 172.21.15.78:0/500484121 shutdown_connections 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.092+0000 7f3272b8a640 1 -- 172.21.15.78:0/500484121 wait complete. 2024-02-19T14:59:12.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.093+0000 7f3272b8a640 1 Processor -- start 2024-02-19T14:59:12.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.093+0000 7f3272b8a640 1 -- start start 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3272b8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 0x7f326c1eaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3272b8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326c0c5270 0x7f326c1eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3272b8a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3272b8a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0bb980 con 0x7f326c0c15b0 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3272b8a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0abce0 con 0x7f326c0c5270 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3272b8a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c150890 con 0x7f326c14ece0 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 0x7f326c1eaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 0x7f326c1eaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33018/0 (socket says 172.21.15.78:33018) 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 -- 172.21.15.78:0/3749716214 learned_addr learned my addr 172.21.15.78:0/3749716214 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f326bfff640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326c0c5270 0x7f326c1eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3271127640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 -- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 msgr2=0x7f326c150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 -- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326c0c5270 msgr2=0x7f326c1eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326c0c5270 0x7f326c1eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3270926640 1 -- 172.21.15.78:0/3749716214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c009400 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.094+0000 7f3271127640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.095+0000 7f3270926640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 0x7f326c1eaf10 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f325c034040 tx=0x7f325c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.095+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c004230 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.095+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c0043b0 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.095+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c14f800 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.095+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c004520 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.095+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c0ac210 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.096+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c150660 con 0x7f326c0c15b0 2024-02-19T14:59:12.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.096+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f325c042070 con 0x7f326c0c15b0 2024-02-19T14:59:12.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.097+0000 7f32697fa640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f325806b810 0x7f325806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.097+0000 7f326bfff640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f325806b810 0x7f325806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.097+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f325c0ac6d0 con 0x7f326c0c15b0 2024-02-19T14:59:12.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.097+0000 7f326bfff640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f325806b810 0x7f325806dcd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f326c142f80 tx=0x7f3260009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.101+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f326c150660 con 0x7f326c0c15b0 2024-02-19T14:59:12.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.268+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f326c0c2150 con 0x7f325806b810 2024-02-19T14:59:12.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.275+0000 7f32697fa640 1 -- 172.21.15.78:0/3749716214 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284544 (secure 0 0 0) 0x7f326000f820 con 0x7f325806b810 2024-02-19T14:59:12.281 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:12.282 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f325806b810 msgr2=0x7f325806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f325806b810 0x7f325806dcd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f326c142f80 tx=0x7f3260009040 comp rx=0 tx=0).stop 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 msgr2=0x7f326c1eaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 0x7f326c1eaf10 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f325c034040 tx=0x7f325c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 shutdown_connections 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f325806b810 0x7f325806dcd0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f326c14ece0 0x7f326c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326c0c5270 0x7f326c1eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.278+0000 7f3272b8a640 1 --2- 172.21.15.78:0/3749716214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f326c0c15b0 0x7f326c1eaf10 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.279+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 >> 172.21.15.78:0/3749716214 conn(0x7f326c06b270 msgr2=0x7f326c145790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.279+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 shutdown_connections 2024-02-19T14:59:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.279+0000 7f3272b8a640 1 -- 172.21.15.78:0/3749716214 wait complete. 2024-02-19T14:59:12.292 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-02-19T14:59:12.144424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268968,"kb_used_data":19512,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754705752,"statfs":{"total":773094113280,"available":772818690048,"internally_reserved":0,"allocated":19980288,"data_stored":10364228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.523288"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627095+0000","last_change":"2024-02-19T14:59:07.627095+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627095+0000","last_undegraded":"2024-02-19T14:59:07.627095+0000","last_fullsized":"2024-02-19T14:59:07.627095+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629907+0000","last_change":"2024-02-19T14:59:07.629907+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629907+0000","last_undegraded":"2024-02-19T14:59:07.629907+0000","last_fullsized":"2024-02-19T14:59:07.629907+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640087+0000","last_change":"2024-02-19T14:59:07.640087+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640087+0000","last_undegraded":"2024-02-19T14:59:07.640087+0000","last_fullsized":"2024-02-19T14:59:07.640087+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626117+0000","last_change":"2024-02-19T14:59:07.626117+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626117+0000","last_undegraded":"2024-02-19T14:59:07.626117+0000","last_fullsized":"2024-02-19T14:59:07.626117+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626189+0000","last_change":"2024-02-19T14:59:07.626189+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626189+0000","last_undegraded":"2024-02-19T14:59:07.626189+0000","last_fullsized":"2024-02-19T14:59:07.626189+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628611+0000","last_change":"2024-02-19T14:59:07.628611+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628611+0000","last_undegraded":"2024-02-19T14:59:07.628611+0000","last_fullsized":"2024-02-19T14:59:07.628611+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640069+0000","last_change":"2024-02-19T14:59:07.640069+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640069+0000","last_undegraded":"2024-02-19T14:59:07.640069+0000","last_fullsized":"2024-02-19T14:59:07.640069+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629382+0000","last_change":"2024-02-19T14:59:07.629382+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629382+0000","last_undegraded":"2024-02-19T14:59:07.629382+0000","last_fullsized":"2024-02-19T14:59:07.629382+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634618+0000","last_change":"2024-02-19T14:59:07.634618+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634618+0000","last_undegraded":"2024-02-19T14:59:07.634618+0000","last_fullsized":"2024-02-19T14:59:07.634618+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637179+0000","last_change":"2024-02-19T14:59:07.637179+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637179+0000","last_undegraded":"2024-02-19T14:59:07.637179+0000","last_fullsized":"2024-02-19T14:59:07.637179+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626760+0000","last_change":"2024-02-19T14:59:07.626760+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626760+0000","last_undegraded":"2024-02-19T14:59:07.626760+0000","last_fullsized":"2024-02-19T14:59:07.626760+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639587+0000","last_change":"2024-02-19T14:59:07.639587+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639587+0000","last_undegraded":"2024-02-19T14:59:07.639587+0000","last_fullsized":"2024-02-19T14:59:07.639587+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628891+0000","last_change":"2024-02-19T14:59:07.628891+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628891+0000","last_undegraded":"2024-02-19T14:59:07.628891+0000","last_fullsized":"2024-02-19T14:59:07.628891+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637655+0000","last_change":"2024-02-19T14:59:07.637655+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637655+0000","last_undegraded":"2024-02-19T14:59:07.637655+0000","last_fullsized":"2024-02-19T14:59:07.637655+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635774+0000","last_change":"2024-02-19T14:59:07.635774+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635774+0000","last_undegraded":"2024-02-19T14:59:07.635774+0000","last_fullsized":"2024-02-19T14:59:07.635774+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628909+0000","last_change":"2024-02-19T14:59:07.628909+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628909+0000","last_undegraded":"2024-02-19T14:59:07.628909+0000","last_fullsized":"2024-02-19T14:59:07.628909+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639159+0000","last_change":"2024-02-19T14:59:07.639159+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639159+0000","last_undegraded":"2024-02-19T14:59:07.639159+0000","last_fullsized":"2024-02-19T14:59:07.639159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636673+0000","last_change":"2024-02-19T14:59:07.636673+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636673+0000","last_undegraded":"2024-02-19T14:59:07.636673+0000","last_fullsized":"2024-02-19T14:59:07.636673+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627251+0000","last_change":"2024-02-19T14:59:07.627251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627251+0000","last_undegraded":"2024-02-19T14:59:07.627251+0000","last_fullsized":"2024-02-19T14:59:07.627251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629094+0000","last_change":"2024-02-19T14:59:07.629094+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629094+0000","last_undegraded":"2024-02-19T14:59:07.629094+0000","last_fullsized":"2024-02-19T14:59:07.629094+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635444+0000","last_change":"2024-02-19T14:59:07.635444+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635444+0000","last_undegraded":"2024-02-19T14:59:07.635444+0000","last_fullsized":"2024-02-19T14:59:07.635444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635646+0000","last_change":"2024-02-19T14:59:07.635646+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635646+0000","last_undegraded":"2024-02-19T14:59:07.635646+0000","last_fullsized":"2024-02-19T14:59:07.635646+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633781+0000","last_change":"2024-02-19T14:59:07.633781+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633781+0000","last_undegraded":"2024-02-19T14:59:07.633781+0000","last_fullsized":"2024-02-19T14:59:07.633781+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626539+0000","last_change":"2024-02-19T14:59:07.626539+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626539+0000","last_undegraded":"2024-02-19T14:59:07.626539+0000","last_fullsized":"2024-02-19T14:59:07.626539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626634+0000","last_change":"2024-02-19T14:59:07.626634+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626634+0000","last_undegraded":"2024-02-19T14:59:07.626634+0000","last_fullsized":"2024-02-19T14:59:07.626634+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634974+0000","last_change":"2024-02-19T14:59:07.634974+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634974+0000","last_undegraded":"2024-02-19T14:59:07.634974+0000","last_fullsized":"2024-02-19T14:59:07.634974+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628304+0000","last_change":"2024-02-19T14:59:07.628304+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628304+0000","last_undegraded":"2024-02-19T14:59:07.628304+0000","last_fullsized":"2024-02-19T14:59:07.628304+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638306+0000","last_change":"2024-02-19T14:59:07.638306+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638306+0000","last_undegraded":"2024-02-19T14:59:07.638306+0000","last_fullsized":"2024-02-19T14:59:07.638306+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629327+0000","last_change":"2024-02-19T14:59:07.629327+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629327+0000","last_undegraded":"2024-02-19T14:59:07.629327+0000","last_fullsized":"2024-02-19T14:59:07.629327+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629097+0000","last_change":"2024-02-19T14:59:07.629097+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629097+0000","last_undegraded":"2024-02-19T14:59:07.629097+0000","last_fullsized":"2024-02-19T14:59:07.629097+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633321+0000","last_change":"2024-02-19T14:59:07.633321+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633321+0000","last_undegraded":"2024-02-19T14:59:07.633321+0000","last_fullsized":"2024-02-19T14:59:07.633321+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628422+0000","last_change":"2024-02-19T14:59:07.628422+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628422+0000","last_undegraded":"2024-02-19T14:59:07.628422+0000","last_fullsized":"2024-02-19T14:59:07.628422+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634580+0000","last_change":"2024-02-19T14:59:07.634580+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634580+0000","last_undegraded":"2024-02-19T14:59:07.634580+0000","last_fullsized":"2024-02-19T14:59:07.634580+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640931+0000","last_change":"2024-02-19T14:59:07.638344+0000","last_active":"2024-02-19T14:59:07.640931+0000","last_peered":"2024-02-19T14:59:07.640931+0000","last_clean":"2024-02-19T14:59:07.640931+0000","last_became_active":"2024-02-19T14:59:07.638065+0000","last_became_peered":"2024-02-19T14:59:07.638065+0000","last_unstale":"2024-02-19T14:59:07.640931+0000","last_undegraded":"2024-02-19T14:59:07.640931+0000","last_fullsized":"2024-02-19T14:59:07.640931+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:55.830792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635080+0000","last_change":"2024-02-19T14:59:07.635080+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635080+0000","last_undegraded":"2024-02-19T14:59:07.635080+0000","last_fullsized":"2024-02-19T14:59:07.635080+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644762+0000","last_change":"2024-02-19T14:59:07.639820+0000","last_active":"2024-02-19T14:59:07.644762+0000","last_peered":"2024-02-19T14:59:07.644762+0000","last_clean":"2024-02-19T14:59:07.644762+0000","last_became_active":"2024-02-19T14:59:07.639571+0000","last_became_peered":"2024-02-19T14:59:07.639571+0000","last_unstale":"2024-02-19T14:59:07.644762+0000","last_undegraded":"2024-02-19T14:59:07.644762+0000","last_fullsized":"2024-02-19T14:59:07.644762+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:26:02.808996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637145+0000","last_change":"2024-02-19T14:59:07.637145+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637145+0000","last_undegraded":"2024-02-19T14:59:07.637145+0000","last_fullsized":"2024-02-19T14:59:07.637145+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642076+0000","last_change":"2024-02-19T14:59:07.641532+0000","last_active":"2024-02-19T14:59:07.642076+0000","last_peered":"2024-02-19T14:59:07.642076+0000","last_clean":"2024-02-19T14:59:07.642076+0000","last_became_active":"2024-02-19T14:59:07.641278+0000","last_became_peered":"2024-02-19T14:59:07.641278+0000","last_unstale":"2024-02-19T14:59:07.642076+0000","last_undegraded":"2024-02-19T14:59:07.642076+0000","last_fullsized":"2024-02-19T14:59:07.642076+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:57.560202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633978+0000","last_change":"2024-02-19T14:59:07.633978+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633978+0000","last_undegraded":"2024-02-19T14:59:07.633978+0000","last_fullsized":"2024-02-19T14:59:07.633978+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646713+0000","last_change":"2024-02-19T14:59:07.646204+0000","last_active":"2024-02-19T14:59:07.646713+0000","last_peered":"2024-02-19T14:59:07.646713+0000","last_clean":"2024-02-19T14:59:07.646713+0000","last_became_active":"2024-02-19T14:59:07.645880+0000","last_became_peered":"2024-02-19T14:59:07.645880+0000","last_unstale":"2024-02-19T14:59:07.646713+0000","last_undegraded":"2024-02-19T14:59:07.646713+0000","last_fullsized":"2024-02-19T14:59:07.646713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637251+0000","last_change":"2024-02-19T14:59:07.637251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637251+0000","last_undegraded":"2024-02-19T14:59:07.637251+0000","last_fullsized":"2024-02-19T14:59:07.637251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644114+0000","last_change":"2024-02-19T14:59:07.642010+0000","last_active":"2024-02-19T14:59:07.644114+0000","last_peered":"2024-02-19T14:59:07.644114+0000","last_clean":"2024-02-19T14:59:07.644114+0000","last_became_active":"2024-02-19T14:59:07.641735+0000","last_became_peered":"2024-02-19T14:59:07.641735+0000","last_unstale":"2024-02-19T14:59:07.644114+0000","last_undegraded":"2024-02-19T14:59:07.644114+0000","last_fullsized":"2024-02-19T14:59:07.644114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:05.299812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635252+0000","last_change":"2024-02-19T14:59:07.635252+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635252+0000","last_undegraded":"2024-02-19T14:59:07.635252+0000","last_fullsized":"2024-02-19T14:59:07.635252+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644650+0000","last_change":"2024-02-19T14:59:07.643447+0000","last_active":"2024-02-19T14:59:07.644650+0000","last_peered":"2024-02-19T14:59:07.644650+0000","last_clean":"2024-02-19T14:59:07.644650+0000","last_became_active":"2024-02-19T14:59:07.643150+0000","last_became_peered":"2024-02-19T14:59:07.643150+0000","last_unstale":"2024-02-19T14:59:07.644650+0000","last_undegraded":"2024-02-19T14:59:07.644650+0000","last_fullsized":"2024-02-19T14:59:07.644650+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:37.234470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634275+0000","last_change":"2024-02-19T14:59:07.634275+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634275+0000","last_undegraded":"2024-02-19T14:59:07.634275+0000","last_fullsized":"2024-02-19T14:59:07.634275+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644014+0000","last_change":"2024-02-19T14:59:07.641207+0000","last_active":"2024-02-19T14:59:07.644014+0000","last_peered":"2024-02-19T14:59:07.644014+0000","last_clean":"2024-02-19T14:59:07.644014+0000","last_became_active":"2024-02-19T14:59:07.640874+0000","last_became_peered":"2024-02-19T14:59:07.640874+0000","last_unstale":"2024-02-19T14:59:07.644014+0000","last_undegraded":"2024-02-19T14:59:07.644014+0000","last_fullsized":"2024-02-19T14:59:07.644014+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:16:13.230015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634281+0000","last_change":"2024-02-19T14:59:07.634281+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634281+0000","last_undegraded":"2024-02-19T14:59:07.634281+0000","last_fullsized":"2024-02-19T14:59:07.634281+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641906+0000","last_change":"2024-02-19T14:59:07.638597+0000","last_active":"2024-02-19T14:59:07.641906+0000","last_peered":"2024-02-19T14:59:07.641906+0000","last_clean":"2024-02-19T14:59:07.641906+0000","last_became_active":"2024-02-19T14:59:07.638358+0000","last_became_peered":"2024-02-19T14:59:07.638358+0000","last_unstale":"2024-02-19T14:59:07.641906+0000","last_undegraded":"2024-02-19T14:59:07.641906+0000","last_fullsized":"2024-02-19T14:59:07.641906+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:21.669002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634738+0000","last_change":"2024-02-19T14:59:07.634738+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634738+0000","last_undegraded":"2024-02-19T14:59:07.634738+0000","last_fullsized":"2024-02-19T14:59:07.634738+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644861+0000","last_change":"2024-02-19T14:59:07.643687+0000","last_active":"2024-02-19T14:59:07.644861+0000","last_peered":"2024-02-19T14:59:07.644861+0000","last_clean":"2024-02-19T14:59:07.644861+0000","last_became_active":"2024-02-19T14:59:07.643136+0000","last_became_peered":"2024-02-19T14:59:07.643136+0000","last_unstale":"2024-02-19T14:59:07.644861+0000","last_undegraded":"2024-02-19T14:59:07.644861+0000","last_fullsized":"2024-02-19T14:59:07.644861+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:28:01.427001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633392+0000","last_change":"2024-02-19T14:59:07.633392+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633392+0000","last_undegraded":"2024-02-19T14:59:07.633392+0000","last_fullsized":"2024-02-19T14:59:07.633392+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641877+0000","last_change":"2024-02-19T14:59:07.638366+0000","last_active":"2024-02-19T14:59:07.641877+0000","last_peered":"2024-02-19T14:59:07.641877+0000","last_clean":"2024-02-19T14:59:07.641877+0000","last_became_active":"2024-02-19T14:59:07.638109+0000","last_became_peered":"2024-02-19T14:59:07.638109+0000","last_unstale":"2024-02-19T14:59:07.641877+0000","last_undegraded":"2024-02-19T14:59:07.641877+0000","last_fullsized":"2024-02-19T14:59:07.641877+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627740+0000","last_change":"2024-02-19T14:59:07.627740+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627740+0000","last_undegraded":"2024-02-19T14:59:07.627740+0000","last_fullsized":"2024-02-19T14:59:07.627740+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647367+0000","last_change":"2024-02-19T14:59:07.646216+0000","last_active":"2024-02-19T14:59:07.647367+0000","last_peered":"2024-02-19T14:59:07.647367+0000","last_clean":"2024-02-19T14:59:07.647367+0000","last_became_active":"2024-02-19T14:59:07.645915+0000","last_became_peered":"2024-02-19T14:59:07.645915+0000","last_unstale":"2024-02-19T14:59:07.647367+0000","last_undegraded":"2024-02-19T14:59:07.647367+0000","last_fullsized":"2024-02-19T14:59:07.647367+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:28:14.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630383+0000","last_change":"2024-02-19T14:59:07.630383+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630383+0000","last_undegraded":"2024-02-19T14:59:07.630383+0000","last_fullsized":"2024-02-19T14:59:07.630383+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647108+0000","last_change":"2024-02-19T14:59:07.643409+0000","last_active":"2024-02-19T14:59:07.647108+0000","last_peered":"2024-02-19T14:59:07.647108+0000","last_clean":"2024-02-19T14:59:07.647108+0000","last_became_active":"2024-02-19T14:59:07.643105+0000","last_became_peered":"2024-02-19T14:59:07.643105+0000","last_unstale":"2024-02-19T14:59:07.647108+0000","last_undegraded":"2024-02-19T14:59:07.647108+0000","last_fullsized":"2024-02-19T14:59:07.647108+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:55.853954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633352+0000","last_change":"2024-02-19T14:59:07.633352+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633352+0000","last_undegraded":"2024-02-19T14:59:07.633352+0000","last_fullsized":"2024-02-19T14:59:07.633352+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640331+0000","last_change":"2024-02-19T14:59:07.639087+0000","last_active":"2024-02-19T14:59:07.640331+0000","last_peered":"2024-02-19T14:59:07.640331+0000","last_clean":"2024-02-19T14:59:07.640331+0000","last_became_active":"2024-02-19T14:59:07.638819+0000","last_became_peered":"2024-02-19T14:59:07.638819+0000","last_unstale":"2024-02-19T14:59:07.640331+0000","last_undegraded":"2024-02-19T14:59:07.640331+0000","last_fullsized":"2024-02-19T14:59:07.640331+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632905+0000","last_change":"2024-02-19T14:59:07.632905+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632905+0000","last_undegraded":"2024-02-19T14:59:07.632905+0000","last_fullsized":"2024-02-19T14:59:07.632905+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637435+0000","last_change":"2024-02-19T14:59:07.636744+0000","last_active":"2024-02-19T14:59:07.637435+0000","last_peered":"2024-02-19T14:59:07.637435+0000","last_clean":"2024-02-19T14:59:07.637435+0000","last_became_active":"2024-02-19T14:59:07.636464+0000","last_became_peered":"2024-02-19T14:59:07.636464+0000","last_unstale":"2024-02-19T14:59:07.637435+0000","last_undegraded":"2024-02-19T14:59:07.637435+0000","last_fullsized":"2024-02-19T14:59:07.637435+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:30.048428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633393+0000","last_change":"2024-02-19T14:59:07.633393+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633393+0000","last_undegraded":"2024-02-19T14:59:07.633393+0000","last_fullsized":"2024-02-19T14:59:07.633393+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646942+0000","last_change":"2024-02-19T14:59:07.643659+0000","last_active":"2024-02-19T14:59:07.646942+0000","last_peered":"2024-02-19T14:59:07.646942+0000","last_clean":"2024-02-19T14:59:07.646942+0000","last_became_active":"2024-02-19T14:59:07.643126+0000","last_became_peered":"2024-02-19T14:59:07.643126+0000","last_unstale":"2024-02-19T14:59:07.646942+0000","last_undegraded":"2024-02-19T14:59:07.646942+0000","last_fullsized":"2024-02-19T14:59:07.646942+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:30.093846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630599+0000","last_change":"2024-02-19T14:59:07.630599+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630599+0000","last_undegraded":"2024-02-19T14:59:07.630599+0000","last_fullsized":"2024-02-19T14:59:07.630599+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644152+0000","last_change":"2024-02-19T14:59:07.642484+0000","last_active":"2024-02-19T14:59:07.644152+0000","last_peered":"2024-02-19T14:59:07.644152+0000","last_clean":"2024-02-19T14:59:07.644152+0000","last_became_active":"2024-02-19T14:59:07.642254+0000","last_became_peered":"2024-02-19T14:59:07.642254+0000","last_unstale":"2024-02-19T14:59:07.644152+0000","last_undegraded":"2024-02-19T14:59:07.644152+0000","last_fullsized":"2024-02-19T14:59:07.644152+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:19.998342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626772+0000","last_change":"2024-02-19T14:59:07.626772+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626772+0000","last_undegraded":"2024-02-19T14:59:07.626772+0000","last_fullsized":"2024-02-19T14:59:07.626772+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.645052+0000","last_change":"2024-02-19T14:59:07.644552+0000","last_active":"2024-02-19T14:59:07.645052+0000","last_peered":"2024-02-19T14:59:07.645052+0000","last_clean":"2024-02-19T14:59:07.645052+0000","last_became_active":"2024-02-19T14:59:07.644274+0000","last_became_peered":"2024-02-19T14:59:07.644274+0000","last_unstale":"2024-02-19T14:59:07.645052+0000","last_undegraded":"2024-02-19T14:59:07.645052+0000","last_fullsized":"2024-02-19T14:59:07.645052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:31:22.200914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632440+0000","last_change":"2024-02-19T14:59:07.632440+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632440+0000","last_undegraded":"2024-02-19T14:59:07.632440+0000","last_fullsized":"2024-02-19T14:59:07.632440+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642826+0000","last_change":"2024-02-19T14:59:07.638149+0000","last_active":"2024-02-19T14:59:07.642826+0000","last_peered":"2024-02-19T14:59:07.642826+0000","last_clean":"2024-02-19T14:59:07.642826+0000","last_became_active":"2024-02-19T14:59:07.637845+0000","last_became_peered":"2024-02-19T14:59:07.637845+0000","last_unstale":"2024-02-19T14:59:07.642826+0000","last_undegraded":"2024-02-19T14:59:07.642826+0000","last_fullsized":"2024-02-19T14:59:07.642826+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:47.116497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634012+0000","last_change":"2024-02-19T14:59:07.634012+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634012+0000","last_undegraded":"2024-02-19T14:59:07.634012+0000","last_fullsized":"2024-02-19T14:59:07.634012+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647966+0000","last_change":"2024-02-19T14:59:07.645385+0000","last_active":"2024-02-19T14:59:07.647966+0000","last_peered":"2024-02-19T14:59:07.647966+0000","last_clean":"2024-02-19T14:59:07.647966+0000","last_became_active":"2024-02-19T14:59:07.644999+0000","last_became_peered":"2024-02-19T14:59:07.644999+0000","last_unstale":"2024-02-19T14:59:07.647966+0000","last_undegraded":"2024-02-19T14:59:07.647966+0000","last_fullsized":"2024-02-19T14:59:07.647966+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:03:16.898728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636317+0000","last_change":"2024-02-19T14:59:07.636317+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636317+0000","last_undegraded":"2024-02-19T14:59:07.636317+0000","last_fullsized":"2024-02-19T14:59:07.636317+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.648114+0000","last_change":"2024-02-19T14:59:07.646936+0000","last_active":"2024-02-19T14:59:07.648114+0000","last_peered":"2024-02-19T14:59:07.648114+0000","last_clean":"2024-02-19T14:59:07.648114+0000","last_became_active":"2024-02-19T14:59:07.645710+0000","last_became_peered":"2024-02-19T14:59:07.645710+0000","last_unstale":"2024-02-19T14:59:07.648114+0000","last_undegraded":"2024-02-19T14:59:07.648114+0000","last_fullsized":"2024-02-19T14:59:07.648114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:44.397134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626291+0000","last_change":"2024-02-19T14:59:07.626291+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626291+0000","last_undegraded":"2024-02-19T14:59:07.626291+0000","last_fullsized":"2024-02-19T14:59:07.626291+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.638399+0000","last_change":"2024-02-19T14:59:07.636425+0000","last_active":"2024-02-19T14:59:07.638399+0000","last_peered":"2024-02-19T14:59:07.638399+0000","last_clean":"2024-02-19T14:59:07.638399+0000","last_became_active":"2024-02-19T14:59:07.636154+0000","last_became_peered":"2024-02-19T14:59:07.636154+0000","last_unstale":"2024-02-19T14:59:07.638399+0000","last_undegraded":"2024-02-19T14:59:07.638399+0000","last_fullsized":"2024-02-19T14:59:07.638399+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:41.446210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630391+0000","last_change":"2024-02-19T14:59:07.630391+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630391+0000","last_undegraded":"2024-02-19T14:59:07.630391+0000","last_fullsized":"2024-02-19T14:59:07.630391+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640425+0000","last_change":"2024-02-19T14:59:07.638740+0000","last_active":"2024-02-19T14:59:07.640425+0000","last_peered":"2024-02-19T14:59:07.640425+0000","last_clean":"2024-02-19T14:59:07.640425+0000","last_became_active":"2024-02-19T14:59:07.638275+0000","last_became_peered":"2024-02-19T14:59:07.638275+0000","last_unstale":"2024-02-19T14:59:07.640425+0000","last_undegraded":"2024-02-19T14:59:07.640425+0000","last_fullsized":"2024-02-19T14:59:07.640425+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631957+0000","last_change":"2024-02-19T14:59:07.631957+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631957+0000","last_undegraded":"2024-02-19T14:59:07.631957+0000","last_fullsized":"2024-02-19T14:59:07.631957+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641644+0000","last_change":"2024-02-19T14:59:07.641182+0000","last_active":"2024-02-19T14:59:07.641644+0000","last_peered":"2024-02-19T14:59:07.641644+0000","last_clean":"2024-02-19T14:59:07.641644+0000","last_became_active":"2024-02-19T14:59:07.640867+0000","last_became_peered":"2024-02-19T14:59:07.640867+0000","last_unstale":"2024-02-19T14:59:07.641644+0000","last_undegraded":"2024-02-19T14:59:07.641644+0000","last_fullsized":"2024-02-19T14:59:07.641644+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:59.660494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629585+0000","last_change":"2024-02-19T14:59:07.629585+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629585+0000","last_undegraded":"2024-02-19T14:59:07.629585+0000","last_fullsized":"2024-02-19T14:59:07.629585+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643474+0000","last_change":"2024-02-19T14:59:07.640656+0000","last_active":"2024-02-19T14:59:07.643474+0000","last_peered":"2024-02-19T14:59:07.643474+0000","last_clean":"2024-02-19T14:59:07.643474+0000","last_became_active":"2024-02-19T14:59:07.640382+0000","last_became_peered":"2024-02-19T14:59:07.640382+0000","last_unstale":"2024-02-19T14:59:07.643474+0000","last_undegraded":"2024-02-19T14:59:07.643474+0000","last_fullsized":"2024-02-19T14:59:07.643474+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:43:14.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626274+0000","last_change":"2024-02-19T14:59:07.626274+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626274+0000","last_undegraded":"2024-02-19T14:59:07.626274+0000","last_fullsized":"2024-02-19T14:59:07.626274+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640387+0000","last_change":"2024-02-19T14:59:07.638470+0000","last_active":"2024-02-19T14:59:07.640387+0000","last_peered":"2024-02-19T14:59:07.640387+0000","last_clean":"2024-02-19T14:59:07.640387+0000","last_became_active":"2024-02-19T14:59:07.638197+0000","last_became_peered":"2024-02-19T14:59:07.638197+0000","last_unstale":"2024-02-19T14:59:07.640387+0000","last_undegraded":"2024-02-19T14:59:07.640387+0000","last_fullsized":"2024-02-19T14:59:07.640387+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:09.994584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630266+0000","last_change":"2024-02-19T14:59:07.630266+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630266+0000","last_undegraded":"2024-02-19T14:59:07.630266+0000","last_fullsized":"2024-02-19T14:59:07.630266+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643869+0000","last_change":"2024-02-19T14:59:07.640562+0000","last_active":"2024-02-19T14:59:07.643869+0000","last_peered":"2024-02-19T14:59:07.643869+0000","last_clean":"2024-02-19T14:59:07.643869+0000","last_became_active":"2024-02-19T14:59:07.640288+0000","last_became_peered":"2024-02-19T14:59:07.640288+0000","last_unstale":"2024-02-19T14:59:07.643869+0000","last_undegraded":"2024-02-19T14:59:07.643869+0000","last_fullsized":"2024-02-19T14:59:07.643869+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634982+0000","last_change":"2024-02-19T14:59:07.634982+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634982+0000","last_undegraded":"2024-02-19T14:59:07.634982+0000","last_fullsized":"2024-02-19T14:59:07.634982+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647753+0000","last_change":"2024-02-19T14:59:07.645290+0000","last_active":"2024-02-19T14:59:07.647753+0000","last_peered":"2024-02-19T14:59:07.647753+0000","last_clean":"2024-02-19T14:59:07.647753+0000","last_became_active":"2024-02-19T14:59:07.644933+0000","last_became_peered":"2024-02-19T14:59:07.644933+0000","last_unstale":"2024-02-19T14:59:07.647753+0000","last_undegraded":"2024-02-19T14:59:07.647753+0000","last_fullsized":"2024-02-19T14:59:07.647753+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:37.251826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630878+0000","last_change":"2024-02-19T14:59:07.630878+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630878+0000","last_undegraded":"2024-02-19T14:59:07.630878+0000","last_fullsized":"2024-02-19T14:59:07.630878+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647675+0000","last_change":"2024-02-19T14:59:07.643990+0000","last_active":"2024-02-19T14:59:07.647675+0000","last_peered":"2024-02-19T14:59:07.647675+0000","last_clean":"2024-02-19T14:59:07.647675+0000","last_became_active":"2024-02-19T14:59:07.643708+0000","last_became_peered":"2024-02-19T14:59:07.643708+0000","last_unstale":"2024-02-19T14:59:07.647675+0000","last_undegraded":"2024-02-19T14:59:07.647675+0000","last_fullsized":"2024-02-19T14:59:07.647675+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:00.752877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631620+0000","last_change":"2024-02-19T14:59:07.631620+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631620+0000","last_undegraded":"2024-02-19T14:59:07.631620+0000","last_fullsized":"2024-02-19T14:59:07.631620+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637822+0000","last_change":"2024-02-19T14:59:07.634059+0000","last_active":"2024-02-19T14:59:07.637822+0000","last_peered":"2024-02-19T14:59:07.637822+0000","last_clean":"2024-02-19T14:59:07.637822+0000","last_became_active":"2024-02-19T14:59:07.633768+0000","last_became_peered":"2024-02-19T14:59:07.633768+0000","last_unstale":"2024-02-19T14:59:07.637822+0000","last_undegraded":"2024-02-19T14:59:07.637822+0000","last_fullsized":"2024-02-19T14:59:07.637822+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:52.944510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633788+0000","last_change":"2024-02-19T14:59:07.633788+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633788+0000","last_undegraded":"2024-02-19T14:59:07.633788+0000","last_fullsized":"2024-02-19T14:59:07.633788+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.636397+0000","last_change":"2024-02-19T14:59:07.632792+0000","last_active":"2024-02-19T14:59:07.636397+0000","last_peered":"2024-02-19T14:59:07.636397+0000","last_clean":"2024-02-19T14:59:07.636397+0000","last_became_active":"2024-02-19T14:59:07.632529+0000","last_became_peered":"2024-02-19T14:59:07.632529+0000","last_unstale":"2024-02-19T14:59:07.636397+0000","last_undegraded":"2024-02-19T14:59:07.636397+0000","last_fullsized":"2024-02-19T14:59:07.636397+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638240+0000","last_change":"2024-02-19T14:59:07.638240+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638240+0000","last_undegraded":"2024-02-19T14:59:07.638240+0000","last_fullsized":"2024-02-19T14:59:07.638240+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646917+0000","last_change":"2024-02-19T14:59:07.643336+0000","last_active":"2024-02-19T14:59:07.646917+0000","last_peered":"2024-02-19T14:59:07.646917+0000","last_clean":"2024-02-19T14:59:07.646917+0000","last_became_active":"2024-02-19T14:59:07.642778+0000","last_became_peered":"2024-02-19T14:59:07.642778+0000","last_unstale":"2024-02-19T14:59:07.646917+0000","last_undegraded":"2024-02-19T14:59:07.646917+0000","last_fullsized":"2024-02-19T14:59:07.646917+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:36.331774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634502+0000","last_change":"2024-02-19T14:59:07.634502+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634502+0000","last_undegraded":"2024-02-19T14:59:07.634502+0000","last_fullsized":"2024-02-19T14:59:07.634502+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641427+0000","last_change":"2024-02-19T14:59:07.636409+0000","last_active":"2024-02-19T14:59:07.641427+0000","last_peered":"2024-02-19T14:59:07.641427+0000","last_clean":"2024-02-19T14:59:07.641427+0000","last_became_active":"2024-02-19T14:59:07.636136+0000","last_became_peered":"2024-02-19T14:59:07.636136+0000","last_unstale":"2024-02-19T14:59:07.641427+0000","last_undegraded":"2024-02-19T14:59:07.641427+0000","last_fullsized":"2024-02-19T14:59:07.641427+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":480,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.627573+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:07.627573+0000","last_peered":"2024-02-19T14:59:07.627573+0000","last_clean":"2024-02-19T14:59:07.627573+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:07.627573+0000","last_undegraded":"2024-02-19T14:59:07.627573+0000","last_fullsized":"2024-02-19T14:59:07.627573+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34060,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337780,"statfs":{"total":96636764160,"available":96601886720,"internally_reserved":0,"allocated":2949120,"data_stored":1741638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.505,"15min":0.495},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51600000000000001,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.748,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69299999999999995,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.627,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.629}]},{"osd":4,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46200000000000002,"15min":0.44900000000000001},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.753,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.499,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.75,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.79000000000000004,"5min":0.871,"15min":0.871},"last":0.498}]},{"osd":6,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.84399999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574939,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69299999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.92400000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.876},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.94899999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.68000000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.313},"min":{"1min":0.217,"5min":0.159,"15min":0.159},"max":{"1min":0.45100000000000001,"5min":3.694,"15min":3.694},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36799999999999999,"15min":0.32600000000000001},"min":{"1min":0.248,"5min":0.192,"15min":0.189},"max":{"1min":0.55300000000000005,"5min":4.835,"15min":4.835},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.41699999999999998,"15min":0.378},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.498,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.314}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41399999999999998,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76200000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.621},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78700000000000003,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.41999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72099999999999997,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.374},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.35299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.312},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.748}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.623},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.85799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.32600000000000001},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32300000000000001,"15min":0.33200000000000002},"min":{"1min":0.24299999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.45600000000000002,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36899999999999999,"15min":0.38500000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42199999999999999,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.54900000000000004},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.309},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.96799999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.44800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.56499999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.82399999999999995,"15min":0.996},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.036},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.71799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34048,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337792,"statfs":{"total":96636764160,"available":96601899008,"internally_reserved":0,"allocated":2936832,"data_stored":1734971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3f is not active+clean 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.627095+0000', 'last_change': '2024-02-19T14:59:07.627095+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.627095+0000', 'last_undegraded': '2024-02-19T14:59:07.627095+0000', 'last_fullsized': '2024-02-19T14:59:07.627095+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3e is not active+clean 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.629907+0000', 'last_change': '2024-02-19T14:59:07.629907+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.629907+0000', 'last_undegraded': '2024-02-19T14:59:07.629907+0000', 'last_fullsized': '2024-02-19T14:59:07.629907+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3d is not active+clean 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.640087+0000', 'last_change': '2024-02-19T14:59:07.640087+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.640087+0000', 'last_undegraded': '2024-02-19T14:59:07.640087+0000', 'last_fullsized': '2024-02-19T14:59:07.640087+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3c is not active+clean 2024-02-19T14:59:12.297 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626117+0000', 'last_change': '2024-02-19T14:59:07.626117+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626117+0000', 'last_undegraded': '2024-02-19T14:59:07.626117+0000', 'last_fullsized': '2024-02-19T14:59:07.626117+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3b is not active+clean 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626189+0000', 'last_change': '2024-02-19T14:59:07.626189+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626189+0000', 'last_undegraded': '2024-02-19T14:59:07.626189+0000', 'last_fullsized': '2024-02-19T14:59:07.626189+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3a is not active+clean 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.628611+0000', 'last_change': '2024-02-19T14:59:07.628611+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.628611+0000', 'last_undegraded': '2024-02-19T14:59:07.628611+0000', 'last_fullsized': '2024-02-19T14:59:07.628611+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.39 is not active+clean 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.640069+0000', 'last_change': '2024-02-19T14:59:07.640069+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.640069+0000', 'last_undegraded': '2024-02-19T14:59:07.640069+0000', 'last_fullsized': '2024-02-19T14:59:07.640069+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.38 is not active+clean 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.629382+0000', 'last_change': '2024-02-19T14:59:07.629382+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.629382+0000', 'last_undegraded': '2024-02-19T14:59:07.629382+0000', 'last_fullsized': '2024-02-19T14:59:07.629382+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.37 is not active+clean 2024-02-19T14:59:12.298 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634618+0000', 'last_change': '2024-02-19T14:59:07.634618+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634618+0000', 'last_undegraded': '2024-02-19T14:59:07.634618+0000', 'last_fullsized': '2024-02-19T14:59:07.634618+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.36 is not active+clean 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.637179+0000', 'last_change': '2024-02-19T14:59:07.637179+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.637179+0000', 'last_undegraded': '2024-02-19T14:59:07.637179+0000', 'last_fullsized': '2024-02-19T14:59:07.637179+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.35 is not active+clean 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626760+0000', 'last_change': '2024-02-19T14:59:07.626760+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626760+0000', 'last_undegraded': '2024-02-19T14:59:07.626760+0000', 'last_fullsized': '2024-02-19T14:59:07.626760+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.34 is not active+clean 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.639587+0000', 'last_change': '2024-02-19T14:59:07.639587+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.639587+0000', 'last_undegraded': '2024-02-19T14:59:07.639587+0000', 'last_fullsized': '2024-02-19T14:59:07.639587+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.33 is not active+clean 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.628891+0000', 'last_change': '2024-02-19T14:59:07.628891+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.628891+0000', 'last_undegraded': '2024-02-19T14:59:07.628891+0000', 'last_fullsized': '2024-02-19T14:59:07.628891+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.32 is not active+clean 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.637655+0000', 'last_change': '2024-02-19T14:59:07.637655+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.637655+0000', 'last_undegraded': '2024-02-19T14:59:07.637655+0000', 'last_fullsized': '2024-02-19T14:59:07.637655+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.299 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.31 is not active+clean 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.635774+0000', 'last_change': '2024-02-19T14:59:07.635774+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.635774+0000', 'last_undegraded': '2024-02-19T14:59:07.635774+0000', 'last_fullsized': '2024-02-19T14:59:07.635774+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.30 is not active+clean 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.628909+0000', 'last_change': '2024-02-19T14:59:07.628909+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.628909+0000', 'last_undegraded': '2024-02-19T14:59:07.628909+0000', 'last_fullsized': '2024-02-19T14:59:07.628909+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2f is not active+clean 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.639159+0000', 'last_change': '2024-02-19T14:59:07.639159+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.639159+0000', 'last_undegraded': '2024-02-19T14:59:07.639159+0000', 'last_fullsized': '2024-02-19T14:59:07.639159+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2e is not active+clean 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.636673+0000', 'last_change': '2024-02-19T14:59:07.636673+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.636673+0000', 'last_undegraded': '2024-02-19T14:59:07.636673+0000', 'last_fullsized': '2024-02-19T14:59:07.636673+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2d is not active+clean 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.627251+0000', 'last_change': '2024-02-19T14:59:07.627251+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.627251+0000', 'last_undegraded': '2024-02-19T14:59:07.627251+0000', 'last_fullsized': '2024-02-19T14:59:07.627251+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2c is not active+clean 2024-02-19T14:59:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.629094+0000', 'last_change': '2024-02-19T14:59:07.629094+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.629094+0000', 'last_undegraded': '2024-02-19T14:59:07.629094+0000', 'last_fullsized': '2024-02-19T14:59:07.629094+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2b is not active+clean 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.635444+0000', 'last_change': '2024-02-19T14:59:07.635444+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.635444+0000', 'last_undegraded': '2024-02-19T14:59:07.635444+0000', 'last_fullsized': '2024-02-19T14:59:07.635444+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2a is not active+clean 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.635646+0000', 'last_change': '2024-02-19T14:59:07.635646+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.635646+0000', 'last_undegraded': '2024-02-19T14:59:07.635646+0000', 'last_fullsized': '2024-02-19T14:59:07.635646+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.29 is not active+clean 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633781+0000', 'last_change': '2024-02-19T14:59:07.633781+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633781+0000', 'last_undegraded': '2024-02-19T14:59:07.633781+0000', 'last_fullsized': '2024-02-19T14:59:07.633781+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.28 is not active+clean 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626539+0000', 'last_change': '2024-02-19T14:59:07.626539+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626539+0000', 'last_undegraded': '2024-02-19T14:59:07.626539+0000', 'last_fullsized': '2024-02-19T14:59:07.626539+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.27 is not active+clean 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626634+0000', 'last_change': '2024-02-19T14:59:07.626634+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626634+0000', 'last_undegraded': '2024-02-19T14:59:07.626634+0000', 'last_fullsized': '2024-02-19T14:59:07.626634+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.26 is not active+clean 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634974+0000', 'last_change': '2024-02-19T14:59:07.634974+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634974+0000', 'last_undegraded': '2024-02-19T14:59:07.634974+0000', 'last_fullsized': '2024-02-19T14:59:07.634974+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.25 is not active+clean 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.628304+0000', 'last_change': '2024-02-19T14:59:07.628304+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.628304+0000', 'last_undegraded': '2024-02-19T14:59:07.628304+0000', 'last_fullsized': '2024-02-19T14:59:07.628304+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.24 is not active+clean 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.638306+0000', 'last_change': '2024-02-19T14:59:07.638306+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.638306+0000', 'last_undegraded': '2024-02-19T14:59:07.638306+0000', 'last_fullsized': '2024-02-19T14:59:07.638306+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.23 is not active+clean 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.629327+0000', 'last_change': '2024-02-19T14:59:07.629327+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.629327+0000', 'last_undegraded': '2024-02-19T14:59:07.629327+0000', 'last_fullsized': '2024-02-19T14:59:07.629327+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.22 is not active+clean 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.629097+0000', 'last_change': '2024-02-19T14:59:07.629097+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.629097+0000', 'last_undegraded': '2024-02-19T14:59:07.629097+0000', 'last_fullsized': '2024-02-19T14:59:07.629097+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.21 is not active+clean 2024-02-19T14:59:12.302 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633321+0000', 'last_change': '2024-02-19T14:59:07.633321+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633321+0000', 'last_undegraded': '2024-02-19T14:59:07.633321+0000', 'last_fullsized': '2024-02-19T14:59:07.633321+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.20 is not active+clean 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.628422+0000', 'last_change': '2024-02-19T14:59:07.628422+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.628422+0000', 'last_undegraded': '2024-02-19T14:59:07.628422+0000', 'last_fullsized': '2024-02-19T14:59:07.628422+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1c is not active+clean 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634580+0000', 'last_change': '2024-02-19T14:59:07.634580+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634580+0000', 'last_undegraded': '2024-02-19T14:59:07.634580+0000', 'last_fullsized': '2024-02-19T14:59:07.634580+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1d is not active+clean 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.635080+0000', 'last_change': '2024-02-19T14:59:07.635080+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.635080+0000', 'last_undegraded': '2024-02-19T14:59:07.635080+0000', 'last_fullsized': '2024-02-19T14:59:07.635080+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1e is not active+clean 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.637145+0000', 'last_change': '2024-02-19T14:59:07.637145+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.637145+0000', 'last_undegraded': '2024-02-19T14:59:07.637145+0000', 'last_fullsized': '2024-02-19T14:59:07.637145+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.18 is not active+clean 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633978+0000', 'last_change': '2024-02-19T14:59:07.633978+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633978+0000', 'last_undegraded': '2024-02-19T14:59:07.633978+0000', 'last_fullsized': '2024-02-19T14:59:07.633978+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.19 is not active+clean 2024-02-19T14:59:12.303 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.637251+0000', 'last_change': '2024-02-19T14:59:07.637251+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.637251+0000', 'last_undegraded': '2024-02-19T14:59:07.637251+0000', 'last_fullsized': '2024-02-19T14:59:07.637251+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1a is not active+clean 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.635252+0000', 'last_change': '2024-02-19T14:59:07.635252+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.635252+0000', 'last_undegraded': '2024-02-19T14:59:07.635252+0000', 'last_fullsized': '2024-02-19T14:59:07.635252+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1b is not active+clean 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634275+0000', 'last_change': '2024-02-19T14:59:07.634275+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634275+0000', 'last_undegraded': '2024-02-19T14:59:07.634275+0000', 'last_fullsized': '2024-02-19T14:59:07.634275+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.14 is not active+clean 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634281+0000', 'last_change': '2024-02-19T14:59:07.634281+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634281+0000', 'last_undegraded': '2024-02-19T14:59:07.634281+0000', 'last_fullsized': '2024-02-19T14:59:07.634281+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.15 is not active+clean 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634738+0000', 'last_change': '2024-02-19T14:59:07.634738+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634738+0000', 'last_undegraded': '2024-02-19T14:59:07.634738+0000', 'last_fullsized': '2024-02-19T14:59:07.634738+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.16 is not active+clean 2024-02-19T14:59:12.304 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633392+0000', 'last_change': '2024-02-19T14:59:07.633392+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633392+0000', 'last_undegraded': '2024-02-19T14:59:07.633392+0000', 'last_fullsized': '2024-02-19T14:59:07.633392+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.17 is not active+clean 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.627740+0000', 'last_change': '2024-02-19T14:59:07.627740+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.627740+0000', 'last_undegraded': '2024-02-19T14:59:07.627740+0000', 'last_fullsized': '2024-02-19T14:59:07.627740+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.10 is not active+clean 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.630383+0000', 'last_change': '2024-02-19T14:59:07.630383+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.630383+0000', 'last_undegraded': '2024-02-19T14:59:07.630383+0000', 'last_fullsized': '2024-02-19T14:59:07.630383+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.11 is not active+clean 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633352+0000', 'last_change': '2024-02-19T14:59:07.633352+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633352+0000', 'last_undegraded': '2024-02-19T14:59:07.633352+0000', 'last_fullsized': '2024-02-19T14:59:07.633352+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.12 is not active+clean 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.632905+0000', 'last_change': '2024-02-19T14:59:07.632905+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.632905+0000', 'last_undegraded': '2024-02-19T14:59:07.632905+0000', 'last_fullsized': '2024-02-19T14:59:07.632905+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.13 is not active+clean 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633393+0000', 'last_change': '2024-02-19T14:59:07.633393+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633393+0000', 'last_undegraded': '2024-02-19T14:59:07.633393+0000', 'last_fullsized': '2024-02-19T14:59:07.633393+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.c is not active+clean 2024-02-19T14:59:12.305 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.630599+0000', 'last_change': '2024-02-19T14:59:07.630599+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.630599+0000', 'last_undegraded': '2024-02-19T14:59:07.630599+0000', 'last_fullsized': '2024-02-19T14:59:07.630599+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.d is not active+clean 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626772+0000', 'last_change': '2024-02-19T14:59:07.626772+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626772+0000', 'last_undegraded': '2024-02-19T14:59:07.626772+0000', 'last_fullsized': '2024-02-19T14:59:07.626772+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.e is not active+clean 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.632440+0000', 'last_change': '2024-02-19T14:59:07.632440+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.632440+0000', 'last_undegraded': '2024-02-19T14:59:07.632440+0000', 'last_fullsized': '2024-02-19T14:59:07.632440+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.f is not active+clean 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634012+0000', 'last_change': '2024-02-19T14:59:07.634012+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634012+0000', 'last_undegraded': '2024-02-19T14:59:07.634012+0000', 'last_fullsized': '2024-02-19T14:59:07.634012+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.8 is not active+clean 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.636317+0000', 'last_change': '2024-02-19T14:59:07.636317+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.636317+0000', 'last_undegraded': '2024-02-19T14:59:07.636317+0000', 'last_fullsized': '2024-02-19T14:59:07.636317+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.9 is not active+clean 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626291+0000', 'last_change': '2024-02-19T14:59:07.626291+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626291+0000', 'last_undegraded': '2024-02-19T14:59:07.626291+0000', 'last_fullsized': '2024-02-19T14:59:07.626291+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.306 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.a is not active+clean 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.630391+0000', 'last_change': '2024-02-19T14:59:07.630391+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.630391+0000', 'last_undegraded': '2024-02-19T14:59:07.630391+0000', 'last_fullsized': '2024-02-19T14:59:07.630391+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.b is not active+clean 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.631957+0000', 'last_change': '2024-02-19T14:59:07.631957+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.631957+0000', 'last_undegraded': '2024-02-19T14:59:07.631957+0000', 'last_fullsized': '2024-02-19T14:59:07.631957+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.4 is not active+clean 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.629585+0000', 'last_change': '2024-02-19T14:59:07.629585+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.629585+0000', 'last_undegraded': '2024-02-19T14:59:07.629585+0000', 'last_fullsized': '2024-02-19T14:59:07.629585+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.5 is not active+clean 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.626274+0000', 'last_change': '2024-02-19T14:59:07.626274+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.626274+0000', 'last_undegraded': '2024-02-19T14:59:07.626274+0000', 'last_fullsized': '2024-02-19T14:59:07.626274+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.6 is not active+clean 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.630266+0000', 'last_change': '2024-02-19T14:59:07.630266+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.630266+0000', 'last_undegraded': '2024-02-19T14:59:07.630266+0000', 'last_fullsized': '2024-02-19T14:59:07.630266+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.7 is not active+clean 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634982+0000', 'last_change': '2024-02-19T14:59:07.634982+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634982+0000', 'last_undegraded': '2024-02-19T14:59:07.634982+0000', 'last_fullsized': '2024-02-19T14:59:07.634982+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:59:12.307 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.0 is not active+clean 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.630878+0000', 'last_change': '2024-02-19T14:59:07.630878+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.630878+0000', 'last_undegraded': '2024-02-19T14:59:07.630878+0000', 'last_fullsized': '2024-02-19T14:59:07.630878+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1 is not active+clean 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.631620+0000', 'last_change': '2024-02-19T14:59:07.631620+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.631620+0000', 'last_undegraded': '2024-02-19T14:59:07.631620+0000', 'last_fullsized': '2024-02-19T14:59:07.631620+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2 is not active+clean 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.633788+0000', 'last_change': '2024-02-19T14:59:07.633788+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.633788+0000', 'last_undegraded': '2024-02-19T14:59:07.633788+0000', 'last_fullsized': '2024-02-19T14:59:07.633788+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3 is not active+clean 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.638240+0000', 'last_change': '2024-02-19T14:59:07.638240+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.638240+0000', 'last_undegraded': '2024-02-19T14:59:07.638240+0000', 'last_fullsized': '2024-02-19T14:59:07.638240+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1f is not active+clean 2024-02-19T14:59:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 186, 'state': 'creating+peering', 'last_fresh': '2024-02-19T14:59:07.634502+0000', 'last_change': '2024-02-19T14:59:07.634502+0000', 'last_active': '2024-02-19T14:59:07.620339+0000', 'last_peered': '2024-02-19T14:59:07.620339+0000', 'last_clean': '2024-02-19T14:59:07.620339+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T14:59:07.634502+0000', 'last_undegraded': '2024-02-19T14:59:07.634502+0000', 'last_fullsized': '2024-02-19T14:59:07.634502+0000', 'mapping_epoch': 186, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 186, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'last_clean_scrub_stamp': '2024-02-19T14:59:07.620339+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T14:59:12.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T14:59:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.429+0000 7faf607f7640 1 Processor -- start 2024-02-19T14:59:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.429+0000 7faf607f7640 1 -- start start 2024-02-19T14:59:12.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf607f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf58158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf607f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf580c34a0 con 0x7faf580c8500 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf607f7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5806c8e0 con 0x7faf580c42e0 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf607f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf5806c090 con 0x7faf580c2f50 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf5e593640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7faf580c8500 0x7faf58158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40002/0 (socket says 172.21.15.78:40002) 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf5e593640 1 -- 172.21.15.78:0/167746119 learned_addr learned my addr 172.21.15.78:0/167746119 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf5ed94640 1 --2- 172.21.15.78:0/167746119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf58158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf5d591640 1 -- 172.21.15.78:0/167746119 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393791993 0 0) 0x7faf580c34a0 con 0x7faf580c8500 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.430+0000 7faf5d591640 1 -- 172.21.15.78:0/167746119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf44003520 con 0x7faf580c8500 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.431+0000 7faf5ed94640 1 -- 172.21.15.78:0/167746119 >> v1:172.21.15.150:6789/0 conn(0x7faf580c42e0 legacy=0x7faf5815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.431+0000 7faf5d591640 1 -- 172.21.15.78:0/167746119 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475094973 0 0) 0x7faf5806c8e0 con 0x7faf580c42e0 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.431+0000 7faf5ed94640 1 -- 172.21.15.78:0/167746119 >> v1:172.21.15.78:6789/0 conn(0x7faf580c8500 legacy=0x7faf58158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.431+0000 7faf5ed94640 1 -- 172.21.15.78:0/167746119 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5806cbc0 con 0x7faf580c2f50 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.431+0000 7faf5ed94640 1 --2- 172.21.15.78:0/167746119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf58158d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faf50009dc0 tx=0x7faf5002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7b4c21819952c90 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.431+0000 7faf5d591640 1 -- 172.21.15.78:0/167746119 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5002cc40 con 0x7faf580c2f50 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf5d591640 1 -- 172.21.15.78:0/167746119 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5002cda0 con 0x7faf580c2f50 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf5d591640 1 -- 172.21.15.78:0/167746119 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50034660 con 0x7faf580c2f50 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 -- 172.21.15.78:0/167746119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 msgr2=0x7faf58158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 --2- 172.21.15.78:0/167746119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf58158d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faf50009dc0 tx=0x7faf5002be90 comp rx=0 tx=0).stop 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 -- 172.21.15.78:0/167746119 shutdown_connections 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 --2- 172.21.15.78:0/167746119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf58158d60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 -- 172.21.15.78:0/167746119 >> 172.21.15.78:0/167746119 conn(0x7faf580aa300 msgr2=0x7faf580aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 -- 172.21.15.78:0/167746119 shutdown_connections 2024-02-19T14:59:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.432+0000 7faf607f7640 1 -- 172.21.15.78:0/167746119 wait complete. 2024-02-19T14:59:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.433+0000 7faf607f7640 1 Processor -- start 2024-02-19T14:59:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.433+0000 7faf607f7640 1 -- start start 2024-02-19T14:59:12.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf607f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf5814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf607f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf580c8500 0x7faf58151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf607f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 0x7faf581eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf607f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf581ea720 con 0x7faf581eab70 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf607f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf580ab3f0 con 0x7faf580c8500 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf607f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf58150480 con 0x7faf580c2f50 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 0x7faf581eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 0x7faf581eb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33034/0 (socket says 172.21.15.78:33034) 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 -- 172.21.15.78:0/3828428070 learned_addr learned my addr 172.21.15.78:0/3828428070 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5e593640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf5814dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5dd92640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf580c8500 0x7faf58151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 -- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 msgr2=0x7faf5814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf5814dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 -- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf580c8500 msgr2=0x7faf58151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf580c8500 0x7faf58151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.434+0000 7faf5ed94640 1 -- 172.21.15.78:0/3828428070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf50009a70 con 0x7faf581eab70 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.435+0000 7faf5ed94640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 0x7faf581eb000 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7faf50009ee0 tx=0x7faf50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.435+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50040650 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.435+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf500407b0 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.435+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf581ee070 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.435+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50040b60 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.435+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf581ee630 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.436+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf581ea890 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.436+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7faf50040cc0 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.436+0000 7faf4effd640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faf3c06b810 0x7faf3c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.437+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faf500ace80 con 0x7faf581eab70 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.437+0000 7faf5e593640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faf3c06b810 0x7faf3c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.437+0000 7faf5e593640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faf3c06b810 0x7faf3c06dcd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faf54005880 tx=0x7faf54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.441+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faf581ea890 con 0x7faf581eab70 2024-02-19T14:59:12.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.685+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7faf581ea890 con 0x7faf581eab70 2024-02-19T14:59:12.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.685+0000 7faf4effd640 1 -- 172.21.15.78:0/3828428070 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1458 (secure 0 0 0) 0x7faf581ea890 con 0x7faf581eab70 2024-02-19T14:59:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:12.688 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":456,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":188,"num_osds":8,"num_up_osds":8,"osd_up_since":1708354307,"num_in_osds":8,"osd_in_since":1708354298,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":275423232,"bytes_avail":772818690048,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":165,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"d","status":"up:active","gid":9831}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-02-19T14:58:58.142183+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T14:59:12.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faf3c06b810 msgr2=0x7faf3c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faf3c06b810 0x7faf3c06dcd0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7faf54005880 tx=0x7faf54009040 comp rx=0 tx=0).stop 2024-02-19T14:59:12.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 msgr2=0x7faf581eb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 0x7faf581eb000 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7faf50009ee0 tx=0x7faf50004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 shutdown_connections 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faf3c06b810 0x7faf3c06dcd0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf581eab70 0x7faf581eb000 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf580c8500 0x7faf58151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 --2- 172.21.15.78:0/3828428070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faf580c2f50 0x7faf5814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 >> 172.21.15.78:0/3828428070 conn(0x7faf580aa300 msgr2=0x7faf58152f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 shutdown_connections 2024-02-19T14:59:12.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.687+0000 7faf607f7640 1 -- 172.21.15.78:0/3828428070 wait complete. 2024-02-19T14:59:12.701 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T14:59:12.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:12.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.798+0000 7fb0c1793640 1 Processor -- start 2024-02-19T14:59:12.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.798+0000 7fb0c1793640 1 -- start start 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0c1793640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc152830 0x7fb0bc0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0c1793640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb0bc0c7990 0x7fb0bc06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0c1793640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc1586a0 con 0x7fb0bc0c5760 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0c1793640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c6550 con 0x7fb0bc0c7990 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0c1793640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc159320 con 0x7fb0bc152830 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb0bc0c7990 0x7fb0bc06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb0bc0c7990 0x7fb0bc06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33044/0 (socket says 172.21.15.78:33044) 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 -- 172.21.15.78:0/3055351737 learned_addr learned my addr 172.21.15.78:0/3055351737 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0baffd640 1 --2- 172.21.15.78:0/3055351737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc152830 0x7fb0bc0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 -- 172.21.15.78:0/3055351737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc152830 msgr2=0x7fb0bc0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:12.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 --2- 172.21.15.78:0/3055351737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc152830 0x7fb0bc0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 -- 172.21.15.78:0/3055351737 >> v1:172.21.15.150:6789/0 conn(0x7fb0bc0c5760 legacy=0x7fb0bc0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.799+0000 7fb0bb7fe640 1 -- 172.21.15.78:0/3055351737 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc158980 con 0x7fb0bc0c7990 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.800+0000 7fb0bb7fe640 1 --2- 172.21.15.78:0/3055351737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc0c7990 0x7fb0bc06c570 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009e80 tx=0x7fb0b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7865a24acfd8f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.800+0000 7fb0b9ffb640 1 -- 172.21.15.78:0/3055351737 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b402ca50 con 0x7fb0bc0c7990 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.800+0000 7fb0b9ffb640 1 -- 172.21.15.78:0/3055351737 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b402cbb0 con 0x7fb0bc0c7990 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.800+0000 7fb0b9ffb640 1 -- 172.21.15.78:0/3055351737 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40343f0 con 0x7fb0bc0c7990 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.800+0000 7fb0c1793640 1 -- 172.21.15.78:0/3055351737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc0c7990 msgr2=0x7fb0bc06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.800+0000 7fb0c1793640 1 --2- 172.21.15.78:0/3055351737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc0c7990 0x7fb0bc06c570 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009e80 tx=0x7fb0b402beb0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.801+0000 7fb0c1793640 1 -- 172.21.15.78:0/3055351737 shutdown_connections 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.801+0000 7fb0c1793640 1 --2- 172.21.15.78:0/3055351737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc0c7990 0x7fb0bc06c570 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.801+0000 7fb0c1793640 1 --2- 172.21.15.78:0/3055351737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc152830 0x7fb0bc0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.801+0000 7fb0c1793640 1 -- 172.21.15.78:0/3055351737 >> 172.21.15.78:0/3055351737 conn(0x7fb0bc0ca8a0 msgr2=0x7fb0bc0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:12.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.801+0000 7fb0c1793640 1 -- 172.21.15.78:0/3055351737 shutdown_connections 2024-02-19T14:59:12.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.801+0000 7fb0c1793640 1 -- 172.21.15.78:0/3055351737 wait complete. 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 Processor -- start 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 -- start start 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc0c5760 0x7fb0bc0c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 0x7fb0bc14b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc0ac5d0 0x7fb0bc14bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c1250 con 0x7fb0bc152830 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc154bc0 con 0x7fb0bc0c5760 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.802+0000 7fb0c1793640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0ad1e0 con 0x7fb0bc0ac5d0 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 0x7fb0bc14b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 0x7fb0bc14b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33058/0 (socket says 172.21.15.78:33058) 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 -- 172.21.15.78:0/2836739122 learned_addr learned my addr 172.21.15.78:0/2836739122 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0baffd640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc0c5760 0x7fb0bc0c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0bb7fe640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc0ac5d0 0x7fb0bc14bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 -- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc0ac5d0 msgr2=0x7fb0bc14bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc0ac5d0 0x7fb0bc14bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 -- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc0c5760 msgr2=0x7fb0bc0c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc0c5760 0x7fb0bc0c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 -- 172.21.15.78:0/2836739122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4009b30 con 0x7fb0bc152830 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0ba7fc640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 0x7fb0bc14b670 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c460 tx=0x7fb0b000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00052d0 con 0x7fb0bc152830 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0005470 con 0x7fb0bc152830 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc0acf40 con 0x7fb0bc152830 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0012870 con 0x7fb0bc152830 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.803+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc1eebb0 con 0x7fb0bc152830 2024-02-19T14:59:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.804+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc074990 con 0x7fb0bc152830 2024-02-19T14:59:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.805+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb0b00129d0 con 0x7fb0bc152830 2024-02-19T14:59:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.805+0000 7fb0a37fe640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb0a806b810 0x7fb0a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.805+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb0b0089c70 con 0x7fb0bc152830 2024-02-19T14:59:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.805+0000 7fb0baffd640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb0a806b810 0x7fb0a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.805+0000 7fb0baffd640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb0a806b810 0x7fb0a806dcd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009aa0 tx=0x7fb0ac009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:12.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.809+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb0b0058170 con 0x7fb0bc152830 2024-02-19T14:59:12.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.977+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0bc06deb0 con 0x7fb0a806b810 2024-02-19T14:59:12.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.984+0000 7fb0a37fe640 1 -- 172.21.15.78:0/2836739122 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284544 (secure 0 0 0) 0x7fb0ac011a00 con 0x7fb0a806b810 2024-02-19T14:59:12.989 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:12.991 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:12.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb0a806b810 msgr2=0x7fb0a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb0a806b810 0x7fb0a806dcd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009aa0 tx=0x7fb0ac009190 comp rx=0 tx=0).stop 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 msgr2=0x7fb0bc14b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 0x7fb0bc14b670 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c460 tx=0x7fb0b000c920 comp rx=0 tx=0).stop 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 shutdown_connections 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb0a806b810 0x7fb0a806dcd0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0bc152830 0x7fb0bc14b670 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb0bc0ac5d0 0x7fb0bc14bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 --2- 172.21.15.78:0/2836739122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0bc0c5760 0x7fb0bc0c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 >> 172.21.15.78:0/2836739122 conn(0x7fb0bc0ca8a0 msgr2=0x7fb0bc14e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 shutdown_connections 2024-02-19T14:59:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:12.987+0000 7fb0c1793640 1 -- 172.21.15.78:0/2836739122 wait complete. 2024-02-19T14:59:13.001 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-02-19T14:59:12.144424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268968,"kb_used_data":19512,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754705752,"statfs":{"total":773094113280,"available":772818690048,"internally_reserved":0,"allocated":19980288,"data_stored":10364228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.523288"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627095+0000","last_change":"2024-02-19T14:59:07.627095+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627095+0000","last_undegraded":"2024-02-19T14:59:07.627095+0000","last_fullsized":"2024-02-19T14:59:07.627095+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629907+0000","last_change":"2024-02-19T14:59:07.629907+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629907+0000","last_undegraded":"2024-02-19T14:59:07.629907+0000","last_fullsized":"2024-02-19T14:59:07.629907+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640087+0000","last_change":"2024-02-19T14:59:07.640087+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640087+0000","last_undegraded":"2024-02-19T14:59:07.640087+0000","last_fullsized":"2024-02-19T14:59:07.640087+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626117+0000","last_change":"2024-02-19T14:59:07.626117+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626117+0000","last_undegraded":"2024-02-19T14:59:07.626117+0000","last_fullsized":"2024-02-19T14:59:07.626117+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626189+0000","last_change":"2024-02-19T14:59:07.626189+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626189+0000","last_undegraded":"2024-02-19T14:59:07.626189+0000","last_fullsized":"2024-02-19T14:59:07.626189+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628611+0000","last_change":"2024-02-19T14:59:07.628611+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628611+0000","last_undegraded":"2024-02-19T14:59:07.628611+0000","last_fullsized":"2024-02-19T14:59:07.628611+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.640069+0000","last_change":"2024-02-19T14:59:07.640069+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.640069+0000","last_undegraded":"2024-02-19T14:59:07.640069+0000","last_fullsized":"2024-02-19T14:59:07.640069+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629382+0000","last_change":"2024-02-19T14:59:07.629382+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629382+0000","last_undegraded":"2024-02-19T14:59:07.629382+0000","last_fullsized":"2024-02-19T14:59:07.629382+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634618+0000","last_change":"2024-02-19T14:59:07.634618+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634618+0000","last_undegraded":"2024-02-19T14:59:07.634618+0000","last_fullsized":"2024-02-19T14:59:07.634618+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637179+0000","last_change":"2024-02-19T14:59:07.637179+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637179+0000","last_undegraded":"2024-02-19T14:59:07.637179+0000","last_fullsized":"2024-02-19T14:59:07.637179+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626760+0000","last_change":"2024-02-19T14:59:07.626760+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626760+0000","last_undegraded":"2024-02-19T14:59:07.626760+0000","last_fullsized":"2024-02-19T14:59:07.626760+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639587+0000","last_change":"2024-02-19T14:59:07.639587+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639587+0000","last_undegraded":"2024-02-19T14:59:07.639587+0000","last_fullsized":"2024-02-19T14:59:07.639587+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628891+0000","last_change":"2024-02-19T14:59:07.628891+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628891+0000","last_undegraded":"2024-02-19T14:59:07.628891+0000","last_fullsized":"2024-02-19T14:59:07.628891+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637655+0000","last_change":"2024-02-19T14:59:07.637655+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637655+0000","last_undegraded":"2024-02-19T14:59:07.637655+0000","last_fullsized":"2024-02-19T14:59:07.637655+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635774+0000","last_change":"2024-02-19T14:59:07.635774+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635774+0000","last_undegraded":"2024-02-19T14:59:07.635774+0000","last_fullsized":"2024-02-19T14:59:07.635774+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628909+0000","last_change":"2024-02-19T14:59:07.628909+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628909+0000","last_undegraded":"2024-02-19T14:59:07.628909+0000","last_fullsized":"2024-02-19T14:59:07.628909+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.639159+0000","last_change":"2024-02-19T14:59:07.639159+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.639159+0000","last_undegraded":"2024-02-19T14:59:07.639159+0000","last_fullsized":"2024-02-19T14:59:07.639159+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636673+0000","last_change":"2024-02-19T14:59:07.636673+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636673+0000","last_undegraded":"2024-02-19T14:59:07.636673+0000","last_fullsized":"2024-02-19T14:59:07.636673+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627251+0000","last_change":"2024-02-19T14:59:07.627251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627251+0000","last_undegraded":"2024-02-19T14:59:07.627251+0000","last_fullsized":"2024-02-19T14:59:07.627251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629094+0000","last_change":"2024-02-19T14:59:07.629094+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629094+0000","last_undegraded":"2024-02-19T14:59:07.629094+0000","last_fullsized":"2024-02-19T14:59:07.629094+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635444+0000","last_change":"2024-02-19T14:59:07.635444+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635444+0000","last_undegraded":"2024-02-19T14:59:07.635444+0000","last_fullsized":"2024-02-19T14:59:07.635444+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635646+0000","last_change":"2024-02-19T14:59:07.635646+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635646+0000","last_undegraded":"2024-02-19T14:59:07.635646+0000","last_fullsized":"2024-02-19T14:59:07.635646+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633781+0000","last_change":"2024-02-19T14:59:07.633781+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633781+0000","last_undegraded":"2024-02-19T14:59:07.633781+0000","last_fullsized":"2024-02-19T14:59:07.633781+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626539+0000","last_change":"2024-02-19T14:59:07.626539+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626539+0000","last_undegraded":"2024-02-19T14:59:07.626539+0000","last_fullsized":"2024-02-19T14:59:07.626539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626634+0000","last_change":"2024-02-19T14:59:07.626634+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626634+0000","last_undegraded":"2024-02-19T14:59:07.626634+0000","last_fullsized":"2024-02-19T14:59:07.626634+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634974+0000","last_change":"2024-02-19T14:59:07.634974+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634974+0000","last_undegraded":"2024-02-19T14:59:07.634974+0000","last_fullsized":"2024-02-19T14:59:07.634974+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628304+0000","last_change":"2024-02-19T14:59:07.628304+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628304+0000","last_undegraded":"2024-02-19T14:59:07.628304+0000","last_fullsized":"2024-02-19T14:59:07.628304+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638306+0000","last_change":"2024-02-19T14:59:07.638306+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638306+0000","last_undegraded":"2024-02-19T14:59:07.638306+0000","last_fullsized":"2024-02-19T14:59:07.638306+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629327+0000","last_change":"2024-02-19T14:59:07.629327+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629327+0000","last_undegraded":"2024-02-19T14:59:07.629327+0000","last_fullsized":"2024-02-19T14:59:07.629327+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629097+0000","last_change":"2024-02-19T14:59:07.629097+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629097+0000","last_undegraded":"2024-02-19T14:59:07.629097+0000","last_fullsized":"2024-02-19T14:59:07.629097+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633321+0000","last_change":"2024-02-19T14:59:07.633321+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633321+0000","last_undegraded":"2024-02-19T14:59:07.633321+0000","last_fullsized":"2024-02-19T14:59:07.633321+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.628422+0000","last_change":"2024-02-19T14:59:07.628422+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.628422+0000","last_undegraded":"2024-02-19T14:59:07.628422+0000","last_fullsized":"2024-02-19T14:59:07.628422+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634580+0000","last_change":"2024-02-19T14:59:07.634580+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634580+0000","last_undegraded":"2024-02-19T14:59:07.634580+0000","last_fullsized":"2024-02-19T14:59:07.634580+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640931+0000","last_change":"2024-02-19T14:59:07.638344+0000","last_active":"2024-02-19T14:59:07.640931+0000","last_peered":"2024-02-19T14:59:07.640931+0000","last_clean":"2024-02-19T14:59:07.640931+0000","last_became_active":"2024-02-19T14:59:07.638065+0000","last_became_peered":"2024-02-19T14:59:07.638065+0000","last_unstale":"2024-02-19T14:59:07.640931+0000","last_undegraded":"2024-02-19T14:59:07.640931+0000","last_fullsized":"2024-02-19T14:59:07.640931+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:22:55.830792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635080+0000","last_change":"2024-02-19T14:59:07.635080+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635080+0000","last_undegraded":"2024-02-19T14:59:07.635080+0000","last_fullsized":"2024-02-19T14:59:07.635080+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644762+0000","last_change":"2024-02-19T14:59:07.639820+0000","last_active":"2024-02-19T14:59:07.644762+0000","last_peered":"2024-02-19T14:59:07.644762+0000","last_clean":"2024-02-19T14:59:07.644762+0000","last_became_active":"2024-02-19T14:59:07.639571+0000","last_became_peered":"2024-02-19T14:59:07.639571+0000","last_unstale":"2024-02-19T14:59:07.644762+0000","last_undegraded":"2024-02-19T14:59:07.644762+0000","last_fullsized":"2024-02-19T14:59:07.644762+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:26:02.808996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637145+0000","last_change":"2024-02-19T14:59:07.637145+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637145+0000","last_undegraded":"2024-02-19T14:59:07.637145+0000","last_fullsized":"2024-02-19T14:59:07.637145+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642076+0000","last_change":"2024-02-19T14:59:07.641532+0000","last_active":"2024-02-19T14:59:07.642076+0000","last_peered":"2024-02-19T14:59:07.642076+0000","last_clean":"2024-02-19T14:59:07.642076+0000","last_became_active":"2024-02-19T14:59:07.641278+0000","last_became_peered":"2024-02-19T14:59:07.641278+0000","last_unstale":"2024-02-19T14:59:07.642076+0000","last_undegraded":"2024-02-19T14:59:07.642076+0000","last_fullsized":"2024-02-19T14:59:07.642076+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:13:57.560202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633978+0000","last_change":"2024-02-19T14:59:07.633978+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633978+0000","last_undegraded":"2024-02-19T14:59:07.633978+0000","last_fullsized":"2024-02-19T14:59:07.633978+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646713+0000","last_change":"2024-02-19T14:59:07.646204+0000","last_active":"2024-02-19T14:59:07.646713+0000","last_peered":"2024-02-19T14:59:07.646713+0000","last_clean":"2024-02-19T14:59:07.646713+0000","last_became_active":"2024-02-19T14:59:07.645880+0000","last_became_peered":"2024-02-19T14:59:07.645880+0000","last_unstale":"2024-02-19T14:59:07.646713+0000","last_undegraded":"2024-02-19T14:59:07.646713+0000","last_fullsized":"2024-02-19T14:59:07.646713+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.637251+0000","last_change":"2024-02-19T14:59:07.637251+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.637251+0000","last_undegraded":"2024-02-19T14:59:07.637251+0000","last_fullsized":"2024-02-19T14:59:07.637251+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644114+0000","last_change":"2024-02-19T14:59:07.642010+0000","last_active":"2024-02-19T14:59:07.644114+0000","last_peered":"2024-02-19T14:59:07.644114+0000","last_clean":"2024-02-19T14:59:07.644114+0000","last_became_active":"2024-02-19T14:59:07.641735+0000","last_became_peered":"2024-02-19T14:59:07.641735+0000","last_unstale":"2024-02-19T14:59:07.644114+0000","last_undegraded":"2024-02-19T14:59:07.644114+0000","last_fullsized":"2024-02-19T14:59:07.644114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:05.299812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.635252+0000","last_change":"2024-02-19T14:59:07.635252+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.635252+0000","last_undegraded":"2024-02-19T14:59:07.635252+0000","last_fullsized":"2024-02-19T14:59:07.635252+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644650+0000","last_change":"2024-02-19T14:59:07.643447+0000","last_active":"2024-02-19T14:59:07.644650+0000","last_peered":"2024-02-19T14:59:07.644650+0000","last_clean":"2024-02-19T14:59:07.644650+0000","last_became_active":"2024-02-19T14:59:07.643150+0000","last_became_peered":"2024-02-19T14:59:07.643150+0000","last_unstale":"2024-02-19T14:59:07.644650+0000","last_undegraded":"2024-02-19T14:59:07.644650+0000","last_fullsized":"2024-02-19T14:59:07.644650+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:06:37.234470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634275+0000","last_change":"2024-02-19T14:59:07.634275+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634275+0000","last_undegraded":"2024-02-19T14:59:07.634275+0000","last_fullsized":"2024-02-19T14:59:07.634275+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644014+0000","last_change":"2024-02-19T14:59:07.641207+0000","last_active":"2024-02-19T14:59:07.644014+0000","last_peered":"2024-02-19T14:59:07.644014+0000","last_clean":"2024-02-19T14:59:07.644014+0000","last_became_active":"2024-02-19T14:59:07.640874+0000","last_became_peered":"2024-02-19T14:59:07.640874+0000","last_unstale":"2024-02-19T14:59:07.644014+0000","last_undegraded":"2024-02-19T14:59:07.644014+0000","last_fullsized":"2024-02-19T14:59:07.644014+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:16:13.230015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634281+0000","last_change":"2024-02-19T14:59:07.634281+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634281+0000","last_undegraded":"2024-02-19T14:59:07.634281+0000","last_fullsized":"2024-02-19T14:59:07.634281+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641906+0000","last_change":"2024-02-19T14:59:07.638597+0000","last_active":"2024-02-19T14:59:07.641906+0000","last_peered":"2024-02-19T14:59:07.641906+0000","last_clean":"2024-02-19T14:59:07.641906+0000","last_became_active":"2024-02-19T14:59:07.638358+0000","last_became_peered":"2024-02-19T14:59:07.638358+0000","last_unstale":"2024-02-19T14:59:07.641906+0000","last_undegraded":"2024-02-19T14:59:07.641906+0000","last_fullsized":"2024-02-19T14:59:07.641906+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:21.669002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634738+0000","last_change":"2024-02-19T14:59:07.634738+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634738+0000","last_undegraded":"2024-02-19T14:59:07.634738+0000","last_fullsized":"2024-02-19T14:59:07.634738+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644861+0000","last_change":"2024-02-19T14:59:07.643687+0000","last_active":"2024-02-19T14:59:07.644861+0000","last_peered":"2024-02-19T14:59:07.644861+0000","last_clean":"2024-02-19T14:59:07.644861+0000","last_became_active":"2024-02-19T14:59:07.643136+0000","last_became_peered":"2024-02-19T14:59:07.643136+0000","last_unstale":"2024-02-19T14:59:07.644861+0000","last_undegraded":"2024-02-19T14:59:07.644861+0000","last_fullsized":"2024-02-19T14:59:07.644861+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:28:01.427001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633392+0000","last_change":"2024-02-19T14:59:07.633392+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633392+0000","last_undegraded":"2024-02-19T14:59:07.633392+0000","last_fullsized":"2024-02-19T14:59:07.633392+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641877+0000","last_change":"2024-02-19T14:59:07.638366+0000","last_active":"2024-02-19T14:59:07.641877+0000","last_peered":"2024-02-19T14:59:07.641877+0000","last_clean":"2024-02-19T14:59:07.641877+0000","last_became_active":"2024-02-19T14:59:07.638109+0000","last_became_peered":"2024-02-19T14:59:07.638109+0000","last_unstale":"2024-02-19T14:59:07.641877+0000","last_undegraded":"2024-02-19T14:59:07.641877+0000","last_fullsized":"2024-02-19T14:59:07.641877+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.627740+0000","last_change":"2024-02-19T14:59:07.627740+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.627740+0000","last_undegraded":"2024-02-19T14:59:07.627740+0000","last_fullsized":"2024-02-19T14:59:07.627740+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647367+0000","last_change":"2024-02-19T14:59:07.646216+0000","last_active":"2024-02-19T14:59:07.647367+0000","last_peered":"2024-02-19T14:59:07.647367+0000","last_clean":"2024-02-19T14:59:07.647367+0000","last_became_active":"2024-02-19T14:59:07.645915+0000","last_became_peered":"2024-02-19T14:59:07.645915+0000","last_unstale":"2024-02-19T14:59:07.647367+0000","last_undegraded":"2024-02-19T14:59:07.647367+0000","last_fullsized":"2024-02-19T14:59:07.647367+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:28:14.831932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630383+0000","last_change":"2024-02-19T14:59:07.630383+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630383+0000","last_undegraded":"2024-02-19T14:59:07.630383+0000","last_fullsized":"2024-02-19T14:59:07.630383+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647108+0000","last_change":"2024-02-19T14:59:07.643409+0000","last_active":"2024-02-19T14:59:07.647108+0000","last_peered":"2024-02-19T14:59:07.647108+0000","last_clean":"2024-02-19T14:59:07.647108+0000","last_became_active":"2024-02-19T14:59:07.643105+0000","last_became_peered":"2024-02-19T14:59:07.643105+0000","last_unstale":"2024-02-19T14:59:07.647108+0000","last_undegraded":"2024-02-19T14:59:07.647108+0000","last_fullsized":"2024-02-19T14:59:07.647108+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:51:55.853954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633352+0000","last_change":"2024-02-19T14:59:07.633352+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633352+0000","last_undegraded":"2024-02-19T14:59:07.633352+0000","last_fullsized":"2024-02-19T14:59:07.633352+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640331+0000","last_change":"2024-02-19T14:59:07.639087+0000","last_active":"2024-02-19T14:59:07.640331+0000","last_peered":"2024-02-19T14:59:07.640331+0000","last_clean":"2024-02-19T14:59:07.640331+0000","last_became_active":"2024-02-19T14:59:07.638819+0000","last_became_peered":"2024-02-19T14:59:07.638819+0000","last_unstale":"2024-02-19T14:59:07.640331+0000","last_undegraded":"2024-02-19T14:59:07.640331+0000","last_fullsized":"2024-02-19T14:59:07.640331+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632905+0000","last_change":"2024-02-19T14:59:07.632905+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632905+0000","last_undegraded":"2024-02-19T14:59:07.632905+0000","last_fullsized":"2024-02-19T14:59:07.632905+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637435+0000","last_change":"2024-02-19T14:59:07.636744+0000","last_active":"2024-02-19T14:59:07.637435+0000","last_peered":"2024-02-19T14:59:07.637435+0000","last_clean":"2024-02-19T14:59:07.637435+0000","last_became_active":"2024-02-19T14:59:07.636464+0000","last_became_peered":"2024-02-19T14:59:07.636464+0000","last_unstale":"2024-02-19T14:59:07.637435+0000","last_undegraded":"2024-02-19T14:59:07.637435+0000","last_fullsized":"2024-02-19T14:59:07.637435+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:30.048428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633393+0000","last_change":"2024-02-19T14:59:07.633393+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633393+0000","last_undegraded":"2024-02-19T14:59:07.633393+0000","last_fullsized":"2024-02-19T14:59:07.633393+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646942+0000","last_change":"2024-02-19T14:59:07.643659+0000","last_active":"2024-02-19T14:59:07.646942+0000","last_peered":"2024-02-19T14:59:07.646942+0000","last_clean":"2024-02-19T14:59:07.646942+0000","last_became_active":"2024-02-19T14:59:07.643126+0000","last_became_peered":"2024-02-19T14:59:07.643126+0000","last_unstale":"2024-02-19T14:59:07.646942+0000","last_undegraded":"2024-02-19T14:59:07.646942+0000","last_fullsized":"2024-02-19T14:59:07.646942+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:15:30.093846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630599+0000","last_change":"2024-02-19T14:59:07.630599+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630599+0000","last_undegraded":"2024-02-19T14:59:07.630599+0000","last_fullsized":"2024-02-19T14:59:07.630599+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.644152+0000","last_change":"2024-02-19T14:59:07.642484+0000","last_active":"2024-02-19T14:59:07.644152+0000","last_peered":"2024-02-19T14:59:07.644152+0000","last_clean":"2024-02-19T14:59:07.644152+0000","last_became_active":"2024-02-19T14:59:07.642254+0000","last_became_peered":"2024-02-19T14:59:07.642254+0000","last_unstale":"2024-02-19T14:59:07.644152+0000","last_undegraded":"2024-02-19T14:59:07.644152+0000","last_fullsized":"2024-02-19T14:59:07.644152+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:19.998342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626772+0000","last_change":"2024-02-19T14:59:07.626772+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626772+0000","last_undegraded":"2024-02-19T14:59:07.626772+0000","last_fullsized":"2024-02-19T14:59:07.626772+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.645052+0000","last_change":"2024-02-19T14:59:07.644552+0000","last_active":"2024-02-19T14:59:07.645052+0000","last_peered":"2024-02-19T14:59:07.645052+0000","last_clean":"2024-02-19T14:59:07.645052+0000","last_became_active":"2024-02-19T14:59:07.644274+0000","last_became_peered":"2024-02-19T14:59:07.644274+0000","last_unstale":"2024-02-19T14:59:07.645052+0000","last_undegraded":"2024-02-19T14:59:07.645052+0000","last_fullsized":"2024-02-19T14:59:07.645052+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:31:22.200914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.632440+0000","last_change":"2024-02-19T14:59:07.632440+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.632440+0000","last_undegraded":"2024-02-19T14:59:07.632440+0000","last_fullsized":"2024-02-19T14:59:07.632440+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.642826+0000","last_change":"2024-02-19T14:59:07.638149+0000","last_active":"2024-02-19T14:59:07.642826+0000","last_peered":"2024-02-19T14:59:07.642826+0000","last_clean":"2024-02-19T14:59:07.642826+0000","last_became_active":"2024-02-19T14:59:07.637845+0000","last_became_peered":"2024-02-19T14:59:07.637845+0000","last_unstale":"2024-02-19T14:59:07.642826+0000","last_undegraded":"2024-02-19T14:59:07.642826+0000","last_fullsized":"2024-02-19T14:59:07.642826+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:47.116497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634012+0000","last_change":"2024-02-19T14:59:07.634012+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634012+0000","last_undegraded":"2024-02-19T14:59:07.634012+0000","last_fullsized":"2024-02-19T14:59:07.634012+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647966+0000","last_change":"2024-02-19T14:59:07.645385+0000","last_active":"2024-02-19T14:59:07.647966+0000","last_peered":"2024-02-19T14:59:07.647966+0000","last_clean":"2024-02-19T14:59:07.647966+0000","last_became_active":"2024-02-19T14:59:07.644999+0000","last_became_peered":"2024-02-19T14:59:07.644999+0000","last_unstale":"2024-02-19T14:59:07.647966+0000","last_undegraded":"2024-02-19T14:59:07.647966+0000","last_fullsized":"2024-02-19T14:59:07.647966+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:03:16.898728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.636317+0000","last_change":"2024-02-19T14:59:07.636317+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.636317+0000","last_undegraded":"2024-02-19T14:59:07.636317+0000","last_fullsized":"2024-02-19T14:59:07.636317+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.648114+0000","last_change":"2024-02-19T14:59:07.646936+0000","last_active":"2024-02-19T14:59:07.648114+0000","last_peered":"2024-02-19T14:59:07.648114+0000","last_clean":"2024-02-19T14:59:07.648114+0000","last_became_active":"2024-02-19T14:59:07.645710+0000","last_became_peered":"2024-02-19T14:59:07.645710+0000","last_unstale":"2024-02-19T14:59:07.648114+0000","last_undegraded":"2024-02-19T14:59:07.648114+0000","last_fullsized":"2024-02-19T14:59:07.648114+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:44.397134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626291+0000","last_change":"2024-02-19T14:59:07.626291+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626291+0000","last_undegraded":"2024-02-19T14:59:07.626291+0000","last_fullsized":"2024-02-19T14:59:07.626291+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.638399+0000","last_change":"2024-02-19T14:59:07.636425+0000","last_active":"2024-02-19T14:59:07.638399+0000","last_peered":"2024-02-19T14:59:07.638399+0000","last_clean":"2024-02-19T14:59:07.638399+0000","last_became_active":"2024-02-19T14:59:07.636154+0000","last_became_peered":"2024-02-19T14:59:07.636154+0000","last_unstale":"2024-02-19T14:59:07.638399+0000","last_undegraded":"2024-02-19T14:59:07.638399+0000","last_fullsized":"2024-02-19T14:59:07.638399+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:41.446210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630391+0000","last_change":"2024-02-19T14:59:07.630391+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630391+0000","last_undegraded":"2024-02-19T14:59:07.630391+0000","last_fullsized":"2024-02-19T14:59:07.630391+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640425+0000","last_change":"2024-02-19T14:59:07.638740+0000","last_active":"2024-02-19T14:59:07.640425+0000","last_peered":"2024-02-19T14:59:07.640425+0000","last_clean":"2024-02-19T14:59:07.640425+0000","last_became_active":"2024-02-19T14:59:07.638275+0000","last_became_peered":"2024-02-19T14:59:07.638275+0000","last_unstale":"2024-02-19T14:59:07.640425+0000","last_undegraded":"2024-02-19T14:59:07.640425+0000","last_fullsized":"2024-02-19T14:59:07.640425+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631957+0000","last_change":"2024-02-19T14:59:07.631957+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631957+0000","last_undegraded":"2024-02-19T14:59:07.631957+0000","last_fullsized":"2024-02-19T14:59:07.631957+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641644+0000","last_change":"2024-02-19T14:59:07.641182+0000","last_active":"2024-02-19T14:59:07.641644+0000","last_peered":"2024-02-19T14:59:07.641644+0000","last_clean":"2024-02-19T14:59:07.641644+0000","last_became_active":"2024-02-19T14:59:07.640867+0000","last_became_peered":"2024-02-19T14:59:07.640867+0000","last_unstale":"2024-02-19T14:59:07.641644+0000","last_undegraded":"2024-02-19T14:59:07.641644+0000","last_fullsized":"2024-02-19T14:59:07.641644+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:24:59.660494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.629585+0000","last_change":"2024-02-19T14:59:07.629585+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.629585+0000","last_undegraded":"2024-02-19T14:59:07.629585+0000","last_fullsized":"2024-02-19T14:59:07.629585+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643474+0000","last_change":"2024-02-19T14:59:07.640656+0000","last_active":"2024-02-19T14:59:07.643474+0000","last_peered":"2024-02-19T14:59:07.643474+0000","last_clean":"2024-02-19T14:59:07.643474+0000","last_became_active":"2024-02-19T14:59:07.640382+0000","last_became_peered":"2024-02-19T14:59:07.640382+0000","last_unstale":"2024-02-19T14:59:07.643474+0000","last_undegraded":"2024-02-19T14:59:07.643474+0000","last_fullsized":"2024-02-19T14:59:07.643474+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:43:14.508468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.626274+0000","last_change":"2024-02-19T14:59:07.626274+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.626274+0000","last_undegraded":"2024-02-19T14:59:07.626274+0000","last_fullsized":"2024-02-19T14:59:07.626274+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.640387+0000","last_change":"2024-02-19T14:59:07.638470+0000","last_active":"2024-02-19T14:59:07.640387+0000","last_peered":"2024-02-19T14:59:07.640387+0000","last_clean":"2024-02-19T14:59:07.640387+0000","last_became_active":"2024-02-19T14:59:07.638197+0000","last_became_peered":"2024-02-19T14:59:07.638197+0000","last_unstale":"2024-02-19T14:59:07.640387+0000","last_undegraded":"2024-02-19T14:59:07.640387+0000","last_fullsized":"2024-02-19T14:59:07.640387+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:09.994584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630266+0000","last_change":"2024-02-19T14:59:07.630266+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630266+0000","last_undegraded":"2024-02-19T14:59:07.630266+0000","last_fullsized":"2024-02-19T14:59:07.630266+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.643869+0000","last_change":"2024-02-19T14:59:07.640562+0000","last_active":"2024-02-19T14:59:07.643869+0000","last_peered":"2024-02-19T14:59:07.643869+0000","last_clean":"2024-02-19T14:59:07.643869+0000","last_became_active":"2024-02-19T14:59:07.640288+0000","last_became_peered":"2024-02-19T14:59:07.640288+0000","last_unstale":"2024-02-19T14:59:07.643869+0000","last_undegraded":"2024-02-19T14:59:07.643869+0000","last_fullsized":"2024-02-19T14:59:07.643869+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634982+0000","last_change":"2024-02-19T14:59:07.634982+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634982+0000","last_undegraded":"2024-02-19T14:59:07.634982+0000","last_fullsized":"2024-02-19T14:59:07.634982+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647753+0000","last_change":"2024-02-19T14:59:07.645290+0000","last_active":"2024-02-19T14:59:07.647753+0000","last_peered":"2024-02-19T14:59:07.647753+0000","last_clean":"2024-02-19T14:59:07.647753+0000","last_became_active":"2024-02-19T14:59:07.644933+0000","last_became_peered":"2024-02-19T14:59:07.644933+0000","last_unstale":"2024-02-19T14:59:07.647753+0000","last_undegraded":"2024-02-19T14:59:07.647753+0000","last_fullsized":"2024-02-19T14:59:07.647753+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:37.251826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.630878+0000","last_change":"2024-02-19T14:59:07.630878+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.630878+0000","last_undegraded":"2024-02-19T14:59:07.630878+0000","last_fullsized":"2024-02-19T14:59:07.630878+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.647675+0000","last_change":"2024-02-19T14:59:07.643990+0000","last_active":"2024-02-19T14:59:07.647675+0000","last_peered":"2024-02-19T14:59:07.647675+0000","last_clean":"2024-02-19T14:59:07.647675+0000","last_became_active":"2024-02-19T14:59:07.643708+0000","last_became_peered":"2024-02-19T14:59:07.643708+0000","last_unstale":"2024-02-19T14:59:07.647675+0000","last_undegraded":"2024-02-19T14:59:07.647675+0000","last_fullsized":"2024-02-19T14:59:07.647675+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:00.752877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.631620+0000","last_change":"2024-02-19T14:59:07.631620+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.631620+0000","last_undegraded":"2024-02-19T14:59:07.631620+0000","last_fullsized":"2024-02-19T14:59:07.631620+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.637822+0000","last_change":"2024-02-19T14:59:07.634059+0000","last_active":"2024-02-19T14:59:07.637822+0000","last_peered":"2024-02-19T14:59:07.637822+0000","last_clean":"2024-02-19T14:59:07.637822+0000","last_became_active":"2024-02-19T14:59:07.633768+0000","last_became_peered":"2024-02-19T14:59:07.633768+0000","last_unstale":"2024-02-19T14:59:07.637822+0000","last_undegraded":"2024-02-19T14:59:07.637822+0000","last_fullsized":"2024-02-19T14:59:07.637822+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:16:52.944510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.633788+0000","last_change":"2024-02-19T14:59:07.633788+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.633788+0000","last_undegraded":"2024-02-19T14:59:07.633788+0000","last_fullsized":"2024-02-19T14:59:07.633788+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.636397+0000","last_change":"2024-02-19T14:59:07.632792+0000","last_active":"2024-02-19T14:59:07.636397+0000","last_peered":"2024-02-19T14:59:07.636397+0000","last_clean":"2024-02-19T14:59:07.636397+0000","last_became_active":"2024-02-19T14:59:07.632529+0000","last_became_peered":"2024-02-19T14:59:07.632529+0000","last_unstale":"2024-02-19T14:59:07.636397+0000","last_undegraded":"2024-02-19T14:59:07.636397+0000","last_fullsized":"2024-02-19T14:59:07.636397+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:58.190651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.638240+0000","last_change":"2024-02-19T14:59:07.638240+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.638240+0000","last_undegraded":"2024-02-19T14:59:07.638240+0000","last_fullsized":"2024-02-19T14:59:07.638240+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.646917+0000","last_change":"2024-02-19T14:59:07.643336+0000","last_active":"2024-02-19T14:59:07.646917+0000","last_peered":"2024-02-19T14:59:07.646917+0000","last_clean":"2024-02-19T14:59:07.646917+0000","last_became_active":"2024-02-19T14:59:07.642778+0000","last_became_peered":"2024-02-19T14:59:07.642778+0000","last_unstale":"2024-02-19T14:59:07.646917+0000","last_undegraded":"2024-02-19T14:59:07.646917+0000","last_fullsized":"2024-02-19T14:59:07.646917+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:36.331774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":186,"state":"creating+peering","last_fresh":"2024-02-19T14:59:07.634502+0000","last_change":"2024-02-19T14:59:07.634502+0000","last_active":"2024-02-19T14:59:07.620339+0000","last_peered":"2024-02-19T14:59:07.620339+0000","last_clean":"2024-02-19T14:59:07.620339+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T14:59:07.634502+0000","last_undegraded":"2024-02-19T14:59:07.634502+0000","last_fullsized":"2024-02-19T14:59:07.634502+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.641427+0000","last_change":"2024-02-19T14:59:07.636409+0000","last_active":"2024-02-19T14:59:07.641427+0000","last_peered":"2024-02-19T14:59:07.641427+0000","last_clean":"2024-02-19T14:59:07.641427+0000","last_became_active":"2024-02-19T14:59:07.636136+0000","last_became_peered":"2024-02-19T14:59:07.636136+0000","last_unstale":"2024-02-19T14:59:07.641427+0000","last_undegraded":"2024-02-19T14:59:07.641427+0000","last_fullsized":"2024-02-19T14:59:07.641427+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:11.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":480,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-19T14:59:07.627573+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:07.627573+0000","last_peered":"2024-02-19T14:59:07.627573+0000","last_clean":"2024-02-19T14:59:07.627573+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:07.627573+0000","last_undegraded":"2024-02-19T14:59:07.627573+0000","last_fullsized":"2024-02-19T14:59:07.627573+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34060,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337780,"statfs":{"total":96636764160,"available":96601886720,"internally_reserved":0,"allocated":2949120,"data_stored":1741638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.505,"15min":0.495},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51600000000000001,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.748,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69299999999999995,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.627,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.629}]},{"osd":4,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46200000000000002,"15min":0.44900000000000001},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.753,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.499,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.75,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.79000000000000004,"5min":0.871,"15min":0.871},"last":0.498}]},{"osd":6,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.84399999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574939,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33476,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338364,"statfs":{"total":96636764160,"available":96602484736,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.63900000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.69299999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.92400000000000004}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.876},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.94899999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.68000000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.313},"min":{"1min":0.217,"5min":0.159,"15min":0.159},"max":{"1min":0.45100000000000001,"5min":3.694,"15min":3.694},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36799999999999999,"15min":0.32600000000000001},"min":{"1min":0.248,"5min":0.192,"15min":0.189},"max":{"1min":0.55300000000000005,"5min":4.835,"15min":4.835},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.41699999999999998,"15min":0.378},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.498,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.314}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41399999999999998,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76200000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.621},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78700000000000003,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.41999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72099999999999997,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.59699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33472,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338368,"statfs":{"total":96636764160,"available":96602488832,"internally_reserved":0,"allocated":2342912,"data_stored":1144603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.374},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.35299999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.312},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.748}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.623},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.85799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33480,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338360,"statfs":{"total":96636764160,"available":96602480640,"internally_reserved":0,"allocated":2355200,"data_stored":1151270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.32600000000000001},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32300000000000001,"15min":0.33200000000000002},"min":{"1min":0.24299999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.45600000000000002,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36899999999999999,"15min":0.38500000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42199999999999999,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.54900000000000004},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.309},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.96799999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.0009999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.44800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.56499999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.82399999999999995,"15min":0.996},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.036},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.71799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34048,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337792,"statfs":{"total":96636764160,"available":96601899008,"internally_reserved":0,"allocated":2936832,"data_stored":1734971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.435},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.85299999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:13.902 INFO:tasks.ceph.mds.d.smithi150.stderr:2024-02-19T14:59:13.899+0000 7fd63d32a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:59:16.006 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:16.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.119+0000 7fc9f85f4640 1 Processor -- start 2024-02-19T14:59:16.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.119+0000 7fc9f85f4640 1 -- start start 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.119+0000 7fc9f85f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f00c8e00 0x7fc9f006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.119+0000 7fc9f85f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f01551f0 con 0x7fc9f0152700 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.119+0000 7fc9f85f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f0155490 con 0x7fc9f006d910 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.119+0000 7fc9f85f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f0155770 con 0x7fc9f00c8e00 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f6390640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc9f0152700 0x7fc9f00c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40014/0 (socket says 172.21.15.78:40014) 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f6390640 1 -- 172.21.15.78:0/1438023128 learned_addr learned my addr 172.21.15.78:0/1438023128 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f5b8f640 1 --2- 172.21.15.78:0/1438023128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f00c8e00 0x7fc9f006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f538e640 1 -- 172.21.15.78:0/1438023128 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137671299 0 0) 0x7fc9f01551f0 con 0x7fc9f0152700 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f538e640 1 -- 172.21.15.78:0/1438023128 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc003520 con 0x7fc9f0152700 2024-02-19T14:59:16.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f538e640 1 -- 172.21.15.78:0/1438023128 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236573215 0 0) 0x7fc9f0155490 con 0x7fc9f006d910 2024-02-19T14:59:16.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f5b8f640 1 -- 172.21.15.78:0/1438023128 >> v1:172.21.15.150:6789/0 conn(0x7fc9f006d910 legacy=0x7fc9f015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f5b8f640 1 -- 172.21.15.78:0/1438023128 >> v1:172.21.15.78:6789/0 conn(0x7fc9f0152700 legacy=0x7fc9f00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.120+0000 7fc9f5b8f640 1 -- 172.21.15.78:0/1438023128 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f0155a50 con 0x7fc9f00c8e00 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.121+0000 7fc9f5b8f640 1 --2- 172.21.15.78:0/1438023128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f00c8e00 0x7fc9f006c2a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00097d0 tx=0x7fc9e002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6a0b797e86a920c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.121+0000 7fc9f538e640 1 -- 172.21.15.78:0/1438023128 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0031800 con 0x7fc9f00c8e00 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.121+0000 7fc9f538e640 1 -- 172.21.15.78:0/1438023128 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e0031960 con 0x7fc9f00c8e00 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.121+0000 7fc9f538e640 1 -- 172.21.15.78:0/1438023128 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0031cd0 con 0x7fc9f00c8e00 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1438023128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f00c8e00 msgr2=0x7fc9f006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1438023128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f00c8e00 0x7fc9f006c2a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00097d0 tx=0x7fc9e002b8a0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1438023128 shutdown_connections 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1438023128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f00c8e00 0x7fc9f006c2a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1438023128 >> 172.21.15.78:0/1438023128 conn(0x7fc9f00cac90 msgr2=0x7fc9f00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:16.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1438023128 shutdown_connections 2024-02-19T14:59:16.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.122+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1438023128 wait complete. 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.123+0000 7fc9f85f4640 1 Processor -- start 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.123+0000 7fc9f85f4640 1 -- start start 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f85f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006d910 0x7fc9f0144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f85f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 0x7fc9f00aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f85f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f0152700 0x7fc9f01ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f85f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f01ead60 con 0x7fc9f00c8e00 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f85f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f01eb5c0 con 0x7fc9f006d910 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f85f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f00abfb0 con 0x7fc9f0152700 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f5b8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 0x7fc9f00aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f5b8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 0x7fc9f00aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33070/0 (socket says 172.21.15.78:33070) 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f5b8f640 1 -- 172.21.15.78:0/1336313955 learned_addr learned my addr 172.21.15.78:0/1336313955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:16.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f0152700 0x7fc9f01ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6390640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006d910 0x7fc9f0144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 -- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006d910 msgr2=0x7fc9f0144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006d910 0x7fc9f0144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 -- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 msgr2=0x7fc9f00aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 0x7fc9f00aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 -- 172.21.15.78:0/1336313955 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e0009400 con 0x7fc9f0152700 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f5b8f640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 0x7fc9f00aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9f6b91640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f0152700 0x7fc9f01ee2d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800c6a0 tx=0x7fc9e800cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.124+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8003b70 con 0x7fc9f0152700 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.125+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e8003cf0 con 0x7fc9f0152700 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.125+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f01450b0 con 0x7fc9f0152700 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.125+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80052b0 con 0x7fc9f0152700 2024-02-19T14:59:16.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.125+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f01eb250 con 0x7fc9f0152700 2024-02-19T14:59:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.126+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f00c4cc0 con 0x7fc9f0152700 2024-02-19T14:59:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.126+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc9e8005410 con 0x7fc9f0152700 2024-02-19T14:59:16.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.126+0000 7fc9e6ffd640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9d406b810 0x7fc9d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.126+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc9e8089f40 con 0x7fc9f0152700 2024-02-19T14:59:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.126+0000 7fc9f6390640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9d406b810 0x7fc9d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.127+0000 7fc9f6390640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9d406b810 0x7fc9d406dcd0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec005880 tx=0x7fc9ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.131+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc9e8054910 con 0x7fc9f0152700 2024-02-19T14:59:16.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.300+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9f006ded0 con 0x7fc9d406b810 2024-02-19T14:59:16.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.307+0000 7fc9e6ffd640 1 -- 172.21.15.78:0/1336313955 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289871 (secure 0 0 0) 0x7fc9ec00f820 con 0x7fc9d406b810 2024-02-19T14:59:16.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:16.314 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:16.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9d406b810 msgr2=0x7fc9d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9d406b810 0x7fc9d406dcd0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec005880 tx=0x7fc9ec009040 comp rx=0 tx=0).stop 2024-02-19T14:59:16.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f0152700 msgr2=0x7fc9f01ee2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f0152700 0x7fc9f01ee2d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800c6a0 tx=0x7fc9e800cb60 comp rx=0 tx=0).stop 2024-02-19T14:59:16.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 shutdown_connections 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc9d406b810 0x7fc9d406dcd0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00c8e00 0x7fc9f00aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9f0152700 0x7fc9f01ee2d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 --2- 172.21.15.78:0/1336313955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006d910 0x7fc9f0144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 >> 172.21.15.78:0/1336313955 conn(0x7fc9f00cac90 msgr2=0x7fc9f006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 shutdown_connections 2024-02-19T14:59:16.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.310+0000 7fc9f85f4640 1 -- 172.21.15.78:0/1336313955 wait complete. 2024-02-19T14:59:16.323 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-02-19T14:59:16.145483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269312,"kb_used_data":19816,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754705408,"statfs":{"total":773094113280,"available":772818337792,"internally_reserved":0,"allocated":20291584,"data_stored":10506588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001780"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043539+0000","last_change":"2024-02-19T14:59:08.673569+0000","last_active":"2024-02-19T14:59:10.043539+0000","last_peered":"2024-02-19T14:59:10.043539+0000","last_clean":"2024-02-19T14:59:10.043539+0000","last_became_active":"2024-02-19T14:59:08.673290+0000","last_became_peered":"2024-02-19T14:59:08.673290+0000","last_unstale":"2024-02-19T14:59:10.043539+0000","last_undegraded":"2024-02-19T14:59:10.043539+0000","last_fullsized":"2024-02-19T14:59:10.043539+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:55:09.731594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042487+0000","last_change":"2024-02-19T14:59:08.664427+0000","last_active":"2024-02-19T14:59:10.042487+0000","last_peered":"2024-02-19T14:59:10.042487+0000","last_clean":"2024-02-19T14:59:10.042487+0000","last_became_active":"2024-02-19T14:59:08.664152+0000","last_became_peered":"2024-02-19T14:59:08.664152+0000","last_unstale":"2024-02-19T14:59:10.042487+0000","last_undegraded":"2024-02-19T14:59:10.042487+0000","last_fullsized":"2024-02-19T14:59:10.042487+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:06.608414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655496+0000","last_change":"2024-02-19T14:59:08.663391+0000","last_active":"2024-02-19T14:59:09.655496+0000","last_peered":"2024-02-19T14:59:09.655496+0000","last_clean":"2024-02-19T14:59:09.655496+0000","last_became_active":"2024-02-19T14:59:08.663042+0000","last_became_peered":"2024-02-19T14:59:08.663042+0000","last_unstale":"2024-02-19T14:59:09.655496+0000","last_undegraded":"2024-02-19T14:59:09.655496+0000","last_fullsized":"2024-02-19T14:59:09.655496+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:03.629327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658649+0000","last_change":"2024-02-19T14:59:08.648997+0000","last_active":"2024-02-19T14:59:09.658649+0000","last_peered":"2024-02-19T14:59:09.658649+0000","last_clean":"2024-02-19T14:59:09.658649+0000","last_became_active":"2024-02-19T14:59:08.648647+0000","last_became_peered":"2024-02-19T14:59:08.648647+0000","last_unstale":"2024-02-19T14:59:09.658649+0000","last_undegraded":"2024-02-19T14:59:09.658649+0000","last_fullsized":"2024-02-19T14:59:09.658649+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:39:38.256650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042933+0000","last_change":"2024-02-19T14:59:08.665199+0000","last_active":"2024-02-19T14:59:10.042933+0000","last_peered":"2024-02-19T14:59:10.042933+0000","last_clean":"2024-02-19T14:59:10.042933+0000","last_became_active":"2024-02-19T14:59:08.664903+0000","last_became_peered":"2024-02-19T14:59:08.664903+0000","last_unstale":"2024-02-19T14:59:10.042933+0000","last_undegraded":"2024-02-19T14:59:10.042933+0000","last_fullsized":"2024-02-19T14:59:10.042933+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:19:30.207085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.660266+0000","last_change":"2024-02-19T14:59:08.650378+0000","last_active":"2024-02-19T14:59:09.660266+0000","last_peered":"2024-02-19T14:59:09.660266+0000","last_clean":"2024-02-19T14:59:09.660266+0000","last_became_active":"2024-02-19T14:59:08.650055+0000","last_became_peered":"2024-02-19T14:59:08.650055+0000","last_unstale":"2024-02-19T14:59:09.660266+0000","last_undegraded":"2024-02-19T14:59:09.660266+0000","last_fullsized":"2024-02-19T14:59:09.660266+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:21.511187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655347+0000","last_change":"2024-02-19T14:59:08.652734+0000","last_active":"2024-02-19T14:59:09.655347+0000","last_peered":"2024-02-19T14:59:09.655347+0000","last_clean":"2024-02-19T14:59:09.655347+0000","last_became_active":"2024-02-19T14:59:08.652475+0000","last_became_peered":"2024-02-19T14:59:08.652475+0000","last_unstale":"2024-02-19T14:59:09.655347+0000","last_undegraded":"2024-02-19T14:59:09.655347+0000","last_fullsized":"2024-02-19T14:59:09.655347+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:59:25.379119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659845+0000","last_change":"2024-02-19T14:59:08.657553+0000","last_active":"2024-02-19T14:59:09.659845+0000","last_peered":"2024-02-19T14:59:09.659845+0000","last_clean":"2024-02-19T14:59:09.659845+0000","last_became_active":"2024-02-19T14:59:08.657198+0000","last_became_peered":"2024-02-19T14:59:08.657198+0000","last_unstale":"2024-02-19T14:59:09.659845+0000","last_undegraded":"2024-02-19T14:59:09.659845+0000","last_fullsized":"2024-02-19T14:59:09.659845+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:44.086234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042943+0000","last_change":"2024-02-19T14:59:08.660309+0000","last_active":"2024-02-19T14:59:10.042943+0000","last_peered":"2024-02-19T14:59:10.042943+0000","last_clean":"2024-02-19T14:59:10.042943+0000","last_became_active":"2024-02-19T14:59:08.660056+0000","last_became_peered":"2024-02-19T14:59:08.660056+0000","last_unstale":"2024-02-19T14:59:10.042943+0000","last_undegraded":"2024-02-19T14:59:10.042943+0000","last_fullsized":"2024-02-19T14:59:10.042943+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:07:36.043058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655809+0000","last_change":"2024-02-19T14:59:08.651796+0000","last_active":"2024-02-19T14:59:09.655809+0000","last_peered":"2024-02-19T14:59:09.655809+0000","last_clean":"2024-02-19T14:59:09.655809+0000","last_became_active":"2024-02-19T14:59:08.651514+0000","last_became_peered":"2024-02-19T14:59:08.651514+0000","last_unstale":"2024-02-19T14:59:09.655809+0000","last_undegraded":"2024-02-19T14:59:09.655809+0000","last_fullsized":"2024-02-19T14:59:09.655809+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:10:23.762875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658992+0000","last_change":"2024-02-19T14:59:08.663540+0000","last_active":"2024-02-19T14:59:09.658992+0000","last_peered":"2024-02-19T14:59:09.658992+0000","last_clean":"2024-02-19T14:59:09.658992+0000","last_became_active":"2024-02-19T14:59:08.663277+0000","last_became_peered":"2024-02-19T14:59:08.663277+0000","last_unstale":"2024-02-19T14:59:09.658992+0000","last_undegraded":"2024-02-19T14:59:09.658992+0000","last_fullsized":"2024-02-19T14:59:09.658992+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:29:45.401958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042773+0000","last_change":"2024-02-19T14:59:08.672300+0000","last_active":"2024-02-19T14:59:10.042773+0000","last_peered":"2024-02-19T14:59:10.042773+0000","last_clean":"2024-02-19T14:59:10.042773+0000","last_became_active":"2024-02-19T14:59:08.671980+0000","last_became_peered":"2024-02-19T14:59:08.671980+0000","last_unstale":"2024-02-19T14:59:10.042773+0000","last_undegraded":"2024-02-19T14:59:10.042773+0000","last_fullsized":"2024-02-19T14:59:10.042773+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:22.673826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659354+0000","last_change":"2024-02-19T14:59:08.649859+0000","last_active":"2024-02-19T14:59:09.659354+0000","last_peered":"2024-02-19T14:59:09.659354+0000","last_clean":"2024-02-19T14:59:09.659354+0000","last_became_active":"2024-02-19T14:59:08.649565+0000","last_became_peered":"2024-02-19T14:59:08.649565+0000","last_unstale":"2024-02-19T14:59:09.659354+0000","last_undegraded":"2024-02-19T14:59:09.659354+0000","last_fullsized":"2024-02-19T14:59:09.659354+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:30.874158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042484+0000","last_change":"2024-02-19T14:59:08.666854+0000","last_active":"2024-02-19T14:59:10.042484+0000","last_peered":"2024-02-19T14:59:10.042484+0000","last_clean":"2024-02-19T14:59:10.042484+0000","last_became_active":"2024-02-19T14:59:08.666559+0000","last_became_peered":"2024-02-19T14:59:08.666559+0000","last_unstale":"2024-02-19T14:59:10.042484+0000","last_undegraded":"2024-02-19T14:59:10.042484+0000","last_fullsized":"2024-02-19T14:59:10.042484+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:42:12.167293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658063+0000","last_change":"2024-02-19T14:59:08.656271+0000","last_active":"2024-02-19T14:59:09.658063+0000","last_peered":"2024-02-19T14:59:09.658063+0000","last_clean":"2024-02-19T14:59:09.658063+0000","last_became_active":"2024-02-19T14:59:08.654343+0000","last_became_peered":"2024-02-19T14:59:08.654343+0000","last_unstale":"2024-02-19T14:59:09.658063+0000","last_undegraded":"2024-02-19T14:59:09.658063+0000","last_fullsized":"2024-02-19T14:59:09.658063+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:31.289032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659081+0000","last_change":"2024-02-19T14:59:08.652001+0000","last_active":"2024-02-19T14:59:09.659081+0000","last_peered":"2024-02-19T14:59:09.659081+0000","last_clean":"2024-02-19T14:59:09.659081+0000","last_became_active":"2024-02-19T14:59:08.651687+0000","last_became_peered":"2024-02-19T14:59:08.651687+0000","last_unstale":"2024-02-19T14:59:09.659081+0000","last_undegraded":"2024-02-19T14:59:09.659081+0000","last_fullsized":"2024-02-19T14:59:09.659081+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:22.673826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043396+0000","last_change":"2024-02-19T14:59:08.668800+0000","last_active":"2024-02-19T14:59:10.043396+0000","last_peered":"2024-02-19T14:59:10.043396+0000","last_clean":"2024-02-19T14:59:10.043396+0000","last_became_active":"2024-02-19T14:59:08.668510+0000","last_became_peered":"2024-02-19T14:59:08.668510+0000","last_unstale":"2024-02-19T14:59:10.043396+0000","last_undegraded":"2024-02-19T14:59:10.043396+0000","last_fullsized":"2024-02-19T14:59:10.043396+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:22:32.957039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655844+0000","last_change":"2024-02-19T14:59:08.648803+0000","last_active":"2024-02-19T14:59:09.655844+0000","last_peered":"2024-02-19T14:59:09.655844+0000","last_clean":"2024-02-19T14:59:09.655844+0000","last_became_active":"2024-02-19T14:59:08.648505+0000","last_became_peered":"2024-02-19T14:59:08.648505+0000","last_unstale":"2024-02-19T14:59:09.655844+0000","last_undegraded":"2024-02-19T14:59:09.655844+0000","last_fullsized":"2024-02-19T14:59:09.655844+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:17:51.133160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043093+0000","last_change":"2024-02-19T14:59:08.665827+0000","last_active":"2024-02-19T14:59:10.043093+0000","last_peered":"2024-02-19T14:59:10.043093+0000","last_clean":"2024-02-19T14:59:10.043093+0000","last_became_active":"2024-02-19T14:59:08.665207+0000","last_became_peered":"2024-02-19T14:59:08.665207+0000","last_unstale":"2024-02-19T14:59:10.043093+0000","last_undegraded":"2024-02-19T14:59:10.043093+0000","last_fullsized":"2024-02-19T14:59:10.043093+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:32:30.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.660345+0000","last_change":"2024-02-19T14:59:08.665421+0000","last_active":"2024-02-19T14:59:09.660345+0000","last_peered":"2024-02-19T14:59:09.660345+0000","last_clean":"2024-02-19T14:59:09.660345+0000","last_became_active":"2024-02-19T14:59:08.664713+0000","last_became_peered":"2024-02-19T14:59:08.664713+0000","last_unstale":"2024-02-19T14:59:09.660345+0000","last_undegraded":"2024-02-19T14:59:09.660345+0000","last_fullsized":"2024-02-19T14:59:09.660345+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:06.414074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655501+0000","last_change":"2024-02-19T14:59:08.663267+0000","last_active":"2024-02-19T14:59:09.655501+0000","last_peered":"2024-02-19T14:59:09.655501+0000","last_clean":"2024-02-19T14:59:09.655501+0000","last_became_active":"2024-02-19T14:59:08.662959+0000","last_became_peered":"2024-02-19T14:59:08.662959+0000","last_unstale":"2024-02-19T14:59:09.655501+0000","last_undegraded":"2024-02-19T14:59:09.655501+0000","last_fullsized":"2024-02-19T14:59:09.655501+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:21:43.309920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658123+0000","last_change":"2024-02-19T14:59:08.658625+0000","last_active":"2024-02-19T14:59:09.658123+0000","last_peered":"2024-02-19T14:59:09.658123+0000","last_clean":"2024-02-19T14:59:09.658123+0000","last_became_active":"2024-02-19T14:59:08.658320+0000","last_became_peered":"2024-02-19T14:59:08.658320+0000","last_unstale":"2024-02-19T14:59:09.658123+0000","last_undegraded":"2024-02-19T14:59:09.658123+0000","last_fullsized":"2024-02-19T14:59:09.658123+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:21.003166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042442+0000","last_change":"2024-02-19T14:59:08.655381+0000","last_active":"2024-02-19T14:59:10.042442+0000","last_peered":"2024-02-19T14:59:10.042442+0000","last_clean":"2024-02-19T14:59:10.042442+0000","last_became_active":"2024-02-19T14:59:08.655124+0000","last_became_peered":"2024-02-19T14:59:08.655124+0000","last_unstale":"2024-02-19T14:59:10.042442+0000","last_undegraded":"2024-02-19T14:59:10.042442+0000","last_fullsized":"2024-02-19T14:59:10.042442+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:52:20.337238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042593+0000","last_change":"2024-02-19T14:59:08.658129+0000","last_active":"2024-02-19T14:59:10.042593+0000","last_peered":"2024-02-19T14:59:10.042593+0000","last_clean":"2024-02-19T14:59:10.042593+0000","last_became_active":"2024-02-19T14:59:08.657860+0000","last_became_peered":"2024-02-19T14:59:08.657860+0000","last_unstale":"2024-02-19T14:59:10.042593+0000","last_undegraded":"2024-02-19T14:59:10.042593+0000","last_fullsized":"2024-02-19T14:59:10.042593+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:37.667731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043581+0000","last_change":"2024-02-19T14:59:08.658129+0000","last_active":"2024-02-19T14:59:10.043581+0000","last_peered":"2024-02-19T14:59:10.043581+0000","last_clean":"2024-02-19T14:59:10.043581+0000","last_became_active":"2024-02-19T14:59:08.657898+0000","last_became_peered":"2024-02-19T14:59:08.657898+0000","last_unstale":"2024-02-19T14:59:10.043581+0000","last_undegraded":"2024-02-19T14:59:10.043581+0000","last_fullsized":"2024-02-19T14:59:10.043581+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:11:29.988303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655885+0000","last_change":"2024-02-19T14:59:08.660706+0000","last_active":"2024-02-19T14:59:09.655885+0000","last_peered":"2024-02-19T14:59:09.655885+0000","last_clean":"2024-02-19T14:59:09.655885+0000","last_became_active":"2024-02-19T14:59:08.660408+0000","last_became_peered":"2024-02-19T14:59:08.660408+0000","last_unstale":"2024-02-19T14:59:09.655885+0000","last_undegraded":"2024-02-19T14:59:09.655885+0000","last_fullsized":"2024-02-19T14:59:09.655885+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:12:43.058084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658837+0000","last_change":"2024-02-19T14:59:08.655376+0000","last_active":"2024-02-19T14:59:09.658837+0000","last_peered":"2024-02-19T14:59:09.658837+0000","last_clean":"2024-02-19T14:59:09.658837+0000","last_became_active":"2024-02-19T14:59:08.655079+0000","last_became_peered":"2024-02-19T14:59:08.655079+0000","last_unstale":"2024-02-19T14:59:09.658837+0000","last_undegraded":"2024-02-19T14:59:09.658837+0000","last_fullsized":"2024-02-19T14:59:09.658837+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:03:17.903552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042914+0000","last_change":"2024-02-19T14:59:08.664982+0000","last_active":"2024-02-19T14:59:10.042914+0000","last_peered":"2024-02-19T14:59:10.042914+0000","last_clean":"2024-02-19T14:59:10.042914+0000","last_became_active":"2024-02-19T14:59:08.663389+0000","last_became_peered":"2024-02-19T14:59:08.663389+0000","last_unstale":"2024-02-19T14:59:10.042914+0000","last_undegraded":"2024-02-19T14:59:10.042914+0000","last_fullsized":"2024-02-19T14:59:10.042914+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:06.304635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042783+0000","last_change":"2024-02-19T14:59:08.659266+0000","last_active":"2024-02-19T14:59:10.042783+0000","last_peered":"2024-02-19T14:59:10.042783+0000","last_clean":"2024-02-19T14:59:10.042783+0000","last_became_active":"2024-02-19T14:59:08.658952+0000","last_became_peered":"2024-02-19T14:59:08.658952+0000","last_unstale":"2024-02-19T14:59:10.042783+0000","last_undegraded":"2024-02-19T14:59:10.042783+0000","last_fullsized":"2024-02-19T14:59:10.042783+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:31.289032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.660295+0000","last_change":"2024-02-19T14:59:08.665667+0000","last_active":"2024-02-19T14:59:09.660295+0000","last_peered":"2024-02-19T14:59:09.660295+0000","last_clean":"2024-02-19T14:59:09.660295+0000","last_became_active":"2024-02-19T14:59:08.665122+0000","last_became_peered":"2024-02-19T14:59:08.665122+0000","last_unstale":"2024-02-19T14:59:09.660295+0000","last_undegraded":"2024-02-19T14:59:09.660295+0000","last_fullsized":"2024-02-19T14:59:09.660295+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:45.870567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042593+0000","last_change":"2024-02-19T14:59:08.658215+0000","last_active":"2024-02-19T14:59:10.042593+0000","last_peered":"2024-02-19T14:59:10.042593+0000","last_clean":"2024-02-19T14:59:10.042593+0000","last_became_active":"2024-02-19T14:59:08.657937+0000","last_became_peered":"2024-02-19T14:59:08.657937+0000","last_unstale":"2024-02-19T14:59:10.042593+0000","last_undegraded":"2024-02-19T14:59:10.042593+0000","last_fullsized":"2024-02-19T14:59:10.042593+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:49:48.790662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659281+0000","last_change":"2024-02-19T14:59:08.671444+0000","last_active":"2024-02-19T14:59:09.659281+0000","last_peered":"2024-02-19T14:59:09.659281+0000","last_clean":"2024-02-19T14:59:09.659281+0000","last_became_active":"2024-02-19T14:59:08.671143+0000","last_became_peered":"2024-02-19T14:59:08.671143+0000","last_unstale":"2024-02-19T14:59:09.659281+0000","last_undegraded":"2024-02-19T14:59:09.659281+0000","last_fullsized":"2024-02-19T14:59:09.659281+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:06.304635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655580+0000","last_change":"2024-02-19T14:59:08.651650+0000","last_active":"2024-02-19T14:59:09.655580+0000","last_peered":"2024-02-19T14:59:09.655580+0000","last_clean":"2024-02-19T14:59:09.655580+0000","last_became_active":"2024-02-19T14:59:08.651347+0000","last_became_peered":"2024-02-19T14:59:08.651347+0000","last_unstale":"2024-02-19T14:59:09.655580+0000","last_undegraded":"2024-02-19T14:59:09.655580+0000","last_fullsized":"2024-02-19T14:59:09.655580+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:05.019083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"187'4","reported_seq":18,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042946+0000","last_change":"2024-02-19T14:59:07.638344+0000","last_active":"2024-02-19T14:59:10.042946+0000","last_peered":"2024-02-19T14:59:10.042946+0000","last_clean":"2024-02-19T14:59:10.042946+0000","last_became_active":"2024-02-19T14:59:07.638065+0000","last_became_peered":"2024-02-19T14:59:07.638065+0000","last_unstale":"2024-02-19T14:59:10.042946+0000","last_undegraded":"2024-02-19T14:59:10.042946+0000","last_fullsized":"2024-02-19T14:59:10.042946+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:46:53.420397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655632+0000","last_change":"2024-02-19T14:59:08.657045+0000","last_active":"2024-02-19T14:59:09.655632+0000","last_peered":"2024-02-19T14:59:09.655632+0000","last_clean":"2024-02-19T14:59:09.655632+0000","last_became_active":"2024-02-19T14:59:08.656775+0000","last_became_peered":"2024-02-19T14:59:08.656775+0000","last_unstale":"2024-02-19T14:59:09.655632+0000","last_undegraded":"2024-02-19T14:59:09.655632+0000","last_fullsized":"2024-02-19T14:59:09.655632+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:52:37.750354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"187'1","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043189+0000","last_change":"2024-02-19T14:59:07.639820+0000","last_active":"2024-02-19T14:59:10.043189+0000","last_peered":"2024-02-19T14:59:10.043189+0000","last_clean":"2024-02-19T14:59:10.043189+0000","last_became_active":"2024-02-19T14:59:07.639571+0000","last_became_peered":"2024-02-19T14:59:07.639571+0000","last_unstale":"2024-02-19T14:59:10.043189+0000","last_undegraded":"2024-02-19T14:59:10.043189+0000","last_fullsized":"2024-02-19T14:59:10.043189+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:38:29.755042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042959+0000","last_change":"2024-02-19T14:59:08.668077+0000","last_active":"2024-02-19T14:59:10.042959+0000","last_peered":"2024-02-19T14:59:10.042959+0000","last_clean":"2024-02-19T14:59:10.042959+0000","last_became_active":"2024-02-19T14:59:08.667800+0000","last_became_peered":"2024-02-19T14:59:08.667800+0000","last_unstale":"2024-02-19T14:59:10.042959+0000","last_undegraded":"2024-02-19T14:59:10.042959+0000","last_fullsized":"2024-02-19T14:59:10.042959+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:30.874158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"187'2","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659230+0000","last_change":"2024-02-19T14:59:07.641532+0000","last_active":"2024-02-19T14:59:09.659230+0000","last_peered":"2024-02-19T14:59:09.659230+0000","last_clean":"2024-02-19T14:59:09.659230+0000","last_became_active":"2024-02-19T14:59:07.641278+0000","last_became_peered":"2024-02-19T14:59:07.641278+0000","last_unstale":"2024-02-19T14:59:09.659230+0000","last_undegraded":"2024-02-19T14:59:09.659230+0000","last_fullsized":"2024-02-19T14:59:09.659230+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:19:18.989638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655336+0000","last_change":"2024-02-19T14:59:08.651916+0000","last_active":"2024-02-19T14:59:09.655336+0000","last_peered":"2024-02-19T14:59:09.655336+0000","last_clean":"2024-02-19T14:59:09.655336+0000","last_became_active":"2024-02-19T14:59:08.651064+0000","last_became_peered":"2024-02-19T14:59:08.651064+0000","last_unstale":"2024-02-19T14:59:09.655336+0000","last_undegraded":"2024-02-19T14:59:09.655336+0000","last_fullsized":"2024-02-19T14:59:09.655336+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:43:09.017151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"187'1","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658239+0000","last_change":"2024-02-19T14:59:07.646204+0000","last_active":"2024-02-19T14:59:09.658239+0000","last_peered":"2024-02-19T14:59:09.658239+0000","last_clean":"2024-02-19T14:59:09.658239+0000","last_became_active":"2024-02-19T14:59:07.645880+0000","last_became_peered":"2024-02-19T14:59:07.645880+0000","last_unstale":"2024-02-19T14:59:09.658239+0000","last_undegraded":"2024-02-19T14:59:09.658239+0000","last_fullsized":"2024-02-19T14:59:09.658239+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:21.669002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042341+0000","last_change":"2024-02-19T14:59:08.658391+0000","last_active":"2024-02-19T14:59:10.042341+0000","last_peered":"2024-02-19T14:59:10.042341+0000","last_clean":"2024-02-19T14:59:10.042341+0000","last_became_active":"2024-02-19T14:59:08.658061+0000","last_became_peered":"2024-02-19T14:59:08.658061+0000","last_unstale":"2024-02-19T14:59:10.042341+0000","last_undegraded":"2024-02-19T14:59:10.042341+0000","last_fullsized":"2024-02-19T14:59:10.042341+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:33:42.451034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"187'1","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042800+0000","last_change":"2024-02-19T14:59:07.642010+0000","last_active":"2024-02-19T14:59:10.042800+0000","last_peered":"2024-02-19T14:59:10.042800+0000","last_clean":"2024-02-19T14:59:10.042800+0000","last_became_active":"2024-02-19T14:59:07.641735+0000","last_became_peered":"2024-02-19T14:59:07.641735+0000","last_unstale":"2024-02-19T14:59:10.042800+0000","last_undegraded":"2024-02-19T14:59:10.042800+0000","last_fullsized":"2024-02-19T14:59:10.042800+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:43.081410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658093+0000","last_change":"2024-02-19T14:59:08.658277+0000","last_active":"2024-02-19T14:59:09.658093+0000","last_peered":"2024-02-19T14:59:09.658093+0000","last_clean":"2024-02-19T14:59:09.658093+0000","last_became_active":"2024-02-19T14:59:08.658030+0000","last_became_peered":"2024-02-19T14:59:08.658030+0000","last_unstale":"2024-02-19T14:59:09.658093+0000","last_undegraded":"2024-02-19T14:59:09.658093+0000","last_fullsized":"2024-02-19T14:59:09.658093+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:16:14.234839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658231+0000","last_change":"2024-02-19T14:59:07.643447+0000","last_active":"2024-02-19T14:59:09.658231+0000","last_peered":"2024-02-19T14:59:09.658231+0000","last_clean":"2024-02-19T14:59:09.658231+0000","last_became_active":"2024-02-19T14:59:07.643150+0000","last_became_peered":"2024-02-19T14:59:07.643150+0000","last_unstale":"2024-02-19T14:59:09.658231+0000","last_undegraded":"2024-02-19T14:59:09.658231+0000","last_fullsized":"2024-02-19T14:59:09.658231+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:43:47.116497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042602+0000","last_change":"2024-02-19T14:59:08.658536+0000","last_active":"2024-02-19T14:59:10.042602+0000","last_peered":"2024-02-19T14:59:10.042602+0000","last_clean":"2024-02-19T14:59:10.042602+0000","last_became_active":"2024-02-19T14:59:08.658279+0000","last_became_peered":"2024-02-19T14:59:08.658279+0000","last_unstale":"2024-02-19T14:59:10.042602+0000","last_undegraded":"2024-02-19T14:59:10.042602+0000","last_fullsized":"2024-02-19T14:59:10.042602+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:06.759729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655431+0000","last_change":"2024-02-19T14:59:07.641207+0000","last_active":"2024-02-19T14:59:09.655431+0000","last_peered":"2024-02-19T14:59:09.655431+0000","last_clean":"2024-02-19T14:59:09.655431+0000","last_became_active":"2024-02-19T14:59:07.640874+0000","last_became_peered":"2024-02-19T14:59:07.640874+0000","last_unstale":"2024-02-19T14:59:09.655431+0000","last_undegraded":"2024-02-19T14:59:09.655431+0000","last_fullsized":"2024-02-19T14:59:09.655431+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:50:20.985844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658320+0000","last_change":"2024-02-19T14:59:08.662720+0000","last_active":"2024-02-19T14:59:09.658320+0000","last_peered":"2024-02-19T14:59:09.658320+0000","last_clean":"2024-02-19T14:59:09.658320+0000","last_became_active":"2024-02-19T14:59:08.662450+0000","last_became_peered":"2024-02-19T14:59:08.662450+0000","last_unstale":"2024-02-19T14:59:09.658320+0000","last_undegraded":"2024-02-19T14:59:09.658320+0000","last_fullsized":"2024-02-19T14:59:09.658320+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:37.667731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043629+0000","last_change":"2024-02-19T14:59:07.638597+0000","last_active":"2024-02-19T14:59:10.043629+0000","last_peered":"2024-02-19T14:59:10.043629+0000","last_clean":"2024-02-19T14:59:10.043629+0000","last_became_active":"2024-02-19T14:59:07.638358+0000","last_became_peered":"2024-02-19T14:59:07.638358+0000","last_unstale":"2024-02-19T14:59:10.043629+0000","last_undegraded":"2024-02-19T14:59:10.043629+0000","last_fullsized":"2024-02-19T14:59:10.043629+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:05.409250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658335+0000","last_change":"2024-02-19T14:59:08.657390+0000","last_active":"2024-02-19T14:59:09.658335+0000","last_peered":"2024-02-19T14:59:09.658335+0000","last_clean":"2024-02-19T14:59:09.658335+0000","last_became_active":"2024-02-19T14:59:08.657002+0000","last_became_peered":"2024-02-19T14:59:08.657002+0000","last_unstale":"2024-02-19T14:59:09.658335+0000","last_undegraded":"2024-02-19T14:59:09.658335+0000","last_fullsized":"2024-02-19T14:59:09.658335+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:11.947832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659445+0000","last_change":"2024-02-19T14:59:07.643687+0000","last_active":"2024-02-19T14:59:09.659445+0000","last_peered":"2024-02-19T14:59:09.659445+0000","last_clean":"2024-02-19T14:59:09.659445+0000","last_became_active":"2024-02-19T14:59:07.643136+0000","last_became_peered":"2024-02-19T14:59:07.643136+0000","last_unstale":"2024-02-19T14:59:09.659445+0000","last_undegraded":"2024-02-19T14:59:09.659445+0000","last_fullsized":"2024-02-19T14:59:09.659445+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:26:02.808996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655996+0000","last_change":"2024-02-19T14:59:08.654101+0000","last_active":"2024-02-19T14:59:09.655996+0000","last_peered":"2024-02-19T14:59:09.655996+0000","last_clean":"2024-02-19T14:59:09.655996+0000","last_became_active":"2024-02-19T14:59:08.652979+0000","last_became_peered":"2024-02-19T14:59:08.652979+0000","last_unstale":"2024-02-19T14:59:09.655996+0000","last_undegraded":"2024-02-19T14:59:09.655996+0000","last_fullsized":"2024-02-19T14:59:09.655996+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:57:55.460654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"187'3","reported_seq":17,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043304+0000","last_change":"2024-02-19T14:59:07.638366+0000","last_active":"2024-02-19T14:59:10.043304+0000","last_peered":"2024-02-19T14:59:10.043304+0000","last_clean":"2024-02-19T14:59:10.043304+0000","last_became_active":"2024-02-19T14:59:07.638109+0000","last_became_peered":"2024-02-19T14:59:07.638109+0000","last_unstale":"2024-02-19T14:59:10.043304+0000","last_undegraded":"2024-02-19T14:59:10.043304+0000","last_fullsized":"2024-02-19T14:59:10.043304+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:44.865744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659350+0000","last_change":"2024-02-19T14:59:08.662867+0000","last_active":"2024-02-19T14:59:09.659350+0000","last_peered":"2024-02-19T14:59:09.659350+0000","last_clean":"2024-02-19T14:59:09.659350+0000","last_became_active":"2024-02-19T14:59:08.662588+0000","last_became_peered":"2024-02-19T14:59:08.662588+0000","last_unstale":"2024-02-19T14:59:09.659350+0000","last_undegraded":"2024-02-19T14:59:09.659350+0000","last_fullsized":"2024-02-19T14:59:09.659350+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:05:58.257368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"187'6","reported_seq":20,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.660378+0000","last_change":"2024-02-19T14:59:07.646216+0000","last_active":"2024-02-19T14:59:09.660378+0000","last_peered":"2024-02-19T14:59:09.660378+0000","last_clean":"2024-02-19T14:59:09.660378+0000","last_became_active":"2024-02-19T14:59:07.645915+0000","last_became_peered":"2024-02-19T14:59:07.645915+0000","last_unstale":"2024-02-19T14:59:09.660378+0000","last_undegraded":"2024-02-19T14:59:09.660378+0000","last_fullsized":"2024-02-19T14:59:09.660378+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:45.811944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042520+0000","last_change":"2024-02-19T14:59:08.663076+0000","last_active":"2024-02-19T14:59:10.042520+0000","last_peered":"2024-02-19T14:59:10.042520+0000","last_clean":"2024-02-19T14:59:10.042520+0000","last_became_active":"2024-02-19T14:59:08.662803+0000","last_became_peered":"2024-02-19T14:59:08.662803+0000","last_unstale":"2024-02-19T14:59:10.042520+0000","last_undegraded":"2024-02-19T14:59:10.042520+0000","last_fullsized":"2024-02-19T14:59:10.042520+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:09.010114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"187'2","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042922+0000","last_change":"2024-02-19T14:59:07.643409+0000","last_active":"2024-02-19T14:59:10.042922+0000","last_peered":"2024-02-19T14:59:10.042922+0000","last_clean":"2024-02-19T14:59:10.042922+0000","last_became_active":"2024-02-19T14:59:07.643105+0000","last_became_peered":"2024-02-19T14:59:07.643105+0000","last_unstale":"2024-02-19T14:59:10.042922+0000","last_undegraded":"2024-02-19T14:59:10.042922+0000","last_fullsized":"2024-02-19T14:59:10.042922+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:21.010267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042367+0000","last_change":"2024-02-19T14:59:08.671163+0000","last_active":"2024-02-19T14:59:10.042367+0000","last_peered":"2024-02-19T14:59:10.042367+0000","last_clean":"2024-02-19T14:59:10.042367+0000","last_became_active":"2024-02-19T14:59:08.670868+0000","last_became_peered":"2024-02-19T14:59:08.670868+0000","last_unstale":"2024-02-19T14:59:10.042367+0000","last_undegraded":"2024-02-19T14:59:10.042367+0000","last_fullsized":"2024-02-19T14:59:10.042367+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:10.999408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042650+0000","last_change":"2024-02-19T14:59:07.639087+0000","last_active":"2024-02-19T14:59:10.042650+0000","last_peered":"2024-02-19T14:59:10.042650+0000","last_clean":"2024-02-19T14:59:10.042650+0000","last_became_active":"2024-02-19T14:59:07.638819+0000","last_became_peered":"2024-02-19T14:59:07.638819+0000","last_unstale":"2024-02-19T14:59:10.042650+0000","last_undegraded":"2024-02-19T14:59:10.042650+0000","last_fullsized":"2024-02-19T14:59:10.042650+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:07:59.233023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655579+0000","last_change":"2024-02-19T14:59:08.653129+0000","last_active":"2024-02-19T14:59:09.655579+0000","last_peered":"2024-02-19T14:59:09.655579+0000","last_clean":"2024-02-19T14:59:09.655579+0000","last_became_active":"2024-02-19T14:59:08.652809+0000","last_became_peered":"2024-02-19T14:59:08.652809+0000","last_unstale":"2024-02-19T14:59:09.655579+0000","last_undegraded":"2024-02-19T14:59:09.655579+0000","last_fullsized":"2024-02-19T14:59:09.655579+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:53:17.148395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659663+0000","last_change":"2024-02-19T14:59:07.636744+0000","last_active":"2024-02-19T14:59:09.659663+0000","last_peered":"2024-02-19T14:59:09.659663+0000","last_clean":"2024-02-19T14:59:09.659663+0000","last_became_active":"2024-02-19T14:59:07.636464+0000","last_became_peered":"2024-02-19T14:59:07.636464+0000","last_unstale":"2024-02-19T14:59:09.659663+0000","last_undegraded":"2024-02-19T14:59:09.659663+0000","last_fullsized":"2024-02-19T14:59:09.659663+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:41:30.326293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658194+0000","last_change":"2024-02-19T14:59:08.663780+0000","last_active":"2024-02-19T14:59:09.658194+0000","last_peered":"2024-02-19T14:59:09.658194+0000","last_clean":"2024-02-19T14:59:09.658194+0000","last_became_active":"2024-02-19T14:59:08.663498+0000","last_became_peered":"2024-02-19T14:59:08.663498+0000","last_unstale":"2024-02-19T14:59:09.658194+0000","last_undegraded":"2024-02-19T14:59:09.658194+0000","last_fullsized":"2024-02-19T14:59:09.658194+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:50:21.990668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"187'8","reported_seq":22,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042317+0000","last_change":"2024-02-19T14:59:07.643659+0000","last_active":"2024-02-19T14:59:10.042317+0000","last_peered":"2024-02-19T14:59:10.042317+0000","last_clean":"2024-02-19T14:59:10.042317+0000","last_became_active":"2024-02-19T14:59:07.643126+0000","last_became_peered":"2024-02-19T14:59:07.643126+0000","last_unstale":"2024-02-19T14:59:10.042317+0000","last_undegraded":"2024-02-19T14:59:10.042317+0000","last_fullsized":"2024-02-19T14:59:10.042317+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:00:28.894453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042446+0000","last_change":"2024-02-19T14:59:08.662126+0000","last_active":"2024-02-19T14:59:10.042446+0000","last_peered":"2024-02-19T14:59:10.042446+0000","last_clean":"2024-02-19T14:59:10.042446+0000","last_became_active":"2024-02-19T14:59:08.661810+0000","last_became_peered":"2024-02-19T14:59:08.661810+0000","last_unstale":"2024-02-19T14:59:10.042446+0000","last_undegraded":"2024-02-19T14:59:10.042446+0000","last_fullsized":"2024-02-19T14:59:10.042446+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:06.414074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"187'1","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655717+0000","last_change":"2024-02-19T14:59:07.642484+0000","last_active":"2024-02-19T14:59:09.655717+0000","last_peered":"2024-02-19T14:59:09.655717+0000","last_clean":"2024-02-19T14:59:09.655717+0000","last_became_active":"2024-02-19T14:59:07.642254+0000","last_became_peered":"2024-02-19T14:59:07.642254+0000","last_unstale":"2024-02-19T14:59:09.655717+0000","last_undegraded":"2024-02-19T14:59:09.655717+0000","last_fullsized":"2024-02-19T14:59:09.655717+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:27:36.662908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043162+0000","last_change":"2024-02-19T14:59:08.662416+0000","last_active":"2024-02-19T14:59:10.043162+0000","last_peered":"2024-02-19T14:59:10.043162+0000","last_clean":"2024-02-19T14:59:10.043162+0000","last_became_active":"2024-02-19T14:59:08.662113+0000","last_became_peered":"2024-02-19T14:59:08.662113+0000","last_unstale":"2024-02-19T14:59:10.043162+0000","last_undegraded":"2024-02-19T14:59:10.043162+0000","last_fullsized":"2024-02-19T14:59:10.043162+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:21:35.658590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"187'2","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042861+0000","last_change":"2024-02-19T14:59:07.644552+0000","last_active":"2024-02-19T14:59:10.042861+0000","last_peered":"2024-02-19T14:59:10.042861+0000","last_clean":"2024-02-19T14:59:10.042861+0000","last_became_active":"2024-02-19T14:59:07.644274+0000","last_became_peered":"2024-02-19T14:59:07.644274+0000","last_unstale":"2024-02-19T14:59:10.042861+0000","last_undegraded":"2024-02-19T14:59:10.042861+0000","last_fullsized":"2024-02-19T14:59:10.042861+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:46:48.013387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655968+0000","last_change":"2024-02-19T14:59:08.662490+0000","last_active":"2024-02-19T14:59:09.655968+0000","last_peered":"2024-02-19T14:59:09.655968+0000","last_clean":"2024-02-19T14:59:09.655968+0000","last_became_active":"2024-02-19T14:59:08.662242+0000","last_became_peered":"2024-02-19T14:59:08.662242+0000","last_unstale":"2024-02-19T14:59:09.655968+0000","last_undegraded":"2024-02-19T14:59:09.655968+0000","last_fullsized":"2024-02-19T14:59:09.655968+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:38:53.810247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"187'1","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043244+0000","last_change":"2024-02-19T14:59:07.638149+0000","last_active":"2024-02-19T14:59:10.043244+0000","last_peered":"2024-02-19T14:59:10.043244+0000","last_clean":"2024-02-19T14:59:10.043244+0000","last_became_active":"2024-02-19T14:59:07.637845+0000","last_became_peered":"2024-02-19T14:59:07.637845+0000","last_unstale":"2024-02-19T14:59:10.043244+0000","last_undegraded":"2024-02-19T14:59:10.043244+0000","last_fullsized":"2024-02-19T14:59:10.043244+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:36:19.529044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043425+0000","last_change":"2024-02-19T14:59:08.667575+0000","last_active":"2024-02-19T14:59:10.043425+0000","last_peered":"2024-02-19T14:59:10.043425+0000","last_clean":"2024-02-19T14:59:10.043425+0000","last_became_active":"2024-02-19T14:59:08.667294+0000","last_became_peered":"2024-02-19T14:59:08.667294+0000","last_unstale":"2024-02-19T14:59:10.043425+0000","last_undegraded":"2024-02-19T14:59:10.043425+0000","last_fullsized":"2024-02-19T14:59:10.043425+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:44.086234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043054+0000","last_change":"2024-02-19T14:59:07.645385+0000","last_active":"2024-02-19T14:59:10.043054+0000","last_peered":"2024-02-19T14:59:10.043054+0000","last_clean":"2024-02-19T14:59:10.043054+0000","last_became_active":"2024-02-19T14:59:07.644999+0000","last_became_peered":"2024-02-19T14:59:07.644999+0000","last_unstale":"2024-02-19T14:59:10.043054+0000","last_undegraded":"2024-02-19T14:59:10.043054+0000","last_fullsized":"2024-02-19T14:59:10.043054+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:29:28.943355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042367+0000","last_change":"2024-02-19T14:59:08.669281+0000","last_active":"2024-02-19T14:59:10.042367+0000","last_peered":"2024-02-19T14:59:10.042367+0000","last_clean":"2024-02-19T14:59:10.042367+0000","last_became_active":"2024-02-19T14:59:08.668236+0000","last_became_peered":"2024-02-19T14:59:08.668236+0000","last_unstale":"2024-02-19T14:59:10.042367+0000","last_undegraded":"2024-02-19T14:59:10.042367+0000","last_fullsized":"2024-02-19T14:59:10.042367+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:01:59.195475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"187'2","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043019+0000","last_change":"2024-02-19T14:59:07.646936+0000","last_active":"2024-02-19T14:59:10.043019+0000","last_peered":"2024-02-19T14:59:10.043019+0000","last_clean":"2024-02-19T14:59:10.043019+0000","last_became_active":"2024-02-19T14:59:07.645710+0000","last_became_peered":"2024-02-19T14:59:07.645710+0000","last_unstale":"2024-02-19T14:59:10.043019+0000","last_undegraded":"2024-02-19T14:59:10.043019+0000","last_fullsized":"2024-02-19T14:59:10.043019+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:57.317171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042627+0000","last_change":"2024-02-19T14:59:08.664552+0000","last_active":"2024-02-19T14:59:10.042627+0000","last_peered":"2024-02-19T14:59:10.042627+0000","last_clean":"2024-02-19T14:59:10.042627+0000","last_became_active":"2024-02-19T14:59:08.664272+0000","last_became_peered":"2024-02-19T14:59:08.664272+0000","last_unstale":"2024-02-19T14:59:10.042627+0000","last_undegraded":"2024-02-19T14:59:10.042627+0000","last_fullsized":"2024-02-19T14:59:10.042627+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:39:11.947832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658198+0000","last_change":"2024-02-19T14:59:07.636425+0000","last_active":"2024-02-19T14:59:09.658198+0000","last_peered":"2024-02-19T14:59:09.658198+0000","last_clean":"2024-02-19T14:59:09.658198+0000","last_became_active":"2024-02-19T14:59:07.636154+0000","last_became_peered":"2024-02-19T14:59:07.636154+0000","last_unstale":"2024-02-19T14:59:09.658198+0000","last_undegraded":"2024-02-19T14:59:09.658198+0000","last_fullsized":"2024-02-19T14:59:09.658198+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:48:29.869334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042502+0000","last_change":"2024-02-19T14:59:08.667958+0000","last_active":"2024-02-19T14:59:10.042502+0000","last_peered":"2024-02-19T14:59:10.042502+0000","last_clean":"2024-02-19T14:59:10.042502+0000","last_became_active":"2024-02-19T14:59:08.667585+0000","last_became_peered":"2024-02-19T14:59:08.667585+0000","last_unstale":"2024-02-19T14:59:10.042502+0000","last_undegraded":"2024-02-19T14:59:10.042502+0000","last_fullsized":"2024-02-19T14:59:10.042502+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:23.474639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658047+0000","last_change":"2024-02-19T14:59:07.638740+0000","last_active":"2024-02-19T14:59:09.658047+0000","last_peered":"2024-02-19T14:59:09.658047+0000","last_clean":"2024-02-19T14:59:09.658047+0000","last_became_active":"2024-02-19T14:59:07.638275+0000","last_became_peered":"2024-02-19T14:59:07.638275+0000","last_unstale":"2024-02-19T14:59:09.658047+0000","last_undegraded":"2024-02-19T14:59:09.658047+0000","last_fullsized":"2024-02-19T14:59:09.658047+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:06:05.299812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655641+0000","last_change":"2024-02-19T14:59:08.670245+0000","last_active":"2024-02-19T14:59:09.655641+0000","last_peered":"2024-02-19T14:59:09.655641+0000","last_clean":"2024-02-19T14:59:09.655641+0000","last_became_active":"2024-02-19T14:59:08.669961+0000","last_became_peered":"2024-02-19T14:59:08.669961+0000","last_unstale":"2024-02-19T14:59:09.655641+0000","last_undegraded":"2024-02-19T14:59:09.655641+0000","last_fullsized":"2024-02-19T14:59:09.655641+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:21:28.687755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042293+0000","last_change":"2024-02-19T14:59:07.641182+0000","last_active":"2024-02-19T14:59:10.042293+0000","last_peered":"2024-02-19T14:59:10.042293+0000","last_clean":"2024-02-19T14:59:10.042293+0000","last_became_active":"2024-02-19T14:59:07.640867+0000","last_became_peered":"2024-02-19T14:59:07.640867+0000","last_unstale":"2024-02-19T14:59:10.042293+0000","last_undegraded":"2024-02-19T14:59:10.042293+0000","last_fullsized":"2024-02-19T14:59:10.042293+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:14:31.036618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042593+0000","last_change":"2024-02-19T14:59:08.656306+0000","last_active":"2024-02-19T14:59:10.042593+0000","last_peered":"2024-02-19T14:59:10.042593+0000","last_clean":"2024-02-19T14:59:10.042593+0000","last_became_active":"2024-02-19T14:59:08.656042+0000","last_became_peered":"2024-02-19T14:59:08.656042+0000","last_unstale":"2024-02-19T14:59:10.042593+0000","last_undegraded":"2024-02-19T14:59:10.042593+0000","last_fullsized":"2024-02-19T14:59:10.042593+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:30:41.458834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"187'2","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043509+0000","last_change":"2024-02-19T14:59:07.640656+0000","last_active":"2024-02-19T14:59:10.043509+0000","last_peered":"2024-02-19T14:59:10.043509+0000","last_clean":"2024-02-19T14:59:10.043509+0000","last_became_active":"2024-02-19T14:59:07.640382+0000","last_became_peered":"2024-02-19T14:59:07.640382+0000","last_unstale":"2024-02-19T14:59:10.043509+0000","last_undegraded":"2024-02-19T14:59:10.043509+0000","last_fullsized":"2024-02-19T14:59:10.043509+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:22:41.660255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043238+0000","last_change":"2024-02-19T14:59:08.669233+0000","last_active":"2024-02-19T14:59:10.043238+0000","last_peered":"2024-02-19T14:59:10.043238+0000","last_clean":"2024-02-19T14:59:10.043238+0000","last_became_active":"2024-02-19T14:59:08.668960+0000","last_became_peered":"2024-02-19T14:59:08.668960+0000","last_unstale":"2024-02-19T14:59:10.043238+0000","last_undegraded":"2024-02-19T14:59:10.043238+0000","last_fullsized":"2024-02-19T14:59:10.043238+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:39:42.581187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"188'1","reported_seq":15,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658407+0000","last_change":"2024-02-19T14:59:07.638470+0000","last_active":"2024-02-19T14:59:09.658407+0000","last_peered":"2024-02-19T14:59:09.658407+0000","last_clean":"2024-02-19T14:59:09.658407+0000","last_became_active":"2024-02-19T14:59:07.638197+0000","last_became_peered":"2024-02-19T14:59:07.638197+0000","last_unstale":"2024-02-19T14:59:09.658407+0000","last_undegraded":"2024-02-19T14:59:09.658407+0000","last_fullsized":"2024-02-19T14:59:09.658407+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:36:43.081410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.659608+0000","last_change":"2024-02-19T14:59:08.664144+0000","last_active":"2024-02-19T14:59:09.659608+0000","last_peered":"2024-02-19T14:59:09.659608+0000","last_clean":"2024-02-19T14:59:09.659608+0000","last_became_active":"2024-02-19T14:59:08.662138+0000","last_became_peered":"2024-02-19T14:59:08.662138+0000","last_unstale":"2024-02-19T14:59:09.659608+0000","last_undegraded":"2024-02-19T14:59:09.659608+0000","last_fullsized":"2024-02-19T14:59:09.659608+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:22:32.957039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658309+0000","last_change":"2024-02-19T14:59:07.640562+0000","last_active":"2024-02-19T14:59:09.658309+0000","last_peered":"2024-02-19T14:59:09.658309+0000","last_clean":"2024-02-19T14:59:09.658309+0000","last_became_active":"2024-02-19T14:59:07.640288+0000","last_became_peered":"2024-02-19T14:59:07.640288+0000","last_unstale":"2024-02-19T14:59:09.658309+0000","last_undegraded":"2024-02-19T14:59:09.658309+0000","last_fullsized":"2024-02-19T14:59:09.658309+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:22:31.952215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042844+0000","last_change":"2024-02-19T14:59:08.663046+0000","last_active":"2024-02-19T14:59:10.042844+0000","last_peered":"2024-02-19T14:59:10.042844+0000","last_clean":"2024-02-19T14:59:10.042844+0000","last_became_active":"2024-02-19T14:59:08.662781+0000","last_became_peered":"2024-02-19T14:59:08.662781+0000","last_unstale":"2024-02-19T14:59:10.042844+0000","last_undegraded":"2024-02-19T14:59:10.042844+0000","last_fullsized":"2024-02-19T14:59:10.042844+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:16:38.881316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043139+0000","last_change":"2024-02-19T14:59:07.645290+0000","last_active":"2024-02-19T14:59:10.043139+0000","last_peered":"2024-02-19T14:59:10.043139+0000","last_clean":"2024-02-19T14:59:10.043139+0000","last_became_active":"2024-02-19T14:59:07.644933+0000","last_became_peered":"2024-02-19T14:59:07.644933+0000","last_unstale":"2024-02-19T14:59:10.043139+0000","last_undegraded":"2024-02-19T14:59:10.043139+0000","last_fullsized":"2024-02-19T14:59:10.043139+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:09:08.578967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655353+0000","last_change":"2024-02-19T14:59:08.669470+0000","last_active":"2024-02-19T14:59:09.655353+0000","last_peered":"2024-02-19T14:59:09.655353+0000","last_clean":"2024-02-19T14:59:09.655353+0000","last_became_active":"2024-02-19T14:59:08.669225+0000","last_became_peered":"2024-02-19T14:59:08.669225+0000","last_unstale":"2024-02-19T14:59:09.655353+0000","last_undegraded":"2024-02-19T14:59:09.655353+0000","last_fullsized":"2024-02-19T14:59:09.655353+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:13:44.516165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.043113+0000","last_change":"2024-02-19T14:59:07.643990+0000","last_active":"2024-02-19T14:59:10.043113+0000","last_peered":"2024-02-19T14:59:10.043113+0000","last_clean":"2024-02-19T14:59:10.043113+0000","last_became_active":"2024-02-19T14:59:07.643708+0000","last_became_peered":"2024-02-19T14:59:07.643708+0000","last_unstale":"2024-02-19T14:59:10.043113+0000","last_undegraded":"2024-02-19T14:59:10.043113+0000","last_fullsized":"2024-02-19T14:59:10.043113+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:10:30.048428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655419+0000","last_change":"2024-02-19T14:59:08.648447+0000","last_active":"2024-02-19T14:59:09.655419+0000","last_peered":"2024-02-19T14:59:09.655419+0000","last_clean":"2024-02-19T14:59:09.655419+0000","last_became_active":"2024-02-19T14:59:08.648141+0000","last_became_peered":"2024-02-19T14:59:08.648141+0000","last_unstale":"2024-02-19T14:59:09.655419+0000","last_undegraded":"2024-02-19T14:59:09.655419+0000","last_fullsized":"2024-02-19T14:59:09.655419+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:39:22.907999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.658338+0000","last_change":"2024-02-19T14:59:07.634059+0000","last_active":"2024-02-19T14:59:09.658338+0000","last_peered":"2024-02-19T14:59:09.658338+0000","last_clean":"2024-02-19T14:59:09.658338+0000","last_became_active":"2024-02-19T14:59:07.633768+0000","last_became_peered":"2024-02-19T14:59:07.633768+0000","last_unstale":"2024-02-19T14:59:09.658338+0000","last_undegraded":"2024-02-19T14:59:09.658338+0000","last_fullsized":"2024-02-19T14:59:09.658338+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:55:11.101309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042608+0000","last_change":"2024-02-19T14:59:08.652102+0000","last_active":"2024-02-19T14:59:10.042608+0000","last_peered":"2024-02-19T14:59:10.042608+0000","last_clean":"2024-02-19T14:59:10.042608+0000","last_became_active":"2024-02-19T14:59:08.651812+0000","last_became_peered":"2024-02-19T14:59:08.651812+0000","last_unstale":"2024-02-19T14:59:10.042608+0000","last_undegraded":"2024-02-19T14:59:10.042608+0000","last_fullsized":"2024-02-19T14:59:10.042608+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:05.351605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042714+0000","last_change":"2024-02-19T14:59:07.632792+0000","last_active":"2024-02-19T14:59:10.042714+0000","last_peered":"2024-02-19T14:59:10.042714+0000","last_clean":"2024-02-19T14:59:10.042714+0000","last_became_active":"2024-02-19T14:59:07.632529+0000","last_became_peered":"2024-02-19T14:59:07.632529+0000","last_unstale":"2024-02-19T14:59:10.042714+0000","last_undegraded":"2024-02-19T14:59:10.042714+0000","last_fullsized":"2024-02-19T14:59:10.042714+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:55:20.506363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042760+0000","last_change":"2024-02-19T14:59:08.667991+0000","last_active":"2024-02-19T14:59:10.042760+0000","last_peered":"2024-02-19T14:59:10.042760+0000","last_clean":"2024-02-19T14:59:10.042760+0000","last_became_active":"2024-02-19T14:59:08.667579+0000","last_became_peered":"2024-02-19T14:59:08.667579+0000","last_unstale":"2024-02-19T14:59:10.042760+0000","last_undegraded":"2024-02-19T14:59:10.042760+0000","last_fullsized":"2024-02-19T14:59:10.042760+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:08:00.237847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042422+0000","last_change":"2024-02-19T14:59:07.643336+0000","last_active":"2024-02-19T14:59:10.042422+0000","last_peered":"2024-02-19T14:59:10.042422+0000","last_clean":"2024-02-19T14:59:10.042422+0000","last_became_active":"2024-02-19T14:59:07.642778+0000","last_became_peered":"2024-02-19T14:59:07.642778+0000","last_unstale":"2024-02-19T14:59:10.042422+0000","last_undegraded":"2024-02-19T14:59:10.042422+0000","last_fullsized":"2024-02-19T14:59:10.042422+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:44:39.391521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:09.655566+0000","last_change":"2024-02-19T14:59:08.668162+0000","last_active":"2024-02-19T14:59:09.655566+0000","last_peered":"2024-02-19T14:59:09.655566+0000","last_clean":"2024-02-19T14:59:09.655566+0000","last_became_active":"2024-02-19T14:59:08.667892+0000","last_became_peered":"2024-02-19T14:59:08.667892+0000","last_unstale":"2024-02-19T14:59:09.655566+0000","last_undegraded":"2024-02-19T14:59:09.655566+0000","last_fullsized":"2024-02-19T14:59:09.655566+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:07.620339+0000","last_clean_scrub_stamp":"2024-02-19T14:59:07.620339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:45:38.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"187'2","reported_seq":16,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042963+0000","last_change":"2024-02-19T14:59:07.636409+0000","last_active":"2024-02-19T14:59:10.042963+0000","last_peered":"2024-02-19T14:59:10.042963+0000","last_clean":"2024-02-19T14:59:10.042963+0000","last_became_active":"2024-02-19T14:59:07.636136+0000","last_became_peered":"2024-02-19T14:59:07.636136+0000","last_unstale":"2024-02-19T14:59:10.042963+0000","last_undegraded":"2024-02-19T14:59:10.042963+0000","last_fullsized":"2024-02-19T14:59:10.042963+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:06.615515+0000","last_clean_scrub_stamp":"2024-02-19T14:59:06.615515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:22:31.952215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":484,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-02-19T14:59:10.042574+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:10.042574+0000","last_peered":"2024-02-19T14:59:10.042574+0000","last_clean":"2024-02-19T14:59:10.042574+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:10.042574+0000","last_undegraded":"2024-02-19T14:59:10.042574+0000","last_fullsized":"2024-02-19T14:59:10.042574+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34100,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337740,"statfs":{"total":96636764160,"available":96601845760,"internally_reserved":0,"allocated":2981888,"data_stored":1756169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.505,"15min":0.495},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51600000000000001,"15min":0.47499999999999998},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.63600000000000001,"5min":0.748,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.35299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69299999999999995,"5min":0.76200000000000001,"15min":0.76400000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.627,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65900000000000003}]},{"osd":3,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.629}]},{"osd":4,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46200000000000002,"15min":0.44900000000000001},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.753,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.49099999999999999,"15min":0.496},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.499,"15min":0.53000000000000003},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.75,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.79000000000000004,"5min":0.871,"15min":0.871},"last":0.498}]},{"osd":6,"last update":"Mon Feb 19 14:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56599999999999995,"15min":0.58999999999999997},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.84399999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.55700000000000005,"15min":0.58299999999999996},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574940,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33536,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338304,"statfs":{"total":96636764160,"available":96602423296,"internally_reserved":0,"allocated":2404352,"data_stored":1166867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.376},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.38500000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.29999999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.442}]}]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33528,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338312,"statfs":{"total":96636764160,"available":96602431488,"internally_reserved":0,"allocated":2396160,"data_stored":1166811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.68000000000000005}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.375}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.498},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.746}]}]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33516,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338324,"statfs":{"total":96636764160,"available":96602443776,"internally_reserved":0,"allocated":2383872,"data_stored":1165219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.504},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.54600000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33516,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338324,"statfs":{"total":96636764160,"available":96602443776,"internally_reserved":0,"allocated":2383872,"data_stored":1165151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34799999999999998,"15min":0.313},"min":{"1min":0.217,"5min":0.159,"15min":0.159},"max":{"1min":0.45100000000000001,"5min":3.694,"15min":3.694},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36799999999999999,"15min":0.32600000000000001},"min":{"1min":0.248,"5min":0.192,"15min":0.189},"max":{"1min":0.55300000000000005,"5min":4.835,"15min":4.835},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.41699999999999998,"15min":0.378},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.498,"5min":6.3810000000000002,"15min":6.3810000000000002},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41399999999999998,"15min":0.38700000000000001},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":3.6320000000000001,"15min":3.6320000000000001},"last":0.312},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.40000000000000002,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":4.9889999999999999,"15min":4.9889999999999999},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55100000000000005,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76200000000000001,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.39600000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.745,"5min":4.9539999999999997,"15min":4.9539999999999997},"last":0.35899999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":3.6669999999999998,"15min":3.6669999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.60999999999999999},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79700000000000004,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68300000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61599999999999999,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78700000000000003,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":0.47399999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.41999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82999999999999996,"5min":4.8799999999999999,"15min":4.8799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.63300000000000001},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.72099999999999997,"5min":3.0760000000000001,"15min":3.0760000000000001},"last":0.44}]}]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33520,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338320,"statfs":{"total":96636764160,"available":96602439680,"internally_reserved":0,"allocated":2387968,"data_stored":1165745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.255},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.35499999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.316},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.434}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.41199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33512,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338328,"statfs":{"total":96636764160,"available":96602447872,"internally_reserved":0,"allocated":2379776,"data_stored":1165129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.32600000000000001},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.53500000000000003,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32300000000000001,"15min":0.33200000000000002},"min":{"1min":0.24299999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.90400000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.309,"15min":0.309},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.45600000000000002,"5min":0.53500000000000003,"15min":0.77400000000000002},"last":0.253},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.33900000000000002,"15min":0.35799999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46899999999999997,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.29399999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.435},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.36899999999999999,"15min":0.38500000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.42199999999999999,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.41599999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.54900000000000004},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.309},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.96799999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53800000000000003,"15min":0.55600000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":1.0009999999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.56000000000000005},"min":{"1min":0.44800000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.373,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.65600000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56000000000000005,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54800000000000004,"15min":0.56499999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.82399999999999995,"15min":0.996},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.57699999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":1.036},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57499999999999996,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.68799999999999994}]}]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34084,"kb_used_data":2904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337756,"statfs":{"total":96636764160,"available":96601862144,"internally_reserved":0,"allocated":2973696,"data_stored":1755497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.251},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.35799999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.40300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.499}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51900000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:16.327 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:59:16.328 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:59:16.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.468+0000 7ffbaed47640 1 Processor -- start 2024-02-19T14:59:16.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.468+0000 7ffbaed47640 1 -- start start 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffbaed47640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffba8152830 0x7ffba80c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffbaed47640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffba80c5760 0x7ffba80c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffbaed47640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba81586a0 con 0x7ffba80c77c0 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffbaed47640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba80c6550 con 0x7ffba80c5760 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffbaed47640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba8159320 con 0x7ffba8152830 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffba7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffba80c5760 0x7ffba80c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffba7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffba80c5760 0x7ffba80c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33074/0 (socket says 172.21.15.78:33074) 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffba7fff640 1 -- 172.21.15.78:0/2376293467 learned_addr learned my addr 172.21.15.78:0/2376293467 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffbacae3640 1 --2- 172.21.15.78:0/2376293467 >> v2:172.21.15.150:3300/0 conn(0x7ffba8152830 0x7ffba80c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffba7fff640 1 -- 172.21.15.78:0/2376293467 >> v2:172.21.15.150:3300/0 conn(0x7ffba8152830 msgr2=0x7ffba80c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.469+0000 7ffba7fff640 1 --2- 172.21.15.78:0/2376293467 >> v2:172.21.15.150:3300/0 conn(0x7ffba8152830 0x7ffba80c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.470+0000 7ffba7fff640 1 -- 172.21.15.78:0/2376293467 >> v1:172.21.15.150:6789/0 conn(0x7ffba80c77c0 legacy=0x7ffba806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.470+0000 7ffba7fff640 1 -- 172.21.15.78:0/2376293467 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8158980 con 0x7ffba80c5760 2024-02-19T14:59:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.470+0000 7ffba7fff640 1 --2- 172.21.15.78:0/2376293467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c5760 0x7ffba80c7280 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7ffba80c7250 tx=0x7ffb9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968e5a1e5d49222 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.470+0000 7ffba77fe640 1 -- 172.21.15.78:0/2376293467 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c031870 con 0x7ffba80c5760 2024-02-19T14:59:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.470+0000 7ffba77fe640 1 -- 172.21.15.78:0/2376293467 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c0319d0 con 0x7ffba80c5760 2024-02-19T14:59:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.470+0000 7ffba77fe640 1 -- 172.21.15.78:0/2376293467 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c031d60 con 0x7ffba80c5760 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 -- 172.21.15.78:0/2376293467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c5760 msgr2=0x7ffba80c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 --2- 172.21.15.78:0/2376293467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c5760 0x7ffba80c7280 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7ffba80c7250 tx=0x7ffb9c02b910 comp rx=0 tx=0).stop 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 -- 172.21.15.78:0/2376293467 shutdown_connections 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 --2- 172.21.15.78:0/2376293467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba80c5760 0x7ffba80c7280 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 --2- 172.21.15.78:0/2376293467 >> v2:172.21.15.150:3300/0 conn(0x7ffba8152830 0x7ffba80c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 -- 172.21.15.78:0/2376293467 >> 172.21.15.78:0/2376293467 conn(0x7ffba80ca8a0 msgr2=0x7ffba80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 -- 172.21.15.78:0/2376293467 shutdown_connections 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.471+0000 7ffbaed47640 1 -- 172.21.15.78:0/2376293467 wait complete. 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.472+0000 7ffbaed47640 1 Processor -- start 2024-02-19T14:59:16.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.472+0000 7ffbaed47640 1 -- start start 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbaed47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba80c5760 0x7ffba80c1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbaed47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c77c0 0x7ffba80c0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbaed47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba8152830 0x7ffba80c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbaed47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80c0fc0 con 0x7ffba8152830 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbaed47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80aab30 con 0x7ffba80c5760 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbaed47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80c3e30 con 0x7ffba80c77c0 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbad2e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba8152830 0x7ffba80c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbacae3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba80c5760 0x7ffba80c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbad2e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba8152830 0x7ffba80c0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33080/0 (socket says 172.21.15.78:33080) 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbad2e4640 1 -- 172.21.15.78:0/3216011993 learned_addr learned my addr 172.21.15.78:0/3216011993 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffba7fff640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c77c0 0x7ffba80c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbacae3640 1 -- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c77c0 msgr2=0x7ffba80c0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbacae3640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c77c0 0x7ffba80c0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbacae3640 1 -- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba8152830 msgr2=0x7ffba80c0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbacae3640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba8152830 0x7ffba80c0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.473+0000 7ffbacae3640 1 -- 172.21.15.78:0/3216011993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c0094d0 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.474+0000 7ffbacae3640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba80c5760 0x7ffba80c1970 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800e8a0 tx=0x7ffb9800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.474+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98009e60 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.474+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9800c940 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.474+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9800ccd0 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.474+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba80c4110 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.474+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba80aa7c0 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.475+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba8074990 con 0x7ffba80c5760 2024-02-19T14:59:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.475+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffb98013470 con 0x7ffba80c5760 2024-02-19T14:59:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.475+0000 7ffba57fa640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb9406b810 0x7ffb9406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.476+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb9808a2f0 con 0x7ffba80c5760 2024-02-19T14:59:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.476+0000 7ffba7fff640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb9406b810 0x7ffb9406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.476+0000 7ffba7fff640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb9406b810 0x7ffb9406dcd0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c036000 tx=0x7ffb9c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.480+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffb98054c20 con 0x7ffba80c5760 2024-02-19T14:59:16.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.722+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ffba806dae0 con 0x7ffba80c5760 2024-02-19T14:59:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.728+0000 7ffba57fa640 1 -- 172.21.15.78:0/3216011993 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7ffb980582e0 con 0x7ffba80c5760 2024-02-19T14:59:16.731 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:59:16.731 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:59:16.731 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:59:16.731 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:59:16.731 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:59:16.732 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:59:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb9406b810 msgr2=0x7ffb9406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb9406b810 0x7ffb9406dcd0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c036000 tx=0x7ffb9c03c040 comp rx=0 tx=0).stop 2024-02-19T14:59:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba80c5760 msgr2=0x7ffba80c1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba80c5760 0x7ffba80c1970 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800e8a0 tx=0x7ffb9800ed60 comp rx=0 tx=0).stop 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 shutdown_connections 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb9406b810 0x7ffb9406dcd0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffba8152830 0x7ffba80c0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffba80c77c0 0x7ffba80c0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 --2- 172.21.15.78:0/3216011993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba80c5760 0x7ffba80c1970 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.730+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 >> 172.21.15.78:0/3216011993 conn(0x7ffba80ca8a0 msgr2=0x7ffba806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.731+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 shutdown_connections 2024-02-19T14:59:16.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.731+0000 7ffbaed47640 1 -- 172.21.15.78:0/3216011993 wait complete. 2024-02-19T14:59:16.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:16.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.841+0000 7f1bab247640 1 Processor -- start 2024-02-19T14:59:16.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.841+0000 7f1bab247640 1 -- start start 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1bab247640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1ba414cd50 0x7f1ba40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1bab247640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1ba41537a0 0x7f1ba40c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1bab247640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba41563b0 con 0x7f1ba40c7b70 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1bab247640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba4153cd0 con 0x7f1ba41537a0 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1bab247640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba406b6e0 con 0x7f1ba414cd50 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1ba3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1ba41537a0 0x7f1ba40c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1ba3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1ba41537a0 0x7f1ba40c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33092/0 (socket says 172.21.15.78:33092) 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1ba3fff640 1 -- 172.21.15.78:0/2564839235 learned_addr learned my addr 172.21.15.78:0/2564839235 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.842+0000 7f1ba8fe3640 1 --2- 172.21.15.78:0/2564839235 >> v2:172.21.15.150:3300/0 conn(0x7f1ba414cd50 0x7f1ba40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.843+0000 7f1ba3fff640 1 -- 172.21.15.78:0/2564839235 >> v2:172.21.15.150:3300/0 conn(0x7f1ba414cd50 msgr2=0x7f1ba40c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.843+0000 7f1ba3fff640 1 --2- 172.21.15.78:0/2564839235 >> v2:172.21.15.150:3300/0 conn(0x7f1ba414cd50 0x7f1ba40c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.843+0000 7f1ba3fff640 1 -- 172.21.15.78:0/2564839235 >> v1:172.21.15.150:6789/0 conn(0x7f1ba40c7b70 legacy=0x7f1ba4156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.843+0000 7f1ba3fff640 1 -- 172.21.15.78:0/2564839235 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba406b980 con 0x7f1ba41537a0 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.843+0000 7f1ba37fe640 1 -- 172.21.15.78:0/2564839235 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134977531 0 0) 0x7f1ba41563b0 con 0x7f1ba40c7b70 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.843+0000 7f1ba3fff640 1 --2- 172.21.15.78:0/2564839235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba41537a0 0x7f1ba40c7630 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ace20 tx=0x7f1b9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7d5044864fa492 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1ba37fe640 1 -- 172.21.15.78:0/2564839235 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b980317e0 con 0x7f1ba41537a0 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1ba37fe640 1 -- 172.21.15.78:0/2564839235 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98031940 con 0x7f1ba41537a0 2024-02-19T14:59:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1ba37fe640 1 -- 172.21.15.78:0/2564839235 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98031cd0 con 0x7f1ba41537a0 2024-02-19T14:59:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 -- 172.21.15.78:0/2564839235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba41537a0 msgr2=0x7f1ba40c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 --2- 172.21.15.78:0/2564839235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba41537a0 0x7f1ba40c7630 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ace20 tx=0x7f1b9802b880 comp rx=0 tx=0).stop 2024-02-19T14:59:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 -- 172.21.15.78:0/2564839235 shutdown_connections 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 --2- 172.21.15.78:0/2564839235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba41537a0 0x7f1ba40c7630 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 --2- 172.21.15.78:0/2564839235 >> v2:172.21.15.150:3300/0 conn(0x7f1ba414cd50 0x7f1ba40c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 -- 172.21.15.78:0/2564839235 >> 172.21.15.78:0/2564839235 conn(0x7f1ba4066e70 msgr2=0x7f1ba40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 -- 172.21.15.78:0/2564839235 shutdown_connections 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.844+0000 7f1bab247640 1 -- 172.21.15.78:0/2564839235 wait complete. 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.845+0000 7f1bab247640 1 Processor -- start 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.845+0000 7f1bab247640 1 -- start start 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1bab247640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 0x7f1ba40c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1bab247640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ba414cd50 0x7f1ba414a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1bab247640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ba41537a0 0x7f1ba414a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1bab247640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41e7750 con 0x7f1ba40c7b70 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1bab247640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41508d0 con 0x7f1ba414cd50 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1bab247640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4149160 con 0x7f1ba41537a0 2024-02-19T14:59:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1ba8fe3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 0x7f1ba40c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1ba8fe3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 0x7f1ba40c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33104/0 (socket says 172.21.15.78:33104) 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1ba8fe3640 1 -- 172.21.15.78:0/2687361661 learned_addr learned my addr 172.21.15.78:0/2687361661 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1ba3fff640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ba414cd50 0x7f1ba414a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.846+0000 7f1ba97e4640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ba41537a0 0x7f1ba414a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba8fe3640 1 -- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ba41537a0 msgr2=0x7f1ba414a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba8fe3640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ba41537a0 0x7f1ba414a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba8fe3640 1 -- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ba414cd50 msgr2=0x7f1ba414a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba8fe3640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ba414cd50 0x7f1ba414a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba8fe3640 1 -- 172.21.15.78:0/2687361661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b980094d0 con 0x7f1ba40c7b70 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba8fe3640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 0x7f1ba40c6410 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f1b94012ab0 tx=0x7f1b94012f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9400d990 con 0x7f1ba40c7b70 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b94013040 con 0x7f1ba40c7b70 2024-02-19T14:59:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba4148ba0 con 0x7f1ba40c7b70 2024-02-19T14:59:16.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94010cc0 con 0x7f1ba40c7b70 2024-02-19T14:59:16.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.847+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba4149b80 con 0x7f1ba40c7b70 2024-02-19T14:59:16.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.848+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba4074990 con 0x7f1ba40c7b70 2024-02-19T14:59:16.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.848+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1b94018480 con 0x7f1ba40c7b70 2024-02-19T14:59:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.849+0000 7f1ba17fa640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b9006b810 0x7f1b9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.849+0000 7f1ba3fff640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b9006b810 0x7f1b9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.849+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b9408e270 con 0x7f1ba40c7b70 2024-02-19T14:59:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.849+0000 7f1ba3fff640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b9006b810 0x7f1b9006dcd0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1b98036000 tx=0x7f1b9803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:16.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:16.853+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b94058bc0 con 0x7f1ba40c7b70 2024-02-19T14:59:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.061+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba40c4230 con 0x7f1ba40c7b70 2024-02-19T14:59:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.062+0000 7f1ba17fa640 1 -- 172.21.15.78:0/2687361661 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4951 (secure 0 0 0) 0x7f1b9405c280 con 0x7f1ba40c7b70 2024-02-19T14:59:17.065 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:17.065 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":165,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:09.985985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 165 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b9006b810 msgr2=0x7f1b9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b9006b810 0x7f1b9006dcd0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1b98036000 tx=0x7f1b9803d040 comp rx=0 tx=0).stop 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 msgr2=0x7f1ba40c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 0x7f1ba40c6410 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f1b94012ab0 tx=0x7f1b94012f70 comp rx=0 tx=0).stop 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 shutdown_connections 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1b9006b810 0x7f1b9006dcd0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ba41537a0 0x7f1ba414a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ba414cd50 0x7f1ba414a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 --2- 172.21.15.78:0/2687361661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba40c7b70 0x7f1ba40c6410 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 >> 172.21.15.78:0/2687361661 conn(0x7f1ba4066e70 msgr2=0x7f1ba4155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 shutdown_connections 2024-02-19T14:59:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.064+0000 7f1bab247640 1 -- 172.21.15.78:0/2687361661 wait complete. 2024-02-19T14:59:17.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:09.985985+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:17.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T14:59:17.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:59:17.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.579+0000 7ff9d56c9640 1 Processor -- start 2024-02-19T14:59:17.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.579+0000 7ff9d56c9640 1 -- start start 2024-02-19T14:59:17.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9d56c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d0152700 0x7ff9d00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9d56c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9d00c8e00 0x7ff9d015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9d56c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d01551f0 con 0x7ff9d00c5730 2024-02-19T14:59:17.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9d56c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d0155490 con 0x7ff9d00c8e00 2024-02-19T14:59:17.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9d56c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0155770 con 0x7ff9d0152700 2024-02-19T14:59:17.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cf7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9d00c8e00 0x7ff9d015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9ceffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d0152700 0x7ff9d00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9ce7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff9d00c5730 0x7ff9d006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58562/0 (socket says 172.21.15.78:58562) 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/2840921074 learned_addr learned my addr 172.21.15.78:0/2840921074 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cdffb640 1 -- 172.21.15.78:0/2840921074 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181308258 0 0) 0x7ff9d01551f0 con 0x7ff9d00c5730 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cdffb640 1 -- 172.21.15.78:0/2840921074 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b8003520 con 0x7ff9d00c5730 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d0152700 msgr2=0x7ff9d00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d0152700 0x7ff9d00c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2840921074 >> v1:172.21.15.150:6789/0 conn(0x7ff9d00c5730 legacy=0x7ff9d006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2840921074 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0155a50 con 0x7ff9d00c8e00 2024-02-19T14:59:17.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.580+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d0152700 0x7ff9d00c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.581+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d00c8e00 0x7ff9d015f980 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47d084eef64e0a59 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.581+0000 7ff9cdffb640 1 -- 172.21.15.78:0/2840921074 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8031830 con 0x7ff9d00c8e00 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.581+0000 7ff9cdffb640 1 -- 172.21.15.78:0/2840921074 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8031990 con 0x7ff9d00c8e00 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.581+0000 7ff9cdffb640 1 -- 172.21.15.78:0/2840921074 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8031d20 con 0x7ff9d00c8e00 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 -- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d00c8e00 msgr2=0x7ff9d015f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d00c8e00 0x7ff9d015f980 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 -- 172.21.15.78:0/2840921074 shutdown_connections 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d00c8e00 0x7ff9d015f980 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/2840921074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d0152700 0x7ff9d00c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 -- 172.21.15.78:0/2840921074 >> 172.21.15.78:0/2840921074 conn(0x7ff9d00cac90 msgr2=0x7ff9d00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 -- 172.21.15.78:0/2840921074 shutdown_connections 2024-02-19T14:59:17.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.582+0000 7ff9d56c9640 1 -- 172.21.15.78:0/2840921074 wait complete. 2024-02-19T14:59:17.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.583+0000 7ff9d56c9640 1 Processor -- start 2024-02-19T14:59:17.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.583+0000 7ff9d56c9640 1 -- start start 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 0x7ff9d01ea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d00c8e00 0x7ff9d00aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d0152700 0x7ff9d00af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00aef70 con 0x7ff9d00c5730 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00ad990 con 0x7ff9d0152700 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00adc70 con 0x7ff9d00c8e00 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 0x7ff9d01ea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 0x7ff9d01ea4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51020/0 (socket says 172.21.15.78:51020) 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 -- 172.21.15.78:0/224379750 learned_addr learned my addr 172.21.15.78:0/224379750 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:17.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d00c8e00 0x7ff9d00aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d0152700 0x7ff9d00af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 -- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d00c8e00 msgr2=0x7ff9d00aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d00c8e00 0x7ff9d00aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 -- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d0152700 msgr2=0x7ff9d00af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d0152700 0x7ff9d00af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 -- 172.21.15.78:0/224379750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0009a70 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 0x7ff9d01ea4c0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000efc0 tx=0x7ff9c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9b77fe640 1 -- 172.21.15.78:0/224379750 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0014680 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9b77fe640 1 -- 172.21.15.78:0/224379750 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c000f040 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9b77fe640 1 -- 172.21.15.78:0/224379750 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0014a00 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.584+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8009400 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.585+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d00ae230 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.586+0000 7ff9b77fe640 1 -- 172.21.15.78:0/224379750 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff9c0014b60 con 0x7ff9d00c5730 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.586+0000 7ff9b77fe640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.586+0000 7ff9b77fe640 1 -- 172.21.15.78:0/224379750 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9c008b150 con 0x7ff9d00c5730 2024-02-19T14:59:17.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.586+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.587+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00c1590 tx=0x7ff9c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.587+0000 7ff9d8377740 1 Processor -- start 2024-02-19T14:59:17.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.587+0000 7ff9d8377740 1 -- start start 2024-02-19T14:59:17.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 0x560da0b97b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b9a8b0 0x560da0b5c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9d8377740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0b17cd0 con 0x560da0b5dae0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0acaf20 con 0x560da0b9a8b0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da0ac8450 con 0x560da0b97740 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 0x560da0b97b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 0x560da0b97b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51022/0 (socket says 172.21.15.78:51022) 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/4152871981 learned_addr learned my addr 172.21.15.78:0/4152871981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b9a8b0 0x560da0b5c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 msgr2=0x560da0b988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b9a8b0 msgr2=0x560da0b5c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b9a8b0 0x560da0b5c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.588+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/4152871981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560da0a78f20 con 0x560da0b5dae0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 0x560da0b97b20 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400f930 tx=0x7ff9c4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40a02f18a6a2a5b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/4152871981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400feb0 con 0x560da0b5dae0 2024-02-19T14:59:17.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/4152871981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4015bf0 con 0x560da0b5dae0 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9b4ff9640 1 -- 172.21.15.78:0/4152871981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4020400 con 0x560da0b5dae0 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 -- 172.21.15.78:0/4152871981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 msgr2=0x560da0b97b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 0x560da0b97b20 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400f930 tx=0x7ff9c4000ed0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 -- 172.21.15.78:0/4152871981 shutdown_connections 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b5dae0 0x560da0b97b20 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b9a8b0 0x560da0b5c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 --2- 172.21.15.78:0/4152871981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.589+0000 7ff9d8377740 1 -- 172.21.15.78:0/4152871981 >> 172.21.15.78:0/4152871981 conn(0x560da0b59120 msgr2=0x560da0b7db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.590+0000 7ff9d8377740 1 -- 172.21.15.78:0/4152871981 shutdown_connections 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.590+0000 7ff9d8377740 1 -- 172.21.15.78:0/4152871981 wait complete. 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.590+0000 7ff9d8377740 1 -- start start 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.590+0000 7ff9d8377740 1 Processor -- start 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b6fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b93960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b5b3a0 0x560da0b5b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0ac3c30 con 0x560da0b9a8b0 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0a68d30 con 0x560da0b5b3a0 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da0797310 con 0x560da0b97740 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 10 client.0 _ll_register_callbacks cb 0x560da0a70d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b93960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b93960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51034/0 (socket says 172.21.15.78:51034) 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/2037948349 learned_addr learned my addr 172.21.15.78:0/2037948349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b6fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b5b3a0 0x560da0b5b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 msgr2=0x560da0b6fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b6fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b5b3a0 msgr2=0x560da0b5b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b5b3a0 0x560da0b5b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 -- 172.21.15.78:0/2037948349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b8003520 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b93960 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4020000 tx=0x7ff9c40010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97cb65c2f5e09a9d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9adffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9adffb640 1 -- 172.21.15.78:0/2037948349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4020da0 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9adffb640 1 -- 172.21.15.78:0/2037948349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9c4006070 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.591+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560da07c36c0 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9adffb640 1 -- 172.21.15.78:0/2037948349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9c4006230 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 10 client.9981 fetch_fsmap learned FSMap version 165 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x560da0ac22b0 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9adffb640 1 -- 172.21.15.78:0/2037948349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560da07c36c0 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9adffb640 1 -- 172.21.15.78:0/2037948349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 165) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff9c4012070 con 0x560da0b9a8b0 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 10 client.9981 fetch_fsmap finished waiting for FSMap version 165 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 10 client.9981 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 20 client.9981 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:17.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0bac360 0x560da0bac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 4 client.9981 mds_command: new command op to 9831 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 --> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560da0814270 con 0x560da0bac360 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0bac360 0x560da0bac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0bac360 0x560da0bac740 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fea9d87786e1d20c server_cookie=ee6d430c54e88dd1 in_seq=0 out_seq=0 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.592+0000 7ff9adffb640 10 client.9981 ms_handle_connect on v2:172.21.15.150:6833/1183887008 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9adffb640 1 -- 172.21.15.78:0/2037948349 <== mds.0 v2:172.21.15.150:6833/1183887008 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560da0814270 con 0x560da0bac360 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9adffb640 10 client.9981 handle_command_reply: tid=0 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 client.9981 shutdown 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 msgr2=0x560da0b93960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b93960 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4020000 tx=0x7ff9c40010a0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 shutdown_connections 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0bac360 0x560da0bac740 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b93960 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9adffb640 0 client.9981 ms_handle_reset on v2:172.21.15.150:6833/1183887008 2024-02-19T14:59:17.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b5b3a0 0x560da0b5b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2037948349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b97740 0x560da0b6fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 >> 172.21.15.78:0/2037948349 conn(0x560da0b5dae0 msgr2=0x560da0b55f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 shutdown_connections 2024-02-19T14:59:17.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.593+0000 7ff9d8377740 1 -- 172.21.15.78:0/2037948349 wait complete. 2024-02-19T14:59:17.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.594+0000 7ff9d8377740 20 client.9981 trim_cache size 0 max 16384 2024-02-19T14:59:17.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.610+0000 7ff9d8377740 1 Processor -- start 2024-02-19T14:59:17.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.610+0000 7ff9d8377740 1 -- start start 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.610+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b93b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.610+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b58180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b5c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9d8377740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0b97410 con 0x560da0b9a8b0 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0a49490 con 0x560da0b97740 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da0b933c0 con 0x560da0b5b3a0 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b5c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b5c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51038/0 (socket says 172.21.15.78:51038) 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2902904783 learned_addr learned my addr 172.21.15.78:0/2902904783 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b58180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b93b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 msgr2=0x560da0b93b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b93b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 msgr2=0x560da0b58180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b58180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/2902904783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560da0a5ea20 con 0x560da0b9a8b0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b5c100 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b522664df64e4389 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9affff640 1 -- 172.21.15.78:0/2902904783 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c803f040 con 0x560da0b9a8b0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9affff640 1 -- 172.21.15.78:0/2902904783 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8040040 con 0x560da0b9a8b0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.611+0000 7ff9affff640 1 -- 172.21.15.78:0/2902904783 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8046440 con 0x560da0b9a8b0 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 -- 172.21.15.78:0/2902904783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 msgr2=0x560da0b5c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b5c100 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c8030000 comp rx=0 tx=0).stop 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 -- 172.21.15.78:0/2902904783 shutdown_connections 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0b5c100 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b58180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 --2- 172.21.15.78:0/2902904783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b93b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 -- 172.21.15.78:0/2902904783 >> 172.21.15.78:0/2902904783 conn(0x560da0b5dae0 msgr2=0x560da0b98e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 -- 172.21.15.78:0/2902904783 shutdown_connections 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 -- 172.21.15.78:0/2902904783 wait complete. 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 -- start start 2024-02-19T14:59:17.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.612+0000 7ff9d8377740 1 Processor -- start 2024-02-19T14:59:17.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b6e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b8eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0c04e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0bb2550 con 0x560da0b9a8b0 2024-02-19T14:59:17.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560da0b8f780 con 0x560da0b97740 2024-02-19T14:59:17.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560da0c07b20 con 0x560da0b5b3a0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9d8377740 10 client.0 _ll_register_callbacks cb 0x7ff9c4023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0c04e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0c04e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51050/0 (socket says 172.21.15.78:51050) 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/505201255 learned_addr learned my addr 172.21.15.78:0/505201255 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b6e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9ce7fc640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b8eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 msgr2=0x560da0b6e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b6e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 msgr2=0x560da0b8eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b8eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.613+0000 7ff9cf7fe640 1 -- 172.21.15.78:0/505201255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d0155a50 con 0x560da0b9a8b0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9cf7fe640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0c04e30 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5396fda5ed72a053 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9adffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9adffb640 1 -- 172.21.15.78:0/505201255 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c803f040 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9adffb640 1 -- 172.21.15.78:0/505201255 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9c8031e30 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9c4015bf0 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9adffb640 1 -- 172.21.15.78:0/505201255 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9c8004020 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 10 client.9987 fetch_fsmap learned FSMap version 165 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x560da0a84780 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9adffb640 1 -- 172.21.15.78:0/505201255 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c4015bf0 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9adffb640 1 -- 172.21.15.78:0/505201255 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 165) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff9c8004180 con 0x560da0b9a8b0 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 10 client.9987 fetch_fsmap finished waiting for FSMap version 165 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 10 client.9987 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 20 client.9987 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0c05370 0x560da0b91fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 4 client.9987 mds_command: new command op to 9831 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 --> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560da0800610 con 0x560da0c05370 2024-02-19T14:59:17.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.614+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0c05370 0x560da0b91fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9adffb640 10 client.9987 ms_handle_connect on v2:172.21.15.150:6833/1183887008 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9ceffd640 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0c05370 0x560da0b91fc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa512541c1880553 server_cookie=29b406bcad7d1610 in_seq=0 out_seq=0 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9adffb640 1 -- 172.21.15.78:0/505201255 <== mds.0 v2:172.21.15.150:6833/1183887008 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x560da0800610 con 0x560da0c05370 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9adffb640 10 client.9987 handle_command_reply: tid=0 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 client.9987 shutdown 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 msgr2=0x560da0c04e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0c04e30 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80097a0 tx=0x7ff9c8004730 comp rx=0 tx=0).stop 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 shutdown_connections 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:6833/1183887008,v1:172.21.15.150:6835/1183887008] conn(0x560da0c05370 0x560da0b91fc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560da0b9a8b0 0x560da0c04e30 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560da0b97740 0x560da0b8eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9adffb640 0 client.9987 ms_handle_reset on v2:172.21.15.150:6833/1183887008 2024-02-19T14:59:17.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 --2- 172.21.15.78:0/505201255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560da0b5b3a0 0x560da0b6e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 >> 172.21.15.78:0/505201255 conn(0x560da0b5dae0 msgr2=0x560da0b90e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 shutdown_connections 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 1 -- 172.21.15.78:0/505201255 wait complete. 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.615+0000 7ff9d8377740 20 client.9987 trim_cache size 0 max 16384 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 9831, 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:59:17.625 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:59:17.626 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 8.9670016930000003, 2024-02-19T14:59:17.626 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 165, 2024-02-19T14:59:17.626 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 188, 2024-02-19T14:59:17.626 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 188, 2024-02-19T14:59:17.626 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 14.098703788 2024-02-19T14:59:17.626 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9bc06b810 msgr2=0x7ff9bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00c1590 tx=0x7ff9c4009040 comp rx=0 tx=0).stop 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 msgr2=0x7ff9d01ea4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 0x7ff9d01ea4c0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000efc0 tx=0x7ff9c000c490 comp rx=0 tx=0).stop 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 shutdown_connections 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d0152700 0x7ff9d00af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9d00c8e00 0x7ff9d00aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 --2- 172.21.15.78:0/224379750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9d00c5730 0x7ff9d01ea4c0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 >> 172.21.15.78:0/224379750 conn(0x7ff9d00cac90 msgr2=0x7ff9d014ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 shutdown_connections 2024-02-19T14:59:17.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.616+0000 7ff9d56c9640 1 -- 172.21.15.78:0/224379750 wait complete. 2024-02-19T14:59:17.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-19T14:59:17.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.731+0000 7ff3a8be9640 1 Processor -- start 2024-02-19T14:59:17.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.731+0000 7ff3a8be9640 1 -- start start 2024-02-19T14:59:17.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a8be9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40ac770 0x7ff3a40c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a8be9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a4158c40 con 0x7ff3a40bdf20 2024-02-19T14:59:17.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a8be9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a40acbb0 con 0x7ff3a40ac390 2024-02-19T14:59:17.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a8be9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40c8de0 con 0x7ff3a40ac770 2024-02-19T14:59:17.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff39bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff3a40bdf20 0x7ff3a415d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39736/0 (socket says 172.21.15.78:39736) 2024-02-19T14:59:17.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff39bfff640 1 -- 172.21.15.78:0/3329059535 learned_addr learned my addr 172.21.15.78:0/3329059535 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a2d9d640 1 --2- 172.21.15.78:0/3329059535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40ac770 0x7ff3a40c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803409897 0 0) 0x7ff3a4158c40 con 0x7ff3a40bdf20 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff390003520 con 0x7ff3a40bdf20 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353321335 0 0) 0x7ff3a40acbb0 con 0x7ff3a40ac390 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a4158c40 con 0x7ff3a40ac390 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a2d9d640 1 -- 172.21.15.78:0/3329059535 >> v1:172.21.15.150:6789/0 conn(0x7ff3a40ac390 legacy=0x7ff3a4158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a2d9d640 1 -- 172.21.15.78:0/3329059535 >> v1:172.21.15.78:6789/0 conn(0x7ff3a40bdf20 legacy=0x7ff3a415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.732+0000 7ff3a2d9d640 1 -- 172.21.15.78:0/3329059535 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a4155a60 con 0x7ff3a40ac770 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.733+0000 7ff3a2d9d640 1 --2- 172.21.15.78:0/3329059535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40ac770 0x7ff3a40c55f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff39c009dc0 tx=0x7ff39c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97617c8fa4ee1873 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.733+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c02cbb0 con 0x7ff3a40ac770 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.733+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c02cd10 con 0x7ff3a40ac770 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.733+0000 7ff3a1d9b640 1 -- 172.21.15.78:0/3329059535 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c034660 con 0x7ff3a40ac770 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 -- 172.21.15.78:0/3329059535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40ac770 msgr2=0x7ff3a40c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/3329059535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40ac770 0x7ff3a40c55f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff39c009dc0 tx=0x7ff39c02be00 comp rx=0 tx=0).stop 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 -- 172.21.15.78:0/3329059535 shutdown_connections 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/3329059535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40ac770 0x7ff3a40c55f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 -- 172.21.15.78:0/3329059535 >> 172.21.15.78:0/3329059535 conn(0x7ff3a40ca890 msgr2=0x7ff3a40caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 -- 172.21.15.78:0/3329059535 shutdown_connections 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.734+0000 7ff3a8be9640 1 -- 172.21.15.78:0/3329059535 wait complete. 2024-02-19T14:59:17.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.735+0000 7ff3a8be9640 1 Processor -- start 2024-02-19T14:59:17.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.735+0000 7ff3a8be9640 1 -- start start 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 0x7ff3a41eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a40ac770 0x7ff3a40c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40bdf20 0x7ff3a40c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40c18d0 con 0x7ff3a40ac390 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40c46e0 con 0x7ff3a40ac770 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40c1ae0 con 0x7ff3a40bdf20 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 0x7ff3a41eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 0x7ff3a41eebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51064/0 (socket says 172.21.15.78:51064) 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 -- 172.21.15.78:0/2229338854 learned_addr learned my addr 172.21.15.78:0/2229338854 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff39bfff640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a40ac770 0x7ff3a40c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a2d9d640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40bdf20 0x7ff3a40c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 -- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40bdf20 msgr2=0x7ff3a40c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40bdf20 0x7ff3a40c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 -- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a40ac770 msgr2=0x7ff3a40c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a40ac770 0x7ff3a40c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 -- 172.21.15.78:0/2229338854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c009a70 con 0x7ff3a40ac390 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a259c640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 0x7ff3a41eebb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff39400c500 tx=0x7ff39400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3940052d0 con 0x7ff3a40ac390 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff394005450 con 0x7ff3a40ac390 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a40c1dc0 con 0x7ff3a40ac390 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.736+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff394012870 con 0x7ff3a40ac390 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.737+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a40c2380 con 0x7ff3a40ac390 2024-02-19T14:59:17.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.738+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a4074990 con 0x7ff3a40ac390 2024-02-19T14:59:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.738+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff3940129d0 con 0x7ff3a40ac390 2024-02-19T14:59:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.738+0000 7ff39affd640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff38406b840 0x7ff38406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.738+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff394089df0 con 0x7ff3a40ac390 2024-02-19T14:59:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.738+0000 7ff39bfff640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff38406b840 0x7ff38406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:17.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.739+0000 7ff39bfff640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff38406b840 0x7ff38406dd00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff3a41586e0 tx=0x7ff38c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:17.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.743+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff394054740 con 0x7ff3a40ac390 2024-02-19T14:59:17.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:17.950+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7ff3a406d2b0 con 0x7ff3a40ac390 2024-02-19T14:59:18.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.676+0000 7ff39affd640 1 -- 172.21.15.78:0/2229338854 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v166) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff394057e00 con 0x7ff3a40ac390 2024-02-19T14:59:18.679 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked down. 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff38406b840 msgr2=0x7ff38406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff38406b840 0x7ff38406dd00 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff3a41586e0 tx=0x7ff38c009040 comp rx=0 tx=0).stop 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 msgr2=0x7ff3a41eebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 0x7ff3a41eebb0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff39400c500 tx=0x7ff39400c9c0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 shutdown_connections 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff38406b840 0x7ff38406dd00 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3a40bdf20 0x7ff3a40c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3a40ac770 0x7ff3a40c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 --2- 172.21.15.78:0/2229338854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3a40ac390 0x7ff3a41eebb0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 >> 172.21.15.78:0/2229338854 conn(0x7ff3a40ca890 msgr2=0x7ff3a406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:18.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 shutdown_connections 2024-02-19T14:59:18.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.679+0000 7ff3a8be9640 1 -- 172.21.15.78:0/2229338854 wait complete. 2024-02-19T14:59:18.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:18.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.793+0000 7faa961c9640 1 Processor -- start 2024-02-19T14:59:18.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.793+0000 7faa961c9640 1 -- start start 2024-02-19T14:59:18.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa961c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faa900c64f0 0x7faa900ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:18.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa961c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa90152d30 con 0x7faa900ad890 2024-02-19T14:59:18.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa961c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa900c7860 con 0x7faa900c49c0 2024-02-19T14:59:18.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa961c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa900c7ce0 con 0x7faa900c64f0 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa8f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7faa900ad890 0x7faa901534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39752/0 (socket says 172.21.15.78:39752) 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa8f7fe640 1 -- 172.21.15.78:0/3475230959 learned_addr learned my addr 172.21.15.78:0/3475230959 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa8ffff640 1 --2- 172.21.15.78:0/3475230959 >> v2:172.21.15.150:3300/0 conn(0x7faa900c64f0 0x7faa900ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085320366 0 0) 0x7faa90152d30 con 0x7faa900ad890 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.794+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa78003520 con 0x7faa900ad890 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140583589 0 0) 0x7faa900c7860 con 0x7faa900c49c0 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa90152d30 con 0x7faa900c49c0 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3328664881 0 0) 0x7faa78003520 con 0x7faa900ad890 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8ffff640 1 -- 172.21.15.78:0/3475230959 >> v1:172.21.15.150:6789/0 conn(0x7faa900c49c0 legacy=0x7faa900c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8ffff640 1 -- 172.21.15.78:0/3475230959 >> v1:172.21.15.78:6789/0 conn(0x7faa900ad890 legacy=0x7faa901534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8ffff640 1 -- 172.21.15.78:0/3475230959 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa900c7f80 con 0x7faa900c64f0 2024-02-19T14:59:18.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.795+0000 7faa8ffff640 1 --2- 172.21.15.78:0/3475230959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900c64f0 0x7faa900ab600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faa88009d90 tx=0x7faa8802bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcd2fa0f0af4c436 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:18.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80001670 con 0x7faa900c64f0 2024-02-19T14:59:18.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8802c820 con 0x7faa900c64f0 2024-02-19T14:59:18.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa8e7fc640 1 -- 172.21.15.78:0/3475230959 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8802cbb0 con 0x7faa900c64f0 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 -- 172.21.15.78:0/3475230959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900c64f0 msgr2=0x7faa900ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 --2- 172.21.15.78:0/3475230959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900c64f0 0x7faa900ab600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faa88009d90 tx=0x7faa8802bcf0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 -- 172.21.15.78:0/3475230959 shutdown_connections 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 --2- 172.21.15.78:0/3475230959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900c64f0 0x7faa900ab600 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 -- 172.21.15.78:0/3475230959 >> 172.21.15.78:0/3475230959 conn(0x7faa9006b700 msgr2=0x7faa900aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 -- 172.21.15.78:0/3475230959 shutdown_connections 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.796+0000 7faa961c9640 1 -- 172.21.15.78:0/3475230959 wait complete. 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.797+0000 7faa961c9640 1 Processor -- start 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.797+0000 7faa961c9640 1 -- start start 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa961c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900ad890 0x7faa90154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa961c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa900c49c0 0x7faa9014b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:18.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa961c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 0x7faa9014aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa961c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9014bf30 con 0x7faa900c64f0 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa961c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa901e9760 con 0x7faa900ad890 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa961c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa901dcc70 con 0x7faa900c49c0 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa8ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 0x7faa9014aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa8ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 0x7faa9014aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51080/0 (socket says 172.21.15.78:51080) 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa8ffff640 1 -- 172.21.15.78:0/4174101287 learned_addr learned my addr 172.21.15.78:0/4174101287 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa8f7fe640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900ad890 0x7faa90154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.798+0000 7faa8effd640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa900c49c0 0x7faa9014b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8ffff640 1 -- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa900c49c0 msgr2=0x7faa9014b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8ffff640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa900c49c0 0x7faa9014b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8ffff640 1 -- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900ad890 msgr2=0x7faa90154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8ffff640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900ad890 0x7faa90154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8ffff640 1 -- 172.21.15.78:0/4174101287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88009a70 con 0x7faa900c64f0 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8f7fe640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900ad890 0x7faa90154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:18.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa8ffff640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 0x7faa9014aee0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7faa88009d90 tx=0x7faa88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88034a00 con 0x7faa900c64f0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa88034b80 con 0x7faa900c64f0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa901e9280 con 0x7faa900c64f0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88003990 con 0x7faa900c64f0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.799+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9014b710 con 0x7faa900c64f0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.800+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa901e9440 con 0x7faa900c64f0 2024-02-19T14:59:18.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.800+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7faa88040450 con 0x7faa900c64f0 2024-02-19T14:59:18.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.801+0000 7faa77fff640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:18.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.801+0000 7faa8f7fe640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:18.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.801+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa880accf0 con 0x7faa900c64f0 2024-02-19T14:59:18.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.801+0000 7faa8f7fe640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faa800011e0 tx=0x7faa80009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:18.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:18.805+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa880b1070 con 0x7faa900c64f0 2024-02-19T14:59:19.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.015+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa90074990 con 0x7faa900c64f0 2024-02-19T14:59:19.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.016+0000 7faa77fff640 1 -- 172.21.15.78:0/4174101287 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7faa88077610 con 0x7faa900c64f0 2024-02-19T14:59:19.019 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:19.019 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:19.019 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:19.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 msgr2=0x7faa7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:19.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faa800011e0 tx=0x7faa80009040 comp rx=0 tx=0).stop 2024-02-19T14:59:19.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 msgr2=0x7faa9014aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:19.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 0x7faa9014aee0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7faa88009d90 tx=0x7faa88004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:19.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 shutdown_connections 2024-02-19T14:59:19.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7faa7c06b810 0x7faa7c06dcd0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:19.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa900c64f0 0x7faa9014aee0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:19.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7faa900c49c0 0x7faa9014b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:19.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 --2- 172.21.15.78:0/4174101287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa900ad890 0x7faa90154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:19.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 >> 172.21.15.78:0/4174101287 conn(0x7faa9006b700 msgr2=0x7faa900c5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:19.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 shutdown_connections 2024-02-19T14:59:19.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:19.018+0000 7faa961c9640 1 -- 172.21.15.78:0/4174101287 wait complete. 2024-02-19T14:59:19.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:19.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:20.435 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.543+0000 7fd59c7d7640 1 Processor -- start 2024-02-19T14:59:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.543+0000 7fd59c7d7640 1 -- start start 2024-02-19T14:59:20.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59c7d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd59406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59c7d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5941588a0 0x7fd594158d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59c7d7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940c6af0 con 0x7fd5940c6d30 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59c7d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940c7940 con 0x7fd5941588a0 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59c7d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5940c7c20 con 0x7fd594155f90 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599d72640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd5940c6d30 0x7fd5940c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39768/0 (socket says 172.21.15.78:39768) 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599d72640 1 -- 172.21.15.78:0/1086199052 learned_addr learned my addr 172.21.15.78:0/1086199052 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59ad74640 1 --2- 172.21.15.78:0/1086199052 >> v2:172.21.15.150:3300/0 conn(0x7fd5941588a0 0x7fd594158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59a573640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd59406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928134386 0 0) 0x7fd5940c6af0 con 0x7fd5940c6d30 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580003520 con 0x7fd5940c6d30 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2981863856 0 0) 0x7fd580003520 con 0x7fd5940c6d30 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 msgr2=0x7fd59406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:20.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd59406d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 >> v2:172.21.15.150:3300/0 conn(0x7fd5941588a0 msgr2=0x7fd594158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 --2- 172.21.15.78:0/1086199052 >> v2:172.21.15.150:3300/0 conn(0x7fd5941588a0 0x7fd594158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940c7f00 con 0x7fd5940c6d30 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fd590001670 con 0x7fd5940c6d30 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 >> v1:172.21.15.78:6789/0 conn(0x7fd5940c6d30 legacy=0x7fd5940c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd580003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd5800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd580004d00 0x7fd580005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5940c7940 con 0x7fd580003970 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5940c7c20 con 0x7fd580004d00 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58000b980 con 0x7fd594155f90 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.544+0000 7fd59a573640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd580003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd599d72640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd5800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59ad74640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd580004d00 0x7fd580005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59a573640 1 -- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 msgr2=0x7fd5800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59a573640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd5800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59a573640 1 -- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd580004d00 msgr2=0x7fd580005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59a573640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd580004d00 0x7fd580005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59a573640 1 -- 172.21.15.78:0/1086199052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd580003520 con 0x7fd580003970 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd599d72640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594155f90 0x7fd5800047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.545+0000 7fd59a573640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd580003d90 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fd584009ac0 tx=0x7fd5800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e37b10c1bb3a84c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58402c6b0 con 0x7fd580003970 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58402c810 con 0x7fd580003970 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd599571640 1 -- 172.21.15.78:0/1086199052 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58402cba0 con 0x7fd580003970 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59c7d7640 1 -- 172.21.15.78:0/1086199052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 msgr2=0x7fd580003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/1086199052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd580003d90 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fd584009ac0 tx=0x7fd5800057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59ad74640 1 -- 172.21.15.78:0/1086199052 reap_dead start 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59c7d7640 1 -- 172.21.15.78:0/1086199052 shutdown_connections 2024-02-19T14:59:20.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59c7d7640 1 -- 172.21.15.78:0/1086199052 >> 172.21.15.78:0/1086199052 conn(0x7fd59406b700 msgr2=0x7fd5940aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59c7d7640 1 -- 172.21.15.78:0/1086199052 shutdown_connections 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.546+0000 7fd59c7d7640 1 -- 172.21.15.78:0/1086199052 wait complete. 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.547+0000 7fd59c7d7640 1 Processor -- start 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.547+0000 7fd59c7d7640 1 -- start start 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59c7d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd5941ede90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59c7d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594153e70 0x7fd5940abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59c7d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5940ac310 0x7fd59406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59c7d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941de7b0 con 0x7fd580003970 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59c7d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941ee6d0 con 0x7fd5940ac310 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59c7d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941ee9b0 con 0x7fd594153e70 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd5941ede90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd5941ede90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51090/0 (socket says 172.21.15.78:51090) 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 -- 172.21.15.78:0/307496054 learned_addr learned my addr 172.21.15.78:0/307496054 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:20.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd599d72640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594153e70 0x7fd5940abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59ad74640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5940ac310 0x7fd59406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 -- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594153e70 msgr2=0x7fd5940abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594153e70 0x7fd5940abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 -- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5940ac310 msgr2=0x7fd59406f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5940ac310 0x7fd59406f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd59a573640 1 -- 172.21.15.78:0/307496054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5941de990 con 0x7fd580003970 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.548+0000 7fd599d72640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594153e70 0x7fd5940abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.549+0000 7fd59a573640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd5941ede90 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fd584000c00 tx=0x7fd584004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.549+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584003810 con 0x7fd580003970 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.549+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd584003970 con 0x7fd580003970 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.549+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5941ee3d0 con 0x7fd580003970 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.549+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584003d00 con 0x7fd580003970 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.549+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd594153850 con 0x7fd580003970 2024-02-19T14:59:20.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.550+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5940ab370 con 0x7fd580003970 2024-02-19T14:59:20.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.550+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd584034440 con 0x7fd580003970 2024-02-19T14:59:20.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.551+0000 7fd58affd640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57806b840 0x7fd57806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:20.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.551+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5840ac230 con 0x7fd580003970 2024-02-19T14:59:20.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.551+0000 7fd599d72640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57806b840 0x7fd57806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:20.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.551+0000 7fd599d72640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57806b840 0x7fd57806dd00 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd5940aa530 tx=0x7fd590001430 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:20.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.555+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd584076c00 con 0x7fd580003970 2024-02-19T14:59:20.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.763+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5941eeb70 con 0x7fd580003970 2024-02-19T14:59:20.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.763+0000 7fd58affd640 1 -- 172.21.15.78:0/307496054 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7fd5941eeb70 con 0x7fd580003970 2024-02-19T14:59:20.766 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:20.766 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:20.767 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:20.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57806b840 msgr2=0x7fd57806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:20.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57806b840 0x7fd57806dd00 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fd5940aa530 tx=0x7fd590001430 comp rx=0 tx=0).stop 2024-02-19T14:59:20.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 msgr2=0x7fd5941ede90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:20.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd5941ede90 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fd584000c00 tx=0x7fd584004730 comp rx=0 tx=0).stop 2024-02-19T14:59:20.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 shutdown_connections 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57806b840 0x7fd57806dd00 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5940ac310 0x7fd59406f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd594153e70 0x7fd5940abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 --2- 172.21.15.78:0/307496054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd580003970 0x7fd5941ede90 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 >> 172.21.15.78:0/307496054 conn(0x7fd59406b700 msgr2=0x7fd5940c1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 shutdown_connections 2024-02-19T14:59:20.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:20.765+0000 7fd59c7d7640 1 -- 172.21.15.78:0/307496054 wait complete. 2024-02-19T14:59:21.182 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:21.182 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:22.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:22.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.292+0000 7fccf7aa4640 1 Processor -- start 2024-02-19T14:59:22.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.292+0000 7fccf7aa4640 1 -- start start 2024-02-19T14:59:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf7aa4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fccf00c25e0 0x7fccf01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf7aa4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf7aa4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf0072250 con 0x7fccf00c29c0 2024-02-19T14:59:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf7aa4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf0152c90 con 0x7fccf00c25e0 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf7aa4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0072480 con 0x7fccf00c5be0 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf5840640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fccf00c25e0 0x7fccf01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf503f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fccf00c29c0 0x7fccf00c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39770/0 (socket says 172.21.15.78:39770) 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf503f640 1 -- 172.21.15.78:0/1318273486 learned_addr learned my addr 172.21.15.78:0/1318273486 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf6041640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf5840640 1 -- 172.21.15.78:0/1318273486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 msgr2=0x7fccf006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf483e640 1 -- 172.21.15.78:0/1318273486 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28328670 0 0) 0x7fccf0072250 con 0x7fccf00c29c0 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.293+0000 7fccf5840640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf5840640 1 -- 172.21.15.78:0/1318273486 >> v1:172.21.15.78:6789/0 conn(0x7fccf00c29c0 legacy=0x7fccf00c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf5840640 1 -- 172.21.15.78:0/1318273486 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf00c4b90 con 0x7fccf00c25e0 2024-02-19T14:59:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf6041640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:22.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf5840640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf01526a0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fccec0097a0 tx=0x7fccec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fb70202d127afb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:22.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf483e640 1 -- 172.21.15.78:0/1318273486 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec02e070 con 0x7fccf00c25e0 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf483e640 1 -- 172.21.15.78:0/1318273486 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec0315a0 con 0x7fccf00c25e0 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.294+0000 7fccf483e640 1 -- 172.21.15.78:0/1318273486 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec031930 con 0x7fccf00c25e0 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 -- 172.21.15.78:0/1318273486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 msgr2=0x7fccf01526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf01526a0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fccec0097a0 tx=0x7fccec02b840 comp rx=0 tx=0).stop 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 -- 172.21.15.78:0/1318273486 shutdown_connections 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/1318273486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf01526a0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 -- 172.21.15.78:0/1318273486 >> 172.21.15.78:0/1318273486 conn(0x7fccf006b270 msgr2=0x7fccf006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 -- 172.21.15.78:0/1318273486 shutdown_connections 2024-02-19T14:59:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.295+0000 7fccf7aa4640 1 -- 172.21.15.78:0/1318273486 wait complete. 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.296+0000 7fccf7aa4640 1 Processor -- start 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.296+0000 7fccf7aa4640 1 -- start start 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf7aa4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf00a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf7aa4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf00c29c0 0x7fccf0153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf7aa4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf00bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf7aa4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0155370 con 0x7fccf00c25e0 2024-02-19T14:59:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf7aa4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00a8ca0 con 0x7fccf00c29c0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf7aa4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01545b0 con 0x7fccf00c5be0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf00a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf00a86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51096/0 (socket says 172.21.15.78:51096) 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 -- 172.21.15.78:0/3332304310 learned_addr learned my addr 172.21.15.78:0/3332304310 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf6041640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf00bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf503f640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf00c29c0 0x7fccf0153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 -- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 msgr2=0x7fccf00bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf00bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 -- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf00c29c0 msgr2=0x7fccf0153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf00c29c0 0x7fccf0153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.297+0000 7fccf5840640 1 -- 172.21.15.78:0/3332304310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec009400 con 0x7fccf00c25e0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.298+0000 7fccf5840640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf00a86c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fccec034040 tx=0x7fccec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.298+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec032840 con 0x7fccf00c25e0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.298+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec0329c0 con 0x7fccf00c25e0 2024-02-19T14:59:22.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.298+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec032d70 con 0x7fccf00c25e0 2024-02-19T14:59:22.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.298+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf0154890 con 0x7fccf00c25e0 2024-02-19T14:59:22.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.298+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf0154e50 con 0x7fccf00c25e0 2024-02-19T14:59:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.299+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf0074990 con 0x7fccf00c25e0 2024-02-19T14:59:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.299+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fccec04e430 con 0x7fccf00c25e0 2024-02-19T14:59:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.300+0000 7fcce67fc640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd406b840 0x7fccd406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.300+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fccec0ac3e0 con 0x7fccf00c25e0 2024-02-19T14:59:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.300+0000 7fccf503f640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd406b840 0x7fccd406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.300+0000 7fccf503f640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd406b840 0x7fccd406dd00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fccf00abd90 tx=0x7fcce0007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.304+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fccec076d00 con 0x7fccf00c25e0 2024-02-19T14:59:22.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.512+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf0155010 con 0x7fccf00c25e0 2024-02-19T14:59:22.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.513+0000 7fcce67fc640 1 -- 172.21.15.78:0/3332304310 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7fccec07a3c0 con 0x7fccf00c25e0 2024-02-19T14:59:22.516 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:22.516 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:22.516 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd406b840 msgr2=0x7fccd406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd406b840 0x7fccd406dd00 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fccf00abd90 tx=0x7fcce0007c00 comp rx=0 tx=0).stop 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 msgr2=0x7fccf00a86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf00a86c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fccec034040 tx=0x7fccec0046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 shutdown_connections 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccd406b840 0x7fccd406dd00 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccf00c5be0 0x7fccf00bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf00c29c0 0x7fccf0153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 --2- 172.21.15.78:0/3332304310 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccf00c25e0 0x7fccf00a86c0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:22.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 >> 172.21.15.78:0/3332304310 conn(0x7fccf006b270 msgr2=0x7fccf006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:22.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 shutdown_connections 2024-02-19T14:59:22.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:22.515+0000 7fccf7aa4640 1 -- 172.21.15.78:0/3332304310 wait complete. 2024-02-19T14:59:22.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:22.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:23.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:24.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.041+0000 7f6c3f125640 1 Processor -- start 2024-02-19T14:59:24.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.041+0000 7f6c3f125640 1 -- start start 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3f125640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c38152830 0x7f6c380c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3f125640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 0x7f6c3806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3f125640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c381586a0 con 0x7f6c380c5760 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3f125640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c380c6550 con 0x7f6c38152830 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3f125640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c38159320 con 0x7f6c380c7990 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3d6c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 0x7f6c3806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3cec1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c38152830 0x7f6c380c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3d6c2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 0x7f6c3806c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:54180/0 (socket says 172.21.15.78:54180) 2024-02-19T14:59:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.042+0000 7f6c3d6c2640 1 -- 172.21.15.78:0/3947166463 learned_addr learned my addr 172.21.15.78:0/3947166463 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.043+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 msgr2=0x7f6c3806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.043+0000 7f6c3cec1640 1 --2- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 0x7f6c3806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.043+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3947166463 >> v1:172.21.15.150:6789/0 conn(0x7f6c380c5760 legacy=0x7f6c380c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.043+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3947166463 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38158980 con 0x7f6c38152830 2024-02-19T14:59:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.043+0000 7f6c3d6c2640 1 --2- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 0x7f6c3806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.043+0000 7f6c3cec1640 1 --2- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c380c5220 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6c280097d0 tx=0x7f6c2802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0265a1a4b350129 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c377fe640 1 -- 172.21.15.78:0/3947166463 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28031770 con 0x7f6c38152830 2024-02-19T14:59:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c377fe640 1 -- 172.21.15.78:0/3947166463 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c280318d0 con 0x7f6c38152830 2024-02-19T14:59:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c377fe640 1 -- 172.21.15.78:0/3947166463 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28031c60 con 0x7f6c38152830 2024-02-19T14:59:24.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 -- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 msgr2=0x7f6c380c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c380c5220 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6c280097d0 tx=0x7f6c2802b810 comp rx=0 tx=0).stop 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 -- 172.21.15.78:0/3947166463 shutdown_connections 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c380c7990 0x7f6c3806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3947166463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c380c5220 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 -- 172.21.15.78:0/3947166463 >> 172.21.15.78:0/3947166463 conn(0x7f6c380ca8a0 msgr2=0x7f6c380cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 -- 172.21.15.78:0/3947166463 shutdown_connections 2024-02-19T14:59:24.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.044+0000 7f6c3f125640 1 -- 172.21.15.78:0/3947166463 wait complete. 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.045+0000 7f6c3f125640 1 Processor -- start 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 -- start start 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 0x7f6c380c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c381ec130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c38147af0 0x7f6c38147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380be740 con 0x7f6c380c5760 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c380c38e0 con 0x7f6c38152830 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.046+0000 7f6c3f125640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c381484e0 con 0x7f6c38147af0 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 0x7f6c380c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:24.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 0x7f6c380c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51104/0 (socket says 172.21.15.78:51104) 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3743724231 learned_addr learned my addr 172.21.15.78:0/3743724231 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3d6c2640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c38147af0 0x7f6c38147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c37fff640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c381ec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c38147af0 msgr2=0x7f6c38147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c38147af0 0x7f6c38147fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 msgr2=0x7f6c381ec130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c381ec130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 -- 172.21.15.78:0/3743724231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30009b30 con 0x7f6c380c5760 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3cec1640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 0x7f6c380c3d00 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f6c280097d0 tx=0x7f6c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28032810 con 0x7f6c380c5760 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c28032990 con 0x7f6c380c5760 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c28009400 con 0x7f6c380c5760 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28032d40 con 0x7f6c380c5760 2024-02-19T14:59:24.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.047+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c380c1300 con 0x7f6c380c5760 2024-02-19T14:59:24.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.048+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c380c4240 con 0x7f6c380c5760 2024-02-19T14:59:24.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.049+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6c2804e440 con 0x7f6c380c5760 2024-02-19T14:59:24.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.049+0000 7f6c357fa640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2406b840 0x7f6c2406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:24.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.049+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c280ac290 con 0x7f6c380c5760 2024-02-19T14:59:24.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.049+0000 7f6c37fff640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2406b840 0x7f6c2406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:24.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.049+0000 7f6c37fff640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2406b840 0x7f6c2406dd00 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6c380c4b80 tx=0x7f6c2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:24.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.053+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c28076b20 con 0x7f6c380c5760 2024-02-19T14:59:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.260+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c38074990 con 0x7f6c380c5760 2024-02-19T14:59:24.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.261+0000 7f6c357fa640 1 -- 172.21.15.78:0/3743724231 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7f6c2807a1e0 con 0x7f6c380c5760 2024-02-19T14:59:24.263 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:24.264 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:24.264 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:24.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2406b840 msgr2=0x7f6c2406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2406b840 0x7f6c2406dd00 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6c380c4b80 tx=0x7f6c2c009040 comp rx=0 tx=0).stop 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 msgr2=0x7f6c380c3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 0x7f6c380c3d00 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f6c280097d0 tx=0x7f6c280046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 shutdown_connections 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6c2406b840 0x7f6c2406dd00 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c38147af0 0x7f6c38147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c38152830 0x7f6c381ec130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 --2- 172.21.15.78:0/3743724231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c380c5760 0x7f6c380c3d00 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 >> 172.21.15.78:0/3743724231 conn(0x7f6c380ca8a0 msgr2=0x7f6c3814dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 shutdown_connections 2024-02-19T14:59:24.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:24.263+0000 7f6c3f125640 1 -- 172.21.15.78:0/3743724231 wait complete. 2024-02-19T14:59:24.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:24.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:25.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.785+0000 7fe60b5c9640 1 Processor -- start 2024-02-19T14:59:25.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.785+0000 7fe60b5c9640 1 -- start start 2024-02-19T14:59:25.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe60b5c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe6041528d0 0x7fe6040c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe60b5c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6040c19c0 0x7fe604155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe60b5c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe604072180 0x7fe6041555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe60b5c9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6040734c0 con 0x7fe604072180 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe60b5c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe604155af0 con 0x7fe6041528d0 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe60b5c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe604072e80 con 0x7fe6040c19c0 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe609b66640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe604072180 0x7fe6041555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe609b66640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe604072180 0x7fe6041555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51116/0 (socket says 172.21.15.78:51116) 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe609b66640 1 -- 172.21.15.78:0/2297680897 learned_addr learned my addr 172.21.15.78:0/2297680897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe608b64640 1 --2- 172.21.15.78:0/2297680897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6040c19c0 0x7fe604155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.786+0000 7fe609365640 1 --2- 172.21.15.78:0/2297680897 >> v2:172.21.15.150:3300/0 conn(0x7fe6041528d0 0x7fe6040c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.787+0000 7fe609b66640 1 -- 172.21.15.78:0/2297680897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6040c19c0 msgr2=0x7fe604155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.787+0000 7fe609b66640 1 --2- 172.21.15.78:0/2297680897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6040c19c0 0x7fe604155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.787+0000 7fe609b66640 1 -- 172.21.15.78:0/2297680897 >> v2:172.21.15.150:3300/0 conn(0x7fe6041528d0 msgr2=0x7fe6040c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.787+0000 7fe609b66640 1 --2- 172.21.15.78:0/2297680897 >> v2:172.21.15.150:3300/0 conn(0x7fe6041528d0 0x7fe6040c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.787+0000 7fe609b66640 1 -- 172.21.15.78:0/2297680897 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60406e990 con 0x7fe604072180 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.787+0000 7fe609b66640 1 --2- 172.21.15.78:0/2297680897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe604072180 0x7fe6041555b0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fe6040c85c0 tx=0x7fe5fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846f396632881cdf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe5fbfff640 1 -- 172.21.15.78:0/2297680897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc02c9e0 con 0x7fe604072180 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe5fbfff640 1 -- 172.21.15.78:0/2297680897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc02cb40 con 0x7fe604072180 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe5fbfff640 1 -- 172.21.15.78:0/2297680897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0343f0 con 0x7fe604072180 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 -- 172.21.15.78:0/2297680897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe604072180 msgr2=0x7fe6041555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/2297680897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe604072180 0x7fe6041555b0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fe6040c85c0 tx=0x7fe5fc02be40 comp rx=0 tx=0).stop 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 -- 172.21.15.78:0/2297680897 shutdown_connections 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/2297680897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe604072180 0x7fe6041555b0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/2297680897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6040c19c0 0x7fe604155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/2297680897 >> v2:172.21.15.150:3300/0 conn(0x7fe6041528d0 0x7fe6040c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 -- 172.21.15.78:0/2297680897 >> 172.21.15.78:0/2297680897 conn(0x7fe6040c7590 msgr2=0x7fe6040c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 -- 172.21.15.78:0/2297680897 shutdown_connections 2024-02-19T14:59:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.788+0000 7fe60b5c9640 1 -- 172.21.15.78:0/2297680897 wait complete. 2024-02-19T14:59:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.789+0000 7fe60b5c9640 1 Processor -- start 2024-02-19T14:59:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.789+0000 7fe60b5c9640 1 -- start start 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe60b5c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe604072180 0x7fe6041de860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe60b5c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c19c0 0x7fe6040acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe60b5c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 0x7fe6040ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe60b5c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6040c5620 con 0x7fe6041528d0 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe60b5c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60414ccd0 con 0x7fe6040c19c0 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe60b5c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60414cf20 con 0x7fe604072180 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 0x7fe6040ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 0x7fe6040ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51124/0 (socket says 172.21.15.78:51124) 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 -- 172.21.15.78:0/4204550457 learned_addr learned my addr 172.21.15.78:0/4204550457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609365640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe604072180 0x7fe6041de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe608b64640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c19c0 0x7fe6040acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 -- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe604072180 msgr2=0x7fe6041de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe604072180 0x7fe6041de860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 -- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c19c0 msgr2=0x7fe6040acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c19c0 0x7fe6040acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe609b66640 1 -- 172.21.15.78:0/4204550457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc009b30 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.790+0000 7fe608b64640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c19c0 0x7fe6040acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.791+0000 7fe609b66640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 0x7fe6040ad010 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc036040 tx=0x7fe5fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.791+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc009510 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.791+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc03e450 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.791+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6041eb3b0 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.791+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc03e7e0 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.791+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe604153060 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.792+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6041eb8d0 con 0x7fe6041528d0 2024-02-19T14:59:25.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.793+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe5fc03e940 con 0x7fe6041528d0 2024-02-19T14:59:25.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.793+0000 7fe5f9ffb640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe5e806b840 0x7fe5e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:25.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.793+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe5fc0ad190 con 0x7fe6041528d0 2024-02-19T14:59:25.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.793+0000 7fe609365640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe5e806b840 0x7fe5e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:25.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.793+0000 7fe609365640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe5e806b840 0x7fe5e806dd00 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe600009aa0 tx=0x7fe600009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:25.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:25.798+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe5fc077bd0 con 0x7fe6041528d0 2024-02-19T14:59:26.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.003+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe604074990 con 0x7fe6041528d0 2024-02-19T14:59:26.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.004+0000 7fe5f9ffb640 1 -- 172.21.15.78:0/4204550457 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7fe5fc07b290 con 0x7fe6041528d0 2024-02-19T14:59:26.006 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:26.006 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:26.007 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:26.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.005+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe5e806b840 msgr2=0x7fe5e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.005+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe5e806b840 0x7fe5e806dd00 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe600009aa0 tx=0x7fe600009190 comp rx=0 tx=0).stop 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.005+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 msgr2=0x7fe6040ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.005+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 0x7fe6040ad010 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc036040 tx=0x7fe5fc004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 shutdown_connections 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe5e806b840 0x7fe5e806dd00 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6041528d0 0x7fe6040ad010 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c19c0 0x7fe6040acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 --2- 172.21.15.78:0/4204550457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe604072180 0x7fe6041de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 >> 172.21.15.78:0/4204550457 conn(0x7fe6040c7590 msgr2=0x7fe604150480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 shutdown_connections 2024-02-19T14:59:26.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:26.006+0000 7fe60b5c9640 1 -- 172.21.15.78:0/4204550457 wait complete. 2024-02-19T14:59:26.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:26.421 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:27.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:59:27.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:59:27.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.536+0000 7f81f6984640 1 Processor -- start 2024-02-19T14:59:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.536+0000 7f81f6984640 1 -- start start 2024-02-19T14:59:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f6984640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81f0152830 0x7f81f00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f6984640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81f00c5760 0x7f81f00c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f6984640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f01586a0 con 0x7f81f00c77c0 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f6984640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f00c6550 con 0x7f81f00c5760 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f6984640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f0159320 con 0x7f81f0152830 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81ef7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81f00c5760 0x7f81f00c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f4f21640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f81f00c77c0 0x7f81f006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49988/0 (socket says 172.21.15.78:49988) 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.537+0000 7f81f4f21640 1 -- 172.21.15.78:0/2105806851 learned_addr learned my addr 172.21.15.78:0/2105806851 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81effff640 1 --2- 172.21.15.78:0/2105806851 >> v2:172.21.15.150:3300/0 conn(0x7f81f0152830 0x7f81f00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81ef7fe640 1 -- 172.21.15.78:0/2105806851 >> v2:172.21.15.150:3300/0 conn(0x7f81f0152830 msgr2=0x7f81f00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81ef7fe640 1 --2- 172.21.15.78:0/2105806851 >> v2:172.21.15.150:3300/0 conn(0x7f81f0152830 0x7f81f00c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81ef7fe640 1 -- 172.21.15.78:0/2105806851 >> v1:172.21.15.78:6789/0 conn(0x7f81f00c77c0 legacy=0x7f81f006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81ef7fe640 1 -- 172.21.15.78:0/2105806851 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f0158980 con 0x7f81f00c5760 2024-02-19T14:59:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81eeffd640 1 -- 172.21.15.78:0/2105806851 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084617867 0 0) 0x7f81f01586a0 con 0x7f81f00c77c0 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.538+0000 7f81ef7fe640 1 --2- 172.21.15.78:0/2105806851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c7280 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f81f00c7250 tx=0x7f81e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caf91c8225c29b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81eeffd640 1 -- 172.21.15.78:0/2105806851 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e402e070 con 0x7f81f00c5760 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81eeffd640 1 -- 172.21.15.78:0/2105806851 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e40315a0 con 0x7f81f00c5760 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81eeffd640 1 -- 172.21.15.78:0/2105806851 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4031930 con 0x7f81f00c5760 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 -- 172.21.15.78:0/2105806851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 msgr2=0x7f81f00c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 --2- 172.21.15.78:0/2105806851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c7280 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f81f00c7250 tx=0x7f81e402b840 comp rx=0 tx=0).stop 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 -- 172.21.15.78:0/2105806851 shutdown_connections 2024-02-19T14:59:27.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 --2- 172.21.15.78:0/2105806851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c7280 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 --2- 172.21.15.78:0/2105806851 >> v2:172.21.15.150:3300/0 conn(0x7f81f0152830 0x7f81f00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 -- 172.21.15.78:0/2105806851 >> 172.21.15.78:0/2105806851 conn(0x7f81f00ca8a0 msgr2=0x7f81f00cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:27.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 -- 172.21.15.78:0/2105806851 shutdown_connections 2024-02-19T14:59:27.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.539+0000 7f81f6984640 1 -- 172.21.15.78:0/2105806851 wait complete. 2024-02-19T14:59:27.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.540+0000 7f81f6984640 1 Processor -- start 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 -- start start 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81f00c77c0 0x7f81f0148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81f0152830 0x7f81f0148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f00ca020 con 0x7f81f00c5760 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01ee2e0 con 0x7f81f00c77c0 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.541+0000 7f81f6984640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f00c1a10 con 0x7f81f0152830 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57304/0 (socket says 172.21.15.78:57304) 2024-02-19T14:59:27.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 -- 172.21.15.78:0/1281470925 learned_addr learned my addr 172.21.15.78:0/1281470925 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81f4f21640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81f0152830 0x7f81f0148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81ef7fe640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81f00c77c0 0x7f81f0148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 -- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81f0152830 msgr2=0x7f81f0148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81f0152830 0x7f81f0148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 -- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81f00c77c0 msgr2=0x7f81f0148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81f00c77c0 0x7f81f0148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 -- 172.21.15.78:0/1281470925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4009400 con 0x7f81f00c5760 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81effff640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c0810 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f81e000d5f0 tx=0x7f81e000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e00125f0 con 0x7f81f00c5760 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e0012750 con 0x7f81f00c5760 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0012b00 con 0x7f81f00c5760 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f00c1cf0 con 0x7f81f00c5760 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.542+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f00c0d50 con 0x7f81f00c5760 2024-02-19T14:59:27.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.543+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f00c2210 con 0x7f81f00c5760 2024-02-19T14:59:27.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.544+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81e0012c60 con 0x7f81f00c5760 2024-02-19T14:59:27.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.544+0000 7f81ecff9640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81dc06b810 0x7f81dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:27.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.544+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81e008a7e0 con 0x7f81f00c5760 2024-02-19T14:59:27.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.544+0000 7f81ef7fe640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81dc06b810 0x7f81dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:27.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.545+0000 7f81ef7fe640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81dc06b810 0x7f81dc06dcd0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f81e4036000 tx=0x7f81e403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:27.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.549+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81e0055100 con 0x7f81f00c5760 2024-02-19T14:59:27.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.755+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81f0074990 con 0x7f81f00c5760 2024-02-19T14:59:27.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.756+0000 7f81ecff9640 1 -- 172.21.15.78:0/1281470925 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7f81e00587c0 con 0x7f81f00c5760 2024-02-19T14:59:27.758 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:27.759 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:27.759 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81dc06b810 msgr2=0x7f81dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81dc06b810 0x7f81dc06dcd0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f81e4036000 tx=0x7f81e403d040 comp rx=0 tx=0).stop 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 msgr2=0x7f81f00c0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c0810 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f81e000d5f0 tx=0x7f81e000dab0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 shutdown_connections 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81dc06b810 0x7f81dc06dcd0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81f0152830 0x7f81f0148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81f00c77c0 0x7f81f0148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 --2- 172.21.15.78:0/1281470925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81f00c5760 0x7f81f00c0810 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 >> 172.21.15.78:0/1281470925 conn(0x7f81f00ca8a0 msgr2=0x7f81f0150c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 shutdown_connections 2024-02-19T14:59:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:27.758+0000 7f81f6984640 1 -- 172.21.15.78:0/1281470925 wait complete. 2024-02-19T14:59:28.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:28.174 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:29.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.282+0000 7f6bd250c640 1 Processor -- start 2024-02-19T14:59:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.282+0000 7f6bd250c640 1 -- start start 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.283+0000 7f6bd250c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 0x7f6bcc071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.283+0000 7f6bd250c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0c1120 con 0x7f6bcc14ece0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.283+0000 7f6bd250c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bcc0c6080 con 0x7f6bcc0c15b0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.283+0000 7f6bd250c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0c57e0 con 0x7f6bcc0719d0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.283+0000 7f6bcbfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6bcc14ece0 0x7f6bcc0c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50004/0 (socket says 172.21.15.78:50004) 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.283+0000 7f6bcbfff640 1 -- 172.21.15.78:0/527905243 learned_addr learned my addr 172.21.15.78:0/527905243 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236754840 0 0) 0x7f6bcc0c1120 con 0x7f6bcc14ece0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb4003520 con 0x7f6bcc14ece0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcb7fe640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 0x7f6bcc071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472024781 0 0) 0x7f6bb4003520 con 0x7f6bcc14ece0 2024-02-19T14:59:29.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 msgr2=0x7f6bcc071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 0x7f6bcc071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 >> v1:172.21.15.150:6789/0 conn(0x7f6bcc0c15b0 legacy=0x7f6bcc0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0c5ac0 con 0x7f6bcc14ece0 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f6bbc001670 con 0x7f6bcc14ece0 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 >> v1:172.21.15.78:6789/0 conn(0x7f6bcc14ece0 legacy=0x7f6bcc0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bb4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 0x7f6bb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bb4004d00 0x7f6bb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0c57e0 con 0x7f6bb40039a0 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb400bb10 con 0x7f6bb4004d00 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb400bca0 con 0x7f6bcc0719d0 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bb4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcb7fe640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 0x7f6bb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bd0aa9640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bb4004d00 0x7f6bb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcbfff640 1 -- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 msgr2=0x7f6bb40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc0719d0 0x7f6bb40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcbfff640 1 -- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bb4004d00 msgr2=0x7f6bb4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bb4004d00 0x7f6bb4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bcbfff640 1 -- 172.21.15.78:0/527905243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4003520 con 0x7f6bb40039a0 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.284+0000 7f6bd0aa9640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bb4004d00 0x7f6bb4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.285+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bb4003dc0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc007ca0 tx=0x7f6bb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8ff5486222c772 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.285+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc0058d0 con 0x7f6bb40039a0 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.285+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc005a30 con 0x7f6bb40039a0 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.285+0000 7f6bcaffd640 1 -- 172.21.15.78:0/527905243 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc005e20 con 0x7f6bb40039a0 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd250c640 1 -- 172.21.15.78:0/527905243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 msgr2=0x7f6bb4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd250c640 1 --2- 172.21.15.78:0/527905243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bb4003dc0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc007ca0 tx=0x7f6bb40057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd0aa9640 1 -- 172.21.15.78:0/527905243 reap_dead start 2024-02-19T14:59:29.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd250c640 1 -- 172.21.15.78:0/527905243 shutdown_connections 2024-02-19T14:59:29.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd250c640 1 -- 172.21.15.78:0/527905243 >> 172.21.15.78:0/527905243 conn(0x7f6bcc06b270 msgr2=0x7f6bcc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:29.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd250c640 1 -- 172.21.15.78:0/527905243 shutdown_connections 2024-02-19T14:59:29.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.286+0000 7f6bd250c640 1 -- 172.21.15.78:0/527905243 wait complete. 2024-02-19T14:59:29.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.287+0000 7f6bd250c640 1 Processor -- start 2024-02-19T14:59:29.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.287+0000 7f6bd250c640 1 -- start start 2024-02-19T14:59:29.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd250c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bcc1eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd250c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bcc151ce0 0x7f6bcc0a9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd250c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc1523a0 0x7f6bcc0bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd250c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1eae60 con 0x7f6bb40039a0 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd250c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1ea870 con 0x7f6bcc151ce0 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd250c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0a85e0 con 0x7f6bcc1523a0 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bcc1eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bcc1eb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57330/0 (socket says 172.21.15.78:57330) 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 -- 172.21.15.78:0/1437545071 learned_addr learned my addr 172.21.15.78:0/1437545071 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcb7fe640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bcc151ce0 0x7f6bcc0a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 -- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc1523a0 msgr2=0x7f6bcc0bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bd0aa9640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc1523a0 0x7f6bcc0bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc1523a0 0x7f6bcc0bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 -- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bcc151ce0 msgr2=0x7f6bcc0a9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bcc151ce0 0x7f6bcc0a9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 -- 172.21.15.78:0/1437545071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc1eb7c0 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcb7fe640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bcc151ce0 0x7f6bcc0a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.288+0000 7f6bcbfff640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bcc1eb280 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00d470 tx=0x7f6bbc001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.289+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc007270 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.289+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc0073d0 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.289+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc007780 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.289+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc0a9d50 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.289+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc0a8c70 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.290+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc0731f0 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.290+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6bbc034070 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.290+0000 7f6bc8ff9640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bb806b840 0x7f6bb806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.291+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6bbc03e070 con 0x7f6bb40039a0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.291+0000 7f6bcb7fe640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bb806b840 0x7f6bb806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:29.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.291+0000 7f6bcb7fe640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bb806b840 0x7f6bb806dd00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc14e1c0 tx=0x7f6bc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:29.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.295+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6bcc0731f0 con 0x7f6bb40039a0 2024-02-19T14:59:29.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.503+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bcc0731f0 con 0x7f6bb40039a0 2024-02-19T14:59:29.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.503+0000 7f6bc8ff9640 1 -- 172.21.15.78:0/1437545071 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7f6bcc0731f0 con 0x7f6bb40039a0 2024-02-19T14:59:29.507 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:29.507 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:29.508 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:29.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.505+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bb806b840 msgr2=0x7f6bb806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.505+0000 7f6bd250c640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bb806b840 0x7f6bb806dd00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc14e1c0 tx=0x7f6bc0008040 comp rx=0 tx=0).stop 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 msgr2=0x7f6bcc1eb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bcc1eb280 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00d470 tx=0x7f6bbc001a00 comp rx=0 tx=0).stop 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 shutdown_connections 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6bb806b840 0x7f6bb806dd00 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6bcc1523a0 0x7f6bcc0bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bcc151ce0 0x7f6bcc0a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 --2- 172.21.15.78:0/1437545071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6bb40039a0 0x7f6bcc1eb280 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 >> 172.21.15.78:0/1437545071 conn(0x7f6bcc06b270 msgr2=0x7f6bcc14b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 shutdown_connections 2024-02-19T14:59:29.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:29.506+0000 7f6bd250c640 1 -- 172.21.15.78:0/1437545071 wait complete. 2024-02-19T14:59:29.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:29.923 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:30.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:31.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.037+0000 7fb69d09a640 1 Processor -- start 2024-02-19T14:59:31.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb69d09a640 1 -- start start 2024-02-19T14:59:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb69d09a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb69806c9b0 0x7fb698160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb69d09a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6981551f0 con 0x7fb698152700 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb69d09a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb698155490 con 0x7fb6980c8e00 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb69d09a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb698155770 con 0x7fb69806c9b0 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb69759e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb69806c9b0 0x7fb698160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb696d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb698152700 0x7fb6980c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50006/0 (socket says 172.21.15.78:50006) 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.038+0000 7fb696d9d640 1 -- 172.21.15.78:0/850065386 learned_addr learned my addr 172.21.15.78:0/850065386 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.039+0000 7fb69759e640 1 -- 172.21.15.78:0/850065386 >> v1:172.21.15.150:6789/0 conn(0x7fb6980c8e00 legacy=0x7fb69806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.039+0000 7fb69759e640 1 -- 172.21.15.78:0/850065386 >> v1:172.21.15.78:6789/0 conn(0x7fb698152700 legacy=0x7fb6980c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:31.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.039+0000 7fb69759e640 1 -- 172.21.15.78:0/850065386 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb698155a50 con 0x7fb69806c9b0 2024-02-19T14:59:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69759e640 1 --2- 172.21.15.78:0/850065386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69806c9b0 0x7fb698160990 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdbc5c243bb7904 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb695d9b640 1 -- 172.21.15.78:0/850065386 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69002cc60 con 0x7fb69806c9b0 2024-02-19T14:59:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb695d9b640 1 -- 172.21.15.78:0/850065386 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69002cdc0 con 0x7fb69806c9b0 2024-02-19T14:59:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb695d9b640 1 -- 172.21.15.78:0/850065386 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690034660 con 0x7fb69806c9b0 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 -- 172.21.15.78:0/850065386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69806c9b0 msgr2=0x7fb698160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 --2- 172.21.15.78:0/850065386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69806c9b0 0x7fb698160990 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69002beb0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 -- 172.21.15.78:0/850065386 shutdown_connections 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 --2- 172.21.15.78:0/850065386 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69806c9b0 0x7fb698160990 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 -- 172.21.15.78:0/850065386 >> 172.21.15.78:0/850065386 conn(0x7fb6980cac90 msgr2=0x7fb6980cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 -- 172.21.15.78:0/850065386 shutdown_connections 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.040+0000 7fb69d09a640 1 -- 172.21.15.78:0/850065386 wait complete. 2024-02-19T14:59:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.042+0000 7fb69d09a640 1 Processor -- start 2024-02-19T14:59:31.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.042+0000 7fb69d09a640 1 -- start start 2024-02-19T14:59:31.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69d09a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69806c9b0 0x7fb6980c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69d09a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6980c8e00 0x7fb6981ec060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69d09a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 0x7fb6981ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69d09a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6980c3ac0 con 0x7fb698152700 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69d09a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6980c23f0 con 0x7fb6980c8e00 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69d09a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6980c26d0 con 0x7fb69806c9b0 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 0x7fb6981ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 0x7fb6981ee3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57354/0 (socket says 172.21.15.78:57354) 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 -- 172.21.15.78:0/685738915 learned_addr learned my addr 172.21.15.78:0/685738915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69659c640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6980c8e00 0x7fb6981ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb696d9d640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69806c9b0 0x7fb6980c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 -- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69806c9b0 msgr2=0x7fb6980c2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:31.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69806c9b0 0x7fb6980c2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 -- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6980c8e00 msgr2=0x7fb6981ec060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6980c8e00 0x7fb6981ec060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 -- 172.21.15.78:0/685738915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb690009a70 con 0x7fb698152700 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb696d9d640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69806c9b0 0x7fb6980c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb69759e640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 0x7fb6981ee3d0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb690004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.043+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690040450 con 0x7fb698152700 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.044+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6900405d0 con 0x7fb698152700 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.044+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6980c29b0 con 0x7fb698152700 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.044+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690040980 con 0x7fb698152700 2024-02-19T14:59:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.044+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6980bddb0 con 0x7fb698152700 2024-02-19T14:59:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.045+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6980c2ba0 con 0x7fb698152700 2024-02-19T14:59:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.045+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb690040ae0 con 0x7fb698152700 2024-02-19T14:59:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.046+0000 7fb68f7fe640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb67806b810 0x7fb67806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.046+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb6900acd50 con 0x7fb698152700 2024-02-19T14:59:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.046+0000 7fb696d9d640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb67806b810 0x7fb67806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.046+0000 7fb696d9d640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb67806b810 0x7fb67806dcd0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb688000f60 tx=0x7fb688009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.050+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb6900b0030 con 0x7fb698152700 2024-02-19T14:59:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.258+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb698074990 con 0x7fb698152700 2024-02-19T14:59:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.259+0000 7fb68f7fe640 1 -- 172.21.15.78:0/685738915 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7fb690077850 con 0x7fb698152700 2024-02-19T14:59:31.262 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:31.263 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb67806b810 msgr2=0x7fb67806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb67806b810 0x7fb67806dcd0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb688000f60 tx=0x7fb688009040 comp rx=0 tx=0).stop 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 msgr2=0x7fb6981ee3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 0x7fb6981ee3d0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb690004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 shutdown_connections 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb67806b810 0x7fb67806dcd0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb698152700 0x7fb6981ee3d0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6980c8e00 0x7fb6981ec060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 --2- 172.21.15.78:0/685738915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69806c9b0 0x7fb6980c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.260+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 >> 172.21.15.78:0/685738915 conn(0x7fb6980cac90 msgr2=0x7fb6980cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.261+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 shutdown_connections 2024-02-19T14:59:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:31.261+0000 7fb69d09a640 1 -- 172.21.15.78:0/685738915 wait complete. 2024-02-19T14:59:31.677 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:31.677 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:32.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:32.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.783+0000 7f29449ac640 1 Processor -- start 2024-02-19T14:59:32.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.783+0000 7f29449ac640 1 -- start start 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f29449ac640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f29400acc50 0x7f29400c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f29449ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2940159130 con 0x7f2940155070 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f29449ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400ad030 con 0x7f29400718f0 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f29449ac640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29400c2cf0 con 0x7f29400acc50 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f29400acc50 0x7f29400c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293dd9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2940155070 0x7f2940155500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50008/0 (socket says 172.21.15.78:50008) 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293dd9b640 1 -- 172.21.15.78:0/790820493 learned_addr learned my addr 172.21.15.78:0/790820493 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293e59c640 1 -- 172.21.15.78:0/790820493 >> v1:172.21.15.150:6789/0 conn(0x7f29400718f0 legacy=0x7f2940071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293e59c640 1 -- 172.21.15.78:0/790820493 >> v1:172.21.15.78:6789/0 conn(0x7f2940155070 legacy=0x7f2940155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:32.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293e59c640 1 -- 172.21.15.78:0/790820493 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29401595c0 con 0x7f29400acc50 2024-02-19T14:59:32.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.784+0000 7f293d59a640 1 -- 172.21.15.78:0/790820493 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464336322 0 0) 0x7f2940159130 con 0x7f2940155070 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.785+0000 7f293e59c640 1 --2- 172.21.15.78:0/790820493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400acc50 0x7f29400c17e0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f2930009d90 tx=0x7f293002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bb7c3da0c76909 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.785+0000 7f293d59a640 1 -- 172.21.15.78:0/790820493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293002cbd0 con 0x7f29400acc50 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.785+0000 7f293d59a640 1 -- 172.21.15.78:0/790820493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293002cd30 con 0x7f29400acc50 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.785+0000 7f293d59a640 1 -- 172.21.15.78:0/790820493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293002cea0 con 0x7f29400acc50 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 -- 172.21.15.78:0/790820493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400acc50 msgr2=0x7f29400c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 --2- 172.21.15.78:0/790820493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400acc50 0x7f29400c17e0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f2930009d90 tx=0x7f293002be20 comp rx=0 tx=0).stop 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 -- 172.21.15.78:0/790820493 shutdown_connections 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 --2- 172.21.15.78:0/790820493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29400acc50 0x7f29400c17e0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 -- 172.21.15.78:0/790820493 >> 172.21.15.78:0/790820493 conn(0x7f29400c7800 msgr2=0x7f29400c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 -- 172.21.15.78:0/790820493 shutdown_connections 2024-02-19T14:59:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.786+0000 7f29449ac640 1 -- 172.21.15.78:0/790820493 wait complete. 2024-02-19T14:59:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.787+0000 7f29449ac640 1 Processor -- start 2024-02-19T14:59:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.787+0000 7f29449ac640 1 -- start start 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f29449ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29400718f0 0x7f29401eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f29449ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29400acc50 0x7f2940151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f29449ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 0x7f2940151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f29449ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c2030 con 0x7f2940155070 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f29449ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401eb960 con 0x7f29400acc50 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f29449ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c5000 con 0x7f29400718f0 2024-02-19T14:59:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 0x7f2940151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 0x7f2940151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57370/0 (socket says 172.21.15.78:57370) 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 -- 172.21.15.78:0/2362321092 learned_addr learned my addr 172.21.15.78:0/2362321092 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293e59c640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29400718f0 0x7f29401eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293dd9b640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29400acc50 0x7f2940151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 -- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29400718f0 msgr2=0x7f29401eb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29400718f0 0x7f29401eb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 -- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29400acc50 msgr2=0x7f2940151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29400acc50 0x7f2940151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 -- 172.21.15.78:0/2362321092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930009a70 con 0x7f2940155070 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293dd9b640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29400acc50 0x7f2940151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.788+0000 7f293ed9d640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 0x7f2940151e90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f293800c390 tx=0x7f293800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.789+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380035f0 con 0x7f2940155070 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.789+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938003750 con 0x7f2940155070 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.789+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29400c49d0 con 0x7f2940155070 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.789+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938003b00 con 0x7f2940155070 2024-02-19T14:59:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.789+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2940152980 con 0x7f2940155070 2024-02-19T14:59:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.790+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294006c820 con 0x7f2940155070 2024-02-19T14:59:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.790+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2938003c60 con 0x7f2940155070 2024-02-19T14:59:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.790+0000 7f292effd640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f292006b840 0x7f292006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.791+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29380898a0 con 0x7f2940155070 2024-02-19T14:59:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.791+0000 7f293e59c640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f292006b840 0x7f292006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:32.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.791+0000 7f293e59c640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f292006b840 0x7f292006dd00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2930036000 tx=0x7f2930030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:32.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:32.796+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29380541c0 con 0x7f2940155070 2024-02-19T14:59:33.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.003+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2940074990 con 0x7f2940155070 2024-02-19T14:59:33.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.004+0000 7f292effd640 1 -- 172.21.15.78:0/2362321092 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+4953 (secure 0 0 0) 0x7f2938017090 con 0x7f2940155070 2024-02-19T14:59:33.008 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:33.008 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":167,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:18.680197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9831":{"gid":9831,"name":"d","rank":0,"incarnation":163,"state":"up:stopping","state_seq":3,"addr":"172.21.15.150:6835/1183887008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1183887008},{"type":"v1","addr":"172.21.15.150:6835","nonce":1183887008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:33.008 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 167 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f292006b840 msgr2=0x7f292006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f292006b840 0x7f292006dd00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2930036000 tx=0x7f2930030000 comp rx=0 tx=0).stop 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 msgr2=0x7f2940151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 0x7f2940151e90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f293800c390 tx=0x7f293800c850 comp rx=0 tx=0).stop 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 shutdown_connections 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f292006b840 0x7f292006dd00 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2940155070 0x7f2940151e90 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29400acc50 0x7f2940151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 --2- 172.21.15.78:0/2362321092 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f29400718f0 0x7f29401eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 >> 172.21.15.78:0/2362321092 conn(0x7f29400c7800 msgr2=0x7f29401548c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 shutdown_connections 2024-02-19T14:59:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:33.006+0000 7f29449ac640 1 -- 172.21.15.78:0/2362321092 wait complete. 2024-02-19T14:59:33.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:18.680197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9831}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9831': {'gid': 9831, 'name': 'd', 'rank': 0, 'incarnation': 163, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.150:6835/1183887008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1183887008}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1183887008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:33.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9831:up:stopping 2024-02-19T14:59:33.721 INFO:tasks.ceph.mds.d.smithi150.stderr: -782> 2024-02-19T14:59:13.899+0000 7fd63d32a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:59:33.748 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:59:34.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:34.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.532+0000 7f8655c9a640 1 Processor -- start 2024-02-19T14:59:34.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.532+0000 7f8655c9a640 1 -- start start 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f8655c9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8650155f90 0x7f865006d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f8655c9a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86501588a0 0x7f8650158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f8655c9a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86500c6af0 con 0x7f86500c6d30 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f8655c9a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86500c7940 con 0x7f86501588a0 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f8655c9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86500c7c20 con 0x7f8650155f90 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86501588a0 0x7f8650158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f86500c6d30 0x7f86500c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50020/0 (socket says 172.21.15.78:50020) 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864effd640 1 -- 172.21.15.78:0/259250594 learned_addr learned my addr 172.21.15.78:0/259250594 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864f7fe640 1 --2- 172.21.15.78:0/259250594 >> v2:172.21.15.150:3300/0 conn(0x7f8650155f90 0x7f865006d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864e7fc640 1 -- 172.21.15.78:0/259250594 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197950570 0 0) 0x7f86500c6af0 con 0x7f86500c6d30 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864e7fc640 1 -- 172.21.15.78:0/259250594 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8638003520 con 0x7f86500c6d30 2024-02-19T14:59:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864ffff640 1 -- 172.21.15.78:0/259250594 >> v2:172.21.15.150:3300/0 conn(0x7f8650155f90 msgr2=0x7f865006d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864ffff640 1 --2- 172.21.15.78:0/259250594 >> v2:172.21.15.150:3300/0 conn(0x7f8650155f90 0x7f865006d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864ffff640 1 -- 172.21.15.78:0/259250594 >> v1:172.21.15.78:6789/0 conn(0x7f86500c6d30 legacy=0x7f86500c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864ffff640 1 -- 172.21.15.78:0/259250594 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86500c7f00 con 0x7f86501588a0 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.533+0000 7f864f7fe640 1 --2- 172.21.15.78:0/259250594 >> v2:172.21.15.150:3300/0 conn(0x7f8650155f90 0x7f865006d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.534+0000 7f864ffff640 1 --2- 172.21.15.78:0/259250594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86501588a0 0x7f8650158ca0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f86500c2a40 tx=0x7f864802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96578d3242c4510 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.534+0000 7f864e7fc640 1 -- 172.21.15.78:0/259250594 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864802cbf0 con 0x7f86501588a0 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.534+0000 7f864e7fc640 1 -- 172.21.15.78:0/259250594 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864802cd50 con 0x7f86501588a0 2024-02-19T14:59:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.534+0000 7f864e7fc640 1 -- 172.21.15.78:0/259250594 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648034660 con 0x7f86501588a0 2024-02-19T14:59:34.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 -- 172.21.15.78:0/259250594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86501588a0 msgr2=0x7f8650158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:34.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 --2- 172.21.15.78:0/259250594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86501588a0 0x7f8650158ca0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f86500c2a40 tx=0x7f864802be40 comp rx=0 tx=0).stop 2024-02-19T14:59:34.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 -- 172.21.15.78:0/259250594 shutdown_connections 2024-02-19T14:59:34.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 --2- 172.21.15.78:0/259250594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86501588a0 0x7f8650158ca0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 --2- 172.21.15.78:0/259250594 >> v2:172.21.15.150:3300/0 conn(0x7f8650155f90 0x7f865006d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 -- 172.21.15.78:0/259250594 >> 172.21.15.78:0/259250594 conn(0x7f865006b700 msgr2=0x7f86500aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 -- 172.21.15.78:0/259250594 shutdown_connections 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.535+0000 7f8655c9a640 1 -- 172.21.15.78:0/259250594 wait complete. 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.536+0000 7f8655c9a640 1 Processor -- start 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.536+0000 7f8655c9a640 1 -- start start 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86500c6d30 0x7f86500ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 0x7f865014e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86501588a0 0x7f865014eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501ee940 con 0x7f8650155f90 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501544d0 con 0x7f86501588a0 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86500715e0 con 0x7f86500c6d30 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 0x7f865014e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 0x7f865014e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57388/0 (socket says 172.21.15.78:57388) 2024-02-19T14:59:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 -- 172.21.15.78:0/2308527350 learned_addr learned my addr 172.21.15.78:0/2308527350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864f7fe640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86500c6d30 0x7f86500ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864ffff640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86501588a0 0x7f865014eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 -- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86500c6d30 msgr2=0x7f86500ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86500c6d30 0x7f86500ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 -- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86501588a0 msgr2=0x7f865014eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86501588a0 0x7f865014eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 -- 172.21.15.78:0/2308527350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8648009a70 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f864effd640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 0x7f865014e760 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f864400c460 tx=0x7f864400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644005ce0 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644005e40 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86501eeb50 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.537+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644007290 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.538+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86500708f0 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.539+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86501ea990 con 0x7f8650155f90 2024-02-19T14:59:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.539+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f86440073f0 con 0x7f8650155f90 2024-02-19T14:59:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.539+0000 7f8637fff640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863c06b840 0x7f863c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.539+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f86440898d0 con 0x7f8650155f90 2024-02-19T14:59:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.539+0000 7f864f7fe640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863c06b840 0x7f863c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.540+0000 7f864f7fe640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863c06b840 0x7f863c06dd00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f864000d8b0 tx=0x7f8640008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.544+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8644054160 con 0x7f8650155f90 2024-02-19T14:59:34.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.754+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86500717a0 con 0x7f8650155f90 2024-02-19T14:59:34.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.754+0000 7f8637fff640 1 -- 172.21.15.78:0/2308527350 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f8644057820 con 0x7f8650155f90 2024-02-19T14:59:34.757 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:34.757 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":169,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8758,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:33.520380+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:34.757 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 169 2024-02-19T14:59:34.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863c06b840 msgr2=0x7f863c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:34.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863c06b840 0x7f863c06dd00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f864000d8b0 tx=0x7f8640008040 comp rx=0 tx=0).stop 2024-02-19T14:59:34.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 msgr2=0x7f865014e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:34.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 0x7f865014e760 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f864400c460 tx=0x7f864400c920 comp rx=0 tx=0).stop 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 shutdown_connections 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f863c06b840 0x7f863c06dd00 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8650155f90 0x7f865014e760 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86501588a0 0x7f865014eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 --2- 172.21.15.78:0/2308527350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86500c6d30 0x7f86500ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.756+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 >> 172.21.15.78:0/2308527350 conn(0x7f865006b700 msgr2=0x7f8650152e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.757+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 shutdown_connections 2024-02-19T14:59:34.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:34.757+0000 7f8655c9a640 1 -- 172.21.15.78:0/2308527350 wait complete. 2024-02-19T14:59:35.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 168, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:33.520380+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:35.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-02-19T14:59:35.172 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:35.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.270+0000 7f9c46825640 1 Processor -- start 2024-02-19T14:59:35.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.270+0000 7f9c46825640 1 -- start start 2024-02-19T14:59:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c46825640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c40152610 0x7f9c400c1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c46825640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c400c1cf0 con 0x7f9c40155cd0 2024-02-19T14:59:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c46825640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c400723b0 con 0x7f9c4006f270 2024-02-19T14:59:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c46825640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c40154fe0 con 0x7f9c40152610 2024-02-19T14:59:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9c40155cd0 0x7f9c40072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50036/0 (socket says 172.21.15.78:50036) 2024-02-19T14:59:35.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3f7fe640 1 -- 172.21.15.78:0/646966221 learned_addr learned my addr 172.21.15.78:0/646966221 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/646966221 >> v2:172.21.15.150:3300/0 conn(0x7f9c40152610 0x7f9c400c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521510961 0 0) 0x7f9c34001110 con 0x7f9c40155cd0 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c28003520 con 0x7f9c40155cd0 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300771216 0 0) 0x7f9c28003520 con 0x7f9c40155cd0 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 >> v2:172.21.15.150:3300/0 conn(0x7f9c40152610 msgr2=0x7f9c400c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 --2- 172.21.15.78:0/646966221 >> v2:172.21.15.150:3300/0 conn(0x7f9c40152610 0x7f9c400c1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 >> v1:172.21.15.150:6789/0 conn(0x7f9c4006f270 legacy=0x7f9c40158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c400c1a10 con 0x7f9c40155cd0 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f9c34001800 con 0x7f9c40155cd0 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 >> v1:172.21.15.78:6789/0 conn(0x7f9c40155cd0 legacy=0x7f9c40072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c28003970 0x7f9c28003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c28004370 0x7f9c28004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c280050e0 0x7f9c2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c40154fe0 con 0x7f9c28003970 2024-02-19T14:59:35.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2800c020 con 0x7f9c28004370 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.271+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2800c1e0 con 0x7f9c280050e0 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c28003970 0x7f9c28003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3f7fe640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c28004370 0x7f9c28004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c44dc2640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c280050e0 0x7f9c2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3ffff640 1 -- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c280050e0 msgr2=0x7f9c2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c280050e0 0x7f9c2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3ffff640 1 -- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c28004370 msgr2=0x7f9c28004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c28004370 0x7f9c28004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c44dc2640 1 -- 172.21.15.78:0/646966221 reap_dead start 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.272+0000 7f9c3ffff640 1 -- 172.21.15.78:0/646966221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c28003520 con 0x7f9c28003970 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c28003970 0x7f9c28003d50 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009ac0 tx=0x7f9c280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e526188d3b96fcf1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3002c6b0 con 0x7f9c28003970 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3002c810 con 0x7f9c28003970 2024-02-19T14:59:35.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c3effd640 1 -- 172.21.15.78:0/646966221 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3002cba0 con 0x7f9c28003970 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 -- 172.21.15.78:0/646966221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c28003970 msgr2=0x7f9c28003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c28003970 0x7f9c28003d50 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009ac0 tx=0x7f9c280057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 -- 172.21.15.78:0/646966221 shutdown_connections 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 --2- 172.21.15.78:0/646966221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c28003970 0x7f9c28003d50 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 -- 172.21.15.78:0/646966221 >> 172.21.15.78:0/646966221 conn(0x7f9c400c7590 msgr2=0x7f9c400c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 -- 172.21.15.78:0/646966221 shutdown_connections 2024-02-19T14:59:35.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.273+0000 7f9c46825640 1 -- 172.21.15.78:0/646966221 wait complete. 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.274+0000 7f9c46825640 1 Processor -- start 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 -- start start 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c28003970 0x7f9c400acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c400ab190 0x7f9c401573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 0x7f9c401edff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c400acb40 con 0x7f9c401578e0 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c400ab610 con 0x7f9c400ab190 2024-02-19T14:59:35.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c46825640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c400a7840 con 0x7f9c28003970 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c44dc2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 0x7f9c401edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c44dc2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 0x7f9c401edff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57418/0 (socket says 172.21.15.78:57418) 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.275+0000 7f9c44dc2640 1 -- 172.21.15.78:0/669917946 learned_addr learned my addr 172.21.15.78:0/669917946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c3f7fe640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c400ab190 0x7f9c401573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c28003970 0x7f9c400acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c44dc2640 1 -- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c28003970 msgr2=0x7f9c400acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c44dc2640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c28003970 0x7f9c400acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c44dc2640 1 -- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c400ab190 msgr2=0x7f9c401573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c44dc2640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c400ab190 0x7f9c401573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c44dc2640 1 -- 172.21.15.78:0/669917946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c400a7b20 con 0x7f9c401578e0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c3f7fe640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c400ab190 0x7f9c401573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c44dc2640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 0x7f9c401edff0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f9c400c7390 tx=0x7f9c3800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3800ee50 con 0x7f9c401578e0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c38003b80 con 0x7f9c401578e0 2024-02-19T14:59:35.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c380050d0 con 0x7f9c401578e0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c400ac4f0 con 0x7f9c401578e0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.276+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c400a74d0 con 0x7f9c401578e0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.277+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c400accb0 con 0x7f9c401578e0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.277+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9c38005230 con 0x7f9c401578e0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.278+0000 7f9c3cff9640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c2c06b840 0x7f9c2c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.278+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9c3808a200 con 0x7f9c401578e0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.278+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c2c06b840 0x7f9c2c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.278+0000 7f9c3ffff640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c2c06b840 0x7f9c2c06dd00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9c30036000 tx=0x7f9c30030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:35.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.282+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9c38054b20 con 0x7f9c401578e0 2024-02-19T14:59:35.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.486+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c40074990 con 0x7f9c401578e0 2024-02-19T14:59:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.486+0000 7f9c3cff9640 1 -- 172.21.15.78:0/669917946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f9c380581e0 con 0x7f9c401578e0 2024-02-19T14:59:35.489 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:35.489 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":169,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8758,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169},{"gid":9816,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:33.520380+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 169 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c2c06b840 msgr2=0x7f9c2c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c2c06b840 0x7f9c2c06dd00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9c30036000 tx=0x7f9c30030000 comp rx=0 tx=0).stop 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 msgr2=0x7f9c401edff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 0x7f9c401edff0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f9c400c7390 tx=0x7f9c3800ec00 comp rx=0 tx=0).stop 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 shutdown_connections 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c2c06b840 0x7f9c2c06dd00 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c401578e0 0x7f9c401edff0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c400ab190 0x7f9c401573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 --2- 172.21.15.78:0/669917946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c28003970 0x7f9c400acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 >> 172.21.15.78:0/669917946 conn(0x7f9c400c7590 msgr2=0x7f9c40151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:35.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 shutdown_connections 2024-02-19T14:59:35.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.488+0000 7f9c46825640 1 -- 172.21.15.78:0/669917946 wait complete. 2024-02-19T14:59:35.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-02-19T14:59:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.601+0000 7efff593e640 1 Processor -- start 2024-02-19T14:59:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.601+0000 7efff593e640 1 -- start start 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7efff593e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efff006e830 0x7efff0154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7efff593e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 0x7efff00c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7efff593e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff014f730 con 0x7efff00c6d60 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7efff593e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff00c1940 con 0x7efff006e830 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7efff593e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff00c16d0 con 0x7efff006ff80 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7effeeffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efff00c6d60 0x7efff00c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50052/0 (socket says 172.21.15.78:50052) 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7effeeffd640 1 -- 172.21.15.78:0/3718641609 learned_addr learned my addr 172.21.15.78:0/3718641609 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7effee7fc640 1 --2- 172.21.15.78:0/3718641609 >> v2:172.21.15.150:3300/0 conn(0x7efff006e830 0x7efff0154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7effef7fe640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 0x7efff00c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972763016 0 0) 0x7efff014f730 con 0x7efff00c6d60 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.602+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd8003520 con 0x7efff00c6d60 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145188771 0 0) 0x7effd8003520 con 0x7efff00c6d60 2024-02-19T14:59:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 msgr2=0x7efff00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 0x7efff00c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 >> v2:172.21.15.150:3300/0 conn(0x7efff006e830 msgr2=0x7efff0154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 --2- 172.21.15.78:0/3718641609 >> v2:172.21.15.150:3300/0 conn(0x7efff006e830 0x7efff0154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff0072f70 con 0x7efff00c6d60 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7effe00019f0 con 0x7efff00c6d60 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 >> v1:172.21.15.78:6789/0 conn(0x7efff00c6d60 legacy=0x7efff00c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8003970 0x7effd8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 0x7effd80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd8004d00 0x7effd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00c1940 con 0x7effd8003970 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff00c16d0 con 0x7effd8004d00 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd800b980 con 0x7efff006ff80 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effeeffd640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8003970 0x7effd8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effee7fc640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 0x7effd80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effef7fe640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd8004d00 0x7effd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effeeffd640 1 -- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 msgr2=0x7effd80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effeeffd640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efff006ff80 0x7effd80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effeeffd640 1 -- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd8004d00 msgr2=0x7effd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effeeffd640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd8004d00 0x7effd8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.603+0000 7effeeffd640 1 -- 172.21.15.78:0/3718641609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd8003520 con 0x7effd8003970 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.604+0000 7effeeffd640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8003970 0x7effd8003d90 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7effe000a860 tx=0x7effd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c208b7168c6e61 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.604+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0034630 con 0x7effd8003970 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.604+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe002f070 con 0x7effd8003970 2024-02-19T14:59:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.604+0000 7effedffb640 1 -- 172.21.15.78:0/3718641609 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe00349f0 con 0x7effd8003970 2024-02-19T14:59:35.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.604+0000 7efff593e640 1 -- 172.21.15.78:0/3718641609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8003970 msgr2=0x7effd8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.604+0000 7efff593e640 1 --2- 172.21.15.78:0/3718641609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd8003970 0x7effd8003d90 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7effe000a860 tx=0x7effd80057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.605+0000 7effef7fe640 1 -- 172.21.15.78:0/3718641609 reap_dead start 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.605+0000 7efff593e640 1 -- 172.21.15.78:0/3718641609 shutdown_connections 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.605+0000 7efff593e640 1 -- 172.21.15.78:0/3718641609 >> 172.21.15.78:0/3718641609 conn(0x7efff006c170 msgr2=0x7efff006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.605+0000 7efff593e640 1 -- 172.21.15.78:0/3718641609 shutdown_connections 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.605+0000 7efff593e640 1 -- 172.21.15.78:0/3718641609 wait complete. 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 Processor -- start 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 -- start start 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd8003970 0x7efff0149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 0x7efff0150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efff01edb40 0x7efff01edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff014a460 con 0x7efff01588a0 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff0153530 con 0x7efff01edb40 2024-02-19T14:59:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.606+0000 7efff593e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff00a75c0 con 0x7effd8003970 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 0x7efff0150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 0x7efff0150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57436/0 (socket says 172.21.15.78:57436) 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 -- 172.21.15.78:0/3928790864 learned_addr learned my addr 172.21.15.78:0/3928790864 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effef7fe640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efff01edb40 0x7efff01edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effeeffd640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd8003970 0x7efff0149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 -- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd8003970 msgr2=0x7efff0149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd8003970 0x7efff0149f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 -- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efff01edb40 msgr2=0x7efff01edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efff01edb40 0x7efff01edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 -- 172.21.15.78:0/3928790864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff00a78a0 con 0x7efff01588a0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effee7fc640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 0x7efff0150c00 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7effe4011980 tx=0x7effe4011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe4004240 con 0x7efff01588a0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe40043a0 con 0x7efff01588a0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff0157dc0 con 0x7efff01588a0 2024-02-19T14:59:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400fae0 con 0x7efff01588a0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.607+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff0158380 con 0x7efff01588a0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.608+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff0074990 con 0x7efff01588a0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.608+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7effe400fc40 con 0x7efff01588a0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.609+0000 7effd7fff640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effdc06b810 0x7effdc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.609+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7effe408a9d0 con 0x7efff01588a0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.609+0000 7effeeffd640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effdc06b810 0x7effdc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.609+0000 7effeeffd640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effdc06b810 0x7effdc06dcd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7effe0034000 tx=0x7effe0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:35.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.613+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7effe4055260 con 0x7efff01588a0 2024-02-19T14:59:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:35.823+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7efff0155fc0 con 0x7efff01588a0 2024-02-19T14:59:36.726 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:59:36.723+0000 7f16560b1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:59:36.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.733+0000 7effd7fff640 1 -- 172.21.15.78:0/3928790864 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v170) v1 ==== 105+0+0 (secure 0 0 0) 0x7effe40230b0 con 0x7efff01588a0 2024-02-19T14:59:36.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effdc06b810 msgr2=0x7effdc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effdc06b810 0x7effdc06dcd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7effe0034000 tx=0x7effe0009030 comp rx=0 tx=0).stop 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 msgr2=0x7efff0150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 0x7efff0150c00 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7effe4011980 tx=0x7effe4011e40 comp rx=0 tx=0).stop 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 shutdown_connections 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effdc06b810 0x7effdc06dcd0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efff01588a0 0x7efff0150c00 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efff01edb40 0x7efff01edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.735+0000 7efff593e640 1 --2- 172.21.15.78:0/3928790864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd8003970 0x7efff0149f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.736+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 >> 172.21.15.78:0/3928790864 conn(0x7efff006c170 msgr2=0x7efff00723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.736+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 shutdown_connections 2024-02-19T14:59:36.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.736+0000 7efff593e640 1 -- 172.21.15.78:0/3928790864 wait complete. 2024-02-19T14:59:36.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:36.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.854+0000 7fc12b7c7640 1 Processor -- start 2024-02-19T14:59:36.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.854+0000 7fc12b7c7640 1 -- start start 2024-02-19T14:59:36.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc12b7c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 0x7fc124156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc12b7c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1240c1dd0 0x7fc1240c2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc12b7c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc124155a50 con 0x7fc124071410 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc12b7c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc124152af0 con 0x7fc1240c1dd0 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc12b7c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc124155f80 con 0x7fc1240c19f0 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc128d62640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc124071410 0x7fc1240c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50058/0 (socket says 172.21.15.78:50058) 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc128d62640 1 -- 172.21.15.78:0/3924112871 learned_addr learned my addr 172.21.15.78:0/3924112871 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537854407 0 0) 0x7fc124155a50 con 0x7fc124071410 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc110003520 con 0x7fc124071410 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc129563640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 0x7fc124156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.855+0000 7fc129d64640 1 --2- 172.21.15.78:0/3924112871 >> v2:172.21.15.150:3300/0 conn(0x7fc1240c1dd0 0x7fc1240c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3850183071 0 0) 0x7fc110003520 con 0x7fc124071410 2024-02-19T14:59:36.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 msgr2=0x7fc124156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 0x7fc124156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 >> v2:172.21.15.150:3300/0 conn(0x7fc1240c1dd0 msgr2=0x7fc1240c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 --2- 172.21.15.78:0/3924112871 >> v2:172.21.15.150:3300/0 conn(0x7fc1240c1dd0 0x7fc1240c2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc124070840 con 0x7fc124071410 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fc114001670 con 0x7fc124071410 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 >> v1:172.21.15.78:6789/0 conn(0x7fc124071410 legacy=0x7fc1240c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 0x7fc110003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110003ef0 0x7fc1100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110004d00 0x7fc110005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124152af0 con 0x7fc110004d00 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124155f80 con 0x7fc110003ef0 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11000ba60 con 0x7fc1240c19f0 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129d64640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110004d00 0x7fc110005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129563640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 0x7fc110003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc128d62640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110003ef0 0x7fc1100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129d64640 1 -- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 msgr2=0x7fc110003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129d64640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1240c19f0 0x7fc110003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129d64640 1 -- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110003ef0 msgr2=0x7fc1100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129d64640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110003ef0 0x7fc1100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.856+0000 7fc129d64640 1 -- 172.21.15.78:0/3924112871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc110003520 con 0x7fc110004d00 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc129d64640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110004d00 0x7fc110005180 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009fc0 tx=0x7fc1100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28cb29ed412f12a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c02cc60 con 0x7fc110004d00 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c00f040 con 0x7fc110004d00 2024-02-19T14:59:36.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc11bfff640 1 -- 172.21.15.78:0/3924112871 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c034440 con 0x7fc110004d00 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3924112871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110004d00 msgr2=0x7fc110005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3924112871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc110004d00 0x7fc110005180 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009fc0 tx=0x7fc1100057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.857+0000 7fc129d64640 1 -- 172.21.15.78:0/3924112871 reap_dead start 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.858+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3924112871 shutdown_connections 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.858+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3924112871 >> 172.21.15.78:0/3924112871 conn(0x7fc1240aa370 msgr2=0x7fc1240aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.858+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3924112871 shutdown_connections 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.858+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3924112871 wait complete. 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 Processor -- start 2024-02-19T14:59:36.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 -- start start 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110004d00 0x7fc1241565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc124150920 0x7fc1240c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 0x7fc1240c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1241550f0 con 0x7fc1240ae300 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124154b50 con 0x7fc110004d00 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc12b7c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1241eeb80 con 0x7fc124150920 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc129d64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 0x7fc1240c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc129d64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 0x7fc1240c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57456/0 (socket says 172.21.15.78:57456) 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.859+0000 7fc129d64640 1 -- 172.21.15.78:0/3528653542 learned_addr learned my addr 172.21.15.78:0/3528653542 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129563640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110004d00 0x7fc1241565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc128d62640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc124150920 0x7fc1240c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129d64640 1 -- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc124150920 msgr2=0x7fc1240c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129d64640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc124150920 0x7fc1240c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129d64640 1 -- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110004d00 msgr2=0x7fc1241565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129d64640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110004d00 0x7fc1241565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129d64640 1 -- 172.21.15.78:0/3528653542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc124154e10 con 0x7fc1240ae300 2024-02-19T14:59:36.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc128d62640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc124150920 0x7fc1240c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc129d64640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 0x7fc1240c3c70 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00e230 tx=0x7fc11c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c043070 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c034b50 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c003990 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc124154110 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.860+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc124154630 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.861+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12406e160 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.862+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc11c03e440 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.862+0000 7fc119ffb640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc10806b810 0x7fc10806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.862+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc11c0abd80 con 0x7fc1240ae300 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.862+0000 7fc129563640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc10806b810 0x7fc10806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:36.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.862+0000 7fc129563640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc10806b810 0x7fc10806dcd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc120009b00 tx=0x7fc120008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:36.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:36.867+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc11c076750 con 0x7fc1240ae300 2024-02-19T14:59:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.077+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1240c8320 con 0x7fc1240ae300 2024-02-19T14:59:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.077+0000 7fc119ffb640 1 -- 172.21.15.78:0/3528653542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+4953 (secure 0 0 0) 0x7fc11c039070 con 0x7fc1240ae300 2024-02-19T14:59:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:37.080 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":171,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8758,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:36.736110+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0],"up":{"mds_0":9816},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:starting","state_seq":1,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 171 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc10806b810 msgr2=0x7fc10806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc10806b810 0x7fc10806dcd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc120009b00 tx=0x7fc120008040 comp rx=0 tx=0).stop 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 msgr2=0x7fc1240c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 0x7fc1240c3c70 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00e230 tx=0x7fc11c009750 comp rx=0 tx=0).stop 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 shutdown_connections 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc10806b810 0x7fc10806dcd0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1240ae300 0x7fc1240c3c70 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc124150920 0x7fc1240c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:37.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 --2- 172.21.15.78:0/3528653542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc110004d00 0x7fc1241565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:37.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 >> 172.21.15.78:0/3528653542 conn(0x7fc1240aa370 msgr2=0x7fc12406cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:37.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 shutdown_connections 2024-02-19T14:59:37.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:37.079+0000 7fc12b7c7640 1 -- 172.21.15.78:0/3528653542 wait complete. 2024-02-19T14:59:37.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 171, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:36.736110+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 9816}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9816': {'gid': 9816, 'name': 'c', 'rank': 0, 'incarnation': 171, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.78:6835/778404633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 778404633}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 778404633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:37.496 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9816:up:starting 2024-02-19T14:59:38.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.596+0000 7f4a7cdbd640 1 Processor -- start 2024-02-19T14:59:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a7cdbd640 1 -- start start 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a7cdbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a7cdbd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a780719e0 0x7f4a780c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a7cdbd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7806ff90 con 0x7f4a780c3d90 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a7cdbd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a78070270 con 0x7f4a780719e0 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a7cdbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780c3970 con 0x7f4a7814d210 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a6ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a780719e0 0x7f4a780c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a6ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a780719e0 0x7f4a780c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35292/0 (socket says 172.21.15.78:35292) 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.597+0000 7f4a6ffff640 1 -- 172.21.15.78:0/3796084625 learned_addr learned my addr 172.21.15.78:0/3796084625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a7659c640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a6ffff640 1 -- 172.21.15.78:0/3796084625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 msgr2=0x7f4a780c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a6ffff640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a6ffff640 1 -- 172.21.15.78:0/3796084625 >> v1:172.21.15.150:6789/0 conn(0x7f4a780c3d90 legacy=0x7f4a78152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a6ffff640 1 -- 172.21.15.78:0/3796084625 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a78156fc0 con 0x7f4a780719e0 2024-02-19T14:59:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a7659c640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.598+0000 7f4a6ffff640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780719e0 0x7f4a780c64d0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009870 tx=0x7f4a6002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835200496bdbc258 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a75d9b640 1 -- 172.21.15.78:0/3796084625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600317c0 con 0x7f4a780719e0 2024-02-19T14:59:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a75d9b640 1 -- 172.21.15.78:0/3796084625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60031920 con 0x7f4a780719e0 2024-02-19T14:59:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a75d9b640 1 -- 172.21.15.78:0/3796084625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60031cb0 con 0x7f4a780719e0 2024-02-19T14:59:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/3796084625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780719e0 msgr2=0x7f4a780c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780719e0 0x7f4a780c64d0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009870 tx=0x7f4a6002b860 comp rx=0 tx=0).stop 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/3796084625 shutdown_connections 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780719e0 0x7f4a780c64d0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/3796084625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/3796084625 >> 172.21.15.78:0/3796084625 conn(0x7f4a780c5050 msgr2=0x7f4a780c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/3796084625 shutdown_connections 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.599+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/3796084625 wait complete. 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.600+0000 7f4a7cdbd640 1 Processor -- start 2024-02-19T14:59:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.600+0000 7f4a7cdbd640 1 -- start start 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7cdbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a780719e0 0x7f4a78155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7cdbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 0x7f4a780c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7cdbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7cdbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a781de450 con 0x7f4a780c3d90 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7cdbd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780ad3c0 con 0x7f4a780719e0 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7cdbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780adb90 con 0x7f4a7814d210 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 0x7f4a780c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 0x7f4a780c1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35304/0 (socket says 172.21.15.78:35304) 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 -- 172.21.15.78:0/2427334265 learned_addr learned my addr 172.21.15.78:0/2427334265 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a76d9d640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7659c640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a780719e0 0x7f4a78155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 -- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 msgr2=0x7f4a780c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 -- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a780719e0 msgr2=0x7f4a78155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a780719e0 0x7f4a78155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a6ffff640 1 -- 172.21.15.78:0/2427334265 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a600094d0 con 0x7f4a780c3d90 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a76d9d640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.601+0000 7f4a7659c640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a780719e0 0x7f4a78155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.602+0000 7f4a6ffff640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 0x7f4a780c1b70 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f4a60000c00 tx=0x7f4a600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.602+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600329a0 con 0x7f4a780c3d90 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.602+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60032b20 con 0x7f4a780c3d90 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.602+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600453f0 con 0x7f4a780c3d90 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.602+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a780c3660 con 0x7f4a780c3d90 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.602+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a780ac810 con 0x7f4a780c3d90 2024-02-19T14:59:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.603+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a78074990 con 0x7f4a780c3d90 2024-02-19T14:59:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.603+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4a6004d3f0 con 0x7f4a780c3d90 2024-02-19T14:59:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.604+0000 7f4a6effd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a5806b810 0x7f4a5806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.604+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a600ac270 con 0x7f4a780c3d90 2024-02-19T14:59:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.604+0000 7f4a7659c640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a5806b810 0x7f4a5806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.604+0000 7f4a7659c640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a5806b810 0x7f4a5806dcd0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4a68006fd0 tx=0x7f4a68008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.608+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a60076b90 con 0x7f4a780c3d90 2024-02-19T14:59:38.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.849+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a780c3100 con 0x7f4a780c3d90 2024-02-19T14:59:38.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.849+0000 7f4a6effd640 1 -- 172.21.15.78:0/2427334265 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4966 (secure 0 0 0) 0x7f4a6007a250 con 0x7f4a780c3d90 2024-02-19T14:59:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:38.852 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:37.764041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9816,"mds_1":8758},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8758":{"gid":8758,"name":"d","rank":1,"incarnation":173,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:38.853 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 174 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a5806b810 msgr2=0x7f4a5806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a5806b810 0x7f4a5806dcd0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4a68006fd0 tx=0x7f4a68008040 comp rx=0 tx=0).stop 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 msgr2=0x7f4a780c1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 0x7f4a780c1b70 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f4a60000c00 tx=0x7f4a600046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 shutdown_connections 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4a5806b810 0x7f4a5806dcd0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a780c3d90 0x7f4a780c1b70 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a7814d210 0x7f4a780c20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 --2- 172.21.15.78:0/2427334265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a780719e0 0x7f4a78155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:38.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 >> 172.21.15.78:0/2427334265 conn(0x7f4a780c5050 msgr2=0x7f4a7814a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:38.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 shutdown_connections 2024-02-19T14:59:38.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:38.851+0000 7f4a7cdbd640 1 -- 172.21.15.78:0/2427334265 wait complete. 2024-02-19T14:59:38.902 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:59:38.899+0000 7f9d05a5a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:59:38.902 INFO:tasks.ceph.mds.c.smithi078.stderr:2024-02-19T14:59:38.899+0000 7f9d05a5a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-19T14:59:39.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 174, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:37.764041+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9816, 'mds_1': 8758}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8758': {'gid': 8758, 'name': 'd', 'rank': 1, 'incarnation': 173, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/2126014034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2126014034}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2126014034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9816': {'gid': 9816, 'name': 'c', 'rank': 0, 'incarnation': 171, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.78:6835/778404633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 778404633}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 778404633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:39.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-02-19T14:59:39.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:59:39.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.368+0000 7fddffeda640 1 Processor -- start 2024-02-19T14:59:39.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.368+0000 7fddffeda640 1 -- start start 2024-02-19T14:59:39.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddffeda640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fddf8152700 0x7fddf80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddffeda640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fddf80c5730 0x7fddf806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddffeda640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddffeda640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf806cd20 con 0x7fddf8152700 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddffeda640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf81551f0 con 0x7fddf80c5730 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddffeda640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8155490 con 0x7fddf80c8e00 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddfdc76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fddf8152700 0x7fddf80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddfdc76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fddf8152700 0x7fddf80c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35308/0 (socket says 172.21.15.78:35308) 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddfd475640 1 --2- 172.21.15.78:0/1477603083 >> v2:172.21.15.150:3300/0 conn(0x7fddf80c5730 0x7fddf806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddfe477640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.369+0000 7fddfdc76640 1 -- 172.21.15.78:0/1477603083 learned_addr learned my addr 172.21.15.78:0/1477603083 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfdc76640 1 -- 172.21.15.78:0/1477603083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 msgr2=0x7fddf806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfdc76640 1 -- 172.21.15.78:0/1477603083 >> v2:172.21.15.150:3300/0 conn(0x7fddf80c5730 msgr2=0x7fddf806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1477603083 >> v2:172.21.15.150:3300/0 conn(0x7fddf80c5730 0x7fddf806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfdc76640 1 -- 172.21.15.78:0/1477603083 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8155770 con 0x7fddf8152700 2024-02-19T14:59:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfd475640 1 --2- 172.21.15.78:0/1477603083 >> v2:172.21.15.150:3300/0 conn(0x7fddf80c5730 0x7fddf806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfe477640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.370+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c51f0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fddf80c7620 tx=0x7fddf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc6a6e256d38dd0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddfcc74640 1 -- 172.21.15.78:0/1477603083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4031870 con 0x7fddf8152700 2024-02-19T14:59:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddfcc74640 1 -- 172.21.15.78:0/1477603083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf40319d0 con 0x7fddf8152700 2024-02-19T14:59:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddfcc74640 1 -- 172.21.15.78:0/1477603083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4031d60 con 0x7fddf8152700 2024-02-19T14:59:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 -- 172.21.15.78:0/1477603083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 msgr2=0x7fddf80c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c51f0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fddf80c7620 tx=0x7fddf402b910 comp rx=0 tx=0).stop 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 -- 172.21.15.78:0/1477603083 shutdown_connections 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 --2- 172.21.15.78:0/1477603083 >> v2:172.21.15.150:3300/0 conn(0x7fddf80c5730 0x7fddf806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 --2- 172.21.15.78:0/1477603083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c51f0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 -- 172.21.15.78:0/1477603083 >> 172.21.15.78:0/1477603083 conn(0x7fddf80cac90 msgr2=0x7fddf80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 -- 172.21.15.78:0/1477603083 shutdown_connections 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.371+0000 7fddffeda640 1 -- 172.21.15.78:0/1477603083 wait complete. 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.372+0000 7fddffeda640 1 Processor -- start 2024-02-19T14:59:39.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.372+0000 7fddffeda640 1 -- start start 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddffeda640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf80c5730 0x7fddf80ac570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddffeda640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf80acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddffeda640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddffeda640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80b00e0 con 0x7fddf8152700 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddffeda640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c23e0 con 0x7fddf80c5730 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddffeda640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c48f0 con 0x7fddf80c8e00 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35312/0 (socket says 172.21.15.78:35312) 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 -- 172.21.15.78:0/633359247 learned_addr learned my addr 172.21.15.78:0/633359247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfdc76640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf80c5730 0x7fddf80ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfd475640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf80acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 -- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 msgr2=0x7fddf80acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf80acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 -- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf80c5730 msgr2=0x7fddf80ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf80c5730 0x7fddf80ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfe477640 1 -- 172.21.15.78:0/633359247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec009400 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.373+0000 7fddfdc76640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf80c5730 0x7fddf80ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.374+0000 7fddfe477640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c2830 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fddf000efc0 tx=0x7fddf0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.374+0000 7fddea7fc640 1 -- 172.21.15.78:0/633359247 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0009d70 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.374+0000 7fddea7fc640 1 -- 172.21.15.78:0/633359247 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf000f040 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.374+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0009a70 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.374+0000 7fddea7fc640 1 -- 172.21.15.78:0/633359247 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0003990 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.374+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf40094d0 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.375+0000 7fddea7fc640 1 -- 172.21.15.78:0/633359247 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fddf00144a0 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.376+0000 7fddea7fc640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdddc06b840 0x7fdddc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.376+0000 7fddea7fc640 1 -- 172.21.15.78:0/633359247 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fddf008ae20 con 0x7fddf8152700 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.376+0000 7fddfdc76640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdddc06b840 0x7fdddc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.376+0000 7fddfdc76640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdddc06b840 0x7fdddc06dd00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fddf4036000 tx=0x7fddf403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.377+0000 7fde02b8b740 1 Processor -- start 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.377+0000 7fde02b8b740 1 -- start start 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.377+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.377+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de07ae0 0x55cd6de41b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.377+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de06860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fde02b8b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6ddc1cd0 con 0x55cd6de448b0 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6dd74f20 con 0x55cd6de07ae0 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd6dd72450 con 0x55cd6de41740 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de06860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de06860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35328/0 (socket says 172.21.15.78:35328) 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 -- 172.21.15.78:0/2067689923 learned_addr learned my addr 172.21.15.78:0/2067689923 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfd475640 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de07ae0 0x55cd6de41b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfdc76640 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 -- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 msgr2=0x55cd6de428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 -- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de07ae0 msgr2=0x55cd6de41b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de07ae0 0x55cd6de41b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 -- 172.21.15.78:0/2067689923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd6dd22f20 con 0x55cd6de448b0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fddfe477640 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de06860 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fddf00615f0 tx=0x7fddf0059640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a9f623f0163e64 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fdddb7fe640 1 -- 172.21.15.78:0/2067689923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0055b30 con 0x55cd6de448b0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fdddb7fe640 1 -- 172.21.15.78:0/2067689923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf0055c90 con 0x55cd6de448b0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.378+0000 7fdddb7fe640 1 -- 172.21.15.78:0/2067689923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf005d2b0 con 0x55cd6de448b0 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 -- 172.21.15.78:0/2067689923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 msgr2=0x55cd6de06860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de06860 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fddf00615f0 tx=0x7fddf0059640 comp rx=0 tx=0).stop 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 -- 172.21.15.78:0/2067689923 shutdown_connections 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de06860 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de07ae0 0x55cd6de41b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2067689923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 -- 172.21.15.78:0/2067689923 >> 172.21.15.78:0/2067689923 conn(0x55cd6de03120 msgr2=0x55cd6de27b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 -- 172.21.15.78:0/2067689923 shutdown_connections 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.379+0000 7fde02b8b740 1 -- 172.21.15.78:0/2067689923 wait complete. 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.380+0000 7fde02b8b740 1 -- start start 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.380+0000 7fde02b8b740 1 Processor -- start 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de27020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de52e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de3d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6dd6dc30 con 0x55cd6de448b0 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6dd12d30 con 0x55cd6de3f610 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd6da41310 con 0x55cd6de41740 2024-02-19T14:59:39.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 10 client.0 _ll_register_callbacks cb 0x55cd6dd1ad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de52e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de52e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35340/0 (socket says 172.21.15.78:35340) 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 -- 172.21.15.78:0/1943660549 learned_addr learned my addr 172.21.15.78:0/1943660549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de27020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfe477640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de3d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 -- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 msgr2=0x55cd6de27020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de27020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 -- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 msgr2=0x55cd6de3d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de3d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 -- 172.21.15.78:0/1943660549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf40094d0 con 0x55cd6de448b0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddfd475640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de52e80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fddec0149d0 tx=0x7fddec014dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368763f70b4b05b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddd8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1943660549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec012b80 con 0x55cd6de448b0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1943660549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fddec012ce0 con 0x55cd6de448b0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd6da6d6c0 con 0x55cd6de448b0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1943660549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddec012ea0 con 0x55cd6de448b0 2024-02-19T14:59:39.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 10 client.10107 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.381+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x55cd6dd6c2b0 con 0x55cd6de448b0 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1943660549 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cd6da6d6c0 con 0x55cd6de448b0 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1943660549 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fddec011040 con 0x55cd6de448b0 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fde02b8b740 10 client.10107 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fde02b8b740 10 client.10107 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fde02b8b740 20 client.10107 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de39c30 0x55cd6de3a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fde02b8b740 4 client.10107 mds_command: new command op to 8758 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 --> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd6dabe270 con 0x55cd6de39c30 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.382+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de39c30 0x55cd6de3a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.383+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de39c30 0x55cd6de3a010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bbda788e18da9293 server_cookie=74385e81f0a99709 in_seq=0 out_seq=0 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.383+0000 7fddd8ff9640 10 client.10107 ms_handle_connect on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.383+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1943660549 <== mds.1 v2:172.21.15.150:6833/2126014034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cd6dabe270 con 0x55cd6de39c30 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.383+0000 7fddd8ff9640 10 client.10107 handle_command_reply: tid=0 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.383+0000 7fde02b8b740 1 client.10107 shutdown 2024-02-19T14:59:39.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 msgr2=0x55cd6de52e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de52e80 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fddec0149d0 tx=0x7fddec014dd0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 shutdown_connections 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de52e80 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de39c30 0x55cd6de3a010 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de3d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1943660549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de27020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fddd8ff9640 0 client.10107 ms_handle_reset on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 >> 172.21.15.78:0/1943660549 conn(0x55cd6de07ae0 msgr2=0x55cd6de01f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 shutdown_connections 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 1 -- 172.21.15.78:0/1943660549 wait complete. 2024-02-19T14:59:39.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.384+0000 7fde02b8b740 20 client.10107 trim_cache size 0 max 16384 2024-02-19T14:59:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.400+0000 7fde02b8b740 1 Processor -- start 2024-02-19T14:59:39.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.400+0000 7fde02b8b740 1 -- start start 2024-02-19T14:59:39.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de42e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de3d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de3d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fde02b8b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6dcf3490 con 0x55cd6de448b0 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6de668c0 con 0x55cd6de3f610 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd6dd9f2e0 con 0x55cd6de41740 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de3d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de3d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35350/0 (socket says 172.21.15.78:35350) 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 -- 172.21.15.78:0/2310011474 learned_addr learned my addr 172.21.15.78:0/2310011474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfd475640 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de3d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfdc76640 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de42e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 -- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 msgr2=0x55cd6de3d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de3d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 -- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 msgr2=0x55cd6de42e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de42e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.401+0000 7fddfe477640 1 -- 172.21.15.78:0/2310011474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd6dd08a20 con 0x55cd6de448b0 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fddfe477640 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de3d910 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fddf0058f90 tx=0x7fddf005b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcc35e2c89d8a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fdddaffd640 1 -- 172.21.15.78:0/2310011474 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0055b30 con 0x55cd6de448b0 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fdddaffd640 1 -- 172.21.15.78:0/2310011474 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf005d0b0 con 0x55cd6de448b0 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fdddaffd640 1 -- 172.21.15.78:0/2310011474 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0058910 con 0x55cd6de448b0 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 -- 172.21.15.78:0/2310011474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 msgr2=0x55cd6de3d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de3d910 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fddf0058f90 tx=0x7fddf005b000 comp rx=0 tx=0).stop 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 -- 172.21.15.78:0/2310011474 shutdown_connections 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de448b0 0x55cd6de3d910 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de41740 0x55cd6de3d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 --2- 172.21.15.78:0/2310011474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6de42e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 -- 172.21.15.78:0/2310011474 >> 172.21.15.78:0/2310011474 conn(0x55cd6de07ae0 msgr2=0x55cd6de43af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 -- 172.21.15.78:0/2310011474 shutdown_connections 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.402+0000 7fde02b8b740 1 -- 172.21.15.78:0/2310011474 wait complete. 2024-02-19T14:59:39.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.403+0000 7fde02b8b740 1 -- start start 2024-02-19T14:59:39.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.403+0000 7fde02b8b740 1 Processor -- start 2024-02-19T14:59:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6deaf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 0x55cd6de3c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de448b0 0x55cd6de55eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6de02920 con 0x55cd6de41740 2024-02-19T14:59:39.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd6de43580 con 0x55cd6de3f610 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd6de3cdc0 con 0x55cd6de448b0 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 10 client.0 _ll_register_callbacks cb 0x7fddec009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 0x55cd6de3c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 0x55cd6de3c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35356/0 (socket says 172.21.15.78:35356) 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 -- 172.21.15.78:0/1662871924 learned_addr learned my addr 172.21.15.78:0/1662871924 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6deaf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfe477640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de448b0 0x55cd6de55eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 -- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de448b0 msgr2=0x55cd6de55eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de448b0 0x55cd6de55eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 -- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 msgr2=0x55cd6deaf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6deaf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 -- 172.21.15.78:0/1662871924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd6dd22f20 con 0x55cd6de41740 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddfd475640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 0x55cd6de3c280 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x55cd6de41520 tx=0x7fddec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a3c41cc2b5057a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddd8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1662871924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec012d80 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1662871924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fddec005990 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.404+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddf0055c90 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1662871924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddec0147d0 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 10 client.10113 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x55cd6dd2e780 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1662871924 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf0055c90 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1662871924 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fddec011040 con 0x55cd6de41740 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 10 client.10113 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 10 client.10113 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 20 client.10113 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de563f0 0x55cd6deb5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 4 client.10113 mds_command: new command op to 8758 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 --> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cd6daaa610 con 0x55cd6de563f0 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de563f0 0x55cd6deb5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fddd8ff9640 10 client.10113 ms_handle_connect on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.405+0000 7fddfdc76640 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de563f0 0x55cd6deb5450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0705e9c80e93781 server_cookie=3bd1b0ba980d87b7 in_seq=0 out_seq=0 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddd8ff9640 1 -- 172.21.15.78:0/1662871924 <== mds.1 v2:172.21.15.150:6833/2126014034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55cd6daaa610 con 0x55cd6de563f0 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddd8ff9640 10 client.10113 handle_command_reply: tid=0 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 client.10113 shutdown 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 msgr2=0x55cd6de3c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 0x55cd6de3c280 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x55cd6de41520 tx=0x7fddec0046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 shutdown_connections 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cd6de41740 0x55cd6de3c280 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x55cd6de563f0 0x55cd6deb5450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55cd6de448b0 0x55cd6de55eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 --2- 172.21.15.78:0/1662871924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cd6de3f610 0x55cd6deaf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddd8ff9640 0 client.10113 ms_handle_reset on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 >> 172.21.15.78:0/1662871924 conn(0x55cd6de07ae0 msgr2=0x55cd6de24950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 shutdown_connections 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 1 -- 172.21.15.78:0/1662871924 wait complete. 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fde02b8b740 20 client.10113 trim_cache size 0 max 16384 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 8758, 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 1.657744254, 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 174, 2024-02-19T14:59:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 188, 2024-02-19T14:59:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 188, 2024-02-19T14:59:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 5.6588472620000001 2024-02-19T14:59:39.416 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdddc06b840 msgr2=0x7fdddc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdddc06b840 0x7fdddc06dd00 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fddf4036000 tx=0x7fddf403c040 comp rx=0 tx=0).stop 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 msgr2=0x7fddf80c2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c2830 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fddf000efc0 tx=0x7fddf0009f90 comp rx=0 tx=0).stop 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 shutdown_connections 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdddc06b840 0x7fdddc06dd00 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fddf8152700 0x7fddf80c2830 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddf80c8e00 0x7fddf80acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 --2- 172.21.15.78:0/633359247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf80c5730 0x7fddf80ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.406+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 >> 172.21.15.78:0/633359247 conn(0x7fddf80cac90 msgr2=0x7fddf814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.407+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 shutdown_connections 2024-02-19T14:59:39.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.407+0000 7fddffeda640 1 -- 172.21.15.78:0/633359247 wait complete. 2024-02-19T14:59:39.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:59:39.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 Processor -- start 2024-02-19T14:59:39.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 -- start start 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c78a0 0x7f57cc0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57cc072ee0 0x7f57cc14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc14f8f0 con 0x7f57cc14ef50 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57cc0c35c0 con 0x7f57cc072ee0 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.519+0000 7f57d20d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc0c38a0 con 0x7f57cc0c78a0 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57cc072ee0 0x7f57cc14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57cc072ee0 0x7f57cc14f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35362/0 (socket says 172.21.15.78:35362) 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 -- 172.21.15.78:0/2382735264 learned_addr learned my addr 172.21.15.78:0/2382735264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c78a0 0x7f57cc0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57caffd640 1 --1- 172.21.15.78:0/2382735264 >> v1:172.21.15.150:6789/0 conn(0x7f57cc14ef50 0x7f57cc0727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51712/0 (socket says 172.21.15.78:51712) 2024-02-19T14:59:39.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 -- 172.21.15.78:0/2382735264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c78a0 msgr2=0x7f57cc0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 --2- 172.21.15.78:0/2382735264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c78a0 0x7f57cc0c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 -- 172.21.15.78:0/2382735264 >> v1:172.21.15.150:6789/0 conn(0x7f57cc14ef50 legacy=0x7f57cc0727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57cbfff640 1 -- 172.21.15.78:0/2382735264 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc0c3b80 con 0x7f57cc072ee0 2024-02-19T14:59:39.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.520+0000 7f57ca7fc640 1 -- 172.21.15.78:0/2382735264 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041566743 0 0) 0x7f57cc14f8f0 con 0x7f57cc14ef50 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.521+0000 7f57cbfff640 1 --2- 172.21.15.78:0/2382735264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc14f3b0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009e50 tx=0x7f57c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a230444594f9bf2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.521+0000 7f57ca7fc640 1 -- 172.21.15.78:0/2382735264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c402cb00 con 0x7f57cc072ee0 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.521+0000 7f57ca7fc640 1 -- 172.21.15.78:0/2382735264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c402cc60 con 0x7f57cc072ee0 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.521+0000 7f57ca7fc640 1 -- 172.21.15.78:0/2382735264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c40343f0 con 0x7f57cc072ee0 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 -- 172.21.15.78:0/2382735264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 msgr2=0x7f57cc14f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 --2- 172.21.15.78:0/2382735264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc14f3b0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009e50 tx=0x7f57c402bf60 comp rx=0 tx=0).stop 2024-02-19T14:59:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 -- 172.21.15.78:0/2382735264 shutdown_connections 2024-02-19T14:59:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 --2- 172.21.15.78:0/2382735264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc14f3b0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 --2- 172.21.15.78:0/2382735264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c78a0 0x7f57cc0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 -- 172.21.15.78:0/2382735264 >> 172.21.15.78:0/2382735264 conn(0x7f57cc066e70 msgr2=0x7f57cc0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 -- 172.21.15.78:0/2382735264 shutdown_connections 2024-02-19T14:59:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.522+0000 7f57d20d4640 1 -- 172.21.15.78:0/2382735264 wait complete. 2024-02-19T14:59:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.523+0000 7f57d20d4640 1 Processor -- start 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.523+0000 7f57d20d4640 1 -- start start 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc1513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c78a0 0x7f57cc1d7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc14ef50 0x7f57cc13dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1d8a30 con 0x7f57cc072ee0 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1ea460 con 0x7f57cc0c78a0 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1ea740 con 0x7f57cc14ef50 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc1513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc1513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35372/0 (socket says 172.21.15.78:35372) 2024-02-19T14:59:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 -- 172.21.15.78:0/3664229803 learned_addr learned my addr 172.21.15.78:0/3664229803 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cbfff640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc14ef50 0x7f57cc13dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 -- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc14ef50 msgr2=0x7f57cc13dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc14ef50 0x7f57cc13dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57caffd640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c78a0 0x7f57cc1d7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 -- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c78a0 msgr2=0x7f57cc1d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c78a0 0x7f57cc1d7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 -- 172.21.15.78:0/3664229803 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4009b30 con 0x7f57cc072ee0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57caffd640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c78a0 0x7f57cc1d7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc1513e0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f57bc010bc0 tx=0x7f57bc00e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57b3fff640 1 -- 172.21.15.78:0/3664229803 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc009670 con 0x7f57cc072ee0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57b3fff640 1 -- 172.21.15.78:0/3664229803 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc011650 con 0x7f57cc072ee0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc1eaa20 con 0x7f57cc072ee0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.524+0000 7f57b3fff640 1 -- 172.21.15.78:0/3664229803 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc011a00 con 0x7f57cc072ee0 2024-02-19T14:59:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.525+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc151920 con 0x7f57cc072ee0 2024-02-19T14:59:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.526+0000 7f57b3fff640 1 -- 172.21.15.78:0/3664229803 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f57bc011b60 con 0x7f57cc072ee0 2024-02-19T14:59:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.527+0000 7f57b3fff640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b806b810 0x7f57b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.527+0000 7f57caffd640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b806b810 0x7f57b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.527+0000 7f57b3fff640 1 -- 172.21.15.78:0/3664229803 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f57bc08c750 con 0x7f57cc072ee0 2024-02-19T14:59:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.527+0000 7f57d4d86740 1 Processor -- start 2024-02-19T14:59:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.527+0000 7f57caffd640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b806b810 0x7f57b806dcd0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f57cc14dfd0 tx=0x7f57c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.527+0000 7f57d4d86740 1 -- start start 2024-02-19T14:59:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567983580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 0x55b56795bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57d4d86740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56791ccd0 con 0x55b567961cb0 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5678cff20 con 0x55b567994140 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5678cd450 con 0x55b567994520 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57cbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 0x55b56795bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57cbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 0x55b56795bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35380/0 (socket says 172.21.15.78:35380) 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57cbfff640 1 -- 172.21.15.78:0/2022203521 learned_addr learned my addr 172.21.15.78:0/2022203521 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567983580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.528+0000 7f57caffd640 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57cbfff640 1 -- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 msgr2=0x55b56799f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57cbfff640 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57cbfff640 1 -- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 msgr2=0x55b567983580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57cbfff640 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567983580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57cbfff640 1 -- 172.21.15.78:0/2022203521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b56787df20 con 0x55b567961cb0 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57cbfff640 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 0x55b56795bfd0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f57c4036040 tx=0x7f57c402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5832857ed8ebf4c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57b17fa640 1 -- 172.21.15.78:0/2022203521 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c402c980 con 0x55b567961cb0 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57b17fa640 1 -- 172.21.15.78:0/2022203521 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c402cae0 con 0x55b567961cb0 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.529+0000 7f57b17fa640 1 -- 172.21.15.78:0/2022203521 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c403f660 con 0x55b567961cb0 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 -- 172.21.15.78:0/2022203521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 msgr2=0x55b56795bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 0x55b56795bfd0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f57c4036040 tx=0x7f57c402e000 comp rx=0 tx=0).stop 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 -- 172.21.15.78:0/2022203521 shutdown_connections 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567961cb0 0x55b56795bfd0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 --2- 172.21.15.78:0/2022203521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567983580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 -- 172.21.15.78:0/2022203521 >> 172.21.15.78:0/2022203521 conn(0x55b56795e120 msgr2=0x55b567982b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 -- 172.21.15.78:0/2022203521 shutdown_connections 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 -- 172.21.15.78:0/2022203521 wait complete. 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 -- start start 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.530+0000 7f57d4d86740 1 Processor -- start 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 0x55b5679603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567981b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567986a00 0x55b567985be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5678c8c30 con 0x55b567994140 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56786dd30 con 0x55b567994520 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b56759c310 con 0x55b567986a00 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57d4d86740 10 client.0 _ll_register_callbacks cb 0x55b567875d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 0x55b5679603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 0x55b5679603f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35382/0 (socket says 172.21.15.78:35382) 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1653626435 learned_addr learned my addr 172.21.15.78:0/1653626435 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57caffd640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567981b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cbfff640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567986a00 0x55b567985be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567986a00 msgr2=0x55b567985be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567986a00 0x55b567985be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 msgr2=0x55b567981b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567981b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.531+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1653626435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc0c3b80 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 0x55b5679603f0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f57bc05ab80 tx=0x7f57bc0571a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c4408fd9c8875 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57aa7fc640 1 -- 172.21.15.78:0/1653626435 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0634e0 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57aa7fc640 1 -- 172.21.15.78:0/1653626435 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f57bc05fe40 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b567619270 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57aa7fc640 1 -- 172.21.15.78:0/1653626435 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57bc011e10 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 10 client.10125 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x55b5678c72b0 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57aa7fc640 1 -- 172.21.15.78:0/1653626435 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc076020 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57aa7fc640 1 -- 172.21.15.78:0/1653626435 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f57bc063050 con 0x55b567994140 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 10 client.10125 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 10 client.10125 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 20 client.10125 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56797fe80 0x55b567980260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 4 client.10125 mds_command: new command op to 9816 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 --> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b567605610 con 0x55b56797fe80 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.532+0000 7f57caffd640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56797fe80 0x55b567980260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.533+0000 7f57caffd640 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56797fe80 0x55b567980260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c9accaadb7d0633 server_cookie=2ce9e925adec46b5 in_seq=0 out_seq=0 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.533+0000 7f57aa7fc640 10 client.10125 ms_handle_connect on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57aa7fc640 1 -- 172.21.15.78:0/1653626435 <== mds.0 v2:172.21.15.78:6834/778404633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b567605610 con 0x55b56797fe80 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57aa7fc640 10 client.10125 handle_command_reply: tid=0 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 client.10125 shutdown 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 msgr2=0x55b5679603f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 0x55b5679603f0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f57bc05ab80 tx=0x7f57bc0571a0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 shutdown_connections 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56797fe80 0x55b567980260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567986a00 0x55b567985be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567981b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1653626435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567994140 0x55b5679603f0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 >> 172.21.15.78:0/1653626435 conn(0x55b567961cb0 msgr2=0x55b5679ab720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57aa7fc640 0 client.10125 ms_handle_reset on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:39.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 shutdown_connections 2024-02-19T14:59:39.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 1 -- 172.21.15.78:0/1653626435 wait complete. 2024-02-19T14:59:39.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.534+0000 7f57d4d86740 20 client.10125 trim_cache size 0 max 16384 2024-02-19T14:59:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.550+0000 7f57d4d86740 1 Processor -- start 2024-02-19T14:59:39.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.550+0000 7f57d4d86740 1 -- start start 2024-02-19T14:59:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b567962ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567961490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57d4d86740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56784e490 con 0x55b567986a00 2024-02-19T14:59:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b56795d4e0 con 0x55b567994140 2024-02-19T14:59:39.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b567981850 con 0x55b567994520 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b567962ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b567962ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35398/0 (socket says 172.21.15.78:35398) 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1497200960 learned_addr learned my addr 172.21.15.78:0/1497200960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cbfff640 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 msgr2=0x55b56799b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 msgr2=0x55b567961490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.551+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567961490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.552+0000 7f57cb7fe640 1 -- 172.21.15.78:0/1497200960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5679b6a60 con 0x55b567986a00 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.552+0000 7f57cbfff640 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.552+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b567962ae0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f57bc010b90 tx=0x7f57bc05d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efffd60b4a8a4f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.552+0000 7f57b0ff9640 1 -- 172.21.15.78:0/1497200960 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0634e0 con 0x55b567986a00 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.552+0000 7f57b0ff9640 1 -- 172.21.15.78:0/1497200960 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc076070 con 0x55b567986a00 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.552+0000 7f57b0ff9640 1 -- 172.21.15.78:0/1497200960 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc011e10 con 0x55b567986a00 2024-02-19T14:59:39.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 -- 172.21.15.78:0/1497200960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 msgr2=0x55b567962ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b567962ae0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f57bc010b90 tx=0x7f57bc05d000 comp rx=0 tx=0).stop 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 -- 172.21.15.78:0/1497200960 shutdown_connections 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994520 0x55b56799b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994140 0x55b567961490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 --2- 172.21.15.78:0/1497200960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b567962ae0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 -- 172.21.15.78:0/1497200960 >> 172.21.15.78:0/1497200960 conn(0x55b567961cb0 msgr2=0x55b567960200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 -- 172.21.15.78:0/1497200960 shutdown_connections 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 -- 172.21.15.78:0/1497200960 wait complete. 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 -- start start 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.553+0000 7f57d4d86740 1 Processor -- start 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b5679c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994140 0x55b567a16090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567a13390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567a16b10 con 0x55b567986a00 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5679a0f10 con 0x55b567994520 2024-02-19T14:59:39.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5679a14d0 con 0x55b567994140 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57d4d86740 10 client.0 _ll_register_callbacks cb 0x7f57bc0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b5679c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b5679c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35412/0 (socket says 172.21.15.78:35412) 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 -- 172.21.15.78:0/558995185 learned_addr learned my addr 172.21.15.78:0/558995185 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cbfff640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567a13390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57caffd640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994140 0x55b567a16090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 -- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994140 msgr2=0x55b567a16090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994140 0x55b567a16090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 -- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 msgr2=0x55b567a13390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567a13390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cb7fe640 1 -- 172.21.15.78:0/558995185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5678c72b0 con 0x55b567986a00 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57cbfff640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567a13390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.554+0000 7f57caffd640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994140 0x55b567a16090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:39.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57cb7fe640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b5679c9420 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00ec30 tx=0x7f57bc08aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abdac7a6a16aa719 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57aa7fc640 1 -- 172.21.15.78:0/558995185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc011e10 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57aa7fc640 1 -- 172.21.15.78:0/558995185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f57bc05fe40 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5675c86c0 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57aa7fc640 1 -- 172.21.15.78:0/558995185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57bc072b60 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 10 client.10131 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x55b567863a20 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57aa7fc640 1 -- 172.21.15.78:0/558995185 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b5675c86c0 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57aa7fc640 1 -- 172.21.15.78:0/558995185 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f57bc072cc0 con 0x55b567986a00 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 10 client.10131 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 10 client.10131 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 20 client.10131 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56795fff0 0x55b567a138d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 4 client.10131 mds_command: new command op to 9816 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:59:39.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 --> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5675745e0 con 0x55b56795fff0 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.555+0000 7f57caffd640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56795fff0 0x55b567a138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57caffd640 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56795fff0 0x55b567a138d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb81a9c38d624d3 server_cookie=b761e18c18b89fc3 in_seq=0 out_seq=0 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57aa7fc640 10 client.10131 ms_handle_connect on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57aa7fc640 1 -- 172.21.15.78:0/558995185 <== mds.0 v2:172.21.15.78:6834/778404633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55b5675745e0 con 0x55b56795fff0 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57aa7fc640 10 client.10131 handle_command_reply: tid=0 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 client.10131 shutdown 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 msgr2=0x55b5679c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b5679c9420 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00ec30 tx=0x7f57bc08aa50 comp rx=0 tx=0).stop 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 shutdown_connections 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x55b56795fff0 0x55b567a138d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b567994520 0x55b567a13390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55b567994140 0x55b567a16090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57aa7fc640 0 client.10131 ms_handle_reset on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 --2- 172.21.15.78:0/558995185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b567986a00 0x55b5679c9420 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 >> 172.21.15.78:0/558995185 conn(0x55b567961cb0 msgr2=0x55b56799f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 shutdown_connections 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 1 -- 172.21.15.78:0/558995185 wait complete. 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.556+0000 7f57d4d86740 20 client.10131 trim_cache size 0 max 16384 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 9816, 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:59:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:59:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 2.8174532160000001, 2024-02-19T14:59:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 174, 2024-02-19T14:59:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 188, 2024-02-19T14:59:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 188, 2024-02-19T14:59:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 36.051695766000002 2024-02-19T14:59:39.567 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b806b810 msgr2=0x7f57b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b806b810 0x7f57b806dcd0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f57cc14dfd0 tx=0x7f57c0009040 comp rx=0 tx=0).stop 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 msgr2=0x7f57cc1513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc1513e0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f57bc010bc0 tx=0x7f57bc00e730 comp rx=0 tx=0).stop 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 shutdown_connections 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b806b810 0x7f57b806dcd0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc14ef50 0x7f57cc13dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c78a0 0x7f57cc1d7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 --2- 172.21.15.78:0/3664229803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc072ee0 0x7f57cc1513e0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 >> 172.21.15.78:0/3664229803 conn(0x7f57cc066e70 msgr2=0x7f57cc150240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 shutdown_connections 2024-02-19T14:59:39.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.557+0000 7f57d20d4640 1 -- 172.21.15.78:0/3664229803 wait complete. 2024-02-19T14:59:39.570 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:39.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.673+0000 7fc5ddedf640 1 Processor -- start 2024-02-19T14:59:39.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.673+0000 7fc5ddedf640 1 -- start start 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ddedf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5d80c7070 0x7fc5d81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ddedf640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8155ef0 con 0x7fc5d8152480 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ddedf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d806dec0 con 0x7fc5d80c87a0 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ddedf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d806d520 con 0x7fc5d80c7070 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5d77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc5d8152480 0x7fc5d806ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39448/0 (socket says 172.21.15.78:39448) 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5d77fe640 1 -- 172.21.15.78:0/1682950160 learned_addr learned my addr 172.21.15.78:0/1682950160 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ceffd640 1 --2- 172.21.15.78:0/1682950160 >> v2:172.21.15.150:3300/0 conn(0x7fc5d80c7070 0x7fc5d81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5d6ffd640 1 -- 172.21.15.78:0/1682950160 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043860346 0 0) 0x7fc5c8001110 con 0x7fc5d8152480 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5d6ffd640 1 -- 172.21.15.78:0/1682950160 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c4003520 con 0x7fc5d8152480 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ceffd640 1 -- 172.21.15.78:0/1682950160 >> v1:172.21.15.150:6789/0 conn(0x7fc5d80c87a0 legacy=0x7fc5d80c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ceffd640 1 -- 172.21.15.78:0/1682950160 >> v1:172.21.15.78:6789/0 conn(0x7fc5d8152480 legacy=0x7fc5d806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.674+0000 7fc5ceffd640 1 -- 172.21.15.78:0/1682950160 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d80c75c0 con 0x7fc5d80c7070 2024-02-19T14:59:39.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.675+0000 7fc5ceffd640 1 --2- 172.21.15.78:0/1682950160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81559b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00097a0 tx=0x7fc5c002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9dd822fb3a94fc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.675+0000 7fc5d6ffd640 1 -- 172.21.15.78:0/1682950160 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0030610 con 0x7fc5d80c7070 2024-02-19T14:59:39.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.675+0000 7fc5d6ffd640 1 -- 172.21.15.78:0/1682950160 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c0030770 con 0x7fc5d80c7070 2024-02-19T14:59:39.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.675+0000 7fc5d6ffd640 1 -- 172.21.15.78:0/1682950160 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0030b00 con 0x7fc5d80c7070 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 -- 172.21.15.78:0/1682950160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 msgr2=0x7fc5d81559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/1682950160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81559b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00097a0 tx=0x7fc5c002b960 comp rx=0 tx=0).stop 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 -- 172.21.15.78:0/1682950160 shutdown_connections 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/1682950160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81559b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 -- 172.21.15.78:0/1682950160 >> 172.21.15.78:0/1682950160 conn(0x7fc5d80ca8a0 msgr2=0x7fc5d80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 -- 172.21.15.78:0/1682950160 shutdown_connections 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.676+0000 7fc5ddedf640 1 -- 172.21.15.78:0/1682950160 wait complete. 2024-02-19T14:59:39.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.677+0000 7fc5ddedf640 1 Processor -- start 2024-02-19T14:59:39.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.678+0000 7fc5ddedf640 1 -- start start 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ddedf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ddedf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 0x7fc5d80ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ddedf640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5d8152480 0x7fc5d80c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ddedf640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d81ee010 con 0x7fc5d80c87a0 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ddedf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d81584f0 con 0x7fc5d80c7070 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ddedf640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8158ed0 con 0x7fc5d8152480 2024-02-19T14:59:39.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 0x7fc5d80ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 0x7fc5d80ac620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35418/0 (socket says 172.21.15.78:35418) 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 -- 172.21.15.78:0/2902003732 learned_addr learned my addr 172.21.15.78:0/2902003732 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5d77fe640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5d7fff640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5d8152480 0x7fc5d80c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 -- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5d8152480 msgr2=0x7fc5d80c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5d8152480 0x7fc5d80c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 -- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 msgr2=0x7fc5d81eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 -- 172.21.15.78:0/2902003732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0009400 con 0x7fc5d80c87a0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5d77fe640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5ceffd640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 0x7fc5d80ac620 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fc5c002bf70 tx=0x7fc5c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.679+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c003e040 con 0x7fc5d80c87a0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.680+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c0039cb0 con 0x7fc5d80c87a0 2024-02-19T14:59:39.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.680+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d81591b0 con 0x7fc5d80c87a0 2024-02-19T14:59:39.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.680+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0044630 con 0x7fc5d80c87a0 2024-02-19T14:59:39.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.680+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d80acb60 con 0x7fc5d80c87a0 2024-02-19T14:59:39.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.681+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d80c17c0 con 0x7fc5d80c87a0 2024-02-19T14:59:39.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.681+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc5c004d3b0 con 0x7fc5d80c87a0 2024-02-19T14:59:39.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.682+0000 7fc5d4ff9640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5b806b810 0x7fc5b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:39.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.682+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5c00ac820 con 0x7fc5d80c87a0 2024-02-19T14:59:39.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.682+0000 7fc5d77fe640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5b806b810 0x7fc5b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:39.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.682+0000 7fc5d77fe640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5b806b810 0x7fc5b806dcd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8001270 tx=0x7fc5c8007bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:39.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.686+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc5c0077160 con 0x7fc5d80c87a0 2024-02-19T14:59:39.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.929+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5d81593a0 con 0x7fc5d80c87a0 2024-02-19T14:59:39.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.930+0000 7fc5d4ff9640 1 -- 172.21.15.78:0/2902003732 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4966 (secure 0 0 0) 0x7fc5c007a820 con 0x7fc5d80c87a0 2024-02-19T14:59:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:39.933 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:37.764041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9816,"mds_1":8758},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8758":{"gid":8758,"name":"d","rank":1,"incarnation":173,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:39.933 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 174 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5b806b810 msgr2=0x7fc5b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5b806b810 0x7fc5b806dcd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8001270 tx=0x7fc5c8007bb0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 msgr2=0x7fc5d80ac620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 0x7fc5d80ac620 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fc5c002bf70 tx=0x7fc5c0004730 comp rx=0 tx=0).stop 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 shutdown_connections 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5b806b810 0x7fc5b806dcd0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5d80c87a0 0x7fc5d80ac620 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5d8152480 0x7fc5d80c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 --2- 172.21.15.78:0/2902003732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5d80c7070 0x7fc5d81eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 >> 172.21.15.78:0/2902003732 conn(0x7fc5d80ca8a0 msgr2=0x7fc5d814feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 shutdown_connections 2024-02-19T14:59:39.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:39.932+0000 7fc5ddedf640 1 -- 172.21.15.78:0/2902003732 wait complete. 2024-02-19T14:59:40.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 174, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:07.976421+0000', 'modified': '2024-02-19T14:59:37.764041+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9816, 'mds_1': 8758}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8758': {'gid': 8758, 'name': 'd', 'rank': 1, 'incarnation': 173, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/2126014034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2126014034}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2126014034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9816': {'gid': 9816, 'name': 'c', 'rank': 0, 'incarnation': 171, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.78:6835/778404633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 778404633}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 778404633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T14:59:40.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-02-19T14:59:40.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T14:59:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.449+0000 7f260d527640 1 Processor -- start 2024-02-19T14:59:40.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.450+0000 7f260d527640 1 -- start start 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.450+0000 7f260d527640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26080c5130 0x7f26080c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.450+0000 7f260d527640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26080c5510 0x7f26081559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.450+0000 7f260d527640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26080c3cf0 con 0x7f2608158f80 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.450+0000 7f260d527640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2608158b60 con 0x7f26080c5510 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.450+0000 7f260d527640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2608155f00 con 0x7f26080c5130 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26080c5130 0x7f26080c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f26077fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26080c5510 0x7f26081559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f26067fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2608158f80 0x7f26080c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51732/0 (socket says 172.21.15.78:51732) 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f26067fc640 1 -- 172.21.15.78:0/409378709 learned_addr learned my addr 172.21.15.78:0/409378709 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2605ffb640 1 -- 172.21.15.78:0/409378709 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739996482 0 0) 0x7f26080c3cf0 con 0x7f2608158f80 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2605ffb640 1 -- 172.21.15.78:0/409378709 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f0003520 con 0x7f2608158f80 2024-02-19T14:59:40.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2606ffd640 1 -- 172.21.15.78:0/409378709 >> v2:172.21.15.150:3300/0 conn(0x7f26080c5510 msgr2=0x7f26081559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2606ffd640 1 --2- 172.21.15.78:0/409378709 >> v2:172.21.15.150:3300/0 conn(0x7f26080c5510 0x7f26081559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2606ffd640 1 -- 172.21.15.78:0/409378709 >> v1:172.21.15.150:6789/0 conn(0x7f2608158f80 legacy=0x7f26080c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f2606ffd640 1 -- 172.21.15.78:0/409378709 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608159400 con 0x7f26080c5130 2024-02-19T14:59:40.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.451+0000 7f26077fe640 1 --2- 172.21.15.78:0/409378709 >> v2:172.21.15.150:3300/0 conn(0x7f26080c5510 0x7f26081559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f2606ffd640 1 --2- 172.21.15.78:0/409378709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26080c5130 0x7f26080c30a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2600009d90 tx=0x7f260002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c107d880d011ad22 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f2605ffb640 1 -- 172.21.15.78:0/409378709 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260002cc50 con 0x7f26080c5130 2024-02-19T14:59:40.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f2605ffb640 1 -- 172.21.15.78:0/409378709 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260002cdb0 con 0x7f26080c5130 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f2605ffb640 1 -- 172.21.15.78:0/409378709 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2600034640 con 0x7f26080c5130 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 -- 172.21.15.78:0/409378709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26080c5130 msgr2=0x7f26080c30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 --2- 172.21.15.78:0/409378709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26080c5130 0x7f26080c30a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2600009d90 tx=0x7f260002bea0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 -- 172.21.15.78:0/409378709 shutdown_connections 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 --2- 172.21.15.78:0/409378709 >> v2:172.21.15.150:3300/0 conn(0x7f26080c5510 0x7f26081559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 --2- 172.21.15.78:0/409378709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26080c5130 0x7f26080c30a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 -- 172.21.15.78:0/409378709 >> 172.21.15.78:0/409378709 conn(0x7f260806b4b0 msgr2=0x7f26080c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 -- 172.21.15.78:0/409378709 shutdown_connections 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.452+0000 7f260d527640 1 -- 172.21.15.78:0/409378709 wait complete. 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 Processor -- start 2024-02-19T14:59:40.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 -- start start 2024-02-19T14:59:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26080c5130 0x7f2608154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 0x7f26081487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2608158f80 0x7f2608148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26080acf10 con 0x7f26080c5510 2024-02-19T14:59:40.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26080a97a0 con 0x7f26080c5130 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.454+0000 7f260d527640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26080aaba0 con 0x7f2608158f80 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 0x7f26081487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 0x7f26081487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35424/0 (socket says 172.21.15.78:35424) 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 -- 172.21.15.78:0/2248234525 learned_addr learned my addr 172.21.15.78:0/2248234525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26077fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2608158f80 0x7f2608148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26080c5130 0x7f2608154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 -- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2608158f80 msgr2=0x7f2608148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2608158f80 0x7f2608148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 -- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26080c5130 msgr2=0x7f2608154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26080c5130 0x7f2608154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 -- 172.21.15.78:0/2248234525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8009400 con 0x7f26080c5510 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26077fe640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2608158f80 0x7f2608148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f26067fc640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 0x7f26081487c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00c390 tx=0x7f25fc00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f25ef7fe640 1 -- 172.21.15.78:0/2248234525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc005ce0 con 0x7f26080c5510 2024-02-19T14:59:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f25ef7fe640 1 -- 172.21.15.78:0/2248234525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc005e40 con 0x7f26080c5510 2024-02-19T14:59:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2600009a70 con 0x7f26080c5510 2024-02-19T14:59:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f25ef7fe640 1 -- 172.21.15.78:0/2248234525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0072f0 con 0x7f26080c5510 2024-02-19T14:59:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.455+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26080a9f70 con 0x7f26080c5510 2024-02-19T14:59:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.457+0000 7f25ef7fe640 1 -- 172.21.15.78:0/2248234525 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f25fc007450 con 0x7f26080c5510 2024-02-19T14:59:40.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.457+0000 7f25ef7fe640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f25f406b810 0x7f25f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.457+0000 7f2606ffd640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f25f406b810 0x7f25f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.457+0000 7f25ef7fe640 1 -- 172.21.15.78:0/2248234525 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f25fc0897d0 con 0x7f26080c5510 2024-02-19T14:59:40.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.458+0000 7f2606ffd640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f25f406b810 0x7f25f406dcd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2600036000 tx=0x7f2600030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.458+0000 7f26101d7740 1 Processor -- start 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.458+0000 7f26101d7740 1 -- start start 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26101d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba10238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba0fe8ae0 0x561ba1022b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba0fe7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26101d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba0fa2cd0 con 0x561ba1022740 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba0f55f20 con 0x561ba10258b0 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ba0f53450 con 0x561ba0fe8ae0 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba10238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba10238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35440/0 (socket says 172.21.15.78:35440) 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 -- 172.21.15.78:0/2698708557 learned_addr learned my addr 172.21.15.78:0/2698708557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26067fc640 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba0fe8ae0 0x561ba1022b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26077fe640 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba0fe7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 -- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba0fe8ae0 msgr2=0x561ba1022b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba0fe8ae0 0x561ba1022b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 -- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 msgr2=0x561ba0fe7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba0fe7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f2606ffd640 1 -- 172.21.15.78:0/2698708557 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ba0f03f20 con 0x561ba1022740 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.459+0000 7f26077fe640 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba0fe7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f2606ffd640 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba10238b0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f260002bee0 tx=0x7f260002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a6af136bc6f7ab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f25ecff9640 1 -- 172.21.15.78:0/2698708557 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2600034e70 con 0x561ba1022740 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f25ecff9640 1 -- 172.21.15.78:0/2698708557 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2600054460 con 0x561ba1022740 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f25ecff9640 1 -- 172.21.15.78:0/2698708557 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26000547d0 con 0x561ba1022740 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 -- 172.21.15.78:0/2698708557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 msgr2=0x561ba10238b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba10238b0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f260002bee0 tx=0x7f260002e000 comp rx=0 tx=0).stop 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 -- 172.21.15.78:0/2698708557 shutdown_connections 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba0fe7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba0fe8ae0 0x561ba1022b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 --2- 172.21.15.78:0/2698708557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba10238b0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.460+0000 7f26101d7740 1 -- 172.21.15.78:0/2698708557 >> 172.21.15.78:0/2698708557 conn(0x561ba0fe4120 msgr2=0x561ba1008b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.461+0000 7f26101d7740 1 -- 172.21.15.78:0/2698708557 shutdown_connections 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.461+0000 7f26101d7740 1 -- 172.21.15.78:0/2698708557 wait complete. 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.461+0000 7f26101d7740 1 -- start start 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.461+0000 7f26101d7740 1 Processor -- start 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1022740 0x561ba1019d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba10053d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba0f4ec30 con 0x561ba1034770 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba0ef3d30 con 0x561ba10258b0 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ba0c22310 con 0x561ba1022740 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26101d7740 10 client.0 _ll_register_callbacks cb 0x561ba0efbd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba10053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1022740 0x561ba1019d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba10053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35446/0 (socket says 172.21.15.78:35446) 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26067fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26077fe640 1 -- 172.21.15.78:0/3788377945 learned_addr learned my addr 172.21.15.78:0/3788377945 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 -- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 msgr2=0x561ba101ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 -- 172.21.15.78:0/3788377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 msgr2=0x561ba10053d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba10053d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 -- 172.21.15.78:0/3788377945 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2600030070 con 0x561ba1022740 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f26077fe640 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba10053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f2606ffd640 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1022740 0x561ba1019d90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x561ba10233f0 tx=0x7f260002e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f39791d4fd018a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.462+0000 7f25e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3301/0 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f25e5ffb640 1 -- 172.21.15.78:0/3788377945 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2600054c10 con 0x561ba1022740 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ba0c4e6c0 con 0x561ba1022740 2024-02-19T14:59:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f25e5ffb640 1 -- 172.21.15.78:0/3788377945 <== mon.2 v2:172.21.15.150:3301/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2600054d70 con 0x561ba1022740 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f25e5ffb640 1 -- 172.21.15.78:0/3788377945 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7f260005f460 con 0x561ba1022740 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 10 client.7565 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x561ba0f4d2b0 con 0x561ba1022740 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f25e5ffb640 1 -- 172.21.15.78:0/3788377945 <== mon.2 v2:172.21.15.150:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ba0c4e6c0 con 0x561ba1022740 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f25e5ffb640 1 -- 172.21.15.78:0/3788377945 <== mon.2 v2:172.21.15.150:3301/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f2600047070 con 0x561ba1022740 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 10 client.7565 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 10 client.7565 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 20 client.7565 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba101f200 0x561ba101f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 4 client.7565 mds_command: new command op to 8758 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 --> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ba0c9f270 con 0x561ba101f200 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.463+0000 7f26067fc640 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba101f200 0x561ba101f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f25e5ffb640 10 client.7565 ms_handle_connect on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f26067fc640 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba101f200 0x561ba101f5e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=685f959c1531c666 server_cookie=5b143a782dc5f45b in_seq=0 out_seq=0 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f25e5ffb640 1 -- 172.21.15.78:0/3788377945 <== mds.1 v2:172.21.15.150:6833/2126014034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ba0c9f270 con 0x561ba101f200 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f25e5ffb640 10 client.7565 handle_command_reply: tid=0 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f26101d7740 1 client.7565 shutdown 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1022740 msgr2=0x561ba1019d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f26101d7740 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1022740 0x561ba1019d90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x561ba10233f0 tx=0x7f260002e070 comp rx=0 tx=0).stop 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 shutdown_connections 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.464+0000 7f26101d7740 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba10053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba101f200 0x561ba101f5e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 1 --2- 172.21.15.78:0/3788377945 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1022740 0x561ba1019d90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f25e5ffb640 0 client.7565 ms_handle_reset on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 >> 172.21.15.78:0/3788377945 conn(0x561ba0fe8ae0 msgr2=0x561ba1023e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 shutdown_connections 2024-02-19T14:59:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 1 -- 172.21.15.78:0/3788377945 wait complete. 2024-02-19T14:59:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.465+0000 7f26101d7740 20 client.7565 trim_cache size 0 max 16384 2024-02-19T14:59:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.481+0000 7f26101d7740 1 Processor -- start 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.481+0000 7f26101d7740 1 -- start start 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba1022740 0x561ba1033d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba10258b0 0x561ba101bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26101d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba101c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26101d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba0ed4490 con 0x561ba1034770 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba1031b20 con 0x561ba1022740 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ba1020a70 con 0x561ba10258b0 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba101c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba101c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35450/0 (socket says 172.21.15.78:35450) 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 -- 172.21.15.78:0/4079728885 learned_addr learned my addr 172.21.15.78:0/4079728885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f2606ffd640 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba1022740 0x561ba1033d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26067fc640 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba10258b0 0x561ba101bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 -- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba10258b0 msgr2=0x561ba101bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba10258b0 0x561ba101bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 -- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba1022740 msgr2=0x561ba1033d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba1022740 0x561ba1033d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 -- 172.21.15.78:0/4079728885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ba0ee9a20 con 0x561ba1034770 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f26077fe640 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba101c130 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f25f8004390 tx=0x7f25f8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3660b90fb6a969b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.482+0000 7f25e7fff640 1 -- 172.21.15.78:0/4079728885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8003820 con 0x561ba1034770 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f25e7fff640 1 -- 172.21.15.78:0/4079728885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f8003980 con 0x561ba1034770 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f25e7fff640 1 -- 172.21.15.78:0/4079728885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8003cf0 con 0x561ba1034770 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 -- 172.21.15.78:0/4079728885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 msgr2=0x561ba101c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba101c130 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f25f8004390 tx=0x7f25f8009190 comp rx=0 tx=0).stop 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 -- 172.21.15.78:0/4079728885 shutdown_connections 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1034770 0x561ba101c130 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba10258b0 0x561ba101bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 --2- 172.21.15.78:0/4079728885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba1022740 0x561ba1033d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 -- 172.21.15.78:0/4079728885 >> 172.21.15.78:0/4079728885 conn(0x561ba0fe8ae0 msgr2=0x561ba0fdd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 -- 172.21.15.78:0/4079728885 shutdown_connections 2024-02-19T14:59:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 -- 172.21.15.78:0/4079728885 wait complete. 2024-02-19T14:59:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.483+0000 7f26101d7740 1 -- start start 2024-02-19T14:59:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 Processor -- start 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba1034070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1034770 0x561ba103e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba108d980 con 0x561ba1022740 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ba10936c0 con 0x561ba10258b0 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ba1037480 con 0x561ba1034770 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26101d7740 10 client.0 _ll_register_callbacks cb 0x7f25fc054110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba1034070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f2606ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba1034070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35466/0 (socket says 172.21.15.78:35466) 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f2606ffd640 1 -- 172.21.15.78:0/659599854 learned_addr learned my addr 172.21.15.78:0/659599854 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26067fc640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.484+0000 7f26077fe640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1034770 0x561ba103e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f2606ffd640 1 -- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1034770 msgr2=0x561ba103e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f2606ffd640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1034770 0x561ba103e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f2606ffd640 1 -- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 msgr2=0x561ba101e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f2606ffd640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f2606ffd640 1 -- 172.21.15.78:0/659599854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f8004040 con 0x561ba1022740 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f2606ffd640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba1034070 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f2600037040 tx=0x7f2600004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586b547cfb43c46d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f25e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f25e5ffb640 1 -- 172.21.15.78:0/659599854 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260005fa70 con 0x561ba1022740 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f25e5ffb640 1 -- 172.21.15.78:0/659599854 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f260005fbd0 con 0x561ba1022740 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2600054460 con 0x561ba1022740 2024-02-19T14:59:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f25e5ffb640 1 -- 172.21.15.78:0/659599854 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7f260005f460 con 0x561ba1022740 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 10 client.10152 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x561ba0f0f780 con 0x561ba1022740 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f25e5ffb640 1 -- 172.21.15.78:0/659599854 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2600054460 con 0x561ba1022740 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f25e5ffb640 1 -- 172.21.15.78:0/659599854 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f2600047040 con 0x561ba1022740 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 10 client.10152 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 10 client.10152 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 20 client.10152 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.485+0000 7f26101d7740 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba103e8c0 0x561ba10478f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 4 client.10152 mds_command: new command op to 8758 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 --> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561ba0c8b610 con 0x561ba103e8c0 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26067fc640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba103e8c0 0x561ba10478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26067fc640 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba103e8c0 0x561ba10478f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=28d64680f405117a server_cookie=cdd33d959796b154 in_seq=0 out_seq=0 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f25e5ffb640 10 client.10152 ms_handle_connect on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f25e5ffb640 1 -- 172.21.15.78:0/659599854 <== mds.1 v2:172.21.15.150:6833/2126014034 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x561ba0c8b610 con 0x561ba103e8c0 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f25e5ffb640 10 client.10152 handle_command_reply: tid=0 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 client.10152 shutdown 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 msgr2=0x561ba1034070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba1034070 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f2600037040 tx=0x7f2600004ab0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 shutdown_connections 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:6833/2126014034,v1:172.21.15.150:6835/2126014034] conn(0x561ba103e8c0 0x561ba10478f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561ba1034770 0x561ba103e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561ba10258b0 0x561ba101e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f25e5ffb640 0 client.10152 ms_handle_reset on v2:172.21.15.150:6833/2126014034 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 --2- 172.21.15.78:0/659599854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561ba1022740 0x561ba1034070 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 >> 172.21.15.78:0/659599854 conn(0x561ba0fe8ae0 msgr2=0x561ba1023b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 shutdown_connections 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.486+0000 7f26101d7740 1 -- 172.21.15.78:0/659599854 wait complete. 2024-02-19T14:59:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f26101d7740 20 client.10152 trim_cache size 0 max 16384 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 8758, 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 2.7383068310000001, 2024-02-19T14:59:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 174, 2024-02-19T14:59:40.498 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 188, 2024-02-19T14:59:40.498 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 188, 2024-02-19T14:59:40.498 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 6.7394096770000003 2024-02-19T14:59:40.498 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f25f406b810 msgr2=0x7f25f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f25f406b810 0x7f25f406dcd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f2600036000 tx=0x7f2600030000 comp rx=0 tx=0).stop 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 msgr2=0x7f26081487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 0x7f26081487c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00c390 tx=0x7f25fc00c790 comp rx=0 tx=0).stop 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 shutdown_connections 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f25f406b810 0x7f25f406dcd0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26080c5510 0x7f26081487c0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2608158f80 0x7f2608148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 --2- 172.21.15.78:0/2248234525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26080c5130 0x7f2608154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 >> 172.21.15.78:0/2248234525 conn(0x7f260806b4b0 msgr2=0x7f26080c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 shutdown_connections 2024-02-19T14:59:40.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.487+0000 7f260d527640 1 -- 172.21.15.78:0/2248234525 wait complete. 2024-02-19T14:59:40.501 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T14:59:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.599+0000 7f4027fd7640 1 Processor -- start 2024-02-19T14:59:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.600+0000 7f4027fd7640 1 -- start start 2024-02-19T14:59:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.600+0000 7f4027fd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f4020159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.600+0000 7f4027fd7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f40200c4d80 0x7f40200c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.600+0000 7f4027fd7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402006ffa0 con 0x7f40200c48f0 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.600+0000 7f4027fd7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f402006b720 con 0x7f40200c4d80 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.600+0000 7f4027fd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402006fc70 con 0x7f40200c78a0 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f40200c4d80 0x7f40200c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f40200c4d80 0x7f40200c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35478/0 (socket says 172.21.15.78:35478) 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 -- 172.21.15.78:0/2191118664 learned_addr learned my addr 172.21.15.78:0/2191118664 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025d73640 1 --2- 172.21.15.78:0/2191118664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f4020159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 -- 172.21.15.78:0/2191118664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 msgr2=0x7f4020159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 --2- 172.21.15.78:0/2191118664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f4020159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 -- 172.21.15.78:0/2191118664 >> v1:172.21.15.150:6789/0 conn(0x7f40200c48f0 legacy=0x7f40201523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.601+0000 7f4025572640 1 -- 172.21.15.78:0/2191118664 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402006b930 con 0x7f40200c4d80 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4025572640 1 --2- 172.21.15.78:0/2191118664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c4d80 0x7f40200c43b0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f4010009870 tx=0x7f401002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a03b18788ea5610 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4024d71640 1 -- 172.21.15.78:0/2191118664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010031640 con 0x7f40200c4d80 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4024d71640 1 -- 172.21.15.78:0/2191118664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40100317a0 con 0x7f40200c4d80 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4024d71640 1 -- 172.21.15.78:0/2191118664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010031b30 con 0x7f40200c4d80 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4027fd7640 1 -- 172.21.15.78:0/2191118664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c4d80 msgr2=0x7f40200c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4027fd7640 1 --2- 172.21.15.78:0/2191118664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c4d80 0x7f40200c43b0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f4010009870 tx=0x7f401002b8f0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4027fd7640 1 -- 172.21.15.78:0/2191118664 shutdown_connections 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4027fd7640 1 --2- 172.21.15.78:0/2191118664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c4d80 0x7f40200c43b0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4027fd7640 1 --2- 172.21.15.78:0/2191118664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f4020159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.602+0000 7f4027fd7640 1 -- 172.21.15.78:0/2191118664 >> 172.21.15.78:0/2191118664 conn(0x7f4020066e70 msgr2=0x7f40200a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.603+0000 7f4027fd7640 1 -- 172.21.15.78:0/2191118664 shutdown_connections 2024-02-19T14:59:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.603+0000 7f4027fd7640 1 -- 172.21.15.78:0/2191118664 wait complete. 2024-02-19T14:59:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 Processor -- start 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 -- start start 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 0x7f402014e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40200c4d80 0x7f402014be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f402014c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40201578f0 con 0x7f40200c48f0 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40201563e0 con 0x7f40200c4d80 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4027fd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40201eb050 con 0x7f40200c78a0 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4025d73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 0x7f402014e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4025d73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 0x7f402014e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35482/0 (socket says 172.21.15.78:35482) 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4025d73640 1 -- 172.21.15.78:0/1123601104 learned_addr learned my addr 172.21.15.78:0/1123601104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4025572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40200c4d80 0x7f402014be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.604+0000 7f4026574640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f402014c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4025d73640 1 -- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 msgr2=0x7f402014c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4025d73640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f402014c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4025d73640 1 -- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40200c4d80 msgr2=0x7f402014be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4025d73640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40200c4d80 0x7f402014be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4025d73640 1 -- 172.21.15.78:0/1123601104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40100094d0 con 0x7f40200c48f0 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4025d73640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 0x7f402014e790 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f401c00e8a0 tx=0x7f401c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f40167fc640 1 -- 172.21.15.78:0/1123601104 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0094f0 con 0x7f40200c48f0 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f40167fc640 1 -- 172.21.15.78:0/1123601104 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c00c940 con 0x7f40200c48f0 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f40167fc640 1 -- 172.21.15.78:0/1123601104 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c00ccf0 con 0x7f40200c48f0 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40200c2140 con 0x7f40200c48f0 2024-02-19T14:59:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.605+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40201eace0 con 0x7f40200c48f0 2024-02-19T14:59:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.607+0000 7f40167fc640 1 -- 172.21.15.78:0/1123601104 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f401c013470 con 0x7f40200c48f0 2024-02-19T14:59:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.608+0000 7f40167fc640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f400406b840 0x7f400406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.608+0000 7f4025572640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f400406b840 0x7f400406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.608+0000 7f40167fc640 1 -- 172.21.15.78:0/1123601104 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f401c08a2d0 con 0x7f40200c48f0 2024-02-19T14:59:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.608+0000 7f402abb5740 1 Processor -- start 2024-02-19T14:59:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.608+0000 7f402abb5740 1 -- start start 2024-02-19T14:59:40.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.608+0000 7f4025572640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f400406b840 0x7f400406dd00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4010036000 tx=0x7f401003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0516fae0 0x560f051a9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f402abb5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f0516e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f402abb5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f05129cd0 con 0x560f051ac8b0 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f050dcf20 con 0x560f0516fae0 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f050da450 con 0x560f051a9740 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f4026574640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f0516e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f4026574640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f0516e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35492/0 (socket says 172.21.15.78:35492) 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f4025572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0516fae0 0x560f051a9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f4026574640 1 -- 172.21.15.78:0/1906479809 learned_addr learned my addr 172.21.15.78:0/1906479809 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.609+0000 7f4025d73640 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4025d73640 1 -- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0516fae0 msgr2=0x560f051a9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4025d73640 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0516fae0 0x560f051a9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4025d73640 1 -- 172.21.15.78:0/1906479809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 msgr2=0x560f0516e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4025d73640 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f0516e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4025d73640 1 -- 172.21.15.78:0/1906479809 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f0508af20 con 0x560f051a9740 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4026574640 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f0516e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f4025d73640 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051aa8b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f401c088750 tx=0x7f401c058420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37706d6f8e806ce5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f40037fe640 1 -- 172.21.15.78:0/1906479809 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c08a0f0 con 0x560f051a9740 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f40037fe640 1 -- 172.21.15.78:0/1906479809 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c061060 con 0x560f051a9740 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.610+0000 7f40037fe640 1 -- 172.21.15.78:0/1906479809 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c00ce50 con 0x560f051a9740 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 -- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 msgr2=0x560f051aa8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051aa8b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f401c088750 tx=0x7f401c058420 comp rx=0 tx=0).stop 2024-02-19T14:59:40.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 -- 172.21.15.78:0/1906479809 shutdown_connections 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f0516e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0516fae0 0x560f051a9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 --2- 172.21.15.78:0/1906479809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051aa8b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 -- 172.21.15.78:0/1906479809 >> 172.21.15.78:0/1906479809 conn(0x560f0516b120 msgr2=0x560f0518fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 -- 172.21.15.78:0/1906479809 shutdown_connections 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 -- 172.21.15.78:0/1906479809 wait complete. 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 -- start start 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.611+0000 7f402abb5740 1 Processor -- start 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f05169390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f050d5c30 con 0x560f0518c970 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f0507ad30 con 0x560f051a9740 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f04da9310 con 0x560f051ac8b0 2024-02-19T14:59:40.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f402abb5740 10 client.0 _ll_register_callbacks cb 0x560f05082d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f4026574640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f4026574640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35504/0 (socket says 172.21.15.78:35504) 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f4026574640 1 -- 172.21.15.78:0/3922507230 learned_addr learned my addr 172.21.15.78:0/3922507230 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f4025d73640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f05169390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.612+0000 7f4025572640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4025572640 1 -- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 msgr2=0x560f05169390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4025572640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f05169390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4025572640 1 -- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 msgr2=0x560f051a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4025572640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4025572640 1 -- 172.21.15.78:0/3922507230 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40201eace0 con 0x560f051ac8b0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4026574640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4025572640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a18b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40200c8450 tx=0x7f401002e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c69f3beee719ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4000ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3301/0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4000ff9640 1 -- 172.21.15.78:0/3922507230 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401004b040 con 0x560f051ac8b0 2024-02-19T14:59:40.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4000ff9640 1 -- 172.21.15.78:0/3922507230 <== mon.2 v2:172.21.15.150:3301/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f401003f070 con 0x560f051ac8b0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f04dd56c0 con 0x560f051ac8b0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4000ff9640 1 -- 172.21.15.78:0/3922507230 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40100448d0 con 0x560f051ac8b0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f402abb5740 10 client.7583 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x560f050d42b0 con 0x560f051ac8b0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4000ff9640 1 -- 172.21.15.78:0/3922507230 <== mon.2 v2:172.21.15.150:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560f04dd56c0 con 0x560f051ac8b0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f4000ff9640 1 -- 172.21.15.78:0/3922507230 <== mon.2 v2:172.21.15.150:3301/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f4010044a30 con 0x560f051ac8b0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f402abb5740 10 client.7583 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.613+0000 7f402abb5740 10 client.7583 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f402abb5740 20 client.7583 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f402abb5740 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f051bdff0 0x560f051be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f402abb5740 4 client.7583 mds_command: new command op to 9816 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 --> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f04e26270 con 0x560f051bdff0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f4025d73640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f051bdff0 0x560f051be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f4000ff9640 10 client.7583 ms_handle_connect on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:40.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.614+0000 7f4025d73640 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f051bdff0 0x560f051be3d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbaac71843423656 server_cookie=a635b55087515273 in_seq=0 out_seq=0 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f4000ff9640 1 -- 172.21.15.78:0/3922507230 <== mds.0 v2:172.21.15.78:6834/778404633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560f04e26270 con 0x560f051bdff0 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f4000ff9640 10 client.7583 handle_command_reply: tid=0 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 client.7583 shutdown 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 msgr2=0x560f051a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a18b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40200c8450 tx=0x7f401002e000 comp rx=0 tx=0).stop 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 shutdown_connections 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f051bdff0 0x560f051be3d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a18b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 --2- 172.21.15.78:0/3922507230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f05169390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f4000ff9640 0 client.7583 ms_handle_reset on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 >> 172.21.15.78:0/3922507230 conn(0x560f0516fae0 msgr2=0x560f0516c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 shutdown_connections 2024-02-19T14:59:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 1 -- 172.21.15.78:0/3922507230 wait complete. 2024-02-19T14:59:40.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.615+0000 7f402abb5740 20 client.7583 trim_cache size 0 max 16384 2024-02-19T14:59:40.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.632+0000 7f402abb5740 1 Processor -- start 2024-02-19T14:59:40.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.632+0000 7f402abb5740 1 -- start start 2024-02-19T14:59:40.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f402abb5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f051ba6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f402abb5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f0505b490 con 0x560f0518c970 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f051a7c90 con 0x560f051a9740 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f051ce580 con 0x560f051ac8b0 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35512/0 (socket says 172.21.15.78:35512) 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 -- 172.21.15.78:0/2773807404 learned_addr learned my addr 172.21.15.78:0/2773807404 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025572640 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f051ba6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4026574640 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 -- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 msgr2=0x560f051a1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 -- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 msgr2=0x560f051ba6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f051ba6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 -- 172.21.15.78:0/2773807404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f05070a20 con 0x560f0518c970 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025572640 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f051ba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4025d73640 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a5470 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f401c058bc0 tx=0x7f401c054dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a5c0f57bab39ad server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4002ffd640 1 -- 172.21.15.78:0/2773807404 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c056d00 con 0x560f0518c970 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4002ffd640 1 -- 172.21.15.78:0/2773807404 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c055070 con 0x560f0518c970 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.633+0000 7f4002ffd640 1 -- 172.21.15.78:0/2773807404 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c088320 con 0x560f0518c970 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 -- 172.21.15.78:0/2773807404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 msgr2=0x560f051a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a5470 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f401c058bc0 tx=0x7f401c054dd0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 -- 172.21.15.78:0/2773807404 shutdown_connections 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051ac8b0 0x560f051a1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f051a9740 0x560f051ba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 --2- 172.21.15.78:0/2773807404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f0518c970 0x560f051a5470 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 -- 172.21.15.78:0/2773807404 >> 172.21.15.78:0/2773807404 conn(0x560f0516fae0 msgr2=0x560f051b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 -- 172.21.15.78:0/2773807404 shutdown_connections 2024-02-19T14:59:40.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.634+0000 7f402abb5740 1 -- 172.21.15.78:0/2773807404 wait complete. 2024-02-19T14:59:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 -- start start 2024-02-19T14:59:40.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 Processor -- start 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0518c970 0x560f0518f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051a1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f05216f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f050d4dd0 con 0x560f051ac8b0 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f051a1d90 con 0x560f0518c970 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f0518ff80 con 0x560f051a9740 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.635+0000 7f402abb5740 10 client.0 _ll_register_callbacks cb 0x7f4010031910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f05216f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4025d73640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0518c970 0x560f0518f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f05216f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35516/0 (socket says 172.21.15.78:35516) 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 -- 172.21.15.78:0/1492883503 learned_addr learned my addr 172.21.15.78:0/1492883503 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4025572640 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 -- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 msgr2=0x560f051a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 -- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0518c970 msgr2=0x560f0518f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0518c970 0x560f0518f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 -- 172.21.15.78:0/1492883503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c070040 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4026574640 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f05216f30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f40180052b0 tx=0x7f4018005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a52e45e66ef63b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4000ff9640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4000ff9640 1 -- 172.21.15.78:0/1492883503 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018006d60 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4000ff9640 1 -- 172.21.15.78:0/1492883503 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4018017990 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f401c061060 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f4000ff9640 1 -- 172.21.15.78:0/1492883503 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=174) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4018017b50 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f402abb5740 10 client.10170 fetch_fsmap learned FSMap version 174 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.636+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=174,monmap=2+}) v3 -- 0x560f05096780 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4000ff9640 1 -- 172.21.15.78:0/1492883503 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c061060 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4000ff9640 1 -- 172.21.15.78:0/1492883503 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 174) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f4018012070 con 0x560f051ac8b0 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 10 client.10170 fetch_fsmap finished waiting for FSMap version 174 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 10 client.10170 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T14:59:40.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 20 client.10170 populate_metadata read hostname 'smithi078' 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f052174e0 0x560f0521d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 4 client.10170 mds_command: new command op to 9816 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 --> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560f04e12610 con 0x560f052174e0 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4025d73640 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f052174e0 0x560f0521d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4000ff9640 10 client.10170 ms_handle_connect on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4025d73640 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f052174e0 0x560f0521d200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e502f5a5b6926f62 server_cookie=4659c3d9c52c3821 in_seq=0 out_seq=0 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4000ff9640 1 -- 172.21.15.78:0/1492883503 <== mds.0 v2:172.21.15.78:6834/778404633 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x560f04e12610 con 0x560f052174e0 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f4000ff9640 10 client.10170 handle_command_reply: tid=0 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 client.10170 shutdown 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 msgr2=0x560f05216f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f05216f30 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f40180052b0 tx=0x7f4018005770 comp rx=0 tx=0).stop 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 shutdown_connections 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.637+0000 7f402abb5740 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f051ac8b0 0x560f05216f30 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.78:6834/778404633,v1:172.21.15.78:6835/778404633] conn(0x560f052174e0 0x560f0521d200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x560f051a9740 0x560f051a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 1 --2- 172.21.15.78:0/1492883503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560f0518c970 0x560f0518f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4000ff9640 0 client.10170 ms_handle_reset on v2:172.21.15.78:6834/778404633 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 >> 172.21.15.78:0/1492883503 conn(0x560f0516fae0 msgr2=0x560f051ba650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 shutdown_connections 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 1 -- 172.21.15.78:0/1492883503 wait complete. 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f402abb5740 20 client.10170 trim_cache size 0 max 16384 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T14:59:40.647 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 9816, 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 3.8990280890000002, 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 174, 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 188, 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 188, 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 37.133270844000002 2024-02-19T14:59:40.648 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f400406b840 msgr2=0x7f400406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f400406b840 0x7f400406dd00 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4010036000 tx=0x7f401003c040 comp rx=0 tx=0).stop 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 msgr2=0x7f402014e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 0x7f402014e790 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f401c00e8a0 tx=0x7f401c00ed60 comp rx=0 tx=0).stop 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 shutdown_connections 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f400406b840 0x7f400406dd00 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40200c78a0 0x7f402014c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40200c4d80 0x7f402014be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 --2- 172.21.15.78:0/1123601104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40200c48f0 0x7f402014e790 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 >> 172.21.15.78:0/1123601104 conn(0x7f4020066e70 msgr2=0x7f4020151190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 shutdown_connections 2024-02-19T14:59:40.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.638+0000 7f4027fd7640 1 -- 172.21.15.78:0/1123601104 wait complete. 2024-02-19T14:59:40.652 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:59:40.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:59:40.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:59:40.739 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:59:40.739 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:59:40.739 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:59:40.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:59:40.754 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:59:40.754 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:59:40.796 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:59:40.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.894+0000 7f7e9c5f7640 1 Processor -- start 2024-02-19T14:59:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.894+0000 7f7e9c5f7640 1 -- start start 2024-02-19T14:59:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9c5f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e94152700 0x7f7e940c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9c5f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c8e00 0x7f7e9415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9c5f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e941551f0 con 0x7f7e940c5730 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9c5f7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e94155490 con 0x7f7e94152700 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9c5f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94155770 con 0x7f7e940c8e00 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9a393640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e94152700 0x7f7e940c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e99b92640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7e940c5730 0x7f7e9406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39450/0 (socket says 172.21.15.78:39450) 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e99b92640 1 -- 172.21.15.78:0/3971079125 learned_addr learned my addr 172.21.15.78:0/3971079125 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.895+0000 7f7e9ab94640 1 --2- 172.21.15.78:0/3971079125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c8e00 0x7f7e9415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e9a393640 1 -- 172.21.15.78:0/3971079125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c8e00 msgr2=0x7f7e9415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e9a393640 1 --2- 172.21.15.78:0/3971079125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c8e00 0x7f7e9415f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e9a393640 1 -- 172.21.15.78:0/3971079125 >> v1:172.21.15.78:6789/0 conn(0x7f7e940c5730 legacy=0x7f7e9406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e99391640 1 -- 172.21.15.78:0/3971079125 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437150448 0 0) 0x7f7e941551f0 con 0x7f7e940c5730 2024-02-19T14:59:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e9a393640 1 -- 172.21.15.78:0/3971079125 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e94155a50 con 0x7f7e94152700 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e9a393640 1 --2- 172.21.15.78:0/3971079125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e94152700 0x7f7e940c51f0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f7e840097a0 tx=0x7f7e8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a490bceba2a1c84 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.896+0000 7f7e99391640 1 -- 172.21.15.78:0/3971079125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84031830 con 0x7f7e94152700 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e99391640 1 -- 172.21.15.78:0/3971079125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e84031990 con 0x7f7e94152700 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e99391640 1 -- 172.21.15.78:0/3971079125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84031d20 con 0x7f7e94152700 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/3971079125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e94152700 msgr2=0x7f7e940c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/3971079125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e94152700 0x7f7e940c51f0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f7e840097a0 tx=0x7f7e8402b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/3971079125 shutdown_connections 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/3971079125 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c8e00 0x7f7e9415f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/3971079125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e94152700 0x7f7e940c51f0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/3971079125 >> 172.21.15.78:0/3971079125 conn(0x7f7e940cac90 msgr2=0x7f7e940cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/3971079125 shutdown_connections 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.897+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/3971079125 wait complete. 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.898+0000 7f7e9c5f7640 1 Processor -- start 2024-02-19T14:59:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.898+0000 7f7e9c5f7640 1 -- start start 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9c5f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c5730 0x7f7e941edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9c5f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 0x7f7e940c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9c5f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e94152700 0x7f7e940c3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9c5f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940ace60 con 0x7f7e940c8e00 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9c5f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941516d0 con 0x7f7e94152700 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9c5f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c38c0 con 0x7f7e940c5730 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 0x7f7e940c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 0x7f7e940c2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35530/0 (socket says 172.21.15.78:35530) 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 -- 172.21.15.78:0/2949602714 learned_addr learned my addr 172.21.15.78:0/2949602714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9a393640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c5730 0x7f7e941edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e9ab94640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e94152700 0x7f7e940c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 -- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c5730 msgr2=0x7f7e941edcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c5730 0x7f7e941edcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 -- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e94152700 msgr2=0x7f7e940c3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e94152700 0x7f7e940c3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.899+0000 7f7e99b92640 1 -- 172.21.15.78:0/2949602714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84009400 con 0x7f7e940c8e00 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.900+0000 7f7e99b92640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 0x7f7e940c2d50 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c4d0 tx=0x7f7e9000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.900+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e900074e0 con 0x7f7e940c8e00 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.900+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e90007660 con 0x7f7e940c8e00 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.900+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e940c3e80 con 0x7f7e940c8e00 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.900+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90004a20 con 0x7f7e940c8e00 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.900+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e941ee630 con 0x7f7e940c8e00 2024-02-19T14:59:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.901+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e940c4010 con 0x7f7e940c8e00 2024-02-19T14:59:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.902+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7e90004b80 con 0x7f7e940c8e00 2024-02-19T14:59:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.902+0000 7f7e8affd640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e7806b810 0x7f7e7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.902+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e900897a0 con 0x7f7e940c8e00 2024-02-19T14:59:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.902+0000 7f7e9a393640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e7806b810 0x7f7e7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.903+0000 7f7e9a393640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e7806b810 0x7f7e7806dcd0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7e84036000 tx=0x7f7e8403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:40.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:40.911+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7e90054170 con 0x7f7e940c8e00 2024-02-19T14:59:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.126+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e94074990 con 0x7f7e940c8e00 2024-02-19T14:59:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.126+0000 7f7e8affd640 1 -- 172.21.15.78:0/2949602714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4966 (secure 0 0 0) 0x7f7e90057830 con 0x7f7e940c8e00 2024-02-19T14:59:41.129 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:41.129 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:37.764041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9816,"mds_1":8758},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8758":{"gid":8758,"name":"d","rank":1,"incarnation":173,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:41.129 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 174 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e7806b810 msgr2=0x7f7e7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e7806b810 0x7f7e7806dcd0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7e84036000 tx=0x7f7e8403c040 comp rx=0 tx=0).stop 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 msgr2=0x7f7e940c2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 0x7f7e940c2d50 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c4d0 tx=0x7f7e9000c990 comp rx=0 tx=0).stop 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 shutdown_connections 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e7806b810 0x7f7e7806dcd0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e940c8e00 0x7f7e940c2d50 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e94152700 0x7f7e940c3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 --2- 172.21.15.78:0/2949602714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e940c5730 0x7f7e941edcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 >> 172.21.15.78:0/2949602714 conn(0x7f7e940cac90 msgr2=0x7f7e9414e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 shutdown_connections 2024-02-19T14:59:41.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.128+0000 7f7e9c5f7640 1 -- 172.21.15.78:0/2949602714 wait complete. 2024-02-19T14:59:41.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:41.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.240+0000 7f297462a640 1 Processor -- start 2024-02-19T14:59:41.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.240+0000 7f297462a640 1 -- start start 2024-02-19T14:59:41.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f297462a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0c4980 0x7f296c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f297462a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f296c0705b0 0x7f296c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f297462a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c0ae670 con 0x7f296c0add70 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f297462a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296c0716b0 con 0x7f296c0705b0 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f297462a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c153780 con 0x7f296c0c4980 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f2972bc7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f296c0705b0 0x7f296c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f29723c6640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f296c0add70 0x7f296c06eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39464/0 (socket says 172.21.15.78:39464) 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f29723c6640 1 -- 172.21.15.78:0/3823997344 learned_addr learned my addr 172.21.15.78:0/3823997344 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.241+0000 7f2971bc5640 1 --2- 172.21.15.78:0/3823997344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0c4980 0x7f296c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.242+0000 7f2972bc7640 1 -- 172.21.15.78:0/3823997344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0c4980 msgr2=0x7f296c0c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.242+0000 7f2972bc7640 1 --2- 172.21.15.78:0/3823997344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0c4980 0x7f296c0c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.242+0000 7f2972bc7640 1 -- 172.21.15.78:0/3823997344 >> v1:172.21.15.78:6789/0 conn(0x7f296c0add70 legacy=0x7f296c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.242+0000 7f2972bc7640 1 -- 172.21.15.78:0/3823997344 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c153a60 con 0x7f296c0705b0 2024-02-19T14:59:41.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.242+0000 7f29713c4640 1 -- 172.21.15.78:0/3823997344 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588504760 0 0) 0x7f296c0ae670 con 0x7f296c0add70 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.243+0000 7f2972bc7640 1 --2- 172.21.15.78:0/3823997344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0705b0 0x7f296c0709c0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f2964009d90 tx=0x7f296402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee3ab984ea101b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.243+0000 7f29713c4640 1 -- 172.21.15.78:0/3823997344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296402cbc0 con 0x7f296c0705b0 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.243+0000 7f29713c4640 1 -- 172.21.15.78:0/3823997344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296402cd20 con 0x7f296c0705b0 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.243+0000 7f29713c4640 1 -- 172.21.15.78:0/3823997344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964034660 con 0x7f296c0705b0 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.243+0000 7f297462a640 1 -- 172.21.15.78:0/3823997344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0705b0 msgr2=0x7f296c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.243+0000 7f297462a640 1 --2- 172.21.15.78:0/3823997344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0705b0 0x7f296c0709c0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f2964009d90 tx=0x7f296402be10 comp rx=0 tx=0).stop 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.244+0000 7f297462a640 1 -- 172.21.15.78:0/3823997344 shutdown_connections 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.244+0000 7f297462a640 1 --2- 172.21.15.78:0/3823997344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0705b0 0x7f296c0709c0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.244+0000 7f297462a640 1 --2- 172.21.15.78:0/3823997344 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0c4980 0x7f296c0c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.244+0000 7f297462a640 1 -- 172.21.15.78:0/3823997344 >> 172.21.15.78:0/3823997344 conn(0x7f296c0c85e0 msgr2=0x7f296c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.244+0000 7f297462a640 1 -- 172.21.15.78:0/3823997344 shutdown_connections 2024-02-19T14:59:41.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.244+0000 7f297462a640 1 -- 172.21.15.78:0/3823997344 wait complete. 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 Processor -- start 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 -- start start 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f296c0705b0 0x7f296c14b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0add70 0x7f296c1ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 0x7f296c1ea570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c1e9ba0 con 0x7f296c0c4980 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c14b5a0 con 0x7f296c0705b0 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.245+0000 7f297462a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c1541d0 con 0x7f296c0add70 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 0x7f296c1ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 0x7f296c1ea570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35544/0 (socket says 172.21.15.78:35544) 2024-02-19T14:59:41.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 -- 172.21.15.78:0/1829637769 learned_addr learned my addr 172.21.15.78:0/1829637769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2971bc5640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0add70 0x7f296c1ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f29723c6640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f296c0705b0 0x7f296c14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 -- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0add70 msgr2=0x7f296c1ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0add70 0x7f296c1ea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 -- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f296c0705b0 msgr2=0x7f296c14b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f296c0705b0 0x7f296c14b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 -- 172.21.15.78:0/1829637769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2964009a70 con 0x7f296c0c4980 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2972bc7640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 0x7f296c1ea570 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f2964009d90 tx=0x7f2964004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964040650 con 0x7f296c0c4980 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296402ce90 con 0x7f296c0c4980 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29640409f0 con 0x7f296c0c4980 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c1544b0 con 0x7f296c0c4980 2024-02-19T14:59:41.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.246+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c1e9830 con 0x7f296c0c4980 2024-02-19T14:59:41.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.248+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2964040b50 con 0x7f296c0c4980 2024-02-19T14:59:41.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.248+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c0676a0 con 0x7f296c0c4980 2024-02-19T14:59:41.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.248+0000 7f2962ffd640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295006b810 0x7f295006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.248+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29640acf10 con 0x7f296c0c4980 2024-02-19T14:59:41.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.248+0000 7f29723c6640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295006b810 0x7f295006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.248+0000 7f29723c6640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295006b810 0x7f295006dcd0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f295c000f60 tx=0x7f295c007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.252+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2964077970 con 0x7f296c0c4980 2024-02-19T14:59:41.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.458+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f296c1e99c0 con 0x7f296c0c4980 2024-02-19T14:59:41.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.460+0000 7f2962ffd640 1 -- 172.21.15.78:0/1829637769 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+16872 (secure 0 0 0) 0x7f2964009450 con 0x7f296c0c4980 2024-02-19T14:59:41.463 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:41.463 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":188,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:09.648574+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:06.192227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:06.947291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295006b810 msgr2=0x7f295006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295006b810 0x7f295006dcd0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f295c000f60 tx=0x7f295c007d50 comp rx=0 tx=0).stop 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 msgr2=0x7f296c1ea570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 0x7f296c1ea570 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f2964009d90 tx=0x7f2964004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 shutdown_connections 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f295006b810 0x7f295006dcd0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f296c0c4980 0x7f296c1ea570 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f296c0add70 0x7f296c1ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 --2- 172.21.15.78:0/1829637769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f296c0705b0 0x7f296c14b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.461+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 >> 172.21.15.78:0/1829637769 conn(0x7f296c0c85e0 msgr2=0x7f296c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.462+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 shutdown_connections 2024-02-19T14:59:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.462+0000 7f297462a640 1 -- 172.21.15.78:0/1829637769 wait complete. 2024-02-19T14:59:41.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:59:06.192227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:59:06.947291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:59:41.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:59:41.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.579+0000 7fd5381bd640 1 Processor -- start 2024-02-19T14:59:41.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.579+0000 7fd5381bd640 1 -- start start 2024-02-19T14:59:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd5381bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5300c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd5381bd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5300c42e0 0x7fd530158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd5381bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300c34a0 con 0x7fd5300c2f50 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd5381bd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53006c090 con 0x7fd5300c42e0 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd5381bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53006d6a0 con 0x7fd5300c8500 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd535758640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5300c42e0 0x7fd530158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd53675a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd5300c2f50 0x7fd530158b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39478/0 (socket says 172.21.15.78:39478) 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd53675a640 1 -- 172.21.15.78:0/2219615986 learned_addr learned my addr 172.21.15.78:0/2219615986 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd535f59640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5300c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd535758640 1 -- 172.21.15.78:0/2219615986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 msgr2=0x7fd5300c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd535758640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5300c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd535758640 1 -- 172.21.15.78:0/2219615986 >> v1:172.21.15.78:6789/0 conn(0x7fd5300c2f50 legacy=0x7fd530158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.580+0000 7fd534f57640 1 -- 172.21.15.78:0/2219615986 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676694779 0 0) 0x7fd5300c34a0 con 0x7fd5300c2f50 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.581+0000 7fd535758640 1 -- 172.21.15.78:0/2219615986 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53006c8e0 con 0x7fd5300c42e0 2024-02-19T14:59:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.581+0000 7fd535f59640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5300c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.581+0000 7fd535758640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c42e0 0x7fd530158650 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097a0 tx=0x7fd52c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeadb4e8e1cd4d50 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.581+0000 7fd534f57640 1 -- 172.21.15.78:0/2219615986 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c02e070 con 0x7fd5300c42e0 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.581+0000 7fd534f57640 1 -- 172.21.15.78:0/2219615986 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c0315a0 con 0x7fd5300c42e0 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.581+0000 7fd534f57640 1 -- 172.21.15.78:0/2219615986 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c031930 con 0x7fd5300c42e0 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 -- 172.21.15.78:0/2219615986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c42e0 msgr2=0x7fd530158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c42e0 0x7fd530158650 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097a0 tx=0x7fd52c02b840 comp rx=0 tx=0).stop 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 -- 172.21.15.78:0/2219615986 shutdown_connections 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c42e0 0x7fd530158650 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2219615986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5300c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 -- 172.21.15.78:0/2219615986 >> 172.21.15.78:0/2219615986 conn(0x7fd5300aa300 msgr2=0x7fd5300aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 -- 172.21.15.78:0/2219615986 shutdown_connections 2024-02-19T14:59:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.582+0000 7fd5381bd640 1 -- 172.21.15.78:0/2219615986 wait complete. 2024-02-19T14:59:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.583+0000 7fd5381bd640 1 Processor -- start 2024-02-19T14:59:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.583+0000 7fd5381bd640 1 -- start start 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd5381bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 0x7fd5301eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd5381bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5301503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd5381bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300ac410 0x7fd53006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd5381bd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5301ebb00 con 0x7fd5300c2f50 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd5381bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd530070490 con 0x7fd5300ac410 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd5381bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300ab9c0 con 0x7fd5300c8500 2024-02-19T14:59:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 0x7fd5301eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 0x7fd5301eb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35568/0 (socket says 172.21.15.78:35568) 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 -- 172.21.15.78:0/2113038858 learned_addr learned my addr 172.21.15.78:0/2113038858 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535758640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5301503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd53675a640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300ac410 0x7fd53006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 -- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 msgr2=0x7fd5301503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5301503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 -- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300ac410 msgr2=0x7fd53006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300ac410 0x7fd53006fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 -- 172.21.15.78:0/2113038858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520009400 con 0x7fd5300c2f50 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.584+0000 7fd535f59640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 0x7fd5301eb5c0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fd52000e960 tx=0x7fd52000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.585+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52000ca00 con 0x7fd5300c2f50 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.585+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52000cb60 con 0x7fd5300c2f50 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.585+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c009400 con 0x7fd5300c2f50 2024-02-19T14:59:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.585+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520013470 con 0x7fd5300c2f50 2024-02-19T14:59:41.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.585+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd530150930 con 0x7fd5300c2f50 2024-02-19T14:59:41.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.586+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd530071c20 con 0x7fd5300c2f50 2024-02-19T14:59:41.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.587+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd5200135d0 con 0x7fd5300c2f50 2024-02-19T14:59:41.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.587+0000 7fd5267fc640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd51406b810 0x7fd51406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.587+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd52008a840 con 0x7fd5300c2f50 2024-02-19T14:59:41.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.587+0000 7fd535758640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd51406b810 0x7fd51406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.587+0000 7fd535758640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd51406b810 0x7fd51406dcd0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd52c036000 tx=0x7fd52c03e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.591+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd520055210 con 0x7fd5300c2f50 2024-02-19T14:59:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.762+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd530074990 con 0x7fd5300c2f50 2024-02-19T14:59:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.763+0000 7fd5267fc640 1 -- 172.21.15.78:0/2113038858 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5200588d0 con 0x7fd5300c2f50 2024-02-19T14:59:41.766 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd51406b810 msgr2=0x7fd51406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd51406b810 0x7fd51406dcd0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd52c036000 tx=0x7fd52c03e040 comp rx=0 tx=0).stop 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 msgr2=0x7fd5301eb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 0x7fd5301eb5c0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fd52000e960 tx=0x7fd52000ee20 comp rx=0 tx=0).stop 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 shutdown_connections 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd51406b810 0x7fd51406dcd0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300ac410 0x7fd53006fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5300c8500 0x7fd5301503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 --2- 172.21.15.78:0/2113038858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5300c2f50 0x7fd5301eb5c0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 >> 172.21.15.78:0/2113038858 conn(0x7fd5300aa300 msgr2=0x7fd5301549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:41.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 shutdown_connections 2024-02-19T14:59:41.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.765+0000 7fd5381bd640 1 -- 172.21.15.78:0/2113038858 wait complete. 2024-02-19T14:59:41.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 Processor -- start 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 -- start start 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc152480 0x7efcbc1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efcbc0c7070 0x7efcbc06ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc155ef0 con 0x7efcbc06d350 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc06dec0 con 0x7efcbc0c7070 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcc2264640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0c75c0 con 0x7efcbc152480 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcbb7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efcbc0c7070 0x7efcbc06ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.884+0000 7efcbb7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efcbc0c7070 0x7efcbc06ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35582/0 (socket says 172.21.15.78:35582) 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbb7fe640 1 -- 172.21.15.78:0/2889040258 learned_addr learned my addr 172.21.15.78:0/2889040258 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbbfff640 1 --2- 172.21.15.78:0/2889040258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc152480 0x7efcbc1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbb7fe640 1 -- 172.21.15.78:0/2889040258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc152480 msgr2=0x7efcbc1559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbb7fe640 1 --2- 172.21.15.78:0/2889040258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc152480 0x7efcbc1559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbb7fe640 1 -- 172.21.15.78:0/2889040258 >> v1:172.21.15.150:6789/0 conn(0x7efcbc06d350 legacy=0x7efcbc0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbb7fe640 1 -- 172.21.15.78:0/2889040258 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc0c78a0 con 0x7efcbc0c7070 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.885+0000 7efcbb7fe640 1 --2- 172.21.15.78:0/2889040258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc0c7070 0x7efcbc06ce10 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7efcb00098a0 tx=0x7efcb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade2bcca4a28f197 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcbaffd640 1 -- 172.21.15.78:0/2889040258 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0030840 con 0x7efcbc0c7070 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcbaffd640 1 -- 172.21.15.78:0/2889040258 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb00309a0 con 0x7efcbc0c7070 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcbaffd640 1 -- 172.21.15.78:0/2889040258 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0030d30 con 0x7efcbc0c7070 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 -- 172.21.15.78:0/2889040258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc0c7070 msgr2=0x7efcbc06ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 --2- 172.21.15.78:0/2889040258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc0c7070 0x7efcbc06ce10 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7efcb00098a0 tx=0x7efcb002b980 comp rx=0 tx=0).stop 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 -- 172.21.15.78:0/2889040258 shutdown_connections 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 --2- 172.21.15.78:0/2889040258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc0c7070 0x7efcbc06ce10 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 --2- 172.21.15.78:0/2889040258 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc152480 0x7efcbc1559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 -- 172.21.15.78:0/2889040258 >> 172.21.15.78:0/2889040258 conn(0x7efcbc0ca8a0 msgr2=0x7efcbc0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 -- 172.21.15.78:0/2889040258 shutdown_connections 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.886+0000 7efcc2264640 1 -- 172.21.15.78:0/2889040258 wait complete. 2024-02-19T14:59:41.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.887+0000 7efcc2264640 1 Processor -- start 2024-02-19T14:59:41.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 -- start start 2024-02-19T14:59:41.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc0c7070 0x7efcbc158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 0x7efcbc158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc0ab570 0x7efcbc0ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc157d20 con 0x7efcbc152480 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0aa280 con 0x7efcbc0ab570 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc2264640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0bdc10 con 0x7efcbc0c7070 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcbb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 0x7efcbc158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcbb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 0x7efcbc158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35594/0 (socket says 172.21.15.78:35594) 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcbb7fe640 1 -- 172.21.15.78:0/911518121 learned_addr learned my addr 172.21.15.78:0/911518121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcbbfff640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc0c7070 0x7efcbc158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.888+0000 7efcc0801640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc0ab570 0x7efcbc0ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcbb7fe640 1 -- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc0c7070 msgr2=0x7efcbc158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcbb7fe640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc0c7070 0x7efcbc158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcbb7fe640 1 -- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc0ab570 msgr2=0x7efcbc0ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcbb7fe640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc0ab570 0x7efcbc0ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcbb7fe640 1 -- 172.21.15.78:0/911518121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00094d0 con 0x7efcbc152480 2024-02-19T14:59:41.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcbb7fe640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 0x7efcbc158740 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7efcb0000c00 tx=0x7efcb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0039c90 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb0039df0 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0043640 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc0aa520 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.889+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc158fc0 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.890+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc157f20 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.890+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efcb00437a0 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.891+0000 7efcb8ff9640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca806b810 0x7efca806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.891+0000 7efcbbfff640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca806b810 0x7efca806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.891+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6217+0+0 (secure 0 0 0) 0x7efcb003e070 con 0x7efcbc152480 2024-02-19T14:59:41.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.891+0000 7efcbbfff640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca806b810 0x7efca806dcd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efcac009aa0 tx=0x7efcac009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:41.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:41.895+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efcbc157f20 con 0x7efcbc152480 2024-02-19T14:59:42.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.066+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcbc157f20 con 0x7efcbc152480 2024-02-19T14:59:42.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.067+0000 7efcb8ff9640 1 -- 172.21.15.78:0/911518121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v189) v1 ==== 104+0+11 (secure 0 0 0) 0x7efcbc157f20 con 0x7efcbc152480 2024-02-19T14:59:42.070 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T14:59:42.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca806b810 msgr2=0x7efca806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca806b810 0x7efca806dcd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efcac009aa0 tx=0x7efcac009190 comp rx=0 tx=0).stop 2024-02-19T14:59:42.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 msgr2=0x7efcbc158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 0x7efcbc158740 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7efcb0000c00 tx=0x7efcb0004730 comp rx=0 tx=0).stop 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 shutdown_connections 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efca806b810 0x7efca806dcd0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcbc152480 0x7efcbc158740 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcbc0ab570 0x7efcbc0ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 --2- 172.21.15.78:0/911518121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcbc0c7070 0x7efcbc158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.068+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 >> 172.21.15.78:0/911518121 conn(0x7efcbc0ca8a0 msgr2=0x7efcbc1541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.069+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 shutdown_connections 2024-02-19T14:59:42.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.069+0000 7efcc2264640 1 -- 172.21.15.78:0/911518121 wait complete. 2024-02-19T14:59:42.086 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T14:59:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f346a4cd640 1 Processor -- start 2024-02-19T14:59:42.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f346a4cd640 1 -- start start 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f346a4cd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f34640c5560 0x7f34640c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f346a4cd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640c65a0 con 0x7f34640c7910 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f346a4cd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640c7530 con 0x7f3464070720 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f346a4cd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34641537b0 con 0x7f34640c5560 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f3463fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f34640c5560 0x7f34640c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f34637fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f34640c7910 0x7f3464154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39490/0 (socket says 172.21.15.78:39490) 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.213+0000 7f34637fe640 1 -- 172.21.15.78:0/3172295480 learned_addr learned my addr 172.21.15.78:0/3172295480 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.214+0000 7f3463fff640 1 -- 172.21.15.78:0/3172295480 >> v1:172.21.15.150:6789/0 conn(0x7f3464070720 legacy=0x7f3464070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.214+0000 7f3463fff640 1 -- 172.21.15.78:0/3172295480 >> v1:172.21.15.78:6789/0 conn(0x7f34640c7910 legacy=0x7f3464154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.214+0000 7f3462ffd640 1 -- 172.21.15.78:0/3172295480 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417148496 0 0) 0x7f34640c7530 con 0x7f3464070720 2024-02-19T14:59:42.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.214+0000 7f3463fff640 1 -- 172.21.15.78:0/3172295480 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3464153a90 con 0x7f34640c5560 2024-02-19T14:59:42.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f3463fff640 1 --2- 172.21.15.78:0/3172295480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c5560 0x7f34640c6ff0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f34540097a0 tx=0x7f345402b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c823dfb83a5db server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f3462ffd640 1 -- 172.21.15.78:0/3172295480 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454030710 con 0x7f34640c5560 2024-02-19T14:59:42.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f3462ffd640 1 -- 172.21.15.78:0/3172295480 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345402e050 con 0x7f34640c5560 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f3462ffd640 1 -- 172.21.15.78:0/3172295480 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454030a90 con 0x7f34640c5560 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 -- 172.21.15.78:0/3172295480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c5560 msgr2=0x7f34640c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3172295480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c5560 0x7f34640c6ff0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f34540097a0 tx=0x7f345402b680 comp rx=0 tx=0).stop 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 -- 172.21.15.78:0/3172295480 shutdown_connections 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3172295480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c5560 0x7f34640c6ff0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 -- 172.21.15.78:0/3172295480 >> 172.21.15.78:0/3172295480 conn(0x7f346406b4b0 msgr2=0x7f34640c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 -- 172.21.15.78:0/3172295480 shutdown_connections 2024-02-19T14:59:42.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.215+0000 7f346a4cd640 1 -- 172.21.15.78:0/3172295480 wait complete. 2024-02-19T14:59:42.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.216+0000 7f346a4cd640 1 Processor -- start 2024-02-19T14:59:42.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 -- start start 2024-02-19T14:59:42.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3464070720 0x7f34640c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34640c5560 0x7f34640a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 0x7f346414c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34641eabc0 con 0x7f34640c7910 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34641eaea0 con 0x7f3464070720 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f346a4cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34641eb180 con 0x7f34640c5560 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f3468a6a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 0x7f346414c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f3468a6a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 0x7f346414c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35614/0 (socket says 172.21.15.78:35614) 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f3468a6a640 1 -- 172.21.15.78:0/3618022636 learned_addr learned my addr 172.21.15.78:0/3618022636 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f34637fe640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34640c5560 0x7f34640a70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.217+0000 7f3463fff640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3464070720 0x7f34640c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3468a6a640 1 -- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34640c5560 msgr2=0x7f34640a70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3468a6a640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34640c5560 0x7f34640a70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3468a6a640 1 -- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3464070720 msgr2=0x7f34640c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3468a6a640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3464070720 0x7f34640c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3468a6a640 1 -- 172.21.15.78:0/3618022636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454009400 con 0x7f34640c7910 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3468a6a640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 0x7f346414c900 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f345c00c780 tx=0x7f345c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c003b70 con 0x7f34640c7910 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c003cd0 con 0x7f34640c7910 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c005270 con 0x7f34640c7910 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34640c5ac0 con 0x7f34640c7910 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.218+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34641eb740 con 0x7f34640c7910 2024-02-19T14:59:42.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.219+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3464074990 con 0x7f34640c7910 2024-02-19T14:59:42.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.219+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f345c0053d0 con 0x7f34640c7910 2024-02-19T14:59:42.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.220+0000 7f3460ff9640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f345006b840 0x7f345006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.220+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f345c089f50 con 0x7f34640c7910 2024-02-19T14:59:42.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.220+0000 7f3463fff640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f345006b840 0x7f345006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.220+0000 7f3463fff640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f345006b840 0x7f345006dd00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3454034000 tx=0x7f3454008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.224+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f345c0548b0 con 0x7f34640c7910 2024-02-19T14:59:42.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.394+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34641502d0 con 0x7f34640c7910 2024-02-19T14:59:42.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.395+0000 7f3460ff9640 1 -- 172.21.15.78:0/3618022636 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v189) v1 ==== 100+0+11 (secure 0 0 0) 0x7f345c057f70 con 0x7f34640c7910 2024-02-19T14:59:42.398 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f345006b840 msgr2=0x7f345006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f345006b840 0x7f345006dd00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3454034000 tx=0x7f3454008030 comp rx=0 tx=0).stop 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 msgr2=0x7f346414c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 0x7f346414c900 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f345c00c780 tx=0x7f345c00cb80 comp rx=0 tx=0).stop 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 shutdown_connections 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f345006b840 0x7f345006dd00 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34640c7910 0x7f346414c900 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34640c5560 0x7f34640a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 --2- 172.21.15.78:0/3618022636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3464070720 0x7f34640c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 >> 172.21.15.78:0/3618022636 conn(0x7f346406b4b0 msgr2=0x7f3464070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:42.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 shutdown_connections 2024-02-19T14:59:42.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.397+0000 7f346a4cd640 1 -- 172.21.15.78:0/3618022636 wait complete. 2024-02-19T14:59:42.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:42.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.508+0000 7f31719e9640 1 Processor -- start 2024-02-19T14:59:42.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f31719e9640 1 -- start start 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f31719e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f316c0c25e0 0x7f316c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f31719e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f31719e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316c072250 con 0x7f316c0c29c0 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f31719e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c152c90 con 0x7f316c0c25e0 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f31719e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c072480 con 0x7f316c0c5be0 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f316affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f316c0c25e0 0x7f316c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f316affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f316c0c25e0 0x7f316c1526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35622/0 (socket says 172.21.15.78:35622) 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f316affd640 1 -- 172.21.15.78:0/1850323133 learned_addr learned my addr 172.21.15.78:0/1850323133 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.509+0000 7f316b7fe640 1 --2- 172.21.15.78:0/1850323133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.510+0000 7f3169ffb640 1 -- 172.21.15.78:0/1850323133 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519440591 0 0) 0x7f316c072250 con 0x7f316c0c29c0 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.510+0000 7f316affd640 1 -- 172.21.15.78:0/1850323133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 msgr2=0x7f316c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.510+0000 7f316affd640 1 --2- 172.21.15.78:0/1850323133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.510+0000 7f316affd640 1 -- 172.21.15.78:0/1850323133 >> v1:172.21.15.150:6789/0 conn(0x7f316c0c29c0 legacy=0x7f316c0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.510+0000 7f316affd640 1 -- 172.21.15.78:0/1850323133 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c0c4b90 con 0x7f316c0c25e0 2024-02-19T14:59:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.511+0000 7f316affd640 1 --2- 172.21.15.78:0/1850323133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c1526a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f315c009870 tx=0x7f315c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b297af2d3e7fb009 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.511+0000 7f3169ffb640 1 -- 172.21.15.78:0/1850323133 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c031640 con 0x7f316c0c25e0 2024-02-19T14:59:42.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.511+0000 7f3169ffb640 1 -- 172.21.15.78:0/1850323133 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c0317a0 con 0x7f316c0c25e0 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.511+0000 7f3169ffb640 1 -- 172.21.15.78:0/1850323133 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c031b30 con 0x7f316c0c25e0 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.511+0000 7f31719e9640 1 -- 172.21.15.78:0/1850323133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 msgr2=0x7f316c1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.511+0000 7f31719e9640 1 --2- 172.21.15.78:0/1850323133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c1526a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f315c009870 tx=0x7f315c02b8f0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.512+0000 7f31719e9640 1 -- 172.21.15.78:0/1850323133 shutdown_connections 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.512+0000 7f31719e9640 1 --2- 172.21.15.78:0/1850323133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.512+0000 7f31719e9640 1 --2- 172.21.15.78:0/1850323133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c1526a0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.512+0000 7f31719e9640 1 -- 172.21.15.78:0/1850323133 >> 172.21.15.78:0/1850323133 conn(0x7f316c06b270 msgr2=0x7f316c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.512+0000 7f31719e9640 1 -- 172.21.15.78:0/1850323133 shutdown_connections 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.512+0000 7f31719e9640 1 -- 172.21.15.78:0/1850323133 wait complete. 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 Processor -- start 2024-02-19T14:59:42.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 -- start start 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c14fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f316c0c29c0 0x7f316c146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c1ea900 con 0x7f316c0c25e0 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0a6660 con 0x7f316c0c29c0 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.513+0000 7f31719e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c1eab50 con 0x7f316c0c5be0 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c14fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c14fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35636/0 (socket says 172.21.15.78:35636) 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 -- 172.21.15.78:0/1640097211 learned_addr learned my addr 172.21.15.78:0/1640097211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316b7fe640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316a7fc640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f316c0c29c0 0x7f316c146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 -- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 msgr2=0x7f316c146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c146d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 -- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f316c0c29c0 msgr2=0x7f316c146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f316c0c29c0 0x7f316c146830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 -- 172.21.15.78:0/1640097211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0094d0 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f316affd640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c14fd30 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f315c034040 tx=0x7f315c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c004230 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c0043b0 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c004520 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c1eae30 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.514+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c150270 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.515+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316c1eb350 con 0x7f316c0c25e0 2024-02-19T14:59:42.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.516+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f315c02f050 con 0x7f316c0c25e0 2024-02-19T14:59:42.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.516+0000 7f3153fff640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f315806b810 0x7f315806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.516+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f315c0aef70 con 0x7f316c0c25e0 2024-02-19T14:59:42.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.516+0000 7f316a7fc640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f315806b810 0x7f315806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.517+0000 7f316a7fc640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f315806b810 0x7f315806dcd0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f316c1e51d0 tx=0x7f3160009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.524+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f315c0e2dc0 con 0x7f316c0c25e0 2024-02-19T14:59:42.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.733+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f316c074990 con 0x7f316c0c25e0 2024-02-19T14:59:42.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.733+0000 7f3153fff640 1 -- 172.21.15.78:0/1640097211 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4966 (secure 0 0 0) 0x7f315c035090 con 0x7f316c0c25e0 2024-02-19T14:59:42.736 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:42.737 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:37.764041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9816,"mds_1":8758},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8758":{"gid":8758,"name":"d","rank":1,"incarnation":173,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 174 2024-02-19T14:59:42.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.735+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f315806b810 msgr2=0x7f315806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f315806b810 0x7f315806dcd0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f316c1e51d0 tx=0x7f3160009040 comp rx=0 tx=0).stop 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 msgr2=0x7f316c14fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c14fd30 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f315c034040 tx=0x7f315c0046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 shutdown_connections 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f315806b810 0x7f315806dcd0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f316c0c5be0 0x7f316c146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f316c0c29c0 0x7f316c146830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 --2- 172.21.15.78:0/1640097211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f316c0c25e0 0x7f316c14fd30 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 >> 172.21.15.78:0/1640097211 conn(0x7f316c06b270 msgr2=0x7f316c151210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 shutdown_connections 2024-02-19T14:59:42.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.736+0000 7f31719e9640 1 -- 172.21.15.78:0/1640097211 wait complete. 2024-02-19T14:59:42.749 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 Processor -- start 2024-02-19T14:59:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 -- start start 2024-02-19T14:59:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c0c7b70 0x7f145c0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f145c06d040 0x7f145c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c06db90 con 0x7f145c155bb0 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145c06d990 con 0x7f145c06d040 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f1461a47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145c154ec0 con 0x7f145c0c7b70 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f145b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f145c06d040 0x7f145c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f145b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f145c06d040 0x7f145c06d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35648/0 (socket says 172.21.15.78:35648) 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.852+0000 7f145b7fe640 1 -- 172.21.15.78:0/945201089 learned_addr learned my addr 172.21.15.78:0/945201089 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.853+0000 7f145affd640 1 --2- 172.21.15.78:0/945201089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c0c7b70 0x7f145c0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.853+0000 7f145b7fe640 1 -- 172.21.15.78:0/945201089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c0c7b70 msgr2=0x7f145c0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.853+0000 7f145b7fe640 1 --2- 172.21.15.78:0/945201089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c0c7b70 0x7f145c0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.853+0000 7f145b7fe640 1 -- 172.21.15.78:0/945201089 >> v1:172.21.15.150:6789/0 conn(0x7f145c155bb0 legacy=0x7f145c1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.853+0000 7f145b7fe640 1 -- 172.21.15.78:0/945201089 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c1550d0 con 0x7f145c06d040 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.853+0000 7f145b7fe640 1 --2- 172.21.15.78:0/945201089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c06d450 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f1454009870 tx=0x7f145402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d903feab467ccf2b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1459ffb640 1 -- 172.21.15.78:0/945201089 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1454031640 con 0x7f145c06d040 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1459ffb640 1 -- 172.21.15.78:0/945201089 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14540317a0 con 0x7f145c06d040 2024-02-19T14:59:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1459ffb640 1 -- 172.21.15.78:0/945201089 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1454031b30 con 0x7f145c06d040 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 -- 172.21.15.78:0/945201089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 msgr2=0x7f145c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 --2- 172.21.15.78:0/945201089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c06d450 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f1454009870 tx=0x7f145402b8f0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 -- 172.21.15.78:0/945201089 shutdown_connections 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 --2- 172.21.15.78:0/945201089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c06d450 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 --2- 172.21.15.78:0/945201089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c0c7b70 0x7f145c0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 -- 172.21.15.78:0/945201089 >> 172.21.15.78:0/945201089 conn(0x7f145c0c5050 msgr2=0x7f145c0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.854+0000 7f1461a47640 1 -- 172.21.15.78:0/945201089 shutdown_connections 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.855+0000 7f1461a47640 1 -- 172.21.15.78:0/945201089 wait complete. 2024-02-19T14:59:42.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 Processor -- start 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 -- start start 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f145c0c7b70 0x7f145c0ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c155bb0 0x7f145c1478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c148590 con 0x7f145c06d040 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0ae300 con 0x7f145c0c7b70 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.856+0000 7f1461a47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145c148c80 con 0x7f145c155bb0 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35662/0 (socket says 172.21.15.78:35662) 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 -- 172.21.15.78:0/1224220498 learned_addr learned my addr 172.21.15.78:0/1224220498 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145b7fe640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c155bb0 0x7f145c1478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145a7fc640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f145c0c7b70 0x7f145c0ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 -- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c155bb0 msgr2=0x7f145c1478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c155bb0 0x7f145c1478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 -- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f145c0c7b70 msgr2=0x7f145c0ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f145c0c7b70 0x7f145c0ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 -- 172.21.15.78:0/1224220498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14540094d0 con 0x7f145c06d040 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f145affd640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c157e60 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f144c00efc0 tx=0x7f144c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c014480 con 0x7f145c06d040 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c00f040 con 0x7f145c06d040 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c014800 con 0x7f145c06d040 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c1487e0 con 0x7f145c06d040 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.857+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c1583a0 con 0x7f145c06d040 2024-02-19T14:59:42.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.858+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145c148a20 con 0x7f145c06d040 2024-02-19T14:59:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.859+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f144c014960 con 0x7f145c06d040 2024-02-19T14:59:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.859+0000 7f1443fff640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f144806b810 0x7f144806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.859+0000 7f145a7fc640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f144806b810 0x7f144806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.859+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f144c08b160 con 0x7f145c06d040 2024-02-19T14:59:42.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.860+0000 7f145a7fc640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f144806b810 0x7f144806dcd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f145c154760 tx=0x7f1450009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:42.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:42.864+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f144c055ac0 con 0x7f145c06d040 2024-02-19T14:59:43.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.047+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f145c074990 con 0x7f145c06d040 2024-02-19T14:59:43.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.049+0000 7f1443fff640 1 -- 172.21.15.78:0/1224220498 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+17067 (secure 0 0 0) 0x7f144c059180 con 0x7f145c06d040 2024-02-19T14:59:43.052 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:43.052 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":190,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:42.772487+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:06.192227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:06.947291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.3","mappings":[{"from":6,"to":0}]},{"pgid":"29.4","mappings":[{"from":6,"to":0}]},{"pgid":"29.7","mappings":[{"from":6,"to":0}]},{"pgid":"29.1d","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:43.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.050+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f144806b810 msgr2=0x7f144806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.050+0000 7f1461a47640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f144806b810 0x7f144806dcd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f145c154760 tx=0x7f1450009040 comp rx=0 tx=0).stop 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.050+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 msgr2=0x7f145c157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.050+0000 7f1461a47640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c157e60 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f144c00efc0 tx=0x7f144c00c490 comp rx=0 tx=0).stop 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 shutdown_connections 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f144806b810 0x7f144806dcd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f145c155bb0 0x7f145c1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f145c0c7b70 0x7f145c0ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 --2- 172.21.15.78:0/1224220498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f145c06d040 0x7f145c157e60 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 >> 172.21.15.78:0/1224220498 conn(0x7f145c0c5050 msgr2=0x7f145c0703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 shutdown_connections 2024-02-19T14:59:43.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.051+0000 7f1461a47640 1 -- 172.21.15.78:0/1224220498 wait complete. 2024-02-19T14:59:43.065 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T14:59:43.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:43.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.163+0000 7f5223e5c640 1 Processor -- start 2024-02-19T14:59:43.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.163+0000 7f5223e5c640 1 -- start start 2024-02-19T14:59:43.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f5223e5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c0c6870 0x7f521c0c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f5223e5c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c0c6460 con 0x7f521c155070 2024-02-19T14:59:43.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f5223e5c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c158720 con 0x7f521c1525c0 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f5223e5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0c1f50 con 0x7f521c0c6870 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f52223f9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f521c155070 0x7f521c155500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39504/0 (socket says 172.21.15.78:39504) 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f52223f9640 1 -- 172.21.15.78:0/1021526082 learned_addr learned my addr 172.21.15.78:0/1021526082 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f52213f7640 1 --2- 172.21.15.78:0/1021526082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c0c6870 0x7f521c0c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f5220bf6640 1 -- 172.21.15.78:0/1021526082 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305441524 0 0) 0x7f521c0c6460 con 0x7f521c155070 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f5220bf6640 1 -- 172.21.15.78:0/1021526082 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5208003520 con 0x7f521c155070 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.164+0000 7f52213f7640 1 -- 172.21.15.78:0/1021526082 >> v1:172.21.15.150:6789/0 conn(0x7f521c1525c0 legacy=0x7f521c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.165+0000 7f52213f7640 1 -- 172.21.15.78:0/1021526082 >> v1:172.21.15.78:6789/0 conn(0x7f521c155070 legacy=0x7f521c155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.165+0000 7f5220bf6640 1 -- 172.21.15.78:0/1021526082 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599895037 0 0) 0x7f521c158720 con 0x7f521c1525c0 2024-02-19T14:59:43.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.165+0000 7f52213f7640 1 -- 172.21.15.78:0/1021526082 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c158a00 con 0x7f521c0c6870 2024-02-19T14:59:43.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.165+0000 7f52213f7640 1 --2- 172.21.15.78:0/1021526082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c0c6870 0x7f521c0c6cd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f52180097d0 tx=0x7f521802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c169fefb26217c70 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.165+0000 7f5220bf6640 1 -- 172.21.15.78:0/1021526082 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218031830 con 0x7f521c0c6870 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5220bf6640 1 -- 172.21.15.78:0/1021526082 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5218031990 con 0x7f521c0c6870 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5220bf6640 1 -- 172.21.15.78:0/1021526082 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218031d20 con 0x7f521c0c6870 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 -- 172.21.15.78:0/1021526082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c0c6870 msgr2=0x7f521c0c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1021526082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c0c6870 0x7f521c0c6cd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f52180097d0 tx=0x7f521802b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 -- 172.21.15.78:0/1021526082 shutdown_connections 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1021526082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c0c6870 0x7f521c0c6cd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 -- 172.21.15.78:0/1021526082 >> 172.21.15.78:0/1021526082 conn(0x7f521c0cab20 msgr2=0x7f521c0caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 -- 172.21.15.78:0/1021526082 shutdown_connections 2024-02-19T14:59:43.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.166+0000 7f5223e5c640 1 -- 172.21.15.78:0/1021526082 wait complete. 2024-02-19T14:59:43.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.167+0000 7f5223e5c640 1 Processor -- start 2024-02-19T14:59:43.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.167+0000 7f5223e5c640 1 -- start start 2024-02-19T14:59:43.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5223e5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f521c0c6870 0x7f521c0aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5223e5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c1525c0 0x7f521c0c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5223e5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 0x7f521c141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5223e5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c1593a0 con 0x7f521c155070 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5223e5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c151eb0 con 0x7f521c0c6870 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5223e5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0aa970 con 0x7f521c1525c0 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 0x7f521c141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 0x7f521c141140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35668/0 (socket says 172.21.15.78:35668) 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 -- 172.21.15.78:0/1737377180 learned_addr learned my addr 172.21.15.78:0/1737377180 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f5221bf8640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f521c0c6870 0x7f521c0aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52213f7640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c1525c0 0x7f521c0c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 -- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c1525c0 msgr2=0x7f521c0c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c1525c0 0x7f521c0c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 -- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f521c0c6870 msgr2=0x7f521c0aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f521c0c6870 0x7f521c0aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 -- 172.21.15.78:0/1737377180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5218009400 con 0x7f521c155070 2024-02-19T14:59:43.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.168+0000 7f52223f9640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 0x7f521c141140 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f521400c9f0 tx=0x7f521400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.169+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400ab70 con 0x7f521c155070 2024-02-19T14:59:43.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.169+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521400acd0 con 0x7f521c155070 2024-02-19T14:59:43.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.169+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c1eb000 con 0x7f521c155070 2024-02-19T14:59:43.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.169+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214012600 con 0x7f521c155070 2024-02-19T14:59:43.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.169+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c1eb5c0 con 0x7f521c155070 2024-02-19T14:59:43.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.170+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c06b240 con 0x7f521c155070 2024-02-19T14:59:43.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.170+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5214007070 con 0x7f521c155070 2024-02-19T14:59:43.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.171+0000 7f52127fc640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520006b840 0x7f520006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.171+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f521408a0c0 con 0x7f521c155070 2024-02-19T14:59:43.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.171+0000 7f5221bf8640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520006b840 0x7f520006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.171+0000 7f5221bf8640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520006b840 0x7f520006dd00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f520c000f60 tx=0x7f520c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.175+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5214054a20 con 0x7f521c155070 2024-02-19T14:59:43.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.497+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f521c1eada0 con 0x7f521c155070 2024-02-19T14:59:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.497+0000 7f52127fc640 1 -- 172.21.15.78:0/1737377180 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4966 (secure 0 0 0) 0x7f52140580e0 con 0x7f521c155070 2024-02-19T14:59:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:43.500 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:37.764041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9816,"mds_1":8758},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8758":{"gid":8758,"name":"d","rank":1,"incarnation":173,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:43.500 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 174 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520006b840 msgr2=0x7f520006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520006b840 0x7f520006dd00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f520c000f60 tx=0x7f520c009040 comp rx=0 tx=0).stop 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 msgr2=0x7f521c141140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 0x7f521c141140 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f521400c9f0 tx=0x7f521400ceb0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 shutdown_connections 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f520006b840 0x7f520006dd00 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f521c155070 0x7f521c141140 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f521c1525c0 0x7f521c0c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 --2- 172.21.15.78:0/1737377180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f521c0c6870 0x7f521c0aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 >> 172.21.15.78:0/1737377180 conn(0x7f521c0cab20 msgr2=0x7f521c0c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:43.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 shutdown_connections 2024-02-19T14:59:43.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.500+0000 7f5223e5c640 1 -- 172.21.15.78:0/1737377180 wait complete. 2024-02-19T14:59:43.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:43.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.611+0000 7f31e37d0640 1 Processor -- start 2024-02-19T14:59:43.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.611+0000 7f31e37d0640 1 -- start start 2024-02-19T14:59:43.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e37d0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31dc1528d0 0x7f31dc0c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e37d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc155780 0x7f31dc15f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e37d0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc072e80 con 0x7f31dc0c19c0 2024-02-19T14:59:43.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e37d0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc0734c0 con 0x7f31dc1528d0 2024-02-19T14:59:43.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e37d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc06e990 con 0x7f31dc155780 2024-02-19T14:59:43.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e0d6b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f31dc0c19c0 0x7f31dc155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39518/0 (socket says 172.21.15.78:39518) 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e0d6b640 1 -- 172.21.15.78:0/1469846285 learned_addr learned my addr 172.21.15.78:0/1469846285 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e1d6d640 1 --2- 172.21.15.78:0/1469846285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc155780 0x7f31dc15f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e156c640 1 --2- 172.21.15.78:0/1469846285 >> v2:172.21.15.150:3300/0 conn(0x7f31dc1528d0 0x7f31dc0c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31cffff640 1 -- 172.21.15.78:0/1469846285 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057653507 0 0) 0x7f31d8001110 con 0x7f31dc0c19c0 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31cffff640 1 -- 172.21.15.78:0/1469846285 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c8003520 con 0x7f31dc0c19c0 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e1d6d640 1 -- 172.21.15.78:0/1469846285 >> v2:172.21.15.150:3300/0 conn(0x7f31dc1528d0 msgr2=0x7f31dc0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e1d6d640 1 --2- 172.21.15.78:0/1469846285 >> v2:172.21.15.150:3300/0 conn(0x7f31dc1528d0 0x7f31dc0c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e1d6d640 1 -- 172.21.15.78:0/1469846285 >> v1:172.21.15.78:6789/0 conn(0x7f31dc0c19c0 legacy=0x7f31dc155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e1d6d640 1 -- 172.21.15.78:0/1469846285 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0731e0 con 0x7f31dc155780 2024-02-19T14:59:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.612+0000 7f31e156c640 1 --2- 172.21.15.78:0/1469846285 >> v2:172.21.15.150:3300/0 conn(0x7f31dc1528d0 0x7f31dc0c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.613+0000 7f31e1d6d640 1 --2- 172.21.15.78:0/1469846285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc155780 0x7f31dc15f910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009d90 tx=0x7f31d402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf356eca3d32954 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.613+0000 7f31cffff640 1 -- 172.21.15.78:0/1469846285 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d402cbc0 con 0x7f31dc155780 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31cffff640 1 -- 172.21.15.78:0/1469846285 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d402cd20 con 0x7f31dc155780 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31cffff640 1 -- 172.21.15.78:0/1469846285 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4034660 con 0x7f31dc155780 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 -- 172.21.15.78:0/1469846285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc155780 msgr2=0x7f31dc15f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 --2- 172.21.15.78:0/1469846285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc155780 0x7f31dc15f910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009d90 tx=0x7f31d402be10 comp rx=0 tx=0).stop 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 -- 172.21.15.78:0/1469846285 shutdown_connections 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 --2- 172.21.15.78:0/1469846285 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc155780 0x7f31dc15f910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 --2- 172.21.15.78:0/1469846285 >> v2:172.21.15.150:3300/0 conn(0x7f31dc1528d0 0x7f31dc0c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 -- 172.21.15.78:0/1469846285 >> 172.21.15.78:0/1469846285 conn(0x7f31dc0c7590 msgr2=0x7f31dc0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:43.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 -- 172.21.15.78:0/1469846285 shutdown_connections 2024-02-19T14:59:43.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.614+0000 7f31e37d0640 1 -- 172.21.15.78:0/1469846285 wait complete. 2024-02-19T14:59:43.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.615+0000 7f31e37d0640 1 Processor -- start 2024-02-19T14:59:43.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.615+0000 7f31e37d0640 1 -- start start 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e37d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc0c19c0 0x7f31dc1e1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e37d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc1528d0 0x7f31dc1e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e37d0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 0x7f31dc0a8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e37d0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc14f500 con 0x7f31dc155780 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e37d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc1e1750 con 0x7f31dc0c19c0 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e37d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc1e0d30 con 0x7f31dc1528d0 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 0x7f31dc0a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 0x7f31dc0a8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35680/0 (socket says 172.21.15.78:35680) 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 -- 172.21.15.78:0/2670153230 learned_addr learned my addr 172.21.15.78:0/2670153230 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e0d6b640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc1528d0 0x7f31dc1e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e156c640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc0c19c0 0x7f31dc1e1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 -- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc1528d0 msgr2=0x7f31dc1e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc1528d0 0x7f31dc1e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 -- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc0c19c0 msgr2=0x7f31dc1e1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc0c19c0 0x7f31dc1e1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 -- 172.21.15.78:0/2670153230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0009400 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.616+0000 7f31e1d6d640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 0x7f31dc0a8310 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009d90 tx=0x7f31d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.617+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4040650 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.617+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d402ce90 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.617+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d4009a70 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.617+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d40409f0 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.617+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc0a7630 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.618+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc1e0f30 con 0x7f31dc155780 2024-02-19T14:59:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.618+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f31d4040b50 con 0x7f31dc155780 2024-02-19T14:59:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.619+0000 7f31cdffb640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c006b810 0x7f31c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.619+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f31d40acfe0 con 0x7f31dc155780 2024-02-19T14:59:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.619+0000 7f31e156c640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c006b810 0x7f31c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.619+0000 7f31e156c640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c006b810 0x7f31c006dcd0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f31dc1e1af0 tx=0x7f31d0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.623+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31d4077a20 con 0x7f31dc155780 2024-02-19T14:59:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.833+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31dc1e0f30 con 0x7f31dc155780 2024-02-19T14:59:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.833+0000 7f31cdffb640 1 -- 172.21.15.78:0/2670153230 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4966 (secure 0 0 0) 0x7f31dc1e0f30 con 0x7f31dc155780 2024-02-19T14:59:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:43.836 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:07.976421+0000","modified":"2024-02-19T14:59:37.764041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9816,"mds_1":8758},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8758":{"gid":8758,"name":"d","rank":1,"incarnation":173,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2126014034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2126014034},{"type":"v1","addr":"172.21.15.150:6835","nonce":2126014034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9816":{"gid":9816,"name":"c","rank":0,"incarnation":171,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6835/778404633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":778404633},{"type":"v1","addr":"172.21.15.78:6835","nonce":778404633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-19T14:59:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 174 2024-02-19T14:59:43.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c006b810 msgr2=0x7f31c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c006b810 0x7f31c006dcd0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f31dc1e1af0 tx=0x7f31d0009190 comp rx=0 tx=0).stop 2024-02-19T14:59:43.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 msgr2=0x7f31dc0a8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 0x7f31dc0a8310 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009d90 tx=0x7f31d4004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 shutdown_connections 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f31c006b810 0x7f31c006dcd0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31dc155780 0x7f31dc0a8310 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31dc1528d0 0x7f31dc1e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 --2- 172.21.15.78:0/2670153230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc0c19c0 0x7f31dc1e1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 >> 172.21.15.78:0/2670153230 conn(0x7f31dc0c7590 msgr2=0x7f31dc0c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 shutdown_connections 2024-02-19T14:59:43.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.835+0000 7f31e37d0640 1 -- 172.21.15.78:0/2670153230 wait complete. 2024-02-19T14:59:43.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:43.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.951+0000 7febbe13e640 1 Processor -- start 2024-02-19T14:59:43.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.951+0000 7febbe13e640 1 -- start start 2024-02-19T14:59:43.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febbe13e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb80c6870 0x7febb80c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febbe13e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febb8155070 0x7febb8155500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febbe13e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80c6460 con 0x7febb81525c0 2024-02-19T14:59:43.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febbe13e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb80c1f50 con 0x7febb8155070 2024-02-19T14:59:43.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febbe13e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb81593a0 con 0x7febb80c6870 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febaeffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb80c6870 0x7febb80c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febb7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febb8155070 0x7febb8155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febb77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7febb81525c0 0x7febb806d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:51796/0 (socket says 172.21.15.78:51796) 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febb77fe640 1 -- 172.21.15.78:0/980198086 learned_addr learned my addr 172.21.15.78:0/980198086 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febaeffd640 1 -- 172.21.15.78:0/980198086 >> v2:172.21.15.150:3300/0 conn(0x7febb8155070 msgr2=0x7febb8155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febaeffd640 1 --2- 172.21.15.78:0/980198086 >> v2:172.21.15.150:3300/0 conn(0x7febb8155070 0x7febb8155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febaeffd640 1 -- 172.21.15.78:0/980198086 >> v1:172.21.15.150:6789/0 conn(0x7febb81525c0 legacy=0x7febb806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febaeffd640 1 -- 172.21.15.78:0/980198086 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8158720 con 0x7febb80c6870 2024-02-19T14:59:43.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.952+0000 7febb7fff640 1 --2- 172.21.15.78:0/980198086 >> v2:172.21.15.150:3300/0 conn(0x7febb8155070 0x7febb8155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:43.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.953+0000 7febaeffd640 1 --2- 172.21.15.78:0/980198086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb80c6870 0x7febb80c6cd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feba00097a0 tx=0x7feba002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c7455d121a5fcf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.953+0000 7febb6ffd640 1 -- 172.21.15.78:0/980198086 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0031780 con 0x7febb80c6870 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.953+0000 7febb6ffd640 1 -- 172.21.15.78:0/980198086 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba00318e0 con 0x7febb80c6870 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.953+0000 7febb6ffd640 1 -- 172.21.15.78:0/980198086 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba0031c70 con 0x7febb80c6870 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 -- 172.21.15.78:0/980198086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb80c6870 msgr2=0x7febb80c6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 --2- 172.21.15.78:0/980198086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb80c6870 0x7febb80c6cd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feba00097a0 tx=0x7feba002b820 comp rx=0 tx=0).stop 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 -- 172.21.15.78:0/980198086 shutdown_connections 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 --2- 172.21.15.78:0/980198086 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb80c6870 0x7febb80c6cd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 --2- 172.21.15.78:0/980198086 >> v2:172.21.15.150:3300/0 conn(0x7febb8155070 0x7febb8155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 -- 172.21.15.78:0/980198086 >> 172.21.15.78:0/980198086 conn(0x7febb80cab20 msgr2=0x7febb80caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 -- 172.21.15.78:0/980198086 shutdown_connections 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.954+0000 7febbe13e640 1 -- 172.21.15.78:0/980198086 wait complete. 2024-02-19T14:59:43.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.955+0000 7febbe13e640 1 Processor -- start 2024-02-19T14:59:43.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.955+0000 7febbe13e640 1 -- start start 2024-02-19T14:59:43.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.956+0000 7febbe13e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 0x7febb80c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.956+0000 7febbe13e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febb81525c0 0x7febb814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.956+0000 7febbe13e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb8155070 0x7febb814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.956+0000 7febbe13e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb814d630 con 0x7febb80c6870 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.956+0000 7febbe13e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb81503e0 con 0x7febb81525c0 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.956+0000 7febbe13e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb814b7c0 con 0x7febb8155070 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 0x7febb80c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 0x7febb80c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35692/0 (socket says 172.21.15.78:35692) 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 -- 172.21.15.78:0/2213908908 learned_addr learned my addr 172.21.15.78:0/2213908908 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb7fff640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb8155070 0x7febb814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febaeffd640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febb81525c0 0x7febb814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 -- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb8155070 msgr2=0x7febb814fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb8155070 0x7febb814fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 -- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febb81525c0 msgr2=0x7febb814f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febb81525c0 0x7febb814f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 -- 172.21.15.78:0/2213908908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb0009a70 con 0x7febb80c6870 2024-02-19T14:59:43.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb77fe640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 0x7febb80c9a00 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7feba800c400 tx=0x7feba800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba8005ce0 con 0x7febb80c6870 2024-02-19T14:59:43.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba8005e40 con 0x7febb80c6870 2024-02-19T14:59:43.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba80072d0 con 0x7febb80c6870 2024-02-19T14:59:43.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0009400 con 0x7febb80c6870 2024-02-19T14:59:43.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.957+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb81ee660 con 0x7febb80c6870 2024-02-19T14:59:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.958+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb8074990 con 0x7febb80c6870 2024-02-19T14:59:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.959+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feba8007430 con 0x7febb80c6870 2024-02-19T14:59:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.959+0000 7febb4ff9640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb9806b840 0x7feb9806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.959+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6217+0+0 (secure 0 0 0) 0x7feba8089890 con 0x7febb80c6870 2024-02-19T14:59:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.959+0000 7febaeffd640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb9806b840 0x7feb9806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.959+0000 7febaeffd640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb9806b840 0x7feb9806dd00 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7feba00097a0 tx=0x7feba003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:43.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:43.963+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feba80541f0 con 0x7febb80c6870 2024-02-19T14:59:44.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.140+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febb80c6650 con 0x7febb80c6870 2024-02-19T14:59:44.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.142+0000 7febb4ff9640 1 -- 172.21.15.78:0/2213908908 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v190) v1 ==== 74+0+17067 (secure 0 0 0) 0x7feba80578b0 con 0x7febb80c6870 2024-02-19T14:59:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:44.145 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":190,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:42.772487+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:06.192227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:06.947291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.3","mappings":[{"from":6,"to":0}]},{"pgid":"29.4","mappings":[{"from":6,"to":0}]},{"pgid":"29.7","mappings":[{"from":6,"to":0}]},{"pgid":"29.1d","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.143+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb9806b840 msgr2=0x7feb9806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.143+0000 7febbe13e640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb9806b840 0x7feb9806dd00 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7feba00097a0 tx=0x7feba003d040 comp rx=0 tx=0).stop 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.143+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 msgr2=0x7febb80c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.143+0000 7febbe13e640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 0x7febb80c9a00 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7feba800c400 tx=0x7feba800c8c0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 shutdown_connections 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7feb9806b840 0x7feb9806dd00 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febb8155070 0x7febb814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febb81525c0 0x7febb814f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 --2- 172.21.15.78:0/2213908908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febb80c6870 0x7febb80c9a00 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 >> 172.21.15.78:0/2213908908 conn(0x7febb80cab20 msgr2=0x7febb806d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 shutdown_connections 2024-02-19T14:59:44.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.144+0000 7febbe13e640 1 -- 172.21.15.78:0/2213908908 wait complete. 2024-02-19T14:59:44.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T14:59:44.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.259+0000 7f8896e46640 1 Processor -- start 2024-02-19T14:59:44.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.259+0000 7f8896e46640 1 -- start start 2024-02-19T14:59:44.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.259+0000 7f8896e46640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 0x7f88901559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.259+0000 7f8896e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c87a0 0x7f88900c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f8896e46640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8890155ef0 con 0x7f8890152480 2024-02-19T14:59:44.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f8896e46640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889006dec0 con 0x7f88900c7070 2024-02-19T14:59:44.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f8896e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889006d520 con 0x7f88900c87a0 2024-02-19T14:59:44.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f888ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 0x7f88901559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f8894be2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8890152480 0x7f889006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39526/0 (socket says 172.21.15.78:39526) 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f888ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 0x7f88901559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35704/0 (socket says 172.21.15.78:35704) 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f8894be2640 1 -- 172.21.15.78:0/2040678273 learned_addr learned my addr 172.21.15.78:0/2040678273 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f88953e3640 1 --2- 172.21.15.78:0/2040678273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c87a0 0x7f88900c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f88953e3640 1 -- 172.21.15.78:0/2040678273 >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 msgr2=0x7f88901559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f88953e3640 1 --2- 172.21.15.78:0/2040678273 >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 0x7f88901559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f88953e3640 1 -- 172.21.15.78:0/2040678273 >> v1:172.21.15.78:6789/0 conn(0x7f8890152480 legacy=0x7f889006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f88953e3640 1 -- 172.21.15.78:0/2040678273 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900c75c0 con 0x7f88900c87a0 2024-02-19T14:59:44.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.260+0000 7f888ffff640 1 --2- 172.21.15.78:0/2040678273 >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 0x7f88901559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:44.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.261+0000 7f88953e3640 1 --2- 172.21.15.78:0/2040678273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c87a0 0x7f88900c5220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8888009d90 tx=0x7f888802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a70712ece6d385d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:44.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.261+0000 7f888f7fe640 1 -- 172.21.15.78:0/2040678273 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888802cc50 con 0x7f88900c87a0 2024-02-19T14:59:44.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.261+0000 7f888f7fe640 1 -- 172.21.15.78:0/2040678273 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888802cdb0 con 0x7f88900c87a0 2024-02-19T14:59:44.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.261+0000 7f888f7fe640 1 -- 172.21.15.78:0/2040678273 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888034660 con 0x7f88900c87a0 2024-02-19T14:59:44.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 -- 172.21.15.78:0/2040678273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c87a0 msgr2=0x7f88900c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 --2- 172.21.15.78:0/2040678273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c87a0 0x7f88900c5220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8888009d90 tx=0x7f888802bea0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 -- 172.21.15.78:0/2040678273 shutdown_connections 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 --2- 172.21.15.78:0/2040678273 >> v2:172.21.15.78:3300/0 conn(0x7f88900c7070 0x7f88901559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 --2- 172.21.15.78:0/2040678273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88900c87a0 0x7f88900c5220 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 -- 172.21.15.78:0/2040678273 >> 172.21.15.78:0/2040678273 conn(0x7f88900ca8a0 msgr2=0x7f88900cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 -- 172.21.15.78:0/2040678273 shutdown_connections 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.262+0000 7f8896e46640 1 -- 172.21.15.78:0/2040678273 wait complete. 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.263+0000 7f8896e46640 1 Processor -- start 2024-02-19T14:59:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.263+0000 7f8896e46640 1 -- start start 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8896e46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c7070 0x7f88901eee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8896e46640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 0x7f88900ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8896e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8890152480 0x7f88900ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8896e46640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900acb60 con 0x7f88900c87a0 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8896e46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88901590c0 con 0x7f88900c7070 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8896e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88900aa1e0 con 0x7f8890152480 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 0x7f88900ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 0x7f88900ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35708/0 (socket says 172.21.15.78:35708) 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 -- 172.21.15.78:0/308592944 learned_addr learned my addr 172.21.15.78:0/308592944 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f8894be2640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c7070 0x7f88901eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f88953e3640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8890152480 0x7f88900ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 -- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8890152480 msgr2=0x7f88900ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8890152480 0x7f88900ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 -- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c7070 msgr2=0x7f88901eee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c7070 0x7f88901eee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.264+0000 7f888ffff640 1 -- 172.21.15.78:0/308592944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c009400 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.265+0000 7f888ffff640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 0x7f88900ac0e0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f887c00ebb0 tx=0x7f887c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.265+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00eeb0 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.265+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c00cc30 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.265+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c013470 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.265+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8888009a70 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.265+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88900aa760 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.266+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890074990 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.266+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f887c0135d0 con 0x7f88900c87a0 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.267+0000 7f888d7fa640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887406b840 0x7f887406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.267+0000 7f8894be2640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887406b840 0x7f887406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.267+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f887c08a540 con 0x7f88900c87a0 2024-02-19T14:59:44.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.267+0000 7f8894be2640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887406b840 0x7f887406dd00 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8884000f60 tx=0x7f8884007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:44.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.271+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f887c054ea0 con 0x7f88900c87a0 2024-02-19T14:59:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.479+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f88900b06e0 con 0x7f88900c87a0 2024-02-19T14:59:44.482 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-02-19T14:59:44.479+0000 7f16538ac640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T14:59:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.480+0000 7f888d7fa640 1 -- 172.21.15.78:0/308592944 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v175) v1 ==== 161+0+0 (secure 0 0 0) 0x7f887c004640 con 0x7f88900c87a0 2024-02-19T14:59:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887406b840 msgr2=0x7f887406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887406b840 0x7f887406dd00 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8884000f60 tx=0x7f8884007e80 comp rx=0 tx=0).stop 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 msgr2=0x7f88900ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 0x7f88900ac0e0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f887c00ebb0 tx=0x7f887c00c520 comp rx=0 tx=0).stop 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 shutdown_connections 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f887406b840 0x7f887406dd00 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88900c87a0 0x7f88900ac0e0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8890152480 0x7f88900ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 --2- 172.21.15.78:0/308592944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88900c7070 0x7f88901eee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 >> 172.21.15.78:0/308592944 conn(0x7f88900ca8a0 msgr2=0x7f8890154bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:44.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 shutdown_connections 2024-02-19T14:59:44.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.484+0000 7f8896e46640 1 -- 172.21.15.78:0/308592944 wait complete. 2024-02-19T14:59:44.488 INFO:tasks.ceph.mds.c.smithi078.stderr: -277> 2024-02-19T14:59:38.899+0000 7f9d05a5a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T14:59:44.488 INFO:tasks.ceph.mds.c.smithi078.stderr: -275> 2024-02-19T14:59:38.899+0000 7f9d05a5a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-19T14:59:44.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T14:59:44.510 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:59:44.512 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:59:44.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.598+0000 7f81d392c640 1 Processor -- start 2024-02-19T14:59:44.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d392c640 1 -- start start 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d392c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81cc155010 0x7f81cc159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d392c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc06ffa0 con 0x7f81cc0c4d80 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d392c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc06fc70 con 0x7f81cc0c78a0 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d392c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81cc06b720 con 0x7f81cc155010 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d0ec7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f81cc0c4d80 0x7f81cc1523d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39534/0 (socket says 172.21.15.78:39534) 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.599+0000 7f81d0ec7640 1 -- 172.21.15.78:0/1360737890 learned_addr learned my addr 172.21.15.78:0/1360737890 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1360737890 >> v2:172.21.15.150:3300/0 conn(0x7f81cc155010 0x7f81cc159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96256818 0 0) 0x7f81cc06ffa0 con 0x7f81cc0c4d80 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b8003520 con 0x7f81cc0c4d80 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958037635 0 0) 0x7f81b8003520 con 0x7f81cc0c4d80 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 >> v2:172.21.15.150:3300/0 conn(0x7f81cc155010 msgr2=0x7f81cc159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 --2- 172.21.15.78:0/1360737890 >> v2:172.21.15.150:3300/0 conn(0x7f81cc155010 0x7f81cc159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 >> v1:172.21.15.150:6789/0 conn(0x7f81cc0c78a0 legacy=0x7f81cc0c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc06b930 con 0x7f81cc0c4d80 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f81c8001670 con 0x7f81cc0c4d80 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 >> v1:172.21.15.78:6789/0 conn(0x7f81cc0c4d80 legacy=0x7f81cc1523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b8003970 0x7f81b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b80047c0 0x7f81b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81b8005180 0x7f81b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc06b720 con 0x7f81b8005180 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b800c120 con 0x7f81b8003970 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b800c2e0 con 0x7f81b80047c0 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81b8005180 0x7f81b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d16c8640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b8003970 0x7f81b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d0ec7640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b80047c0 0x7f81b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b80047c0 msgr2=0x7f81b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:44.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b80047c0 0x7f81b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b8003970 msgr2=0x7f81b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b8003970 0x7f81b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1360737890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b8003520 con 0x7f81b8005180 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.600+0000 7f81d16c8640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b8003970 0x7f81b8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.601+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1360737890 reap_dead start 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.601+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81b8005180 0x7f81b800b980 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009fc0 tx=0x7f81b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b6154b758b228a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.601+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c402cc60 con 0x7f81b8005180 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.601+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c400f040 con 0x7f81b8005180 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.601+0000 7f81c3fff640 1 -- 172.21.15.78:0/1360737890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4003990 con 0x7f81b8005180 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 -- 172.21.15.78:0/1360737890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81b8005180 msgr2=0x7f81b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81b8005180 0x7f81b800b980 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009fc0 tx=0x7f81b80057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 -- 172.21.15.78:0/1360737890 shutdown_connections 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 --2- 172.21.15.78:0/1360737890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81b8005180 0x7f81b800b980 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 -- 172.21.15.78:0/1360737890 >> 172.21.15.78:0/1360737890 conn(0x7f81cc066e70 msgr2=0x7f81cc0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:44.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 -- 172.21.15.78:0/1360737890 shutdown_connections 2024-02-19T14:59:44.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.602+0000 7f81d392c640 1 -- 172.21.15.78:0/1360737890 wait complete. 2024-02-19T14:59:44.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.603+0000 7f81d392c640 1 Processor -- start 2024-02-19T14:59:44.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.603+0000 7f81d392c640 1 -- start start 2024-02-19T14:59:44.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d392c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8005180 0x7f81cc157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d392c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0c3c60 0x7f81cc1ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d392c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 0x7f81cc0c2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d392c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc0c24b0 con 0x7f81cc0abb40 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d392c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc1ee6a0 con 0x7f81cc0c3c60 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d392c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc156370 con 0x7f81b8005180 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 0x7f81cc0c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 0x7f81cc0c2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35746/0 (socket says 172.21.15.78:35746) 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1041059579 learned_addr learned my addr 172.21.15.78:0/1041059579 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d16c8640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8005180 0x7f81cc157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d0ec7640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0c3c60 0x7f81cc1ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8005180 msgr2=0x7f81cc157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8005180 0x7f81cc157000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0c3c60 msgr2=0x7f81cc1ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0c3c60 0x7f81cc1ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 -- 172.21.15.78:0/1041059579 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc156650 con 0x7f81cc0abb40 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81d1ec9640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 0x7f81cc0c2940 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f81c402c6d0 tx=0x7f81c4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:44.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4003da0 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.604+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4035950 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.605+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc0c2140 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.605+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4035d40 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.605+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc0c2e80 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.606+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc0c3070 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.606+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81c4004030 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.606+0000 7f81c1ffb640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b006b840 0x7f81b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.606+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f81c40acda0 con 0x7f81cc0abb40 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.606+0000 7f81d16c8640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b006b840 0x7f81b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:44.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.607+0000 7f81d16c8640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b006b840 0x7f81b006dd00 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f81bc000c00 tx=0x7f81bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:44.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.611+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81c40775a0 con 0x7f81cc0abb40 2024-02-19T14:59:44.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:44.819+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f81cc0c1fa0 con 0x7f81cc0abb40 2024-02-19T14:59:45.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.482+0000 7f81c1ffb640 1 -- 172.21.15.78:0/1041059579 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v176) v1 ==== 104+0+0 (secure 0 0 0) 0x7f81cc0c1fa0 con 0x7f81cc0abb40 2024-02-19T14:59:45.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b006b840 msgr2=0x7f81b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:45.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b006b840 0x7f81b006dd00 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f81bc000c00 tx=0x7f81bc009040 comp rx=0 tx=0).stop 2024-02-19T14:59:45.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 msgr2=0x7f81cc0c2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:45.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 0x7f81cc0c2940 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f81c402c6d0 tx=0x7f81c4009750 comp rx=0 tx=0).stop 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 shutdown_connections 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b006b840 0x7f81b006dd00 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81cc0abb40 0x7f81cc0c2940 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0c3c60 0x7f81cc1ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 --2- 172.21.15.78:0/1041059579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81b8005180 0x7f81cc157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 >> 172.21.15.78:0/1041059579 conn(0x7f81cc066e70 msgr2=0x7f81cc152e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 shutdown_connections 2024-02-19T14:59:45.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.484+0000 7f81d392c640 1 -- 172.21.15.78:0/1041059579 wait complete. 2024-02-19T14:59:45.500 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T14:59:45.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.646+0000 7f4d15d81640 1 Processor -- start 2024-02-19T14:59:45.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.646+0000 7f4d15d81640 1 -- start start 2024-02-19T14:59:45.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d15d81640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4d100c8bb0 0x7f4d100c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:45.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d15d81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d10153380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d15d81640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100712a0 con 0x7f4d100c7d50 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d15d81640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d10070690 con 0x7f4d100c8bb0 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d15d81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10070970 con 0x7f4d10152f10 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4d100c8bb0 0x7f4d100c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4d100c7d50 0x7f4d10150130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39536/0 (socket says 172.21.15.78:39536) 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0effd640 1 -- 172.21.15.78:0/1766510568 learned_addr learned my addr 172.21.15.78:0/1766510568 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0ffff640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d10153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/1766510568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 msgr2=0x7f4d10153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0f7fe640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d10153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/1766510568 >> v1:172.21.15.78:6789/0 conn(0x7f4d100c7d50 legacy=0x7f4d10150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/1766510568 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d10070c50 con 0x7f4d100c8bb0 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0e7fc640 1 -- 172.21.15.78:0/1766510568 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627674725 0 0) 0x7f4d100712a0 con 0x7f4d100c7d50 2024-02-19T14:59:45.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.647+0000 7f4d0ffff640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d10153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:45.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d0f7fe640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c8bb0 0x7f4d100c70e0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f4d000097a0 tx=0x7f4d0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaac4271fbce4736 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:45.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d0e7fc640 1 -- 172.21.15.78:0/1766510568 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00031830 con 0x7f4d100c8bb0 2024-02-19T14:59:45.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d0e7fc640 1 -- 172.21.15.78:0/1766510568 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d00031990 con 0x7f4d100c8bb0 2024-02-19T14:59:45.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d0e7fc640 1 -- 172.21.15.78:0/1766510568 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00031d20 con 0x7f4d100c8bb0 2024-02-19T14:59:45.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 -- 172.21.15.78:0/1766510568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c8bb0 msgr2=0x7f4d100c70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c8bb0 0x7f4d100c70e0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f4d000097a0 tx=0x7f4d0002b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 -- 172.21.15.78:0/1766510568 shutdown_connections 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d10153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 --2- 172.21.15.78:0/1766510568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c8bb0 0x7f4d100c70e0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 -- 172.21.15.78:0/1766510568 >> 172.21.15.78:0/1766510568 conn(0x7f4d1006b060 msgr2=0x7f4d1006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 -- 172.21.15.78:0/1766510568 shutdown_connections 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.648+0000 7f4d15d81640 1 -- 172.21.15.78:0/1766510568 wait complete. 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.649+0000 7f4d15d81640 1 Processor -- start 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 -- start start 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 0x7f4d10156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d100c8bb0 0x7f4d1014ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:45.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d101eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10155420 con 0x7f4d100c7d50 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100abab0 con 0x7f4d100c8bb0 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d15d81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10149590 con 0x7f4d10152f10 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d0f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 0x7f4d10156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d0f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 0x7f4d10156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35768/0 (socket says 172.21.15.78:35768) 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.650+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/670707013 learned_addr learned my addr 172.21.15.78:0/670707013 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0ffff640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d101eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0effd640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d100c8bb0 0x7f4d1014ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 msgr2=0x7f4d101eb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0f7fe640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d101eb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d100c8bb0 msgr2=0x7f4d1014ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0f7fe640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d100c8bb0 0x7f4d1014ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0f7fe640 1 -- 172.21.15.78:0/670707013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d00009400 con 0x7f4d100c7d50 2024-02-19T14:59:45.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d0f7fe640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 0x7f4d10156d40 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009890 tx=0x7f4d000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0003baa0 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0003bc20 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d10157280 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d000433f0 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.651+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d101e8af0 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.652+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d10074990 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.652+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4d0004c3b0 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.653+0000 7f4cf7fff640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4cfc06b810 0x7f4cfc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.653+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f4d000ac3c0 con 0x7f4d100c7d50 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.653+0000 7f4d0effd640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4cfc06b810 0x7f4cfc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:45.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.653+0000 7f4d0effd640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4cfc06b810 0x7f4cfc06dcd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4d101ec3f0 tx=0x7f4d04007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:45.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.657+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4d00076b30 con 0x7f4d100c7d50 2024-02-19T14:59:45.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:45.829+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d101eb2d0 con 0x7f4d100c7d50 2024-02-19T14:59:46.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.786+0000 7f4cf7fff640 1 -- 172.21.15.78:0/670707013 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v192) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4d0007a1f0 con 0x7f4d100c7d50 2024-02-19T14:59:46.789 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4cfc06b810 msgr2=0x7f4cfc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4cfc06b810 0x7f4cfc06dcd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4d101ec3f0 tx=0x7f4d04007b50 comp rx=0 tx=0).stop 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 msgr2=0x7f4d10156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 0x7f4d10156d40 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009890 tx=0x7f4d000046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 shutdown_connections 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4cfc06b810 0x7f4cfc06dcd0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d10152f10 0x7f4d101eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d100c8bb0 0x7f4d1014ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 --2- 172.21.15.78:0/670707013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d100c7d50 0x7f4d10156d40 secure :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009890 tx=0x7f4d000046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.791+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 >> 172.21.15.78:0/670707013 conn(0x7f4d1006b060 msgr2=0x7f4d100c41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.792+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 shutdown_connections 2024-02-19T14:59:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.792+0000 7f4d15d81640 1 -- 172.21.15.78:0/670707013 wait complete. 2024-02-19T14:59:46.809 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T14:59:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.915+0000 7f28ea84e640 1 Processor -- start 2024-02-19T14:59:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28ea84e640 1 -- start start 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28ea84e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 0x7f28e4155720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28ea84e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e4159210 con 0x7f28e406d000 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28ea84e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e40c6420 con 0x7f28e4152620 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28ea84e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40c1f00 con 0x7f28e406d440 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28e37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f28e406d000 0x7f28e4155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39548/0 (socket says 172.21.15.78:39548) 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.916+0000 7f28e37fe640 1 -- 172.21.15.78:0/2840535893 learned_addr learned my addr 172.21.15.78:0/2840535893 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e8deb640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 0x7f28e4155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658713368 0 0) 0x7f28e4159210 con 0x7f28e406d000 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28cc003520 con 0x7f28e406d000 2024-02-19T14:59:46.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896740034 0 0) 0x7f28cc003520 con 0x7f28e406d000 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 msgr2=0x7f28e4155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 0x7f28e4155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 >> v1:172.21.15.150:6789/0 conn(0x7f28e4152620 legacy=0x7f28e40c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e406d8d0 con 0x7f28e406d000 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f28d4001670 con 0x7f28e406d000 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 >> v1:172.21.15.78:6789/0 conn(0x7f28e406d000 legacy=0x7f28e4155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 0x7f28cc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28cc003ee0 0x7f28cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28cc004d00 0x7f28cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc00ba60 con 0x7f28cc003ee0 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc00bbd0 con 0x7f28cc004d00 2024-02-19T14:59:46.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc00bd90 con 0x7f28e406d440 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e37fe640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28cc003ee0 0x7f28cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e3fff640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 0x7f28cc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e8deb640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28cc004d00 0x7f28cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e37fe640 1 -- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 msgr2=0x7f28cc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e37fe640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28e406d440 0x7f28cc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e37fe640 1 -- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28cc004d00 msgr2=0x7f28cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e37fe640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28cc004d00 0x7f28cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.917+0000 7f28e37fe640 1 -- 172.21.15.78:0/2840535893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc003520 con 0x7f28cc003ee0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.918+0000 7f28e37fe640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28cc003ee0 0x7f28cc0047c0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f28d400c240 tx=0x7f28cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65d59d4bbf762f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.918+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4005ad0 con 0x7f28cc003ee0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.918+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d4005c30 con 0x7f28cc003ee0 2024-02-19T14:59:46.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.918+0000 7f28e2ffd640 1 -- 172.21.15.78:0/2840535893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d402c8a0 con 0x7f28cc003ee0 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28ea84e640 1 -- 172.21.15.78:0/2840535893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28cc003ee0 msgr2=0x7f28cc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2840535893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28cc003ee0 0x7f28cc0047c0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f28d400c240 tx=0x7f28cc0057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28e8deb640 1 -- 172.21.15.78:0/2840535893 reap_dead start 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28ea84e640 1 -- 172.21.15.78:0/2840535893 shutdown_connections 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28ea84e640 1 -- 172.21.15.78:0/2840535893 >> 172.21.15.78:0/2840535893 conn(0x7f28e40caae0 msgr2=0x7f28e40caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28ea84e640 1 -- 172.21.15.78:0/2840535893 shutdown_connections 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.919+0000 7f28ea84e640 1 -- 172.21.15.78:0/2840535893 wait complete. 2024-02-19T14:59:46.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.920+0000 7f28ea84e640 1 Processor -- start 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.920+0000 7f28ea84e640 1 -- start start 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28ea84e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28cc003ee0 0x7f28e41530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28ea84e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e4147a20 0x7f28e4147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28ea84e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 0x7f28e41ea590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28ea84e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e41480f0 con 0x7f28e4153e80 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28ea84e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4154ef0 con 0x7f28e4147a20 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28ea84e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4155190 con 0x7f28cc003ee0 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 0x7f28e41ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 0x7f28e41ea590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35790/0 (socket says 172.21.15.78:35790) 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 -- 172.21.15.78:0/2893914980 learned_addr learned my addr 172.21.15.78:0/2893914980 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e3fff640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28cc003ee0 0x7f28e41530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e37fe640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e4147a20 0x7f28e4147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 -- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28cc003ee0 msgr2=0x7f28e41530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28cc003ee0 0x7f28e41530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 -- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e4147a20 msgr2=0x7f28e4147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e4147a20 0x7f28e4147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 -- 172.21.15.78:0/2893914980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4149f40 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.921+0000 7f28e8deb640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 0x7f28e41ea590 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f28e40c8430 tx=0x7f28dc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.922+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00e040 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.922+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc00f040 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.922+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e414a220 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.922+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc015430 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.922+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e41477b0 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.923+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e4147030 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.923+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f28dc015590 con 0x7f28e4153e80 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.923+0000 7f28e0ff9640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28d006b810 0x7f28d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:46.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.923+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f28dc08ab50 con 0x7f28e4153e80 2024-02-19T14:59:46.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.923+0000 7f28e3fff640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28d006b810 0x7f28d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:46.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.924+0000 7f28e3fff640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28d006b810 0x7f28d006dcd0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f28d8001910 tx=0x7f28d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:46.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:46.928+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f28dc055630 con 0x7f28e4153e80 2024-02-19T14:59:47.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.102+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f28e414db30 con 0x7f28e4153e80 2024-02-19T14:59:47.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.788+0000 7f28e0ff9640 1 -- 172.21.15.78:0/2893914980 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v193) v1 ==== 169+0+0 (secure 0 0 0) 0x7f28dc058cf0 con 0x7f28e4153e80 2024-02-19T14:59:47.791 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T14:59:47.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28d006b810 msgr2=0x7f28d006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:47.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28d006b810 0x7f28d006dcd0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f28d8001910 tx=0x7f28d8009040 comp rx=0 tx=0).stop 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 msgr2=0x7f28e41ea590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 0x7f28e41ea590 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f28e40c8430 tx=0x7f28dc00f580 comp rx=0 tx=0).stop 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 shutdown_connections 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f28d006b810 0x7f28d006dcd0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e4153e80 0x7f28e41ea590 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e4147a20 0x7f28e4147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 --2- 172.21.15.78:0/2893914980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28cc003ee0 0x7f28e41530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.791+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 >> 172.21.15.78:0/2893914980 conn(0x7f28e40caae0 msgr2=0x7f28e40c3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.792+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 shutdown_connections 2024-02-19T14:59:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.792+0000 7f28ea84e640 1 -- 172.21.15.78:0/2893914980 wait complete. 2024-02-19T14:59:47.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T14:59:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.916+0000 7f2c8ef0c640 1 Processor -- start 2024-02-19T14:59:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.916+0000 7f2c8ef0c640 1 -- start start 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8ef0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c880c49c0 0x7f2c880c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8ef0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c880ab600 0x7f2c880c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8ef0c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c880aba60 con 0x7f2c880ad890 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8ef0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c88152d30 con 0x7f2c880c49c0 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8ef0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c880c7860 con 0x7f2c880ab600 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c880c49c0 0x7f2c880c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c880c49c0 0x7f2c880c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39290/0 (socket says 172.21.15.78:39290) 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 -- 172.21.15.78:0/215788083 learned_addr learned my addr 172.21.15.78:0/215788083 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8d4a9640 1 --2- 172.21.15.78:0/215788083 >> v2:172.21.15.150:3300/0 conn(0x7f2c880ab600 0x7f2c880c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 -- 172.21.15.78:0/215788083 >> v2:172.21.15.150:3300/0 conn(0x7f2c880ab600 msgr2=0x7f2c880c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 --2- 172.21.15.78:0/215788083 >> v2:172.21.15.150:3300/0 conn(0x7f2c880ab600 0x7f2c880c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 -- 172.21.15.78:0/215788083 >> v1:172.21.15.150:6789/0 conn(0x7f2c880ad890 legacy=0x7f2c881534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c87fff640 1 -- 172.21.15.78:0/215788083 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c880c7ce0 con 0x7f2c880c49c0 2024-02-19T14:59:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.917+0000 7f2c8d4a9640 1 --2- 172.21.15.78:0/215788083 >> v2:172.21.15.150:3300/0 conn(0x7f2c880ab600 0x7f2c880c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.918+0000 7f2c87fff640 1 --2- 172.21.15.78:0/215788083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880c49c0 0x7f2c880c64f0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f2c88071a80 tx=0x7f2c7402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7252ef757cbb7a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.918+0000 7f2c877fe640 1 -- 172.21.15.78:0/215788083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74031750 con 0x7f2c880c49c0 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.918+0000 7f2c877fe640 1 -- 172.21.15.78:0/215788083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c740318b0 con 0x7f2c880c49c0 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.918+0000 7f2c877fe640 1 -- 172.21.15.78:0/215788083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74031c40 con 0x7f2c880c49c0 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/215788083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880c49c0 msgr2=0x7f2c880c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/215788083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880c49c0 0x7f2c880c64f0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f2c88071a80 tx=0x7f2c7402b7f0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/215788083 shutdown_connections 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/215788083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880c49c0 0x7f2c880c64f0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/215788083 >> v2:172.21.15.150:3300/0 conn(0x7f2c880ab600 0x7f2c880c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/215788083 >> 172.21.15.78:0/215788083 conn(0x7f2c8806b700 msgr2=0x7f2c880aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/215788083 shutdown_connections 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.919+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/215788083 wait complete. 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.920+0000 7f2c8ef0c640 1 Processor -- start 2024-02-19T14:59:47.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.920+0000 7f2c8ef0c640 1 -- start start 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.920+0000 7f2c8ef0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c880ab600 0x7f2c88070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.920+0000 7f2c8ef0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 0x7f2c8814f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.920+0000 7f2c8ef0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c880c49c0 0x7f2c881ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c8ef0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c881ec030 con 0x7f2c880ad890 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c8ef0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c880715a0 con 0x7f2c880ab600 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c8ef0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c880700c0 con 0x7f2c880c49c0 2024-02-19T14:59:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 0x7f2c8814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 0x7f2c8814f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39306/0 (socket says 172.21.15.78:39306) 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 -- 172.21.15.78:0/1009001485 learned_addr learned my addr 172.21.15.78:0/1009001485 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 -- 172.21.15.78:0/1009001485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c880c49c0 msgr2=0x7f2c881ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c880c49c0 0x7f2c881ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 -- 172.21.15.78:0/1009001485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c880ab600 msgr2=0x7f2c88070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c880ab600 0x7f2c88070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 -- 172.21.15.78:0/1009001485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c740094d0 con 0x7f2c880ad890 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c87fff640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 0x7f2c8814f390 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f2c74000c00 tx=0x7f2c740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74033c80 con 0x7f2c880ad890 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c74033de0 con 0x7f2c880ad890 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c880703a0 con 0x7f2c880ad890 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7403c620 con 0x7f2c880ad890 2024-02-19T14:59:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.921+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8814f8d0 con 0x7f2c880ad890 2024-02-19T14:59:47.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.922+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c88070200 con 0x7f2c880ad890 2024-02-19T14:59:47.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.923+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2c7403a040 con 0x7f2c880ad890 2024-02-19T14:59:47.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.923+0000 7f2c857fa640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6c06b810 0x7f2c6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:47.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.923+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2c740abe90 con 0x7f2c880ad890 2024-02-19T14:59:47.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.923+0000 7f2c8cca8640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6c06b810 0x7f2c6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:47.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.923+0000 7f2c8cca8640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6c06b810 0x7f2c6c06dcd0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c006fd0 tx=0x7f2c7c006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:47.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:47.927+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c88070200 con 0x7f2c880ad890 2024-02-19T14:59:48.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.100+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2c88070200 con 0x7f2c880ad890 2024-02-19T14:59:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.103+0000 7f2c857fa640 1 -- 172.21.15.78:0/1009001485 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2c88070200 con 0x7f2c880ad890 2024-02-19T14:59:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6c06b810 msgr2=0x7f2c6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6c06b810 0x7f2c6c06dcd0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c006fd0 tx=0x7f2c7c006bc0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 msgr2=0x7f2c8814f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 0x7f2c8814f390 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f2c74000c00 tx=0x7f2c740046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 shutdown_connections 2024-02-19T14:59:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6c06b810 0x7f2c6c06dcd0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c880ad890 0x7f2c8814f390 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c880c49c0 0x7f2c881ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 --2- 172.21.15.78:0/1009001485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c880ab600 0x7f2c88070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 >> 172.21.15.78:0/1009001485 conn(0x7f2c8806b700 msgr2=0x7f2c880a8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 shutdown_connections 2024-02-19T14:59:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.105+0000 7f2c8ef0c640 1 -- 172.21.15.78:0/1009001485 wait complete. 2024-02-19T14:59:48.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-02-19T14:59:48.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.220+0000 7f2c7a623640 1 Processor -- start 2024-02-19T14:59:48.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.220+0000 7f2c7a623640 1 -- start start 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.220+0000 7f2c7a623640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 0x7f2c74071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.220+0000 7f2c7a623640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7406fe40 con 0x7f2c740c8b60 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.220+0000 7f2c7a623640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7406f880 con 0x7f2c740c29e0 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.220+0000 7f2c7a623640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740c32c0 con 0x7f2c740713a0 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c78bc0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2c740c8b60 0x7f2c741536b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38274/0 (socket says 172.21.15.78:38274) 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c78bc0640 1 -- 172.21.15.78:0/4163757658 learned_addr learned my addr 172.21.15.78:0/4163757658 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c73fff640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 0x7f2c74071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503268130 0 0) 0x7f2c7406fe40 con 0x7f2c740c8b60 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c5c003520 con 0x7f2c740c8b60 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091534912 0 0) 0x7f2c5c003520 con 0x7f2c740c8b60 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 msgr2=0x7f2c74071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 0x7f2c74071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 >> v1:172.21.15.150:6789/0 conn(0x7f2c740c29e0 legacy=0x7f2c740c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7406fb60 con 0x7f2c740c8b60 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f2c6c0019f0 con 0x7f2c740c8b60 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 >> v1:172.21.15.78:6789/0 conn(0x7f2c740c8b60 legacy=0x7f2c741536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c5c0039a0 0x7f2c5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 0x7f2c5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c5c004d00 0x7f2c5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c00ba80 con 0x7f2c5c0039a0 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c00bbf0 con 0x7f2c5c004d00 2024-02-19T14:59:48.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c00bdb0 con 0x7f2c740713a0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c73fff640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c5c0039a0 0x7f2c5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c737fe640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 0x7f2c5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.221+0000 7f2c78bc0640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c5c004d00 0x7f2c5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c73fff640 1 -- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 msgr2=0x7f2c5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c73fff640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740713a0 0x7f2c5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c73fff640 1 -- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c5c004d00 msgr2=0x7f2c5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c73fff640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c5c004d00 0x7f2c5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c73fff640 1 -- 172.21.15.78:0/4163757658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c003520 con 0x7f2c5c0039a0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c78bc0640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c5c004d00 0x7f2c5c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c73fff640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c5c0039a0 0x7f2c5c003dc0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009ac0 tx=0x7f2c5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85baf1a9d1b5ec36 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6402c600 con 0x7f2c5c0039a0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6402c760 con 0x7f2c5c0039a0 2024-02-19T14:59:48.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.222+0000 7f2c72ffd640 1 -- 172.21.15.78:0/4163757658 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6402caf0 con 0x7f2c5c0039a0 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c7a623640 1 -- 172.21.15.78:0/4163757658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c5c0039a0 msgr2=0x7f2c5c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c7a623640 1 --2- 172.21.15.78:0/4163757658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c5c0039a0 0x7f2c5c003dc0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009ac0 tx=0x7f2c5c0057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c78bc0640 1 -- 172.21.15.78:0/4163757658 reap_dead start 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c7a623640 1 -- 172.21.15.78:0/4163757658 shutdown_connections 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c7a623640 1 -- 172.21.15.78:0/4163757658 >> 172.21.15.78:0/4163757658 conn(0x7f2c740aa300 msgr2=0x7f2c740aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c7a623640 1 -- 172.21.15.78:0/4163757658 shutdown_connections 2024-02-19T14:59:48.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.223+0000 7f2c7a623640 1 -- 172.21.15.78:0/4163757658 wait complete. 2024-02-19T14:59:48.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.224+0000 7f2c7a623640 1 Processor -- start 2024-02-19T14:59:48.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.224+0000 7f2c7a623640 1 -- start start 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c7a623640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 0x7f2c7414d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c7a623640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7414cdc0 0x7f2c7414c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c7a623640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c741563b0 0x7f2c74156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c7a623640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c741dece0 con 0x7f2c7414c3b0 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c7a623640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c741e9470 con 0x7f2c741563b0 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c7a623640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c741e9750 con 0x7f2c7414cdc0 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 0x7f2c7414d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 0x7f2c7414d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39330/0 (socket says 172.21.15.78:39330) 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 -- 172.21.15.78:0/3490609729 learned_addr learned my addr 172.21.15.78:0/3490609729 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c737fe640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7414cdc0 0x7f2c7414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c78bc0640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c741563b0 0x7f2c74156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 -- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7414cdc0 msgr2=0x7f2c7414c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7414cdc0 0x7f2c7414c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 -- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c741563b0 msgr2=0x7f2c74156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c741563b0 0x7f2c74156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 -- 172.21.15.78:0/3490609729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c741dfae0 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c78bc0640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c741563b0 0x7f2c74156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.225+0000 7f2c73fff640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 0x7f2c7414d310 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f2c64000c00 tx=0x7f2c64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.226+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64003810 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.226+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c64003970 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.226+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c741dfdc0 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.226+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64003d00 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.226+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c74156fe0 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.227+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7406f290 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.227+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2c64034440 con 0x7f2c7414c3b0 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.227+0000 7f2c70ff9640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6006b840 0x7f2c6006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.227+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2c640abda0 con 0x7f2c7414c3b0 2024-02-19T14:59:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.227+0000 7f2c737fe640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6006b840 0x7f2c6006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.228+0000 7f2c737fe640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6006b840 0x7f2c6006dd00 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2c74156140 tx=0x7f2c68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:48.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.232+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c64076b40 con 0x7f2c7414c3b0 2024-02-19T14:59:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.468+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f2c74074990 con 0x7f2c7414c3b0 2024-02-19T14:59:48.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.794+0000 7f2c70ff9640 1 -- 172.21.15.78:0/3490609729 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v458) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2c640044e0 con 0x7f2c7414c3b0 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6006b840 msgr2=0x7f2c6006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6006b840 0x7f2c6006dd00 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2c74156140 tx=0x7f2c68009040 comp rx=0 tx=0).stop 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 msgr2=0x7f2c7414d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 0x7f2c7414d310 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f2c64000c00 tx=0x7f2c64004730 comp rx=0 tx=0).stop 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 shutdown_connections 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c6006b840 0x7f2c6006dd00 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c741563b0 0x7f2c74156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7414cdc0 0x7f2c7414c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 --2- 172.21.15.78:0/3490609729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c7414c3b0 0x7f2c7414d310 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 >> 172.21.15.78:0/3490609729 conn(0x7f2c740aa300 msgr2=0x7f2c740a8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:48.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 shutdown_connections 2024-02-19T14:59:48.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.797+0000 7f2c7a623640 1 -- 172.21.15.78:0/3490609729 wait complete. 2024-02-19T14:59:48.811 INFO:tasks.cephfs_test_runner:That setting max_mds undoes down. ... ok 2024-02-19T14:59:48.811 INFO:tasks.cephfs_test_runner:Starting test: test_down_health (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down does not generate a health warning 2024-02-19T14:59:48.811 INFO:tasks.cephfs_test_runner:test_down_health (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T14:59:48.812 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:48.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.911+0000 7f231d2c7640 1 Processor -- start 2024-02-19T14:59:48.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.911+0000 7f231d2c7640 1 -- start start 2024-02-19T14:59:48.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.911+0000 7f231d2c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f23180c7070 0x7f23181559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.911+0000 7f231d2c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2318155ef0 con 0x7f23180c87a0 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.911+0000 7f231d2c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231806dec0 con 0x7f2318152480 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.911+0000 7f231d2c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231806d520 con 0x7f23180c7070 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f23167fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f23180c7070 0x7f23181559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f23177fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f23180c87a0 0x7f23180c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38288/0 (socket says 172.21.15.78:38288) 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f23177fe640 1 -- 172.21.15.78:0/1264924885 learned_addr learned my addr 172.21.15.78:0/1264924885 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f23167fc640 1 -- 172.21.15.78:0/1264924885 >> v1:172.21.15.150:6789/0 conn(0x7f2318152480 legacy=0x7f231806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f23167fc640 1 -- 172.21.15.78:0/1264924885 >> v1:172.21.15.78:6789/0 conn(0x7f23180c87a0 legacy=0x7f23180c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f23167fc640 1 -- 172.21.15.78:0/1264924885 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180c75c0 con 0x7f23180c7070 2024-02-19T14:59:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.912+0000 7f2315ffb640 1 -- 172.21.15.78:0/1264924885 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546529850 0 0) 0x7f2308001110 con 0x7f23180c87a0 2024-02-19T14:59:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f23167fc640 1 --2- 172.21.15.78:0/1264924885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23180c7070 0x7f23181559b0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f230c009d90 tx=0x7f230c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b59c46017743b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f2315ffb640 1 -- 172.21.15.78:0/1264924885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c02cd00 con 0x7f23180c7070 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f2315ffb640 1 -- 172.21.15.78:0/1264924885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c02ce60 con 0x7f23180c7070 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f2315ffb640 1 -- 172.21.15.78:0/1264924885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c034660 con 0x7f23180c7070 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 -- 172.21.15.78:0/1264924885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23180c7070 msgr2=0x7f23181559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1264924885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23180c7070 0x7f23181559b0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f230c009d90 tx=0x7f230c02bf50 comp rx=0 tx=0).stop 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 -- 172.21.15.78:0/1264924885 shutdown_connections 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1264924885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23180c7070 0x7f23181559b0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 -- 172.21.15.78:0/1264924885 >> 172.21.15.78:0/1264924885 conn(0x7f23180ca8a0 msgr2=0x7f23180cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 -- 172.21.15.78:0/1264924885 shutdown_connections 2024-02-19T14:59:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.913+0000 7f231d2c7640 1 -- 172.21.15.78:0/1264924885 wait complete. 2024-02-19T14:59:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 Processor -- start 2024-02-19T14:59:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 -- start start 2024-02-19T14:59:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23180c7070 0x7f23181ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23180c87a0 0x7f23181ee540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 0x7f231814d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180b0700 con 0x7f2318152480 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180ad9b0 con 0x7f23180c87a0 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f231d2c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23181eea80 con 0x7f23180c7070 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f23177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 0x7f231814d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f23177fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 0x7f231814d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39346/0 (socket says 172.21.15.78:39346) 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.915+0000 7f23177fe640 1 -- 172.21.15.78:0/1132443364 learned_addr learned my addr 172.21.15.78:0/1132443364 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23167fc640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23180c87a0 0x7f23181ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23177fe640 1 -- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23180c7070 msgr2=0x7f23181ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23177fe640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23180c7070 0x7f23181ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23177fe640 1 -- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23180c87a0 msgr2=0x7f23181ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23177fe640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23180c87a0 0x7f23181ee540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23177fe640 1 -- 172.21.15.78:0/1132443364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c009a70 con 0x7f2318152480 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f23177fe640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 0x7f231814d670 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f230800c780 tx=0x7f230800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:48.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23080072d0 con 0x7f2318152480 2024-02-19T14:59:48.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2308007430 con 0x7f2318152480 2024-02-19T14:59:48.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308001d10 con 0x7f2318152480 2024-02-19T14:59:48.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23180b0140 con 0x7f2318152480 2024-02-19T14:59:48.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.916+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23180ad640 con 0x7f2318152480 2024-02-19T14:59:48.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.917+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23180ad4a0 con 0x7f2318152480 2024-02-19T14:59:48.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.917+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f23080047e0 con 0x7f2318152480 2024-02-19T14:59:48.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.918+0000 7f22ff7fe640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f230406b810 0x7f230406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:48.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.918+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2308089340 con 0x7f2318152480 2024-02-19T14:59:48.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.918+0000 7f2316ffd640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f230406b810 0x7f230406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:48.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.918+0000 7f2316ffd640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f230406b810 0x7f230406dcd0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2310000f60 tx=0x7f23100079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:48.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:48.922+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f23180ad4a0 con 0x7f2318152480 2024-02-19T14:59:49.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.093+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2318155ef0 con 0x7f2318152480 2024-02-19T14:59:49.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.093+0000 7f22ff7fe640 1 -- 172.21.15.78:0/1132443364 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+13421 (secure 0 0 0) 0x7f2318155ef0 con 0x7f2318152480 2024-02-19T14:59:49.096 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:49.096 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":193,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:47.787511+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6833/2126014034":"2024-02-20T14:59:44.480511+0000","172.21.15.78:6835/778404633":"2024-02-20T14:59:44.480523+0000","172.21.15.150:6835/2126014034":"2024-02-20T14:59:44.480511+0000","172.21.15.78:6834/778404633":"2024-02-20T14:59:44.480523+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:49.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f230406b810 msgr2=0x7f230406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f230406b810 0x7f230406dcd0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2310000f60 tx=0x7f23100079b0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 msgr2=0x7f231814d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 0x7f231814d670 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f230800c780 tx=0x7f230800cc40 comp rx=0 tx=0).stop 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 shutdown_connections 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f230406b810 0x7f230406dcd0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318152480 0x7f231814d670 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23180c87a0 0x7f23181ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 --2- 172.21.15.78:0/1132443364 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23180c7070 0x7f23181ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 >> 172.21.15.78:0/1132443364 conn(0x7f23180ca8a0 msgr2=0x7f2318151630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 shutdown_connections 2024-02-19T14:59:49.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.095+0000 7f231d2c7640 1 -- 172.21.15.78:0/1132443364 wait complete. 2024-02-19T14:59:49.109 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:59:49.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.209+0000 7efd7e088640 1 Processor -- start 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.209+0000 7efd7e088640 1 -- start start 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd7e088640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efd781528d0 0x7efd780c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd7e088640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd78072e80 con 0x7efd780c19c0 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd7e088640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7806e990 con 0x7efd78155780 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd7e088640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd780734c0 con 0x7efd781528d0 2024-02-19T14:59:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd777fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efd781528d0 0x7efd780c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd76ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efd780c19c0 0x7efd78155070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38294/0 (socket says 172.21.15.78:38294) 2024-02-19T14:59:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd76ffd640 1 -- 172.21.15.78:0/706907308 learned_addr learned my addr 172.21.15.78:0/706907308 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd777fe640 1 -- 172.21.15.78:0/706907308 >> v1:172.21.15.150:6789/0 conn(0x7efd78155780 legacy=0x7efd7815f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd777fe640 1 -- 172.21.15.78:0/706907308 >> v1:172.21.15.78:6789/0 conn(0x7efd780c19c0 legacy=0x7efd78155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd767fc640 1 -- 172.21.15.78:0/706907308 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597884105 0 0) 0x7efd6c001110 con 0x7efd780c19c0 2024-02-19T14:59:49.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.210+0000 7efd777fe640 1 -- 172.21.15.78:0/706907308 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780731e0 con 0x7efd781528d0 2024-02-19T14:59:49.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.211+0000 7efd777fe640 1 --2- 172.21.15.78:0/706907308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd780c1480 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7efd680097a0 tx=0x7efd6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c76d124675f7715 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.211+0000 7efd767fc640 1 -- 172.21.15.78:0/706907308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6802e070 con 0x7efd781528d0 2024-02-19T14:59:49.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.211+0000 7efd767fc640 1 -- 172.21.15.78:0/706907308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd680315a0 con 0x7efd781528d0 2024-02-19T14:59:49.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.211+0000 7efd767fc640 1 -- 172.21.15.78:0/706907308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68031930 con 0x7efd781528d0 2024-02-19T14:59:49.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 -- 172.21.15.78:0/706907308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 msgr2=0x7efd780c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 --2- 172.21.15.78:0/706907308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd780c1480 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7efd680097a0 tx=0x7efd6802b840 comp rx=0 tx=0).stop 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 -- 172.21.15.78:0/706907308 shutdown_connections 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 --2- 172.21.15.78:0/706907308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd780c1480 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 -- 172.21.15.78:0/706907308 >> 172.21.15.78:0/706907308 conn(0x7efd780c7590 msgr2=0x7efd780c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 -- 172.21.15.78:0/706907308 shutdown_connections 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.212+0000 7efd7e088640 1 -- 172.21.15.78:0/706907308 wait complete. 2024-02-19T14:59:49.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.213+0000 7efd7e088640 1 Processor -- start 2024-02-19T14:59:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.213+0000 7efd7e088640 1 -- start start 2024-02-19T14:59:49.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd780c19c0 0x7efd780c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd78156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78155780 0x7efd78157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd781de5f0 con 0x7efd781528d0 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd781deb20 con 0x7efd78155780 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd78158340 con 0x7efd780c19c0 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd78156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd78156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39364/0 (socket says 172.21.15.78:39364) 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 -- 172.21.15.78:0/3055688178 learned_addr learned my addr 172.21.15.78:0/3055688178 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd77fff640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78155780 0x7efd78157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd777fe640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd780c19c0 0x7efd780c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 -- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd780c19c0 msgr2=0x7efd780c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd780c19c0 0x7efd780c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 -- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78155780 msgr2=0x7efd78157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78155780 0x7efd78157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 -- 172.21.15.78:0/3055688178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68009400 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd77fff640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78155780 0x7efd78157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd76ffd640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd78156df0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00c430 tx=0x7efd6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0074e0 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c007660 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.214+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd781de840 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.215+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c004a20 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.215+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd78157fd0 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.215+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd781593c0 con 0x7efd781528d0 2024-02-19T14:59:49.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.216+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efd6c004b80 con 0x7efd781528d0 2024-02-19T14:59:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.216+0000 7efd5ffff640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6406b840 0x7efd6406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.217+0000 7efd777fe640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6406b840 0x7efd6406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.217+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7efd6c089c80 con 0x7efd781528d0 2024-02-19T14:59:49.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.217+0000 7efd777fe640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6406b840 0x7efd6406dd00 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7efd68036000 tx=0x7efd6803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.221+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efd6c054850 con 0x7efd781528d0 2024-02-19T14:59:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.394+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd780c1260 con 0x7efd781528d0 2024-02-19T14:59:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.394+0000 7efd5ffff640 1 -- 172.21.15.78:0/3055688178 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd6c057f10 con 0x7efd781528d0 2024-02-19T14:59:49.397 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6406b840 msgr2=0x7efd6406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6406b840 0x7efd6406dd00 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7efd68036000 tx=0x7efd6803d040 comp rx=0 tx=0).stop 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 msgr2=0x7efd78156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd78156df0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00c430 tx=0x7efd6c00c8f0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 shutdown_connections 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd6406b840 0x7efd6406dd00 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd781528d0 0x7efd78156df0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd78155780 0x7efd78157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 --2- 172.21.15.78:0/3055688178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd780c19c0 0x7efd780c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 >> 172.21.15.78:0/3055688178 conn(0x7efd780c7590 msgr2=0x7efd78152df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 shutdown_connections 2024-02-19T14:59:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.396+0000 7efd7e088640 1 -- 172.21.15.78:0/3055688178 wait complete. 2024-02-19T14:59:49.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-02-19T14:59:49.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.506+0000 7fbe0c56c640 1 Processor -- start 2024-02-19T14:59:49.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.506+0000 7fbe0c56c640 1 -- start start 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe0c56c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe040c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe0c56c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe040c7910 0x7fbe04154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe0c56c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe040c65a0 con 0x7fbe04070720 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe0c56c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe040c7530 con 0x7fbe040c7910 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe0c56c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe041537b0 con 0x7fbe040c5560 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe0a308640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe040c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.507+0000 7fbe09b07640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe040c7910 0x7fbe04154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe0a308640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe040c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:57274/0 (socket says 172.21.15.78:57274) 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe09b07640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe040c7910 0x7fbe04154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:56562/0 (socket says 172.21.15.78:56562) 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe0a308640 1 -- 172.21.15.78:0/2168364388 learned_addr learned my addr 172.21.15.78:0/2168364388 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:49.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe09b07640 1 -- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 msgr2=0x7fbe040c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:49.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe09b07640 1 --2- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe040c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe09b07640 1 -- 172.21.15.78:0/2168364388 >> v1:172.21.15.150:6789/0 conn(0x7fbe04070720 legacy=0x7fbe04070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe09b07640 1 -- 172.21.15.78:0/2168364388 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04153a90 con 0x7fbe040c7910 2024-02-19T14:59:49.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.508+0000 7fbe0a308640 1 --2- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe040c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:49.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe09b07640 1 --2- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe04154180 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40097a0 tx=0x7fbdf402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f61ecd2ae7ecfd6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe09306640 1 -- 172.21.15.78:0/2168364388 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf402f050 con 0x7fbe040c7910 2024-02-19T14:59:49.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe09306640 1 -- 172.21.15.78:0/2168364388 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4031700 con 0x7fbe040c7910 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe09306640 1 -- 172.21.15.78:0/2168364388 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4031a90 con 0x7fbe040c7910 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 -- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 msgr2=0x7fbe04154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe04154180 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40097a0 tx=0x7fbdf402b7a0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 -- 172.21.15.78:0/2168364388 shutdown_connections 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe04154180 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/2168364388 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe040c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 -- 172.21.15.78:0/2168364388 >> 172.21.15.78:0/2168364388 conn(0x7fbe0406b4b0 msgr2=0x7fbe040c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 -- 172.21.15.78:0/2168364388 shutdown_connections 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.509+0000 7fbe0c56c640 1 -- 172.21.15.78:0/2168364388 wait complete. 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.510+0000 7fbe0c56c640 1 Processor -- start 2024-02-19T14:59:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 -- start start 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 0x7fbe0414a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe0414bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe041ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0414c8e0 con 0x7fbe04070720 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe041ee420 con 0x7fbe040c7910 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0c56c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0414fae0 con 0x7fbe040c5560 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0a308640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 0x7fbe0414a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0a308640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 0x7fbe0414a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39366/0 (socket says 172.21.15.78:39366) 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.511+0000 7fbe0a308640 1 -- 172.21.15.78:0/1496573224 learned_addr learned my addr 172.21.15.78:0/1496573224 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe09b07640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe0414bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0ab09640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe041ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0a308640 1 -- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 msgr2=0x7fbe0414bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0a308640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe0414bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0a308640 1 -- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 msgr2=0x7fbe041ee8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0a308640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe041ee8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0a308640 1 -- 172.21.15.78:0/1496573224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00009b30 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0a308640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 0x7fbe0414a2e0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009e80 tx=0x7fbe0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0000ce60 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0000f040 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf4009400 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00014680 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.512+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0414c600 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.513+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe040c2e00 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.513+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbe000147e0 con 0x7fbe04070720 2024-02-19T14:59:49.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.514+0000 7fbdfaffd640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbde806b810 0x7fbde806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.514+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fbe0008a8e0 con 0x7fbe04070720 2024-02-19T14:59:49.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.514+0000 7fbe09b07640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbde806b810 0x7fbde806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.514+0000 7fbe09b07640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbde806b810 0x7fbde806dcd0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4035000 tx=0x7fbdf4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.518+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbe040c2e00 con 0x7fbe04070720 2024-02-19T14:59:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.756+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7fbe040c2e00 con 0x7fbe04070720 2024-02-19T14:59:49.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.796+0000 7fbdfaffd640 1 -- 172.21.15.78:0/1496573224 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v459) v1 ==== 143+0+0 (secure 0 0 0) 0x7fbe040c2e00 con 0x7fbe04070720 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.797+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbde806b810 msgr2=0x7fbde806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.797+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbde806b810 0x7fbde806dcd0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4035000 tx=0x7fbdf4008030 comp rx=0 tx=0).stop 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.797+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 msgr2=0x7fbe0414a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.797+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 0x7fbe0414a2e0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009e80 tx=0x7fbe0000c490 comp rx=0 tx=0).stop 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 shutdown_connections 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbde806b810 0x7fbde806dcd0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe040c7910 0x7fbe041ee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe040c5560 0x7fbe0414bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 --2- 172.21.15.78:0/1496573224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe04070720 0x7fbe0414a2e0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 >> 172.21.15.78:0/1496573224 conn(0x7fbe0406b4b0 msgr2=0x7fbe04154840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 shutdown_connections 2024-02-19T14:59:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.798+0000 7fbe0c56c640 1 -- 172.21.15.78:0/1496573224 wait complete. 2024-02-19T14:59:49.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T14:59:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.911+0000 7fd954d9a640 1 Processor -- start 2024-02-19T14:59:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.911+0000 7fd954d9a640 1 -- start start 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd954d9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 0x7fd9500c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd954d9a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd950158c40 con 0x7fd9500ac390 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd954d9a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9500acbb0 con 0x7fd9500bdf20 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd954d9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500c8de0 con 0x7fd9500ac770 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd94e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd9500ac390 0x7fd950158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38302/0 (socket says 172.21.15.78:38302) 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd94e59c640 1 -- 172.21.15.78:0/1592964376 learned_addr learned my addr 172.21.15.78:0/1592964376 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd94ed9d640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 0x7fd9500c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.912+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975672057 0 0) 0x7fd950158c40 con 0x7fd9500ac390 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd938003520 con 0x7fd9500ac390 2024-02-19T14:59:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433737946 0 0) 0x7fd938003520 con 0x7fd9500ac390 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 msgr2=0x7fd9500c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 0x7fd9500c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 >> v1:172.21.15.150:6789/0 conn(0x7fd9500bdf20 legacy=0x7fd95015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950155a60 con 0x7fd9500ac390 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fd940001a10 con 0x7fd9500ac390 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 >> v1:172.21.15.78:6789/0 conn(0x7fd9500ac390 legacy=0x7fd950158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9380039a0 0x7fd938003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9380047c0 0x7fd938004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 0x7fd938005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500c8de0 con 0x7fd9380047c0 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93800bb70 con 0x7fd9380039a0 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93800bd30 con 0x7fd9500ac770 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94dd9b640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9380047c0 0x7fd938004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94e59c640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9380039a0 0x7fd938003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94ed9d640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 0x7fd938005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94dd9b640 1 -- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 msgr2=0x7fd938005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94dd9b640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9500ac770 0x7fd938005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94dd9b640 1 -- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9380039a0 msgr2=0x7fd938003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94dd9b640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9380039a0 0x7fd938003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.913+0000 7fd94dd9b640 1 -- 172.21.15.78:0/1592964376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd938003520 con 0x7fd9380047c0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.914+0000 7fd94dd9b640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9380047c0 0x7fd938004c40 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fd944001720 tx=0x7fd9380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f7f4c7f3f22f9e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.914+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9440052d0 con 0x7fd9380047c0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.914+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94402e050 con 0x7fd9380047c0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.914+0000 7fd94d59a640 1 -- 172.21.15.78:0/1592964376 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9440035f0 con 0x7fd9380047c0 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd954d9a640 1 -- 172.21.15.78:0/1592964376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9380047c0 msgr2=0x7fd938004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd954d9a640 1 --2- 172.21.15.78:0/1592964376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9380047c0 0x7fd938004c40 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fd944001720 tx=0x7fd9380057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd94ed9d640 1 -- 172.21.15.78:0/1592964376 reap_dead start 2024-02-19T14:59:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd954d9a640 1 -- 172.21.15.78:0/1592964376 shutdown_connections 2024-02-19T14:59:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd954d9a640 1 -- 172.21.15.78:0/1592964376 >> 172.21.15.78:0/1592964376 conn(0x7fd9500ca890 msgr2=0x7fd9500caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd954d9a640 1 -- 172.21.15.78:0/1592964376 shutdown_connections 2024-02-19T14:59:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.915+0000 7fd954d9a640 1 -- 172.21.15.78:0/1592964376 wait complete. 2024-02-19T14:59:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.916+0000 7fd954d9a640 1 Processor -- start 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.916+0000 7fd954d9a640 1 -- start start 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9380047c0 0x7fd9500c0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd950157410 0x7fd95014ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 0x7fd9501eab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500af420 con 0x7fd95014f350 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500af140 con 0x7fd950157410 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9501581a0 con 0x7fd9380047c0 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 0x7fd9501eab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 0x7fd9501eab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39386/0 (socket says 172.21.15.78:39386) 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 -- 172.21.15.78:0/2135061634 learned_addr learned my addr 172.21.15.78:0/2135061634 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94e59c640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9380047c0 0x7fd9500c0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 -- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9380047c0 msgr2=0x7fd9500c0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94dd9b640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd950157410 0x7fd95014ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9380047c0 0x7fd9500c0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 -- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd950157410 msgr2=0x7fd95014ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd950157410 0x7fd95014ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 -- 172.21.15.78:0/2135061634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9500aeeb0 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd94ed9d640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 0x7fd9501eab50 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd9500ca830 tx=0x7fd948011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948011ee0 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd948004240 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9501eeb50 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94800f770 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.917+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95014f7d0 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.918+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950074990 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.919+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd94800f8d0 con 0x7fd95014f350 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.919+0000 7fd93effd640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd93006b810 0x7fd93006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.919+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd94808a320 con 0x7fd95014f350 2024-02-19T14:59:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.919+0000 7fd94e59c640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd93006b810 0x7fd93006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.919+0000 7fd94e59c640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd93006b810 0x7fd93006dcd0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd940002230 tx=0x7fd940006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:49.923+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd948054e60 con 0x7fd95014f350 2024-02-19T14:59:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.098+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd95006da00 con 0x7fd95014f350 2024-02-19T14:59:50.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.101+0000 7fd93effd640 1 -- 172.21.15.78:0/2135061634 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v61)=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd948000e60 con 0x7fd95014f350 2024-02-19T14:59:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd93006b810 msgr2=0x7fd93006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd93006b810 0x7fd93006dcd0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd940002230 tx=0x7fd940006040 comp rx=0 tx=0).stop 2024-02-19T14:59:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 msgr2=0x7fd9501eab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 0x7fd9501eab50 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd9500ca830 tx=0x7fd948011cf0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 shutdown_connections 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd93006b810 0x7fd93006dcd0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd95014f350 0x7fd9501eab50 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd950157410 0x7fd95014ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 --2- 172.21.15.78:0/2135061634 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9380047c0 0x7fd9500c0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 >> 172.21.15.78:0/2135061634 conn(0x7fd9500ca890 msgr2=0x7fd9501585e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 shutdown_connections 2024-02-19T14:59:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.103+0000 7fd954d9a640 1 -- 172.21.15.78:0/2135061634 wait complete. 2024-02-19T14:59:50.117 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T14:59:50.117 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:59:50.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T14:59:50.204 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T14:59:50.205 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T14:59:50.205 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:59:50.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T14:59:50.220 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T14:59:50.220 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T14:59:50.237 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T14:59:50.237 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T14:59:50.237 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.356+0000 7fd16729d640 1 Processor -- start 2024-02-19T14:59:50.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.356+0000 7fd16729d640 1 -- start start 2024-02-19T14:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16729d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16729d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd1600c7910 0x7fd160154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16729d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd160070720 0x7fd160070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16729d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1600c65a0 con 0x7fd160070720 2024-02-19T14:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16729d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1600c7530 con 0x7fd1600c7910 2024-02-19T14:59:50.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16729d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601537b0 con 0x7fd1600c5560 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16583a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd160070720 0x7fd160070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16583a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd160070720 0x7fd160070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39396/0 (socket says 172.21.15.78:39396) 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.357+0000 7fd16583a640 1 -- 172.21.15.78:0/712131885 learned_addr learned my addr 172.21.15.78:0/712131885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd165039640 1 --2- 172.21.15.78:0/712131885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd164838640 1 --2- 172.21.15.78:0/712131885 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c7910 0x7fd160154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd16583a640 1 -- 172.21.15.78:0/712131885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 msgr2=0x7fd1600c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd16583a640 1 --2- 172.21.15.78:0/712131885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd16583a640 1 -- 172.21.15.78:0/712131885 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c7910 msgr2=0x7fd160154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd16583a640 1 --2- 172.21.15.78:0/712131885 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c7910 0x7fd160154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd16583a640 1 -- 172.21.15.78:0/712131885 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd160153a90 con 0x7fd160070720 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd164838640 1 --2- 172.21.15.78:0/712131885 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c7910 0x7fd160154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd16583a640 1 --2- 172.21.15.78:0/712131885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd160070720 0x7fd160070b70 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd1600c8600 tx=0x7fd15802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9fb1effe379e63 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.358+0000 7fd157fff640 1 -- 172.21.15.78:0/712131885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158031740 con 0x7fd160070720 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd157fff640 1 -- 172.21.15.78:0/712131885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1580318a0 con 0x7fd160070720 2024-02-19T14:59:50.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd157fff640 1 -- 172.21.15.78:0/712131885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158031c30 con 0x7fd160070720 2024-02-19T14:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 -- 172.21.15.78:0/712131885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd160070720 msgr2=0x7fd160070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 --2- 172.21.15.78:0/712131885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd160070720 0x7fd160070b70 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd1600c8600 tx=0x7fd15802b7e0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 -- 172.21.15.78:0/712131885 shutdown_connections 2024-02-19T14:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 --2- 172.21.15.78:0/712131885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd160070720 0x7fd160070b70 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 --2- 172.21.15.78:0/712131885 >> v2:172.21.15.150:3300/0 conn(0x7fd1600c7910 0x7fd160154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 --2- 172.21.15.78:0/712131885 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 -- 172.21.15.78:0/712131885 >> 172.21.15.78:0/712131885 conn(0x7fd16006b4b0 msgr2=0x7fd1600c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 -- 172.21.15.78:0/712131885 shutdown_connections 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.359+0000 7fd16729d640 1 -- 172.21.15.78:0/712131885 wait complete. 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.360+0000 7fd16729d640 1 Processor -- start 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.360+0000 7fd16729d640 1 -- start start 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16729d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd160070720 0x7fd1601e9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16729d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16729d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 0x7fd1601ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16729d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601e9940 con 0x7fd1600c7910 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16729d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601ec120 con 0x7fd160070720 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16729d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600aa820 con 0x7fd1600c5560 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 0x7fd1601ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 0x7fd1601ec5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39406/0 (socket says 172.21.15.78:39406) 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 -- 172.21.15.78:0/3554463459 learned_addr learned my addr 172.21.15.78:0/3554463459 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd164838640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd165039640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd160070720 0x7fd1601e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 -- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 msgr2=0x7fd1600c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 -- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd160070720 msgr2=0x7fd1601e9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd160070720 0x7fd1601e9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd16583a640 1 -- 172.21.15.78:0/3554463459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580094d0 con 0x7fd1600c7910 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.361+0000 7fd165039640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd160070720 0x7fd1601e9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.362+0000 7fd16583a640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 0x7fd1601ec5b0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fd1580090b0 tx=0x7fd1580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.362+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158033c70 con 0x7fd1600c7910 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.362+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158033df0 con 0x7fd1600c7910 2024-02-19T14:59:50.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.362+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1600aab00 con 0x7fd1600c7910 2024-02-19T14:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.362+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15803c640 con 0x7fd1600c7910 2024-02-19T14:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.362+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1600c2810 con 0x7fd1600c7910 2024-02-19T14:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.363+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1600ab020 con 0x7fd1600c7910 2024-02-19T14:59:50.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.363+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd15803a040 con 0x7fd1600c7910 2024-02-19T14:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.364+0000 7fd155ffb640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14406b810 0x7fd14406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.364+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd1580ab9b0 con 0x7fd1600c7910 2024-02-19T14:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.364+0000 7fd165039640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14406b810 0x7fd14406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.364+0000 7fd165039640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14406b810 0x7fd14406dcd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd1600aa370 tx=0x7fd15c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:50.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.368+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1600ab020 con 0x7fd1600c7910 2024-02-19T14:59:50.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.579+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1601ebef0 con 0x7fd1600c7910 2024-02-19T14:59:50.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.580+0000 7fd155ffb640 1 -- 172.21.15.78:0/3554463459 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+3711 (secure 0 0 0) 0x7fd1601ebef0 con 0x7fd1600c7910 2024-02-19T14:59:50.583 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:50.583 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":176,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7355,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3243372122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":3243372122},{"type":"v1","addr":"172.21.15.78:6837","nonce":3243372122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/3161033484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3161033484},{"type":"v1","addr":"172.21.15.150:6834","nonce":3161033484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162},{"gid":8926,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3471411726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":3471411726},{"type":"v1","addr":"172.21.15.150:6835","nonce":3471411726}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176},{"gid":10263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2005216823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2005216823},{"type":"v1","addr":"172.21.15.78:6835","nonce":2005216823}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":176}],"filesystems":[]} 2024-02-19T14:59:50.583 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 176 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14406b810 msgr2=0x7fd14406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14406b810 0x7fd14406dcd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fd1600aa370 tx=0x7fd15c00b040 comp rx=0 tx=0).stop 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 msgr2=0x7fd1601ec5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 0x7fd1601ec5b0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fd1580090b0 tx=0x7fd1580046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 shutdown_connections 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd14406b810 0x7fd14406dcd0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1600c7910 0x7fd1601ec5b0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1600c5560 0x7fd1600c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 --2- 172.21.15.78:0/3554463459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd160070720 0x7fd1601e9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 >> 172.21.15.78:0/3554463459 conn(0x7fd16006b4b0 msgr2=0x7fd160153ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 shutdown_connections 2024-02-19T14:59:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.582+0000 7fd16729d640 1 -- 172.21.15.78:0/3554463459 wait complete. 2024-02-19T14:59:50.596 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T14:59:50.596 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T14:59:50.596 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T14:59:50.596 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:50.596 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T14:59:50.597 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T14:59:50.597 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T14:59:50.597 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:50.597 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T14:59:50.597 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T14:59:50.597 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T14:59:50.597 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:50.597 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T14:59:50.598 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T14:59:50.598 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T14:59:50.598 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T14:59:50.605 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T14:59:50.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T14:59:50.606 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T14:59:50.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T14:59:50.609 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T14:59:50.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T14:59:50.610 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T14:59:50.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T14:59:50.648 INFO:tasks.ceph.mds.a:Started 2024-02-19T14:59:50.648 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:50.648 INFO:tasks.ceph.mds.c:Started 2024-02-19T14:59:50.649 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:50.651 INFO:tasks.ceph.mds.b:Started 2024-02-19T14:59:50.651 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:50.652 INFO:tasks.ceph.mds.d:Started 2024-02-19T14:59:50.652 DEBUG:teuthology.parallel:result is None 2024-02-19T14:59:50.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T14:59:50.725 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T14:59:50.726 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T14:59:50.734 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T14:59:50.734 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T14:59:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.755+0000 7fa7bd360640 1 Processor -- start 2024-02-19T14:59:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.755+0000 7fa7bd360640 1 -- start start 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7bd360640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7b8152700 0x7fa7b80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7bd360640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa7b80c5730 0x7fa7b806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7bd360640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b81551f0 con 0x7fa7b80c8e00 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7bd360640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b8155490 con 0x7fa7b80c5730 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7bd360640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b8155770 con 0x7fa7b8152700 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa7b80c5730 0x7fa7b806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa7b80c5730 0x7fa7b806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39446/0 (socket says 172.21.15.78:39446) 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 -- 172.21.15.78:0/193630526 learned_addr learned my addr 172.21.15.78:0/193630526 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b77fe640 1 --1- 172.21.15.78:0/193630526 >> v1:172.21.15.78:6789/0 conn(0x7fa7b80c8e00 0x7fa7b8158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38324/0 (socket says 172.21.15.78:38324) 2024-02-19T14:59:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b6ffd640 1 --2- 172.21.15.78:0/193630526 >> v2:172.21.15.150:3300/0 conn(0x7fa7b8152700 0x7fa7b80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 -- 172.21.15.78:0/193630526 >> v2:172.21.15.150:3300/0 conn(0x7fa7b8152700 msgr2=0x7fa7b80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b5ffb640 1 -- 172.21.15.78:0/193630526 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800267351 0 0) 0x7fa7b81551f0 con 0x7fa7b80c8e00 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 --2- 172.21.15.78:0/193630526 >> v2:172.21.15.150:3300/0 conn(0x7fa7b8152700 0x7fa7b80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 -- 172.21.15.78:0/193630526 >> v1:172.21.15.78:6789/0 conn(0x7fa7b80c8e00 legacy=0x7fa7b8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.756+0000 7fa7b67fc640 1 -- 172.21.15.78:0/193630526 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8155a50 con 0x7fa7b80c5730 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.757+0000 7fa7b6ffd640 1 --2- 172.21.15.78:0/193630526 >> v2:172.21.15.150:3300/0 conn(0x7fa7b8152700 0x7fa7b80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.757+0000 7fa7b67fc640 1 --2- 172.21.15.78:0/193630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c5730 0x7fa7b806c2a0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fa7b808bad0 tx=0x7fa7ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2de950a7a9f2f62 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.757+0000 7fa7b5ffb640 1 -- 172.21.15.78:0/193630526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac02ca80 con 0x7fa7b80c5730 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.757+0000 7fa7b5ffb640 1 -- 172.21.15.78:0/193630526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac02cbe0 con 0x7fa7b80c5730 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.757+0000 7fa7b5ffb640 1 -- 172.21.15.78:0/193630526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0343f0 con 0x7fa7b80c5730 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 -- 172.21.15.78:0/193630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c5730 msgr2=0x7fa7b806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 --2- 172.21.15.78:0/193630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c5730 0x7fa7b806c2a0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fa7b808bad0 tx=0x7fa7ac02bee0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 -- 172.21.15.78:0/193630526 shutdown_connections 2024-02-19T14:59:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 --2- 172.21.15.78:0/193630526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c5730 0x7fa7b806c2a0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 --2- 172.21.15.78:0/193630526 >> v2:172.21.15.150:3300/0 conn(0x7fa7b8152700 0x7fa7b80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 -- 172.21.15.78:0/193630526 >> 172.21.15.78:0/193630526 conn(0x7fa7b80cac90 msgr2=0x7fa7b80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 -- 172.21.15.78:0/193630526 shutdown_connections 2024-02-19T14:59:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.758+0000 7fa7bd360640 1 -- 172.21.15.78:0/193630526 wait complete. 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.759+0000 7fa7bd360640 1 Processor -- start 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.759+0000 7fa7bd360640 1 -- start start 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7bd360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b80c5730 0x7fa7b81ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7bd360640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 0x7fa7b8148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7bd360640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7b8152700 0x7fa7b81e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7bd360640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8157770 con 0x7fa7b80c8e00 2024-02-19T14:59:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7bd360640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81eeb40 con 0x7fa7b80c5730 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7bd360640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81eb020 con 0x7fa7b8152700 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 0x7fa7b8148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 0x7fa7b8148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39452/0 (socket says 172.21.15.78:39452) 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 -- 172.21.15.78:0/969356578 learned_addr learned my addr 172.21.15.78:0/969356578 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b6ffd640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b80c5730 0x7fa7b81ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b77fe640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7b8152700 0x7fa7b81e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 -- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7b8152700 msgr2=0x7fa7b81e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7b8152700 0x7fa7b81e7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 -- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b80c5730 msgr2=0x7fa7b81ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.760+0000 7fa7b67fc640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b80c5730 0x7fa7b81ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa7b67fc640 1 -- 172.21.15.78:0/969356578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac009b30 con 0x7fa7b80c8e00 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa7b77fe640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7b8152700 0x7fa7b81e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa7b67fc640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 0x7fa7b8148db0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac036040 tx=0x7fa7ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac009510 con 0x7fa7b80c8e00 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac03e450 con 0x7fa7b80c8e00 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b80aa4a0 con 0x7fa7b80c8e00 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac03e5e0 con 0x7fa7b80c8e00 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.761+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b81ea6f0 con 0x7fa7b80c8e00 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.762+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b80ca370 con 0x7fa7b80c8e00 2024-02-19T14:59:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.762+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa7ac02cd50 con 0x7fa7b80c8e00 2024-02-19T14:59:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.763+0000 7fa79f7fe640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa7a406b840 0x7fa7a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.763+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fa7ac0ac690 con 0x7fa7b80c8e00 2024-02-19T14:59:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.763+0000 7fa7b6ffd640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa7a406b840 0x7fa7a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.763+0000 7fa7b6ffd640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa7a406b840 0x7fa7a406dd00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0006e80 tx=0x7fa7b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.767+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa7ac0771d0 con 0x7fa7b80c8e00 2024-02-19T14:59:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:50.952+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa7b8074990 con 0x7fa7b80c8e00 2024-02-19T14:59:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.808+0000 7fa79f7fe640 1 -- 172.21.15.78:0/969356578 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v198) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa7ac07a890 con 0x7fa7b80c8e00 2024-02-19T14:59:51.811 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T14:59:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.811+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa7a406b840 msgr2=0x7fa7a406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.811+0000 7fa7bd360640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa7a406b840 0x7fa7a406dd00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0006e80 tx=0x7fa7b0008040 comp rx=0 tx=0).stop 2024-02-19T14:59:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.811+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 msgr2=0x7fa7b8148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.811+0000 7fa7bd360640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 0x7fa7b8148db0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac036040 tx=0x7fa7ac004a40 comp rx=0 tx=0).stop 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.811+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 shutdown_connections 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.811+0000 7fa7bd360640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa7a406b840 0x7fa7a406dd00 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.812+0000 7fa7bd360640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa7b80c8e00 0x7fa7b8148db0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.812+0000 7fa7bd360640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7b8152700 0x7fa7b81e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.812+0000 7fa7bd360640 1 --2- 172.21.15.78:0/969356578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b80c5730 0x7fa7b81ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.812+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 >> 172.21.15.78:0/969356578 conn(0x7fa7b80cac90 msgr2=0x7fa7b8151520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.812+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 shutdown_connections 2024-02-19T14:59:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.812+0000 7fa7bd360640 1 -- 172.21.15.78:0/969356578 wait complete. 2024-02-19T14:59:51.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T14:59:51.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.930+0000 7fbf10fac640 1 Processor -- start 2024-02-19T14:59:51.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.930+0000 7fbf10fac640 1 -- start start 2024-02-19T14:59:51.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf10fac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:51.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf10fac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf0c0c2f50 0x7fbf0c158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf10fac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0c0c34a0 con 0x7fbf0c0c42e0 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf10fac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c06c090 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf10fac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c06d6a0 con 0x7fbf0c0c8500 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf0ad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf0c0c2f50 0x7fbf0c158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf09d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbf0c0c42e0 0x7fbf0c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:56956/0 (socket says 172.21.15.78:56956) 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf09d9b640 1 -- 172.21.15.78:0/1541036957 learned_addr learned my addr 172.21.15.78:0/1541036957 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.931+0000 7fbf0a59c640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c0c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:57376/0 (socket says 172.21.15.78:57376) 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0959a640 1 -- 172.21.15.78:0/1541036957 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302842129 0 0) 0x7fbf0c0c34a0 con 0x7fbf0c0c42e0 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0959a640 1 -- 172.21.15.78:0/1541036957 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef4003520 con 0x7fbf0c0c42e0 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0ad9d640 1 -- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 msgr2=0x7fbf0c0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0ad9d640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0ad9d640 1 -- 172.21.15.78:0/1541036957 >> v1:172.21.15.150:6789/0 conn(0x7fbf0c0c42e0 legacy=0x7fbf0c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0ad9d640 1 -- 172.21.15.78:0/1541036957 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c06c8e0 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.932+0000 7fbf0a59c640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf0ad9d640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c158d60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009d90 tx=0x7fbf0402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d6d009dc8d574b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf0959a640 1 -- 172.21.15.78:0/1541036957 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0402cbc0 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf0959a640 1 -- 172.21.15.78:0/1541036957 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0402cd20 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf0959a640 1 -- 172.21.15.78:0/1541036957 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04034660 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 -- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c2f50 msgr2=0x7fbf0c158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c158d60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009d90 tx=0x7fbf0402be10 comp rx=0 tx=0).stop 2024-02-19T14:59:51.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 -- 172.21.15.78:0/1541036957 shutdown_connections 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c158d60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 --2- 172.21.15.78:0/1541036957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 -- 172.21.15.78:0/1541036957 >> 172.21.15.78:0/1541036957 conn(0x7fbf0c0aa300 msgr2=0x7fbf0c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 -- 172.21.15.78:0/1541036957 shutdown_connections 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.933+0000 7fbf10fac640 1 -- 172.21.15.78:0/1541036957 wait complete. 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 Processor -- start 2024-02-19T14:59:51.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 -- start start 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c0703e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c149580 0x7fbf0c14aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c149410 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c06f330 con 0x7fbf0c149580 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf10fac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c14be70 con 0x7fbf0c0c8500 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c0703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:51.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c0703e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39462/0 (socket says 172.21.15.78:39462) 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.935+0000 7fbf0a59c640 1 -- 172.21.15.78:0/4074265946 learned_addr learned my addr 172.21.15.78:0/4074265946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0ad9d640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c149580 0x7fbf0c14aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf09d9b640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0a59c640 1 -- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 msgr2=0x7fbf0c06faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0a59c640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c06faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0a59c640 1 -- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c149580 msgr2=0x7fbf0c14aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0a59c640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c149580 0x7fbf0c14aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0a59c640 1 -- 172.21.15.78:0/4074265946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc009400 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf09d9b640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c06faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0ad9d640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c149580 0x7fbf0c14aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf0a59c640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c0703e0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00e960 tx=0x7fbefc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc00ca00 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc00cb60 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf04009a70 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc013470 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.936+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c06f5d0 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.937+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0c071c20 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.937+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbefc0135d0 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.938+0000 7fbefaffd640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbeec06b810 0x7fbeec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.938+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbefc08a2b0 con 0x7fbf0c0c2f50 2024-02-19T14:59:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.938+0000 7fbf09d9b640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbeec06b810 0x7fbeec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:51.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.938+0000 7fbf09d9b640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbeec06b810 0x7fbeec06dcd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0a7f60 tx=0x7fbf00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:51.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:51.942+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbefc054ea0 con 0x7fbf0c0c2f50 2024-02-19T14:59:52.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.175+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbf0c074990 con 0x7fbf0c0c2f50 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.176+0000 7fbefaffd640 1 -- 172.21.15.78:0/4074265946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fbefc058560 con 0x7fbf0c0c2f50 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:52.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:52.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.183 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T14:59:52.184 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.186 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.187 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T14:59:52.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T14:59:52.190 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbeec06b810 msgr2=0x7fbeec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbeec06b810 0x7fbeec06dcd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0a7f60 tx=0x7fbf00009040 comp rx=0 tx=0).stop 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 msgr2=0x7fbf0c0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c0703e0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00e960 tx=0x7fbefc00ee20 comp rx=0 tx=0).stop 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 shutdown_connections 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbeec06b810 0x7fbeec06dcd0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c149580 0x7fbf0c14aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf0c0c8500 0x7fbf0c06faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 --2- 172.21.15.78:0/4074265946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbf0c0c2f50 0x7fbf0c0703e0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 >> 172.21.15.78:0/4074265946 conn(0x7fbf0c0aa300 msgr2=0x7fbf0c155400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 shutdown_connections 2024-02-19T14:59:52.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.178+0000 7fbf10fac640 1 -- 172.21.15.78:0/4074265946 wait complete. 2024-02-19T14:59:52.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:52.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.290+0000 7fbc8869a640 1 Processor -- start 2024-02-19T14:59:52.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.290+0000 7fbc8869a640 1 -- start start 2024-02-19T14:59:52.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc8869a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc800c25e0 0x7fbc801526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc8869a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc80072250 con 0x7fbc800c29c0 2024-02-19T14:59:52.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc8869a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc80072480 con 0x7fbc800c5be0 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc8869a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc80152c90 con 0x7fbc800c25e0 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc86436640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc800c25e0 0x7fbc801526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc85c35640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbc800c29c0 0x7fbc800c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38332/0 (socket says 172.21.15.78:38332) 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.291+0000 7fbc85c35640 1 -- 172.21.15.78:0/1879213165 learned_addr learned my addr 172.21.15.78:0/1879213165 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.292+0000 7fbc86436640 1 -- 172.21.15.78:0/1879213165 >> v1:172.21.15.150:6789/0 conn(0x7fbc800c5be0 legacy=0x7fbc8006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.292+0000 7fbc86436640 1 -- 172.21.15.78:0/1879213165 >> v1:172.21.15.78:6789/0 conn(0x7fbc800c29c0 legacy=0x7fbc800c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.292+0000 7fbc86436640 1 -- 172.21.15.78:0/1879213165 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800c4b90 con 0x7fbc800c25e0 2024-02-19T14:59:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.292+0000 7fbc85434640 1 -- 172.21.15.78:0/1879213165 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399366282 0 0) 0x7fbc80072250 con 0x7fbc800c29c0 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.292+0000 7fbc86436640 1 --2- 172.21.15.78:0/1879213165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c25e0 0x7fbc801526a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0097a0 tx=0x7fbc7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5728fecb2c3e71a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc85434640 1 -- 172.21.15.78:0/1879213165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c031830 con 0x7fbc800c25e0 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc85434640 1 -- 172.21.15.78:0/1879213165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c031990 con 0x7fbc800c25e0 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc85434640 1 -- 172.21.15.78:0/1879213165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c031d20 con 0x7fbc800c25e0 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 -- 172.21.15.78:0/1879213165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c25e0 msgr2=0x7fbc801526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 --2- 172.21.15.78:0/1879213165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c25e0 0x7fbc801526a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0097a0 tx=0x7fbc7c02b8d0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 -- 172.21.15.78:0/1879213165 shutdown_connections 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 --2- 172.21.15.78:0/1879213165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c25e0 0x7fbc801526a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 -- 172.21.15.78:0/1879213165 >> 172.21.15.78:0/1879213165 conn(0x7fbc8006b270 msgr2=0x7fbc8006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 -- 172.21.15.78:0/1879213165 shutdown_connections 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.293+0000 7fbc8869a640 1 -- 172.21.15.78:0/1879213165 wait complete. 2024-02-19T14:59:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.294+0000 7fbc8869a640 1 Processor -- start 2024-02-19T14:59:52.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.294+0000 7fbc8869a640 1 -- start start 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc8869a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc800c25e0 0x7fbc800c0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc8869a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 0x7fbc801ed800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc8869a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc800c5be0 0x7fbc8006d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc8869a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c0220 con 0x7fbc800c29c0 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc8869a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c0500 con 0x7fbc800c25e0 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc8869a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c07e0 con 0x7fbc800c5be0 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 0x7fbc801ed800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 0x7fbc801ed800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39478/0 (socket says 172.21.15.78:39478) 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 -- 172.21.15.78:0/3176861052 learned_addr learned my addr 172.21.15.78:0/3176861052 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc86436640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc800c25e0 0x7fbc800c0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 -- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc800c5be0 msgr2=0x7fbc8006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc800c5be0 0x7fbc8006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 -- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc800c25e0 msgr2=0x7fbc800c0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc800c25e0 0x7fbc800c0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 -- 172.21.15.78:0/3176861052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c009400 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.295+0000 7fbc85c35640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 0x7fbc801ed800 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c4a0 tx=0x7fbc7000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.296+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc700074e0 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.296+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc70007640 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.296+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc800a7170 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.296+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70004a00 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.296+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc80153770 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.297+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc800a9d80 con 0x7fbc800c29c0 2024-02-19T14:59:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.297+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbc70004b60 con 0x7fbc800c29c0 2024-02-19T14:59:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.298+0000 7fbc76ffd640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc6406b810 0x7fbc6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.298+0000 7fbc86436640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc6406b810 0x7fbc6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.298+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbc700894b0 con 0x7fbc800c29c0 2024-02-19T14:59:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.298+0000 7fbc86436640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc6406b810 0x7fbc6406dcd0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c036000 tx=0x7fbc7c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.302+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbc70054120 con 0x7fbc800c29c0 2024-02-19T14:59:52.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.473+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc800a7360 con 0x7fbc800c29c0 2024-02-19T14:59:52.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.474+0000 7fbc76ffd640 1 -- 172.21.15.78:0/3176861052 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+13162 (secure 0 0 0) 0x7fbc700577e0 con 0x7fbc800c29c0 2024-02-19T14:59:52.477 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:52.477 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":198,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:51.808191+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc6406b810 msgr2=0x7fbc6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc6406b810 0x7fbc6406dcd0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c036000 tx=0x7fbc7c008030 comp rx=0 tx=0).stop 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 msgr2=0x7fbc801ed800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 0x7fbc801ed800 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000c4a0 tx=0x7fbc7000c960 comp rx=0 tx=0).stop 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 shutdown_connections 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbc6406b810 0x7fbc6406dcd0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc800c29c0 0x7fbc801ed800 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc800c5be0 0x7fbc8006d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 --2- 172.21.15.78:0/3176861052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc800c25e0 0x7fbc800c0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 >> 172.21.15.78:0/3176861052 conn(0x7fbc8006b270 msgr2=0x7fbc800c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 shutdown_connections 2024-02-19T14:59:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.476+0000 7fbc8869a640 1 -- 172.21.15.78:0/3176861052 wait complete. 2024-02-19T14:59:52.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T14:59:52.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T14:59:52.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb64a5e9640 1 Processor -- start 2024-02-19T14:59:52.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb64a5e9640 1 -- start start 2024-02-19T14:59:52.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb64a5e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 0x7fb64406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb64a5e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6441551f0 con 0x7fb64406d910 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb64a5e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644155490 con 0x7fb644152700 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb64a5e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb644155770 con 0x7fb6440c8e00 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb648b86640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb64406d910 0x7fb64415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38344/0 (socket says 172.21.15.78:38344) 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.590+0000 7fb648b86640 1 -- 172.21.15.78:0/3157624984 learned_addr learned my addr 172.21.15.78:0/3157624984 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970723081 0 0) 0x7fb6441551f0 con 0x7fb64406d910 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb630003520 con 0x7fb64406d910 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 0x7fb64406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 137907536 0 0) 0x7fb630003520 con 0x7fb64406d910 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 msgr2=0x7fb64406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 0x7fb64406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 >> v1:172.21.15.150:6789/0 conn(0x7fb644152700 legacy=0x7fb6440c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb644155a50 con 0x7fb64406d910 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb63c0019f0 con 0x7fb64406d910 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 >> v1:172.21.15.78:6789/0 conn(0x7fb64406d910 legacy=0x7fb64415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300039a0 0x7fb630003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300047c0 0x7fb630004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 0x7fb630005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644155770 con 0x7fb6300047c0 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63000bb10 con 0x7fb6300039a0 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63000bcd0 con 0x7fb6440c8e00 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300047c0 0x7fb630004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb643fff640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300039a0 0x7fb630003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb648b86640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 0x7fb630005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 -- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 msgr2=0x7fb630005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6440c8e00 0x7fb630005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 -- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300039a0 msgr2=0x7fb630003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300039a0 0x7fb630003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb6437fe640 1 -- 172.21.15.78:0/3157624984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630003520 con 0x7fb6300047c0 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.591+0000 7fb643fff640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300039a0 0x7fb630003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.592+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300047c0 0x7fb630004c40 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fb634009ac0 tx=0x7fb6300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209bbbef1e3044d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.592+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63402c6b0 con 0x7fb6300047c0 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.592+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63402c810 con 0x7fb6300047c0 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.592+0000 7fb642ffd640 1 -- 172.21.15.78:0/3157624984 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63402cba0 con 0x7fb6300047c0 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3157624984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300047c0 msgr2=0x7fb630004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3157624984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300047c0 0x7fb630004c40 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fb634009ac0 tx=0x7fb6300057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb648b86640 1 -- 172.21.15.78:0/3157624984 reap_dead start 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3157624984 shutdown_connections 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3157624984 >> 172.21.15.78:0/3157624984 conn(0x7fb6440cac90 msgr2=0x7fb6440cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3157624984 shutdown_connections 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.593+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3157624984 wait complete. 2024-02-19T14:59:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.594+0000 7fb64a5e9640 1 Processor -- start 2024-02-19T14:59:52.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.594+0000 7fb64a5e9640 1 -- start start 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300047c0 0x7fb6441425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 0x7fb6440bf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6441edfd0 0x7fb6441ee460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644142af0 con 0x7fb6441ebf10 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440c06a0 con 0x7fb6441edfd0 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440bfce0 con 0x7fb6300047c0 2024-02-19T14:59:52.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 0x7fb6440bf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 0x7fb6440bf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39488/0 (socket says 172.21.15.78:39488) 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 -- 172.21.15.78:0/3973603711 learned_addr learned my addr 172.21.15.78:0/3973603711 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb643fff640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300047c0 0x7fb6441425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 -- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300047c0 msgr2=0x7fb6441425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300047c0 0x7fb6441425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 -- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6441edfd0 msgr2=0x7fb6441ee460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb648b86640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6441edfd0 0x7fb6441ee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6441edfd0 0x7fb6441ee460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 -- 172.21.15.78:0/3973603711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6441ec470 con 0x7fb6441ebf10 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb6437fe640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 0x7fb6440bf7a0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb634037040 tx=0x7fb634004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634003810 con 0x7fb6441ebf10 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634003970 con 0x7fb6441ebf10 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634003d00 con 0x7fb6441ebf10 2024-02-19T14:59:52.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6440c0930 con 0x7fb6441ebf10 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.595+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6440befa0 con 0x7fb6441ebf10 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.596+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb644142af0 con 0x7fb6441ebf10 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.597+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb634047450 con 0x7fb6441ebf10 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.597+0000 7fb640ff9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb62406b810 0x7fb62406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.598+0000 7fb643fff640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb62406b810 0x7fb62406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.598+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb6340ac2c0 con 0x7fb6441ebf10 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.598+0000 7fb643fff640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb62406b810 0x7fb62406dcd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb6440c5eb0 tx=0x7fb62c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.602+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb644142af0 con 0x7fb6441ebf10 2024-02-19T14:59:52.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.771+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb644142af0 con 0x7fb6441ebf10 2024-02-19T14:59:52.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.772+0000 7fb640ff9640 1 -- 172.21.15.78:0/3973603711 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v198) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb644142af0 con 0x7fb6441ebf10 2024-02-19T14:59:52.775 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb62406b810 msgr2=0x7fb62406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb62406b810 0x7fb62406dcd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb6440c5eb0 tx=0x7fb62c009040 comp rx=0 tx=0).stop 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 msgr2=0x7fb6440bf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 0x7fb6440bf7a0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb634037040 tx=0x7fb634004730 comp rx=0 tx=0).stop 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 shutdown_connections 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb62406b810 0x7fb62406dcd0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6441ebf10 0x7fb6440bf7a0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6441edfd0 0x7fb6441ee460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.773+0000 7fb64a5e9640 1 --2- 172.21.15.78:0/3973603711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300047c0 0x7fb6441425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.774+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 >> 172.21.15.78:0/3973603711 conn(0x7fb6440cac90 msgr2=0x7fb64406c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.774+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 shutdown_connections 2024-02-19T14:59:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.774+0000 7fb64a5e9640 1 -- 172.21.15.78:0/3973603711 wait complete. 2024-02-19T14:59:52.787 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T14:59:52.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.885+0000 7f62946c0640 1 Processor -- start 2024-02-19T14:59:52.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.885+0000 7f62946c0640 1 -- start start 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.886+0000 7f62946c0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f628c158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.886+0000 7f62946c0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0c6af0 con 0x7f628c155f90 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.886+0000 7f62946c0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0c7940 con 0x7f628c06d550 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.886+0000 7f62946c0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0c7c20 con 0x7f628c0c6d30 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.886+0000 7f629245c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f628c155f90 0x7f628c0c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38354/0 (socket says 172.21.15.78:38354) 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.886+0000 7f629245c640 1 -- 172.21.15.78:0/486734030 learned_addr learned my addr 172.21.15.78:0/486734030 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755446087 0 0) 0x7f628c0c6af0 con 0x7f628c155f90 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6278003520 con 0x7f628c155f90 2024-02-19T14:59:52.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f628c158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1671773268 0 0) 0x7f6278003520 con 0x7f628c155f90 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 msgr2=0x7f628c158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f628c158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 >> v1:172.21.15.150:6789/0 conn(0x7f628c06d550 legacy=0x7f628c1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0c7f00 con 0x7f628c155f90 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f6288001a10 con 0x7f628c155f90 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 >> v1:172.21.15.78:6789/0 conn(0x7f628c155f90 legacy=0x7f628c0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62780039a0 0x7f6278003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f62780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6278004d00 0x7f6278005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0c7c20 con 0x7f6278004d00 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627800bab0 con 0x7f62780039a0 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627800bc70 con 0x7f628c0c6d30 2024-02-19T14:59:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6278004d00 0x7f6278005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629245c640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62780039a0 0x7f6278003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6291c5b640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f62780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 -- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 msgr2=0x7f62780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f62780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 -- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62780039a0 msgr2=0x7f6278003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62780039a0 0x7f6278003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6292c5d640 1 -- 172.21.15.78:0/486734030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278003520 con 0x7f6278004d00 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f6291c5b640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c0c6d30 0x7f62780047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.887+0000 7f629245c640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62780039a0 0x7f6278003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.888+0000 7f6292c5d640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6278004d00 0x7f6278005180 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f6284009ac0 tx=0x7f62780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c5b9b56d31e5c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.888+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628402c6b0 con 0x7f6278004d00 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.888+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628402c810 con 0x7f6278004d00 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.888+0000 7f629145a640 1 -- 172.21.15.78:0/486734030 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628402cba0 con 0x7f6278004d00 2024-02-19T14:59:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f62946c0640 1 -- 172.21.15.78:0/486734030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6278004d00 msgr2=0x7f6278005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f62946c0640 1 --2- 172.21.15.78:0/486734030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6278004d00 0x7f6278005180 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f6284009ac0 tx=0x7f62780057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f6292c5d640 1 -- 172.21.15.78:0/486734030 reap_dead start 2024-02-19T14:59:52.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f62946c0640 1 -- 172.21.15.78:0/486734030 shutdown_connections 2024-02-19T14:59:52.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f62946c0640 1 -- 172.21.15.78:0/486734030 >> 172.21.15.78:0/486734030 conn(0x7f628c06b700 msgr2=0x7f628c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:52.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f62946c0640 1 -- 172.21.15.78:0/486734030 shutdown_connections 2024-02-19T14:59:52.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.889+0000 7f62946c0640 1 -- 172.21.15.78:0/486734030 wait complete. 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.890+0000 7f62946c0640 1 Processor -- start 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.890+0000 7f62946c0640 1 -- start start 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.890+0000 7f62946c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6278004d00 0x7f628c1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.890+0000 7f62946c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 0x7f628c14e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.890+0000 7f62946c0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c14ee30 0x7f628c1eab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f62946c0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0702e0 con 0x7f628c06f5f0 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f62946c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c099a60 con 0x7f6278004d00 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f62946c0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c09a270 con 0x7f628c14ee30 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 0x7f628c14e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 0x7f628c14e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39508/0 (socket says 172.21.15.78:39508) 2024-02-19T14:59:52.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 -- 172.21.15.78:0/107860224 learned_addr learned my addr 172.21.15.78:0/107860224 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 -- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c14ee30 msgr2=0x7f628c1eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c14ee30 0x7f628c1eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 -- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6278004d00 msgr2=0x7f628c1546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f629245c640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6278004d00 0x7f628c1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6278004d00 0x7f628c1546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 -- 172.21.15.78:0/107860224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c1ea6d0 con 0x7f628c06f5f0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f629245c640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6278004d00 0x7f628c1546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6291c5b640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 0x7f628c14e8f0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f627c00e580 tx=0x7f627c00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c0052e0 con 0x7f628c06f5f0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c005440 con 0x7f628c06f5f0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c003810 con 0x7f628c06f5f0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c14f2b0 con 0x7f628c06f5f0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.891+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c1eeab0 con 0x7f628c06f5f0 2024-02-19T14:59:52.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.892+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c074990 con 0x7f628c06f5f0 2024-02-19T14:59:52.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.893+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f627c003970 con 0x7f628c06f5f0 2024-02-19T14:59:52.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.893+0000 7f6282ffd640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f627006b810 0x7f627006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:52.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.893+0000 7f629245c640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f627006b810 0x7f627006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:52.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.893+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f627c089dc0 con 0x7f628c06f5f0 2024-02-19T14:59:52.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.893+0000 7f629245c640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f627006b810 0x7f627006dcd0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6288001390 tx=0x7f6288002190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:52.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:52.897+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f627c0549b0 con 0x7f628c06f5f0 2024-02-19T14:59:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.070+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f628c0c75a0 con 0x7f628c06f5f0 2024-02-19T14:59:53.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.071+0000 7f6282ffd640 1 -- 172.21.15.78:0/107860224 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v198) v1 ==== 84+0+65 (secure 0 0 0) 0x7f627c058070 con 0x7f628c06f5f0 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T14:59:53.074 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f627006b810 msgr2=0x7f627006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f627006b810 0x7f627006dcd0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6288001390 tx=0x7f6288002190 comp rx=0 tx=0).stop 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 msgr2=0x7f628c14e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 0x7f628c14e8f0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f627c00e580 tx=0x7f627c00ea40 comp rx=0 tx=0).stop 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 shutdown_connections 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f627006b810 0x7f627006dcd0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f628c06f5f0 0x7f628c14e8f0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f628c14ee30 0x7f628c1eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 --2- 172.21.15.78:0/107860224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6278004d00 0x7f628c1546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 >> 172.21.15.78:0/107860224 conn(0x7f628c06b700 msgr2=0x7f628c1582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:53.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 shutdown_connections 2024-02-19T14:59:53.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.073+0000 7f62946c0640 1 -- 172.21.15.78:0/107860224 wait complete. 2024-02-19T14:59:53.086 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T14:59:53.086 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T14:59:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.187+0000 7f4893f65640 1 Processor -- start 2024-02-19T14:59:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.188+0000 7f4893f65640 1 -- start start 2024-02-19T14:59:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.188+0000 7f4893f65640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f488c1527d0 0x7f488c0c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.188+0000 7f4893f65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f488c0c5360 0x7f488c1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.188+0000 7f4893f65640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c06fa90 con 0x7f488c0c7f10 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.188+0000 7f4893f65640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c1592b0 con 0x7f488c1527d0 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.188+0000 7f4893f65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c071280 con 0x7f488c0c5360 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f488c1527d0 0x7f488c0c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f488c1527d0 0x7f488c0c4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39510/0 (socket says 172.21.15.78:39510) 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4892502640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f488c0c7f10 0x7f488c155610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38356/0 (socket says 172.21.15.78:38356) 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 -- 172.21.15.78:0/2513616792 learned_addr learned my addr 172.21.15.78:0/2513616792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891500640 1 --2- 172.21.15.78:0/2513616792 >> v2:172.21.15.150:3300/0 conn(0x7f488c0c5360 0x7f488c1550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 -- 172.21.15.78:0/2513616792 >> v2:172.21.15.150:3300/0 conn(0x7f488c0c5360 msgr2=0x7f488c1550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 --2- 172.21.15.78:0/2513616792 >> v2:172.21.15.150:3300/0 conn(0x7f488c0c5360 0x7f488c1550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 -- 172.21.15.78:0/2513616792 >> v1:172.21.15.78:6789/0 conn(0x7f488c0c7f10 legacy=0x7f488c155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4891d01640 1 -- 172.21.15.78:0/2513616792 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c06f4d0 con 0x7f488c1527d0 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.189+0000 7f4890cff640 1 -- 172.21.15.78:0/2513616792 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666688909 0 0) 0x7f488c06fa90 con 0x7f488c0c7f10 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4891d01640 1 --2- 172.21.15.78:0/2513616792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0c4e20 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f488c072650 tx=0x7f488802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c408eaef3c9806 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4890cff640 1 -- 172.21.15.78:0/2513616792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888030860 con 0x7f488c1527d0 2024-02-19T14:59:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4890cff640 1 -- 172.21.15.78:0/2513616792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48880309c0 con 0x7f488c1527d0 2024-02-19T14:59:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4890cff640 1 -- 172.21.15.78:0/2513616792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888030d50 con 0x7f488c1527d0 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 -- 172.21.15.78:0/2513616792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 msgr2=0x7f488c0c4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 --2- 172.21.15.78:0/2513616792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0c4e20 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f488c072650 tx=0x7f488802b9a0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 -- 172.21.15.78:0/2513616792 shutdown_connections 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 --2- 172.21.15.78:0/2513616792 >> v2:172.21.15.150:3300/0 conn(0x7f488c0c5360 0x7f488c1550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 --2- 172.21.15.78:0/2513616792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0c4e20 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 -- 172.21.15.78:0/2513616792 >> 172.21.15.78:0/2513616792 conn(0x7f488c0aa370 msgr2=0x7f488c0aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 -- 172.21.15.78:0/2513616792 shutdown_connections 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.190+0000 7f4893f65640 1 -- 172.21.15.78:0/2513616792 wait complete. 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.191+0000 7f4893f65640 1 Processor -- start 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.191+0000 7f4893f65640 1 -- start start 2024-02-19T14:59:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4893f65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0c5360 0x7f488c0c1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4893f65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f488c0c7f10 0x7f488c153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4893f65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4893f65640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c14f6c0 con 0x7f488c1527d0 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4893f65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0ac120 con 0x7f488c0c5360 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4893f65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0bfc70 con 0x7f488c0c7f10 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0ac5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39522/0 (socket says 172.21.15.78:39522) 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 -- 172.21.15.78:0/3709111975 learned_addr learned my addr 172.21.15.78:0/3709111975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4891d01640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0c5360 0x7f488c0c1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4891500640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f488c0c7f10 0x7f488c153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 -- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f488c0c7f10 msgr2=0x7f488c153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f488c0c7f10 0x7f488c153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 -- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0c5360 msgr2=0x7f488c0c1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0c5360 0x7f488c0c1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.192+0000 7f4892502640 1 -- 172.21.15.78:0/3709111975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48880094d0 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.193+0000 7f4892502640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0ac5b0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f488400c7a0 tx=0x7f488400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.193+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400a640 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.193+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488400a7a0 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.193+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c0ab5a0 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.193+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400abd0 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.193+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c1ee5b0 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.194+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c06e160 con 0x7f488c1527d0 2024-02-19T14:59:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.194+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f488400ad30 con 0x7f488c1527d0 2024-02-19T14:59:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.195+0000 7f487e7fc640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f487006b810 0x7f487006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.195+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4884089890 con 0x7f488c1527d0 2024-02-19T14:59:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.195+0000 7f4891d01640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f487006b810 0x7f487006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.195+0000 7f4891d01640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f487006b810 0x7f487006dcd0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4888036000 tx=0x7f488803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.199+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4884054470 con 0x7f488c1527d0 2024-02-19T14:59:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.371+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f488c074990 con 0x7f488c1527d0 2024-02-19T14:59:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.820+0000 7f487e7fc640 1 -- 172.21.15.78:0/3709111975 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v199) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4884018020 con 0x7f488c1527d0 2024-02-19T14:59:53.824 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T14:59:53.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f487006b810 msgr2=0x7f487006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f487006b810 0x7f487006dcd0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4888036000 tx=0x7f488803b040 comp rx=0 tx=0).stop 2024-02-19T14:59:53.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 msgr2=0x7f488c0ac5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0ac5b0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f488400c7a0 tx=0x7f488400cc60 comp rx=0 tx=0).stop 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 shutdown_connections 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f487006b810 0x7f487006dcd0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f488c1527d0 0x7f488c0ac5b0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f488c0c7f10 0x7f488c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 --2- 172.21.15.78:0/3709111975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0c5360 0x7f488c0c1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 >> 172.21.15.78:0/3709111975 conn(0x7f488c0aa370 msgr2=0x7f488c0aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 shutdown_connections 2024-02-19T14:59:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.822+0000 7f4893f65640 1 -- 172.21.15.78:0/3709111975 wait complete. 2024-02-19T14:59:53.839 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T14:59:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.943+0000 7f5c9a0f4640 1 Processor -- start 2024-02-19T14:59:53.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.943+0000 7f5c9a0f4640 1 -- start start 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c9a0f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 0x7f5c94155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c9a0f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5c940c6110 0x7f5c940c2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c9a0f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c940c2570 0x7f5c941523e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c9a0f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c9406fbe0 con 0x7f5c940c2570 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c9a0f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c94151cb0 con 0x7f5c940c6110 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c9a0f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941561f0 con 0x7f5c9414f1e0 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c93fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c940c2570 0x7f5c941523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c93fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5c940c2570 0x7f5c941523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39536/0 (socket says 172.21.15.78:39536) 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c93fff640 1 -- 172.21.15.78:0/97741211 learned_addr learned my addr 172.21.15.78:0/97741211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.944+0000 7f5c92ffd640 1 --2- 172.21.15.78:0/97741211 >> v2:172.21.15.150:3300/0 conn(0x7f5c940c6110 0x7f5c940c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c93fff640 1 -- 172.21.15.78:0/97741211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 msgr2=0x7f5c94155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c93fff640 1 --2- 172.21.15.78:0/97741211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 0x7f5c94155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c93fff640 1 -- 172.21.15.78:0/97741211 >> v2:172.21.15.150:3300/0 conn(0x7f5c940c6110 msgr2=0x7f5c940c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:53.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c93fff640 1 --2- 172.21.15.78:0/97741211 >> v2:172.21.15.150:3300/0 conn(0x7f5c940c6110 0x7f5c940c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c93fff640 1 -- 172.21.15.78:0/97741211 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9406f060 con 0x7f5c940c2570 2024-02-19T14:59:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c93fff640 1 --2- 172.21.15.78:0/97741211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c941523e0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f5c94070d00 tx=0x7f5c8c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8474b93c04c6597 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.945+0000 7f5c927fc640 1 -- 172.21.15.78:0/97741211 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c02c9e0 con 0x7f5c940c2570 2024-02-19T14:59:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c927fc640 1 -- 172.21.15.78:0/97741211 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c02cb40 con 0x7f5c940c2570 2024-02-19T14:59:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c927fc640 1 -- 172.21.15.78:0/97741211 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c0343f0 con 0x7f5c940c2570 2024-02-19T14:59:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/97741211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 msgr2=0x7f5c941523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/97741211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c941523e0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f5c94070d00 tx=0x7f5c8c02be40 comp rx=0 tx=0).stop 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/97741211 shutdown_connections 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/97741211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c941523e0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/97741211 >> v2:172.21.15.150:3300/0 conn(0x7f5c940c6110 0x7f5c940c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/97741211 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 0x7f5c94155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/97741211 >> 172.21.15.78:0/97741211 conn(0x7f5c940c7630 msgr2=0x7f5c940c7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/97741211 shutdown_connections 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.946+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/97741211 wait complete. 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.947+0000 7f5c9a0f4640 1 Processor -- start 2024-02-19T14:59:53.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.947+0000 7f5c9a0f4640 1 -- start start 2024-02-19T14:59:53.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c9a0f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c940aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c9a0f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940c6110 0x7f5c94147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c9a0f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 0x7f5c941eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c9a0f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940aaaa0 con 0x7f5c940c2570 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c9a0f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c94154950 con 0x7f5c940c6110 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c9a0f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c940aa770 con 0x7f5c9414f1e0 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c940aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c940aaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39546/0 (socket says 172.21.15.78:39546) 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 -- 172.21.15.78:0/3517830658 learned_addr learned my addr 172.21.15.78:0/3517830658 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c92ffd640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940c6110 0x7f5c94147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 -- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 msgr2=0x7f5c941eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 0x7f5c941eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 -- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940c6110 msgr2=0x7f5c94147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940c6110 0x7f5c94147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:53.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c937fe640 1 -- 172.21.15.78:0/3517830658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c009b30 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.948+0000 7f5c92ffd640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940c6110 0x7f5c94147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.949+0000 7f5c937fe640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c940aaf30 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400b4a0 tx=0x7f5c8400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.949+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84010610 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.949+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c84010770 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.949+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c94154bf0 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.949+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84010b20 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.949+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c94154ed0 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.950+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c94074990 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.950+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5c84010c80 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.951+0000 7f5c7bfff640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c8006b810 0x7f5c8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.951+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f5c84089d20 con 0x7f5c940c2570 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.951+0000 7f5c92ffd640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c8006b810 0x7f5c8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:53.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.951+0000 7f5c92ffd640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c8006b810 0x7f5c8006dcd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5c941ec660 tx=0x7f5c88008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:53.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:53.955+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5c84054820 con 0x7f5c940c2570 2024-02-19T14:59:54.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.126+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5c940c5530 con 0x7f5c940c2570 2024-02-19T14:59:54.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.828+0000 7f5c7bfff640 1 -- 172.21.15.78:0/3517830658 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v200) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5c8401e0d0 con 0x7f5c940c2570 2024-02-19T14:59:54.831 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T14:59:54.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.830+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c8006b810 msgr2=0x7f5c8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:54.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.830+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c8006b810 0x7f5c8006dcd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5c941ec660 tx=0x7f5c88008040 comp rx=0 tx=0).stop 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.830+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 msgr2=0x7f5c940aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.830+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c940aaf30 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400b4a0 tx=0x7f5c8400b960 comp rx=0 tx=0).stop 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 shutdown_connections 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5c8006b810 0x7f5c8006dcd0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c9414f1e0 0x7f5c941eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c940c6110 0x7f5c94147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 --2- 172.21.15.78:0/3517830658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940c2570 0x7f5c940aaf30 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 >> 172.21.15.78:0/3517830658 conn(0x7f5c940c7630 msgr2=0x7f5c9406fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 shutdown_connections 2024-02-19T14:59:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.831+0000 7f5c9a0f4640 1 -- 172.21.15.78:0/3517830658 wait complete. 2024-02-19T14:59:54.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T14:59:54.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.957+0000 7effec120640 1 Processor -- start 2024-02-19T14:59:54.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.957+0000 7effec120640 1 -- start start 2024-02-19T14:59:54.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effec120640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effe41528d0 0x7effe40c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:54.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effec120640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe4155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effec120640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4072e80 con 0x7effe4072180 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effec120640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe40734c0 con 0x7effe41528d0 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effec120640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe406e990 con 0x7effe40c19c0 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effea6bd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7effe4072180 0x7effe406ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38358/0 (socket says 172.21.15.78:38358) 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effea6bd640 1 -- 172.21.15.78:0/2127464559 learned_addr learned my addr 172.21.15.78:0/2127464559 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effe96bb640 1 --2- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe4155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effe9ebc640 1 --2- 172.21.15.78:0/2127464559 >> v2:172.21.15.150:3300/0 conn(0x7effe41528d0 0x7effe40c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effe8eba640 1 -- 172.21.15.78:0/2127464559 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416080875 0 0) 0x7effdc001490 con 0x7effe4072180 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.958+0000 7effe8eba640 1 -- 172.21.15.78:0/2127464559 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd0003520 con 0x7effe4072180 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.959+0000 7effe9ebc640 1 -- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 msgr2=0x7effe4155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.959+0000 7effe9ebc640 1 --2- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe4155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.959+0000 7effe8eba640 1 -- 172.21.15.78:0/2127464559 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1543802588 0 0) 0x7effd0003520 con 0x7effe4072180 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.959+0000 7effe9ebc640 1 -- 172.21.15.78:0/2127464559 >> v1:172.21.15.78:6789/0 conn(0x7effe4072180 legacy=0x7effe406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.959+0000 7effe9ebc640 1 -- 172.21.15.78:0/2127464559 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe40731e0 con 0x7effe41528d0 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.959+0000 7effe9ebc640 1 --2- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe40c1480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7effe00097a0 tx=0x7effe002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dfdb748a021c6ee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effe8eba640 1 -- 172.21.15.78:0/2127464559 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc001b80 con 0x7effe41528d0 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effe8eba640 1 -- 172.21.15.78:0/2127464559 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe002e050 con 0x7effe41528d0 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effe8eba640 1 -- 172.21.15.78:0/2127464559 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0030710 con 0x7effe41528d0 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 -- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 msgr2=0x7effe40c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 --2- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe40c1480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7effe00097a0 tx=0x7effe002b710 comp rx=0 tx=0).stop 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 -- 172.21.15.78:0/2127464559 shutdown_connections 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 --2- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe4155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 --2- 172.21.15.78:0/2127464559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe40c1480 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 -- 172.21.15.78:0/2127464559 >> 172.21.15.78:0/2127464559 conn(0x7effe40c7590 msgr2=0x7effe40c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:54.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 -- 172.21.15.78:0/2127464559 shutdown_connections 2024-02-19T14:59:54.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.960+0000 7effec120640 1 -- 172.21.15.78:0/2127464559 wait complete. 2024-02-19T14:59:54.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.961+0000 7effec120640 1 Processor -- start 2024-02-19T14:59:54.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.961+0000 7effec120640 1 -- start start 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effec120640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 0x7effe40a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effec120640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe41e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effec120640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe41e8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effec120640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe40a7ba0 con 0x7effe4072180 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effec120640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe4156590 con 0x7effe41528d0 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effec120640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe4155fd0 con 0x7effe40c19c0 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 0x7effe40a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 0x7effe40a7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39554/0 (socket says 172.21.15.78:39554) 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 -- 172.21.15.78:0/2111480587 learned_addr learned my addr 172.21.15.78:0/2111480587 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe96bb640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe41e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effea6bd640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe41e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:54.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 -- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 msgr2=0x7effe41e7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe41e7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 -- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 msgr2=0x7effe41e8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe41e8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.962+0000 7effe9ebc640 1 -- 172.21.15.78:0/2111480587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0009400 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.963+0000 7effe9ebc640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 0x7effe40a7fa0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7effe002b750 tx=0x7effe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.963+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0003ea0 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.963+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe0030710 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.963+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe41562b0 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.963+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0030ac0 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.963+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe40a84e0 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.964+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe4074990 con 0x7effe4072180 2024-02-19T14:59:54.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.964+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7effe0030c20 con 0x7effe4072180 2024-02-19T14:59:54.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.965+0000 7effda7fc640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc806b810 0x7effc806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:54.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.965+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6001+0+0 (secure 0 0 0) 0x7effe00ac340 con 0x7effe4072180 2024-02-19T14:59:54.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.965+0000 7effe96bb640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc806b810 0x7effc806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:54.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.965+0000 7effe96bb640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc806b810 0x7effc806dcd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7effe41556a0 tx=0x7effd4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:54.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:54.969+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7effe0076da0 con 0x7effe4072180 2024-02-19T14:59:55.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.176+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7effe40c1260 con 0x7effe4072180 2024-02-19T14:59:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.829+0000 7effda7fc640 1 -- 172.21.15.78:0/2111480587 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v178) v1 ==== 174+0+0 (secure 0 0 0) 0x7effe0030ed0 con 0x7effe4072180 2024-02-19T14:59:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 31 and data pool 32 2024-02-19T14:59:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc806b810 msgr2=0x7effc806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc806b810 0x7effc806dcd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7effe41556a0 tx=0x7effd4008040 comp rx=0 tx=0).stop 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 msgr2=0x7effe40a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 0x7effe40a7fa0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7effe002b750 tx=0x7effe00046c0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 shutdown_connections 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc806b810 0x7effc806dcd0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe41528d0 0x7effe41e8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe40c19c0 0x7effe41e7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 --2- 172.21.15.78:0/2111480587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effe4072180 0x7effe40a7fa0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 >> 172.21.15.78:0/2111480587 conn(0x7effe40c7590 msgr2=0x7effe4150800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.833+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 shutdown_connections 2024-02-19T14:59:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.834+0000 7effec120640 1 -- 172.21.15.78:0/2111480587 wait complete. 2024-02-19T14:59:55.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:55.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.965+0000 7f95ed379640 1 Processor -- start 2024-02-19T14:59:55.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95ed379640 1 -- start start 2024-02-19T14:59:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95ed379640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95e8152700 0x7f95e80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95ed379640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95e80c5730 0x7f95e806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95ed379640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e81551f0 con 0x7f95e80c8e00 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95ed379640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e8155490 con 0x7f95e80c5730 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95ed379640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e8155770 con 0x7f95e8152700 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95e67fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95e80c5730 0x7f95e806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95e67fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f95e80c5730 0x7f95e806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39560/0 (socket says 172.21.15.78:39560) 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.966+0000 7f95e67fc640 1 -- 172.21.15.78:0/3685914061 learned_addr learned my addr 172.21.15.78:0/3685914061 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e6ffd640 1 --2- 172.21.15.78:0/3685914061 >> v2:172.21.15.150:3300/0 conn(0x7f95e8152700 0x7f95e80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e67fc640 1 -- 172.21.15.78:0/3685914061 >> v2:172.21.15.150:3300/0 conn(0x7f95e8152700 msgr2=0x7f95e80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e67fc640 1 --2- 172.21.15.78:0/3685914061 >> v2:172.21.15.150:3300/0 conn(0x7f95e8152700 0x7f95e80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e67fc640 1 -- 172.21.15.78:0/3685914061 >> v1:172.21.15.150:6789/0 conn(0x7f95e80c8e00 legacy=0x7f95e8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e67fc640 1 -- 172.21.15.78:0/3685914061 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e8155a50 con 0x7f95e80c5730 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e6ffd640 1 --2- 172.21.15.78:0/3685914061 >> v2:172.21.15.150:3300/0 conn(0x7f95e8152700 0x7f95e80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.967+0000 7f95e67fc640 1 --2- 172.21.15.78:0/3685914061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e806c2a0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f95e808bad0 tx=0x7f95d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c961f021d433ed3d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95e5ffb640 1 -- 172.21.15.78:0/3685914061 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d00317e0 con 0x7f95e80c5730 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95e5ffb640 1 -- 172.21.15.78:0/3685914061 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d0031940 con 0x7f95e80c5730 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95e5ffb640 1 -- 172.21.15.78:0/3685914061 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0031cd0 con 0x7f95e80c5730 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 -- 172.21.15.78:0/3685914061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 msgr2=0x7f95e806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 --2- 172.21.15.78:0/3685914061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e806c2a0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f95e808bad0 tx=0x7f95d002b880 comp rx=0 tx=0).stop 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 -- 172.21.15.78:0/3685914061 shutdown_connections 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 --2- 172.21.15.78:0/3685914061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e806c2a0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 --2- 172.21.15.78:0/3685914061 >> v2:172.21.15.150:3300/0 conn(0x7f95e8152700 0x7f95e80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 -- 172.21.15.78:0/3685914061 >> 172.21.15.78:0/3685914061 conn(0x7f95e80cac90 msgr2=0x7f95e80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 -- 172.21.15.78:0/3685914061 shutdown_connections 2024-02-19T14:59:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.968+0000 7f95ed379640 1 -- 172.21.15.78:0/3685914061 wait complete. 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.969+0000 7f95ed379640 1 Processor -- start 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.969+0000 7f95ed379640 1 -- start start 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95ed379640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e80c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95ed379640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80c8e00 0x7f95e81eb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95ed379640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95e8152700 0x7f95e814ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95ed379640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e80c32e0 con 0x7f95e80c5730 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95ed379640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e80c2350 con 0x7f95e80c8e00 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95ed379640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e80c2910 con 0x7f95e8152700 2024-02-19T14:59:55.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e80c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e80c2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39566/0 (socket says 172.21.15.78:39566) 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 -- 172.21.15.78:0/812376736 learned_addr learned my addr 172.21.15.78:0/812376736 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e67fc640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80c8e00 0x7f95e81eb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e77fe640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95e8152700 0x7f95e814ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 -- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95e8152700 msgr2=0x7f95e814ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95e8152700 0x7f95e814ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 -- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80c8e00 msgr2=0x7f95e81eb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80c8e00 0x7f95e81eb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e6ffd640 1 -- 172.21.15.78:0/812376736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00094d0 con 0x7f95e80c5730 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e77fe640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95e8152700 0x7f95e814ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.970+0000 7f95e67fc640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80c8e00 0x7f95e81eb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:55.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.971+0000 7f95e6ffd640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e80c2da0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c820 tx=0x7f95d800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.971+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8007a30 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.971+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d8007b90 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.971+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e8153f30 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.971+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d800ce10 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.971+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e80c3ea0 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.972+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e8074990 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.972+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f95d80113f0 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.973+0000 7f95df7fe640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95c806b810 0x7f95c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.973+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95d808a3f0 con 0x7f95e80c5730 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.973+0000 7f95e67fc640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95c806b810 0x7f95c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:55.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.973+0000 7f95e67fc640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95c806b810 0x7f95c806dcd0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f95d0036000 tx=0x7f95d003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:55.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:55.977+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f95d8054d10 con 0x7f95e80c5730 2024-02-19T14:59:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.149+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95e80c8770 con 0x7f95e80c5730 2024-02-19T14:59:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.150+0000 7f95df7fe640 1 -- 172.21.15.78:0/812376736 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+16953 (secure 0 0 0) 0x7f95d80583d0 con 0x7f95e80c5730 2024-02-19T14:59:56.153 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:56.153 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:55.177295+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:53.390533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:54.145380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95c806b810 msgr2=0x7f95c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95c806b810 0x7f95c806dcd0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f95d0036000 tx=0x7f95d003d040 comp rx=0 tx=0).stop 2024-02-19T14:59:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 msgr2=0x7f95e80c2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e80c2da0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c820 tx=0x7f95d800cce0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 shutdown_connections 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95c806b810 0x7f95c806dcd0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95e8152700 0x7f95e814ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80c8e00 0x7f95e81eb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 --2- 172.21.15.78:0/812376736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95e80c5730 0x7f95e80c2da0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 >> 172.21.15.78:0/812376736 conn(0x7f95e80cac90 msgr2=0x7f95e806c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 shutdown_connections 2024-02-19T14:59:56.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.152+0000 7f95ed379640 1 -- 172.21.15.78:0/812376736 wait complete. 2024-02-19T14:59:56.166 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.264+0000 7fb280ba0640 1 Processor -- start 2024-02-19T14:59:56.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.264+0000 7fb280ba0640 1 -- start start 2024-02-19T14:59:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb280ba0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb27c14d110 0x7fb27c0ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb280ba0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c06bae0 0x7fb27c06f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb280ba0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27c06b620 con 0x7fb27c073220 2024-02-19T14:59:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb280ba0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27c06ea00 con 0x7fb27c14d110 2024-02-19T14:59:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb280ba0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c157230 con 0x7fb27c06bae0 2024-02-19T14:59:56.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb27a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb27c14d110 0x7fb27c0ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb27ad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c06bae0 0x7fb27c06f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb273fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb27c073220 0x7fb27c06ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:57004/0 (socket says 172.21.15.78:57004) 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.265+0000 7fb273fff640 1 -- 172.21.15.78:0/3700730446 learned_addr learned my addr 172.21.15.78:0/3700730446 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.266+0000 7fb27ad9d640 1 -- 172.21.15.78:0/3700730446 >> v2:172.21.15.150:3300/0 conn(0x7fb27c14d110 msgr2=0x7fb27c0ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.266+0000 7fb27ad9d640 1 --2- 172.21.15.78:0/3700730446 >> v2:172.21.15.150:3300/0 conn(0x7fb27c14d110 0x7fb27c0ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.266+0000 7fb27ad9d640 1 -- 172.21.15.78:0/3700730446 >> v1:172.21.15.150:6789/0 conn(0x7fb27c073220 legacy=0x7fb27c06ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.266+0000 7fb27ad9d640 1 -- 172.21.15.78:0/3700730446 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c072180 con 0x7fb27c06bae0 2024-02-19T14:59:56.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.266+0000 7fb27a59c640 1 --2- 172.21.15.78:0/3700730446 >> v2:172.21.15.150:3300/0 conn(0x7fb27c14d110 0x7fb27c0ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.266+0000 7fb27ad9d640 1 --2- 172.21.15.78:0/3700730446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c06bae0 0x7fb27c06f530 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb274009d90 tx=0x7fb27402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9da6c464e8b6e173 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb279d9b640 1 -- 172.21.15.78:0/3700730446 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27402cbc0 con 0x7fb27c06bae0 2024-02-19T14:59:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb279d9b640 1 -- 172.21.15.78:0/3700730446 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27402cd20 con 0x7fb27c06bae0 2024-02-19T14:59:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb279d9b640 1 -- 172.21.15.78:0/3700730446 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274034660 con 0x7fb27c06bae0 2024-02-19T14:59:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 -- 172.21.15.78:0/3700730446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c06bae0 msgr2=0x7fb27c06f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3700730446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c06bae0 0x7fb27c06f530 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb274009d90 tx=0x7fb27402be10 comp rx=0 tx=0).stop 2024-02-19T14:59:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 -- 172.21.15.78:0/3700730446 shutdown_connections 2024-02-19T14:59:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3700730446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c06bae0 0x7fb27c06f530 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3700730446 >> v2:172.21.15.150:3300/0 conn(0x7fb27c14d110 0x7fb27c0ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 -- 172.21.15.78:0/3700730446 >> 172.21.15.78:0/3700730446 conn(0x7fb27c0c7590 msgr2=0x7fb27c0c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 -- 172.21.15.78:0/3700730446 shutdown_connections 2024-02-19T14:59:56.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.267+0000 7fb280ba0640 1 -- 172.21.15.78:0/3700730446 wait complete. 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.268+0000 7fb280ba0640 1 Processor -- start 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.268+0000 7fb280ba0640 1 -- start start 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb280ba0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 0x7fb27c1e86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb280ba0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c073220 0x7fb27c1ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb280ba0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c14d110 0x7fb27c145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb280ba0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c150490 con 0x7fb27c06bae0 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb280ba0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c0c4e70 con 0x7fb27c14d110 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb280ba0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1e8c20 con 0x7fb27c073220 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb27a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 0x7fb27c1e86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb27a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 0x7fb27c1e86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39578/0 (socket says 172.21.15.78:39578) 2024-02-19T14:59:56.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb27a59c640 1 -- 172.21.15.78:0/3688435792 learned_addr learned my addr 172.21.15.78:0/3688435792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb27ad9d640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c14d110 0x7fb27c145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.269+0000 7fb273fff640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c073220 0x7fb27c1ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb27a59c640 1 -- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c073220 msgr2=0x7fb27c1ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb27a59c640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c073220 0x7fb27c1ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb27a59c640 1 -- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c14d110 msgr2=0x7fb27c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb27a59c640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c14d110 0x7fb27c145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb27a59c640 1 -- 172.21.15.78:0/3688435792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c009400 con 0x7fb27c06bae0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb27a59c640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 0x7fb27c1e86e0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00ec10 tx=0x7fb26c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c009760 con 0x7fb27c06bae0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c00cc30 con 0x7fb27c06bae0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c013470 con 0x7fb27c06bae0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb274009a70 con 0x7fb27c06bae0 2024-02-19T14:59:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.270+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c0c4b90 con 0x7fb27c06bae0 2024-02-19T14:59:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.271+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c156e40 con 0x7fb27c06bae0 2024-02-19T14:59:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.271+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb26c0135d0 con 0x7fb27c06bae0 2024-02-19T14:59:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.272+0000 7fb272ffd640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb25c06b810 0x7fb25c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.272+0000 7fb273fff640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb25c06b810 0x7fb25c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.272+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb26c08a8a0 con 0x7fb27c06bae0 2024-02-19T14:59:56.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.272+0000 7fb273fff640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb25c06b810 0x7fb25c06dcd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb264007fd0 tx=0x7fb264007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.276+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb27c156e40 con 0x7fb27c06bae0 2024-02-19T14:59:56.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.448+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb27c156e40 con 0x7fb27c06bae0 2024-02-19T14:59:56.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.449+0000 7fb272ffd640 1 -- 172.21.15.78:0/3688435792 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+16953 (secure 0 0 0) 0x7fb27c156e40 con 0x7fb27c06bae0 2024-02-19T14:59:56.452 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:56.452 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:55.177295+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:53.390533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:54.145380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb25c06b810 msgr2=0x7fb25c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb25c06b810 0x7fb25c06dcd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb264007fd0 tx=0x7fb264007e80 comp rx=0 tx=0).stop 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 msgr2=0x7fb27c1e86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 0x7fb27c1e86e0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00ec10 tx=0x7fb26c00c520 comp rx=0 tx=0).stop 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 shutdown_connections 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb25c06b810 0x7fb25c06dcd0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb27c14d110 0x7fb27c145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27c073220 0x7fb27c1ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 --2- 172.21.15.78:0/3688435792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb27c06bae0 0x7fb27c1e86e0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 >> 172.21.15.78:0/3688435792 conn(0x7fb27c0c7590 msgr2=0x7fb27c14c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 shutdown_connections 2024-02-19T14:59:56.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.451+0000 7fb280ba0640 1 -- 172.21.15.78:0/3688435792 wait complete. 2024-02-19T14:59:56.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T14:59:56.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.564+0000 7fc2cb4d4640 1 Processor -- start 2024-02-19T14:59:56.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.564+0000 7fc2cb4d4640 1 -- start start 2024-02-19T14:59:56.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.564+0000 7fc2cb4d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc2c4071410 0x7fc2c4156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.564+0000 7fc2cb4d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc2c40c1dd0 0x7fc2c40c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2cb4d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c4155a50 con 0x7fc2c40c19f0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2cb4d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c4152af0 con 0x7fc2c40c1dd0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2cb4d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c4155f80 con 0x7fc2c4071410 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc2c40c1dd0 0x7fc2c40c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc2c40c1dd0 0x7fc2c40c21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39586/0 (socket says 172.21.15.78:39586) 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9270640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc2c40c19f0 0x7fc2c40c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38368/0 (socket says 172.21.15.78:38368) 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 -- 172.21.15.78:0/2337275113 learned_addr learned my addr 172.21.15.78:0/2337275113 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c8a6f640 1 --2- 172.21.15.78:0/2337275113 >> v2:172.21.15.150:3300/0 conn(0x7fc2c4071410 0x7fc2c4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 -- 172.21.15.78:0/2337275113 >> v2:172.21.15.150:3300/0 conn(0x7fc2c4071410 msgr2=0x7fc2c4156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 --2- 172.21.15.78:0/2337275113 >> v2:172.21.15.150:3300/0 conn(0x7fc2c4071410 0x7fc2c4156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 -- 172.21.15.78:0/2337275113 >> v1:172.21.15.78:6789/0 conn(0x7fc2c40c19f0 legacy=0x7fc2c40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c9a71640 1 -- 172.21.15.78:0/2337275113 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4070840 con 0x7fc2c40c1dd0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2bbfff640 1 -- 172.21.15.78:0/2337275113 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476670710 0 0) 0x7fc2c4155a50 con 0x7fc2c40c19f0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.565+0000 7fc2c8a6f640 1 --2- 172.21.15.78:0/2337275113 >> v2:172.21.15.150:3300/0 conn(0x7fc2c4071410 0x7fc2c4156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2c9a71640 1 --2- 172.21.15.78:0/2337275113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c1dd0 0x7fc2c40c21e0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40a8290 tx=0x7fc2bc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8786cc1f276d3485 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2bbfff640 1 -- 172.21.15.78:0/2337275113 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc02c9f0 con 0x7fc2c40c1dd0 2024-02-19T14:59:56.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2bbfff640 1 -- 172.21.15.78:0/2337275113 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc02cb50 con 0x7fc2c40c1dd0 2024-02-19T14:59:56.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2bbfff640 1 -- 172.21.15.78:0/2337275113 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0343f0 con 0x7fc2c40c1dd0 2024-02-19T14:59:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/2337275113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c1dd0 msgr2=0x7fc2c40c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/2337275113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c1dd0 0x7fc2c40c21e0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40a8290 tx=0x7fc2bc02be50 comp rx=0 tx=0).stop 2024-02-19T14:59:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/2337275113 shutdown_connections 2024-02-19T14:59:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/2337275113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c1dd0 0x7fc2c40c21e0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/2337275113 >> v2:172.21.15.150:3300/0 conn(0x7fc2c4071410 0x7fc2c4156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.566+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/2337275113 >> 172.21.15.78:0/2337275113 conn(0x7fc2c40aa370 msgr2=0x7fc2c40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.567+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/2337275113 shutdown_connections 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.567+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/2337275113 wait complete. 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 Processor -- start 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 -- start start 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2c4071410 0x7fc2c40c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 0x7fc2c414d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2c40c1dd0 0x7fc2c414dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40abaf0 con 0x7fc2c40c19f0 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41ec220 con 0x7fc2c4071410 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2cb4d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c414e100 con 0x7fc2c40c1dd0 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2c8a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 0x7fc2c414d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2c8a6f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 0x7fc2c414d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39594/0 (socket says 172.21.15.78:39594) 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.568+0000 7fc2c8a6f640 1 -- 172.21.15.78:0/1132927428 learned_addr learned my addr 172.21.15.78:0/1132927428 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:56.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c9270640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2c4071410 0x7fc2c40c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c9a71640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2c40c1dd0 0x7fc2c414dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c8a6f640 1 -- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2c40c1dd0 msgr2=0x7fc2c414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c8a6f640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2c40c1dd0 0x7fc2c414dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c8a6f640 1 -- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2c4071410 msgr2=0x7fc2c40c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c8a6f640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2c4071410 0x7fc2c40c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c8a6f640 1 -- 172.21.15.78:0/1132927428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc009b30 con 0x7fc2c40c19f0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c9a71640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2c40c1dd0 0x7fc2c414dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2c8a6f640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 0x7fc2c414d680 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fc2b400d7d0 tx=0x7fc2b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4004030 con 0x7fc2c40c19f0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b4004190 con 0x7fc2c40c19f0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c40ab530 con 0x7fc2c40c19f0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4004520 con 0x7fc2c40c19f0 2024-02-19T14:59:56.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.569+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c40c4270 con 0x7fc2c40c19f0 2024-02-19T14:59:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.570+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c40c8320 con 0x7fc2c40c19f0 2024-02-19T14:59:56.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.570+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc2b40096a0 con 0x7fc2c40c19f0 2024-02-19T14:59:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.571+0000 7fc2b9ffb640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc2a806b810 0x7fc2a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.571+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc2b408a3d0 con 0x7fc2c40c19f0 2024-02-19T14:59:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.571+0000 7fc2c9270640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc2a806b810 0x7fc2a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.571+0000 7fc2c9270640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc2a806b810 0x7fc2a806dcd0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0007fd0 tx=0x7fc2c0007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.575+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc2b4054cf0 con 0x7fc2c40c19f0 2024-02-19T14:59:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.781+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc2c4074990 con 0x7fc2c40c19f0 2024-02-19T14:59:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.842+0000 7fc2b9ffb640 1 -- 172.21.15.78:0/1132927428 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v179) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc2b40583b0 con 0x7fc2c40c19f0 2024-02-19T14:59:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.844+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc2a806b810 msgr2=0x7fc2a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.844+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc2a806b810 0x7fc2a806dcd0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0007fd0 tx=0x7fc2c0007de0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.844+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 msgr2=0x7fc2c414d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.844+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 0x7fc2c414d680 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fc2b400d7d0 tx=0x7fc2b400dc90 comp rx=0 tx=0).stop 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 shutdown_connections 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc2a806b810 0x7fc2a806dcd0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c40c19f0 0x7fc2c414d680 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2c40c1dd0 0x7fc2c414dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 --2- 172.21.15.78:0/1132927428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2c4071410 0x7fc2c40c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 >> 172.21.15.78:0/1132927428 conn(0x7fc2c40aa370 msgr2=0x7fc2c406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 shutdown_connections 2024-02-19T14:59:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.845+0000 7fc2cb4d4640 1 -- 172.21.15.78:0/1132927428 wait complete. 2024-02-19T14:59:56.858 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T14:59:56.858 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T14:59:56.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.971+0000 7f8cb9cb5640 1 Processor -- start 2024-02-19T14:59:56.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.971+0000 7f8cb9cb5640 1 -- start start 2024-02-19T14:59:56.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb9cb5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8cb40c25e0 0x7f8cb41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb9cb5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cb40c29c0 0x7f8cb40c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb9cb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c49c0 0x7f8cb406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb9cb5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb4152c90 con 0x7f8cb40c29c0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb9cb5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb4072250 con 0x7f8cb40c25e0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb9cb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb4072480 con 0x7f8cb40c49c0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cb40c29c0 0x7f8cb40c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8cb40c29c0 0x7f8cb40c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39598/0 (socket says 172.21.15.78:39598) 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 -- 172.21.15.78:0/2271125532 learned_addr learned my addr 172.21.15.78:0/2271125532 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb37fe640 1 --2- 172.21.15.78:0/2271125532 >> v2:172.21.15.150:3300/0 conn(0x7f8cb40c25e0 0x7f8cb41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 -- 172.21.15.78:0/2271125532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c49c0 msgr2=0x7f8cb406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb3fff640 1 --2- 172.21.15.78:0/2271125532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c49c0 0x7f8cb406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 --2- 172.21.15.78:0/2271125532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c49c0 0x7f8cb406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 -- 172.21.15.78:0/2271125532 >> v2:172.21.15.150:3300/0 conn(0x7f8cb40c25e0 msgr2=0x7f8cb41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 --2- 172.21.15.78:0/2271125532 >> v2:172.21.15.150:3300/0 conn(0x7f8cb40c25e0 0x7f8cb41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb2ffd640 1 -- 172.21.15.78:0/2271125532 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb40c3eb0 con 0x7f8cb40c29c0 2024-02-19T14:59:56.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.972+0000 7f8cb37fe640 1 --2- 172.21.15.78:0/2271125532 >> v2:172.21.15.150:3300/0 conn(0x7f8cb40c25e0 0x7f8cb41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.973+0000 7f8cb2ffd640 1 --2- 172.21.15.78:0/2271125532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb40c29c0 0x7f8cb40c4480 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f8cb406f120 tx=0x7f8ca402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f855f81dc6f90c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.973+0000 7f8cb27fc640 1 -- 172.21.15.78:0/2271125532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40317c0 con 0x7f8cb40c29c0 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.973+0000 7f8cb27fc640 1 -- 172.21.15.78:0/2271125532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca4031920 con 0x7f8cb40c29c0 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.973+0000 7f8cb27fc640 1 -- 172.21.15.78:0/2271125532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4031cb0 con 0x7f8cb40c29c0 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/2271125532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb40c29c0 msgr2=0x7f8cb40c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/2271125532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb40c29c0 0x7f8cb40c4480 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f8cb406f120 tx=0x7f8ca402b860 comp rx=0 tx=0).stop 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/2271125532 shutdown_connections 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/2271125532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c49c0 0x7f8cb406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/2271125532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb40c29c0 0x7f8cb40c4480 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/2271125532 >> v2:172.21.15.150:3300/0 conn(0x7f8cb40c25e0 0x7f8cb41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/2271125532 >> 172.21.15.78:0/2271125532 conn(0x7f8cb406b270 msgr2=0x7f8cb406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/2271125532 shutdown_connections 2024-02-19T14:59:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.974+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/2271125532 wait complete. 2024-02-19T14:59:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.975+0000 7f8cb9cb5640 1 Processor -- start 2024-02-19T14:59:56.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.975+0000 7f8cb9cb5640 1 -- start start 2024-02-19T14:59:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb9cb5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c25e0 0x7f8cb414acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb9cb5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c29c0 0x7f8cb4151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb9cb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 0x7f8cb41eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb9cb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb4155cb0 con 0x7f8cb41ea590 2024-02-19T14:59:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb9cb5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb4155f90 con 0x7f8cb40c25e0 2024-02-19T14:59:56.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb9cb5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40ab890 con 0x7f8cb40c29c0 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 0x7f8cb41eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 0x7f8cb41eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39614/0 (socket says 172.21.15.78:39614) 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 -- 172.21.15.78:0/1661619205 learned_addr learned my addr 172.21.15.78:0/1661619205 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb2ffd640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c29c0 0x7f8cb4151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb37fe640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c25e0 0x7f8cb414acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 -- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c29c0 msgr2=0x7f8cb4151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c29c0 0x7f8cb4151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 -- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c25e0 msgr2=0x7f8cb414acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c25e0 0x7f8cb414acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb3fff640 1 -- 172.21.15.78:0/1661619205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40094d0 con 0x7f8cb41ea590 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.976+0000 7f8cb37fe640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c25e0 0x7f8cb414acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.977+0000 7f8cb3fff640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 0x7f8cb41eaa20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00cc80 tx=0x7f8cac00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.977+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00ce90 con 0x7f8cb41ea590 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.977+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac00ae80 con 0x7f8cb41ea590 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.977+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac003bd0 con 0x7f8cb41ea590 2024-02-19T14:59:56.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.977+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb40abb70 con 0x7f8cb41ea590 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.977+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb40ac130 con 0x7f8cb41ea590 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.978+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb406f1a0 con 0x7f8cb41ea590 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.978+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8cac00f3f0 con 0x7f8cb41ea590 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.978+0000 7f8c9bfff640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ca006b840 0x7f8ca006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.979+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8cac08a750 con 0x7f8cb41ea590 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.979+0000 7f8cb37fe640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ca006b840 0x7f8ca006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:56.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.979+0000 7f8cb37fe640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ca006b840 0x7f8ca006dd00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4151980 tx=0x7f8ca8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:56.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:56.983+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8cac054fe0 con 0x7f8cb41ea590 2024-02-19T14:59:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.198+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8cb40ac320 con 0x7f8cb41ea590 2024-02-19T14:59:57.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:59:57.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T14:59:57.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.847+0000 7f8c9bfff640 1 -- 172.21.15.78:0/1661619205 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v180) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8cac0586a0 con 0x7f8cb41ea590 2024-02-19T14:59:57.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ca006b840 msgr2=0x7f8ca006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ca006b840 0x7f8ca006dd00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4151980 tx=0x7f8ca8008040 comp rx=0 tx=0).stop 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 msgr2=0x7f8cb41eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 0x7f8cb41eaa20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00cc80 tx=0x7f8cac00a5c0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 shutdown_connections 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ca006b840 0x7f8ca006dd00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cb41ea590 0x7f8cb41eaa20 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cb40c29c0 0x7f8cb4151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 --2- 172.21.15.78:0/1661619205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c25e0 0x7f8cb414acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 >> 172.21.15.78:0/1661619205 conn(0x7f8cb406b270 msgr2=0x7f8cb406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 shutdown_connections 2024-02-19T14:59:57.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.850+0000 7f8cb9cb5640 1 -- 172.21.15.78:0/1661619205 wait complete. 2024-02-19T14:59:57.864 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T14:59:57.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:57.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.965+0000 7fccdf4ed640 1 Processor -- start 2024-02-19T14:59:57.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.965+0000 7fccdf4ed640 1 -- start start 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdf4ed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccd80add70 0x7fccd80c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdf4ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd80ae670 con 0x7fccd80705b0 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdf4ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd80716b0 con 0x7fccd80c2f60 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdf4ed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd80c33d0 con 0x7fccd80add70 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdda8a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fccd80705b0 0x7fccd80709c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40282/0 (socket says 172.21.15.78:40282) 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdda8a640 1 -- 172.21.15.78:0/2748489043 learned_addr learned my addr 172.21.15.78:0/2748489043 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fccdd289640 1 --2- 172.21.15.78:0/2748489043 >> v2:172.21.15.150:3300/0 conn(0x7fccd80add70 0x7fccd80c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389651743 0 0) 0x7fccd80ae670 con 0x7fccd80705b0 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc4003520 con 0x7fccd80705b0 2024-02-19T14:59:57.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727402961 0 0) 0x7fccc4003520 con 0x7fccd80705b0 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 >> v2:172.21.15.150:3300/0 conn(0x7fccd80add70 msgr2=0x7fccd80c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 --2- 172.21.15.78:0/2748489043 >> v2:172.21.15.150:3300/0 conn(0x7fccd80add70 0x7fccd80c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 >> v1:172.21.15.150:6789/0 conn(0x7fccd80c2f60 legacy=0x7fccd806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8153780 con 0x7fccd80705b0 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fccd4001670 con 0x7fccd80705b0 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.966+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 >> v1:172.21.15.78:6789/0 conn(0x7fccd80705b0 legacy=0x7fccd80709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fcccffff640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fcccffff640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fcccffff640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4005180 0x7fccc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd80c33d0 con 0x7fccc4003970 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc400c120 con 0x7fccc4005180 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc400c2e0 con 0x7fccc40047c0 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdd289640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdda8a640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4005180 0x7fccc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdca88640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdd289640 1 -- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 msgr2=0x7fccc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdd289640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdd289640 1 -- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4005180 msgr2=0x7fccc400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdd289640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4005180 0x7fccc400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdd289640 1 -- 172.21.15.78:0/2748489043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4003520 con 0x7fccc4003970 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdca88640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccc40047c0 0x7fccc4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdda8a640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4005180 0x7fccc400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.967+0000 7fccdda8a640 1 -- 172.21.15.78:0/2748489043 reap_dead start 2024-02-19T14:59:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fccdd289640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009fc0 tx=0x7fccc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60fbed40f365ccf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd002cc20 con 0x7fccc4003970 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd000f040 con 0x7fccc4003970 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fcccffff640 1 -- 172.21.15.78:0/2748489043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0034440 con 0x7fccc4003970 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fccdf4ed640 1 -- 172.21.15.78:0/2748489043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc4003970 msgr2=0x7fccc4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fccd0009fc0 tx=0x7fccc40057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fccdf4ed640 1 -- 172.21.15.78:0/2748489043 shutdown_connections 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/2748489043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccc4003970 0x7fccc4003d90 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.968+0000 7fccdf4ed640 1 -- 172.21.15.78:0/2748489043 >> 172.21.15.78:0/2748489043 conn(0x7fccd80c85e0 msgr2=0x7fccd80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.969+0000 7fccdf4ed640 1 -- 172.21.15.78:0/2748489043 shutdown_connections 2024-02-19T14:59:57.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.969+0000 7fccdf4ed640 1 -- 172.21.15.78:0/2748489043 wait complete. 2024-02-19T14:59:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 Processor -- start 2024-02-19T14:59:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 -- start start 2024-02-19T14:59:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccd80ad150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccd80a7d50 0x7fccd8155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 0x7fccd81eda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd8154050 con 0x7fccd80c1b60 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd81eb270 con 0x7fccc4003970 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdf4ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd806c030 con 0x7fccd80a7d50 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdda8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 0x7fccd81eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdda8a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 0x7fccd81eda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35984/0 (socket says 172.21.15.78:35984) 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.970+0000 7fccdda8a640 1 -- 172.21.15.78:0/1184237815 learned_addr learned my addr 172.21.15.78:0/1184237815 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdd289640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccd80ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdca88640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccd80a7d50 0x7fccd8155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdda8a640 1 -- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccd80a7d50 msgr2=0x7fccd8155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdda8a640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccd80a7d50 0x7fccd8155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdda8a640 1 -- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 msgr2=0x7fccd80ad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdda8a640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccd80ad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdda8a640 1 -- 172.21.15.78:0/1184237815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8154230 con 0x7fccd80c1b60 2024-02-19T14:59:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdca88640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccd80a7d50 0x7fccd8155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdda8a640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 0x7fccd81eda50 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fccd806ee20 tx=0x7fccd400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400c630 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd400c7b0 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd81eaf00 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400cb60 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.971+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd80c1fc0 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.972+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd8074990 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.973+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fccd400ccc0 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.973+0000 7fcccdffb640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccbc06b840 0x7fccbc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.973+0000 7fccdd289640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccbc06b840 0x7fccbc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.973+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fccd408a0a0 con 0x7fccd80c1b60 2024-02-19T14:59:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.974+0000 7fccdd289640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccbc06b840 0x7fccbc06dd00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fccd0036000 tx=0x7fccd002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:57.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:57.977+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fccd40549c0 con 0x7fccd80c1b60 2024-02-19T14:59:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.192+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccd8071490 con 0x7fccd80c1b60 2024-02-19T14:59:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.193+0000 7fcccdffb640 1 -- 172.21.15.78:0/1184237815 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4949 (secure 0 0 0) 0x7fccd4058080 con 0x7fccd80c1b60 2024-02-19T14:59:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:58.196 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":180,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:55.177591+0000","modified":"2024-02-19T14:59:57.199274+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10344":{"gid":10344,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3213692827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3213692827},{"type":"v1","addr":"172.21.15.78:6837","nonce":3213692827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-19T14:59:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 180 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccbc06b840 msgr2=0x7fccbc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccbc06b840 0x7fccbc06dd00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fccd0036000 tx=0x7fccd002e000 comp rx=0 tx=0).stop 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 msgr2=0x7fccd81eda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 0x7fccd81eda50 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fccd806ee20 tx=0x7fccd400eb90 comp rx=0 tx=0).stop 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 shutdown_connections 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fccbc06b840 0x7fccbc06dd00 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fccd80c1b60 0x7fccd81eda50 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccd80a7d50 0x7fccd8155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 --2- 172.21.15.78:0/1184237815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccc4003970 0x7fccd80ad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 >> 172.21.15.78:0/1184237815 conn(0x7fccd80c85e0 msgr2=0x7fccd814fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 shutdown_connections 2024-02-19T14:59:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.196+0000 7fccdf4ed640 1 -- 172.21.15.78:0/1184237815 wait complete. 2024-02-19T14:59:58.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T14:59:58.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.306+0000 7f86fc691640 1 Processor -- start 2024-02-19T14:59:58.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.307+0000 7f86fc691640 1 -- start start 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.307+0000 7f86fc691640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86f40c5130 0x7f86f40c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.307+0000 7f86fc691640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86f40c5510 0x7f86f41559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.307+0000 7f86fc691640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f40c3cf0 con 0x7f86f4158f80 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.307+0000 7f86fc691640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f4158b60 con 0x7f86f40c5510 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.307+0000 7f86fc691640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f4155f00 con 0x7f86f40c5130 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86f40c5510 0x7f86f41559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f86f40c5510 0x7f86f41559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35990/0 (socket says 172.21.15.78:35990) 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 -- 172.21.15.78:0/2888560234 learned_addr learned my addr 172.21.15.78:0/2888560234 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86f9c2c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f86f4158f80 0x7f86f40c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40290/0 (socket says 172.21.15.78:40290) 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fa42d640 1 --2- 172.21.15.78:0/2888560234 >> v2:172.21.15.150:3300/0 conn(0x7f86f40c5130 0x7f86f40c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 -- 172.21.15.78:0/2888560234 >> v2:172.21.15.150:3300/0 conn(0x7f86f40c5130 msgr2=0x7f86f40c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 --2- 172.21.15.78:0/2888560234 >> v2:172.21.15.150:3300/0 conn(0x7f86f40c5130 0x7f86f40c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 -- 172.21.15.78:0/2888560234 >> v1:172.21.15.78:6789/0 conn(0x7f86f4158f80 legacy=0x7f86f40c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fac2e640 1 -- 172.21.15.78:0/2888560234 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4159400 con 0x7f86f40c5510 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86f942b640 1 -- 172.21.15.78:0/2888560234 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261474257 0 0) 0x7f86f40c3cf0 con 0x7f86f4158f80 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.308+0000 7f86fa42d640 1 --2- 172.21.15.78:0/2888560234 >> v2:172.21.15.150:3300/0 conn(0x7f86f40c5130 0x7f86f40c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.309+0000 7f86fac2e640 1 --2- 172.21.15.78:0/2888560234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5510 0x7f86f41559c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f86f40c9130 tx=0x7f86ec02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65b6072d8c7306d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.309+0000 7f86f942b640 1 -- 172.21.15.78:0/2888560234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec02cb20 con 0x7f86f40c5510 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.309+0000 7f86f942b640 1 -- 172.21.15.78:0/2888560234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec02cc80 con 0x7f86f40c5510 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.309+0000 7f86f942b640 1 -- 172.21.15.78:0/2888560234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0343f0 con 0x7f86f40c5510 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.309+0000 7f86fc691640 1 -- 172.21.15.78:0/2888560234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5510 msgr2=0x7f86f41559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.309+0000 7f86fc691640 1 --2- 172.21.15.78:0/2888560234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5510 0x7f86f41559c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f86f40c9130 tx=0x7f86ec02bf80 comp rx=0 tx=0).stop 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.310+0000 7f86fc691640 1 -- 172.21.15.78:0/2888560234 shutdown_connections 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.310+0000 7f86fc691640 1 --2- 172.21.15.78:0/2888560234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5510 0x7f86f41559c0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.310+0000 7f86fc691640 1 --2- 172.21.15.78:0/2888560234 >> v2:172.21.15.150:3300/0 conn(0x7f86f40c5130 0x7f86f40c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.310+0000 7f86fc691640 1 -- 172.21.15.78:0/2888560234 >> 172.21.15.78:0/2888560234 conn(0x7f86f406b4b0 msgr2=0x7f86f40c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.310+0000 7f86fc691640 1 -- 172.21.15.78:0/2888560234 shutdown_connections 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.310+0000 7f86fc691640 1 -- 172.21.15.78:0/2888560234 wait complete. 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 Processor -- start 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 -- start start 2024-02-19T14:59:58.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 0x7f86f41eaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86f40c5510 0x7f86f4070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86f4158f80 0x7f86f4157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f414f700 con 0x7f86f40c5130 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f4150800 con 0x7f86f40c5510 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fc691640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f41eb410 con 0x7f86f4158f80 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fa42d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 0x7f86f41eaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fa42d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 0x7f86f41eaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36006/0 (socket says 172.21.15.78:36006) 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86fa42d640 1 -- 172.21.15.78:0/4251313323 learned_addr learned my addr 172.21.15.78:0/4251313323 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.311+0000 7f86f9c2c640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86f40c5510 0x7f86f4070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fac2e640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86f4158f80 0x7f86f4157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fa42d640 1 -- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86f4158f80 msgr2=0x7f86f4157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fa42d640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86f4158f80 0x7f86f4157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fa42d640 1 -- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86f40c5510 msgr2=0x7f86f4070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fa42d640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86f40c5510 0x7f86f4070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fa42d640 1 -- 172.21.15.78:0/4251313323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec009b30 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fa42d640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 0x7f86f41eaed0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c7f0 tx=0x7f86f000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0007a30 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0007b90 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f000ce40 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f41467e0 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.312+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f4146da0 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.313+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f4074990 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.313+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f86f00113f0 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.314+0000 7f86eaffd640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86d806b810 0x7f86d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.314+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f86f008a3d0 con 0x7f86f40c5130 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.314+0000 7f86f9c2c640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86d806b810 0x7f86d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.314+0000 7f86f9c2c640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86d806b810 0x7f86d806dcd0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f86f41e8400 tx=0x7f86e4007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.318+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f86f0054da0 con 0x7f86f40c5130 2024-02-19T14:59:58.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.496+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86f40c9400 con 0x7f86f40c5130 2024-02-19T14:59:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.497+0000 7f86eaffd640 1 -- 172.21.15.78:0/4251313323 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16940 (secure 0 0 0) 0x7f86f0058460 con 0x7f86f40c5130 2024-02-19T14:59:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:58.500 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":202,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:56.830521+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:53.390533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:54.145380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T14:59:58.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86d806b810 msgr2=0x7f86d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86d806b810 0x7f86d806dcd0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f86f41e8400 tx=0x7f86e4007de0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 msgr2=0x7f86f41eaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 0x7f86f41eaed0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c7f0 tx=0x7f86f000cbf0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 shutdown_connections 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f86d806b810 0x7f86d806dcd0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86f4158f80 0x7f86f4157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86f40c5510 0x7f86f4070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 --2- 172.21.15.78:0/4251313323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86f40c5130 0x7f86f41eaed0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.499+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 >> 172.21.15.78:0/4251313323 conn(0x7f86f406b4b0 msgr2=0x7f86f40c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.500+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 shutdown_connections 2024-02-19T14:59:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.500+0000 7f86fc691640 1 -- 172.21.15.78:0/4251313323 wait complete. 2024-02-19T14:59:58.517 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T14:59:58.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.614+0000 7fc5c8eed640 1 Processor -- start 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.615+0000 7fc5c8eed640 1 -- start start 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c8eed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 0x7fc5c40c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c8eed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c406fcd0 con 0x7fc5c40c7630 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c8eed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c406ff70 con 0x7fc5c414f2c0 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c8eed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c4151ce0 con 0x7fc5c41529b0 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc5c40c7630 0x7fc5c415c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40296/0 (socket says 172.21.15.78:40296) 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c2d9d640 1 -- 172.21.15.78:0/2100904162 learned_addr learned my addr 172.21.15.78:0/2100904162 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221718632 0 0) 0x7fc5c406fcd0 con 0x7fc5c40c7630 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003520 con 0x7fc5c40c7630 2024-02-19T14:59:58.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5bbfff640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 0x7fc5c40c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943065722 0 0) 0x7fc5b0003520 con 0x7fc5c40c7630 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 msgr2=0x7fc5c40c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 0x7fc5c40c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 >> v1:172.21.15.150:6789/0 conn(0x7fc5c414f2c0 legacy=0x7fc5c41552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4151ed0 con 0x7fc5c40c7630 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fc5bc0019f0 con 0x7fc5c40c7630 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 >> v1:172.21.15.78:6789/0 conn(0x7fc5c40c7630 legacy=0x7fc5c415c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.616+0000 7fc5c1d9b640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c1d9b640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 0x7fc5b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c1d9b640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0004d50 0x7fc5b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000bc00 con 0x7fc5b00039a0 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000bd70 con 0x7fc5b0004d50 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b000bf30 con 0x7fc5c41529b0 2024-02-19T14:59:58.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c259c640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5bbfff640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 0x7fc5b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c2d9d640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0004d50 0x7fc5b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c259c640 1 -- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 msgr2=0x7fc5b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c259c640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c41529b0 0x7fc5b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c259c640 1 -- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0004d50 msgr2=0x7fc5b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c259c640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0004d50 0x7fc5b00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.617+0000 7fc5c259c640 1 -- 172.21.15.78:0/2100904162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0003520 con 0x7fc5b00039a0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c259c640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4001720 tx=0x7fc5b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f287882975dee6c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40052d0 con 0x7fc5b00039a0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b402e050 con 0x7fc5b00039a0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c1d9b640 1 -- 172.21.15.78:0/2100904162 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40035f0 con 0x7fc5b00039a0 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c8eed640 1 -- 172.21.15.78:0/2100904162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 msgr2=0x7fc5b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/2100904162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5b0003dc0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4001720 tx=0x7fc5b00057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c2d9d640 1 -- 172.21.15.78:0/2100904162 reap_dead start 2024-02-19T14:59:58.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c8eed640 1 -- 172.21.15.78:0/2100904162 shutdown_connections 2024-02-19T14:59:58.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c8eed640 1 -- 172.21.15.78:0/2100904162 >> 172.21.15.78:0/2100904162 conn(0x7fc5c406ea10 msgr2=0x7fc5c40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:58.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c8eed640 1 -- 172.21.15.78:0/2100904162 shutdown_connections 2024-02-19T14:59:58.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.618+0000 7fc5c8eed640 1 -- 172.21.15.78:0/2100904162 wait complete. 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 Processor -- start 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 -- start start 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5c41ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c41493f0 0x7fc5c41545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c40c3420 0x7fc5c41df9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c41ee4b0 con 0x7fc5b00039a0 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40aa520 con 0x7fc5c41493f0 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c8eed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c40aa800 con 0x7fc5c40c3420 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5c41ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5c41ee8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36030/0 (socket says 172.21.15.78:36030) 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.620+0000 7fc5c259c640 1 -- 172.21.15.78:0/1493068827 learned_addr learned my addr 172.21.15.78:0/1493068827 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:58.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5bbfff640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c41493f0 0x7fc5c41545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c2d9d640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c40c3420 0x7fc5c41df9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c259c640 1 -- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c40c3420 msgr2=0x7fc5c41df9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c259c640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c40c3420 0x7fc5c41df9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c259c640 1 -- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c41493f0 msgr2=0x7fc5c41545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c259c640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c41493f0 0x7fc5c41545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c259c640 1 -- 172.21.15.78:0/1493068827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c41eddf0 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5bbfff640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c41493f0 0x7fc5c41545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c259c640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5c41ee8d0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4008fd0 tx=0x7fc5b4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40076c0 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4007a30 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4006c00 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c41ee0d0 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.621+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c4154f80 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.622+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c4074990 con 0x7fc5b00039a0 2024-02-19T14:59:58.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.623+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc5b40035f0 con 0x7fc5b00039a0 2024-02-19T14:59:58.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.623+0000 7fc5baffd640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5a406b810 0x7fc5a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.623+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5b40ac110 con 0x7fc5b00039a0 2024-02-19T14:59:58.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.623+0000 7fc5bbfff640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5a406b810 0x7fc5a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.623+0000 7fc5bbfff640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5a406b810 0x7fc5a406dcd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40a8940 tx=0x7fc5ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.627+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc5b4076a30 con 0x7fc5b00039a0 2024-02-19T14:59:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.801+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5c40c1b30 con 0x7fc5a406b810 2024-02-19T14:59:58.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.808+0000 7fc5baffd640 1 -- 172.21.15.78:0/1493068827 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289989 (secure 0 0 0) 0x7fc5c40c1b30 con 0x7fc5a406b810 2024-02-19T14:59:58.814 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:58.816 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5a406b810 msgr2=0x7fc5a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5a406b810 0x7fc5a406dcd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40a8940 tx=0x7fc5ac008040 comp rx=0 tx=0).stop 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 msgr2=0x7fc5c41ee8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5c41ee8d0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4008fd0 tx=0x7fc5b4001120 comp rx=0 tx=0).stop 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 shutdown_connections 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc5a406b810 0x7fc5a406dcd0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc5c40c3420 0x7fc5c41df9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c41493f0 0x7fc5c41545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 --2- 172.21.15.78:0/1493068827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5b00039a0 0x7fc5c41ee8d0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 >> 172.21.15.78:0/1493068827 conn(0x7fc5c406ea10 msgr2=0x7fc5c414d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 shutdown_connections 2024-02-19T14:59:58.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.812+0000 7fc5c8eed640 1 -- 172.21.15.78:0/1493068827 wait complete. 2024-02-19T14:59:58.826 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-02-19T14:59:58.153585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270700,"kb_used_data":21224,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754704020,"statfs":{"total":773094113280,"available":772816916480,"internally_reserved":0,"allocated":21733376,"data_stored":11203604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.329357"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950725+0000","last_change":"2024-02-19T14:59:55.860557+0000","last_active":"2024-02-19T14:59:56.950725+0000","last_peered":"2024-02-19T14:59:56.950725+0000","last_clean":"2024-02-19T14:59:56.950725+0000","last_became_active":"2024-02-19T14:59:55.860294+0000","last_became_peered":"2024-02-19T14:59:55.860294+0000","last_unstale":"2024-02-19T14:59:56.950725+0000","last_undegraded":"2024-02-19T14:59:56.950725+0000","last_fullsized":"2024-02-19T14:59:56.950725+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:17:48.596442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951199+0000","last_change":"2024-02-19T14:59:55.859202+0000","last_active":"2024-02-19T14:59:56.951199+0000","last_peered":"2024-02-19T14:59:56.951199+0000","last_clean":"2024-02-19T14:59:56.951199+0000","last_became_active":"2024-02-19T14:59:55.858082+0000","last_became_peered":"2024-02-19T14:59:55.858082+0000","last_unstale":"2024-02-19T14:59:56.951199+0000","last_undegraded":"2024-02-19T14:59:56.951199+0000","last_fullsized":"2024-02-19T14:59:56.951199+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:18.254704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833643+0000","last_change":"2024-02-19T14:59:55.854227+0000","last_active":"2024-02-19T14:59:56.833643+0000","last_peered":"2024-02-19T14:59:56.833643+0000","last_clean":"2024-02-19T14:59:56.833643+0000","last_became_active":"2024-02-19T14:59:55.853972+0000","last_became_peered":"2024-02-19T14:59:55.853972+0000","last_unstale":"2024-02-19T14:59:56.833643+0000","last_undegraded":"2024-02-19T14:59:56.833643+0000","last_fullsized":"2024-02-19T14:59:56.833643+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:27:32.442528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833623+0000","last_change":"2024-02-19T14:59:55.861264+0000","last_active":"2024-02-19T14:59:56.833623+0000","last_peered":"2024-02-19T14:59:56.833623+0000","last_clean":"2024-02-19T14:59:56.833623+0000","last_became_active":"2024-02-19T14:59:55.860144+0000","last_became_peered":"2024-02-19T14:59:55.860144+0000","last_unstale":"2024-02-19T14:59:56.833623+0000","last_undegraded":"2024-02-19T14:59:56.833623+0000","last_fullsized":"2024-02-19T14:59:56.833623+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:21.621871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950990+0000","last_change":"2024-02-19T14:59:55.850506+0000","last_active":"2024-02-19T14:59:56.950990+0000","last_peered":"2024-02-19T14:59:56.950990+0000","last_clean":"2024-02-19T14:59:56.950990+0000","last_became_active":"2024-02-19T14:59:55.849976+0000","last_became_peered":"2024-02-19T14:59:55.849976+0000","last_unstale":"2024-02-19T14:59:56.950990+0000","last_undegraded":"2024-02-19T14:59:56.950990+0000","last_fullsized":"2024-02-19T14:59:56.950990+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:24.951807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950374+0000","last_change":"2024-02-19T14:59:55.847289+0000","last_active":"2024-02-19T14:59:56.950374+0000","last_peered":"2024-02-19T14:59:56.950374+0000","last_clean":"2024-02-19T14:59:56.950374+0000","last_became_active":"2024-02-19T14:59:55.847014+0000","last_became_peered":"2024-02-19T14:59:55.847014+0000","last_unstale":"2024-02-19T14:59:56.950374+0000","last_undegraded":"2024-02-19T14:59:56.950374+0000","last_fullsized":"2024-02-19T14:59:56.950374+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:14.218443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835174+0000","last_change":"2024-02-19T14:59:55.856571+0000","last_active":"2024-02-19T14:59:56.835174+0000","last_peered":"2024-02-19T14:59:56.835174+0000","last_clean":"2024-02-19T14:59:56.835174+0000","last_became_active":"2024-02-19T14:59:55.856243+0000","last_became_peered":"2024-02-19T14:59:55.856243+0000","last_unstale":"2024-02-19T14:59:56.835174+0000","last_undegraded":"2024-02-19T14:59:56.835174+0000","last_fullsized":"2024-02-19T14:59:56.835174+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:25.963752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950173+0000","last_change":"2024-02-19T14:59:55.853857+0000","last_active":"2024-02-19T14:59:56.950173+0000","last_peered":"2024-02-19T14:59:56.950173+0000","last_clean":"2024-02-19T14:59:56.950173+0000","last_became_active":"2024-02-19T14:59:55.853599+0000","last_became_peered":"2024-02-19T14:59:55.853599+0000","last_unstale":"2024-02-19T14:59:56.950173+0000","last_undegraded":"2024-02-19T14:59:56.950173+0000","last_fullsized":"2024-02-19T14:59:56.950173+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:30.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950864+0000","last_change":"2024-02-19T14:59:55.846715+0000","last_active":"2024-02-19T14:59:56.950864+0000","last_peered":"2024-02-19T14:59:56.950864+0000","last_clean":"2024-02-19T14:59:56.950864+0000","last_became_active":"2024-02-19T14:59:55.846452+0000","last_became_peered":"2024-02-19T14:59:55.846452+0000","last_unstale":"2024-02-19T14:59:56.950864+0000","last_undegraded":"2024-02-19T14:59:56.950864+0000","last_fullsized":"2024-02-19T14:59:56.950864+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:54.892526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833751+0000","last_change":"2024-02-19T14:59:55.856472+0000","last_active":"2024-02-19T14:59:56.833751+0000","last_peered":"2024-02-19T14:59:56.833751+0000","last_clean":"2024-02-19T14:59:56.833751+0000","last_became_active":"2024-02-19T14:59:55.856191+0000","last_became_peered":"2024-02-19T14:59:55.856191+0000","last_unstale":"2024-02-19T14:59:56.833751+0000","last_undegraded":"2024-02-19T14:59:56.833751+0000","last_fullsized":"2024-02-19T14:59:56.833751+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:18:39.380631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833714+0000","last_change":"2024-02-19T14:59:55.856271+0000","last_active":"2024-02-19T14:59:56.833714+0000","last_peered":"2024-02-19T14:59:56.833714+0000","last_clean":"2024-02-19T14:59:56.833714+0000","last_became_active":"2024-02-19T14:59:55.855990+0000","last_became_peered":"2024-02-19T14:59:55.855990+0000","last_unstale":"2024-02-19T14:59:56.833714+0000","last_undegraded":"2024-02-19T14:59:56.833714+0000","last_fullsized":"2024-02-19T14:59:56.833714+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:35.018838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950259+0000","last_change":"2024-02-19T14:59:55.838067+0000","last_active":"2024-02-19T14:59:56.950259+0000","last_peered":"2024-02-19T14:59:56.950259+0000","last_clean":"2024-02-19T14:59:56.950259+0000","last_became_active":"2024-02-19T14:59:55.837761+0000","last_became_peered":"2024-02-19T14:59:55.837761+0000","last_unstale":"2024-02-19T14:59:56.950259+0000","last_undegraded":"2024-02-19T14:59:56.950259+0000","last_fullsized":"2024-02-19T14:59:56.950259+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:15.435031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835811+0000","last_change":"2024-02-19T14:59:55.860660+0000","last_active":"2024-02-19T14:59:56.835811+0000","last_peered":"2024-02-19T14:59:56.835811+0000","last_clean":"2024-02-19T14:59:56.835811+0000","last_became_active":"2024-02-19T14:59:55.860371+0000","last_became_peered":"2024-02-19T14:59:55.860371+0000","last_unstale":"2024-02-19T14:59:56.835811+0000","last_undegraded":"2024-02-19T14:59:56.835811+0000","last_fullsized":"2024-02-19T14:59:56.835811+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:18.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949763+0000","last_change":"2024-02-19T14:59:55.849652+0000","last_active":"2024-02-19T14:59:56.949763+0000","last_peered":"2024-02-19T14:59:56.949763+0000","last_clean":"2024-02-19T14:59:56.949763+0000","last_became_active":"2024-02-19T14:59:55.848746+0000","last_became_peered":"2024-02-19T14:59:55.848746+0000","last_unstale":"2024-02-19T14:59:56.949763+0000","last_undegraded":"2024-02-19T14:59:56.949763+0000","last_fullsized":"2024-02-19T14:59:56.949763+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:27.807111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950461+0000","last_change":"2024-02-19T14:59:55.855006+0000","last_active":"2024-02-19T14:59:56.950461+0000","last_peered":"2024-02-19T14:59:56.950461+0000","last_clean":"2024-02-19T14:59:56.950461+0000","last_became_active":"2024-02-19T14:59:55.854728+0000","last_became_peered":"2024-02-19T14:59:55.854728+0000","last_unstale":"2024-02-19T14:59:56.950461+0000","last_undegraded":"2024-02-19T14:59:56.950461+0000","last_fullsized":"2024-02-19T14:59:56.950461+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:51.831198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833361+0000","last_change":"2024-02-19T14:59:55.860301+0000","last_active":"2024-02-19T14:59:56.833361+0000","last_peered":"2024-02-19T14:59:56.833361+0000","last_clean":"2024-02-19T14:59:56.833361+0000","last_became_active":"2024-02-19T14:59:55.858671+0000","last_became_peered":"2024-02-19T14:59:55.858671+0000","last_unstale":"2024-02-19T14:59:56.833361+0000","last_undegraded":"2024-02-19T14:59:56.833361+0000","last_fullsized":"2024-02-19T14:59:56.833361+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:51.831198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835816+0000","last_change":"2024-02-19T14:59:55.852800+0000","last_active":"2024-02-19T14:59:56.835816+0000","last_peered":"2024-02-19T14:59:56.835816+0000","last_clean":"2024-02-19T14:59:56.835816+0000","last_became_active":"2024-02-19T14:59:55.849421+0000","last_became_peered":"2024-02-19T14:59:55.849421+0000","last_unstale":"2024-02-19T14:59:56.835816+0000","last_undegraded":"2024-02-19T14:59:56.835816+0000","last_fullsized":"2024-02-19T14:59:56.835816+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:08.204619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833758+0000","last_change":"2024-02-19T14:59:55.855281+0000","last_active":"2024-02-19T14:59:56.833758+0000","last_peered":"2024-02-19T14:59:56.833758+0000","last_clean":"2024-02-19T14:59:56.833758+0000","last_became_active":"2024-02-19T14:59:55.855011+0000","last_became_peered":"2024-02-19T14:59:55.855011+0000","last_unstale":"2024-02-19T14:59:56.833758+0000","last_undegraded":"2024-02-19T14:59:56.833758+0000","last_fullsized":"2024-02-19T14:59:56.833758+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:09.338353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950500+0000","last_change":"2024-02-19T14:59:55.852244+0000","last_active":"2024-02-19T14:59:56.950500+0000","last_peered":"2024-02-19T14:59:56.950500+0000","last_clean":"2024-02-19T14:59:56.950500+0000","last_became_active":"2024-02-19T14:59:55.851901+0000","last_became_peered":"2024-02-19T14:59:55.851901+0000","last_unstale":"2024-02-19T14:59:56.950500+0000","last_undegraded":"2024-02-19T14:59:56.950500+0000","last_fullsized":"2024-02-19T14:59:56.950500+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:29:55.479033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951326+0000","last_change":"2024-02-19T14:59:55.858207+0000","last_active":"2024-02-19T14:59:56.951326+0000","last_peered":"2024-02-19T14:59:56.951326+0000","last_clean":"2024-02-19T14:59:56.951326+0000","last_became_active":"2024-02-19T14:59:55.856282+0000","last_became_peered":"2024-02-19T14:59:55.856282+0000","last_unstale":"2024-02-19T14:59:56.951326+0000","last_undegraded":"2024-02-19T14:59:56.951326+0000","last_fullsized":"2024-02-19T14:59:56.951326+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:03.038209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835285+0000","last_change":"2024-02-19T14:59:55.846714+0000","last_active":"2024-02-19T14:59:56.835285+0000","last_peered":"2024-02-19T14:59:56.835285+0000","last_clean":"2024-02-19T14:59:56.835285+0000","last_became_active":"2024-02-19T14:59:55.846422+0000","last_became_peered":"2024-02-19T14:59:55.846422+0000","last_unstale":"2024-02-19T14:59:56.835285+0000","last_undegraded":"2024-02-19T14:59:56.835285+0000","last_fullsized":"2024-02-19T14:59:56.835285+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:52.220536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835051+0000","last_change":"2024-02-19T14:59:55.841606+0000","last_active":"2024-02-19T14:59:56.835051+0000","last_peered":"2024-02-19T14:59:56.835051+0000","last_clean":"2024-02-19T14:59:56.835051+0000","last_became_active":"2024-02-19T14:59:55.840425+0000","last_became_peered":"2024-02-19T14:59:55.840425+0000","last_unstale":"2024-02-19T14:59:56.835051+0000","last_undegraded":"2024-02-19T14:59:56.835051+0000","last_fullsized":"2024-02-19T14:59:56.835051+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:10.676092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950490+0000","last_change":"2024-02-19T14:59:55.852748+0000","last_active":"2024-02-19T14:59:56.950490+0000","last_peered":"2024-02-19T14:59:56.950490+0000","last_clean":"2024-02-19T14:59:56.950490+0000","last_became_active":"2024-02-19T14:59:55.852296+0000","last_became_peered":"2024-02-19T14:59:55.852296+0000","last_unstale":"2024-02-19T14:59:56.950490+0000","last_undegraded":"2024-02-19T14:59:56.950490+0000","last_fullsized":"2024-02-19T14:59:56.950490+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:30.266982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.834787+0000","last_change":"2024-02-19T14:59:55.838192+0000","last_active":"2024-02-19T14:59:56.834787+0000","last_peered":"2024-02-19T14:59:56.834787+0000","last_clean":"2024-02-19T14:59:56.834787+0000","last_became_active":"2024-02-19T14:59:55.837866+0000","last_became_peered":"2024-02-19T14:59:55.837866+0000","last_unstale":"2024-02-19T14:59:56.834787+0000","last_undegraded":"2024-02-19T14:59:56.834787+0000","last_fullsized":"2024-02-19T14:59:56.834787+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:53.615527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951491+0000","last_change":"2024-02-19T14:59:55.858422+0000","last_active":"2024-02-19T14:59:56.951491+0000","last_peered":"2024-02-19T14:59:56.951491+0000","last_clean":"2024-02-19T14:59:56.951491+0000","last_became_active":"2024-02-19T14:59:55.855797+0000","last_became_peered":"2024-02-19T14:59:55.855797+0000","last_unstale":"2024-02-19T14:59:56.951491+0000","last_undegraded":"2024-02-19T14:59:56.951491+0000","last_fullsized":"2024-02-19T14:59:56.951491+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:18.532570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835105+0000","last_change":"2024-02-19T14:59:55.857690+0000","last_active":"2024-02-19T14:59:56.835105+0000","last_peered":"2024-02-19T14:59:56.835105+0000","last_clean":"2024-02-19T14:59:56.835105+0000","last_became_active":"2024-02-19T14:59:55.857423+0000","last_became_peered":"2024-02-19T14:59:55.857423+0000","last_unstale":"2024-02-19T14:59:56.835105+0000","last_undegraded":"2024-02-19T14:59:56.835105+0000","last_fullsized":"2024-02-19T14:59:56.835105+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:50.830780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950735+0000","last_change":"2024-02-19T14:59:55.845581+0000","last_active":"2024-02-19T14:59:56.950735+0000","last_peered":"2024-02-19T14:59:56.950735+0000","last_clean":"2024-02-19T14:59:56.950735+0000","last_became_active":"2024-02-19T14:59:55.845247+0000","last_became_peered":"2024-02-19T14:59:55.845247+0000","last_unstale":"2024-02-19T14:59:56.950735+0000","last_undegraded":"2024-02-19T14:59:56.950735+0000","last_fullsized":"2024-02-19T14:59:56.950735+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:52.220536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950400+0000","last_change":"2024-02-19T14:59:55.847106+0000","last_active":"2024-02-19T14:59:56.950400+0000","last_peered":"2024-02-19T14:59:56.950400+0000","last_clean":"2024-02-19T14:59:56.950400+0000","last_became_active":"2024-02-19T14:59:55.846827+0000","last_became_peered":"2024-02-19T14:59:55.846827+0000","last_unstale":"2024-02-19T14:59:56.950400+0000","last_undegraded":"2024-02-19T14:59:56.950400+0000","last_fullsized":"2024-02-19T14:59:56.950400+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:01:52.501855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950123+0000","last_change":"2024-02-19T14:59:55.848307+0000","last_active":"2024-02-19T14:59:56.950123+0000","last_peered":"2024-02-19T14:59:56.950123+0000","last_clean":"2024-02-19T14:59:56.950123+0000","last_became_active":"2024-02-19T14:59:55.848034+0000","last_became_peered":"2024-02-19T14:59:55.848034+0000","last_unstale":"2024-02-19T14:59:56.950123+0000","last_undegraded":"2024-02-19T14:59:56.950123+0000","last_fullsized":"2024-02-19T14:59:56.950123+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:33.068323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949890+0000","last_change":"2024-02-19T14:59:55.847389+0000","last_active":"2024-02-19T14:59:56.949890+0000","last_peered":"2024-02-19T14:59:56.949890+0000","last_clean":"2024-02-19T14:59:56.949890+0000","last_became_active":"2024-02-19T14:59:55.846627+0000","last_became_peered":"2024-02-19T14:59:55.846627+0000","last_unstale":"2024-02-19T14:59:56.949890+0000","last_undegraded":"2024-02-19T14:59:56.949890+0000","last_fullsized":"2024-02-19T14:59:56.949890+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:06.037741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950746+0000","last_change":"2024-02-19T14:59:55.852061+0000","last_active":"2024-02-19T14:59:56.950746+0000","last_peered":"2024-02-19T14:59:56.950746+0000","last_clean":"2024-02-19T14:59:56.950746+0000","last_became_active":"2024-02-19T14:59:55.851791+0000","last_became_peered":"2024-02-19T14:59:55.851791+0000","last_unstale":"2024-02-19T14:59:56.950746+0000","last_undegraded":"2024-02-19T14:59:56.950746+0000","last_fullsized":"2024-02-19T14:59:56.950746+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:45:27.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949876+0000","last_change":"2024-02-19T14:59:55.847738+0000","last_active":"2024-02-19T14:59:56.949876+0000","last_peered":"2024-02-19T14:59:56.949876+0000","last_clean":"2024-02-19T14:59:56.949876+0000","last_became_active":"2024-02-19T14:59:55.847438+0000","last_became_peered":"2024-02-19T14:59:55.847438+0000","last_unstale":"2024-02-19T14:59:56.949876+0000","last_undegraded":"2024-02-19T14:59:56.949876+0000","last_fullsized":"2024-02-19T14:59:56.949876+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:12:17.209037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950231+0000","last_change":"2024-02-19T14:59:55.858050+0000","last_active":"2024-02-19T14:59:56.950231+0000","last_peered":"2024-02-19T14:59:56.950231+0000","last_clean":"2024-02-19T14:59:56.950231+0000","last_became_active":"2024-02-19T14:59:55.857754+0000","last_became_peered":"2024-02-19T14:59:55.857754+0000","last_unstale":"2024-02-19T14:59:56.950231+0000","last_undegraded":"2024-02-19T14:59:56.950231+0000","last_fullsized":"2024-02-19T14:59:56.950231+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:17.100730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"201'4","reported_seq":18,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950746+0000","last_change":"2024-02-19T14:59:54.850831+0000","last_active":"2024-02-19T14:59:56.950746+0000","last_peered":"2024-02-19T14:59:56.950746+0000","last_clean":"2024-02-19T14:59:56.950746+0000","last_became_active":"2024-02-19T14:59:54.850434+0000","last_became_peered":"2024-02-19T14:59:54.850434+0000","last_unstale":"2024-02-19T14:59:56.950746+0000","last_undegraded":"2024-02-19T14:59:56.950746+0000","last_fullsized":"2024-02-19T14:59:56.950746+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:53.548615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950117+0000","last_change":"2024-02-19T14:59:55.847052+0000","last_active":"2024-02-19T14:59:56.950117+0000","last_peered":"2024-02-19T14:59:56.950117+0000","last_clean":"2024-02-19T14:59:56.950117+0000","last_became_active":"2024-02-19T14:59:55.846788+0000","last_became_peered":"2024-02-19T14:59:55.846788+0000","last_unstale":"2024-02-19T14:59:56.950117+0000","last_undegraded":"2024-02-19T14:59:56.950117+0000","last_fullsized":"2024-02-19T14:59:56.950117+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:06.037741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.834901+0000","last_change":"2024-02-19T14:59:54.841297+0000","last_active":"2024-02-19T14:59:56.834901+0000","last_peered":"2024-02-19T14:59:56.834901+0000","last_clean":"2024-02-19T14:59:56.834901+0000","last_became_active":"2024-02-19T14:59:54.840553+0000","last_became_peered":"2024-02-19T14:59:54.840553+0000","last_unstale":"2024-02-19T14:59:56.834901+0000","last_undegraded":"2024-02-19T14:59:56.834901+0000","last_fullsized":"2024-02-19T14:59:56.834901+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:37.496659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950779+0000","last_change":"2024-02-19T14:59:55.846747+0000","last_active":"2024-02-19T14:59:56.950779+0000","last_peered":"2024-02-19T14:59:56.950779+0000","last_clean":"2024-02-19T14:59:56.950779+0000","last_became_active":"2024-02-19T14:59:55.846473+0000","last_became_peered":"2024-02-19T14:59:55.846473+0000","last_unstale":"2024-02-19T14:59:56.950779+0000","last_undegraded":"2024-02-19T14:59:56.950779+0000","last_fullsized":"2024-02-19T14:59:56.950779+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:52:29.692660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950408+0000","last_change":"2024-02-19T14:59:54.849731+0000","last_active":"2024-02-19T14:59:56.950408+0000","last_peered":"2024-02-19T14:59:56.950408+0000","last_clean":"2024-02-19T14:59:56.950408+0000","last_became_active":"2024-02-19T14:59:54.849492+0000","last_became_peered":"2024-02-19T14:59:54.849492+0000","last_unstale":"2024-02-19T14:59:56.950408+0000","last_undegraded":"2024-02-19T14:59:56.950408+0000","last_fullsized":"2024-02-19T14:59:56.950408+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:14.881873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950338+0000","last_change":"2024-02-19T14:59:55.858699+0000","last_active":"2024-02-19T14:59:56.950338+0000","last_peered":"2024-02-19T14:59:56.950338+0000","last_clean":"2024-02-19T14:59:56.950338+0000","last_became_active":"2024-02-19T14:59:55.858414+0000","last_became_peered":"2024-02-19T14:59:55.858414+0000","last_unstale":"2024-02-19T14:59:56.950338+0000","last_undegraded":"2024-02-19T14:59:56.950338+0000","last_fullsized":"2024-02-19T14:59:56.950338+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:51.831198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950786+0000","last_change":"2024-02-19T14:59:54.846189+0000","last_active":"2024-02-19T14:59:56.950786+0000","last_peered":"2024-02-19T14:59:56.950786+0000","last_clean":"2024-02-19T14:59:56.950786+0000","last_became_active":"2024-02-19T14:59:54.845872+0000","last_became_peered":"2024-02-19T14:59:54.845872+0000","last_unstale":"2024-02-19T14:59:56.950786+0000","last_undegraded":"2024-02-19T14:59:56.950786+0000","last_fullsized":"2024-02-19T14:59:56.950786+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:09.956992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835566+0000","last_change":"2024-02-19T14:59:55.845947+0000","last_active":"2024-02-19T14:59:56.835566+0000","last_peered":"2024-02-19T14:59:56.835566+0000","last_clean":"2024-02-19T14:59:56.835566+0000","last_became_active":"2024-02-19T14:59:55.845670+0000","last_became_peered":"2024-02-19T14:59:55.845670+0000","last_unstale":"2024-02-19T14:59:56.835566+0000","last_undegraded":"2024-02-19T14:59:56.835566+0000","last_fullsized":"2024-02-19T14:59:56.835566+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:15.889208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949986+0000","last_change":"2024-02-19T14:59:54.844743+0000","last_active":"2024-02-19T14:59:56.949986+0000","last_peered":"2024-02-19T14:59:56.949986+0000","last_clean":"2024-02-19T14:59:56.949986+0000","last_became_active":"2024-02-19T14:59:54.844479+0000","last_became_peered":"2024-02-19T14:59:54.844479+0000","last_unstale":"2024-02-19T14:59:56.949986+0000","last_undegraded":"2024-02-19T14:59:56.949986+0000","last_fullsized":"2024-02-19T14:59:56.949986+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:37.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833586+0000","last_change":"2024-02-19T14:59:55.857968+0000","last_active":"2024-02-19T14:59:56.833586+0000","last_peered":"2024-02-19T14:59:56.833586+0000","last_clean":"2024-02-19T14:59:56.833586+0000","last_became_active":"2024-02-19T14:59:55.857663+0000","last_became_peered":"2024-02-19T14:59:55.857663+0000","last_unstale":"2024-02-19T14:59:56.833586+0000","last_undegraded":"2024-02-19T14:59:56.833586+0000","last_fullsized":"2024-02-19T14:59:56.833586+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:33.622046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833308+0000","last_change":"2024-02-19T14:59:54.851699+0000","last_active":"2024-02-19T14:59:56.833308+0000","last_peered":"2024-02-19T14:59:56.833308+0000","last_clean":"2024-02-19T14:59:56.833308+0000","last_became_active":"2024-02-19T14:59:54.851328+0000","last_became_peered":"2024-02-19T14:59:54.851328+0000","last_unstale":"2024-02-19T14:59:56.833308+0000","last_undegraded":"2024-02-19T14:59:56.833308+0000","last_fullsized":"2024-02-19T14:59:56.833308+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:23.944471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951094+0000","last_change":"2024-02-19T14:59:55.850552+0000","last_active":"2024-02-19T14:59:56.951094+0000","last_peered":"2024-02-19T14:59:56.951094+0000","last_clean":"2024-02-19T14:59:56.951094+0000","last_became_active":"2024-02-19T14:59:55.850232+0000","last_became_peered":"2024-02-19T14:59:55.850232+0000","last_unstale":"2024-02-19T14:59:56.951094+0000","last_undegraded":"2024-02-19T14:59:56.951094+0000","last_fullsized":"2024-02-19T14:59:56.951094+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:10.964328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949860+0000","last_change":"2024-02-19T14:59:54.851004+0000","last_active":"2024-02-19T14:59:56.949860+0000","last_peered":"2024-02-19T14:59:56.949860+0000","last_clean":"2024-02-19T14:59:56.949860+0000","last_became_active":"2024-02-19T14:59:54.850773+0000","last_became_peered":"2024-02-19T14:59:54.850773+0000","last_unstale":"2024-02-19T14:59:56.949860+0000","last_undegraded":"2024-02-19T14:59:56.949860+0000","last_fullsized":"2024-02-19T14:59:56.949860+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:35:20.441122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.834969+0000","last_change":"2024-02-19T14:59:55.844315+0000","last_active":"2024-02-19T14:59:56.834969+0000","last_peered":"2024-02-19T14:59:56.834969+0000","last_clean":"2024-02-19T14:59:56.834969+0000","last_became_active":"2024-02-19T14:59:55.843969+0000","last_became_peered":"2024-02-19T14:59:55.843969+0000","last_unstale":"2024-02-19T14:59:56.834969+0000","last_undegraded":"2024-02-19T14:59:56.834969+0000","last_fullsized":"2024-02-19T14:59:56.834969+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:10.109452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949923+0000","last_change":"2024-02-19T14:59:54.845253+0000","last_active":"2024-02-19T14:59:56.949923+0000","last_peered":"2024-02-19T14:59:56.949923+0000","last_clean":"2024-02-19T14:59:56.949923+0000","last_became_active":"2024-02-19T14:59:54.844951+0000","last_became_peered":"2024-02-19T14:59:54.844951+0000","last_unstale":"2024-02-19T14:59:56.949923+0000","last_undegraded":"2024-02-19T14:59:56.949923+0000","last_fullsized":"2024-02-19T14:59:56.949923+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:23.944471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835568+0000","last_change":"2024-02-19T14:59:55.841047+0000","last_active":"2024-02-19T14:59:56.835568+0000","last_peered":"2024-02-19T14:59:56.835568+0000","last_clean":"2024-02-19T14:59:56.835568+0000","last_became_active":"2024-02-19T14:59:55.840715+0000","last_became_peered":"2024-02-19T14:59:55.840715+0000","last_unstale":"2024-02-19T14:59:56.835568+0000","last_undegraded":"2024-02-19T14:59:56.835568+0000","last_fullsized":"2024-02-19T14:59:56.835568+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:28.660287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833804+0000","last_change":"2024-02-19T14:59:54.846606+0000","last_active":"2024-02-19T14:59:56.833804+0000","last_peered":"2024-02-19T14:59:56.833804+0000","last_clean":"2024-02-19T14:59:56.833804+0000","last_became_active":"2024-02-19T14:59:54.844365+0000","last_became_peered":"2024-02-19T14:59:54.844365+0000","last_unstale":"2024-02-19T14:59:56.833804+0000","last_undegraded":"2024-02-19T14:59:56.833804+0000","last_fullsized":"2024-02-19T14:59:56.833804+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:03.342512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833395+0000","last_change":"2024-02-19T14:59:55.855006+0000","last_active":"2024-02-19T14:59:56.833395+0000","last_peered":"2024-02-19T14:59:56.833395+0000","last_clean":"2024-02-19T14:59:56.833395+0000","last_became_active":"2024-02-19T14:59:55.854717+0000","last_became_peered":"2024-02-19T14:59:55.854717+0000","last_unstale":"2024-02-19T14:59:56.833395+0000","last_undegraded":"2024-02-19T14:59:56.833395+0000","last_fullsized":"2024-02-19T14:59:56.833395+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:29:55.479033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"201'3","reported_seq":17,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950950+0000","last_change":"2024-02-19T14:59:54.846344+0000","last_active":"2024-02-19T14:59:56.950950+0000","last_peered":"2024-02-19T14:59:56.950950+0000","last_clean":"2024-02-19T14:59:56.950950+0000","last_became_active":"2024-02-19T14:59:54.846035+0000","last_became_peered":"2024-02-19T14:59:54.846035+0000","last_unstale":"2024-02-19T14:59:56.950950+0000","last_undegraded":"2024-02-19T14:59:56.950950+0000","last_fullsized":"2024-02-19T14:59:56.950950+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:45:39.339962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835134+0000","last_change":"2024-02-19T14:59:55.841234+0000","last_active":"2024-02-19T14:59:56.835134+0000","last_peered":"2024-02-19T14:59:56.835134+0000","last_clean":"2024-02-19T14:59:56.835134+0000","last_became_active":"2024-02-19T14:59:55.840935+0000","last_became_peered":"2024-02-19T14:59:55.840935+0000","last_unstale":"2024-02-19T14:59:56.835134+0000","last_undegraded":"2024-02-19T14:59:56.835134+0000","last_fullsized":"2024-02-19T14:59:56.835134+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:30.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"201'6","reported_seq":20,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950506+0000","last_change":"2024-02-19T14:59:54.847336+0000","last_active":"2024-02-19T14:59:56.950506+0000","last_peered":"2024-02-19T14:59:56.950506+0000","last_clean":"2024-02-19T14:59:56.950506+0000","last_became_active":"2024-02-19T14:59:54.847089+0000","last_became_peered":"2024-02-19T14:59:54.847089+0000","last_unstale":"2024-02-19T14:59:56.950506+0000","last_undegraded":"2024-02-19T14:59:56.950506+0000","last_fullsized":"2024-02-19T14:59:56.950506+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:31.781028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951242+0000","last_change":"2024-02-19T14:59:55.857384+0000","last_active":"2024-02-19T14:59:56.951242+0000","last_peered":"2024-02-19T14:59:56.951242+0000","last_clean":"2024-02-19T14:59:56.951242+0000","last_became_active":"2024-02-19T14:59:55.857143+0000","last_became_peered":"2024-02-19T14:59:55.857143+0000","last_unstale":"2024-02-19T14:59:56.951242+0000","last_undegraded":"2024-02-19T14:59:56.951242+0000","last_fullsized":"2024-02-19T14:59:56.951242+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:45.766479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950793+0000","last_change":"2024-02-19T14:59:54.848997+0000","last_active":"2024-02-19T14:59:56.950793+0000","last_peered":"2024-02-19T14:59:56.950793+0000","last_clean":"2024-02-19T14:59:56.950793+0000","last_became_active":"2024-02-19T14:59:54.848723+0000","last_became_peered":"2024-02-19T14:59:54.848723+0000","last_unstale":"2024-02-19T14:59:56.950793+0000","last_undegraded":"2024-02-19T14:59:56.950793+0000","last_fullsized":"2024-02-19T14:59:56.950793+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:37.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835546+0000","last_change":"2024-02-19T14:59:55.850638+0000","last_active":"2024-02-19T14:59:56.835546+0000","last_peered":"2024-02-19T14:59:56.835546+0000","last_clean":"2024-02-19T14:59:56.835546+0000","last_became_active":"2024-02-19T14:59:55.850379+0000","last_became_peered":"2024-02-19T14:59:55.850379+0000","last_unstale":"2024-02-19T14:59:56.835546+0000","last_undegraded":"2024-02-19T14:59:56.835546+0000","last_fullsized":"2024-02-19T14:59:56.835546+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:24.951807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950430+0000","last_change":"2024-02-19T14:59:54.843069+0000","last_active":"2024-02-19T14:59:56.950430+0000","last_peered":"2024-02-19T14:59:56.950430+0000","last_clean":"2024-02-19T14:59:56.950430+0000","last_became_active":"2024-02-19T14:59:54.842801+0000","last_became_peered":"2024-02-19T14:59:54.842801+0000","last_unstale":"2024-02-19T14:59:56.950430+0000","last_undegraded":"2024-02-19T14:59:56.950430+0000","last_fullsized":"2024-02-19T14:59:56.950430+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:35.960893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950398+0000","last_change":"2024-02-19T14:59:55.854497+0000","last_active":"2024-02-19T14:59:56.950398+0000","last_peered":"2024-02-19T14:59:56.950398+0000","last_clean":"2024-02-19T14:59:56.950398+0000","last_became_active":"2024-02-19T14:59:55.854186+0000","last_became_peered":"2024-02-19T14:59:55.854186+0000","last_unstale":"2024-02-19T14:59:56.950398+0000","last_undegraded":"2024-02-19T14:59:56.950398+0000","last_fullsized":"2024-02-19T14:59:56.950398+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:44:09.953528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950716+0000","last_change":"2024-02-19T14:59:54.847939+0000","last_active":"2024-02-19T14:59:56.950716+0000","last_peered":"2024-02-19T14:59:56.950716+0000","last_clean":"2024-02-19T14:59:56.950716+0000","last_became_active":"2024-02-19T14:59:54.847477+0000","last_became_peered":"2024-02-19T14:59:54.847477+0000","last_unstale":"2024-02-19T14:59:56.950716+0000","last_undegraded":"2024-02-19T14:59:56.950716+0000","last_fullsized":"2024-02-19T14:59:56.950716+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:46:51.013136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951012+0000","last_change":"2024-02-19T14:59:55.863635+0000","last_active":"2024-02-19T14:59:56.951012+0000","last_peered":"2024-02-19T14:59:56.951012+0000","last_clean":"2024-02-19T14:59:56.951012+0000","last_became_active":"2024-02-19T14:59:55.863315+0000","last_became_peered":"2024-02-19T14:59:55.863315+0000","last_unstale":"2024-02-19T14:59:56.951012+0000","last_undegraded":"2024-02-19T14:59:56.951012+0000","last_fullsized":"2024-02-19T14:59:56.951012+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:15.889208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"201'8","reported_seq":22,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950279+0000","last_change":"2024-02-19T14:59:54.843398+0000","last_active":"2024-02-19T14:59:56.950279+0000","last_peered":"2024-02-19T14:59:56.950279+0000","last_clean":"2024-02-19T14:59:56.950279+0000","last_became_active":"2024-02-19T14:59:54.843107+0000","last_became_peered":"2024-02-19T14:59:54.843107+0000","last_unstale":"2024-02-19T14:59:56.950279+0000","last_undegraded":"2024-02-19T14:59:56.950279+0000","last_fullsized":"2024-02-19T14:59:56.950279+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:24.450767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950051+0000","last_change":"2024-02-19T14:59:55.863946+0000","last_active":"2024-02-19T14:59:56.950051+0000","last_peered":"2024-02-19T14:59:56.950051+0000","last_clean":"2024-02-19T14:59:56.950051+0000","last_became_active":"2024-02-19T14:59:55.863617+0000","last_became_peered":"2024-02-19T14:59:55.863617+0000","last_unstale":"2024-02-19T14:59:56.950051+0000","last_undegraded":"2024-02-19T14:59:56.950051+0000","last_fullsized":"2024-02-19T14:59:56.950051+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:50.830780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950838+0000","last_change":"2024-02-19T14:59:54.844905+0000","last_active":"2024-02-19T14:59:56.950838+0000","last_peered":"2024-02-19T14:59:56.950838+0000","last_clean":"2024-02-19T14:59:56.950838+0000","last_became_active":"2024-02-19T14:59:54.844645+0000","last_became_peered":"2024-02-19T14:59:54.844645+0000","last_unstale":"2024-02-19T14:59:56.950838+0000","last_undegraded":"2024-02-19T14:59:56.950838+0000","last_fullsized":"2024-02-19T14:59:56.950838+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:50.571117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950750+0000","last_change":"2024-02-19T14:59:55.851312+0000","last_active":"2024-02-19T14:59:56.950750+0000","last_peered":"2024-02-19T14:59:56.950750+0000","last_clean":"2024-02-19T14:59:56.950750+0000","last_became_active":"2024-02-19T14:59:55.851046+0000","last_became_peered":"2024-02-19T14:59:55.851046+0000","last_unstale":"2024-02-19T14:59:56.950750+0000","last_undegraded":"2024-02-19T14:59:56.950750+0000","last_fullsized":"2024-02-19T14:59:56.950750+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:04.349848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951436+0000","last_change":"2024-02-19T14:59:54.842198+0000","last_active":"2024-02-19T14:59:56.951436+0000","last_peered":"2024-02-19T14:59:56.951436+0000","last_clean":"2024-02-19T14:59:56.951436+0000","last_became_active":"2024-02-19T14:59:54.841950+0000","last_became_peered":"2024-02-19T14:59:54.841950+0000","last_unstale":"2024-02-19T14:59:56.951436+0000","last_undegraded":"2024-02-19T14:59:56.951436+0000","last_fullsized":"2024-02-19T14:59:56.951436+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:47.951818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950825+0000","last_change":"2024-02-19T14:59:55.856810+0000","last_active":"2024-02-19T14:59:56.950825+0000","last_peered":"2024-02-19T14:59:56.950825+0000","last_clean":"2024-02-19T14:59:56.950825+0000","last_became_active":"2024-02-19T14:59:55.856227+0000","last_became_peered":"2024-02-19T14:59:55.856227+0000","last_unstale":"2024-02-19T14:59:56.950825+0000","last_undegraded":"2024-02-19T14:59:56.950825+0000","last_fullsized":"2024-02-19T14:59:56.950825+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:18:09.216544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835020+0000","last_change":"2024-02-19T14:59:54.839629+0000","last_active":"2024-02-19T14:59:56.835020+0000","last_peered":"2024-02-19T14:59:56.835020+0000","last_clean":"2024-02-19T14:59:56.835020+0000","last_became_active":"2024-02-19T14:59:54.839357+0000","last_became_peered":"2024-02-19T14:59:54.839357+0000","last_unstale":"2024-02-19T14:59:56.835020+0000","last_undegraded":"2024-02-19T14:59:56.835020+0000","last_fullsized":"2024-02-19T14:59:56.835020+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:34:28.645152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833411+0000","last_change":"2024-02-19T14:59:55.857108+0000","last_active":"2024-02-19T14:59:56.833411+0000","last_peered":"2024-02-19T14:59:56.833411+0000","last_clean":"2024-02-19T14:59:56.833411+0000","last_became_active":"2024-02-19T14:59:55.856850+0000","last_became_peered":"2024-02-19T14:59:55.856850+0000","last_unstale":"2024-02-19T14:59:56.833411+0000","last_undegraded":"2024-02-19T14:59:56.833411+0000","last_fullsized":"2024-02-19T14:59:56.833411+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:41:29.199791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950198+0000","last_change":"2024-02-19T14:59:54.848485+0000","last_active":"2024-02-19T14:59:56.950198+0000","last_peered":"2024-02-19T14:59:56.950198+0000","last_clean":"2024-02-19T14:59:56.950198+0000","last_became_active":"2024-02-19T14:59:54.848149+0000","last_became_peered":"2024-02-19T14:59:54.848149+0000","last_unstale":"2024-02-19T14:59:56.950198+0000","last_undegraded":"2024-02-19T14:59:56.950198+0000","last_fullsized":"2024-02-19T14:59:56.950198+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:14.881873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833581+0000","last_change":"2024-02-19T14:59:55.862519+0000","last_active":"2024-02-19T14:59:56.833581+0000","last_peered":"2024-02-19T14:59:56.833581+0000","last_clean":"2024-02-19T14:59:56.833581+0000","last_became_active":"2024-02-19T14:59:55.861517+0000","last_became_peered":"2024-02-19T14:59:55.861517+0000","last_unstale":"2024-02-19T14:59:56.833581+0000","last_undegraded":"2024-02-19T14:59:56.833581+0000","last_fullsized":"2024-02-19T14:59:56.833581+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:50:32.012617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833469+0000","last_change":"2024-02-19T14:59:54.843051+0000","last_active":"2024-02-19T14:59:56.833469+0000","last_peered":"2024-02-19T14:59:56.833469+0000","last_clean":"2024-02-19T14:59:56.833469+0000","last_became_active":"2024-02-19T14:59:54.842799+0000","last_became_peered":"2024-02-19T14:59:54.842799+0000","last_unstale":"2024-02-19T14:59:56.833469+0000","last_undegraded":"2024-02-19T14:59:56.833469+0000","last_fullsized":"2024-02-19T14:59:56.833469+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:09.956992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951092+0000","last_change":"2024-02-19T14:59:55.846683+0000","last_active":"2024-02-19T14:59:56.951092+0000","last_peered":"2024-02-19T14:59:56.951092+0000","last_clean":"2024-02-19T14:59:56.951092+0000","last_became_active":"2024-02-19T14:59:55.846413+0000","last_became_peered":"2024-02-19T14:59:55.846413+0000","last_unstale":"2024-02-19T14:59:56.951092+0000","last_undegraded":"2024-02-19T14:59:56.951092+0000","last_fullsized":"2024-02-19T14:59:56.951092+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:09:45.523448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835458+0000","last_change":"2024-02-19T14:59:54.849250+0000","last_active":"2024-02-19T14:59:56.835458+0000","last_peered":"2024-02-19T14:59:56.835458+0000","last_clean":"2024-02-19T14:59:56.835458+0000","last_became_active":"2024-02-19T14:59:54.848890+0000","last_became_peered":"2024-02-19T14:59:54.848890+0000","last_unstale":"2024-02-19T14:59:56.835458+0000","last_undegraded":"2024-02-19T14:59:56.835458+0000","last_fullsized":"2024-02-19T14:59:56.835458+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:57.193525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835190+0000","last_change":"2024-02-19T14:59:55.847294+0000","last_active":"2024-02-19T14:59:56.835190+0000","last_peered":"2024-02-19T14:59:56.835190+0000","last_clean":"2024-02-19T14:59:56.835190+0000","last_became_active":"2024-02-19T14:59:55.846990+0000","last_became_peered":"2024-02-19T14:59:55.846990+0000","last_unstale":"2024-02-19T14:59:56.835190+0000","last_undegraded":"2024-02-19T14:59:56.835190+0000","last_fullsized":"2024-02-19T14:59:56.835190+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:56.211567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835515+0000","last_change":"2024-02-19T14:59:54.842710+0000","last_active":"2024-02-19T14:59:56.835515+0000","last_peered":"2024-02-19T14:59:56.835515+0000","last_clean":"2024-02-19T14:59:56.835515+0000","last_became_active":"2024-02-19T14:59:54.842417+0000","last_became_peered":"2024-02-19T14:59:54.842417+0000","last_unstale":"2024-02-19T14:59:56.835515+0000","last_undegraded":"2024-02-19T14:59:56.835515+0000","last_fullsized":"2024-02-19T14:59:56.835515+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:45.389592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835805+0000","last_change":"2024-02-19T14:59:55.849204+0000","last_active":"2024-02-19T14:59:56.835805+0000","last_peered":"2024-02-19T14:59:56.835805+0000","last_clean":"2024-02-19T14:59:56.835805+0000","last_became_active":"2024-02-19T14:59:55.848928+0000","last_became_peered":"2024-02-19T14:59:55.848928+0000","last_unstale":"2024-02-19T14:59:56.835805+0000","last_undegraded":"2024-02-19T14:59:56.835805+0000","last_fullsized":"2024-02-19T14:59:56.835805+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:17:01.436292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949670+0000","last_change":"2024-02-19T14:59:54.840932+0000","last_active":"2024-02-19T14:59:56.949670+0000","last_peered":"2024-02-19T14:59:56.949670+0000","last_clean":"2024-02-19T14:59:56.949670+0000","last_became_active":"2024-02-19T14:59:54.840661+0000","last_became_peered":"2024-02-19T14:59:54.840661+0000","last_unstale":"2024-02-19T14:59:56.949670+0000","last_undegraded":"2024-02-19T14:59:56.949670+0000","last_fullsized":"2024-02-19T14:59:56.949670+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:03.342512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835367+0000","last_change":"2024-02-19T14:59:55.857244+0000","last_active":"2024-02-19T14:59:56.835367+0000","last_peered":"2024-02-19T14:59:56.835367+0000","last_clean":"2024-02-19T14:59:56.835367+0000","last_became_active":"2024-02-19T14:59:55.856943+0000","last_became_peered":"2024-02-19T14:59:55.856943+0000","last_unstale":"2024-02-19T14:59:56.835367+0000","last_undegraded":"2024-02-19T14:59:56.835367+0000","last_fullsized":"2024-02-19T14:59:56.835367+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:04.349848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833789+0000","last_change":"2024-02-19T14:59:54.846407+0000","last_active":"2024-02-19T14:59:56.833789+0000","last_peered":"2024-02-19T14:59:56.833789+0000","last_clean":"2024-02-19T14:59:56.833789+0000","last_became_active":"2024-02-19T14:59:54.846137+0000","last_became_peered":"2024-02-19T14:59:54.846137+0000","last_unstale":"2024-02-19T14:59:56.833789+0000","last_undegraded":"2024-02-19T14:59:56.833789+0000","last_fullsized":"2024-02-19T14:59:56.833789+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:51.213200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950249+0000","last_change":"2024-02-19T14:59:55.856334+0000","last_active":"2024-02-19T14:59:56.950249+0000","last_peered":"2024-02-19T14:59:56.950249+0000","last_clean":"2024-02-19T14:59:56.950249+0000","last_became_active":"2024-02-19T14:59:55.855981+0000","last_became_peered":"2024-02-19T14:59:55.855981+0000","last_unstale":"2024-02-19T14:59:56.950249+0000","last_undegraded":"2024-02-19T14:59:56.950249+0000","last_fullsized":"2024-02-19T14:59:56.950249+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:33.068323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"202'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:57.836399+0000","last_change":"2024-02-19T14:59:54.843777+0000","last_active":"2024-02-19T14:59:57.836399+0000","last_peered":"2024-02-19T14:59:57.836399+0000","last_clean":"2024-02-19T14:59:57.836399+0000","last_became_active":"2024-02-19T14:59:54.842585+0000","last_became_peered":"2024-02-19T14:59:54.842585+0000","last_unstale":"2024-02-19T14:59:57.836399+0000","last_undegraded":"2024-02-19T14:59:57.836399+0000","last_fullsized":"2024-02-19T14:59:57.836399+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:31.183050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833374+0000","last_change":"2024-02-19T14:59:55.858567+0000","last_active":"2024-02-19T14:59:56.833374+0000","last_peered":"2024-02-19T14:59:56.833374+0000","last_clean":"2024-02-19T14:59:56.833374+0000","last_became_active":"2024-02-19T14:59:55.858315+0000","last_became_peered":"2024-02-19T14:59:55.858315+0000","last_unstale":"2024-02-19T14:59:56.833374+0000","last_undegraded":"2024-02-19T14:59:56.833374+0000","last_fullsized":"2024-02-19T14:59:56.833374+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:30.266982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950147+0000","last_change":"2024-02-19T14:59:54.845201+0000","last_active":"2024-02-19T14:59:56.950147+0000","last_peered":"2024-02-19T14:59:56.950147+0000","last_clean":"2024-02-19T14:59:56.950147+0000","last_became_active":"2024-02-19T14:59:54.844910+0000","last_became_peered":"2024-02-19T14:59:54.844910+0000","last_unstale":"2024-02-19T14:59:56.950147+0000","last_undegraded":"2024-02-19T14:59:56.950147+0000","last_fullsized":"2024-02-19T14:59:56.950147+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:13:29.252202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950626+0000","last_change":"2024-02-19T14:59:55.863719+0000","last_active":"2024-02-19T14:59:56.950626+0000","last_peered":"2024-02-19T14:59:56.950626+0000","last_clean":"2024-02-19T14:59:56.950626+0000","last_became_active":"2024-02-19T14:59:55.863372+0000","last_became_peered":"2024-02-19T14:59:55.863372+0000","last_unstale":"2024-02-19T14:59:56.950626+0000","last_undegraded":"2024-02-19T14:59:56.950626+0000","last_fullsized":"2024-02-19T14:59:56.950626+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:25.963752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950431+0000","last_change":"2024-02-19T14:59:54.845513+0000","last_active":"2024-02-19T14:59:56.950431+0000","last_peered":"2024-02-19T14:59:56.950431+0000","last_clean":"2024-02-19T14:59:56.950431+0000","last_became_active":"2024-02-19T14:59:54.845245+0000","last_became_peered":"2024-02-19T14:59:54.845245+0000","last_unstale":"2024-02-19T14:59:56.950431+0000","last_undegraded":"2024-02-19T14:59:56.950431+0000","last_fullsized":"2024-02-19T14:59:56.950431+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:09.956992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950292+0000","last_change":"2024-02-19T14:59:55.848520+0000","last_active":"2024-02-19T14:59:56.950292+0000","last_peered":"2024-02-19T14:59:56.950292+0000","last_clean":"2024-02-19T14:59:56.950292+0000","last_became_active":"2024-02-19T14:59:55.847551+0000","last_became_peered":"2024-02-19T14:59:55.847551+0000","last_unstale":"2024-02-19T14:59:56.950292+0000","last_undegraded":"2024-02-19T14:59:56.950292+0000","last_fullsized":"2024-02-19T14:59:56.950292+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:44:09.953528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835861+0000","last_change":"2024-02-19T14:59:54.849318+0000","last_active":"2024-02-19T14:59:56.835861+0000","last_peered":"2024-02-19T14:59:56.835861+0000","last_clean":"2024-02-19T14:59:56.835861+0000","last_became_active":"2024-02-19T14:59:54.849012+0000","last_became_peered":"2024-02-19T14:59:54.849012+0000","last_unstale":"2024-02-19T14:59:56.835861+0000","last_undegraded":"2024-02-19T14:59:56.835861+0000","last_fullsized":"2024-02-19T14:59:56.835861+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:01.000065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950537+0000","last_change":"2024-02-19T14:59:55.849469+0000","last_active":"2024-02-19T14:59:56.950537+0000","last_peered":"2024-02-19T14:59:56.950537+0000","last_clean":"2024-02-19T14:59:56.950537+0000","last_became_active":"2024-02-19T14:59:55.849216+0000","last_became_peered":"2024-02-19T14:59:55.849216+0000","last_unstale":"2024-02-19T14:59:56.950537+0000","last_undegraded":"2024-02-19T14:59:56.950537+0000","last_fullsized":"2024-02-19T14:59:56.950537+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:16.092292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950476+0000","last_change":"2024-02-19T14:59:54.844701+0000","last_active":"2024-02-19T14:59:56.950476+0000","last_peered":"2024-02-19T14:59:56.950476+0000","last_clean":"2024-02-19T14:59:56.950476+0000","last_became_active":"2024-02-19T14:59:54.844435+0000","last_became_peered":"2024-02-19T14:59:54.844435+0000","last_unstale":"2024-02-19T14:59:56.950476+0000","last_undegraded":"2024-02-19T14:59:56.950476+0000","last_fullsized":"2024-02-19T14:59:56.950476+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:39.607914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951338+0000","last_change":"2024-02-19T14:59:55.859416+0000","last_active":"2024-02-19T14:59:56.951338+0000","last_peered":"2024-02-19T14:59:56.951338+0000","last_clean":"2024-02-19T14:59:56.951338+0000","last_became_active":"2024-02-19T14:59:55.857730+0000","last_became_peered":"2024-02-19T14:59:55.857730+0000","last_unstale":"2024-02-19T14:59:56.951338+0000","last_undegraded":"2024-02-19T14:59:56.951338+0000","last_fullsized":"2024-02-19T14:59:56.951338+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:47.689226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833373+0000","last_change":"2024-02-19T14:59:54.851476+0000","last_active":"2024-02-19T14:59:56.833373+0000","last_peered":"2024-02-19T14:59:56.833373+0000","last_clean":"2024-02-19T14:59:56.833373+0000","last_became_active":"2024-02-19T14:59:54.851201+0000","last_became_peered":"2024-02-19T14:59:54.851201+0000","last_unstale":"2024-02-19T14:59:56.833373+0000","last_undegraded":"2024-02-19T14:59:56.833373+0000","last_fullsized":"2024-02-19T14:59:56.833373+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:14.881873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835626+0000","last_change":"2024-02-19T14:59:55.842508+0000","last_active":"2024-02-19T14:59:56.835626+0000","last_peered":"2024-02-19T14:59:56.835626+0000","last_clean":"2024-02-19T14:59:56.835626+0000","last_became_active":"2024-02-19T14:59:55.842145+0000","last_became_peered":"2024-02-19T14:59:55.842145+0000","last_unstale":"2024-02-19T14:59:56.835626+0000","last_undegraded":"2024-02-19T14:59:56.835626+0000","last_fullsized":"2024-02-19T14:59:56.835626+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:53.809867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835267+0000","last_change":"2024-02-19T14:59:54.843469+0000","last_active":"2024-02-19T14:59:56.835267+0000","last_peered":"2024-02-19T14:59:56.835267+0000","last_clean":"2024-02-19T14:59:56.835267+0000","last_became_active":"2024-02-19T14:59:54.843130+0000","last_became_peered":"2024-02-19T14:59:54.843130+0000","last_unstale":"2024-02-19T14:59:56.835267+0000","last_undegraded":"2024-02-19T14:59:56.835267+0000","last_fullsized":"2024-02-19T14:59:56.835267+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:48.144190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950044+0000","last_change":"2024-02-19T14:59:55.847781+0000","last_active":"2024-02-19T14:59:56.950044+0000","last_peered":"2024-02-19T14:59:56.950044+0000","last_clean":"2024-02-19T14:59:56.950044+0000","last_became_active":"2024-02-19T14:59:55.845992+0000","last_became_peered":"2024-02-19T14:59:55.845992+0000","last_unstale":"2024-02-19T14:59:56.950044+0000","last_undegraded":"2024-02-19T14:59:56.950044+0000","last_fullsized":"2024-02-19T14:59:56.950044+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:17.056685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835206+0000","last_change":"2024-02-19T14:59:54.843958+0000","last_active":"2024-02-19T14:59:56.835206+0000","last_peered":"2024-02-19T14:59:56.835206+0000","last_clean":"2024-02-19T14:59:56.835206+0000","last_became_active":"2024-02-19T14:59:54.843503+0000","last_became_peered":"2024-02-19T14:59:54.843503+0000","last_unstale":"2024-02-19T14:59:56.835206+0000","last_undegraded":"2024-02-19T14:59:56.835206+0000","last_fullsized":"2024-02-19T14:59:56.835206+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:45:39.339962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":510,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949900+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:56.949900+0000","last_peered":"2024-02-19T14:59:56.949900+0000","last_clean":"2024-02-19T14:59:56.949900+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:56.949900+0000","last_undegraded":"2024-02-19T14:59:56.949900+0000","last_fullsized":"2024-02-19T14:59:56.949900+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34272,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337568,"statfs":{"total":96636764160,"available":96601669632,"internally_reserved":0,"allocated":3162112,"data_stored":1842748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.441}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.379},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33692,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338148,"statfs":{"total":96636764160,"available":96602263552,"internally_reserved":0,"allocated":2568192,"data_stored":1252368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.62}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51100000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.433},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.374}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.39100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33684,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338156,"statfs":{"total":96636764160,"available":96602271744,"internally_reserved":0,"allocated":2560000,"data_stored":1252256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.753},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.67000000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.59199999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.71599999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.91600000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.89200000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574950,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33688,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338152,"statfs":{"total":96636764160,"available":96602267648,"internally_reserved":0,"allocated":2564096,"data_stored":1252326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.441},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.51100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33708,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338132,"statfs":{"total":96636764160,"available":96602247168,"internally_reserved":0,"allocated":2584576,"data_stored":1254506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.315},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.28999999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33692,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338148,"statfs":{"total":96636764160,"available":96602263552,"internally_reserved":0,"allocated":2568192,"data_stored":1252908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.308},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.501},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33684,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338156,"statfs":{"total":96636764160,"available":96602271744,"internally_reserved":0,"allocated":2560000,"data_stored":1252256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.214},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.246}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":1844236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.253},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.23300000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.308}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:58.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T14:59:58.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.965+0000 7fb4150c7640 1 Processor -- start 2024-02-19T14:59:58.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.965+0000 7fb4150c7640 1 -- start start 2024-02-19T14:59:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.966+0000 7fb4150c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 0x7fb41006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.966+0000 7fb4150c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 0x7fb41006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.966+0000 7fb4150c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41006cd20 con 0x7fb410152700 2024-02-19T14:59:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.966+0000 7fb4150c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4101551f0 con 0x7fb4100c8e00 2024-02-19T14:59:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.966+0000 7fb4150c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb410155490 con 0x7fb41006d910 2024-02-19T14:59:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb407fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 0x7fb41006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40f59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 0x7fb41006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40ed9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb410152700 0x7fb4100c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46500/0 (socket says 172.21.15.78:46500) 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb407fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 0x7fb41006c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:49662/0 (socket says 172.21.15.78:49662) 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40ed9d640 1 -- 172.21.15.78:0/3551768337 learned_addr learned my addr 172.21.15.78:0/3551768337 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40f59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 0x7fb41006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:34206/0 (socket says 172.21.15.78:34206) 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40f59e640 1 -- 172.21.15.78:0/3551768337 >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 msgr2=0x7fb41006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40f59e640 1 --2- 172.21.15.78:0/3551768337 >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 0x7fb41006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40f59e640 1 -- 172.21.15.78:0/3551768337 >> v1:172.21.15.150:6789/0 conn(0x7fb410152700 legacy=0x7fb4100c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb40f59e640 1 -- 172.21.15.78:0/3551768337 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb410155770 con 0x7fb41006d910 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.967+0000 7fb407fff640 1 --2- 172.21.15.78:0/3551768337 >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 0x7fb41006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T14:59:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.968+0000 7fb40f59e640 1 --2- 172.21.15.78:0/3551768337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 0x7fb41006c7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb408009d90 tx=0x7fb40802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df49d0abd121cb3f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.968+0000 7fb40e59c640 1 -- 172.21.15.78:0/3551768337 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40802cbc0 con 0x7fb41006d910 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.968+0000 7fb40e59c640 1 -- 172.21.15.78:0/3551768337 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40802cd20 con 0x7fb41006d910 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.968+0000 7fb40e59c640 1 -- 172.21.15.78:0/3551768337 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408034660 con 0x7fb41006d910 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 -- 172.21.15.78:0/3551768337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 msgr2=0x7fb41006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 --2- 172.21.15.78:0/3551768337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 0x7fb41006c7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb408009d90 tx=0x7fb40802be10 comp rx=0 tx=0).stop 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 -- 172.21.15.78:0/3551768337 shutdown_connections 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 --2- 172.21.15.78:0/3551768337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb41006d910 0x7fb41006c7e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 --2- 172.21.15.78:0/3551768337 >> v2:172.21.15.150:3300/0 conn(0x7fb4100c8e00 0x7fb41006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 -- 172.21.15.78:0/3551768337 >> 172.21.15.78:0/3551768337 conn(0x7fb4100cac90 msgr2=0x7fb4100cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 -- 172.21.15.78:0/3551768337 shutdown_connections 2024-02-19T14:59:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.969+0000 7fb4150c7640 1 -- 172.21.15.78:0/3551768337 wait complete. 2024-02-19T14:59:58.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.971+0000 7fb4150c7640 1 Processor -- start 2024-02-19T14:59:58.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.971+0000 7fb4150c7640 1 -- start start 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb4150c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb41006d910 0x7fb4101eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb4150c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 0x7fb4100ab830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb4150c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb410152700 0x7fb41014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb4150c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4101eb2f0 con 0x7fb4100c8e00 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb4150c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4101590d0 con 0x7fb41006d910 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb4150c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4101593b0 con 0x7fb410152700 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb407fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 0x7fb4100ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb407fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 0x7fb4100ab830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36036/0 (socket says 172.21.15.78:36036) 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb407fff640 1 -- 172.21.15.78:0/1280243384 learned_addr learned my addr 172.21.15.78:0/1280243384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb40ed9d640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb41006d910 0x7fb4101eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.972+0000 7fb40f59e640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb410152700 0x7fb41014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb407fff640 1 -- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb410152700 msgr2=0x7fb41014cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb407fff640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb410152700 0x7fb41014cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb407fff640 1 -- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb41006d910 msgr2=0x7fb4101eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb407fff640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb41006d910 0x7fb4101eadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb407fff640 1 -- 172.21.15.78:0/1280243384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8009400 con 0x7fb4100c8e00 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb40ed9d640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb41006d910 0x7fb4101eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb407fff640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 0x7fb4100ab830 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800ebe0 tx=0x7fb3f800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8009760 con 0x7fb4100c8e00 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f800cc30 con 0x7fb4100c8e00 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb408009a70 con 0x7fb4100c8e00 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f800eee0 con 0x7fb4100c8e00 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.973+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb410158410 con 0x7fb4100c8e00 2024-02-19T14:59:58.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.974+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb410152ae0 con 0x7fb4100c8e00 2024-02-19T14:59:58.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.975+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb3f8013470 con 0x7fb4100c8e00 2024-02-19T14:59:58.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.975+0000 7fb4077fe640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3f006b810 0x7fb3f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:58.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.975+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb3f808a5a0 con 0x7fb4100c8e00 2024-02-19T14:59:58.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.975+0000 7fb40ed9d640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3f006b810 0x7fb3f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:58.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.976+0000 7fb40ed9d640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3f006b810 0x7fb3f006dcd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb400000f60 tx=0x7fb400009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:58.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:58.984+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb3f8054f70 con 0x7fb4100c8e00 2024-02-19T14:59:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.158+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb41006ded0 con 0x7fb3f006b810 2024-02-19T14:59:59.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.165+0000 7fb4077fe640 1 -- 172.21.15.78:0/1280243384 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289989 (secure 0 0 0) 0x7fb400001370 con 0x7fb3f006b810 2024-02-19T14:59:59.171 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3f006b810 msgr2=0x7fb3f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3f006b810 0x7fb3f006dcd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb400000f60 tx=0x7fb400009040 comp rx=0 tx=0).stop 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 msgr2=0x7fb4100ab830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 0x7fb4100ab830 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800ebe0 tx=0x7fb3f800c520 comp rx=0 tx=0).stop 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 shutdown_connections 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3f006b810 0x7fb3f006dcd0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4100c8e00 0x7fb4100ab830 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb410152700 0x7fb41014cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 --2- 172.21.15.78:0/1280243384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb41006d910 0x7fb4101eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.168+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 >> 172.21.15.78:0/1280243384 conn(0x7fb4100cac90 msgr2=0x7fb4100cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.169+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 shutdown_connections 2024-02-19T14:59:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.169+0000 7fb4150c7640 1 -- 172.21.15.78:0/1280243384 wait complete. 2024-02-19T14:59:59.182 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-02-19T14:59:58.153585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270700,"kb_used_data":21224,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754704020,"statfs":{"total":773094113280,"available":772816916480,"internally_reserved":0,"allocated":21733376,"data_stored":11203604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.329357"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950725+0000","last_change":"2024-02-19T14:59:55.860557+0000","last_active":"2024-02-19T14:59:56.950725+0000","last_peered":"2024-02-19T14:59:56.950725+0000","last_clean":"2024-02-19T14:59:56.950725+0000","last_became_active":"2024-02-19T14:59:55.860294+0000","last_became_peered":"2024-02-19T14:59:55.860294+0000","last_unstale":"2024-02-19T14:59:56.950725+0000","last_undegraded":"2024-02-19T14:59:56.950725+0000","last_fullsized":"2024-02-19T14:59:56.950725+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:17:48.596442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951199+0000","last_change":"2024-02-19T14:59:55.859202+0000","last_active":"2024-02-19T14:59:56.951199+0000","last_peered":"2024-02-19T14:59:56.951199+0000","last_clean":"2024-02-19T14:59:56.951199+0000","last_became_active":"2024-02-19T14:59:55.858082+0000","last_became_peered":"2024-02-19T14:59:55.858082+0000","last_unstale":"2024-02-19T14:59:56.951199+0000","last_undegraded":"2024-02-19T14:59:56.951199+0000","last_fullsized":"2024-02-19T14:59:56.951199+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:11:18.254704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833643+0000","last_change":"2024-02-19T14:59:55.854227+0000","last_active":"2024-02-19T14:59:56.833643+0000","last_peered":"2024-02-19T14:59:56.833643+0000","last_clean":"2024-02-19T14:59:56.833643+0000","last_became_active":"2024-02-19T14:59:55.853972+0000","last_became_peered":"2024-02-19T14:59:55.853972+0000","last_unstale":"2024-02-19T14:59:56.833643+0000","last_undegraded":"2024-02-19T14:59:56.833643+0000","last_fullsized":"2024-02-19T14:59:56.833643+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:27:32.442528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833623+0000","last_change":"2024-02-19T14:59:55.861264+0000","last_active":"2024-02-19T14:59:56.833623+0000","last_peered":"2024-02-19T14:59:56.833623+0000","last_clean":"2024-02-19T14:59:56.833623+0000","last_became_active":"2024-02-19T14:59:55.860144+0000","last_became_peered":"2024-02-19T14:59:55.860144+0000","last_unstale":"2024-02-19T14:59:56.833623+0000","last_undegraded":"2024-02-19T14:59:56.833623+0000","last_fullsized":"2024-02-19T14:59:56.833623+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:21.621871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950990+0000","last_change":"2024-02-19T14:59:55.850506+0000","last_active":"2024-02-19T14:59:56.950990+0000","last_peered":"2024-02-19T14:59:56.950990+0000","last_clean":"2024-02-19T14:59:56.950990+0000","last_became_active":"2024-02-19T14:59:55.849976+0000","last_became_peered":"2024-02-19T14:59:55.849976+0000","last_unstale":"2024-02-19T14:59:56.950990+0000","last_undegraded":"2024-02-19T14:59:56.950990+0000","last_fullsized":"2024-02-19T14:59:56.950990+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:24.951807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950374+0000","last_change":"2024-02-19T14:59:55.847289+0000","last_active":"2024-02-19T14:59:56.950374+0000","last_peered":"2024-02-19T14:59:56.950374+0000","last_clean":"2024-02-19T14:59:56.950374+0000","last_became_active":"2024-02-19T14:59:55.847014+0000","last_became_peered":"2024-02-19T14:59:55.847014+0000","last_unstale":"2024-02-19T14:59:56.950374+0000","last_undegraded":"2024-02-19T14:59:56.950374+0000","last_fullsized":"2024-02-19T14:59:56.950374+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:36:14.218443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835174+0000","last_change":"2024-02-19T14:59:55.856571+0000","last_active":"2024-02-19T14:59:56.835174+0000","last_peered":"2024-02-19T14:59:56.835174+0000","last_clean":"2024-02-19T14:59:56.835174+0000","last_became_active":"2024-02-19T14:59:55.856243+0000","last_became_peered":"2024-02-19T14:59:55.856243+0000","last_unstale":"2024-02-19T14:59:56.835174+0000","last_undegraded":"2024-02-19T14:59:56.835174+0000","last_fullsized":"2024-02-19T14:59:56.835174+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:25.963752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950173+0000","last_change":"2024-02-19T14:59:55.853857+0000","last_active":"2024-02-19T14:59:56.950173+0000","last_peered":"2024-02-19T14:59:56.950173+0000","last_clean":"2024-02-19T14:59:56.950173+0000","last_became_active":"2024-02-19T14:59:55.853599+0000","last_became_peered":"2024-02-19T14:59:55.853599+0000","last_unstale":"2024-02-19T14:59:56.950173+0000","last_undegraded":"2024-02-19T14:59:56.950173+0000","last_fullsized":"2024-02-19T14:59:56.950173+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:30.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950864+0000","last_change":"2024-02-19T14:59:55.846715+0000","last_active":"2024-02-19T14:59:56.950864+0000","last_peered":"2024-02-19T14:59:56.950864+0000","last_clean":"2024-02-19T14:59:56.950864+0000","last_became_active":"2024-02-19T14:59:55.846452+0000","last_became_peered":"2024-02-19T14:59:55.846452+0000","last_unstale":"2024-02-19T14:59:56.950864+0000","last_undegraded":"2024-02-19T14:59:56.950864+0000","last_fullsized":"2024-02-19T14:59:56.950864+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:03:54.892526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833751+0000","last_change":"2024-02-19T14:59:55.856472+0000","last_active":"2024-02-19T14:59:56.833751+0000","last_peered":"2024-02-19T14:59:56.833751+0000","last_clean":"2024-02-19T14:59:56.833751+0000","last_became_active":"2024-02-19T14:59:55.856191+0000","last_became_peered":"2024-02-19T14:59:55.856191+0000","last_unstale":"2024-02-19T14:59:56.833751+0000","last_undegraded":"2024-02-19T14:59:56.833751+0000","last_fullsized":"2024-02-19T14:59:56.833751+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:18:39.380631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833714+0000","last_change":"2024-02-19T14:59:55.856271+0000","last_active":"2024-02-19T14:59:56.833714+0000","last_peered":"2024-02-19T14:59:56.833714+0000","last_clean":"2024-02-19T14:59:56.833714+0000","last_became_active":"2024-02-19T14:59:55.855990+0000","last_became_peered":"2024-02-19T14:59:55.855990+0000","last_unstale":"2024-02-19T14:59:56.833714+0000","last_undegraded":"2024-02-19T14:59:56.833714+0000","last_fullsized":"2024-02-19T14:59:56.833714+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:08:35.018838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950259+0000","last_change":"2024-02-19T14:59:55.838067+0000","last_active":"2024-02-19T14:59:56.950259+0000","last_peered":"2024-02-19T14:59:56.950259+0000","last_clean":"2024-02-19T14:59:56.950259+0000","last_became_active":"2024-02-19T14:59:55.837761+0000","last_became_peered":"2024-02-19T14:59:55.837761+0000","last_unstale":"2024-02-19T14:59:56.950259+0000","last_undegraded":"2024-02-19T14:59:56.950259+0000","last_fullsized":"2024-02-19T14:59:56.950259+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:09:15.435031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835811+0000","last_change":"2024-02-19T14:59:55.860660+0000","last_active":"2024-02-19T14:59:56.835811+0000","last_peered":"2024-02-19T14:59:56.835811+0000","last_clean":"2024-02-19T14:59:56.835811+0000","last_became_active":"2024-02-19T14:59:55.860371+0000","last_became_peered":"2024-02-19T14:59:55.860371+0000","last_unstale":"2024-02-19T14:59:56.835811+0000","last_undegraded":"2024-02-19T14:59:56.835811+0000","last_fullsized":"2024-02-19T14:59:56.835811+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:41:18.490485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949763+0000","last_change":"2024-02-19T14:59:55.849652+0000","last_active":"2024-02-19T14:59:56.949763+0000","last_peered":"2024-02-19T14:59:56.949763+0000","last_clean":"2024-02-19T14:59:56.949763+0000","last_became_active":"2024-02-19T14:59:55.848746+0000","last_became_peered":"2024-02-19T14:59:55.848746+0000","last_unstale":"2024-02-19T14:59:56.949763+0000","last_undegraded":"2024-02-19T14:59:56.949763+0000","last_fullsized":"2024-02-19T14:59:56.949763+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:27.807111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950461+0000","last_change":"2024-02-19T14:59:55.855006+0000","last_active":"2024-02-19T14:59:56.950461+0000","last_peered":"2024-02-19T14:59:56.950461+0000","last_clean":"2024-02-19T14:59:56.950461+0000","last_became_active":"2024-02-19T14:59:55.854728+0000","last_became_peered":"2024-02-19T14:59:55.854728+0000","last_unstale":"2024-02-19T14:59:56.950461+0000","last_undegraded":"2024-02-19T14:59:56.950461+0000","last_fullsized":"2024-02-19T14:59:56.950461+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:51.831198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833361+0000","last_change":"2024-02-19T14:59:55.860301+0000","last_active":"2024-02-19T14:59:56.833361+0000","last_peered":"2024-02-19T14:59:56.833361+0000","last_clean":"2024-02-19T14:59:56.833361+0000","last_became_active":"2024-02-19T14:59:55.858671+0000","last_became_peered":"2024-02-19T14:59:55.858671+0000","last_unstale":"2024-02-19T14:59:56.833361+0000","last_undegraded":"2024-02-19T14:59:56.833361+0000","last_fullsized":"2024-02-19T14:59:56.833361+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:51.831198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835816+0000","last_change":"2024-02-19T14:59:55.852800+0000","last_active":"2024-02-19T14:59:56.835816+0000","last_peered":"2024-02-19T14:59:56.835816+0000","last_clean":"2024-02-19T14:59:56.835816+0000","last_became_active":"2024-02-19T14:59:55.849421+0000","last_became_peered":"2024-02-19T14:59:55.849421+0000","last_unstale":"2024-02-19T14:59:56.835816+0000","last_undegraded":"2024-02-19T14:59:56.835816+0000","last_fullsized":"2024-02-19T14:59:56.835816+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:25:08.204619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833758+0000","last_change":"2024-02-19T14:59:55.855281+0000","last_active":"2024-02-19T14:59:56.833758+0000","last_peered":"2024-02-19T14:59:56.833758+0000","last_clean":"2024-02-19T14:59:56.833758+0000","last_became_active":"2024-02-19T14:59:55.855011+0000","last_became_peered":"2024-02-19T14:59:55.855011+0000","last_unstale":"2024-02-19T14:59:56.833758+0000","last_undegraded":"2024-02-19T14:59:56.833758+0000","last_fullsized":"2024-02-19T14:59:56.833758+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:09.338353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950500+0000","last_change":"2024-02-19T14:59:55.852244+0000","last_active":"2024-02-19T14:59:56.950500+0000","last_peered":"2024-02-19T14:59:56.950500+0000","last_clean":"2024-02-19T14:59:56.950500+0000","last_became_active":"2024-02-19T14:59:55.851901+0000","last_became_peered":"2024-02-19T14:59:55.851901+0000","last_unstale":"2024-02-19T14:59:56.950500+0000","last_undegraded":"2024-02-19T14:59:56.950500+0000","last_fullsized":"2024-02-19T14:59:56.950500+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:29:55.479033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951326+0000","last_change":"2024-02-19T14:59:55.858207+0000","last_active":"2024-02-19T14:59:56.951326+0000","last_peered":"2024-02-19T14:59:56.951326+0000","last_clean":"2024-02-19T14:59:56.951326+0000","last_became_active":"2024-02-19T14:59:55.856282+0000","last_became_peered":"2024-02-19T14:59:55.856282+0000","last_unstale":"2024-02-19T14:59:56.951326+0000","last_undegraded":"2024-02-19T14:59:56.951326+0000","last_fullsized":"2024-02-19T14:59:56.951326+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:29:03.038209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835285+0000","last_change":"2024-02-19T14:59:55.846714+0000","last_active":"2024-02-19T14:59:56.835285+0000","last_peered":"2024-02-19T14:59:56.835285+0000","last_clean":"2024-02-19T14:59:56.835285+0000","last_became_active":"2024-02-19T14:59:55.846422+0000","last_became_peered":"2024-02-19T14:59:55.846422+0000","last_unstale":"2024-02-19T14:59:56.835285+0000","last_undegraded":"2024-02-19T14:59:56.835285+0000","last_fullsized":"2024-02-19T14:59:56.835285+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:52.220536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835051+0000","last_change":"2024-02-19T14:59:55.841606+0000","last_active":"2024-02-19T14:59:56.835051+0000","last_peered":"2024-02-19T14:59:56.835051+0000","last_clean":"2024-02-19T14:59:56.835051+0000","last_became_active":"2024-02-19T14:59:55.840425+0000","last_became_peered":"2024-02-19T14:59:55.840425+0000","last_unstale":"2024-02-19T14:59:56.835051+0000","last_undegraded":"2024-02-19T14:59:56.835051+0000","last_fullsized":"2024-02-19T14:59:56.835051+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:06:10.676092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950490+0000","last_change":"2024-02-19T14:59:55.852748+0000","last_active":"2024-02-19T14:59:56.950490+0000","last_peered":"2024-02-19T14:59:56.950490+0000","last_clean":"2024-02-19T14:59:56.950490+0000","last_became_active":"2024-02-19T14:59:55.852296+0000","last_became_peered":"2024-02-19T14:59:55.852296+0000","last_unstale":"2024-02-19T14:59:56.950490+0000","last_undegraded":"2024-02-19T14:59:56.950490+0000","last_fullsized":"2024-02-19T14:59:56.950490+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:30.266982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.834787+0000","last_change":"2024-02-19T14:59:55.838192+0000","last_active":"2024-02-19T14:59:56.834787+0000","last_peered":"2024-02-19T14:59:56.834787+0000","last_clean":"2024-02-19T14:59:56.834787+0000","last_became_active":"2024-02-19T14:59:55.837866+0000","last_became_peered":"2024-02-19T14:59:55.837866+0000","last_unstale":"2024-02-19T14:59:56.834787+0000","last_undegraded":"2024-02-19T14:59:56.834787+0000","last_fullsized":"2024-02-19T14:59:56.834787+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:07:53.615527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951491+0000","last_change":"2024-02-19T14:59:55.858422+0000","last_active":"2024-02-19T14:59:56.951491+0000","last_peered":"2024-02-19T14:59:56.951491+0000","last_clean":"2024-02-19T14:59:56.951491+0000","last_became_active":"2024-02-19T14:59:55.855797+0000","last_became_peered":"2024-02-19T14:59:55.855797+0000","last_unstale":"2024-02-19T14:59:56.951491+0000","last_undegraded":"2024-02-19T14:59:56.951491+0000","last_fullsized":"2024-02-19T14:59:56.951491+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:42:18.532570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835105+0000","last_change":"2024-02-19T14:59:55.857690+0000","last_active":"2024-02-19T14:59:56.835105+0000","last_peered":"2024-02-19T14:59:56.835105+0000","last_clean":"2024-02-19T14:59:56.835105+0000","last_became_active":"2024-02-19T14:59:55.857423+0000","last_became_peered":"2024-02-19T14:59:55.857423+0000","last_unstale":"2024-02-19T14:59:56.835105+0000","last_undegraded":"2024-02-19T14:59:56.835105+0000","last_fullsized":"2024-02-19T14:59:56.835105+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:50.830780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950735+0000","last_change":"2024-02-19T14:59:55.845581+0000","last_active":"2024-02-19T14:59:56.950735+0000","last_peered":"2024-02-19T14:59:56.950735+0000","last_clean":"2024-02-19T14:59:56.950735+0000","last_became_active":"2024-02-19T14:59:55.845247+0000","last_became_peered":"2024-02-19T14:59:55.845247+0000","last_unstale":"2024-02-19T14:59:56.950735+0000","last_undegraded":"2024-02-19T14:59:56.950735+0000","last_fullsized":"2024-02-19T14:59:56.950735+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:52.220536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950400+0000","last_change":"2024-02-19T14:59:55.847106+0000","last_active":"2024-02-19T14:59:56.950400+0000","last_peered":"2024-02-19T14:59:56.950400+0000","last_clean":"2024-02-19T14:59:56.950400+0000","last_became_active":"2024-02-19T14:59:55.846827+0000","last_became_peered":"2024-02-19T14:59:55.846827+0000","last_unstale":"2024-02-19T14:59:56.950400+0000","last_undegraded":"2024-02-19T14:59:56.950400+0000","last_fullsized":"2024-02-19T14:59:56.950400+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:01:52.501855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950123+0000","last_change":"2024-02-19T14:59:55.848307+0000","last_active":"2024-02-19T14:59:56.950123+0000","last_peered":"2024-02-19T14:59:56.950123+0000","last_clean":"2024-02-19T14:59:56.950123+0000","last_became_active":"2024-02-19T14:59:55.848034+0000","last_became_peered":"2024-02-19T14:59:55.848034+0000","last_unstale":"2024-02-19T14:59:56.950123+0000","last_undegraded":"2024-02-19T14:59:56.950123+0000","last_fullsized":"2024-02-19T14:59:56.950123+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:33.068323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949890+0000","last_change":"2024-02-19T14:59:55.847389+0000","last_active":"2024-02-19T14:59:56.949890+0000","last_peered":"2024-02-19T14:59:56.949890+0000","last_clean":"2024-02-19T14:59:56.949890+0000","last_became_active":"2024-02-19T14:59:55.846627+0000","last_became_peered":"2024-02-19T14:59:55.846627+0000","last_unstale":"2024-02-19T14:59:56.949890+0000","last_undegraded":"2024-02-19T14:59:56.949890+0000","last_fullsized":"2024-02-19T14:59:56.949890+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:06.037741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950746+0000","last_change":"2024-02-19T14:59:55.852061+0000","last_active":"2024-02-19T14:59:56.950746+0000","last_peered":"2024-02-19T14:59:56.950746+0000","last_clean":"2024-02-19T14:59:56.950746+0000","last_became_active":"2024-02-19T14:59:55.851791+0000","last_became_peered":"2024-02-19T14:59:55.851791+0000","last_unstale":"2024-02-19T14:59:56.950746+0000","last_undegraded":"2024-02-19T14:59:56.950746+0000","last_fullsized":"2024-02-19T14:59:56.950746+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:45:27.597797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949876+0000","last_change":"2024-02-19T14:59:55.847738+0000","last_active":"2024-02-19T14:59:56.949876+0000","last_peered":"2024-02-19T14:59:56.949876+0000","last_clean":"2024-02-19T14:59:56.949876+0000","last_became_active":"2024-02-19T14:59:55.847438+0000","last_became_peered":"2024-02-19T14:59:55.847438+0000","last_unstale":"2024-02-19T14:59:56.949876+0000","last_undegraded":"2024-02-19T14:59:56.949876+0000","last_fullsized":"2024-02-19T14:59:56.949876+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:12:17.209037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950231+0000","last_change":"2024-02-19T14:59:55.858050+0000","last_active":"2024-02-19T14:59:56.950231+0000","last_peered":"2024-02-19T14:59:56.950231+0000","last_clean":"2024-02-19T14:59:56.950231+0000","last_became_active":"2024-02-19T14:59:55.857754+0000","last_became_peered":"2024-02-19T14:59:55.857754+0000","last_unstale":"2024-02-19T14:59:56.950231+0000","last_undegraded":"2024-02-19T14:59:56.950231+0000","last_fullsized":"2024-02-19T14:59:56.950231+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:01:17.100730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"201'4","reported_seq":18,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950746+0000","last_change":"2024-02-19T14:59:54.850831+0000","last_active":"2024-02-19T14:59:56.950746+0000","last_peered":"2024-02-19T14:59:56.950746+0000","last_clean":"2024-02-19T14:59:56.950746+0000","last_became_active":"2024-02-19T14:59:54.850434+0000","last_became_peered":"2024-02-19T14:59:54.850434+0000","last_unstale":"2024-02-19T14:59:56.950746+0000","last_undegraded":"2024-02-19T14:59:56.950746+0000","last_fullsized":"2024-02-19T14:59:56.950746+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:58:53.548615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950117+0000","last_change":"2024-02-19T14:59:55.847052+0000","last_active":"2024-02-19T14:59:56.950117+0000","last_peered":"2024-02-19T14:59:56.950117+0000","last_clean":"2024-02-19T14:59:56.950117+0000","last_became_active":"2024-02-19T14:59:55.846788+0000","last_became_peered":"2024-02-19T14:59:55.846788+0000","last_unstale":"2024-02-19T14:59:56.950117+0000","last_undegraded":"2024-02-19T14:59:56.950117+0000","last_fullsized":"2024-02-19T14:59:56.950117+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:06.037741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.834901+0000","last_change":"2024-02-19T14:59:54.841297+0000","last_active":"2024-02-19T14:59:56.834901+0000","last_peered":"2024-02-19T14:59:56.834901+0000","last_clean":"2024-02-19T14:59:56.834901+0000","last_became_active":"2024-02-19T14:59:54.840553+0000","last_became_peered":"2024-02-19T14:59:54.840553+0000","last_unstale":"2024-02-19T14:59:56.834901+0000","last_undegraded":"2024-02-19T14:59:56.834901+0000","last_fullsized":"2024-02-19T14:59:56.834901+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:37.496659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950779+0000","last_change":"2024-02-19T14:59:55.846747+0000","last_active":"2024-02-19T14:59:56.950779+0000","last_peered":"2024-02-19T14:59:56.950779+0000","last_clean":"2024-02-19T14:59:56.950779+0000","last_became_active":"2024-02-19T14:59:55.846473+0000","last_became_peered":"2024-02-19T14:59:55.846473+0000","last_unstale":"2024-02-19T14:59:56.950779+0000","last_undegraded":"2024-02-19T14:59:56.950779+0000","last_fullsized":"2024-02-19T14:59:56.950779+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:52:29.692660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950408+0000","last_change":"2024-02-19T14:59:54.849731+0000","last_active":"2024-02-19T14:59:56.950408+0000","last_peered":"2024-02-19T14:59:56.950408+0000","last_clean":"2024-02-19T14:59:56.950408+0000","last_became_active":"2024-02-19T14:59:54.849492+0000","last_became_peered":"2024-02-19T14:59:54.849492+0000","last_unstale":"2024-02-19T14:59:56.950408+0000","last_undegraded":"2024-02-19T14:59:56.950408+0000","last_fullsized":"2024-02-19T14:59:56.950408+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:14.881873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950338+0000","last_change":"2024-02-19T14:59:55.858699+0000","last_active":"2024-02-19T14:59:56.950338+0000","last_peered":"2024-02-19T14:59:56.950338+0000","last_clean":"2024-02-19T14:59:56.950338+0000","last_became_active":"2024-02-19T14:59:55.858414+0000","last_became_peered":"2024-02-19T14:59:55.858414+0000","last_unstale":"2024-02-19T14:59:56.950338+0000","last_undegraded":"2024-02-19T14:59:56.950338+0000","last_fullsized":"2024-02-19T14:59:56.950338+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:23:51.831198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950786+0000","last_change":"2024-02-19T14:59:54.846189+0000","last_active":"2024-02-19T14:59:56.950786+0000","last_peered":"2024-02-19T14:59:56.950786+0000","last_clean":"2024-02-19T14:59:56.950786+0000","last_became_active":"2024-02-19T14:59:54.845872+0000","last_became_peered":"2024-02-19T14:59:54.845872+0000","last_unstale":"2024-02-19T14:59:56.950786+0000","last_undegraded":"2024-02-19T14:59:56.950786+0000","last_fullsized":"2024-02-19T14:59:56.950786+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:09.956992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835566+0000","last_change":"2024-02-19T14:59:55.845947+0000","last_active":"2024-02-19T14:59:56.835566+0000","last_peered":"2024-02-19T14:59:56.835566+0000","last_clean":"2024-02-19T14:59:56.835566+0000","last_became_active":"2024-02-19T14:59:55.845670+0000","last_became_peered":"2024-02-19T14:59:55.845670+0000","last_unstale":"2024-02-19T14:59:56.835566+0000","last_undegraded":"2024-02-19T14:59:56.835566+0000","last_fullsized":"2024-02-19T14:59:56.835566+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:15.889208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949986+0000","last_change":"2024-02-19T14:59:54.844743+0000","last_active":"2024-02-19T14:59:56.949986+0000","last_peered":"2024-02-19T14:59:56.949986+0000","last_clean":"2024-02-19T14:59:56.949986+0000","last_became_active":"2024-02-19T14:59:54.844479+0000","last_became_peered":"2024-02-19T14:59:54.844479+0000","last_unstale":"2024-02-19T14:59:56.949986+0000","last_undegraded":"2024-02-19T14:59:56.949986+0000","last_fullsized":"2024-02-19T14:59:56.949986+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:37.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833586+0000","last_change":"2024-02-19T14:59:55.857968+0000","last_active":"2024-02-19T14:59:56.833586+0000","last_peered":"2024-02-19T14:59:56.833586+0000","last_clean":"2024-02-19T14:59:56.833586+0000","last_became_active":"2024-02-19T14:59:55.857663+0000","last_became_peered":"2024-02-19T14:59:55.857663+0000","last_unstale":"2024-02-19T14:59:56.833586+0000","last_undegraded":"2024-02-19T14:59:56.833586+0000","last_fullsized":"2024-02-19T14:59:56.833586+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:24:33.622046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833308+0000","last_change":"2024-02-19T14:59:54.851699+0000","last_active":"2024-02-19T14:59:56.833308+0000","last_peered":"2024-02-19T14:59:56.833308+0000","last_clean":"2024-02-19T14:59:56.833308+0000","last_became_active":"2024-02-19T14:59:54.851328+0000","last_became_peered":"2024-02-19T14:59:54.851328+0000","last_unstale":"2024-02-19T14:59:56.833308+0000","last_undegraded":"2024-02-19T14:59:56.833308+0000","last_fullsized":"2024-02-19T14:59:56.833308+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:23.944471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951094+0000","last_change":"2024-02-19T14:59:55.850552+0000","last_active":"2024-02-19T14:59:56.951094+0000","last_peered":"2024-02-19T14:59:56.951094+0000","last_clean":"2024-02-19T14:59:56.951094+0000","last_became_active":"2024-02-19T14:59:55.850232+0000","last_became_peered":"2024-02-19T14:59:55.850232+0000","last_unstale":"2024-02-19T14:59:56.951094+0000","last_undegraded":"2024-02-19T14:59:56.951094+0000","last_fullsized":"2024-02-19T14:59:56.951094+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:10.964328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949860+0000","last_change":"2024-02-19T14:59:54.851004+0000","last_active":"2024-02-19T14:59:56.949860+0000","last_peered":"2024-02-19T14:59:56.949860+0000","last_clean":"2024-02-19T14:59:56.949860+0000","last_became_active":"2024-02-19T14:59:54.850773+0000","last_became_peered":"2024-02-19T14:59:54.850773+0000","last_unstale":"2024-02-19T14:59:56.949860+0000","last_undegraded":"2024-02-19T14:59:56.949860+0000","last_fullsized":"2024-02-19T14:59:56.949860+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:35:20.441122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.834969+0000","last_change":"2024-02-19T14:59:55.844315+0000","last_active":"2024-02-19T14:59:56.834969+0000","last_peered":"2024-02-19T14:59:56.834969+0000","last_clean":"2024-02-19T14:59:56.834969+0000","last_became_active":"2024-02-19T14:59:55.843969+0000","last_became_peered":"2024-02-19T14:59:55.843969+0000","last_unstale":"2024-02-19T14:59:56.834969+0000","last_undegraded":"2024-02-19T14:59:56.834969+0000","last_fullsized":"2024-02-19T14:59:56.834969+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:40:10.109452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949923+0000","last_change":"2024-02-19T14:59:54.845253+0000","last_active":"2024-02-19T14:59:56.949923+0000","last_peered":"2024-02-19T14:59:56.949923+0000","last_clean":"2024-02-19T14:59:56.949923+0000","last_became_active":"2024-02-19T14:59:54.844951+0000","last_became_peered":"2024-02-19T14:59:54.844951+0000","last_unstale":"2024-02-19T14:59:56.949923+0000","last_undegraded":"2024-02-19T14:59:56.949923+0000","last_fullsized":"2024-02-19T14:59:56.949923+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:23.944471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835568+0000","last_change":"2024-02-19T14:59:55.841047+0000","last_active":"2024-02-19T14:59:56.835568+0000","last_peered":"2024-02-19T14:59:56.835568+0000","last_clean":"2024-02-19T14:59:56.835568+0000","last_became_active":"2024-02-19T14:59:55.840715+0000","last_became_peered":"2024-02-19T14:59:55.840715+0000","last_unstale":"2024-02-19T14:59:56.835568+0000","last_undegraded":"2024-02-19T14:59:56.835568+0000","last_fullsized":"2024-02-19T14:59:56.835568+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:31:28.660287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833804+0000","last_change":"2024-02-19T14:59:54.846606+0000","last_active":"2024-02-19T14:59:56.833804+0000","last_peered":"2024-02-19T14:59:56.833804+0000","last_clean":"2024-02-19T14:59:56.833804+0000","last_became_active":"2024-02-19T14:59:54.844365+0000","last_became_peered":"2024-02-19T14:59:54.844365+0000","last_unstale":"2024-02-19T14:59:56.833804+0000","last_undegraded":"2024-02-19T14:59:56.833804+0000","last_fullsized":"2024-02-19T14:59:56.833804+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:03.342512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833395+0000","last_change":"2024-02-19T14:59:55.855006+0000","last_active":"2024-02-19T14:59:56.833395+0000","last_peered":"2024-02-19T14:59:56.833395+0000","last_clean":"2024-02-19T14:59:56.833395+0000","last_became_active":"2024-02-19T14:59:55.854717+0000","last_became_peered":"2024-02-19T14:59:55.854717+0000","last_unstale":"2024-02-19T14:59:56.833395+0000","last_undegraded":"2024-02-19T14:59:56.833395+0000","last_fullsized":"2024-02-19T14:59:56.833395+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:29:55.479033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"201'3","reported_seq":17,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950950+0000","last_change":"2024-02-19T14:59:54.846344+0000","last_active":"2024-02-19T14:59:56.950950+0000","last_peered":"2024-02-19T14:59:56.950950+0000","last_clean":"2024-02-19T14:59:56.950950+0000","last_became_active":"2024-02-19T14:59:54.846035+0000","last_became_peered":"2024-02-19T14:59:54.846035+0000","last_unstale":"2024-02-19T14:59:56.950950+0000","last_undegraded":"2024-02-19T14:59:56.950950+0000","last_fullsized":"2024-02-19T14:59:56.950950+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:45:39.339962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835134+0000","last_change":"2024-02-19T14:59:55.841234+0000","last_active":"2024-02-19T14:59:56.835134+0000","last_peered":"2024-02-19T14:59:56.835134+0000","last_clean":"2024-02-19T14:59:56.835134+0000","last_became_active":"2024-02-19T14:59:55.840935+0000","last_became_peered":"2024-02-19T14:59:55.840935+0000","last_unstale":"2024-02-19T14:59:56.835134+0000","last_undegraded":"2024-02-19T14:59:56.835134+0000","last_fullsized":"2024-02-19T14:59:56.835134+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:22:30.511373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"201'6","reported_seq":20,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950506+0000","last_change":"2024-02-19T14:59:54.847336+0000","last_active":"2024-02-19T14:59:56.950506+0000","last_peered":"2024-02-19T14:59:56.950506+0000","last_clean":"2024-02-19T14:59:56.950506+0000","last_became_active":"2024-02-19T14:59:54.847089+0000","last_became_peered":"2024-02-19T14:59:54.847089+0000","last_unstale":"2024-02-19T14:59:56.950506+0000","last_undegraded":"2024-02-19T14:59:56.950506+0000","last_fullsized":"2024-02-19T14:59:56.950506+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:00:31.781028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951242+0000","last_change":"2024-02-19T14:59:55.857384+0000","last_active":"2024-02-19T14:59:56.951242+0000","last_peered":"2024-02-19T14:59:56.951242+0000","last_clean":"2024-02-19T14:59:56.951242+0000","last_became_active":"2024-02-19T14:59:55.857143+0000","last_became_peered":"2024-02-19T14:59:55.857143+0000","last_unstale":"2024-02-19T14:59:56.951242+0000","last_undegraded":"2024-02-19T14:59:56.951242+0000","last_fullsized":"2024-02-19T14:59:56.951242+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:45.766479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950793+0000","last_change":"2024-02-19T14:59:54.848997+0000","last_active":"2024-02-19T14:59:56.950793+0000","last_peered":"2024-02-19T14:59:56.950793+0000","last_clean":"2024-02-19T14:59:56.950793+0000","last_became_active":"2024-02-19T14:59:54.848723+0000","last_became_peered":"2024-02-19T14:59:54.848723+0000","last_unstale":"2024-02-19T14:59:56.950793+0000","last_undegraded":"2024-02-19T14:59:56.950793+0000","last_fullsized":"2024-02-19T14:59:56.950793+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:18:37.327278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835546+0000","last_change":"2024-02-19T14:59:55.850638+0000","last_active":"2024-02-19T14:59:56.835546+0000","last_peered":"2024-02-19T14:59:56.835546+0000","last_clean":"2024-02-19T14:59:56.835546+0000","last_became_active":"2024-02-19T14:59:55.850379+0000","last_became_peered":"2024-02-19T14:59:55.850379+0000","last_unstale":"2024-02-19T14:59:56.835546+0000","last_undegraded":"2024-02-19T14:59:56.835546+0000","last_fullsized":"2024-02-19T14:59:56.835546+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:53:24.951807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950430+0000","last_change":"2024-02-19T14:59:54.843069+0000","last_active":"2024-02-19T14:59:56.950430+0000","last_peered":"2024-02-19T14:59:56.950430+0000","last_clean":"2024-02-19T14:59:56.950430+0000","last_became_active":"2024-02-19T14:59:54.842801+0000","last_became_peered":"2024-02-19T14:59:54.842801+0000","last_unstale":"2024-02-19T14:59:56.950430+0000","last_undegraded":"2024-02-19T14:59:56.950430+0000","last_fullsized":"2024-02-19T14:59:56.950430+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:29:35.960893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950398+0000","last_change":"2024-02-19T14:59:55.854497+0000","last_active":"2024-02-19T14:59:56.950398+0000","last_peered":"2024-02-19T14:59:56.950398+0000","last_clean":"2024-02-19T14:59:56.950398+0000","last_became_active":"2024-02-19T14:59:55.854186+0000","last_became_peered":"2024-02-19T14:59:55.854186+0000","last_unstale":"2024-02-19T14:59:56.950398+0000","last_undegraded":"2024-02-19T14:59:56.950398+0000","last_fullsized":"2024-02-19T14:59:56.950398+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:44:09.953528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950716+0000","last_change":"2024-02-19T14:59:54.847939+0000","last_active":"2024-02-19T14:59:56.950716+0000","last_peered":"2024-02-19T14:59:56.950716+0000","last_clean":"2024-02-19T14:59:56.950716+0000","last_became_active":"2024-02-19T14:59:54.847477+0000","last_became_peered":"2024-02-19T14:59:54.847477+0000","last_unstale":"2024-02-19T14:59:56.950716+0000","last_undegraded":"2024-02-19T14:59:56.950716+0000","last_fullsized":"2024-02-19T14:59:56.950716+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:46:51.013136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951012+0000","last_change":"2024-02-19T14:59:55.863635+0000","last_active":"2024-02-19T14:59:56.951012+0000","last_peered":"2024-02-19T14:59:56.951012+0000","last_clean":"2024-02-19T14:59:56.951012+0000","last_became_active":"2024-02-19T14:59:55.863315+0000","last_became_peered":"2024-02-19T14:59:55.863315+0000","last_unstale":"2024-02-19T14:59:56.951012+0000","last_undegraded":"2024-02-19T14:59:56.951012+0000","last_fullsized":"2024-02-19T14:59:56.951012+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:15.889208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"201'8","reported_seq":22,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950279+0000","last_change":"2024-02-19T14:59:54.843398+0000","last_active":"2024-02-19T14:59:56.950279+0000","last_peered":"2024-02-19T14:59:56.950279+0000","last_clean":"2024-02-19T14:59:56.950279+0000","last_became_active":"2024-02-19T14:59:54.843107+0000","last_became_peered":"2024-02-19T14:59:54.843107+0000","last_unstale":"2024-02-19T14:59:56.950279+0000","last_undegraded":"2024-02-19T14:59:56.950279+0000","last_fullsized":"2024-02-19T14:59:56.950279+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:40:24.450767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950051+0000","last_change":"2024-02-19T14:59:55.863946+0000","last_active":"2024-02-19T14:59:56.950051+0000","last_peered":"2024-02-19T14:59:56.950051+0000","last_clean":"2024-02-19T14:59:56.950051+0000","last_became_active":"2024-02-19T14:59:55.863617+0000","last_became_peered":"2024-02-19T14:59:55.863617+0000","last_unstale":"2024-02-19T14:59:56.950051+0000","last_undegraded":"2024-02-19T14:59:56.950051+0000","last_fullsized":"2024-02-19T14:59:56.950051+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:00:50.830780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950838+0000","last_change":"2024-02-19T14:59:54.844905+0000","last_active":"2024-02-19T14:59:56.950838+0000","last_peered":"2024-02-19T14:59:56.950838+0000","last_clean":"2024-02-19T14:59:56.950838+0000","last_became_active":"2024-02-19T14:59:54.844645+0000","last_became_peered":"2024-02-19T14:59:54.844645+0000","last_unstale":"2024-02-19T14:59:56.950838+0000","last_undegraded":"2024-02-19T14:59:56.950838+0000","last_fullsized":"2024-02-19T14:59:56.950838+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:28:50.571117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950750+0000","last_change":"2024-02-19T14:59:55.851312+0000","last_active":"2024-02-19T14:59:56.950750+0000","last_peered":"2024-02-19T14:59:56.950750+0000","last_clean":"2024-02-19T14:59:56.950750+0000","last_became_active":"2024-02-19T14:59:55.851046+0000","last_became_peered":"2024-02-19T14:59:55.851046+0000","last_unstale":"2024-02-19T14:59:56.950750+0000","last_undegraded":"2024-02-19T14:59:56.950750+0000","last_fullsized":"2024-02-19T14:59:56.950750+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:04.349848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951436+0000","last_change":"2024-02-19T14:59:54.842198+0000","last_active":"2024-02-19T14:59:56.951436+0000","last_peered":"2024-02-19T14:59:56.951436+0000","last_clean":"2024-02-19T14:59:56.951436+0000","last_became_active":"2024-02-19T14:59:54.841950+0000","last_became_peered":"2024-02-19T14:59:54.841950+0000","last_unstale":"2024-02-19T14:59:56.951436+0000","last_undegraded":"2024-02-19T14:59:56.951436+0000","last_fullsized":"2024-02-19T14:59:56.951436+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:47.951818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950825+0000","last_change":"2024-02-19T14:59:55.856810+0000","last_active":"2024-02-19T14:59:56.950825+0000","last_peered":"2024-02-19T14:59:56.950825+0000","last_clean":"2024-02-19T14:59:56.950825+0000","last_became_active":"2024-02-19T14:59:55.856227+0000","last_became_peered":"2024-02-19T14:59:55.856227+0000","last_unstale":"2024-02-19T14:59:56.950825+0000","last_undegraded":"2024-02-19T14:59:56.950825+0000","last_fullsized":"2024-02-19T14:59:56.950825+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:18:09.216544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835020+0000","last_change":"2024-02-19T14:59:54.839629+0000","last_active":"2024-02-19T14:59:56.835020+0000","last_peered":"2024-02-19T14:59:56.835020+0000","last_clean":"2024-02-19T14:59:56.835020+0000","last_became_active":"2024-02-19T14:59:54.839357+0000","last_became_peered":"2024-02-19T14:59:54.839357+0000","last_unstale":"2024-02-19T14:59:56.835020+0000","last_undegraded":"2024-02-19T14:59:56.835020+0000","last_fullsized":"2024-02-19T14:59:56.835020+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:34:28.645152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833411+0000","last_change":"2024-02-19T14:59:55.857108+0000","last_active":"2024-02-19T14:59:56.833411+0000","last_peered":"2024-02-19T14:59:56.833411+0000","last_clean":"2024-02-19T14:59:56.833411+0000","last_became_active":"2024-02-19T14:59:55.856850+0000","last_became_peered":"2024-02-19T14:59:55.856850+0000","last_unstale":"2024-02-19T14:59:56.833411+0000","last_undegraded":"2024-02-19T14:59:56.833411+0000","last_fullsized":"2024-02-19T14:59:56.833411+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:41:29.199791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950198+0000","last_change":"2024-02-19T14:59:54.848485+0000","last_active":"2024-02-19T14:59:56.950198+0000","last_peered":"2024-02-19T14:59:56.950198+0000","last_clean":"2024-02-19T14:59:56.950198+0000","last_became_active":"2024-02-19T14:59:54.848149+0000","last_became_peered":"2024-02-19T14:59:54.848149+0000","last_unstale":"2024-02-19T14:59:56.950198+0000","last_undegraded":"2024-02-19T14:59:56.950198+0000","last_fullsized":"2024-02-19T14:59:56.950198+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:14.881873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833581+0000","last_change":"2024-02-19T14:59:55.862519+0000","last_active":"2024-02-19T14:59:56.833581+0000","last_peered":"2024-02-19T14:59:56.833581+0000","last_clean":"2024-02-19T14:59:56.833581+0000","last_became_active":"2024-02-19T14:59:55.861517+0000","last_became_peered":"2024-02-19T14:59:55.861517+0000","last_unstale":"2024-02-19T14:59:56.833581+0000","last_undegraded":"2024-02-19T14:59:56.833581+0000","last_fullsized":"2024-02-19T14:59:56.833581+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:50:32.012617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833469+0000","last_change":"2024-02-19T14:59:54.843051+0000","last_active":"2024-02-19T14:59:56.833469+0000","last_peered":"2024-02-19T14:59:56.833469+0000","last_clean":"2024-02-19T14:59:56.833469+0000","last_became_active":"2024-02-19T14:59:54.842799+0000","last_became_peered":"2024-02-19T14:59:54.842799+0000","last_unstale":"2024-02-19T14:59:56.833469+0000","last_undegraded":"2024-02-19T14:59:56.833469+0000","last_fullsized":"2024-02-19T14:59:56.833469+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:09.956992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951092+0000","last_change":"2024-02-19T14:59:55.846683+0000","last_active":"2024-02-19T14:59:56.951092+0000","last_peered":"2024-02-19T14:59:56.951092+0000","last_clean":"2024-02-19T14:59:56.951092+0000","last_became_active":"2024-02-19T14:59:55.846413+0000","last_became_peered":"2024-02-19T14:59:55.846413+0000","last_unstale":"2024-02-19T14:59:56.951092+0000","last_undegraded":"2024-02-19T14:59:56.951092+0000","last_fullsized":"2024-02-19T14:59:56.951092+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:09:45.523448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835458+0000","last_change":"2024-02-19T14:59:54.849250+0000","last_active":"2024-02-19T14:59:56.835458+0000","last_peered":"2024-02-19T14:59:56.835458+0000","last_clean":"2024-02-19T14:59:56.835458+0000","last_became_active":"2024-02-19T14:59:54.848890+0000","last_became_peered":"2024-02-19T14:59:54.848890+0000","last_unstale":"2024-02-19T14:59:56.835458+0000","last_undegraded":"2024-02-19T14:59:56.835458+0000","last_fullsized":"2024-02-19T14:59:56.835458+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:15:57.193525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835190+0000","last_change":"2024-02-19T14:59:55.847294+0000","last_active":"2024-02-19T14:59:56.835190+0000","last_peered":"2024-02-19T14:59:56.835190+0000","last_clean":"2024-02-19T14:59:56.835190+0000","last_became_active":"2024-02-19T14:59:55.846990+0000","last_became_peered":"2024-02-19T14:59:55.846990+0000","last_unstale":"2024-02-19T14:59:56.835190+0000","last_undegraded":"2024-02-19T14:59:56.835190+0000","last_fullsized":"2024-02-19T14:59:56.835190+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:56.211567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835515+0000","last_change":"2024-02-19T14:59:54.842710+0000","last_active":"2024-02-19T14:59:56.835515+0000","last_peered":"2024-02-19T14:59:56.835515+0000","last_clean":"2024-02-19T14:59:56.835515+0000","last_became_active":"2024-02-19T14:59:54.842417+0000","last_became_peered":"2024-02-19T14:59:54.842417+0000","last_unstale":"2024-02-19T14:59:56.835515+0000","last_undegraded":"2024-02-19T14:59:56.835515+0000","last_fullsized":"2024-02-19T14:59:56.835515+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T18:02:45.389592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835805+0000","last_change":"2024-02-19T14:59:55.849204+0000","last_active":"2024-02-19T14:59:56.835805+0000","last_peered":"2024-02-19T14:59:56.835805+0000","last_clean":"2024-02-19T14:59:56.835805+0000","last_became_active":"2024-02-19T14:59:55.848928+0000","last_became_peered":"2024-02-19T14:59:55.848928+0000","last_unstale":"2024-02-19T14:59:56.835805+0000","last_undegraded":"2024-02-19T14:59:56.835805+0000","last_fullsized":"2024-02-19T14:59:56.835805+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:17:01.436292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949670+0000","last_change":"2024-02-19T14:59:54.840932+0000","last_active":"2024-02-19T14:59:56.949670+0000","last_peered":"2024-02-19T14:59:56.949670+0000","last_clean":"2024-02-19T14:59:56.949670+0000","last_became_active":"2024-02-19T14:59:54.840661+0000","last_became_peered":"2024-02-19T14:59:54.840661+0000","last_unstale":"2024-02-19T14:59:56.949670+0000","last_undegraded":"2024-02-19T14:59:56.949670+0000","last_fullsized":"2024-02-19T14:59:56.949670+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:03.342512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835367+0000","last_change":"2024-02-19T14:59:55.857244+0000","last_active":"2024-02-19T14:59:56.835367+0000","last_peered":"2024-02-19T14:59:56.835367+0000","last_clean":"2024-02-19T14:59:56.835367+0000","last_became_active":"2024-02-19T14:59:55.856943+0000","last_became_peered":"2024-02-19T14:59:55.856943+0000","last_unstale":"2024-02-19T14:59:56.835367+0000","last_undegraded":"2024-02-19T14:59:56.835367+0000","last_fullsized":"2024-02-19T14:59:56.835367+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:54:04.349848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"201'2","reported_seq":16,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833789+0000","last_change":"2024-02-19T14:59:54.846407+0000","last_active":"2024-02-19T14:59:56.833789+0000","last_peered":"2024-02-19T14:59:56.833789+0000","last_clean":"2024-02-19T14:59:56.833789+0000","last_became_active":"2024-02-19T14:59:54.846137+0000","last_became_peered":"2024-02-19T14:59:54.846137+0000","last_unstale":"2024-02-19T14:59:56.833789+0000","last_undegraded":"2024-02-19T14:59:56.833789+0000","last_fullsized":"2024-02-19T14:59:56.833789+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:40:51.213200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950249+0000","last_change":"2024-02-19T14:59:55.856334+0000","last_active":"2024-02-19T14:59:56.950249+0000","last_peered":"2024-02-19T14:59:56.950249+0000","last_clean":"2024-02-19T14:59:56.950249+0000","last_became_active":"2024-02-19T14:59:55.855981+0000","last_became_peered":"2024-02-19T14:59:55.855981+0000","last_unstale":"2024-02-19T14:59:56.950249+0000","last_undegraded":"2024-02-19T14:59:56.950249+0000","last_fullsized":"2024-02-19T14:59:56.950249+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:33:33.068323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"202'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:57.836399+0000","last_change":"2024-02-19T14:59:54.843777+0000","last_active":"2024-02-19T14:59:57.836399+0000","last_peered":"2024-02-19T14:59:57.836399+0000","last_clean":"2024-02-19T14:59:57.836399+0000","last_became_active":"2024-02-19T14:59:54.842585+0000","last_became_peered":"2024-02-19T14:59:54.842585+0000","last_unstale":"2024-02-19T14:59:57.836399+0000","last_undegraded":"2024-02-19T14:59:57.836399+0000","last_fullsized":"2024-02-19T14:59:57.836399+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:17:31.183050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833374+0000","last_change":"2024-02-19T14:59:55.858567+0000","last_active":"2024-02-19T14:59:56.833374+0000","last_peered":"2024-02-19T14:59:56.833374+0000","last_clean":"2024-02-19T14:59:56.833374+0000","last_became_active":"2024-02-19T14:59:55.858315+0000","last_became_peered":"2024-02-19T14:59:55.858315+0000","last_unstale":"2024-02-19T14:59:56.833374+0000","last_undegraded":"2024-02-19T14:59:56.833374+0000","last_fullsized":"2024-02-19T14:59:56.833374+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:20:30.266982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950147+0000","last_change":"2024-02-19T14:59:54.845201+0000","last_active":"2024-02-19T14:59:56.950147+0000","last_peered":"2024-02-19T14:59:56.950147+0000","last_clean":"2024-02-19T14:59:56.950147+0000","last_became_active":"2024-02-19T14:59:54.844910+0000","last_became_peered":"2024-02-19T14:59:54.844910+0000","last_unstale":"2024-02-19T14:59:56.950147+0000","last_undegraded":"2024-02-19T14:59:56.950147+0000","last_fullsized":"2024-02-19T14:59:56.950147+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:13:29.252202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950626+0000","last_change":"2024-02-19T14:59:55.863719+0000","last_active":"2024-02-19T14:59:56.950626+0000","last_peered":"2024-02-19T14:59:56.950626+0000","last_clean":"2024-02-19T14:59:56.950626+0000","last_became_active":"2024-02-19T14:59:55.863372+0000","last_became_peered":"2024-02-19T14:59:55.863372+0000","last_unstale":"2024-02-19T14:59:56.950626+0000","last_undegraded":"2024-02-19T14:59:56.950626+0000","last_fullsized":"2024-02-19T14:59:56.950626+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:46:25.963752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950431+0000","last_change":"2024-02-19T14:59:54.845513+0000","last_active":"2024-02-19T14:59:56.950431+0000","last_peered":"2024-02-19T14:59:56.950431+0000","last_clean":"2024-02-19T14:59:56.950431+0000","last_became_active":"2024-02-19T14:59:54.845245+0000","last_became_peered":"2024-02-19T14:59:54.845245+0000","last_unstale":"2024-02-19T14:59:56.950431+0000","last_undegraded":"2024-02-19T14:59:56.950431+0000","last_fullsized":"2024-02-19T14:59:56.950431+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:11:09.956992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950292+0000","last_change":"2024-02-19T14:59:55.848520+0000","last_active":"2024-02-19T14:59:56.950292+0000","last_peered":"2024-02-19T14:59:56.950292+0000","last_clean":"2024-02-19T14:59:56.950292+0000","last_became_active":"2024-02-19T14:59:55.847551+0000","last_became_peered":"2024-02-19T14:59:55.847551+0000","last_unstale":"2024-02-19T14:59:56.950292+0000","last_undegraded":"2024-02-19T14:59:56.950292+0000","last_fullsized":"2024-02-19T14:59:56.950292+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:44:09.953528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835861+0000","last_change":"2024-02-19T14:59:54.849318+0000","last_active":"2024-02-19T14:59:56.835861+0000","last_peered":"2024-02-19T14:59:56.835861+0000","last_clean":"2024-02-19T14:59:56.835861+0000","last_became_active":"2024-02-19T14:59:54.849012+0000","last_became_peered":"2024-02-19T14:59:54.849012+0000","last_unstale":"2024-02-19T14:59:56.835861+0000","last_undegraded":"2024-02-19T14:59:56.835861+0000","last_fullsized":"2024-02-19T14:59:56.835861+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:01.000065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950537+0000","last_change":"2024-02-19T14:59:55.849469+0000","last_active":"2024-02-19T14:59:56.950537+0000","last_peered":"2024-02-19T14:59:56.950537+0000","last_clean":"2024-02-19T14:59:56.950537+0000","last_became_active":"2024-02-19T14:59:55.849216+0000","last_became_peered":"2024-02-19T14:59:55.849216+0000","last_unstale":"2024-02-19T14:59:56.950537+0000","last_undegraded":"2024-02-19T14:59:56.950537+0000","last_fullsized":"2024-02-19T14:59:56.950537+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:39:16.092292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950476+0000","last_change":"2024-02-19T14:59:54.844701+0000","last_active":"2024-02-19T14:59:56.950476+0000","last_peered":"2024-02-19T14:59:56.950476+0000","last_clean":"2024-02-19T14:59:56.950476+0000","last_became_active":"2024-02-19T14:59:54.844435+0000","last_became_peered":"2024-02-19T14:59:54.844435+0000","last_unstale":"2024-02-19T14:59:56.950476+0000","last_undegraded":"2024-02-19T14:59:56.950476+0000","last_fullsized":"2024-02-19T14:59:56.950476+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:14:39.607914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.951338+0000","last_change":"2024-02-19T14:59:55.859416+0000","last_active":"2024-02-19T14:59:56.951338+0000","last_peered":"2024-02-19T14:59:56.951338+0000","last_clean":"2024-02-19T14:59:56.951338+0000","last_became_active":"2024-02-19T14:59:55.857730+0000","last_became_peered":"2024-02-19T14:59:55.857730+0000","last_unstale":"2024-02-19T14:59:56.951338+0000","last_undegraded":"2024-02-19T14:59:56.951338+0000","last_fullsized":"2024-02-19T14:59:56.951338+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:04:47.689226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.833373+0000","last_change":"2024-02-19T14:59:54.851476+0000","last_active":"2024-02-19T14:59:56.833373+0000","last_peered":"2024-02-19T14:59:56.833373+0000","last_clean":"2024-02-19T14:59:56.833373+0000","last_became_active":"2024-02-19T14:59:54.851201+0000","last_became_peered":"2024-02-19T14:59:54.851201+0000","last_unstale":"2024-02-19T14:59:56.833373+0000","last_undegraded":"2024-02-19T14:59:56.833373+0000","last_fullsized":"2024-02-19T14:59:56.833373+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:22:14.881873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835626+0000","last_change":"2024-02-19T14:59:55.842508+0000","last_active":"2024-02-19T14:59:56.835626+0000","last_peered":"2024-02-19T14:59:56.835626+0000","last_clean":"2024-02-19T14:59:56.835626+0000","last_became_active":"2024-02-19T14:59:55.842145+0000","last_became_peered":"2024-02-19T14:59:55.842145+0000","last_unstale":"2024-02-19T14:59:56.835626+0000","last_undegraded":"2024-02-19T14:59:56.835626+0000","last_fullsized":"2024-02-19T14:59:56.835626+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:06:53.809867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835267+0000","last_change":"2024-02-19T14:59:54.843469+0000","last_active":"2024-02-19T14:59:56.835267+0000","last_peered":"2024-02-19T14:59:56.835267+0000","last_clean":"2024-02-19T14:59:56.835267+0000","last_became_active":"2024-02-19T14:59:54.843130+0000","last_became_peered":"2024-02-19T14:59:54.843130+0000","last_unstale":"2024-02-19T14:59:56.835267+0000","last_undegraded":"2024-02-19T14:59:56.835267+0000","last_fullsized":"2024-02-19T14:59:56.835267+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:49:48.144190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.950044+0000","last_change":"2024-02-19T14:59:55.847781+0000","last_active":"2024-02-19T14:59:56.950044+0000","last_peered":"2024-02-19T14:59:56.950044+0000","last_clean":"2024-02-19T14:59:56.950044+0000","last_became_active":"2024-02-19T14:59:55.845992+0000","last_became_peered":"2024-02-19T14:59:55.845992+0000","last_unstale":"2024-02-19T14:59:56.950044+0000","last_undegraded":"2024-02-19T14:59:56.950044+0000","last_fullsized":"2024-02-19T14:59:56.950044+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:54.821792+0000","last_clean_scrub_stamp":"2024-02-19T14:59:54.821792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:17.056685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"201'1","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.835206+0000","last_change":"2024-02-19T14:59:54.843958+0000","last_active":"2024-02-19T14:59:56.835206+0000","last_peered":"2024-02-19T14:59:56.835206+0000","last_clean":"2024-02-19T14:59:56.835206+0000","last_became_active":"2024-02-19T14:59:54.843503+0000","last_became_peered":"2024-02-19T14:59:54.843503+0000","last_unstale":"2024-02-19T14:59:56.835206+0000","last_undegraded":"2024-02-19T14:59:56.835206+0000","last_fullsized":"2024-02-19T14:59:56.835206+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:59:53.814456+0000","last_clean_scrub_stamp":"2024-02-19T14:59:53.814456+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:45:39.339962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":510,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-02-19T14:59:56.949900+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T14:59:56.949900+0000","last_peered":"2024-02-19T14:59:56.949900+0000","last_clean":"2024-02-19T14:59:56.949900+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T14:59:56.949900+0000","last_undegraded":"2024-02-19T14:59:56.949900+0000","last_fullsized":"2024-02-19T14:59:56.949900+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574950,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34272,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337568,"statfs":{"total":96636764160,"available":96601669632,"internally_reserved":0,"allocated":3162112,"data_stored":1842748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.441}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.379},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33692,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338148,"statfs":{"total":96636764160,"available":96602263552,"internally_reserved":0,"allocated":2568192,"data_stored":1252368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.48199999999999998,"15min":0.46899999999999997},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.68899999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.496},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.70999999999999996,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.374,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.81999999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.747,"5min":0.80500000000000005,"15min":0.88100000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.70499999999999996,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.90500000000000003},"last":0.62}]},{"osd":3,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73199999999999998,"5min":0.753,"15min":0.84799999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.52800000000000002},"min":{"1min":0.375,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64800000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51100000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47299999999999998,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.216,"15min":0.216},"max":{"1min":0.79300000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56100000000000005,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.52600000000000002},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.748,"5min":0.753,"15min":0.77400000000000002},"last":0.433},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82099999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.374}]},{"osd":7,"last update":"Mon Feb 19 14:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.56100000000000005},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79500000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.39100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33684,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338156,"statfs":{"total":96636764160,"available":96602271744,"internally_reserved":0,"allocated":2560000,"data_stored":1252256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.755,"5min":0.79600000000000004,"15min":0.89100000000000001},"last":0.753},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78300000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.67000000000000004}]},{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.499},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.872},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29399999999999998},"max":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.92600000000000005},"last":0.59199999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71799999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.55500000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77100000000000002,"5min":0.77300000000000002,"15min":0.85899999999999999},"last":0.71599999999999997}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.70499999999999996,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66200000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48299999999999998,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.72499999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51700000000000002,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.91600000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.54500000000000004},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.80600000000000005,"5min":0.877,"15min":0.877},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.55600000000000005},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.72799999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55000000000000004,"15min":0.58699999999999997},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.749,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.61599999999999999},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76500000000000001,"5min":1.03,"15min":1.03},"last":0.89200000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574950,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33688,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338152,"statfs":{"total":96636764160,"available":96602267648,"internally_reserved":0,"allocated":2564096,"data_stored":1252326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82999999999999996,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":1.341,"15min":1.341},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67000000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76500000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":10.629,"15min":10.629},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.52100000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72399999999999998,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.441},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60799999999999998,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85299999999999998,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.51000000000000001},"min":{"1min":0.31,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50900000000000001,"15min":0.51400000000000001},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.871,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.29399999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.92400000000000004,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60799999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89000000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.54400000000000004,"15min":0.56399999999999995},"min":{"1min":0.34699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.79100000000000004,"5min":1.246,"15min":1.246},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.36399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.024,"5min":2.621,"15min":2.621},"last":0.51100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33708,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338132,"statfs":{"total":96636764160,"available":96602247168,"internally_reserved":0,"allocated":2584576,"data_stored":1254506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.315},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.28999999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33692,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338148,"statfs":{"total":96636764160,"available":96602263552,"internally_reserved":0,"allocated":2568192,"data_stored":1252908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32200000000000001,"15min":0.317},"min":{"1min":0.19600000000000001,"5min":0.182,"15min":0.17399999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.308},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32600000000000001,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.51800000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.32200000000000001}]},{"osd":1,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58599999999999997,"5min":1.512,"15min":1.512},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38,"15min":0.376},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.624,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.746,"5min":1.133,"15min":1.133},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.83299999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81599999999999995,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71099999999999997,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86599999999999999,"5min":1.115,"15min":1.115},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.72299999999999998,"5min":0.88500000000000001,"15min":1.236},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.76200000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.501},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.73499999999999999,"5min":0.90200000000000002,"15min":0.995},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33684,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338156,"statfs":{"total":96636764160,"available":96602271744,"internally_reserved":0,"allocated":2560000,"data_stored":1252256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.214},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.246}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":1844236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.30499999999999999},"min":{"1min":0.24399999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.253},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.23699999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.23300000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.23999999999999999,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.36699999999999999,"15min":0.35599999999999998},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.308}]},{"osd":3,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.501,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.379,"15min":0.373},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70599999999999996,"5min":0.78500000000000003,"15min":0.78900000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55800000000000005,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.84799999999999998,"15min":0.90100000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56200000000000006,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":1.1339999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54800000000000004,"15min":0.57499999999999996},"min":{"1min":0.46300000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.67100000000000004}]},{"osd":6,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57199999999999995,"15min":0.59299999999999997},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.45800000000000002,"5min":0.34599999999999997,"15min":0.34100000000000003},"max":{"1min":0.78100000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 19 14:59:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.79300000000000004,"5min":0.91600000000000004,"15min":1.0149999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55200000000000005,"15min":0.58099999999999996},"min":{"1min":0.42699999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71499999999999997,"5min":0.81100000000000005,"15min":0.96099999999999997},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T14:59:59.187 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T14:59:59.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T14:59:59.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.331+0000 7f0d819d7640 1 Processor -- start 2024-02-19T14:59:59.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.331+0000 7f0d819d7640 1 -- start start 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d819d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d7c06ea50 0x7f0d7c072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d819d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c0c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d819d7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0c5970 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d819d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7c0c5750 con 0x7f0d7c06ea50 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d819d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0702d0 con 0x7f0d7c152680 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7a7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d7c06ea50 0x7f0d7c072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c0c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7affd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0d7c14f1a0 0x7f0d7c06fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46508/0 (socket says 172.21.15.78:46508) 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7affd640 1 -- 172.21.15.78:0/1791251082 learned_addr learned my addr 172.21.15.78:0/1791251082 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7b7fe640 1 -- 172.21.15.78:0/1791251082 >> v2:172.21.15.150:3300/0 conn(0x7f0d7c06ea50 msgr2=0x7f0d7c072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:59.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7b7fe640 1 --2- 172.21.15.78:0/1791251082 >> v2:172.21.15.150:3300/0 conn(0x7f0d7c06ea50 0x7f0d7c072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7b7fe640 1 -- 172.21.15.78:0/1791251082 >> v1:172.21.15.150:6789/0 conn(0x7f0d7c14f1a0 legacy=0x7f0d7c06fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.333+0000 7f0d7b7fe640 1 -- 172.21.15.78:0/1791251082 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0c5bc0 con 0x7f0d7c152680 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.334+0000 7f0d7b7fe640 1 --2- 172.21.15.78:0/1791251082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c0c78a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009d90 tx=0x7f0d7402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1027d6c10e9587 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.334+0000 7f0d79ffb640 1 -- 172.21.15.78:0/1791251082 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7402cb30 con 0x7f0d7c152680 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.334+0000 7f0d79ffb640 1 -- 172.21.15.78:0/1791251082 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7402cc90 con 0x7f0d7c152680 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.334+0000 7f0d79ffb640 1 -- 172.21.15.78:0/1791251082 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74034440 con 0x7f0d7c152680 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 -- 172.21.15.78:0/1791251082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 msgr2=0x7f0d7c0c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 --2- 172.21.15.78:0/1791251082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c0c78a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009d90 tx=0x7f0d7402bd80 comp rx=0 tx=0).stop 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 -- 172.21.15.78:0/1791251082 shutdown_connections 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 --2- 172.21.15.78:0/1791251082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c0c78a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 --2- 172.21.15.78:0/1791251082 >> v2:172.21.15.150:3300/0 conn(0x7f0d7c06ea50 0x7f0d7c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 -- 172.21.15.78:0/1791251082 >> 172.21.15.78:0/1791251082 conn(0x7f0d7c06c170 msgr2=0x7f0d7c06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:59.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 -- 172.21.15.78:0/1791251082 shutdown_connections 2024-02-19T14:59:59.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.335+0000 7f0d819d7640 1 -- 172.21.15.78:0/1791251082 wait complete. 2024-02-19T14:59:59.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.337+0000 7f0d819d7640 1 Processor -- start 2024-02-19T14:59:59.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.337+0000 7f0d819d7640 1 -- start start 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d819d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d7c06ea50 0x7f0d7c153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d819d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 0x7f0d7c1e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d819d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c1e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d819d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c154020 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d819d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c1e5130 con 0x7f0d7c06ea50 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d819d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c154660 con 0x7f0d7c152680 2024-02-19T14:59:59.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d7a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 0x7f0d7c1e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d7a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 0x7f0d7c1e46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36040/0 (socket says 172.21.15.78:36040) 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d7a7fc640 1 -- 172.21.15.78:0/3511465063 learned_addr learned my addr 172.21.15.78:0/3511465063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d7b7fe640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c1e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.338+0000 7f0d7affd640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d7c06ea50 0x7f0d7c153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7a7fc640 1 -- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 msgr2=0x7f0d7c1e4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7a7fc640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c1e4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7a7fc640 1 -- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d7c06ea50 msgr2=0x7f0d7c153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7a7fc640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d7c06ea50 0x7f0d7c153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7a7fc640 1 -- 172.21.15.78:0/3511465063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d74009a70 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7affd640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d7c06ea50 0x7f0d7c153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d7a7fc640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 0x7f0d7c1e46b0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009f50 tx=0x7f0d6c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0094f0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c0c21e0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c00c8c0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c00cc70 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.339+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c0c27a0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.340+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c06c9d0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.341+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0d6c011450 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.341+0000 7f0d63fff640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d6806b840 0x7f0d6806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.341+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d6c08bfe0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.341+0000 7f0d7affd640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d6806b840 0x7f0d6806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.342+0000 7f0d7affd640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d6806b840 0x7f0d6806dd00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f0d70005880 tx=0x7f0d70009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:59.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.345+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0d6c056930 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.592+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0d7c074990 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.594+0000 7f0d63fff640 1 -- 172.21.15.78:0/3511465063 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0d6c059ff0 con 0x7f0d7c14f1a0 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T14:59:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T14:59:59.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d6806b840 msgr2=0x7f0d6806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d6806b840 0x7f0d6806dd00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f0d70005880 tx=0x7f0d70009040 comp rx=0 tx=0).stop 2024-02-19T14:59:59.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 msgr2=0x7f0d7c1e46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 0x7f0d7c1e46b0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009f50 tx=0x7f0d6c00ed00 comp rx=0 tx=0).stop 2024-02-19T14:59:59.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 shutdown_connections 2024-02-19T14:59:59.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d6806b840 0x7f0d6806dd00 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d7c14f1a0 0x7f0d7c1e46b0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d7c152680 0x7f0d7c1e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 --2- 172.21.15.78:0/3511465063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d7c06ea50 0x7f0d7c153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 >> 172.21.15.78:0/3511465063 conn(0x7f0d7c06c170 msgr2=0x7f0d7c06fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 shutdown_connections 2024-02-19T14:59:59.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.597+0000 7f0d819d7640 1 -- 172.21.15.78:0/3511465063 wait complete. 2024-02-19T14:59:59.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T14:59:59.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.711+0000 7ffb23eff640 1 Processor -- start 2024-02-19T14:59:59.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.711+0000 7ffb23eff640 1 -- start start 2024-02-19T14:59:59.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.712+0000 7ffb23eff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb1c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.712+0000 7ffb23eff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0c5730 0x7ffb1c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.712+0000 7ffb23eff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c1551f0 con 0x7ffb1c0c8e00 2024-02-19T14:59:59.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.712+0000 7ffb23eff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c155490 con 0x7ffb1c0c5730 2024-02-19T14:59:59.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.712+0000 7ffb23eff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c155770 con 0x7ffb1c152700 2024-02-19T14:59:59.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb2249c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ffb1c0c8e00 0x7ffb1c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40306/0 (socket says 172.21.15.78:40306) 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb2249c640 1 -- 172.21.15.78:0/2849035345 learned_addr learned my addr 172.21.15.78:0/2849035345 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb2149a640 1 --2- 172.21.15.78:0/2849035345 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0c5730 0x7ffb1c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb1c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117121136 0 0) 0x7ffb1c1551f0 con 0x7ffb1c0c8e00 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb08003520 con 0x7ffb1c0c8e00 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127477247 0 0) 0x7ffb08003520 con 0x7ffb1c0c8e00 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 msgr2=0x7ffb1c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb1c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0c5730 msgr2=0x7ffb1c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 --2- 172.21.15.78:0/2849035345 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0c5730 0x7ffb1c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c155a50 con 0x7ffb1c0c8e00 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7ffb140019f0 con 0x7ffb1c0c8e00 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 >> v1:172.21.15.78:6789/0 conn(0x7ffb1c0c8e00 legacy=0x7ffb1c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb080047c0 0x7ffb08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c155490 con 0x7ffb08003970 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c155770 con 0x7ffb080047c0 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0800b980 con 0x7ffb1c152700 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb2149a640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb080047c0 0x7ffb08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.713+0000 7ffb2249c640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb21c9b640 1 -- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 msgr2=0x7ffb08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb08005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb21c9b640 1 -- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb080047c0 msgr2=0x7ffb08004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb080047c0 0x7ffb08004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb21c9b640 1 -- 172.21.15.78:0/2849035345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb08003520 con 0x7ffb08003970 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb2149a640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb080047c0 0x7ffb08004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb2249c640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c152700 0x7ffb08005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T14:59:59.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb08003d90 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009ac0 tx=0x7ffb080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b77c5355a8a11da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c02c6b0 con 0x7ffb08003970 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c02c810 con 0x7ffb08003970 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.714+0000 7ffb20c99640 1 -- 172.21.15.78:0/2849035345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c02cba0 con 0x7ffb08003970 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb23eff640 1 -- 172.21.15.78:0/2849035345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 msgr2=0x7ffb08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb23eff640 1 --2- 172.21.15.78:0/2849035345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb08003d90 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009ac0 tx=0x7ffb080057f0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb2249c640 1 -- 172.21.15.78:0/2849035345 reap_dead start 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb23eff640 1 -- 172.21.15.78:0/2849035345 shutdown_connections 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb23eff640 1 -- 172.21.15.78:0/2849035345 >> 172.21.15.78:0/2849035345 conn(0x7ffb1c0cac90 msgr2=0x7ffb1c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb23eff640 1 -- 172.21.15.78:0/2849035345 shutdown_connections 2024-02-19T14:59:59.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.715+0000 7ffb23eff640 1 -- 172.21.15.78:0/2849035345 wait complete. 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.716+0000 7ffb23eff640 1 Processor -- start 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.716+0000 7ffb23eff640 1 -- start start 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb23eff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb1c0ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb23eff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0acaf0 0x7ffb1c1ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb23eff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c156ce0 0x7ffb1c1e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb23eff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1eece0 con 0x7ffb08003970 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb23eff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c158e40 con 0x7ffb1c156ce0 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb23eff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1568c0 con 0x7ffb1c0acaf0 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb1c0ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb1c0ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36054/0 (socket says 172.21.15.78:36054) 2024-02-19T14:59:59.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 -- 172.21.15.78:0/1008636167 learned_addr learned my addr 172.21.15.78:0/1008636167 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb2249c640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c156ce0 0x7ffb1c1e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 -- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0acaf0 msgr2=0x7ffb1c1ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb2149a640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0acaf0 0x7ffb1c1ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0acaf0 0x7ffb1c1ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 -- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c156ce0 msgr2=0x7ffb1c1e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c156ce0 0x7ffb1c1e7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 -- 172.21.15.78:0/1008636167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c159090 con 0x7ffb08003970 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb2149a640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0acaf0 0x7ffb1c1ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb2249c640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c156ce0 0x7ffb1c1e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb21c9b640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb1c0ab660 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009be0 tx=0x7ffb0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.717+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c003810 con 0x7ffb08003970 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.718+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c003970 con 0x7ffb08003970 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.718+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c003d20 con 0x7ffb08003970 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.718+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c1eea00 con 0x7ffb08003970 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.718+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c0ab140 con 0x7ffb08003970 2024-02-19T14:59:59.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.719+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c0aacc0 con 0x7ffb08003970 2024-02-19T14:59:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.719+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ffb0c040450 con 0x7ffb08003970 2024-02-19T14:59:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.719+0000 7ffb127fc640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb0006b840 0x7ffb0006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T14:59:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.720+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb0c0ac970 con 0x7ffb08003970 2024-02-19T14:59:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.720+0000 7ffb2149a640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb0006b840 0x7ffb0006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T14:59:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.720+0000 7ffb2149a640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb0006b840 0x7ffb0006dd00 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0ad1b0 tx=0x7ffb18008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T14:59:59.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.724+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffb1c0aacc0 con 0x7ffb08003970 2024-02-19T14:59:59.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.938+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb1c0aacc0 con 0x7ffb08003970 2024-02-19T14:59:59.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.939+0000 7ffb127fc640 1 -- 172.21.15.78:0/1008636167 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4949 (secure 0 0 0) 0x7ffb1c0aacc0 con 0x7ffb08003970 2024-02-19T14:59:59.942 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T14:59:59.942 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":180,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:55.177591+0000","modified":"2024-02-19T14:59:57.199274+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10344},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10344":{"gid":10344,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3213692827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3213692827},{"type":"v1","addr":"172.21.15.78:6837","nonce":3213692827}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-19T14:59:59.942 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 180 2024-02-19T14:59:59.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb0006b840 msgr2=0x7ffb0006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb0006b840 0x7ffb0006dd00 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0ad1b0 tx=0x7ffb18008040 comp rx=0 tx=0).stop 2024-02-19T14:59:59.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 msgr2=0x7ffb1c0ab660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T14:59:59.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb1c0ab660 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009be0 tx=0x7ffb0c004730 comp rx=0 tx=0).stop 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 shutdown_connections 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ffb0006b840 0x7ffb0006dd00 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c156ce0 0x7ffb1c1e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0acaf0 0x7ffb1c1ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 --2- 172.21.15.78:0/1008636167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb08003970 0x7ffb1c0ab660 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 >> 172.21.15.78:0/1008636167 conn(0x7ffb1c0cac90 msgr2=0x7ffb1c14fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 shutdown_connections 2024-02-19T14:59:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T14:59:59.941+0000 7ffb23eff640 1 -- 172.21.15.78:0/1008636167 wait complete. 2024-02-19T15:00:00.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 180, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T14:59:55.177591+0000', 'modified': '2024-02-19T14:59:57.199274+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10344}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10344': {'gid': 10344, 'name': 'a', 'rank': 0, 'incarnation': 178, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3213692827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3213692827}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3213692827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:00:00.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:00:00.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-19T15:00:00.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.457+0000 7f771f3e2640 1 Processor -- start 2024-02-19T15:00:00.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 -- start start 2024-02-19T15:00:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7718152670 0x7f7718071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7718152a50 0x7f7718155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f7718158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7718071bf0 con 0x7f7718152a50 2024-02-19T15:00:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7718158a20 con 0x7f7718152670 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771f3e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7718155550 con 0x7f77180702d0 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771c97d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7718152a50 0x7f7718155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771c97d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7718152a50 0x7f7718155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36068/0 (socket says 172.21.15.78:36068) 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.458+0000 7f771c97d640 1 -- 172.21.15.78:0/2789454176 learned_addr learned my addr 172.21.15.78:0/2789454176 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771d17e640 1 --2- 172.21.15.78:0/2789454176 >> v2:172.21.15.150:3300/0 conn(0x7f7718152670 0x7f7718071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771d97f640 1 --2- 172.21.15.78:0/2789454176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f7718158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771c97d640 1 -- 172.21.15.78:0/2789454176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 msgr2=0x7f7718158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771c97d640 1 --2- 172.21.15.78:0/2789454176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f7718158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771c97d640 1 -- 172.21.15.78:0/2789454176 >> v2:172.21.15.150:3300/0 conn(0x7f7718152670 msgr2=0x7f7718071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771c97d640 1 --2- 172.21.15.78:0/2789454176 >> v2:172.21.15.150:3300/0 conn(0x7f7718152670 0x7f7718071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771c97d640 1 -- 172.21.15.78:0/2789454176 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180c6780 con 0x7f7718152a50 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.459+0000 7f771c97d640 1 --2- 172.21.15.78:0/2789454176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718155010 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f771806e600 tx=0x7f770002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487200fac775c8d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f7717fff640 1 -- 172.21.15.78:0/2789454176 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770002ca70 con 0x7f7718152a50 2024-02-19T15:00:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f7717fff640 1 -- 172.21.15.78:0/2789454176 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770002cbd0 con 0x7f7718152a50 2024-02-19T15:00:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f7717fff640 1 -- 172.21.15.78:0/2789454176 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77000343f0 con 0x7f7718152a50 2024-02-19T15:00:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 -- 172.21.15.78:0/2789454176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 msgr2=0x7f7718155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2789454176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718155010 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f771806e600 tx=0x7f770002bed0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 -- 172.21.15.78:0/2789454176 shutdown_connections 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2789454176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f7718158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2789454176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718155010 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2789454176 >> v2:172.21.15.150:3300/0 conn(0x7f7718152670 0x7f7718071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 -- 172.21.15.78:0/2789454176 >> 172.21.15.78:0/2789454176 conn(0x7f77180c85e0 msgr2=0x7f77180c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 -- 172.21.15.78:0/2789454176 shutdown_connections 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.460+0000 7f771f3e2640 1 -- 172.21.15.78:0/2789454176 wait complete. 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 Processor -- start 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 -- start start 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f771814d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7718152670 0x7f771806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7718144990 con 0x7f7718152a50 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77181eb6d0 con 0x7f7718152670 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771f3e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77181eb9b0 con 0x7f77180702d0 2024-02-19T15:00:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771d97f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771d97f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718153e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36080/0 (socket says 172.21.15.78:36080) 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771d97f640 1 -- 172.21.15.78:0/2443718194 learned_addr learned my addr 172.21.15.78:0/2443718194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.462+0000 7f771d17e640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f771814d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771c97d640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7718152670 0x7f771806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771d97f640 1 -- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 msgr2=0x7f771814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771d97f640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f771814d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771d97f640 1 -- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7718152670 msgr2=0x7f771806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771d97f640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7718152670 0x7f771806bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771d97f640 1 -- 172.21.15.78:0/2443718194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7700009b30 con 0x7f7718152a50 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771c97d640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7718152670 0x7f771806bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771d97f640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718153e80 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f770800d5d0 tx=0x7f770800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f7715ffb640 1 -- 172.21.15.78:0/2443718194 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708014620 con 0x7f7718152a50 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f7715ffb640 1 -- 172.21.15.78:0/2443718194 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77080147a0 con 0x7f7718152a50 2024-02-19T15:00:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f7715ffb640 1 -- 172.21.15.78:0/2443718194 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708014b50 con 0x7f7718152a50 2024-02-19T15:00:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77181533a0 con 0x7f7718152a50 2024-02-19T15:00:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.463+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7718153960 con 0x7f7718152a50 2024-02-19T15:00:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.465+0000 7f7715ffb640 1 -- 172.21.15.78:0/2443718194 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7708014cb0 con 0x7f7718152a50 2024-02-19T15:00:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.465+0000 7f7715ffb640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f76f806b840 0x7f76f806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.465+0000 7f7715ffb640 1 -- 172.21.15.78:0/2443718194 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f770808a630 con 0x7f7718152a50 2024-02-19T15:00:00.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.465+0000 7f771d17e640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f76f806b840 0x7f76f806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.466+0000 7f771d17e640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f76f806b840 0x7f76f806dd00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f77100079a0 tx=0x7f7710008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.466+0000 7f7721fc1740 1 Processor -- start 2024-02-19T15:00:00.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.466+0000 7f7721fc1740 1 -- start start 2024-02-19T15:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.467+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa11f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11baae0 0x557aa11f4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa11b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f7721fc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa1174cd0 con 0x557aa11f4740 2024-02-19T15:00:00.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa1127f20 con 0x557aa11baae0 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x557aa1125450 con 0x557aa11f78b0 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa11f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa11f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36086/0 (socket says 172.21.15.78:36086) 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 -- 172.21.15.78:0/2159786785 learned_addr learned my addr 172.21.15.78:0/2159786785 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 -- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 msgr2=0x557aa11b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d97f640 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa11b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa11b9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 -- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11baae0 msgr2=0x557aa11f4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771c97d640 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11baae0 0x557aa11f4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11baae0 0x557aa11f4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 -- 172.21.15.78:0/2159786785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557aa10d5f20 con 0x557aa11f4740 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771c97d640 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11baae0 0x557aa11f4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f771d17e640 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa11f58b0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f7710010af0 tx=0x7f7710013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2111b06d9c65ad5f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f770effd640 1 -- 172.21.15.78:0/2159786785 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710016cd0 con 0x557aa11f4740 2024-02-19T15:00:00.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f770effd640 1 -- 172.21.15.78:0/2159786785 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771001a070 con 0x557aa11f4740 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.468+0000 7f770effd640 1 -- 172.21.15.78:0/2159786785 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710021640 con 0x557aa11f4740 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 -- 172.21.15.78:0/2159786785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 msgr2=0x557aa11f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa11f58b0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f7710010af0 tx=0x7f7710013000 comp rx=0 tx=0).stop 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 -- 172.21.15.78:0/2159786785 shutdown_connections 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa11b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11baae0 0x557aa11f4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 --2- 172.21.15.78:0/2159786785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa11f58b0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 -- 172.21.15.78:0/2159786785 >> 172.21.15.78:0/2159786785 conn(0x557aa11b6120 msgr2=0x557aa11dab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 -- 172.21.15.78:0/2159786785 shutdown_connections 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 -- 172.21.15.78:0/2159786785 wait complete. 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 -- start start 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.469+0000 7f7721fc1740 1 Processor -- start 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f4740 0x557aa11f2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 0x557aa11b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11eca70 0x557aa11cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa1120c30 con 0x557aa11f78b0 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa10c5d30 con 0x557aa11f4740 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x557aa0df4310 con 0x557aa11eca70 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f7721fc1740 10 client.0 _ll_register_callbacks cb 0x557aa10cdd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f771c97d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 0x557aa11b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f771c97d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 0x557aa11b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36096/0 (socket says 172.21.15.78:36096) 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.470+0000 7f771c97d640 1 -- 172.21.15.78:0/565352356 learned_addr learned my addr 172.21.15.78:0/565352356 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771d17e640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f4740 0x557aa11f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771d97f640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11eca70 0x557aa11cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771c97d640 1 -- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11eca70 msgr2=0x557aa11cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771c97d640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11eca70 0x557aa11cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771c97d640 1 -- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f4740 msgr2=0x557aa11f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771c97d640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f4740 0x557aa11f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771c97d640 1 -- 172.21.15.78:0/565352356 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771000e040 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f771c97d640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 0x557aa11b88e0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f7700009f40 tx=0x7f7700004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1af9b9f9969543d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f76f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f76f7fff640 1 -- 172.21.15.78:0/565352356 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700034c30 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f76f7fff640 1 -- 172.21.15.78:0/565352356 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7700044070 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557aa0e206c0 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f76f7fff640 1 -- 172.21.15.78:0/565352356 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7700034d90 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f7721fc1740 10 client.10482 fetch_fsmap learned FSMap version 180 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x557aa111f2b0 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f76f7fff640 1 -- 172.21.15.78:0/565352356 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557aa0e206c0 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.471+0000 7f76f7fff640 1 -- 172.21.15.78:0/565352356 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 180) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7700048430 con 0x557aa11f78b0 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f7721fc1740 10 client.10482 fetch_fsmap finished waiting for FSMap version 180 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f7721fc1740 10 client.10482 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f7721fc1740 20 client.10482 populate_metadata read hostname 'smithi078' 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f7721fc1740 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa12090a0 0x557aa1209480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f7721fc1740 4 client.10482 mds_command: new command op to 10344 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 --> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557aa0e71270 con 0x557aa12090a0 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f771d17e640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa12090a0 0x557aa1209480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f76f7fff640 10 client.10482 ms_handle_connect on v2:172.21.15.78:6835/3213692827 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.472+0000 7f771d17e640 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa12090a0 0x557aa1209480 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e49340eb6f8b25 server_cookie=6454e1d8701cc13e in_seq=0 out_seq=0 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f76f7fff640 1 -- 172.21.15.78:0/565352356 <== mds.0 v2:172.21.15.78:6835/3213692827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557aa0e71270 con 0x557aa12090a0 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f76f7fff640 10 client.10482 handle_command_reply: tid=0 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 client.10482 shutdown 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 msgr2=0x557aa11b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 0x557aa11b88e0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f7700009f40 tx=0x7f7700004ab0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 shutdown_connections 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f78b0 0x557aa11b88e0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa12090a0 0x557aa1209480 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11eca70 0x557aa11cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 --2- 172.21.15.78:0/565352356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f4740 0x557aa11f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f76f7fff640 0 client.10482 ms_handle_reset on v2:172.21.15.78:6835/3213692827 2024-02-19T15:00:00.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 >> 172.21.15.78:0/565352356 conn(0x557aa11baae0 msgr2=0x557aa11b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 shutdown_connections 2024-02-19T15:00:00.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 1 -- 172.21.15.78:0/565352356 wait complete. 2024-02-19T15:00:00.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.473+0000 7f7721fc1740 20 client.10482 trim_cache size 0 max 16384 2024-02-19T15:00:00.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.490+0000 7f7721fc1740 1 Processor -- start 2024-02-19T15:00:00.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.490+0000 7f7721fc1740 1 -- start start 2024-02-19T15:00:00.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 0x557aa11b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f4740 0x557aa11edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f78b0 0x557aa11b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f7721fc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa11b9e20 con 0x557aa11eca70 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa10a6490 con 0x557aa11f78b0 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x557aa11f2d40 con 0x557aa11f4740 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 0x557aa11b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 0x557aa11b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36102/0 (socket says 172.21.15.78:36102) 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 -- 172.21.15.78:0/3335144981 learned_addr learned my addr 172.21.15.78:0/3335144981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 -- 172.21.15.78:0/3335144981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f4740 msgr2=0x557aa11edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f4740 0x557aa11edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 -- 172.21.15.78:0/3335144981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f78b0 msgr2=0x557aa11b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f78b0 0x557aa11b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.491+0000 7f771d17e640 1 -- 172.21.15.78:0/3335144981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718154ba0 con 0x557aa11eca70 2024-02-19T15:00:00.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f771d17e640 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 0x557aa11b2f40 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f7710022000 tx=0x7f7710015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e50731a2e289861 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f770e7fc640 1 -- 172.21.15.78:0/3335144981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100352d0 con 0x557aa11eca70 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f770e7fc640 1 -- 172.21.15.78:0/3335144981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7710033040 con 0x557aa11eca70 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f770e7fc640 1 -- 172.21.15.78:0/3335144981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710005b90 con 0x557aa11eca70 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 -- 172.21.15.78:0/3335144981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 msgr2=0x557aa11b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 0x557aa11b2f40 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f7710022000 tx=0x7f7710015000 comp rx=0 tx=0).stop 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 -- 172.21.15.78:0/3335144981 shutdown_connections 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11f78b0 0x557aa11b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f4740 0x557aa11edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3335144981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11eca70 0x557aa11b2f40 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.492+0000 7f7721fc1740 1 -- 172.21.15.78:0/3335144981 >> 172.21.15.78:0/3335144981 conn(0x557aa11baae0 msgr2=0x557aa11f5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.493+0000 7f7721fc1740 1 -- 172.21.15.78:0/3335144981 shutdown_connections 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.493+0000 7f7721fc1740 1 -- 172.21.15.78:0/3335144981 wait complete. 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.493+0000 7f7721fc1740 1 -- start start 2024-02-19T15:00:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.493+0000 7f7721fc1740 1 Processor -- start 2024-02-19T15:00:00.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11eca70 0x557aa11b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa1218b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa1262310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa11f29d0 con 0x557aa11f4740 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557aa11f26f0 con 0x557aa11eca70 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x557aa1265b10 con 0x557aa11f78b0 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f7721fc1740 10 client.0 _ll_register_callbacks cb 0x7f77000095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.494+0000 7f771c97d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa1218b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa1218b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36114/0 (socket says 172.21.15.78:36114) 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 -- 172.21.15.78:0/3583800179 learned_addr learned my addr 172.21.15.78:0/3583800179 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771d97f640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa1262310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771d17e640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11eca70 0x557aa11b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 -- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 msgr2=0x557aa1262310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa1262310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 -- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11eca70 msgr2=0x557aa11b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11eca70 0x557aa11b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 -- 172.21.15.78:0/3583800179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557aa10d5f20 con 0x557aa11f4740 2024-02-19T15:00:00.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f771c97d640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa1218b20 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f770002cba0 tx=0x7f7700004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a840d13bcdd60a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f76f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f76f7fff640 1 -- 172.21.15.78:0/3583800179 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770005a070 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f76f7fff640 1 -- 172.21.15.78:0/3583800179 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7700044040 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f771001a070 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f76f7fff640 1 -- 172.21.15.78:0/3583800179 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7700034600 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 10 client.10488 fetch_fsmap learned FSMap version 180 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x557aa10bba20 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f76f7fff640 1 -- 172.21.15.78:0/3583800179 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771001a070 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f76f7fff640 1 -- 172.21.15.78:0/3583800179 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 180) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7700034a30 con 0x557aa11f4740 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 10 client.10488 fetch_fsmap finished waiting for FSMap version 180 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 10 client.10488 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 20 client.10488 populate_metadata read hostname 'smithi078' 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa1219120 0x557aa1262850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 4 client.10488 mds_command: new command op to 10344 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.495+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 --> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557aa0e5d610 con 0x557aa1219120 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f771d17e640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa1219120 0x557aa1262850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f76f7fff640 10 client.10488 ms_handle_connect on v2:172.21.15.78:6835/3213692827 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f771d17e640 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa1219120 0x557aa1262850 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7464dc9d4235184 server_cookie=fced4e810bc889dd in_seq=0 out_seq=0 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f76f7fff640 1 -- 172.21.15.78:0/3583800179 <== mds.0 v2:172.21.15.78:6835/3213692827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x557aa0e5d610 con 0x557aa1219120 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f76f7fff640 10 client.10488 handle_command_reply: tid=0 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 client.10488 shutdown 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 msgr2=0x557aa1218b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa1218b20 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f770002cba0 tx=0x7f7700004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 shutdown_connections 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557aa11f4740 0x557aa1218b20 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.78:6835/3213692827,v1:172.21.15.78:6837/3213692827] conn(0x557aa1219120 0x557aa1262850 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x557aa11f78b0 0x557aa1262310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 --2- 172.21.15.78:0/3583800179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557aa11eca70 0x557aa11b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f76f7fff640 0 client.10488 ms_handle_reset on v2:172.21.15.78:6835/3213692827 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.496+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 >> 172.21.15.78:0/3583800179 conn(0x557aa11baae0 msgr2=0x557aa11f5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 shutdown_connections 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f7721fc1740 1 -- 172.21.15.78:0/3583800179 wait complete. 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f7721fc1740 20 client.10488 trim_cache size 0 max 16384 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 10344, 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T15:00:00.505 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T15:00:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 4.6664873890000003, 2024-02-19T15:00:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 180, 2024-02-19T15:00:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 202, 2024-02-19T15:00:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 202, 2024-02-19T15:00:00.506 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 9.7721008600000001 2024-02-19T15:00:00.506 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f76f806b840 msgr2=0x7f76f806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f76f806b840 0x7f76f806dd00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f77100079a0 tx=0x7f7710008040 comp rx=0 tx=0).stop 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 msgr2=0x7f7718153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718153e80 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f770800d5d0 tx=0x7f770800da90 comp rx=0 tx=0).stop 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 shutdown_connections 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f76f806b840 0x7f76f806dd00 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7718152a50 0x7f7718153e80 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7718152670 0x7f771806bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 --2- 172.21.15.78:0/2443718194 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f77180702d0 0x7f771814d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 >> 172.21.15.78:0/2443718194 conn(0x7f77180c85e0 msgr2=0x7f7718156d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 shutdown_connections 2024-02-19T15:00:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.497+0000 7f771f3e2640 1 -- 172.21.15.78:0/2443718194 wait complete. 2024-02-19T15:00:00.511 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-19T15:00:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.612+0000 7ff3292ed640 1 Processor -- start 2024-02-19T15:00:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.612+0000 7ff3292ed640 1 -- start start 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.613+0000 7ff3292ed640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff324072ee0 0x7ff32414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.613+0000 7ff3292ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240c35c0 con 0x7ff32414ef50 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3292ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3240c38a0 con 0x7ff3240c78a0 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3292ed640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3240c4bc0 con 0x7ff324072ee0 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3237fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff324072ee0 0x7ff32414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff31bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff32414ef50 0x7ff3240727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40314/0 (socket says 172.21.15.78:40314) 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3237fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff324072ee0 0x7ff32414f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36126/0 (socket says 172.21.15.78:36126) 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff31bfff640 1 -- 172.21.15.78:0/3992593992 learned_addr learned my addr 172.21.15.78:0/3992593992 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3237fe640 1 -- 172.21.15.78:0/3992593992 >> v1:172.21.15.150:6789/0 conn(0x7ff3240c78a0 legacy=0x7ff3240c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3237fe640 1 -- 172.21.15.78:0/3992593992 >> v1:172.21.15.78:6789/0 conn(0x7ff32414ef50 legacy=0x7ff3240727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.614+0000 7ff3237fe640 1 -- 172.21.15.78:0/3992593992 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240c3b80 con 0x7ff324072ee0 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3237fe640 1 --2- 172.21.15.78:0/3992593992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324072ee0 0x7ff32414f3e0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009d90 tx=0x7ff31c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fd545de966d354 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3227fc640 1 -- 172.21.15.78:0/3992593992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c02cc60 con 0x7ff324072ee0 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3227fc640 1 -- 172.21.15.78:0/3992593992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c02cdc0 con 0x7ff324072ee0 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3227fc640 1 -- 172.21.15.78:0/3992593992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c034660 con 0x7ff324072ee0 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 -- 172.21.15.78:0/3992593992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324072ee0 msgr2=0x7ff32414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3992593992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324072ee0 0x7ff32414f3e0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009d90 tx=0x7ff31c02beb0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 -- 172.21.15.78:0/3992593992 shutdown_connections 2024-02-19T15:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3992593992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324072ee0 0x7ff32414f3e0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 -- 172.21.15.78:0/3992593992 >> 172.21.15.78:0/3992593992 conn(0x7ff324066e70 msgr2=0x7ff3240a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 -- 172.21.15.78:0/3992593992 shutdown_connections 2024-02-19T15:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.615+0000 7ff3292ed640 1 -- 172.21.15.78:0/3992593992 wait complete. 2024-02-19T15:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.616+0000 7ff3292ed640 1 Processor -- start 2024-02-19T15:00:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 -- start start 2024-02-19T15:00:00.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff324072ee0 0x7ff32406c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 0x7ff3240aa750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32414ef50 0x7ff3240ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324149b60 con 0x7ff3240c78a0 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241e8d10 con 0x7ff32414ef50 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff3292ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff324149880 con 0x7ff324072ee0 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff31bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 0x7ff3240aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff31bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 0x7ff3240aa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36128/0 (socket says 172.21.15.78:36128) 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.617+0000 7ff31bfff640 1 -- 172.21.15.78:0/3595470333 learned_addr learned my addr 172.21.15.78:0/3595470333 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31bfff640 1 -- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff324072ee0 msgr2=0x7ff32406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff322ffd640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff324072ee0 0x7ff32406c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff3237fe640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32414ef50 0x7ff3240ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31bfff640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff324072ee0 0x7ff32406c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31bfff640 1 -- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32414ef50 msgr2=0x7ff3240ab3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31bfff640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32414ef50 0x7ff3240ab3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31bfff640 1 -- 172.21.15.78:0/3595470333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c009a70 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff3237fe640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32414ef50 0x7ff3240ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff322ffd640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff324072ee0 0x7ff32406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31bfff640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 0x7ff3240aa750 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c3c0 tx=0x7ff30c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c005ce0 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c005e40 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3241e86c0 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0072d0 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.618+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32406c680 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.619+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3241e8520 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.619+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff30c0074f0 con 0x7ff3240c78a0 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.620+0000 7ff31b7fe640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff30406b810 0x7ff30406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:00.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.620+0000 7ff322ffd640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff30406b810 0x7ff30406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:00.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.620+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff30c089870 con 0x7ff3240c78a0 2024-02-19T15:00:00.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.620+0000 7ff322ffd640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff30406b810 0x7ff30406dcd0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff32415d290 tx=0x7ff314009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:00.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.624+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff3241e8520 con 0x7ff3240c78a0 2024-02-19T15:00:00.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:00.943+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7ff3241e8520 con 0x7ff3240c78a0 2024-02-19T15:00:01.117 INFO:tasks.ceph.mds.a.smithi078.stderr:2024-02-19T15:00:01.115+0000 7f101cd7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:00:01.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.859+0000 7ff31b7fe640 1 -- 172.21.15.78:0/3595470333 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v181) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff3241e8520 con 0x7ff3240c78a0 2024-02-19T15:00:01.862 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked down. 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff30406b810 msgr2=0x7ff30406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff30406b810 0x7ff30406dcd0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff32415d290 tx=0x7ff314009040 comp rx=0 tx=0).stop 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 msgr2=0x7ff3240aa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 0x7ff3240aa750 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00c3c0 tx=0x7ff30c00c880 comp rx=0 tx=0).stop 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 shutdown_connections 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff30406b810 0x7ff30406dcd0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3240c78a0 0x7ff3240aa750 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff32414ef50 0x7ff3240ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 --2- 172.21.15.78:0/3595470333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff324072ee0 0x7ff32406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 >> 172.21.15.78:0/3595470333 conn(0x7ff324066e70 msgr2=0x7ff32414faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 shutdown_connections 2024-02-19T15:00:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.861+0000 7ff3292ed640 1 -- 172.21.15.78:0/3595470333 wait complete. 2024-02-19T15:00:01.876 INFO:tasks.ceph_test_case:waiting 30s for health warning matching 2024-02-19T15:00:01.876 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.971+0000 7f05ff6ae640 1 Processor -- start 2024-02-19T15:00:01.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.971+0000 7f05ff6ae640 1 -- start start 2024-02-19T15:00:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05ff6ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f05f8152a50 0x7f05f8071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05ff6ae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05f80702d0 0x7f05f8158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05ff6ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f8071bf0 con 0x7f05f8152670 2024-02-19T15:00:01.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05ff6ae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f8158a20 con 0x7f05f80702d0 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05ff6ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f80c6780 con 0x7f05f8152a50 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05fdc4b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05f80702d0 0x7f05f8158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05fdc4b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05f80702d0 0x7f05f8158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36144/0 (socket says 172.21.15.78:36144) 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05fdc4b640 1 -- 172.21.15.78:0/336505241 learned_addr learned my addr 172.21.15.78:0/336505241 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05fd44a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f05f8152670 0x7f05f8155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40316/0 (socket says 172.21.15.78:40316) 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.972+0000 7f05fdc4b640 1 -- 172.21.15.78:0/336505241 >> v2:172.21.15.150:3300/0 conn(0x7f05f8152a50 msgr2=0x7f05f8071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05fdc4b640 1 --2- 172.21.15.78:0/336505241 >> v2:172.21.15.150:3300/0 conn(0x7f05f8152a50 0x7f05f8071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05fdc4b640 1 -- 172.21.15.78:0/336505241 >> v1:172.21.15.78:6789/0 conn(0x7f05f8152670 legacy=0x7f05f8155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05fdc4b640 1 -- 172.21.15.78:0/336505241 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f8071630 con 0x7f05f80702d0 2024-02-19T15:00:01.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05f7fff640 1 -- 172.21.15.78:0/336505241 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170979776 0 0) 0x7f05f8071bf0 con 0x7f05f8152670 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05fdc4b640 1 --2- 172.21.15.78:0/336505241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f80702d0 0x7f05f8158eb0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f05f806e770 tx=0x7f05f002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33613f519d83ba0f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05f7fff640 1 -- 172.21.15.78:0/336505241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f002f850 con 0x7f05f80702d0 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05f7fff640 1 -- 172.21.15.78:0/336505241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f002f9b0 con 0x7f05f80702d0 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.973+0000 7f05f7fff640 1 -- 172.21.15.78:0/336505241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f002fd40 con 0x7f05f80702d0 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 -- 172.21.15.78:0/336505241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f80702d0 msgr2=0x7f05f8158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/336505241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f80702d0 0x7f05f8158eb0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f05f806e770 tx=0x7f05f002ba30 comp rx=0 tx=0).stop 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 -- 172.21.15.78:0/336505241 shutdown_connections 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/336505241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f80702d0 0x7f05f8158eb0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/336505241 >> v2:172.21.15.150:3300/0 conn(0x7f05f8152a50 0x7f05f8071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 -- 172.21.15.78:0/336505241 >> 172.21.15.78:0/336505241 conn(0x7f05f80c85e0 msgr2=0x7f05f80c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 -- 172.21.15.78:0/336505241 shutdown_connections 2024-02-19T15:00:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.974+0000 7f05ff6ae640 1 -- 172.21.15.78:0/336505241 wait complete. 2024-02-19T15:00:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.975+0000 7f05ff6ae640 1 Processor -- start 2024-02-19T15:00:01.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.975+0000 7f05ff6ae640 1 -- start start 2024-02-19T15:00:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05ff6ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05f80702d0 0x7f05f81ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05ff6ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05f8152670 0x7f05f8153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05ff6ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 0x7f05f81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05ff6ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f8153890 con 0x7f05f8152a50 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05ff6ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f81def40 con 0x7f05f8152670 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05ff6ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f81eec10 con 0x7f05f80702d0 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 0x7f05f81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 0x7f05f81542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36146/0 (socket says 172.21.15.78:36146) 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 -- 172.21.15.78:0/1673547633 learned_addr learned my addr 172.21.15.78:0/1673547633 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fcc49640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05f8152670 0x7f05f8153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fd44a640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05f80702d0 0x7f05f81ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 -- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05f80702d0 msgr2=0x7f05f81ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05f80702d0 0x7f05f81ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 -- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05f8152670 msgr2=0x7f05f8153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05f8152670 0x7f05f8153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 -- 172.21.15.78:0/1673547633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f00094d0 con 0x7f05f8152a50 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fcc49640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05f8152670 0x7f05f8153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:01.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.976+0000 7f05fdc4b640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 0x7f05f81542b0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f05f00090b0 tx=0x7f05f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.977+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f003c040 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.977+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0037dd0 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.977+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0004270 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.977+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f80ab8e0 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.977+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f80abea0 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.978+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f80676a0 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.978+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f05f00043d0 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.979+0000 7f05f5ffb640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05e406b810 0x7f05e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.979+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f05f00ac9a0 con 0x7f05f8152a50 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.979+0000 7f05fd44a640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05e406b810 0x7f05e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.979+0000 7f05fd44a640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05e406b810 0x7f05e406dcd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f05e80012e0 tx=0x7f05e8007ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:01.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:01.983+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05f00772c0 con 0x7f05f8152a50 2024-02-19T15:00:02.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.191+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f05f8153a60 con 0x7f05f8152a50 2024-02-19T15:00:02.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.191+0000 7f05f5ffb640 1 -- 172.21.15.78:0/1673547633 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f05f0047080 con 0x7f05f8152a50 2024-02-19T15:00:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:02.195 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05e406b810 msgr2=0x7f05e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05e406b810 0x7f05e406dcd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f05e80012e0 tx=0x7f05e8007ed0 comp rx=0 tx=0).stop 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 msgr2=0x7f05f81542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 0x7f05f81542b0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f05f00090b0 tx=0x7f05f0004730 comp rx=0 tx=0).stop 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 shutdown_connections 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f05e406b810 0x7f05e406dcd0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05f8152a50 0x7f05f81542b0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05f8152670 0x7f05f8153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 --2- 172.21.15.78:0/1673547633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05f80702d0 0x7f05f81ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.193+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 >> 172.21.15.78:0/1673547633 conn(0x7f05f80c85e0 msgr2=0x7f05f8156800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.194+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 shutdown_connections 2024-02-19T15:00:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:02.194+0000 7f05ff6ae640 1 -- 172.21.15.78:0/1673547633 wait complete. 2024-02-19T15:00:02.207 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:02.207 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T15:00:07.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:07.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.322+0000 7f52665a2640 1 Processor -- start 2024-02-19T15:00:07.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.323+0000 7f52665a2640 1 -- start start 2024-02-19T15:00:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.323+0000 7f52665a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 0x7f526006d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f52665a2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5260155e00 con 0x7f52600c7150 2024-02-19T15:00:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f52665a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52600c7580 con 0x7f52601559c0 2024-02-19T15:00:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f52665a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52600c8d50 con 0x7f52601524b0 2024-02-19T15:00:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5265da1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f52600c7150 0x7f52600c50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40326/0 (socket says 172.21.15.78:40326) 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5265da1640 1 -- 172.21.15.78:0/3618234281 learned_addr learned my addr 172.21.15.78:0/3618234281 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f52655a0640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 0x7f526006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571236632 0 0) 0x7f5260155e00 con 0x7f52600c7150 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f524c003520 con 0x7f52600c7150 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047968755 0 0) 0x7f52600c7580 con 0x7f52601559c0 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260155e00 con 0x7f52601559c0 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1235288811 0 0) 0x7f524c003520 con 0x7f52600c7150 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 msgr2=0x7f526006d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 0x7f526006d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 >> v1:172.21.15.150:6789/0 conn(0x7f52601559c0 legacy=0x7f52600c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52600c7850 con 0x7f52600c7150 2024-02-19T15:00:07.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f52580019f0 con 0x7f52600c7150 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 >> v1:172.21.15.78:6789/0 conn(0x7f52600c7150 legacy=0x7f52600c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 0x7f524c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f524c004090 0x7f524c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004d00 0x7f524c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52600c8d50 con 0x7f524c004090 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c00bb10 con 0x7f524c004d00 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.324+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c00bca0 con 0x7f52601524b0 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5264d9f640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f524c004090 0x7f524c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f52655a0640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 0x7f524c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5265da1640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004d00 0x7f524c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5264d9f640 1 -- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 msgr2=0x7f524c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5264d9f640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52601524b0 0x7f524c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5264d9f640 1 -- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004d00 msgr2=0x7f524c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5264d9f640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004d00 0x7f524c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.325+0000 7f5264d9f640 1 -- 172.21.15.78:0/3618234281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c003520 con 0x7f524c004090 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f5264d9f640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f524c004090 0x7f524c0047c0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f525c0018d0 tx=0x7f524c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1306babdf1a949a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0052d0 con 0x7f524c004090 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c005430 con 0x7f524c004090 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f5257fff640 1 -- 172.21.15.78:0/3618234281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c003810 con 0x7f524c004090 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f52665a2640 1 -- 172.21.15.78:0/3618234281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f524c004090 msgr2=0x7f524c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f52665a2640 1 --2- 172.21.15.78:0/3618234281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f524c004090 0x7f524c0047c0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f525c0018d0 tx=0x7f524c0057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f5265da1640 1 -- 172.21.15.78:0/3618234281 reap_dead start 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f52665a2640 1 -- 172.21.15.78:0/3618234281 shutdown_connections 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f52665a2640 1 -- 172.21.15.78:0/3618234281 >> 172.21.15.78:0/3618234281 conn(0x7f52600ac250 msgr2=0x7f52600ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.326+0000 7f52665a2640 1 -- 172.21.15.78:0/3618234281 shutdown_connections 2024-02-19T15:00:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.327+0000 7f52665a2640 1 -- 172.21.15.78:0/3618234281 wait complete. 2024-02-19T15:00:07.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 Processor -- start 2024-02-19T15:00:07.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 -- start start 2024-02-19T15:00:07.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004090 0x7f526014e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 0x7f52601541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5260147920 0x7f5260147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526014f550 con 0x7f526014d740 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526014cb10 con 0x7f524c004090 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f52665a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52600c4600 con 0x7f5260147920 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f5264d9f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 0x7f52601541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f5264d9f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 0x7f52601541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36172/0 (socket says 172.21.15.78:36172) 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.328+0000 7f5264d9f640 1 -- 172.21.15.78:0/1288650594 learned_addr learned my addr 172.21.15.78:0/1288650594 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5265da1640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5260147920 0x7f5260147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f52655a0640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004090 0x7f526014e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5264d9f640 1 -- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5260147920 msgr2=0x7f5260147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5264d9f640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5260147920 0x7f5260147da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5264d9f640 1 -- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004090 msgr2=0x7f526014e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5264d9f640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004090 0x7f526014e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5264d9f640 1 -- 172.21.15.78:0/1288650594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526014e970 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f52655a0640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004090 0x7f526014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5264d9f640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 0x7f52601541a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f525c0019f0 tx=0x7f525c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c006ea0 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c007450 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0035f0 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52601482e0 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.329+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526014f170 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.331+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526014efb0 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.331+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f525c003750 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.331+0000 7f5255ffb640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f524406b810 0x7f524406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.332+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f525c0ac090 con 0x7f526014d740 2024-02-19T15:00:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.332+0000 7f52655a0640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f524406b810 0x7f524406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.332+0000 7f52655a0640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f524406b810 0x7f524406dcd0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5250009ae0 tx=0x7f5250008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.336+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f525c076a60 con 0x7f526014d740 2024-02-19T15:00:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.545+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5260074990 con 0x7f526014d740 2024-02-19T15:00:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.546+0000 7f5255ffb640 1 -- 172.21.15.78:0/1288650594 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f525c04b040 con 0x7f526014d740 2024-02-19T15:00:07.548 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:07.549 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.547+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f524406b810 msgr2=0x7f524406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.547+0000 7f52665a2640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f524406b810 0x7f524406dcd0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5250009ae0 tx=0x7f5250008040 comp rx=0 tx=0).stop 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.547+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 msgr2=0x7f52601541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.547+0000 7f52665a2640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 0x7f52601541a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f525c0019f0 tx=0x7f525c0012d0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 shutdown_connections 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f524406b810 0x7f524406dcd0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f526014d740 0x7f52601541a0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5260147920 0x7f5260147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 --2- 172.21.15.78:0/1288650594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f524c004090 0x7f526014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 >> 172.21.15.78:0/1288650594 conn(0x7f52600ac250 msgr2=0x7f526006d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 shutdown_connections 2024-02-19T15:00:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:07.548+0000 7f52665a2640 1 -- 172.21.15.78:0/1288650594 wait complete. 2024-02-19T15:00:07.561 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:07.561 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T15:00:12.562 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:12.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.668+0000 7efc473ac640 1 Processor -- start 2024-02-19T15:00:12.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.669+0000 7efc473ac640 1 -- start start 2024-02-19T15:00:12.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.669+0000 7efc473ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc4006c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.669+0000 7efc473ac640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400717b0 con 0x7efc40156050 2024-02-19T15:00:12.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.669+0000 7efc473ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40071a00 con 0x7efc400c6a50 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc473ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c7560 con 0x7efc400c2f70 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc45949640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efc40156050 0x7efc4006cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47690/0 (socket says 172.21.15.78:47690) 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc45949640 1 -- 172.21.15.78:0/313740322 learned_addr learned my addr 172.21.15.78:0/313740322 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc44947640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc4006c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474547924 0 0) 0x7efc400717b0 con 0x7efc40156050 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c003520 con 0x7efc40156050 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370923478 0 0) 0x7efc2c003520 con 0x7efc40156050 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 msgr2=0x7efc4006c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc4006c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 >> v1:172.21.15.150:6789/0 conn(0x7efc400c6a50 legacy=0x7efc40155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400c8030 con 0x7efc40156050 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7efc380019f0 con 0x7efc40156050 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 >> v1:172.21.15.78:6789/0 conn(0x7efc40156050 legacy=0x7efc4006cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc2c0039a0 0x7efc2c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc2c004d00 0x7efc2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c7560 con 0x7efc2c0039a0 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c00ba30 con 0x7efc2c004d00 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c00bbf0 con 0x7efc400c2f70 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc45148640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc2c0039a0 0x7efc2c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.670+0000 7efc44947640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc45949640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc2c004d00 0x7efc2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc44947640 1 -- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc2c004d00 msgr2=0x7efc2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:12.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc44947640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc2c004d00 0x7efc2c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc44947640 1 -- 172.21.15.78:0/313740322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc2c0039a0 msgr2=0x7efc2c003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc44947640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc2c0039a0 0x7efc2c003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc44947640 1 -- 172.21.15.78:0/313740322 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c003520 con 0x7efc400c2f70 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.671+0000 7efc45148640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc2c0039a0 0x7efc2c003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc44947640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc2c0047c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efc400c3350 tx=0x7efc2c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffe28bece51a921e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c02c680 con 0x7efc400c2f70 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c02c7e0 con 0x7efc400c2f70 2024-02-19T15:00:12.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc37fff640 1 -- 172.21.15.78:0/313740322 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c02cb70 con 0x7efc400c2f70 2024-02-19T15:00:12.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc473ac640 1 -- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 msgr2=0x7efc2c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc473ac640 1 --2- 172.21.15.78:0/313740322 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c2f70 0x7efc2c0047c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efc400c3350 tx=0x7efc2c0057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc45949640 1 -- 172.21.15.78:0/313740322 reap_dead start 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc473ac640 1 -- 172.21.15.78:0/313740322 shutdown_connections 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc473ac640 1 -- 172.21.15.78:0/313740322 >> 172.21.15.78:0/313740322 conn(0x7efc4006b700 msgr2=0x7efc400aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc473ac640 1 -- 172.21.15.78:0/313740322 shutdown_connections 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.672+0000 7efc473ac640 1 -- 172.21.15.78:0/313740322 wait complete. 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.673+0000 7efc473ac640 1 Processor -- start 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 -- start start 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2f70 0x7efc4014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc401590f0 0x7efc400ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 0x7efc4014d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4014dee0 con 0x7efc40158ba0 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc401ee560 con 0x7efc400c2f70 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc473ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc401ee9f0 con 0x7efc401590f0 2024-02-19T15:00:12.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc45949640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 0x7efc4014d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc45949640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 0x7efc4014d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36378/0 (socket says 172.21.15.78:36378) 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc45949640 1 -- 172.21.15.78:0/1243764188 learned_addr learned my addr 172.21.15.78:0/1243764188 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc45148640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2f70 0x7efc4014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.674+0000 7efc44947640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc401590f0 0x7efc400ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc45949640 1 -- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc401590f0 msgr2=0x7efc400ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc45949640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc401590f0 0x7efc400ac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc45949640 1 -- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2f70 msgr2=0x7efc4014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc45949640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2f70 0x7efc4014efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc45949640 1 -- 172.21.15.78:0/1243764188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc40070b30 con 0x7efc40158ba0 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc44947640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc401590f0 0x7efc400ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:12.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc45949640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 0x7efc4014d450 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7efc40156460 tx=0x7efc38007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3800aa80 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3800abe0 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc40158650 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380133f0 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.675+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc40158250 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.676+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc40158410 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.677+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efc38013550 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.677+0000 7efc35ffb640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc2406b810 0x7efc2406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.677+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efc3808a680 con 0x7efc40158ba0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.677+0000 7efc45148640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc2406b810 0x7efc2406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:12.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.677+0000 7efc45148640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc2406b810 0x7efc2406dcd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7efc30000c00 tx=0x7efc30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:12.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.681+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efc40158410 con 0x7efc40158ba0 2024-02-19T15:00:12.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.890+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc401583e0 con 0x7efc40158ba0 2024-02-19T15:00:12.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.890+0000 7efc35ffb640 1 -- 172.21.15.78:0/1243764188 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7efc38054fa0 con 0x7efc40158ba0 2024-02-19T15:00:12.893 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:12.893 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:12.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc2406b810 msgr2=0x7efc2406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc2406b810 0x7efc2406dcd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7efc30000c00 tx=0x7efc30009040 comp rx=0 tx=0).stop 2024-02-19T15:00:12.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 msgr2=0x7efc4014d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 0x7efc4014d450 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7efc40156460 tx=0x7efc38007b20 comp rx=0 tx=0).stop 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 shutdown_connections 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efc2406b810 0x7efc2406dcd0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc40158ba0 0x7efc4014d450 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc401590f0 0x7efc400ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 --2- 172.21.15.78:0/1243764188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2f70 0x7efc4014efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 >> 172.21.15.78:0/1243764188 conn(0x7efc4006b700 msgr2=0x7efc4006c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 shutdown_connections 2024-02-19T15:00:12.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:12.892+0000 7efc473ac640 1 -- 172.21.15.78:0/1243764188 wait complete. 2024-02-19T15:00:12.906 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:12.906 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T15:00:15.897 INFO:tasks.ceph.mds.a.smithi078.stderr: -602> 2024-02-19T15:00:01.115+0000 7f101cd7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:00:15.923 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T15:00:17.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:18.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.012+0000 7f75cd5a2640 1 Processor -- start 2024-02-19T15:00:18.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.012+0000 7f75cd5a2640 1 -- start start 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75cd5a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f75c806b0f0 0x7f75c814eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75cd5a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75c80c2350 0x7f75c8152360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75cd5a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c8155250 con 0x7f75c80c1ed0 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75cd5a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c80a9670 con 0x7f75c80c2350 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75cd5a2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c8155530 con 0x7f75c806b0f0 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75c80c2350 0x7f75c8152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:18.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75c80c2350 0x7f75c8152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43974/0 (socket says 172.21.15.78:43974) 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 -- 172.21.15.78:0/3519064378 learned_addr learned my addr 172.21.15.78:0/3519064378 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75c7fff640 1 --2- 172.21.15.78:0/3519064378 >> v2:172.21.15.150:3300/0 conn(0x7f75c806b0f0 0x7f75c814eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 -- 172.21.15.78:0/3519064378 >> v2:172.21.15.150:3300/0 conn(0x7f75c806b0f0 msgr2=0x7f75c814eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 --2- 172.21.15.78:0/3519064378 >> v2:172.21.15.150:3300/0 conn(0x7f75c806b0f0 0x7f75c814eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 -- 172.21.15.78:0/3519064378 >> v1:172.21.15.150:6789/0 conn(0x7f75c80c1ed0 legacy=0x7f75c8151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.013+0000 7f75ccda1640 1 -- 172.21.15.78:0/3519064378 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8155810 con 0x7f75c80c2350 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.014+0000 7f75ccda1640 1 --2- 172.21.15.78:0/3519064378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8152360 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f75c80c7180 tx=0x7f75c002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f9a5ddce0d0404 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.014+0000 7f75c6ffd640 1 -- 172.21.15.78:0/3519064378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c002c9f0 con 0x7f75c80c2350 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.014+0000 7f75c6ffd640 1 -- 172.21.15.78:0/3519064378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c002cb50 con 0x7f75c80c2350 2024-02-19T15:00:18.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.014+0000 7f75c6ffd640 1 -- 172.21.15.78:0/3519064378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00343f0 con 0x7f75c80c2350 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3519064378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 msgr2=0x7f75c8152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3519064378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8152360 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f75c80c7180 tx=0x7f75c002be50 comp rx=0 tx=0).stop 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3519064378 shutdown_connections 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3519064378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8152360 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3519064378 >> v2:172.21.15.150:3300/0 conn(0x7f75c806b0f0 0x7f75c814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3519064378 >> 172.21.15.78:0/3519064378 conn(0x7f75c80c6410 msgr2=0x7f75c80c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3519064378 shutdown_connections 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.015+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3519064378 wait complete. 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.016+0000 7f75cd5a2640 1 Processor -- start 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.016+0000 7f75cd5a2640 1 -- start start 2024-02-19T15:00:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75cd5a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75c806b0f0 0x7f75c81eac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75cd5a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c80c1ed0 0x7f75c814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75cd5a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75cd5a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c8146d00 con 0x7f75c80c2350 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75cd5a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81ea1c0 con 0x7f75c80c1ed0 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75cd5a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81ee1d0 con 0x7f75c806b0f0 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43976/0 (socket says 172.21.15.78:43976) 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 -- 172.21.15.78:0/3977345138 learned_addr learned my addr 172.21.15.78:0/3977345138 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75c77fe640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c80c1ed0 0x7f75c814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75c7fff640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75c806b0f0 0x7f75c81eac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 -- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75c806b0f0 msgr2=0x7f75c81eac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75c806b0f0 0x7f75c81eac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 -- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c80c1ed0 msgr2=0x7f75c814c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c80c1ed0 0x7f75c814c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 -- 172.21.15.78:0/3977345138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c0009b30 con 0x7f75c80c2350 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75c7fff640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75c806b0f0 0x7f75c81eac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.017+0000 7f75ccda1640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8153f20 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036040 tx=0x7f75c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.018+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0009510 con 0x7f75c80c2350 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.018+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c003e450 con 0x7f75c80c2350 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.018+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c81ea460 con 0x7f75c80c2350 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.018+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c003e7e0 con 0x7f75c80c2350 2024-02-19T15:00:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.018+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81ee4b0 con 0x7f75c80c2350 2024-02-19T15:00:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.019+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c80a9db0 con 0x7f75c80c2350 2024-02-19T15:00:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.019+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f75c003e940 con 0x7f75c80c2350 2024-02-19T15:00:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.019+0000 7f75c4ff9640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f75b406b810 0x7f75b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.019+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f75c00ad000 con 0x7f75c80c2350 2024-02-19T15:00:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.019+0000 7f75c7fff640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f75b406b810 0x7f75b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.020+0000 7f75c7fff640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f75b406b810 0x7f75b406dcd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f75b8007a50 tx=0x7f75b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:18.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.024+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f75c0077920 con 0x7f75c80c2350 2024-02-19T15:00:18.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.230+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f75c81ea980 con 0x7f75c80c2350 2024-02-19T15:00:18.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.231+0000 7f75c4ff9640 1 -- 172.21.15.78:0/3977345138 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f75c81ea980 con 0x7f75c80c2350 2024-02-19T15:00:18.234 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f75b406b810 msgr2=0x7f75b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f75b406b810 0x7f75b406dcd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f75b8007a50 tx=0x7f75b8008040 comp rx=0 tx=0).stop 2024-02-19T15:00:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 msgr2=0x7f75c8153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8153f20 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036040 tx=0x7f75c0004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 shutdown_connections 2024-02-19T15:00:18.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f75b406b810 0x7f75b406dcd0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75c80c2350 0x7f75c8153f20 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c80c1ed0 0x7f75c814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 --2- 172.21.15.78:0/3977345138 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f75c806b0f0 0x7f75c81eac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:18.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 >> 172.21.15.78:0/3977345138 conn(0x7f75c80c6410 msgr2=0x7f75c80c00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:18.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 shutdown_connections 2024-02-19T15:00:18.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:18.233+0000 7f75cd5a2640 1 -- 172.21.15.78:0/3977345138 wait complete. 2024-02-19T15:00:18.246 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:18.246 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T15:00:23.247 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:23.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.354+0000 7f44622b5640 1 Processor -- start 2024-02-19T15:00:23.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.354+0000 7f44622b5640 1 -- start start 2024-02-19T15:00:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f44622b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f445c06d550 0x7f445c0c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:23.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f44622b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f445c1588a0 0x7f445c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f44622b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c0c6af0 con 0x7f445c155f90 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f44622b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0c7270 con 0x7f445c06d550 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f44622b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0c7940 con 0x7f445c1588a0 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f445c155f90 0x7f445c0c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46678/0 (socket says 172.21.15.78:46678) 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445bfff640 1 -- 172.21.15.78:0/665667296 learned_addr learned my addr 172.21.15.78:0/665667296 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445b7fe640 1 --2- 172.21.15.78:0/665667296 >> v2:172.21.15.78:3300/0 conn(0x7f445c06d550 0x7f445c0c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445b7fe640 1 -- 172.21.15.78:0/665667296 >> v2:172.21.15.150:3300/0 conn(0x7f445c1588a0 msgr2=0x7f445c158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445b7fe640 1 --2- 172.21.15.78:0/665667296 >> v2:172.21.15.150:3300/0 conn(0x7f445c1588a0 0x7f445c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445b7fe640 1 -- 172.21.15.78:0/665667296 >> v1:172.21.15.78:6789/0 conn(0x7f445c155f90 legacy=0x7f445c0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445b7fe640 1 -- 172.21.15.78:0/665667296 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c0c7c20 con 0x7f445c06d550 2024-02-19T15:00:23.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.355+0000 7f445affd640 1 -- 172.21.15.78:0/665667296 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634035975 0 0) 0x7f445c0c6af0 con 0x7f445c155f90 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.356+0000 7f445b7fe640 1 --2- 172.21.15.78:0/665667296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0c6d30 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f445c0c6560 tx=0x7f445002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb117c54a5e1065 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.356+0000 7f445affd640 1 -- 172.21.15.78:0/665667296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445002e070 con 0x7f445c06d550 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.356+0000 7f445affd640 1 -- 172.21.15.78:0/665667296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44500315a0 con 0x7f445c06d550 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.356+0000 7f445affd640 1 -- 172.21.15.78:0/665667296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450031930 con 0x7f445c06d550 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 -- 172.21.15.78:0/665667296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 msgr2=0x7f445c0c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 --2- 172.21.15.78:0/665667296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0c6d30 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f445c0c6560 tx=0x7f445002b840 comp rx=0 tx=0).stop 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 -- 172.21.15.78:0/665667296 shutdown_connections 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 --2- 172.21.15.78:0/665667296 >> v2:172.21.15.150:3300/0 conn(0x7f445c1588a0 0x7f445c158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 --2- 172.21.15.78:0/665667296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0c6d30 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 -- 172.21.15.78:0/665667296 >> 172.21.15.78:0/665667296 conn(0x7f445c06b700 msgr2=0x7f445c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 -- 172.21.15.78:0/665667296 shutdown_connections 2024-02-19T15:00:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.357+0000 7f44622b5640 1 -- 172.21.15.78:0/665667296 wait complete. 2024-02-19T15:00:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.358+0000 7f44622b5640 1 Processor -- start 2024-02-19T15:00:23.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.358+0000 7f44622b5640 1 -- start start 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f44622b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f44622b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c155f90 0x7f445c151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f44622b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c1588a0 0x7f445c1edae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f44622b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1eb4c0 con 0x7f445c06d550 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f44622b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1eaf90 con 0x7f445c155f90 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f44622b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1eb1e0 con 0x7f445c1588a0 2024-02-19T15:00:23.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43984/0 (socket says 172.21.15.78:43984) 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 -- 172.21.15.78:0/2142923155 learned_addr learned my addr 172.21.15.78:0/2142923155 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445b7fe640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c155f90 0x7f445c151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f4460852640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c1588a0 0x7f445c1edae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 -- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c1588a0 msgr2=0x7f445c1edae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c1588a0 0x7f445c1edae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 -- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c155f90 msgr2=0x7f445c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c155f90 0x7f445c151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.359+0000 7f445bfff640 1 -- 172.21.15.78:0/2142923155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009400 con 0x7f445c06d550 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.360+0000 7f445bfff640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0ad6c0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c4d0 tx=0x7f444c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.360+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c0074e0 con 0x7f445c06d550 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.360+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c007660 con 0x7f445c06d550 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.360+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c0abe10 con 0x7f445c06d550 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.360+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c004a20 con 0x7f445c06d550 2024-02-19T15:00:23.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.360+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c070620 con 0x7f445c06d550 2024-02-19T15:00:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.361+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c0abc70 con 0x7f445c06d550 2024-02-19T15:00:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.361+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f444c004b80 con 0x7f445c06d550 2024-02-19T15:00:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.362+0000 7f4458ff9640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444806b810 0x7f444806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.362+0000 7f445b7fe640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444806b810 0x7f444806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.362+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f444c089870 con 0x7f445c06d550 2024-02-19T15:00:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.362+0000 7f445b7fe640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444806b810 0x7f444806dcd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4450036000 tx=0x7f445003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:23.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.366+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f445c0abc70 con 0x7f445c06d550 2024-02-19T15:00:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.576+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f445c0abc70 con 0x7f445c06d550 2024-02-19T15:00:23.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.577+0000 7f4458ff9640 1 -- 172.21.15.78:0/2142923155 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f445c0abc70 con 0x7f445c06d550 2024-02-19T15:00:23.581 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:23.581 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:23.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.578+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444806b810 msgr2=0x7f444806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.578+0000 7f44622b5640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444806b810 0x7f444806dcd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4450036000 tx=0x7f445003d040 comp rx=0 tx=0).stop 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.578+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 msgr2=0x7f445c0ad6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.578+0000 7f44622b5640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0ad6c0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c4d0 tx=0x7f444c00c990 comp rx=0 tx=0).stop 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.578+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 shutdown_connections 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.578+0000 7f44622b5640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444806b810 0x7f444806dcd0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.579+0000 7f44622b5640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c1588a0 0x7f445c1edae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.579+0000 7f44622b5640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c155f90 0x7f445c151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.579+0000 7f44622b5640 1 --2- 172.21.15.78:0/2142923155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c06d550 0x7f445c0ad6c0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.579+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 >> 172.21.15.78:0/2142923155 conn(0x7f445c06b700 msgr2=0x7f445c0aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.579+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 shutdown_connections 2024-02-19T15:00:23.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:23.579+0000 7f44622b5640 1 -- 172.21.15.78:0/2142923155 wait complete. 2024-02-19T15:00:23.593 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:23.593 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T15:00:27.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:00:27.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:00:28.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:28.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.703+0000 7f271e4ae640 1 Processor -- start 2024-02-19T15:00:28.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.703+0000 7f271e4ae640 1 -- start start 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.703+0000 7f271e4ae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f271806c9b0 0x7f2718160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.703+0000 7f271e4ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27181551f0 con 0x7f2718152700 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.703+0000 7f271e4ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2718155490 con 0x7f27180c8e00 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.703+0000 7f271e4ae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2718155770 con 0x7f271806c9b0 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f271ca4b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f271806c9b0 0x7f2718160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f2717fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2718152700 0x7f27180c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41776/0 (socket says 172.21.15.78:41776) 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f271ca4b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f271806c9b0 0x7f2718160990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46112/0 (socket says 172.21.15.78:46112) 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f2717fff640 1 -- 172.21.15.78:0/771528888 learned_addr learned my addr 172.21.15.78:0/771528888 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f271ca4b640 1 -- 172.21.15.78:0/771528888 >> v1:172.21.15.150:6789/0 conn(0x7f27180c8e00 legacy=0x7f271806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f271ca4b640 1 -- 172.21.15.78:0/771528888 >> v1:172.21.15.78:6789/0 conn(0x7f2718152700 legacy=0x7f27180c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f271ca4b640 1 -- 172.21.15.78:0/771528888 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2718155a50 con 0x7f271806c9b0 2024-02-19T15:00:28.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.704+0000 7f2716ffd640 1 -- 172.21.15.78:0/771528888 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759308776 0 0) 0x7f27181551f0 con 0x7f2718152700 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271ca4b640 1 --2- 172.21.15.78:0/771528888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f271806c9b0 0x7f2718160990 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f2710009d90 tx=0x7f271002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f934424dc71f9eb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f2716ffd640 1 -- 172.21.15.78:0/771528888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271002cc60 con 0x7f271806c9b0 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f2716ffd640 1 -- 172.21.15.78:0/771528888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271002cdc0 con 0x7f271806c9b0 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f2716ffd640 1 -- 172.21.15.78:0/771528888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710034660 con 0x7f271806c9b0 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 -- 172.21.15.78:0/771528888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f271806c9b0 msgr2=0x7f2718160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 --2- 172.21.15.78:0/771528888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f271806c9b0 0x7f2718160990 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f2710009d90 tx=0x7f271002beb0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 -- 172.21.15.78:0/771528888 shutdown_connections 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 --2- 172.21.15.78:0/771528888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f271806c9b0 0x7f2718160990 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 -- 172.21.15.78:0/771528888 >> 172.21.15.78:0/771528888 conn(0x7f27180cac90 msgr2=0x7f27180cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 -- 172.21.15.78:0/771528888 shutdown_connections 2024-02-19T15:00:28.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.705+0000 7f271e4ae640 1 -- 172.21.15.78:0/771528888 wait complete. 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 Processor -- start 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 -- start start 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f271806c9b0 0x7f2718156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27180c8e00 0x7f27181572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 0x7f2718144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718156990 con 0x7f2718152700 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718158f50 con 0x7f27180c8e00 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.707+0000 7f271e4ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2718159230 con 0x7f271806c9b0 2024-02-19T15:00:28.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 0x7f2718144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 0x7f2718144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46120/0 (socket says 172.21.15.78:46120) 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 -- 172.21.15.78:0/3100885827 learned_addr learned my addr 172.21.15.78:0/3100885827 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 -- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f271806c9b0 msgr2=0x7f2718156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f271806c9b0 0x7f2718156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f27177fe640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27180c8e00 0x7f27181572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 -- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27180c8e00 msgr2=0x7f27181572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27180c8e00 0x7f27181572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 -- 172.21.15.78:0/3100885827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710009a70 con 0x7f2718152700 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f27177fe640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27180c8e00 0x7f27181572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271ca4b640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 0x7f2718144ee0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f2710009d90 tx=0x7f2710004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271003e610 con 0x7f2718152700 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271003e790 con 0x7f2718152700 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2718154190 con 0x7f2718152700 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271003eb40 con 0x7f2718152700 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.708+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2718154750 con 0x7f2718152700 2024-02-19T15:00:28.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.709+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2718154940 con 0x7f2718152700 2024-02-19T15:00:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.710+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f271003b050 con 0x7f2718152700 2024-02-19T15:00:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.710+0000 7f2714ff9640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b810 0x7f270406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.710+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f27100acd50 con 0x7f2718152700 2024-02-19T15:00:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.710+0000 7f2717fff640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b810 0x7f270406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:28.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.711+0000 7f2717fff640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b810 0x7f270406dcd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2708007fd0 tx=0x7f2708007960 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.715+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f27100b0030 con 0x7f2718152700 2024-02-19T15:00:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.923+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2718074990 con 0x7f2718152700 2024-02-19T15:00:28.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.923+0000 7f2714ff9640 1 -- 172.21.15.78:0/3100885827 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2710077670 con 0x7f2718152700 2024-02-19T15:00:28.926 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:28.926 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:28.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b810 msgr2=0x7f270406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:28.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b810 0x7f270406dcd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2708007fd0 tx=0x7f2708007960 comp rx=0 tx=0).stop 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 msgr2=0x7f2718144ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 0x7f2718144ee0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f2710009d90 tx=0x7f2710004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 shutdown_connections 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f270406b810 0x7f270406dcd0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2718152700 0x7f2718144ee0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27180c8e00 0x7f27181572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 --2- 172.21.15.78:0/3100885827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f271806c9b0 0x7f2718156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.925+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 >> 172.21.15.78:0/3100885827 conn(0x7f27180cac90 msgr2=0x7f27180cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.926+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 shutdown_connections 2024-02-19T15:00:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:28.926+0000 7f271e4ae640 1 -- 172.21.15.78:0/3100885827 wait complete. 2024-02-19T15:00:28.939 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:28.939 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-19T15:00:33.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-19T15:00:34.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.051+0000 7f49b2ea4640 1 Processor -- start 2024-02-19T15:00:34.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b2ea4640 1 -- start start 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b2ea4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b2ea4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49ac0705b0 0x7f49ac0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b2ea4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0ae670 con 0x7f49ac0c2f60 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b2ea4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac0716b0 con 0x7f49ac0705b0 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b2ea4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c33a0 con 0x7f49ac0add70 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b1441640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49ac0705b0 0x7f49ac0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b1441640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f49ac0705b0 0x7f49ac0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46136/0 (socket says 172.21.15.78:46136) 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.052+0000 7f49b1441640 1 -- 172.21.15.78:0/3715310120 learned_addr learned my addr 172.21.15.78:0/3715310120 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.053+0000 7f49b0c40640 1 --2- 172.21.15.78:0/3715310120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.053+0000 7f49b1441640 1 -- 172.21.15.78:0/3715310120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 msgr2=0x7f49ac0c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:34.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.053+0000 7f49b1441640 1 --2- 172.21.15.78:0/3715310120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.053+0000 7f49b1441640 1 -- 172.21.15.78:0/3715310120 >> v1:172.21.15.150:6789/0 conn(0x7f49ac0c2f60 legacy=0x7f49ac06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:34.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.053+0000 7f49b1441640 1 -- 172.21.15.78:0/3715310120 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac153780 con 0x7f49ac0705b0 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b1441640 1 --2- 172.21.15.78:0/3715310120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac0709c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009e50 tx=0x7f49a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e22b2187765605 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49ab7fe640 1 -- 172.21.15.78:0/3715310120 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a402c9e0 con 0x7f49ac0705b0 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49ab7fe640 1 -- 172.21.15.78:0/3715310120 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a402cb40 con 0x7f49ac0705b0 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49ab7fe640 1 -- 172.21.15.78:0/3715310120 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40343f0 con 0x7f49ac0705b0 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 -- 172.21.15.78:0/3715310120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 msgr2=0x7f49ac0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/3715310120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac0709c0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009e50 tx=0x7f49a402be40 comp rx=0 tx=0).stop 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 -- 172.21.15.78:0/3715310120 shutdown_connections 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/3715310120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac0709c0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/3715310120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 -- 172.21.15.78:0/3715310120 >> 172.21.15.78:0/3715310120 conn(0x7f49ac0c85e0 msgr2=0x7f49ac0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 -- 172.21.15.78:0/3715310120 shutdown_connections 2024-02-19T15:00:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.054+0000 7f49b2ea4640 1 -- 172.21.15.78:0/3715310120 wait complete. 2024-02-19T15:00:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.055+0000 7f49b2ea4640 1 Processor -- start 2024-02-19T15:00:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.055+0000 7f49b2ea4640 1 -- start start 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b2ea4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b2ea4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b2ea4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49ac0c2f60 0x7f49ac156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b2ea4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac155a70 con 0x7f49ac0705b0 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b2ea4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac152530 con 0x7f49ac0c2f60 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b2ea4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1527d0 con 0x7f49ac0add70 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b0c40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b0c40640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac153e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46150/0 (socket says 172.21.15.78:46150) 2024-02-19T15:00:34.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.056+0000 7f49b0c40640 1 -- 172.21.15.78:0/2657723877 learned_addr learned my addr 172.21.15.78:0/2657723877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b0c40640 1 -- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 msgr2=0x7f49ac0c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b1441640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49ac0c2f60 0x7f49ac156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49abfff640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b0c40640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b0c40640 1 -- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49ac0c2f60 msgr2=0x7f49ac156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b0c40640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49ac0c2f60 0x7f49ac156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b0c40640 1 -- 172.21.15.78:0/2657723877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4009b30 con 0x7f49ac0705b0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b1441640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49ac0c2f60 0x7f49ac156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49abfff640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b0c40640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac153e70 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c7f0 tx=0x7f499c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c004220 con 0x7f49ac0705b0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c0043a0 con 0x7f49ac0705b0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac152ab0 con 0x7f49ac0705b0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c007860 con 0x7f49ac0705b0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.057+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac06cb90 con 0x7f49ac0705b0 2024-02-19T15:00:34.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.058+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac0676a0 con 0x7f49ac0705b0 2024-02-19T15:00:34.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.058+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f499c007bd0 con 0x7f49ac0705b0 2024-02-19T15:00:34.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.059+0000 7f49a97fa640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499806b840 0x7f499806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.059+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f499c08a2d0 con 0x7f49ac0705b0 2024-02-19T15:00:34.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.059+0000 7f49abfff640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499806b840 0x7f499806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.059+0000 7f49abfff640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499806b840 0x7f499806dd00 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f49ac14f1f0 tx=0x7f49a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.063+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f499c054b60 con 0x7f49ac0705b0 2024-02-19T15:00:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.270+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f49ac074990 con 0x7f49ac0705b0 2024-02-19T15:00:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.271+0000 7f49a97fa640 1 -- 172.21.15.78:0/2657723877 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f499c058220 con 0x7f49ac0705b0 2024-02-19T15:00:34.274 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T15:00:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.272+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499806b840 msgr2=0x7f499806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.272+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499806b840 0x7f499806dd00 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f49ac14f1f0 tx=0x7f49a0009040 comp rx=0 tx=0).stop 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.272+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 msgr2=0x7f49ac153e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.272+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac153e70 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c7f0 tx=0x7f499c00ccb0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 shutdown_connections 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f499806b840 0x7f499806dd00 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49ac0c2f60 0x7f49ac156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49ac0add70 0x7f49ac0c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 --2- 172.21.15.78:0/2657723877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49ac0705b0 0x7f49ac153e70 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 >> 172.21.15.78:0/2657723877 conn(0x7f49ac0c85e0 msgr2=0x7f49ac14fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 shutdown_connections 2024-02-19T15:00:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.273+0000 7f49b2ea4640 1 -- 172.21.15.78:0/2657723877 wait complete. 2024-02-19T15:00:34.286 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-19T15:00:34.286 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:00:34.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:00:34.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:00:34.367 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:00:34.368 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T15:00:34.368 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:00:34.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T15:00:34.383 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T15:00:34.383 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:00:34.425 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T15:00:34.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:34.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.523+0000 7fc7329a2640 1 Processor -- start 2024-02-19T15:00:34.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.523+0000 7fc7329a2640 1 -- start start 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.523+0000 7fc7329a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc72c1524b0 0x7fc72c06d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7329a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c0c7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7329a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c155e00 con 0x7fc72c0c8640 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7329a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c0c7850 con 0x7fc72c1524b0 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7329a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c0c7b30 con 0x7fc72c1559c0 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc72c1524b0 0x7fc72c06d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc72c1524b0 0x7fc72c06d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46162/0 (socket says 172.21.15.78:46162) 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 -- 172.21.15.78:0/475454726 learned_addr learned my addr 172.21.15.78:0/475454726 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc73119f640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c0c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 -- 172.21.15.78:0/475454726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 msgr2=0x7fc72c0c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:34.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c0c7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 -- 172.21.15.78:0/475454726 >> v1:172.21.15.150:6789/0 conn(0x7fc72c0c8640 legacy=0x7fc72c0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:34.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc7319a0640 1 -- 172.21.15.78:0/475454726 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c06cc60 con 0x7fc72c1524b0 2024-02-19T15:00:34.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.524+0000 7fc73119f640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c0c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7319a0640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c06d740 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fc728009870 tx=0x7fc72802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596816109b627639 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc73099e640 1 -- 172.21.15.78:0/475454726 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7280317c0 con 0x7fc72c1524b0 2024-02-19T15:00:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc73099e640 1 -- 172.21.15.78:0/475454726 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc728031920 con 0x7fc72c1524b0 2024-02-19T15:00:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc73099e640 1 -- 172.21.15.78:0/475454726 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728031cb0 con 0x7fc72c1524b0 2024-02-19T15:00:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7329a2640 1 -- 172.21.15.78:0/475454726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 msgr2=0x7fc72c06d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7329a2640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c06d740 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fc728009870 tx=0x7fc72802b860 comp rx=0 tx=0).stop 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7329a2640 1 -- 172.21.15.78:0/475454726 shutdown_connections 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7329a2640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c0c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7329a2640 1 --2- 172.21.15.78:0/475454726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c06d740 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.525+0000 7fc7329a2640 1 -- 172.21.15.78:0/475454726 >> 172.21.15.78:0/475454726 conn(0x7fc72c0ac250 msgr2=0x7fc72c0ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.526+0000 7fc7329a2640 1 -- 172.21.15.78:0/475454726 shutdown_connections 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.526+0000 7fc7329a2640 1 -- 172.21.15.78:0/475454726 wait complete. 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 Processor -- start 2024-02-19T15:00:34.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 -- start start 2024-02-19T15:00:34.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c0c8640 0x7fc72c0c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c0c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c1ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c1ee480 con 0x7fc72c1524b0 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c0c2090 con 0x7fc72c0c8640 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc7329a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c0c18b0 con 0x7fc72c1559c0 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc73119f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c0c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc73119f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c0c94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46166/0 (socket says 172.21.15.78:46166) 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.527+0000 7fc73119f640 1 -- 172.21.15.78:0/1825160816 learned_addr learned my addr 172.21.15.78:0/1825160816 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc73119f640 1 -- 172.21.15.78:0/1825160816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 msgr2=0x7fc72c1ea7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc73119f640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c1ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc73119f640 1 -- 172.21.15.78:0/1825160816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c0c8640 msgr2=0x7fc72c0c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc73119f640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c0c8640 0x7fc72c0c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc73119f640 1 -- 172.21.15.78:0/1825160816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280094d0 con 0x7fc72c1524b0 2024-02-19T15:00:34.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc73119f640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c0c94a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00c7f0 tx=0x7fc71c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c007820 con 0x7fc72c1524b0 2024-02-19T15:00:34.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c007980 con 0x7fc72c1524b0 2024-02-19T15:00:34.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c1ede00 con 0x7fc72c1524b0 2024-02-19T15:00:34.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c007cf0 con 0x7fc72c1524b0 2024-02-19T15:00:34.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.528+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c0c99e0 con 0x7fc72c1524b0 2024-02-19T15:00:34.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.529+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c074990 con 0x7fc72c1524b0 2024-02-19T15:00:34.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.529+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc71c0113f0 con 0x7fc72c1524b0 2024-02-19T15:00:34.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.530+0000 7fc7227fc640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71006b810 0x7fc71006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.530+0000 7fc7319a0640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71006b810 0x7fc71006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.530+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc71c08a6e0 con 0x7fc72c1524b0 2024-02-19T15:00:34.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.530+0000 7fc7319a0640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71006b810 0x7fc71006dcd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc728036000 tx=0x7fc728030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.534+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc71c054f70 con 0x7fc72c1524b0 2024-02-19T15:00:34.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.739+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc72c1ee5c0 con 0x7fc72c1524b0 2024-02-19T15:00:34.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.739+0000 7fc7227fc640 1 -- 172.21.15.78:0/1825160816 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fc72c1ee5c0 con 0x7fc72c1524b0 2024-02-19T15:00:34.742 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:34.742 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":184,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10524,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2906585066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2906585066},{"type":"v1","addr":"172.21.15.78:6837","nonce":2906585066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:55.177591+0000","modified":"2024-02-19T15:00:15.727959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-19T15:00:34.743 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 184 2024-02-19T15:00:34.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71006b810 msgr2=0x7fc71006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71006b810 0x7fc71006dcd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc728036000 tx=0x7fc728030000 comp rx=0 tx=0).stop 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 msgr2=0x7fc72c0c94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c0c94a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00c7f0 tx=0x7fc71c00ccb0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 shutdown_connections 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc71006b810 0x7fc71006dcd0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc72c1524b0 0x7fc72c0c94a0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc72c1559c0 0x7fc72c1ea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 --2- 172.21.15.78:0/1825160816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc72c0c8640 0x7fc72c0c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 >> 172.21.15.78:0/1825160816 conn(0x7fc72c0ac250 msgr2=0x7fc72c154460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 shutdown_connections 2024-02-19T15:00:34.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.741+0000 7fc7329a2640 1 -- 172.21.15.78:0/1825160816 wait complete. 2024-02-19T15:00:34.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.853+0000 7fe3bc3dd640 1 Processor -- start 2024-02-19T15:00:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.853+0000 7fe3bc3dd640 1 -- start start 2024-02-19T15:00:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3bc3dd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe3b40c25e0 0x7fe3b41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3bc3dd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3b40c29c0 0x7fe3b40c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3bc3dd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b4072250 con 0x7fe3b40c49c0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3bc3dd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b4152c90 con 0x7fe3b40c29c0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3bc3dd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b4072480 con 0x7fe3b40c25e0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9978640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3b40c29c0 0x7fe3b40c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3ba97a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe3b40c49c0 0x7fe3b406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41780/0 (socket says 172.21.15.78:41780) 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9978640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3b40c29c0 0x7fe3b40c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46170/0 (socket says 172.21.15.78:46170) 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3ba97a640 1 -- 172.21.15.78:0/3008630955 learned_addr learned my addr 172.21.15.78:0/3008630955 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3ba179640 1 --2- 172.21.15.78:0/3008630955 >> v2:172.21.15.150:3300/0 conn(0x7fe3b40c25e0 0x7fe3b41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9978640 1 -- 172.21.15.78:0/3008630955 >> v2:172.21.15.150:3300/0 conn(0x7fe3b40c25e0 msgr2=0x7fe3b41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9978640 1 --2- 172.21.15.78:0/3008630955 >> v2:172.21.15.150:3300/0 conn(0x7fe3b40c25e0 0x7fe3b41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9978640 1 -- 172.21.15.78:0/3008630955 >> v1:172.21.15.78:6789/0 conn(0x7fe3b40c49c0 legacy=0x7fe3b406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9177640 1 -- 172.21.15.78:0/3008630955 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630235719 0 0) 0x7fe3b4072250 con 0x7fe3b40c49c0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.854+0000 7fe3b9978640 1 -- 172.21.15.78:0/3008630955 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b40c3eb0 con 0x7fe3b40c29c0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3b9978640 1 --2- 172.21.15.78:0/3008630955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c29c0 0x7fe3b40c4480 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fe3b406f120 tx=0x7fe3a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f2a730a1b53211 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3b9177640 1 -- 172.21.15.78:0/3008630955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a402e070 con 0x7fe3b40c29c0 2024-02-19T15:00:34.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3b9177640 1 -- 172.21.15.78:0/3008630955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a40315a0 con 0x7fe3b40c29c0 2024-02-19T15:00:34.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3b9177640 1 -- 172.21.15.78:0/3008630955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4031930 con 0x7fe3b40c29c0 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/3008630955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c29c0 msgr2=0x7fe3b40c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/3008630955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c29c0 0x7fe3b40c4480 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fe3b406f120 tx=0x7fe3a402b840 comp rx=0 tx=0).stop 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/3008630955 shutdown_connections 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/3008630955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c29c0 0x7fe3b40c4480 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/3008630955 >> v2:172.21.15.150:3300/0 conn(0x7fe3b40c25e0 0x7fe3b41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.855+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/3008630955 >> 172.21.15.78:0/3008630955 conn(0x7fe3b406b270 msgr2=0x7fe3b406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.856+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/3008630955 shutdown_connections 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.856+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/3008630955 wait complete. 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 Processor -- start 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 -- start start 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 0x7fe3b414c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b40c29c0 0x7fe3b4149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4149850 0x7fe3b406d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41e5b40 con 0x7fe3b40c25e0 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41e8df0 con 0x7fe3b40c29c0 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3bc3dd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41e8270 con 0x7fe3b4149850 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3ba179640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 0x7fe3b414c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3ba179640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 0x7fe3b414c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46176/0 (socket says 172.21.15.78:46176) 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.857+0000 7fe3ba179640 1 -- 172.21.15.78:0/2306146053 learned_addr learned my addr 172.21.15.78:0/2306146053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3b9978640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b40c29c0 0x7fe3b4149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3ba179640 1 -- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4149850 msgr2=0x7fe3b406d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3ba179640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4149850 0x7fe3b406d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3ba179640 1 -- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b40c29c0 msgr2=0x7fe3b4149310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3ba179640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b40c29c0 0x7fe3b4149310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3ba179640 1 -- 172.21.15.78:0/2306146053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4009400 con 0x7fe3b40c25e0 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3b9978640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b40c29c0 0x7fe3b4149310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3ba179640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 0x7fe3b414c190 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9b0 tx=0x7fe3b000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0009670 con 0x7fe3b40c25e0 2024-02-19T15:00:34.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b000c9a0 con 0x7fe3b40c25e0 2024-02-19T15:00:34.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000cd50 con 0x7fe3b40c25e0 2024-02-19T15:00:34.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b41e8550 con 0x7fe3b40c25e0 2024-02-19T15:00:34.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.858+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b41e8b10 con 0x7fe3b40c25e0 2024-02-19T15:00:34.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.859+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b4074970 con 0x7fe3b40c25e0 2024-02-19T15:00:34.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.859+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe3b0013470 con 0x7fe3b40c25e0 2024-02-19T15:00:34.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.860+0000 7fe3aaffd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe39806b840 0x7fe39806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:34.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.860+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe3b008a4f0 con 0x7fe3b40c25e0 2024-02-19T15:00:34.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.860+0000 7fe3b9978640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe39806b840 0x7fe39806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:34.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.860+0000 7fe3b9978640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe39806b840 0x7fe39806dd00 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4036000 tx=0x7fe3a403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:34.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:34.864+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe3b0054e10 con 0x7fe3b40c25e0 2024-02-19T15:00:35.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.038+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3b406b7f0 con 0x7fe3b40c25e0 2024-02-19T15:00:35.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.039+0000 7fe3aaffd640 1 -- 172.21.15.78:0/2306146053 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16940 (secure 0 0 0) 0x7fe3b00584d0 con 0x7fe3b40c25e0 2024-02-19T15:00:35.042 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:35.043 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":202,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:56.830521+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:53.390533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:54.145380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:35.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.041+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe39806b840 msgr2=0x7fe39806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.041+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe39806b840 0x7fe39806dd00 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4036000 tx=0x7fe3a403d040 comp rx=0 tx=0).stop 2024-02-19T15:00:35.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 msgr2=0x7fe3b414c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 0x7fe3b414c190 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9b0 tx=0x7fe3b000ee70 comp rx=0 tx=0).stop 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 shutdown_connections 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe39806b840 0x7fe39806dd00 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4149850 0x7fe3b406d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b40c29c0 0x7fe3b4149310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 --2- 172.21.15.78:0/2306146053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3b40c25e0 0x7fe3b414c190 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 >> 172.21.15.78:0/2306146053 conn(0x7fe3b406b270 msgr2=0x7fe3b40c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 shutdown_connections 2024-02-19T15:00:35.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.042+0000 7fe3bc3dd640 1 -- 172.21.15.78:0/2306146053 wait complete. 2024-02-19T15:00:35.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T14:59:53.390533+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T14:59:54.145380+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '202', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:00:35.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:00:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.155+0000 7feaee902640 1 Processor -- start 2024-02-19T15:00:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.155+0000 7feaee902640 1 -- start start 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feaee902640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feae80c78a0 0x7feae80c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feaee902640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feae814ef50 0x7feae814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feaee902640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae80c35c0 con 0x7feae80727d0 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feaee902640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae814f920 con 0x7feae80c78a0 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feaee902640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae80c38a0 con 0x7feae814ef50 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feae7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feae80c78a0 0x7feae80c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feae7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feae80c78a0 0x7feae80c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46184/0 (socket says 172.21.15.78:46184) 2024-02-19T15:00:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feae7fff640 1 -- 172.21.15.78:0/2021485511 learned_addr learned my addr 172.21.15.78:0/2021485511 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feae77fe640 1 --2- 172.21.15.78:0/2021485511 >> v2:172.21.15.150:3300/0 conn(0x7feae814ef50 0x7feae814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feae7fff640 1 -- 172.21.15.78:0/2021485511 >> v2:172.21.15.150:3300/0 conn(0x7feae814ef50 msgr2=0x7feae814f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.156+0000 7feae7fff640 1 --2- 172.21.15.78:0/2021485511 >> v2:172.21.15.150:3300/0 conn(0x7feae814ef50 0x7feae814f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.157+0000 7feae7fff640 1 -- 172.21.15.78:0/2021485511 >> v1:172.21.15.150:6789/0 conn(0x7feae80727d0 legacy=0x7feae8072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.157+0000 7feae7fff640 1 -- 172.21.15.78:0/2021485511 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80c3b80 con 0x7feae80c78a0 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.157+0000 7feae7fff640 1 --2- 172.21.15.78:0/2021485511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae80c44b0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7feae80c1eb0 tx=0x7fead802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2050f916c72f6d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.157+0000 7feae6ffd640 1 -- 172.21.15.78:0/2021485511 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8030860 con 0x7feae80c78a0 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.157+0000 7feae6ffd640 1 -- 172.21.15.78:0/2021485511 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80309c0 con 0x7feae80c78a0 2024-02-19T15:00:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.157+0000 7feae6ffd640 1 -- 172.21.15.78:0/2021485511 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8030d50 con 0x7feae80c78a0 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 -- 172.21.15.78:0/2021485511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 msgr2=0x7feae80c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 --2- 172.21.15.78:0/2021485511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae80c44b0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7feae80c1eb0 tx=0x7fead802b9a0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 -- 172.21.15.78:0/2021485511 shutdown_connections 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 --2- 172.21.15.78:0/2021485511 >> v2:172.21.15.150:3300/0 conn(0x7feae814ef50 0x7feae814f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 --2- 172.21.15.78:0/2021485511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae80c44b0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 -- 172.21.15.78:0/2021485511 >> 172.21.15.78:0/2021485511 conn(0x7feae8066e70 msgr2=0x7feae80a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 -- 172.21.15.78:0/2021485511 shutdown_connections 2024-02-19T15:00:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.158+0000 7feaee902640 1 -- 172.21.15.78:0/2021485511 wait complete. 2024-02-19T15:00:35.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.159+0000 7feaee902640 1 Processor -- start 2024-02-19T15:00:35.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.159+0000 7feaee902640 1 -- start start 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaee902640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae80727d0 0x7feae8145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaee902640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae806c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaee902640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae814ef50 0x7feae81ed750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaee902640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae81462e0 con 0x7feae80c78a0 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaee902640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80aa8f0 con 0x7feae80727d0 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaee902640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80aabd0 con 0x7feae814ef50 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae806c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae806c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46200/0 (socket says 172.21.15.78:46200) 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 -- 172.21.15.78:0/2782479890 learned_addr learned my addr 172.21.15.78:0/2782479890 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae7fff640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae80727d0 0x7feae8145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaece9f640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae814ef50 0x7feae81ed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 -- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae814ef50 msgr2=0x7feae81ed750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae814ef50 0x7feae81ed750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 -- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae80727d0 msgr2=0x7feae8145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae80727d0 0x7feae8145da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feae77fe640 1 -- 172.21.15.78:0/2782479890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead80094d0 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.160+0000 7feaece9f640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae814ef50 0x7feae81ed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.161+0000 7feae77fe640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae806c600 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7feadc00d5f0 tx=0x7feadc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.161+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc0125f0 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.161+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc012750 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.161+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae8149d60 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.161+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc012b00 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.161+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae814a320 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.162+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae80aad90 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.162+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feadc012c60 con 0x7feae80c78a0 2024-02-19T15:00:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.162+0000 7feae4ff9640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.163+0000 7feae7fff640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.163+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feadc08a750 con 0x7feae80c78a0 2024-02-19T15:00:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.163+0000 7feae7fff640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fead8036000 tx=0x7fead8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.167+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feadc054fe0 con 0x7feae80c78a0 2024-02-19T15:00:35.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.341+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feae8074990 con 0x7feae80c78a0 2024-02-19T15:00:35.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.341+0000 7feae4ff9640 1 -- 172.21.15.78:0/2782479890 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v202) v1 ==== 93+0+10 (secure 0 0 0) 0x7feadc0586a0 con 0x7feae80c78a0 2024-02-19T15:00:35.344 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T15:00:35.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 msgr2=0x7fead406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fead8036000 tx=0x7fead8008030 comp rx=0 tx=0).stop 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 msgr2=0x7feae806c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae806c600 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7feadc00d5f0 tx=0x7feadc00dab0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 shutdown_connections 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fead406b840 0x7fead406dd00 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feae80c78a0 0x7feae806c600 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feae814ef50 0x7feae81ed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 --2- 172.21.15.78:0/2782479890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feae80727d0 0x7feae8145da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 >> 172.21.15.78:0/2782479890 conn(0x7feae8066e70 msgr2=0x7feae8150a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 shutdown_connections 2024-02-19T15:00:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.343+0000 7feaee902640 1 -- 172.21.15.78:0/2782479890 wait complete. 2024-02-19T15:00:35.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:00:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.457+0000 7fd6d570d640 1 Processor -- start 2024-02-19T15:00:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.457+0000 7fd6d570d640 1 -- start start 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6d570d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd6d0152700 0x7fd6d00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6d570d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6d00c8e00 0x7fd6d015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6d570d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d01551f0 con 0x7fd6d00c5730 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6d570d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d0155490 con 0x7fd6d0152700 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6d570d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d0155770 con 0x7fd6d00c8e00 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd6d0152700 0x7fd6d00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd6d0152700 0x7fd6d00c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46204/0 (socket says 172.21.15.78:46204) 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 -- 172.21.15.78:0/122088013 learned_addr learned my addr 172.21.15.78:0/122088013 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6cf7fe640 1 --2- 172.21.15.78:0/122088013 >> v2:172.21.15.150:3300/0 conn(0x7fd6d00c8e00 0x7fd6d015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 -- 172.21.15.78:0/122088013 >> v2:172.21.15.150:3300/0 conn(0x7fd6d00c8e00 msgr2=0x7fd6d015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 --2- 172.21.15.78:0/122088013 >> v2:172.21.15.150:3300/0 conn(0x7fd6d00c8e00 0x7fd6d015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 -- 172.21.15.78:0/122088013 >> v1:172.21.15.150:6789/0 conn(0x7fd6d00c5730 legacy=0x7fd6d006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.458+0000 7fd6ceffd640 1 -- 172.21.15.78:0/122088013 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0155a50 con 0x7fd6d0152700 2024-02-19T15:00:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.459+0000 7fd6ceffd640 1 --2- 172.21.15.78:0/122088013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d0152700 0x7fd6d00c51f0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fd6d01596f0 tx=0x7fd6c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a1587b68ec8300 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.459+0000 7fd6ce7fc640 1 -- 172.21.15.78:0/122088013 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00317e0 con 0x7fd6d0152700 2024-02-19T15:00:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.459+0000 7fd6ce7fc640 1 -- 172.21.15.78:0/122088013 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c0031940 con 0x7fd6d0152700 2024-02-19T15:00:35.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.459+0000 7fd6ce7fc640 1 -- 172.21.15.78:0/122088013 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0031cd0 con 0x7fd6d0152700 2024-02-19T15:00:35.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 -- 172.21.15.78:0/122088013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d0152700 msgr2=0x7fd6d00c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 --2- 172.21.15.78:0/122088013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d0152700 0x7fd6d00c51f0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fd6d01596f0 tx=0x7fd6c002b880 comp rx=0 tx=0).stop 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 -- 172.21.15.78:0/122088013 shutdown_connections 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 --2- 172.21.15.78:0/122088013 >> v2:172.21.15.150:3300/0 conn(0x7fd6d00c8e00 0x7fd6d015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 --2- 172.21.15.78:0/122088013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d0152700 0x7fd6d00c51f0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 -- 172.21.15.78:0/122088013 >> 172.21.15.78:0/122088013 conn(0x7fd6d00cac90 msgr2=0x7fd6d00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 -- 172.21.15.78:0/122088013 shutdown_connections 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.460+0000 7fd6d570d640 1 -- 172.21.15.78:0/122088013 wait complete. 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 Processor -- start 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 -- start start 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6d00c5730 0x7fd6d00ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 0x7fd6d01ede20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6d0152700 0x7fd6d01e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00c41c0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d01ee660 con 0x7fd6d00c5730 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.461+0000 7fd6d570d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d01ee900 con 0x7fd6d0152700 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 0x7fd6d01ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 0x7fd6d01ede20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46214/0 (socket says 172.21.15.78:46214) 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 -- 172.21.15.78:0/3529198433 learned_addr learned my addr 172.21.15.78:0/3529198433 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6ceffd640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6d00c5730 0x7fd6d00ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 -- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6d0152700 msgr2=0x7fd6d01e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6d0152700 0x7fd6d01e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 -- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6d00c5730 msgr2=0x7fd6d00ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6d00c5730 0x7fd6d00ac5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 -- 172.21.15.78:0/3529198433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c00094d0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c7fff640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 0x7fd6d01ede20 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c430 tx=0x7fd6b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8011790 con 0x7fd6d00c8e00 2024-02-19T15:00:35.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b80118f0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8011ce0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d00accd0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.462+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d01ee360 con 0x7fd6d00c8e00 2024-02-19T15:00:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.463+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d0151ad0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.464+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd6b801c460 con 0x7fd6d00c8e00 2024-02-19T15:00:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.464+0000 7fd6c77fe640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd6b006b810 0x7fd6b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.464+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd6b808a350 con 0x7fd6d00c8e00 2024-02-19T15:00:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.464+0000 7fd6ceffd640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd6b006b810 0x7fd6b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.464+0000 7fd6ceffd640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd6b006b810 0x7fd6b006dcd0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0036000 tx=0x7fd6c003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.468+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd6b8054be0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.639+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6d00acb20 con 0x7fd6d00c8e00 2024-02-19T15:00:35.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.639+0000 7fd6c77fe640 1 -- 172.21.15.78:0/3529198433 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v202) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6b80582a0 con 0x7fd6d00c8e00 2024-02-19T15:00:35.642 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T15:00:35.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd6b006b810 msgr2=0x7fd6b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd6b006b810 0x7fd6b006dcd0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0036000 tx=0x7fd6c003d040 comp rx=0 tx=0).stop 2024-02-19T15:00:35.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 msgr2=0x7fd6d01ede20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 0x7fd6d01ede20 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c430 tx=0x7fd6b800c8f0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 shutdown_connections 2024-02-19T15:00:35.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd6b006b810 0x7fd6b006dcd0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6d00c8e00 0x7fd6d01ede20 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6d0152700 0x7fd6d01e7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 --2- 172.21.15.78:0/3529198433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6d00c5730 0x7fd6d00ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 >> 172.21.15.78:0/3529198433 conn(0x7fd6d00cac90 msgr2=0x7fd6d00cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 shutdown_connections 2024-02-19T15:00:35.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.641+0000 7fd6d570d640 1 -- 172.21.15.78:0/3529198433 wait complete. 2024-02-19T15:00:35.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:00:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.754+0000 7f604264e640 1 Processor -- start 2024-02-19T15:00:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f604264e640 1 -- start start 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f604264e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f603c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f604264e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f603c06d910 0x7f603c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f604264e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c06cd20 con 0x7f603c152700 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f604264e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c1551f0 con 0x7f603c06d910 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f604264e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c155490 con 0x7f603c0c8e00 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f603bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f603c152700 0x7f603c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41788/0 (socket says 172.21.15.78:41788) 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.755+0000 7f603bfff640 1 -- 172.21.15.78:0/4229474031 learned_addr learned my addr 172.21.15.78:0/4229474031 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790931409 0 0) 0x7f603c06cd20 con 0x7f603c152700 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f6040beb640 1 --2- 172.21.15.78:0/4229474031 >> v2:172.21.15.150:3300/0 conn(0x7f603c06d910 0x7f603c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024003520 con 0x7f603c152700 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603b7fe640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f603c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1003077636 0 0) 0x7f6024003520 con 0x7f603c152700 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 msgr2=0x7f603c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f603c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 >> v2:172.21.15.150:3300/0 conn(0x7f603c06d910 msgr2=0x7f603c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 --2- 172.21.15.78:0/4229474031 >> v2:172.21.15.150:3300/0 conn(0x7f603c06d910 0x7f603c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c155770 con 0x7f603c152700 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f602c001670 con 0x7f603c152700 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 >> v1:172.21.15.78:6789/0 conn(0x7f603c152700 legacy=0x7f603c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f6024003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f60240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6024004d00 0x7f6024005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c1551f0 con 0x7f6024003970 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c155490 con 0x7f6024004d00 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602400b980 con 0x7f603c0c8e00 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603bfff640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f6024003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f6040beb640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6024004d00 0x7f6024005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603b7fe640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f60240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603bfff640 1 -- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 msgr2=0x7f60240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603bfff640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f60240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603bfff640 1 -- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6024004d00 msgr2=0x7f6024005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603bfff640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6024004d00 0x7f6024005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603bfff640 1 -- 172.21.15.78:0/4229474031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6024003520 con 0x7f6024003970 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.756+0000 7f603b7fe640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c0c8e00 0x7f60240047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.757+0000 7f603bfff640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f6024003d90 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f602c00c210 tx=0x7f60240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fea65214b831b3a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.757+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c005ad0 con 0x7f6024003970 2024-02-19T15:00:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.757+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c005c30 con 0x7f6024003970 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.757+0000 7f603affd640 1 -- 172.21.15.78:0/4229474031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c02c8d0 con 0x7f6024003970 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f604264e640 1 -- 172.21.15.78:0/4229474031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 msgr2=0x7f6024003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f604264e640 1 --2- 172.21.15.78:0/4229474031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f6024003d90 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f602c00c210 tx=0x7f60240057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f6040beb640 1 -- 172.21.15.78:0/4229474031 reap_dead start 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f604264e640 1 -- 172.21.15.78:0/4229474031 shutdown_connections 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f604264e640 1 -- 172.21.15.78:0/4229474031 >> 172.21.15.78:0/4229474031 conn(0x7f603c0cac90 msgr2=0x7f603c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f604264e640 1 -- 172.21.15.78:0/4229474031 shutdown_connections 2024-02-19T15:00:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.758+0000 7f604264e640 1 -- 172.21.15.78:0/4229474031 wait complete. 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.759+0000 7f604264e640 1 Processor -- start 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.759+0000 7f604264e640 1 -- start start 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f603c1edd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c1ee270 0x7f603c1ea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f603c0c4630 0x7f603c0c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c0aea50 con 0x7f6024003970 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c1ed570 con 0x7f603c0c4630 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c1ea3a0 con 0x7f603c1ee270 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f603c1edd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f603c1edd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46228/0 (socket says 172.21.15.78:46228) 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 -- 172.21.15.78:0/1808217936 learned_addr learned my addr 172.21.15.78:0/1808217936 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603b7fe640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c1ee270 0x7f603c1ea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f6040beb640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f603c0c4630 0x7f603c0c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 -- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c1ee270 msgr2=0x7f603c1ea7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c1ee270 0x7f603c1ea7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 -- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f603c0c4630 msgr2=0x7f603c0c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f603c0c4630 0x7f603c0c0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 -- 172.21.15.78:0/1808217936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0af600 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f603bfff640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f603c1edd30 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f602c001930 tx=0x7f603c06d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c039070 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c02dd90 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c0072b0 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c0af8e0 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.760+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c1ed810 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.761+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c1ed9a0 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.762+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f602c032070 con 0x7f6024003970 2024-02-19T15:00:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.762+0000 7f6038ff9640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f602806b810 0x7f602806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.762+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f602c03c040 con 0x7f6024003970 2024-02-19T15:00:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.762+0000 7f603b7fe640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f602806b810 0x7f602806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.763+0000 7f603b7fe640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f602806b810 0x7f602806dcd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f603c0aff50 tx=0x7f6030008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.767+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f602c076110 con 0x7f6024003970 2024-02-19T15:00:35.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.945+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f603c074990 con 0x7f6024003970 2024-02-19T15:00:35.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.945+0000 7f6038ff9640 1 -- 172.21.15.78:0/1808217936 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v202) v1 ==== 100+0+11 (secure 0 0 0) 0x7f602c0797d0 con 0x7f6024003970 2024-02-19T15:00:35.948 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T15:00:35.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f602806b810 msgr2=0x7f602806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f602806b810 0x7f602806dcd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f603c0aff50 tx=0x7f6030008040 comp rx=0 tx=0).stop 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 msgr2=0x7f603c1edd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f603c1edd30 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f602c001930 tx=0x7f603c06d6b0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 shutdown_connections 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f602806b810 0x7f602806dcd0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f603c0c4630 0x7f603c0c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f603c1ee270 0x7f603c1ea7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 --2- 172.21.15.78:0/1808217936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6024003970 0x7f603c1edd30 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.947+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 >> 172.21.15.78:0/1808217936 conn(0x7f603c0cac90 msgr2=0x7f603c06c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.948+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 shutdown_connections 2024-02-19T15:00:35.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:35.948+0000 7f604264e640 1 -- 172.21.15.78:0/1808217936 wait complete. 2024-02-19T15:00:35.961 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:36.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.058+0000 7fc748347640 1 Processor -- start 2024-02-19T15:00:36.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.058+0000 7fc748347640 1 -- start start 2024-02-19T15:00:36.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc748347640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7401559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc748347640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7400c87a0 0x7fc7400c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc748347640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc740155ef0 con 0x7fc740152480 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc748347640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74006dec0 con 0x7fc7400c87a0 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc748347640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74006d520 con 0x7fc7400c7070 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7468e4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7400c87a0 0x7fc7400c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7460e3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc740152480 0x7fc74006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41800/0 (socket says 172.21.15.78:41800) 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7468e4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7400c87a0 0x7fc7400c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46234/0 (socket says 172.21.15.78:46234) 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7460e3640 1 -- 172.21.15.78:0/343976709 learned_addr learned my addr 172.21.15.78:0/343976709 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7468e4640 1 -- 172.21.15.78:0/343976709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 msgr2=0x7fc7401559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7458e2640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7401559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7450e1640 1 -- 172.21.15.78:0/343976709 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408921008 0 0) 0x7fc73c001110 con 0x7fc740152480 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7468e4640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7401559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7468e4640 1 -- 172.21.15.78:0/343976709 >> v1:172.21.15.78:6789/0 conn(0x7fc740152480 legacy=0x7fc74006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7468e4640 1 -- 172.21.15.78:0/343976709 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7400c75c0 con 0x7fc7400c87a0 2024-02-19T15:00:36.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.059+0000 7fc7458e2640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7401559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:36.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc7468e4640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7400c5220 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fc738009d90 tx=0x7fc73802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44eb1ac73aaadfd0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc7450e1640 1 -- 172.21.15.78:0/343976709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73802cc50 con 0x7fc7400c87a0 2024-02-19T15:00:36.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc7450e1640 1 -- 172.21.15.78:0/343976709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73802cdb0 con 0x7fc7400c87a0 2024-02-19T15:00:36.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc7450e1640 1 -- 172.21.15.78:0/343976709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738034660 con 0x7fc7400c87a0 2024-02-19T15:00:36.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc748347640 1 -- 172.21.15.78:0/343976709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 msgr2=0x7fc7400c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc748347640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7400c5220 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fc738009d90 tx=0x7fc73802bea0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc748347640 1 -- 172.21.15.78:0/343976709 shutdown_connections 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc748347640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7400c5220 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc748347640 1 --2- 172.21.15.78:0/343976709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7401559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.060+0000 7fc748347640 1 -- 172.21.15.78:0/343976709 >> 172.21.15.78:0/343976709 conn(0x7fc7400ca8a0 msgr2=0x7fc7400cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.061+0000 7fc748347640 1 -- 172.21.15.78:0/343976709 shutdown_connections 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.061+0000 7fc748347640 1 -- 172.21.15.78:0/343976709 wait complete. 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 Processor -- start 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 -- start start 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7400afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7401564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc740152480 0x7fc7401569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740157c80 con 0x7fc7400c87a0 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740158490 con 0x7fc740152480 2024-02-19T15:00:36.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc748347640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400c1510 con 0x7fc7400c7070 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc7458e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7401564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc7458e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7401564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46236/0 (socket says 172.21.15.78:46236) 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.062+0000 7fc7458e2640 1 -- 172.21.15.78:0/1021083313 learned_addr learned my addr 172.21.15.78:0/1021083313 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7460e3640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7400afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7468e4640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc740152480 0x7fc7401569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7458e2640 1 -- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 msgr2=0x7fc7400afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7458e2640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7400afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7458e2640 1 -- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc740152480 msgr2=0x7fc7401569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7458e2640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc740152480 0x7fc7401569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7458e2640 1 -- 172.21.15.78:0/1021083313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738009a70 con 0x7fc7400c87a0 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7468e4640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc740152480 0x7fc7401569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc7458e2640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7401564a0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fc73000c7f0 tx=0x7fc73000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73000cea0 con 0x7fc7400c87a0 2024-02-19T15:00:36.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc730004230 con 0x7fc7400c87a0 2024-02-19T15:00:36.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc740157ed0 con 0x7fc7400c87a0 2024-02-19T15:00:36.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730007830 con 0x7fc7400c87a0 2024-02-19T15:00:36.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.063+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7400b0330 con 0x7fc7400c87a0 2024-02-19T15:00:36.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.064+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc740074990 con 0x7fc7400c87a0 2024-02-19T15:00:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.064+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc730007990 con 0x7fc7400c87a0 2024-02-19T15:00:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.065+0000 7fc736ffd640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc72406b840 0x7fc72406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.065+0000 7fc7460e3640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc72406b840 0x7fc72406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.065+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc73008a280 con 0x7fc7400c87a0 2024-02-19T15:00:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.065+0000 7fc7460e3640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc72406b840 0x7fc72406dd00 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fc73c000f60 tx=0x7fc73c007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.069+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc730054ba0 con 0x7fc7400c87a0 2024-02-19T15:00:36.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.275+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc740067ed0 con 0x7fc7400c87a0 2024-02-19T15:00:36.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.275+0000 7fc736ffd640 1 -- 172.21.15.78:0/1021083313 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fc730058260 con 0x7fc7400c87a0 2024-02-19T15:00:36.278 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:36.278 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":184,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10524,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2906585066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2906585066},{"type":"v1","addr":"172.21.15.78:6837","nonce":2906585066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:55.177591+0000","modified":"2024-02-19T15:00:15.727959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-19T15:00:36.279 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 184 2024-02-19T15:00:36.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc72406b840 msgr2=0x7fc72406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc72406b840 0x7fc72406dd00 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fc73c000f60 tx=0x7fc73c007c00 comp rx=0 tx=0).stop 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 msgr2=0x7fc7401564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7401564a0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fc73000c7f0 tx=0x7fc73000ccb0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 shutdown_connections 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc72406b840 0x7fc72406dd00 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7400c87a0 0x7fc7401564a0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc740152480 0x7fc7401569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 --2- 172.21.15.78:0/1021083313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7400c7070 0x7fc7400afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 >> 172.21.15.78:0/1021083313 conn(0x7fc7400ca8a0 msgr2=0x7fc740153980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 shutdown_connections 2024-02-19T15:00:36.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.277+0000 7fc748347640 1 -- 172.21.15.78:0/1021083313 wait complete. 2024-02-19T15:00:36.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.388+0000 7f0a40748640 1 Processor -- start 2024-02-19T15:00:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.388+0000 7f0a40748640 1 -- start start 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.389+0000 7f0a40748640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a38072ee0 0x7f0a3814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.389+0000 7f0a40748640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a380c35c0 con 0x7f0a3814ef50 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.389+0000 7f0a40748640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a380c38a0 con 0x7f0a380c78a0 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.389+0000 7f0a40748640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a380c4bc0 con 0x7f0a38072ee0 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.389+0000 7f0a3dce3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0a3814ef50 0x7f0a380727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41802/0 (socket says 172.21.15.78:41802) 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.389+0000 7f0a3dce3640 1 -- 172.21.15.78:0/97340857 learned_addr learned my addr 172.21.15.78:0/97340857 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3ece5640 1 --2- 172.21.15.78:0/97340857 >> v2:172.21.15.150:3300/0 conn(0x7f0a38072ee0 0x7f0a3814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785311989 0 0) 0x7f0a380c35c0 con 0x7f0a3814ef50 2024-02-19T15:00:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a24003520 con 0x7f0a3814ef50 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126466857 0 0) 0x7f0a24003520 con 0x7f0a3814ef50 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 >> v2:172.21.15.150:3300/0 conn(0x7f0a38072ee0 msgr2=0x7f0a3814f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 --2- 172.21.15.78:0/97340857 >> v2:172.21.15.150:3300/0 conn(0x7f0a38072ee0 0x7f0a3814f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 >> v1:172.21.15.150:6789/0 conn(0x7f0a380c78a0 legacy=0x7f0a380c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a380c3b80 con 0x7f0a3814ef50 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f0a34001670 con 0x7f0a3814ef50 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 >> v1:172.21.15.78:6789/0 conn(0x7f0a3814ef50 legacy=0x7f0a380727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a24003970 0x7f0a24003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a24004370 0x7f0a24004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24005040 0x7f0a2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a380c4bc0 con 0x7f0a24003970 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2400bfa0 con 0x7f0a24005040 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2400c160 con 0x7f0a24004370 2024-02-19T15:00:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a24003970 0x7f0a24003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3ece5640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24005040 0x7f0a2400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3dce3640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a24004370 0x7f0a24004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3e4e4640 1 -- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a24004370 msgr2=0x7f0a24004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a24004370 0x7f0a24004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3e4e4640 1 -- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24005040 msgr2=0x7f0a2400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24005040 0x7f0a2400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3ece5640 1 -- 172.21.15.78:0/97340857 reap_dead start 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.390+0000 7f0a3e4e4640 1 -- 172.21.15.78:0/97340857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a24003520 con 0x7f0a24003970 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.391+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a24003970 0x7f0a24003d50 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c001860 tx=0x7f0a240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a1e59b974dab9d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.391+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c033410 con 0x7f0a24003970 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.391+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2c033590 con 0x7f0a24003970 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.391+0000 7f0a3d4e2640 1 -- 172.21.15.78:0/97340857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c033940 con 0x7f0a24003970 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 -- 172.21.15.78:0/97340857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a24003970 msgr2=0x7f0a24003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a24003970 0x7f0a24003d50 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c001860 tx=0x7f0a240057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 -- 172.21.15.78:0/97340857 shutdown_connections 2024-02-19T15:00:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 --2- 172.21.15.78:0/97340857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a24003970 0x7f0a24003d50 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 -- 172.21.15.78:0/97340857 >> 172.21.15.78:0/97340857 conn(0x7f0a38066e70 msgr2=0x7f0a380a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 -- 172.21.15.78:0/97340857 shutdown_connections 2024-02-19T15:00:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.392+0000 7f0a40748640 1 -- 172.21.15.78:0/97340857 wait complete. 2024-02-19T15:00:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.393+0000 7f0a40748640 1 Processor -- start 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.393+0000 7f0a40748640 1 -- start start 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24003970 0x7f0a380c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a381d7e30 0x7f0a3814a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 0x7f0a3814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a381daa30 con 0x7f0a3814b5b0 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3814b430 con 0x7f0a24003970 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a380ac1b0 con 0x7f0a381d7e30 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 0x7f0a3814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 0x7f0a3814aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46246/0 (socket says 172.21.15.78:46246) 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 -- 172.21.15.78:0/129219319 learned_addr learned my addr 172.21.15.78:0/129219319 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24003970 0x7f0a380c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3dce3640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a381d7e30 0x7f0a3814a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 -- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a381d7e30 msgr2=0x7f0a3814a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a381d7e30 0x7f0a3814a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 -- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24003970 msgr2=0x7f0a380c0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24003970 0x7f0a380c0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 -- 172.21.15.78:0/129219319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3814b190 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3dce3640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a381d7e30 0x7f0a3814a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24003970 0x7f0a380c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a3ece5640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 0x7f0a3814aa90 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f0a380aa4f0 tx=0x7f0a300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3000e070 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a30011e40 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a381eaed0 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a30003990 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.394+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a380c66c0 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.395+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a380ac5e0 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.396+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0a30003af0 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.396+0000 7f0a2affd640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0a1c06b840 0x7f0a1c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.396+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0a3008a930 con 0x7f0a3814b5b0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.396+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0a1c06b840 0x7f0a1c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.397+0000 7f0a3e4e4640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0a1c06b840 0x7f0a1c06dd00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c035000 tx=0x7f0a2c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.401+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0a380ac5e0 con 0x7f0a3814b5b0 2024-02-19T15:00:36.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.570+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a380ac320 con 0x7f0a3814b5b0 2024-02-19T15:00:36.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.571+0000 7f0a2affd640 1 -- 172.21.15.78:0/129219319 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16940 (secure 0 0 0) 0x7f0a380ac320 con 0x7f0a3814b5b0 2024-02-19T15:00:36.574 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:36.574 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":202,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:56.830521+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:53.390533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:54.145380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:36.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0a1c06b840 msgr2=0x7f0a1c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0a1c06b840 0x7f0a1c06dd00 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c035000 tx=0x7f0a2c009030 comp rx=0 tx=0).stop 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 msgr2=0x7f0a3814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 0x7f0a3814aa90 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f0a380aa4f0 tx=0x7f0a300115a0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 shutdown_connections 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0a1c06b840 0x7f0a1c06dd00 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a3814b5b0 0x7f0a3814aa90 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a381d7e30 0x7f0a3814a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 --2- 172.21.15.78:0/129219319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a24003970 0x7f0a380c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 >> 172.21.15.78:0/129219319 conn(0x7f0a38066e70 msgr2=0x7f0a380a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 shutdown_connections 2024-02-19T15:00:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.573+0000 7f0a40748640 1 -- 172.21.15.78:0/129219319 wait complete. 2024-02-19T15:00:36.587 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:00:36.588 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.685+0000 7f6fb5548640 1 Processor -- start 2024-02-19T15:00:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6fb5548640 1 -- start start 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6fb5548640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6fb5548640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6fb006d910 0x7f6fb006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6fb5548640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb006cd20 con 0x7f6fb0152700 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6fb5548640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb01551f0 con 0x7f6fb006d910 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6fb5548640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0155490 con 0x7f6fb00c8e00 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6faf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6fb006d910 0x7f6fb006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6faeffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6fb0152700 0x7f6fb00c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41818/0 (socket says 172.21.15.78:41818) 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.686+0000 7f6faeffd640 1 -- 172.21.15.78:0/1075363954 learned_addr learned my addr 172.21.15.78:0/1075363954 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6fae7fc640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1075363954 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 msgr2=0x7f6fb006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6faf7fe640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1075363954 >> v1:172.21.15.78:6789/0 conn(0x7f6fb0152700 legacy=0x7f6fb00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6fadffb640 1 -- 172.21.15.78:0/1075363954 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840025868 0 0) 0x7f6fb006cd20 con 0x7f6fb0152700 2024-02-19T15:00:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1075363954 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0155770 con 0x7f6fb006d910 2024-02-19T15:00:36.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6fae7fc640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.687+0000 7f6faf7fe640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb006d910 0x7f6fb006c7e0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009d90 tx=0x7f6fa802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5ac5db437e4ed4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fadffb640 1 -- 172.21.15.78:0/1075363954 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802cbc0 con 0x7f6fb006d910 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fadffb640 1 -- 172.21.15.78:0/1075363954 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802cd20 con 0x7f6fb006d910 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fadffb640 1 -- 172.21.15.78:0/1075363954 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8034660 con 0x7f6fb006d910 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 -- 172.21.15.78:0/1075363954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb006d910 msgr2=0x7f6fb006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb006d910 0x7f6fb006c7e0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009d90 tx=0x7f6fa802be10 comp rx=0 tx=0).stop 2024-02-19T15:00:36.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 -- 172.21.15.78:0/1075363954 shutdown_connections 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb006d910 0x7f6fb006c7e0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1075363954 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 -- 172.21.15.78:0/1075363954 >> 172.21.15.78:0/1075363954 conn(0x7f6fb00cac90 msgr2=0x7f6fb00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 -- 172.21.15.78:0/1075363954 shutdown_connections 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.688+0000 7f6fb5548640 1 -- 172.21.15.78:0/1075363954 wait complete. 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.689+0000 7f6fb5548640 1 Processor -- start 2024-02-19T15:00:36.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 -- start start 2024-02-19T15:00:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fb006d910 0x7f6fb00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb01ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 0x7f6fb0153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00c3930 con 0x7f6fb0152700 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00c3ce0 con 0x7f6fb006d910 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fb5548640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00c4950 con 0x7f6fb00c8e00 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 0x7f6fb0153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 0x7f6fb0153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46268/0 (socket says 172.21.15.78:46268) 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1693513342 learned_addr learned my addr 172.21.15.78:0/1693513342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6fae7fc640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb01ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.690+0000 7f6faeffd640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fb006d910 0x7f6fb00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 msgr2=0x7f6fb01ea900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faf7fe640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb01ea900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fb006d910 msgr2=0x7f6fb00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faf7fe640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fb006d910 0x7f6fb00c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faf7fe640 1 -- 172.21.15.78:0/1693513342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8009a70 con 0x7f6fb0152700 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faeffd640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fb006d910 0x7f6fb00c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:36.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6fae7fc640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb01ea900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6faf7fe640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 0x7f6fb0153cb0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8037040 tx=0x7f6fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003b90 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802ce90 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00ac1c0 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8004030 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.691+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb00c2540 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.692+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb0074990 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.692+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6fa8004190 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.693+0000 7f6f977fe640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f9c06b840 0x7f6f9c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.693+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6fa80aca90 con 0x7f6fb0152700 2024-02-19T15:00:36.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.693+0000 7f6faeffd640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f9c06b840 0x7f6f9c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:36.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.693+0000 7f6faeffd640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f9c06b840 0x7f6f9c06dd00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0000f60 tx=0x7f6fa0007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:36.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.697+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6fa80773b0 con 0x7f6fb0152700 2024-02-19T15:00:36.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.903+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb00c8770 con 0x7f6fb0152700 2024-02-19T15:00:36.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.904+0000 7f6f977fe640 1 -- 172.21.15.78:0/1693513342 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6fa807aa70 con 0x7f6fb0152700 2024-02-19T15:00:36.907 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:36.907 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":184,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10524,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2906585066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2906585066},{"type":"v1","addr":"172.21.15.78:6837","nonce":2906585066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:55.177591+0000","modified":"2024-02-19T15:00:15.727959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-19T15:00:36.907 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 184 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f9c06b840 msgr2=0x7f6f9c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f9c06b840 0x7f6f9c06dd00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0000f60 tx=0x7f6fa0007ca0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 msgr2=0x7f6fb0153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 0x7f6fb0153cb0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8037040 tx=0x7f6fa8004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 shutdown_connections 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6f9c06b840 0x7f6f9c06dd00 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fb0152700 0x7f6fb0153cb0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6fb00c8e00 0x7f6fb01ea900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 --2- 172.21.15.78:0/1693513342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fb006d910 0x7f6fb00c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 >> 172.21.15.78:0/1693513342 conn(0x7f6fb00cac90 msgr2=0x7f6fb006c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:36.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 shutdown_connections 2024-02-19T15:00:36.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:36.906+0000 7f6fb5548640 1 -- 172.21.15.78:0/1693513342 wait complete. 2024-02-19T15:00:36.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:37.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.017+0000 7f2eef1ac640 1 Processor -- start 2024-02-19T15:00:37.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.017+0000 7f2eef1ac640 1 -- start start 2024-02-19T15:00:37.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eef1ac640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ee80c8500 0x7f2ee80c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eef1ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2ee80c42e0 0x7f2ee8158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eef1ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80c2f50 0x7f2ee8158b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eef1ac640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee806c090 con 0x7f2ee80c8500 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eef1ac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee806d6a0 con 0x7f2ee80c42e0 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eef1ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80c34a0 con 0x7f2ee80c2f50 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ee80c8500 0x7f2ee80c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ee80c8500 0x7f2ee80c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46272/0 (socket says 172.21.15.78:46272) 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 -- 172.21.15.78:0/715981160 learned_addr learned my addr 172.21.15.78:0/715981160 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2edffff640 1 --2- 172.21.15.78:0/715981160 >> v2:172.21.15.150:3300/0 conn(0x7f2ee80c42e0 0x7f2ee8158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eed749640 1 --2- 172.21.15.78:0/715981160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80c2f50 0x7f2ee8158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 -- 172.21.15.78:0/715981160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80c2f50 msgr2=0x7f2ee8158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 --2- 172.21.15.78:0/715981160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80c2f50 0x7f2ee8158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 -- 172.21.15.78:0/715981160 >> v2:172.21.15.150:3300/0 conn(0x7f2ee80c42e0 msgr2=0x7f2ee8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:37.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 --2- 172.21.15.78:0/715981160 >> v2:172.21.15.150:3300/0 conn(0x7f2ee80c42e0 0x7f2ee8158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.018+0000 7f2eecf48640 1 -- 172.21.15.78:0/715981160 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee806c8e0 con 0x7f2ee80c8500 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.019+0000 7f2eecf48640 1 --2- 172.21.15.78:0/715981160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee80c88e0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80c9320 tx=0x7f2ee402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdb85f6a302df19 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.019+0000 7f2edf7fe640 1 -- 172.21.15.78:0/715981160 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40317c0 con 0x7f2ee80c8500 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.019+0000 7f2edf7fe640 1 -- 172.21.15.78:0/715981160 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee4031920 con 0x7f2ee80c8500 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.019+0000 7f2edf7fe640 1 -- 172.21.15.78:0/715981160 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4031cb0 con 0x7f2ee80c8500 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 -- 172.21.15.78:0/715981160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 msgr2=0x7f2ee80c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/715981160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee80c88e0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80c9320 tx=0x7f2ee402b860 comp rx=0 tx=0).stop 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 -- 172.21.15.78:0/715981160 shutdown_connections 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/715981160 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80c2f50 0x7f2ee8158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/715981160 >> v2:172.21.15.150:3300/0 conn(0x7f2ee80c42e0 0x7f2ee8158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/715981160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee80c88e0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 -- 172.21.15.78:0/715981160 >> 172.21.15.78:0/715981160 conn(0x7f2ee80aa300 msgr2=0x7f2ee80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 -- 172.21.15.78:0/715981160 shutdown_connections 2024-02-19T15:00:37.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.020+0000 7f2eef1ac640 1 -- 172.21.15.78:0/715981160 wait complete. 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.021+0000 7f2eef1ac640 1 Processor -- start 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.021+0000 7f2eef1ac640 1 -- start start 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ee80c2f50 0x7f2ee81503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee8151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80ab560 0x7f2ee80a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8150b30 con 0x7f2ee80c8500 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee814db70 con 0x7f2ee80c2f50 2024-02-19T15:00:37.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee81eb6c0 con 0x7f2ee80ab560 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee8151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee8151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46288/0 (socket says 172.21.15.78:46288) 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 -- 172.21.15.78:0/2715502629 learned_addr learned my addr 172.21.15.78:0/2715502629 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eecf48640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ee80c2f50 0x7f2ee81503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eed749640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80ab560 0x7f2ee80a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 -- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80ab560 msgr2=0x7f2ee80a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80ab560 0x7f2ee80a9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 -- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ee80c2f50 msgr2=0x7f2ee81503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ee80c2f50 0x7f2ee81503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 -- 172.21.15.78:0/2715502629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40094d0 con 0x7f2ee80c8500 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edffff640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee8151c20 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800d800 tx=0x7f2ed800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed800b6c0 con 0x7f2ee80c8500 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed800b820 con 0x7f2ee80c8500 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee814de10 con 0x7f2ee80c8500 2024-02-19T15:00:37.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed800bbb0 con 0x7f2ee80c8500 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.022+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee80a9a50 con 0x7f2ee80c8500 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.023+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee8071c20 con 0x7f2ee80c8500 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.024+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2ed800bd10 con 0x7f2ee80c8500 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.024+0000 7f2edd7fa640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ed406b810 0x7f2ed406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.024+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2ed808a7e0 con 0x7f2ee80c8500 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.024+0000 7f2eecf48640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ed406b810 0x7f2ed406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.025+0000 7f2eecf48640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ed406b810 0x7f2ed406dcd0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4036000 tx=0x7f2ee403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.029+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ed8055100 con 0x7f2ee80c8500 2024-02-19T15:00:37.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.238+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ee814dfa0 con 0x7f2ee80c8500 2024-02-19T15:00:37.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.239+0000 7f2edd7fa640 1 -- 172.21.15.78:0/2715502629 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f2ed80587c0 con 0x7f2ee80c8500 2024-02-19T15:00:37.242 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:37.242 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":184,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10524,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2906585066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2906585066},{"type":"v1","addr":"172.21.15.78:6837","nonce":2906585066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T14:59:55.177591+0000","modified":"2024-02-19T15:00:15.727959+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-19T15:00:37.242 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 184 2024-02-19T15:00:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ed406b810 msgr2=0x7f2ed406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ed406b810 0x7f2ed406dcd0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4036000 tx=0x7f2ee403d040 comp rx=0 tx=0).stop 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 msgr2=0x7f2ee8151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee8151c20 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800d800 tx=0x7f2ed800dcc0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 shutdown_connections 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ed406b810 0x7f2ed406dcd0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ee80c8500 0x7f2ee8151c20 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ee80ab560 0x7f2ee80a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 --2- 172.21.15.78:0/2715502629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ee80c2f50 0x7f2ee81503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 >> 172.21.15.78:0/2715502629 conn(0x7f2ee80aa300 msgr2=0x7f2ee80c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 shutdown_connections 2024-02-19T15:00:37.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.241+0000 7f2eef1ac640 1 -- 172.21.15.78:0/2715502629 wait complete. 2024-02-19T15:00:37.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:37.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 Processor -- start 2024-02-19T15:00:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 -- start start 2024-02-19T15:00:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fad700ac390 0x7fad700ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad700c55f0 0x7fad7015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad70158c40 con 0x7fad700bdf20 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad700c8de0 con 0x7fad700ac390 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.351+0000 7fad761bc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad700c5a30 con 0x7fad700c55f0 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fad700ac390 0x7fad700ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fad700ac390 0x7fad700ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46296/0 (socket says 172.21.15.78:46296) 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 -- 172.21.15.78:0/1733630403 learned_addr learned my addr 172.21.15.78:0/1733630403 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6ffff640 1 --2- 172.21.15.78:0/1733630403 >> v2:172.21.15.150:3300/0 conn(0x7fad700c55f0 0x7fad7015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 -- 172.21.15.78:0/1733630403 >> v2:172.21.15.150:3300/0 conn(0x7fad700c55f0 msgr2=0x7fad7015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 --2- 172.21.15.78:0/1733630403 >> v2:172.21.15.150:3300/0 conn(0x7fad700c55f0 0x7fad7015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 -- 172.21.15.78:0/1733630403 >> v1:172.21.15.150:6789/0 conn(0x7fad700bdf20 legacy=0x7fad70158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.352+0000 7fad6f7fe640 1 -- 172.21.15.78:0/1733630403 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad70155a60 con 0x7fad700ac390 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad6f7fe640 1 --2- 172.21.15.78:0/1733630403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700ac390 0x7fad700ac770 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fad7006d9a0 tx=0x7fad6002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a796427055cc89 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad6e7fc640 1 -- 172.21.15.78:0/1733630403 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad60031750 con 0x7fad700ac390 2024-02-19T15:00:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad6e7fc640 1 -- 172.21.15.78:0/1733630403 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad600318b0 con 0x7fad700ac390 2024-02-19T15:00:37.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad6e7fc640 1 -- 172.21.15.78:0/1733630403 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad60031c40 con 0x7fad700ac390 2024-02-19T15:00:37.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 -- 172.21.15.78:0/1733630403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700ac390 msgr2=0x7fad700ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 --2- 172.21.15.78:0/1733630403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700ac390 0x7fad700ac770 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fad7006d9a0 tx=0x7fad6002b7f0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 -- 172.21.15.78:0/1733630403 shutdown_connections 2024-02-19T15:00:37.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 --2- 172.21.15.78:0/1733630403 >> v2:172.21.15.150:3300/0 conn(0x7fad700c55f0 0x7fad7015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 --2- 172.21.15.78:0/1733630403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700ac390 0x7fad700ac770 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 -- 172.21.15.78:0/1733630403 >> 172.21.15.78:0/1733630403 conn(0x7fad700ca890 msgr2=0x7fad700caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 -- 172.21.15.78:0/1733630403 shutdown_connections 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.353+0000 7fad761bc640 1 -- 172.21.15.78:0/1733630403 wait complete. 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.354+0000 7fad761bc640 1 Processor -- start 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 -- start start 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad700ac390 0x7fad70150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700bdf20 0x7fad700c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 0x7fad700c2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad70156330 con 0x7fad700c55f0 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7006cba0 con 0x7fad700bdf20 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad761bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad70156580 con 0x7fad700ac390 2024-02-19T15:00:37.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 0x7fad700c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad6ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 0x7fad700c2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46310/0 (socket says 172.21.15.78:46310) 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.355+0000 7fad6ffff640 1 -- 172.21.15.78:0/1534827304 learned_addr learned my addr 172.21.15.78:0/1534827304 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6effd640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700bdf20 0x7fad700c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6f7fe640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad700ac390 0x7fad70150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6ffff640 1 -- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad700ac390 msgr2=0x7fad70150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6ffff640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad700ac390 0x7fad70150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6ffff640 1 -- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700bdf20 msgr2=0x7fad700c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6ffff640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700bdf20 0x7fad700c1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6ffff640 1 -- 172.21.15.78:0/1534827304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad600094d0 con 0x7fad700c55f0 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad6ffff640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 0x7fad700c2320 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fad68007e60 tx=0x7fad6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68014420 con 0x7fad700c55f0 2024-02-19T15:00:37.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad680145a0 con 0x7fad700c55f0 2024-02-19T15:00:37.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad70156860 con 0x7fad700c55f0 2024-02-19T15:00:37.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68014950 con 0x7fad700c55f0 2024-02-19T15:00:37.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.356+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad70156e20 con 0x7fad700c55f0 2024-02-19T15:00:37.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.357+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad70074990 con 0x7fad700c55f0 2024-02-19T15:00:37.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.357+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fad68014ab0 con 0x7fad700c55f0 2024-02-19T15:00:37.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.358+0000 7fad57fff640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad5c06b810 0x7fad5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.358+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad6808abf0 con 0x7fad700c55f0 2024-02-19T15:00:37.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.358+0000 7fad6f7fe640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad5c06b810 0x7fad5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.358+0000 7fad6f7fe640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad5c06b810 0x7fad5c06dcd0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fad60035000 tx=0x7fad60008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.362+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fad68055510 con 0x7fad700c55f0 2024-02-19T15:00:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.532+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad7014fb90 con 0x7fad700c55f0 2024-02-19T15:00:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.533+0000 7fad57fff640 1 -- 172.21.15.78:0/1534827304 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+16940 (secure 0 0 0) 0x7fad68058bd0 con 0x7fad700c55f0 2024-02-19T15:00:37.536 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:37.537 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":202,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T14:59:56.830521+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-19T14:59:53.390533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-19T14:59:54.145380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:37.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad5c06b810 msgr2=0x7fad5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad5c06b810 0x7fad5c06dcd0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fad60035000 tx=0x7fad60008030 comp rx=0 tx=0).stop 2024-02-19T15:00:37.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 msgr2=0x7fad700c2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 0x7fad700c2320 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fad68007e60 tx=0x7fad6800e4b0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 shutdown_connections 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fad5c06b810 0x7fad5c06dcd0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad700c55f0 0x7fad700c2320 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700bdf20 0x7fad700c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 --2- 172.21.15.78:0/1534827304 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad700ac390 0x7fad70150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.535+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 >> 172.21.15.78:0/1534827304 conn(0x7fad700ca890 msgr2=0x7fad70154800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.536+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 shutdown_connections 2024-02-19T15:00:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.536+0000 7fad761bc640 1 -- 172.21.15.78:0/1534827304 wait complete. 2024-02-19T15:00:37.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:00:37.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.646+0000 7f92c57a1640 1 Processor -- start 2024-02-19T15:00:37.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.646+0000 7f92c57a1640 1 -- start start 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92c57a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92c00c78a0 0x7f92c00c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92c57a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c014f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92c57a1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f92c00727d0 0x7f92c0072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92c57a1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c014f8f0 con 0x7f92c00727d0 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92c57a1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00c35c0 con 0x7f92c00c78a0 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92c57a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00c38a0 con 0x7f92c014ef50 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f92c00727d0 0x7f92c0072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f92c00727d0 0x7f92c0072bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36928/0 (socket says 172.21.15.78:36928) 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 -- 172.21.15.78:0/3967048605 learned_addr learned my addr 172.21.15.78:0/3967048605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92beffd640 1 --2- 172.21.15.78:0/3967048605 >> v2:172.21.15.150:3300/0 conn(0x7f92c00c78a0 0x7f92c00c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92be7fc640 1 --2- 172.21.15.78:0/3967048605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c014f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 -- 172.21.15.78:0/3967048605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 msgr2=0x7f92c014f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 --2- 172.21.15.78:0/3967048605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c014f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 -- 172.21.15.78:0/3967048605 >> v2:172.21.15.150:3300/0 conn(0x7f92c00c78a0 msgr2=0x7f92c00c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 --2- 172.21.15.78:0/3967048605 >> v2:172.21.15.150:3300/0 conn(0x7f92c00c78a0 0x7f92c00c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.647+0000 7f92bf7fe640 1 -- 172.21.15.78:0/3967048605 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00c3b80 con 0x7f92c00727d0 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.648+0000 7f92bf7fe640 1 --2- 172.21.15.78:0/3967048605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00727d0 0x7f92c0072bf0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c80d0 tx=0x7f92b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30a5cc66e7ad7a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.648+0000 7f92bdffb640 1 -- 172.21.15.78:0/3967048605 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8031870 con 0x7f92c00727d0 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.648+0000 7f92bdffb640 1 -- 172.21.15.78:0/3967048605 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b80319d0 con 0x7f92c00727d0 2024-02-19T15:00:37.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.648+0000 7f92bdffb640 1 -- 172.21.15.78:0/3967048605 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8031d60 con 0x7f92c00727d0 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 -- 172.21.15.78:0/3967048605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00727d0 msgr2=0x7f92c0072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 --2- 172.21.15.78:0/3967048605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00727d0 0x7f92c0072bf0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c80d0 tx=0x7f92b802b910 comp rx=0 tx=0).stop 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 -- 172.21.15.78:0/3967048605 shutdown_connections 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 --2- 172.21.15.78:0/3967048605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00727d0 0x7f92c0072bf0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 --2- 172.21.15.78:0/3967048605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c014f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 --2- 172.21.15.78:0/3967048605 >> v2:172.21.15.150:3300/0 conn(0x7f92c00c78a0 0x7f92c00c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 -- 172.21.15.78:0/3967048605 >> 172.21.15.78:0/3967048605 conn(0x7f92c0066e70 msgr2=0x7f92c00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 -- 172.21.15.78:0/3967048605 shutdown_connections 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.649+0000 7f92c57a1640 1 -- 172.21.15.78:0/3967048605 wait complete. 2024-02-19T15:00:37.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.650+0000 7f92c57a1640 1 Processor -- start 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.650+0000 7f92c57a1640 1 -- start start 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92c57a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92c00727d0 0x7f92c013f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92c57a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 0x7f92c01ebb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92c57a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c01ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92c57a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c013c410 con 0x7f92c00c78a0 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92c57a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c01eb6c0 con 0x7f92c00727d0 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92c57a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c01ec5d0 con 0x7f92c014ef50 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 0x7f92c01ebb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 0x7f92c01ebb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36940/0 (socket says 172.21.15.78:36940) 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 -- 172.21.15.78:0/166572012 learned_addr learned my addr 172.21.15.78:0/166572012 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92beffd640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92c00727d0 0x7f92c013f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92bf7fe640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c01ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 -- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 msgr2=0x7f92c01ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c01ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 -- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92c00727d0 msgr2=0x7f92c013f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92c00727d0 0x7f92c013f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 -- 172.21.15.78:0/166572012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b80094d0 con 0x7f92c00c78a0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92be7fc640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 0x7f92c01ebb50 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f92b400d5f0 tx=0x7f92b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b40125f0 con 0x7f92c00c78a0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.651+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b4012750 con 0x7f92c00c78a0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.652+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c00abf60 con 0x7f92c00c78a0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.652+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012b00 con 0x7f92c00c78a0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.652+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c00ac520 con 0x7f92c00c78a0 2024-02-19T15:00:37.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.652+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c00ac6b0 con 0x7f92c00c78a0 2024-02-19T15:00:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.653+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f92b4012c60 con 0x7f92c00c78a0 2024-02-19T15:00:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.653+0000 7f92a77fe640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f92ac06b840 0x7f92ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.653+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92b408a760 con 0x7f92c00c78a0 2024-02-19T15:00:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.653+0000 7f92beffd640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f92ac06b840 0x7f92ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.654+0000 7f92beffd640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f92ac06b840 0x7f92ac06dd00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f92b0004410 tx=0x7f92b00039f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.658+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f92b4055080 con 0x7f92c00c78a0 2024-02-19T15:00:37.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:37.864+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f92c00730e0 con 0x7f92c00c78a0 2024-02-19T15:00:38.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.229+0000 7f92a77fe640 1 -- 172.21.15.78:0/166572012 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v185) v1 ==== 161+0+0 (secure 0 0 0) 0x7f92b4058740 con 0x7f92c00c78a0 2024-02-19T15:00:38.232 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f92ac06b840 msgr2=0x7f92ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f92ac06b840 0x7f92ac06dd00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f92b0004410 tx=0x7f92b00039f0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 msgr2=0x7f92c01ebb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 0x7f92c01ebb50 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f92b400d5f0 tx=0x7f92b400dab0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 shutdown_connections 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f92ac06b840 0x7f92ac06dd00 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92c00c78a0 0x7f92c01ebb50 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92c014ef50 0x7f92c01ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 --2- 172.21.15.78:0/166572012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92c00727d0 0x7f92c013f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 >> 172.21.15.78:0/166572012 conn(0x7f92c0066e70 msgr2=0x7f92c014faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 shutdown_connections 2024-02-19T15:00:38.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.232+0000 7f92c57a1640 1 -- 172.21.15.78:0/166572012 wait complete. 2024-02-19T15:00:38.246 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:00:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.346+0000 7f48ca12c640 1 Processor -- start 2024-02-19T15:00:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.346+0000 7f48ca12c640 1 -- start start 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48ca12c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48c414cd50 0x7f48c40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48ca12c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f48c41537a0 0x7f48c40c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48ca12c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c41563b0 con 0x7f48c40c7b70 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48ca12c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c4153cd0 con 0x7f48c41537a0 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48ca12c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c406b6e0 con 0x7f48c414cd50 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48c2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f48c41537a0 0x7f48c40c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48c3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f48c40c7b70 0x7f48c4156860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41980/0 (socket says 172.21.15.78:41980) 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48c2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f48c41537a0 0x7f48c40c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36946/0 (socket says 172.21.15.78:36946) 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.347+0000 7f48c3fff640 1 -- 172.21.15.78:0/1671497063 learned_addr learned my addr 172.21.15.78:0/1671497063 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c37fe640 1 --2- 172.21.15.78:0/1671497063 >> v2:172.21.15.150:3300/0 conn(0x7f48c414cd50 0x7f48c40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c2ffd640 1 -- 172.21.15.78:0/1671497063 >> v2:172.21.15.150:3300/0 conn(0x7f48c414cd50 msgr2=0x7f48c40c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c2ffd640 1 --2- 172.21.15.78:0/1671497063 >> v2:172.21.15.150:3300/0 conn(0x7f48c414cd50 0x7f48c40c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c2ffd640 1 -- 172.21.15.78:0/1671497063 >> v1:172.21.15.78:6789/0 conn(0x7f48c40c7b70 legacy=0x7f48c4156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c27fc640 1 -- 172.21.15.78:0/1671497063 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765827637 0 0) 0x7f48c41563b0 con 0x7f48c40c7b70 2024-02-19T15:00:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c2ffd640 1 -- 172.21.15.78:0/1671497063 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c406b980 con 0x7f48c41537a0 2024-02-19T15:00:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c37fe640 1 --2- 172.21.15.78:0/1671497063 >> v2:172.21.15.150:3300/0 conn(0x7f48c414cd50 0x7f48c40c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.348+0000 7f48c2ffd640 1 --2- 172.21.15.78:0/1671497063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c41537a0 0x7f48c40c7630 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f48c40ace20 tx=0x7f48b802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240b52703675cca5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48c27fc640 1 -- 172.21.15.78:0/1671497063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8031710 con 0x7f48c41537a0 2024-02-19T15:00:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48c27fc640 1 -- 172.21.15.78:0/1671497063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b802f050 con 0x7f48c41537a0 2024-02-19T15:00:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48c27fc640 1 -- 172.21.15.78:0/1671497063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8031a90 con 0x7f48c41537a0 2024-02-19T15:00:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 -- 172.21.15.78:0/1671497063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c41537a0 msgr2=0x7f48c40c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 --2- 172.21.15.78:0/1671497063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c41537a0 0x7f48c40c7630 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f48c40ace20 tx=0x7f48b802b7b0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 -- 172.21.15.78:0/1671497063 shutdown_connections 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 --2- 172.21.15.78:0/1671497063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c41537a0 0x7f48c40c7630 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 --2- 172.21.15.78:0/1671497063 >> v2:172.21.15.150:3300/0 conn(0x7f48c414cd50 0x7f48c40c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 -- 172.21.15.78:0/1671497063 >> 172.21.15.78:0/1671497063 conn(0x7f48c4066e70 msgr2=0x7f48c40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 -- 172.21.15.78:0/1671497063 shutdown_connections 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.349+0000 7f48ca12c640 1 -- 172.21.15.78:0/1671497063 wait complete. 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.350+0000 7f48ca12c640 1 Processor -- start 2024-02-19T15:00:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.350+0000 7f48ca12c640 1 -- start start 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48ca12c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c40c7b70 0x7f48c40c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48ca12c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 0x7f48c41e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48ca12c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48c41537a0 0x7f48c41e9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48ca12c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c41e86c0 con 0x7f48c414cd50 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48ca12c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c4150e30 con 0x7f48c40c7b70 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48ca12c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c414e600 con 0x7f48c41537a0 2024-02-19T15:00:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 0x7f48c41e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 0x7f48c41e8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36960/0 (socket says 172.21.15.78:36960) 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 -- 172.21.15.78:0/553389888 learned_addr learned my addr 172.21.15.78:0/553389888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c37fe640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c40c7b70 0x7f48c40c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c3fff640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48c41537a0 0x7f48c41e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 -- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48c41537a0 msgr2=0x7f48c41e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48c41537a0 0x7f48c41e9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 -- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c40c7b70 msgr2=0x7f48c40c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c40c7b70 0x7f48c40c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c2ffd640 1 -- 172.21.15.78:0/553389888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b8009400 con 0x7f48c414cd50 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.351+0000 7f48c37fe640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c40c7b70 0x7f48c40c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.352+0000 7f48c2ffd640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 0x7f48c41e8df0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f48b8036040 tx=0x7f48b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.352+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8004230 con 0x7f48c414cd50 2024-02-19T15:00:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.352+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8004390 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.352+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c41ec760 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.352+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8033970 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.352+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c41e88d0 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.353+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c41e8ac0 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.353+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f48b803a040 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.354+0000 7f48abfff640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f48b006b810 0x7f48b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.354+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48b80abb60 con 0x7f48c414cd50 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.354+0000 7f48c37fe640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f48b006b810 0x7f48b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.354+0000 7f48c37fe640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f48b006b810 0x7f48b006dcd0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f48b400a990 tx=0x7f48b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.358+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f48b80765a0 con 0x7f48c414cd50 2024-02-19T15:00:38.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:38.565+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f48c4074990 con 0x7f48c414cd50 2024-02-19T15:00:39.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.235+0000 7f48abfff640 1 -- 172.21.15.78:0/553389888 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v186) v1 ==== 104+0+0 (secure 0 0 0) 0x7f48b8079c60 con 0x7f48c414cd50 2024-02-19T15:00:39.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.237+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f48b006b810 msgr2=0x7f48b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.237+0000 7f48ca12c640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f48b006b810 0x7f48b006dcd0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f48b400a990 tx=0x7f48b4008040 comp rx=0 tx=0).stop 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.237+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 msgr2=0x7f48c41e8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.237+0000 7f48ca12c640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 0x7f48c41e8df0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f48b8036040 tx=0x7f48b80046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 shutdown_connections 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f48b006b810 0x7f48b006dcd0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48c414cd50 0x7f48c41e8df0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48c41537a0 0x7f48c41e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 --2- 172.21.15.78:0/553389888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48c40c7b70 0x7f48c40c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 >> 172.21.15.78:0/553389888 conn(0x7f48c4066e70 msgr2=0x7f48c40c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 shutdown_connections 2024-02-19T15:00:39.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.238+0000 7f48ca12c640 1 -- 172.21.15.78:0/553389888 wait complete. 2024-02-19T15:00:39.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:00:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.356+0000 7fcd29676640 1 Processor -- start 2024-02-19T15:00:39.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.356+0000 7fcd29676640 1 -- start start 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd29676640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd24152620 0x7fcd240c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd29676640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd24159210 con 0x7fcd24155010 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd29676640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd240c6420 con 0x7fcd240c6d50 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd29676640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd240c1f00 con 0x7fcd24152620 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd22ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcd24152620 0x7fcd240c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd237fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcd24155010 0x7fcd24155460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:41984/0 (socket says 172.21.15.78:41984) 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd237fe640 1 -- 172.21.15.78:0/3487761204 learned_addr learned my addr 172.21.15.78:0/3487761204 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd22ffd640 1 -- 172.21.15.78:0/3487761204 >> v1:172.21.15.150:6789/0 conn(0x7fcd240c6d50 legacy=0x7fcd2406d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd22ffd640 1 -- 172.21.15.78:0/3487761204 >> v1:172.21.15.78:6789/0 conn(0x7fcd24155010 legacy=0x7fcd24155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd22ffd640 1 -- 172.21.15.78:0/3487761204 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24158600 con 0x7fcd24152620 2024-02-19T15:00:39.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.357+0000 7fcd21ffb640 1 -- 172.21.15.78:0/3487761204 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067478133 0 0) 0x7fcd24159210 con 0x7fcd24155010 2024-02-19T15:00:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.358+0000 7fcd22ffd640 1 --2- 172.21.15.78:0/3487761204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c6810 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fcd14009d90 tx=0x7fcd1402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddfc3d4e3e76a86 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.358+0000 7fcd21ffb640 1 -- 172.21.15.78:0/3487761204 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1402cb40 con 0x7fcd24152620 2024-02-19T15:00:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.358+0000 7fcd21ffb640 1 -- 172.21.15.78:0/3487761204 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1402cca0 con 0x7fcd24152620 2024-02-19T15:00:39.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.358+0000 7fcd21ffb640 1 -- 172.21.15.78:0/3487761204 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14034660 con 0x7fcd24152620 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 -- 172.21.15.78:0/3487761204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 msgr2=0x7fcd240c6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 --2- 172.21.15.78:0/3487761204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c6810 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fcd14009d90 tx=0x7fcd1402bd90 comp rx=0 tx=0).stop 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 -- 172.21.15.78:0/3487761204 shutdown_connections 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 --2- 172.21.15.78:0/3487761204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c6810 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 -- 172.21.15.78:0/3487761204 >> 172.21.15.78:0/3487761204 conn(0x7fcd240caae0 msgr2=0x7fcd240caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 -- 172.21.15.78:0/3487761204 shutdown_connections 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.359+0000 7fcd29676640 1 -- 172.21.15.78:0/3487761204 wait complete. 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.360+0000 7fcd29676640 1 Processor -- start 2024-02-19T15:00:39.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.360+0000 7fcd29676640 1 -- start start 2024-02-19T15:00:39.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd240c6d50 0x7fcd24157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd24155010 0x7fcd240c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240c9660 con 0x7fcd24152620 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2414af80 con 0x7fcd240c6d50 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd241eaa50 con 0x7fcd24155010 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36970/0 (socket says 172.21.15.78:36970) 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 -- 172.21.15.78:0/4186905584 learned_addr learned my addr 172.21.15.78:0/4186905584 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd22ffd640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd240c6d50 0x7fcd24157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd237fe640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd24155010 0x7fcd240c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 -- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd24155010 msgr2=0x7fcd240c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd24155010 0x7fcd240c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 -- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd240c6d50 msgr2=0x7fcd24157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd240c6d50 0x7fcd24157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 -- 172.21.15.78:0/4186905584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd14009a70 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd227fc640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c8be0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800c430 tx=0x7fcd1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18011790 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd180118f0 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18011ce0 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.361+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd241ead30 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.362+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd241eb2f0 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.363+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd24074990 con 0x7fcd24152620 2024-02-19T15:00:39.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.363+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcd1801c460 con 0x7fcd24152620 2024-02-19T15:00:39.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.363+0000 7fcd0b7fe640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd1006b840 0x7fcd1006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:39.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.364+0000 7fcd22ffd640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd1006b840 0x7fcd1006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:39.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.364+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcd1808a290 con 0x7fcd24152620 2024-02-19T15:00:39.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.364+0000 7fcd22ffd640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd1006b840 0x7fcd1006dd00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcd14036000 tx=0x7fcd14030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:39.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.368+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd18054bb0 con 0x7fcd24152620 2024-02-19T15:00:39.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:39.538+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcd240c65f0 con 0x7fcd24152620 2024-02-19T15:00:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.243+0000 7fcd0b7fe640 1 -- 172.21.15.78:0/4186905584 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v203) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcd18058270 con 0x7fcd24152620 2024-02-19T15:00:40.246 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:00:40.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd1006b840 msgr2=0x7fcd1006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:40.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd1006b840 0x7fcd1006dd00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcd14036000 tx=0x7fcd14030000 comp rx=0 tx=0).stop 2024-02-19T15:00:40.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 msgr2=0x7fcd240c8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:40.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c8be0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800c430 tx=0x7fcd1800c8f0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 shutdown_connections 2024-02-19T15:00:40.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcd1006b840 0x7fcd1006dd00 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd24152620 0x7fcd240c8be0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd24155010 0x7fcd240c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 --2- 172.21.15.78:0/4186905584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd240c6d50 0x7fcd24157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 >> 172.21.15.78:0/4186905584 conn(0x7fcd240caae0 msgr2=0x7fcd240c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:40.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 shutdown_connections 2024-02-19T15:00:40.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.246+0000 7fcd29676640 1 -- 172.21.15.78:0/4186905584 wait complete. 2024-02-19T15:00:40.261 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:00:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.364+0000 7f406d183640 1 Processor -- start 2024-02-19T15:00:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.364+0000 7f406d183640 1 -- start start 2024-02-19T15:00:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406d183640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40680c8730 0x7f406806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406d183640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4068154ea0 0x7f4068155330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406d183640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406806dc70 con 0x7f4068152670 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406d183640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4068158ef0 con 0x7f4068154ea0 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406d183640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40680be190 con 0x7f40680c8730 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4068154ea0 0x7f4068155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4068154ea0 0x7f4068155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36976/0 (socket says 172.21.15.78:36976) 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 -- 172.21.15.78:0/1721018219 learned_addr learned my addr 172.21.15.78:0/1721018219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 -- 172.21.15.78:0/1721018219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40680c8730 msgr2=0x7f406806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 --2- 172.21.15.78:0/1721018219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40680c8730 0x7f406806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 -- 172.21.15.78:0/1721018219 >> v1:172.21.15.150:6789/0 conn(0x7f4068152670 legacy=0x7f40680c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.365+0000 7f406759e640 1 -- 172.21.15.78:0/1721018219 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406806dea0 con 0x7f4068154ea0 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406759e640 1 --2- 172.21.15.78:0/1721018219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f4068155330 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f4060009e50 tx=0x7f406002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dcf9bf8fb08c54 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406659c640 1 -- 172.21.15.78:0/1721018219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406002c950 con 0x7f4068154ea0 2024-02-19T15:00:40.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406659c640 1 -- 172.21.15.78:0/1721018219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406002cab0 con 0x7f4068154ea0 2024-02-19T15:00:40.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406659c640 1 -- 172.21.15.78:0/1721018219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406002ce40 con 0x7f4068154ea0 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406d183640 1 -- 172.21.15.78:0/1721018219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 msgr2=0x7f4068155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406d183640 1 --2- 172.21.15.78:0/1721018219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f4068155330 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f4060009e50 tx=0x7f406002bdb0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406d183640 1 -- 172.21.15.78:0/1721018219 shutdown_connections 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406d183640 1 --2- 172.21.15.78:0/1721018219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f4068155330 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406d183640 1 --2- 172.21.15.78:0/1721018219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40680c8730 0x7f406806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.366+0000 7f406d183640 1 -- 172.21.15.78:0/1721018219 >> 172.21.15.78:0/1721018219 conn(0x7f4068072790 msgr2=0x7f4068072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.367+0000 7f406d183640 1 -- 172.21.15.78:0/1721018219 shutdown_connections 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.367+0000 7f406d183640 1 -- 172.21.15.78:0/1721018219 wait complete. 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 Processor -- start 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 -- start start 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680c8730 0x7f40681ea9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:40.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4068152670 0x7f406814d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f40681ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4068158320 con 0x7f4068154ea0 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4068157a80 con 0x7f40680c8730 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406d183640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4068157d60 con 0x7f4068152670 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f40681ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.368+0000 7f406759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f40681ee600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36986/0 (socket says 172.21.15.78:36986) 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 -- 172.21.15.78:0/200795427 learned_addr learned my addr 172.21.15.78:0/200795427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f4066d9d640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680c8730 0x7f40681ea9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f405ffff640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4068152670 0x7f406814d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 -- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4068152670 msgr2=0x7f406814d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4068152670 0x7f406814d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 -- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680c8730 msgr2=0x7f40681ea9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680c8730 0x7f40681ea9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 -- 172.21.15.78:0/200795427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060009b30 con 0x7f4068154ea0 2024-02-19T15:00:40.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406759e640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f40681ee600 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f4060036040 tx=0x7f4060004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:40.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060040670 con 0x7f4068154ea0 2024-02-19T15:00:40.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40600407f0 con 0x7f4068154ea0 2024-02-19T15:00:40.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4068158040 con 0x7f4068154ea0 2024-02-19T15:00:40.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060040ba0 con 0x7f4068154ea0 2024-02-19T15:00:40.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.369+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406814d8a0 con 0x7f4068154ea0 2024-02-19T15:00:40.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.370+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40681edf40 con 0x7f4068154ea0 2024-02-19T15:00:40.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.371+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4060040d00 con 0x7f4068154ea0 2024-02-19T15:00:40.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.371+0000 7f405f7fe640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f404806b840 0x7f404806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:40.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.371+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5663+0+0 (secure 0 0 0) 0x7f40600ad090 con 0x7f4068154ea0 2024-02-19T15:00:40.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.371+0000 7f4066d9d640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f404806b840 0x7f404806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:40.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.371+0000 7f4066d9d640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f404806b840 0x7f404806dd00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4058006fd0 tx=0x7f4058009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:40.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.375+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f40681edf40 con 0x7f4068154ea0 2024-02-19T15:00:40.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:40.545+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f40681edf40 con 0x7f4068154ea0 2024-02-19T15:00:41.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.251+0000 7f405f7fe640 1 -- 172.21.15.78:0/200795427 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v204) v1 ==== 169+0+0 (secure 0 0 0) 0x7f40681edf40 con 0x7f4068154ea0 2024-02-19T15:00:41.254 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.254+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f404806b840 msgr2=0x7f404806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.254+0000 7f406d183640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f404806b840 0x7f404806dd00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4058006fd0 tx=0x7f4058009060 comp rx=0 tx=0).stop 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.254+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 msgr2=0x7f40681ee600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.254+0000 7f406d183640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f40681ee600 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f4060036040 tx=0x7f4060004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 shutdown_connections 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f404806b840 0x7f404806dd00 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068154ea0 0x7f40681ee600 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4068152670 0x7f406814d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 --2- 172.21.15.78:0/200795427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680c8730 0x7f40681ea9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 >> 172.21.15.78:0/200795427 conn(0x7f4068072790 msgr2=0x7f4068072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 shutdown_connections 2024-02-19T15:00:41.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.255+0000 7f406d183640 1 -- 172.21.15.78:0/200795427 wait complete. 2024-02-19T15:00:41.272 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:00:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.376+0000 7f61ff1a2640 1 Processor -- start 2024-02-19T15:00:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.376+0000 7f61ff1a2640 1 -- start start 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61ff1a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f61f80c8dd0 0x7f61f8155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61ff1a2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f81595d0 con 0x7f61f80c7c60 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61ff1a2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8158810 con 0x7f61f8152720 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61ff1a2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f80c6090 con 0x7f61f80c8dd0 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61fe9a1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f61f80c8dd0 0x7f61f8155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61fd99f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f61f80c7c60 0x7f61f8155100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42000/0 (socket says 172.21.15.78:42000) 2024-02-19T15:00:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.377+0000 7f61fd99f640 1 -- 172.21.15.78:0/2203868024 learned_addr learned my addr 172.21.15.78:0/2203868024 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2203868024 >> v1:172.21.15.150:6789/0 conn(0x7f61f8152720 legacy=0x7f61f80c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2203868024 >> v1:172.21.15.78:6789/0 conn(0x7f61f80c7c60 legacy=0x7f61f8155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2203868024 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8158af0 con 0x7f61f80c8dd0 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fe9a1640 1 --2- 172.21.15.78:0/2203868024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f80c8dd0 0x7f61f8155810 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097a0 tx=0x7f61f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2785a728d12cb70b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fd19e640 1 -- 172.21.15.78:0/2203868024 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f002e070 con 0x7f61f80c8dd0 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fd19e640 1 -- 172.21.15.78:0/2203868024 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f00315a0 con 0x7f61f80c8dd0 2024-02-19T15:00:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.378+0000 7f61fd19e640 1 -- 172.21.15.78:0/2203868024 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0031930 con 0x7f61f80c8dd0 2024-02-19T15:00:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2203868024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f80c8dd0 msgr2=0x7f61f8155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2203868024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f80c8dd0 0x7f61f8155810 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097a0 tx=0x7f61f002b840 comp rx=0 tx=0).stop 2024-02-19T15:00:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2203868024 shutdown_connections 2024-02-19T15:00:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2203868024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f80c8dd0 0x7f61f8155810 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2203868024 >> 172.21.15.78:0/2203868024 conn(0x7f61f80ac290 msgr2=0x7f61f80ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2203868024 shutdown_connections 2024-02-19T15:00:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.379+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2203868024 wait complete. 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.380+0000 7f61ff1a2640 1 Processor -- start 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.380+0000 7f61ff1a2640 1 -- start start 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61ff1a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61f80c7c60 0x7f61f80ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61ff1a2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61f80c8dd0 0x7f61f81459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61ff1a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 0x7f61f8145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61ff1a2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80c4bc0 con 0x7f61f8152720 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61ff1a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f81eb860 con 0x7f61f80c7c60 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61ff1a2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80be7b0 con 0x7f61f80c8dd0 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 0x7f61f8145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 0x7f61f8145f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37000/0 (socket says 172.21.15.78:37000) 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2229357746 learned_addr learned my addr 172.21.15.78:0/2229357746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe1a0640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61f80c7c60 0x7f61f80ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fd99f640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61f80c8dd0 0x7f61f81459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61f80c8dd0 msgr2=0x7f61f81459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61f80c8dd0 0x7f61f81459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61f80c7c60 msgr2=0x7f61f80ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61f80c7c60 0x7f61f80ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 -- 172.21.15.78:0/2229357746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f0009400 con 0x7f61f8152720 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fd99f640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61f80c8dd0 0x7f61f81459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.381+0000 7f61fe9a1640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 0x7f61f8145f20 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f61f00096f0 tx=0x7f61f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.382+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0032840 con 0x7f61f8152720 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.382+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f00329a0 con 0x7f61f8152720 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.382+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0032d50 con 0x7f61f8152720 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.382+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f81eb530 con 0x7f61f8152720 2024-02-19T15:00:41.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.382+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f8158090 con 0x7f61f8152720 2024-02-19T15:00:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.383+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f8074990 con 0x7f61f8152720 2024-02-19T15:00:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.383+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f61f00453f0 con 0x7f61f8152720 2024-02-19T15:00:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.384+0000 7f61eeffd640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61dc06b810 0x7f61dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.384+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f61f00abf90 con 0x7f61f8152720 2024-02-19T15:00:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.384+0000 7f61fe1a0640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61dc06b810 0x7f61dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.385+0000 7f61fe1a0640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61dc06b810 0x7f61dc06dcd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f61f400b450 tx=0x7f61f400d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:41.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.390+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f61f00dfdc0 con 0x7f61f8152720 2024-02-19T15:00:41.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.560+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f61f81eb9a0 con 0x7f61f8152720 2024-02-19T15:00:41.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.564+0000 7f61eeffd640 1 -- 172.21.15.78:0/2229357746 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f61f0076d20 con 0x7f61f8152720 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61dc06b810 msgr2=0x7f61dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61dc06b810 0x7f61dc06dcd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f61f400b450 tx=0x7f61f400d040 comp rx=0 tx=0).stop 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 msgr2=0x7f61f8145f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 0x7f61f8145f20 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f61f00096f0 tx=0x7f61f00046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 shutdown_connections 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f61dc06b810 0x7f61dc06dcd0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61f8152720 0x7f61f8145f20 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61f80c8dd0 0x7f61f81459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 --2- 172.21.15.78:0/2229357746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61f80c7c60 0x7f61f80ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 >> 172.21.15.78:0/2229357746 conn(0x7f61f80ac290 msgr2=0x7f61f814d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 shutdown_connections 2024-02-19T15:00:41.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.566+0000 7f61ff1a2640 1 -- 172.21.15.78:0/2229357746 wait complete. 2024-02-19T15:00:41.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-02-19T15:00:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.681+0000 7f72f8c8f640 1 Processor -- start 2024-02-19T15:00:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.681+0000 7f72f8c8f640 1 -- start start 2024-02-19T15:00:41.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f8c8f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f72f414f1a0 0x7f72f4072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f8c8f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72f4152680 0x7f72f40c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f8c8f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f4152ab0 con 0x7f72f406ea50 2024-02-19T15:00:41.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f8c8f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40c5970 con 0x7f72f414f1a0 2024-02-19T15:00:41.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f8c8f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f40c5750 con 0x7f72f4152680 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f72f414f1a0 0x7f72f4072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f1d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f72f406ea50 0x7f72f406fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42008/0 (socket says 172.21.15.78:42008) 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f1d9b640 1 -- 172.21.15.78:0/3377730249 learned_addr learned my addr 172.21.15.78:0/3377730249 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.682+0000 7f72f259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f72f414f1a0 0x7f72f4072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37016/0 (socket says 172.21.15.78:37016) 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f2d9d640 1 --2- 172.21.15.78:0/3377730249 >> v2:172.21.15.150:3300/0 conn(0x7f72f4152680 0x7f72f40c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f259c640 1 -- 172.21.15.78:0/3377730249 >> v2:172.21.15.150:3300/0 conn(0x7f72f4152680 msgr2=0x7f72f40c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f259c640 1 --2- 172.21.15.78:0/3377730249 >> v2:172.21.15.150:3300/0 conn(0x7f72f4152680 0x7f72f40c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f259c640 1 -- 172.21.15.78:0/3377730249 >> v1:172.21.15.78:6789/0 conn(0x7f72f406ea50 legacy=0x7f72f406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f259c640 1 -- 172.21.15.78:0/3377730249 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40702d0 con 0x7f72f414f1a0 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f159a640 1 -- 172.21.15.78:0/3377730249 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944298917 0 0) 0x7f72f4152ab0 con 0x7f72f406ea50 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.683+0000 7f72f2d9d640 1 --2- 172.21.15.78:0/3377730249 >> v2:172.21.15.150:3300/0 conn(0x7f72f4152680 0x7f72f40c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f259c640 1 --2- 172.21.15.78:0/3377730249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f414f1a0 0x7f72f4072d70 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f72f406de00 tx=0x7f72ec02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc82483668a284d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f159a640 1 -- 172.21.15.78:0/3377730249 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec031710 con 0x7f72f414f1a0 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f159a640 1 -- 172.21.15.78:0/3377730249 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec02f050 con 0x7f72f414f1a0 2024-02-19T15:00:41.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f159a640 1 -- 172.21.15.78:0/3377730249 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec031a90 con 0x7f72f414f1a0 2024-02-19T15:00:41.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 -- 172.21.15.78:0/3377730249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f414f1a0 msgr2=0x7f72f4072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/3377730249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f414f1a0 0x7f72f4072d70 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f72f406de00 tx=0x7f72ec02b7b0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 -- 172.21.15.78:0/3377730249 shutdown_connections 2024-02-19T15:00:41.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/3377730249 >> v2:172.21.15.150:3300/0 conn(0x7f72f4152680 0x7f72f40c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/3377730249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f414f1a0 0x7f72f4072d70 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 -- 172.21.15.78:0/3377730249 >> 172.21.15.78:0/3377730249 conn(0x7f72f406c170 msgr2=0x7f72f406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 -- 172.21.15.78:0/3377730249 shutdown_connections 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.684+0000 7f72f8c8f640 1 -- 172.21.15.78:0/3377730249 wait complete. 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 Processor -- start 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 -- start start 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f406ea50 0x7f72f41e9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72f414f1a0 0x7f72f41556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 0x7f72f4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f4156150 con 0x7f72f4152680 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f4151810 con 0x7f72f406ea50 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f8c8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f41ea1c0 con 0x7f72f414f1a0 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 0x7f72f4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 0x7f72f4155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37032/0 (socket says 172.21.15.78:37032) 2024-02-19T15:00:41.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.686+0000 7f72f2d9d640 1 -- 172.21.15.78:0/1753933314 learned_addr learned my addr 172.21.15.78:0/1753933314 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f259c640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f406ea50 0x7f72f41e9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f2d9d640 1 -- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72f414f1a0 msgr2=0x7f72f41556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f1d9b640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72f414f1a0 0x7f72f41556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f2d9d640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72f414f1a0 0x7f72f41556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f2d9d640 1 -- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f406ea50 msgr2=0x7f72f41e9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f2d9d640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f406ea50 0x7f72f41e9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f2d9d640 1 -- 172.21.15.78:0/1753933314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec009400 con 0x7f72f4152680 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f2d9d640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 0x7f72f4155c10 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f72e400c7c0 tx=0x7f72e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4004230 con 0x7f72f4152680 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4004390 con 0x7f72f4152680 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f41ea4a0 con 0x7f72f4152680 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4007860 con 0x7f72f4152680 2024-02-19T15:00:41.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.687+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f40c2750 con 0x7f72f4152680 2024-02-19T15:00:41.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.688+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f4150300 con 0x7f72f4152680 2024-02-19T15:00:41.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.688+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f72e40079c0 con 0x7f72f4152680 2024-02-19T15:00:41.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.689+0000 7f72e2ffd640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72d406b810 0x7f72d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:41.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.689+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f72e408a230 con 0x7f72f4152680 2024-02-19T15:00:41.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.689+0000 7f72f259c640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72d406b810 0x7f72d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:41.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.689+0000 7f72f259c640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72d406b810 0x7f72d406dcd0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f72ec035000 tx=0x7f72ec008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:41.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.693+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f72e4054f60 con 0x7f72f4152680 2024-02-19T15:00:41.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:41.930+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7f72f4074990 con 0x7f72f4152680 2024-02-19T15:00:42.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.252+0000 7f72e2ffd640 1 -- 172.21.15.78:0/1753933314 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v498) v1 ==== 140+0+0 (secure 0 0 0) 0x7f72e4058620 con 0x7f72f4152680 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72d406b810 msgr2=0x7f72d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72d406b810 0x7f72d406dcd0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f72ec035000 tx=0x7f72ec008030 comp rx=0 tx=0).stop 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 msgr2=0x7f72f4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 0x7f72f4155c10 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f72e400c7c0 tx=0x7f72e400cc80 comp rx=0 tx=0).stop 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 shutdown_connections 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f72d406b810 0x7f72d406dcd0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72f4152680 0x7f72f4155c10 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72f414f1a0 0x7f72f41556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 --2- 172.21.15.78:0/1753933314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f406ea50 0x7f72f41e9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 >> 172.21.15.78:0/1753933314 conn(0x7f72f406c170 msgr2=0x7f72f414c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:42.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 shutdown_connections 2024-02-19T15:00:42.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.254+0000 7f72f8c8f640 1 -- 172.21.15.78:0/1753933314 wait complete. 2024-02-19T15:00:42.268 INFO:tasks.cephfs_test_runner:That marking a FS down does not generate a health warning ... ok 2024-02-19T15:00:42.268 INFO:tasks.cephfs_test_runner:Starting test: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down twice does not wipe old_max_mds. 2024-02-19T15:00:42.268 INFO:tasks.cephfs_test_runner:test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T15:00:42.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:42.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.366+0000 7f74272d4640 1 Processor -- start 2024-02-19T15:00:42.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.366+0000 7f74272d4640 1 -- start start 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.366+0000 7f74272d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 0x7f74200c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.366+0000 7f74272d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7420158c40 con 0x7f74200ac390 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f74272d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200acbb0 con 0x7f74200bdf20 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f74272d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74200c8de0 con 0x7f74200ac770 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7425070640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f74200ac390 0x7f7420158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42022/0 (socket says 172.21.15.78:42022) 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7425070640 1 -- 172.21.15.78:0/710865689 learned_addr learned my addr 172.21.15.78:0/710865689 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433040073 0 0) 0x7f7420158c40 con 0x7f74200ac390 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740c003520 con 0x7f74200ac390 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613786650 0 0) 0x7f740c003520 con 0x7f74200ac390 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 msgr2=0x7f74200c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 0x7f74200c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 >> v1:172.21.15.150:6789/0 conn(0x7f74200bdf20 legacy=0x7f742015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420155a60 con 0x7f74200ac390 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f7414001690 con 0x7f74200ac390 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 >> v1:172.21.15.78:6789/0 conn(0x7f74200ac390 legacy=0x7f7420158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f740c0039a0 0x7f740c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f740c0047c0 0x7f740c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 0x7f740c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74200c8de0 con 0x7f740c0047c0 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c0055f0 con 0x7f740c0039a0 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c00bec0 con 0x7f74200ac770 2024-02-19T15:00:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f742486f640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f740c0047c0 0x7f740c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7425070640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f740c0039a0 0x7f740c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.367+0000 7f7425871640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 0x7f740c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f742486f640 1 -- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 msgr2=0x7f740c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f742486f640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f74200ac770 0x7f740c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f742486f640 1 -- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f740c0039a0 msgr2=0x7f740c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f742486f640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f740c0039a0 0x7f740c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f742486f640 1 -- 172.21.15.78:0/710865689 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c003520 con 0x7f740c0047c0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f7425070640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f740c0039a0 0x7f740c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f742486f640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f740c0047c0 0x7f740c004c40 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f741c003d80 tx=0x7f740c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22cc32548cf0b3c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c032600 con 0x7f740c0047c0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c02e050 con 0x7f740c0047c0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.368+0000 7f7413fff640 1 -- 172.21.15.78:0/710865689 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0329a0 con 0x7f740c0047c0 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f74272d4640 1 -- 172.21.15.78:0/710865689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f740c0047c0 msgr2=0x7f740c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f74272d4640 1 --2- 172.21.15.78:0/710865689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f740c0047c0 0x7f740c004c40 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f741c003d80 tx=0x7f740c0057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f7425871640 1 -- 172.21.15.78:0/710865689 reap_dead start 2024-02-19T15:00:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f74272d4640 1 -- 172.21.15.78:0/710865689 shutdown_connections 2024-02-19T15:00:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f74272d4640 1 -- 172.21.15.78:0/710865689 >> 172.21.15.78:0/710865689 conn(0x7f74200ca890 msgr2=0x7f74200caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f74272d4640 1 -- 172.21.15.78:0/710865689 shutdown_connections 2024-02-19T15:00:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.369+0000 7f74272d4640 1 -- 172.21.15.78:0/710865689 wait complete. 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.370+0000 7f74272d4640 1 Processor -- start 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.370+0000 7f74272d4640 1 -- start start 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f74272d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f740c0047c0 0x7f74200c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f74272d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 0x7f74200be330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f74272d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7420151c80 0x7f74200ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f74272d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74201ee070 con 0x7f74200c11b0 2024-02-19T15:00:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f74272d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74200c4170 con 0x7f7420151c80 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f74272d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74200c4410 con 0x7f740c0047c0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 0x7f74200be330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 0x7f74200be330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37054/0 (socket says 172.21.15.78:37054) 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 -- 172.21.15.78:0/994346453 learned_addr learned my addr 172.21.15.78:0/994346453 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f7425871640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7420151c80 0x7f74200ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f7425070640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f740c0047c0 0x7f74200c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 -- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f740c0047c0 msgr2=0x7f74200c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f740c0047c0 0x7f74200c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 -- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7420151c80 msgr2=0x7f74200ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7420151c80 0x7f74200ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 -- 172.21.15.78:0/994346453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74200c46f0 con 0x7f74200c11b0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f7425871640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7420151c80 0x7f74200ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f7425070640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f740c0047c0 0x7f74200c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.371+0000 7f742486f640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 0x7f74200be330 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f741c035040 tx=0x7f741c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.372+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c03d420 con 0x7f74200c11b0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.372+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c03d5a0 con 0x7f74200c11b0 2024-02-19T15:00:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.372+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c03d950 con 0x7f74200c11b0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.372+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74201edd00 con 0x7f74200c11b0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.372+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74201ee500 con 0x7f74200c11b0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.373+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74201ee1e0 con 0x7f74200c11b0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.373+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f741c03dab0 con 0x7f74200c11b0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.374+0000 7f7411ffb640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740406b810 0x7f740406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.374+0000 7f7425070640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740406b810 0x7f740406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.374+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f741c0abb80 con 0x7f74200c11b0 2024-02-19T15:00:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.374+0000 7f7425070640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740406b810 0x7f740406dcd0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7414001cd0 tx=0x7f7414001520 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.378+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f741c0768a0 con 0x7f74200c11b0 2024-02-19T15:00:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.551+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7420074990 con 0x7f74200c11b0 2024-02-19T15:00:42.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.551+0000 7f7411ffb640 1 -- 172.21.15.78:0/994346453 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f741c037090 con 0x7f74200c11b0 2024-02-19T15:00:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:42.554 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":204,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:00:41.244493+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:42.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740406b810 msgr2=0x7f740406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740406b810 0x7f740406dcd0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7414001cd0 tx=0x7f7414001520 comp rx=0 tx=0).stop 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 msgr2=0x7f74200be330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 0x7f74200be330 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f741c035040 tx=0x7f741c0036f0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 shutdown_connections 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f740406b810 0x7f740406dcd0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74200c11b0 0x7f74200be330 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7420151c80 0x7f74200ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 --2- 172.21.15.78:0/994346453 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f740c0047c0 0x7f74200c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.553+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 >> 172.21.15.78:0/994346453 conn(0x7f74200ca890 msgr2=0x7f74200c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.554+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 shutdown_connections 2024-02-19T15:00:42.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.554+0000 7f74272d4640 1 -- 172.21.15.78:0/994346453 wait complete. 2024-02-19T15:00:42.567 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:00:42.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:00:42.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.667+0000 7f259889a640 1 Processor -- start 2024-02-19T15:00:42.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.667+0000 7f259889a640 1 -- start start 2024-02-19T15:00:42.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259889a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2594155070 0x7f25941554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259889a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25940718f0 0x7f2594071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259889a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2594159130 con 0x7f25940acc50 2024-02-19T15:00:42.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259889a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25940c2cf0 con 0x7f25940718f0 2024-02-19T15:00:42.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259889a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25940c2030 con 0x7f2594155070 2024-02-19T15:00:42.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2592d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25940718f0 0x7f2594071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259259c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f25940acc50 0x7f25940c17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42030/0 (socket says 172.21.15.78:42030) 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259259c640 1 -- 172.21.15.78:0/2019408012 learned_addr learned my addr 172.21.15.78:0/2019408012 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2591d9b640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2594155070 0x7f25941554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2592d9d640 1 -- 172.21.15.78:0/2019408012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2594155070 msgr2=0x7f25941554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2592d9d640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2594155070 0x7f25941554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2592d9d640 1 -- 172.21.15.78:0/2019408012 >> v1:172.21.15.78:6789/0 conn(0x7f25940acc50 legacy=0x7f25940c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2592d9d640 1 -- 172.21.15.78:0/2019408012 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25940ad030 con 0x7f25940718f0 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f259159a640 1 -- 172.21.15.78:0/2019408012 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609363333 0 0) 0x7f2594159130 con 0x7f25940acc50 2024-02-19T15:00:42.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.668+0000 7f2591d9b640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2594155070 0x7f25941554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:42.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.669+0000 7f2592d9d640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25940718f0 0x7f2594071d80 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097d0 tx=0x7f258c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86da0044895071c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.669+0000 7f259159a640 1 -- 172.21.15.78:0/2019408012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c031770 con 0x7f25940718f0 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.669+0000 7f259159a640 1 -- 172.21.15.78:0/2019408012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c0318d0 con 0x7f25940718f0 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.669+0000 7f259159a640 1 -- 172.21.15.78:0/2019408012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c031c60 con 0x7f25940718f0 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 -- 172.21.15.78:0/2019408012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25940718f0 msgr2=0x7f2594071d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25940718f0 0x7f2594071d80 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097d0 tx=0x7f258c02b810 comp rx=0 tx=0).stop 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 -- 172.21.15.78:0/2019408012 shutdown_connections 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25940718f0 0x7f2594071d80 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 --2- 172.21.15.78:0/2019408012 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2594155070 0x7f25941554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 -- 172.21.15.78:0/2019408012 >> 172.21.15.78:0/2019408012 conn(0x7f25940c7800 msgr2=0x7f25940c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 -- 172.21.15.78:0/2019408012 shutdown_connections 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.670+0000 7f259889a640 1 -- 172.21.15.78:0/2019408012 wait complete. 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.671+0000 7f259889a640 1 Processor -- start 2024-02-19T15:00:42.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.671+0000 7f259889a640 1 -- start start 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25940718f0 0x7f25940c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25940acc50 0x7f25940a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 0x7f25940a9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25940a8b10 con 0x7f2594155070 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25941ee6f0 con 0x7f25940acc50 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25941ee190 con 0x7f25940718f0 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 0x7f25940a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 0x7f25940a9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37062/0 (socket says 172.21.15.78:37062) 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 -- 172.21.15.78:0/2183689361 learned_addr learned my addr 172.21.15.78:0/2183689361 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259259c640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25940718f0 0x7f25940c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2591d9b640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25940acc50 0x7f25940a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 -- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25940718f0 msgr2=0x7f25940c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25940718f0 0x7f25940c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 -- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25940acc50 msgr2=0x7f25940a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25940acc50 0x7f25940a9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 -- 172.21.15.78:0/2183689361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c009400 con 0x7f2594155070 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2591d9b640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25940acc50 0x7f25940a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259259c640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25940718f0 0x7f25940c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2592d9d640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 0x7f25940a9540 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f258c034040 tx=0x7f258c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c032810 con 0x7f2594155070 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c0329b0 con 0x7f2594155070 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.672+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25940c6b30 con 0x7f2594155070 2024-02-19T15:00:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.673+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c032d60 con 0x7f2594155070 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.673+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25941ee410 con 0x7f2594155070 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.674+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25941595a0 con 0x7f2594155070 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.674+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f258c04e450 con 0x7f2594155070 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.674+0000 7f2582ffd640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f257406b810 0x7f257406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.674+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f258c0abfa0 con 0x7f2594155070 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.674+0000 7f259259c640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f257406b810 0x7f257406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.675+0000 7f259259c640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f257406b810 0x7f257406dcd0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f25840012e0 tx=0x7f2584007f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.679+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f258c0dfdc0 con 0x7f2594155070 2024-02-19T15:00:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.851+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f259406c820 con 0x7f2594155070 2024-02-19T15:00:42.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.852+0000 7f2582ffd640 1 -- 172.21.15.78:0/2183689361 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v204) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25941595a0 con 0x7f2594155070 2024-02-19T15:00:42.855 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f257406b810 msgr2=0x7f257406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f257406b810 0x7f257406dcd0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f25840012e0 tx=0x7f2584007f90 comp rx=0 tx=0).stop 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 msgr2=0x7f25940a9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 0x7f25940a9540 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f258c034040 tx=0x7f258c0046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 shutdown_connections 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f257406b810 0x7f257406dcd0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2594155070 0x7f25940a9540 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25940acc50 0x7f25940a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 --2- 172.21.15.78:0/2183689361 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25940718f0 0x7f25940c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 >> 172.21.15.78:0/2183689361 conn(0x7f25940c7800 msgr2=0x7f2594153170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 shutdown_connections 2024-02-19T15:00:42.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.854+0000 7f259889a640 1 -- 172.21.15.78:0/2183689361 wait complete. 2024-02-19T15:00:42.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-02-19T15:00:42.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.965+0000 7f8e19aed640 1 Processor -- start 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.965+0000 7f8e19aed640 1 -- start start 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e19aed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e140c2f70 0x7f8e1406c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e19aed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 0x7f8e1406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e19aed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e140717b0 con 0x7f8e140c6a50 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e19aed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e140c7560 con 0x7f8e140c2f70 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e19aed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e14071a00 con 0x7f8e14156050 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 0x7f8e1406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e12ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e140c2f70 0x7f8e1406c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 0x7f8e1406cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50876/0 (socket says 172.21.15.78:50876) 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e13fff640 1 -- 172.21.15.78:0/4223118828 learned_addr learned my addr 172.21.15.78:0/4223118828 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.966+0000 7f8e137fe640 1 --1- 172.21.15.78:0/4223118828 >> v1:172.21.15.150:6789/0 conn(0x7f8e140c6a50 0x7f8e14155940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:32834/0 (socket says 172.21.15.78:32834) 2024-02-19T15:00:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e127fc640 1 -- 172.21.15.78:0/4223118828 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001155073 0 0) 0x7f8e140717b0 con 0x7f8e140c6a50 2024-02-19T15:00:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e127fc640 1 -- 172.21.15.78:0/4223118828 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dfc003520 con 0x7f8e140c6a50 2024-02-19T15:00:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e12ffd640 1 -- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 msgr2=0x7f8e1406cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e12ffd640 1 --2- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 0x7f8e1406cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e12ffd640 1 -- 172.21.15.78:0/4223118828 >> v1:172.21.15.150:6789/0 conn(0x7f8e140c6a50 legacy=0x7f8e14155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e12ffd640 1 -- 172.21.15.78:0/4223118828 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140c8030 con 0x7f8e140c2f70 2024-02-19T15:00:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e13fff640 1 --2- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 0x7f8e1406cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:42.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.967+0000 7f8e12ffd640 1 --2- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e140c2f70 0x7f8e1406c9e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8e080097a0 tx=0x7f8e0802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b18b1939002038c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e127fc640 1 -- 172.21.15.78:0/4223118828 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0802e070 con 0x7f8e140c2f70 2024-02-19T15:00:42.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e127fc640 1 -- 172.21.15.78:0/4223118828 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e080315a0 con 0x7f8e140c2f70 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e127fc640 1 -- 172.21.15.78:0/4223118828 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08031910 con 0x7f8e140c2f70 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 -- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e140c2f70 msgr2=0x7f8e1406c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 --2- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e140c2f70 0x7f8e1406c9e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8e080097a0 tx=0x7f8e0802b840 comp rx=0 tx=0).stop 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 -- 172.21.15.78:0/4223118828 shutdown_connections 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 --2- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e14156050 0x7f8e1406cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 --2- 172.21.15.78:0/4223118828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e140c2f70 0x7f8e1406c9e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 -- 172.21.15.78:0/4223118828 >> 172.21.15.78:0/4223118828 conn(0x7f8e1406b700 msgr2=0x7f8e140aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 -- 172.21.15.78:0/4223118828 shutdown_connections 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.968+0000 7f8e19aed640 1 -- 172.21.15.78:0/4223118828 wait complete. 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.969+0000 7f8e19aed640 1 Processor -- start 2024-02-19T15:00:42.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 -- start start 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e140c2f70 0x7f8e141eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 0x7f8e140a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e14156050 0x7f8e140ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e141ea200 con 0x7f8e140c6a50 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140ac5c0 con 0x7f8e14156050 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e19aed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140a7ca0 con 0x7f8e140c2f70 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e12ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 0x7f8e140a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e12ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 0x7f8e140a9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37072/0 (socket says 172.21.15.78:37072) 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.970+0000 7f8e12ffd640 1 -- 172.21.15.78:0/3551956286 learned_addr learned my addr 172.21.15.78:0/3551956286 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e137fe640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e140c2f70 0x7f8e141eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e13fff640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e14156050 0x7f8e140ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e12ffd640 1 -- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e140c2f70 msgr2=0x7f8e141eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e12ffd640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e140c2f70 0x7f8e141eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e12ffd640 1 -- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e14156050 msgr2=0x7f8e140ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e12ffd640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e14156050 0x7f8e140ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e12ffd640 1 -- 172.21.15.78:0/3551956286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c009b30 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e137fe640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e140c2f70 0x7f8e141eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e13fff640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e14156050 0x7f8e140ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e12ffd640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 0x7f8e140a9890 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f8e08034040 tx=0x7f8e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08032630 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08032790 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e08009400 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08032b00 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.971+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e141ea780 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.972+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14074990 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.972+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8e08032c60 con 0x7f8e140c6a50 2024-02-19T15:00:42.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.973+0000 7f8dfbfff640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e0006b840 0x7f8e0006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:42.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.973+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8e080abea0 con 0x7f8e140c6a50 2024-02-19T15:00:42.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.973+0000 7f8e137fe640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e0006b840 0x7f8e0006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:42.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.973+0000 7f8e137fe640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e0006b840 0x7f8e0006dd00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8e04000f60 tx=0x7f8e04009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:42.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:42.977+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8e08076b40 con 0x7f8e140c6a50 2024-02-19T15:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.212+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f8e140c65c0 con 0x7f8e140c6a50 2024-02-19T15:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.253+0000 7f8dfbfff640 1 -- 172.21.15.78:0/3551956286 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v499) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8e0807a200 con 0x7f8e140c6a50 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e0006b840 msgr2=0x7f8e0006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e0006b840 0x7f8e0006dd00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8e04000f60 tx=0x7f8e04009040 comp rx=0 tx=0).stop 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 msgr2=0x7f8e140a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 0x7f8e140a9890 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f8e08034040 tx=0x7f8e080046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 shutdown_connections 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e0006b840 0x7f8e0006dd00 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e140c6a50 0x7f8e140a9890 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e14156050 0x7f8e140ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 --2- 172.21.15.78:0/3551956286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e140c2f70 0x7f8e141eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 >> 172.21.15.78:0/3551956286 conn(0x7f8e1406b700 msgr2=0x7f8e140c2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 shutdown_connections 2024-02-19T15:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.255+0000 7f8e19aed640 1 -- 172.21.15.78:0/3551956286 wait complete. 2024-02-19T15:00:43.269 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-19T15:00:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb2925640 1 Processor -- start 2024-02-19T15:00:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb2925640 1 -- start start 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb2925640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcbac157040 0x7fcbac0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb2925640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac071050 con 0x7fcbac06f170 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb2925640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac0c6ea0 con 0x7fcbac156550 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb2925640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac0c7120 con 0x7fcbac157040 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb0ec2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcbac157040 0x7fcbac0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb0ec2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcbac157040 0x7fcbac0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37078/0 (socket says 172.21.15.78:37078) 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbab7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcbac06f170 0x7fcbac156930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42036/0 (socket says 172.21.15.78:42036) 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.369+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/2535731699 learned_addr learned my addr 172.21.15.78:0/2535731699 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.370+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/2535731699 >> v1:172.21.15.150:6789/0 conn(0x7fcbac156550 legacy=0x7fcbac0ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.370+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/2535731699 >> v1:172.21.15.78:6789/0 conn(0x7fcbac06f170 legacy=0x7fcbac156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.370+0000 7fcbaaffd640 1 -- 172.21.15.78:0/2535731699 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324069390 0 0) 0x7fcbac071050 con 0x7fcbac06f170 2024-02-19T15:00:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.370+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/2535731699 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0c76c0 con 0x7fcbac157040 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.370+0000 7fcbb0ec2640 1 --2- 172.21.15.78:0/2535731699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac157040 0x7fcbac0c7dd0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009e50 tx=0x7fcba402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f45acae118ccfd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbaaffd640 1 -- 172.21.15.78:0/2535731699 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba402ca80 con 0x7fcbac157040 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbaaffd640 1 -- 172.21.15.78:0/2535731699 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba402cbe0 con 0x7fcbac157040 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbaaffd640 1 -- 172.21.15.78:0/2535731699 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba40343f0 con 0x7fcbac157040 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 -- 172.21.15.78:0/2535731699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac157040 msgr2=0x7fcbac0c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 --2- 172.21.15.78:0/2535731699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac157040 0x7fcbac0c7dd0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009e50 tx=0x7fcba402bee0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 -- 172.21.15.78:0/2535731699 shutdown_connections 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 --2- 172.21.15.78:0/2535731699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac157040 0x7fcbac0c7dd0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 -- 172.21.15.78:0/2535731699 >> 172.21.15.78:0/2535731699 conn(0x7fcbac0aa3e0 msgr2=0x7fcbac0aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 -- 172.21.15.78:0/2535731699 shutdown_connections 2024-02-19T15:00:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.371+0000 7fcbb2925640 1 -- 172.21.15.78:0/2535731699 wait complete. 2024-02-19T15:00:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.372+0000 7fcbb2925640 1 Processor -- start 2024-02-19T15:00:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.372+0000 7fcbb2925640 1 -- start start 2024-02-19T15:00:43.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb2925640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac06f170 0x7fcbac1515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb2925640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbac156550 0x7fcbac0ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb2925640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 0x7fcbac14bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb2925640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1e84e0 con 0x7fcbac14b940 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb2925640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac151da0 con 0x7fcbac06f170 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb2925640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac14c310 con 0x7fcbac156550 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb0ec2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 0x7fcbac14bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb0ec2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 0x7fcbac14bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37084/0 (socket says 172.21.15.78:37084) 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/1861870039 learned_addr learned my addr 172.21.15.78:0/1861870039 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.373+0000 7fcbabfff640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac06f170 0x7fcbac1515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbac156550 msgr2=0x7fcbac0ad4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb0ec2640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbac156550 0x7fcbac0ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac06f170 msgr2=0x7fcbac1515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb0ec2640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac06f170 0x7fcbac1515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb0ec2640 1 -- 172.21.15.78:0/1861870039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba4009b30 con 0x7fcbac14b940 2024-02-19T15:00:43.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb0ec2640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 0x7fcbac14bdd0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fcba4036040 tx=0x7fcba4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba403e450 con 0x7fcbac14b940 2024-02-19T15:00:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba403e5d0 con 0x7fcbac14b940 2024-02-19T15:00:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba403e980 con 0x7fcbac14b940 2024-02-19T15:00:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac151b00 con 0x7fcbac14b940 2024-02-19T15:00:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.374+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac1e89d0 con 0x7fcbac14b940 2024-02-19T15:00:43.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.375+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac074990 con 0x7fcbac14b940 2024-02-19T15:00:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.375+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcba402cd50 con 0x7fcbac14b940 2024-02-19T15:00:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.376+0000 7fcba8ff9640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb9806b810 0x7fcb9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.376+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcba40acf00 con 0x7fcbac14b940 2024-02-19T15:00:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.376+0000 7fcbabfff640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb9806b810 0x7fcb9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.376+0000 7fcbabfff640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb9806b810 0x7fcb9806dcd0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0015d0 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:43.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.380+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcba4077b70 con 0x7fcbac14b940 2024-02-19T15:00:43.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.551+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcbac0ab640 con 0x7fcbac14b940 2024-02-19T15:00:43.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.557+0000 7fcba8ff9640 1 -- 172.21.15.78:0/1861870039 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v63)=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcba4047e90 con 0x7fcbac14b940 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb9806b810 msgr2=0x7fcb9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb9806b810 0x7fcb9806dcd0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0015d0 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 msgr2=0x7fcbac14bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 0x7fcbac14bdd0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fcba4036040 tx=0x7fcba4004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 shutdown_connections 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcb9806b810 0x7fcb9806dcd0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbac14b940 0x7fcbac14bdd0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbac156550 0x7fcbac0ad4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 --2- 172.21.15.78:0/1861870039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac06f170 0x7fcbac1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 >> 172.21.15.78:0/1861870039 conn(0x7fcbac0aa3e0 msgr2=0x7fcbac0a9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 shutdown_connections 2024-02-19T15:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.559+0000 7fcbb2925640 1 -- 172.21.15.78:0/1861870039 wait complete. 2024-02-19T15:00:43.572 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:00:43.572 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:00:43.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:00:43.670 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:00:43.670 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T15:00:43.670 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:00:43.685 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T15:00:43.685 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T15:00:43.685 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:00:43.701 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T15:00:43.701 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-19T15:00:43.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:43.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.821+0000 7f133749a640 1 Processor -- start 2024-02-19T15:00:43.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.821+0000 7f133749a640 1 -- start start 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f133749a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f133006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f133749a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f133006d910 0x7f133006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f133749a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133006cd20 con 0x7f1330152700 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f133749a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13301551f0 con 0x7f133006d910 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f133749a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1330155490 con 0x7f13300c8e00 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f1335a37640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f133006d910 0x7f133006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f1335236640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1330152700 0x7f13300c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42052/0 (socket says 172.21.15.78:42052) 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f1335a37640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f133006d910 0x7f133006c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37100/0 (socket says 172.21.15.78:37100) 2024-02-19T15:00:43.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.822+0000 7f1335236640 1 -- 172.21.15.78:0/320524690 learned_addr learned my addr 172.21.15.78:0/320524690 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1334a35640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f133006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1335a37640 1 -- 172.21.15.78:0/320524690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 msgr2=0x7f133006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1335a37640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f133006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1335a37640 1 -- 172.21.15.78:0/320524690 >> v1:172.21.15.78:6789/0 conn(0x7f1330152700 legacy=0x7f13300c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1335a37640 1 -- 172.21.15.78:0/320524690 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1330155770 con 0x7f133006d910 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1334a35640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f133006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1335a37640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f133006d910 0x7f133006c7e0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f13280097a0 tx=0x7f132802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6dc8b0f96ebd954 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.823+0000 7f1327fff640 1 -- 172.21.15.78:0/320524690 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132802e070 con 0x7f133006d910 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f1327fff640 1 -- 172.21.15.78:0/320524690 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13280315a0 con 0x7f133006d910 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f1327fff640 1 -- 172.21.15.78:0/320524690 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328031930 con 0x7f133006d910 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 -- 172.21.15.78:0/320524690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f133006d910 msgr2=0x7f133006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f133006d910 0x7f133006c7e0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f13280097a0 tx=0x7f132802b840 comp rx=0 tx=0).stop 2024-02-19T15:00:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 -- 172.21.15.78:0/320524690 shutdown_connections 2024-02-19T15:00:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f133006d910 0x7f133006c7e0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 --2- 172.21.15.78:0/320524690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f133006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 -- 172.21.15.78:0/320524690 >> 172.21.15.78:0/320524690 conn(0x7f13300cac90 msgr2=0x7f13300cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 -- 172.21.15.78:0/320524690 shutdown_connections 2024-02-19T15:00:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.824+0000 7f133749a640 1 -- 172.21.15.78:0/320524690 wait complete. 2024-02-19T15:00:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.825+0000 7f133749a640 1 Processor -- start 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.825+0000 7f133749a640 1 -- start start 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f133749a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f133006d910 0x7f13301edc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f133749a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f13301eb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f133749a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 0x7f13300c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f133749a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1330151300 con 0x7f1330152700 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f133749a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13300aad60 con 0x7f133006d910 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f133749a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13301ee550 con 0x7f13300c8e00 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 0x7f13300c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 0x7f13300c0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37106/0 (socket says 172.21.15.78:37106) 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 -- 172.21.15.78:0/2321268140 learned_addr learned my addr 172.21.15.78:0/2321268140 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1334a35640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f13301eb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335236640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f133006d910 0x7f13301edc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 -- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 msgr2=0x7f13301eb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f13301eb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 -- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f133006d910 msgr2=0x7f13301edc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f133006d910 0x7f13301edc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335a37640 1 -- 172.21.15.78:0/2321268140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328009400 con 0x7f1330152700 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1334a35640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f13301eb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.826+0000 7f1335236640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f133006d910 0x7f13301edc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.827+0000 7f1335a37640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 0x7f13300c0ad0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f1328034040 tx=0x7f13280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.827+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328032840 con 0x7f1330152700 2024-02-19T15:00:43.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.827+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13280329c0 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.827+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13300ab000 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.827+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328032d70 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.827+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13300abdc0 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.828+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13300ab520 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.828+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f132804e430 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.829+0000 7f1325ffb640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f131406b840 0x7f131406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.829+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f13280ac080 con 0x7f1330152700 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.829+0000 7f1335236640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f131406b840 0x7f131406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:43.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.829+0000 7f1335236640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f131406b840 0x7f131406dd00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f13300c4af0 tx=0x7f132c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:43.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:43.833+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f13300ab520 con 0x7f1330152700 2024-02-19T15:00:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.041+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13300ab520 con 0x7f1330152700 2024-02-19T15:00:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.042+0000 7f1325ffb640 1 -- 172.21.15.78:0/2321268140 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+3703 (secure 0 0 0) 0x7f13300ab520 con 0x7f1330152700 2024-02-19T15:00:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:44.045 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":186,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8992,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/485928225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":485928225},{"type":"v1","addr":"172.21.15.150:6834","nonce":485928225}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":8995,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/459710574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":459710574},{"type":"v1","addr":"172.21.15.150:6835","nonce":459710574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/302197543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":302197543},{"type":"v1","addr":"172.21.15.78:6836","nonce":302197543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":177},{"gid":10524,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/2906585066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":2906585066},{"type":"v1","addr":"172.21.15.78:6837","nonce":2906585066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[]} 2024-02-19T15:00:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 186 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f131406b840 msgr2=0x7f131406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f131406b840 0x7f131406dd00 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f13300c4af0 tx=0x7f132c009040 comp rx=0 tx=0).stop 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 msgr2=0x7f13300c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 0x7f13300c0ad0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f1328034040 tx=0x7f13280046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 shutdown_connections 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f131406b840 0x7f131406dd00 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1330152700 0x7f13300c0ad0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13300c8e00 0x7f13301eb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 --2- 172.21.15.78:0/2321268140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f133006d910 0x7f13301edc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 >> 172.21.15.78:0/2321268140 conn(0x7f13300cac90 msgr2=0x7f1330151480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 shutdown_connections 2024-02-19T15:00:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.044+0000 7f133749a640 1 -- 172.21.15.78:0/2321268140 wait complete. 2024-02-19T15:00:44.058 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-19T15:00:44.058 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-19T15:00:44.058 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:00:44.058 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:00:44.058 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-19T15:00:44.058 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-19T15:00:44.058 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:00:44.059 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:00:44.059 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-19T15:00:44.059 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-19T15:00:44.059 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:00:44.059 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:00:44.059 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-19T15:00:44.059 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-19T15:00:44.060 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:00:44.060 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:00:44.067 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:00:44.067 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-19T15:00:44.068 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:00:44.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-19T15:00:44.073 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:00:44.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-19T15:00:44.074 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:00:44.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-19T15:00:44.110 INFO:tasks.ceph.mds.a:Started 2024-02-19T15:00:44.110 DEBUG:teuthology.parallel:result is None 2024-02-19T15:00:44.111 INFO:tasks.ceph.mds.c:Started 2024-02-19T15:00:44.111 DEBUG:teuthology.parallel:result is None 2024-02-19T15:00:44.117 INFO:tasks.ceph.mds.b:Started 2024-02-19T15:00:44.117 DEBUG:teuthology.parallel:result is None 2024-02-19T15:00:44.117 INFO:tasks.ceph.mds.d:Started 2024-02-19T15:00:44.117 DEBUG:teuthology.parallel:result is None 2024-02-19T15:00:44.118 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-19T15:00:44.185 INFO:tasks.ceph.mds.a.smithi078.stdout:starting mds.a at 2024-02-19T15:00:44.186 INFO:tasks.ceph.mds.c.smithi078.stdout:starting mds.c at 2024-02-19T15:00:44.204 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T15:00:44.204 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2024-02-19T15:00:44.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.221+0000 7f7ddb24e640 1 Processor -- start 2024-02-19T15:00:44.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.221+0000 7f7ddb24e640 1 -- start start 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7ddb24e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7dd40c3260 0x7f7dd406e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7ddb24e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd4155590 con 0x7f7dd40a84c0 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7dd40a84c0 0x7f7dd4072410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42082/0 (socket says 172.21.15.78:42082) 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd3fff640 1 -- 172.21.15.78:0/164434787 learned_addr learned my addr 172.21.15.78:0/164434787 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd97eb640 1 --2- 172.21.15.78:0/164434787 >> v2:172.21.15.78:3300/0 conn(0x7f7dd40c3260 0x7f7dd406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd4155870 con 0x7f7dd414b620 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd4152410 con 0x7f7dd40c3260 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd37fe640 1 -- 172.21.15.78:0/164434787 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124252939 0 0) 0x7f7dd4155590 con 0x7f7dd40a84c0 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd37fe640 1 -- 172.21.15.78:0/164434787 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc4004630 con 0x7f7dd40a84c0 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd37fe640 1 -- 172.21.15.78:0/164434787 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850544358 0 0) 0x7f7dc4004630 con 0x7f7dd40a84c0 2024-02-19T15:00:44.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd97eb640 1 -- 172.21.15.78:0/164434787 >> v1:172.21.15.150:6789/0 conn(0x7f7dd414b620 legacy=0x7f7dd4151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd97eb640 1 -- 172.21.15.78:0/164434787 >> v1:172.21.15.78:6789/0 conn(0x7f7dd40a84c0 legacy=0x7f7dd4072410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.222+0000 7f7dd97eb640 1 -- 172.21.15.78:0/164434787 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd4151500 con 0x7f7dd40c3260 2024-02-19T15:00:44.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.223+0000 7f7dd97eb640 1 --2- 172.21.15.78:0/164434787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd40c3260 0x7f7dd406e310 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009d90 tx=0x7f7dcc02bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6791a604780ffd48 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:44.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.223+0000 7f7dd37fe640 1 -- 172.21.15.78:0/164434787 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0001670 con 0x7f7dd40c3260 2024-02-19T15:00:44.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.223+0000 7f7dd37fe640 1 -- 172.21.15.78:0/164434787 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc02d690 con 0x7f7dd40c3260 2024-02-19T15:00:44.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.223+0000 7f7dd37fe640 1 -- 172.21.15.78:0/164434787 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02da20 con 0x7f7dd40c3260 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd40c3260 msgr2=0x7f7dd406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/164434787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd40c3260 0x7f7dd406e310 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009d90 tx=0x7f7dcc02bb50 comp rx=0 tx=0).stop 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 shutdown_connections 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/164434787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd40c3260 0x7f7dd406e310 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 >> 172.21.15.78:0/164434787 conn(0x7f7dd4071920 msgr2=0x7f7dd4071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 shutdown_connections 2024-02-19T15:00:44.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.224+0000 7f7ddb24e640 1 -- 172.21.15.78:0/164434787 wait complete. 2024-02-19T15:00:44.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.225+0000 7f7ddb24e640 1 Processor -- start 2024-02-19T15:00:44.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.225+0000 7f7ddb24e640 1 -- start start 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7ddb24e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a84c0 0x7f7dd4153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7ddb24e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7dd40c3260 0x7f7dd40a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7ddb24e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 0x7f7dd40ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7ddb24e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4146f00 con 0x7f7dd414b620 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7ddb24e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4146570 con 0x7f7dd40a84c0 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7ddb24e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd41e9bb0 con 0x7f7dd40c3260 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 0x7f7dd40ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd3fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7dd40c3260 0x7f7dd40a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd8fea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a84c0 0x7f7dd4153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 0x7f7dd40ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37146/0 (socket says 172.21.15.78:37146) 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 -- 172.21.15.78:0/3125880084 learned_addr learned my addr 172.21.15.78:0/3125880084 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 -- 172.21.15.78:0/3125880084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7dd40c3260 msgr2=0x7f7dd40a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7dd40c3260 0x7f7dd40a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 -- 172.21.15.78:0/3125880084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a84c0 msgr2=0x7f7dd4153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.226+0000 7f7dd97eb640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a84c0 0x7f7dd4153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7dd97eb640 1 -- 172.21.15.78:0/3125880084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc009a70 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7dd97eb640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 0x7f7dd40ac090 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009d90 tx=0x7f7dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc02ea20 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc02eb80 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc003990 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd41e9e90 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.227+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd4153fe0 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.228+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7dcc049460 con 0x7f7dd414b620 2024-02-19T15:00:44.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.228+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd41e9a10 con 0x7f7dd414b620 2024-02-19T15:00:44.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.228+0000 7f7dd17fa640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7db806fc20 0x7f7db80720e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:44.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.229+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7dcc0ac9c0 con 0x7f7dd414b620 2024-02-19T15:00:44.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.229+0000 7f7dd8fea640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7db806fc20 0x7f7db80720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:44.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.229+0000 7f7dd8fea640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7db806fc20 0x7f7db80720e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8000f60 tx=0x7f7dc8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:44.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.233+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7dd41e9a10 con 0x7f7dd414b620 2024-02-19T15:00:44.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:44.420+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7dd41ea3b0 con 0x7f7dd414b620 2024-02-19T15:00:45.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.279+0000 7f7dd17fa640 1 -- 172.21.15.78:0/3125880084 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v209) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7dd41ea3b0 con 0x7f7dd414b620 2024-02-19T15:00:45.282 INFO:teuthology.orchestra.run.smithi078.stderr: removed all blocklist entries 2024-02-19T15:00:45.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.281+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7db806fc20 msgr2=0x7f7db80720e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.281+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7db806fc20 0x7f7db80720e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8000f60 tx=0x7f7dc8009040 comp rx=0 tx=0).stop 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.281+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 msgr2=0x7f7dd40ac090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.281+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 0x7f7dd40ac090 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009d90 tx=0x7f7dcc004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 shutdown_connections 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7db806fc20 0x7f7db80720e0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7dd414b620 0x7f7dd40ac090 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7dd40c3260 0x7f7dd40a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 --2- 172.21.15.78:0/3125880084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a84c0 0x7f7dd4153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 >> 172.21.15.78:0/3125880084 conn(0x7f7dd4071920 msgr2=0x7f7dd406e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 shutdown_connections 2024-02-19T15:00:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.282+0000 7f7ddb24e640 1 -- 172.21.15.78:0/3125880084 wait complete. 2024-02-19T15:00:45.297 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-19T15:00:45.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.399+0000 7f3f13fbc640 1 Processor -- start 2024-02-19T15:00:45.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.399+0000 7f3f13fbc640 1 -- start start 2024-02-19T15:00:45.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f13fbc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3f0c0acc50 0x7f3f0c0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f13fbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f13fbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0c159130 con 0x7f3f0c0718f0 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f13fbc640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0c0c2cf0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f13fbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0c2030 con 0x7f3f0c155070 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3f0c0acc50 0x7f3f0c0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3f0c0acc50 0x7f3f0c0c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37156/0 (socket says 172.21.15.78:37156) 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 -- 172.21.15.78:0/2713556993 learned_addr learned my addr 172.21.15.78:0/2713556993 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11557640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 -- 172.21.15.78:0/2713556993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 msgr2=0x7f3f0c1554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 -- 172.21.15.78:0/2713556993 >> v1:172.21.15.150:6789/0 conn(0x7f3f0c0718f0 legacy=0x7f3f0c071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11d58640 1 -- 172.21.15.78:0/2713556993 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c0ad030 con 0x7f3f0c0acc50 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.400+0000 7f3f11557640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:45.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.401+0000 7f3f11d58640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c17e0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f3f080098a0 tx=0x7f3f0802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3403a17f64b1a602 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:45.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.401+0000 7f3f10d56640 1 -- 172.21.15.78:0/2713556993 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08031640 con 0x7f3f0c0acc50 2024-02-19T15:00:45.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.401+0000 7f3f10d56640 1 -- 172.21.15.78:0/2713556993 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f080317a0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.401+0000 7f3f10d56640 1 -- 172.21.15.78:0/2713556993 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08031b30 con 0x7f3f0c0acc50 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.401+0000 7f3f13fbc640 1 -- 172.21.15.78:0/2713556993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 msgr2=0x7f3f0c0c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.401+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c17e0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f3f080098a0 tx=0x7f3f0802b8f0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.402+0000 7f3f13fbc640 1 -- 172.21.15.78:0/2713556993 shutdown_connections 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.402+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.402+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/2713556993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c17e0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.402+0000 7f3f13fbc640 1 -- 172.21.15.78:0/2713556993 >> 172.21.15.78:0/2713556993 conn(0x7f3f0c0c7800 msgr2=0x7f3f0c0c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:45.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.402+0000 7f3f13fbc640 1 -- 172.21.15.78:0/2713556993 shutdown_connections 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.402+0000 7f3f13fbc640 1 -- 172.21.15.78:0/2713556993 wait complete. 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 Processor -- start 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 -- start start 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f0c0718f0 0x7f3f0c134900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c0c5860 con 0x7f3f0c0acc50 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c137300 con 0x7f3f0c0718f0 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f13fbc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c136d40 con 0x7f3f0c155070 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f11557640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f11557640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37158/0 (socket says 172.21.15.78:37158) 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.403+0000 7f3f11557640 1 -- 172.21.15.78:0/3556814230 learned_addr learned my addr 172.21.15.78:0/3556814230 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:45.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f12559640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11d58640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f0c0718f0 0x7f3f0c134900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11557640 1 -- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 msgr2=0x7f3f0c1e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11557640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11557640 1 -- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f0c0718f0 msgr2=0x7f3f0c134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11557640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f0c0718f0 0x7f3f0c134900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11557640 1 -- 172.21.15.78:0/3556814230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f080094d0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f11557640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c5c80 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00d7d0 tx=0x7f3efc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc004030 con 0x7f3f0c0acc50 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc004190 con 0x7f3f0c0acc50 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc004520 con 0x7f3f0c0acc50 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c137020 con 0x7f3f0c0acc50 2024-02-19T15:00:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.404+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c0c61c0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.405+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c1595a0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.405+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3efc0096a0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.406+0000 7f3f027fc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ef006b810 0x7f3ef006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.406+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3efc08a1b0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.406+0000 7f3f11d58640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ef006b810 0x7f3ef006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.406+0000 7f3f11d58640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ef006b810 0x7f3ef006dcd0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3f08036000 tx=0x7f3f0803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:45.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.410+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f0c1595a0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.646+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3f0c0c59a0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.647+0000 7f3f027fc640 1 -- 172.21.15.78:0/3556814230 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f3f0c0c59a0 con 0x7f3f0c0acc50 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_dump": [ 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.a", 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQVCkHBAAWbCtZt15KlJc5HKD9c9T3g==", 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.b", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlaKTzKBAATTLlU6zNB/wdC/itIXNnlg==", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.c", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlfnfzIxAA6KwOMh/Zvi9Q05ucUqCC7g==", 2024-02-19T15:00:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mds.d", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlE65TMBAA4o3+WebuMY5iowXi0c6OPg==", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T15:00:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.0", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDVatNlk5X/CBAAUjiE+9CrY8JTKEfJIz1gFA==", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.1", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDYatNl0FJ1BxAABj5O5s5lC52bdewI/s38Sg==", 2024-02-19T15:00:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.2", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDbatNliEyjBRAATpIOaLRr7u8fMw5om2bkDA==", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.3", 2024-02-19T15:00:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDeatNl+haDAxAABMKDi4hrdtjbW4hk0PPHJg==", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.4", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDkatNl+z7cChAAbGJKgFhSuAWrGBrSBA7+gg==", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.5", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDnatNllJdUCRAAxjmgbg6YY4c2ZjDCARhJpA==", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.6", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDqatNlNmPfBxAAkXM7e9VgBPObbFhw/9/Wog==", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "osd.7", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDtatNl5ibLBhAAJ/OjyZMurMI+/vZfoj1tqQ==", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow profile osd", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile osd", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.0", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg==", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow r", 2024-02-19T15:00:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.1", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNl7tL4NxAACXawJnbV+0eB24JP7qNCZw==", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow r", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow rw", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow rwx" 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "client.admin", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlpcoCABAA46nc8/NRbOAUla6tmsDGvQ==", 2024-02-19T15:00:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": "allow *", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow *", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.x", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlO3w8GRAAxX1cXgGqLBMedQFvEU+M3A==", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "entity": "mgr.y", 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "AQDRatNlQddXFRAAur6PMdm0FEijGttlQPPGgQ==", 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": { 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": "allow *", 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": "allow profile mgr", 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": "allow *" 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-02-19T15:00:45.660 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:00:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ef006b810 msgr2=0x7f3ef006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ef006b810 0x7f3ef006dcd0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3f08036000 tx=0x7f3f0803c040 comp rx=0 tx=0).stop 2024-02-19T15:00:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 msgr2=0x7f3f0c0c5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c5c80 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00d7d0 tx=0x7f3efc00dc90 comp rx=0 tx=0).stop 2024-02-19T15:00:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 shutdown_connections 2024-02-19T15:00:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3ef006b810 0x7f3ef006dcd0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f0c0acc50 0x7f3f0c0c5c80 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f0c155070 0x7f3f0c1e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 --2- 172.21.15.78:0/3556814230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f0c0718f0 0x7f3f0c134900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 >> 172.21.15.78:0/3556814230 conn(0x7f3f0c0c7800 msgr2=0x7f3f0c1545b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 shutdown_connections 2024-02-19T15:00:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.649+0000 7f3f13fbc640 1 -- 172.21.15.78:0/3556814230 wait complete. 2024-02-19T15:00:45.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:45.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.760+0000 7f371dbbd640 1 Processor -- start 2024-02-19T15:00:45.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.760+0000 7f371dbbd640 1 -- start start 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.760+0000 7f371dbbd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f37180c59f0 0x7f37181608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.760+0000 7f371dbbd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180c9200 con 0x7f37180c5590 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.760+0000 7f371dbbd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180cb720 con 0x7f37180c51b0 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.760+0000 7f371dbbd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3718158ff0 con 0x7f37180c59f0 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f3716ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f37180c5590 0x7f37180c9c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42088/0 (socket says 172.21.15.78:42088) 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f3716ffd640 1 -- 172.21.15.78:0/4187400632 learned_addr learned my addr 172.21.15.78:0/4187400632 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:45.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f3717fff640 1 --2- 172.21.15.78:0/4187400632 >> v2:172.21.15.150:3300/0 conn(0x7f37180c59f0 0x7f37181608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206159805 0 0) 0x7f37180c9200 con 0x7f37180c5590 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3700003520 con 0x7f37180c5590 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832199336 0 0) 0x7f37180cb720 con 0x7f37180c51b0 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37180c9200 con 0x7f37180c51b0 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3624362393 0 0) 0x7f3700003520 con 0x7f37180c5590 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 >> v2:172.21.15.150:3300/0 conn(0x7f37180c59f0 msgr2=0x7f37181608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 --2- 172.21.15.78:0/4187400632 >> v2:172.21.15.150:3300/0 conn(0x7f37180c59f0 0x7f37181608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 >> v1:172.21.15.150:6789/0 conn(0x7f37180c51b0 legacy=0x7f37180c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37181592d0 con 0x7f37180c5590 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f370c001670 con 0x7f37180c5590 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f3717fff640 1 --2- 172.21.15.78:0/4187400632 >> v2:172.21.15.150:3300/0 conn(0x7f37180c59f0 0x7f37181608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 >> v1:172.21.15.78:6789/0 conn(0x7f37180c5590 legacy=0x7f37180c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3700003b00 0x7f3700003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37000047c0 0x7f3700004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700005180 0x7f370000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3718158ff0 con 0x7f3700003b00 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370000c000 con 0x7f37000047c0 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370000c1c0 con 0x7f3700005180 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f37177fe640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3700003b00 0x7f3700003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.761+0000 7f3716ffd640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37000047c0 0x7f3700004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f3717fff640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700005180 0x7f370000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37177fe640 1 -- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700005180 msgr2=0x7f370000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37177fe640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700005180 0x7f370000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37177fe640 1 -- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37000047c0 msgr2=0x7f3700004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37177fe640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37000047c0 0x7f3700004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f3717fff640 1 -- 172.21.15.78:0/4187400632 reap_dead start 2024-02-19T15:00:45.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37177fe640 1 -- 172.21.15.78:0/4187400632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3700003520 con 0x7f3700003b00 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f3716ffd640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37000047c0 0x7f3700004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37177fe640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3700003b00 0x7f3700003f00 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f37080018d0 tx=0x7f37000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70acc280f979f81c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37080052d0 con 0x7f3700003b00 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3708005430 con 0x7f3700003b00 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.762+0000 7f37167fc640 1 -- 172.21.15.78:0/4187400632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708003810 con 0x7f3700003b00 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 -- 172.21.15.78:0/4187400632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3700003b00 msgr2=0x7f3700003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3700003b00 0x7f3700003f00 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f37080018d0 tx=0x7f37000057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 -- 172.21.15.78:0/4187400632 shutdown_connections 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 --2- 172.21.15.78:0/4187400632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3700003b00 0x7f3700003f00 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 -- 172.21.15.78:0/4187400632 >> 172.21.15.78:0/4187400632 conn(0x7f37180ca830 msgr2=0x7f37180cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 -- 172.21.15.78:0/4187400632 shutdown_connections 2024-02-19T15:00:45.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.763+0000 7f371dbbd640 1 -- 172.21.15.78:0/4187400632 wait complete. 2024-02-19T15:00:45.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.764+0000 7f371dbbd640 1 Processor -- start 2024-02-19T15:00:45.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.764+0000 7f371dbbd640 1 -- start start 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f371dbbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700003b00 0x7f37181df710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f371dbbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 0x7f371814f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f371dbbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37181deb00 0x7f371814f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f371dbbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371814fdd0 con 0x7f37181de5b0 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f371dbbd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180c1900 con 0x7f37181deb00 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f371dbbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180c1b50 con 0x7f3700003b00 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 0x7f371814f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 0x7f371814f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37178/0 (socket says 172.21.15.78:37178) 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 -- 172.21.15.78:0/1459482459 learned_addr learned my addr 172.21.15.78:0/1459482459 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f37177fe640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700003b00 0x7f37181df710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3717fff640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37181deb00 0x7f371814f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 -- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700003b00 msgr2=0x7f37181df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700003b00 0x7f37181df710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 -- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37181deb00 msgr2=0x7f371814f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37181deb00 0x7f371814f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3716ffd640 1 -- 172.21.15.78:0/1459482459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180c1e30 con 0x7f37181de5b0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.765+0000 7f3717fff640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37181deb00 0x7f371814f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.766+0000 7f3716ffd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 0x7f371814f350 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e7b0 tx=0x7f370c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.766+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c00eec0 con 0x7f37181de5b0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.766+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370c00c870 con 0x7f37181de5b0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.766+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c00cbe0 con 0x7f37181de5b0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.766+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37180c2110 con 0x7f37181de5b0 2024-02-19T15:00:45.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.766+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37181dfc50 con 0x7f37181de5b0 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.767+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37180c2630 con 0x7f37181de5b0 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.767+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f370c00cd40 con 0x7f37181de5b0 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.767+0000 7f36fffff640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f370406b810 0x7f370406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.768+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f370c089f00 con 0x7f37181de5b0 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.768+0000 7f37177fe640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f370406b810 0x7f370406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.768+0000 7f37177fe640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f370406b810 0x7f370406dcd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3708034000 tx=0x7f3708031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:45.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.772+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f37180c2630 con 0x7f37181de5b0 2024-02-19T15:00:45.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.945+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37180c2630 con 0x7f37181de5b0 2024-02-19T15:00:45.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.945+0000 7f36fffff640 1 -- 172.21.15.78:0/1459482459 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+13162 (secure 0 0 0) 0x7f37180c2630 con 0x7f37181de5b0 2024-02-19T15:00:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:45.948 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":209,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:00:45.278377+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":200,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:45.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f370406b810 msgr2=0x7f370406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f370406b810 0x7f370406dcd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3708034000 tx=0x7f3708031000 comp rx=0 tx=0).stop 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 msgr2=0x7f371814f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 0x7f371814f350 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e7b0 tx=0x7f370c00ec70 comp rx=0 tx=0).stop 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 shutdown_connections 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f370406b810 0x7f370406dcd0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37181de5b0 0x7f371814f350 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37181deb00 0x7f371814f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 --2- 172.21.15.78:0/1459482459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3700003b00 0x7f37181df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.947+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 >> 172.21.15.78:0/1459482459 conn(0x7f37180ca830 msgr2=0x7f3718151270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.948+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 shutdown_connections 2024-02-19T15:00:45.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:45.948+0000 7f371dbbd640 1 -- 172.21.15.78:0/1459482459 wait complete. 2024-02-19T15:00:45.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:00:45.962 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:00:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.062+0000 7fd90ecc7640 1 Processor -- start 2024-02-19T15:00:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.062+0000 7fd90ecc7640 1 -- start start 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90ecc7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd90806caa0 0x7fd9081609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90ecc7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9080c7df0 con 0x7fd908155e00 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90ecc7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9080c77c0 con 0x7fd9080c67e0 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90ecc7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9080c4310 con 0x7fd90806caa0 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90d264640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd90806caa0 0x7fd9081609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90ca63640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd908155e00 0x7fd9080c34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42092/0 (socket says 172.21.15.78:42092) 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90ca63640 1 -- 172.21.15.78:0/2978954473 learned_addr learned my addr 172.21.15.78:0/2978954473 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90d264640 1 -- 172.21.15.78:0/2978954473 >> v1:172.21.15.150:6789/0 conn(0x7fd9080c67e0 legacy=0x7fd90806c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90d264640 1 -- 172.21.15.78:0/2978954473 >> v1:172.21.15.78:6789/0 conn(0x7fd908155e00 legacy=0x7fd9080c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd90d264640 1 -- 172.21.15.78:0/2978954473 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080c70d0 con 0x7fd90806caa0 2024-02-19T15:00:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.063+0000 7fd9077fe640 1 -- 172.21.15.78:0/2978954473 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086892748 0 0) 0x7fd9080c7df0 con 0x7fd908155e00 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd90d264640 1 --2- 172.21.15.78:0/2978954473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd90806caa0 0x7fd9081609d0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fd900009d90 tx=0x7fd90002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3673d47b060626 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd9077fe640 1 -- 172.21.15.78:0/2978954473 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90002cbd0 con 0x7fd90806caa0 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd9077fe640 1 -- 172.21.15.78:0/2978954473 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90002cd30 con 0x7fd90806caa0 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd9077fe640 1 -- 172.21.15.78:0/2978954473 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90002cea0 con 0x7fd90806caa0 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd90ecc7640 1 -- 172.21.15.78:0/2978954473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd90806caa0 msgr2=0x7fd9081609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/2978954473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd90806caa0 0x7fd9081609d0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fd900009d90 tx=0x7fd90002be20 comp rx=0 tx=0).stop 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd90ecc7640 1 -- 172.21.15.78:0/2978954473 shutdown_connections 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/2978954473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd90806caa0 0x7fd9081609d0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.064+0000 7fd90ecc7640 1 -- 172.21.15.78:0/2978954473 >> 172.21.15.78:0/2978954473 conn(0x7fd90806b700 msgr2=0x7fd9080aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.065+0000 7fd90ecc7640 1 -- 172.21.15.78:0/2978954473 shutdown_connections 2024-02-19T15:00:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.065+0000 7fd90ecc7640 1 -- 172.21.15.78:0/2978954473 wait complete. 2024-02-19T15:00:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 Processor -- start 2024-02-19T15:00:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 -- start start 2024-02-19T15:00:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd90806caa0 0x7fd9081eb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 0x7fd9080c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd908155e00 0x7fd9081ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9081582d0 con 0x7fd9080c67e0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908158aa0 con 0x7fd908155e00 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd90ecc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd908159370 con 0x7fd90806caa0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.066+0000 7fd907fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 0x7fd9080c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 0x7fd9080c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37204/0 (socket says 172.21.15.78:37204) 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 -- 172.21.15.78:0/3955382446 learned_addr learned my addr 172.21.15.78:0/3955382446 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 -- 172.21.15.78:0/3955382446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd90806caa0 msgr2=0x7fd9081eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd90806caa0 0x7fd9081eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 -- 172.21.15.78:0/3955382446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd908155e00 msgr2=0x7fd9081ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd908155e00 0x7fd9081ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 -- 172.21.15.78:0/3955382446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009a70 con 0x7fd9080c67e0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd907fff640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 0x7fd9080c8c60 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400b420 tx=0x7fd8f400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4011830 con 0x7fd9080c67e0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4011990 con 0x7fd9080c67e0 2024-02-19T15:00:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9081584e0 con 0x7fd9080c67e0 2024-02-19T15:00:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4011d40 con 0x7fd9080c67e0 2024-02-19T15:00:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.067+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9081edf30 con 0x7fd9080c67e0 2024-02-19T15:00:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.068+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9081ee150 con 0x7fd9080c67e0 2024-02-19T15:00:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.069+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd8f401b460 con 0x7fd9080c67e0 2024-02-19T15:00:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.069+0000 7fd9057fa640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8ec06b810 0x7fd8ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.069+0000 7fd90ca63640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8ec06b810 0x7fd8ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.069+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd8f4089b10 con 0x7fd9080c67e0 2024-02-19T15:00:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.070+0000 7fd90ca63640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8ec06b810 0x7fd8ec06dcd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc007fd0 tx=0x7fd8fc007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.073+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd8f4054780 con 0x7fd9080c67e0 2024-02-19T15:00:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.244+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9080c6cb0 con 0x7fd9080c67e0 2024-02-19T15:00:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.244+0000 7fd9057fa640 1 -- 172.21.15.78:0/3955382446 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8f4057e40 con 0x7fd9080c67e0 2024-02-19T15:00:46.247 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T15:00:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8ec06b810 msgr2=0x7fd8ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8ec06b810 0x7fd8ec06dcd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc007fd0 tx=0x7fd8fc007920 comp rx=0 tx=0).stop 2024-02-19T15:00:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 msgr2=0x7fd9080c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 0x7fd9080c8c60 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400b420 tx=0x7fd8f400b8e0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 shutdown_connections 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd8ec06b810 0x7fd8ec06dcd0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9080c67e0 0x7fd9080c8c60 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd908155e00 0x7fd9081ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 --2- 172.21.15.78:0/3955382446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd90806caa0 0x7fd9081eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 >> 172.21.15.78:0/3955382446 conn(0x7fd90806b700 msgr2=0x7fd9080aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 shutdown_connections 2024-02-19T15:00:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.246+0000 7fd90ecc7640 1 -- 172.21.15.78:0/3955382446 wait complete. 2024-02-19T15:00:46.260 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.357+0000 7f7a7265c640 1 Processor -- start 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a7265c640 1 -- start start 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a7265c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a6c0c8e00 0x7f7a6c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a7265c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c1551f0 con 0x7f7a6c152700 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a7265c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c155490 con 0x7f7a6c06d910 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a7265c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6c155770 con 0x7f7a6c0c8e00 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a6bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7a6c152700 0x7f7a6c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42108/0 (socket says 172.21.15.78:42108) 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.358+0000 7f7a6bfff640 1 -- 172.21.15.78:0/3639855117 learned_addr learned my addr 172.21.15.78:0/3639855117 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:46.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6b7fe640 1 --2- 172.21.15.78:0/3639855117 >> v2:172.21.15.150:3300/0 conn(0x7f7a6c0c8e00 0x7f7a6c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097312087 0 0) 0x7f7a6c1551f0 con 0x7f7a6c152700 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a58003520 con 0x7f7a6c152700 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609103874 0 0) 0x7f7a58003520 con 0x7f7a6c152700 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 >> v2:172.21.15.150:3300/0 conn(0x7f7a6c0c8e00 msgr2=0x7f7a6c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 --2- 172.21.15.78:0/3639855117 >> v2:172.21.15.150:3300/0 conn(0x7f7a6c0c8e00 0x7f7a6c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 >> v1:172.21.15.150:6789/0 conn(0x7f7a6c06d910 legacy=0x7f7a6c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c155a50 con 0x7f7a6c152700 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f7a54001670 con 0x7f7a6c152700 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 >> v1:172.21.15.78:6789/0 conn(0x7f7a6c152700 legacy=0x7f7a6c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a58003970 0x7f7a58003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a58004370 0x7f7a58004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58005040 0x7f7a5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c155770 con 0x7f7a58003970 2024-02-19T15:00:46.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5800bfa0 con 0x7f7a58004370 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5800c160 con 0x7f7a58005040 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a58003970 0x7f7a58003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6b7fe640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a58004370 0x7f7a58004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a70bf9640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58005040 0x7f7a5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6bfff640 1 -- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58005040 msgr2=0x7f7a5800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58005040 0x7f7a5800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6bfff640 1 -- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a58004370 msgr2=0x7f7a58004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a58004370 0x7f7a58004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a6bfff640 1 -- 172.21.15.78:0/3639855117 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a58003520 con 0x7f7a58003970 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.359+0000 7f7a70bf9640 1 -- 172.21.15.78:0/3639855117 reap_dead start 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.360+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a58003970 0x7f7a58003d50 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400c160 tx=0x7f7a580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8bcc0075d10cc7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.360+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54005ad0 con 0x7f7a58003970 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.360+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a54005c50 con 0x7f7a58003970 2024-02-19T15:00:46.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.360+0000 7f7a6affd640 1 -- 172.21.15.78:0/3639855117 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5402c850 con 0x7f7a58003970 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 -- 172.21.15.78:0/3639855117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a58003970 msgr2=0x7f7a58003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a58003970 0x7f7a58003d50 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400c160 tx=0x7f7a580057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 -- 172.21.15.78:0/3639855117 shutdown_connections 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 --2- 172.21.15.78:0/3639855117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a58003970 0x7f7a58003d50 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 -- 172.21.15.78:0/3639855117 >> 172.21.15.78:0/3639855117 conn(0x7f7a6c0cac90 msgr2=0x7f7a6c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 -- 172.21.15.78:0/3639855117 shutdown_connections 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.361+0000 7f7a7265c640 1 -- 172.21.15.78:0/3639855117 wait complete. 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.362+0000 7f7a7265c640 1 Processor -- start 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.362+0000 7f7a7265c640 1 -- start start 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a7265c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58003970 0x7f7a6c0aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a7265c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a6c1ea650 0x7f7a6c1ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a7265c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 0x7f7a6c0bf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a7265c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0bf170 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a7265c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0be4f0 con 0x7f7a6c1ea650 2024-02-19T15:00:46.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a7265c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0be7d0 con 0x7f7a58003970 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 0x7f7a6c0bf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 0x7f7a6c0bf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37216/0 (socket says 172.21.15.78:37216) 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 -- 172.21.15.78:0/2436938921 learned_addr learned my addr 172.21.15.78:0/2436938921 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a6b7fe640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a6c1ea650 0x7f7a6c1ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58003970 0x7f7a6c0aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 -- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58003970 msgr2=0x7f7a6c0aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58003970 0x7f7a6c0aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 -- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a6c1ea650 msgr2=0x7f7a6c1ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a6c1ea650 0x7f7a6c1ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 -- 172.21.15.78:0/2436938921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c0beab0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.363+0000 7f7a70bf9640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 0x7f7a6c0bf590 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c08c490 tx=0x7f7a6400eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.364+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64003970 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.364+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a64003af0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.364+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c0bed90 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.364+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64003ea0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.364+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c0c1940 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.365+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c0bf2b0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.365+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7a640050d0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.365+0000 7f7a68ff9640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a4c06b810 0x7f7a4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.365+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7a64089cc0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.365+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a4c06b810 0x7f7a4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.366+0000 7f7a6bfff640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a4c06b810 0x7f7a4c06dcd0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7a54034000 tx=0x7f7a54009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.370+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7a6c0bf2b0 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.542+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7a6c0be910 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.543+0000 7f7a68ff9640 1 -- 172.21.15.78:0/2436938921 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v209) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7a6c0be910 con 0x7f7a6c0c0ca0 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-02-19T15:00:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a4c06b810 msgr2=0x7f7a4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a4c06b810 0x7f7a4c06dcd0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f7a54034000 tx=0x7f7a54009030 comp rx=0 tx=0).stop 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 msgr2=0x7f7a6c0bf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 0x7f7a6c0bf590 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c08c490 tx=0x7f7a6400eb40 comp rx=0 tx=0).stop 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 shutdown_connections 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7a4c06b810 0x7f7a4c06dcd0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a6c0c0ca0 0x7f7a6c0bf590 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a6c1ea650 0x7f7a6c1ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 --2- 172.21.15.78:0/2436938921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a58003970 0x7f7a6c0aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 >> 172.21.15.78:0/2436938921 conn(0x7f7a6c0cac90 msgr2=0x7f7a6c06c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 shutdown_connections 2024-02-19T15:00:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.545+0000 7f7a7265c640 1 -- 172.21.15.78:0/2436938921 wait complete. 2024-02-19T15:00:46.558 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-19T15:00:46.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-19T15:00:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.659+0000 7fdef029a640 1 Processor -- start 2024-02-19T15:00:46.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.659+0000 7fdef029a640 1 -- start start 2024-02-19T15:00:46.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdef029a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdee80719d0 0x7fdee815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdef029a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee80c65a0 con 0x7fdee81503a0 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdef029a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee80a8100 con 0x7fdee80c2ba0 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdef029a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee8150780 con 0x7fdee80719d0 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdeee036640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdee81503a0 0x7fdee80c5130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42110/0 (socket says 172.21.15.78:42110) 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdeee837640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdee80719d0 0x7fdee815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.660+0000 7fdeee036640 1 -- 172.21.15.78:0/1662431269 learned_addr learned my addr 172.21.15.78:0/1662431269 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeed034640 1 -- 172.21.15.78:0/1662431269 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595667512 0 0) 0x7fdee80c65a0 con 0x7fdee81503a0 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeed034640 1 -- 172.21.15.78:0/1662431269 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded4003520 con 0x7fdee81503a0 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeee837640 1 -- 172.21.15.78:0/1662431269 >> v1:172.21.15.150:6789/0 conn(0x7fdee80c2ba0 legacy=0x7fdee8156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeee837640 1 -- 172.21.15.78:0/1662431269 >> v1:172.21.15.78:6789/0 conn(0x7fdee81503a0 legacy=0x7fdee80c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeee837640 1 -- 172.21.15.78:0/1662431269 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80c67f0 con 0x7fdee80719d0 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeee837640 1 --2- 172.21.15.78:0/1662431269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee80719d0 0x7fdee815e6f0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fdee00097a0 tx=0x7fdee002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd720ff04123ce99 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.661+0000 7fdeed034640 1 -- 172.21.15.78:0/1662431269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee002f050 con 0x7fdee80719d0 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdeed034640 1 -- 172.21.15.78:0/1662431269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee0031700 con 0x7fdee80719d0 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdeed034640 1 -- 172.21.15.78:0/1662431269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0031a70 con 0x7fdee80719d0 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 -- 172.21.15.78:0/1662431269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee80719d0 msgr2=0x7fdee815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 --2- 172.21.15.78:0/1662431269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee80719d0 0x7fdee815e6f0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fdee00097a0 tx=0x7fdee002b7a0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 -- 172.21.15.78:0/1662431269 shutdown_connections 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 --2- 172.21.15.78:0/1662431269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee80719d0 0x7fdee815e6f0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 -- 172.21.15.78:0/1662431269 >> 172.21.15.78:0/1662431269 conn(0x7fdee80c86a0 msgr2=0x7fdee80c8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 -- 172.21.15.78:0/1662431269 shutdown_connections 2024-02-19T15:00:46.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.662+0000 7fdef029a640 1 -- 172.21.15.78:0/1662431269 wait complete. 2024-02-19T15:00:46.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.663+0000 7fdef029a640 1 Processor -- start 2024-02-19T15:00:46.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.663+0000 7fdef029a640 1 -- start start 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdef029a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee80719d0 0x7fdee80adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdef029a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdee80c2ba0 0x7fdee8146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdef029a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 0x7fdee8146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdef029a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80aaa60 con 0x7fdee81503a0 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdef029a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8146ec0 con 0x7fdee80719d0 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdef029a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee81e9870 con 0x7fdee80c2ba0 2024-02-19T15:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 0x7fdee8146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 0x7fdee8146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37238/0 (socket says 172.21.15.78:37238) 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 -- 172.21.15.78:0/2279271384 learned_addr learned my addr 172.21.15.78:0/2279271384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeed835640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdee80c2ba0 0x7fdee8146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee036640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee80719d0 0x7fdee80adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 -- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdee80c2ba0 msgr2=0x7fdee8146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdee80c2ba0 0x7fdee8146440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 -- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee80719d0 msgr2=0x7fdee80adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee80719d0 0x7fdee80adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 -- 172.21.15.78:0/2279271384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0009400 con 0x7fdee81503a0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee036640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee80719d0 0x7fdee80adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.664+0000 7fdeee837640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 0x7fdee8146980 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fdee002b7e0 tx=0x7fdee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.665+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee00339b0 con 0x7fdee81503a0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.665+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee0033b10 con 0x7fdee81503a0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.665+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee81e8cf0 con 0x7fdee81503a0 2024-02-19T15:00:46.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.665+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee003c400 con 0x7fdee81503a0 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.665+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee81e92b0 con 0x7fdee81503a0 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.666+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee81541e0 con 0x7fdee81503a0 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.666+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdee003a040 con 0x7fdee81503a0 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.666+0000 7fdedeffd640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdecc06b810 0x7fdecc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.666+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fdee00ab860 con 0x7fdee81503a0 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.667+0000 7fdeee036640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdecc06b810 0x7fdecc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.667+0000 7fdeee036640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdecc06b810 0x7fdecc06dcd0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdee4005c00 tx=0x7fdee4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:46.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.671+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdee00765f0 con 0x7fdee81503a0 2024-02-19T15:00:46.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:46.841+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdee8074990 con 0x7fdee81503a0 2024-02-19T15:00:47.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.293+0000 7fdedeffd640 1 -- 172.21.15.78:0/2279271384 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v210) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdee00380b0 con 0x7fdee81503a0 2024-02-19T15:00:47.296 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdecc06b810 msgr2=0x7fdecc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdecc06b810 0x7fdecc06dcd0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdee4005c00 tx=0x7fdee4009040 comp rx=0 tx=0).stop 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 msgr2=0x7fdee8146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 0x7fdee8146980 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fdee002b7e0 tx=0x7fdee00046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 shutdown_connections 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdecc06b810 0x7fdecc06dcd0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdee81503a0 0x7fdee8146980 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdee80c2ba0 0x7fdee8146440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 --2- 172.21.15.78:0/2279271384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee80719d0 0x7fdee80adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 >> 172.21.15.78:0/2279271384 conn(0x7fdee80c86a0 msgr2=0x7fdee814e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 shutdown_connections 2024-02-19T15:00:47.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.296+0000 7fdef029a640 1 -- 172.21.15.78:0/2279271384 wait complete. 2024-02-19T15:00:47.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-19T15:00:47.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.417+0000 7f0d09fe9640 1 Processor -- start 2024-02-19T15:00:47.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.417+0000 7f0d09fe9640 1 -- start start 2024-02-19T15:00:47.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d09fe9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d040ace90 0x7f0d040c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d09fe9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d040c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d09fe9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d04152a60 con 0x7f0d04152680 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d09fe9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d040aa050 con 0x7f0d040ace90 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d09fe9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d041557e0 con 0x7f0d040c8ee0 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d03fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d040c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d02ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d040ace90 0x7f0d040c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d02ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d040ace90 0x7f0d040c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:36002/0 (socket says 172.21.15.78:36002) 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d037fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0d04152680 0x7f0d04155000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:32936/0 (socket says 172.21.15.78:32936) 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d02ffd640 1 -- 172.21.15.78:0/2307185798 learned_addr learned my addr 172.21.15.78:0/2307185798 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.418+0000 7f0d03fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d040c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:51062/0 (socket says 172.21.15.78:51062) 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.419+0000 7f0d03fff640 1 -- 172.21.15.78:0/2307185798 >> v2:172.21.15.150:3300/0 conn(0x7f0d040ace90 msgr2=0x7f0d040c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.419+0000 7f0d03fff640 1 --2- 172.21.15.78:0/2307185798 >> v2:172.21.15.150:3300/0 conn(0x7f0d040ace90 0x7f0d040c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.419+0000 7f0d03fff640 1 -- 172.21.15.78:0/2307185798 >> v1:172.21.15.150:6789/0 conn(0x7f0d04152680 legacy=0x7f0d04155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:47.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.419+0000 7f0d03fff640 1 -- 172.21.15.78:0/2307185798 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04155ac0 con 0x7f0d040c8ee0 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.419+0000 7f0d03fff640 1 --2- 172.21.15.78:0/2307185798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d040c9b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009d90 tx=0x7f0cfc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a4cb0c218d911c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d027fc640 1 -- 172.21.15.78:0/2307185798 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc02cbc0 con 0x7f0d040c8ee0 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d027fc640 1 -- 172.21.15.78:0/2307185798 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc02cd20 con 0x7f0d040c8ee0 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d027fc640 1 -- 172.21.15.78:0/2307185798 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc034660 con 0x7f0d040c8ee0 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2307185798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 msgr2=0x7f0d040c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2307185798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d040c9b80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009d90 tx=0x7f0cfc02be10 comp rx=0 tx=0).stop 2024-02-19T15:00:47.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2307185798 shutdown_connections 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2307185798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d040c9b80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2307185798 >> v2:172.21.15.150:3300/0 conn(0x7f0d040ace90 0x7f0d040c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2307185798 >> 172.21.15.78:0/2307185798 conn(0x7f0d040ac370 msgr2=0x7f0d040ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2307185798 shutdown_connections 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.420+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2307185798 wait complete. 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.421+0000 7f0d09fe9640 1 Processor -- start 2024-02-19T15:00:47.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 -- start start 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d040ace90 0x7f0d04159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d0414b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 0x7f0d0406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0414bec0 con 0x7f0d04152680 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d040c1160 con 0x7f0d040ace90 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d09fe9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d04157b40 con 0x7f0d040c8ee0 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d03fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 0x7f0d0406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d03fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 0x7f0d0406ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37240/0 (socket says 172.21.15.78:37240) 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.422+0000 7f0d03fff640 1 -- 172.21.15.78:0/2136271387 learned_addr learned my addr 172.21.15.78:0/2136271387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:47.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d02ffd640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d0414b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d037fe640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d040ace90 0x7f0d04159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d03fff640 1 -- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 msgr2=0x7f0d0414b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d03fff640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d0414b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d03fff640 1 -- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d040ace90 msgr2=0x7f0d04159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d03fff640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d040ace90 0x7f0d04159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d03fff640 1 -- 172.21.15.78:0/2136271387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc009a70 con 0x7f0d04152680 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d037fe640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d040ace90 0x7f0d04159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d03fff640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 0x7f0d0406ca50 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009eb0 tx=0x7f0cfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc040650 con 0x7f0d04152680 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc02ce90 con 0x7f0d04152680 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc0409f0 con 0x7f0d04152680 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d040c09a0 con 0x7f0d04152680 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.423+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0414c3b0 con 0x7f0d04152680 2024-02-19T15:00:47.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.424+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d04074990 con 0x7f0d04152680 2024-02-19T15:00:47.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.425+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0cfc040b50 con 0x7f0d04152680 2024-02-19T15:00:47.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.425+0000 7f0cebfff640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0cf006b810 0x7f0cf006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:47.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.425+0000 7f0d037fe640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0cf006b810 0x7f0cf006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:47.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.425+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f0cfc0ace00 con 0x7f0d04152680 2024-02-19T15:00:47.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.425+0000 7f0d037fe640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0cf006b810 0x7f0cf006dcd0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4000f60 tx=0x7f0cf4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:47.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.430+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0cfc077900 con 0x7f0d04152680 2024-02-19T15:00:47.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:47.605+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0d040c8780 con 0x7f0d04152680 2024-02-19T15:00:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.298+0000 7f0cebfff640 1 -- 172.21.15.78:0/2136271387 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v211) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0cfc0440d0 con 0x7f0d04152680 2024-02-19T15:00:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0cf006b810 msgr2=0x7f0cf006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0cf006b810 0x7f0cf006dcd0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4000f60 tx=0x7f0cf4009040 comp rx=0 tx=0).stop 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 msgr2=0x7f0d0406ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 0x7f0d0406ca50 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009eb0 tx=0x7f0cfc004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 shutdown_connections 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0cf006b810 0x7f0cf006dcd0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d04152680 0x7f0d0406ca50 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d040c8ee0 0x7f0d0414b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 --2- 172.21.15.78:0/2136271387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d040ace90 0x7f0d04159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 >> 172.21.15.78:0/2136271387 conn(0x7f0d040ac370 msgr2=0x7f0d0414f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 shutdown_connections 2024-02-19T15:00:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.301+0000 7f0d09fe9640 1 -- 172.21.15.78:0/2136271387 wait complete. 2024-02-19T15:00:48.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-19T15:00:48.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.424+0000 7f9790d02640 1 Processor -- start 2024-02-19T15:00:48.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.424+0000 7f9790d02640 1 -- start start 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f9790d02640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f978c152700 0x7f978c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f9790d02640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f978c0c5730 0x7f978c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f9790d02640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c1551f0 con 0x7f978c0c8e00 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f9790d02640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c155490 con 0x7f978c0c5730 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f9790d02640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c155770 con 0x7f978c152700 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f9789d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f978c0c5730 0x7f978c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f978ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f978c0c8e00 0x7f978c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59834/0 (socket says 172.21.15.78:59834) 2024-02-19T15:00:48.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f978ad9d640 1 -- 172.21.15.78:0/3261682728 learned_addr learned my addr 172.21.15.78:0/3261682728 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.425+0000 7f978a59c640 1 --2- 172.21.15.78:0/3261682728 >> v2:172.21.15.150:3300/0 conn(0x7f978c152700 0x7f978c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f9789d9b640 1 -- 172.21.15.78:0/3261682728 >> v2:172.21.15.150:3300/0 conn(0x7f978c152700 msgr2=0x7f978c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f9789d9b640 1 --2- 172.21.15.78:0/3261682728 >> v2:172.21.15.150:3300/0 conn(0x7f978c152700 0x7f978c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f9789d9b640 1 -- 172.21.15.78:0/3261682728 >> v1:172.21.15.78:6789/0 conn(0x7f978c0c8e00 legacy=0x7f978c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f9789d9b640 1 -- 172.21.15.78:0/3261682728 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c155a50 con 0x7f978c0c5730 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f978959a640 1 -- 172.21.15.78:0/3261682728 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741105099 0 0) 0x7f978c1551f0 con 0x7f978c0c8e00 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f9789d9b640 1 --2- 172.21.15.78:0/3261682728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c0c5730 0x7f978c06c2a0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f978c08bad0 tx=0x7f978002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13664d80f567167 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.426+0000 7f978959a640 1 -- 172.21.15.78:0/3261682728 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978002e070 con 0x7f978c0c5730 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f978959a640 1 -- 172.21.15.78:0/3261682728 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97800315a0 con 0x7f978c0c5730 2024-02-19T15:00:48.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f978959a640 1 -- 172.21.15.78:0/3261682728 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780031930 con 0x7f978c0c5730 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 -- 172.21.15.78:0/3261682728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c0c5730 msgr2=0x7f978c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 --2- 172.21.15.78:0/3261682728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c0c5730 0x7f978c06c2a0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f978c08bad0 tx=0x7f978002b840 comp rx=0 tx=0).stop 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 -- 172.21.15.78:0/3261682728 shutdown_connections 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 --2- 172.21.15.78:0/3261682728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c0c5730 0x7f978c06c2a0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 --2- 172.21.15.78:0/3261682728 >> v2:172.21.15.150:3300/0 conn(0x7f978c152700 0x7f978c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 -- 172.21.15.78:0/3261682728 >> 172.21.15.78:0/3261682728 conn(0x7f978c0cac90 msgr2=0x7f978c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 -- 172.21.15.78:0/3261682728 shutdown_connections 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.427+0000 7f9790d02640 1 -- 172.21.15.78:0/3261682728 wait complete. 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.428+0000 7f9790d02640 1 Processor -- start 2024-02-19T15:00:48.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.428+0000 7f9790d02640 1 -- start start 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9790d02640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0c5730 0x7f978c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9790d02640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c0c8e00 0x7f978c14a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9790d02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 0x7f978c0ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9790d02640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0ad210 con 0x7f978c152700 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9790d02640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0aa400 con 0x7f978c0c8e00 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9790d02640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1eb670 con 0x7f978c0c5730 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 0x7f978c0ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 0x7f978c0ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46050/0 (socket says 172.21.15.78:46050) 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 -- 172.21.15.78:0/3218497674 learned_addr learned my addr 172.21.15.78:0/3218497674 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978a59c640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0c5730 0x7f978c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9789d9b640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c0c8e00 0x7f978c14a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 -- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0c5730 msgr2=0x7f978c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0c5730 0x7f978c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 -- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c0c8e00 msgr2=0x7f978c14a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c0c8e00 0x7f978c14a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f978ad9d640 1 -- 172.21.15.78:0/3218497674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9780009400 con 0x7f978c152700 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.429+0000 7f9789d9b640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c0c8e00 0x7f978c14a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.430+0000 7f978ad9d640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 0x7f978c0ac180 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f978400c450 tx=0x7f978400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.430+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840074e0 con 0x7f978c152700 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.430+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784007660 con 0x7f978c152700 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.430+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c1eb950 con 0x7f978c152700 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.430+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784004a20 con 0x7f978c152700 2024-02-19T15:00:48.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.430+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c159350 con 0x7f978c152700 2024-02-19T15:00:48.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.431+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0aa260 con 0x7f978c152700 2024-02-19T15:00:48.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.431+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9784004b80 con 0x7f978c152700 2024-02-19T15:00:48.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.432+0000 7f977affd640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:48.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.432+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f9784089720 con 0x7f978c152700 2024-02-19T15:00:48.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.432+0000 7f978a59c640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:48.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.432+0000 7f978a59c640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f977c009f60 tx=0x7f977c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:48.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.436+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f978c0aa260 con 0x7f978c152700 2024-02-19T15:00:48.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:48.646+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f978c0aa260 con 0x7f978c152700 2024-02-19T15:00:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.300+0000 7f977affd640 1 -- 172.21.15.78:0/3218497674 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 33 and data pool 34 v188) v1 ==== 174+0+0 (secure 0 0 0) 0x7f978c0aa260 con 0x7f978c152700 2024-02-19T15:00:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 33 and data pool 34 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.305+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 msgr2=0x7f976c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.305+0000 7f9790d02640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f977c009f60 tx=0x7f977c00b040 comp rx=0 tx=0).stop 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.305+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 msgr2=0x7f978c0ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.305+0000 7f9790d02640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 0x7f978c0ac180 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f978400c450 tx=0x7f978400c910 comp rx=0 tx=0).stop 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.306+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 shutdown_connections 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.306+0000 7f9790d02640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f976c06b810 0x7f976c06dcd0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.306+0000 7f9790d02640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f978c152700 0x7f978c0ac180 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.306+0000 7f9790d02640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c0c8e00 0x7f978c14a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.306+0000 7f9790d02640 1 --2- 172.21.15.78:0/3218497674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0c5730 0x7f978c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.306+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 >> 172.21.15.78:0/3218497674 conn(0x7f978c0cac90 msgr2=0x7f978c0cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:49.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.308+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 shutdown_connections 2024-02-19T15:00:49.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.309+0000 7f9790d02640 1 -- 172.21.15.78:0/3218497674 wait complete. 2024-02-19T15:00:49.342 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:49.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.440+0000 7fe9e8a15640 1 Processor -- start 2024-02-19T15:00:49.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.440+0000 7fe9e8a15640 1 -- start start 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e8a15640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe9e4152670 0x7fe9e4071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e8a15640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152a50 0x7fe9e4155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e8a15640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9e40702d0 0x7fe9e4158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e8a15640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e4071bf0 con 0x7fe9e4152670 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e8a15640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e4158a20 con 0x7fe9e40702d0 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e8a15640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4155550 con 0x7fe9e4152a50 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe9e4152670 0x7fe9e4071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe9e4152670 0x7fe9e4071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46054/0 (socket says 172.21.15.78:46054) 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.441+0000 7fe9e259c640 1 -- 172.21.15.78:0/2101838743 learned_addr learned my addr 172.21.15.78:0/2101838743 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e1d9b640 1 --2- 172.21.15.78:0/2101838743 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152a50 0x7fe9e4155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e259c640 1 -- 172.21.15.78:0/2101838743 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152a50 msgr2=0x7fe9e4155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e259c640 1 --2- 172.21.15.78:0/2101838743 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152a50 0x7fe9e4155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e259c640 1 -- 172.21.15.78:0/2101838743 >> v2:172.21.15.150:3300/0 conn(0x7fe9e40702d0 msgr2=0x7fe9e4158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e2d9d640 1 --2- 172.21.15.78:0/2101838743 >> v2:172.21.15.150:3300/0 conn(0x7fe9e40702d0 0x7fe9e4158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e259c640 1 --2- 172.21.15.78:0/2101838743 >> v2:172.21.15.150:3300/0 conn(0x7fe9e40702d0 0x7fe9e4158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e259c640 1 -- 172.21.15.78:0/2101838743 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e40c6780 con 0x7fe9e4152670 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e2d9d640 1 --2- 172.21.15.78:0/2101838743 >> v2:172.21.15.150:3300/0 conn(0x7fe9e40702d0 0x7fe9e4158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e259c640 1 --2- 172.21.15.78:0/2101838743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e4152670 0x7fe9e4071050 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4070df0 tx=0x7fe9d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d985135d7963e123 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.442+0000 7fe9e159a640 1 -- 172.21.15.78:0/2101838743 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d402cb00 con 0x7fe9e4152670 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e159a640 1 -- 172.21.15.78:0/2101838743 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d402cc60 con 0x7fe9e4152670 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e159a640 1 -- 172.21.15.78:0/2101838743 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d40343f0 con 0x7fe9e4152670 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2101838743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e4152670 msgr2=0x7fe9e4071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2101838743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e4152670 0x7fe9e4071050 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4070df0 tx=0x7fe9d402bf60 comp rx=0 tx=0).stop 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2101838743 shutdown_connections 2024-02-19T15:00:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2101838743 >> v2:172.21.15.150:3300/0 conn(0x7fe9e40702d0 0x7fe9e4158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2101838743 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152a50 0x7fe9e4155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2101838743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e4152670 0x7fe9e4071050 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2101838743 >> 172.21.15.78:0/2101838743 conn(0x7fe9e40c85e0 msgr2=0x7fe9e40c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:49.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2101838743 shutdown_connections 2024-02-19T15:00:49.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.443+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2101838743 wait complete. 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.444+0000 7fe9e8a15640 1 Processor -- start 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.444+0000 7fe9e8a15640 1 -- start start 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e8a15640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 0x7fe9e40ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e8a15640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152670 0x7fe9e4151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e8a15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e4152a50 0x7fe9e4151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e8a15640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e414ff00 con 0x7fe9e40702d0 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e8a15640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41ea5f0 con 0x7fe9e4152a50 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e8a15640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41ea8d0 con 0x7fe9e4152670 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 0x7fe9e40ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 0x7fe9e40ab9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46056/0 (socket says 172.21.15.78:46056) 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 -- 172.21.15.78:0/2780690891 learned_addr learned my addr 172.21.15.78:0/2780690891 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 -- 172.21.15.78:0/2780690891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152670 msgr2=0x7fe9e4151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152670 0x7fe9e4151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 -- 172.21.15.78:0/2780690891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e4152a50 msgr2=0x7fe9e4151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e4152a50 0x7fe9e4151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.445+0000 7fe9e259c640 1 -- 172.21.15.78:0/2780690891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4009b30 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.446+0000 7fe9e259c640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 0x7fe9e40ab9f0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40093e0 tx=0x7fe9d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.446+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d403e450 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.446+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d403e5b0 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.446+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e41eabb0 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.446+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d403e960 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.446+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e41eb170 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.447+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e40676a0 con 0x7fe9e40702d0 2024-02-19T15:00:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.447+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe9d403eac0 con 0x7fe9e40702d0 2024-02-19T15:00:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.448+0000 7fe9daffd640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9c406b810 0x7fe9c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.448+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe9d40ad290 con 0x7fe9e40702d0 2024-02-19T15:00:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.448+0000 7fe9e1d9b640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9c406b810 0x7fe9c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.448+0000 7fe9e1d9b640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9c406b810 0x7fe9c406dcd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc006fd0 tx=0x7fe9cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.452+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9d4077bb0 con 0x7fe9e40702d0 2024-02-19T15:00:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.625+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9e41eaa10 con 0x7fe9e40702d0 2024-02-19T15:00:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.626+0000 7fe9daffd640 1 -- 172.21.15.78:0/2780690891 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+16889 (secure 0 0 0) 0x7fe9e41eaa10 con 0x7fe9e40702d0 2024-02-19T15:00:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:49.629 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":212,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:00:48.648185+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:00:46.861039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-19T15:00:47.624123+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9c406b810 msgr2=0x7fe9c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9c406b810 0x7fe9c406dcd0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc006fd0 tx=0x7fe9cc008040 comp rx=0 tx=0).stop 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 msgr2=0x7fe9e40ab9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 0x7fe9e40ab9f0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40093e0 tx=0x7fe9d4004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 shutdown_connections 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe9c406b810 0x7fe9c406dcd0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e4152a50 0x7fe9e4151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe9e4152670 0x7fe9e4151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 --2- 172.21.15.78:0/2780690891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e40702d0 0x7fe9e40ab9f0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 >> 172.21.15.78:0/2780690891 conn(0x7fe9e40c85e0 msgr2=0x7fe9e40a86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 shutdown_connections 2024-02-19T15:00:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.628+0000 7fe9e8a15640 1 -- 172.21.15.78:0/2780690891 wait complete. 2024-02-19T15:00:49.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.739+0000 7fd45f430640 1 Processor -- start 2024-02-19T15:00:49.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.739+0000 7fd45f430640 1 -- start start 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45f430640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45f430640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4580730a0 0x7fd458158550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45f430640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd458072ec0 con 0x7fd4580c19a0 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45f430640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45806fa40 con 0x7fd4580730a0 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45f430640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45806f270 con 0x7fd458152610 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45c9cb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4580730a0 0x7fd458158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:59436/0 (socket says 172.21.15.78:59436) 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 -- 172.21.15.78:0/2273965635 learned_addr learned my addr 172.21.15.78:0/2273965635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 -- 172.21.15.78:0/2273965635 >> v2:172.21.15.150:3300/0 conn(0x7fd4580730a0 msgr2=0x7fd458158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 --2- 172.21.15.78:0/2273965635 >> v2:172.21.15.150:3300/0 conn(0x7fd4580730a0 0x7fd458158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 -- 172.21.15.78:0/2273965635 >> v1:172.21.15.150:6789/0 conn(0x7fd4580c19a0 legacy=0x7fd45806ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.740+0000 7fd45d1cc640 1 -- 172.21.15.78:0/2273965635 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45806f4c0 con 0x7fd458152610 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.741+0000 7fd45d1cc640 1 --2- 172.21.15.78:0/2273965635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580c1460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd454009870 tx=0x7fd45402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a53f78c53f4bd3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.741+0000 7fd44ffff640 1 -- 172.21.15.78:0/2273965635 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454031640 con 0x7fd458152610 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.741+0000 7fd44ffff640 1 -- 172.21.15.78:0/2273965635 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4540317a0 con 0x7fd458152610 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.741+0000 7fd44ffff640 1 -- 172.21.15.78:0/2273965635 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454031b30 con 0x7fd458152610 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 -- 172.21.15.78:0/2273965635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 msgr2=0x7fd4580c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 --2- 172.21.15.78:0/2273965635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580c1460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd454009870 tx=0x7fd45402b8f0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 -- 172.21.15.78:0/2273965635 shutdown_connections 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 --2- 172.21.15.78:0/2273965635 >> v2:172.21.15.150:3300/0 conn(0x7fd4580730a0 0x7fd458158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 --2- 172.21.15.78:0/2273965635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580c1460 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 -- 172.21.15.78:0/2273965635 >> 172.21.15.78:0/2273965635 conn(0x7fd4580c7590 msgr2=0x7fd4580c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 -- 172.21.15.78:0/2273965635 shutdown_connections 2024-02-19T15:00:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.742+0000 7fd45f430640 1 -- 172.21.15.78:0/2273965635 wait complete. 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.743+0000 7fd45f430640 1 Processor -- start 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.743+0000 7fd45f430640 1 -- start start 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 0x7fd45814b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4580c19a0 0x7fd45814ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45814ff00 con 0x7fd4580730a0 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd458157670 con 0x7fd4580c19a0 2024-02-19T15:00:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4581eeae0 con 0x7fd458152610 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 0x7fd45814b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 0x7fd45814b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46060/0 (socket says 172.21.15.78:46060) 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 -- 172.21.15.78:0/3733518672 learned_addr learned my addr 172.21.15.78:0/3733518672 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d9cd640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45c9cb640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4580c19a0 0x7fd45814ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 -- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 msgr2=0x7fd4580a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 -- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4580c19a0 msgr2=0x7fd45814ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4580c19a0 0x7fd45814ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 -- 172.21.15.78:0/3733518672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4540094d0 con 0x7fd4580730a0 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45d1cc640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 0x7fd45814b4e0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fd45402b930 tx=0x7fd4540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45403a930 con 0x7fd4580730a0 2024-02-19T15:00:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45403aab0 con 0x7fd4580730a0 2024-02-19T15:00:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4581eedc0 con 0x7fd4580730a0 2024-02-19T15:00:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.744+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45403ae60 con 0x7fd4580730a0 2024-02-19T15:00:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.745+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd458158b20 con 0x7fd4580730a0 2024-02-19T15:00:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.746+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4580a6eb0 con 0x7fd4580730a0 2024-02-19T15:00:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.746+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd45402f050 con 0x7fd4580730a0 2024-02-19T15:00:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.746+0000 7fd44dffb640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd43c06b810 0x7fd43c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.746+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4540ac1e0 con 0x7fd4580730a0 2024-02-19T15:00:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.747+0000 7fd45c9cb640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd43c06b810 0x7fd43c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.747+0000 7fd45c9cb640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd43c06b810 0x7fd43c06dcd0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd4580ad000 tx=0x7fd448008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.751+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4540769e0 con 0x7fd4580730a0 2024-02-19T15:00:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.922+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd458074990 con 0x7fd4580730a0 2024-02-19T15:00:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.923+0000 7fd44dffb640 1 -- 172.21.15.78:0/3733518672 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+16889 (secure 0 0 0) 0x7fd45407a0a0 con 0x7fd4580730a0 2024-02-19T15:00:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:49.926 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":212,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:00:48.648185+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:00:46.861039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-19T15:00:47.624123+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd43c06b810 msgr2=0x7fd43c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd43c06b810 0x7fd43c06dcd0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd4580ad000 tx=0x7fd448008040 comp rx=0 tx=0).stop 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 msgr2=0x7fd45814b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 0x7fd45814b4e0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7fd45402b930 tx=0x7fd4540046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 shutdown_connections 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd43c06b810 0x7fd43c06dcd0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd458152610 0x7fd4580a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4580c19a0 0x7fd45814ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 --2- 172.21.15.78:0/3733518672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4580730a0 0x7fd45814b4e0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 >> 172.21.15.78:0/3733518672 conn(0x7fd4580c7590 msgr2=0x7fd4580c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 shutdown_connections 2024-02-19T15:00:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:49.925+0000 7fd45f430640 1 -- 172.21.15.78:0/3733518672 wait complete. 2024-02-19T15:00:49.939 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-19T15:00:50.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.040+0000 7f4252927640 1 Processor -- start 2024-02-19T15:00:50.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.040+0000 7f4252927640 1 -- start start 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f4252927640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f424c152480 0x7f424c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f4252927640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c155ef0 con 0x7f424c0c87a0 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f4252927640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424c06dec0 con 0x7f424c0c7070 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f4252927640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424c06d520 con 0x7f424c152480 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f4250ec4640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f424c0c87a0 0x7f424c0c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59846/0 (socket says 172.21.15.78:59846) 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f4250ec4640 1 -- 172.21.15.78:0/2513025572 learned_addr learned my addr 172.21.15.78:0/2513025572 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424bfff640 1 --2- 172.21.15.78:0/2513025572 >> v2:172.21.15.150:3300/0 conn(0x7f424c152480 0x7f424c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576409396 0 0) 0x7f4244001490 con 0x7f424c0c87a0 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234003520 con 0x7f424c0c87a0 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80668211 0 0) 0x7f424c06dec0 con 0x7f424c0c7070 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4244001490 con 0x7f424c0c7070 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3033257045 0 0) 0x7f4234003520 con 0x7f424c0c87a0 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 >> v2:172.21.15.150:3300/0 conn(0x7f424c152480 msgr2=0x7f424c1559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 --2- 172.21.15.78:0/2513025572 >> v2:172.21.15.150:3300/0 conn(0x7f424c152480 0x7f424c1559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 >> v1:172.21.15.150:6789/0 conn(0x7f424c0c7070 legacy=0x7f424c06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c0c75c0 con 0x7f424c0c87a0 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4244001b80 con 0x7f424c0c87a0 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 >> v1:172.21.15.78:6789/0 conn(0x7f424c0c87a0 legacy=0x7f424c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4234003b50 0x7f4234003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42340047c0 0x7f4234004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f423400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c06d520 con 0x7f4234005180 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423400c050 con 0x7f42340047c0 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.041+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423400c210 con 0x7f4234003b50 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f423400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f424bfff640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4234003b50 0x7f4234003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f424b7fe640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42340047c0 0x7f4234004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 -- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4234003b50 msgr2=0x7f4234003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4234003b50 0x7f4234003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 -- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42340047c0 msgr2=0x7f4234004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42340047c0 0x7f4234004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 -- 172.21.15.78:0/2513025572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234003520 con 0x7f4234005180 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.042+0000 7f4250ec4640 1 -- 172.21.15.78:0/2513025572 reap_dead start 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4250ec4640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f423400b980 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f424400a6e0 tx=0x7f42340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330b68f81803f758 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244034630 con 0x7f4234005180 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424402f070 con 0x7f4234005180 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f424affd640 1 -- 172.21.15.78:0/2513025572 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244034a10 con 0x7f4234005180 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 -- 172.21.15.78:0/2513025572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 msgr2=0x7f423400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f423400b980 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f424400a6e0 tx=0x7f42340057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 -- 172.21.15.78:0/2513025572 shutdown_connections 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 --2- 172.21.15.78:0/2513025572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f423400b980 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 -- 172.21.15.78:0/2513025572 >> 172.21.15.78:0/2513025572 conn(0x7f424c0ca8a0 msgr2=0x7f424c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 -- 172.21.15.78:0/2513025572 shutdown_connections 2024-02-19T15:00:50.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.043+0000 7f4252927640 1 -- 172.21.15.78:0/2513025572 wait complete. 2024-02-19T15:00:50.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.044+0000 7f4252927640 1 Processor -- start 2024-02-19T15:00:50.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 -- start start 2024-02-19T15:00:50.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f424c0acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f424c0c4bb0 0x7f424c1ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f424c151250 0x7f424c0be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c14ba30 con 0x7f4234005180 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c0abc50 con 0x7f424c151250 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f4252927640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424c1593a0 con 0x7f424c0c4bb0 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f424bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f424c0acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f424bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f424c0acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46076/0 (socket says 172.21.15.78:46076) 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.045+0000 7f424bfff640 1 -- 172.21.15.78:0/1511311228 learned_addr learned my addr 172.21.15.78:0/1511311228 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f4250ec4640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f424c151250 0x7f424c0be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424b7fe640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f424c0c4bb0 0x7f424c1ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424bfff640 1 -- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f424c0c4bb0 msgr2=0x7f424c1ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424bfff640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f424c0c4bb0 0x7f424c1ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424bfff640 1 -- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f424c151250 msgr2=0x7f424c0be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424bfff640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f424c151250 0x7f424c0be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424bfff640 1 -- 172.21.15.78:0/1511311228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c0abea0 con 0x7f4234005180 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f424bfff640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f424c0acc60 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f423c0118e0 tx=0x7f423c011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c00faf0 con 0x7f4234005180 2024-02-19T15:00:50.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c00fc50 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c014440 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424c0ac180 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.046+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424c0ac740 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.047+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424c074990 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.047+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f423c0145a0 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.048+0000 7f4248ff9640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f423806b810 0x7f423806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.048+0000 7f424b7fe640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f423806b810 0x7f423806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.048+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f423c08ac50 con 0x7f4234005180 2024-02-19T15:00:50.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.048+0000 7f424b7fe640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f423806b810 0x7f423806dcd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4240000c00 tx=0x7f4240009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:50.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.052+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f423c090020 con 0x7f4234005180 2024-02-19T15:00:50.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.259+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f424c06dae0 con 0x7f4234005180 2024-02-19T15:00:50.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.318+0000 7f4248ff9640 1 -- 172.21.15.78:0/1511311228 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v189) v1 ==== 118+0+0 (secure 0 0 0) 0x7f423c0554e0 con 0x7f4234005180 2024-02-19T15:00:50.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f423806b810 msgr2=0x7f423806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f423806b810 0x7f423806dcd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4240000c00 tx=0x7f4240009040 comp rx=0 tx=0).stop 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 msgr2=0x7f424c0acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f424c0acc60 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f423c0118e0 tx=0x7f423c011da0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 shutdown_connections 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f423806b810 0x7f423806dcd0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f424c151250 0x7f424c0be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f424c0c4bb0 0x7f424c1ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 --2- 172.21.15.78:0/1511311228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4234005180 0x7f424c0acc60 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.320+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 >> 172.21.15.78:0/1511311228 conn(0x7f424c0ca8a0 msgr2=0x7f424c1541b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.321+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 shutdown_connections 2024-02-19T15:00:50.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.321+0000 7f4252927640 1 -- 172.21.15.78:0/1511311228 wait complete. 2024-02-19T15:00:50.334 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-19T15:00:50.334 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-19T15:00:50.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.434+0000 7f516d548640 1 Processor -- start 2024-02-19T15:00:50.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.434+0000 7f516d548640 1 -- start start 2024-02-19T15:00:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f516d548640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51681537a0 0x7f51680c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f516d548640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5168156860 0x7f5168156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f516d548640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51681563b0 con 0x7f516814cd50 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f516d548640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5168153cd0 con 0x7f5168156860 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f516d548640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516806b6e0 con 0x7f51681537a0 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f5166ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f516814cd50 0x7f51680c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59856/0 (socket says 172.21.15.78:59856) 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f5166ffd640 1 -- 172.21.15.78:0/906091380 learned_addr learned my addr 172.21.15.78:0/906091380 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f51677fe640 1 --2- 172.21.15.78:0/906091380 >> v2:172.21.15.150:3300/0 conn(0x7f5168156860 0x7f5168156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f51667fc640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51681537a0 0x7f51680c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f5165ffb640 1 -- 172.21.15.78:0/906091380 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871267177 0 0) 0x7f51681563b0 con 0x7f516814cd50 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.435+0000 7f5165ffb640 1 -- 172.21.15.78:0/906091380 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5154003520 con 0x7f516814cd50 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.436+0000 7f51677fe640 1 -- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51681537a0 msgr2=0x7f51680c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.436+0000 7f51677fe640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51681537a0 0x7f51680c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.436+0000 7f51677fe640 1 -- 172.21.15.78:0/906091380 >> v1:172.21.15.78:6789/0 conn(0x7f516814cd50 legacy=0x7f51680c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.436+0000 7f51677fe640 1 -- 172.21.15.78:0/906091380 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5168157230 con 0x7f5168156860 2024-02-19T15:00:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.436+0000 7f51667fc640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51681537a0 0x7f51680c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:50.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.436+0000 7f51677fe640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5168156860 0x7f5168156cf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f51600097a0 tx=0x7f516002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6da98b34753fd368 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f5165ffb640 1 -- 172.21.15.78:0/906091380 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516002f050 con 0x7f5168156860 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f5165ffb640 1 -- 172.21.15.78:0/906091380 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5160031700 con 0x7f5168156860 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f5165ffb640 1 -- 172.21.15.78:0/906091380 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160031a90 con 0x7f5168156860 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 -- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5168156860 msgr2=0x7f5168156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5168156860 0x7f5168156cf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f51600097a0 tx=0x7f516002b7a0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 -- 172.21.15.78:0/906091380 shutdown_connections 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5168156860 0x7f5168156cf0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 --2- 172.21.15.78:0/906091380 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f51681537a0 0x7f51680c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 -- 172.21.15.78:0/906091380 >> 172.21.15.78:0/906091380 conn(0x7f5168066e70 msgr2=0x7f51680a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 -- 172.21.15.78:0/906091380 shutdown_connections 2024-02-19T15:00:50.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.437+0000 7f516d548640 1 -- 172.21.15.78:0/906091380 wait complete. 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.438+0000 7f516d548640 1 Processor -- start 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.438+0000 7f516d548640 1 -- start start 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f516d548640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f516814cd50 0x7f51681460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f516d548640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51681537a0 0x7f5168146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f516d548640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 0x7f51680c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f516d548640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5168146b70 con 0x7f5168156860 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f516d548640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516814dd80 con 0x7f51681537a0 2024-02-19T15:00:50.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f516d548640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5168149ad0 con 0x7f516814cd50 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f51677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 0x7f51680c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f51677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 0x7f51680c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46078/0 (socket says 172.21.15.78:46078) 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f51677fe640 1 -- 172.21.15.78:0/2888399175 learned_addr learned my addr 172.21.15.78:0/2888399175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f51667fc640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51681537a0 0x7f5168146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f5166ffd640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f516814cd50 0x7f51681460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f51667fc640 1 -- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f516814cd50 msgr2=0x7f51681460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.439+0000 7f51667fc640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f516814cd50 0x7f51681460f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f51667fc640 1 -- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 msgr2=0x7f51680c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f51667fc640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 0x7f51680c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f51667fc640 1 -- 172.21.15.78:0/2888399175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5150009400 con 0x7f51681537a0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f51677fe640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 0x7f51680c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f51667fc640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51681537a0 0x7f5168146630 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f515000e840 tx=0x7f515000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000c940 con 0x7f51681537a0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515000caa0 con 0x7f51681537a0 2024-02-19T15:00:50.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5160009400 con 0x7f51681537a0 2024-02-19T15:00:50.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000ce50 con 0x7f51681537a0 2024-02-19T15:00:50.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.440+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5168149d70 con 0x7f51681537a0 2024-02-19T15:00:50.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.441+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5168074990 con 0x7f51681537a0 2024-02-19T15:00:50.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.441+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5150013470 con 0x7f51681537a0 2024-02-19T15:00:50.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.442+0000 7f515f7fe640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f514806b840 0x7f514806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:50.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.442+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f515008a3f0 con 0x7f51681537a0 2024-02-19T15:00:50.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.442+0000 7f5166ffd640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f514806b840 0x7f514806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:50.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.442+0000 7f5166ffd640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f514806b840 0x7f514806dd00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5158005c00 tx=0x7f5158009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:50.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.446+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5150054d30 con 0x7f51681537a0 2024-02-19T15:00:50.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:50.655+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5168152ad0 con 0x7f51681537a0 2024-02-19T15:00:51.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.322+0000 7f515f7fe640 1 -- 172.21.15.78:0/2888399175 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v190) v1 ==== 122+0+0 (secure 0 0 0) 0x7f51500583f0 con 0x7f51681537a0 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f514806b840 msgr2=0x7f514806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f514806b840 0x7f514806dd00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5158005c00 tx=0x7f5158009040 comp rx=0 tx=0).stop 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51681537a0 msgr2=0x7f5168146630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51681537a0 0x7f5168146630 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f515000e840 tx=0x7f515000ed00 comp rx=0 tx=0).stop 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 shutdown_connections 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f514806b840 0x7f514806dd00 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5168156860 0x7f51680c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51681537a0 0x7f5168146630 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 --2- 172.21.15.78:0/2888399175 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f516814cd50 0x7f51681460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 >> 172.21.15.78:0/2888399175 conn(0x7f5168066e70 msgr2=0x7f51681544f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:51.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 shutdown_connections 2024-02-19T15:00:51.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.325+0000 7f516d548640 1 -- 172.21.15.78:0/2888399175 wait complete. 2024-02-19T15:00:51.341 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-19T15:00:51.341 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:51.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.442+0000 7fbdbd3e4640 1 Processor -- start 2024-02-19T15:00:51.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.442+0000 7fbdbd3e4640 1 -- start start 2024-02-19T15:00:51.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.442+0000 7fbdbd3e4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbdb80c8e00 0x7fbdb806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdbd3e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdbd3e4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb806cd20 con 0x7fbdb8152700 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdbd3e4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb81551f0 con 0x7fbdb80c8e00 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdbd3e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8155490 con 0x7fbdb806d910 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdaffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbdb80c8e00 0x7fbdb806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbdb8152700 0x7fbdb80c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53036/0 (socket says 172.21.15.78:53036) 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb6ffd640 1 -- 172.21.15.78:0/3768530857 learned_addr learned my addr 172.21.15.78:0/3768530857 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb806c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:59482/0 (socket says 172.21.15.78:59482) 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb67fc640 1 -- 172.21.15.78:0/3768530857 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938834628 0 0) 0x7fbdb806cd20 con 0x7fbdb8152700 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb67fc640 1 -- 172.21.15.78:0/3768530857 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda4003520 con 0x7fbdb8152700 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdaffff640 1 -- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 msgr2=0x7fbdb806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:51.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdaffff640 1 --2- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdaffff640 1 -- 172.21.15.78:0/3768530857 >> v1:172.21.15.150:6789/0 conn(0x7fbdb8152700 legacy=0x7fbdb80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdaffff640 1 -- 172.21.15.78:0/3768530857 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8155770 con 0x7fbdb80c8e00 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.443+0000 7fbdb77fe640 1 --2- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.444+0000 7fbdaffff640 1 --2- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb80c8e00 0x7fbdb806c2a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbda00097a0 tx=0x7fbda002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9195e6904a52f826 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.444+0000 7fbdb67fc640 1 -- 172.21.15.78:0/3768530857 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda002e070 con 0x7fbdb80c8e00 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.444+0000 7fbdb67fc640 1 -- 172.21.15.78:0/3768530857 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda00315a0 con 0x7fbdb80c8e00 2024-02-19T15:00:51.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.444+0000 7fbdb67fc640 1 -- 172.21.15.78:0/3768530857 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0031930 con 0x7fbdb80c8e00 2024-02-19T15:00:51.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb80c8e00 msgr2=0x7fbdb806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb80c8e00 0x7fbdb806c2a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbda00097a0 tx=0x7fbda002b840 comp rx=0 tx=0).stop 2024-02-19T15:00:51.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/3768530857 shutdown_connections 2024-02-19T15:00:51.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/3768530857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb80c8e00 0x7fbdb806c2a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/3768530857 >> 172.21.15.78:0/3768530857 conn(0x7fbdb80cac90 msgr2=0x7fbdb80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/3768530857 shutdown_connections 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.445+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/3768530857 wait complete. 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 Processor -- start 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 -- start start 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb814a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 0x7fbdb8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb8152700 0x7fbdb8158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb81eb320 con 0x7fbdb80c8e00 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb81ea7a0 con 0x7fbdb8152700 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.446+0000 7fbdbd3e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb81eaa80 con 0x7fbdb806d910 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 0x7fbdb8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 0x7fbdb8156410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46092/0 (socket says 172.21.15.78:46092) 2024-02-19T15:00:51.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 -- 172.21.15.78:0/2969944975 learned_addr learned my addr 172.21.15.78:0/2969944975 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdb6ffd640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb814a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdb77fe640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb8152700 0x7fbdb8158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 -- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 msgr2=0x7fbdb814a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb814a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 -- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb8152700 msgr2=0x7fbdb8158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb8152700 0x7fbdb8158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 -- 172.21.15.78:0/2969944975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0009a70 con 0x7fbdb80c8e00 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaffff640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 0x7fbdb8156410 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fbda0034040 tx=0x7fbda00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0032630 con 0x7fbdb80c8e00 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda0032790 con 0x7fbdb80c8e00 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda0009400 con 0x7fbdb80c8e00 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0032b40 con 0x7fbdb80c8e00 2024-02-19T15:00:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.447+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb81eb040 con 0x7fbdb80c8e00 2024-02-19T15:00:51.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.448+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb8074990 con 0x7fbdb80c8e00 2024-02-19T15:00:51.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.449+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbda0032ca0 con 0x7fbdb80c8e00 2024-02-19T15:00:51.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.449+0000 7fbdaf7fe640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9806b810 0x7fbd9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.449+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbda00ac120 con 0x7fbdb80c8e00 2024-02-19T15:00:51.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.449+0000 7fbdb6ffd640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9806b810 0x7fbd9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.450+0000 7fbdb6ffd640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9806b810 0x7fbd9806dcd0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005880 tx=0x7fbda8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:51.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.454+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbda00769b0 con 0x7fbdb80c8e00 2024-02-19T15:00:51.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.663+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdb80c8770 con 0x7fbdb80c8e00 2024-02-19T15:00:51.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.663+0000 7fbdaf7fe640 1 -- 172.21.15.78:0/2969944975 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4956 (secure 0 0 0) 0x7fbda007a070 con 0x7fbdb80c8e00 2024-02-19T15:00:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:51.666 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":190,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:00:50.657189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:00:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 190 2024-02-19T15:00:51.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9806b810 msgr2=0x7fbd9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9806b810 0x7fbd9806dcd0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005880 tx=0x7fbda8009040 comp rx=0 tx=0).stop 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 msgr2=0x7fbdb8156410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 0x7fbdb8156410 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fbda0034040 tx=0x7fbda00046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 shutdown_connections 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbd9806b810 0x7fbd9806dcd0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdb80c8e00 0x7fbdb8156410 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdb8152700 0x7fbdb8158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 --2- 172.21.15.78:0/2969944975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdb806d910 0x7fbdb814a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 >> 172.21.15.78:0/2969944975 conn(0x7fbdb80cac90 msgr2=0x7fbdb80c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 shutdown_connections 2024-02-19T15:00:51.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.665+0000 7fbdbd3e4640 1 -- 172.21.15.78:0/2969944975 wait complete. 2024-02-19T15:00:51.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:00:51.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.775+0000 7fbccdd1e640 1 Processor -- start 2024-02-19T15:00:51.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.775+0000 7fbccdd1e640 1 -- start start 2024-02-19T15:00:51.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbccdd1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbcc80c30a0 0x7fbcc8158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbccdd1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5510 0x7fbcc81559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbccdd1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc80c3cf0 con 0x7fbcc80c5130 2024-02-19T15:00:51.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbccdd1e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc8158b60 con 0x7fbcc80c30a0 2024-02-19T15:00:51.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbccdd1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80c3520 con 0x7fbcc80c5510 2024-02-19T15:00:51.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbcc6ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbcc80c30a0 0x7fbcc8158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbcc77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbcc80c5130 0x7fbcc80c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59868/0 (socket says 172.21.15.78:59868) 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbcc77fe640 1 -- 172.21.15.78:0/1179760358 learned_addr learned my addr 172.21.15.78:0/1179760358 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbcc7fff640 1 --2- 172.21.15.78:0/1179760358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5510 0x7fbcc81559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.776+0000 7fbcc67fc640 1 -- 172.21.15.78:0/1179760358 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051479436 0 0) 0x7fbcc80c3cf0 con 0x7fbcc80c5130 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc67fc640 1 -- 172.21.15.78:0/1179760358 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb0003520 con 0x7fbcc80c5130 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc6ffd640 1 -- 172.21.15.78:0/1179760358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5510 msgr2=0x7fbcc81559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc6ffd640 1 --2- 172.21.15.78:0/1179760358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5510 0x7fbcc81559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc6ffd640 1 -- 172.21.15.78:0/1179760358 >> v1:172.21.15.78:6789/0 conn(0x7fbcc80c5130 legacy=0x7fbcc80c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc6ffd640 1 -- 172.21.15.78:0/1179760358 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc80c59a0 con 0x7fbcc80c30a0 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc6ffd640 1 --2- 172.21.15.78:0/1179760358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c30a0 0x7fbcc8158f80 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0097a0 tx=0x7fbcbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f3281739cfb2a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc67fc640 1 -- 172.21.15.78:0/1179760358 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc031830 con 0x7fbcc80c30a0 2024-02-19T15:00:51.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc67fc640 1 -- 172.21.15.78:0/1179760358 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc031990 con 0x7fbcc80c30a0 2024-02-19T15:00:51.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.777+0000 7fbcc67fc640 1 -- 172.21.15.78:0/1179760358 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc031d20 con 0x7fbcc80c30a0 2024-02-19T15:00:51.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1179760358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c30a0 msgr2=0x7fbcc8158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1179760358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c30a0 0x7fbcc8158f80 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0097a0 tx=0x7fbcbc02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1179760358 shutdown_connections 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1179760358 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c30a0 0x7fbcc8158f80 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1179760358 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5510 0x7fbcc81559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1179760358 >> 172.21.15.78:0/1179760358 conn(0x7fbcc806b4b0 msgr2=0x7fbcc80c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1179760358 shutdown_connections 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.778+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1179760358 wait complete. 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.779+0000 7fbccdd1e640 1 Processor -- start 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.779+0000 7fbccdd1e640 1 -- start start 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbccdd1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc80c30a0 0x7fbcc80aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbccdd1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5130 0x7fbcc81eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbccdd1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 0x7fbcc8153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbccdd1e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80ad4c0 con 0x7fbcc80c5510 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbccdd1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80a9da0 con 0x7fbcc80c30a0 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbccdd1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80a9540 con 0x7fbcc80c5130 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 0x7fbcc8153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 0x7fbcc8153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46110/0 (socket says 172.21.15.78:46110) 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 -- 172.21.15.78:0/1854275946 learned_addr learned my addr 172.21.15.78:0/1854275946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc6ffd640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5130 0x7fbcc81eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc77fe640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc80c30a0 0x7fbcc80aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 -- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5130 msgr2=0x7fbcc81eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5130 0x7fbcc81eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 -- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc80c30a0 msgr2=0x7fbcc80aa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc80c30a0 0x7fbcc80aa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.780+0000 7fbcc7fff640 1 -- 172.21.15.78:0/1854275946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc009400 con 0x7fbcc80c5510 2024-02-19T15:00:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.781+0000 7fbcc7fff640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 0x7fbcc8153fc0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000e7f0 tx=0x7fbcc000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.781+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0009510 con 0x7fbcc80c5510 2024-02-19T15:00:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.781+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc000ee40 con 0x7fbcc80c5510 2024-02-19T15:00:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.781+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc81eb770 con 0x7fbcc80c5510 2024-02-19T15:00:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.781+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000cb80 con 0x7fbcc80c5510 2024-02-19T15:00:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.781+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc80a9ac0 con 0x7fbcc80c5510 2024-02-19T15:00:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.782+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc8074990 con 0x7fbcc80c5510 2024-02-19T15:00:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.782+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbcc000cce0 con 0x7fbcc80c5510 2024-02-19T15:00:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.783+0000 7fbcaffff640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcb406b840 0x7fbcb406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.783+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbcc008a390 con 0x7fbcc80c5510 2024-02-19T15:00:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.783+0000 7fbcc77fe640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcb406b840 0x7fbcb406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.783+0000 7fbcc77fe640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcb406b840 0x7fbcb406dd00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8005c00 tx=0x7fbcb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:51.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.787+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbcc0054cd0 con 0x7fbcc80c5510 2024-02-19T15:00:51.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.959+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcc80c9400 con 0x7fbcc80c5510 2024-02-19T15:00:51.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.960+0000 7fbcaffff640 1 -- 172.21.15.78:0/1854275946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16876 (secure 0 0 0) 0x7fbcc0058390 con 0x7fbcc80c5510 2024-02-19T15:00:51.963 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:51.963 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":213,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:00:50.301199+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:00:46.861039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-19T15:00:47.624123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcb406b840 msgr2=0x7fbcb406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcb406b840 0x7fbcb406dd00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8005c00 tx=0x7fbcb8009040 comp rx=0 tx=0).stop 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 msgr2=0x7fbcc8153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 0x7fbcc8153fc0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000e7f0 tx=0x7fbcc000ebf0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 shutdown_connections 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbcb406b840 0x7fbcb406dd00 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcc80c5510 0x7fbcc8153fc0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcc80c5130 0x7fbcc81eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 --2- 172.21.15.78:0/1854275946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcc80c30a0 0x7fbcc80aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:51.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 >> 172.21.15.78:0/1854275946 conn(0x7fbcc806b4b0 msgr2=0x7fbcc8157150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:51.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 shutdown_connections 2024-02-19T15:00:51.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:51.962+0000 7fbccdd1e640 1 -- 172.21.15.78:0/1854275946 wait complete. 2024-02-19T15:00:51.976 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-19T15:00:51.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:00:52.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.074+0000 7f48a361c640 1 Processor -- start 2024-02-19T15:00:52.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.074+0000 7f48a361c640 1 -- start start 2024-02-19T15:00:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a361c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f489c152670 0x7f489c071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a361c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f489c152a50 0x7f489c155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a361c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c071bf0 con 0x7f489c0702d0 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a361c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c158a20 con 0x7f489c152a50 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a361c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c155550 con 0x7f489c152670 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a0bb7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f489c152a50 0x7f489c155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a0bb7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f489c152a50 0x7f489c155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46122/0 (socket says 172.21.15.78:46122) 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a0bb7640 1 -- 172.21.15.78:0/1764708815 learned_addr learned my addr 172.21.15.78:0/1764708815 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.075+0000 7f48a13b8640 1 --2- 172.21.15.78:0/1764708815 >> v2:172.21.15.150:3300/0 conn(0x7f489c152670 0x7f489c071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f48a0bb7640 1 -- 172.21.15.78:0/1764708815 >> v2:172.21.15.150:3300/0 conn(0x7f489c152670 msgr2=0x7f489c071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f48a0bb7640 1 --2- 172.21.15.78:0/1764708815 >> v2:172.21.15.150:3300/0 conn(0x7f489c152670 0x7f489c071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f48a0bb7640 1 -- 172.21.15.78:0/1764708815 >> v1:172.21.15.150:6789/0 conn(0x7f489c0702d0 legacy=0x7f489c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f48a0bb7640 1 -- 172.21.15.78:0/1764708815 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0c6780 con 0x7f489c152a50 2024-02-19T15:00:52.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f48a0bb7640 1 --2- 172.21.15.78:0/1764708815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c152a50 0x7f489c155010 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f489c06e600 tx=0x7f489002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb5d6e5038ab0b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f489bfff640 1 -- 172.21.15.78:0/1764708815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489002f850 con 0x7f489c152a50 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f489bfff640 1 -- 172.21.15.78:0/1764708815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489002f9b0 con 0x7f489c152a50 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.076+0000 7f489bfff640 1 -- 172.21.15.78:0/1764708815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489002fd40 con 0x7f489c152a50 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 -- 172.21.15.78:0/1764708815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c152a50 msgr2=0x7f489c155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 --2- 172.21.15.78:0/1764708815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c152a50 0x7f489c155010 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f489c06e600 tx=0x7f489002ba30 comp rx=0 tx=0).stop 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 -- 172.21.15.78:0/1764708815 shutdown_connections 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 --2- 172.21.15.78:0/1764708815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c152a50 0x7f489c155010 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 --2- 172.21.15.78:0/1764708815 >> v2:172.21.15.150:3300/0 conn(0x7f489c152670 0x7f489c071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 -- 172.21.15.78:0/1764708815 >> 172.21.15.78:0/1764708815 conn(0x7f489c0c85e0 msgr2=0x7f489c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 -- 172.21.15.78:0/1764708815 shutdown_connections 2024-02-19T15:00:52.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.077+0000 7f48a361c640 1 -- 172.21.15.78:0/1764708815 wait complete. 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.078+0000 7f48a361c640 1 Processor -- start 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.078+0000 7f48a361c640 1 -- start start 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a361c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 0x7f489c150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a361c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f489c152670 0x7f489c153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a361c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f489c152a50 0x7f489c153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a361c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c154ae0 con 0x7f489c0702d0 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a361c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1eb310 con 0x7f489c152670 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a361c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c154d80 con 0x7f489c152a50 2024-02-19T15:00:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 0x7f489c150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 0x7f489c150ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46128/0 (socket says 172.21.15.78:46128) 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 -- 172.21.15.78:0/298324867 learned_addr learned my addr 172.21.15.78:0/298324867 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a0bb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f489c152670 0x7f489c153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a1bb9640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f489c152a50 0x7f489c153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 -- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f489c152a50 msgr2=0x7f489c153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f489c152a50 0x7f489c153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 -- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f489c152670 msgr2=0x7f489c153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f489c152670 0x7f489c153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 -- 172.21.15.78:0/298324867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900094d0 con 0x7f489c0702d0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.079+0000 7f48a13b8640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 0x7f489c150ea0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f488c00e9b0 tx=0x7f488c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.080+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c009670 con 0x7f489c0702d0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.080+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c00c9a0 con 0x7f489c0702d0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.080+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c1513e0 con 0x7f489c0702d0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.080+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c00cd50 con 0x7f489c0702d0 2024-02-19T15:00:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.080+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0ab730 con 0x7f489c0702d0 2024-02-19T15:00:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.081+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c154f40 con 0x7f489c0702d0 2024-02-19T15:00:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.081+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f488c013470 con 0x7f489c0702d0 2024-02-19T15:00:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.082+0000 7f4899ffb640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f488806b810 0x7f488806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.082+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f488c08a810 con 0x7f489c0702d0 2024-02-19T15:00:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.082+0000 7f48a0bb7640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f488806b810 0x7f488806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.082+0000 7f48a0bb7640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f488806b810 0x7f488806dcd0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4890036000 tx=0x7f4890008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.086+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f488c0551e0 con 0x7f489c0702d0 2024-02-19T15:00:52.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.255+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f489c1586b0 con 0x7f488806b810 2024-02-19T15:00:52.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.262+0000 7f4899ffb640 1 -- 172.21.15.78:0/298324867 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277981 (secure 0 0 0) 0x7f489002f650 con 0x7f488806b810 2024-02-19T15:00:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:52.269 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:00:52.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f488806b810 msgr2=0x7f488806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f488806b810 0x7f488806dcd0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4890036000 tx=0x7f4890008030 comp rx=0 tx=0).stop 2024-02-19T15:00:52.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 msgr2=0x7f489c150ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 0x7f489c150ea0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f488c00e9b0 tx=0x7f488c00ee70 comp rx=0 tx=0).stop 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 shutdown_connections 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f488806b810 0x7f488806dcd0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f489c152a50 0x7f489c153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f489c152670 0x7f489c153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 --2- 172.21.15.78:0/298324867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f489c0702d0 0x7f489c150ea0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 >> 172.21.15.78:0/298324867 conn(0x7f489c0c85e0 msgr2=0x7f489c06d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 shutdown_connections 2024-02-19T15:00:52.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.265+0000 7f48a361c640 1 -- 172.21.15.78:0/298324867 wait complete. 2024-02-19T15:00:52.279 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-02-19T15:00:52.165571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271428,"kb_used_data":21980,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754703292,"statfs":{"total":773094113280,"available":772816171008,"internally_reserved":0,"allocated":22507520,"data_stored":11557647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.865246"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302970+0000","last_change":"2024-02-19T15:00:47.302970+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302970+0000","last_undegraded":"2024-02-19T15:00:47.302970+0000","last_fullsized":"2024-02-19T15:00:47.302970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302378+0000","last_change":"2024-02-19T15:00:47.302378+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302378+0000","last_undegraded":"2024-02-19T15:00:47.302378+0000","last_fullsized":"2024-02-19T15:00:47.302378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302000+0000","last_change":"2024-02-19T15:00:47.302000+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302000+0000","last_undegraded":"2024-02-19T15:00:47.302000+0000","last_fullsized":"2024-02-19T15:00:47.302000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302352+0000","last_change":"2024-02-19T15:00:47.302352+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302352+0000","last_undegraded":"2024-02-19T15:00:47.302352+0000","last_fullsized":"2024-02-19T15:00:47.302352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298756+0000","last_change":"2024-02-19T15:00:47.298756+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298756+0000","last_undegraded":"2024-02-19T15:00:47.298756+0000","last_fullsized":"2024-02-19T15:00:47.298756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302263+0000","last_change":"2024-02-19T15:00:47.302263+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302263+0000","last_undegraded":"2024-02-19T15:00:47.302263+0000","last_fullsized":"2024-02-19T15:00:47.302263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301716+0000","last_change":"2024-02-19T15:00:47.301716+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301716+0000","last_undegraded":"2024-02-19T15:00:47.301716+0000","last_fullsized":"2024-02-19T15:00:47.301716+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298531+0000","last_change":"2024-02-19T15:00:47.298531+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298531+0000","last_undegraded":"2024-02-19T15:00:47.298531+0000","last_fullsized":"2024-02-19T15:00:47.298531+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300164+0000","last_change":"2024-02-19T15:00:47.300164+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300164+0000","last_undegraded":"2024-02-19T15:00:47.300164+0000","last_fullsized":"2024-02-19T15:00:47.300164+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301485+0000","last_change":"2024-02-19T15:00:47.301485+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301485+0000","last_undegraded":"2024-02-19T15:00:47.301485+0000","last_fullsized":"2024-02-19T15:00:47.301485+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299593+0000","last_change":"2024-02-19T15:00:47.299593+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299593+0000","last_undegraded":"2024-02-19T15:00:47.299593+0000","last_fullsized":"2024-02-19T15:00:47.299593+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303789+0000","last_change":"2024-02-19T15:00:47.303789+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303789+0000","last_undegraded":"2024-02-19T15:00:47.303789+0000","last_fullsized":"2024-02-19T15:00:47.303789+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298402+0000","last_change":"2024-02-19T15:00:47.298402+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298402+0000","last_undegraded":"2024-02-19T15:00:47.298402+0000","last_fullsized":"2024-02-19T15:00:47.298402+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299538+0000","last_change":"2024-02-19T15:00:47.299538+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299538+0000","last_undegraded":"2024-02-19T15:00:47.299538+0000","last_fullsized":"2024-02-19T15:00:47.299538+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300012+0000","last_change":"2024-02-19T15:00:47.300012+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300012+0000","last_undegraded":"2024-02-19T15:00:47.300012+0000","last_fullsized":"2024-02-19T15:00:47.300012+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299961+0000","last_change":"2024-02-19T15:00:47.299961+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299961+0000","last_undegraded":"2024-02-19T15:00:47.299961+0000","last_fullsized":"2024-02-19T15:00:47.299961+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301196+0000","last_change":"2024-02-19T15:00:47.301196+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301196+0000","last_undegraded":"2024-02-19T15:00:47.301196+0000","last_fullsized":"2024-02-19T15:00:47.301196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304210+0000","last_change":"2024-02-19T15:00:47.304210+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304210+0000","last_undegraded":"2024-02-19T15:00:47.304210+0000","last_fullsized":"2024-02-19T15:00:47.304210+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304793+0000","last_change":"2024-02-19T15:00:47.304793+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304793+0000","last_undegraded":"2024-02-19T15:00:47.304793+0000","last_fullsized":"2024-02-19T15:00:47.304793+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303248+0000","last_change":"2024-02-19T15:00:47.303248+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303248+0000","last_undegraded":"2024-02-19T15:00:47.303248+0000","last_fullsized":"2024-02-19T15:00:47.303248+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298045+0000","last_change":"2024-02-19T15:00:47.298045+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298045+0000","last_undegraded":"2024-02-19T15:00:47.298045+0000","last_fullsized":"2024-02-19T15:00:47.298045+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298032+0000","last_change":"2024-02-19T15:00:47.298032+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298032+0000","last_undegraded":"2024-02-19T15:00:47.298032+0000","last_fullsized":"2024-02-19T15:00:47.298032+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301770+0000","last_change":"2024-02-19T15:00:47.301770+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301770+0000","last_undegraded":"2024-02-19T15:00:47.301770+0000","last_fullsized":"2024-02-19T15:00:47.301770+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299522+0000","last_change":"2024-02-19T15:00:47.299522+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299522+0000","last_undegraded":"2024-02-19T15:00:47.299522+0000","last_fullsized":"2024-02-19T15:00:47.299522+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299568+0000","last_change":"2024-02-19T15:00:47.299568+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299568+0000","last_undegraded":"2024-02-19T15:00:47.299568+0000","last_fullsized":"2024-02-19T15:00:47.299568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303829+0000","last_change":"2024-02-19T15:00:47.303829+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303829+0000","last_undegraded":"2024-02-19T15:00:47.303829+0000","last_fullsized":"2024-02-19T15:00:47.303829+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304340+0000","last_change":"2024-02-19T15:00:47.304340+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304340+0000","last_undegraded":"2024-02-19T15:00:47.304340+0000","last_fullsized":"2024-02-19T15:00:47.304340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304306+0000","last_change":"2024-02-19T15:00:47.304306+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304306+0000","last_undegraded":"2024-02-19T15:00:47.304306+0000","last_fullsized":"2024-02-19T15:00:47.304306+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302782+0000","last_change":"2024-02-19T15:00:47.302782+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302782+0000","last_undegraded":"2024-02-19T15:00:47.302782+0000","last_fullsized":"2024-02-19T15:00:47.302782+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302732+0000","last_change":"2024-02-19T15:00:47.302732+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302732+0000","last_undegraded":"2024-02-19T15:00:47.302732+0000","last_fullsized":"2024-02-19T15:00:47.302732+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304345+0000","last_change":"2024-02-19T15:00:47.304345+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304345+0000","last_undegraded":"2024-02-19T15:00:47.304345+0000","last_fullsized":"2024-02-19T15:00:47.304345+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302819+0000","last_change":"2024-02-19T15:00:47.302819+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302819+0000","last_undegraded":"2024-02-19T15:00:47.302819+0000","last_fullsized":"2024-02-19T15:00:47.302819+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":523,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-02-19T15:00:47.300822+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:00:47.300822+0000","last_peered":"2024-02-19T15:00:47.300822+0000","last_clean":"2024-02-19T15:00:47.300822+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:00:47.300822+0000","last_undegraded":"2024-02-19T15:00:47.300822+0000","last_fullsized":"2024-02-19T15:00:47.300822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34364,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337476,"statfs":{"total":96636764160,"available":96601575424,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.629},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68899999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.432},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.42399999999999999,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34599999999999997,"15min":0.32400000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.90500000000000003},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.42599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.503,"15min":0.52600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.627,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47799999999999998,"15min":0.52200000000000002},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.51900000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50700000000000001,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.502,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.55800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.51900000000000002,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.86799999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.872},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.42799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.62},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71599999999999997,"5min":0.77100000000000002,"15min":0.85899999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.55300000000000005},"min":{"1min":0.34499999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.50800000000000001},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.82999999999999996,"5min":0.84399999999999997,"15min":0.877},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.497,"15min":0.55100000000000005},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.58299999999999996},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.90000000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.61299999999999999},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.03},"last":0.755}]}]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68799999999999994,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":1.341,"15min":1.341},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67500000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70299999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.442}]},{"osd":2,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":10.629,"15min":10.629},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59699999999999998,"15min":0.61599999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.53100000000000003,"15min":0.50800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.5},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77000000000000002,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.29099999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.75,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58199999999999996,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.70399999999999996,"5min":1.246,"15min":1.246},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81000000000000005,"5min":2.621,"15min":2.621},"last":0.41799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574960,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33776,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338064,"statfs":{"total":96636764160,"available":96602177536,"internally_reserved":0,"allocated":2654208,"data_stored":1291681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.77100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":0.40799999999999997,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.308,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.39900000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.247}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":0.45200000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.42599999999999999,"5min":0.66400000000000003,"15min":1.512},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.41899999999999998,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.45800000000000002,"5min":0.67600000000000005,"15min":0.76700000000000002},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.879,"15min":1.133},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.85999999999999999,"15min":1.1539999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36899999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.86699999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":1.115},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.76100000000000001,"15min":1.236},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.86099999999999999,"15min":1.1830000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.995},"last":0.66100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.32800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.496}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34368,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337472,"statfs":{"total":96636764160,"available":96601571328,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.17899999999999999},"max":{"1min":0.504,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.312},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.58099999999999996,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.317}]},{"osd":2,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.47499999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.25800000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34100000000000003},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56100000000000005,"5min":0.62,"15min":0.62},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55300000000000005,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.35999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56399999999999995,"15min":0.59199999999999997},"min":{"1min":0.40300000000000002,"5min":0.38500000000000001,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.503},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.34100000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.85099999999999998},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:00:52.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:00:52.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.417+0000 7f71f8960640 1 Processor -- start 2024-02-19T15:00:52.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.417+0000 7f71f8960640 1 -- start start 2024-02-19T15:00:52.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f8960640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f414ece0 0x7f71f40c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f8960640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71f40c15b0 0x7f71f40c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f8960640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f40c1120 con 0x7f71f40c5270 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f8960640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f40c6080 con 0x7f71f40c15b0 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f8960640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40c57e0 con 0x7f71f414ece0 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f2d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71f40c15b0 0x7f71f40c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f2d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f71f40c15b0 0x7f71f40c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46140/0 (socket says 172.21.15.78:46140) 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f2d9d640 1 -- 172.21.15.78:0/3629075915 learned_addr learned my addr 172.21.15.78:0/3629075915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.418+0000 7f71f259c640 1 --2- 172.21.15.78:0/3629075915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f414ece0 0x7f71f40c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f2d9d640 1 -- 172.21.15.78:0/3629075915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f414ece0 msgr2=0x7f71f40c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f2d9d640 1 --2- 172.21.15.78:0/3629075915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f414ece0 0x7f71f40c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f2d9d640 1 -- 172.21.15.78:0/3629075915 >> v1:172.21.15.150:6789/0 conn(0x7f71f40c5270 legacy=0x7f71f40719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f2d9d640 1 -- 172.21.15.78:0/3629075915 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40c5ac0 con 0x7f71f40c15b0 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f2d9d640 1 --2- 172.21.15.78:0/3629075915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f40c15b0 0x7f71f40c1a40 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f71ec009870 tx=0x7f71ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574d8eef6c913fe1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f159a640 1 -- 172.21.15.78:0/3629075915 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec030840 con 0x7f71f40c15b0 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f159a640 1 -- 172.21.15.78:0/3629075915 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec0309a0 con 0x7f71f40c15b0 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.419+0000 7f71f159a640 1 -- 172.21.15.78:0/3629075915 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec030d30 con 0x7f71f40c15b0 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 -- 172.21.15.78:0/3629075915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f40c15b0 msgr2=0x7f71f40c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 --2- 172.21.15.78:0/3629075915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f40c15b0 0x7f71f40c1a40 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f71ec009870 tx=0x7f71ec02b980 comp rx=0 tx=0).stop 2024-02-19T15:00:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 -- 172.21.15.78:0/3629075915 shutdown_connections 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 --2- 172.21.15.78:0/3629075915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f40c15b0 0x7f71f40c1a40 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 --2- 172.21.15.78:0/3629075915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f414ece0 0x7f71f40c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 -- 172.21.15.78:0/3629075915 >> 172.21.15.78:0/3629075915 conn(0x7f71f406b270 msgr2=0x7f71f406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 -- 172.21.15.78:0/3629075915 shutdown_connections 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.420+0000 7f71f8960640 1 -- 172.21.15.78:0/3629075915 wait complete. 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.421+0000 7f71f8960640 1 Processor -- start 2024-02-19T15:00:52.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.421+0000 7f71f8960640 1 -- start start 2024-02-19T15:00:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f8960640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f40c15b0 0x7f71f41ed320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f8960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f40c5270 0x7f71f41edb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f8960640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 0x7f71f41ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f8960640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f4146e60 con 0x7f71f414ece0 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f8960640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f41db6d0 con 0x7f71f40c5270 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f8960640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f4148680 con 0x7f71f40c15b0 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 0x7f71f41ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 0x7f71f41ee0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46156/0 (socket says 172.21.15.78:46156) 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 -- 172.21.15.78:0/1577483000 learned_addr learned my addr 172.21.15.78:0/1577483000 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f1d9b640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f40c5270 0x7f71f41edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f259c640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f40c15b0 0x7f71f41ed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 -- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f40c15b0 msgr2=0x7f71f41ed320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f40c15b0 0x7f71f41ed320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 -- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f40c5270 msgr2=0x7f71f41edb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f40c5270 0x7f71f41edb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.422+0000 7f71f2d9d640 1 -- 172.21.15.78:0/1577483000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec0094d0 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.423+0000 7f71f2d9d640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 0x7f71f41ee0d0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f71ec02b9c0 tx=0x7f71ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.423+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec039a90 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.423+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec039bf0 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.423+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec0433f0 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.423+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f4146b80 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.423+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f41eea00 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.424+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f4148880 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.424+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f71ec04c3b0 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.424+0000 7f71e2ffd640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.425+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f71ec0ac650 con 0x7f71f414ece0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.425+0000 7f71f259c640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.425+0000 7f71f259c640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f71f4149f80 tx=0x7f71e400a750 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.429+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f71f4148880 con 0x7f71f414ece0 2024-02-19T15:00:52.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.597+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71f4073210 con 0x7f71d406b810 2024-02-19T15:00:52.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.604+0000 7f71e2ffd640 1 -- 172.21.15.78:0/1577483000 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277981 (secure 0 0 0) 0x7f71e4010040 con 0x7f71d406b810 2024-02-19T15:00:52.609 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:52.610 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:00:52.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 msgr2=0x7f71d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f71f4149f80 tx=0x7f71e400a750 comp rx=0 tx=0).stop 2024-02-19T15:00:52.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 msgr2=0x7f71f41ee0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 0x7f71f41ee0d0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f71ec02b9c0 tx=0x7f71ec004730 comp rx=0 tx=0).stop 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 shutdown_connections 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f71d406b810 0x7f71d406dcd0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f414ece0 0x7f71f41ee0d0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71f40c5270 0x7f71f41edb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 --2- 172.21.15.78:0/1577483000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71f40c15b0 0x7f71f41ed320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 >> 172.21.15.78:0/1577483000 conn(0x7f71f406b270 msgr2=0x7f71f414bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 shutdown_connections 2024-02-19T15:00:52.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.607+0000 7f71f8960640 1 -- 172.21.15.78:0/1577483000 wait complete. 2024-02-19T15:00:52.620 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-02-19T15:00:52.165571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271428,"kb_used_data":21980,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754703292,"statfs":{"total":773094113280,"available":772816171008,"internally_reserved":0,"allocated":22507520,"data_stored":11557647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.865246"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302970+0000","last_change":"2024-02-19T15:00:47.302970+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302970+0000","last_undegraded":"2024-02-19T15:00:47.302970+0000","last_fullsized":"2024-02-19T15:00:47.302970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302378+0000","last_change":"2024-02-19T15:00:47.302378+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302378+0000","last_undegraded":"2024-02-19T15:00:47.302378+0000","last_fullsized":"2024-02-19T15:00:47.302378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302000+0000","last_change":"2024-02-19T15:00:47.302000+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302000+0000","last_undegraded":"2024-02-19T15:00:47.302000+0000","last_fullsized":"2024-02-19T15:00:47.302000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302352+0000","last_change":"2024-02-19T15:00:47.302352+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302352+0000","last_undegraded":"2024-02-19T15:00:47.302352+0000","last_fullsized":"2024-02-19T15:00:47.302352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298756+0000","last_change":"2024-02-19T15:00:47.298756+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298756+0000","last_undegraded":"2024-02-19T15:00:47.298756+0000","last_fullsized":"2024-02-19T15:00:47.298756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302263+0000","last_change":"2024-02-19T15:00:47.302263+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302263+0000","last_undegraded":"2024-02-19T15:00:47.302263+0000","last_fullsized":"2024-02-19T15:00:47.302263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301716+0000","last_change":"2024-02-19T15:00:47.301716+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301716+0000","last_undegraded":"2024-02-19T15:00:47.301716+0000","last_fullsized":"2024-02-19T15:00:47.301716+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298531+0000","last_change":"2024-02-19T15:00:47.298531+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298531+0000","last_undegraded":"2024-02-19T15:00:47.298531+0000","last_fullsized":"2024-02-19T15:00:47.298531+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300164+0000","last_change":"2024-02-19T15:00:47.300164+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300164+0000","last_undegraded":"2024-02-19T15:00:47.300164+0000","last_fullsized":"2024-02-19T15:00:47.300164+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301485+0000","last_change":"2024-02-19T15:00:47.301485+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301485+0000","last_undegraded":"2024-02-19T15:00:47.301485+0000","last_fullsized":"2024-02-19T15:00:47.301485+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299593+0000","last_change":"2024-02-19T15:00:47.299593+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299593+0000","last_undegraded":"2024-02-19T15:00:47.299593+0000","last_fullsized":"2024-02-19T15:00:47.299593+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303789+0000","last_change":"2024-02-19T15:00:47.303789+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303789+0000","last_undegraded":"2024-02-19T15:00:47.303789+0000","last_fullsized":"2024-02-19T15:00:47.303789+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298402+0000","last_change":"2024-02-19T15:00:47.298402+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298402+0000","last_undegraded":"2024-02-19T15:00:47.298402+0000","last_fullsized":"2024-02-19T15:00:47.298402+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299538+0000","last_change":"2024-02-19T15:00:47.299538+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299538+0000","last_undegraded":"2024-02-19T15:00:47.299538+0000","last_fullsized":"2024-02-19T15:00:47.299538+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300012+0000","last_change":"2024-02-19T15:00:47.300012+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300012+0000","last_undegraded":"2024-02-19T15:00:47.300012+0000","last_fullsized":"2024-02-19T15:00:47.300012+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299961+0000","last_change":"2024-02-19T15:00:47.299961+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299961+0000","last_undegraded":"2024-02-19T15:00:47.299961+0000","last_fullsized":"2024-02-19T15:00:47.299961+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301196+0000","last_change":"2024-02-19T15:00:47.301196+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301196+0000","last_undegraded":"2024-02-19T15:00:47.301196+0000","last_fullsized":"2024-02-19T15:00:47.301196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304210+0000","last_change":"2024-02-19T15:00:47.304210+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304210+0000","last_undegraded":"2024-02-19T15:00:47.304210+0000","last_fullsized":"2024-02-19T15:00:47.304210+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304793+0000","last_change":"2024-02-19T15:00:47.304793+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304793+0000","last_undegraded":"2024-02-19T15:00:47.304793+0000","last_fullsized":"2024-02-19T15:00:47.304793+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303248+0000","last_change":"2024-02-19T15:00:47.303248+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303248+0000","last_undegraded":"2024-02-19T15:00:47.303248+0000","last_fullsized":"2024-02-19T15:00:47.303248+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298045+0000","last_change":"2024-02-19T15:00:47.298045+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298045+0000","last_undegraded":"2024-02-19T15:00:47.298045+0000","last_fullsized":"2024-02-19T15:00:47.298045+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298032+0000","last_change":"2024-02-19T15:00:47.298032+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298032+0000","last_undegraded":"2024-02-19T15:00:47.298032+0000","last_fullsized":"2024-02-19T15:00:47.298032+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301770+0000","last_change":"2024-02-19T15:00:47.301770+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301770+0000","last_undegraded":"2024-02-19T15:00:47.301770+0000","last_fullsized":"2024-02-19T15:00:47.301770+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299522+0000","last_change":"2024-02-19T15:00:47.299522+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299522+0000","last_undegraded":"2024-02-19T15:00:47.299522+0000","last_fullsized":"2024-02-19T15:00:47.299522+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299568+0000","last_change":"2024-02-19T15:00:47.299568+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299568+0000","last_undegraded":"2024-02-19T15:00:47.299568+0000","last_fullsized":"2024-02-19T15:00:47.299568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303829+0000","last_change":"2024-02-19T15:00:47.303829+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303829+0000","last_undegraded":"2024-02-19T15:00:47.303829+0000","last_fullsized":"2024-02-19T15:00:47.303829+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304340+0000","last_change":"2024-02-19T15:00:47.304340+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304340+0000","last_undegraded":"2024-02-19T15:00:47.304340+0000","last_fullsized":"2024-02-19T15:00:47.304340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304306+0000","last_change":"2024-02-19T15:00:47.304306+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304306+0000","last_undegraded":"2024-02-19T15:00:47.304306+0000","last_fullsized":"2024-02-19T15:00:47.304306+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302782+0000","last_change":"2024-02-19T15:00:47.302782+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302782+0000","last_undegraded":"2024-02-19T15:00:47.302782+0000","last_fullsized":"2024-02-19T15:00:47.302782+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302732+0000","last_change":"2024-02-19T15:00:47.302732+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302732+0000","last_undegraded":"2024-02-19T15:00:47.302732+0000","last_fullsized":"2024-02-19T15:00:47.302732+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304345+0000","last_change":"2024-02-19T15:00:47.304345+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304345+0000","last_undegraded":"2024-02-19T15:00:47.304345+0000","last_fullsized":"2024-02-19T15:00:47.304345+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302819+0000","last_change":"2024-02-19T15:00:47.302819+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302819+0000","last_undegraded":"2024-02-19T15:00:47.302819+0000","last_fullsized":"2024-02-19T15:00:47.302819+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":523,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-02-19T15:00:47.300822+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:00:47.300822+0000","last_peered":"2024-02-19T15:00:47.300822+0000","last_clean":"2024-02-19T15:00:47.300822+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:00:47.300822+0000","last_undegraded":"2024-02-19T15:00:47.300822+0000","last_fullsized":"2024-02-19T15:00:47.300822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34364,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337476,"statfs":{"total":96636764160,"available":96601575424,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.629},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68899999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.432},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.42399999999999999,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34599999999999997,"15min":0.32400000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.90500000000000003},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.42599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.503,"15min":0.52600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.627,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47799999999999998,"15min":0.52200000000000002},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.51900000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50700000000000001,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.502,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.55800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.51900000000000002,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.86799999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.872},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.42799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.62},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71599999999999997,"5min":0.77100000000000002,"15min":0.85899999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.55300000000000005},"min":{"1min":0.34499999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.50800000000000001},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.82999999999999996,"5min":0.84399999999999997,"15min":0.877},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.497,"15min":0.55100000000000005},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.58299999999999996},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.90000000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.61299999999999999},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.03},"last":0.755}]}]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68799999999999994,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":1.341,"15min":1.341},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67500000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70299999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.442}]},{"osd":2,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":10.629,"15min":10.629},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59699999999999998,"15min":0.61599999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.53100000000000003,"15min":0.50800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.5},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77000000000000002,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.29099999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.75,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58199999999999996,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.70399999999999996,"5min":1.246,"15min":1.246},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81000000000000005,"5min":2.621,"15min":2.621},"last":0.41799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574960,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33776,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338064,"statfs":{"total":96636764160,"available":96602177536,"internally_reserved":0,"allocated":2654208,"data_stored":1291681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.77100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":0.40799999999999997,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.308,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.39900000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.247}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":0.45200000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.42599999999999999,"5min":0.66400000000000003,"15min":1.512},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.41899999999999998,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.45800000000000002,"5min":0.67600000000000005,"15min":0.76700000000000002},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.879,"15min":1.133},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.85999999999999999,"15min":1.1539999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36899999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.86699999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":1.115},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.76100000000000001,"15min":1.236},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.86099999999999999,"15min":1.1830000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.995},"last":0.66100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.32800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.496}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34368,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337472,"statfs":{"total":96636764160,"available":96601571328,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.17899999999999999},"max":{"1min":0.504,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.312},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.58099999999999996,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.317}]},{"osd":2,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.47499999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.25800000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34100000000000003},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56100000000000005,"5min":0.62,"15min":0.62},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55300000000000005,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.35999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56399999999999995,"15min":0.59199999999999997},"min":{"1min":0.40300000000000002,"5min":0.38500000000000001,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.503},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.34100000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.85099999999999998},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:00:52.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:00:52.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 Processor -- start 2024-02-19T15:00:52.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 -- start start 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56400c87a0 0x7f56400c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5640155ef0 con 0x7f56400c7070 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564006dec0 con 0x7f56400c87a0 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.758+0000 7f5646c47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564006d520 con 0x7f5640152480 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f563ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f56400c7070 0x7f564006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59874/0 (socket says 172.21.15.78:59874) 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f563ffff640 1 -- 172.21.15.78:0/2154019331 learned_addr learned my addr 172.21.15.78:0/2154019331 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56451e4640 1 --2- 172.21.15.78:0/2154019331 >> v2:172.21.15.150:3300/0 conn(0x7f56400c87a0 0x7f56400c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56449e3640 1 --2- 172.21.15.78:0/2154019331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56449e3640 1 -- 172.21.15.78:0/2154019331 >> v2:172.21.15.150:3300/0 conn(0x7f56400c87a0 msgr2=0x7f56400c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56449e3640 1 --2- 172.21.15.78:0/2154019331 >> v2:172.21.15.150:3300/0 conn(0x7f56400c87a0 0x7f56400c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56449e3640 1 -- 172.21.15.78:0/2154019331 >> v1:172.21.15.78:6789/0 conn(0x7f56400c7070 legacy=0x7f564006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f563f7fe640 1 -- 172.21.15.78:0/2154019331 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234888931 0 0) 0x7f562c001110 con 0x7f56400c7070 2024-02-19T15:00:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56449e3640 1 -- 172.21.15.78:0/2154019331 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400c75c0 con 0x7f5640152480 2024-02-19T15:00:52.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.759+0000 7f56451e4640 1 --2- 172.21.15.78:0/2154019331 >> v2:172.21.15.150:3300/0 conn(0x7f56400c87a0 0x7f56400c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f56449e3640 1 --2- 172.21.15.78:0/2154019331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401559b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56340097a0 tx=0x7f563402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2bd6702f67a0a95 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f563f7fe640 1 -- 172.21.15.78:0/2154019331 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634031830 con 0x7f5640152480 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f563f7fe640 1 -- 172.21.15.78:0/2154019331 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5634031990 con 0x7f5640152480 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f563f7fe640 1 -- 172.21.15.78:0/2154019331 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634031d20 con 0x7f5640152480 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f5646c47640 1 -- 172.21.15.78:0/2154019331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 msgr2=0x7f56401559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f5646c47640 1 --2- 172.21.15.78:0/2154019331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401559b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56340097a0 tx=0x7f563402b8d0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f5646c47640 1 -- 172.21.15.78:0/2154019331 shutdown_connections 2024-02-19T15:00:52.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f5646c47640 1 --2- 172.21.15.78:0/2154019331 >> v2:172.21.15.150:3300/0 conn(0x7f56400c87a0 0x7f56400c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f5646c47640 1 --2- 172.21.15.78:0/2154019331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401559b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.760+0000 7f5646c47640 1 -- 172.21.15.78:0/2154019331 >> 172.21.15.78:0/2154019331 conn(0x7f56400ca8a0 msgr2=0x7f56400cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:52.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.761+0000 7f5646c47640 1 -- 172.21.15.78:0/2154019331 shutdown_connections 2024-02-19T15:00:52.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.761+0000 7f5646c47640 1 -- 172.21.15.78:0/2154019331 wait complete. 2024-02-19T15:00:52.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 Processor -- start 2024-02-19T15:00:52.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 -- start start 2024-02-19T15:00:52.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 0x7f5640149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56400c87a0 0x7f5640149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401eba10 con 0x7f56400c7070 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401eea50 con 0x7f56400c87a0 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.762+0000 7f5646c47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400acd50 con 0x7f5640152480 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 0x7f5640149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 0x7f5640149650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46168/0 (socket says 172.21.15.78:46168) 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 -- 172.21.15.78:0/2321328881 learned_addr learned my addr 172.21.15.78:0/2321328881 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f563ffff640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56400c87a0 0x7f5640149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56451e4640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 -- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 msgr2=0x7f56401edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401edb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 -- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56400c87a0 msgr2=0x7f5640149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56400c87a0 0x7f5640149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 -- 172.21.15.78:0/2321328881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638009a70 con 0x7f56400c7070 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f56449e3640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 0x7f5640149650 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f56340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634004230 con 0x7f56400c7070 2024-02-19T15:00:52.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5634004390 con 0x7f56400c7070 2024-02-19T15:00:52.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563403a8a0 con 0x7f56400c7070 2024-02-19T15:00:52.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634009400 con 0x7f56400c7070 2024-02-19T15:00:52.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.763+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5640158e40 con 0x7f56400c7070 2024-02-19T15:00:52.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.764+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640074990 con 0x7f56400c7070 2024-02-19T15:00:52.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.765+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f563403aa00 con 0x7f56400c7070 2024-02-19T15:00:52.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.765+0000 7f563d7fa640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f562406b810 0x7f562406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:52.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.765+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f56340ac6d0 con 0x7f56400c7070 2024-02-19T15:00:52.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.765+0000 7f563ffff640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f562406b810 0x7f562406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:52.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.766+0000 7f563ffff640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f562406b810 0x7f562406dcd0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f56400c9110 tx=0x7f562c007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.770+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56340770a0 con 0x7f56400c7070 2024-02-19T15:00:52.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.938+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56400c7b80 con 0x7f562406b810 2024-02-19T15:00:52.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.944+0000 7f563d7fa640 1 -- 172.21.15.78:0/2321328881 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277981 (secure 0 0 0) 0x7f562c004c70 con 0x7f562406b810 2024-02-19T15:00:52.950 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:52.951 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:00:52.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f562406b810 msgr2=0x7f562406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f562406b810 0x7f562406dcd0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f56400c9110 tx=0x7f562c007e40 comp rx=0 tx=0).stop 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 msgr2=0x7f5640149650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 0x7f5640149650 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f5634000c00 tx=0x7f56340046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 shutdown_connections 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f562406b810 0x7f562406dcd0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5640152480 0x7f56401edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56400c87a0 0x7f5640149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 --2- 172.21.15.78:0/2321328881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56400c7070 0x7f5640149650 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 >> 172.21.15.78:0/2321328881 conn(0x7f56400ca8a0 msgr2=0x7f56401513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.947+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 shutdown_connections 2024-02-19T15:00:52.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:52.948+0000 7f5646c47640 1 -- 172.21.15.78:0/2321328881 wait complete. 2024-02-19T15:00:52.961 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-02-19T15:00:52.165571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271428,"kb_used_data":21980,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754703292,"statfs":{"total":773094113280,"available":772816171008,"internally_reserved":0,"allocated":22507520,"data_stored":11557647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.865246"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302970+0000","last_change":"2024-02-19T15:00:47.302970+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302970+0000","last_undegraded":"2024-02-19T15:00:47.302970+0000","last_fullsized":"2024-02-19T15:00:47.302970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302378+0000","last_change":"2024-02-19T15:00:47.302378+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302378+0000","last_undegraded":"2024-02-19T15:00:47.302378+0000","last_fullsized":"2024-02-19T15:00:47.302378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302000+0000","last_change":"2024-02-19T15:00:47.302000+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302000+0000","last_undegraded":"2024-02-19T15:00:47.302000+0000","last_fullsized":"2024-02-19T15:00:47.302000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302352+0000","last_change":"2024-02-19T15:00:47.302352+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302352+0000","last_undegraded":"2024-02-19T15:00:47.302352+0000","last_fullsized":"2024-02-19T15:00:47.302352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298756+0000","last_change":"2024-02-19T15:00:47.298756+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298756+0000","last_undegraded":"2024-02-19T15:00:47.298756+0000","last_fullsized":"2024-02-19T15:00:47.298756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302263+0000","last_change":"2024-02-19T15:00:47.302263+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302263+0000","last_undegraded":"2024-02-19T15:00:47.302263+0000","last_fullsized":"2024-02-19T15:00:47.302263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301716+0000","last_change":"2024-02-19T15:00:47.301716+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301716+0000","last_undegraded":"2024-02-19T15:00:47.301716+0000","last_fullsized":"2024-02-19T15:00:47.301716+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298531+0000","last_change":"2024-02-19T15:00:47.298531+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298531+0000","last_undegraded":"2024-02-19T15:00:47.298531+0000","last_fullsized":"2024-02-19T15:00:47.298531+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300164+0000","last_change":"2024-02-19T15:00:47.300164+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300164+0000","last_undegraded":"2024-02-19T15:00:47.300164+0000","last_fullsized":"2024-02-19T15:00:47.300164+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301485+0000","last_change":"2024-02-19T15:00:47.301485+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301485+0000","last_undegraded":"2024-02-19T15:00:47.301485+0000","last_fullsized":"2024-02-19T15:00:47.301485+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299593+0000","last_change":"2024-02-19T15:00:47.299593+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299593+0000","last_undegraded":"2024-02-19T15:00:47.299593+0000","last_fullsized":"2024-02-19T15:00:47.299593+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303789+0000","last_change":"2024-02-19T15:00:47.303789+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303789+0000","last_undegraded":"2024-02-19T15:00:47.303789+0000","last_fullsized":"2024-02-19T15:00:47.303789+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298402+0000","last_change":"2024-02-19T15:00:47.298402+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298402+0000","last_undegraded":"2024-02-19T15:00:47.298402+0000","last_fullsized":"2024-02-19T15:00:47.298402+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299538+0000","last_change":"2024-02-19T15:00:47.299538+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299538+0000","last_undegraded":"2024-02-19T15:00:47.299538+0000","last_fullsized":"2024-02-19T15:00:47.299538+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300012+0000","last_change":"2024-02-19T15:00:47.300012+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300012+0000","last_undegraded":"2024-02-19T15:00:47.300012+0000","last_fullsized":"2024-02-19T15:00:47.300012+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299961+0000","last_change":"2024-02-19T15:00:47.299961+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299961+0000","last_undegraded":"2024-02-19T15:00:47.299961+0000","last_fullsized":"2024-02-19T15:00:47.299961+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301196+0000","last_change":"2024-02-19T15:00:47.301196+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301196+0000","last_undegraded":"2024-02-19T15:00:47.301196+0000","last_fullsized":"2024-02-19T15:00:47.301196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304210+0000","last_change":"2024-02-19T15:00:47.304210+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304210+0000","last_undegraded":"2024-02-19T15:00:47.304210+0000","last_fullsized":"2024-02-19T15:00:47.304210+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304793+0000","last_change":"2024-02-19T15:00:47.304793+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304793+0000","last_undegraded":"2024-02-19T15:00:47.304793+0000","last_fullsized":"2024-02-19T15:00:47.304793+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303248+0000","last_change":"2024-02-19T15:00:47.303248+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303248+0000","last_undegraded":"2024-02-19T15:00:47.303248+0000","last_fullsized":"2024-02-19T15:00:47.303248+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298045+0000","last_change":"2024-02-19T15:00:47.298045+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298045+0000","last_undegraded":"2024-02-19T15:00:47.298045+0000","last_fullsized":"2024-02-19T15:00:47.298045+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298032+0000","last_change":"2024-02-19T15:00:47.298032+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298032+0000","last_undegraded":"2024-02-19T15:00:47.298032+0000","last_fullsized":"2024-02-19T15:00:47.298032+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301770+0000","last_change":"2024-02-19T15:00:47.301770+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301770+0000","last_undegraded":"2024-02-19T15:00:47.301770+0000","last_fullsized":"2024-02-19T15:00:47.301770+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299522+0000","last_change":"2024-02-19T15:00:47.299522+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299522+0000","last_undegraded":"2024-02-19T15:00:47.299522+0000","last_fullsized":"2024-02-19T15:00:47.299522+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299568+0000","last_change":"2024-02-19T15:00:47.299568+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299568+0000","last_undegraded":"2024-02-19T15:00:47.299568+0000","last_fullsized":"2024-02-19T15:00:47.299568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303829+0000","last_change":"2024-02-19T15:00:47.303829+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303829+0000","last_undegraded":"2024-02-19T15:00:47.303829+0000","last_fullsized":"2024-02-19T15:00:47.303829+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304340+0000","last_change":"2024-02-19T15:00:47.304340+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304340+0000","last_undegraded":"2024-02-19T15:00:47.304340+0000","last_fullsized":"2024-02-19T15:00:47.304340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304306+0000","last_change":"2024-02-19T15:00:47.304306+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304306+0000","last_undegraded":"2024-02-19T15:00:47.304306+0000","last_fullsized":"2024-02-19T15:00:47.304306+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302782+0000","last_change":"2024-02-19T15:00:47.302782+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302782+0000","last_undegraded":"2024-02-19T15:00:47.302782+0000","last_fullsized":"2024-02-19T15:00:47.302782+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302732+0000","last_change":"2024-02-19T15:00:47.302732+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302732+0000","last_undegraded":"2024-02-19T15:00:47.302732+0000","last_fullsized":"2024-02-19T15:00:47.302732+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304345+0000","last_change":"2024-02-19T15:00:47.304345+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304345+0000","last_undegraded":"2024-02-19T15:00:47.304345+0000","last_fullsized":"2024-02-19T15:00:47.304345+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302819+0000","last_change":"2024-02-19T15:00:47.302819+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302819+0000","last_undegraded":"2024-02-19T15:00:47.302819+0000","last_fullsized":"2024-02-19T15:00:47.302819+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":523,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-02-19T15:00:47.300822+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:00:47.300822+0000","last_peered":"2024-02-19T15:00:47.300822+0000","last_clean":"2024-02-19T15:00:47.300822+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:00:47.300822+0000","last_undegraded":"2024-02-19T15:00:47.300822+0000","last_fullsized":"2024-02-19T15:00:47.300822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34364,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337476,"statfs":{"total":96636764160,"available":96601575424,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.629},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68899999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.432},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.42399999999999999,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34599999999999997,"15min":0.32400000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.90500000000000003},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.42599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.503,"15min":0.52600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.627,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47799999999999998,"15min":0.52200000000000002},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.51900000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50700000000000001,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.502,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.55800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.51900000000000002,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.86799999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.872},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.42799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.62},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71599999999999997,"5min":0.77100000000000002,"15min":0.85899999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.55300000000000005},"min":{"1min":0.34499999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.50800000000000001},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.82999999999999996,"5min":0.84399999999999997,"15min":0.877},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.497,"15min":0.55100000000000005},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.58299999999999996},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.90000000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.61299999999999999},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.03},"last":0.755}]}]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68799999999999994,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":1.341,"15min":1.341},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67500000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70299999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.442}]},{"osd":2,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":10.629,"15min":10.629},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59699999999999998,"15min":0.61599999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.53100000000000003,"15min":0.50800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.5},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77000000000000002,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.29099999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.75,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58199999999999996,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.70399999999999996,"5min":1.246,"15min":1.246},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81000000000000005,"5min":2.621,"15min":2.621},"last":0.41799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574960,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33776,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338064,"statfs":{"total":96636764160,"available":96602177536,"internally_reserved":0,"allocated":2654208,"data_stored":1291681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.77100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":0.40799999999999997,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.308,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.39900000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.247}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":0.45200000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.42599999999999999,"5min":0.66400000000000003,"15min":1.512},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.41899999999999998,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.45800000000000002,"5min":0.67600000000000005,"15min":0.76700000000000002},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.879,"15min":1.133},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.85999999999999999,"15min":1.1539999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36899999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.86699999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":1.115},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.76100000000000001,"15min":1.236},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.86099999999999999,"15min":1.1830000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.995},"last":0.66100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.32800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.496}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34368,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337472,"statfs":{"total":96636764160,"available":96601571328,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.17899999999999999},"max":{"1min":0.504,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.312},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.58099999999999996,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.317}]},{"osd":2,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.47499999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.25800000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34100000000000003},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56100000000000005,"5min":0.62,"15min":0.62},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55300000000000005,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.35999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56399999999999995,"15min":0.59199999999999997},"min":{"1min":0.40300000000000002,"5min":0.38500000000000001,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.503},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.34100000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.85099999999999998},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3f is not active+clean 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3e is not active+clean 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3d is not active+clean 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3c is not active+clean 2024-02-19T15:00:52.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3b is not active+clean 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3a is not active+clean 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.39 is not active+clean 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.38 is not active+clean 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.37 is not active+clean 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.36 is not active+clean 2024-02-19T15:00:52.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.35 is not active+clean 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.34 is not active+clean 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.33 is not active+clean 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.32 is not active+clean 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.31 is not active+clean 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.30 is not active+clean 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2f is not active+clean 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2e is not active+clean 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2d is not active+clean 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2c is not active+clean 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2b is not active+clean 2024-02-19T15:00:52.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2a is not active+clean 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.29 is not active+clean 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.28 is not active+clean 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.27 is not active+clean 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.26 is not active+clean 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.25 is not active+clean 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.24 is not active+clean 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.22 is not active+clean 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.21 is not active+clean 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.20 is not active+clean 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1c is not active+clean 2024-02-19T15:00:52.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1f is not active+clean 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302970+0000', 'last_change': '2024-02-19T15:00:47.302970+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302970+0000', 'last_undegraded': '2024-02-19T15:00:47.302970+0000', 'last_fullsized': '2024-02-19T15:00:47.302970+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1d is not active+clean 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1e is not active+clean 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302378+0000', 'last_change': '2024-02-19T15:00:47.302378+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302378+0000', 'last_undegraded': '2024-02-19T15:00:47.302378+0000', 'last_fullsized': '2024-02-19T15:00:47.302378+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1e is not active+clean 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1d is not active+clean 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302000+0000', 'last_change': '2024-02-19T15:00:47.302000+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302000+0000', 'last_undegraded': '2024-02-19T15:00:47.302000+0000', 'last_fullsized': '2024-02-19T15:00:47.302000+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1f is not active+clean 2024-02-19T15:00:52.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1c is not active+clean 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302352+0000', 'last_change': '2024-02-19T15:00:47.302352+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302352+0000', 'last_undegraded': '2024-02-19T15:00:47.302352+0000', 'last_fullsized': '2024-02-19T15:00:47.302352+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.18 is not active+clean 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1b is not active+clean 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.298756+0000', 'last_change': '2024-02-19T15:00:47.298756+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.298756+0000', 'last_undegraded': '2024-02-19T15:00:47.298756+0000', 'last_fullsized': '2024-02-19T15:00:47.298756+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.19 is not active+clean 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1a is not active+clean 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302263+0000', 'last_change': '2024-02-19T15:00:47.302263+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302263+0000', 'last_undegraded': '2024-02-19T15:00:47.302263+0000', 'last_fullsized': '2024-02-19T15:00:47.302263+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1a is not active+clean 2024-02-19T15:00:52.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.19 is not active+clean 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.301716+0000', 'last_change': '2024-02-19T15:00:47.301716+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.301716+0000', 'last_undegraded': '2024-02-19T15:00:47.301716+0000', 'last_fullsized': '2024-02-19T15:00:47.301716+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1b is not active+clean 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.18 is not active+clean 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.298531+0000', 'last_change': '2024-02-19T15:00:47.298531+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.298531+0000', 'last_undegraded': '2024-02-19T15:00:47.298531+0000', 'last_fullsized': '2024-02-19T15:00:47.298531+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.14 is not active+clean 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.17 is not active+clean 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.300164+0000', 'last_change': '2024-02-19T15:00:47.300164+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.300164+0000', 'last_undegraded': '2024-02-19T15:00:47.300164+0000', 'last_fullsized': '2024-02-19T15:00:47.300164+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.15 is not active+clean 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.16 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.301485+0000', 'last_change': '2024-02-19T15:00:47.301485+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.301485+0000', 'last_undegraded': '2024-02-19T15:00:47.301485+0000', 'last_fullsized': '2024-02-19T15:00:47.301485+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.16 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.15 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.299593+0000', 'last_change': '2024-02-19T15:00:47.299593+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.299593+0000', 'last_undegraded': '2024-02-19T15:00:47.299593+0000', 'last_fullsized': '2024-02-19T15:00:47.299593+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.17 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.14 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.303789+0000', 'last_change': '2024-02-19T15:00:47.303789+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.303789+0000', 'last_undegraded': '2024-02-19T15:00:47.303789+0000', 'last_fullsized': '2024-02-19T15:00:47.303789+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.10 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.13 is not active+clean 2024-02-19T15:00:52.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.298402+0000', 'last_change': '2024-02-19T15:00:47.298402+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.298402+0000', 'last_undegraded': '2024-02-19T15:00:47.298402+0000', 'last_fullsized': '2024-02-19T15:00:47.298402+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.11 is not active+clean 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.12 is not active+clean 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.299538+0000', 'last_change': '2024-02-19T15:00:47.299538+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.299538+0000', 'last_undegraded': '2024-02-19T15:00:47.299538+0000', 'last_fullsized': '2024-02-19T15:00:47.299538+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.12 is not active+clean 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.11 is not active+clean 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.300012+0000', 'last_change': '2024-02-19T15:00:47.300012+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.300012+0000', 'last_undegraded': '2024-02-19T15:00:47.300012+0000', 'last_fullsized': '2024-02-19T15:00:47.300012+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.13 is not active+clean 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.10 is not active+clean 2024-02-19T15:00:52.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.299961+0000', 'last_change': '2024-02-19T15:00:47.299961+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.299961+0000', 'last_undegraded': '2024-02-19T15:00:47.299961+0000', 'last_fullsized': '2024-02-19T15:00:47.299961+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.c is not active+clean 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.f is not active+clean 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.301196+0000', 'last_change': '2024-02-19T15:00:47.301196+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.301196+0000', 'last_undegraded': '2024-02-19T15:00:47.301196+0000', 'last_fullsized': '2024-02-19T15:00:47.301196+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.d is not active+clean 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.e is not active+clean 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.304210+0000', 'last_change': '2024-02-19T15:00:47.304210+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.304210+0000', 'last_undegraded': '2024-02-19T15:00:47.304210+0000', 'last_fullsized': '2024-02-19T15:00:47.304210+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.e is not active+clean 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.d is not active+clean 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.304793+0000', 'last_change': '2024-02-19T15:00:47.304793+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.304793+0000', 'last_undegraded': '2024-02-19T15:00:47.304793+0000', 'last_fullsized': '2024-02-19T15:00:47.304793+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:00:52.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.f is not active+clean 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.c is not active+clean 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.303248+0000', 'last_change': '2024-02-19T15:00:47.303248+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.303248+0000', 'last_undegraded': '2024-02-19T15:00:47.303248+0000', 'last_fullsized': '2024-02-19T15:00:47.303248+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.8 is not active+clean 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.b is not active+clean 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.298045+0000', 'last_change': '2024-02-19T15:00:47.298045+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.298045+0000', 'last_undegraded': '2024-02-19T15:00:47.298045+0000', 'last_fullsized': '2024-02-19T15:00:47.298045+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.9 is not active+clean 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.a is not active+clean 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.298032+0000', 'last_change': '2024-02-19T15:00:47.298032+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.298032+0000', 'last_undegraded': '2024-02-19T15:00:47.298032+0000', 'last_fullsized': '2024-02-19T15:00:47.298032+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-19T15:00:52.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.a is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.9 is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.301770+0000', 'last_change': '2024-02-19T15:00:47.301770+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.301770+0000', 'last_undegraded': '2024-02-19T15:00:47.301770+0000', 'last_fullsized': '2024-02-19T15:00:47.301770+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.b is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.8 is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.299522+0000', 'last_change': '2024-02-19T15:00:47.299522+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.299522+0000', 'last_undegraded': '2024-02-19T15:00:47.299522+0000', 'last_fullsized': '2024-02-19T15:00:47.299522+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.4 is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.7 is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.299568+0000', 'last_change': '2024-02-19T15:00:47.299568+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.299568+0000', 'last_undegraded': '2024-02-19T15:00:47.299568+0000', 'last_fullsized': '2024-02-19T15:00:47.299568+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.5 is not active+clean 2024-02-19T15:00:52.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.6 is not active+clean 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.303829+0000', 'last_change': '2024-02-19T15:00:47.303829+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.303829+0000', 'last_undegraded': '2024-02-19T15:00:47.303829+0000', 'last_fullsized': '2024-02-19T15:00:47.303829+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.6 is not active+clean 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.5 is not active+clean 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.304340+0000', 'last_change': '2024-02-19T15:00:47.304340+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.304340+0000', 'last_undegraded': '2024-02-19T15:00:47.304340+0000', 'last_fullsized': '2024-02-19T15:00:47.304340+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.7 is not active+clean 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.4 is not active+clean 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.304306+0000', 'last_change': '2024-02-19T15:00:47.304306+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.304306+0000', 'last_undegraded': '2024-02-19T15:00:47.304306+0000', 'last_fullsized': '2024-02-19T15:00:47.304306+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:00:52.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.0 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.3 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302782+0000', 'last_change': '2024-02-19T15:00:47.302782+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302782+0000', 'last_undegraded': '2024-02-19T15:00:47.302782+0000', 'last_fullsized': '2024-02-19T15:00:47.302782+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.2 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302732+0000', 'last_change': '2024-02-19T15:00:47.302732+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302732+0000', 'last_undegraded': '2024-02-19T15:00:47.302732+0000', 'last_fullsized': '2024-02-19T15:00:47.302732+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.1 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.304345+0000', 'last_change': '2024-02-19T15:00:47.304345+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.304345+0000', 'last_undegraded': '2024-02-19T15:00:47.304345+0000', 'last_fullsized': '2024-02-19T15:00:47.304345+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3 is not active+clean 2024-02-19T15:00:52.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.0 is not active+clean 2024-02-19T15:00:52.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 210, 'state': 'creating+peering', 'last_fresh': '2024-02-19T15:00:47.302819+0000', 'last_change': '2024-02-19T15:00:47.302819+0000', 'last_active': '2024-02-19T15:00:47.287509+0000', 'last_peered': '2024-02-19T15:00:47.287509+0000', 'last_clean': '2024-02-19T15:00:47.287509+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:47.302819+0000', 'last_undegraded': '2024-02-19T15:00:47.302819+0000', 'last_fullsized': '2024-02-19T15:00:47.302819+0000', 'mapping_epoch': 210, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 210, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:47.287509+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-19T15:00:52.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.23 is not active+clean 2024-02-19T15:00:52.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-02-19T15:00:48.294950+0000', 'last_change': '2024-02-19T15:00:48.294950+0000', 'last_active': '2024-02-19T15:00:48.294950+0000', 'last_peered': '2024-02-19T15:00:48.294950+0000', 'last_clean': '2024-02-19T15:00:48.294950+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-19T15:00:48.294950+0000', 'last_undegraded': '2024-02-19T15:00:48.294950+0000', 'last_fullsized': '2024-02-19T15:00:48.294950+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'last_clean_scrub_stamp': '2024-02-19T15:00:48.294950+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-02-19T15:00:52.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-19T15:00:53.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 Processor -- start 2024-02-19T15:00:53.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 -- start start 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f30e00add70 0x7f30e00c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00c2f60 0x7f30e006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00ae670 con 0x7f30e006f530 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e00716b0 con 0x7f30e00add70 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.098+0000 7f30e62c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00c33a0 con 0x7f30e00c2f60 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30dffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f30e00add70 0x7f30e00c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30e4864640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f30e006f530 0x7f30e00705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59878/0 (socket says 172.21.15.78:59878) 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30e4864640 1 -- 172.21.15.78:0/1719526354 learned_addr learned my addr 172.21.15.78:0/1719526354 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30df7fe640 1 --2- 172.21.15.78:0/1719526354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00c2f60 0x7f30e006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30dffff640 1 -- 172.21.15.78:0/1719526354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00c2f60 msgr2=0x7f30e006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30dffff640 1 --2- 172.21.15.78:0/1719526354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00c2f60 0x7f30e006eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30dffff640 1 -- 172.21.15.78:0/1719526354 >> v1:172.21.15.78:6789/0 conn(0x7f30e006f530 legacy=0x7f30e00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30deffd640 1 -- 172.21.15.78:0/1719526354 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807568418 0 0) 0x7f30e00ae670 con 0x7f30e006f530 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.099+0000 7f30dffff640 1 -- 172.21.15.78:0/1719526354 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e0153780 con 0x7f30e00add70 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30dffff640 1 --2- 172.21.15.78:0/1719526354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00add70 0x7f30e00c4980 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f30d40097a0 tx=0x7f30d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933c303f5c18d36d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30deffd640 1 -- 172.21.15.78:0/1719526354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402e070 con 0x7f30e00add70 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30deffd640 1 -- 172.21.15.78:0/1719526354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d40315a0 con 0x7f30e00add70 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30deffd640 1 -- 172.21.15.78:0/1719526354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4031930 con 0x7f30e00add70 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 -- 172.21.15.78:0/1719526354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00add70 msgr2=0x7f30e00c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 --2- 172.21.15.78:0/1719526354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00add70 0x7f30e00c4980 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f30d40097a0 tx=0x7f30d402b840 comp rx=0 tx=0).stop 2024-02-19T15:00:53.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 -- 172.21.15.78:0/1719526354 shutdown_connections 2024-02-19T15:00:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 --2- 172.21.15.78:0/1719526354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00c2f60 0x7f30e006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 --2- 172.21.15.78:0/1719526354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00add70 0x7f30e00c4980 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 -- 172.21.15.78:0/1719526354 >> 172.21.15.78:0/1719526354 conn(0x7f30e00c85e0 msgr2=0x7f30e00c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 -- 172.21.15.78:0/1719526354 shutdown_connections 2024-02-19T15:00:53.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.100+0000 7f30e62c7640 1 -- 172.21.15.78:0/1719526354 wait complete. 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.101+0000 7f30e62c7640 1 Processor -- start 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 -- start start 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006f530 0x7f30e006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00add70 0x7f30e01ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 0x7f30e00c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e009a8d0 con 0x7f30e00c2f60 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00c68e0 con 0x7f30e006f530 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e62c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00c6e10 con 0x7f30e00add70 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e4864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 0x7f30e00c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e4864640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 0x7f30e00c1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46174/0 (socket says 172.21.15.78:46174) 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30e4864640 1 -- 172.21.15.78:0/436774186 learned_addr learned my addr 172.21.15.78:0/436774186 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:53.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30dffff640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006f530 0x7f30e006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.102+0000 7f30df7fe640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00add70 0x7f30e01ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e4864640 1 -- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00add70 msgr2=0x7f30e01ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e4864640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00add70 0x7f30e01ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e4864640 1 -- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006f530 msgr2=0x7f30e006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e4864640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006f530 0x7f30e006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e4864640 1 -- 172.21.15.78:0/436774186 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4009400 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e4864640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 0x7f30e00c1b60 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c7d0 tx=0x7f30d800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d800a640 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d800a7a0 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e01ee670 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d800ab90 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.103+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e00c6b30 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.104+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00676a0 con 0x7f30e00c2f60 2024-02-19T15:00:53.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.104+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f30d800acf0 con 0x7f30e00c2f60 2024-02-19T15:00:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.105+0000 7f30dcff9640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f30cc06b810 0x7f30cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.105+0000 7f30dffff640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f30cc06b810 0x7f30cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.105+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f30d8089ad0 con 0x7f30e00c2f60 2024-02-19T15:00:53.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.105+0000 7f30dffff640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f30cc06b810 0x7f30cc06dcd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f30d4036000 tx=0x7f30d403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:53.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.109+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f30d80544a0 con 0x7f30e00c2f60 2024-02-19T15:00:53.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.349+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f30e0074990 con 0x7f30e00c2f60 2024-02-19T15:00:53.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.349+0000 7f30dcff9640 1 -- 172.21.15.78:0/436774186 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1534 (secure 0 0 0) 0x7f30d8018080 con 0x7f30e00c2f60 2024-02-19T15:00:53.352 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:53.353 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":213,"num_osds":8,"num_up_osds":8,"osd_up_since":1708354307,"num_in_osds":8,"osd_in_since":1708354298,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":277942272,"bytes_avail":772816171008,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":190,"id":16,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":16,"rank":0,"name":"d","status":"up:active","gid":10731}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-02-19T15:00:16.157722+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-19T15:00:53.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f30cc06b810 msgr2=0x7f30cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f30cc06b810 0x7f30cc06dcd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f30d4036000 tx=0x7f30d403d040 comp rx=0 tx=0).stop 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 msgr2=0x7f30e00c1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 0x7f30e00c1b60 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c7d0 tx=0x7f30d800cc90 comp rx=0 tx=0).stop 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 shutdown_connections 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f30cc06b810 0x7f30cc06dcd0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30e00c2f60 0x7f30e00c1b60 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30e00add70 0x7f30e01ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 --2- 172.21.15.78:0/436774186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30e006f530 0x7f30e006c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 >> 172.21.15.78:0/436774186 conn(0x7f30e00c85e0 msgr2=0x7f30e014fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 shutdown_connections 2024-02-19T15:00:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.351+0000 7f30e62c7640 1 -- 172.21.15.78:0/436774186 wait complete. 2024-02-19T15:00:53.365 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-19T15:00:53.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:00:53.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.462+0000 7f6abba39640 1 Processor -- start 2024-02-19T15:00:53.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.462+0000 7f6abba39640 1 -- start start 2024-02-19T15:00:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6abba39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 0x7f6ab40c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6abba39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ab40c42e0 0x7f6ab4158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6abba39640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab40c34a0 con 0x7f6ab40c2f50 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6abba39640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab406c090 con 0x7f6ab40c42e0 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6abba39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab406d6a0 con 0x7f6ab40c8500 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6ab97d5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 0x7f6ab40c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6ab8fd4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ab40c42e0 0x7f6ab4158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6ab9fd6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6ab40c2f50 0x7f6ab4158b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:53066/0 (socket says 172.21.15.78:53066) 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6ab97d5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 0x7f6ab40c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:59592/0 (socket says 172.21.15.78:59592) 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6ab8fd4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ab40c42e0 0x7f6ab4158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:40254/0 (socket says 172.21.15.78:40254) 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.463+0000 7f6ab9fd6640 1 -- 172.21.15.78:0/3653203795 learned_addr learned my addr 172.21.15.78:0/3653203795 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.464+0000 7f6ab97d5640 1 -- 172.21.15.78:0/3653203795 >> v2:172.21.15.150:3300/0 conn(0x7f6ab40c42e0 msgr2=0x7f6ab4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.464+0000 7f6ab97d5640 1 --2- 172.21.15.78:0/3653203795 >> v2:172.21.15.150:3300/0 conn(0x7f6ab40c42e0 0x7f6ab4158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.464+0000 7f6ab97d5640 1 -- 172.21.15.78:0/3653203795 >> v1:172.21.15.150:6789/0 conn(0x7f6ab40c2f50 legacy=0x7f6ab4158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.464+0000 7f6ab97d5640 1 -- 172.21.15.78:0/3653203795 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab406c8e0 con 0x7f6ab40c8500 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.464+0000 7f6ab97d5640 1 --2- 172.21.15.78:0/3653203795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 0x7f6ab40c88e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00097a0 tx=0x7f6ab002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85ba7d1c15efd2cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.464+0000 7f6aabfff640 1 -- 172.21.15.78:0/3653203795 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab002e070 con 0x7f6ab40c8500 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6aabfff640 1 -- 172.21.15.78:0/3653203795 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab00315a0 con 0x7f6ab40c8500 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6aabfff640 1 -- 172.21.15.78:0/3653203795 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0031930 con 0x7f6ab40c8500 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 -- 172.21.15.78:0/3653203795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 msgr2=0x7f6ab40c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 --2- 172.21.15.78:0/3653203795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 0x7f6ab40c88e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00097a0 tx=0x7f6ab002b840 comp rx=0 tx=0).stop 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 -- 172.21.15.78:0/3653203795 shutdown_connections 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 --2- 172.21.15.78:0/3653203795 >> v2:172.21.15.150:3300/0 conn(0x7f6ab40c42e0 0x7f6ab4158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 --2- 172.21.15.78:0/3653203795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c8500 0x7f6ab40c88e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 -- 172.21.15.78:0/3653203795 >> 172.21.15.78:0/3653203795 conn(0x7f6ab40aa300 msgr2=0x7f6ab40aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 -- 172.21.15.78:0/3653203795 shutdown_connections 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.465+0000 7f6abba39640 1 -- 172.21.15.78:0/3653203795 wait complete. 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.466+0000 7f6abba39640 1 Processor -- start 2024-02-19T15:00:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.466+0000 7f6abba39640 1 -- start start 2024-02-19T15:00:53.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6abba39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c2f50 0x7f6ab4070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6abba39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 0x7f6ab414a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6abba39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab414b530 0x7f6ab414ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6abba39640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41eaa10 con 0x7f6ab40c8500 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6abba39640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41ebe80 con 0x7f6ab414b530 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6abba39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41533e0 con 0x7f6ab40c2f50 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 0x7f6ab414a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 0x7f6ab414a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46188/0 (socket says 172.21.15.78:46188) 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 -- 172.21.15.78:0/592395955 learned_addr learned my addr 172.21.15.78:0/592395955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab97d5640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c2f50 0x7f6ab4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab9fd6640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab414b530 0x7f6ab414ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 -- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c2f50 msgr2=0x7f6ab4070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c2f50 0x7f6ab4070550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 -- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab414b530 msgr2=0x7f6ab414ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab414b530 0x7f6ab414ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.467+0000 7f6ab8fd4640 1 -- 172.21.15.78:0/592395955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0009400 con 0x7f6ab40c8500 2024-02-19T15:00:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.468+0000 7f6ab8fd4640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 0x7f6ab414a850 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9b0 tx=0x7f6aa400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.468+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4009670 con 0x7f6ab40c8500 2024-02-19T15:00:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.468+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa400c9a0 con 0x7f6ab40c8500 2024-02-19T15:00:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.468+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab41522b0 con 0x7f6ab40c8500 2024-02-19T15:00:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.468+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400cd50 con 0x7f6ab40c8500 2024-02-19T15:00:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.468+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab4153630 con 0x7f6ab40c8500 2024-02-19T15:00:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.469+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab4071c20 con 0x7f6ab40c8500 2024-02-19T15:00:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.470+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6aa4013470 con 0x7f6ab40c8500 2024-02-19T15:00:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.470+0000 7f6aa9ffb640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a9806b810 0x7f6a9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.470+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6aa408a840 con 0x7f6ab40c8500 2024-02-19T15:00:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.470+0000 7f6ab97d5640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a9806b810 0x7f6a9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:53.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.471+0000 7f6ab97d5640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a9806b810 0x7f6a9806dcd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0036000 tx=0x7f6ab003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:53.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.475+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6aa4055210 con 0x7f6ab40c8500 2024-02-19T15:00:53.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.644+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ab406cea0 con 0x7f6a9806b810 2024-02-19T15:00:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.650+0000 7f6aa9ffb640 1 -- 172.21.15.78:0/592395955 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277981 (secure 0 0 0) 0x7f6ab0031a30 con 0x7f6a9806b810 2024-02-19T15:00:53.656 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a9806b810 msgr2=0x7f6a9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a9806b810 0x7f6a9806dcd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0036000 tx=0x7f6ab003d040 comp rx=0 tx=0).stop 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 msgr2=0x7f6ab414a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 0x7f6ab414a850 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9b0 tx=0x7f6aa400ee70 comp rx=0 tx=0).stop 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 shutdown_connections 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6a9806b810 0x7f6a9806dcd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ab40c8500 0x7f6ab414a850 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ab414b530 0x7f6ab414ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 --2- 172.21.15.78:0/592395955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ab40c2f50 0x7f6ab4070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 >> 172.21.15.78:0/592395955 conn(0x7f6ab40aa300 msgr2=0x7f6ab40c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.653+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 shutdown_connections 2024-02-19T15:00:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:53.654+0000 7f6abba39640 1 -- 172.21.15.78:0/592395955 wait complete. 2024-02-19T15:00:53.667 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-02-19T15:00:52.165571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271428,"kb_used_data":21980,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754703292,"statfs":{"total":773094113280,"available":772816171008,"internally_reserved":0,"allocated":22507520,"data_stored":11557647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.865246"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302970+0000","last_change":"2024-02-19T15:00:47.302970+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302970+0000","last_undegraded":"2024-02-19T15:00:47.302970+0000","last_fullsized":"2024-02-19T15:00:47.302970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302378+0000","last_change":"2024-02-19T15:00:47.302378+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302378+0000","last_undegraded":"2024-02-19T15:00:47.302378+0000","last_fullsized":"2024-02-19T15:00:47.302378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302000+0000","last_change":"2024-02-19T15:00:47.302000+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302000+0000","last_undegraded":"2024-02-19T15:00:47.302000+0000","last_fullsized":"2024-02-19T15:00:47.302000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302352+0000","last_change":"2024-02-19T15:00:47.302352+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302352+0000","last_undegraded":"2024-02-19T15:00:47.302352+0000","last_fullsized":"2024-02-19T15:00:47.302352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298756+0000","last_change":"2024-02-19T15:00:47.298756+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298756+0000","last_undegraded":"2024-02-19T15:00:47.298756+0000","last_fullsized":"2024-02-19T15:00:47.298756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302263+0000","last_change":"2024-02-19T15:00:47.302263+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302263+0000","last_undegraded":"2024-02-19T15:00:47.302263+0000","last_fullsized":"2024-02-19T15:00:47.302263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301716+0000","last_change":"2024-02-19T15:00:47.301716+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301716+0000","last_undegraded":"2024-02-19T15:00:47.301716+0000","last_fullsized":"2024-02-19T15:00:47.301716+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298531+0000","last_change":"2024-02-19T15:00:47.298531+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298531+0000","last_undegraded":"2024-02-19T15:00:47.298531+0000","last_fullsized":"2024-02-19T15:00:47.298531+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300164+0000","last_change":"2024-02-19T15:00:47.300164+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300164+0000","last_undegraded":"2024-02-19T15:00:47.300164+0000","last_fullsized":"2024-02-19T15:00:47.300164+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301485+0000","last_change":"2024-02-19T15:00:47.301485+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301485+0000","last_undegraded":"2024-02-19T15:00:47.301485+0000","last_fullsized":"2024-02-19T15:00:47.301485+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299593+0000","last_change":"2024-02-19T15:00:47.299593+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299593+0000","last_undegraded":"2024-02-19T15:00:47.299593+0000","last_fullsized":"2024-02-19T15:00:47.299593+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303789+0000","last_change":"2024-02-19T15:00:47.303789+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303789+0000","last_undegraded":"2024-02-19T15:00:47.303789+0000","last_fullsized":"2024-02-19T15:00:47.303789+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298402+0000","last_change":"2024-02-19T15:00:47.298402+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298402+0000","last_undegraded":"2024-02-19T15:00:47.298402+0000","last_fullsized":"2024-02-19T15:00:47.298402+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299538+0000","last_change":"2024-02-19T15:00:47.299538+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299538+0000","last_undegraded":"2024-02-19T15:00:47.299538+0000","last_fullsized":"2024-02-19T15:00:47.299538+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.300012+0000","last_change":"2024-02-19T15:00:47.300012+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.300012+0000","last_undegraded":"2024-02-19T15:00:47.300012+0000","last_fullsized":"2024-02-19T15:00:47.300012+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299961+0000","last_change":"2024-02-19T15:00:47.299961+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299961+0000","last_undegraded":"2024-02-19T15:00:47.299961+0000","last_fullsized":"2024-02-19T15:00:47.299961+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301196+0000","last_change":"2024-02-19T15:00:47.301196+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301196+0000","last_undegraded":"2024-02-19T15:00:47.301196+0000","last_fullsized":"2024-02-19T15:00:47.301196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304210+0000","last_change":"2024-02-19T15:00:47.304210+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304210+0000","last_undegraded":"2024-02-19T15:00:47.304210+0000","last_fullsized":"2024-02-19T15:00:47.304210+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304793+0000","last_change":"2024-02-19T15:00:47.304793+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304793+0000","last_undegraded":"2024-02-19T15:00:47.304793+0000","last_fullsized":"2024-02-19T15:00:47.304793+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303248+0000","last_change":"2024-02-19T15:00:47.303248+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303248+0000","last_undegraded":"2024-02-19T15:00:47.303248+0000","last_fullsized":"2024-02-19T15:00:47.303248+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298045+0000","last_change":"2024-02-19T15:00:47.298045+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298045+0000","last_undegraded":"2024-02-19T15:00:47.298045+0000","last_fullsized":"2024-02-19T15:00:47.298045+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.298032+0000","last_change":"2024-02-19T15:00:47.298032+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.298032+0000","last_undegraded":"2024-02-19T15:00:47.298032+0000","last_fullsized":"2024-02-19T15:00:47.298032+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.301770+0000","last_change":"2024-02-19T15:00:47.301770+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.301770+0000","last_undegraded":"2024-02-19T15:00:47.301770+0000","last_fullsized":"2024-02-19T15:00:47.301770+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299522+0000","last_change":"2024-02-19T15:00:47.299522+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299522+0000","last_undegraded":"2024-02-19T15:00:47.299522+0000","last_fullsized":"2024-02-19T15:00:47.299522+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.299568+0000","last_change":"2024-02-19T15:00:47.299568+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.299568+0000","last_undegraded":"2024-02-19T15:00:47.299568+0000","last_fullsized":"2024-02-19T15:00:47.299568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.303829+0000","last_change":"2024-02-19T15:00:47.303829+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.303829+0000","last_undegraded":"2024-02-19T15:00:47.303829+0000","last_fullsized":"2024-02-19T15:00:47.303829+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304340+0000","last_change":"2024-02-19T15:00:47.304340+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304340+0000","last_undegraded":"2024-02-19T15:00:47.304340+0000","last_fullsized":"2024-02-19T15:00:47.304340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304306+0000","last_change":"2024-02-19T15:00:47.304306+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304306+0000","last_undegraded":"2024-02-19T15:00:47.304306+0000","last_fullsized":"2024-02-19T15:00:47.304306+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302782+0000","last_change":"2024-02-19T15:00:47.302782+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302782+0000","last_undegraded":"2024-02-19T15:00:47.302782+0000","last_fullsized":"2024-02-19T15:00:47.302782+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302732+0000","last_change":"2024-02-19T15:00:47.302732+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302732+0000","last_undegraded":"2024-02-19T15:00:47.302732+0000","last_fullsized":"2024-02-19T15:00:47.302732+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.304345+0000","last_change":"2024-02-19T15:00:47.304345+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.304345+0000","last_undegraded":"2024-02-19T15:00:47.304345+0000","last_fullsized":"2024-02-19T15:00:47.304345+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1,"reported_epoch":210,"state":"creating+peering","last_fresh":"2024-02-19T15:00:47.302819+0000","last_change":"2024-02-19T15:00:47.302819+0000","last_active":"2024-02-19T15:00:47.287509+0000","last_peered":"2024-02-19T15:00:47.287509+0000","last_clean":"2024-02-19T15:00:47.287509+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:47.302819+0000","last_undegraded":"2024-02-19T15:00:47.302819+0000","last_fullsized":"2024-02-19T15:00:47.302819+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T15:00:48.294950+0000","last_change":"2024-02-19T15:00:48.294950+0000","last_active":"2024-02-19T15:00:48.294950+0000","last_peered":"2024-02-19T15:00:48.294950+0000","last_clean":"2024-02-19T15:00:48.294950+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T15:00:48.294950+0000","last_undegraded":"2024-02-19T15:00:48.294950+0000","last_fullsized":"2024-02-19T15:00:48.294950+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":523,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-02-19T15:00:47.300822+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:00:47.300822+0000","last_peered":"2024-02-19T15:00:47.300822+0000","last_clean":"2024-02-19T15:00:47.300822+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:00:47.300822+0000","last_undegraded":"2024-02-19T15:00:47.300822+0000","last_fullsized":"2024-02-19T15:00:47.300822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34364,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337476,"statfs":{"total":96636764160,"available":96601575424,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.629},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68899999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.432},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.42399999999999999,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34599999999999997,"15min":0.32400000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.90500000000000003},"last":0.65200000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.42599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.503,"15min":0.52600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.627,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47799999999999998,"15min":0.52200000000000002},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.51900000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.748},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50700000000000001,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.502,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.55800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.69699999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.51900000000000002,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.86799999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.872},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.42799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.62},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71599999999999997,"5min":0.77100000000000002,"15min":0.85899999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.55300000000000005},"min":{"1min":0.34499999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.50800000000000001},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.82999999999999996,"5min":0.84399999999999997,"15min":0.877},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.497,"15min":0.55100000000000005},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.58299999999999996},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.90000000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.61299999999999999},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.03},"last":0.755}]}]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68799999999999994,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":1.341,"15min":1.341},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67500000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70299999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.442}]},{"osd":2,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":10.629,"15min":10.629},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59699999999999998,"15min":0.61599999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.53100000000000003,"15min":0.50800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.5},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77000000000000002,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.29099999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.75,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58199999999999996,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.70399999999999996,"5min":1.246,"15min":1.246},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81000000000000005,"5min":2.621,"15min":2.621},"last":0.41799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574960,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33776,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338064,"statfs":{"total":96636764160,"available":96602177536,"internally_reserved":0,"allocated":2654208,"data_stored":1291681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.78100000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.77100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":0.40799999999999997,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.308,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.39900000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.247}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":0.45200000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.42599999999999999,"5min":0.66400000000000003,"15min":1.512},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.41899999999999998,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.45800000000000002,"5min":0.67600000000000005,"15min":0.76700000000000002},"last":0.29999999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.879,"15min":1.133},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.85999999999999999,"15min":1.1539999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36899999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.86699999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":1.115},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.76100000000000001,"15min":1.236},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.86099999999999999,"15min":1.1830000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.995},"last":0.66100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2666496,"data_stored":1297890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.32800000000000001}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.496}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34368,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337472,"statfs":{"total":96636764160,"available":96601571328,"internally_reserved":0,"allocated":3260416,"data_stored":1888258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.17899999999999999},"max":{"1min":0.504,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.312},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.58099999999999996,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.317}]},{"osd":2,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.47499999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.25800000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34100000000000003},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56100000000000005,"5min":0.62,"15min":0.62},"last":0.27400000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55300000000000005,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.35999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56399999999999995,"15min":0.59199999999999997},"min":{"1min":0.40300000000000002,"5min":0.38500000000000001,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.503},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.34100000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.85099999999999998},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:00:54.569 INFO:tasks.ceph.mds.d.smithi150.stderr:2024-02-19T15:00:54.566+0000 7fc2f1360640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:00:56.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:00:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.781+0000 7fbb66ca4640 1 Processor -- start 2024-02-19T15:00:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.781+0000 7fbb66ca4640 1 -- start start 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb66ca4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c8500 0x7fbb600c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb66ca4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb600c2f50 0x7fbb60158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb66ca4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb600c34a0 con 0x7fbb600c42e0 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb66ca4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6006c090 con 0x7fbb600c2f50 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb66ca4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6006d6a0 con 0x7fbb600c8500 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb5ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbb600c42e0 0x7fbb60158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59890/0 (socket says 172.21.15.78:59890) 2024-02-19T15:00:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb5ffff640 1 -- 172.21.15.78:0/2404918438 learned_addr learned my addr 172.21.15.78:0/2404918438 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb64a40640 1 --2- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c8500 0x7fbb600c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb65241640 1 --2- 172.21.15.78:0/2404918438 >> v2:172.21.15.150:3300/0 conn(0x7fbb600c2f50 0x7fbb60158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb5f7fe640 1 -- 172.21.15.78:0/2404918438 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287401846 0 0) 0x7fbb600c34a0 con 0x7fbb600c42e0 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb5f7fe640 1 -- 172.21.15.78:0/2404918438 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003520 con 0x7fbb600c42e0 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb65241640 1 -- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c8500 msgr2=0x7fbb600c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb65241640 1 --2- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c8500 0x7fbb600c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb65241640 1 -- 172.21.15.78:0/2404918438 >> v1:172.21.15.78:6789/0 conn(0x7fbb600c42e0 legacy=0x7fbb60158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.782+0000 7fbb65241640 1 -- 172.21.15.78:0/2404918438 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6006c8e0 con 0x7fbb600c2f50 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.783+0000 7fbb65241640 1 --2- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c2f50 0x7fbb60158d60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009d90 tx=0x7fbb5802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=521ad96079009236 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.783+0000 7fbb5f7fe640 1 -- 172.21.15.78:0/2404918438 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5802cbc0 con 0x7fbb600c2f50 2024-02-19T15:00:56.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.783+0000 7fbb5f7fe640 1 -- 172.21.15.78:0/2404918438 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5802cd20 con 0x7fbb600c2f50 2024-02-19T15:00:56.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.783+0000 7fbb5f7fe640 1 -- 172.21.15.78:0/2404918438 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58034660 con 0x7fbb600c2f50 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 -- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c2f50 msgr2=0x7fbb60158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c2f50 0x7fbb60158d60 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009d90 tx=0x7fbb5802be10 comp rx=0 tx=0).stop 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 -- 172.21.15.78:0/2404918438 shutdown_connections 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c2f50 0x7fbb60158d60 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/2404918438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c8500 0x7fbb600c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 -- 172.21.15.78:0/2404918438 >> 172.21.15.78:0/2404918438 conn(0x7fbb600aa300 msgr2=0x7fbb600aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 -- 172.21.15.78:0/2404918438 shutdown_connections 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.784+0000 7fbb66ca4640 1 -- 172.21.15.78:0/2404918438 wait complete. 2024-02-19T15:00:56.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.785+0000 7fbb66ca4640 1 Processor -- start 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.785+0000 7fbb66ca4640 1 -- start start 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb66ca4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c2f50 0x7fbb6014b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb66ca4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c8500 0x7fbb601ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb66ca4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 0x7fbb601ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb66ca4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb60071480 con 0x7fbb60145ce0 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb66ca4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb600711b0 con 0x7fbb600c8500 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb66ca4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb600acb00 con 0x7fbb600c2f50 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 0x7fbb601ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 0x7fbb601ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46194/0 (socket says 172.21.15.78:46194) 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 -- 172.21.15.78:0/4050385016 learned_addr learned my addr 172.21.15.78:0/4050385016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb64a40640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c2f50 0x7fbb6014b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb5ffff640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c8500 0x7fbb601ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:56.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 -- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c2f50 msgr2=0x7fbb6014b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c2f50 0x7fbb6014b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 -- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c8500 msgr2=0x7fbb601ee2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c8500 0x7fbb601ee2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.786+0000 7fbb65241640 1 -- 172.21.15.78:0/4050385016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb58009a70 con 0x7fbb60145ce0 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.787+0000 7fbb65241640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 0x7fbb601ee810 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009d90 tx=0x7fbb58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.787+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58040650 con 0x7fbb60145ce0 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.787+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5802ce90 con 0x7fbb60145ce0 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.787+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb601eee80 con 0x7fbb60145ce0 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.787+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb580409f0 con 0x7fbb60145ce0 2024-02-19T15:00:56.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.787+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb600ad080 con 0x7fbb60145ce0 2024-02-19T15:00:56.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.788+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb60071c20 con 0x7fbb60145ce0 2024-02-19T15:00:56.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.788+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbb58040b50 con 0x7fbb60145ce0 2024-02-19T15:00:56.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.789+0000 7fbb5d7fa640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbb4406b840 0x7fbb4406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:56.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.789+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbb580aca00 con 0x7fbb60145ce0 2024-02-19T15:00:56.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.789+0000 7fbb64a40640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbb4406b840 0x7fbb4406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:56.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.789+0000 7fbb64a40640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbb4406b840 0x7fbb4406dd00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009aa0 tx=0x7fbb54009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:56.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.793+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbb580773d0 con 0x7fbb60145ce0 2024-02-19T15:00:56.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.961+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb60156150 con 0x7fbb4406b840 2024-02-19T15:00:56.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.968+0000 7fbb5d7fa640 1 -- 172.21.15.78:0/4050385016 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290154 (secure 0 0 0) 0x7fbb54011a00 con 0x7fbb4406b840 2024-02-19T15:00:56.974 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:56.975 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:00:56.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.971+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbb4406b840 msgr2=0x7fbb4406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:56.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.971+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbb4406b840 0x7fbb4406dd00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009aa0 tx=0x7fbb54009190 comp rx=0 tx=0).stop 2024-02-19T15:00:56.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.971+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 msgr2=0x7fbb601ee810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:56.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.971+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 0x7fbb601ee810 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009d90 tx=0x7fbb58004a40 comp rx=0 tx=0).stop 2024-02-19T15:00:56.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 shutdown_connections 2024-02-19T15:00:56.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbb4406b840 0x7fbb4406dd00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb60145ce0 0x7fbb601ee810 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb600c8500 0x7fbb601ee2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 --2- 172.21.15.78:0/4050385016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb600c2f50 0x7fbb6014b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 >> 172.21.15.78:0/4050385016 conn(0x7fbb600aa300 msgr2=0x7fbb600ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 shutdown_connections 2024-02-19T15:00:56.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:56.972+0000 7fbb66ca4640 1 -- 172.21.15.78:0/4050385016 wait complete. 2024-02-19T15:00:56.985 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-02-19T15:00:56.166501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271788,"kb_used_data":22344,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754702932,"statfs":{"total":773094113280,"available":772815802368,"internally_reserved":0,"allocated":22880256,"data_stored":11732884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.861282"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306755+0000","last_change":"2024-02-19T15:00:49.314319+0000","last_active":"2024-02-19T15:00:50.306755+0000","last_peered":"2024-02-19T15:00:50.306755+0000","last_clean":"2024-02-19T15:00:50.306755+0000","last_became_active":"2024-02-19T15:00:49.313768+0000","last_became_peered":"2024-02-19T15:00:49.313768+0000","last_unstale":"2024-02-19T15:00:50.306755+0000","last_undegraded":"2024-02-19T15:00:50.306755+0000","last_fullsized":"2024-02-19T15:00:50.306755+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:34:26.541480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457703+0000","last_change":"2024-02-19T15:00:49.327204+0000","last_active":"2024-02-19T15:00:51.457703+0000","last_peered":"2024-02-19T15:00:51.457703+0000","last_clean":"2024-02-19T15:00:51.457703+0000","last_became_active":"2024-02-19T15:00:49.326829+0000","last_became_peered":"2024-02-19T15:00:49.326829+0000","last_unstale":"2024-02-19T15:00:51.457703+0000","last_undegraded":"2024-02-19T15:00:51.457703+0000","last_fullsized":"2024-02-19T15:00:51.457703+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:55:53.428498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305672+0000","last_change":"2024-02-19T15:00:49.325995+0000","last_active":"2024-02-19T15:00:50.305672+0000","last_peered":"2024-02-19T15:00:50.305672+0000","last_clean":"2024-02-19T15:00:50.305672+0000","last_became_active":"2024-02-19T15:00:49.325729+0000","last_became_peered":"2024-02-19T15:00:49.325729+0000","last_unstale":"2024-02-19T15:00:50.305672+0000","last_undegraded":"2024-02-19T15:00:50.305672+0000","last_fullsized":"2024-02-19T15:00:50.305672+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:11:20.791134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458007+0000","last_change":"2024-02-19T15:00:49.320907+0000","last_active":"2024-02-19T15:00:51.458007+0000","last_peered":"2024-02-19T15:00:51.458007+0000","last_clean":"2024-02-19T15:00:51.458007+0000","last_became_active":"2024-02-19T15:00:49.320643+0000","last_became_peered":"2024-02-19T15:00:49.320643+0000","last_unstale":"2024-02-19T15:00:51.458007+0000","last_undegraded":"2024-02-19T15:00:51.458007+0000","last_fullsized":"2024-02-19T15:00:51.458007+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:17:21.071765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304146+0000","last_change":"2024-02-19T15:00:49.320408+0000","last_active":"2024-02-19T15:00:50.304146+0000","last_peered":"2024-02-19T15:00:50.304146+0000","last_clean":"2024-02-19T15:00:50.304146+0000","last_became_active":"2024-02-19T15:00:49.320110+0000","last_became_peered":"2024-02-19T15:00:49.320110+0000","last_unstale":"2024-02-19T15:00:50.304146+0000","last_undegraded":"2024-02-19T15:00:50.304146+0000","last_fullsized":"2024-02-19T15:00:50.304146+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.674082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310439+0000","last_change":"2024-02-19T15:00:49.328728+0000","last_active":"2024-02-19T15:00:51.310439+0000","last_peered":"2024-02-19T15:00:51.310439+0000","last_clean":"2024-02-19T15:00:51.310439+0000","last_became_active":"2024-02-19T15:00:49.328446+0000","last_became_peered":"2024-02-19T15:00:49.328446+0000","last_unstale":"2024-02-19T15:00:51.310439+0000","last_undegraded":"2024-02-19T15:00:51.310439+0000","last_fullsized":"2024-02-19T15:00:51.310439+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:40:09.565450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305191+0000","last_change":"2024-02-19T15:00:49.324736+0000","last_active":"2024-02-19T15:00:50.305191+0000","last_peered":"2024-02-19T15:00:50.305191+0000","last_clean":"2024-02-19T15:00:50.305191+0000","last_became_active":"2024-02-19T15:00:49.321304+0000","last_became_peered":"2024-02-19T15:00:49.321304+0000","last_unstale":"2024-02-19T15:00:50.305191+0000","last_undegraded":"2024-02-19T15:00:50.305191+0000","last_fullsized":"2024-02-19T15:00:50.305191+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:28.564452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457723+0000","last_change":"2024-02-19T15:00:49.326199+0000","last_active":"2024-02-19T15:00:51.457723+0000","last_peered":"2024-02-19T15:00:51.457723+0000","last_clean":"2024-02-19T15:00:51.457723+0000","last_became_active":"2024-02-19T15:00:49.325915+0000","last_became_peered":"2024-02-19T15:00:49.325915+0000","last_unstale":"2024-02-19T15:00:51.457723+0000","last_undegraded":"2024-02-19T15:00:51.457723+0000","last_fullsized":"2024-02-19T15:00:51.457723+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:00:48.246667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310736+0000","last_change":"2024-02-19T15:00:49.328439+0000","last_active":"2024-02-19T15:00:51.310736+0000","last_peered":"2024-02-19T15:00:51.310736+0000","last_clean":"2024-02-19T15:00:51.310736+0000","last_became_active":"2024-02-19T15:00:49.328164+0000","last_became_peered":"2024-02-19T15:00:49.328164+0000","last_unstale":"2024-02-19T15:00:51.310736+0000","last_undegraded":"2024-02-19T15:00:51.310736+0000","last_fullsized":"2024-02-19T15:00:51.310736+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:13:28.562730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603605+0000","last_change":"2024-02-19T15:00:49.322607+0000","last_active":"2024-02-19T15:00:50.603605+0000","last_peered":"2024-02-19T15:00:50.603605+0000","last_clean":"2024-02-19T15:00:50.603605+0000","last_became_active":"2024-02-19T15:00:49.322289+0000","last_became_peered":"2024-02-19T15:00:49.322289+0000","last_unstale":"2024-02-19T15:00:50.603605+0000","last_undegraded":"2024-02-19T15:00:50.603605+0000","last_fullsized":"2024-02-19T15:00:50.603605+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:03:59.466246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306859+0000","last_change":"2024-02-19T15:00:49.323418+0000","last_active":"2024-02-19T15:00:50.306859+0000","last_peered":"2024-02-19T15:00:50.306859+0000","last_clean":"2024-02-19T15:00:50.306859+0000","last_became_active":"2024-02-19T15:00:49.323166+0000","last_became_peered":"2024-02-19T15:00:49.323166+0000","last_unstale":"2024-02-19T15:00:50.306859+0000","last_undegraded":"2024-02-19T15:00:50.306859+0000","last_fullsized":"2024-02-19T15:00:50.306859+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:23.740140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310554+0000","last_change":"2024-02-19T15:00:49.329258+0000","last_active":"2024-02-19T15:00:51.310554+0000","last_peered":"2024-02-19T15:00:51.310554+0000","last_clean":"2024-02-19T15:00:51.310554+0000","last_became_active":"2024-02-19T15:00:49.328910+0000","last_became_peered":"2024-02-19T15:00:49.328910+0000","last_unstale":"2024-02-19T15:00:51.310554+0000","last_undegraded":"2024-02-19T15:00:51.310554+0000","last_fullsized":"2024-02-19T15:00:51.310554+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:23.165818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458333+0000","last_change":"2024-02-19T15:00:49.327607+0000","last_active":"2024-02-19T15:00:51.458333+0000","last_peered":"2024-02-19T15:00:51.458333+0000","last_clean":"2024-02-19T15:00:51.458333+0000","last_became_active":"2024-02-19T15:00:49.326063+0000","last_became_peered":"2024-02-19T15:00:49.326063+0000","last_unstale":"2024-02-19T15:00:51.458333+0000","last_undegraded":"2024-02-19T15:00:51.458333+0000","last_fullsized":"2024-02-19T15:00:51.458333+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:12:04.869512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457607+0000","last_change":"2024-02-19T15:00:49.317370+0000","last_active":"2024-02-19T15:00:51.457607+0000","last_peered":"2024-02-19T15:00:51.457607+0000","last_clean":"2024-02-19T15:00:51.457607+0000","last_became_active":"2024-02-19T15:00:49.317112+0000","last_became_peered":"2024-02-19T15:00:49.317112+0000","last_unstale":"2024-02-19T15:00:51.457607+0000","last_undegraded":"2024-02-19T15:00:51.457607+0000","last_fullsized":"2024-02-19T15:00:51.457607+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:00.786553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458169+0000","last_change":"2024-02-19T15:00:49.320519+0000","last_active":"2024-02-19T15:00:51.458169+0000","last_peered":"2024-02-19T15:00:51.458169+0000","last_clean":"2024-02-19T15:00:51.458169+0000","last_became_active":"2024-02-19T15:00:49.320271+0000","last_became_peered":"2024-02-19T15:00:49.320271+0000","last_unstale":"2024-02-19T15:00:51.458169+0000","last_undegraded":"2024-02-19T15:00:51.458169+0000","last_fullsized":"2024-02-19T15:00:51.458169+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:05.509217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603451+0000","last_change":"2024-02-19T15:00:49.317618+0000","last_active":"2024-02-19T15:00:50.603451+0000","last_peered":"2024-02-19T15:00:50.603451+0000","last_clean":"2024-02-19T15:00:50.603451+0000","last_became_active":"2024-02-19T15:00:49.317349+0000","last_became_peered":"2024-02-19T15:00:49.317349+0000","last_unstale":"2024-02-19T15:00:50.603451+0000","last_undegraded":"2024-02-19T15:00:50.603451+0000","last_fullsized":"2024-02-19T15:00:50.603451+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:47:16.404746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603656+0000","last_change":"2024-02-19T15:00:49.320569+0000","last_active":"2024-02-19T15:00:50.603656+0000","last_peered":"2024-02-19T15:00:50.603656+0000","last_clean":"2024-02-19T15:00:50.603656+0000","last_became_active":"2024-02-19T15:00:49.319981+0000","last_became_peered":"2024-02-19T15:00:49.319981+0000","last_unstale":"2024-02-19T15:00:50.603656+0000","last_undegraded":"2024-02-19T15:00:50.603656+0000","last_fullsized":"2024-02-19T15:00:50.603656+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:25:53.776453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308453+0000","last_change":"2024-02-19T15:00:49.315566+0000","last_active":"2024-02-19T15:00:51.308453+0000","last_peered":"2024-02-19T15:00:51.308453+0000","last_clean":"2024-02-19T15:00:51.308453+0000","last_became_active":"2024-02-19T15:00:49.315256+0000","last_became_peered":"2024-02-19T15:00:49.315256+0000","last_unstale":"2024-02-19T15:00:51.308453+0000","last_undegraded":"2024-02-19T15:00:51.308453+0000","last_fullsized":"2024-02-19T15:00:51.308453+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:46:33.820455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603635+0000","last_change":"2024-02-19T15:00:49.325928+0000","last_active":"2024-02-19T15:00:50.603635+0000","last_peered":"2024-02-19T15:00:50.603635+0000","last_clean":"2024-02-19T15:00:50.603635+0000","last_became_active":"2024-02-19T15:00:49.325661+0000","last_became_peered":"2024-02-19T15:00:49.325661+0000","last_unstale":"2024-02-19T15:00:50.603635+0000","last_undegraded":"2024-02-19T15:00:50.603635+0000","last_fullsized":"2024-02-19T15:00:50.603635+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:08:32.283312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310589+0000","last_change":"2024-02-19T15:00:49.330692+0000","last_active":"2024-02-19T15:00:51.310589+0000","last_peered":"2024-02-19T15:00:51.310589+0000","last_clean":"2024-02-19T15:00:51.310589+0000","last_became_active":"2024-02-19T15:00:49.328345+0000","last_became_peered":"2024-02-19T15:00:49.328345+0000","last_unstale":"2024-02-19T15:00:51.310589+0000","last_undegraded":"2024-02-19T15:00:51.310589+0000","last_fullsized":"2024-02-19T15:00:51.310589+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:48.365684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304158+0000","last_change":"2024-02-19T15:00:49.317214+0000","last_active":"2024-02-19T15:00:50.304158+0000","last_peered":"2024-02-19T15:00:50.304158+0000","last_clean":"2024-02-19T15:00:50.304158+0000","last_became_active":"2024-02-19T15:00:49.316922+0000","last_became_peered":"2024-02-19T15:00:49.316922+0000","last_unstale":"2024-02-19T15:00:50.304158+0000","last_undegraded":"2024-02-19T15:00:50.304158+0000","last_fullsized":"2024-02-19T15:00:50.304158+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:16:11.962877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306694+0000","last_change":"2024-02-19T15:00:49.319748+0000","last_active":"2024-02-19T15:00:50.306694+0000","last_peered":"2024-02-19T15:00:50.306694+0000","last_clean":"2024-02-19T15:00:50.306694+0000","last_became_active":"2024-02-19T15:00:49.318523+0000","last_became_peered":"2024-02-19T15:00:49.318523+0000","last_unstale":"2024-02-19T15:00:50.306694+0000","last_undegraded":"2024-02-19T15:00:50.306694+0000","last_fullsized":"2024-02-19T15:00:50.306694+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:21.280269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458299+0000","last_change":"2024-02-19T15:00:49.320860+0000","last_active":"2024-02-19T15:00:51.458299+0000","last_peered":"2024-02-19T15:00:51.458299+0000","last_clean":"2024-02-19T15:00:51.458299+0000","last_became_active":"2024-02-19T15:00:49.319953+0000","last_became_peered":"2024-02-19T15:00:49.319953+0000","last_unstale":"2024-02-19T15:00:51.458299+0000","last_undegraded":"2024-02-19T15:00:51.458299+0000","last_fullsized":"2024-02-19T15:00:51.458299+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:16:11.962877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306321+0000","last_change":"2024-02-19T15:00:49.325073+0000","last_active":"2024-02-19T15:00:50.306321+0000","last_peered":"2024-02-19T15:00:50.306321+0000","last_clean":"2024-02-19T15:00:50.306321+0000","last_became_active":"2024-02-19T15:00:49.319248+0000","last_became_peered":"2024-02-19T15:00:49.319248+0000","last_unstale":"2024-02-19T15:00:50.306321+0000","last_undegraded":"2024-02-19T15:00:50.306321+0000","last_fullsized":"2024-02-19T15:00:50.306321+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:34:46.319421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305564+0000","last_change":"2024-02-19T15:00:49.324441+0000","last_active":"2024-02-19T15:00:50.305564+0000","last_peered":"2024-02-19T15:00:50.305564+0000","last_clean":"2024-02-19T15:00:50.305564+0000","last_became_active":"2024-02-19T15:00:49.320767+0000","last_became_peered":"2024-02-19T15:00:49.320767+0000","last_unstale":"2024-02-19T15:00:50.305564+0000","last_undegraded":"2024-02-19T15:00:50.305564+0000","last_fullsized":"2024-02-19T15:00:50.305564+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:50.361532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457847+0000","last_change":"2024-02-19T15:00:49.314931+0000","last_active":"2024-02-19T15:00:51.457847+0000","last_peered":"2024-02-19T15:00:51.457847+0000","last_clean":"2024-02-19T15:00:51.457847+0000","last_became_active":"2024-02-19T15:00:49.314661+0000","last_became_peered":"2024-02-19T15:00:49.314661+0000","last_unstale":"2024-02-19T15:00:51.457847+0000","last_undegraded":"2024-02-19T15:00:51.457847+0000","last_fullsized":"2024-02-19T15:00:51.457847+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:02:24.047436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306900+0000","last_change":"2024-02-19T15:00:49.312608+0000","last_active":"2024-02-19T15:00:50.306900+0000","last_peered":"2024-02-19T15:00:50.306900+0000","last_clean":"2024-02-19T15:00:50.306900+0000","last_became_active":"2024-02-19T15:00:49.312339+0000","last_became_peered":"2024-02-19T15:00:49.312339+0000","last_unstale":"2024-02-19T15:00:50.306900+0000","last_undegraded":"2024-02-19T15:00:50.306900+0000","last_fullsized":"2024-02-19T15:00:50.306900+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:30:48.952191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306665+0000","last_change":"2024-02-19T15:00:49.322930+0000","last_active":"2024-02-19T15:00:50.306665+0000","last_peered":"2024-02-19T15:00:50.306665+0000","last_clean":"2024-02-19T15:00:50.306665+0000","last_became_active":"2024-02-19T15:00:49.322351+0000","last_became_peered":"2024-02-19T15:00:49.322351+0000","last_unstale":"2024-02-19T15:00:50.306665+0000","last_undegraded":"2024-02-19T15:00:50.306665+0000","last_fullsized":"2024-02-19T15:00:50.306665+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:22:04.173516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310421+0000","last_change":"2024-02-19T15:00:49.328981+0000","last_active":"2024-02-19T15:00:51.310421+0000","last_peered":"2024-02-19T15:00:51.310421+0000","last_clean":"2024-02-19T15:00:51.310421+0000","last_became_active":"2024-02-19T15:00:49.327923+0000","last_became_peered":"2024-02-19T15:00:49.327923+0000","last_unstale":"2024-02-19T15:00:51.310421+0000","last_undegraded":"2024-02-19T15:00:51.310421+0000","last_fullsized":"2024-02-19T15:00:51.310421+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:02:45.975013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458463+0000","last_change":"2024-02-19T15:00:49.318372+0000","last_active":"2024-02-19T15:00:51.458463+0000","last_peered":"2024-02-19T15:00:51.458463+0000","last_clean":"2024-02-19T15:00:51.458463+0000","last_became_active":"2024-02-19T15:00:49.317950+0000","last_became_peered":"2024-02-19T15:00:49.317950+0000","last_unstale":"2024-02-19T15:00:51.458463+0000","last_undegraded":"2024-02-19T15:00:51.458463+0000","last_fullsized":"2024-02-19T15:00:51.458463+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:06:47.430763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458294+0000","last_change":"2024-02-19T15:00:49.319050+0000","last_active":"2024-02-19T15:00:51.458294+0000","last_peered":"2024-02-19T15:00:51.458294+0000","last_clean":"2024-02-19T15:00:51.458294+0000","last_became_active":"2024-02-19T15:00:49.318784+0000","last_became_peered":"2024-02-19T15:00:49.318784+0000","last_unstale":"2024-02-19T15:00:51.458294+0000","last_undegraded":"2024-02-19T15:00:51.458294+0000","last_fullsized":"2024-02-19T15:00:51.458294+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:37:50.674082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603595+0000","last_change":"2024-02-19T15:00:49.323210+0000","last_active":"2024-02-19T15:00:50.603595+0000","last_peered":"2024-02-19T15:00:50.603595+0000","last_clean":"2024-02-19T15:00:50.603595+0000","last_became_active":"2024-02-19T15:00:49.322920+0000","last_became_peered":"2024-02-19T15:00:49.322920+0000","last_unstale":"2024-02-19T15:00:50.603595+0000","last_undegraded":"2024-02-19T15:00:50.603595+0000","last_fullsized":"2024-02-19T15:00:50.603595+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:11:20.791134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"212'4","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308486+0000","last_change":"2024-02-19T15:00:48.321784+0000","last_active":"2024-02-19T15:00:51.308486+0000","last_peered":"2024-02-19T15:00:51.308486+0000","last_clean":"2024-02-19T15:00:51.308486+0000","last_became_active":"2024-02-19T15:00:48.321510+0000","last_became_peered":"2024-02-19T15:00:48.321510+0000","last_unstale":"2024-02-19T15:00:51.308486+0000","last_undegraded":"2024-02-19T15:00:51.308486+0000","last_fullsized":"2024-02-19T15:00:51.308486+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:21:09.874255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308354+0000","last_change":"2024-02-19T15:00:49.318097+0000","last_active":"2024-02-19T15:00:51.308354+0000","last_peered":"2024-02-19T15:00:51.308354+0000","last_clean":"2024-02-19T15:00:51.308354+0000","last_became_active":"2024-02-19T15:00:49.317840+0000","last_became_peered":"2024-02-19T15:00:49.317840+0000","last_unstale":"2024-02-19T15:00:51.308354+0000","last_undegraded":"2024-02-19T15:00:51.308354+0000","last_fullsized":"2024-02-19T15:00:51.308354+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:15:34.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603563+0000","last_change":"2024-02-19T15:00:48.319906+0000","last_active":"2024-02-19T15:00:50.603563+0000","last_peered":"2024-02-19T15:00:50.603563+0000","last_clean":"2024-02-19T15:00:50.603563+0000","last_became_active":"2024-02-19T15:00:48.319623+0000","last_became_peered":"2024-02-19T15:00:48.319623+0000","last_unstale":"2024-02-19T15:00:50.603563+0000","last_undegraded":"2024-02-19T15:00:50.603563+0000","last_fullsized":"2024-02-19T15:00:50.603563+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:42.772235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308520+0000","last_change":"2024-02-19T15:00:49.319056+0000","last_active":"2024-02-19T15:00:51.308520+0000","last_peered":"2024-02-19T15:00:51.308520+0000","last_clean":"2024-02-19T15:00:51.308520+0000","last_became_active":"2024-02-19T15:00:49.318765+0000","last_became_peered":"2024-02-19T15:00:49.318765+0000","last_unstale":"2024-02-19T15:00:51.308520+0000","last_undegraded":"2024-02-19T15:00:51.308520+0000","last_fullsized":"2024-02-19T15:00:51.308520+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:18:25.663543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306967+0000","last_change":"2024-02-19T15:00:48.320208+0000","last_active":"2024-02-19T15:00:50.306967+0000","last_peered":"2024-02-19T15:00:50.306967+0000","last_clean":"2024-02-19T15:00:50.306967+0000","last_became_active":"2024-02-19T15:00:48.319187+0000","last_became_peered":"2024-02-19T15:00:48.319187+0000","last_unstale":"2024-02-19T15:00:50.306967+0000","last_undegraded":"2024-02-19T15:00:50.306967+0000","last_fullsized":"2024-02-19T15:00:50.306967+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:41:02.575169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458167+0000","last_change":"2024-02-19T15:00:49.319592+0000","last_active":"2024-02-19T15:00:51.458167+0000","last_peered":"2024-02-19T15:00:51.458167+0000","last_clean":"2024-02-19T15:00:51.458167+0000","last_became_active":"2024-02-19T15:00:49.319322+0000","last_became_peered":"2024-02-19T15:00:49.319322+0000","last_unstale":"2024-02-19T15:00:51.458167+0000","last_undegraded":"2024-02-19T15:00:51.458167+0000","last_fullsized":"2024-02-19T15:00:51.458167+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:15:33.996325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603374+0000","last_change":"2024-02-19T15:00:48.321119+0000","last_active":"2024-02-19T15:00:50.603374+0000","last_peered":"2024-02-19T15:00:50.603374+0000","last_clean":"2024-02-19T15:00:50.603374+0000","last_became_active":"2024-02-19T15:00:48.320836+0000","last_became_peered":"2024-02-19T15:00:48.320836+0000","last_unstale":"2024-02-19T15:00:50.603374+0000","last_undegraded":"2024-02-19T15:00:50.603374+0000","last_fullsized":"2024-02-19T15:00:50.603374+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:13.914175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457567+0000","last_change":"2024-02-19T15:00:49.319591+0000","last_active":"2024-02-19T15:00:51.457567+0000","last_peered":"2024-02-19T15:00:51.457567+0000","last_clean":"2024-02-19T15:00:51.457567+0000","last_became_active":"2024-02-19T15:00:49.319342+0000","last_became_peered":"2024-02-19T15:00:49.319342+0000","last_unstale":"2024-02-19T15:00:51.457567+0000","last_undegraded":"2024-02-19T15:00:51.457567+0000","last_fullsized":"2024-02-19T15:00:51.457567+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:37:08.869658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304215+0000","last_change":"2024-02-19T15:00:48.315611+0000","last_active":"2024-02-19T15:00:50.304215+0000","last_peered":"2024-02-19T15:00:50.304215+0000","last_clean":"2024-02-19T15:00:50.304215+0000","last_became_active":"2024-02-19T15:00:48.315229+0000","last_became_peered":"2024-02-19T15:00:48.315229+0000","last_unstale":"2024-02-19T15:00:50.304215+0000","last_undegraded":"2024-02-19T15:00:50.304215+0000","last_fullsized":"2024-02-19T15:00:50.304215+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:41.062159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458482+0000","last_change":"2024-02-19T15:00:49.316944+0000","last_active":"2024-02-19T15:00:51.458482+0000","last_peered":"2024-02-19T15:00:51.458482+0000","last_clean":"2024-02-19T15:00:51.458482+0000","last_became_active":"2024-02-19T15:00:49.316663+0000","last_became_peered":"2024-02-19T15:00:49.316663+0000","last_unstale":"2024-02-19T15:00:51.458482+0000","last_undegraded":"2024-02-19T15:00:51.458482+0000","last_fullsized":"2024-02-19T15:00:51.458482+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:39:10.996844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457774+0000","last_change":"2024-02-19T15:00:48.317094+0000","last_active":"2024-02-19T15:00:51.457774+0000","last_peered":"2024-02-19T15:00:51.457774+0000","last_clean":"2024-02-19T15:00:51.457774+0000","last_became_active":"2024-02-19T15:00:48.316824+0000","last_became_peered":"2024-02-19T15:00:48.316824+0000","last_unstale":"2024-02-19T15:00:51.457774+0000","last_undegraded":"2024-02-19T15:00:51.457774+0000","last_fullsized":"2024-02-19T15:00:51.457774+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:48:27.557011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305416+0000","last_change":"2024-02-19T15:00:49.325958+0000","last_active":"2024-02-19T15:00:50.305416+0000","last_peered":"2024-02-19T15:00:50.305416+0000","last_clean":"2024-02-19T15:00:50.305416+0000","last_became_active":"2024-02-19T15:00:49.325661+0000","last_became_peered":"2024-02-19T15:00:49.325661+0000","last_unstale":"2024-02-19T15:00:50.305416+0000","last_undegraded":"2024-02-19T15:00:50.305416+0000","last_fullsized":"2024-02-19T15:00:50.305416+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:22.766683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603242+0000","last_change":"2024-02-19T15:00:48.320289+0000","last_active":"2024-02-19T15:00:50.603242+0000","last_peered":"2024-02-19T15:00:50.603242+0000","last_clean":"2024-02-19T15:00:50.603242+0000","last_became_active":"2024-02-19T15:00:48.319537+0000","last_became_peered":"2024-02-19T15:00:48.319537+0000","last_unstale":"2024-02-19T15:00:50.603242+0000","last_undegraded":"2024-02-19T15:00:50.603242+0000","last_fullsized":"2024-02-19T15:00:50.603242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:27.855316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458599+0000","last_change":"2024-02-19T15:00:49.325762+0000","last_active":"2024-02-19T15:00:51.458599+0000","last_peered":"2024-02-19T15:00:51.458599+0000","last_clean":"2024-02-19T15:00:51.458599+0000","last_became_active":"2024-02-19T15:00:49.325191+0000","last_became_peered":"2024-02-19T15:00:49.325191+0000","last_unstale":"2024-02-19T15:00:51.458599+0000","last_undegraded":"2024-02-19T15:00:51.458599+0000","last_fullsized":"2024-02-19T15:00:51.458599+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:15:09.856315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304069+0000","last_change":"2024-02-19T15:00:48.320722+0000","last_active":"2024-02-19T15:00:50.304069+0000","last_peered":"2024-02-19T15:00:50.304069+0000","last_clean":"2024-02-19T15:00:50.304069+0000","last_became_active":"2024-02-19T15:00:48.320447+0000","last_became_peered":"2024-02-19T15:00:48.320447+0000","last_unstale":"2024-02-19T15:00:50.304069+0000","last_undegraded":"2024-02-19T15:00:50.304069+0000","last_fullsized":"2024-02-19T15:00:50.304069+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:02:44.967572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457768+0000","last_change":"2024-02-19T15:00:49.321562+0000","last_active":"2024-02-19T15:00:51.457768+0000","last_peered":"2024-02-19T15:00:51.457768+0000","last_clean":"2024-02-19T15:00:51.457768+0000","last_became_active":"2024-02-19T15:00:49.321298+0000","last_became_peered":"2024-02-19T15:00:49.321298+0000","last_unstale":"2024-02-19T15:00:51.457768+0000","last_undegraded":"2024-02-19T15:00:51.457768+0000","last_fullsized":"2024-02-19T15:00:51.457768+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:57:07.695905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457813+0000","last_change":"2024-02-19T15:00:48.313473+0000","last_active":"2024-02-19T15:00:51.457813+0000","last_peered":"2024-02-19T15:00:51.457813+0000","last_clean":"2024-02-19T15:00:51.457813+0000","last_became_active":"2024-02-19T15:00:48.313178+0000","last_became_peered":"2024-02-19T15:00:48.313178+0000","last_unstale":"2024-02-19T15:00:51.457813+0000","last_undegraded":"2024-02-19T15:00:51.457813+0000","last_fullsized":"2024-02-19T15:00:51.457813+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:00:43.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308403+0000","last_change":"2024-02-19T15:00:49.317532+0000","last_active":"2024-02-19T15:00:51.308403+0000","last_peered":"2024-02-19T15:00:51.308403+0000","last_clean":"2024-02-19T15:00:51.308403+0000","last_became_active":"2024-02-19T15:00:49.317251+0000","last_became_peered":"2024-02-19T15:00:49.317251+0000","last_unstale":"2024-02-19T15:00:51.308403+0000","last_undegraded":"2024-02-19T15:00:51.308403+0000","last_fullsized":"2024-02-19T15:00:51.308403+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:47:45.493630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603644+0000","last_change":"2024-02-19T15:00:48.322464+0000","last_active":"2024-02-19T15:00:50.603644+0000","last_peered":"2024-02-19T15:00:50.603644+0000","last_clean":"2024-02-19T15:00:50.603644+0000","last_became_active":"2024-02-19T15:00:48.322171+0000","last_became_peered":"2024-02-19T15:00:48.322171+0000","last_unstale":"2024-02-19T15:00:50.603644+0000","last_undegraded":"2024-02-19T15:00:50.603644+0000","last_fullsized":"2024-02-19T15:00:50.603644+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:37.493631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310632+0000","last_change":"2024-02-19T15:00:49.324672+0000","last_active":"2024-02-19T15:00:51.310632+0000","last_peered":"2024-02-19T15:00:51.310632+0000","last_clean":"2024-02-19T15:00:51.310632+0000","last_became_active":"2024-02-19T15:00:49.324369+0000","last_became_peered":"2024-02-19T15:00:49.324369+0000","last_unstale":"2024-02-19T15:00:51.310632+0000","last_undegraded":"2024-02-19T15:00:51.310632+0000","last_fullsized":"2024-02-19T15:00:51.310632+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:37:08.869658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"212'3","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458141+0000","last_change":"2024-02-19T15:00:48.316060+0000","last_active":"2024-02-19T15:00:51.458141+0000","last_peered":"2024-02-19T15:00:51.458141+0000","last_clean":"2024-02-19T15:00:51.458141+0000","last_became_active":"2024-02-19T15:00:48.315785+0000","last_became_peered":"2024-02-19T15:00:48.315785+0000","last_unstale":"2024-02-19T15:00:51.458141+0000","last_undegraded":"2024-02-19T15:00:51.458141+0000","last_fullsized":"2024-02-19T15:00:51.458141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:22:55.737710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306782+0000","last_change":"2024-02-19T15:00:49.314048+0000","last_active":"2024-02-19T15:00:50.306782+0000","last_peered":"2024-02-19T15:00:50.306782+0000","last_clean":"2024-02-19T15:00:50.306782+0000","last_became_active":"2024-02-19T15:00:49.313459+0000","last_became_peered":"2024-02-19T15:00:49.313459+0000","last_unstale":"2024-02-19T15:00:50.306782+0000","last_undegraded":"2024-02-19T15:00:50.306782+0000","last_fullsized":"2024-02-19T15:00:50.306782+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:45:03.426685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"212'6","reported_seq":20,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308194+0000","last_change":"2024-02-19T15:00:48.319663+0000","last_active":"2024-02-19T15:00:51.308194+0000","last_peered":"2024-02-19T15:00:51.308194+0000","last_clean":"2024-02-19T15:00:51.308194+0000","last_became_active":"2024-02-19T15:00:48.319431+0000","last_became_peered":"2024-02-19T15:00:48.319431+0000","last_unstale":"2024-02-19T15:00:51.308194+0000","last_undegraded":"2024-02-19T15:00:51.308194+0000","last_fullsized":"2024-02-19T15:00:51.308194+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:40:51.615002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304107+0000","last_change":"2024-02-19T15:00:49.314226+0000","last_active":"2024-02-19T15:00:50.304107+0000","last_peered":"2024-02-19T15:00:50.304107+0000","last_clean":"2024-02-19T15:00:50.304107+0000","last_became_active":"2024-02-19T15:00:49.313938+0000","last_became_peered":"2024-02-19T15:00:49.313938+0000","last_unstale":"2024-02-19T15:00:50.304107+0000","last_undegraded":"2024-02-19T15:00:50.304107+0000","last_fullsized":"2024-02-19T15:00:50.304107+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:15:33.996325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304312+0000","last_change":"2024-02-19T15:00:48.322824+0000","last_active":"2024-02-19T15:00:50.304312+0000","last_peered":"2024-02-19T15:00:50.304312+0000","last_clean":"2024-02-19T15:00:50.304312+0000","last_became_active":"2024-02-19T15:00:48.322538+0000","last_became_peered":"2024-02-19T15:00:48.322538+0000","last_unstale":"2024-02-19T15:00:50.304312+0000","last_undegraded":"2024-02-19T15:00:50.304312+0000","last_fullsized":"2024-02-19T15:00:50.304312+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:53:22.158377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306626+0000","last_change":"2024-02-19T15:00:49.315946+0000","last_active":"2024-02-19T15:00:50.306626+0000","last_peered":"2024-02-19T15:00:50.306626+0000","last_clean":"2024-02-19T15:00:50.306626+0000","last_became_active":"2024-02-19T15:00:49.315658+0000","last_became_peered":"2024-02-19T15:00:49.315658+0000","last_unstale":"2024-02-19T15:00:50.306626+0000","last_undegraded":"2024-02-19T15:00:50.306626+0000","last_fullsized":"2024-02-19T15:00:50.306626+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:13:57.399664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458007+0000","last_change":"2024-02-19T15:00:48.323025+0000","last_active":"2024-02-19T15:00:51.458007+0000","last_peered":"2024-02-19T15:00:51.458007+0000","last_clean":"2024-02-19T15:00:51.458007+0000","last_became_active":"2024-02-19T15:00:48.322729+0000","last_became_peered":"2024-02-19T15:00:48.322729+0000","last_unstale":"2024-02-19T15:00:51.458007+0000","last_undegraded":"2024-02-19T15:00:51.458007+0000","last_fullsized":"2024-02-19T15:00:51.458007+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:40:04.322543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304116+0000","last_change":"2024-02-19T15:00:49.311531+0000","last_active":"2024-02-19T15:00:50.304116+0000","last_peered":"2024-02-19T15:00:50.304116+0000","last_clean":"2024-02-19T15:00:50.304116+0000","last_became_active":"2024-02-19T15:00:49.308788+0000","last_became_peered":"2024-02-19T15:00:49.308788+0000","last_unstale":"2024-02-19T15:00:50.304116+0000","last_undegraded":"2024-02-19T15:00:50.304116+0000","last_fullsized":"2024-02-19T15:00:50.304116+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:08:05.509217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457609+0000","last_change":"2024-02-19T15:00:48.316332+0000","last_active":"2024-02-19T15:00:51.457609+0000","last_peered":"2024-02-19T15:00:51.457609+0000","last_clean":"2024-02-19T15:00:51.457609+0000","last_became_active":"2024-02-19T15:00:48.316064+0000","last_became_peered":"2024-02-19T15:00:48.316064+0000","last_unstale":"2024-02-19T15:00:51.457609+0000","last_undegraded":"2024-02-19T15:00:51.457609+0000","last_fullsized":"2024-02-19T15:00:51.457609+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:17:08.909011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304306+0000","last_change":"2024-02-19T15:00:49.308620+0000","last_active":"2024-02-19T15:00:50.304306+0000","last_peered":"2024-02-19T15:00:50.304306+0000","last_clean":"2024-02-19T15:00:50.304306+0000","last_became_active":"2024-02-19T15:00:49.308361+0000","last_became_peered":"2024-02-19T15:00:49.308361+0000","last_unstale":"2024-02-19T15:00:50.304306+0000","last_undegraded":"2024-02-19T15:00:50.304306+0000","last_fullsized":"2024-02-19T15:00:50.304306+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:05.243375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"212'8","reported_seq":22,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458090+0000","last_change":"2024-02-19T15:00:48.312889+0000","last_active":"2024-02-19T15:00:51.458090+0000","last_peered":"2024-02-19T15:00:51.458090+0000","last_clean":"2024-02-19T15:00:51.458090+0000","last_became_active":"2024-02-19T15:00:48.312252+0000","last_became_peered":"2024-02-19T15:00:48.312252+0000","last_unstale":"2024-02-19T15:00:51.458090+0000","last_undegraded":"2024-02-19T15:00:51.458090+0000","last_fullsized":"2024-02-19T15:00:51.458090+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:59:15.134381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306828+0000","last_change":"2024-02-19T15:00:49.324844+0000","last_active":"2024-02-19T15:00:50.306828+0000","last_peered":"2024-02-19T15:00:50.306828+0000","last_clean":"2024-02-19T15:00:50.306828+0000","last_became_active":"2024-02-19T15:00:49.324181+0000","last_became_peered":"2024-02-19T15:00:49.324181+0000","last_unstale":"2024-02-19T15:00:50.306828+0000","last_undegraded":"2024-02-19T15:00:50.306828+0000","last_fullsized":"2024-02-19T15:00:50.306828+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:24:45.304356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310786+0000","last_change":"2024-02-19T15:00:48.323634+0000","last_active":"2024-02-19T15:00:51.310786+0000","last_peered":"2024-02-19T15:00:51.310786+0000","last_clean":"2024-02-19T15:00:51.310786+0000","last_became_active":"2024-02-19T15:00:48.323443+0000","last_became_peered":"2024-02-19T15:00:48.323443+0000","last_unstale":"2024-02-19T15:00:51.310786+0000","last_undegraded":"2024-02-19T15:00:51.310786+0000","last_fullsized":"2024-02-19T15:00:51.310786+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:21.759242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.307000+0000","last_change":"2024-02-19T15:00:49.316376+0000","last_active":"2024-02-19T15:00:50.307000+0000","last_peered":"2024-02-19T15:00:50.307000+0000","last_clean":"2024-02-19T15:00:50.307000+0000","last_became_active":"2024-02-19T15:00:49.316057+0000","last_became_peered":"2024-02-19T15:00:49.316057+0000","last_unstale":"2024-02-19T15:00:50.307000+0000","last_undegraded":"2024-02-19T15:00:50.307000+0000","last_fullsized":"2024-02-19T15:00:50.307000+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:51:25.485775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308493+0000","last_change":"2024-02-19T15:00:48.319670+0000","last_active":"2024-02-19T15:00:51.308493+0000","last_peered":"2024-02-19T15:00:51.308493+0000","last_clean":"2024-02-19T15:00:51.308493+0000","last_became_active":"2024-02-19T15:00:48.319342+0000","last_became_peered":"2024-02-19T15:00:48.319342+0000","last_unstale":"2024-02-19T15:00:51.308493+0000","last_undegraded":"2024-02-19T15:00:51.308493+0000","last_fullsized":"2024-02-19T15:00:51.308493+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:34:10.202315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457742+0000","last_change":"2024-02-19T15:00:49.327357+0000","last_active":"2024-02-19T15:00:51.457742+0000","last_peered":"2024-02-19T15:00:51.457742+0000","last_clean":"2024-02-19T15:00:51.457742+0000","last_became_active":"2024-02-19T15:00:49.327107+0000","last_became_peered":"2024-02-19T15:00:49.327107+0000","last_unstale":"2024-02-19T15:00:51.457742+0000","last_undegraded":"2024-02-19T15:00:51.457742+0000","last_fullsized":"2024-02-19T15:00:51.457742+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:41:22.935930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305678+0000","last_change":"2024-02-19T15:00:48.318405+0000","last_active":"2024-02-19T15:00:50.305678+0000","last_peered":"2024-02-19T15:00:50.305678+0000","last_clean":"2024-02-19T15:00:50.305678+0000","last_became_active":"2024-02-19T15:00:48.318114+0000","last_became_peered":"2024-02-19T15:00:48.318114+0000","last_unstale":"2024-02-19T15:00:50.305678+0000","last_undegraded":"2024-02-19T15:00:50.305678+0000","last_fullsized":"2024-02-19T15:00:50.305678+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:36:13.914175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306951+0000","last_change":"2024-02-19T15:00:49.314514+0000","last_active":"2024-02-19T15:00:50.306951+0000","last_peered":"2024-02-19T15:00:50.306951+0000","last_clean":"2024-02-19T15:00:50.306951+0000","last_became_active":"2024-02-19T15:00:49.314160+0000","last_became_peered":"2024-02-19T15:00:49.314160+0000","last_unstale":"2024-02-19T15:00:50.306951+0000","last_undegraded":"2024-02-19T15:00:50.306951+0000","last_fullsized":"2024-02-19T15:00:50.306951+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:42:22.672949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308196+0000","last_change":"2024-02-19T15:00:48.319840+0000","last_active":"2024-02-19T15:00:51.308196+0000","last_peered":"2024-02-19T15:00:51.308196+0000","last_clean":"2024-02-19T15:00:51.308196+0000","last_became_active":"2024-02-19T15:00:48.318633+0000","last_became_peered":"2024-02-19T15:00:48.318633+0000","last_unstale":"2024-02-19T15:00:51.308196+0000","last_undegraded":"2024-02-19T15:00:51.308196+0000","last_fullsized":"2024-02-19T15:00:51.308196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:29:17.334901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457697+0000","last_change":"2024-02-19T15:00:49.321639+0000","last_active":"2024-02-19T15:00:51.457697+0000","last_peered":"2024-02-19T15:00:51.457697+0000","last_clean":"2024-02-19T15:00:51.457697+0000","last_became_active":"2024-02-19T15:00:49.321405+0000","last_became_peered":"2024-02-19T15:00:49.321405+0000","last_unstale":"2024-02-19T15:00:51.457697+0000","last_undegraded":"2024-02-19T15:00:51.457697+0000","last_fullsized":"2024-02-19T15:00:51.457697+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:13:28.562730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304363+0000","last_change":"2024-02-19T15:00:48.309076+0000","last_active":"2024-02-19T15:00:50.304363+0000","last_peered":"2024-02-19T15:00:50.304363+0000","last_clean":"2024-02-19T15:00:50.304363+0000","last_became_active":"2024-02-19T15:00:48.308807+0000","last_became_peered":"2024-02-19T15:00:48.308807+0000","last_unstale":"2024-02-19T15:00:50.304363+0000","last_undegraded":"2024-02-19T15:00:50.304363+0000","last_fullsized":"2024-02-19T15:00:50.304363+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:04:47.358243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.306690+0000","last_change":"2024-02-19T15:00:49.320189+0000","last_active":"2024-02-19T15:00:50.306690+0000","last_peered":"2024-02-19T15:00:50.306690+0000","last_clean":"2024-02-19T15:00:50.306690+0000","last_became_active":"2024-02-19T15:00:49.319936+0000","last_became_peered":"2024-02-19T15:00:49.319936+0000","last_unstale":"2024-02-19T15:00:50.306690+0000","last_undegraded":"2024-02-19T15:00:50.306690+0000","last_fullsized":"2024-02-19T15:00:50.306690+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T16:09:59.510898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.304037+0000","last_change":"2024-02-19T15:00:48.310013+0000","last_active":"2024-02-19T15:00:50.304037+0000","last_peered":"2024-02-19T15:00:50.304037+0000","last_clean":"2024-02-19T15:00:50.304037+0000","last_became_active":"2024-02-19T15:00:48.309741+0000","last_became_peered":"2024-02-19T15:00:48.309741+0000","last_unstale":"2024-02-19T15:00:50.304037+0000","last_undegraded":"2024-02-19T15:00:50.304037+0000","last_fullsized":"2024-02-19T15:00:50.304037+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:40:08.558009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458053+0000","last_change":"2024-02-19T15:00:49.323027+0000","last_active":"2024-02-19T15:00:51.458053+0000","last_peered":"2024-02-19T15:00:51.458053+0000","last_clean":"2024-02-19T15:00:51.458053+0000","last_became_active":"2024-02-19T15:00:49.321696+0000","last_became_peered":"2024-02-19T15:00:49.321696+0000","last_unstale":"2024-02-19T15:00:51.458053+0000","last_undegraded":"2024-02-19T15:00:51.458053+0000","last_fullsized":"2024-02-19T15:00:51.458053+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:35:02.341224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458527+0000","last_change":"2024-02-19T15:00:48.323665+0000","last_active":"2024-02-19T15:00:51.458527+0000","last_peered":"2024-02-19T15:00:51.458527+0000","last_clean":"2024-02-19T15:00:51.458527+0000","last_became_active":"2024-02-19T15:00:48.323351+0000","last_became_peered":"2024-02-19T15:00:48.323351+0000","last_unstale":"2024-02-19T15:00:51.458527+0000","last_undegraded":"2024-02-19T15:00:51.458527+0000","last_fullsized":"2024-02-19T15:00:51.458527+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:37:21.759242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458452+0000","last_change":"2024-02-19T15:00:49.319399+0000","last_active":"2024-02-19T15:00:51.458452+0000","last_peered":"2024-02-19T15:00:51.458452+0000","last_clean":"2024-02-19T15:00:51.458452+0000","last_became_active":"2024-02-19T15:00:49.319145+0000","last_became_peered":"2024-02-19T15:00:49.319145+0000","last_unstale":"2024-02-19T15:00:51.458452+0000","last_undegraded":"2024-02-19T15:00:51.458452+0000","last_fullsized":"2024-02-19T15:00:51.458452+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T17:07:05.243375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457767+0000","last_change":"2024-02-19T15:00:48.321531+0000","last_active":"2024-02-19T15:00:51.457767+0000","last_peered":"2024-02-19T15:00:51.457767+0000","last_clean":"2024-02-19T15:00:51.457767+0000","last_became_active":"2024-02-19T15:00:48.321235+0000","last_became_peered":"2024-02-19T15:00:48.321235+0000","last_unstale":"2024-02-19T15:00:51.457767+0000","last_undegraded":"2024-02-19T15:00:51.457767+0000","last_fullsized":"2024-02-19T15:00:51.457767+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:31:58.705898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310682+0000","last_change":"2024-02-19T15:00:49.329479+0000","last_active":"2024-02-19T15:00:51.310682+0000","last_peered":"2024-02-19T15:00:51.310682+0000","last_clean":"2024-02-19T15:00:51.310682+0000","last_became_active":"2024-02-19T15:00:49.328699+0000","last_became_peered":"2024-02-19T15:00:49.328699+0000","last_unstale":"2024-02-19T15:00:51.310682+0000","last_undegraded":"2024-02-19T15:00:51.310682+0000","last_fullsized":"2024-02-19T15:00:51.310682+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:21:00.786553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457898+0000","last_change":"2024-02-19T15:00:48.316523+0000","last_active":"2024-02-19T15:00:51.457898+0000","last_peered":"2024-02-19T15:00:51.457898+0000","last_clean":"2024-02-19T15:00:51.457898+0000","last_became_active":"2024-02-19T15:00:48.316260+0000","last_became_peered":"2024-02-19T15:00:48.316260+0000","last_unstale":"2024-02-19T15:00:51.457898+0000","last_undegraded":"2024-02-19T15:00:51.457898+0000","last_fullsized":"2024-02-19T15:00:51.457898+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:30:01.025835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310513+0000","last_change":"2024-02-19T15:00:49.322641+0000","last_active":"2024-02-19T15:00:51.310513+0000","last_peered":"2024-02-19T15:00:51.310513+0000","last_clean":"2024-02-19T15:00:51.310513+0000","last_became_active":"2024-02-19T15:00:49.322419+0000","last_became_peered":"2024-02-19T15:00:49.322419+0000","last_unstale":"2024-02-19T15:00:51.310513+0000","last_undegraded":"2024-02-19T15:00:51.310513+0000","last_fullsized":"2024-02-19T15:00:51.310513+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:18:42.069599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"213'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308631+0000","last_change":"2024-02-19T15:00:48.319403+0000","last_active":"2024-02-19T15:00:51.308631+0000","last_peered":"2024-02-19T15:00:51.308631+0000","last_clean":"2024-02-19T15:00:51.308631+0000","last_became_active":"2024-02-19T15:00:48.318959+0000","last_became_peered":"2024-02-19T15:00:48.318959+0000","last_unstale":"2024-02-19T15:00:51.308631+0000","last_undegraded":"2024-02-19T15:00:51.308631+0000","last_fullsized":"2024-02-19T15:00:51.308631+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:23:15.325760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305625+0000","last_change":"2024-02-19T15:00:49.318010+0000","last_active":"2024-02-19T15:00:50.305625+0000","last_peered":"2024-02-19T15:00:50.305625+0000","last_clean":"2024-02-19T15:00:50.305625+0000","last_became_active":"2024-02-19T15:00:49.317730+0000","last_became_peered":"2024-02-19T15:00:49.317730+0000","last_unstale":"2024-02-19T15:00:50.305625+0000","last_undegraded":"2024-02-19T15:00:50.305625+0000","last_fullsized":"2024-02-19T15:00:50.305625+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T19:47:16.404746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305727+0000","last_change":"2024-02-19T15:00:48.315754+0000","last_active":"2024-02-19T15:00:50.305727+0000","last_peered":"2024-02-19T15:00:50.305727+0000","last_clean":"2024-02-19T15:00:50.305727+0000","last_became_active":"2024-02-19T15:00:48.315428+0000","last_became_peered":"2024-02-19T15:00:48.315428+0000","last_unstale":"2024-02-19T15:00:50.305727+0000","last_undegraded":"2024-02-19T15:00:50.305727+0000","last_fullsized":"2024-02-19T15:00:50.305727+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:05:37.493631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310759+0000","last_change":"2024-02-19T15:00:49.330441+0000","last_active":"2024-02-19T15:00:51.310759+0000","last_peered":"2024-02-19T15:00:51.310759+0000","last_clean":"2024-02-19T15:00:51.310759+0000","last_became_active":"2024-02-19T15:00:49.329743+0000","last_became_peered":"2024-02-19T15:00:49.329743+0000","last_unstale":"2024-02-19T15:00:51.310759+0000","last_undegraded":"2024-02-19T15:00:51.310759+0000","last_fullsized":"2024-02-19T15:00:51.310759+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:55:53.428498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305546+0000","last_change":"2024-02-19T15:00:48.317777+0000","last_active":"2024-02-19T15:00:50.305546+0000","last_peered":"2024-02-19T15:00:50.305546+0000","last_clean":"2024-02-19T15:00:50.305546+0000","last_became_active":"2024-02-19T15:00:48.316581+0000","last_became_peered":"2024-02-19T15:00:48.316581+0000","last_unstale":"2024-02-19T15:00:50.305546+0000","last_undegraded":"2024-02-19T15:00:50.305546+0000","last_fullsized":"2024-02-19T15:00:50.305546+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:49:42.772235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457494+0000","last_change":"2024-02-19T15:00:49.322978+0000","last_active":"2024-02-19T15:00:51.457494+0000","last_peered":"2024-02-19T15:00:51.457494+0000","last_clean":"2024-02-19T15:00:51.457494+0000","last_became_active":"2024-02-19T15:00:49.322702+0000","last_became_peered":"2024-02-19T15:00:49.322702+0000","last_unstale":"2024-02-19T15:00:51.457494+0000","last_undegraded":"2024-02-19T15:00:51.457494+0000","last_fullsized":"2024-02-19T15:00:51.457494+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:18:44.993990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308185+0000","last_change":"2024-02-19T15:00:48.319089+0000","last_active":"2024-02-19T15:00:51.308185+0000","last_peered":"2024-02-19T15:00:51.308185+0000","last_clean":"2024-02-19T15:00:51.308185+0000","last_became_active":"2024-02-19T15:00:48.318791+0000","last_became_peered":"2024-02-19T15:00:48.318791+0000","last_unstale":"2024-02-19T15:00:51.308185+0000","last_undegraded":"2024-02-19T15:00:51.308185+0000","last_fullsized":"2024-02-19T15:00:51.308185+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T22:52:01.657838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458588+0000","last_change":"2024-02-19T15:00:49.319979+0000","last_active":"2024-02-19T15:00:51.458588+0000","last_peered":"2024-02-19T15:00:51.458588+0000","last_clean":"2024-02-19T15:00:51.458588+0000","last_became_active":"2024-02-19T15:00:49.318724+0000","last_became_peered":"2024-02-19T15:00:49.318724+0000","last_unstale":"2024-02-19T15:00:51.458588+0000","last_undegraded":"2024-02-19T15:00:51.458588+0000","last_fullsized":"2024-02-19T15:00:51.458588+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:17:21.120049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308103+0000","last_change":"2024-02-19T15:00:48.314769+0000","last_active":"2024-02-19T15:00:51.308103+0000","last_peered":"2024-02-19T15:00:51.308103+0000","last_clean":"2024-02-19T15:00:51.308103+0000","last_became_active":"2024-02-19T15:00:48.314487+0000","last_became_peered":"2024-02-19T15:00:48.314487+0000","last_unstale":"2024-02-19T15:00:51.308103+0000","last_undegraded":"2024-02-19T15:00:51.308103+0000","last_fullsized":"2024-02-19T15:00:51.308103+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T23:26:00.670336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.457727+0000","last_change":"2024-02-19T15:00:49.316544+0000","last_active":"2024-02-19T15:00:51.457727+0000","last_peered":"2024-02-19T15:00:51.457727+0000","last_clean":"2024-02-19T15:00:51.457727+0000","last_became_active":"2024-02-19T15:00:49.316276+0000","last_became_peered":"2024-02-19T15:00:49.316276+0000","last_unstale":"2024-02-19T15:00:51.457727+0000","last_undegraded":"2024-02-19T15:00:51.457727+0000","last_fullsized":"2024-02-19T15:00:51.457727+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T21:03:15.303114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.305273+0000","last_change":"2024-02-19T15:00:48.317656+0000","last_active":"2024-02-19T15:00:50.305273+0000","last_peered":"2024-02-19T15:00:50.305273+0000","last_clean":"2024-02-19T15:00:50.305273+0000","last_became_active":"2024-02-19T15:00:48.316016+0000","last_became_peered":"2024-02-19T15:00:48.316016+0000","last_unstale":"2024-02-19T15:00:50.305273+0000","last_undegraded":"2024-02-19T15:00:50.305273+0000","last_fullsized":"2024-02-19T15:00:50.305273+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T15:20:27.855316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:50.603444+0000","last_change":"2024-02-19T15:00:49.320992+0000","last_active":"2024-02-19T15:00:50.603444+0000","last_peered":"2024-02-19T15:00:50.603444+0000","last_clean":"2024-02-19T15:00:50.603444+0000","last_became_active":"2024-02-19T15:00:49.319971+0000","last_became_peered":"2024-02-19T15:00:49.319971+0000","last_unstale":"2024-02-19T15:00:50.603444+0000","last_undegraded":"2024-02-19T15:00:50.603444+0000","last_fullsized":"2024-02-19T15:00:50.603444+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:50:50.361532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.308276+0000","last_change":"2024-02-19T15:00:48.320851+0000","last_active":"2024-02-19T15:00:51.308276+0000","last_peered":"2024-02-19T15:00:51.308276+0000","last_clean":"2024-02-19T15:00:51.308276+0000","last_became_active":"2024-02-19T15:00:48.320606+0000","last_became_peered":"2024-02-19T15:00:48.320606+0000","last_unstale":"2024-02-19T15:00:51.308276+0000","last_undegraded":"2024-02-19T15:00:51.308276+0000","last_fullsized":"2024-02-19T15:00:51.308276+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:47.287509+0000","last_clean_scrub_stamp":"2024-02-19T15:00:47.287509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:17:53.902009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.458461+0000","last_change":"2024-02-19T15:00:49.327976+0000","last_active":"2024-02-19T15:00:51.458461+0000","last_peered":"2024-02-19T15:00:51.458461+0000","last_clean":"2024-02-19T15:00:51.458461+0000","last_became_active":"2024-02-19T15:00:49.325517+0000","last_became_peered":"2024-02-19T15:00:49.325517+0000","last_unstale":"2024-02-19T15:00:51.458461+0000","last_undegraded":"2024-02-19T15:00:51.458461+0000","last_fullsized":"2024-02-19T15:00:51.458461+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T15:00:48.294950+0000","last_clean_scrub_stamp":"2024-02-19T15:00:48.294950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-20T20:23:55.963426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":529,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-02-19T15:00:51.310213+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:00:51.310213+0000","last_peered":"2024-02-19T15:00:51.310213+0000","last_clean":"2024-02-19T15:00:51.310213+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:00:51.310213+0000","last_undegraded":"2024-02-19T15:00:51.310213+0000","last_fullsized":"2024-02-19T15:00:51.310213+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34408,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337432,"statfs":{"total":96636764160,"available":96601530368,"internally_reserved":0,"allocated":3305472,"data_stored":1909388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48699999999999999,"15min":0.47299999999999998},"min":{"1min":0.39900000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.26700000000000002},"max":{"1min":0.70699999999999996,"5min":0.71699999999999997,"15min":0.79500000000000004},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.47999999999999998},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.27200000000000002},"max":{"1min":0.69199999999999995,"5min":0.748,"15min":0.83099999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.505,"15min":0.503},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":0.76400000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.26700000000000002},"max":{"1min":0.64400000000000002,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.504},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72899999999999998,"5min":0.76900000000000002,"15min":0.77500000000000002},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71299999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.41499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76800000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45900000000000002,"15min":0.45200000000000001},"min":{"1min":0.23499999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80400000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49199999999999999,"15min":0.52900000000000003},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76000000000000001,"5min":0.871,"15min":0.871},"last":0.63400000000000001}]},{"osd":6,"last update":"Mon Feb 19 14:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55500000000000005,"15min":0.58699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75800000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.753},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574960,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2707456,"data_stored":1318450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.42399999999999999,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":0.72899999999999998,"5min":0.79600000000000004,"15min":0.95699999999999996},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52900000000000003,"15min":0.51700000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56200000000000006,"15min":0.53900000000000003},"min":{"1min":0.40999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.311},"max":{"1min":0.80700000000000005,"5min":0.81499999999999995,"15min":0.93600000000000005},"last":0.504},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.50600000000000001,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34599999999999997,"15min":0.32400000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.90500000000000003},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.42599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.503,"15min":0.52600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.627,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47799999999999998,"15min":0.52200000000000002},"min":{"1min":0.26300000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45800000000000002,"15min":0.51900000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50700000000000001,"15min":0.54900000000000004},"min":{"1min":0.29399999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.502,"15min":0.56299999999999994},"min":{"1min":0.34200000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.78900000000000003,"5min":0.81299999999999994,"15min":0.89600000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.55800000000000005},"min":{"1min":0.26700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.37}]}]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2707456,"data_stored":1318486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.51900000000000002,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.86799999999999999},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53300000000000003,"15min":0.50600000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.872},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.42799999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81299999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71599999999999997,"5min":0.77100000000000002,"15min":0.85899999999999999},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.36399999999999999},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.83799999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54300000000000004,"15min":0.55300000000000005},"min":{"1min":0.34499999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75600000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.50800000000000001},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54600000000000004},"min":{"1min":0.28199999999999997,"5min":0.26800000000000002,"15min":0.249},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.50800000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26100000000000001},"max":{"1min":0.82999999999999996,"5min":0.84399999999999997,"15min":0.877},"last":0.443},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.497,"15min":0.55100000000000005},"min":{"1min":0.30499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.58299999999999996},"min":{"1min":0.32600000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.90000000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.61299999999999999},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.03},"last":0.55400000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33832,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338008,"statfs":{"total":96636764160,"available":96602120192,"internally_reserved":0,"allocated":2715648,"data_stored":1320028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55800000000000005,"15min":0.54000000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68799999999999994,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":1.341,"15min":1.341},"last":0.439}]},{"osd":1,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67500000000000004,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.56100000000000005,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70299999999999996,"5min":4.7290000000000001,"15min":4.7290000000000001},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.32200000000000001},"max":{"1min":0.70999999999999996,"5min":10.629,"15min":10.629},"last":0.502}]},{"osd":3,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69199999999999995,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59699999999999998,"15min":0.61599999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.42799999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.53100000000000003,"15min":0.50800000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77000000000000002,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.374}]},{"osd":6,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52400000000000002,"15min":0.54200000000000004},"min":{"1min":0.29099999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.75,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58199999999999996,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":4.6239999999999997,"15min":4.6239999999999997},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52200000000000002,"15min":0.55900000000000005},"min":{"1min":0.33800000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.70399999999999996,"5min":1.246,"15min":1.246},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59399999999999997,"15min":0.62},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81000000000000005,"5min":2.621,"15min":2.621},"last":0.39500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574961,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":2707456,"data_stored":1318506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.316,"15min":0.314},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.159},"max":{"1min":0.439,"5min":0.61699999999999999,"15min":3.694},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.315,"5min":0.32200000000000001,"15min":0.32700000000000001},"min":{"1min":0.23000000000000001,"5min":0.192,"15min":0.189},"max":{"1min":0.44400000000000001,"5min":0.60399999999999998,"15min":4.835},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32100000000000001,"15min":0.315},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.40300000000000002,"5min":0.629,"15min":0.76100000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35499999999999998,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":6.3810000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":3.6320000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23699999999999999},"max":{"1min":0.46300000000000002,"5min":0.57999999999999996,"15min":4.9889999999999999},"last":0.317}]},{"osd":4,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.55500000000000005},"min":{"1min":0.315,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81000000000000005,"5min":0.82899999999999996,"15min":0.83799999999999997},"last":0.371},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":4.9539999999999997},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.59599999999999997},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82099999999999995,"5min":0.91000000000000003,"15min":3.6669999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.60799999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.82899999999999996,"5min":0.98099999999999998,"15min":4.9130000000000003},"last":0.46800000000000003}]},{"osd":6,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54800000000000004,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.82799999999999996,"15min":0.86799999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.63500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79000000000000004,"5min":0.877,"15min":3.7189999999999999},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Feb 19 14:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.82999999999999996,"15min":4.8799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56699999999999995,"15min":0.627},"min":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73499999999999999,"5min":0.84799999999999998,"15min":3.0760000000000001},"last":0.436}]}]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33840,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338000,"statfs":{"total":96636764160,"available":96602112000,"internally_reserved":0,"allocated":2723840,"data_stored":1320632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.32100000000000001,"15min":0.317},"min":{"1min":0.224,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":0.40799999999999997,"5min":0.66300000000000003,"15min":0.68899999999999995},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.308,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.39900000000000002,"5min":0.56799999999999995,"15min":0.57699999999999996},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":0.45200000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.42599999999999999,"5min":0.66400000000000003,"15min":1.512},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.19700000000000001},"max":{"1min":0.41899999999999998,"5min":0.60999999999999999,"15min":0.65700000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.375,"15min":0.374},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.45800000000000002,"5min":0.67600000000000005,"15min":0.76700000000000002},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81299999999999994,"5min":0.879,"15min":1.133},"last":0.44},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":0.85999999999999999,"15min":1.1539999999999999},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.36899999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78100000000000003,"5min":0.89900000000000002,"15min":0.94999999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84099999999999997,"5min":0.84599999999999997,"15min":0.86699999999999999},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.40799999999999997,"5min":0.375,"15min":0.36099999999999999},"max":{"1min":0.73699999999999999,"5min":0.76100000000000001,"15min":1.236},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76800000000000002,"5min":0.86099999999999999,"15min":1.1830000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.41199999999999998,"5min":0.40100000000000002,"15min":0.374},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.995},"last":0.48699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33828,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338012,"statfs":{"total":96636764160,"available":96602124288,"internally_reserved":0,"allocated":2711552,"data_stored":1318540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.312,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.80600000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33300000000000002},"min":{"1min":0.255,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.90400000000000003},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.32100000000000001,"15min":0.312},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.17699999999999999},"max":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.77400000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.35699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.53200000000000003,"5min":0.58599999999999997,"15min":0.76000000000000001},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40999999999999998,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.60899999999999999,"15min":0.72199999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.38500000000000001},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52200000000000002,"5min":0.57699999999999996,"15min":0.93300000000000005},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.54900000000000004,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":1.0009999999999999},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56000000000000005,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76200000000000001,"5min":0.81699999999999995,"15min":1.0229999999999999},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.80300000000000005,"5min":0.80900000000000005,"15min":1.026},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.996},"last":0.68999999999999995}]},{"osd":7,"last update":"Mon Feb 19 14:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.036},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.83399999999999996,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.67000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574961,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34408,"kb_used_data":3224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337432,"statfs":{"total":96636764160,"available":96601530368,"internally_reserved":0,"allocated":3301376,"data_stored":1908854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33000000000000002,"15min":0.308},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.17899999999999999},"max":{"1min":0.504,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33700000000000002,"15min":0.312},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.58099999999999996,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.33400000000000002},"min":{"1min":0.23499999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.47499999999999998,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.30599999999999999,"5min":0.23599999999999999,"15min":0.23300000000000001},"max":{"1min":0.53800000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34100000000000003},"min":{"1min":0.26700000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.379},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38700000000000001,"15min":0.375},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56100000000000005,"5min":0.62,"15min":0.62},"last":0.27700000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55300000000000005,"15min":0.56999999999999995},"min":{"1min":0.39900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.35999999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.317}]},{"osd":5,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56399999999999995,"15min":0.59199999999999997},"min":{"1min":0.40300000000000002,"5min":0.38500000000000001,"15min":0.28399999999999997},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.40600000000000003}]},{"osd":6,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.59199999999999997},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.41499999999999998,"5min":0.40300000000000002,"15min":0.34100000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.85099999999999998},"last":0.39300000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.38},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.443},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56200000000000006,"15min":0.58099999999999996},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.43099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:00:56.990 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-19T15:00:56.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-19T15:00:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.129+0000 7f9077525640 1 Processor -- start 2024-02-19T15:00:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.130+0000 7f9077525640 1 -- start start 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.130+0000 7f9077525640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9070158f80 0x7f90700c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.130+0000 7f9077525640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700c3cf0 con 0x7f90700c5130 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.130+0000 7f9077525640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9070159410 con 0x7f90700c30a0 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.130+0000 7f9077525640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9070158b60 con 0x7f9070158f80 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f90752c1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f90700c5130 0x7f90700c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:59896/0 (socket says 172.21.15.78:59896) 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f90752c1640 1 -- 172.21.15.78:0/804580036 learned_addr learned my addr 172.21.15.78:0/804580036 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9075ac2640 1 --2- 172.21.15.78:0/804580036 >> v2:172.21.15.150:3300/0 conn(0x7f9070158f80 0x7f90700c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949804078 0 0) 0x7f90700c3cf0 con 0x7f90700c5130 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c003520 con 0x7f90700c5130 2024-02-19T15:00:57.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814371350 0 0) 0x7f9070159410 con 0x7f90700c30a0 2024-02-19T15:00:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90700c3cf0 con 0x7f90700c30a0 2024-02-19T15:00:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9075ac2640 1 -- 172.21.15.78:0/804580036 >> v1:172.21.15.150:6789/0 conn(0x7f90700c30a0 legacy=0x7f907015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9075ac2640 1 -- 172.21.15.78:0/804580036 >> v1:172.21.15.78:6789/0 conn(0x7f90700c5130 legacy=0x7f90700c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.131+0000 7f9075ac2640 1 -- 172.21.15.78:0/804580036 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070158530 con 0x7f9070158f80 2024-02-19T15:00:57.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9075ac2640 1 --2- 172.21.15.78:0/804580036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90700c5510 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9068009d90 tx=0x7f906802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ef80889403b33a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:57.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906802cd00 con 0x7f9070158f80 2024-02-19T15:00:57.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906802ce60 con 0x7f9070158f80 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9063fff640 1 -- 172.21.15.78:0/804580036 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068034660 con 0x7f9070158f80 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9077525640 1 -- 172.21.15.78:0/804580036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 msgr2=0x7f90700c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9077525640 1 --2- 172.21.15.78:0/804580036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90700c5510 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9068009d90 tx=0x7f906802bf50 comp rx=0 tx=0).stop 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9077525640 1 -- 172.21.15.78:0/804580036 shutdown_connections 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9077525640 1 --2- 172.21.15.78:0/804580036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90700c5510 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.132+0000 7f9077525640 1 -- 172.21.15.78:0/804580036 >> 172.21.15.78:0/804580036 conn(0x7f907006b4b0 msgr2=0x7f90700c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.133+0000 7f9077525640 1 -- 172.21.15.78:0/804580036 shutdown_connections 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.133+0000 7f9077525640 1 -- 172.21.15.78:0/804580036 wait complete. 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 Processor -- start 2024-02-19T15:00:57.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 -- start start 2024-02-19T15:00:57.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 0x7f90700c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90700c5130 0x7f90701eb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90701eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907014ec50 con 0x7f90700c30a0 2024-02-19T15:00:57.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90701eea40 con 0x7f9070158f80 2024-02-19T15:00:57.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9077525640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90701eece0 con 0x7f90700c5130 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f90752c1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 0x7f90700c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f90752c1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 0x7f90700c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46202/0 (socket says 172.21.15.78:46202) 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f90752c1640 1 -- 172.21.15.78:0/1669492909 learned_addr learned my addr 172.21.15.78:0/1669492909 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9075ac2640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90701eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.134+0000 7f9074ac0640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90700c5130 0x7f90701eb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f90752c1640 1 -- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90700c5130 msgr2=0x7f90701eb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f90752c1640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90700c5130 0x7f90701eb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f90752c1640 1 -- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 msgr2=0x7f90701eb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f90752c1640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90701eb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f90752c1640 1 -- 172.21.15.78:0/1669492909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068009a70 con 0x7f90700c30a0 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f90752c1640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 0x7f90700c75c0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f906400c500 tx=0x7f906400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:57.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064005ce0 con 0x7f90700c30a0 2024-02-19T15:00:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064005e40 con 0x7f90700c30a0 2024-02-19T15:00:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90701572e0 con 0x7f90700c30a0 2024-02-19T15:00:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90640072f0 con 0x7f90700c30a0 2024-02-19T15:00:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.135+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90701582e0 con 0x7f90700c30a0 2024-02-19T15:00:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.136+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9070074990 con 0x7f90700c30a0 2024-02-19T15:00:57.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.137+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9064007450 con 0x7f90700c30a0 2024-02-19T15:00:57.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.137+0000 7f9061ffb640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f905406b810 0x7f905406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.137+0000 7f9074ac0640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f905406b810 0x7f905406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.137+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90640897d0 con 0x7f90700c30a0 2024-02-19T15:00:57.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.137+0000 7f9074ac0640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f905406b810 0x7f905406dcd0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f906c007fd0 tx=0x7f906c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:57.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.141+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f90640541a0 con 0x7f90700c30a0 2024-02-19T15:00:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.382+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f90700c9400 con 0x7f90700c30a0 2024-02-19T15:00:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.385+0000 7f9061ffb640 1 -- 172.21.15.78:0/1669492909 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+147 (secure 0 0 0) 0x7f90640170b0 con 0x7f90700c30a0 2024-02-19T15:00:57.388 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-02-19T15:00:57.388 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDRatNlhJSaNBAAzs80W3GaEpOMHCEiUHVmKg== 2024-02-19T15:00:57.388 INFO:teuthology.orchestra.run.smithi078.stdout: caps mds = "allow" 2024-02-19T15:00:57.388 INFO:teuthology.orchestra.run.smithi078.stdout: caps mon = "allow r" 2024-02-19T15:00:57.388 INFO:teuthology.orchestra.run.smithi078.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-19T15:00:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:updated caps for client.0 2024-02-19T15:00:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.386+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f905406b810 msgr2=0x7f905406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.386+0000 7f9077525640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f905406b810 0x7f905406dcd0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f906c007fd0 tx=0x7f906c009040 comp rx=0 tx=0).stop 2024-02-19T15:00:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.386+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 msgr2=0x7f90700c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.386+0000 7f9077525640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 0x7f90700c75c0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f906400c500 tx=0x7f906400c9c0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 shutdown_connections 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f905406b810 0x7f905406dcd0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9070158f80 0x7f90701eb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90700c5130 0x7f90701eb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 --2- 172.21.15.78:0/1669492909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90700c30a0 0x7f90700c75c0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 >> 172.21.15.78:0/1669492909 conn(0x7f907006b4b0 msgr2=0x7f9070147790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 shutdown_connections 2024-02-19T15:00:57.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.387+0000 7f9077525640 1 -- 172.21.15.78:0/1669492909 wait complete. 2024-02-19T15:00:57.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.497+0000 7f48765f4640 1 Processor -- start 2024-02-19T15:00:57.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.497+0000 7f48765f4640 1 -- start start 2024-02-19T15:00:57.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f48765f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48700c25e0 0x7f48701526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f48765f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4870072250 con 0x7f48700c5be0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f48765f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4870072480 con 0x7f48700c29c0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f48765f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4870152c90 con 0x7f48700c25e0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f4874b91640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f48700c5be0 0x7f487006e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48760/0 (socket says 172.21.15.78:48760) 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f4874b91640 1 -- 172.21.15.78:0/1042004249 learned_addr learned my addr 172.21.15.78:0/1042004249 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486ffff640 1 --2- 172.21.15.78:0/1042004249 >> v2:172.21.15.150:3300/0 conn(0x7f48700c25e0 0x7f48701526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957478966 0 0) 0x7f4870072250 con 0x7f48700c5be0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4858003520 con 0x7f48700c5be0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166229178 0 0) 0x7f4870072480 con 0x7f48700c29c0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4870072250 con 0x7f48700c29c0 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486ffff640 1 -- 172.21.15.78:0/1042004249 >> v1:172.21.15.150:6789/0 conn(0x7f48700c29c0 legacy=0x7f48700c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486ffff640 1 -- 172.21.15.78:0/1042004249 >> v1:172.21.15.78:6789/0 conn(0x7f48700c5be0 legacy=0x7f487006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433966033 0 0) 0x7f4858003520 con 0x7f48700c5be0 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.498+0000 7f486ffff640 1 -- 172.21.15.78:0/1042004249 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700c4b90 con 0x7f48700c25e0 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.499+0000 7f486ffff640 1 --2- 172.21.15.78:0/1042004249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c25e0 0x7f48701526a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f48600097a0 tx=0x7f486002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7bfcb9842a3b3d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.499+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680019f0 con 0x7f48700c25e0 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.499+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486002e050 con 0x7f48700c25e0 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.499+0000 7f486effd640 1 -- 172.21.15.78:0/1042004249 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860030710 con 0x7f48700c25e0 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 -- 172.21.15.78:0/1042004249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c25e0 msgr2=0x7f48701526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 --2- 172.21.15.78:0/1042004249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c25e0 0x7f48701526a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f48600097a0 tx=0x7f486002b790 comp rx=0 tx=0).stop 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 -- 172.21.15.78:0/1042004249 shutdown_connections 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 --2- 172.21.15.78:0/1042004249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c25e0 0x7f48701526a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 -- 172.21.15.78:0/1042004249 >> 172.21.15.78:0/1042004249 conn(0x7f487006b270 msgr2=0x7f487006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 -- 172.21.15.78:0/1042004249 shutdown_connections 2024-02-19T15:00:57.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.500+0000 7f48765f4640 1 -- 172.21.15.78:0/1042004249 wait complete. 2024-02-19T15:00:57.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.501+0000 7f48765f4640 1 Processor -- start 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.501+0000 7f48765f4640 1 -- start start 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f48765f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48700c25e0 0x7f48700bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f48765f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c29c0 0x7f48700aa570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f48765f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 0x7f48700aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f48765f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701ea9b0 con 0x7f48700c5be0 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f48765f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48700a8d70 con 0x7f48700c29c0 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f48765f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48701eb580 con 0x7f48700c25e0 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 0x7f48700aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 0x7f48700aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50086/0 (socket says 172.21.15.78:50086) 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 -- 172.21.15.78:0/3549212313 learned_addr learned my addr 172.21.15.78:0/3549212313 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f486ffff640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48700c25e0 0x7f48700bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f486f7fe640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c29c0 0x7f48700aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 -- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48700c25e0 msgr2=0x7f48700bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48700c25e0 0x7f48700bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 -- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c29c0 msgr2=0x7f48700aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c29c0 0x7f48700aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 -- 172.21.15.78:0/3549212313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009400 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.502+0000 7f4874b91640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 0x7f48700aaab0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f486800cc50 tx=0x7f486800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.503+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800ce60 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.503+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868012400 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.503+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48700a97d0 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.503+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868012770 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.503+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48700a9d90 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.504+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4870074990 con 0x7f48700c5be0 2024-02-19T15:00:57.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.504+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f48680128d0 con 0x7f48700c5be0 2024-02-19T15:00:57.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.504+0000 7f486cff9640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f485c06b810 0x7f485c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:57.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.505+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f486808a890 con 0x7f48700c5be0 2024-02-19T15:00:57.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.505+0000 7f486ffff640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f485c06b810 0x7f485c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:57.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.505+0000 7f486ffff640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f485c06b810 0x7f485c06dcd0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4860034000 tx=0x7f486002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:57.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.509+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4868055260 con 0x7f48700c5be0 2024-02-19T15:00:57.656 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:00:57.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:00:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.716+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48700c0bd0 con 0x7f48700c5be0 2024-02-19T15:00:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.716+0000 7f486cff9640 1 -- 172.21.15.78:0/3549212313 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f4868058920 con 0x7f48700c5be0 2024-02-19T15:00:57.719 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:57.719 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":190,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:00:50.657189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:00:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 190 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f485c06b810 msgr2=0x7f485c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f485c06b810 0x7f485c06dcd0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4860034000 tx=0x7f486002f000 comp rx=0 tx=0).stop 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 msgr2=0x7f48700aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 0x7f48700aaab0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f486800cc50 tx=0x7f486800a430 comp rx=0 tx=0).stop 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 shutdown_connections 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f485c06b810 0x7f485c06dcd0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48700c5be0 0x7f48700aaab0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48700c29c0 0x7f48700aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 --2- 172.21.15.78:0/3549212313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48700c25e0 0x7f48700bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 >> 172.21.15.78:0/3549212313 conn(0x7f487006b270 msgr2=0x7f487014ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 shutdown_connections 2024-02-19T15:00:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:57.719+0000 7f48765f4640 1 -- 172.21.15.78:0/3549212313 wait complete. 2024-02-19T15:00:58.135 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 190, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:00:50.657189+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10731}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:00:58.135 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-19T15:00:58.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T15:00:58.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.235+0000 7fc0998ff640 1 Processor -- start 2024-02-19T15:00:58.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.235+0000 7fc0998ff640 1 -- start start 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0998ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc0940c8e00 0x7fc09406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0998ff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0941551f0 con 0x7fc09406d910 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0998ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc094155490 con 0x7fc094152700 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0998ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc094155770 con 0x7fc0940c8e00 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0937fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc09406d910 0x7fc09415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48762/0 (socket says 172.21.15.78:48762) 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0937fe640 1 -- 172.21.15.78:0/1908000952 learned_addr learned my addr 172.21.15.78:0/1908000952 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1908000952 >> v2:172.21.15.150:3300/0 conn(0x7fc0940c8e00 0x7fc09406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc091ffb640 1 -- 172.21.15.78:0/1908000952 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849795842 0 0) 0x7fc0941551f0 con 0x7fc09406d910 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc091ffb640 1 -- 172.21.15.78:0/1908000952 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07c003520 con 0x7fc09406d910 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0927fc640 1 -- 172.21.15.78:0/1908000952 >> v1:172.21.15.150:6789/0 conn(0x7fc094152700 legacy=0x7fc0940c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0927fc640 1 -- 172.21.15.78:0/1908000952 >> v1:172.21.15.78:6789/0 conn(0x7fc09406d910 legacy=0x7fc09415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.236+0000 7fc0927fc640 1 -- 172.21.15.78:0/1908000952 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc094155a50 con 0x7fc0940c8e00 2024-02-19T15:00:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.237+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1908000952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0940c8e00 0x7fc09406c2a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc0880097a0 tx=0x7fc08802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38769f7f05e8d4ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.237+0000 7fc091ffb640 1 -- 172.21.15.78:0/1908000952 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088030610 con 0x7fc0940c8e00 2024-02-19T15:00:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.237+0000 7fc091ffb640 1 -- 172.21.15.78:0/1908000952 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc088030770 con 0x7fc0940c8e00 2024-02-19T15:00:58.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.237+0000 7fc091ffb640 1 -- 172.21.15.78:0/1908000952 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088030ae0 con 0x7fc0940c8e00 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 -- 172.21.15.78:0/1908000952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0940c8e00 msgr2=0x7fc09406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1908000952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0940c8e00 0x7fc09406c2a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc0880097a0 tx=0x7fc08802b960 comp rx=0 tx=0).stop 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 -- 172.21.15.78:0/1908000952 shutdown_connections 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1908000952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0940c8e00 0x7fc09406c2a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 -- 172.21.15.78:0/1908000952 >> 172.21.15.78:0/1908000952 conn(0x7fc0940cac90 msgr2=0x7fc0940cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 -- 172.21.15.78:0/1908000952 shutdown_connections 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.238+0000 7fc0998ff640 1 -- 172.21.15.78:0/1908000952 wait complete. 2024-02-19T15:00:58.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.239+0000 7fc0998ff640 1 Processor -- start 2024-02-19T15:00:58.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.239+0000 7fc0998ff640 1 -- start start 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0998ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc09406d910 0x7fc0940aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0998ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 0x7fc094145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0998ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc094152700 0x7fc094146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0998ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940acbe0 con 0x7fc0940c8e00 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0998ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940aa370 con 0x7fc09406d910 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0998ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0940ace80 con 0x7fc094152700 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 0x7fc094145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 0x7fc094145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50098/0 (socket says 172.21.15.78:50098) 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0937fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc094152700 0x7fc094146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 -- 172.21.15.78:0/1134594867 learned_addr learned my addr 172.21.15.78:0/1134594867 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc09406d910 0x7fc0940aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 -- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc094152700 msgr2=0x7fc094146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc094152700 0x7fc094146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 -- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc09406d910 msgr2=0x7fc0940aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc09406d910 0x7fc0940aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 -- 172.21.15.78:0/1134594867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088009400 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc094152700 0x7fc094146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 0x7fc094145ac0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fc088000c00 tx=0x7fc0880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.240+0000 7fc07bfff640 1 -- 172.21.15.78:0/1134594867 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088039c90 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.241+0000 7fc07bfff640 1 -- 172.21.15.78:0/1134594867 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08803f040 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.241+0000 7fc07bfff640 1 -- 172.21.15.78:0/1134594867 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088006e10 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.241+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0941540b0 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.241+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0940bddd0 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.242+0000 7fc07bfff640 1 -- 172.21.15.78:0/1134594867 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc088004030 con 0x7fc0940c8e00 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.242+0000 7fc07bfff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc08006b810 0x7fc08006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.242+0000 7fc07bfff640 1 -- 172.21.15.78:0/1134594867 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc0880ac2a0 con 0x7fc0940c8e00 2024-02-19T15:00:58.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.242+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc08006b810 0x7fc08006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.243+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc08006b810 0x7fc08006dcd0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc09414a270 tx=0x7fc084009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.243+0000 7fc09c5b2740 1 Processor -- start 2024-02-19T15:00:58.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.243+0000 7fc09c5b2740 1 -- start start 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd5072b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 0x55bd5072ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd506ef860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd506aacd0 con 0x55bd506f0ae0 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd5065df20 con 0x55bd5072a740 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd5065b450 con 0x55bd5072d8b0 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 0x55bd5072ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 0x55bd5072ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50104/0 (socket says 172.21.15.78:50104) 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 -- 172.21.15.78:0/1932732579 learned_addr learned my addr 172.21.15.78:0/1932732579 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd506ef860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd5072b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 -- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 msgr2=0x55bd506ef860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd506ef860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 -- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 msgr2=0x55bd5072b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd5072b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0927fc640 1 -- 172.21.15.78:0/1932732579 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd5060bf20 con 0x55bd506f0ae0 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.244+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd506ef860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.245+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 0x55bd5072ab20 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fc088082a70 tx=0x7fc08807a940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2f37b585285880 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.245+0000 7fc0797fa640 1 -- 172.21.15.78:0/1932732579 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088082fa0 con 0x55bd506f0ae0 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.245+0000 7fc0797fa640 1 -- 172.21.15.78:0/1932732579 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0880789e0 con 0x55bd506f0ae0 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.245+0000 7fc0797fa640 1 -- 172.21.15.78:0/1932732579 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088076fb0 con 0x55bd506f0ae0 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.245+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1932732579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 msgr2=0x55bd5072ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.245+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 0x55bd5072ab20 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fc088082a70 tx=0x7fc08807a940 comp rx=0 tx=0).stop 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1932732579 shutdown_connections 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd506f0ae0 0x55bd5072ab20 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd506ef860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1932732579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd5072b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1932732579 >> 172.21.15.78:0/1932732579 conn(0x55bd506ec120 msgr2=0x55bd50710b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1932732579 shutdown_connections 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1932732579 wait complete. 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 -- start start 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.246+0000 7fc09c5b2740 1 Processor -- start 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506eed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5073b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd5073f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd50656c30 con 0x55bd50723680 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd505fbd30 con 0x55bd5072a740 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd5032a310 con 0x55bd5072d8b0 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc09c5b2740 10 client.0 _ll_register_callbacks cb 0x55bd50603d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd5073f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd5073f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50114/0 (socket says 172.21.15.78:50114) 2024-02-19T15:00:58.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5073b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 -- 172.21.15.78:0/1584956057 learned_addr learned my addr 172.21.15.78:0/1584956057 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 -- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 msgr2=0x55bd5073b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5073b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 -- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 msgr2=0x55bd506eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 -- 172.21.15.78:0/1584956057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0940bddd0 con 0x55bd50723680 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5073b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd5073f6c0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fc08c007aa0 tx=0x7fc08c012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9943c334fa7cf513 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.247+0000 7fc0727fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc0727fc640 1 -- 172.21.15.78:0/1584956057 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c015440 con 0x55bd50723680 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc0727fc640 1 -- 172.21.15.78:0/1584956057 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc08c0155a0 con 0x55bd50723680 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd503566c0 con 0x55bd50723680 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc0727fc640 1 -- 172.21.15.78:0/1584956057 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=190) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc08c015760 con 0x55bd50723680 2024-02-19T15:00:58.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 10 client.10881 fetch_fsmap learned FSMap version 190 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=190,monmap=2+}) v3 -- 0x55bd506552b0 con 0x55bd50723680 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc0727fc640 1 -- 172.21.15.78:0/1584956057 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bd503566c0 con 0x55bd50723680 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc0727fc640 1 -- 172.21.15.78:0/1584956057 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 190) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc08c0158c0 con 0x55bd50723680 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 10 client.10881 fetch_fsmap finished waiting for FSMap version 190 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 10 client.10881 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 20 client.10881 populate_metadata read hostname 'smithi078' 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd5073bd60 0x55bd5070d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 4 client.10881 mds_command: new command op to 10731 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 --> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd503a7270 con 0x55bd5073bd60 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.248+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd5073bd60 0x55bd5070d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc0727fc640 10 client.10881 ms_handle_connect on v2:172.21.15.150:6833/1952320195 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd5073bd60 0x55bd5070d8a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60829bc6bb7cdd4 server_cookie=d0e116690f8b9c19 in_seq=0 out_seq=0 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc0727fc640 1 -- 172.21.15.78:0/1584956057 <== mds.0 v2:172.21.15.150:6833/1952320195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bd503a7270 con 0x55bd5073bd60 2024-02-19T15:00:58.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc0727fc640 10 client.10881 handle_command_reply: tid=0 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 client.10881 shutdown 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 msgr2=0x55bd5073f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd5073f6c0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fc08c007aa0 tx=0x7fc08c012690 comp rx=0 tx=0).stop 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 shutdown_connections 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd5073bd60 0x55bd5070d8a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd5073f6c0 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5073b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc0727fc640 0 client.10881 ms_handle_reset on v2:172.21.15.150:6833/1952320195 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1584956057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 >> 172.21.15.78:0/1584956057 conn(0x55bd506f0ae0 msgr2=0x55bd50710ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 shutdown_connections 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.249+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1584956057 wait complete. 2024-02-19T15:00:58.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.250+0000 7fc09c5b2740 20 client.10881 trim_cache size 0 max 16384 2024-02-19T15:00:58.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.266+0000 7fc09c5b2740 1 Processor -- start 2024-02-19T15:00:58.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.266+0000 7fc09c5b2740 1 -- start start 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd506eed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5072cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd505dc490 con 0x55bd50723680 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd506f0390 con 0x55bd5072a740 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd50688380 con 0x55bd5072d8b0 2024-02-19T15:00:58.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd506eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd506eed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50126/0 (socket says 172.21.15.78:50126) 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 -- 172.21.15.78:0/1295456206 learned_addr learned my addr 172.21.15.78:0/1295456206 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc0927fc640 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5072cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 -- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 msgr2=0x55bd5072cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5072cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 -- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 msgr2=0x55bd506e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 -- 172.21.15.78:0/1295456206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd505f1a20 con 0x55bd50723680 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc0937fe640 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5072cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc092ffd640 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd506eed10 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x55bd50722a40 tx=0x7fc08400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2c37ffa373a569 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc078ff9640 1 -- 172.21.15.78:0/1295456206 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084028040 con 0x55bd50723680 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc078ff9640 1 -- 172.21.15.78:0/1295456206 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08402a2c0 con 0x55bd50723680 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.267+0000 7fc078ff9640 1 -- 172.21.15.78:0/1295456206 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084016310 con 0x55bd50723680 2024-02-19T15:00:58.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1295456206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 msgr2=0x55bd506eed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd506eed10 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x55bd50722a40 tx=0x7fc08400a000 comp rx=0 tx=0).stop 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1295456206 shutdown_connections 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd5072d8b0 0x55bd5072cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072a740 0x55bd506e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/1295456206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd50723680 0x55bd506eed10 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1295456206 >> 172.21.15.78:0/1295456206 conn(0x55bd506f0ae0 msgr2=0x55bd5073f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1295456206 shutdown_connections 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.268+0000 7fc09c5b2740 1 -- 172.21.15.78:0/1295456206 wait complete. 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 -- start start 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 Processor -- start 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd50723680 0x55bd506ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 0x55bd5079a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072d8b0 0x55bd5079adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd5079b610 con 0x55bd5072a740 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd50723b80 con 0x55bd5072d8b0 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bd507242b0 con 0x55bd50723680 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc09c5b2740 10 client.0 _ll_register_callbacks cb 0x7fc08c006ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 0x55bd5079a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc0927fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 0x55bd5079a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50140/0 (socket says 172.21.15.78:50140) 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc0927fc640 1 -- 172.21.15.78:0/362022825 learned_addr learned my addr 172.21.15.78:0/362022825 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc0937fe640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072d8b0 0x55bd5079adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.269+0000 7fc092ffd640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd50723680 0x55bd506ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0927fc640 1 -- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd50723680 msgr2=0x55bd506ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0927fc640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd50723680 0x55bd506ea0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0927fc640 1 -- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072d8b0 msgr2=0x55bd5079adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0927fc640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072d8b0 0x55bd5079adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0927fc640 1 -- 172.21.15.78:0/362022825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc094155490 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0927fc640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 0x55bd5079a880 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fc0880aa610 tx=0x7fc08803b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc34d9f872d5002b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0727fc640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0727fc640 1 -- 172.21.15.78:0/362022825 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088082fa0 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0727fc640 1 -- 172.21.15.78:0/362022825 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc088092190 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0880789e0 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0727fc640 1 -- 172.21.15.78:0/362022825 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=190) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc088092350 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 10 client.10887 fetch_fsmap learned FSMap version 190 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=190,monmap=2+}) v3 -- 0x55bd50617780 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0727fc640 1 -- 172.21.15.78:0/362022825 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0880789e0 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc0727fc640 1 -- 172.21.15.78:0/362022825 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 190) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc088042070 con 0x55bd5072a740 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 10 client.10887 fetch_fsmap finished waiting for FSMap version 190 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 10 client.10887 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 20 client.10887 populate_metadata read hostname 'smithi078' 2024-02-19T15:00:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd50797d10 0x55bd507980f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 4 client.10887 mds_command: new command op to 10731 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.270+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 --> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bd50393610 con 0x55bd50797d10 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc092ffd640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd50797d10 0x55bd507980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc0727fc640 10 client.10887 ms_handle_connect on v2:172.21.15.150:6833/1952320195 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc092ffd640 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd50797d10 0x55bd507980f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dd1ce1ffb215c15 server_cookie=b05477eb31caacfd in_seq=0 out_seq=0 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc0727fc640 1 -- 172.21.15.78:0/362022825 <== mds.0 v2:172.21.15.150:6833/1952320195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55bd50393610 con 0x55bd50797d10 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc0727fc640 10 client.10887 handle_command_reply: tid=0 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 client.10887 shutdown 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 msgr2=0x55bd5079a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 0x55bd5079a880 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fc0880aa610 tx=0x7fc08803b070 comp rx=0 tx=0).stop 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 shutdown_connections 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x55bd50797d10 0x55bd507980f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bd5072a740 0x55bd5079a880 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc0727fc640 0 client.10887 ms_handle_reset on v2:172.21.15.150:6833/1952320195 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd5072d8b0 0x55bd5079adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 --2- 172.21.15.78:0/362022825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55bd50723680 0x55bd506ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 >> 172.21.15.78:0/362022825 conn(0x55bd506f0ae0 msgr2=0x55bd5072be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 shutdown_connections 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 1 -- 172.21.15.78:0/362022825 wait complete. 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.271+0000 7fc09c5b2740 20 client.10887 trim_cache size 0 max 16384 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 10731, 2024-02-19T15:00:58.283 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 8.9706248219999996, 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 190, 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 213, 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 213, 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 14.068316762 2024-02-19T15:00:58.284 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:00:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc08006b810 msgr2=0x7fc08006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc08006b810 0x7fc08006dcd0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc09414a270 tx=0x7fc084009040 comp rx=0 tx=0).stop 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 msgr2=0x7fc094145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 0x7fc094145ac0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fc088000c00 tx=0x7fc0880046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 shutdown_connections 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc08006b810 0x7fc08006dcd0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0940c8e00 0x7fc094145ac0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc094152700 0x7fc094146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 --2- 172.21.15.78:0/1134594867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc09406d910 0x7fc0940aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 >> 172.21.15.78:0/1134594867 conn(0x7fc0940cac90 msgr2=0x7fc09414a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 shutdown_connections 2024-02-19T15:00:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.272+0000 7fc0998ff640 1 -- 172.21.15.78:0/1134594867 wait complete. 2024-02-19T15:00:58.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:58.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.384+0000 7f119aced640 1 Processor -- start 2024-02-19T15:00:58.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.384+0000 7f119aced640 1 -- start start 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119aced640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11940ac390 0x7f11940ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119aced640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11940c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119aced640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1194158530 0x7f119406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119aced640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11940c8de0 con 0x7f11940ac390 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119aced640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1194155a60 con 0x7f1194158530 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119aced640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1194155d40 con 0x7f11940bdf20 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f1198a89640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11940ac390 0x7f11940ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f1198a89640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f11940ac390 0x7f11940ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50148/0 (socket says 172.21.15.78:50148) 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f1198a89640 1 -- 172.21.15.78:0/2092169712 learned_addr learned my addr 172.21.15.78:0/2092169712 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f1193fff640 1 --2- 172.21.15.78:0/2092169712 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11940c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.385+0000 7f119928a640 1 --2- 172.21.15.78:0/2092169712 >> v2:172.21.15.150:3300/0 conn(0x7f1194158530 0x7f119406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.386+0000 7f1198a89640 1 -- 172.21.15.78:0/2092169712 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 msgr2=0x7f11940c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.386+0000 7f1198a89640 1 --2- 172.21.15.78:0/2092169712 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11940c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.386+0000 7f1198a89640 1 -- 172.21.15.78:0/2092169712 >> v2:172.21.15.150:3300/0 conn(0x7f1194158530 msgr2=0x7f119406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.386+0000 7f1198a89640 1 --2- 172.21.15.78:0/2092169712 >> v2:172.21.15.150:3300/0 conn(0x7f1194158530 0x7f119406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.386+0000 7f1198a89640 1 -- 172.21.15.78:0/2092169712 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940c90a0 con 0x7f11940ac390 2024-02-19T15:00:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.386+0000 7f1198a89640 1 --2- 172.21.15.78:0/2092169712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11940ac770 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f119408bb70 tx=0x7f118402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0f19a4747e61a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.387+0000 7f11937fe640 1 -- 172.21.15.78:0/2092169712 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184031750 con 0x7f11940ac390 2024-02-19T15:00:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.387+0000 7f11937fe640 1 -- 172.21.15.78:0/2092169712 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11840318b0 con 0x7f11940ac390 2024-02-19T15:00:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.387+0000 7f11937fe640 1 -- 172.21.15.78:0/2092169712 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184031c40 con 0x7f11940ac390 2024-02-19T15:00:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 -- 172.21.15.78:0/2092169712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 msgr2=0x7f11940ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 --2- 172.21.15.78:0/2092169712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11940ac770 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f119408bb70 tx=0x7f118402b7f0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 -- 172.21.15.78:0/2092169712 shutdown_connections 2024-02-19T15:00:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 --2- 172.21.15.78:0/2092169712 >> v2:172.21.15.150:3300/0 conn(0x7f1194158530 0x7f119406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 --2- 172.21.15.78:0/2092169712 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11940c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 --2- 172.21.15.78:0/2092169712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11940ac770 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 -- 172.21.15.78:0/2092169712 >> 172.21.15.78:0/2092169712 conn(0x7f11940ca890 msgr2=0x7f11940caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 -- 172.21.15.78:0/2092169712 shutdown_connections 2024-02-19T15:00:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.388+0000 7f119aced640 1 -- 172.21.15.78:0/2092169712 wait complete. 2024-02-19T15:00:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.389+0000 7f119aced640 1 Processor -- start 2024-02-19T15:00:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.389+0000 7f119aced640 1 -- start start 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119aced640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11941eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119aced640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11941eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119aced640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1194158530 0x7f11940aa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119aced640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11941ee6b0 con 0x7f11940ac390 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119aced640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194156570 con 0x7f1194158530 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119aced640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11940c3b00 con 0x7f11940bdf20 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11941eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11941eead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50152/0 (socket says 172.21.15.78:50152) 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 -- 172.21.15.78:0/23174219 learned_addr learned my addr 172.21.15.78:0/23174219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1193fff640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11941eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f119928a640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1194158530 0x7f11940aa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 -- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 msgr2=0x7f11941eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11941eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 -- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1194158530 msgr2=0x7f11940aa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1194158530 0x7f11940aa9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 -- 172.21.15.78:0/23174219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840094d0 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.390+0000 7f1198a89640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11941eead0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f118402b830 tx=0x7f11840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.391+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11840338d0 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.391+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1184033a30 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.391+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11940c3de0 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.391+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184033de0 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.391+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11941ee2d0 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.392+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1194156330 con 0x7f11940ac390 2024-02-19T15:00:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.392+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f118403a040 con 0x7f11940ac390 2024-02-19T15:00:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.392+0000 7f11917fa640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118006b810 0x7f118006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.392+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11840ac080 con 0x7f11940ac390 2024-02-19T15:00:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.392+0000 7f1193fff640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118006b810 0x7f118006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.393+0000 7f1193fff640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118006b810 0x7f118006dcd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f11941ebbb0 tx=0x7f1188009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.397+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1184076a50 con 0x7f11940ac390 2024-02-19T15:00:58.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.603+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1194074990 con 0x7f11940ac390 2024-02-19T15:00:58.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.604+0000 7f11917fa640 1 -- 172.21.15.78:0/23174219 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4956 (secure 0 0 0) 0x7f118404d090 con 0x7f11940ac390 2024-02-19T15:00:58.607 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:58.607 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":190,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:00:50.657189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:00:58.607 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 190 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118006b810 msgr2=0x7f118006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118006b810 0x7f118006dcd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f11941ebbb0 tx=0x7f1188009190 comp rx=0 tx=0).stop 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 msgr2=0x7f11941eead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11941eead0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f118402b830 tx=0x7f11840046c0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 shutdown_connections 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f118006b810 0x7f118006dcd0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1194158530 0x7f11940aa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f11940bdf20 0x7f11941eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 --2- 172.21.15.78:0/23174219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11940ac390 0x7f11941eead0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 >> 172.21.15.78:0/23174219 conn(0x7f11940ca890 msgr2=0x7f11940c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 shutdown_connections 2024-02-19T15:00:58.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.606+0000 7f119aced640 1 -- 172.21.15.78:0/23174219 wait complete. 2024-02-19T15:00:58.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-02-19T15:00:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.720+0000 7fb823581640 1 Processor -- start 2024-02-19T15:00:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.720+0000 7fb823581640 1 -- start start 2024-02-19T15:00:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.720+0000 7fb823581640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb81c0c42e0 0x7fb81c158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb823581640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c0c34a0 con 0x7fb81c0c8500 2024-02-19T15:00:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb823581640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c06c8e0 con 0x7fb81c0c2f50 2024-02-19T15:00:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb823581640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81c06c090 con 0x7fb81c0c42e0 2024-02-19T15:00:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb82131d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb81c0c8500 0x7fb81c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48778/0 (socket says 172.21.15.78:48778) 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb82131d640 1 -- 172.21.15.78:0/524591355 learned_addr learned my addr 172.21.15.78:0/524591355 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb820b1c640 1 --2- 172.21.15.78:0/524591355 >> v2:172.21.15.150:3300/0 conn(0x7fb81c0c42e0 0x7fb81c158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61339894 0 0) 0x7fb81c0c34a0 con 0x7fb81c0c8500 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808003520 con 0x7fb81c0c8500 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1167877653 0 0) 0x7fb808003520 con 0x7fb81c0c8500 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 >> v2:172.21.15.150:3300/0 conn(0x7fb81c0c42e0 msgr2=0x7fb81c158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 --2- 172.21.15.78:0/524591355 >> v2:172.21.15.150:3300/0 conn(0x7fb81c0c42e0 0x7fb81c158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 >> v1:172.21.15.150:6789/0 conn(0x7fb81c0c2f50 legacy=0x7fb81c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c06cbc0 con 0x7fb81c0c8500 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fb818001670 con 0x7fb81c0c8500 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 >> v1:172.21.15.78:6789/0 conn(0x7fb81c0c8500 legacy=0x7fb81c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb808003970 0x7fb808003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8080047c0 0x7fb808004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808005180 0x7fb80800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80800c0f0 con 0x7fb808003970 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80800c260 con 0x7fb808005180 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80800c420 con 0x7fb8080047c0 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb82131d640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb808003970 0x7fb808003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb821b1e640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808005180 0x7fb80800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.721+0000 7fb820b1c640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8080047c0 0x7fb808004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb82131d640 1 -- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8080047c0 msgr2=0x7fb808004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb82131d640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8080047c0 0x7fb808004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb82131d640 1 -- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808005180 msgr2=0x7fb80800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb82131d640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808005180 0x7fb80800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb82131d640 1 -- 172.21.15.78:0/524591355 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808003520 con 0x7fb808003970 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb821b1e640 1 -- 172.21.15.78:0/524591355 reap_dead start 2024-02-19T15:00:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb820b1c640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8080047c0 0x7fb808004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb82131d640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb808003970 0x7fb808003d90 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fb81800c240 tx=0x7fb8080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28de347a0516467c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818005ad0 con 0x7fb808003970 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb818005c30 con 0x7fb808003970 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.722+0000 7fb813fff640 1 -- 172.21.15.78:0/524591355 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81802caf0 con 0x7fb808003970 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 -- 172.21.15.78:0/524591355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb808003970 msgr2=0x7fb808003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb808003970 0x7fb808003d90 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fb81800c240 tx=0x7fb8080057f0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 -- 172.21.15.78:0/524591355 shutdown_connections 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 --2- 172.21.15.78:0/524591355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb808003970 0x7fb808003d90 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 -- 172.21.15.78:0/524591355 >> 172.21.15.78:0/524591355 conn(0x7fb81c0aa300 msgr2=0x7fb81c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 -- 172.21.15.78:0/524591355 shutdown_connections 2024-02-19T15:00:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.723+0000 7fb823581640 1 -- 172.21.15.78:0/524591355 wait complete. 2024-02-19T15:00:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.724+0000 7fb823581640 1 Processor -- start 2024-02-19T15:00:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.724+0000 7fb823581640 1 -- start start 2024-02-19T15:00:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb808003970 0x7fb81c06fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 0x7fb81c14db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0ab770 0x7fb81c148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c06fa40 con 0x7fb81c158a30 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c06f410 con 0x7fb81c0ab770 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c1ee7d0 con 0x7fb808003970 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 0x7fb81c14db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 0x7fb81c14db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50180/0 (socket says 172.21.15.78:50180) 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 -- 172.21.15.78:0/509017083 learned_addr learned my addr 172.21.15.78:0/509017083 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb821b1e640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0ab770 0x7fb81c148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb82131d640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb808003970 0x7fb81c06fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 -- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb808003970 msgr2=0x7fb81c06fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb808003970 0x7fb81c06fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 -- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0ab770 msgr2=0x7fb81c148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0ab770 0x7fb81c148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 -- 172.21.15.78:0/509017083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c1eeab0 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb820b1c640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 0x7fb81c14db60 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fb80c011530 tx=0x7fb80c0119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c015870 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c0159d0 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.725+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c06f6d0 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.726+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c015d60 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.726+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c14e430 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.727+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c074990 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.727+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb80c01d400 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.727+0000 7fb811ffb640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb80006b810 0x7fb80006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.727+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb80c08a8f0 con 0x7fb81c158a30 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.727+0000 7fb82131d640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb80006b810 0x7fb80006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.728+0000 7fb82131d640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb80006b810 0x7fb80006dcd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb818034000 tx=0x7fb818009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:58.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.732+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb80c0552c0 con 0x7fb81c158a30 2024-02-19T15:00:58.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:58.941+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fb81c071ba0 con 0x7fb81c158a30 2024-02-19T15:00:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.387+0000 7fb811ffb640 1 -- 172.21.15.78:0/509017083 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v191) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb80c058980 con 0x7fb81c158a30 2024-02-19T15:00:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb80006b810 msgr2=0x7fb80006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb80006b810 0x7fb80006dcd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb818034000 tx=0x7fb818009030 comp rx=0 tx=0).stop 2024-02-19T15:00:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 msgr2=0x7fb81c14db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 0x7fb81c14db60 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fb80c011530 tx=0x7fb80c0119f0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 shutdown_connections 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb80006b810 0x7fb80006dcd0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81c158a30 0x7fb81c14db60 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb81c0ab770 0x7fb81c148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 --2- 172.21.15.78:0/509017083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb808003970 0x7fb81c06fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 >> 172.21.15.78:0/509017083 conn(0x7fb81c0aa300 msgr2=0x7fb81c154cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 shutdown_connections 2024-02-19T15:00:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.389+0000 7fb823581640 1 -- 172.21.15.78:0/509017083 wait complete. 2024-02-19T15:00:59.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:00:59.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.508+0000 7f7d49fa1640 1 Processor -- start 2024-02-19T15:00:59.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.508+0000 7f7d49fa1640 1 -- start start 2024-02-19T15:00:59.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d49fa1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d440c78a0 0x7f7d440c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:59.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d49fa1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d44072ee0 0x7f7d4414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:59.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d49fa1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d440c35c0 con 0x7f7d4414ef50 2024-02-19T15:00:59.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d49fa1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4414f920 con 0x7f7d440c78a0 2024-02-19T15:00:59.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d49fa1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440c38a0 con 0x7f7d44072ee0 2024-02-19T15:00:59.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d3affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7d4414ef50 0x7f7d440727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48794/0 (socket says 172.21.15.78:48794) 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d3affd640 1 -- 172.21.15.78:0/1035759438 learned_addr learned my addr 172.21.15.78:0/1035759438 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d437fe640 1 --2- 172.21.15.78:0/1035759438 >> v2:172.21.15.150:3300/0 conn(0x7f7d440c78a0 0x7f7d440c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.509+0000 7f7d43fff640 1 --2- 172.21.15.78:0/1035759438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d44072ee0 0x7f7d4414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d42ffd640 1 -- 172.21.15.78:0/1035759438 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777831012 0 0) 0x7f7d440c35c0 con 0x7f7d4414ef50 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d42ffd640 1 -- 172.21.15.78:0/1035759438 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d30003520 con 0x7f7d4414ef50 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d43fff640 1 -- 172.21.15.78:0/1035759438 >> v2:172.21.15.150:3300/0 conn(0x7f7d440c78a0 msgr2=0x7f7d440c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d43fff640 1 --2- 172.21.15.78:0/1035759438 >> v2:172.21.15.150:3300/0 conn(0x7f7d440c78a0 0x7f7d440c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d43fff640 1 -- 172.21.15.78:0/1035759438 >> v1:172.21.15.78:6789/0 conn(0x7f7d4414ef50 legacy=0x7f7d440727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d43fff640 1 -- 172.21.15.78:0/1035759438 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440c3b80 con 0x7f7d44072ee0 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d437fe640 1 --2- 172.21.15.78:0/1035759438 >> v2:172.21.15.150:3300/0 conn(0x7f7d440c78a0 0x7f7d440c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.510+0000 7f7d43fff640 1 --2- 172.21.15.78:0/1035759438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d44072ee0 0x7f7d4414f3e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009d90 tx=0x7f7d3c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4318ef19a790754b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d42ffd640 1 -- 172.21.15.78:0/1035759438 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c02cc50 con 0x7f7d44072ee0 2024-02-19T15:00:59.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d42ffd640 1 -- 172.21.15.78:0/1035759438 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c02cdb0 con 0x7f7d44072ee0 2024-02-19T15:00:59.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d42ffd640 1 -- 172.21.15.78:0/1035759438 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c034660 con 0x7f7d44072ee0 2024-02-19T15:00:59.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 -- 172.21.15.78:0/1035759438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d44072ee0 msgr2=0x7f7d4414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/1035759438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d44072ee0 0x7f7d4414f3e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009d90 tx=0x7f7d3c02bea0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 -- 172.21.15.78:0/1035759438 shutdown_connections 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/1035759438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d44072ee0 0x7f7d4414f3e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/1035759438 >> v2:172.21.15.150:3300/0 conn(0x7f7d440c78a0 0x7f7d440c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 -- 172.21.15.78:0/1035759438 >> 172.21.15.78:0/1035759438 conn(0x7f7d44066e70 msgr2=0x7f7d440a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 -- 172.21.15.78:0/1035759438 shutdown_connections 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.511+0000 7f7d49fa1640 1 -- 172.21.15.78:0/1035759438 wait complete. 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.512+0000 7f7d49fa1640 1 Processor -- start 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.512+0000 7f7d49fa1640 1 -- start start 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d49fa1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 0x7f7d441210d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d49fa1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d440c78a0 0x7f7d440c0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d49fa1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d4414ef50 0x7f7d440bd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d49fa1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d44122720 con 0x7f7d44072ee0 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d49fa1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440c1040 con 0x7f7d440c78a0 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d49fa1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440c0dd0 con 0x7f7d4414ef50 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 0x7f7d441210d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d437fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 0x7f7d441210d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50188/0 (socket says 172.21.15.78:50188) 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d437fe640 1 -- 172.21.15.78:0/3339075076 learned_addr learned my addr 172.21.15.78:0/3339075076 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d3affd640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d440c78a0 0x7f7d440c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d43fff640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d4414ef50 0x7f7d440bd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.513+0000 7f7d437fe640 1 -- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d4414ef50 msgr2=0x7f7d440bd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d437fe640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d4414ef50 0x7f7d440bd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d437fe640 1 -- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d440c78a0 msgr2=0x7f7d440c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d437fe640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d440c78a0 0x7f7d440c0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d437fe640 1 -- 172.21.15.78:0/3339075076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34009400 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d3affd640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d440c78a0 0x7f7d440c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d437fe640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 0x7f7d441210d0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f7d34010e20 tx=0x7f7d3400e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34009760 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34011860 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34011bf0 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c009a70 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.514+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d441eb430 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.515+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d44074990 con 0x7f7d44072ee0 2024-02-19T15:00:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.515+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7d34011d50 con 0x7f7d44072ee0 2024-02-19T15:00:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.516+0000 7f7d40ff9640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d2406b810 0x7f7d2406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:00:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.516+0000 7f7d3affd640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d2406b810 0x7f7d2406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:00:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.516+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d3408c8c0 con 0x7f7d44072ee0 2024-02-19T15:00:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.516+0000 7f7d3affd640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d2406b810 0x7f7d2406dcd0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c007fd0 tx=0x7f7d2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:00:59.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.520+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7d34057290 con 0x7f7d44072ee0 2024-02-19T15:00:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.726+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d440bdc50 con 0x7f7d44072ee0 2024-02-19T15:00:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.727+0000 7f7d40ff9640 1 -- 172.21.15.78:0/3339075076 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f7d3405a950 con 0x7f7d44072ee0 2024-02-19T15:00:59.730 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:00:59.730 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":192,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:00:59.390106+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:creating","state_seq":1,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:00:59.730 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 192 2024-02-19T15:00:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d2406b810 msgr2=0x7f7d2406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d2406b810 0x7f7d2406dcd0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c007fd0 tx=0x7f7d2c009040 comp rx=0 tx=0).stop 2024-02-19T15:00:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 msgr2=0x7f7d441210d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:00:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 0x7f7d441210d0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f7d34010e20 tx=0x7f7d3400e730 comp rx=0 tx=0).stop 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 shutdown_connections 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7d2406b810 0x7f7d2406dcd0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d4414ef50 0x7f7d440bd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d440c78a0 0x7f7d440c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 --2- 172.21.15.78:0/3339075076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d44072ee0 0x7f7d441210d0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 >> 172.21.15.78:0/3339075076 conn(0x7f7d44066e70 msgr2=0x7f7d4414faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 shutdown_connections 2024-02-19T15:00:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:00:59.729+0000 7f7d49fa1640 1 -- 172.21.15.78:0/3339075076 wait complete. 2024-02-19T15:01:00.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 192, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:00:59.390106+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:00.145 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:creating 2024-02-19T15:01:01.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:01.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.246+0000 7f55d09c9640 1 Processor -- start 2024-02-19T15:01:01.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.246+0000 7f55d09c9640 1 -- start start 2024-02-19T15:01:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55d09c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc1537a0 0x7f55cc0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55d09c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55cc156860 0x7f55cc156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55d09c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc1563b0 con 0x7f55cc14cd50 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55d09c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc153cd0 con 0x7f55cc156860 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55d09c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc06b6e0 con 0x7f55cc1537a0 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55cad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55cc156860 0x7f55cc156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55c9d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc1537a0 0x7f55cc0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55cad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55cc156860 0x7f55cc156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:58096/0 (socket says 172.21.15.78:58096) 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55ca59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f55cc14cd50 0x7f55cc0c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:40270/0 (socket says 172.21.15.78:40270) 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.247+0000 7f55cad9d640 1 -- 172.21.15.78:0/3637990191 learned_addr learned my addr 172.21.15.78:0/3637990191 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.248+0000 7f55cad9d640 1 -- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc1537a0 msgr2=0x7f55cc0c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.248+0000 7f55cad9d640 1 --2- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc1537a0 0x7f55cc0c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.248+0000 7f55cad9d640 1 -- 172.21.15.78:0/3637990191 >> v1:172.21.15.150:6789/0 conn(0x7f55cc14cd50 legacy=0x7f55cc0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.248+0000 7f55cad9d640 1 -- 172.21.15.78:0/3637990191 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc157230 con 0x7f55cc156860 2024-02-19T15:01:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.248+0000 7f55c9d9b640 1 --2- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc1537a0 0x7f55cc0c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.248+0000 7f55cad9d640 1 --2- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc156cf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009870 tx=0x7f55bc02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f169cd3eeca2269 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55c959a640 1 -- 172.21.15.78:0/3637990191 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0317c0 con 0x7f55cc156860 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55c959a640 1 -- 172.21.15.78:0/3637990191 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc031920 con 0x7f55cc156860 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55c959a640 1 -- 172.21.15.78:0/3637990191 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc031cb0 con 0x7f55cc156860 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 -- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 msgr2=0x7f55cc156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 --2- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc156cf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009870 tx=0x7f55bc02b860 comp rx=0 tx=0).stop 2024-02-19T15:01:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 -- 172.21.15.78:0/3637990191 shutdown_connections 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 --2- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc1537a0 0x7f55cc0c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 --2- 172.21.15.78:0/3637990191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc156cf0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 -- 172.21.15.78:0/3637990191 >> 172.21.15.78:0/3637990191 conn(0x7f55cc066e70 msgr2=0x7f55cc0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 -- 172.21.15.78:0/3637990191 shutdown_connections 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.249+0000 7f55d09c9640 1 -- 172.21.15.78:0/3637990191 wait complete. 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.250+0000 7f55d09c9640 1 Processor -- start 2024-02-19T15:01:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 -- start start 2024-02-19T15:01:01.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc14cd50 0x7f55cc1ef240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:01.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 0x7f55cc0c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:01.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:01.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0aae40 con 0x7f55cc1537a0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0c1680 con 0x7f55cc156860 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55d09c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc1ef780 con 0x7f55cc14cd50 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55c9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 0x7f55cc0c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55ca59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc14cd50 0x7f55cc1ef240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55c9d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 0x7f55cc0c0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50190/0 (socket says 172.21.15.78:50190) 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55c9d9b640 1 -- 172.21.15.78:0/2937935645 learned_addr learned my addr 172.21.15.78:0/2937935645 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.251+0000 7f55cad9d640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55c9d9b640 1 -- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc14cd50 msgr2=0x7f55cc1ef240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55c9d9b640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc14cd50 0x7f55cc1ef240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55c9d9b640 1 -- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 msgr2=0x7f55cc149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55c9d9b640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55c9d9b640 1 -- 172.21.15.78:0/2937935645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c0009a70 con 0x7f55cc1537a0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55c9d9b640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 0x7f55cc0c0d40 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f55c000efc0 tx=0x7f55c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0009cd0 con 0x7f55cc1537a0 2024-02-19T15:01:01.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c000f040 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc0094d0 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00148c0 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.252+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc14e100 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.253+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc072560 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.253+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f55c0014a20 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.254+0000 7f55baffd640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55ac06b810 0x7f55ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.254+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55c008ae10 con 0x7f55cc1537a0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.254+0000 7f55ca59c640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55ac06b810 0x7f55ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.254+0000 7f55ca59c640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55ac06b810 0x7f55ac06dcd0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f55c4007fd0 tx=0x7f55c4007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:01.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.258+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f55c00557e0 con 0x7f55cc1537a0 2024-02-19T15:01:01.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.475+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55cc072b20 con 0x7f55cc1537a0 2024-02-19T15:01:01.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.475+0000 7f55baffd640 1 -- 172.21.15.78:0/2937935645 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+4972 (secure 0 0 0) 0x7f55c0058ea0 con 0x7f55cc1537a0 2024-02-19T15:01:01.478 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:01.479 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":193,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:00:59.408846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:active","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:01.479 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 193 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55ac06b810 msgr2=0x7f55ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55ac06b810 0x7f55ac06dcd0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f55c4007fd0 tx=0x7f55c4007e80 comp rx=0 tx=0).stop 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 msgr2=0x7f55cc0c0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 0x7f55cc0c0d40 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f55c000efc0 tx=0x7f55c000c490 comp rx=0 tx=0).stop 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 shutdown_connections 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55ac06b810 0x7f55ac06dcd0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55cc1537a0 0x7f55cc0c0d40 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55cc156860 0x7f55cc149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 --2- 172.21.15.78:0/2937935645 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55cc14cd50 0x7f55cc1ef240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.477+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 >> 172.21.15.78:0/2937935645 conn(0x7f55cc066e70 msgr2=0x7f55cc0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.478+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 shutdown_connections 2024-02-19T15:01:01.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:01.478+0000 7f55d09c9640 1 -- 172.21.15.78:0/2937935645 wait complete. 2024-02-19T15:01:01.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 193, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:00:59.408846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:01.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-02-19T15:01:01.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-19T15:01:02.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 Processor -- start 2024-02-19T15:01:02.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 -- start start 2024-02-19T15:01:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd8780add70 0x7fd8780c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8780c2f60 0x7fd87806eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8780ae670 con 0x7fd87806f530 2024-02-19T15:01:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8780716b0 con 0x7fd8780add70 2024-02-19T15:01:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.000+0000 7fd880607640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780c33a0 con 0x7fd8780c2f60 2024-02-19T15:01:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd8780add70 0x7fd8780c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd8780add70 0x7fd8780c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50206/0 (socket says 172.21.15.78:50206) 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 -- 172.21.15.78:0/3845728632 learned_addr learned my addr 172.21.15.78:0/3845728632 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87dba2640 1 --2- 172.21.15.78:0/3845728632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8780c2f60 0x7fd87806eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 -- 172.21.15.78:0/3845728632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8780c2f60 msgr2=0x7fd87806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/3845728632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8780c2f60 0x7fd87806eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 -- 172.21.15.78:0/3845728632 >> v1:172.21.15.150:6789/0 conn(0x7fd87806f530 legacy=0x7fd8780705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.001+0000 7fd87e3a3640 1 -- 172.21.15.78:0/3845728632 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878153780 con 0x7fd8780add70 2024-02-19T15:01:02.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.002+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/3845728632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd8780c4980 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd874009870 tx=0x7fd87402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e4c9c64965ab92 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.002+0000 7fd87d3a1640 1 -- 172.21.15.78:0/3845728632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874031640 con 0x7fd8780add70 2024-02-19T15:01:02.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.002+0000 7fd87d3a1640 1 -- 172.21.15.78:0/3845728632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8740317a0 con 0x7fd8780add70 2024-02-19T15:01:02.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.002+0000 7fd87d3a1640 1 -- 172.21.15.78:0/3845728632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874031b30 con 0x7fd8780add70 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 -- 172.21.15.78:0/3845728632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 msgr2=0x7fd8780c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 --2- 172.21.15.78:0/3845728632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd8780c4980 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd874009870 tx=0x7fd87402b8f0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 -- 172.21.15.78:0/3845728632 shutdown_connections 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 --2- 172.21.15.78:0/3845728632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8780c2f60 0x7fd87806eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 --2- 172.21.15.78:0/3845728632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd8780c4980 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 -- 172.21.15.78:0/3845728632 >> 172.21.15.78:0/3845728632 conn(0x7fd8780c85e0 msgr2=0x7fd8780c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 -- 172.21.15.78:0/3845728632 shutdown_connections 2024-02-19T15:01:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.003+0000 7fd880607640 1 -- 172.21.15.78:0/3845728632 wait complete. 2024-02-19T15:01:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.004+0000 7fd880607640 1 Processor -- start 2024-02-19T15:01:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.004+0000 7fd880607640 1 -- start start 2024-02-19T15:01:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd87806f530 0x7fd8780bde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd878156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8780c2f60 0x7fd8781ebdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878156550 con 0x7fd8780add70 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878156200 con 0x7fd8780c2f60 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd878154430 con 0x7fd87806f530 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd878156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd878156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50214/0 (socket says 172.21.15.78:50214) 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 -- 172.21.15.78:0/4224828807 learned_addr learned my addr 172.21.15.78:0/4224828807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd87806f530 0x7fd8780bde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87eba4640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8780c2f60 0x7fd8781ebdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 -- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd87806f530 msgr2=0x7fd8780bde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd87806f530 0x7fd8780bde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 -- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8780c2f60 msgr2=0x7fd8781ebdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8780c2f60 0x7fd8781ebdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 -- 172.21.15.78:0/4224828807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740094d0 con 0x7fd8780add70 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd87dba2640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd878156970 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e870 tx=0x7fd86800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd86effd640 1 -- 172.21.15.78:0/4224828807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8680094f0 con 0x7fd8780add70 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd86effd640 1 -- 172.21.15.78:0/4224828807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86800c940 con 0x7fd8780add70 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.005+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8780be3b0 con 0x7fd8780add70 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.006+0000 7fd86effd640 1 -- 172.21.15.78:0/4224828807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86800ccf0 con 0x7fd8780add70 2024-02-19T15:01:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.006+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8781549b0 con 0x7fd8780add70 2024-02-19T15:01:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.007+0000 7fd86effd640 1 -- 172.21.15.78:0/4224828807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd868013470 con 0x7fd8780add70 2024-02-19T15:01:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.008+0000 7fd86effd640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd85c06b810 0x7fd85c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.008+0000 7fd86effd640 1 -- 172.21.15.78:0/4224828807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd86808a2d0 con 0x7fd8780add70 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.008+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd85c06b810 0x7fd85c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.009+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd85c06b810 0x7fd85c06dcd0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fd874036000 tx=0x7fd87403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.009+0000 7fd8831e5740 1 Processor -- start 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.009+0000 7fd8831e5740 1 -- start start 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 0x55dfd5b9eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b63860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd8831e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5b1ecd0 con 0x55dfd5b64ae0 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5ad1f20 con 0x55dfd5b9e740 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5acf450 con 0x55dfd5ba18b0 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 0x55dfd5b9eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 0x55dfd5b9eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50224/0 (socket says 172.21.15.78:50224) 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 -- 172.21.15.78:0/2717584616 learned_addr learned my addr 172.21.15.78:0/2717584616 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87eba4640 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b63860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 -- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 msgr2=0x55dfd5b63860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b63860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 -- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 msgr2=0x55dfd5b9f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 -- 172.21.15.78:0/2717584616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfd5a7ff20 con 0x55dfd5b64ae0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd87dba2640 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 0x55dfd5b9eb20 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fd868088670 tx=0x7fd868058420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de14d4e9dafcbdee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd85bfff640 1 -- 172.21.15.78:0/2717584616 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868061060 con 0x55dfd5b64ae0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd85bfff640 1 -- 172.21.15.78:0/2717584616 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86808a0e0 con 0x55dfd5b64ae0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.010+0000 7fd85bfff640 1 -- 172.21.15.78:0/2717584616 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868055070 con 0x55dfd5b64ae0 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 -- 172.21.15.78:0/2717584616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 msgr2=0x55dfd5b9eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 0x55dfd5b9eb20 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fd868088670 tx=0x7fd868058420 comp rx=0 tx=0).stop 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 -- 172.21.15.78:0/2717584616 shutdown_connections 2024-02-19T15:01:02.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b64ae0 0x55dfd5b9eb20 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b63860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2717584616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 -- 172.21.15.78:0/2717584616 >> 172.21.15.78:0/2717584616 conn(0x55dfd5b60120 msgr2=0x55dfd5b84b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 -- 172.21.15.78:0/2717584616 shutdown_connections 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.011+0000 7fd8831e5740 1 -- 172.21.15.78:0/2717584616 wait complete. 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.012+0000 7fd8831e5740 1 -- start start 2024-02-19T15:01:02.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.012+0000 7fd8831e5740 1 Processor -- start 2024-02-19T15:01:02.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.012+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b62400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5b9c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5acac30 con 0x55dfd5b81f80 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5a6fd30 con 0x55dfd5b9e740 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dfd579e310 con 0x55dfd5ba18b0 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 10 client.0 _ll_register_callbacks cb 0x55dfd5a77d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5b9c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5b9c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50236/0 (socket says 172.21.15.78:50236) 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 -- 172.21.15.78:0/1225697912 learned_addr learned my addr 172.21.15.78:0/1225697912 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87dba2640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b62400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 -- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 msgr2=0x55dfd5b9c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 -- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 msgr2=0x55dfd5b62400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b62400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 -- 172.21.15.78:0/1225697912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8781549b0 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd87eba4640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5b9c810 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fd870012660 tx=0x7fd870012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a8a50e7d3728f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8597fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8597fa640 1 -- 172.21.15.78:0/1225697912 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87001a030 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8597fa640 1 -- 172.21.15.78:0/1225697912 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd87001b070 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dfd57ca6c0 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8597fa640 1 -- 172.21.15.78:0/1225697912 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd870006b50 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 10 client.10920 fetch_fsmap learned FSMap version 193 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.013+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) v3 -- 0x55dfd5ac92b0 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8597fa640 1 -- 172.21.15.78:0/1225697912 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd870025020 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8597fa640 1 -- 172.21.15.78:0/1225697912 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 193) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fd870006cb0 con 0x55dfd5b81f80 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8831e5740 10 client.10920 fetch_fsmap finished waiting for FSMap version 193 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8831e5740 10 client.10920 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8831e5740 20 client.10920 populate_metadata read hostname 'smithi078' 2024-02-19T15:01:02.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8831e5740 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5bb3180 0x55dfd5bb3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8831e5740 4 client.10920 mds_command: new command op to 10719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 --> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dfd581b270 con 0x55dfd5bb3180 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5bb3180 0x55dfd5bb3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd8597fa640 10 client.10920 ms_handle_connect on v2:172.21.15.78:6836/159409097 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.014+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5bb3180 0x55dfd5bb3560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d544103879fd0514 server_cookie=89eb8dc567a5649b in_seq=0 out_seq=0 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8597fa640 1 -- 172.21.15.78:0/1225697912 <== mds.1 v2:172.21.15.78:6836/159409097 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dfd581b270 con 0x55dfd5bb3180 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8597fa640 10 client.10920 handle_command_reply: tid=0 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 client.10920 shutdown 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 msgr2=0x55dfd5b9c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5b9c810 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fd870012660 tx=0x7fd870012b20 comp rx=0 tx=0).stop 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 shutdown_connections 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5bb3180 0x55dfd5bb3560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5b9c810 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8597fa640 0 client.10920 ms_handle_reset on v2:172.21.15.78:6836/159409097 2024-02-19T15:01:02.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 --2- 172.21.15.78:0/1225697912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b62400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 >> 172.21.15.78:0/1225697912 conn(0x55dfd5b64ae0 msgr2=0x55dfd5b9f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 shutdown_connections 2024-02-19T15:01:02.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 1 -- 172.21.15.78:0/1225697912 wait complete. 2024-02-19T15:01:02.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.015+0000 7fd8831e5740 20 client.10920 trim_cache size 0 max 16384 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 Processor -- start 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 -- start start 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5bb95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5a50490 con 0x55dfd5b81f80 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5b9aaf0 con 0x55dfd5b9e740 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5b82410 con 0x55dfd5ba18b0 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd87e3a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5bb95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd87e3a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5bb95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50240/0 (socket says 172.21.15.78:50240) 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd87e3a3640 1 -- 172.21.15.78:0/2479240381 learned_addr learned my addr 172.21.15.78:0/2479240381 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.032+0000 7fd87dba2640 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87eba4640 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87e3a3640 1 -- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 msgr2=0x55dfd5b9c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87e3a3640 1 -- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 msgr2=0x55dfd5b9bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87e3a3640 1 -- 172.21.15.78:0/2479240381 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfd5a65a20 con 0x55dfd5b81f80 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87eba4640 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5bb95b0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fd874009990 tx=0x7fd874030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af7e54045f8b3df server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd85b7fe640 1 -- 172.21.15.78:0/2479240381 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87404b030 con 0x55dfd5b81f80 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd85b7fe640 1 -- 172.21.15.78:0/2479240381 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87404c040 con 0x55dfd5b81f80 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.033+0000 7fd85b7fe640 1 -- 172.21.15.78:0/2479240381 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87403aaf0 con 0x55dfd5b81f80 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 -- 172.21.15.78:0/2479240381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 msgr2=0x55dfd5bb95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5bb95b0 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7fd874009990 tx=0x7fd874030000 comp rx=0 tx=0).stop 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 -- 172.21.15.78:0/2479240381 shutdown_connections 2024-02-19T15:01:02.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5ba18b0 0x55dfd5b9c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b9bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2479240381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5b81f80 0x55dfd5bb95b0 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 -- 172.21.15.78:0/2479240381 >> 172.21.15.78:0/2479240381 conn(0x55dfd5b64ae0 msgr2=0x55dfd5b97110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 -- 172.21.15.78:0/2479240381 shutdown_connections 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 -- 172.21.15.78:0/2479240381 wait complete. 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 -- start start 2024-02-19T15:01:02.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.034+0000 7fd8831e5740 1 Processor -- start 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5b81f80 0x55dfd5c0f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b61240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 0x55dfd5b61780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5bb99e0 con 0x55dfd5ba18b0 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5b61cc0 con 0x55dfd5b9e740 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dfd5b84e00 con 0x55dfd5b81f80 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd8831e5740 10 client.0 _ll_register_callbacks cb 0x7fd870001800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd87eba4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 0x55dfd5b61780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd87eba4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 0x55dfd5b61780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50254/0 (socket says 172.21.15.78:50254) 2024-02-19T15:01:02.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd87eba4640 1 -- 172.21.15.78:0/2592705433 learned_addr learned my addr 172.21.15.78:0/2592705433 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd87dba2640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b61240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.035+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5b81f80 0x55dfd5c0f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87eba4640 1 -- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5b81f80 msgr2=0x55dfd5c0f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87eba4640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5b81f80 0x55dfd5c0f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87eba4640 1 -- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 msgr2=0x55dfd5b61240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87eba4640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b61240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87eba4640 1 -- 172.21.15.78:0/2592705433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd878153780 con 0x55dfd5ba18b0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87dba2640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b61240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87eba4640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 0x55dfd5b61780 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fd870000f80 tx=0x7fd8700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd8ef033905fbe2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8597fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8597fa640 1 -- 172.21.15.78:0/2592705433 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87001a040 con 0x55dfd5ba18b0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8597fa640 1 -- 172.21.15.78:0/2592705433 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd870026080 con 0x55dfd5ba18b0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd86808a0e0 con 0x55dfd5ba18b0 2024-02-19T15:01:02.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8597fa640 1 -- 172.21.15.78:0/2592705433 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd870010610 con 0x55dfd5ba18b0 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 10 client.10926 fetch_fsmap learned FSMap version 193 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) v3 -- 0x55dfd5a8b780 con 0x55dfd5ba18b0 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8597fa640 1 -- 172.21.15.78:0/2592705433 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86808a0e0 con 0x55dfd5ba18b0 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8597fa640 1 -- 172.21.15.78:0/2592705433 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 193) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fd870010820 con 0x55dfd5ba18b0 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 10 client.10926 fetch_fsmap finished waiting for FSMap version 193 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 10 client.10926 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 20 client.10926 populate_metadata read hostname 'smithi078' 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5b84620 0x55dfd5c11ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 4 client.10926 mds_command: new command op to 10719 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 --> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dfd5807610 con 0x55dfd5b84620 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.036+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5b84620 0x55dfd5c11ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8597fa640 10 client.10926 ms_handle_connect on v2:172.21.15.78:6836/159409097 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd87e3a3640 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5b84620 0x55dfd5c11ce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fd2fbfcc9f0027d server_cookie=8ed3ceb72c03c3f2 in_seq=0 out_seq=0 2024-02-19T15:01:02.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8597fa640 1 -- 172.21.15.78:0/2592705433 <== mds.1 v2:172.21.15.78:6836/159409097 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55dfd5807610 con 0x55dfd5b84620 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8597fa640 10 client.10926 handle_command_reply: tid=0 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 client.10926 shutdown 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 msgr2=0x55dfd5b61780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 0x55dfd5b61780 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fd870000f80 tx=0x7fd8700014a0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 shutdown_connections 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:6836/159409097,v1:172.21.15.78:6837/159409097] conn(0x55dfd5b84620 0x55dfd5c11ce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dfd5ba18b0 0x55dfd5b61780 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dfd5b9e740 0x55dfd5b61240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 --2- 172.21.15.78:0/2592705433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55dfd5b81f80 0x55dfd5c0f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8597fa640 0 client.10926 ms_handle_reset on v2:172.21.15.78:6836/159409097 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 >> 172.21.15.78:0/2592705433 conn(0x55dfd5b64ae0 msgr2=0x55dfd5b97060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 shutdown_connections 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 1 -- 172.21.15.78:0/2592705433 wait complete. 2024-02-19T15:01:02.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.037+0000 7fd8831e5740 20 client.10926 trim_cache size 0 max 16384 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 10719, 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 2.643463637, 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 193, 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 213, 2024-02-19T15:01:02.047 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 213, 2024-02-19T15:01:02.048 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17.851747003 2024-02-19T15:01:02.048 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:01:02.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd85c06b810 msgr2=0x7fd85c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd85c06b810 0x7fd85c06dcd0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fd874036000 tx=0x7fd87403c040 comp rx=0 tx=0).stop 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 msgr2=0x7fd878156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd878156970 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fd86800e870 tx=0x7fd86800ed30 comp rx=0 tx=0).stop 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 shutdown_connections 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd85c06b810 0x7fd85c06dcd0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8780add70 0x7fd878156970 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8780c2f60 0x7fd8781ebdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 --2- 172.21.15.78:0/4224828807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd87806f530 0x7fd8780bde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 >> 172.21.15.78:0/4224828807 conn(0x7fd8780c85e0 msgr2=0x7fd87814fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 shutdown_connections 2024-02-19T15:01:02.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.038+0000 7fd880607640 1 -- 172.21.15.78:0/4224828807 wait complete. 2024-02-19T15:01:02.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-19T15:01:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.154+0000 7fd230b0c640 1 Processor -- start 2024-02-19T15:01:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd230b0c640 1 -- start start 2024-02-19T15:01:02.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd230b0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c072ee0 0x7fd22c14f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd230b0c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0c35c0 con 0x7fd22c0c78a0 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd230b0c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0c38a0 con 0x7fd22c14ef50 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd230b0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c4bc0 con 0x7fd22c072ee0 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd22a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd22c0c78a0 0x7fd22c0c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48804/0 (socket says 172.21.15.78:48804) 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd22a59c640 1 -- 172.21.15.78:0/2899149034 learned_addr learned my addr 172.21.15.78:0/2899149034 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.155+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/2899149034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c072ee0 0x7fd22c14f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667078454 0 0) 0x7fd22c0c38a0 con 0x7fd22c14ef50 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd218003520 con 0x7fd22c14ef50 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496783878 0 0) 0x7fd22c0c35c0 con 0x7fd22c0c78a0 2024-02-19T15:01:02.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22c0c38a0 con 0x7fd22c0c78a0 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd22ad9d640 1 -- 172.21.15.78:0/2899149034 >> v1:172.21.15.150:6789/0 conn(0x7fd22c14ef50 legacy=0x7fd22c0727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd22ad9d640 1 -- 172.21.15.78:0/2899149034 >> v1:172.21.15.78:6789/0 conn(0x7fd22c0c78a0 legacy=0x7fd22c0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.156+0000 7fd22ad9d640 1 -- 172.21.15.78:0/2899149034 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0c3b80 con 0x7fd22c072ee0 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/2899149034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c072ee0 0x7fd22c14f3e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd224009dc0 tx=0x7fd22402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2dbd5f172264c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22402ccd0 con 0x7fd22c072ee0 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22402ce30 con 0x7fd22c072ee0 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd229d9b640 1 -- 172.21.15.78:0/2899149034 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224034640 con 0x7fd22c072ee0 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 -- 172.21.15.78:0/2899149034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c072ee0 msgr2=0x7fd22c14f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2899149034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c072ee0 0x7fd22c14f3e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd224009dc0 tx=0x7fd22402bf20 comp rx=0 tx=0).stop 2024-02-19T15:01:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 -- 172.21.15.78:0/2899149034 shutdown_connections 2024-02-19T15:01:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2899149034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c072ee0 0x7fd22c14f3e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 -- 172.21.15.78:0/2899149034 >> 172.21.15.78:0/2899149034 conn(0x7fd22c066e70 msgr2=0x7fd22c0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 -- 172.21.15.78:0/2899149034 shutdown_connections 2024-02-19T15:01:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.157+0000 7fd230b0c640 1 -- 172.21.15.78:0/2899149034 wait complete. 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 Processor -- start 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 -- start start 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 0x7fd22c1515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd22c0c78a0 0x7fd22c1ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c14ef50 0x7fd22c0be8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c11c0 con 0x7fd22c072ee0 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c0640 con 0x7fd22c0c78a0 2024-02-19T15:01:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd230b0c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c0920 con 0x7fd22c14ef50 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd22a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 0x7fd22c1515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.159+0000 7fd22a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 0x7fd22c1515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50264/0 (socket says 172.21.15.78:50264) 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 -- 172.21.15.78:0/2063663914 learned_addr learned my addr 172.21.15.78:0/2063663914 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd223fff640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd22c0c78a0 0x7fd22c1ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c14ef50 0x7fd22c0be8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 -- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c14ef50 msgr2=0x7fd22c0be8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c14ef50 0x7fd22c0be8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 -- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd22c0c78a0 msgr2=0x7fd22c1ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd22c0c78a0 0x7fd22c1ebd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 -- 172.21.15.78:0/2063663914 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd224009a70 con 0x7fd22c072ee0 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd22a59c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 0x7fd22c1515a0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001eb0 tx=0x7fd21c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd222ffd640 1 -- 172.21.15.78:0/2063663914 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c005ce0 con 0x7fd22c072ee0 2024-02-19T15:01:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd222ffd640 1 -- 172.21.15.78:0/2063663914 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c005e40 con 0x7fd22c072ee0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd222ffd640 1 -- 172.21.15.78:0/2063663914 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0072b0 con 0x7fd22c072ee0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c0c0c00 con 0x7fd22c072ee0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.160+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c1eb7e0 con 0x7fd22c072ee0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.162+0000 7fd222ffd640 1 -- 172.21.15.78:0/2063663914 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd21c007410 con 0x7fd22c072ee0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.162+0000 7fd222ffd640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd20c06b810 0x7fd20c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.162+0000 7fd222ffd640 1 -- 172.21.15.78:0/2063663914 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd21c089800 con 0x7fd22c072ee0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.162+0000 7fd223fff640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd20c06b810 0x7fd20c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.163+0000 7fd223fff640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd20c06b810 0x7fd20c06dcd0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fd214007fd0 tx=0x7fd214009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.163+0000 7fd2337c8740 1 Processor -- start 2024-02-19T15:01:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.163+0000 7fd2337c8740 1 -- start start 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e29450 0x561fd6e69720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e69c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e27fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd2337c8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6de8cd0 con 0x561fd6e2eae0 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6d9bf20 con 0x561fd6e29450 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd6d99450 con 0x561fd6e2dc20 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e69c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e69c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50276/0 (socket says 172.21.15.78:50276) 2024-02-19T15:01:02.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 -- 172.21.15.78:0/565912352 learned_addr learned my addr 172.21.15.78:0/565912352 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd22a59c640 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e29450 0x561fd6e69720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e27fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 -- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 msgr2=0x561fd6e27fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e27fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 -- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e29450 msgr2=0x561fd6e69720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e29450 0x561fd6e69720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.164+0000 7fd223fff640 1 -- 172.21.15.78:0/565912352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fd6d49f20 con 0x561fd6e2eae0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd223fff640 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e69c60 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fd214007a00 tx=0x7fd214012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0100fe47ce4a75 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd20bfff640 1 -- 172.21.15.78:0/565912352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2140149a0 con 0x561fd6e2eae0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd20bfff640 1 -- 172.21.15.78:0/565912352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd214014b00 con 0x561fd6e2eae0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd20bfff640 1 -- 172.21.15.78:0/565912352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214014e70 con 0x561fd6e2eae0 2024-02-19T15:01:02.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 -- 172.21.15.78:0/565912352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 msgr2=0x561fd6e69c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e69c60 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fd214007a00 tx=0x7fd214012000 comp rx=0 tx=0).stop 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 -- 172.21.15.78:0/565912352 shutdown_connections 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e69c60 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e27fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 --2- 172.21.15.78:0/565912352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e29450 0x561fd6e69720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.165+0000 7fd2337c8740 1 -- 172.21.15.78:0/565912352 >> 172.21.15.78:0/565912352 conn(0x561fd6e2a120 msgr2=0x561fd6e4eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.166+0000 7fd2337c8740 1 -- 172.21.15.78:0/565912352 shutdown_connections 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.166+0000 7fd2337c8740 1 -- 172.21.15.78:0/565912352 wait complete. 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.166+0000 7fd2337c8740 1 -- start start 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.166+0000 7fd2337c8740 1 Processor -- start 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e7dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e2eae0 0x561fd6e51400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e51940 0x561fd6e4e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6d94c30 con 0x561fd6e2dc20 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6d39d30 con 0x561fd6e2eae0 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd6a68310 con 0x561fd6e51940 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd2337c8740 10 client.0 _ll_register_callbacks cb 0x561fd6d41d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd22a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e7dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd22a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e7dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50278/0 (socket says 172.21.15.78:50278) 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd22a59c640 1 -- 172.21.15.78:0/2568089063 learned_addr learned my addr 172.21.15.78:0/2568089063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e51940 0x561fd6e4e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.167+0000 7fd223fff640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e2eae0 0x561fd6e51400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd22a59c640 1 -- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e51940 msgr2=0x561fd6e4e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd22a59c640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e51940 0x561fd6e4e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd22a59c640 1 -- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e2eae0 msgr2=0x561fd6e51400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd22a59c640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e2eae0 0x561fd6e51400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd22a59c640 1 -- 172.21.15.78:0/2568089063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd218003520 con 0x561fd6e2dc20 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd22a59c640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e7dee0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fd21c000f60 tx=0x7fd21c060590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0528166906b83 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2097fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2097fa640 1 -- 172.21.15.78:0/2568089063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c06f070 con 0x561fd6e2dc20 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2097fa640 1 -- 172.21.15.78:0/2568089063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd21c06be40 con 0x561fd6e2dc20 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561fd6a946c0 con 0x561fd6e2dc20 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2097fa640 1 -- 172.21.15.78:0/2568089063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd21c0562f0 con 0x561fd6e2dc20 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 10 client.10938 fetch_fsmap learned FSMap version 193 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) v3 -- 0x561fd6d932b0 con 0x561fd6e2dc20 2024-02-19T15:01:02.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2097fa640 1 -- 172.21.15.78:0/2568089063 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c073020 con 0x561fd6e2dc20 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2097fa640 1 -- 172.21.15.78:0/2568089063 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 193) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fd21c060100 con 0x561fd6e2dc20 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 10 client.10938 fetch_fsmap finished waiting for FSMap version 193 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 10 client.10938 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 20 client.10938 populate_metadata read hostname 'smithi078' 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e81300 0x561fd6e816e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 4 client.10938 mds_command: new command op to 10731 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.168+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 --> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fd6ae5270 con 0x561fd6e81300 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.169+0000 7fd223fff640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e81300 0x561fd6e816e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.169+0000 7fd2097fa640 10 client.10938 ms_handle_connect on v2:172.21.15.150:6833/1952320195 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.169+0000 7fd223fff640 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e81300 0x561fd6e816e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a860638f7ed1efe2 server_cookie=d46a749fe07700a0 in_seq=0 out_seq=0 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.169+0000 7fd2097fa640 1 -- 172.21.15.78:0/2568089063 <== mds.0 v2:172.21.15.150:6833/1952320195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561fd6ae5270 con 0x561fd6e81300 2024-02-19T15:01:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.169+0000 7fd2097fa640 10 client.10938 handle_command_reply: tid=0 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.169+0000 7fd2337c8740 1 client.10938 shutdown 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 msgr2=0x561fd6e7dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e7dee0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fd21c000f60 tx=0x7fd21c060590 comp rx=0 tx=0).stop 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 shutdown_connections 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e81300 0x561fd6e816e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e51940 0x561fd6e4e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e2eae0 0x561fd6e51400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 --2- 172.21.15.78:0/2568089063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e7dee0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2097fa640 0 client.10938 ms_handle_reset on v2:172.21.15.150:6833/1952320195 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 >> 172.21.15.78:0/2568089063 conn(0x561fd6e29450 msgr2=0x561fd6e6b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 shutdown_connections 2024-02-19T15:01:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 1 -- 172.21.15.78:0/2568089063 wait complete. 2024-02-19T15:01:02.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.170+0000 7fd2337c8740 20 client.10938 trim_cache size 0 max 16384 2024-02-19T15:01:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.186+0000 7fd2337c8740 1 Processor -- start 2024-02-19T15:01:02.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.186+0000 7fd2337c8740 1 -- start start 2024-02-19T15:01:02.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e818d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2eae0 0x561fd6e27ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e27ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd2337c8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6d1a490 con 0x561fd6e2dc20 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6d93d30 con 0x561fd6e51940 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd6e531a0 con 0x561fd6e2eae0 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e818d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e818d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50284/0 (socket says 172.21.15.78:50284) 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 -- 172.21.15.78:0/1141189216 learned_addr learned my addr 172.21.15.78:0/1141189216 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd223fff640 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2eae0 0x561fd6e27ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e27ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 -- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2eae0 msgr2=0x561fd6e27ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2eae0 0x561fd6e27ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 -- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 msgr2=0x561fd6e27ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e27ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 -- 172.21.15.78:0/1141189216 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fd6e6a240 con 0x561fd6e2dc20 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e27ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.187+0000 7fd22a59c640 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e818d0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00ce60 tx=0x7fd21c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851adafb37d411a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd20b7fe640 1 -- 172.21.15.78:0/1141189216 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c06f040 con 0x561fd6e2dc20 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd20b7fe640 1 -- 172.21.15.78:0/1141189216 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c073030 con 0x561fd6e2dc20 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd20b7fe640 1 -- 172.21.15.78:0/1141189216 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c00e040 con 0x561fd6e2dc20 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 -- 172.21.15.78:0/1141189216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 msgr2=0x561fd6e818d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e818d0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00ce60 tx=0x7fd21c014000 comp rx=0 tx=0).stop 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 -- 172.21.15.78:0/1141189216 shutdown_connections 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e27ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2eae0 0x561fd6e27ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 --2- 172.21.15.78:0/1141189216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2dc20 0x561fd6e818d0 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 -- 172.21.15.78:0/1141189216 >> 172.21.15.78:0/1141189216 conn(0x561fd6e29450 msgr2=0x561fd6e23660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 -- 172.21.15.78:0/1141189216 shutdown_connections 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.188+0000 7fd2337c8740 1 -- 172.21.15.78:0/1141189216 wait complete. 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.189+0000 7fd2337c8740 1 -- start start 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.189+0000 7fd2337c8740 1 Processor -- start 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e84490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e90ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e6d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6e6dd00 con 0x561fd6e2eae0 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd6e6cdb0 con 0x561fd6e51940 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fd6ee1080 con 0x561fd6e2dc20 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 10 client.0 _ll_register_callbacks cb 0x7fd21c057890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e90ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e90ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50296/0 (socket says 172.21.15.78:50296) 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 -- 172.21.15.78:0/3142092329 learned_addr learned my addr 172.21.15.78:0/3142092329 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd22a59c640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e84490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd22ad9d640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e6d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 -- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 msgr2=0x561fd6e84490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e84490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 -- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 msgr2=0x561fd6e6d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e6d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 -- 172.21.15.78:0/3142092329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fd6d932b0 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd223fff640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e90ae0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fd2140015e0 tx=0x7fd214000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6736948c43cc4695 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2097fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2097fa640 1 -- 172.21.15.78:0/3142092329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214032070 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2097fa640 1 -- 172.21.15.78:0/3142092329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd21401c040 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd214014b00 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2097fa640 1 -- 172.21.15.78:0/3142092329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd214014920 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 10 client.10944 fetch_fsmap learned FSMap version 193 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.190+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) v3 -- 0x561fd6d2fa20 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2097fa640 1 -- 172.21.15.78:0/3142092329 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd214014b00 con 0x561fd6e2eae0 2024-02-19T15:01:02.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2097fa640 1 -- 172.21.15.78:0/3142092329 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 193) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fd214012070 con 0x561fd6e2eae0 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 10 client.10944 fetch_fsmap finished waiting for FSMap version 193 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 10 client.10944 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 20 client.10944 populate_metadata read hostname 'smithi078' 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e28230 0x561fd6eda1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 4 client.10944 mds_command: new command op to 10731 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 --> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561fd6ad1610 con 0x561fd6e28230 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd22a59c640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e28230 0x561fd6eda1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2097fa640 10 client.10944 ms_handle_connect on v2:172.21.15.150:6833/1952320195 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd22a59c640 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e28230 0x561fd6eda1b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7093d07cae6bb535 server_cookie=370f626249cb93c0 in_seq=0 out_seq=0 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2097fa640 1 -- 172.21.15.78:0/3142092329 <== mds.0 v2:172.21.15.150:6833/1952320195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x561fd6ad1610 con 0x561fd6e28230 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2097fa640 10 client.10944 handle_command_reply: tid=0 2024-02-19T15:01:02.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.191+0000 7fd2337c8740 1 client.10944 shutdown 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 msgr2=0x561fd6e90ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e90ae0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fd2140015e0 tx=0x7fd214000ed0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 shutdown_connections 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:6833/1952320195,v1:172.21.15.150:6835/1952320195] conn(0x561fd6e28230 0x561fd6eda1b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fd6e2eae0 0x561fd6e90ae0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561fd6e51940 0x561fd6e6d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2097fa640 0 client.10944 ms_handle_reset on v2:172.21.15.150:6833/1952320195 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 --2- 172.21.15.78:0/3142092329 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x561fd6e2dc20 0x561fd6e84490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 >> 172.21.15.78:0/3142092329 conn(0x561fd6e29450 msgr2=0x561fd6e818d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 shutdown_connections 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 1 -- 172.21.15.78:0/3142092329 wait complete. 2024-02-19T15:01:02.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd2337c8740 20 client.10944 trim_cache size 0 max 16384 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "7cde3b0a-2943-4d9b-b265-05e609d5a6b3", 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 10731, 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 12.890951279999999, 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 193, 2024-02-19T15:01:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 213, 2024-02-19T15:01:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 213, 2024-02-19T15:01:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17.988642758000001 2024-02-19T15:01:02.204 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd20c06b810 msgr2=0x7fd20c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd20c06b810 0x7fd20c06dcd0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fd214007fd0 tx=0x7fd214009040 comp rx=0 tx=0).stop 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 msgr2=0x7fd22c1515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 0x7fd22c1515a0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001eb0 tx=0x7fd21c00c960 comp rx=0 tx=0).stop 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 shutdown_connections 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd20c06b810 0x7fd20c06dcd0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd22c14ef50 0x7fd22c0be8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd22c0c78a0 0x7fd22c1ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 --2- 172.21.15.78:0/2063663914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd22c072ee0 0x7fd22c1515a0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.192+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 >> 172.21.15.78:0/2063663914 conn(0x7fd22c066e70 msgr2=0x7fd22c0a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.193+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 shutdown_connections 2024-02-19T15:01:02.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.193+0000 7fd230b0c640 1 -- 172.21.15.78:0/2063663914 wait complete. 2024-02-19T15:01:02.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-19T15:01:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.314+0000 7fbe6c0c9640 1 Processor -- start 2024-02-19T15:01:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.314+0000 7fbe6c0c9640 1 -- start start 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe6c0c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640c8e00 0x7fbe6406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe6c0c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe641551f0 con 0x7fbe6406d910 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe6c0c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe64155490 con 0x7fbe64152700 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe6c0c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64155770 con 0x7fbe640c8e00 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe6a666640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbe6406d910 0x7fbe6415f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48812/0 (socket says 172.21.15.78:48812) 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe6a666640 1 -- 172.21.15.78:0/3620475908 learned_addr learned my addr 172.21.15.78:0/3620475908 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe69664640 1 --2- 172.21.15.78:0/3620475908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640c8e00 0x7fbe6406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593428182 0 0) 0x7fbe641551f0 con 0x7fbe6406d910 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe50003520 con 0x7fbe6406d910 2024-02-19T15:01:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084528810 0 0) 0x7fbe64155490 con 0x7fbe64152700 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe641551f0 con 0x7fbe64152700 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe69664640 1 -- 172.21.15.78:0/3620475908 >> v1:172.21.15.150:6789/0 conn(0x7fbe64152700 legacy=0x7fbe640c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe69664640 1 -- 172.21.15.78:0/3620475908 >> v1:172.21.15.78:6789/0 conn(0x7fbe6406d910 legacy=0x7fbe6415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.315+0000 7fbe69664640 1 -- 172.21.15.78:0/3620475908 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64155a50 con 0x7fbe640c8e00 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.316+0000 7fbe69664640 1 --2- 172.21.15.78:0/3620475908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640c8e00 0x7fbe6406c2a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbe540097d0 tx=0x7fbe5402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31e613bd28c9444 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.316+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54031800 con 0x7fbe640c8e00 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.316+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe54031960 con 0x7fbe640c8e00 2024-02-19T15:01:02.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.316+0000 7fbe68e63640 1 -- 172.21.15.78:0/3620475908 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54031cf0 con 0x7fbe640c8e00 2024-02-19T15:01:02.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3620475908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640c8e00 msgr2=0x7fbe6406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3620475908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640c8e00 0x7fbe6406c2a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbe540097d0 tx=0x7fbe5402b8a0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3620475908 shutdown_connections 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3620475908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640c8e00 0x7fbe6406c2a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3620475908 >> 172.21.15.78:0/3620475908 conn(0x7fbe640cac90 msgr2=0x7fbe640cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3620475908 shutdown_connections 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.317+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3620475908 wait complete. 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.319+0000 7fbe6c0c9640 1 Processor -- start 2024-02-19T15:01:02.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.319+0000 7fbe6c0c9640 1 -- start start 2024-02-19T15:01:02.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 0x7fbe64154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe640c8e00 0x7fbe641ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe64152700 0x7fbe641ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641ede10 con 0x7fbe6406d910 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c1fa0 con 0x7fbe640c8e00 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640c43c0 con 0x7fbe64152700 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 0x7fbe64154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 0x7fbe64154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50298/0 (socket says 172.21.15.78:50298) 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 -- 172.21.15.78:0/3644021014 learned_addr learned my addr 172.21.15.78:0/3644021014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69664640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe640c8e00 0x7fbe641ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6a666640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe64152700 0x7fbe641ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 -- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe64152700 msgr2=0x7fbe641ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe64152700 0x7fbe641ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 -- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe640c8e00 msgr2=0x7fbe641ee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:02.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe640c8e00 0x7fbe641ee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 -- 172.21.15.78:0/3644021014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54009400 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe69e65640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 0x7fbe64154210 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c460 tx=0x7fbe6000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe600052d0 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60005450 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640c19e0 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60003830 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.320+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe64154750 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.321+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64074990 con 0x7fbe6406d910 2024-02-19T15:01:02.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.322+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbe60003990 con 0x7fbe6406d910 2024-02-19T15:01:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.322+0000 7fbe5a7fc640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe4806b810 0x7fbe4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.322+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbe60089a00 con 0x7fbe6406d910 2024-02-19T15:01:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.322+0000 7fbe69664640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe4806b810 0x7fbe4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:02.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.323+0000 7fbe69664640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe4806b810 0x7fbe4806dcd0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fbe54036000 tx=0x7fbe5403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:02.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.327+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbe60054230 con 0x7fbe6406d910 2024-02-19T15:01:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:02.542+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7fbe6414b700 con 0x7fbe6406d910 2024-02-19T15:01:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.412+0000 7fbe5a7fc640 1 -- 172.21.15.78:0/3644021014 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v194) v1 ==== 125+0+0 (secure 0 0 0) 0x7fbe600170b0 con 0x7fbe6406d910 2024-02-19T15:01:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked down. 2024-02-19T15:01:03.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe4806b810 msgr2=0x7fbe4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe4806b810 0x7fbe4806dcd0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fbe54036000 tx=0x7fbe5403c040 comp rx=0 tx=0).stop 2024-02-19T15:01:03.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 msgr2=0x7fbe64154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 0x7fbe64154210 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c460 tx=0x7fbe6000c920 comp rx=0 tx=0).stop 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 shutdown_connections 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe4806b810 0x7fbe4806dcd0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe64152700 0x7fbe641ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe640c8e00 0x7fbe641ee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 --2- 172.21.15.78:0/3644021014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe6406d910 0x7fbe64154210 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 >> 172.21.15.78:0/3644021014 conn(0x7fbe640cac90 msgr2=0x7fbe6414a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 shutdown_connections 2024-02-19T15:01:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.414+0000 7fbe6c0c9640 1 -- 172.21.15.78:0/3644021014 wait complete. 2024-02-19T15:01:03.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:03.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.526+0000 7f0d5938a640 1 Processor -- start 2024-02-19T15:01:03.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.526+0000 7f0d5938a640 1 -- start start 2024-02-19T15:01:03.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d5938a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d54152700 0x7f0d540c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:03.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d5938a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d540c8e00 0x7f0d5415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:03.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d5938a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d541551f0 con 0x7f0d540c5730 2024-02-19T15:01:03.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d5938a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d54155490 con 0x7f0d540c8e00 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d5938a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d54155770 con 0x7f0d54152700 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d540c8e00 0x7f0d5415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d540c8e00 0x7f0d5415f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50308/0 (socket says 172.21.15.78:50308) 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 -- 172.21.15.78:0/1181478907 learned_addr learned my addr 172.21.15.78:0/1181478907 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d52ffd640 1 --2- 172.21.15.78:0/1181478907 >> v2:172.21.15.150:3300/0 conn(0x7f0d54152700 0x7f0d540c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 -- 172.21.15.78:0/1181478907 >> v2:172.21.15.150:3300/0 conn(0x7f0d54152700 msgr2=0x7f0d540c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 --2- 172.21.15.78:0/1181478907 >> v2:172.21.15.150:3300/0 conn(0x7f0d54152700 0x7f0d540c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 -- 172.21.15.78:0/1181478907 >> v1:172.21.15.150:6789/0 conn(0x7f0d540c5730 legacy=0x7f0d5406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.527+0000 7f0d537fe640 1 -- 172.21.15.78:0/1181478907 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d54155a50 con 0x7f0d540c8e00 2024-02-19T15:01:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.528+0000 7f0d537fe640 1 --2- 172.21.15.78:0/1181478907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5415f980 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0d5408bad0 tx=0x7f0d4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25eb692c6d167c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.528+0000 7f0d51ffb640 1 -- 172.21.15.78:0/1181478907 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c02ca80 con 0x7f0d540c8e00 2024-02-19T15:01:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.528+0000 7f0d51ffb640 1 -- 172.21.15.78:0/1181478907 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c02cbe0 con 0x7f0d540c8e00 2024-02-19T15:01:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.528+0000 7f0d51ffb640 1 -- 172.21.15.78:0/1181478907 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0343f0 con 0x7f0d540c8e00 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 -- 172.21.15.78:0/1181478907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 msgr2=0x7f0d5415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 --2- 172.21.15.78:0/1181478907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5415f980 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0d5408bad0 tx=0x7f0d4c02bee0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 -- 172.21.15.78:0/1181478907 shutdown_connections 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 --2- 172.21.15.78:0/1181478907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5415f980 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 --2- 172.21.15.78:0/1181478907 >> v2:172.21.15.150:3300/0 conn(0x7f0d54152700 0x7f0d540c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 -- 172.21.15.78:0/1181478907 >> 172.21.15.78:0/1181478907 conn(0x7f0d540cac90 msgr2=0x7f0d540cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 -- 172.21.15.78:0/1181478907 shutdown_connections 2024-02-19T15:01:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.529+0000 7f0d5938a640 1 -- 172.21.15.78:0/1181478907 wait complete. 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.530+0000 7f0d5938a640 1 Processor -- start 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.530+0000 7f0d5938a640 1 -- start start 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d540c5730 0x7f0d54149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5414a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d54152700 0x7f0d541ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5414a560 con 0x7f0d540c8e00 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540c3b70 con 0x7f0d540c5730 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540c3e50 con 0x7f0d54152700 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5414a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5414a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50316/0 (socket says 172.21.15.78:50316) 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 -- 172.21.15.78:0/2984836487 learned_addr learned my addr 172.21.15.78:0/2984836487 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d52ffd640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d540c5730 0x7f0d54149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d537fe640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d54152700 0x7f0d541ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 -- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d54152700 msgr2=0x7f0d541ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d54152700 0x7f0d541ee450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 -- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d540c5730 msgr2=0x7f0d54149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d540c5730 0x7f0d54149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 -- 172.21.15.78:0/2984836487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c009b30 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d527fc640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5414a020 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0d4800c400 tx=0x7f0d4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480052d0 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d48005450 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48007260 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d540c4130 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.531+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d540c46f0 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.532+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54074990 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.533+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0d480073c0 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.533+0000 7f0d3b7fe640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d4006b810 0x7f0d4006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.533+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d480896d0 con 0x7f0d540c8e00 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.533+0000 7f0d52ffd640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d4006b810 0x7f0d4006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.533+0000 7f0d52ffd640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d4006b810 0x7f0d4006dcd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009aa0 tx=0x7f0d44009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.537+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0d48054010 con 0x7f0d540c8e00 2024-02-19T15:01:03.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.741+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d540c8160 con 0x7f0d540c8e00 2024-02-19T15:01:03.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.741+0000 7f0d3b7fe640 1 -- 172.21.15.78:0/2984836487 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f0d480576d0 con 0x7f0d540c8e00 2024-02-19T15:01:03.744 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:03.745 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:03.745 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:03.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d4006b810 msgr2=0x7f0d4006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d4006b810 0x7f0d4006dcd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009aa0 tx=0x7f0d44009190 comp rx=0 tx=0).stop 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 msgr2=0x7f0d5414a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5414a020 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0d4800c400 tx=0x7f0d4800c8c0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 shutdown_connections 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d4006b810 0x7f0d4006dcd0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d540c8e00 0x7f0d5414a020 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d54152700 0x7f0d541ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 --2- 172.21.15.78:0/2984836487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d540c5730 0x7f0d54149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 >> 172.21.15.78:0/2984836487 conn(0x7f0d540cac90 msgr2=0x7f0d5406cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 shutdown_connections 2024-02-19T15:01:03.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:03.743+0000 7f0d5938a640 1 -- 172.21.15.78:0/2984836487 wait complete. 2024-02-19T15:01:04.159 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:04.160 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:05.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:05.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.269+0000 7efdaa3b5640 1 Processor -- start 2024-02-19T15:01:05.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.269+0000 7efdaa3b5640 1 -- start start 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efdaa3b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efda4073220 0x7efda406ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efdaa3b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda414d430 con 0x7efda414d050 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efdaa3b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda4072540 con 0x7efda406ba10 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efdaa3b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda406ff20 con 0x7efda4073220 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efda37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7efda4073220 0x7efda406ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efda3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efda414d050 0x7efda40ac950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48818/0 (socket says 172.21.15.78:48818) 2024-02-19T15:01:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efda3fff640 1 -- 172.21.15.78:0/3769815859 learned_addr learned my addr 172.21.15.78:0/3769815859 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efda37fe640 1 -- 172.21.15.78:0/3769815859 >> v1:172.21.15.150:6789/0 conn(0x7efda406ba10 legacy=0x7efda406f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efda37fe640 1 -- 172.21.15.78:0/3769815859 >> v1:172.21.15.78:6789/0 conn(0x7efda414d050 legacy=0x7efda40ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.270+0000 7efda37fe640 1 -- 172.21.15.78:0/3769815859 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4157010 con 0x7efda4073220 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.271+0000 7efda37fe640 1 --2- 172.21.15.78:0/3769815859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda4073220 0x7efda406ee20 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7efd980097a0 tx=0x7efd9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438d4ba95490a5bd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.271+0000 7efda2ffd640 1 -- 172.21.15.78:0/3769815859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98031830 con 0x7efda4073220 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.271+0000 7efda2ffd640 1 -- 172.21.15.78:0/3769815859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd98031990 con 0x7efda4073220 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.271+0000 7efda2ffd640 1 -- 172.21.15.78:0/3769815859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98031d20 con 0x7efda4073220 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3769815859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda4073220 msgr2=0x7efda406ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3769815859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda4073220 0x7efda406ee20 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7efd980097a0 tx=0x7efd9802b8d0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3769815859 shutdown_connections 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3769815859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda4073220 0x7efda406ee20 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3769815859 >> 172.21.15.78:0/3769815859 conn(0x7efda40c7590 msgr2=0x7efda40c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:05.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3769815859 shutdown_connections 2024-02-19T15:01:05.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.272+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3769815859 wait complete. 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.273+0000 7efdaa3b5640 1 Processor -- start 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.273+0000 7efdaa3b5640 1 -- start start 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efda406ba10 0x7efda4155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efda4073220 0x7efda41dd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 0x7efda41ddbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4072820 con 0x7efda414d050 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda41e9740 con 0x7efda406ba10 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda41e8ac0 con 0x7efda4073220 2024-02-19T15:01:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 0x7efda41ddbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 0x7efda41ddbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50332/0 (socket says 172.21.15.78:50332) 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 -- 172.21.15.78:0/3722060896 learned_addr learned my addr 172.21.15.78:0/3722060896 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda3fff640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efda406ba10 0x7efda4155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda37fe640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efda4073220 0x7efda41dd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 -- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efda4073220 msgr2=0x7efda41dd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efda4073220 0x7efda41dd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 -- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efda406ba10 msgr2=0x7efda4155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efda406ba10 0x7efda4155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 -- 172.21.15.78:0/3722060896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd98009400 con 0x7efda414d050 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda8952640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 0x7efda41ddbd0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00c7d0 tx=0x7efd9c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c003b70 con 0x7efda414d050 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9c003cf0 con 0x7efda414d050 2024-02-19T15:01:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00aa90 con 0x7efda414d050 2024-02-19T15:01:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda41e8da0 con 0x7efda414d050 2024-02-19T15:01:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.274+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda41e9360 con 0x7efda414d050 2024-02-19T15:01:05.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.275+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda41e9880 con 0x7efda414d050 2024-02-19T15:01:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.276+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efd9c00abf0 con 0x7efda414d050 2024-02-19T15:01:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.276+0000 7efda0ff9640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd9006b810 0x7efd9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.276+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efd9c089fb0 con 0x7efda414d050 2024-02-19T15:01:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.276+0000 7efda3fff640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd9006b810 0x7efd9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.277+0000 7efda3fff640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd9006b810 0x7efd9006dcd0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7efd94005880 tx=0x7efd94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:05.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.281+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efd9c0bddc0 con 0x7efda414d050 2024-02-19T15:01:05.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.492+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efda4074990 con 0x7efda414d050 2024-02-19T15:01:05.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.492+0000 7efda0ff9640 1 -- 172.21.15.78:0/3722060896 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7efd9c003e60 con 0x7efda414d050 2024-02-19T15:01:05.495 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:05.496 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:05.496 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:05.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd9006b810 msgr2=0x7efd9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd9006b810 0x7efd9006dcd0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7efd94005880 tx=0x7efd94009040 comp rx=0 tx=0).stop 2024-02-19T15:01:05.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 msgr2=0x7efda41ddbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:05.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 0x7efda41ddbd0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00c7d0 tx=0x7efd9c00cc90 comp rx=0 tx=0).stop 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 shutdown_connections 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efd9006b810 0x7efd9006dcd0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efda414d050 0x7efda41ddbd0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efda4073220 0x7efda41dd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 --2- 172.21.15.78:0/3722060896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efda406ba10 0x7efda4155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 >> 172.21.15.78:0/3722060896 conn(0x7efda40c7590 msgr2=0x7efda4145bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 shutdown_connections 2024-02-19T15:01:05.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:05.494+0000 7efdaa3b5640 1 -- 172.21.15.78:0/3722060896 wait complete. 2024-02-19T15:01:05.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:05.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:06.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:07.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.019+0000 7f7e4cc39640 1 Processor -- start 2024-02-19T15:01:07.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.019+0000 7f7e4cc39640 1 -- start start 2024-02-19T15:01:07.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e4cc39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e48152830 0x7f7e480c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e4cc39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7e480c7990 0x7f7e4806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e4cc39640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e481586a0 con 0x7f7e480c5760 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e4cc39640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e480c6550 con 0x7f7e480c7990 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e4cc39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e48159320 con 0x7f7e48152830 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e45d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7e480c5760 0x7f7e480c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:40318/0 (socket says 172.21.15.78:40318) 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e45d9b640 1 -- 172.21.15.78:0/1701679099 learned_addr learned my addr 172.21.15.78:0/1701679099 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e46d9d640 1 --2- 172.21.15.78:0/1701679099 >> v2:172.21.15.150:3300/0 conn(0x7f7e480c7990 0x7f7e4806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.020+0000 7f7e4659c640 1 --2- 172.21.15.78:0/1701679099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e48152830 0x7f7e480c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4559a640 1 -- 172.21.15.78:0/1701679099 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873566469 0 0) 0x7f7e481586a0 con 0x7f7e480c5760 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4559a640 1 -- 172.21.15.78:0/1701679099 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e30003520 con 0x7f7e480c5760 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4659c640 1 -- 172.21.15.78:0/1701679099 >> v2:172.21.15.150:3300/0 conn(0x7f7e480c7990 msgr2=0x7f7e4806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4659c640 1 --2- 172.21.15.78:0/1701679099 >> v2:172.21.15.150:3300/0 conn(0x7f7e480c7990 0x7f7e4806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4659c640 1 -- 172.21.15.78:0/1701679099 >> v1:172.21.15.150:6789/0 conn(0x7f7e480c5760 legacy=0x7f7e480c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:07.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4659c640 1 -- 172.21.15.78:0/1701679099 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48158980 con 0x7f7e48152830 2024-02-19T15:01:07.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e46d9d640 1 --2- 172.21.15.78:0/1701679099 >> v2:172.21.15.150:3300/0 conn(0x7f7e480c7990 0x7f7e4806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.021+0000 7f7e4659c640 1 --2- 172.21.15.78:0/1701679099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e48152830 0x7f7e480c5220 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009d90 tx=0x7f7e3802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0130f72e66db62a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4559a640 1 -- 172.21.15.78:0/1701679099 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3802cbc0 con 0x7f7e48152830 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4559a640 1 -- 172.21.15.78:0/1701679099 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3802cd20 con 0x7f7e48152830 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4559a640 1 -- 172.21.15.78:0/1701679099 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38034660 con 0x7f7e48152830 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 -- 172.21.15.78:0/1701679099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e48152830 msgr2=0x7f7e480c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/1701679099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e48152830 0x7f7e480c5220 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009d90 tx=0x7f7e3802be10 comp rx=0 tx=0).stop 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 -- 172.21.15.78:0/1701679099 shutdown_connections 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/1701679099 >> v2:172.21.15.150:3300/0 conn(0x7f7e480c7990 0x7f7e4806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/1701679099 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e48152830 0x7f7e480c5220 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 -- 172.21.15.78:0/1701679099 >> 172.21.15.78:0/1701679099 conn(0x7f7e480ca8a0 msgr2=0x7f7e480cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:07.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 -- 172.21.15.78:0/1701679099 shutdown_connections 2024-02-19T15:01:07.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.022+0000 7f7e4cc39640 1 -- 172.21.15.78:0/1701679099 wait complete. 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 Processor -- start 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 -- start start 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e480c5760 0x7f7e480c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 0x7f7e480bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e480bf300 0x7f7e480c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e481eb310 con 0x7f7e48152830 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e480c1f80 con 0x7f7e480c5760 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e4cc39640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e481ea700 con 0x7f7e480bf300 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.024+0000 7f7e45d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 0x7f7e480bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:07.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e4659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e480c5760 0x7f7e480c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 0x7f7e480bedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50348/0 (socket says 172.21.15.78:50348) 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 -- 172.21.15.78:0/154620865 learned_addr learned my addr 172.21.15.78:0/154620865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e46d9d640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e480bf300 0x7f7e480c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 -- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e480bf300 msgr2=0x7f7e480c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e480bf300 0x7f7e480c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 -- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e480c5760 msgr2=0x7f7e480c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e480c5760 0x7f7e480c0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 -- 172.21.15.78:0/154620865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40009400 con 0x7f7e48152830 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e45d9b640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 0x7f7e480bedc0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0010e0 tx=0x7f7e3c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c005ce0 con 0x7f7e48152830 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c005e60 con 0x7f7e48152830 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c007310 con 0x7f7e48152830 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e38009a70 con 0x7f7e48152830 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.025+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e481eacc0 con 0x7f7e48152830 2024-02-19T15:01:07.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.026+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e48074990 con 0x7f7e48152830 2024-02-19T15:01:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.027+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7e3c007530 con 0x7f7e48152830 2024-02-19T15:01:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.027+0000 7f7e36ffd640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e2806b810 0x7f7e2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.027+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e3c089710 con 0x7f7e48152830 2024-02-19T15:01:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.027+0000 7f7e4659c640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e2806b810 0x7f7e2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:07.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.028+0000 7f7e4659c640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e2806b810 0x7f7e2806dcd0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009d90 tx=0x7f7e38030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:07.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.032+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7e3c0540e0 con 0x7f7e48152830 2024-02-19T15:01:07.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.239+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e481583e0 con 0x7f7e48152830 2024-02-19T15:01:07.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.239+0000 7f7e36ffd640 1 -- 172.21.15.78:0/154620865 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f7e3c0577a0 con 0x7f7e48152830 2024-02-19T15:01:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:07.243 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:07.243 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e2806b810 msgr2=0x7f7e2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e2806b810 0x7f7e2806dcd0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009d90 tx=0x7f7e38030000 comp rx=0 tx=0).stop 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 msgr2=0x7f7e480bedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 0x7f7e480bedc0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0010e0 tx=0x7f7e3c00c8a0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 shutdown_connections 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7e2806b810 0x7f7e2806dcd0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e48152830 0x7f7e480bedc0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7e480bf300 0x7f7e480c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 --2- 172.21.15.78:0/154620865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e480c5760 0x7f7e480c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 >> 172.21.15.78:0/154620865 conn(0x7f7e480ca8a0 msgr2=0x7f7e480cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 shutdown_connections 2024-02-19T15:01:07.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:07.241+0000 7f7e4cc39640 1 -- 172.21.15.78:0/154620865 wait complete. 2024-02-19T15:01:07.658 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:07.658 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:08.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:08.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 Processor -- start 2024-02-19T15:01:08.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 -- start start 2024-02-19T15:01:08.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f85600c7070 0x7f85601559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85600c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8560155ef0 con 0x7f8560152480 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856006dec0 con 0x7f85600c7070 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.774+0000 7f856658a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856006d520 con 0x7f85600c87a0 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8560152480 0x7f856006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36848/0 (socket says 172.21.15.78:36848) 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855ffff640 1 -- 172.21.15.78:0/980907714 learned_addr learned my addr 172.21.15.78:0/980907714 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855f7fe640 1 --2- 172.21.15.78:0/980907714 >> v2:172.21.15.150:3300/0 conn(0x7f85600c7070 0x7f85601559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f8564b27640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85600c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855effd640 1 -- 172.21.15.78:0/980907714 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564559086 0 0) 0x7f8550001110 con 0x7f8560152480 2024-02-19T15:01:08.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855effd640 1 -- 172.21.15.78:0/980907714 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8548003520 con 0x7f8560152480 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855f7fe640 1 -- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 msgr2=0x7f85600c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855f7fe640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85600c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855f7fe640 1 -- 172.21.15.78:0/980907714 >> v1:172.21.15.78:6789/0 conn(0x7f8560152480 legacy=0x7f856006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f855f7fe640 1 -- 172.21.15.78:0/980907714 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85600c75c0 con 0x7f85600c7070 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.775+0000 7f8564b27640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85600c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.776+0000 7f855f7fe640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85600c7070 0x7f85601559b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f85540097a0 tx=0x7f855402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62fa32b92449023f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.776+0000 7f855effd640 1 -- 172.21.15.78:0/980907714 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554031830 con 0x7f85600c7070 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.776+0000 7f855effd640 1 -- 172.21.15.78:0/980907714 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554031990 con 0x7f85600c7070 2024-02-19T15:01:08.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.776+0000 7f855effd640 1 -- 172.21.15.78:0/980907714 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554031d00 con 0x7f85600c7070 2024-02-19T15:01:08.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 -- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85600c7070 msgr2=0x7f85601559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:08.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85600c7070 0x7f85601559b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f85540097a0 tx=0x7f855402b8d0 comp rx=0 tx=0).stop 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 -- 172.21.15.78:0/980907714 shutdown_connections 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85600c7070 0x7f85601559b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 --2- 172.21.15.78:0/980907714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85600c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 -- 172.21.15.78:0/980907714 >> 172.21.15.78:0/980907714 conn(0x7f85600ca8a0 msgr2=0x7f85600cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 -- 172.21.15.78:0/980907714 shutdown_connections 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.777+0000 7f856658a640 1 -- 172.21.15.78:0/980907714 wait complete. 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.778+0000 7f856658a640 1 Processor -- start 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.778+0000 7f856658a640 1 -- start start 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f856658a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 0x7f8560151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:08.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f856658a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85601eadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f856658a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8560152480 0x7f85601eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f856658a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85600ab7c0 con 0x7f85600c7070 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f856658a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85601eeeb0 con 0x7f8560152480 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f856658a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85601ee330 con 0x7f85600c87a0 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 0x7f8560151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 0x7f8560151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57092/0 (socket says 172.21.15.78:57092) 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 -- 172.21.15.78:0/3274822772 learned_addr learned my addr 172.21.15.78:0/3274822772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855f7fe640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85601eadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f8564b27640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8560152480 0x7f85601eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 -- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 msgr2=0x7f85601eadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:08.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85601eadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 -- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8560152480 msgr2=0x7f85601eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8560152480 0x7f85601eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 -- 172.21.15.78:0/3274822772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558009a70 con 0x7f85600c7070 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f8564b27640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8560152480 0x7f85601eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.779+0000 7f855ffff640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 0x7f8560151560 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f855000c370 tx=0x7f855000c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.780+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8550005ce0 con 0x7f85600c7070 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.780+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8550005e40 con 0x7f85600c7070 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.780+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8550007310 con 0x7f85600c7070 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.780+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8554009400 con 0x7f85600c7070 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.780+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85601ee8f0 con 0x7f85600c7070 2024-02-19T15:01:08.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.781+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8560074990 con 0x7f85600c7070 2024-02-19T15:01:08.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.781+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8550007470 con 0x7f85600c7070 2024-02-19T15:01:08.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.782+0000 7f855cff9640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f854c06b810 0x7f854c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:08.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.782+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8550089680 con 0x7f85600c7070 2024-02-19T15:01:08.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.782+0000 7f855f7fe640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f854c06b810 0x7f854c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:08.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.782+0000 7f855f7fe640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f854c06b810 0x7f854c06dcd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8554036000 tx=0x7f855403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:08.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.786+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8550053fc0 con 0x7f85600c7070 2024-02-19T15:01:08.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.995+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8560155ef0 con 0x7f85600c7070 2024-02-19T15:01:08.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.995+0000 7f855cff9640 1 -- 172.21.15.78:0/3274822772 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f8560155ef0 con 0x7f85600c7070 2024-02-19T15:01:08.998 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:08.998 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:08.999 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:09.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f854c06b810 msgr2=0x7f854c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:09.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f854c06b810 0x7f854c06dcd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f8554036000 tx=0x7f855403c040 comp rx=0 tx=0).stop 2024-02-19T15:01:09.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 msgr2=0x7f8560151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 0x7f8560151560 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f855000c370 tx=0x7f855000c770 comp rx=0 tx=0).stop 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 shutdown_connections 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f854c06b810 0x7f854c06dcd0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8560152480 0x7f85601eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85600c87a0 0x7f85601eadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 --2- 172.21.15.78:0/3274822772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85600c7070 0x7f8560151560 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 >> 172.21.15.78:0/3274822772 conn(0x7f85600ca8a0 msgr2=0x7f8560154f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 shutdown_connections 2024-02-19T15:01:09.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:08.997+0000 7f856658a640 1 -- 172.21.15.78:0/3274822772 wait complete. 2024-02-19T15:01:09.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:09.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:10.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:10.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.526+0000 7f12c6c9a640 1 Processor -- start 2024-02-19T15:01:10.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c6c9a640 1 -- start start 2024-02-19T15:01:10.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c6c9a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12c01537a0 0x7f12c00c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:10.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c6c9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12c0156860 0x7f12c0156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:10.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c6c9a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c01563b0 con 0x7f12c014cd50 2024-02-19T15:01:10.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c6c9a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c0153cd0 con 0x7f12c01537a0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c6c9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c006b6e0 con 0x7f12c0156860 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12bffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f12c01537a0 0x7f12c00c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c4a36640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f12c014cd50 0x7f12c00c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36864/0 (socket says 172.21.15.78:36864) 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.527+0000 7f12c4a36640 1 -- 172.21.15.78:0/3261344720 learned_addr learned my addr 172.21.15.78:0/3261344720 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.528+0000 7f12c5237640 1 --2- 172.21.15.78:0/3261344720 >> v2:172.21.15.150:3300/0 conn(0x7f12c0156860 0x7f12c0156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.528+0000 7f12bffff640 1 -- 172.21.15.78:0/3261344720 >> v2:172.21.15.150:3300/0 conn(0x7f12c0156860 msgr2=0x7f12c0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.528+0000 7f12bffff640 1 --2- 172.21.15.78:0/3261344720 >> v2:172.21.15.150:3300/0 conn(0x7f12c0156860 0x7f12c0156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.528+0000 7f12bffff640 1 -- 172.21.15.78:0/3261344720 >> v1:172.21.15.78:6789/0 conn(0x7f12c014cd50 legacy=0x7f12c00c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.528+0000 7f12bf7fe640 1 -- 172.21.15.78:0/3261344720 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745158593 0 0) 0x7f12c01563b0 con 0x7f12c014cd50 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.528+0000 7f12bffff640 1 -- 172.21.15.78:0/3261344720 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c006b980 con 0x7f12c01537a0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12bffff640 1 --2- 172.21.15.78:0/3261344720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c01537a0 0x7f12c00c4cd0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f12c0152d20 tx=0x7f12ac02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7714f04c233d3ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12bf7fe640 1 -- 172.21.15.78:0/3261344720 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac031710 con 0x7f12c01537a0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12bf7fe640 1 -- 172.21.15.78:0/3261344720 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac02f050 con 0x7f12c01537a0 2024-02-19T15:01:10.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12bf7fe640 1 -- 172.21.15.78:0/3261344720 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac031a90 con 0x7f12c01537a0 2024-02-19T15:01:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 -- 172.21.15.78:0/3261344720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c01537a0 msgr2=0x7f12c00c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/3261344720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c01537a0 0x7f12c00c4cd0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f12c0152d20 tx=0x7f12ac02b7b0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 -- 172.21.15.78:0/3261344720 shutdown_connections 2024-02-19T15:01:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/3261344720 >> v2:172.21.15.150:3300/0 conn(0x7f12c0156860 0x7f12c0156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/3261344720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c01537a0 0x7f12c00c4cd0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 -- 172.21.15.78:0/3261344720 >> 172.21.15.78:0/3261344720 conn(0x7f12c0066e70 msgr2=0x7f12c00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 -- 172.21.15.78:0/3261344720 shutdown_connections 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.529+0000 7f12c6c9a640 1 -- 172.21.15.78:0/3261344720 wait complete. 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.530+0000 7f12c6c9a640 1 Processor -- start 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.530+0000 7f12c6c9a640 1 -- start start 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c6c9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c014cd50 0x7f12c01ebea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c6c9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c01537a0 0x7f12c01dd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c6c9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 0x7f12c01dd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c6c9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01df2a0 con 0x7f12c0156860 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c6c9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01dcc70 con 0x7f12c01537a0 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c6c9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c01ecb70 con 0x7f12c014cd50 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c5237640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 0x7f12c01dd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.531+0000 7f12c5237640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 0x7f12c01dd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57114/0 (socket says 172.21.15.78:57114) 2024-02-19T15:01:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 -- 172.21.15.78:0/1359382669 learned_addr learned my addr 172.21.15.78:0/1359382669 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c4a36640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c014cd50 0x7f12c01ebea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12bffff640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c01537a0 0x7f12c01dd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 -- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c014cd50 msgr2=0x7f12c01ebea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c014cd50 0x7f12c01ebea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 -- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c01537a0 msgr2=0x7f12c01dd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c01537a0 0x7f12c01dd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 -- 172.21.15.78:0/1359382669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac009400 con 0x7f12c0156860 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c5237640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 0x7f12c01dd790 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009d30 tx=0x7f12b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8009ba0 con 0x7f12c0156860 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b800f040 con 0x7f12c0156860 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c01dcec0 con 0x7f12c0156860 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b80146c0 con 0x7f12c0156860 2024-02-19T15:01:10.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.532+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c01dc4b0 con 0x7f12c0156860 2024-02-19T15:01:10.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.533+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c0074990 con 0x7f12c0156860 2024-02-19T15:01:10.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.534+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f12b8014820 con 0x7f12c0156860 2024-02-19T15:01:10.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.534+0000 7f12bd7fa640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a406b810 0x7f12a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:10.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.534+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12b808ac80 con 0x7f12c0156860 2024-02-19T15:01:10.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.534+0000 7f12c4a36640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a406b810 0x7f12a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:10.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.534+0000 7f12c4a36640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a406b810 0x7f12a406dcd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f12b4007fd0 tx=0x7f12b4007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:10.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.538+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f12b8055650 con 0x7f12c0156860 2024-02-19T15:01:10.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.743+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12c01dc6a0 con 0x7f12c0156860 2024-02-19T15:01:10.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.744+0000 7f12bd7fa640 1 -- 172.21.15.78:0/1359382669 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f12b8018020 con 0x7f12c0156860 2024-02-19T15:01:10.748 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:10.748 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:10.749 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:10.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a406b810 msgr2=0x7f12a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a406b810 0x7f12a406dcd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f12b4007fd0 tx=0x7f12b4007de0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 msgr2=0x7f12c01dd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 0x7f12c01dd790 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009d30 tx=0x7f12b800c490 comp rx=0 tx=0).stop 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 shutdown_connections 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f12a406b810 0x7f12a406dcd0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12c0156860 0x7f12c01dd790 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c01537a0 0x7f12c01dd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 --2- 172.21.15.78:0/1359382669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12c014cd50 0x7f12c01ebea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.745+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 >> 172.21.15.78:0/1359382669 conn(0x7f12c0066e70 msgr2=0x7f12c0151f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.746+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 shutdown_connections 2024-02-19T15:01:10.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:10.746+0000 7f12c6c9a640 1 -- 172.21.15.78:0/1359382669 wait complete. 2024-02-19T15:01:11.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:11.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:12.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:12.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.279+0000 7f9c3b7d7640 1 Processor -- start 2024-02-19T15:01:12.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.279+0000 7f9c3b7d7640 1 -- start start 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c3b7d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c340c7070 0x7f9c341559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c3b7d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 0x7f9c340c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c3b7d7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c34155ef0 con 0x7f9c34152480 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c3b7d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3406dec0 con 0x7f9c340c7070 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c3b7d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3406d520 con 0x7f9c340c87a0 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c340c7070 0x7f9c341559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c39d74640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 0x7f9c340c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c340c7070 0x7f9c341559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57682/0 (socket says 172.21.15.78:57682) 2024-02-19T15:01:12.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c39d74640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 0x7f9c340c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:58922/0 (socket says 172.21.15.78:58922) 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 -- 172.21.15.78:0/1792892457 learned_addr learned my addr 172.21.15.78:0/1792892457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c39573640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9c34152480 0x7f9c3406ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:41310/0 (socket says 172.21.15.78:41310) 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 -- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 msgr2=0x7f9c340c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 --2- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 0x7f9c340c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 -- 172.21.15.78:0/1792892457 >> v1:172.21.15.150:6789/0 conn(0x7f9c34152480 legacy=0x7f9c3406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c38d72640 1 -- 172.21.15.78:0/1792892457 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340c75c0 con 0x7f9c340c7070 2024-02-19T15:01:12.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.280+0000 7f9c39d74640 1 --2- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 0x7f9c340c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.281+0000 7f9c38d72640 1 --2- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341559b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009870 tx=0x7f9c3002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7db31342f8a0a38c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.281+0000 7f9c2bfff640 1 -- 172.21.15.78:0/1792892457 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30030840 con 0x7f9c340c7070 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.281+0000 7f9c2bfff640 1 -- 172.21.15.78:0/1792892457 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c300309a0 con 0x7f9c340c7070 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.281+0000 7f9c2bfff640 1 -- 172.21.15.78:0/1792892457 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30030d30 con 0x7f9c340c7070 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 msgr2=0x7f9c341559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341559b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009870 tx=0x7f9c3002b980 comp rx=0 tx=0).stop 2024-02-19T15:01:12.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/1792892457 shutdown_connections 2024-02-19T15:01:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c340c87a0 0x7f9c340c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/1792892457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341559b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/1792892457 >> 172.21.15.78:0/1792892457 conn(0x7f9c340ca8a0 msgr2=0x7f9c340cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/1792892457 shutdown_connections 2024-02-19T15:01:12.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.282+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/1792892457 wait complete. 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.283+0000 7f9c3b7d7640 1 Processor -- start 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.283+0000 7f9c3b7d7640 1 -- start start 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c3b7d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c3b7d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 0x7f9c3414c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c3b7d7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 0x7f9c34149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c3b7d7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341eae70 con 0x7f9c340c87a0 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c3b7d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341eb150 con 0x7f9c340c7070 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c3b7d7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341eb430 con 0x7f9c34152480 2024-02-19T15:01:12.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c38d72640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 0x7f9c3414c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39d74640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 0x7f9c34149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c38d72640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 0x7f9c3414c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57122/0 (socket says 172.21.15.78:57122) 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39d74640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 0x7f9c34149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:58936/0 (socket says 172.21.15.78:58936) 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341df9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:57686/0 (socket says 172.21.15.78:57686) 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c38d72640 1 -- 172.21.15.78:0/640916335 learned_addr learned my addr 172.21.15.78:0/640916335 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 -- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 msgr2=0x7f9c34149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 0x7f9c34149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 -- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 msgr2=0x7f9c3414c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 0x7f9c3414c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39573640 1 -- 172.21.15.78:0/640916335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c009a70 con 0x7f9c340c7070 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c38d72640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 0x7f9c3414c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.284+0000 7f9c39d74640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 0x7f9c34149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.285+0000 7f9c39573640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341df9a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9c340ca780 tx=0x7f9c2400c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:12.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.285+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240074e0 con 0x7f9c340c7070 2024-02-19T15:01:12.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.285+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c24007640 con 0x7f9c340c7070 2024-02-19T15:01:12.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.285+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c300094d0 con 0x7f9c340c7070 2024-02-19T15:01:12.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.285+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24004a00 con 0x7f9c340c7070 2024-02-19T15:01:12.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.285+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c341eb9f0 con 0x7f9c340c7070 2024-02-19T15:01:12.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.286+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34074990 con 0x7f9c340c7070 2024-02-19T15:01:12.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.287+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9c24004b60 con 0x7f9c340c7070 2024-02-19T15:01:12.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.287+0000 7f9c29ffb640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c1806b810 0x7f9c1806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:12.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.287+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9c24089890 con 0x7f9c340c7070 2024-02-19T15:01:12.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.287+0000 7f9c38d72640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c1806b810 0x7f9c1806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:12.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.288+0000 7f9c38d72640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c1806b810 0x7f9c1806dcd0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009870 tx=0x7f9c30008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:12.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.296+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9c240541b0 con 0x7f9c340c7070 2024-02-19T15:01:12.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.507+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c34155ef0 con 0x7f9c340c7070 2024-02-19T15:01:12.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.508+0000 7f9c29ffb640 1 -- 172.21.15.78:0/640916335 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f9c34155ef0 con 0x7f9c340c7070 2024-02-19T15:01:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:12.511 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:12.511 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.509+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c1806b810 msgr2=0x7f9c1806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.509+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c1806b810 0x7f9c1806dcd0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009870 tx=0x7f9c30008030 comp rx=0 tx=0).stop 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 msgr2=0x7f9c341df9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341df9a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9c340ca780 tx=0x7f9c2400c760 comp rx=0 tx=0).stop 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 shutdown_connections 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c1806b810 0x7f9c1806dcd0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c340c87a0 0x7f9c3414c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c34152480 0x7f9c34149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 --2- 172.21.15.78:0/640916335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c340c7070 0x7f9c341df9a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 >> 172.21.15.78:0/640916335 conn(0x7f9c340ca8a0 msgr2=0x7f9c340c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:12.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 shutdown_connections 2024-02-19T15:01:12.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:12.510+0000 7f9c3b7d7640 1 -- 172.21.15.78:0/640916335 wait complete. 2024-02-19T15:01:12.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:12.925 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:13.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.026+0000 7fa6c0b16640 1 Processor -- start 2024-02-19T15:01:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.026+0000 7fa6c0b16640 1 -- start start 2024-02-19T15:01:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6c0b16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c25e0 0x7fa6bc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6c0b16640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6bc0c5be0 0x7fa6bc06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6c0b16640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc072250 con 0x7fa6bc0c29c0 2024-02-19T15:01:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6c0b16640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc152c90 con 0x7fa6bc0c5be0 2024-02-19T15:01:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6c0b16640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc072480 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6bad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6bc0c5be0 0x7fa6bc06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6bad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6bc0c5be0 0x7fa6bc06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57124/0 (socket says 172.21.15.78:57124) 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6bad9d640 1 -- 172.21.15.78:0/2246712663 learned_addr learned my addr 172.21.15.78:0/2246712663 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.027+0000 7fa6ba59c640 1 --2- 172.21.15.78:0/2246712663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c25e0 0x7fa6bc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.028+0000 7fa6bad9d640 1 -- 172.21.15.78:0/2246712663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c25e0 msgr2=0x7fa6bc1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.028+0000 7fa6bad9d640 1 --2- 172.21.15.78:0/2246712663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c25e0 0x7fa6bc1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.028+0000 7fa6bad9d640 1 -- 172.21.15.78:0/2246712663 >> v1:172.21.15.150:6789/0 conn(0x7fa6bc0c29c0 legacy=0x7fa6bc0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.028+0000 7fa6bad9d640 1 -- 172.21.15.78:0/2246712663 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc0c4b90 con 0x7fa6bc0c5be0 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.028+0000 7fa6bad9d640 1 --2- 172.21.15.78:0/2246712663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc06e420 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009e80 tx=0x7fa6b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c7469e7cc2e9d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6b959a640 1 -- 172.21.15.78:0/2246712663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b402ca50 con 0x7fa6bc0c5be0 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6b959a640 1 -- 172.21.15.78:0/2246712663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b402cbb0 con 0x7fa6bc0c5be0 2024-02-19T15:01:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6b959a640 1 -- 172.21.15.78:0/2246712663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40343f0 con 0x7fa6bc0c5be0 2024-02-19T15:01:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 -- 172.21.15.78:0/2246712663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c5be0 msgr2=0x7fa6bc06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/2246712663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc06e420 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009e80 tx=0x7fa6b402beb0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 -- 172.21.15.78:0/2246712663 shutdown_connections 2024-02-19T15:01:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/2246712663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc06e420 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/2246712663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c25e0 0x7fa6bc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 -- 172.21.15.78:0/2246712663 >> 172.21.15.78:0/2246712663 conn(0x7fa6bc06b270 msgr2=0x7fa6bc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 -- 172.21.15.78:0/2246712663 shutdown_connections 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.029+0000 7fa6c0b16640 1 -- 172.21.15.78:0/2246712663 wait complete. 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.030+0000 7fa6c0b16640 1 Processor -- start 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 -- start start 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 0x7fa6bc148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c29c0 0x7fa6bc13e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc1e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc13d3e0 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc13ee10 con 0x7fa6bc0c5be0 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6c0b16640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc147d80 con 0x7fa6bc0c29c0 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6ba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 0x7fa6bc148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6ba59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 0x7fa6bc148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57136/0 (socket says 172.21.15.78:57136) 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6ba59c640 1 -- 172.21.15.78:0/727911369 learned_addr learned my addr 172.21.15.78:0/727911369 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6b9d9b640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c29c0 0x7fa6bc13e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.031+0000 7fa6bad9d640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc1e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6ba59c640 1 -- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c29c0 msgr2=0x7fa6bc13e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6ba59c640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c29c0 0x7fa6bc13e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6ba59c640 1 -- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6bc0c5be0 msgr2=0x7fa6bc1e9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6ba59c640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc1e9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6ba59c640 1 -- 172.21.15.78:0/727911369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4009b30 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6ba59c640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 0x7fa6bc148210 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000e8a0 tx=0x7fa6b000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b00094f0 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b000c940 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b000ccf0 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc13f320 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.032+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc13f040 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.033+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6bc0c5a00 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.033+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa6b0013470 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.034+0000 7fa6aaffd640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa69c06b810 0x7fa69c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.034+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6b008a320 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.034+0000 7fa6b9d9b640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa69c06b810 0x7fa69c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.034+0000 7fa6b9d9b640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa69c06b810 0x7fa69c06dcd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc144610 tx=0x7fa6ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.038+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa6b0054cf0 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.245+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6bc074990 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.246+0000 7fa6aaffd640 1 -- 172.21.15.78:0/727911369 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7fa6b00583b0 con 0x7fa6bc0c25e0 2024-02-19T15:01:14.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:14.249 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:14.249 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa69c06b810 msgr2=0x7fa69c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa69c06b810 0x7fa69c06dcd0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc144610 tx=0x7fa6ac009040 comp rx=0 tx=0).stop 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 msgr2=0x7fa6bc148210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 0x7fa6bc148210 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000e8a0 tx=0x7fa6b000ed60 comp rx=0 tx=0).stop 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 shutdown_connections 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa69c06b810 0x7fa69c06dcd0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6bc0c5be0 0x7fa6bc1e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6bc0c29c0 0x7fa6bc13e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 --2- 172.21.15.78:0/727911369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6bc0c25e0 0x7fa6bc148210 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 >> 172.21.15.78:0/727911369 conn(0x7fa6bc06b270 msgr2=0x7fa6bc14a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 shutdown_connections 2024-02-19T15:01:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:14.248+0000 7fa6c0b16640 1 -- 172.21.15.78:0/727911369 wait complete. 2024-02-19T15:01:14.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:14.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:15.665 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.772+0000 7f80a90a4640 1 Processor -- start 2024-02-19T15:01:15.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.772+0000 7f80a90a4640 1 -- start start 2024-02-19T15:01:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a90a4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80a406ae80 0x7f80a406b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:15.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a90a4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80a4155af0 0x7f80a406f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a90a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a4072180 con 0x7f80a40c19f0 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a90a4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a406fdd0 con 0x7f80a4155af0 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a90a4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a4073440 con 0x7f80a406ae80 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f80a4155af0 0x7f80a406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a359e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f80a40c19f0 0x7f80a41585e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36880/0 (socket says 172.21.15.78:36880) 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a359e640 1 -- 172.21.15.78:0/2676422035 learned_addr learned my addr 172.21.15.78:0/2676422035 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a2d9d640 1 --2- 172.21.15.78:0/2676422035 >> v2:172.21.15.150:3300/0 conn(0x7f80a406ae80 0x7f80a406b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a259c640 1 -- 172.21.15.78:0/2676422035 >> v2:172.21.15.150:3300/0 conn(0x7f80a406ae80 msgr2=0x7f80a406b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a259c640 1 --2- 172.21.15.78:0/2676422035 >> v2:172.21.15.150:3300/0 conn(0x7f80a406ae80 0x7f80a406b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a259c640 1 -- 172.21.15.78:0/2676422035 >> v1:172.21.15.78:6789/0 conn(0x7f80a40c19f0 legacy=0x7f80a41585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a259c640 1 -- 172.21.15.78:0/2676422035 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a414f810 con 0x7f80a4155af0 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.773+0000 7f80a1d9b640 1 -- 172.21.15.78:0/2676422035 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928932388 0 0) 0x7f80a4072180 con 0x7f80a40c19f0 2024-02-19T15:01:15.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.774+0000 7f80a259c640 1 --2- 172.21.15.78:0/2676422035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a406f190 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f80a40c0b50 tx=0x7f809802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf3685e8cc53839 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.774+0000 7f80a1d9b640 1 -- 172.21.15.78:0/2676422035 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098031710 con 0x7f80a4155af0 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.774+0000 7f80a1d9b640 1 -- 172.21.15.78:0/2676422035 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809802f050 con 0x7f80a4155af0 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.774+0000 7f80a1d9b640 1 -- 172.21.15.78:0/2676422035 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098031a90 con 0x7f80a4155af0 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 -- 172.21.15.78:0/2676422035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 msgr2=0x7f80a406f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 --2- 172.21.15.78:0/2676422035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a406f190 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f80a40c0b50 tx=0x7f809802b7b0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 -- 172.21.15.78:0/2676422035 shutdown_connections 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 --2- 172.21.15.78:0/2676422035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a406f190 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 --2- 172.21.15.78:0/2676422035 >> v2:172.21.15.150:3300/0 conn(0x7f80a406ae80 0x7f80a406b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 -- 172.21.15.78:0/2676422035 >> 172.21.15.78:0/2676422035 conn(0x7f80a40c7590 msgr2=0x7f80a40c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 -- 172.21.15.78:0/2676422035 shutdown_connections 2024-02-19T15:01:15.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.775+0000 7f80a90a4640 1 -- 172.21.15.78:0/2676422035 wait complete. 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.776+0000 7f80a90a4640 1 Processor -- start 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.776+0000 7f80a90a4640 1 -- start start 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a90a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80a406ae80 0x7f80a4149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a90a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80a40c19f0 0x7f80a41472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a90a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a4147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a90a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a41486c0 con 0x7f80a4155af0 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a90a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a4147f50 con 0x7f80a406ae80 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a90a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a41eb5c0 con 0x7f80a40c19f0 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a4147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a4147830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57148/0 (socket says 172.21.15.78:57148) 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a359e640 1 -- 172.21.15.78:0/3381966468 learned_addr learned my addr 172.21.15.78:0/3381966468 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a2d9d640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80a406ae80 0x7f80a4149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:15.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.777+0000 7f80a259c640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80a40c19f0 0x7f80a41472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a359e640 1 -- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80a40c19f0 msgr2=0x7f80a41472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a359e640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80a40c19f0 0x7f80a41472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a359e640 1 -- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80a406ae80 msgr2=0x7f80a4149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a359e640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80a406ae80 0x7f80a4149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a359e640 1 -- 172.21.15.78:0/3381966468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8098009400 con 0x7f80a4155af0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a359e640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a4147830 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f809c00c7e0 tx=0x7f809c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00a640 con 0x7f80a4155af0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c00a7c0 con 0x7f80a4155af0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a40a9c90 con 0x7f80a4155af0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00ab70 con 0x7f80a4155af0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.778+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a41eb8a0 con 0x7f80a4155af0 2024-02-19T15:01:15.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.779+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a40c4c50 con 0x7f80a4155af0 2024-02-19T15:01:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.779+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f809c00acd0 con 0x7f80a4155af0 2024-02-19T15:01:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.780+0000 7f808b7fe640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f809006b810 0x7f809006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.780+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f809c089ad0 con 0x7f80a4155af0 2024-02-19T15:01:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.780+0000 7f80a2d9d640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f809006b810 0x7f809006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:15.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.780+0000 7f80a2d9d640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f809006b810 0x7f809006dcd0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f80a414a8a0 tx=0x7f8094009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:15.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.784+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f809c0544a0 con 0x7f80a4155af0 2024-02-19T15:01:15.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.992+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a4074990 con 0x7f80a4155af0 2024-02-19T15:01:15.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.992+0000 7f808b7fe640 1 -- 172.21.15.78:0/3381966468 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f809c018020 con 0x7f80a4155af0 2024-02-19T15:01:15.995 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:15.995 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:15.996 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f809006b810 msgr2=0x7f809006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f809006b810 0x7f809006dcd0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f80a414a8a0 tx=0x7f8094009190 comp rx=0 tx=0).stop 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 msgr2=0x7f80a4147830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a4147830 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f809c00c7e0 tx=0x7f809c00cca0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 shutdown_connections 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f809006b810 0x7f809006dcd0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f80a4155af0 0x7f80a4147830 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80a40c19f0 0x7f80a41472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 --2- 172.21.15.78:0/3381966468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80a406ae80 0x7f80a4149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 >> 172.21.15.78:0/3381966468 conn(0x7f80a40c7590 msgr2=0x7f80a4154c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 shutdown_connections 2024-02-19T15:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:15.994+0000 7f80a90a4640 1 -- 172.21.15.78:0/3381966468 wait complete. 2024-02-19T15:01:16.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:16.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:17.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:17.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.524+0000 7f0bc3b02640 1 Processor -- start 2024-02-19T15:01:17.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.524+0000 7f0bc3b02640 1 -- start start 2024-02-19T15:01:17.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc3b02640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bbc0c8500 0x7f0bbc0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:17.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc3b02640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc0c42e0 0x7f0bbc158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:17.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc3b02640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0c34a0 con 0x7f0bbc0c2f50 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc3b02640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc06c090 con 0x7f0bbc0c8500 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc3b02640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc06d6a0 con 0x7f0bbc0c42e0 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc189e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bbc0c8500 0x7f0bbc0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc189e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bbc0c8500 0x7f0bbc0c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50364/0 (socket says 172.21.15.78:50364) 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc189e640 1 -- 172.21.15.78:0/4009978837 learned_addr learned my addr 172.21.15.78:0/4009978837 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.525+0000 7f0bc109d640 1 --2- 172.21.15.78:0/4009978837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc0c42e0 0x7f0bbc158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.526+0000 7f0bc189e640 1 -- 172.21.15.78:0/4009978837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc0c42e0 msgr2=0x7f0bbc158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.526+0000 7f0bc189e640 1 --2- 172.21.15.78:0/4009978837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc0c42e0 0x7f0bbc158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.526+0000 7f0bc189e640 1 -- 172.21.15.78:0/4009978837 >> v1:172.21.15.150:6789/0 conn(0x7f0bbc0c2f50 legacy=0x7f0bbc158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:17.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.526+0000 7f0bc189e640 1 -- 172.21.15.78:0/4009978837 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc06c8e0 con 0x7f0bbc0c8500 2024-02-19T15:01:17.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.526+0000 7f0bc189e640 1 --2- 172.21.15.78:0/4009978837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc0c88e0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009870 tx=0x7f0bb802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7500a18d2730da server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:17.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.526+0000 7f0bc089c640 1 -- 172.21.15.78:0/4009978837 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8031640 con 0x7f0bbc0c8500 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc089c640 1 -- 172.21.15.78:0/4009978837 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb80317a0 con 0x7f0bbc0c8500 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc089c640 1 -- 172.21.15.78:0/4009978837 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8031b30 con 0x7f0bbc0c8500 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 -- 172.21.15.78:0/4009978837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 msgr2=0x7f0bbc0c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/4009978837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc0c88e0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009870 tx=0x7f0bb802b8f0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 -- 172.21.15.78:0/4009978837 shutdown_connections 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/4009978837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc0c42e0 0x7f0bbc158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/4009978837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc0c88e0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 -- 172.21.15.78:0/4009978837 >> 172.21.15.78:0/4009978837 conn(0x7f0bbc0aa300 msgr2=0x7f0bbc0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 -- 172.21.15.78:0/4009978837 shutdown_connections 2024-02-19T15:01:17.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.527+0000 7f0bc3b02640 1 -- 172.21.15.78:0/4009978837 wait complete. 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.528+0000 7f0bc3b02640 1 Processor -- start 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.528+0000 7f0bc3b02640 1 -- start start 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc3b02640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0c2f50 0x7f0bbc070970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc3b02640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc14b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc3b02640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc14ba70 0x7f0bbc14a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc3b02640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1e08d0 con 0x7f0bbc0c8500 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc3b02640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1e22d0 con 0x7f0bbc0c2f50 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc3b02640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1578f0 con 0x7f0bbc14ba70 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc109d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc14b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc109d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc14b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50374/0 (socket says 172.21.15.78:50374) 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc109d640 1 -- 172.21.15.78:0/449497567 learned_addr learned my addr 172.21.15.78:0/449497567 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc189e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0c2f50 0x7f0bbc070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:17.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.529+0000 7f0bc209f640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc14ba70 0x7f0bbc14a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc109d640 1 -- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc14ba70 msgr2=0x7f0bbc14a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc109d640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc14ba70 0x7f0bbc14a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc109d640 1 -- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0c2f50 msgr2=0x7f0bbc070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc109d640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0c2f50 0x7f0bbc070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc109d640 1 -- 172.21.15.78:0/449497567 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80094d0 con 0x7f0bbc0c8500 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc189e640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0c2f50 0x7f0bbc070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc109d640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc14b530 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e9e0 tx=0x7f0bac00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac009670 con 0x7f0bbc0c8500 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00c9d0 con 0x7f0bbc0c8500 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc1e19c0 con 0x7f0bbc0c8500 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00cd80 con 0x7f0bbc0c8500 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.530+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc1e1c80 con 0x7f0bbc0c8500 2024-02-19T15:01:17.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.531+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc071c20 con 0x7f0bbc0c8500 2024-02-19T15:01:17.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.532+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0bac013470 con 0x7f0bbc0c8500 2024-02-19T15:01:17.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.532+0000 7f0bb27fc640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b840 0x7f0ba006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:17.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.532+0000 7f0bc189e640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b840 0x7f0ba006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:17.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.532+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0bac08a780 con 0x7f0bbc0c8500 2024-02-19T15:01:17.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.533+0000 7f0bc189e640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b840 0x7f0ba006dd00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8036000 tx=0x7f0bb803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:17.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.537+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0bac055150 con 0x7f0bbc0c8500 2024-02-19T15:01:17.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.744+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bbc074990 con 0x7f0bbc0c8500 2024-02-19T15:01:17.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.745+0000 7f0bb27fc640 1 -- 172.21.15.78:0/449497567 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f0bac058810 con 0x7f0bbc0c8500 2024-02-19T15:01:17.749 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:17.749 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:17.749 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b840 msgr2=0x7f0ba006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b840 0x7f0ba006dd00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8036000 tx=0x7f0bb803c040 comp rx=0 tx=0).stop 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 msgr2=0x7f0bbc14b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc14b530 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e9e0 tx=0x7f0bac00eea0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 shutdown_connections 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ba006b840 0x7f0ba006dd00 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bbc0c8500 0x7f0bbc14b530 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bbc14ba70 0x7f0bbc14a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 --2- 172.21.15.78:0/449497567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bbc0c2f50 0x7f0bbc070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:17.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 >> 172.21.15.78:0/449497567 conn(0x7f0bbc0aa300 msgr2=0x7f0bbc14f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:17.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 shutdown_connections 2024-02-19T15:01:17.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:17.747+0000 7f0bc3b02640 1 -- 172.21.15.78:0/449497567 wait complete. 2024-02-19T15:01:18.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:18.164 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:19.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:19.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.270+0000 7f466af25640 1 Processor -- start 2024-02-19T15:01:19.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 -- start start 2024-02-19T15:01:19.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4664152670 0x7f4664071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4664152a50 0x7f4664155010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46640702d0 0x7f4664158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4664071bf0 con 0x7f4664152a50 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4664158a20 con 0x7f4664152670 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f466af25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4664155550 con 0x7f46640702d0 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f4663fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4664152a50 0x7f4664155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f4663fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4664152a50 0x7f4664155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50376/0 (socket says 172.21.15.78:50376) 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f4663fff640 1 -- 172.21.15.78:0/251332695 learned_addr learned my addr 172.21.15.78:0/251332695 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f46694c2640 1 --2- 172.21.15.78:0/251332695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46640702d0 0x7f4664158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.271+0000 7f4668cc1640 1 --2- 172.21.15.78:0/251332695 >> v2:172.21.15.150:3300/0 conn(0x7f4664152670 0x7f4664071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4663fff640 1 -- 172.21.15.78:0/251332695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46640702d0 msgr2=0x7f4664158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:19.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4663fff640 1 --2- 172.21.15.78:0/251332695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46640702d0 0x7f4664158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4663fff640 1 -- 172.21.15.78:0/251332695 >> v2:172.21.15.150:3300/0 conn(0x7f4664152670 msgr2=0x7f4664071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4663fff640 1 --2- 172.21.15.78:0/251332695 >> v2:172.21.15.150:3300/0 conn(0x7f4664152670 0x7f4664071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4663fff640 1 -- 172.21.15.78:0/251332695 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640c6780 con 0x7f4664152a50 2024-02-19T15:01:19.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4668cc1640 1 --2- 172.21.15.78:0/251332695 >> v2:172.21.15.150:3300/0 conn(0x7f4664152670 0x7f4664071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.272+0000 7f4663fff640 1 --2- 172.21.15.78:0/251332695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4664152a50 0x7f4664155010 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f466406e600 tx=0x7f465802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b86cea390b71c4d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f46637fe640 1 -- 172.21.15.78:0/251332695 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658031870 con 0x7f4664152a50 2024-02-19T15:01:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f46637fe640 1 -- 172.21.15.78:0/251332695 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46580319d0 con 0x7f4664152a50 2024-02-19T15:01:19.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f46637fe640 1 -- 172.21.15.78:0/251332695 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658031d60 con 0x7f4664152a50 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 -- 172.21.15.78:0/251332695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4664152a50 msgr2=0x7f4664155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 --2- 172.21.15.78:0/251332695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4664152a50 0x7f4664155010 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f466406e600 tx=0x7f465802b910 comp rx=0 tx=0).stop 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 -- 172.21.15.78:0/251332695 shutdown_connections 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 --2- 172.21.15.78:0/251332695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46640702d0 0x7f4664158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 --2- 172.21.15.78:0/251332695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4664152a50 0x7f4664155010 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 --2- 172.21.15.78:0/251332695 >> v2:172.21.15.150:3300/0 conn(0x7f4664152670 0x7f4664071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 -- 172.21.15.78:0/251332695 >> 172.21.15.78:0/251332695 conn(0x7f46640c85e0 msgr2=0x7f46640c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 -- 172.21.15.78:0/251332695 shutdown_connections 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.273+0000 7f466af25640 1 -- 172.21.15.78:0/251332695 wait complete. 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.274+0000 7f466af25640 1 Processor -- start 2024-02-19T15:01:19.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.274+0000 7f466af25640 1 -- start start 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f466af25640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 0x7f46641de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f466af25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4664152670 0x7f4664157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f466af25640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4664152a50 0x7f46641ece60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f466af25640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640ab850 con 0x7f46640702d0 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f466af25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466414d7b0 con 0x7f4664152670 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f466af25640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640ab570 con 0x7f4664152a50 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f4668cc1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 0x7f46641de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f4668cc1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 0x7f46641de4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50382/0 (socket says 172.21.15.78:50382) 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f4668cc1640 1 -- 172.21.15.78:0/3161977202 learned_addr learned my addr 172.21.15.78:0/3161977202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f46694c2640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4664152a50 0x7f46641ece60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.275+0000 7f4663fff640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4664152670 0x7f4664157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f4668cc1640 1 -- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4664152a50 msgr2=0x7f46641ece60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f4668cc1640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4664152a50 0x7f46641ece60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f4668cc1640 1 -- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4664152670 msgr2=0x7f4664157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f4668cc1640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4664152670 0x7f4664157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f4668cc1640 1 -- 172.21.15.78:0/3161977202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4654009400 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f4668cc1640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 0x7f46641de4d0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f465400ebe0 tx=0x7f465400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654009760 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465400cc30 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46580094d0 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465400eee0 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.276+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46641ee4e0 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.277+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46640676a0 con 0x7f46640702d0 2024-02-19T15:01:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.277+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4654013470 con 0x7f46640702d0 2024-02-19T15:01:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.278+0000 7f46617fa640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f465006b810 0x7f465006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.278+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f465408a530 con 0x7f46640702d0 2024-02-19T15:01:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.278+0000 7f4663fff640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f465006b810 0x7f465006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:19.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.278+0000 7f4663fff640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f465006b810 0x7f465006dcd0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4658036000 tx=0x7f465803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:19.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.282+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4654054f00 con 0x7f46640702d0 2024-02-19T15:01:19.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.490+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4664154fb0 con 0x7f46640702d0 2024-02-19T15:01:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.491+0000 7f46617fa640 1 -- 172.21.15.78:0/3161977202 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f4664154fb0 con 0x7f46640702d0 2024-02-19T15:01:19.495 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f465006b810 msgr2=0x7f465006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f465006b810 0x7f465006dcd0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4658036000 tx=0x7f465803c040 comp rx=0 tx=0).stop 2024-02-19T15:01:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 msgr2=0x7f46641de4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 0x7f46641de4d0 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f465400ebe0 tx=0x7f465400c520 comp rx=0 tx=0).stop 2024-02-19T15:01:19.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 shutdown_connections 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f465006b810 0x7f465006dcd0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4664152a50 0x7f46641ece60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4664152670 0x7f4664157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 --2- 172.21.15.78:0/3161977202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46640702d0 0x7f46641de4d0 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 >> 172.21.15.78:0/3161977202 conn(0x7f46640c85e0 msgr2=0x7f46640c47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 shutdown_connections 2024-02-19T15:01:19.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:19.493+0000 7f466af25640 1 -- 172.21.15.78:0/3161977202 wait complete. 2024-02-19T15:01:19.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:19.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:20.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.027+0000 7f65ba27b640 1 Processor -- start 2024-02-19T15:01:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.027+0000 7f65ba27b640 1 -- start start 2024-02-19T15:01:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65ba27b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65ba27b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65b40c1dd0 0x7f65b40c2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65ba27b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b4155a50 con 0x7f65b40c19f0 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65ba27b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b4152af0 con 0x7f65b40c1dd0 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65ba27b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4155f80 con 0x7f65b4071410 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b3fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f65b40c19f0 0x7f65b40c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:54392/0 (socket says 172.21.15.78:54392) 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b3fff640 1 -- 172.21.15.78:0/505086970 learned_addr learned my addr 172.21.15.78:0/505086970 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b8818640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65b40c1dd0 0x7f65b40c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b2ffd640 1 -- 172.21.15.78:0/505086970 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801231622 0 0) 0x7f65b4155a50 con 0x7f65b40c19f0 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b2ffd640 1 -- 172.21.15.78:0/505086970 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659c003520 con 0x7f65b40c19f0 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b37fe640 1 -- 172.21.15.78:0/505086970 >> v2:172.21.15.150:3300/0 conn(0x7f65b40c1dd0 msgr2=0x7f65b40c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b37fe640 1 --2- 172.21.15.78:0/505086970 >> v2:172.21.15.150:3300/0 conn(0x7f65b40c1dd0 0x7f65b40c2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.028+0000 7f65b37fe640 1 -- 172.21.15.78:0/505086970 >> v1:172.21.15.150:6789/0 conn(0x7f65b40c19f0 legacy=0x7f65b40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.029+0000 7f65b37fe640 1 -- 172.21.15.78:0/505086970 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4070840 con 0x7f65b4071410 2024-02-19T15:01:21.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.029+0000 7f65b8818640 1 --2- 172.21.15.78:0/505086970 >> v2:172.21.15.150:3300/0 conn(0x7f65b40c1dd0 0x7f65b40c2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.029+0000 7f65b37fe640 1 --2- 172.21.15.78:0/505086970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4156440 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f65a80097a0 tx=0x7f65a802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5101ea316590b08 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65b2ffd640 1 -- 172.21.15.78:0/505086970 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a802f050 con 0x7f65b4071410 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65b2ffd640 1 -- 172.21.15.78:0/505086970 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8031700 con 0x7f65b4071410 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65b2ffd640 1 -- 172.21.15.78:0/505086970 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8031a90 con 0x7f65b4071410 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 -- 172.21.15.78:0/505086970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 msgr2=0x7f65b4156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 --2- 172.21.15.78:0/505086970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4156440 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f65a80097a0 tx=0x7f65a802b7a0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 -- 172.21.15.78:0/505086970 shutdown_connections 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 --2- 172.21.15.78:0/505086970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4156440 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 --2- 172.21.15.78:0/505086970 >> v2:172.21.15.150:3300/0 conn(0x7f65b40c1dd0 0x7f65b40c2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 -- 172.21.15.78:0/505086970 >> 172.21.15.78:0/505086970 conn(0x7f65b40aa370 msgr2=0x7f65b40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 -- 172.21.15.78:0/505086970 shutdown_connections 2024-02-19T15:01:21.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.030+0000 7f65ba27b640 1 -- 172.21.15.78:0/505086970 wait complete. 2024-02-19T15:01:21.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.031+0000 7f65ba27b640 1 Processor -- start 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 -- start start 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65b40c19f0 0x7f65b40c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65b40c1dd0 0x7f65b41ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b41e2420 con 0x7f65b40c1dd0 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b41e1f30 con 0x7f65b40c19f0 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.032+0000 7f65ba27b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4151b20 con 0x7f65b4071410 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b8818640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65b40c1dd0 0x7f65b41ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4151e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:50114/0 (socket says 172.21.15.78:50114) 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65b40c19f0 0x7f65b40c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 -- 172.21.15.78:0/316345528 learned_addr learned my addr 172.21.15.78:0/316345528 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:21.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 -- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65b40c19f0 msgr2=0x7f65b40c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65b40c19f0 0x7f65b40c4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 -- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65b40c1dd0 msgr2=0x7f65b41ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65b40c1dd0 0x7f65b41ee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 -- 172.21.15.78:0/316345528 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac009a70 con 0x7f65b4071410 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b8818640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65b40c1dd0 0x7f65b41ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b3fff640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4151e00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f65a400c430 tx=0x7f65a400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4005ce0 con 0x7f65b4071410 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a4005e40 con 0x7f65b4071410 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.033+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a8009400 con 0x7f65b4071410 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.034+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a40072f0 con 0x7f65b4071410 2024-02-19T15:01:21.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.034+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b40ad260 con 0x7f65b4071410 2024-02-19T15:01:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.034+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b40c8320 con 0x7f65b4071410 2024-02-19T15:01:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.035+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f65a4007450 con 0x7f65b4071410 2024-02-19T15:01:21.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.035+0000 7f65b0ff9640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f65a006b840 0x7f65a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.035+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f65a4089830 con 0x7f65b4071410 2024-02-19T15:01:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.035+0000 7f65b37fe640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f65a006b840 0x7f65a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:21.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.036+0000 7f65b37fe640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f65a006b840 0x7f65a006dd00 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f65a8035000 tx=0x7f65a8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.041+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f65a40541a0 con 0x7f65b4071410 2024-02-19T15:01:21.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.248+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b40ad0c0 con 0x7f65b4071410 2024-02-19T15:01:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.249+0000 7f65b0ff9640 1 -- 172.21.15.78:0/316345528 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f65b40ad0c0 con 0x7f65b4071410 2024-02-19T15:01:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:21.253 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f65a006b840 msgr2=0x7f65a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f65a006b840 0x7f65a006dd00 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f65a8035000 tx=0x7f65a8008030 comp rx=0 tx=0).stop 2024-02-19T15:01:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 msgr2=0x7f65b4151e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4151e00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f65a400c430 tx=0x7f65a400c830 comp rx=0 tx=0).stop 2024-02-19T15:01:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 shutdown_connections 2024-02-19T15:01:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f65a006b840 0x7f65a006dd00 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65b40c1dd0 0x7f65b41ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65b40c19f0 0x7f65b40c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 --2- 172.21.15.78:0/316345528 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65b4071410 0x7f65b4151e00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 >> 172.21.15.78:0/316345528 conn(0x7f65b40aa370 msgr2=0x7f65b406d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 shutdown_connections 2024-02-19T15:01:21.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:21.251+0000 7f65ba27b640 1 -- 172.21.15.78:0/316345528 wait complete. 2024-02-19T15:01:21.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:21.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:22.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:22.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.780+0000 7f3d9ede4640 1 Processor -- start 2024-02-19T15:01:22.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.780+0000 7f3d9ede4640 1 -- start start 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9ede4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3d980c8500 0x7f3d980c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9ede4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d980c34a0 con 0x7f3d980c2f50 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9ede4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9806c8e0 con 0x7f3d980c42e0 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9ede4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9806c090 con 0x7f3d980c8500 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9cb80640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3d980c8500 0x7f3d980c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9d381640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3d980c2f50 0x7f3d9815f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37824/0 (socket says 172.21.15.78:37824) 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9d381640 1 -- 172.21.15.78:0/580063852 learned_addr learned my addr 172.21.15.78:0/580063852 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:22.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9cb80640 1 -- 172.21.15.78:0/580063852 >> v1:172.21.15.150:6789/0 conn(0x7f3d980c42e0 legacy=0x7f3d98158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:22.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9cb80640 1 -- 172.21.15.78:0/580063852 >> v1:172.21.15.78:6789/0 conn(0x7f3d980c2f50 legacy=0x7f3d9815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:22.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d9cb80640 1 -- 172.21.15.78:0/580063852 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9806cbc0 con 0x7f3d980c8500 2024-02-19T15:01:22.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.781+0000 7f3d977fe640 1 -- 172.21.15.78:0/580063852 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604506139 0 0) 0x7f3d980c34a0 con 0x7f3d980c2f50 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.782+0000 7f3d9cb80640 1 --2- 172.21.15.78:0/580063852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c8500 0x7f3d980c88e0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b18540fc8215e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.782+0000 7f3d977fe640 1 -- 172.21.15.78:0/580063852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c031830 con 0x7f3d980c8500 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.782+0000 7f3d977fe640 1 -- 172.21.15.78:0/580063852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c031990 con 0x7f3d980c8500 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.782+0000 7f3d977fe640 1 -- 172.21.15.78:0/580063852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c031d20 con 0x7f3d980c8500 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 -- 172.21.15.78:0/580063852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c8500 msgr2=0x7f3d980c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/580063852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c8500 0x7f3d980c88e0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0097a0 tx=0x7f3d8c02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 -- 172.21.15.78:0/580063852 shutdown_connections 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/580063852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c8500 0x7f3d980c88e0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:22.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 -- 172.21.15.78:0/580063852 >> 172.21.15.78:0/580063852 conn(0x7f3d980aa300 msgr2=0x7f3d980aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:22.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 -- 172.21.15.78:0/580063852 shutdown_connections 2024-02-19T15:01:22.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.783+0000 7f3d9ede4640 1 -- 172.21.15.78:0/580063852 wait complete. 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.784+0000 7f3d9ede4640 1 Processor -- start 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.784+0000 7f3d9ede4640 1 -- start start 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9ede4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 0x7f3d981eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9ede4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d980c8500 0x7f3d981e0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9ede4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d981df070 0x7f3d9814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9ede4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d981dfec0 con 0x7f3d980c2f50 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9ede4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9806fa10 con 0x7f3d980c8500 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9ede4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d980a8250 con 0x7f3d981df070 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 0x7f3d981eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 0x7f3d981eee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50404/0 (socket says 172.21.15.78:50404) 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 -- 172.21.15.78:0/13154323 learned_addr learned my addr 172.21.15.78:0/13154323 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:22.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9d381640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d981df070 0x7f3d9814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d97fff640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d980c8500 0x7f3d981e0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 -- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d981df070 msgr2=0x7f3d9814b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d981df070 0x7f3d9814b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 -- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d980c8500 msgr2=0x7f3d981e0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d980c8500 0x7f3d981e0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 -- 172.21.15.78:0/13154323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c009400 con 0x7f3d980c2f50 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d97fff640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d980c8500 0x7f3d981e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9d381640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d981df070 0x7f3d9814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.785+0000 7f3d9cb80640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 0x7f3d981eee60 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c000c00 tx=0x7f3d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.786+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c03aa60 con 0x7f3d980c2f50 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.786+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c03abc0 con 0x7f3d980c2f50 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.786+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d981ef3a0 con 0x7f3d980c2f50 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.786+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c0443f0 con 0x7f3d980c2f50 2024-02-19T15:01:22.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.786+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d981de660 con 0x7f3d980c2f50 2024-02-19T15:01:22.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.787+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d98071c20 con 0x7f3d980c2f50 2024-02-19T15:01:22.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.787+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3d8c04d450 con 0x7f3d980c2f50 2024-02-19T15:01:22.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.787+0000 7f3d957fa640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d7c06b810 0x7f3d7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:22.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.788+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d8c0ac420 con 0x7f3d980c2f50 2024-02-19T15:01:22.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.788+0000 7f3d97fff640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d7c06b810 0x7f3d7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:22.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.788+0000 7f3d97fff640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d7c06b810 0x7f3d7c06dcd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3d84007fd0 tx=0x7f3d84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:22.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.793+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d8c076e60 con 0x7f3d980c2f50 2024-02-19T15:01:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.999+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d981ef3a0 con 0x7f3d980c2f50 2024-02-19T15:01:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:22.999+0000 7f3d957fa640 1 -- 172.21.15.78:0/13154323 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f3d981ef3a0 con 0x7f3d980c2f50 2024-02-19T15:01:23.003 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d7c06b810 msgr2=0x7f3d7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d7c06b810 0x7f3d7c06dcd0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3d84007fd0 tx=0x7f3d84009040 comp rx=0 tx=0).stop 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 msgr2=0x7f3d981eee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 0x7f3d981eee60 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c000c00 tx=0x7f3d8c0046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 shutdown_connections 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3d7c06b810 0x7f3d7c06dcd0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d981df070 0x7f3d9814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d980c8500 0x7f3d981e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.001+0000 7f3d9ede4640 1 --2- 172.21.15.78:0/13154323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d980c2f50 0x7f3d981eee60 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.002+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 >> 172.21.15.78:0/13154323 conn(0x7f3d980aa300 msgr2=0x7f3d98154c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.002+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 shutdown_connections 2024-02-19T15:01:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:23.002+0000 7f3d9ede4640 1 -- 172.21.15.78:0/13154323 wait complete. 2024-02-19T15:01:23.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:23.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:24.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:24.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.526+0000 7f7f9912c640 1 Processor -- start 2024-02-19T15:01:24.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.526+0000 7f7f9912c640 1 -- start start 2024-02-19T15:01:24.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9912c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f9406b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:24.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9912c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f940c19f0 0x7f7f9415f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9912c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f94072180 con 0x7f7f9406ae80 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9912c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9406fdd0 con 0x7f7f940c19f0 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9912c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f94073440 con 0x7f7f94155af0 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f940c19f0 0x7f7f9415f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9259c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f9406b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f92d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7f9406ae80 0x7f7f9406f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:54406/0 (socket says 172.21.15.78:54406) 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f92d9d640 1 -- 172.21.15.78:0/1707942235 learned_addr learned my addr 172.21.15.78:0/1707942235 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f91d9b640 1 -- 172.21.15.78:0/1707942235 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430364438 0 0) 0x7f7f94072180 con 0x7f7f9406ae80 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f91d9b640 1 -- 172.21.15.78:0/1707942235 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c003520 con 0x7f7f9406ae80 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9359e640 1 -- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 msgr2=0x7f7f9406b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9359e640 1 --2- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f9406b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9359e640 1 -- 172.21.15.78:0/1707942235 >> v1:172.21.15.150:6789/0 conn(0x7f7f9406ae80 legacy=0x7f7f9406f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9359e640 1 -- 172.21.15.78:0/1707942235 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9414f810 con 0x7f7f940c19f0 2024-02-19T15:01:24.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.527+0000 7f7f9259c640 1 --2- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f9406b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.528+0000 7f7f9359e640 1 --2- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f9415f910 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c009d90 tx=0x7f7f8c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1386333f5104a489 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.528+0000 7f7f91d9b640 1 -- 172.21.15.78:0/1707942235 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c02cb30 con 0x7f7f940c19f0 2024-02-19T15:01:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.528+0000 7f7f91d9b640 1 -- 172.21.15.78:0/1707942235 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c02cc90 con 0x7f7f940c19f0 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.528+0000 7f7f91d9b640 1 -- 172.21.15.78:0/1707942235 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c034440 con 0x7f7f940c19f0 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 -- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 msgr2=0x7f7f9415f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 --2- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f9415f910 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c009d90 tx=0x7f7f8c02bd80 comp rx=0 tx=0).stop 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 -- 172.21.15.78:0/1707942235 shutdown_connections 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 --2- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f9406b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 --2- 172.21.15.78:0/1707942235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f9415f910 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 -- 172.21.15.78:0/1707942235 >> 172.21.15.78:0/1707942235 conn(0x7f7f940c7590 msgr2=0x7f7f940c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 -- 172.21.15.78:0/1707942235 shutdown_connections 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.529+0000 7f7f9912c640 1 -- 172.21.15.78:0/1707942235 wait complete. 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.530+0000 7f7f9912c640 1 Processor -- start 2024-02-19T15:01:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.530+0000 7f7f9912c640 1 -- start start 2024-02-19T15:01:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9912c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 0x7f7f940a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9912c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f94150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9912c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f940a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9912c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f94159590 con 0x7f7f9406ae80 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9912c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940a78c0 con 0x7f7f940c19f0 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9912c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f94158fd0 con 0x7f7f94155af0 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f92d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 0x7f7f940a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f92d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 0x7f7f940a8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50418/0 (socket says 172.21.15.78:50418) 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f92d9d640 1 -- 172.21.15.78:0/454465394 learned_addr learned my addr 172.21.15.78:0/454465394 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9259c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f94150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.531+0000 7f7f9359e640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f940a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f92d9d640 1 -- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 msgr2=0x7f7f940a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f92d9d640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f940a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f92d9d640 1 -- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 msgr2=0x7f7f94150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f92d9d640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f94150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f92d9d640 1 -- 172.21.15.78:0/454465394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88009400 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f9259c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f94150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f92d9d640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 0x7f7f940a8610 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400c430 tx=0x7f7f8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f84005ce0 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f84005e40 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c009a70 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f840072f0 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.532+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9414a910 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.533+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f940a6f00 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.533+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7f84007450 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.534+0000 7f7f7b7fe640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f8006b840 0x7f7f8006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.534+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7f84089880 con 0x7f7f9406ae80 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.534+0000 7f7f9259c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f8006b840 0x7f7f8006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:24.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.534+0000 7f7f9259c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f8006b840 0x7f7f8006dd00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7f9414b010 tx=0x7f7f88009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:24.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.538+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7f84054250 con 0x7f7f9406ae80 2024-02-19T15:01:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.748+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f94074990 con 0x7f7f9406ae80 2024-02-19T15:01:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.749+0000 7f7f7b7fe640 1 -- 172.21.15.78:0/454465394 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f7f84017090 con 0x7f7f9406ae80 2024-02-19T15:01:24.752 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:24.752 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f8006b840 msgr2=0x7f7f8006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f8006b840 0x7f7f8006dd00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7f9414b010 tx=0x7f7f88009190 comp rx=0 tx=0).stop 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 msgr2=0x7f7f940a8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 0x7f7f940a8610 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400c430 tx=0x7f7f8400c8f0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 shutdown_connections 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7f8006b840 0x7f7f8006dd00 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f94155af0 0x7f7f940a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f940c19f0 0x7f7f94150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 --2- 172.21.15.78:0/454465394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f9406ae80 0x7f7f940a8610 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 >> 172.21.15.78:0/454465394 conn(0x7f7f940c7590 msgr2=0x7f7f94153130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 shutdown_connections 2024-02-19T15:01:24.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:24.751+0000 7f7f9912c640 1 -- 172.21.15.78:0/454465394 wait complete. 2024-02-19T15:01:25.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:25.167 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:26.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.277+0000 7f5f0336c640 1 Processor -- start 2024-02-19T15:01:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f0336c640 1 -- start start 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f0336c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5efc14cd50 0x7f5efc0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f0336c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f0336c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc1563b0 con 0x7f5efc0c7b70 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f0336c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5efc153cd0 con 0x7f5efc14cd50 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f0336c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc06b6e0 con 0x7f5efc1537a0 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f01108640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5efc14cd50 0x7f5efc0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f01909640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5efc0c7b70 0x7f5efc156860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37828/0 (socket says 172.21.15.78:37828) 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f01108640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5efc14cd50 0x7f5efc0c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50422/0 (socket says 172.21.15.78:50422) 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.278+0000 7f5f01909640 1 -- 172.21.15.78:0/3842968710 learned_addr learned my addr 172.21.15.78:0/3842968710 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.279+0000 7f5f00907640 1 --2- 172.21.15.78:0/3842968710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.279+0000 7f5f01108640 1 -- 172.21.15.78:0/3842968710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 msgr2=0x7f5efc0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.279+0000 7f5f01108640 1 --2- 172.21.15.78:0/3842968710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc0c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.279+0000 7f5f01108640 1 -- 172.21.15.78:0/3842968710 >> v1:172.21.15.78:6789/0 conn(0x7f5efc0c7b70 legacy=0x7f5efc156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.279+0000 7f5f01108640 1 -- 172.21.15.78:0/3842968710 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc06b980 con 0x7f5efc14cd50 2024-02-19T15:01:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.279+0000 7f5ef3fff640 1 -- 172.21.15.78:0/3842968710 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186157022 0 0) 0x7f5efc1563b0 con 0x7f5efc0c7b70 2024-02-19T15:01:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f01108640 1 --2- 172.21.15.78:0/3842968710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc0c4cd0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009dc0 tx=0x7f5ef802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cec0e69a7f70d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5ef3fff640 1 -- 172.21.15.78:0/3842968710 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef802cb20 con 0x7f5efc14cd50 2024-02-19T15:01:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5ef3fff640 1 -- 172.21.15.78:0/3842968710 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef802cc80 con 0x7f5efc14cd50 2024-02-19T15:01:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5ef3fff640 1 -- 172.21.15.78:0/3842968710 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8034440 con 0x7f5efc14cd50 2024-02-19T15:01:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 -- 172.21.15.78:0/3842968710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 msgr2=0x7f5efc0c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 --2- 172.21.15.78:0/3842968710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc0c4cd0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009dc0 tx=0x7f5ef802bd70 comp rx=0 tx=0).stop 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 -- 172.21.15.78:0/3842968710 shutdown_connections 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 --2- 172.21.15.78:0/3842968710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc0c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 --2- 172.21.15.78:0/3842968710 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc0c4cd0 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 -- 172.21.15.78:0/3842968710 >> 172.21.15.78:0/3842968710 conn(0x7f5efc066e70 msgr2=0x7f5efc0a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 -- 172.21.15.78:0/3842968710 shutdown_connections 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.280+0000 7f5f0336c640 1 -- 172.21.15.78:0/3842968710 wait complete. 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 Processor -- start 2024-02-19T15:01:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 -- start start 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5efc0c7b70 0x7f5efc14a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc1dedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc1df310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1ea1a0 con 0x7f5efc14cd50 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1ea760 con 0x7f5efc0c7b70 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f0336c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc1df850 con 0x7f5efc1537a0 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f00907640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc1dedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f00907640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc1dedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50434/0 (socket says 172.21.15.78:50434) 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f00907640 1 -- 172.21.15.78:0/2229442896 learned_addr learned my addr 172.21.15.78:0/2229442896 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.282+0000 7f5f01108640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5efc0c7b70 0x7f5efc14a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f01909640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc1df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f00907640 1 -- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 msgr2=0x7f5efc1df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f00907640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc1df310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f00907640 1 -- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5efc0c7b70 msgr2=0x7f5efc14a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f00907640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5efc0c7b70 0x7f5efc14a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f00907640 1 -- 172.21.15.78:0/2229442896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef8009a70 con 0x7f5efc14cd50 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f00907640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc1dedd0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00e870 tx=0x7f5eec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec0094f0 con 0x7f5efc14cd50 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eec00c8e0 con 0x7f5efc14cd50 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc1ecd50 con 0x7f5efc14cd50 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eec00cc70 con 0x7f5efc14cd50 2024-02-19T15:01:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.283+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc1ea3f0 con 0x7f5efc14cd50 2024-02-19T15:01:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.284+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc1555b0 con 0x7f5efc14cd50 2024-02-19T15:01:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.285+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f5eec013470 con 0x7f5efc14cd50 2024-02-19T15:01:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.285+0000 7f5ef1ffb640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ee006b810 0x7f5ee006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:26.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.285+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5eec08a370 con 0x7f5efc14cd50 2024-02-19T15:01:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.285+0000 7f5f01108640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ee006b810 0x7f5ee006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:26.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.286+0000 7f5f01108640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ee006b810 0x7f5ee006dcd0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8036000 tx=0x7f5ef8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:26.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.290+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5eec054cb0 con 0x7f5efc14cd50 2024-02-19T15:01:26.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.498+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5efc0c4650 con 0x7f5efc14cd50 2024-02-19T15:01:26.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.498+0000 7f5ef1ffb640 1 -- 172.21.15.78:0/2229442896 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f5eec058370 con 0x7f5efc14cd50 2024-02-19T15:01:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:26.501 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:26.501 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ee006b810 msgr2=0x7f5ee006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ee006b810 0x7f5ee006dcd0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8036000 tx=0x7f5ef8030000 comp rx=0 tx=0).stop 2024-02-19T15:01:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 msgr2=0x7f5efc1dedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc1dedd0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00e870 tx=0x7f5eec00ed30 comp rx=0 tx=0).stop 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 shutdown_connections 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f5ee006b810 0x7f5ee006dcd0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5efc14cd50 0x7f5efc1dedd0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5efc1537a0 0x7f5efc1df310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 --2- 172.21.15.78:0/2229442896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5efc0c7b70 0x7f5efc14a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 >> 172.21.15.78:0/2229442896 conn(0x7f5efc066e70 msgr2=0x7f5efc1522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 shutdown_connections 2024-02-19T15:01:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:26.500+0000 7f5f0336c640 1 -- 172.21.15.78:0/2229442896 wait complete. 2024-02-19T15:01:26.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:26.917 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:27.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:01:27.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:01:27.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:28.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.031+0000 7fb6a23bd640 1 Processor -- start 2024-02-19T15:01:28.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.031+0000 7fb6a23bd640 1 -- start start 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a23bd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb69c156070 0x7fb69c0c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a23bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c071c20 con 0x7fb69c0c50a0 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a23bd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0c7fe0 con 0x7fb69c071790 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a23bd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69c0c82c0 con 0x7fb69c156070 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb69bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb69c0c50a0 0x7fb69c06cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38042/0 (socket says 172.21.15.78:38042) 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb69bfff640 1 -- 172.21.15.78:0/3350971519 learned_addr learned my addr 172.21.15.78:0/3350971519 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a095a640 1 --2- 172.21.15.78:0/3350971519 >> v2:172.21.15.150:3300/0 conn(0x7fb69c156070 0x7fb69c0c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251299512 0 0) 0x7fb69c071c20 con 0x7fb69c0c50a0 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb684003520 con 0x7fb69c0c50a0 2024-02-19T15:01:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908331065 0 0) 0x7fb69c0c7fe0 con 0x7fb69c071790 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c071c20 con 0x7fb69c071790 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a095a640 1 -- 172.21.15.78:0/3350971519 >> v1:172.21.15.150:6789/0 conn(0x7fb69c071790 legacy=0x7fb69c155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a095a640 1 -- 172.21.15.78:0/3350971519 >> v1:172.21.15.78:6789/0 conn(0x7fb69c0c50a0 legacy=0x7fb69c06cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.032+0000 7fb6a095a640 1 -- 172.21.15.78:0/3350971519 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c06c700 con 0x7fb69c156070 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.033+0000 7fb6a095a640 1 --2- 172.21.15.78:0/3350971519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c156070 0x7fb69c0c73e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb694009d90 tx=0x7fb69402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a49d752c287c18c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.033+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69402cbd0 con 0x7fb69c156070 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.033+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69402cd30 con 0x7fb69c156070 2024-02-19T15:01:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.033+0000 7fb69affd640 1 -- 172.21.15.78:0/3350971519 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69402cea0 con 0x7fb69c156070 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 -- 172.21.15.78:0/3350971519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c156070 msgr2=0x7fb69c0c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/3350971519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c156070 0x7fb69c0c73e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb694009d90 tx=0x7fb69402be20 comp rx=0 tx=0).stop 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 -- 172.21.15.78:0/3350971519 shutdown_connections 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/3350971519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c156070 0x7fb69c0c73e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 -- 172.21.15.78:0/3350971519 >> 172.21.15.78:0/3350971519 conn(0x7fb69c06b700 msgr2=0x7fb69c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 -- 172.21.15.78:0/3350971519 shutdown_connections 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.034+0000 7fb6a23bd640 1 -- 172.21.15.78:0/3350971519 wait complete. 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.035+0000 7fb6a23bd640 1 Processor -- start 2024-02-19T15:01:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.035+0000 7fb6a23bd640 1 -- start start 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a23bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c0c50a0 0x7fb69c0ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a23bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 0x7fb69c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a23bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69c1eb060 0x7fb69c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a23bd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c1ea370 con 0x7fb69c156070 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a23bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c159580 con 0x7fb69c0c50a0 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a23bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c1eabd0 con 0x7fb69c1eb060 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 0x7fb69c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c0c50a0 0x7fb69c0ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 0x7fb69c158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33216/0 (socket says 172.21.15.78:33216) 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 -- 172.21.15.78:0/1489111061 learned_addr learned my addr 172.21.15.78:0/1489111061 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb6a095a640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69c1eb060 0x7fb69c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:28.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 -- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69c1eb060 msgr2=0x7fb69c1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69c1eb060 0x7fb69c1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 -- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c0c50a0 msgr2=0x7fb69c0ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c0c50a0 0x7fb69c0ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.036+0000 7fb69b7fe640 1 -- 172.21.15.78:0/1489111061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb694009a70 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.037+0000 7fb69b7fe640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 0x7fb69c158070 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c400 tx=0x7fb69000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.037+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6900052d0 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.037+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb690005430 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.037+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c0a9b80 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.037+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690003810 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.037+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c1ea580 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.038+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c1ead80 con 0x7fb69c156070 2024-02-19T15:01:28.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.038+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb690003970 con 0x7fb69c156070 2024-02-19T15:01:28.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.039+0000 7fb698ff9640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb68806b810 0x7fb68806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:28.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.039+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb690089920 con 0x7fb69c156070 2024-02-19T15:01:28.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.039+0000 7fb69bfff640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb68806b810 0x7fb68806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:28.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.039+0000 7fb69bfff640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb68806b810 0x7fb68806dcd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb68c000f60 tx=0x7fb68c007d70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:28.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.043+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb69c1ead80 con 0x7fb69c156070 2024-02-19T15:01:28.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.253+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb69c1ead40 con 0x7fb69c156070 2024-02-19T15:01:28.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.253+0000 7fb698ff9640 1 -- 172.21.15.78:0/1489111061 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7fb690054260 con 0x7fb69c156070 2024-02-19T15:01:28.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:28.256 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:28.257 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb68806b810 msgr2=0x7fb68806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb68806b810 0x7fb68806dcd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb68c000f60 tx=0x7fb68c007d70 comp rx=0 tx=0).stop 2024-02-19T15:01:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 msgr2=0x7fb69c158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 0x7fb69c158070 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c400 tx=0x7fb69000c8c0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 shutdown_connections 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb68806b810 0x7fb68806dcd0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb69c156070 0x7fb69c158070 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb69c1eb060 0x7fb69c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 --2- 172.21.15.78:0/1489111061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c0c50a0 0x7fb69c0ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 >> 172.21.15.78:0/1489111061 conn(0x7fb69c06b700 msgr2=0x7fb69c1533e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 shutdown_connections 2024-02-19T15:01:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:28.255+0000 7fb6a23bd640 1 -- 172.21.15.78:0/1489111061 wait complete. 2024-02-19T15:01:28.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:28.672 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:29.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:29.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.786+0000 7f8df00b5640 1 Processor -- start 2024-02-19T15:01:29.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.786+0000 7f8df00b5640 1 -- start start 2024-02-19T15:01:29.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8df00b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8de8152700 0x7f8de80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:29.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8df00b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8de80c5730 0x7f8de806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:29.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8df00b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de81551f0 con 0x7f8de80c8e00 2024-02-19T15:01:29.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8df00b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de8155490 con 0x7f8de8152700 2024-02-19T15:01:29.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8df00b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de8155770 con 0x7f8de80c5730 2024-02-19T15:01:29.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8dede51640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8de8152700 0x7f8de80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8dede51640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8de8152700 0x7f8de80c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33222/0 (socket says 172.21.15.78:33222) 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8dede51640 1 -- 172.21.15.78:0/2629820611 learned_addr learned my addr 172.21.15.78:0/2629820611 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8ded650640 1 --2- 172.21.15.78:0/2629820611 >> v2:172.21.15.150:3300/0 conn(0x7f8de80c5730 0x7f8de806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.787+0000 7f8dede51640 1 -- 172.21.15.78:0/2629820611 >> v2:172.21.15.150:3300/0 conn(0x7f8de80c5730 msgr2=0x7f8de806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.788+0000 7f8dede51640 1 --2- 172.21.15.78:0/2629820611 >> v2:172.21.15.150:3300/0 conn(0x7f8de80c5730 0x7f8de806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.788+0000 7f8dede51640 1 -- 172.21.15.78:0/2629820611 >> v1:172.21.15.150:6789/0 conn(0x7f8de80c8e00 legacy=0x7f8de8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:29.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.788+0000 7f8dede51640 1 -- 172.21.15.78:0/2629820611 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8155a50 con 0x7f8de8152700 2024-02-19T15:01:29.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.788+0000 7f8dede51640 1 --2- 172.21.15.78:0/2629820611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c51f0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f8de80c7620 tx=0x7f8de402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b1eb3fe8d17d5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:29.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8dece4f640 1 -- 172.21.15.78:0/2629820611 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de402ca80 con 0x7f8de8152700 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8dece4f640 1 -- 172.21.15.78:0/2629820611 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de402cbe0 con 0x7f8de8152700 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8dece4f640 1 -- 172.21.15.78:0/2629820611 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40343f0 con 0x7f8de8152700 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8df00b5640 1 -- 172.21.15.78:0/2629820611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 msgr2=0x7f8de80c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8df00b5640 1 --2- 172.21.15.78:0/2629820611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c51f0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f8de80c7620 tx=0x7f8de402bee0 comp rx=0 tx=0).stop 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8df00b5640 1 -- 172.21.15.78:0/2629820611 shutdown_connections 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8df00b5640 1 --2- 172.21.15.78:0/2629820611 >> v2:172.21.15.150:3300/0 conn(0x7f8de80c5730 0x7f8de806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8df00b5640 1 --2- 172.21.15.78:0/2629820611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c51f0 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.789+0000 7f8df00b5640 1 -- 172.21.15.78:0/2629820611 >> 172.21.15.78:0/2629820611 conn(0x7f8de80cac90 msgr2=0x7f8de80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.790+0000 7f8df00b5640 1 -- 172.21.15.78:0/2629820611 shutdown_connections 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.790+0000 7f8df00b5640 1 -- 172.21.15.78:0/2629820611 wait complete. 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 Processor -- start 2024-02-19T15:01:29.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 -- start start 2024-02-19T15:01:29.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de80c5730 0x7f8de81eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:29.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8de80c8e00 0x7f8de80c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80be340 con 0x7f8de8152700 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de80bdf60 con 0x7f8de80c5730 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.791+0000 7f8df00b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81eb540 con 0x7f8de80c8e00 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dede51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de80c5730 0x7f8de81eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33228/0 (socket says 172.21.15.78:33228) 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 -- 172.21.15.78:0/3770367773 learned_addr learned my addr 172.21.15.78:0/3770367773 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8ded650640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8de80c8e00 0x7f8de80c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 -- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8de80c8e00 msgr2=0x7f8de80c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8de80c8e00 0x7f8de80c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 -- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de80c5730 msgr2=0x7f8de81eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de80c5730 0x7f8de81eb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 -- 172.21.15.78:0/3770367773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4009b30 con 0x7f8de8152700 2024-02-19T15:01:29.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dede51640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de80c5730 0x7f8de81eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dee652640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c2950 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f8de0001eb0 tx=0x7f8de000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de00052d0 con 0x7f8de8152700 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de0005430 con 0x7f8de8152700 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de80be5b0 con 0x7f8de8152700 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0003810 con 0x7f8de8152700 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.792+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de80c03c0 con 0x7f8de8152700 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.793+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de80688f0 con 0x7f8de8152700 2024-02-19T15:01:29.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.794+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8de0003970 con 0x7f8de8152700 2024-02-19T15:01:29.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.794+0000 7f8dde7fc640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8dcc06b840 0x7f8dcc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:29.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.794+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8de0089700 con 0x7f8de8152700 2024-02-19T15:01:29.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.794+0000 7f8dede51640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8dcc06b840 0x7f8dcc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:29.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.795+0000 7f8dede51640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8dcc06b840 0x7f8dcc06dd00 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8de4035000 tx=0x7f8de4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:29.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:29.799+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8de00540d0 con 0x7f8de8152700 2024-02-19T15:01:30.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.005+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8de80be0a0 con 0x7f8de8152700 2024-02-19T15:01:30.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.005+0000 7f8dde7fc640 1 -- 172.21.15.78:0/3770367773 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4974 (secure 0 0 0) 0x7f8de0057790 con 0x7f8de8152700 2024-02-19T15:01:30.009 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:30.009 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:03.415074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10719":{"gid":10719,"name":"c","rank":1,"incarnation":192,"state":"up:stopping","state_seq":5,"addr":"172.21.15.78:6837/159409097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":159409097},{"type":"v1","addr":"172.21.15.78:6837","nonce":159409097}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:30.009 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 195 2024-02-19T15:01:30.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8dcc06b840 msgr2=0x7f8dcc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:30.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8dcc06b840 0x7f8dcc06dd00 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8de4035000 tx=0x7f8de4030000 comp rx=0 tx=0).stop 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 msgr2=0x7f8de80c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c2950 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f8de0001eb0 tx=0x7f8de000c890 comp rx=0 tx=0).stop 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 shutdown_connections 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8dcc06b840 0x7f8dcc06dd00 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8de8152700 0x7f8de80c2950 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8de80c8e00 0x7f8de80c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 --2- 172.21.15.78:0/3770367773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8de80c5730 0x7f8de81eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.007+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 >> 172.21.15.78:0/3770367773 conn(0x7f8de80cac90 msgr2=0x7f8de80cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.008+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 shutdown_connections 2024-02-19T15:01:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:30.008+0000 7f8df00b5640 1 -- 172.21.15.78:0/3770367773 wait complete. 2024-02-19T15:01:30.424 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:03.415074+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10719}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10719': {'gid': 10719, 'name': 'c', 'rank': 1, 'incarnation': 192, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.78:6837/159409097', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6836', 'nonce': 159409097}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 159409097}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:30.424 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10719:up:stopping 2024-02-19T15:01:31.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.532+0000 7f3fc9947640 1 Processor -- start 2024-02-19T15:01:31.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc9947640 1 -- start start 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc9947640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3fc40c7700 0x7f3fc406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc9947640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc41586a0 con 0x7f3fc4152830 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc9947640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc4158980 con 0x7f3fc40c7280 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc9947640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc40c6550 con 0x7f3fc40c7700 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc2ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3fc4152830 0x7f3fc40c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38044/0 (socket says 172.21.15.78:38044) 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.533+0000 7f3fc2ffd640 1 -- 172.21.15.78:0/2356245461 learned_addr learned my addr 172.21.15.78:0/2356245461 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc37fe640 1 --2- 172.21.15.78:0/2356245461 >> v2:172.21.15.150:3300/0 conn(0x7f3fc40c7700 0x7f3fc406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837308995 0 0) 0x7f3fc41586a0 con 0x7f3fc4152830 2024-02-19T15:01:31.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fac003520 con 0x7f3fc4152830 2024-02-19T15:01:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007089195 0 0) 0x7f3fc4158980 con 0x7f3fc40c7280 2024-02-19T15:01:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc41586a0 con 0x7f3fc40c7280 2024-02-19T15:01:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc37fe640 1 -- 172.21.15.78:0/2356245461 >> v1:172.21.15.150:6789/0 conn(0x7f3fc40c7280 legacy=0x7f3fc406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc37fe640 1 -- 172.21.15.78:0/2356245461 >> v1:172.21.15.78:6789/0 conn(0x7f3fc4152830 legacy=0x7f3fc40c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.534+0000 7f3fc37fe640 1 -- 172.21.15.78:0/2356245461 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc4158c60 con 0x7f3fc40c7700 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc37fe640 1 --2- 172.21.15.78:0/2356245461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc40c7700 0x7f3fc406cc80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0097a0 tx=0x7f3fbc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13d8b10d1f3d2ac4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc031830 con 0x7f3fc40c7700 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fbc031990 con 0x7f3fc40c7700 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc1ffb640 1 -- 172.21.15.78:0/2356245461 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc031d20 con 0x7f3fc40c7700 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 -- 172.21.15.78:0/2356245461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc40c7700 msgr2=0x7f3fc406cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 --2- 172.21.15.78:0/2356245461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc40c7700 0x7f3fc406cc80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0097a0 tx=0x7f3fbc02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 -- 172.21.15.78:0/2356245461 shutdown_connections 2024-02-19T15:01:31.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 --2- 172.21.15.78:0/2356245461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc40c7700 0x7f3fc406cc80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 -- 172.21.15.78:0/2356245461 >> 172.21.15.78:0/2356245461 conn(0x7f3fc40ca8a0 msgr2=0x7f3fc40cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:31.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 -- 172.21.15.78:0/2356245461 shutdown_connections 2024-02-19T15:01:31.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.535+0000 7f3fc9947640 1 -- 172.21.15.78:0/2356245461 wait complete. 2024-02-19T15:01:31.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.536+0000 7f3fc9947640 1 Processor -- start 2024-02-19T15:01:31.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 -- start start 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fc40c7280 0x7f3fc40abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 0x7f3fc4148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc4152830 0x7f3fc4148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4159320 con 0x7f3fc40c7700 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c3c40 con 0x7f3fc4152830 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc9947640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c0c30 con 0x7f3fc40c7280 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.537+0000 7f3fc27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 0x7f3fc4148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 0x7f3fc4148a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33242/0 (socket says 172.21.15.78:33242) 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 -- 172.21.15.78:0/324302441 learned_addr learned my addr 172.21.15.78:0/324302441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc37fe640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc4152830 0x7f3fc4148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc2ffd640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fc40c7280 0x7f3fc40abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:31.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 -- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fc40c7280 msgr2=0x7f3fc40abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fc40c7280 0x7f3fc40abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 -- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc4152830 msgr2=0x7f3fc4148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc4152830 0x7f3fc4148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 -- 172.21.15.78:0/324302441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc009400 con 0x7f3fc40c7700 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc27fc640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 0x7f3fc4148a00 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c500 tx=0x7f3fb800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80052d0 con 0x7f3fc40c7700 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb8005450 con 0x7f3fc40c7700 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc40c3960 con 0x7f3fc40c7700 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8012870 con 0x7f3fc40c7700 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.538+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc40c1140 con 0x7f3fc40c7700 2024-02-19T15:01:31.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.539+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc406d140 con 0x7f3fc40c7700 2024-02-19T15:01:31.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.540+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3fb80129d0 con 0x7f3fc40c7700 2024-02-19T15:01:31.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.540+0000 7f3fabfff640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b810 0x7f3fb006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:31.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.540+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3fb8089ea0 con 0x7f3fc40c7700 2024-02-19T15:01:31.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.540+0000 7f3fc2ffd640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b810 0x7f3fb006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:31.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.540+0000 7f3fc2ffd640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b810 0x7f3fb006dcd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4001560 tx=0x7f3fb40019f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:31.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.544+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3fb80547a0 con 0x7f3fc40c7700 2024-02-19T15:01:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.753+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc4074990 con 0x7f3fc40c7700 2024-02-19T15:01:31.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.753+0000 7f3fabfff640 1 -- 172.21.15.78:0/324302441 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+4186 (secure 0 0 0) 0x7f3fb8057e60 con 0x7f3fc40c7700 2024-02-19T15:01:31.756 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:31.756 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":197,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:30.644801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:resolve","state_seq":13,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 197 2024-02-19T15:01:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b810 msgr2=0x7f3fb006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b810 0x7f3fb006dcd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4001560 tx=0x7f3fb40019f0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 msgr2=0x7f3fc4148a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 0x7f3fc4148a00 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800c500 tx=0x7f3fb800c9c0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 shutdown_connections 2024-02-19T15:01:31.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3fb006b810 0x7f3fb006dcd0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc40c7700 0x7f3fc4148a00 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fc4152830 0x7f3fc4148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 --2- 172.21.15.78:0/324302441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3fc40c7280 0x7f3fc40abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 >> 172.21.15.78:0/324302441 conn(0x7f3fc40ca8a0 msgr2=0x7f3fc4150eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 shutdown_connections 2024-02-19T15:01:31.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:31.755+0000 7f3fc9947640 1 -- 172.21.15.78:0/324302441 wait complete. 2024-02-19T15:01:32.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:30.644801+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:resolve', 'state_seq': 13, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:32.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:resolve 2024-02-19T15:01:33.173 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.281+0000 7f58fc41c640 1 Processor -- start 2024-02-19T15:01:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.281+0000 7f58fc41c640 1 -- start start 2024-02-19T15:01:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fc41c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f58f4155f90 0x7f58f406d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fc41c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f58f40c6d30 0x7f58f40c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fc41c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f40c6af0 con 0x7f58f40c3aa0 2024-02-19T15:01:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fc41c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f40c7940 con 0x7f58f40c6d30 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fc41c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f40c7c20 con 0x7f58f4155f90 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58f99b7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f58f40c6d30 0x7f58f40c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fa9b9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f58f40c3aa0 0x7f58f41588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38048/0 (socket says 172.21.15.78:38048) 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fa9b9640 1 -- 172.21.15.78:0/2132462304 learned_addr learned my addr 172.21.15.78:0/2132462304 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58fa1b8640 1 --2- 172.21.15.78:0/2132462304 >> v2:172.21.15.150:3300/0 conn(0x7f58f4155f90 0x7f58f406d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58f99b7640 1 -- 172.21.15.78:0/2132462304 >> v2:172.21.15.150:3300/0 conn(0x7f58f4155f90 msgr2=0x7f58f406d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58f99b7640 1 --2- 172.21.15.78:0/2132462304 >> v2:172.21.15.150:3300/0 conn(0x7f58f4155f90 0x7f58f406d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.282+0000 7f58f99b7640 1 -- 172.21.15.78:0/2132462304 >> v1:172.21.15.78:6789/0 conn(0x7f58f40c3aa0 legacy=0x7f58f41588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.283+0000 7f58f99b7640 1 -- 172.21.15.78:0/2132462304 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40c7f00 con 0x7f58f40c6d30 2024-02-19T15:01:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.283+0000 7f58f91b6640 1 -- 172.21.15.78:0/2132462304 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058154320 0 0) 0x7f58f40c6af0 con 0x7f58f40c3aa0 2024-02-19T15:01:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.283+0000 7f58f99b7640 1 --2- 172.21.15.78:0/2132462304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f40c3560 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f58f40c2a40 tx=0x7f58e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5c9fda0c9419fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.283+0000 7f58f91b6640 1 -- 172.21.15.78:0/2132462304 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e402e070 con 0x7f58f40c6d30 2024-02-19T15:01:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.283+0000 7f58f91b6640 1 -- 172.21.15.78:0/2132462304 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e40315a0 con 0x7f58f40c6d30 2024-02-19T15:01:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.283+0000 7f58f91b6640 1 -- 172.21.15.78:0/2132462304 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4031930 con 0x7f58f40c6d30 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 -- 172.21.15.78:0/2132462304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 msgr2=0x7f58f40c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2132462304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f40c3560 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f58f40c2a40 tx=0x7f58e402b840 comp rx=0 tx=0).stop 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 -- 172.21.15.78:0/2132462304 shutdown_connections 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2132462304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f40c3560 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2132462304 >> v2:172.21.15.150:3300/0 conn(0x7f58f4155f90 0x7f58f406d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 -- 172.21.15.78:0/2132462304 >> 172.21.15.78:0/2132462304 conn(0x7f58f406b700 msgr2=0x7f58f40aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 -- 172.21.15.78:0/2132462304 shutdown_connections 2024-02-19T15:01:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.284+0000 7f58fc41c640 1 -- 172.21.15.78:0/2132462304 wait complete. 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.285+0000 7f58fc41c640 1 Processor -- start 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.285+0000 7f58fc41c640 1 -- start start 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fc41c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58f40c3aa0 0x7f58f4099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fc41c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fc41c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58f4155f90 0x7f58f40ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fc41c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f4156db0 con 0x7f58f40c6d30 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fc41c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f406f820 con 0x7f58f4155f90 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fc41c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f41eac30 con 0x7f58f40c3aa0 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f406f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33260/0 (socket says 172.21.15.78:33260) 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 -- 172.21.15.78:0/2181557615 learned_addr learned my addr 172.21.15.78:0/2181557615 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fa1b8640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58f40c3aa0 0x7f58f4099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58fa9b9640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58f4155f90 0x7f58f40ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 -- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58f40c3aa0 msgr2=0x7f58f4099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58f40c3aa0 0x7f58f4099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 -- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58f4155f90 msgr2=0x7f58f40ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58f4155f90 0x7f58f40ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.286+0000 7f58f99b7640 1 -- 172.21.15.78:0/2181557615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4009400 con 0x7f58f40c6d30 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.287+0000 7f58f99b7640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f406f2e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009890 tx=0x7f58e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.287+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4004230 con 0x7f58f40c6d30 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.287+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e4004390 con 0x7f58f40c6d30 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.287+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f4156410 con 0x7f58f40c6d30 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.287+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4003830 con 0x7f58f40c6d30 2024-02-19T15:01:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.287+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f41569d0 con 0x7f58f40c6d30 2024-02-19T15:01:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.288+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f4156ef0 con 0x7f58f40c6d30 2024-02-19T15:01:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.288+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f58e4003990 con 0x7f58f40c6d30 2024-02-19T15:01:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.289+0000 7f58eaffd640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d806b810 0x7f58d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.289+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f58e40ac390 con 0x7f58f40c6d30 2024-02-19T15:01:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.289+0000 7f58fa1b8640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d806b810 0x7f58d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.289+0000 7f58fa1b8640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d806b810 0x7f58d806dcd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009aa0 tx=0x7f58f0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:33.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.293+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f58f4156ef0 con 0x7f58f40c6d30 2024-02-19T15:01:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.498+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58f4156ef0 con 0x7f58f40c6d30 2024-02-19T15:01:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.499+0000 7f58eaffd640 1 -- 172.21.15.78:0/2181557615 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+4188 (secure 0 0 0) 0x7f58f4156ef0 con 0x7f58f40c6d30 2024-02-19T15:01:33.502 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:33.502 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":198,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:31.637809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 198 2024-02-19T15:01:33.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.500+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d806b810 msgr2=0x7f58d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.500+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d806b810 0x7f58d806dcd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009aa0 tx=0x7f58f0009190 comp rx=0 tx=0).stop 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.500+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 msgr2=0x7f58f406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.500+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f406f2e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009890 tx=0x7f58e40046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.500+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 shutdown_connections 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.500+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f58d806b810 0x7f58d806dcd0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.501+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58f40c6d30 0x7f58f406f2e0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.501+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58f4155f90 0x7f58f40ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.501+0000 7f58fc41c640 1 --2- 172.21.15.78:0/2181557615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58f40c3aa0 0x7f58f4099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.501+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 >> 172.21.15.78:0/2181557615 conn(0x7f58f406b700 msgr2=0x7f58f4155220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.501+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 shutdown_connections 2024-02-19T15:01:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:33.501+0000 7f58fc41c640 1 -- 172.21.15.78:0/2181557615 wait complete. 2024-02-19T15:01:33.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 198, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:31.637809+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:reconnect', 'state_seq': 14, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:33.917 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:reconnect 2024-02-19T15:01:34.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:35.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.025+0000 7fbdee33a640 1 Processor -- start 2024-02-19T15:01:35.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.025+0000 7fbdee33a640 1 -- start start 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdee33a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c25e0 0x7fbde81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdee33a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbde80c5be0 0x7fbde806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdee33a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde8072250 con 0x7fbde80c29c0 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdee33a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde8152c90 con 0x7fbde80c5be0 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdee33a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8072480 con 0x7fbde80c25e0 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdec8d7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbde80c5be0 0x7fbde806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdec8d7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbde80c5be0 0x7fbde806e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33262/0 (socket says 172.21.15.78:33262) 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbdec8d7640 1 -- 172.21.15.78:0/3712702922 learned_addr learned my addr 172.21.15.78:0/3712702922 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.026+0000 7fbde7fff640 1 --2- 172.21.15.78:0/3712702922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c25e0 0x7fbde81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbdec8d7640 1 -- 172.21.15.78:0/3712702922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c25e0 msgr2=0x7fbde81526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbdec8d7640 1 --2- 172.21.15.78:0/3712702922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c25e0 0x7fbde81526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbdec8d7640 1 -- 172.21.15.78:0/3712702922 >> v1:172.21.15.150:6789/0 conn(0x7fbde80c29c0 legacy=0x7fbde80c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbdec8d7640 1 -- 172.21.15.78:0/3712702922 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80c4b90 con 0x7fbde80c5be0 2024-02-19T15:01:35.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbdec8d7640 1 --2- 172.21.15.78:0/3712702922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde806e420 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d180 tx=0x7fbde002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a8f5f9ceb81a3d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbde6ffd640 1 -- 172.21.15.78:0/3712702922 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde002ea60 con 0x7fbde80c5be0 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbde6ffd640 1 -- 172.21.15.78:0/3712702922 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde002ebc0 con 0x7fbde80c5be0 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.027+0000 7fbde6ffd640 1 -- 172.21.15.78:0/3712702922 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0009600 con 0x7fbde80c5be0 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 -- 172.21.15.78:0/3712702922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 msgr2=0x7fbde806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 --2- 172.21.15.78:0/3712702922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde806e420 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d180 tx=0x7fbde002e0c0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 -- 172.21.15.78:0/3712702922 shutdown_connections 2024-02-19T15:01:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 --2- 172.21.15.78:0/3712702922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde806e420 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 --2- 172.21.15.78:0/3712702922 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c25e0 0x7fbde81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 -- 172.21.15.78:0/3712702922 >> 172.21.15.78:0/3712702922 conn(0x7fbde806b270 msgr2=0x7fbde806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 -- 172.21.15.78:0/3712702922 shutdown_connections 2024-02-19T15:01:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.028+0000 7fbdee33a640 1 -- 172.21.15.78:0/3712702922 wait complete. 2024-02-19T15:01:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.029+0000 7fbdee33a640 1 Processor -- start 2024-02-19T15:01:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.029+0000 7fbdee33a640 1 -- start start 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdee33a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbde80c25e0 0x7fbde81490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdee33a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c29c0 0x7fbde80a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdee33a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde8145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdee33a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81ee230 con 0x7fbde80c5be0 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdee33a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81ea8c0 con 0x7fbde80c25e0 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdee33a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81edf50 con 0x7fbde80c29c0 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdec8d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde8145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdec8d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde8145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33274/0 (socket says 172.21.15.78:33274) 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbdec8d7640 1 -- 172.21.15.78:0/765916635 learned_addr learned my addr 172.21.15.78:0/765916635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:35.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbde7fff640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbde80c25e0 0x7fbde81490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.030+0000 7fbde77fe640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c29c0 0x7fbde80a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdec8d7640 1 -- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c29c0 msgr2=0x7fbde80a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdec8d7640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c29c0 0x7fbde80a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdec8d7640 1 -- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbde80c25e0 msgr2=0x7fbde81490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdec8d7640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbde80c25e0 0x7fbde81490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdec8d7640 1 -- 172.21.15.78:0/765916635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde000bd40 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdec8d7640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde8145820 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fbde0038040 tx=0x7fbde0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde002ea60 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0036820 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde81eaad0 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0036bd0 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.031+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde81eb090 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.032+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde8074990 con 0x7fbde80c5be0 2024-02-19T15:01:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.032+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbde0036d30 con 0x7fbde80c5be0 2024-02-19T15:01:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.033+0000 7fbde4ff9640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdd406b810 0x7fbdd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.033+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fbde00ad550 con 0x7fbde80c5be0 2024-02-19T15:01:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.033+0000 7fbde7fff640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdd406b810 0x7fbdd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.033+0000 7fbde7fff640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdd406b810 0x7fbdd406dcd0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009aa0 tx=0x7fbdd8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:35.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.037+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbde0077f70 con 0x7fbde80c5be0 2024-02-19T15:01:35.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.244+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbde80c5770 con 0x7fbde80c5be0 2024-02-19T15:01:35.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.245+0000 7fbde4ff9640 1 -- 172.21.15.78:0/765916635 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7fbde000b720 con 0x7fbde80c5be0 2024-02-19T15:01:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:35.248 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:35.249 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:35.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdd406b810 msgr2=0x7fbdd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:35.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdd406b810 0x7fbdd406dcd0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009aa0 tx=0x7fbdd8009190 comp rx=0 tx=0).stop 2024-02-19T15:01:35.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 msgr2=0x7fbde8145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde8145820 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fbde0038040 tx=0x7fbde0004a40 comp rx=0 tx=0).stop 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 shutdown_connections 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbdd406b810 0x7fbdd406dcd0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbde80c5be0 0x7fbde8145820 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbde80c29c0 0x7fbde80a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 --2- 172.21.15.78:0/765916635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbde80c25e0 0x7fbde81490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 >> 172.21.15.78:0/765916635 conn(0x7fbde806b270 msgr2=0x7fbde8074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 shutdown_connections 2024-02-19T15:01:35.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:35.247+0000 7fbdee33a640 1 -- 172.21.15.78:0/765916635 wait complete. 2024-02-19T15:01:35.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:35.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:36.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:36.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.775+0000 7f56a5b83640 1 Processor -- start 2024-02-19T15:01:36.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.775+0000 7f56a5b83640 1 -- start start 2024-02-19T15:01:36.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f56a5b83640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56a0071410 0x7f56a0156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:36.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f56a5b83640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56a00c1dd0 0x7f56a00c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:36.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f56a5b83640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a0155a50 con 0x7f56a00c19f0 2024-02-19T15:01:36.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f56a5b83640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a0152af0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f56a5b83640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a0155f80 con 0x7f56a0071410 2024-02-19T15:01:36.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56a00c1dd0 0x7f56a00c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56a00c1dd0 0x7f56a00c21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33290/0 (socket says 172.21.15.78:33290) 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 -- 172.21.15.78:0/2866529269 learned_addr learned my addr 172.21.15.78:0/2866529269 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569effd640 1 --2- 172.21.15.78:0/2866529269 >> v2:172.21.15.150:3300/0 conn(0x7f56a0071410 0x7f56a0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 -- 172.21.15.78:0/2866529269 >> v2:172.21.15.150:3300/0 conn(0x7f56a0071410 msgr2=0x7f56a0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 --2- 172.21.15.78:0/2866529269 >> v2:172.21.15.150:3300/0 conn(0x7f56a0071410 0x7f56a0156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 -- 172.21.15.78:0/2866529269 >> v1:172.21.15.150:6789/0 conn(0x7f56a00c19f0 legacy=0x7f56a00c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:36.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.776+0000 7f569ffff640 1 -- 172.21.15.78:0/2866529269 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a0070840 con 0x7f56a00c1dd0 2024-02-19T15:01:36.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.777+0000 7f569ffff640 1 --2- 172.21.15.78:0/2866529269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a00c21e0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f56a00a8290 tx=0x7f569802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb69fa0a0798c608 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.777+0000 7f569e7fc640 1 -- 172.21.15.78:0/2866529269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569802c9f0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.777+0000 7f569e7fc640 1 -- 172.21.15.78:0/2866529269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569802cb50 con 0x7f56a00c1dd0 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.777+0000 7f569e7fc640 1 -- 172.21.15.78:0/2866529269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56980343f0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 -- 172.21.15.78:0/2866529269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 msgr2=0x7f56a00c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2866529269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a00c21e0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f56a00a8290 tx=0x7f569802be50 comp rx=0 tx=0).stop 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 -- 172.21.15.78:0/2866529269 shutdown_connections 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2866529269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a00c21e0 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2866529269 >> v2:172.21.15.150:3300/0 conn(0x7f56a0071410 0x7f56a0156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 -- 172.21.15.78:0/2866529269 >> 172.21.15.78:0/2866529269 conn(0x7f56a00aa370 msgr2=0x7f56a00aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 -- 172.21.15.78:0/2866529269 shutdown_connections 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.778+0000 7f56a5b83640 1 -- 172.21.15.78:0/2866529269 wait complete. 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.779+0000 7f56a5b83640 1 Processor -- start 2024-02-19T15:01:36.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.779+0000 7f56a5b83640 1 -- start start 2024-02-19T15:01:36.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56a0071410 0x7f56a01502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:36.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56a00c19f0 0x7f56a0158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:36.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:36.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a01ee1f0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a01ee4d0 con 0x7f56a0071410 2024-02-19T15:01:36.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a01ee7b0 con 0x7f56a00c19f0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a014d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33296/0 (socket says 172.21.15.78:33296) 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 -- 172.21.15.78:0/2900404609 learned_addr learned my addr 172.21.15.78:0/2900404609 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569effd640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56a00c19f0 0x7f56a0158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569f7fe640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56a0071410 0x7f56a01502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 -- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56a00c19f0 msgr2=0x7f56a0158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56a00c19f0 0x7f56a0158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 -- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56a0071410 msgr2=0x7f56a01502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56a0071410 0x7f56a01502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 -- 172.21.15.78:0/2900404609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5698009b30 con 0x7f56a00c1dd0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f569ffff640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a014d810 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f5698036040 tx=0x7f5698004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698009510 con 0x7f56a00c1dd0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569803e450 con 0x7f56a00c1dd0 2024-02-19T15:01:36.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a01eea90 con 0x7f56a00c1dd0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569803e800 con 0x7f56a00c1dd0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.780+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a01507f0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.782+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a00c8320 con 0x7f56a00c1dd0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.782+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f569803e960 con 0x7f56a00c1dd0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.782+0000 7f5687fff640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f568c06b840 0x7f568c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.783+0000 7f569f7fe640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f568c06b840 0x7f568c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.783+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f56980acf90 con 0x7f56a00c1dd0 2024-02-19T15:01:36.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.783+0000 7f569f7fe640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f568c06b840 0x7f568c06dd00 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f56a01ebdb0 tx=0x7f5690009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:36.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.787+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56980e0dc0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.992+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56a01eefb0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.993+0000 7f5687fff640 1 -- 172.21.15.78:0/2900404609 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f56a01eefb0 con 0x7f56a00c1dd0 2024-02-19T15:01:36.996 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:36.996 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:36.996 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f568c06b840 msgr2=0x7f568c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f568c06b840 0x7f568c06dd00 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f56a01ebdb0 tx=0x7f5690009040 comp rx=0 tx=0).stop 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 msgr2=0x7f56a014d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a014d810 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f5698036040 tx=0x7f5698004a40 comp rx=0 tx=0).stop 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 shutdown_connections 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f568c06b840 0x7f568c06dd00 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56a00c1dd0 0x7f56a014d810 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56a00c19f0 0x7f56a0158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 --2- 172.21.15.78:0/2900404609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56a0071410 0x7f56a01502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:36.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 >> 172.21.15.78:0/2900404609 conn(0x7f56a00aa370 msgr2=0x7f56a006cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:36.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 shutdown_connections 2024-02-19T15:01:36.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:36.995+0000 7f56a5b83640 1 -- 172.21.15.78:0/2900404609 wait complete. 2024-02-19T15:01:37.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:37.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:38.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:38.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.520+0000 7f9ffe12e640 1 Processor -- start 2024-02-19T15:01:38.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.520+0000 7f9ffe12e640 1 -- start start 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ffe12e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ad890 0x7f9ff80c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ffe12e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff8152d30 con 0x7f9ff80c49c0 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ffe12e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff80c7860 con 0x7f9ff80ab600 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ffe12e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80aba90 con 0x7f9ff80ad890 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ff6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9ff80c49c0 0x7f9ff81534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55014/0 (socket says 172.21.15.78:55014) 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ff6ffd640 1 -- 172.21.15.78:0/1903312127 learned_addr learned my addr 172.21.15.78:0/1903312127 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ff77fe640 1 --2- 172.21.15.78:0/1903312127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ad890 0x7f9ff80c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.521+0000 7f9ff67fc640 1 -- 172.21.15.78:0/1903312127 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413923498 0 0) 0x7f9ff8152d30 con 0x7f9ff80c49c0 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.522+0000 7f9ff67fc640 1 -- 172.21.15.78:0/1903312127 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe4003520 con 0x7f9ff80c49c0 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.522+0000 7f9ff77fe640 1 -- 172.21.15.78:0/1903312127 >> v1:172.21.15.150:6789/0 conn(0x7f9ff80ab600 legacy=0x7f9ff80c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.522+0000 7f9ff77fe640 1 -- 172.21.15.78:0/1903312127 >> v1:172.21.15.78:6789/0 conn(0x7f9ff80c49c0 legacy=0x7f9ff81534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.522+0000 7f9ff67fc640 1 -- 172.21.15.78:0/1903312127 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984354326 0 0) 0x7f9ff80c7860 con 0x7f9ff80ab600 2024-02-19T15:01:38.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.522+0000 7f9ff77fe640 1 -- 172.21.15.78:0/1903312127 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff80c7ce0 con 0x7f9ff80ad890 2024-02-19T15:01:38.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.522+0000 7f9ff77fe640 1 --2- 172.21.15.78:0/1903312127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ad890 0x7f9ff80c64f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80097d0 tx=0x7f9fe802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa8962999966755 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:38.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ff67fc640 1 -- 172.21.15.78:0/1903312127 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8031770 con 0x7f9ff80ad890 2024-02-19T15:01:38.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ff67fc640 1 -- 172.21.15.78:0/1903312127 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe80318d0 con 0x7f9ff80ad890 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ff67fc640 1 -- 172.21.15.78:0/1903312127 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8031c60 con 0x7f9ff80ad890 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 -- 172.21.15.78:0/1903312127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ad890 msgr2=0x7f9ff80c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/1903312127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ad890 0x7f9ff80c64f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80097d0 tx=0x7f9fe802b810 comp rx=0 tx=0).stop 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 -- 172.21.15.78:0/1903312127 shutdown_connections 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/1903312127 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ad890 0x7f9ff80c64f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 -- 172.21.15.78:0/1903312127 >> 172.21.15.78:0/1903312127 conn(0x7f9ff806b700 msgr2=0x7f9ff80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 -- 172.21.15.78:0/1903312127 shutdown_connections 2024-02-19T15:01:38.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.523+0000 7f9ffe12e640 1 -- 172.21.15.78:0/1903312127 wait complete. 2024-02-19T15:01:38.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.524+0000 7f9ffe12e640 1 Processor -- start 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.524+0000 7f9ffe12e640 1 -- start start 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ffe12e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ab600 0x7f9ff81e8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ffe12e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 0x7f9ff8154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ffe12e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80c49c0 0x7f9ff8155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ffe12e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8154b30 con 0x7f9ff80ad890 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ffe12e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff81e9110 con 0x7f9ff80c49c0 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ffe12e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8155a40 con 0x7f9ff80ab600 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ff6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 0x7f9ff8154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ff6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 0x7f9ff8154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42868/0 (socket says 172.21.15.78:42868) 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ff6ffd640 1 -- 172.21.15.78:0/3013984240 learned_addr learned my addr 172.21.15.78:0/3013984240 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ff77fe640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ab600 0x7f9ff81e8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.525+0000 7f9ff7fff640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80c49c0 0x7f9ff8155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ff6ffd640 1 -- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ab600 msgr2=0x7f9ff81e8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ff6ffd640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ab600 0x7f9ff81e8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ff6ffd640 1 -- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80c49c0 msgr2=0x7f9ff8155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ff6ffd640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80c49c0 0x7f9ff8155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ff6ffd640 1 -- 172.21.15.78:0/3013984240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe8009400 con 0x7f9ff80ad890 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ff6ffd640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 0x7f9ff8154fc0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000c400 tx=0x7f9fe000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0005ce0 con 0x7f9ff80ad890 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe0005e60 con 0x7f9ff80ad890 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0007310 con 0x7f9ff80ad890 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff81e8e30 con 0x7f9ff80ad890 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.526+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff8157100 con 0x7f9ff80ad890 2024-02-19T15:01:38.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.527+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff8154ce0 con 0x7f9ff80ad890 2024-02-19T15:01:38.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.527+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9fe0007470 con 0x7f9ff80ad890 2024-02-19T15:01:38.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.528+0000 7f9feffff640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:38.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.528+0000 7f9ff77fe640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:38.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.528+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9fe00898a0 con 0x7f9ff80ad890 2024-02-19T15:01:38.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.528+0000 7f9ff77fe640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8036000 tx=0x7f9fe803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:38.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.532+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9ff8154ce0 con 0x7f9ff80ad890 2024-02-19T15:01:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.739+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ff8154ce0 con 0x7f9ff80ad890 2024-02-19T15:01:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.740+0000 7f9feffff640 1 -- 172.21.15.78:0/3013984240 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f9ff8154ce0 con 0x7f9ff80ad890 2024-02-19T15:01:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:38.743 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 msgr2=0x7f9fd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8036000 tx=0x7f9fe803d040 comp rx=0 tx=0).stop 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 msgr2=0x7f9ff8154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 0x7f9ff8154fc0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000c400 tx=0x7f9fe000c8c0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 shutdown_connections 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9fd806b810 0x7f9fd806dcd0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff80ad890 0x7f9ff8154fc0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80c49c0 0x7f9ff8155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 --2- 172.21.15.78:0/3013984240 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ff80ab600 0x7f9ff81e8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 >> 172.21.15.78:0/3013984240 conn(0x7f9ff806b700 msgr2=0x7f9ff8070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 shutdown_connections 2024-02-19T15:01:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:38.742+0000 7f9ffe12e640 1 -- 172.21.15.78:0/3013984240 wait complete. 2024-02-19T15:01:39.159 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:39.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:40.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.273+0000 7ff698e09640 1 Processor -- start 2024-02-19T15:01:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 -- start start 2024-02-19T15:01:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff694152700 0x7ff6940c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff6940c5730 0x7ff69406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c8e00 0x7ff69406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69406cd20 con 0x7ff6940c5730 2024-02-19T15:01:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6941551f0 con 0x7ff694152700 2024-02-19T15:01:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.274+0000 7ff698e09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff694155490 con 0x7ff6940c8e00 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff6940c5730 0x7ff69406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff6940c5730 0x7ff69406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42872/0 (socket says 172.21.15.78:42872) 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 -- 172.21.15.78:0/4276237022 learned_addr learned my addr 172.21.15.78:0/4276237022 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff692d9d640 1 --2- 172.21.15.78:0/4276237022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c8e00 0x7ff69406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff69259c640 1 --2- 172.21.15.78:0/4276237022 >> v2:172.21.15.150:3300/0 conn(0x7ff694152700 0x7ff6940c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 -- 172.21.15.78:0/4276237022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c8e00 msgr2=0x7ff69406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 --2- 172.21.15.78:0/4276237022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c8e00 0x7ff69406c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 -- 172.21.15.78:0/4276237022 >> v2:172.21.15.150:3300/0 conn(0x7ff694152700 msgr2=0x7ff6940c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 --2- 172.21.15.78:0/4276237022 >> v2:172.21.15.150:3300/0 conn(0x7ff694152700 0x7ff6940c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.275+0000 7ff691d9b640 1 -- 172.21.15.78:0/4276237022 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694155770 con 0x7ff6940c5730 2024-02-19T15:01:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff691d9b640 1 --2- 172.21.15.78:0/4276237022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6940c5730 0x7ff69406c2a0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7ff69408bad0 tx=0x7ff68802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741c70b81de87c88 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff69159a640 1 -- 172.21.15.78:0/4276237022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688031870 con 0x7ff6940c5730 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff69159a640 1 -- 172.21.15.78:0/4276237022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6880319d0 con 0x7ff6940c5730 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff69159a640 1 -- 172.21.15.78:0/4276237022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688031d60 con 0x7ff6940c5730 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 -- 172.21.15.78:0/4276237022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6940c5730 msgr2=0x7ff69406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 --2- 172.21.15.78:0/4276237022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6940c5730 0x7ff69406c2a0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7ff69408bad0 tx=0x7ff68802b910 comp rx=0 tx=0).stop 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 -- 172.21.15.78:0/4276237022 shutdown_connections 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 --2- 172.21.15.78:0/4276237022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c8e00 0x7ff69406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 --2- 172.21.15.78:0/4276237022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6940c5730 0x7ff69406c2a0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 --2- 172.21.15.78:0/4276237022 >> v2:172.21.15.150:3300/0 conn(0x7ff694152700 0x7ff6940c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 -- 172.21.15.78:0/4276237022 >> 172.21.15.78:0/4276237022 conn(0x7ff6940cac90 msgr2=0x7ff6940cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 -- 172.21.15.78:0/4276237022 shutdown_connections 2024-02-19T15:01:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.276+0000 7ff698e09640 1 -- 172.21.15.78:0/4276237022 wait complete. 2024-02-19T15:01:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.278+0000 7ff698e09640 1 Processor -- start 2024-02-19T15:01:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.278+0000 7ff698e09640 1 -- start start 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.278+0000 7ff698e09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c5730 0x7ff6940ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.278+0000 7ff698e09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6940c8e00 0x7ff6940ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.278+0000 7ff698e09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 0x7ff6940ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff698e09640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941eed90 con 0x7ff694152700 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff698e09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940aae40 con 0x7ff6940c8e00 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff698e09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941eeab0 con 0x7ff6940c5730 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 0x7ff6940ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 0x7ff6940ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42882/0 (socket says 172.21.15.78:42882) 2024-02-19T15:01:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 -- 172.21.15.78:0/1553437115 learned_addr learned my addr 172.21.15.78:0/1553437115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff69259c640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c5730 0x7ff6940ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff691d9b640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6940c8e00 0x7ff6940ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 -- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c5730 msgr2=0x7ff6940ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c5730 0x7ff6940ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 -- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6940c8e00 msgr2=0x7ff6940ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6940c8e00 0x7ff6940ab600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 -- 172.21.15.78:0/1553437115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6880094d0 con 0x7ff694152700 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff692d9d640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 0x7ff6940ae260 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00e930 tx=0x7ff68c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c009690 con 0x7ff694152700 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c00c980 con 0x7ff694152700 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6940ab0e0 con 0x7ff694152700 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00cd30 con 0x7ff694152700 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.279+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6940ae7a0 con 0x7ff694152700 2024-02-19T15:01:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.280+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6940ae9f0 con 0x7ff694152700 2024-02-19T15:01:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.281+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff68c013470 con 0x7ff694152700 2024-02-19T15:01:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.281+0000 7ff682ffd640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff67406b810 0x7ff67406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.281+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff68c08a4e0 con 0x7ff694152700 2024-02-19T15:01:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.281+0000 7ff69259c640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff67406b810 0x7ff67406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.282+0000 7ff69259c640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff67406b810 0x7ff67406dcd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ff684011340 tx=0x7ff68400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:40.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.286+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff68c054e90 con 0x7ff694152700 2024-02-19T15:01:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.493+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6940aaca0 con 0x7ff694152700 2024-02-19T15:01:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.494+0000 7ff682ffd640 1 -- 172.21.15.78:0/1553437115 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7ff6940aaca0 con 0x7ff694152700 2024-02-19T15:01:40.497 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:40.497 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:40.497 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff67406b810 msgr2=0x7ff67406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff67406b810 0x7ff67406dcd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ff684011340 tx=0x7ff68400b040 comp rx=0 tx=0).stop 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 msgr2=0x7ff6940ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 0x7ff6940ae260 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00e930 tx=0x7ff68c00edf0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 shutdown_connections 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff67406b810 0x7ff67406dcd0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff694152700 0x7ff6940ae260 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6940c8e00 0x7ff6940ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 --2- 172.21.15.78:0/1553437115 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6940c5730 0x7ff6940ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 >> 172.21.15.78:0/1553437115 conn(0x7ff6940cac90 msgr2=0x7ff69414d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 shutdown_connections 2024-02-19T15:01:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:40.496+0000 7ff698e09640 1 -- 172.21.15.78:0/1553437115 wait complete. 2024-02-19T15:01:40.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:40.912 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:41.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.027+0000 7f39bb70c640 1 Processor -- start 2024-02-19T15:01:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.028+0000 7f39bb70c640 1 -- start start 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.028+0000 7f39bb70c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39b40c5560 0x7f39b40c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.028+0000 7f39bb70c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b40c65a0 con 0x7f39b40c7910 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.028+0000 7f39bb70c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b40c7530 con 0x7f39b4070720 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.028+0000 7f39bb70c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b41537b0 con 0x7f39b40c5560 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b94a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39b40c5560 0x7f39b40c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b8ca7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f39b40c7910 0x7f39b4154180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55026/0 (socket says 172.21.15.78:55026) 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b94a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f39b40c5560 0x7f39b40c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42894/0 (socket says 172.21.15.78:42894) 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b94a8640 1 -- 172.21.15.78:0/1136877388 learned_addr learned my addr 172.21.15.78:0/1136877388 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b94a8640 1 -- 172.21.15.78:0/1136877388 >> v1:172.21.15.150:6789/0 conn(0x7f39b4070720 legacy=0x7f39b4070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b94a8640 1 -- 172.21.15.78:0/1136877388 >> v1:172.21.15.78:6789/0 conn(0x7f39b40c7910 legacy=0x7f39b4154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39abfff640 1 -- 172.21.15.78:0/1136877388 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268629644 0 0) 0x7f39b40c7530 con 0x7f39b4070720 2024-02-19T15:01:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.029+0000 7f39b94a8640 1 -- 172.21.15.78:0/1136877388 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4153a90 con 0x7f39b40c5560 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39b94a8640 1 --2- 172.21.15.78:0/1136877388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c5560 0x7f39b40c6ff0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009870 tx=0x7f39b002b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3ec14ac50efb09 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39abfff640 1 -- 172.21.15.78:0/1136877388 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0030710 con 0x7f39b40c5560 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39abfff640 1 -- 172.21.15.78:0/1136877388 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b002e050 con 0x7f39b40c5560 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39abfff640 1 -- 172.21.15.78:0/1136877388 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0030a90 con 0x7f39b40c5560 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 -- 172.21.15.78:0/1136877388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c5560 msgr2=0x7f39b40c6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 --2- 172.21.15.78:0/1136877388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c5560 0x7f39b40c6ff0 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009870 tx=0x7f39b002b740 comp rx=0 tx=0).stop 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 -- 172.21.15.78:0/1136877388 shutdown_connections 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 --2- 172.21.15.78:0/1136877388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c5560 0x7f39b40c6ff0 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 -- 172.21.15.78:0/1136877388 >> 172.21.15.78:0/1136877388 conn(0x7f39b406b4b0 msgr2=0x7f39b40c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 -- 172.21.15.78:0/1136877388 shutdown_connections 2024-02-19T15:01:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.030+0000 7f39bb70c640 1 -- 172.21.15.78:0/1136877388 wait complete. 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 Processor -- start 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 -- start start 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b4070720 0x7f39b40a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39b40c5560 0x7f39b41e8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 0x7f39b41e8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b40ab2f0 con 0x7f39b40c7910 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b40ad830 con 0x7f39b4070720 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39bb70c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b40ab050 con 0x7f39b40c5560 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39b9ca9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 0x7f39b41e8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39b9ca9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 0x7f39b41e8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42910/0 (socket says 172.21.15.78:42910) 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.032+0000 7f39b9ca9640 1 -- 172.21.15.78:0/3317383688 learned_addr learned my addr 172.21.15.78:0/3317383688 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b94a8640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b4070720 0x7f39b40a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b8ca7640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39b40c5560 0x7f39b41e8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b9ca9640 1 -- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39b40c5560 msgr2=0x7f39b41e8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b9ca9640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39b40c5560 0x7f39b41e8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b9ca9640 1 -- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b4070720 msgr2=0x7f39b40a9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b9ca9640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b4070720 0x7f39b40a9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b9ca9640 1 -- 172.21.15.78:0/3317383688 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00094d0 con 0x7f39b40c7910 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39b9ca9640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 0x7f39b41e8c70 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c780 tx=0x7f39ac00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac003b70 con 0x7f39b40c7910 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac003cd0 con 0x7f39b40c7910 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b40c5ac0 con 0x7f39b40c7910 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac005270 con 0x7f39b40c7910 2024-02-19T15:01:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.033+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b41dd3d0 con 0x7f39b40c7910 2024-02-19T15:01:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.034+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b4074990 con 0x7f39b40c7910 2024-02-19T15:01:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.035+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f39ac0053d0 con 0x7f39b40c7910 2024-02-19T15:01:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.035+0000 7f39a9ffb640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f399806b810 0x7f399806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:42.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.035+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f39ac089fa0 con 0x7f39b40c7910 2024-02-19T15:01:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.035+0000 7f39b94a8640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f399806b810 0x7f399806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.036+0000 7f39b94a8640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f399806b810 0x7f399806dcd0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f39b0034000 tx=0x7f39b0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.040+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f39ac0bddc0 con 0x7f39b40c7910 2024-02-19T15:01:42.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.247+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39b4070250 con 0x7f39b40c7910 2024-02-19T15:01:42.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.248+0000 7f39a9ffb640 1 -- 172.21.15.78:0/3317383688 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f39ac054920 con 0x7f39b40c7910 2024-02-19T15:01:42.251 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:42.251 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:42.251 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f399806b810 msgr2=0x7f399806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f399806b810 0x7f399806dcd0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f39b0034000 tx=0x7f39b0008030 comp rx=0 tx=0).stop 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 msgr2=0x7f39b41e8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 0x7f39b41e8c70 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c780 tx=0x7f39ac00cb80 comp rx=0 tx=0).stop 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 shutdown_connections 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f399806b810 0x7f399806dcd0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b40c7910 0x7f39b41e8c70 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39b40c5560 0x7f39b41e8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 --2- 172.21.15.78:0/3317383688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b4070720 0x7f39b40a9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:42.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 >> 172.21.15.78:0/3317383688 conn(0x7f39b406b4b0 msgr2=0x7f39b41514f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:42.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 shutdown_connections 2024-02-19T15:01:42.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:42.250+0000 7f39bb70c640 1 -- 172.21.15.78:0/3317383688 wait complete. 2024-02-19T15:01:42.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:42.667 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:43.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:43.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.774+0000 7f3e7ba47640 1 Processor -- start 2024-02-19T15:01:43.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.774+0000 7f3e7ba47640 1 -- start start 2024-02-19T15:01:43.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e7ba47640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e740c2f50 0x7f3e74158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e7ba47640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e740c34a0 con 0x7f3e740c42e0 2024-02-19T15:01:43.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e7ba47640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7406c8e0 con 0x7f3e740c8500 2024-02-19T15:01:43.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e7ba47640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e7406c090 con 0x7f3e740c2f50 2024-02-19T15:01:43.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e78fe2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3e740c42e0 0x7f3e7415d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55040/0 (socket says 172.21.15.78:55040) 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e78fe2640 1 -- 172.21.15.78:0/1324536219 learned_addr learned my addr 172.21.15.78:0/1324536219 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e79fe4640 1 --2- 172.21.15.78:0/1324536219 >> v2:172.21.15.150:3300/0 conn(0x7f3e740c2f50 0x7f3e74158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574148570 0 0) 0x7f3e740c34a0 con 0x7f3e740c42e0 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60003520 con 0x7f3e740c42e0 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375054325 0 0) 0x7f3e7406c8e0 con 0x7f3e740c8500 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e740c34a0 con 0x7f3e740c8500 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1901485805 0 0) 0x7f3e60003520 con 0x7f3e740c42e0 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 >> v2:172.21.15.150:3300/0 conn(0x7f3e740c2f50 msgr2=0x7f3e74158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 --2- 172.21.15.78:0/1324536219 >> v2:172.21.15.150:3300/0 conn(0x7f3e740c2f50 0x7f3e74158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 >> v1:172.21.15.150:6789/0 conn(0x7f3e740c8500 legacy=0x7f3e74158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7406cbc0 con 0x7f3e740c42e0 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f3e64001670 con 0x7f3e740c42e0 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 >> v1:172.21.15.78:6789/0 conn(0x7f3e740c42e0 legacy=0x7f3e7415d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:43.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60003b00 0x7f3e60003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e600047c0 0x7f3e60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e60005180 0x7f3e60014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60014890 con 0x7f3e60005180 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60014a00 con 0x7f3e60003b00 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.775+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60014bc0 con 0x7f3e600047c0 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e60005180 0x7f3e60014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e797e3640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60003b00 0x7f3e60003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e78fe2640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e600047c0 0x7f3e60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 -- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e600047c0 msgr2=0x7f3e60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e600047c0 0x7f3e60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 -- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60003b00 msgr2=0x7f3e60003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60003b00 0x7f3e60003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 -- 172.21.15.78:0/1324536219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e60003520 con 0x7f3e60005180 2024-02-19T15:01:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.776+0000 7f3e79fe4640 1 -- 172.21.15.78:0/1324536219 reap_dead start 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e79fe4640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e60005180 0x7f3e60014190 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009fc0 tx=0x7f3e600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd372c4c8ddd16b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c02cc60 con 0x7f3e60005180 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c00f040 con 0x7f3e60005180 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e6bfff640 1 -- 172.21.15.78:0/1324536219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c003990 con 0x7f3e60005180 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1324536219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e60005180 msgr2=0x7f3e60014190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e60005180 0x7f3e60014190 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009fc0 tx=0x7f3e600057f0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1324536219 shutdown_connections 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1324536219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e60005180 0x7f3e60014190 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1324536219 >> 172.21.15.78:0/1324536219 conn(0x7f3e740aa300 msgr2=0x7f3e740aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1324536219 shutdown_connections 2024-02-19T15:01:43.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.777+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1324536219 wait complete. 2024-02-19T15:01:43.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.778+0000 7f3e7ba47640 1 Processor -- start 2024-02-19T15:01:43.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.778+0000 7f3e7ba47640 1 -- start start 2024-02-19T15:01:43.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e7ba47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60005180 0x7f3e7414d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e7ba47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 0x7f3e740acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e7ba47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e7414a900 0x7f3e741e7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e7ba47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74152770 con 0x7f3e7414a400 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e7ba47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7414d060 con 0x7f3e60005180 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e7ba47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7414dc40 con 0x7f3e7414a900 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e78fe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 0x7f3e740acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e78fe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 0x7f3e740acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42930/0 (socket says 172.21.15.78:42930) 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e78fe2640 1 -- 172.21.15.78:0/1585829664 learned_addr learned my addr 172.21.15.78:0/1585829664 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e797e3640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60005180 0x7f3e7414d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.779+0000 7f3e79fe4640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e7414a900 0x7f3e741e7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e78fe2640 1 -- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e7414a900 msgr2=0x7f3e741e7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e78fe2640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e7414a900 0x7f3e741e7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e78fe2640 1 -- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60005180 msgr2=0x7f3e7414d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e78fe2640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60005180 0x7f3e7414d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:43.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e78fe2640 1 -- 172.21.15.78:0/1585829664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7414df20 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e78fe2640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 0x7f3e740acb30 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f3e64004f20 tx=0x7f3e64007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64001040 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6400f750 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7414c180 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6400fae0 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.780+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e740ad070 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.781+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e74071c20 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.781+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3e6400fc40 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.782+0000 7f3e69ffb640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5806b840 0x7f3e5806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.782+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f3e6408a200 con 0x7f3e7414a400 2024-02-19T15:01:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.782+0000 7f3e797e3640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5806b840 0x7f3e5806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.782+0000 7f3e797e3640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5806b840 0x7f3e5806dd00 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3e741eac30 tx=0x7f3e70009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.786+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3e64054a70 con 0x7f3e7414a400 2024-02-19T15:01:43.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.996+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e7414bfe0 con 0x7f3e7414a400 2024-02-19T15:01:43.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.996+0000 7f3e69ffb640 1 -- 172.21.15.78:0/1585829664 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f3e7414bfe0 con 0x7f3e7414a400 2024-02-19T15:01:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:43.999 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:44.000 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:44.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5806b840 msgr2=0x7f3e5806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:44.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5806b840 0x7f3e5806dd00 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3e741eac30 tx=0x7f3e70009040 comp rx=0 tx=0).stop 2024-02-19T15:01:44.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 msgr2=0x7f3e740acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:44.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 0x7f3e740acb30 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f3e64004f20 tx=0x7f3e64007430 comp rx=0 tx=0).stop 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 shutdown_connections 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3e5806b840 0x7f3e5806dd00 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e7414a400 0x7f3e740acb30 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e7414a900 0x7f3e741e7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 --2- 172.21.15.78:0/1585829664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e60005180 0x7f3e7414d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 >> 172.21.15.78:0/1585829664 conn(0x7f3e740aa300 msgr2=0x7f3e740a8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 shutdown_connections 2024-02-19T15:01:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:43.998+0000 7f3e7ba47640 1 -- 172.21.15.78:0/1585829664 wait complete. 2024-02-19T15:01:44.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:44.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:45.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.530+0000 7f2af0207640 1 Processor -- start 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 -- start start 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae8152700 0x7f2ae80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ae80c5730 0x7f2ae806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2ae80c8e00 0x7f2ae806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae806cd20 con 0x7f2ae80c5730 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae81551f0 con 0x7f2ae80c8e00 2024-02-19T15:01:45.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.531+0000 7f2af0207640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae8155490 con 0x7f2ae8152700 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ae80c5730 0x7f2ae806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ae80c5730 0x7f2ae806c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42944/0 (socket says 172.21.15.78:42944) 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 -- 172.21.15.78:0/183959485 learned_addr learned my addr 172.21.15.78:0/183959485 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aedfa3640 1 --2- 172.21.15.78:0/183959485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae8152700 0x7f2ae80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aee7a4640 1 --2- 172.21.15.78:0/183959485 >> v2:172.21.15.150:3300/0 conn(0x7f2ae80c8e00 0x7f2ae806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 -- 172.21.15.78:0/183959485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae8152700 msgr2=0x7f2ae80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 --2- 172.21.15.78:0/183959485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae8152700 0x7f2ae80c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 -- 172.21.15.78:0/183959485 >> v2:172.21.15.150:3300/0 conn(0x7f2ae80c8e00 msgr2=0x7f2ae806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 --2- 172.21.15.78:0/183959485 >> v2:172.21.15.150:3300/0 conn(0x7f2ae80c8e00 0x7f2ae806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.532+0000 7f2aed7a2640 1 -- 172.21.15.78:0/183959485 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8155770 con 0x7f2ae80c5730 2024-02-19T15:01:45.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2aed7a2640 1 --2- 172.21.15.78:0/183959485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c5730 0x7f2ae806c2a0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80c8bc0 tx=0x7f2ad802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fe1272aae54620 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2aecfa1640 1 -- 172.21.15.78:0/183959485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8031870 con 0x7f2ae80c5730 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2aecfa1640 1 -- 172.21.15.78:0/183959485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad80319d0 con 0x7f2ae80c5730 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2aecfa1640 1 -- 172.21.15.78:0/183959485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8031d60 con 0x7f2ae80c5730 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 -- 172.21.15.78:0/183959485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c5730 msgr2=0x7f2ae806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 --2- 172.21.15.78:0/183959485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c5730 0x7f2ae806c2a0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80c8bc0 tx=0x7f2ad802b910 comp rx=0 tx=0).stop 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 -- 172.21.15.78:0/183959485 shutdown_connections 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 --2- 172.21.15.78:0/183959485 >> v2:172.21.15.150:3300/0 conn(0x7f2ae80c8e00 0x7f2ae806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 --2- 172.21.15.78:0/183959485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c5730 0x7f2ae806c2a0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 --2- 172.21.15.78:0/183959485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae8152700 0x7f2ae80c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.533+0000 7f2af0207640 1 -- 172.21.15.78:0/183959485 >> 172.21.15.78:0/183959485 conn(0x7f2ae80cac90 msgr2=0x7f2ae80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:45.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.534+0000 7f2af0207640 1 -- 172.21.15.78:0/183959485 shutdown_connections 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.534+0000 7f2af0207640 1 -- 172.21.15.78:0/183959485 wait complete. 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 Processor -- start 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 -- start start 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae80c5730 0x7f2ae80aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 0x7f2ae81eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae8152700 0x7f2ae806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae81ede60 con 0x7f2ae80c8e00 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae81eb420 con 0x7f2ae8152700 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2af0207640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80c3b70 con 0x7f2ae80c5730 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2aed7a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 0x7f2ae81eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.535+0000 7f2aed7a2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 0x7f2ae81eaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42960/0 (socket says 172.21.15.78:42960) 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 -- 172.21.15.78:0/287353677 learned_addr learned my addr 172.21.15.78:0/287353677 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:45.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aedfa3640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae80c5730 0x7f2ae80aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aee7a4640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae8152700 0x7f2ae806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 -- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae80c5730 msgr2=0x7f2ae80aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae80c5730 0x7f2ae80aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 -- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae8152700 msgr2=0x7f2ae806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae8152700 0x7f2ae806c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 -- 172.21.15.78:0/287353677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad80094d0 con 0x7f2ae80c8e00 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2aed7a2640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 0x7f2ae81eaee0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f2ad802b950 tx=0x7f2ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad803ac60 con 0x7f2ae80c8e00 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad803adc0 con 0x7f2ae80c8e00 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae80c3e10 con 0x7f2ae80c8e00 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8044630 con 0x7f2ae80c8e00 2024-02-19T15:01:45.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.536+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae81edaf0 con 0x7f2ae80c8e00 2024-02-19T15:01:45.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.537+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae81edc80 con 0x7f2ae80c8e00 2024-02-19T15:01:45.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.538+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2ad8044790 con 0x7f2ae80c8e00 2024-02-19T15:01:45.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.538+0000 7f2ade7fc640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2acc06b810 0x7f2acc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:45.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.538+0000 7f2aedfa3640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2acc06b810 0x7f2acc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:45.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.539+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f2ad802d040 con 0x7f2ae80c8e00 2024-02-19T15:01:45.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.539+0000 7f2aedfa3640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2acc06b810 0x7f2acc06dcd0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009a20 tx=0x7f2ae4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:45.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.543+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ad8077180 con 0x7f2ae80c8e00 2024-02-19T15:01:45.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.750+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ae8074990 con 0x7f2ae80c8e00 2024-02-19T15:01:45.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.750+0000 7f2ade7fc640 1 -- 172.21.15.78:0/287353677 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f2ad807a840 con 0x7f2ae80c8e00 2024-02-19T15:01:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:45.754 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:45.754 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2acc06b810 msgr2=0x7f2acc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2acc06b810 0x7f2acc06dcd0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009a20 tx=0x7f2ae4009190 comp rx=0 tx=0).stop 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 msgr2=0x7f2ae81eaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 0x7f2ae81eaee0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f2ad802b950 tx=0x7f2ad80046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 shutdown_connections 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2acc06b810 0x7f2acc06dcd0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae80c8e00 0x7f2ae81eaee0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae8152700 0x7f2ae806c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 --2- 172.21.15.78:0/287353677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae80c5730 0x7f2ae80aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 >> 172.21.15.78:0/287353677 conn(0x7f2ae80cac90 msgr2=0x7f2ae8150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 shutdown_connections 2024-02-19T15:01:45.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:45.752+0000 7f2af0207640 1 -- 172.21.15.78:0/287353677 wait complete. 2024-02-19T15:01:46.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:46.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:47.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:47.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.277+0000 7f8e5544e640 1 Processor -- start 2024-02-19T15:01:47.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.277+0000 7f8e5544e640 1 -- start start 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e5544e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e500c8e00 0x7f8e5006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e5544e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e501551f0 con 0x7f8e5006d910 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e5544e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50155490 con 0x7f8e50152700 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e5544e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e50155770 con 0x7f8e500c8e00 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e47fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e500c8e00 0x7f8e5006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e4f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8e5006d910 0x7f8e5015f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55056/0 (socket says 172.21.15.78:55056) 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e4f7fe640 1 -- 172.21.15.78:0/3696835326 learned_addr learned my addr 172.21.15.78:0/3696835326 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e47fff640 1 -- 172.21.15.78:0/3696835326 >> v1:172.21.15.150:6789/0 conn(0x7f8e50152700 legacy=0x7f8e500c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e47fff640 1 -- 172.21.15.78:0/3696835326 >> v1:172.21.15.78:6789/0 conn(0x7f8e5006d910 legacy=0x7f8e5015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e4e7fc640 1 -- 172.21.15.78:0/3696835326 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989137390 0 0) 0x7f8e50155490 con 0x7f8e50152700 2024-02-19T15:01:47.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.278+0000 7f8e47fff640 1 -- 172.21.15.78:0/3696835326 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50155a50 con 0x7f8e500c8e00 2024-02-19T15:01:47.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.279+0000 7f8e47fff640 1 --2- 172.21.15.78:0/3696835326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e5006c2a0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f8e380097a0 tx=0x7f8e3802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bb010d839c059f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:47.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.279+0000 7f8e4e7fc640 1 -- 172.21.15.78:0/3696835326 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38030710 con 0x7f8e500c8e00 2024-02-19T15:01:47.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.279+0000 7f8e4e7fc640 1 -- 172.21.15.78:0/3696835326 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3802e050 con 0x7f8e500c8e00 2024-02-19T15:01:47.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.279+0000 7f8e4e7fc640 1 -- 172.21.15.78:0/3696835326 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38030a90 con 0x7f8e500c8e00 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 -- 172.21.15.78:0/3696835326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 msgr2=0x7f8e5006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 --2- 172.21.15.78:0/3696835326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e5006c2a0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f8e380097a0 tx=0x7f8e3802b790 comp rx=0 tx=0).stop 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 -- 172.21.15.78:0/3696835326 shutdown_connections 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 --2- 172.21.15.78:0/3696835326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e5006c2a0 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 -- 172.21.15.78:0/3696835326 >> 172.21.15.78:0/3696835326 conn(0x7f8e500cac90 msgr2=0x7f8e500cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 -- 172.21.15.78:0/3696835326 shutdown_connections 2024-02-19T15:01:47.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.280+0000 7f8e5544e640 1 -- 172.21.15.78:0/3696835326 wait complete. 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.281+0000 7f8e5544e640 1 Processor -- start 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.281+0000 7f8e5544e640 1 -- start start 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e5006d910 0x7f8e500bfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e500c2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e50152700 0x7f8e501edf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500bf6e0 con 0x7f8e500c8e00 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500c2640 con 0x7f8e50152700 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500c3160 con 0x7f8e5006d910 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e500c2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e500c2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42984/0 (socket says 172.21.15.78:42984) 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 -- 172.21.15.78:0/875162183 learned_addr learned my addr 172.21.15.78:0/875162183 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:47.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e4effd640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e5006d910 0x7f8e500bfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e4f7fe640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e50152700 0x7f8e501edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 -- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e5006d910 msgr2=0x7f8e500bfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e5006d910 0x7f8e500bfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 -- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e50152700 msgr2=0x7f8e501edf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e50152700 0x7f8e501edf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 -- 172.21.15.78:0/875162183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38009400 con 0x7f8e500c8e00 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e4effd640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e5006d910 0x7f8e500bfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e47fff640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e500c2a60 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f8e380098c0 tx=0x7f8e380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e380329d0 con 0x7f8e500c8e00 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38032b50 con 0x7f8e500c8e00 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e501eeac0 con 0x7f8e500c8e00 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3803c470 con 0x7f8e500c8e00 2024-02-19T15:01:47.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.282+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e500bf090 con 0x7f8e500c8e00 2024-02-19T15:01:47.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.284+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e50074990 con 0x7f8e500c8e00 2024-02-19T15:01:47.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.284+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8e38039040 con 0x7f8e500c8e00 2024-02-19T15:01:47.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.284+0000 7f8e477fe640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e3006b810 0x7f8e3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:47.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.284+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f8e380ac0a0 con 0x7f8e500c8e00 2024-02-19T15:01:47.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.284+0000 7f8e4effd640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e3006b810 0x7f8e3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:47.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.285+0000 7f8e4effd640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e3006b810 0x7f8e3006dcd0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000f60 tx=0x7f8e40009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:47.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.289+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8e38076910 con 0x7f8e500c8e00 2024-02-19T15:01:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.493+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e500c1600 con 0x7f8e500c8e00 2024-02-19T15:01:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.494+0000 7f8e477fe640 1 -- 172.21.15.78:0/875162183 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f8e38079fd0 con 0x7f8e500c8e00 2024-02-19T15:01:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:47.497 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.495+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e3006b810 msgr2=0x7f8e3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.495+0000 7f8e5544e640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e3006b810 0x7f8e3006dcd0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000f60 tx=0x7f8e40009040 comp rx=0 tx=0).stop 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.495+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 msgr2=0x7f8e500c2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.495+0000 7f8e5544e640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e500c2a60 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f8e380098c0 tx=0x7f8e380046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 shutdown_connections 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e3006b810 0x7f8e3006dcd0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e500c8e00 0x7f8e500c2a60 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e50152700 0x7f8e501edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 --2- 172.21.15.78:0/875162183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e5006d910 0x7f8e500bfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 >> 172.21.15.78:0/875162183 conn(0x7f8e500cac90 msgr2=0x7f8e500c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 shutdown_connections 2024-02-19T15:01:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:47.496+0000 7f8e5544e640 1 -- 172.21.15.78:0/875162183 wait complete. 2024-02-19T15:01:47.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:47.912 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:48.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:49.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.024+0000 7f44bf5c7640 1 Processor -- start 2024-02-19T15:01:49.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.025+0000 7f44bf5c7640 1 -- start start 2024-02-19T15:01:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.025+0000 7f44bf5c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44b8152830 0x7f44b80c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.025+0000 7f44bf5c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c7990 0x7f44b806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.025+0000 7f44bf5c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b81586a0 con 0x7f44b80c5760 2024-02-19T15:01:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.025+0000 7f44bf5c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b80c6550 con 0x7f44b8152830 2024-02-19T15:01:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.025+0000 7f44bf5c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b8159320 con 0x7f44b80c7990 2024-02-19T15:01:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bcb62640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f44b80c5760 0x7f44b80c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36384/0 (socket says 172.21.15.78:36384) 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bcb62640 1 -- 172.21.15.78:0/1072587023 learned_addr learned my addr 172.21.15.78:0/1072587023 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bd363640 1 --2- 172.21.15.78:0/1072587023 >> v2:172.21.15.150:3300/0 conn(0x7f44b8152830 0x7f44b80c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bdb64640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c7990 0x7f44b806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44affff640 1 -- 172.21.15.78:0/1072587023 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738186357 0 0) 0x7f44b81586a0 con 0x7f44b80c5760 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44affff640 1 -- 172.21.15.78:0/1072587023 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a4003520 con 0x7f44b80c5760 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bd363640 1 -- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c7990 msgr2=0x7f44b806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bd363640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c7990 0x7f44b806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bd363640 1 -- 172.21.15.78:0/1072587023 >> v1:172.21.15.78:6789/0 conn(0x7f44b80c5760 legacy=0x7f44b80c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bd363640 1 -- 172.21.15.78:0/1072587023 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8158980 con 0x7f44b8152830 2024-02-19T15:01:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.026+0000 7f44bdb64640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c7990 0x7f44b806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:49.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.027+0000 7f44bd363640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b8152830 0x7f44b80c5220 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f44b40097d0 tx=0x7f44b402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=370fea838f9bd04a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.027+0000 7f44affff640 1 -- 172.21.15.78:0/1072587023 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4031770 con 0x7f44b8152830 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.027+0000 7f44affff640 1 -- 172.21.15.78:0/1072587023 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b40318d0 con 0x7f44b8152830 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.027+0000 7f44affff640 1 -- 172.21.15.78:0/1072587023 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4031c40 con 0x7f44b8152830 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 -- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b8152830 msgr2=0x7f44b80c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b8152830 0x7f44b80c5220 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f44b40097d0 tx=0x7f44b402b810 comp rx=0 tx=0).stop 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 -- 172.21.15.78:0/1072587023 shutdown_connections 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c7990 0x7f44b806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/1072587023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b8152830 0x7f44b80c5220 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 -- 172.21.15.78:0/1072587023 >> 172.21.15.78:0/1072587023 conn(0x7f44b80ca8a0 msgr2=0x7f44b80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 -- 172.21.15.78:0/1072587023 shutdown_connections 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.028+0000 7f44bf5c7640 1 -- 172.21.15.78:0/1072587023 wait complete. 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 Processor -- start 2024-02-19T15:01:49.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 -- start start 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c5760 0x7f44b80c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 0x7f44b80c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b80bf280 0x7f44b81e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b81e84a0 con 0x7f44b8152830 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.029+0000 7f44bf5c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80bfff0 con 0x7f44b80bf280 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bf5c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80bf730 con 0x7f44b80c5760 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 0x7f44b80c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 0x7f44b80c2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53994/0 (socket says 172.21.15.78:53994) 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 -- 172.21.15.78:0/133827604 learned_addr learned my addr 172.21.15.78:0/133827604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bdb64640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b80bf280 0x7f44b81e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bd363640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c5760 0x7f44b80c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 -- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c5760 msgr2=0x7f44b80c0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c5760 0x7f44b80c0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 -- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b80bf280 msgr2=0x7f44b81e7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b80bf280 0x7f44b81e7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 -- 172.21.15.78:0/133827604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0009a70 con 0x7f44b8152830 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bcb62640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 0x7f44b80c2480 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f44a800c430 tx=0x7f44a800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a8005ce0 con 0x7f44b8152830 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a8005e40 con 0x7f44b8152830 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a80072f0 con 0x7f44b8152830 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.030+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b4009400 con 0x7f44b8152830 2024-02-19T15:01:49.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.031+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b80c1a10 con 0x7f44b8152830 2024-02-19T15:01:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.032+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b8074990 con 0x7f44b8152830 2024-02-19T15:01:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.032+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f44a8007450 con 0x7f44b8152830 2024-02-19T15:01:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.032+0000 7f44adffb640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f449c06b840 0x7f449c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.032+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f44a8089910 con 0x7f44b8152830 2024-02-19T15:01:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.032+0000 7f44bd363640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f449c06b840 0x7f449c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:49.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.033+0000 7f44bd363640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f449c06b840 0x7f449c06dd00 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f44b4036000 tx=0x7f44b4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:49.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.037+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f44a80541a0 con 0x7f44b8152830 2024-02-19T15:01:49.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.244+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44b806dae0 con 0x7f44b8152830 2024-02-19T15:01:49.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.244+0000 7f44adffb640 1 -- 172.21.15.78:0/133827604 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f44a8017090 con 0x7f44b8152830 2024-02-19T15:01:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:49.247 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:49.247 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:49.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f449c06b840 msgr2=0x7f449c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:49.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f449c06b840 0x7f449c06dd00 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f44b4036000 tx=0x7f44b4008030 comp rx=0 tx=0).stop 2024-02-19T15:01:49.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 msgr2=0x7f44b80c2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:49.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 0x7f44b80c2480 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f44a800c430 tx=0x7f44a800c830 comp rx=0 tx=0).stop 2024-02-19T15:01:49.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 shutdown_connections 2024-02-19T15:01:49.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f449c06b840 0x7f449c06dd00 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44b8152830 0x7f44b80c2480 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44b80bf280 0x7f44b81e7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 --2- 172.21.15.78:0/133827604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44b80c5760 0x7f44b80c0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 >> 172.21.15.78:0/133827604 conn(0x7f44b80ca8a0 msgr2=0x7f44b80c8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 shutdown_connections 2024-02-19T15:01:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:49.246+0000 7f44bf5c7640 1 -- 172.21.15.78:0/133827604 wait complete. 2024-02-19T15:01:49.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:49.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:50.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:50.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.776+0000 7f30b428f640 1 Processor -- start 2024-02-19T15:01:50.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.776+0000 7f30b428f640 1 -- start start 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b428f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30ac0c8500 0x7f30ac0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b428f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c42e0 0x7f30ac158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b428f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0c34a0 con 0x7f30ac0c2f50 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b428f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac06c090 con 0x7f30ac0c8500 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b428f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac06d6a0 con 0x7f30ac0c42e0 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b282c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f30ac0c2f50 0x7f30ac158b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36390/0 (socket says 172.21.15.78:36390) 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b282c640 1 -- 172.21.15.78:0/86580312 learned_addr learned my addr 172.21.15.78:0/86580312 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b182a640 1 --2- 172.21.15.78:0/86580312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c42e0 0x7f30ac158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.777+0000 7f30b202b640 1 --2- 172.21.15.78:0/86580312 >> v2:172.21.15.150:3300/0 conn(0x7f30ac0c8500 0x7f30ac0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b1029640 1 -- 172.21.15.78:0/86580312 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411400144 0 0) 0x7f30ac0c34a0 con 0x7f30ac0c2f50 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b1029640 1 -- 172.21.15.78:0/86580312 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003520 con 0x7f30ac0c2f50 2024-02-19T15:01:50.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b182a640 1 -- 172.21.15.78:0/86580312 >> v2:172.21.15.150:3300/0 conn(0x7f30ac0c8500 msgr2=0x7f30ac0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:50.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b182a640 1 --2- 172.21.15.78:0/86580312 >> v2:172.21.15.150:3300/0 conn(0x7f30ac0c8500 0x7f30ac0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:50.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b182a640 1 -- 172.21.15.78:0/86580312 >> v1:172.21.15.78:6789/0 conn(0x7f30ac0c2f50 legacy=0x7f30ac158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:50.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b182a640 1 -- 172.21.15.78:0/86580312 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac06c8e0 con 0x7f30ac0c42e0 2024-02-19T15:01:50.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.778+0000 7f30b202b640 1 --2- 172.21.15.78:0/86580312 >> v2:172.21.15.150:3300/0 conn(0x7f30ac0c8500 0x7f30ac0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b182a640 1 --2- 172.21.15.78:0/86580312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c42e0 0x7f30ac158650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f309c0097a0 tx=0x7f309c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86ce604912ef85ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b1029640 1 -- 172.21.15.78:0/86580312 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c02e070 con 0x7f30ac0c42e0 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b1029640 1 -- 172.21.15.78:0/86580312 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c0315a0 con 0x7f30ac0c42e0 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b1029640 1 -- 172.21.15.78:0/86580312 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c031930 con 0x7f30ac0c42e0 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 -- 172.21.15.78:0/86580312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c42e0 msgr2=0x7f30ac158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 --2- 172.21.15.78:0/86580312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c42e0 0x7f30ac158650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f309c0097a0 tx=0x7f309c02b840 comp rx=0 tx=0).stop 2024-02-19T15:01:50.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 -- 172.21.15.78:0/86580312 shutdown_connections 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 --2- 172.21.15.78:0/86580312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c42e0 0x7f30ac158650 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 --2- 172.21.15.78:0/86580312 >> v2:172.21.15.150:3300/0 conn(0x7f30ac0c8500 0x7f30ac0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 -- 172.21.15.78:0/86580312 >> 172.21.15.78:0/86580312 conn(0x7f30ac0aa300 msgr2=0x7f30ac0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 -- 172.21.15.78:0/86580312 shutdown_connections 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.779+0000 7f30b428f640 1 -- 172.21.15.78:0/86580312 wait complete. 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 Processor -- start 2024-02-19T15:01:50.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 -- start start 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c2f50 0x7f30ac1eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30ac0c8500 0x7f30ac1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 0x7f30ac1503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac1eaa30 con 0x7f30ac070aa0 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac071010 con 0x7f30ac0c8500 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b428f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0712f0 con 0x7f30ac0c2f50 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b282c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 0x7f30ac1503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b202b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c2f50 0x7f30ac1eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b282c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 0x7f30ac1503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54010/0 (socket says 172.21.15.78:54010) 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b282c640 1 -- 172.21.15.78:0/1646003045 learned_addr learned my addr 172.21.15.78:0/1646003045 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:50.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.781+0000 7f30b182a640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30ac0c8500 0x7f30ac1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b282c640 1 -- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c2f50 msgr2=0x7f30ac1eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b282c640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c2f50 0x7f30ac1eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b282c640 1 -- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30ac0c8500 msgr2=0x7f30ac1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b282c640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30ac0c8500 0x7f30ac1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b282c640 1 -- 172.21.15.78:0/1646003045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a8009400 con 0x7f30ac070aa0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b202b640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c2f50 0x7f30ac1eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b282c640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 0x7f30ac1503f0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f30a400cb40 tx=0x7f30a400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a400aca0 con 0x7f30ac070aa0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a400ae00 con 0x7f30ac070aa0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c009400 con 0x7f30ac070aa0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4012620 con 0x7f30ac070aa0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.782+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1eafd0 con 0x7f30ac070aa0 2024-02-19T15:01:50.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.783+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac0714b0 con 0x7f30ac070aa0 2024-02-19T15:01:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.784+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f30a4007070 con 0x7f30ac070aa0 2024-02-19T15:01:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.784+0000 7f30a2ffd640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f309006b840 0x7f309006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.784+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f30a408a350 con 0x7f30ac070aa0 2024-02-19T15:01:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.784+0000 7f30b202b640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f309006b840 0x7f309006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:50.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.784+0000 7f30b202b640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f309006b840 0x7f309006dd00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f30ac070720 tx=0x7f30a8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:50.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.789+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f30a4054c70 con 0x7f30ac070aa0 2024-02-19T15:01:50.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.996+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30ac071c20 con 0x7f30ac070aa0 2024-02-19T15:01:50.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.997+0000 7f30a2ffd640 1 -- 172.21.15.78:0/1646003045 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f30a4058330 con 0x7f30ac070aa0 2024-02-19T15:01:51.000 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:51.000 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:51.000 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f309006b840 msgr2=0x7f309006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f309006b840 0x7f309006dd00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f30ac070720 tx=0x7f30a8009190 comp rx=0 tx=0).stop 2024-02-19T15:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 msgr2=0x7f30ac1503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 0x7f30ac1503f0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f30a400cb40 tx=0x7f30a400a060 comp rx=0 tx=0).stop 2024-02-19T15:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 shutdown_connections 2024-02-19T15:01:51.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f309006b840 0x7f309006dd00 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30ac070aa0 0x7f30ac1503f0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30ac0c8500 0x7f30ac1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 --2- 172.21.15.78:0/1646003045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30ac0c2f50 0x7f30ac1eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 >> 172.21.15.78:0/1646003045 conn(0x7f30ac0aa300 msgr2=0x7f30ac0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 shutdown_connections 2024-02-19T15:01:51.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:50.999+0000 7f30b428f640 1 -- 172.21.15.78:0/1646003045 wait complete. 2024-02-19T15:01:51.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:51.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:52.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:52.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.530+0000 7f0b7e7f6640 1 Processor -- start 2024-02-19T15:01:52.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.530+0000 7f0b7e7f6640 1 -- start start 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7e7f6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b781528d0 0x7f0b780c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7e7f6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b780c19c0 0x7f0b78155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7e7f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b78072180 0x7f0b781555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7e7f6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780734c0 con 0x7f0b780c19c0 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7e7f6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b78155af0 con 0x7f0b781528d0 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7e7f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78072e80 con 0x7f0b78072180 2024-02-19T15:01:52.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b780c19c0 0x7f0b78155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b780c19c0 0x7f0b78155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54014/0 (socket says 172.21.15.78:54014) 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 -- 172.21.15.78:0/346589618 learned_addr learned my addr 172.21.15.78:0/346589618 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b7cd93640 1 --2- 172.21.15.78:0/346589618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b78072180 0x7f0b781555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b77fff640 1 --2- 172.21.15.78:0/346589618 >> v2:172.21.15.150:3300/0 conn(0x7f0b781528d0 0x7f0b780c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 -- 172.21.15.78:0/346589618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b78072180 msgr2=0x7f0b781555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 --2- 172.21.15.78:0/346589618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b78072180 0x7f0b781555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 -- 172.21.15.78:0/346589618 >> v2:172.21.15.150:3300/0 conn(0x7f0b781528d0 msgr2=0x7f0b780c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 --2- 172.21.15.78:0/346589618 >> v2:172.21.15.150:3300/0 conn(0x7f0b781528d0 0x7f0b780c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.531+0000 7f0b777fe640 1 -- 172.21.15.78:0/346589618 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7806e990 con 0x7f0b780c19c0 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.532+0000 7f0b777fe640 1 --2- 172.21.15.78:0/346589618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b780c19c0 0x7f0b78155070 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f0b780c85c0 tx=0x7f0b6c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e4b2a53b8c820 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.532+0000 7f0b76ffd640 1 -- 172.21.15.78:0/346589618 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0317e0 con 0x7f0b780c19c0 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.532+0000 7f0b76ffd640 1 -- 172.21.15.78:0/346589618 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c031940 con 0x7f0b780c19c0 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.532+0000 7f0b76ffd640 1 -- 172.21.15.78:0/346589618 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c031cd0 con 0x7f0b780c19c0 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/346589618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b780c19c0 msgr2=0x7f0b78155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/346589618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b780c19c0 0x7f0b78155070 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f0b780c85c0 tx=0x7f0b6c02b880 comp rx=0 tx=0).stop 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/346589618 shutdown_connections 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/346589618 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b78072180 0x7f0b781555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/346589618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b780c19c0 0x7f0b78155070 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/346589618 >> v2:172.21.15.150:3300/0 conn(0x7f0b781528d0 0x7f0b780c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/346589618 >> 172.21.15.78:0/346589618 conn(0x7f0b780c7590 msgr2=0x7f0b780c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/346589618 shutdown_connections 2024-02-19T15:01:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.533+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/346589618 wait complete. 2024-02-19T15:01:52.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.534+0000 7f0b7e7f6640 1 Processor -- start 2024-02-19T15:01:52.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.534+0000 7f0b7e7f6640 1 -- start start 2024-02-19T15:01:52.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7e7f6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 0x7f0b781e0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7e7f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780c19c0 0x7f0b7814e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7e7f6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b781528d0 0x7f0b7814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7e7f6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780a89d0 con 0x7f0b78072180 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7e7f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780a7270 con 0x7f0b780c19c0 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7e7f6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b781eb550 con 0x7f0b781528d0 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 0x7f0b781e0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 0x7f0b781e0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54026/0 (socket says 172.21.15.78:54026) 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 -- 172.21.15.78:0/2850810910 learned_addr learned my addr 172.21.15.78:0/2850810910 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b777fe640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780c19c0 0x7f0b7814e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b7cd93640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b781528d0 0x7f0b7814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 -- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b781528d0 msgr2=0x7f0b7814e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b781528d0 0x7f0b7814e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 -- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780c19c0 msgr2=0x7f0b7814e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780c19c0 0x7f0b7814e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b77fff640 1 -- 172.21.15.78:0/2850810910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0094d0 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.535+0000 7f0b777fe640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780c19c0 0x7f0b7814e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.536+0000 7f0b77fff640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 0x7f0b781e0650 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f0b6800e870 tx=0x7f0b6800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.536+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680094f0 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.536+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6800c940 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.536+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6800ccf0 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.536+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b781eba80 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.536+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b781eb7a0 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.537+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b78074990 con 0x7f0b78072180 2024-02-19T15:01:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.537+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0b68013470 con 0x7f0b78072180 2024-02-19T15:01:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.538+0000 7f0b74ff9640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b6406b810 0x7f0b6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.538+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f0b6808a380 con 0x7f0b78072180 2024-02-19T15:01:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.538+0000 7f0b777fe640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b6406b810 0x7f0b6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.538+0000 7f0b777fe640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b6406b810 0x7f0b6406dcd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c036000 tx=0x7f0b6c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:52.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.542+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0b68054ca0 con 0x7f0b78072180 2024-02-19T15:01:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.751+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b78072c10 con 0x7f0b78072180 2024-02-19T15:01:52.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.751+0000 7f0b74ff9640 1 -- 172.21.15.78:0/2850810910 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f0b68058360 con 0x7f0b78072180 2024-02-19T15:01:52.754 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:52.754 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:52.754 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b6406b810 msgr2=0x7f0b6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b6406b810 0x7f0b6406dcd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c036000 tx=0x7f0b6c03d040 comp rx=0 tx=0).stop 2024-02-19T15:01:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 msgr2=0x7f0b781e0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 0x7f0b781e0650 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f0b6800e870 tx=0x7f0b6800ed30 comp rx=0 tx=0).stop 2024-02-19T15:01:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 shutdown_connections 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0b6406b810 0x7f0b6406dcd0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b781528d0 0x7f0b7814e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780c19c0 0x7f0b7814e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 --2- 172.21.15.78:0/2850810910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b78072180 0x7f0b781e0650 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 >> 172.21.15.78:0/2850810910 conn(0x7f0b780c7590 msgr2=0x7f0b780c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 shutdown_connections 2024-02-19T15:01:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:52.753+0000 7f0b7e7f6640 1 -- 172.21.15.78:0/2850810910 wait complete. 2024-02-19T15:01:53.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:53.169 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:54.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:54.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.281+0000 7f0e620d4640 1 Processor -- start 2024-02-19T15:01:54.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.281+0000 7f0e620d4640 1 -- start start 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e620d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0e5c0bdf20 0x7f0e5c0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e620d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c158c40 con 0x7f0e5c0ac390 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e620d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0c8de0 con 0x7f0e5c0c55f0 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e620d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c0c5a30 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0e5c0bdf20 0x7f0e5c0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0e5c0ac390 0x7f0e5c158530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36392/0 (socket says 172.21.15.78:36392) 2024-02-19T15:01:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5b7fe640 1 -- 172.21.15.78:0/4246536680 learned_addr learned my addr 172.21.15.78:0/4246536680 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5affd640 1 -- 172.21.15.78:0/4246536680 >> v1:172.21.15.150:6789/0 conn(0x7f0e5c0c55f0 legacy=0x7f0e5c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5affd640 1 -- 172.21.15.78:0/4246536680 >> v1:172.21.15.78:6789/0 conn(0x7f0e5c0ac390 legacy=0x7f0e5c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5affd640 1 -- 172.21.15.78:0/4246536680 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c155a60 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.282+0000 7f0e5a7fc640 1 -- 172.21.15.78:0/4246536680 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233735387 0 0) 0x7f0e5c158c40 con 0x7f0e5c0ac390 2024-02-19T15:01:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.283+0000 7f0e5affd640 1 --2- 172.21.15.78:0/4246536680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c0ac770 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f0e500097a0 tx=0x7f0e5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3506499d0b7d5812 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.283+0000 7f0e5a7fc640 1 -- 172.21.15.78:0/4246536680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5002e070 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.283+0000 7f0e5a7fc640 1 -- 172.21.15.78:0/4246536680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e500315a0 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.283+0000 7f0e5a7fc640 1 -- 172.21.15.78:0/4246536680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50031930 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 -- 172.21.15.78:0/4246536680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 msgr2=0x7f0e5c0ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 --2- 172.21.15.78:0/4246536680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c0ac770 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f0e500097a0 tx=0x7f0e5002b840 comp rx=0 tx=0).stop 2024-02-19T15:01:54.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 -- 172.21.15.78:0/4246536680 shutdown_connections 2024-02-19T15:01:54.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 --2- 172.21.15.78:0/4246536680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c0ac770 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 -- 172.21.15.78:0/4246536680 >> 172.21.15.78:0/4246536680 conn(0x7f0e5c0ca890 msgr2=0x7f0e5c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:54.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 -- 172.21.15.78:0/4246536680 shutdown_connections 2024-02-19T15:01:54.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.284+0000 7f0e620d4640 1 -- 172.21.15.78:0/4246536680 wait complete. 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.285+0000 7f0e620d4640 1 Processor -- start 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.285+0000 7f0e620d4640 1 -- start start 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e620d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e5c0ac390 0x7f0e5c1ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e620d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c1ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e620d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e5c0c55f0 0x7f0e5c0c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e620d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0c07e0 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e620d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0c1970 con 0x7f0e5c0c55f0 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e620d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0c4790 con 0x7f0e5c0ac390 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e5affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c1ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e5affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c1ebcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54046/0 (socket says 172.21.15.78:54046) 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e5affd640 1 -- 172.21.15.78:0/605602043 learned_addr learned my addr 172.21.15.78:0/605602043 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e5b7fe640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e5c0ac390 0x7f0e5c1ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.286+0000 7f0e5bfff640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e5c0c55f0 0x7f0e5c0c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e5affd640 1 -- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e5c0ac390 msgr2=0x7f0e5c1ea7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e5affd640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e5c0ac390 0x7f0e5c1ea7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e5affd640 1 -- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e5c0c55f0 msgr2=0x7f0e5c0c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e5affd640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e5c0c55f0 0x7f0e5c0c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e5affd640 1 -- 172.21.15.78:0/605602043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50009400 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e5affd640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c1ebcc0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f0e50034040 tx=0x7f0e500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50032980 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e50032ae0 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c0c1c30 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50032e90 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.287+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c0c21f0 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.288+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c074990 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.288+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0e5003b3f0 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.289+0000 7f0e43fff640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e4806b810 0x7f0e4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.289+0000 7f0e5b7fe640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e4806b810 0x7f0e4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.289+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f0e500ac340 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.289+0000 7f0e5b7fe640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e4806b810 0x7f0e4806dcd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c007f70 tx=0x7f0e4c007f00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:54.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.293+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0e50076c60 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.499+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e5c0abfc0 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.500+0000 7f0e43fff640 1 -- 172.21.15.78:0/605602043 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f0e5007a320 con 0x7f0e5c0bdf20 2024-02-19T15:01:54.503 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:54.503 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.501+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e4806b810 msgr2=0x7f0e4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.501+0000 7f0e620d4640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e4806b810 0x7f0e4806dcd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c007f70 tx=0x7f0e4c007f00 comp rx=0 tx=0).stop 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.501+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 msgr2=0x7f0e5c1ebcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.501+0000 7f0e620d4640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c1ebcc0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f0e50034040 tx=0x7f0e500046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 shutdown_connections 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e4806b810 0x7f0e4806dcd0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e5c0bdf20 0x7f0e5c1ebcc0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e5c0c55f0 0x7f0e5c0c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 --2- 172.21.15.78:0/605602043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e5c0ac390 0x7f0e5c1ea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 >> 172.21.15.78:0/605602043 conn(0x7f0e5c0ca890 msgr2=0x7f0e5c0cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 shutdown_connections 2024-02-19T15:01:54.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:54.502+0000 7f0e620d4640 1 -- 172.21.15.78:0/605602043 wait complete. 2024-02-19T15:01:54.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:54.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:55.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:56.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.027+0000 7f57c5c76640 1 Processor -- start 2024-02-19T15:01:56.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.027+0000 7f57c5c76640 1 -- start start 2024-02-19T15:01:56.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.027+0000 7f57c5c76640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57c00c30a0 0x7f57c0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:56.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.027+0000 7f57c5c76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c00c3cf0 con 0x7f57c00c5130 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.027+0000 7f57c5c76640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c00c59a0 con 0x7f57c00c5510 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.027+0000 7f57c5c76640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c0158b60 con 0x7f57c00c30a0 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.028+0000 7f57bf7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f57c00c5130 0x7f57c00c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36398/0 (socket says 172.21.15.78:36398) 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.028+0000 7f57b7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57c00c30a0 0x7f57c0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.028+0000 7f57bf7fe640 1 -- 172.21.15.78:0/2206505178 learned_addr learned my addr 172.21.15.78:0/2206505178 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.028+0000 7f57b7fff640 1 -- 172.21.15.78:0/2206505178 >> v1:172.21.15.150:6789/0 conn(0x7f57c00c5510 legacy=0x7f57c015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.028+0000 7f57b7fff640 1 -- 172.21.15.78:0/2206505178 >> v1:172.21.15.78:6789/0 conn(0x7f57c00c5130 legacy=0x7f57c00c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.028+0000 7f57b7fff640 1 -- 172.21.15.78:0/2206505178 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0158530 con 0x7f57c00c30a0 2024-02-19T15:01:56.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57b7fff640 1 --2- 172.21.15.78:0/2206505178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c30a0 0x7f57c0158f80 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f57a80097a0 tx=0x7f57a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86121a5004ef99b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:56.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57beffd640 1 -- 172.21.15.78:0/2206505178 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8030610 con 0x7f57c00c30a0 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57beffd640 1 -- 172.21.15.78:0/2206505178 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a8030770 con 0x7f57c00c30a0 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57beffd640 1 -- 172.21.15.78:0/2206505178 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8030b00 con 0x7f57c00c30a0 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 -- 172.21.15.78:0/2206505178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c30a0 msgr2=0x7f57c0158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 --2- 172.21.15.78:0/2206505178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c30a0 0x7f57c0158f80 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f57a80097a0 tx=0x7f57a802b960 comp rx=0 tx=0).stop 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 -- 172.21.15.78:0/2206505178 shutdown_connections 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 --2- 172.21.15.78:0/2206505178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c30a0 0x7f57c0158f80 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 -- 172.21.15.78:0/2206505178 >> 172.21.15.78:0/2206505178 conn(0x7f57c006b4b0 msgr2=0x7f57c00c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 -- 172.21.15.78:0/2206505178 shutdown_connections 2024-02-19T15:01:56.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.029+0000 7f57c5c76640 1 -- 172.21.15.78:0/2206505178 wait complete. 2024-02-19T15:01:56.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.030+0000 7f57c5c76640 1 Processor -- start 2024-02-19T15:01:56.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 -- start start 2024-02-19T15:01:56.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57c00c30a0 0x7f57c014d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 0x7f57c01eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57c00c5510 0x7f57c0156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c014c790 con 0x7f57c00c5130 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01eb9a0 con 0x7f57c00c30a0 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57c5c76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c006d5e0 con 0x7f57c00c5510 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 0x7f57c01eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 0x7f57c01eb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:54066/0 (socket says 172.21.15.78:54066) 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57b7fff640 1 -- 172.21.15.78:0/1317737441 learned_addr learned my addr 172.21.15.78:0/1317737441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57bf7fe640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57c00c30a0 0x7f57c014d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.031+0000 7f57bffff640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57c00c5510 0x7f57c0156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57b7fff640 1 -- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57c00c5510 msgr2=0x7f57c0156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57b7fff640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57c00c5510 0x7f57c0156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57b7fff640 1 -- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57c00c30a0 msgr2=0x7f57c014d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57b7fff640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57c00c30a0 0x7f57c014d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57b7fff640 1 -- 172.21.15.78:0/1317737441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a8009400 con 0x7f57c00c5130 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57b7fff640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 0x7f57c01eb460 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f57a802bf70 tx=0x7f57a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a803e040 con 0x7f57c00c5130 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a8039cb0 con 0x7f57c00c5130 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8044630 con 0x7f57c00c5130 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c006d830 con 0x7f57c00c5130 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.032+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c006dbc0 con 0x7f57c00c5130 2024-02-19T15:01:56.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.033+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c0074990 con 0x7f57c00c5130 2024-02-19T15:01:56.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.033+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f57a804d3b0 con 0x7f57c00c5130 2024-02-19T15:01:56.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.034+0000 7f57bcff9640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57a006b840 0x7f57a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:56.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.034+0000 7f57bf7fe640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57a006b840 0x7f57a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:56.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.034+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f57a80ac8f0 con 0x7f57c00c5130 2024-02-19T15:01:56.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.034+0000 7f57bf7fe640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57a006b840 0x7f57a006dd00 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f57b0005880 tx=0x7f57b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:56.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.038+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57a8077180 con 0x7f57c00c5130 2024-02-19T15:01:56.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.246+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c00c65a0 con 0x7f57c00c5130 2024-02-19T15:01:56.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.247+0000 7f57bcff9640 1 -- 172.21.15.78:0/1317737441 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f57a807a840 con 0x7f57c00c5130 2024-02-19T15:01:56.250 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:56.250 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:56.250 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57a006b840 msgr2=0x7f57a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57a006b840 0x7f57a006dd00 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f57b0005880 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 msgr2=0x7f57c01eb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 0x7f57c01eb460 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f57a802bf70 tx=0x7f57a8004730 comp rx=0 tx=0).stop 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 shutdown_connections 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57a006b840 0x7f57a006dd00 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57c00c5130 0x7f57c01eb460 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57c00c5510 0x7f57c0156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 --2- 172.21.15.78:0/1317737441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57c00c30a0 0x7f57c014d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 >> 172.21.15.78:0/1317737441 conn(0x7f57c006b4b0 msgr2=0x7f57c0147f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:56.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 shutdown_connections 2024-02-19T15:01:56.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:56.249+0000 7f57c5c76640 1 -- 172.21.15.78:0/1317737441 wait complete. 2024-02-19T15:01:56.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:56.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:57.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.774+0000 7effd998a640 1 Processor -- start 2024-02-19T15:01:57.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.774+0000 7effd998a640 1 -- start start 2024-02-19T15:01:57.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd998a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7effd40c8e00 0x7effd406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:57.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd998a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:57.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd998a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd406cd20 con 0x7effd4152700 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd998a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd41551f0 con 0x7effd40c8e00 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd998a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd4155490 con 0x7effd406d910 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd27fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7effd40c8e00 0x7effd406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd2ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7effd4152700 0x7effd40c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50804/0 (socket says 172.21.15.78:50804) 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd2ffd640 1 -- 172.21.15.78:0/3656510450 learned_addr learned my addr 172.21.15.78:0/3656510450 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd37fe640 1 --2- 172.21.15.78:0/3656510450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd1ffb640 1 -- 172.21.15.78:0/3656510450 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322067087 0 0) 0x7effd406cd20 con 0x7effd4152700 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd27fc640 1 -- 172.21.15.78:0/3656510450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 msgr2=0x7effd406c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd27fc640 1 --2- 172.21.15.78:0/3656510450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd406c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.775+0000 7effd27fc640 1 -- 172.21.15.78:0/3656510450 >> v1:172.21.15.150:6789/0 conn(0x7effd4152700 legacy=0x7effd40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:57.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.776+0000 7effd27fc640 1 -- 172.21.15.78:0/3656510450 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4155770 con 0x7effd40c8e00 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.776+0000 7effd27fc640 1 --2- 172.21.15.78:0/3656510450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd406c2a0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7effc80097a0 tx=0x7effc802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f5b592aaf4101 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.776+0000 7effd1ffb640 1 -- 172.21.15.78:0/3656510450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc802e070 con 0x7effd40c8e00 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.776+0000 7effd1ffb640 1 -- 172.21.15.78:0/3656510450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc80315a0 con 0x7effd40c8e00 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.776+0000 7effd1ffb640 1 -- 172.21.15.78:0/3656510450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8031930 con 0x7effd40c8e00 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 -- 172.21.15.78:0/3656510450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 msgr2=0x7effd406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 --2- 172.21.15.78:0/3656510450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd406c2a0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7effc80097a0 tx=0x7effc802b840 comp rx=0 tx=0).stop 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 -- 172.21.15.78:0/3656510450 shutdown_connections 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 --2- 172.21.15.78:0/3656510450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd406c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 --2- 172.21.15.78:0/3656510450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd406c2a0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:57.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 -- 172.21.15.78:0/3656510450 >> 172.21.15.78:0/3656510450 conn(0x7effd40cac90 msgr2=0x7effd40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 -- 172.21.15.78:0/3656510450 shutdown_connections 2024-02-19T15:01:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.777+0000 7effd998a640 1 -- 172.21.15.78:0/3656510450 wait complete. 2024-02-19T15:01:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.778+0000 7effd998a640 1 Processor -- start 2024-02-19T15:01:57.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.778+0000 7effd998a640 1 -- start start 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd998a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd998a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd414d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd998a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd4152700 0x7effd4147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd998a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40c27e0 con 0x7effd40c8e00 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd998a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40c5d20 con 0x7effd4152700 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd998a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd40c35e0 con 0x7effd406d910 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd414d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd414d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57386/0 (socket says 172.21.15.78:57386) 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 -- 172.21.15.78:0/1319611675 learned_addr learned my addr 172.21.15.78:0/1319611675 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd37fe640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd4152700 0x7effd4147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd2ffd640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd40c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 -- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 msgr2=0x7effd40c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:57.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd40c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 -- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd4152700 msgr2=0x7effd4147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd4152700 0x7effd4147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.779+0000 7effd27fc640 1 -- 172.21.15.78:0/1319611675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8009400 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.780+0000 7effd27fc640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd414d650 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7effc8034040 tx=0x7effc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.780+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8032840 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.780+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc80329c0 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.780+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd40c1e80 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.780+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8032d70 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.780+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd40c3810 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.781+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd4074990 con 0x7effd40c8e00 2024-02-19T15:01:57.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.781+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7effc804e430 con 0x7effd40c8e00 2024-02-19T15:01:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.781+0000 7effbbfff640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc006b840 0x7effc006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.782+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7effc80ac260 con 0x7effd40c8e00 2024-02-19T15:01:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.782+0000 7effd2ffd640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc006b840 0x7effc006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:57.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.782+0000 7effd2ffd640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc006b840 0x7effc006dd00 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7effc40012e0 tx=0x7effc4001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:57.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.786+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7effc8076af0 con 0x7effd40c8e00 2024-02-19T15:01:57.896 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:01:57.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:01:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.995+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effd40c8770 con 0x7effd40c8e00 2024-02-19T15:01:57.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.996+0000 7effbbfff640 1 -- 172.21.15.78:0/1319611675 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7effc807a1b0 con 0x7effd40c8e00 2024-02-19T15:01:57.999 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:57.999 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:57.999 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc006b840 msgr2=0x7effc006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc006b840 0x7effc006dd00 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7effc40012e0 tx=0x7effc4001630 comp rx=0 tx=0).stop 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 msgr2=0x7effd414d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd414d650 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7effc8034040 tx=0x7effc80046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 shutdown_connections 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7effc006b840 0x7effc006dd00 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effd40c8e00 0x7effd414d650 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effd4152700 0x7effd4147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 --2- 172.21.15.78:0/1319611675 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effd406d910 0x7effd40c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:58.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 >> 172.21.15.78:0/1319611675 conn(0x7effd40cac90 msgr2=0x7effd40cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:58.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 shutdown_connections 2024-02-19T15:01:58.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:57.998+0000 7effd998a640 1 -- 172.21.15.78:0/1319611675 wait complete. 2024-02-19T15:01:58.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:01:58.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:01:59.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:01:59.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.523+0000 7f9c52902640 1 Processor -- start 2024-02-19T15:01:59.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.523+0000 7f9c52902640 1 -- start start 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c52902640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c152700 0x7f9c4c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c52902640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9c4c0c5730 0x7f9c4c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c52902640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c4c0c8e00 0x7f9c4c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c52902640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c06cd20 con 0x7f9c4c0c5730 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c52902640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c1551f0 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c52902640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c155490 con 0x7f9c4c152700 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9c4c0c5730 0x7f9c4c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9c4c0c5730 0x7f9c4c06c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57402/0 (socket says 172.21.15.78:57402) 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3119333293 learned_addr learned my addr 172.21.15.78:0/3119333293 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c50e9f640 1 --2- 172.21.15.78:0/3119333293 >> v2:172.21.15.150:3300/0 conn(0x7f9c4c0c8e00 0x7f9c4c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4bfff640 1 --2- 172.21.15.78:0/3119333293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c152700 0x7f9c4c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3119333293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c152700 msgr2=0x7f9c4c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 --2- 172.21.15.78:0/3119333293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c152700 0x7f9c4c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3119333293 >> v2:172.21.15.150:3300/0 conn(0x7f9c4c0c8e00 msgr2=0x7f9c4c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 --2- 172.21.15.78:0/3119333293 >> v2:172.21.15.150:3300/0 conn(0x7f9c4c0c8e00 0x7f9c4c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.524+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3119333293 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c155770 con 0x7f9c4c0c5730 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.525+0000 7f9c4b7fe640 1 --2- 172.21.15.78:0/3119333293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c5730 0x7f9c4c06c2a0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c0c8bc0 tx=0x7f9c4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b1503981510236 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.525+0000 7f9c4affd640 1 -- 172.21.15.78:0/3119333293 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40031870 con 0x7f9c4c0c5730 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.525+0000 7f9c4affd640 1 -- 172.21.15.78:0/3119333293 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c400319d0 con 0x7f9c4c0c5730 2024-02-19T15:01:59.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.525+0000 7f9c4affd640 1 -- 172.21.15.78:0/3119333293 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40031d60 con 0x7f9c4c0c5730 2024-02-19T15:01:59.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 -- 172.21.15.78:0/3119333293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c5730 msgr2=0x7f9c4c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:59.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 --2- 172.21.15.78:0/3119333293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c5730 0x7f9c4c06c2a0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c0c8bc0 tx=0x7f9c4002b910 comp rx=0 tx=0).stop 2024-02-19T15:01:59.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 -- 172.21.15.78:0/3119333293 shutdown_connections 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 --2- 172.21.15.78:0/3119333293 >> v2:172.21.15.150:3300/0 conn(0x7f9c4c0c8e00 0x7f9c4c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 --2- 172.21.15.78:0/3119333293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c5730 0x7f9c4c06c2a0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 --2- 172.21.15.78:0/3119333293 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c152700 0x7f9c4c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 -- 172.21.15.78:0/3119333293 >> 172.21.15.78:0/3119333293 conn(0x7f9c4c0cac90 msgr2=0x7f9c4c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 -- 172.21.15.78:0/3119333293 shutdown_connections 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.526+0000 7f9c52902640 1 -- 172.21.15.78:0/3119333293 wait complete. 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.527+0000 7f9c52902640 1 Processor -- start 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.527+0000 7f9c52902640 1 -- start start 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c52902640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c0c5730 0x7f9c4c0aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c52902640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 0x7f9c4c1eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c52902640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c152700 0x7f9c4c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c52902640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1ede60 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c52902640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1eb420 con 0x7f9c4c152700 2024-02-19T15:01:59.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c52902640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c0c3b70 con 0x7f9c4c0c5730 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 0x7f9c4c1eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 0x7f9c4c1eaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57410/0 (socket says 172.21.15.78:57410) 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3916355460 learned_addr learned my addr 172.21.15.78:0/3916355460 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4bfff640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c0c5730 0x7f9c4c0aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c50e9f640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c152700 0x7f9c4c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c0c5730 msgr2=0x7f9c4c0aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c0c5730 0x7f9c4c0aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c152700 msgr2=0x7f9c4c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c152700 0x7f9c4c06c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 -- 172.21.15.78:0/3916355460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c400094d0 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4bfff640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c0c5730 0x7f9c4c0aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c50e9f640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c152700 0x7f9c4c06c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c4b7fe640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 0x7f9c4c1eaee0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f9c4002b950 tx=0x7f9c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.528+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4003ac60 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.529+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4003adc0 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.529+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c0c3e10 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.529+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40044630 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.529+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c1edaf0 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.529+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c074990 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.530+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9c40044790 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.530+0000 7f9c48ff9640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c3806b810 0x7f9c3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:01:59.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.530+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f9c4002d040 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.530+0000 7f9c4bfff640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c3806b810 0x7f9c3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:01:59.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.531+0000 7f9c4bfff640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c3806b810 0x7f9c3806dcd0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009e10 tx=0x7f9c3c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:01:59.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.535+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9c40077140 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.746+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c4c0c8770 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.746+0000 7f9c48ff9640 1 -- 172.21.15.78:0/3916355460 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f9c4007fe60 con 0x7f9c4c0c8e00 2024-02-19T15:01:59.749 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:01:59.749 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:01:59.750 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c3806b810 msgr2=0x7f9c3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c3806b810 0x7f9c3806dcd0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009e10 tx=0x7f9c3c009190 comp rx=0 tx=0).stop 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 msgr2=0x7f9c4c1eaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 0x7f9c4c1eaee0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f9c4002b950 tx=0x7f9c400046c0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 shutdown_connections 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9c3806b810 0x7f9c3806dcd0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c0c8e00 0x7f9c4c1eaee0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c4c152700 0x7f9c4c06c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 --2- 172.21.15.78:0/3916355460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9c4c0c5730 0x7f9c4c0aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:01:59.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 >> 172.21.15.78:0/3916355460 conn(0x7f9c4c0cac90 msgr2=0x7f9c4c150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:01:59.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 shutdown_connections 2024-02-19T15:01:59.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:01:59.748+0000 7f9c52902640 1 -- 172.21.15.78:0/3916355460 wait complete. 2024-02-19T15:02:00.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:00.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:02:01.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:01.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.274+0000 7f4de97c7640 1 Processor -- start 2024-02-19T15:02:01.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.274+0000 7f4de97c7640 1 -- start start 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de97c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de4152700 0x7f4de40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de97c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4de40c5730 0x7f4de406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de97c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de41551f0 con 0x7f4de40c8e00 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de97c7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de4155490 con 0x7f4de40c5730 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de97c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4155770 con 0x7f4de4152700 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4de40c5730 0x7f4de406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4de40c5730 0x7f4de406c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57422/0 (socket says 172.21.15.78:57422) 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4de40c8e00 0x7f4de4158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:50820/0 (socket says 172.21.15.78:50820) 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 -- 172.21.15.78:0/1706987201 learned_addr learned my addr 172.21.15.78:0/1706987201 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de2ffd640 1 --2- 172.21.15.78:0/1706987201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de4152700 0x7f4de40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:01.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4de27fc640 1 -- 172.21.15.78:0/1706987201 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652889786 0 0) 0x7f4de41551f0 con 0x7f4de40c8e00 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 -- 172.21.15.78:0/1706987201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de4152700 msgr2=0x7f4de40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/1706987201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de4152700 0x7f4de40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 -- 172.21.15.78:0/1706987201 >> v1:172.21.15.150:6789/0 conn(0x7f4de40c8e00 legacy=0x7f4de4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.275+0000 7f4ddbfff640 1 -- 172.21.15.78:0/1706987201 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4155a50 con 0x7f4de40c5730 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.276+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/1706987201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c5730 0x7f4de406c2a0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009870 tx=0x7f4dcc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a99455a9ece076e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.276+0000 7f4de27fc640 1 -- 172.21.15.78:0/1706987201 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc030840 con 0x7f4de40c5730 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.276+0000 7f4de27fc640 1 -- 172.21.15.78:0/1706987201 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc0309a0 con 0x7f4de40c5730 2024-02-19T15:02:01.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.276+0000 7f4de27fc640 1 -- 172.21.15.78:0/1706987201 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc030d30 con 0x7f4de40c5730 2024-02-19T15:02:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 -- 172.21.15.78:0/1706987201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c5730 msgr2=0x7f4de406c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 --2- 172.21.15.78:0/1706987201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c5730 0x7f4de406c2a0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009870 tx=0x7f4dcc02b980 comp rx=0 tx=0).stop 2024-02-19T15:02:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 -- 172.21.15.78:0/1706987201 shutdown_connections 2024-02-19T15:02:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 --2- 172.21.15.78:0/1706987201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c5730 0x7f4de406c2a0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 --2- 172.21.15.78:0/1706987201 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de4152700 0x7f4de40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 -- 172.21.15.78:0/1706987201 >> 172.21.15.78:0/1706987201 conn(0x7f4de40cac90 msgr2=0x7f4de40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 -- 172.21.15.78:0/1706987201 shutdown_connections 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.277+0000 7f4de97c7640 1 -- 172.21.15.78:0/1706987201 wait complete. 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.278+0000 7f4de97c7640 1 Processor -- start 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.278+0000 7f4de97c7640 1 -- start start 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de40c5730 0x7f4de40c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 0x7f4de4154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de4152700 0x7f4de41ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40ace70 con 0x7f4de40c8e00 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de414e6d0 con 0x7f4de4152700 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40c2ac0 con 0x7f4de40c5730 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 0x7f4de4154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 0x7f4de4154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57430/0 (socket says 172.21.15.78:57430) 2024-02-19T15:02:01.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 -- 172.21.15.78:0/3565516738 learned_addr learned my addr 172.21.15.78:0/3565516738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de2ffd640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de40c5730 0x7f4de40c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de37fe640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de4152700 0x7f4de41ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 -- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de40c5730 msgr2=0x7f4de40c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de40c5730 0x7f4de40c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 -- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de4152700 msgr2=0x7f4de41ec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de4152700 0x7f4de41ec5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 -- 172.21.15.78:0/3565516738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0094d0 con 0x7f4de40c8e00 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddbfff640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 0x7f4de4154670 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc000c00 tx=0x7f4dcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc039c90 con 0x7f4de40c8e00 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc039df0 con 0x7f4de40c8e00 2024-02-19T15:02:01.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc043640 con 0x7f4de40c8e00 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de41ebb00 con 0x7f4de40c8e00 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.279+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de41ec0c0 con 0x7f4de40c8e00 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.280+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de41eb960 con 0x7f4de40c8e00 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.281+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4dcc0437a0 con 0x7f4de40c8e00 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.281+0000 7f4ddb7fe640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dc406b810 0x7f4dc406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.281+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f4dcc03e070 con 0x7f4de40c8e00 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.281+0000 7f4de2ffd640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dc406b810 0x7f4dc406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.282+0000 7f4de2ffd640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dc406b810 0x7f4dc406dcd0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009aa0 tx=0x7f4dd4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:01.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.286+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4de41eb960 con 0x7f4de40c8e00 2024-02-19T15:02:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.494+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de41eb960 con 0x7f4de40c8e00 2024-02-19T15:02:01.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.494+0000 7f4ddb7fe640 1 -- 172.21.15.78:0/3565516738 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f4de41eb960 con 0x7f4de40c8e00 2024-02-19T15:02:01.497 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:01.498 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:01.498 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:02:01.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dc406b810 msgr2=0x7f4dc406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:01.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dc406b810 0x7f4dc406dcd0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009aa0 tx=0x7f4dd4009190 comp rx=0 tx=0).stop 2024-02-19T15:02:01.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 msgr2=0x7f4de4154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:01.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 0x7f4de4154670 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc000c00 tx=0x7f4dcc004730 comp rx=0 tx=0).stop 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 shutdown_connections 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4dc406b810 0x7f4dc406dcd0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de40c8e00 0x7f4de4154670 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de4152700 0x7f4de41ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 --2- 172.21.15.78:0/3565516738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4de40c5730 0x7f4de40c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 >> 172.21.15.78:0/3565516738 conn(0x7f4de40cac90 msgr2=0x7f4de41509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 shutdown_connections 2024-02-19T15:02:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:01.496+0000 7f4de97c7640 1 -- 172.21.15.78:0/3565516738 wait complete. 2024-02-19T15:02:01.913 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:01.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:02:02.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:03.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.022+0000 7f1fdb823640 1 Processor -- start 2024-02-19T15:02:03.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.022+0000 7f1fdb823640 1 -- start start 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fdb823640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1fd40719d0 0x7f1fd4071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fdb823640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40c1120 con 0x7f1fd40c32c0 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fdb823640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40c6080 con 0x7f1fd414ece0 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fdb823640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd40c57e0 con 0x7f1fd40719d0 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fd8dbe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1fd40719d0 0x7f1fd4071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fd9dc0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1fd40c32c0 0x7f1fd40c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48054/0 (socket says 172.21.15.78:48054) 2024-02-19T15:02:03.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fd9dc0640 1 -- 172.21.15.78:0/1083848089 learned_addr learned my addr 172.21.15.78:0/1083848089 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:03.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fd8dbe640 1 -- 172.21.15.78:0/1083848089 >> v1:172.21.15.150:6789/0 conn(0x7f1fd414ece0 legacy=0x7f1fd40c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fd8dbe640 1 -- 172.21.15.78:0/1083848089 >> v1:172.21.15.78:6789/0 conn(0x7f1fd40c32c0 legacy=0x7f1fd40c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fd8dbe640 1 -- 172.21.15.78:0/1083848089 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd40c5ac0 con 0x7f1fd40719d0 2024-02-19T15:02:03.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.023+0000 7f1fcbfff640 1 -- 172.21.15.78:0/1083848089 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192911498 0 0) 0x7f1fd40c1120 con 0x7f1fd40c32c0 2024-02-19T15:02:03.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.024+0000 7f1fd8dbe640 1 --2- 172.21.15.78:0/1083848089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd40719d0 0x7f1fd4071e50 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40097a0 tx=0x7f1fc402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b597c6112ba003 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:03.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.024+0000 7f1fcbfff640 1 -- 172.21.15.78:0/1083848089 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4030610 con 0x7f1fd40719d0 2024-02-19T15:02:03.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.024+0000 7f1fcbfff640 1 -- 172.21.15.78:0/1083848089 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4030770 con 0x7f1fd40719d0 2024-02-19T15:02:03.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.024+0000 7f1fcbfff640 1 -- 172.21.15.78:0/1083848089 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4030b00 con 0x7f1fd40719d0 2024-02-19T15:02:03.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 -- 172.21.15.78:0/1083848089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd40719d0 msgr2=0x7f1fd4071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 --2- 172.21.15.78:0/1083848089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd40719d0 0x7f1fd4071e50 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40097a0 tx=0x7f1fc402b960 comp rx=0 tx=0).stop 2024-02-19T15:02:03.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 -- 172.21.15.78:0/1083848089 shutdown_connections 2024-02-19T15:02:03.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 --2- 172.21.15.78:0/1083848089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd40719d0 0x7f1fd4071e50 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 -- 172.21.15.78:0/1083848089 >> 172.21.15.78:0/1083848089 conn(0x7f1fd406b270 msgr2=0x7f1fd406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:03.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 -- 172.21.15.78:0/1083848089 shutdown_connections 2024-02-19T15:02:03.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.025+0000 7f1fdb823640 1 -- 172.21.15.78:0/1083848089 wait complete. 2024-02-19T15:02:03.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.026+0000 7f1fdb823640 1 Processor -- start 2024-02-19T15:02:03.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.026+0000 7f1fdb823640 1 -- start start 2024-02-19T15:02:03.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fdb823640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fd40719d0 0x7f1fd40bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:03.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fdb823640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd40c32c0 0x7f1fd41522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fdb823640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 0x7f1fd41ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fdb823640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd40bbdb0 con 0x7f1fd414ece0 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fdb823640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd41486c0 con 0x7f1fd40c32c0 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fdb823640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd4152810 con 0x7f1fd40719d0 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 0x7f1fd41ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 0x7f1fd41ea4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57446/0 (socket says 172.21.15.78:57446) 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 -- 172.21.15.78:0/4124975807 learned_addr learned my addr 172.21.15.78:0/4124975807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd8dbe640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd40c32c0 0x7f1fd41522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd95bf640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fd40719d0 0x7f1fd40bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 -- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fd40719d0 msgr2=0x7f1fd40bb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fd40719d0 0x7f1fd40bb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 -- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd40c32c0 msgr2=0x7f1fd41522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd40c32c0 0x7f1fd41522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 -- 172.21.15.78:0/4124975807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4009400 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fd9dc0640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 0x7f1fd41ea4f0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00ead0 tx=0x7f1fcc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc004d70 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.027+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc00c9d0 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.028+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd414fcd0 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.028+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc00cd80 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.028+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd4150290 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.029+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd4150420 con 0x7f1fd414ece0 2024-02-19T15:02:03.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.029+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1fcc0070d0 con 0x7f1fd414ece0 2024-02-19T15:02:03.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.029+0000 7f1fc9ffb640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fb806b810 0x7f1fb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:03.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.029+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f1fcc08c870 con 0x7f1fd414ece0 2024-02-19T15:02:03.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.029+0000 7f1fd95bf640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fb806b810 0x7f1fb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:03.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.030+0000 7f1fd95bf640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fb806b810 0x7f1fb806dcd0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0000f60 tx=0x7f1fd0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:03.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.034+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1fcc057190 con 0x7f1fd414ece0 2024-02-19T15:02:03.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.243+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fd4074990 con 0x7f1fd414ece0 2024-02-19T15:02:03.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.243+0000 7f1fc9ffb640 1 -- 172.21.15.78:0/4124975807 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7f1fcc05a850 con 0x7f1fd414ece0 2024-02-19T15:02:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:03.246 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:03.247 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:02:03.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fb806b810 msgr2=0x7f1fb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fb806b810 0x7f1fb806dcd0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0000f60 tx=0x7f1fd0009040 comp rx=0 tx=0).stop 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 msgr2=0x7f1fd41ea4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 0x7f1fd41ea4f0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00ead0 tx=0x7f1fcc00ef90 comp rx=0 tx=0).stop 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 shutdown_connections 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fb806b810 0x7f1fb806dcd0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fd414ece0 0x7f1fd41ea4f0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd40c32c0 0x7f1fd41522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 --2- 172.21.15.78:0/4124975807 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1fd40719d0 0x7f1fd40bb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 >> 172.21.15.78:0/4124975807 conn(0x7f1fd406b270 msgr2=0x7f1fd414d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 shutdown_connections 2024-02-19T15:02:03.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:03.245+0000 7f1fdb823640 1 -- 172.21.15.78:0/4124975807 wait complete. 2024-02-19T15:02:03.662 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:03.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:02:04.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:04.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.772+0000 7ff60835e640 1 Processor -- start 2024-02-19T15:02:04.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.772+0000 7ff60835e640 1 -- start start 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff60835e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6000c5560 0x7ff6000c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff60835e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff6000c7910 0x7ff600154180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff60835e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6000c65a0 con 0x7ff600070720 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff60835e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6000c7530 con 0x7ff6000c7910 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff60835e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6001537b0 con 0x7ff6000c5560 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff6058f9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff6000c7910 0x7ff600154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff6068fb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff600070720 0x7ff600070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48058/0 (socket says 172.21.15.78:48058) 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff6068fb640 1 -- 172.21.15.78:0/1452034468 learned_addr learned my addr 172.21.15.78:0/1452034468 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.773+0000 7ff6060fa640 1 --2- 172.21.15.78:0/1452034468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6000c5560 0x7ff6000c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:04.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.774+0000 7ff6058f9640 1 -- 172.21.15.78:0/1452034468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6000c5560 msgr2=0x7ff6000c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.774+0000 7ff6058f9640 1 --2- 172.21.15.78:0/1452034468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6000c5560 0x7ff6000c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.774+0000 7ff6058f9640 1 -- 172.21.15.78:0/1452034468 >> v1:172.21.15.78:6789/0 conn(0x7ff600070720 legacy=0x7ff600070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.774+0000 7ff6050f8640 1 -- 172.21.15.78:0/1452034468 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108174796 0 0) 0x7ff6000c65a0 con 0x7ff600070720 2024-02-19T15:02:04.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.774+0000 7ff6058f9640 1 -- 172.21.15.78:0/1452034468 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600153a90 con 0x7ff6000c7910 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.774+0000 7ff6058f9640 1 --2- 172.21.15.78:0/1452034468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff600154180 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097a0 tx=0x7ff5f002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1ce462673b2cc9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff6050f8640 1 -- 172.21.15.78:0/1452034468 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f002f050 con 0x7ff6000c7910 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff6050f8640 1 -- 172.21.15.78:0/1452034468 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f0031700 con 0x7ff6000c7910 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff6050f8640 1 -- 172.21.15.78:0/1452034468 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0031a90 con 0x7ff6000c7910 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 -- 172.21.15.78:0/1452034468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 msgr2=0x7ff600154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 --2- 172.21.15.78:0/1452034468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff600154180 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097a0 tx=0x7ff5f002b7a0 comp rx=0 tx=0).stop 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 -- 172.21.15.78:0/1452034468 shutdown_connections 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 --2- 172.21.15.78:0/1452034468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff600154180 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 --2- 172.21.15.78:0/1452034468 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6000c5560 0x7ff6000c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:04.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 -- 172.21.15.78:0/1452034468 >> 172.21.15.78:0/1452034468 conn(0x7ff60006b4b0 msgr2=0x7ff6000c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 -- 172.21.15.78:0/1452034468 shutdown_connections 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.775+0000 7ff60835e640 1 -- 172.21.15.78:0/1452034468 wait complete. 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.776+0000 7ff60835e640 1 Processor -- start 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.776+0000 7ff60835e640 1 -- start start 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff60835e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff600070720 0x7ff6000aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff60835e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6000c5560 0x7ff6000a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff60835e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff60014c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff60835e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6000bf7a0 con 0x7ff6000c7910 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff60835e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6000a8fd0 con 0x7ff6000c5560 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff60835e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6000ab210 con 0x7ff600070720 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff60014c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff60014c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57468/0 (socket says 172.21.15.78:57468) 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6060fa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff600070720 0x7ff6000aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 -- 172.21.15.78:0/2739820511 learned_addr learned my addr 172.21.15.78:0/2739820511 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:04.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6058f9640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6000c5560 0x7ff6000a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 -- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff600070720 msgr2=0x7ff6000aacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff600070720 0x7ff6000aacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 -- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6000c5560 msgr2=0x7ff6000a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6000c5560 0x7ff6000a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.777+0000 7ff6068fb640 1 -- 172.21.15.78:0/2739820511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0009400 con 0x7ff6000c7910 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.778+0000 7ff6068fb640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff60014c820 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c410 tx=0x7ff5f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.778+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80074e0 con 0x7ff6000c7910 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.778+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8007640 con 0x7ff6000c7910 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.778+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6001ebb00 con 0x7ff6000c7910 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.778+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8004a20 con 0x7ff6000c7910 2024-02-19T15:02:04.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.778+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6001ebe70 con 0x7ff6000c7910 2024-02-19T15:02:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.779+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff600074990 con 0x7ff6000c7910 2024-02-19T15:02:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.779+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff5f8004b80 con 0x7ff6000c7910 2024-02-19T15:02:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.780+0000 7ff5f6ffd640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff5e406b810 0x7ff5e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.780+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6274+0+0 (secure 0 0 0) 0x7ff5f8089f80 con 0x7ff6000c7910 2024-02-19T15:02:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.780+0000 7ff6060fa640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff5e406b810 0x7ff5e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:04.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.780+0000 7ff6060fa640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff5e406b810 0x7ff5e406dcd0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc011340 tx=0x7ff5fc00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:04.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.784+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff5f80bddc0 con 0x7ff6000c7910 2024-02-19T15:02:04.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.994+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60006f8b0 con 0x7ff6000c7910 2024-02-19T15:02:04.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.994+0000 7ff5f6ffd640 1 -- 172.21.15.78:0/2739820511 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4187 (secure 0 0 0) 0x7ff5f80583d0 con 0x7ff6000c7910 2024-02-19T15:02:04.998 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:04.998 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:01:34.651492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":214,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":10716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10716":{"gid":10716,"name":"a","rank":1,"incarnation":196,"state":"up:stopping","state_seq":16,"addr":"172.21.15.78:6835/3736272032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3736272032},{"type":"v1","addr":"172.21.15.78:6835","nonce":3736272032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:04.999 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 201 2024-02-19T15:02:05.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff5e406b810 msgr2=0x7ff5e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:05.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff5e406b810 0x7ff5e406dcd0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc011340 tx=0x7ff5fc00b040 comp rx=0 tx=0).stop 2024-02-19T15:02:05.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 msgr2=0x7ff60014c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:05.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff60014c820 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c410 tx=0x7ff5f800c8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:05.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 shutdown_connections 2024-02-19T15:02:05.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff5e406b810 0x7ff5e406dcd0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:05.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6000c7910 0x7ff60014c820 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6000c5560 0x7ff6000a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 --2- 172.21.15.78:0/2739820511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff600070720 0x7ff6000aacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 >> 172.21.15.78:0/2739820511 conn(0x7ff60006b4b0 msgr2=0x7ff600070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.996+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 shutdown_connections 2024-02-19T15:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:04.997+0000 7ff60835e640 1 -- 172.21.15.78:0/2739820511 wait complete. 2024-02-19T15:02:05.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:01:34.651492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 214, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 10716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10716': {'gid': 10716, 'name': 'a', 'rank': 1, 'incarnation': 196, 'state': 'up:stopping', 'state_seq': 16, 'addr': '172.21.15.78:6835/3736272032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6834', 'nonce': 3736272032}, {'type': 'v1', 'addr': '172.21.15.78:6835', 'nonce': 3736272032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:05.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10716:up:stopping 2024-02-19T15:02:06.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:06.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.521+0000 7f96d9b48640 1 Processor -- start 2024-02-19T15:02:06.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d9b48640 1 -- start start 2024-02-19T15:02:06.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d9b48640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f96d4152700 0x7f96d40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:06.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d9b48640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d40c5730 0x7f96d406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d9b48640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d41551f0 con 0x7f96d40c8e00 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d9b48640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d4155490 con 0x7f96d4152700 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d9b48640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d4155770 con 0x7f96d40c5730 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f96d4152700 0x7f96d40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f96d4152700 0x7f96d40c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57478/0 (socket says 172.21.15.78:57478) 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.522+0000 7f96d37fe640 1 -- 172.21.15.78:0/1426282332 learned_addr learned my addr 172.21.15.78:0/1426282332 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.523+0000 7f96caffd640 1 --2- 172.21.15.78:0/1426282332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d40c5730 0x7f96d406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.523+0000 7f96d37fe640 1 -- 172.21.15.78:0/1426282332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d40c5730 msgr2=0x7f96d406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.523+0000 7f96d37fe640 1 --2- 172.21.15.78:0/1426282332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d40c5730 0x7f96d406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.523+0000 7f96d37fe640 1 -- 172.21.15.78:0/1426282332 >> v1:172.21.15.150:6789/0 conn(0x7f96d40c8e00 legacy=0x7f96d4158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.523+0000 7f96d2ffd640 1 -- 172.21.15.78:0/1426282332 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646847317 0 0) 0x7f96d41551f0 con 0x7f96d40c8e00 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.523+0000 7f96d37fe640 1 -- 172.21.15.78:0/1426282332 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d4155a50 con 0x7f96d4152700 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d37fe640 1 --2- 172.21.15.78:0/1426282332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d4152700 0x7f96d40c51f0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009870 tx=0x7f96c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93024368a7515fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d2ffd640 1 -- 172.21.15.78:0/1426282332 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4030840 con 0x7f96d4152700 2024-02-19T15:02:06.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d2ffd640 1 -- 172.21.15.78:0/1426282332 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c40309a0 con 0x7f96d4152700 2024-02-19T15:02:06.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d2ffd640 1 -- 172.21.15.78:0/1426282332 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4030d30 con 0x7f96d4152700 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 -- 172.21.15.78:0/1426282332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d4152700 msgr2=0x7f96d40c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1426282332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d4152700 0x7f96d40c51f0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009870 tx=0x7f96c402b980 comp rx=0 tx=0).stop 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 -- 172.21.15.78:0/1426282332 shutdown_connections 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1426282332 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d40c5730 0x7f96d406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1426282332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d4152700 0x7f96d40c51f0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 -- 172.21.15.78:0/1426282332 >> 172.21.15.78:0/1426282332 conn(0x7f96d40cac90 msgr2=0x7f96d40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 -- 172.21.15.78:0/1426282332 shutdown_connections 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.524+0000 7f96d9b48640 1 -- 172.21.15.78:0/1426282332 wait complete. 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.525+0000 7f96d9b48640 1 Processor -- start 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 -- start start 2024-02-19T15:02:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 0x7f96d41e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d40c8e00 0x7f96d41ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d4152700 0x7f96d41ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d41eb2a0 con 0x7f96d40c5730 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d40ab520 con 0x7f96d40c8e00 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d9b48640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d4149740 con 0x7f96d4152700 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 0x7f96d41e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 0x7f96d41e7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57494/0 (socket says 172.21.15.78:57494) 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.526+0000 7f96d37fe640 1 -- 172.21.15.78:0/1713515438 learned_addr learned my addr 172.21.15.78:0/1713515438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96caffd640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d40c8e00 0x7f96d41ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d3fff640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d4152700 0x7f96d41ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d37fe640 1 -- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d4152700 msgr2=0x7f96d41ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d37fe640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d4152700 0x7f96d41ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d37fe640 1 -- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d40c8e00 msgr2=0x7f96d41ea4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d37fe640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d40c8e00 0x7f96d41ea4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d37fe640 1 -- 172.21.15.78:0/1713515438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c40094d0 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d37fe640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 0x7f96d41e7b90 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f96c402b9c0 tx=0x7f96c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4039a90 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4039bf0 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4043410 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d41eafc0 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.527+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d40ab140 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.528+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d4074990 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.529+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f96c404c3b0 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.529+0000 7f96d0ff9640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96b406b810 0x7f96b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.529+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f96c40ac890 con 0x7f96d40c5730 2024-02-19T15:02:06.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.529+0000 7f96caffd640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96b406b810 0x7f96b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:06.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.529+0000 7f96caffd640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96b406b810 0x7f96b406dcd0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f96d40c1590 tx=0x7f96bc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:06.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.533+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f96c4077280 con 0x7f96d40c5730 2024-02-19T15:02:06.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.744+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96d40aac50 con 0x7f96d40c5730 2024-02-19T15:02:06.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.744+0000 7f96d0ff9640 1 -- 172.21.15.78:0/1713515438 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+3392 (secure 0 0 0) 0x7f96c4034d50 con 0x7f96d40c5730 2024-02-19T15:02:06.747 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:06.747 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":203,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:05.654376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:resolve","state_seq":22,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:06.748 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 203 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.746+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96b406b810 msgr2=0x7f96b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.746+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96b406b810 0x7f96b406dcd0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f96d40c1590 tx=0x7f96bc009190 comp rx=0 tx=0).stop 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.746+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 msgr2=0x7f96d41e7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.746+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 0x7f96d41e7b90 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f96c402b9c0 tx=0x7f96c4004730 comp rx=0 tx=0).stop 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 shutdown_connections 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96b406b810 0x7f96b406dcd0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96d4152700 0x7f96d41ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d40c8e00 0x7f96d41ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 --2- 172.21.15.78:0/1713515438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96d40c5730 0x7f96d41e7b90 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 >> 172.21.15.78:0/1713515438 conn(0x7f96d40cac90 msgr2=0x7f96d4150a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 shutdown_connections 2024-02-19T15:02:06.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:06.747+0000 7f96d9b48640 1 -- 172.21.15.78:0/1713515438 wait complete. 2024-02-19T15:02:07.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:05.654376+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:resolve', 'state_seq': 22, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:07.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:resolve 2024-02-19T15:02:08.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:08.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.275+0000 7f98e13b5640 1 Processor -- start 2024-02-19T15:02:08.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.275+0000 7f98e13b5640 1 -- start start 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98e13b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152700 0x7f98dc0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98e13b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c8e00 0x7f98dc15f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98e13b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc1551f0 con 0x7f98dc0c5730 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98e13b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98dc155490 con 0x7f98dc0c8e00 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98e13b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc155770 con 0x7f98dc152700 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98d3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f98dc0c5730 0x7f98dc06c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54232/0 (socket says 172.21.15.78:54232) 2024-02-19T15:02:08.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98d3fff640 1 -- 172.21.15.78:0/4003012550 learned_addr learned my addr 172.21.15.78:0/4003012550 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98daffd640 1 --2- 172.21.15.78:0/4003012550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152700 0x7f98dc0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98db7fe640 1 --2- 172.21.15.78:0/4003012550 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c8e00 0x7f98dc15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98da7fc640 1 -- 172.21.15.78:0/4003012550 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634498973 0 0) 0x7f98dc1551f0 con 0x7f98dc0c5730 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98da7fc640 1 -- 172.21.15.78:0/4003012550 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c8003520 con 0x7f98dc0c5730 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98daffd640 1 -- 172.21.15.78:0/4003012550 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c8e00 msgr2=0x7f98dc15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98daffd640 1 --2- 172.21.15.78:0/4003012550 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c8e00 0x7f98dc15f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98daffd640 1 -- 172.21.15.78:0/4003012550 >> v1:172.21.15.78:6789/0 conn(0x7f98dc0c5730 legacy=0x7f98dc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98daffd640 1 -- 172.21.15.78:0/4003012550 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc155a50 con 0x7f98dc152700 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.276+0000 7f98db7fe640 1 --2- 172.21.15.78:0/4003012550 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c8e00 0x7f98dc15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.277+0000 7f98daffd640 1 --2- 172.21.15.78:0/4003012550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152700 0x7f98dc0c51f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097a0 tx=0x7f98cc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f9ad5963e0ed79c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.277+0000 7f98da7fc640 1 -- 172.21.15.78:0/4003012550 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc031830 con 0x7f98dc152700 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.277+0000 7f98da7fc640 1 -- 172.21.15.78:0/4003012550 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc031990 con 0x7f98dc152700 2024-02-19T15:02:08.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.277+0000 7f98da7fc640 1 -- 172.21.15.78:0/4003012550 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc031d20 con 0x7f98dc152700 2024-02-19T15:02:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 -- 172.21.15.78:0/4003012550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152700 msgr2=0x7f98dc0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 --2- 172.21.15.78:0/4003012550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152700 0x7f98dc0c51f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097a0 tx=0x7f98cc02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 -- 172.21.15.78:0/4003012550 shutdown_connections 2024-02-19T15:02:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 --2- 172.21.15.78:0/4003012550 >> v2:172.21.15.150:3300/0 conn(0x7f98dc0c8e00 0x7f98dc15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 --2- 172.21.15.78:0/4003012550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc152700 0x7f98dc0c51f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 -- 172.21.15.78:0/4003012550 >> 172.21.15.78:0/4003012550 conn(0x7f98dc0cac90 msgr2=0x7f98dc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 -- 172.21.15.78:0/4003012550 shutdown_connections 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.278+0000 7f98e13b5640 1 -- 172.21.15.78:0/4003012550 wait complete. 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.279+0000 7f98e13b5640 1 Processor -- start 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.279+0000 7f98e13b5640 1 -- start start 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98e13b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c5730 0x7f98dc0c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98e13b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 0x7f98dc0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98e13b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc152700 0x7f98dc0c3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98e13b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0abdf0 con 0x7f98dc0c8e00 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98e13b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0aae60 con 0x7f98dc152700 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98e13b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0ab140 con 0x7f98dc0c5730 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 0x7f98dc0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 0x7f98dc0ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40410/0 (socket says 172.21.15.78:40410) 2024-02-19T15:02:08.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 -- 172.21.15.78:0/3325741308 learned_addr learned my addr 172.21.15.78:0/3325741308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98daffd640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c5730 0x7f98dc0c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98db7fe640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc152700 0x7f98dc0c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 -- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c5730 msgr2=0x7f98dc0c1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c5730 0x7f98dc0c1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 -- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc152700 msgr2=0x7f98dc0c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.280+0000 7f98d3fff640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc152700 0x7f98dc0c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98d3fff640 1 -- 172.21.15.78:0/3325741308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4009a70 con 0x7f98dc0c8e00 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98d3fff640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 0x7f98dc0ac960 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c430 tx=0x7f98c400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4005ce0 con 0x7f98dc0c8e00 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4005e40 con 0x7f98dc0c8e00 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc009400 con 0x7f98dc0c8e00 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40072f0 con 0x7f98dc0c8e00 2024-02-19T15:02:08.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.281+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc0ab700 con 0x7f98dc0c8e00 2024-02-19T15:02:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.282+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc074990 con 0x7f98dc0c8e00 2024-02-19T15:02:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.282+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f98c4007450 con 0x7f98dc0c8e00 2024-02-19T15:02:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.283+0000 7f98d37fe640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98bc06b840 0x7f98bc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.283+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f98c4089980 con 0x7f98dc0c8e00 2024-02-19T15:02:08.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.283+0000 7f98daffd640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98bc06b840 0x7f98bc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:08.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.283+0000 7f98daffd640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98bc06b840 0x7f98bc06dd00 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f98cc036000 tx=0x7f98cc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.287+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98c40541c0 con 0x7f98dc0c8e00 2024-02-19T15:02:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.496+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98dc0c8770 con 0x7f98dc0c8e00 2024-02-19T15:02:08.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.496+0000 7f98d37fe640 1 -- 172.21.15.78:0/3325741308 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+3394 (secure 0 0 0) 0x7f98c4017090 con 0x7f98dc0c8e00 2024-02-19T15:02:08.499 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:08.500 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":204,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:06.643201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:08.500 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 204 2024-02-19T15:02:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98bc06b840 msgr2=0x7f98bc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98bc06b840 0x7f98bc06dd00 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f98cc036000 tx=0x7f98cc03c040 comp rx=0 tx=0).stop 2024-02-19T15:02:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 msgr2=0x7f98dc0ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:08.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 0x7f98dc0ac960 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c430 tx=0x7f98c400c830 comp rx=0 tx=0).stop 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 shutdown_connections 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98bc06b840 0x7f98bc06dd00 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98dc0c8e00 0x7f98dc0ac960 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98dc152700 0x7f98dc0c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 --2- 172.21.15.78:0/3325741308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f98dc0c5730 0x7f98dc0c1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 >> 172.21.15.78:0/3325741308 conn(0x7f98dc0cac90 msgr2=0x7f98dc14e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 shutdown_connections 2024-02-19T15:02:08.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:08.498+0000 7f98e13b5640 1 -- 172.21.15.78:0/3325741308 wait complete. 2024-02-19T15:02:08.915 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 204, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:06.643201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:reconnect', 'state_seq': 23, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:08.915 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:reconnect 2024-02-19T15:02:09.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.028+0000 7f677b6c9640 1 Processor -- start 2024-02-19T15:02:10.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.029+0000 7f677b6c9640 1 -- start start 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.029+0000 7f677b6c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67740c4d30 0x7f67740c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.029+0000 7f677b6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.029+0000 7f677b6c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67740c1120 con 0x7f677414ece0 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.029+0000 7f677b6c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67740c39d0 con 0x7f67740c4d30 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.029+0000 7f677b6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740c6080 con 0x7f67740719d0 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779c66640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6778c64640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67740c4d30 0x7f67740c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779465640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f677414ece0 0x7f67740c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:56506/0 (socket says 172.21.15.78:56506) 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779465640 1 -- 172.21.15.78:0/204675399 learned_addr learned my addr 172.21.15.78:0/204675399 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779c66640 1 -- 172.21.15.78:0/204675399 >> v2:172.21.15.150:3300/0 conn(0x7f67740c4d30 msgr2=0x7f67740c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779c66640 1 --2- 172.21.15.78:0/204675399 >> v2:172.21.15.150:3300/0 conn(0x7f67740c4d30 0x7f67740c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779c66640 1 -- 172.21.15.78:0/204675399 >> v1:172.21.15.150:6789/0 conn(0x7f677414ece0 legacy=0x7f67740c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6779c66640 1 -- 172.21.15.78:0/204675399 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740c57e0 con 0x7f67740719d0 2024-02-19T15:02:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f6778c64640 1 --2- 172.21.15.78:0/204675399 >> v2:172.21.15.150:3300/0 conn(0x7f67740c4d30 0x7f67740c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:10.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.030+0000 7f676bfff640 1 -- 172.21.15.78:0/204675399 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665572250 0 0) 0x7f67740c1120 con 0x7f677414ece0 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.031+0000 7f6779c66640 1 --2- 172.21.15.78:0/204675399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774071e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f676c009d90 tx=0x7f676c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=654ba8217ee96e55 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.031+0000 7f676bfff640 1 -- 172.21.15.78:0/204675399 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c02cc50 con 0x7f67740719d0 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.031+0000 7f676bfff640 1 -- 172.21.15.78:0/204675399 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c02cdb0 con 0x7f67740719d0 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.031+0000 7f676bfff640 1 -- 172.21.15.78:0/204675399 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c034660 con 0x7f67740719d0 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 -- 172.21.15.78:0/204675399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 msgr2=0x7f6774071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 --2- 172.21.15.78:0/204675399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774071e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f676c009d90 tx=0x7f676c02bea0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 -- 172.21.15.78:0/204675399 shutdown_connections 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 --2- 172.21.15.78:0/204675399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774071e60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 --2- 172.21.15.78:0/204675399 >> v2:172.21.15.150:3300/0 conn(0x7f67740c4d30 0x7f67740c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 -- 172.21.15.78:0/204675399 >> 172.21.15.78:0/204675399 conn(0x7f677406b270 msgr2=0x7f677406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 -- 172.21.15.78:0/204675399 shutdown_connections 2024-02-19T15:02:10.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.032+0000 7f677b6c9640 1 -- 172.21.15.78:0/204675399 wait complete. 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.033+0000 7f677b6c9640 1 Processor -- start 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.033+0000 7f677b6c9640 1 -- start start 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f677b6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f677b6c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 0x7f67740abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f677b6c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677414ece0 0x7f677414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f677b6c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677414f950 con 0x7f67740c4d30 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f677b6c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677414f360 con 0x7f677414ece0 2024-02-19T15:02:10.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f677b6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740bfde0 con 0x7f67740719d0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6778c64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 0x7f67740abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6778c64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 0x7f67740abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40414/0 (socket says 172.21.15.78:40414) 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6778c64640 1 -- 172.21.15.78:0/2734090400 learned_addr learned my addr 172.21.15.78:0/2734090400 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779c66640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677414ece0 0x7f677414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 -- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677414ece0 msgr2=0x7f677414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677414ece0 0x7f677414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 -- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 msgr2=0x7f67740abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 0x7f67740abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 -- 172.21.15.78:0/2734090400 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6764009400 con 0x7f67740719d0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6778c64640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 0x7f67740abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6779465640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774150fa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f677000c410 tx=0x7f677000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700074e0 con 0x7f67740719d0 2024-02-19T15:02:10.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.034+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770007640 con 0x7f67740719d0 2024-02-19T15:02:10.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.035+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c009a70 con 0x7f67740719d0 2024-02-19T15:02:10.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.035+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770004a20 con 0x7f67740719d0 2024-02-19T15:02:10.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.035+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67740c03a0 con 0x7f67740719d0 2024-02-19T15:02:10.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.036+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6774074990 con 0x7f67740719d0 2024-02-19T15:02:10.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.036+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6770004b80 con 0x7f67740719d0 2024-02-19T15:02:10.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.036+0000 7f6769ffb640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675806b810 0x7f675806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:10.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.037+0000 7f6778c64640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675806b810 0x7f675806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:10.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.037+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f677008a0c0 con 0x7f67740719d0 2024-02-19T15:02:10.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.037+0000 7f6778c64640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675806b810 0x7f675806dcd0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f67741ec6a0 tx=0x7f6764009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:10.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.046+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f67700549a0 con 0x7f67740719d0 2024-02-19T15:02:10.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.253+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67740aa470 con 0x7f67740719d0 2024-02-19T15:02:10.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.253+0000 7f6769ffb640 1 -- 172.21.15.78:0/2734090400 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f6770058060 con 0x7f67740719d0 2024-02-19T15:02:10.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:10.256 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:10.257 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675806b810 msgr2=0x7f675806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:10.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675806b810 0x7f675806dcd0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f67741ec6a0 tx=0x7f6764009190 comp rx=0 tx=0).stop 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 msgr2=0x7f6774150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774150fa0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f677000c410 tx=0x7f677000c8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 shutdown_connections 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f675806b810 0x7f675806dcd0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67740c4d30 0x7f67740abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677414ece0 0x7f677414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 --2- 172.21.15.78:0/2734090400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740719d0 0x7f6774150fa0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.255+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 >> 172.21.15.78:0/2734090400 conn(0x7f677406b270 msgr2=0x7f67741dc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.256+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 shutdown_connections 2024-02-19T15:02:10.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:10.256+0000 7f677b6c9640 1 -- 172.21.15.78:0/2734090400 wait complete. 2024-02-19T15:02:10.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:10.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:11.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:11.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.782+0000 7f8ede4ff640 1 Processor -- start 2024-02-19T15:02:11.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.782+0000 7f8ede4ff640 1 -- start start 2024-02-19T15:02:11.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ede4ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed8153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:11.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ede4ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ed80aca90 0x7f8ed806ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:11.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ede4ff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed81535a0 con 0x7f8ed806b3c0 2024-02-19T15:02:11.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ede4ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80ac590 con 0x7f8ed80aca90 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ede4ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8152f90 con 0x7f8ed8071b60 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8edca9c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8ed806b3c0 0x7f8ed80ab840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54242/0 (socket says 172.21.15.78:54242) 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8edca9c640 1 -- 172.21.15.78:0/4063201118 learned_addr learned my addr 172.21.15.78:0/4063201118 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ed7fff640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed8153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ed77fe640 1 --2- 172.21.15.78:0/4063201118 >> v2:172.21.15.150:3300/0 conn(0x7f8ed80aca90 0x7f8ed806ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ed6ffd640 1 -- 172.21.15.78:0/4063201118 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851133804 0 0) 0x7f8ed81535a0 con 0x7f8ed806b3c0 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.783+0000 7f8ed6ffd640 1 -- 172.21.15.78:0/4063201118 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec0003520 con 0x7f8ed806b3c0 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.784+0000 7f8ed77fe640 1 -- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 msgr2=0x7f8ed8153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.784+0000 7f8ed77fe640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed8153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.784+0000 7f8ed77fe640 1 -- 172.21.15.78:0/4063201118 >> v1:172.21.15.78:6789/0 conn(0x7f8ed806b3c0 legacy=0x7f8ed80ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.784+0000 7f8ed77fe640 1 -- 172.21.15.78:0/4063201118 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed80ac2b0 con 0x7f8ed80aca90 2024-02-19T15:02:11.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.784+0000 7f8ed7fff640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed8153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:11.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.784+0000 7f8ed77fe640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed80aca90 0x7f8ed806ae80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0097a0 tx=0x7f8ecc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=893ad49145c9fd38 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ed6ffd640 1 -- 172.21.15.78:0/4063201118 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc02d040 con 0x7f8ed80aca90 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ed6ffd640 1 -- 172.21.15.78:0/4063201118 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc031810 con 0x7f8ed80aca90 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ed6ffd640 1 -- 172.21.15.78:0/4063201118 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc031ba0 con 0x7f8ed80aca90 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 -- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed80aca90 msgr2=0x7f8ed806ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed80aca90 0x7f8ed806ae80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0097a0 tx=0x7f8ecc02b8b0 comp rx=0 tx=0).stop 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 -- 172.21.15.78:0/4063201118 shutdown_connections 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed80aca90 0x7f8ed806ae80 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/4063201118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed8153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 -- 172.21.15.78:0/4063201118 >> 172.21.15.78:0/4063201118 conn(0x7f8ed80c7800 msgr2=0x7f8ed80c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 -- 172.21.15.78:0/4063201118 shutdown_connections 2024-02-19T15:02:11.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.785+0000 7f8ede4ff640 1 -- 172.21.15.78:0/4063201118 wait complete. 2024-02-19T15:02:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.786+0000 7f8ede4ff640 1 Processor -- start 2024-02-19T15:02:11.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 -- start start 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 0x7f8ed814bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed8145b00 0x7f8ed814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed814c940 con 0x7f8ed80aca90 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81ebf40 con 0x7f8ed8145b00 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ede4ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81ec220 con 0x7f8ed8071b60 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 0x7f8ed814bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 0x7f8ed814bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40420/0 (socket says 172.21.15.78:40420) 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.787+0000 7f8ed77fe640 1 -- 172.21.15.78:0/3479722317 learned_addr learned my addr 172.21.15.78:0/3479722317 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8edca9c640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed8145b00 0x7f8ed814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 -- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed8145b00 msgr2=0x7f8ed814c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:11.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed8145b00 0x7f8ed814c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 -- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 msgr2=0x7f8ed814bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 0x7f8ed814bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 -- 172.21.15.78:0/3479722317 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8009400 con 0x7f8ed8071b60 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed77fe640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 0x7f8ed814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed7fff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed814ed00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e930 tx=0x7f8ec800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800ca00 con 0x7f8ed8071b60 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec800cb60 con 0x7f8ed8071b60 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc009400 con 0x7f8ed8071b60 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8013470 con 0x7f8ed8071b60 2024-02-19T15:02:11.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.788+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed81ec7e0 con 0x7f8ed8071b60 2024-02-19T15:02:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.789+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed8074990 con 0x7f8ed8071b60 2024-02-19T15:02:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.790+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8ec80135d0 con 0x7f8ed8071b60 2024-02-19T15:02:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.790+0000 7f8ed4ff9640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ec406b840 0x7f8ec406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.790+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8ec808a5d0 con 0x7f8ed8071b60 2024-02-19T15:02:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.790+0000 7f8ed77fe640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ec406b840 0x7f8ec406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:11.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.791+0000 7f8ed77fe640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ec406b840 0x7f8ec406dd00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc02b8f0 tx=0x7f8ecc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:11.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:11.799+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8ec8054e00 con 0x7f8ed8071b60 2024-02-19T15:02:12.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.006+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ed81ec360 con 0x7f8ed8071b60 2024-02-19T15:02:12.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.008+0000 7f8ed4ff9640 1 -- 172.21.15.78:0/3479722317 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f8ec8017d60 con 0x7f8ed8071b60 2024-02-19T15:02:12.011 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:12.011 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ec406b840 msgr2=0x7f8ec406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ec406b840 0x7f8ec406dd00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc02b8f0 tx=0x7f8ecc03c040 comp rx=0 tx=0).stop 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 msgr2=0x7f8ed814ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed814ed00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e930 tx=0x7f8ec800edf0 comp rx=0 tx=0).stop 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 shutdown_connections 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ec406b840 0x7f8ec406dd00 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80aca90 0x7f8ed814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed8145b00 0x7f8ed814c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 --2- 172.21.15.78:0/3479722317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ed8071b60 0x7f8ed814ed00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:12.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 >> 172.21.15.78:0/3479722317 conn(0x7f8ed80c7800 msgr2=0x7f8ed8149040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:12.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 shutdown_connections 2024-02-19T15:02:12.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:12.009+0000 7f8ede4ff640 1 -- 172.21.15.78:0/3479722317 wait complete. 2024-02-19T15:02:12.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:12.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:13.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.534+0000 7f0995793640 1 Processor -- start 2024-02-19T15:02:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.534+0000 7f0995793640 1 -- start start 2024-02-19T15:02:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0995793640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f099006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0995793640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09901551f0 con 0x7f0990152700 2024-02-19T15:02:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0995793640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0990155490 con 0x7f099006d910 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0995793640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0990155770 con 0x7f09900c8e00 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f098effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0990152700 0x7f09900c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54256/0 (socket says 172.21.15.78:54256) 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f098effd640 1 -- 172.21.15.78:0/2260297250 learned_addr learned my addr 172.21.15.78:0/2260297250 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0987fff640 1 --2- 172.21.15.78:0/2260297250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f099006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856535702 0 0) 0x7f09901551f0 con 0x7f0990152700 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f097c003520 con 0x7f0990152700 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948067711 0 0) 0x7f0990155490 con 0x7f099006d910 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09901551f0 con 0x7f099006d910 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0987fff640 1 -- 172.21.15.78:0/2260297250 >> v1:172.21.15.150:6789/0 conn(0x7f099006d910 legacy=0x7f099015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0987fff640 1 -- 172.21.15.78:0/2260297250 >> v1:172.21.15.78:6789/0 conn(0x7f0990152700 legacy=0x7f09900c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.535+0000 7f0987fff640 1 -- 172.21.15.78:0/2260297250 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990155a50 con 0x7f09900c8e00 2024-02-19T15:02:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.536+0000 7f0987fff640 1 --2- 172.21.15.78:0/2260297250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f099006c2a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f09780097d0 tx=0x7f097802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13bf14e8fcef9c42 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.536+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978031800 con 0x7f09900c8e00 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.536+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0978031960 con 0x7f09900c8e00 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.536+0000 7f098e7fc640 1 -- 172.21.15.78:0/2260297250 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978031cd0 con 0x7f09900c8e00 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 -- 172.21.15.78:0/2260297250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 msgr2=0x7f099006c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 --2- 172.21.15.78:0/2260297250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f099006c2a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f09780097d0 tx=0x7f097802b8a0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 -- 172.21.15.78:0/2260297250 shutdown_connections 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 --2- 172.21.15.78:0/2260297250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f099006c2a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 -- 172.21.15.78:0/2260297250 >> 172.21.15.78:0/2260297250 conn(0x7f09900cac90 msgr2=0x7f09900cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 -- 172.21.15.78:0/2260297250 shutdown_connections 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.537+0000 7f0995793640 1 -- 172.21.15.78:0/2260297250 wait complete. 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.538+0000 7f0995793640 1 Processor -- start 2024-02-19T15:02:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.538+0000 7f0995793640 1 -- start start 2024-02-19T15:02:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 0x7f0990154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f09900c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0990152700 0x7f09900c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0990148940 con 0x7f099006d910 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09901ef180 con 0x7f0990152700 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09901eecf0 con 0x7f09900c8e00 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 0x7f0990154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 0x7f0990154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40428/0 (socket says 172.21.15.78:40428) 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 -- 172.21.15.78:0/1188761731 learned_addr learned my addr 172.21.15.78:0/1188761731 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0987fff640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f09900c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098f7fe640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0990152700 0x7f09900c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 -- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 msgr2=0x7f09900c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f09900c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 -- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0990152700 msgr2=0x7f09900c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0990152700 0x7f09900c41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 -- 172.21.15.78:0/1188761731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0978009400 con 0x7f099006d910 2024-02-19T15:02:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f098effd640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 0x7f0990154210 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f098000c440 tx=0x7f098000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980005ce0 con 0x7f099006d910 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0980005e40 con 0x7f099006d910 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980007270 con 0x7f099006d910 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.539+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09900c4710 con 0x7f099006d910 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.540+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09901ee6a0 con 0x7f099006d910 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.541+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0990074990 con 0x7f099006d910 2024-02-19T15:02:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.541+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f09800073d0 con 0x7f099006d910 2024-02-19T15:02:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.541+0000 7f09877fe640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f097006b840 0x7f097006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.541+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f09800897d0 con 0x7f099006d910 2024-02-19T15:02:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.541+0000 7f0987fff640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f097006b840 0x7f097006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.542+0000 7f0987fff640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f097006b840 0x7f097006dd00 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0978036000 tx=0x7f097803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.546+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f09800540a0 con 0x7f099006d910 2024-02-19T15:02:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.755+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09900c1600 con 0x7f099006d910 2024-02-19T15:02:13.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.756+0000 7f09877fe640 1 -- 172.21.15.78:0/1188761731 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f0980017d60 con 0x7f099006d910 2024-02-19T15:02:13.760 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:13.760 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:13.760 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f097006b840 msgr2=0x7f097006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f097006b840 0x7f097006dd00 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0978036000 tx=0x7f097803c040 comp rx=0 tx=0).stop 2024-02-19T15:02:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 msgr2=0x7f0990154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 0x7f0990154210 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f098000c440 tx=0x7f098000c900 comp rx=0 tx=0).stop 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 shutdown_connections 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f097006b840 0x7f097006dd00 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0990152700 0x7f09900c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09900c8e00 0x7f09900c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 --2- 172.21.15.78:0/1188761731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f099006d910 0x7f0990154210 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 >> 172.21.15.78:0/1188761731 conn(0x7f09900cac90 msgr2=0x7f09901509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 shutdown_connections 2024-02-19T15:02:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:13.758+0000 7f0995793640 1 -- 172.21.15.78:0/1188761731 wait complete. 2024-02-19T15:02:14.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:14.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:15.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:15.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.281+0000 7f2a04c88640 1 Processor -- start 2024-02-19T15:02:15.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.281+0000 7f2a04c88640 1 -- start start 2024-02-19T15:02:15.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f2a04c88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a000c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f2a04c88640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2a000c19c0 0x7f2a00155070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f2a04c88640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2a00072180 0x7f2a001555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f2a04c88640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a000734c0 con 0x7f2a000c19c0 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f2a04c88640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a00155af0 con 0x7f2a00072180 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f2a04c88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00072e80 con 0x7f2a001528d0 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.282+0000 7f29fdd9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2a000c19c0 0x7f2a00155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2a000c19c0 0x7f2a00155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40442/0 (socket says 172.21.15.78:40442) 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 -- 172.21.15.78:0/1191007718 learned_addr learned my addr 172.21.15.78:0/1191007718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fed9d640 1 --2- 172.21.15.78:0/1191007718 >> v2:172.21.15.150:3300/0 conn(0x7f2a00072180 0x7f2a001555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fe59c640 1 --2- 172.21.15.78:0/1191007718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a000c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 -- 172.21.15.78:0/1191007718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 msgr2=0x7f2a000c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 --2- 172.21.15.78:0/1191007718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a000c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 -- 172.21.15.78:0/1191007718 >> v2:172.21.15.150:3300/0 conn(0x7f2a00072180 msgr2=0x7f2a001555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:15.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 --2- 172.21.15.78:0/1191007718 >> v2:172.21.15.150:3300/0 conn(0x7f2a00072180 0x7f2a001555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.283+0000 7f29fdd9b640 1 -- 172.21.15.78:0/1191007718 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0006e990 con 0x7f2a000c19c0 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f29fdd9b640 1 --2- 172.21.15.78:0/1191007718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a00155070 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f2a000c1230 tx=0x7f29e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311cc34cea4190ef server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f29fd59a640 1 -- 172.21.15.78:0/1191007718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e80317e0 con 0x7f2a000c19c0 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f29fd59a640 1 -- 172.21.15.78:0/1191007718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e8031940 con 0x7f2a000c19c0 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f29fd59a640 1 -- 172.21.15.78:0/1191007718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8031cd0 con 0x7f2a000c19c0 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 -- 172.21.15.78:0/1191007718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 msgr2=0x7f2a00155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 --2- 172.21.15.78:0/1191007718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a00155070 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f2a000c1230 tx=0x7f29e802b880 comp rx=0 tx=0).stop 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 -- 172.21.15.78:0/1191007718 shutdown_connections 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 --2- 172.21.15.78:0/1191007718 >> v2:172.21.15.150:3300/0 conn(0x7f2a00072180 0x7f2a001555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 --2- 172.21.15.78:0/1191007718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a00155070 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 --2- 172.21.15.78:0/1191007718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a000c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 -- 172.21.15.78:0/1191007718 >> 172.21.15.78:0/1191007718 conn(0x7f2a000c7590 msgr2=0x7f2a000c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:15.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 -- 172.21.15.78:0/1191007718 shutdown_connections 2024-02-19T15:02:15.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.284+0000 7f2a04c88640 1 -- 172.21.15.78:0/1191007718 wait complete. 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 Processor -- start 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 -- start start 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00072180 0x7f2a00150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a001ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a001eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000ab630 con 0x7f2a000c19c0 2024-02-19T15:02:15.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001ee400 con 0x7f2a00072180 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.286+0000 7f2a04c88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000ac450 con 0x7f2a001528d0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a001ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a001ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40454/0 (socket says 172.21.15.78:40454) 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 -- 172.21.15.78:0/396216599 learned_addr learned my addr 172.21.15.78:0/396216599 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fe59c640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00072180 0x7f2a00150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fed9d640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a001eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 -- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 msgr2=0x7f2a001eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a001eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 -- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00072180 msgr2=0x7f2a00150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00072180 0x7f2a00150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 -- 172.21.15.78:0/396216599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80094d0 con 0x7f2a000c19c0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29fdd9b640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a001ee890 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f29e8000c00 tx=0x7f29e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8032670 con 0x7f2a000c19c0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e80327d0 con 0x7f2a000c19c0 2024-02-19T15:02:15.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a001ef310 con 0x7f2a000c19c0 2024-02-19T15:02:15.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8032b60 con 0x7f2a000c19c0 2024-02-19T15:02:15.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.287+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a000ad1f0 con 0x7f2a000c19c0 2024-02-19T15:02:15.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.288+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a001ee570 con 0x7f2a000c19c0 2024-02-19T15:02:15.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.289+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f29e8032cc0 con 0x7f2a000c19c0 2024-02-19T15:02:15.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.289+0000 7f29f6ffd640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29e006b810 0x7f29e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.289+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f29e80ac5f0 con 0x7f2a000c19c0 2024-02-19T15:02:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.289+0000 7f29fe59c640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29e006b810 0x7f29e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.290+0000 7f29fe59c640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29e006b810 0x7f29e006dcd0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f29f8011340 tx=0x7f29f800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:15.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.294+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29e8076fe0 con 0x7f2a000c19c0 2024-02-19T15:02:15.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.499+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a00074990 con 0x7f2a000c19c0 2024-02-19T15:02:15.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.499+0000 7f29f6ffd640 1 -- 172.21.15.78:0/396216599 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f29e807a6a0 con 0x7f2a000c19c0 2024-02-19T15:02:15.503 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:15.503 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:15.504 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:15.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29e006b810 msgr2=0x7f29e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:15.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29e006b810 0x7f29e006dcd0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f29f8011340 tx=0x7f29f800b040 comp rx=0 tx=0).stop 2024-02-19T15:02:15.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 msgr2=0x7f2a001ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:15.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a001ee890 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f29e8000c00 tx=0x7f29e80046c0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 shutdown_connections 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f29e006b810 0x7f29e006dcd0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a000c19c0 0x7f2a001ee890 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2a001528d0 0x7f2a001eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 --2- 172.21.15.78:0/396216599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a00072180 0x7f2a00150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 >> 172.21.15.78:0/396216599 conn(0x7f2a000c7590 msgr2=0x7f2a000c2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 shutdown_connections 2024-02-19T15:02:15.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:15.501+0000 7f2a04c88640 1 -- 172.21.15.78:0/396216599 wait complete. 2024-02-19T15:02:15.919 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:15.919 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:16.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:17.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.029+0000 7f051ff3a640 1 Processor -- start 2024-02-19T15:02:17.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051ff3a640 1 -- start start 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051ff3a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f05180add70 0x7f05180c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051ff3a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180ae670 con 0x7f05180c2f60 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051ff3a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180716b0 con 0x7f05180705b0 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051ff3a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05180c33d0 con 0x7f05180add70 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051d4d5640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f05180c2f60 0x7f051806eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:54266/0 (socket says 172.21.15.78:54266) 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.030+0000 7f051d4d5640 1 -- 172.21.15.78:0/374629022 learned_addr learned my addr 172.21.15.78:0/374629022 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051dcd6640 1 --2- 172.21.15.78:0/374629022 >> v2:172.21.15.150:3300/0 conn(0x7f05180add70 0x7f05180c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589133236 0 0) 0x7f05180ae670 con 0x7f05180c2f60 2024-02-19T15:02:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504003520 con 0x7f05180c2f60 2024-02-19T15:02:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998504863 0 0) 0x7f05180716b0 con 0x7f05180705b0 2024-02-19T15:02:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05180ae670 con 0x7f05180705b0 2024-02-19T15:02:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051dcd6640 1 -- 172.21.15.78:0/374629022 >> v1:172.21.15.150:6789/0 conn(0x7f05180705b0 legacy=0x7f05180709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051dcd6640 1 -- 172.21.15.78:0/374629022 >> v1:172.21.15.78:6789/0 conn(0x7f05180c2f60 legacy=0x7f051806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.031+0000 7f051dcd6640 1 -- 172.21.15.78:0/374629022 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518153780 con 0x7f05180add70 2024-02-19T15:02:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.032+0000 7f051dcd6640 1 --2- 172.21.15.78:0/374629022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05180c4980 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0514009d90 tx=0x7f051402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a8e65108ae20bc1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.032+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051402cc60 con 0x7f05180add70 2024-02-19T15:02:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.032+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051402cdc0 con 0x7f05180add70 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.032+0000 7f051ccd4640 1 -- 172.21.15.78:0/374629022 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514034660 con 0x7f05180add70 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.032+0000 7f051ff3a640 1 -- 172.21.15.78:0/374629022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 msgr2=0x7f05180c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.032+0000 7f051ff3a640 1 --2- 172.21.15.78:0/374629022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05180c4980 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0514009d90 tx=0x7f051402beb0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.033+0000 7f051ff3a640 1 -- 172.21.15.78:0/374629022 shutdown_connections 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.033+0000 7f051ff3a640 1 --2- 172.21.15.78:0/374629022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05180c4980 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.033+0000 7f051ff3a640 1 -- 172.21.15.78:0/374629022 >> 172.21.15.78:0/374629022 conn(0x7f05180c85e0 msgr2=0x7f05180c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.033+0000 7f051ff3a640 1 -- 172.21.15.78:0/374629022 shutdown_connections 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.033+0000 7f051ff3a640 1 -- 172.21.15.78:0/374629022 wait complete. 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 Processor -- start 2024-02-19T15:02:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 -- start start 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05180705b0 0x7f05181466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05181ec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 0x7f05181ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518146280 con 0x7f05180c2f60 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518145690 con 0x7f05180add70 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.034+0000 7f051ff3a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0518147530 con 0x7f05180705b0 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 0x7f05181ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 0x7f05181ec5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40468/0 (socket says 172.21.15.78:40468) 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 -- 172.21.15.78:0/1951545828 learned_addr learned my addr 172.21.15.78:0/1951545828 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051dcd6640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05180705b0 0x7f05181466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051d4d5640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05181ec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 -- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05180705b0 msgr2=0x7f05181466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05180705b0 0x7f05181466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 -- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 msgr2=0x7f05181ec0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05181ec0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 -- 172.21.15.78:0/1951545828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514009a70 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051e4d7640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 0x7f05181ec5e0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0510001eb0 tx=0x7f051000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100052d0 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510005430 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100037f0 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05181458e0 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.035+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518145ea0 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.036+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518074990 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.037+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0510003950 con 0x7f05180c2f60 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.037+0000 7f050e7fc640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f04fc06b810 0x7f04fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:17.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.037+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0510089820 con 0x7f05180c2f60 2024-02-19T15:02:17.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.037+0000 7f051dcd6640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f04fc06b810 0x7f04fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:17.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.038+0000 7f051dcd6640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f04fc06b810 0x7f04fc06dcd0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0514036000 tx=0x7f0514030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:17.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.042+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05100540f0 con 0x7f05180c2f60 2024-02-19T15:02:17.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.247+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0518071490 con 0x7f05180c2f60 2024-02-19T15:02:17.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.248+0000 7f050e7fc640 1 -- 172.21.15.78:0/1951545828 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f0510017d60 con 0x7f05180c2f60 2024-02-19T15:02:17.252 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:17.252 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:17.252 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:17.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f04fc06b810 msgr2=0x7f04fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f04fc06b810 0x7f04fc06dcd0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0514036000 tx=0x7f0514030000 comp rx=0 tx=0).stop 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 msgr2=0x7f05181ec5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 0x7f05181ec5e0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0510001eb0 tx=0x7f051000c820 comp rx=0 tx=0).stop 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 shutdown_connections 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f04fc06b810 0x7f04fc06dcd0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05180c2f60 0x7f05181ec5e0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05180add70 0x7f05181ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.249+0000 7f051ff3a640 1 --2- 172.21.15.78:0/1951545828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05180705b0 0x7f05181466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.250+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 >> 172.21.15.78:0/1951545828 conn(0x7f05180c85e0 msgr2=0x7f0518152cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.250+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 shutdown_connections 2024-02-19T15:02:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:17.250+0000 7f051ff3a640 1 -- 172.21.15.78:0/1951545828 wait complete. 2024-02-19T15:02:17.666 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:17.667 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:18.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:18.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.775+0000 7fa52413e640 1 Processor -- start 2024-02-19T15:02:18.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.776+0000 7fa52413e640 1 -- start start 2024-02-19T15:02:18.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.776+0000 7fa52413e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa51c0c25e0 0x7fa51c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.776+0000 7fa52413e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa51c0c29c0 0x7fa51c0c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.776+0000 7fa52413e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c072250 con 0x7fa51c0c49c0 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.776+0000 7fa52413e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51c152c90 con 0x7fa51c0c29c0 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.776+0000 7fa52413e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51c072480 con 0x7fa51c0c25e0 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa51c0c29c0 0x7fa51c0c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa51c0c29c0 0x7fa51c0c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49834/0 (socket says 172.21.15.78:49834) 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 -- 172.21.15.78:0/2227593204 learned_addr learned my addr 172.21.15.78:0/2227593204 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa521eda640 1 --2- 172.21.15.78:0/2227593204 >> v2:172.21.15.150:3300/0 conn(0x7fa51c0c25e0 0x7fa51c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 -- 172.21.15.78:0/2227593204 >> v2:172.21.15.150:3300/0 conn(0x7fa51c0c25e0 msgr2=0x7fa51c1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 --2- 172.21.15.78:0/2227593204 >> v2:172.21.15.150:3300/0 conn(0x7fa51c0c25e0 0x7fa51c1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 -- 172.21.15.78:0/2227593204 >> v1:172.21.15.150:6789/0 conn(0x7fa51c0c49c0 legacy=0x7fa51c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:18.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.777+0000 7fa5216d9640 1 -- 172.21.15.78:0/2227593204 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0c3eb0 con 0x7fa51c0c29c0 2024-02-19T15:02:18.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa5216d9640 1 --2- 172.21.15.78:0/2227593204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c0c29c0 0x7fa51c0c4480 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fa51c06f120 tx=0x7fa51002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89236f7c464d060e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa520ed8640 1 -- 172.21.15.78:0/2227593204 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510031870 con 0x7fa51c0c29c0 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa520ed8640 1 -- 172.21.15.78:0/2227593204 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100319d0 con 0x7fa51c0c29c0 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa520ed8640 1 -- 172.21.15.78:0/2227593204 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510031d60 con 0x7fa51c0c29c0 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 -- 172.21.15.78:0/2227593204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c0c29c0 msgr2=0x7fa51c0c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 --2- 172.21.15.78:0/2227593204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c0c29c0 0x7fa51c0c4480 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fa51c06f120 tx=0x7fa51002b910 comp rx=0 tx=0).stop 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 -- 172.21.15.78:0/2227593204 shutdown_connections 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 --2- 172.21.15.78:0/2227593204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c0c29c0 0x7fa51c0c4480 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 --2- 172.21.15.78:0/2227593204 >> v2:172.21.15.150:3300/0 conn(0x7fa51c0c25e0 0x7fa51c1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 -- 172.21.15.78:0/2227593204 >> 172.21.15.78:0/2227593204 conn(0x7fa51c06b270 msgr2=0x7fa51c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 -- 172.21.15.78:0/2227593204 shutdown_connections 2024-02-19T15:02:18.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.778+0000 7fa52413e640 1 -- 172.21.15.78:0/2227593204 wait complete. 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.779+0000 7fa52413e640 1 Processor -- start 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.779+0000 7fa52413e640 1 -- start start 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa52413e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa51c0c25e0 0x7fa51c13dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa52413e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa51c0c29c0 0x7fa51c154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa52413e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 0x7fa51c1487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa52413e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c154090 con 0x7fa51c147080 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa52413e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c148310 con 0x7fa51c0c25e0 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa52413e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c153860 con 0x7fa51c0c29c0 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa5226db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 0x7fa51c1487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa5226db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 0x7fa51c1487d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49850/0 (socket says 172.21.15.78:49850) 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa5226db640 1 -- 172.21.15.78:0/727243457 learned_addr learned my addr 172.21.15.78:0/727243457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:18.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa521eda640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa51c0c25e0 0x7fa51c13dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.780+0000 7fa5216d9640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa51c0c29c0 0x7fa51c154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa5226db640 1 -- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa51c0c29c0 msgr2=0x7fa51c154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa5226db640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa51c0c29c0 0x7fa51c154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa5226db640 1 -- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa51c0c25e0 msgr2=0x7fa51c13dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa5226db640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa51c0c25e0 0x7fa51c13dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa5226db640 1 -- 172.21.15.78:0/727243457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5100094d0 con 0x7fa51c147080 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa5226db640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 0x7fa51c1487d0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00c3f0 tx=0x7fa50c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c0052d0 con 0x7fa51c147080 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c005430 con 0x7fa51c147080 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c153ad0 con 0x7fa51c147080 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c003830 con 0x7fa51c147080 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.781+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c13e110 con 0x7fa51c147080 2024-02-19T15:02:18.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.782+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c1484f0 con 0x7fa51c147080 2024-02-19T15:02:18.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.782+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa50c003990 con 0x7fa51c147080 2024-02-19T15:02:18.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.783+0000 7fa51a7fc640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa50806b810 0x7fa50806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:18.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.783+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa50c089810 con 0x7fa51c147080 2024-02-19T15:02:18.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.783+0000 7fa521eda640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa50806b810 0x7fa50806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:18.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.783+0000 7fa521eda640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa50806b810 0x7fa50806dcd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa51c146270 tx=0x7fa51400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:18.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.787+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa51c1484f0 con 0x7fa51c147080 2024-02-19T15:02:18.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.995+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa51c1484f0 con 0x7fa51c147080 2024-02-19T15:02:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.996+0000 7fa51a7fc640 1 -- 172.21.15.78:0/727243457 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7fa50c017d60 con 0x7fa51c147080 2024-02-19T15:02:18.999 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:18.999 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:18.999 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:19.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa50806b810 msgr2=0x7fa50806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa50806b810 0x7fa50806dcd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa51c146270 tx=0x7fa51400b040 comp rx=0 tx=0).stop 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 msgr2=0x7fa51c1487d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 0x7fa51c1487d0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00c3f0 tx=0x7fa50c00c7f0 comp rx=0 tx=0).stop 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 shutdown_connections 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa50806b810 0x7fa50806dcd0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa51c147080 0x7fa51c1487d0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa51c0c29c0 0x7fa51c154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 --2- 172.21.15.78:0/727243457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa51c0c25e0 0x7fa51c13dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 >> 172.21.15.78:0/727243457 conn(0x7fa51c06b270 msgr2=0x7fa51c0c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 shutdown_connections 2024-02-19T15:02:19.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:18.998+0000 7fa52413e640 1 -- 172.21.15.78:0/727243457 wait complete. 2024-02-19T15:02:19.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:19.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:20.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:20.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.522+0000 7f1f5fcf4640 1 Processor -- start 2024-02-19T15:02:20.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.522+0000 7f1f5fcf4640 1 -- start start 2024-02-19T15:02:20.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5fcf4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f580ac390 0x7f1f580ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:20.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5fcf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580bdf20 0x7f1f580c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:20.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5fcf4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580c8de0 con 0x7f1f58158530 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5fcf4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f58155a60 con 0x7f1f580ac390 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5fcf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58155d40 con 0x7f1f580bdf20 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5da90640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f580ac390 0x7f1f580ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5d28f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580bdf20 0x7f1f580c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5e291640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1f58158530 0x7f1f5806c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:59108/0 (socket says 172.21.15.78:59108) 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5e291640 1 -- 172.21.15.78:0/473042094 learned_addr learned my addr 172.21.15.78:0/473042094 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5ca8e640 1 -- 172.21.15.78:0/473042094 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947973927 0 0) 0x7f1f580c8de0 con 0x7f1f58158530 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5ca8e640 1 -- 172.21.15.78:0/473042094 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f44003520 con 0x7f1f58158530 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5d28f640 1 -- 172.21.15.78:0/473042094 >> v2:172.21.15.150:3300/0 conn(0x7f1f580ac390 msgr2=0x7f1f580ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5d28f640 1 --2- 172.21.15.78:0/473042094 >> v2:172.21.15.150:3300/0 conn(0x7f1f580ac390 0x7f1f580ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5d28f640 1 -- 172.21.15.78:0/473042094 >> v1:172.21.15.150:6789/0 conn(0x7f1f58158530 legacy=0x7f1f5806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5d28f640 1 -- 172.21.15.78:0/473042094 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580c90a0 con 0x7f1f580bdf20 2024-02-19T15:02:20.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.523+0000 7f1f5da90640 1 --2- 172.21.15.78:0/473042094 >> v2:172.21.15.150:3300/0 conn(0x7f1f580ac390 0x7f1f580ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.524+0000 7f1f5d28f640 1 --2- 172.21.15.78:0/473042094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580bdf20 0x7f1f580c55f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1f480097a0 tx=0x7f1f4802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=544326fd385e5acf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.524+0000 7f1f5ca8e640 1 -- 172.21.15.78:0/473042094 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4802f050 con 0x7f1f580bdf20 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.524+0000 7f1f5ca8e640 1 -- 172.21.15.78:0/473042094 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f48031700 con 0x7f1f580bdf20 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.524+0000 7f1f5ca8e640 1 -- 172.21.15.78:0/473042094 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f48031a90 con 0x7f1f580bdf20 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/473042094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580bdf20 msgr2=0x7f1f580c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/473042094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580bdf20 0x7f1f580c55f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1f480097a0 tx=0x7f1f4802b7a0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/473042094 shutdown_connections 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/473042094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580bdf20 0x7f1f580c55f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/473042094 >> v2:172.21.15.150:3300/0 conn(0x7f1f580ac390 0x7f1f580ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/473042094 >> 172.21.15.78:0/473042094 conn(0x7f1f580ca890 msgr2=0x7f1f580caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:20.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/473042094 shutdown_connections 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.525+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/473042094 wait complete. 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.526+0000 7f1f5fcf4640 1 Processor -- start 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.526+0000 7f1f5fcf4640 1 -- start start 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5fcf4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580ac390 0x7f1f5814f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5fcf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580bdf20 0x7f1f5814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5fcf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 0x7f1f581ea370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5fcf4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f580c2750 con 0x7f1f58158530 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5fcf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f580c4600 con 0x7f1f580bdf20 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5fcf4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f580c1ca0 con 0x7f1f580ac390 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 0x7f1f581ea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5da90640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580ac390 0x7f1f5814f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 0x7f1f581ea370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49864/0 (socket says 172.21.15.78:49864) 2024-02-19T15:02:20.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 -- 172.21.15.78:0/2202918804 learned_addr learned my addr 172.21.15.78:0/2202918804 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5d28f640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580bdf20 0x7f1f5814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 -- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580ac390 msgr2=0x7f1f5814f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580ac390 0x7f1f5814f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 -- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580bdf20 msgr2=0x7f1f5814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580bdf20 0x7f1f5814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 -- 172.21.15.78:0/2202918804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f54009400 con 0x7f1f58158530 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5da90640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580ac390 0x7f1f5814f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.527+0000 7f1f5e291640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 0x7f1f581ea370 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000caa0 tx=0x7f1f5000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.528+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000ac00 con 0x7f1f58158530 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.528+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5000ad60 con 0x7f1f58158530 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.528+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f48009400 con 0x7f1f58158530 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.528+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50012620 con 0x7f1f58158530 2024-02-19T15:02:20.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.528+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f581eb270 con 0x7f1f58158530 2024-02-19T15:02:20.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.528+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f580c1e10 con 0x7f1f58158530 2024-02-19T15:02:20.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.529+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1f50007070 con 0x7f1f58158530 2024-02-19T15:02:20.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.529+0000 7f1f4e7fc640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f3c06b840 0x7f1f3c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:20.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.530+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1f5008a2d0 con 0x7f1f58158530 2024-02-19T15:02:20.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.530+0000 7f1f5da90640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f3c06b840 0x7f1f3c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:20.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.530+0000 7f1f5da90640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f3c06b840 0x7f1f3c06dd00 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1f54011380 tx=0x7f1f54009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:20.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.534+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1f50054ba0 con 0x7f1f58158530 2024-02-19T15:02:20.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.742+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f58074990 con 0x7f1f58158530 2024-02-19T15:02:20.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.743+0000 7f1f4e7fc640 1 -- 172.21.15.78:0/2202918804 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f1f50017d60 con 0x7f1f58158530 2024-02-19T15:02:20.746 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:20.746 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:20.746 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f3c06b840 msgr2=0x7f1f3c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f3c06b840 0x7f1f3c06dd00 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1f54011380 tx=0x7f1f54009190 comp rx=0 tx=0).stop 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 msgr2=0x7f1f581ea370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 0x7f1f581ea370 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000caa0 tx=0x7f1f5000cf60 comp rx=0 tx=0).stop 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 shutdown_connections 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1f3c06b840 0x7f1f3c06dd00 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f58158530 0x7f1f581ea370 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580bdf20 0x7f1f5814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 --2- 172.21.15.78:0/2202918804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580ac390 0x7f1f5814f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:20.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 >> 172.21.15.78:0/2202918804 conn(0x7f1f580ca890 msgr2=0x7f1f580cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:20.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 shutdown_connections 2024-02-19T15:02:20.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:20.745+0000 7f1f5fcf4640 1 -- 172.21.15.78:0/2202918804 wait complete. 2024-02-19T15:02:21.160 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:21.161 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:22.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.276+0000 7fa8a152c640 1 Processor -- start 2024-02-19T15:02:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.276+0000 7fa8a152c640 1 -- start start 2024-02-19T15:02:22.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa8a152c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa89c06ed20 0x7fa89c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:22.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa8a152c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c150820 con 0x7fa89c150440 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa8a152c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c1537d0 con 0x7fa89c06d750 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa8a152c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89c0c1b60 con 0x7fa89c06ed20 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa89affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa89c150440 0x7fa89c0c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42624/0 (socket says 172.21.15.78:42624) 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa89affd640 1 -- 172.21.15.78:0/2338236876 learned_addr learned my addr 172.21.15.78:0/2338236876 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa893fff640 1 --2- 172.21.15.78:0/2338236876 >> v2:172.21.15.150:3300/0 conn(0x7fa89c06ed20 0x7fa89c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660441545 0 0) 0x7fa89c150820 con 0x7fa89c150440 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888003520 con 0x7fa89c150440 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351267179 0 0) 0x7fa89c1537d0 con 0x7fa89c06d750 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89c150820 con 0x7fa89c06d750 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa893fff640 1 -- 172.21.15.78:0/2338236876 >> v1:172.21.15.150:6789/0 conn(0x7fa89c06d750 legacy=0x7fa89c15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa893fff640 1 -- 172.21.15.78:0/2338236876 >> v1:172.21.15.78:6789/0 conn(0x7fa89c150440 legacy=0x7fa89c0c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.277+0000 7fa893fff640 1 -- 172.21.15.78:0/2338236876 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c153d10 con 0x7fa89c06ed20 2024-02-19T15:02:22.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.278+0000 7fa893fff640 1 --2- 172.21.15.78:0/2338236876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c06d210 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa89c153960 tx=0x7fa88402b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aece47a1164ad809 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:22.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.278+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884030890 con 0x7fa89c06ed20 2024-02-19T15:02:22.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.278+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8840309f0 con 0x7fa89c06ed20 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.278+0000 7fa89a7fc640 1 -- 172.21.15.78:0/2338236876 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884030d80 con 0x7fa89c06ed20 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 -- 172.21.15.78:0/2338236876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 msgr2=0x7fa89c06d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 --2- 172.21.15.78:0/2338236876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c06d210 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa89c153960 tx=0x7fa88402b9d0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 -- 172.21.15.78:0/2338236876 shutdown_connections 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 --2- 172.21.15.78:0/2338236876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c06d210 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 -- 172.21.15.78:0/2338236876 >> 172.21.15.78:0/2338236876 conn(0x7fa89c06b270 msgr2=0x7fa89c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 -- 172.21.15.78:0/2338236876 shutdown_connections 2024-02-19T15:02:22.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.279+0000 7fa8a152c640 1 -- 172.21.15.78:0/2338236876 wait complete. 2024-02-19T15:02:22.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.280+0000 7fa8a152c640 1 Processor -- start 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.280+0000 7fa8a152c640 1 -- start start 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa8a152c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa89c06d750 0x7fa89c0bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa8a152c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c0c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa8a152c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 0x7fa89c1ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa8a152c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c154d80 con 0x7fa89c150440 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa8a152c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c154a30 con 0x7fa89c06ed20 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa8a152c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1edac0 con 0x7fa89c06d750 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 0x7fa89c1ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 0x7fa89c1ee1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49878/0 (socket says 172.21.15.78:49878) 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 -- 172.21.15.78:0/3177875177 learned_addr learned my addr 172.21.15.78:0/3177875177 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa893fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c0c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:22.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89affd640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa89c06d750 0x7fa89c0bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 -- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa89c06d750 msgr2=0x7fa89c0bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa89c06d750 0x7fa89c0bffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 -- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 msgr2=0x7fa89c0c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c0c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.281+0000 7fa89b7fe640 1 -- 172.21.15.78:0/3177875177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884009400 con 0x7fa89c150440 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.282+0000 7fa89b7fe640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 0x7fa89c1ee1f0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c790 tx=0x7fa89400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.282+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89400cea0 con 0x7fa89c150440 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.282+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894003b70 con 0x7fa89c150440 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.282+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c1e81f0 con 0x7fa89c150440 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.282+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89400a7f0 con 0x7fa89c150440 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.282+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c154240 con 0x7fa89c150440 2024-02-19T15:02:22.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.283+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c074990 con 0x7fa89c150440 2024-02-19T15:02:22.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.283+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa89400a950 con 0x7fa89c150440 2024-02-19T15:02:22.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.284+0000 7fa8937fe640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa87c06b810 0x7fa87c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:22.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.284+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa89408a100 con 0x7fa89c150440 2024-02-19T15:02:22.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.284+0000 7fa89affd640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa87c06b810 0x7fa87c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:22.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.284+0000 7fa89affd640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa87c06b810 0x7fa87c06dcd0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000f60 tx=0x7fa88c007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:22.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.288+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa8940549d0 con 0x7fa89c150440 2024-02-19T15:02:22.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.495+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa89c06eb00 con 0x7fa89c150440 2024-02-19T15:02:22.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.495+0000 7fa8937fe640 1 -- 172.21.15.78:0/3177875177 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7fa894058090 con 0x7fa89c150440 2024-02-19T15:02:22.498 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:22.498 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:22.499 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:22.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa87c06b810 msgr2=0x7fa87c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa87c06b810 0x7fa87c06dcd0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000f60 tx=0x7fa88c007d50 comp rx=0 tx=0).stop 2024-02-19T15:02:22.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 msgr2=0x7fa89c1ee1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:22.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 0x7fa89c1ee1f0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c790 tx=0x7fa89400cc50 comp rx=0 tx=0).stop 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 shutdown_connections 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa87c06b810 0x7fa87c06dcd0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa89c150440 0x7fa89c1ee1f0 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa89c06ed20 0x7fa89c0c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 --2- 172.21.15.78:0/3177875177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa89c06d750 0x7fa89c0bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 >> 172.21.15.78:0/3177875177 conn(0x7fa89c06b270 msgr2=0x7fa89c14e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 shutdown_connections 2024-02-19T15:02:22.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:22.497+0000 7fa8a152c640 1 -- 172.21.15.78:0/3177875177 wait complete. 2024-02-19T15:02:22.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:22.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:23.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.030+0000 7f17212cd640 1 Processor -- start 2024-02-19T15:02:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.030+0000 7f17212cd640 1 -- start start 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f17212cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c0c8e00 0x7f171c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f17212cd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f171c06d910 0x7f171c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f17212cd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171c06cd20 con 0x7f171c152700 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f17212cd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c1551f0 con 0x7f171c06d910 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f17212cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c155490 con 0x7f171c0c8e00 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f171c06d910 0x7f171c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f171c152700 0x7f171c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42632/0 (socket says 172.21.15.78:42632) 2024-02-19T15:02:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171affd640 1 -- 172.21.15.78:0/1766449811 learned_addr learned my addr 172.21.15.78:0/1766449811 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171a7fc640 1 --2- 172.21.15.78:0/1766449811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c0c8e00 0x7f171c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f1719ffb640 1 -- 172.21.15.78:0/1766449811 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488682869 0 0) 0x7f171c06cd20 con 0x7f171c152700 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f1719ffb640 1 -- 172.21.15.78:0/1766449811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704003520 con 0x7f171c152700 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171b7fe640 1 -- 172.21.15.78:0/1766449811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c0c8e00 msgr2=0x7f171c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171b7fe640 1 --2- 172.21.15.78:0/1766449811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c0c8e00 0x7f171c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171b7fe640 1 -- 172.21.15.78:0/1766449811 >> v1:172.21.15.78:6789/0 conn(0x7f171c152700 legacy=0x7f171c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.031+0000 7f171b7fe640 1 -- 172.21.15.78:0/1766449811 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c155770 con 0x7f171c06d910 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.032+0000 7f171b7fe640 1 --2- 172.21.15.78:0/1766449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c06d910 0x7f171c06c7e0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f1714009d90 tx=0x7f171402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd346491809f5c6b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.032+0000 7f1719ffb640 1 -- 172.21.15.78:0/1766449811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171402cbc0 con 0x7f171c06d910 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.032+0000 7f1719ffb640 1 -- 172.21.15.78:0/1766449811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171402cd20 con 0x7f171c06d910 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.032+0000 7f1719ffb640 1 -- 172.21.15.78:0/1766449811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714034660 con 0x7f171c06d910 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 -- 172.21.15.78:0/1766449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c06d910 msgr2=0x7f171c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 --2- 172.21.15.78:0/1766449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c06d910 0x7f171c06c7e0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f1714009d90 tx=0x7f171402be10 comp rx=0 tx=0).stop 2024-02-19T15:02:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 -- 172.21.15.78:0/1766449811 shutdown_connections 2024-02-19T15:02:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 --2- 172.21.15.78:0/1766449811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c06d910 0x7f171c06c7e0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 --2- 172.21.15.78:0/1766449811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c0c8e00 0x7f171c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 -- 172.21.15.78:0/1766449811 >> 172.21.15.78:0/1766449811 conn(0x7f171c0cac90 msgr2=0x7f171c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 -- 172.21.15.78:0/1766449811 shutdown_connections 2024-02-19T15:02:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.033+0000 7f17212cd640 1 -- 172.21.15.78:0/1766449811 wait complete. 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.034+0000 7f17212cd640 1 Processor -- start 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.034+0000 7f17212cd640 1 -- start start 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f17212cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c06d910 0x7f171c1e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f17212cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 0x7f171c0c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f17212cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f171c152700 0x7f171c148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f17212cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c0c42f0 con 0x7f171c0c8e00 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f17212cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c147790 con 0x7f171c152700 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f17212cd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c1ea4a0 con 0x7f171c06d910 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 0x7f171c0c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 0x7f171c0c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49896/0 (socket says 172.21.15.78:49896) 2024-02-19T15:02:24.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 -- 172.21.15.78:0/152041651 learned_addr learned my addr 172.21.15.78:0/152041651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171affd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c06d910 0x7f171c1e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171b7fe640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f171c152700 0x7f171c148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 -- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c06d910 msgr2=0x7f171c1e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c06d910 0x7f171c1e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 -- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f171c152700 msgr2=0x7f171c148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f171c152700 0x7f171c148dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 -- 172.21.15.78:0/152041651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714009a70 con 0x7f171c0c8e00 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.035+0000 7f171a7fc640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 0x7f171c0c1940 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f171000e9e0 tx=0x7f171000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.036+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1710009670 con 0x7f171c0c8e00 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.036+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171000c9d0 con 0x7f171c0c8e00 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.036+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c0c3540 con 0x7f171c0c8e00 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.036+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171000cd80 con 0x7f171c0c8e00 2024-02-19T15:02:24.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.036+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c0c3a50 con 0x7f171c0c8e00 2024-02-19T15:02:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.037+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c0688f0 con 0x7f171c0c8e00 2024-02-19T15:02:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.037+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1710013470 con 0x7f171c0c8e00 2024-02-19T15:02:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.038+0000 7f17037fe640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170806b810 0x7f170806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.038+0000 7f171affd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170806b810 0x7f170806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.038+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f171008a5e0 con 0x7f171c0c8e00 2024-02-19T15:02:24.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.038+0000 7f171affd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170806b810 0x7f170806dcd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f170c005880 tx=0x7f170c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:24.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.042+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1710054e20 con 0x7f171c0c8e00 2024-02-19T15:02:24.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.252+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f171c074990 con 0x7f171c0c8e00 2024-02-19T15:02:24.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.253+0000 7f17037fe640 1 -- 172.21.15.78:0/152041651 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f1710017d60 con 0x7f171c0c8e00 2024-02-19T15:02:24.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:24.256 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:24.256 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:24.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.254+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170806b810 msgr2=0x7f170806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.254+0000 7f17212cd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170806b810 0x7f170806dcd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f170c005880 tx=0x7f170c009040 comp rx=0 tx=0).stop 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.254+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 msgr2=0x7f171c0c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.254+0000 7f17212cd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 0x7f171c0c1940 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f171000e9e0 tx=0x7f171000eea0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 shutdown_connections 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f170806b810 0x7f170806dcd0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f171c0c8e00 0x7f171c0c1940 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f171c152700 0x7f171c148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 --2- 172.21.15.78:0/152041651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f171c06d910 0x7f171c1e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 >> 172.21.15.78:0/152041651 conn(0x7f171c0cac90 msgr2=0x7f171c14bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 shutdown_connections 2024-02-19T15:02:24.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:24.255+0000 7f17212cd640 1 -- 172.21.15.78:0/152041651 wait complete. 2024-02-19T15:02:24.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:24.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:25.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:25.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.781+0000 7ff2612ed640 1 Processor -- start 2024-02-19T15:02:25.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.781+0000 7ff2612ed640 1 -- start start 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff2612ed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff25c152700 0x7ff25c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff2612ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c0c5730 0x7ff25c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff2612ed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c1551f0 con 0x7ff25c0c8e00 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff2612ed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c155490 con 0x7ff25c152700 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff2612ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c155770 con 0x7ff25c0c5730 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff25b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff25c0c8e00 0x7ff25c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42638/0 (socket says 172.21.15.78:42638) 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff25b7fe640 1 -- 172.21.15.78:0/3884734001 learned_addr learned my addr 172.21.15.78:0/3884734001 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff25affd640 1 --2- 172.21.15.78:0/3884734001 >> v2:172.21.15.150:3300/0 conn(0x7ff25c152700 0x7ff25c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:25.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff253fff640 1 --2- 172.21.15.78:0/3884734001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c0c5730 0x7ff25c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff25a7fc640 1 -- 172.21.15.78:0/3884734001 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988182340 0 0) 0x7ff25c1551f0 con 0x7ff25c0c8e00 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff25a7fc640 1 -- 172.21.15.78:0/3884734001 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248003520 con 0x7ff25c0c8e00 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff253fff640 1 -- 172.21.15.78:0/3884734001 >> v2:172.21.15.150:3300/0 conn(0x7ff25c152700 msgr2=0x7ff25c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff253fff640 1 --2- 172.21.15.78:0/3884734001 >> v2:172.21.15.150:3300/0 conn(0x7ff25c152700 0x7ff25c0c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff253fff640 1 -- 172.21.15.78:0/3884734001 >> v1:172.21.15.78:6789/0 conn(0x7ff25c0c8e00 legacy=0x7ff25c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff253fff640 1 -- 172.21.15.78:0/3884734001 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c155a50 con 0x7ff25c0c5730 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.782+0000 7ff25affd640 1 --2- 172.21.15.78:0/3884734001 >> v2:172.21.15.150:3300/0 conn(0x7ff25c152700 0x7ff25c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.783+0000 7ff253fff640 1 --2- 172.21.15.78:0/3884734001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c0c5730 0x7ff25c06c2a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097a0 tx=0x7ff24402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd951d0f1afd2b3b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.783+0000 7ff25a7fc640 1 -- 172.21.15.78:0/3884734001 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244031830 con 0x7ff25c0c5730 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.783+0000 7ff25a7fc640 1 -- 172.21.15.78:0/3884734001 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244031990 con 0x7ff25c0c5730 2024-02-19T15:02:25.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.783+0000 7ff25a7fc640 1 -- 172.21.15.78:0/3884734001 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244031d20 con 0x7ff25c0c5730 2024-02-19T15:02:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 -- 172.21.15.78:0/3884734001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c0c5730 msgr2=0x7ff25c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:25.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 --2- 172.21.15.78:0/3884734001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c0c5730 0x7ff25c06c2a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097a0 tx=0x7ff24402b8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 -- 172.21.15.78:0/3884734001 shutdown_connections 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 --2- 172.21.15.78:0/3884734001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c0c5730 0x7ff25c06c2a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 --2- 172.21.15.78:0/3884734001 >> v2:172.21.15.150:3300/0 conn(0x7ff25c152700 0x7ff25c0c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 -- 172.21.15.78:0/3884734001 >> 172.21.15.78:0/3884734001 conn(0x7ff25c0cac90 msgr2=0x7ff25c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 -- 172.21.15.78:0/3884734001 shutdown_connections 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.784+0000 7ff2612ed640 1 -- 172.21.15.78:0/3884734001 wait complete. 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.785+0000 7ff2612ed640 1 Processor -- start 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.785+0000 7ff2612ed640 1 -- start start 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25c0c5730 0x7ff25c149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 0x7ff25c1e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c152700 0x7ff25c1e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c1e84c0 con 0x7ff25c0c8e00 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c1eac50 con 0x7ff25c0c5730 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c1eaf30 con 0x7ff25c152700 2024-02-19T15:02:25.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 0x7ff25c1e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 0x7ff25c1e7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49908/0 (socket says 172.21.15.78:49908) 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff25affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25c0c5730 0x7ff25c149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 -- 172.21.15.78:0/2530526609 learned_addr learned my addr 172.21.15.78:0/2530526609 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff25b7fe640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c152700 0x7ff25c1e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 -- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c152700 msgr2=0x7ff25c1e7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c152700 0x7ff25c1e7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 -- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25c0c5730 msgr2=0x7ff25c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25c0c5730 0x7ff25c149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 -- 172.21.15.78:0/2530526609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c009400 con 0x7ff25c0c8e00 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff253fff640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 0x7ff25c1e7a40 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7ff2440098c0 tx=0x7ff2440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24403a630 con 0x7ff25c0c8e00 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24403a7b0 con 0x7ff25c0c8e00 2024-02-19T15:02:25.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24403ab60 con 0x7ff25c0c8e00 2024-02-19T15:02:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.786+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff244009400 con 0x7ff25c0c8e00 2024-02-19T15:02:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.787+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c1eb4f0 con 0x7ff25c0c8e00 2024-02-19T15:02:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.788+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c074990 con 0x7ff25c0c8e00 2024-02-19T15:02:25.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.788+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff24403acc0 con 0x7ff25c0c8e00 2024-02-19T15:02:25.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.788+0000 7ff2537fe640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff23c06b810 0x7ff23c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:25.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.788+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff2440abf20 con 0x7ff25c0c8e00 2024-02-19T15:02:25.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.788+0000 7ff25affd640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff23c06b810 0x7ff23c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:25.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.789+0000 7ff25affd640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff23c06b810 0x7ff23c06dcd0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009c30 tx=0x7ff24c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:25.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.793+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff2440767f0 con 0x7ff25c0c8e00 2024-02-19T15:02:26.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.998+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25c0c8770 con 0x7ff25c0c8e00 2024-02-19T15:02:26.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:25.999+0000 7ff2537fe640 1 -- 172.21.15.78:0/2530526609 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7ff244048d60 con 0x7ff25c0c8e00 2024-02-19T15:02:26.002 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:26.002 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:26.002 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff23c06b810 msgr2=0x7ff23c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff23c06b810 0x7ff23c06dcd0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009c30 tx=0x7ff24c009190 comp rx=0 tx=0).stop 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 msgr2=0x7ff25c1e7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 0x7ff25c1e7a40 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7ff2440098c0 tx=0x7ff2440046c0 comp rx=0 tx=0).stop 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 shutdown_connections 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff23c06b810 0x7ff23c06dcd0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:26.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff25c0c8e00 0x7ff25c1e7a40 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:26.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff25c152700 0x7ff25c1e7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:26.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 --2- 172.21.15.78:0/2530526609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25c0c5730 0x7ff25c149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:26.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 >> 172.21.15.78:0/2530526609 conn(0x7ff25c0cac90 msgr2=0x7ff25c1512f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:26.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 shutdown_connections 2024-02-19T15:02:26.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:26.001+0000 7ff2612ed640 1 -- 172.21.15.78:0/2530526609 wait complete. 2024-02-19T15:02:26.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:26.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:27.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:27.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.526+0000 7f0d80701640 1 Processor -- start 2024-02-19T15:02:27.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.526+0000 7f0d80701640 1 -- start start 2024-02-19T15:02:27.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d80701640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d780c4980 0x7f0d780c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:27.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d80701640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d80701640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d780ae670 con 0x7f0d780add70 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d80701640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d780716b0 con 0x7f0d780c4980 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d80701640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78153780 con 0x7f0d780705b0 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d7dc9c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0d780c4980 0x7f0d780c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d7e49d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0d780add70 0x7f0d7806eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47004/0 (socket says 172.21.15.78:47004) 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d7e49d640 1 -- 172.21.15.78:0/4192126839 learned_addr learned my addr 172.21.15.78:0/4192126839 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.527+0000 7f0d7ec9e640 1 --2- 172.21.15.78:0/4192126839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.528+0000 7f0d7dc9c640 1 -- 172.21.15.78:0/4192126839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 msgr2=0x7f0d78070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.528+0000 7f0d7dc9c640 1 --2- 172.21.15.78:0/4192126839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.528+0000 7f0d7dc9c640 1 -- 172.21.15.78:0/4192126839 >> v1:172.21.15.78:6789/0 conn(0x7f0d780add70 legacy=0x7f0d7806eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.528+0000 7f0d7d49b640 1 -- 172.21.15.78:0/4192126839 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873528802 0 0) 0x7f0d780ae670 con 0x7f0d780add70 2024-02-19T15:02:27.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.528+0000 7f0d7dc9c640 1 -- 172.21.15.78:0/4192126839 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78153a60 con 0x7f0d780c4980 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.528+0000 7f0d7dc9c640 1 --2- 172.21.15.78:0/4192126839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d780c2f60 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f0d680097a0 tx=0x7f0d6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d0c5a9c58289e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d7d49b640 1 -- 172.21.15.78:0/4192126839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6802e070 con 0x7f0d780c4980 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d7d49b640 1 -- 172.21.15.78:0/4192126839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d680315a0 con 0x7f0d780c4980 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d7d49b640 1 -- 172.21.15.78:0/4192126839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68031930 con 0x7f0d780c4980 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 -- 172.21.15.78:0/4192126839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 msgr2=0x7f0d780c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 --2- 172.21.15.78:0/4192126839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d780c2f60 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f0d680097a0 tx=0x7f0d6802b840 comp rx=0 tx=0).stop 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 -- 172.21.15.78:0/4192126839 shutdown_connections 2024-02-19T15:02:27.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 --2- 172.21.15.78:0/4192126839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d780c2f60 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 --2- 172.21.15.78:0/4192126839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 -- 172.21.15.78:0/4192126839 >> 172.21.15.78:0/4192126839 conn(0x7f0d780c85e0 msgr2=0x7f0d780c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 -- 172.21.15.78:0/4192126839 shutdown_connections 2024-02-19T15:02:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.529+0000 7f0d80701640 1 -- 172.21.15.78:0/4192126839 wait complete. 2024-02-19T15:02:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.530+0000 7f0d80701640 1 Processor -- start 2024-02-19T15:02:27.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.530+0000 7f0d80701640 1 -- start start 2024-02-19T15:02:27.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d80701640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:27.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d80701640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d780add70 0x7f0d781eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:27.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d80701640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d781ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:27.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d80701640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780bfe30 con 0x7f0d780c4980 2024-02-19T15:02:27.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d80701640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7806fc10 con 0x7f0d780add70 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d80701640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d78157080 con 0x7f0d780705b0 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d7ec9e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d781ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d7ec9e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d781ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44516/0 (socket says 172.21.15.78:44516) 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d7ec9e640 1 -- 172.21.15.78:0/356630105 learned_addr learned my addr 172.21.15.78:0/356630105 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d7dc9c640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d780add70 0x7f0d781eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.531+0000 7f0d7e49d640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d7ec9e640 1 -- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 msgr2=0x7f0d78156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d7ec9e640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d7ec9e640 1 -- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d780add70 msgr2=0x7f0d781eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d7ec9e640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d780add70 0x7f0d781eb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d7ec9e640 1 -- 172.21.15.78:0/356630105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d68009400 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d7ec9e640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d781ebbb0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000e960 tx=0x7f0d7000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70009690 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7000c920 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d781ec0f0 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000ccd0 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.532+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d78155a20 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.533+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780676a0 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.533+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0d70013470 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.534+0000 7f0d6effd640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d5c06b810 0x7f0d5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.534+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0d7008a630 con 0x7f0d780c4980 2024-02-19T15:02:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.534+0000 7f0d7e49d640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d5c06b810 0x7f0d5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.534+0000 7f0d7e49d640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d5c06b810 0x7f0d5c06dcd0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0d740012e0 tx=0x7f0d74001700 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:27.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.538+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0d70054de0 con 0x7f0d780c4980 2024-02-19T15:02:27.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.745+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d780c78b0 con 0x7f0d780c4980 2024-02-19T15:02:27.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.745+0000 7f0d6effd640 1 -- 172.21.15.78:0/356630105 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f0d70017d60 con 0x7f0d780c4980 2024-02-19T15:02:27.748 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:27.748 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:27.748 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d5c06b810 msgr2=0x7f0d5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d5c06b810 0x7f0d5c06dcd0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0d740012e0 tx=0x7f0d74001700 comp rx=0 tx=0).stop 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 msgr2=0x7f0d781ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d781ebbb0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000e960 tx=0x7f0d7000ee20 comp rx=0 tx=0).stop 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 shutdown_connections 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0d5c06b810 0x7f0d5c06dcd0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d780c4980 0x7f0d781ebbb0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d780add70 0x7f0d781eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 --2- 172.21.15.78:0/356630105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d780705b0 0x7f0d78156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 >> 172.21.15.78:0/356630105 conn(0x7f0d780c85e0 msgr2=0x7f0d780c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 shutdown_connections 2024-02-19T15:02:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:27.747+0000 7f0d80701640 1 -- 172.21.15.78:0/356630105 wait complete. 2024-02-19T15:02:28.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:02:28.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:02:28.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:28.164 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:29.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.276+0000 7f87b7883640 1 Processor -- start 2024-02-19T15:02:29.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.276+0000 7f87b7883640 1 -- start start 2024-02-19T15:02:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b7883640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87b0155750 0x7f87b01609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:29.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b7883640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b0159000 con 0x7f87b0152500 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b7883640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b00c4f30 con 0x7f87b00c47e0 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b7883640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b0155b80 con 0x7f87b0155750 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b561f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f87b0152500 0x7f87b006e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47014/0 (socket says 172.21.15.78:47014) 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b561f640 1 -- 172.21.15.78:0/664343111 learned_addr learned my addr 172.21.15.78:0/664343111 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.277+0000 7f87b5e20640 1 --2- 172.21.15.78:0/664343111 >> v2:172.21.15.150:3300/0 conn(0x7f87b0155750 0x7f87b01609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.278+0000 7f87a7fff640 1 -- 172.21.15.78:0/664343111 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383587808 0 0) 0x7f87b0159000 con 0x7f87b0152500 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.278+0000 7f87a7fff640 1 -- 172.21.15.78:0/664343111 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879c003520 con 0x7f87b0152500 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.278+0000 7f87b5e20640 1 -- 172.21.15.78:0/664343111 >> v1:172.21.15.150:6789/0 conn(0x7f87b00c47e0 legacy=0x7f87b0155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.278+0000 7f87a7fff640 1 -- 172.21.15.78:0/664343111 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205172290 0 0) 0x7f87b00c4f30 con 0x7f87b00c47e0 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.278+0000 7f87b5e20640 1 -- 172.21.15.78:0/664343111 >> v1:172.21.15.78:6789/0 conn(0x7f87b0152500 legacy=0x7f87b006e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.278+0000 7f87b5e20640 1 -- 172.21.15.78:0/664343111 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b00c4c70 con 0x7f87b0155750 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b5e20640 1 --2- 172.21.15.78:0/664343111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0155750 0x7f87b01609d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f87a80097a0 tx=0x7f87a802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d366a93ea453c75 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87a7fff640 1 -- 172.21.15.78:0/664343111 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8030610 con 0x7f87b0155750 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87a7fff640 1 -- 172.21.15.78:0/664343111 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8030770 con 0x7f87b0155750 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87a7fff640 1 -- 172.21.15.78:0/664343111 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8030b00 con 0x7f87b0155750 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 -- 172.21.15.78:0/664343111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0155750 msgr2=0x7f87b01609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 --2- 172.21.15.78:0/664343111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0155750 0x7f87b01609d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f87a80097a0 tx=0x7f87a802b960 comp rx=0 tx=0).stop 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 -- 172.21.15.78:0/664343111 shutdown_connections 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 --2- 172.21.15.78:0/664343111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0155750 0x7f87b01609d0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 -- 172.21.15.78:0/664343111 >> 172.21.15.78:0/664343111 conn(0x7f87b006d9e0 msgr2=0x7f87b00a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 -- 172.21.15.78:0/664343111 shutdown_connections 2024-02-19T15:02:29.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.279+0000 7f87b7883640 1 -- 172.21.15.78:0/664343111 wait complete. 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.280+0000 7f87b7883640 1 Processor -- start 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 -- start start 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87b00c47e0 0x7f87b0153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0152500 0x7f87b01ede80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 0x7f87b01ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00c3160 con 0x7f87b0155750 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0152eb0 con 0x7f87b0152500 2024-02-19T15:02:29.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b7883640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00c1a20 con 0x7f87b00c47e0 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b5e20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 0x7f87b01ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b5e20640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 0x7f87b01ee3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44524/0 (socket says 172.21.15.78:44524) 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b5e20640 1 -- 172.21.15.78:0/3558765050 learned_addr learned my addr 172.21.15.78:0/3558765050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b561f640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87b00c47e0 0x7f87b0153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.281+0000 7f87b4e1e640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0152500 0x7f87b01ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b5e20640 1 -- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87b00c47e0 msgr2=0x7f87b0153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b5e20640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87b00c47e0 0x7f87b0153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b5e20640 1 -- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0152500 msgr2=0x7f87b01ede80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b5e20640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0152500 0x7f87b01ede80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b5e20640 1 -- 172.21.15.78:0/3558765050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a8009400 con 0x7f87b0155750 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b5e20640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 0x7f87b01ee3c0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f87a80098c0 tx=0x7f87a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a803e070 con 0x7f87b0155750 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8030e10 con 0x7f87b0155750 2024-02-19T15:02:29.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8004270 con 0x7f87b0155750 2024-02-19T15:02:29.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b01ee900 con 0x7f87b0155750 2024-02-19T15:02:29.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.282+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b00c0630 con 0x7f87b0155750 2024-02-19T15:02:29.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.283+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b00c0490 con 0x7f87b0155750 2024-02-19T15:02:29.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.284+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f87a8042070 con 0x7f87b0155750 2024-02-19T15:02:29.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.284+0000 7f87a5ffb640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f879406b810 0x7f879406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:29.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.284+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f87a80abed0 con 0x7f87b0155750 2024-02-19T15:02:29.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.284+0000 7f87b561f640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f879406b810 0x7f879406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:29.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.285+0000 7f87b561f640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f879406b810 0x7f879406dcd0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005c00 tx=0x7f87ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.289+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f87b00c0490 con 0x7f87b0155750 2024-02-19T15:02:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.494+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b00c0490 con 0x7f87b0155750 2024-02-19T15:02:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.495+0000 7f87a5ffb640 1 -- 172.21.15.78:0/3558765050 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f87b00c0490 con 0x7f87b0155750 2024-02-19T15:02:29.498 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:29.498 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f879406b810 msgr2=0x7f879406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f879406b810 0x7f879406dcd0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005c00 tx=0x7f87ac009040 comp rx=0 tx=0).stop 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 msgr2=0x7f87b01ee3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 0x7f87b01ee3c0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f87a80098c0 tx=0x7f87a80046c0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 shutdown_connections 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f879406b810 0x7f879406dcd0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87b0155750 0x7f87b01ee3c0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87b0152500 0x7f87b01ede80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 --2- 172.21.15.78:0/3558765050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87b00c47e0 0x7f87b0153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 >> 172.21.15.78:0/3558765050 conn(0x7f87b006d9e0 msgr2=0x7f87b00c6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 shutdown_connections 2024-02-19T15:02:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:29.497+0000 7f87b7883640 1 -- 172.21.15.78:0/3558765050 wait complete. 2024-02-19T15:02:29.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:29.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:30.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.026+0000 7fbe6a21c640 1 Processor -- start 2024-02-19T15:02:31.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe6a21c640 1 -- start start 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe6a21c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe64152a50 0x7fbe64071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe6a21c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe640702d0 0x7fbe64158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe6a21c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe64071bf0 con 0x7fbe64152670 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe6a21c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe64158a20 con 0x7fbe64152a50 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe6a21c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe640c6780 con 0x7fbe640702d0 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe62ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe64152a50 0x7fbe64071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe637fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbe64152670 0x7fbe64155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47024/0 (socket says 172.21.15.78:47024) 2024-02-19T15:02:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.027+0000 7fbe637fe640 1 -- 172.21.15.78:0/3156890034 learned_addr learned my addr 172.21.15.78:0/3156890034 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.028+0000 7fbe63fff640 1 --2- 172.21.15.78:0/3156890034 >> v2:172.21.15.150:3300/0 conn(0x7fbe640702d0 0x7fbe64158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.028+0000 7fbe62ffd640 1 -- 172.21.15.78:0/3156890034 >> v2:172.21.15.150:3300/0 conn(0x7fbe640702d0 msgr2=0x7fbe64158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.028+0000 7fbe62ffd640 1 --2- 172.21.15.78:0/3156890034 >> v2:172.21.15.150:3300/0 conn(0x7fbe640702d0 0x7fbe64158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.028+0000 7fbe62ffd640 1 -- 172.21.15.78:0/3156890034 >> v1:172.21.15.78:6789/0 conn(0x7fbe64152670 legacy=0x7fbe64155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.028+0000 7fbe62ffd640 1 -- 172.21.15.78:0/3156890034 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64071630 con 0x7fbe64152a50 2024-02-19T15:02:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.028+0000 7fbe627fc640 1 -- 172.21.15.78:0/3156890034 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287940520 0 0) 0x7fbe64071bf0 con 0x7fbe64152670 2024-02-19T15:02:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe62ffd640 1 --2- 172.21.15.78:0/3156890034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe64071050 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fbe64070df0 tx=0x7fbe5802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab10b548ffd5735 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe627fc640 1 -- 172.21.15.78:0/3156890034 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58030610 con 0x7fbe64152a50 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe627fc640 1 -- 172.21.15.78:0/3156890034 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58030770 con 0x7fbe64152a50 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe627fc640 1 -- 172.21.15.78:0/3156890034 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58030b00 con 0x7fbe64152a50 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 -- 172.21.15.78:0/3156890034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 msgr2=0x7fbe64071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/3156890034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe64071050 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fbe64070df0 tx=0x7fbe5802b960 comp rx=0 tx=0).stop 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 -- 172.21.15.78:0/3156890034 shutdown_connections 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/3156890034 >> v2:172.21.15.150:3300/0 conn(0x7fbe640702d0 0x7fbe64158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/3156890034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe64071050 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 -- 172.21.15.78:0/3156890034 >> 172.21.15.78:0/3156890034 conn(0x7fbe640c85e0 msgr2=0x7fbe640c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 -- 172.21.15.78:0/3156890034 shutdown_connections 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.029+0000 7fbe6a21c640 1 -- 172.21.15.78:0/3156890034 wait complete. 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.030+0000 7fbe6a21c640 1 Processor -- start 2024-02-19T15:02:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 -- start start 2024-02-19T15:02:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640702d0 0x7fbe641543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe64152670 0x7fbe641eab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe641eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640ad500 con 0x7fbe64152a50 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640acef0 con 0x7fbe64152670 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe6a21c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641de420 con 0x7fbe640702d0 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.031+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe641eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe641eb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44548/0 (socket says 172.21.15.78:44548) 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 -- 172.21.15.78:0/2164756794 learned_addr learned my addr 172.21.15.78:0/2164756794 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe637fe640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640702d0 0x7fbe641543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe62ffd640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe64152670 0x7fbe641eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 -- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640702d0 msgr2=0x7fbe641543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640702d0 0x7fbe641543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 -- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe64152670 msgr2=0x7fbe641eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe64152670 0x7fbe641eab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 -- 172.21.15.78:0/2164756794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58009400 con 0x7fbe64152a50 2024-02-19T15:02:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe63fff640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe641eb0d0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00efc0 tx=0x7fbe5c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c009ba0 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c00f040 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640ad190 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c0146c0 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.032+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641548e0 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.033+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe640676a0 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.034+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fbe5c014820 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.034+0000 7fbe4bfff640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe5006b810 0x7fbe5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.034+0000 7fbe637fe640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe5006b810 0x7fbe5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.034+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbe5c08add0 con 0x7fbe64152a50 2024-02-19T15:02:31.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.035+0000 7fbe637fe640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe5006b810 0x7fbe5006dcd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbe54000f60 tx=0x7fbe54007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:31.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.039+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbe5c0556a0 con 0x7fbe64152a50 2024-02-19T15:02:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.245+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe64070120 con 0x7fbe64152a50 2024-02-19T15:02:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.246+0000 7fbe4bfff640 1 -- 172.21.15.78:0/2164756794 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7fbe5c08c090 con 0x7fbe64152a50 2024-02-19T15:02:31.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:31.249 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:31.249 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe5006b810 msgr2=0x7fbe5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe5006b810 0x7fbe5006dcd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbe54000f60 tx=0x7fbe54007d50 comp rx=0 tx=0).stop 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 msgr2=0x7fbe641eb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe641eb0d0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00efc0 tx=0x7fbe5c00c490 comp rx=0 tx=0).stop 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 shutdown_connections 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fbe5006b810 0x7fbe5006dcd0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe64152a50 0x7fbe641eb0d0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe64152670 0x7fbe641eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 --2- 172.21.15.78:0/2164756794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe640702d0 0x7fbe641543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 >> 172.21.15.78:0/2164756794 conn(0x7fbe640c85e0 msgr2=0x7fbe64156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 shutdown_connections 2024-02-19T15:02:31.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:31.248+0000 7fbe6a21c640 1 -- 172.21.15.78:0/2164756794 wait complete. 2024-02-19T15:02:31.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:31.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:32.665 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:32.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.783+0000 7f7899bbd640 1 Processor -- start 2024-02-19T15:02:32.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.784+0000 7f7899bbd640 1 -- start start 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.784+0000 7f7899bbd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f789406d920 0x7f789415e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.784+0000 7f7899bbd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7894150820 con 0x7f789406ed20 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.784+0000 7f7899bbd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78941537d0 con 0x7f7894150440 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.784+0000 7f7899bbd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78940c1b60 con 0x7f789406d920 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f7893fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f789406d920 0x7f789415e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f7892ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f789406ed20 0x7f789406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47040/0 (socket says 172.21.15.78:47040) 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f7892ffd640 1 -- 172.21.15.78:0/649493470 learned_addr learned my addr 172.21.15.78:0/649493470 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f78927fc640 1 -- 172.21.15.78:0/649493470 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337477073 0 0) 0x7f78941537d0 con 0x7f7894150440 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f78927fc640 1 -- 172.21.15.78:0/649493470 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f787c003520 con 0x7f7894150440 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f7893fff640 1 -- 172.21.15.78:0/649493470 >> v1:172.21.15.150:6789/0 conn(0x7f7894150440 legacy=0x7f78940c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f7893fff640 1 -- 172.21.15.78:0/649493470 >> v1:172.21.15.78:6789/0 conn(0x7f789406ed20 legacy=0x7f789406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:32.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.785+0000 7f7893fff640 1 -- 172.21.15.78:0/649493470 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894153d10 con 0x7f789406d920 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.786+0000 7f7893fff640 1 --2- 172.21.15.78:0/649493470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f789406d920 0x7f789415e730 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f7894153960 tx=0x7f788c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd062bcb6c40350 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.786+0000 7f78927fc640 1 -- 172.21.15.78:0/649493470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c02cd80 con 0x7f789406d920 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.786+0000 7f78927fc640 1 -- 172.21.15.78:0/649493470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c034440 con 0x7f789406d920 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.786+0000 7f78927fc640 1 -- 172.21.15.78:0/649493470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0347b0 con 0x7f789406d920 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 -- 172.21.15.78:0/649493470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f789406d920 msgr2=0x7f789415e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 --2- 172.21.15.78:0/649493470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f789406d920 0x7f789415e730 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f7894153960 tx=0x7f788c02bfd0 comp rx=0 tx=0).stop 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 -- 172.21.15.78:0/649493470 shutdown_connections 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 --2- 172.21.15.78:0/649493470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f789406d920 0x7f789415e730 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 -- 172.21.15.78:0/649493470 >> 172.21.15.78:0/649493470 conn(0x7f789406b270 msgr2=0x7f789406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 -- 172.21.15.78:0/649493470 shutdown_connections 2024-02-19T15:02:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.787+0000 7f7899bbd640 1 -- 172.21.15.78:0/649493470 wait complete. 2024-02-19T15:02:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.788+0000 7f7899bbd640 1 Processor -- start 2024-02-19T15:02:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 -- start start 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789406d920 0x7f78941e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f789406ed20 0x7f78940bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 0x7f78941edb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7894152150 con 0x7f7894150440 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78941529d0 con 0x7f789406d920 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7899bbd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78940c6030 con 0x7f789406ed20 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7893fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 0x7f78941edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7893fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 0x7f78941edb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44568/0 (socket says 172.21.15.78:44568) 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7893fff640 1 -- 172.21.15.78:0/1852145050 learned_addr learned my addr 172.21.15.78:0/1852145050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f7892ffd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f789406ed20 0x7f78940bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.789+0000 7f78937fe640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789406d920 0x7f78941e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7893fff640 1 -- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f789406ed20 msgr2=0x7f78940bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7893fff640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f789406ed20 0x7f78940bf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7893fff640 1 -- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789406d920 msgr2=0x7f78941e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7893fff640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789406d920 0x7f78941e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7893fff640 1 -- 172.21.15.78:0/1852145050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c009a70 con 0x7f7894150440 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7893fff640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 0x7f78941edb30 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f788c037040 tx=0x7f788c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c003b90 con 0x7f7894150440 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c003cf0 con 0x7f7894150440 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c004250 con 0x7f7894150440 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789413a990 con 0x7f7894150440 2024-02-19T15:02:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.790+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7894152660 con 0x7f7894150440 2024-02-19T15:02:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.792+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7894152820 con 0x7f7894150440 2024-02-19T15:02:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.792+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f788c0043b0 con 0x7f7894150440 2024-02-19T15:02:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.793+0000 7f787bfff640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f788006b810 0x7f788006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:32.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.793+0000 7f78937fe640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f788006b810 0x7f788006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:32.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.793+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f788c0acb60 con 0x7f7894150440 2024-02-19T15:02:32.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.793+0000 7f78937fe640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f788006b810 0x7f788006dcd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f7884007fd0 tx=0x7f7884009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:32.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:32.797+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f788c077430 con 0x7f7894150440 2024-02-19T15:02:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.007+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78941ee830 con 0x7f7894150440 2024-02-19T15:02:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.008+0000 7f787bfff640 1 -- 172.21.15.78:0/1852145050 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f788c039d60 con 0x7f7894150440 2024-02-19T15:02:33.011 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:33.011 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f788006b810 msgr2=0x7f788006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f788006b810 0x7f788006dcd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f7884007fd0 tx=0x7f7884009040 comp rx=0 tx=0).stop 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 msgr2=0x7f78941edb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 0x7f78941edb30 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f788c037040 tx=0x7f788c004a40 comp rx=0 tx=0).stop 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 shutdown_connections 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f788006b810 0x7f788006dcd0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7894150440 0x7f78941edb30 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f789406ed20 0x7f78940bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 --2- 172.21.15.78:0/1852145050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789406d920 0x7f78941e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:33.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 >> 172.21.15.78:0/1852145050 conn(0x7f789406b270 msgr2=0x7f789406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:33.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 shutdown_connections 2024-02-19T15:02:33.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:33.010+0000 7f7899bbd640 1 -- 172.21.15.78:0/1852145050 wait complete. 2024-02-19T15:02:33.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:33.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:34.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:34.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.538+0000 7f054432c640 1 Processor -- start 2024-02-19T15:02:34.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.538+0000 7f054432c640 1 -- start start 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f054432c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053c0c42e0 0x7f053c158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f054432c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0c34a0 con 0x7f053c0c2f50 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f054432c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c06c8e0 con 0x7f053c0c8500 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f054432c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c06c090 con 0x7f053c0c42e0 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05418c7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f053c0c42e0 0x7f053c158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05428c9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f053c0c2f50 0x7f053c15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47048/0 (socket says 172.21.15.78:47048) 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05428c9640 1 -- 172.21.15.78:0/1053971369 learned_addr learned my addr 172.21.15.78:0/1053971369 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05410c6640 1 -- 172.21.15.78:0/1053971369 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385222376 0 0) 0x7f053c0c34a0 con 0x7f053c0c2f50 2024-02-19T15:02:34.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05410c6640 1 -- 172.21.15.78:0/1053971369 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0528003520 con 0x7f053c0c2f50 2024-02-19T15:02:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05418c7640 1 -- 172.21.15.78:0/1053971369 >> v1:172.21.15.150:6789/0 conn(0x7f053c0c8500 legacy=0x7f053c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05410c6640 1 -- 172.21.15.78:0/1053971369 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393855419 0 0) 0x7f053c06c8e0 con 0x7f053c0c8500 2024-02-19T15:02:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05418c7640 1 -- 172.21.15.78:0/1053971369 >> v1:172.21.15.78:6789/0 conn(0x7f053c0c2f50 legacy=0x7f053c15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.539+0000 7f05418c7640 1 -- 172.21.15.78:0/1053971369 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c06cbc0 con 0x7f053c0c42e0 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.540+0000 7f05418c7640 1 --2- 172.21.15.78:0/1053971369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c0c42e0 0x7f053c158d60 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f052c0097a0 tx=0x7f052c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a6c99b5b629a60 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.540+0000 7f05410c6640 1 -- 172.21.15.78:0/1053971369 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c031830 con 0x7f053c0c42e0 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.540+0000 7f05410c6640 1 -- 172.21.15.78:0/1053971369 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c031990 con 0x7f053c0c42e0 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.540+0000 7f05410c6640 1 -- 172.21.15.78:0/1053971369 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c031d00 con 0x7f053c0c42e0 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 -- 172.21.15.78:0/1053971369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c0c42e0 msgr2=0x7f053c158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 --2- 172.21.15.78:0/1053971369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c0c42e0 0x7f053c158d60 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f052c0097a0 tx=0x7f052c02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 -- 172.21.15.78:0/1053971369 shutdown_connections 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 --2- 172.21.15.78:0/1053971369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c0c42e0 0x7f053c158d60 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 -- 172.21.15.78:0/1053971369 >> 172.21.15.78:0/1053971369 conn(0x7f053c0aa300 msgr2=0x7f053c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 -- 172.21.15.78:0/1053971369 shutdown_connections 2024-02-19T15:02:34.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.541+0000 7f054432c640 1 -- 172.21.15.78:0/1053971369 wait complete. 2024-02-19T15:02:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.542+0000 7f054432c640 1 Processor -- start 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.542+0000 7f054432c640 1 -- start start 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f054432c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f053c0c2f50 0x7f053c152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f054432c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053c0c8500 0x7f053c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f054432c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 0x7f053c06fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f054432c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c06f290 con 0x7f053c14b280 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f054432c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c074430 con 0x7f053c0c8500 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f054432c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1ee800 con 0x7f053c0c2f50 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 0x7f053c06fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 0x7f053c06fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44592/0 (socket says 172.21.15.78:44592) 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 -- 172.21.15.78:0/2366483901 learned_addr learned my addr 172.21.15.78:0/2366483901 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05420c8640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f053c0c2f50 0x7f053c152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05418c7640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053c0c8500 0x7f053c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 -- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f053c0c2f50 msgr2=0x7f053c152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f053c0c2f50 0x7f053c152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 -- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053c0c8500 msgr2=0x7f053c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053c0c8500 0x7f053c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 -- 172.21.15.78:0/2366483901 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009400 con 0x7f053c14b280 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05418c7640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053c0c8500 0x7f053c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05420c8640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f053c0c2f50 0x7f053c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f05428c9640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 0x7f053c06fce0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f053400cb40 tx=0x7f053400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.543+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053400aca0 con 0x7f053c14b280 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.544+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053400ae00 con 0x7f053c14b280 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.544+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0534012620 con 0x7f053c14b280 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.544+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c1eeae0 con 0x7f053c14b280 2024-02-19T15:02:34.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.544+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c06f7c0 con 0x7f053c14b280 2024-02-19T15:02:34.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.545+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c071c20 con 0x7f053c14b280 2024-02-19T15:02:34.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.545+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0534007070 con 0x7f053c14b280 2024-02-19T15:02:34.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.545+0000 7f0532ffd640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f052006b810 0x7f052006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:34.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.546+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f053408a5b0 con 0x7f053c14b280 2024-02-19T15:02:34.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.546+0000 7f05420c8640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f052006b810 0x7f052006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:34.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.546+0000 7f05420c8640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f052006b810 0x7f052006dcd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0538000f60 tx=0x7f0538009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:34.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.550+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0534054cb0 con 0x7f053c14b280 2024-02-19T15:02:34.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.758+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f053c06f9b0 con 0x7f053c14b280 2024-02-19T15:02:34.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.759+0000 7f0532ffd640 1 -- 172.21.15.78:0/2366483901 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f0534017090 con 0x7f053c14b280 2024-02-19T15:02:34.762 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:34.762 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:34.762 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f052006b810 msgr2=0x7f052006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f052006b810 0x7f052006dcd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0538000f60 tx=0x7f0538009040 comp rx=0 tx=0).stop 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 msgr2=0x7f053c06fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 0x7f053c06fce0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f053400cb40 tx=0x7f053400a060 comp rx=0 tx=0).stop 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 shutdown_connections 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f052006b810 0x7f052006dcd0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f053c14b280 0x7f053c06fce0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f053c0c8500 0x7f053c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 --2- 172.21.15.78:0/2366483901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f053c0c2f50 0x7f053c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:34.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 >> 172.21.15.78:0/2366483901 conn(0x7f053c0aa300 msgr2=0x7f053c0a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:34.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 shutdown_connections 2024-02-19T15:02:34.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:34.761+0000 7f054432c640 1 -- 172.21.15.78:0/2366483901 wait complete. 2024-02-19T15:02:35.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:35.178 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:36.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:36.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.293+0000 7fcbe4377640 1 Processor -- start 2024-02-19T15:02:36.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.293+0000 7fcbe4377640 1 -- start start 2024-02-19T15:02:36.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe4377640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcbdc0c25e0 0x7fcbdc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:36.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe4377640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcbdc0c29c0 0x7fcbdc0c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe4377640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc072250 con 0x7fcbdc0c49c0 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe4377640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbdc152c90 con 0x7fcbdc0c29c0 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe4377640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbdc072480 con 0x7fcbdc0c25e0 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe1912640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcbdc0c29c0 0x7fcbdc0c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe2914640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcbdc0c49c0 0x7fcbdc06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47052/0 (socket says 172.21.15.78:47052) 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe2914640 1 -- 172.21.15.78:0/1870655875 learned_addr learned my addr 172.21.15.78:0/1870655875 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.294+0000 7fcbe2113640 1 --2- 172.21.15.78:0/1870655875 >> v2:172.21.15.150:3300/0 conn(0x7fcbdc0c25e0 0x7fcbdc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.295+0000 7fcbe1912640 1 -- 172.21.15.78:0/1870655875 >> v2:172.21.15.150:3300/0 conn(0x7fcbdc0c25e0 msgr2=0x7fcbdc1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.295+0000 7fcbe1912640 1 --2- 172.21.15.78:0/1870655875 >> v2:172.21.15.150:3300/0 conn(0x7fcbdc0c25e0 0x7fcbdc1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.295+0000 7fcbe1912640 1 -- 172.21.15.78:0/1870655875 >> v1:172.21.15.78:6789/0 conn(0x7fcbdc0c49c0 legacy=0x7fcbdc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.295+0000 7fcbe1912640 1 -- 172.21.15.78:0/1870655875 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc0c3eb0 con 0x7fcbdc0c29c0 2024-02-19T15:02:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.295+0000 7fcbe1111640 1 -- 172.21.15.78:0/1870655875 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113616131 0 0) 0x7fcbdc072250 con 0x7fcbdc0c49c0 2024-02-19T15:02:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.295+0000 7fcbe1912640 1 --2- 172.21.15.78:0/1870655875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc0c4480 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc06f120 tx=0x7fcbcc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8b69250d1c1aaa server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe1111640 1 -- 172.21.15.78:0/1870655875 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc02e070 con 0x7fcbdc0c29c0 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe1111640 1 -- 172.21.15.78:0/1870655875 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0315a0 con 0x7fcbdc0c29c0 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe1111640 1 -- 172.21.15.78:0/1870655875 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc031930 con 0x7fcbdc0c29c0 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 -- 172.21.15.78:0/1870655875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc0c29c0 msgr2=0x7fcbdc0c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 --2- 172.21.15.78:0/1870655875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc0c4480 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc06f120 tx=0x7fcbcc02b840 comp rx=0 tx=0).stop 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 -- 172.21.15.78:0/1870655875 shutdown_connections 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 --2- 172.21.15.78:0/1870655875 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc0c4480 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 --2- 172.21.15.78:0/1870655875 >> v2:172.21.15.150:3300/0 conn(0x7fcbdc0c25e0 0x7fcbdc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 -- 172.21.15.78:0/1870655875 >> 172.21.15.78:0/1870655875 conn(0x7fcbdc06b270 msgr2=0x7fcbdc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 -- 172.21.15.78:0/1870655875 shutdown_connections 2024-02-19T15:02:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.296+0000 7fcbe4377640 1 -- 172.21.15.78:0/1870655875 wait complete. 2024-02-19T15:02:36.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.297+0000 7fcbe4377640 1 Processor -- start 2024-02-19T15:02:36.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 -- start start 2024-02-19T15:02:36.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbdc0c25e0 0x7fcbdc146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 0x7fcbdc145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc1551c0 con 0x7fcbdc145290 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc145c80 con 0x7fcbdc0c29c0 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe4377640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc1ee390 con 0x7fcbdc0c25e0 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe2914640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 0x7fcbdc145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe2914640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 0x7fcbdc145740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44602/0 (socket says 172.21.15.78:44602) 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe2914640 1 -- 172.21.15.78:0/3937944411 learned_addr learned my addr 172.21.15.78:0/3937944411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe2113640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbdc0c25e0 0x7fcbdc146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.298+0000 7fcbe1912640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe2914640 1 -- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbdc0c25e0 msgr2=0x7fcbdc146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe2914640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbdc0c25e0 0x7fcbdc146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe2914640 1 -- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbdc0c29c0 msgr2=0x7fcbdc149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe2914640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe2914640 1 -- 172.21.15.78:0/3937944411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc009400 con 0x7fcbdc145290 2024-02-19T15:02:36.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe2914640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 0x7fcbdc145740 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400c730 tx=0x7fcbd400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:36.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400ce40 con 0x7fcbdc145290 2024-02-19T15:02:36.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd400a640 con 0x7fcbdc145290 2024-02-19T15:02:36.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400a9f0 con 0x7fcbdc145290 2024-02-19T15:02:36.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc1ee8c0 con 0x7fcbdc145290 2024-02-19T15:02:36.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.299+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc0a83e0 con 0x7fcbdc145290 2024-02-19T15:02:36.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.300+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc0a8e10 con 0x7fcbdc145290 2024-02-19T15:02:36.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.301+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcbd400ab50 con 0x7fcbdc145290 2024-02-19T15:02:36.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.301+0000 7fcbd2ffd640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbc006b810 0x7fcbc006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:36.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.301+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcbd4089a40 con 0x7fcbdc145290 2024-02-19T15:02:36.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.301+0000 7fcbe2113640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbc006b810 0x7fcbc006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:36.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.302+0000 7fcbe2113640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbc006b810 0x7fcbc006dcd0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc14edf0 tx=0x7fcbd8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:36.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.306+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcbd408be60 con 0x7fcbdc145290 2024-02-19T15:02:36.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.512+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbdc0a8570 con 0x7fcbdc145290 2024-02-19T15:02:36.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.513+0000 7fcbd2ffd640 1 -- 172.21.15.78:0/3937944411 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7fcbd4018020 con 0x7fcbdc145290 2024-02-19T15:02:36.516 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:36.516 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:36.516 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbc006b810 msgr2=0x7fcbc006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbc006b810 0x7fcbc006dcd0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc14edf0 tx=0x7fcbd8009190 comp rx=0 tx=0).stop 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 msgr2=0x7fcbdc145740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 0x7fcbdc145740 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400c730 tx=0x7fcbd400cbf0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 shutdown_connections 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcbc006b810 0x7fcbc006dcd0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbdc145290 0x7fcbdc145740 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbdc0c29c0 0x7fcbdc149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 --2- 172.21.15.78:0/3937944411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcbdc0c25e0 0x7fcbdc146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 >> 172.21.15.78:0/3937944411 conn(0x7fcbdc06b270 msgr2=0x7fcbdc150c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 shutdown_connections 2024-02-19T15:02:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:36.515+0000 7fcbe4377640 1 -- 172.21.15.78:0/3937944411 wait complete. 2024-02-19T15:02:36.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:36.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:37.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.042+0000 7f8bb94d4640 1 Processor -- start 2024-02-19T15:02:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.042+0000 7f8bb94d4640 1 -- start start 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb94d4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8bb406c9b0 0x7f8bb4160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb94d4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb41551f0 con 0x7f8bb40c8e00 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb94d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb4155490 con 0x7f8bb4152700 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb94d4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb4155770 con 0x7f8bb406c9b0 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8bb406c9b0 0x7f8bb4160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb27fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8bb40c8e00 0x7f8bb406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50708/0 (socket says 172.21.15.78:50708) 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb27fc640 1 -- 172.21.15.78:0/1103492041 learned_addr learned my addr 172.21.15.78:0/1103492041 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb37fe640 1 --2- 172.21.15.78:0/1103492041 >> v2:172.21.15.78:3300/0 conn(0x7f8bb406c9b0 0x7f8bb4160990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59318/0 (socket says 172.21.15.78:59318) 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb37fe640 1 -- 172.21.15.78:0/1103492041 >> v1:172.21.15.150:6789/0 conn(0x7f8bb4152700 legacy=0x7f8bb40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb37fe640 1 -- 172.21.15.78:0/1103492041 >> v1:172.21.15.78:6789/0 conn(0x7f8bb40c8e00 legacy=0x7f8bb406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.043+0000 7f8bb37fe640 1 -- 172.21.15.78:0/1103492041 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb4155a50 con 0x7f8bb406c9b0 2024-02-19T15:02:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.044+0000 7f8bb37fe640 1 --2- 172.21.15.78:0/1103492041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb406c9b0 0x7f8bb4160990 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009d90 tx=0x7f8bac02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d09c018222269a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.044+0000 7f8bb1ffb640 1 -- 172.21.15.78:0/1103492041 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac02cd00 con 0x7f8bb406c9b0 2024-02-19T15:02:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.044+0000 7f8bb1ffb640 1 -- 172.21.15.78:0/1103492041 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac02ce60 con 0x7f8bb406c9b0 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.044+0000 7f8bb1ffb640 1 -- 172.21.15.78:0/1103492041 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac034660 con 0x7f8bb406c9b0 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.044+0000 7f8bb94d4640 1 -- 172.21.15.78:0/1103492041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb406c9b0 msgr2=0x7f8bb4160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.044+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/1103492041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb406c9b0 0x7f8bb4160990 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009d90 tx=0x7f8bac02bf50 comp rx=0 tx=0).stop 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.045+0000 7f8bb94d4640 1 -- 172.21.15.78:0/1103492041 shutdown_connections 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.045+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/1103492041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb406c9b0 0x7f8bb4160990 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.045+0000 7f8bb94d4640 1 -- 172.21.15.78:0/1103492041 >> 172.21.15.78:0/1103492041 conn(0x7f8bb40cac90 msgr2=0x7f8bb40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.045+0000 7f8bb94d4640 1 -- 172.21.15.78:0/1103492041 shutdown_connections 2024-02-19T15:02:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.045+0000 7f8bb94d4640 1 -- 172.21.15.78:0/1103492041 wait complete. 2024-02-19T15:02:38.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.046+0000 7f8bb94d4640 1 Processor -- start 2024-02-19T15:02:38.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.046+0000 7f8bb94d4640 1 -- start start 2024-02-19T15:02:38.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.046+0000 7f8bb94d4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8bb406c9b0 0x7f8bb41edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:38.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bb40c8e00 0x7f8bb4157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:38.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 0x7f8bb4157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb40aced0 con 0x7f8bb4152700 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb40c16b0 con 0x7f8bb40c8e00 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb4157320 con 0x7f8bb406c9b0 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 0x7f8bb4157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 0x7f8bb4157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59332/0 (socket says 172.21.15.78:59332) 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 -- 172.21.15.78:0/202161605 learned_addr learned my addr 172.21.15.78:0/202161605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb27fc640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bb40c8e00 0x7f8bb4157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb2ffd640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8bb406c9b0 0x7f8bb41edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 -- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8bb406c9b0 msgr2=0x7f8bb41edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8bb406c9b0 0x7f8bb41edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 -- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bb40c8e00 msgr2=0x7f8bb4157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bb40c8e00 0x7f8bb4157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 -- 172.21.15.78:0/202161605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac009a70 con 0x7f8bb4152700 2024-02-19T15:02:38.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb37fe640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 0x7f8bb4157f70 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009eb0 tx=0x7f8bac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:38.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac03d600 con 0x7f8bb4152700 2024-02-19T15:02:38.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac03d780 con 0x7f8bb4152700 2024-02-19T15:02:38.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac03db30 con 0x7f8bb4152700 2024-02-19T15:02:38.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb4157530 con 0x7f8bb4152700 2024-02-19T15:02:38.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.047+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb41ee940 con 0x7f8bb4152700 2024-02-19T15:02:38.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.048+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb4074990 con 0x7f8bb4152700 2024-02-19T15:02:38.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.049+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8bac03b040 con 0x7f8bb4152700 2024-02-19T15:02:38.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.049+0000 7f8b9b7fe640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ba006b840 0x7f8ba006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:38.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.049+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8bac0acc70 con 0x7f8bb4152700 2024-02-19T15:02:38.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.049+0000 7f8bb2ffd640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ba006b840 0x7f8ba006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:38.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.050+0000 7f8bb2ffd640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ba006b840 0x7f8ba006dd00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40016e0 tx=0x7f8ba4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:38.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.053+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8bac077540 con 0x7f8bb4152700 2024-02-19T15:02:38.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.262+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bb40c8770 con 0x7f8bb4152700 2024-02-19T15:02:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.262+0000 7f8b9b7fe640 1 -- 172.21.15.78:0/202161605 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f8bac039020 con 0x7f8bb4152700 2024-02-19T15:02:38.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:38.265 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:38.266 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:38.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ba006b840 msgr2=0x7f8ba006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ba006b840 0x7f8ba006dd00 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40016e0 tx=0x7f8ba4009040 comp rx=0 tx=0).stop 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 msgr2=0x7f8bb4157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 0x7f8bb4157f70 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009eb0 tx=0x7f8bac004a40 comp rx=0 tx=0).stop 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 shutdown_connections 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ba006b840 0x7f8ba006dd00 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8bb4152700 0x7f8bb4157f70 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bb40c8e00 0x7f8bb4157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 --2- 172.21.15.78:0/202161605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8bb406c9b0 0x7f8bb41edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 >> 172.21.15.78:0/202161605 conn(0x7f8bb40cac90 msgr2=0x7f8bb414cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 shutdown_connections 2024-02-19T15:02:38.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:38.264+0000 7f8bb94d4640 1 -- 172.21.15.78:0/202161605 wait complete. 2024-02-19T15:02:38.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:38.680 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:39.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.789+0000 7f200d1e9640 1 Processor -- start 2024-02-19T15:02:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.789+0000 7f200d1e9640 1 -- start start 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f200d1e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2008152480 0x7f20081559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f200d1e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f20080c7070 0x7f200806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f200d1e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2008155ef0 con 0x7f200806d350 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f200d1e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200806dec0 con 0x7f20080c7070 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f200d1e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080c75c0 con 0x7f2008152480 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f20080c7070 0x7f200806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f20080c7070 0x7f200806ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59336/0 (socket says 172.21.15.78:59336) 2024-02-19T15:02:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 -- 172.21.15.78:0/4172067556 learned_addr learned my addr 172.21.15.78:0/4172067556 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f2006d9d640 1 --2- 172.21.15.78:0/4172067556 >> v2:172.21.15.150:3300/0 conn(0x7f2008152480 0x7f20081559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 -- 172.21.15.78:0/4172067556 >> v2:172.21.15.150:3300/0 conn(0x7f2008152480 msgr2=0x7f20081559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 --2- 172.21.15.78:0/4172067556 >> v2:172.21.15.150:3300/0 conn(0x7f2008152480 0x7f20081559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 -- 172.21.15.78:0/4172067556 >> v1:172.21.15.150:6789/0 conn(0x7f200806d350 legacy=0x7f20080c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.790+0000 7f1ffffff640 1 -- 172.21.15.78:0/4172067556 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080c78a0 con 0x7f20080c7070 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.791+0000 7f1ffffff640 1 --2- 172.21.15.78:0/4172067556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20080c7070 0x7f200806ce10 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f20080aa1b0 tx=0x7f1ff002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc26ba52f48dc74d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.791+0000 7f200659c640 1 -- 172.21.15.78:0/4172067556 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0030860 con 0x7f20080c7070 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.791+0000 7f200659c640 1 -- 172.21.15.78:0/4172067556 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff00309c0 con 0x7f20080c7070 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.791+0000 7f200659c640 1 -- 172.21.15.78:0/4172067556 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0030d50 con 0x7f20080c7070 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 -- 172.21.15.78:0/4172067556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20080c7070 msgr2=0x7f200806ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 --2- 172.21.15.78:0/4172067556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20080c7070 0x7f200806ce10 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f20080aa1b0 tx=0x7f1ff002b9a0 comp rx=0 tx=0).stop 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 -- 172.21.15.78:0/4172067556 shutdown_connections 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 --2- 172.21.15.78:0/4172067556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20080c7070 0x7f200806ce10 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 --2- 172.21.15.78:0/4172067556 >> v2:172.21.15.150:3300/0 conn(0x7f2008152480 0x7f20081559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 -- 172.21.15.78:0/4172067556 >> 172.21.15.78:0/4172067556 conn(0x7f20080ca8a0 msgr2=0x7f20080cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 -- 172.21.15.78:0/4172067556 shutdown_connections 2024-02-19T15:02:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.792+0000 7f200d1e9640 1 -- 172.21.15.78:0/4172067556 wait complete. 2024-02-19T15:02:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.793+0000 7f200d1e9640 1 Processor -- start 2024-02-19T15:02:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.793+0000 7f200d1e9640 1 -- start start 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.793+0000 7f200d1e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20080c7070 0x7f200814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 0x7f20080ac380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20080ac8c0 0x7f20081edb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2008156f90 con 0x7f2008152480 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2008156800 con 0x7f20080ac8c0 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2008156280 con 0x7f20080c7070 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 0x7f20080ac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 0x7f20080ac380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59350/0 (socket says 172.21.15.78:59350) 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 -- 172.21.15.78:0/3300134764 learned_addr learned my addr 172.21.15.78:0/3300134764 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f2006d9d640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20080c7070 0x7f200814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200759e640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20080ac8c0 0x7f20081edb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 -- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20080c7070 msgr2=0x7f200814d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20080c7070 0x7f200814d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 -- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20080ac8c0 msgr2=0x7f20081edb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20080ac8c0 0x7f20081edb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 -- 172.21.15.78:0/3300134764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00094d0 con 0x7f2008152480 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1ffffff640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 0x7f20080ac380 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f1ff002b9e0 tx=0x7f1ff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0039c90 con 0x7f2008152480 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0039e10 con 0x7f2008152480 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2008156520 con 0x7f2008152480 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0043640 con 0x7f2008152480 2024-02-19T15:02:39.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.794+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20080c1490 con 0x7f2008152480 2024-02-19T15:02:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.795+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2008157140 con 0x7f2008152480 2024-02-19T15:02:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.796+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1ff00437a0 con 0x7f2008152480 2024-02-19T15:02:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.796+0000 7f1fff7fe640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe806b810 0x7f1fe806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.796+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1ff003e070 con 0x7f2008152480 2024-02-19T15:02:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.796+0000 7f2006d9d640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe806b810 0x7f1fe806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.797+0000 7f2006d9d640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe806b810 0x7f1fe806dcd0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f20080c1790 tx=0x7f1ff8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:39.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:39.801+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ff0077c60 con 0x7f2008152480 2024-02-19T15:02:40.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.009+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2008074990 con 0x7f2008152480 2024-02-19T15:02:40.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.009+0000 7f1fff7fe640 1 -- 172.21.15.78:0/3300134764 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3393 (secure 0 0 0) 0x7f1ff0034d60 con 0x7f2008152480 2024-02-19T15:02:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:40.012 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:09.653117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 207 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe806b810 msgr2=0x7f1fe806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe806b810 0x7f1fe806dcd0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f20080c1790 tx=0x7f1ff8009190 comp rx=0 tx=0).stop 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 msgr2=0x7f20080ac380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 0x7f20080ac380 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f1ff002b9e0 tx=0x7f1ff0004730 comp rx=0 tx=0).stop 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 shutdown_connections 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe806b810 0x7f1fe806dcd0 secure :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f20080c1790 tx=0x7f1ff8009190 comp rx=0 tx=0).stop 2024-02-19T15:02:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2008152480 0x7f20080ac380 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20080ac8c0 0x7f20081edb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 --2- 172.21.15.78:0/3300134764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20080c7070 0x7f200814d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 >> 172.21.15.78:0/3300134764 conn(0x7f20080ca8a0 msgr2=0x7f20080af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 shutdown_connections 2024-02-19T15:02:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:40.011+0000 7f200d1e9640 1 -- 172.21.15.78:0/3300134764 wait complete. 2024-02-19T15:02:40.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:09.653117+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:40.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:41.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:41.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.536+0000 7fd1d7b81640 1 Processor -- start 2024-02-19T15:02:41.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 -- start start 2024-02-19T15:02:41.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd1d00c25e0 0x7fd1d01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd1d00c29c0 0x7fd1d00c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d00c49c0 0x7fd1d006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d0152c90 con 0x7fd1d00c29c0 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d0072250 con 0x7fd1d00c25e0 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.537+0000 7fd1d7b81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d0072480 con 0x7fd1d00c49c0 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd1d00c29c0 0x7fd1d00c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd1d00c29c0 0x7fd1d00c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59356/0 (socket says 172.21.15.78:59356) 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 -- 172.21.15.78:0/2122320952 learned_addr learned my addr 172.21.15.78:0/2122320952 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d611e640 1 --2- 172.21.15.78:0/2122320952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d00c49c0 0x7fd1d006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d591d640 1 --2- 172.21.15.78:0/2122320952 >> v2:172.21.15.150:3300/0 conn(0x7fd1d00c25e0 0x7fd1d01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 -- 172.21.15.78:0/2122320952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d00c49c0 msgr2=0x7fd1d006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 --2- 172.21.15.78:0/2122320952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d00c49c0 0x7fd1d006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 -- 172.21.15.78:0/2122320952 >> v2:172.21.15.150:3300/0 conn(0x7fd1d00c25e0 msgr2=0x7fd1d01526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:41.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 --2- 172.21.15.78:0/2122320952 >> v2:172.21.15.150:3300/0 conn(0x7fd1d00c25e0 0x7fd1d01526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.538+0000 7fd1d511c640 1 -- 172.21.15.78:0/2122320952 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00c3eb0 con 0x7fd1d00c29c0 2024-02-19T15:02:41.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d511c640 1 --2- 172.21.15.78:0/2122320952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c29c0 0x7fd1d00c4480 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fd1d006f120 tx=0x7fd1c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4409b2fea1674574 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:41.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d491b640 1 -- 172.21.15.78:0/2122320952 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40317c0 con 0x7fd1d00c29c0 2024-02-19T15:02:41.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d491b640 1 -- 172.21.15.78:0/2122320952 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c4031920 con 0x7fd1d00c29c0 2024-02-19T15:02:41.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d491b640 1 -- 172.21.15.78:0/2122320952 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4031cb0 con 0x7fd1d00c29c0 2024-02-19T15:02:41.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d7b81640 1 -- 172.21.15.78:0/2122320952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c29c0 msgr2=0x7fd1d00c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:41.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/2122320952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c29c0 0x7fd1d00c4480 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fd1d006f120 tx=0x7fd1c402b860 comp rx=0 tx=0).stop 2024-02-19T15:02:41.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d7b81640 1 -- 172.21.15.78:0/2122320952 shutdown_connections 2024-02-19T15:02:41.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/2122320952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d00c49c0 0x7fd1d006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/2122320952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c29c0 0x7fd1d00c4480 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.539+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/2122320952 >> v2:172.21.15.150:3300/0 conn(0x7fd1d00c25e0 0x7fd1d01526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.540+0000 7fd1d7b81640 1 -- 172.21.15.78:0/2122320952 >> 172.21.15.78:0/2122320952 conn(0x7fd1d006b270 msgr2=0x7fd1d006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.540+0000 7fd1d7b81640 1 -- 172.21.15.78:0/2122320952 shutdown_connections 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.540+0000 7fd1d7b81640 1 -- 172.21.15.78:0/2122320952 wait complete. 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 Processor -- start 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 -- start start 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 0x7fd1d01eb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1d00c29c0 0x7fd1d0149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d0149850 0x7fd1d00a8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d01dee70 con 0x7fd1d00c25e0 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d00c0070 con 0x7fd1d00c29c0 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d7b81640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d01ebe40 con 0x7fd1d0149850 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d591d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 0x7fd1d01eb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d591d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 0x7fd1d01eb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59360/0 (socket says 172.21.15.78:59360) 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.541+0000 7fd1d591d640 1 -- 172.21.15.78:0/913906994 learned_addr learned my addr 172.21.15.78:0/913906994 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d611e640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d0149850 0x7fd1d00a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d511c640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1d00c29c0 0x7fd1d0149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d591d640 1 -- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d0149850 msgr2=0x7fd1d00a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d591d640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d0149850 0x7fd1d00a8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d591d640 1 -- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1d00c29c0 msgr2=0x7fd1d0149310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d591d640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1d00c29c0 0x7fd1d0149310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d591d640 1 -- 172.21.15.78:0/913906994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c40094d0 con 0x7fd1d00c25e0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d591d640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 0x7fd1d01eb4e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800e8c0 tx=0x7fd1c800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800c940 con 0x7fd1d00c25e0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c800caa0 con 0x7fd1d00c25e0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800ce30 con 0x7fd1d00c25e0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d01deb90 con 0x7fd1d00c25e0 2024-02-19T15:02:41.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.542+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d006e130 con 0x7fd1d00c25e0 2024-02-19T15:02:41.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.543+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d0074990 con 0x7fd1d00c25e0 2024-02-19T15:02:41.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.544+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd1c8013470 con 0x7fd1d00c25e0 2024-02-19T15:02:41.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.544+0000 7fd1ce7fc640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd1bc06b810 0x7fd1bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:41.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.544+0000 7fd1d511c640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd1bc06b810 0x7fd1bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:41.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.544+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd1c808a510 con 0x7fd1d00c25e0 2024-02-19T15:02:41.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.545+0000 7fd1d511c640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd1bc06b810 0x7fd1bc06dcd0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4036000 tx=0x7fd1c403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:41.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.549+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1c8054de0 con 0x7fd1d00c25e0 2024-02-19T15:02:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.756+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1d006c560 con 0x7fd1d00c25e0 2024-02-19T15:02:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.756+0000 7fd1ce7fc640 1 -- 172.21.15.78:0/913906994 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fd1c8017d90 con 0x7fd1d00c25e0 2024-02-19T15:02:41.759 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:41.759 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd1bc06b810 msgr2=0x7fd1bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd1bc06b810 0x7fd1bc06dcd0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4036000 tx=0x7fd1c403d040 comp rx=0 tx=0).stop 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 msgr2=0x7fd1d01eb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 0x7fd1d01eb4e0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800e8c0 tx=0x7fd1c800ed80 comp rx=0 tx=0).stop 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 shutdown_connections 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd1bc06b810 0x7fd1bc06dcd0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd1d0149850 0x7fd1d00a8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1d00c29c0 0x7fd1d0149310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 --2- 172.21.15.78:0/913906994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1d00c25e0 0x7fd1d01eb4e0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:41.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 >> 172.21.15.78:0/913906994 conn(0x7fd1d006b270 msgr2=0x7fd1d00c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 shutdown_connections 2024-02-19T15:02:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:41.758+0000 7fd1d7b81640 1 -- 172.21.15.78:0/913906994 wait complete. 2024-02-19T15:02:42.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:42.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:43.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 Processor -- start 2024-02-19T15:02:43.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 -- start start 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01340c5130 0x7f01340c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0134158f80 0x7f01340c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01340c3cf0 con 0x7f01341559c0 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0134158b60 con 0x7f0134158f80 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.292+0000 7f0139c95640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0134159400 con 0x7f01340c5130 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f012affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0134158f80 0x7f01340c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f0133fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f01341559c0 0x7f01340c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:52088/0 (socket says 172.21.15.78:52088) 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f01337fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01340c5130 0x7f01340c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f0133fff640 1 -- 172.21.15.78:0/420099919 learned_addr learned my addr 172.21.15.78:0/420099919 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f012affd640 1 -- 172.21.15.78:0/420099919 >> v2:172.21.15.150:3300/0 conn(0x7f01340c5130 msgr2=0x7f01340c30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:43.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f012affd640 1 --2- 172.21.15.78:0/420099919 >> v2:172.21.15.150:3300/0 conn(0x7f01340c5130 0x7f01340c30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f012affd640 1 -- 172.21.15.78:0/420099919 >> v1:172.21.15.150:6789/0 conn(0x7f01341559c0 legacy=0x7f01340c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:43.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.293+0000 7f012affd640 1 -- 172.21.15.78:0/420099919 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0134158530 con 0x7f0134158f80 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.294+0000 7f012affd640 1 --2- 172.21.15.78:0/420099919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0134158f80 0x7f01340c5510 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f01340c9130 tx=0x7f011c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707db31256f9f67b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.294+0000 7f0132ffd640 1 -- 172.21.15.78:0/420099919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c031830 con 0x7f0134158f80 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.294+0000 7f0132ffd640 1 -- 172.21.15.78:0/420099919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c031990 con 0x7f0134158f80 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.294+0000 7f0132ffd640 1 -- 172.21.15.78:0/420099919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c031d20 con 0x7f0134158f80 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.294+0000 7f0139c95640 1 -- 172.21.15.78:0/420099919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0134158f80 msgr2=0x7f01340c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.294+0000 7f0139c95640 1 --2- 172.21.15.78:0/420099919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0134158f80 0x7f01340c5510 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f01340c9130 tx=0x7f011c02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.295+0000 7f0139c95640 1 -- 172.21.15.78:0/420099919 shutdown_connections 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.295+0000 7f0139c95640 1 --2- 172.21.15.78:0/420099919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0134158f80 0x7f01340c5510 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.295+0000 7f0139c95640 1 --2- 172.21.15.78:0/420099919 >> v2:172.21.15.150:3300/0 conn(0x7f01340c5130 0x7f01340c30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.295+0000 7f0139c95640 1 -- 172.21.15.78:0/420099919 >> 172.21.15.78:0/420099919 conn(0x7f013406b4b0 msgr2=0x7f01340c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.295+0000 7f0139c95640 1 -- 172.21.15.78:0/420099919 shutdown_connections 2024-02-19T15:02:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.295+0000 7f0139c95640 1 -- 172.21.15.78:0/420099919 wait complete. 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.296+0000 7f0139c95640 1 Processor -- start 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.296+0000 7f0139c95640 1 -- start start 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0139c95640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340c5130 0x7f01341eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0139c95640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 0x7f0134154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0139c95640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0134158f80 0x7f0134151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0139c95640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341ea8e0 con 0x7f01341559c0 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0139c95640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013414f390 con 0x7f0134158f80 2024-02-19T15:02:43.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0139c95640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013414eb80 con 0x7f01340c5130 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 0x7f0134154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 0x7f0134154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59378/0 (socket says 172.21.15.78:59378) 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 -- 172.21.15.78:0/2913559731 learned_addr learned my addr 172.21.15.78:0/2913559731 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f01337fe640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340c5130 0x7f01341eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f0133fff640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0134158f80 0x7f0134151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 -- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340c5130 msgr2=0x7f01341eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340c5130 0x7f01341eb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 -- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0134158f80 msgr2=0x7f0134151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0134158f80 0x7f0134151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 -- 172.21.15.78:0/2913559731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c009400 con 0x7f01341559c0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.297+0000 7f012affd640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 0x7f0134154070 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f011c02bee0 tx=0x7f011c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.298+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c03a6f0 con 0x7f01341559c0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.298+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c03a870 con 0x7f01341559c0 2024-02-19T15:02:43.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.298+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01341eae60 con 0x7f01341559c0 2024-02-19T15:02:43.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.298+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c03ac20 con 0x7f01341559c0 2024-02-19T15:02:43.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.298+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013414edd0 con 0x7f01341559c0 2024-02-19T15:02:43.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.299+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01341eaff0 con 0x7f01341559c0 2024-02-19T15:02:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.299+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f011c043410 con 0x7f01341559c0 2024-02-19T15:02:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.300+0000 7f0130ff9640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f011406b810 0x7f011406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.300+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f011c0ac100 con 0x7f01341559c0 2024-02-19T15:02:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.300+0000 7f01337fe640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f011406b810 0x7f011406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.300+0000 7f01337fe640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f011406b810 0x7f011406dcd0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f0134155260 tx=0x7f0124009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:43.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.304+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f011c076af0 con 0x7f01341559c0 2024-02-19T15:02:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.512+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0134074990 con 0x7f01341559c0 2024-02-19T15:02:43.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.513+0000 7f0130ff9640 1 -- 172.21.15.78:0/2913559731 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f011c07a1b0 con 0x7f01341559c0 2024-02-19T15:02:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:43.516 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:43.516 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f011406b810 msgr2=0x7f011406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f011406b810 0x7f011406dcd0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f0134155260 tx=0x7f0124009190 comp rx=0 tx=0).stop 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 msgr2=0x7f0134154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 0x7f0134154070 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f011c02bee0 tx=0x7f011c0046c0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 shutdown_connections 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f011406b810 0x7f011406dcd0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01341559c0 0x7f0134154070 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0134158f80 0x7f0134151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 --2- 172.21.15.78:0/2913559731 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01340c5130 0x7f01341eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 >> 172.21.15.78:0/2913559731 conn(0x7f013406b4b0 msgr2=0x7f01340c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 shutdown_connections 2024-02-19T15:02:43.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:43.515+0000 7f0139c95640 1 -- 172.21.15.78:0/2913559731 wait complete. 2024-02-19T15:02:43.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:43.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:44.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:45.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.039+0000 7f472ae9a640 1 Processor -- start 2024-02-19T15:02:45.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f472ae9a640 1 -- start start 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f472ae9a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47240c6a50 0x7f4724155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f472ae9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47240c5920 0x7f4724155e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f472ae9a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240717b0 con 0x7f47240c2f70 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f472ae9a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47240c7560 con 0x7f47240c6a50 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f472ae9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4724071a00 con 0x7f47240c5920 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f4728c36640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47240c6a50 0x7f4724155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f4723fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f47240c2f70 0x7f472406c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50722/0 (socket says 172.21.15.78:50722) 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.040+0000 7f4723fff640 1 -- 172.21.15.78:0/11854740 learned_addr learned my addr 172.21.15.78:0/11854740 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.041+0000 7f4729437640 1 --2- 172.21.15.78:0/11854740 >> v2:172.21.15.150:3300/0 conn(0x7f47240c5920 0x7f4724155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.041+0000 7f4728c36640 1 -- 172.21.15.78:0/11854740 >> v2:172.21.15.150:3300/0 conn(0x7f47240c5920 msgr2=0x7f4724155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.041+0000 7f4728c36640 1 --2- 172.21.15.78:0/11854740 >> v2:172.21.15.150:3300/0 conn(0x7f47240c5920 0x7f4724155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.041+0000 7f4728c36640 1 -- 172.21.15.78:0/11854740 >> v1:172.21.15.78:6789/0 conn(0x7f47240c2f70 legacy=0x7f472406c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:45.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.041+0000 7f4728c36640 1 -- 172.21.15.78:0/11854740 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240c8030 con 0x7f47240c6a50 2024-02-19T15:02:45.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.041+0000 7f47237fe640 1 -- 172.21.15.78:0/11854740 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914174589 0 0) 0x7f47240717b0 con 0x7f47240c2f70 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f4728c36640 1 --2- 172.21.15.78:0/11854740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f4724155940 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f472406dd90 tx=0x7f471402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588c92e707d00d33 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f47237fe640 1 -- 172.21.15.78:0/11854740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471402e070 con 0x7f47240c6a50 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f47237fe640 1 -- 172.21.15.78:0/11854740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47140315a0 con 0x7f47240c6a50 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f47237fe640 1 -- 172.21.15.78:0/11854740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714031930 con 0x7f47240c6a50 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 -- 172.21.15.78:0/11854740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 msgr2=0x7f4724155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 --2- 172.21.15.78:0/11854740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f4724155940 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f472406dd90 tx=0x7f471402b840 comp rx=0 tx=0).stop 2024-02-19T15:02:45.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 -- 172.21.15.78:0/11854740 shutdown_connections 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 --2- 172.21.15.78:0/11854740 >> v2:172.21.15.150:3300/0 conn(0x7f47240c5920 0x7f4724155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 --2- 172.21.15.78:0/11854740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f4724155940 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 -- 172.21.15.78:0/11854740 >> 172.21.15.78:0/11854740 conn(0x7f472406b700 msgr2=0x7f47240aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 -- 172.21.15.78:0/11854740 shutdown_connections 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.042+0000 7f472ae9a640 1 -- 172.21.15.78:0/11854740 wait complete. 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.043+0000 7f472ae9a640 1 Processor -- start 2024-02-19T15:02:45.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.043+0000 7f472ae9a640 1 -- start start 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f472ae9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47240c2f70 0x7f4724156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f472ae9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47240c5920 0x7f47241eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f472ae9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f472414fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f472ae9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47241ee740 con 0x7f47240c6a50 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f472ae9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4724070f40 con 0x7f47240c2f70 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f472ae9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47241ebbb0 con 0x7f47240c5920 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.044+0000 7f4729437640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f472414fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f472414fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59392/0 (socket says 172.21.15.78:59392) 2024-02-19T15:02:45.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4723fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47240c5920 0x7f47241eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 -- 172.21.15.78:0/1796993806 learned_addr learned my addr 172.21.15.78:0/1796993806 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4728c36640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47240c2f70 0x7f4724156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 -- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47240c5920 msgr2=0x7f47241eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47240c5920 0x7f47241eb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 -- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47240c2f70 msgr2=0x7f4724156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47240c2f70 0x7f4724156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 -- 172.21.15.78:0/1796993806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714009400 con 0x7f47240c6a50 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f4729437640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f472414fca0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f471c009dd0 tx=0x7f471c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c014480 con 0x7f47240c6a50 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c00f040 con 0x7f47240c6a50 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47240a8df0 con 0x7f47240c6a50 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c014840 con 0x7f47240c6a50 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.045+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47240a9090 con 0x7f47240c6a50 2024-02-19T15:02:45.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.046+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47240a92b0 con 0x7f47240c6a50 2024-02-19T15:02:45.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.047+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f471c0149a0 con 0x7f47240c6a50 2024-02-19T15:02:45.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.047+0000 7f47217fa640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f471006b810 0x7f471006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:45.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.047+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f471c08b090 con 0x7f47240c6a50 2024-02-19T15:02:45.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.047+0000 7f4728c36640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f471006b810 0x7f471006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:45.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.047+0000 7f4728c36640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f471006b810 0x7f471006dcd0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f4714036000 tx=0x7f471403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:45.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.051+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f471c055960 con 0x7f47240c6a50 2024-02-19T15:02:45.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.259+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47240a8c20 con 0x7f47240c6a50 2024-02-19T15:02:45.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.260+0000 7f47217fa640 1 -- 172.21.15.78:0/1796993806 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f471c018d90 con 0x7f47240c6a50 2024-02-19T15:02:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:45.263 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:45.263 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.261+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f471006b810 msgr2=0x7f471006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.261+0000 7f472ae9a640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f471006b810 0x7f471006dcd0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f4714036000 tx=0x7f471403d040 comp rx=0 tx=0).stop 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.261+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 msgr2=0x7f472414fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.261+0000 7f472ae9a640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f472414fca0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f471c009dd0 tx=0x7f471c00c490 comp rx=0 tx=0).stop 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 shutdown_connections 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f471006b810 0x7f471006dcd0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47240c6a50 0x7f472414fca0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47240c5920 0x7f47241eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 --2- 172.21.15.78:0/1796993806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47240c2f70 0x7f4724156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 >> 172.21.15.78:0/1796993806 conn(0x7f472406b700 msgr2=0x7f47240c1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 shutdown_connections 2024-02-19T15:02:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:45.262+0000 7f472ae9a640 1 -- 172.21.15.78:0/1796993806 wait complete. 2024-02-19T15:02:45.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:45.678 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:46.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:46.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.797+0000 7fb63725c640 1 Processor -- start 2024-02-19T15:02:46.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.797+0000 7fb63725c640 1 -- start start 2024-02-19T15:02:46.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb63725c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6300ad890 0x7fb6300c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:46.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb63725c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6300ab600 0x7fb6300c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:46.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb63725c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300aba60 con 0x7fb6300c49c0 2024-02-19T15:02:46.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb63725c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb630152d30 con 0x7fb6300ad890 2024-02-19T15:02:46.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb63725c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6300c7860 con 0x7fb6300ab600 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6300ad890 0x7fb6300c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb6300ad890 0x7fb6300c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59396/0 (socket says 172.21.15.78:59396) 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 -- 172.21.15.78:0/1927397530 learned_addr learned my addr 172.21.15.78:0/1927397530 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb6357f9640 1 --2- 172.21.15.78:0/1927397530 >> v2:172.21.15.150:3300/0 conn(0x7fb6300ab600 0x7fb6300c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 -- 172.21.15.78:0/1927397530 >> v2:172.21.15.150:3300/0 conn(0x7fb6300ab600 msgr2=0x7fb6300c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 --2- 172.21.15.78:0/1927397530 >> v2:172.21.15.150:3300/0 conn(0x7fb6300ab600 0x7fb6300c5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 -- 172.21.15.78:0/1927397530 >> v1:172.21.15.150:6789/0 conn(0x7fb6300c49c0 legacy=0x7fb6301534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb62f7fe640 1 -- 172.21.15.78:0/1927397530 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921120989 0 0) 0x7fb6300aba60 con 0x7fb6300c49c0 2024-02-19T15:02:46.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.798+0000 7fb634ff8640 1 -- 172.21.15.78:0/1927397530 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300c7ce0 con 0x7fb6300ad890 2024-02-19T15:02:46.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.799+0000 7fb634ff8640 1 --2- 172.21.15.78:0/1927397530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ad890 0x7fb6300c64f0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fb630071a80 tx=0x7fb62002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b16604f27c29e0c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:46.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.799+0000 7fb62f7fe640 1 -- 172.21.15.78:0/1927397530 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6200317e0 con 0x7fb6300ad890 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.799+0000 7fb62f7fe640 1 -- 172.21.15.78:0/1927397530 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb620031940 con 0x7fb6300ad890 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.799+0000 7fb62f7fe640 1 -- 172.21.15.78:0/1927397530 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620031cd0 con 0x7fb6300ad890 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 -- 172.21.15.78:0/1927397530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ad890 msgr2=0x7fb6300c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 --2- 172.21.15.78:0/1927397530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ad890 0x7fb6300c64f0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fb630071a80 tx=0x7fb62002b880 comp rx=0 tx=0).stop 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 -- 172.21.15.78:0/1927397530 shutdown_connections 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 --2- 172.21.15.78:0/1927397530 >> v2:172.21.15.150:3300/0 conn(0x7fb6300ab600 0x7fb6300c5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 --2- 172.21.15.78:0/1927397530 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ad890 0x7fb6300c64f0 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 -- 172.21.15.78:0/1927397530 >> 172.21.15.78:0/1927397530 conn(0x7fb63006b700 msgr2=0x7fb6300aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 -- 172.21.15.78:0/1927397530 shutdown_connections 2024-02-19T15:02:46.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.800+0000 7fb63725c640 1 -- 172.21.15.78:0/1927397530 wait complete. 2024-02-19T15:02:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.801+0000 7fb63725c640 1 Processor -- start 2024-02-19T15:02:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.801+0000 7fb63725c640 1 -- start start 2024-02-19T15:02:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb63725c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 0x7fb630154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb63725c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300ad890 0x7fb6300bd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb63725c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300c49c0 0x7fb63014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb63725c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6301552d0 con 0x7fb6300ab600 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb63725c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6301e6050 con 0x7fb6300c49c0 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb63725c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6301e9280 con 0x7fb6300ad890 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb634ff8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 0x7fb630154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb634ff8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 0x7fb630154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59404/0 (socket says 172.21.15.78:59404) 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb634ff8640 1 -- 172.21.15.78:0/257546197 learned_addr learned my addr 172.21.15.78:0/257546197 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb6357f9640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300c49c0 0x7fb63014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.802+0000 7fb62ffff640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300ad890 0x7fb6300bd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb634ff8640 1 -- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300ad890 msgr2=0x7fb6300bd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb634ff8640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300ad890 0x7fb6300bd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb634ff8640 1 -- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300c49c0 msgr2=0x7fb63014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb634ff8640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300c49c0 0x7fb63014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb634ff8640 1 -- 172.21.15.78:0/257546197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6200094d0 con 0x7fb6300ab600 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb634ff8640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 0x7fb630154d90 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fb62002b8c0 tx=0x7fb6200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620032880 con 0x7fb6300ab600 2024-02-19T15:02:46.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb620032a00 con 0x7fb6300ab600 2024-02-19T15:02:46.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb630155530 con 0x7fb6300ab600 2024-02-19T15:02:46.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620032db0 con 0x7fb6300ab600 2024-02-19T15:02:46.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.803+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6301e87a0 con 0x7fb6300ab600 2024-02-19T15:02:46.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.804+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb630074990 con 0x7fb6300ab600 2024-02-19T15:02:46.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.805+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb62004e450 con 0x7fb6300ab600 2024-02-19T15:02:46.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.805+0000 7fb62d7fa640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb61c06b810 0x7fb61c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:46.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.805+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb6200ac380 con 0x7fb6300ab600 2024-02-19T15:02:46.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.805+0000 7fb62ffff640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb61c06b810 0x7fb61c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:46.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.806+0000 7fb62ffff640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb61c06b810 0x7fb61c06dcd0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb6300c5a80 tx=0x7fb624009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:46.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:46.809+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb620076d70 con 0x7fb6300ab600 2024-02-19T15:02:47.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.017+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6300c7ac0 con 0x7fb6300ab600 2024-02-19T15:02:47.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.017+0000 7fb62d7fa640 1 -- 172.21.15.78:0/257546197 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fb62007a430 con 0x7fb6300ab600 2024-02-19T15:02:47.020 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:47.020 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:47.021 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:47.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb61c06b810 msgr2=0x7fb61c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:47.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb61c06b810 0x7fb61c06dcd0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb6300c5a80 tx=0x7fb624009040 comp rx=0 tx=0).stop 2024-02-19T15:02:47.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 msgr2=0x7fb630154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:47.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 0x7fb630154d90 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fb62002b8c0 tx=0x7fb6200046c0 comp rx=0 tx=0).stop 2024-02-19T15:02:47.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 shutdown_connections 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb61c06b810 0x7fb61c06dcd0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6300c49c0 0x7fb63014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6300ad890 0x7fb6300bd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 --2- 172.21.15.78:0/257546197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6300ab600 0x7fb630154d90 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 >> 172.21.15.78:0/257546197 conn(0x7fb63006b700 msgr2=0x7fb6300c5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 shutdown_connections 2024-02-19T15:02:47.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:47.019+0000 7fb63725c640 1 -- 172.21.15.78:0/257546197 wait complete. 2024-02-19T15:02:47.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:47.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:48.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.545+0000 7f82ddcd4640 1 Processor -- start 2024-02-19T15:02:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.545+0000 7f82ddcd4640 1 -- start start 2024-02-19T15:02:48.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82ddcd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d806e590 0x7f82d806ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82ddcd4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f82d806b3c0 0x7f82d8152d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82ddcd4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d80c6400 con 0x7f82d8150420 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82ddcd4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d8150800 con 0x7f82d806b3c0 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82ddcd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80c6210 con 0x7f82d806e590 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f82d806b3c0 0x7f82d8152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f82d8150420 0x7f82d80c5b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55210/0 (socket says 172.21.15.78:55210) 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d77fe640 1 -- 172.21.15.78:0/1582469063 learned_addr learned my addr 172.21.15.78:0/1582469063 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d7fff640 1 --2- 172.21.15.78:0/1582469063 >> v2:172.21.15.78:3300/0 conn(0x7f82d806b3c0 0x7f82d8152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40852/0 (socket says 172.21.15.78:40852) 2024-02-19T15:02:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d6ffd640 1 --2- 172.21.15.78:0/1582469063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d806e590 0x7f82d806ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d7fff640 1 -- 172.21.15.78:0/1582469063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d806e590 msgr2=0x7f82d806ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d7fff640 1 --2- 172.21.15.78:0/1582469063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d806e590 0x7f82d806ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d7fff640 1 -- 172.21.15.78:0/1582469063 >> v1:172.21.15.78:6789/0 conn(0x7f82d8150420 legacy=0x7f82d80c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d67fc640 1 -- 172.21.15.78:0/1582469063 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902460692 0 0) 0x7f82d80c6400 con 0x7f82d8150420 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.546+0000 7f82d7fff640 1 -- 172.21.15.78:0/1582469063 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d80c2580 con 0x7f82d806b3c0 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.547+0000 7f82d7fff640 1 --2- 172.21.15.78:0/1582469063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806b3c0 0x7f82d8152d10 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009d90 tx=0x7f82d002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81c553b37d0dfbe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.547+0000 7f82d67fc640 1 -- 172.21.15.78:0/1582469063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d002cbc0 con 0x7f82d806b3c0 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.547+0000 7f82d67fc640 1 -- 172.21.15.78:0/1582469063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d002cd20 con 0x7f82d806b3c0 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.547+0000 7f82d67fc640 1 -- 172.21.15.78:0/1582469063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0034660 con 0x7f82d806b3c0 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 -- 172.21.15.78:0/1582469063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806b3c0 msgr2=0x7f82d8152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/1582469063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806b3c0 0x7f82d8152d10 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009d90 tx=0x7f82d002be10 comp rx=0 tx=0).stop 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 -- 172.21.15.78:0/1582469063 shutdown_connections 2024-02-19T15:02:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/1582469063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806b3c0 0x7f82d8152d10 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/1582469063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d806e590 0x7f82d806ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 -- 172.21.15.78:0/1582469063 >> 172.21.15.78:0/1582469063 conn(0x7f82d806d9e0 msgr2=0x7f82d80a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 -- 172.21.15.78:0/1582469063 shutdown_connections 2024-02-19T15:02:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.548+0000 7f82ddcd4640 1 -- 172.21.15.78:0/1582469063 wait complete. 2024-02-19T15:02:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.549+0000 7f82ddcd4640 1 Processor -- start 2024-02-19T15:02:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.549+0000 7f82ddcd4640 1 -- start start 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82ddcd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82d806b3c0 0x7f82d80a78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82ddcd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 0x7f82d8146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82ddcd4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d8150420 0x7f82d81470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82ddcd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8147630 con 0x7f82d806e590 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82ddcd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80aa2c0 con 0x7f82d806b3c0 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82ddcd4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d80aa5a0 con 0x7f82d8150420 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 0x7f82d8146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 0x7f82d8146bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40868/0 (socket says 172.21.15.78:40868) 2024-02-19T15:02:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 -- 172.21.15.78:0/979643420 learned_addr learned my addr 172.21.15.78:0/979643420 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d77fe640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82d806b3c0 0x7f82d80a78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d7fff640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d8150420 0x7f82d81470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 -- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d8150420 msgr2=0x7f82d81470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d8150420 0x7f82d81470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 -- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82d806b3c0 msgr2=0x7f82d80a78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82d806b3c0 0x7f82d80a78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 -- 172.21.15.78:0/979643420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0009a70 con 0x7f82d806e590 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82d6ffd640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 0x7f82d8146bb0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00e980 tx=0x7f82cc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc009670 con 0x7f82d806e590 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.550+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc00ca00 con 0x7f82d806e590 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.551+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d814a6c0 con 0x7f82d806e590 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.551+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00cd90 con 0x7f82d806e590 2024-02-19T15:02:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.551+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d814ac80 con 0x7f82d806e590 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.552+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d8074990 con 0x7f82d806e590 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.552+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f82cc013470 con 0x7f82d806e590 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.553+0000 7f82bffff640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f82c406b810 0x7f82c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.553+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f82cc08a860 con 0x7f82d806e590 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.553+0000 7f82d77fe640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f82c406b810 0x7f82c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.553+0000 7f82d77fe640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f82c406b810 0x7f82c406dcd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f82c8007fd0 tx=0x7f82c8007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.557+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f82cc055130 con 0x7f82d806e590 2024-02-19T15:02:48.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.763+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82d80a5e50 con 0x7f82d806e590 2024-02-19T15:02:48.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.763+0000 7f82bffff640 1 -- 172.21.15.78:0/979643420 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f82cc021d90 con 0x7f82d806e590 2024-02-19T15:02:48.766 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:48.766 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:48.766 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:48.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f82c406b810 msgr2=0x7f82c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:48.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f82c406b810 0x7f82c406dcd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f82c8007fd0 tx=0x7f82c8007de0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 msgr2=0x7f82d8146bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:48.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 0x7f82d8146bb0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00e980 tx=0x7f82cc00ee40 comp rx=0 tx=0).stop 2024-02-19T15:02:48.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 shutdown_connections 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f82c406b810 0x7f82c406dcd0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d806e590 0x7f82d8146bb0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82d8150420 0x7f82d81470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 --2- 172.21.15.78:0/979643420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82d806b3c0 0x7f82d80a78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 >> 172.21.15.78:0/979643420 conn(0x7f82d806d9e0 msgr2=0x7f82d81529f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 shutdown_connections 2024-02-19T15:02:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:48.765+0000 7f82ddcd4640 1 -- 172.21.15.78:0/979643420 wait complete. 2024-02-19T15:02:49.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:49.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:50.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:50.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.291+0000 7f3b590c9640 1 Processor -- start 2024-02-19T15:02:50.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.291+0000 7f3b590c9640 1 -- start start 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.291+0000 7f3b590c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3b54152830 0x7f3b540c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.291+0000 7f3b590c9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b541586a0 con 0x7f3b540c7990 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.291+0000 7f3b590c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b54158980 con 0x7f3b540c5760 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.291+0000 7f3b590c9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b540c6550 con 0x7f3b54152830 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.292+0000 7f3b52d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3b54152830 0x7f3b540c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.292+0000 7f3b5359e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3b540c7990 0x7f3b5406c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55226/0 (socket says 172.21.15.78:55226) 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.292+0000 7f3b5359e640 1 -- 172.21.15.78:0/3516774262 learned_addr learned my addr 172.21.15.78:0/3516774262 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.292+0000 7f3b52d9d640 1 -- 172.21.15.78:0/3516774262 >> v1:172.21.15.150:6789/0 conn(0x7f3b540c5760 legacy=0x7f3b540c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.292+0000 7f3b52d9d640 1 -- 172.21.15.78:0/3516774262 >> v1:172.21.15.78:6789/0 conn(0x7f3b540c7990 legacy=0x7f3b5406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.292+0000 7f3b52d9d640 1 -- 172.21.15.78:0/3516774262 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54158c60 con 0x7f3b54152830 2024-02-19T15:02:50.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b52d9d640 1 --2- 172.21.15.78:0/3516774262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54152830 0x7f3b540c5220 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0097a0 tx=0x7f3b3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7a990d37c4f4ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b51d9b640 1 -- 172.21.15.78:0/3516774262 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c031830 con 0x7f3b54152830 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b51d9b640 1 -- 172.21.15.78:0/3516774262 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c031990 con 0x7f3b54152830 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b51d9b640 1 -- 172.21.15.78:0/3516774262 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c031d20 con 0x7f3b54152830 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 -- 172.21.15.78:0/3516774262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54152830 msgr2=0x7f3b540c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3516774262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54152830 0x7f3b540c5220 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0097a0 tx=0x7f3b3c02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 -- 172.21.15.78:0/3516774262 shutdown_connections 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3516774262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54152830 0x7f3b540c5220 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 -- 172.21.15.78:0/3516774262 >> 172.21.15.78:0/3516774262 conn(0x7f3b540ca8a0 msgr2=0x7f3b540cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 -- 172.21.15.78:0/3516774262 shutdown_connections 2024-02-19T15:02:50.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.293+0000 7f3b590c9640 1 -- 172.21.15.78:0/3516774262 wait complete. 2024-02-19T15:02:50.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.295+0000 7f3b590c9640 1 Processor -- start 2024-02-19T15:02:50.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.295+0000 7f3b590c9640 1 -- start start 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b540c5760 0x7f3b54155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b54152830 0x7f3b540c2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 0x7f3b541ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54159320 con 0x7f3b540c42d0 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54157450 con 0x7f3b54152830 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54157cb0 con 0x7f3b540c5760 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 0x7f3b541ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 0x7f3b541ee040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40894/0 (socket says 172.21.15.78:40894) 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 -- 172.21.15.78:0/3370815937 learned_addr learned my addr 172.21.15.78:0/3370815937 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b52d9d640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b540c5760 0x7f3b54155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5259c640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b54152830 0x7f3b540c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:50.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 -- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b540c5760 msgr2=0x7f3b54155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b540c5760 0x7f3b54155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 -- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b54152830 msgr2=0x7f3b540c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b54152830 0x7f3b540c2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 -- 172.21.15.78:0/3370815937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c009400 con 0x7f3b540c42d0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b5359e640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 0x7f3b541ee040 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00cb20 tx=0x7f3b4c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00ac40 con 0x7f3b540c42d0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c00ada0 con 0x7f3b540c42d0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b540c4860 con 0x7f3b540c42d0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c012600 con 0x7f3b540c42d0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.296+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b54157940 con 0x7f3b540c42d0 2024-02-19T15:02:50.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.297+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b54074990 con 0x7f3b540c42d0 2024-02-19T15:02:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.298+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3b4c012760 con 0x7f3b540c42d0 2024-02-19T15:02:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.298+0000 7f3b4b7fe640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b3406b810 0x7f3b3406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.298+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3b4c08a2a0 con 0x7f3b540c42d0 2024-02-19T15:02:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.298+0000 7f3b52d9d640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b3406b810 0x7f3b3406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:50.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.299+0000 7f3b52d9d640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b3406b810 0x7f3b3406dcd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c036000 tx=0x7f3b3c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:50.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.303+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3b4c054ae0 con 0x7f3b540c42d0 2024-02-19T15:02:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.511+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b54157df0 con 0x7f3b540c42d0 2024-02-19T15:02:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.511+0000 7f3b4b7fe640 1 -- 172.21.15.78:0/3370815937 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f3b4c017d90 con 0x7f3b540c42d0 2024-02-19T15:02:50.514 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:50.515 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b3406b810 msgr2=0x7f3b3406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b3406b810 0x7f3b3406dcd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c036000 tx=0x7f3b3c03c040 comp rx=0 tx=0).stop 2024-02-19T15:02:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 msgr2=0x7f3b541ee040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 0x7f3b541ee040 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00cb20 tx=0x7f3b4c00a060 comp rx=0 tx=0).stop 2024-02-19T15:02:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 shutdown_connections 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3b3406b810 0x7f3b3406dcd0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b540c42d0 0x7f3b541ee040 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b54152830 0x7f3b540c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 --2- 172.21.15.78:0/3370815937 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b540c5760 0x7f3b54155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 >> 172.21.15.78:0/3370815937 conn(0x7f3b540ca8a0 msgr2=0x7f3b540cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 shutdown_connections 2024-02-19T15:02:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:50.513+0000 7f3b590c9640 1 -- 172.21.15.78:0/3370815937 wait complete. 2024-02-19T15:02:50.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:50.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:51.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 Processor -- start 2024-02-19T15:02:52.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 -- start start 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f614806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f614806d910 0x7f614806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614806cd20 con 0x7f6148152700 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61481551f0 con 0x7f614806d910 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.041+0000 7f614ecae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6148155490 con 0x7f61480c8e00 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f614ca4a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6148152700 0x7f61480c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55238/0 (socket says 172.21.15.78:55238) 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f614ca4a640 1 -- 172.21.15.78:0/1396277048 learned_addr learned my addr 172.21.15.78:0/1396277048 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f6147fff640 1 --2- 172.21.15.78:0/1396277048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f614806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f614d24b640 1 --2- 172.21.15.78:0/1396277048 >> v2:172.21.15.150:3300/0 conn(0x7f614806d910 0x7f614806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:52.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f61477fe640 1 -- 172.21.15.78:0/1396277048 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935591794 0 0) 0x7f614806cd20 con 0x7f6148152700 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f61477fe640 1 -- 172.21.15.78:0/1396277048 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6138003520 con 0x7f6148152700 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f6147fff640 1 -- 172.21.15.78:0/1396277048 >> v2:172.21.15.150:3300/0 conn(0x7f614806d910 msgr2=0x7f614806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f6147fff640 1 --2- 172.21.15.78:0/1396277048 >> v2:172.21.15.150:3300/0 conn(0x7f614806d910 0x7f614806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f6147fff640 1 -- 172.21.15.78:0/1396277048 >> v1:172.21.15.78:6789/0 conn(0x7f6148152700 legacy=0x7f61480c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f6147fff640 1 -- 172.21.15.78:0/1396277048 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148155770 con 0x7f61480c8e00 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.042+0000 7f614d24b640 1 --2- 172.21.15.78:0/1396277048 >> v2:172.21.15.150:3300/0 conn(0x7f614806d910 0x7f614806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.043+0000 7f6147fff640 1 --2- 172.21.15.78:0/1396277048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f614806c2a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f61340097a0 tx=0x7f613402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f95b6491d8f6455 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.043+0000 7f61477fe640 1 -- 172.21.15.78:0/1396277048 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613402e070 con 0x7f61480c8e00 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.043+0000 7f61477fe640 1 -- 172.21.15.78:0/1396277048 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61340315a0 con 0x7f61480c8e00 2024-02-19T15:02:52.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.043+0000 7f61477fe640 1 -- 172.21.15.78:0/1396277048 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134031930 con 0x7f61480c8e00 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.043+0000 7f614ecae640 1 -- 172.21.15.78:0/1396277048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 msgr2=0x7f614806c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.043+0000 7f614ecae640 1 --2- 172.21.15.78:0/1396277048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f614806c2a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f61340097a0 tx=0x7f613402b840 comp rx=0 tx=0).stop 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.044+0000 7f614ecae640 1 -- 172.21.15.78:0/1396277048 shutdown_connections 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.044+0000 7f614ecae640 1 --2- 172.21.15.78:0/1396277048 >> v2:172.21.15.150:3300/0 conn(0x7f614806d910 0x7f614806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.044+0000 7f614ecae640 1 --2- 172.21.15.78:0/1396277048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f614806c2a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.044+0000 7f614ecae640 1 -- 172.21.15.78:0/1396277048 >> 172.21.15.78:0/1396277048 conn(0x7f61480cac90 msgr2=0x7f61480cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.044+0000 7f614ecae640 1 -- 172.21.15.78:0/1396277048 shutdown_connections 2024-02-19T15:02:52.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.044+0000 7f614ecae640 1 -- 172.21.15.78:0/1396277048 wait complete. 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.045+0000 7f614ecae640 1 Processor -- start 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.045+0000 7f614ecae640 1 -- start start 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 0x7f6148154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f61480c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6148152700 0x7f6148147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61480c4640 con 0x7f614806d910 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61481ee550 con 0x7f6148152700 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61481eea80 con 0x7f61480c8e00 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 0x7f6148154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:52.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 0x7f6148154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40902/0 (socket says 172.21.15.78:40902) 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 -- 172.21.15.78:0/845013440 learned_addr learned my addr 172.21.15.78:0/845013440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614d24b640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6148152700 0x7f6148147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f6147fff640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f61480c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 -- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 msgr2=0x7f61480c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f61480c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 -- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6148152700 msgr2=0x7f6148147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6148152700 0x7f6148147d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 -- 172.21.15.78:0/845013440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6140009a70 con 0x7f614806d910 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ca4a640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 0x7f6148154210 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f613c00c3c0 tx=0x7f613c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c005ce0 con 0x7f614806d910 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613c005e40 con 0x7f614806d910 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6134009400 con 0x7f614806d910 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613c0072f0 con 0x7f614806d910 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.046+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61480c42d0 con 0x7f614806d910 2024-02-19T15:02:52.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.047+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61480c4110 con 0x7f614806d910 2024-02-19T15:02:52.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.048+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f613c007450 con 0x7f614806d910 2024-02-19T15:02:52.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.048+0000 7f61457fa640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f612c06b840 0x7f612c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:52.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.048+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f613c0897b0 con 0x7f614806d910 2024-02-19T15:02:52.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.048+0000 7f6147fff640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f612c06b840 0x7f612c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:52.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.049+0000 7f6147fff640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f612c06b840 0x7f612c06dd00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6134036000 tx=0x7f613403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:52.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.053+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f613c054080 con 0x7f614806d910 2024-02-19T15:02:52.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.260+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6148074990 con 0x7f614806d910 2024-02-19T15:02:52.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.261+0000 7f61457fa640 1 -- 172.21.15.78:0/845013440 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f613c017d90 con 0x7f614806d910 2024-02-19T15:02:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:52.264 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:52.264 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:52.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f612c06b840 msgr2=0x7f612c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:52.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f612c06b840 0x7f612c06dd00 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6134036000 tx=0x7f613403d040 comp rx=0 tx=0).stop 2024-02-19T15:02:52.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 msgr2=0x7f6148154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:52.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 0x7f6148154210 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f613c00c3c0 tx=0x7f613c00c7c0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 shutdown_connections 2024-02-19T15:02:52.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f612c06b840 0x7f612c06dd00 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6148152700 0x7f6148147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61480c8e00 0x7f61480c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 --2- 172.21.15.78:0/845013440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614806d910 0x7f6148154210 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:52.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 >> 172.21.15.78:0/845013440 conn(0x7f61480cac90 msgr2=0x7f61480ca110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:52.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 shutdown_connections 2024-02-19T15:02:52.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:52.263+0000 7f614ecae640 1 -- 172.21.15.78:0/845013440 wait complete. 2024-02-19T15:02:52.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:52.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:53.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:53.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.788+0000 7f8f5030f640 1 Processor -- start 2024-02-19T15:02:53.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.788+0000 7f8f5030f640 1 -- start start 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.789+0000 7f8f5030f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8f480c78a0 0x7f8f48159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.789+0000 7f8f5030f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f48155010 0x7f8f481523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.789+0000 7f8f5030f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4806ffa0 con 0x7f8f480c4d80 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.789+0000 7f8f5030f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f4806b720 con 0x7f8f480c78a0 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.789+0000 7f8f5030f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4806fc70 con 0x7f8f48155010 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4d8aa640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8f480c4d80 0x7f8f480c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55252/0 (socket says 172.21.15.78:55252) 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4d8aa640 1 -- 172.21.15.78:0/1930176451 learned_addr learned my addr 172.21.15.78:0/1930176451 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e8ac640 1 --2- 172.21.15.78:0/1930176451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f48155010 0x7f8f481523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e0ab640 1 --2- 172.21.15.78:0/1930176451 >> v2:172.21.15.150:3300/0 conn(0x7f8f480c78a0 0x7f8f48159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:53.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4d0a9640 1 -- 172.21.15.78:0/1930176451 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916187305 0 0) 0x7f8f4806ffa0 con 0x7f8f480c4d80 2024-02-19T15:02:53.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4d0a9640 1 -- 172.21.15.78:0/1930176451 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34003520 con 0x7f8f480c4d80 2024-02-19T15:02:53.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e8ac640 1 -- 172.21.15.78:0/1930176451 >> v2:172.21.15.150:3300/0 conn(0x7f8f480c78a0 msgr2=0x7f8f48159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:53.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e8ac640 1 --2- 172.21.15.78:0/1930176451 >> v2:172.21.15.150:3300/0 conn(0x7f8f480c78a0 0x7f8f48159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:53.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e8ac640 1 -- 172.21.15.78:0/1930176451 >> v1:172.21.15.78:6789/0 conn(0x7f8f480c4d80 legacy=0x7f8f480c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:53.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e8ac640 1 -- 172.21.15.78:0/1930176451 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4806b930 con 0x7f8f48155010 2024-02-19T15:02:53.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.790+0000 7f8f4e0ab640 1 --2- 172.21.15.78:0/1930176451 >> v2:172.21.15.150:3300/0 conn(0x7f8f480c78a0 0x7f8f48159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.791+0000 7f8f4e8ac640 1 --2- 172.21.15.78:0/1930176451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f48155010 0x7f8f481523d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009d90 tx=0x7f8f4002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e15580dc63ab8cdd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.791+0000 7f8f4d0a9640 1 -- 172.21.15.78:0/1930176451 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4002cbc0 con 0x7f8f48155010 2024-02-19T15:02:53.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.791+0000 7f8f4d0a9640 1 -- 172.21.15.78:0/1930176451 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4002cd20 con 0x7f8f48155010 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.791+0000 7f8f4d0a9640 1 -- 172.21.15.78:0/1930176451 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40034660 con 0x7f8f48155010 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.791+0000 7f8f5030f640 1 -- 172.21.15.78:0/1930176451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f48155010 msgr2=0x7f8f481523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.791+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1930176451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f48155010 0x7f8f481523d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009d90 tx=0x7f8f4002be10 comp rx=0 tx=0).stop 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.792+0000 7f8f5030f640 1 -- 172.21.15.78:0/1930176451 shutdown_connections 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.792+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1930176451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f48155010 0x7f8f481523d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.792+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1930176451 >> v2:172.21.15.150:3300/0 conn(0x7f8f480c78a0 0x7f8f48159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.792+0000 7f8f5030f640 1 -- 172.21.15.78:0/1930176451 >> 172.21.15.78:0/1930176451 conn(0x7f8f48066e70 msgr2=0x7f8f480a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.792+0000 7f8f5030f640 1 -- 172.21.15.78:0/1930176451 shutdown_connections 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.792+0000 7f8f5030f640 1 -- 172.21.15.78:0/1930176451 wait complete. 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 Processor -- start 2024-02-19T15:02:53.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 -- start start 2024-02-19T15:02:53.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f480c4d80 0x7f8f480c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:53.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 0x7f8f48155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f48155010 0x7f8f481ea730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f481ed660 con 0x7f8f480c78a0 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480c3d10 con 0x7f8f48155010 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f5030f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480c3fd0 con 0x7f8f480c4d80 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f4d8aa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 0x7f8f48155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f4d8aa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 0x7f8f48155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40914/0 (socket says 172.21.15.78:40914) 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f4d8aa640 1 -- 172.21.15.78:0/1129118242 learned_addr learned my addr 172.21.15.78:0/1129118242 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f4e0ab640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f480c4d80 0x7f8f480c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.793+0000 7f8f4e8ac640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f48155010 0x7f8f481ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f4d8aa640 1 -- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f480c4d80 msgr2=0x7f8f480c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f4d8aa640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f480c4d80 0x7f8f480c1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f4d8aa640 1 -- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f48155010 msgr2=0x7f8f481ea730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f4d8aa640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f48155010 0x7f8f481ea730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:53.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f4d8aa640 1 -- 172.21.15.78:0/1129118242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f38009400 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f4d8aa640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 0x7f8f48155420 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400c390 tx=0x7f8f4400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44005ce0 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44005e40 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f40009a70 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440072f0 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.794+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f480c4590 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.795+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f481ed860 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.795+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8f44007450 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.796+0000 7f8f3effd640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f2c06b810 0x7f8f2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.796+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8f44089970 con 0x7f8f480c78a0 2024-02-19T15:02:53.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.796+0000 7f8f4e0ab640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f2c06b810 0x7f8f2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:53.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.796+0000 7f8f4e0ab640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f2c06b810 0x7f8f2c06dcd0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009c30 tx=0x7f8f38009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:53.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:53.800+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8f481ed860 con 0x7f8f480c78a0 2024-02-19T15:02:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.005+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f48074990 con 0x7f8f480c78a0 2024-02-19T15:02:54.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.006+0000 7f8f3effd640 1 -- 172.21.15.78:0/1129118242 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f8f44017090 con 0x7f8f480c78a0 2024-02-19T15:02:54.009 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:54.009 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:54.009 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:54.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f2c06b810 msgr2=0x7f8f2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:54.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f2c06b810 0x7f8f2c06dcd0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009c30 tx=0x7f8f38009190 comp rx=0 tx=0).stop 2024-02-19T15:02:54.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 msgr2=0x7f8f48155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:54.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 0x7f8f48155420 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400c390 tx=0x7f8f4400c850 comp rx=0 tx=0).stop 2024-02-19T15:02:54.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 shutdown_connections 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8f2c06b810 0x7f8f2c06dcd0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f480c78a0 0x7f8f48155420 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f48155010 0x7f8f481ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 --2- 172.21.15.78:0/1129118242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f480c4d80 0x7f8f480c1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 >> 172.21.15.78:0/1129118242 conn(0x7f8f48066e70 msgr2=0x7f8f480a88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 shutdown_connections 2024-02-19T15:02:54.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:54.008+0000 7f8f5030f640 1 -- 172.21.15.78:0/1129118242 wait complete. 2024-02-19T15:02:54.424 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:54.424 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:55.425 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:55.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.541+0000 7fdb9f847640 1 Processor -- start 2024-02-19T15:02:55.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.541+0000 7fdb9f847640 1 -- start start 2024-02-19T15:02:55.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9f847640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb980c5130 0x7fdb980c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9f847640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 0x7fdb980c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9f847640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb981559c0 0x7fdb980c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9f847640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb98158b60 con 0x7fdb98158f80 2024-02-19T15:02:55.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9f847640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb980c3cf0 con 0x7fdb981559c0 2024-02-19T15:02:55.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9f847640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb98159400 con 0x7fdb980c5130 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9cde2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 0x7fdb980c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9d5e3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb980c5130 0x7fdb980c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9dde4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb981559c0 0x7fdb980c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9cde2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 0x7fdb980c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40920/0 (socket says 172.21.15.78:40920) 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9cde2640 1 -- 172.21.15.78:0/4139598531 learned_addr learned my addr 172.21.15.78:0/4139598531 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9d5e3640 1 -- 172.21.15.78:0/4139598531 >> v2:172.21.15.150:3300/0 conn(0x7fdb981559c0 msgr2=0x7fdb980c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9d5e3640 1 --2- 172.21.15.78:0/4139598531 >> v2:172.21.15.150:3300/0 conn(0x7fdb981559c0 0x7fdb980c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9d5e3640 1 -- 172.21.15.78:0/4139598531 >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 msgr2=0x7fdb980c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9d5e3640 1 --2- 172.21.15.78:0/4139598531 >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 0x7fdb980c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9d5e3640 1 -- 172.21.15.78:0/4139598531 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98158530 con 0x7fdb980c5130 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9cde2640 1 --2- 172.21.15.78:0/4139598531 >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 0x7fdb980c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:55.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.542+0000 7fdb9dde4640 1 --2- 172.21.15.78:0/4139598531 >> v2:172.21.15.150:3300/0 conn(0x7fdb981559c0 0x7fdb980c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.543+0000 7fdb9d5e3640 1 --2- 172.21.15.78:0/4139598531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb980c5130 0x7fdb980c30a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdb980732c0 tx=0x7fdb9002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a01cdf870ec0fa9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.543+0000 7fdb97fff640 1 -- 172.21.15.78:0/4139598531 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9002cbd0 con 0x7fdb980c5130 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.543+0000 7fdb97fff640 1 -- 172.21.15.78:0/4139598531 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9002cd30 con 0x7fdb980c5130 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.543+0000 7fdb97fff640 1 -- 172.21.15.78:0/4139598531 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9002cea0 con 0x7fdb980c5130 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 -- 172.21.15.78:0/4139598531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb980c5130 msgr2=0x7fdb980c30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 --2- 172.21.15.78:0/4139598531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb980c5130 0x7fdb980c30a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdb980732c0 tx=0x7fdb9002be20 comp rx=0 tx=0).stop 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 -- 172.21.15.78:0/4139598531 shutdown_connections 2024-02-19T15:02:55.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 --2- 172.21.15.78:0/4139598531 >> v2:172.21.15.150:3300/0 conn(0x7fdb981559c0 0x7fdb980c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 --2- 172.21.15.78:0/4139598531 >> v2:172.21.15.78:3300/0 conn(0x7fdb98158f80 0x7fdb980c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 --2- 172.21.15.78:0/4139598531 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb980c5130 0x7fdb980c30a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 -- 172.21.15.78:0/4139598531 >> 172.21.15.78:0/4139598531 conn(0x7fdb9806b4b0 msgr2=0x7fdb980c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 -- 172.21.15.78:0/4139598531 shutdown_connections 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.544+0000 7fdb9f847640 1 -- 172.21.15.78:0/4139598531 wait complete. 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.545+0000 7fdb9f847640 1 Processor -- start 2024-02-19T15:02:55.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.545+0000 7fdb9f847640 1 -- start start 2024-02-19T15:02:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9f847640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb980c5130 0x7fdb980a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9f847640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb981559c0 0x7fdb981eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9f847640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 0x7fdb98154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9f847640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb981eb820 con 0x7fdb98158f80 2024-02-19T15:02:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9f847640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980a9b20 con 0x7fdb980c5130 2024-02-19T15:02:55.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9f847640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980a9560 con 0x7fdb981559c0 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 0x7fdb98154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 0x7fdb98154130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40928/0 (socket says 172.21.15.78:40928) 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 -- 172.21.15.78:0/1566586469 learned_addr learned my addr 172.21.15.78:0/1566586469 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9cde2640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb981559c0 0x7fdb981eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9d5e3640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb980c5130 0x7fdb980a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 -- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb981559c0 msgr2=0x7fdb981eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb981559c0 0x7fdb981eb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 -- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb980c5130 msgr2=0x7fdb980a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb980c5130 0x7fdb980a8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.546+0000 7fdb9dde4640 1 -- 172.21.15.78:0/1566586469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88009400 con 0x7fdb98158f80 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.547+0000 7fdb9dde4640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 0x7fdb98154130 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800e960 tx=0x7fdb8800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.547+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88004020 con 0x7fdb98158f80 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.547+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb880041a0 con 0x7fdb98158f80 2024-02-19T15:02:55.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.547+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c0094d0 con 0x7fdb98158f80 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.547+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8800ca00 con 0x7fdb98158f80 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.547+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb90009a70 con 0x7fdb98158f80 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.548+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb98074990 con 0x7fdb98158f80 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.548+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdb8800cb60 con 0x7fdb98158f80 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.549+0000 7fdb95ffb640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb8406b810 0x7fdb8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.549+0000 7fdb9d5e3640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb8406b810 0x7fdb8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:55.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.549+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdb8808a5a0 con 0x7fdb98158f80 2024-02-19T15:02:55.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.549+0000 7fdb9d5e3640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb8406b810 0x7fdb8406dcd0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fdb90036000 tx=0x7fdb90030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:55.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.553+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdb88054e70 con 0x7fdb98158f80 2024-02-19T15:02:55.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.761+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb980c9400 con 0x7fdb98158f80 2024-02-19T15:02:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.762+0000 7fdb95ffb640 1 -- 172.21.15.78:0/1566586469 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fdb88020d90 con 0x7fdb98158f80 2024-02-19T15:02:55.765 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:55.765 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:55.765 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb8406b810 msgr2=0x7fdb8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb8406b810 0x7fdb8406dcd0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fdb90036000 tx=0x7fdb90030000 comp rx=0 tx=0).stop 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 msgr2=0x7fdb98154130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 0x7fdb98154130 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800e960 tx=0x7fdb8800ee20 comp rx=0 tx=0).stop 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 shutdown_connections 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb8406b810 0x7fdb8406dcd0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb98158f80 0x7fdb98154130 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb981559c0 0x7fdb981eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 --2- 172.21.15.78:0/1566586469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb980c5130 0x7fdb980a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 >> 172.21.15.78:0/1566586469 conn(0x7fdb9806b4b0 msgr2=0x7fdb9806bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 shutdown_connections 2024-02-19T15:02:55.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:55.765+0000 7fdb9f847640 1 -- 172.21.15.78:0/1566586469 wait complete. 2024-02-19T15:02:56.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:56.182 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:57.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:57.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 Processor -- start 2024-02-19T15:02:57.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 -- start start 2024-02-19T15:02:57.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f99a8152700 0x7f99a80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99a80c8e00 0x7f99a815f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a81551f0 con 0x7f99a80c5730 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a8155490 con 0x7f99a8152700 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.293+0000 7f99aceeb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a8155770 con 0x7f99a80c8e00 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a659c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f99a8152700 0x7f99a80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f999ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f99a80c5730 0x7f99a806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55262/0 (socket says 172.21.15.78:55262) 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f999ffff640 1 -- 172.21.15.78:0/3420715382 learned_addr learned my addr 172.21.15.78:0/3420715382 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a6d9d640 1 --2- 172.21.15.78:0/3420715382 >> v2:172.21.15.150:3300/0 conn(0x7f99a80c8e00 0x7f99a815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a659c640 1 -- 172.21.15.78:0/3420715382 >> v2:172.21.15.150:3300/0 conn(0x7f99a80c8e00 msgr2=0x7f99a815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a659c640 1 --2- 172.21.15.78:0/3420715382 >> v2:172.21.15.150:3300/0 conn(0x7f99a80c8e00 0x7f99a815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a659c640 1 -- 172.21.15.78:0/3420715382 >> v1:172.21.15.78:6789/0 conn(0x7f99a80c5730 legacy=0x7f99a806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:57.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a5d9b640 1 -- 172.21.15.78:0/3420715382 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025963205 0 0) 0x7f99a81551f0 con 0x7f99a80c5730 2024-02-19T15:02:57.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.294+0000 7f99a659c640 1 -- 172.21.15.78:0/3420715382 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8155a50 con 0x7f99a8152700 2024-02-19T15:02:57.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99a659c640 1 --2- 172.21.15.78:0/3420715382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a8152700 0x7f99a80c51f0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f99a81596f0 tx=0x7f999802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73a452b484110ce server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:57.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99a5d9b640 1 -- 172.21.15.78:0/3420715382 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999802e070 con 0x7f99a8152700 2024-02-19T15:02:57.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99a5d9b640 1 -- 172.21.15.78:0/3420715382 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99980315a0 con 0x7f99a8152700 2024-02-19T15:02:57.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99a5d9b640 1 -- 172.21.15.78:0/3420715382 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998031930 con 0x7f99a8152700 2024-02-19T15:02:57.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 -- 172.21.15.78:0/3420715382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a8152700 msgr2=0x7f99a80c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3420715382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a8152700 0x7f99a80c51f0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f99a81596f0 tx=0x7f999802b840 comp rx=0 tx=0).stop 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 -- 172.21.15.78:0/3420715382 shutdown_connections 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3420715382 >> v2:172.21.15.150:3300/0 conn(0x7f99a80c8e00 0x7f99a815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3420715382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a8152700 0x7f99a80c51f0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 -- 172.21.15.78:0/3420715382 >> 172.21.15.78:0/3420715382 conn(0x7f99a80cac90 msgr2=0x7f99a80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 -- 172.21.15.78:0/3420715382 shutdown_connections 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.295+0000 7f99aceeb640 1 -- 172.21.15.78:0/3420715382 wait complete. 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 Processor -- start 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 -- start start 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 0x7f99a80c3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80c8e00 0x7f99a80c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99a8152700 0x7f99a81ea610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a81497d0 con 0x7f99a80c5730 2024-02-19T15:02:57.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80c1cb0 con 0x7f99a80c8e00 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.297+0000 7f99aceeb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80c1f90 con 0x7f99a8152700 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 0x7f99a80c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 0x7f99a80c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40942/0 (socket says 172.21.15.78:40942) 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 -- 172.21.15.78:0/3908386385 learned_addr learned my addr 172.21.15.78:0/3908386385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f999ffff640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80c8e00 0x7f99a80c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a6d9d640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99a8152700 0x7f99a81ea610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 -- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99a8152700 msgr2=0x7f99a81ea610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99a8152700 0x7f99a81ea610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 -- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80c8e00 msgr2=0x7f99a80c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80c8e00 0x7f99a80c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 -- 172.21.15.78:0/3908386385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9998009400 con 0x7f99a80c5730 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99a659c640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 0x7f99a80c3560 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f9998000c00 tx=0x7f99980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998032840 con 0x7f99a80c5730 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99980329c0 con 0x7f99a80c5730 2024-02-19T15:02:57.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a80c0ba0 con 0x7f99a80c5730 2024-02-19T15:02:57.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998032d70 con 0x7f99a80c5730 2024-02-19T15:02:57.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.298+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a80c1160 con 0x7f99a80c5730 2024-02-19T15:02:57.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.299+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a8074990 con 0x7f99a80c5730 2024-02-19T15:02:57.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.300+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f999804d3f0 con 0x7f99a80c5730 2024-02-19T15:02:57.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.300+0000 7f999effd640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f998806b840 0x7f998806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:57.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.300+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f99980abdb0 con 0x7f99a80c5730 2024-02-19T15:02:57.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.300+0000 7f999ffff640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f998806b840 0x7f998806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:57.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.301+0000 7f999ffff640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f998806b840 0x7f998806dd00 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f99a80b0670 tx=0x7f9990007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:57.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.305+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9998076680 con 0x7f99a80c5730 2024-02-19T15:02:57.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.515+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99a80c1600 con 0x7f99a80c5730 2024-02-19T15:02:57.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.515+0000 7f999effd640 1 -- 172.21.15.78:0/3908386385 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f9998079d40 con 0x7f99a80c5730 2024-02-19T15:02:57.519 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:57.519 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:57.520 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:57.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f998806b840 msgr2=0x7f998806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:57.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f998806b840 0x7f998806dd00 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f99a80b0670 tx=0x7f9990007d50 comp rx=0 tx=0).stop 2024-02-19T15:02:57.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 msgr2=0x7f99a80c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:57.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 0x7f99a80c3560 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f9998000c00 tx=0x7f99980046c0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 shutdown_connections 2024-02-19T15:02:57.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f998806b840 0x7f998806dd00 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99a8152700 0x7f99a81ea610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80c8e00 0x7f99a80c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 --2- 172.21.15.78:0/3908386385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99a80c5730 0x7f99a80c3560 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:57.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 >> 172.21.15.78:0/3908386385 conn(0x7f99a80cac90 msgr2=0x7f99a814e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:57.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 shutdown_connections 2024-02-19T15:02:57.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:57.517+0000 7f99aceeb640 1 -- 172.21.15.78:0/3908386385 wait complete. 2024-02-19T15:02:57.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:57.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:02:58.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:02:58.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:02:58.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:02:59.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.041+0000 7fecb1777640 1 Processor -- start 2024-02-19T15:02:59.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.041+0000 7fecb1777640 1 -- start start 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecb1777640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fecac0c8e00 0x7fecac06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecb1777640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac1551f0 con 0x7fecac06d910 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecb1777640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac155490 con 0x7fecac152700 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecb1777640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac155770 con 0x7fecac0c8e00 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecaa7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fecac0c8e00 0x7fecac06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecab7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fecac06d910 0x7fecac15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52322/0 (socket says 172.21.15.78:52322) 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecab7fe640 1 -- 172.21.15.78:0/1252375431 learned_addr learned my addr 172.21.15.78:0/1252375431 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:02:59.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecaa7fc640 1 -- 172.21.15.78:0/1252375431 >> v1:172.21.15.150:6789/0 conn(0x7fecac152700 legacy=0x7fecac0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecaa7fc640 1 -- 172.21.15.78:0/1252375431 >> v1:172.21.15.78:6789/0 conn(0x7fecac06d910 legacy=0x7fecac15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7feca9ffb640 1 -- 172.21.15.78:0/1252375431 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460030735 0 0) 0x7fecac1551f0 con 0x7fecac06d910 2024-02-19T15:02:59.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.042+0000 7fecaa7fc640 1 -- 172.21.15.78:0/1252375431 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac155a50 con 0x7fecac0c8e00 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.043+0000 7fecaa7fc640 1 --2- 172.21.15.78:0/1252375431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac06c2a0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7feca00097a0 tx=0x7feca002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e5a1af6e1e41b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.043+0000 7feca9ffb640 1 -- 172.21.15.78:0/1252375431 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0030610 con 0x7fecac0c8e00 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.043+0000 7feca9ffb640 1 -- 172.21.15.78:0/1252375431 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca0030770 con 0x7fecac0c8e00 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.043+0000 7feca9ffb640 1 -- 172.21.15.78:0/1252375431 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0030b00 con 0x7fecac0c8e00 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 -- 172.21.15.78:0/1252375431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 msgr2=0x7fecac06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 --2- 172.21.15.78:0/1252375431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac06c2a0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7feca00097a0 tx=0x7feca002b960 comp rx=0 tx=0).stop 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 -- 172.21.15.78:0/1252375431 shutdown_connections 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 --2- 172.21.15.78:0/1252375431 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac06c2a0 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 -- 172.21.15.78:0/1252375431 >> 172.21.15.78:0/1252375431 conn(0x7fecac0cac90 msgr2=0x7fecac0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:59.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 -- 172.21.15.78:0/1252375431 shutdown_connections 2024-02-19T15:02:59.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.044+0000 7fecb1777640 1 -- 172.21.15.78:0/1252375431 wait complete. 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.045+0000 7fecb1777640 1 Processor -- start 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.045+0000 7fecb1777640 1 -- start start 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecac06d910 0x7fecac0c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac0ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac152700 0x7fecac147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0aa280 con 0x7fecac0c8e00 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0acdf0 con 0x7fecac152700 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0ad0d0 con 0x7fecac06d910 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac0ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac0ab020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35620/0 (socket says 172.21.15.78:35620) 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 -- 172.21.15.78:0/3304307238 learned_addr learned my addr 172.21.15.78:0/3304307238 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaaffd640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecac06d910 0x7fecac0c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:59.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecab7fe640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac152700 0x7fecac147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 -- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecac06d910 msgr2=0x7fecac0c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecac06d910 0x7fecac0c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 -- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac152700 msgr2=0x7fecac147cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac152700 0x7fecac147cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 -- 172.21.15.78:0/3304307238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0009400 con 0x7fecac0c8e00 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecab7fe640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac152700 0x7fecac147cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecaa7fc640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac0ab020 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7feca00098c0 tx=0x7feca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca003e040 con 0x7fecac0c8e00 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca0039cb0 con 0x7fecac0c8e00 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0044630 con 0x7fecac0c8e00 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.046+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac0aa490 con 0x7fecac0c8e00 2024-02-19T15:02:59.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.047+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac0ab560 con 0x7fecac0c8e00 2024-02-19T15:02:59.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.048+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecac0aa9b0 con 0x7fecac0c8e00 2024-02-19T15:02:59.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.048+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7feca004d3b0 con 0x7fecac0c8e00 2024-02-19T15:02:59.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.048+0000 7fec937fe640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec9806b810 0x7fec9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:02:59.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.048+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7feca00ac8f0 con 0x7fecac0c8e00 2024-02-19T15:02:59.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.048+0000 7fecaaffd640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec9806b810 0x7fec9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:02:59.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.049+0000 7fecaaffd640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec9806b810 0x7fec9806dcd0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000f60 tx=0x7fec9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:02:59.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.053+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feca00771c0 con 0x7fecac0c8e00 2024-02-19T15:02:59.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.260+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecac074990 con 0x7fecac0c8e00 2024-02-19T15:02:59.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.261+0000 7fec937fe640 1 -- 172.21.15.78:0/3304307238 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7feca0034d90 con 0x7fecac0c8e00 2024-02-19T15:02:59.264 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:02:59.264 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:02:59.264 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec9806b810 msgr2=0x7fec9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec9806b810 0x7fec9806dcd0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000f60 tx=0x7fec9c009040 comp rx=0 tx=0).stop 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 msgr2=0x7fecac0ab020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac0ab020 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7feca00098c0 tx=0x7feca0004730 comp rx=0 tx=0).stop 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 shutdown_connections 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fec9806b810 0x7fec9806dcd0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecac0c8e00 0x7fecac0ab020 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecac152700 0x7fecac147cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 --2- 172.21.15.78:0/3304307238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecac06d910 0x7fecac0c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 >> 172.21.15.78:0/3304307238 conn(0x7fecac0cac90 msgr2=0x7fecac06d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 shutdown_connections 2024-02-19T15:02:59.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:02:59.263+0000 7fecb1777640 1 -- 172.21.15.78:0/3304307238 wait complete. 2024-02-19T15:02:59.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:02:59.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:00.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:00.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.787+0000 7f79d296c640 1 Processor -- start 2024-02-19T15:03:00.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.787+0000 7f79d296c640 1 -- start start 2024-02-19T15:03:00.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d296c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c42e0 0x7f79cc158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d296c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f79cc0c2f50 0x7f79cc0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d296c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79cc0c34a0 con 0x7f79cc0c8500 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d296c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79cc06c090 con 0x7f79cc0c2f50 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d296c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06d6a0 con 0x7f79cc0c42e0 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d0f09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f79cc0c2f50 0x7f79cc0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79cbfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f79cc0c8500 0x7f79cc158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52328/0 (socket says 172.21.15.78:52328) 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79cbfff640 1 -- 172.21.15.78:0/2466220525 learned_addr learned my addr 172.21.15.78:0/2466220525 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79cb7fe640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c42e0 0x7f79cc158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d0f09640 1 -- 172.21.15.78:0/2466220525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c42e0 msgr2=0x7f79cc158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d0f09640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c42e0 0x7f79cc158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d0f09640 1 -- 172.21.15.78:0/2466220525 >> v1:172.21.15.78:6789/0 conn(0x7f79cc0c8500 legacy=0x7f79cc158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79d0f09640 1 -- 172.21.15.78:0/2466220525 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc06c8e0 con 0x7f79cc0c2f50 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79caffd640 1 -- 172.21.15.78:0/2466220525 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912867940 0 0) 0x7f79cc0c34a0 con 0x7f79cc0c8500 2024-02-19T15:03:00.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.788+0000 7f79cb7fe640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c42e0 0x7f79cc158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.789+0000 7f79d0f09640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c2f50 0x7f79cc0c88e0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009d90 tx=0x7f79c402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2040845857d7df server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.789+0000 7f79caffd640 1 -- 172.21.15.78:0/2466220525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c402cbc0 con 0x7f79cc0c2f50 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.789+0000 7f79caffd640 1 -- 172.21.15.78:0/2466220525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c402cd20 con 0x7f79cc0c2f50 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.789+0000 7f79caffd640 1 -- 172.21.15.78:0/2466220525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4034660 con 0x7f79cc0c2f50 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 -- 172.21.15.78:0/2466220525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c2f50 msgr2=0x7f79cc0c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c2f50 0x7f79cc0c88e0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009d90 tx=0x7f79c402be10 comp rx=0 tx=0).stop 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 -- 172.21.15.78:0/2466220525 shutdown_connections 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c2f50 0x7f79cc0c88e0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 --2- 172.21.15.78:0/2466220525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c42e0 0x7f79cc158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 -- 172.21.15.78:0/2466220525 >> 172.21.15.78:0/2466220525 conn(0x7f79cc0aa300 msgr2=0x7f79cc0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 -- 172.21.15.78:0/2466220525 shutdown_connections 2024-02-19T15:03:00.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.790+0000 7f79d296c640 1 -- 172.21.15.78:0/2466220525 wait complete. 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.791+0000 7f79d296c640 1 Processor -- start 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.791+0000 7f79d296c640 1 -- start start 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d296c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c2f50 0x7f79cc070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d296c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 0x7f79cc1eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d296c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0ad280 0x7f79cc14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d296c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06f330 con 0x7f79cc0c8500 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d296c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06f610 con 0x7f79cc0ad280 2024-02-19T15:03:00.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d296c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc06f8f0 con 0x7f79cc0c2f50 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 0x7f79cc1eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 0x7f79cc1eab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35636/0 (socket says 172.21.15.78:35636) 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 -- 172.21.15.78:0/1625293348 learned_addr learned my addr 172.21.15.78:0/1625293348 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cbfff640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c2f50 0x7f79cc070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79d0f09640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0ad280 0x7f79cc14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 -- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c2f50 msgr2=0x7f79cc070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c2f50 0x7f79cc070610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 -- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0ad280 msgr2=0x7f79cc14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0ad280 0x7f79cc14da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 -- 172.21.15.78:0/1625293348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c4009a70 con 0x7f79cc0c8500 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79cb7fe640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 0x7f79cc1eab10 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f79c000e730 tx=0x7f79c000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c0004230 con 0x7f79cc0c8500 2024-02-19T15:03:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.792+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c0004390 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.793+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c000c8f0 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.793+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79cc1ee080 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.793+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79cc14df80 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.793+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79cc071c20 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.794+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f79c000ca50 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.794+0000 7f79c8ff9640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f79b806b840 0x7f79b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.794+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f79c0089f40 con 0x7f79cc0c8500 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.794+0000 7f79cbfff640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f79b806b840 0x7f79b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:00.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.795+0000 7f79cbfff640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f79b806b840 0x7f79b806dd00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f79bc007fd0 tx=0x7f79bc007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:00.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:00.799+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f79c0054810 con 0x7f79cc0c8500 2024-02-19T15:03:01.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.006+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79cc06f470 con 0x7f79cc0c8500 2024-02-19T15:03:01.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.007+0000 7f79c8ff9640 1 -- 172.21.15.78:0/1625293348 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f79cc06f470 con 0x7f79cc0c8500 2024-02-19T15:03:01.010 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:01.010 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:01.010 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f79b806b840 msgr2=0x7f79b806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f79b806b840 0x7f79b806dd00 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f79bc007fd0 tx=0x7f79bc007de0 comp rx=0 tx=0).stop 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 msgr2=0x7f79cc1eab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 0x7f79cc1eab10 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f79c000e730 tx=0x7f79c000ebf0 comp rx=0 tx=0).stop 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 shutdown_connections 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f79b806b840 0x7f79b806dd00 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:01.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79cc0c8500 0x7f79cc1eab10 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:01.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0ad280 0x7f79cc14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:01.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 --2- 172.21.15.78:0/1625293348 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79cc0c2f50 0x7f79cc070610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:01.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 >> 172.21.15.78:0/1625293348 conn(0x7f79cc0aa300 msgr2=0x7f79cc0c7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:01.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 shutdown_connections 2024-02-19T15:03:01.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:01.009+0000 7f79d296c640 1 -- 172.21.15.78:0/1625293348 wait complete. 2024-02-19T15:03:01.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:01.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:02.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d20a46640 1 Processor -- start 2024-02-19T15:03:02.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d20a46640 1 -- start start 2024-02-19T15:03:02.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d20a46640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7d1c0ac770 0x7f7d1c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d20a46640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c158c40 con 0x7f7d1c0bdf20 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d20a46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c0acbb0 con 0x7f7d1c0ac390 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d20a46640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d1c0c8de0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d1ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7d1c0ac770 0x7f7d1c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.536+0000 7f7d13fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7d1c0bdf20 0x7f7d1c15d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52342/0 (socket says 172.21.15.78:52342) 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.537+0000 7f7d13fff640 1 -- 172.21.15.78:0/1380028753 learned_addr learned my addr 172.21.15.78:0/1380028753 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.537+0000 7f7d19d9b640 1 -- 172.21.15.78:0/1380028753 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464676157 0 0) 0x7f7d1c0acbb0 con 0x7f7d1c0ac390 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.537+0000 7f7d1ad9d640 1 -- 172.21.15.78:0/1380028753 >> v1:172.21.15.150:6789/0 conn(0x7f7d1c0ac390 legacy=0x7f7d1c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.537+0000 7f7d1ad9d640 1 -- 172.21.15.78:0/1380028753 >> v1:172.21.15.78:6789/0 conn(0x7f7d1c0bdf20 legacy=0x7f7d1c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.537+0000 7f7d1ad9d640 1 -- 172.21.15.78:0/1380028753 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c155a60 con 0x7f7d1c0ac770 2024-02-19T15:03:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d1ad9d640 1 --2- 172.21.15.78:0/1380028753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c0c55f0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009d90 tx=0x7f7d1402bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf48a078951f7171 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d19d9b640 1 -- 172.21.15.78:0/1380028753 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1402c990 con 0x7f7d1c0ac770 2024-02-19T15:03:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d19d9b640 1 -- 172.21.15.78:0/1380028753 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1402caf0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d19d9b640 1 -- 172.21.15.78:0/1380028753 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1402ce80 con 0x7f7d1c0ac770 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 -- 172.21.15.78:0/1380028753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 msgr2=0x7f7d1c0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 --2- 172.21.15.78:0/1380028753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c0c55f0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009d90 tx=0x7f7d1402bc60 comp rx=0 tx=0).stop 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 -- 172.21.15.78:0/1380028753 shutdown_connections 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 --2- 172.21.15.78:0/1380028753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c0c55f0 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 -- 172.21.15.78:0/1380028753 >> 172.21.15.78:0/1380028753 conn(0x7f7d1c0ca890 msgr2=0x7f7d1c0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 -- 172.21.15.78:0/1380028753 shutdown_connections 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.538+0000 7f7d20a46640 1 -- 172.21.15.78:0/1380028753 wait complete. 2024-02-19T15:03:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 Processor -- start 2024-02-19T15:03:02.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 -- start start 2024-02-19T15:03:02.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d1c0ac390 0x7f7d1c1ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:02.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:02.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d1c0bdf20 0x7f7d1c0c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:02.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0af170 con 0x7f7d1c0ac770 2024-02-19T15:03:02.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0af410 con 0x7f7d1c0bdf20 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d20a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0af6f0 con 0x7f7d1c0ac390 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.540+0000 7f7d13fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35652/0 (socket says 172.21.15.78:35652) 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 -- 172.21.15.78:0/32948839 learned_addr learned my addr 172.21.15.78:0/32948839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d1ad9d640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d1c0bdf20 0x7f7d1c0c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d1a59c640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d1c0ac390 0x7f7d1c1ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 -- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d1c0ac390 msgr2=0x7f7d1c1ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d1c0ac390 0x7f7d1c1ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 -- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d1c0bdf20 msgr2=0x7f7d1c0c2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d1c0bdf20 0x7f7d1c0c2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 -- 172.21.15.78:0/32948839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d14009a70 con 0x7f7d1c0ac770 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d13fff640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c151450 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400c780 tx=0x7f7d0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040072d0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d04007450 con 0x7f7d1c0ac770 2024-02-19T15:03:02.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040047e0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1c0c22f0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.541+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1c1eed70 con 0x7f7d1c0ac770 2024-02-19T15:03:02.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.542+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d1c074990 con 0x7f7d1c0ac770 2024-02-19T15:03:02.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.543+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7d04004940 con 0x7f7d1c0ac770 2024-02-19T15:03:02.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.543+0000 7f7d12ffd640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7cfc06b840 0x7f7cfc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:02.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.543+0000 7f7d1a59c640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7cfc06b840 0x7f7cfc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:02.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.543+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7d040898d0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.544+0000 7f7d1a59c640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7cfc06b840 0x7f7cfc06dd00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000f60 tx=0x7f7d0c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:02.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.548+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7d040541a0 con 0x7f7d1c0ac770 2024-02-19T15:03:02.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.753+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d1c06da00 con 0x7f7d1c0ac770 2024-02-19T15:03:02.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.754+0000 7f7d12ffd640 1 -- 172.21.15.78:0/32948839 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f7d04018d90 con 0x7f7d1c0ac770 2024-02-19T15:03:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:02.757 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:02.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.755+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7cfc06b840 msgr2=0x7f7cfc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.755+0000 7f7d20a46640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7cfc06b840 0x7f7cfc06dd00 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c000f60 tx=0x7f7d0c009040 comp rx=0 tx=0).stop 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.755+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 msgr2=0x7f7d1c151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.755+0000 7f7d20a46640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c151450 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400c780 tx=0x7f7d0400cc40 comp rx=0 tx=0).stop 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 shutdown_connections 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7cfc06b840 0x7f7cfc06dd00 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d1c0ac770 0x7f7d1c151450 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d1c0bdf20 0x7f7d1c0c2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 --2- 172.21.15.78:0/32948839 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d1c0ac390 0x7f7d1c1ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 >> 172.21.15.78:0/32948839 conn(0x7f7d1c0ca890 msgr2=0x7f7d1c0cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 shutdown_connections 2024-02-19T15:03:02.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:02.756+0000 7f7d20a46640 1 -- 172.21.15.78:0/32948839 wait complete. 2024-02-19T15:03:03.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:03.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:04.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.284+0000 7f9de05e9640 1 Processor -- start 2024-02-19T15:03:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.285+0000 7f9de05e9640 1 -- start start 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.285+0000 7f9de05e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9dd80c8e00 0x7f9dd806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.285+0000 7f9de05e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9dd806d910 0x7f9dd806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.285+0000 7f9de05e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd806cd20 con 0x7f9dd8152700 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.285+0000 7f9de05e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd81551f0 con 0x7f9dd806d910 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.285+0000 7f9de05e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd8155490 con 0x7f9dd80c8e00 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddeb86640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9dd806d910 0x7f9dd806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9dde385640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9dd8152700 0x7f9dd80c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52350/0 (socket says 172.21.15.78:52350) 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9dde385640 1 -- 172.21.15.78:0/3828954108 learned_addr learned my addr 172.21.15.78:0/3828954108 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9dddb84640 1 --2- 172.21.15.78:0/3828954108 >> v2:172.21.15.150:3300/0 conn(0x7f9dd80c8e00 0x7f9dd806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddd383640 1 -- 172.21.15.78:0/3828954108 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246732617 0 0) 0x7f9dd806cd20 con 0x7f9dd8152700 2024-02-19T15:03:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddd383640 1 -- 172.21.15.78:0/3828954108 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc0003520 con 0x7f9dd8152700 2024-02-19T15:03:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddeb86640 1 -- 172.21.15.78:0/3828954108 >> v2:172.21.15.150:3300/0 conn(0x7f9dd80c8e00 msgr2=0x7f9dd806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddeb86640 1 --2- 172.21.15.78:0/3828954108 >> v2:172.21.15.150:3300/0 conn(0x7f9dd80c8e00 0x7f9dd806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddeb86640 1 -- 172.21.15.78:0/3828954108 >> v1:172.21.15.78:6789/0 conn(0x7f9dd8152700 legacy=0x7f9dd80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.286+0000 7f9ddeb86640 1 -- 172.21.15.78:0/3828954108 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd8155770 con 0x7f9dd806d910 2024-02-19T15:03:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.287+0000 7f9ddeb86640 1 --2- 172.21.15.78:0/3828954108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd806c7e0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80c7620 tx=0x7f9dc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f6ece8be9f85d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.287+0000 7f9ddd383640 1 -- 172.21.15.78:0/3828954108 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc802cc80 con 0x7f9dd806d910 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.287+0000 7f9ddd383640 1 -- 172.21.15.78:0/3828954108 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc802cde0 con 0x7f9dd806d910 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.287+0000 7f9ddd383640 1 -- 172.21.15.78:0/3828954108 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8034640 con 0x7f9dd806d910 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.287+0000 7f9de05e9640 1 -- 172.21.15.78:0/3828954108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 msgr2=0x7f9dd806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.287+0000 7f9de05e9640 1 --2- 172.21.15.78:0/3828954108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd806c7e0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80c7620 tx=0x7f9dc802bed0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.288+0000 7f9de05e9640 1 -- 172.21.15.78:0/3828954108 shutdown_connections 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.288+0000 7f9de05e9640 1 --2- 172.21.15.78:0/3828954108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd806c7e0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.288+0000 7f9de05e9640 1 --2- 172.21.15.78:0/3828954108 >> v2:172.21.15.150:3300/0 conn(0x7f9dd80c8e00 0x7f9dd806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.288+0000 7f9de05e9640 1 -- 172.21.15.78:0/3828954108 >> 172.21.15.78:0/3828954108 conn(0x7f9dd80cac90 msgr2=0x7f9dd80cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.288+0000 7f9de05e9640 1 -- 172.21.15.78:0/3828954108 shutdown_connections 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.288+0000 7f9de05e9640 1 -- 172.21.15.78:0/3828954108 wait complete. 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 Processor -- start 2024-02-19T15:03:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 -- start start 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd80c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd80c8e00 0x7f9dd80c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd8152700 0x7f9dd80aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80ad210 con 0x7f9dd806d910 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd81ee2d0 con 0x7f9dd8152700 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9de05e9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd81ed6c0 con 0x7f9dd80c8e00 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.289+0000 7f9dde385640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd80c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd80c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35682/0 (socket says 172.21.15.78:35682) 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 -- 172.21.15.78:0/287070273 learned_addr learned my addr 172.21.15.78:0/287070273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dddb84640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd80c8e00 0x7f9dd80c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9ddeb86640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd8152700 0x7f9dd80aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 -- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd80c8e00 msgr2=0x7f9dd80c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd80c8e00 0x7f9dd80c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 -- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd8152700 msgr2=0x7f9dd80aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd8152700 0x7f9dd80aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 -- 172.21.15.78:0/287070273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8009a70 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dde385640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd80c4010 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c390 tx=0x7f9dd000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0005ce0 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0005e40 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00072f0 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd81ed9a0 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.290+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd81edf60 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.291+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd81ee0f0 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.292+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9dd0007450 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.292+0000 7f9dd6ffd640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9dc406b840 0x7f9dc406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.292+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9dd00898d0 con 0x7f9dd806d910 2024-02-19T15:03:04.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.292+0000 7f9dddb84640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9dc406b840 0x7f9dc406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.293+0000 7f9dddb84640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9dc406b840 0x7f9dc406dd00 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80c5ae0 tx=0x7f9dcc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.297+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9dd00541a0 con 0x7f9dd806d910 2024-02-19T15:03:04.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.512+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd8074990 con 0x7f9dd806d910 2024-02-19T15:03:04.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.512+0000 7f9dd6ffd640 1 -- 172.21.15.78:0/287070273 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f9dd0017d90 con 0x7f9dd806d910 2024-02-19T15:03:04.516 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:04.516 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:04.516 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9dc406b840 msgr2=0x7f9dc406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9dc406b840 0x7f9dc406dd00 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80c5ae0 tx=0x7f9dcc009190 comp rx=0 tx=0).stop 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 msgr2=0x7f9dd80c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd80c4010 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c390 tx=0x7f9dd000c850 comp rx=0 tx=0).stop 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 shutdown_connections 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9dc406b840 0x7f9dc406dd00 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dd8152700 0x7f9dd80aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9dd80c8e00 0x7f9dd80c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 --2- 172.21.15.78:0/287070273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9dd806d910 0x7f9dd80c4010 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 >> 172.21.15.78:0/287070273 conn(0x7f9dd80cac90 msgr2=0x7f9dd8151450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 shutdown_connections 2024-02-19T15:03:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:04.515+0000 7f9de05e9640 1 -- 172.21.15.78:0/287070273 wait complete. 2024-02-19T15:03:04.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:04.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:05.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:06.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 Processor -- start 2024-02-19T15:03:06.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 -- start start 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 0x7f99100c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99100c2f60 0x7f991006eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f991006f530 0x7f99100705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99100ae670 con 0x7f991006f530 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99100716b0 con 0x7f99100add70 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.045+0000 7f9916ceb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99100c33a0 con 0x7f99100c2f60 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f991006f530 0x7f99100705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9914a87640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 0x7f99100c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f990ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99100c2f60 0x7f991006eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9914a87640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 0x7f99100c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:47280/0 (socket says 172.21.15.78:47280) 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f991006f530 0x7f99100705b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35696/0 (socket says 172.21.15.78:35696) 2024-02-19T15:03:06.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9914a87640 1 -- 172.21.15.78:0/251744430 learned_addr learned my addr 172.21.15.78:0/251744430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 -- 172.21.15.78:0/251744430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99100c2f60 msgr2=0x7f991006eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 --2- 172.21.15.78:0/251744430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99100c2f60 0x7f991006eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 -- 172.21.15.78:0/251744430 >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 msgr2=0x7f99100c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 --2- 172.21.15.78:0/251744430 >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 0x7f99100c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9915288640 1 -- 172.21.15.78:0/251744430 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9910153780 con 0x7f991006f530 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f990ffff640 1 --2- 172.21.15.78:0/251744430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99100c2f60 0x7f991006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.046+0000 7f9914a87640 1 --2- 172.21.15.78:0/251744430 >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 0x7f99100c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.047+0000 7f9915288640 1 --2- 172.21.15.78:0/251744430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f991006f530 0x7f99100705b0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f99100c4fc0 tx=0x7f990802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67841f3e082f7459 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.047+0000 7f990f7fe640 1 -- 172.21.15.78:0/251744430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990802cb40 con 0x7f991006f530 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.047+0000 7f990f7fe640 1 -- 172.21.15.78:0/251744430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990802cca0 con 0x7f991006f530 2024-02-19T15:03:06.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.047+0000 7f990f7fe640 1 -- 172.21.15.78:0/251744430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908034660 con 0x7f991006f530 2024-02-19T15:03:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.047+0000 7f9916ceb640 1 -- 172.21.15.78:0/251744430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f991006f530 msgr2=0x7f99100705b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.047+0000 7f9916ceb640 1 --2- 172.21.15.78:0/251744430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f991006f530 0x7f99100705b0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f99100c4fc0 tx=0x7f990802bd90 comp rx=0 tx=0).stop 2024-02-19T15:03:06.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 -- 172.21.15.78:0/251744430 shutdown_connections 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 --2- 172.21.15.78:0/251744430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f991006f530 0x7f99100705b0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 --2- 172.21.15.78:0/251744430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99100c2f60 0x7f991006eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 --2- 172.21.15.78:0/251744430 >> v2:172.21.15.150:3300/0 conn(0x7f99100add70 0x7f99100c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 -- 172.21.15.78:0/251744430 >> 172.21.15.78:0/251744430 conn(0x7f99100c85e0 msgr2=0x7f99100c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 -- 172.21.15.78:0/251744430 shutdown_connections 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.048+0000 7f9916ceb640 1 -- 172.21.15.78:0/251744430 wait complete. 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.049+0000 7f9916ceb640 1 Processor -- start 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.049+0000 7f9916ceb640 1 -- start start 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9916ceb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f991006f530 0x7f9910150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9916ceb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 0x7f9910155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9916ceb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99100c2f60 0x7f99100ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9916ceb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9910154140 con 0x7f99100add70 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9916ceb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9910155f50 con 0x7f99100c2f60 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9916ceb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99101543e0 con 0x7f991006f530 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 0x7f9910155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 0x7f9910155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35702/0 (socket says 172.21.15.78:35702) 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 -- 172.21.15.78:0/2926866418 learned_addr learned my addr 172.21.15.78:0/2926866418 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9914a87640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f991006f530 0x7f9910150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f9915288640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99100c2f60 0x7f99100ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 -- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f991006f530 msgr2=0x7f9910150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f991006f530 0x7f9910150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 -- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99100c2f60 msgr2=0x7f99100ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:06.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99100c2f60 0x7f99100ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 -- 172.21.15.78:0/2926866418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040094d0 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990ffff640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 0x7f9910155a10 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00ebe0 tx=0x7f98fc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.050+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc009760 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.051+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc00cc30 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.051+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00eee0 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.051+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc009400 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.051+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9908009a70 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.052+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99100676a0 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.052+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f98fc013470 con 0x7f99100add70 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.052+0000 7f990d7fa640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98f406b810 0x7f98f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.053+0000 7f9914a87640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98f406b810 0x7f98f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:06.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.053+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f98fc08a9a0 con 0x7f99100add70 2024-02-19T15:03:06.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.053+0000 7f9914a87640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98f406b810 0x7f98f406dcd0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9904006ca0 tx=0x7f9904009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:06.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.057+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98fc0551e0 con 0x7f99100add70 2024-02-19T15:03:06.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.265+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9910074990 con 0x7f99100add70 2024-02-19T15:03:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.266+0000 7f990d7fa640 1 -- 172.21.15.78:0/2926866418 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f98fc017d90 con 0x7f99100add70 2024-02-19T15:03:06.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:06.269 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:06.269 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98f406b810 msgr2=0x7f98f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98f406b810 0x7f98f406dcd0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9904006ca0 tx=0x7f9904009240 comp rx=0 tx=0).stop 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 msgr2=0x7f9910155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 0x7f9910155a10 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00ebe0 tx=0x7f98fc00c520 comp rx=0 tx=0).stop 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 shutdown_connections 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f98f406b810 0x7f98f406dcd0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99100add70 0x7f9910155a10 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99100c2f60 0x7f99100ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 --2- 172.21.15.78:0/2926866418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f991006f530 0x7f9910150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 >> 172.21.15.78:0/2926866418 conn(0x7f99100c85e0 msgr2=0x7f99100a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 shutdown_connections 2024-02-19T15:03:06.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:06.268+0000 7f9916ceb640 1 -- 172.21.15.78:0/2926866418 wait complete. 2024-02-19T15:03:06.684 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:06.684 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:07.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:07.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.795+0000 7f6854c23640 1 Processor -- start 2024-02-19T15:03:07.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.795+0000 7f6854c23640 1 -- start start 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f6854c23640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014f1a0 0x7f6850072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f6854c23640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f685006ea50 0x7f6850152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f6854c23640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 0x7f68500c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f6854c23640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500c5970 con 0x7f685006fbc0 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f6854c23640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500c5750 con 0x7f685006ea50 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f6854c23640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68500c5bc0 con 0x7f685014f1a0 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014f1a0 0x7f6850072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684dd9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f685006ea50 0x7f6850152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 0x7f68500c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684dd9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f685006ea50 0x7f6850152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:42322/0 (socket says 172.21.15.78:42322) 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014f1a0 0x7f6850072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:53102/0 (socket says 172.21.15.78:53102) 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684dd9b640 1 -- 172.21.15.78:0/3642370687 learned_addr learned my addr 172.21.15.78:0/3642370687 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.796+0000 7f684ed9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 0x7f68500c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33240/0 (socket says 172.21.15.78:33240) 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684dd9b640 1 -- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014f1a0 msgr2=0x7f6850072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684dd9b640 1 --2- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014f1a0 0x7f6850072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684dd9b640 1 -- 172.21.15.78:0/3642370687 >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 msgr2=0x7f68500c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684dd9b640 1 --2- 172.21.15.78:0/3642370687 >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 0x7f68500c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684dd9b640 1 -- 172.21.15.78:0/3642370687 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500c5ea0 con 0x7f685006ea50 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684ed9d640 1 --2- 172.21.15.78:0/3642370687 >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 0x7f68500c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:07.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.797+0000 7f684dd9b640 1 --2- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f6850152680 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f68500c8650 tx=0x7f684402dfc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3076b7bcc25788a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.798+0000 7f684d59a640 1 -- 172.21.15.78:0/3642370687 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684402ed70 con 0x7f685006ea50 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.798+0000 7f684d59a640 1 -- 172.21.15.78:0/3642370687 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844036440 con 0x7f685006ea50 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.798+0000 7f684d59a640 1 -- 172.21.15.78:0/3642370687 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68440367d0 con 0x7f685006ea50 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 -- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 msgr2=0x7f6850152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 --2- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f6850152680 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f68500c8650 tx=0x7f684402dfc0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 -- 172.21.15.78:0/3642370687 shutdown_connections 2024-02-19T15:03:07.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 --2- 172.21.15.78:0/3642370687 >> v2:172.21.15.78:3300/0 conn(0x7f685006fbc0 0x7f68500c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 --2- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f6850152680 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 --2- 172.21.15.78:0/3642370687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685014f1a0 0x7f6850072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 -- 172.21.15.78:0/3642370687 >> 172.21.15.78:0/3642370687 conn(0x7f685006c170 msgr2=0x7f685006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 -- 172.21.15.78:0/3642370687 shutdown_connections 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.799+0000 7f6854c23640 1 -- 172.21.15.78:0/3642370687 wait complete. 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 Processor -- start 2024-02-19T15:03:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 -- start start 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f68501e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685006fbc0 0x7f6850154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 0x7f6850154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850153200 con 0x7f685014f1a0 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68501ea530 con 0x7f685006ea50 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f6854c23640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68500c4790 con 0x7f685006fbc0 2024-02-19T15:03:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.801+0000 7f684ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 0x7f6850154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 0x7f6850154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33252/0 (socket says 172.21.15.78:33252) 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 -- 172.21.15.78:0/413865154 learned_addr learned my addr 172.21.15.78:0/413865154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f68501e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684dd9b640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685006fbc0 0x7f6850154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 -- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685006fbc0 msgr2=0x7f6850154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685006fbc0 0x7f6850154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 -- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 msgr2=0x7f68501e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f68501e9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 -- 172.21.15.78:0/413865154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848009a70 con 0x7f685014f1a0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f684ed9d640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 0x7f6850154ed0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f684800efc0 tx=0x7f6848009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848009d70 con 0x7f685014f1a0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684800f040 con 0x7f685014f1a0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848003990 con 0x7f685014f1a0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684400b770 con 0x7f685014f1a0 2024-02-19T15:03:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.802+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6850155410 con 0x7f685014f1a0 2024-02-19T15:03:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.803+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68500c4cb0 con 0x7f685014f1a0 2024-02-19T15:03:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.804+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f68480144a0 con 0x7f685014f1a0 2024-02-19T15:03:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.804+0000 7f683effd640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683006b810 0x7f683006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.804+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f684808af80 con 0x7f685014f1a0 2024-02-19T15:03:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.804+0000 7f684e59c640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683006b810 0x7f683006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.805+0000 7f684e59c640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683006b810 0x7f683006dcd0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6840009a20 tx=0x7f6840009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:07.809+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68480bedc0 con 0x7f685014f1a0 2024-02-19T15:03:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.016+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6850074990 con 0x7f685014f1a0 2024-02-19T15:03:08.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.017+0000 7f683effd640 1 -- 172.21.15.78:0/413865154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f6848018090 con 0x7f685014f1a0 2024-02-19T15:03:08.021 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:08.021 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:08.021 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683006b810 msgr2=0x7f683006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683006b810 0x7f683006dcd0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6840009a20 tx=0x7f6840009190 comp rx=0 tx=0).stop 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 msgr2=0x7f6850154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 0x7f6850154ed0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f684800efc0 tx=0x7f6848009f90 comp rx=0 tx=0).stop 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 shutdown_connections 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f683006b810 0x7f683006dcd0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f685014f1a0 0x7f6850154ed0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f685006fbc0 0x7f6850154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 --2- 172.21.15.78:0/413865154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f685006ea50 0x7f68501e9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 >> 172.21.15.78:0/413865154 conn(0x7f685006c170 msgr2=0x7f6850072b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 shutdown_connections 2024-02-19T15:03:08.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:08.019+0000 7f6854c23640 1 -- 172.21.15.78:0/413865154 wait complete. 2024-02-19T15:03:08.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:08.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:09.437 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:09.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.547+0000 7fed747c7640 1 Processor -- start 2024-02-19T15:03:09.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.547+0000 7fed747c7640 1 -- start start 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.548+0000 7fed747c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed6c0c8e00 0x7fed6c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.548+0000 7fed747c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c06d910 0x7fed6c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.548+0000 7fed747c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c06cd20 con 0x7fed6c152700 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.548+0000 7fed747c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c1551f0 con 0x7fed6c0c8e00 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.548+0000 7fed747c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c155490 con 0x7fed6c06d910 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed72563640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fed6c152700 0x7fed6c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56554/0 (socket says 172.21.15.78:56554) 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed72563640 1 -- 172.21.15.78:0/1836438684 learned_addr learned my addr 172.21.15.78:0/1836438684 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed72d64640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c06d910 0x7fed6c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71d62640 1 --2- 172.21.15.78:0/1836438684 >> v2:172.21.15.150:3300/0 conn(0x7fed6c0c8e00 0x7fed6c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71561640 1 -- 172.21.15.78:0/1836438684 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645502986 0 0) 0x7fed6c06cd20 con 0x7fed6c152700 2024-02-19T15:03:09.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71561640 1 -- 172.21.15.78:0/1836438684 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed58003520 con 0x7fed6c152700 2024-02-19T15:03:09.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71d62640 1 -- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c06d910 msgr2=0x7fed6c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:09.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71d62640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c06d910 0x7fed6c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71d62640 1 -- 172.21.15.78:0/1836438684 >> v1:172.21.15.78:6789/0 conn(0x7fed6c152700 legacy=0x7fed6c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:09.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed71d62640 1 -- 172.21.15.78:0/1836438684 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c155770 con 0x7fed6c0c8e00 2024-02-19T15:03:09.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.549+0000 7fed72d64640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c06d910 0x7fed6c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.550+0000 7fed71d62640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c0c8e00 0x7fed6c06c2a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0097a0 tx=0x7fed5c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e81e16a3fe4c37a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.550+0000 7fed71561640 1 -- 172.21.15.78:0/1836438684 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c031780 con 0x7fed6c0c8e00 2024-02-19T15:03:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.550+0000 7fed71561640 1 -- 172.21.15.78:0/1836438684 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c0318e0 con 0x7fed6c0c8e00 2024-02-19T15:03:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.550+0000 7fed71561640 1 -- 172.21.15.78:0/1836438684 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c031c50 con 0x7fed6c0c8e00 2024-02-19T15:03:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 -- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c0c8e00 msgr2=0x7fed6c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c0c8e00 0x7fed6c06c2a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0097a0 tx=0x7fed5c02b820 comp rx=0 tx=0).stop 2024-02-19T15:03:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 -- 172.21.15.78:0/1836438684 shutdown_connections 2024-02-19T15:03:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c0c8e00 0x7fed6c06c2a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 --2- 172.21.15.78:0/1836438684 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c06d910 0x7fed6c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 -- 172.21.15.78:0/1836438684 >> 172.21.15.78:0/1836438684 conn(0x7fed6c0cac90 msgr2=0x7fed6c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 -- 172.21.15.78:0/1836438684 shutdown_connections 2024-02-19T15:03:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.551+0000 7fed747c7640 1 -- 172.21.15.78:0/1836438684 wait complete. 2024-02-19T15:03:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 Processor -- start 2024-02-19T15:03:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 -- start start 2024-02-19T15:03:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 0x7fed6c146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c0c8e00 0x7fed6c0aa640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c152700 0x7fed6c0ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c156770 con 0x7fed6c06d910 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c159340 con 0x7fed6c152700 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.552+0000 7fed747c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c1eb750 con 0x7fed6c0c8e00 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 0x7fed6c146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 0x7fed6c146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33268/0 (socket says 172.21.15.78:33268) 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 -- 172.21.15.78:0/2584677546 learned_addr learned my addr 172.21.15.78:0/2584677546 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72d64640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c152700 0x7fed6c0ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed71d62640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c0c8e00 0x7fed6c0aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 -- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c0c8e00 msgr2=0x7fed6c0aa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c0c8e00 0x7fed6c0aa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 -- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c152700 msgr2=0x7fed6c0ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c152700 0x7fed6c0ac3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 -- 172.21.15.78:0/2584677546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64009a70 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed72563640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 0x7fed6c146ee0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fed6800c390 tx=0x7fed6800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68005ce0 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68005e40 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680072f0 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5c009400 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.553+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c158d80 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.554+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c1eb520 con 0x7fed6c06d910 2024-02-19T15:03:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.555+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fed68007450 con 0x7fed6c06d910 2024-02-19T15:03:09.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.555+0000 7fed62ffd640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed5006b840 0x7fed5006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:09.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.555+0000 7fed71d62640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed5006b840 0x7fed5006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:09.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.555+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fed68089800 con 0x7fed6c06d910 2024-02-19T15:03:09.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.556+0000 7fed71d62640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed5006b840 0x7fed5006dd00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fed5c036000 tx=0x7fed5c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:09.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.560+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fed6c1eb520 con 0x7fed6c06d910 2024-02-19T15:03:09.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.766+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed6c1eb520 con 0x7fed6c06d910 2024-02-19T15:03:09.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.766+0000 7fed62ffd640 1 -- 172.21.15.78:0/2584677546 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fed68017d90 con 0x7fed6c06d910 2024-02-19T15:03:09.770 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:09.770 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:09.771 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed5006b840 msgr2=0x7fed5006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed5006b840 0x7fed5006dd00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fed5c036000 tx=0x7fed5c03d040 comp rx=0 tx=0).stop 2024-02-19T15:03:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 msgr2=0x7fed6c146ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:09.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 0x7fed6c146ee0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fed6800c390 tx=0x7fed6800c790 comp rx=0 tx=0).stop 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 shutdown_connections 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fed5006b840 0x7fed5006dd00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed6c152700 0x7fed6c0ac3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed6c0c8e00 0x7fed6c0aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 --2- 172.21.15.78:0/2584677546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed6c06d910 0x7fed6c146ee0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 >> 172.21.15.78:0/2584677546 conn(0x7fed6c0cac90 msgr2=0x7fed6c150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 shutdown_connections 2024-02-19T15:03:09.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:09.768+0000 7fed747c7640 1 -- 172.21.15.78:0/2584677546 wait complete. 2024-02-19T15:03:10.186 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:10.186 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:11.187 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:11.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.293+0000 7fc094272640 1 Processor -- start 2024-02-19T15:03:11.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.293+0000 7fc094272640 1 -- start start 2024-02-19T15:03:11.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc094272640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c150440 0x7fc08c0c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:11.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc094272640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c150820 con 0x7fc08c0c3800 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc094272640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c1537d0 con 0x7fc08c06ed20 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc094272640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0c1b60 con 0x7fc08c150440 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc09280f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc08c0c3800 0x7fc08c06d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56556/0 (socket says 172.21.15.78:56556) 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc09280f640 1 -- 172.21.15.78:0/635188695 learned_addr learned my addr 172.21.15.78:0/635188695 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.294+0000 7fc09200e640 1 --2- 172.21.15.78:0/635188695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c150440 0x7fc08c0c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996653594 0 0) 0x7fc08c150820 con 0x7fc08c0c3800 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078003520 con 0x7fc08c0c3800 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684524989 0 0) 0x7fc08c1537d0 con 0x7fc08c06ed20 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08c150820 con 0x7fc08c06ed20 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09200e640 1 -- 172.21.15.78:0/635188695 >> v1:172.21.15.150:6789/0 conn(0x7fc08c06ed20 legacy=0x7fc08c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09200e640 1 -- 172.21.15.78:0/635188695 >> v1:172.21.15.78:6789/0 conn(0x7fc08c0c3800 legacy=0x7fc08c06d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.295+0000 7fc09200e640 1 -- 172.21.15.78:0/635188695 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c153d10 con 0x7fc08c150440 2024-02-19T15:03:11.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc09200e640 1 --2- 172.21.15.78:0/635188695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c150440 0x7fc08c0c32c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc088009dc0 tx=0x7fc08802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a574841a76f2b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08802ccd0 con 0x7fc08c150440 2024-02-19T15:03:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08802ce30 con 0x7fc08c150440 2024-02-19T15:03:11.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc09100c640 1 -- 172.21.15.78:0/635188695 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088034660 con 0x7fc08c150440 2024-02-19T15:03:11.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 -- 172.21.15.78:0/635188695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c150440 msgr2=0x7fc08c0c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 --2- 172.21.15.78:0/635188695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c150440 0x7fc08c0c32c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc088009dc0 tx=0x7fc08802bf20 comp rx=0 tx=0).stop 2024-02-19T15:03:11.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 -- 172.21.15.78:0/635188695 shutdown_connections 2024-02-19T15:03:11.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 --2- 172.21.15.78:0/635188695 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c150440 0x7fc08c0c32c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 -- 172.21.15.78:0/635188695 >> 172.21.15.78:0/635188695 conn(0x7fc08c06b270 msgr2=0x7fc08c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 -- 172.21.15.78:0/635188695 shutdown_connections 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.296+0000 7fc094272640 1 -- 172.21.15.78:0/635188695 wait complete. 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.297+0000 7fc094272640 1 Processor -- start 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 -- start start 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c06ed20 0x7fc08c1e8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc08c0c3800 0x7fc08c0a8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 0x7fc08c0a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0a62a0 con 0x7fc08c150440 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1e91a0 con 0x7fc08c0c3800 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc094272640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1e9480 con 0x7fc08c06ed20 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc09280f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 0x7fc08c0a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:11.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc09280f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 0x7fc08c0a6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33272/0 (socket says 172.21.15.78:33272) 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc09280f640 1 -- 172.21.15.78:0/3735912432 learned_addr learned my addr 172.21.15.78:0/3735912432 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc09200e640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c06ed20 0x7fc08c1e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.298+0000 7fc09180d640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc08c0c3800 0x7fc08c0a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc09280f640 1 -- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c06ed20 msgr2=0x7fc08c1e8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc09280f640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c06ed20 0x7fc08c1e8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc09280f640 1 -- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc08c0c3800 msgr2=0x7fc08c0a8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc09280f640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc08c0c3800 0x7fc08c0a8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc09280f640 1 -- 172.21.15.78:0/3735912432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088009a70 con 0x7fc08c150440 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc09280f640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 0x7fc08c0a6a10 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c430 tx=0x7fc08400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084005ce0 con 0x7fc08c150440 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc084005e40 con 0x7fc08c150440 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c1e9760 con 0x7fc08c150440 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840072f0 con 0x7fc08c150440 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.299+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c1e8e90 con 0x7fc08c150440 2024-02-19T15:03:11.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.300+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c0a6730 con 0x7fc08c150440 2024-02-19T15:03:11.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.300+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc084007450 con 0x7fc08c150440 2024-02-19T15:03:11.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.301+0000 7fc082ffd640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc07006b810 0x7fc07006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:11.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.301+0000 7fc09200e640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc07006b810 0x7fc07006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:11.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.301+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc084089970 con 0x7fc08c150440 2024-02-19T15:03:11.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.301+0000 7fc09200e640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc07006b810 0x7fc07006dcd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc088036000 tx=0x7fc088030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:11.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.305+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc084054240 con 0x7fc08c150440 2024-02-19T15:03:11.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.509+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc08c074990 con 0x7fc08c150440 2024-02-19T15:03:11.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.510+0000 7fc082ffd640 1 -- 172.21.15.78:0/3735912432 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fc084017d90 con 0x7fc08c150440 2024-02-19T15:03:11.514 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:11.514 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:11.514 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc07006b810 msgr2=0x7fc07006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc07006b810 0x7fc07006dcd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc088036000 tx=0x7fc088030000 comp rx=0 tx=0).stop 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 msgr2=0x7fc08c0a6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 0x7fc08c0a6a10 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c430 tx=0x7fc08400c8f0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 shutdown_connections 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc07006b810 0x7fc07006dcd0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc08c150440 0x7fc08c0a6a10 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc08c0c3800 0x7fc08c0a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 --2- 172.21.15.78:0/3735912432 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc08c06ed20 0x7fc08c1e8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 >> 172.21.15.78:0/3735912432 conn(0x7fc08c06b270 msgr2=0x7fc08c14abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 shutdown_connections 2024-02-19T15:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:11.512+0000 7fc094272640 1 -- 172.21.15.78:0/3735912432 wait complete. 2024-02-19T15:03:11.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:11.929 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:12.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.038+0000 7efde8a10640 1 Processor -- start 2024-02-19T15:03:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.038+0000 7efde8a10640 1 -- start start 2024-02-19T15:03:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde8a10640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efde406ed20 0x7efde406d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde8a10640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde415d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde8a10640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde4150820 con 0x7efde4150440 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde8a10640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde40c1b60 con 0x7efde406ed20 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde8a10640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde41537d0 con 0x7efde406d750 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde1d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efde406ed20 0x7efde406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efde4150440 0x7efde40c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:47560/0 (socket says 172.21.15.78:47560) 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde259c640 1 -- 172.21.15.78:0/462360930 learned_addr learned my addr 172.21.15.78:0/462360930 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde2d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde415d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde2d9d640 1 -- 172.21.15.78:0/462360930 >> v2:172.21.15.150:3300/0 conn(0x7efde406ed20 msgr2=0x7efde406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde2d9d640 1 --2- 172.21.15.78:0/462360930 >> v2:172.21.15.150:3300/0 conn(0x7efde406ed20 0x7efde406d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde2d9d640 1 -- 172.21.15.78:0/462360930 >> v1:172.21.15.150:6789/0 conn(0x7efde4150440 legacy=0x7efde40c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde159a640 1 -- 172.21.15.78:0/462360930 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734279182 0 0) 0x7efde4150820 con 0x7efde4150440 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.039+0000 7efde2d9d640 1 -- 172.21.15.78:0/462360930 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4153d10 con 0x7efde406d750 2024-02-19T15:03:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.040+0000 7efde1d9b640 1 --2- 172.21.15.78:0/462360930 >> v2:172.21.15.150:3300/0 conn(0x7efde406ed20 0x7efde406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.040+0000 7efde2d9d640 1 --2- 172.21.15.78:0/462360930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde415d720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efde4153f00 tx=0x7efddc02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c11f413ad16d20fe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.040+0000 7efde159a640 1 -- 172.21.15.78:0/462360930 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc02ccd0 con 0x7efde406d750 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.040+0000 7efde159a640 1 -- 172.21.15.78:0/462360930 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc02ce30 con 0x7efde406d750 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.040+0000 7efde159a640 1 -- 172.21.15.78:0/462360930 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc034660 con 0x7efde406d750 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 -- 172.21.15.78:0/462360930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 msgr2=0x7efde415d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 --2- 172.21.15.78:0/462360930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde415d720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efde4153f00 tx=0x7efddc02bf20 comp rx=0 tx=0).stop 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 -- 172.21.15.78:0/462360930 shutdown_connections 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 --2- 172.21.15.78:0/462360930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde415d720 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 --2- 172.21.15.78:0/462360930 >> v2:172.21.15.150:3300/0 conn(0x7efde406ed20 0x7efde406d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 -- 172.21.15.78:0/462360930 >> 172.21.15.78:0/462360930 conn(0x7efde406b270 msgr2=0x7efde406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 -- 172.21.15.78:0/462360930 shutdown_connections 2024-02-19T15:03:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.041+0000 7efde8a10640 1 -- 172.21.15.78:0/462360930 wait complete. 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.042+0000 7efde8a10640 1 Processor -- start 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.042+0000 7efde8a10640 1 -- start start 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde8a10640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde4155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde8a10640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde406ed20 0x7efde414b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde8a10640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 0x7efde414bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde8a10640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4155ef0 con 0x7efde4150440 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde8a10640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde41527a0 con 0x7efde406ed20 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde8a10640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde4152c90 con 0x7efde406d750 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 0x7efde414bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 0x7efde414bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33278/0 (socket says 172.21.15.78:33278) 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde2d9d640 1 -- 172.21.15.78:0/3661499558 learned_addr learned my addr 172.21.15.78:0/3661499558 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde1d9b640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde406ed20 0x7efde414b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.043+0000 7efde259c640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde4155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde2d9d640 1 -- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 msgr2=0x7efde4155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde2d9d640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde4155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde2d9d640 1 -- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde406ed20 msgr2=0x7efde414b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde2d9d640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde406ed20 0x7efde414b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde2d9d640 1 -- 172.21.15.78:0/3661499558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009400 con 0x7efde4150440 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde2d9d640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 0x7efde414bd70 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7efddc037040 tx=0x7efddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc03d600 con 0x7efde4150440 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc03d780 con 0x7efde4150440 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efddc009a70 con 0x7efde4150440 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc03db30 con 0x7efde4150440 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.044+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde41529b0 con 0x7efde4150440 2024-02-19T15:03:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.045+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde4074990 con 0x7efde4150440 2024-02-19T15:03:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.046+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efddc03b040 con 0x7efde4150440 2024-02-19T15:03:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.046+0000 7efdd2ffd640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efdc406b810 0x7efdc406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.046+0000 7efde259c640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efdc406b810 0x7efdc406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.046+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7efddc0ac8a0 con 0x7efde4150440 2024-02-19T15:03:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.046+0000 7efde259c640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efdc406b810 0x7efdc406dcd0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efdd4007fd0 tx=0x7efdd4007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.050+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efddc077290 con 0x7efde4150440 2024-02-19T15:03:13.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.255+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efde40c4a90 con 0x7efde4150440 2024-02-19T15:03:13.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.256+0000 7efdd2ffd640 1 -- 172.21.15.78:0/3661499558 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7efddc039d90 con 0x7efde4150440 2024-02-19T15:03:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:13.259 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:13.259 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efdc406b810 msgr2=0x7efdc406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efdc406b810 0x7efdc406dcd0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efdd4007fd0 tx=0x7efdd4007de0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 msgr2=0x7efde414bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 0x7efde414bd70 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7efddc037040 tx=0x7efddc004a40 comp rx=0 tx=0).stop 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 shutdown_connections 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efdc406b810 0x7efdc406dcd0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4150440 0x7efde414bd70 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde406ed20 0x7efde414b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 --2- 172.21.15.78:0/3661499558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde406d750 0x7efde4155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 >> 172.21.15.78:0/3661499558 conn(0x7efde406b270 msgr2=0x7efde414ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 shutdown_connections 2024-02-19T15:03:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:13.258+0000 7efde8a10640 1 -- 172.21.15.78:0/3661499558 wait complete. 2024-02-19T15:03:13.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:13.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:14.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.786+0000 7f39cbbd7640 1 Processor -- start 2024-02-19T15:03:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.787+0000 7f39cbbd7640 1 -- start start 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.787+0000 7f39cbbd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c4152480 0x7f39c41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.787+0000 7f39cbbd7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39c40c87a0 0x7f39c40c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39cbbd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c4155ef0 con 0x7f39c40c7070 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39cbbd7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c406dec0 con 0x7f39c40c87a0 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39cbbd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c406d520 con 0x7f39c4152480 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9172640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f39c40c7070 0x7f39c406ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56566/0 (socket says 172.21.15.78:56566) 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9172640 1 -- 172.21.15.78:0/2285410825 learned_addr learned my addr 172.21.15.78:0/2285410825 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39ca174640 1 --2- 172.21.15.78:0/2285410825 >> v2:172.21.15.150:3300/0 conn(0x7f39c40c87a0 0x7f39c40c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9973640 1 --2- 172.21.15.78:0/2285410825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c4152480 0x7f39c41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c8971640 1 -- 172.21.15.78:0/2285410825 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544740000 0 0) 0x7f39c0001110 con 0x7f39c40c7070 2024-02-19T15:03:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c8971640 1 -- 172.21.15.78:0/2285410825 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0003520 con 0x7f39c40c7070 2024-02-19T15:03:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9973640 1 -- 172.21.15.78:0/2285410825 >> v2:172.21.15.150:3300/0 conn(0x7f39c40c87a0 msgr2=0x7f39c40c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9973640 1 --2- 172.21.15.78:0/2285410825 >> v2:172.21.15.150:3300/0 conn(0x7f39c40c87a0 0x7f39c40c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9973640 1 -- 172.21.15.78:0/2285410825 >> v1:172.21.15.78:6789/0 conn(0x7f39c40c7070 legacy=0x7f39c406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.788+0000 7f39c9973640 1 -- 172.21.15.78:0/2285410825 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40c75c0 con 0x7f39c4152480 2024-02-19T15:03:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.789+0000 7f39c9973640 1 --2- 172.21.15.78:0/2285410825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c4152480 0x7f39c41559b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39b40097a0 tx=0x7f39b402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae73a88d9cbc652 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.789+0000 7f39c8971640 1 -- 172.21.15.78:0/2285410825 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4031830 con 0x7f39c4152480 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.789+0000 7f39c8971640 1 -- 172.21.15.78:0/2285410825 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4031990 con 0x7f39c4152480 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.789+0000 7f39c8971640 1 -- 172.21.15.78:0/2285410825 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4031d00 con 0x7f39c4152480 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 -- 172.21.15.78:0/2285410825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c4152480 msgr2=0x7f39c41559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/2285410825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c4152480 0x7f39c41559b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39b40097a0 tx=0x7f39b402b8d0 comp rx=0 tx=0).stop 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 -- 172.21.15.78:0/2285410825 shutdown_connections 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/2285410825 >> v2:172.21.15.150:3300/0 conn(0x7f39c40c87a0 0x7f39c40c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/2285410825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c4152480 0x7f39c41559b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 -- 172.21.15.78:0/2285410825 >> 172.21.15.78:0/2285410825 conn(0x7f39c40ca8a0 msgr2=0x7f39c40cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 -- 172.21.15.78:0/2285410825 shutdown_connections 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.790+0000 7f39cbbd7640 1 -- 172.21.15.78:0/2285410825 wait complete. 2024-02-19T15:03:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.791+0000 7f39cbbd7640 1 Processor -- start 2024-02-19T15:03:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.791+0000 7f39cbbd7640 1 -- start start 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40c7070 0x7f39c41ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 0x7f39c40ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c4152480 0x7f39c40afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41eec70 con 0x7f39c40c87a0 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40c10e0 con 0x7f39c4152480 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40ae6d0 con 0x7f39c40c7070 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 0x7f39c40ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 0x7f39c40ac620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33292/0 (socket says 172.21.15.78:33292) 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 -- 172.21.15.78:0/3232662397 learned_addr learned my addr 172.21.15.78:0/3232662397 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9973640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40c7070 0x7f39c41ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39ca174640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c4152480 0x7f39c40afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 -- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40c7070 msgr2=0x7f39c41ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40c7070 0x7f39c41ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 -- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c4152480 msgr2=0x7f39c40afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c4152480 0x7f39c40afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 -- 172.21.15.78:0/3232662397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4009400 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39c9172640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 0x7f39c40ac620 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c3c0 tx=0x7f39c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0005ce0 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c0005e40 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.792+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c414b450 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.793+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c00072f0 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.793+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c40af3e0 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.793+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c414b970 con 0x7f39c40c87a0 2024-02-19T15:03:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.794+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f39c0007450 con 0x7f39c40c87a0 2024-02-19T15:03:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.794+0000 7f39ba7fc640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f39a806b840 0x7f39a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.795+0000 7f39c9973640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f39a806b840 0x7f39a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.795+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f39c0089850 con 0x7f39c40c87a0 2024-02-19T15:03:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.795+0000 7f39c9973640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f39a806b840 0x7f39a806dd00 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f39b4036000 tx=0x7f39b403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:14.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:14.799+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f39c0054010 con 0x7f39c40c87a0 2024-02-19T15:03:15.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.006+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39c4155ef0 con 0x7f39c40c87a0 2024-02-19T15:03:15.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.006+0000 7f39ba7fc640 1 -- 172.21.15.78:0/3232662397 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f39c0017d90 con 0x7f39c40c87a0 2024-02-19T15:03:15.009 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:15.010 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:15.010 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f39a806b840 msgr2=0x7f39a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f39a806b840 0x7f39a806dd00 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f39b4036000 tx=0x7f39b403c040 comp rx=0 tx=0).stop 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 msgr2=0x7f39c40ac620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 0x7f39c40ac620 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c3c0 tx=0x7f39c000c880 comp rx=0 tx=0).stop 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 shutdown_connections 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f39a806b840 0x7f39a806dd00 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39c40c87a0 0x7f39c40ac620 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c4152480 0x7f39c40afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 --2- 172.21.15.78:0/3232662397 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40c7070 0x7f39c41ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 >> 172.21.15.78:0/3232662397 conn(0x7f39c40ca8a0 msgr2=0x7f39c4153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 shutdown_connections 2024-02-19T15:03:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:15.008+0000 7f39cbbd7640 1 -- 172.21.15.78:0/3232662397 wait complete. 2024-02-19T15:03:15.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:15.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:16.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.533+0000 7fab1d042640 1 Processor -- start 2024-02-19T15:03:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.533+0000 7fab1d042640 1 -- start start 2024-02-19T15:03:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1d042640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152a50 0x7fab18071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1d042640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fab180702d0 0x7fab18158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1d042640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab18071bf0 con 0x7fab18152670 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1d042640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab18158a20 con 0x7fab180702d0 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1d042640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab180c6780 con 0x7fab18152a50 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1759e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fab180702d0 0x7fab18158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab16d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fab18152670 0x7fab18155010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:56570/0 (socket says 172.21.15.78:56570) 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab16d9d640 1 -- 172.21.15.78:0/1470510282 learned_addr learned my addr 172.21.15.78:0/1470510282 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1759e640 1 --2- 172.21.15.78:0/1470510282 >> v2:172.21.15.78:3300/0 conn(0x7fab180702d0 0x7fab18158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33308/0 (socket says 172.21.15.78:33308) 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1659c640 1 --2- 172.21.15.78:0/1470510282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152a50 0x7fab18071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1759e640 1 -- 172.21.15.78:0/1470510282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152a50 msgr2=0x7fab18071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1759e640 1 --2- 172.21.15.78:0/1470510282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152a50 0x7fab18071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.534+0000 7fab1759e640 1 -- 172.21.15.78:0/1470510282 >> v1:172.21.15.78:6789/0 conn(0x7fab18152670 legacy=0x7fab18155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.535+0000 7fab1759e640 1 -- 172.21.15.78:0/1470510282 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18071630 con 0x7fab180702d0 2024-02-19T15:03:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.535+0000 7fab15d9b640 1 -- 172.21.15.78:0/1470510282 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701460063 0 0) 0x7fab18071bf0 con 0x7fab18152670 2024-02-19T15:03:16.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.535+0000 7fab1759e640 1 --2- 172.21.15.78:0/1470510282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab180702d0 0x7fab18158eb0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fab10009d90 tx=0x7fab1002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3843df8aff05897 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.535+0000 7fab15d9b640 1 -- 172.21.15.78:0/1470510282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1002cce0 con 0x7fab180702d0 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.535+0000 7fab15d9b640 1 -- 172.21.15.78:0/1470510282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1002ce40 con 0x7fab180702d0 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.535+0000 7fab15d9b640 1 -- 172.21.15.78:0/1470510282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab10034660 con 0x7fab180702d0 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 -- 172.21.15.78:0/1470510282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab180702d0 msgr2=0x7fab18158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 --2- 172.21.15.78:0/1470510282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab180702d0 0x7fab18158eb0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fab10009d90 tx=0x7fab1002bf30 comp rx=0 tx=0).stop 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 -- 172.21.15.78:0/1470510282 shutdown_connections 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 --2- 172.21.15.78:0/1470510282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab180702d0 0x7fab18158eb0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 --2- 172.21.15.78:0/1470510282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152a50 0x7fab18071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 -- 172.21.15.78:0/1470510282 >> 172.21.15.78:0/1470510282 conn(0x7fab180c85e0 msgr2=0x7fab180c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 -- 172.21.15.78:0/1470510282 shutdown_connections 2024-02-19T15:03:16.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.536+0000 7fab1d042640 1 -- 172.21.15.78:0/1470510282 wait complete. 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.537+0000 7fab1d042640 1 Processor -- start 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.537+0000 7fab1d042640 1 -- start start 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1d042640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab180702d0 0x7fab18154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1d042640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152670 0x7fab181eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1d042640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 0x7fab181eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1d042640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1814f840 con 0x7fab18152a50 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1d042640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab180addf0 con 0x7fab180702d0 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1d042640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab181eedf0 con 0x7fab18152670 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 0x7fab181eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 0x7fab181eb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33310/0 (socket says 172.21.15.78:33310) 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1759e640 1 -- 172.21.15.78:0/188204057 learned_addr learned my addr 172.21.15.78:0/188204057 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab16d9d640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab180702d0 0x7fab18154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.538+0000 7fab1659c640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152670 0x7fab181eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1759e640 1 -- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152670 msgr2=0x7fab181eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:16.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1759e640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152670 0x7fab181eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1759e640 1 -- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab180702d0 msgr2=0x7fab18154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1759e640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab180702d0 0x7fab18154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1759e640 1 -- 172.21.15.78:0/188204057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab10009a70 con 0x7fab18152a50 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1759e640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 0x7fab181eb690 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fab10009d90 tx=0x7fab10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1003e670 con 0x7fab18152a50 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1003e7f0 con 0x7fab18152a50 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab180ae000 con 0x7fab18152a50 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1003eba0 con 0x7fab18152a50 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.539+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab18154680 con 0x7fab18152a50 2024-02-19T15:03:16.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.540+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab180676a0 con 0x7fab18152a50 2024-02-19T15:03:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.540+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fab1003b040 con 0x7fab18152a50 2024-02-19T15:03:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.541+0000 7faaff7fe640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab0406b810 0x7fab0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.541+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fab100acd90 con 0x7fab18152a50 2024-02-19T15:03:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.541+0000 7fab16d9d640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab0406b810 0x7fab0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:16.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.541+0000 7fab16d9d640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab0406b810 0x7fab0406dcd0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fab08000f60 tx=0x7fab08007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:16.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.545+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fab100775d0 con 0x7fab18152a50 2024-02-19T15:03:16.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.750+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab18070120 con 0x7fab18152a50 2024-02-19T15:03:16.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.750+0000 7faaff7fe640 1 -- 172.21.15.78:0/188204057 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fab1004f090 con 0x7fab18152a50 2024-02-19T15:03:16.757 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:16.757 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab0406b810 msgr2=0x7fab0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab0406b810 0x7fab0406dcd0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fab08000f60 tx=0x7fab08007d50 comp rx=0 tx=0).stop 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 msgr2=0x7fab181eb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 0x7fab181eb690 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fab10009d90 tx=0x7fab10004a40 comp rx=0 tx=0).stop 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 shutdown_connections 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fab0406b810 0x7fab0406dcd0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab18152a50 0x7fab181eb690 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab18152670 0x7fab181eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 --2- 172.21.15.78:0/188204057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab180702d0 0x7fab18154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 >> 172.21.15.78:0/188204057 conn(0x7fab180c85e0 msgr2=0x7fab18070860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 shutdown_connections 2024-02-19T15:03:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:16.752+0000 7fab1d042640 1 -- 172.21.15.78:0/188204057 wait complete. 2024-02-19T15:03:17.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:17.169 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:18.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:18.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.280+0000 7f69f1a64640 1 Processor -- start 2024-02-19T15:03:18.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.280+0000 7f69f1a64640 1 -- start start 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69f1a64640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0bdf20 0x7f69ec0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69f1a64640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69ec0c55f0 0x7f69ec15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69f1a64640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ec158c40 con 0x7f69ec0ac390 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69f1a64640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ec0c5a60 con 0x7f69ec0c55f0 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69f1a64640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec0c8de0 con 0x7f69ec0bdf20 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69ec0c55f0 0x7f69ec15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69ec0c55f0 0x7f69ec15f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44574/0 (socket says 172.21.15.78:44574) 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 -- 172.21.15.78:0/1675629595 learned_addr learned my addr 172.21.15.78:0/1675629595 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69eaffd640 1 --2- 172.21.15.78:0/1675629595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0bdf20 0x7f69ec0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 -- 172.21.15.78:0/1675629595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0bdf20 msgr2=0x7f69ec0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 --2- 172.21.15.78:0/1675629595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0bdf20 0x7f69ec0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 -- 172.21.15.78:0/1675629595 >> v1:172.21.15.150:6789/0 conn(0x7f69ec0ac390 legacy=0x7f69ec158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ebfff640 1 -- 172.21.15.78:0/1675629595 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec155a60 con 0x7f69ec0c55f0 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.281+0000 7f69ea7fc640 1 -- 172.21.15.78:0/1675629595 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850163324 0 0) 0x7f69ec158c40 con 0x7f69ec0ac390 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.282+0000 7f69ebfff640 1 --2- 172.21.15.78:0/1675629595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0c55f0 0x7f69ec15f8c0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009e50 tx=0x7f69e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1403b0032c22c00c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.282+0000 7f69ea7fc640 1 -- 172.21.15.78:0/1675629595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e402c9e0 con 0x7f69ec0c55f0 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.282+0000 7f69ea7fc640 1 -- 172.21.15.78:0/1675629595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e402cb40 con 0x7f69ec0c55f0 2024-02-19T15:03:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.282+0000 7f69ea7fc640 1 -- 172.21.15.78:0/1675629595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e40343f0 con 0x7f69ec0c55f0 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 -- 172.21.15.78:0/1675629595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0c55f0 msgr2=0x7f69ec15f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1675629595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0c55f0 0x7f69ec15f8c0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009e50 tx=0x7f69e402be40 comp rx=0 tx=0).stop 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 -- 172.21.15.78:0/1675629595 shutdown_connections 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1675629595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0c55f0 0x7f69ec15f8c0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1675629595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0bdf20 0x7f69ec0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 -- 172.21.15.78:0/1675629595 >> 172.21.15.78:0/1675629595 conn(0x7f69ec0ca890 msgr2=0x7f69ec0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:18.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 -- 172.21.15.78:0/1675629595 shutdown_connections 2024-02-19T15:03:18.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.283+0000 7f69f1a64640 1 -- 172.21.15.78:0/1675629595 wait complete. 2024-02-19T15:03:18.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.284+0000 7f69f1a64640 1 Processor -- start 2024-02-19T15:03:18.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.284+0000 7f69f1a64640 1 -- start start 2024-02-19T15:03:18.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69f1a64640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 0x7f69ec0aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:18.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69f1a64640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec0bdf20 0x7f69ec151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:18.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69f1a64640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0c55f0 0x7f69ec151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:18.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69f1a64640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec152520 con 0x7f69ec0ac390 2024-02-19T15:03:18.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69f1a64640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec0c1650 con 0x7f69ec0bdf20 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69f1a64640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec1ea810 con 0x7f69ec0c55f0 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 0x7f69ec0aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 0x7f69ec0aed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44584/0 (socket says 172.21.15.78:44584) 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 -- 172.21.15.78:0/1498620385 learned_addr learned my addr 172.21.15.78:0/1498620385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec0bdf20 0x7f69ec151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69ebfff640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0c55f0 0x7f69ec151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 -- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0c55f0 msgr2=0x7f69ec151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0c55f0 0x7f69ec151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 -- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec0bdf20 msgr2=0x7f69ec151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec0bdf20 0x7f69ec151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 -- 172.21.15.78:0/1498620385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4009b30 con 0x7f69ec0ac390 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.285+0000 7f69eb7fe640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 0x7f69ec0aed10 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00c4b0 tx=0x7f69dc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.286+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc005480 con 0x7f69ec0ac390 2024-02-19T15:03:18.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.286+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc0035f0 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.286+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc0039e0 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.286+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec1eaaf0 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.286+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec1eb0b0 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.287+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec074990 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.287+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f69dc003b40 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.287+0000 7f69d3fff640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f69d806b810 0x7f69d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.287+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f69dc0899e0 con 0x7f69ec0ac390 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.288+0000 7f69eaffd640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f69d806b810 0x7f69d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:18.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.288+0000 7f69eaffd640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f69d806b810 0x7f69d806dcd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0ab3e0 tx=0x7f69e0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.292+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f69dc054220 con 0x7f69ec0ac390 2024-02-19T15:03:18.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.500+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69ec06da00 con 0x7f69ec0ac390 2024-02-19T15:03:18.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.500+0000 7f69d3fff640 1 -- 172.21.15.78:0/1498620385 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f69dc0578e0 con 0x7f69ec0ac390 2024-02-19T15:03:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:18.504 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f69d806b810 msgr2=0x7f69d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f69d806b810 0x7f69d806dcd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0ab3e0 tx=0x7f69e0009190 comp rx=0 tx=0).stop 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 msgr2=0x7f69ec0aed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 0x7f69ec0aed10 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f69dc00c4b0 tx=0x7f69dc00c970 comp rx=0 tx=0).stop 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 shutdown_connections 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f69d806b810 0x7f69d806dcd0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69ec0c55f0 0x7f69ec151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec0bdf20 0x7f69ec151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 --2- 172.21.15.78:0/1498620385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec0ac390 0x7f69ec0aed10 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 >> 172.21.15.78:0/1498620385 conn(0x7f69ec0ca890 msgr2=0x7f69ec0cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 shutdown_connections 2024-02-19T15:03:18.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:18.502+0000 7f69f1a64640 1 -- 172.21.15.78:0/1498620385 wait complete. 2024-02-19T15:03:18.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:18.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:19.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.024+0000 7f937a3f4640 1 Processor -- start 2024-02-19T15:03:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f937a3f4640 1 -- start start 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f937a3f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f93740c29c0 0x7f93741526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f937a3f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93740c5be0 0x7f937406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f937a3f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374072250 con 0x7f93740c25e0 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f937a3f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9374152c90 con 0x7f93740c29c0 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f937a3f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9374072480 con 0x7f93740c5be0 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f93737fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f93740c29c0 0x7f93741526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f9373fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f93740c25e0 0x7f93740c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44858/0 (socket says 172.21.15.78:44858) 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.025+0000 7f9373fff640 1 -- 172.21.15.78:0/371908235 learned_addr learned my addr 172.21.15.78:0/371908235 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.026+0000 7f9378991640 1 --2- 172.21.15.78:0/371908235 >> v2:172.21.15.150:3300/0 conn(0x7f93740c5be0 0x7f937406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.026+0000 7f93737fe640 1 -- 172.21.15.78:0/371908235 >> v2:172.21.15.150:3300/0 conn(0x7f93740c5be0 msgr2=0x7f937406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:20.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.026+0000 7f93737fe640 1 --2- 172.21.15.78:0/371908235 >> v2:172.21.15.150:3300/0 conn(0x7f93740c5be0 0x7f937406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.026+0000 7f93737fe640 1 -- 172.21.15.78:0/371908235 >> v1:172.21.15.78:6789/0 conn(0x7f93740c25e0 legacy=0x7f93740c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:20.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.026+0000 7f93737fe640 1 -- 172.21.15.78:0/371908235 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740c4b90 con 0x7f93740c29c0 2024-02-19T15:03:20.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.026+0000 7f9372ffd640 1 -- 172.21.15.78:0/371908235 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986296249 0 0) 0x7f9374072250 con 0x7f93740c25e0 2024-02-19T15:03:20.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f93737fe640 1 --2- 172.21.15.78:0/371908235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f93741526a0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f9374151c50 tx=0x7f936802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fba22df2bb2805 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f9372ffd640 1 -- 172.21.15.78:0/371908235 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936802e070 con 0x7f93740c29c0 2024-02-19T15:03:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f9372ffd640 1 -- 172.21.15.78:0/371908235 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93680315a0 con 0x7f93740c29c0 2024-02-19T15:03:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f9372ffd640 1 -- 172.21.15.78:0/371908235 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368031930 con 0x7f93740c29c0 2024-02-19T15:03:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 -- 172.21.15.78:0/371908235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 msgr2=0x7f93741526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 --2- 172.21.15.78:0/371908235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f93741526a0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f9374151c50 tx=0x7f936802b840 comp rx=0 tx=0).stop 2024-02-19T15:03:20.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 -- 172.21.15.78:0/371908235 shutdown_connections 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 --2- 172.21.15.78:0/371908235 >> v2:172.21.15.150:3300/0 conn(0x7f93740c5be0 0x7f937406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 --2- 172.21.15.78:0/371908235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f93741526a0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 -- 172.21.15.78:0/371908235 >> 172.21.15.78:0/371908235 conn(0x7f937406b270 msgr2=0x7f937406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 -- 172.21.15.78:0/371908235 shutdown_connections 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.027+0000 7f937a3f4640 1 -- 172.21.15.78:0/371908235 wait complete. 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.028+0000 7f937a3f4640 1 Processor -- start 2024-02-19T15:03:20.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 -- start start 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93740c25e0 0x7f9374149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f937414bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93740c5be0 0x7f93740a72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937414c2b0 con 0x7f93740c29c0 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740a84c0 con 0x7f93740c25e0 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f937a3f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93740a87a0 con 0x7f93740c5be0 2024-02-19T15:03:20.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f93737fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f937414bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f93737fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f937414bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44616/0 (socket says 172.21.15.78:44616) 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f93737fe640 1 -- 172.21.15.78:0/1956984563 learned_addr learned my addr 172.21.15.78:0/1956984563 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f9378991640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93740c5be0 0x7f93740a72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.029+0000 7f9373fff640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93740c25e0 0x7f9374149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f93737fe640 1 -- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93740c5be0 msgr2=0x7f93740a72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f93737fe640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93740c5be0 0x7f93740a72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f93737fe640 1 -- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93740c25e0 msgr2=0x7f9374149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f93737fe640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93740c25e0 0x7f9374149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f93737fe640 1 -- 172.21.15.78:0/1956984563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9368009400 con 0x7f93740c29c0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f93737fe640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f937414bd70 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f9368034040 tx=0x7f93680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368004230 con 0x7f93740c29c0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368004390 con 0x7f93740c29c0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93740a8a80 con 0x7f93740c29c0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368003830 con 0x7f93740c29c0 2024-02-19T15:03:20.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.030+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93740a6d90 con 0x7f93740c29c0 2024-02-19T15:03:20.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.031+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9374074990 con 0x7f93740c29c0 2024-02-19T15:03:20.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.032+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9368003990 con 0x7f93740c29c0 2024-02-19T15:03:20.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.032+0000 7f9370ff9640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f936006b840 0x7f936006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:20.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.032+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f93680ac2c0 con 0x7f93740c29c0 2024-02-19T15:03:20.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.032+0000 7f9373fff640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f936006b840 0x7f936006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:20.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.032+0000 7f9373fff640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f936006b840 0x7f936006dd00 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9364007fd0 tx=0x7f9364007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:20.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.036+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9368076cb0 con 0x7f93740c29c0 2024-02-19T15:03:20.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.243+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93740a6f20 con 0x7f93740c29c0 2024-02-19T15:03:20.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.243+0000 7f9370ff9640 1 -- 172.21.15.78:0/1956984563 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f936807a370 con 0x7f93740c29c0 2024-02-19T15:03:20.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:20.246 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:20.247 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:20.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f936006b840 msgr2=0x7f936006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f936006b840 0x7f936006dd00 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9364007fd0 tx=0x7f9364007de0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 msgr2=0x7f937414bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f937414bd70 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f9368034040 tx=0x7f93680046c0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 shutdown_connections 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f936006b840 0x7f936006dd00 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93740c29c0 0x7f937414bd70 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93740c5be0 0x7f93740a72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 --2- 172.21.15.78:0/1956984563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93740c25e0 0x7f9374149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.245+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 >> 172.21.15.78:0/1956984563 conn(0x7f937406b270 msgr2=0x7f9374155050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.246+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 shutdown_connections 2024-02-19T15:03:20.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:20.246+0000 7f937a3f4640 1 -- 172.21.15.78:0/1956984563 wait complete. 2024-02-19T15:03:20.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:20.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:21.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.772+0000 7fa4a3577640 1 Processor -- start 2024-02-19T15:03:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.772+0000 7fa4a3577640 1 -- start start 2024-02-19T15:03:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a3577640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa49c155e00 0x7fa49c0c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a3577640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0c7df0 con 0x7fa49c0c39e0 2024-02-19T15:03:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a3577640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0c77c0 con 0x7fa49c0c67e0 2024-02-19T15:03:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a3577640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49c0c4310 con 0x7fa49c155e00 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a1b14640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa49c0c39e0 0x7fa49c15f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44862/0 (socket says 172.21.15.78:44862) 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a1b14640 1 -- 172.21.15.78:0/1039785114 learned_addr learned my addr 172.21.15.78:0/1039785114 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a1313640 1 --2- 172.21.15.78:0/1039785114 >> v2:172.21.15.150:3300/0 conn(0x7fa49c155e00 0x7fa49c0c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168180164 0 0) 0x7fa49c0c7df0 con 0x7fa49c0c39e0 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa488003520 con 0x7fa49c0c39e0 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008637518 0 0) 0x7fa49c0c77c0 con 0x7fa49c0c67e0 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c0c7df0 con 0x7fa49c0c67e0 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a1313640 1 -- 172.21.15.78:0/1039785114 >> v1:172.21.15.150:6789/0 conn(0x7fa49c0c67e0 legacy=0x7fa49c06c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a1313640 1 -- 172.21.15.78:0/1039785114 >> v1:172.21.15.78:6789/0 conn(0x7fa49c0c39e0 legacy=0x7fa49c15f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.773+0000 7fa4a1313640 1 -- 172.21.15.78:0/1039785114 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0c70d0 con 0x7fa49c155e00 2024-02-19T15:03:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.774+0000 7fa4a1313640 1 --2- 172.21.15.78:0/1039785114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c155e00 0x7fa49c0c34a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa4980097a0 tx=0x7fa49802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e209396a8f0fd247 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.774+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498031830 con 0x7fa49c155e00 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.774+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa498031990 con 0x7fa49c155e00 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.774+0000 7fa493fff640 1 -- 172.21.15.78:0/1039785114 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498031d20 con 0x7fa49c155e00 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 -- 172.21.15.78:0/1039785114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c155e00 msgr2=0x7fa49c0c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1039785114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c155e00 0x7fa49c0c34a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa4980097a0 tx=0x7fa49802b8d0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 -- 172.21.15.78:0/1039785114 shutdown_connections 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1039785114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c155e00 0x7fa49c0c34a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 -- 172.21.15.78:0/1039785114 >> 172.21.15.78:0/1039785114 conn(0x7fa49c06b700 msgr2=0x7fa49c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 -- 172.21.15.78:0/1039785114 shutdown_connections 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.775+0000 7fa4a3577640 1 -- 172.21.15.78:0/1039785114 wait complete. 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.776+0000 7fa4a3577640 1 Processor -- start 2024-02-19T15:03:21.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.776+0000 7fa4a3577640 1 -- start start 2024-02-19T15:03:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a3577640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c0c39e0 0x7fa49c099cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a3577640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa49c0c67e0 0x7fa49c0a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a3577640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 0x7fa49c0a9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a3577640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0ad220 con 0x7fa49c155e00 2024-02-19T15:03:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a3577640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c1ea400 con 0x7fa49c0c39e0 2024-02-19T15:03:21.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a3577640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0a95d0 con 0x7fa49c0c67e0 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 0x7fa49c0a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1313640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c0c39e0 0x7fa49c099cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 0x7fa49c0a9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44622/0 (socket says 172.21.15.78:44622) 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 -- 172.21.15.78:0/1741079656 learned_addr learned my addr 172.21.15.78:0/1741079656 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a0b12640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa49c0c67e0 0x7fa49c0a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 -- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa49c0c67e0 msgr2=0x7fa49c0a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa49c0c67e0 0x7fa49c0a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 -- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c0c39e0 msgr2=0x7fa49c099cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c0c39e0 0x7fa49c099cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 -- 172.21.15.78:0/1741079656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498009400 con 0x7fa49c155e00 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa4a1b14640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 0x7fa49c0a9090 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7fa49400caa0 tx=0x7fa49400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.777+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49400ac00 con 0x7fa49c155e00 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.778+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49400ad80 con 0x7fa49c155e00 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.778+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494012640 con 0x7fa49c155e00 2024-02-19T15:03:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.778+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c1ea6c0 con 0x7fa49c155e00 2024-02-19T15:03:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.778+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c1eac80 con 0x7fa49c155e00 2024-02-19T15:03:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.779+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49c0c6cb0 con 0x7fa49c155e00 2024-02-19T15:03:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.779+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa494007070 con 0x7fa49c155e00 2024-02-19T15:03:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.779+0000 7fa491ffb640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa48006b810 0x7fa48006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.780+0000 7fa4a1313640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa48006b810 0x7fa48006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.780+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa49408a370 con 0x7fa49c155e00 2024-02-19T15:03:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.780+0000 7fa4a1313640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa48006b810 0x7fa48006dcd0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa498036000 tx=0x7fa49803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:21.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.784+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa494054bb0 con 0x7fa49c155e00 2024-02-19T15:03:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.990+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa49c074990 con 0x7fa49c155e00 2024-02-19T15:03:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.990+0000 7fa491ffb640 1 -- 172.21.15.78:0/1741079656 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fa494017d90 con 0x7fa49c155e00 2024-02-19T15:03:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:21.994 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:21.994 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa48006b810 msgr2=0x7fa48006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa48006b810 0x7fa48006dcd0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa498036000 tx=0x7fa49803c040 comp rx=0 tx=0).stop 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 msgr2=0x7fa49c0a9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 0x7fa49c0a9090 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7fa49400caa0 tx=0x7fa49400cf60 comp rx=0 tx=0).stop 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 shutdown_connections 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa48006b810 0x7fa48006dcd0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa49c155e00 0x7fa49c0a9090 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa49c0c67e0 0x7fa49c0a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 --2- 172.21.15.78:0/1741079656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa49c0c39e0 0x7fa49c099cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 >> 172.21.15.78:0/1741079656 conn(0x7fa49c06b700 msgr2=0x7fa49c0aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 shutdown_connections 2024-02-19T15:03:21.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:21.993+0000 7fa4a3577640 1 -- 172.21.15.78:0/1741079656 wait complete. 2024-02-19T15:03:22.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:22.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:23.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:23.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.517+0000 7f39092f4640 1 Processor -- start 2024-02-19T15:03:23.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.517+0000 7f39092f4640 1 -- start start 2024-02-19T15:03:23.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f39092f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3904150440 0x7f39040c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f39092f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3904150820 con 0x7f390406ed20 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f39092f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39041537d0 con 0x7f39040c3800 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f39092f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040c1b60 con 0x7f3904150440 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f39027fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f390406ed20 0x7f390406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44868/0 (socket says 172.21.15.78:44868) 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f39027fc640 1 -- 172.21.15.78:0/1611414906 learned_addr learned my addr 172.21.15.78:0/1611414906 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f3902ffd640 1 --2- 172.21.15.78:0/1611414906 >> v2:172.21.15.150:3300/0 conn(0x7f3904150440 0x7f39040c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f3901ffb640 1 -- 172.21.15.78:0/1611414906 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634851951 0 0) 0x7f3904150820 con 0x7f390406ed20 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f3901ffb640 1 -- 172.21.15.78:0/1611414906 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec003520 con 0x7f390406ed20 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f3902ffd640 1 -- 172.21.15.78:0/1611414906 >> v1:172.21.15.150:6789/0 conn(0x7f39040c3800 legacy=0x7f390406d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f3902ffd640 1 -- 172.21.15.78:0/1611414906 >> v1:172.21.15.78:6789/0 conn(0x7f390406ed20 legacy=0x7f390406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.518+0000 7f3902ffd640 1 -- 172.21.15.78:0/1611414906 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904153d10 con 0x7f3904150440 2024-02-19T15:03:23.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.519+0000 7f3902ffd640 1 --2- 172.21.15.78:0/1611414906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904150440 0x7f39040c32c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3904153960 tx=0x7f38f402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3efb4ea24de2a660 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.519+0000 7f3901ffb640 1 -- 172.21.15.78:0/1611414906 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f402cce0 con 0x7f3904150440 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.519+0000 7f3901ffb640 1 -- 172.21.15.78:0/1611414906 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f402ce40 con 0x7f3904150440 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f3901ffb640 1 -- 172.21.15.78:0/1611414906 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4034660 con 0x7f3904150440 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 -- 172.21.15.78:0/1611414906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904150440 msgr2=0x7f39040c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 --2- 172.21.15.78:0/1611414906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904150440 0x7f39040c32c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3904153960 tx=0x7f38f402bf30 comp rx=0 tx=0).stop 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 -- 172.21.15.78:0/1611414906 shutdown_connections 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 --2- 172.21.15.78:0/1611414906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3904150440 0x7f39040c32c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 -- 172.21.15.78:0/1611414906 >> 172.21.15.78:0/1611414906 conn(0x7f390406b270 msgr2=0x7f390406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 -- 172.21.15.78:0/1611414906 shutdown_connections 2024-02-19T15:03:23.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.520+0000 7f39092f4640 1 -- 172.21.15.78:0/1611414906 wait complete. 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.521+0000 7f39092f4640 1 Processor -- start 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.521+0000 7f39092f4640 1 -- start start 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39092f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 0x7f39041471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39092f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39040c3800 0x7f39041564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39092f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3904150440 0x7f3904152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39092f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39041de930 con 0x7f390406ed20 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39092f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39041e8a50 con 0x7f39040c3800 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39092f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3904153230 con 0x7f3904150440 2024-02-19T15:03:23.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f3902ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 0x7f39041471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f3902ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 0x7f39041471b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44638/0 (socket says 172.21.15.78:44638) 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f3902ffd640 1 -- 172.21.15.78:0/1974510375 learned_addr learned my addr 172.21.15.78:0/1974510375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39027fc640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39040c3800 0x7f39041564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.522+0000 7f39037fe640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3904150440 0x7f3904152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f3902ffd640 1 -- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3904150440 msgr2=0x7f3904152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f3902ffd640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3904150440 0x7f3904152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f3902ffd640 1 -- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39040c3800 msgr2=0x7f39041564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f3902ffd640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39040c3800 0x7f39041564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f3902ffd640 1 -- 172.21.15.78:0/1974510375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4009a70 con 0x7f390406ed20 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f3902ffd640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 0x7f39041471b0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f38f4037040 tx=0x7f38f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f403d600 con 0x7f390406ed20 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f403d760 con 0x7f390406ed20 2024-02-19T15:03:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39041551e0 con 0x7f390406ed20 2024-02-19T15:03:23.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f403db10 con 0x7f390406ed20 2024-02-19T15:03:23.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.523+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39041dfd80 con 0x7f390406ed20 2024-02-19T15:03:23.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.524+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3904074990 con 0x7f390406ed20 2024-02-19T15:03:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.525+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f38f403b040 con 0x7f390406ed20 2024-02-19T15:03:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.525+0000 7f38eb7fe640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f006b810 0x7f38f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.525+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f38f40ac240 con 0x7f390406ed20 2024-02-19T15:03:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.525+0000 7f39027fc640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f006b810 0x7f38f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:23.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.525+0000 7f39027fc640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f006b810 0x7f38f006dcd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f39041476f0 tx=0x7f38f8007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.529+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f38f4076b10 con 0x7f390406ed20 2024-02-19T15:03:23.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.734+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39040c4a90 con 0x7f390406ed20 2024-02-19T15:03:23.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.734+0000 7f38eb7fe640 1 -- 172.21.15.78:0/1974510375 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f38f4039d90 con 0x7f390406ed20 2024-02-19T15:03:23.737 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:23.737 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:23.738 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:23.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f006b810 msgr2=0x7f38f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f006b810 0x7f38f006dcd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f39041476f0 tx=0x7f38f8007d30 comp rx=0 tx=0).stop 2024-02-19T15:03:23.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 msgr2=0x7f39041471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 0x7f39041471b0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f38f4037040 tx=0x7f38f4004a40 comp rx=0 tx=0).stop 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 shutdown_connections 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f38f006b810 0x7f38f006dcd0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3904150440 0x7f3904152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39040c3800 0x7f39041564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 --2- 172.21.15.78:0/1974510375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f390406ed20 0x7f39041471b0 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 >> 172.21.15.78:0/1974510375 conn(0x7f390406b270 msgr2=0x7f390414ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 shutdown_connections 2024-02-19T15:03:23.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:23.736+0000 7f39092f4640 1 -- 172.21.15.78:0/1974510375 wait complete. 2024-02-19T15:03:24.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:24.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:25.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:25.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.269+0000 7f50541c9640 1 Processor -- start 2024-02-19T15:03:25.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.270+0000 7f50541c9640 1 -- start start 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.270+0000 7f50541c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f504c156b90 0x7f504c06d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.270+0000 7f50541c9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f504c0c8e90 0x7f504c0c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f50541c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c156f70 con 0x7f504c0c7b40 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f50541c9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c15d7a0 con 0x7f504c0c8e90 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f50541c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c15cb20 con 0x7f504c156b90 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f504c0c8e90 0x7f504c0c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f504c0c8e90 0x7f504c0c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44652/0 (socket says 172.21.15.78:44652) 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 -- 172.21.15.78:0/1625632115 learned_addr learned my addr 172.21.15.78:0/1625632115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051f65640 1 --2- 172.21.15.78:0/1625632115 >> v2:172.21.15.150:3300/0 conn(0x7f504c156b90 0x7f504c06d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 -- 172.21.15.78:0/1625632115 >> v2:172.21.15.150:3300/0 conn(0x7f504c156b90 msgr2=0x7f504c06d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:25.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 --2- 172.21.15.78:0/1625632115 >> v2:172.21.15.150:3300/0 conn(0x7f504c156b90 0x7f504c06d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 -- 172.21.15.78:0/1625632115 >> v1:172.21.15.150:6789/0 conn(0x7f504c0c7b40 legacy=0x7f504c06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:25.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.271+0000 7f5051764640 1 -- 172.21.15.78:0/1625632115 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c15ce00 con 0x7f504c0c8e90 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.272+0000 7f5051764640 1 --2- 172.21.15.78:0/1625632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c0c8e90 0x7f504c0c7600 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f504c08ba90 tx=0x7f504002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c432f7244b363625 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.272+0000 7f5050f63640 1 -- 172.21.15.78:0/1625632115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040031870 con 0x7f504c0c8e90 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.272+0000 7f5050f63640 1 -- 172.21.15.78:0/1625632115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50400319d0 con 0x7f504c0c8e90 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.272+0000 7f5050f63640 1 -- 172.21.15.78:0/1625632115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040031d60 con 0x7f504c0c8e90 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 -- 172.21.15.78:0/1625632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c0c8e90 msgr2=0x7f504c0c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 --2- 172.21.15.78:0/1625632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c0c8e90 0x7f504c0c7600 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f504c08ba90 tx=0x7f504002b910 comp rx=0 tx=0).stop 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 -- 172.21.15.78:0/1625632115 shutdown_connections 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 --2- 172.21.15.78:0/1625632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c0c8e90 0x7f504c0c7600 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 --2- 172.21.15.78:0/1625632115 >> v2:172.21.15.150:3300/0 conn(0x7f504c156b90 0x7f504c06d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 -- 172.21.15.78:0/1625632115 >> 172.21.15.78:0/1625632115 conn(0x7f504c0cac90 msgr2=0x7f504c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:25.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 -- 172.21.15.78:0/1625632115 shutdown_connections 2024-02-19T15:03:25.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.273+0000 7f50541c9640 1 -- 172.21.15.78:0/1625632115 wait complete. 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.274+0000 7f50541c9640 1 Processor -- start 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.274+0000 7f50541c9640 1 -- start start 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f50541c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f504c0c7b40 0x7f504c159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f50541c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f504c0c8e90 0x7f504c1efa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f50541c9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 0x7f504c0ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f50541c9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c159800 con 0x7f504c156b90 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f50541c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c1eef40 con 0x7f504c0c8e90 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f50541c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c1f2f60 con 0x7f504c0c7b40 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 0x7f504c0ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 0x7f504c0ac4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44666/0 (socket says 172.21.15.78:44666) 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 -- 172.21.15.78:0/530732758 learned_addr learned my addr 172.21.15.78:0/530732758 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5051f65640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f504c0c7b40 0x7f504c159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5051764640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f504c0c8e90 0x7f504c1efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 -- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f504c0c7b40 msgr2=0x7f504c159190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f504c0c7b40 0x7f504c159190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 -- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f504c0c8e90 msgr2=0x7f504c1efa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f504c0c8e90 0x7f504c1efa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 -- 172.21.15.78:0/530732758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50400094d0 con 0x7f504c156b90 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.275+0000 7f5052766640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 0x7f504c0ac4c0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f503c001eb0 tx=0x7f503c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.276+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0052d0 con 0x7f504c156b90 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.276+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c005430 con 0x7f504c156b90 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.276+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c1f3240 con 0x7f504c156b90 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.276+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c003810 con 0x7f504c156b90 2024-02-19T15:03:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.276+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504c1ef4e0 con 0x7f504c156b90 2024-02-19T15:03:25.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.277+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504c074990 con 0x7f504c156b90 2024-02-19T15:03:25.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.277+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f503c003970 con 0x7f504c156b90 2024-02-19T15:03:25.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.278+0000 7f504a7fc640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503806b810 0x7f503806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:25.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.278+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f503c089920 con 0x7f504c156b90 2024-02-19T15:03:25.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.278+0000 7f5051f65640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503806b810 0x7f503806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:25.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.278+0000 7f5051f65640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503806b810 0x7f503806dcd0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5044011340 tx=0x7f504400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:25.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.282+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f503c0541f0 con 0x7f504c156b90 2024-02-19T15:03:25.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.488+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f504c0b07f0 con 0x7f504c156b90 2024-02-19T15:03:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.489+0000 7f504a7fc640 1 -- 172.21.15.78:0/530732758 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f503c017d90 con 0x7f504c156b90 2024-02-19T15:03:25.492 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:25.492 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503806b810 msgr2=0x7f503806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503806b810 0x7f503806dcd0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5044011340 tx=0x7f504400b040 comp rx=0 tx=0).stop 2024-02-19T15:03:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 msgr2=0x7f504c0ac4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 0x7f504c0ac4c0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f503c001eb0 tx=0x7f503c00c890 comp rx=0 tx=0).stop 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 shutdown_connections 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f503806b810 0x7f503806dcd0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f504c156b90 0x7f504c0ac4c0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f504c0c8e90 0x7f504c1efa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 --2- 172.21.15.78:0/530732758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f504c0c7b40 0x7f504c159190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 >> 172.21.15.78:0/530732758 conn(0x7f504c0cac90 msgr2=0x7f504c15a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 shutdown_connections 2024-02-19T15:03:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:25.491+0000 7f50541c9640 1 -- 172.21.15.78:0/530732758 wait complete. 2024-02-19T15:03:25.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:25.907 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:26.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:27.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.016+0000 7f9604da1640 1 Processor -- start 2024-02-19T15:03:27.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.016+0000 7f9604da1640 1 -- start start 2024-02-19T15:03:27.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f9604da1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 0x7f96000c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:27.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f9604da1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96000c5270 0x7f96000719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:27.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f9604da1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96000c1120 con 0x7f9600071f10 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f9604da1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96000c6080 con 0x7f96000c5270 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f9604da1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96000c57e0 con 0x7f960014ece0 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95f7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96000c5270 0x7f96000719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 0x7f96000c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95fed9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9600071f10 0x7f96000c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:58594/0 (socket says 172.21.15.78:58594) 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 0x7f96000c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:56232/0 (socket says 172.21.15.78:56232) 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95f7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96000c5270 0x7f96000719d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:52792/0 (socket says 172.21.15.78:52792) 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95fed9d640 1 -- 172.21.15.78:0/2438583529 learned_addr learned my addr 172.21.15.78:0/2438583529 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95f7fff640 1 -- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 msgr2=0x7f96000c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95f7fff640 1 --2- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 0x7f96000c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95f7fff640 1 -- 172.21.15.78:0/2438583529 >> v1:172.21.15.150:6789/0 conn(0x7f9600071f10 legacy=0x7f96000c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:27.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95fdd9b640 1 -- 172.21.15.78:0/2438583529 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466494331 0 0) 0x7f96000c1120 con 0x7f9600071f10 2024-02-19T15:03:27.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95f7fff640 1 -- 172.21.15.78:0/2438583529 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96000c5ac0 con 0x7f96000c5270 2024-02-19T15:03:27.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.017+0000 7f95fe59c640 1 --2- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 0x7f96000c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.018+0000 7f95f7fff640 1 --2- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96000c5270 0x7f96000719d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f95e80097a0 tx=0x7f95e802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3881dabec879b579 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.018+0000 7f95fdd9b640 1 -- 172.21.15.78:0/2438583529 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8031830 con 0x7f96000c5270 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.018+0000 7f95fdd9b640 1 -- 172.21.15.78:0/2438583529 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e8031990 con 0x7f96000c5270 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.018+0000 7f95fdd9b640 1 -- 172.21.15.78:0/2438583529 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8031d20 con 0x7f96000c5270 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 -- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96000c5270 msgr2=0x7f96000719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 --2- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96000c5270 0x7f96000719d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f95e80097a0 tx=0x7f95e802b8d0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 -- 172.21.15.78:0/2438583529 shutdown_connections 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 --2- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96000c5270 0x7f96000719d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 --2- 172.21.15.78:0/2438583529 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f960014ece0 0x7f96000c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 -- 172.21.15.78:0/2438583529 >> 172.21.15.78:0/2438583529 conn(0x7f960006b270 msgr2=0x7f960006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 -- 172.21.15.78:0/2438583529 shutdown_connections 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.019+0000 7f9604da1640 1 -- 172.21.15.78:0/2438583529 wait complete. 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.020+0000 7f9604da1640 1 Processor -- start 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.020+0000 7f9604da1640 1 -- start start 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f9604da1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 0x7f96000bd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f9604da1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96000c5270 0x7f9600143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f9604da1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f960014ece0 0x7f9600142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f9604da1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96000bd8c0 con 0x7f960014ece0 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f9604da1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001433b0 con 0x7f9600071f10 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f9604da1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960014d530 con 0x7f96000c5270 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 0x7f96000bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f960014ece0 0x7f9600142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 0x7f96000bd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:52794/0 (socket says 172.21.15.78:52794) 2024-02-19T15:03:27.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95f7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96000c5270 0x7f9600143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 -- 172.21.15.78:0/1231566702 learned_addr learned my addr 172.21.15.78:0/1231566702 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 -- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96000c5270 msgr2=0x7f9600143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96000c5270 0x7f9600143840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 -- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f960014ece0 msgr2=0x7f9600142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f960014ece0 0x7f9600142930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 -- 172.21.15.78:0/1231566702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f0009400 con 0x7f9600071f10 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.021+0000 7f95fe59c640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 0x7f96000bd380 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f95f000e990 tx=0x7f95f000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.022+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f000c9e0 con 0x7f9600071f10 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.022+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f000cb40 con 0x7f9600071f10 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.022+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f000ced0 con 0x7f9600071f10 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.022+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e8009400 con 0x7f9600071f10 2024-02-19T15:03:27.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.022+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96000bbc00 con 0x7f9600071f10 2024-02-19T15:03:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.023+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9600074990 con 0x7f9600071f10 2024-02-19T15:03:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.023+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f95f0013470 con 0x7f9600071f10 2024-02-19T15:03:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.023+0000 7f95f6ffd640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95e006b810 0x7f95e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:27.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.023+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f95f008a620 con 0x7f9600071f10 2024-02-19T15:03:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.023+0000 7f95f7fff640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95e006b810 0x7f95e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:27.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.024+0000 7f95f7fff640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95e006b810 0x7f95e006dcd0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f95e8036000 tx=0x7f95e803e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:27.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.028+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f95f0054e40 con 0x7f9600071f10 2024-02-19T15:03:27.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.233+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f960006b650 con 0x7f9600071f10 2024-02-19T15:03:27.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.234+0000 7f95f6ffd640 1 -- 172.21.15.78:0/1231566702 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f95f0017d90 con 0x7f9600071f10 2024-02-19T15:03:27.237 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:27.237 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:27.237 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95e006b810 msgr2=0x7f95e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95e006b810 0x7f95e006dcd0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f95e8036000 tx=0x7f95e803e040 comp rx=0 tx=0).stop 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 msgr2=0x7f96000bd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 0x7f96000bd380 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f95f000e990 tx=0x7f95f000ee50 comp rx=0 tx=0).stop 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 shutdown_connections 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f95e006b810 0x7f95e006dcd0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f960014ece0 0x7f9600142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96000c5270 0x7f9600143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 --2- 172.21.15.78:0/1231566702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9600071f10 0x7f96000bd380 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:27.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 >> 172.21.15.78:0/1231566702 conn(0x7f960006b270 msgr2=0x7f96000c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:27.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 shutdown_connections 2024-02-19T15:03:27.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:27.236+0000 7f9604da1640 1 -- 172.21.15.78:0/1231566702 wait complete. 2024-02-19T15:03:27.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:27.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:28.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:03:28.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:03:28.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:28.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.765+0000 7fb3f89bd640 1 Processor -- start 2024-02-19T15:03:28.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.765+0000 7fb3f89bd640 1 -- start start 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f89bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4071850 0x7fb3f41536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f89bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f4156e90 con 0x7fb3f4070180 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f89bd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40c2fc0 con 0x7fb3f4150320 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f89bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4157150 con 0x7fb3f4071850 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb3f4070180 0x7fb3f4070610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48676/0 (socket says 172.21.15.78:48676) 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f2d9d640 1 -- 172.21.15.78:0/4157959973 learned_addr learned my addr 172.21.15.78:0/4157959973 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.766+0000 7fb3f1d9b640 1 --2- 172.21.15.78:0/4157959973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4071850 0x7fb3f41536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347811772 0 0) 0x7fb3f4156e90 con 0x7fb3f4070180 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3dc003520 con 0x7fb3f4070180 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386169542 0 0) 0x7fb3f40c2fc0 con 0x7fb3f4150320 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f4156e90 con 0x7fb3f4150320 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/4157959973 >> v1:172.21.15.150:6789/0 conn(0x7fb3f4150320 legacy=0x7fb3f4152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/4157959973 >> v1:172.21.15.78:6789/0 conn(0x7fb3f4070180 legacy=0x7fb3f4070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.767+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/4157959973 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f40c3280 con 0x7fb3f4071850 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f1d9b640 1 --2- 172.21.15.78:0/4157959973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4071850 0x7fb3f41536b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80097d0 tx=0x7fb3e802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eabdf5ff0fd64ec server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8031800 con 0x7fb3f4071850 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e8031960 con 0x7fb3f4071850 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f159a640 1 -- 172.21.15.78:0/4157959973 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8031cf0 con 0x7fb3f4071850 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f89bd640 1 -- 172.21.15.78:0/4157959973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4071850 msgr2=0x7fb3f41536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/4157959973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4071850 0x7fb3f41536b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80097d0 tx=0x7fb3e802b8a0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f89bd640 1 -- 172.21.15.78:0/4157959973 shutdown_connections 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/4157959973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4071850 0x7fb3f41536b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f89bd640 1 -- 172.21.15.78:0/4157959973 >> 172.21.15.78:0/4157959973 conn(0x7fb3f40aa370 msgr2=0x7fb3f40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.768+0000 7fb3f89bd640 1 -- 172.21.15.78:0/4157959973 shutdown_connections 2024-02-19T15:03:28.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.769+0000 7fb3f89bd640 1 -- 172.21.15.78:0/4157959973 wait complete. 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.770+0000 7fb3f89bd640 1 Processor -- start 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.770+0000 7fb3f89bd640 1 -- start start 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f89bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4070180 0x7fb3f414e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f89bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 0x7fb3f4150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f89bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3f4150320 0x7fb3f4154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f89bd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f414f6e0 con 0x7fb3f4071850 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f89bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f41e9270 con 0x7fb3f4150320 2024-02-19T15:03:28.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f89bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f414f120 con 0x7fb3f4070180 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 0x7fb3f4150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 0x7fb3f4150b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46092/0 (socket says 172.21.15.78:46092) 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/2207342781 learned_addr learned my addr 172.21.15.78:0/2207342781 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f259c640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4070180 0x7fb3f414e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f2d9d640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3f4150320 0x7fb3f4154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4070180 msgr2=0x7fb3f414e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4070180 0x7fb3f414e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3f4150320 msgr2=0x7fb3f4154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3f4150320 0x7fb3f4154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 -- 172.21.15.78:0/2207342781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e8009400 con 0x7fb3f4071850 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f259c640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4070180 0x7fb3f414e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3f1d9b640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 0x7fb3f4150b80 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80098f0 tx=0x7fb3e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e803bc70 con 0x7fb3f4071850 2024-02-19T15:03:28.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.771+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e803be10 con 0x7fb3f4071850 2024-02-19T15:03:28.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.772+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f414f400 con 0x7fb3f4071850 2024-02-19T15:03:28.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.772+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8045630 con 0x7fb3f4071850 2024-02-19T15:03:28.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.772+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f4152a80 con 0x7fb3f4071850 2024-02-19T15:03:28.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.773+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4074990 con 0x7fb3f4071850 2024-02-19T15:03:28.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.773+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb3e804e450 con 0x7fb3f4071850 2024-02-19T15:03:28.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.773+0000 7fb3e2ffd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3d406b840 0x7fb3d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:28.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.774+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb3e802e070 con 0x7fb3f4071850 2024-02-19T15:03:28.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.774+0000 7fb3f259c640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3d406b840 0x7fb3d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:28.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.774+0000 7fb3f259c640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3d406b840 0x7fb3d406dd00 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4007fd0 tx=0x7fb3e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:28.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.778+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb3e8076da0 con 0x7fb3f4071850 2024-02-19T15:03:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.991+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f4071630 con 0x7fb3f4071850 2024-02-19T15:03:28.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.992+0000 7fb3e2ffd640 1 -- 172.21.15.78:0/2207342781 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fb3e8035d90 con 0x7fb3f4071850 2024-02-19T15:03:28.995 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:28.995 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:28.996 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3d406b840 msgr2=0x7fb3d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3d406b840 0x7fb3d406dd00 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4007fd0 tx=0x7fb3e4009040 comp rx=0 tx=0).stop 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 msgr2=0x7fb3f4150b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 0x7fb3f4150b80 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80098f0 tx=0x7fb3e80046c0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 shutdown_connections 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3d406b840 0x7fb3d406dd00 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb3f4071850 0x7fb3f4150b80 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3f4150320 0x7fb3f4154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 --2- 172.21.15.78:0/2207342781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3f4070180 0x7fb3f414e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:28.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 >> 172.21.15.78:0/2207342781 conn(0x7fb3f40aa370 msgr2=0x7fb3f40c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:28.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 shutdown_connections 2024-02-19T15:03:28.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:28.994+0000 7fb3f89bd640 1 -- 172.21.15.78:0/2207342781 wait complete. 2024-02-19T15:03:29.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:29.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:30.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:30.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.518+0000 7fd62c023640 1 Processor -- start 2024-02-19T15:03:30.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd62c023640 1 -- start start 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd62c023640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd62414ef50 0x7fd62414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd62c023640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6240c35c0 con 0x7fd6240c78a0 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd62c023640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6240c38a0 con 0x7fd6240727d0 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd62c023640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6240c4bc0 con 0x7fd62414ef50 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd629dbf640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd6240c78a0 0x7fd6240c44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:48678/0 (socket says 172.21.15.78:48678) 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd629dbf640 1 -- 172.21.15.78:0/3681124175 learned_addr learned my addr 172.21.15.78:0/3681124175 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:30.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.519+0000 7fd6295be640 1 --2- 172.21.15.78:0/3681124175 >> v2:172.21.15.150:3300/0 conn(0x7fd62414ef50 0x7fd62414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377609153 0 0) 0x7fd6240c35c0 con 0x7fd6240c78a0 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd610003520 con 0x7fd6240c78a0 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983187550 0 0) 0x7fd6240c38a0 con 0x7fd6240727d0 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6240c35c0 con 0x7fd6240727d0 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd6295be640 1 -- 172.21.15.78:0/3681124175 >> v1:172.21.15.150:6789/0 conn(0x7fd6240727d0 legacy=0x7fd624072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd6295be640 1 -- 172.21.15.78:0/3681124175 >> v1:172.21.15.78:6789/0 conn(0x7fd6240c78a0 legacy=0x7fd6240c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.520+0000 7fd6295be640 1 -- 172.21.15.78:0/3681124175 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240c3b80 con 0x7fd62414ef50 2024-02-19T15:03:30.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd6295be640 1 --2- 172.21.15.78:0/3681124175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd62414ef50 0x7fd62414f3e0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd6140097a0 tx=0x7fd61402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc41a09f802cdae6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:30.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614030610 con 0x7fd62414ef50 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd614030770 con 0x7fd62414ef50 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd628dbd640 1 -- 172.21.15.78:0/3681124175 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd614030b00 con 0x7fd62414ef50 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd62c023640 1 -- 172.21.15.78:0/3681124175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd62414ef50 msgr2=0x7fd62414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd62c023640 1 --2- 172.21.15.78:0/3681124175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd62414ef50 0x7fd62414f3e0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd6140097a0 tx=0x7fd61402b960 comp rx=0 tx=0).stop 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd62c023640 1 -- 172.21.15.78:0/3681124175 shutdown_connections 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd62c023640 1 --2- 172.21.15.78:0/3681124175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd62414ef50 0x7fd62414f3e0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.521+0000 7fd62c023640 1 -- 172.21.15.78:0/3681124175 >> 172.21.15.78:0/3681124175 conn(0x7fd624066e70 msgr2=0x7fd6240a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.522+0000 7fd62c023640 1 -- 172.21.15.78:0/3681124175 shutdown_connections 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.522+0000 7fd62c023640 1 -- 172.21.15.78:0/3681124175 wait complete. 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 Processor -- start 2024-02-19T15:03:30.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 -- start start 2024-02-19T15:03:30.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 0x7fd6240becc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6240c78a0 0x7fd6241e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd62414ef50 0x7fd62406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6241de500 con 0x7fd6240727d0 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6241deca0 con 0x7fd6240c78a0 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62c023640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6240bf200 con 0x7fd62414ef50 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd629dbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 0x7fd6240becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd629dbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 0x7fd6240becc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46096/0 (socket says 172.21.15.78:46096) 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd629dbf640 1 -- 172.21.15.78:0/528191440 learned_addr learned my addr 172.21.15.78:0/528191440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd6295be640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6240c78a0 0x7fd6241e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.523+0000 7fd62a5c0640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd62414ef50 0x7fd62406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd629dbf640 1 -- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd62414ef50 msgr2=0x7fd62406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd629dbf640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd62414ef50 0x7fd62406ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd629dbf640 1 -- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6240c78a0 msgr2=0x7fd6241e8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd629dbf640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6240c78a0 0x7fd6241e8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd629dbf640 1 -- 172.21.15.78:0/528191440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd614009400 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd629dbf640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 0x7fd6240becc0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7fd62000cbf0 tx=0x7fd62000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62000ad50 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd620012400 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6241de6e0 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620012790 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.524+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6241e8660 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.525+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd624074990 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.526+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd620007070 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.526+0000 7fd61a7fc640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd60806b810 0x7fd60806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.526+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd62008a410 con 0x7fd6240727d0 2024-02-19T15:03:30.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.526+0000 7fd6295be640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd60806b810 0x7fd60806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:30.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.526+0000 7fd6295be640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd60806b810 0x7fd60806dcd0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd614036000 tx=0x7fd61403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:30.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.530+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd620054ce0 con 0x7fd6240727d0 2024-02-19T15:03:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.737+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd624152320 con 0x7fd6240727d0 2024-02-19T15:03:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.738+0000 7fd61a7fc640 1 -- 172.21.15.78:0/528191440 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fd620017d90 con 0x7fd6240727d0 2024-02-19T15:03:30.741 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:30.741 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd60806b810 msgr2=0x7fd60806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd60806b810 0x7fd60806dcd0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd614036000 tx=0x7fd61403b040 comp rx=0 tx=0).stop 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 msgr2=0x7fd6240becc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 0x7fd6240becc0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7fd62000cbf0 tx=0x7fd62000a430 comp rx=0 tx=0).stop 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 shutdown_connections 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd60806b810 0x7fd60806dcd0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd62414ef50 0x7fd62406ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6240c78a0 0x7fd6241e8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 --2- 172.21.15.78:0/528191440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240727d0 0x7fd6240becc0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 >> 172.21.15.78:0/528191440 conn(0x7fd624066e70 msgr2=0x7fd624150080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 shutdown_connections 2024-02-19T15:03:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:30.740+0000 7fd62c023640 1 -- 172.21.15.78:0/528191440 wait complete. 2024-02-19T15:03:31.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:31.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:32.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.265+0000 7f22d348f640 1 Processor -- start 2024-02-19T15:03:32.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.265+0000 7f22d348f640 1 -- start start 2024-02-19T15:03:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d348f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f22cc0ac390 0x7f22cc0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d348f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22cc0bdf20 0x7f22cc0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d348f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc158530 0x7f22cc06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d348f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22cc0c8de0 con 0x7f22cc0bdf20 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d348f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22cc155a60 con 0x7f22cc0ac390 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d348f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc155d40 con 0x7f22cc158530 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d0a2a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22cc0bdf20 0x7f22cc0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d0a2a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f22cc0bdf20 0x7f22cc0c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46102/0 (socket says 172.21.15.78:46102) 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d0a2a640 1 -- 172.21.15.78:0/1228459319 learned_addr learned my addr 172.21.15.78:0/1228459319 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d1a2c640 1 --2- 172.21.15.78:0/1228459319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc158530 0x7f22cc06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.266+0000 7f22d122b640 1 --2- 172.21.15.78:0/1228459319 >> v2:172.21.15.150:3300/0 conn(0x7f22cc0ac390 0x7f22cc0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22d0a2a640 1 -- 172.21.15.78:0/1228459319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc158530 msgr2=0x7f22cc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22d0a2a640 1 --2- 172.21.15.78:0/1228459319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc158530 0x7f22cc06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22d0a2a640 1 -- 172.21.15.78:0/1228459319 >> v2:172.21.15.150:3300/0 conn(0x7f22cc0ac390 msgr2=0x7f22cc0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22d0a2a640 1 --2- 172.21.15.78:0/1228459319 >> v2:172.21.15.150:3300/0 conn(0x7f22cc0ac390 0x7f22cc0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22d0a2a640 1 -- 172.21.15.78:0/1228459319 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc0c90a0 con 0x7f22cc0bdf20 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22d0a2a640 1 --2- 172.21.15.78:0/1228459319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc0bdf20 0x7f22cc0c55f0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0ad2a0 tx=0x7f22bc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603fb9646e4c32e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.267+0000 7f22c3fff640 1 -- 172.21.15.78:0/1228459319 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc031740 con 0x7f22cc0bdf20 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22c3fff640 1 -- 172.21.15.78:0/1228459319 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22bc0318a0 con 0x7f22cc0bdf20 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22c3fff640 1 -- 172.21.15.78:0/1228459319 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22bc031c30 con 0x7f22cc0bdf20 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 -- 172.21.15.78:0/1228459319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc0bdf20 msgr2=0x7f22cc0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 --2- 172.21.15.78:0/1228459319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc0bdf20 0x7f22cc0c55f0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0ad2a0 tx=0x7f22bc02b7e0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 -- 172.21.15.78:0/1228459319 shutdown_connections 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 --2- 172.21.15.78:0/1228459319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc158530 0x7f22cc06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 --2- 172.21.15.78:0/1228459319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc0bdf20 0x7f22cc0c55f0 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 --2- 172.21.15.78:0/1228459319 >> v2:172.21.15.150:3300/0 conn(0x7f22cc0ac390 0x7f22cc0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 -- 172.21.15.78:0/1228459319 >> 172.21.15.78:0/1228459319 conn(0x7f22cc0ca890 msgr2=0x7f22cc0caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 -- 172.21.15.78:0/1228459319 shutdown_connections 2024-02-19T15:03:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.268+0000 7f22d348f640 1 -- 172.21.15.78:0/1228459319 wait complete. 2024-02-19T15:03:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.269+0000 7f22d348f640 1 Processor -- start 2024-02-19T15:03:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.269+0000 7f22d348f640 1 -- start start 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d348f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22cc0ac390 0x7f22cc0c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d348f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc0bdf20 0x7f22cc0aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d348f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 0x7f22cc14ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d348f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc1eaa90 con 0x7f22cc158530 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d348f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc1eea70 con 0x7f22cc0ac390 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d348f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc1eefa0 con 0x7f22cc0bdf20 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 0x7f22cc14ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 0x7f22cc14ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46104/0 (socket says 172.21.15.78:46104) 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 -- 172.21.15.78:0/2630297169 learned_addr learned my addr 172.21.15.78:0/2630297169 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d0a2a640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc0bdf20 0x7f22cc0aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d122b640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22cc0ac390 0x7f22cc0c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 -- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc0bdf20 msgr2=0x7f22cc0aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc0bdf20 0x7f22cc0aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 -- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22cc0ac390 msgr2=0x7f22cc0c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.270+0000 7f22d1a2c640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22cc0ac390 0x7f22cc0c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22d1a2c640 1 -- 172.21.15.78:0/2630297169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc0094d0 con 0x7f22cc158530 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22d1a2c640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 0x7f22cc14ed80 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f22c400efc0 tx=0x7f22c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c4009ba0 con 0x7f22cc158530 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c400f040 con 0x7f22cc158530 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c40146c0 con 0x7f22cc158530 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc1eabf0 con 0x7f22cc158530 2024-02-19T15:03:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.271+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc1549e0 con 0x7f22cc158530 2024-02-19T15:03:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.272+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22cc1ead80 con 0x7f22cc158530 2024-02-19T15:03:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.272+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f22c4014820 con 0x7f22cc158530 2024-02-19T15:03:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.273+0000 7f22c1ffb640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22b006b810 0x7f22b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.273+0000 7f22d122b640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22b006b810 0x7f22b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.273+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f22c408ad70 con 0x7f22cc158530 2024-02-19T15:03:32.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.273+0000 7f22d122b640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22b006b810 0x7f22b006dcd0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009ac0 tx=0x7f22c8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:32.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.277+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f22c4055640 con 0x7f22cc158530 2024-02-19T15:03:32.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.487+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22cc074990 con 0x7f22cc158530 2024-02-19T15:03:32.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.487+0000 7f22c1ffb640 1 -- 172.21.15.78:0/2630297169 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f22c4018020 con 0x7f22cc158530 2024-02-19T15:03:32.490 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:32.491 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:32.491 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22b006b810 msgr2=0x7f22b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22b006b810 0x7f22b006dcd0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f22c8009ac0 tx=0x7f22c8009190 comp rx=0 tx=0).stop 2024-02-19T15:03:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 msgr2=0x7f22cc14ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 0x7f22cc14ed80 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f22c400efc0 tx=0x7f22c400c490 comp rx=0 tx=0).stop 2024-02-19T15:03:32.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 shutdown_connections 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f22b006b810 0x7f22b006dcd0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22cc158530 0x7f22cc14ed80 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22cc0bdf20 0x7f22cc0aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 --2- 172.21.15.78:0/2630297169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22cc0ac390 0x7f22cc0c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 >> 172.21.15.78:0/2630297169 conn(0x7f22cc0ca890 msgr2=0x7f22cc0c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 shutdown_connections 2024-02-19T15:03:32.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:32.489+0000 7f22d348f640 1 -- 172.21.15.78:0/2630297169 wait complete. 2024-02-19T15:03:32.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:32.905 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:33.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:34.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.011+0000 7f07ffbe2640 1 Processor -- start 2024-02-19T15:03:34.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.011+0000 7f07ffbe2640 1 -- start start 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07ffbe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f07f8155f90 0x7f07f806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07ffbe2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07f80c6d30 0x7f07f80c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07ffbe2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f80c6af0 con 0x7f07f80c3aa0 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07ffbe2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f80c7940 con 0x7f07f8155f90 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07ffbe2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f80c7c20 con 0x7f07f80c6d30 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f07f8155f90 0x7f07f806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f07f8155f90 0x7f07f806d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46114/0 (socket says 172.21.15.78:46114) 2024-02-19T15:03:34.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 -- 172.21.15.78:0/2348983329 learned_addr learned my addr 172.21.15.78:0/2348983329 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:34.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd17d640 1 --2- 172.21.15.78:0/2348983329 >> v2:172.21.15.150:3300/0 conn(0x7f07f80c6d30 0x7f07f80c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:34.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 -- 172.21.15.78:0/2348983329 >> v2:172.21.15.150:3300/0 conn(0x7f07f80c6d30 msgr2=0x7f07f80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:34.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 --2- 172.21.15.78:0/2348983329 >> v2:172.21.15.150:3300/0 conn(0x7f07f80c6d30 0x7f07f80c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 -- 172.21.15.78:0/2348983329 >> v1:172.21.15.150:6789/0 conn(0x7f07f80c3aa0 legacy=0x7f07f81588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:34.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.012+0000 7f07fd97e640 1 -- 172.21.15.78:0/2348983329 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80c7f00 con 0x7f07f8155f90 2024-02-19T15:03:34.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.013+0000 7f07fd97e640 1 --2- 172.21.15.78:0/2348983329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f806d550 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f07f80c2b60 tx=0x7f07f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2064c0f4eeb0f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:34.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.013+0000 7f07fc97c640 1 -- 172.21.15.78:0/2348983329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4031870 con 0x7f07f8155f90 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.013+0000 7f07fc97c640 1 -- 172.21.15.78:0/2348983329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f40319d0 con 0x7f07f8155f90 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.013+0000 7f07fc97c640 1 -- 172.21.15.78:0/2348983329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4031d60 con 0x7f07f8155f90 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2348983329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 msgr2=0x7f07f806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2348983329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f806d550 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f07f80c2b60 tx=0x7f07f402b910 comp rx=0 tx=0).stop 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2348983329 shutdown_connections 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2348983329 >> v2:172.21.15.150:3300/0 conn(0x7f07f80c6d30 0x7f07f80c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2348983329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f806d550 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2348983329 >> 172.21.15.78:0/2348983329 conn(0x7f07f806b700 msgr2=0x7f07f80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2348983329 shutdown_connections 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.014+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2348983329 wait complete. 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 Processor -- start 2024-02-19T15:03:34.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 -- start start 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f80c3aa0 0x7f07f809bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f80c6d30 0x7f07f80ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8071450 con 0x7f07f8155f90 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f809a140 con 0x7f07f80c3aa0 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.015+0000 7f07ffbe2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8070d90 con 0x7f07f80c6d30 2024-02-19T15:03:34.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f814f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46118/0 (socket says 172.21.15.78:46118) 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 -- 172.21.15.78:0/2288996714 learned_addr learned my addr 172.21.15.78:0/2288996714 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fd97e640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f80c3aa0 0x7f07f809bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fd17d640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f80c6d30 0x7f07f80ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 -- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f80c6d30 msgr2=0x7f07f80ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f80c6d30 0x7f07f80ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 -- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f80c3aa0 msgr2=0x7f07f809bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f80c3aa0 0x7f07f809bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 -- 172.21.15.78:0/2288996714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40094d0 con 0x7f07f8155f90 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07fe17f640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f814f870 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f07f000c7a0 tx=0x7f07f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0003b70 con 0x7f07f8155f90 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f0003cf0 con 0x7f07f8155f90 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f8071070 con 0x7f07f8155f90 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0003ea0 con 0x7f07f8155f90 2024-02-19T15:03:34.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.016+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f80a9cb0 con 0x7f07f8155f90 2024-02-19T15:03:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.017+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f8071590 con 0x7f07f8155f90 2024-02-19T15:03:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.018+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f07f000aa90 con 0x7f07f8155f90 2024-02-19T15:03:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.018+0000 7f07ee7fc640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07dc06b840 0x7f07dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.018+0000 7f07fd97e640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07dc06b840 0x7f07dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.018+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f07f008a140 con 0x7f07f8155f90 2024-02-19T15:03:34.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.019+0000 7f07fd97e640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07dc06b840 0x7f07dc06dd00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f07f4036000 tx=0x7f07f403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:34.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.023+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f07f8071590 con 0x7f07f8155f90 2024-02-19T15:03:34.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.228+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f8071590 con 0x7f07f8155f90 2024-02-19T15:03:34.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.229+0000 7f07ee7fc640 1 -- 172.21.15.78:0/2288996714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f07f8071590 con 0x7f07f8155f90 2024-02-19T15:03:34.232 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:34.232 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:34.232 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07dc06b840 msgr2=0x7f07dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07dc06b840 0x7f07dc06dd00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f07f4036000 tx=0x7f07f403c040 comp rx=0 tx=0).stop 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 msgr2=0x7f07f814f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f814f870 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f07f000c7a0 tx=0x7f07f000cc60 comp rx=0 tx=0).stop 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 shutdown_connections 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f07dc06b840 0x7f07dc06dd00 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07f8155f90 0x7f07f814f870 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07f80c6d30 0x7f07f80ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 --2- 172.21.15.78:0/2288996714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07f80c3aa0 0x7f07f809bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 >> 172.21.15.78:0/2288996714 conn(0x7f07f806b700 msgr2=0x7f07f80c8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 shutdown_connections 2024-02-19T15:03:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:34.231+0000 7f07ffbe2640 1 -- 172.21.15.78:0/2288996714 wait complete. 2024-02-19T15:03:34.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:34.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:35.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:35.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.756+0000 7f0e99165640 1 Processor -- start 2024-02-19T15:03:35.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.756+0000 7f0e99165640 1 -- start start 2024-02-19T15:03:35.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e99165640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e94152700 0x7f0e940c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e99165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e9415f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e99165640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e941551f0 con 0x7f0e940c5730 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e99165640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e94155490 con 0x7f0e94152700 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e99165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e94155770 con 0x7f0e940c8e00 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e9359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e9415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e92d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e94152700 0x7f0e940c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e9259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0e940c5730 0x7f0e9406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:34624/0 (socket says 172.21.15.78:34624) 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.757+0000 7f0e9259c640 1 -- 172.21.15.78:0/1931136946 learned_addr learned my addr 172.21.15.78:0/1931136946 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e9359e640 1 -- 172.21.15.78:0/1931136946 >> v2:172.21.15.150:3300/0 conn(0x7f0e94152700 msgr2=0x7f0e940c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e9359e640 1 --2- 172.21.15.78:0/1931136946 >> v2:172.21.15.150:3300/0 conn(0x7f0e94152700 0x7f0e940c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e9359e640 1 -- 172.21.15.78:0/1931136946 >> v1:172.21.15.150:6789/0 conn(0x7f0e940c5730 legacy=0x7f0e9406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e91d9b640 1 -- 172.21.15.78:0/1931136946 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833305805 0 0) 0x7f0e941551f0 con 0x7f0e940c5730 2024-02-19T15:03:35.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e9359e640 1 -- 172.21.15.78:0/1931136946 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94155a50 con 0x7f0e940c8e00 2024-02-19T15:03:35.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e92d9d640 1 --2- 172.21.15.78:0/1931136946 >> v2:172.21.15.150:3300/0 conn(0x7f0e94152700 0x7f0e940c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.758+0000 7f0e9359e640 1 --2- 172.21.15.78:0/1931136946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e9415f980 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da6c085e6b86375a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e91d9b640 1 -- 172.21.15.78:0/1931136946 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c031830 con 0x7f0e940c8e00 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e91d9b640 1 -- 172.21.15.78:0/1931136946 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c031990 con 0x7f0e940c8e00 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e91d9b640 1 -- 172.21.15.78:0/1931136946 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c031d20 con 0x7f0e940c8e00 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 -- 172.21.15.78:0/1931136946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 msgr2=0x7f0e9415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 --2- 172.21.15.78:0/1931136946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e9415f980 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 -- 172.21.15.78:0/1931136946 shutdown_connections 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 --2- 172.21.15.78:0/1931136946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e9415f980 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 --2- 172.21.15.78:0/1931136946 >> v2:172.21.15.150:3300/0 conn(0x7f0e94152700 0x7f0e940c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 -- 172.21.15.78:0/1931136946 >> 172.21.15.78:0/1931136946 conn(0x7f0e940cac90 msgr2=0x7f0e940cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:35.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 -- 172.21.15.78:0/1931136946 shutdown_connections 2024-02-19T15:03:35.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.759+0000 7f0e99165640 1 -- 172.21.15.78:0/1931136946 wait complete. 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.761+0000 7f0e99165640 1 Processor -- start 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.761+0000 7f0e99165640 1 -- start start 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.761+0000 7f0e99165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e940c5730 0x7f0e940ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.761+0000 7f0e99165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e940c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.761+0000 7f0e99165640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 0x7f0e94147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e99165640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e940c4420 con 0x7f0e94152700 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e99165640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e94150d70 con 0x7f0e940c5730 2024-02-19T15:03:35.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e99165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e940c3f40 con 0x7f0e940c8e00 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 0x7f0e94147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9259c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e940c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 0x7f0e94147930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46130/0 (socket says 172.21.15.78:46130) 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e92d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e940c5730 0x7f0e940ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 -- 172.21.15.78:0/1661834170 learned_addr learned my addr 172.21.15.78:0/1661834170 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 -- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 msgr2=0x7f0e940c2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e940c2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 -- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e940c5730 msgr2=0x7f0e940ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e940c5730 0x7f0e940ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 -- 172.21.15.78:0/1661834170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84009a70 con 0x7f0e94152700 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e92d9d640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e940c5730 0x7f0e940ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e9359e640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 0x7f0e94147930 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c004230 con 0x7f0e94152700 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c004390 con 0x7f0e94152700 2024-02-19T15:03:35.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c009400 con 0x7f0e94152700 2024-02-19T15:03:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.762+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c03a8a0 con 0x7f0e94152700 2024-02-19T15:03:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.763+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9414b550 con 0x7f0e94152700 2024-02-19T15:03:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.764+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e94074990 con 0x7f0e94152700 2024-02-19T15:03:35.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.764+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0e8c03aa00 con 0x7f0e94152700 2024-02-19T15:03:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.764+0000 7f0e7b7fe640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e8006b810 0x7f0e8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.764+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0e8c0ac910 con 0x7f0e94152700 2024-02-19T15:03:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.764+0000 7f0e92d9d640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e8006b810 0x7f0e8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:35.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.765+0000 7f0e92d9d640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e8006b810 0x7f0e8006dcd0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009fd0 tx=0x7f0e8400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:35.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.769+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0e8c0771f0 con 0x7f0e94152700 2024-02-19T15:03:35.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.977+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e940c8770 con 0x7f0e94152700 2024-02-19T15:03:35.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.978+0000 7f0e7b7fe640 1 -- 172.21.15.78:0/1661834170 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f0e8c035d90 con 0x7f0e94152700 2024-02-19T15:03:35.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:35.981 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:35.981 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e8006b810 msgr2=0x7f0e8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e8006b810 0x7f0e8006dcd0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f0e84009fd0 tx=0x7f0e8400b040 comp rx=0 tx=0).stop 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 msgr2=0x7f0e94147930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 0x7f0e94147930 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c0046c0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 shutdown_connections 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0e8006b810 0x7f0e8006dcd0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e94152700 0x7f0e94147930 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e940c8e00 0x7f0e940c2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 --2- 172.21.15.78:0/1661834170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e940c5730 0x7f0e940ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 >> 172.21.15.78:0/1661834170 conn(0x7f0e940cac90 msgr2=0x7f0e9414e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 shutdown_connections 2024-02-19T15:03:35.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:35.980+0000 7f0e99165640 1 -- 172.21.15.78:0/1661834170 wait complete. 2024-02-19T15:03:36.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:36.397 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:37.398 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:37.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.513+0000 7f0097e12640 1 Processor -- start 2024-02-19T15:03:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.513+0000 7f0097e12640 1 -- start start 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0097e12640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0090152500 0x7f00900c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0097e12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0090155750 0x7f009006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0097e12640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0090159000 con 0x7f009006e810 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0097e12640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0090155b80 con 0x7f0090152500 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0097e12640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900c5f10 con 0x7f0090155750 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0090152500 0x7f00900c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0090152500 0x7f00900c47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37666/0 (socket says 172.21.15.78:37666) 2024-02-19T15:03:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 -- 172.21.15.78:0/2633866068 learned_addr learned my addr 172.21.15.78:0/2633866068 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f00963af640 1 --2- 172.21.15.78:0/2633866068 >> v2:172.21.15.150:3300/0 conn(0x7f0090155750 0x7f009006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 -- 172.21.15.78:0/2633866068 >> v2:172.21.15.150:3300/0 conn(0x7f0090155750 msgr2=0x7f009006eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 --2- 172.21.15.78:0/2633866068 >> v2:172.21.15.150:3300/0 conn(0x7f0090155750 0x7f009006eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 -- 172.21.15.78:0/2633866068 >> v1:172.21.15.150:6789/0 conn(0x7f009006e810 legacy=0x7f0090155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.514+0000 7f0095bae640 1 -- 172.21.15.78:0/2633866068 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00900c4f30 con 0x7f0090152500 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.515+0000 7f0095bae640 1 --2- 172.21.15.78:0/2633866068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090152500 0x7f00900c47e0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f00901561d0 tx=0x7f008c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dd55e6c0f0c246 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.515+0000 7f0094bac640 1 -- 172.21.15.78:0/2633866068 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c030860 con 0x7f0090152500 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.515+0000 7f0094bac640 1 -- 172.21.15.78:0/2633866068 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c0309c0 con 0x7f0090152500 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.515+0000 7f0094bac640 1 -- 172.21.15.78:0/2633866068 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c030d50 con 0x7f0090152500 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 -- 172.21.15.78:0/2633866068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090152500 msgr2=0x7f00900c47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 --2- 172.21.15.78:0/2633866068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090152500 0x7f00900c47e0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f00901561d0 tx=0x7f008c02b9a0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 -- 172.21.15.78:0/2633866068 shutdown_connections 2024-02-19T15:03:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 --2- 172.21.15.78:0/2633866068 >> v2:172.21.15.150:3300/0 conn(0x7f0090155750 0x7f009006eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 --2- 172.21.15.78:0/2633866068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090152500 0x7f00900c47e0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 -- 172.21.15.78:0/2633866068 >> 172.21.15.78:0/2633866068 conn(0x7f009006d9e0 msgr2=0x7f00900a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 -- 172.21.15.78:0/2633866068 shutdown_connections 2024-02-19T15:03:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.516+0000 7f0097e12640 1 -- 172.21.15.78:0/2633866068 wait complete. 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.517+0000 7f0097e12640 1 Processor -- start 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.517+0000 7f0097e12640 1 -- start start 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f009006e810 0x7f00900a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0090152500 0x7f0090148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 0x7f0090149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900abef0 con 0x7f0090155750 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009014a0c0 con 0x7f009006e810 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0090149930 con 0x7f0090152500 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 0x7f0090149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 0x7f0090149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37678/0 (socket says 172.21.15.78:37678) 2024-02-19T15:03:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 -- 172.21.15.78:0/3080222273 learned_addr learned my addr 172.21.15.78:0/3080222273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0095bae640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f009006e810 0x7f00900a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00953ad640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0090152500 0x7f0090148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 -- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0090152500 msgr2=0x7f0090148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0090152500 0x7f0090148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 -- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f009006e810 msgr2=0x7f00900a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f009006e810 0x7f00900a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 -- 172.21.15.78:0/3080222273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0094d0 con 0x7f0090155750 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00953ad640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0090152500 0x7f0090148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00963af640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 0x7f0090149260 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f008800efc0 tx=0x7f008800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088009ba0 con 0x7f0090155750 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008800f040 con 0x7f0090155750 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.518+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00900a98b0 con 0x7f0090155750 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.519+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00880146c0 con 0x7f0090155750 2024-02-19T15:03:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.519+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00900a7e80 con 0x7f0090155750 2024-02-19T15:03:37.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.520+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0090159470 con 0x7f0090155750 2024-02-19T15:03:37.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.520+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0088014820 con 0x7f0090155750 2024-02-19T15:03:37.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.520+0000 7f00867fc640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f007406b810 0x7f007406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.521+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f008808b110 con 0x7f0090155750 2024-02-19T15:03:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.521+0000 7f0095bae640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f007406b810 0x7f007406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.521+0000 7f0095bae640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f007406b810 0x7f007406dcd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f008c036000 tx=0x7f008c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:37.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.525+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0090159470 con 0x7f0090155750 2024-02-19T15:03:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.734+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00901589b0 con 0x7f0090155750 2024-02-19T15:03:37.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.734+0000 7f00867fc640 1 -- 172.21.15.78:0/3080222273 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f0088018d90 con 0x7f0090155750 2024-02-19T15:03:37.737 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:37.737 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:37.738 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f007406b810 msgr2=0x7f007406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f007406b810 0x7f007406dcd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f008c036000 tx=0x7f008c008030 comp rx=0 tx=0).stop 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 msgr2=0x7f0090149260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 0x7f0090149260 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f008800efc0 tx=0x7f008800c490 comp rx=0 tx=0).stop 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 shutdown_connections 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f007406b810 0x7f007406dcd0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0090155750 0x7f0090149260 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0090152500 0x7f0090148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 --2- 172.21.15.78:0/3080222273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f009006e810 0x7f00900a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 >> 172.21.15.78:0/3080222273 conn(0x7f009006d9e0 msgr2=0x7f0090071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 shutdown_connections 2024-02-19T15:03:37.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:37.736+0000 7f0097e12640 1 -- 172.21.15.78:0/3080222273 wait complete. 2024-02-19T15:03:38.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:38.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:39.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:39.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.257+0000 7fd59718c640 1 Processor -- start 2024-02-19T15:03:39.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.257+0000 7fd59718c640 1 -- start start 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd59718c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5900c50a0 0x7fd5900c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd59718c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd590071790 0x7fd5900c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd59718c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd590071c20 con 0x7fd5900c5480 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd59718c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5900c7fe0 con 0x7fd5900c50a0 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd59718c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900c82c0 con 0x7fd590071790 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd594f28640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5900c50a0 0x7fd5900c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd595729640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd5900c5480 0x7fd59006cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42080/0 (socket says 172.21.15.78:42080) 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd595729640 1 -- 172.21.15.78:0/3115894709 learned_addr learned my addr 172.21.15.78:0/3115894709 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd58ffff640 1 --2- 172.21.15.78:0/3115894709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd590071790 0x7fd5900c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd594f28640 1 -- 172.21.15.78:0/3115894709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd590071790 msgr2=0x7fd5900c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd594f28640 1 --2- 172.21.15.78:0/3115894709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd590071790 0x7fd5900c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd594f28640 1 -- 172.21.15.78:0/3115894709 >> v1:172.21.15.78:6789/0 conn(0x7fd5900c5480 legacy=0x7fd59006cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd594f28640 1 -- 172.21.15.78:0/3115894709 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59006c700 con 0x7fd5900c50a0 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.258+0000 7fd58f7fe640 1 -- 172.21.15.78:0/3115894709 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685817700 0 0) 0x7fd590071c20 con 0x7fd5900c5480 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.259+0000 7fd594f28640 1 --2- 172.21.15.78:0/3115894709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5900c73e0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd5800097a0 tx=0x7fd58002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14aeb1a3df669c46 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.259+0000 7fd58f7fe640 1 -- 172.21.15.78:0/3115894709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58002f050 con 0x7fd5900c50a0 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.259+0000 7fd58f7fe640 1 -- 172.21.15.78:0/3115894709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd580031700 con 0x7fd5900c50a0 2024-02-19T15:03:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.259+0000 7fd58f7fe640 1 -- 172.21.15.78:0/3115894709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580031a90 con 0x7fd5900c50a0 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 -- 172.21.15.78:0/3115894709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 msgr2=0x7fd5900c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 --2- 172.21.15.78:0/3115894709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5900c73e0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd5800097a0 tx=0x7fd58002b7a0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 -- 172.21.15.78:0/3115894709 shutdown_connections 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 --2- 172.21.15.78:0/3115894709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd590071790 0x7fd5900c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 --2- 172.21.15.78:0/3115894709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5900c73e0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 -- 172.21.15.78:0/3115894709 >> 172.21.15.78:0/3115894709 conn(0x7fd59006b700 msgr2=0x7fd5900aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 -- 172.21.15.78:0/3115894709 shutdown_connections 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.260+0000 7fd59718c640 1 -- 172.21.15.78:0/3115894709 wait complete. 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.261+0000 7fd59718c640 1 Processor -- start 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.261+0000 7fd59718c640 1 -- start start 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd59718c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5901ead10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd59718c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5900c5480 0x7fd59009a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd59718c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59009a7a0 0x7fd5900a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd59718c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59009d140 con 0x7fd5900c50a0 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd59718c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5901573c0 con 0x7fd59009a7a0 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd59718c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd590099c60 con 0x7fd5900c5480 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5901ead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5901ead10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37696/0 (socket says 172.21.15.78:37696) 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 -- 172.21.15.78:0/2546195066 learned_addr learned my addr 172.21.15.78:0/2546195066 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd58ffff640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5900c5480 0x7fd59009a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd595729640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59009a7a0 0x7fd5900a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 -- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5900c5480 msgr2=0x7fd59009a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5900c5480 0x7fd59009a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 -- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59009a7a0 msgr2=0x7fd5900a88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59009a7a0 0x7fd5900a88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 -- 172.21.15.78:0/2546195066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd580009400 con 0x7fd5900c50a0 2024-02-19T15:03:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd594f28640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5901ead10 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fd580000c00 tx=0x7fd5800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.262+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5800339b0 con 0x7fd5900c50a0 2024-02-19T15:03:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.263+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd580033b10 con 0x7fd5900c50a0 2024-02-19T15:03:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.263+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5900a9a90 con 0x7fd5900c50a0 2024-02-19T15:03:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.263+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58003c400 con 0x7fd5900c50a0 2024-02-19T15:03:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.263+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5900aa050 con 0x7fd5900c50a0 2024-02-19T15:03:39.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.264+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd590074990 con 0x7fd5900c50a0 2024-02-19T15:03:39.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.264+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd58003a040 con 0x7fd5900c50a0 2024-02-19T15:03:39.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.265+0000 7fd58d7fa640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57c06b810 0x7fd57c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:39.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.265+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd5800abb40 con 0x7fd5900c50a0 2024-02-19T15:03:39.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.265+0000 7fd58ffff640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57c06b810 0x7fd57c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:39.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.265+0000 7fd58ffff640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57c06b810 0x7fd57c06dcd0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd5900aa830 tx=0x7fd584009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:39.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.269+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd580076410 con 0x7fd5900c50a0 2024-02-19T15:03:39.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.474+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5900c6550 con 0x7fd5900c50a0 2024-02-19T15:03:39.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.474+0000 7fd58d7fa640 1 -- 172.21.15.78:0/2546195066 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fd580043d90 con 0x7fd5900c50a0 2024-02-19T15:03:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:39.477 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:39.478 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:39.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57c06b810 msgr2=0x7fd57c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:39.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57c06b810 0x7fd57c06dcd0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd5900aa830 tx=0x7fd584009190 comp rx=0 tx=0).stop 2024-02-19T15:03:39.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 msgr2=0x7fd5901ead10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:39.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5901ead10 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fd580000c00 tx=0x7fd5800046c0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 shutdown_connections 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd57c06b810 0x7fd57c06dcd0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59009a7a0 0x7fd5900a88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5900c5480 0x7fd59009a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 --2- 172.21.15.78:0/2546195066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5900c50a0 0x7fd5901ead10 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 >> 172.21.15.78:0/2546195066 conn(0x7fd59006b700 msgr2=0x7fd590154610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 shutdown_connections 2024-02-19T15:03:39.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:39.476+0000 7fd59718c640 1 -- 172.21.15.78:0/2546195066 wait complete. 2024-02-19T15:03:39.892 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:39.892 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:40.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:41.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.002+0000 7f095618c640 1 Processor -- start 2024-02-19T15:03:41.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f095618c640 1 -- start start 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f095618c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f095014ece0 0x7f09500c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f095618c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09500c5270 0x7f09500719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f095618c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09500c1120 con 0x7f0950071f10 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f095618c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09500c6080 con 0x7f095014ece0 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f095618c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09500c57e0 con 0x7f09500c5270 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f094f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f095014ece0 0x7f09500c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f094ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0950071f10 0x7f09500c15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42086/0 (socket says 172.21.15.78:42086) 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.003+0000 7f094ffff640 1 -- 172.21.15.78:0/3482173429 learned_addr learned my addr 172.21.15.78:0/3482173429 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.004+0000 7f0947fff640 1 --2- 172.21.15.78:0/3482173429 >> v2:172.21.15.150:3300/0 conn(0x7f09500c5270 0x7f09500719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.004+0000 7f094f7fe640 1 -- 172.21.15.78:0/3482173429 >> v2:172.21.15.150:3300/0 conn(0x7f09500c5270 msgr2=0x7f09500719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.004+0000 7f094f7fe640 1 --2- 172.21.15.78:0/3482173429 >> v2:172.21.15.150:3300/0 conn(0x7f09500c5270 0x7f09500719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.004+0000 7f094f7fe640 1 -- 172.21.15.78:0/3482173429 >> v1:172.21.15.78:6789/0 conn(0x7f0950071f10 legacy=0x7f09500c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:41.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.004+0000 7f094effd640 1 -- 172.21.15.78:0/3482173429 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949866606 0 0) 0x7f09500c1120 con 0x7f0950071f10 2024-02-19T15:03:41.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.004+0000 7f094f7fe640 1 -- 172.21.15.78:0/3482173429 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500c5ac0 con 0x7f095014ece0 2024-02-19T15:03:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f094f7fe640 1 --2- 172.21.15.78:0/3482173429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09500c4d30 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f0950155b80 tx=0x7f094002cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d987aeb94fbb2d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f094effd640 1 -- 172.21.15.78:0/3482173429 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094002db50 con 0x7f095014ece0 2024-02-19T15:03:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f094effd640 1 -- 172.21.15.78:0/3482173429 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094002dcb0 con 0x7f095014ece0 2024-02-19T15:03:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f094effd640 1 -- 172.21.15.78:0/3482173429 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940035660 con 0x7f095014ece0 2024-02-19T15:03:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 -- 172.21.15.78:0/3482173429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 msgr2=0x7f09500c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 --2- 172.21.15.78:0/3482173429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09500c4d30 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f0950155b80 tx=0x7f094002cda0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 -- 172.21.15.78:0/3482173429 shutdown_connections 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 --2- 172.21.15.78:0/3482173429 >> v2:172.21.15.150:3300/0 conn(0x7f09500c5270 0x7f09500719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 --2- 172.21.15.78:0/3482173429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09500c4d30 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 -- 172.21.15.78:0/3482173429 >> 172.21.15.78:0/3482173429 conn(0x7f095006b270 msgr2=0x7f095006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 -- 172.21.15.78:0/3482173429 shutdown_connections 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.005+0000 7f095618c640 1 -- 172.21.15.78:0/3482173429 wait complete. 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 Processor -- start 2024-02-19T15:03:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 -- start start 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0950071f10 0x7f09500bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09500c5270 0x7f09501ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09501eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09501ee4f0 con 0x7f095014ece0 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095015de30 con 0x7f0950071f10 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f095618c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095015d870 con 0x7f09500c5270 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f094ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09501eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f094ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09501eabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37722/0 (socket says 172.21.15.78:37722) 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f094ffff640 1 -- 172.21.15.78:0/1007795626 learned_addr learned my addr 172.21.15.78:0/1007795626 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f094f7fe640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0950071f10 0x7f09500bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.007+0000 7f0947fff640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09500c5270 0x7f09501ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094ffff640 1 -- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09500c5270 msgr2=0x7f09501ea6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094ffff640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09500c5270 0x7f09501ea6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094ffff640 1 -- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0950071f10 msgr2=0x7f09500bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094ffff640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0950071f10 0x7f09500bf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094ffff640 1 -- 172.21.15.78:0/1007795626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094000a5a0 con 0x7f095014ece0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094ffff640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09501eabe0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f094800c7d0 tx=0x7f094800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800a640 con 0x7f095014ece0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800a7a0 con 0x7f095014ece0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800ab90 con 0x7f095014ece0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095015db50 con 0x7f095014ece0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.008+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0950152930 con 0x7f095014ece0 2024-02-19T15:03:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.009+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0950152b50 con 0x7f095014ece0 2024-02-19T15:03:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.010+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f094800acf0 con 0x7f095014ece0 2024-02-19T15:03:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.010+0000 7f094cff9640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f093006b810 0x7f093006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.010+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0948089b80 con 0x7f095014ece0 2024-02-19T15:03:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.010+0000 7f094f7fe640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f093006b810 0x7f093006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.011+0000 7f094f7fe640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f093006b810 0x7f093006dcd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0940037000 tx=0x7f0940008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:41.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.014+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0948054450 con 0x7f095014ece0 2024-02-19T15:03:41.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.221+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09501e9fb0 con 0x7f095014ece0 2024-02-19T15:03:41.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.221+0000 7f094cff9640 1 -- 172.21.15.78:0/1007795626 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f0948018020 con 0x7f095014ece0 2024-02-19T15:03:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:41.224 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:41.224 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f093006b810 msgr2=0x7f093006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f093006b810 0x7f093006dcd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f0940037000 tx=0x7f0940008040 comp rx=0 tx=0).stop 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 msgr2=0x7f09501eabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09501eabe0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f094800c7d0 tx=0x7f094800cc90 comp rx=0 tx=0).stop 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 shutdown_connections 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f093006b810 0x7f093006dcd0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f095014ece0 0x7f09501eabe0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09500c5270 0x7f09501ea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 --2- 172.21.15.78:0/1007795626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0950071f10 0x7f09500bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 >> 172.21.15.78:0/1007795626 conn(0x7f095006b270 msgr2=0x7f09500722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 shutdown_connections 2024-02-19T15:03:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:41.223+0000 7f095618c640 1 -- 172.21.15.78:0/1007795626 wait complete. 2024-02-19T15:03:41.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:41.639 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:42.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:42.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.748+0000 7f9cf05f7640 1 Processor -- start 2024-02-19T15:03:42.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.748+0000 7f9cf05f7640 1 -- start start 2024-02-19T15:03:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cf05f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce806ea50 0x7f9ce8072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cf05f7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9ce8152680 0x7f9ce80c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cf05f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce80c5970 con 0x7f9ce814f1a0 2024-02-19T15:03:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cf05f7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce80c5750 con 0x7f9ce8152680 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cf05f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80702d0 con 0x7f9ce806ea50 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cee393640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9ce814f1a0 0x7f9ce806fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42088/0 (socket says 172.21.15.78:42088) 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9ceeb94640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9ce8152680 0x7f9ce80c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cee393640 1 -- 172.21.15.78:0/1495776813 learned_addr learned my addr 172.21.15.78:0/1495776813 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9cedb92640 1 --2- 172.21.15.78:0/1495776813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce806ea50 0x7f9ce8072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9ceeb94640 1 -- 172.21.15.78:0/1495776813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce806ea50 msgr2=0x7f9ce8072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9ceeb94640 1 --2- 172.21.15.78:0/1495776813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce806ea50 0x7f9ce8072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9ceeb94640 1 -- 172.21.15.78:0/1495776813 >> v1:172.21.15.78:6789/0 conn(0x7f9ce814f1a0 legacy=0x7f9ce806fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.750+0000 7f9ceeb94640 1 -- 172.21.15.78:0/1495776813 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80c5bc0 con 0x7f9ce8152680 2024-02-19T15:03:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.749+0000 7f9ced391640 1 -- 172.21.15.78:0/1495776813 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071669522 0 0) 0x7f9ce80c5970 con 0x7f9ce814f1a0 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.750+0000 7f9ceeb94640 1 --2- 172.21.15.78:0/1495776813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce8152680 0x7f9ce80c78a0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00097a0 tx=0x7f9ce002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bb94e6f16bf823 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.750+0000 7f9ced391640 1 -- 172.21.15.78:0/1495776813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce002f050 con 0x7f9ce8152680 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.750+0000 7f9ced391640 1 -- 172.21.15.78:0/1495776813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce0031700 con 0x7f9ce8152680 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.750+0000 7f9ced391640 1 -- 172.21.15.78:0/1495776813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0031a90 con 0x7f9ce8152680 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 -- 172.21.15.78:0/1495776813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce8152680 msgr2=0x7f9ce80c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/1495776813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce8152680 0x7f9ce80c78a0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00097a0 tx=0x7f9ce002b7a0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 -- 172.21.15.78:0/1495776813 shutdown_connections 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/1495776813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce8152680 0x7f9ce80c78a0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/1495776813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce806ea50 0x7f9ce8072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 -- 172.21.15.78:0/1495776813 >> 172.21.15.78:0/1495776813 conn(0x7f9ce806c170 msgr2=0x7f9ce806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 -- 172.21.15.78:0/1495776813 shutdown_connections 2024-02-19T15:03:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.751+0000 7f9cf05f7640 1 -- 172.21.15.78:0/1495776813 wait complete. 2024-02-19T15:03:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.752+0000 7f9cf05f7640 1 Processor -- start 2024-02-19T15:03:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.752+0000 7f9cf05f7640 1 -- start start 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cf05f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ce806ea50 0x7f9ce8153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cf05f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 0x7f9ce8155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cf05f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce8152680 0x7f9ce80c3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cf05f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81561a0 con 0x7f9ce814f1a0 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cf05f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8149540 con 0x7f9ce806ea50 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cf05f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81ec770 con 0x7f9ce8152680 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 0x7f9ce8155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 0x7f9ce8155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37736/0 (socket says 172.21.15.78:37736) 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 -- 172.21.15.78:0/2122674888 learned_addr learned my addr 172.21.15.78:0/2122674888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cee393640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ce806ea50 0x7f9ce8153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:42.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9ceeb94640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce8152680 0x7f9ce80c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 -- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce8152680 msgr2=0x7f9ce80c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce8152680 0x7f9ce80c3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 -- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ce806ea50 msgr2=0x7f9ce8153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ce806ea50 0x7f9ce8153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 -- 172.21.15.78:0/2122674888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0009400 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.753+0000 7f9cedb92640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 0x7f9ce8155c60 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800e8a0 tx=0x7f9cd800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.754+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80094f0 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.754+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd800c940 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.754+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce81ec490 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.754+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd800ccf0 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.754+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce80c2e40 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.755+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce806f290 con 0x7f9ce814f1a0 2024-02-19T15:03:42.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.755+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9cd8013470 con 0x7f9ce814f1a0 2024-02-19T15:03:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.756+0000 7f9cdeffd640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ccc06b810 0x7f9ccc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.756+0000 7f9cee393640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ccc06b810 0x7f9ccc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.756+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9cd808a420 con 0x7f9ce814f1a0 2024-02-19T15:03:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.756+0000 7f9cee393640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ccc06b810 0x7f9ccc06dcd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40012e0 tx=0x7f9ce4001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.760+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9cd8054cf0 con 0x7f9ce814f1a0 2024-02-19T15:03:42.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.968+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ce8074990 con 0x7f9ce814f1a0 2024-02-19T15:03:42.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.969+0000 7f9cdeffd640 1 -- 172.21.15.78:0/2122674888 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f9cd8017d90 con 0x7f9ce814f1a0 2024-02-19T15:03:42.972 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:42.972 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:42.972 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ccc06b810 msgr2=0x7f9ccc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ccc06b810 0x7f9ccc06dcd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40012e0 tx=0x7f9ce4001630 comp rx=0 tx=0).stop 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 msgr2=0x7f9ce8155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 0x7f9ce8155c60 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800e8a0 tx=0x7f9cd800ed60 comp rx=0 tx=0).stop 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 shutdown_connections 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ccc06b810 0x7f9ccc06dcd0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ce814f1a0 0x7f9ce8155c60 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ce8152680 0x7f9ce80c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 --2- 172.21.15.78:0/2122674888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ce806ea50 0x7f9ce8153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:42.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 >> 172.21.15.78:0/2122674888 conn(0x7f9ce806c170 msgr2=0x7f9ce81497f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:42.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 shutdown_connections 2024-02-19T15:03:42.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:42.971+0000 7f9cf05f7640 1 -- 172.21.15.78:0/2122674888 wait complete. 2024-02-19T15:03:43.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:43.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:44.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.496+0000 7fb1c4883640 1 Processor -- start 2024-02-19T15:03:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.496+0000 7fb1c4883640 1 -- start start 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1c4883640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c29c0 0x7fb1c01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1c4883640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb1c00c5be0 0x7fb1c006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1c4883640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c0072250 con 0x7fb1c00c25e0 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1c4883640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c0152c90 con 0x7fb1c00c5be0 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1c4883640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c0072480 con 0x7fb1c00c29c0 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1be59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb1c00c25e0 0x7fb1c00c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42100/0 (socket says 172.21.15.78:42100) 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1be59c640 1 -- 172.21.15.78:0/1676991311 learned_addr learned my addr 172.21.15.78:0/1676991311 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1b7fff640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c29c0 0x7fb1c01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bed9d640 1 --2- 172.21.15.78:0/1676991311 >> v2:172.21.15.150:3300/0 conn(0x7fb1c00c5be0 0x7fb1c006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bdd9b640 1 -- 172.21.15.78:0/1676991311 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908892360 0 0) 0x7fb1c0072250 con 0x7fb1c00c25e0 2024-02-19T15:03:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bdd9b640 1 -- 172.21.15.78:0/1676991311 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ac003520 con 0x7fb1c00c25e0 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bed9d640 1 -- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c29c0 msgr2=0x7fb1c01526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bed9d640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c29c0 0x7fb1c01526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bed9d640 1 -- 172.21.15.78:0/1676991311 >> v1:172.21.15.78:6789/0 conn(0x7fb1c00c25e0 legacy=0x7fb1c00c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1bed9d640 1 -- 172.21.15.78:0/1676991311 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c00c4b90 con 0x7fb1c00c5be0 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.497+0000 7fb1b7fff640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c29c0 0x7fb1c01526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.498+0000 7fb1bed9d640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c5be0 0x7fb1c006e420 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009dc0 tx=0x7fb1b802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=946b29163d02c7ca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.498+0000 7fb1bdd9b640 1 -- 172.21.15.78:0/1676991311 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b802cbb0 con 0x7fb1c00c5be0 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.498+0000 7fb1bdd9b640 1 -- 172.21.15.78:0/1676991311 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b802cd10 con 0x7fb1c00c5be0 2024-02-19T15:03:44.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.498+0000 7fb1bdd9b640 1 -- 172.21.15.78:0/1676991311 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8034640 con 0x7fb1c00c5be0 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 -- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c5be0 msgr2=0x7fb1c006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c5be0 0x7fb1c006e420 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009dc0 tx=0x7fb1b802be00 comp rx=0 tx=0).stop 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 -- 172.21.15.78:0/1676991311 shutdown_connections 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c5be0 0x7fb1c006e420 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1676991311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c29c0 0x7fb1c01526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 -- 172.21.15.78:0/1676991311 >> 172.21.15.78:0/1676991311 conn(0x7fb1c006b270 msgr2=0x7fb1c006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 -- 172.21.15.78:0/1676991311 shutdown_connections 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.499+0000 7fb1c4883640 1 -- 172.21.15.78:0/1676991311 wait complete. 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.500+0000 7fb1c4883640 1 Processor -- start 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.500+0000 7fb1c4883640 1 -- start start 2024-02-19T15:03:44.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1c4883640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c25e0 0x7fb1c01ec200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1c4883640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 0x7fb1c0149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1c4883640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c5be0 0x7fb1c0155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1c4883640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c00a6920 con 0x7fb1c00c29c0 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1c4883640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c014a2e0 con 0x7fb1c00c25e0 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1c4883640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c01eb5e0 con 0x7fb1c00c5be0 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 0x7fb1c0149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 0x7fb1c0149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37738/0 (socket says 172.21.15.78:37738) 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 -- 172.21.15.78:0/1191354418 learned_addr learned my addr 172.21.15.78:0/1191354418 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1be59c640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c25e0 0x7fb1c01ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1bed9d640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c5be0 0x7fb1c0155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 -- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c5be0 msgr2=0x7fb1c0155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c5be0 0x7fb1c0155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 -- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c25e0 msgr2=0x7fb1c01ec200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c25e0 0x7fb1c01ec200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 -- 172.21.15.78:0/1191354418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a8009400 con 0x7fb1c00c29c0 2024-02-19T15:03:44.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b7fff640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 0x7fb1c0149580 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800e990 tx=0x7fb1a800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:44.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.501+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a800c980 con 0x7fb1c00c29c0 2024-02-19T15:03:44.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.502+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a800cae0 con 0x7fb1c00c29c0 2024-02-19T15:03:44.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.502+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b8009a70 con 0x7fb1c00c29c0 2024-02-19T15:03:44.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.502+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a800ce70 con 0x7fb1c00c29c0 2024-02-19T15:03:44.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.502+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c01ec740 con 0x7fb1c00c29c0 2024-02-19T15:03:44.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.503+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c0149ac0 con 0x7fb1c00c29c0 2024-02-19T15:03:44.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.503+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb1a8013470 con 0x7fb1c00c29c0 2024-02-19T15:03:44.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.504+0000 7fb1b6ffd640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb1a006b840 0x7fb1a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:44.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.504+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb1a808a6e0 con 0x7fb1c00c29c0 2024-02-19T15:03:44.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.504+0000 7fb1be59c640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb1a006b840 0x7fb1a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:44.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.504+0000 7fb1be59c640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb1a006b840 0x7fb1a006dd00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005880 tx=0x7fb1b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:44.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.508+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb1a8054fb0 con 0x7fb1c00c29c0 2024-02-19T15:03:44.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.715+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1c01ec740 con 0x7fb1c00c29c0 2024-02-19T15:03:44.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.716+0000 7fb1b6ffd640 1 -- 172.21.15.78:0/1191354418 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fb1a8017d90 con 0x7fb1c00c29c0 2024-02-19T15:03:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:44.719 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:44.719 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb1a006b840 msgr2=0x7fb1a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb1a006b840 0x7fb1a006dd00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005880 tx=0x7fb1b0009040 comp rx=0 tx=0).stop 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 msgr2=0x7fb1c0149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 0x7fb1c0149580 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800e990 tx=0x7fb1a800ee50 comp rx=0 tx=0).stop 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 shutdown_connections 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb1a006b840 0x7fb1a006dd00 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c00c29c0 0x7fb1c0149580 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1c00c5be0 0x7fb1c0155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 --2- 172.21.15.78:0/1191354418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c00c25e0 0x7fb1c01ec200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 >> 172.21.15.78:0/1191354418 conn(0x7fb1c006b270 msgr2=0x7fb1c014d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 shutdown_connections 2024-02-19T15:03:44.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:44.718+0000 7fb1c4883640 1 -- 172.21.15.78:0/1191354418 wait complete. 2024-02-19T15:03:45.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:45.134 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:46.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:46.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.242+0000 7f8abc56c640 1 Processor -- start 2024-02-19T15:03:46.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 -- start start 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab40c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ab4155070 0x7f8ab41554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ab40718f0 0x7f8ab4071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab40c2cf0 con 0x7f8ab40718f0 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab40c2030 con 0x7f8ab4155070 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.243+0000 7f8abc56c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4159130 con 0x7f8ab40acc50 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ab40718f0 0x7f8ab4071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8ab40718f0 0x7f8ab4071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37750/0 (socket says 172.21.15.78:37750) 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 -- 172.21.15.78:0/3044199772 learned_addr learned my addr 172.21.15.78:0/3044199772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8ab9b07640 1 --2- 172.21.15.78:0/3044199772 >> v2:172.21.15.150:3300/0 conn(0x7f8ab4155070 0x7f8ab41554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8aba308640 1 --2- 172.21.15.78:0/3044199772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab40c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 -- 172.21.15.78:0/3044199772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 msgr2=0x7f8ab40c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 --2- 172.21.15.78:0/3044199772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab40c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 -- 172.21.15.78:0/3044199772 >> v2:172.21.15.150:3300/0 conn(0x7f8ab4155070 msgr2=0x7f8ab41554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 --2- 172.21.15.78:0/3044199772 >> v2:172.21.15.150:3300/0 conn(0x7f8ab4155070 0x7f8ab41554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.244+0000 7f8abab09640 1 -- 172.21.15.78:0/3044199772 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab40ad030 con 0x7f8ab40718f0 2024-02-19T15:03:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abab09640 1 --2- 172.21.15.78:0/3044199772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab40718f0 0x7f8ab4071d80 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4072a10 tx=0x7f8aac02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff09948f9ff6bc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8ab9306640 1 -- 172.21.15.78:0/3044199772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac02c9e0 con 0x7f8ab40718f0 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8ab9306640 1 -- 172.21.15.78:0/3044199772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac02cb40 con 0x7f8ab40718f0 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8ab9306640 1 -- 172.21.15.78:0/3044199772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac0343f0 con 0x7f8ab40718f0 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 -- 172.21.15.78:0/3044199772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab40718f0 msgr2=0x7f8ab4071d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 --2- 172.21.15.78:0/3044199772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab40718f0 0x7f8ab4071d80 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4072a10 tx=0x7f8aac02be40 comp rx=0 tx=0).stop 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 -- 172.21.15.78:0/3044199772 shutdown_connections 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 --2- 172.21.15.78:0/3044199772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab40718f0 0x7f8ab4071d80 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 --2- 172.21.15.78:0/3044199772 >> v2:172.21.15.150:3300/0 conn(0x7f8ab4155070 0x7f8ab41554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 --2- 172.21.15.78:0/3044199772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab40c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.245+0000 7f8abc56c640 1 -- 172.21.15.78:0/3044199772 >> 172.21.15.78:0/3044199772 conn(0x7f8ab40c7800 msgr2=0x7f8ab40c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.246+0000 7f8abc56c640 1 -- 172.21.15.78:0/3044199772 shutdown_connections 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.246+0000 7f8abc56c640 1 -- 172.21.15.78:0/3044199772 wait complete. 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 Processor -- start 2024-02-19T15:03:46.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 -- start start 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ab40718f0 0x7f8ab4156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab41394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 0x7f8ab4138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4137f30 con 0x7f8ab4155070 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab41384f0 con 0x7f8ab40718f0 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abc56c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab41eb390 con 0x7f8ab40acc50 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abab09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 0x7f8ab4138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abab09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 0x7f8ab4138980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37760/0 (socket says 172.21.15.78:37760) 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8abab09640 1 -- 172.21.15.78:0/2751550882 learned_addr learned my addr 172.21.15.78:0/2751550882 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.247+0000 7f8aba308640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ab40718f0 0x7f8ab4156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8ab9b07640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab41394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abab09640 1 -- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 msgr2=0x7f8ab41394a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abab09640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab41394a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abab09640 1 -- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ab40718f0 msgr2=0x7f8ab4156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abab09640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ab40718f0 0x7f8ab4156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abab09640 1 -- 172.21.15.78:0/2751550882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac009b30 con 0x7f8ab4155070 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abab09640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 0x7f8ab4138980 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f8aac036040 tx=0x7f8aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac009510 con 0x7f8ab4155070 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac03e450 con 0x7f8ab4155070 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab41eb670 con 0x7f8ab4155070 2024-02-19T15:03:46.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac03e800 con 0x7f8ab4155070 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.248+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab41399e0 con 0x7f8ab4155070 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.249+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab4138680 con 0x7f8ab4155070 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.250+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8aac03e960 con 0x7f8ab4155070 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.250+0000 7f8aa6ffd640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a9806b810 0x7f8a9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.250+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8aac0acef0 con 0x7f8ab4155070 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.250+0000 7f8aba308640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a9806b810 0x7f8a9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.251+0000 7f8aba308640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a9806b810 0x7f8a9806dcd0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009ac0 tx=0x7f8ab0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:46.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.254+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8aac0777c0 con 0x7f8ab4155070 2024-02-19T15:03:46.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.461+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ab406c820 con 0x7f8ab4155070 2024-02-19T15:03:46.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.462+0000 7f8aa6ffd640 1 -- 172.21.15.78:0/2751550882 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f8aac043d90 con 0x7f8ab4155070 2024-02-19T15:03:46.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:46.465 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:46.465 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a9806b810 msgr2=0x7f8a9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a9806b810 0x7f8a9806dcd0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009ac0 tx=0x7f8ab0009190 comp rx=0 tx=0).stop 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 msgr2=0x7f8ab4138980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 0x7f8ab4138980 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f8aac036040 tx=0x7f8aac004a40 comp rx=0 tx=0).stop 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 shutdown_connections 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8a9806b810 0x7f8a9806dcd0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ab4155070 0x7f8ab4138980 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ab40acc50 0x7f8ab41394a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 --2- 172.21.15.78:0/2751550882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ab40718f0 0x7f8ab4156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:46.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 >> 172.21.15.78:0/2751550882 conn(0x7f8ab40c7800 msgr2=0x7f8ab4071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:46.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 shutdown_connections 2024-02-19T15:03:46.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:46.464+0000 7f8abc56c640 1 -- 172.21.15.78:0/2751550882 wait complete. 2024-02-19T15:03:46.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:46.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:47.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.992+0000 7f4d400c7640 1 Processor -- start 2024-02-19T15:03:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.992+0000 7f4d400c7640 1 -- start start 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d400c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d380c25e0 0x7f4d381526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d400c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d38072250 con 0x7f4d380c5be0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d400c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d38072480 con 0x7f4d380c29c0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d400c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d38152c90 con 0x7f4d380c25e0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3e664640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4d380c5be0 0x7f4d3806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50860/0 (socket says 172.21.15.78:50860) 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3e664640 1 -- 172.21.15.78:0/1794095015 learned_addr learned my addr 172.21.15.78:0/1794095015 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3de63640 1 --2- 172.21.15.78:0/1794095015 >> v2:172.21.15.150:3300/0 conn(0x7f4d380c25e0 0x7f4d381526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052931785 0 0) 0x7f4d38072250 con 0x7f4d380c5be0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d24003520 con 0x7f4d380c5be0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91633246 0 0) 0x7f4d38072480 con 0x7f4d380c29c0 2024-02-19T15:03:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d38072250 con 0x7f4d380c29c0 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3de63640 1 -- 172.21.15.78:0/1794095015 >> v1:172.21.15.150:6789/0 conn(0x7f4d380c29c0 legacy=0x7f4d380c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3de63640 1 -- 172.21.15.78:0/1794095015 >> v1:172.21.15.78:6789/0 conn(0x7f4d380c5be0 legacy=0x7f4d3806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.993+0000 7f4d3de63640 1 -- 172.21.15.78:0/1794095015 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380c4b90 con 0x7f4d380c25e0 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.994+0000 7f4d3de63640 1 --2- 172.21.15.78:0/1794095015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c25e0 0x7f4d381526a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4d280097a0 tx=0x7f4d2802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd3ec7a85454c61c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.994+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28031830 con 0x7f4d380c25e0 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.994+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d28031990 con 0x7f4d380c25e0 2024-02-19T15:03:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.994+0000 7f4d3ce61640 1 -- 172.21.15.78:0/1794095015 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28031d20 con 0x7f4d380c25e0 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 -- 172.21.15.78:0/1794095015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c25e0 msgr2=0x7f4d381526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1794095015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c25e0 0x7f4d381526a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4d280097a0 tx=0x7f4d2802b8d0 comp rx=0 tx=0).stop 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 -- 172.21.15.78:0/1794095015 shutdown_connections 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1794095015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c25e0 0x7f4d381526a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 -- 172.21.15.78:0/1794095015 >> 172.21.15.78:0/1794095015 conn(0x7f4d3806b270 msgr2=0x7f4d3806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 -- 172.21.15.78:0/1794095015 shutdown_connections 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.995+0000 7f4d400c7640 1 -- 172.21.15.78:0/1794095015 wait complete. 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.996+0000 7f4d400c7640 1 Processor -- start 2024-02-19T15:03:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.996+0000 7f4d400c7640 1 -- start start 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d400c7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d380c25e0 0x7f4d380a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d400c7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 0x7f4d3814a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d400c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c5be0 0x7f4d3814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d400c7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3814afd0 con 0x7f4d380c29c0 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d400c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380a9200 con 0x7f4d380c5be0 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d400c7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380a6800 con 0x7f4d380c25e0 2024-02-19T15:03:48.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 0x7f4d3814a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 0x7f4d3814a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34552/0 (socket says 172.21.15.78:34552) 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 -- 172.21.15.78:0/1288255848 learned_addr learned my addr 172.21.15.78:0/1288255848 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3de63640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d380c25e0 0x7f4d380a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3e664640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c5be0 0x7f4d3814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 -- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d380c25e0 msgr2=0x7f4d380a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d380c25e0 0x7f4d380a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 -- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c5be0 msgr2=0x7f4d3814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c5be0 0x7f4d3814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 -- 172.21.15.78:0/1288255848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28009400 con 0x7f4d380c29c0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d3d662640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 0x7f4d3814a550 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c500 tx=0x7f4d3400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.997+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d340052d0 con 0x7f4d380c29c0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.998+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d34005450 con 0x7f4d380c29c0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.998+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d381e7ac0 con 0x7f4d380c29c0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.998+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d34012870 con 0x7f4d380c29c0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.998+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d380a6a50 con 0x7f4d380c29c0 2024-02-19T15:03:48.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.998+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d38074990 con 0x7f4d380c29c0 2024-02-19T15:03:48.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.999+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4d340129d0 con 0x7f4d380c29c0 2024-02-19T15:03:48.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:47.999+0000 7f4d2e7fc640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1c06b810 0x7f4d1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:48.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.000+0000 7f4d3de63640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1c06b810 0x7f4d1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:48.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.000+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4d34089f90 con 0x7f4d380c29c0 2024-02-19T15:03:48.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.000+0000 7f4d3de63640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1c06b810 0x7f4d1c06dcd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4d28036000 tx=0x7f4d2803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:48.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.004+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4d340bddc0 con 0x7f4d380c29c0 2024-02-19T15:03:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.210+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d3814ed60 con 0x7f4d380c29c0 2024-02-19T15:03:48.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.210+0000 7f4d2e7fc640 1 -- 172.21.15.78:0/1288255848 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f4d34058310 con 0x7f4d380c29c0 2024-02-19T15:03:48.214 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:48.214 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:48.214 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1c06b810 msgr2=0x7f4d1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1c06b810 0x7f4d1c06dcd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4d28036000 tx=0x7f4d2803c040 comp rx=0 tx=0).stop 2024-02-19T15:03:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 msgr2=0x7f4d3814a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:48.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 0x7f4d3814a550 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c500 tx=0x7f4d3400c9c0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 shutdown_connections 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4d1c06b810 0x7f4d1c06dcd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d380c29c0 0x7f4d3814a550 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d380c5be0 0x7f4d3814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 --2- 172.21.15.78:0/1288255848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d380c25e0 0x7f4d380a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 >> 172.21.15.78:0/1288255848 conn(0x7f4d3806b270 msgr2=0x7f4d3806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.212+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 shutdown_connections 2024-02-19T15:03:48.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:48.213+0000 7f4d400c7640 1 -- 172.21.15.78:0/1288255848 wait complete. 2024-02-19T15:03:48.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:48.629 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:49.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.742+0000 7f7080e76640 1 Processor -- start 2024-02-19T15:03:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.742+0000 7f7080e76640 1 -- start start 2024-02-19T15:03:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7080e76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c150320 0x7f707c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7080e76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707c156e90 con 0x7f707c070180 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7080e76640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707c0c2fc0 con 0x7f707c071850 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7080e76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c157150 con 0x7f707c150320 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f707ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f707c070180 0x7f707c070610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50872/0 (socket says 172.21.15.78:50872) 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f707ad9d640 1 -- 172.21.15.78:0/1526543791 learned_addr learned my addr 172.21.15.78:0/1526543791 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f707a59c640 1 --2- 172.21.15.78:0/1526543791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c150320 0x7f707c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63936505 0 0) 0x7f707c156e90 con 0x7f707c070180 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7068003520 con 0x7f707c070180 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.743+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635663477 0 0) 0x7f707c0c2fc0 con 0x7f707c071850 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.744+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f707c156e90 con 0x7f707c071850 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.744+0000 7f707a59c640 1 -- 172.21.15.78:0/1526543791 >> v1:172.21.15.150:6789/0 conn(0x7f707c071850 legacy=0x7f707c1534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.744+0000 7f707a59c640 1 -- 172.21.15.78:0/1526543791 >> v1:172.21.15.78:6789/0 conn(0x7f707c070180 legacy=0x7f707c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.744+0000 7f707a59c640 1 -- 172.21.15.78:0/1526543791 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0c3280 con 0x7f707c150320 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.744+0000 7f707a59c640 1 --2- 172.21.15.78:0/1526543791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c150320 0x7f707c152fa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f706c0097d0 tx=0x7f706c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfe872732bc5a60d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c031800 con 0x7f707c150320 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c031960 con 0x7f707c150320 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7079d9b640 1 -- 172.21.15.78:0/1526543791 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c031cf0 con 0x7f707c150320 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 -- 172.21.15.78:0/1526543791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c150320 msgr2=0x7f707c152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 --2- 172.21.15.78:0/1526543791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c150320 0x7f707c152fa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f706c0097d0 tx=0x7f706c02b8a0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 -- 172.21.15.78:0/1526543791 shutdown_connections 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 --2- 172.21.15.78:0/1526543791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c150320 0x7f707c152fa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 -- 172.21.15.78:0/1526543791 >> 172.21.15.78:0/1526543791 conn(0x7f707c0aa370 msgr2=0x7f707c0aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 -- 172.21.15.78:0/1526543791 shutdown_connections 2024-02-19T15:03:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.745+0000 7f7080e76640 1 -- 172.21.15.78:0/1526543791 wait complete. 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 Processor -- start 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 -- start start 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c070180 0x7f707c150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f707c071850 0x7f707c146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 0x7f707c146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c154a30 con 0x7f707c150320 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c152830 con 0x7f707c071850 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7080e76640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c1474e0 con 0x7f707c070180 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f707ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 0x7f707c146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f707ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 0x7f707c146fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34562/0 (socket says 172.21.15.78:34562) 2024-02-19T15:03:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f707ad9d640 1 -- 172.21.15.78:0/1142830295 learned_addr learned my addr 172.21.15.78:0/1142830295 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.747+0000 7f7073fff640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f707c071850 0x7f707c146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707a59c640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c070180 0x7f707c150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707ad9d640 1 -- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c070180 msgr2=0x7f707c150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707ad9d640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c070180 0x7f707c150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707ad9d640 1 -- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f707c071850 msgr2=0x7f707c146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707ad9d640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f707c071850 0x7f707c146a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707ad9d640 1 -- 172.21.15.78:0/1142830295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c009400 con 0x7f707c150320 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f707ad9d640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 0x7f707c146fa0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f707400cbd0 tx=0x7f707400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707400ace0 con 0x7f707c150320 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707400ae40 con 0x7f707c150320 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074012620 con 0x7f707c150320 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c154c80 con 0x7f707c150320 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.748+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c1521e0 con 0x7f707c150320 2024-02-19T15:03:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.749+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c074990 con 0x7f707c150320 2024-02-19T15:03:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.750+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7074007070 con 0x7f707c150320 2024-02-19T15:03:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.750+0000 7f7072ffd640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705c06b810 0x7f705c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.750+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f707408a620 con 0x7f707c150320 2024-02-19T15:03:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.750+0000 7f707a59c640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705c06b810 0x7f705c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.751+0000 7f707a59c640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705c06b810 0x7f705c06dcd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f706c036000 tx=0x7f706c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.755+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7074054e70 con 0x7f707c150320 2024-02-19T15:03:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.965+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f707c152970 con 0x7f707c150320 2024-02-19T15:03:49.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.966+0000 7f7072ffd640 1 -- 172.21.15.78:0/1142830295 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f7074017d90 con 0x7f707c150320 2024-02-19T15:03:49.969 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:49.969 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705c06b810 msgr2=0x7f705c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705c06b810 0x7f705c06dcd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f706c036000 tx=0x7f706c03c040 comp rx=0 tx=0).stop 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 msgr2=0x7f707c146fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 0x7f707c146fa0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f707400cbd0 tx=0x7f707400a430 comp rx=0 tx=0).stop 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 shutdown_connections 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f705c06b810 0x7f705c06dcd0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f707c150320 0x7f707c146fa0 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f707c071850 0x7f707c146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 --2- 172.21.15.78:0/1142830295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f707c070180 0x7f707c150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 >> 172.21.15.78:0/1142830295 conn(0x7f707c0aa370 msgr2=0x7f707c14b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:49.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 shutdown_connections 2024-02-19T15:03:49.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:49.968+0000 7f7080e76640 1 -- 172.21.15.78:0/1142830295 wait complete. 2024-02-19T15:03:50.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:50.384 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:51.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:51.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.494+0000 7f64b3ec3640 1 Processor -- start 2024-02-19T15:03:51.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.494+0000 7f64b3ec3640 1 -- start start 2024-02-19T15:03:51.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b3ec3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64ac150560 0x7f64ac0c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:51.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b3ec3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac156e80 con 0x7f64ac0721a0 2024-02-19T15:03:51.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b3ec3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac072d70 con 0x7f64ac06bb00 2024-02-19T15:03:51.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b3ec3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ac06ffc0 con 0x7f64ac150560 2024-02-19T15:03:51.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b1c5f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64ac150560 0x7f64ac0c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:51.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b2460640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f64ac0721a0 0x7f64ac15d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50878/0 (socket says 172.21.15.78:50878) 2024-02-19T15:03:51.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.495+0000 7f64b2460640 1 -- 172.21.15.78:0/2605488874 learned_addr learned my addr 172.21.15.78:0/2605488874 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:51.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.496+0000 7f64b1c5f640 1 -- 172.21.15.78:0/2605488874 >> v1:172.21.15.150:6789/0 conn(0x7f64ac06bb00 legacy=0x7f64ac0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.496+0000 7f64b1c5f640 1 -- 172.21.15.78:0/2605488874 >> v1:172.21.15.78:6789/0 conn(0x7f64ac0721a0 legacy=0x7f64ac15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.496+0000 7f64b1c5f640 1 -- 172.21.15.78:0/2605488874 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac073050 con 0x7f64ac150560 2024-02-19T15:03:51.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.496+0000 7f64b0c5d640 1 -- 172.21.15.78:0/2605488874 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715228863 0 0) 0x7f64ac156e80 con 0x7f64ac0721a0 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.496+0000 7f64b1c5f640 1 --2- 172.21.15.78:0/2605488874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac150560 0x7f64ac0c77d0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009d90 tx=0x7f64a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fddff30efc39228 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b0c5d640 1 -- 172.21.15.78:0/2605488874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a802cc60 con 0x7f64ac150560 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b0c5d640 1 -- 172.21.15.78:0/2605488874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a802cdc0 con 0x7f64ac150560 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b0c5d640 1 -- 172.21.15.78:0/2605488874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8034660 con 0x7f64ac150560 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 -- 172.21.15.78:0/2605488874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac150560 msgr2=0x7f64ac0c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/2605488874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac150560 0x7f64ac0c77d0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009d90 tx=0x7f64a802beb0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 -- 172.21.15.78:0/2605488874 shutdown_connections 2024-02-19T15:03:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/2605488874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac150560 0x7f64ac0c77d0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 -- 172.21.15.78:0/2605488874 >> 172.21.15.78:0/2605488874 conn(0x7f64ac06c170 msgr2=0x7f64ac06c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 -- 172.21.15.78:0/2605488874 shutdown_connections 2024-02-19T15:03:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.497+0000 7f64b3ec3640 1 -- 172.21.15.78:0/2605488874 wait complete. 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.498+0000 7f64b3ec3640 1 Processor -- start 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.498+0000 7f64b3ec3640 1 -- start start 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b3ec3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64ac06bb00 0x7f64ac154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b3ec3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 0x7f64ac0a7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b3ec3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64ac150560 0x7f64ac154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b3ec3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac156050 con 0x7f64ac0721a0 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b3ec3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac1e8c90 con 0x7f64ac150560 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b3ec3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac1e8f70 con 0x7f64ac06bb00 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 0x7f64ac0a7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 0x7f64ac0a7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34580/0 (socket says 172.21.15.78:34580) 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 -- 172.21.15.78:0/1723754616 learned_addr learned my addr 172.21.15.78:0/1723754616 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:51.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b2460640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64ac150560 0x7f64ac154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b1c5f640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64ac06bb00 0x7f64ac154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 -- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64ac06bb00 msgr2=0x7f64ac154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64ac06bb00 0x7f64ac154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 -- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64ac150560 msgr2=0x7f64ac154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64ac150560 0x7f64ac154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.499+0000 7f64b145e640 1 -- 172.21.15.78:0/1723754616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8009a70 con 0x7f64ac0721a0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.500+0000 7f64b145e640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 0x7f64ac0a7600 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c400 tx=0x7f649c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.500+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0052d0 con 0x7f64ac0721a0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.500+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c005430 con 0x7f64ac0721a0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.500+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac155ce0 con 0x7f64ac0721a0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.500+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c003830 con 0x7f64ac0721a0 2024-02-19T15:03:51.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.500+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac155560 con 0x7f64ac0721a0 2024-02-19T15:03:51.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.501+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac156200 con 0x7f64ac0721a0 2024-02-19T15:03:51.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.501+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f649c003990 con 0x7f64ac0721a0 2024-02-19T15:03:51.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.502+0000 7f64a27fc640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f649006b840 0x7f649006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:51.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.502+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f649c089910 con 0x7f64ac0721a0 2024-02-19T15:03:51.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.502+0000 7f64b1c5f640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f649006b840 0x7f649006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:51.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.502+0000 7f64b1c5f640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f649006b840 0x7f649006dd00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f64a8036000 tx=0x7f64a8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:51.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.506+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64ac156200 con 0x7f64ac0721a0 2024-02-19T15:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.713+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64ac0ad9a0 con 0x7f64ac0721a0 2024-02-19T15:03:51.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.713+0000 7f64a27fc640 1 -- 172.21.15.78:0/1723754616 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f649c017d90 con 0x7f64ac0721a0 2024-02-19T15:03:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:51.717 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:51.717 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f649006b840 msgr2=0x7f649006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f649006b840 0x7f649006dd00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f64a8036000 tx=0x7f64a8030000 comp rx=0 tx=0).stop 2024-02-19T15:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 msgr2=0x7f64ac0a7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 0x7f64ac0a7600 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c400 tx=0x7f649c00c8c0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 shutdown_connections 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f649006b840 0x7f649006dd00 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ac0721a0 0x7f64ac0a7600 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64ac150560 0x7f64ac154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 --2- 172.21.15.78:0/1723754616 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64ac06bb00 0x7f64ac154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 >> 172.21.15.78:0/1723754616 conn(0x7f64ac06c170 msgr2=0x7f64ac14e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 shutdown_connections 2024-02-19T15:03:51.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:51.715+0000 7f64b3ec3640 1 -- 172.21.15.78:0/1723754616 wait complete. 2024-02-19T15:03:52.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:52.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:53.132 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:53.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.244+0000 7fc775f55640 1 Processor -- start 2024-02-19T15:03:53.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.244+0000 7fc775f55640 1 -- start start 2024-02-19T15:03:53.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc775f55640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7700c5590 0x7fc7700c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:53.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc775f55640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7700c9200 con 0x7fc7700c51b0 2024-02-19T15:03:53.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc775f55640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7700cb720 con 0x7fc7700c59f0 2024-02-19T15:03:53.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc775f55640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc770158ff0 con 0x7fc7700c5590 2024-02-19T15:03:53.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc76effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc7700c5590 0x7fc7700c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:53.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc76f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc7700c51b0 0x7fc7700c8950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50894/0 (socket says 172.21.15.78:50894) 2024-02-19T15:03:53.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.245+0000 7fc76f7fe640 1 -- 172.21.15.78:0/1284801972 learned_addr learned my addr 172.21.15.78:0/1284801972 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:53.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.246+0000 7fc76effd640 1 -- 172.21.15.78:0/1284801972 >> v1:172.21.15.150:6789/0 conn(0x7fc7700c59f0 legacy=0x7fc77015f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.246+0000 7fc76effd640 1 -- 172.21.15.78:0/1284801972 >> v1:172.21.15.78:6789/0 conn(0x7fc7700c51b0 legacy=0x7fc7700c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.246+0000 7fc76effd640 1 -- 172.21.15.78:0/1284801972 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7701592d0 con 0x7fc7700c5590 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.246+0000 7fc76effd640 1 --2- 172.21.15.78:0/1284801972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c5590 0x7fc7700c9c50 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fc7640097a0 tx=0x7fc76402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443466ce87c2afc0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc76e7fc640 1 -- 172.21.15.78:0/1284801972 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764030610 con 0x7fc7700c5590 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc76e7fc640 1 -- 172.21.15.78:0/1284801972 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764030770 con 0x7fc7700c5590 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc76e7fc640 1 -- 172.21.15.78:0/1284801972 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764030b00 con 0x7fc7700c5590 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 -- 172.21.15.78:0/1284801972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c5590 msgr2=0x7fc7700c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 --2- 172.21.15.78:0/1284801972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c5590 0x7fc7700c9c50 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fc7640097a0 tx=0x7fc76402b960 comp rx=0 tx=0).stop 2024-02-19T15:03:53.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 -- 172.21.15.78:0/1284801972 shutdown_connections 2024-02-19T15:03:53.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 --2- 172.21.15.78:0/1284801972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c5590 0x7fc7700c9c50 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 -- 172.21.15.78:0/1284801972 >> 172.21.15.78:0/1284801972 conn(0x7fc7700ca830 msgr2=0x7fc7700cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:53.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 -- 172.21.15.78:0/1284801972 shutdown_connections 2024-02-19T15:03:53.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.247+0000 7fc775f55640 1 -- 172.21.15.78:0/1284801972 wait complete. 2024-02-19T15:03:53.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.248+0000 7fc775f55640 1 Processor -- start 2024-02-19T15:03:53.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 -- start start 2024-02-19T15:03:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7700c51b0 0x7fc770156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7700c5590 0x7fc7701572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 0x7fc7701506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770156340 con 0x7fc7700c59f0 2024-02-19T15:03:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770151670 con 0x7fc7700c5590 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc775f55640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7701565e0 con 0x7fc7700c51b0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 0x7fc7701506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 0x7fc7701506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34604/0 (socket says 172.21.15.78:34604) 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc76ffff640 1 -- 172.21.15.78:0/1499406411 learned_addr learned my addr 172.21.15.78:0/1499406411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc76effd640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7700c5590 0x7fc7701572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.249+0000 7fc76f7fe640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7700c51b0 0x7fc770156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc76ffff640 1 -- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7700c51b0 msgr2=0x7fc770156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc76ffff640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7700c51b0 0x7fc770156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc76ffff640 1 -- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7700c5590 msgr2=0x7fc7701572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc76ffff640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7700c5590 0x7fc7701572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc76ffff640 1 -- 172.21.15.78:0/1499406411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764009400 con 0x7fc7700c59f0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc76ffff640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 0x7fc7701506b0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c6c0 tx=0x7fc76800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768003b70 con 0x7fc7700c59f0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc768003cd0 con 0x7fc7700c59f0 2024-02-19T15:03:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc770151910 con 0x7fc7700c59f0 2024-02-19T15:03:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680052f0 con 0x7fc7700c59f0 2024-02-19T15:03:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.250+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7701568a0 con 0x7fc7700c59f0 2024-02-19T15:03:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.251+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc770151b00 con 0x7fc7700c59f0 2024-02-19T15:03:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.252+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc768005450 con 0x7fc7700c59f0 2024-02-19T15:03:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.252+0000 7fc757fff640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc75c06b810 0x7fc75c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.252+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc76808a020 con 0x7fc7700c59f0 2024-02-19T15:03:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.252+0000 7fc76f7fe640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc75c06b810 0x7fc75c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.252+0000 7fc76f7fe640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc75c06b810 0x7fc75c06dcd0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc760005880 tx=0x7fc760009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:53.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.256+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc7680548f0 con 0x7fc7700c59f0 2024-02-19T15:03:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.467+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc770074990 con 0x7fc7700c59f0 2024-02-19T15:03:53.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.467+0000 7fc757fff640 1 -- 172.21.15.78:0/1499406411 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fc768057fb0 con 0x7fc7700c59f0 2024-02-19T15:03:53.470 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:53.470 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:53.471 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc75c06b810 msgr2=0x7fc75c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc75c06b810 0x7fc75c06dcd0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc760005880 tx=0x7fc760009040 comp rx=0 tx=0).stop 2024-02-19T15:03:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 msgr2=0x7fc7701506b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 0x7fc7701506b0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c6c0 tx=0x7fc76800cb80 comp rx=0 tx=0).stop 2024-02-19T15:03:53.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 shutdown_connections 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc75c06b810 0x7fc75c06dcd0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7700c59f0 0x7fc7701506b0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7700c5590 0x7fc7701572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 --2- 172.21.15.78:0/1499406411 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7700c51b0 0x7fc770156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 >> 172.21.15.78:0/1499406411 conn(0x7fc7700ca830 msgr2=0x7fc77006c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 shutdown_connections 2024-02-19T15:03:53.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:53.469+0000 7fc775f55640 1 -- 172.21.15.78:0/1499406411 wait complete. 2024-02-19T15:03:53.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:53.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:54.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:54.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.993+0000 7fb2c936c640 1 Processor -- start 2024-02-19T15:03:54.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.993+0000 7fb2c936c640 1 -- start start 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c936c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2c40c78a0 0x7fb2c40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c936c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c414f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c936c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40c35c0 con 0x7fb2c414ef50 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c936c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c414f920 con 0x7fb2c40c78a0 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c936c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40c38a0 con 0x7fb2c4072ee0 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c27fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb2c414ef50 0x7fb2c40727d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50900/0 (socket says 172.21.15.78:50900) 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c27fc640 1 -- 172.21.15.78:0/218344949 learned_addr learned my addr 172.21.15.78:0/218344949 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c37fe640 1 --2- 172.21.15.78:0/218344949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c414f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.994+0000 7fb2c2ffd640 1 --2- 172.21.15.78:0/218344949 >> v2:172.21.15.150:3300/0 conn(0x7fb2c40c78a0 0x7fb2c40c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.995+0000 7fb2c1ffb640 1 -- 172.21.15.78:0/218344949 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667906639 0 0) 0x7fb2c40c35c0 con 0x7fb2c414ef50 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.995+0000 7fb2c1ffb640 1 -- 172.21.15.78:0/218344949 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac003520 con 0x7fb2c414ef50 2024-02-19T15:03:54.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.995+0000 7fb2c37fe640 1 -- 172.21.15.78:0/218344949 >> v2:172.21.15.150:3300/0 conn(0x7fb2c40c78a0 msgr2=0x7fb2c40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:54.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.995+0000 7fb2c37fe640 1 --2- 172.21.15.78:0/218344949 >> v2:172.21.15.150:3300/0 conn(0x7fb2c40c78a0 0x7fb2c40c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:54.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.995+0000 7fb2c37fe640 1 -- 172.21.15.78:0/218344949 >> v1:172.21.15.78:6789/0 conn(0x7fb2c414ef50 legacy=0x7fb2c40727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:54.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.995+0000 7fb2c37fe640 1 -- 172.21.15.78:0/218344949 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40c3b80 con 0x7fb2c4072ee0 2024-02-19T15:03:55.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c37fe640 1 --2- 172.21.15.78:0/218344949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c414f3e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0097a0 tx=0x7fb2bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2da999b8fe38a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c1ffb640 1 -- 172.21.15.78:0/218344949 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc031830 con 0x7fb2c4072ee0 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c1ffb640 1 -- 172.21.15.78:0/218344949 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc031990 con 0x7fb2c4072ee0 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c1ffb640 1 -- 172.21.15.78:0/218344949 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc031d20 con 0x7fb2c4072ee0 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 -- 172.21.15.78:0/218344949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 msgr2=0x7fb2c414f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 --2- 172.21.15.78:0/218344949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c414f3e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0097a0 tx=0x7fb2bc02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 -- 172.21.15.78:0/218344949 shutdown_connections 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 --2- 172.21.15.78:0/218344949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c414f3e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 --2- 172.21.15.78:0/218344949 >> v2:172.21.15.150:3300/0 conn(0x7fb2c40c78a0 0x7fb2c40c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 -- 172.21.15.78:0/218344949 >> 172.21.15.78:0/218344949 conn(0x7fb2c4066e70 msgr2=0x7fb2c40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 -- 172.21.15.78:0/218344949 shutdown_connections 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.996+0000 7fb2c936c640 1 -- 172.21.15.78:0/218344949 wait complete. 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 Processor -- start 2024-02-19T15:03:55.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 -- start start 2024-02-19T15:03:55.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c40c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:55.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 0x7fb2c41eda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:55.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c414ef50 0x7fb2c4144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:55.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40c7350 con 0x7fb2c40c78a0 2024-02-19T15:03:55.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40beeb0 con 0x7fb2c414ef50 2024-02-19T15:03:55.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c936c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c40be930 con 0x7fb2c4072ee0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 0x7fb2c41eda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 0x7fb2c41eda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34616/0 (socket says 172.21.15.78:34616) 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c27fc640 1 -- 172.21.15.78:0/3829388079 learned_addr learned my addr 172.21.15.78:0/3829388079 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c37fe640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c414ef50 0x7fb2c4144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.998+0000 7fb2c2ffd640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c40c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c27fc640 1 -- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 msgr2=0x7fb2c40c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c27fc640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c40c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c27fc640 1 -- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c414ef50 msgr2=0x7fb2c4144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c27fc640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c414ef50 0x7fb2c4144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c27fc640 1 -- 172.21.15.78:0/3829388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc009400 con 0x7fb2c40c78a0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c27fc640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 0x7fb2c41eda40 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800c390 tx=0x7fb2b800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8005ce0 con 0x7fb2c40c78a0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b8005e40 con 0x7fb2c40c78a0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80072f0 con 0x7fb2c40c78a0 2024-02-19T15:03:55.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c41ee360 con 0x7fb2c40c78a0 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:54.999+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c41ee640 con 0x7fb2c40c78a0 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.000+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c40c5d30 con 0x7fb2c40c78a0 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.001+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb2b8007450 con 0x7fb2c40c78a0 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.001+0000 7fb2ab7fe640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b006b810 0x7fb2b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.001+0000 7fb2c2ffd640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b006b810 0x7fb2b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.001+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb2b8089970 con 0x7fb2c40c78a0 2024-02-19T15:03:55.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.002+0000 7fb2c2ffd640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b006b810 0x7fb2b006dcd0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40113e0 tx=0x7fb2b400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:55.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.006+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2c40c5d30 con 0x7fb2c40c78a0 2024-02-19T15:03:55.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.213+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2c40c5d30 con 0x7fb2c40c78a0 2024-02-19T15:03:55.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.213+0000 7fb2ab7fe640 1 -- 172.21.15.78:0/3829388079 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fb2c40c5d30 con 0x7fb2c40c78a0 2024-02-19T15:03:55.216 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:55.217 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:55.217 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:55.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b006b810 msgr2=0x7fb2b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b006b810 0x7fb2b006dcd0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40113e0 tx=0x7fb2b400b040 comp rx=0 tx=0).stop 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 msgr2=0x7fb2c41eda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 0x7fb2c41eda40 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800c390 tx=0x7fb2b800c850 comp rx=0 tx=0).stop 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 shutdown_connections 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b006b810 0x7fb2b006dcd0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c40c78a0 0x7fb2c41eda40 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c414ef50 0x7fb2c4144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.215+0000 7fb2c936c640 1 --2- 172.21.15.78:0/3829388079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c4072ee0 0x7fb2c40c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.216+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 >> 172.21.15.78:0/3829388079 conn(0x7fb2c4066e70 msgr2=0x7fb2c414faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.216+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 shutdown_connections 2024-02-19T15:03:55.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:55.216+0000 7fb2c936c640 1 -- 172.21.15.78:0/3829388079 wait complete. 2024-02-19T15:03:55.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:55.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:56.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:56.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.741+0000 7f4b97d77640 1 Processor -- start 2024-02-19T15:03:56.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.742+0000 7f4b97d77640 1 -- start start 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.742+0000 7f4b97d77640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b90152700 0x7f4b900c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.742+0000 7f4b97d77640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b9015f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.742+0000 7f4b97d77640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b901551f0 con 0x7f4b900c5730 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.742+0000 7f4b97d77640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b90155490 con 0x7f4b90152700 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.742+0000 7f4b97d77640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90155770 con 0x7f4b900c8e00 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b96314640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b9015f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95b13640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b90152700 0x7f4b900c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95312640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4b900c5730 0x7f4b9006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46108/0 (socket says 172.21.15.78:46108) 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95312640 1 -- 172.21.15.78:0/469934711 learned_addr learned my addr 172.21.15.78:0/469934711 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95b13640 1 -- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 msgr2=0x7f4b9015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95b13640 1 --2- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b9015f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95b13640 1 -- 172.21.15.78:0/469934711 >> v1:172.21.15.150:6789/0 conn(0x7f4b900c5730 legacy=0x7f4b9006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:56.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b95b13640 1 -- 172.21.15.78:0/469934711 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90155a50 con 0x7f4b90152700 2024-02-19T15:03:56.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b94b11640 1 -- 172.21.15.78:0/469934711 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854750571 0 0) 0x7f4b901551f0 con 0x7f4b900c5730 2024-02-19T15:03:56.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.743+0000 7f4b96314640 1 --2- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b9015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.744+0000 7f4b95b13640 1 --2- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b900c51f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009d90 tx=0x7f4b8802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d35bc6171089ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.744+0000 7f4b94b11640 1 -- 172.21.15.78:0/469934711 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8802cc50 con 0x7f4b90152700 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.744+0000 7f4b94b11640 1 -- 172.21.15.78:0/469934711 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8802cdb0 con 0x7f4b90152700 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.744+0000 7f4b94b11640 1 -- 172.21.15.78:0/469934711 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88034660 con 0x7f4b90152700 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 -- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 msgr2=0x7f4b900c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 --2- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b900c51f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009d90 tx=0x7f4b8802bea0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 -- 172.21.15.78:0/469934711 shutdown_connections 2024-02-19T15:03:56.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 --2- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b9015f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 --2- 172.21.15.78:0/469934711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b900c51f0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 -- 172.21.15.78:0/469934711 >> 172.21.15.78:0/469934711 conn(0x7f4b900cac90 msgr2=0x7f4b900cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:56.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 -- 172.21.15.78:0/469934711 shutdown_connections 2024-02-19T15:03:56.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.745+0000 7f4b97d77640 1 -- 172.21.15.78:0/469934711 wait complete. 2024-02-19T15:03:56.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.746+0000 7f4b97d77640 1 Processor -- start 2024-02-19T15:03:56.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.746+0000 7f4b97d77640 1 -- start start 2024-02-19T15:03:56.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 0x7f4b901582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:56.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b90157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b901eb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90156e60 con 0x7f4b900c5730 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b901588b0 con 0x7f4b90152700 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b901570b0 con 0x7f4b900c8e00 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 0x7f4b901582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 0x7f4b901582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34618/0 (socket says 172.21.15.78:34618) 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 -- 172.21.15.78:0/1917521742 learned_addr learned my addr 172.21.15.78:0/1917521742 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95312640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b90157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b96314640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b901eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 -- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 msgr2=0x7f4b90157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b90157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 -- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 msgr2=0x7f4b901eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b901eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 -- 172.21.15.78:0/1917521742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c009400 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b95b13640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 0x7f4b901582d0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009d90 tx=0x7f4b88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88040650 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b880407b0 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b88009a70 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88040b60 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.747+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b90157670 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.748+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b90074990 con 0x7f4b900c5730 2024-02-19T15:03:56.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.749+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b88040cc0 con 0x7f4b900c5730 2024-02-19T15:03:56.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.749+0000 7f4b867fc640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7406b810 0x7f4b7406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:56.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.749+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4b880ad090 con 0x7f4b900c5730 2024-02-19T15:03:56.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.749+0000 7f4b95312640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7406b810 0x7f4b7406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:56.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.750+0000 7f4b95312640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7406b810 0x7f4b7406dcd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4b80007f70 tx=0x7f4b80007f00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:56.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.754+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b880779d0 con 0x7f4b900c5730 2024-02-19T15:03:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.963+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b900c8770 con 0x7f4b900c5730 2024-02-19T15:03:56.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.964+0000 7f4b867fc640 1 -- 172.21.15.78:0/1917521742 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f4b88044d90 con 0x7f4b900c5730 2024-02-19T15:03:56.967 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:56.967 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:56.968 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7406b810 msgr2=0x7f4b7406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7406b810 0x7f4b7406dcd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f4b80007f70 tx=0x7f4b80007f00 comp rx=0 tx=0).stop 2024-02-19T15:03:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 msgr2=0x7f4b901582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 0x7f4b901582d0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009d90 tx=0x7f4b88004a40 comp rx=0 tx=0).stop 2024-02-19T15:03:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 shutdown_connections 2024-02-19T15:03:56.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b7406b810 0x7f4b7406dcd0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b90152700 0x7f4b901eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b900c8e00 0x7f4b90157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 --2- 172.21.15.78:0/1917521742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b900c5730 0x7f4b901582d0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 >> 172.21.15.78:0/1917521742 conn(0x7f4b900cac90 msgr2=0x7f4b900cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 shutdown_connections 2024-02-19T15:03:56.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:56.966+0000 7f4b97d77640 1 -- 172.21.15.78:0/1917521742 wait complete. 2024-02-19T15:03:57.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:57.382 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:03:58.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:03:58.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.491+0000 7fc1c5720640 1 Processor -- start 2024-02-19T15:03:58.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.491+0000 7fc1c5720640 1 -- start start 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1c5720640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc1c006c9b0 0x7fc1c0160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1c5720640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c01551f0 con 0x7fc1c00c8e00 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1c5720640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c0155490 con 0x7fc1c0152700 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1c5720640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c0155770 con 0x7fc1c006c9b0 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1bf7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc1c006c9b0 0x7fc1c0160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1be7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc1c00c8e00 0x7fc1c006c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47694/0 (socket says 172.21.15.78:47694) 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1be7fc640 1 -- 172.21.15.78:0/2713619763 learned_addr learned my addr 172.21.15.78:0/2713619763 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:03:58.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1bf7fe640 1 --2- 172.21.15.78:0/2713619763 >> v2:172.21.15.78:3300/0 conn(0x7fc1c006c9b0 0x7fc1c0160990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38442/0 (socket says 172.21.15.78:38442) 2024-02-19T15:03:58.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1bf7fe640 1 -- 172.21.15.78:0/2713619763 >> v1:172.21.15.150:6789/0 conn(0x7fc1c0152700 legacy=0x7fc1c00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1bf7fe640 1 -- 172.21.15.78:0/2713619763 >> v1:172.21.15.78:6789/0 conn(0x7fc1c00c8e00 legacy=0x7fc1c006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1bdffb640 1 -- 172.21.15.78:0/2713619763 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283780613 0 0) 0x7fc1c01551f0 con 0x7fc1c00c8e00 2024-02-19T15:03:58.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.492+0000 7fc1bf7fe640 1 -- 172.21.15.78:0/2713619763 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0155a50 con 0x7fc1c006c9b0 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.493+0000 7fc1bf7fe640 1 --2- 172.21.15.78:0/2713619763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c0160990 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009d90 tx=0x7fc1b802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1871178c6fbe0ed6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.493+0000 7fc1bdffb640 1 -- 172.21.15.78:0/2713619763 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b802cd00 con 0x7fc1c006c9b0 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.493+0000 7fc1bdffb640 1 -- 172.21.15.78:0/2713619763 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b802ce60 con 0x7fc1c006c9b0 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.493+0000 7fc1bdffb640 1 -- 172.21.15.78:0/2713619763 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8034660 con 0x7fc1c006c9b0 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 -- 172.21.15.78:0/2713619763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 msgr2=0x7fc1c0160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2713619763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c0160990 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009d90 tx=0x7fc1b802bf50 comp rx=0 tx=0).stop 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 -- 172.21.15.78:0/2713619763 shutdown_connections 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2713619763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c0160990 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 -- 172.21.15.78:0/2713619763 >> 172.21.15.78:0/2713619763 conn(0x7fc1c00cac90 msgr2=0x7fc1c00cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:58.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 -- 172.21.15.78:0/2713619763 shutdown_connections 2024-02-19T15:03:58.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.494+0000 7fc1c5720640 1 -- 172.21.15.78:0/2713619763 wait complete. 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.495+0000 7fc1c5720640 1 Processor -- start 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.495+0000 7fc1c5720640 1 -- start start 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1c5720640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c00ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1c5720640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c00c8e00 0x7fc1c01576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1c5720640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0152700 0x7fc1c01ede00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1c5720640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00acc50 con 0x7fc1c006c9b0 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1c5720640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00abce0 con 0x7fc1c0152700 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1c5720640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00c1d70 con 0x7fc1c00c8e00 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c00ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c00ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38448/0 (socket says 172.21.15.78:38448) 2024-02-19T15:03:58.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 -- 172.21.15.78:0/2248817940 learned_addr learned my addr 172.21.15.78:0/2248817940 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1be7fc640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c00c8e00 0x7fc1c01576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1bf7fe640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0152700 0x7fc1c01ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 -- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c00c8e00 msgr2=0x7fc1c01576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c00c8e00 0x7fc1c01576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 -- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0152700 msgr2=0x7fc1c01ede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0152700 0x7fc1c01ede00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 -- 172.21.15.78:0/2248817940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8009a70 con 0x7fc1c006c9b0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.496+0000 7fc1beffd640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c00ac710 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c430 tx=0x7fc1b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.497+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b00052d0 con 0x7fc1c006c9b0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.497+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0005430 con 0x7fc1c006c9b0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.497+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c01572b0 con 0x7fc1c006c9b0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.497+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0003830 con 0x7fc1c006c9b0 2024-02-19T15:03:58.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.497+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c00c22c0 con 0x7fc1c006c9b0 2024-02-19T15:03:58.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:03:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.498+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0074990 con 0x7fc1c006c9b0 2024-02-19T15:03:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.498+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc1b0003990 con 0x7fc1c006c9b0 2024-02-19T15:03:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.499+0000 7fc1a77fe640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1ac06b810 0x7fc1ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:03:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.499+0000 7fc1be7fc640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1ac06b810 0x7fc1ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:03:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.499+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc1b0089890 con 0x7fc1c006c9b0 2024-02-19T15:03:58.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.499+0000 7fc1be7fc640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1ac06b810 0x7fc1ac06dcd0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4007f70 tx=0x7fc1b4007f00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:03:58.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:03:58.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.503+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1b0054160 con 0x7fc1c006c9b0 2024-02-19T15:03:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.717+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c00c8770 con 0x7fc1c006c9b0 2024-02-19T15:03:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.717+0000 7fc1a77fe640 1 -- 172.21.15.78:0/2248817940 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fc1b0017d90 con 0x7fc1c006c9b0 2024-02-19T15:03:58.720 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:03:58.720 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:03:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:03:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1ac06b810 msgr2=0x7fc1ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1ac06b810 0x7fc1ac06dcd0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4007f70 tx=0x7fc1b4007f00 comp rx=0 tx=0).stop 2024-02-19T15:03:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 msgr2=0x7fc1c00ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:03:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c00ac710 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000c430 tx=0x7fc1b000c8f0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 shutdown_connections 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc1ac06b810 0x7fc1ac06dcd0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c0152700 0x7fc1c01ede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1c00c8e00 0x7fc1c01576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 --2- 172.21.15.78:0/2248817940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1c006c9b0 0x7fc1c00ac710 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 >> 172.21.15.78:0/2248817940 conn(0x7fc1c00cac90 msgr2=0x7fc1c00cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 shutdown_connections 2024-02-19T15:03:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:03:58.719+0000 7fc1c5720640 1 -- 172.21.15.78:0/2248817940 wait complete. 2024-02-19T15:03:59.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:03:59.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:00.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.246+0000 7f57d38f4640 1 Processor -- start 2024-02-19T15:04:00.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.246+0000 7f57d38f4640 1 -- start start 2024-02-19T15:04:00.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d38f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c8500 0x7f57cc0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:00.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d38f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57cc0c2f50 0x7f57cc158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:00.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d38f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc0c34a0 con 0x7f57cc0c42e0 2024-02-19T15:04:00.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d38f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57cc06c090 con 0x7f57cc0c2f50 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d38f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc06d6a0 con 0x7f57cc0c8500 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d1e91640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57cc0c2f50 0x7f57cc158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d1e91640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f57cc0c2f50 0x7f57cc158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38450/0 (socket says 172.21.15.78:38450) 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d1e91640 1 -- 172.21.15.78:0/93909300 learned_addr learned my addr 172.21.15.78:0/93909300 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d1690640 1 --2- 172.21.15.78:0/93909300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c8500 0x7f57cc0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.247+0000 7f57d1e91640 1 -- 172.21.15.78:0/93909300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c8500 msgr2=0x7f57cc0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57d1e91640 1 --2- 172.21.15.78:0/93909300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c8500 0x7f57cc0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57d1e91640 1 -- 172.21.15.78:0/93909300 >> v1:172.21.15.150:6789/0 conn(0x7f57cc0c42e0 legacy=0x7f57cc158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:00.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57d1e91640 1 -- 172.21.15.78:0/93909300 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc06c8e0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57d1e91640 1 --2- 172.21.15.78:0/93909300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc158d60 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009e50 tx=0x7f57c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6defc3f078946e08 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57c3fff640 1 -- 172.21.15.78:0/93909300 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c402ca70 con 0x7f57cc0c2f50 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57c3fff640 1 -- 172.21.15.78:0/93909300 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c402cbd0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.248+0000 7f57c3fff640 1 -- 172.21.15.78:0/93909300 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c40343f0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 -- 172.21.15.78:0/93909300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 msgr2=0x7f57cc158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 --2- 172.21.15.78:0/93909300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc158d60 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009e50 tx=0x7f57c402bed0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 -- 172.21.15.78:0/93909300 shutdown_connections 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 --2- 172.21.15.78:0/93909300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc158d60 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 --2- 172.21.15.78:0/93909300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c8500 0x7f57cc0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 -- 172.21.15.78:0/93909300 >> 172.21.15.78:0/93909300 conn(0x7f57cc0aa300 msgr2=0x7f57cc0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 -- 172.21.15.78:0/93909300 shutdown_connections 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.249+0000 7f57d38f4640 1 -- 172.21.15.78:0/93909300 wait complete. 2024-02-19T15:04:00.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.250+0000 7f57d38f4640 1 Processor -- start 2024-02-19T15:04:00.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.250+0000 7f57d38f4640 1 -- start start 2024-02-19T15:04:00.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d38f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc1eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:00.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d38f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c8500 0x7f57cc14f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d38f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c38a0 0x7f57cc1eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d38f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1ee010 con 0x7f57cc0c2f50 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d38f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1593b0 con 0x7f57cc0c8500 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d38f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1eef60 con 0x7f57cc0c38a0 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc1eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc1eea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38460/0 (socket says 172.21.15.78:38460) 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 -- 172.21.15.78:0/178589337 learned_addr learned my addr 172.21.15.78:0/178589337 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d0e8f640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c8500 0x7f57cc14f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1e91640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c38a0 0x7f57cc1eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 -- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c38a0 msgr2=0x7f57cc1eaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c38a0 0x7f57cc1eaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 -- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c8500 msgr2=0x7f57cc14f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c8500 0x7f57cc14f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 -- 172.21.15.78:0/178589337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4009b30 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.251+0000 7f57d1690640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc1eea20 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f57c800e8a0 tx=0x7f57c800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.252+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c80094f0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.252+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c800c8e0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.252+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c800cc90 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.252+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc0713a0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.252+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc0ad1a0 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.253+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc071c20 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.253+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f57c8013470 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.254+0000 7f57c1ffb640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b006b810 0x7f57b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.254+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f57c808a420 con 0x7f57cc0c2f50 2024-02-19T15:04:00.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.254+0000 7f57d0e8f640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b006b810 0x7f57b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:00.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.254+0000 7f57d0e8f640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b006b810 0x7f57b006dcd0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0a8020 tx=0x7f57bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:00.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.258+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57c8054c70 con 0x7f57cc0c2f50 2024-02-19T15:04:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.468+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57cc074990 con 0x7f57cc0c2f50 2024-02-19T15:04:00.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.469+0000 7f57c1ffb640 1 -- 172.21.15.78:0/178589337 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f57c8017d90 con 0x7f57cc0c2f50 2024-02-19T15:04:00.472 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:00.472 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:00.472 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b006b810 msgr2=0x7f57b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b006b810 0x7f57b006dcd0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0a8020 tx=0x7f57bc009040 comp rx=0 tx=0).stop 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 msgr2=0x7f57cc1eea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc1eea20 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f57c800e8a0 tx=0x7f57c800ed60 comp rx=0 tx=0).stop 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 shutdown_connections 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f57b006b810 0x7f57b006dcd0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57cc0c38a0 0x7f57cc1eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57cc0c8500 0x7f57cc14f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 --2- 172.21.15.78:0/178589337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57cc0c2f50 0x7f57cc1eea20 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 >> 172.21.15.78:0/178589337 conn(0x7f57cc0aa300 msgr2=0x7f57cc152ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 shutdown_connections 2024-02-19T15:04:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:00.471+0000 7f57d38f4640 1 -- 172.21.15.78:0/178589337 wait complete. 2024-02-19T15:04:00.887 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:00.887 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:01.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:02.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:01.999+0000 7fc13b055640 1 Processor -- start 2024-02-19T15:04:02.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:01.999+0000 7fc13b055640 1 -- start start 2024-02-19T15:04:02.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc13b055640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340ad890 0x7fc1340c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc13b055640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc1340c49c0 0x7fc1340ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc13b055640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc134152d30 con 0x7fc1341534d0 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc13b055640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1340c7860 con 0x7fc1340c49c0 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc13b055640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340c7ce0 con 0x7fc1340ad890 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc133fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc1340c49c0 0x7fc1340ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc138df1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340ad890 0x7fc1340c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc133fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc1340c49c0 0x7fc1340ab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38472/0 (socket says 172.21.15.78:38472) 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc133fff640 1 -- 172.21.15.78:0/3442605614 learned_addr learned my addr 172.21.15.78:0/3442605614 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.000+0000 7fc1395f2640 1 --1- 172.21.15.78:0/3442605614 >> v1:172.21.15.150:6789/0 conn(0x7fc1341534d0 0x7fc1340c5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42510/0 (socket says 172.21.15.78:42510) 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc133fff640 1 -- 172.21.15.78:0/3442605614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340ad890 msgr2=0x7fc1340c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc133fff640 1 --2- 172.21.15.78:0/3442605614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340ad890 0x7fc1340c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc1337fe640 1 -- 172.21.15.78:0/3442605614 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685319135 0 0) 0x7fc134152d30 con 0x7fc1341534d0 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc133fff640 1 -- 172.21.15.78:0/3442605614 >> v1:172.21.15.150:6789/0 conn(0x7fc1341534d0 legacy=0x7fc1340c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:02.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc133fff640 1 -- 172.21.15.78:0/3442605614 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1340c7f80 con 0x7fc1340c49c0 2024-02-19T15:04:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc133fff640 1 --2- 172.21.15.78:0/3442605614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1340c49c0 0x7fc1340ab600 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7fc128009870 tx=0x7fc12802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9b0195c0a1591a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc1337fe640 1 -- 172.21.15.78:0/3442605614 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280317c0 con 0x7fc1340c49c0 2024-02-19T15:04:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc1337fe640 1 -- 172.21.15.78:0/3442605614 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128031920 con 0x7fc1340c49c0 2024-02-19T15:04:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.001+0000 7fc1337fe640 1 -- 172.21.15.78:0/3442605614 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128031cb0 con 0x7fc1340c49c0 2024-02-19T15:04:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 -- 172.21.15.78:0/3442605614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1340c49c0 msgr2=0x7fc1340ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:02.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 --2- 172.21.15.78:0/3442605614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1340c49c0 0x7fc1340ab600 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7fc128009870 tx=0x7fc12802b860 comp rx=0 tx=0).stop 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 -- 172.21.15.78:0/3442605614 shutdown_connections 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 --2- 172.21.15.78:0/3442605614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1340c49c0 0x7fc1340ab600 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 --2- 172.21.15.78:0/3442605614 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340ad890 0x7fc1340c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 -- 172.21.15.78:0/3442605614 >> 172.21.15.78:0/3442605614 conn(0x7fc13406b700 msgr2=0x7fc1340aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 -- 172.21.15.78:0/3442605614 shutdown_connections 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.002+0000 7fc13b055640 1 -- 172.21.15.78:0/3442605614 wait complete. 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.003+0000 7fc13b055640 1 Processor -- start 2024-02-19T15:04:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.003+0000 7fc13b055640 1 -- start start 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc13b055640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1340ad890 0x7fc1341ec630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc13b055640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340c49c0 0x7fc1340a9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc13b055640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 0x7fc13414b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc13b055640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340a9db0 con 0x7fc1341534d0 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc13b055640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134154c90 con 0x7fc1340ad890 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc13b055640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1341554a0 con 0x7fc1340c49c0 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc1395f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 0x7fc13414b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc1395f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 0x7fc13414b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38482/0 (socket says 172.21.15.78:38482) 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc1395f2640 1 -- 172.21.15.78:0/3738841893 learned_addr learned my addr 172.21.15.78:0/3738841893 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc133fff640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340c49c0 0x7fc1340a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:02.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.004+0000 7fc138df1640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1340ad890 0x7fc1341ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1395f2640 1 -- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340c49c0 msgr2=0x7fc1340a9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1395f2640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340c49c0 0x7fc1340a9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1395f2640 1 -- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1340ad890 msgr2=0x7fc1341ec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1395f2640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1340ad890 0x7fc1341ec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1395f2640 1 -- 172.21.15.78:0/3738841893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280094d0 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1395f2640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 0x7fc13414b7a0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00c7b0 tx=0x7fc12c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c005da0 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c00a640 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1340bdd50 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00a9d0 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.005+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1340bda70 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.006+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1340a7ee0 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.006+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc12c00ab30 con 0x7fc1341534d0 2024-02-19T15:04:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.007+0000 7fc1317fa640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc12006b810 0x7fc12006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.007+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc12c08a040 con 0x7fc1341534d0 2024-02-19T15:04:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.007+0000 7fc138df1640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc12006b810 0x7fc12006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.007+0000 7fc138df1640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc12006b810 0x7fc12006dcd0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc124009b40 tx=0x7fc124009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:02.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.011+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc12c054910 con 0x7fc1341534d0 2024-02-19T15:04:02.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.218+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc134074990 con 0x7fc1341534d0 2024-02-19T15:04:02.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.218+0000 7fc1317fa640 1 -- 172.21.15.78:0/3738841893 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fc12c057fd0 con 0x7fc1341534d0 2024-02-19T15:04:02.223 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:02.223 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:02.223 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:02.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.220+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc12006b810 msgr2=0x7fc12006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.220+0000 7fc13b055640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc12006b810 0x7fc12006dcd0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc124009b40 tx=0x7fc124009190 comp rx=0 tx=0).stop 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.220+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 msgr2=0x7fc13414b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.220+0000 7fc13b055640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 0x7fc13414b7a0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00c7b0 tx=0x7fc12c00cbb0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 shutdown_connections 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc12006b810 0x7fc12006dcd0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1341534d0 0x7fc13414b7a0 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1340c49c0 0x7fc1340a9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 --2- 172.21.15.78:0/3738841893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1340ad890 0x7fc1341ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 >> 172.21.15.78:0/3738841893 conn(0x7fc13406b700 msgr2=0x7fc134150ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 shutdown_connections 2024-02-19T15:04:02.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:02.221+0000 7fc13b055640 1 -- 172.21.15.78:0/3738841893 wait complete. 2024-02-19T15:04:02.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:02.638 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:03.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:03.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.752+0000 7f1426343640 1 Processor -- start 2024-02-19T15:04:03.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.752+0000 7f1426343640 1 -- start start 2024-02-19T15:04:03.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f1426343640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1420155070 0x7f1420155500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f1426343640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1420159130 con 0x7f14200acc50 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f1426343640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200ad030 con 0x7f14200718f0 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f1426343640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200c2cf0 con 0x7f1420155070 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f141ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f14200acc50 0x7f14200c17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47702/0 (socket says 172.21.15.78:47702) 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f141ffff640 1 -- 172.21.15.78:0/2293876003 learned_addr learned my addr 172.21.15.78:0/2293876003 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.753+0000 7f141f7fe640 1 --2- 172.21.15.78:0/2293876003 >> v2:172.21.15.78:3300/0 conn(0x7f1420155070 0x7f1420155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.754+0000 7f141f7fe640 1 -- 172.21.15.78:0/2293876003 >> v1:172.21.15.150:6789/0 conn(0x7f14200718f0 legacy=0x7f1420071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.754+0000 7f141f7fe640 1 -- 172.21.15.78:0/2293876003 >> v1:172.21.15.78:6789/0 conn(0x7f14200acc50 legacy=0x7f14200c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.754+0000 7f141effd640 1 -- 172.21.15.78:0/2293876003 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377859335 0 0) 0x7f1420159130 con 0x7f14200acc50 2024-02-19T15:04:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.754+0000 7f141f7fe640 1 -- 172.21.15.78:0/2293876003 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14201595c0 con 0x7f1420155070 2024-02-19T15:04:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.754+0000 7f141f7fe640 1 --2- 172.21.15.78:0/2293876003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1420155070 0x7f1420155500 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f14140097a0 tx=0x7f141402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67b219021c71b95 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f141effd640 1 -- 172.21.15.78:0/2293876003 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414031830 con 0x7f1420155070 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f141effd640 1 -- 172.21.15.78:0/2293876003 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1414031990 con 0x7f1420155070 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f141effd640 1 -- 172.21.15.78:0/2293876003 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414031d20 con 0x7f1420155070 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 -- 172.21.15.78:0/2293876003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1420155070 msgr2=0x7f1420155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 --2- 172.21.15.78:0/2293876003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1420155070 0x7f1420155500 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f14140097a0 tx=0x7f141402b8d0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 -- 172.21.15.78:0/2293876003 shutdown_connections 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 --2- 172.21.15.78:0/2293876003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1420155070 0x7f1420155500 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 -- 172.21.15.78:0/2293876003 >> 172.21.15.78:0/2293876003 conn(0x7f14200c7800 msgr2=0x7f14200c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 -- 172.21.15.78:0/2293876003 shutdown_connections 2024-02-19T15:04:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.755+0000 7f1426343640 1 -- 172.21.15.78:0/2293876003 wait complete. 2024-02-19T15:04:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.756+0000 7f1426343640 1 Processor -- start 2024-02-19T15:04:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 -- start start 2024-02-19T15:04:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14200718f0 0x7f14201eb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 0x7f1420135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1420155070 0x7f1420151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200c2030 con 0x7f14200acc50 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201eb6b0 con 0x7f14200718f0 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f1426343640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1420134990 con 0x7f1420155070 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.757+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 0x7f1420135cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 0x7f1420135cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38502/0 (socket says 172.21.15.78:38502) 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 -- 172.21.15.78:0/1741736717 learned_addr learned my addr 172.21.15.78:0/1741736717 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141ffff640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14200718f0 0x7f14201eb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f14248e0640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1420155070 0x7f1420151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 -- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1420155070 msgr2=0x7f1420151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1420155070 0x7f1420151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 -- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14200718f0 msgr2=0x7f14201eb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14200718f0 0x7f14201eb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 -- 172.21.15.78:0/1741736717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1414009400 con 0x7f14200acc50 2024-02-19T15:04:03.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141f7fe640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 0x7f1420135cd0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f1414034040 tx=0x7f14140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:03.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141403ba70 con 0x7f14200acc50 2024-02-19T15:04:03.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141403bbd0 con 0x7f14200acc50 2024-02-19T15:04:03.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1420134380 con 0x7f14200acc50 2024-02-19T15:04:03.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140453f0 con 0x7f14200acc50 2024-02-19T15:04:03.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.758+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1420134620 con 0x7f14200acc50 2024-02-19T15:04:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.759+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142006c820 con 0x7f14200acc50 2024-02-19T15:04:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.760+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f141404e450 con 0x7f14200acc50 2024-02-19T15:04:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.760+0000 7f141cff9640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f140c06b810 0x7f140c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.760+0000 7f141ffff640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f140c06b810 0x7f140c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.760+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f14140ac490 con 0x7f14200acc50 2024-02-19T15:04:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.761+0000 7f141ffff640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f140c06b810 0x7f140c06dcd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f1410007f70 tx=0x7f1410007f00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:03.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.765+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1414076d60 con 0x7f14200acc50 2024-02-19T15:04:03.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.973+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1420074990 con 0x7f14200acc50 2024-02-19T15:04:03.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.973+0000 7f141cff9640 1 -- 172.21.15.78:0/1741736717 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f1414035090 con 0x7f14200acc50 2024-02-19T15:04:03.978 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:03.978 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:03.978 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:03.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f140c06b810 msgr2=0x7f140c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f140c06b810 0x7f140c06dcd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f1410007f70 tx=0x7f1410007f00 comp rx=0 tx=0).stop 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 msgr2=0x7f1420135cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 0x7f1420135cd0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f1414034040 tx=0x7f14140046c0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 shutdown_connections 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f140c06b810 0x7f140c06dcd0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14200acc50 0x7f1420135cd0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1420155070 0x7f1420151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 --2- 172.21.15.78:0/1741736717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14200718f0 0x7f14201eb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.975+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 >> 172.21.15.78:0/1741736717 conn(0x7f14200c7800 msgr2=0x7f142006be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.976+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 shutdown_connections 2024-02-19T15:04:03.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:03.976+0000 7f1426343640 1 -- 172.21.15.78:0/1741736717 wait complete. 2024-02-19T15:04:04.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:04.393 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:05.393 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:05.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.502+0000 7f05c1a1c640 1 Processor -- start 2024-02-19T15:04:05.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.502+0000 7f05c1a1c640 1 -- start start 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05c1a1c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f05bc152700 0x7f05bc0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05c1a1c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc1551f0 con 0x7f05bc0c8e00 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05c1a1c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc155490 con 0x7f05bc0c5730 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05c1a1c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc155770 con 0x7f05bc152700 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05bb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f05bc0c8e00 0x7f05bc15f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:47704/0 (socket says 172.21.15.78:47704) 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05bb7fe640 1 -- 172.21.15.78:0/3496650793 learned_addr learned my addr 172.21.15.78:0/3496650793 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05baffd640 1 --2- 172.21.15.78:0/3496650793 >> v2:172.21.15.150:3300/0 conn(0x7f05bc152700 0x7f05bc0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774607509 0 0) 0x7f05bc1551f0 con 0x7f05bc0c8e00 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a8003520 con 0x7f05bc0c8e00 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425182348 0 0) 0x7f05bc155490 con 0x7f05bc0c5730 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05bc1551f0 con 0x7f05bc0c5730 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05baffd640 1 -- 172.21.15.78:0/3496650793 >> v1:172.21.15.150:6789/0 conn(0x7f05bc0c5730 legacy=0x7f05bc06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05baffd640 1 -- 172.21.15.78:0/3496650793 >> v1:172.21.15.78:6789/0 conn(0x7f05bc0c8e00 legacy=0x7f05bc15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.503+0000 7f05baffd640 1 -- 172.21.15.78:0/3496650793 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc155a50 con 0x7f05bc152700 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.504+0000 7f05baffd640 1 --2- 172.21.15.78:0/3496650793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc152700 0x7f05bc0c51f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0097a0 tx=0x7f05ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a87db9990a76083 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.504+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac031830 con 0x7f05bc152700 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.504+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ac031990 con 0x7f05bc152700 2024-02-19T15:04:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.504+0000 7f05ba7fc640 1 -- 172.21.15.78:0/3496650793 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac031d00 con 0x7f05bc152700 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 -- 172.21.15.78:0/3496650793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc152700 msgr2=0x7f05bc0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/3496650793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc152700 0x7f05bc0c51f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0097a0 tx=0x7f05ac02b8d0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 -- 172.21.15.78:0/3496650793 shutdown_connections 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/3496650793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc152700 0x7f05bc0c51f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 -- 172.21.15.78:0/3496650793 >> 172.21.15.78:0/3496650793 conn(0x7f05bc0cac90 msgr2=0x7f05bc0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 -- 172.21.15.78:0/3496650793 shutdown_connections 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.505+0000 7f05c1a1c640 1 -- 172.21.15.78:0/3496650793 wait complete. 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.506+0000 7f05c1a1c640 1 Processor -- start 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.506+0000 7f05c1a1c640 1 -- start start 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05c1a1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc0c5730 0x7f05bc1eb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05c1a1c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 0x7f05bc0acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05c1a1c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05bc152700 0x7f05bc0ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05c1a1c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0aa270 con 0x7f05bc0c8e00 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05c1a1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0aa510 con 0x7f05bc0c5730 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05c1a1c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0aa7f0 con 0x7f05bc152700 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 0x7f05bc0acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 0x7f05bc0acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38516/0 (socket says 172.21.15.78:38516) 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 -- 172.21.15.78:0/1630039272 learned_addr learned my addr 172.21.15.78:0/1630039272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05baffd640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc0c5730 0x7f05bc1eb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05bb7fe640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05bc152700 0x7f05bc0ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 -- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05bc152700 msgr2=0x7f05bc0ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05bc152700 0x7f05bc0ab2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 -- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc0c5730 msgr2=0x7f05bc1eb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc0c5730 0x7f05bc1eb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 -- 172.21.15.78:0/1630039272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac009400 con 0x7f05bc0c8e00 2024-02-19T15:04:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.507+0000 7f05b3fff640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 0x7f05bc0acb10 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c500 tx=0x7f05a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.508+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40052d0 con 0x7f05bc0c8e00 2024-02-19T15:04:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.508+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4005470 con 0x7f05bc0c8e00 2024-02-19T15:04:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.508+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc0aaad0 con 0x7f05bc0c8e00 2024-02-19T15:04:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.508+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4003830 con 0x7f05bc0c8e00 2024-02-19T15:04:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.508+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc0ad050 con 0x7f05bc0c8e00 2024-02-19T15:04:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.509+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc074990 con 0x7f05bc0c8e00 2024-02-19T15:04:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.509+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f05a4003990 con 0x7f05bc0c8e00 2024-02-19T15:04:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.509+0000 7f05b8ff9640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f059c06b810 0x7f059c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.510+0000 7f05baffd640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f059c06b810 0x7f059c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.510+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f05a4089910 con 0x7f05bc0c8e00 2024-02-19T15:04:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.510+0000 7f05baffd640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f059c06b810 0x7f059c06dcd0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f05ac036000 tx=0x7f05ac03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:05.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.514+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05a4054160 con 0x7f05bc0c8e00 2024-02-19T15:04:05.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.719+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05bc0c8770 con 0x7f05bc0c8e00 2024-02-19T15:04:05.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.719+0000 7f05b8ff9640 1 -- 172.21.15.78:0/1630039272 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f05a4017d90 con 0x7f05bc0c8e00 2024-02-19T15:04:05.724 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:05.724 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f059c06b810 msgr2=0x7f059c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f059c06b810 0x7f059c06dcd0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f05ac036000 tx=0x7f05ac03c040 comp rx=0 tx=0).stop 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 msgr2=0x7f05bc0acb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 0x7f05bc0acb10 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c500 tx=0x7f05a400c9c0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 shutdown_connections 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f059c06b810 0x7f059c06dcd0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05bc0c8e00 0x7f05bc0acb10 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f05bc152700 0x7f05bc0ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.721+0000 7f05c1a1c640 1 --2- 172.21.15.78:0/1630039272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05bc0c5730 0x7f05bc1eb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.722+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 >> 172.21.15.78:0/1630039272 conn(0x7f05bc0cac90 msgr2=0x7f05bc0cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.722+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 shutdown_connections 2024-02-19T15:04:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:05.722+0000 7f05c1a1c640 1 -- 172.21.15.78:0/1630039272 wait complete. 2024-02-19T15:04:06.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:06.139 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:07.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.244+0000 7fea6236c640 1 Processor -- start 2024-02-19T15:04:07.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.244+0000 7fea6236c640 1 -- start start 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.244+0000 7fea6236c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fea5c0c19f0 0x7fea5c156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.244+0000 7fea6236c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fea5c071410 0x7fea5c0c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.244+0000 7fea6236c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c7570 0x7fea5c0c1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.244+0000 7fea6236c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c152af0 con 0x7fea5c071410 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea6236c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c155a50 con 0x7fea5c0c19f0 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea6236c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c155f80 con 0x7fea5c0c7570 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fea5c071410 0x7fea5c0c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fea5c071410 0x7fea5c0c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38530/0 (socket says 172.21.15.78:38530) 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 -- 172.21.15.78:0/1513550951 learned_addr learned my addr 172.21.15.78:0/1513550951 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5bfff640 1 --2- 172.21.15.78:0/1513550951 >> v2:172.21.15.150:3300/0 conn(0x7fea5c0c19f0 0x7fea5c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea60909640 1 --2- 172.21.15.78:0/1513550951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c7570 0x7fea5c0c1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 -- 172.21.15.78:0/1513550951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c7570 msgr2=0x7fea5c0c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 --2- 172.21.15.78:0/1513550951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c7570 0x7fea5c0c1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 -- 172.21.15.78:0/1513550951 >> v2:172.21.15.150:3300/0 conn(0x7fea5c0c19f0 msgr2=0x7fea5c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 --2- 172.21.15.78:0/1513550951 >> v2:172.21.15.150:3300/0 conn(0x7fea5c0c19f0 0x7fea5c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.245+0000 7fea5b7fe640 1 -- 172.21.15.78:0/1513550951 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c070840 con 0x7fea5c071410 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea5b7fe640 1 --2- 172.21.15.78:0/1513550951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c7030 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0c2ca0 tx=0x7fea5002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c98c2be6fd232bd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea5affd640 1 -- 172.21.15.78:0/1513550951 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50031750 con 0x7fea5c071410 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea5affd640 1 -- 172.21.15.78:0/1513550951 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea500318b0 con 0x7fea5c071410 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea5affd640 1 -- 172.21.15.78:0/1513550951 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50031c40 con 0x7fea5c071410 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 -- 172.21.15.78:0/1513550951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 msgr2=0x7fea5c0c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 --2- 172.21.15.78:0/1513550951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c7030 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0c2ca0 tx=0x7fea5002b7f0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 -- 172.21.15.78:0/1513550951 shutdown_connections 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 --2- 172.21.15.78:0/1513550951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c7570 0x7fea5c0c1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 --2- 172.21.15.78:0/1513550951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c7030 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 --2- 172.21.15.78:0/1513550951 >> v2:172.21.15.150:3300/0 conn(0x7fea5c0c19f0 0x7fea5c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 -- 172.21.15.78:0/1513550951 >> 172.21.15.78:0/1513550951 conn(0x7fea5c0aa370 msgr2=0x7fea5c0aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 -- 172.21.15.78:0/1513550951 shutdown_connections 2024-02-19T15:04:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.246+0000 7fea6236c640 1 -- 172.21.15.78:0/1513550951 wait complete. 2024-02-19T15:04:07.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.247+0000 7fea6236c640 1 Processor -- start 2024-02-19T15:04:07.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 -- start start 2024-02-19T15:04:07.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c19f0 0x7fea5c1ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea5c0c7570 0x7fea5c1eee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c1ef350 con 0x7fea5c071410 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c1ee440 con 0x7fea5c0c7570 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea6236c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c1e10d0 con 0x7fea5c0c19f0 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea5bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38546/0 (socket says 172.21.15.78:38546) 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.248+0000 7fea5bfff640 1 -- 172.21.15.78:0/1882762540 learned_addr learned my addr 172.21.15.78:0/1882762540 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5b7fe640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c19f0 0x7fea5c1ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea60909640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea5c0c7570 0x7fea5c1eee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5bfff640 1 -- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c19f0 msgr2=0x7fea5c1ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5bfff640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c19f0 0x7fea5c1ee8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5bfff640 1 -- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea5c0c7570 msgr2=0x7fea5c1eee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5bfff640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea5c0c7570 0x7fea5c1eee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5bfff640 1 -- 172.21.15.78:0/1882762540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea500094d0 con 0x7fea5c071410 2024-02-19T15:04:07.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea5bfff640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c4c90 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00e890 tx=0x7fea4c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00c940 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c00caa0 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c1eafc0 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c00ce30 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.249+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c1dfe80 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.250+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c0c8320 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.250+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fea4c013470 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.251+0000 7fea58ff9640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4806b810 0x7fea4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.251+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fea4c08a520 con 0x7fea5c071410 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.251+0000 7fea5b7fe640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4806b810 0x7fea4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:07.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.251+0000 7fea5b7fe640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4806b810 0x7fea4806dcd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fea50035000 tx=0x7fea50008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:07.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.255+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fea4c054df0 con 0x7fea5c071410 2024-02-19T15:04:07.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.464+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea5c074990 con 0x7fea5c071410 2024-02-19T15:04:07.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.465+0000 7fea58ff9640 1 -- 172.21.15.78:0/1882762540 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fea4c017d90 con 0x7fea5c071410 2024-02-19T15:04:07.468 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:07.468 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:07.468 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4806b810 msgr2=0x7fea4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4806b810 0x7fea4806dcd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fea50035000 tx=0x7fea50008030 comp rx=0 tx=0).stop 2024-02-19T15:04:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 msgr2=0x7fea5c0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c4c90 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00e890 tx=0x7fea4c00ed50 comp rx=0 tx=0).stop 2024-02-19T15:04:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 shutdown_connections 2024-02-19T15:04:07.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fea4806b810 0x7fea4806dcd0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea5c0c7570 0x7fea5c1eee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea5c0c19f0 0x7fea5c1ee8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 --2- 172.21.15.78:0/1882762540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea5c071410 0x7fea5c0c4c90 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 >> 172.21.15.78:0/1882762540 conn(0x7fea5c0aa370 msgr2=0x7fea5c06ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 shutdown_connections 2024-02-19T15:04:07.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:07.467+0000 7fea6236c640 1 -- 172.21.15.78:0/1882762540 wait complete. 2024-02-19T15:04:07.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:07.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:08.884 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:09.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.000+0000 7f81cf039640 1 Processor -- start 2024-02-19T15:04:09.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.000+0000 7f81cf039640 1 -- start start 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81cf039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81cf039640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8071c20 con 0x7f81c8071790 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81cf039640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80c7fe0 con 0x7f81c80c5480 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81cf039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c80c82c0 con 0x7f81c80c50a0 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81c7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f81c8071790 0x7f81c806cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52554/0 (socket says 172.21.15.78:52554) 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81c7fff640 1 -- 172.21.15.78:0/2315417058 learned_addr learned my addr 172.21.15.78:0/2315417058 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.001+0000 7f81ccdd5640 1 --2- 172.21.15.78:0/2315417058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706082605 0 0) 0x7f81c8071c20 con 0x7f81c8071790 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0003520 con 0x7f81c8071790 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858945266 0 0) 0x7f81c80c7fe0 con 0x7f81c80c5480 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c8071c20 con 0x7f81c80c5480 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81ccdd5640 1 -- 172.21.15.78:0/2315417058 >> v1:172.21.15.150:6789/0 conn(0x7f81c80c5480 legacy=0x7f81c8155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81ccdd5640 1 -- 172.21.15.78:0/2315417058 >> v1:172.21.15.78:6789/0 conn(0x7f81c8071790 legacy=0x7f81c806cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.002+0000 7f81ccdd5640 1 -- 172.21.15.78:0/2315417058 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c806c700 con 0x7f81c80c50a0 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81ccdd5640 1 --2- 172.21.15.78:0/2315417058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c73e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097d0 tx=0x7f81b802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b9396eb7eec320 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8031770 con 0x7f81c80c50a0 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b80318d0 con 0x7f81c80c50a0 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81c77fe640 1 -- 172.21.15.78:0/2315417058 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8031c40 con 0x7f81c80c50a0 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81cf039640 1 -- 172.21.15.78:0/2315417058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 msgr2=0x7f81c80c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81cf039640 1 --2- 172.21.15.78:0/2315417058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c73e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f81b80097d0 tx=0x7f81b802b810 comp rx=0 tx=0).stop 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81cf039640 1 -- 172.21.15.78:0/2315417058 shutdown_connections 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81cf039640 1 --2- 172.21.15.78:0/2315417058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c73e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.003+0000 7f81cf039640 1 -- 172.21.15.78:0/2315417058 >> 172.21.15.78:0/2315417058 conn(0x7f81c806b700 msgr2=0x7f81c80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.004+0000 7f81cf039640 1 -- 172.21.15.78:0/2315417058 shutdown_connections 2024-02-19T15:04:09.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.004+0000 7f81cf039640 1 -- 172.21.15.78:0/2315417058 wait complete. 2024-02-19T15:04:09.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.005+0000 7f81cf039640 1 Processor -- start 2024-02-19T15:04:09.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.005+0000 7f81cf039640 1 -- start start 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cf039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cf039640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 0x7f81c8156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cf039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c809a120 0x7f81c8157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cf039640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c809ae50 con 0x7f81c80c5480 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cf039640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81566a0 con 0x7f81c809a120 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cf039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c809bc50 con 0x7f81c80c50a0 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 0x7f81c8156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 0x7f81c8156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58190/0 (socket says 172.21.15.78:58190) 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 -- 172.21.15.78:0/3371361886 learned_addr learned my addr 172.21.15.78:0/3371361886 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81ccdd5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81cd5d6640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c809a120 0x7f81c8157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:09.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 -- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 msgr2=0x7f81c80c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 -- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c809a120 msgr2=0x7f81c8157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c809a120 0x7f81c8157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 -- 172.21.15.78:0/3371361886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b8009400 con 0x7f81c80c5480 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81ccdd5640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c7fff640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 0x7f81c8156b30 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c500 tx=0x7f81bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.006+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc005ce0 con 0x7f81c80c5480 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.007+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc005e40 con 0x7f81c80c5480 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.007+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c809a5b0 con 0x7f81c80c5480 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.007+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0072f0 con 0x7f81c80c5480 2024-02-19T15:04:09.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.007+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c809ab70 con 0x7f81c80c5480 2024-02-19T15:04:09.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.008+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c8156830 con 0x7f81c80c5480 2024-02-19T15:04:09.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.008+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f81bc007450 con 0x7f81c80c5480 2024-02-19T15:04:09.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.009+0000 7f81c57fa640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b406b840 0x7f81b406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:09.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.009+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f81bc0898b0 con 0x7f81c80c5480 2024-02-19T15:04:09.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.009+0000 7f81ccdd5640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b406b840 0x7f81b406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:09.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.009+0000 7f81ccdd5640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b406b840 0x7f81b406dd00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f81b8036000 tx=0x7f81b803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:09.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.013+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81bc054240 con 0x7f81c80c5480 2024-02-19T15:04:09.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.224+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c8074990 con 0x7f81c80c5480 2024-02-19T15:04:09.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.224+0000 7f81c57fa640 1 -- 172.21.15.78:0/3371361886 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f81bc017d90 con 0x7f81c80c5480 2024-02-19T15:04:09.227 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:09.227 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:09.227 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:09.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b406b840 msgr2=0x7f81b406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b406b840 0x7f81b406dd00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f81b8036000 tx=0x7f81b803d040 comp rx=0 tx=0).stop 2024-02-19T15:04:09.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 msgr2=0x7f81c8156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:09.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 0x7f81c8156b30 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c500 tx=0x7f81bc00c9c0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 shutdown_connections 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f81b406b840 0x7f81b406dd00 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81c80c5480 0x7f81c8156b30 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c809a120 0x7f81c8157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 --2- 172.21.15.78:0/3371361886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81c80c50a0 0x7f81c80c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 >> 172.21.15.78:0/3371361886 conn(0x7f81c806b700 msgr2=0x7f81c8155730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 shutdown_connections 2024-02-19T15:04:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:09.226+0000 7f81cf039640 1 -- 172.21.15.78:0/3371361886 wait complete. 2024-02-19T15:04:09.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:09.643 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:10.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:10.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.756+0000 7f09138a4640 1 Processor -- start 2024-02-19T15:04:10.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.757+0000 7f09138a4640 1 -- start start 2024-02-19T15:04:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.757+0000 7f09138a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c7070 0x7f090c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.757+0000 7f09138a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c155ef0 con 0x7f090c0c87a0 2024-02-19T15:04:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.757+0000 7f09138a4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c06dec0 con 0x7f090c152480 2024-02-19T15:04:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.757+0000 7f09138a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c06d520 con 0x7f090c0c7070 2024-02-19T15:04:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0911e41640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f090c0c87a0 0x7f090c0c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52556/0 (socket says 172.21.15.78:52556) 2024-02-19T15:04:10.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0911e41640 1 -- 172.21.15.78:0/176971617 learned_addr learned my addr 172.21.15.78:0/176971617 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0910e3f640 1 --2- 172.21.15.78:0/176971617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c7070 0x7f090c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222683035 0 0) 0x7f0904001490 con 0x7f090c0c87a0 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8003520 con 0x7f090c0c87a0 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955646797 0 0) 0x7f090c06dec0 con 0x7f090c152480 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904001490 con 0x7f090c152480 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0910e3f640 1 -- 172.21.15.78:0/176971617 >> v1:172.21.15.150:6789/0 conn(0x7f090c152480 legacy=0x7f090c06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0910e3f640 1 -- 172.21.15.78:0/176971617 >> v1:172.21.15.78:6789/0 conn(0x7f090c0c87a0 legacy=0x7f090c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.758+0000 7f0910e3f640 1 -- 172.21.15.78:0/176971617 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c0c75c0 con 0x7f090c0c7070 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.759+0000 7f0910e3f640 1 --2- 172.21.15.78:0/176971617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c7070 0x7f090c1559b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0097d0 tx=0x7f08fc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f2155006e232b69 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.759+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc031890 con 0x7f090c0c7070 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.759+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc0319f0 con 0x7f090c0c7070 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.759+0000 7f0903fff640 1 -- 172.21.15.78:0/176971617 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc031d80 con 0x7f090c0c7070 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 -- 172.21.15.78:0/176971617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c7070 msgr2=0x7f090c1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 --2- 172.21.15.78:0/176971617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c7070 0x7f090c1559b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0097d0 tx=0x7f08fc02b930 comp rx=0 tx=0).stop 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 -- 172.21.15.78:0/176971617 shutdown_connections 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 --2- 172.21.15.78:0/176971617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c7070 0x7f090c1559b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 -- 172.21.15.78:0/176971617 >> 172.21.15.78:0/176971617 conn(0x7f090c0ca8a0 msgr2=0x7f090c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 -- 172.21.15.78:0/176971617 shutdown_connections 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.760+0000 7f09138a4640 1 -- 172.21.15.78:0/176971617 wait complete. 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.761+0000 7f09138a4640 1 Processor -- start 2024-02-19T15:04:10.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.761+0000 7f09138a4640 1 -- start start 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f090c0c7070 0x7f090c0ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c87a0 0x7f090c1eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 0x7f090c1eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c157c20 con 0x7f090c152480 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c1ebbd0 con 0x7f090c0c7070 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c1eac50 con 0x7f090c0c87a0 2024-02-19T15:04:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 0x7f090c1eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 0x7f090c1eb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58194/0 (socket says 172.21.15.78:58194) 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 -- 172.21.15.78:0/368186887 learned_addr learned my addr 172.21.15.78:0/368186887 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0910e3f640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c87a0 0x7f090c1eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911640640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f090c0c7070 0x7f090c0ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 -- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c87a0 msgr2=0x7f090c1eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c87a0 0x7f090c1eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 -- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f090c0c7070 msgr2=0x7f090c0ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f090c0c7070 0x7f090c0ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 -- 172.21.15.78:0/368186887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc009400 con 0x7f090c152480 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0911e41640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 0x7f090c1eb690 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f090400cfb0 tx=0x7f090400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904012400 con 0x7f090c152480 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0904012560 con 0x7f090c152480 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c1eef70 con 0x7f090c152480 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.762+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09040128d0 con 0x7f090c152480 2024-02-19T15:04:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.763+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c0af440 con 0x7f090c152480 2024-02-19T15:04:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.764+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c1eae10 con 0x7f090c152480 2024-02-19T15:04:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.764+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0904007070 con 0x7f090c152480 2024-02-19T15:04:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.765+0000 7f0901ffb640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f08f006b810 0x7f08f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.765+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f090408a330 con 0x7f090c152480 2024-02-19T15:04:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.765+0000 7f0911640640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f08f006b810 0x7f08f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.765+0000 7f0911640640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f08f006b810 0x7f08f006dcd0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0908007fd0 tx=0x7f0908009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.769+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0904054b70 con 0x7f090c152480 2024-02-19T15:04:10.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.978+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f090c074990 con 0x7f090c152480 2024-02-19T15:04:10.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.978+0000 7f0901ffb640 1 -- 172.21.15.78:0/368186887 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f0904017d90 con 0x7f090c152480 2024-02-19T15:04:10.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:10.981 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:10.981 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:10.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f08f006b810 msgr2=0x7f08f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f08f006b810 0x7f08f006dcd0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0908007fd0 tx=0x7f0908009040 comp rx=0 tx=0).stop 2024-02-19T15:04:10.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 msgr2=0x7f090c1eb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:10.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 0x7f090c1eb690 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7f090400cfb0 tx=0x7f090400a430 comp rx=0 tx=0).stop 2024-02-19T15:04:10.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 shutdown_connections 2024-02-19T15:04:10.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f08f006b810 0x7f08f006dcd0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f090c152480 0x7f090c1eb690 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f090c0c87a0 0x7f090c1eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 --2- 172.21.15.78:0/368186887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f090c0c7070 0x7f090c0ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 >> 172.21.15.78:0/368186887 conn(0x7f090c0ca8a0 msgr2=0x7f090c0c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 shutdown_connections 2024-02-19T15:04:10.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:10.980+0000 7f09138a4640 1 -- 172.21.15.78:0/368186887 wait complete. 2024-02-19T15:04:11.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:11.396 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:12.397 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:12.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.507+0000 7f817661c640 1 Processor -- start 2024-02-19T15:04:12.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.507+0000 7f817661c640 1 -- start start 2024-02-19T15:04:12.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.507+0000 7f817661c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81700c25e0 0x7f81701526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f817661c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81700c5be0 0x7f817006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f817661c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8170072250 con 0x7f81700c29c0 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f817661c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8170152c90 con 0x7f81700c25e0 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f817661c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8170072480 con 0x7f81700c5be0 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81700c25e0 0x7f81701526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f81700c25e0 0x7f81701526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58202/0 (socket says 172.21.15.78:58202) 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 -- 172.21.15.78:0/1498362283 learned_addr learned my addr 172.21.15.78:0/1498362283 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f8174bb9640 1 --2- 172.21.15.78:0/1498362283 >> v2:172.21.15.150:3300/0 conn(0x7f81700c5be0 0x7f817006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 -- 172.21.15.78:0/1498362283 >> v2:172.21.15.150:3300/0 conn(0x7f81700c5be0 msgr2=0x7f817006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816effd640 1 -- 172.21.15.78:0/1498362283 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543764961 0 0) 0x7f8170072250 con 0x7f81700c29c0 2024-02-19T15:04:12.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 --2- 172.21.15.78:0/1498362283 >> v2:172.21.15.150:3300/0 conn(0x7f81700c5be0 0x7f817006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 -- 172.21.15.78:0/1498362283 >> v1:172.21.15.150:6789/0 conn(0x7f81700c29c0 legacy=0x7f81700c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.508+0000 7f816ffff640 1 -- 172.21.15.78:0/1498362283 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700c4b90 con 0x7f81700c25e0 2024-02-19T15:04:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.509+0000 7f816ffff640 1 --2- 172.21.15.78:0/1498362283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c25e0 0x7f81701526a0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f8170151c50 tx=0x7f816002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ced4e862bccf65 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.509+0000 7f816effd640 1 -- 172.21.15.78:0/1498362283 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160031870 con 0x7f81700c25e0 2024-02-19T15:04:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.509+0000 7f816effd640 1 -- 172.21.15.78:0/1498362283 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81600319d0 con 0x7f81700c25e0 2024-02-19T15:04:12.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.509+0000 7f816effd640 1 -- 172.21.15.78:0/1498362283 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160031d60 con 0x7f81700c25e0 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 -- 172.21.15.78:0/1498362283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c25e0 msgr2=0x7f81701526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 --2- 172.21.15.78:0/1498362283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c25e0 0x7f81701526a0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7f8170151c50 tx=0x7f816002b910 comp rx=0 tx=0).stop 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 -- 172.21.15.78:0/1498362283 shutdown_connections 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 --2- 172.21.15.78:0/1498362283 >> v2:172.21.15.150:3300/0 conn(0x7f81700c5be0 0x7f817006e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 --2- 172.21.15.78:0/1498362283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c25e0 0x7f81701526a0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 -- 172.21.15.78:0/1498362283 >> 172.21.15.78:0/1498362283 conn(0x7f817006b270 msgr2=0x7f817006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 -- 172.21.15.78:0/1498362283 shutdown_connections 2024-02-19T15:04:12.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.510+0000 7f817661c640 1 -- 172.21.15.78:0/1498362283 wait complete. 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.511+0000 7f817661c640 1 Processor -- start 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.511+0000 7f817661c640 1 -- start start 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81700c25e0 0x7f81700a7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81700c29c0 0x7f8170154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 0x7f8170154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8170154270 con 0x7f81700c5be0 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8170155a00 con 0x7f81700c25e0 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81701ec6c0 con 0x7f81700c29c0 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 0x7f8170154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 0x7f8170154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58206/0 (socket says 172.21.15.78:58206) 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 -- 172.21.15.78:0/3936496339 learned_addr learned my addr 172.21.15.78:0/3936496339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:12.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f816f7fe640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81700c29c0 0x7f8170154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f816ffff640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81700c25e0 0x7f81700a7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 -- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81700c29c0 msgr2=0x7f8170154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81700c29c0 0x7f8170154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 -- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81700c25e0 msgr2=0x7f81700a7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81700c25e0 0x7f81700a7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 -- 172.21.15.78:0/3936496339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81600094d0 con 0x7f81700c5be0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f8174bb9640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 0x7f8170154bd0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f816800efc0 tx=0x7f816800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168014480 con 0x7f81700c5be0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816800f040 con 0x7f81700c5be0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168014820 con 0x7f81700c5be0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.512+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8170155c50 con 0x7f81700c5be0 2024-02-19T15:04:12.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.513+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81700c0850 con 0x7f81700c5be0 2024-02-19T15:04:12.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.514+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81701543b0 con 0x7f81700c5be0 2024-02-19T15:04:12.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.514+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8168014980 con 0x7f81700c5be0 2024-02-19T15:04:12.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.514+0000 7f816cff9640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f815c06b810 0x7f815c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:12.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.514+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f816808b0f0 con 0x7f81700c5be0 2024-02-19T15:04:12.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.514+0000 7f816ffff640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f815c06b810 0x7f815c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:12.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.515+0000 7f816ffff640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f815c06b810 0x7f815c06dcd0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8160036000 tx=0x7f816003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:12.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.519+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81701543b0 con 0x7f81700c5be0 2024-02-19T15:04:12.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.727+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81701543b0 con 0x7f81700c5be0 2024-02-19T15:04:12.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.727+0000 7f816cff9640 1 -- 172.21.15.78:0/3936496339 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f8168018d90 con 0x7f81700c5be0 2024-02-19T15:04:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:12.731 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:12.733 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f815c06b810 msgr2=0x7f815c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f815c06b810 0x7f815c06dcd0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8160036000 tx=0x7f816003c040 comp rx=0 tx=0).stop 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 msgr2=0x7f8170154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 0x7f8170154bd0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f816800efc0 tx=0x7f816800c490 comp rx=0 tx=0).stop 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 shutdown_connections 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f815c06b810 0x7f815c06dcd0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81700c5be0 0x7f8170154bd0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81700c29c0 0x7f8170154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 --2- 172.21.15.78:0/3936496339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81700c25e0 0x7f81700a7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:12.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 >> 172.21.15.78:0/3936496339 conn(0x7f817006b270 msgr2=0x7f81700ab840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:12.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 shutdown_connections 2024-02-19T15:04:12.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:12.729+0000 7f817661c640 1 -- 172.21.15.78:0/3936496339 wait complete. 2024-02-19T15:04:13.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:13.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:14.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.250+0000 7fc65781c640 1 Processor -- start 2024-02-19T15:04:14.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.250+0000 7fc65781c640 1 -- start start 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc65781c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc6500c19f0 0x7fc650156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc65781c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc6500c1dd0 0x7fc6500c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc65781c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc650155a50 con 0x7fc650071410 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc65781c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc650152af0 con 0x7fc6500c1dd0 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc65781c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc650155f80 con 0x7fc6500c19f0 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc655db9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc6500c1dd0 0x7fc6500c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:14.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc654db7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc650071410 0x7fc6500c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52572/0 (socket says 172.21.15.78:52572) 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc654db7640 1 -- 172.21.15.78:0/2362598424 learned_addr learned my addr 172.21.15.78:0/2362598424 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc6555b8640 1 --2- 172.21.15.78:0/2362598424 >> v2:172.21.15.150:3300/0 conn(0x7fc6500c19f0 0x7fc650156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc655db9640 1 -- 172.21.15.78:0/2362598424 >> v2:172.21.15.150:3300/0 conn(0x7fc6500c19f0 msgr2=0x7fc650156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc655db9640 1 --2- 172.21.15.78:0/2362598424 >> v2:172.21.15.150:3300/0 conn(0x7fc6500c19f0 0x7fc650156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc655db9640 1 -- 172.21.15.78:0/2362598424 >> v1:172.21.15.78:6789/0 conn(0x7fc650071410 legacy=0x7fc6500c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.251+0000 7fc655db9640 1 -- 172.21.15.78:0/2362598424 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650070840 con 0x7fc6500c1dd0 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc655db9640 1 --2- 172.21.15.78:0/2362598424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6500c1dd0 0x7fc6500c21e0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fc6500c2ca0 tx=0x7fc64802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1275038db6a505a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc647fff640 1 -- 172.21.15.78:0/2362598424 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64802cb60 con 0x7fc6500c1dd0 2024-02-19T15:04:14.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc647fff640 1 -- 172.21.15.78:0/2362598424 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64802ccc0 con 0x7fc6500c1dd0 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc647fff640 1 -- 172.21.15.78:0/2362598424 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc648034660 con 0x7fc6500c1dd0 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc65781c640 1 -- 172.21.15.78:0/2362598424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6500c1dd0 msgr2=0x7fc6500c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc65781c640 1 --2- 172.21.15.78:0/2362598424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6500c1dd0 0x7fc6500c21e0 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fc6500c2ca0 tx=0x7fc64802bdb0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc65781c640 1 -- 172.21.15.78:0/2362598424 shutdown_connections 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc65781c640 1 --2- 172.21.15.78:0/2362598424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6500c1dd0 0x7fc6500c21e0 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc65781c640 1 --2- 172.21.15.78:0/2362598424 >> v2:172.21.15.150:3300/0 conn(0x7fc6500c19f0 0x7fc650156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.252+0000 7fc65781c640 1 -- 172.21.15.78:0/2362598424 >> 172.21.15.78:0/2362598424 conn(0x7fc6500aa370 msgr2=0x7fc6500aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.253+0000 7fc65781c640 1 -- 172.21.15.78:0/2362598424 shutdown_connections 2024-02-19T15:04:14.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.253+0000 7fc65781c640 1 -- 172.21.15.78:0/2362598424 wait complete. 2024-02-19T15:04:14.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 Processor -- start 2024-02-19T15:04:14.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 -- start start 2024-02-19T15:04:14.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 0x7fc6501eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6500c19f0 0x7fc6501e0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6500c1dd0 0x7fc650156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501ee460 con 0x7fc650071410 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500c4df0 con 0x7fc6500c19f0 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc65781c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501ee110 con 0x7fc6500c1dd0 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc6555b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 0x7fc6501eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc6555b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 0x7fc6501eaba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58238/0 (socket says 172.21.15.78:58238) 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc6555b8640 1 -- 172.21.15.78:0/2399129556 learned_addr learned my addr 172.21.15.78:0/2399129556 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc654db7640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6500c19f0 0x7fc6501e0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.254+0000 7fc655db9640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6500c1dd0 0x7fc650156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc6555b8640 1 -- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6500c1dd0 msgr2=0x7fc650156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:14.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc6555b8640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6500c1dd0 0x7fc650156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc6555b8640 1 -- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6500c19f0 msgr2=0x7fc6501e0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc6555b8640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6500c19f0 0x7fc6501e0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc6555b8640 1 -- 172.21.15.78:0/2399129556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc648009a70 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc6555b8640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 0x7fc6501eaba0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00e8a0 tx=0x7fc64c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0094f0 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c00c940 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6501e1380 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c00ccf0 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.255+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6500c5320 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.256+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6500c8320 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.256+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc64c013470 con 0x7fc650071410 2024-02-19T15:04:14.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.257+0000 7fc645ffb640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc63406b810 0x7fc63406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:14.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.257+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc64c08a420 con 0x7fc650071410 2024-02-19T15:04:14.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.257+0000 7fc654db7640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc63406b810 0x7fc63406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:14.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.257+0000 7fc654db7640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc63406b810 0x7fc63406dcd0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc6500c4bb0 tx=0x7fc640007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:14.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.261+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc64c054cf0 con 0x7fc650071410 2024-02-19T15:04:14.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.467+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc65006c7f0 con 0x7fc650071410 2024-02-19T15:04:14.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.467+0000 7fc645ffb640 1 -- 172.21.15.78:0/2399129556 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fc64c017d90 con 0x7fc650071410 2024-02-19T15:04:14.471 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:14.471 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:14.471 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.469+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc63406b810 msgr2=0x7fc63406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.469+0000 7fc65781c640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc63406b810 0x7fc63406dcd0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fc6500c4bb0 tx=0x7fc640007e80 comp rx=0 tx=0).stop 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.469+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 msgr2=0x7fc6501eaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.469+0000 7fc65781c640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 0x7fc6501eaba0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00e8a0 tx=0x7fc64c00ed60 comp rx=0 tx=0).stop 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 shutdown_connections 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc63406b810 0x7fc63406dcd0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6500c1dd0 0x7fc650156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6500c19f0 0x7fc6501e0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 --2- 172.21.15.78:0/2399129556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc650071410 0x7fc6501eaba0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 >> 172.21.15.78:0/2399129556 conn(0x7fc6500aa370 msgr2=0x7fc650153eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 shutdown_connections 2024-02-19T15:04:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:14.470+0000 7fc65781c640 1 -- 172.21.15.78:0/2399129556 wait complete. 2024-02-19T15:04:14.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:14.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:15.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.000+0000 7fe77a9f4640 1 Processor -- start 2024-02-19T15:04:16.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.000+0000 7fe77a9f4640 1 -- start start 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe77a9f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe77406ff80 0x7fe774154f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe77a9f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77414f730 con 0x7fe7740c6d60 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe77a9f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740c16d0 con 0x7fe77406e830 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe77a9f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740c1940 con 0x7fe77406ff80 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe773fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe7740c6d60 0x7fe7740c0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:52584/0 (socket says 172.21.15.78:52584) 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe773fff640 1 -- 172.21.15.78:0/110785478 learned_addr learned my addr 172.21.15.78:0/110785478 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.001+0000 7fe778f91640 1 --2- 172.21.15.78:0/110785478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe77406ff80 0x7fe774154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246631737 0 0) 0x7fe77414f730 con 0x7fe7740c6d60 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c003520 con 0x7fe7740c6d60 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755532820 0 0) 0x7fe7740c16d0 con 0x7fe77406e830 2024-02-19T15:04:16.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77414f730 con 0x7fe77406e830 2024-02-19T15:04:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe778f91640 1 -- 172.21.15.78:0/110785478 >> v1:172.21.15.150:6789/0 conn(0x7fe77406e830 legacy=0x7fe7740c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe778f91640 1 -- 172.21.15.78:0/110785478 >> v1:172.21.15.78:6789/0 conn(0x7fe7740c6d60 legacy=0x7fe7740c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.002+0000 7fe778f91640 1 -- 172.21.15.78:0/110785478 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774072f70 con 0x7fe77406ff80 2024-02-19T15:04:16.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe778f91640 1 --2- 172.21.15.78:0/110785478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe77406ff80 0x7fe774154f70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009dc0 tx=0x7fe76c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ec4396b0e2dd60 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:16.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c02cc40 con 0x7fe77406ff80 2024-02-19T15:04:16.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c02cda0 con 0x7fe77406ff80 2024-02-19T15:04:16.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe772ffd640 1 -- 172.21.15.78:0/110785478 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c034660 con 0x7fe77406ff80 2024-02-19T15:04:16.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 -- 172.21.15.78:0/110785478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe77406ff80 msgr2=0x7fe774154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/110785478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe77406ff80 0x7fe774154f70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009dc0 tx=0x7fe76c02be90 comp rx=0 tx=0).stop 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 -- 172.21.15.78:0/110785478 shutdown_connections 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/110785478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe77406ff80 0x7fe774154f70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 -- 172.21.15.78:0/110785478 >> 172.21.15.78:0/110785478 conn(0x7fe77406c170 msgr2=0x7fe77406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 -- 172.21.15.78:0/110785478 shutdown_connections 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.003+0000 7fe77a9f4640 1 -- 172.21.15.78:0/110785478 wait complete. 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.004+0000 7fe77a9f4640 1 Processor -- start 2024-02-19T15:04:16.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 -- start start 2024-02-19T15:04:16.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 0x7fe774149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:16.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77406ff80 0x7fe7741ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7740c6d60 0x7fe7741eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7741486c0 con 0x7fe77406e830 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740a9cf0 con 0x7fe77406ff80 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe77a9f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7740a7270 con 0x7fe7740c6d60 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe773fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 0x7fe774149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe773fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 0x7fe774149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58252/0 (socket says 172.21.15.78:58252) 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.005+0000 7fe773fff640 1 -- 172.21.15.78:0/3614361837 learned_addr learned my addr 172.21.15.78:0/3614361837 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe778f91640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7740c6d60 0x7fe7741eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe7737fe640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77406ff80 0x7fe7741ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe773fff640 1 -- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7740c6d60 msgr2=0x7fe7741eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe773fff640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7740c6d60 0x7fe7741eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe773fff640 1 -- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77406ff80 msgr2=0x7fe7741ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe773fff640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77406ff80 0x7fe7741ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe773fff640 1 -- 172.21.15.78:0/3614361837 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c009a70 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe773fff640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 0x7fe774149e20 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c500 tx=0x7fe76400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764005ce0 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe764005e40 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7640072f0 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7740a9f00 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.006+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7740a7760 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.007+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe774074990 con 0x7fe77406e830 2024-02-19T15:04:16.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.008+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fe764007450 con 0x7fe77406e830 2024-02-19T15:04:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.008+0000 7fe770ff9640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe76006b840 0x7fe76006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.008+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe764089910 con 0x7fe77406e830 2024-02-19T15:04:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.008+0000 7fe7737fe640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe76006b840 0x7fe76006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:16.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.009+0000 7fe7737fe640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe76006b840 0x7fe76006dd00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe768007fd0 tx=0x7fe768009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:16.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.013+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe7640541e0 con 0x7fe77406e830 2024-02-19T15:04:16.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.219+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7741eefa0 con 0x7fe77406e830 2024-02-19T15:04:16.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.220+0000 7fe770ff9640 1 -- 172.21.15.78:0/3614361837 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fe764017d90 con 0x7fe77406e830 2024-02-19T15:04:16.223 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:16.223 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:16.223 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe76006b840 msgr2=0x7fe76006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe76006b840 0x7fe76006dd00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fe768007fd0 tx=0x7fe768009040 comp rx=0 tx=0).stop 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 msgr2=0x7fe774149e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 0x7fe774149e20 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c500 tx=0x7fe76400c9c0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 shutdown_connections 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fe76006b840 0x7fe76006dd00 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7740c6d60 0x7fe7741eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77406ff80 0x7fe7741ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 --2- 172.21.15.78:0/3614361837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe77406e830 0x7fe774149e20 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 >> 172.21.15.78:0/3614361837 conn(0x7fe77406c170 msgr2=0x7fe77406c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 shutdown_connections 2024-02-19T15:04:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:16.222+0000 7fe77a9f4640 1 -- 172.21.15.78:0/3614361837 wait complete. 2024-02-19T15:04:16.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:16.638 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:17.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:17.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.750+0000 7f0732bd7640 1 Processor -- start 2024-02-19T15:04:17.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.750+0000 7f0732bd7640 1 -- start start 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0732bd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0732bd7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f072c06d910 0x7f072c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0732bd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c06cd20 con 0x7f072c152700 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0732bd7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c1551f0 con 0x7f072c06d910 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0732bd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c155490 con 0x7f072c0c8e00 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0730973640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f072c152700 0x7f072c0c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42772/0 (socket says 172.21.15.78:42772) 2024-02-19T15:04:17.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0730973640 1 -- 172.21.15.78:0/1856295501 learned_addr learned my addr 172.21.15.78:0/1856295501 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072bfff640 1 --2- 172.21.15.78:0/1856295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f0731174640 1 --2- 172.21.15.78:0/1856295501 >> v2:172.21.15.150:3300/0 conn(0x7f072c06d910 0x7f072c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072b7fe640 1 -- 172.21.15.78:0/1856295501 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667985181 0 0) 0x7f072c06cd20 con 0x7f072c152700 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072b7fe640 1 -- 172.21.15.78:0/1856295501 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f071c003520 con 0x7f072c152700 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072bfff640 1 -- 172.21.15.78:0/1856295501 >> v2:172.21.15.150:3300/0 conn(0x7f072c06d910 msgr2=0x7f072c06c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072bfff640 1 --2- 172.21.15.78:0/1856295501 >> v2:172.21.15.150:3300/0 conn(0x7f072c06d910 0x7f072c06c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072bfff640 1 -- 172.21.15.78:0/1856295501 >> v1:172.21.15.78:6789/0 conn(0x7f072c152700 legacy=0x7f072c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.751+0000 7f072bfff640 1 -- 172.21.15.78:0/1856295501 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c155770 con 0x7f072c0c8e00 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.752+0000 7f072bfff640 1 --2- 172.21.15.78:0/1856295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c06c2a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07180097a0 tx=0x7f071802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f9e9f374f54d539 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.752+0000 7f072b7fe640 1 -- 172.21.15.78:0/1856295501 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071802e070 con 0x7f072c0c8e00 2024-02-19T15:04:17.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.752+0000 7f072b7fe640 1 -- 172.21.15.78:0/1856295501 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07180315a0 con 0x7f072c0c8e00 2024-02-19T15:04:17.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.752+0000 7f072b7fe640 1 -- 172.21.15.78:0/1856295501 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718031930 con 0x7f072c0c8e00 2024-02-19T15:04:17.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 -- 172.21.15.78:0/1856295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 msgr2=0x7f072c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:17.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 --2- 172.21.15.78:0/1856295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c06c2a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07180097a0 tx=0x7f071802b840 comp rx=0 tx=0).stop 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 -- 172.21.15.78:0/1856295501 shutdown_connections 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 --2- 172.21.15.78:0/1856295501 >> v2:172.21.15.150:3300/0 conn(0x7f072c06d910 0x7f072c06c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 --2- 172.21.15.78:0/1856295501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c06c2a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 -- 172.21.15.78:0/1856295501 >> 172.21.15.78:0/1856295501 conn(0x7f072c0cac90 msgr2=0x7f072c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 -- 172.21.15.78:0/1856295501 shutdown_connections 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.753+0000 7f0732bd7640 1 -- 172.21.15.78:0/1856295501 wait complete. 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.754+0000 7f0732bd7640 1 Processor -- start 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.754+0000 7f0732bd7640 1 -- start start 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0732bd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f072c06d910 0x7f072c1ea610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0732bd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0732bd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 0x7f072c1e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0732bd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c1ee170 con 0x7f072c152700 2024-02-19T15:04:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0732bd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c1ee450 con 0x7f072c06d910 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0732bd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c1ee730 con 0x7f072c0c8e00 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 0x7f072c1e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 0x7f072c1e7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33586/0 (socket says 172.21.15.78:33586) 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 -- 172.21.15.78:0/3459778651 learned_addr learned my addr 172.21.15.78:0/3459778651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0730973640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f072c06d910 0x7f072c1ea610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f072bfff640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 -- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 msgr2=0x7f072c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 -- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f072c06d910 msgr2=0x7f072c1ea610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f072c06d910 0x7f072c1ea610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 -- 172.21.15.78:0/3459778651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718009400 con 0x7f072c152700 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.755+0000 7f0731174640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 0x7f072c1e7920 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f072400efc0 tx=0x7f072400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.756+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724014480 con 0x7f072c152700 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.756+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072400f040 con 0x7f072c152700 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.756+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c1eea10 con 0x7f072c152700 2024-02-19T15:04:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.756+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724014820 con 0x7f072c152700 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.756+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c14a040 con 0x7f072c152700 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.757+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072c153db0 con 0x7f072c152700 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.757+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0724014980 con 0x7f072c152700 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.758+0000 7f07297fa640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f071006b810 0x7f071006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.758+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f072408ae20 con 0x7f072c152700 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.758+0000 7f0730973640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f071006b810 0x7f071006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:17.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.758+0000 7f0730973640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f071006b810 0x7f071006dcd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0720005880 tx=0x7f0720009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.762+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f07240556f0 con 0x7f072c152700 2024-02-19T15:04:17.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.969+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f072c074990 con 0x7f072c152700 2024-02-19T15:04:17.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.970+0000 7f07297fa640 1 -- 172.21.15.78:0/3459778651 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f0724018020 con 0x7f072c152700 2024-02-19T15:04:17.973 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:17.973 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:17.973 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f071006b810 msgr2=0x7f071006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f071006b810 0x7f071006dcd0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0720005880 tx=0x7f0720009040 comp rx=0 tx=0).stop 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 msgr2=0x7f072c1e7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 0x7f072c1e7920 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f072400efc0 tx=0x7f072400c490 comp rx=0 tx=0).stop 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 shutdown_connections 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f071006b810 0x7f071006dcd0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c152700 0x7f072c1e7920 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f072c0c8e00 0x7f072c149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 --2- 172.21.15.78:0/3459778651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f072c06d910 0x7f072c1ea610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:17.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 >> 172.21.15.78:0/3459778651 conn(0x7f072c0cac90 msgr2=0x7f072c06c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:17.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 shutdown_connections 2024-02-19T15:04:17.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:17.972+0000 7f0732bd7640 1 -- 172.21.15.78:0/3459778651 wait complete. 2024-02-19T15:04:18.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:18.389 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:19.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:19.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.502+0000 7f83b93bd640 1 Processor -- start 2024-02-19T15:04:19.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.502+0000 7f83b93bd640 1 -- start start 2024-02-19T15:04:19.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b93bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b40c8e00 0x7f83b406c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:19.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b93bd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83b406d910 0x7f83b406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b93bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b406cd20 con 0x7f83b4152700 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b93bd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b41551f0 con 0x7f83b406d910 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b93bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4155490 con 0x7f83b40c8e00 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83b406d910 0x7f83b406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b2ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f83b4152700 0x7f83b40c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42784/0 (socket says 172.21.15.78:42784) 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b2ffd640 1 -- 172.21.15.78:0/2436039122 learned_addr learned my addr 172.21.15.78:0/2436039122 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.503+0000 7f83b27fc640 1 --2- 172.21.15.78:0/2436039122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b40c8e00 0x7f83b406c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.504+0000 7f83b37fe640 1 -- 172.21.15.78:0/2436039122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b40c8e00 msgr2=0x7f83b406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.504+0000 7f83b37fe640 1 --2- 172.21.15.78:0/2436039122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b40c8e00 0x7f83b406c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.504+0000 7f83b37fe640 1 -- 172.21.15.78:0/2436039122 >> v1:172.21.15.78:6789/0 conn(0x7f83b4152700 legacy=0x7f83b40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.504+0000 7f83b37fe640 1 -- 172.21.15.78:0/2436039122 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4155770 con 0x7f83b406d910 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.504+0000 7f83b1ffb640 1 -- 172.21.15.78:0/2436039122 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942537202 0 0) 0x7f83b406cd20 con 0x7f83b4152700 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.504+0000 7f83b37fe640 1 --2- 172.21.15.78:0/2436039122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b406d910 0x7f83b406c7e0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0097a0 tx=0x7f83ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e77ca091b2e35d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:19.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b1ffb640 1 -- 172.21.15.78:0/2436039122 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac02e070 con 0x7f83b406d910 2024-02-19T15:04:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b1ffb640 1 -- 172.21.15.78:0/2436039122 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac0315a0 con 0x7f83b406d910 2024-02-19T15:04:19.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b1ffb640 1 -- 172.21.15.78:0/2436039122 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac031930 con 0x7f83b406d910 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 -- 172.21.15.78:0/2436039122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b406d910 msgr2=0x7f83b406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2436039122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b406d910 0x7f83b406c7e0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0097a0 tx=0x7f83ac02b840 comp rx=0 tx=0).stop 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 -- 172.21.15.78:0/2436039122 shutdown_connections 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2436039122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b406d910 0x7f83b406c7e0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2436039122 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b40c8e00 0x7f83b406c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 -- 172.21.15.78:0/2436039122 >> 172.21.15.78:0/2436039122 conn(0x7f83b40cac90 msgr2=0x7f83b40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 -- 172.21.15.78:0/2436039122 shutdown_connections 2024-02-19T15:04:19.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.505+0000 7f83b93bd640 1 -- 172.21.15.78:0/2436039122 wait complete. 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 Processor -- start 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 -- start start 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b406d910 0x7f83b40c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 0x7f83b40c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b4152700 0x7f83b41eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41eaef0 con 0x7f83b40c8e00 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40c3200 con 0x7f83b406d910 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b93bd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41eb420 con 0x7f83b4152700 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 0x7f83b40c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.507+0000 7f83b27fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 0x7f83b40c5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33594/0 (socket says 172.21.15.78:33594) 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 -- 172.21.15.78:0/2463819195 learned_addr learned my addr 172.21.15.78:0/2463819195 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b2ffd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b406d910 0x7f83b40c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:19.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b37fe640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b4152700 0x7f83b41eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 -- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b4152700 msgr2=0x7f83b41eb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b4152700 0x7f83b41eb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 -- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b406d910 msgr2=0x7f83b40c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b406d910 0x7f83b40c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 -- 172.21.15.78:0/2463819195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac009400 con 0x7f83b40c8e00 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b37fe640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b4152700 0x7f83b41eb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b27fc640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 0x7f83b40c5360 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f83a800e9b0 tx=0x7f83a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8009670 con 0x7f83b40c8e00 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a800c9a0 con 0x7f83b40c8e00 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b40c2bf0 con 0x7f83b40c8e00 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800cd50 con 0x7f83b40c8e00 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.508+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b40c2e90 con 0x7f83b40c8e00 2024-02-19T15:04:19.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.509+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b40c3370 con 0x7f83b40c8e00 2024-02-19T15:04:19.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.510+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f83a8013470 con 0x7f83b40c8e00 2024-02-19T15:04:19.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.510+0000 7f839b7fe640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f83a006b810 0x7f83a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:19.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.510+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f83a808a570 con 0x7f83b40c8e00 2024-02-19T15:04:19.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.510+0000 7f83b2ffd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f83a006b810 0x7f83a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:19.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.511+0000 7f83b2ffd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f83a006b810 0x7f83a006dcd0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f83a40012e0 tx=0x7f83a4001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:19.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.515+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f83a8054e40 con 0x7f83b40c8e00 2024-02-19T15:04:19.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.722+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b4074990 con 0x7f83b40c8e00 2024-02-19T15:04:19.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.722+0000 7f839b7fe640 1 -- 172.21.15.78:0/2463819195 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f83a8017d90 con 0x7f83b40c8e00 2024-02-19T15:04:19.726 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:19.726 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:19.726 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.724+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f83a006b810 msgr2=0x7f83a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.724+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f83a006b810 0x7f83a006dcd0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f83a40012e0 tx=0x7f83a4001630 comp rx=0 tx=0).stop 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.724+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 msgr2=0x7f83b40c5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.724+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 0x7f83b40c5360 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f83a800e9b0 tx=0x7f83a800ee70 comp rx=0 tx=0).stop 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 shutdown_connections 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f83a006b810 0x7f83a006dcd0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83b40c8e00 0x7f83b40c5360 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83b4152700 0x7f83b41eb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 --2- 172.21.15.78:0/2463819195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b406d910 0x7f83b40c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 >> 172.21.15.78:0/2463819195 conn(0x7f83b40cac90 msgr2=0x7f83b4151670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 shutdown_connections 2024-02-19T15:04:19.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:19.725+0000 7f83b93bd640 1 -- 172.21.15.78:0/2463819195 wait complete. 2024-02-19T15:04:20.141 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:20.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:21.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.250+0000 7efeac85b640 1 Processor -- start 2024-02-19T15:04:21.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.250+0000 7efeac85b640 1 -- start start 2024-02-19T15:04:21.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efeac85b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efea8155f90 0x7efea806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efeac85b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea80c6d30 0x7efea80c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efeac85b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea80c6af0 con 0x7efea80c3aa0 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efeac85b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea80c7940 con 0x7efea8155f90 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efeac85b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea80c7c20 con 0x7efea80c6d30 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efea5d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea80c6d30 0x7efea80c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efea659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efea8155f90 0x7efea806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efea6d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efea80c3aa0 0x7efea81588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42742/0 (socket says 172.21.15.78:42742) 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.251+0000 7efea6d9d640 1 -- 172.21.15.78:0/2149280112 learned_addr learned my addr 172.21.15.78:0/2149280112 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea659c640 1 -- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea80c6d30 msgr2=0x7efea80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea659c640 1 --2- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea80c6d30 0x7efea80c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea659c640 1 -- 172.21.15.78:0/2149280112 >> v1:172.21.15.150:6789/0 conn(0x7efea80c3aa0 legacy=0x7efea81588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea559a640 1 -- 172.21.15.78:0/2149280112 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232093844 0 0) 0x7efea80c6af0 con 0x7efea80c3aa0 2024-02-19T15:04:21.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea659c640 1 -- 172.21.15.78:0/2149280112 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea80c7f00 con 0x7efea8155f90 2024-02-19T15:04:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea659c640 1 --2- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea8155f90 0x7efea806d550 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7efe980097a0 tx=0x7efe9802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e249a9351d19b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.252+0000 7efea559a640 1 -- 172.21.15.78:0/2149280112 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9802e070 con 0x7efea8155f90 2024-02-19T15:04:21.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efea559a640 1 -- 172.21.15.78:0/2149280112 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe980315a0 con 0x7efea8155f90 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efea559a640 1 -- 172.21.15.78:0/2149280112 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98031930 con 0x7efea8155f90 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 -- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea8155f90 msgr2=0x7efea806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 --2- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea8155f90 0x7efea806d550 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7efe980097a0 tx=0x7efe9802b840 comp rx=0 tx=0).stop 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 -- 172.21.15.78:0/2149280112 shutdown_connections 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 --2- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea80c6d30 0x7efea80c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 --2- 172.21.15.78:0/2149280112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea8155f90 0x7efea806d550 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 -- 172.21.15.78:0/2149280112 >> 172.21.15.78:0/2149280112 conn(0x7efea806b700 msgr2=0x7efea80aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 -- 172.21.15.78:0/2149280112 shutdown_connections 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.253+0000 7efeac85b640 1 -- 172.21.15.78:0/2149280112 wait complete. 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 Processor -- start 2024-02-19T15:04:21.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 -- start start 2024-02-19T15:04:21.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea80c3aa0 0x7efea809ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:21.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 0x7efea81ed1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:21.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea8155f90 0x7efea80ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea809e410 con 0x7efea80c6d30 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8099a60 con 0x7efea80c3aa0 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efeac85b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea81eafa0 con 0x7efea8155f90 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efea5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 0x7efea81ed1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efea5d9b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 0x7efea81ed1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33596/0 (socket says 172.21.15.78:33596) 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efea5d9b640 1 -- 172.21.15.78:0/2011467603 learned_addr learned my addr 172.21.15.78:0/2011467603 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efea6d9d640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea8155f90 0x7efea80ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.255+0000 7efea659c640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea80c3aa0 0x7efea809ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efea5d9b640 1 -- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea8155f90 msgr2=0x7efea80ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efea5d9b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea8155f90 0x7efea80ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efea5d9b640 1 -- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea80c3aa0 msgr2=0x7efea809ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efea5d9b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea80c3aa0 0x7efea809ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efea5d9b640 1 -- 172.21.15.78:0/2011467603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98009400 con 0x7efea80c6d30 2024-02-19T15:04:21.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efea5d9b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 0x7efea81ed1e0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00e9b0 tx=0x7efe9c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c009670 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c00c9a0 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea81eb280 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00cd50 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.256+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea80ad470 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.257+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea80c9560 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.258+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7efe9c013470 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.258+0000 7efe96ffd640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe8806b810 0x7efe8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.258+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7efe9c08a540 con 0x7efea80c6d30 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.258+0000 7efea659c640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe8806b810 0x7efe8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:21.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.258+0000 7efea659c640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe8806b810 0x7efe8806dcd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7efe98036000 tx=0x7efe9803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:21.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.262+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efe9c054e10 con 0x7efea80c6d30 2024-02-19T15:04:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.467+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efea8074990 con 0x7efea80c6d30 2024-02-19T15:04:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.468+0000 7efe96ffd640 1 -- 172.21.15.78:0/2011467603 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7efe9c017d90 con 0x7efea80c6d30 2024-02-19T15:04:21.471 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:21.471 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.469+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe8806b810 msgr2=0x7efe8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.469+0000 7efeac85b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe8806b810 0x7efe8806dcd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7efe98036000 tx=0x7efe9803d040 comp rx=0 tx=0).stop 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.469+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 msgr2=0x7efea81ed1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.469+0000 7efeac85b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 0x7efea81ed1e0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00e9b0 tx=0x7efe9c00ee70 comp rx=0 tx=0).stop 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 shutdown_connections 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7efe8806b810 0x7efe8806dcd0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efea80c6d30 0x7efea81ed1e0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efea8155f90 0x7efea80ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 --2- 172.21.15.78:0/2011467603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efea80c3aa0 0x7efea809ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 >> 172.21.15.78:0/2011467603 conn(0x7efea806b700 msgr2=0x7efea8158100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 shutdown_connections 2024-02-19T15:04:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:21.470+0000 7efeac85b640 1 -- 172.21.15.78:0/2011467603 wait complete. 2024-02-19T15:04:21.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:21.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:22.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:23.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.997+0000 7fef3c377640 1 Processor -- start 2024-02-19T15:04:23.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.997+0000 7fef3c377640 1 -- start start 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.998+0000 7fef3c377640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef34070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.998+0000 7fef3c377640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef340ae670 con 0x7fef340c4980 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.998+0000 7fef3c377640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef340716b0 con 0x7fef340add70 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.998+0000 7fef3c377640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef340c33b0 con 0x7fef340c2f60 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.998+0000 7fef39912640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fef340c4980 0x7fef340705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42792/0 (socket says 172.21.15.78:42792) 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.998+0000 7fef39912640 1 -- 172.21.15.78:0/730864083 learned_addr learned my addr 172.21.15.78:0/730864083 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef3a914640 1 --2- 172.21.15.78:0/730864083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef34070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40593739 0 0) 0x7fef340ae670 con 0x7fef340c4980 2024-02-19T15:04:23.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef20003520 con 0x7fef340c4980 2024-02-19T15:04:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986127936 0 0) 0x7fef340716b0 con 0x7fef340add70 2024-02-19T15:04:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef340ae670 con 0x7fef340add70 2024-02-19T15:04:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef3a914640 1 -- 172.21.15.78:0/730864083 >> v1:172.21.15.150:6789/0 conn(0x7fef340add70 legacy=0x7fef3406eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef3a914640 1 -- 172.21.15.78:0/730864083 >> v1:172.21.15.78:6789/0 conn(0x7fef340c4980 legacy=0x7fef340705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:22.999+0000 7fef3a914640 1 -- 172.21.15.78:0/730864083 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef34153780 con 0x7fef340c2f60 2024-02-19T15:04:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.000+0000 7fef3a914640 1 --2- 172.21.15.78:0/730864083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef34070cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef300097d0 tx=0x7fef3002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6536dbee54680cdc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.000+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30031800 con 0x7fef340c2f60 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.000+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef30031960 con 0x7fef340c2f60 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.000+0000 7fef39111640 1 -- 172.21.15.78:0/730864083 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30031cf0 con 0x7fef340c2f60 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.000+0000 7fef3c377640 1 -- 172.21.15.78:0/730864083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 msgr2=0x7fef34070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.000+0000 7fef3c377640 1 --2- 172.21.15.78:0/730864083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef34070cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef300097d0 tx=0x7fef3002b8a0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.001+0000 7fef3c377640 1 -- 172.21.15.78:0/730864083 shutdown_connections 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.001+0000 7fef3c377640 1 --2- 172.21.15.78:0/730864083 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef34070cc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.001+0000 7fef3c377640 1 -- 172.21.15.78:0/730864083 >> 172.21.15.78:0/730864083 conn(0x7fef340c85e0 msgr2=0x7fef340c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.001+0000 7fef3c377640 1 -- 172.21.15.78:0/730864083 shutdown_connections 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.001+0000 7fef3c377640 1 -- 172.21.15.78:0/730864083 wait complete. 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 Processor -- start 2024-02-19T15:04:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 -- start start 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef340add70 0x7fef340c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef341ed750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 0x7fef341edc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef34154a10 con 0x7fef340c4980 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341ea160 con 0x7fef340add70 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3c377640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef34154bf0 con 0x7fef340c2f60 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.002+0000 7fef3a914640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 0x7fef341edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 0x7fef341edc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33612/0 (socket says 172.21.15.78:33612) 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 -- 172.21.15.78:0/724814390 learned_addr learned my addr 172.21.15.78:0/724814390 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a113640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef340add70 0x7fef340c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef39912640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef341ed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 -- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 msgr2=0x7fef341ed750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef341ed750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 -- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef340add70 msgr2=0x7fef340c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef340add70 0x7fef340c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 -- 172.21.15.78:0/724814390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef30009400 con 0x7fef340c4980 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3a914640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 0x7fef341edc90 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fef300097d0 tx=0x7fef300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3003bbf0 con 0x7fef340c4980 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3003bd50 con 0x7fef340c4980 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef341ea420 con 0x7fef340c4980 2024-02-19T15:04:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30045630 con 0x7fef340c4980 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.003+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef341ea9e0 con 0x7fef340c4980 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.004+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef34074990 con 0x7fef340c4980 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.005+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fef3004e450 con 0x7fef340c4980 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.005+0000 7fef2affd640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fef1806b810 0x7fef1806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.005+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fef3002e070 con 0x7fef340c4980 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.005+0000 7fef3a113640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fef1806b810 0x7fef1806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.006+0000 7fef3a113640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fef1806b810 0x7fef1806dcd0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fef340c7420 tx=0x7fef2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:23.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.010+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fef30076e00 con 0x7fef340c4980 2024-02-19T15:04:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.217+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef34071490 con 0x7fef340c4980 2024-02-19T15:04:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.217+0000 7fef2affd640 1 -- 172.21.15.78:0/724814390 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fef30035d90 con 0x7fef340c4980 2024-02-19T15:04:23.220 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:23.221 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:23.221 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fef1806b810 msgr2=0x7fef1806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fef1806b810 0x7fef1806dcd0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fef340c7420 tx=0x7fef2c009040 comp rx=0 tx=0).stop 2024-02-19T15:04:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 msgr2=0x7fef341edc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 0x7fef341edc90 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fef300097d0 tx=0x7fef300046c0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 shutdown_connections 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fef1806b810 0x7fef1806dcd0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef340c4980 0x7fef341edc90 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef340c2f60 0x7fef341ed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 --2- 172.21.15.78:0/724814390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef340add70 0x7fef340c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 >> 172.21.15.78:0/724814390 conn(0x7fef340c85e0 msgr2=0x7fef340c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 shutdown_connections 2024-02-19T15:04:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:23.219+0000 7fef3c377640 1 -- 172.21.15.78:0/724814390 wait complete. 2024-02-19T15:04:23.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:23.636 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:24.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:24.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.740+0000 7fb408423640 1 Processor -- start 2024-02-19T15:04:24.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.740+0000 7fb408423640 1 -- start start 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb408423640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb400156b90 0x7fb40006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb408423640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb40006c330 0x7fb40006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb408423640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb400156f70 con 0x7fb4000c8e90 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb408423640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40015d7a0 con 0x7fb400156b90 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb408423640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40015cb20 con 0x7fb40006c330 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb4061bf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb400156b90 0x7fb40006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb4061bf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb400156b90 0x7fb40006d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33618/0 (socket says 172.21.15.78:33618) 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb4059be640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb4000c8e90 0x7fb4000c7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42802/0 (socket says 172.21.15.78:42802) 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.741+0000 7fb4061bf640 1 -- 172.21.15.78:0/3670389782 learned_addr learned my addr 172.21.15.78:0/3670389782 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4069c0640 1 --2- 172.21.15.78:0/3670389782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb40006c330 0x7fb40006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4061bf640 1 -- 172.21.15.78:0/3670389782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb40006c330 msgr2=0x7fb40006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4061bf640 1 --2- 172.21.15.78:0/3670389782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb40006c330 0x7fb40006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4061bf640 1 -- 172.21.15.78:0/3670389782 >> v1:172.21.15.78:6789/0 conn(0x7fb4000c8e90 legacy=0x7fb4000c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4061bf640 1 -- 172.21.15.78:0/3670389782 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40015ce00 con 0x7fb400156b90 2024-02-19T15:04:24.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4051bd640 1 -- 172.21.15.78:0/3670389782 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382079314 0 0) 0x7fb400156f70 con 0x7fb4000c8e90 2024-02-19T15:04:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.742+0000 7fb4061bf640 1 --2- 172.21.15.78:0/3670389782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb40006d8d0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009870 tx=0x7fb3f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f87a00ec41cb5cd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb4051bd640 1 -- 172.21.15.78:0/3670389782 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8031640 con 0x7fb400156b90 2024-02-19T15:04:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb4051bd640 1 -- 172.21.15.78:0/3670389782 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f80317a0 con 0x7fb400156b90 2024-02-19T15:04:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb4051bd640 1 -- 172.21.15.78:0/3670389782 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8031b30 con 0x7fb400156b90 2024-02-19T15:04:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 -- 172.21.15.78:0/3670389782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 msgr2=0x7fb40006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 --2- 172.21.15.78:0/3670389782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb40006d8d0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009870 tx=0x7fb3f802b8f0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 -- 172.21.15.78:0/3670389782 shutdown_connections 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 --2- 172.21.15.78:0/3670389782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb40006c330 0x7fb40006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 --2- 172.21.15.78:0/3670389782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb40006d8d0 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 -- 172.21.15.78:0/3670389782 >> 172.21.15.78:0/3670389782 conn(0x7fb4000cac90 msgr2=0x7fb4000cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 -- 172.21.15.78:0/3670389782 shutdown_connections 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.743+0000 7fb408423640 1 -- 172.21.15.78:0/3670389782 wait complete. 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.744+0000 7fb408423640 1 Processor -- start 2024-02-19T15:04:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.744+0000 7fb408423640 1 -- start start 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb408423640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40006c330 0x7fb4001f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb408423640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4000c8e90 0x7fb4000ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb408423640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb4001ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb408423640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4001590d0 con 0x7fb400156b90 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb408423640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000af330 con 0x7fb40006c330 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb408423640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000afb00 con 0x7fb4000c8e90 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb4069c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb4001ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb4069c0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb4001ef3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33626/0 (socket says 172.21.15.78:33626) 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb4069c0640 1 -- 172.21.15.78:0/3312819532 learned_addr learned my addr 172.21.15.78:0/3312819532 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb4061bf640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40006c330 0x7fb4001f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.745+0000 7fb4059be640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4000c8e90 0x7fb4000ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4069c0640 1 -- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4000c8e90 msgr2=0x7fb4000ac7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4069c0640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4000c8e90 0x7fb4000ac7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4069c0640 1 -- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40006c330 msgr2=0x7fb4001f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4069c0640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40006c330 0x7fb4001f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4069c0640 1 -- 172.21.15.78:0/3312819532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f80094d0 con 0x7fb400156b90 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4059be640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4000c8e90 0x7fb4000ac7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb4069c0640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb4001ef3f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00efc0 tx=0x7fb3fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc014480 con 0x7fb400156b90 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc00f040 con 0x7fb400156b90 2024-02-19T15:04:24.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4001eef60 con 0x7fb400156b90 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc014820 con 0x7fb400156b90 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.746+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4000af820 con 0x7fb400156b90 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.747+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4001eed30 con 0x7fb400156b90 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.747+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb3fc014980 con 0x7fb400156b90 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.748+0000 7fb3f6ffd640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3e406b810 0x7fb3e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.748+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb3fc08b1a0 con 0x7fb400156b90 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.748+0000 7fb4061bf640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3e406b810 0x7fb3e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:24.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.748+0000 7fb4061bf640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3e406b810 0x7fb3e406dcd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8036000 tx=0x7fb3f803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:24.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.752+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4001eed30 con 0x7fb400156b90 2024-02-19T15:04:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.960+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4001eed30 con 0x7fb400156b90 2024-02-19T15:04:24.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.960+0000 7fb3f6ffd640 1 -- 172.21.15.78:0/3312819532 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fb3fc018d90 con 0x7fb400156b90 2024-02-19T15:04:24.963 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:24.963 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:24.964 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3e406b810 msgr2=0x7fb3e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3e406b810 0x7fb3e406dcd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8036000 tx=0x7fb3f803c040 comp rx=0 tx=0).stop 2024-02-19T15:04:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 msgr2=0x7fb4001ef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb4001ef3f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00efc0 tx=0x7fb3fc00c490 comp rx=0 tx=0).stop 2024-02-19T15:04:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 shutdown_connections 2024-02-19T15:04:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb3e406b810 0x7fb3e406dcd0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb400156b90 0x7fb4001ef3f0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4000c8e90 0x7fb4000ac7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 --2- 172.21.15.78:0/3312819532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40006c330 0x7fb4001f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 >> 172.21.15.78:0/3312819532 conn(0x7fb4000cac90 msgr2=0x7fb4000c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 shutdown_connections 2024-02-19T15:04:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:24.962+0000 7fb408423640 1 -- 172.21.15.78:0/3312819532 wait complete. 2024-02-19T15:04:25.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:25.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:26.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:26.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.484+0000 7f6b735b5640 1 Processor -- start 2024-02-19T15:04:26.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.484+0000 7f6b735b5640 1 -- start start 2024-02-19T15:04:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b735b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6b6c0c7070 0x7f6b6c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b735b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c155ef0 con 0x7f6b6c0c87a0 2024-02-19T15:04:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b735b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c06dec0 con 0x7f6b6c152480 2024-02-19T15:04:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b735b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6c06d520 con 0x7f6b6c0c7070 2024-02-19T15:04:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b70b50640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6b6c0c7070 0x7f6b6c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b71b52640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6b6c0c87a0 0x7f6b6c0c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42810/0 (socket says 172.21.15.78:42810) 2024-02-19T15:04:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b71b52640 1 -- 172.21.15.78:0/1665666014 learned_addr learned my addr 172.21.15.78:0/1665666014 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b70b50640 1 -- 172.21.15.78:0/1665666014 >> v1:172.21.15.150:6789/0 conn(0x7f6b6c152480 legacy=0x7f6b6c06ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b70b50640 1 -- 172.21.15.78:0/1665666014 >> v1:172.21.15.78:6789/0 conn(0x7f6b6c0c87a0 legacy=0x7f6b6c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b70b50640 1 -- 172.21.15.78:0/1665666014 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c0c75c0 con 0x7f6b6c0c7070 2024-02-19T15:04:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.485+0000 7f6b63fff640 1 -- 172.21.15.78:0/1665666014 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729954982 0 0) 0x7f6b64001490 con 0x7f6b6c0c87a0 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.486+0000 7f6b70b50640 1 --2- 172.21.15.78:0/1665666014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c0c7070 0x7f6b6c1559b0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0097a0 tx=0x7f6b5c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0df78d1fba6cb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.486+0000 7f6b63fff640 1 -- 172.21.15.78:0/1665666014 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c030610 con 0x7f6b6c0c7070 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.486+0000 7f6b63fff640 1 -- 172.21.15.78:0/1665666014 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c030770 con 0x7f6b6c0c7070 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.486+0000 7f6b63fff640 1 -- 172.21.15.78:0/1665666014 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c030b00 con 0x7f6b6c0c7070 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 -- 172.21.15.78:0/1665666014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c0c7070 msgr2=0x7f6b6c1559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 --2- 172.21.15.78:0/1665666014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c0c7070 0x7f6b6c1559b0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0097a0 tx=0x7f6b5c02b960 comp rx=0 tx=0).stop 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 -- 172.21.15.78:0/1665666014 shutdown_connections 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 --2- 172.21.15.78:0/1665666014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c0c7070 0x7f6b6c1559b0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 -- 172.21.15.78:0/1665666014 >> 172.21.15.78:0/1665666014 conn(0x7f6b6c0ca8a0 msgr2=0x7f6b6c0cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 -- 172.21.15.78:0/1665666014 shutdown_connections 2024-02-19T15:04:26.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.487+0000 7f6b735b5640 1 -- 172.21.15.78:0/1665666014 wait complete. 2024-02-19T15:04:26.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.488+0000 7f6b735b5640 1 Processor -- start 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.488+0000 7f6b735b5640 1 -- start start 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b735b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b6c0c7070 0x7f6b6c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b735b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b6c0c87a0 0x7f6b6c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b735b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 0x7f6b6c1ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b735b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c1ee1d0 con 0x7f6b6c152480 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b735b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0b0300 con 0x7f6b6c0c87a0 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b735b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0aeee0 con 0x7f6b6c0c7070 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 0x7f6b6c1ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 0x7f6b6c1ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33650/0 (socket says 172.21.15.78:33650) 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 -- 172.21.15.78:0/3005961960 learned_addr learned my addr 172.21.15.78:0/3005961960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71351640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b6c0c7070 0x7f6b6c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b70b50640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b6c0c87a0 0x7f6b6c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 -- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b6c0c7070 msgr2=0x7f6b6c0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b6c0c7070 0x7f6b6c0afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 -- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b6c0c87a0 msgr2=0x7f6b6c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b6c0c87a0 0x7f6b6c1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 -- 172.21.15.78:0/3005961960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c009400 con 0x7f6b6c152480 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b71b52640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 0x7f6b6c1ee5f0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400c7f0 tx=0x7f6b6400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6400a5e0 con 0x7f6b6c152480 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6400a760 con 0x7f6b6c152480 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.489+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6400ab10 con 0x7f6b6c152480 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.490+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c0af1c0 con 0x7f6b6c152480 2024-02-19T15:04:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.490+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c158a20 con 0x7f6b6c152480 2024-02-19T15:04:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.491+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c1ee310 con 0x7f6b6c152480 2024-02-19T15:04:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.491+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f6b6400ac70 con 0x7f6b6c152480 2024-02-19T15:04:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.491+0000 7f6b61ffb640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b5006b810 0x7f6b5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.491+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6b640899e0 con 0x7f6b6c152480 2024-02-19T15:04:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.491+0000 7f6b71351640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b5006b810 0x7f6b5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.492+0000 7f6b71351640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b5006b810 0x7f6b5006dcd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6b68000f60 tx=0x7f6b68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:26.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.496+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6b6c1ee310 con 0x7f6b6c152480 2024-02-19T15:04:26.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.704+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b6c155ef0 con 0x7f6b6c152480 2024-02-19T15:04:26.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.704+0000 7f6b61ffb640 1 -- 172.21.15.78:0/3005961960 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f6b6c155ef0 con 0x7f6b6c152480 2024-02-19T15:04:26.708 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:26.708 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:26.708 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:26.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b5006b810 msgr2=0x7f6b5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b5006b810 0x7f6b5006dcd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6b68000f60 tx=0x7f6b68009040 comp rx=0 tx=0).stop 2024-02-19T15:04:26.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 msgr2=0x7f6b6c1ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:26.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 0x7f6b6c1ee5f0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400c7f0 tx=0x7f6b6400ccb0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 shutdown_connections 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f6b5006b810 0x7f6b5006dcd0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b6c152480 0x7f6b6c1ee5f0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b6c0c87a0 0x7f6b6c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 --2- 172.21.15.78:0/3005961960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b6c0c7070 0x7f6b6c0afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 >> 172.21.15.78:0/3005961960 conn(0x7f6b6c0ca8a0 msgr2=0x7f6b6c0c9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 shutdown_connections 2024-02-19T15:04:26.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:26.706+0000 7f6b735b5640 1 -- 172.21.15.78:0/3005961960 wait complete. 2024-02-19T15:04:27.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:27.122 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:28.123 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:28.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.238+0000 7fc2b05e2640 1 Processor -- start 2024-02-19T15:04:28.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.238+0000 7fc2b05e2640 1 -- start start 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2b05e2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc2a8152480 0x7fc2a81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2b05e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a80c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2b05e2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a8155ef0 con 0x7fc2a80c7070 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2b05e2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a806dec0 con 0x7fc2a8152480 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2b05e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a806d520 con 0x7fc2a80c87a0 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2aeb7f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a80c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2ae37e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc2a8152480 0x7fc2a81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2adb7d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc2a80c7070 0x7fc2a806ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:39284/0 (socket says 172.21.15.78:39284) 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.239+0000 7fc2adb7d640 1 -- 172.21.15.78:0/3969084280 learned_addr learned my addr 172.21.15.78:0/3969084280 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2ad37c640 1 -- 172.21.15.78:0/3969084280 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100854935 0 0) 0x7fc298001110 con 0x7fc2a80c7070 2024-02-19T15:04:28.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2ad37c640 1 -- 172.21.15.78:0/3969084280 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc294003520 con 0x7fc2a80c7070 2024-02-19T15:04:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/3969084280 >> v2:172.21.15.150:3300/0 conn(0x7fc2a8152480 msgr2=0x7fc2a81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2aeb7f640 1 --2- 172.21.15.78:0/3969084280 >> v2:172.21.15.150:3300/0 conn(0x7fc2a8152480 0x7fc2a81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/3969084280 >> v1:172.21.15.150:6789/0 conn(0x7fc2a80c7070 legacy=0x7fc2a806ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/3969084280 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80c75c0 con 0x7fc2a80c87a0 2024-02-19T15:04:28.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.240+0000 7fc2ae37e640 1 --2- 172.21.15.78:0/3969084280 >> v2:172.21.15.150:3300/0 conn(0x7fc2a8152480 0x7fc2a81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:28.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2aeb7f640 1 --2- 172.21.15.78:0/3969084280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a80c5220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009d90 tx=0x7fc2a002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f251861fc3393ed server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:28.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2ad37c640 1 -- 172.21.15.78:0/3969084280 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a002cc50 con 0x7fc2a80c87a0 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2ad37c640 1 -- 172.21.15.78:0/3969084280 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a002cdb0 con 0x7fc2a80c87a0 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2ad37c640 1 -- 172.21.15.78:0/3969084280 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0034660 con 0x7fc2a80c87a0 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2b05e2640 1 -- 172.21.15.78:0/3969084280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 msgr2=0x7fc2a80c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/3969084280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a80c5220 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009d90 tx=0x7fc2a002bea0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2b05e2640 1 -- 172.21.15.78:0/3969084280 shutdown_connections 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/3969084280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a80c5220 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/3969084280 >> v2:172.21.15.150:3300/0 conn(0x7fc2a8152480 0x7fc2a81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.241+0000 7fc2b05e2640 1 -- 172.21.15.78:0/3969084280 >> 172.21.15.78:0/3969084280 conn(0x7fc2a80ca8a0 msgr2=0x7fc2a80cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.242+0000 7fc2b05e2640 1 -- 172.21.15.78:0/3969084280 shutdown_connections 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.242+0000 7fc2b05e2640 1 -- 172.21.15.78:0/3969084280 wait complete. 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 Processor -- start 2024-02-19T15:04:28.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 -- start start 2024-02-19T15:04:28.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80c7070 0x7fc2a80ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:28.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a81ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 0x7fc2a81ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8157ed0 con 0x7fc2a8152480 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a81eaba0 con 0x7fc2a80c7070 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2b05e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80af080 con 0x7fc2a80c87a0 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.243+0000 7fc2aeb7f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 0x7fc2a81ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2ae37e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80c7070 0x7fc2a80ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 0x7fc2a81ee930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33554/0 (socket says 172.21.15.78:33554) 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/4121889192 learned_addr learned my addr 172.21.15.78:0/4121889192 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2adb7d640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a81ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 msgr2=0x7fc2a81ee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a81ee3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80c7070 msgr2=0x7fc2a80ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80c7070 0x7fc2a80ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 -- 172.21.15.78:0/4121889192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a4009400 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2adb7d640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a81ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2ae37e640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80c7070 0x7fc2a80ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2aeb7f640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 0x7fc2a81ee930 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009eb0 tx=0x7fc2a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0040650 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a00407b0 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0040b60 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a0009a70 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.244+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a80ad640 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.245+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a80ad4a0 con 0x7fc2a8152480 2024-02-19T15:04:28.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.246+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fc2a0040cc0 con 0x7fc2a8152480 2024-02-19T15:04:28.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.246+0000 7fc29effd640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc28c06b810 0x7fc28c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:28.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.246+0000 7fc2ae37e640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc28c06b810 0x7fc28c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:28.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.247+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc2a00ad060 con 0x7fc2a8152480 2024-02-19T15:04:28.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.247+0000 7fc2ae37e640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc28c06b810 0x7fc28c06dcd0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8159260 tx=0x7fc2a4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:28.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.251+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc2a80ad4a0 con 0x7fc2a8152480 2024-02-19T15:04:28.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.460+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a81ee110 con 0x7fc2a8152480 2024-02-19T15:04:28.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.460+0000 7fc29effd640 1 -- 172.21.15.78:0/4121889192 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fc2a0039d90 con 0x7fc2a8152480 2024-02-19T15:04:28.463 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:28.463 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:28.463 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:28.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc28c06b810 msgr2=0x7fc28c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:28.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc28c06b810 0x7fc28c06dcd0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8159260 tx=0x7fc2a4009190 comp rx=0 tx=0).stop 2024-02-19T15:04:28.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 msgr2=0x7fc2a81ee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:28.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 0x7fc2a81ee930 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009eb0 tx=0x7fc2a0004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:28.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 shutdown_connections 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fc28c06b810 0x7fc28c06dcd0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2a8152480 0x7fc2a81ee930 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc2a80c87a0 0x7fc2a81ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 --2- 172.21.15.78:0/4121889192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80c7070 0x7fc2a80ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 >> 172.21.15.78:0/4121889192 conn(0x7fc2a80ca8a0 msgr2=0x7fc2a8152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 shutdown_connections 2024-02-19T15:04:28.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:28.462+0000 7fc2b05e2640 1 -- 172.21.15.78:0/4121889192 wait complete. 2024-02-19T15:04:28.564 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:04:28.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:04:28.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:28.879 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:29.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:29.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.994+0000 7f16e8aec640 1 Processor -- start 2024-02-19T15:04:29.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.994+0000 7f16e8aec640 1 -- start start 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e8aec640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16e4150340 0x7f16e4072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e8aec640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16e4073230 0x7f16e40ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e8aec640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e4153a60 con 0x7f16e40ace90 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e8aec640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e4152e50 con 0x7f16e4150340 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e8aec640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e4153130 con 0x7f16e4073230 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16e4150340 0x7f16e4072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:29.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16e4150340 0x7f16e4072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33558/0 (socket says 172.21.15.78:33558) 2024-02-19T15:04:29.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 -- 172.21.15.78:0/1794358194 learned_addr learned my addr 172.21.15.78:0/1794358194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:29.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16dbfff640 1 --2- 172.21.15.78:0/1794358194 >> v2:172.21.15.150:3300/0 conn(0x7f16e4073230 0x7f16e40ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:29.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 -- 172.21.15.78:0/1794358194 >> v2:172.21.15.150:3300/0 conn(0x7f16e4073230 msgr2=0x7f16e40ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:29.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 --2- 172.21.15.78:0/1794358194 >> v2:172.21.15.150:3300/0 conn(0x7f16e4073230 0x7f16e40ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:29.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 -- 172.21.15.78:0/1794358194 >> v1:172.21.15.150:6789/0 conn(0x7f16e40ace90 legacy=0x7f16e4156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:29.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.995+0000 7f16e259c640 1 -- 172.21.15.78:0/1794358194 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4153410 con 0x7f16e4150340 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.996+0000 7f16e259c640 1 --2- 172.21.15.78:0/1794358194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4072cf0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f16e406b550 tx=0x7f16d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5dd83096c1884fc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.996+0000 7f16e1d9b640 1 -- 172.21.15.78:0/1794358194 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d4031870 con 0x7f16e4150340 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.996+0000 7f16e1d9b640 1 -- 172.21.15.78:0/1794358194 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d40319d0 con 0x7f16e4150340 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.996+0000 7f16e1d9b640 1 -- 172.21.15.78:0/1794358194 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d4031d60 con 0x7f16e4150340 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 -- 172.21.15.78:0/1794358194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 msgr2=0x7f16e4072cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 --2- 172.21.15.78:0/1794358194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4072cf0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f16e406b550 tx=0x7f16d402b910 comp rx=0 tx=0).stop 2024-02-19T15:04:30.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 -- 172.21.15.78:0/1794358194 shutdown_connections 2024-02-19T15:04:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 --2- 172.21.15.78:0/1794358194 >> v2:172.21.15.150:3300/0 conn(0x7f16e4073230 0x7f16e40ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 --2- 172.21.15.78:0/1794358194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4072cf0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 -- 172.21.15.78:0/1794358194 >> 172.21.15.78:0/1794358194 conn(0x7f16e40c7590 msgr2=0x7f16e40c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 -- 172.21.15.78:0/1794358194 shutdown_connections 2024-02-19T15:04:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.997+0000 7f16e8aec640 1 -- 172.21.15.78:0/1794358194 wait complete. 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.998+0000 7f16e8aec640 1 Processor -- start 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.998+0000 7f16e8aec640 1 -- start start 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e8aec640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16e4073230 0x7f16e414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e8aec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e40ace90 0x7f16e4150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e8aec640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e8aec640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4154fe0 con 0x7f16e4150340 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e8aec640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4156140 con 0x7f16e40ace90 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e8aec640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e41ec3e0 con 0x7f16e4073230 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4144af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33570/0 (socket says 172.21.15.78:33570) 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e259c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16e4073230 0x7f16e414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 -- 172.21.15.78:0/4292839994 learned_addr learned my addr 172.21.15.78:0/4292839994 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e40ace90 0x7f16e4150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 -- 172.21.15.78:0/4292839994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16e4073230 msgr2=0x7f16e414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16e4073230 0x7f16e414ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 -- 172.21.15.78:0/4292839994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e40ace90 msgr2=0x7f16e4150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e40ace90 0x7f16e4150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:29.999+0000 7f16e2d9d640 1 -- 172.21.15.78:0/4292839994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d40094d0 con 0x7f16e4150340 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.000+0000 7f16e2d9d640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4144af0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00c710 tx=0x7f16dc00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.000+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc006d50 con 0x7f16e4150340 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.000+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc00ce20 con 0x7f16e4150340 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.000+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e41ec6c0 con 0x7f16e4150340 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.000+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16dc003bb0 con 0x7f16e4150340 2024-02-19T15:04:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.000+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e4155b10 con 0x7f16e4150340 2024-02-19T15:04:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.001+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e414dbf0 con 0x7f16e4150340 2024-02-19T15:04:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.001+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f16dc003d10 con 0x7f16e4150340 2024-02-19T15:04:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.002+0000 7f16daffd640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c406b810 0x7f16c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.002+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f16dc089e80 con 0x7f16e4150340 2024-02-19T15:04:30.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.002+0000 7f16e259c640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c406b810 0x7f16c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:30.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.002+0000 7f16e259c640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c406b810 0x7f16c406dcd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f16d4036000 tx=0x7f16d403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:30.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.006+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f16e414dbf0 con 0x7f16e4150340 2024-02-19T15:04:30.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.212+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16e414dbf0 con 0x7f16e4150340 2024-02-19T15:04:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.213+0000 7f16daffd640 1 -- 172.21.15.78:0/4292839994 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f16e414dbf0 con 0x7f16e4150340 2024-02-19T15:04:30.216 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:30.216 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:30.216 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c406b810 msgr2=0x7f16c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c406b810 0x7f16c406dcd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f16d4036000 tx=0x7f16d403c040 comp rx=0 tx=0).stop 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 msgr2=0x7f16e4144af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4144af0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00c710 tx=0x7f16dc00cbd0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 shutdown_connections 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f16c406b810 0x7f16c406dcd0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16e4150340 0x7f16e4144af0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e40ace90 0x7f16e4150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 --2- 172.21.15.78:0/4292839994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f16e4073230 0x7f16e414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:30.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 >> 172.21.15.78:0/4292839994 conn(0x7f16e40c7590 msgr2=0x7f16e41487c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 shutdown_connections 2024-02-19T15:04:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:30.215+0000 7f16e8aec640 1 -- 172.21.15.78:0/4292839994 wait complete. 2024-02-19T15:04:30.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:30.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:31.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:31.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.742+0000 7f2894d27640 1 Processor -- start 2024-02-19T15:04:31.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.742+0000 7f2894d27640 1 -- start start 2024-02-19T15:04:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f2894d27640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac770 0x7f28900c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f2894d27640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2890158c40 con 0x7f28900bdf20 2024-02-19T15:04:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f2894d27640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28900acbb0 con 0x7f28900ac390 2024-02-19T15:04:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f2894d27640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28900c8de0 con 0x7f28900ac770 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288dd9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f28900bdf20 0x7f289015d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42488/0 (socket says 172.21.15.78:42488) 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288dd9b640 1 -- 172.21.15.78:0/4017918996 learned_addr learned my addr 172.21.15.78:0/4017918996 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288ed9d640 1 --2- 172.21.15.78:0/4017918996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac770 0x7f28900c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081055086 0 0) 0x7f2890158c40 con 0x7f28900bdf20 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878003520 con 0x7f28900bdf20 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191508911 0 0) 0x7f28900acbb0 con 0x7f28900ac390 2024-02-19T15:04:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.743+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2890158c40 con 0x7f28900ac390 2024-02-19T15:04:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.744+0000 7f288ed9d640 1 -- 172.21.15.78:0/4017918996 >> v1:172.21.15.150:6789/0 conn(0x7f28900ac390 legacy=0x7f2890158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.744+0000 7f288ed9d640 1 -- 172.21.15.78:0/4017918996 >> v1:172.21.15.78:6789/0 conn(0x7f28900bdf20 legacy=0x7f289015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.744+0000 7f288ed9d640 1 -- 172.21.15.78:0/4017918996 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2890155a60 con 0x7f28900ac770 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.744+0000 7f288ed9d640 1 --2- 172.21.15.78:0/4017918996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac770 0x7f28900c55f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2888009dc0 tx=0x7f288802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1e824b5b7e640a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.744+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288802cbb0 con 0x7f28900ac770 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288802cd10 con 0x7f28900ac770 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f288d59a640 1 -- 172.21.15.78:0/4017918996 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888034640 con 0x7f28900ac770 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 -- 172.21.15.78:0/4017918996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac770 msgr2=0x7f28900c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 --2- 172.21.15.78:0/4017918996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac770 0x7f28900c55f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2888009dc0 tx=0x7f288802be00 comp rx=0 tx=0).stop 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 -- 172.21.15.78:0/4017918996 shutdown_connections 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 --2- 172.21.15.78:0/4017918996 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac770 0x7f28900c55f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 -- 172.21.15.78:0/4017918996 >> 172.21.15.78:0/4017918996 conn(0x7f28900ca890 msgr2=0x7f28900caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 -- 172.21.15.78:0/4017918996 shutdown_connections 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.745+0000 7f2894d27640 1 -- 172.21.15.78:0/4017918996 wait complete. 2024-02-19T15:04:31.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.746+0000 7f2894d27640 1 Processor -- start 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.746+0000 7f2894d27640 1 -- start start 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f2894d27640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac390 0x7f2890157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f2894d27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28900ac770 0x7f28900bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f2894d27640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 0x7f28900c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f2894d27640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2890151a80 con 0x7f28900bdf20 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f2894d27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2890151c10 con 0x7f28900ac770 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f2894d27640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28901eb590 con 0x7f28900ac390 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 0x7f28900c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f288ed9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 0x7f28900c01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33588/0 (socket says 172.21.15.78:33588) 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f288ed9d640 1 -- 172.21.15.78:0/556860670 learned_addr learned my addr 172.21.15.78:0/556860670 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f288dd9b640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28900ac770 0x7f28900bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.747+0000 7f288e59c640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac390 0x7f2890157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:31.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f288ed9d640 1 -- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac390 msgr2=0x7f2890157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f288ed9d640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac390 0x7f2890157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f288ed9d640 1 -- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28900ac770 msgr2=0x7f28900bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f288ed9d640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28900ac770 0x7f28900bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f288ed9d640 1 -- 172.21.15.78:0/556860670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2888009a70 con 0x7f28900bdf20 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f288ed9d640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 0x7f28900c01b0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f288802c410 tx=0x7f2888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888007000 con 0x7f28900bdf20 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2888040650 con 0x7f28900bdf20 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888040a00 con 0x7f28900bdf20 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28900c0bc0 con 0x7f28900bdf20 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.748+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28901eaf40 con 0x7f28900bdf20 2024-02-19T15:04:31.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.749+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28900ac1b0 con 0x7f28900bdf20 2024-02-19T15:04:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.749+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2888040b60 con 0x7f28900bdf20 2024-02-19T15:04:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.750+0000 7f287effd640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287006b810 0x7f287006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.750+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f28880ad010 con 0x7f28900bdf20 2024-02-19T15:04:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.750+0000 7f288e59c640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287006b810 0x7f287006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:31.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.750+0000 7f288e59c640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287006b810 0x7f287006dcd0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f2880007fd0 tx=0x7f2880009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:31.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.754+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2888077ac0 con 0x7f28900bdf20 2024-02-19T15:04:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.961+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2890074990 con 0x7f28900bdf20 2024-02-19T15:04:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.962+0000 7f287effd640 1 -- 172.21.15.78:0/556860670 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f2888044d90 con 0x7f28900bdf20 2024-02-19T15:04:31.965 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:31.965 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287006b810 msgr2=0x7f287006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287006b810 0x7f287006dcd0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f2880007fd0 tx=0x7f2880009040 comp rx=0 tx=0).stop 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 msgr2=0x7f28900c01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 0x7f28900c01b0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f288802c410 tx=0x7f2888004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 shutdown_connections 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f287006b810 0x7f287006dcd0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28900bdf20 0x7f28900c01b0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28900ac770 0x7f28900bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 --2- 172.21.15.78:0/556860670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28900ac390 0x7f2890157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 >> 172.21.15.78:0/556860670 conn(0x7f28900ca890 msgr2=0x7f28900ab1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 shutdown_connections 2024-02-19T15:04:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:31.964+0000 7f2894d27640 1 -- 172.21.15.78:0/556860670 wait complete. 2024-02-19T15:04:32.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:32.380 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:33.381 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:33.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.494+0000 7fdb90d6b640 1 Processor -- start 2024-02-19T15:04:33.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.494+0000 7fdb90d6b640 1 -- start start 2024-02-19T15:04:33.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb90d6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb8c0c8500 0x7fdb8c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:33.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb90d6b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb8c0c2f50 0x7fdb8c158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:33.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb90d6b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c0c34a0 con 0x7fdb8c0c42e0 2024-02-19T15:04:33.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb90d6b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c06c090 con 0x7fdb8c0c2f50 2024-02-19T15:04:33.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb90d6b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c06d6a0 con 0x7fdb8c0c8500 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb8ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb8c0c2f50 0x7fdb8c158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb83fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdb8c0c42e0 0x7fdb8c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42490/0 (socket says 172.21.15.78:42490) 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb83fff640 1 -- 172.21.15.78:0/2751650863 learned_addr learned my addr 172.21.15.78:0/2751650863 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb8a59c640 1 --2- 172.21.15.78:0/2751650863 >> v2:172.21.15.150:3300/0 conn(0x7fdb8c0c8500 0x7fdb8c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/2751650863 >> v2:172.21.15.150:3300/0 conn(0x7fdb8c0c8500 msgr2=0x7fdb8c0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb8ad9d640 1 --2- 172.21.15.78:0/2751650863 >> v2:172.21.15.150:3300/0 conn(0x7fdb8c0c8500 0x7fdb8c0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/2751650863 >> v1:172.21.15.78:6789/0 conn(0x7fdb8c0c42e0 legacy=0x7fdb8c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb89d9b640 1 -- 172.21.15.78:0/2751650863 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929754275 0 0) 0x7fdb8c0c34a0 con 0x7fdb8c0c42e0 2024-02-19T15:04:33.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.495+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/2751650863 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c06c8e0 con 0x7fdb8c0c2f50 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.496+0000 7fdb8ad9d640 1 --2- 172.21.15.78:0/2751650863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c158d60 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0c9350 tx=0x7fdb8402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd02c8790f6d57f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.496+0000 7fdb89d9b640 1 -- 172.21.15.78:0/2751650863 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402cbd0 con 0x7fdb8c0c2f50 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.496+0000 7fdb89d9b640 1 -- 172.21.15.78:0/2751650863 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8402cd30 con 0x7fdb8c0c2f50 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.496+0000 7fdb89d9b640 1 -- 172.21.15.78:0/2751650863 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402cea0 con 0x7fdb8c0c2f50 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 -- 172.21.15.78:0/2751650863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c0c2f50 msgr2=0x7fdb8c158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/2751650863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c158d60 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0c9350 tx=0x7fdb8402be20 comp rx=0 tx=0).stop 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 -- 172.21.15.78:0/2751650863 shutdown_connections 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/2751650863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c158d60 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/2751650863 >> v2:172.21.15.150:3300/0 conn(0x7fdb8c0c8500 0x7fdb8c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 -- 172.21.15.78:0/2751650863 >> 172.21.15.78:0/2751650863 conn(0x7fdb8c0aa300 msgr2=0x7fdb8c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 -- 172.21.15.78:0/2751650863 shutdown_connections 2024-02-19T15:04:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.497+0000 7fdb90d6b640 1 -- 172.21.15.78:0/2751650863 wait complete. 2024-02-19T15:04:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.498+0000 7fdb90d6b640 1 Processor -- start 2024-02-19T15:04:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.498+0000 7fdb90d6b640 1 -- start start 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb90d6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c0acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb90d6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb8c0c8500 0x7fdb8c14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb90d6b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 0x7fdb8c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb90d6b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0abe00 con 0x7fdb8c159030 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb90d6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0c5800 con 0x7fdb8c0c2f50 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb90d6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c1493f0 con 0x7fdb8c0c8500 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 0x7fdb8c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 0x7fdb8c156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33602/0 (socket says 172.21.15.78:33602) 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/798271862 learned_addr learned my addr 172.21.15.78:0/798271862 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb83fff640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb8c0c8500 0x7fdb8c14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8a59c640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c0acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb8c0c8500 msgr2=0x7fdb8c14af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb8c0c8500 0x7fdb8c14af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb8c0c2f50 msgr2=0x7fdb8c0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c0acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb8ad9d640 1 -- 172.21.15.78:0/798271862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb84009a70 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.499+0000 7fdb83fff640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb8c0c8500 0x7fdb8c14af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.500+0000 7fdb8ad9d640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 0x7fdb8c156a80 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009d90 tx=0x7fdb84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.500+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8402cea0 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.500+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb84003bb0 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.500+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c0ac050 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.500+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84009450 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.500+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c0ac610 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.501+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb8c071c20 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.501+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fdb84004030 con 0x7fdb8c159030 2024-02-19T15:04:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.502+0000 7fdb82ffd640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb6c06b840 0x7fdb6c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:33.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.502+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdb840aca50 con 0x7fdb8c159030 2024-02-19T15:04:33.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.502+0000 7fdb8a59c640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb6c06b840 0x7fdb6c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:33.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.502+0000 7fdb8a59c640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb6c06b840 0x7fdb6c06dd00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009aa0 tx=0x7fdb7c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.506+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdb84077320 con 0x7fdb8c159030 2024-02-19T15:04:33.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.712+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb8c074990 con 0x7fdb8c159030 2024-02-19T15:04:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.712+0000 7fdb82ffd640 1 -- 172.21.15.78:0/798271862 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fdb84039d90 con 0x7fdb8c159030 2024-02-19T15:04:33.715 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:33.715 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb6c06b840 msgr2=0x7fdb6c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb6c06b840 0x7fdb6c06dd00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009aa0 tx=0x7fdb7c009190 comp rx=0 tx=0).stop 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 msgr2=0x7fdb8c156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 0x7fdb8c156a80 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009d90 tx=0x7fdb84004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 shutdown_connections 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fdb6c06b840 0x7fdb6c06dd00 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb8c159030 0x7fdb8c156a80 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb8c0c8500 0x7fdb8c14af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 --2- 172.21.15.78:0/798271862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb8c0c2f50 0x7fdb8c0acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 >> 172.21.15.78:0/798271862 conn(0x7fdb8c0aa300 msgr2=0x7fdb8c155410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 shutdown_connections 2024-02-19T15:04:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:33.714+0000 7fdb90d6b640 1 -- 172.21.15.78:0/798271862 wait complete. 2024-02-19T15:04:34.130 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-19T15:00:48.648504+0000', 'modified': '2024-02-19T15:02:40.643214+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 217, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 10731, 'mds_1': 9217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9217': {'gid': 9217, 'name': 'b', 'rank': 1, 'incarnation': 202, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.150:6834/4270793546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 4270793546}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 4270793546}]}, 'join_fscid': -1, 'laggy_since': '2024-02-19T15:02:40.643214+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 188, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/1952320195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 1952320195}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 1952320195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-19T15:04:34.130 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9217:up:stopping 2024-02-19T15:04:35.132 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 16, "epoch": 208, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 16, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-19T15:00:48.648504+0000", "damaged": [], "data_pools": [ 34 ], "enabled": true, "epoch": 208, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_10731": { "addr": "172.21.15.150:6835/1952320195", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6833", "nonce": 1952320195, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1952320195, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 10731, "incarnation": 188, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 }, "gid_9217": { "addr": "172.21.15.150:6834/4270793546", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 4270793546, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 4270793546, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9217, "incarnation": 202, "join_fscid": -1, "laggy_since": "2024-02-19T15:02:40.643214+0000", "name": "b", "rank": 1, "state": "up:stopping", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 217, "max_file_size": 1099511627776, "max_mds": 0, "max_xattr_size": 65536, "metadata_pool": 33, "modified": "2024-02-19T15:02:40.643214+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10731, "mds_1": 9217 } } } ], "standbys": [] } 2024-02-19T15:04:35.132 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-19T15:04:35.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:04:35.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-19T15:04:35.230 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-19T15:04:35.230 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T15:04:35.230 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:04:35.248 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T15:04:35.248 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T15:04:35.248 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:04:35.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T15:04:35.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:35.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 Processor -- start 2024-02-19T15:04:35.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 -- start start 2024-02-19T15:04:35.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4734155e00 0x7f47340c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47340c39e0 0x7f473415f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340c7df0 con 0x7f47340c67e0 2024-02-19T15:04:35.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340c4310 con 0x7f47340c39e0 2024-02-19T15:04:35.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.389+0000 7f4739b9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340c77c0 con 0x7f4734155e00 2024-02-19T15:04:35.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47340c39e0 0x7f473415f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f47340c39e0 0x7f473415f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33608/0 (socket says 172.21.15.78:33608) 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 -- 172.21.15.78:0/741918204 learned_addr learned my addr 172.21.15.78:0/741918204 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f47337fe640 1 --2- 172.21.15.78:0/741918204 >> v2:172.21.15.150:3300/0 conn(0x7f4734155e00 0x7f47340c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 -- 172.21.15.78:0/741918204 >> v2:172.21.15.150:3300/0 conn(0x7f4734155e00 msgr2=0x7f47340c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 --2- 172.21.15.78:0/741918204 >> v2:172.21.15.150:3300/0 conn(0x7f4734155e00 0x7f47340c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 -- 172.21.15.78:0/741918204 >> v1:172.21.15.150:6789/0 conn(0x7f47340c67e0 legacy=0x7f473406c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.390+0000 7f4733fff640 1 -- 172.21.15.78:0/741918204 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340c70d0 con 0x7f47340c39e0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4733fff640 1 --2- 172.21.15.78:0/741918204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c39e0 0x7f473415f9c0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f473406be60 tx=0x7f472c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec4272344bce929 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f47327fc640 1 -- 172.21.15.78:0/741918204 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c02ca80 con 0x7f47340c39e0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f47327fc640 1 -- 172.21.15.78:0/741918204 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c02cbe0 con 0x7f47340c39e0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f47327fc640 1 -- 172.21.15.78:0/741918204 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c0343f0 con 0x7f47340c39e0 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 -- 172.21.15.78:0/741918204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c39e0 msgr2=0x7f473415f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 --2- 172.21.15.78:0/741918204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c39e0 0x7f473415f9c0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f473406be60 tx=0x7f472c02bee0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 -- 172.21.15.78:0/741918204 shutdown_connections 2024-02-19T15:04:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 --2- 172.21.15.78:0/741918204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c39e0 0x7f473415f9c0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 --2- 172.21.15.78:0/741918204 >> v2:172.21.15.150:3300/0 conn(0x7f4734155e00 0x7f47340c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 -- 172.21.15.78:0/741918204 >> 172.21.15.78:0/741918204 conn(0x7f473406b700 msgr2=0x7f47340aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 -- 172.21.15.78:0/741918204 shutdown_connections 2024-02-19T15:04:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.391+0000 7f4739b9a640 1 -- 172.21.15.78:0/741918204 wait complete. 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.392+0000 7f4739b9a640 1 Processor -- start 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 -- start start 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47340c39e0 0x7f47341ead30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 0x7f473414e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4734155e00 0x7f473414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47340ad780 con 0x7f47340c67e0 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47341eb270 con 0x7f4734155e00 2024-02-19T15:04:35.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4739b9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4734099b00 con 0x7f47340c39e0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4732ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 0x7f473414e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4732ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 0x7f473414e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33622/0 (socket says 172.21.15.78:33622) 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4732ffd640 1 -- 172.21.15.78:0/1013906225 learned_addr learned my addr 172.21.15.78:0/1013906225 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f4733fff640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4734155e00 0x7f473414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.393+0000 7f47337fe640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47340c39e0 0x7f47341ead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4732ffd640 1 -- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47340c39e0 msgr2=0x7f47341ead30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4732ffd640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47340c39e0 0x7f47341ead30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4732ffd640 1 -- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4734155e00 msgr2=0x7f473414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4732ffd640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4734155e00 0x7f473414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4732ffd640 1 -- 172.21.15.78:0/1013906225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c009b30 con 0x7f47340c67e0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4733fff640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4734155e00 0x7f473414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4732ffd640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 0x7f473414e960 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f472800c400 tx=0x7f472800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47280052d0 con 0x7f47340c67e0 2024-02-19T15:04:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4728005450 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4734099de0 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728007260 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.394+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47340a9e20 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.395+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47340aa040 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.395+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f47280073c0 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.396+0000 7f471bfff640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f472006b810 0x7f472006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.396+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4728089770 con 0x7f47340c67e0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.396+0000 7f47337fe640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f472006b810 0x7f472006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.396+0000 7f47337fe640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f472006b810 0x7f472006dcd0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4724009b40 tx=0x7f4724009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.400+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4728054040 con 0x7f47340c67e0 2024-02-19T15:04:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.605+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47340c6cb0 con 0x7f47340c67e0 2024-02-19T15:04:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.606+0000 7f471bfff640 1 -- 172.21.15.78:0/1013906225 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f4728057700 con 0x7f47340c67e0 2024-02-19T15:04:35.609 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:35.609 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f472006b810 msgr2=0x7f472006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f472006b810 0x7f472006dcd0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4724009b40 tx=0x7f4724009190 comp rx=0 tx=0).stop 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 msgr2=0x7f473414e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 0x7f473414e960 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f472800c400 tx=0x7f472800c8c0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 shutdown_connections 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f472006b810 0x7f472006dcd0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47340c67e0 0x7f473414e960 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4734155e00 0x7f473414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 --2- 172.21.15.78:0/1013906225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47340c39e0 0x7f47341ead30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 >> 172.21.15.78:0/1013906225 conn(0x7f473406b700 msgr2=0x7f47340aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:35.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 shutdown_connections 2024-02-19T15:04:35.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.608+0000 7f4739b9a640 1 -- 172.21.15.78:0/1013906225 wait complete. 2024-02-19T15:04:35.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:04:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.717+0000 7f55ffc4e640 1 Processor -- start 2024-02-19T15:04:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.718+0000 7f55ffc4e640 1 -- start start 2024-02-19T15:04:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.718+0000 7f55ffc4e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55f806ed20 0x7f55f806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.718+0000 7f55ffc4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f806d750 0x7f55f815d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.718+0000 7f55ffc4e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f8150820 con 0x7f55f8150440 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.718+0000 7f55ffc4e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f80c1b60 con 0x7f55f806ed20 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.718+0000 7f55ffc4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f81537d0 con 0x7f55f806d750 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd1e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f55f806ed20 0x7f55f806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd9ea640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f55f8150440 0x7f55f80c32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42504/0 (socket says 172.21.15.78:42504) 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd9ea640 1 -- 172.21.15.78:0/2387980613 learned_addr learned my addr 172.21.15.78:0/2387980613 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fe1eb640 1 --2- 172.21.15.78:0/2387980613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f806d750 0x7f55f815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fc9e8640 1 -- 172.21.15.78:0/2387980613 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913315833 0 0) 0x7f55f8150820 con 0x7f55f8150440 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fc9e8640 1 -- 172.21.15.78:0/2387980613 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e4003520 con 0x7f55f8150440 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd1e9640 1 -- 172.21.15.78:0/2387980613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f806d750 msgr2=0x7f55f815d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd1e9640 1 --2- 172.21.15.78:0/2387980613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f806d750 0x7f55f815d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd1e9640 1 -- 172.21.15.78:0/2387980613 >> v1:172.21.15.78:6789/0 conn(0x7f55f8150440 legacy=0x7f55f80c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.719+0000 7f55fd1e9640 1 -- 172.21.15.78:0/2387980613 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f8153d10 con 0x7f55f806ed20 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55fd1e9640 1 --2- 172.21.15.78:0/2387980613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806ed20 0x7f55f806d210 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f55f8153f00 tx=0x7f55e802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e063bf0ae79b50d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55fc9e8640 1 -- 172.21.15.78:0/2387980613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8031890 con 0x7f55f806ed20 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55fc9e8640 1 -- 172.21.15.78:0/2387980613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e80319f0 con 0x7f55f806ed20 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55fc9e8640 1 -- 172.21.15.78:0/2387980613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8031d80 con 0x7f55f806ed20 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2387980613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806ed20 msgr2=0x7f55f806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2387980613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806ed20 0x7f55f806d210 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f55f8153f00 tx=0x7f55e802b930 comp rx=0 tx=0).stop 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2387980613 shutdown_connections 2024-02-19T15:04:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2387980613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806ed20 0x7f55f806d210 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2387980613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f806d750 0x7f55f815d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2387980613 >> 172.21.15.78:0/2387980613 conn(0x7f55f806b270 msgr2=0x7f55f806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2387980613 shutdown_connections 2024-02-19T15:04:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.720+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2387980613 wait complete. 2024-02-19T15:04:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.721+0000 7f55ffc4e640 1 Processor -- start 2024-02-19T15:04:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.721+0000 7f55ffc4e640 1 -- start start 2024-02-19T15:04:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55ffc4e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 0x7f55f81568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55ffc4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55f806ed20 0x7f55f8150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55ffc4e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f8150440 0x7f55f81ec030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55ffc4e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f8154190 con 0x7f55f806d750 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55ffc4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f8154470 con 0x7f55f806ed20 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55ffc4e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f81e9820 con 0x7f55f8150440 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55fd9ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 0x7f55f81568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55fd9ea640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 0x7f55f81568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33640/0 (socket says 172.21.15.78:33640) 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55fd9ea640 1 -- 172.21.15.78:0/2511910363 learned_addr learned my addr 172.21.15.78:0/2511910363 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55fe1eb640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f8150440 0x7f55f81ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.722+0000 7f55fd1e9640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55f806ed20 0x7f55f8150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55fd9ea640 1 -- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f8150440 msgr2=0x7f55f81ec030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55fd9ea640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f8150440 0x7f55f81ec030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55fd9ea640 1 -- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55f806ed20 msgr2=0x7f55f8150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55fd9ea640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55f806ed20 0x7f55f8150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55fd9ea640 1 -- 172.21.15.78:0/2511910363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e8009400 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55fd9ea640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 0x7f55f81568e0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f55f000caa0 tx=0x7f55f000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f000ac00 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f000ad60 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f8156e20 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0012620 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.723+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f813f9c0 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.724+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f81e99e0 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.724+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f55f0007070 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.725+0000 7f55ee7fc640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55dc06b840 0x7f55dc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.725+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f55f008a290 con 0x7f55f806d750 2024-02-19T15:04:35.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.725+0000 7f55fd1e9640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55dc06b840 0x7f55dc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:35.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.725+0000 7f55fd1e9640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55dc06b840 0x7f55dc06dd00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:35.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.729+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f55f0054b60 con 0x7f55f806d750 2024-02-19T15:04:35.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.900+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55f81545e0 con 0x7f55f806d750 2024-02-19T15:04:35.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.901+0000 7f55ee7fc640 1 -- 172.21.15.78:0/2511910363 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+17273 (secure 0 0 0) 0x7f55f0001b60 con 0x7f55f806d750 2024-02-19T15:04:35.904 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:35.904 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":217,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:02:05.637768+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:00:46.861039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-19T15:00:47.624123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.2","mappings":[{"from":0,"to":2}]},{"pgid":"34.b","mappings":[{"from":0,"to":1}]},{"pgid":"34.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6835/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6834/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6837/159409097":"2024-02-20T15:01:30.632749+0000","172.21.15.78:6836/159409097":"2024-02-20T15:01:30.632749+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:04:35.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55dc06b840 msgr2=0x7f55dc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55dc06b840 0x7f55dc06dd00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e803c040 comp rx=0 tx=0).stop 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 msgr2=0x7f55f81568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 0x7f55f81568e0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f55f000caa0 tx=0x7f55f000cf60 comp rx=0 tx=0).stop 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 shutdown_connections 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f55dc06b840 0x7f55dc06dd00 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55f8150440 0x7f55f81ec030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55f806ed20 0x7f55f8150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 --2- 172.21.15.78:0/2511910363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55f806d750 0x7f55f81568e0 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 >> 172.21.15.78:0/2511910363 conn(0x7f55f806b270 msgr2=0x7f55f806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 shutdown_connections 2024-02-19T15:04:35.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:35.903+0000 7f55ffc4e640 1 -- 172.21.15.78:0/2511910363 wait complete. 2024-02-19T15:04:35.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T14:51:48.105873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-19T15:00:46.861039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2024-02-19T15:00:47.624123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T15:04:35.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T15:04:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.017+0000 7f9e5219a640 1 Processor -- start 2024-02-19T15:04:36.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.017+0000 7f9e5219a640 1 -- start start 2024-02-19T15:04:36.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e5219a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c6a50 0x7f9e4c155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e5219a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9e4c0c5920 0x7f9e4c155e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e5219a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4c0717b0 con 0x7f9e4c0c2f70 2024-02-19T15:04:36.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e5219a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e4c0c7560 con 0x7f9e4c0c5920 2024-02-19T15:04:36.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e5219a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c071a00 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9e4c0c5920 0x7f9e4c155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9e4c0c2f70 0x7f9e4c06c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42516/0 (socket says 172.21.15.78:42516) 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4affd640 1 -- 172.21.15.78:0/4140807500 learned_addr learned my addr 172.21.15.78:0/4140807500 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4b7fe640 1 --2- 172.21.15.78:0/4140807500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c6a50 0x7f9e4c155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4140807500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c6a50 msgr2=0x7f9e4c155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4bfff640 1 --2- 172.21.15.78:0/4140807500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c6a50 0x7f9e4c155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4140807500 >> v1:172.21.15.78:6789/0 conn(0x7f9e4c0c2f70 legacy=0x7f9e4c06c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4140807500 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c0c8030 con 0x7f9e4c0c5920 2024-02-19T15:04:36.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.018+0000 7f9e4a7fc640 1 -- 172.21.15.78:0/4140807500 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173101068 0 0) 0x7f9e4c0717b0 con 0x7f9e4c0c2f70 2024-02-19T15:04:36.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.019+0000 7f9e4bfff640 1 --2- 172.21.15.78:0/4140807500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c155e80 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009d90 tx=0x7f9e4402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fdd26106092e99 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.019+0000 7f9e4a7fc640 1 -- 172.21.15.78:0/4140807500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4402cb30 con 0x7f9e4c0c5920 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.019+0000 7f9e4a7fc640 1 -- 172.21.15.78:0/4140807500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4402cc90 con 0x7f9e4c0c5920 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.019+0000 7f9e4a7fc640 1 -- 172.21.15.78:0/4140807500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44034440 con 0x7f9e4c0c5920 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 -- 172.21.15.78:0/4140807500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c5920 msgr2=0x7f9e4c155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4140807500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c155e80 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009d90 tx=0x7f9e4402bd80 comp rx=0 tx=0).stop 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 -- 172.21.15.78:0/4140807500 shutdown_connections 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4140807500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c155e80 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4140807500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c6a50 0x7f9e4c155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 -- 172.21.15.78:0/4140807500 >> 172.21.15.78:0/4140807500 conn(0x7f9e4c06b700 msgr2=0x7f9e4c0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 -- 172.21.15.78:0/4140807500 shutdown_connections 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.020+0000 7f9e5219a640 1 -- 172.21.15.78:0/4140807500 wait complete. 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.021+0000 7f9e5219a640 1 Processor -- start 2024-02-19T15:04:36.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.021+0000 7f9e5219a640 1 -- start start 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c2f70 0x7f9e4c0a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c1eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 0x7f9e4c1ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c0a8c50 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c156530 con 0x7f9e4c0c5920 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c156810 con 0x7f9e4c0c2f70 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 0x7f9e4c1ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 0x7f9e4c1ea6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33672/0 (socket says 172.21.15.78:33672) 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4023206488 learned_addr learned my addr 172.21.15.78:0/4023206488 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4b7fe640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c2f70 0x7f9e4c0a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4affd640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c1eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c2f70 msgr2=0x7f9e4c0a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c2f70 0x7f9e4c0a96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4c0c5920 msgr2=0x7f9e4c1eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c1eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 -- 172.21.15.78:0/4023206488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e44009a70 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e4bfff640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 0x7f9e4c1ea6e0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009d90 tx=0x7f9e44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44034e40 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e44034440 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c0a8ea0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e440408a0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.022+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c0aa6c0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.023+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c0a93c0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.024+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9e44040a00 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.024+0000 7f9e33fff640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e3806b810 0x7f9e3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.024+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9e440acec0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.024+0000 7f9e4b7fe640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e3806b810 0x7f9e3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.025+0000 7f9e4b7fe640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e3806b810 0x7f9e3806dcd0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009b40 tx=0x7f9e3c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.029+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9e4c0a93c0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.198+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e4c0a93c0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.198+0000 7f9e33fff640 1 -- 172.21.15.78:0/4023206488 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e4c0a93c0 con 0x7f9e4c0c6a50 2024-02-19T15:04:36.209 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-02-19T15:04:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e3806b810 msgr2=0x7f9e3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e3806b810 0x7f9e3806dcd0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009b40 tx=0x7f9e3c009190 comp rx=0 tx=0).stop 2024-02-19T15:04:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 msgr2=0x7f9e4c1ea6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 0x7f9e4c1ea6e0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009d90 tx=0x7f9e44004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 shutdown_connections 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9e3806b810 0x7f9e3806dcd0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e4c0c6a50 0x7f9e4c1ea6e0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e4c0c5920 0x7f9e4c1eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 --2- 172.21.15.78:0/4023206488 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e4c0c2f70 0x7f9e4c0a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 >> 172.21.15.78:0/4023206488 conn(0x7f9e4c06b700 msgr2=0x7f9e4c154670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 shutdown_connections 2024-02-19T15:04:36.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.200+0000 7f9e5219a640 1 -- 172.21.15.78:0/4023206488 wait complete. 2024-02-19T15:04:36.214 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-19T15:04:36.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.314+0000 7f885ce6b640 1 Processor -- start 2024-02-19T15:04:36.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.314+0000 7f885ce6b640 1 -- start start 2024-02-19T15:04:36.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f885ce6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88580c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f885ce6b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88581563b0 con 0x7f885814cd50 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f885ce6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8858153cd0 con 0x7f8858156860 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f885ce6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885806b6e0 con 0x7f88581537a0 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f885659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f885814cd50 0x7f88580c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42526/0 (socket says 172.21.15.78:42526) 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f885659c640 1 -- 172.21.15.78:0/3052155200 learned_addr learned my addr 172.21.15.78:0/3052155200 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.315+0000 7f884ffff640 1 --2- 172.21.15.78:0/3052155200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88580c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416218093 0 0) 0x7f88581563b0 con 0x7f885814cd50 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8844003520 con 0x7f885814cd50 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970327449 0 0) 0x7f8858153cd0 con 0x7f8858156860 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88581563b0 con 0x7f8858156860 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91488288 0 0) 0x7f8844003520 con 0x7f885814cd50 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f884ffff640 1 -- 172.21.15.78:0/3052155200 >> v1:172.21.15.150:6789/0 conn(0x7f8858156860 legacy=0x7f8858156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f884ffff640 1 -- 172.21.15.78:0/3052155200 >> v1:172.21.15.78:6789/0 conn(0x7f885814cd50 legacy=0x7f88580c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.316+0000 7f884ffff640 1 -- 172.21.15.78:0/3052155200 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885806b980 con 0x7f88581537a0 2024-02-19T15:04:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f884ffff640 1 --2- 172.21.15.78:0/3052155200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88580c4cd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f88400097d0 tx=0x7f884002b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40a98d9974ad4f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848001670 con 0x7f88581537a0 2024-02-19T15:04:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884002e050 con 0x7f88581537a0 2024-02-19T15:04:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f8855d9b640 1 -- 172.21.15.78:0/3052155200 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840030730 con 0x7f88581537a0 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 -- 172.21.15.78:0/3052155200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 msgr2=0x7f88580c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 --2- 172.21.15.78:0/3052155200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88580c4cd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f88400097d0 tx=0x7f884002b6e0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 -- 172.21.15.78:0/3052155200 shutdown_connections 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 --2- 172.21.15.78:0/3052155200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88580c4cd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 -- 172.21.15.78:0/3052155200 >> 172.21.15.78:0/3052155200 conn(0x7f8858066e70 msgr2=0x7f88580a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 -- 172.21.15.78:0/3052155200 shutdown_connections 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.317+0000 7f885ce6b640 1 -- 172.21.15.78:0/3052155200 wait complete. 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 Processor -- start 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 -- start start 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 0x7f88581510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88581dfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8858156860 0x7f88581edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88580ac100 con 0x7f885814cd50 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8858150c30 con 0x7f8858156860 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885ce6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88580ab1e0 con 0x7f88581537a0 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 0x7f88581510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 0x7f88581510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33686/0 (socket says 172.21.15.78:33686) 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f885659c640 1 -- 172.21.15.78:0/2042613969 learned_addr learned my addr 172.21.15.78:0/2042613969 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f884ffff640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88581dfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.319+0000 7f8856d9d640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8858156860 0x7f88581edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885659c640 1 -- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 msgr2=0x7f88581dfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885659c640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88581dfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885659c640 1 -- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8858156860 msgr2=0x7f88581edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885659c640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8858156860 0x7f88581edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885659c640 1 -- 172.21.15.78:0/2042613969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8840009400 con 0x7f885814cd50 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885659c640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 0x7f88581510c0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f884800c700 tx=0x7f884800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848005ad0 con 0x7f885814cd50 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8848005c30 con 0x7f885814cd50 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848007070 con 0x7f885814cd50 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580ab4c0 con 0x7f885814cd50 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.320+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88580ab7a0 con 0x7f885814cd50 2024-02-19T15:04:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.321+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8858074990 con 0x7f885814cd50 2024-02-19T15:04:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.322+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f88480071d0 con 0x7f885814cd50 2024-02-19T15:04:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.322+0000 7f884effd640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f883806b810 0x7f883806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.322+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f88480899e0 con 0x7f885814cd50 2024-02-19T15:04:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.322+0000 7f884ffff640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f883806b810 0x7f883806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.322+0000 7f884ffff640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f883806b810 0x7f883806dcd0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8840034000 tx=0x7f8840008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.326+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f88480542b0 con 0x7f885814cd50 2024-02-19T15:04:36.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.498+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88580c4230 con 0x7f885814cd50 2024-02-19T15:04:36.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.498+0000 7f884effd640 1 -- 172.21.15.78:0/2042613969 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v217) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8848057970 con 0x7f885814cd50 2024-02-19T15:04:36.501 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f883806b810 msgr2=0x7f883806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f883806b810 0x7f883806dcd0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8840034000 tx=0x7f8840008030 comp rx=0 tx=0).stop 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 msgr2=0x7f88581510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 0x7f88581510c0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f884800c700 tx=0x7f884800cbc0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 shutdown_connections 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f883806b810 0x7f883806dcd0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8858156860 0x7f88581edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88581537a0 0x7f88581dfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 --2- 172.21.15.78:0/2042613969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f885814cd50 0x7f88581510c0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 >> 172.21.15.78:0/2042613969 conn(0x7f8858066e70 msgr2=0x7f8858156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 shutdown_connections 2024-02-19T15:04:36.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.500+0000 7f885ce6b640 1 -- 172.21.15.78:0/2042613969 wait complete. 2024-02-19T15:04:36.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-19T15:04:36.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 Processor -- start 2024-02-19T15:04:36.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 -- start start 2024-02-19T15:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f574c1523a0 0x7f574c155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f574c0c5bc0 0x7f574c06bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0ca140 con 0x7f574c06c4b0 2024-02-19T15:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0c5a50 con 0x7f574c1523a0 2024-02-19T15:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5753223640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0c51b0 con 0x7f574c0c5bc0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5750fbf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f574c1523a0 0x7f574c155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5750fbf640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f574c1523a0 0x7f574c155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33696/0 (socket says 172.21.15.78:33696) 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.613+0000 7f5750fbf640 1 -- 172.21.15.78:0/4289658585 learned_addr learned my addr 172.21.15.78:0/4289658585 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.614+0000 7f574bfff640 1 --2- 172.21.15.78:0/4289658585 >> v2:172.21.15.150:3300/0 conn(0x7f574c0c5bc0 0x7f574c06bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.614+0000 7f5750fbf640 1 -- 172.21.15.78:0/4289658585 >> v2:172.21.15.150:3300/0 conn(0x7f574c0c5bc0 msgr2=0x7f574c06bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.614+0000 7f5750fbf640 1 --2- 172.21.15.78:0/4289658585 >> v2:172.21.15.150:3300/0 conn(0x7f574c0c5bc0 0x7f574c06bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.614+0000 7f5750fbf640 1 -- 172.21.15.78:0/4289658585 >> v1:172.21.15.150:6789/0 conn(0x7f574c06c4b0 legacy=0x7f574c0c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.614+0000 7f5750fbf640 1 -- 172.21.15.78:0/4289658585 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0c5490 con 0x7f574c1523a0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.614+0000 7f5750fbf640 1 --2- 172.21.15.78:0/4289658585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c1523a0 0x7f574c155950 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f574c0b09a0 tx=0x7f574402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3f094d75fe2836 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f574b7fe640 1 -- 172.21.15.78:0/4289658585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574402ca80 con 0x7f574c1523a0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f574b7fe640 1 -- 172.21.15.78:0/4289658585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574402cbe0 con 0x7f574c1523a0 2024-02-19T15:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f574b7fe640 1 -- 172.21.15.78:0/4289658585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57440343f0 con 0x7f574c1523a0 2024-02-19T15:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 -- 172.21.15.78:0/4289658585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c1523a0 msgr2=0x7f574c155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 --2- 172.21.15.78:0/4289658585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c1523a0 0x7f574c155950 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f574c0b09a0 tx=0x7f574402bee0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 -- 172.21.15.78:0/4289658585 shutdown_connections 2024-02-19T15:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 --2- 172.21.15.78:0/4289658585 >> v2:172.21.15.150:3300/0 conn(0x7f574c0c5bc0 0x7f574c06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 --2- 172.21.15.78:0/4289658585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c1523a0 0x7f574c155950 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 -- 172.21.15.78:0/4289658585 >> 172.21.15.78:0/4289658585 conn(0x7f574c0ac5a0 msgr2=0x7f574c0ac9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 -- 172.21.15.78:0/4289658585 shutdown_connections 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.615+0000 7f5753223640 1 -- 172.21.15.78:0/4289658585 wait complete. 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.616+0000 7f5753223640 1 Processor -- start 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.616+0000 7f5753223640 1 -- start start 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5753223640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 0x7f574c1eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5753223640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f574c0c5bc0 0x7f574c1ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5753223640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574c1523a0 0x7f574c0c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5753223640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c14c840 con 0x7f574c06c4b0 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5753223640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c159330 con 0x7f574c0c5bc0 2024-02-19T15:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5753223640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c158da0 con 0x7f574c1523a0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 0x7f574c1eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 0x7f574c1eb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33710/0 (socket says 172.21.15.78:33710) 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 -- 172.21.15.78:0/2487540823 learned_addr learned my addr 172.21.15.78:0/2487540823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f574bfff640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f574c0c5bc0 0x7f574c1ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f57517c0640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574c1523a0 0x7f574c0c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 -- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574c1523a0 msgr2=0x7f574c0c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574c1523a0 0x7f574c0c3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 -- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f574c0c5bc0 msgr2=0x7f574c1ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f574c0c5bc0 0x7f574c1ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.617+0000 7f5750fbf640 1 -- 172.21.15.78:0/2487540823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744009b30 con 0x7f574c06c4b0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.618+0000 7f5750fbf640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 0x7f574c1eb9c0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f5744036040 tx=0x7f5744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.618+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744009510 con 0x7f574c06c4b0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.618+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574403e450 con 0x7f574c06c4b0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.618+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c159050 con 0x7f574c06c4b0 2024-02-19T15:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.618+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574403e800 con 0x7f574c06c4b0 2024-02-19T15:04:36.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.618+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c1570c0 con 0x7f574c06c4b0 2024-02-19T15:04:36.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.619+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c074990 con 0x7f574c06c4b0 2024-02-19T15:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.619+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f574402cd50 con 0x7f574c06c4b0 2024-02-19T15:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.619+0000 7f57497fa640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b840 0x7f572c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.620+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f57440ace70 con 0x7f574c06c4b0 2024-02-19T15:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.620+0000 7f574bfff640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b840 0x7f572c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.620+0000 7f574bfff640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b840 0x7f572c06dd00 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f574c154740 tx=0x7f573c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.624+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57440777e0 con 0x7f574c06c4b0 2024-02-19T15:04:36.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.794+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f574c0ca3a0 con 0x7f574c06c4b0 2024-02-19T15:04:36.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.795+0000 7f57497fa640 1 -- 172.21.15.78:0/2487540823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v217) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5744043020 con 0x7f574c06c4b0 2024-02-19T15:04:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.796+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b840 msgr2=0x7f572c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.796+0000 7f5753223640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b840 0x7f572c06dd00 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f574c154740 tx=0x7f573c009190 comp rx=0 tx=0).stop 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.796+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 msgr2=0x7f574c1eb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.796+0000 7f5753223640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 0x7f574c1eb9c0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f5744036040 tx=0x7f5744004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 shutdown_connections 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f572c06b840 0x7f572c06dd00 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574c1523a0 0x7f574c0c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f574c0c5bc0 0x7f574c1ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 --2- 172.21.15.78:0/2487540823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f574c06c4b0 0x7f574c1eb9c0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 >> 172.21.15.78:0/2487540823 conn(0x7f574c0ac5a0 msgr2=0x7f574c0c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 shutdown_connections 2024-02-19T15:04:36.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.797+0000 7f5753223640 1 -- 172.21.15.78:0/2487540823 wait complete. 2024-02-19T15:04:36.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.906+0000 7f2ae8623640 1 Processor -- start 2024-02-19T15:04:36.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.906+0000 7f2ae8623640 1 -- start start 2024-02-19T15:04:36.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae8623640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae00c4d80 0x7f2ae0159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae8623640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ae0155010 0x7f2ae01523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae8623640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae006ffa0 con 0x7f2ae00c78a0 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae8623640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae006b720 con 0x7f2ae0155010 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae8623640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae006fc70 con 0x7f2ae00c4d80 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae6bc0640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ae0155010 0x7f2ae01523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae63bf640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2ae00c78a0 0x7f2ae00c43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42534/0 (socket says 172.21.15.78:42534) 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.907+0000 7f2ae63bf640 1 -- 172.21.15.78:0/1309377745 learned_addr learned my addr 172.21.15.78:0/1309377745 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae5bbe640 1 --2- 172.21.15.78:0/1309377745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae00c4d80 0x7f2ae0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae6bc0640 1 -- 172.21.15.78:0/1309377745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae00c4d80 msgr2=0x7f2ae0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae6bc0640 1 --2- 172.21.15.78:0/1309377745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae00c4d80 0x7f2ae0159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae6bc0640 1 -- 172.21.15.78:0/1309377745 >> v1:172.21.15.78:6789/0 conn(0x7f2ae00c78a0 legacy=0x7f2ae00c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae6bc0640 1 -- 172.21.15.78:0/1309377745 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae006b930 con 0x7f2ae0155010 2024-02-19T15:04:36.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae53bd640 1 -- 172.21.15.78:0/1309377745 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579503675 0 0) 0x7f2ae006ffa0 con 0x7f2ae00c78a0 2024-02-19T15:04:36.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.908+0000 7f2ae6bc0640 1 --2- 172.21.15.78:0/1309377745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae0155010 0x7f2ae01523d0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009d90 tx=0x7f2ad802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c6dd4a2cbbe1c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae53bd640 1 -- 172.21.15.78:0/1309377745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad802cbc0 con 0x7f2ae0155010 2024-02-19T15:04:36.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae53bd640 1 -- 172.21.15.78:0/1309377745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad802cd20 con 0x7f2ae0155010 2024-02-19T15:04:36.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae53bd640 1 -- 172.21.15.78:0/1309377745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8034660 con 0x7f2ae0155010 2024-02-19T15:04:36.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 -- 172.21.15.78:0/1309377745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae0155010 msgr2=0x7f2ae01523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 --2- 172.21.15.78:0/1309377745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae0155010 0x7f2ae01523d0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009d90 tx=0x7f2ad802be10 comp rx=0 tx=0).stop 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 -- 172.21.15.78:0/1309377745 shutdown_connections 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 --2- 172.21.15.78:0/1309377745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae0155010 0x7f2ae01523d0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 --2- 172.21.15.78:0/1309377745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae00c4d80 0x7f2ae0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 -- 172.21.15.78:0/1309377745 >> 172.21.15.78:0/1309377745 conn(0x7f2ae0066e70 msgr2=0x7f2ae00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 -- 172.21.15.78:0/1309377745 shutdown_connections 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.909+0000 7f2ae8623640 1 -- 172.21.15.78:0/1309377745 wait complete. 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.910+0000 7f2ae8623640 1 Processor -- start 2024-02-19T15:04:36.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.910+0000 7f2ae8623640 1 -- start start 2024-02-19T15:04:36.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae8623640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 0x7f2ae0157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae8623640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae00c78a0 0x7f2ae0152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae8623640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae0155010 0x7f2ae00c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae8623640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0154110 con 0x7f2ae00c4d80 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae8623640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0150f20 con 0x7f2ae00c78a0 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae8623640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0151200 con 0x7f2ae0155010 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 0x7f2ae0157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 0x7f2ae0157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33734/0 (socket says 172.21.15.78:33734) 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 -- 172.21.15.78:0/2903610769 learned_addr learned my addr 172.21.15.78:0/2903610769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae6bc0640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae0155010 0x7f2ae00c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae5bbe640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae00c78a0 0x7f2ae0152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 -- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae0155010 msgr2=0x7f2ae00c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae0155010 0x7f2ae00c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 -- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae00c78a0 msgr2=0x7f2ae0152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae00c78a0 0x7f2ae0152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.911+0000 7f2ae63bf640 1 -- 172.21.15.78:0/2903610769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8009a70 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.912+0000 7f2ae63bf640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 0x7f2ae0157bf0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000c430 tx=0x7f2ad000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.912+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad00074e0 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.912+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad0007660 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.912+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae01514e0 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.912+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0004a20 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.912+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae01edcb0 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.913+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae0074990 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.913+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2ad0004b80 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.914+0000 7f2ad6ffd640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ac406b840 0x7f2ac406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.914+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2ad008a050 con 0x7f2ae00c4d80 2024-02-19T15:04:36.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.914+0000 7f2ae5bbe640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ac406b840 0x7f2ac406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:36.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.914+0000 7f2ae5bbe640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ac406b840 0x7f2ac406dd00 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2ae014cde0 tx=0x7f2adc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:36.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:36.918+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ad0054920 con 0x7f2ae00c4d80 2024-02-19T15:04:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.124+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ae0072a30 con 0x7f2ae00c4d80 2024-02-19T15:04:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.125+0000 7f2ad6ffd640 1 -- 172.21.15.78:0/2903610769 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7f2ad0057fe0 con 0x7f2ae00c4d80 2024-02-19T15:04:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:37.128 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:37.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ac406b840 msgr2=0x7f2ac406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ac406b840 0x7f2ac406dd00 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2ae014cde0 tx=0x7f2adc009190 comp rx=0 tx=0).stop 2024-02-19T15:04:37.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 msgr2=0x7f2ae0157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 0x7f2ae0157bf0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000c430 tx=0x7f2ad000c8f0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 shutdown_connections 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2ac406b840 0x7f2ac406dd00 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ae0155010 0x7f2ae00c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ae00c78a0 0x7f2ae0152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 --2- 172.21.15.78:0/2903610769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ae00c4d80 0x7f2ae0157bf0 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 >> 172.21.15.78:0/2903610769 conn(0x7f2ae0066e70 msgr2=0x7f2ae00c0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 shutdown_connections 2024-02-19T15:04:37.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.127+0000 7f2ae8623640 1 -- 172.21.15.78:0/2903610769 wait complete. 2024-02-19T15:04:37.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:04:37.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.238+0000 7f089981c640 1 Processor -- start 2024-02-19T15:04:37.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.238+0000 7f089981c640 1 -- start start 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f089981c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0894158f80 0x7f08940c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f089981c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08940c3cf0 con 0x7f08940c5130 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f089981c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0894159410 con 0x7f08940c30a0 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f089981c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0894158b60 con 0x7f0894158f80 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f0892ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f08940c5130 0x7f08940c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42542/0 (socket says 172.21.15.78:42542) 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f0892ffd640 1 -- 172.21.15.78:0/2197748216 learned_addr learned my addr 172.21.15.78:0/2197748216 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f08937fe640 1 --2- 172.21.15.78:0/2197748216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0894158f80 0x7f08940c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.239+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317053167 0 0) 0x7f08940c3cf0 con 0x7f08940c5130 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c003520 con 0x7f08940c5130 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897773394 0 0) 0x7f0894159410 con 0x7f08940c30a0 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08940c3cf0 con 0x7f08940c30a0 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f08937fe640 1 -- 172.21.15.78:0/2197748216 >> v1:172.21.15.150:6789/0 conn(0x7f08940c30a0 legacy=0x7f089415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f08937fe640 1 -- 172.21.15.78:0/2197748216 >> v1:172.21.15.78:6789/0 conn(0x7f08940c5130 legacy=0x7f08940c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f08937fe640 1 -- 172.21.15.78:0/2197748216 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0894158530 con 0x7f0894158f80 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.240+0000 7f08937fe640 1 --2- 172.21.15.78:0/2197748216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0894158f80 0x7f08940c5510 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097d0 tx=0x7f088c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2571f17821ab701 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c031890 con 0x7f0894158f80 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c0319f0 con 0x7f0894158f80 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f0891ffb640 1 -- 172.21.15.78:0/2197748216 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c031d80 con 0x7f0894158f80 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 -- 172.21.15.78:0/2197748216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0894158f80 msgr2=0x7f08940c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 --2- 172.21.15.78:0/2197748216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0894158f80 0x7f08940c5510 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097d0 tx=0x7f088c02b930 comp rx=0 tx=0).stop 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 -- 172.21.15.78:0/2197748216 shutdown_connections 2024-02-19T15:04:37.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 --2- 172.21.15.78:0/2197748216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0894158f80 0x7f08940c5510 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 -- 172.21.15.78:0/2197748216 >> 172.21.15.78:0/2197748216 conn(0x7f089406b4b0 msgr2=0x7f08940c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:37.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 -- 172.21.15.78:0/2197748216 shutdown_connections 2024-02-19T15:04:37.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.241+0000 7f089981c640 1 -- 172.21.15.78:0/2197748216 wait complete. 2024-02-19T15:04:37.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.242+0000 7f089981c640 1 Processor -- start 2024-02-19T15:04:37.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.242+0000 7f089981c640 1 -- start start 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f089981c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08940c30a0 0x7f08940ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f089981c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08940c5130 0x7f0894148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f089981c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 0x7f08940a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f089981c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941550c0 con 0x7f0894158f80 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f089981c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941eeee0 con 0x7f08940c30a0 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f089981c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0894155380 con 0x7f08940c5130 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 0x7f08940a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 0x7f08940a8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33750/0 (socket says 172.21.15.78:33750) 2024-02-19T15:04:37.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 -- 172.21.15.78:0/2787833770 learned_addr learned my addr 172.21.15.78:0/2787833770 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f0892ffd640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08940c30a0 0x7f08940ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08927fc640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08940c5130 0x7f0894148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 -- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08940c5130 msgr2=0x7f0894148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08940c5130 0x7f0894148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 -- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08940c30a0 msgr2=0x7f08940ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08940c30a0 0x7f08940ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.243+0000 7f08937fe640 1 -- 172.21.15.78:0/2787833770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c009400 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.244+0000 7f08937fe640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 0x7f08940a8a20 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f088c02b970 tx=0x7f088c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.244+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c03aae0 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.244+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c03ac60 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.244+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0443f0 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.244+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08940a8500 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.244+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08941eeab0 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.245+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08941eeca0 con 0x7f0894158f80 2024-02-19T15:04:37.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.245+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f088c04d450 con 0x7f0894158f80 2024-02-19T15:04:37.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.246+0000 7f087bfff640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f088006b810 0x7f088006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.246+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f088c0ac860 con 0x7f0894158f80 2024-02-19T15:04:37.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.246+0000 7f0892ffd640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f088006b810 0x7f088006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.246+0000 7f0892ffd640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f088006b810 0x7f088006dcd0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f08840012e0 tx=0x7f0884001ad0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.250+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f08941eeca0 con 0x7f0894158f80 2024-02-19T15:04:37.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.422+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08941eeca0 con 0x7f0894158f80 2024-02-19T15:04:37.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.424+0000 7f087bfff640 1 -- 172.21.15.78:0/2787833770 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+17273 (secure 0 0 0) 0x7f08941eeca0 con 0x7f0894158f80 2024-02-19T15:04:37.427 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:37.427 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":217,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:02:05.637768+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:00:46.861039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-19T15:00:47.624123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.2","mappings":[{"from":0,"to":2}]},{"pgid":"34.b","mappings":[{"from":0,"to":1}]},{"pgid":"34.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6835/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6834/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6837/159409097":"2024-02-20T15:01:30.632749+0000","172.21.15.78:6836/159409097":"2024-02-20T15:01:30.632749+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f088006b810 msgr2=0x7f088006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f088006b810 0x7f088006dcd0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f08840012e0 tx=0x7f0884001ad0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 msgr2=0x7f08940a8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 0x7f08940a8a20 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f088c02b970 tx=0x7f088c0046c0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 shutdown_connections 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f088006b810 0x7f088006dcd0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0894158f80 0x7f08940a8a20 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08940c5130 0x7f0894148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 --2- 172.21.15.78:0/2787833770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08940c30a0 0x7f08940ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 >> 172.21.15.78:0/2787833770 conn(0x7f089406b4b0 msgr2=0x7f08940c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 shutdown_connections 2024-02-19T15:04:37.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.426+0000 7f089981c640 1 -- 172.21.15.78:0/2787833770 wait complete. 2024-02-19T15:04:37.440 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:04:37.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.537+0000 7fb4cfde4640 1 Processor -- start 2024-02-19T15:04:37.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.537+0000 7fb4cfde4640 1 -- start start 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cfde4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb4c80c8500 0x7fb4c80c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cfde4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4c80c42e0 0x7fb4c8158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cfde4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80c34a0 con 0x7fb4c80c2f50 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cfde4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c806c090 con 0x7fb4c80c8500 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cfde4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c806d6a0 con 0x7fb4c80c42e0 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cdb80640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb4c80c8500 0x7fb4c80c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4ce381640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb4c80c2f50 0x7fb4c8158b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50596/0 (socket says 172.21.15.78:50596) 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4ce381640 1 -- 172.21.15.78:0/2262242272 learned_addr learned my addr 172.21.15.78:0/2262242272 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cd37f640 1 --2- 172.21.15.78:0/2262242272 >> v2:172.21.15.150:3300/0 conn(0x7fb4c80c42e0 0x7fb4c8158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cdb80640 1 -- 172.21.15.78:0/2262242272 >> v2:172.21.15.150:3300/0 conn(0x7fb4c80c42e0 msgr2=0x7fb4c8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:37.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cdb80640 1 --2- 172.21.15.78:0/2262242272 >> v2:172.21.15.150:3300/0 conn(0x7fb4c80c42e0 0x7fb4c8158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cdb80640 1 -- 172.21.15.78:0/2262242272 >> v1:172.21.15.78:6789/0 conn(0x7fb4c80c2f50 legacy=0x7fb4c8158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4cdb80640 1 -- 172.21.15.78:0/2262242272 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c806c8e0 con 0x7fb4c80c8500 2024-02-19T15:04:37.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.538+0000 7fb4ccb7e640 1 -- 172.21.15.78:0/2262242272 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112562969 0 0) 0x7fb4c80c34a0 con 0x7fb4c80c2f50 2024-02-19T15:04:37.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cdb80640 1 --2- 172.21.15.78:0/2262242272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c80c88e0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80c9320 tx=0x7fb4c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb13fa1aacc59e6d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4ccb7e640 1 -- 172.21.15.78:0/2262242272 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c402e070 con 0x7fb4c80c8500 2024-02-19T15:04:37.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4ccb7e640 1 -- 172.21.15.78:0/2262242272 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c40315a0 con 0x7fb4c80c8500 2024-02-19T15:04:37.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4ccb7e640 1 -- 172.21.15.78:0/2262242272 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4031930 con 0x7fb4c80c8500 2024-02-19T15:04:37.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2262242272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 msgr2=0x7fb4c80c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2262242272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c80c88e0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80c9320 tx=0x7fb4c402b840 comp rx=0 tx=0).stop 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2262242272 shutdown_connections 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2262242272 >> v2:172.21.15.150:3300/0 conn(0x7fb4c80c42e0 0x7fb4c8158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2262242272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c80c88e0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.539+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2262242272 >> 172.21.15.78:0/2262242272 conn(0x7fb4c80aa300 msgr2=0x7fb4c80aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.540+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2262242272 shutdown_connections 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.540+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2262242272 wait complete. 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 Processor -- start 2024-02-19T15:04:37.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 -- start start 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4c80c2f50 0x7fb4c814b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c8070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c8070e90 0x7fb4c81eb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81eeef0 con 0x7fb4c80c8500 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81ee280 con 0x7fb4c8070e90 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cfde4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81ef3e0 con 0x7fb4c80c2f50 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cd37f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c8070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cd37f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c8070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46262/0 (socket says 172.21.15.78:46262) 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cd37f640 1 -- 172.21.15.78:0/2805844915 learned_addr learned my addr 172.21.15.78:0/2805844915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4cdb80640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4c80c2f50 0x7fb4c814b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.541+0000 7fb4ce381640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c8070e90 0x7fb4c81eb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cd37f640 1 -- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4c80c2f50 msgr2=0x7fb4c814b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cd37f640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4c80c2f50 0x7fb4c814b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cd37f640 1 -- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c8070e90 msgr2=0x7fb4c81eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cd37f640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c8070e90 0x7fb4c81eb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cd37f640 1 -- 172.21.15.78:0/2805844915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c4009400 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cd37f640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c8070950 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800e9b0 tx=0x7fb4b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8009670 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b800c9a0 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c8158400 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800cd50 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.542+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c81ef100 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.543+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c8071c20 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.543+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb4b8013470 con 0x7fb4c80c8500 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.544+0000 7fb4be7fc640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4ac06b840 0x7fb4ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.544+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb4b808a540 con 0x7fb4c80c8500 2024-02-19T15:04:37.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.544+0000 7fb4cdb80640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4ac06b840 0x7fb4ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.544+0000 7fb4cdb80640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4ac06b840 0x7fb4ac06dd00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4036000 tx=0x7fb4c403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.548+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4b8054e10 con 0x7fb4c80c8500 2024-02-19T15:04:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.754+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c8074990 con 0x7fb4c80c8500 2024-02-19T15:04:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.754+0000 7fb4be7fc640 1 -- 172.21.15.78:0/2805844915 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7fb4b8017d90 con 0x7fb4c80c8500 2024-02-19T15:04:37.757 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:37.757 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4ac06b840 msgr2=0x7fb4ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4ac06b840 0x7fb4ac06dd00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4036000 tx=0x7fb4c403d040 comp rx=0 tx=0).stop 2024-02-19T15:04:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 msgr2=0x7fb4c8070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c8070950 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800e9b0 tx=0x7fb4b800ee70 comp rx=0 tx=0).stop 2024-02-19T15:04:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 shutdown_connections 2024-02-19T15:04:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb4ac06b840 0x7fb4ac06dd00 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4c80c8500 0x7fb4c8070950 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c8070e90 0x7fb4c81eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 --2- 172.21.15.78:0/2805844915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4c80c2f50 0x7fb4c814b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 >> 172.21.15.78:0/2805844915 conn(0x7fb4c80aa300 msgr2=0x7fb4c8153970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 shutdown_connections 2024-02-19T15:04:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.756+0000 7fb4cfde4640 1 -- 172.21.15.78:0/2805844915 wait complete. 2024-02-19T15:04:37.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:04:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 Processor -- start 2024-02-19T15:04:37.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 -- start start 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b00c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0b0154fd0 0x7ff0b006f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b006b470 con 0x7ff0b01526b0 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b0152a90 con 0x7ff0b0154fd0 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.867+0000 7ff0b4e47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b006b9a0 con 0x7ff0b006f490 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0a7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b00c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0ae59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff0b01526b0 0x7ff0b0158ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42864/0 (socket says 172.21.15.78:42864) 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0ae59c640 1 -- 172.21.15.78:0/1947925011 learned_addr learned my addr 172.21.15.78:0/1947925011 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0aed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0b0154fd0 0x7ff0b006f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0aed9d640 1 -- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 msgr2=0x7ff0b00c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b00c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0aed9d640 1 -- 172.21.15.78:0/1947925011 >> v1:172.21.15.150:6789/0 conn(0x7ff0b01526b0 legacy=0x7ff0b0158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0add9b640 1 -- 172.21.15.78:0/1947925011 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206264473 0 0) 0x7ff0b006b470 con 0x7ff0b01526b0 2024-02-19T15:04:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0aed9d640 1 -- 172.21.15.78:0/1947925011 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0155460 con 0x7ff0b0154fd0 2024-02-19T15:04:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.868+0000 7ff0a7fff640 1 --2- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b00c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b006f9e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009d90 tx=0x7ff0a802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=340e32954ffdd869 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0add9b640 1 -- 172.21.15.78:0/1947925011 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a802cb30 con 0x7ff0b0154fd0 2024-02-19T15:04:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0add9b640 1 -- 172.21.15.78:0/1947925011 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a802cc90 con 0x7ff0b0154fd0 2024-02-19T15:04:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0add9b640 1 -- 172.21.15.78:0/1947925011 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8034440 con 0x7ff0b0154fd0 2024-02-19T15:04:37.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 -- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 msgr2=0x7ff0b006f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b006f9e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009d90 tx=0x7ff0a802bd80 comp rx=0 tx=0).stop 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 -- 172.21.15.78:0/1947925011 shutdown_connections 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b00c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/1947925011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b006f9e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 -- 172.21.15.78:0/1947925011 >> 172.21.15.78:0/1947925011 conn(0x7ff0b006ea10 msgr2=0x7ff0b00a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 -- 172.21.15.78:0/1947925011 shutdown_connections 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.869+0000 7ff0b4e47640 1 -- 172.21.15.78:0/1947925011 wait complete. 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 Processor -- start 2024-02-19T15:04:37.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 -- start start 2024-02-19T15:04:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b01eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 0x7ff0b0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b0141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b01edb90 con 0x7ff0b01526b0 2024-02-19T15:04:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b01eb550 con 0x7ff0b0154fd0 2024-02-19T15:04:37.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0b4e47640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00aad20 con 0x7ff0b006f490 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0a7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 0x7ff0b0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0a7fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 0x7ff0b0072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46264/0 (socket says 172.21.15.78:46264) 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0a7fff640 1 -- 172.21.15.78:0/604737154 learned_addr learned my addr 172.21.15.78:0/604737154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0ae59c640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b01eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.871+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b0141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a7fff640 1 -- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 msgr2=0x7ff0b01eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a7fff640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b01eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a7fff640 1 -- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 msgr2=0x7ff0b0141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a7fff640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b0141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a7fff640 1 -- 172.21.15.78:0/604737154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff098009400 con 0x7ff0b01526b0 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0ae59c640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b01eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0aed9d640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b0141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a7fff640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 0x7ff0b0072e40 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7ff09800ebe0 tx=0x7ff09800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098009760 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09800cc30 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09800eee0 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a8009a70 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.872+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b00a9c40 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.873+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b0074990 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.873+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff098013470 con 0x7ff0b01526b0 2024-02-19T15:04:37.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.874+0000 7ff0a6ffd640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff09006b810 0x7ff09006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:37.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.874+0000 7ff0ae59c640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff09006b810 0x7ff09006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:37.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.874+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff09808a600 con 0x7ff0b01526b0 2024-02-19T15:04:37.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.874+0000 7ff0ae59c640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff09006b810 0x7ff09006dcd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0000f60 tx=0x7ff0a0007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:37.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:37.878+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff098054ed0 con 0x7ff0b01526b0 2024-02-19T15:04:38.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.085+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0b00c7a20 con 0x7ff0b01526b0 2024-02-19T15:04:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.085+0000 7ff0a6ffd640 1 -- 172.21.15.78:0/604737154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3441 (secure 0 0 0) 0x7ff098017d90 con 0x7ff0b01526b0 2024-02-19T15:04:38.088 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:38.088 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T15:00:48.648504+0000","modified":"2024-02-19T15:02:40.643214+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":217,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":10731,"mds_1":9217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9217":{"gid":9217,"name":"b","rank":1,"incarnation":202,"state":"up:stopping","state_seq":25,"addr":"172.21.15.150:6834/4270793546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4270793546},{"type":"v1","addr":"172.21.15.150:6834","nonce":4270793546}]},"join_fscid":-1,"laggy_since":"2024-02-19T15:02:40.643214+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":188,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/1952320195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":1952320195},{"type":"v1","addr":"172.21.15.150:6835","nonce":1952320195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-19T15:04:38.089 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 208 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff09006b810 msgr2=0x7ff09006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff09006b810 0x7ff09006dcd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0000f60 tx=0x7ff0a0007d50 comp rx=0 tx=0).stop 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 msgr2=0x7ff0b0072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 0x7ff0b0072e40 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7ff09800ebe0 tx=0x7ff09800c520 comp rx=0 tx=0).stop 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 shutdown_connections 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff09006b810 0x7ff09006dcd0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0b01526b0 0x7ff0b0072e40 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0b0154fd0 0x7ff0b0141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 --2- 172.21.15.78:0/604737154 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0b006f490 0x7ff0b01eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 >> 172.21.15.78:0/604737154 conn(0x7ff0b006ea10 msgr2=0x7ff0b0074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 shutdown_connections 2024-02-19T15:04:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.087+0000 7ff0b4e47640 1 -- 172.21.15.78:0/604737154 wait complete. 2024-02-19T15:04:38.101 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:04:38.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.198+0000 7f7bd34b5640 1 Processor -- start 2024-02-19T15:04:38.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.199+0000 7f7bd34b5640 1 -- start start 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.199+0000 7f7bd34b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c29c0 0x7f7bcc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.199+0000 7f7bd34b5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7bcc0c5be0 0x7f7bcc06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.199+0000 7f7bd34b5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc072250 con 0x7f7bcc0c25e0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.199+0000 7f7bd34b5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc152c90 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.199+0000 7f7bd34b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc072480 con 0x7f7bcc0c29c0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd1251640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7bcc0c25e0 0x7f7bcc0c4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50606/0 (socket says 172.21.15.78:50606) 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd1251640 1 -- 172.21.15.78:0/2897317063 learned_addr learned my addr 172.21.15.78:0/2897317063 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd1a52640 1 --2- 172.21.15.78:0/2897317063 >> v2:172.21.15.78:3300/0 conn(0x7f7bcc0c5be0 0x7f7bcc06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd0a50640 1 --2- 172.21.15.78:0/2897317063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c29c0 0x7f7bcc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bc3fff640 1 -- 172.21.15.78:0/2897317063 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312283649 0 0) 0x7f7bcc072250 con 0x7f7bcc0c25e0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bc3fff640 1 -- 172.21.15.78:0/2897317063 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb8003520 con 0x7f7bcc0c25e0 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd0a50640 1 -- 172.21.15.78:0/2897317063 >> v2:172.21.15.78:3300/0 conn(0x7f7bcc0c5be0 msgr2=0x7f7bcc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:38.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd0a50640 1 --2- 172.21.15.78:0/2897317063 >> v2:172.21.15.78:3300/0 conn(0x7f7bcc0c5be0 0x7f7bcc06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd0a50640 1 -- 172.21.15.78:0/2897317063 >> v1:172.21.15.78:6789/0 conn(0x7f7bcc0c25e0 legacy=0x7f7bcc0c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd0a50640 1 -- 172.21.15.78:0/2897317063 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0c4b90 con 0x7f7bcc0c29c0 2024-02-19T15:04:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.200+0000 7f7bd1a52640 1 --2- 172.21.15.78:0/2897317063 >> v2:172.21.15.78:3300/0 conn(0x7f7bcc0c5be0 0x7f7bcc06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd0a50640 1 --2- 172.21.15.78:0/2897317063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c29c0 0x7f7bcc1526a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0097d0 tx=0x7f7bbc02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13540d1498fd9ae7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bc3fff640 1 -- 172.21.15.78:0/2897317063 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc031770 con 0x7f7bcc0c29c0 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bc3fff640 1 -- 172.21.15.78:0/2897317063 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc0318d0 con 0x7f7bcc0c29c0 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bc3fff640 1 -- 172.21.15.78:0/2897317063 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc031c60 con 0x7f7bcc0c29c0 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2897317063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c29c0 msgr2=0x7f7bcc1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2897317063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c29c0 0x7f7bcc1526a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0097d0 tx=0x7f7bbc02b810 comp rx=0 tx=0).stop 2024-02-19T15:04:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2897317063 shutdown_connections 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2897317063 >> v2:172.21.15.78:3300/0 conn(0x7f7bcc0c5be0 0x7f7bcc06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2897317063 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c29c0 0x7f7bcc1526a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.201+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2897317063 >> 172.21.15.78:0/2897317063 conn(0x7f7bcc06b270 msgr2=0x7f7bcc06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.202+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2897317063 shutdown_connections 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.202+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2897317063 wait complete. 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 Processor -- start 2024-02-19T15:04:38.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 -- start start 2024-02-19T15:04:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c25e0 0x7f7bcc0a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bcc0c29c0 0x7f7bcc0a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 0x7f7bcc14b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc0c3840 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc1476b0 con 0x7f7bcc0c29c0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd34b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc1e68c0 con 0x7f7bcc0c25e0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd1a52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 0x7f7bcc14b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd1a52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 0x7f7bcc14b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46286/0 (socket says 172.21.15.78:46286) 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd1a52640 1 -- 172.21.15.78:0/2911237187 learned_addr learned my addr 172.21.15.78:0/2911237187 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd1251640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c25e0 0x7f7bcc0a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.203+0000 7f7bd0a50640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bcc0c29c0 0x7f7bcc0a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd1a52640 1 -- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c25e0 msgr2=0x7f7bcc0a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd1a52640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c25e0 0x7f7bcc0a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd1a52640 1 -- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bcc0c29c0 msgr2=0x7f7bcc0a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd1a52640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bcc0c29c0 0x7f7bcc0a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd1a52640 1 -- 172.21.15.78:0/2911237187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8009400 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd1a52640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 0x7f7bcc14b9d0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800e930 tx=0x7f7bc800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8009700 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc800ca00 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc009400 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc800cd90 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.204+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc1e6df0 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.205+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc074990 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.205+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f7bc8013470 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.206+0000 7f7bc1ffb640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bb006b840 0x7f7bb006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.206+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7bc8089f60 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.206+0000 7f7bd1251640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bb006b840 0x7f7bb006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.206+0000 7f7bd1251640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bb006b840 0x7f7bb006dd00 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005c00 tx=0x7f7bc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.210+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7bc80548f0 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.383+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bcc14fce0 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.384+0000 7f7bc1ffb640 1 -- 172.21.15.78:0/2911237187 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+17273 (secure 0 0 0) 0x7f7bc8057fb0 con 0x7f7bcc0c5be0 2024-02-19T15:04:38.387 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:38.387 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":217,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:02:05.637768+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-19T15:00:46.861039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-19T15:00:47.624123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.2","mappings":[{"from":0,"to":2}]},{"pgid":"34.b","mappings":[{"from":0,"to":1}]},{"pgid":"34.26","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6835/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6834/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6837/159409097":"2024-02-20T15:01:30.632749+0000","172.21.15.78:6836/159409097":"2024-02-20T15:01:30.632749+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bb006b840 msgr2=0x7f7bb006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bb006b840 0x7f7bb006dd00 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4005c00 tx=0x7f7bc4009040 comp rx=0 tx=0).stop 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 msgr2=0x7f7bcc14b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 0x7f7bcc14b9d0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800e930 tx=0x7f7bc800edf0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 shutdown_connections 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f7bb006b840 0x7f7bb006dd00 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7bcc0c5be0 0x7f7bcc14b9d0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bcc0c29c0 0x7f7bcc0a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 --2- 172.21.15.78:0/2911237187 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bcc0c25e0 0x7f7bcc0a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 >> 172.21.15.78:0/2911237187 conn(0x7f7bcc06b270 msgr2=0x7f7bcc06d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 shutdown_connections 2024-02-19T15:04:38.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.386+0000 7f7bd34b5640 1 -- 172.21.15.78:0/2911237187 wait complete. 2024-02-19T15:04:38.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-19T15:04:38.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.497+0000 7f656e1a1640 1 Processor -- start 2024-02-19T15:04:38.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.497+0000 7f656e1a1640 1 -- start start 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f656e1a1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65680c78a0 0x7f6568159010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f656e1a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c4d80 0x7f65680c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f656e1a1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656806ffa0 con 0x7f65680c48f0 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f656e1a1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656806b720 con 0x7f65680c78a0 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f656e1a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656806fc70 con 0x7f65680c4d80 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f65677fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65680c78a0 0x7f6568159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f6567fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f65680c48f0 0x7f65681523d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50608/0 (socket says 172.21.15.78:50608) 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f65677fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f65680c78a0 0x7f6568159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46288/0 (socket says 172.21.15.78:46288) 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f6567fff640 1 -- 172.21.15.78:0/3344138974 learned_addr learned my addr 172.21.15.78:0/3344138974 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f6566ffd640 1 --2- 172.21.15.78:0/3344138974 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c4d80 0x7f65680c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f65677fe640 1 -- 172.21.15.78:0/3344138974 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c4d80 msgr2=0x7f65680c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f65677fe640 1 --2- 172.21.15.78:0/3344138974 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c4d80 0x7f65680c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.498+0000 7f65677fe640 1 -- 172.21.15.78:0/3344138974 >> v1:172.21.15.78:6789/0 conn(0x7f65680c48f0 legacy=0x7f65681523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.499+0000 7f65677fe640 1 -- 172.21.15.78:0/3344138974 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656806b930 con 0x7f65680c78a0 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.499+0000 7f65667fc640 1 -- 172.21.15.78:0/3344138974 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307658767 0 0) 0x7f656806ffa0 con 0x7f65680c48f0 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.499+0000 7f65677fe640 1 --2- 172.21.15.78:0/3344138974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c78a0 0x7f6568159010 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6558009d90 tx=0x7f655802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fd70479fb9729d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.499+0000 7f65667fc640 1 -- 172.21.15.78:0/3344138974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655802cbc0 con 0x7f65680c78a0 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.499+0000 7f65667fc640 1 -- 172.21.15.78:0/3344138974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655802cd20 con 0x7f65680c78a0 2024-02-19T15:04:38.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.499+0000 7f65667fc640 1 -- 172.21.15.78:0/3344138974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6558034660 con 0x7f65680c78a0 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 -- 172.21.15.78:0/3344138974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c78a0 msgr2=0x7f6568159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 --2- 172.21.15.78:0/3344138974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c78a0 0x7f6568159010 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6558009d90 tx=0x7f655802be10 comp rx=0 tx=0).stop 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 -- 172.21.15.78:0/3344138974 shutdown_connections 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 --2- 172.21.15.78:0/3344138974 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c4d80 0x7f65680c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 --2- 172.21.15.78:0/3344138974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c78a0 0x7f6568159010 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 -- 172.21.15.78:0/3344138974 >> 172.21.15.78:0/3344138974 conn(0x7f6568066e70 msgr2=0x7f65680a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 -- 172.21.15.78:0/3344138974 shutdown_connections 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.500+0000 7f656e1a1640 1 -- 172.21.15.78:0/3344138974 wait complete. 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.501+0000 7f656e1a1640 1 Processor -- start 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.501+0000 7f656e1a1640 1 -- start start 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f656e1a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c48f0 0x7f6568158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f656e1a1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 0x7f65681ed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f656e1a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65680c78a0 0x7f65681e7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f656e1a1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6568156b60 con 0x7f65680c4d80 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f656e1a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65680c4010 con 0x7f65680c78a0 2024-02-19T15:04:38.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f656e1a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65680c3d30 con 0x7f65680c48f0 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 0x7f65681ed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 0x7f65681ed8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46292/0 (socket says 172.21.15.78:46292) 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 -- 172.21.15.78:0/1342178553 learned_addr learned my addr 172.21.15.78:0/1342178553 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f65677fe640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c48f0 0x7f6568158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6567fff640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65680c78a0 0x7f65681e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 -- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c48f0 msgr2=0x7f6568158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c48f0 0x7f6568158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 -- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65680c78a0 msgr2=0x7f65681e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65680c78a0 0x7f65681e7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.502+0000 7f6566ffd640 1 -- 172.21.15.78:0/1342178553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558009a70 con 0x7f65680c4d80 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.503+0000 7f6566ffd640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 0x7f65681ed8b0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f655c00e9b0 tx=0x7f655c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.503+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c009670 con 0x7f65680c4d80 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.503+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c00c9a0 con 0x7f65680c4d80 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.503+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6568157e30 con 0x7f65680c4d80 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.503+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c00cd50 con 0x7f65680c4d80 2024-02-19T15:04:38.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.503+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65680ab1f0 con 0x7f65680c4d80 2024-02-19T15:04:38.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.504+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65680ab410 con 0x7f65680c4d80 2024-02-19T15:04:38.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.504+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f655c013470 con 0x7f65680c4d80 2024-02-19T15:04:38.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.504+0000 7f654ffff640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f655406b810 0x7f655406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.504+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f655c08a540 con 0x7f65680c4d80 2024-02-19T15:04:38.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.504+0000 7f65677fe640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f655406b810 0x7f655406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.505+0000 7f65677fe640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f655406b810 0x7f655406dcd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6558036000 tx=0x7f6558030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.509+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f655c054e10 con 0x7f65680c4d80 2024-02-19T15:04:38.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.713+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6568157c70 con 0x7f65680c4d80 2024-02-19T15:04:38.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.714+0000 7f654ffff640 1 -- 172.21.15.78:0/1342178553 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v209) v1 ==== 161+0+0 (secure 0 0 0) 0x7f655c0584d0 con 0x7f65680c4d80 2024-02-19T15:04:38.717 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f655406b810 msgr2=0x7f655406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f655406b810 0x7f655406dcd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f6558036000 tx=0x7f6558030000 comp rx=0 tx=0).stop 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 msgr2=0x7f65681ed8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 0x7f65681ed8b0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f655c00e9b0 tx=0x7f655c00ee70 comp rx=0 tx=0).stop 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 shutdown_connections 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f655406b810 0x7f655406dcd0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65680c4d80 0x7f65681ed8b0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65680c78a0 0x7f65681e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 --2- 172.21.15.78:0/1342178553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65680c48f0 0x7f6568158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 >> 172.21.15.78:0/1342178553 conn(0x7f6568066e70 msgr2=0x7f65680c0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:38.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 shutdown_connections 2024-02-19T15:04:38.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.717+0000 7f656e1a1640 1 -- 172.21.15.78:0/1342178553 wait complete. 2024-02-19T15:04:38.723 INFO:tasks.ceph.mds.d.smithi150.stderr: -8571> 2024-02-19T15:00:54.566+0000 7fc2f1360640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-19T15:04:38.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-19T15:04:38.754 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2024-02-19T15:04:38.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.829+0000 7fcb069f4640 1 Processor -- start 2024-02-19T15:04:38.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcb069f4640 1 -- start start 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcb069f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcb000ac390 0x7fcb000ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcb069f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcb000c55f0 0x7fcb0015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcb069f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb00158c40 con 0x7fcb000bdf20 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcb069f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb000c8de0 con 0x7fcb000ac390 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcb069f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb000c5a30 con 0x7fcb000c55f0 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcafffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcb000ac390 0x7fcb000ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcafffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fcb000ac390 0x7fcb000ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46302/0 (socket says 172.21.15.78:46302) 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.830+0000 7fcafffff640 1 -- 172.21.15.78:0/185812142 learned_addr learned my addr 172.21.15.78:0/185812142 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:38.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.831+0000 7fcb04f91640 1 --2- 172.21.15.78:0/185812142 >> v2:172.21.15.150:3300/0 conn(0x7fcb000c55f0 0x7fcb0015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.831+0000 7fcafffff640 1 -- 172.21.15.78:0/185812142 >> v2:172.21.15.150:3300/0 conn(0x7fcb000c55f0 msgr2=0x7fcb0015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.831+0000 7fcafffff640 1 --2- 172.21.15.78:0/185812142 >> v2:172.21.15.150:3300/0 conn(0x7fcb000c55f0 0x7fcb0015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.831+0000 7fcafffff640 1 -- 172.21.15.78:0/185812142 >> v1:172.21.15.150:6789/0 conn(0x7fcb000bdf20 legacy=0x7fcb00158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.831+0000 7fcafffff640 1 -- 172.21.15.78:0/185812142 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb00155a60 con 0x7fcb000ac390 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.831+0000 7fcafffff640 1 --2- 172.21.15.78:0/185812142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000ac770 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fcb0006d9a0 tx=0x7fcaf002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf82022042d7ccd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcafeffd640 1 -- 172.21.15.78:0/185812142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf00317e0 con 0x7fcb000ac390 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcafeffd640 1 -- 172.21.15.78:0/185812142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf0031940 con 0x7fcb000ac390 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcafeffd640 1 -- 172.21.15.78:0/185812142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0031cd0 con 0x7fcb000ac390 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 -- 172.21.15.78:0/185812142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 msgr2=0x7fcb000ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 --2- 172.21.15.78:0/185812142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000ac770 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fcb0006d9a0 tx=0x7fcaf002b880 comp rx=0 tx=0).stop 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 -- 172.21.15.78:0/185812142 shutdown_connections 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 --2- 172.21.15.78:0/185812142 >> v2:172.21.15.150:3300/0 conn(0x7fcb000c55f0 0x7fcb0015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 --2- 172.21.15.78:0/185812142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000ac770 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 -- 172.21.15.78:0/185812142 >> 172.21.15.78:0/185812142 conn(0x7fcb000ca890 msgr2=0x7fcb000caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:38.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 -- 172.21.15.78:0/185812142 shutdown_connections 2024-02-19T15:04:38.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.832+0000 7fcb069f4640 1 -- 172.21.15.78:0/185812142 wait complete. 2024-02-19T15:04:38.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.833+0000 7fcb069f4640 1 Processor -- start 2024-02-19T15:04:38.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.833+0000 7fcb069f4640 1 -- start start 2024-02-19T15:04:38.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb069f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb069f4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb000bdf20 0x7fcb000ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb069f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb000c55f0 0x7fcb000aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb069f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb001ebbd0 con 0x7fcb000ac390 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb069f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb000c3370 con 0x7fcb000c55f0 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb069f4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb001e2080 con 0x7fcb000bdf20 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46308/0 (socket says 172.21.15.78:46308) 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 -- 172.21.15.78:0/1386485570 learned_addr learned my addr 172.21.15.78:0/1386485570 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcaff7fe640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb000bdf20 0x7fcb000ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcb04f91640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb000c55f0 0x7fcb000aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 -- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb000bdf20 msgr2=0x7fcb000ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb000bdf20 0x7fcb000ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 -- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb000c55f0 msgr2=0x7fcb000aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb000c55f0 0x7fcb000aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:38.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.834+0000 7fcafffff640 1 -- 172.21.15.78:0/1386485570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00094d0 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.835+0000 7fcafffff640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000c2e30 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0000c00 tx=0x7fcaf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.835+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0032880 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.835+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf0032a00 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.835+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb000c3730 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.835+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0032db0 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.835+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb001e2340 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.836+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb000c38c0 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.836+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fcaf00453f0 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.836+0000 7fcafcff9640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcaec06b810 0x7fcaec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.837+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcaf00ac680 con 0x7fcb000ac390 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.837+0000 7fcaff7fe640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcaec06b810 0x7fcaec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:38.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.837+0000 7fcaff7fe640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcaec06b810 0x7fcaec06dcd0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcb000c1880 tx=0x7fcaf4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:38.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:38.841+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcb000c38c0 con 0x7fcb000ac390 2024-02-19T15:04:39.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.047+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcb000c38c0 con 0x7fcb000ac390 2024-02-19T15:04:39.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.716+0000 7fcafcff9640 1 -- 172.21.15.78:0/1386485570 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v210) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcb000c38c0 con 0x7fcb000ac390 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcaec06b810 msgr2=0x7fcaec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcaec06b810 0x7fcaec06dcd0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcb000c1880 tx=0x7fcaf4009040 comp rx=0 tx=0).stop 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 msgr2=0x7fcb000c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000c2e30 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0000c00 tx=0x7fcaf00046c0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 shutdown_connections 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fcaec06b810 0x7fcaec06dcd0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb000c55f0 0x7fcb000aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb000bdf20 0x7fcb000ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 --2- 172.21.15.78:0/1386485570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb000ac390 0x7fcb000c2e30 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 >> 172.21.15.78:0/1386485570 conn(0x7fcb000ca890 msgr2=0x7fcb000c9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:39.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 shutdown_connections 2024-02-19T15:04:39.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.718+0000 7fcb069f4640 1 -- 172.21.15.78:0/1386485570 wait complete. 2024-02-19T15:04:39.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-19T15:04:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.838+0000 7fa3dae88640 1 Processor -- start 2024-02-19T15:04:39.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.838+0000 7fa3dae88640 1 -- start start 2024-02-19T15:04:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3dae88640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3d40c79d0 0x7fa3d40c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3dae88640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3d4152ac0 0x7fa3d4158600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3dae88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d406c570 con 0x7fa3d40c59c0 2024-02-19T15:04:39.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3dae88640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d406caa0 con 0x7fa3d4152ac0 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3dae88640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d40c52d0 con 0x7fa3d40c79d0 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3d4152ac0 0x7fa3d4158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3d4152ac0 0x7fa3d4158600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46314/0 (socket says 172.21.15.78:46314) 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa3d40c59c0 0x7fa3d41523b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50616/0 (socket says 172.21.15.78:50616) 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 -- 172.21.15.78:0/3369170695 learned_addr learned my addr 172.21.15.78:0/3369170695 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d8c24640 1 --2- 172.21.15.78:0/3369170695 >> v2:172.21.15.150:3300/0 conn(0x7fa3d40c79d0 0x7fa3d40c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 -- 172.21.15.78:0/3369170695 >> v2:172.21.15.150:3300/0 conn(0x7fa3d40c79d0 msgr2=0x7fa3d40c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 --2- 172.21.15.78:0/3369170695 >> v2:172.21.15.150:3300/0 conn(0x7fa3d40c79d0 0x7fa3d40c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 -- 172.21.15.78:0/3369170695 >> v1:172.21.15.78:6789/0 conn(0x7fa3d40c59c0 legacy=0x7fa3d41523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d9425640 1 -- 172.21.15.78:0/3369170695 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d40c5760 con 0x7fa3d4152ac0 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d37fe640 1 -- 172.21.15.78:0/3369170695 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909179895 0 0) 0x7fa3d406c570 con 0x7fa3d40c59c0 2024-02-19T15:04:39.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.839+0000 7fa3d8c24640 1 --2- 172.21.15.78:0/3369170695 >> v2:172.21.15.150:3300/0 conn(0x7fa3d40c79d0 0x7fa3d40c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.840+0000 7fa3d9425640 1 --2- 172.21.15.78:0/3369170695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d4152ac0 0x7fa3d4158600 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40cb8c0 tx=0x7fa3cc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73aa58bd9adfc24 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.840+0000 7fa3d37fe640 1 -- 172.21.15.78:0/3369170695 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc02ca80 con 0x7fa3d4152ac0 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.840+0000 7fa3d37fe640 1 -- 172.21.15.78:0/3369170695 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc02cbe0 con 0x7fa3d4152ac0 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.840+0000 7fa3d37fe640 1 -- 172.21.15.78:0/3369170695 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc0343f0 con 0x7fa3d4152ac0 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 -- 172.21.15.78:0/3369170695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d4152ac0 msgr2=0x7fa3d4158600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 --2- 172.21.15.78:0/3369170695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d4152ac0 0x7fa3d4158600 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40cb8c0 tx=0x7fa3cc02bee0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 -- 172.21.15.78:0/3369170695 shutdown_connections 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 --2- 172.21.15.78:0/3369170695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d4152ac0 0x7fa3d4158600 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 --2- 172.21.15.78:0/3369170695 >> v2:172.21.15.150:3300/0 conn(0x7fa3d40c79d0 0x7fa3d40c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 -- 172.21.15.78:0/3369170695 >> 172.21.15.78:0/3369170695 conn(0x7fa3d40ac750 msgr2=0x7fa3d40acb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 -- 172.21.15.78:0/3369170695 shutdown_connections 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.841+0000 7fa3dae88640 1 -- 172.21.15.78:0/3369170695 wait complete. 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.842+0000 7fa3dae88640 1 Processor -- start 2024-02-19T15:04:39.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.842+0000 7fa3dae88640 1 -- start start 2024-02-19T15:04:39.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3dae88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 0x7fa3d414ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3dae88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d40c79d0 0x7fa3d414acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3dae88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d4152ac0 0x7fa3d414b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3dae88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4150de0 con 0x7fa3d40c59c0 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3dae88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40af300 con 0x7fa3d40c79d0 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3dae88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ad650 con 0x7fa3d4152ac0 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 0x7fa3d414ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 0x7fa3d414ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46330/0 (socket says 172.21.15.78:46330) 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 -- 172.21.15.78:0/1189017929 learned_addr learned my addr 172.21.15.78:0/1189017929 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d3fff640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d40c79d0 0x7fa3d414acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d9425640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d4152ac0 0x7fa3d414b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 -- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d4152ac0 msgr2=0x7fa3d414b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:39.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d4152ac0 0x7fa3d414b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 -- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d40c79d0 msgr2=0x7fa3d414acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d40c79d0 0x7fa3d414acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 -- 172.21.15.78:0/1189017929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc009b30 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.843+0000 7fa3d8c24640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 0x7fa3d414ddb0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400e730 tx=0x7fa3c400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.844+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4004230 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.844+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4004390 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.844+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d40ad930 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.844+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c400c8f0 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.844+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d4151990 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.845+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d4074990 con 0x7fa3d40c59c0 2024-02-19T15:04:39.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.845+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa3c400ca50 con 0x7fa3d40c59c0 2024-02-19T15:04:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.845+0000 7fa3d17fa640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa3c006b840 0x7fa3c006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.845+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa3c408a0a0 con 0x7fa3d40c59c0 2024-02-19T15:04:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.845+0000 7fa3d3fff640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa3c006b840 0x7fa3c006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:39.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.846+0000 7fa3d3fff640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa3c006b840 0x7fa3c006dd00 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa3d414c640 tx=0x7fa3c8007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:39.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:39.850+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa3c4054810 con 0x7fa3d40c59c0 2024-02-19T15:04:40.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.020+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3d41526b0 con 0x7fa3d40c59c0 2024-02-19T15:04:40.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.721+0000 7fa3d17fa640 1 -- 172.21.15.78:0/1189017929 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v219) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa3c4057ed0 con 0x7fa3d40c59c0 2024-02-19T15:04:40.724 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-02-19T15:04:40.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.723+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa3c006b840 msgr2=0x7fa3c006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:40.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.723+0000 7fa3dae88640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa3c006b840 0x7fa3c006dd00 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa3d414c640 tx=0x7fa3c8007b50 comp rx=0 tx=0).stop 2024-02-19T15:04:40.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.723+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 msgr2=0x7fa3d414ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.723+0000 7fa3dae88640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 0x7fa3d414ddb0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400e730 tx=0x7fa3c400ebf0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.724+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 shutdown_connections 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.724+0000 7fa3dae88640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa3c006b840 0x7fa3c006dd00 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.724+0000 7fa3dae88640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d4152ac0 0x7fa3d414b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.724+0000 7fa3dae88640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d40c79d0 0x7fa3d414acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.724+0000 7fa3dae88640 1 --2- 172.21.15.78:0/1189017929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3d40c59c0 0x7fa3d414ddb0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.724+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 >> 172.21.15.78:0/1189017929 conn(0x7fa3d40ac750 msgr2=0x7fa3d4153bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.726+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 shutdown_connections 2024-02-19T15:04:40.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.727+0000 7fa3dae88640 1 -- 172.21.15.78:0/1189017929 wait complete. 2024-02-19T15:04:40.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-19T15:04:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.844+0000 7f2008f60640 1 Processor -- start 2024-02-19T15:04:40.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.844+0000 7f2008f60640 1 -- start start 2024-02-19T15:04:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2008f60640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20040c7f10 0x7f20041550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2008f60640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004155610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:40.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2008f60640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200406fa90 con 0x7f20041527d0 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2008f60640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20041592b0 con 0x7f20040c7f10 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2008f60640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2004071280 con 0x7f20040bf0c0 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2002d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2001d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20040c7f10 0x7f20041550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f200259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f20041527d0 0x7f20040c4e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42874/0 (socket says 172.21.15.78:42874) 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2002d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:47564/0 (socket says 172.21.15.78:47564) 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f2001d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20040c7f10 0x7f20041550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:46352/0 (socket says 172.21.15.78:46352) 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.845+0000 7f200259c640 1 -- 172.21.15.78:0/1368699581 learned_addr learned my addr 172.21.15.78:0/1368699581 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.846+0000 7f2002d9d640 1 -- 172.21.15.78:0/1368699581 >> v2:172.21.15.150:3300/0 conn(0x7f20040c7f10 msgr2=0x7f20041550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.846+0000 7f2002d9d640 1 --2- 172.21.15.78:0/1368699581 >> v2:172.21.15.150:3300/0 conn(0x7f20040c7f10 0x7f20041550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.846+0000 7f2002d9d640 1 -- 172.21.15.78:0/1368699581 >> v1:172.21.15.150:6789/0 conn(0x7f20041527d0 legacy=0x7f20040c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:40.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.846+0000 7f2002d9d640 1 -- 172.21.15.78:0/1368699581 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200406f4d0 con 0x7f20040bf0c0 2024-02-19T15:04:40.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.846+0000 7f2002d9d640 1 --2- 172.21.15.78:0/1368699581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004155610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d90 tx=0x7f1ffc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c0df57340a36c51 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f200159a640 1 -- 172.21.15.78:0/1368699581 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc02cc50 con 0x7f20040bf0c0 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f200159a640 1 -- 172.21.15.78:0/1368699581 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc02cdb0 con 0x7f20040bf0c0 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f200159a640 1 -- 172.21.15.78:0/1368699581 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc034660 con 0x7f20040bf0c0 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 -- 172.21.15.78:0/1368699581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 msgr2=0x7f2004155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 --2- 172.21.15.78:0/1368699581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004155610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d90 tx=0x7f1ffc02bea0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 -- 172.21.15.78:0/1368699581 shutdown_connections 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 --2- 172.21.15.78:0/1368699581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004155610 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 --2- 172.21.15.78:0/1368699581 >> v2:172.21.15.150:3300/0 conn(0x7f20040c7f10 0x7f20041550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 -- 172.21.15.78:0/1368699581 >> 172.21.15.78:0/1368699581 conn(0x7f20040aa370 msgr2=0x7f20040aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 -- 172.21.15.78:0/1368699581 shutdown_connections 2024-02-19T15:04:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.847+0000 7f2008f60640 1 -- 172.21.15.78:0/1368699581 wait complete. 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 Processor -- start 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 -- start start 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20040c7f10 0x7f20041edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 0x7f2004140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20040c27f0 con 0x7f20041527d0 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041ee9d0 con 0x7f20040c7f10 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2008f60640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20040c2230 con 0x7f20040bf0c0 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2002d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 0x7f2004140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2002d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 0x7f2004140770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46342/0 (socket says 172.21.15.78:46342) 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.849+0000 7f2002d9d640 1 -- 172.21.15.78:0/668377377 learned_addr learned my addr 172.21.15.78:0/668377377 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:40.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f200259c640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2001d9b640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20040c7f10 0x7f20041edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2002d9d640 1 -- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 msgr2=0x7f2004156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2002d9d640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2002d9d640 1 -- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20040c7f10 msgr2=0x7f20041edf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2002d9d640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20040c7f10 0x7f20041edf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2002d9d640 1 -- 172.21.15.78:0/668377377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc009a70 con 0x7f20041527d0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2001d9b640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20040c7f10 0x7f20041edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2002d9d640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 0x7f2004140770 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d90 tx=0x7f1ffc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc040650 con 0x7f20041527d0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc0407b0 con 0x7f20041527d0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20040c2510 con 0x7f20041527d0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc040b60 con 0x7f20041527d0 2024-02-19T15:04:40.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.850+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20040bfa00 con 0x7f20041527d0 2024-02-19T15:04:40.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.851+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20040bfd80 con 0x7f20041527d0 2024-02-19T15:04:40.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.852+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1ffc040cc0 con 0x7f20041527d0 2024-02-19T15:04:40.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.852+0000 7f1ff2ffd640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe406b810 0x7f1fe406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:40.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.852+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f1ffc0ad110 con 0x7f20041527d0 2024-02-19T15:04:40.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.852+0000 7f200259c640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe406b810 0x7f1fe406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:40.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.853+0000 7f200259c640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe406b810 0x7f1fe406dcd0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4000f60 tx=0x7f1ff4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:40.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:40.857+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f20040bfd80 con 0x7f20041527d0 2024-02-19T15:04:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.030+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f20040bfd80 con 0x7f20041527d0 2024-02-19T15:04:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.730+0000 7f1ff2ffd640 1 -- 172.21.15.78:0/668377377 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v220) v1 ==== 169+0+0 (secure 0 0 0) 0x7f20040bfd80 con 0x7f20041527d0 2024-02-19T15:04:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-02-19T15:04:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe406b810 msgr2=0x7f1fe406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe406b810 0x7f1fe406dcd0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4000f60 tx=0x7f1ff4009040 comp rx=0 tx=0).stop 2024-02-19T15:04:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 msgr2=0x7f2004140770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 0x7f2004140770 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009d90 tx=0x7f1ffc004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 shutdown_connections 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f1fe406b810 0x7f1fe406dcd0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20041527d0 0x7f2004140770 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20040c7f10 0x7f20041edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 --2- 172.21.15.78:0/668377377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20040bf0c0 0x7f2004156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 >> 172.21.15.78:0/668377377 conn(0x7f20040aa370 msgr2=0x7f2004155a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 shutdown_connections 2024-02-19T15:04:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.732+0000 7f2008f60640 1 -- 172.21.15.78:0/668377377 wait complete. 2024-02-19T15:04:41.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-19T15:04:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.852+0000 7f44640ff640 1 Processor -- start 2024-02-19T15:04:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.852+0000 7f44640ff640 1 -- start start 2024-02-19T15:04:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f44640ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f445c0add70 0x7f445c0c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f44640ff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f445c0705b0 0x7f445c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f44640ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c0ae670 con 0x7f445c0c2f60 2024-02-19T15:04:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f44640ff640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0716b0 con 0x7f445c0705b0 2024-02-19T15:04:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f44640ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c0c33d0 con 0x7f445c0add70 2024-02-19T15:04:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f446269c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f445c0705b0 0x7f445c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f446269c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f445c0705b0 0x7f445c0709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46346/0 (socket says 172.21.15.78:46346) 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.853+0000 7f446269c640 1 -- 172.21.15.78:0/2747492478 learned_addr learned my addr 172.21.15.78:0/2747492478 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f446269c640 1 -- 172.21.15.78:0/2747492478 >> v2:172.21.15.150:3300/0 conn(0x7f445c0add70 msgr2=0x7f445c0c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f4461e9b640 1 --2- 172.21.15.78:0/2747492478 >> v2:172.21.15.150:3300/0 conn(0x7f445c0add70 0x7f445c0c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f446269c640 1 --2- 172.21.15.78:0/2747492478 >> v2:172.21.15.150:3300/0 conn(0x7f445c0add70 0x7f445c0c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f446269c640 1 -- 172.21.15.78:0/2747492478 >> v1:172.21.15.150:6789/0 conn(0x7f445c0c2f60 legacy=0x7f445c06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f446269c640 1 -- 172.21.15.78:0/2747492478 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c153780 con 0x7f445c0705b0 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f4461e9b640 1 --2- 172.21.15.78:0/2747492478 >> v2:172.21.15.150:3300/0 conn(0x7f445c0add70 0x7f445c0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.854+0000 7f446269c640 1 --2- 172.21.15.78:0/2747492478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0705b0 0x7f445c0709c0 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f445c0c4fc0 tx=0x7f445402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ef99e37b043f4b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f4460e99640 1 -- 172.21.15.78:0/2747492478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44540317e0 con 0x7f445c0705b0 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f4460e99640 1 -- 172.21.15.78:0/2747492478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4454031940 con 0x7f445c0705b0 2024-02-19T15:04:41.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f4460e99640 1 -- 172.21.15.78:0/2747492478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454031cd0 con 0x7f445c0705b0 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 -- 172.21.15.78:0/2747492478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0705b0 msgr2=0x7f445c0709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 --2- 172.21.15.78:0/2747492478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0705b0 0x7f445c0709c0 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f445c0c4fc0 tx=0x7f445402b880 comp rx=0 tx=0).stop 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 -- 172.21.15.78:0/2747492478 shutdown_connections 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 --2- 172.21.15.78:0/2747492478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0705b0 0x7f445c0709c0 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 --2- 172.21.15.78:0/2747492478 >> v2:172.21.15.150:3300/0 conn(0x7f445c0add70 0x7f445c0c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 -- 172.21.15.78:0/2747492478 >> 172.21.15.78:0/2747492478 conn(0x7f445c0c85e0 msgr2=0x7f445c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 -- 172.21.15.78:0/2747492478 shutdown_connections 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.855+0000 7f44640ff640 1 -- 172.21.15.78:0/2747492478 wait complete. 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.856+0000 7f44640ff640 1 Processor -- start 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.856+0000 7f44640ff640 1 -- start start 2024-02-19T15:04:41.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f44640ff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c0705b0 0x7f445c09c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f44640ff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 0x7f445c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f44640ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0c2f60 0x7f445c0c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f44640ff640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1eb550 con 0x7f445c0add70 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f44640ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c099fa0 con 0x7f445c0c2f60 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f44640ff640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c1eaf90 con 0x7f445c0705b0 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 0x7f445c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 0x7f445c099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46356/0 (socket says 172.21.15.78:46356) 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 -- 172.21.15.78:0/1194880519 learned_addr learned my addr 172.21.15.78:0/1194880519 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f4461e9b640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c0705b0 0x7f445c09c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446269c640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0c2f60 0x7f445c0c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 -- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c0705b0 msgr2=0x7f445c09c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c0705b0 0x7f445c09c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 -- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0c2f60 msgr2=0x7f445c0c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0c2f60 0x7f445c0c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:41.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 -- 172.21.15.78:0/1194880519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540094d0 con 0x7f445c0add70 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f4461e9b640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c0705b0 0x7f445c09c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.857+0000 7f446169a640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 0x7f445c099a60 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c460 tx=0x7f444c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.858+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c012830 con 0x7f445c0add70 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.858+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c0129d0 con 0x7f445c0add70 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.858+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c1eb270 con 0x7f445c0add70 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.858+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c012d80 con 0x7f445c0add70 2024-02-19T15:04:41.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.858+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c06c410 con 0x7f445c0add70 2024-02-19T15:04:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.859+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c0676a0 con 0x7f445c0add70 2024-02-19T15:04:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.859+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f444c01a3b0 con 0x7f445c0add70 2024-02-19T15:04:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.860+0000 7f44527fc640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444006b810 0x7f444006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.860+0000 7f4461e9b640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444006b810 0x7f444006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.860+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f444c08a030 con 0x7f445c0add70 2024-02-19T15:04:41.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.860+0000 7f4461e9b640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444006b810 0x7f444006dcd0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f445800e550 tx=0x7f4458007ab0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:41.864+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f444c054b50 con 0x7f445c0add70 2024-02-19T15:04:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.036+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f445c06bf90 con 0x7f445c0add70 2024-02-19T15:04:42.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.040+0000 7f44527fc640 1 -- 172.21.15.78:0/1194880519 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f445c06bf90 con 0x7f445c0add70 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444006b810 msgr2=0x7f444006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444006b810 0x7f444006dcd0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f445800e550 tx=0x7f4458007ab0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 msgr2=0x7f445c099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 0x7f445c099a60 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f444c00c460 tx=0x7f444c00c920 comp rx=0 tx=0).stop 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 shutdown_connections 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f444006b810 0x7f444006dcd0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f445c0add70 0x7f445c099a60 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0c2f60 0x7f445c0c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 --2- 172.21.15.78:0/1194880519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f445c0705b0 0x7f445c09c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 >> 172.21.15.78:0/1194880519 conn(0x7f445c0c85e0 msgr2=0x7f445c14fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 shutdown_connections 2024-02-19T15:04:42.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.042+0000 7f44640ff640 1 -- 172.21.15.78:0/1194880519 wait complete. 2024-02-19T15:04:42.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-02-19T15:04:42.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.156+0000 7f9cb913e640 1 Processor -- start 2024-02-19T15:04:42.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.156+0000 7f9cb913e640 1 -- start start 2024-02-19T15:04:42.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb913e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb40c19f0 0x7f9cb4156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:42.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb913e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 0x7f9cb40c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:42.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb913e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb4155a50 con 0x7f9cb40c7570 2024-02-19T15:04:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb913e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb4152af0 con 0x7f9cb4071410 2024-02-19T15:04:42.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb913e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb4155f80 con 0x7f9cb40c19f0 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb2d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb40c19f0 0x7f9cb4156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 0x7f9cb40c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 0x7f9cb40c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:46394/0 (socket says 172.21.15.78:46394) 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb359e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9cb40c7570 0x7f9cb40c1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42884/0 (socket says 172.21.15.78:42884) 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb259c640 1 -- 172.21.15.78:0/1029974360 learned_addr learned my addr 172.21.15.78:0/1029974360 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb2d9d640 1 -- 172.21.15.78:0/1029974360 >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 msgr2=0x7f9cb40c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb2d9d640 1 --2- 172.21.15.78:0/1029974360 >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 0x7f9cb40c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.157+0000 7f9cb2d9d640 1 -- 172.21.15.78:0/1029974360 >> v1:172.21.15.150:6789/0 conn(0x7f9cb40c7570 legacy=0x7f9cb40c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.158+0000 7f9cb2d9d640 1 -- 172.21.15.78:0/1029974360 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4070840 con 0x7f9cb40c19f0 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.158+0000 7f9cb1d9b640 1 -- 172.21.15.78:0/1029974360 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797882231 0 0) 0x7f9cb4155a50 con 0x7f9cb40c7570 2024-02-19T15:04:42.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.158+0000 7f9cb259c640 1 --2- 172.21.15.78:0/1029974360 >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 0x7f9cb40c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.158+0000 7f9cb2d9d640 1 --2- 172.21.15.78:0/1029974360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb40c19f0 0x7f9cb4156440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40097a0 tx=0x7f9ca402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a248b2341e201c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb1d9b640 1 -- 172.21.15.78:0/1029974360 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca402f050 con 0x7f9cb40c19f0 2024-02-19T15:04:42.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb1d9b640 1 -- 172.21.15.78:0/1029974360 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca4031700 con 0x7f9cb40c19f0 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb1d9b640 1 -- 172.21.15.78:0/1029974360 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4031a90 con 0x7f9cb40c19f0 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 -- 172.21.15.78:0/1029974360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb40c19f0 msgr2=0x7f9cb4156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 --2- 172.21.15.78:0/1029974360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb40c19f0 0x7f9cb4156440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40097a0 tx=0x7f9ca402b7a0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 -- 172.21.15.78:0/1029974360 shutdown_connections 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 --2- 172.21.15.78:0/1029974360 >> v2:172.21.15.150:3300/0 conn(0x7f9cb4071410 0x7f9cb40c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 --2- 172.21.15.78:0/1029974360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb40c19f0 0x7f9cb4156440 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 -- 172.21.15.78:0/1029974360 >> 172.21.15.78:0/1029974360 conn(0x7f9cb40aa370 msgr2=0x7f9cb40aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 -- 172.21.15.78:0/1029974360 shutdown_connections 2024-02-19T15:04:42.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.159+0000 7f9cb913e640 1 -- 172.21.15.78:0/1029974360 wait complete. 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.160+0000 7f9cb913e640 1 Processor -- start 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.160+0000 7f9cb913e640 1 -- start start 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb913e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb4071410 0x7f9cb41eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb913e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9cb40c19f0 0x7f9cb4151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb913e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 0x7f9cb4153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb913e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb41530e0 con 0x7f9cb40c7570 2024-02-19T15:04:42.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb913e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb41eba30 con 0x7f9cb40c19f0 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb913e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb41543e0 con 0x7f9cb4071410 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 0x7f9cb4153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb359e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 0x7f9cb4153500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46358/0 (socket says 172.21.15.78:46358) 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb359e640 1 -- 172.21.15.78:0/3831680647 learned_addr learned my addr 172.21.15.78:0/3831680647 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb259c640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9cb40c19f0 0x7f9cb4151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.161+0000 7f9cb2d9d640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb4071410 0x7f9cb41eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb359e640 1 -- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb4071410 msgr2=0x7f9cb41eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb359e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb4071410 0x7f9cb41eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb359e640 1 -- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9cb40c19f0 msgr2=0x7f9cb4151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb359e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9cb40c19f0 0x7f9cb4151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb359e640 1 -- 172.21.15.78:0/3831680647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80094d0 con 0x7f9cb40c7570 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb259c640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9cb40c19f0 0x7f9cb4151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb359e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 0x7f9cb4153500 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00c700 tx=0x7f9cac00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00ce10 con 0x7f9cb40c7570 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac00a5e0 con 0x7f9cb40c7570 2024-02-19T15:04:42.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca4009400 con 0x7f9cb40c7570 2024-02-19T15:04:42.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00a9d0 con 0x7f9cb40c7570 2024-02-19T15:04:42.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.162+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb41549a0 con 0x7f9cb40c7570 2024-02-19T15:04:42.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.163+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb40c8320 con 0x7f9cb40c7570 2024-02-19T15:04:42.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.163+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f9cac00ab30 con 0x7f9cb40c7570 2024-02-19T15:04:42.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.164+0000 7f9c9b7fe640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ca006b810 0x7f9ca006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:42.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.164+0000 7f9cb2d9d640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ca006b810 0x7f9ca006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:42.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.164+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f9cac089750 con 0x7f9cb40c7570 2024-02-19T15:04:42.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.164+0000 7f9cb2d9d640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ca006b810 0x7f9ca006dcd0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4035000 tx=0x7f9ca4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:42.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.168+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9cac054270 con 0x7f9cb40c7570 2024-02-19T15:04:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.407+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f9cb4153220 con 0x7f9cb40c7570 2024-02-19T15:04:42.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.731+0000 7f9c9b7fe640 1 -- 172.21.15.78:0/3831680647 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v694) v1 ==== 139+0+0 (secure 0 0 0) 0x7f9cb4153220 con 0x7f9cb40c7570 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ca006b810 msgr2=0x7f9ca006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ca006b810 0x7f9ca006dcd0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4035000 tx=0x7f9ca4008030 comp rx=0 tx=0).stop 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 msgr2=0x7f9cb4153500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 0x7f9cb4153500 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00c700 tx=0x7f9cac00cbc0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 shutdown_connections 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f9ca006b810 0x7f9ca006dcd0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cb40c7570 0x7f9cb4153500 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9cb40c19f0 0x7f9cb4151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 --2- 172.21.15.78:0/3831680647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9cb4071410 0x7f9cb41eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:42.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 >> 172.21.15.78:0/3831680647 conn(0x7f9cb40aa370 msgr2=0x7f9cb40c1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:42.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 shutdown_connections 2024-02-19T15:04:42.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:42.734+0000 7f9cb913e640 1 -- 172.21.15.78:0/3831680647 wait complete. 2024-02-19T15:04:42.749 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. ... ERROR 2024-02-19T15:04:42.749 INFO:tasks.cephfs_test_runner: 2024-02-19T15:04:42.749 INFO:tasks.cephfs_test_runner:====================================================================== 2024-02-19T15:04:42.749 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T15:04:42.749 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-02-19T15:04:42.749 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:Ran 12 tests in 724.076s 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner: 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:====================================================================== 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T15:04:42.750 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-02-19T15:04:42.751 INFO:tasks.cephfs_test_runner: 2024-02-19T15:04:42.751 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T15:04:42.944 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=26d011ce065f481687ca22f81383b9ff Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9f8908eed2b2d7e8c316bd5be7ed6e49024931ee/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-19T15:04:42.945 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-02-19T15:04:42.954 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-19T15:04:42.967 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-19T15:04:42.967 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-02-19T15:04:42.967 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:04:42.996 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2322 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497664k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-19T15:04:42.996 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-19T15:04:42.996 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T15:04:43.015 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056371,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14944 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-19T15:04:43.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:04:43.015 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-02-19T15:04:43.051 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-19T15:04:43.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:04:43.052 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T15:04:43.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:04:43.120 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-02-19T15:04:43.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:04:43.226 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-02-19T15:04:43.245 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-02-19T15:04:43.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:04:43.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T15:04:43.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T15:04:43.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-02-19T15:04:43.413 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-19T15:04:43.423 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-19T15:04:43.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:04:43.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.523+0000 7f900a023640 1 Processor -- start 2024-02-19T15:04:43.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.523+0000 7f900a023640 1 -- start start 2024-02-19T15:04:43.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.523+0000 7f900a023640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c25e0 0x7f90041526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.523+0000 7f900a023640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90040c29c0 0x7f90040c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f900a023640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004072250 con 0x7f90040c49c0 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f900a023640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9004152c90 con 0x7f90040c29c0 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f900a023640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9004072480 con 0x7f90040c25e0 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f90037fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c25e0 0x7f90041526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9002ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90040c29c0 0x7f90040c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9003fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f90040c49c0 0x7f900406e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42890/0 (socket says 172.21.15.78:42890) 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9003fff640 1 -- 172.21.15.78:0/4105075107 learned_addr learned my addr 172.21.15.78:0/4105075107 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9002ffd640 1 -- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c25e0 msgr2=0x7f90041526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:43.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9002ffd640 1 --2- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c25e0 0x7f90041526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9002ffd640 1 -- 172.21.15.78:0/4105075107 >> v1:172.21.15.150:6789/0 conn(0x7f90040c49c0 legacy=0x7f900406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f9002ffd640 1 -- 172.21.15.78:0/4105075107 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040c3eb0 con 0x7f90040c29c0 2024-02-19T15:04:43.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f90037fe640 1 --2- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c25e0 0x7f90041526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:43.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.524+0000 7f90027fc640 1 -- 172.21.15.78:0/4105075107 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336613262 0 0) 0x7f9004072250 con 0x7f90040c49c0 2024-02-19T15:04:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.525+0000 7f9002ffd640 1 --2- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c29c0 0x7f90040c4480 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8ff80097a0 tx=0x7f8ff802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4559000e16c27bc7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.525+0000 7f90027fc640 1 -- 172.21.15.78:0/4105075107 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff802e070 con 0x7f90040c29c0 2024-02-19T15:04:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.525+0000 7f90027fc640 1 -- 172.21.15.78:0/4105075107 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff80315a0 con 0x7f90040c29c0 2024-02-19T15:04:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.525+0000 7f90027fc640 1 -- 172.21.15.78:0/4105075107 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8031930 con 0x7f90040c29c0 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 -- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c29c0 msgr2=0x7f90040c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 --2- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c29c0 0x7f90040c4480 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8ff80097a0 tx=0x7f8ff802b840 comp rx=0 tx=0).stop 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 -- 172.21.15.78:0/4105075107 shutdown_connections 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 --2- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c29c0 0x7f90040c4480 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 --2- 172.21.15.78:0/4105075107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c25e0 0x7f90041526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 -- 172.21.15.78:0/4105075107 >> 172.21.15.78:0/4105075107 conn(0x7f900406b270 msgr2=0x7f900406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 -- 172.21.15.78:0/4105075107 shutdown_connections 2024-02-19T15:04:43.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.526+0000 7f900a023640 1 -- 172.21.15.78:0/4105075107 wait complete. 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.527+0000 7f900a023640 1 Processor -- start 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.527+0000 7f900a023640 1 -- start start 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f900a023640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c25e0 0x7f9004150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f900a023640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c29c0 0x7f9004145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f900a023640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 0x7f90040a8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f900a023640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040747f0 con 0x7f90040741a0 2024-02-19T15:04:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f900a023640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90040abaa0 con 0x7f90040c25e0 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f900a023640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9004155cd0 con 0x7f90040c29c0 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 0x7f90040a8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 0x7f90040a8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46360/0 (socket says 172.21.15.78:46360) 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 -- 172.21.15.78:0/1224727643 learned_addr learned my addr 172.21.15.78:0/1224727643 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f90037fe640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c25e0 0x7f9004150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9002ffd640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c29c0 0x7f9004145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 -- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c29c0 msgr2=0x7f9004145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c29c0 0x7f9004145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 -- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c25e0 msgr2=0x7f9004150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c25e0 0x7f9004150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 -- 172.21.15.78:0/1224727643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4009400 con 0x7f90040741a0 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f90037fe640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c25e0 0x7f9004150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f9003fff640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 0x7f90040a8180 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c780 tx=0x7f8ffc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00ce90 con 0x7f90040741a0 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.528+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc00a640 con 0x7f90040741a0 2024-02-19T15:04:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.529+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff8009400 con 0x7f90040741a0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.529+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00aa70 con 0x7f90040741a0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.529+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90040abd60 con 0x7f90040741a0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.529+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004074990 con 0x7f90040741a0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.530+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8ffc00abd0 con 0x7f90040741a0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.530+0000 7f8febfff640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ff006b840 0x7f8ff006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.530+0000 7f90037fe640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ff006b840 0x7f8ff006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.530+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8ffc089910 con 0x7f90040741a0 2024-02-19T15:04:43.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.531+0000 7f90037fe640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ff006b840 0x7f8ff006dd00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009fd0 tx=0x7f8ff4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:43.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.535+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8ffc054450 con 0x7f90040741a0 2024-02-19T15:04:43.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.703+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9004072ce0 con 0x7f8ff006b840 2024-02-19T15:04:43.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.705+0000 7f8febfff640 1 -- 172.21.15.78:0/1224727643 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50707 (secure 0 0 0) 0x7f8ff4014440 con 0x7f8ff006b840 2024-02-19T15:04:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:43.709 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:04:43.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ff006b840 msgr2=0x7f8ff006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ff006b840 0x7f8ff006dd00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009fd0 tx=0x7f8ff4009190 comp rx=0 tx=0).stop 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 msgr2=0x7f90040a8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 0x7f90040a8180 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00c780 tx=0x7f8ffc00cc40 comp rx=0 tx=0).stop 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 shutdown_connections 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8ff006b840 0x7f8ff006dd00 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90040741a0 0x7f90040a8180 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90040c29c0 0x7f9004145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 --2- 172.21.15.78:0/1224727643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90040c25e0 0x7f9004150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 >> 172.21.15.78:0/1224727643 conn(0x7f900406b270 msgr2=0x7f900406d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 shutdown_connections 2024-02-19T15:04:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.707+0000 7f900a023640 1 -- 172.21.15.78:0/1224727643 wait complete. 2024-02-19T15:04:43.721 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-02-19T15:04:42.218036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272216,"kb_used_data":22776,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754702504,"statfs":{"total":773094113280,"available":772815364096,"internally_reserved":0,"allocated":23322624,"data_stored":11945056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-52,"num_read_kb":-58,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001007"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":539,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-19T15:02:05.655593+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:02:05.655593+0000","last_peered":"2024-02-19T15:02:05.655593+0000","last_clean":"2024-02-19T15:02:05.655593+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:02:05.655593+0000","last_undegraded":"2024-02-19T15:02:05.655593+0000","last_fullsized":"2024-02-19T15:02:05.655593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575006,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34464,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337376,"statfs":{"total":96636764160,"available":96601473024,"internally_reserved":0,"allocated":3362816,"data_stored":1936180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.65100000000000002,"5min":0.747,"15min":0.747},"last":0.496},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.79500000000000004},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.40999999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.66900000000000004,"5min":0.71999999999999997,"15min":0.83099999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59499999999999997,"5min":0.70899999999999996,"15min":0.76400000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47999999999999998,"15min":0.49099999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.627,"5min":0.76000000000000001,"15min":0.80000000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.34799999999999998,"5min":0.30399999999999999,"15min":0.26500000000000001},"max":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.77500000000000002},"last":0.374}]},{"osd":3,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.72399999999999998,"15min":0.84199999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.61499999999999999,"5min":0.76900000000000002,"15min":0.81100000000000005},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.36499999999999999,"15min":0.42299999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.432,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.38200000000000001,"15min":0.45600000000000002},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.44,"5min":0.80400000000000005,"15min":0.86899999999999999},"last":0.29199999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.39500000000000002,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41299999999999998,"15min":0.49399999999999999},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.45600000000000002,"5min":0.76000000000000001,"15min":0.871},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51800000000000002,"5min":0.79400000000000004,"15min":0.90200000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.44600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.48399999999999999,"5min":0.78000000000000003,"15min":0.90500000000000003},"last":0.36099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":2756608,"data_stored":1344620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.22700000000000001},"max":{"1min":0.50900000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.502,"15min":0.499},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.78100000000000003,"15min":0.95699999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.311},"max":{"1min":0.625,"5min":0.80700000000000005,"15min":0.93600000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78500000000000003,"15min":0.90500000000000003},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.59499999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.318,"15min":0.318},"max":{"1min":0.61399999999999999,"5min":0.70799999999999996,"15min":0.77700000000000002},"last":0.628}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.374,"15min":0.46400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.40600000000000003,"5min":0.874,"15min":0.874},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.44500000000000001,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47799999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.29999999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.371,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.39200000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.316},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41699999999999998,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.47299999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.23100000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.503},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.45900000000000002,"5min":0.78900000000000003,"15min":0.89600000000000002},"last":0.254},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.42499999999999999,"15min":0.50900000000000001},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":0.42599999999999999,"5min":0.874,"15min":0.874},"last":0.27100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33876,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337964,"statfs":{"total":96636764160,"available":96602075136,"internally_reserved":0,"allocated":2760704,"data_stored":1344678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.58199999999999996,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.878,"15min":0.878},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.35699999999999998,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.81000000000000005,"15min":0.872},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49299999999999999,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64600000000000002,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.81299999999999994,"15min":0.81699999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.56499999999999995,"5min":0.79400000000000004,"15min":0.85899999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66200000000000003,"5min":0.82799999999999996,"15min":0.83799999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.503,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.75600000000000001,"15min":0.83899999999999997},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35399999999999998,"15min":0.45300000000000001},"min":{"1min":0.252,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":0.41099999999999998,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.311},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.249},"max":{"1min":0.51600000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.40000000000000002,"15min":0.49199999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.44,"5min":0.85999999999999999,"15min":0.877},"last":0.378},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.33500000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42699999999999999,"15min":0.52500000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.57999999999999996,"5min":0.876,"15min":0.90000000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45600000000000002,"15min":0.55600000000000005},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52900000000000003,"5min":0.96299999999999997,"15min":1.03},"last":0.27100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575006,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33880,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2764800,"data_stored":1346198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.505,"15min":0.53300000000000003},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70799999999999996,"5min":0.79200000000000004,"15min":4.6879999999999997},"last":0.497},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":1.341},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48399999999999999,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.53700000000000003,"5min":0.83399999999999996,"15min":2.6549999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48999999999999999,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":0.64900000000000002,"5min":0.82899999999999996,"15min":4.7290000000000001},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":2.5800000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.48999999999999999,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.56899999999999995,"5min":0.79100000000000004,"15min":10.629},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":1.2050000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84099999999999997,"15min":2.7010000000000001},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36499999999999999,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.40600000000000003,"5min":0.76400000000000001,"15min":6.6989999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38600000000000001,"15min":0.47099999999999997},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40000000000000002,"5min":0.77000000000000002,"15min":1.4470000000000001},"last":0.36899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38600000000000001,"15min":0.495},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.36399999999999999,"5min":0.85199999999999998,"15min":2.7349999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41499999999999998,"15min":0.52900000000000003},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.41499999999999998,"5min":0.83599999999999997,"15min":4.6239999999999997},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.41299999999999998,"15min":0.51000000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.41799999999999998,"5min":0.80300000000000005,"15min":1.246},"last":0.44},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.441,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46999999999999997,"5min":0.81999999999999995,"15min":2.621},"last":0.52300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575006,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33880,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2764800,"data_stored":1345348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.32700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.159},"max":{"1min":0.441,"5min":0.67900000000000005,"15min":3.694},"last":0.308},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":4.835},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.18099999999999999},"max":{"1min":0.58799999999999997,"5min":0.64200000000000002,"15min":0.76100000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.20000000000000001},"max":{"1min":0.501,"5min":0.66400000000000003,"15min":6.3810000000000002},"last":0.375}]},{"osd":2,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.40200000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":3.6320000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.372},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57799999999999996,"5min":0.68300000000000005,"15min":4.9889999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48499999999999999,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.27900000000000003,"15min":0.26600000000000001},"max":{"1min":0.58199999999999996,"5min":0.874,"15min":0.874},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60499999999999998,"5min":0.86199999999999999,"15min":4.9539999999999997},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53100000000000003,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63800000000000001,"5min":0.85699999999999998,"15min":3.6669999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.65900000000000003,"5min":0.88600000000000001,"15min":4.9130000000000003},"last":0.436}]},{"osd":6,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.80700000000000005,"15min":0.86799999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67800000000000005,"5min":0.82999999999999996,"15min":3.7189999999999999},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30499999999999999},"max":{"1min":0.65000000000000002,"5min":0.82199999999999995,"15min":4.8799999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.69299999999999995,"5min":0.871,"15min":3.0760000000000001},"last":0.48699999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575007,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33892,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337948,"statfs":{"total":96636764160,"available":96602058752,"internally_reserved":0,"allocated":2777088,"data_stored":1346892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34000000000000002,"15min":0.32700000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":0.52000000000000002,"5min":0.621,"15min":0.68899999999999995},"last":0.443},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.32800000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.57699999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.63,"5min":0.67200000000000004,"15min":1.512},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.35099999999999998},"min":{"1min":0.25600000000000001,"5min":0.22900000000000001,"15min":0.19700000000000001},"max":{"1min":0.52900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.38900000000000001},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":0.59099999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61299999999999999,"5min":0.84599999999999997,"15min":1.133},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51100000000000001,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61199999999999999,"5min":0.748,"15min":1.1539999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50700000000000001,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59699999999999998,"5min":0.78700000000000003,"15min":0.94999999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52700000000000002,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.627}]},{"osd":6,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.60299999999999998,"5min":0.82999999999999996,"15min":1.115},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":1.236},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.60299999999999998,"5min":0.81299999999999994,"15min":1.1830000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54300000000000004,"15min":0.56499999999999995},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.623,"5min":0.85799999999999998,"15min":0.995},"last":0.66100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33896,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337944,"statfs":{"total":96636764160,"available":96602054656,"internally_reserved":0,"allocated":2781184,"data_stored":1346026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.80600000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.223,"5min":0.218,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.90400000000000003},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.56100000000000005,"5min":0.624,"15min":0.77400000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.622,"5min":0.70199999999999996,"15min":0.76000000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.41399999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.72199999999999998},"last":0.371},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42599999999999999,"15min":0.39500000000000002},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.93300000000000005},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59099999999999997,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.314},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.88200000000000001,"15min":1.0009999999999999},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.77300000000000002,"15min":1.0189999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.53800000000000003},"min":{"1min":0.36299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.625,"5min":0.85599999999999998,"15min":1.0229999999999999},"last":0.39100000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.53700000000000003,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.31},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":1.026},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.307},"max":{"1min":0.65400000000000003,"5min":0.81399999999999995,"15min":0.996},"last":0.41799999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79200000000000004,"5min":0.90800000000000003,"15min":1.036},"last":0.499},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.83399999999999996,"15min":1.0740000000000001},"last":0.48099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34456,"kb_used_data":3276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337384,"statfs":{"total":96636764160,"available":96601481216,"internally_reserved":0,"allocated":3354624,"data_stored":1935114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.33700000000000002},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.17399999999999999},"max":{"1min":0.50900000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38300000000000001,"15min":0.35299999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.48399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52300000000000002,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.56699999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52900000000000003,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.35999999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.32400000000000001,"15min":0.28399999999999997},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53300000000000003,"15min":0.55500000000000005},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.442}]},{"osd":6,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45800000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.58299999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:04:43.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:04:43.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.820+0000 7fb2ce6ae640 1 Processor -- start 2024-02-19T15:04:43.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.820+0000 7fb2ce6ae640 1 -- start start 2024-02-19T15:04:43.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.820+0000 7fb2ce6ae640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2c80c25e0 0x7fb2c81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.820+0000 7fb2ce6ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c8072250 con 0x7fb2c80c5be0 2024-02-19T15:04:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.820+0000 7fb2ce6ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c8072480 con 0x7fb2c80c29c0 2024-02-19T15:04:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.820+0000 7fb2ce6ae640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c8152c90 con 0x7fb2c80c25e0 2024-02-19T15:04:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2c7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2c80c25e0 0x7fb2c81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2ccc4b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb2c80c5be0 0x7fb2c806e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50630/0 (socket says 172.21.15.78:50630) 2024-02-19T15:04:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2ccc4b640 1 -- 172.21.15.78:0/4268798554 learned_addr learned my addr 172.21.15.78:0/4268798554 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:43.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2c7fff640 1 -- 172.21.15.78:0/4268798554 >> v1:172.21.15.150:6789/0 conn(0x7fb2c80c29c0 legacy=0x7fb2c80c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2c7fff640 1 -- 172.21.15.78:0/4268798554 >> v1:172.21.15.78:6789/0 conn(0x7fb2c80c5be0 legacy=0x7fb2c806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2c7fff640 1 -- 172.21.15.78:0/4268798554 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c80c4b90 con 0x7fb2c80c25e0 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.821+0000 7fb2c6ffd640 1 -- 172.21.15.78:0/4268798554 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053904388 0 0) 0x7fb2c8072250 con 0x7fb2c80c5be0 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2c7fff640 1 --2- 172.21.15.78:0/4268798554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c25e0 0x7fb2c81526a0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097a0 tx=0x7fb2b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e542a09d5d8ac3f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2c6ffd640 1 -- 172.21.15.78:0/4268798554 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b802e070 con 0x7fb2c80c25e0 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2c6ffd640 1 -- 172.21.15.78:0/4268798554 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b80315a0 con 0x7fb2c80c25e0 2024-02-19T15:04:43.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2c6ffd640 1 -- 172.21.15.78:0/4268798554 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8031930 con 0x7fb2c80c25e0 2024-02-19T15:04:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/4268798554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c25e0 msgr2=0x7fb2c81526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/4268798554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c25e0 0x7fb2c81526a0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097a0 tx=0x7fb2b802b840 comp rx=0 tx=0).stop 2024-02-19T15:04:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/4268798554 shutdown_connections 2024-02-19T15:04:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/4268798554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c25e0 0x7fb2c81526a0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/4268798554 >> 172.21.15.78:0/4268798554 conn(0x7fb2c806b270 msgr2=0x7fb2c806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/4268798554 shutdown_connections 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.822+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/4268798554 wait complete. 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.823+0000 7fb2ce6ae640 1 Processor -- start 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 -- start start 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c80c25e0 0x7fb2c8145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c80c29c0 0x7fb2c8145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 0x7fb2c8152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8154fa0 con 0x7fb2c80c5be0 2024-02-19T15:04:43.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8155280 con 0x7fb2c80c29c0 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ce6ae640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8155560 con 0x7fb2c80c25e0 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ccc4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 0x7fb2c8152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ccc4b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 0x7fb2c8152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46374/0 (socket says 172.21.15.78:46374) 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.824+0000 7fb2ccc4b640 1 -- 172.21.15.78:0/3776222559 learned_addr learned my addr 172.21.15.78:0/3776222559 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2c7fff640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c80c25e0 0x7fb2c8145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2c77fe640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c80c29c0 0x7fb2c8145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ccc4b640 1 -- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c80c25e0 msgr2=0x7fb2c8145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ccc4b640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c80c25e0 0x7fb2c8145840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ccc4b640 1 -- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c80c29c0 msgr2=0x7fb2c8145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ccc4b640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c80c29c0 0x7fb2c8145d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ccc4b640 1 -- 172.21.15.78:0/3776222559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8009400 con 0x7fb2c80c5be0 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ccc4b640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 0x7fb2c8152ff0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c700 tx=0x7fb2c000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000ce10 con 0x7fb2c80c5be0 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c000a640 con 0x7fb2c80c5be0 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000aa70 con 0x7fb2c80c5be0 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c8145320 con 0x7fb2c80c5be0 2024-02-19T15:04:43.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.825+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c8144ff0 con 0x7fb2c80c5be0 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.826+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c8074990 con 0x7fb2c80c5be0 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.827+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fb2c000abd0 con 0x7fb2c80c5be0 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.827+0000 7fb2c4ff9640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b406b810 0x7fb2b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.827+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fb2c0089860 con 0x7fb2c80c5be0 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.827+0000 7fb2c7fff640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b406b810 0x7fb2b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.827+0000 7fb2c7fff640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b406b810 0x7fb2b406dcd0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036000 tx=0x7fb2b803d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:43.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.831+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2c0054380 con 0x7fb2c80c5be0 2024-02-19T15:04:44.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:43.999+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2c8156120 con 0x7fb2b406b810 2024-02-19T15:04:44.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.001+0000 7fb2c4ff9640 1 -- 172.21.15.78:0/3776222559 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50707 (secure 0 0 0) 0x7fb2c8156120 con 0x7fb2b406b810 2024-02-19T15:04:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:44.006 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:04:44.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b406b810 msgr2=0x7fb2b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b406b810 0x7fb2b406dcd0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036000 tx=0x7fb2b803d040 comp rx=0 tx=0).stop 2024-02-19T15:04:44.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 msgr2=0x7fb2c8152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 0x7fb2c8152ff0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c700 tx=0x7fb2c000cbc0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 shutdown_connections 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fb2b406b810 0x7fb2b406dcd0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2c80c5be0 0x7fb2c8152ff0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2c80c29c0 0x7fb2c8145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 --2- 172.21.15.78:0/3776222559 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2c80c25e0 0x7fb2c8145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 >> 172.21.15.78:0/3776222559 conn(0x7fb2c806b270 msgr2=0x7fb2c806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 shutdown_connections 2024-02-19T15:04:44.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.003+0000 7fb2ce6ae640 1 -- 172.21.15.78:0/3776222559 wait complete. 2024-02-19T15:04:44.016 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-02-19T15:04:42.218036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272216,"kb_used_data":22776,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754702504,"statfs":{"total":773094113280,"available":772815364096,"internally_reserved":0,"allocated":23322624,"data_stored":11945056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-52,"num_read_kb":-58,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001007"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":539,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-19T15:02:05.655593+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:02:05.655593+0000","last_peered":"2024-02-19T15:02:05.655593+0000","last_clean":"2024-02-19T15:02:05.655593+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:02:05.655593+0000","last_undegraded":"2024-02-19T15:02:05.655593+0000","last_fullsized":"2024-02-19T15:02:05.655593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575006,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34464,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337376,"statfs":{"total":96636764160,"available":96601473024,"internally_reserved":0,"allocated":3362816,"data_stored":1936180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.65100000000000002,"5min":0.747,"15min":0.747},"last":0.496},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.79500000000000004},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.40999999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.66900000000000004,"5min":0.71999999999999997,"15min":0.83099999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59499999999999997,"5min":0.70899999999999996,"15min":0.76400000000000001},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47999999999999998,"15min":0.49099999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.627,"5min":0.76000000000000001,"15min":0.80000000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.34799999999999998,"5min":0.30399999999999999,"15min":0.26500000000000001},"max":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.77500000000000002},"last":0.374}]},{"osd":3,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.72399999999999998,"15min":0.84199999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.61499999999999999,"5min":0.76900000000000002,"15min":0.81100000000000005},"last":0.45800000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.36499999999999999,"15min":0.42299999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.432,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.38200000000000001,"15min":0.45600000000000002},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.44,"5min":0.80400000000000005,"15min":0.86899999999999999},"last":0.29199999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.39500000000000002,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41299999999999998,"15min":0.49399999999999999},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.45600000000000002,"5min":0.76000000000000001,"15min":0.871},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51800000000000002,"5min":0.79400000000000004,"15min":0.90200000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.44600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.48399999999999999,"5min":0.78000000000000003,"15min":0.90500000000000003},"last":0.36099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":2756608,"data_stored":1344620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.22700000000000001},"max":{"1min":0.50900000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.502,"15min":0.499},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.78100000000000003,"15min":0.95699999999999996},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.311},"max":{"1min":0.625,"5min":0.80700000000000005,"15min":0.93600000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78500000000000003,"15min":0.90500000000000003},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.59499999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.318,"15min":0.318},"max":{"1min":0.61399999999999999,"5min":0.70799999999999996,"15min":0.77700000000000002},"last":0.628}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.374,"15min":0.46400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.40600000000000003,"5min":0.874,"15min":0.874},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.44500000000000001,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47799999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.29999999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.371,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.39200000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.316},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41699999999999998,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.47299999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.23100000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.503},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.45900000000000002,"5min":0.78900000000000003,"15min":0.89600000000000002},"last":0.254},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.42499999999999999,"15min":0.50900000000000001},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":0.42599999999999999,"5min":0.874,"15min":0.874},"last":0.27100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575006,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33876,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337964,"statfs":{"total":96636764160,"available":96602075136,"internally_reserved":0,"allocated":2760704,"data_stored":1344678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.58199999999999996,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.878,"15min":0.878},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.35699999999999998,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.81000000000000005,"15min":0.872},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49299999999999999,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64600000000000002,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.81299999999999994,"15min":0.81699999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.56499999999999995,"5min":0.79400000000000004,"15min":0.85899999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66200000000000003,"5min":0.82799999999999996,"15min":0.83799999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.503,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.75600000000000001,"15min":0.83899999999999997},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35399999999999998,"15min":0.45300000000000001},"min":{"1min":0.252,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":0.41099999999999998,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.311},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.249},"max":{"1min":0.51600000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.40000000000000002,"15min":0.49199999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.44,"5min":0.85999999999999999,"15min":0.877},"last":0.378},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.33500000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42699999999999999,"15min":0.52500000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.57999999999999996,"5min":0.876,"15min":0.90000000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45600000000000002,"15min":0.55600000000000005},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52900000000000003,"5min":0.96299999999999997,"15min":1.03},"last":0.27100000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575006,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33880,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2764800,"data_stored":1346198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.505,"15min":0.53300000000000003},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70799999999999996,"5min":0.79200000000000004,"15min":4.6879999999999997},"last":0.497},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":1.341},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48399999999999999,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.53700000000000003,"5min":0.83399999999999996,"15min":2.6549999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48999999999999999,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":0.64900000000000002,"5min":0.82899999999999996,"15min":4.7290000000000001},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":2.5800000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.48999999999999999,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.56899999999999995,"5min":0.79100000000000004,"15min":10.629},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":1.2050000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84099999999999997,"15min":2.7010000000000001},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36499999999999999,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.40600000000000003,"5min":0.76400000000000001,"15min":6.6989999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38600000000000001,"15min":0.47099999999999997},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40000000000000002,"5min":0.77000000000000002,"15min":1.4470000000000001},"last":0.36899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38600000000000001,"15min":0.495},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.36399999999999999,"5min":0.85199999999999998,"15min":2.7349999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41499999999999998,"15min":0.52900000000000003},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.41499999999999998,"5min":0.83599999999999997,"15min":4.6239999999999997},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.41299999999999998,"15min":0.51000000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.41799999999999998,"5min":0.80300000000000005,"15min":1.246},"last":0.44},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.441,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46999999999999997,"5min":0.81999999999999995,"15min":2.621},"last":0.52300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575006,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33880,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337960,"statfs":{"total":96636764160,"available":96602071040,"internally_reserved":0,"allocated":2764800,"data_stored":1345348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.32700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.159},"max":{"1min":0.441,"5min":0.67900000000000005,"15min":3.694},"last":0.308},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":4.835},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.18099999999999999},"max":{"1min":0.58799999999999997,"5min":0.64200000000000002,"15min":0.76100000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.20000000000000001},"max":{"1min":0.501,"5min":0.66400000000000003,"15min":6.3810000000000002},"last":0.375}]},{"osd":2,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.40200000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":3.6320000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.372},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57799999999999996,"5min":0.68300000000000005,"15min":4.9889999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48499999999999999,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.27900000000000003,"15min":0.26600000000000001},"max":{"1min":0.58199999999999996,"5min":0.874,"15min":0.874},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60499999999999998,"5min":0.86199999999999999,"15min":4.9539999999999997},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53100000000000003,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63800000000000001,"5min":0.85699999999999998,"15min":3.6669999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.65900000000000003,"5min":0.88600000000000001,"15min":4.9130000000000003},"last":0.436}]},{"osd":6,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.80700000000000005,"15min":0.86799999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67800000000000005,"5min":0.82999999999999996,"15min":3.7189999999999999},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30499999999999999},"max":{"1min":0.65000000000000002,"5min":0.82199999999999995,"15min":4.8799999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.69299999999999995,"5min":0.871,"15min":3.0760000000000001},"last":0.48699999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575007,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33892,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337948,"statfs":{"total":96636764160,"available":96602058752,"internally_reserved":0,"allocated":2777088,"data_stored":1346892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34000000000000002,"15min":0.32700000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":0.52000000000000002,"5min":0.621,"15min":0.68899999999999995},"last":0.443},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.32800000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.57699999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.63,"5min":0.67200000000000004,"15min":1.512},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.35099999999999998},"min":{"1min":0.25600000000000001,"5min":0.22900000000000001,"15min":0.19700000000000001},"max":{"1min":0.52900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.38900000000000001},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":0.59099999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61299999999999999,"5min":0.84599999999999997,"15min":1.133},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51100000000000001,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61199999999999999,"5min":0.748,"15min":1.1539999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50700000000000001,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59699999999999998,"5min":0.78700000000000003,"15min":0.94999999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52700000000000002,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.627}]},{"osd":6,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.60299999999999998,"5min":0.82999999999999996,"15min":1.115},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":1.236},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.60299999999999998,"5min":0.81299999999999994,"15min":1.1830000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54300000000000004,"15min":0.56499999999999995},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.623,"5min":0.85799999999999998,"15min":0.995},"last":0.66100000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575007,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33896,"kb_used_data":2716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337944,"statfs":{"total":96636764160,"available":96602054656,"internally_reserved":0,"allocated":2781184,"data_stored":1346026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.80600000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.223,"5min":0.218,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.90400000000000003},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.56100000000000005,"5min":0.624,"15min":0.77400000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.622,"5min":0.70199999999999996,"15min":0.76000000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.41399999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.72199999999999998},"last":0.371},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42599999999999999,"15min":0.39500000000000002},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.93300000000000005},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59099999999999997,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.314},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.88200000000000001,"15min":1.0009999999999999},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.77300000000000002,"15min":1.0189999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.53800000000000003},"min":{"1min":0.36299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.625,"5min":0.85599999999999998,"15min":1.0229999999999999},"last":0.39100000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.53700000000000003,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.31},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":1.026},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.307},"max":{"1min":0.65400000000000003,"5min":0.81399999999999995,"15min":0.996},"last":0.41799999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79200000000000004,"5min":0.90800000000000003,"15min":1.036},"last":0.499},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.83399999999999996,"15min":1.0740000000000001},"last":0.48099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34456,"kb_used_data":3276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337384,"statfs":{"total":96636764160,"available":96601481216,"internally_reserved":0,"allocated":3354624,"data_stored":1935114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.33700000000000002},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.17399999999999999},"max":{"1min":0.50900000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38300000000000001,"15min":0.35299999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.48399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52300000000000002,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.56699999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52900000000000003,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.35999999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.32400000000000001,"15min":0.28399999999999997},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53300000000000003,"15min":0.55500000000000005},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.442}]},{"osd":6,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45800000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.58299999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:04:44.018 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-19T15:04:44.018 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.115+0000 7f57b1012640 1 Processor -- start 2024-02-19T15:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57b1012640 1 -- start start 2024-02-19T15:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57b1012640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 0x7f57ac0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57b1012640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ac152d30 con 0x7f57ac0c49c0 2024-02-19T15:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57b1012640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ac0c7860 con 0x7f57ac0ab600 2024-02-19T15:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57b1012640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac0aba90 con 0x7f57ac0ad890 2024-02-19T15:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57aa59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f57ac0c49c0 0x7f57ac1534d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50638/0 (socket says 172.21.15.78:50638) 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.116+0000 7f57aa59c640 1 -- 172.21.15.78:0/2491438503 learned_addr learned my addr 172.21.15.78:0/2491438503 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124986912 0 0) 0x7f57ac152d30 con 0x7f57ac0c49c0 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aad9d640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 0x7f57ac0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5794003520 con 0x7f57ac0c49c0 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887094645 0 0) 0x7f5794003520 con 0x7f57ac0c49c0 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 msgr2=0x7f57ac0c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 0x7f57ac0c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 >> v1:172.21.15.150:6789/0 conn(0x7f57ac0ab600 legacy=0x7f57ac0c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac0c7ce0 con 0x7f57ac0c49c0 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f579c001690 con 0x7f57ac0c49c0 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 >> v1:172.21.15.78:6789/0 conn(0x7f57ac0c49c0 legacy=0x7f57ac1534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 0x7f57940039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794004d00 0x7f5794005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac0aba90 con 0x7f5794003f20 2024-02-19T15:04:44.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579400bbc0 con 0x7f5794004d00 2024-02-19T15:04:44.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579400bd80 con 0x7f57ac0ad890 2024-02-19T15:04:44.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aa59c640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aad9d640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 0x7f57940039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57ab59e640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794004d00 0x7f5794005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aa59c640 1 -- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 msgr2=0x7f57940039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aa59c640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0ad890 0x7f57940039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aa59c640 1 -- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794004d00 msgr2=0x7f5794005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aa59c640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794004d00 0x7f5794005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.117+0000 7f57aa59c640 1 -- 172.21.15.78:0/2491438503 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794003520 con 0x7f5794003f20 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.118+0000 7f57aa59c640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57940047c0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f579c00c210 tx=0x7f57940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5854412a25b4f6fd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.118+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c005ad0 con 0x7f5794003f20 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.118+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c005c30 con 0x7f5794003f20 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.118+0000 7f57a9d9b640 1 -- 172.21.15.78:0/2491438503 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c02c870 con 0x7f5794003f20 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57b1012640 1 -- 172.21.15.78:0/2491438503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 msgr2=0x7f57940047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57b1012640 1 --2- 172.21.15.78:0/2491438503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57940047c0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f579c00c210 tx=0x7f57940057f0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57ab59e640 1 -- 172.21.15.78:0/2491438503 reap_dead start 2024-02-19T15:04:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57b1012640 1 -- 172.21.15.78:0/2491438503 shutdown_connections 2024-02-19T15:04:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57b1012640 1 -- 172.21.15.78:0/2491438503 >> 172.21.15.78:0/2491438503 conn(0x7f57ac06b700 msgr2=0x7f57ac0aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57b1012640 1 -- 172.21.15.78:0/2491438503 shutdown_connections 2024-02-19T15:04:44.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.119+0000 7f57b1012640 1 -- 172.21.15.78:0/2491438503 wait complete. 2024-02-19T15:04:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.120+0000 7f57b1012640 1 Processor -- start 2024-02-19T15:04:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.120+0000 7f57b1012640 1 -- start start 2024-02-19T15:04:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57ac1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57ac14bf80 0x7f57ac1ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0bd570 0x7f57ac14b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac1ecba0 con 0x7f5794003f20 2024-02-19T15:04:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac156eb0 con 0x7f57ac14bf80 2024-02-19T15:04:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ac154670 con 0x7f57ac0bd570 2024-02-19T15:04:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57ac1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57ac1551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46388/0 (socket says 172.21.15.78:46388) 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 -- 172.21.15.78:0/3219483873 learned_addr learned my addr 172.21.15.78:0/3219483873 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57ab59e640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0bd570 0x7f57ac14b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aa59c640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57ac14bf80 0x7f57ac1ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 -- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0bd570 msgr2=0x7f57ac14b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0bd570 0x7f57ac14b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 -- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57ac14bf80 msgr2=0x7f57ac1ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57ac14bf80 0x7f57ac1ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 -- 172.21.15.78:0/3219483873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac0be020 con 0x7f5794003f20 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57ab59e640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0bd570 0x7f57ac14b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57aad9d640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57ac1551b0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f57a0011600 tx=0x7f57a0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a000f920 con 0x7f5794003f20 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a000fa80 con 0x7f5794003f20 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ac156ad0 con 0x7f5794003f20 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.121+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a000fdf0 con 0x7f5794003f20 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.122+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ac0a8a20 con 0x7f5794003f20 2024-02-19T15:04:44.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.123+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ac156c60 con 0x7f5794003f20 2024-02-19T15:04:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.123+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f57a0015460 con 0x7f5794003f20 2024-02-19T15:04:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.124+0000 7f57937fe640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f579806b810 0x7f579806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.124+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f57a008a7d0 con 0x7f5794003f20 2024-02-19T15:04:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.124+0000 7f57aa59c640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f579806b810 0x7f579806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.125+0000 7f57aa59c640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f579806b810 0x7f579806dcd0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f579c034000 tx=0x7f579c037040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:44.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.129+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57a00552f0 con 0x7f5794003f20 2024-02-19T15:04:44.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.299+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57ac0c8260 con 0x7f579806b810 2024-02-19T15:04:44.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.301+0000 7f57937fe640 1 -- 172.21.15.78:0/3219483873 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50860 (secure 0 0 0) 0x7f579c0010e0 con 0x7f579806b810 2024-02-19T15:04:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:44.304 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:04:44.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f579806b810 msgr2=0x7f579806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f579806b810 0x7f579806dcd0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f579c034000 tx=0x7f579c037040 comp rx=0 tx=0).stop 2024-02-19T15:04:44.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 msgr2=0x7f57ac1551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57ac1551b0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f57a0011600 tx=0x7f57a0011ac0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 shutdown_connections 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f579806b810 0x7f579806dcd0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57ac0bd570 0x7f57ac14b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57ac14bf80 0x7f57ac1ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 --2- 172.21.15.78:0/3219483873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794003f20 0x7f57ac1551b0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 >> 172.21.15.78:0/3219483873 conn(0x7f57ac06b700 msgr2=0x7f57ac06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 shutdown_connections 2024-02-19T15:04:44.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.303+0000 7f57b1012640 1 -- 172.21.15.78:0/3219483873 wait complete. 2024-02-19T15:04:44.316 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2024-02-19T15:04:44.218372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":272480,"kb_used_data":23016,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754702240,"statfs":{"total":773094113280,"available":772815093760,"internally_reserved":0,"allocated":23568384,"data_stored":12096704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-52,"num_read_kb":-58,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001075"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":545,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-02-19T15:04:42.116479+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:04:42.116479+0000","last_peered":"2024-02-19T15:04:42.116479+0000","last_clean":"2024-02-19T15:04:42.116479+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:04:42.116479+0000","last_undegraded":"2024-02-19T15:04:42.116479+0000","last_fullsized":"2024-02-19T15:04:42.116479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34496,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337344,"statfs":{"total":96636764160,"available":96601440256,"internally_reserved":0,"allocated":3391488,"data_stored":1954864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.65100000000000002,"5min":0.747,"15min":0.747},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.79500000000000004},"last":0.502}]},{"osd":1,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.40999999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.66900000000000004,"5min":0.71999999999999997,"15min":0.83099999999999996},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59499999999999997,"5min":0.70899999999999996,"15min":0.76400000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47999999999999998,"15min":0.49099999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.627,"5min":0.76000000000000001,"15min":0.80000000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.34799999999999998,"5min":0.30399999999999999,"15min":0.26500000000000001},"max":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.77500000000000002},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.72399999999999998,"15min":0.84199999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.61499999999999999,"5min":0.76900000000000002,"15min":0.81100000000000005},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.36499999999999999,"15min":0.42299999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.432,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.436},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.38200000000000001,"15min":0.45600000000000002},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.44,"5min":0.80400000000000005,"15min":0.86899999999999999},"last":0.58699999999999997}]},{"osd":5,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.39500000000000002,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41299999999999998,"15min":0.49399999999999999},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.45600000000000002,"5min":0.76000000000000001,"15min":0.871},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51800000000000002,"5min":0.79400000000000004,"15min":0.90200000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.44600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.48399999999999999,"5min":0.78000000000000003,"15min":0.90500000000000003},"last":0.55100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.22700000000000001},"max":{"1min":0.50900000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.502,"15min":0.499},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.78100000000000003,"15min":0.95699999999999996},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.311},"max":{"1min":0.625,"5min":0.80700000000000005,"15min":0.93600000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78500000000000003,"15min":0.90500000000000003},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.59499999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.318,"15min":0.318},"max":{"1min":0.61399999999999999,"5min":0.70799999999999996,"15min":0.77700000000000002},"last":0.34999999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.374,"15min":0.46400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.40600000000000003,"5min":0.874,"15min":0.874},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.44500000000000001,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47799999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.376}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.371,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.39200000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.316},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41699999999999998,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.47299999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.29399999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.503},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.45900000000000002,"5min":0.78900000000000003,"15min":0.89600000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.42499999999999999,"15min":0.50900000000000001},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":0.42599999999999999,"5min":0.874,"15min":0.874},"last":0.27700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575007,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.58199999999999996,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.878,"15min":0.878},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.35699999999999998,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.81000000000000005,"15min":0.872},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49299999999999999,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64600000000000002,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.628}]},{"osd":2,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.81299999999999994,"15min":0.81699999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.56499999999999995,"5min":0.79400000000000004,"15min":0.85899999999999999},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66200000000000003,"5min":0.82799999999999996,"15min":0.83799999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.503,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.75600000000000001,"15min":0.83899999999999997},"last":0.498}]},{"osd":4,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35399999999999998,"15min":0.45300000000000001},"min":{"1min":0.252,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":0.41099999999999998,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.249},"max":{"1min":0.51600000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64700000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.40000000000000002,"15min":0.49199999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.44,"5min":0.85999999999999999,"15min":0.877},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.437}]},{"osd":7,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42699999999999999,"15min":0.52500000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.57999999999999996,"5min":0.876,"15min":0.90000000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45600000000000002,"15min":0.55600000000000005},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52900000000000003,"5min":0.96299999999999997,"15min":1.03},"last":0.60299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.505,"15min":0.53300000000000003},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70799999999999996,"5min":0.79200000000000004,"15min":4.6879999999999997},"last":0.629},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":1.341},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48399999999999999,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.53700000000000003,"5min":0.83399999999999996,"15min":2.6549999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48999999999999999,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":0.64900000000000002,"5min":0.82899999999999996,"15min":4.7290000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":2.5800000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.48999999999999999,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.56899999999999995,"5min":0.79100000000000004,"15min":10.629},"last":0.57799999999999996}]},{"osd":3,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":1.2050000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84099999999999997,"15min":2.7010000000000001},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36499999999999999,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.40600000000000003,"5min":0.76400000000000001,"15min":6.6989999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38600000000000001,"15min":0.47099999999999997},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40000000000000002,"5min":0.77000000000000002,"15min":1.4470000000000001},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38600000000000001,"15min":0.495},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.36399999999999999,"5min":0.85199999999999998,"15min":2.7349999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41499999999999998,"15min":0.52900000000000003},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.41499999999999998,"5min":0.83599999999999997,"15min":4.6239999999999997},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.41299999999999998,"15min":0.51000000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.41799999999999998,"5min":0.80300000000000005,"15min":1.246},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.441,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46999999999999997,"5min":0.81999999999999995,"15min":2.621},"last":0.65800000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575007,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.32700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.159},"max":{"1min":0.441,"5min":0.67900000000000005,"15min":3.694},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":4.835},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.18099999999999999},"max":{"1min":0.58799999999999997,"5min":0.64200000000000002,"15min":0.76100000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.20000000000000001},"max":{"1min":0.501,"5min":0.66400000000000003,"15min":6.3810000000000002},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.40200000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":3.6320000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.372},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57799999999999996,"5min":0.68300000000000005,"15min":4.9889999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48499999999999999,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.27900000000000003,"15min":0.26600000000000001},"max":{"1min":0.58199999999999996,"5min":0.874,"15min":0.874},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60499999999999998,"5min":0.86199999999999999,"15min":4.9539999999999997},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53100000000000003,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63800000000000001,"5min":0.85699999999999998,"15min":3.6669999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.65900000000000003,"5min":0.88600000000000001,"15min":4.9130000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.80700000000000005,"15min":0.86799999999999999},"last":0.495},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67800000000000005,"5min":0.82999999999999996,"15min":3.7189999999999999},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30499999999999999},"max":{"1min":0.65000000000000002,"5min":0.82199999999999995,"15min":4.8799999999999999},"last":0.432},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.69299999999999995,"5min":0.871,"15min":3.0760000000000001},"last":0.56100000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34000000000000002,"15min":0.32700000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":0.52000000000000002,"5min":0.621,"15min":0.68899999999999995},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.32800000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.57699999999999996},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.63,"5min":0.67200000000000004,"15min":1.512},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.35099999999999998},"min":{"1min":0.25600000000000001,"5min":0.22900000000000001,"15min":0.19700000000000001},"max":{"1min":0.52900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.38900000000000001},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":0.59099999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61299999999999999,"5min":0.84599999999999997,"15min":1.133},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51100000000000001,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61199999999999999,"5min":0.748,"15min":1.1539999999999999},"last":0.625}]},{"osd":5,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50700000000000001,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59699999999999998,"5min":0.78700000000000003,"15min":0.94999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52700000000000002,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.60299999999999998,"5min":0.82999999999999996,"15min":1.115},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":1.236},"last":0.55000000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.60299999999999998,"5min":0.81299999999999994,"15min":1.1830000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54300000000000004,"15min":0.56499999999999995},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.623,"5min":0.85799999999999998,"15min":0.995},"last":0.61299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.80600000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.223,"5min":0.218,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.90400000000000003},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.56100000000000005,"5min":0.624,"15min":0.77400000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.622,"5min":0.70199999999999996,"15min":0.76000000000000001},"last":0.44}]},{"osd":3,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.41399999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.72199999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42599999999999999,"15min":0.39500000000000002},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.93300000000000005},"last":0.34599999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59099999999999997,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.88200000000000001,"15min":1.0009999999999999},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.77300000000000002,"15min":1.0189999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.53800000000000003},"min":{"1min":0.36299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.625,"5min":0.85599999999999998,"15min":1.0229999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.53700000000000003,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.31},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":1.026},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.307},"max":{"1min":0.65400000000000003,"5min":0.81399999999999995,"15min":0.996},"last":0.77500000000000002}]},{"osd":7,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79200000000000004,"5min":0.90800000000000003,"15min":1.036},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.83399999999999996,"15min":1.0740000000000001},"last":0.67000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575007,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34496,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337344,"statfs":{"total":96636764160,"available":96601440256,"internally_reserved":0,"allocated":3391488,"data_stored":1954864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.33700000000000002},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.17399999999999999},"max":{"1min":0.50900000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.27900000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38300000000000001,"15min":0.35299999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.48399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52300000000000002,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.56699999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52900000000000003,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63500000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.32400000000000001,"15min":0.28399999999999997},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.53300000000000003,"15min":0.55500000000000005},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.504},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.624}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.58299999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.64700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:04:44.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T15:04:44.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.414+0000 7f2badaed640 1 Processor -- start 2024-02-19T15:04:44.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.414+0000 7f2badaed640 1 -- start start 2024-02-19T15:04:44.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2badaed640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ba8071e50 0x7f2ba80c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2badaed640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80c1120 con 0x7f2ba814ece0 2024-02-19T15:04:44.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2badaed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80c6080 con 0x7f2ba80719d0 2024-02-19T15:04:44.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2badaed640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba80722a0 con 0x7f2ba8071e50 2024-02-19T15:04:44.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2ba77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2ba814ece0 0x7f2ba80c4d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50640/0 (socket says 172.21.15.78:50640) 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2ba77fe640 1 -- 172.21.15.78:0/1388345165 learned_addr learned my addr 172.21.15.78:0/1388345165 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2ba7fff640 1 --2- 172.21.15.78:0/1388345165 >> v2:172.21.15.78:3300/0 conn(0x7f2ba8071e50 0x7f2ba80c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.415+0000 7f2ba7fff640 1 -- 172.21.15.78:0/1388345165 >> v1:172.21.15.150:6789/0 conn(0x7f2ba80719d0 legacy=0x7f2ba80c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba7fff640 1 -- 172.21.15.78:0/1388345165 >> v1:172.21.15.78:6789/0 conn(0x7f2ba814ece0 legacy=0x7f2ba80c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba7fff640 1 -- 172.21.15.78:0/1388345165 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80c57e0 con 0x7f2ba8071e50 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba67fc640 1 -- 172.21.15.78:0/1388345165 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364601706 0 0) 0x7f2ba80c1120 con 0x7f2ba814ece0 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba7fff640 1 --2- 172.21.15.78:0/1388345165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8071e50 0x7f2ba80c1cc0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009d90 tx=0x7f2ba002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b2b298390ec023 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:44.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba67fc640 1 -- 172.21.15.78:0/1388345165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba002cc60 con 0x7f2ba8071e50 2024-02-19T15:04:44.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba67fc640 1 -- 172.21.15.78:0/1388345165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba002cdc0 con 0x7f2ba8071e50 2024-02-19T15:04:44.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.416+0000 7f2ba67fc640 1 -- 172.21.15.78:0/1388345165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0034660 con 0x7f2ba8071e50 2024-02-19T15:04:44.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 -- 172.21.15.78:0/1388345165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8071e50 msgr2=0x7f2ba80c1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 --2- 172.21.15.78:0/1388345165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8071e50 0x7f2ba80c1cc0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009d90 tx=0x7f2ba002beb0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 -- 172.21.15.78:0/1388345165 shutdown_connections 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 --2- 172.21.15.78:0/1388345165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8071e50 0x7f2ba80c1cc0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 -- 172.21.15.78:0/1388345165 >> 172.21.15.78:0/1388345165 conn(0x7f2ba806b270 msgr2=0x7f2ba806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 -- 172.21.15.78:0/1388345165 shutdown_connections 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.417+0000 7f2badaed640 1 -- 172.21.15.78:0/1388345165 wait complete. 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.418+0000 7f2badaed640 1 Processor -- start 2024-02-19T15:04:44.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.418+0000 7f2badaed640 1 -- start start 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2badaed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 0x7f2ba8149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2badaed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ba8071e50 0x7f2ba81db5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2badaed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ba814ece0 0x7f2ba81e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2badaed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba80bec70 con 0x7f2ba80719d0 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2badaed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba8152ba0 con 0x7f2ba814ece0 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2badaed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba814a0d0 con 0x7f2ba8071e50 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 0x7f2ba8149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 0x7f2ba8149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46410/0 (socket says 172.21.15.78:46410) 2024-02-19T15:04:44.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 -- 172.21.15.78:0/3115839008 learned_addr learned my addr 172.21.15.78:0/3115839008 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba7fff640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ba814ece0 0x7f2ba81e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba6ffd640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ba8071e50 0x7f2ba81db5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 -- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ba8071e50 msgr2=0x7f2ba81db5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ba8071e50 0x7f2ba81db5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 -- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ba814ece0 msgr2=0x7f2ba81e9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ba814ece0 0x7f2ba81e9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 -- 172.21.15.78:0/3115839008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba0009a70 con 0x7f2ba80719d0 2024-02-19T15:04:44.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba7fff640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ba814ece0 0x7f2ba81e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.419+0000 7f2ba77fe640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 0x7f2ba8149b90 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800c4d0 tx=0x7f2b9800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.420+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980074e0 con 0x7f2ba80719d0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.420+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b98007660 con 0x7f2ba80719d0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.420+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba81528c0 con 0x7f2ba80719d0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.420+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98004a20 con 0x7f2ba80719d0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.420+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba81e6ad0 con 0x7f2ba80719d0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.421+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba81e6620 con 0x7f2ba80719d0 2024-02-19T15:04:44.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.421+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2b98004b80 con 0x7f2ba80719d0 2024-02-19T15:04:44.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.421+0000 7f2b8ffff640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b9406b810 0x7f2b9406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:44.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.421+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2b980895e0 con 0x7f2ba80719d0 2024-02-19T15:04:44.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.421+0000 7f2ba6ffd640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b9406b810 0x7f2b9406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:44.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.422+0000 7f2ba6ffd640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b9406b810 0x7f2b9406dcd0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006fd0 tx=0x7f2b9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:44.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.426+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2b98054100 con 0x7f2ba80719d0 2024-02-19T15:04:44.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.598+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ba80c4170 con 0x7f2ba80719d0 2024-02-19T15:04:44.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.599+0000 7f2b8ffff640 1 -- 172.21.15.78:0/3115839008 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+13683 (secure 0 0 0) 0x7f2b980577c0 con 0x7f2ba80719d0 2024-02-19T15:04:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:44.602 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":220,"fsid":"7cde3b0a-2943-4d9b-b265-05e609d5a6b3","created":"2024-02-19T14:51:35.818531+0000","modified":"2024-02-19T15:04:41.722287+0000","last_up_change":"2024-02-19T14:51:47.110557+0000","last_in_change":"2024-02-19T14:51:38.054002+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T14:51:48.105873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfeee3e6-366d-43a1-ae94-10a089e7e8c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6817","nonce":2796609917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6819","nonce":2796609917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6823","nonce":2796609917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2796609917},{"type":"v1","addr":"172.21.15.78:6821","nonce":2796609917}]},"public_addr":"172.21.15.78:6817/2796609917","cluster_addr":"172.21.15.78:6819/2796609917","heartbeat_back_addr":"172.21.15.78:6823/2796609917","heartbeat_front_addr":"172.21.15.78:6821/2796609917","state":["exists","up"]},{"osd":1,"uuid":"eb08ce5d-c579-413d-8058-91c426ab9dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6804","nonce":289106328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6808","nonce":289106328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6815","nonce":289106328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":289106328},{"type":"v1","addr":"172.21.15.78:6812","nonce":289106328}]},"public_addr":"172.21.15.78:6804/289106328","cluster_addr":"172.21.15.78:6808/289106328","heartbeat_back_addr":"172.21.15.78:6815/289106328","heartbeat_front_addr":"172.21.15.78:6812/289106328","state":["exists","up"]},{"osd":2,"uuid":"068ab687-b58f-475b-9b39-e16546079e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6801","nonce":3664023701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6805","nonce":3664023701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6813","nonce":3664023701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3664023701},{"type":"v1","addr":"172.21.15.78:6809","nonce":3664023701}]},"public_addr":"172.21.15.78:6801/3664023701","cluster_addr":"172.21.15.78:6805/3664023701","heartbeat_back_addr":"172.21.15.78:6813/3664023701","heartbeat_front_addr":"172.21.15.78:6809/3664023701","state":["exists","up"]},{"osd":3,"uuid":"043d280c-df76-4b38-b704-cb2fc622c5b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6825","nonce":3716944111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6827","nonce":3716944111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6831","nonce":3716944111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":3716944111},{"type":"v1","addr":"172.21.15.78:6829","nonce":3716944111}]},"public_addr":"172.21.15.78:6825/3716944111","cluster_addr":"172.21.15.78:6827/3716944111","heartbeat_back_addr":"172.21.15.78:6831/3716944111","heartbeat_front_addr":"172.21.15.78:6829/3716944111","state":["exists","up"]},{"osd":4,"uuid":"ca904970-a7f2-4b96-bab2-fc2dc65022ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6817","nonce":1744914063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6820","nonce":1744914063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6828","nonce":1744914063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1744914063},{"type":"v1","addr":"172.21.15.150:6824","nonce":1744914063}]},"public_addr":"172.21.15.150:6817/1744914063","cluster_addr":"172.21.15.150:6820/1744914063","heartbeat_back_addr":"172.21.15.150:6828/1744914063","heartbeat_front_addr":"172.21.15.150:6824/1744914063","state":["exists","up"]},{"osd":5,"uuid":"294f6374-334d-4b72-8374-40cf962e05f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6801","nonce":911852656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6803","nonce":911852656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6807","nonce":911852656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":911852656},{"type":"v1","addr":"172.21.15.150:6805","nonce":911852656}]},"public_addr":"172.21.15.150:6801/911852656","cluster_addr":"172.21.15.150:6803/911852656","heartbeat_back_addr":"172.21.15.150:6807/911852656","heartbeat_front_addr":"172.21.15.150:6805/911852656","state":["exists","up"]},{"osd":6,"uuid":"32d93613-4035-426d-8554-d3b5dddd289d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6809","nonce":3532732509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6811","nonce":3532732509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6815","nonce":3532732509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3532732509},{"type":"v1","addr":"172.21.15.150:6813","nonce":3532732509}]},"public_addr":"172.21.15.150:6809/3532732509","cluster_addr":"172.21.15.150:6811/3532732509","heartbeat_back_addr":"172.21.15.150:6815/3532732509","heartbeat_front_addr":"172.21.15.150:6813/3532732509","state":["exists","up"]},{"osd":7,"uuid":"7115141e-af2d-4ee8-84bf-8876f91b641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6821","nonce":1386990918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6825","nonce":1386990918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6831","nonce":1386990918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":1386990918},{"type":"v1","addr":"172.21.15.150:6829","nonce":1386990918}]},"public_addr":"172.21.15.150:6821/1386990918","cluster_addr":"172.21.15.150:6825/1386990918","heartbeat_back_addr":"172.21.15.150:6831/1386990918","heartbeat_front_addr":"172.21.15.150:6829/1386990918","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.223853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.248505+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.247781+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.203320+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.215457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.178622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.254067+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T14:51:43.239809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/1952320195":"2024-02-20T15:04:38.714272+0000","172.21.15.150:6833/1952320195":"2024-02-20T15:04:38.714272+0000","172.21.15.150:6832/4270793546":"2024-02-20T15:04:38.714259+0000","172.21.15.150:6834/4270793546":"2024-02-20T15:04:38.714259+0000","172.21.15.78:6835/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6834/3736272032":"2024-02-20T15:02:05.637692+0000","172.21.15.78:6837/159409097":"2024-02-20T15:01:30.632749+0000","172.21.15.78:6836/159409097":"2024-02-20T15:01:30.632749+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T15:04:44.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b9406b810 msgr2=0x7f2b9406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b9406b810 0x7f2b9406dcd0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c006fd0 tx=0x7f2b9c008040 comp rx=0 tx=0).stop 2024-02-19T15:04:44.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 msgr2=0x7f2ba8149b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:44.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 0x7f2ba8149b90 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800c4d0 tx=0x7f2b9800c990 comp rx=0 tx=0).stop 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 shutdown_connections 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2b9406b810 0x7f2b9406dcd0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ba814ece0 0x7f2ba81e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ba8071e50 0x7f2ba81db5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 --2- 172.21.15.78:0/3115839008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba80719d0 0x7f2ba8149b90 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 >> 172.21.15.78:0/3115839008 conn(0x7f2ba806b270 msgr2=0x7f2ba814da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 shutdown_connections 2024-02-19T15:04:44.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:44.601+0000 7f2badaed640 1 -- 172.21.15.78:0/3115839008 wait complete. 2024-02-19T15:04:45.615 INFO:tasks.ceph:Scrubbing osd.0 2024-02-19T15:04:45.616 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:45.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.727+0000 7f3a81b93640 1 Processor -- start 2024-02-19T15:04:45.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a81b93640 1 -- start start 2024-02-19T15:04:45.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a81b93640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3a7c152620 0x7f3a7c0c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a81b93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a7c155010 0x7f3a7c155460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a81b93640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c159210 con 0x7f3a7c0c6d50 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a81b93640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c0c1f00 con 0x7f3a7c155010 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a81b93640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c0c6420 con 0x7f3a7c152620 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a7bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a7c155010 0x7f3a7c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a7bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a7c155010 0x7f3a7c155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46420/0 (socket says 172.21.15.78:46420) 2024-02-19T15:04:45.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.728+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1441993567 learned_addr learned my addr 172.21.15.78:0/1441993567 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7b7fe640 1 --2- 172.21.15.78:0/1441993567 >> v2:172.21.15.150:3300/0 conn(0x7f3a7c152620 0x7f3a7c0c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1441993567 >> v2:172.21.15.150:3300/0 conn(0x7f3a7c152620 msgr2=0x7f3a7c0c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7bfff640 1 --2- 172.21.15.78:0/1441993567 >> v2:172.21.15.150:3300/0 conn(0x7f3a7c152620 0x7f3a7c0c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1441993567 >> v1:172.21.15.78:6789/0 conn(0x7f3a7c0c6d50 legacy=0x7f3a7c06d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1441993567 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c158600 con 0x7f3a7c155010 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7affd640 1 -- 172.21.15.78:0/1441993567 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112255197 0 0) 0x7f3a7c159210 con 0x7f3a7c0c6d50 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.729+0000 7f3a7bfff640 1 --2- 172.21.15.78:0/1441993567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c155460 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c08c5b0 tx=0x7f3a7402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95632dc8ef44420 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a7affd640 1 -- 172.21.15.78:0/1441993567 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7402c9f0 con 0x7f3a7c155010 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a7affd640 1 -- 172.21.15.78:0/1441993567 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7402cb50 con 0x7f3a7c155010 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a7affd640 1 -- 172.21.15.78:0/1441993567 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a740343f0 con 0x7f3a7c155010 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 -- 172.21.15.78:0/1441993567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 msgr2=0x7f3a7c155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 --2- 172.21.15.78:0/1441993567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c155460 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c08c5b0 tx=0x7f3a7402be50 comp rx=0 tx=0).stop 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 -- 172.21.15.78:0/1441993567 shutdown_connections 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 --2- 172.21.15.78:0/1441993567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c155460 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 --2- 172.21.15.78:0/1441993567 >> v2:172.21.15.150:3300/0 conn(0x7f3a7c152620 0x7f3a7c0c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 -- 172.21.15.78:0/1441993567 >> 172.21.15.78:0/1441993567 conn(0x7f3a7c0caae0 msgr2=0x7f3a7c0caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:45.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 -- 172.21.15.78:0/1441993567 shutdown_connections 2024-02-19T15:04:45.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.730+0000 7f3a81b93640 1 -- 172.21.15.78:0/1441993567 wait complete. 2024-02-19T15:04:45.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.731+0000 7f3a81b93640 1 Processor -- start 2024-02-19T15:04:45.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 -- start start 2024-02-19T15:04:45.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a7c0c6d50 0x7f3a7c1ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a7c152620 0x7f3a7c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1de990 con 0x7f3a7c155010 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1579d0 con 0x7f3a7c0c6d50 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a81b93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c157d90 con 0x7f3a7c152620 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46426/0 (socket says 172.21.15.78:46426) 2024-02-19T15:04:45.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1088694718 learned_addr learned my addr 172.21.15.78:0/1088694718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.732+0000 7f3a7b7fe640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a7c0c6d50 0x7f3a7c1ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a72ffd640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a7c152620 0x7f3a7c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a7c152620 msgr2=0x7f3a7c156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a7bfff640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a7c152620 0x7f3a7c156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a7c0c6d50 msgr2=0x7f3a7c1ee120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a7bfff640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a7c0c6d50 0x7f3a7c1ee120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1088694718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74009b30 con 0x7f3a7c155010 2024-02-19T15:04:45.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a72ffd640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3a7c152620 0x7f3a7c156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a7bfff640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c157490 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f3a74036040 tx=0x7f3a74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7403e450 con 0x7f3a7c155010 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7403e5b0 con 0x7f3a7c155010 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c1ebdb0 con 0x7f3a7c155010 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7403e960 con 0x7f3a7c155010 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.733+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c1deba0 con 0x7f3a7c155010 2024-02-19T15:04:45.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.734+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a7c158ea0 con 0x7f3a7c155010 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f3a7403eac0 con 0x7f3a7c155010 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a78ff9640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a5c06b810 0x7f3a5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f3a740acec0 con 0x7f3a7c155010 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a7b7fe640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a5c06b810 0x7f3a5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a78ff9640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7f3a5c071e90 0x7f3a5c0742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 --> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a5c0749a0 con 0x7f3a5c071e90 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a740af580 con 0x7f3a7c155010 2024-02-19T15:04:45.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a72ffd640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7f3a5c071e90 0x7f3a5c0742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.735+0000 7f3a7b7fe640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a5c06b810 0x7f3a5c06dcd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0079a0 tx=0x7f3a6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.736+0000 7f3a72ffd640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7f3a5c071e90 0x7f3a5c0742d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.736+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== osd.0 v2:172.21.15.78:6816/2796609917 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a5c0749a0 con 0x7f3a5c071e90 2024-02-19T15:04:45.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.749+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 --> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3a7c158bc0 con 0x7f3a5c071e90 2024-02-19T15:04:45.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a78ff9640 1 -- 172.21.15.78:0/1088694718 <== osd.0 v2:172.21.15.78:6816/2796609917 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3a640050e0 con 0x7f3a5c071e90 2024-02-19T15:04:45.753 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:45.753 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:45.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7f3a5c071e90 msgr2=0x7f3a5c0742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6816/2796609917,v1:172.21.15.78:6817/2796609917] conn(0x7f3a5c071e90 0x7f3a5c0742d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a5c06b810 msgr2=0x7f3a5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f3a5c06b810 0x7f3a5c06dcd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0079a0 tx=0x7f3a6c008040 comp rx=0 tx=0).stop 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 msgr2=0x7f3a7c157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 --2- 172.21.15.78:0/1088694718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a7c155010 0x7f3a7c157490 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f3a74036040 tx=0x7f3a74004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a7bfff640 1 -- 172.21.15.78:0/1088694718 reap_dead start 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 shutdown_connections 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 >> 172.21.15.78:0/1088694718 conn(0x7f3a7c0caae0 msgr2=0x7f3a7c06d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 shutdown_connections 2024-02-19T15:04:45.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.750+0000 7f3a81b93640 1 -- 172.21.15.78:0/1088694718 wait complete. 2024-02-19T15:04:45.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-02-19T15:04:45.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.862+0000 7f15d5039640 1 Processor -- start 2024-02-19T15:04:45.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.862+0000 7f15d5039640 1 -- start start 2024-02-19T15:04:45.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15d5039640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f15d0152480 0x7f15d01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15d5039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15d00c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15d5039640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d0155ef0 con 0x7f15d00c7070 2024-02-19T15:04:45.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15d5039640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d006dec0 con 0x7f15d0152480 2024-02-19T15:04:45.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15d5039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d006d520 con 0x7f15d00c87a0 2024-02-19T15:04:45.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15ce59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f15d00c7070 0x7f15d006ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50668/0 (socket says 172.21.15.78:50668) 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15ce59c640 1 -- 172.21.15.78:0/1929521973 learned_addr learned my addr 172.21.15.78:0/1929521973 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cf59e640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15d00c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15ced9d640 1 --2- 172.21.15.78:0/1929521973 >> v2:172.21.15.150:3300/0 conn(0x7f15d0152480 0x7f15d01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917537305 0 0) 0x7f15b8001110 con 0x7f15d00c7070 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15bc003520 con 0x7f15d00c7070 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2838834652 0 0) 0x7f15bc003520 con 0x7f15d00c7070 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 msgr2=0x7f15d00c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:45.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15d00c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 >> v2:172.21.15.150:3300/0 conn(0x7f15d0152480 msgr2=0x7f15d01559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 --2- 172.21.15.78:0/1929521973 >> v2:172.21.15.150:3300/0 conn(0x7f15d0152480 0x7f15d01559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d00c75c0 con 0x7f15d00c7070 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f15b8001820 con 0x7f15d00c7070 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cf59e640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15d00c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15ced9d640 1 --2- 172.21.15.78:0/1929521973 >> v2:172.21.15.150:3300/0 conn(0x7f15d0152480 0x7f15d01559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 >> v1:172.21.15.78:6789/0 conn(0x7f15d00c7070 legacy=0x7f15d006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15bc003970 0x7f15bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0047c0 0x7f15bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.863+0000 7f15cdd9b640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d006dec0 con 0x7f15bc003970 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d006d520 con 0x7f15bc0047c0 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc00b980 con 0x7f15d00c87a0 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ced9d640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15bc003970 0x7f15bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ce59c640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0047c0 0x7f15bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15cf59e640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ced9d640 1 -- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 msgr2=0x7f15bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ced9d640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15d00c87a0 0x7f15bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ced9d640 1 -- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0047c0 msgr2=0x7f15bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ced9d640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0047c0 0x7f15bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.864+0000 7f15ced9d640 1 -- 172.21.15.78:0/1929521973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c0009400 con 0x7f15bc003970 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15ced9d640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15bc003970 0x7f15bc003d90 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f15c800e350 tx=0x7f15bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e2640cdaa60354 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c802ce00 con 0x7f15bc003970 2024-02-19T15:04:45.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c800f040 con 0x7f15bc003970 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15cdd9b640 1 -- 172.21.15.78:0/1929521973 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c8034640 con 0x7f15bc003970 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15d5039640 1 -- 172.21.15.78:0/1929521973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15bc003970 msgr2=0x7f15bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15d5039640 1 --2- 172.21.15.78:0/1929521973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15bc003970 0x7f15bc003d90 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f15c800e350 tx=0x7f15bc0057f0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15cf59e640 1 -- 172.21.15.78:0/1929521973 reap_dead start 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15d5039640 1 -- 172.21.15.78:0/1929521973 shutdown_connections 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15d5039640 1 -- 172.21.15.78:0/1929521973 >> 172.21.15.78:0/1929521973 conn(0x7f15d00ca8a0 msgr2=0x7f15d00cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15d5039640 1 -- 172.21.15.78:0/1929521973 shutdown_connections 2024-02-19T15:04:45.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.865+0000 7f15d5039640 1 -- 172.21.15.78:0/1929521973 wait complete. 2024-02-19T15:04:45.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 Processor -- start 2024-02-19T15:04:45.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 -- start start 2024-02-19T15:04:45.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15bc003970 0x7f15d00aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d00ac410 0x7f15d01ee170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 0x7f15d0149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00ad860 con 0x7f15d01493a0 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00abc50 con 0x7f15d00ac410 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15d5039640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d01eeab0 con 0x7f15bc003970 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15cf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 0x7f15d0149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15cf59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 0x7f15d0149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46454/0 (socket says 172.21.15.78:46454) 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15cf59e640 1 -- 172.21.15.78:0/3296404549 learned_addr learned my addr 172.21.15.78:0/3296404549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:45.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15ce59c640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d00ac410 0x7f15d01ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.867+0000 7f15ced9d640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15bc003970 0x7f15d00aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15cf59e640 1 -- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15bc003970 msgr2=0x7f15d00aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15cf59e640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15bc003970 0x7f15d00aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15cf59e640 1 -- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d00ac410 msgr2=0x7f15d01ee170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15cf59e640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d00ac410 0x7f15d01ee170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15cf59e640 1 -- 172.21.15.78:0/3296404549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d00abef0 con 0x7f15d01493a0 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15cf59e640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 0x7f15d0149820 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f15d00be3a0 tx=0x7f15c00119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0004440 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c0003610 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0003980 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d01eed90 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.868+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d014a380 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.869+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d00c17c0 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.869+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f15c0003ae0 con 0x7f15d01493a0 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.870+0000 7f15c77fe640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15b006b840 0x7f15b006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:45.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.870+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f15c0089f80 con 0x7f15d01493a0 2024-02-19T15:04:45.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.870+0000 7f15ced9d640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15b006b840 0x7f15b006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:45.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.870+0000 7f15ced9d640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15b006b840 0x7f15b006dd00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f15c8036000 tx=0x7f15c802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:45.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:45.874+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f15c0058540 con 0x7f15d01493a0 2024-02-19T15:04:46.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.045+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f15d006c610 con 0x7f15b006b840 2024-02-19T15:04:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.046+0000 7f15c77fe640 1 -- 172.21.15.78:0/3296404549 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f15c80348f0 con 0x7f15b006b840 2024-02-19T15:04:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 0 to deep-scrub 2024-02-19T15:04:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15b006b840 msgr2=0x7f15b006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15b006b840 0x7f15b006dd00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f15c8036000 tx=0x7f15c802e000 comp rx=0 tx=0).stop 2024-02-19T15:04:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 msgr2=0x7f15d0149820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 0x7f15d0149820 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f15d00be3a0 tx=0x7f15c00119a0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 shutdown_connections 2024-02-19T15:04:46.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f15b006b840 0x7f15b006dd00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15d01493a0 0x7f15d0149820 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15d00ac410 0x7f15d01ee170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 --2- 172.21.15.78:0/3296404549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15bc003970 0x7f15d00aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 >> 172.21.15.78:0/3296404549 conn(0x7f15d00ca8a0 msgr2=0x7f15d0074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 shutdown_connections 2024-02-19T15:04:46.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.048+0000 7f15d5039640 1 -- 172.21.15.78:0/3296404549 wait complete. 2024-02-19T15:04:46.061 INFO:tasks.ceph:Scrubbing osd.1 2024-02-19T15:04:46.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:46.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.162+0000 7f4c6de5c640 1 Processor -- start 2024-02-19T15:04:46.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c6de5c640 1 -- start start 2024-02-19T15:04:46.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c6de5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c680c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c6de5c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c680c6fa0 con 0x7f4c680c34b0 2024-02-19T15:04:46.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c6de5c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c681528e0 con 0x7f4c68152500 2024-02-19T15:04:46.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c6de5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680c6cc0 con 0x7f4c681590e0 2024-02-19T15:04:46.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c67fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4c680c34b0 0x7f4c6806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50670/0 (socket says 172.21.15.78:50670) 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.163+0000 7f4c67fff640 1 -- 172.21.15.78:0/1274911441 learned_addr learned my addr 172.21.15.78:0/1274911441 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827314389 0 0) 0x7f4c680c6fa0 con 0x7f4c680c34b0 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c54003520 con 0x7f4c680c34b0 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4226527967 0 0) 0x7f4c54003520 con 0x7f4c680c34b0 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 msgr2=0x7f4c680c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c680c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c680c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 >> v1:172.21.15.150:6789/0 conn(0x7f4c68152500 legacy=0x7f4c68155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68158600 con 0x7f4c680c34b0 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f4c600019f0 con 0x7f4c680c34b0 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c680c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 >> v1:172.21.15.78:6789/0 conn(0x7f4c680c34b0 legacy=0x7f4c6806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c54003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c54004d00 0x7f4c54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680c6cc0 con 0x7f4c540039a0 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5400bb90 con 0x7f4c54004d00 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5400bd50 con 0x7f4c681590e0 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c677fe640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c54003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c67fff640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c54004d00 0x7f4c54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c677fe640 1 -- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 msgr2=0x7f4c540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:46.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c677fe640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c677fe640 1 -- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c54004d00 msgr2=0x7f4c54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c677fe640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c54004d00 0x7f4c54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c677fe640 1 -- 172.21.15.78:0/1274911441 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c54003520 con 0x7f4c540039a0 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c67fff640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c54004d00 0x7f4c54005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.164+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c681590e0 0x7f4c540047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.165+0000 7f4c677fe640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c54003d80 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f4c58001860 tx=0x7f4c540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7d82c63434775e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.165+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58033630 con 0x7f4c540039a0 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.165+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c580337b0 con 0x7f4c540039a0 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.165+0000 7f4c667fc640 1 -- 172.21.15.78:0/1274911441 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58033b60 con 0x7f4c540039a0 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c6de5c640 1 -- 172.21.15.78:0/1274911441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 msgr2=0x7f4c54003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c6de5c640 1 --2- 172.21.15.78:0/1274911441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c54003d80 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f4c58001860 tx=0x7f4c540057f0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c67fff640 1 -- 172.21.15.78:0/1274911441 reap_dead start 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c6de5c640 1 -- 172.21.15.78:0/1274911441 shutdown_connections 2024-02-19T15:04:46.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c6de5c640 1 -- 172.21.15.78:0/1274911441 >> 172.21.15.78:0/1274911441 conn(0x7f4c680ca910 msgr2=0x7f4c680cad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c6de5c640 1 -- 172.21.15.78:0/1274911441 shutdown_connections 2024-02-19T15:04:46.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.166+0000 7f4c6de5c640 1 -- 172.21.15.78:0/1274911441 wait complete. 2024-02-19T15:04:46.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.167+0000 7f4c6de5c640 1 Processor -- start 2024-02-19T15:04:46.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.167+0000 7f4c6de5c640 1 -- start start 2024-02-19T15:04:46.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c680c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c680c18f0 0x7f4c68157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c68143f20 0x7f4c681443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681e2420 con 0x7f4c540039a0 2024-02-19T15:04:46.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681ed710 con 0x7f4c68143f20 2024-02-19T15:04:46.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68157c00 con 0x7f4c680c18f0 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c680c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c680c0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46470/0 (socket says 172.21.15.78:46470) 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 -- 172.21.15.78:0/2891567828 learned_addr learned my addr 172.21.15.78:0/2891567828 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c67fff640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c68143f20 0x7f4c681443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c680c18f0 0x7f4c68157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 -- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c680c18f0 msgr2=0x7f4c68157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c680c18f0 0x7f4c68157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 -- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c68143f20 msgr2=0x7f4c681443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c68143f20 0x7f4c681443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 -- 172.21.15.78:0/2891567828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c681ed960 con 0x7f4c540039a0 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c680c18f0 0x7f4c68157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c677fe640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c680c0d60 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f4c58008fd0 tx=0x7f4c58001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c580035f0 con 0x7f4c540039a0 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58003750 con 0x7f4c540039a0 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c681edc40 con 0x7f4c540039a0 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58003b00 con 0x7f4c540039a0 2024-02-19T15:04:46.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.168+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c680c12a0 con 0x7f4c540039a0 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.169+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c680b0910 con 0x7f4c540039a0 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4c58003c60 con 0x7f4c540039a0 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c5ffff640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4806b810 0x7f4c4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4806b810 0x7f4c4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f4c580ac1d0 con 0x7f4c540039a0 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c5ffff640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f4c48071e90 0x7f4c480742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 --> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c480749a0 con 0x7f4c48071e90 2024-02-19T15:04:46.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c580ae890 con 0x7f4c540039a0 2024-02-19T15:04:46.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.170+0000 7f4c67fff640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f4c48071e90 0x7f4c480742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.171+0000 7f4c66ffd640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4806b810 0x7f4c4806dcd0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4c680c31d0 tx=0x7f4c50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.171+0000 7f4c67fff640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f4c48071e90 0x7f4c480742d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.171+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== osd.1 v2:172.21.15.78:6802/289106328 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c480749a0 con 0x7f4c48071e90 2024-02-19T15:04:46.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.184+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 --> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4c68158bc0 con 0x7f4c48071e90 2024-02-19T15:04:46.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.184+0000 7f4c5ffff640 1 -- 172.21.15.78:0/2891567828 <== osd.1 v2:172.21.15.78:6802/289106328 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4c60007170 con 0x7f4c48071e90 2024-02-19T15:04:46.188 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:46.188 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f4c48071e90 msgr2=0x7f4c480742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6802/289106328,v1:172.21.15.78:6804/289106328] conn(0x7f4c48071e90 0x7f4c480742d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4806b810 msgr2=0x7f4c4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4c4806b810 0x7f4c4806dcd0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4c680c31d0 tx=0x7f4c50008040 comp rx=0 tx=0).stop 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 msgr2=0x7f4c680c0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 --2- 172.21.15.78:0/2891567828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c540039a0 0x7f4c680c0d60 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f4c58008fd0 tx=0x7f4c58001230 comp rx=0 tx=0).stop 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c67fff640 1 -- 172.21.15.78:0/2891567828 reap_dead start 2024-02-19T15:04:46.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 shutdown_connections 2024-02-19T15:04:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 >> 172.21.15.78:0/2891567828 conn(0x7f4c680ca910 msgr2=0x7f4c68154f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 shutdown_connections 2024-02-19T15:04:46.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.185+0000 7f4c6de5c640 1 -- 172.21.15.78:0/2891567828 wait complete. 2024-02-19T15:04:46.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-02-19T15:04:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 Processor -- start 2024-02-19T15:04:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 -- start start 2024-02-19T15:04:46.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b28155f90 0x7f4b2806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b28158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b280c6af0 con 0x7f4b280c6d30 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b280c7940 con 0x7f4b28155f90 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b2dbd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280c7c20 con 0x7f4b281588a0 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.296+0000 7f4b277fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b28155f90 0x7f4b2806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b26ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4b280c6d30 0x7f4b280c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50684/0 (socket says 172.21.15.78:50684) 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b277fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b28155f90 0x7f4b2806d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46472/0 (socket says 172.21.15.78:46472) 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b26ffd640 1 -- 172.21.15.78:0/508706814 learned_addr learned my addr 172.21.15.78:0/508706814 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:46.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b27fff640 1 --2- 172.21.15.78:0/508706814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b28158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b277fe640 1 -- 172.21.15.78:0/508706814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 msgr2=0x7f4b28158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b277fe640 1 --2- 172.21.15.78:0/508706814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b28158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b277fe640 1 -- 172.21.15.78:0/508706814 >> v1:172.21.15.78:6789/0 conn(0x7f4b280c6d30 legacy=0x7f4b280c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b277fe640 1 -- 172.21.15.78:0/508706814 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280c7f00 con 0x7f4b28155f90 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.297+0000 7f4b267fc640 1 -- 172.21.15.78:0/508706814 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345797767 0 0) 0x7f4b280c6af0 con 0x7f4b280c6d30 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b277fe640 1 --2- 172.21.15.78:0/508706814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b28155f90 0x7f4b2806d550 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f4b180097a0 tx=0x7f4b1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23299301703ea8a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b267fc640 1 -- 172.21.15.78:0/508706814 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1802e070 con 0x7f4b28155f90 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b267fc640 1 -- 172.21.15.78:0/508706814 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b180315a0 con 0x7f4b28155f90 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b267fc640 1 -- 172.21.15.78:0/508706814 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18031930 con 0x7f4b28155f90 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/508706814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b28155f90 msgr2=0x7f4b2806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/508706814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b28155f90 0x7f4b2806d550 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f4b180097a0 tx=0x7f4b1802b840 comp rx=0 tx=0).stop 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/508706814 shutdown_connections 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/508706814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b28158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/508706814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b28155f90 0x7f4b2806d550 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/508706814 >> 172.21.15.78:0/508706814 conn(0x7f4b2806b700 msgr2=0x7f4b280aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/508706814 shutdown_connections 2024-02-19T15:04:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.298+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/508706814 wait complete. 2024-02-19T15:04:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.299+0000 7f4b2dbd7640 1 Processor -- start 2024-02-19T15:04:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 -- start start 2024-02-19T15:04:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 0x7f4b281536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b28155f90 0x7f4b280a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b280a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b281ee6d0 con 0x7f4b280c6d30 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280a98e0 con 0x7f4b28155f90 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b2dbd7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b281eec00 con 0x7f4b281588a0 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 0x7f4b281536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b277fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 0x7f4b281536c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46476/0 (socket says 172.21.15.78:46476) 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b277fe640 1 -- 172.21.15.78:0/3197141202 learned_addr learned my addr 172.21.15.78:0/3197141202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b27fff640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b280a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.300+0000 7f4b26ffd640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b28155f90 0x7f4b280a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b277fe640 1 -- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 msgr2=0x7f4b280a93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b277fe640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b280a93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b277fe640 1 -- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b28155f90 msgr2=0x7f4b280a8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b277fe640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b28155f90 0x7f4b280a8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b277fe640 1 -- 172.21.15.78:0/3197141202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18009400 con 0x7f4b280c6d30 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b277fe640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 0x7f4b281536c0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f4b18034040 tx=0x7f4b180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18032900 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18032a80 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b280ad670 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18032e30 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.301+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b281ee920 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.302+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b28074990 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.302+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4b1804e430 con 0x7f4b280c6d30 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.303+0000 7f4b0ffff640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b840 0x7f4b1406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.303+0000 7f4b26ffd640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b840 0x7f4b1406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.303+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f4b180ac010 con 0x7f4b280c6d30 2024-02-19T15:04:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.303+0000 7f4b26ffd640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b840 0x7f4b1406dd00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4b280c9710 tx=0x7f4b1c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.307+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b18076be0 con 0x7f4b280c6d30 2024-02-19T15:04:46.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.479+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b280c3430 con 0x7f4b1406b840 2024-02-19T15:04:46.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.479+0000 7f4b0ffff640 1 -- 172.21.15.78:0/3197141202 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4b280c3430 con 0x7f4b1406b840 2024-02-19T15:04:46.482 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 1 to deep-scrub 2024-02-19T15:04:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b840 msgr2=0x7f4b1406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b840 0x7f4b1406dd00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4b280c9710 tx=0x7f4b1c009040 comp rx=0 tx=0).stop 2024-02-19T15:04:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 msgr2=0x7f4b281536c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 0x7f4b281536c0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f4b18034040 tx=0x7f4b180046c0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 shutdown_connections 2024-02-19T15:04:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f4b1406b840 0x7f4b1406dd00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b281588a0 0x7f4b280a93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b28155f90 0x7f4b280a8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 --2- 172.21.15.78:0/3197141202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b280c6d30 0x7f4b281536c0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 >> 172.21.15.78:0/3197141202 conn(0x7f4b2806b700 msgr2=0x7f4b28154290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 shutdown_connections 2024-02-19T15:04:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.481+0000 7f4b2dbd7640 1 -- 172.21.15.78:0/3197141202 wait complete. 2024-02-19T15:04:46.494 INFO:tasks.ceph:Scrubbing osd.2 2024-02-19T15:04:46.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:46.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.595+0000 7f8ffda88640 1 Processor -- start 2024-02-19T15:04:46.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.595+0000 7f8ffda88640 1 -- start start 2024-02-19T15:04:46.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ffda88640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ff806c330 0x7f8ff80c0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ffda88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8067030 0x7f8ff80c0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ffda88640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff806f690 con 0x7f8ff8152820 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ffda88640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff80c2470 con 0x7f8ff806c330 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ffda88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff8155000 con 0x7f8ff8067030 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ff6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ff806c330 0x7f8ff80c0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8067030 0x7f8ff80c0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ff77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8ff8152820 0x7f8ff80c18e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42954/0 (socket says 172.21.15.78:42954) 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.596+0000 7f8ff77fe640 1 -- 172.21.15.78:0/4090965730 learned_addr learned my addr 172.21.15.78:0/4090965730 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:46.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.597+0000 7f8ff6ffd640 1 -- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8067030 msgr2=0x7f8ff80c0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:46.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.597+0000 7f8ff6ffd640 1 --2- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8067030 0x7f8ff80c0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.597+0000 7f8ff6ffd640 1 -- 172.21.15.78:0/4090965730 >> v1:172.21.15.150:6789/0 conn(0x7f8ff8152820 legacy=0x7f8ff80c18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.597+0000 7f8ff67fc640 1 -- 172.21.15.78:0/4090965730 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378855673 0 0) 0x7f8ff806f690 con 0x7f8ff8152820 2024-02-19T15:04:46.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.597+0000 7f8ff6ffd640 1 -- 172.21.15.78:0/4090965730 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff806f8e0 con 0x7f8ff806c330 2024-02-19T15:04:46.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.597+0000 7f8ff6ffd640 1 --2- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 0x7f8ff80c0310 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0097a0 tx=0x7f8fec02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10d05263769a7f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ff67fc640 1 -- 172.21.15.78:0/4090965730 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec02e070 con 0x7f8ff806c330 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ff67fc640 1 -- 172.21.15.78:0/4090965730 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec0315a0 con 0x7f8ff806c330 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ff67fc640 1 -- 172.21.15.78:0/4090965730 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec031930 con 0x7f8ff806c330 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 -- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 msgr2=0x7f8ff80c0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 --2- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 0x7f8ff80c0310 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0097a0 tx=0x7f8fec02b840 comp rx=0 tx=0).stop 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 -- 172.21.15.78:0/4090965730 shutdown_connections 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 --2- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8067030 0x7f8ff80c0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 --2- 172.21.15.78:0/4090965730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 0x7f8ff80c0310 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 -- 172.21.15.78:0/4090965730 >> 172.21.15.78:0/4090965730 conn(0x7f8ff806e960 msgr2=0x7f8ff806ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 -- 172.21.15.78:0/4090965730 shutdown_connections 2024-02-19T15:04:46.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.598+0000 7f8ffda88640 1 -- 172.21.15.78:0/4090965730 wait complete. 2024-02-19T15:04:46.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.599+0000 7f8ffda88640 1 Processor -- start 2024-02-19T15:04:46.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 -- start start 2024-02-19T15:04:46.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff8067030 0x7f8ff8158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 0x7f8ff8143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8152820 0x7f8ff8144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff81eaf20 con 0x7f8ff8067030 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80c4d50 con 0x7f8ff806c330 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ffda88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff81ed530 con 0x7f8ff8152820 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ff77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff8067030 0x7f8ff8158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ff77fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff8067030 0x7f8ff8158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46482/0 (socket says 172.21.15.78:46482) 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ff77fe640 1 -- 172.21.15.78:0/405636517 learned_addr learned my addr 172.21.15.78:0/405636517 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ff6ffd640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 0x7f8ff8143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.600+0000 7f8ff7fff640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8152820 0x7f8ff8144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ff77fe640 1 -- 172.21.15.78:0/405636517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8152820 msgr2=0x7f8ff8144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ff77fe640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ff8152820 0x7f8ff8144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ff77fe640 1 -- 172.21.15.78:0/405636517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 msgr2=0x7f8ff8143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ff77fe640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ff806c330 0x7f8ff8143f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ff77fe640 1 -- 172.21.15.78:0/405636517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec009400 con 0x7f8ff8067030 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ff77fe640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff8067030 0x7f8ff8158200 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c500 tx=0x7f8fe800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8007560 con 0x7f8ff8067030 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe80047e0 con 0x7f8ff8067030 2024-02-19T15:04:46.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff80a9b70 con 0x7f8ff8067030 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8004b90 con 0x7f8ff8067030 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.601+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff80aa130 con 0x7f8ff8067030 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.602+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ff8152c00 con 0x7f8ff8067030 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.602+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8fe8004cf0 con 0x7f8ff8067030 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8fdffff640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fe406b810 0x7f8fe406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8fe80895b0 con 0x7f8ff8067030 2024-02-19T15:04:46.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8ff6ffd640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fe406b810 0x7f8fe406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8fdffff640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f8fe4071e90 0x7f8fe40742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 --> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fe40749a0 con 0x7f8fe4071e90 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fe808bc70 con 0x7f8ff8067030 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8ff7fff640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f8fe4071e90 0x7f8fe40742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8ff6ffd640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fe406b810 0x7f8fe406dcd0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec03e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.603+0000 7f8ff7fff640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f8fe4071e90 0x7f8fe40742d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.604+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== osd.2 v2:172.21.15.78:6800/3664023701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8fe40749a0 con 0x7f8fe4071e90 2024-02-19T15:04:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 --> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8ff81592e0 con 0x7f8fe4071e90 2024-02-19T15:04:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8fdffff640 1 -- 172.21.15.78:0/405636517 <== osd.2 v2:172.21.15.78:6800/3664023701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8ff000f040 con 0x7f8fe4071e90 2024-02-19T15:04:46.620 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:46.620 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:46.620 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f8fe4071e90 msgr2=0x7f8fe40742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6800/3664023701,v1:172.21.15.78:6801/3664023701] conn(0x7f8fe4071e90 0x7f8fe40742d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fe406b810 msgr2=0x7f8fe406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8fe406b810 0x7f8fe406dcd0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec03e040 comp rx=0 tx=0).stop 2024-02-19T15:04:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff8067030 msgr2=0x7f8ff8158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 --2- 172.21.15.78:0/405636517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ff8067030 0x7f8ff8158200 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c500 tx=0x7f8fe800c9c0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ff7fff640 1 -- 172.21.15.78:0/405636517 reap_dead start 2024-02-19T15:04:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 shutdown_connections 2024-02-19T15:04:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 >> 172.21.15.78:0/405636517 conn(0x7f8ff806e960 msgr2=0x7f8ff80c0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 shutdown_connections 2024-02-19T15:04:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.617+0000 7f8ffda88640 1 -- 172.21.15.78:0/405636517 wait complete. 2024-02-19T15:04:46.631 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-02-19T15:04:46.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e6e10d640 1 Processor -- start 2024-02-19T15:04:46.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e6e10d640 1 -- start start 2024-02-19T15:04:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e6e10d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e68152700 0x7f2e680c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e6e10d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e681551f0 con 0x7f2e680c5730 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e6e10d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e68155490 con 0x7f2e680c8e00 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e6e10d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e68155770 con 0x7f2e68152700 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.729+0000 7f2e66ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2e680c5730 0x7f2e6806c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50692/0 (socket says 172.21.15.78:50692) 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e66ffd640 1 -- 172.21.15.78:0/3512346371 learned_addr learned my addr 172.21.15.78:0/3512346371 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e677fe640 1 --2- 172.21.15.78:0/3512346371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e68152700 0x7f2e680c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401740215 0 0) 0x7f2e681551f0 con 0x7f2e680c5730 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e50003520 con 0x7f2e680c5730 2024-02-19T15:04:46.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744110069 0 0) 0x7f2e68155490 con 0x7f2e680c8e00 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e681551f0 con 0x7f2e680c8e00 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e677fe640 1 -- 172.21.15.78:0/3512346371 >> v1:172.21.15.150:6789/0 conn(0x7f2e680c8e00 legacy=0x7f2e6815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e677fe640 1 -- 172.21.15.78:0/3512346371 >> v1:172.21.15.78:6789/0 conn(0x7f2e680c5730 legacy=0x7f2e6806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560583274 0 0) 0x7f2e50003520 con 0x7f2e680c5730 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.730+0000 7f2e677fe640 1 -- 172.21.15.78:0/3512346371 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68155a50 con 0x7f2e68152700 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.731+0000 7f2e677fe640 1 --2- 172.21.15.78:0/3512346371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e68152700 0x7f2e680c51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e580097d0 tx=0x7f2e5802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af06ccdb12ce0907 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.731+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c001670 con 0x7f2e68152700 2024-02-19T15:04:46.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.731+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5802e050 con 0x7f2e68152700 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.731+0000 7f2e667fc640 1 -- 172.21.15.78:0/3512346371 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58030730 con 0x7f2e68152700 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.731+0000 7f2e6e10d640 1 -- 172.21.15.78:0/3512346371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e68152700 msgr2=0x7f2e680c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.731+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/3512346371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e68152700 0x7f2e680c51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e580097d0 tx=0x7f2e5802b800 comp rx=0 tx=0).stop 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.732+0000 7f2e6e10d640 1 -- 172.21.15.78:0/3512346371 shutdown_connections 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.732+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/3512346371 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e68152700 0x7f2e680c51f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.732+0000 7f2e6e10d640 1 -- 172.21.15.78:0/3512346371 >> 172.21.15.78:0/3512346371 conn(0x7f2e680cac90 msgr2=0x7f2e680cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.732+0000 7f2e6e10d640 1 -- 172.21.15.78:0/3512346371 shutdown_connections 2024-02-19T15:04:46.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.732+0000 7f2e6e10d640 1 -- 172.21.15.78:0/3512346371 wait complete. 2024-02-19T15:04:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 Processor -- start 2024-02-19T15:04:46.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 -- start start 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e680c5730 0x7f2e680ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e680c8e00 0x7f2e681ebf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 0x7f2e681ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680c4520 con 0x7f2e68152700 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e681eef50 con 0x7f2e680c8e00 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.733+0000 7f2e6e10d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680abb30 con 0x7f2e680c5730 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 0x7f2e681ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 0x7f2e681ee280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46488/0 (socket says 172.21.15.78:46488) 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 -- 172.21.15.78:0/1688735577 learned_addr learned my addr 172.21.15.78:0/1688735577 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e677fe640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e680c5730 0x7f2e680ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e66ffd640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e680c8e00 0x7f2e681ebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 -- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e680c5730 msgr2=0x7f2e680ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e680c5730 0x7f2e680ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 -- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e680c8e00 msgr2=0x7f2e681ebf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e680c8e00 0x7f2e681ebf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 -- 172.21.15.78:0/1688735577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e58009400 con 0x7f2e68152700 2024-02-19T15:04:46.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e66ffd640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e680c8e00 0x7f2e681ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e677fe640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e680c5730 0x7f2e680ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e67fff640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 0x7f2e681ee280 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000c7c0 tx=0x7f2e6000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60003b70 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60003cd0 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000aa50 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e68149b30 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.734+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e680aadf0 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.735+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e68074990 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.735+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2e6000abb0 con 0x7f2e68152700 2024-02-19T15:04:46.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.736+0000 7f2e4ffff640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e5406b840 0x7f2e5406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:46.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.736+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2e60089f50 con 0x7f2e68152700 2024-02-19T15:04:46.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.736+0000 7f2e677fe640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e5406b840 0x7f2e5406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:46.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.736+0000 7f2e677fe640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e5406b840 0x7f2e5406dd00 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f2e58034000 tx=0x7f2e58008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:46.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.740+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2e60054a70 con 0x7f2e68152700 2024-02-19T15:04:46.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.912+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e6806ded0 con 0x7f2e5406b840 2024-02-19T15:04:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.913+0000 7f2e4ffff640 1 -- 172.21.15.78:0/1688735577 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2e580307e0 con 0x7f2e5406b840 2024-02-19T15:04:46.916 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 2 to deep-scrub 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e5406b840 msgr2=0x7f2e5406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e5406b840 0x7f2e5406dd00 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f2e58034000 tx=0x7f2e58008030 comp rx=0 tx=0).stop 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 msgr2=0x7f2e681ee280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 0x7f2e681ee280 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000c7c0 tx=0x7f2e6000cc80 comp rx=0 tx=0).stop 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 shutdown_connections 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2e5406b840 0x7f2e5406dd00 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e68152700 0x7f2e681ee280 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e680c8e00 0x7f2e681ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 --2- 172.21.15.78:0/1688735577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e680c5730 0x7f2e680ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:46.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 >> 172.21.15.78:0/1688735577 conn(0x7f2e680cac90 msgr2=0x7f2e680ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:46.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 shutdown_connections 2024-02-19T15:04:46.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:46.915+0000 7f2e6e10d640 1 -- 172.21.15.78:0/1688735577 wait complete. 2024-02-19T15:04:46.928 INFO:tasks.ceph:Scrubbing osd.3 2024-02-19T15:04:46.928 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:47.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.029+0000 7f2c41ec7640 1 Processor -- start 2024-02-19T15:04:47.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.029+0000 7f2c41ec7640 1 -- start start 2024-02-19T15:04:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c41ec7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c3c0bf0c0 0x7f2c3c1557e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c41ec7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c06fa90 con 0x7f2c3c1527d0 2024-02-19T15:04:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c41ec7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3c1592b0 con 0x7f2c3c0c7f10 2024-02-19T15:04:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c41ec7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3c071280 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c3c0bf0c0 0x7f2c3c1557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c3c0bf0c0 0x7f2c3c1557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46490/0 (socket says 172.21.15.78:46490) 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2c3c1527d0 0x7f2c3c0c4e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50698/0 (socket says 172.21.15.78:50698) 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3bfff640 1 -- 172.21.15.78:0/315490265 learned_addr learned my addr 172.21.15.78:0/315490265 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3bfff640 1 -- 172.21.15.78:0/315490265 >> v1:172.21.15.150:6789/0 conn(0x7f2c3c0c7f10 legacy=0x7f2c3c1550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3bfff640 1 -- 172.21.15.78:0/315490265 >> v1:172.21.15.78:6789/0 conn(0x7f2c3c1527d0 legacy=0x7f2c3c0c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3bfff640 1 -- 172.21.15.78:0/315490265 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c06f4d0 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.030+0000 7f2c3a7fc640 1 -- 172.21.15.78:0/315490265 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546011332 0 0) 0x7f2c3c06fa90 con 0x7f2c3c1527d0 2024-02-19T15:04:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c3bfff640 1 --2- 172.21.15.78:0/315490265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c1557e0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009e50 tx=0x7f2c3402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e630b6fa77bad22a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c3a7fc640 1 -- 172.21.15.78:0/315490265 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3402cb20 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c3a7fc640 1 -- 172.21.15.78:0/315490265 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3402cc80 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c3a7fc640 1 -- 172.21.15.78:0/315490265 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c340343f0 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c41ec7640 1 -- 172.21.15.78:0/315490265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 msgr2=0x7f2c3c1557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c41ec7640 1 --2- 172.21.15.78:0/315490265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c1557e0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009e50 tx=0x7f2c3402bf80 comp rx=0 tx=0).stop 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c41ec7640 1 -- 172.21.15.78:0/315490265 shutdown_connections 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c41ec7640 1 --2- 172.21.15.78:0/315490265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c1557e0 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.031+0000 7f2c41ec7640 1 -- 172.21.15.78:0/315490265 >> 172.21.15.78:0/315490265 conn(0x7f2c3c0aa370 msgr2=0x7f2c3c0aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.032+0000 7f2c41ec7640 1 -- 172.21.15.78:0/315490265 shutdown_connections 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.032+0000 7f2c41ec7640 1 -- 172.21.15.78:0/315490265 wait complete. 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 Processor -- start 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 -- start start 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c3c0c7f10 0x7f2c3c150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c3c1527d0 0x7f2c3c150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0c16c0 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0ad620 con 0x7f2c3c1527d0 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c41ec7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c0bf7d0 con 0x7f2c3c0c7f10 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.033+0000 7f2c3b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46494/0 (socket says 172.21.15.78:46494) 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 -- 172.21.15.78:0/1624027870 learned_addr learned my addr 172.21.15.78:0/1624027870 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3bfff640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c3c1527d0 0x7f2c3c150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 -- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c3c0c7f10 msgr2=0x7f2c3c150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3affd640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c3c0c7f10 0x7f2c3c150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c3c0c7f10 0x7f2c3c150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 -- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c3c1527d0 msgr2=0x7f2c3c150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c3c1527d0 0x7f2c3c150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 -- 172.21.15.78:0/1624027870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c34009b30 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3affd640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c3c0c7f10 0x7f2c3c150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3bfff640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c3c1527d0 0x7f2c3c150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c3b7fe640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c158090 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00c460 tx=0x7f2c2c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c0074e0 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2c007660 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c0ace10 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c004a20 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.034+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c0ad060 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.035+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c3c070f10 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f2c2c004b80 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c23fff640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c2806b810 0x7f2c2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c3affd640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c2806b810 0x7f2c2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2c2c0895c0 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c23fff640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7f2c28071e90 0x7f2c280742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 --> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c280749a0 con 0x7f2c28071e90 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c2c08bc80 con 0x7f2c3c0bf0c0 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.036+0000 7f2c3bfff640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7f2c28071e90 0x7f2c280742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.037+0000 7f2c3affd640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c2806b810 0x7f2c2806dcd0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0ae5f0 tx=0x7f2c30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.037+0000 7f2c3bfff640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7f2c28071e90 0x7f2c280742d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.037+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== osd.3 v2:172.21.15.78:6824/3716944111 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2c280749a0 con 0x7f2c28071e90 2024-02-19T15:04:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.050+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 --> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2c3c155d20 con 0x7f2c28071e90 2024-02-19T15:04:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.050+0000 7f2c23fff640 1 -- 172.21.15.78:0/1624027870 <== osd.3 v2:172.21.15.78:6824/3716944111 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2c34040410 con 0x7f2c28071e90 2024-02-19T15:04:47.053 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:47.054 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:47.054 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:47.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7f2c28071e90 msgr2=0x7f2c280742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6824/3716944111,v1:172.21.15.78:6825/3716944111] conn(0x7f2c28071e90 0x7f2c280742d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c2806b810 msgr2=0x7f2c2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f2c2806b810 0x7f2c2806dcd0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0ae5f0 tx=0x7f2c30009040 comp rx=0 tx=0).stop 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 msgr2=0x7f2c3c158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 --2- 172.21.15.78:0/1624027870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c3c0bf0c0 0x7f2c3c158090 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c00c460 tx=0x7f2c2c00c920 comp rx=0 tx=0).stop 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c3bfff640 1 -- 172.21.15.78:0/1624027870 reap_dead start 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 shutdown_connections 2024-02-19T15:04:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 >> 172.21.15.78:0/1624027870 conn(0x7f2c3c0aa370 msgr2=0x7f2c3c151dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 shutdown_connections 2024-02-19T15:04:47.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.051+0000 7f2c41ec7640 1 -- 172.21.15.78:0/1624027870 wait complete. 2024-02-19T15:04:47.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-02-19T15:04:47.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 Processor -- start 2024-02-19T15:04:47.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 -- start start 2024-02-19T15:04:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 0x7f96fc0c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96fc0c59f0 0x7f96fc15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96fc0c9200 con 0x7f96fc0c5590 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0cb720 con 0x7f96fc0c59f0 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.162+0000 7f9703f0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc158ff0 con 0x7f96fc0c51b0 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f9701cab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 0x7f96fc0c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97024ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96fc0c59f0 0x7f96fc15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f9701cab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 0x7f96fc0c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:47824/0 (socket says 172.21.15.78:47824) 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97024ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96fc0c59f0 0x7f96fc15f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:46586/0 (socket says 172.21.15.78:46586) 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97014aa640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f96fc0c5590 0x7f96fc0c9c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:42994/0 (socket says 172.21.15.78:42994) 2024-02-19T15:04:47.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f9701cab640 1 -- 172.21.15.78:0/2131524466 learned_addr learned my addr 172.21.15.78:0/2131524466 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97024ac640 1 -- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 msgr2=0x7f96fc0c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97024ac640 1 --2- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 0x7f96fc0c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97024ac640 1 -- 172.21.15.78:0/2131524466 >> v1:172.21.15.150:6789/0 conn(0x7f96fc0c5590 legacy=0x7f96fc0c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f9700ca9640 1 -- 172.21.15.78:0/2131524466 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935415234 0 0) 0x7f96fc0c9200 con 0x7f96fc0c5590 2024-02-19T15:04:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f97024ac640 1 -- 172.21.15.78:0/2131524466 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc1592d0 con 0x7f96fc0c59f0 2024-02-19T15:04:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.163+0000 7f9701cab640 1 --2- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 0x7f96fc0c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:47.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.164+0000 7f97024ac640 1 --2- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc15f8d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f96f40097a0 tx=0x7f96f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6846a5f3cb594617 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.164+0000 7f9700ca9640 1 -- 172.21.15.78:0/2131524466 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4031830 con 0x7f96fc0c59f0 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.164+0000 7f9700ca9640 1 -- 172.21.15.78:0/2131524466 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4031990 con 0x7f96fc0c59f0 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.164+0000 7f9700ca9640 1 -- 172.21.15.78:0/2131524466 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4031d20 con 0x7f96fc0c59f0 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 -- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 msgr2=0x7f96fc15f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 --2- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc15f8d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f96f40097a0 tx=0x7f96f402b8d0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 -- 172.21.15.78:0/2131524466 shutdown_connections 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 --2- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc15f8d0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 --2- 172.21.15.78:0/2131524466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c51b0 0x7f96fc0c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 -- 172.21.15.78:0/2131524466 >> 172.21.15.78:0/2131524466 conn(0x7f96fc0ca830 msgr2=0x7f96fc0cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 -- 172.21.15.78:0/2131524466 shutdown_connections 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.165+0000 7f9703f0f640 1 -- 172.21.15.78:0/2131524466 wait complete. 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.166+0000 7f9703f0f640 1 Processor -- start 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.166+0000 7f9703f0f640 1 -- start start 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.166+0000 7f9703f0f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 0x7f96fc1573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.166+0000 7f9703f0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c5590 0x7f96fc151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9703f0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9703f0f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc148210 con 0x7f96fc0c51b0 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9703f0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc151490 con 0x7f96fc0c59f0 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9703f0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc147c50 con 0x7f96fc0c5590 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 0x7f96fc1573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 0x7f96fc1573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46504/0 (socket says 172.21.15.78:46504) 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 -- 172.21.15.78:0/3359622888 learned_addr learned my addr 172.21.15.78:0/3359622888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f97014aa640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c5590 0x7f96fc151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f97024ac640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 -- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c5590 msgr2=0x7f96fc151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c5590 0x7f96fc151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 -- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 msgr2=0x7f96fc157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 -- 172.21.15.78:0/3359622888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8009b30 con 0x7f96fc0c51b0 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9701cab640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 0x7f96fc1573f0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f96f800efc0 tx=0x7f96f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f800ce60 con 0x7f96fc0c51b0 2024-02-19T15:04:47.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f800f040 con 0x7f96fc0c51b0 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8014680 con 0x7f96fc0c51b0 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f4009400 con 0x7f96fc0c51b0 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.167+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc156ed0 con 0x7f96fc0c51b0 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.168+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc074990 con 0x7f96fc0c51b0 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.169+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f96f80147e0 con 0x7f96fc0c51b0 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.169+0000 7f96f27fc640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96e006b810 0x7f96e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.169+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f96f808ab80 con 0x7f96fc0c51b0 2024-02-19T15:04:47.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.169+0000 7f97014aa640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96e006b810 0x7f96e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.170+0000 7f97014aa640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96e006b810 0x7f96e006dcd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0c8b30 tx=0x7f96ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.173+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f96f80556a0 con 0x7f96fc0c51b0 2024-02-19T15:04:47.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.346+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f96fc14c700 con 0x7f96e006b810 2024-02-19T15:04:47.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.347+0000 7f96f27fc640 1 -- 172.21.15.78:0/3359622888 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f96fc14c700 con 0x7f96e006b810 2024-02-19T15:04:47.350 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 3 to deep-scrub 2024-02-19T15:04:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96e006b810 msgr2=0x7f96e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96e006b810 0x7f96e006dcd0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0c8b30 tx=0x7f96ec009040 comp rx=0 tx=0).stop 2024-02-19T15:04:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 msgr2=0x7f96fc1573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 0x7f96fc1573f0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f96f800efc0 tx=0x7f96f800c490 comp rx=0 tx=0).stop 2024-02-19T15:04:47.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 shutdown_connections 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f96e006b810 0x7f96e006dcd0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96fc0c59f0 0x7f96fc157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96fc0c5590 0x7f96fc151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 --2- 172.21.15.78:0/3359622888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96fc0c51b0 0x7f96fc1573f0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 >> 172.21.15.78:0/3359622888 conn(0x7f96fc0ca830 msgr2=0x7f96fc0ab300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 shutdown_connections 2024-02-19T15:04:47.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.349+0000 7f9703f0f640 1 -- 172.21.15.78:0/3359622888 wait complete. 2024-02-19T15:04:47.362 INFO:tasks.ceph:Scrubbing osd.4 2024-02-19T15:04:47.362 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:47.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.463+0000 7fd12caed640 1 Processor -- start 2024-02-19T15:04:47.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.463+0000 7fd12caed640 1 -- start start 2024-02-19T15:04:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd12caed640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 0x7fd12806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd12caed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd128155330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd12caed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12806dc70 con 0x7fd128152670 2024-02-19T15:04:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd12caed640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd128158ef0 con 0x7fd1280c8730 2024-02-19T15:04:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd12caed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1280be190 con 0x7fd128154ea0 2024-02-19T15:04:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd128155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd11ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 0x7fd12806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd128155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:45384/0 (socket says 172.21.15.78:45384) 2024-02-19T15:04:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd11ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 0x7fd12806d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:47466/0 (socket says 172.21.15.78:47466) 2024-02-19T15:04:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 -- 172.21.15.78:0/1899408565 learned_addr learned my addr 172.21.15.78:0/1899408565 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd12659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd128152670 0x7fd1280c51b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46714/0 (socket says 172.21.15.78:46714) 2024-02-19T15:04:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 -- 172.21.15.78:0/1899408565 >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 msgr2=0x7fd12806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 --2- 172.21.15.78:0/1899408565 >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 0x7fd12806d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 -- 172.21.15.78:0/1899408565 >> v1:172.21.15.150:6789/0 conn(0x7fd128152670 legacy=0x7fd1280c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd126d9d640 1 -- 172.21.15.78:0/1899408565 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12806dea0 con 0x7fd128154ea0 2024-02-19T15:04:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.464+0000 7fd11ffff640 1 --2- 172.21.15.78:0/1899408565 >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 0x7fd12806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.465+0000 7fd126d9d640 1 --2- 172.21.15.78:0/1899408565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd128155330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd120009e50 tx=0x7fd12002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7ed2421a40cf3bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.465+0000 7fd125d9b640 1 -- 172.21.15.78:0/1899408565 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12002c9e0 con 0x7fd128154ea0 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.465+0000 7fd125d9b640 1 -- 172.21.15.78:0/1899408565 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12002cb40 con 0x7fd128154ea0 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.465+0000 7fd125d9b640 1 -- 172.21.15.78:0/1899408565 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200343f0 con 0x7fd128154ea0 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 -- 172.21.15.78:0/1899408565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 msgr2=0x7fd128155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 --2- 172.21.15.78:0/1899408565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd128155330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd120009e50 tx=0x7fd12002be40 comp rx=0 tx=0).stop 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 -- 172.21.15.78:0/1899408565 shutdown_connections 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 --2- 172.21.15.78:0/1899408565 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd128155330 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 --2- 172.21.15.78:0/1899408565 >> v2:172.21.15.150:3300/0 conn(0x7fd1280c8730 0x7fd12806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 -- 172.21.15.78:0/1899408565 >> 172.21.15.78:0/1899408565 conn(0x7fd128072790 msgr2=0x7fd128072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 -- 172.21.15.78:0/1899408565 shutdown_connections 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.466+0000 7fd12caed640 1 -- 172.21.15.78:0/1899408565 wait complete. 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.467+0000 7fd12caed640 1 Processor -- start 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.467+0000 7fd12caed640 1 -- start start 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1280c8730 0x7fd1280c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd128152670 0x7fd1280ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd1281ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12814e080 con 0x7fd128152670 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1280acda0 con 0x7fd1280c8730 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1280ad060 con 0x7fd128154ea0 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd128152670 0x7fd1280ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd128152670 0x7fd1280ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52952/0 (socket says 172.21.15.78:52952) 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 -- 172.21.15.78:0/388216718 learned_addr learned my addr 172.21.15.78:0/388216718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd126d9d640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd1281ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12659c640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1280c8730 0x7fd1280c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 -- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 msgr2=0x7fd1281ea6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd1281ea6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 -- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1280c8730 msgr2=0x7fd1280c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1280c8730 0x7fd1280c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 -- 172.21.15.78:0/388216718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd110009400 con 0x7fd128152670 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12659c640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1280c8730 0x7fd1280c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd126d9d640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd128154ea0 0x7fd1281ea6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11ffff640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd128152670 0x7fd1280ae9e0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fd11000e990 tx=0x7fd11000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11000c9e0 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11000cb40 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd120009b30 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11000ced0 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.468+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1280c3740 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.469+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1281588a0 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.470+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fd110013470 con 0x7fd128152670 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.470+0000 7fd11effd640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd10806b840 0x7fd10806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.470+0000 7fd12659c640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd10806b840 0x7fd10806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.470+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fd11008a300 con 0x7fd128152670 2024-02-19T15:04:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.470+0000 7fd11effd640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd108071d50 0x7fd108074190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.470+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 --> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd108074860 con 0x7fd108071d50 2024-02-19T15:04:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.471+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd11008c9c0 con 0x7fd128152670 2024-02-19T15:04:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.471+0000 7fd12659c640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd10806b840 0x7fd10806dd00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd128154510 tx=0x7fd118009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.471+0000 7fd126d9d640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd108071d50 0x7fd108074190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.471+0000 7fd126d9d640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd108071d50 0x7fd108074190 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.472+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== osd.4 v2:172.21.15.150:6816/1744914063 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd108074860 con 0x7fd108071d50 2024-02-19T15:04:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.485+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 --> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd1281585c0 con 0x7fd108071d50 2024-02-19T15:04:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd11effd640 1 -- 172.21.15.78:0/388216718 <== osd.4 v2:172.21.15.150:6816/1744914063 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd120040410 con 0x7fd108071d50 2024-02-19T15:04:47.489 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:47.489 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:47.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd108071d50 msgr2=0x7fd108074190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.150:6816/1744914063,v1:172.21.15.150:6817/1744914063] conn(0x7fd108071d50 0x7fd108074190 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd10806b840 msgr2=0x7fd10806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fd10806b840 0x7fd10806dd00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd128154510 tx=0x7fd118009040 comp rx=0 tx=0).stop 2024-02-19T15:04:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd128152670 msgr2=0x7fd1280ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 --2- 172.21.15.78:0/388216718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd128152670 0x7fd1280ae9e0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fd11000e990 tx=0x7fd11000ee50 comp rx=0 tx=0).stop 2024-02-19T15:04:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd126d9d640 1 -- 172.21.15.78:0/388216718 reap_dead start 2024-02-19T15:04:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 shutdown_connections 2024-02-19T15:04:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 >> 172.21.15.78:0/388216718 conn(0x7fd128072790 msgr2=0x7fd1281552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 shutdown_connections 2024-02-19T15:04:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.486+0000 7fd12caed640 1 -- 172.21.15.78:0/388216718 wait complete. 2024-02-19T15:04:47.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-02-19T15:04:47.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.597+0000 7f8e70b1e640 1 Processor -- start 2024-02-19T15:04:47.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.597+0000 7f8e70b1e640 1 -- start start 2024-02-19T15:04:47.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e70b1e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e6c0c8500 0x7f8e6c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e70b1e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e6c0c2f50 0x7f8e6c158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e70b1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c0c34a0 con 0x7f8e6c0c42e0 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e70b1e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c06c090 con 0x7f8e6c0c2f50 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e70b1e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c06d6a0 con 0x7f8e6c0c8500 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e6c0c2f50 0x7f8e6c158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e63fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8e6c0c42e0 0x7f8e6c158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35678/0 (socket says 172.21.15.78:35678) 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8e6c0c2f50 0x7f8e6c158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52956/0 (socket says 172.21.15.78:52956) 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e63fff640 1 -- 172.21.15.78:0/3304016642 learned_addr learned my addr 172.21.15.78:0/3304016642 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:47.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6a59c640 1 --2- 172.21.15.78:0/3304016642 >> v2:172.21.15.150:3300/0 conn(0x7f8e6c0c8500 0x7f8e6c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6ad9d640 1 -- 172.21.15.78:0/3304016642 >> v2:172.21.15.150:3300/0 conn(0x7f8e6c0c8500 msgr2=0x7f8e6c0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6ad9d640 1 --2- 172.21.15.78:0/3304016642 >> v2:172.21.15.150:3300/0 conn(0x7f8e6c0c8500 0x7f8e6c0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6ad9d640 1 -- 172.21.15.78:0/3304016642 >> v1:172.21.15.78:6789/0 conn(0x7f8e6c0c42e0 legacy=0x7f8e6c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e69d9b640 1 -- 172.21.15.78:0/3304016642 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211021535 0 0) 0x7f8e6c0c34a0 con 0x7f8e6c0c42e0 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.598+0000 7f8e6ad9d640 1 -- 172.21.15.78:0/3304016642 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c06c8e0 con 0x7f8e6c0c2f50 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e6ad9d640 1 --2- 172.21.15.78:0/3304016642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c158d60 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0c9350 tx=0x7f8e6402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89a69eee0dbbd3a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e69d9b640 1 -- 172.21.15.78:0/3304016642 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6402cbd0 con 0x7f8e6c0c2f50 2024-02-19T15:04:47.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e69d9b640 1 -- 172.21.15.78:0/3304016642 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6402cd30 con 0x7f8e6c0c2f50 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e69d9b640 1 -- 172.21.15.78:0/3304016642 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6402cea0 con 0x7f8e6c0c2f50 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e70b1e640 1 -- 172.21.15.78:0/3304016642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c2f50 msgr2=0x7f8e6c158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/3304016642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c158d60 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0c9350 tx=0x7f8e6402be20 comp rx=0 tx=0).stop 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e70b1e640 1 -- 172.21.15.78:0/3304016642 shutdown_connections 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/3304016642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c158d60 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/3304016642 >> v2:172.21.15.150:3300/0 conn(0x7f8e6c0c8500 0x7f8e6c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.599+0000 7f8e70b1e640 1 -- 172.21.15.78:0/3304016642 >> 172.21.15.78:0/3304016642 conn(0x7f8e6c0aa300 msgr2=0x7f8e6c0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.600+0000 7f8e70b1e640 1 -- 172.21.15.78:0/3304016642 shutdown_connections 2024-02-19T15:04:47.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.600+0000 7f8e70b1e640 1 -- 172.21.15.78:0/3304016642 wait complete. 2024-02-19T15:04:47.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 Processor -- start 2024-02-19T15:04:47.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 -- start start 2024-02-19T15:04:47.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c14da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 0x7f8e6c06fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e6c070700 0x7f8e6c14b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c070c00 con 0x7f8e6c0c8500 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c070ee0 con 0x7f8e6c0c2f50 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e70b1e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c0711c0 con 0x7f8e6c070700 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 0x7f8e6c06fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 0x7f8e6c06fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52972/0 (socket says 172.21.15.78:52972) 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.601+0000 7f8e63fff640 1 -- 172.21.15.78:0/2681987790 learned_addr learned my addr 172.21.15.78:0/2681987790 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e6a59c640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e6ad9d640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e6c070700 0x7f8e6c14b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e63fff640 1 -- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e6c070700 msgr2=0x7f8e6c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e63fff640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e6c070700 0x7f8e6c14b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e63fff640 1 -- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e6c0c2f50 msgr2=0x7f8e6c14da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e63fff640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c14da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e63fff640 1 -- 172.21.15.78:0/2681987790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e64009a70 con 0x7f8e6c0c8500 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e6a59c640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c14da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e63fff640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 0x7f8e6c06fc70 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400c4d0 tx=0x7f8e5400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e540074e0 con 0x7f8e6c0c8500 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e54007660 con 0x7f8e6c0c8500 2024-02-19T15:04:47.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c150ae0 con 0x7f8e6c0c8500 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54004a20 con 0x7f8e6c0c8500 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.602+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c14df70 con 0x7f8e6c0c8500 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.603+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c071c20 con 0x7f8e6c0c8500 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.604+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f8e54004b80 con 0x7f8e6c0c8500 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.604+0000 7f8e62ffd640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e4c06b810 0x7f8e4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.604+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8e54089670 con 0x7f8e6c0c8500 2024-02-19T15:04:47.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.604+0000 7f8e6a59c640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e4c06b810 0x7f8e4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.604+0000 7f8e6a59c640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e4c06b810 0x7f8e4c06dcd0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009a20 tx=0x7f8e5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.608+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8e54054100 con 0x7f8e6c0c8500 2024-02-19T15:04:47.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.782+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e6c06cea0 con 0x7f8e4c06b810 2024-02-19T15:04:47.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.782+0000 7f8e62ffd640 1 -- 172.21.15.78:0/2681987790 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e5c004030 con 0x7f8e4c06b810 2024-02-19T15:04:47.785 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 4 to deep-scrub 2024-02-19T15:04:47.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e4c06b810 msgr2=0x7f8e4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e4c06b810 0x7f8e4c06dcd0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009a20 tx=0x7f8e5c008040 comp rx=0 tx=0).stop 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 msgr2=0x7f8e6c06fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 0x7f8e6c06fc70 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400c4d0 tx=0x7f8e5400c990 comp rx=0 tx=0).stop 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 shutdown_connections 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f8e4c06b810 0x7f8e4c06dcd0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0c8500 0x7f8e6c06fc70 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e6c070700 0x7f8e6c14b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 --2- 172.21.15.78:0/2681987790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e6c0c2f50 0x7f8e6c14da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 >> 172.21.15.78:0/2681987790 conn(0x7f8e6c0aa300 msgr2=0x7f8e6c0c4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 shutdown_connections 2024-02-19T15:04:47.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.784+0000 7f8e70b1e640 1 -- 172.21.15.78:0/2681987790 wait complete. 2024-02-19T15:04:47.798 INFO:tasks.ceph:Scrubbing osd.5 2024-02-19T15:04:47.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.899+0000 7ff6df1b6640 1 Processor -- start 2024-02-19T15:04:47.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.899+0000 7ff6df1b6640 1 -- start start 2024-02-19T15:04:47.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6df1b6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 0x7ff6d806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6df1b6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 0x7ff6d80c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6df1b6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d806dc70 con 0x7ff6d80c56f0 2024-02-19T15:04:47.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6df1b6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d8158ef0 con 0x7ff6d8152670 2024-02-19T15:04:47.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6df1b6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d80be190 con 0x7ff6d80c8730 2024-02-19T15:04:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6dcf52640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 0x7ff6d806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6d7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 0x7ff6d80c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6d7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 0x7ff6d80c51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:45412/0 (socket says 172.21.15.78:45412) 2024-02-19T15:04:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6dcf52640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 0x7ff6d806d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.78:47502/0 (socket says 172.21.15.78:47502) 2024-02-19T15:04:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6d7fff640 1 -- 172.21.15.78:0/2125219759 learned_addr learned my addr 172.21.15.78:0/2125219759 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.900+0000 7ff6dd753640 1 --1- 172.21.15.78:0/2125219759 >> v1:172.21.15.150:6789/0 conn(0x7ff6d80c56f0 0x7ff6d8154ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46730/0 (socket says 172.21.15.78:46730) 2024-02-19T15:04:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.901+0000 7ff6d7fff640 1 -- 172.21.15.78:0/2125219759 >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 msgr2=0x7ff6d806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.901+0000 7ff6d7fff640 1 --2- 172.21.15.78:0/2125219759 >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 0x7ff6d806d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.901+0000 7ff6d7fff640 1 -- 172.21.15.78:0/2125219759 >> v1:172.21.15.150:6789/0 conn(0x7ff6d80c56f0 legacy=0x7ff6d8154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.901+0000 7ff6d7fff640 1 -- 172.21.15.78:0/2125219759 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d806dea0 con 0x7ff6d80c8730 2024-02-19T15:04:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.901+0000 7ff6dcf52640 1 --2- 172.21.15.78:0/2125219759 >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 0x7ff6d806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:47.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6d7fff640 1 --2- 172.21.15.78:0/2125219759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 0x7ff6d80c51b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009870 tx=0x7ff6cc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a647aee2c20e21 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6d77fe640 1 -- 172.21.15.78:0/2125219759 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0317c0 con 0x7ff6d80c8730 2024-02-19T15:04:47.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6d77fe640 1 -- 172.21.15.78:0/2125219759 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc031920 con 0x7ff6d80c8730 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6d77fe640 1 -- 172.21.15.78:0/2125219759 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc031cb0 con 0x7ff6d80c8730 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 -- 172.21.15.78:0/2125219759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 msgr2=0x7ff6d80c51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 --2- 172.21.15.78:0/2125219759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 0x7ff6d80c51b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009870 tx=0x7ff6cc02b860 comp rx=0 tx=0).stop 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 -- 172.21.15.78:0/2125219759 shutdown_connections 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 --2- 172.21.15.78:0/2125219759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c8730 0x7ff6d80c51b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 --2- 172.21.15.78:0/2125219759 >> v2:172.21.15.150:3300/0 conn(0x7ff6d8152670 0x7ff6d806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 -- 172.21.15.78:0/2125219759 >> 172.21.15.78:0/2125219759 conn(0x7ff6d8072790 msgr2=0x7ff6d8072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 -- 172.21.15.78:0/2125219759 shutdown_connections 2024-02-19T15:04:47.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.902+0000 7ff6df1b6640 1 -- 172.21.15.78:0/2125219759 wait complete. 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.903+0000 7ff6df1b6640 1 Processor -- start 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.903+0000 7ff6df1b6640 1 -- start start 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6df1b6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c56f0 0x7ff6d81dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6df1b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d80c8730 0x7ff6d80aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6df1b6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6d8152670 0x7ff6d81e0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6df1b6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8157b20 con 0x7ff6d8152670 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6df1b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d81de340 con 0x7ff6d80c8730 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6df1b6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d81de5e0 con 0x7ff6d80c56f0 2024-02-19T15:04:47.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6dd753640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6d8152670 0x7ff6d81e0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6dd753640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6d8152670 0x7ff6d81e0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52984/0 (socket says 172.21.15.78:52984) 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.904+0000 7ff6dd753640 1 -- 172.21.15.78:0/3007263167 learned_addr learned my addr 172.21.15.78:0/3007263167 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6d7fff640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d80c8730 0x7ff6d80aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dcf52640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c56f0 0x7ff6d81dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dd753640 1 -- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c56f0 msgr2=0x7ff6d81dfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dd753640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff6d80c56f0 0x7ff6d81dfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dd753640 1 -- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d80c8730 msgr2=0x7ff6d80aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dd753640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6d80c8730 0x7ff6d80aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dd753640 1 -- 172.21.15.78:0/3007263167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c8009400 con 0x7ff6d8152670 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6dd753640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6d8152670 0x7ff6d81e0730 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c730 tx=0x7ff6d000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0004d60 con 0x7ff6d8152670 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000a640 con 0x7ff6d8152670 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc0094d0 con 0x7ff6d8152670 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000a9f0 con 0x7ff6d8152670 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.905+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d81dec50 con 0x7ff6d8152670 2024-02-19T15:04:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.906+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6d81588a0 con 0x7ff6d8152670 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.906+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff6d000ab50 con 0x7ff6d8152670 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6d57fa640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff6c406b810 0x7ff6c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff6d00897a0 con 0x7ff6d8152670 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6dcf52640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff6c406b810 0x7ff6c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6d57fa640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7ff6c4071e90 0x7ff6c40742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 --> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6c40749a0 con 0x7ff6c4071e90 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6d008be60 con 0x7ff6d8152670 2024-02-19T15:04:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6d7fff640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7ff6c4071e90 0x7ff6c40742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.907+0000 7ff6dcf52640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff6c406b810 0x7ff6c406dcd0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009bc0 tx=0x7ff6c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.908+0000 7ff6d7fff640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7ff6c4071e90 0x7ff6c40742d0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:47.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.908+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== osd.5 v2:172.21.15.150:6800/911852656 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff6c40749a0 con 0x7ff6c4071e90 2024-02-19T15:04:47.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.921+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 --> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff6d81585c0 con 0x7ff6c4071e90 2024-02-19T15:04:47.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.922+0000 7ff6d57fa640 1 -- 172.21.15.78:0/3007263167 <== osd.5 v2:172.21.15.150:6800/911852656 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff6cc031b10 con 0x7ff6c4071e90 2024-02-19T15:04:47.925 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:47.925 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:47.925 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.922+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7ff6c4071e90 msgr2=0x7ff6c40742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.922+0000 7ff6df1b6640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.150:6800/911852656,v1:172.21.15.150:6801/911852656] conn(0x7ff6c4071e90 0x7ff6c40742d0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:47.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.922+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff6c406b810 msgr2=0x7ff6c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff6c406b810 0x7ff6c406dcd0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009bc0 tx=0x7ff6c8008040 comp rx=0 tx=0).stop 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6d8152670 msgr2=0x7ff6d81e0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 --2- 172.21.15.78:0/3007263167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6d8152670 0x7ff6d81e0730 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7ff6d000c730 tx=0x7ff6d000cb30 comp rx=0 tx=0).stop 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6dd753640 1 -- 172.21.15.78:0/3007263167 reap_dead start 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 shutdown_connections 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 >> 172.21.15.78:0/3007263167 conn(0x7ff6d8072790 msgr2=0x7ff6d8072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 shutdown_connections 2024-02-19T15:04:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:47.923+0000 7ff6df1b6640 1 -- 172.21.15.78:0/3007263167 wait complete. 2024-02-19T15:04:47.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-02-19T15:04:48.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.033+0000 7f62efe0e640 1 Processor -- start 2024-02-19T15:04:48.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.033+0000 7f62efe0e640 1 -- start start 2024-02-19T15:04:48.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.033+0000 7f62efe0e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62e80ac390 0x7f62e80ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62efe0e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62e80c55f0 0x7f62e815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62efe0e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e8158c40 con 0x7f62e80bdf20 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62efe0e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e80c8de0 con 0x7f62e80c55f0 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62efe0e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e80c5a30 con 0x7f62e80ac390 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62e80c55f0 0x7f62e815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f62e80c55f0 0x7f62e815f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52986/0 (socket says 172.21.15.78:52986) 2024-02-19T15:04:48.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 -- 172.21.15.78:0/1861502646 learned_addr learned my addr 172.21.15.78:0/1861502646 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62edbaa640 1 --2- 172.21.15.78:0/1861502646 >> v2:172.21.15.150:3300/0 conn(0x7f62e80ac390 0x7f62e80ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 -- 172.21.15.78:0/1861502646 >> v2:172.21.15.150:3300/0 conn(0x7f62e80ac390 msgr2=0x7f62e80ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 --2- 172.21.15.78:0/1861502646 >> v2:172.21.15.150:3300/0 conn(0x7f62e80ac390 0x7f62e80ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 -- 172.21.15.78:0/1861502646 >> v1:172.21.15.150:6789/0 conn(0x7f62e80bdf20 legacy=0x7f62e8158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.034+0000 7f62ee3ab640 1 -- 172.21.15.78:0/1861502646 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e8155a60 con 0x7f62e80c55f0 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62ee3ab640 1 --2- 172.21.15.78:0/1861502646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e815f8c0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f62e80ad2a0 tx=0x7f62e002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ef6e4eaaaad0df server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62ecba8640 1 -- 172.21.15.78:0/1861502646 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e002c9f0 con 0x7f62e80c55f0 2024-02-19T15:04:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62ecba8640 1 -- 172.21.15.78:0/1861502646 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e002cb50 con 0x7f62e80c55f0 2024-02-19T15:04:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62ecba8640 1 -- 172.21.15.78:0/1861502646 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e00343f0 con 0x7f62e80c55f0 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62efe0e640 1 -- 172.21.15.78:0/1861502646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 msgr2=0x7f62e815f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62efe0e640 1 --2- 172.21.15.78:0/1861502646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e815f8c0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f62e80ad2a0 tx=0x7f62e002be50 comp rx=0 tx=0).stop 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62efe0e640 1 -- 172.21.15.78:0/1861502646 shutdown_connections 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62efe0e640 1 --2- 172.21.15.78:0/1861502646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e815f8c0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62efe0e640 1 --2- 172.21.15.78:0/1861502646 >> v2:172.21.15.150:3300/0 conn(0x7f62e80ac390 0x7f62e80ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.035+0000 7f62efe0e640 1 -- 172.21.15.78:0/1861502646 >> 172.21.15.78:0/1861502646 conn(0x7f62e80ca890 msgr2=0x7f62e80caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.036+0000 7f62efe0e640 1 -- 172.21.15.78:0/1861502646 shutdown_connections 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.036+0000 7f62efe0e640 1 -- 172.21.15.78:0/1861502646 wait complete. 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 Processor -- start 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 -- start start 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62e80ac390 0x7f62e8151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e80bdf20 0x7f62e80c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e80bfa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e8152010 con 0x7f62e80c55f0 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e81edce0 con 0x7f62e80bdf20 2024-02-19T15:04:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62efe0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e81edfc0 con 0x7f62e80ac390 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62ee3ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e80bfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62ee3ab640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e80bfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53000/0 (socket says 172.21.15.78:53000) 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.037+0000 7f62ee3ab640 1 -- 172.21.15.78:0/2236998632 learned_addr learned my addr 172.21.15.78:0/2236998632 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62edbaa640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62e80ac390 0x7f62e8151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ed3a9640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e80bdf20 0x7f62e80c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ee3ab640 1 -- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62e80ac390 msgr2=0x7f62e8151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ee3ab640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62e80ac390 0x7f62e8151ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ee3ab640 1 -- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e80bdf20 msgr2=0x7f62e80c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ee3ab640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e80bdf20 0x7f62e80c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ee3ab640 1 -- 172.21.15.78:0/2236998632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e0009b30 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62ee3ab640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e80bfa30 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009f40 tx=0x7f62e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0009510 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e003e450 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e81ee2a0 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e003e820 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.038+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e81ee860 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.039+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e8074990 con 0x7f62e80c55f0 2024-02-19T15:04:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.039+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f62e003e980 con 0x7f62e80c55f0 2024-02-19T15:04:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.040+0000 7f62de7fc640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62cc06b840 0x7f62cc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.040+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f62e00ace40 con 0x7f62e80c55f0 2024-02-19T15:04:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.040+0000 7f62edbaa640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62cc06b840 0x7f62cc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.040+0000 7f62edbaa640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62cc06b840 0x7f62cc06dd00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009a10 tx=0x7f62e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.044+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62e00779f0 con 0x7f62e80c55f0 2024-02-19T15:04:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.215+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f62e80b0760 con 0x7f62cc06b840 2024-02-19T15:04:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.215+0000 7f62de7fc640 1 -- 172.21.15.78:0/2236998632 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f62e4010920 con 0x7f62cc06b840 2024-02-19T15:04:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 5 to deep-scrub 2024-02-19T15:04:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62cc06b840 msgr2=0x7f62cc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62cc06b840 0x7f62cc06dd00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009a10 tx=0x7f62e4008040 comp rx=0 tx=0).stop 2024-02-19T15:04:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 msgr2=0x7f62e80bfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e80bfa30 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009f40 tx=0x7f62e0004a40 comp rx=0 tx=0).stop 2024-02-19T15:04:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 shutdown_connections 2024-02-19T15:04:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f62cc06b840 0x7f62cc06dd00 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62e80c55f0 0x7f62e80bfa30 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e80bdf20 0x7f62e80c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 --2- 172.21.15.78:0/2236998632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f62e80ac390 0x7f62e8151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 >> 172.21.15.78:0/2236998632 conn(0x7f62e80ca890 msgr2=0x7f62e80cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 shutdown_connections 2024-02-19T15:04:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.217+0000 7f62efe0e640 1 -- 172.21.15.78:0/2236998632 wait complete. 2024-02-19T15:04:48.230 INFO:tasks.ceph:Scrubbing osd.6 2024-02-19T15:04:48.231 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.332+0000 7ff8d7a8f640 1 Processor -- start 2024-02-19T15:04:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.332+0000 7ff8d7a8f640 1 -- start start 2024-02-19T15:04:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d7a8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 0x7ff8d00c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d7a8f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8d0157050 0x7ff8d0152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d7a8f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d0070f10 con 0x7ff8d00c5360 2024-02-19T15:04:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d7a8f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d00bf540 con 0x7ff8d0157050 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d7a8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0071420 con 0x7ff8d00becb0 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 0x7ff8d00c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 0x7ff8d00c4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.78:45426/0 (socket says 172.21.15.78:45426) 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d502a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8d0157050 0x7ff8d0152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 -- 172.21.15.78:0/2531444710 learned_addr learned my addr 172.21.15.78:0/2531444710 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d602c640 1 --1- 172.21.15.78:0/2531444710 >> v1:172.21.15.150:6789/0 conn(0x7ff8d00c5360 0x7ff8d00c6600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.78:46744/0 (socket says 172.21.15.78:46744) 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 -- 172.21.15.78:0/2531444710 >> v2:172.21.15.150:3300/0 conn(0x7ff8d0157050 msgr2=0x7ff8d0152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 --2- 172.21.15.78:0/2531444710 >> v2:172.21.15.150:3300/0 conn(0x7ff8d0157050 0x7ff8d0152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 -- 172.21.15.78:0/2531444710 >> v1:172.21.15.150:6789/0 conn(0x7ff8d00c5360 legacy=0x7ff8d00c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d582b640 1 -- 172.21.15.78:0/2531444710 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d00c5970 con 0x7ff8d00becb0 2024-02-19T15:04:48.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.333+0000 7ff8d502a640 1 --2- 172.21.15.78:0/2531444710 >> v2:172.21.15.150:3300/0 conn(0x7ff8d0157050 0x7ff8d0152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T15:04:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.334+0000 7ff8d582b640 1 --2- 172.21.15.78:0/2531444710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 0x7ff8d00c4e20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009870 tx=0x7ff8cc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699bec86511ca54e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.334+0000 7ff8d4829640 1 -- 172.21.15.78:0/2531444710 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc030840 con 0x7ff8d00becb0 2024-02-19T15:04:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.334+0000 7ff8d4829640 1 -- 172.21.15.78:0/2531444710 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc0309a0 con 0x7ff8d00becb0 2024-02-19T15:04:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.334+0000 7ff8d4829640 1 -- 172.21.15.78:0/2531444710 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc030d30 con 0x7ff8d00becb0 2024-02-19T15:04:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/2531444710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 msgr2=0x7ff8d00c4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 --2- 172.21.15.78:0/2531444710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 0x7ff8d00c4e20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009870 tx=0x7ff8cc02b980 comp rx=0 tx=0).stop 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/2531444710 shutdown_connections 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 --2- 172.21.15.78:0/2531444710 >> v2:172.21.15.150:3300/0 conn(0x7ff8d0157050 0x7ff8d0152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 --2- 172.21.15.78:0/2531444710 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00becb0 0x7ff8d00c4e20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/2531444710 >> 172.21.15.78:0/2531444710 conn(0x7ff8d00aa370 msgr2=0x7ff8d00aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/2531444710 shutdown_connections 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.335+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/2531444710 wait complete. 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.336+0000 7ff8d7a8f640 1 Processor -- start 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.336+0000 7ff8d7a8f640 1 -- start start 2024-02-19T15:04:48.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d00becb0 0x7ff8d00abee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d00c5360 0x7ff8d01ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d0157050 0x7ff8d01e5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01510d0 con 0x7ff8d00becb0 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0152fb0 con 0x7ff8d00c5360 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01ecac0 con 0x7ff8d0157050 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d00becb0 0x7ff8d00abee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d00becb0 0x7ff8d00abee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53010/0 (socket says 172.21.15.78:53010) 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 -- 172.21.15.78:0/3117387101 learned_addr learned my addr 172.21.15.78:0/3117387101 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d502a640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d00c5360 0x7ff8d01ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d602c640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d0157050 0x7ff8d01e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 -- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d0157050 msgr2=0x7ff8d01e5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d0157050 0x7ff8d01e5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 -- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d00c5360 msgr2=0x7ff8d01ebd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d00c5360 0x7ff8d01ebd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 -- 172.21.15.78:0/3117387101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c0009400 con 0x7ff8d00becb0 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d582b640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d00becb0 0x7ff8d00abee0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc000c00 tx=0x7ff8cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc039c90 con 0x7ff8d00becb0 2024-02-19T15:04:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc039df0 con 0x7ff8d00becb0 2024-02-19T15:04:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc0094d0 con 0x7ff8d00becb0 2024-02-19T15:04:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc043640 con 0x7ff8d00becb0 2024-02-19T15:04:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.337+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d01ec7e0 con 0x7ff8d00becb0 2024-02-19T15:04:48.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.338+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8d00bf1d0 con 0x7ff8d00becb0 2024-02-19T15:04:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7ff8cc0437a0 con 0x7ff8d00becb0 2024-02-19T15:04:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8c67fc640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff8b406b810 0x7ff8b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff8cc03e070 con 0x7ff8d00becb0 2024-02-19T15:04:48.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8d502a640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff8b406b810 0x7ff8b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8c67fc640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff8b4071e90 0x7ff8b40742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 --> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8b40749a0 con 0x7ff8b4071e90 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.339+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8cc0af4e0 con 0x7ff8d00becb0 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.340+0000 7ff8d602c640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff8b4071e90 0x7ff8b40742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.340+0000 7ff8d502a640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff8b406b810 0x7ff8b406dcd0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00c2790 tx=0x7ff8c0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.340+0000 7ff8d602c640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff8b4071e90 0x7ff8b40742d0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.341+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== osd.6 v2:172.21.15.150:6808/3532732509 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff8b40749a0 con 0x7ff8b4071e90 2024-02-19T15:04:48.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.354+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 --> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff8d006f670 con 0x7ff8b4071e90 2024-02-19T15:04:48.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.354+0000 7ff8c67fc640 1 -- 172.21.15.78:0/3117387101 <== osd.6 v2:172.21.15.150:6808/3532732509 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff8c8004de0 con 0x7ff8b4071e90 2024-02-19T15:04:48.357 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:48.358 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:48.358 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff8b4071e90 msgr2=0x7ff8b40742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.150:6808/3532732509,v1:172.21.15.150:6809/3532732509] conn(0x7ff8b4071e90 0x7ff8b40742d0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff8b406b810 msgr2=0x7ff8b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7ff8b406b810 0x7ff8b406dcd0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00c2790 tx=0x7ff8c0009190 comp rx=0 tx=0).stop 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d00becb0 msgr2=0x7ff8d00abee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 --2- 172.21.15.78:0/3117387101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8d00becb0 0x7ff8d00abee0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc000c00 tx=0x7ff8cc004730 comp rx=0 tx=0).stop 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d602c640 1 -- 172.21.15.78:0/3117387101 reap_dead start 2024-02-19T15:04:48.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 shutdown_connections 2024-02-19T15:04:48.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 >> 172.21.15.78:0/3117387101 conn(0x7ff8d00aa370 msgr2=0x7ff8d00c6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 shutdown_connections 2024-02-19T15:04:48.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.355+0000 7ff8d7a8f640 1 -- 172.21.15.78:0/3117387101 wait complete. 2024-02-19T15:04:48.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-02-19T15:04:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.465+0000 7f152a6ed640 1 Processor -- start 2024-02-19T15:04:48.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f152a6ed640 1 -- start start 2024-02-19T15:04:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f152a6ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15240c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f152a6ed640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f15240c15b0 0x7f15240c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f152a6ed640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15240c1120 con 0x7f15240c5270 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f152a6ed640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15240c6080 con 0x7f15240c15b0 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f152a6ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15240c57e0 con 0x7f152414ece0 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f1528c8a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f15240c15b0 0x7f15240c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f1528c8a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f15240c15b0 0x7f15240c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53018/0 (socket says 172.21.15.78:53018) 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.466+0000 7f1528c8a640 1 -- 172.21.15.78:0/3762786082 learned_addr learned my addr 172.21.15.78:0/3762786082 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.467+0000 7f1523fff640 1 --2- 172.21.15.78:0/3762786082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15240c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.467+0000 7f1528c8a640 1 -- 172.21.15.78:0/3762786082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 msgr2=0x7f15240c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.467+0000 7f1528c8a640 1 --2- 172.21.15.78:0/3762786082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15240c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.467+0000 7f1528c8a640 1 -- 172.21.15.78:0/3762786082 >> v1:172.21.15.150:6789/0 conn(0x7f15240c5270 legacy=0x7f15240719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.467+0000 7f1528c8a640 1 -- 172.21.15.78:0/3762786082 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240c5ac0 con 0x7f15240c15b0 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.467+0000 7f1528c8a640 1 --2- 172.21.15.78:0/3762786082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15240c1a40 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f151c009e50 tx=0x7f151c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541df3b3245e679f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f1522ffd640 1 -- 172.21.15.78:0/3762786082 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c02cb00 con 0x7f15240c15b0 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f1522ffd640 1 -- 172.21.15.78:0/3762786082 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c02cc60 con 0x7f15240c15b0 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f1522ffd640 1 -- 172.21.15.78:0/3762786082 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0343f0 con 0x7f15240c15b0 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 -- 172.21.15.78:0/3762786082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 msgr2=0x7f15240c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 --2- 172.21.15.78:0/3762786082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15240c1a40 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f151c009e50 tx=0x7f151c02bf60 comp rx=0 tx=0).stop 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 -- 172.21.15.78:0/3762786082 shutdown_connections 2024-02-19T15:04:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 --2- 172.21.15.78:0/3762786082 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15240c1a40 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 --2- 172.21.15.78:0/3762786082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15240c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 -- 172.21.15.78:0/3762786082 >> 172.21.15.78:0/3762786082 conn(0x7f152406b270 msgr2=0x7f152406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 -- 172.21.15.78:0/3762786082 shutdown_connections 2024-02-19T15:04:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.468+0000 7f152a6ed640 1 -- 172.21.15.78:0/3762786082 wait complete. 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.469+0000 7f152a6ed640 1 Processor -- start 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.469+0000 7f152a6ed640 1 -- start start 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f152a6ed640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15241e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f152a6ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15240c5270 0x7f1524142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f152a6ed640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15241e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f152a6ed640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524151ff0 con 0x7f15240c15b0 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f152a6ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15241e9480 con 0x7f15240c5270 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f152a6ed640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152414a620 con 0x7f152414ece0 2024-02-19T15:04:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15241e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15241e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53022/0 (socket says 172.21.15.78:53022) 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 -- 172.21.15.78:0/1184035442 learned_addr learned my addr 172.21.15.78:0/1184035442 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f15237fe640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15240c5270 0x7f1524142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1528c8a640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15241e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 -- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 msgr2=0x7f15241e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15241e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 -- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15240c5270 msgr2=0x7f1524142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15240c5270 0x7f1524142c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.470+0000 7f1523fff640 1 -- 172.21.15.78:0/1184035442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c009b30 con 0x7f15240c15b0 2024-02-19T15:04:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.471+0000 7f1523fff640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15241e8f40 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f151400d5d0 tx=0x7f151400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.471+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15140125f0 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.471+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514012770 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.471+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1524073390 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.471+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514012b20 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.471+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1524143160 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.472+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1524074990 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.472+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f1514012c80 con 0x7f15240c15b0 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.472+0000 7f1520ff9640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f151006b810 0x7f151006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.473+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f151408a560 con 0x7f15240c15b0 2024-02-19T15:04:48.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.473+0000 7f15237fe640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f151006b810 0x7f151006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.473+0000 7f15237fe640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f151006b810 0x7f151006dcd0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f15240bff10 tx=0x7f1518009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.477+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1514055080 con 0x7f15240c15b0 2024-02-19T15:04:48.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.648+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f1524073210 con 0x7f151006b810 2024-02-19T15:04:48.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.648+0000 7f1520ff9640 1 -- 172.21.15.78:0/1184035442 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1518001c70 con 0x7f151006b810 2024-02-19T15:04:48.651 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 6 to deep-scrub 2024-02-19T15:04:48.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f151006b810 msgr2=0x7f151006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f151006b810 0x7f151006dcd0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f15240bff10 tx=0x7f1518009040 comp rx=0 tx=0).stop 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 msgr2=0x7f15241e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15241e8f40 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f151400d5d0 tx=0x7f151400da90 comp rx=0 tx=0).stop 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 shutdown_connections 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f151006b810 0x7f151006dcd0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f152414ece0 0x7f15241e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15240c5270 0x7f1524142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 --2- 172.21.15.78:0/1184035442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15240c15b0 0x7f15241e8f40 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 >> 172.21.15.78:0/1184035442 conn(0x7f152406b270 msgr2=0x7f152414d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.650+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 shutdown_connections 2024-02-19T15:04:48.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.651+0000 7f152a6ed640 1 -- 172.21.15.78:0/1184035442 wait complete. 2024-02-19T15:04:48.664 INFO:tasks.ceph:Scrubbing osd.7 2024-02-19T15:04:48.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-02-19T15:04:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.766+0000 7f4173125640 1 Processor -- start 2024-02-19T15:04:48.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.766+0000 7f4173125640 1 -- start start 2024-02-19T15:04:48.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f4173125640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f416c152ac0 0x7f416c0c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f4173125640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c06c570 con 0x7f416c0c59c0 2024-02-19T15:04:48.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f4173125640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0c52d0 con 0x7f416c0c79d0 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f4173125640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c06caa0 con 0x7f416c152ac0 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f41716c2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f416c0c59c0 0x7f416c158600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35680/0 (socket says 172.21.15.78:35680) 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f41716c2640 1 -- 172.21.15.78:0/3572726171 learned_addr learned my addr 172.21.15.78:0/3572726171 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676638773 0 0) 0x7f416c06c570 con 0x7f416c0c59c0 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c003520 con 0x7f416c0c59c0 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416bfff640 1 --2- 172.21.15.78:0/3572726171 >> v2:172.21.15.150:3300/0 conn(0x7f416c152ac0 0x7f416c0c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183255724 0 0) 0x7f415c003520 con 0x7f416c0c59c0 2024-02-19T15:04:48.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 >> v2:172.21.15.150:3300/0 conn(0x7f416c152ac0 msgr2=0x7f416c0c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 --2- 172.21.15.78:0/3572726171 >> v2:172.21.15.150:3300/0 conn(0x7f416c152ac0 0x7f416c0c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 >> v1:172.21.15.150:6789/0 conn(0x7f416c0c79d0 legacy=0x7f416c1523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0c5760 con 0x7f416c0c59c0 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f41640019f0 con 0x7f416c0c59c0 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 >> v1:172.21.15.78:6789/0 conn(0x7f416c0c59c0 legacy=0x7f416c158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f415c003970 0x7f415c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c0047c0 0x7f415c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c005180 0x7f415c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c06caa0 con 0x7f415c0047c0 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c00c120 con 0x7f415c003970 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.767+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c00c2e0 con 0x7f415c005180 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f416bfff640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c0047c0 0x7f415c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f4170ec1640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f415c003970 0x7f415c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f41716c2640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c005180 0x7f415c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f416bfff640 1 -- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c005180 msgr2=0x7f415c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f416bfff640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c005180 0x7f415c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f416bfff640 1 -- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f415c003970 msgr2=0x7f415c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f416bfff640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f415c003970 0x7f415c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f416bfff640 1 -- 172.21.15.78:0/3572726171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c003520 con 0x7f415c0047c0 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f41716c2640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c005180 0x7f415c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.768+0000 7f41716c2640 1 -- 172.21.15.78:0/3572726171 reap_dead start 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f416bfff640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c0047c0 0x7f415c004c40 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f4158009ac0 tx=0x7f415c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fffff031fa81d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415802c6b0 con 0x7f415c0047c0 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415802c810 con 0x7f415c0047c0 2024-02-19T15:04:48.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f416b7fe640 1 -- 172.21.15.78:0/3572726171 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415802cba0 con 0x7f415c0047c0 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 -- 172.21.15.78:0/3572726171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c0047c0 msgr2=0x7f415c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c0047c0 0x7f415c004c40 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f4158009ac0 tx=0x7f415c0057f0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 -- 172.21.15.78:0/3572726171 shutdown_connections 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 --2- 172.21.15.78:0/3572726171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c0047c0 0x7f415c004c40 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 -- 172.21.15.78:0/3572726171 >> 172.21.15.78:0/3572726171 conn(0x7f416c0ac750 msgr2=0x7f416c0acb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 -- 172.21.15.78:0/3572726171 shutdown_connections 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.769+0000 7f4173125640 1 -- 172.21.15.78:0/3572726171 wait complete. 2024-02-19T15:04:48.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.770+0000 7f4173125640 1 Processor -- start 2024-02-19T15:04:48.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.770+0000 7f4173125640 1 -- start start 2024-02-19T15:04:48.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4173125640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c0047c0 0x7f416c0bddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4173125640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0af7b0 0x7f416c0ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4173125640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f416c0adf80 0x7f416c1edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4173125640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0adaf0 con 0x7f416c0af7b0 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4173125640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c1eb020 con 0x7f416c0adf80 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4173125640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c1eb550 con 0x7f415c0047c0 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0af7b0 0x7f416c0ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0af7b0 0x7f416c0ab4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53036/0 (socket says 172.21.15.78:53036) 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 -- 172.21.15.78:0/2714086738 learned_addr learned my addr 172.21.15.78:0/2714086738 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f41716c2640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f416c0adf80 0x7f416c1edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f4170ec1640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c0047c0 0x7f416c0bddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 -- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c0047c0 msgr2=0x7f416c0bddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f415c0047c0 0x7f416c0bddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 -- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f416c0adf80 msgr2=0x7f416c1edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.771+0000 7f416bfff640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f416c0adf80 0x7f416c1edb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f416bfff640 1 -- 172.21.15.78:0/2714086738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0c0730 con 0x7f416c0af7b0 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f416bfff640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0af7b0 0x7f416c0ab4c0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f4158037040 tx=0x7f4158004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158003810 con 0x7f416c0af7b0 2024-02-19T15:04:48.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158003970 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158003d20 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c0aba00 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.772+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c0ad780 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.773+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f416c0c7030 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.773+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f4158034440 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41697fa640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f415006b810 0x7f415006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f41580ac010 con 0x7f416c0af7b0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f4170ec1640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f415006b810 0x7f415006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41697fa640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f4150071e90 0x7f41500742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 --> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41500749a0 con 0x7f4150071e90 2024-02-19T15:04:48.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=220) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41580ae6d0 con 0x7f416c0af7b0 2024-02-19T15:04:48.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41716c2640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f4150071e90 0x7f41500742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f4170ec1640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f415006b810 0x7f415006dcd0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4160000c00 tx=0x7f4160009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.774+0000 7f41716c2640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f4150071e90 0x7f41500742d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.775+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== osd.7 v2:172.21.15.150:6819/1386990918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f41500749a0 con 0x7f4150071e90 2024-02-19T15:04:48.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.788+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 --> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f416c0c6d50 con 0x7f4150071e90 2024-02-19T15:04:48.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f41697fa640 1 -- 172.21.15.78:0/2714086738 <== osd.7 v2:172.21.15.150:6819/1386990918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4164004b60 con 0x7f4150071e90 2024-02-19T15:04:48.792 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-02-19T15:04:48.792 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-19T15:04:48.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-02-19T15:04:48.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f4150071e90 msgr2=0x7f41500742d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.150:6819/1386990918,v1:172.21.15.150:6821/1386990918] conn(0x7f4150071e90 0x7f41500742d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f415006b810 msgr2=0x7f415006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f415006b810 0x7f415006dcd0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f4160000c00 tx=0x7f4160009040 comp rx=0 tx=0).stop 2024-02-19T15:04:48.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0af7b0 msgr2=0x7f416c0ab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 --2- 172.21.15.78:0/2714086738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f416c0af7b0 0x7f416c0ab4c0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f4158037040 tx=0x7f4158004730 comp rx=0 tx=0).stop 2024-02-19T15:04:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f41716c2640 1 -- 172.21.15.78:0/2714086738 reap_dead start 2024-02-19T15:04:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 shutdown_connections 2024-02-19T15:04:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 >> 172.21.15.78:0/2714086738 conn(0x7f416c0ac750 msgr2=0x7f416c151870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 shutdown_connections 2024-02-19T15:04:48.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.789+0000 7f4173125640 1 -- 172.21.15.78:0/2714086738 wait complete. 2024-02-19T15:04:48.802 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-02-19T15:04:48.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.901+0000 7effaf51e640 1 Processor -- start 2024-02-19T15:04:48.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.901+0000 7effaf51e640 1 -- start start 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effaf51e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7effa8156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effaf51e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa8155a50 con 0x7effa80c19f0 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effaf51e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa8155f80 con 0x7effa80c1dd0 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effaf51e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa8152af0 con 0x7effa8071410 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effad2ba640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7effa80c19f0 0x7effa80c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35682/0 (socket says 172.21.15.78:35682) 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effad2ba640 1 -- 172.21.15.78:0/1519824046 learned_addr learned my addr 172.21.15.78:0/1519824046 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7effacab9640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7effa8156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861002192 0 0) 0x7effa8155a50 con 0x7effa80c19f0 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff94003520 con 0x7effa80c19f0 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844925235 0 0) 0x7effa8155f80 con 0x7effa80c1dd0 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa8155a50 con 0x7effa80c1dd0 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067557510 0 0) 0x7eff94003520 con 0x7effa80c19f0 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 msgr2=0x7effa8156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7effa8156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 >> v1:172.21.15.150:6789/0 conn(0x7effa80c1dd0 legacy=0x7effa80c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa8070840 con 0x7effa80c19f0 2024-02-19T15:04:48.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7effa00019f0 con 0x7effa80c19f0 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.902+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 >> v1:172.21.15.78:6789/0 conn(0x7effa80c19f0 legacy=0x7effa80c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7eff9ffff640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff94003b30 0x7eff94003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7eff9ffff640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7eff940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7eff9ffff640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff94004d00 0x7eff94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8152af0 con 0x7eff94003b30 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9400bb70 con 0x7eff94004d00 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9400bd30 con 0x7effa8071410 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effad2ba640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff94003b30 0x7eff94003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effacab9640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7eff940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effadabb640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff94004d00 0x7eff94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effad2ba640 1 -- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 msgr2=0x7eff940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effad2ba640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7eff940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effad2ba640 1 -- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff94004d00 msgr2=0x7eff94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effad2ba640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff94004d00 0x7eff94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effad2ba640 1 -- 172.21.15.78:0/1519824046 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff94003520 con 0x7eff94003b30 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.903+0000 7effacab9640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effa8071410 0x7eff940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effad2ba640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff94003b30 0x7eff94003f30 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7effa000a7c0 tx=0x7eff940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18566a87c17ebdee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0034630 con 0x7eff94003b30 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa002f070 con 0x7eff94003b30 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7eff9ffff640 1 -- 172.21.15.78:0/1519824046 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa00349f0 con 0x7eff94003b30 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effaf51e640 1 -- 172.21.15.78:0/1519824046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff94003b30 msgr2=0x7eff94003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effaf51e640 1 --2- 172.21.15.78:0/1519824046 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff94003b30 0x7eff94003f30 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7effa000a7c0 tx=0x7eff940057f0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effadabb640 1 -- 172.21.15.78:0/1519824046 reap_dead start 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effaf51e640 1 -- 172.21.15.78:0/1519824046 shutdown_connections 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effaf51e640 1 -- 172.21.15.78:0/1519824046 >> 172.21.15.78:0/1519824046 conn(0x7effa80aa370 msgr2=0x7effa80aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effaf51e640 1 -- 172.21.15.78:0/1519824046 shutdown_connections 2024-02-19T15:04:48.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.904+0000 7effaf51e640 1 -- 172.21.15.78:0/1519824046 wait complete. 2024-02-19T15:04:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.905+0000 7effaf51e640 1 Processor -- start 2024-02-19T15:04:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 -- start start 2024-02-19T15:04:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff94003b30 0x7effa81eba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 0x7effa81e1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa80c5690 0x7effa81ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80ad840 con 0x7effa80ad3c0 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80ada50 con 0x7effa80c5690 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effaf51e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80add30 con 0x7eff94003b30 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 0x7effa81e1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 0x7effa81e1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53066/0 (socket says 172.21.15.78:53066) 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 -- 172.21.15.78:0/2961625817 learned_addr learned my addr 172.21.15.78:0/2961625817 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effad2ba640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff94003b30 0x7effa81eba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effadabb640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa80c5690 0x7effa81ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 -- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff94003b30 msgr2=0x7effa81eba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff94003b30 0x7effa81eba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 -- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa80c5690 msgr2=0x7effa81ee430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa80c5690 0x7effa81ee430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effacab9640 1 -- 172.21.15.78:0/2961625817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa81eaf70 con 0x7effa80ad3c0 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.906+0000 7effadabb640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa80c5690 0x7effa81ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.907+0000 7effacab9640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 0x7effa81e1fc0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7eff980116d0 tx=0x7eff98011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.907+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800f9c0 con 0x7effa80ad3c0 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.907+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9800fb20 con 0x7effa80ad3c0 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.907+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa81eb250 con 0x7effa80ad3c0 2024-02-19T15:04:48.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.907+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9800fed0 con 0x7effa80ad3c0 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.907+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa80c5b10 con 0x7effa80ad3c0 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.908+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80c8320 con 0x7effa80ad3c0 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.908+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7eff98015460 con 0x7effa80ad3c0 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.909+0000 7eff9dffb640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff8c06b840 0x7eff8c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.909+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7eff9808a970 con 0x7effa80ad3c0 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.909+0000 7effad2ba640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff8c06b840 0x7eff8c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:48.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.909+0000 7effad2ba640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff8c06b840 0x7eff8c06dd00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7effa0034000 tx=0x7effa0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:48.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:48.913+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7eff98055490 con 0x7effa80ad3c0 2024-02-19T15:04:49.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.085+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7effa8157d70 con 0x7eff8c06b840 2024-02-19T15:04:49.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.086+0000 7eff9dffb640 1 -- 172.21.15.78:0/2961625817 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7effa8157d70 con 0x7eff8c06b840 2024-02-19T15:04:49.088 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 7 to deep-scrub 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.087+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff8c06b840 msgr2=0x7eff8c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.087+0000 7effaf51e640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff8c06b840 0x7eff8c06dd00 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7effa0034000 tx=0x7effa0009030 comp rx=0 tx=0).stop 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 msgr2=0x7effa81e1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 0x7effa81e1fc0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7eff980116d0 tx=0x7eff98011b90 comp rx=0 tx=0).stop 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 shutdown_connections 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7eff8c06b840 0x7eff8c06dd00 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7effa80ad3c0 0x7effa81e1fc0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa80c5690 0x7effa81ee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 --2- 172.21.15.78:0/2961625817 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7eff94003b30 0x7effa81eba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 >> 172.21.15.78:0/2961625817 conn(0x7effa80aa370 msgr2=0x7effa80c21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:49.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 shutdown_connections 2024-02-19T15:04:49.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.088+0000 7effaf51e640 1 -- 172.21.15.78:0/2961625817 wait complete. 2024-02-19T15:04:49.101 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:04:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.199+0000 7f1411748640 1 Processor -- start 2024-02-19T15:04:49.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.199+0000 7f1411748640 1 -- start start 2024-02-19T15:04:49.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f1411748640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c0c8e00 0x7f140c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:49.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f1411748640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f140c06d910 0x7f140c06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:49.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f1411748640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140c06cd20 con 0x7f140c152700 2024-02-19T15:04:49.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f1411748640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140c1551f0 con 0x7f140c06d910 2024-02-19T15:04:49.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f1411748640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140c155490 con 0x7f140c0c8e00 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f140c06d910 0x7f140c06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f140c06d910 0x7f140c06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53074/0 (socket says 172.21.15.78:53074) 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 -- 172.21.15.78:0/2623118095 learned_addr learned my addr 172.21.15.78:0/2623118095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f1403fff640 1 --2- 172.21.15.78:0/2623118095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c0c8e00 0x7f140c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 -- 172.21.15.78:0/2623118095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c0c8e00 msgr2=0x7f140c06c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 --2- 172.21.15.78:0/2623118095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c0c8e00 0x7f140c06c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 -- 172.21.15.78:0/2623118095 >> v1:172.21.15.150:6789/0 conn(0x7f140c152700 legacy=0x7f140c0c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:04:49.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.200+0000 7f140b7fe640 1 -- 172.21.15.78:0/2623118095 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c155770 con 0x7f140c06d910 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.201+0000 7f140b7fe640 1 --2- 172.21.15.78:0/2623118095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c06c7e0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f1404009e50 tx=0x7f140402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc10ddbefe645c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.201+0000 7f140a7fc640 1 -- 172.21.15.78:0/2623118095 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140402ca70 con 0x7f140c06d910 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.201+0000 7f140a7fc640 1 -- 172.21.15.78:0/2623118095 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140402cbd0 con 0x7f140c06d910 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.201+0000 7f140a7fc640 1 -- 172.21.15.78:0/2623118095 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14040343f0 con 0x7f140c06d910 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 -- 172.21.15.78:0/2623118095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 msgr2=0x7f140c06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 --2- 172.21.15.78:0/2623118095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c06c7e0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f1404009e50 tx=0x7f140402bed0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 -- 172.21.15.78:0/2623118095 shutdown_connections 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 --2- 172.21.15.78:0/2623118095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c06c7e0 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 --2- 172.21.15.78:0/2623118095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c0c8e00 0x7f140c06c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 -- 172.21.15.78:0/2623118095 >> 172.21.15.78:0/2623118095 conn(0x7f140c0cac90 msgr2=0x7f140c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 -- 172.21.15.78:0/2623118095 shutdown_connections 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.202+0000 7f1411748640 1 -- 172.21.15.78:0/2623118095 wait complete. 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 Processor -- start 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 -- start start 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c1eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:49.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f140c0c8e00 0x7f140c0c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c152700 0x7f140c158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c0aa2b0 con 0x7f140c06d910 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c1eb4b0 con 0x7f140c0c8e00 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.203+0000 7f1411748640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140c0ac980 con 0x7f140c152700 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c1eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c1eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:53084/0 (socket says 172.21.15.78:53084) 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 -- 172.21.15.78:0/2187389148 learned_addr learned my addr 172.21.15.78:0/2187389148 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 -- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c152700 msgr2=0x7f140c158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c152700 0x7f140c158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 -- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f140c0c8e00 msgr2=0x7f140c0c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f1403fff640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f140c0c8e00 0x7f140c0c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f140c0c8e00 0x7f140c0c3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 -- 172.21.15.78:0/2187389148 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1404009b30 con 0x7f140c06d910 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f1403fff640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f140c0c8e00 0x7f140c0c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f140affd640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c1eaf70 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c3f0 tx=0x7f13fc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc0052d0 con 0x7f140c06d910 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc005450 con 0x7f140c06d910 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140c0acc60 con 0x7f140c06d910 2024-02-19T15:04:49.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc007260 con 0x7f140c06d910 2024-02-19T15:04:49.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.204+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c0aa7e0 con 0x7f140c06d910 2024-02-19T15:04:49.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.205+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c074990 con 0x7f140c06d910 2024-02-19T15:04:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.206+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f13fc0073c0 con 0x7f140c06d910 2024-02-19T15:04:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.206+0000 7f14037fe640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f13ec06b810 0x7f13ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:04:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.206+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f13fc089480 con 0x7f140c06d910 2024-02-19T15:04:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.206+0000 7f1403fff640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f13ec06b810 0x7f13ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:04:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.207+0000 7f1403fff640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f13ec06b810 0x7f13ec06dcd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f140c0689a0 tx=0x7f13f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:04:49.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.210+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f13fc053fa0 con 0x7f140c06d910 2024-02-19T15:04:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.380+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f140c06ded0 con 0x7f13ec06b810 2024-02-19T15:04:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.382+0000 7f14037fe640 1 -- 172.21.15.78:0/2187389148 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50782 (secure 0 0 0) 0x7f13f4009e10 con 0x7f13ec06b810 2024-02-19T15:04:49.385 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:04:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f13ec06b810 msgr2=0x7f13ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f13ec06b810 0x7f13ec06dcd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f140c0689a0 tx=0x7f13f4008040 comp rx=0 tx=0).stop 2024-02-19T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 msgr2=0x7f140c1eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c1eaf70 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c3f0 tx=0x7f13fc00c8b0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 shutdown_connections 2024-02-19T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f13ec06b810 0x7f13ec06dcd0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f140c152700 0x7f140c158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f140c0c8e00 0x7f140c0c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 --2- 172.21.15.78:0/2187389148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f140c06d910 0x7f140c1eaf70 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:04:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 >> 172.21.15.78:0/2187389148 conn(0x7f140c0cac90 msgr2=0x7f140c14e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:04:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 shutdown_connections 2024-02-19T15:04:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:04:49.384+0000 7f1411748640 1 -- 172.21.15.78:0/2187389148 wait complete. 2024-02-19T15:04:49.397 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":603,"stamp":"2024-02-19T15:04:48.218935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":272480,"kb_used_data":23016,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754702240,"statfs":{"total":773094113280,"available":772815093760,"internally_reserved":0,"allocated":23568384,"data_stored":12096704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-52,"num_read_kb":-58,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001209"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":545,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-02-19T15:04:42.116479+0000","last_change":"2024-02-19T14:51:49.168155+0000","last_active":"2024-02-19T15:04:42.116479+0000","last_peered":"2024-02-19T15:04:42.116479+0000","last_clean":"2024-02-19T15:04:42.116479+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:04:42.116479+0000","last_undegraded":"2024-02-19T15:04:42.116479+0000","last_fullsized":"2024-02-19T15:04:42.116479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T14:51:48.112297+0000","last_clean_scrub_stamp":"2024-02-19T14:51:48.112297+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:07:35.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34496,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337344,"statfs":{"total":96636764160,"available":96601440256,"internally_reserved":0,"allocated":3391488,"data_stored":1954864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.65100000000000002,"5min":0.747,"15min":0.747},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.67800000000000005,"5min":0.70699999999999996,"15min":0.79500000000000004},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.40999999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.66900000000000004,"5min":0.71999999999999997,"15min":0.83099999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59499999999999997,"5min":0.70899999999999996,"15min":0.76400000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47999999999999998,"15min":0.49099999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.627,"5min":0.76000000000000001,"15min":0.80000000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.498,"15min":0.504},"min":{"1min":0.34799999999999998,"5min":0.30399999999999999,"15min":0.26500000000000001},"max":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.77500000000000002},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66200000000000003,"5min":0.72399999999999998,"15min":0.84199999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.56200000000000006},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.61499999999999999,"5min":0.76900000000000002,"15min":0.81100000000000005},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.36499999999999999,"15min":0.42299999999999999},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.432,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.38200000000000001,"15min":0.45600000000000002},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.44,"5min":0.80400000000000005,"15min":0.86899999999999999},"last":0.33500000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.39500000000000002,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":0.40100000000000002,"5min":0.80600000000000005,"15min":0.92400000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41299999999999998,"15min":0.49399999999999999},"min":{"1min":0.27500000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.45600000000000002,"5min":0.76000000000000001,"15min":0.871},"last":0.41499999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:04:02 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.53700000000000003},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51800000000000002,"5min":0.79400000000000004,"15min":0.90200000000000002},"last":0.443},{"interface":"front","average":{"1min":0.372,"5min":0.44600000000000001,"15min":0.53100000000000003},"min":{"1min":0.27000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.48399999999999999,"5min":0.78000000000000003,"15min":0.90500000000000003},"last":0.47799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.22700000000000001},"max":{"1min":0.50900000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.504},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.502,"15min":0.499},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.78100000000000003,"15min":0.95699999999999996},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.311},"max":{"1min":0.625,"5min":0.80700000000000005,"15min":0.93600000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.441,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78500000000000003,"15min":0.90500000000000003},"last":0.58799999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.59499999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.318,"15min":0.318},"max":{"1min":0.61399999999999999,"5min":0.70799999999999996,"15min":0.77700000000000002},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.374,"15min":0.46400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.40600000000000003,"5min":0.874,"15min":0.874},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.44500000000000001,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47799999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.627}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.371,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.39200000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41699999999999998,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.47299999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.503},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.45900000000000002,"5min":0.78900000000000003,"15min":0.89600000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.42499999999999999,"15min":0.50900000000000001},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":0.42599999999999999,"5min":0.874,"15min":0.874},"last":0.52000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.58199999999999996,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.878,"15min":0.878},"last":0.498}]},{"osd":1,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.35699999999999998,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.81000000000000005,"15min":0.872},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49299999999999999,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64600000000000002,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.81299999999999994,"15min":0.81699999999999995},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.56499999999999995,"5min":0.79400000000000004,"15min":0.85899999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66200000000000003,"5min":0.82799999999999996,"15min":0.83799999999999997},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.503,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.75600000000000001,"15min":0.83899999999999997},"last":0.432}]},{"osd":4,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35399999999999998,"15min":0.45300000000000001},"min":{"1min":0.252,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":0.41099999999999998,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.249},"max":{"1min":0.51600000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.72699999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.40000000000000002,"15min":0.49199999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.44,"5min":0.85999999999999999,"15min":0.877},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42699999999999999,"15min":0.52500000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.57999999999999996,"5min":0.876,"15min":0.90000000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45600000000000002,"15min":0.55600000000000005},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52900000000000003,"5min":0.96299999999999997,"15min":1.03},"last":0.46600000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.505,"15min":0.53300000000000003},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70799999999999996,"5min":0.79200000000000004,"15min":4.6879999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":1.341},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48399999999999999,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.53700000000000003,"5min":0.83399999999999996,"15min":2.6549999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48999999999999999,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":0.64900000000000002,"5min":0.82899999999999996,"15min":4.7290000000000001},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":2.5800000000000001},"last":0.437},{"interface":"front","average":{"1min":0.434,"5min":0.48999999999999999,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.56899999999999995,"5min":0.79100000000000004,"15min":10.629},"last":0.39000000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":1.2050000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84099999999999997,"15min":2.7010000000000001},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36499999999999999,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.40600000000000003,"5min":0.76400000000000001,"15min":6.6989999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38600000000000001,"15min":0.47099999999999997},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40000000000000002,"5min":0.77000000000000002,"15min":1.4470000000000001},"last":0.35599999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38600000000000001,"15min":0.495},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.36399999999999999,"5min":0.85199999999999998,"15min":2.7349999999999999},"last":0.378},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41499999999999998,"15min":0.52900000000000003},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.41499999999999998,"5min":0.83599999999999997,"15min":4.6239999999999997},"last":0.34399999999999997}]},{"osd":7,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.41299999999999998,"15min":0.51000000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.41799999999999998,"5min":0.80300000000000005,"15min":1.246},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.441,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46999999999999997,"5min":0.81999999999999995,"15min":2.621},"last":0.32300000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.32700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.159},"max":{"1min":0.441,"5min":0.67900000000000005,"15min":3.694},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":4.835},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.18099999999999999},"max":{"1min":0.58799999999999997,"5min":0.64200000000000002,"15min":0.76100000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.20000000000000001},"max":{"1min":0.501,"5min":0.66400000000000003,"15min":6.3810000000000002},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.40200000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":3.6320000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.372},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57799999999999996,"5min":0.68300000000000005,"15min":4.9889999999999999},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48499999999999999,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.27900000000000003,"15min":0.26600000000000001},"max":{"1min":0.58199999999999996,"5min":0.874,"15min":0.874},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60499999999999998,"5min":0.86199999999999999,"15min":4.9539999999999997},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53100000000000003,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63800000000000001,"5min":0.85699999999999998,"15min":3.6669999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.65900000000000003,"5min":0.88600000000000001,"15min":4.9130000000000003},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.80700000000000005,"15min":0.86799999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67800000000000005,"5min":0.82999999999999996,"15min":3.7189999999999999},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30499999999999999},"max":{"1min":0.65000000000000002,"5min":0.82199999999999995,"15min":4.8799999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.69299999999999995,"5min":0.871,"15min":3.0760000000000001},"last":0.56599999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34000000000000002,"15min":0.32700000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":0.52000000000000002,"5min":0.621,"15min":0.68899999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.32800000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.57699999999999996},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.309},{"interface":"front","average":{"1min":0.37,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.63,"5min":0.67200000000000004,"15min":1.512},"last":0.38900000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.35099999999999998},"min":{"1min":0.25600000000000001,"5min":0.22900000000000001,"15min":0.19700000000000001},"max":{"1min":0.52900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.38900000000000001},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":0.59099999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61299999999999999,"5min":0.84599999999999997,"15min":1.133},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51100000000000001,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61199999999999999,"5min":0.748,"15min":1.1539999999999999},"last":0.45200000000000001}]},{"osd":5,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50700000000000001,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59699999999999998,"5min":0.78700000000000003,"15min":0.94999999999999996},"last":0.434},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52700000000000002,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.60299999999999998,"5min":0.82999999999999996,"15min":1.115},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":1.236},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.60299999999999998,"5min":0.81299999999999994,"15min":1.1830000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54300000000000004,"15min":0.56499999999999995},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.623,"5min":0.85799999999999998,"15min":0.995},"last":0.36899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.80600000000000005},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.223,"5min":0.218,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.90400000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.56100000000000005,"5min":0.624,"15min":0.77400000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.622,"5min":0.70199999999999996,"15min":0.76000000000000001},"last":0.32000000000000001}]},{"osd":3,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.41399999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.72199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42599999999999999,"15min":0.39500000000000002},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.93300000000000005},"last":0.375}]},{"osd":4,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59099999999999997,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.88200000000000001,"15min":1.0009999999999999},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.77300000000000002,"15min":1.0189999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.53800000000000003},"min":{"1min":0.36299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.625,"5min":0.85599999999999998,"15min":1.0229999999999999},"last":0.40999999999999998}]},{"osd":6,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.53700000000000003,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.31},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":1.026},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.307},"max":{"1min":0.65400000000000003,"5min":0.81399999999999995,"15min":0.996},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79200000000000004,"5min":0.90800000000000003,"15min":1.036},"last":0.436},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.83399999999999996,"15min":1.0740000000000001},"last":0.51400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34496,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337344,"statfs":{"total":96636764160,"available":96601440256,"internally_reserved":0,"allocated":3391488,"data_stored":1954864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.33700000000000002},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.17399999999999999},"max":{"1min":0.50900000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.372},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38300000000000001,"15min":0.35299999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.48399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52300000000000002,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.56699999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52900000000000003,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.32400000000000001,"15min":0.28399999999999997},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.53300000000000003,"15min":0.55500000000000005},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.70399999999999996}]},{"osd":6,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.78000000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.58299999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.747},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.65000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:04:49.400 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-02-19T14:51:48.112297+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=19, tm_hour=14, tm_min=51, tm_sec=48, tm_wday=0, tm_yday=50, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=19, tm_hour=15, tm_min=4, tm_sec=44, tm_wday=0, tm_yday=50, tm_isdst=0) 2024-02-19T15:04:49.401 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-02-19T15:04:58.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:04:58.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:05:09.401 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-19T15:05:09.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.505+0000 7fa9dc5e2640 1 Processor -- start 2024-02-19T15:05:09.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.505+0000 7fa9dc5e2640 1 -- start start 2024-02-19T15:05:09.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.506+0000 7fa9dc5e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9d406f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.506+0000 7fa9dc5e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d406b620 con 0x7fa9d4073220 2024-02-19T15:05:09.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.506+0000 7fa9dc5e2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d4157230 con 0x7fa9d414d110 2024-02-19T15:05:09.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.506+0000 7fa9dc5e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d406ea00 con 0x7fa9d406bae0 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.506+0000 7fa9d9b7d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa9d4073220 0x7fa9d406ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51270/0 (socket says 172.21.15.78:51270) 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.506+0000 7fa9d9b7d640 1 -- 172.21.15.78:0/2082401399 learned_addr learned my addr 172.21.15.78:0/2082401399 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325456605 0 0) 0x7fa9d406b620 con 0x7fa9d4073220 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0003520 con 0x7fa9d4073220 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9d406f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959447310 0 0) 0x7fa9c0003520 con 0x7fa9d4073220 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 msgr2=0x7fa9d406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:09.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9d406f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 >> v1:172.21.15.150:6789/0 conn(0x7fa9d414d110 legacy=0x7fa9d40ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4072180 con 0x7fa9d4073220 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7fa9c4001670 con 0x7fa9d4073220 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 >> v1:172.21.15.78:6789/0 conn(0x7fa9d4073220 legacy=0x7fa9d406ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c00039a0 0x7fa9c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d406ea00 con 0x7fa9c0004d00 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c000bb60 con 0x7fa9c00039a0 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c000bd20 con 0x7fa9d406bae0 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9da37e640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c00039a0 0x7fa9c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9d9b7d640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 -- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 msgr2=0x7fa9c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 -- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c00039a0 msgr2=0x7fa9c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:09.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c00039a0 0x7fa9c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.507+0000 7fa9dab7f640 1 -- 172.21.15.78:0/2082401399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0003520 con 0x7fa9c0004d00 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.508+0000 7fa9d9b7d640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d406bae0 0x7fa9c00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.508+0000 7fa9dab7f640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9c0005180 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0007360 tx=0x7fa9c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7176e8f82597fc51 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.508+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d002ccd0 con 0x7fa9c0004d00 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.508+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d002ce30 con 0x7fa9c0004d00 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.508+0000 7fa9d937c640 1 -- 172.21.15.78:0/2082401399 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0034660 con 0x7fa9c0004d00 2024-02-19T15:05:09.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/2082401399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 msgr2=0x7fa9c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/2082401399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9c0005180 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0007360 tx=0x7fa9c00057f0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dab7f640 1 -- 172.21.15.78:0/2082401399 reap_dead start 2024-02-19T15:05:09.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/2082401399 shutdown_connections 2024-02-19T15:05:09.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/2082401399 >> 172.21.15.78:0/2082401399 conn(0x7fa9d40c7590 msgr2=0x7fa9d40c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:05:09.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/2082401399 shutdown_connections 2024-02-19T15:05:09.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.509+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/2082401399 wait complete. 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.510+0000 7fa9dc5e2640 1 Processor -- start 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.510+0000 7fa9dc5e2640 1 -- start start 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dc5e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9d414edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dc5e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d41ea020 0x7fa9d4146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dc5e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d41ea4a0 0x7fa9d4146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dc5e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d4147500 con 0x7fa9c0004d00 2024-02-19T15:05:09.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dc5e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d414e940 con 0x7fa9d41ea4a0 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dc5e2640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d414e350 con 0x7fa9d41ea020 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9d414edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9d414edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39384/0 (socket says 172.21.15.78:39384) 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 -- 172.21.15.78:0/1544307898 learned_addr learned my addr 172.21.15.78:0/1544307898 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9dab7f640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d41ea4a0 0x7fa9d4146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9d9b7d640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d41ea020 0x7fa9d4146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 -- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d41ea020 msgr2=0x7fa9d4146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d41ea020 0x7fa9d4146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 -- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d41ea4a0 msgr2=0x7fa9d4146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d41ea4a0 0x7fa9d4146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 -- 172.21.15.78:0/1544307898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4151120 con 0x7fa9c0004d00 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9da37e640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9d414edd0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00f850 tx=0x7fa9cc00fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.511+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00fe40 con 0x7fa9c0004d00 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.512+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc0052e0 con 0x7fa9c0004d00 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.512+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d4151400 con 0x7fa9c0004d00 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.512+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00d970 con 0x7fa9c0004d00 2024-02-19T15:05:09.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.512+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d414f9f0 con 0x7fa9c0004d00 2024-02-19T15:05:09.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.513+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d4072de0 con 0x7fa9c0004d00 2024-02-19T15:05:09.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.513+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7fa9cc00dad0 con 0x7fa9c0004d00 2024-02-19T15:05:09.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.513+0000 7fa9caffd640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9b806b810 0x7fa9b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.513+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa9cc08a020 con 0x7fa9c0004d00 2024-02-19T15:05:09.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.513+0000 7fa9d9b7d640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9b806b810 0x7fa9b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.514+0000 7fa9d9b7d640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9b806b810 0x7fa9b806dcd0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4000ff0 tx=0x7fa9c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:09.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.518+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9cc054b40 con 0x7fa9c0004d00 2024-02-19T15:05:09.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.685+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 --> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9d4156e60 con 0x7fa9b806b810 2024-02-19T15:05:09.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.687+0000 7fa9caffd640 1 -- 172.21.15.78:0/1544307898 <== mgr.4110 v2:172.21.15.78:6832/359840476 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50844 (secure 0 0 0) 0x7fa9d4156e60 con 0x7fa9b806b810 2024-02-19T15:05:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:05:09.691 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-02-19T15:05:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9b806b810 msgr2=0x7fa9b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9b806b810 0x7fa9b806dcd0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4000ff0 tx=0x7fa9c4009040 comp rx=0 tx=0).stop 2024-02-19T15:05:09.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 msgr2=0x7fa9d414edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9d414edd0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00f850 tx=0x7fa9cc00fd10 comp rx=0 tx=0).stop 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 shutdown_connections 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7fa9b806b810 0x7fa9b806dcd0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9d41ea4a0 0x7fa9d4146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9d41ea020 0x7fa9d4146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 --2- 172.21.15.78:0/1544307898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa9c0004d00 0x7fa9d414edd0 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 >> 172.21.15.78:0/1544307898 conn(0x7fa9d40c7590 msgr2=0x7fa9d40c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:05:09.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 shutdown_connections 2024-02-19T15:05:09.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.689+0000 7fa9dc5e2640 1 -- 172.21.15.78:0/1544307898 wait complete. 2024-02-19T15:05:09.702 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2024-02-19T15:05:08.223088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":272464,"kb_used_data":23016,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754702256,"statfs":{"total":773094113280,"available":772815110144,"internally_reserved":0,"allocated":23568384,"data_stored":12096704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002701"},"pg_stats":[{"pgid":"1.0","version":"220'119","reported_seq":556,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-02-19T15:04:49.562381+0000","last_change":"2024-02-19T15:04:49.562416+0000","last_active":"2024-02-19T15:04:49.562381+0000","last_peered":"2024-02-19T15:04:49.562381+0000","last_clean":"2024-02-19T15:04:49.562381+0000","last_became_active":"2024-02-19T14:51:49.167847+0000","last_became_peered":"2024-02-19T14:51:49.167847+0000","last_unstale":"2024-02-19T15:04:49.562381+0000","last_undegraded":"2024-02-19T15:04:49.562381+0000","last_fullsized":"2024-02-19T15:04:49.562381+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'119","last_scrub_stamp":"2024-02-19T15:04:49.562282+0000","last_deep_scrub":"220'119","last_deep_scrub_stamp":"2024-02-19T15:04:49.562282+0000","last_clean_scrub_stamp":"2024-02-19T15:04:49.562282+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T01:23:16.443890+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575012,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34488,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337352,"statfs":{"total":96636764160,"available":96601448448,"internally_reserved":0,"allocated":3391488,"data_stored":1954864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.33200000000000002,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":0.65200000000000002,"5min":0.747,"15min":0.747},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.32000000000000001,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.79500000000000004},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.499,"15min":0.497},"min":{"1min":0.27900000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.83099999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.33400000000000002},"max":{"1min":0.70499999999999996,"5min":0.70899999999999996,"15min":0.76400000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47299999999999998,"15min":0.48799999999999999},"min":{"1min":0.34899999999999998,"5min":0.28299999999999997,"15min":0.26700000000000002},"max":{"1min":0.68500000000000005,"5min":0.76000000000000001,"15min":0.80000000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.48799999999999999,"15min":0.503},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.26500000000000001},"max":{"1min":0.64800000000000002,"5min":0.66900000000000004,"15min":0.77500000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.53200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63700000000000001,"5min":0.72399999999999998,"15min":0.84199999999999997},"last":0.626},{"interface":"front","average":{"1min":0.505,"5min":0.52700000000000002,"15min":0.55700000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.746,"5min":0.76900000000000002,"15min":0.81100000000000005},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.34200000000000003,"15min":0.41699999999999998},"min":{"1min":0.23999999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.63600000000000001,"5min":0.70799999999999996,"15min":0.82899999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.44900000000000001},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60499999999999998,"5min":0.72499999999999998,"15min":0.86899999999999999},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.36699999999999999,"15min":0.46800000000000003},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":0.66400000000000003,"5min":0.76200000000000001,"15min":0.92400000000000004},"last":0.372},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.39500000000000002,"15min":0.48499999999999999},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63700000000000001,"5min":0.748,"15min":0.871},"last":0.39900000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.434,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76000000000000001,"5min":0.79400000000000004,"15min":0.90200000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.42599999999999999,"15min":0.52200000000000002},"min":{"1min":0.317,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65200000000000002,"5min":0.78000000000000003,"15min":0.90500000000000003},"last":0.46999999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.48699999999999999,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.28699999999999998,"15min":0.22700000000000001},"max":{"1min":0.50900000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.502,"15min":0.499},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.78100000000000003,"15min":0.95699999999999996},"last":0.70299999999999996}]},{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29999999999999999},"max":{"1min":0.66200000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73999999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.311},"max":{"1min":0.625,"5min":0.80700000000000005,"15min":0.93600000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.32400000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.78500000000000003,"15min":0.90500000000000003},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.59499999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.49199999999999999,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.318,"15min":0.318},"max":{"1min":0.61399999999999999,"5min":0.70799999999999996,"15min":0.77700000000000002},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.374,"15min":0.46400000000000002},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.40600000000000003,"5min":0.874,"15min":0.874},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.44500000000000001,"15min":0.54000000000000004},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47799999999999998,"5min":0.85299999999999998,"15min":0.85999999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.371,"15min":0.46300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.39200000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.41699999999999998,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22600000000000001},"max":{"1min":0.47299999999999998,"5min":0.97699999999999998,"15min":1.4279999999999999},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40100000000000002,"15min":0.503},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.45900000000000002,"5min":0.78900000000000003,"15min":0.89600000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.42499999999999999,"15min":0.50900000000000001},"min":{"1min":0.29099999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":0.42599999999999999,"5min":0.874,"15min":0.874},"last":0.435}]}]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.58199999999999996,"5min":0.79800000000000004,"15min":0.89100000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.41199999999999998,"5min":0.30399999999999999,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.878,"15min":0.878},"last":0.505}]},{"osd":1,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.35699999999999998,"5min":0.32500000000000001,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.81000000000000005,"15min":0.872},"last":0.496},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.49299999999999999,"15min":0.502},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64600000000000002,"5min":0.89000000000000001,"15min":0.92600000000000005},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.81299999999999994,"15min":0.81699999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50700000000000001,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.56499999999999995,"5min":0.79400000000000004,"15min":0.85899999999999999},"last":0.52600000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66200000000000003,"5min":0.82799999999999996,"15min":0.83799999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.503,"15min":0.53300000000000003},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.75600000000000001,"15min":0.83899999999999997},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35399999999999998,"15min":0.45300000000000001},"min":{"1min":0.252,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":0.41099999999999998,"5min":0.90700000000000003,"15min":0.92400000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.39700000000000002,"15min":0.49099999999999999},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.249},"max":{"1min":0.51600000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.43099999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.40000000000000002,"15min":0.49199999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.44,"5min":0.85999999999999999,"15min":0.877},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.39800000000000002,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Feb 19 15:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42699999999999999,"15min":0.52500000000000002},"min":{"1min":0.28299999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.57999999999999996,"5min":0.876,"15min":0.90000000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.45600000000000002,"15min":0.55600000000000005},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.52900000000000003,"5min":0.96299999999999997,"15min":1.03},"last":0.55400000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.505,"15min":0.53300000000000003},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70799999999999996,"5min":0.79200000000000004,"15min":4.6879999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51400000000000001,"15min":0.52800000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28599999999999998},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":1.341},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.48399999999999999,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.53700000000000003,"5min":0.83399999999999996,"15min":2.6549999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48999999999999999,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.24299999999999999},"max":{"1min":0.64900000000000002,"5min":0.82899999999999996,"15min":4.7290000000000001},"last":0.72099999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.625,"5min":0.81000000000000005,"15min":2.5800000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.48999999999999999,"15min":0.55800000000000005},"min":{"1min":0.33500000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.56899999999999995,"5min":0.79100000000000004,"15min":10.629},"last":0.68400000000000005}]},{"osd":3,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":1.2050000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.58899999999999997},"min":{"1min":0.40400000000000003,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.84099999999999997,"15min":2.7010000000000001},"last":0.755}]},{"osd":5,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.36499999999999999,"15min":0.46999999999999997},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.40600000000000003,"5min":0.76400000000000001,"15min":6.6989999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.38600000000000001,"15min":0.47099999999999997},"min":{"1min":0.26700000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40000000000000002,"5min":0.77000000000000002,"15min":1.4470000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38600000000000001,"15min":0.495},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.36399999999999999,"5min":0.85199999999999998,"15min":2.7349999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.41499999999999998,"15min":0.52900000000000003},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.41499999999999998,"5min":0.83599999999999997,"15min":4.6239999999999997},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 15:04:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.41299999999999998,"15min":0.51000000000000001},"min":{"1min":0.249,"5min":0.247,"15min":0.247},"max":{"1min":0.41799999999999998,"5min":0.80300000000000005,"15min":1.246},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.441,"15min":0.56200000000000006},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46999999999999997,"5min":0.81999999999999995,"15min":2.621},"last":0.70699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.32700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.159},"max":{"1min":0.441,"5min":0.67900000000000005,"15min":3.694},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.34799999999999998},"min":{"1min":0.254,"5min":0.23000000000000001,"15min":0.189},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":4.835},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.23499999999999999,"5min":0.21099999999999999,"15min":0.18099999999999999},"max":{"1min":0.58799999999999997,"5min":0.64200000000000002,"15min":0.76100000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.379,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.20000000000000001},"max":{"1min":0.501,"5min":0.66400000000000003,"15min":6.3810000000000002},"last":0.59499999999999997}]},{"osd":2,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.40200000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":3.6320000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.372},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57799999999999996,"5min":0.68300000000000005,"15min":4.9889999999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.48499999999999999,"15min":0.53100000000000003},"min":{"1min":0.34300000000000003,"5min":0.27900000000000003,"15min":0.26600000000000001},"max":{"1min":0.58199999999999996,"5min":0.874,"15min":0.874},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.504,"15min":0.55900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60499999999999998,"5min":0.86199999999999999,"15min":4.9539999999999997},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.53100000000000003,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63800000000000001,"5min":0.85699999999999998,"15min":3.6669999999999998},"last":0.629},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.33900000000000002},"max":{"1min":0.65900000000000003,"5min":0.88600000000000001,"15min":4.9130000000000003},"last":0.68200000000000005}]},{"osd":6,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51100000000000001,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.80700000000000005,"15min":0.86799999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53200000000000003,"15min":0.59899999999999998},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67800000000000005,"5min":0.82999999999999996,"15min":3.7189999999999999},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Feb 19 15:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.30499999999999999},"max":{"1min":0.65000000000000002,"5min":0.82199999999999995,"15min":4.8799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53800000000000003,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.38400000000000001},"max":{"1min":0.69299999999999995,"5min":0.871,"15min":3.0760000000000001},"last":0.70299999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33916,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34000000000000002,"15min":0.32700000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":0.52000000000000002,"5min":0.621,"15min":0.68899999999999995},"last":0.44},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35999999999999999,"15min":0.32800000000000001},"min":{"1min":0.23499999999999999,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.51000000000000001,"5min":0.57399999999999995,"15min":0.57699999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20599999999999999},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.22800000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.63,"5min":0.67200000000000004,"15min":1.512},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.371,"15min":0.35099999999999998},"min":{"1min":0.25600000000000001,"5min":0.22900000000000001,"15min":0.19700000000000001},"max":{"1min":0.52900000000000003,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.38900000000000001},"min":{"1min":0.28299999999999997,"5min":0.27200000000000002,"15min":0.254},"max":{"1min":0.59099999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"last":0.497}]},{"osd":4,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61299999999999999,"5min":0.84599999999999997,"15min":1.133},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51100000000000001,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61199999999999999,"5min":0.748,"15min":1.1539999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50700000000000001,"15min":0.53800000000000003},"min":{"1min":0.34799999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59699999999999998,"5min":0.78700000000000003,"15min":0.94999999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.52700000000000002,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.86699999999999999},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.60299999999999998,"5min":0.82999999999999996,"15min":1.115},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54200000000000004,"15min":0.56399999999999995},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61799999999999999,"5min":0.83099999999999996,"15min":1.236},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Feb 19 15:04:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.57999999999999996},"min":{"1min":0.435,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.60299999999999998,"5min":0.81299999999999994,"15min":1.1830000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.54300000000000004,"15min":0.56499999999999995},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.623,"5min":0.85799999999999998,"15min":0.995},"last":0.68100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33912,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2797568,"data_stored":1364496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34399999999999997,"15min":0.32700000000000001},"min":{"1min":0.184,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.80600000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.223,"5min":0.218,"15min":0.17999999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.90400000000000003},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35599999999999998,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.187,"15min":0.17699999999999999},"max":{"1min":0.56100000000000005,"5min":0.624,"15min":0.77400000000000002},"last":0.31},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23000000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.622,"5min":0.70199999999999996,"15min":0.76000000000000001},"last":0.33100000000000002}]},{"osd":3,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.41399999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.72199999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42599999999999999,"15min":0.39500000000000002},"min":{"1min":0.30399999999999999,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":0.61199999999999999,"5min":0.61799999999999999,"15min":0.93300000000000005},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.53000000000000003},"min":{"1min":0.32100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.59099999999999997,"5min":0.80800000000000005,"15min":0.96799999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.28399999999999997},"max":{"1min":0.67200000000000004,"5min":0.88200000000000001,"15min":1.0009999999999999},"last":0.60799999999999998}]},{"osd":5,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.30199999999999999},"max":{"1min":0.68100000000000005,"5min":0.77300000000000002,"15min":1.0189999999999999},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.53800000000000003},"min":{"1min":0.36299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.625,"5min":0.85599999999999998,"15min":1.0229999999999999},"last":0.625}]},{"osd":6,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.53700000000000003,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.34599999999999997,"15min":0.31},"max":{"1min":0.70599999999999996,"5min":0.80300000000000005,"15min":1.026},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.53800000000000003,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.307},"max":{"1min":0.65400000000000003,"5min":0.81399999999999995,"15min":0.996},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 15:04:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79200000000000004,"5min":0.90800000000000003,"15min":1.036},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.83399999999999996,"15min":1.0740000000000001},"last":0.55700000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575012,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34488,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337352,"statfs":{"total":96636764160,"available":96601448448,"internally_reserved":0,"allocated":3391488,"data_stored":1954864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36199999999999999,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.20300000000000001,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.44},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38300000000000001,"15min":0.33700000000000002},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.17399999999999999},"max":{"1min":0.50900000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.217},"max":{"1min":0.60299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40400000000000003,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.218,"15min":0.218},"max":{"1min":0.59199999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.41499999999999998}]},{"osd":3,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38300000000000001,"15min":0.35299999999999998},"min":{"1min":0.253,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.48399999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.52300000000000002,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33300000000000002},"max":{"1min":0.56699999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.52900000000000003,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.32400000000000001,"15min":0.28399999999999997},"max":{"1min":0.68799999999999994,"5min":0.86699999999999999,"15min":1.1339999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53300000000000003,"15min":0.55500000000000005},"min":{"1min":0.41199999999999998,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.76700000000000002}]},{"osd":6,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.627},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36899999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"max":{"1min":0.74099999999999999,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.78400000000000003}]},{"osd":7,"last update":"Mon Feb 19 15:04:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.58299999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71799999999999997,"5min":0.89300000000000002,"15min":1.0149999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.92300000000000004,"15min":0.96099999999999997},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T15:05:09.704 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-02-19T15:05:09.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 Processor -- start 2024-02-19T15:05:09.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 -- start start 2024-02-19T15:05:09.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0ebc0c8500 0x7f0ebc0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0ebc0c42e0 0x7f0ebc158650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0c34a0 con 0x7f0ebc0c2f50 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc06c090 con 0x7f0ebc0c42e0 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ec2295640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc06d6a0 con 0x7f0ebc0c8500 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.802+0000 7f0ebb7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0ebc0c42e0 0x7f0ebc158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebb7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0ebc0c42e0 0x7f0ebc158650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39390/0 (socket says 172.21.15.78:39390) 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebb7fe640 1 -- 172.21.15.78:0/1306302664 learned_addr learned my addr 172.21.15.78:0/1306302664 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebb7fe640 1 -- 172.21.15.78:0/1306302664 >> v2:172.21.15.150:3300/0 conn(0x7f0ebc0c8500 msgr2=0x7f0ebc0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:09.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebbfff640 1 --2- 172.21.15.78:0/1306302664 >> v2:172.21.15.150:3300/0 conn(0x7f0ebc0c8500 0x7f0ebc0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebb7fe640 1 --2- 172.21.15.78:0/1306302664 >> v2:172.21.15.150:3300/0 conn(0x7f0ebc0c8500 0x7f0ebc0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebb7fe640 1 -- 172.21.15.78:0/1306302664 >> v1:172.21.15.150:6789/0 conn(0x7f0ebc0c2f50 legacy=0x7f0ebc158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebb7fe640 1 -- 172.21.15.78:0/1306302664 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc06c8e0 con 0x7f0ebc0c42e0 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.803+0000 7f0ebbfff640 1 --2- 172.21.15.78:0/1306302664 >> v2:172.21.15.150:3300/0 conn(0x7f0ebc0c8500 0x7f0ebc0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ebb7fe640 1 --2- 172.21.15.78:0/1306302664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc0c42e0 0x7f0ebc158650 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0c9350 tx=0x7f0eb002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80216896222f4c55 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ebaffd640 1 -- 172.21.15.78:0/1306302664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00317e0 con 0x7f0ebc0c42e0 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ebaffd640 1 -- 172.21.15.78:0/1306302664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb0031940 con 0x7f0ebc0c42e0 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ebaffd640 1 -- 172.21.15.78:0/1306302664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0031cd0 con 0x7f0ebc0c42e0 2024-02-19T15:05:09.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 -- 172.21.15.78:0/1306302664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc0c42e0 msgr2=0x7f0ebc158650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 --2- 172.21.15.78:0/1306302664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc0c42e0 0x7f0ebc158650 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0c9350 tx=0x7f0eb002b880 comp rx=0 tx=0).stop 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 -- 172.21.15.78:0/1306302664 shutdown_connections 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 --2- 172.21.15.78:0/1306302664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc0c42e0 0x7f0ebc158650 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 --2- 172.21.15.78:0/1306302664 >> v2:172.21.15.150:3300/0 conn(0x7f0ebc0c8500 0x7f0ebc0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 -- 172.21.15.78:0/1306302664 >> 172.21.15.78:0/1306302664 conn(0x7f0ebc0aa300 msgr2=0x7f0ebc0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 -- 172.21.15.78:0/1306302664 shutdown_connections 2024-02-19T15:05:09.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.804+0000 7f0ec2295640 1 -- 172.21.15.78:0/1306302664 wait complete. 2024-02-19T15:05:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.805+0000 7f0ec2295640 1 Processor -- start 2024-02-19T15:05:09.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.805+0000 7f0ec2295640 1 -- start start 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec2295640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ebc0c2f50 0x7f0ebc0ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec2295640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ebc0c8500 0x7f0ebc0acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec2295640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 0x7f0ebc0ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec2295640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc150e90 con 0x7f0ebc1591e0 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec2295640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc152ff0 con 0x7f0ebc0c2f50 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec2295640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc0ad760 con 0x7f0ebc0c8500 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec0832640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 0x7f0ebc0ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec0832640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 0x7f0ebc0ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39404/0 (socket says 172.21.15.78:39404) 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ec0832640 1 -- 172.21.15.78:0/9216335 learned_addr learned my addr 172.21.15.78:0/9216335 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ebbfff640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ebc0c2f50 0x7f0ebc0ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.806+0000 7f0ebb7fe640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ebc0c8500 0x7f0ebc0acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec0832640 1 -- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ebc0c8500 msgr2=0x7f0ebc0acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec0832640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ebc0c8500 0x7f0ebc0acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec0832640 1 -- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ebc0c2f50 msgr2=0x7f0ebc0ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec0832640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ebc0c2f50 0x7f0ebc0ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec0832640 1 -- 172.21.15.78:0/9216335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00094d0 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ebbfff640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ebc0c2f50 0x7f0ebc0ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec0832640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 0x7f0ebc0ad220 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400c780 tx=0x7f0eb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4004b60 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb400ce90 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc070e10 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb400a850 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.807+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc1510a0 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.808+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc0c6690 con 0x7f0ebc1591e0 2024-02-19T15:05:09.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.808+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f0eb400a9b0 con 0x7f0ebc1591e0 2024-02-19T15:05:09.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.809+0000 7f0eb8ff9640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ea806b810 0x7f0ea806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:09.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.809+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0eb408a2e0 con 0x7f0ebc1591e0 2024-02-19T15:05:09.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.809+0000 7f0ebbfff640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ea806b810 0x7f0ea806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:09.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.809+0000 7f0ebbfff640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ea806b810 0x7f0ea806dcd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc158a50 tx=0x7f0eac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:09.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.813+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0eb4054e00 con 0x7f0ebc1591e0 2024-02-19T15:05:09.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.985+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f0ebc071c20 con 0x7f0ebc1591e0 2024-02-19T15:05:09.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.988+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v65)=0 v65) v1 ==== 123+0+0 (secure 0 0 0) 0x7f0eb40584c0 con 0x7f0ebc1591e0 2024-02-19T15:05:09.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.988+0000 7f0eb8ff9640 1 -- 172.21.15.78:0/9216335 <== mon.0 v2:172.21.15.78:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0eb4054e00 con 0x7f0ebc1591e0 2024-02-19T15:05:09.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ea806b810 msgr2=0x7f0ea806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ea806b810 0x7f0ea806dcd0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc158a50 tx=0x7f0eac008040 comp rx=0 tx=0).stop 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 msgr2=0x7f0ebc0ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 0x7f0ebc0ad220 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400c780 tx=0x7f0eb400cc40 comp rx=0 tx=0).stop 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 shutdown_connections 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f0ea806b810 0x7f0ea806dcd0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ebc1591e0 0x7f0ebc0ad220 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0ebc0c8500 0x7f0ebc0acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 --2- 172.21.15.78:0/9216335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ebc0c2f50 0x7f0ebc0ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 >> 172.21.15.78:0/9216335 conn(0x7f0ebc0aa300 msgr2=0x7f0ebc0aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:05:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 shutdown_connections 2024-02-19T15:05:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:09.990+0000 7f0ec2295640 1 -- 172.21.15.78:0/9216335 wait complete. 2024-02-19T15:05:10.004 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-19T15:05:10.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T15:05:10.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.101+0000 7f8334ac9640 1 Processor -- start 2024-02-19T15:05:10.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.101+0000 7f8334ac9640 1 -- start start 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8334ac9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8330155f90 0x7f833006d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8334ac9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 0x7f8330158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8334ac9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83300c6af0 con 0x7f83300c6d30 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8334ac9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83300c7940 con 0x7f8330155f90 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8334ac9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83300c7c20 con 0x7f83301588a0 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8327fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f83300c6d30 0x7f83300c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51284/0 (socket says 172.21.15.78:51284) 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f8327fff640 1 -- 172.21.15.78:0/1790330442 learned_addr learned my addr 172.21.15.78:0/1790330442 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-02-19T15:05:10.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832ed9d640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 0x7f8330158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832e59c640 1 --2- 172.21.15.78:0/1790330442 >> v2:172.21.15.150:3300/0 conn(0x7f8330155f90 0x7f833006d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847615125 0 0) 0x7f83300c6af0 con 0x7f83300c6d30 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831c003520 con 0x7f83300c6d30 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 242381958 0 0) 0x7f831c003520 con 0x7f83300c6d30 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 msgr2=0x7f8330158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 0x7f8330158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 >> v2:172.21.15.150:3300/0 conn(0x7f8330155f90 msgr2=0x7f833006d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 --2- 172.21.15.78:0/1790330442 >> v2:172.21.15.150:3300/0 conn(0x7f8330155f90 0x7f833006d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83300c7f00 con 0x7f83300c6d30 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1090223067 0 0) 0x7f8318001690 con 0x7f83300c6d30 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 >> v1:172.21.15.78:6789/0 conn(0x7f83300c6d30 legacy=0x7f83300c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c003970 0x7f831c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 0x7f831c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f831c004d00 0x7f831c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300c7940 con 0x7f831c004d00 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300c7c20 con 0x7f831c003970 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.102+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c00b980 con 0x7f83301588a0 2024-02-19T15:05:10.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832ed9d640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f831c004d00 0x7f831c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832e59c640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c003970 0x7f831c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f8327fff640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 0x7f831c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832ed9d640 1 -- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 msgr2=0x7f831c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832ed9d640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83301588a0 0x7f831c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832ed9d640 1 -- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c003970 msgr2=0x7f831c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832ed9d640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c003970 0x7f831c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.103+0000 7f832ed9d640 1 -- 172.21.15.78:0/1790330442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c003520 con 0x7f831c004d00 2024-02-19T15:05:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f832ed9d640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f831c004d00 0x7f831c005180 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f8328009fc0 tx=0x7f831c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cfd6443ebc1606 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832802cc20 con 0x7f831c004d00 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f832800f040 con 0x7f831c004d00 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f832dd9b640 1 -- 172.21.15.78:0/1790330442 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328034440 con 0x7f831c004d00 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f8334ac9640 1 -- 172.21.15.78:0/1790330442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f831c004d00 msgr2=0x7f831c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f8334ac9640 1 --2- 172.21.15.78:0/1790330442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f831c004d00 0x7f831c005180 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f8328009fc0 tx=0x7f831c0057f0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f832ed9d640 1 -- 172.21.15.78:0/1790330442 reap_dead start 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f8334ac9640 1 -- 172.21.15.78:0/1790330442 shutdown_connections 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f8334ac9640 1 -- 172.21.15.78:0/1790330442 >> 172.21.15.78:0/1790330442 conn(0x7f833006b700 msgr2=0x7f83300aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:05:10.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f8334ac9640 1 -- 172.21.15.78:0/1790330442 shutdown_connections 2024-02-19T15:05:10.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.104+0000 7f8334ac9640 1 -- 172.21.15.78:0/1790330442 wait complete. 2024-02-19T15:05:10.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 Processor -- start 2024-02-19T15:05:10.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 -- start start 2024-02-19T15:05:10.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 0x7f83301edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833006fcf0 0x7f83301eb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f833014ea00 0x7f833014ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833006f4f0 con 0x7f8330158ef0 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301ee2d0 con 0x7f833006fcf0 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f8334ac9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8330150c10 con 0x7f833014ea00 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f832e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 0x7f83301edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f832e59c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 0x7f83301edd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39432/0 (socket says 172.21.15.78:39432) 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.106+0000 7f832e59c640 1 -- 172.21.15.78:0/4143256855 learned_addr learned my addr 172.21.15.78:0/4143256855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f8327fff640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833006fcf0 0x7f83301eb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832ed9d640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f833014ea00 0x7f833014ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832e59c640 1 -- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f833014ea00 msgr2=0x7f833014ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832e59c640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f833014ea00 0x7f833014ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832e59c640 1 -- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833006fcf0 msgr2=0x7f83301eb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832e59c640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833006fcf0 0x7f83301eb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832e59c640 1 -- 172.21.15.78:0/4143256855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833006f9b0 con 0x7f8330158ef0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832ed9d640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f833014ea00 0x7f833014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f832e59c640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 0x7f83301edd90 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f8320011990 tx=0x7f8320011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000fad0 con 0x7f8330158ef0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f832000fc30 con 0x7f8330158ef0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833006f6d0 con 0x7f8330158ef0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320014440 con 0x7f8330158ef0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.107+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83301eab90 con 0x7f8330158ef0 2024-02-19T15:05:10.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.108+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8330074990 con 0x7f8330158ef0 2024-02-19T15:05:10.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.108+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 89788+0+0 (secure 0 0 0) 0x7f83200145a0 con 0x7f8330158ef0 2024-02-19T15:05:10.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.109+0000 7f8326ffd640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f831006b810 0x7f831006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T15:05:10.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.109+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f832008aa20 con 0x7f8330158ef0 2024-02-19T15:05:10.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.109+0000 7f8327fff640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f831006b810 0x7f831006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T15:05:10.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.109+0000 7f8327fff640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f831006b810 0x7f831006dcd0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f8318004840 tx=0x7f8318009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T15:05:10.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.113+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f83200554b0 con 0x7f8330158ef0 2024-02-19T15:05:10.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.319+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833006fb40 con 0x7f8330158ef0 2024-02-19T15:05:10.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.319+0000 7f8326ffd640 1 -- 172.21.15.78:0/4143256855 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+1329 (secure 0 0 0) 0x7f8320058b70 con 0x7f8330158ef0 2024-02-19T15:05:10.323 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-02-19T15:05:10.323 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":210,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/538483950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":538483950},{"type":"v1","addr":"172.21.15.150:6835","nonce":538483950}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[]} 2024-02-19T15:05:10.323 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 210 2024-02-19T15:05:10.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f831006b810 msgr2=0x7f831006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:10.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f831006b810 0x7f831006dcd0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f8318004840 tx=0x7f8318009040 comp rx=0 tx=0).stop 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 msgr2=0x7f83301edd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 0x7f83301edd90 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f8320011990 tx=0x7f8320011e50 comp rx=0 tx=0).stop 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 shutdown_connections 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:6832/359840476,v1:172.21.15.78:6833/359840476] conn(0x7f831006b810 0x7f831006dcd0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f833014ea00 0x7f833014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833006fcf0 0x7f83301eb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 --2- 172.21.15.78:0/4143256855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8330158ef0 0x7f83301edd90 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T15:05:10.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.321+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 >> 172.21.15.78:0/4143256855 conn(0x7f833006b700 msgr2=0x7f83300a8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T15:05:10.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.322+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 shutdown_connections 2024-02-19T15:05:10.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-02-19T15:05:10.322+0000 7f8334ac9640 1 -- 172.21.15.78:0/4143256855 wait complete. 2024-02-19T15:05:10.335 DEBUG:tasks.cephfs.filesystem:already dead... 2024-02-19T15:05:10.336 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-19T15:05:10.336 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-19T15:05:10.336 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.345 INFO:tasks.ceph.mds.a:Stopped 2024-02-19T15:05:10.346 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-19T15:05:10.346 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.395 INFO:tasks.ceph.mds.c:Stopped 2024-02-19T15:05:10.395 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-19T15:05:10.395 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.407 INFO:tasks.ceph.mds.b:Stopped 2024-02-19T15:05:10.407 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-19T15:05:10.407 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.459 INFO:tasks.ceph.mds.d:Stopped 2024-02-19T15:05:10.460 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-19T15:05:10.460 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-02-19T15:05:10.460 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.486 INFO:tasks.ceph.osd.0:Stopped 2024-02-19T15:05:10.486 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-02-19T15:05:10.486 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.555 INFO:tasks.ceph.osd.1:Stopped 2024-02-19T15:05:10.555 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-02-19T15:05:10.555 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.628 INFO:tasks.ceph.osd.2:Stopped 2024-02-19T15:05:10.628 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-02-19T15:05:10.628 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.702 INFO:tasks.ceph.osd.3:Stopped 2024-02-19T15:05:10.702 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-02-19T15:05:10.702 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.731 INFO:tasks.ceph.osd.4:Stopped 2024-02-19T15:05:10.731 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-02-19T15:05:10.731 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.804 INFO:tasks.ceph.osd.5:Stopped 2024-02-19T15:05:10.804 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-02-19T15:05:10.804 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.876 INFO:tasks.ceph.osd.6:Stopped 2024-02-19T15:05:10.876 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-02-19T15:05:10.876 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.946 INFO:tasks.ceph.osd.7:Stopped 2024-02-19T15:05:10.946 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-19T15:05:10.946 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-02-19T15:05:10.946 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:10.972 INFO:tasks.ceph.mgr.y:Stopped 2024-02-19T15:05:10.973 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-02-19T15:05:10.973 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:11.013 INFO:tasks.ceph.mgr.x:Stopped 2024-02-19T15:05:11.013 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-19T15:05:11.014 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-02-19T15:05:11.014 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:11.025 INFO:tasks.ceph.mon.a:Stopped 2024-02-19T15:05:11.025 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-02-19T15:05:11.025 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:11.066 INFO:tasks.ceph.mon.b:Stopped 2024-02-19T15:05:11.066 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-02-19T15:05:11.066 INFO:teuthology.orchestra.run:waiting for 300 2024-02-19T15:05:11.119 INFO:tasks.ceph.mon.c:Stopped 2024-02-19T15:05:11.119 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-19T15:05:11.119 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'Replacing daemon mds' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(FS_DEGRADED\)' | head -n 1 2024-02-19T15:05:11.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T15:05:11.167 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-02-19T15:05:11.339 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T15:05:11.339 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-02-19T15:05:11.421 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T15:05:11.421 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-02-19T15:05:11.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T15:05:11.481 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-02-19T15:05:11.555 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T15:05:11.555 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-02-19T15:05:11.711 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T15:05:11.711 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-02-19T15:05:11.801 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T15:05:11.801 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-02-19T15:05:11.854 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T15:05:11.855 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-02-19T15:05:11.914 INFO:tasks.ceph:Archiving mon data... 2024-02-19T15:05:11.915 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/data/mon.a.tgz 2024-02-19T15:05:11.915 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-02-19T15:05:11.930 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.vMDS9O3uOE 2024-02-19T15:05:11.931 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.vMDS9O3uOE 2024-02-19T15:05:12.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.vMDS9O3uOE 2024-02-19T15:05:12.198 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.vMDS9O3uOE is 548KB 2024-02-19T15:05:12.298 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.vMDS9O3uOE 2024-02-19T15:05:12.312 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/data/mon.b.tgz 2024-02-19T15:05:12.312 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-19T15:05:12.329 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.tqOREHhIoc 2024-02-19T15:05:12.330 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tqOREHhIoc 2024-02-19T15:05:12.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.tqOREHhIoc 2024-02-19T15:05:12.606 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.tqOREHhIoc is 559KB 2024-02-19T15:05:12.703 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.tqOREHhIoc 2024-02-19T15:05:12.718 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/data/mon.c.tgz 2024-02-19T15:05:12.718 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-19T15:05:12.772 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.De4HcIMWI7 2024-02-19T15:05:12.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.De4HcIMWI7 2024-02-19T15:05:12.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.De4HcIMWI7 2024-02-19T15:05:13.028 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.De4HcIMWI7 is 559KB 2024-02-19T15:05:13.098 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.De4HcIMWI7 2024-02-19T15:05:13.113 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-19T15:05:13.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-19T15:05:13.115 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-19T15:05:13.198 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-19T15:05:13.233 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-19T15:05:13.234 INFO:tasks.ceph:Archiving crash dumps... 2024-02-19T15:05:13.235 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/remote/smithi078/crash 2024-02-19T15:05:13.236 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-19T15:05:13.267 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/remote/smithi150/crash 2024-02-19T15:05:13.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-19T15:05:13.300 INFO:tasks.ceph:Shutting down logrotate 2024-02-19T15:05:13.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:05:13.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-19T15:05:13.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-02-19T15:05:13.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-02-19T15:05:13.444 INFO:tasks.ceph:Compressing logs... 2024-02-19T15:05:13.444 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T15:05:13.460 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T15:05:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38143.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38143.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38143.log.gz 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38023.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44310.log 2024-02-19T15:05:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38183.log 2024-02-19T15:05:13.538 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38023.log.gz 2024-02-19T15:05:13.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.1.44310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38223.log 2024-02-19T15:05:13.538 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44310.log.gz 2024-02-19T15:05:13.538 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-02-19T15:05:13.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38183.log: /var/log/ceph/ceph-client.admin.38223.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38183.log.gz 2024-02-19T15:05:13.538 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-osd.5.log 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38223.log.gz 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38103.log 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38303.log 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38263.log 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38103.log.gz 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38303.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38303.log.gz 2024-02-19T15:05:13.539 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38263.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38263.log.gz 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.x.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.b.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-19T15:05:13.540 INFO:teuthology.orchestra.run.smithi150.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-02-19T15:05:13.546 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38063.log 2024-02-19T15:05:13.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-02-19T15:05:13.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38063.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38063.log.gz 2024-02-19T15:05:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-02-19T15:05:13.622 INFO:teuthology.orchestra.run.smithi150.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-02-19T15:05:13.923 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62272.log 2024-02-19T15:05:13.923 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69361.log 2024-02-19T15:05:13.923 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69067.log 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69361.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62272.log.gz 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71621.log 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69361.log.gz 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69067.log.gz 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66008.log 2024-02-19T15:05:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88980.log 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71621.log: /var/log/ceph/ceph-client.admin.66008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66963.log 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71621.log.gz 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74058.log 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66008.log.gz 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63030.log 88.5% 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88980.log.gz 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61547.log 2024-02-19T15:05:13.925 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66963.log.gz 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63030.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74058.log.gz 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78944.log 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63030.log.gz 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48026.log 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78944.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61547.log.gz 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84520.log 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69960.log 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78944.log.gz 2024-02-19T15:05:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58994.log 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48026.log.gz 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74184.log 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84520.log.gz 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58994.log: gzip -5 --verbose 87.8% -- /var/log/ceph/ceph-client.admin.55351.log -- replaced with /var/log/ceph/ceph-client.admin.69960.log.gz 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74184.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58994.log.gz 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81960.log 2024-02-19T15:05:13.927 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74184.log.gz 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56262.log 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60026.log 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55351.log.gz 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67526.log 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81960.log.gz 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70254.log 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56262.log.gz 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67526.log: gzip -5 89.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.60026.log.gz /var/log/ceph/ceph-mon.a.log 2024-02-19T15:05:13.928 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47544.log 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67526.log.gz 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.a.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70254.log.gz 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58141.log 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87054.log 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58141.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47544.log.gz 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66795.log 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58141.log.gz 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62759.log 2024-02-19T15:05:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50434.log 89.6% 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87054.log.gz 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62759.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66795.log.gz 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43943.log 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63342.log 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43943.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.62759.log.gz 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50434.log.gz 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49768.log 2024-02-19T15:05:13.930 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43943.log.gz 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88518.log 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49768.log: gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.63342.log.gz /var/log/ceph/ceph-client.admin.52854.log 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85953.log 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49768.log.gz 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52854.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88518.log.gz 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2024-02-19T15:05:13.931 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52854.log.gz 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67186.log 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92611.log 88.1% 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85953.log.gz 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6%/var/log/ceph/ceph-client.admin.67186.log: -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67144.log 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61300.log 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67186.log.gz 2024-02-19T15:05:13.932 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67144.log: gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.92611.log.gz /var/log/ceph/ceph-client.admin.85815.log 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67144.log.gz 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61300.log.gz 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61342.log 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85815.log.gz 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81361.log 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80598.log 2024-02-19T15:05:13.933 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81361.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61342.log.gz 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76539.log 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81361.log.gz 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81463.log 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76539.log: gzip 88.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.80598.log.gz -- 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.68310.log 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81463.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76539.log.gz -5 2024-02-19T15:05:13.934 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86164.log 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68310.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81463.log.gz 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63384.log 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86164.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68310.log.gz 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69109.log 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68012.log 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86164.log.gz 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69109.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63384.log.gz 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74350.log 2024-02-19T15:05:13.935 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69109.log.gz 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51389.log 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90840.log 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51389.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74350.log.gz 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75708.log 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.68012.log.gz 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51389.log.gz 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56930.log 2024-02-19T15:05:13.936 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1%/var/log/ceph/ceph-client.admin.75708.log: -- replaced with /var/log/ceph/ceph-client.admin.90840.log.gz 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71747.log 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75708.log.gz 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50770.log 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7%/var/log/ceph/ceph-client.admin.71747.log: -- replaced with /var/log/ceph/ceph-client.admin.56930.log.gz 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58952.log 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50770.log: gzip -5 --verbose 88.6% -- -- replaced with /var/log/ceph/ceph-client.admin.71747.log.gz /var/log/ceph/ceph-client.admin.72768.log 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.937 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58952.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50770.log.gz 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52312.log 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58952.log.gz 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91465.log 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52312.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.72768.log.gz -5 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90696.log 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88206.log 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90696.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91465.log.gz 2024-02-19T15:05:13.938 INFO:teuthology.orchestra.run.smithi078.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.52312.log.gz 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46830.log 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90696.log.gz 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69686.log 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4%/var/log/ceph/ceph-client.admin.46830.log: -- replaced with /var/log/ceph/ceph-client.admin.88206.log.gz 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62946.log 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86206.log 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46830.log.gz 2024-02-19T15:05:13.939 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62946.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69686.log.gz 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50957.log 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62946.log.gz 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50140.log 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51515.log 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86206.log.gz 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50957.log.gz 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54432.log 2024-02-19T15:05:13.940 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50140.log.gz 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86826.log 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74915.log 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51515.log.gz 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54432.log.gz 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60483.log 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86826.log.gz 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70891.log 2024-02-19T15:05:13.941 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67358.log 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74915.log.gz 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60483.log.gz 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79978.log 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70891.log.gz 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52451.log 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78617.log 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67358.log.gz 2024-02-19T15:05:13.942 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79978.log.gz 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70698.log 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52451.log.gz 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71663.log 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65527.log 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78617.log.gz 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3%/var/log/ceph/ceph-client.admin.71663.log: -- replaced with /var/log/ceph/ceph-client.admin.70698.log.gz 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55439.log 2024-02-19T15:05:13.943 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0%/var/log/ceph/ceph-client.admin.65527.log: -- replaced with /var/log/ceph/ceph-client.admin.71663.log.gz 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46295.log 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89838.log 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65527.log.gz 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46295.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55439.log.gz 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79194.log 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46295.log.gz 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73974.log 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79194.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89838.log.gz 2024-02-19T15:05:13.944 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83269.log 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73974.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79194.log.gz 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86616.log 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73974.log.gz 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64087.log/var/log/ceph/ceph-client.admin.83269.log: 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75434.log 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83269.log.gz 2024-02-19T15:05:13.945 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7%/var/log/ceph/ceph-client.admin.64087.log: -- replaced with /var/log/ceph/ceph-client.admin.86616.log.gz 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92427.log 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64087.log.gz 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63552.log 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92427.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75434.log.gz 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85995.log 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92427.log.gz 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70002.log 2024-02-19T15:05:13.946 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63552.log.gz 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57969.log 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84025.log 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85995.log.gz 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70002.log.gz 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69235.log 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57969.log.gz 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50350.log 2024-02-19T15:05:13.947 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88686.log 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84025.log.gz 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69235.log.gz 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89586.log 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88686.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50350.log.gz 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45308.log 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89922.log 2024-02-19T15:05:13.948 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.88686.log.gz 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89586.log.gz 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71224.log 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45308.log.gz 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.89922.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.88728.log 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47292.log 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89922.log.gz 2024-02-19T15:05:13.949 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71224.log.gz 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60754.log 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.88728.log.gz 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80552.log 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46926.log 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47292.log.gz 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60754.log.gz 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81193.log 2024-02-19T15:05:13.950 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47088.log 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46926.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80552.log.gz 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76044.log 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46926.log.gz 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47088.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81193.log.gz 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59900.log 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84067.log 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47088.log.gz 2024-02-19T15:05:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59900.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76044.log.gz 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59078.log 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59900.log.gz 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47166.log 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72229.log 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84067.log.gz 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47166.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59078.log.gz 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47166.log.gz 2024-02-19T15:05:13.952 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86403.log 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91339.log 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72229.log.gz 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86403.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gz 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59659.log 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86403.log.gz 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51037.log 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77520.log 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91339.log.gz 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51037.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59659.log.gz 2024-02-19T15:05:13.953 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58210.log 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51037.log.gz 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60278.log 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65316.log 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77520.log.gz 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60278.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58210.log.gz 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65756.log 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60278.log.gz 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91549.log 2024-02-19T15:05:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91965.log 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65316.log.gz 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91549.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65756.log.gz 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45948.log 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91549.log.gz 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70824.log 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82479.log 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91965.log.gz 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70824.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45948.log.gz 2024-02-19T15:05:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55128.log 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74831.log 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.70824.log.gz 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55128.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82479.log.gz 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43714.log 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49894.log 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55128.log.gz 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74831.log: /var/log/ceph/ceph-client.admin.43714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75136.log 2024-02-19T15:05:13.956 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74831.log.gz 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49894.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43714.log.gz 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52270.log 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75136.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49894.log.gz 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73309.log 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52270.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75136.log.gz 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73351.log 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52270.log.gz 2024-02-19T15:05:13.957 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46634.log 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65358.log 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73309.log.gz 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46634.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73351.log.gz 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45990.log 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46634.log.gz 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75665.log 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71537.log 2024-02-19T15:05:13.958 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65358.log.gz 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75665.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45990.log.gz 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65882.log 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75665.log.gz 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52559.log 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71537.log: /var/log/ceph/ceph-client.admin.65882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88416.log 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71537.log.gz 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52559.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65882.log.gz 2024-02-19T15:05:13.959 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54516.log 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52559.log.gz 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78659.log 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92511.log 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88416.log.gz 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78659.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54516.log.gz 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70782.log 2024-02-19T15:05:13.960 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78659.log.gz 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48553.log 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70782.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92511.log.gz 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63174.log 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48553.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70782.log.gz 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88374.log 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48553.log.gz 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2024-02-19T15:05:13.961 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71369.log 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63174.log.gz 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48471.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88374.log.gz 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92259.log 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81151.log 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83479.log 2024-02-19T15:05:13.962 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71369.log.gz 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81151.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92259.log.gz 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58424.log 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81151.log.gz 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.83479.log: --verbose -- /var/log/ceph/ceph-client.admin.80062.log 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55086.log 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83479.log.gz 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80062.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58424.log.gz 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76411.log 2024-02-19T15:05:13.963 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80062.log.gz 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66428.log 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46631.log 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55086.log.gz 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66428.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76411.log.gz 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49402.log 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66428.log.gz 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71892.log 2024-02-19T15:05:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83647.log 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46631.log.gz 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71892.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49402.log.gz 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51601.log 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71892.log.gz 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87600.log 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72903.log 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83647.log.gz 2024-02-19T15:05:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87600.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51601.log.gz 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61468.log 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87600.log.gz 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58868.log 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72903.log: /var/log/ceph/ceph-client.admin.61468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58508.log 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72903.log.gz 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58868.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61468.log.gz 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73806.log 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58868.log.gz 2024-02-19T15:05:13.966 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69520.log 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73806.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.68398.log -- replaced with /var/log/ceph/ceph-client.admin.58508.log.gz 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69520.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73806.log.gz 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91839.log 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69520.log.gz 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56430.log 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59036.log 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56430.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91839.log.gz 2024-02-19T15:05:13.967 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83857.log 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.68398.log.gz 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56430.log.gz 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44382.log 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59036.log.gz 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87912.log 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68192.log 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83857.log.gz 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87912.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44382.log.gz 2024-02-19T15:05:13.968 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57182.log 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87912.log.gz 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76652.log 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56720.log 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68192.log.gz 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57182.log.gz 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70212.log 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76652.log.gz 2024-02-19T15:05:13.969 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72596.log 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85857.log 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56720.log.gz 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72596.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70212.log.gz 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86079.log 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72596.log.gz 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90024.log/var/log/ceph/ceph-client.admin.85857.log: 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73607.log 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85857.log.gz 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86079.log.gz 2024-02-19T15:05:13.970 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77076.log 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90024.log.gz 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85647.log 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78034.log 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73607.log.gz 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51826.log 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.77076.log.gz 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85647.log.gz 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69319.log 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51826.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78034.log.gz 2024-02-19T15:05:13.971 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57602.log 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51826.log.gz 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67586.log 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69319.log.gz 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53460.log 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84942.log 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57602.log.gz 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53460.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67586.log.gz 2024-02-19T15:05:13.972 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83395.log 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53460.log.gz 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69876.log 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73700.log 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84942.log.gz 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69876.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83395.log.gz 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86361.log 2024-02-19T15:05:13.973 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69876.log.gz 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82985.log 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73700.log.gz 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60236.log 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45266.log 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86361.log.gz 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60236.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82985.log.gz 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62230.log 2024-02-19T15:05:13.974 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60236.log.gz 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54642.log 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70656.log 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45266.log.gz 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54642.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62230.log.gz 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83731.log 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54642.log.gz 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46298.log 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83731.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70656.log.gz 2024-02-19T15:05:13.975 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76583.log 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83731.log.gz 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50392.log 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46298.log.gz 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65569.log 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50392.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76583.log.gz 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84434.log 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50392.log.gz 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59389.log 2024-02-19T15:05:13.976 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62100.log 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59389.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65569.log.gz 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84434.log.gz 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66470.log 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86910.log 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59389.log.gz 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62100.log: /var/log/ceph/ceph-client.admin.66470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74579.log 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62100.log.gz 2024-02-19T15:05:13.977 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86910.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66470.log.gz 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90234.log 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74579.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86910.log.gz 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62548.log 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74579.log.gz 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63799.log 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90234.log.gz/var/log/ceph/ceph-client.admin.62548.log: 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70128.log 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79026.log 87.9% 2024-02-19T15:05:13.978 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62548.log.gz 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70128.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63799.log.gz 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85278.log 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70128.log.gz 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51738.log 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85278.log: gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.80146.log -- replaced with /var/log/ceph/ceph-client.admin.79026.log.gz 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51738.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85278.log.gz 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91507.log 2024-02-19T15:05:13.979 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51738.log.gz 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62398.log 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91507.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80146.log.gz 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47586.log 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91507.log.gz 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57927.log 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62398.log.gz 2024-02-19T15:05:13.980 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.47586.log: --verbose -- /var/log/ceph/ceph-client.admin.64440.log 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74663.log 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47586.log.gz 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64440.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57927.log.gz 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79660.log 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64440.log.gz 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84774.log 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66879.log 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74663.log.gz 2024-02-19T15:05:13.981 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79660.log.gz 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59779.log 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84774.log.gz 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75304.log 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59779.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.66879.log.gz -5 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74100.log 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75304.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59779.log.gz 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79786.log 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75304.log.gz 2024-02-19T15:05:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90276.log 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79786.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.74100.log.gz -- 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.56556.log 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55793.log 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79786.log.gz 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56556.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90276.log.gz 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71041.log 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55793.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56556.log.gz 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49978.log 2024-02-19T15:05:13.983 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89376.log 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55793.log.gz 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49978.log: gzip 89.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.71041.log.gz -- 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.60320.log 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89376.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49978.log.gz 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58742.log 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89376.log.gz 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78160.log 2024-02-19T15:05:13.984 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79744.log 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60320.log.gz 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78160.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58742.log.gz 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62632.log 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78160.log.gz 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66344.log 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50830.log 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79744.log.gz 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68149.log 2024-02-19T15:05:13.985 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66344.log.gz 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr: 91.4%/var/log/ceph/ceph-client.admin.50830.log: -- replaced with /var/log/ceph/ceph-client.admin.62632.log.gz 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50644.log 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68149.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50830.log.gz 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89250.log 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68149.log.gz 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48721.log 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50644.log.gz 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.89250.log: /var/log/ceph/ceph-client.admin.51263.log 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59858.log 2024-02-19T15:05:13.986 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89250.log.gz 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51263.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48721.log.gz 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70999.log 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51263.log.gz 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55649.log 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49573.log 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59858.log.gz 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55649.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70999.log.gz 2024-02-19T15:05:13.987 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77310.log 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55649.log.gz 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76327.log 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69025.log 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49573.log.gz 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76327.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77310.log.gz 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80640.log 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76327.log.gz 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69025.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31677.log 2024-02-19T15:05:13.988 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48340.log 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69025.log.gz 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.tmp-client.admin.31677.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31677.log.gz 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92301.log 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.80640.log.gz 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.0.48340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83046.log 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48340.log.gz 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63720.log 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83046.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92301.log.gz 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74747.log 2024-02-19T15:05:13.989 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83046.log.gz 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.63720.log: /var/log/ceph/ceph-client.admin.58910.log 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54079.log 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63720.log.gz 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74747.log.gz 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52643.log 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58910.log.gz 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57056.log 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50602.log 2024-02-19T15:05:13.990 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54079.log.gz 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66386.log 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52643.log.gz 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50602.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57056.log.gz 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73932.log 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50056.log 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50602.log.gz 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66386.log.gz 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76736.log 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73932.log.gz 2024-02-19T15:05:13.991 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67274.log 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82210.log 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50056.log.gz 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67274.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76736.log.gz 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73201.log 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67274.log.gz 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79238.log 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87828.log 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82210.log.gz 2024-02-19T15:05:13.992 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79238.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73201.log.gz 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90444.log 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79238.log.gz 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67712.log 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74016.log 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87828.log.gz 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67712.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90444.log.gz 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92175.log 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60964.log 2024-02-19T15:05:13.993 INFO:teuthology.orchestra.run.smithi078.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.67712.log.gz 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92175.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74016.log.gz 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61006.log 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46032.log 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60964.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92175.log.gz 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79828.log 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60964.log.gz 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46032.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61006.log.gz 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92133.log 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84350.log 2024-02-19T15:05:13.994 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46032.log.gz 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92133.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79828.log.gz 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83190.log 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92133.log.gz 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80416.log 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48889.log 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84350.log.gz 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80416.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83190.log.gz 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71411.log 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80416.log.gz 2024-02-19T15:05:13.995 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58592.log 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55309.log 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48889.log.gz 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58592.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71411.log.gz 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45906.log 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58592.log.gz 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49657.log 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67778.log 2024-02-19T15:05:13.996 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55309.log.gz 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49657.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45906.log.gz 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47502.log 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49657.log.gz 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65400.log 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78076.log 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67778.log.gz 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65400.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47502.log.gz 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71579.log 2024-02-19T15:05:13.997 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65400.log.gz 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.78076.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64818.log 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68669.log 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78076.log.gz 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64818.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71579.log.gz 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73159.log 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64818.log.gz 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49810.log 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68669.log: /var/log/ceph/ceph-client.admin.73159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43987.log 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68669.log.gz 2024-02-19T15:05:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49810.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73159.log.gz 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45576.log 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49810.log.gz 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46297.log 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82707.log 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43987.log.gz 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46297.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45576.log.gz 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85371.log 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46297.log.gz 2024-02-19T15:05:13.999 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79702.log 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74999.log 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82707.log.gz 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79702.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85371.log.gz 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53022.log 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79702.log.gz 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82254.log 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62016.log 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74999.log.gz 2024-02-19T15:05:14.000 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82254.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53022.log.gz 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78785.log 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82254.log.gz 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84308.log 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81880.log 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62016.log.gz 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84308.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78785.log.gz 2024-02-19T15:05:14.001 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88038.log 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84308.log.gz 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45822.log 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81880.log: /var/log/ceph/ceph-client.admin.88038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70296.log 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.81880.log.gz 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45822.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88038.log.gz 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84226.log 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45822.log.gz 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66921.log 2024-02-19T15:05:14.002 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58784.log 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70296.log.gz 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66921.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84226.log.gz 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76495.log 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66921.log.gz 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68838.log 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58784.log: /var/log/ceph/ceph-client.admin.76495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64398.log 2024-02-19T15:05:14.003 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58784.log.gz 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68838.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76495.log.gz 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52896.log 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68838.log.gz 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89460.log 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78202.log 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64398.log.gz 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89460.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52896.log.gz 2024-02-19T15:05:14.004 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54750.log 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89460.log.gz 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78328.log 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64482.log 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78202.log.gz 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78328.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54750.log.gz 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62862.log 2024-02-19T15:05:14.005 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78328.log.gz 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44214.log 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57308.log 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64482.log.gz 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44214.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62862.log.gz 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81506.log 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44214.log.gz 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66633.log 2024-02-19T15:05:14.006 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60526.log 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57308.log.gz 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66633.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81506.log.gz 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59721.log 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66633.log.gz 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53418.log 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76992.log 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60526.log.gz 2024-02-19T15:05:14.007 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59721.log.gz 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53418.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53418.log.gz 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70506.log 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75519.log 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76992.log.gz 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83088.log 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79576.log 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75519.log.gz 2024-02-19T15:05:14.008 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53502.log 91.0% 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70506.log.gz 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr:gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83088.log.gz -5 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80500.log 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79576.log: /var/log/ceph/ceph-client.admin.53502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60880.log 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79576.log.gz 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80500.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53502.log.gz 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53106.log 2024-02-19T15:05:14.009 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82665.log 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80500.log.gz/var/log/ceph/ceph-client.admin.60880.log: 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56094.log 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60880.log.gz 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53106.log.gz 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91713.log 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82665.log: /var/log/ceph/ceph-client.admin.56094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46928.log 2024-02-19T15:05:14.010 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82665.log.gz 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91713.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56094.log.gz 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59259.log 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91713.log.gz 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70740.log 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61132.log 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3%/var/log/ceph/ceph-client.admin.59259.log: -- replaced with /var/log/ceph/ceph-client.admin.46928.log.gz 2024-02-19T15:05:14.011 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59599.log 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59259.log.gz 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70740.log.gz 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55523.log 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56178.log 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61132.log.gz 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7%/var/log/ceph/ceph-client.admin.55523.log: -- replaced with /var/log/ceph/ceph-client.admin.59599.log.gz 2024-02-19T15:05:14.012 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89334.log 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56178.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55523.log.gz 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86308.log 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89334.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56178.log.gz 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63468.log 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89334.log.gz 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90961.log 2024-02-19T15:05:14.013 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68267.log 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86308.log.gz 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90961.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63468.log.gz 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58550.log 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90961.log.gz 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53712.log/var/log/ceph/ceph-client.admin.68267.log: 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57560.log 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68267.log.gz 2024-02-19T15:05:14.014 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53712.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58550.log.gz 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66512.log 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53712.log.gz 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60838.log 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46967.log 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57560.log.gz 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60838.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66512.log.gz 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56472.log 2024-02-19T15:05:14.015 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60838.log.gz 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70044.log 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72638.log 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46967.log.gz 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56472.log.gz 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78118.log 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70044.log.gz 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60586.log 2024-02-19T15:05:14.016 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80332.log 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72638.log.gz 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60586.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78118.log.gz 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60586.log.gz 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54834.log 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48511.log: gzip -5 --verbose 87.7% -- -- replaced with /var/log/ceph/ceph-client.admin.80332.log.gz /var/log/ceph/ceph-client.admin.74458.log 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54834.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2024-02-19T15:05:14.017 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90756.log 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54834.log.gz 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37626.log 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73243.log 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74458.log.gz 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90756.log.gz 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77352.log 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63636.log 2024-02-19T15:05:14.018 INFO:teuthology.orchestra.run.smithi078.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.37626.log.gz 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50728.log 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63636.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77352.log.gz 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45350.log 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.73243.log.gz 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63636.log.gz 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44298.log 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50728.log.gz 2024-02-19T15:05:14.019 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66302.log 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60068.log 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45350.log.gz 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44298.log.gz 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77675.log 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66302.log.gz 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46296.log 2024-02-19T15:05:14.020 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60422.log 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60068.log.gz 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77675.log.gz 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83311.log 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46296.log.gz 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51177.log 2024-02-19T15:05:14.021 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60422.log.gz 2024-02-19T15:05:14.022 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83311.log.gz 2024-02-19T15:05:14.022 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90108.log 2024-02-19T15:05:14.022 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2024-02-19T15:05:14.022 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89964.log 2024-02-19T15:05:14.022 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84984.log 2024-02-19T15:05:14.022 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51177.log.gz 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90108.log.gz 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57266.log 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89964.log.gz 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92049.log 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75476.log 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84984.log.gz 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2%/var/log/ceph/ceph-client.admin.92049.log: -- replaced with /var/log/ceph/ceph-client.admin.57266.log.gz 2024-02-19T15:05:14.023 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80458.log 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92049.log.gz 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51645.log 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75476.log.gz 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80458.log.gz 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82168.log 2024-02-19T15:05:14.024 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92343.log 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73477.log 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51645.log.gz 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82168.log.gz 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57885.log 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92343.log.gz 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44340.log 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73477.log: /var/log/ceph/ceph-client.admin.57885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52036.log 2024-02-19T15:05:14.025 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73477.log.gz 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57885.log.gz 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88122.log 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44340.log.gz 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72103.log 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52036.log.gz 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88122.log.gz 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46930.log 2024-02-19T15:05:14.026 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74537.log 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48637.log 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72103.log.gz 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46930.log.gz 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71977.log 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74537.log.gz 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86742.log 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46292.log 2024-02-19T15:05:14.027 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48637.log.gz 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71977.log.gz/var/log/ceph/ceph-client.admin.86742.log: 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54558.log 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86742.log.gz 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82901.log 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54558.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46292.log.gz 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64268.log 2024-02-19T15:05:14.028 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54558.log.gz 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52938.log 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82901.log.gz 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51305.log 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64268.log.gz 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61384.log 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52938.log.gz 2024-02-19T15:05:14.029 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51305.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51305.log.gz 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87996.log 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77268.log 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87996.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61384.log.gz 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.67442.log 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87996.log.gz 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81319.log 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77268.log.gz 2024-02-19T15:05:14.030 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60922.log 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81319.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.67442.log.gz 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.79366.log 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81319.log.gz 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89628.log 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60922.log.gz 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68796.log 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89628.log: gzip 88.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.79366.log.gz -- 2024-02-19T15:05:14.031 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.85473.log 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68796.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89628.log.gz 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55216.log 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68796.log.gz 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45408.log 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82042.log 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55216.log.gz 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45408.log: gzip 91.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.85473.log.gz -- 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.52751.log 2024-02-19T15:05:14.032 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87558.log 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82042.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45408.log.gz 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64860.log 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82042.log.gz 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82859.log 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64860.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87558.log.gz 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82084.log 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52751.log.gz 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90066.log/var/log/ceph/ceph-client.admin.82859.log: 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.64860.log.gz 2024-02-19T15:05:14.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82084.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82859.log.gz 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81590.log 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82084.log.gz 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91213.log 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90066.log.gz 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87264.log 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81590.log.gz/var/log/ceph/ceph-client.admin.91213.log: 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80020.log 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87264.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.91213.log.gz -5 2024-02-19T15:05:14.034 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.82623.log 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80020.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87264.log.gz 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73658.log 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82623.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80020.log.gz 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84476.log 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73658.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82623.log.gz 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56972.log 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73658.log.gz 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72187.log 2024-02-19T15:05:14.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69277.log 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84476.log.gz 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72187.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56972.log.gz 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49615.log 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72187.log.gz 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75792.log 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84732.log 2024-02-19T15:05:14.036 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69277.log.gz 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75792.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49615.log.gz 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77478.log 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75792.log.gz 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62590.log 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54876.log 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84732.log.gz 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62590.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77478.log.gz 2024-02-19T15:05:14.037 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72991.log 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62590.log.gz 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60194.log 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79110.log 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54876.log.gz 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72991.log.gz 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83815.log 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60194.log.gz 2024-02-19T15:05:14.038 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63961.log 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83941.log 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79110.log.gz 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63961.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83815.log.gz 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86574.log 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63961.log.gz 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63594.log 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53376.log 2024-02-19T15:05:14.039 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83941.log.gz 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.86574.log.gz 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63594.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63594.log.gz 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43901.log 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56804.log 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78388.log 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53376.log.gz 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56804.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43901.log.gz 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67820.log 2024-02-19T15:05:14.040 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56804.log.gz 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91297.log 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65714.log 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78388.log.gz 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91297.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67820.log.gz 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79534.log 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91297.log.gz 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.65714.log: -- /var/log/ceph/ceph-client.admin.91255.log 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89670.log 2024-02-19T15:05:14.041 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65714.log.gz 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91255.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79534.log.gz 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54390.log 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91255.log.gz 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65137.log 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74307.log 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.89670.log.gz 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54390.log.gz/var/log/ceph/ceph-client.admin.65137.log: 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62188.log 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65137.log.gz 2024-02-19T15:05:14.042 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70380.log 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65179.log 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74307.log.gz 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9%/var/log/ceph/ceph-client.admin.70380.log: -- replaced with /var/log/ceph/ceph-client.admin.62188.log.gz 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71789.log 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70380.log.gz 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71453.log 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82391.log 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71789.log.gz 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71453.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-02-19T15:05:14.043 INFO:teuthology.orchestra.run.smithi078.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.65179.log.gz 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52980.log 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82391.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71453.log.gz 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60796.log 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82391.log.gz 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61877.log 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60796.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52980.log.gz 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58294.log 2024-02-19T15:05:14.044 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52493.log 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5%/var/log/ceph/ceph-client.admin.61877.log: -- replaced with /var/log/ceph/ceph-client.admin.60796.log.gz 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58294.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61877.log.gz 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77992.log 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58294.log.gz 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52493.log: /var/log/ceph/ceph-client.admin.77992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52409.log 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77992.log.gz/var/log/ceph/ceph-client.admin.46927.log: 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89880.log 2024-02-19T15:05:14.045 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52493.log.gz 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70572.log 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80374.log 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52409.log.gz 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89880.log: /var/log/ceph/ceph-client.admin.70572.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89880.log.gz 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55172.log 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70572.log.gz 2024-02-19T15:05:14.046 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63879.log 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80374.log: /var/log/ceph/ceph-client.admin.55172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52601.log 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80374.log.gz 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55172.log.gz 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69151.log 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63879.log.gz 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76453.log 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83563.log 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52601.log.gz 2024-02-19T15:05:14.047 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69151.log: /var/log/ceph/ceph-client.admin.76453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88896.log 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69151.log.gz 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54209.log 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83563.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76453.log.gz 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81235.log 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83563.log.gz 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88896.log.gz 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2024-02-19T15:05:14.048 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81235.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54209.log.gz 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91423.log 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81235.log.gz 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48931.log: /var/log/ceph/ceph-client.admin.91423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82539.log 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46630.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91423.log.gz 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75178.log 2024-02-19T15:05:14.049 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65010.log 2024-02-19T15:05:14.050 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2024-02-19T15:05:14.050 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82539.log: /var/log/ceph/ceph-client.admin.75178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72724.log 2024-02-19T15:05:14.050 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82539.log.gz 2024-02-19T15:05:14.050 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75178.log.gz 2024-02-19T15:05:14.050 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53791.log 2024-02-19T15:05:14.050 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65010.log: /var/log/ceph/ceph-client.admin.72724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68627.log 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65010.log.gz 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72724.log.gz 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54684.log 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53791.log: /var/log/ceph/ceph-client.admin.68627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55853.log 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53791.log.gz 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68627.log.gz 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74262.log 2024-02-19T15:05:14.051 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55267.log 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55853.log: /var/log/ceph/ceph-client.admin.74262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68078.log 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55853.log.gz 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54684.log.gz 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74262.log.gz 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89544.log 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53292.log 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68078.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55267.log.gz 2024-02-19T15:05:14.052 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66176.log 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89544.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68078.log.gz 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83521.log 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89544.log.gz 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53292.log.gz 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80983.log 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66176.log.gz 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81067.log 2024-02-19T15:05:14.053 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83521.log: /var/log/ceph/ceph-client.admin.80983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51347.log 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83521.log.gz 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81067.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80983.log.gz 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73848.log 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81067.log.gz 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66092.log 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51347.log: /var/log/ceph/ceph-client.admin.73848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-02-19T15:05:14.054 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51347.log.gz 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66092.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73848.log.gz 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69746.log 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66092.log.gz 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67400.log 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37876.log: /var/log/ceph/ceph-client.admin.69746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85320.log 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67400.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69746.log.gz 2024-02-19T15:05:14.055 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59557.log 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67400.log.gz 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88602.log 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85320.log: /var/log/ceph/ceph-client.admin.59557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90318.log 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85320.log.gz 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88602.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59557.log.gz 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72819.log 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88602.log.gz 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63510.log 2024-02-19T15:05:14.056 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90318.log: /var/log/ceph/ceph-client.admin.72819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61965.log 2024-02-19T15:05:14.057 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90318.log.gz 2024-02-19T15:05:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63510.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72819.log.gz 2024-02-19T15:05:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46294.log 2024-02-19T15:05:14.057 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63510.log.gz 2024-02-19T15:05:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82943.log 2024-02-19T15:05:14.057 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56846.log 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61965.log.gz 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82943.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46294.log.gz 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55565.log 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79282.log 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56846.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82943.log.gz 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91881.log 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56846.log.gz 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79282.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55565.log.gz 2024-02-19T15:05:14.058 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60110.log 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91881.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79282.log.gz 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45224.log 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91881.log.gz 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64650.log 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60110.log: /var/log/ceph/ceph-client.admin.45224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57723.log 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60110.log.gz 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1%/var/log/ceph/ceph-client.admin.64650.log: -- replaced with /var/log/ceph/ceph-client.admin.45224.log.gz 2024-02-19T15:05:14.059 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75220.log 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64650.log.gz 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56888.log 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50266.log 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57723.log.gz 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75220.log: /var/log/ceph/ceph-client.admin.56888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85911.log 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75220.log.gz 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56888.log.gz 2024-02-19T15:05:14.060 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62146.log 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77226.log 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50266.log.gz 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62146.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85911.log.gz 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77580.log 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77226.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62146.log.gz 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62988.log 2024-02-19T15:05:14.061 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77580.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77226.log.gz 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45450.log 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77580.log.gz 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86490.log 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76824.log 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7%/var/log/ceph/ceph-client.admin.45450.log: -- replaced with /var/log/ceph/ceph-client.admin.62988.log.gz 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86490.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45450.log.gzgzip 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68585.log 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86490.log.gz 2024-02-19T15:05:14.062 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77805.log 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76824.log: /var/log/ceph/ceph-client.admin.68585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65798.log 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76824.log.gz 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77805.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68585.log.gz 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57098.log 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68880.log 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.77805.log.gz 2024-02-19T15:05:14.063 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57098.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65798.log.gz 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81548.log 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57098.log.gz 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82792.log 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68880.log: /var/log/ceph/ceph-client.admin.81548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92469.log 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68880.log.gz 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82792.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81548.log.gz 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77034.log 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78864.log 2024-02-19T15:05:14.064 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92469.log: /var/log/ceph/ceph-client.admin.77034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54960.log 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.82792.log.gz 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92469.log.gz 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77034.log.gz 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89292.log 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53334.log 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78864.log.gz 2024-02-19T15:05:14.065 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89292.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54960.log.gz 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49488.log 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89292.log.gz 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84648.log 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91797.log 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49488.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53334.log.gz 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63678.log 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49488.log.gz 2024-02-19T15:05:14.066 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84648.log.gz 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67670.log 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63678.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91797.log.gz 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88290.log 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63678.log.gz 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46929.log 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67670.log: /var/log/ceph/ceph-client.admin.88290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58340.log 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67670.log.gz 2024-02-19T15:05:14.067 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46929.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88290.log.gz 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64045.log 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46929.log.gz 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48763.log 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58340.log.gz 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74789.log 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64045.log: /var/log/ceph/ceph-client.admin.48763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77950.log 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64045.log.gz 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74789.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48763.log.gz 2024-02-19T15:05:14.068 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59942.log 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2%/var/log/ceph/ceph-client.admin.77950.log: -- replaced with /var/log/ceph/ceph-client.admin.74789.log.gz 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56304.log 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77950.log.gz 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61835.log 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59942.log: /var/log/ceph/ceph-client.admin.56304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89418.log 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59942.log.gz 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61835.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56304.log.gz 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58252.log 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.61835.log.gz -5 2024-02-19T15:05:14.069 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90570.log 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89418.log: /var/log/ceph/ceph-client.admin.58252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57392.log 88.2% 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89418.log.gz 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58252.log.gz 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61258.log 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90570.log: /var/log/ceph/ceph-client.admin.57392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51952.log 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90570.log.gz 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57392.log.gz 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51910.log 2024-02-19T15:05:14.070 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61258.log.gz 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55397.log 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88458.log 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51952.log.gz 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51910.log: /var/log/ceph/ceph-client.admin.55397.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51910.log.gz 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90654.log 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55397.log.gz 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72019.log 2024-02-19T15:05:14.071 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88458.log: /var/log/ceph/ceph-client.admin.90654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75918.log 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88458.log.gz 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90654.log.gz 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88644.log 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75918.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72019.log.gz 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56616.log 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84900.log 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88644.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75918.log.gz 2024-02-19T15:05:14.072 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56616.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.69918.log -- replaced with /var/log/ceph/ceph-client.admin.88644.log.gz 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56616.log.gz 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64968.log 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84900.log: /var/log/ceph/ceph-client.admin.69918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61793.log 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84900.log.gz 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64968.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69918.log.gz 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67970.log 2024-02-19T15:05:14.073 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64968.log.gz 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55002.log 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69832.log 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61793.log.gz 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67970.log: /var/log/ceph/ceph-client.admin.55002.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.67970.log.gz --verbose 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.55915.log 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55002.log.gz 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45738.log 2024-02-19T15:05:14.074 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69832.log: /var/log/ceph/ceph-client.admin.55915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75563.log 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69832.log.gz 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45738.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55915.log.gz 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37979.log 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45738.log.gz 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46289.log 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37979.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75563.log.gz 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46289.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.37979.log.gz 2024-02-19T15:05:14.075 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51117.log 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46289.log.gz 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60670.log 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50224.log: /var/log/ceph/ceph-client.admin.51117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77436.log 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51117.log.gz 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55969.log 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77436.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60670.log.gz 2024-02-19T15:05:14.076 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61921.log 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59347.log 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55969.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77436.log.gz 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61174.log 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55969.log.gz 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46074.log 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59347.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61921.log.gz 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61174.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59347.log.gz 2024-02-19T15:05:14.077 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56220.log 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83437.log 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46074.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61174.log.gz 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56220.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46074.log.gz 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83148.log 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83437.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56220.log.gz 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64776.log 2024-02-19T15:05:14.078 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83437.log.gz 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66050.log 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64776.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.47814.log 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83148.log.gz 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66050.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64776.log.gz 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78743.log 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66050.log.gz 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78533.log 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78743.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47814.log.gz 2024-02-19T15:05:14.079 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86448.log 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78743.log.gz 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45492.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.78533.log.gz 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64173.log 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85068.log 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr:gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.86448.log.gz --verbose 2024-02-19T15:05:14.080 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.78448.log 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64173.log: /var/log/ceph/ceph-client.admin.85068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71083.log 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64173.log.gz 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78448.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85068.log.gz 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53628.log 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78448.log.gz 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86868.log 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56388.log 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53628.log: /var/log/ceph/ceph-client.admin.86868.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53628.log.gz 2024-02-19T15:05:14.081 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61426.log 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71083.log.gz 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56388.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86868.log.gz 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73117.log 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56388.log.gz 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58011.log 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61426.log: /var/log/ceph/ceph-client.admin.73117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84146.log 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61426.log.gz 2024-02-19T15:05:14.082 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73117.log.gz 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61048.log 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64352.log 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84146.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58011.log.gz 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89502.log 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84146.log.gz 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64352.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61048.log.gz 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2024-02-19T15:05:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88332.log 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64352.log.gz 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89502.log.gz 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57434.log 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88812.log 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88332.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90486.log 2024-02-19T15:05:14.084 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88332.log.gz 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88812.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57434.log.gz 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47334.log 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6%gzip/var/log/ceph/ceph-client.admin.90486.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.79450.log 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88812.log.gz 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47334.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90486.log.gz 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68711.log 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43672.log 2024-02-19T15:05:14.085 INFO:teuthology.orchestra.run.smithi078.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47334.log.gz 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72949.log 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79450.log.gz 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68711.log.gz 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70933.log 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76165.log 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43672.log.gz 2024-02-19T15:05:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72949.log: /var/log/ceph/ceph-client.admin.70933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46619.log 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72949.log.gz 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84564.log 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76165.log.gz 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72430.log 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.70933.log.gz 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65245.log 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84564.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46619.log.gz 2024-02-19T15:05:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72430.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84564.log.gz 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.62356.log 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72430.log.gz 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88248.log 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65245.log.gz 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47376.log 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78575.log 2024-02-19T15:05:14.088 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62356.log.gz 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88248.log.gz 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51219.log 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83773.log 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78575.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47376.log.gz 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87390.log 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78575.log.gz 2024-02-19T15:05:14.089 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71705.log 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83773.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51219.log.gz 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82349.log 87.5% 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83773.log.gz 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75876.log 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87390.log.gz 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65840.log 2024-02-19T15:05:14.090 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71705.log.gz 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82349.log.gz 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50873.log 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66134.log 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65840.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75876.log.gz 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59122.log 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65840.log.gz 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66134.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50873.log.gz 2024-02-19T15:05:14.091 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75041.log 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76782.log 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66134.log.gz 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75041.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.44466.log 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59122.log.gz 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75041.log.gz 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90360.log 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44466.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76782.log.gz 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85539.log 2024-02-19T15:05:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69602.log 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.44466.log.gz 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85539.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43495.log 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90360.log.gz 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82749.log 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69602.log.gz 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53064.log 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.85539.log.gz 2024-02-19T15:05:14.093 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.82749.log: -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43495.log.gz 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91129.log 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53064.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.82749.log.gz 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65052.log 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53064.log.gz 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70086.log 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91129.log.gz 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65966.log 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51557.log 2024-02-19T15:05:14.094 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.65052.log.gz 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65966.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70086.log.gzgzip 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51431.log 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.51557.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53148.log 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65966.log.gz 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51431.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51557.log.gz 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47772.log 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51431.log.gz 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84690.log 2024-02-19T15:05:14.095 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53148.log.gz 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72350.log 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47772.log: /var/log/ceph/ceph-client.admin.84690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62506.log 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47772.log.gz 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84690.log.gz 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73890.log 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63258.log 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62506.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72350.log.gz 2024-02-19T15:05:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56346.log 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62506.log.gz 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73890.log.gz 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53670.log 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82433.log 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63258.log.gz 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56346.log.gz 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53670.log.gz 2024-02-19T15:05:14.097 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53953.log 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44128.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82433.log.gz 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52186.log 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.53953.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.72861.log 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8%/var/log/ceph/ceph-client.admin.52186.log: -- replaced with /var/log/ceph/ceph-client.admin.53953.log.gz 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57803.log 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47688.log 2024-02-19T15:05:14.098 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72861.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52186.log.gz 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58826.log 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72861.log.gz 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57803.log.gz 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49183.log 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89022.log 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58826.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47688.log.gz 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70614.log 2024-02-19T15:05:14.099 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58826.log.gz 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89022.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49183.log.gz 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81800.log 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89022.log.gz 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76245.log 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81800.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.70614.log.gz /var/log/ceph/ceph-client.admin.70170.log 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78244.log 2024-02-19T15:05:14.100 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81800.log.gz 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74621.log 89.3% 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76245.log.gz 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44948.log 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78244.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70170.log.gz 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54474.log 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78244.log.gz 2024-02-19T15:05:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55607.log 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74621.log.gz 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46628.log 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64310.log 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54474.log.gz 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55607.log.gz 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.46628.log: -- /var/log/ceph/ceph-client.admin.48679.log 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64310.log: gzip -5 --verbose 88.0% -- /var/log/ceph/ceph-client.admin.78491.log 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46628.log.gz 2024-02-19T15:05:14.102 INFO:teuthology.orchestra.run.smithi078.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.44948.log.gz 87.8% 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64310.log.gz 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85152.log 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61709.log 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78491.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48679.log.gz 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86784.log 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78491.log.gz 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68543.log 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85152.log.gz 2024-02-19T15:05:14.103 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86784.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61709.log.gz 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91381.log 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60628.log 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86784.log.gz 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91381.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68543.log.gz 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44424.log 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91381.log.gz 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66218.log 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72145.log 2024-02-19T15:05:14.104 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44424.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60628.log.gz 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86037.log 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44424.log.gz 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66218.log.gz 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67232.log 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89064.log 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86037.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72145.log.gz 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2024-02-19T15:05:14.105 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86037.log.gz 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89064.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67232.log.gz 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62314.log 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89064.log.gz 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60362.log 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66713.log 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62314.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2024-02-19T15:05:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60362.log: gzip -5 --verbose 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62314.log.gz 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.71849.log 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66713.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60362.log.gz 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81109.log 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66713.log.gz 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54600.log 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71849.log: /var/log/ceph/ceph-client.admin.81109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84816.log 2024-02-19T15:05:14.107 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71849.log.gz 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81109.log.gz 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75834.log 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45140.log 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84816.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54600.log.gz 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76369.log 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84816.log.gz 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75834.log.gz 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90798.log 2024-02-19T15:05:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76369.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45140.log.gz 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90798.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76369.log.gz 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46622.log 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mgr.y.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90798.log.gz 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64129.log 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60712.log 2024-02-19T15:05:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46622.log: /var/log/ceph/ceph-client.admin.64129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68356.log 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46622.log.gz 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60712.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64129.log.gz 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67051.log 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68356.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60712.log.gz 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45864.log 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67051.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68356.log.gz 2024-02-19T15:05:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82298.log 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67051.log.gz 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83689.log 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45864.log: /var/log/ceph/ceph-client.admin.82298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61627.log 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45864.log.gz 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82298.log.gz 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45182.log 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83689.log.gz 2024-02-19T15:05:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89148.log/var/log/ceph/ceph-client.admin.61627.log: 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45182.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61627.log.gz 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74392.log 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89148.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45182.log.gz 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49057.log 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89148.log.gz 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74705.log 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74392.log: /var/log/ceph/ceph-client.admin.49057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88854.log 2024-02-19T15:05:14.112 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0%/var/log/ceph/ceph-client.admin.74705.log: -- replaced with /var/log/ceph/ceph-client.admin.49057.log.gz 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52228.log 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74392.log.gz 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74705.log.gz 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54121.log 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52228.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88854.log.gz 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47418.log 2024-02-19T15:05:14.113 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54121.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52228.log.gz 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57224.log 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54121.log.gz 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47418.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47418.log.gz 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87870.log 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88080.log 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87870.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57224.log.gz 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76086.log 2024-02-19T15:05:14.114 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87870.log.gz 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44082.log 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88080.log.gz 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63072.log 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76086.log.gz 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67484.log 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.63072.log: -5 --verbose 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44082.log.gz 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.85731.log 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67484.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63072.log.gz 2024-02-19T15:05:14.115 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58053.log 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67484.log.gz 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88560.log 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85731.log: /var/log/ceph/ceph-client.admin.58053.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85731.log.gz 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.92653.log 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88560.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58053.log.gz 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68501.log 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88560.log.gz 2024-02-19T15:05:14.116 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.92653.log: --verbose -- /var/log/ceph/ceph-client.admin.80290.log 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68501.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92653.log.gz 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83983.log 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68501.log.gz 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47856.log 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80290.log: /var/log/ceph/ceph-client.admin.83983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77184.log 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6%/var/log/ceph/ceph-client.admin.47856.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.83983.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.85110.log 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.117 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80290.log.gz 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77184.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47856.log.gz 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50098.log 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85110.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77184.log.gz 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90402.log 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85110.log.gz 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90150.log 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90402.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50098.log.gz 2024-02-19T15:05:14.118 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55751.log 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90150.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90402.log.gz 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77394.log 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90150.log.gz 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50686.log 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77394.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55751.log.gz 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.67316.log 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50686.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77394.log.gz 2024-02-19T15:05:14.119 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83899.log 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50686.log.gz 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86266.log 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83899.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67316.log.gz 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87786.log 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86266.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83899.log.gz 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69644.log 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86266.log.gz 2024-02-19T15:05:14.120 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78701.log 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69644.log: gzip 87.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.47898.log 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87786.log.gz 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78701.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69644.log.gz 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79912.log 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78701.log.gz 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87474.log 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47898.log: /var/log/ceph/ceph-client.admin.79912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87138.log 2024-02-19T15:05:14.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63132.log 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.47898.log.gz 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5%/var/log/ceph/ceph-client.admin.87138.log: -- replaced with /var/log/ceph/ceph-client.admin.87474.log.gz 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79912.log.gz 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87138.log.gz 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56136.log 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67628.log 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63132.log: /var/log/ceph/ceph-client.admin.56136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87348.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63132.log.gz 2024-02-19T15:05:14.122 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67628.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56136.log.gz 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76950.log 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67628.log.gz 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.87348.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47730.log 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76950.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87348.log.gz 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58466.log 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76950.log.gz 2024-02-19T15:05:14.123 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87684.log 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.58466.log: --verbose -- /var/log/ceph/ceph-client.admin.57644.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47730.log.gz 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87684.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84858.log 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58466.log.gz 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87684.log.gz 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54302.log 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84858.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57644.log.gz 2024-02-19T15:05:14.124 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.85416.log -- replaced with /var/log/ceph/ceph-client.admin.84858.log.gz 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72680.log 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54302.log.gz/var/log/ceph/ceph-client.admin.85416.log: 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70338.log 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85416.log.gz 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54792.log 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.70338.log: -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72680.log.gz 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.79492.log 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54792.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70338.log.gz 2024-02-19T15:05:14.125 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88938.log 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54792.log.gz 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80880.log 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79492.log.gz 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76694.log 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88938.log.gz 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64608.log 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr:gzip/var/log/ceph/ceph-client.admin.76694.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54165.log 2024-02-19T15:05:14.126 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80104.log 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.80880.log.gz 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76694.log.gz 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73075.log 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64608.log.gz 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54165.log: /var/log/ceph/ceph-client.admin.80104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72271.log 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54165.log.gz 2024-02-19T15:05:14.127 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75750.log 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80104.log.gz 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73075.log: /var/log/ceph/ceph-client.admin.72271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56514.log 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75750.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72271.log.gz 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77622.log 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73075.log.gz 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56514.log: gzip -5 --verbose -- 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75750.log.gz 2024-02-19T15:05:14.128 INFO:teuthology.orchestra.run.smithi078.stderr: /var/log/ceph/ceph-client.admin.80814.log 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77622.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56514.log.gz 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64003.log 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77622.log.gz 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68982.log 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47208.log 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.80814.log.gz 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64003.log.gz 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68982.log: 2024-02-19T15:05:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57518.log 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68982.log.gz 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47208.log: /var/log/ceph/ceph-client.admin.57518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75262.log 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1%/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51994.log 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57518.log.gz 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47208.log.gz 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55044.log 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68922.log 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75262.log.gz 2024-02-19T15:05:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55044.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51994.log.gz 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70464.log 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75346.log 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55044.log.gz 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68922.log.gz/var/log/ceph/ceph-client.admin.70464.log: 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73033.log 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56052.log 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70464.log.gz 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73033.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75346.log.gz 2024-02-19T15:05:14.131 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53544.log 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73033.log.gz 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.56052.log: -- /var/log/ceph/ceph-client.admin.59301.log 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53544.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91087.log 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56052.log.gz 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53544.log.gz 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51473.log 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91087.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59301.log.gz 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71495.log 2024-02-19T15:05:14.132 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91087.log.gz 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90919.log 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51473.log: /var/log/ceph/ceph-client.admin.71495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61090.log 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90919.log: 88.5% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71495.log.gz 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51473.log.gz 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90919.log.gz 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64692.log 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59984.log 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61090.log.gz 2024-02-19T15:05:14.133 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54037.log 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64692.log.gz 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64524.log 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59984.log.gz 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57476.log 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92571.log 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64524.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54037.log.gz 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57476.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64524.log.gz 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57476.log.gz 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75083.log 2024-02-19T15:05:14.134 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50308.log 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8%/var/log/ceph/ceph-client.admin.75083.log: -- replaced with /var/log/ceph/ceph-client.admin.92571.log.gz 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81716.log 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75083.log.gz 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63300.log 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91671.log 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81716.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50308.log.gz 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63300.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81716.log.gz 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63300.log.gz 2024-02-19T15:05:14.135 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84392.log 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87726.log 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91671.log.gz 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49225.log 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84392.log.gz/var/log/ceph/ceph-client.admin.87726.log: 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92091.log 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87726.log.gz 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89106.log 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49225.log: /var/log/ceph/ceph-client.admin.92091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80706.log 2024-02-19T15:05:14.136 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89106.log: 87.7% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49225.log.gz 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92091.log.gz 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89106.log.gz 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74873.log 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.80706.log: --verbose -- /var/log/ceph/ceph-client.admin.43591.log 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59431.log 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74873.log.gz 91.0% -- replaced with /var/log/ceph/ceph-client.admin.80706.log.gz 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43591.log: gzip -5 89.2% --verbose -- /var/log/ceph/ceph-client.admin.47628.log 2024-02-19T15:05:14.137 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43591.log.gz 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48595.log 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6%/var/log/ceph/ceph-client.admin.47628.log: -- replaced with /var/log/ceph/ceph-client.admin.59431.log.gz 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59217.log 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47628.log.gz 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79870.log 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48595.log: /var/log/ceph/ceph-client.admin.59217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44256.log 2024-02-19T15:05:14.138 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79870.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48595.log.gz 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59217.log.gz 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79870.log.gz 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62717.log 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86700.log 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62717.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44256.log.gz 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44172.log 2024-02-19T15:05:14.139 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62717.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50560.log 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86700.log: 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80248.log 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86700.log.gz 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44172.log: /var/log/ceph/ceph-client.admin.50560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54918.log 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44172.log.gz 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80248.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50560.log.gz 2024-02-19T15:05:14.140 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56677.log 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80248.log.gz 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.54918.log: --verbose -- /var/log/ceph/ceph-client.admin.48847.log 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56677.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54918.log.gz 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56677.log.gz/var/log/ceph/ceph-client.admin.48847.log: 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79618.log 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92217.log 2024-02-19T15:05:14.141 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48847.log.gz 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64734.log 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79618.log.gz 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91003.log 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92217.log.gz 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64734.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64734.log.gz 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87642.log 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49267.log 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91003.log.gz 2024-02-19T15:05:14.142 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89712.log 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87642.log.gz 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53208.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49267.log.gz 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87306.log 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89712.log.gz 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53586.log 2024-02-19T15:05:14.143 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53208.log.gz 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71935.log 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53586.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87306.log.gz 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.2.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53586.log.gz 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90192.log 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81632.log 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71935.log.gz 2024-02-19T15:05:14.144 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90192.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90192.log.gz 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86658.log 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66837.log 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81632.log.gz/var/log/ceph/ceph-client.admin.86658.log: 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0%/var/log/ceph/ceph-client.admin.66837.log: -- replaced with /var/log/ceph/ceph-client.admin.86658.log.gz 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66837.log.gz 2024-02-19T15:05:14.145 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45780.log 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64926.log 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.45780.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45780.log.gz 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81277.log 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64926.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64926.log.gz 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72554.log 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81277.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81277.log.gz 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73435.log 2024-02-19T15:05:14.146 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72554.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72554.log.gz 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46629.log 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73435.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73435.log.gz 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57350.log 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46629.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46629.log.gz 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87954.log 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57350.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57350.log.gz 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56762.log 2024-02-19T15:05:14.147 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87954.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87954.log.gz 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89796.log 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.56762.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56762.log.gz 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91171.log 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89796.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89796.log.gz 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55481.log 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91171.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91171.log.gz 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79152.log 2024-02-19T15:05:14.148 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55481.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55481.log.gz 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67904.log 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79152.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79152.log.gz 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52120.log 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67904.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67904.log.gz 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85212.log 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52120.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.52120.log.gz 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49352.log 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85212.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.85212.log.gz 2024-02-19T15:05:14.149 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54344.log 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49352.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49352.log.gz 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54344.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54344.log.gz 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81674.log 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53250.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74957.log 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81674.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81674.log.gz 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90528.log 2024-02-19T15:05:14.150 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74957.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74957.log.gz 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79408.log 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90528.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90528.log.gz 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62904.log 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79408.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79408.log.gz 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65672.log 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62904.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62904.log.gz 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58382.log 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65672.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65672.log.gz 2024-02-19T15:05:14.151 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59515.log 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58382.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58382.log.gz 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59166.log 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59515.log.gz 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46116.log 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59166.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59166.log.gz 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65442.log 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46116.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46116.log.gz 2024-02-19T15:05:14.152 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83353.log 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65442.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.65442.log.gz 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53871.log 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.83353.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83353.log.gz 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49852.log 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53871.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53871.log.gz 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91755.log 2024-02-19T15:05:14.153 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49852.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49852.log.gz 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87222.log 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91755.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91755.log.gz 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58634.log 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87222.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87222.log.gz 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80188.log 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58634.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.58634.log.gz 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78286.log 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80188.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80188.log.gz 2024-02-19T15:05:14.154 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57014.log 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.78286.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78286.log.gz 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76002.log 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57014.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57014.log.gz 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70422.log 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76002.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76002.log.gz 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46288.log 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.70422.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70422.log.gz 2024-02-19T15:05:14.155 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81758.log 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46288.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46288.log.gz 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63426.log 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81758.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81758.log.gz 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87180.log 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63426.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63426.log.gz 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73519.log 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87180.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87180.log.gz 2024-02-19T15:05:14.156 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86121.log 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73519.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73519.log.gz 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85689.log 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86121.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86121.log.gz 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86994.log 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85689.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85689.log.gz 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61751.log 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86994.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86994.log.gz 2024-02-19T15:05:14.157 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48805.log 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61751.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61751.log.gz 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67007.log 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48805.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48805.log.gz 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53995.log 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67007.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67007.log.gz 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47460.log 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.53995.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53995.log.gz 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60152.log 2024-02-19T15:05:14.158 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47460.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47460.log.gz 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92007.log 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.60152.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60152.log.gz 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81025.log 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92007.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92007.log.gz 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85773.log 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81025.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81025.log.gz 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79324.log 2024-02-19T15:05:14.159 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85773.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85773.log.gz 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88164.log 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79324.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79324.log.gz 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69193.log 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88164.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88164.log.gz 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44031.log 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69193.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69193.log.gz 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77717.log 2024-02-19T15:05:14.160 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44031.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44031.log.gz 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49141.log 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77717.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77717.log.gz 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92385.log 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49141.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49141.log.gz 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49936.log 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.92385.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92385.log.gz 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71266.log 2024-02-19T15:05:14.161 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49936.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49936.log.gz 2024-02-19T15:05:14.162 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72061.log 2024-02-19T15:05:14.162 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71266.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.71266.log.gz 2024-02-19T15:05:14.162 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89190.log 2024-02-19T15:05:14.162 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72061.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72061.log.gz 2024-02-19T15:05:14.162 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59473.log 2024-02-19T15:05:14.162 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89190.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89190.log.gz 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73393.log 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.59473.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59473.log.gz 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77871.log 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73393.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73393.log.gz 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68753.log 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77871.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77871.log.gz 2024-02-19T15:05:14.163 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66554.log 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.68753.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68753.log.gz 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44532.log 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66554.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66554.log.gz 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73563.log 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.44532.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44532.log.gz 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71178.log 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73563.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73563.log.gz 2024-02-19T15:05:14.164 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75960.log 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.71178.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71178.log.gz 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77763.log 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75960.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75960.log.gz 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69788.log 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77763.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77763.log.gz 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72512.log 2024-02-19T15:05:14.165 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69788.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69788.log.gz 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65924.log 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.72512.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72512.log.gz 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49531.log 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.65924.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65924.log.gz 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89754.log 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49531.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49531.log.gz 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81403.log 2024-02-19T15:05:14.166 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.89754.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89754.log.gz 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75605.log 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.81403.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81403.log.gz 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75605.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75605.log.gz 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62058.log 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.48973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47250.log 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62058.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62058.log.gz 2024-02-19T15:05:14.167 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64566.log 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.47250.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47250.log.gz 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43775.log 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64566.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64566.log.gz 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64217.log 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43775.log.gz 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.64217.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64217.log.gz 2024-02-19T15:05:14.168 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86532.log 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.46626.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85026.log 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86532.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86532.log.gz 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90612.log 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85026.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85026.log.gz 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87432.log 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.90612.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90612.log.gz 2024-02-19T15:05:14.169 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67862.log 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87432.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87432.log.gz 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54260.log 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67862.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67862.log.gz 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49310.log 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.54260.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54260.log.gz 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87516.log 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49310.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49310.log.gz 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67102.log 2024-02-19T15:05:14.170 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87516.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87516.log.gz 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58097.log 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.67102.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67102.log.gz 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82126.log 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58097.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58097.log.gz 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66260.log 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82126.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82126.log.gz 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74142.log 2024-02-19T15:05:14.171 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.66260.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66260.log.gz 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82581.log 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.74142.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74142.log.gz 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76908.log 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.82581.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82581.log.gz 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69440.log 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76908.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76908.log.gz 2024-02-19T15:05:14.172 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76866.log 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.69440.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69440.log.gz 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63216.log 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.76866.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76866.log.gz 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77142.log 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.63216.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63216.log.gz 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86952.log 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.77142.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77142.log.gz 2024-02-19T15:05:14.173 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85605.log 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.86952.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86952.log.gz 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73742.log 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.85605.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85605.log.gz 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.73742.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73742.log.gz 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80772.log 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62440.log 2024-02-19T15:05:14.174 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.80772.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80772.log.gz 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50518.log 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.62440.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.62440.log.gz 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75392.log 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50518.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50518.log.gz 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50915.log 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.75392.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75392.log.gz 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50182.log 2024-02-19T15:05:14.175 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50915.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50915.log.gz 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61216.log 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50182.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50182.log.gz 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50476.log 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.61216.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61216.log.gz 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57140.log 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.50476.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50476.log.gz 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51868.log 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.57140.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57140.log.gz 2024-02-19T15:05:14.176 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87096.log 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.51868.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51868.log.gz 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88770.log 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.87096.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87096.log.gz 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91045.log 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.88770.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88770.log.gz 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52078.log 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91045.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91045.log.gz 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43403.log 2024-02-19T15:05:14.177 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.52078.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52078.log.gz 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84606.log 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.43403.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43403.log.gz 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55691.log 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.84606.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84606.log.gz 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.55691.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55691.log.gz 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91923.log 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.49446.log: 90.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-02-19T15:05:14.178 INFO:teuthology.orchestra.run.smithi078.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79068.log 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.91923.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83605.log 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91923.log.gz 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58700.log 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.79068.log: /var/log/ceph/ceph-client.admin.83605.log: 89.3% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79068.log.gz 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83605.log.gz 2024-02-19T15:05:14.179 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.58700.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58700.log.gz 2024-02-19T15:05:14.180 INFO:teuthology.orchestra.run.smithi078.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-02-19T15:05:14.180 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-02-19T15:05:14.180 INFO:teuthology.orchestra.run.smithi150.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-02-19T15:05:14.508 INFO:teuthology.orchestra.run.smithi078.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-02-19T15:05:15.008 INFO:teuthology.orchestra.run.smithi150.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-02-19T15:05:15.092 INFO:teuthology.orchestra.run.smithi150.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-02-19T15:05:15.119 INFO:teuthology.orchestra.run.smithi150.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-02-19T15:05:15.137 INFO:teuthology.orchestra.run.smithi078.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-02-19T15:05:15.161 INFO:teuthology.orchestra.run.smithi150.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-02-19T15:05:15.163 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-02-19T15:05:15.163 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m1.661s 2024-02-19T15:05:15.164 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m7.160s 2024-02-19T15:05:15.164 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.252s 2024-02-19T15:05:15.232 INFO:teuthology.orchestra.run.smithi078.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-02-19T15:05:15.396 INFO:teuthology.orchestra.run.smithi078.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-02-19T15:05:15.470 INFO:teuthology.orchestra.run.smithi078.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-02-19T15:05:15.472 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-02-19T15:05:15.472 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m2.000s 2024-02-19T15:05:15.472 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m8.029s 2024-02-19T15:05:15.472 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.870s 2024-02-19T15:05:15.472 INFO:tasks.ceph:Archiving logs... 2024-02-19T15:05:15.473 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/remote/smithi078/log 2024-02-19T15:05:15.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T15:05:17.537 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/remote/smithi150/log 2024-02-19T15:05:17.538 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T15:05:18.566 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T15:05:18.575 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T15:05:18.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T15:05:18.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T15:05:18.667 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T15:05:18.676 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T15:05:18.677 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T15:05:18.679 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T15:05:18.693 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-02-19T15:05:18.695 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-19T15:05:18.698 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T15:05:18.698 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-02-19T15:05:18.699 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1586us[+1586us] +/- 105ms 2024-02-19T15:05:18.699 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 +197us[ +212us] +/- 52ms 2024-02-19T15:05:18.699 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -1153us[-1153us] +/- 71ms 2024-02-19T15:05:18.699 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T15:05:18.700 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T15:05:18.701 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-19T15:05:18.701 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1539us[+1525us] +/- 105ms 2024-02-19T15:05:18.701 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 +150us[ +136us] +/- 52ms 2024-02-19T15:05:18.701 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -1194us[-1208us] +/- 71ms 2024-02-19T15:05:18.701 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T15:05:18.701 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T15:05:18.710 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T15:05:18.710 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T15:05:18.719 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T15:05:18.838 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T15:05:18.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T15:05:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:05:18.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T15:05:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:05:18.977 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T15:05:18.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T15:05:18.998 INFO:teuthology.task.internal:Duration was 1259.555099 seconds 2024-02-19T15:05:18.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T15:05:19.008 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T15:05:19.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T15:05:19.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T15:05:19.054 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T15:05:19.063 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T15:05:19.395 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T15:05:19.395 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-02-19T15:05:19.396 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T15:05:19.417 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-02-19T15:05:19.417 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T15:05:19.443 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T15:05:19.443 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T15:05:19.460 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T15:05:19.512 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T15:05:19.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T15:05:19.530 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T15:05:19.668 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T15:05:19.678 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T15:05:19.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T15:05:19.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T15:05:19.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T15:05:19.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T15:05:19.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T15:05:19.769 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-02-19T15:05:19.797 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-02-19T15:05:19.813 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T15:05:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:05:19.839 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T15:05:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T15:05:19.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T15:05:19.877 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T15:05:19.878 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/remote/smithi078 2024-02-19T15:05:19.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T15:05:19.918 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468/remote/smithi150 2024-02-19T15:05:19.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T15:05:19.953 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T15:05:19.953 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T15:05:19.956 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T15:05:20.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T15:05:20.014 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T15:05:20.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T15:05:20.024 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T15:05:20.024 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T15:05:20.027 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T15:05:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 15:05 /home/ubuntu/cephtest 2024-02-19T15:05:20.057 INFO:teuthology.orchestra.run.smithi150.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 15:05 /home/ubuntu/cephtest 2024-02-19T15:05:20.058 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T15:05:20.068 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T15:05:20.086 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T15:05:20.112 DEBUG:teuthology.nuke:shortname: smithi078 2024-02-19T15:05:20.113 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T15:05:20.133 DEBUG:teuthology.nuke:shortname: smithi150 2024-02-19T15:05:20.133 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T15:05:20.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 14:34:19.578089', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEFJ716bM4kXdY3SWaq4E/9AKO2aR9oARi5WQKiL9KctLMdfkOuM0QpNZUyOUbjV2DvVftK/40LaKvuq62Mw7tA='} 2024-02-19T15:05:20.159 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-19_14:33:01-fs-wip-lusov-qdb-distro-default-smithi/7566468', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 14:34:19.577033', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2zy5WOe4j8G1rIhiUDwy2DrvGN7z8LqJEVTU1UzoJPp1BXF97WPZD0W3wQVGOZ1sfaE4HPcOT5L0Db8pL1jFQ='} 2024-02-19T15:05:20.187 INFO:teuthology.orchestra.console.smithi078:Power off 2024-02-19T15:05:20.187 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T15:05:20.213 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-02-19T15:05:20.213 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:20.229 INFO:teuthology.orchestra.console.smithi150:Power off 2024-02-19T15:05:20.229 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T15:05:20.241 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-02-19T15:05:20.254 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-02-19T15:05:20.254 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:20.278 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-19T15:05:24.242 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:24.279 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:24.353 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-02-19T15:05:24.389 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-19T15:05:28.357 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:28.390 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:28.468 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-02-19T15:05:28.500 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-02-19T15:05:28.600 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-02-19T15:05:32.471 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T15:05:32.580 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-02-19T15:05:32.681 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-02-19T15:05:32.785 INFO:teuthology.run:Summary data: description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} duration: 1259.5550985336304 failure_reason: 'Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=26d011ce065f481687ca22f81383b9ff status: fail success: false 2024-02-19T15:05:32.785 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T15:05:32.879 INFO:teuthology.run:FAIL